From general-return-2133-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 01 15:07:28 2010
Return-Path: <general-return-2133-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98655 invoked from network); 1 Oct 2010 15:07:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Oct 2010 15:07:28 -0000
Received: (qmail 25490 invoked by uid 500); 1 Oct 2010 15:07:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25064 invoked by uid 500); 1 Oct 2010 15:07:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25056 invoked by uid 99); 1 Oct 2010 15:07:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Oct 2010 15:07:22 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of marcoscba@gmail.com designates 74.125.82.42 as permitted sender)
Received: from [74.125.82.42] (HELO mail-ww0-f42.google.com) (74.125.82.42)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Oct 2010 15:07:14 +0000
Received: by wwb13 with SMTP id 13so490553wwb.5
        for <general@hadoop.apache.org>; Fri, 01 Oct 2010 08:06:54 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=LLeRkFnW2xmWuQ0PKv0hSNY+S4lPpRMOVHh8Holkf2c=;
        b=MMjKqWH/CSIwYynGokX5798bCBV4zcc4VYqMlnAblxbUEGsKUi0gEsaFiuNQjppzcW
         YgbPrM/daMEIdm9/pA13qrErOjjb6mKaZG0P1fO/g6/tzzk35k9tuvaVJZtT5sc1Irpa
         89bCoAhXQasIV4KM4ESHDHQaOx1i5xHAsjCi0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=WT1fOPKv4AmzJWfv/gDcl/Q4EuUsvmz/sd/IhaonFDZbzmaPVcupp36cYnLM6k6kHq
         v4OlGa7dZf2SXaFo9gTChhkijt8VdsJfEQ+HhSqie3Bc/91fnc0cc/Wi+KZO6uxYmmOP
         TzeLLy2FavKDX1db/OreJ+1O/u5qXs8b0M0xg=
MIME-Version: 1.0
Received: by 10.216.153.140 with SMTP id f12mr2211173wek.111.1285945614642;
 Fri, 01 Oct 2010 08:06:54 -0700 (PDT)
Received: by 10.216.51.145 with HTTP; Fri, 1 Oct 2010 08:06:54 -0700 (PDT)
Date: Fri, 1 Oct 2010 12:06:54 -0300
Message-ID: <AANLkTin6ZUrWYUscAXmq0bOG8mWA35xcAfvLD-jYVzwu@mail.gmail.com>
Subject: Is it possible write directly to datanode's?
From: Marcos Pinto <marcoscba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e65b54dc8fb24c04918f8bcf
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e65b54dc8fb24c04918f8bcf
Content-Type: text/plain; charset=ISO-8859-1

Hi guys, I gotta a question and I really would appreciate some help,
suggestion, or advices.
Is it possible write directly to the datanode's and eliminate the bottleneck
to write into HDFS?
for example: I gotta 40 writers and I writes th data direct to namenode, i
was wondering if it would become a bottleneck so I would like to know
if it is possible to write the data direct to the datanode's.
I've read that Hadoop doesnt support multiple writes so how can we deal with
big cluster? I mean I am really going to have this bottleneck???

Thanks in advanced.

--0016e65b54dc8fb24c04918f8bcf--

From general-return-2134-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 02 04:28:00 2010
Return-Path: <general-return-2134-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94628 invoked from network); 2 Oct 2010 04:28:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Oct 2010 04:28:00 -0000
Received: (qmail 2672 invoked by uid 500); 2 Oct 2010 04:27:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2122 invoked by uid 500); 2 Oct 2010 04:27:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2114 invoked by uid 99); 2 Oct 2010 04:27:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 04:27:53 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of rmalviya@apple.com designates 17.254.13.23 as permitted sender)
Received: from [17.254.13.23] (HELO mail-out4.apple.com) (17.254.13.23)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 04:27:44 +0000
Received: from relay16.apple.com (relay16.apple.com [17.128.113.55])
	by mail-out4.apple.com (Postfix) with ESMTP id CBEB9B33BCBA
	for <general@hadoop.apache.org>; Fri,  1 Oct 2010 21:27:23 -0700 (PDT)
X-AuditID: 11807137-b7b43ae00000547d-60-4ca6b4aa26f9
Received: from [17.151.95.191] (Unknown_Domain [17.151.95.191])
	(using TLS with cipher AES128-SHA (AES128-SHA/128 bits))
	(Client did not present a certificate)
	by relay16.apple.com (Apple SCV relay) with SMTP id 39.69.21629.BA4B6AC4; Fri,  1 Oct 2010 21:27:23 -0700 (PDT)
From: rahul <rmalviya@apple.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Date: Fri, 1 Oct 2010 21:27:22 -0700
Subject: Total Space Available on Hadoop Cluster Or Hadoop version of "df".
To: general@hadoop.apache.org
Message-Id: <1C4C43FE-05F2-464B-9F04-AC972988C326@apple.com>
Mime-Version: 1.0 (Apple Message framework v1081)
X-Mailer: Apple Mail (2.1081)
X-Brightmail-Tracker: AAAAAA==

Hi,

I am using Hadoop 0.20.2 version for data processing by setting up =
Hadoop Cluster on two nodes.=20

And I am continuously adding more space to the nodes.

Can some body let me know how to get the total space available on the =
hadoop cluster using command line.

or=20

Hadoop version "df", Unix command.

Any input is helpful.

Thanks
Rahul=

From general-return-2135-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 02 06:14:59 2010
Return-Path: <general-return-2135-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12845 invoked from network); 2 Oct 2010 06:14:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Oct 2010 06:14:59 -0000
Received: (qmail 43982 invoked by uid 500); 2 Oct 2010 06:14:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 43504 invoked by uid 500); 2 Oct 2010 06:14:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 43496 invoked by uid 99); 2 Oct 2010 06:14:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 06:14:53 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of glenn.gore@melbourneit.com.au designates 203.147.157.18 as permitted sender)
Received: from [203.147.157.18] (HELO bne3-0002mri.server-mail.com) (203.147.157.18)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 06:14:46 +0000
Received: from wic002mz.server-mail.com (wic002mz.server-mail.com [210.247.173.2])
	by bne3-0002mri.server-mail.com (Postfix) with ESMTP id A9B88118009
	for <general@hadoop.apache.org>; Sat,  2 Oct 2010 16:14:21 +1000 (EST)
Received: from wic002mitef.messaging.mit ([172.31.0.12])
	by wic002mz.server-mail.com with -
	id DiE91f0070FY6JL3NiEM2e; Sat, 02 Oct 2010 16:14:21 +1000
Received: from WIC001MITEBCLV1.messaging.mit ([172.31.0.20]) by wic002mitef.messaging.mit with Microsoft SMTPSVC(6.0.3790.3959);
	 Sat, 2 Oct 2010 16:14:09 +1000
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/mixed;
	boundary="----_=_NextPart_001_01CB61F8.F74C3C33"
Subject: RE: Total Space Available on Hadoop Cluster Or Hadoop version of "df".
Date: Sat, 2 Oct 2010 16:13:18 +1000
Message-ID: <C586A8404F8B304E807B4B8BF73B376EA600AC@WIC001MITEBCLV1.messaging.mit>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: <C586A8404F8B304E807B4B8BF73B376EA600AC@WIC001MITEBCLV1.messaging.mit>
Thread-Topic: Total Space Available on Hadoop Cluster Or Hadoop version of "df".
Thread-Index: Acth6jNxZ+WCuCqDS0Sxho1TgErmUAADrR6s
References: <1C4C43FE-05F2-464B-9F04-AC972988C326@apple.com>
From: "Glenn Gore" <Glenn.Gore@melbourneit.com.au>
To: <general@hadoop.apache.org>
X-OriginalArrivalTime: 02 Oct 2010 06:14:09.0821 (UTC) FILETIME=[06C054D0:01CB61F9]

------_=_NextPart_001_01CB61F8.F74C3C33
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

hadoop dfsadmin -report

Regards

Glenn


-----Original Message-----
From: rahul [mailto:rmalviya@apple.com]
Sent: Sat 10/2/2010 2:27 PM
To: general@hadoop.apache.org
Subject: Total Space Available on Hadoop Cluster Or Hadoop version of =
"df".
=20
Hi,

I am using Hadoop 0.20.2 version for data processing by setting up =
Hadoop Cluster on two nodes.=20

And I am continuously adding more space to the nodes.

Can some body let me know how to get the total space available on the =
hadoop cluster using command line.

or=20

Hadoop version "df", Unix command.

Any input is helpful.

Thanks
Rahul


------_=_NextPart_001_01CB61F8.F74C3C33--

From general-return-2136-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 02 12:13:21 2010
Return-Path: <general-return-2136-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83142 invoked from network); 2 Oct 2010 12:13:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Oct 2010 12:13:21 -0000
Received: (qmail 85826 invoked by uid 500); 2 Oct 2010 12:13:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85361 invoked by uid 500); 2 Oct 2010 12:13:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85353 invoked by uid 99); 2 Oct 2010 12:13:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 12:13:16 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of palmercliff@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 12:13:10 +0000
Received: by wye20 with SMTP id 20so720732wye.35
        for <general@hadoop.apache.org>; Sat, 02 Oct 2010 05:12:49 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=g1iODifIfhHpJepOL44Y5fz2VMttuP1gkcN0H7lXpsc=;
        b=YfgIFtP9igCK1YjCoKFJXkae5XcmHBY+Kf7j4txrHyeaR4i8Y9dKzf1FWILqv7VINM
         QtcNfCauyMbQO8/d97ELxPPYG63Cc3INxUdN+7jVjDrnTY4iZmImzR3VgT6O15eGLKNL
         0yHYMWeJuLqQjS1en+j+pjv7DG+csjAqm598s=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=ems/kYOVy/gqXvld04rTCFAtxW+K4wVN+401ukhgRETroGgaOIQByF7btw4kIRrxqn
         /bBnXgFdFgLUN3V9Cye04XSnR8XRkCR33vD+mOHNdyFIcQJN/A2OBIcx+XabNIgHTFGw
         mlx4za301RNvH7b0HvSvl9BCL0FN0pRFIxdrM=
MIME-Version: 1.0
Received: by 10.216.11.205 with SMTP id 55mr5583702wex.51.1286021568974; Sat,
 02 Oct 2010 05:12:48 -0700 (PDT)
Received: by 10.216.79.15 with HTTP; Sat, 2 Oct 2010 05:12:48 -0700 (PDT)
In-Reply-To: <AANLkTin6ZUrWYUscAXmq0bOG8mWA35xcAfvLD-jYVzwu@mail.gmail.com>
References: <AANLkTin6ZUrWYUscAXmq0bOG8mWA35xcAfvLD-jYVzwu@mail.gmail.com>
Date: Sat, 2 Oct 2010 08:12:48 -0400
Message-ID: <AANLkTimtEM4yzQSC5F16j4r0eqd0MeDb+7FF3FT1v3Yk@mail.gmail.com>
Subject: Re: Is it possible write directly to datanode's?
From: cliff palmer <palmercliff@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364d2cbdcad04a0491a13add

--0016364d2cbdcad04a0491a13add
Content-Type: text/plain; charset=ISO-8859-1

Marcos, NFS will let you write almost anywhere.....

On Fri, Oct 1, 2010 at 11:06 AM, Marcos Pinto <marcoscba@gmail.com> wrote:

> Hi guys, I gotta a question and I really would appreciate some help,
> suggestion, or advices.
> Is it possible write directly to the datanode's and eliminate the
> bottleneck
> to write into HDFS?
> for example: I gotta 40 writers and I writes th data direct to namenode, i
> was wondering if it would become a bottleneck so I would like to know
> if it is possible to write the data direct to the datanode's.
> I've read that Hadoop doesnt support multiple writes so how can we deal
> with
> big cluster? I mean I am really going to have this bottleneck???
>
> Thanks in advanced.
>

--0016364d2cbdcad04a0491a13add--

From general-return-2137-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 02 14:39:43 2010
Return-Path: <general-return-2137-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19848 invoked from network); 2 Oct 2010 14:39:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Oct 2010 14:39:42 -0000
Received: (qmail 49915 invoked by uid 500); 2 Oct 2010 14:39:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49385 invoked by uid 500); 2 Oct 2010 14:39:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49377 invoked by uid 99); 2 Oct 2010 14:39:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 14:39:36 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of marcoscba@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 14:39:30 +0000
Received: by wwb22 with SMTP id 22so4535615wwb.29
        for <general@hadoop.apache.org>; Sat, 02 Oct 2010 07:39:09 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=SDhW64ZvG2fdEpbYobNL/QYcKgLpChxjzX9AvvwA4pU=;
        b=e4kYcOmheZ47OBdZbGPlpCmo3Hjc0b8u991SMxrCARJavNGVgLHW54HaXoWVqoSF1o
         9e+EbN9NXA89+sXRh+bDfpz+KGo3mRlkfvXUNCAq1ulFkXMWJGY+SZ7NrJ4C+JBUcAO3
         Gn92YtJT7mGtJDcvSEO57WKCIUoeDndLfCrrc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=fY3CVUVHwAyRV5kRZar9jFFhDDVktCAiEM97spEEhEzbFGbS9NCX5ZXVcfcKS+vbuj
         1xxL88eMJXfTiBIzPx19SoI5luREqWxhUP48QxrcP1uFRkK24RxcMkf7zF9PsKYvKimN
         gfCQ4qBCOW/n2B+fM/3B8/sL35cjxC07RUo1w=
MIME-Version: 1.0
Received: by 10.216.184.19 with SMTP id r19mr5709718wem.36.1286030349700; Sat,
 02 Oct 2010 07:39:09 -0700 (PDT)
Received: by 10.216.51.145 with HTTP; Sat, 2 Oct 2010 07:39:09 -0700 (PDT)
In-Reply-To: <C586A8404F8B304E807B4B8BF73B376EA600AC@WIC001MITEBCLV1.messaging.mit>
References: <1C4C43FE-05F2-464B-9F04-AC972988C326@apple.com>
	<C586A8404F8B304E807B4B8BF73B376EA600AC@WIC001MITEBCLV1.messaging.mit>
Date: Sat, 2 Oct 2010 10:39:09 -0400
Message-ID: <AANLkTikT3H2y_EJ-tVgPvYjSVN7NouHvFTXXYR5vi044@mail.gmail.com>
Subject: Re: Total Space Available on Hadoop Cluster Or Hadoop version of "df".
From: Marcos Pinto <marcoscba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64ea56a2a0d6a0491a3460c
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e64ea56a2a0d6a0491a3460c
Content-Type: text/plain; charset=ISO-8859-1

I gotte the same problem, I remember it was something realted to user's
partition.
for example I created hadoop user so HDFS took the closest partition to
user.
I dont remenber exaclty but it was something like that. I hope it helps u in
someway.

On Sat, Oct 2, 2010 at 2:13 AM, Glenn Gore <Glenn.Gore@melbourneit.com.au>wrote:

> hadoop dfsadmin -report
>
> Regards
>
> Glenn
>
>
> -----Original Message-----
> From: rahul [mailto:rmalviya@apple.com]
> Sent: Sat 10/2/2010 2:27 PM
> To: general@hadoop.apache.org
> Subject: Total Space Available on Hadoop Cluster Or Hadoop version of "df".
>
> Hi,
>
> I am using Hadoop 0.20.2 version for data processing by setting up Hadoop
> Cluster on two nodes.
>
> And I am continuously adding more space to the nodes.
>
> Can some body let me know how to get the total space available on the
> hadoop cluster using command line.
>
> or
>
> Hadoop version "df", Unix command.
>
> Any input is helpful.
>
> Thanks
> Rahul
>
>

--0016e64ea56a2a0d6a0491a3460c--

From general-return-2138-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 02 16:53:13 2010
Return-Path: <general-return-2138-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55594 invoked from network); 2 Oct 2010 16:53:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Oct 2010 16:53:13 -0000
Received: (qmail 29035 invoked by uid 500); 2 Oct 2010 16:53:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28907 invoked by uid 500); 2 Oct 2010 16:53:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28899 invoked by uid 99); 2 Oct 2010 16:53:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 16:53:11 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of rmalviya@apple.com designates 17.254.13.23 as permitted sender)
Received: from [17.254.13.23] (HELO mail-out4.apple.com) (17.254.13.23)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 16:53:04 +0000
Received: from relay15.apple.com (relay15.apple.com [17.128.113.54])
	by mail-out4.apple.com (Postfix) with ESMTP id B1211B353051
	for <general@hadoop.apache.org>; Sat,  2 Oct 2010 09:52:43 -0700 (PDT)
X-AuditID: 11807136-b7b3eae0000066cf-f5-4ca7635bc402
Received: from [17.151.77.246] (Unknown_Domain [17.151.77.246])
	(using TLS with cipher AES128-SHA (AES128-SHA/128 bits))
	(Client did not present a certificate)
	by relay15.apple.com (Apple SCV relay) with SMTP id 53.6C.26319.B5367AC4; Sat,  2 Oct 2010 09:52:43 -0700 (PDT)
Subject: Re: Total Space Available on Hadoop Cluster Or Hadoop version of "df".
Mime-Version: 1.0 (Apple Message framework v1081)
Content-Type: text/plain; charset=us-ascii
From: rahul <rmalviya@apple.com>
In-Reply-To: <AANLkTikT3H2y_EJ-tVgPvYjSVN7NouHvFTXXYR5vi044@mail.gmail.com>
Date: Sat, 2 Oct 2010 09:52:42 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <4DDC93DB-A7B8-400C-9618-F72E3125BD3B@apple.com>
References: <1C4C43FE-05F2-464B-9F04-AC972988C326@apple.com> <C586A8404F8B304E807B4B8BF73B376EA600AC@WIC001MITEBCLV1.messaging.mit> <AANLkTikT3H2y_EJ-tVgPvYjSVN7NouHvFTXXYR5vi044@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1081)
X-Brightmail-Tracker: AAAAAA==

Hi Marcos,

Same thing is happening for me as well.=20

I have multiple disks mounted to my system but by default when i format =
it took the nearest/ disk in which hadoop binary is present.

Is there a way in which I can format all the drives mounted to my system =
?

So can we control in some way the drives or the places which we want to =
format for hdfs?

Thanks,
Rahul

On Oct 2, 2010, at 7:39 AM, Marcos Pinto wrote:

> I gotte the same problem, I remember it was something realted to =
user's
> partition.
> for example I created hadoop user so HDFS took the closest partition =
to
> user.
> I dont remenber exaclty but it was something like that. I hope it =
helps u in
> someway.
>=20
> On Sat, Oct 2, 2010 at 2:13 AM, Glenn Gore =
<Glenn.Gore@melbourneit.com.au>wrote:
>=20
>> hadoop dfsadmin -report
>>=20
>> Regards
>>=20
>> Glenn
>>=20
>>=20
>> -----Original Message-----
>> From: rahul [mailto:rmalviya@apple.com]
>> Sent: Sat 10/2/2010 2:27 PM
>> To: general@hadoop.apache.org
>> Subject: Total Space Available on Hadoop Cluster Or Hadoop version of =
"df".
>>=20
>> Hi,
>>=20
>> I am using Hadoop 0.20.2 version for data processing by setting up =
Hadoop
>> Cluster on two nodes.
>>=20
>> And I am continuously adding more space to the nodes.
>>=20
>> Can some body let me know how to get the total space available on the
>> hadoop cluster using command line.
>>=20
>> or
>>=20
>> Hadoop version "df", Unix command.
>>=20
>> Any input is helpful.
>>=20
>> Thanks
>> Rahul
>>=20
>>=20


From general-return-2139-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 02 16:55:32 2010
Return-Path: <general-return-2139-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55791 invoked from network); 2 Oct 2010 16:55:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Oct 2010 16:55:32 -0000
Received: (qmail 30385 invoked by uid 500); 2 Oct 2010 16:55:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30286 invoked by uid 500); 2 Oct 2010 16:55:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30278 invoked by uid 99); 2 Oct 2010 16:55:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 16:55:30 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of rmalviya@apple.com designates 17.254.13.23 as permitted sender)
Received: from [17.254.13.23] (HELO mail-out4.apple.com) (17.254.13.23)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 16:55:20 +0000
Received: from relay13.apple.com (relay13.apple.com [17.128.113.29])
	by mail-out4.apple.com (Postfix) with ESMTP id E7F5CB35319A
	for <general@hadoop.apache.org>; Sat,  2 Oct 2010 09:54:59 -0700 (PDT)
X-AuditID: 1180711d-b7b8eae0000035ac-b0-4ca763e38c32
Received: from [17.151.77.246] (Unknown_Domain [17.151.77.246])
	(using TLS with cipher AES128-SHA (AES128-SHA/128 bits))
	(Client did not present a certificate)
	by relay13.apple.com (Apple SCV relay) with SMTP id 4F.27.13740.3E367AC4; Sat,  2 Oct 2010 09:54:59 -0700 (PDT)
Subject: Re: Total Space Available on Hadoop Cluster Or Hadoop version of "df".
Mime-Version: 1.0 (Apple Message framework v1081)
Content-Type: text/plain; charset=us-ascii
From: rahul <rmalviya@apple.com>
In-Reply-To: <C586A8404F8B304E807B4B8BF73B376EA600AC@WIC001MITEBCLV1.messaging.mit>
Date: Sat, 2 Oct 2010 09:54:59 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <F5AFC2C6-70C6-4504-A781-A9C910E0FB98@apple.com>
References: <1C4C43FE-05F2-464B-9F04-AC972988C326@apple.com> <C586A8404F8B304E807B4B8BF73B376EA600AC@WIC001MITEBCLV1.messaging.mit>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1081)
X-Brightmail-Tracker: AAAAAA==
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks Glenn

On Oct 1, 2010, at 11:13 PM, Glenn Gore wrote:

> hadoop dfsadmin -report
>=20
> Regards
>=20
> Glenn
>=20
>=20
> -----Original Message-----
> From: rahul [mailto:rmalviya@apple.com]
> Sent: Sat 10/2/2010 2:27 PM
> To: general@hadoop.apache.org
> Subject: Total Space Available on Hadoop Cluster Or Hadoop version of =
"df".
>=20
> Hi,
>=20
> I am using Hadoop 0.20.2 version for data processing by setting up =
Hadoop Cluster on two nodes.=20
>=20
> And I am continuously adding more space to the nodes.
>=20
> Can some body let me know how to get the total space available on the =
hadoop cluster using command line.
>=20
> or=20
>=20
> Hadoop version "df", Unix command.
>=20
> Any input is helpful.
>=20
> Thanks
> Rahul
>=20


From general-return-2140-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 02 18:58:21 2010
Return-Path: <general-return-2140-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1956 invoked from network); 2 Oct 2010 18:58:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Oct 2010 18:58:21 -0000
Received: (qmail 45748 invoked by uid 500); 2 Oct 2010 18:58:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45683 invoked by uid 500); 2 Oct 2010 18:58:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45675 invoked by uid 99); 2 Oct 2010 18:58:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 18:58:19 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ryanobjc@gmail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 18:58:11 +0000
Received: by iwn6 with SMTP id 6so4126577iwn.35
        for <general@hadoop.apache.org>; Sat, 02 Oct 2010 11:57:50 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=JDDm6S3GhrhpnUyTGo1CB+ueVDYPENupnxMt2Gaqf1s=;
        b=JBIJlN5qtB4YEsEQGrbKxasMZMYnmcN5mXBtRg/NHG6iFdeKayqe7UIH3BNwS9q3Zh
         q3XiU+Nd4vDbpBb0fHza9ErH9XmeLk4AurOc60Lt1Y1t60x/TsRECJNmo0ekMVcKPDb2
         nUr3atzvyeLYqHNXUZn8K4J9qu95LLxhJOjww=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=K6eEdFR/9tXz1FtOdNKfkC+YGSOV1udFg2iKPjV0r0iAIfgjPeWSo2Fam7B3eF3G02
         HPFX8lMMRaRtZ06OY48yhdAZDbyAlbop/ZtWLKgExNdMvymgjWFGCtjoDi2wUNkiidWR
         440zlQOvO9Zabb1v+rBh7Kf6xAQJ+0j3r+2cA=
MIME-Version: 1.0
Received: by 10.231.152.143 with SMTP id g15mr7497238ibw.76.1286045870543;
 Sat, 02 Oct 2010 11:57:50 -0700 (PDT)
Received: by 10.231.12.140 with HTTP; Sat, 2 Oct 2010 11:57:50 -0700 (PDT)
In-Reply-To: <AANLkTin6ZUrWYUscAXmq0bOG8mWA35xcAfvLD-jYVzwu@mail.gmail.com>
References: <AANLkTin6ZUrWYUscAXmq0bOG8mWA35xcAfvLD-jYVzwu@mail.gmail.com>
Date: Sat, 2 Oct 2010 11:57:50 -0700
Message-ID: <AANLkTik_dZctLp6Af1k9C75oJ+JhboRGZOXGsYOfwZLK@mail.gmail.com>
Subject: Re: Is it possible write directly to datanode's?
From: Ryan Rawson <ryanobjc@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

I suggest you read this:
http://hadoop.apache.org/common/docs/r0.20.0/hdfs_design.html

-ryan

On Fri, Oct 1, 2010 at 8:06 AM, Marcos Pinto <marcoscba@gmail.com> wrote:
> Hi guys, I gotta a question and I really would appreciate some help,
> suggestion, or advices.
> Is it possible write directly to the datanode's and eliminate the bottleneck
> to write into HDFS?
> for example: I gotta 40 writers and I writes th data direct to namenode, i
> was wondering if it would become a bottleneck so I would like to know
> if it is possible to write the data direct to the datanode's.
> I've read that Hadoop doesnt support multiple writes so how can we deal with
> big cluster? I mean I am really going to have this bottleneck???
>
> Thanks in advanced.
>

From general-return-2141-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 02 21:16:38 2010
Return-Path: <general-return-2141-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34106 invoked from network); 2 Oct 2010 21:16:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Oct 2010 21:16:37 -0000
Received: (qmail 13839 invoked by uid 500); 2 Oct 2010 21:16:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13679 invoked by uid 500); 2 Oct 2010 21:16:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13671 invoked by uid 99); 2 Oct 2010 21:16:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 21:16:36 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of marcoscba@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 21:16:29 +0000
Received: by wye20 with SMTP id 20so1053225wye.35
        for <general@hadoop.apache.org>; Sat, 02 Oct 2010 14:16:08 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=NYATcC8ORh8kyWOzVXWk+KoRGk/Qi2TTHNYNM7tAyS0=;
        b=EKo17oVsUfTWmf+5bNwBz7ona63hvOdU4ap56Vz3QAvF4OoE6KHh/ivoTGn5yfel5p
         P2Rp6/76CMR51E8s333qthaAWuvrhrih9tHuYP7V/6eX8aH/tvM8sWoHh6ZBuR4/188m
         OXqOM/cDO4mlgwakVlSsUEhbolnM1UKH5N7Nw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=mrAFsDCV6+FDT4UTGvrbEvHvZnaiiCv7NeK78B+jYEhbaEN4Z9NkAUku/SWZfujtub
         2eCrBf290iZfs64Q8xV8a/bwO6Wg2lcBudwqC3N/8O+cr6/oE4uCJv3+KErgymXSKV4e
         Z00TbZLADUBgcaE+sYz1TkpmbWknjrmiuz1pc=
MIME-Version: 1.0
Received: by 10.216.26.211 with SMTP id c61mr1139103wea.26.1286054168551; Sat,
 02 Oct 2010 14:16:08 -0700 (PDT)
Received: by 10.216.51.145 with HTTP; Sat, 2 Oct 2010 14:16:08 -0700 (PDT)
In-Reply-To: <AANLkTik_dZctLp6Af1k9C75oJ+JhboRGZOXGsYOfwZLK@mail.gmail.com>
References: <AANLkTin6ZUrWYUscAXmq0bOG8mWA35xcAfvLD-jYVzwu@mail.gmail.com>
	<AANLkTik_dZctLp6Af1k9C75oJ+JhboRGZOXGsYOfwZLK@mail.gmail.com>
Date: Sat, 2 Oct 2010 17:16:08 -0400
Message-ID: <AANLkTi=+r_qR093O=O0+2C=rRfLsoJRz=x5AWuRKgHoF@mail.gmail.com>
Subject: Re: Is it possible write directly to datanode's?
From: Marcos Pinto <marcoscba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636d34693e0e26e0491a8d163

--001636d34693e0e26e0491a8d163
Content-Type: text/plain; charset=ISO-8859-1

Thanks Guys, I misunderstood the picture in the hadoop site. Now I know that
the namenode says in which datanode should be written and which part of
file. So I dont need to worry about it. Thanks.

On Sat, Oct 2, 2010 at 2:57 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:

> I suggest you read this:
> http://hadoop.apache.org/common/docs/r0.20.0/hdfs_design.html
>
> -ryan
>
> On Fri, Oct 1, 2010 at 8:06 AM, Marcos Pinto <marcoscba@gmail.com> wrote:
> > Hi guys, I gotta a question and I really would appreciate some help,
> > suggestion, or advices.
> > Is it possible write directly to the datanode's and eliminate the
> bottleneck
> > to write into HDFS?
> > for example: I gotta 40 writers and I writes th data direct to namenode,
> i
> > was wondering if it would become a bottleneck so I would like to know
> > if it is possible to write the data direct to the datanode's.
> > I've read that Hadoop doesnt support multiple writes so how can we deal
> with
> > big cluster? I mean I am really going to have this bottleneck???
> >
> > Thanks in advanced.
> >
>

--001636d34693e0e26e0491a8d163--

From general-return-2142-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Oct 03 04:33:04 2010
Return-Path: <general-return-2142-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49113 invoked from network); 3 Oct 2010 04:33:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Oct 2010 04:33:03 -0000
Received: (qmail 48999 invoked by uid 500); 3 Oct 2010 04:33:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48700 invoked by uid 500); 3 Oct 2010 04:32:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48692 invoked by uid 99); 3 Oct 2010 04:32:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 03 Oct 2010 04:32:58 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jgray@facebook.com designates 69.63.178.183 as permitted sender)
Received: from [69.63.178.183] (HELO mx-out.facebook.com) (69.63.178.183)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 03 Oct 2010 04:32:51 +0000
Received: from [10.18.255.176] ([10.18.255.176:27669] helo=mail.thefacebook.com)
	by mta015.snc1.facebook.com (envelope-from <jgray@facebook.com>)
	(ecelerity 2.2.2.45 r(34067)) with ESMTP
	id 13/4E-13534-E5708AC4; Sat, 02 Oct 2010 21:32:30 -0700
Received: from SC-MBX04.TheFacebook.com ([169.254.3.231]) by
 sc-hub03.TheFacebook.com ([fe80::1cfe:1f6b:8b35:cf7f%11]) with mapi; Sat, 2
 Oct 2010 21:32:30 -0700
From: Jonathan Gray <jgray@facebook.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: RE: Total Space Available on Hadoop Cluster Or Hadoop version of
 "df".
Thread-Topic: Total Space Available on Hadoop Cluster Or Hadoop version of
 "df".
Thread-Index: AQHLYeoyLan6Ia1UEk6/C4gnBUqBgQADrR6sky4TQICAACVQAIAATLFw
Date: Sun, 3 Oct 2010 04:32:52 +0000
Message-ID: <5A76F6CE309AD049AAF9A039A39242820F01F1CA@sc-mbx04.TheFacebook.com>
References: <1C4C43FE-05F2-464B-9F04-AC972988C326@apple.com>
 <C586A8404F8B304E807B4B8BF73B376EA600AC@WIC001MITEBCLV1.messaging.mit>
 <AANLkTikT3H2y_EJ-tVgPvYjSVN7NouHvFTXXYR5vi044@mail.gmail.com>
 <4DDC93DB-A7B8-400C-9618-F72E3125BD3B@apple.com>
In-Reply-To: <4DDC93DB-A7B8-400C-9618-F72E3125BD3B@apple.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Rahul,

There is a ton of documentation available for Hadoop (including books).

Best place to start is the wiki: http://wiki.apache.org/hadoop/

On your specific issue, you need to configure Hadoop to tell it what direct=
ories to store data.

The configuration parameter name is 'dfs.data.dir' and you need to put in a=
 comma-delimited list of directories to use to store data.

JG

> -----Original Message-----
> From: rahul [mailto:rmalviya@apple.com]
> Sent: Saturday, October 02, 2010 9:53 AM
> To: general@hadoop.apache.org
> Subject: Re: Total Space Available on Hadoop Cluster Or Hadoop version
> of "df".
>=20
> Hi Marcos,
>=20
> Same thing is happening for me as well.
>=20
> I have multiple disks mounted to my system but by default when i format
> it took the nearest/ disk in which hadoop binary is present.
>=20
> Is there a way in which I can format all the drives mounted to my
> system ?
>=20
> So can we control in some way the drives or the places which we want to
> format for hdfs?
>=20
> Thanks,
> Rahul
>=20
> On Oct 2, 2010, at 7:39 AM, Marcos Pinto wrote:
>=20
> > I gotte the same problem, I remember it was something realted to
> user's
> > partition.
> > for example I created hadoop user so HDFS took the closest partition
> to
> > user.
> > I dont remenber exaclty but it was something like that. I hope it
> helps u in
> > someway.
> >
> > On Sat, Oct 2, 2010 at 2:13 AM, Glenn Gore
> <Glenn.Gore@melbourneit.com.au>wrote:
> >
> >> hadoop dfsadmin -report
> >>
> >> Regards
> >>
> >> Glenn
> >>
> >>
> >> -----Original Message-----
> >> From: rahul [mailto:rmalviya@apple.com]
> >> Sent: Sat 10/2/2010 2:27 PM
> >> To: general@hadoop.apache.org
> >> Subject: Total Space Available on Hadoop Cluster Or Hadoop version
> of "df".
> >>
> >> Hi,
> >>
> >> I am using Hadoop 0.20.2 version for data processing by setting up
> Hadoop
> >> Cluster on two nodes.
> >>
> >> And I am continuously adding more space to the nodes.
> >>
> >> Can some body let me know how to get the total space available on
> the
> >> hadoop cluster using command line.
> >>
> >> or
> >>
> >> Hadoop version "df", Unix command.
> >>
> >> Any input is helpful.
> >>
> >> Thanks
> >> Rahul
> >>
> >>


From general-return-2143-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Oct 03 17:58:45 2010
Return-Path: <general-return-2143-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21374 invoked from network); 3 Oct 2010 17:58:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Oct 2010 17:58:45 -0000
Received: (qmail 68961 invoked by uid 500); 3 Oct 2010 17:58:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68889 invoked by uid 500); 3 Oct 2010 17:58:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68881 invoked by uid 99); 3 Oct 2010 17:58:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 03 Oct 2010 17:58:42 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of rmalviya@apple.com designates 17.254.13.22 as permitted sender)
Received: from [17.254.13.22] (HELO mail-out3.apple.com) (17.254.13.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 03 Oct 2010 17:58:35 +0000
Received: from relay11.apple.com (relay11.apple.com [17.128.113.48])
	by mail-out3.apple.com (Postfix) with ESMTP id CB987ACE0A17
	for <general@hadoop.apache.org>; Sun,  3 Oct 2010 10:58:14 -0700 (PDT)
X-AuditID: 11807130-b7cf8ae0000058d2-31-4ca8c43693b0
Received: from [17.151.97.209] (Unknown_Domain [17.151.97.209])
	(using TLS with cipher AES128-SHA (AES128-SHA/128 bits))
	(Client did not present a certificate)
	by relay11.apple.com (Apple SCV relay) with SMTP id 02.04.22738.634C8AC4; Sun,  3 Oct 2010 10:58:14 -0700 (PDT)
Subject: Re: Total Space Available on Hadoop Cluster Or Hadoop version of "df".
Mime-Version: 1.0 (Apple Message framework v1081)
Content-Type: text/plain; charset=us-ascii
From: rahul <rmalviya@apple.com>
In-Reply-To: <5A76F6CE309AD049AAF9A039A39242820F01F1CA@sc-mbx04.TheFacebook.com>
Date: Sun, 3 Oct 2010 10:58:13 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <1DB033BE-79E4-45DC-B4C5-046DF165CF59@apple.com>
References: <1C4C43FE-05F2-464B-9F04-AC972988C326@apple.com> <C586A8404F8B304E807B4B8BF73B376EA600AC@WIC001MITEBCLV1.messaging.mit> <AANLkTikT3H2y_EJ-tVgPvYjSVN7NouHvFTXXYR5vi044@mail.gmail.com> <4DDC93DB-A7B8-400C-9618-F72E3125BD3B@apple.com> <5A76F6CE309AD049AAF9A039A39242820F01F1CA@sc-mbx04.TheFacebook.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1081)
X-Brightmail-Tracker: AAAAAA==

Thanks Jonathan,

Its really of great help.

Rahul
On Oct 2, 2010, at 9:32 PM, Jonathan Gray wrote:

> Rahul,
>=20
> There is a ton of documentation available for Hadoop (including =
books).
>=20
> Best place to start is the wiki: http://wiki.apache.org/hadoop/
>=20
> On your specific issue, you need to configure Hadoop to tell it what =
directories to store data.
>=20
> The configuration parameter name is 'dfs.data.dir' and you need to put =
in a comma-delimited list of directories to use to store data.
>=20
> JG
>=20
>> -----Original Message-----
>> From: rahul [mailto:rmalviya@apple.com]
>> Sent: Saturday, October 02, 2010 9:53 AM
>> To: general@hadoop.apache.org
>> Subject: Re: Total Space Available on Hadoop Cluster Or Hadoop =
version
>> of "df".
>>=20
>> Hi Marcos,
>>=20
>> Same thing is happening for me as well.
>>=20
>> I have multiple disks mounted to my system but by default when i =
format
>> it took the nearest/ disk in which hadoop binary is present.
>>=20
>> Is there a way in which I can format all the drives mounted to my
>> system ?
>>=20
>> So can we control in some way the drives or the places which we want =
to
>> format for hdfs?
>>=20
>> Thanks,
>> Rahul
>>=20
>> On Oct 2, 2010, at 7:39 AM, Marcos Pinto wrote:
>>=20
>>> I gotte the same problem, I remember it was something realted to
>> user's
>>> partition.
>>> for example I created hadoop user so HDFS took the closest partition
>> to
>>> user.
>>> I dont remenber exaclty but it was something like that. I hope it
>> helps u in
>>> someway.
>>>=20
>>> On Sat, Oct 2, 2010 at 2:13 AM, Glenn Gore
>> <Glenn.Gore@melbourneit.com.au>wrote:
>>>=20
>>>> hadoop dfsadmin -report
>>>>=20
>>>> Regards
>>>>=20
>>>> Glenn
>>>>=20
>>>>=20
>>>> -----Original Message-----
>>>> From: rahul [mailto:rmalviya@apple.com]
>>>> Sent: Sat 10/2/2010 2:27 PM
>>>> To: general@hadoop.apache.org
>>>> Subject: Total Space Available on Hadoop Cluster Or Hadoop version
>> of "df".
>>>>=20
>>>> Hi,
>>>>=20
>>>> I am using Hadoop 0.20.2 version for data processing by setting up
>> Hadoop
>>>> Cluster on two nodes.
>>>>=20
>>>> And I am continuously adding more space to the nodes.
>>>>=20
>>>> Can some body let me know how to get the total space available on
>> the
>>>> hadoop cluster using command line.
>>>>=20
>>>> or
>>>>=20
>>>> Hadoop version "df", Unix command.
>>>>=20
>>>> Any input is helpful.
>>>>=20
>>>> Thanks
>>>> Rahul
>>>>=20
>>>>=20
>=20


From general-return-2144-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 04 12:49:27 2010
Return-Path: <general-return-2144-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5812 invoked from network); 4 Oct 2010 12:49:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Oct 2010 12:49:26 -0000
Received: (qmail 79341 invoked by uid 500); 4 Oct 2010 12:49:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78933 invoked by uid 500); 4 Oct 2010 12:49:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78925 invoked by uid 99); 4 Oct 2010 12:49:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 12:49:21 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of marcoscba@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 12:49:14 +0000
Received: by wwb22 with SMTP id 22so6461665wwb.29
        for <general@hadoop.apache.org>; Mon, 04 Oct 2010 05:48:54 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=0GDAZSZu9UfXbWkcTugxDN7x+fItTZRqYjTzODQdSI4=;
        b=tO4ZDdolni6hcJzoiW2MRbbphqfx634hayk5jgjN9V8ReT7ScBm+G/rt8C3dC2rGaH
         CcI8v3CM3hDSepqytAdH4e3i0ZPGMcHTlD8/Dg9FeyNgdiJj/uYzIMUp5mxWxfmymLJX
         mX3EyDZwpLaVlVOB1yi+zE5DvFIboD81+WS2M=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=RGrBcEKpjJbcmp81BhemN/eeIFhcuyQ0FkeiIppbKKsr4/aqcTvtrtEKrZiOO+eQ5/
         d4jFJaHdl5M59BFfNY9rco1TEiSFMDsKMy2cciQiA5bhd5xOTrou63nvKoMIdIQfc2fX
         fuvFuAvmk2vbJ6FIdr4li849qgVRMK7WXGE+g=
MIME-Version: 1.0
Received: by 10.216.22.70 with SMTP id s48mr5235847wes.27.1286196534197; Mon,
 04 Oct 2010 05:48:54 -0700 (PDT)
Received: by 10.216.51.145 with HTTP; Mon, 4 Oct 2010 05:48:54 -0700 (PDT)
In-Reply-To: <1DB033BE-79E4-45DC-B4C5-046DF165CF59@apple.com>
References: <1C4C43FE-05F2-464B-9F04-AC972988C326@apple.com>
	<C586A8404F8B304E807B4B8BF73B376EA600AC@WIC001MITEBCLV1.messaging.mit>
	<AANLkTikT3H2y_EJ-tVgPvYjSVN7NouHvFTXXYR5vi044@mail.gmail.com>
	<4DDC93DB-A7B8-400C-9618-F72E3125BD3B@apple.com>
	<5A76F6CE309AD049AAF9A039A39242820F01F1CA@sc-mbx04.TheFacebook.com>
	<1DB033BE-79E4-45DC-B4C5-046DF165CF59@apple.com>
Date: Mon, 4 Oct 2010 09:48:54 -0300
Message-ID: <AANLkTimoNaVwQWAiGanJwUDrxYSRGAFbxTgyiW6MV1Mr@mail.gmail.com>
Subject: Re: Total Space Available on Hadoop Cluster Or Hadoop version of "df".
From: Marcos Pinto <marcoscba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364c72bb885e7b0491c9f73e
X-Virus-Checked: Checked by ClamAV on apache.org

--0016364c72bb885e7b0491c9f73e
Content-Type: text/plain; charset=ISO-8859-1

HDFS takes the memory available from the closest partition to your
hadoop.tmp.dir(core-site.xml).
I hope it helps you.


user hadoop => /home/hadoop

On Sun, Oct 3, 2010 at 2:58 PM, rahul <rmalviya@apple.com> wrote:

> Thanks Jonathan,
>
> Its really of great help.
>
> Rahul
> On Oct 2, 2010, at 9:32 PM, Jonathan Gray wrote:
>
> > Rahul,
> >
> > There is a ton of documentation available for Hadoop (including books).
> >
> > Best place to start is the wiki: http://wiki.apache.org/hadoop/
> >
> > On your specific issue, you need to configure Hadoop to tell it what
> directories to store data.
> >
> > The configuration parameter name is 'dfs.data.dir' and you need to put in
> a comma-delimited list of directories to use to store data.
> >
> > JG
> >
> >> -----Original Message-----
> >> From: rahul [mailto:rmalviya@apple.com]
> >> Sent: Saturday, October 02, 2010 9:53 AM
> >> To: general@hadoop.apache.org
> >> Subject: Re: Total Space Available on Hadoop Cluster Or Hadoop version
> >> of "df".
> >>
> >> Hi Marcos,
> >>
> >> Same thing is happening for me as well.
> >>
> >> I have multiple disks mounted to my system but by default when i format
> >> it took the nearest/ disk in which hadoop binary is present.
> >>
> >> Is there a way in which I can format all the drives mounted to my
> >> system ?
> >>
> >> So can we control in some way the drives or the places which we want to
> >> format for hdfs?
> >>
> >> Thanks,
> >> Rahul
> >>
> >> On Oct 2, 2010, at 7:39 AM, Marcos Pinto wrote:
> >>
> >>> I gotte the same problem, I remember it was something realted to
> >> user's
> >>> partition.
> >>> for example I created hadoop user so HDFS took the closest partition
> >> to
> >>> user.
> >>> I dont remenber exaclty but it was something like that. I hope it
> >> helps u in
> >>> someway.
> >>>
> >>> On Sat, Oct 2, 2010 at 2:13 AM, Glenn Gore
> >> <Glenn.Gore@melbourneit.com.au>wrote:
> >>>
> >>>> hadoop dfsadmin -report
> >>>>
> >>>> Regards
> >>>>
> >>>> Glenn
> >>>>
> >>>>
> >>>> -----Original Message-----
> >>>> From: rahul [mailto:rmalviya@apple.com]
> >>>> Sent: Sat 10/2/2010 2:27 PM
> >>>> To: general@hadoop.apache.org
> >>>> Subject: Total Space Available on Hadoop Cluster Or Hadoop version
> >> of "df".
> >>>>
> >>>> Hi,
> >>>>
> >>>> I am using Hadoop 0.20.2 version for data processing by setting up
> >> Hadoop
> >>>> Cluster on two nodes.
> >>>>
> >>>> And I am continuously adding more space to the nodes.
> >>>>
> >>>> Can some body let me know how to get the total space available on
> >> the
> >>>> hadoop cluster using command line.
> >>>>
> >>>> or
> >>>>
> >>>> Hadoop version "df", Unix command.
> >>>>
> >>>> Any input is helpful.
> >>>>
> >>>> Thanks
> >>>> Rahul
> >>>>
> >>>>
> >
>
>

--0016364c72bb885e7b0491c9f73e--

From general-return-2145-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 06 20:17:47 2010
Return-Path: <general-return-2145-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 89350 invoked from network); 6 Oct 2010 20:17:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Oct 2010 20:17:46 -0000
Received: (qmail 61644 invoked by uid 500); 6 Oct 2010 20:17:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61565 invoked by uid 500); 6 Oct 2010 20:17:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61557 invoked by uid 99); 6 Oct 2010 20:17:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 20:17:44 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jdcryans@gmail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 20:17:38 +0000
Received: by iwn6 with SMTP id 6so10787296iwn.35
        for <general@hadoop.apache.org>; Wed, 06 Oct 2010 13:17:17 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        bh=12YPfLx4XV2yOFkMF78jRw6lcvhb8VzUdPAKG8458rU=;
        b=M4PMAMYr+IdCbi6p3A6fHvjM/QloTn/+QZYQU06ZCLz5tzxDUJ1DrYeer4bi3hNVWo
         Y7WhBMmOlmJG4p4Kgseq/N6/RR3GBrFKim0pU0webdA9nzY0n5mmHaxa7kpLv35fmDlm
         NC8EtqAhNmxRnde8Jhkk6hU3TuA7KRj0y6lAw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        b=NGoUcfP9tLuJVDMlLCgg/vED9sFjLqyq9AFRhv5wl5mu2WPa5VWtYzxEt1xiO+oJwa
         g+fgLs8ib3LCHD3Ja+XlDl6wFKoKdQYxd4GtJYx7UbNSNHUH56l+4zZb+gz9Lq7SbsEb
         kWRHTY+GETqMYQEI3beZisclvn76buWngmo4g=
MIME-Version: 1.0
Received: by 10.231.149.140 with SMTP id t12mr14510345ibv.100.1286396237458;
 Wed, 06 Oct 2010 13:17:17 -0700 (PDT)
Sender: jdcryans@gmail.com
Received: by 10.231.19.137 with HTTP; Wed, 6 Oct 2010 13:17:17 -0700 (PDT)
Date: Wed, 6 Oct 2010 13:17:17 -0700
X-Google-Sender-Auth: ACRzZXto3gBOzxrC-iLr_wFT94s
Message-ID: <AANLkTinj-MOcTLZvg3k=Zha3GHwp68SvH1m3p9XSYWm=@mail.gmail.com>
Subject: [ANN] HBase-0.89.20100924, our third 'developer release', is now
 available for download
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: general@hadoop.apache.org, user@hbase.apache.org
Content-Type: text/plain; charset=ISO-8859-1

The HBase Team would like to announce the availability of the third
in our 'developer release' series, hbase-0.89.20100924.  Binary
and source tar balls are available here:

 http://www.apache.org/dyn/closer.cgi/hbase/

You can browse the documentation here:

 http://hbase.apache.org/docs/r0.89.20100924/

This release contains a rollback of some features that were added to
the master in 0726 that made it more unstable, but also contains about
40 fixes for bugs (including more reliable region server failovers)
and performance improvements.

Our 'developer releases' are for those who can tolerate risk and who
are willing to give feedback in advance of our next major release.
We're not making any guarantees that this is bug free. Its definitely
not for production deploys.  See
http://wiki.apache.org/hadoop/Hbase/HBaseVersions for more on the new
hbase versoning and our developer releases series. Also, please have a
look at the KNOWN_ISSUES.txt file next to the tar balls for critical
issues you need to be aware of.

Thanks to all who contributed to this release.

Yours,

The HBase Team

From general-return-2146-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 08 20:01:44 2010
Return-Path: <general-return-2146-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1269 invoked from network); 8 Oct 2010 20:01:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Oct 2010 20:01:43 -0000
Received: (qmail 62587 invoked by uid 500); 8 Oct 2010 20:01:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62504 invoked by uid 500); 8 Oct 2010 20:01:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62496 invoked by uid 99); 8 Oct 2010 20:01:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Oct 2010 20:01:41 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of davidlary@gmail.com designates 209.85.213.48 as permitted sender)
Received: from [209.85.213.48] (HELO mail-yw0-f48.google.com) (209.85.213.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Oct 2010 20:01:33 +0000
Received: by ywo32 with SMTP id 32so619032ywo.35
        for <general@hadoop.apache.org>; Fri, 08 Oct 2010 13:01:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:from:content-type
         :content-transfer-encoding:subject:date:message-id:cc:to
         :mime-version:x-mailer;
        bh=7bSlKHZ9QyMTVE8v0hie5xnJ/L27HeaayLt6jDu+HA0=;
        b=wPI1GLGZZI628IO+N1mc8R3pMtgSnh+Avf7aBICKcP/gK9GHxr1NwRS5XgaqWSj5x/
         iyNPQPQg93i44PKp3TwLAuCOZ+nftnEaQWy0OVLC8StPCVp5bHxnHBw8+eWY6++KsW4T
         63jFeBpRd/IwctGyWpKmpi4KPdaQLoUaq+7ME=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=from:content-type:content-transfer-encoding:subject:date:message-id
         :cc:to:mime-version:x-mailer;
        b=I1wIcABGPMmjvL31hSl+p1jL4JO/GIjw3w8mAhH412lDkX646JyccLWKEFSzj/b4Yq
         zUDmLSjVEcFqc6ky9jIBdy+9iheBn2RorT1sJm2GyHyOjijfawCK6MsW7/onkM/9KpMw
         BnLd7facrnmNXFN+uqhjSdpoz0LV/8+ytAcRY=
Received: by 10.236.109.44 with SMTP id r32mr6203951yhg.16.1286568072477;
        Fri, 08 Oct 2010 13:01:12 -0700 (PDT)
Received: from [192.168.1.174] (utdpat241007.utdallas.edu [129.110.241.7])
        by mx.google.com with ESMTPS id m25sm2646895yha.43.2010.10.08.13.01.11
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Fri, 08 Oct 2010 13:01:11 -0700 (PDT)
From: David Lary <davidlary@gmail.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: DFS error 
Date: Fri, 8 Oct 2010 15:01:09 -0500
Message-Id: <3C526026-7FFD-4E20-819B-58DE5738B577@gmail.com>
Cc: David Lary <davidlary@gmail.com>
To: general@hadoop.apache.org
Mime-Version: 1.0 (Apple Message framework v1081)
X-Mailer: Apple Mail (2.1081)
X-Virus-Checked: Checked by ClamAV on apache.org

Please can someone help.

Tried to set up pseudo-distributed mode for DFS. When I do start-dfs.sh =
I get the error message below, and am not sure what it means. Any help =
would be most appreciated.

Thanks

David

The conf/hdfs-site.xml file contains:
<configuration>
	<property>
		<name>fs.defaults.name</name>
		<value>hdfs://localhost:8020</value>
	</property>
</configuration>



$ start-dfs.sh
starting namenode, logging to =
/Users/davidlary/hadoop-0.20.2/bin/../logs/hadoop-davidlary-namenode-129.1=
10.31.40.out
localhost: starting datanode, logging to =
/Users/davidlary/hadoop-0.20.2/bin/../logs/hadoop-davidlary-datanode-129.1=
10.31.40.out
localhost: starting secondarynamenode, logging to =
/Users/davidlary/hadoop-0.20.2/bin/../logs/hadoop-davidlary-secondarynamen=
ode-129.110.31.40.out
localhost: Exception in thread "main" java.lang.NullPointerException
localhost: 	at =
org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:134)
localhost: 	at =
org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:1=
56)
localhost: 	at =
org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:1=
60)
localhost: 	at =
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(Second=
aryNameNode.java:131)
localhost: 	at =
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryN=
ameNode.java:115)
localhost: 	at =
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNam=
eNode.java:469)




From general-return-2147-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 09 11:15:19 2010
Return-Path: <general-return-2147-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34617 invoked from network); 9 Oct 2010 11:15:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Oct 2010 11:15:19 -0000
Received: (qmail 32471 invoked by uid 500); 9 Oct 2010 11:15:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32305 invoked by uid 500); 9 Oct 2010 11:15:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32297 invoked by uid 99); 9 Oct 2010 11:15:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 09 Oct 2010 11:15:14 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eltonsky9404@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 09 Oct 2010 11:15:07 +0000
Received: by wyb38 with SMTP id 38so944836wyb.35
        for <general@hadoop.apache.org>; Sat, 09 Oct 2010 04:14:47 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=nH/1HP5d8k9t7Tvalv4xEuUWXLRkqsm6PcLU7zWNvQg=;
        b=FJUQZXNXaLSzF3i4gGAZy5YluIJC+ird2KHZyfS5k5fPCYwORTpZmYHbP+p3rbbPBK
         pqzK1DgGypg66UxJj7ttOL6Jxgj58+P37y7jGYRqeoBQmr0eHFpqQoZu1B0ABP0nROzn
         sbleExlSLP/G0V+PuDNr1dAtempbKtU4RDjcM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=HCqz7TbmFNIlUKlXpzTa+SRbAdu+ZMQl2xJ8Ol1ZLgJlUsR09lI50XFElmPThQJfZS
         6Zja7T+iWn1Meh4rp3MVyYnbwq0PNGlJlkzxiUMiNp0eRq/MDbMznajcYZRMZmnCo1KY
         PKmkoFK2VmKIBfJegiep7uIdS38IVqWwwmAGo=
MIME-Version: 1.0
Received: by 10.227.128.196 with SMTP id l4mr3641330wbs.32.1286622886784; Sat,
 09 Oct 2010 04:14:46 -0700 (PDT)
Received: by 10.227.147.136 with HTTP; Sat, 9 Oct 2010 04:14:46 -0700 (PDT)
In-Reply-To: <3C526026-7FFD-4E20-819B-58DE5738B577@gmail.com>
References: <3C526026-7FFD-4E20-819B-58DE5738B577@gmail.com>
Date: Sat, 9 Oct 2010 22:14:46 +1100
Message-ID: <AANLkTi=YjmiK-0qE5M8C9_+CZay=RMCFGLr3PHKj0cBy@mail.gmail.com>
Subject: Re: DFS error
From: elton sky <eltonsky9404@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e65b400c206fe104922d3ced
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e65b400c206fe104922d3ced
Content-Type: text/plain; charset=ISO-8859-1

looks like hadoop couldn't find the uri of your second name node. what's in
your conf/master?

On Sat, Oct 9, 2010 at 7:01 AM, David Lary <davidlary@gmail.com> wrote:

> Please can someone help.
>
> Tried to set up pseudo-distributed mode for DFS. When I do start-dfs.sh I
> get the error message below, and am not sure what it means. Any help would
> be most appreciated.
>
> Thanks
>
> David
>
> The conf/hdfs-site.xml file contains:
> <configuration>
>        <property>
>                <name>fs.defaults.name</name>
>                <value>hdfs://localhost:8020</value>
>        </property>
> </configuration>
>
>
>
> $ start-dfs.sh
> starting namenode, logging to
> /Users/davidlary/hadoop-0.20.2/bin/../logs/hadoop-davidlary-namenode-129.110.31.40.out
> localhost: starting datanode, logging to
> /Users/davidlary/hadoop-0.20.2/bin/../logs/hadoop-davidlary-datanode-129.110.31.40.out
> localhost: starting secondarynamenode, logging to
> /Users/davidlary/hadoop-0.20.2/bin/../logs/hadoop-davidlary-secondarynamenode-129.110.31.40.out
> localhost: Exception in thread "main" java.lang.NullPointerException
> localhost:      at
> org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:134)
> localhost:      at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:156)
> localhost:      at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:160)
> localhost:      at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:131)
> localhost:      at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:115)
> localhost:      at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:469)
>
>
>
>

--0016e65b400c206fe104922d3ced--

From general-return-2148-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Oct 10 03:02:00 2010
Return-Path: <general-return-2148-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 90290 invoked from network); 10 Oct 2010 03:01:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Oct 2010 03:01:59 -0000
Received: (qmail 36607 invoked by uid 500); 10 Oct 2010 03:01:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36526 invoked by uid 500); 10 Oct 2010 03:01:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36518 invoked by uid 99); 10 Oct 2010 03:01:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Oct 2010 03:01:55 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of davidlary@gmail.com designates 209.85.161.176 as permitted sender)
Received: from [209.85.161.176] (HELO mail-gx0-f176.google.com) (209.85.161.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Oct 2010 03:01:48 +0000
Received: by gxk7 with SMTP id 7so1006887gxk.35
        for <general@hadoop.apache.org>; Sat, 09 Oct 2010 20:01:28 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:subject:mime-version
         :content-type:from:in-reply-to:date:cc:content-transfer-encoding
         :message-id:references:to:x-mailer;
        bh=6pBGICbX5piJE7zWRGQ4jSfKjiIOl1WQrCUhuMa2omM=;
        b=Zf0mspWBCCS5cBpyXD9bK+Lfzdl/3LfBavxnd673Qwm3Xp4Lm45CXWALaz5dRRv1wN
         HUPBjaRQxwsW6sE2PXqvk5V1pFKoZa5C87tg5YDYyenoML+ta0tzm+6V+viFuHrd3IjS
         cMThHQ22/hvdKOpfKmEMVYrL+B93N++UZ9h54=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=subject:mime-version:content-type:from:in-reply-to:date:cc
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=v7hE4VgywvHsIzbnczAgNZyii6tYmINSU+yf5Puwalv2kaWhRYR/P0IXkklRSfzFTx
         ePEnD7lXXx/d1mlSM3SKXf45R0eLLBEEvYrDghSzKQVyHicxsHpOU/uM/6SJ5WN3yUao
         C3ELh4ln7la8uH+t1Zy8KIv0YvPOB6+EjWUcg=
Received: by 10.151.27.6 with SMTP id e6mr1393056ybj.404.1286679687599;
        Sat, 09 Oct 2010 20:01:27 -0700 (PDT)
Received: from [10.0.1.5] (cpe-70-123-100-2.tx.res.rr.com [70.123.100.2])
        by mx.google.com with ESMTPS id q29sm4494587yba.6.2010.10.09.20.01.26
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Sat, 09 Oct 2010 20:01:26 -0700 (PDT)
Subject: Re: DFS error
Mime-Version: 1.0 (Apple Message framework v1081)
Content-Type: text/plain; charset=us-ascii
From: David Lary <davidlary@gmail.com>
In-Reply-To: <AANLkTi=YjmiK-0qE5M8C9_+CZay=RMCFGLr3PHKj0cBy@mail.gmail.com>
Date: Sat, 9 Oct 2010 22:01:24 -0500
Cc: David Lary <davidlary@gmail.com>
Content-Transfer-Encoding: quoted-printable
Message-Id: <EC9F6EDD-0FDA-4F46-9B00-3761D364894F@gmail.com>
References: <3C526026-7FFD-4E20-819B-58DE5738B577@gmail.com> <AANLkTi=YjmiK-0qE5M8C9_+CZay=RMCFGLr3PHKj0cBy@mail.gmail.com>
To: general@hadoop.apache.org,
 elton sky <eltonsky9404@gmail.com>
X-Mailer: Apple Mail (2.1081)

Thanks,=20

conf/masters contains:

localhost

David

On Oct 9, 2010, at 6:14 AM, elton sky wrote:

> looks like hadoop couldn't find the uri of your second name node. =
what's in
> your conf/master?
>=20
> On Sat, Oct 9, 2010 at 7:01 AM, David Lary <davidlary@gmail.com> =
wrote:
>=20
>> Please can someone help.
>>=20
>> Tried to set up pseudo-distributed mode for DFS. When I do =
start-dfs.sh I
>> get the error message below, and am not sure what it means. Any help =
would
>> be most appreciated.
>>=20
>> Thanks
>>=20
>> David
>>=20
>> The conf/hdfs-site.xml file contains:
>> <configuration>
>>       <property>
>>               <name>fs.defaults.name</name>
>>               <value>hdfs://localhost:8020</value>
>>       </property>
>> </configuration>
>>=20
>>=20
>>=20
>> $ start-dfs.sh
>> starting namenode, logging to
>> =
/Users/davidlary/hadoop-0.20.2/bin/../logs/hadoop-davidlary-namenode-129.1=
10.31.40.out
>> localhost: starting datanode, logging to
>> =
/Users/davidlary/hadoop-0.20.2/bin/../logs/hadoop-davidlary-datanode-129.1=
10.31.40.out
>> localhost: starting secondarynamenode, logging to
>> =
/Users/davidlary/hadoop-0.20.2/bin/../logs/hadoop-davidlary-secondarynamen=
ode-129.110.31.40.out
>> localhost: Exception in thread "main" java.lang.NullPointerException
>> localhost:      at
>> org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:134)
>> localhost:      at
>> =
org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:1=
56)
>> localhost:      at
>> =
org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:1=
60)
>> localhost:      at
>> =
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(Second=
aryNameNode.java:131)
>> localhost:      at
>> =
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryN=
ameNode.java:115)
>> localhost:      at
>> =
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNam=
eNode.java:469)
>>=20
>>=20
>>=20
>>=20


From general-return-2149-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 11 06:38:54 2010
Return-Path: <general-return-2149-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 74047 invoked from network); 11 Oct 2010 06:38:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Oct 2010 06:38:54 -0000
Received: (qmail 32911 invoked by uid 500); 11 Oct 2010 06:38:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32571 invoked by uid 500); 11 Oct 2010 06:38:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32563 invoked by uid 99); 11 Oct 2010 06:38:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Oct 2010 06:38:49 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=FREEMAIL_FROM,FROM_LOCAL_NOVOWEL,HK_RANDOM_ENVFROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of sssssssenator@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Oct 2010 06:38:43 +0000
Received: by wyb38 with SMTP id 38so2283643wyb.35
        for <general@hadoop.apache.org>; Sun, 10 Oct 2010 23:38:22 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=RaYJKTR8F6y+0uDZgncl4ndwh7ck9Ssy2oXcqFPuzXc=;
        b=skWwj7wAoXbTK0pIxB1wwGSR0rrmm33c4R8ea7+rDm4gaDTtCPrRep09aIB6Oh0ryb
         rU7pldmYqOWk6Ph1/S6XziORtPrf7nSPcivOSreodX1n5YjlG/+9iPoH8N+/7dmD9CVF
         8XlbCqbVYGQWAdELW0JSCLsGNLKrIkszpj+qs=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=GWp7Kba1J2O2Ppq+A/+3K6FHi2vkJlknvFaQmm8jH75A+1vdRSAVxdK60K9YfXHEZN
         ZMFZJV+JAbPgg7OD+jzYwXARfpOGNk3t8/Opej8JGThzO72TiZThGK4/hgwMM7IM8SlQ
         dLXcm8gEkaE6m056xUnuyth979Mkoy6hGxZt8=
MIME-Version: 1.0
Received: by 10.216.238.37 with SMTP id z37mr3292248weq.26.1286779101634; Sun,
 10 Oct 2010 23:38:21 -0700 (PDT)
Received: by 10.216.185.207 with HTTP; Sun, 10 Oct 2010 23:38:21 -0700 (PDT)
Date: Mon, 11 Oct 2010 12:08:21 +0530
Message-ID: <AANLkTi=ChkwrBMnGN-pVRw6ntxAhb2pJnP3B3W+XRvJe@mail.gmail.com>
Subject: Task tracker failed to start
From: vaibhav negi <sssssssenator@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/mixed; boundary=001517401fa441e1700492519b7f

--001517401fa441e1700492519b7f
Content-Type: multipart/alternative; boundary=001517401fa441e1620492519b7d

--001517401fa441e1620492519b7d
Content-Type: text/plain; charset=ISO-8859-1

Hi ,

I am using hadoop-0.20.2 in 2 Nodes cluster. But Tasktracker in dedicated
datanode failed to start.
Error i can see is :-

" INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Cannot initialize JVM
Metrics with processName=TaskTracker, sessionId= - already initialized "

Detailed error log is attached.
Where is the problem?

Thanks and Regards

Vaibhav Negi

--001517401fa441e1620492519b7d
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi ,<br><br>I am using hadoop-0.20.2 in 2 Nodes cluster. But Tasktracker in=
 dedicated datanode failed to start. <br>Error i can see is :- <br><br>&quo=
t; INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Cannot initialize JVM Met=
rics with processName=3DTaskTracker, sessionId=3D - already initialized &qu=
ot;<br>
<br>Detailed error log is attached.<br>Where is the problem?<br><br>Thanks =
and Regards<br><br>Vaibhav Negi<br>

--001517401fa441e1620492519b7d--
--001517401fa441e1700492519b7f--

From general-return-2150-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 11 11:52:43 2010
Return-Path: <general-return-2150-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 51226 invoked from network); 11 Oct 2010 11:52:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Oct 2010 11:52:42 -0000
Received: (qmail 12867 invoked by uid 500); 11 Oct 2010 11:52:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12745 invoked by uid 500); 11 Oct 2010 11:52:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12737 invoked by uid 99); 11 Oct 2010 11:52:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Oct 2010 11:52:37 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of hadoop.li@gmail.com designates 209.85.210.48 as permitted sender)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Oct 2010 11:52:30 +0000
Received: by pzk4 with SMTP id 4so1208896pzk.35
        for <general@hadoop.apache.org>; Mon, 11 Oct 2010 04:52:08 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=N4pCFgHl4kNK9QnguiPTml7tWMDzo0gAqTbNpdLKml8=;
        b=IMz9QP3SF3HuT2Svk5bktWOYHHDqqwiFkJuqYzbl608xJBir3VnUFObr9HUZJbKD07
         W1pfRJd7DA0fmE8ByNJorNK/YgjXFfEj7KfsSDgi0KJyhJKqIWiPvrZVx4lCYbcZ4oGF
         QjhURhDEbY4PIwlxM20Id9QESj8pBoUy1Pn40=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=E+b9zPV+v9MLh+gi0WcGfQ0+4idWNgEXQyK6rLR7UaE1C83pIdBDOVeRXpky2TJ3Kc
         EeC0h7HOKqNsKbJD0yUm4rlK4knLKdafqf6w7r/T5ea/DBw6VMccb0OdENaWlaIRZn9p
         elGR52haVDPT9gTzRxsVxW7iG8GDPTLecpOro=
MIME-Version: 1.0
Received: by 10.142.212.9 with SMTP id k9mr2499970wfg.77.1286797928584; Mon,
 11 Oct 2010 04:52:08 -0700 (PDT)
Received: by 10.142.210.7 with HTTP; Mon, 11 Oct 2010 04:52:08 -0700 (PDT)
In-Reply-To: <EC9F6EDD-0FDA-4F46-9B00-3761D364894F@gmail.com>
References: <3C526026-7FFD-4E20-819B-58DE5738B577@gmail.com>
	<AANLkTi=YjmiK-0qE5M8C9_+CZay=RMCFGLr3PHKj0cBy@mail.gmail.com>
	<EC9F6EDD-0FDA-4F46-9B00-3761D364894F@gmail.com>
Date: Mon, 11 Oct 2010 19:52:08 +0800
Message-ID: <AANLkTikE5H63kK_rsQYdBy5=Rizxa1V_rJmbLj2RU1Qb@mail.gmail.com>
Subject: Re: DFS error
From: Weiwei Li <hadoop.li@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd331ee6e5712049255fd55
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd331ee6e5712049255fd55
Content-Type: text/plain; charset=ISO-8859-1

How about change localhost to 129.110.31.40?

2010/10/10 David Lary <davidlary@gmail.com>

> Thanks,
>
> conf/masters contains:
>
> localhost
>
> David
>
> On Oct 9, 2010, at 6:14 AM, elton sky wrote:
>
> > looks like hadoop couldn't find the uri of your second name node. what's
> in
> > your conf/master?
> >
> > On Sat, Oct 9, 2010 at 7:01 AM, David Lary <davidlary@gmail.com> wrote:
> >
> >> Please can someone help.
> >>
> >> Tried to set up pseudo-distributed mode for DFS. When I do start-dfs.sh
> I
> >> get the error message below, and am not sure what it means. Any help
> would
> >> be most appreciated.
> >>
> >> Thanks
> >>
> >> David
> >>
> >> The conf/hdfs-site.xml file contains:
> >> <configuration>
> >>       <property>
> >>               <name>fs.defaults.name</name>
> >>               <value>hdfs://localhost:8020</value>
> >>       </property>
> >> </configuration>
> >>
> >>
> >>
> >> $ start-dfs.sh
> >> starting namenode, logging to
> >>
> /Users/davidlary/hadoop-0.20.2/bin/../logs/hadoop-davidlary-namenode-129.110.31.40.out
> >> localhost: starting datanode, logging to
> >>
> /Users/davidlary/hadoop-0.20.2/bin/../logs/hadoop-davidlary-datanode-129.110.31.40.out
> >> localhost: starting secondarynamenode, logging to
> >>
> /Users/davidlary/hadoop-0.20.2/bin/../logs/hadoop-davidlary-secondarynamenode-129.110.31.40.out
> >> localhost: Exception in thread "main" java.lang.NullPointerException
> >> localhost:      at
> >> org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:134)
> >> localhost:      at
> >>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:156)
> >> localhost:      at
> >>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:160)
> >> localhost:      at
> >>
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:131)
> >> localhost:      at
> >>
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:115)
> >> localhost:      at
> >>
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:469)
> >>
> >>
> >>
> >>
>
>

--000e0cd331ee6e5712049255fd55--

From general-return-2151-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 11 12:21:31 2010
Return-Path: <general-return-2151-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 61165 invoked from network); 11 Oct 2010 12:21:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Oct 2010 12:21:30 -0000
Received: (qmail 35112 invoked by uid 500); 11 Oct 2010 12:21:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34680 invoked by uid 500); 11 Oct 2010 12:21:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34662 invoked by uid 99); 11 Oct 2010 12:21:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Oct 2010 12:21:25 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of magawake@gmail.com designates 209.85.213.176 as permitted sender)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Oct 2010 12:21:18 +0000
Received: by yxg6 with SMTP id 6so1309689yxg.35
        for <general@hadoop.apache.org>; Mon, 11 Oct 2010 05:20:57 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=5WwQDbJ31VPuPbkZ6xb9PukjLT/cEOsBMHfyrfh7mEI=;
        b=UZUK2RK9kEHIk7DFq9SC0oCKIYLgGIBgnWPJi/HM8KNqk0JHpZJoKquy19E2fqtUmu
         /+2UisEsvIG+h4jgon6IFifrSOKpaWD5egsdasOzmx/aAzBrZ6L1tuaYqbZb9OjvIJei
         qshaR8SC252av5qezAWeqNBoI3foh4z5FHyNo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=XFPxqtHazBOh0yIoga350qxp/j7N631htYO83zrgq7itDxAbN53g5tTaz2wIFuZTtw
         EEtYVzUEJWJPbHxg9OcHXC0lJc5B2XOTTJW2tkqsO2qhcMkeoDVgYwqEXFoivFGyQC8g
         XXmwIbTt0RhYiNDlQdfWGVGecOczv9lvTkfvc=
MIME-Version: 1.0
Received: by 10.151.83.5 with SMTP id k5mr372884ybl.295.1286799656808; Mon, 11
 Oct 2010 05:20:56 -0700 (PDT)
Received: by 10.150.211.17 with HTTP; Mon, 11 Oct 2010 05:20:56 -0700 (PDT)
Date: Mon, 11 Oct 2010 08:20:56 -0400
Message-ID: <AANLkTi=bONk3wMFt918DV066RToe06gsUDbk1KBOXzH+@mail.gmail.com>
Subject: newbie setup question
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
X-Virus-Checked: Checked by ClamAV on apache.org

I am planning to setup Hadoop in our environment. We only want the
hdfs portion and not really interested in mapreduce.

I've been using,
http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Multi-Node_Cluster)
to setup my cluster and so far its going well. I was wondering if
there is any newer documentation or howto I should follow.

TIA

From general-return-2152-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 12 13:11:35 2010
Return-Path: <general-return-2152-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66463 invoked from network); 12 Oct 2010 13:11:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Oct 2010 13:11:34 -0000
Received: (qmail 68829 invoked by uid 500); 12 Oct 2010 13:11:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68687 invoked by uid 500); 12 Oct 2010 13:11:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68676 invoked by uid 99); 12 Oct 2010 13:11:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Oct 2010 13:11:30 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Oct 2010 13:11:23 +0000
Received: by wyb38 with SMTP id 38so3922939wyb.35
        for <general@hadoop.apache.org>; Tue, 12 Oct 2010 06:11:03 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.181.84 with SMTP id k62mr6774474wem.76.1286889062842; Tue,
 12 Oct 2010 06:11:02 -0700 (PDT)
Received: by 10.216.178.210 with HTTP; Tue, 12 Oct 2010 06:11:02 -0700 (PDT)
In-Reply-To: <AANLkTi=bONk3wMFt918DV066RToe06gsUDbk1KBOXzH+@mail.gmail.com>
References: <AANLkTi=bONk3wMFt918DV066RToe06gsUDbk1KBOXzH+@mail.gmail.com>
Date: Tue, 12 Oct 2010 06:11:02 -0700
Message-ID: <AANLkTikKJ_FTr95E1QLLWBnSN7Mcn3n-aMpnNDXqsp9d@mail.gmail.com>
Subject: Re: newbie setup question
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163642768574c15d04926b352a
X-Virus-Checked: Checked by ClamAV on apache.org

--00163642768574c15d04926b352a
Content-Type: text/plain; charset=ISO-8859-1

Hey,

To get started with the version of HDFS that's found in CDH3, see
https://docs.cloudera.com/display/DOC/Hadoop+Installation+Documentation+for+CDH3
.

Regards,
Jeff

On Mon, Oct 11, 2010 at 5:20 AM, Mag Gam <magawake@gmail.com> wrote:

> I am planning to setup Hadoop in our environment. We only want the
> hdfs portion and not really interested in mapreduce.
>
> I've been using,
>
> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Multi-Node_Cluster)
> to setup my cluster and so far its going well. I was wondering if
> there is any newer documentation or howto I should follow.
>
> TIA
>

--00163642768574c15d04926b352a--

From general-return-2153-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 13 04:01:26 2010
Return-Path: <general-return-2153-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50701 invoked from network); 13 Oct 2010 04:01:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Oct 2010 04:01:26 -0000
Received: (qmail 27695 invoked by uid 500); 13 Oct 2010 04:01:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26426 invoked by uid 500); 13 Oct 2010 04:01:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26403 invoked by uid 99); 13 Oct 2010 04:01:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 04:01:16 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [98.139.91.204] (HELO nm5-vm0.bullet.mail.sp2.yahoo.com) (98.139.91.204)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 13 Oct 2010 04:01:07 +0000
Received: from [98.139.91.63] by nm5.bullet.mail.sp2.yahoo.com with NNFMP; 13 Oct 2010 04:00:46 -0000
Received: from [98.139.91.10] by tm3.bullet.mail.sp2.yahoo.com with NNFMP; 13 Oct 2010 04:00:46 -0000
Received: from [127.0.0.1] by omp1010.mail.sp2.yahoo.com with NNFMP; 13 Oct 2010 04:00:46 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 205679.62479.bm@omp1010.mail.sp2.yahoo.com
Received: (qmail 64909 invoked by uid 60001); 13 Oct 2010 04:00:45 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1286942445; bh=K6ESYR+xC/rzVGOgXDghXbzvNHO+fm78n/aF7Leg32U=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=SkQ1QiHfRAWD4QwMo44ZyTxxitRo5iPaxPKM4lJATyp3qPvbyTrxp8xnLbpswQUihoK7VhLflqrz9NEEF7KztngMlKbYUtZ5yzE/5pqbGfj4H/mXNAjJGaqV4niMPznmRuhU91a1ov8YXPbwXmjjjT2T8hZbPZ//Xx0VoKPsuKs=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=FyB63MSw4FgJeHawsnpyun+vvx4BQNh46hbJ8RmiY6UxqCqOS6oyQRzFIihqFkIXo1+YRLn+a08iCUgCmbyBxgn60Vd4R8VFIyWWaG796Eugtcbcu2wao40D1ETBHx36avISdVwJ1/MbYETN8v7Ss82DzS3jjteMmwoWmp5hixs=;
Message-ID: <680770.64071.qm@web110202.mail.gq1.yahoo.com>
X-YMail-OSG: Gpm4hBAVM1k98D07_uPwpM7ChWsXRna09pLb6zhfHoHYzSG
 2KQe.6X8alLkeVny6UmfFaySjHR90IQ.ryrNVow3J48cMVZtVB1p0dORAdRa
 715Xo4NLfwiKRoLvA7lU5bTqpeDMsUtYUUZJ2.bLg36NoGj4MghoFYbdNoub
 rPk2Zh9AqNqLLvn4PwQ50lM7c6pu.CI_g5FRIXXvnAU9figElejOH1zOl6gA
 ZqkvhMoSFd094D5ulh8YCyVCwuW4sfx7cIoCFrzJQxGqpyK.h9fcuGQ6Db0J
 S_w4P4TP0k7uUlMnEKeTw6goY4k.z_oN9NIZp1jzbjz83gVnyyjS3ZIA-
Received: from [71.80.224.31] by web110202.mail.gq1.yahoo.com via HTTP; Tue, 12 Oct 2010 21:00:45 PDT
X-Mailer: YahooMailRC/504.5 YahooMailWebService/0.8.106.282862
Date: Tue, 12 Oct 2010 21:00:45 -0700 (PDT)
From: Roger Smith <rogersmith1711@yahoo.com>
Subject: Mailing list for posting a job opportunity
To: common-user@hadoop.apache.org, general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-417116068-1286942445=:64071"

--0-417116068-1286942445=:64071
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

I am starting a new cloud computing technology startup to develop a Hadoop-=
based =0Asoftware for CleanTech applications and looking for a developer wi=
th deep Hadoop =0Aexperience to join the founding team. I would like to kno=
w if Apache has a =0Amailing list to post opportunities of such nature, one=
 similar to Eclipse Job =0AForum mailing list. If yes, I would very much ap=
preciate a pointer. If not, I =0Awant to gauge level of interest from this =
community to start one (a job mailing =0Alist). =A0=A0=A0=0A=A0=0ARoger Smi=
th
--0-417116068-1286942445=:64071--

From general-return-2154-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 13 04:02:32 2010
Return-Path: <general-return-2154-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50981 invoked from network); 13 Oct 2010 04:02:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Oct 2010 04:02:32 -0000
Received: (qmail 32961 invoked by uid 500); 13 Oct 2010 04:02:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32038 invoked by uid 500); 13 Oct 2010 04:02:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32024 invoked by uid 99); 13 Oct 2010 04:02:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 04:02:27 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of paliwalashish@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 04:02:21 +0000
Received: by wyb38 with SMTP id 38so5028328wyb.35
        for <multiple recipients>; Tue, 12 Oct 2010 21:02:01 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:cc:content-type;
        bh=WrLY+CAFY2Yg1dZZiBsFeioBO2XTeBk3tAnQuE78w1Q=;
        b=RfvfklmnEb94Q3kclLNSIfRP2bMNEdt+Hu2t1QQwBXB2ugF754r1gnYPpJGXTYqxDj
         RQDmiJYO8dy61Rv5Fw+qg0noxujc4c/xULnibS21sr/t1/6u+j4iWqcSL0zvCVQHqoow
         037lAQgX1XOaQde3mBBRgEHn4IW2E9OkwiNT0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=aobqZeiir1PuwHbNsgCGYaVhtYY2FihA6tJfbxmvAzzo0NxONd0Nm8ONAjmc2gR0RZ
         yH/gDPFi25uGOkNObVhWUjRLVqg/E6IDlhEa4tkOgsJNw/qn+9lBbkBXWefQDeabKCMc
         IjkST35c3Gr1Bns2Eo22lOmiuURsTWmHOrBDE=
MIME-Version: 1.0
Received: by 10.216.22.203 with SMTP id t53mr7689210wet.37.1286942520966; Tue,
 12 Oct 2010 21:02:00 -0700 (PDT)
Received: by 10.216.73.14 with HTTP; Tue, 12 Oct 2010 21:02:00 -0700 (PDT)
In-Reply-To: <680770.64071.qm@web110202.mail.gq1.yahoo.com>
References: <680770.64071.qm@web110202.mail.gq1.yahoo.com>
Date: Wed, 13 Oct 2010 09:32:00 +0530
Message-ID: <AANLkTikBYt9Y4A2t4at0U37vPJdxUEncYbTXSas7+PEc@mail.gmail.com>
Subject: Re: Mailing list for posting a job opportunity
From: Ashish <paliwalashish@gmail.com>
To: general@hadoop.apache.org
Cc: common-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

jobs@apache.org

On Wed, Oct 13, 2010 at 9:30 AM, Roger Smith <rogersmith1711@yahoo.com> wrote:
> I am starting a new cloud computing technology startup to develop a Hadoop-based
> software for CleanTech applications and looking for a developer with deep Hadoop
> experience to join the founding team. I would like to know if Apache has a
> mailing list to post opportunities of such nature, one similar to Eclipse Job
> Forum mailing list. If yes, I would very much appreciate a pointer. If not, I
> want to gauge level of interest from this community to start one (a job mailing
> list).
>
> Roger Smith



-- 
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal

From general-return-2155-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 13 12:58:05 2010
Return-Path: <general-return-2155-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 32858 invoked from network); 13 Oct 2010 12:58:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Oct 2010 12:58:00 -0000
Received: (qmail 2378 invoked by uid 500); 13 Oct 2010 12:57:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1916 invoked by uid 500); 13 Oct 2010 12:57:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1267 invoked by uid 99); 13 Oct 2010 12:57:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 12:57:54 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [217.72.192.221] (HELO fmmailgate01.web.de) (217.72.192.221)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 12:57:47 +0000
Received:  from smtp04.web.de  ( [172.20.0.225])
	by fmmailgate01.web.de (Postfix) with ESMTP id 07FC016EC84D9
	for <general@hadoop.apache.org>; Wed, 13 Oct 2010 14:57:27 +0200 (CEST)
Received: from [89.246.34.1] (helo=[192.168.0.58])
	by smtp04.web.de with asmtp (WEB.DE 4.110 #24)
	id 1P60ta-0005ky-00
	for general@hadoop.apache.org; Wed, 13 Oct 2010 14:57:26 +0200
User-Agent: Microsoft-Entourage/12.23.0.091001
Date: Wed, 13 Oct 2010 14:57:25 +0200
Subject: Architecture
From: Andre Reiter <a.reiter@web.de>
To: <general@hadoop.apache.org>
Message-ID: <C8DB7955.17EB%a.reiter@web.de>
Thread-Topic: Architecture
Thread-Index: Actq1i6+hc039LXzJkiOs5oPzinFPA==
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
Sender: a.reiter@web.de
X-Sender: a.reiter@web.de
X-Provags-ID: V01U2FsdGVkX18GXCbMHRQUIhOBAeK0UVlOe7mD4oiwI0j0QfnL
	jS4xS0vub1PUxGmEW+NM1BO1s1QMR3QBIB9WF0A5epNU+Ex7Dl
	AiKN+mPjQ=
X-Virus-Checked: Checked by ClamAV on apache.org

Hello everybody,

i'm evaluating hadoop for a new platform.
the application is a web based application, and the challange is to handle
requests up to 1.000.000 times per second. this can actually be solved by
load ballanced webservers.
The problem now is to persist user data at real time. Let's say every
request appends some data to the user record, i.e. 100 bytes are appended.
The record would have an identifier placed in the cookie.

The problem now is to look up the user record for every request at runtime.
Is it possible to get the user record with the information about all former
requests in a short time i.e. 50 ms ???

i hove no experience with hadoop yet

every help would be very appreciated
thanks in advance

best regards
andre



From general-return-2156-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 13 13:10:06 2010
Return-Path: <general-return-2156-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 40583 invoked from network); 13 Oct 2010 13:10:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Oct 2010 13:10:05 -0000
Received: (qmail 18517 invoked by uid 500); 13 Oct 2010 13:10:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18350 invoked by uid 500); 13 Oct 2010 13:10:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18342 invoked by uid 99); 13 Oct 2010 13:10:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 13:10:01 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of timrobertson100@gmail.com designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 13:09:55 +0000
Received: by ewy28 with SMTP id 28so1892036ewy.35
        for <general@hadoop.apache.org>; Wed, 13 Oct 2010 06:09:33 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=o4IhwstMQbMGI2FT24+a2RjPZ88mr9ZSIw05XVLL36U=;
        b=StNBa8ftXVEcmJMynL+dwh3n5RLWjaf2rvZPqUCahEDuzjVisGxyBhtLJM5NaftEH8
         RFzNhfYA29/sUEh7i6W1vITI/Ni9looWkd3NruzS4xcNm2L2ksy6uWO2EGpg/Tp+gmVR
         NxjcYuWsYOhuIHN6bGh5FQJ/Hkykz9ibAe5yg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=pwIt/VxsYmr0UTGZqMjn7+LoweQb0qvhMnragDo/904RfZkzdqaoK9RCBaUVCUocy/
         1eLqwCOlNGWd1fWAUGEzwBXi2BPwxrwcli7KBMlRRNFF9t8IZ24HR5277oaBTH+W5/8r
         uTUq9Mu37U0Qprhazi8hkjgvUMAL2onRMsrRM=
MIME-Version: 1.0
Received: by 10.213.19.8 with SMTP id y8mr542677eba.94.1286975373666; Wed, 13
 Oct 2010 06:09:33 -0700 (PDT)
Received: by 10.14.127.137 with HTTP; Wed, 13 Oct 2010 06:09:33 -0700 (PDT)
In-Reply-To: <C8DB7955.17EB%a.reiter@web.de>
References: <C8DB7955.17EB%a.reiter@web.de>
Date: Wed, 13 Oct 2010 15:09:33 +0200
Message-ID: <AANLkTikK0=epR6TupiS1Vv+3ZwxurfM5QbKDX4L341=r@mail.gmail.com>
Subject: Re: Architecture
From: Tim Robertson <timrobertson100@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

This sounds like you should be looking to HBase.  HBase is a data
store that builds on top of Hadoop to meet those kind of real time
storage and retrieval requirements.

HTH,
Tim

On Wed, Oct 13, 2010 at 2:57 PM, Andre Reiter <a.reiter@web.de> wrote:
> Hello everybody,
>
> i'm evaluating hadoop for a new platform.
> the application is a web based application, and the challange is to handle
> requests up to 1.000.000 times per second. this can actually be solved by
> load ballanced webservers.
> The problem now is to persist user data at real time. Let's say every
> request appends some data to the user record, i.e. 100 bytes are appended.
> The record would have an identifier placed in the cookie.
>
> The problem now is to look up the user record for every request at runtime.
> Is it possible to get the user record with the information about all former
> requests in a short time i.e. 50 ms ???
>
> i hove no experience with hadoop yet
>
> every help would be very appreciated
> thanks in advance
>
> best regards
> andre
>
>
>

From general-return-2157-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 13 14:03:44 2010
Return-Path: <general-return-2157-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59297 invoked from network); 13 Oct 2010 14:03:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Oct 2010 14:03:43 -0000
Received: (qmail 13667 invoked by uid 500); 13 Oct 2010 14:03:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13189 invoked by uid 500); 13 Oct 2010 14:03:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13168 invoked by uid 99); 13 Oct 2010 14:03:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 14:03:38 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jon.creasy@announcemedia.com designates 67.192.241.160 as permitted sender)
Received: from [67.192.241.160] (HELO smtp160.dfw.emailsrvr.com) (67.192.241.160)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 14:03:31 +0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by smtp16.relay.dfw1a.emailsrvr.com (SMTP Server) with ESMTP id 5D20640CEC
	for <general@hadoop.apache.org>; Wed, 13 Oct 2010 10:03:10 -0400 (EDT)
X-Orig-To: general@hadoop.apache.org
X-Virus-Scanned: OK
Received: from smtp192.mex07a.mlsrvr.com (smtp192.mex07a.mlsrvr.com [67.192.133.192])
	by smtp16.relay.dfw1a.emailsrvr.com (SMTP Server) with ESMTPS id 2A9C740D1E;
	Wed, 13 Oct 2010 10:03:08 -0400 (EDT)
Received: from DFW1MBX20.mex07a.mlsrvr.com ([169.254.2.105]) by
 207037-HUB09.mex07a.mlsrvr.com ([192.168.1.202]) with mapi; Wed, 13 Oct 2010
 09:03:07 -0500
From: Jonathan Creasy <jon.creasy@announcemedia.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 13 Oct 2010 09:02:40 -0500
Subject: Re: Architecture
Thread-Topic: Architecture
Thread-Index: Actq31whAvM7GV4lRdiRdVunjAYwIQ==
Message-ID: <80D1F44F-5D35-4764-8911-94907EAEAA36@announcemedia.com>
References: <C8DB7955.17EB%a.reiter@web.de>
 <AANLkTikK0=epR6TupiS1Vv+3ZwxurfM5QbKDX4L341=r@mail.gmail.com>
In-Reply-To: <AANLkTikK0=epR6TupiS1Vv+3ZwxurfM5QbKDX4L341=r@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Not to sound argumentative but because perhaps I have misunderstood, Hbase =
doesn't seem well suited to realtime requests at all. Perhaps you could ela=
borate on the architecture you would use to support this application?

data starts here, goes here, this is done, it is accessed by a webserver he=
re, etc

Thank you!

On Oct 13, 2010, at 8:10 AM, "Tim Robertson" <timrobertson100@gmail.com> wr=
ote:

> This sounds like you should be looking to HBase.  HBase is a data
> store that builds on top of Hadoop to meet those kind of real time
> storage and retrieval requirements.
>=20
> HTH,
> Tim
>=20
> On Wed, Oct 13, 2010 at 2:57 PM, Andre Reiter <a.reiter@web.de> wrote:
>> Hello everybody,
>>=20
>> i'm evaluating hadoop for a new platform.
>> the application is a web based application, and the challange is to hand=
le
>> requests up to 1.000.000 times per second. this can actually be solved b=
y
>> load ballanced webservers.
>> The problem now is to persist user data at real time. Let's say every
>> request appends some data to the user record, i.e. 100 bytes are appende=
d.
>> The record would have an identifier placed in the cookie.
>>=20
>> The problem now is to look up the user record for every request at runti=
me.
>> Is it possible to get the user record with the information about all for=
mer
>> requests in a short time i.e. 50 ms ???
>>=20
>> i hove no experience with hadoop yet
>>=20
>> every help would be very appreciated
>> thanks in advance
>>=20
>> best regards
>> andre
>>=20
>>=20
>>=20

From general-return-2158-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 13 14:38:05 2010
Return-Path: <general-return-2158-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 74264 invoked from network); 13 Oct 2010 14:38:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Oct 2010 14:38:04 -0000
Received: (qmail 73068 invoked by uid 500); 13 Oct 2010 14:38:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72701 invoked by uid 500); 13 Oct 2010 14:38:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72693 invoked by uid 99); 13 Oct 2010 14:38:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 14:38:00 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of lars.francke@gmail.com designates 74.125.83.48 as permitted sender)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 14:37:52 +0000
Received: by gwj23 with SMTP id 23so637302gwj.35
        for <general@hadoop.apache.org>; Wed, 13 Oct 2010 07:37:31 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type;
        bh=e19S1h+wlVY6obpdVPEGODQK9EDIFarSIuVJZqDr26Y=;
        b=dDT+1AvxCgjTBbbLW6lTqGAMaY1eMsOqJwP2shtDlaR+/DyZzMoQ0agwgqA2hk8Mww
         85mJr9jW4zkJKnERgnbtjSCo418BvQ4wneeNBNjD3JlxLaP8zsyEggiGxrZQy+/0FehU
         JzvdVGM0qo3LtcI6wxcOsUf5nS0gTg2alUXes=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=jYcths4RBp7RFiE0sjtbUKKFx9k0YTsHW8ZX5p9IIAqqcFEApJZEPpRO9TIBO/YlQO
         iyfeeyXjdBk6dt202BZQTNl9e/cTqAb3oRCrIpsl6j7UDKOBqDrkhMiA+wK7Y+U5s0qe
         QsNm2aHzDr3Aeqn0O96vSfvrg2GYUxZ0PK/ok=
Received: by 10.150.131.16 with SMTP id e16mr1114036ybd.415.1286980649315;
 Wed, 13 Oct 2010 07:37:29 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.150.212.20 with HTTP; Wed, 13 Oct 2010 07:36:47 -0700 (PDT)
In-Reply-To: <80D1F44F-5D35-4764-8911-94907EAEAA36@announcemedia.com>
References: <C8DB7955.17EB%a.reiter@web.de> <AANLkTikK0=epR6TupiS1Vv+3ZwxurfM5QbKDX4L341=r@mail.gmail.com>
 <80D1F44F-5D35-4764-8911-94907EAEAA36@announcemedia.com>
From: Lars Francke <lars.francke@gmail.com>
Date: Wed, 13 Oct 2010 16:36:47 +0200
Message-ID: <AANLkTin0s0pg==Y35oGpdinia91LU7Ea8LULrNum-Nng@mail.gmail.com>
Subject: Re: Architecture
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
X-Virus-Checked: Checked by ClamAV on apache.org

> Not to sound argumentative but because perhaps I have misunderstood, Hbase doesn't seem well suited to realtime requests at all. Perhaps you could elaborate on the architecture you would use to support this application?

Quite the opposite. HBase is all about realtime requests. It's
optimized for it. Here's the description from the homepage:
"HBase is the Hadoop database. Use it when you need random, realtime
read/write access to your Big Data. This project's goal is the hosting
of very large tables -- billions of rows X millions of columns -- atop
clusters of commodity hardware.

Cheers,
Lars

From general-return-2159-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 13 14:46:29 2010
Return-Path: <general-return-2159-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78179 invoked from network); 13 Oct 2010 14:46:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Oct 2010 14:46:28 -0000
Received: (qmail 91837 invoked by uid 500); 13 Oct 2010 14:46:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91705 invoked by uid 500); 13 Oct 2010 14:46:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91690 invoked by uid 99); 13 Oct 2010 14:46:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 14:46:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 13 Oct 2010 14:46:24 +0000
Received: (qmail 78127 invoked by uid 99); 13 Oct 2010 14:46:03 -0000
Received: from localhost.apache.org (HELO mail-bw0-f48.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 14:46:03 +0000
Received: by bwz5 with SMTP id 5so3171107bwz.35
        for <multiple recipients>; Wed, 13 Oct 2010 07:46:01 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.131.200 with SMTP id y8mr7409433bks.107.1286981161211;
 Wed, 13 Oct 2010 07:46:01 -0700 (PDT)
Received: by 10.204.77.11 with HTTP; Wed, 13 Oct 2010 07:46:01 -0700 (PDT)
In-Reply-To: <AANLkTin0s0pg==Y35oGpdinia91LU7Ea8LULrNum-Nng@mail.gmail.com>
References: <C8DB7955.17EB%a.reiter@web.de>
	<AANLkTikK0=epR6TupiS1Vv+3ZwxurfM5QbKDX4L341=r@mail.gmail.com>
	<80D1F44F-5D35-4764-8911-94907EAEAA36@announcemedia.com>
	<AANLkTin0s0pg==Y35oGpdinia91LU7Ea8LULrNum-Nng@mail.gmail.com>
Date: Wed, 13 Oct 2010 07:46:01 -0700
Message-ID: <AANLkTikibABvQSCkaukmV9e+yXhcJ6VgBCEpvE6uG7nD@mail.gmail.com>
Subject: Re: Architecture
From: "Owen O'Malley" <omalley@apache.org>
To: common-user@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8

Here is a presentation from Hadoop Summit 2009 "HBase goes Realtime"
that gives numbers for latency with HBase. Redirecting to common-user.

http://bit.ly/aJEwYj

-- Owen

From general-return-2160-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 13 14:55:24 2010
Return-Path: <general-return-2160-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81307 invoked from network); 13 Oct 2010 14:55:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Oct 2010 14:55:23 -0000
Received: (qmail 18621 invoked by uid 500); 13 Oct 2010 14:55:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18549 invoked by uid 500); 13 Oct 2010 14:55:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18540 invoked by uid 99); 13 Oct 2010 14:55:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 14:55:20 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of dsikar@gmail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 14:55:13 +0000
Received: by iwn3 with SMTP id 3so8884251iwn.35
        for <general@hadoop.apache.org>; Wed, 13 Oct 2010 07:54:52 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=SZbgjxQ5RtExSHiEQ7O0+ayDjzr2Jdz8XZY7KC4cBLg=;
        b=gdlqek69pG60ZDpPgAWGVSl02u3IlkE+4yw8E39ZD274iPk8qvRAJYgZRGYlh1ymqm
         oxznQqaW99hPZBiZYiW4AsWtMOgBqqUeCpx8WGmRUwcCSTFaam2hFSQsY75h9IE7YIaw
         pYMvS6Q2G7WWrvlAovERZB76OXbf8P/6Rooxc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=QMIkzJqx8s36K68g6YVTxQpuR/kkyBKk4MNl7mjWQAa4jhfz5EI9jjo1LKB95bCkOB
         2mXlZVR5JNNOtpL2ZGsLMSJUQ8NehXOMwbY46yGJ6Ol5b5hpZbvrX7Q8URSua2ztKL7h
         nYeeQu7d59KEXc0mSrBaGoQUNJO1yOoQsJX44=
MIME-Version: 1.0
Received: by 10.231.16.76 with SMTP id n12mr1808998iba.97.1286981691766; Wed,
 13 Oct 2010 07:54:51 -0700 (PDT)
Received: by 10.231.172.211 with HTTP; Wed, 13 Oct 2010 07:54:51 -0700 (PDT)
In-Reply-To: <80D1F44F-5D35-4764-8911-94907EAEAA36@announcemedia.com>
References: <C8DB7955.17EB%a.reiter@web.de>
	<AANLkTikK0=epR6TupiS1Vv+3ZwxurfM5QbKDX4L341=r@mail.gmail.com>
	<80D1F44F-5D35-4764-8911-94907EAEAA36@announcemedia.com>
Date: Wed, 13 Oct 2010 17:54:51 +0300
Message-ID: <AANLkTikPec8pmgC3f0xGH0SGwAJmaz+KmBTfc=J=FrZA@mail.gmail.com>
Subject: Re: Architecture
From: daniel sikar <dsikar@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Perhaps in-memory data grids could be considered?

On 13 October 2010 17:02, Jonathan Creasy <jon.creasy@announcemedia.com> wr=
ote:
> Not to sound argumentative but because perhaps I have misunderstood, Hbas=
e doesn't seem well suited to realtime requests at all. Perhaps you could e=
laborate on the architecture you would use to support this application?
>
> data starts here, goes here, this is done, it is accessed by a webserver =
here, etc
>
> Thank you!
>
> On Oct 13, 2010, at 8:10 AM, "Tim Robertson" <timrobertson100@gmail.com> =
wrote:
>
>> This sounds like you should be looking to HBase. =A0HBase is a data
>> store that builds on top of Hadoop to meet those kind of real time
>> storage and retrieval requirements.
>>
>> HTH,
>> Tim
>>
>> On Wed, Oct 13, 2010 at 2:57 PM, Andre Reiter <a.reiter@web.de> wrote:
>>> Hello everybody,
>>>
>>> i'm evaluating hadoop for a new platform.
>>> the application is a web based application, and the challange is to han=
dle
>>> requests up to 1.000.000 times per second. this can actually be solved =
by
>>> load ballanced webservers.
>>> The problem now is to persist user data at real time. Let's say every
>>> request appends some data to the user record, i.e. 100 bytes are append=
ed.
>>> The record would have an identifier placed in the cookie.
>>>
>>> The problem now is to look up the user record for every request at runt=
ime.
>>> Is it possible to get the user record with the information about all fo=
rmer
>>> requests in a short time i.e. 50 ms ???
>>>
>>> i hove no experience with hadoop yet
>>>
>>> every help would be very appreciated
>>> thanks in advance
>>>
>>> best regards
>>> andre
>>>
>>>
>>>
>

From general-return-2161-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 13 23:12:39 2010
Return-Path: <general-return-2161-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21672 invoked from network); 13 Oct 2010 23:12:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Oct 2010 23:12:39 -0000
Received: (qmail 74357 invoked by uid 500); 13 Oct 2010 23:12:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74244 invoked by uid 500); 13 Oct 2010 23:12:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74236 invoked by uid 99); 13 Oct 2010 23:12:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 23:12:36 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mmoores@real.com designates 207.188.23.7 as permitted sender)
Received: from [207.188.23.7] (HELO cir-el.real.com) (207.188.23.7)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 23:12:31 +0000
Received: from seacas01.corp.real.com ([::ffff:192.168.139.56])
  (TLS: TLSv1/SSLv3,128bits,AES128-SHA)
  by cir-el.real.com with esmtp; Wed, 13 Oct 2010 16:12:11 -0700
  id 001FC1A1.4CB63CCB.00000ECB
Received: from seambx.corp.real.com ([fe80::2d15:fda7:b3b8:e268]) by
 seacas01.corp.real.com ([192.168.139.56]) with mapi; Wed, 13 Oct 2010
 16:12:10 -0700
From: Michael Moores <mmoores@real.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 13 Oct 2010 16:12:09 -0700
Subject: Specifying the InputFormat class that exists in a JAR on the hdfs
Thread-Topic: Specifying the InputFormat class that exists in a JAR on the
 hdfs
Thread-Index: ActrLBA21b1czM9DQzSf8HOBnG1wdQ==
Message-ID: <B2EA53A1-B10A-4FE8-84E4-313EFEE4CF39@real.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_B2EA53A1B10A4FE884E4313EFEE4CF39realcom_"
MIME-Version: 1.0

--_000_B2EA53A1B10A4FE884E4313EFEE4CF39realcom_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

I have specified my InputFormat to be the cassandra ColumnFamilyInputFormat=
, and also
added the cassandra JAR to my classpath via a call to DistributedCache.addF=
ileToClassPath().
The JAR exists on the HDFS.
When I run my jar I get  java.lang.NoClassDefFoundError: org/apache/cassand=
ra/hadoop/ColumnFamilyInputFormat at the line that
makes the job.setInputFormatClass() call.

I execute the job with "hadoop jar <myjar>".

Will I need to put my cassandra JAR on each machine and add it to the JVM s=
tartup options???

Here is a code snippet:

public class MyStats extends Configured implements Tool {
...
   public static void main(String[] args) throws Exception {
        // Let ToolRunner handle generic command-line options
        Configuration configuration =3D new Configuration();
        Path path =3D new Path("/user/hadoop/profilestats/cassandra-0.7.0-b=
eta2.jar");
        log.info("main: adding jars...");
        DistributedCache.addFileToClassPath(path, configuration);





        ToolRunner.run(configuration, new MyStats(), args);
        System.exit(0);
    }

   public int run(String[] args) throws Exception {
      Job job =3D new Job(getConf(), "myjob");
      job.setInputFormatClass(org.apache.cassandra.hadoop.ColumnFamilyInput=
Format.class);
      ..
      job.waitForCompletion(true);
   }


FILE LISTING from HDFS:

[hadoop@kv-app02 ~]$ hadoop dfs -lsr
10/10/13 14:57:47 INFO security.Groups: Group mapping impl=3Dorg.apache.had=
oop.security.ShellBasedUnixGroupsMapping; cacheTimeout=3D300000
10/10/13 14:57:48 WARN conf.Configuration: mapred.task.id is deprecated. In=
stead, use mapreduce.task.attempt.id
drwxr-xr-x   - hadoop supergroup          0 2010-10-13 14:34 /user/hadoop/p=
rofilestats
-rw-r--r--   3 hadoop supergroup    1841467 2010-10-13 14:34 /user/hadoop/p=
rofilestats/cassandra-0.7.0-beta2.jar

--_000_B2EA53A1B10A4FE884E4313EFEE4CF39realcom_--

From general-return-2162-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 13 23:16:15 2010
Return-Path: <general-return-2162-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24875 invoked from network); 13 Oct 2010 23:16:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Oct 2010 23:16:14 -0000
Received: (qmail 78429 invoked by uid 500); 13 Oct 2010 23:16:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78383 invoked by uid 500); 13 Oct 2010 23:16:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78375 invoked by uid 99); 13 Oct 2010 23:16:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 23:16:13 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mmoores@real.com designates 207.188.23.7 as permitted sender)
Received: from [207.188.23.7] (HELO cir-el.real.com) (207.188.23.7)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 23:16:06 +0000
Received: from seacas01.corp.real.com ([::ffff:192.168.139.56])
  (TLS: TLSv1/SSLv3,128bits,AES128-SHA)
  by cir-el.real.com with esmtp; Wed, 13 Oct 2010 16:15:45 -0700
  id 001FC18F.4CB63DA1.0000163E
Received: from seambx.corp.real.com ([fe80::2d15:fda7:b3b8:e268]) by
 seacas01.corp.real.com ([192.168.139.56]) with mapi; Wed, 13 Oct 2010
 16:15:44 -0700
From: Michael Moores <mmoores@real.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 13 Oct 2010 16:15:43 -0700
Subject: Re: Specifying the InputFormat class that exists in a JAR on the
 hdfs
Thread-Topic: Specifying the InputFormat class that exists in a JAR on the
 hdfs
Thread-Index: ActrLI/clJ+XArMuTBaK1xfxJjkT9w==
Message-ID: <2EB7C48D-7640-4B76-8135-9120E39031A6@real.com>
References: <B2EA53A1-B10A-4FE8-84E4-313EFEE4CF39@real.com>
In-Reply-To: <B2EA53A1-B10A-4FE8-84E4-313EFEE4CF39@real.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

fyi- I also tried thr archive version--=20

calling DistributedCache.addArchiveToClassPath(path, configuration);

On Oct 13, 2010, at 4:12 PM, Michael Moores wrote:

> I have specified my InputFormat to be the cassandra ColumnFamilyInputForm=
at, and also
> added the cassandra JAR to my classpath via a call to DistributedCache.ad=
dFileToClassPath().
> The JAR exists on the HDFS.
> When I run my jar I get  java.lang.NoClassDefFoundError: org/apache/cassa=
ndra/hadoop/ColumnFamilyInputFormat at the line that
> makes the job.setInputFormatClass() call.
>=20
> I execute the job with "hadoop jar <myjar>".
>=20
> Will I need to put my cassandra JAR on each machine and add it to the JVM=
 startup options???
>=20
> Here is a code snippet:
>=20
> public class MyStats extends Configured implements Tool {
> ...
>   public static void main(String[] args) throws Exception {
>        // Let ToolRunner handle generic command-line options
>        Configuration configuration =3D new Configuration();
>        Path path =3D new Path("/user/hadoop/profilestats/cassandra-0.7.0-=
beta2.jar");
>        log.info("main: adding jars...");
>        DistributedCache.addFileToClassPath(path, configuration);
>=20
>=20
>=20
>=20
>=20
>        ToolRunner.run(configuration, new MyStats(), args);
>        System.exit(0);
>    }
>=20
>   public int run(String[] args) throws Exception {
>      Job job =3D new Job(getConf(), "myjob");
>      job.setInputFormatClass(org.apache.cassandra.hadoop.ColumnFamilyInpu=
tFormat.class);
>      ..
>      job.waitForCompletion(true);
>   }
>=20
>=20
> FILE LISTING from HDFS:
>=20
> [hadoop@kv-app02 ~]$ hadoop dfs -lsr
> 10/10/13 14:57:47 INFO security.Groups: Group mapping impl=3Dorg.apache.h=
adoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=3D300000
> 10/10/13 14:57:48 WARN conf.Configuration: mapred.task.id is deprecated. =
Instead, use mapreduce.task.attempt.id
> drwxr-xr-x   - hadoop supergroup          0 2010-10-13 14:34 /user/hadoop=
/profilestats
> -rw-r--r--   3 hadoop supergroup    1841467 2010-10-13 14:34 /user/hadoop=
/profilestats/cassandra-0.7.0-beta2.jar


From general-return-2163-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 13 23:22:07 2010
Return-Path: <general-return-2163-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29417 invoked from network); 13 Oct 2010 23:22:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Oct 2010 23:22:06 -0000
Received: (qmail 89092 invoked by uid 500); 13 Oct 2010 23:22:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89044 invoked by uid 500); 13 Oct 2010 23:22:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89036 invoked by uid 99); 13 Oct 2010 23:22:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 23:22:05 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of shrijeet@rocketfuelinc.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 23:22:00 +0000
Received: by qwf7 with SMTP id 7so1493192qwf.35
        for <general@hadoop.apache.org>; Wed, 13 Oct 2010 16:21:39 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.236.133 with SMTP id kk5mr8136069qcb.191.1287012099582;
 Wed, 13 Oct 2010 16:21:39 -0700 (PDT)
Received: by 10.229.250.21 with HTTP; Wed, 13 Oct 2010 16:21:39 -0700 (PDT)
In-Reply-To: <2EB7C48D-7640-4B76-8135-9120E39031A6@real.com>
References: <B2EA53A1-B10A-4FE8-84E4-313EFEE4CF39@real.com>
	<2EB7C48D-7640-4B76-8135-9120E39031A6@real.com>
Date: Wed, 13 Oct 2010 16:21:39 -0700
Message-ID: <AANLkTim=PWyk_Pp4rC4uJ3S-bYP1_Pk9w3yoSgajqmWX@mail.gmail.com>
Subject: Re: Specifying the InputFormat class that exists in a JAR on the hdfs
From: Shrijeet Paliwal <shrijeet@rocketfuel.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64caedc047264049287db65

--0016e64caedc047264049287db65
Content-Type: text/plain; charset=ISO-8859-1

How about adding it to HADOOP_CLASSPATH if not already.

On Wed, Oct 13, 2010 at 4:15 PM, Michael Moores <mmoores@real.com> wrote:

> fyi- I also tried thr archive version--
>
> calling DistributedCache.addArchiveToClassPath(path, configuration);
>
> On Oct 13, 2010, at 4:12 PM, Michael Moores wrote:
>
> > I have specified my InputFormat to be the cassandra
> ColumnFamilyInputFormat, and also
> > added the cassandra JAR to my classpath via a call to
> DistributedCache.addFileToClassPath().
> > The JAR exists on the HDFS.
> > When I run my jar I get  java.lang.NoClassDefFoundError:
> org/apache/cassandra/hadoop/ColumnFamilyInputFormat at the line that
> > makes the job.setInputFormatClass() call.
> >
> > I execute the job with "hadoop jar <myjar>".
> >
> > Will I need to put my cassandra JAR on each machine and add it to the JVM
> startup options???
> >
> > Here is a code snippet:
> >
> > public class MyStats extends Configured implements Tool {
> > ...
> >   public static void main(String[] args) throws Exception {
> >        // Let ToolRunner handle generic command-line options
> >        Configuration configuration = new Configuration();
> >        Path path = new
> Path("/user/hadoop/profilestats/cassandra-0.7.0-beta2.jar");
> >        log.info("main: adding jars...");
> >        DistributedCache.addFileToClassPath(path, configuration);
> >
> >
> >
> >
> >
> >        ToolRunner.run(configuration, new MyStats(), args);
> >        System.exit(0);
> >    }
> >
> >   public int run(String[] args) throws Exception {
> >      Job job = new Job(getConf(), "myjob");
> >
>  job.setInputFormatClass(org.apache.cassandra.hadoop.ColumnFamilyInputFormat.class);
> >      ..
> >      job.waitForCompletion(true);
> >   }
> >
> >
> > FILE LISTING from HDFS:
> >
> > [hadoop@kv-app02 ~]$ hadoop dfs -lsr
> > 10/10/13 14:57:47 INFO security.Groups: Group mapping
> impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
> cacheTimeout=300000
> > 10/10/13 14:57:48 WARN conf.Configuration: mapred.task.id is deprecated.
> Instead, use mapreduce.task.attempt.id
> > drwxr-xr-x   - hadoop supergroup          0 2010-10-13 14:34
> /user/hadoop/profilestats
> > -rw-r--r--   3 hadoop supergroup    1841467 2010-10-13 14:34
> /user/hadoop/profilestats/cassandra-0.7.0-beta2.jar
>
>

--0016e64caedc047264049287db65--

From general-return-2164-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 13 23:39:05 2010
Return-Path: <general-return-2164-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34263 invoked from network); 13 Oct 2010 23:39:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Oct 2010 23:39:05 -0000
Received: (qmail 12934 invoked by uid 500); 13 Oct 2010 23:39:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12864 invoked by uid 500); 13 Oct 2010 23:39:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12856 invoked by uid 99); 13 Oct 2010 23:39:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 23:39:03 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mmoores@real.com designates 207.188.23.7 as permitted sender)
Received: from [207.188.23.7] (HELO cir-el.real.com) (207.188.23.7)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 23:38:58 +0000
Received: from seacas01.corp.real.com ([::ffff:192.168.139.56])
  (TLS: TLSv1/SSLv3,128bits,AES128-SHA)
  by cir-el.real.com with esmtp; Wed, 13 Oct 2010 16:38:38 -0700
  id 001FC19E.4CB642FE.000033D7
Received: from seambx.corp.real.com ([fe80::2d15:fda7:b3b8:e268]) by
 seacas01.corp.real.com ([192.168.139.56]) with mapi; Wed, 13 Oct 2010
 16:38:37 -0700
From: Michael Moores <mmoores@real.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 13 Oct 2010 16:38:35 -0700
Subject: Re: Specifying the InputFormat class that exists in a JAR on the
 hdfs
Thread-Topic: Specifying the InputFormat class that exists in a JAR on the
 hdfs
Thread-Index: ActrL8IzR9NPEpvzRAaAgcgLUdK2wg==
Message-ID: <70A4FF20-6FB5-41AC-B05D-5DBA51FA90D4@real.com>
References: <B2EA53A1-B10A-4FE8-84E4-313EFEE4CF39@real.com>
 <2EB7C48D-7640-4B76-8135-9120E39031A6@real.com>
 <AANLkTim=PWyk_Pp4rC4uJ3S-bYP1_Pk9w3yoSgajqmWX@mail.gmail.com>
In-Reply-To: <AANLkTim=PWyk_Pp4rC4uJ3S-bYP1_Pk9w3yoSgajqmWX@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Add it to HADOOP_CLASSPATH on all machines running the task?
I can try that, but I'd like users to be able to execute jobs using jars fr=
om their own hdfs directory.


On Oct 13, 2010, at 4:21 PM, Shrijeet Paliwal wrote:

> How about adding it to HADOOP_CLASSPATH if not already.
>=20
> On Wed, Oct 13, 2010 at 4:15 PM, Michael Moores <mmoores@real.com> wrote:
>=20
>> fyi- I also tried thr archive version--
>>=20
>> calling DistributedCache.addArchiveToClassPath(path, configuration);
>>=20
>> On Oct 13, 2010, at 4:12 PM, Michael Moores wrote:
>>=20
>>> I have specified my InputFormat to be the cassandra
>> ColumnFamilyInputFormat, and also
>>> added the cassandra JAR to my classpath via a call to
>> DistributedCache.addFileToClassPath().
>>> The JAR exists on the HDFS.
>>> When I run my jar I get  java.lang.NoClassDefFoundError:
>> org/apache/cassandra/hadoop/ColumnFamilyInputFormat at the line that
>>> makes the job.setInputFormatClass() call.
>>>=20
>>> I execute the job with "hadoop jar <myjar>".
>>>=20
>>> Will I need to put my cassandra JAR on each machine and add it to the J=
VM
>> startup options???
>>>=20
>>> Here is a code snippet:
>>>=20
>>> public class MyStats extends Configured implements Tool {
>>> ...
>>>  public static void main(String[] args) throws Exception {
>>>       // Let ToolRunner handle generic command-line options
>>>       Configuration configuration =3D new Configuration();
>>>       Path path =3D new
>> Path("/user/hadoop/profilestats/cassandra-0.7.0-beta2.jar");
>>>       log.info("main: adding jars...");
>>>       DistributedCache.addFileToClassPath(path, configuration);
>>>=20
>>>=20
>>>=20
>>>=20
>>>=20
>>>       ToolRunner.run(configuration, new MyStats(), args);
>>>       System.exit(0);
>>>   }
>>>=20
>>>  public int run(String[] args) throws Exception {
>>>     Job job =3D new Job(getConf(), "myjob");
>>>=20
>> job.setInputFormatClass(org.apache.cassandra.hadoop.ColumnFamilyInputFor=
mat.class);
>>>     ..
>>>     job.waitForCompletion(true);
>>>  }
>>>=20
>>>=20
>>> FILE LISTING from HDFS:
>>>=20
>>> [hadoop@kv-app02 ~]$ hadoop dfs -lsr
>>> 10/10/13 14:57:47 INFO security.Groups: Group mapping
>> impl=3Dorg.apache.hadoop.security.ShellBasedUnixGroupsMapping;
>> cacheTimeout=3D300000
>>> 10/10/13 14:57:48 WARN conf.Configuration: mapred.task.id is deprecated=
.
>> Instead, use mapreduce.task.attempt.id
>>> drwxr-xr-x   - hadoop supergroup          0 2010-10-13 14:34
>> /user/hadoop/profilestats
>>> -rw-r--r--   3 hadoop supergroup    1841467 2010-10-13 14:34
>> /user/hadoop/profilestats/cassandra-0.7.0-beta2.jar
>>=20
>>=20


From general-return-2165-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 13 23:41:50 2010
Return-Path: <general-return-2165-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34498 invoked from network); 13 Oct 2010 23:41:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Oct 2010 23:41:50 -0000
Received: (qmail 15603 invoked by uid 500); 13 Oct 2010 23:41:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15549 invoked by uid 500); 13 Oct 2010 23:41:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15541 invoked by uid 99); 13 Oct 2010 23:41:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 23:41:49 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of shrijeet@rocketfuelinc.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 23:41:42 +0000
Received: by qwf7 with SMTP id 7so1504194qwf.35
        for <general@hadoop.apache.org>; Wed, 13 Oct 2010 16:41:21 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.236.17 with SMTP id ki17mr8172657qcb.37.1287013280585;
 Wed, 13 Oct 2010 16:41:20 -0700 (PDT)
Received: by 10.229.250.21 with HTTP; Wed, 13 Oct 2010 16:41:20 -0700 (PDT)
In-Reply-To: <70A4FF20-6FB5-41AC-B05D-5DBA51FA90D4@real.com>
References: <B2EA53A1-B10A-4FE8-84E4-313EFEE4CF39@real.com>
	<2EB7C48D-7640-4B76-8135-9120E39031A6@real.com>
	<AANLkTim=PWyk_Pp4rC4uJ3S-bYP1_Pk9w3yoSgajqmWX@mail.gmail.com>
	<70A4FF20-6FB5-41AC-B05D-5DBA51FA90D4@real.com>
Date: Wed, 13 Oct 2010 16:41:20 -0700
Message-ID: <AANLkTimK_Dj=TuyAfbRuFOyuZhP=u+A2GtTDGO9tPKy7@mail.gmail.com>
Subject: Re: Specifying the InputFormat class that exists in a JAR on the hdfs
From: Shrijeet Paliwal <shrijeet@rocketfuel.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163630f87369189c049288211d
X-Virus-Checked: Checked by ClamAV on apache.org

--00163630f87369189c049288211d
Content-Type: text/plain; charset=ISO-8859-1

Do that only on the machine which is launching the job.

On Wed, Oct 13, 2010 at 4:38 PM, Michael Moores <mmoores@real.com> wrote:

> Add it to HADOOP_CLASSPATH on all machines running the task?
> I can try that, but I'd like users to be able to execute jobs using jars
> from their own hdfs directory.
>
>
> On Oct 13, 2010, at 4:21 PM, Shrijeet Paliwal wrote:
>
> > How about adding it to HADOOP_CLASSPATH if not already.
> >
> > On Wed, Oct 13, 2010 at 4:15 PM, Michael Moores <mmoores@real.com>
> wrote:
> >
> >> fyi- I also tried thr archive version--
> >>
> >> calling DistributedCache.addArchiveToClassPath(path, configuration);
> >>
> >> On Oct 13, 2010, at 4:12 PM, Michael Moores wrote:
> >>
> >>> I have specified my InputFormat to be the cassandra
> >> ColumnFamilyInputFormat, and also
> >>> added the cassandra JAR to my classpath via a call to
> >> DistributedCache.addFileToClassPath().
> >>> The JAR exists on the HDFS.
> >>> When I run my jar I get  java.lang.NoClassDefFoundError:
> >> org/apache/cassandra/hadoop/ColumnFamilyInputFormat at the line that
> >>> makes the job.setInputFormatClass() call.
> >>>
> >>> I execute the job with "hadoop jar <myjar>".
> >>>
> >>> Will I need to put my cassandra JAR on each machine and add it to the
> JVM
> >> startup options???
> >>>
> >>> Here is a code snippet:
> >>>
> >>> public class MyStats extends Configured implements Tool {
> >>> ...
> >>>  public static void main(String[] args) throws Exception {
> >>>       // Let ToolRunner handle generic command-line options
> >>>       Configuration configuration = new Configuration();
> >>>       Path path = new
> >> Path("/user/hadoop/profilestats/cassandra-0.7.0-beta2.jar");
> >>>       log.info("main: adding jars...");
> >>>       DistributedCache.addFileToClassPath(path, configuration);
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>       ToolRunner.run(configuration, new MyStats(), args);
> >>>       System.exit(0);
> >>>   }
> >>>
> >>>  public int run(String[] args) throws Exception {
> >>>     Job job = new Job(getConf(), "myjob");
> >>>
> >>
> job.setInputFormatClass(org.apache.cassandra.hadoop.ColumnFamilyInputFormat.class);
> >>>     ..
> >>>     job.waitForCompletion(true);
> >>>  }
> >>>
> >>>
> >>> FILE LISTING from HDFS:
> >>>
> >>> [hadoop@kv-app02 ~]$ hadoop dfs -lsr
> >>> 10/10/13 14:57:47 INFO security.Groups: Group mapping
> >> impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
> >> cacheTimeout=300000
> >>> 10/10/13 14:57:48 WARN conf.Configuration: mapred.task.id is
> deprecated.
> >> Instead, use mapreduce.task.attempt.id
> >>> drwxr-xr-x   - hadoop supergroup          0 2010-10-13 14:34
> >> /user/hadoop/profilestats
> >>> -rw-r--r--   3 hadoop supergroup    1841467 2010-10-13 14:34
> >> /user/hadoop/profilestats/cassandra-0.7.0-beta2.jar
> >>
> >>
>
>

--00163630f87369189c049288211d--

From general-return-2166-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 13 23:48:00 2010
Return-Path: <general-return-2166-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35854 invoked from network); 13 Oct 2010 23:47:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Oct 2010 23:47:59 -0000
Received: (qmail 20211 invoked by uid 500); 13 Oct 2010 23:47:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20162 invoked by uid 500); 13 Oct 2010 23:47:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 20154 invoked by uid 99); 13 Oct 2010 23:47:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 23:47:58 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of shrijeet@rocketfuelinc.com designates 209.85.216.176 as permitted sender)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 23:47:54 +0000
Received: by qyk29 with SMTP id 29so8916736qyk.14
        for <general@hadoop.apache.org>; Wed, 13 Oct 2010 16:47:33 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.224.197.5 with SMTP id ei5mr7332409qab.35.1287013653213; Wed,
 13 Oct 2010 16:47:33 -0700 (PDT)
Received: by 10.229.250.21 with HTTP; Wed, 13 Oct 2010 16:47:33 -0700 (PDT)
In-Reply-To: <AANLkTimK_Dj=TuyAfbRuFOyuZhP=u+A2GtTDGO9tPKy7@mail.gmail.com>
References: <B2EA53A1-B10A-4FE8-84E4-313EFEE4CF39@real.com>
	<2EB7C48D-7640-4B76-8135-9120E39031A6@real.com>
	<AANLkTim=PWyk_Pp4rC4uJ3S-bYP1_Pk9w3yoSgajqmWX@mail.gmail.com>
	<70A4FF20-6FB5-41AC-B05D-5DBA51FA90D4@real.com>
	<AANLkTimK_Dj=TuyAfbRuFOyuZhP=u+A2GtTDGO9tPKy7@mail.gmail.com>
Date: Wed, 13 Oct 2010 16:47:33 -0700
Message-ID: <AANLkTimuJ5q4_54bsHQ7XQFPo2-ZVzKfvUw_8WqB0Cau@mail.gmail.com>
Subject: Re: Specifying the InputFormat class that exists in a JAR on the hdfs
From: Shrijeet Paliwal <shrijeet@rocketfuel.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf300fb12b9ef6910492883799

--20cf300fb12b9ef6910492883799
Content-Type: text/plain; charset=ISO-8859-1

Also you dont necessarily need to use DistributedCache API from your
application. You can supply  libjars flag from command line to supply
additional jars to mappers and reducers.

Take a look :
http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html#Usage  (look
for libjars option)

On Wed, Oct 13, 2010 at 4:41 PM, Shrijeet Paliwal
<shrijeet@rocketfuel.com>wrote:

> Do that only on the machine which is launching the job.
>
>
> On Wed, Oct 13, 2010 at 4:38 PM, Michael Moores <mmoores@real.com> wrote:
>
>> Add it to HADOOP_CLASSPATH on all machines running the task?
>> I can try that, but I'd like users to be able to execute jobs using jars
>> from their own hdfs directory.
>>
>>
>> On Oct 13, 2010, at 4:21 PM, Shrijeet Paliwal wrote:
>>
>> > How about adding it to HADOOP_CLASSPATH if not already.
>> >
>> > On Wed, Oct 13, 2010 at 4:15 PM, Michael Moores <mmoores@real.com>
>> wrote:
>> >
>> >> fyi- I also tried thr archive version--
>> >>
>> >> calling DistributedCache.addArchiveToClassPath(path, configuration);
>> >>
>> >> On Oct 13, 2010, at 4:12 PM, Michael Moores wrote:
>> >>
>> >>> I have specified my InputFormat to be the cassandra
>> >> ColumnFamilyInputFormat, and also
>> >>> added the cassandra JAR to my classpath via a call to
>> >> DistributedCache.addFileToClassPath().
>> >>> The JAR exists on the HDFS.
>> >>> When I run my jar I get  java.lang.NoClassDefFoundError:
>> >> org/apache/cassandra/hadoop/ColumnFamilyInputFormat at the line that
>> >>> makes the job.setInputFormatClass() call.
>> >>>
>> >>> I execute the job with "hadoop jar <myjar>".
>> >>>
>> >>> Will I need to put my cassandra JAR on each machine and add it to the
>> JVM
>> >> startup options???
>> >>>
>> >>> Here is a code snippet:
>> >>>
>> >>> public class MyStats extends Configured implements Tool {
>> >>> ...
>> >>>  public static void main(String[] args) throws Exception {
>> >>>       // Let ToolRunner handle generic command-line options
>> >>>       Configuration configuration = new Configuration();
>> >>>       Path path = new
>> >> Path("/user/hadoop/profilestats/cassandra-0.7.0-beta2.jar");
>> >>>       log.info("main: adding jars...");
>> >>>       DistributedCache.addFileToClassPath(path, configuration);
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>       ToolRunner.run(configuration, new MyStats(), args);
>> >>>       System.exit(0);
>> >>>   }
>> >>>
>> >>>  public int run(String[] args) throws Exception {
>> >>>     Job job = new Job(getConf(), "myjob");
>> >>>
>> >>
>> job.setInputFormatClass(org.apache.cassandra.hadoop.ColumnFamilyInputFormat.class);
>> >>>     ..
>> >>>     job.waitForCompletion(true);
>> >>>  }
>> >>>
>> >>>
>> >>> FILE LISTING from HDFS:
>> >>>
>> >>> [hadoop@kv-app02 ~]$ hadoop dfs -lsr
>> >>> 10/10/13 14:57:47 INFO security.Groups: Group mapping
>> >> impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
>> >> cacheTimeout=300000
>> >>> 10/10/13 14:57:48 WARN conf.Configuration: mapred.task.id is
>> deprecated.
>> >> Instead, use mapreduce.task.attempt.id
>> >>> drwxr-xr-x   - hadoop supergroup          0 2010-10-13 14:34
>> >> /user/hadoop/profilestats
>> >>> -rw-r--r--   3 hadoop supergroup    1841467 2010-10-13 14:34
>> >> /user/hadoop/profilestats/cassandra-0.7.0-beta2.jar
>> >>
>> >>
>>
>>
>

--20cf300fb12b9ef6910492883799--

From general-return-2167-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 14 17:18:15 2010
Return-Path: <general-return-2167-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 3785 invoked from network); 14 Oct 2010 17:18:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Oct 2010 17:18:14 -0000
Received: (qmail 81618 invoked by uid 500); 14 Oct 2010 17:18:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81555 invoked by uid 500); 14 Oct 2010 17:18:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81547 invoked by uid 99); 14 Oct 2010 17:18:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Oct 2010 17:18:12 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mmoores@real.com designates 207.188.23.4 as permitted sender)
Received: from [207.188.23.4] (HELO kal-el.real.com) (207.188.23.4)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Oct 2010 17:18:07 +0000
Received: from seacas01.corp.real.com ([::ffff:192.168.139.56])
  (TLS: TLSv1/SSLv3,128bits,AES128-SHA)
  by kal-el.real.com with esmtp; Thu, 14 Oct 2010 10:17:47 -0700
  id 000806B1.4CB73B3B.00005431
Received: from seambx.corp.real.com ([fe80::2d15:fda7:b3b8:e268]) by
 seacas01.corp.real.com ([192.168.139.56]) with mapi; Thu, 14 Oct 2010
 10:17:46 -0700
From: Michael Moores <mmoores@real.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 14 Oct 2010 10:17:45 -0700
Subject: Re: Specifying the InputFormat class that exists in a JAR on the
 hdfs
Thread-Topic: Specifying the InputFormat class that exists in a JAR on the
 hdfs
Thread-Index: Actrw7gYBsQ1mYCuShOQR67Jjeo6sQ==
Message-ID: <8147FF31-729A-484D-B0B3-81F9DF96E793@real.com>
References: <B2EA53A1-B10A-4FE8-84E4-313EFEE4CF39@real.com>
 <2EB7C48D-7640-4B76-8135-9120E39031A6@real.com>
 <AANLkTim=PWyk_Pp4rC4uJ3S-bYP1_Pk9w3yoSgajqmWX@mail.gmail.com>
 <70A4FF20-6FB5-41AC-B05D-5DBA51FA90D4@real.com>
 <AANLkTimK_Dj=TuyAfbRuFOyuZhP=u+A2GtTDGO9tPKy7@mail.gmail.com>
 <AANLkTimuJ5q4_54bsHQ7XQFPo2-ZVzKfvUw_8WqB0Cau@mail.gmail.com>
In-Reply-To: <AANLkTimuJ5q4_54bsHQ7XQFPo2-ZVzKfvUw_8WqB0Cau@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

I moved back from hadoop 21.0 to 20.2 and things look better.

But I'm a little confused on how things are working:

My InputFormat class attempts to connect to cassandra on localhost.
I have JobTracker/NameNode running on one server, and TaskTracker/DataNode =
running on 8 other machines (slaves).
I also have cassandra running on those hadoop slaves. =20

I execute the hadoop job on the JobTracker machine, and I get a connection =
refused exception attempting to connect to cassandra.
I expected the InputFormat to run on the 8 TaskTracker machines..  but it l=
ooks like it's just running locally.






On Oct 13, 2010, at 4:47 PM, Shrijeet Paliwal wrote:

> Also you dont necessarily need to use DistributedCache API from your
> application. You can supply  libjars flag from command line to supply
> additional jars to mappers and reducers.
>=20
> Take a look :
> http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html#Usage  =
(look
> for libjars option)
>=20
> On Wed, Oct 13, 2010 at 4:41 PM, Shrijeet Paliwal
> <shrijeet@rocketfuel.com>wrote:
>=20
>> Do that only on the machine which is launching the job.
>>=20
>>=20
>> On Wed, Oct 13, 2010 at 4:38 PM, Michael Moores <mmoores@real.com> wrote=
:
>>=20
>>> Add it to HADOOP_CLASSPATH on all machines running the task?
>>> I can try that, but I'd like users to be able to execute jobs using jar=
s
>>> from their own hdfs directory.
>>>=20
>>>=20
>>> On Oct 13, 2010, at 4:21 PM, Shrijeet Paliwal wrote:
>>>=20
>>>> How about adding it to HADOOP_CLASSPATH if not already.
>>>>=20
>>>> On Wed, Oct 13, 2010 at 4:15 PM, Michael Moores <mmoores@real.com>
>>> wrote:
>>>>=20
>>>>> fyi- I also tried thr archive version--
>>>>>=20
>>>>> calling DistributedCache.addArchiveToClassPath(path, configuration);
>>>>>=20
>>>>> On Oct 13, 2010, at 4:12 PM, Michael Moores wrote:
>>>>>=20
>>>>>> I have specified my InputFormat to be the cassandra
>>>>> ColumnFamilyInputFormat, and also
>>>>>> added the cassandra JAR to my classpath via a call to
>>>>> DistributedCache.addFileToClassPath().
>>>>>> The JAR exists on the HDFS.
>>>>>> When I run my jar I get  java.lang.NoClassDefFoundError:
>>>>> org/apache/cassandra/hadoop/ColumnFamilyInputFormat at the line that
>>>>>> makes the job.setInputFormatClass() call.
>>>>>>=20
>>>>>> I execute the job with "hadoop jar <myjar>".
>>>>>>=20
>>>>>> Will I need to put my cassandra JAR on each machine and add it to th=
e
>>> JVM
>>>>> startup options???
>>>>>>=20
>>>>>> Here is a code snippet:
>>>>>>=20
>>>>>> public class MyStats extends Configured implements Tool {
>>>>>> ...
>>>>>> public static void main(String[] args) throws Exception {
>>>>>>      // Let ToolRunner handle generic command-line options
>>>>>>      Configuration configuration =3D new Configuration();
>>>>>>      Path path =3D new
>>>>> Path("/user/hadoop/profilestats/cassandra-0.7.0-beta2.jar");
>>>>>>      log.info("main: adding jars...");
>>>>>>      DistributedCache.addFileToClassPath(path, configuration);
>>>>>>=20
>>>>>>=20
>>>>>>=20
>>>>>>=20
>>>>>>=20
>>>>>>      ToolRunner.run(configuration, new MyStats(), args);
>>>>>>      System.exit(0);
>>>>>>  }
>>>>>>=20
>>>>>> public int run(String[] args) throws Exception {
>>>>>>    Job job =3D new Job(getConf(), "myjob");
>>>>>>=20
>>>>>=20
>>> job.setInputFormatClass(org.apache.cassandra.hadoop.ColumnFamilyInputFo=
rmat.class);
>>>>>>    ..
>>>>>>    job.waitForCompletion(true);
>>>>>> }
>>>>>>=20
>>>>>>=20
>>>>>> FILE LISTING from HDFS:
>>>>>>=20
>>>>>> [hadoop@kv-app02 ~]$ hadoop dfs -lsr
>>>>>> 10/10/13 14:57:47 INFO security.Groups: Group mapping
>>>>> impl=3Dorg.apache.hadoop.security.ShellBasedUnixGroupsMapping;
>>>>> cacheTimeout=3D300000
>>>>>> 10/10/13 14:57:48 WARN conf.Configuration: mapred.task.id is
>>> deprecated.
>>>>> Instead, use mapreduce.task.attempt.id
>>>>>> drwxr-xr-x   - hadoop supergroup          0 2010-10-13 14:34
>>>>> /user/hadoop/profilestats
>>>>>> -rw-r--r--   3 hadoop supergroup    1841467 2010-10-13 14:34
>>>>> /user/hadoop/profilestats/cassandra-0.7.0-beta2.jar
>>>>>=20
>>>>>=20
>>>=20
>>>=20
>>=20


From general-return-2168-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 15 20:36:30 2010
Return-Path: <general-return-2168-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78368 invoked from network); 15 Oct 2010 20:36:29 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Oct 2010 20:36:29 -0000
Received: (qmail 12325 invoked by uid 500); 15 Oct 2010 20:36:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12274 invoked by uid 500); 15 Oct 2010 20:36:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12266 invoked by uid 99); 15 Oct 2010 20:36:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 20:36:27 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of goutham.patnaik@gmail.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 20:36:22 +0000
Received: by gyh3 with SMTP id 3so715455gyh.35
        for <general@hadoop.apache.org>; Fri, 15 Oct 2010 13:36:01 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:from:date
         :message-id:subject:to:content-type;
        bh=ov4WXJxI1893UMYsRPOgyv1Akr3314mi6bUZgAg/r5A=;
        b=Lazv41sP69M/jH/Zq4sJj3nLDwEvqlw6O6QZLCcDopgV1Xa9UHPzYwOAIbnWncJFiV
         yFIN7sA+GD0557Wb2v+CLmMr4wZXWMaSfAyw+yOWMYhIeqLJMlsqQ32nQhourl3SMBCJ
         7/JEqw3ceW3M5rE17nBxNXDvlGBEhqx2sVr8s=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type;
        b=jW+9KFOaPFaxf2oVjtb+UUQeOHqe5yqksK6lGkWTRwWSE52ICDQR7+reeu6VFXDMib
         X+XeDvvgN8401FlQlh/3OtqeED4nQNZQy422n3A1NoGyC2BB5SUy6Mj7Hnuvheto0twI
         GwcMBjHtmD2njtaa1AobgJ2aglTvKCQxtovh0=
Received: by 10.42.28.9 with SMTP id l9mr1137190icc.104.1287174959776; Fri, 15
 Oct 2010 13:35:59 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.220.164.16 with HTTP; Fri, 15 Oct 2010 13:35:38 -0700 (PDT)
From: goutham patnaik <goutham.patnaik@gmail.com>
Date: Fri, 15 Oct 2010 13:35:38 -0700
Message-ID: <AANLkTinyb6B5m0ZYe5Uwoms138XosSioBAWM6qB5Eh5J@mail.gmail.com>
Subject: building mapreduce project on eclipse
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf301d41c03dc39f0492adc647

--20cf301d41c03dc39f0492adc647
Content-Type: text/plain; charset=ISO-8859-1

i checked out (from http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk) the
mapreduce project into eclipse. Following the instructions from the hadoop
screencast (http://vimeo.com/4193623) and the hadoop eclipse setup
instructions (http://wiki.apache.org/hadoop/EclipseEnvironment), i did an
ant build with the following targets selected -

clean
compile-mapred-test
eclipse-files

The ant build and the corresponding Build Project were successful, but
eclipse still doesnt recognize classes / libraries from the hadoop-common
project - which i also checked out and built (successfully). is there a way
to include the hadoop-common project inside the mapreduce project ?

Also, im looking to contribute to the hadoop project in the near future - do
most committers out there checkout from and use eclipse for this ? or is
there a more efficient way you've found to build your hadoop projects ?


Thanks in advance


Goutham

--20cf301d41c03dc39f0492adc647--

From general-return-2169-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 15 20:52:47 2010
Return-Path: <general-return-2169-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83285 invoked from network); 15 Oct 2010 20:52:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Oct 2010 20:52:47 -0000
Received: (qmail 37442 invoked by uid 500); 15 Oct 2010 20:52:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37400 invoked by uid 500); 15 Oct 2010 20:52:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37390 invoked by uid 99); 15 Oct 2010 20:52:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 20:52:46 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 20:52:39 +0000
Received: by qwf7 with SMTP id 7so690319qwf.35
        for <general@hadoop.apache.org>; Fri, 15 Oct 2010 13:52:18 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.183.21 with SMTP id ce21mr1136739qcb.197.1287175928766;
 Fri, 15 Oct 2010 13:52:08 -0700 (PDT)
Received: by 10.229.237.131 with HTTP; Fri, 15 Oct 2010 13:52:08 -0700 (PDT)
In-Reply-To: <AANLkTinyb6B5m0ZYe5Uwoms138XosSioBAWM6qB5Eh5J@mail.gmail.com>
References: <AANLkTinyb6B5m0ZYe5Uwoms138XosSioBAWM6qB5Eh5J@mail.gmail.com>
Date: Fri, 15 Oct 2010 13:52:08 -0700
Message-ID: <AANLkTi=bdKm0UZX2Oqfc9kfmOQ3ULDGE-H1RNtG2MKxV@mail.gmail.com>
Subject: Re: building mapreduce project on eclipse
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hey Goutham,

Add separate common, hdfs and mr projects to your workspace. Then for
the hdfs and mapreduce projects, select Build Path -> Configure Build
Path, in the Projects tab of the Java Build Path add dependencies on
the other projects. hdfs should depend on common, and mapreduce should
depend on hdfs and common.

Thanks,
Eli

On Fri, Oct 15, 2010 at 1:35 PM, goutham patnaik
<goutham.patnaik@gmail.com> wrote:
> i checked out (from http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk) the
> mapreduce project into eclipse. Following the instructions from the hadoop
> screencast (http://vimeo.com/4193623) and the hadoop eclipse setup
> instructions (http://wiki.apache.org/hadoop/EclipseEnvironment), i did an
> ant build with the following targets selected -
>
> clean
> compile-mapred-test
> eclipse-files
>
> The ant build and the corresponding Build Project were successful, but
> eclipse still doesnt recognize classes / libraries from the hadoop-common
> project - which i also checked out and built (successfully). is there a way
> to include the hadoop-common project inside the mapreduce project ?
>
> Also, im looking to contribute to the hadoop project in the near future - do
> most committers out there checkout from and use eclipse for this ? or is
> there a more efficient way you've found to build your hadoop projects ?
>
>
> Thanks in advance
>
>
> Goutham
>

From general-return-2170-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 15 21:28:27 2010
Return-Path: <general-return-2170-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1779 invoked from network); 15 Oct 2010 21:28:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Oct 2010 21:28:17 -0000
Received: (qmail 937 invoked by uid 500); 15 Oct 2010 21:28:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 871 invoked by uid 500); 15 Oct 2010 21:28:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 301 invoked by uid 99); 15 Oct 2010 21:28:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 21:28:15 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 15 Oct 2010 21:28:13 +0000
Received: (qmail 1373 invoked by uid 99); 15 Oct 2010 21:27:51 -0000
Received: from localhost.apache.org (HELO [10.0.0.149]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 21:27:51 +0000
Message-ID: <4CB8C756.5050806@apache.org>
Date: Fri, 15 Oct 2010 14:27:50 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.12) Gecko/20100915 Thunderbird/3.0.8
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
In-Reply-To: <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 08/30/2010 02:14 PM, Arun C Murthy wrote:
> Since most people seemed to think of it as a reasonable idea, I'm going
> to create the hadoop-0.20-security branch and start the necessary work.

I don't yet see this branch.  Are you still intending to do this?

Doug

From general-return-2171-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 16 02:06:22 2010
Return-Path: <general-return-2171-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91430 invoked from network); 16 Oct 2010 02:06:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Oct 2010 02:06:21 -0000
Received: (qmail 88239 invoked by uid 500); 16 Oct 2010 02:06:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88187 invoked by uid 500); 16 Oct 2010 02:06:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88179 invoked by uid 99); 16 Oct 2010 02:06:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 16 Oct 2010 02:06:19 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 16 Oct 2010 02:06:15 +0000
Received: by fxm13 with SMTP id 13so1338501fxm.35
        for <general@hadoop.apache.org>; Fri, 15 Oct 2010 19:05:54 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:received:date
         :message-id:subject:from:to:content-type;
        bh=uptBvhm0iUW1wrKhQmafuJnOUQDWsBwF9dJiphacLLo=;
        b=MvdyKmDDcQfB0UzomCZfIptK+HhHZK8Y9fOeNtiRf7UMoNHG9Z71rBc576RrfJ0UKj
         beEm5IBjQ6CpKaSv9PvroemYDKXLIcMZlLSkRo+HePw/JZTE5r+96CNXkQu7QuVXgXav
         V1IsbQ4F54dieH79/CTubTdVCYcYwq9zULdRw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=sST591Uk+vHU+vVaOmpBz1O+V+3mFAjvVAyq1YKkdw6qhltbXnp+HC+ozU+QhwpL2D
         0ktmW73ABHLX3vaWobLX7FdGGiTL2bKOOnS6owfpHpICyPsSo6iiqyJiiEE0/kapa7ae
         DdAOaPxmsUBSoA64lR6cIt9mfKfbHaecZAt1A=
MIME-Version: 1.0
Received: by 10.102.83.1 with SMTP id g1mr850324mub.47.1287194753790; Fri, 15
 Oct 2010 19:05:53 -0700 (PDT)
Received: by 10.223.110.147 with HTTP; Fri, 15 Oct 2010 19:05:53 -0700 (PDT)
Received: by 10.223.110.147 with HTTP; Fri, 15 Oct 2010 19:05:53 -0700 (PDT)
Date: Sat, 16 Oct 2010 07:35:53 +0530
Message-ID: <AANLkTi=VTj=ae0PrybQpnSWoKv79hnFq4rgchTaf=NJL@mail.gmail.com>
Subject: Re: building mapreduce project on eclipse
From: Harsh J <qwertymaniac@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636416ed70e72cd0492b2622e

--001636416ed70e72cd0492b2622e
Content-Type: text/plain; charset=ISO-8859-1

Sometimes there's faults in the .classpath file among the jars that are on
the build path. Correcting those manually should fix this issue as am sure
the ant build pulled down proper hadoop common and hdfs jars. They might
just have wrong version names in .classpath sometimes (like say hadoop-core
instead of common). Check it out and fix it.

--
Harsh J
http://www.harshj.com

On Oct 16, 2010 2:06 AM, "goutham patnaik" <goutham.patnaik@gmail.com>
wrote:

i checked out (from http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk)
the
mapreduce project into eclipse. Following the instructions from the hadoop
screencast (http://vimeo.com/4193623) and the hadoop eclipse setup
instructions (http://wiki.apache.org/hadoop/EclipseEnvironment), i did an
ant build with the following targets selected -

clean
compile-mapred-test
eclipse-files

The ant build and the corresponding Build Project were successful, but
eclipse still doesnt recognize classes / libraries from the hadoop-common
project - which i also checked out and built (successfully). is there a way
to include the hadoop-common project inside the mapreduce project ?

Also, im looking to contribute to the hadoop project in the near future - do
most committers out there checkout from and use eclipse for this ? or is
there a more efficient way you've found to build your hadoop projects ?


Thanks in advance


Goutham

--001636416ed70e72cd0492b2622e--

From general-return-2172-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 16 08:47:13 2010
Return-Path: <general-return-2172-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98692 invoked from network); 16 Oct 2010 08:47:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Oct 2010 08:47:13 -0000
Received: (qmail 35233 invoked by uid 500); 16 Oct 2010 08:47:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34896 invoked by uid 500); 16 Oct 2010 08:47:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34875 invoked by uid 99); 16 Oct 2010 08:47:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 16 Oct 2010 08:47:08 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of goutham.patnaik@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 16 Oct 2010 08:47:00 +0000
Received: by vws15 with SMTP id 15so870403vws.35
        for <general@hadoop.apache.org>; Sat, 16 Oct 2010 01:46:39 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:from:date
         :message-id:subject:to:content-type;
        bh=GgS0oNj+gO59vzsbj+z8Xy+XKEfXsEeXXBqPwXFQOGM=;
        b=F6//h/Oa48JR9iBYFFX7vtDczRDzDGeeL0NaHAODnu1xddPJrarYI/XjCoMqAhCsuJ
         PrzdUehqXL15zueohcyAjrswXj5e/VtpLpg38QvFBfUTqgSKX+ZZaq62Y93e84K/o0D4
         TBp6C6gni8TlYfCvEujSAakYKIJJlXwnRtxjM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type;
        b=HxkMXbd+7FK4d88OEyc3dxOIVcN/CWQ9lmkj/LgsrJC+bX0ml7pKR83P4KHeb0jPHp
         tjVrB2GsJ4lE3oLf7RjKOXenvSgRvzKb62r5HmQzQL8X/UvGkmtbjqRr3bijLdpnbLy4
         KcBJ7M1ODYYN5HrLhYTOUuROSLieH/glMCw0o=
Received: by 10.220.202.74 with SMTP id fd10mr266245vcb.226.1287218797477;
 Sat, 16 Oct 2010 01:46:37 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.220.164.16 with HTTP; Sat, 16 Oct 2010 01:46:17 -0700 (PDT)
From: goutham patnaik <goutham.patnaik@gmail.com>
Date: Sat, 16 Oct 2010 01:46:17 -0700
Message-ID: <AANLkTimiDVHRAScjxZjPU0tddTLZh4hDwRJD+Vu=C9yB@mail.gmail.com>
Subject: building hadoop projects (common, hdfs, mapreduce) on eclipse
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba53a97c2bff4c0492b7fba6
X-Virus-Checked: Checked by ClamAV on apache.org

--90e6ba53a97c2bff4c0492b7fba6
Content-Type: text/plain; charset=ISO-8859-1

When trying to build these projects on eclipse following instructions from
http://wiki.apache.org/hadoop/EclipseEnvironment, when trying to do an ant
build, i get a http 502 error when trying to run the ivy-resolve target :

[ivy:resolve] :::: ERRORS

[ivy:resolve] SERVER ERROR: Bad Gateway url=
https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.22.0-SNAPSHOT/maven-metadata.xml

[ivy:resolve] SERVER ERROR: Bad Gateway url=
https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.22.0-SNAPSHOT/hadoop-common-0.22.0-SNAPSHOT.pom

[ivy:resolve] SERVER ERROR: Bad Gateway url=
https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.22.0-SNAPSHOT/hadoop-common-0.22.0-SNAPSHOT.jar

[ivy:resolve]

[ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS


BUILD FAILED

/Users/goutham/hadoop_src/hdfs/build.xml:1552: impossible to resolve
dependencies:

resolve failed - see output for details


ive only been seeing this error for the last 3 hours or so - the ant build
worked perfectly fine earlier in the day. Other https sites work fine for me



Goutham

--90e6ba53a97c2bff4c0492b7fba6--

From general-return-2173-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 16 12:54:33 2010
Return-Path: <general-return-2173-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66012 invoked from network); 16 Oct 2010 12:54:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Oct 2010 12:54:33 -0000
Received: (qmail 69579 invoked by uid 500); 16 Oct 2010 12:54:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69240 invoked by uid 500); 16 Oct 2010 12:54:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69232 invoked by uid 99); 16 Oct 2010 12:54:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 16 Oct 2010 12:54:29 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of magawake@gmail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 16 Oct 2010 12:54:21 +0000
Received: by iwn3 with SMTP id 3so2517360iwn.35
        for <general@hadoop.apache.org>; Sat, 16 Oct 2010 05:54:00 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=cm7F3FhQlLQrhpWgTfMbacbtzjeRNOLjdRdkPFKnkaw=;
        b=qZePgBSeKtJ5MzRQWOEDt35950J5XC34ZgywqWXKjOFKXtSY5Y37Q5Okp8M4gM57Uf
         H4K4zUwwhnKmVdXUJnkkCHn1FgQHaHe8FoNY6dwqgNPg4ka65Do8ksBPMDraRhpgTbnS
         ghfTyLe7SnBC9ZvHg5/aZLNvfRRcz3pR+f1TI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=ONA1A7bSElrxgPELp/xcVY28dVn9mhJ0v7ErHBVbdT5EegalpS7XpvFpoeGROck7Nc
         QL2g6l+EDudffXaAYiq8u9IYuaRqGkwsCCv3oGAvlQR0ujP2Hoen+Xiu0VQ074xhHEMU
         cUCnuVIBNcZ5jRcJwmDjiCNWzAkzW0l1/qeBk=
MIME-Version: 1.0
Received: by 10.231.30.72 with SMTP id t8mr1607718ibc.46.1287233640498; Sat,
 16 Oct 2010 05:54:00 -0700 (PDT)
Received: by 10.231.189.154 with HTTP; Sat, 16 Oct 2010 05:54:00 -0700 (PDT)
Date: Sat, 16 Oct 2010 08:54:00 -0400
Message-ID: <AANLkTim7hcNQc3LbOkUjwzQXM35BUAnFqTzXW_3N62C3@mail.gmail.com>
Subject: name node and secondary name node configuration query
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
X-Virus-Checked: Checked by ClamAV on apache.org

Hello,
I am trying to trim down my configuration files to avoid confusion.

If I were to setup something like this: 1 name node (192.168.0.1) , 1
secondary name node (192.168.0.2), and 10 clients. The name node and
secondary namenode will not serve as a data node.

The clients will access thru a shared NFS configuration, core-site.xml
The datanode will only have, dfs.data.dir in hdfs-site.xml
The name node will have, dfs.name.dir in hdfs-site.xml
I am really not sure what I must do for my secondary name node. What
should I place there?

The idea is I want to reduce my configuration files. Any advise?

From general-return-2174-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 16 17:56:40 2010
Return-Path: <general-return-2174-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50768 invoked from network); 16 Oct 2010 17:56:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Oct 2010 17:56:40 -0000
Received: (qmail 11550 invoked by uid 500); 16 Oct 2010 17:56:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11492 invoked by uid 500); 16 Oct 2010 17:56:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11484 invoked by uid 99); 16 Oct 2010 17:56:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 16 Oct 2010 17:56:37 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 16 Oct 2010 17:56:31 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=Q6GKqKOsy3odblwyH1wPwe0TBe3tHP0NDb8HGbOKTgTeZjT/fEoWsm7A
   NzMQZcXE8NV/jXed9P10GYQJkKrRISY61UzfSRpqWDbTLApMAsROvLcaU
   QJ/TQsZUFJv3eXw;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1287251791; x=1318787791;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20name=20node=20and=20secondary=20name=20
   node=20configuration=20query|Date:=20Sat,=2016=20Oct=2020
   10=2017:56:09=20+0000|Message-ID:=20<6DA1C439-03E1-4EFF-A
   788-52F90CDD47C0@linkedin.com>|To:=20"<general@hadoop.apa
   che.org>"=20<general@hadoop.apache.org>|MIME-Version:=201
   .0|Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<D15B2FDAC2F6AB49BAA6E8EF771BDC41@linkedin
   .com>|In-Reply-To:=20<AANLkTim7hcNQc3LbOkUjwzQXM35BUAnFqT
   zXW_3N62C3@mail.gmail.com>|References:=20<AANLkTim7hcNQc3
   LbOkUjwzQXM35BUAnFqTzXW_3N62C3@mail.gmail.com>;
  bh=yftDvUA8wpTvwo0lW7WJ4ss6yF8AOGcfEh9AAjkfG/Q=;
  b=HnozB2wV4LuhEhrf+qVOyk6dkmdScYslIFC54zGWa8DWOYe3Z4UqvOKH
   4ZkrnEUDZfDDjAdKXBYiIFo7c3/4QFUPs+GYP5rO3ywzI/xvskrwdktql
   uCfL3nHpIqvaRZv;
X-IronPort-AV: E=Sophos;i="4.57,340,1283756400"; 
   d="scan'208";a="16135346"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi id 14.01.0218.012; Sat,
 16 Oct 2010 10:56:09 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: name node and secondary name node configuration query
Thread-Topic: name node and secondary name node configuration query
Thread-Index: AQHLbTFIsX+V10TdKkuM3YkfPyNLVZNEUcmA
Date: Sat, 16 Oct 2010 17:56:09 +0000
Message-ID: <6DA1C439-03E1-4EFF-A788-52F90CDD47C0@linkedin.com>
References: <AANLkTim7hcNQc3LbOkUjwzQXM35BUAnFqTzXW_3N62C3@mail.gmail.com>
In-Reply-To: <AANLkTim7hcNQc3LbOkUjwzQXM35BUAnFqTzXW_3N62C3@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <D15B2FDAC2F6AB49BAA6E8EF771BDC41@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


On Oct 16, 2010, at 5:54 AM, Mag Gam wrote:

> Hello,
> I am trying to trim down my configuration files to avoid confusion.
>=20
> If I were to setup something like this: 1 name node (192.168.0.1) , 1
> secondary name node (192.168.0.2), and 10 clients. The name node and
> secondary namenode will not serve as a data node.
>=20
> The clients will access thru a shared NFS configuration, core-site.xml
> The datanode will only have, dfs.data.dir in hdfs-site.xml
> The name node will have, dfs.name.dir in hdfs-site.xml
> I am really not sure what I must do for my secondary name node. What
> should I place there?
>=20
> The idea is I want to reduce my configuration files. Any advise?



Short Story: =20

	You are fighting a losing battle.  Throw in the towel now and plan on push=
ing huge balls of configs.  Work on a strategy for clients that are out of =
sync.

Longer Story:

	a) Any production Hadoop deployment should expect to have somewhere around=
 3-4 different sets of configuration files if you configure even some of th=
e intermediate features. [Hello separate audit logs!]

	b) You should expect to have all the config files for all the different se=
rvices.  The devs are just now starting to separate what needs what so hope=
fully by 0.22 we won't have to have worry whether we have all of our bases =
covered.  Remember: if it isn't defined, it will use a default and therefor=
e still work...just not the way you intend.

	c) Don't use NFS for pushing configuration files unless it is an HA-NFS co=
nfiguration,  full/redundant connectivity to every switch, etc.  The first =
time the NFS server falls out from underneath the grid, you're going to be =
in a world of hurt.

=09

	We really need https://issues.apache.org/jira/browse/HADOOP-5670 to make t=
his whole mess go away.  Until it is nailed down which components need whic=
h config vars, it is unlikely to either a) happen or b) work in a way that =
is actually operable for any reasonable production deployment.


From general-return-2175-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Oct 17 02:19:22 2010
Return-Path: <general-return-2175-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73386 invoked from network); 17 Oct 2010 02:19:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Oct 2010 02:19:21 -0000
Received: (qmail 63544 invoked by uid 500); 17 Oct 2010 02:19:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63289 invoked by uid 500); 17 Oct 2010 02:19:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63281 invoked by uid 99); 17 Oct 2010 02:19:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 17 Oct 2010 02:19:20 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of goutham.patnaik@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 17 Oct 2010 02:19:15 +0000
Received: by vws15 with SMTP id 15so1205099vws.35
        for <general@hadoop.apache.org>; Sat, 16 Oct 2010 19:18:54 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type;
        bh=nl2UrxZB16BB2J3pMd+26DxxPrggbmOu9AlNxiNnjQk=;
        b=pex/S8PzFAE4eUMMbWToxczmT8qfGzgSVG5uu6aTQ40wBYOSTNomYVY6rtqgNuXtEY
         vGkm0ifNe9YnZhR3xvmrJ0iXpH4J6byFs1b/lb0Vb+7T5rWMWi+P0dd9q+58No+iPRvl
         NzY95IcGmn3n8OiWUe6NNIRkPv/nLLPeg48xA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=cIU03lfScRP+FrbaNc0Y3W1EXOU8UzOZGT2PvluDQHa8m6NQyYfW6u3ZPz4H9jjKR1
         ItPUPWLsCGNqHnLnK+r7o+2xTLqbBX16ZMdvQ8tjUxI653K3hPR2xqa8YaWS48hMU+KT
         2uDzJA7qHmkgOHd1wz/IhuPD/00JK5FYzrKOs=
Received: by 10.220.203.12 with SMTP id fg12mr573338vcb.133.1287281934376;
 Sat, 16 Oct 2010 19:18:54 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.220.164.16 with HTTP; Sat, 16 Oct 2010 19:18:34 -0700 (PDT)
In-Reply-To: <AANLkTi=bdKm0UZX2Oqfc9kfmOQ3ULDGE-H1RNtG2MKxV@mail.gmail.com>
References: <AANLkTinyb6B5m0ZYe5Uwoms138XosSioBAWM6qB5Eh5J@mail.gmail.com> <AANLkTi=bdKm0UZX2Oqfc9kfmOQ3ULDGE-H1RNtG2MKxV@mail.gmail.com>
From: goutham patnaik <goutham.patnaik@gmail.com>
Date: Sat, 16 Oct 2010 19:18:34 -0700
Message-ID: <AANLkTik_Qy60Yj3-Dwak-xtFMaw4XzzE9nOx09GRUJ-p@mail.gmail.com>
Subject: Re: building mapreduce project on eclipse
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba53a1b86c9e510492c6aee2

--90e6ba53a1b86c9e510492c6aee2
Content-Type: text/plain; charset=ISO-8859-1

Thanks Eli - that seemed to have worked


Goutham

On Fri, Oct 15, 2010 at 1:52 PM, Eli Collins <eli@cloudera.com> wrote:

> Hey Goutham,
>
> Add separate common, hdfs and mr projects to your workspace. Then for
> the hdfs and mapreduce projects, select Build Path -> Configure Build
> Path, in the Projects tab of the Java Build Path add dependencies on
> the other projects. hdfs should depend on common, and mapreduce should
> depend on hdfs and common.
>
> Thanks,
> Eli
>
> On Fri, Oct 15, 2010 at 1:35 PM, goutham patnaik
> <goutham.patnaik@gmail.com> wrote:
> > i checked out (from
> http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk) the
> > mapreduce project into eclipse. Following the instructions from the
> hadoop
> > screencast (http://vimeo.com/4193623) and the hadoop eclipse setup
> > instructions (http://wiki.apache.org/hadoop/EclipseEnvironment), i did
> an
> > ant build with the following targets selected -
> >
> > clean
> > compile-mapred-test
> > eclipse-files
> >
> > The ant build and the corresponding Build Project were successful, but
> > eclipse still doesnt recognize classes / libraries from the hadoop-common
> > project - which i also checked out and built (successfully). is there a
> way
> > to include the hadoop-common project inside the mapreduce project ?
> >
> > Also, im looking to contribute to the hadoop project in the near future -
> do
> > most committers out there checkout from and use eclipse for this ? or is
> > there a more efficient way you've found to build your hadoop projects ?
> >
> >
> > Thanks in advance
> >
> >
> > Goutham
> >
>

--90e6ba53a1b86c9e510492c6aee2--

From general-return-2176-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 18 22:18:23 2010
Return-Path: <general-return-2176-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 51947 invoked from network); 18 Oct 2010 22:18:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Oct 2010 22:18:23 -0000
Received: (qmail 35733 invoked by uid 500); 18 Oct 2010 22:18:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 35544 invoked by uid 500); 18 Oct 2010 22:18:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 35536 invoked by uid 99); 18 Oct 2010 22:18:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Oct 2010 22:18:21 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of amp@opendns.com designates 67.215.68.163 as permitted sender)
Received: from [67.215.68.163] (HELO mail.opendns.com) (67.215.68.163)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Oct 2010 22:18:12 +0000
Received: from Adams-Desktop.local ([67.215.69.84])
	(authenticated bits=0)
	by mail.opendns.com (8.14.3/8.14.3/Debian-5) with ESMTP id o9IMHoIo002256
	(version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Mon, 18 Oct 2010 22:17:50 GMT
Message-ID: <4CBCC78E.4090006@opendns.com>
Date: Mon, 18 Oct 2010 15:17:50 -0700
From: Adam Phelps <amp@opendns.com>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.9) Gecko/20100915 Thunderbird/3.1.4
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Checking current configuration values?
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Is there a means of looking up the current value of a configuration 
variable, in particular one that is not explicitly set in one of the 
.conf files?

For instance if I wanted to find the value of "io.file.buffer.size" 
(just a random example) how would I go about it?

Thanks
- Adam Phelps

From general-return-2177-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 18 22:22:42 2010
Return-Path: <general-return-2177-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53832 invoked from network); 18 Oct 2010 22:22:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Oct 2010 22:22:42 -0000
Received: (qmail 41915 invoked by uid 500); 18 Oct 2010 22:22:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 41860 invoked by uid 500); 18 Oct 2010 22:22:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 41852 invoked by uid 99); 18 Oct 2010 22:22:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Oct 2010 22:22:40 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Oct 2010 22:22:34 +0000
Received: by qyk34 with SMTP id 34so143039qyk.14
        for <general@hadoop.apache.org>; Mon, 18 Oct 2010 15:22:13 -0700 (PDT)
Received: by 10.224.131.153 with SMTP id x25mr1552820qas.185.1287440532424;
 Mon, 18 Oct 2010 15:22:12 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.214.68 with HTTP; Mon, 18 Oct 2010 15:21:51 -0700 (PDT)
In-Reply-To: <4CBCC78E.4090006@opendns.com>
References: <4CBCC78E.4090006@opendns.com>
From: Philip Zeyliger <philip@cloudera.com>
Date: Mon, 18 Oct 2010 15:21:51 -0700
Message-ID: <AANLkTi=_b9rU8R2t4QSk7FN2_NLzETbCw3m2NhPNcExZ@mail.gmail.com>
Subject: Re: Checking current configuration values?
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0022150473bf9ac7d60492eb9b47
X-Virus-Checked: Checked by ClamAV on apache.org

--0022150473bf9ac7d60492eb9b47
Content-Type: text/plain; charset=ISO-8859-1

Trunk has a configuration servlet (/conf) that exposes configuration.  I
think it's in CDH3b3 (Cloudera's Distribution) as well.

https://issues.apache.org/jira/browse/HADOOP-6408

Cheers,

-- Philip

On Mon, Oct 18, 2010 at 3:17 PM, Adam Phelps <amp@opendns.com> wrote:

> Is there a means of looking up the current value of a configuration
> variable, in particular one that is not explicitly set in one of the .conf
> files?
>
> For instance if I wanted to find the value of "io.file.buffer.size" (just a
> random example) how would I go about it?
>
> Thanks
> - Adam Phelps
>

--0022150473bf9ac7d60492eb9b47--

From general-return-2178-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 18 22:33:11 2010
Return-Path: <general-return-2178-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56587 invoked from network); 18 Oct 2010 22:33:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Oct 2010 22:33:11 -0000
Received: (qmail 55771 invoked by uid 500); 18 Oct 2010 22:33:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55673 invoked by uid 500); 18 Oct 2010 22:33:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55665 invoked by uid 99); 18 Oct 2010 22:33:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Oct 2010 22:33:09 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Oct 2010 22:33:05 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=hvKSfapyDNYftD8HNjAMHaKfT5V0aA37XvPh3gJsHd5+1VCgCS9W6FYc
   hCo7T1HBPziytDZEf/rDX0oRrSSGyL2K55qe9vkaTzUZSs5/rB0+PGeaN
   Ajy2NlRqRW6ofvZ;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1287441185; x=1318977185;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20Checking=20current=20configuration=20va
   lues?|Date:=20Mon,=2018=20Oct=202010=2022:32:44=20+0000
   |Message-ID:=20<E062B443-B097-40EF-812D-06AD18BEDE46@link
   edin.com>|To:=20"<general@hadoop.apache.org>"=20<general@
   hadoop.apache.org>|MIME-Version:=201.0
   |Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<44473CABDF1733499B0CB90001BE9021@linkedin
   .com>|In-Reply-To:=20<AANLkTi=3D_b9rU8R2t4QSk7FN2_NLzETbC
   w3m2NhPNcExZ@mail.gmail.com>|References:=20<4CBCC78E.4090
   006@opendns.com>=0D=0A=20<AANLkTi=3D_b9rU8R2t4QSk7FN2_NLz
   ETbCw3m2NhPNcExZ@mail.gmail.com>;
  bh=JuXT3dnajX2CIAdwDH60ylyLfeIOSVgaQAAihEfS5QA=;
  b=jBELZHM/NrB/AXpLJnoQySu2U3uGJvssxm0wtKIjO9YKzwt5EgaWyQ+L
   Ix9swUqlkDJMQsE/eY3vY80F6E57zm/LAtyRuHzAiCwZWYEPL+sHAuVGL
   fGqvNeW8AIW03YG;
X-IronPort-AV: E=Sophos;i="4.57,346,1283756400"; 
   d="scan'208";a="16181648"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi id 14.01.0218.012; Mon,
 18 Oct 2010 15:32:45 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: Checking current configuration values?
Thread-Topic: Checking current configuration values?
Thread-Index: AQHLbxJgEcmlRVxMaU6nZeXGAKCkAZNHvO2AgAADCgA=
Date: Mon, 18 Oct 2010 22:32:44 +0000
Message-ID: <E062B443-B097-40EF-812D-06AD18BEDE46@linkedin.com>
References: <4CBCC78E.4090006@opendns.com>
 <AANLkTi=_b9rU8R2t4QSk7FN2_NLzETbCw3m2NhPNcExZ@mail.gmail.com>
In-Reply-To: <AANLkTi=_b9rU8R2t4QSk7FN2_NLzETbCw3m2NhPNcExZ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <44473CABDF1733499B0CB90001BE9021@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


If you just want a quick visual inspection, you can use the jobconf.jsp by =
clicking on the xml at the top of a job page in the jobtracker.


On Oct 18, 2010, at 3:21 PM, Philip Zeyliger wrote:

> Trunk has a configuration servlet (/conf) that exposes configuration.  I
> think it's in CDH3b3 (Cloudera's Distribution) as well.
>=20
> https://issues.apache.org/jira/browse/HADOOP-6408
>=20
> Cheers,
>=20
> -- Philip
>=20
> On Mon, Oct 18, 2010 at 3:17 PM, Adam Phelps <amp@opendns.com> wrote:
>=20
>> Is there a means of looking up the current value of a configuration
>> variable, in particular one that is not explicitly set in one of the .co=
nf
>> files?
>>=20
>> For instance if I wanted to find the value of "io.file.buffer.size" (jus=
t a
>> random example) how would I go about it?
>>=20
>> Thanks
>> - Adam Phelps
>>=20


From general-return-2179-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 19 07:33:49 2010
Return-Path: <general-return-2179-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27625 invoked from network); 19 Oct 2010 07:33:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Oct 2010 07:33:49 -0000
Received: (qmail 13212 invoked by uid 500); 19 Oct 2010 07:33:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13027 invoked by uid 500); 19 Oct 2010 07:33:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13017 invoked by uid 99); 19 Oct 2010 07:33:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Oct 2010 07:33:44 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [188.40.108.78] (HELO mail.affinitas.de) (188.40.108.78)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Oct 2010 07:33:35 +0000
Received: from localhost (localhost [127.0.0.1])
	by mail.affinitas.de (Postfix) with ESMTP id 7BA73C0077B
	for <general@hadoop.apache.org>; Tue, 19 Oct 2010 09:37:08 +0200 (CEST)
X-Virus-Scanned: Debian amavisd-new at mail.affinitas.de
Received: from mail.affinitas.de ([127.0.0.1])
	by localhost (mail.affinitas.de [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id bxlyqB2IoEFL for <general@hadoop.apache.org>;
	Tue, 19 Oct 2010 09:37:07 +0200 (CEST)
Received: from [192.168.1.20] (unknown [91.102.12.2])
	by mail.affinitas.de (Postfix) with ESMTPA
	for <general@hadoop.apache.org>; Tue, 19 Oct 2010 09:37:07 +0200 (CEST)
Message-ID: <4CBD49B9.4090304@affinitas.de>
Date: Tue, 19 Oct 2010 09:33:13 +0200
From: Martin Kuhn <martin.kuhn@affinitas.de>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.12) Gecko/20100915 Lightning/1.0b1 Thunderbird/3.0.8
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Checking current configuration values?
References: <4CBCC78E.4090006@opendns.com>
In-Reply-To: <4CBCC78E.4090006@opendns.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

> Is there a means of looking up the current value of a configuration
> variable, in particular one that is not explicitly set in one of the
> .conf files?

http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/conf/Configuration.html#get%28java.lang.String%29


> For instance if I wanted to find the value of "io.file.buffer.size"
> (just a random example) how would I go about it?

public class MyApp extends Configured implements Tool {

     public int run(final String[] args) throws Exception {
         final Configuration conf = getConf();
         ...
         final String property = conf.get("io.file.buffer.size");
         ...
     }

     public static void main(final String[] args) throws Exception {
         final int res = ToolRunner.run(new Configuration(), new 
MyApp(), args);
         System.exit(res);
     }

}

From general-return-2180-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 19 18:31:43 2010
Return-Path: <general-return-2180-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72129 invoked from network); 19 Oct 2010 18:31:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Oct 2010 18:31:42 -0000
Received: (qmail 44511 invoked by uid 500); 19 Oct 2010 18:31:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44307 invoked by uid 500); 19 Oct 2010 18:31:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44297 invoked by uid 99); 19 Oct 2010 18:31:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Oct 2010 18:31:40 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of anil@kitenga.com designates 205.209.188.5 as permitted sender)
Received: from [205.209.188.5] (HELO kitenga.com) (205.209.188.5)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Oct 2010 18:31:32 +0000
Received: from smlucid3lt (55.tensv.org [71.6.14.55] (may be forged))
	(authenticated bits=0)
	by kitenga.com (8.13.1/8.13.1) with ESMTP id o9JIV60I001211
	for <general@hadoop.apache.org>; Tue, 19 Oct 2010 11:31:07 -0700
From: "Anil Uberoi" <anil@kitenga.com>
To: <general@hadoop.apache.org>
References: <C8DB7955.17EB%a.reiter@web.de>
In-Reply-To: <C8DB7955.17EB%a.reiter@web.de>
Subject: RE: Architecture
Date: Tue, 19 Oct 2010 11:31:06 -0700
Message-ID: <001401cb6fbb$cc09e880$641db980$@com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Mailer: Microsoft Office Outlook 12.0
Thread-Index: Actq1i6+hc039LXzJkiOs5oPzinFPAE5W9Ig
Content-Language: en-us

Andre,

Hope all is well.
I 'd like to invite you to check out www.kitenga.com
We'd be more than happy to help.
Best regards,
Anil

------------------------------------------------------------------
Anil Uberoi | CEO | Kitenga | Reinventing Information
www.kitenga.com | 510.507.3399
 


-----Original Message-----
From: a.reiter@web.de [mailto:a.reiter@web.de] 
Sent: Wednesday, October 13, 2010 5:57 AM
To: general@hadoop.apache.org
Subject: Architecture

Hello everybody,

i'm evaluating hadoop for a new platform.
the application is a web based application, and the challange is to handle
requests up to 1.000.000 times per second. this can actually be solved by
load ballanced webservers.
The problem now is to persist user data at real time. Let's say every
request appends some data to the user record, i.e. 100 bytes are appended.
The record would have an identifier placed in the cookie.

The problem now is to look up the user record for every request at runtime.
Is it possible to get the user record with the information about all former
requests in a short time i.e. 50 ms ???

i hove no experience with hadoop yet

every help would be very appreciated
thanks in advance

best regards
andre


From general-return-2181-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 19 21:13:35 2010
Return-Path: <general-return-2181-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46252 invoked from network); 19 Oct 2010 21:13:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Oct 2010 21:13:35 -0000
Received: (qmail 69670 invoked by uid 500); 19 Oct 2010 21:13:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69598 invoked by uid 500); 19 Oct 2010 21:13:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69590 invoked by uid 99); 19 Oct 2010 21:13:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Oct 2010 21:13:33 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.27.243] (HELO qmta13.emeryville.ca.mail.comcast.net) (76.96.27.243)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Oct 2010 21:13:23 +0000
Received: from omta09.emeryville.ca.mail.comcast.net ([76.96.30.20])
	by qmta13.emeryville.ca.mail.comcast.net with comcast
	id Ljtd1f0050S2fkCADlD1JB; Tue, 19 Oct 2010 21:13:01 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta09.emeryville.ca.mail.comcast.net with comcast
	id LlCs1f00B2SbwD58VlCuDb; Tue, 19 Oct 2010 21:12:59 +0000
Message-Id: <63A1C7A9-727B-4492-9998-01F6B6958EDD@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: [DISCUSS] Proposed bylaws for Hadoop
Date: Tue, 19 Oct 2010 14:12:52 -0700
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

Apache Hadoop Project Bylaws

This document defines the bylaws under which the Apache Hadoop project
operates. It defines the roles and responsibilities of the project,
who may vote, how voting works, how conflicts are resolved, etc.

Hadoop is a project of the <a
href="http://www.apache.org/foundation/">Apache Software
Foundation</a>.  The foundation holds the trademark on the name
"Hadoop" and copyright on Apache code
including the code in the Hadoop codebase. The <a
href="http://www.apache.org/foundation/faq.html">foundation FAQ</a>
explains the operation and background of the foundation.

Hadoop is typical of Apache projects in that it operates under a set
of principles, known collectively as the &quot;Apache Way&quot;. If
you are new to Apache development, please refer to the <a
href="http://incubator.apache.org">Incubator project</a> for more
information on how Apache projects operate.

Roles and Responsibilities

Apache projects define a set of roles with associated rights and
responsibilities. These roles govern what tasks an individual may
perform within the project. The roles are defined in the following
sections

    * Users

         The most important participants in the project are people who
         use our software. The majority of our developers start out as
         users and guide their development efforts from the user's
         perspective.

        Users contribute to the Apache projects by providing feedback
         to developers in the form of bug reports and feature
         suggestions. As well, users participate in the Apache
         community by helping other users on mailing lists and user
         support forums.

    * Contributors

         All of the volunteers who are contributing time, code,
         documentation, or resources to the Hadoop Project. A  
contributor
         that makes sustained, welcome contributions to the project may
         be invited to become a Committer, though the exact timing of
         such invitations depends on many factors.

     * Committers

        The project's Committers are responsible for the project's
         technical management. Committers have access to a specified
         set of subprojects' subversion repositories. Committers on
         subprojects may cast binding votes on any technical discussion
         regarding that subproject.

        Committer access is by invitation only and must be approved by
         lazy consensus of the active PMC members. A Committer is
         considered emeritus by their own declaration or by not
         contributing in any form to the project for over six
         months. An emeritus committer may request reinstatement of
         commit access from the PMC. Such reinstatement is subject to
         lazy consensus of active PMC members.

       All Apache committers are required to have a signed Contributor
         License Agreement (CLA) on file with the Apache Software
         Foundation. There is a <a
         href="http://www.apache.org/dev/committers.html">Committer
         FAQ</a> which provides more details on the requirements for
         Committers

        A committer who makes a sustained contribution to the project
         may be invited to become a member of the PMC. The form of
         contribution is not limited to code. It can also include code
         review, helping out users on the mailing lists, documentation,
         testing, etc.

    * Project Management Committee

        The Project Management Committee (PMC) for Apache Hadoop was
         created by the Apache Board in January 2008 when Hadoop moved
         out of Lucene and became a top level project at Apache. The
         PMC is responsible to the board and the ASF for the management
         and oversight of the Apache Hadoop codebase. The
         responsibilities of the PMC include

        * Deciding what is distributed as products of the Apache Hadoop
         project.  In particular all releases must be approved by the
         PMC

         * Maintaining the project's shared resources, including the  
codebase
             repository, mailing lists, websites.

         * Speaking on behalf of the project.

        * Resolving license disputes regarding products of the project

         * Nominating new PMC members and committers

         * Maintaining these bylaws and other guidelines of the project

        Membership of the PMC is by invitation only and must be
         approved by a lazy consensus of active PMC members. A PMC
         member is considered &quot;emeritus&quot; by their own
         declaration or by not contributing in any form to the project
         for over six months. An emeritus member may request
         reinstatement to the PMC. Such reinstatement is subject to
         lazy consensus of the active PMC members.

        The chair of the PMC is appointed by the ASF board. The chair
         is an office holder of the Apache Software Foundation (Vice
         President, Apache Hadoop) and has primary responsibility to
         the board for the management of the projects within the scope
         of the Hadoop PMC. The chair reports to the board quarterly on
         developments within the Hadoop project.

	When the current chair of the PMC resigns, the PMC votes to
	recommend a new chair using lazy consensus, but the decision
	must be ratified by the Apache board.

Decision Making

       Within the Hadoop project, different types of decisions require
       different forms of approval. For example, the <a href="#Roles
       and Responsibilities">previous section</a> describes several
       decisions which require &quot;lazy consensus&quot;
       approval. This section defines how voting is performed, the
       types of approvals, and which types of decision require which
       type of approval.

     * Voting

         Decisions regarding the project are made by votes on the
         primary project development mailing list
         (general@hadoop.apache.org).  Where necessary, PMC voting may
         take place on the private Hadoop PMC mailing list.  Votes are
         clearly indicated by subject line starting with [VOTE].  Votes
         may contain multiple items for approval and these should be
         clearly separated. Voting is carried out by replying to the
         vote mail. Voting may take four flavours

         * +1 &quot;Yes,&quot; &quot;Agree,&quot; or &quot;the action  
should be
             performed.&quot; In general, this vote also indicates a  
willingness
             on the behalf of the voter in &quot;making it happen&quot;

          * +0 This vote indicates a willingness for the action under
             consideration to go ahead. The voter, however will not be  
able
             to help.

          * -0 This vote indicates that the voter does not, in  
general, agree with
             the proposed action but is not concerned enough to  
prevent the
             action going ahead.

           * -1 This is a negative vote. On issues where consensus is  
required,
             this vote counts as a <strong>veto</strong>. All vetoes  
must
             contain an explanation of why the veto is appropriate.  
Vetoes with
             no explanation are void. It may also be appropriate for a  
-1 vote
             to include an alternative course of action.

         All participants in the Hadoop project are encouraged to show  
their
         agreement with or against a particular action by voting. For  
technical
         decisions, only the votes of active committers are binding.  
Non binding
         votes are still useful for those with binding votes to  
understand the
         perception of an action in the wider Hadoop community. For  
PMC decisions,
         only the votes of PMC members are binding.

        Voting can also be applied to changes made to the Hadoop  
codebase. These
         typically take the form of a veto (-1) in reply to the commit  
message
         sent when the commit is made.

     * Approvals

         These are the types of approvals that can be sought.  
Different actions
         require different types of approvals

         * Lazy Consensus - Lazy consensus requires 3 binding +1 votes
            and no binding vetoes.

          * Lazy Majority - A lazy majority vote requires 3 binding +1
             votes and more binding +1 votes than -1 votes.

         * Lazy 2/3 Majority - Lazy 2/3 majority votes requires at
            least 3 votes and twice as many +1 votes as -1 votes.

     * Vetoes

         A valid, binding veto cannot be overruled. If a veto is cast,
         it must be accompanied by a valid reason explaining the
         reasons for the veto. The validity of a veto, if challenged,
         can be confirmed by anyone who has a binding vote. This does
         not necessarily signify agreement with the veto - merely that
         the veto is valid.

        If you disagree with a valid veto, you must lobby the person  
casting
         the veto to withdraw their veto. If a veto is not withdrawn,  
any action
         that has been vetoed must be reversed in a timely manner.

     * Actions

         This section describes the various actions which are  
undertaken within
         the project, the corresponding approval required for that  
action and
         those who have binding votes over the action.

          * Code Change

             A change made to a codebase of the project and committed
             by a committer. This includes source code, documentation,  
website
             content, etc.

             Lazy consensus of active committers.

          * Release Plan

             Defines the timetable and actions for a release. The plan  
also
             nominates a Release Manager.

             Lazy majority of active committers

          * Product Release

             When a release of one of the project's products is ready,  
a vote is
             required to accept the release as an official release of  
the
             project.

            Lazy Majority of active PMC members

           * Adoption of New Codebase

             When the codebase for an existing, released product is to  
be
             replaced with an alternative codebase. If such a vote  
fails to
             gain approval, the existing code base will continue.

            This also covers the creation of new sub-projects
             within the project

            Lazy 2/3 majority of PMC members

           * New Committer

             When a new committer is proposed for the project

             Lazy consensus of active PMC members

           * New PMC Member

             When a committer is proposed for the PMC

            Lazy consensus of active PMC members

           * Committer Removal

            When removal of commit privileges is sought.  Note: Such
            actions will also be referred to the ASF board by the PMC
            chair

             Majority of active PMC members (excluding the committer  
in question if a
             member of the PMC).

           * PMC Member Removal

             When removal of a PMC member is sought.  Note: Such
             actions will also be referred to the ASF board by the PMC
             chair.

             Majority of active PMC members (excluding the member in  
question)

           * Modifying Bylaws

             Modifying this document.

             Majority of active PMC members

    * Voting Timeframes

         Votes are open for a period of 7 days to allow all active
         voters time to consider the vote. Votes relating to code
         changes are not subject to a strict timetable but should be
         made as timely as possible.


From general-return-2182-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 19 21:36:12 2010
Return-Path: <general-return-2182-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55968 invoked from network); 19 Oct 2010 21:36:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Oct 2010 21:36:12 -0000
Received: (qmail 1041 invoked by uid 500); 19 Oct 2010 21:36:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 973 invoked by uid 500); 19 Oct 2010 21:36:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 964 invoked by uid 99); 19 Oct 2010 21:36:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Oct 2010 21:36:10 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Oct 2010 21:36:03 +0000
Received: by wyg36 with SMTP id 36so1810414wyg.35
        for <general@hadoop.apache.org>; Tue, 19 Oct 2010 14:35:41 -0700 (PDT)
Received: by 10.227.145.148 with SMTP id d20mr6808319wbv.117.1287524141424;
        Tue, 19 Oct 2010 14:35:41 -0700 (PDT)
Received: from tp ([38.104.141.102])
        by mx.google.com with ESMTPS id a17sm13168711wbe.0.2010.10.19.14.35.39
        (version=SSLv3 cipher=RC4-MD5);
        Tue, 19 Oct 2010 14:35:40 -0700 (PDT)
Sender: Konstantin Boudnik <cos@boudnik.org>
Date: Tue, 19 Oct 2010 14:35:35 -0700
From: Konstantin Boudnik <cos@apache.org>
To: general@hadoop.apache.org
Subject: Re: [DISCUSS] Proposed bylaws for Hadoop
Message-ID: <20101019213535.GC2075@tp>
References: <63A1C7A9-727B-4492-9998-01F6B6958EDD@apache.org>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="jRHKVT23PllUwdXP"
Content-Disposition: inline
In-Reply-To: <63A1C7A9-727B-4492-9998-01F6B6958EDD@apache.org>
X-Organization: It's something of 'Cos
X-PGP-Key: http://www.boudnik.org/~cos/pubkey.asc
User-Agent: Mutt/1.5.20 (2009-06-14)

--jRHKVT23PllUwdXP
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Being a relatively new member of Hadoop community (just slightly over year
and a half) and never been exposed to its politics I can't help but to wond=
er
if such document didn't exist before?

If it did, then what's the difference between the current version and the
previous one? What's causing that change to be introduced?

If it didn't, then how Hadoop community has been governed beforehand?
By the general ASF rules or differently? If the rules were the same for ASF
and Hadoop why we need to deviate from them now and why?

Clearly, having written bylaws is handy in case of a dispute. However it'd =
be
great to know the history (or just a story) of this particular discussion.

Appreciate any information on the topic. Thanks in advance.
  Cos

On Tue, Oct 19, 2010 at 02:12PM, Owen O'Malley wrote:
> Apache Hadoop Project Bylaws
>=20
> This document defines the bylaws under which the Apache Hadoop project
> operates. It defines the roles and responsibilities of the project,
> who may vote, how voting works, how conflicts are resolved, etc.
>=20
> Hadoop is a project of the <a
> href=3D"http://www.apache.org/foundation/">Apache Software
> Foundation</a>.  The foundation holds the trademark on the name
> "Hadoop" and copyright on Apache code
> including the code in the Hadoop codebase. The <a
> href=3D"http://www.apache.org/foundation/faq.html">foundation FAQ</a>
> explains the operation and background of the foundation.
>=20
> Hadoop is typical of Apache projects in that it operates under a set
> of principles, known collectively as the &quot;Apache Way&quot;. If
> you are new to Apache development, please refer to the <a
> href=3D"http://incubator.apache.org">Incubator project</a> for more
> information on how Apache projects operate.
>=20
> Roles and Responsibilities
>=20
> Apache projects define a set of roles with associated rights and
> responsibilities. These roles govern what tasks an individual may
> perform within the project. The roles are defined in the following
> sections
>=20
>    * Users
>=20
>         The most important participants in the project are people who
>         use our software. The majority of our developers start out as
>         users and guide their development efforts from the user's
>         perspective.
>=20
>        Users contribute to the Apache projects by providing feedback
>         to developers in the form of bug reports and feature
>         suggestions. As well, users participate in the Apache
>         community by helping other users on mailing lists and user
>         support forums.
>=20
>    * Contributors
>=20
>         All of the volunteers who are contributing time, code,
>         documentation, or resources to the Hadoop Project. A
> contributor
>         that makes sustained, welcome contributions to the project may
>         be invited to become a Committer, though the exact timing of
>         such invitations depends on many factors.
>=20
>     * Committers
>=20
>        The project's Committers are responsible for the project's
>         technical management. Committers have access to a specified
>         set of subprojects' subversion repositories. Committers on
>         subprojects may cast binding votes on any technical discussion
>         regarding that subproject.
>=20
>        Committer access is by invitation only and must be approved by
>         lazy consensus of the active PMC members. A Committer is
>         considered emeritus by their own declaration or by not
>         contributing in any form to the project for over six
>         months. An emeritus committer may request reinstatement of
>         commit access from the PMC. Such reinstatement is subject to
>         lazy consensus of active PMC members.
>=20
>       All Apache committers are required to have a signed Contributor
>         License Agreement (CLA) on file with the Apache Software
>         Foundation. There is a <a
>         href=3D"http://www.apache.org/dev/committers.html">Committer
>         FAQ</a> which provides more details on the requirements for
>         Committers
>=20
>        A committer who makes a sustained contribution to the project
>         may be invited to become a member of the PMC. The form of
>         contribution is not limited to code. It can also include code
>         review, helping out users on the mailing lists, documentation,
>         testing, etc.
>=20
>    * Project Management Committee
>=20
>        The Project Management Committee (PMC) for Apache Hadoop was
>         created by the Apache Board in January 2008 when Hadoop moved
>         out of Lucene and became a top level project at Apache. The
>         PMC is responsible to the board and the ASF for the management
>         and oversight of the Apache Hadoop codebase. The
>         responsibilities of the PMC include
>=20
>        * Deciding what is distributed as products of the Apache Hadoop
>         project.  In particular all releases must be approved by the
>         PMC
>=20
>         * Maintaining the project's shared resources, including the
> codebase
>             repository, mailing lists, websites.
>=20
>         * Speaking on behalf of the project.
>=20
>        * Resolving license disputes regarding products of the project
>=20
>         * Nominating new PMC members and committers
>=20
>         * Maintaining these bylaws and other guidelines of the project
>=20
>        Membership of the PMC is by invitation only and must be
>         approved by a lazy consensus of active PMC members. A PMC
>         member is considered &quot;emeritus&quot; by their own
>         declaration or by not contributing in any form to the project
>         for over six months. An emeritus member may request
>         reinstatement to the PMC. Such reinstatement is subject to
>         lazy consensus of the active PMC members.
>=20
>        The chair of the PMC is appointed by the ASF board. The chair
>         is an office holder of the Apache Software Foundation (Vice
>         President, Apache Hadoop) and has primary responsibility to
>         the board for the management of the projects within the scope
>         of the Hadoop PMC. The chair reports to the board quarterly on
>         developments within the Hadoop project.
>=20
> 	When the current chair of the PMC resigns, the PMC votes to
> 	recommend a new chair using lazy consensus, but the decision
> 	must be ratified by the Apache board.
>=20
> Decision Making
>=20
>       Within the Hadoop project, different types of decisions require
>       different forms of approval. For example, the <a href=3D"#Roles
>       and Responsibilities">previous section</a> describes several
>       decisions which require &quot;lazy consensus&quot;
>       approval. This section defines how voting is performed, the
>       types of approvals, and which types of decision require which
>       type of approval.
>=20
>     * Voting
>=20
>         Decisions regarding the project are made by votes on the
>         primary project development mailing list
>         (general@hadoop.apache.org).  Where necessary, PMC voting may
>         take place on the private Hadoop PMC mailing list.  Votes are
>         clearly indicated by subject line starting with [VOTE].  Votes
>         may contain multiple items for approval and these should be
>         clearly separated. Voting is carried out by replying to the
>         vote mail. Voting may take four flavours
>=20
>         * +1 &quot;Yes,&quot; &quot;Agree,&quot; or &quot;the action
> should be
>             performed.&quot; In general, this vote also indicates a
> willingness
>             on the behalf of the voter in &quot;making it happen&quot;
>=20
>          * +0 This vote indicates a willingness for the action under
>             consideration to go ahead. The voter, however will not
> be able
>             to help.
>=20
>          * -0 This vote indicates that the voter does not, in
> general, agree with
>             the proposed action but is not concerned enough to
> prevent the
>             action going ahead.
>=20
>           * -1 This is a negative vote. On issues where consensus is
> required,
>             this vote counts as a <strong>veto</strong>. All vetoes
> must
>             contain an explanation of why the veto is appropriate.
> Vetoes with
>             no explanation are void. It may also be appropriate for
> a -1 vote
>             to include an alternative course of action.
>=20
>         All participants in the Hadoop project are encouraged to
> show their
>         agreement with or against a particular action by voting. For
> technical
>         decisions, only the votes of active committers are binding.
> Non binding
>         votes are still useful for those with binding votes to
> understand the
>         perception of an action in the wider Hadoop community. For
> PMC decisions,
>         only the votes of PMC members are binding.
>=20
>        Voting can also be applied to changes made to the Hadoop
> codebase. These
>         typically take the form of a veto (-1) in reply to the
> commit message
>         sent when the commit is made.
>=20
>     * Approvals
>=20
>         These are the types of approvals that can be sought.
> Different actions
>         require different types of approvals
>=20
>         * Lazy Consensus - Lazy consensus requires 3 binding +1 votes
>            and no binding vetoes.
>=20
>          * Lazy Majority - A lazy majority vote requires 3 binding +1
>             votes and more binding +1 votes than -1 votes.
>=20
>         * Lazy 2/3 Majority - Lazy 2/3 majority votes requires at
>            least 3 votes and twice as many +1 votes as -1 votes.
>=20
>     * Vetoes
>=20
>         A valid, binding veto cannot be overruled. If a veto is cast,
>         it must be accompanied by a valid reason explaining the
>         reasons for the veto. The validity of a veto, if challenged,
>         can be confirmed by anyone who has a binding vote. This does
>         not necessarily signify agreement with the veto - merely that
>         the veto is valid.
>=20
>        If you disagree with a valid veto, you must lobby the person
> casting
>         the veto to withdraw their veto. If a veto is not withdrawn,
> any action
>         that has been vetoed must be reversed in a timely manner.
>=20
>     * Actions
>=20
>         This section describes the various actions which are
> undertaken within
>         the project, the corresponding approval required for that
> action and
>         those who have binding votes over the action.
>=20
>          * Code Change
>=20
>             A change made to a codebase of the project and committed
>             by a committer. This includes source code,
> documentation, website
>             content, etc.
>=20
>             Lazy consensus of active committers.
>=20
>          * Release Plan
>=20
>             Defines the timetable and actions for a release. The
> plan also
>             nominates a Release Manager.
>=20
>             Lazy majority of active committers
>=20
>          * Product Release
>=20
>             When a release of one of the project's products is
> ready, a vote is
>             required to accept the release as an official release of
> the
>             project.
>=20
>            Lazy Majority of active PMC members
>=20
>           * Adoption of New Codebase
>=20
>             When the codebase for an existing, released product is
> to be
>             replaced with an alternative codebase. If such a vote
> fails to
>             gain approval, the existing code base will continue.
>=20
>            This also covers the creation of new sub-projects
>             within the project
>=20
>            Lazy 2/3 majority of PMC members
>=20
>           * New Committer
>=20
>             When a new committer is proposed for the project
>=20
>             Lazy consensus of active PMC members
>=20
>           * New PMC Member
>=20
>             When a committer is proposed for the PMC
>=20
>            Lazy consensus of active PMC members
>=20
>           * Committer Removal
>=20
>            When removal of commit privileges is sought.  Note: Such
>            actions will also be referred to the ASF board by the PMC
>            chair
>=20
>             Majority of active PMC members (excluding the committer
> in question if a
>             member of the PMC).
>=20
>           * PMC Member Removal
>=20
>             When removal of a PMC member is sought.  Note: Such
>             actions will also be referred to the ASF board by the PMC
>             chair.
>=20
>             Majority of active PMC members (excluding the member in
> question)
>=20
>           * Modifying Bylaws
>=20
>             Modifying this document.
>=20
>             Majority of active PMC members
>=20
>    * Voting Timeframes
>=20
>         Votes are open for a period of 7 days to allow all active
>         voters time to consider the vote. Votes relating to code
>         changes are not subject to a strict timetable but should be
>         made as timely as possible.
>=20

--jRHKVT23PllUwdXP
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEARECAAYFAky+DycACgkQIg9pgB8n5iKiqgCfSj59WPJe/GZ9PIEcCbMBOf15
N1kAn3+cXTvwAIpVIznExdfo2YlJg+uU
=Vu5R
-----END PGP SIGNATURE-----

--jRHKVT23PllUwdXP--

From general-return-2183-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 19 21:40:34 2010
Return-Path: <general-return-2183-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56756 invoked from network); 19 Oct 2010 21:40:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Oct 2010 21:40:34 -0000
Received: (qmail 5584 invoked by uid 500); 19 Oct 2010 21:40:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5529 invoked by uid 500); 19 Oct 2010 21:40:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5521 invoked by uid 99); 19 Oct 2010 21:40:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Oct 2010 21:40:32 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.30.96] (HELO qmta09.emeryville.ca.mail.comcast.net) (76.96.30.96)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Oct 2010 21:40:24 +0000
Received: from omta22.emeryville.ca.mail.comcast.net ([76.96.30.89])
	by qmta09.emeryville.ca.mail.comcast.net with comcast
	id Ldcw1f0031vN32cA9lg4mA; Tue, 19 Oct 2010 21:40:04 +0000
Received: from wlan-snve-154-225.corp.yahoo.com ([209.131.62.115])
	by omta22.emeryville.ca.mail.comcast.net with comcast
	id Llfv1f0012VBGtd8ilfxTi; Tue, 19 Oct 2010 21:40:01 +0000
Message-Id: <1CDC6360-5991-4946-A3D9-0632AD214474@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <20101019213535.GC2075@tp>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSS] Proposed bylaws for Hadoop
Date: Tue, 19 Oct 2010 14:39:54 -0700
References: <63A1C7A9-727B-4492-9998-01F6B6958EDD@apache.org> <20101019213535.GC2075@tp>
X-Mailer: Apple Mail (2.936)


On Oct 19, 2010, at 2:35 PM, Konstantin Boudnik wrote:

> Being a relatively new member of Hadoop community (just slightly  
> over year
> and a half) and never been exposed to its politics I can't help but  
> to wonder
> if such document didn't exist before?

No, it should have been done when we form as a TLP, but it wasn't done.

-- Owen


From general-return-2184-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 19 21:46:57 2010
Return-Path: <general-return-2184-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57814 invoked from network); 19 Oct 2010 21:46:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Oct 2010 21:46:57 -0000
Received: (qmail 13323 invoked by uid 500); 19 Oct 2010 21:46:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13277 invoked by uid 500); 19 Oct 2010 21:46:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13269 invoked by uid 99); 19 Oct 2010 21:46:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Oct 2010 21:46:55 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Oct 2010 21:46:49 +0000
Received: by qyk34 with SMTP id 34so807211qyk.14
        for <general@hadoop.apache.org>; Tue, 19 Oct 2010 14:46:28 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.224.174.3 with SMTP id r3mr3917151qaz.260.1287524788162; Tue,
 19 Oct 2010 14:46:28 -0700 (PDT)
Received: by 10.229.237.131 with HTTP; Tue, 19 Oct 2010 14:46:27 -0700 (PDT)
In-Reply-To: <20101019213535.GC2075@tp>
References: <63A1C7A9-727B-4492-9998-01F6B6958EDD@apache.org>
	<20101019213535.GC2075@tp>
Date: Tue, 19 Oct 2010 14:46:27 -0700
Message-ID: <AANLkTinB-uoHpxLU-=CVJZU5zsbkYV+ujNEbmS2DnHvn@mail.gmail.com>
Subject: Re: [DISCUSS] Proposed bylaws for Hadoop
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hey Cos,

There's some background in the proposal thread from last year.

http://mail-archives.apache.org/mod_mbox/hadoop-general/200911.mbox/%3C284E=
0999-F935-47A2-B051-7241BF8A4B39@apache.org%3E

Thanks,
Eli

On Tue, Oct 19, 2010 at 2:35 PM, Konstantin Boudnik <cos@apache.org> wrote:
> Being a relatively new member of Hadoop community (just slightly over yea=
r
> and a half) and never been exposed to its politics I can't help but to wo=
nder
> if such document didn't exist before?
>
> If it did, then what's the difference between the current version and the
> previous one? What's causing that change to be introduced?
>
> If it didn't, then how Hadoop community has been governed beforehand?
> By the general ASF rules or differently? If the rules were the same for A=
SF
> and Hadoop why we need to deviate from them now and why?
>
> Clearly, having written bylaws is handy in case of a dispute. However it'=
d be
> great to know the history (or just a story) of this particular discussion=
.
>
> Appreciate any information on the topic. Thanks in advance.
> =A0Cos
>
> On Tue, Oct 19, 2010 at 02:12PM, Owen O'Malley wrote:
>> Apache Hadoop Project Bylaws
>>
>> This document defines the bylaws under which the Apache Hadoop project
>> operates. It defines the roles and responsibilities of the project,
>> who may vote, how voting works, how conflicts are resolved, etc.
>>
>> Hadoop is a project of the <a
>> href=3D"http://www.apache.org/foundation/">Apache Software
>> Foundation</a>. =A0The foundation holds the trademark on the name
>> "Hadoop" and copyright on Apache code
>> including the code in the Hadoop codebase. The <a
>> href=3D"http://www.apache.org/foundation/faq.html">foundation FAQ</a>
>> explains the operation and background of the foundation.
>>
>> Hadoop is typical of Apache projects in that it operates under a set
>> of principles, known collectively as the &quot;Apache Way&quot;. If
>> you are new to Apache development, please refer to the <a
>> href=3D"http://incubator.apache.org">Incubator project</a> for more
>> information on how Apache projects operate.
>>
>> Roles and Responsibilities
>>
>> Apache projects define a set of roles with associated rights and
>> responsibilities. These roles govern what tasks an individual may
>> perform within the project. The roles are defined in the following
>> sections
>>
>> =A0 =A0* Users
>>
>> =A0 =A0 =A0 =A0 The most important participants in the project are peopl=
e who
>> =A0 =A0 =A0 =A0 use our software. The majority of our developers start o=
ut as
>> =A0 =A0 =A0 =A0 users and guide their development efforts from the user'=
s
>> =A0 =A0 =A0 =A0 perspective.
>>
>> =A0 =A0 =A0 =A0Users contribute to the Apache projects by providing feed=
back
>> =A0 =A0 =A0 =A0 to developers in the form of bug reports and feature
>> =A0 =A0 =A0 =A0 suggestions. As well, users participate in the Apache
>> =A0 =A0 =A0 =A0 community by helping other users on mailing lists and us=
er
>> =A0 =A0 =A0 =A0 support forums.
>>
>> =A0 =A0* Contributors
>>
>> =A0 =A0 =A0 =A0 All of the volunteers who are contributing time, code,
>> =A0 =A0 =A0 =A0 documentation, or resources to the Hadoop Project. A
>> contributor
>> =A0 =A0 =A0 =A0 that makes sustained, welcome contributions to the proje=
ct may
>> =A0 =A0 =A0 =A0 be invited to become a Committer, though the exact timin=
g of
>> =A0 =A0 =A0 =A0 such invitations depends on many factors.
>>
>> =A0 =A0 * Committers
>>
>> =A0 =A0 =A0 =A0The project's Committers are responsible for the project'=
s
>> =A0 =A0 =A0 =A0 technical management. Committers have access to a specif=
ied
>> =A0 =A0 =A0 =A0 set of subprojects' subversion repositories. Committers =
on
>> =A0 =A0 =A0 =A0 subprojects may cast binding votes on any technical disc=
ussion
>> =A0 =A0 =A0 =A0 regarding that subproject.
>>
>> =A0 =A0 =A0 =A0Committer access is by invitation only and must be approv=
ed by
>> =A0 =A0 =A0 =A0 lazy consensus of the active PMC members. A Committer is
>> =A0 =A0 =A0 =A0 considered emeritus by their own declaration or by not
>> =A0 =A0 =A0 =A0 contributing in any form to the project for over six
>> =A0 =A0 =A0 =A0 months. An emeritus committer may request reinstatement =
of
>> =A0 =A0 =A0 =A0 commit access from the PMC. Such reinstatement is subjec=
t to
>> =A0 =A0 =A0 =A0 lazy consensus of active PMC members.
>>
>> =A0 =A0 =A0 All Apache committers are required to have a signed Contribu=
tor
>> =A0 =A0 =A0 =A0 License Agreement (CLA) on file with the Apache Software
>> =A0 =A0 =A0 =A0 Foundation. There is a <a
>> =A0 =A0 =A0 =A0 href=3D"http://www.apache.org/dev/committers.html">Commi=
tter
>> =A0 =A0 =A0 =A0 FAQ</a> which provides more details on the requirements =
for
>> =A0 =A0 =A0 =A0 Committers
>>
>> =A0 =A0 =A0 =A0A committer who makes a sustained contribution to the pro=
ject
>> =A0 =A0 =A0 =A0 may be invited to become a member of the PMC. The form o=
f
>> =A0 =A0 =A0 =A0 contribution is not limited to code. It can also include=
 code
>> =A0 =A0 =A0 =A0 review, helping out users on the mailing lists, document=
ation,
>> =A0 =A0 =A0 =A0 testing, etc.
>>
>> =A0 =A0* Project Management Committee
>>
>> =A0 =A0 =A0 =A0The Project Management Committee (PMC) for Apache Hadoop =
was
>> =A0 =A0 =A0 =A0 created by the Apache Board in January 2008 when Hadoop =
moved
>> =A0 =A0 =A0 =A0 out of Lucene and became a top level project at Apache. =
The
>> =A0 =A0 =A0 =A0 PMC is responsible to the board and the ASF for the mana=
gement
>> =A0 =A0 =A0 =A0 and oversight of the Apache Hadoop codebase. The
>> =A0 =A0 =A0 =A0 responsibilities of the PMC include
>>
>> =A0 =A0 =A0 =A0* Deciding what is distributed as products of the Apache =
Hadoop
>> =A0 =A0 =A0 =A0 project. =A0In particular all releases must be approved =
by the
>> =A0 =A0 =A0 =A0 PMC
>>
>> =A0 =A0 =A0 =A0 * Maintaining the project's shared resources, including =
the
>> codebase
>> =A0 =A0 =A0 =A0 =A0 =A0 repository, mailing lists, websites.
>>
>> =A0 =A0 =A0 =A0 * Speaking on behalf of the project.
>>
>> =A0 =A0 =A0 =A0* Resolving license disputes regarding products of the pr=
oject
>>
>> =A0 =A0 =A0 =A0 * Nominating new PMC members and committers
>>
>> =A0 =A0 =A0 =A0 * Maintaining these bylaws and other guidelines of the p=
roject
>>
>> =A0 =A0 =A0 =A0Membership of the PMC is by invitation only and must be
>> =A0 =A0 =A0 =A0 approved by a lazy consensus of active PMC members. A PM=
C
>> =A0 =A0 =A0 =A0 member is considered &quot;emeritus&quot; by their own
>> =A0 =A0 =A0 =A0 declaration or by not contributing in any form to the pr=
oject
>> =A0 =A0 =A0 =A0 for over six months. An emeritus member may request
>> =A0 =A0 =A0 =A0 reinstatement to the PMC. Such reinstatement is subject =
to
>> =A0 =A0 =A0 =A0 lazy consensus of the active PMC members.
>>
>> =A0 =A0 =A0 =A0The chair of the PMC is appointed by the ASF board. The c=
hair
>> =A0 =A0 =A0 =A0 is an office holder of the Apache Software Foundation (V=
ice
>> =A0 =A0 =A0 =A0 President, Apache Hadoop) and has primary responsibility=
 to
>> =A0 =A0 =A0 =A0 the board for the management of the projects within the =
scope
>> =A0 =A0 =A0 =A0 of the Hadoop PMC. The chair reports to the board quarte=
rly on
>> =A0 =A0 =A0 =A0 developments within the Hadoop project.
>>
>> =A0 =A0 =A0 When the current chair of the PMC resigns, the PMC votes to
>> =A0 =A0 =A0 recommend a new chair using lazy consensus, but the decision
>> =A0 =A0 =A0 must be ratified by the Apache board.
>>
>> Decision Making
>>
>> =A0 =A0 =A0 Within the Hadoop project, different types of decisions requ=
ire
>> =A0 =A0 =A0 different forms of approval. For example, the <a href=3D"#Ro=
les
>> =A0 =A0 =A0 and Responsibilities">previous section</a> describes several
>> =A0 =A0 =A0 decisions which require &quot;lazy consensus&quot;
>> =A0 =A0 =A0 approval. This section defines how voting is performed, the
>> =A0 =A0 =A0 types of approvals, and which types of decision require whic=
h
>> =A0 =A0 =A0 type of approval.
>>
>> =A0 =A0 * Voting
>>
>> =A0 =A0 =A0 =A0 Decisions regarding the project are made by votes on the
>> =A0 =A0 =A0 =A0 primary project development mailing list
>> =A0 =A0 =A0 =A0 (general@hadoop.apache.org). =A0Where necessary, PMC vot=
ing may
>> =A0 =A0 =A0 =A0 take place on the private Hadoop PMC mailing list. =A0Vo=
tes are
>> =A0 =A0 =A0 =A0 clearly indicated by subject line starting with [VOTE]. =
=A0Votes
>> =A0 =A0 =A0 =A0 may contain multiple items for approval and these should=
 be
>> =A0 =A0 =A0 =A0 clearly separated. Voting is carried out by replying to =
the
>> =A0 =A0 =A0 =A0 vote mail. Voting may take four flavours
>>
>> =A0 =A0 =A0 =A0 * +1 &quot;Yes,&quot; &quot;Agree,&quot; or &quot;the ac=
tion
>> should be
>> =A0 =A0 =A0 =A0 =A0 =A0 performed.&quot; In general, this vote also indi=
cates a
>> willingness
>> =A0 =A0 =A0 =A0 =A0 =A0 on the behalf of the voter in &quot;making it ha=
ppen&quot;
>>
>> =A0 =A0 =A0 =A0 =A0* +0 This vote indicates a willingness for the action=
 under
>> =A0 =A0 =A0 =A0 =A0 =A0 consideration to go ahead. The voter, however wi=
ll not
>> be able
>> =A0 =A0 =A0 =A0 =A0 =A0 to help.
>>
>> =A0 =A0 =A0 =A0 =A0* -0 This vote indicates that the voter does not, in
>> general, agree with
>> =A0 =A0 =A0 =A0 =A0 =A0 the proposed action but is not concerned enough =
to
>> prevent the
>> =A0 =A0 =A0 =A0 =A0 =A0 action going ahead.
>>
>> =A0 =A0 =A0 =A0 =A0 * -1 This is a negative vote. On issues where consen=
sus is
>> required,
>> =A0 =A0 =A0 =A0 =A0 =A0 this vote counts as a <strong>veto</strong>. All=
 vetoes
>> must
>> =A0 =A0 =A0 =A0 =A0 =A0 contain an explanation of why the veto is approp=
riate.
>> Vetoes with
>> =A0 =A0 =A0 =A0 =A0 =A0 no explanation are void. It may also be appropri=
ate for
>> a -1 vote
>> =A0 =A0 =A0 =A0 =A0 =A0 to include an alternative course of action.
>>
>> =A0 =A0 =A0 =A0 All participants in the Hadoop project are encouraged to
>> show their
>> =A0 =A0 =A0 =A0 agreement with or against a particular action by voting.=
 For
>> technical
>> =A0 =A0 =A0 =A0 decisions, only the votes of active committers are bindi=
ng.
>> Non binding
>> =A0 =A0 =A0 =A0 votes are still useful for those with binding votes to
>> understand the
>> =A0 =A0 =A0 =A0 perception of an action in the wider Hadoop community. F=
or
>> PMC decisions,
>> =A0 =A0 =A0 =A0 only the votes of PMC members are binding.
>>
>> =A0 =A0 =A0 =A0Voting can also be applied to changes made to the Hadoop
>> codebase. These
>> =A0 =A0 =A0 =A0 typically take the form of a veto (-1) in reply to the
>> commit message
>> =A0 =A0 =A0 =A0 sent when the commit is made.
>>
>> =A0 =A0 * Approvals
>>
>> =A0 =A0 =A0 =A0 These are the types of approvals that can be sought.
>> Different actions
>> =A0 =A0 =A0 =A0 require different types of approvals
>>
>> =A0 =A0 =A0 =A0 * Lazy Consensus - Lazy consensus requires 3 binding +1 =
votes
>> =A0 =A0 =A0 =A0 =A0 =A0and no binding vetoes.
>>
>> =A0 =A0 =A0 =A0 =A0* Lazy Majority - A lazy majority vote requires 3 bin=
ding +1
>> =A0 =A0 =A0 =A0 =A0 =A0 votes and more binding +1 votes than -1 votes.
>>
>> =A0 =A0 =A0 =A0 * Lazy 2/3 Majority - Lazy 2/3 majority votes requires a=
t
>> =A0 =A0 =A0 =A0 =A0 =A0least 3 votes and twice as many +1 votes as -1 vo=
tes.
>>
>> =A0 =A0 * Vetoes
>>
>> =A0 =A0 =A0 =A0 A valid, binding veto cannot be overruled. If a veto is =
cast,
>> =A0 =A0 =A0 =A0 it must be accompanied by a valid reason explaining the
>> =A0 =A0 =A0 =A0 reasons for the veto. The validity of a veto, if challen=
ged,
>> =A0 =A0 =A0 =A0 can be confirmed by anyone who has a binding vote. This =
does
>> =A0 =A0 =A0 =A0 not necessarily signify agreement with the veto - merely=
 that
>> =A0 =A0 =A0 =A0 the veto is valid.
>>
>> =A0 =A0 =A0 =A0If you disagree with a valid veto, you must lobby the per=
son
>> casting
>> =A0 =A0 =A0 =A0 the veto to withdraw their veto. If a veto is not withdr=
awn,
>> any action
>> =A0 =A0 =A0 =A0 that has been vetoed must be reversed in a timely manner=
.
>>
>> =A0 =A0 * Actions
>>
>> =A0 =A0 =A0 =A0 This section describes the various actions which are
>> undertaken within
>> =A0 =A0 =A0 =A0 the project, the corresponding approval required for tha=
t
>> action and
>> =A0 =A0 =A0 =A0 those who have binding votes over the action.
>>
>> =A0 =A0 =A0 =A0 =A0* Code Change
>>
>> =A0 =A0 =A0 =A0 =A0 =A0 A change made to a codebase of the project and c=
ommitted
>> =A0 =A0 =A0 =A0 =A0 =A0 by a committer. This includes source code,
>> documentation, website
>> =A0 =A0 =A0 =A0 =A0 =A0 content, etc.
>>
>> =A0 =A0 =A0 =A0 =A0 =A0 Lazy consensus of active committers.
>>
>> =A0 =A0 =A0 =A0 =A0* Release Plan
>>
>> =A0 =A0 =A0 =A0 =A0 =A0 Defines the timetable and actions for a release.=
 The
>> plan also
>> =A0 =A0 =A0 =A0 =A0 =A0 nominates a Release Manager.
>>
>> =A0 =A0 =A0 =A0 =A0 =A0 Lazy majority of active committers
>>
>> =A0 =A0 =A0 =A0 =A0* Product Release
>>
>> =A0 =A0 =A0 =A0 =A0 =A0 When a release of one of the project's products =
is
>> ready, a vote is
>> =A0 =A0 =A0 =A0 =A0 =A0 required to accept the release as an official re=
lease of
>> the
>> =A0 =A0 =A0 =A0 =A0 =A0 project.
>>
>> =A0 =A0 =A0 =A0 =A0 =A0Lazy Majority of active PMC members
>>
>> =A0 =A0 =A0 =A0 =A0 * Adoption of New Codebase
>>
>> =A0 =A0 =A0 =A0 =A0 =A0 When the codebase for an existing, released prod=
uct is
>> to be
>> =A0 =A0 =A0 =A0 =A0 =A0 replaced with an alternative codebase. If such a=
 vote
>> fails to
>> =A0 =A0 =A0 =A0 =A0 =A0 gain approval, the existing code base will conti=
nue.
>>
>> =A0 =A0 =A0 =A0 =A0 =A0This also covers the creation of new sub-projects
>> =A0 =A0 =A0 =A0 =A0 =A0 within the project
>>
>> =A0 =A0 =A0 =A0 =A0 =A0Lazy 2/3 majority of PMC members
>>
>> =A0 =A0 =A0 =A0 =A0 * New Committer
>>
>> =A0 =A0 =A0 =A0 =A0 =A0 When a new committer is proposed for the project
>>
>> =A0 =A0 =A0 =A0 =A0 =A0 Lazy consensus of active PMC members
>>
>> =A0 =A0 =A0 =A0 =A0 * New PMC Member
>>
>> =A0 =A0 =A0 =A0 =A0 =A0 When a committer is proposed for the PMC
>>
>> =A0 =A0 =A0 =A0 =A0 =A0Lazy consensus of active PMC members
>>
>> =A0 =A0 =A0 =A0 =A0 * Committer Removal
>>
>> =A0 =A0 =A0 =A0 =A0 =A0When removal of commit privileges is sought. =A0N=
ote: Such
>> =A0 =A0 =A0 =A0 =A0 =A0actions will also be referred to the ASF board by=
 the PMC
>> =A0 =A0 =A0 =A0 =A0 =A0chair
>>
>> =A0 =A0 =A0 =A0 =A0 =A0 Majority of active PMC members (excluding the co=
mmitter
>> in question if a
>> =A0 =A0 =A0 =A0 =A0 =A0 member of the PMC).
>>
>> =A0 =A0 =A0 =A0 =A0 * PMC Member Removal
>>
>> =A0 =A0 =A0 =A0 =A0 =A0 When removal of a PMC member is sought. =A0Note:=
 Such
>> =A0 =A0 =A0 =A0 =A0 =A0 actions will also be referred to the ASF board b=
y the PMC
>> =A0 =A0 =A0 =A0 =A0 =A0 chair.
>>
>> =A0 =A0 =A0 =A0 =A0 =A0 Majority of active PMC members (excluding the me=
mber in
>> question)
>>
>> =A0 =A0 =A0 =A0 =A0 * Modifying Bylaws
>>
>> =A0 =A0 =A0 =A0 =A0 =A0 Modifying this document.
>>
>> =A0 =A0 =A0 =A0 =A0 =A0 Majority of active PMC members
>>
>> =A0 =A0* Voting Timeframes
>>
>> =A0 =A0 =A0 =A0 Votes are open for a period of 7 days to allow all activ=
e
>> =A0 =A0 =A0 =A0 voters time to consider the vote. Votes relating to code
>> =A0 =A0 =A0 =A0 changes are not subject to a strict timetable but should=
 be
>> =A0 =A0 =A0 =A0 made as timely as possible.
>>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
>
> iEYEARECAAYFAky+DycACgkQIg9pgB8n5iKiqgCfSj59WPJe/GZ9PIEcCbMBOf15
> N1kAn3+cXTvwAIpVIznExdfo2YlJg+uU
> =3DVu5R
> -----END PGP SIGNATURE-----
>
>

From general-return-2185-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 19 21:47:09 2010
Return-Path: <general-return-2185-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57920 invoked from network); 19 Oct 2010 21:47:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Oct 2010 21:47:08 -0000
Received: (qmail 14600 invoked by uid 500); 19 Oct 2010 21:47:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14541 invoked by uid 500); 19 Oct 2010 21:47:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14533 invoked by uid 99); 19 Oct 2010 21:47:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Oct 2010 21:47:07 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 19 Oct 2010 21:47:06 +0000
Received: (qmail 57800 invoked by uid 99); 19 Oct 2010 21:46:46 -0000
Received: from localhost.apache.org (HELO [192.168.168.103]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Oct 2010 21:46:46 +0000
Message-ID: <4CBE11C5.301@apache.org>
Date: Tue, 19 Oct 2010 14:46:45 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.12) Gecko/20100915 Thunderbird/3.0.8
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSS] Proposed bylaws for Hadoop
References: <63A1C7A9-727B-4492-9998-01F6B6958EDD@apache.org>
In-Reply-To: <63A1C7A9-727B-4492-9998-01F6B6958EDD@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 10/19/2010 02:12 PM, Owen O'Malley wrote:
> * Committer Removal
>
> When removal of commit privileges is sought. Note: Such
> actions will also be referred to the ASF board by the PMC
> chair
>
> Majority of active PMC members (excluding the committer in question if a
> member of the PMC).
>
> * PMC Member Removal
>
> When removal of a PMC member is sought. Note: Such
> actions will also be referred to the ASF board by the PMC
> chair.
>
> Majority of active PMC members (excluding the member in question)

With the exception of these two bullets, these bylaws seem equivalent to 
those posted by several other projects.  Most projects use 
consensus-but-one for committer and PMC removal.  Was this change 
intentional or accidental?

Doug

From general-return-2186-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 20 15:56:31 2010
Return-Path: <general-return-2186-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29993 invoked from network); 20 Oct 2010 15:56:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Oct 2010 15:56:30 -0000
Received: (qmail 58434 invoked by uid 500); 20 Oct 2010 15:56:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58345 invoked by uid 500); 20 Oct 2010 15:56:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58336 invoked by uid 99); 20 Oct 2010 15:56:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 15:56:28 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 15:56:20 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 159191BA369
	for <general@hadoop.apache.org>; Wed, 20 Oct 2010 16:55:58 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id awBg1buJ1PZj for <general@hadoop.apache.org>;
	Wed, 20 Oct 2010 16:55:57 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by colossus.hpl.hp.com (Postfix) with ESMTPS id 7189F1BA5CD
	for <general@hadoop.apache.org>; Wed, 20 Oct 2010 16:55:57 +0100 (BST)
MailScanner-NULL-Check: 1288194936.53823@TruYPxKisLNG6Rb1eyA7EA
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o9KFtaL6020568
	for <general@hadoop.apache.org>; Wed, 20 Oct 2010 16:55:36 +0100 (BST)
Message-ID: <4CBF10F8.6090107@apache.org>
Date: Wed, 20 Oct 2010 16:55:36 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.12) Gecko/20100915 Thunderbird/3.0.8
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSS] Proposed bylaws for Hadoop
References: <63A1C7A9-727B-4492-9998-01F6B6958EDD@apache.org>
In-Reply-To: <63A1C7A9-727B-4492-9998-01F6B6958EDD@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o9KFtaL6020568
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 19/10/10 22:12, Owen O'Malley wrote:
> Apache Hadoop Project Bylaws
>
> This document defines the bylaws under which the Apache Hadoop project
> operates. It defines the roles and responsibilities of the project,
> who may vote, how voting works, how conflicts are resolved, etc.


+1, looks like the standard ASF process

That said
  -the JIRA/hudson review process isn't so common on other projects, 
where it's usually lazy post-commit-veto. That doesn't mean the Hadoop 
process is bad, only that you may want to say "the process used for 
getting code in is stricter than many existing ASF processes due to the 
value of data stored in Hadoop clusters. The exact process for 
committing code will be defined by the PMC and voted on" -then you can 
propose/vote the existing process, but have the ability to change things 
without reconstituting.

-the proposal for big changes that was discussed  may also be something 
that this would cover

-Vetoes can kill a project. It's the situation you don't want to get, 
where someone starts -1 rejecting everything. It might be good to lay 
out what the escalation process is for resolving vetoes that cannot be 
resolved -is there a way to raise with the PMC, etc.?

-steve

>
> Hadoop is a project of the <a
> href="http://www.apache.org/foundation/">Apache Software
> Foundation</a>. The foundation holds the trademark on the name
> "Hadoop" and copyright on Apache code
> including the code in the Hadoop codebase. The <a
> href="http://www.apache.org/foundation/faq.html">foundation FAQ</a>
> explains the operation and background of the foundation.
>
> Hadoop is typical of Apache projects in that it operates under a set
> of principles, known collectively as the &quot;Apache Way&quot;. If
> you are new to Apache development, please refer to the <a
> href="http://incubator.apache.org">Incubator project</a> for more
> information on how Apache projects operate.
>
> Roles and Responsibilities
>
> Apache projects define a set of roles with associated rights and
> responsibilities. These roles govern what tasks an individual may
> perform within the project. The roles are defined in the following
> sections
>
> * Users
>
> The most important participants in the project are people who
> use our software. The majority of our developers start out as
> users and guide their development efforts from the user's
> perspective.
>
> Users contribute to the Apache projects by providing feedback
> to developers in the form of bug reports and feature
> suggestions. As well, users participate in the Apache
> community by helping other users on mailing lists and user
> support forums.
>
> * Contributors
>
> All of the volunteers who are contributing time, code,
> documentation, or resources to the Hadoop Project. A contributor
> that makes sustained, welcome contributions to the project may
> be invited to become a Committer, though the exact timing of
> such invitations depends on many factors.
>
> * Committers
>
> The project's Committers are responsible for the project's
> technical management. Committers have access to a specified
> set of subprojects' subversion repositories. Committers on
> subprojects may cast binding votes on any technical discussion
> regarding that subproject.
>
> Committer access is by invitation only and must be approved by
> lazy consensus of the active PMC members. A Committer is
> considered emeritus by their own declaration or by not
> contributing in any form to the project for over six
> months. An emeritus committer may request reinstatement of
> commit access from the PMC. Such reinstatement is subject to
> lazy consensus of active PMC members.
>
> All Apache committers are required to have a signed Contributor
> License Agreement (CLA) on file with the Apache Software
> Foundation. There is a <a
> href="http://www.apache.org/dev/committers.html">Committer
> FAQ</a> which provides more details on the requirements for
> Committers
>
> A committer who makes a sustained contribution to the project
> may be invited to become a member of the PMC. The form of
> contribution is not limited to code. It can also include code
> review, helping out users on the mailing lists, documentation,
> testing, etc.
>
> * Project Management Committee
>
> The Project Management Committee (PMC) for Apache Hadoop was
> created by the Apache Board in January 2008 when Hadoop moved
> out of Lucene and became a top level project at Apache. The
> PMC is responsible to the board and the ASF for the management
> and oversight of the Apache Hadoop codebase. The
> responsibilities of the PMC include
>
> * Deciding what is distributed as products of the Apache Hadoop
> project. In particular all releases must be approved by the
> PMC
>
> * Maintaining the project's shared resources, including the codebase
> repository, mailing lists, websites.
>
> * Speaking on behalf of the project.
>
> * Resolving license disputes regarding products of the project
>
> * Nominating new PMC members and committers
>
> * Maintaining these bylaws and other guidelines of the project
>
> Membership of the PMC is by invitation only and must be
> approved by a lazy consensus of active PMC members. A PMC
> member is considered &quot;emeritus&quot; by their own
> declaration or by not contributing in any form to the project
> for over six months. An emeritus member may request
> reinstatement to the PMC. Such reinstatement is subject to
> lazy consensus of the active PMC members.
>
> The chair of the PMC is appointed by the ASF board. The chair
> is an office holder of the Apache Software Foundation (Vice
> President, Apache Hadoop) and has primary responsibility to
> the board for the management of the projects within the scope
> of the Hadoop PMC. The chair reports to the board quarterly on
> developments within the Hadoop project.
>
> When the current chair of the PMC resigns, the PMC votes to
> recommend a new chair using lazy consensus, but the decision
> must be ratified by the Apache board.
>
> Decision Making
>
> Within the Hadoop project, different types of decisions require
> different forms of approval. For example, the <a href="#Roles
> and Responsibilities">previous section</a> describes several
> decisions which require &quot;lazy consensus&quot;
> approval. This section defines how voting is performed, the
> types of approvals, and which types of decision require which
> type of approval.
>
> * Voting
>
> Decisions regarding the project are made by votes on the
> primary project development mailing list
> (general@hadoop.apache.org). Where necessary, PMC voting may
> take place on the private Hadoop PMC mailing list. Votes are
> clearly indicated by subject line starting with [VOTE]. Votes
> may contain multiple items for approval and these should be
> clearly separated. Voting is carried out by replying to the
> vote mail. Voting may take four flavours
>
> * +1 &quot;Yes,&quot; &quot;Agree,&quot; or &quot;the action should be
> performed.&quot; In general, this vote also indicates a willingness
> on the behalf of the voter in &quot;making it happen&quot;
>
> * +0 This vote indicates a willingness for the action under
> consideration to go ahead. The voter, however will not be able
> to help.
>
> * -0 This vote indicates that the voter does not, in general, agree with
> the proposed action but is not concerned enough to prevent the
> action going ahead.
>
> * -1 This is a negative vote. On issues where consensus is required,
> this vote counts as a <strong>veto</strong>. All vetoes must
> contain an explanation of why the veto is appropriate. Vetoes with
> no explanation are void. It may also be appropriate for a -1 vote
> to include an alternative course of action.
>
> All participants in the Hadoop project are encouraged to show their
> agreement with or against a particular action by voting. For technical
> decisions, only the votes of active committers are binding. Non binding
> votes are still useful for those with binding votes to understand the
> perception of an action in the wider Hadoop community. For PMC decisions,
> only the votes of PMC members are binding.
>
> Voting can also be applied to changes made to the Hadoop codebase. These
> typically take the form of a veto (-1) in reply to the commit message
> sent when the commit is made.
>
> * Approvals
>
> These are the types of approvals that can be sought. Different actions
> require different types of approvals
>
> * Lazy Consensus - Lazy consensus requires 3 binding +1 votes
> and no binding vetoes.
>
> * Lazy Majority - A lazy majority vote requires 3 binding +1
> votes and more binding +1 votes than -1 votes.
>
> * Lazy 2/3 Majority - Lazy 2/3 majority votes requires at
> least 3 votes and twice as many +1 votes as -1 votes.
>
> * Vetoes
>
> A valid, binding veto cannot be overruled. If a veto is cast,
> it must be accompanied by a valid reason explaining the
> reasons for the veto. The validity of a veto, if challenged,
> can be confirmed by anyone who has a binding vote. This does
> not necessarily signify agreement with the veto - merely that
> the veto is valid.
>
> If you disagree with a valid veto, you must lobby the person casting
> the veto to withdraw their veto. If a veto is not withdrawn, any action
> that has been vetoed must be reversed in a timely manner.
>
> * Actions
>
> This section describes the various actions which are undertaken within
> the project, the corresponding approval required for that action and
> those who have binding votes over the action.
>
> * Code Change
>
> A change made to a codebase of the project and committed
> by a committer. This includes source code, documentation, website
> content, etc.
>
> Lazy consensus of active committers.
>
> * Release Plan
>
> Defines the timetable and actions for a release. The plan also
> nominates a Release Manager.
>
> Lazy majority of active committers
>
> * Product Release
>
> When a release of one of the project's products is ready, a vote is
> required to accept the release as an official release of the
> project.
>
> Lazy Majority of active PMC members
>
> * Adoption of New Codebase
>
> When the codebase for an existing, released product is to be
> replaced with an alternative codebase. If such a vote fails to
> gain approval, the existing code base will continue.
>
> This also covers the creation of new sub-projects
> within the project
>
> Lazy 2/3 majority of PMC members
>
> * New Committer
>
> When a new committer is proposed for the project
>
> Lazy consensus of active PMC members
>
> * New PMC Member
>
> When a committer is proposed for the PMC
>
> Lazy consensus of active PMC members
>
> * Committer Removal
>
> When removal of commit privileges is sought. Note: Such
> actions will also be referred to the ASF board by the PMC
> chair
>
> Majority of active PMC members (excluding the committer in question if a
> member of the PMC).
>
> * PMC Member Removal
>
> When removal of a PMC member is sought. Note: Such
> actions will also be referred to the ASF board by the PMC
> chair.
>
> Majority of active PMC members (excluding the member in question)
>
> * Modifying Bylaws
>
> Modifying this document.
>
> Majority of active PMC members
>
> * Voting Timeframes
>
> Votes are open for a period of 7 days to allow all active
> voters time to consider the vote. Votes relating to code
> changes are not subject to a strict timetable but should be
> made as timely as possible.
>


From general-return-2187-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 20 16:54:44 2010
Return-Path: <general-return-2187-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 51274 invoked from network); 20 Oct 2010 16:54:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Oct 2010 16:54:43 -0000
Received: (qmail 60731 invoked by uid 500); 20 Oct 2010 16:54:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60626 invoked by uid 500); 20 Oct 2010 16:54:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60618 invoked by uid 99); 20 Oct 2010 16:54:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 16:54:42 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.27.211] (HELO qmta11.emeryville.ca.mail.comcast.net) (76.96.27.211)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 16:54:35 +0000
Received: from omta17.emeryville.ca.mail.comcast.net ([76.96.30.73])
	by qmta11.emeryville.ca.mail.comcast.net with comcast
	id M0Zk1f0021afHeLAB4uEBU; Wed, 20 Oct 2010 16:54:14 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta17.emeryville.ca.mail.comcast.net with comcast
	id M4u51f00D2SbwD58d4u8Ue; Wed, 20 Oct 2010 16:54:12 +0000
Message-Id: <CE8E008F-D08E-4564-80F0-B0326246B768@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <4CBE11C5.301@apache.org>
Content-Type: multipart/alternative; boundary=Apple-Mail-5-48800315
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSS] Proposed bylaws for Hadoop
Date: Wed, 20 Oct 2010 09:54:05 -0700
References: <63A1C7A9-727B-4492-9998-01F6B6958EDD@apache.org> <4CBE11C5.301@apache.org>
X-Mailer: Apple Mail (2.936)

--Apple-Mail-5-48800315
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


On Oct 19, 2010, at 2:46 PM, Doug Cutting wrote:

> With the exception of these two bullets, these bylaws seem  
> equivalent to those posted by several other projects.  Most projects  
> use consensus-but-one for committer and PMC removal.  Was this  
> change intentional or accidental?

It was intentional. In my survey of other Apache project's bylaws, I  
was originally surprised to find some such as HTTP Components (http://hc.apache.org/bylaws.html 
) that use majority votes for removing people. However, the question  
is whether a small minority should be able to drain a project's  
attention and energy.

For anyone who hasn't already seen it, there is an outstanding  
presentation on "Open Source Projects and Poisonous People" that was  
done by Brian Fitzpatrick and Ben Collins-Sussman. I'd highly  
recommend it. http://www.youtube.com/watch?v=ZSFDm3UYkeE. The  
presentation's central thesis is that a project's attention and energy  
are its most valuable resource. People that cause long emotional  
debates without contributing to the project are extremely destructive  
to the project and must occasionally be asked to leave.

Of course everyone hopes to avoid these cases, but the question is  
whether the project should have the mechanism to fix itself. I feel  
that it must.

-- Owen
--Apple-Mail-5-48800315--

From general-return-2188-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 20 17:15:47 2010
Return-Path: <general-return-2188-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 64117 invoked from network); 20 Oct 2010 17:15:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Oct 2010 17:15:47 -0000
Received: (qmail 97912 invoked by uid 500); 20 Oct 2010 17:15:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97860 invoked by uid 500); 20 Oct 2010 17:15:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97852 invoked by uid 99); 20 Oct 2010 17:15:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 17:15:45 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.102 as permitted sender)
Received: from [17.148.16.102] (HELO asmtpout027.mac.com) (17.148.16.102)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 17:15:38 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp027.mac.com (Oracle Communications Messaging Exchange Server 7u4-18.01
 64bit (built Jul 15 2010)) with ESMTPSA id <0LAL001AOMKO7G90@asmtp027.mac.com>
 for general@hadoop.apache.org; Wed, 20 Oct 2010 10:14:49 -0700 (PDT)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2010-10-20_08:2010-10-20,2010-10-20,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=5 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1004200000 definitions=main-1010200090
Subject: Re: [DISCUSS] Proposed bylaws for Hadoop
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <CE8E008F-D08E-4564-80F0-B0326246B768@apache.org>
Date: Wed, 20 Oct 2010 10:14:48 -0700
Message-id: <580DC943-D250-405D-A524-82452CBC4E03@mac.com>
References: <63A1C7A9-727B-4492-9998-01F6B6958EDD@apache.org>
 <4CBE11C5.301@apache.org> <CE8E008F-D08E-4564-80F0-B0326246B768@apache.org>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1081)
X-Virus-Checked: Checked by ClamAV on apache.org


On Oct 20, 2010, at 9:54 AM, Owen O'Malley wrote:

> 
> On Oct 19, 2010, at 2:46 PM, Doug Cutting wrote:
> 
>> With the exception of these two bullets, these bylaws seem equivalent to those posted by several other projects.  Most projects use consensus-but-one for committer and PMC removal.  Was this change intentional or accidental?
> 
> It was intentional. In my survey of other Apache project's bylaws, I was originally surprised to find some such as HTTP Components (http://hc.apache.org/bylaws.html) that use majority votes for removing people. However, the question is whether a small minority should be able to drain a project's attention and energy.
> 
> For anyone who hasn't already seen it, there is an outstanding presentation on "Open Source Projects and Poisonous People" that was done by Brian Fitzpatrick and Ben Collins-Sussman. I'd highly recommend it. http://www.youtube.com/watch?v=ZSFDm3UYkeE. The presentation's central thesis is that a project's attention and energy are its most valuable resource. People that cause long emotional debates without contributing to the project are extremely destructive to the project and must occasionally be asked to leave.
> 
> Of course everyone hopes to avoid these cases, but the question is whether the project should have the mechanism to fix itself. I feel that it must.

FWIW, "PMC does not generally operate by majority but by consensus." was given as a rationale when explaining to an existing PMC member why it was ok to approve so many new PMC members from a single company.

Nige

From general-return-2189-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 20 19:48:18 2010
Return-Path: <general-return-2189-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 26408 invoked from network); 20 Oct 2010 19:48:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Oct 2010 19:48:18 -0000
Received: (qmail 68385 invoked by uid 500); 20 Oct 2010 19:48:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68328 invoked by uid 500); 20 Oct 2010 19:48:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68320 invoked by uid 99); 20 Oct 2010 19:48:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 19:48:16 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.105 as permitted sender)
Received: from [17.148.16.105] (HELO asmtpout030.mac.com) (17.148.16.105)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 19:48:07 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; charset=us-ascii
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp030.mac.com (Sun Java(tm) System Messaging Server 6.3-8.01 (built Dec
 16 2008; 32bit)) with ESMTPSA id <0LAL00ACGTMRXF60@asmtp030.mac.com> for
 general@hadoop.apache.org; Wed, 20 Oct 2010 12:47:17 -0700 (PDT)
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=5 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1004200000 definitions=main-1010200111
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2010-10-20_08:2010-10-20,2010-10-20,1970-01-01 signatures=0
From: Nigel Daley <ndaley@mac.com>
Subject: Patch testing
Date: Wed, 20 Oct 2010 12:47:15 -0700
Message-id: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1081)
X-Virus-Checked: Checked by ClamAV on apache.org

Folks, 

I'm working to get the pre-commit patch testing running again for HDFS, HADOOP, and MAPREDUCE patches.  Since I've been on a bit of a hiatus from day-to-day involvement w/ the project over the last 6 months (new job), I want to check in and make sure folks would still find this pre-commit testing useful.  Also, happy to hear any suggested improvement.  Let me know.  

Cheers,
Nige

From general-return-2190-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 20 19:53:28 2010
Return-Path: <general-return-2190-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 28048 invoked from network); 20 Oct 2010 19:53:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Oct 2010 19:53:28 -0000
Received: (qmail 82449 invoked by uid 500); 20 Oct 2010 19:53:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82389 invoked by uid 500); 20 Oct 2010 19:53:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82381 invoked by uid 99); 20 Oct 2010 19:53:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 19:53:27 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 19:53:20 +0000
Received: by qwg8 with SMTP id 8so2770648qwg.35
        for <general@hadoop.apache.org>; Wed, 20 Oct 2010 12:52:59 -0700 (PDT)
Received: by 10.224.49.72 with SMTP id u8mr5059956qaf.9.1287604379813; Wed, 20
 Oct 2010 12:52:59 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.79.193 with HTTP; Wed, 20 Oct 2010 12:52:38 -0700 (PDT)
In-Reply-To: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com>
From: Aaron Myers <atm@cloudera.com>
Date: Wed, 20 Oct 2010 15:52:38 -0400
Message-ID: <AANLkTi=A+GdC4FUgZgF=UdC86YDhGEUEfqahmdqOyexe@mail.gmail.com>
Subject: Re: Patch testing
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485e7c404ab94fb049311c1a1

--001485e7c404ab94fb049311c1a1
Content-Type: text/plain; charset=ISO-8859-1

+1

There have been several tests failing on trunk for entirely too long, and I
believe this will help prevent further regressions while we address those
issues.

Thanks a lot for volunteering to do this work, Nigel.

Aaron

On Wed, Oct 20, 2010 at 3:47 PM, Nigel Daley <ndaley@mac.com> wrote:

> Folks,
>
> I'm working to get the pre-commit patch testing running again for HDFS,
> HADOOP, and MAPREDUCE patches.  Since I've been on a bit of a hiatus from
> day-to-day involvement w/ the project over the last 6 months (new job), I
> want to check in and make sure folks would still find this pre-commit
> testing useful.  Also, happy to hear any suggested improvement.  Let me
> know.
>
> Cheers,
> Nige
>

--001485e7c404ab94fb049311c1a1--

From general-return-2191-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 20 19:54:59 2010
Return-Path: <general-return-2191-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 28351 invoked from network); 20 Oct 2010 19:54:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Oct 2010 19:54:57 -0000
Received: (qmail 86240 invoked by uid 500); 20 Oct 2010 19:54:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86175 invoked by uid 500); 20 Oct 2010 19:54:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86167 invoked by uid 99); 20 Oct 2010 19:54:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 19:54:56 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 19:54:48 +0000
Received: by gwb15 with SMTP id 15so2137915gwb.35
        for <general@hadoop.apache.org>; Wed, 20 Oct 2010 12:54:27 -0700 (PDT)
Received: by 10.103.251.4 with SMTP id d4mr1388422mus.131.1287604465627;
        Wed, 20 Oct 2010 12:54:25 -0700 (PDT)
Received: from tp (c-24-4-185-157.hsd1.ca.comcast.net [24.4.185.157])
        by mx.google.com with ESMTPS id k4sm388450faa.8.2010.10.20.12.54.24
        (version=SSLv3 cipher=RC4-MD5);
        Wed, 20 Oct 2010 12:54:24 -0700 (PDT)
Sender: Konstantin Boudnik <cos@boudnik.org>
Date: Wed, 20 Oct 2010 12:54:20 -0700
From: Konstantin Boudnik <cos@apache.org>
To: general@hadoop.apache.org
Subject: Re: Patch testing
Message-ID: <20101020195420.GG2075@tp>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="//IivP0gvsAy3Can"
Content-Disposition: inline
In-Reply-To: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com>
X-Organization: It's something of 'Cos
X-PGP-Key: http://www.boudnik.org/~cos/pubkey.asc
User-Agent: Mutt/1.5.20 (2009-06-14)

--//IivP0gvsAy3Can
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Thanks for getting to it, Nigel. This is absolutely useful and allows to ke=
ep
weeds out of the trunk up to a certain degree.

There were at least two slightly different sources of information about wha=
t's
going on with test-patch process. Would you mind to post a brief about the
situation so comments can be more substantial?

Thanks,
  Cos

On Wed, Oct 20, 2010 at 12:47PM, Nigel Daley wrote:
> Folks,=20
>=20
> I'm working to get the pre-commit patch testing running again for HDFS,
> HADOOP, and MAPREDUCE patches.  Since I've been on a bit of a hiatus from
> day-to-day involvement w/ the project over the last 6 months (new job), I
> want to check in and make sure folks would still find this pre-commit
> testing useful.  Also, happy to hear any suggested improvement.  Let me
> know. =20
>=20
> Cheers,
> Nige

--//IivP0gvsAy3Can
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEARECAAYFAky/SOwACgkQIg9pgB8n5iJn3gCfUrUvlP+lvELkEgKAjACytwJU
cCQAn0SO7c+pE3I6msad866NFcTmRH1D
=AvyQ
-----END PGP SIGNATURE-----

--//IivP0gvsAy3Can--

From general-return-2192-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 20 20:19:02 2010
Return-Path: <general-return-2192-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39774 invoked from network); 20 Oct 2010 20:19:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Oct 2010 20:19:01 -0000
Received: (qmail 32685 invoked by uid 500); 20 Oct 2010 20:19:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32646 invoked by uid 500); 20 Oct 2010 20:19:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32638 invoked by uid 99); 20 Oct 2010 20:19:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 20:19:00 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 20:18:52 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id o9KKHjNo013255
	for <general@hadoop.apache.org>; Wed, 20 Oct 2010 13:17:45 -0700 (PDT)
Received: from SP2-EX07VS02.ds.corp.yahoo.com ([98.137.59.30]) by
 SP2-EX07CAS01.ds.corp.yahoo.com ([98.137.59.37]) with mapi; Wed, 20 Oct 2010
 13:17:44 -0700
From: "Giridharan  Kesavan" <gkesavan@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 20 Oct 2010 13:17:42 -0700
Subject: Re: Patch testing
Thread-Topic: Patch testing
Thread-Index: Actwk9tpjvwOWloSQUyPNAqCo7QFtg==
Message-ID: <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com>
 <20101020195420.GG2075@tp>
In-Reply-To: <20101020195420.GG2075@tp>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


Old times:
The hudson patch-testing admin job used to run on the master machine hudson=
.zones.apache.org as this machine used to parse the emails from jira and cr=
eate the patch queue for testing.

Current situation:=20
hudson.zones.apache.org machine is no more a master and we have a new machi=
ne called aegis.apache.org as the master machine.=20

This machine is configured to process the email and create the patch queue.=
  <done>
Though we are able to get the email and create the patch queue on the new a=
egis.apache.org machine, still we wont be able to trigger patch admin job o=
n the hudson master as this is not allowed in the new setup. =20

Infra team doesn't want us to run any builds on the hudson master. Instead =
they want all the builds to run on the slave machine. =20

I got the password-less access for hudson user from hudson slave h1.grid.sp=
2.yahoo.net to aegis.apache.org like 2 weeks back. <done>  =20

The plan here is to read the patch queue on aegis from h1 and schedule buil=
ds.   <I'm working on getting this to work>

Longterm solution:=20
email patch process and queue creation is un-reliable. As a long term solut=
ion we need to use jira-cli command line tool to read patch directly from j=
ira.=20
And doing this would require introducing a new workflow in jira. Apart from=
 the current status called "patch-available"

Thanks,
Giri

On Oct 20, 2010, at 12:54 PM, Konstantin Boudnik wrote:

> Thanks for getting to it, Nigel. This is absolutely useful and allows to =
keep
> weeds out of the trunk up to a certain degree.
>=20
> There were at least two slightly different sources of information about w=
hat's
> going on with test-patch process. Would you mind to post a brief about th=
e
> situation so comments can be more substantial?
>=20
> Thanks,
>  Cos
>=20
> On Wed, Oct 20, 2010 at 12:47PM, Nigel Daley wrote:
>> Folks,=20
>>=20
>> I'm working to get the pre-commit patch testing running again for HDFS,
>> HADOOP, and MAPREDUCE patches.  Since I've been on a bit of a hiatus fro=
m
>> day-to-day involvement w/ the project over the last 6 months (new job), =
I
>> want to check in and make sure folks would still find this pre-commit
>> testing useful.  Also, happy to hear any suggested improvement.  Let me
>> know. =20
>>=20
>> Cheers,
>> Nige


From general-return-2193-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 20 20:24:12 2010
Return-Path: <general-return-2193-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 41411 invoked from network); 20 Oct 2010 20:24:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Oct 2010 20:24:11 -0000
Received: (qmail 47196 invoked by uid 500); 20 Oct 2010 20:24:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47155 invoked by uid 500); 20 Oct 2010 20:24:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47147 invoked by uid 99); 20 Oct 2010 20:24:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 20:24:10 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.62.96] (HELO qmta09.westchester.pa.mail.comcast.net) (76.96.62.96)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 20:24:00 +0000
Received: from omta04.westchester.pa.mail.comcast.net ([76.96.62.35])
	by qmta09.westchester.pa.mail.comcast.net with comcast
	id LzSB1f0070ldTLk598PgzQ; Wed, 20 Oct 2010 20:23:40 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta04.westchester.pa.mail.comcast.net with comcast
	id M8PW1f00B2SbwD53Q8PZt7; Wed, 20 Oct 2010 20:23:38 +0000
Message-Id: <FC0C4DE2-8B5B-41B6-8970-629F8E311AB1@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Patch testing
Date: Wed, 20 Oct 2010 13:23:29 -0700
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


On Oct 20, 2010, at 12:47 PM, Nigel Daley wrote:

> I'm working to get the pre-commit patch testing running again for  
> HDFS, HADOOP, and MAPREDUCE patches.

That would be great, Nigel.

Thanks,
    Owen

From general-return-2194-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 20 20:28:35 2010
Return-Path: <general-return-2194-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42242 invoked from network); 20 Oct 2010 20:28:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Oct 2010 20:28:35 -0000
Received: (qmail 54228 invoked by uid 500); 20 Oct 2010 20:28:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54124 invoked by uid 500); 20 Oct 2010 20:28:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54116 invoked by uid 99); 20 Oct 2010 20:28:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 20:28:34 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 20:28:25 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o9KKRrhn087227
	for <general@hadoop.apache.org>; Wed, 20 Oct 2010 13:27:53 -0700 (PDT)
Received: from SP2-EX07VS04.ds.corp.yahoo.com ([98.137.59.33]) by
 SP2-EX07CAS01.ds.corp.yahoo.com ([98.137.59.37]) with mapi; Wed, 20 Oct 2010
 13:27:53 -0700
From: Mahadev Konar <mahadev@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 20 Oct 2010 13:27:51 -0700
Subject: Re: Patch testing
Thread-Topic: Patch testing
Thread-Index: ActwlPfdtcX18DVET92IT55U0GbJWwAAEyES
Message-ID: <C8E49ED7.447A0%mahadev@yahoo-inc.com>
In-Reply-To: <FC0C4DE2-8B5B-41B6-8970-629F8E311AB1@apache.org>
Accept-Language: en, en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en, en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Great.=20

Nigel,=20
 Can you please document in somewhere on how you fixed it? We=B9d like to f=
ix
it for ZooKeeper as well.

Thanks
mahadev


On 10/20/10 1:23 PM, "Owen O'Malley" <omalley@apache.org> wrote:

>=20
>=20
> On Oct 20, 2010, at 12:47 PM, Nigel Daley wrote:
>=20
>> I'm working to get the pre-commit patch testing running again for
>> HDFS, HADOOP, and MAPREDUCE patches.
>=20
> That would be great, Nigel.
>=20
> Thanks,
>     Owen
>=20


From general-return-2195-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 20 20:31:37 2010
Return-Path: <general-return-2195-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42947 invoked from network); 20 Oct 2010 20:31:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Oct 2010 20:31:37 -0000
Received: (qmail 58186 invoked by uid 500); 20 Oct 2010 20:31:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58017 invoked by uid 500); 20 Oct 2010 20:31:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58009 invoked by uid 99); 20 Oct 2010 20:31:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 20:31:35 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.100 as permitted sender)
Received: from [17.148.16.100] (HELO asmtpout025.mac.com) (17.148.16.100)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 20:31:28 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; charset=us-ascii
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp025.mac.com (Sun Java(tm) System Messaging Server 6.3-7.04 (built Sep
 26 2008; 64bit)) with ESMTPSA id <0LAL000LAVMP3Q10@asmtp025.mac.com> for
 general@hadoop.apache.org; Wed, 20 Oct 2010 13:30:26 -0700 (PDT)
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=5 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1004200000 definitions=main-1010200120
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2010-10-20_08:2010-10-20,2010-10-20,1970-01-01 signatures=0
Subject: Re: Patch testing
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com>
Date: Wed, 20 Oct 2010 13:30:25 -0700
Message-id: <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com>
 <20101020195420.GG2075@tp> <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1081)
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks Giri.

> Longterm solution: 
> email patch process and queue creation is un-reliable. As a long term solution we need to use jira-cli command line tool to read patch directly from jira. 

Ya, that's what I've got mostly working on my machine.  Any open Jira's for this work?  If not, I'll open one.

> And doing this would require introducing a new workflow in jira. Apart from the current status called "patch-available"

Not sure that's necessary.  I think if we change the patch testing model a little, we can keep the current workflow.  Also, given some recent advances in Hudson, I think we can reduce all the patch jobs down to 1 admin job covering *all* projects and 1 job per project -- and we should be able to drastically improve utilization of the slaves we have.  Let's move this discussion to a ticket.  LMK if I need to open one.

Cheers,
Nige

On Oct 20, 2010, at 1:17 PM, Giridharan Kesavan wrote:

> 
> Old times:
> The hudson patch-testing admin job used to run on the master machine hudson.zones.apache.org as this machine used to parse the emails from jira and create the patch queue for testing.
> 
> Current situation: 
> hudson.zones.apache.org machine is no more a master and we have a new machine called aegis.apache.org as the master machine. 
> 
> This machine is configured to process the email and create the patch queue.  <done>
> Though we are able to get the email and create the patch queue on the new aegis.apache.org machine, still we wont be able to trigger patch admin job on the hudson master as this is not allowed in the new setup.  
> 
> Infra team doesn't want us to run any builds on the hudson master. Instead they want all the builds to run on the slave machine.  
> 
> I got the password-less access for hudson user from hudson slave h1.grid.sp2.yahoo.net to aegis.apache.org like 2 weeks back. <done>   
> 
> The plan here is to read the patch queue on aegis from h1 and schedule builds.   <I'm working on getting this to work>
> 
> Longterm solution: 
> email patch process and queue creation is un-reliable. As a long term solution we need to use jira-cli command line tool to read patch directly from jira. 
> And doing this would require introducing a new workflow in jira. Apart from the current status called "patch-available"
> 
> Thanks,
> Giri
> 
> On Oct 20, 2010, at 12:54 PM, Konstantin Boudnik wrote:
> 
>> Thanks for getting to it, Nigel. This is absolutely useful and allows to keep
>> weeds out of the trunk up to a certain degree.
>> 
>> There were at least two slightly different sources of information about what's
>> going on with test-patch process. Would you mind to post a brief about the
>> situation so comments can be more substantial?
>> 
>> Thanks,
>> Cos
>> 
>> On Wed, Oct 20, 2010 at 12:47PM, Nigel Daley wrote:
>>> Folks, 
>>> 
>>> I'm working to get the pre-commit patch testing running again for HDFS,
>>> HADOOP, and MAPREDUCE patches.  Since I've been on a bit of a hiatus from
>>> day-to-day involvement w/ the project over the last 6 months (new job), I
>>> want to check in and make sure folks would still find this pre-commit
>>> testing useful.  Also, happy to hear any suggested improvement.  Let me
>>> know.  
>>> 
>>> Cheers,
>>> Nige
> 


From general-return-2196-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 20 20:32:06 2010
Return-Path: <general-return-2196-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43127 invoked from network); 20 Oct 2010 20:32:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Oct 2010 20:32:06 -0000
Received: (qmail 59745 invoked by uid 500); 20 Oct 2010 20:32:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59667 invoked by uid 500); 20 Oct 2010 20:32:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59659 invoked by uid 99); 20 Oct 2010 20:32:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 20:32:04 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 20:31:58 +0000
Received: by fxm13 with SMTP id 13so3257337fxm.35
        for <general@hadoop.apache.org>; Wed, 20 Oct 2010 13:31:37 -0700 (PDT)
Received: by 10.103.171.9 with SMTP id y9mr1921258muo.40.1287606697171;
        Wed, 20 Oct 2010 13:31:37 -0700 (PDT)
Received: from tp (c-24-4-185-157.hsd1.ca.comcast.net [24.4.185.157])
        by mx.google.com with ESMTPS id j8sm413740fah.30.2010.10.20.13.31.35
        (version=SSLv3 cipher=RC4-MD5);
        Wed, 20 Oct 2010 13:31:36 -0700 (PDT)
Sender: Konstantin Boudnik <cos@boudnik.org>
Date: Wed, 20 Oct 2010 13:31:31 -0700
From: Konstantin Boudnik <cos@apache.org>
To: general@hadoop.apache.org
Subject: Re: Patch testing
Message-ID: <20101020203131.GA17884@tp>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com>
 <20101020195420.GG2075@tp>
 <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="yrj/dFKFPuw6o+aM"
Content-Disposition: inline
In-Reply-To: <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com>
X-Organization: It's something of 'Cos
X-PGP-Key: http://www.boudnik.org/~cos/pubkey.asc
User-Agent: Mutt/1.5.20 (2009-06-14)

--yrj/dFKFPuw6o+aM
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Thanks Giri!

I'm totally agree that we need to move away from the email triggering.=20
And in order to make changes to JIRA's workflow someone needs to work with
INFRA folks or a person with admin access should suffice?=20

I'm offering my help for either way, so lemme know.

Cos

On Wed, Oct 20, 2010 at 01:17PM, Giridharan  Kesavan wrote:
>=20
> Old times:
> The hudson patch-testing admin job used to run on the master machine
> hudson.zones.apache.org as this machine used to parse the emails from jira
> and create the patch queue for testing.
>=20
> Current situation:=20
> hudson.zones.apache.org machine is no more a master and we have a new
> machine called aegis.apache.org as the master machine.=20
>=20
> This machine is configured to process the email and create the patch queu=
e.  <done>
> Though we are able to get the email and create the patch queue on the new
> aegis.apache.org machine, still we wont be able to trigger patch admin job
> on the hudson master as this is not allowed in the new setup. =20
>=20
> Infra team doesn't want us to run any builds on the hudson master. Instead
> they want all the builds to run on the slave machine. =20
>=20
> I got the password-less access for hudson user from hudson slave
> h1.grid.sp2.yahoo.net to aegis.apache.org like 2 weeks back. <done>  =20
>=20
> The plan here is to read the patch queue on aegis from h1 and schedule
> builds.   <I'm working on getting this to work>
>=20
> Longterm solution:=20
> email patch process and queue creation is un-reliable. As a long term
> solution we need to use jira-cli command line tool to read patch directly
> from jira.  And doing this would require introducing a new workflow in ji=
ra.
> Apart from the current status called "patch-available"
>=20
> Thanks,
> Giri
>=20
> On Oct 20, 2010, at 12:54 PM, Konstantin Boudnik wrote:
>=20
> > Thanks for getting to it, Nigel. This is absolutely useful and allows t=
o keep
> > weeds out of the trunk up to a certain degree.
> >=20
> > There were at least two slightly different sources of information about=
 what's
> > going on with test-patch process. Would you mind to post a brief about =
the
> > situation so comments can be more substantial?
> >=20
> > Thanks,
> >  Cos
> >=20
> > On Wed, Oct 20, 2010 at 12:47PM, Nigel Daley wrote:
> >> Folks,=20
> >>=20
> >> I'm working to get the pre-commit patch testing running again for HDFS,
> >> HADOOP, and MAPREDUCE patches.  Since I've been on a bit of a hiatus f=
rom
> >> day-to-day involvement w/ the project over the last 6 months (new job)=
, I
> >> want to check in and make sure folks would still find this pre-commit
> >> testing useful.  Also, happy to hear any suggested improvement.  Let me
> >> know. =20
> >>=20
> >> Cheers,
> >> Nige
>=20

--yrj/dFKFPuw6o+aM
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEARECAAYFAky/UaMACgkQIg9pgB8n5iLwqwCgk5Voh3Cyyq8CGNAgvVtezlsQ
ERsAn1aQpQdO30aBH5Fmg5g02QwjpfDh
=ssWS
-----END PGP SIGNATURE-----

--yrj/dFKFPuw6o+aM--

From general-return-2197-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 20 20:32:32 2010
Return-Path: <general-return-2197-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43259 invoked from network); 20 Oct 2010 20:32:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Oct 2010 20:32:32 -0000
Received: (qmail 61029 invoked by uid 500); 20 Oct 2010 20:32:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60982 invoked by uid 500); 20 Oct 2010 20:32:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60974 invoked by uid 99); 20 Oct 2010 20:32:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 20:32:30 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.98 as permitted sender)
Received: from [17.148.16.98] (HELO asmtpout023.mac.com) (17.148.16.98)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 20:32:21 +0000
MIME-version: 1.0
Content-type: text/plain; charset=iso-8859-1
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp023.mac.com (Sun Java(tm) System Messaging Server 6.3-8.01 (built Dec
 16 2008; 32bit)) with ESMTPSA id <0LAL00HG7VPA2S40@asmtp023.mac.com> for
 general@hadoop.apache.org; Wed, 20 Oct 2010 13:31:58 -0700 (PDT)
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=5 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1004200000 definitions=main-1010200120
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2010-10-20_08:2010-10-20,2010-10-20,1970-01-01 signatures=0
Subject: Re: Patch testing
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <C8E49ED7.447A0%mahadev@yahoo-inc.com>
Date: Wed, 20 Oct 2010 13:31:57 -0700
Content-transfer-encoding: quoted-printable
Message-id: <0C437F3B-1577-49E1-9C6A-7D4AA53ED9D9@mac.com>
References: <C8E49ED7.447A0%mahadev@yahoo-inc.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1081)
X-Virus-Checked: Checked by ClamAV on apache.org

I think ZK, PIG, etc will also be included in the update I'm working on.

Cheers,
Nige

On Oct 20, 2010, at 1:27 PM, Mahadev Konar wrote:

> Great.=20
>=20
> Nigel,=20
> Can you please document in somewhere on how you fixed it? We=B9d like =
to fix
> it for ZooKeeper as well.
>=20
> Thanks
> mahadev
>=20
>=20
> On 10/20/10 1:23 PM, "Owen O'Malley" <omalley@apache.org> wrote:
>=20
>>=20
>>=20
>> On Oct 20, 2010, at 12:47 PM, Nigel Daley wrote:
>>=20
>>> I'm working to get the pre-commit patch testing running again for
>>> HDFS, HADOOP, and MAPREDUCE patches.
>>=20
>> That would be great, Nigel.
>>=20
>> Thanks,
>>    Owen
>>=20
>=20


From general-return-2198-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 20 20:38:14 2010
Return-Path: <general-return-2198-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44820 invoked from network); 20 Oct 2010 20:38:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Oct 2010 20:38:14 -0000
Received: (qmail 70227 invoked by uid 500); 20 Oct 2010 20:38:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70175 invoked by uid 500); 20 Oct 2010 20:38:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70167 invoked by uid 99); 20 Oct 2010 20:38:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 20:38:12 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 20:38:06 +0000
Received: by qwg8 with SMTP id 8so2814333qwg.35
        for <general@hadoop.apache.org>; Wed, 20 Oct 2010 13:37:45 -0700 (PDT)
Received: by 10.224.3.137 with SMTP id 9mr2756071qan.142.1287607065401; Wed,
 20 Oct 2010 13:37:45 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.79.193 with HTTP; Wed, 20 Oct 2010 13:37:25 -0700 (PDT)
In-Reply-To: <C8E49ED7.447A0%mahadev@yahoo-inc.com>
References: <FC0C4DE2-8B5B-41B6-8970-629F8E311AB1@apache.org> <C8E49ED7.447A0%mahadev@yahoo-inc.com>
From: Aaron Myers <atm@cloudera.com>
Date: Wed, 20 Oct 2010 16:37:25 -0400
Message-ID: <AANLkTikQWqKtY4-15PxV0fhe=CxJjRo3OHQ1WaUh-9ou@mail.gmail.com>
Subject: Re: Patch testing
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001517510c4cbe59e80493126193

--001517510c4cbe59e80493126193
Content-Type: text/plain; charset=ISO-8859-1

One feature request, which may be difficult to implement, and may not be
worth it:

It would be nice to somehow support running tests for patches which require
changes that span core/MR/HDFS. As it stands, I believe an HDFS patch will
be built and run against a trunk jar, but if this HDFS patch requires some
not-yet-committed work that's being done in a HADOOP JIRA, the HDFS tests
will fail.

Obviously, just getting hudson QA running again is a necessary pre-requisite
for this. :)

Aaron

--001517510c4cbe59e80493126193--

From general-return-2199-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 20 20:56:53 2010
Return-Path: <general-return-2199-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 65375 invoked from network); 20 Oct 2010 20:56:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Oct 2010 20:56:52 -0000
Received: (qmail 7279 invoked by uid 500); 20 Oct 2010 20:56:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7213 invoked by uid 500); 20 Oct 2010 20:56:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7205 invoked by uid 99); 20 Oct 2010 20:56:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 20:56:51 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 20:56:43 +0000
Received: from SP2-EX07CAS02.ds.corp.yahoo.com (sp2-ex07cas02.corp.sp2.yahoo.com [98.137.59.38])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id o9KKu1Tm097903
	for <general@hadoop.apache.org>; Wed, 20 Oct 2010 13:56:01 -0700 (PDT)
Received: from SP2-EX07VS02.ds.corp.yahoo.com ([98.137.59.30]) by
 SP2-EX07CAS02.ds.corp.yahoo.com ([98.137.59.38]) with mapi; Wed, 20 Oct 2010
 13:56:01 -0700
From: "Giridharan  Kesavan" <gkesavan@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 20 Oct 2010 13:55:57 -0700
Subject: Re: Patch testing
Thread-Topic: Patch testing
Thread-Index: ActwmTO1OsXh6BaaTsuN7U3w0XQSIw==
Message-ID: <F8DA5EB8-37A6-4124-A339-4A79DE1FE417@yahoo-inc.com>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com>
 <20101020195420.GG2075@tp>
 <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com>
 <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
In-Reply-To: <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

I ve just filed this jira for discussions about patch-testing.

https://issues.apache.org/jira/browse/HADOOP-7003

Thanks,
-Giri


On Oct 20, 2010, at 1:30 PM, Nigel Daley wrote:

> Thanks Giri.
>=20
>> Longterm solution:=20
>> email patch process and queue creation is un-reliable. As a long term so=
lution we need to use jira-cli command line tool to read patch directly fro=
m jira.=20
>=20
> Ya, that's what I've got mostly working on my machine.  Any open Jira's f=
or this work?  If not, I'll open one.
>=20
>> And doing this would require introducing a new workflow in jira. Apart f=
rom the current status called "patch-available"
>=20
> Not sure that's necessary.  I think if we change the patch testing model =
a little, we can keep the current workflow.  Also, given some recent advanc=
es in Hudson, I think we can reduce all the patch jobs down to 1 admin job =
covering *all* projects and 1 job per project -- and we should be able to d=
rastically improve utilization of the slaves we have.  Let's move this disc=
ussion to a ticket.  LMK if I need to open one.
>=20
> Cheers,
> Nige
>=20
> On Oct 20, 2010, at 1:17 PM, Giridharan Kesavan wrote:
>=20
>>=20
>> Old times:
>> The hudson patch-testing admin job used to run on the master machine hud=
son.zones.apache.org as this machine used to parse the emails from jira and=
 create the patch queue for testing.
>>=20
>> Current situation:=20
>> hudson.zones.apache.org machine is no more a master and we have a new ma=
chine called aegis.apache.org as the master machine.=20
>>=20
>> This machine is configured to process the email and create the patch que=
ue.  <done>
>> Though we are able to get the email and create the patch queue on the ne=
w aegis.apache.org machine, still we wont be able to trigger patch admin jo=
b on the hudson master as this is not allowed in the new setup. =20
>>=20
>> Infra team doesn't want us to run any builds on the hudson master. Inste=
ad they want all the builds to run on the slave machine. =20
>>=20
>> I got the password-less access for hudson user from hudson slave h1.grid=
.sp2.yahoo.net to aegis.apache.org like 2 weeks back. <done>  =20
>>=20
>> The plan here is to read the patch queue on aegis from h1 and schedule b=
uilds.   <I'm working on getting this to work>
>>=20
>> Longterm solution:=20
>> email patch process and queue creation is un-reliable. As a long term so=
lution we need to use jira-cli command line tool to read patch directly fro=
m jira.=20
>> And doing this would require introducing a new workflow in jira. Apart f=
rom the current status called "patch-available"
>>=20
>> Thanks,
>> Giri
>>=20
>> On Oct 20, 2010, at 12:54 PM, Konstantin Boudnik wrote:
>>=20
>>> Thanks for getting to it, Nigel. This is absolutely useful and allows t=
o keep
>>> weeds out of the trunk up to a certain degree.
>>>=20
>>> There were at least two slightly different sources of information about=
 what's
>>> going on with test-patch process. Would you mind to post a brief about =
the
>>> situation so comments can be more substantial?
>>>=20
>>> Thanks,
>>> Cos
>>>=20
>>> On Wed, Oct 20, 2010 at 12:47PM, Nigel Daley wrote:
>>>> Folks,=20
>>>>=20
>>>> I'm working to get the pre-commit patch testing running again for HDFS=
,
>>>> HADOOP, and MAPREDUCE patches.  Since I've been on a bit of a hiatus f=
rom
>>>> day-to-day involvement w/ the project over the last 6 months (new job)=
, I
>>>> want to check in and make sure folks would still find this pre-commit
>>>> testing useful.  Also, happy to hear any suggested improvement.  Let m=
e
>>>> know. =20
>>>>=20
>>>> Cheers,
>>>> Nige
>>=20
>=20


From general-return-2200-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 20 21:13:40 2010
Return-Path: <general-return-2200-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73192 invoked from network); 20 Oct 2010 21:13:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Oct 2010 21:13:39 -0000
Received: (qmail 27364 invoked by uid 500); 20 Oct 2010 21:13:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27291 invoked by uid 500); 20 Oct 2010 21:13:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27283 invoked by uid 99); 20 Oct 2010 21:13:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 21:13:38 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of tom.e.white@gmail.com designates 209.85.213.48 as permitted sender)
Received: from [209.85.213.48] (HELO mail-yw0-f48.google.com) (209.85.213.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 21:13:32 +0000
Received: by ywj3 with SMTP id 3so158659ywj.35
        for <general@hadoop.apache.org>; Wed, 20 Oct 2010 14:13:11 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=7WTj9ZsQbRzQL/LgXQMhmK+billWfLyP8Rn3c70htb0=;
        b=qGHsKiwDrlpUa+e3gCNfeA2Q3ABuLa4gnQ344TMstNGsSzgrIClRLiAfFr+qIueWU4
         qiUvpMYiC/T+uCJk5EVD+FjHsd+2tXBLppcR+QFcf8zyBMS033Pmbm+LgRXw2zjc8wzd
         iTUVBiotxGa679XXIRAsF2xXqnZLfn4+Pe97s=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=Utqy7f1BT9Mb1q7A0FCm4BJwgIz2xbjPVNa/ylnirAIHq1Oc3KpCJ4fme0Z1YpGwcv
         kTcvkhd707400UiksoH2slFZl/BwD3xQdgODK+ROnot6Hf+v6Vq8KXsJ5lFB+sPYw39H
         rB8Rg5BuAyrLmyMNhHWYJT24z8eyKB6RE7SIQ=
Received: by 10.42.208.205 with SMTP id gd13mr5757124icb.308.1287609190726;
 Wed, 20 Oct 2010 14:13:10 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.220.70.147 with HTTP; Wed, 20 Oct 2010 14:12:50 -0700 (PDT)
In-Reply-To: <CE8E008F-D08E-4564-80F0-B0326246B768@apache.org>
References: <63A1C7A9-727B-4492-9998-01F6B6958EDD@apache.org>
 <4CBE11C5.301@apache.org> <CE8E008F-D08E-4564-80F0-B0326246B768@apache.org>
From: Tom White <tom.e.white@gmail.com>
Date: Wed, 20 Oct 2010 14:12:50 -0700
Message-ID: <AANLkTinj0b2_mVZeY=_irWReMMJ2dth6u8__dALAVYvr@mail.gmail.com>
Subject: Re: [DISCUSS] Proposed bylaws for Hadoop
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks for the explanation Owen. The HttpComponents bylaws define
"majority" as "A majority decision passes if there is a minimum of
three binding +1 votes and at least 75% of the binding votes are +1."
So this is not the same as the 50% majority defined the the draft.
(Actually, I notice that only "lazy majority" is defined, not
"majority" in the approvals section.) I suggest we make this stronger.
By way of comparison, the recently enacted bylaws for Pig
(http://pig.apache.org/bylaws.html) have consensus, for example.

Also, Pig has a minimum length of time for each action, rather than
the same fixed number for each. Might be worth considering.

Cheers,
Tom

On Wed, Oct 20, 2010 at 9:54 AM, Owen O'Malley <omalley@apache.org> wrote:
>
> On Oct 19, 2010, at 2:46 PM, Doug Cutting wrote:
>
>> With the exception of these two bullets, these bylaws seem equivalent to
>> those posted by several other projects. =A0Most projects use consensus-b=
ut-one
>> for committer and PMC removal. =A0Was this change intentional or acciden=
tal?
>
> It was intentional. In my survey of other Apache project's bylaws, I was
> originally surprised to find some such as HTTP Components
> (http://hc.apache.org/bylaws.html) that use majority votes for removing
> people. However, the question is whether a small minority should be able =
to
> drain a project's attention and energy.
>
> For anyone who hasn't already seen it, there is an outstanding presentati=
on
> on "Open Source Projects and Poisonous People" that was done by Brian
> Fitzpatrick and Ben Collins-Sussman. I'd highly recommend it.
> http://www.youtube.com/watch?v=3DZSFDm3UYkeE. The presentation's central
> thesis is that a project's attention and energy are its most valuable
> resource. People that cause long emotional debates without contributing t=
o
> the project are extremely destructive to the project and must occasionall=
y
> be asked to leave.
>
> Of course everyone hopes to avoid these cases, but the question is whethe=
r
> the project should have the mechanism to fix itself. I feel that it must.
>
> -- Owen

From general-return-2201-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 20 21:13:47 2010
Return-Path: <general-return-2201-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73295 invoked from network); 20 Oct 2010 21:13:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Oct 2010 21:13:47 -0000
Received: (qmail 28608 invoked by uid 500); 20 Oct 2010 21:13:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28562 invoked by uid 500); 20 Oct 2010 21:13:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28554 invoked by uid 99); 20 Oct 2010 21:13:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 21:13:45 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 21:13:38 +0000
Received: by bwz5 with SMTP id 5so3380861bwz.35
        for <general@hadoop.apache.org>; Wed, 20 Oct 2010 14:13:16 -0700 (PDT)
Received: by 10.103.181.15 with SMTP id i15mr6202636mup.16.1287609196316;
        Wed, 20 Oct 2010 14:13:16 -0700 (PDT)
Received: from tp (c-24-4-185-157.hsd1.ca.comcast.net [24.4.185.157])
        by mx.google.com with ESMTPS id p2sm438168fak.22.2010.10.20.14.13.14
        (version=SSLv3 cipher=RC4-MD5);
        Wed, 20 Oct 2010 14:13:15 -0700 (PDT)
Sender: Konstantin Boudnik <cos@boudnik.org>
Date: Wed, 20 Oct 2010 14:13:10 -0700
From: Konstantin Boudnik <cos@apache.org>
To: general@hadoop.apache.org
Subject: Re: Patch testing
Message-ID: <20101020211310.GB17884@tp>
References: <FC0C4DE2-8B5B-41B6-8970-629F8E311AB1@apache.org>
 <C8E49ED7.447A0%mahadev@yahoo-inc.com>
 <AANLkTikQWqKtY4-15PxV0fhe=CxJjRo3OHQ1WaUh-9ou@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="oLBj+sq0vYjzfsbl"
Content-Disposition: inline
In-Reply-To: <AANLkTikQWqKtY4-15PxV0fhe=CxJjRo3OHQ1WaUh-9ou@mail.gmail.com>
X-Organization: It's something of 'Cos
X-PGP-Key: http://www.boudnik.org/~cos/pubkey.asc
User-Agent: Mutt/1.5.20 (2009-06-14)

--oLBj+sq0vYjzfsbl
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Actually, there's a way to do it, but is isn't simple as you said and requi=
re
certain JIRA management discipline like linking JIRAs properly, for instanc=
e.

Another veritable here is to submit patches for all involved project at onc=
e or
to be able to postpone a patch validation if patch-A isn't available when
patch-B is submitted (project B depends on A, obviously).

After that everything else is real easy: build dependencies, mvn-install
artifacts, build the dependent one with locally installed artifacts, clean =
mvn
cache after all. The tricky part though is to guarantee that these chained
builds are executed on the same machine. Otherwise, local mvn repo
synchronization will be one huge mess ;)

So, in other words, let's restore the status quo first ;) IMO
  Cos

On Wed, Oct 20, 2010 at 04:37PM, Aaron Myers wrote:
> One feature request, which may be difficult to implement, and may not be
> worth it:
>=20
> It would be nice to somehow support running tests for patches which requi=
re
> changes that span core/MR/HDFS. As it stands, I believe an HDFS patch will
> be built and run against a trunk jar, but if this HDFS patch requires some
> not-yet-committed work that's being done in a HADOOP JIRA, the HDFS tests
> will fail.
>=20
> Obviously, just getting hudson QA running again is a necessary pre-requis=
ite
> for this. :)
>=20
> Aaron

--oLBj+sq0vYjzfsbl
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEARECAAYFAky/W2YACgkQIg9pgB8n5iKlwwCfXAMBk3ke6W5NQ/Giu1rOYpe6
u5kAn1GzDowSUemOpVFSUTvPPk0TGvwa
=Tncj
-----END PGP SIGNATURE-----

--oLBj+sq0vYjzfsbl--

From general-return-2202-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 20 21:21:44 2010
Return-Path: <general-return-2202-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76860 invoked from network); 20 Oct 2010 21:21:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Oct 2010 21:21:44 -0000
Received: (qmail 44396 invoked by uid 500); 20 Oct 2010 21:21:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44221 invoked by uid 500); 20 Oct 2010 21:21:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44213 invoked by uid 99); 20 Oct 2010 21:21:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 21:21:42 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of dhruba@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 21:21:36 +0000
Received: by wwb18 with SMTP id 18so3216116wwb.29
        for <general@hadoop.apache.org>; Wed, 20 Oct 2010 14:21:16 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=S6RN5NqLfUhgoyBY8SWh8++mGBAHjzZ1b0EN3hZGW24=;
        b=nXoMLoNxsvE2ElODj4f55N2Rq+Nre30wwomtwvguszKWkkon4t6mrlggDE5z8bVptR
         hc20B2wzRK7O2cnr6SReyLf/f4/+B2sNS3rht70+t9obxSji3gD04SreFLbYQGTHeWNF
         +WsYgL/vqv2u16H7iIHeBPnq8Iv4vRRXK2Dkk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=T0TcNGtsueEDybbI1nUaqD7y7CDJSek2RNXmP3egTqYxUS4mziirEQeaFL/KsTXqwR
         LcOtxnhty3kf53difKquwI2OFaam4g9aDPP1x1gi/pYBcs2aImhsN1cHrX+qx8Q6Pox1
         Q5GnWFxa6IBZIjfWmVOSAN5JkuU3GnhjpNAew=
MIME-Version: 1.0
Received: by 10.227.152.9 with SMTP id e9mr102411wbw.169.1287609676310; Wed,
 20 Oct 2010 14:21:16 -0700 (PDT)
Received: by 10.216.183.85 with HTTP; Wed, 20 Oct 2010 14:21:16 -0700 (PDT)
In-Reply-To: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com>
Date: Wed, 20 Oct 2010 14:21:16 -0700
Message-ID: <AANLkTikOb=sr5CUgU92+4AaBXy_3NbgZ=DEYKep0fLUy@mail.gmail.com>
Subject: Re: Patch testing
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f6cbfc5da703049312fdea
X-Virus-Checked: Checked by ClamAV on apache.org

--001485f6cbfc5da703049312fdea
Content-Type: text/plain; charset=ISO-8859-1

Awesome Nigel! That will be of real help.

thanks,
dhruba


On Wed, Oct 20, 2010 at 12:47 PM, Nigel Daley <ndaley@mac.com> wrote:

> Folks,
>
> I'm working to get the pre-commit patch testing running again for HDFS,
> HADOOP, and MAPREDUCE patches.  Since I've been on a bit of a hiatus from
> day-to-day involvement w/ the project over the last 6 months (new job), I
> want to check in and make sure folks would still find this pre-commit
> testing useful.  Also, happy to hear any suggested improvement.  Let me
> know.
>
> Cheers,
> Nige
>



-- 
Connect to me at http://www.facebook.com/dhruba

--001485f6cbfc5da703049312fdea--

From general-return-2203-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 20 21:46:47 2010
Return-Path: <general-return-2203-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83315 invoked from network); 20 Oct 2010 21:46:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Oct 2010 21:46:47 -0000
Received: (qmail 76906 invoked by uid 500); 20 Oct 2010 21:46:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76654 invoked by uid 500); 20 Oct 2010 21:46:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76639 invoked by uid 99); 20 Oct 2010 21:46:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 21:46:45 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 21:46:39 +0000
Received: by qwg8 with SMTP id 8so2895069qwg.35
        for <general@hadoop.apache.org>; Wed, 20 Oct 2010 14:46:18 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.224.19.205 with SMTP id c13mr2674069qab.90.1287611178382; Wed,
 20 Oct 2010 14:46:18 -0700 (PDT)
Received: by 10.229.237.131 with HTTP; Wed, 20 Oct 2010 14:46:18 -0700 (PDT)
In-Reply-To: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com>
Date: Wed, 20 Oct 2010 14:46:18 -0700
Message-ID: <AANLkTimNHsMZcywk+_e45n8YnJTccoHYWamqdryr3h8K@mail.gmail.com>
Subject: Re: Patch testing
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Yes!  Thanks Nigel.

On Wed, Oct 20, 2010 at 12:47 PM, Nigel Daley <ndaley@mac.com> wrote:
> Folks,
>
> I'm working to get the pre-commit patch testing running again for HDFS, H=
ADOOP, and MAPREDUCE patches. =A0Since I've been on a bit of a hiatus from =
day-to-day involvement w/ the project over the last 6 months (new job), I w=
ant to check in and make sure folks would still find this pre-commit testin=
g useful. =A0Also, happy to hear any suggested improvement. =A0Let me know.
>
> Cheers,
> Nige
>

From general-return-2204-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 20 22:16:06 2010
Return-Path: <general-return-2204-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97071 invoked from network); 20 Oct 2010 22:16:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Oct 2010 22:16:06 -0000
Received: (qmail 16192 invoked by uid 500); 20 Oct 2010 22:16:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16140 invoked by uid 500); 20 Oct 2010 22:16:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16132 invoked by uid 99); 20 Oct 2010 22:16:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 22:16:04 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 20 Oct 2010 22:16:02 +0000
Received: (qmail 96906 invoked by uid 99); 20 Oct 2010 22:15:40 -0000
Received: from localhost.apache.org (HELO mail-yx0-f176.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 22:15:40 +0000
Received: by yxn35 with SMTP id 35so1440137yxn.35
        for <general@hadoop.apache.org>; Wed, 20 Oct 2010 15:15:38 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.42.1.6 with SMTP id 6mr3202945ice.100.1287612937085; Wed, 20
 Oct 2010 15:15:37 -0700 (PDT)
Received: by 10.231.148.7 with HTTP; Wed, 20 Oct 2010 15:15:37 -0700 (PDT)
In-Reply-To: <580DC943-D250-405D-A524-82452CBC4E03@mac.com>
References: <63A1C7A9-727B-4492-9998-01F6B6958EDD@apache.org>
	<4CBE11C5.301@apache.org>
	<CE8E008F-D08E-4564-80F0-B0326246B768@apache.org>
	<580DC943-D250-405D-A524-82452CBC4E03@mac.com>
Date: Wed, 20 Oct 2010 15:15:37 -0700
Message-ID: <AANLkTikR5uY0_zsT_owo-90yivuVE0PMUPFoEOx3U_8H@mail.gmail.com>
Subject: Re: [DISCUSS] Proposed bylaws for Hadoop
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Wed, Oct 20, 2010 at 10:14 AM, Nigel Daley <ndaley@mac.com> wrote:
> FWIW, "PMC does not generally operate by majority but by consensus." was given as a rationale when explaining to an existing PMC member why it was ok to approve so many new PMC members from a single company.

FWIW, working at the same company doesn't imbue contributors with a
shared agenda, neither does it rob them of their ability to
compromise, relieve their concern for the project's health, nor does
it diminish their respect for other contributors. If individuals from
that company display any of those traits, then the PMC should address
them. Equating association and collusion a priori is inadmissible. And
insulting.

On Wed, Oct 20, 2010 at 2:12 PM, Tom White <tom.e.white@gmail.com> wrote:
> I suggest we make this stronger.
> By way of comparison, the recently enacted bylaws for Pig
> (http://pig.apache.org/bylaws.html) have consensus, for example.

Consensus is equivalent to making the PMC a permanent appointment.
Discussions would be more civil and more likely to offer compromise-
we would actually work toward consensus- if intransigence and
hostility had consequences. Removing people is a last resort. Moving
the barrier closer doesn't make it more likely, but it aligns
incentives so rational people will reign in their more intemperate
comments and, instead, find ways to make progress.

Many projects require supermajorities or even 3/4 majorities. It's
sufficiently damning if more than half the PMC thinks an individual is
so destructive that less damage would be done by ejecting them, in my
opinion, but again: this is not a facility we should use, or have to
use. Throwing someone out is an expensive failure for the project, and
an conspicuous failure of its governance and community. Personally, I
don't care if it's a majority or a supermajority, but consensus is a
fig leaf.

> Also, Pig has a minimum length of time for each action, rather than
> the same fixed number for each. Might be worth considering.

IIRC, the reasoning for a full week in the last thread was to prevent
national holidays from affecting the definition of "working days." We
might expedite votes for security/patch releases. -C

From general-return-2205-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 20 22:42:01 2010
Return-Path: <general-return-2205-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5346 invoked from network); 20 Oct 2010 22:42:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Oct 2010 22:42:00 -0000
Received: (qmail 41835 invoked by uid 500); 20 Oct 2010 22:41:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 41756 invoked by uid 500); 20 Oct 2010 22:41:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 41747 invoked by uid 99); 20 Oct 2010 22:41:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 22:41:59 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mmoores@real.com designates 207.188.23.6 as permitted sender)
Received: from [207.188.23.6] (HELO jor-el.real.com) (207.188.23.6)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 22:41:53 +0000
Received: from seacas02.corp.real.com ([::ffff:192.168.139.57])
  (TLS: TLSv1/SSLv3,128bits,AES128-SHA)
  by jor-el.real.com with esmtp; Wed, 20 Oct 2010 15:41:32 -0700
  id 00094022.4CBF701C.00004FF7
Received: from seambx.corp.real.com ([fe80::2d15:fda7:b3b8:e268]) by
 seacas02.corp.real.com ([::1]) with mapi; Wed, 20 Oct 2010 15:41:32 -0700
From: Michael Moores <mmoores@real.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 20 Oct 2010 15:41:31 -0700
Subject: Limiting concurrent maps
Thread-Topic: Limiting concurrent maps
Thread-Index: Actwp/Fb/Y5FxTcKSeujJFhFELqJlg==
Message-ID: <8AE653F8-6EA6-4D50-A7B2-CA1B0CC75649@real.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

I have been playing with mapreduce.tasktracker.map.tasks.maximum to reduce =
the load
on my Cassandra cluster (using the Cassandra ColumnFamilyInputFormat).  I'd=
 like to find ways of throttling the map operations
in the case I may be affecting OLTP activity on the cluster.

What parameters can I use to limit the number of map tasks running concurre=
ntly across the whole cluster?  mapreduce.tasktracker.map.tasks.maximum=20
limits the number of concurrent maps per task tracker.  But can i do this a=
t the job level?=20

Should I look at the "fair" scheduler?

regards,Michael=

From general-return-2206-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 20 23:33:36 2010
Return-Path: <general-return-2206-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19987 invoked from network); 20 Oct 2010 23:33:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Oct 2010 23:33:35 -0000
Received: (qmail 94428 invoked by uid 500); 20 Oct 2010 23:33:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94353 invoked by uid 500); 20 Oct 2010 23:33:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94345 invoked by uid 99); 20 Oct 2010 23:33:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 23:33:34 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE
X-Spam-Check-By: apache.org
Received-SPF: unknown ip:63.236.97.88 (nike.apache.org: encountered unrecognized mechanism during SPF processing of domain of james@rockyou.com)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 23:33:26 +0000
Received: by wwb18 with SMTP id 18so3325850wwb.29
        for <general@hadoop.apache.org>; Wed, 20 Oct 2010 16:33:06 -0700 (PDT)
Received: by 10.227.137.71 with SMTP id v7mr234637wbt.149.1287617586380; Wed,
 20 Oct 2010 16:33:06 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.227.154.78 with HTTP; Wed, 20 Oct 2010 16:32:26 -0700 (PDT)
In-Reply-To: <8AE653F8-6EA6-4D50-A7B2-CA1B0CC75649@real.com>
References: <8AE653F8-6EA6-4D50-A7B2-CA1B0CC75649@real.com>
From: james warren <james@rockyou.com>
Date: Wed, 20 Oct 2010 16:32:26 -0700
Message-ID: <AANLkTin2o+gGdoRCNCA0sEXr7EpNhv5+GEo19j+tFf3Y@mail.gmail.com>
Subject: Re: Limiting concurrent maps
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e659f8dad7bf67049314d4fa
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e659f8dad7bf67049314d4fa
Content-Type: text/plain; charset=ISO-8859-1

Hi Michael,

Any of the tasktracker configs affect the local tasktracker daemon and not
other servers in your cluster.  Moreover, they can't be overridden by a job
configuration.  Sounds like you're in need of a job scheduler; I personally
prefer use the Fair Scheduler but I'm sure the Capacity Scheduler would suit
your needs as well.

cheers,
-James

On Wed, Oct 20, 2010 at 3:41 PM, Michael Moores <mmoores@real.com> wrote:

> I have been playing with mapreduce.tasktracker.map.tasks.maximum to reduce
> the load
> on my Cassandra cluster (using the Cassandra ColumnFamilyInputFormat).  I'd
> like to find ways of throttling the map operations
> in the case I may be affecting OLTP activity on the cluster.
>
> What parameters can I use to limit the number of map tasks running
> concurrently across the whole cluster?
>  mapreduce.tasktracker.map.tasks.maximum
> limits the number of concurrent maps per task tracker.  But can i do this
> at the job level?
>
> Should I look at the "fair" scheduler?
>
> regards,Michael

--0016e659f8dad7bf67049314d4fa--

From general-return-2207-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 17:11:20 2010
Return-Path: <general-return-2207-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76385 invoked from network); 21 Oct 2010 17:11:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 17:11:19 -0000
Received: (qmail 15868 invoked by uid 500); 21 Oct 2010 17:11:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15611 invoked by uid 500); 21 Oct 2010 17:11:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15594 invoked by uid 99); 21 Oct 2010 17:11:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 17:11:16 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of tom.e.white@gmail.com designates 209.85.213.176 as permitted sender)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 17:11:10 +0000
Received: by yxn35 with SMTP id 35so2091848yxn.35
        for <general@hadoop.apache.org>; Thu, 21 Oct 2010 10:10:49 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type;
        bh=RJca5fyRhoSadSmxYYTvyu3pb+MYygc0KrEPwscIXcg=;
        b=tlHHdkvPnPzuhGaBdsJig0N7zbfYJpNG1gMORyEZ2UreuMe7OduRXheEXEbVvwH/7N
         SdqgUfce+X69FOfAv237BW9iPE9TWYLLaJHIdAfqw/AFCehPA9d2b1N7gA/8k7L7DYF5
         vXpUWNgB4FZW5zzWX1jFGWjqzSMjGqPm11R4o=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=Gruc8ipOYB8hoBIf6/0uBvhmxM1QzF16NuNOfmK9P/U6gRwW3g+Ts43yiecDzGlMN+
         o6v7b85v5h4ZmT2BHgdry0QiqL+7HGQBSqkXF43DeaDxP1LAu2TXMdtOYgteD4UgZdLv
         Z1gCJmBOKvk2ns0MIAgUdV7EOSlLgc+DwsF/c=
Received: by 10.239.190.134 with SMTP id x6mr372433hbh.70.1287681048083; Thu,
 21 Oct 2010 10:10:48 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.220.70.147 with HTTP; Thu, 21 Oct 2010 10:10:27 -0700 (PDT)
In-Reply-To: <AANLkTikR5uY0_zsT_owo-90yivuVE0PMUPFoEOx3U_8H@mail.gmail.com>
References: <63A1C7A9-727B-4492-9998-01F6B6958EDD@apache.org>
 <4CBE11C5.301@apache.org> <CE8E008F-D08E-4564-80F0-B0326246B768@apache.org>
 <580DC943-D250-405D-A524-82452CBC4E03@mac.com> <AANLkTikR5uY0_zsT_owo-90yivuVE0PMUPFoEOx3U_8H@mail.gmail.com>
From: Tom White <tom.e.white@gmail.com>
Date: Thu, 21 Oct 2010 10:10:27 -0700
Message-ID: <AANLkTinE2TzQ88NBDrGUxfNbPu3Q-0k1fR-MbRT6xvd-@mail.gmail.com>
Subject: Re: [DISCUSS] Proposed bylaws for Hadoop
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Wed, Oct 20, 2010 at 3:15 PM, Chris Douglas <cdouglas@apache.org> wrote:
> On Wed, Oct 20, 2010 at 10:14 AM, Nigel Daley <ndaley@mac.com> wrote:
>> FWIW, "PMC does not generally operate by majority but by consensus." was given as a rationale when explaining to an existing PMC member why it was ok to approve so many new PMC members from a single company.
>
> FWIW, working at the same company doesn't imbue contributors with a
> shared agenda, neither does it rob them of their ability to
> compromise, relieve their concern for the project's health, nor does
> it diminish their respect for other contributors. If individuals from
> that company display any of those traits, then the PMC should address
> them. Equating association and collusion a priori is inadmissible. And
> insulting.
>
> On Wed, Oct 20, 2010 at 2:12 PM, Tom White <tom.e.white@gmail.com> wrote:
>> I suggest we make this stronger.
>> By way of comparison, the recently enacted bylaws for Pig
>> (http://pig.apache.org/bylaws.html) have consensus, for example.
>
> Consensus is equivalent to making the PMC a permanent appointment.
> Discussions would be more civil and more likely to offer compromise-
> we would actually work toward consensus- if intransigence and
> hostility had consequences. Removing people is a last resort. Moving
> the barrier closer doesn't make it more likely, but it aligns
> incentives so rational people will reign in their more intemperate
> comments and, instead, find ways to make progress.
>
> Many projects require supermajorities or even 3/4 majorities. It's
> sufficiently damning if more than half the PMC thinks an individual is
> so destructive that less damage would be done by ejecting them, in my
> opinion, but again: this is not a facility we should use, or have to
> use. Throwing someone out is an expensive failure for the project, and
> an conspicuous failure of its governance and community. Personally, I
> don't care if it's a majority or a supermajority, but consensus is a
> fig leaf.

Fair point. For a large PMC, that's probably true. This matter was
discussed on the Pig list, and there was a range of opinion there:
http://search-hadoop.com/m/jk35b2tZCuy&subj=RE+DISCUSS+Apache+Pig+bylaws.
Personally, I think a supermajority strikes the right balance.

Tom

>
>> Also, Pig has a minimum length of time for each action, rather than
>> the same fixed number for each. Might be worth considering.
>
> IIRC, the reasoning for a full week in the last thread was to prevent
> national holidays from affecting the definition of "working days." We
> might expedite votes for security/patch releases. -C
>

From general-return-2208-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 19:13:47 2010
Return-Path: <general-return-2208-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 36875 invoked from network); 21 Oct 2010 19:13:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 19:13:46 -0000
Received: (qmail 27358 invoked by uid 500); 21 Oct 2010 19:13:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27304 invoked by uid 500); 21 Oct 2010 19:13:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27296 invoked by uid 99); 21 Oct 2010 19:13:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 19:13:44 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 19:13:37 +0000
Received: by wwb34 with SMTP id 34so70852wwb.29
        for <general@hadoop.apache.org>; Thu, 21 Oct 2010 12:13:16 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.227.153.14 with SMTP id i14mr1600180wbw.142.1287688396049;
 Thu, 21 Oct 2010 12:13:16 -0700 (PDT)
Received: by 10.216.87.78 with HTTP; Thu, 21 Oct 2010 12:13:16 -0700 (PDT)
Date: Thu, 21 Oct 2010 15:13:16 -0400
Message-ID: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>
Subject: bringing the codebases back in line
From: Ian Holsman <hadoop@holsman.net>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016363b9f826d96b104932551ae
X-Virus-Checked: Checked by ClamAV on apache.org

--0016363b9f826d96b104932551ae
Content-Type: text/plain; charset=ISO-8859-1

Hi guys.

I wanted to start a conversation about how we could merge the the cloudera +
yahoo distribtutions of hadoop into our codebase,
and what would be required.

--0016363b9f826d96b104932551ae--

From general-return-2209-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 19:30:53 2010
Return-Path: <general-return-2209-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43196 invoked from network); 21 Oct 2010 19:30:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 19:30:53 -0000
Received: (qmail 60807 invoked by uid 500); 21 Oct 2010 19:30:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60757 invoked by uid 500); 21 Oct 2010 19:30:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60749 invoked by uid 99); 21 Oct 2010 19:30:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 19:30:51 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 19:30:45 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=j/7GijcjSs4UEb+KXZQfzTJQ4poURGA31+3/sejZCRgl/1nZyZCgHcLd
   ePfDHst2BdWrg63qGz15shsjQvrikf6S99qalAx1xyh2RZ3aLlO9t5SOU
   0umB79NZ2e4fgcf;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1287689444; x=1319225444;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20bringing=20the=20codebases=20back=20in
   =20line|Date:=20Thu,=2021=20Oct=202010=2019:30:23=20+0000
   |Message-ID:=20<D9F0255E-7517-4CDB-9CCD-912B842F9E7C@link
   edin.com>|To:=20"<general@hadoop.apache.org>"=20<general@
   hadoop.apache.org>|MIME-Version:=201.0
   |Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<0B850FE2A152BF429ED5FABC8EFFBB2C@linkedin
   .com>|In-Reply-To:=20<AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYE
   ThpsCSTeYT@mail.gmail.com>|References:=20<AANLkTik_fT+NSC
   D1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>;
  bh=oki8y0fjwZT9ZHMa8cfW/BowFyRd4lEWGd0QXL/NaDM=;
  b=EXLMEQ/d2uogfPVaSRXoXamM+fbU8rUmFQDLCxJOOxkCX0dbCUIgJQhF
   8CFGRkpqbGLD3AvCBD4BTKW/nSX7n9VtsX4Q33/xXuQX+FzFiwVKCir5I
   9HVzvyNeVDE6N5q;
X-IronPort-AV: E=Sophos;i="4.58,219,1286175600"; 
   d="scan'208";a="15855466"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi id 14.01.0218.012; Thu,
 21 Oct 2010 12:30:24 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: bringing the codebases back in line
Thread-Topic: bringing the codebases back in line
Thread-Index: AQHLcVQVuqCpAAO/eUm8zw+IHmCpg5NMP34A
Date: Thu, 21 Oct 2010 19:30:23 +0000
Message-ID: <D9F0255E-7517-4CDB-9CCD-912B842F9E7C@linkedin.com>
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>
In-Reply-To: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <0B850FE2A152BF429ED5FABC8EFFBB2C@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


On Oct 21, 2010, at 12:13 PM, Ian Holsman wrote:

> Hi guys.
>=20
> I wanted to start a conversation about how we could merge the the clouder=
a +
> yahoo distribtutions of hadoop into our codebase,
> and what would be required.


*grabs popcorn*


From general-return-2210-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 20:00:44 2010
Return-Path: <general-return-2210-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59329 invoked from network); 21 Oct 2010 20:00:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 20:00:44 -0000
Received: (qmail 30262 invoked by uid 500); 21 Oct 2010 20:00:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30036 invoked by uid 500); 21 Oct 2010 20:00:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30028 invoked by uid 99); 21 Oct 2010 20:00:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 20:00:42 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.30.17] (HELO qmta10.emeryville.ca.mail.comcast.net) (76.96.30.17)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 20:00:33 +0000
Received: from omta05.emeryville.ca.mail.comcast.net ([76.96.30.43])
	by qmta10.emeryville.ca.mail.comcast.net with comcast
	id MVAB1f00A0vp7WLAAY0BiY; Thu, 21 Oct 2010 20:00:11 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta05.emeryville.ca.mail.comcast.net with comcast
	id MY011f00B2SbwD58RY04gA; Thu, 21 Oct 2010 20:00:09 +0000
Message-Id: <A2ED1DE3-B131-4D83-AE0A-382F6A64C06F@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: bringing the codebases back in line
Date: Thu, 21 Oct 2010 13:00:01 -0700
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


On Oct 21, 2010, at 12:13 PM, Ian Holsman wrote:

> I wanted to start a conversation about how we could merge the the  
> cloudera +
> yahoo distribtutions of hadoop into our codebase,
> and what would be required.

All of the patches that are the "yahoo distribution of hadoop" have  
been in Apache's trunk for months.

-- Owen

From general-return-2211-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 21:01:01 2010
Return-Path: <general-return-2211-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91441 invoked from network); 21 Oct 2010 21:01:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 21:01:01 -0000
Received: (qmail 46001 invoked by uid 500); 21 Oct 2010 21:00:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45784 invoked by uid 500); 21 Oct 2010 21:00:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45776 invoked by uid 99); 21 Oct 2010 21:00:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 21:00:59 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 21:00:53 +0000
Received: by wyg36 with SMTP id 36so47639wyg.35
        for <general@hadoop.apache.org>; Thu, 21 Oct 2010 14:00:31 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.227.156.210 with SMTP id y18mr1797234wbw.63.1287694831469;
 Thu, 21 Oct 2010 14:00:31 -0700 (PDT)
Received: by 10.216.87.78 with HTTP; Thu, 21 Oct 2010 14:00:31 -0700 (PDT)
In-Reply-To: <A2ED1DE3-B131-4D83-AE0A-382F6A64C06F@apache.org>
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>
	<A2ED1DE3-B131-4D83-AE0A-382F6A64C06F@apache.org>
Date: Thu, 21 Oct 2010 17:00:31 -0400
Message-ID: <AANLkTikFo-XS3c4T+6S1Sp25dE0Aq+ZZNX_T1C57E7ca@mail.gmail.com>
Subject: Re: bringing the codebases back in line
From: Ian Holsman <hadoop@holsman.net>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e65a01f602661d049326d16d

--0016e65a01f602661d049326d16d
Content-Type: text/plain; charset=ISO-8859-1

so what do you think is required to get them into a release?

On Thu, Oct 21, 2010 at 4:00 PM, Owen O'Malley <omalley@apache.org> wrote:

>
> On Oct 21, 2010, at 12:13 PM, Ian Holsman wrote:
>
>  I wanted to start a conversation about how we could merge the the cloudera
>> +
>> yahoo distribtutions of hadoop into our codebase,
>> and what would be required.
>>
>
> All of the patches that are the "yahoo distribution of hadoop" have been in
> Apache's trunk for months.
>
> -- Owen
>

--0016e65a01f602661d049326d16d--

From general-return-2212-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 21:42:07 2010
Return-Path: <general-return-2212-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 20592 invoked from network); 21 Oct 2010 21:42:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 21:42:06 -0000
Received: (qmail 13582 invoked by uid 500); 21 Oct 2010 21:42:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13536 invoked by uid 500); 21 Oct 2010 21:42:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13528 invoked by uid 99); 21 Oct 2010 21:42:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 21:42:05 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.27.211] (HELO qmta11.emeryville.ca.mail.comcast.net) (76.96.27.211)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 21:41:57 +0000
Received: from omta02.emeryville.ca.mail.comcast.net ([76.96.30.19])
	by qmta11.emeryville.ca.mail.comcast.net with comcast
	id MQNh1f0090QkzPwABZhcKM; Thu, 21 Oct 2010 21:41:36 +0000
Received: from [172.21.154.225] ([209.131.62.146])
	by omta02.emeryville.ca.mail.comcast.net with comcast
	id MZhT1f00F39K7Mt8NZhWYw; Thu, 21 Oct 2010 21:41:34 +0000
Message-Id: <5A40C123-83C3-4F64-91E8-2C93FB6C053C@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <AANLkTikFo-XS3c4T+6S1Sp25dE0Aq+ZZNX_T1C57E7ca@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: bringing the codebases back in line
Date: Thu, 21 Oct 2010 14:41:26 -0700
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com> <A2ED1DE3-B131-4D83-AE0A-382F6A64C06F@apache.org> <AANLkTikFo-XS3c4T+6S1Sp25dE0Aq+ZZNX_T1C57E7ca@mail.gmail.com>
X-Mailer: Apple Mail (2.936)


On Oct 21, 2010, at 2:00 PM, Ian Holsman wrote:

> so what do you think is required to get them into a release?

I'd planned to start making a release next month.

-- Owen

From general-return-2213-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 21:54:14 2010
Return-Path: <general-return-2213-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30189 invoked from network); 21 Oct 2010 21:54:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 21:54:13 -0000
Received: (qmail 34074 invoked by uid 500); 21 Oct 2010 21:54:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34011 invoked by uid 500); 21 Oct 2010 21:54:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34003 invoked by uid 99); 21 Oct 2010 21:54:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 21:54:12 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 21:54:05 +0000
Received: by wyg36 with SMTP id 36so97803wyg.35
        for <general@hadoop.apache.org>; Thu, 21 Oct 2010 14:53:45 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.50.11 with SMTP id y11mr1612067web.57.1287698025465; Thu,
 21 Oct 2010 14:53:45 -0700 (PDT)
Received: by 10.216.87.78 with HTTP; Thu, 21 Oct 2010 14:53:45 -0700 (PDT)
In-Reply-To: <D9F0255E-7517-4CDB-9CCD-912B842F9E7C@linkedin.com>
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>
	<D9F0255E-7517-4CDB-9CCD-912B842F9E7C@linkedin.com>
Date: Thu, 21 Oct 2010 17:53:45 -0400
Message-ID: <AANLkTimkd_t2k168JpejruPHVDRSBhE1XzPsEtV9rq3M@mail.gmail.com>
Subject: Re: bringing the codebases back in line
From: Ian Holsman <hadoop@holsman.net>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6dbe80262c57e0493278f1d
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6dbe80262c57e0493278f1d
Content-Type: text/plain; charset=ISO-8859-1

yep.. I've heard it's a source of contention...

but I'd like to see how we can get it so the amount of patches that the
large companies apply on top of the current production apache release gets
minimized, and the large installations are all running nearly identical code
on their clusters, and that we wouldn't need to have a yahoo or cloudera
repo of their patch sets made available.

So Ideally I'd like to hear what kind of things apache needs to do help get
these kind of things less divergent.

In discussing it with people, I've heard that a major issue (not the only
one i'm sure) is lack of resources to actually test the apache releases on
large clusters, and that it is very hard getting this done in short cycles
(hence the large gap between 20.x and 21).

So I thought I would start the thread to see if we could at least identify
what the people think are the problems are.


On Thu, Oct 21, 2010 at 3:30 PM, Allen Wittenauer
<awittenauer@linkedin.com>wrote:

>
> On Oct 21, 2010, at 12:13 PM, Ian Holsman wrote:
>
> > Hi guys.
> >
> > I wanted to start a conversation about how we could merge the the
> cloudera +
> > yahoo distribtutions of hadoop into our codebase,
> > and what would be required.
>
>
> *grabs popcorn*
>
>

--0016e6dbe80262c57e0493278f1d--

From general-return-2214-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 21:54:57 2010
Return-Path: <general-return-2214-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30311 invoked from network); 21 Oct 2010 21:54:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 21:54:56 -0000
Received: (qmail 35457 invoked by uid 500); 21 Oct 2010 21:54:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 35355 invoked by uid 500); 21 Oct 2010 21:54:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 35347 invoked by uid 99); 21 Oct 2010 21:54:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 21:54:55 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 21:54:49 +0000
Received: by qwc9 with SMTP id 9so84972qwc.35
        for <general@hadoop.apache.org>; Thu, 21 Oct 2010 14:54:26 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.241.12 with SMTP id lc12mr1296738qcb.178.1287698065939;
 Thu, 21 Oct 2010 14:54:25 -0700 (PDT)
Received: by 10.229.237.131 with HTTP; Thu, 21 Oct 2010 14:54:25 -0700 (PDT)
In-Reply-To: <A2ED1DE3-B131-4D83-AE0A-382F6A64C06F@apache.org>
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>
	<A2ED1DE3-B131-4D83-AE0A-382F6A64C06F@apache.org>
Date: Thu, 21 Oct 2010 14:54:25 -0700
Message-ID: <AANLkTimnqhcG_nazMjSjwvXnvw_udDXJkFq3VbNwXZda@mail.gmail.com>
Subject: Re: bringing the codebases back in line
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Thu, Oct 21, 2010 at 1:00 PM, Owen O'Malley <omalley@apache.org> wrote:
>
> On Oct 21, 2010, at 12:13 PM, Ian Holsman wrote:
>
>> I wanted to start a conversation about how we could merge the the cloudera
>> +
>> yahoo distribtutions of hadoop into our codebase,
>> and what would be required.
>
> All of the patches that are the "yahoo distribution of hadoop" have been in
> Apache's trunk for months.

It's worth double checking.  When we added the YDH patch set to CDH3
we ran a script to see which patches were in YDH but not yet in trunk
and it turned up around 100 or so patches.  A fair number of those may
have been included in trunk but under a different jira, however some
(eg MR-1088, MR-1100) are definitely not in trunk.  Also, if I
remember correctly some of the 20-based patches are substantially
different than the versions for trunk.

Thanks,
Eli

From general-return-2215-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 22:20:10 2010
Return-Path: <general-return-2215-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 48840 invoked from network); 21 Oct 2010 22:20:10 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 22:20:10 -0000
Received: (qmail 69211 invoked by uid 500); 21 Oct 2010 22:20:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69150 invoked by uid 500); 21 Oct 2010 22:20:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69142 invoked by uid 99); 21 Oct 2010 22:20:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 22:20:09 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 21 Oct 2010 22:20:07 +0000
Received: (qmail 48781 invoked by uid 99); 21 Oct 2010 22:19:45 -0000
Received: from localhost.apache.org (HELO [192.168.168.130]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 22:19:45 +0000
Message-ID: <4CC0BC80.6050303@apache.org>
Date: Thu, 21 Oct 2010 15:19:44 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.14) Gecko/20101006 Thunderbird/3.0.9
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: bringing the codebases back in line
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>
In-Reply-To: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 10/21/2010 12:13 PM, Ian Holsman wrote:
> I wanted to start a conversation about how we could merge the the cloudera +
> yahoo distribtutions of hadoop into our codebase,
> and what would be required.

Cloudera's distribution is based on Y!'s 0.20 distribution, together 
with patches from the Apache 0.20-append branch, patches cherry-picked 
from trunk and patches that have been submitted but not yet committed.

Please note that both Cloudera and Y!'s distributions are 0.20-based 
distributions.  Patches may have been committed to trunk, but nearly 
everyone is using 0.20 in production and will be for some time.  There 
is currently no 0.20 branch at Apache that contains what most folks are 
using in production.  There have been proposals to create such branches, 
most recently by Arun.  Do you think this is worthwhile?

Doug

From general-return-2216-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 22:20:28 2010
Return-Path: <general-return-2216-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 48949 invoked from network); 21 Oct 2010 22:20:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 22:20:28 -0000
Received: (qmail 70503 invoked by uid 500); 21 Oct 2010 22:20:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70459 invoked by uid 500); 21 Oct 2010 22:20:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70451 invoked by uid 99); 21 Oct 2010 22:20:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 22:20:26 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.30.96] (HELO qmta09.emeryville.ca.mail.comcast.net) (76.96.30.96)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 22:20:17 +0000
Received: from omta15.emeryville.ca.mail.comcast.net ([76.96.30.71])
	by qmta09.emeryville.ca.mail.comcast.net with comcast
	id MXea1f0021Y3wxoA9aKvrf; Thu, 21 Oct 2010 22:19:55 +0000
Received: from wlan-snve-154-225.corp.yahoo.com ([209.131.62.115])
	by omta15.emeryville.ca.mail.comcast.net with comcast
	id MaKm1f00C2VBGtd8baKpbo; Thu, 21 Oct 2010 22:19:53 +0000
Message-Id: <03C46405-ECE7-44CE-925D-12E1989D4DD0@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <AANLkTimnqhcG_nazMjSjwvXnvw_udDXJkFq3VbNwXZda@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: bringing the codebases back in line
Date: Thu, 21 Oct 2010 15:19:45 -0700
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com> <A2ED1DE3-B131-4D83-AE0A-382F6A64C06F@apache.org> <AANLkTimnqhcG_nazMjSjwvXnvw_udDXJkFq3VbNwXZda@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

On Oct 21, 2010, at 2:54 PM, Eli Collins wrote:

> It's worth double checking.  When we added the YDH patch set to CDH3
> we ran a script to see which patches were in YDH but not yet in trunk
> and it turned up around 100 or so patches.

If you could generate a list, that would be useful for tracking down  
the differences. Certainly, everyone was supposed to have gotten  
everything checked in. I see that I managed to forget h-6832. I just  
uploaded the patch and marked it patch available.

-- Owen

From general-return-2217-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 22:32:14 2010
Return-Path: <general-return-2217-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54560 invoked from network); 21 Oct 2010 22:32:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 22:32:14 -0000
Received: (qmail 85233 invoked by uid 500); 21 Oct 2010 22:32:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85182 invoked by uid 500); 21 Oct 2010 22:32:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85174 invoked by uid 99); 21 Oct 2010 22:32:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 22:32:12 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 22:32:08 +0000
Received: from [192.168.0.121] (snvvpn1-10-72-244-c102.hq.corp.yahoo.com [10.72.244.102])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id o9LMU5h9034829
	for <general@hadoop.apache.org>; Thu, 21 Oct 2010 15:30:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1287700206;
	bh=kkhrrUP9TdLZdVoj1LRtytMCCfabcBau0N3e20r4Ytk=;
	h=Message-ID:Date:From:MIME-Version:To:Subject:References:
	 In-Reply-To:Content-Type:Content-Transfer-Encoding;
	b=iJ5um9qUfk0fV/H/BEeC8Xex0Q4bsLbmlO8ERfN5OJHWywN9EA0mK85m2BvuQyMkK
	 d+kSMEBW/8TAs5bBaHbqBliJr6ZiubLlPRj6Ln7vmtbf8weAHEkUq6hbUCyWKCxDpP
	 9fqFIS+Oz1DJAdwESfCPXCasSeqNBNlcP763WVQQ=
Message-ID: <4CC0BEED.2020200@yahoo-inc.com>
Date: Thu, 21 Oct 2010 15:30:05 -0700
From: Jakob Homan <jhoman@yahoo-inc.com>
User-Agent: Thunderbird 2.0.0.12 (Macintosh/20080213)
MIME-Version: 1.0
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: bringing the codebases back in line
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>	<A2ED1DE3-B131-4D83-AE0A-382F6A64C06F@apache.org> <AANLkTimnqhcG_nazMjSjwvXnvw_udDXJkFq3VbNwXZda@mail.gmail.com>
In-Reply-To: <AANLkTimnqhcG_nazMjSjwvXnvw_udDXJkFq3VbNwXZda@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

 > It's worth double checking.  When we added the YDH patch set to CDH3
 > we ran a script to see which patches were in YDH but not yet in trunk
 > and it turned up around 100 or so patches.

If the patch was just checking 1:1 Jira to patch, it would certainly not 
work.  We were uploading multiple patches to the same JIRA to avoid 
opening extraneous issues before generating patches for trunk. 
Venerable old HDFS-1150, for instance, went through about different 
patches applied to Y!'s branch before the final version.
-jg


Eli Collins wrote:
> On Thu, Oct 21, 2010 at 1:00 PM, Owen O'Malley <omalley@apache.org> wrote:
>> On Oct 21, 2010, at 12:13 PM, Ian Holsman wrote:
>>
>>> I wanted to start a conversation about how we could merge the the cloudera
>>> +
>>> yahoo distribtutions of hadoop into our codebase,
>>> and what would be required.
>> All of the patches that are the "yahoo distribution of hadoop" have been in
>> Apache's trunk for months.
> 
  fair number of those may
> have been included in trunk but under a different jira, however some
> (eg MR-1088, MR-1100) are definitely not in trunk.  Also, if I
> remember correctly some of the 20-based patches are substantially
> different than the versions for trunk.
> 
> Thanks,
> Eli


From general-return-2218-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 22:43:06 2010
Return-Path: <general-return-2218-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58973 invoked from network); 21 Oct 2010 22:43:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 22:43:05 -0000
Received: (qmail 97356 invoked by uid 500); 21 Oct 2010 22:43:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97283 invoked by uid 500); 21 Oct 2010 22:43:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97275 invoked by uid 99); 21 Oct 2010 22:43:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 22:43:04 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.62.56] (HELO qmta06.westchester.pa.mail.comcast.net) (76.96.62.56)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 22:42:56 +0000
Received: from omta17.westchester.pa.mail.comcast.net ([76.96.62.89])
	by qmta06.westchester.pa.mail.comcast.net with comcast
	id MXaa1f0011vXlb856aicmp; Thu, 21 Oct 2010 22:42:36 +0000
Received: from wlan-snve-154-225.corp.yahoo.com ([209.131.62.115])
	by omta17.westchester.pa.mail.comcast.net with comcast
	id MaiS1f00M2VBGtd3daiVRv; Thu, 21 Oct 2010 22:42:34 +0000
Message-Id: <211EF221-CFC5-42A5-99AD-339F120333D1@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <4CC0BC80.6050303@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: bringing the codebases back in line
Date: Thu, 21 Oct 2010 15:42:24 -0700
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com> <4CC0BC80.6050303@apache.org>
X-Mailer: Apple Mail (2.936)


On Oct 21, 2010, at 3:19 PM, Doug Cutting wrote:

> Cloudera's distribution is based on Y!'s 0.20 distribution, together  
> with patches from the Apache 0.20-append branch,

Cloudera's Distribution of Hadoop includes many tools from outside of  
Hadoop and even outside of Apache.

-- Owen


From general-return-2219-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 23:38:11 2010
Return-Path: <general-return-2219-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85222 invoked from network); 21 Oct 2010 23:38:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 23:38:11 -0000
Received: (qmail 43557 invoked by uid 500); 21 Oct 2010 23:38:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 43503 invoked by uid 500); 21 Oct 2010 23:38:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 43495 invoked by uid 99); 21 Oct 2010 23:38:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 23:38:09 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 23:38:04 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=orakKMd8SJugbec1iW7+uRsIJDUHKpiqfb3fudVCsgNLF8QVCzPGaBke
   cKPIV4bfWuUyIJkuT2KaU4jLb1VxEnGUV9zuLDSTOFFYy/RAJ7A8p1lXT
   VjfIKJ20Nj9L6EX;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1287704284; x=1319240284;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20bringing=20the=20codebases=20back=20in
   =20line|Date:=20Thu,=2021=20Oct=202010=2023:37:44=20+0000
   |Message-ID:=20<FDE1E4F4-D765-45F8-B529-47166881AB95@link
   edin.com>|To:=20"<general@hadoop.apache.org>"=20<general@
   hadoop.apache.org>|MIME-Version:=201.0
   |Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<1DF7DBA6CCA12744BA525EB728175D43@linkedin
   .com>|In-Reply-To:=20<AANLkTimkd_t2k168JpejruPHVDRSBhE1Xz
   PsEtV9rq3M@mail.gmail.com>|References:=20<AANLkTik_fT+NSC
   D1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>=0D=0A=20<D
   9F0255E-7517-4CDB-9CCD-912B842F9E7C@linkedin.com>=0D=0A
   =20<AANLkTimkd_t2k168JpejruPHVDRSBhE1XzPsEtV9rq3M@mail.gm
   ail.com>;
  bh=QVrBuptr9bKfWFd1Z1WFUcFGiSBGIQXPhgQ/pzK3UfY=;
  b=rp4QFevb/wExVJvD4N2Q7XESxPOwLLBLrSy5H/LBMqUE37zwZxhB7gc1
   pihCVzjE3s/1SzwhywWIP6rZsPicqpTnUQ9/ghk/NnKYZnyOttgsX312F
   +SqNvmtN5zKLFzJ;
X-IronPort-AV: E=Sophos;i="4.58,220,1286175600"; 
   d="scan'208";a="16281347"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi id 14.01.0218.012; Thu,
 21 Oct 2010 16:37:44 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: bringing the codebases back in line
Thread-Topic: bringing the codebases back in line
Thread-Index: AQHLcVQVuqCpAAO/eUm8zw+IHmCpg5NMP34AgAAoEICAAB0MgA==
Date: Thu, 21 Oct 2010 23:37:44 +0000
Message-ID: <FDE1E4F4-D765-45F8-B529-47166881AB95@linkedin.com>
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>
 <D9F0255E-7517-4CDB-9CCD-912B842F9E7C@linkedin.com>
 <AANLkTimkd_t2k168JpejruPHVDRSBhE1XzPsEtV9rq3M@mail.gmail.com>
In-Reply-To: <AANLkTimkd_t2k168JpejruPHVDRSBhE1XzPsEtV9rq3M@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <1DF7DBA6CCA12744BA525EB728175D43@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


On Oct 21, 2010, at 2:53 PM, Ian Holsman wrote:

> yep.. I've heard it's a source of contention...

	Sure. Maybe like 8 months ago to anyone who was paying attention.

> In discussing it with people, I've heard that a major issue (not the only
> one i'm sure) is lack of resources to actually test the apache releases o=
n
> large clusters, and
>=20
> So I thought I would start the thread to see if we could at least identif=
y
> what the people think are the problems are.

	I think you've asked the wrong question to begin with and I think the disc=
ussion you've started is going to be completely counter productive and put =
everyone on the defensive.  [It is pretty clear it already has.]   The team=
 is finally making progress towards a common goal (0.22) and (mostly) getti=
ng along.=20

	This question/issue

>  that it is very hard getting this done in short cycles
> (hence the large gap between 20.x and 21).

	is really the cruxt of the problem.

	We wouldn't have had major diversions between the distributions if the mai=
nline Apache distribution had been making more frequent releases.   IMNSHO,=
 as long as the distributions don't break compatibility (forward or backwar=
d) in without a word of warning... who cares?


From general-return-2220-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 23:51:02 2010
Return-Path: <general-return-2220-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 89533 invoked from network); 21 Oct 2010 23:51:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 23:51:02 -0000
Received: (qmail 54578 invoked by uid 500); 21 Oct 2010 23:51:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54487 invoked by uid 500); 21 Oct 2010 23:51:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54479 invoked by uid 99); 21 Oct 2010 23:51:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 23:51:01 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 23:50:55 +0000
Received: by wyg36 with SMTP id 36so194386wyg.35
        for <general@hadoop.apache.org>; Thu, 21 Oct 2010 16:50:33 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.142.131 with SMTP id i3mr10445829wej.5.1287705032816; Thu,
 21 Oct 2010 16:50:32 -0700 (PDT)
Received: by 10.216.87.78 with HTTP; Thu, 21 Oct 2010 16:50:32 -0700 (PDT)
In-Reply-To: <211EF221-CFC5-42A5-99AD-339F120333D1@apache.org>
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>
	<4CC0BC80.6050303@apache.org>
	<211EF221-CFC5-42A5-99AD-339F120333D1@apache.org>
Date: Thu, 21 Oct 2010 19:50:32 -0400
Message-ID: <AANLkTim1ygQZEMhCpGRG0u86kDf0xRr8QWAuo=8TtLqa@mail.gmail.com>
Subject: Re: bringing the codebases back in line
From: Ian Holsman <hadoop@holsman.net>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d7e9300e83bc0493293171

--0016e6d7e9300e83bc0493293171
Content-Type: text/plain; charset=ISO-8859-1

right.. Cloudera is bundling it's add-ons into a single tarball to make it
easier to install.

but my main bone of contention here is not in the bundling, but that in
order for those tools to work, they need to make changes to the base hadoop
package.

In my ideal world, I'd like to be able to just download/buy any of those
tools and have them run on a released apache hadoop tarball. and then if
someone else comes along with a competing tool I would be free to choose it
and have it also run on my apache hadoop tarball, not have to go through the
pain of saying XXX tool needs their customized version of hadoop so I can't
use it. (ie remove the lock-in that comes from a forked base).

but the other question I have which hopefully you guys can answer is does
the yahoo distribution have ALL the patches from the trunk on it? because if
it doesn't I think that is problematic as well for other reasons.

so what I'd like to see is both cloudera and yahoo running a minimal set of
patches as a 'superset' of the apache hadoop stuff, with the apache hadoop
very close to both of these. the only patches being in either being to fix
bugs or performance issues that would be available in the next release of
a-hadoop.
And when a new release of a-hadoop comes, it the vendors would switch to
using that a-hadoop version as their baseline.

I don't want to get into the situation that linux is in with redhat in that
their kernel is dramatically different to the one on kernel.org.

does that make sense?




On Thu, Oct 21, 2010 at 6:42 PM, Owen O'Malley <omalley@apache.org> wrote:

>
> On Oct 21, 2010, at 3:19 PM, Doug Cutting wrote:
>
>  Cloudera's distribution is based on Y!'s 0.20 distribution, together with
>> patches from the Apache 0.20-append branch,
>>
>
> Cloudera's Distribution of Hadoop includes many tools from outside of
> Hadoop and even outside of Apache.
>
> -- Owen
>
>

--0016e6d7e9300e83bc0493293171--

From general-return-2221-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 00:11:33 2010
Return-Path: <general-return-2221-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98605 invoked from network); 22 Oct 2010 00:11:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 00:11:33 -0000
Received: (qmail 65895 invoked by uid 500); 22 Oct 2010 00:11:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65807 invoked by uid 500); 22 Oct 2010 00:11:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65799 invoked by uid 99); 22 Oct 2010 00:11:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 00:11:31 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.161.176] (HELO mail-gx0-f176.google.com) (209.85.161.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 00:11:26 +0000
Received: by gxk24 with SMTP id 24so148059gxk.35
        for <general@hadoop.apache.org>; Thu, 21 Oct 2010 17:11:04 -0700 (PDT)
Received: by 10.42.153.138 with SMTP id m10mr1343950icw.261.1287706264374;
        Thu, 21 Oct 2010 17:11:04 -0700 (PDT)
Received: from tp ([38.104.141.102])
        by mx.google.com with ESMTPS id j7sm871276vcr.39.2010.10.21.17.11.02
        (version=SSLv3 cipher=RC4-MD5);
        Thu, 21 Oct 2010 17:11:03 -0700 (PDT)
Sender: Konstantin Boudnik <cos@boudnik.org>
Date: Thu, 21 Oct 2010 17:10:57 -0700
From: Konstantin Boudnik <cos@apache.org>
To: general@hadoop.apache.org
Subject: Re: bringing the codebases back in line
Message-ID: <20101022001057.GA2710@tp>
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>
 <D9F0255E-7517-4CDB-9CCD-912B842F9E7C@linkedin.com>
 <AANLkTimkd_t2k168JpejruPHVDRSBhE1XzPsEtV9rq3M@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="sm4nu43k4a2Rpi4c"
Content-Disposition: inline
In-Reply-To: <AANLkTimkd_t2k168JpejruPHVDRSBhE1XzPsEtV9rq3M@mail.gmail.com>
X-Organization: It's something of 'Cos
X-PGP-Key: http://www.boudnik.org/~cos/pubkey.asc
User-Agent: Mutt/1.5.20 (2009-06-14)

--sm4nu43k4a2Rpi4c
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Thu, Oct 21, 2010 at 05:53PM, Ian Holsman wrote:
> In discussing it with people, I've heard that a major issue (not the only
> one i'm sure) is lack of resources to actually test the apache releases on
> large clusters, and that it is very hard getting this done in short cycles
> (hence the large gap between 20.x and 21).

I do agree the lack of resources for testing Hadoop is a problem. However,
there might be some slight difference in the meaning of word 'resources' ;)

The only way, IMO, to have a reasonable testing done on a system as complex=
 as
Hadoop is to invest into automatic validation of builds at system level. Th=
is
requires a few things (resources, if you will):
  - extra hardware (the easiest and cheapest problem)
  - automatic deployment, testing, and analysis
  - system tests development which able to control and observe a cluster
    behavior (in other words something more sophisticated than just shell
    scripts)

And for the semi-adequate system testing you don't need a large cluster: 10=
-20
nodes will be sufficient in most cases. But the automation of all the
processes starting from deployment is the key. Testing automation is in a
little better shape for Hadoop has that system test framework called Herriot
(part of Hadoop code base for about 7 months now), but it still needs furth=
er
extending.

Hopefully this briefs you about the cluster testing side of the issue.
  Cos

> So I thought I would start the thread to see if we could at least identify
> what the people think are the problems are.
>=20
>=20
> On Thu, Oct 21, 2010 at 3:30 PM, Allen Wittenauer
> <awittenauer@linkedin.com>wrote:
>=20
> >
> > On Oct 21, 2010, at 12:13 PM, Ian Holsman wrote:
> >
> > > Hi guys.
> > >
> > > I wanted to start a conversation about how we could merge the the
> > cloudera +
> > > yahoo distribtutions of hadoop into our codebase,
> > > and what would be required.
> >
> >
> > *grabs popcorn*
> >
> >

--sm4nu43k4a2Rpi4c
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEARECAAYFAkzA1pEACgkQIg9pgB8n5iKaXgCfbiEmKXnr4hBd0DZrOSFEWGtw
GLQAoJCIDjLCSEjX4uHJDRgCLswlG/JX
=LWKM
-----END PGP SIGNATURE-----

--sm4nu43k4a2Rpi4c--

From general-return-2222-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 00:18:22 2010
Return-Path: <general-return-2222-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 730 invoked from network); 22 Oct 2010 00:18:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 00:18:22 -0000
Received: (qmail 69978 invoked by uid 500); 22 Oct 2010 00:18:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69935 invoked by uid 500); 22 Oct 2010 00:18:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69927 invoked by uid 99); 22 Oct 2010 00:18:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 00:18:20 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 00:18:15 +0000
Received: by qwc9 with SMTP id 9so153074qwc.35
        for <general@hadoop.apache.org>; Thu, 21 Oct 2010 17:17:54 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.235.146 with SMTP id kg18mr1411959qcb.205.1287706674670;
 Thu, 21 Oct 2010 17:17:54 -0700 (PDT)
Received: by 10.229.237.131 with HTTP; Thu, 21 Oct 2010 17:17:54 -0700 (PDT)
In-Reply-To: <4CC0BEED.2020200@yahoo-inc.com>
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>
	<A2ED1DE3-B131-4D83-AE0A-382F6A64C06F@apache.org>
	<AANLkTimnqhcG_nazMjSjwvXnvw_udDXJkFq3VbNwXZda@mail.gmail.com>
	<4CC0BEED.2020200@yahoo-inc.com>
Date: Thu, 21 Oct 2010 17:17:54 -0700
Message-ID: <AANLkTi=WF85J47-JcRghD7AzeaA7qZmScGVQt2fn2NHV@mail.gmail.com>
Subject: Re: bringing the codebases back in line
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Thu, Oct 21, 2010 at 3:30 PM, Jakob Homan <jhoman@yahoo-inc.com> wrote:
>> It's worth double checking. =A0When we added the YDH patch set to CDH3
>> we ran a script to see which patches were in YDH but not yet in trunk
>> and it turned up around 100 or so patches.
>
> If the patch was just checking 1:1 Jira to patch, it would certainly not
> work. =A0We were uploading multiple patches to the same JIRA to avoid ope=
ning
> extraneous issues before generating patches for trunk. Venerable old
> HDFS-1150, for instance, went through about different patches applied to
> Y!'s branch before the final version.

That's what I meant by saying "a fair number of those may have been
included in trunk but under a different jira".  There seemed to be a
number of patches that are not in trunk under any jira (eg MR-1088,
MR-1100 where the jira is still open).  We need to go through the
patches in YDH and CDH and get them reviewed and checked in.

Thanks,
Eli


> -jg
>
>
> Eli Collins wrote:
>>
>> On Thu, Oct 21, 2010 at 1:00 PM, Owen O'Malley <omalley@apache.org> wrot=
e:
>>>
>>> On Oct 21, 2010, at 12:13 PM, Ian Holsman wrote:
>>>
>>>> I wanted to start a conversation about how we could merge the the
>>>> cloudera
>>>> +
>>>> yahoo distribtutions of hadoop into our codebase,
>>>> and what would be required.
>>>
>>> All of the patches that are the "yahoo distribution of hadoop" have bee=
n
>>> in
>>> Apache's trunk for months.
>>
> =A0fair number of those may
>>
>> have been included in trunk but under a different jira, however some
>> (eg MR-1088, MR-1100) are definitely not in trunk. =A0Also, if I
>> remember correctly some of the 20-based patches are substantially
>> different than the versions for trunk.
>>
>> Thanks,
>> Eli
>
>

From general-return-2223-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 00:31:18 2010
Return-Path: <general-return-2223-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 8907 invoked from network); 22 Oct 2010 00:31:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 00:31:18 -0000
Received: (qmail 76777 invoked by uid 500); 22 Oct 2010 00:31:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76732 invoked by uid 500); 22 Oct 2010 00:31:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76724 invoked by uid 99); 22 Oct 2010 00:31:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 00:31:17 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mmoores@real.com designates 207.188.23.6 as permitted sender)
Received: from [207.188.23.6] (HELO jor-el.real.com) (207.188.23.6)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 00:31:09 +0000
Received: from seacas01.corp.real.com ([::ffff:192.168.139.56])
  (TLS: TLSv1/SSLv3,128bits,AES128-SHA)
  by jor-el.real.com with esmtp; Thu, 21 Oct 2010 17:30:41 -0700
  id 00094069.4CC0DB31.00007768
Received: from seambx.corp.real.com ([fe80::2d15:fda7:b3b8:e268]) by
 seacas01.corp.real.com ([192.168.139.56]) with mapi; Thu, 21 Oct 2010
 17:30:41 -0700
From: Michael Moores <mmoores@real.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 21 Oct 2010 17:30:40 -0700
Subject: Re: Limiting concurrent maps
Thread-Topic: Limiting concurrent maps
Thread-Index: ActxgFtaIc3s8+yySDaKEBykfejieQ==
Message-ID: <ECA2730C-2F85-4ECA-BA37-3E996D007938@real.com>
References: <8AE653F8-6EA6-4D50-A7B2-CA1B0CC75649@real.com>
 <AANLkTin2o+gGdoRCNCA0sEXr7EpNhv5+GEo19j+tFf3Y@mail.gmail.com>
In-Reply-To: <AANLkTin2o+gGdoRCNCA0sEXr7EpNhv5+GEo19j+tFf3Y@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_ECA2730C2F854ECABA373E996D007938realcom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_ECA2730C2F854ECABA373E996D007938realcom_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

I don't see how the capacity scheduler could limit the number of maps runni=
ng concurrently across  the whole cluster,
even if this is the only job running.

but, maybe with the fair scheduler mapred.fairscheduler.loadmanager extensi=
on point:


mapred.fairscheduler.loadmanager        An extensibility point that lets yo=
u specify a class that determines how many maps and reduces can run on a gi=
ven TaskTracker. This class should implement the LoadManager interface. By =
default the task caps in the Hadoop config file are used, but this option c=
ould be used to make the load based on available memory and CPU utilization=
 for example.


On Oct 20, 2010, at 4:32 PM, james warren wrote:

Hi Michael,

Any of the tasktracker configs affect the local tasktracker daemon and not
other servers in your cluster.  Moreover, they can't be overridden by a job
configuration.  Sounds like you're in need of a job scheduler; I personally
prefer use the Fair Scheduler but I'm sure the Capacity Scheduler would sui=
t
your needs as well.

cheers,
-James

On Wed, Oct 20, 2010 at 3:41 PM, Michael Moores <mmoores@real.com<mailto:mm=
oores@real.com>> wrote:

I have been playing with mapreduce.tasktracker.map.tasks.maximum to reduce
the load
on my Cassandra cluster (using the Cassandra ColumnFamilyInputFormat).  I'd
like to find ways of throttling the map operations
in the case I may be affecting OLTP activity on the cluster.

What parameters can I use to limit the number of map tasks running
concurrently across the whole cluster?
mapreduce.tasktracker.map.tasks.maximum
limits the number of concurrent maps per task tracker.  But can i do this
at the job level?

Should I look at the "fair" scheduler?

regards,Michael


--_000_ECA2730C2F854ECABA373E996D007938realcom_--

From general-return-2224-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 00:31:48 2010
Return-Path: <general-return-2224-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9096 invoked from network); 22 Oct 2010 00:31:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 00:31:47 -0000
Received: (qmail 78311 invoked by uid 500); 22 Oct 2010 00:31:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78258 invoked by uid 500); 22 Oct 2010 00:31:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78250 invoked by uid 99); 22 Oct 2010 00:31:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 00:31:46 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 00:31:41 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id o9M0UMUR074026
	for <general@hadoop.apache.org>; Thu, 21 Oct 2010 17:30:22 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1287707422;
	bh=HR0/9JPxuCN/MHi4fP8sniTfr6z5byHvLIfAyxMVb5g=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=PIIqRnX3U8WNnIzxZF8Ys7im6+xc1jgOxOkL1w9xW+3Lascs/2M+GjPDNtdeMHOCg
	 s+oNRLpKEp/UrWA+D155Fc556r6trxGt2H3MvZM8y9uIBd8qdkzlkiN2Kbhpp74oLR
	 tWnC4eToWZrvWjZHx0R5aGYLHjG6LxRG5BcvFyr4=
Message-Id: <C86BB6A4-B881-425C-BA28-63C59B4DC08E@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <AANLkTim1ygQZEMhCpGRG0u86kDf0xRr8QWAuo=8TtLqa@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: bringing the codebases back in line
Date: Thu, 21 Oct 2010 17:30:22 -0700
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com> <4CC0BC80.6050303@apache.org> <211EF221-CFC5-42A5-99AD-339F120333D1@apache.org> <AANLkTim1ygQZEMhCpGRG0u86kDf0xRr8QWAuo=8TtLqa@mail.gmail.com>
X-Mailer: Apple Mail (2.936)

Ian,

On Oct 21, 2010, at 4:50 PM, Ian Holsman wrote:

> but the other question I have which hopefully you guys can answer is  
> does
> the yahoo distribution have ALL the patches from the trunk on it?  
> because if
> it doesn't I think that is problematic as well for other reasons.

Yahoo put security on Apache Hadoop-0.20.

Apache Hadoop trunk is very far from hadoop-0.20, there are lots of  
features in trunk which aren't part of yahoo-hadoop-0.20 simply  
because there wasn't a need or it wasn't worth our effort to backport  
them etc. I know, since I have a big hand in deciding it.

However, we have been very religious about porting all our changes to  
trunk, we might have missed a couple due to time pressure, human  
mistake etc.

Thus, it isn't feasible for yahoo distribution to be a superset of  
trunk. Even more because it takes a *huge* amount of effort to qualify  
trunk... we at Yahoo qualified Apache Hadoop 0.20 and have stuck with  
it for over a year now, same as Cloudera, Facebook etc. Again, I'll  
point out that we have been very good at porting nearly 4000 internal  
commits to trunk throughout this time.

Hope that helps.

thanks,
Arun


From general-return-2225-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 00:32:38 2010
Return-Path: <general-return-2225-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9274 invoked from network); 22 Oct 2010 00:32:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 00:32:37 -0000
Received: (qmail 79619 invoked by uid 500); 22 Oct 2010 00:32:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79572 invoked by uid 500); 22 Oct 2010 00:32:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 79564 invoked by uid 99); 22 Oct 2010 00:32:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 00:32:36 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 69.147.107.21 is neither permitted nor denied by domain of acm@yahoo-inc.com)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 00:32:31 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id o9M0Vj7L074614
	for <general@hadoop.apache.org>; Thu, 21 Oct 2010 17:31:46 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1287707506;
	bh=jqVh7TFdvJ3Fiy0hsp8xQGLBWxJzufwFJLhuE3ondzs=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=djkI+yAwPBSQuvTY4g9qC9+bTN3h5waa3jh967SCAMq852p9RuQ4esgvfIkNdfqBo
	 X28QOE+8kj8TfX5IoI7FIn/lBvUpnsiv2q3URx5R9fduAg/bpmkGjaxlZ6j7jvA7cm
	 4MG1UWPRZv964Z370iJagUgcVH+WcP8YXa8aeshM=
Message-Id: <06380AD8-CC00-4983-A93D-7DE01EE60753@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <AANLkTi=WF85J47-JcRghD7AzeaA7qZmScGVQt2fn2NHV@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: bringing the codebases back in line
Date: Thu, 21 Oct 2010 17:31:45 -0700
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com> <A2ED1DE3-B131-4D83-AE0A-382F6A64C06F@apache.org> <AANLkTimnqhcG_nazMjSjwvXnvw_udDXJkFq3VbNwXZda@mail.gmail.com> <4CC0BEED.2020200@yahoo-inc.com> <AANLkTi=WF85J47-JcRghD7AzeaA7qZmScGVQt2fn2NHV@mail.gmail.com>
X-Mailer: Apple Mail (2.936)


On Oct 21, 2010, at 5:17 PM, Eli Collins wrote:

> On Thu, Oct 21, 2010 at 3:30 PM, Jakob Homan <jhoman@yahoo-inc.com>  
> wrote:
>> If the patch was just checking 1:1 Jira to patch, it would  
>> certainly not
>> work.  We were uploading multiple patches to the same JIRA to avoid  
>> opening
>> extraneous issues before generating patches for trunk. Venerable old
>> HDFS-1150, for instance, went through about different patches  
>> applied to
>> Y!'s branch before the final version.
>
> That's what I meant by saying "a fair number of those may have been
> included in trunk but under a different jira".  There seemed to be a
> number of patches that are not in trunk under any jira (eg MR-1088,
> MR-1100 where the jira is still open).  We need to go through the
> patches in YDH and CDH and get them reviewed and checked in.

It would be great to get HADOOP-6255, the functionality for creating  
RPMs, from CDH to Apache Hadoop.

Arun



From general-return-2226-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 00:34:26 2010
Return-Path: <general-return-2226-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9872 invoked from network); 22 Oct 2010 00:34:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 00:34:26 -0000
Received: (qmail 83758 invoked by uid 500); 22 Oct 2010 00:34:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83704 invoked by uid 500); 22 Oct 2010 00:34:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83696 invoked by uid 99); 22 Oct 2010 00:34:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 00:34:24 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 00:34:18 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id o9M0XRhK075188
	for <general@hadoop.apache.org>; Thu, 21 Oct 2010 17:33:27 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1287707607;
	bh=yMxJUvRKY38cJ7/ScnzwZJxmI1kCRW6vf0YKH3aiVmk=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=lA+ccwRaeD8qEf3iu6WMNuECVJGIwxItLVx4SzOV68ur3yQRcKOywCDpkupJATLav
	 1SozKwRgTuIJXTwCy4G3gw5Fvnk0OsLH/kr6Lj2Ja3UT2W4OjbqGyEHbgzZA8sQQJS
	 5Qq94aF3WBimVb44utMkb0r5dYe1m4RZxo9+uv6c=
Message-Id: <360ECE7D-48B4-4B74-8F4A-605B51C21043@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <ECA2730C-2F85-4ECA-BA37-3E996D007938@real.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Limiting concurrent maps
Date: Thu, 21 Oct 2010 17:33:27 -0700
References: <8AE653F8-6EA6-4D50-A7B2-CA1B0CC75649@real.com> <AANLkTin2o+gGdoRCNCA0sEXr7EpNhv5+GEo19j+tFf3Y@mail.gmail.com> <ECA2730C-2F85-4ECA-BA37-3E996D007938@real.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


On Oct 21, 2010, at 5:30 PM, Michael Moores wrote:

> I don't see how the capacity scheduler could limit the number of  
> maps running concurrently across  the whole cluster,
> even if this is the only job running.
>

Easy, set a maximum limit on the queue.

Arun


From general-return-2227-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 00:38:09 2010
Return-Path: <general-return-2227-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 10980 invoked from network); 22 Oct 2010 00:38:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 00:38:09 -0000
Received: (qmail 87493 invoked by uid 500); 22 Oct 2010 00:38:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87441 invoked by uid 500); 22 Oct 2010 00:38:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87433 invoked by uid 99); 22 Oct 2010 00:38:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 00:38:07 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 00:38:01 +0000
Received: by yxn22 with SMTP id 22so180043yxn.35
        for <general@hadoop.apache.org>; Thu, 21 Oct 2010 17:37:40 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.185.208 with SMTP id cp16mr1422485qcb.213.1287707860137;
 Thu, 21 Oct 2010 17:37:40 -0700 (PDT)
Received: by 10.229.237.131 with HTTP; Thu, 21 Oct 2010 17:37:40 -0700 (PDT)
In-Reply-To: <AANLkTim1ygQZEMhCpGRG0u86kDf0xRr8QWAuo=8TtLqa@mail.gmail.com>
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>
	<4CC0BC80.6050303@apache.org>
	<211EF221-CFC5-42A5-99AD-339F120333D1@apache.org>
	<AANLkTim1ygQZEMhCpGRG0u86kDf0xRr8QWAuo=8TtLqa@mail.gmail.com>
Date: Thu, 21 Oct 2010 17:37:40 -0700
Message-ID: <AANLkTinPugs3ecK39rsy2+iMwPP1nwfWvQyfXu2XS5wN@mail.gmail.com>
Subject: Re: bringing the codebases back in line
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Thu, Oct 21, 2010 at 4:50 PM, Ian Holsman <hadoop@holsman.net> wrote:
> right.. Cloudera is bundling it's add-ons into a single tarball to make i=
t
> easier to install.

CDH contains a number of different projects, however each project has
a distinct tarball (and packages). The tarball is essentially an
Apache release (tarball) plus a directory that has a set of patches
that we've applied to the Apache release (our build process downloads
the Apache release and applies our set of patches).  For each version
of CDH we rebase our patch set on the latest Apache dot release
available at the time to minimize our delta with upstream.  Here's an
example tarball:
http://archive.cloudera.com/cdh/3/hadoop-0.20.2+737.tar.gz

> In my ideal world, I'd like to be able to just download/buy any of those
> tools and have them run on a released apache hadoop tarball. and then if
> someone else comes along with a competing tool I would be free to choose =
it
> and have it also run on my apache hadoop tarball, not have to go through =
the
> pain of saying XXX tool needs their customized version of hadoop so I can=
't
> use it. (ie remove the lock-in that comes from a forked base).

All of our Apache projects are an Apache release plus a set of
patches, these are typically backports of bug fixes in trunk but not a
dot release. Except for Hadoop, the set of additional patches is very
small. Here's an example, the 16 changes not in Pig 0.7 that we've
included: http://archive.cloudera.com/cdh/3/pig-0.7.0+16.CHANGES.txt

> so what I'd like to see is both cloudera and yahoo running a minimal set =
of
> patches as a 'superset' of the apache hadoop stuff, with the apache hadoo=
p
> very close to both of these. the only patches being in either being to fi=
x
> bugs or performance issues that would be available in the next release of
> a-hadoop.

That's our goal as well. For all the Apache projects in CDH, except
for Hadoop, that is the case today.  For CDH3 we ended up adding large
additional patch sets (the  security patch set the append patch set to
support HBase), but for Apache 22 the majority of the delta that CDH
and YDH have against Apache 20 will go away (thanks to Y! contributing
security and append to trunk).

> And when a new release of a-hadoop comes, it the vendors would switch to
> using that a-hadoop version as their baseline.
>
> I don't want to get into the situation that linux is in with redhat in th=
at
> their kernel is dramatically different to the one on kernel.org.
>
> does that make sense?

Definitely.

Thanks,
Eli
>
>
>
>
> On Thu, Oct 21, 2010 at 6:42 PM, Owen O'Malley <omalley@apache.org> wrote=
:
>
>>
>> On Oct 21, 2010, at 3:19 PM, Doug Cutting wrote:
>>
>> =A0Cloudera's distribution is based on Y!'s 0.20 distribution, together =
with
>>> patches from the Apache 0.20-append branch,
>>>
>>
>> Cloudera's Distribution of Hadoop includes many tools from outside of
>> Hadoop and even outside of Apache.
>>
>> -- Owen
>>
>>
>

From general-return-2228-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 00:42:49 2010
Return-Path: <general-return-2228-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12299 invoked from network); 22 Oct 2010 00:42:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 00:42:49 -0000
Received: (qmail 91540 invoked by uid 500); 22 Oct 2010 00:42:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91474 invoked by uid 500); 22 Oct 2010 00:42:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91441 invoked by uid 99); 22 Oct 2010 00:42:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 00:42:45 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 00:42:37 +0000
Received: from sp1-ex07cas01.ds.corp.yahoo.com (sp1-ex07cas01.ds.corp.yahoo.com [216.252.116.137])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o9M0g346040917
	for <general@hadoop.apache.org>; Thu, 21 Oct 2010 17:42:03 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1287708123;
	bh=w2ObZruhrHrYLtj1o7h5UavQBq0DfMG83DFQJAL30gQ=;
	h=From:To:Date:Subject:Message-ID:References:In-Reply-To:
	 Content-Type:Content-Transfer-Encoding:MIME-Version;
	b=AaEQAtaWD/DlExp08NROXLqdCjzxy6lci3Nm0YK7nqz6YGZwYCMq7zFdvdO2kSlkj
	 uI8JnMZt3+7ruzTbnacq6wgF2KboqgE1BVeC0a4qm32eFvtYsjn7OaXv9zQ7AhbQZX
	 ExeFGQzzjoTJR57Zb090WQjdORAn+aG5XWfZXb/k=
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas01.ds.corp.yahoo.com ([216.252.116.137]) with mapi; Thu, 21 Oct
 2010 17:42:03 -0700
From: Milind A Bhandarkar <milindb@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 21 Oct 2010 17:42:02 -0700
Subject: Re: bringing the codebases back in line
Thread-Topic: bringing the codebases back in line
Thread-Index: ActxgfGGYCFzr2FGS/affJInvqGXnQ==
Message-ID: <E398F71E-867A-4682-A736-A3CB2FE05B05@yahoo-inc.com>
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>
 <4CC0BC80.6050303@apache.org>
 <211EF221-CFC5-42A5-99AD-339F120333D1@apache.org>
 <AANLkTim1ygQZEMhCpGRG0u86kDf0xRr8QWAuo=8TtLqa@mail.gmail.com>
In-Reply-To: <AANLkTim1ygQZEMhCpGRG0u86kDf0xRr8QWAuo=8TtLqa@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


>=20
> but the other question I have which hopefully you guys can answer is does
> the yahoo distribution have ALL the patches from the trunk on it? because=
 if
> it doesn't I think that is problematic as well for other reasons.


What are these "other" reasons ?

yahoo distribution runs on our production clusters, and I do not see why an=
y production cluster should run code from trunk.

- Milind

--
Milind Bhandarkar
(mailto:milindb@yahoo-inc.com)
(phone: 408-203-5213 W)



From general-return-2229-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 00:52:12 2010
Return-Path: <general-return-2229-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14229 invoked from network); 22 Oct 2010 00:52:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 00:52:12 -0000
Received: (qmail 533 invoked by uid 500); 22 Oct 2010 00:52:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 480 invoked by uid 500); 22 Oct 2010 00:52:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 472 invoked by uid 99); 22 Oct 2010 00:52:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 00:52:11 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 00:52:06 +0000
Received: by yxn22 with SMTP id 22so189545yxn.35
        for <general@hadoop.apache.org>; Thu, 21 Oct 2010 17:51:45 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.236.193 with SMTP id kl1mr540184qcb.37.1287708705320; Thu,
 21 Oct 2010 17:51:45 -0700 (PDT)
Received: by 10.229.237.131 with HTTP; Thu, 21 Oct 2010 17:51:45 -0700 (PDT)
In-Reply-To: <06380AD8-CC00-4983-A93D-7DE01EE60753@yahoo-inc.com>
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>
	<A2ED1DE3-B131-4D83-AE0A-382F6A64C06F@apache.org>
	<AANLkTimnqhcG_nazMjSjwvXnvw_udDXJkFq3VbNwXZda@mail.gmail.com>
	<4CC0BEED.2020200@yahoo-inc.com>
	<AANLkTi=WF85J47-JcRghD7AzeaA7qZmScGVQt2fn2NHV@mail.gmail.com>
	<06380AD8-CC00-4983-A93D-7DE01EE60753@yahoo-inc.com>
Date: Thu, 21 Oct 2010 17:51:45 -0700
Message-ID: <AANLkTik79syC6croM4Mw2ir6VJPeJXMTgb9wa-t5QfBw@mail.gmail.com>
Subject: Re: bringing the codebases back in line
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Thu, Oct 21, 2010 at 5:31 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:
>
> On Oct 21, 2010, at 5:17 PM, Eli Collins wrote:
>
>> On Thu, Oct 21, 2010 at 3:30 PM, Jakob Homan <jhoman@yahoo-inc.com> wrot=
e:
>>>
>>> If the patch was just checking 1:1 Jira to patch, it would certainly no=
t
>>> work. =A0We were uploading multiple patches to the same JIRA to avoid
>>> opening
>>> extraneous issues before generating patches for trunk. Venerable old
>>> HDFS-1150, for instance, went through about different patches applied t=
o
>>> Y!'s branch before the final version.
>>
>> That's what I meant by saying "a fair number of those may have been
>> included in trunk but under a different jira". =A0There seemed to be a
>> number of patches that are not in trunk under any jira (eg MR-1088,
>> MR-1100 where the jira is still open). =A0We need to go through the
>> patches in YDH and CDH and get them reviewed and checked in.
>
> It would be great to get HADOOP-6255, the functionality for creating RPMs=
,
> from CDH to Apache Hadoop.
>

There are some challenges in contributing packaging up stream:
- Packaging is typically versioned independently from the project code
(we share packaging code across project major versions). This is why
most projects have a separate repository for the packaging, and the
packaging is done by a distribution.
- The packaging source shares code across the 10 or so components,
which is useful since we continuously make the packaging more
consistent, so hosting the code on any single project repository
doesn't fit well.
- The packaging is Linux specific, we've gotten push back when trying
to contribute modifications upstream with Linuxisms since Apache
supports non-Linux platforms (namely Solaris).

All of our packaging is Apache licensed (and we publish source RPMs)
so there's no issue from that perspective.  But this is a digression
from the subject at hand so we should probably table for a separate
discussion.

Thanks,
Eli

From general-return-2230-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 00:54:51 2010
Return-Path: <general-return-2230-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 15152 invoked from network); 22 Oct 2010 00:54:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 00:54:51 -0000
Received: (qmail 3043 invoked by uid 500); 22 Oct 2010 00:54:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3004 invoked by uid 500); 22 Oct 2010 00:54:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2996 invoked by uid 99); 22 Oct 2010 00:54:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 00:54:49 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 00:54:44 +0000
Received: by wyg36 with SMTP id 36so240887wyg.35
        for <general@hadoop.apache.org>; Thu, 21 Oct 2010 17:54:23 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.49.212 with SMTP id x62mr1768607web.55.1287708863171; Thu,
 21 Oct 2010 17:54:23 -0700 (PDT)
Received: by 10.216.87.78 with HTTP; Thu, 21 Oct 2010 17:54:23 -0700 (PDT)
In-Reply-To: <E398F71E-867A-4682-A736-A3CB2FE05B05@yahoo-inc.com>
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>
	<4CC0BC80.6050303@apache.org>
	<211EF221-CFC5-42A5-99AD-339F120333D1@apache.org>
	<AANLkTim1ygQZEMhCpGRG0u86kDf0xRr8QWAuo=8TtLqa@mail.gmail.com>
	<E398F71E-867A-4682-A736-A3CB2FE05B05@yahoo-inc.com>
Date: Thu, 21 Oct 2010 20:54:23 -0400
Message-ID: <AANLkTi=JFzv0gyOg6pzk7be+KEBjNGO+JQZRbqwunSvh@mail.gmail.com>
Subject: Re: bringing the codebases back in line
From: Ian Holsman <hadoop@holsman.net>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f5ce5c5d0f3704932a1535

--001485f5ce5c5d0f3704932a1535
Content-Type: text/plain; charset=ISO-8859-1

On Thu, Oct 21, 2010 at 8:42 PM, Milind A Bhandarkar
<milindb@yahoo-inc.com>wrote:

>
> >
> > but the other question I have which hopefully you guys can answer is does
> > the yahoo distribution have ALL the patches from the trunk on it? because
> if
> > it doesn't I think that is problematic as well for other reasons.
>
>
> What are these "other" reasons ?
>

yahoo distribution runs on our production clusters, and I do not see why any
> production cluster should run code from trunk.
>
>
right.. the trunk is not for production use.  I wasn't suggesting that.

but the trunk is what will eventually become the next release.

Then someone in yahoo will have to decide if they are going to move to
rebase their production cluster to 0.21, or just continue back-porting what
they need to the version they are running on their clusters.

and if yahoo fixes a bug in their version, it would need to be
forward-ported over to the current trunk. which will get harder and harder
as the paths diverge.

I'm sure you've seen it happen on other projects when a major branch lands
on the trunk, and the amount of effort it takes to reconcile them.
  - Milind

--
Milind Bhandarkar
(mailto:milindb@yahoo-inc.com)
(phone: 408-203-5213 W)

--001485f5ce5c5d0f3704932a1535--

From general-return-2231-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 00:58:11 2010
Return-Path: <general-return-2231-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 15519 invoked from network); 22 Oct 2010 00:58:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 00:58:11 -0000
Received: (qmail 6757 invoked by uid 500); 22 Oct 2010 00:58:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6703 invoked by uid 500); 22 Oct 2010 00:58:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6695 invoked by uid 99); 22 Oct 2010 00:58:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 00:58:09 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 00:58:05 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id o9M0vN7V031305
	for <general@hadoop.apache.org>; Thu, 21 Oct 2010 17:57:23 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1287709044;
	bh=dgHQ7p2KQNWAZxh1zhqzZb0QNt44VOhgPkyMiJO+QS0=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=ZvaLarxVjr11QSF/wNYur9g7qsmE030zVAVRHoU6chfywYDPwjsPugVbS3EuLK+F9
	 N1fe45K8tDOyQDeWulmBwOT7DWO/CnJalKXsJDEP4c3MT27mkTS+5CSXZY329lE+ls
	 HB0OR8Gf/joSEBYwQRxDSYPOuUmf2C99RuZe4v7s=
Message-Id: <3FF18D23-A2C3-4CD0-BD37-E1DF0C4F5E61@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <AANLkTik79syC6croM4Mw2ir6VJPeJXMTgb9wa-t5QfBw@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: bringing the codebases back in line
Date: Thu, 21 Oct 2010 17:57:23 -0700
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com> <A2ED1DE3-B131-4D83-AE0A-382F6A64C06F@apache.org> <AANLkTimnqhcG_nazMjSjwvXnvw_udDXJkFq3VbNwXZda@mail.gmail.com> <4CC0BEED.2020200@yahoo-inc.com> <AANLkTi=WF85J47-JcRghD7AzeaA7qZmScGVQt2fn2NHV@mail.gmail.com> <06380AD8-CC00-4983-A93D-7DE01EE60753@yahoo-inc.com> <AANLkTik79syC6croM4Mw2ir6VJPeJXMTgb9wa-t5QfBw@mail.gmail.com>
X-Mailer: Apple Mail (2.936)

I was merely pointing out, given the number of interested parties on  
that jira, that having Hadoop RPMs for Linux is very desirable.

We could have a technical discussion on the ways to go about doing  
RPMs, debs etc., but it is clear that there is a need for something  
more than tgz releases, even if they are Linux specific. This is  
particularly true given Linux specific components such as  
LinuxTaskController, jsvc for security etc.

Arun


On Oct 21, 2010, at 5:51 PM, Eli Collins wrote:

> On Thu, Oct 21, 2010 at 5:31 PM, Arun C Murthy <acm@yahoo-inc.com>  
> wrote:
>>
>> On Oct 21, 2010, at 5:17 PM, Eli Collins wrote:
>>
>>> On Thu, Oct 21, 2010 at 3:30 PM, Jakob Homan <jhoman@yahoo- 
>>> inc.com> wrote:
>>>>
>>>> If the patch was just checking 1:1 Jira to patch, it would  
>>>> certainly not
>>>> work.  We were uploading multiple patches to the same JIRA to avoid
>>>> opening
>>>> extraneous issues before generating patches for trunk. Venerable  
>>>> old
>>>> HDFS-1150, for instance, went through about different patches  
>>>> applied to
>>>> Y!'s branch before the final version.
>>>
>>> That's what I meant by saying "a fair number of those may have been
>>> included in trunk but under a different jira".  There seemed to be a
>>> number of patches that are not in trunk under any jira (eg MR-1088,
>>> MR-1100 where the jira is still open).  We need to go through the
>>> patches in YDH and CDH and get them reviewed and checked in.
>>
>> It would be great to get HADOOP-6255, the functionality for  
>> creating RPMs,
>> from CDH to Apache Hadoop.
>>
>
> There are some challenges in contributing packaging up stream:
> - Packaging is typically versioned independently from the project code
> (we share packaging code across project major versions). This is why
> most projects have a separate repository for the packaging, and the
> packaging is done by a distribution.
> - The packaging source shares code across the 10 or so components,
> which is useful since we continuously make the packaging more
> consistent, so hosting the code on any single project repository
> doesn't fit well.
> - The packaging is Linux specific, we've gotten push back when trying
> to contribute modifications upstream with Linuxisms since Apache
> supports non-Linux platforms (namely Solaris).
>
> All of our packaging is Apache licensed (and we publish source RPMs)
> so there's no issue from that perspective.  But this is a digression
> from the subject at hand so we should probably table for a separate
> discussion.
>
> Thanks,
> Eli


From general-return-2232-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 04:30:31 2010
Return-Path: <general-return-2232-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 74462 invoked from network); 22 Oct 2010 04:30:31 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 04:30:31 -0000
Received: (qmail 4014 invoked by uid 500); 22 Oct 2010 04:30:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3777 invoked by uid 500); 22 Oct 2010 04:30:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3769 invoked by uid 99); 22 Oct 2010 04:30:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 04:30:26 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 04:30:20 +0000
Received: from sp1-ex07cas01.ds.corp.yahoo.com (sp1-ex07cas01.ds.corp.yahoo.com [216.252.116.137])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id o9M4TV35085162
	for <general@hadoop.apache.org>; Thu, 21 Oct 2010 21:29:31 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1287721771;
	bh=wfhVtbFNbrJzmFj0ofwsWYHFmlD4xWmBIzhMR0Y2CX4=;
	h=From:To:Date:Subject:Message-ID:References:In-Reply-To:
	 Content-Type:Content-Transfer-Encoding:MIME-Version;
	b=Hf6Lgk6iRfc8ON6wWvVSfXQQCoUySzoYTTHxOQCsD7+FDioDYolEgDuLSdB+YKVCj
	 MER6J3Ye3lYKbXOWOHBWeyHYa1avJoGhJvtloT/+gUgwXOarWhUo+aV3pekG/y+Ez2
	 ataevIuGq0qommilPfJmp+768/spF5x7u7UTQOK8=
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas01.ds.corp.yahoo.com ([216.252.116.137]) with mapi; Thu, 21 Oct
 2010 21:29:31 -0700
From: Milind A Bhandarkar <milindb@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 21 Oct 2010 21:29:30 -0700
Subject: Re: bringing the codebases back in line
Thread-Topic: bringing the codebases back in line
Thread-Index: Actxobh0SglnEJLYSFWf6kjVnGvYhw==
Message-ID: <D3904CD6-27C9-44B8-B526-27A16717E5F5@yahoo-inc.com>
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>
 <4CC0BC80.6050303@apache.org>
 <211EF221-CFC5-42A5-99AD-339F120333D1@apache.org>
 <AANLkTim1ygQZEMhCpGRG0u86kDf0xRr8QWAuo=8TtLqa@mail.gmail.com>
 <E398F71E-867A-4682-A736-A3CB2FE05B05@yahoo-inc.com>
 <AANLkTi=JFzv0gyOg6pzk7be+KEBjNGO+JQZRbqwunSvh@mail.gmail.com>
In-Reply-To: <AANLkTi=JFzv0gyOg6pzk7be+KEBjNGO+JQZRbqwunSvh@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

>>=20
> right.. the trunk is not for production use.  I wasn't suggesting that.

So, what are you suggesting ? That Yahoo distribution of Hadoop should *not=
* be the version we run on our production clusters ?

>=20
> but the trunk is what will eventually become the next release.

>=20
> Then someone in yahoo will have to decide if they are going to move to
> rebase their production cluster to 0.21, or just continue back-porting wh=
at
> they need to the version they are running on their clusters.

Yes, that is what we do now. If there are committed patches in trunk that d=
o not scale for our needs, or break existing applications, or are deemed no=
t worth the efforts needed to backport, we do not include them in our deplo=
yments, and therefore do not include in Yahoo distribution.

>=20
> and if yahoo fixes a bug in their version, it would need to be
> forward-ported over to the current trunk. which will get harder and harde=
r
> as the paths diverge.

Yes, indeed. So, care must be taken that paths do not diverge too much. I h=
ave seen some cases where the bug fixes did not need to be forward ported, =
because that piece of code was completely re-written in trunk.

>=20
> I'm sure you've seen it happen on other projects when a major branch land=
s
> on the trunk, and the amount of effort it takes to reconcile them.

Yes. And that results in delayed releases. An unexpected benefit for applic=
ation developers was that they could spend time adding features to their ap=
plications, rather than porting same applications from release-to-release, =
and validating releases. So, it's not always bad.

- Milind


--
Milind Bhandarkar
(mailto:milindb@yahoo-inc.com)
(phone: 408-203-5213 W)



From general-return-2233-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 06:40:46 2010
Return-Path: <general-return-2233-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 13829 invoked from network); 22 Oct 2010 06:40:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 06:40:45 -0000
Received: (qmail 14935 invoked by uid 500); 22 Oct 2010 06:40:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14638 invoked by uid 500); 22 Oct 2010 06:40:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14630 invoked by uid 99); 22 Oct 2010 06:40:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 06:40:41 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.102 as permitted sender)
Received: from [17.148.16.102] (HELO asmtpout027.mac.com) (17.148.16.102)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 06:40:35 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp027.mac.com (Oracle Communications Messaging Exchange Server 7u4-18.01
 64bit (built Jul 15 2010)) with ESMTPSA id <0LAO00HZSIIFP130@asmtp027.mac.com>
 for general@hadoop.apache.org; Thu, 21 Oct 2010 23:39:52 -0700 (PDT)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2010-10-22_03:2010-10-22,2010-10-22,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1004200000 definitions=main-1010210287
Subject: Re: [DISCUSS] Proposed bylaws for Hadoop
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <AANLkTinE2TzQ88NBDrGUxfNbPu3Q-0k1fR-MbRT6xvd-@mail.gmail.com>
Date: Thu, 21 Oct 2010 23:39:51 -0700
Message-id: <B5B20A56-E6EC-4E3D-8C61-F6EA7ED8A6BB@mac.com>
References: <63A1C7A9-727B-4492-9998-01F6B6958EDD@apache.org>
 <4CBE11C5.301@apache.org> <CE8E008F-D08E-4564-80F0-B0326246B768@apache.org>
 <580DC943-D250-405D-A524-82452CBC4E03@mac.com>
 <AANLkTikR5uY0_zsT_owo-90yivuVE0PMUPFoEOx3U_8H@mail.gmail.com>
 <AANLkTinE2TzQ88NBDrGUxfNbPu3Q-0k1fR-MbRT6xvd-@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1081)


On Oct 21, 2010, at 10:10 AM, Tom White wrote:
> On Wed, Oct 20, 2010 at 3:15 PM, Chris Douglas <cdouglas@apache.org> wrote:

>>> I suggest we make this stronger.
>>> By way of comparison, the recently enacted bylaws for Pig
>>> (http://pig.apache.org/bylaws.html) have consensus, for example.
>> 
>> Consensus is equivalent to making the PMC a permanent appointment.
>> Discussions would be more civil and more likely to offer compromise-
>> we would actually work toward consensus- if intransigence and
>> hostility had consequences. Removing people is a last resort. Moving
>> the barrier closer doesn't make it more likely, but it aligns
>> incentives so rational people will reign in their more intemperate
>> comments and, instead, find ways to make progress.
>> 
>> Many projects require supermajorities or even 3/4 majorities. It's
>> sufficiently damning if more than half the PMC thinks an individual is
>> so destructive that less damage would be done by ejecting them, in my
>> opinion, but again: this is not a facility we should use, or have to
>> use. Throwing someone out is an expensive failure for the project, and
>> an conspicuous failure of its governance and community. Personally, I
>> don't care if it's a majority or a supermajority, but consensus is a
>> fig leaf.
> 
> Fair point. For a large PMC, that's probably true. This matter was
> discussed on the Pig list, and there was a range of opinion there:
> http://search-hadoop.com/m/jk35b2tZCuy&subj=RE+DISCUSS+Apache+Pig+bylaws.
> Personally, I think a supermajority strikes the right balance.
> 
> Tom

Yup, all good points Chris. I, too, agree that supermajority strikes a good balance.

Nige

From general-return-2234-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 06:46:42 2010
Return-Path: <general-return-2234-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14603 invoked from network); 22 Oct 2010 06:46:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 06:46:41 -0000
Received: (qmail 19153 invoked by uid 500); 22 Oct 2010 06:46:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18890 invoked by uid 500); 22 Oct 2010 06:46:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18882 invoked by uid 99); 22 Oct 2010 06:46:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 06:46:38 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 06:46:32 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=qgpTGuX/SXKpVO9eF+o8G/bIDv8PgD0xFRjfEqjye4fZkjvN9A8GK92e
   iM+F7wNp+pkmpuYoKjh0PG80gJOb8wpUla2vSwPapYSiJQnId8lrJ/4zi
   NOwKbzKXvEDQXgS;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1287729992; x=1319265992;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20bringing=20the=20codebases=20back=20in
   =20line|Date:=20Fri,=2022=20Oct=202010=2006:46:11=20+0000
   |Message-ID:=20<253774F2-2590-4E67-BB5B-86FE5BD5A555@link
   edin.com>|To:=20"<general@hadoop.apache.org>"=20<general@
   hadoop.apache.org>|MIME-Version:=201.0
   |Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<D0060A378177024CA31ADF2518459F26@linkedin
   .com>|In-Reply-To:=20<AANLkTik79syC6croM4Mw2ir6VJPeJXMTgb
   9wa-t5QfBw@mail.gmail.com>|References:=20<AANLkTik_fT+NSC
   D1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>=0D=0A=20<A
   2ED1DE3-B131-4D83-AE0A-382F6A64C06F@apache.org>=0D=0A=20<
   AANLkTimnqhcG_nazMjSjwvXnvw_udDXJkFq3VbNwXZda@mail.gmail.
   com>=0D=0A=20<4CC0BEED.2020200@yahoo-inc.com>=0D=0A=20<AA
   NLkTi=3DWF85J47-JcRghD7AzeaA7qZmScGVQt2fn2NHV@mail.gmail.
   com>=0D=0A=20<06380AD8-CC00-4983-A93D-7DE01EE60753@yahoo-
   inc.com>=0D=0A=20<AANLkTik79syC6croM4Mw2ir6VJPeJXMTgb9wa-
   t5QfBw@mail.gmail.com>;
  bh=oflcqlVo1UXk/m4FkuTVWyh01jj8MEFXRo8Nk2BQNjk=;
  b=JFFOx13PME2AO+6P/hCMt7I54ZHPp4aScj10uV1FfjtlOiXvbpgLvfmO
   KAru9tH/CanIJ95HlmbUgZaAJbH3mQmCvithaPw0edQE+pATrER7IJtIb
   +K/1f65nMSCkDYs;
X-IronPort-AV: E=Sophos;i="4.58,222,1286175600"; 
   d="scan'208";a="16287934"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi id 14.01.0218.012; Thu,
 21 Oct 2010 23:46:11 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: bringing the codebases back in line
Thread-Topic: bringing the codebases back in line
Thread-Index: AQHLcVQVuqCpAAO/eUm8zw+IHmCpg5NMR8eAgAAf9oCAAAn3gIAAHiAAgAAD34CAAAWWgIAAYwYA
Date: Fri, 22 Oct 2010 06:46:11 +0000
Message-ID: <253774F2-2590-4E67-BB5B-86FE5BD5A555@linkedin.com>
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>
 <A2ED1DE3-B131-4D83-AE0A-382F6A64C06F@apache.org>
 <AANLkTimnqhcG_nazMjSjwvXnvw_udDXJkFq3VbNwXZda@mail.gmail.com>
 <4CC0BEED.2020200@yahoo-inc.com>
 <AANLkTi=WF85J47-JcRghD7AzeaA7qZmScGVQt2fn2NHV@mail.gmail.com>
 <06380AD8-CC00-4983-A93D-7DE01EE60753@yahoo-inc.com>
 <AANLkTik79syC6croM4Mw2ir6VJPeJXMTgb9wa-t5QfBw@mail.gmail.com>
In-Reply-To: <AANLkTik79syC6croM4Mw2ir6VJPeJXMTgb9wa-t5QfBw@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <D0060A378177024CA31ADF2518459F26@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


On Oct 21, 2010, at 5:51 PM, Eli Collins wrote:
>=20
> - The packaging is Linux specific, we've gotten push back when trying
> to contribute modifications upstream with Linuxisms since Apache
> supports non-Linux platforms (namely Solaris).

	Oh come now Eli.  Just say it: I push everyone really hard on making thing=
s multiple platform to the point of being as trollish as this Ian fellow th=
at started this thread.

	Most of the Linux-specific modifications that I've seen that have been sub=
mitted are specifically to avoid configuring the system properly*, built in=
 such a way that they weren't pluggable for other operating systems, or cou=
ld be done in a much more POSIX/OS-agnostic way.  The fact that there are p=
arts of Hadoop that don't work properly on Mac OS X (despite the overwhelmi=
ng number of Mac laptops in use by Hadoop core devs) always struck me as pa=
rticularly funny when people get frustrated with me when I point this probl=
ems out.

	It is also worth mentioning that, AFAIK, only Linux and AIX have OS-specif=
ic code in Hadoop.  Attempts to get fixes (not even features!) for specific=
 issues for other operating systems have been fully rejected with the cry o=
f "we the community don't want to support specific OS issues in the core", =
even tho some of them are direct hinderance to the proper operation of Hado=
op on those OSes.  We have a very big double standard at work here.

*  e.g., searching through x LInux distribution directories looking for a w=
orking JRE in the primary hadoop script to protect users from putting the l=
ocation in hadoop-env.sh.   To me the great irony here is that this is the =
type of code that'd be great for a (mostly) single use tool do... like, say=
, a packaging system....=

From general-return-2235-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 11:38:11 2010
Return-Path: <general-return-2235-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 2292 invoked from network); 22 Oct 2010 11:38:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 11:38:11 -0000
Received: (qmail 84207 invoked by uid 500); 22 Oct 2010 11:38:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84094 invoked by uid 500); 22 Oct 2010 11:38:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84086 invoked by uid 99); 22 Oct 2010 11:38:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 11:38:05 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 11:37:57 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id A256BB7DF0
	for <general@hadoop.apache.org>; Fri, 22 Oct 2010 12:37:35 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id dE1uI6tX6JEg for <general@hadoop.apache.org>;
	Fri, 22 Oct 2010 12:37:35 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id D6310B7D6F
	for <general@hadoop.apache.org>; Fri, 22 Oct 2010 12:37:34 +0100 (BST)
MailScanner-NULL-Check: 1288352237.81233@sQlDbTcssYuoo8iVNWnMMQ
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o9MBbH3X024370
	for <general@hadoop.apache.org>; Fri, 22 Oct 2010 12:37:17 +0100 (BST)
Message-ID: <4CC1776D.50901@apache.org>
Date: Fri, 22 Oct 2010 12:37:17 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.12) Gecko/20100915 Thunderbird/3.0.8
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: bringing the codebases back in line
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>	<D9F0255E-7517-4CDB-9CCD-912B842F9E7C@linkedin.com> <AANLkTimkd_t2k168JpejruPHVDRSBhE1XzPsEtV9rq3M@mail.gmail.com>
In-Reply-To: <AANLkTimkd_t2k168JpejruPHVDRSBhE1XzPsEtV9rq3M@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o9MBbH3X024370
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 21/10/10 22:53, Ian Holsman wrote:
> yep.. I've heard it's a source of contention...
>
> but I'd like to see how we can get it so the amount of patches that the
> large companies apply on top of the current production apache release gets
> minimized, and the large installations are all running nearly identical code
> on their clusters, and that we wouldn't need to have a yahoo or cloudera
> repo of their patch sets made available.
>
> So Ideally I'd like to hear what kind of things apache needs to do help get
> these kind of things less divergent.
>
> In discussing it with people, I've heard that a major issue (not the only
> one i'm sure) is lack of resources to actually test the apache releases on
> large clusters, and that it is very hard getting this done in short cycles
> (hence the large gap between 20.x and 21).
>
> So I thought I would start the thread to see if we could at least identify
> what the people think are the problems are.

A big issue is the $ value of the data in a production cluster, the size 
of the large clusters, and the fact that they are in use. The only way 
to test on a few hundred nodes -especially now that 12-24TB/node is 
possible, is when you are bringing up a cluster of this size, which is a 
pretty rare event. Lots of us have small real or virtual clusters, but 
they don't stress the NN or JT, don't raise emergent problems like the 
increasing cost of rebalancing 24TB nodes if they go down, etc.


From general-return-2236-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 11:41:16 2010
Return-Path: <general-return-2236-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 2982 invoked from network); 22 Oct 2010 11:41:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 11:41:16 -0000
Received: (qmail 87244 invoked by uid 500); 22 Oct 2010 11:41:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87054 invoked by uid 500); 22 Oct 2010 11:41:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87046 invoked by uid 99); 22 Oct 2010 11:41:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 11:41:13 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 11:41:04 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 1B9241BA78E
	for <general@hadoop.apache.org>; Fri, 22 Oct 2010 12:40:43 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id csccMdV7x6Mv for <general@hadoop.apache.org>;
	Fri, 22 Oct 2010 12:40:42 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by colossus.hpl.hp.com (Postfix) with ESMTPS id 7F5141BA5F4
	for <general@hadoop.apache.org>; Fri, 22 Oct 2010 12:40:42 +0100 (BST)
MailScanner-NULL-Check: 1288352428.33402@jU+9iWBP8MBFkmRrAaAsuA
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o9MBeRCv024581
	for <general@hadoop.apache.org>; Fri, 22 Oct 2010 12:40:28 +0100 (BST)
Message-ID: <4CC1782B.8040504@apache.org>
Date: Fri, 22 Oct 2010 12:40:27 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.12) Gecko/20100915 Thunderbird/3.0.8
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: bringing the codebases back in line
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com> <D9F0255E-7517-4CDB-9CCD-912B842F9E7C@linkedin.com> <AANLkTimkd_t2k168JpejruPHVDRSBhE1XzPsEtV9rq3M@mail.gmail.com> <20101022001057.GA2710@tp>
In-Reply-To: <20101022001057.GA2710@tp>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o9MBeRCv024581
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 22/10/10 01:10, Konstantin Boudnik wrote:
> On Thu, Oct 21, 2010 at 05:53PM, Ian Holsman wrote:
>> In discussing it with people, I've heard that a major issue (not the only
>> one i'm sure) is lack of resources to actually test the apache releases on
>> large clusters, and that it is very hard getting this done in short cycles
>> (hence the large gap between 20.x and 21).
>
> I do agree the lack of resources for testing Hadoop is a problem. However,
> there might be some slight difference in the meaning of word 'resources' ;)
>
> The only way, IMO, to have a reasonable testing done on a system as complex as
> Hadoop is to invest into automatic validation of builds at system level. This
> requires a few things (resources, if you will):
>    - extra hardware (the easiest and cheapest problem)
>    - automatic deployment, testing, and analysis
>    - system tests development which able to control and observe a cluster
>      behavior (in other words something more sophisticated than just shell
>      scripts)
>
> And for the semi-adequate system testing you don't need a large cluster: 10-20
> nodes will be sufficient in most cases. But the automation of all the
> processes starting from deployment is the key. Testing automation is in a
> little better shape for Hadoop has that system test framework called Herriot
> (part of Hadoop code base for about 7 months now), but it still needs further
> extending.
>

+1 for testing, I would like to help with this, but my test stuff 
depends on my lifecycle stuff which I need to sit down, sync up with 
trunk and work out how to get in.

One thing you can do in a virtual world which you can't do in the 
physical space is reconfigure the LAN on the fly, to see what happens. 
For example, I could set up VLANs of two racks and a switch between 
them, then partition the two  and see what happens -while a simulated 
external load (separate issue) hits the NN with the same amount of 
traffic. Fun things.

From general-return-2237-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 11:46:17 2010
Return-Path: <general-return-2237-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4550 invoked from network); 22 Oct 2010 11:46:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 11:46:17 -0000
Received: (qmail 92200 invoked by uid 500); 22 Oct 2010 11:46:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92034 invoked by uid 500); 22 Oct 2010 11:46:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 92026 invoked by uid 99); 22 Oct 2010 11:46:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 11:46:15 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 11:46:05 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 35E5FB7D4C
	for <general@hadoop.apache.org>; Fri, 22 Oct 2010 12:45:43 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id OHu0A+Tzdcvx for <general@hadoop.apache.org>;
	Fri, 22 Oct 2010 12:45:43 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 6427DB7D41
	for <general@hadoop.apache.org>; Fri, 22 Oct 2010 12:45:43 +0100 (BST)
MailScanner-NULL-Check: 1288352729.29833@yBON/f2Gs1qdpZqMPyvT8g
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o9MBjSQm024768
	for <general@hadoop.apache.org>; Fri, 22 Oct 2010 12:45:28 +0100 (BST)
Message-ID: <4CC17958.3090301@apache.org>
Date: Fri, 22 Oct 2010 12:45:28 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.12) Gecko/20100915 Thunderbird/3.0.8
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: bringing the codebases back in line
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com> <4CC0BC80.6050303@apache.org> <211EF221-CFC5-42A5-99AD-339F120333D1@apache.org> <AANLkTim1ygQZEMhCpGRG0u86kDf0xRr8QWAuo=8TtLqa@mail.gmail.com> <E398F71E-867A-4682-A736-A3CB2FE05B05@yahoo-inc.com>
In-Reply-To: <E398F71E-867A-4682-A736-A3CB2FE05B05@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o9MBjSQm024768
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 22/10/10 01:42, Milind A Bhandarkar wrote:
>
>>
>> but the other question I have which hopefully you guys can answer is does
>> the yahoo distribution have ALL the patches from the trunk on it? because if
>> it doesn't I think that is problematic as well for other reasons.
>
>
> What are these "other" reasons ?
>
> yahoo distribution runs on our production clusters, and I do not see why any production cluster should run code from trunk.

Transient virtual clusters where the FS only exists for a couple of 
hours can do this, but big live physical ones shouldn't, too much data 
at risk. So it depends on your view of "production". If it is "someone 
wants a cluster for 3h to analyse their nightly logs", then you can get 
away with it -it's the best testing you can do.

One problem I hit doing this is that if you do upgrade every week, if 
the app starts failing, you can waste a lot of time trying to decide 
whether its the app code that's changed or trunk, and then if its trunk, 
whether that's a regression or a change that's caused a bug in the app 
code to surface. To be really strict, you should A/B test your virtual 
clusters on the stable and trunk versions, see which finishes faster 
-and whether there are any differences in the output.


that would be very slick for testing indeed.



From general-return-2238-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 11:48:42 2010
Return-Path: <general-return-2238-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4953 invoked from network); 22 Oct 2010 11:48:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 11:48:42 -0000
Received: (qmail 93983 invoked by uid 500); 22 Oct 2010 11:48:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93743 invoked by uid 500); 22 Oct 2010 11:48:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93735 invoked by uid 99); 22 Oct 2010 11:48:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 11:48:40 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 11:48:31 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 8515E1BA614
	for <general@hadoop.apache.org>; Fri, 22 Oct 2010 12:48:10 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id p008M62uC7Dr for <general@hadoop.apache.org>;
	Fri, 22 Oct 2010 12:48:10 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by colossus.hpl.hp.com (Postfix) with ESMTPS id 09E751BA59D
	for <general@hadoop.apache.org>; Fri, 22 Oct 2010 12:48:09 +0100 (BST)
MailScanner-NULL-Check: 1288352874.42009@8PA3pykAuWszfpbKWB4KNg
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o9MBlrF3024906
	for <general@hadoop.apache.org>; Fri, 22 Oct 2010 12:47:53 +0100 (BST)
Message-ID: <4CC179E9.4050307@apache.org>
Date: Fri, 22 Oct 2010 12:47:53 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.12) Gecko/20100915 Thunderbird/3.0.8
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: bringing the codebases back in line
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>	<A2ED1DE3-B131-4D83-AE0A-382F6A64C06F@apache.org>	<AANLkTimnqhcG_nazMjSjwvXnvw_udDXJkFq3VbNwXZda@mail.gmail.com>	<4CC0BEED.2020200@yahoo-inc.com>	<AANLkTi=WF85J47-JcRghD7AzeaA7qZmScGVQt2fn2NHV@mail.gmail.com>	<06380AD8-CC00-4983-A93D-7DE01EE60753@yahoo-inc.com> <AANLkTik79syC6croM4Mw2ir6VJPeJXMTgb9wa-t5QfBw@mail.gmail.com>
In-Reply-To: <AANLkTik79syC6croM4Mw2ir6VJPeJXMTgb9wa-t5QfBw@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o9MBlrF3024906
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 22/10/10 01:51, Eli Collins wrote:

>
> There are some challenges in contributing packaging up stream:
> - Packaging is typically versioned independently from the project code
> (we share packaging code across project major versions). This is why
> most projects have a separate repository for the packaging, and the
> packaging is done by a distribution.

which is why I proposed a downstream integration project

> - The packaging source shares code across the 10 or so components,
> which is useful since we continuously make the packaging more
> consistent, so hosting the code on any single project repository
> doesn't fit well.

same reason

> - The packaging is Linux specific, we've gotten push back when trying
> to contribute modifications upstream with Linuxisms since Apache
> supports non-Linux platforms (namely Solaris).

I think the apple and oracle Java issues may allow those issues to be 
re-evaluated.

Again, something in the ASF that's downstream -and which then does the 
testing- would be good.

One thing people don't realise is how much effort goes into good RPM 
packaging -Eli does- but others may not. It is hard work to get right, 
very hard to test things like RPM upgrades of an existing installation, etc.

-steve

From general-return-2239-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 16:06:42 2010
Return-Path: <general-return-2239-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22397 invoked from network); 22 Oct 2010 16:06:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 16:06:42 -0000
Received: (qmail 70018 invoked by uid 500); 22 Oct 2010 16:06:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69958 invoked by uid 500); 22 Oct 2010 16:06:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69950 invoked by uid 99); 22 Oct 2010 16:06:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 16:06:39 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 16:06:32 +0000
Received: by pzk1 with SMTP id 1so271769pzk.35
        for <general@hadoop.apache.org>; Fri, 22 Oct 2010 09:06:10 -0700 (PDT)
Received: by 10.103.165.7 with SMTP id s7mr3738086muo.51.1287763568551;
        Fri, 22 Oct 2010 09:06:08 -0700 (PDT)
Received: from tp (c-24-4-185-157.hsd1.ca.comcast.net [24.4.185.157])
        by mx.google.com with ESMTPS id c25sm1525075fac.33.2010.10.22.09.06.06
        (version=SSLv3 cipher=RC4-MD5);
        Fri, 22 Oct 2010 09:06:06 -0700 (PDT)
Sender: Konstantin Boudnik <cos@boudnik.org>
Date: Fri, 22 Oct 2010 09:06:02 -0700
From: Konstantin Boudnik <cos@apache.org>
To: general@hadoop.apache.org
Subject: Re: [DISCUSS] Proposed bylaws for Hadoop
Message-ID: <20101022160602.GA17626@tp>
References: <63A1C7A9-727B-4492-9998-01F6B6958EDD@apache.org>
 <4CBE11C5.301@apache.org>
 <CE8E008F-D08E-4564-80F0-B0326246B768@apache.org>
 <580DC943-D250-405D-A524-82452CBC4E03@mac.com>
 <AANLkTikR5uY0_zsT_owo-90yivuVE0PMUPFoEOx3U_8H@mail.gmail.com>
 <AANLkTinE2TzQ88NBDrGUxfNbPu3Q-0k1fR-MbRT6xvd-@mail.gmail.com>
 <B5B20A56-E6EC-4E3D-8C61-F6EA7ED8A6BB@mac.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="bp/iNruPH9dso1Pn"
Content-Disposition: inline
In-Reply-To: <B5B20A56-E6EC-4E3D-8C61-F6EA7ED8A6BB@mac.com>
X-Organization: It's something of 'Cos
X-PGP-Key: http://www.boudnik.org/~cos/pubkey.asc
User-Agent: Mutt/1.5.20 (2009-06-14)
X-Virus-Checked: Checked by ClamAV on apache.org

--bp/iNruPH9dso1Pn
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

+1 on supermajority - enough "people democracy".

On Thu, Oct 21, 2010 at 11:39PM, Nigel Daley wrote:
>=20
> On Oct 21, 2010, at 10:10 AM, Tom White wrote:
> > On Wed, Oct 20, 2010 at 3:15 PM, Chris Douglas <cdouglas@apache.org> wr=
ote:
>=20
> >>> I suggest we make this stronger.
> >>> By way of comparison, the recently enacted bylaws for Pig
> >>> (http://pig.apache.org/bylaws.html) have consensus, for example.
> >>=20
> >> Consensus is equivalent to making the PMC a permanent appointment.
> >> Discussions would be more civil and more likely to offer compromise-
> >> we would actually work toward consensus- if intransigence and
> >> hostility had consequences. Removing people is a last resort. Moving
> >> the barrier closer doesn't make it more likely, but it aligns
> >> incentives so rational people will reign in their more intemperate
> >> comments and, instead, find ways to make progress.
> >>=20
> >> Many projects require supermajorities or even 3/4 majorities. It's
> >> sufficiently damning if more than half the PMC thinks an individual is
> >> so destructive that less damage would be done by ejecting them, in my
> >> opinion, but again: this is not a facility we should use, or have to
> >> use. Throwing someone out is an expensive failure for the project, and
> >> an conspicuous failure of its governance and community. Personally, I
> >> don't care if it's a majority or a supermajority, but consensus is a
> >> fig leaf.
> >=20
> > Fair point. For a large PMC, that's probably true. This matter was
> > discussed on the Pig list, and there was a range of opinion there:
> > http://search-hadoop.com/m/jk35b2tZCuy&subj=3DRE+DISCUSS+Apache+Pig+byl=
aws.
> > Personally, I think a supermajority strikes the right balance.
> >=20
> > Tom
>=20
> Yup, all good points Chris. I, too, agree that supermajority strikes a go=
od balance.
>=20
> Nige

--bp/iNruPH9dso1Pn
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEARECAAYFAkzBtmoACgkQIg9pgB8n5iKTfQCffNTTmE30oRzr3p8puuPmmTl2
jUQAmwStgdGgdTVyv4ivVuwijh47C80/
=vzWn
-----END PGP SIGNATURE-----

--bp/iNruPH9dso1Pn--

From general-return-2240-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 16:37:35 2010
Return-Path: <general-return-2240-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31756 invoked from network); 22 Oct 2010 16:37:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 16:37:34 -0000
Received: (qmail 17903 invoked by uid 500); 22 Oct 2010 16:37:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17854 invoked by uid 500); 22 Oct 2010 16:37:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17846 invoked by uid 99); 22 Oct 2010 16:37:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 16:37:32 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 16:37:24 +0000
Received: by pzk1 with SMTP id 1so293993pzk.35
        for <general@hadoop.apache.org>; Fri, 22 Oct 2010 09:37:02 -0700 (PDT)
Received: by 10.103.108.12 with SMTP id k12mr988716mum.68.1287765420639;
        Fri, 22 Oct 2010 09:37:00 -0700 (PDT)
Received: from tp (c-24-4-185-157.hsd1.ca.comcast.net [24.4.185.157])
        by mx.google.com with ESMTPS id a10sm1547878fak.3.2010.10.22.09.36.59
        (version=SSLv3 cipher=RC4-MD5);
        Fri, 22 Oct 2010 09:36:59 -0700 (PDT)
Sender: Konstantin Boudnik <cos@boudnik.org>
Date: Fri, 22 Oct 2010 09:36:55 -0700
From: Konstantin Boudnik <cos@apache.org>
To: general@hadoop.apache.org
Subject: Re: bringing the codebases back in line
Message-ID: <20101022163655.GB17626@tp>
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>
 <D9F0255E-7517-4CDB-9CCD-912B842F9E7C@linkedin.com>
 <AANLkTimkd_t2k168JpejruPHVDRSBhE1XzPsEtV9rq3M@mail.gmail.com>
 <20101022001057.GA2710@tp>
 <4CC1782B.8040504@apache.org>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="wq9mPyueHGvFACwf"
Content-Disposition: inline
In-Reply-To: <4CC1782B.8040504@apache.org>
X-Organization: It's something of 'Cos
X-PGP-Key: http://www.boudnik.org/~cos/pubkey.asc
User-Agent: Mutt/1.5.20 (2009-06-14)
X-Virus-Checked: Checked by ClamAV on apache.org

--wq9mPyueHGvFACwf
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Fri, Oct 22, 2010 at 12:40PM, Steve Loughran wrote:
> On 22/10/10 01:10, Konstantin Boudnik wrote:
> >
> >The only way, IMO, to have a reasonable testing done on a system as comp=
lex as
> >Hadoop is to invest into automatic validation of builds at system level.=
 This
> >requires a few things (resources, if you will):
> >   - extra hardware (the easiest and cheapest problem)
> >   - automatic deployment, testing, and analysis
> >   - system tests development which able to control and observe a cluster
> >     behavior (in other words something more sophisticated than just she=
ll
> >     scripts)
> >
> +1 for testing, I would like to help with this, but my test stuff
> depends on my lifecycle stuff which I need to sit down, sync up with
> trunk and work out how to get in.
>=20
> One thing you can do in a virtual world which you can't do in the
> physical space is reconfigure the LAN on the fly, to see what
> happens. For example, I could set up VLANs of two racks and a switch
> between them, then partition the two  and see what happens -while a
> simulated external load (separate issue) hits the NN with the same
> amount of traffic. Fun things.

Awesome idea! I guess it is well aligned with Herriot's abilities to do fau=
lt
injections on real (or virual) hardware.


--wq9mPyueHGvFACwf
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEARECAAYFAkzBvacACgkQIg9pgB8n5iI9VwCgjnCcLKEoYZsxZL58Qd12hw4n
pFcAniea7mi3PML8vV22KeNp4/RvPkAT
=O+VV
-----END PGP SIGNATURE-----

--wq9mPyueHGvFACwf--

From general-return-2241-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 17:36:59 2010
Return-Path: <general-return-2241-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58419 invoked from network); 22 Oct 2010 17:36:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 17:36:58 -0000
Received: (qmail 19005 invoked by uid 500); 22 Oct 2010 17:36:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18941 invoked by uid 500); 22 Oct 2010 17:36:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18933 invoked by uid 99); 22 Oct 2010 17:36:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 17:36:56 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of shv.hadoop@gmail.com designates 209.85.210.48 as permitted sender)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 17:36:52 +0000
Received: by pzk1 with SMTP id 1so327737pzk.35
        for <general@hadoop.apache.org>; Fri, 22 Oct 2010 10:36:31 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=3G0aEj+A++bYSssJIo3sHz3IERZ54asgdc1uULRJUpo=;
        b=dNnOVHNQEbHiIt/yJAUhNEktU/3c7rUila9BSELSXWxlIOVHg/Vl7OROQqFsbnktg1
         YSwmjh+MpHpBCsP4orx2ju5+msOtLxHpZDFUF+hW2DNYIbdMDU6e9XzNvEAiUJbIDDjc
         TUyJZVuttSQgP9RvMrtrgJ+9fowKXBDxX37cE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=ViWPqh6sns6FGhQ3MvvwuWSPI/vLZmOKPsShBv168HZkBMHRAuyfeyAwo61tS0FFcp
         tt+1Dg61RQxRd8dFscTMvWgg1U77ra7R2JL2y8XbtSAMuk79gCG6U3Zcz3jN6uVT6Ej6
         bfCwOkLL9qwkxezZZR0Tt2Hz1jX3HqYi0eM20=
MIME-Version: 1.0
Received: by 10.142.245.11 with SMTP id s11mr2175231wfh.392.1287768991855;
 Fri, 22 Oct 2010 10:36:31 -0700 (PDT)
Received: by 10.142.88.15 with HTTP; Fri, 22 Oct 2010 10:36:31 -0700 (PDT)
In-Reply-To: <D3904CD6-27C9-44B8-B526-27A16717E5F5@yahoo-inc.com>
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>
	<4CC0BC80.6050303@apache.org>
	<211EF221-CFC5-42A5-99AD-339F120333D1@apache.org>
	<AANLkTim1ygQZEMhCpGRG0u86kDf0xRr8QWAuo=8TtLqa@mail.gmail.com>
	<E398F71E-867A-4682-A736-A3CB2FE05B05@yahoo-inc.com>
	<AANLkTi=JFzv0gyOg6pzk7be+KEBjNGO+JQZRbqwunSvh@mail.gmail.com>
	<D3904CD6-27C9-44B8-B526-27A16717E5F5@yahoo-inc.com>
Date: Fri, 22 Oct 2010 10:36:31 -0700
Message-ID: <AANLkTinuCh5iKHzMpaGp+i6W4wqUNBfgcJqJh0bMhN-H@mail.gmail.com>
Subject: Re: bringing the codebases back in line
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd171f24ff46f049338156a

--000e0cd171f24ff46f049338156a
Content-Type: text/plain; charset=ISO-8859-1

Milind's point is valid, the PMC cannot demand or control what Yahoo,
Facebook, et. al. run in their productions, or what Couldera sells to their
customers  AS  LONG  AS  it is within the Apache licensing requirements.

What Apache Hadoop can and should provide is a *steady* stream of base
A-releases.

I think that a single fact that we missed to release Hadoop 0.21 late last
year got us into the state we are in now. As it let different Hadoop
installations to diverge drastically from each other, whether it was based
on production or commercial reasons.

Now that we have that, it would not be feasible or worthwhile to find the
common denominator based on the old 0.20 version, unless we want to spend
another year looking for it and diverging the individual installations even
more in the process.

So the question imo is not "how we merge the cloudera and yahoo
distributions", but when/how do we make the new 0.22 release.
And how do we provide a steady release cycle after that.

--Konstantin

On Thu, Oct 21, 2010 at 9:29 PM, Milind A Bhandarkar
<milindb@yahoo-inc.com>wrote:

> >>
> > right.. the trunk is not for production use.  I wasn't suggesting that.
>
> So, what are you suggesting ? That Yahoo distribution of Hadoop should
> *not* be the version we run on our production clusters ?
>
> >
> > but the trunk is what will eventually become the next release.
>
> >
> > Then someone in yahoo will have to decide if they are going to move to
> > rebase their production cluster to 0.21, or just continue back-porting
> what
> > they need to the version they are running on their clusters.
>
> Yes, that is what we do now. If there are committed patches in trunk that
> do not scale for our needs, or break existing applications, or are deemed
> not worth the efforts needed to backport, we do not include them in our
> deployments, and therefore do not include in Yahoo distribution.
>
> >
> > and if yahoo fixes a bug in their version, it would need to be
> > forward-ported over to the current trunk. which will get harder and
> harder
> > as the paths diverge.
>
> Yes, indeed. So, care must be taken that paths do not diverge too much. I
> have seen some cases where the bug fixes did not need to be forward ported,
> because that piece of code was completely re-written in trunk.
>
> >
> > I'm sure you've seen it happen on other projects when a major branch
> lands
> > on the trunk, and the amount of effort it takes to reconcile them.
>
> Yes. And that results in delayed releases. An unexpected benefit for
> application developers was that they could spend time adding features to
> their applications, rather than porting same applications from
> release-to-release, and validating releases. So, it's not always bad.
>
> - Milind
>
>
> --
> Milind Bhandarkar
> (mailto:milindb@yahoo-inc.com)
> (phone: 408-203-5213 W)
>
>
>

--000e0cd171f24ff46f049338156a--

From general-return-2242-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 17:49:31 2010
Return-Path: <general-return-2242-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 64017 invoked from network); 22 Oct 2010 17:49:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 17:49:30 -0000
Received: (qmail 36508 invoked by uid 500); 22 Oct 2010 17:49:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36357 invoked by uid 500); 22 Oct 2010 17:49:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36349 invoked by uid 99); 22 Oct 2010 17:49:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 17:49:28 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 17:49:22 +0000
Received: from sp1-ex07cas01.ds.corp.yahoo.com (sp1-ex07cas01.ds.corp.yahoo.com [216.252.116.137])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o9MHlm3x047771
	for <general@hadoop.apache.org>; Fri, 22 Oct 2010 10:47:48 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1287769668;
	bh=4Tw7BtLpf/yCTI6Yj0+gf/gjnRpb61pMxvMXzCbPDWA=;
	h=From:To:Date:Subject:Message-ID:References:In-Reply-To:
	 Content-Type:Content-Transfer-Encoding:MIME-Version;
	b=j6DN2lQWwD5hQeDTyd27IBZfI9D0lFepz81A2OmNAu5k0syx1DdNFJ3cBtN45JEL/
	 wvAs87hliuN6zTHUYPEeDQTrEzzeTat8I1EJwvKXQDz/n0E53lDmDcZzgErcYcqOMq
	 51ltJfTgfwfxTb38IO0Fz0/pdoqgCLIsNQlL+Rig=
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas01.ds.corp.yahoo.com ([216.252.116.137]) with mapi; Fri, 22 Oct
 2010 10:47:48 -0700
From: Milind A Bhandarkar <milindb@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 22 Oct 2010 10:47:48 -0700
Subject: Re: bringing the codebases back in line
Thread-Topic: bringing the codebases back in line
Thread-Index: ActyET3B84mdT6OASLOKHiXYDoGRUA==
Message-ID: <3235F7D9-AD2C-4508-B921-D1CE527D3535@yahoo-inc.com>
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>
 <4CC0BC80.6050303@apache.org>
 <211EF221-CFC5-42A5-99AD-339F120333D1@apache.org>
 <AANLkTim1ygQZEMhCpGRG0u86kDf0xRr8QWAuo=8TtLqa@mail.gmail.com>
 <E398F71E-867A-4682-A736-A3CB2FE05B05@yahoo-inc.com>
 <AANLkTi=JFzv0gyOg6pzk7be+KEBjNGO+JQZRbqwunSvh@mail.gmail.com>
 <D3904CD6-27C9-44B8-B526-27A16717E5F5@yahoo-inc.com>
 <AANLkTinuCh5iKHzMpaGp+i6W4wqUNBfgcJqJh0bMhN-H@mail.gmail.com>
In-Reply-To: <AANLkTinuCh5iKHzMpaGp+i6W4wqUNBfgcJqJh0bMhN-H@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

+1

On Oct 22, 2010, at 10:36 AM, Konstantin Shvachko wrote:

> So the question imo is not "how we merge the cloudera and yahoo
> distributions", but when/how do we make the new 0.22 release.
> And how do we provide a steady release cycle after that.


--
Milind Bhandarkar
(mailto:milindb@yahoo-inc.com)
(phone: 408-203-5213 W)



From general-return-2243-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 20:07:29 2010
Return-Path: <general-return-2243-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16498 invoked from network); 22 Oct 2010 20:07:29 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 20:07:29 -0000
Received: (qmail 89222 invoked by uid 500); 22 Oct 2010 20:07:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89105 invoked by uid 500); 22 Oct 2010 20:07:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89092 invoked by uid 99); 22 Oct 2010 20:07:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 20:07:27 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 20:07:22 +0000
Received: from oceanfarearth-lm.corp.yahoo.com (oceanfarearth-lm.corp.yahoo.com [10.72.113.156])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o9MK6TEM097572
	for <general@hadoop.apache.org>; Fri, 22 Oct 2010 13:06:29 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1287777989;
	bh=qmXeWYNjlTWTlUfx1NvIfjyLwfqK0/m61ev1pWfIEl4=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=J3vuEUh/DGGPbrea9uKUtcsPAj/u6uXHZ1JZ1Z26ZeLQMwvA2JqgI6JD3ZQwN084K
	 pPLUL1rlM8TQeJlN1BxCENoC2logZkVJ90ZJWF+IBRPqk4MdR+f0Bj5vsw4gSCqc3Z
	 gEw0BKKDTTUnuSdjqcDlSqsXtVvhyIoCka5ZvQjo=
Message-Id: <6ADD878D-CEEC-4A88-87D2-717D134F8E91@yahoo-inc.com>
From: Sanjay Radia <sradia@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <AANLkTinuCh5iKHzMpaGp+i6W4wqUNBfgcJqJh0bMhN-H@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: bringing the codebases back in line
Date: Fri, 22 Oct 2010 13:06:29 -0700
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com> <4CC0BC80.6050303@apache.org> <211EF221-CFC5-42A5-99AD-339F120333D1@apache.org> <AANLkTim1ygQZEMhCpGRG0u86kDf0xRr8QWAuo=8TtLqa@mail.gmail.com> <E398F71E-867A-4682-A736-A3CB2FE05B05@yahoo-inc.com> <AANLkTi=JFzv0gyOg6pzk7be+KEBjNGO+JQZRbqwunSvh@mail.gmail.com> <D3904CD6-27C9-44B8-B526-27A16717E5F5@yahoo-inc.com> <AANLkTinuCh5iKHzMpaGp+i6W4wqUNBfgcJqJh0bMhN-H@mail.gmail.com>
X-Mailer: Apple Mail (2.936)


On Oct 22, 2010, at 10:36 AM, Konstantin Shvachko wrote:

> Milind's point is valid, the PMC cannot demand or control what Yahoo,
> Facebook, et. al. run in their productions, or what Couldera sells  
> to their
> customers  AS  LONG  AS  it is within the Apache licensing  
> requirements.
>
> What Apache Hadoop can and should provide is a *steady* stream of base
> A-releases.
>
> I think that a single fact that we missed to release Hadoop 0.21  
> late last
> year got us into the state we are in now. As it let different Hadoop
> installations to diverge drastically from each other, whether it was  
> based
> on production or commercial reasons.
>
> Now that we have that, it would not be feasible or worthwhile to  
> find the
> common denominator based on the old 0.20 version, unless we want to  
> spend
> another year looking for it and diverging the individual  
> installations even
> more in the process.
>
> So the question imo is not "how we merge the cloudera and yahoo
> distributions", but when/how do we make the new 0.22 release.
> And how do we provide a steady release cycle after that.

+1

sanjay


>
> --Konstantin
>
> On Thu, Oct 21, 2010 at 9:29 PM, Milind A Bhandarkar
> <milindb@yahoo-inc.com>wrote:
>
>>>>
>>> right.. the trunk is not for production use.  I wasn't suggesting  
>>> that.
>>
>> So, what are you suggesting ? That Yahoo distribution of Hadoop  
>> should
>> *not* be the version we run on our production clusters ?
>>
>>>
>>> but the trunk is what will eventually become the next release.
>>
>>>
>>> Then someone in yahoo will have to decide if they are going to  
>>> move to
>>> rebase their production cluster to 0.21, or just continue back- 
>>> porting
>> what
>>> they need to the version they are running on their clusters.
>>
>> Yes, that is what we do now. If there are committed patches in  
>> trunk that
>> do not scale for our needs, or break existing applications, or are  
>> deemed
>> not worth the efforts needed to backport, we do not include them in  
>> our
>> deployments, and therefore do not include in Yahoo distribution.
>>
>>>
>>> and if yahoo fixes a bug in their version, it would need to be
>>> forward-ported over to the current trunk. which will get harder and
>> harder
>>> as the paths diverge.
>>
>> Yes, indeed. So, care must be taken that paths do not diverge too  
>> much. I
>> have seen some cases where the bug fixes did not need to be forward  
>> ported,
>> because that piece of code was completely re-written in trunk.
>>
>>>
>>> I'm sure you've seen it happen on other projects when a major branch
>> lands
>>> on the trunk, and the amount of effort it takes to reconcile them.
>>
>> Yes. And that results in delayed releases. An unexpected benefit for
>> application developers was that they could spend time adding  
>> features to
>> their applications, rather than porting same applications from
>> release-to-release, and validating releases. So, it's not always bad.
>>
>> - Milind
>>
>>
>> --
>> Milind Bhandarkar
>> (mailto:milindb@yahoo-inc.com)
>> (phone: 408-203-5213 W)
>>
>>
>>


From general-return-2244-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 21:52:28 2010
Return-Path: <general-return-2244-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44665 invoked from network); 22 Oct 2010 21:52:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 21:52:28 -0000
Received: (qmail 26151 invoked by uid 500); 22 Oct 2010 21:52:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26113 invoked by uid 500); 22 Oct 2010 21:52:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26105 invoked by uid 99); 22 Oct 2010 21:52:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 21:52:26 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 21:52:22 +0000
Received: by gwb15 with SMTP id 15so1675719gwb.35
        for <general@hadoop.apache.org>; Fri, 22 Oct 2010 14:52:01 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.231.3 with SMTP id jo3mr2784777qcb.21.1287784321143; Fri,
 22 Oct 2010 14:52:01 -0700 (PDT)
Received: by 10.229.237.131 with HTTP; Fri, 22 Oct 2010 14:52:01 -0700 (PDT)
In-Reply-To: <AANLkTi=JFzv0gyOg6pzk7be+KEBjNGO+JQZRbqwunSvh@mail.gmail.com>
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>
	<4CC0BC80.6050303@apache.org>
	<211EF221-CFC5-42A5-99AD-339F120333D1@apache.org>
	<AANLkTim1ygQZEMhCpGRG0u86kDf0xRr8QWAuo=8TtLqa@mail.gmail.com>
	<E398F71E-867A-4682-A736-A3CB2FE05B05@yahoo-inc.com>
	<AANLkTi=JFzv0gyOg6pzk7be+KEBjNGO+JQZRbqwunSvh@mail.gmail.com>
Date: Fri, 22 Oct 2010 14:52:01 -0700
Message-ID: <AANLkTi=MMxb=z9K9x-kmUgwixX=5_z5tQpmWBedngvYq@mail.gmail.com>
Subject: Re: bringing the codebases back in line
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Thu, Oct 21, 2010 at 5:54 PM, Ian Holsman <hadoop@holsman.net> wrote:
> On Thu, Oct 21, 2010 at 8:42 PM, Milind A Bhandarkar
> <milindb@yahoo-inc.com>wrote:
>
>>
>> >
>> > but the other question I have which hopefully you guys can answer is d=
oes
>> > the yahoo distribution have ALL the patches from the trunk on it? beca=
use
>> if
>> > it doesn't I think that is problematic as well for other reasons.
>>
>>
>> What are these "other" reasons ?
>>
>
> yahoo distribution runs on our production clusters, and I do not see why =
any
>> production cluster should run code from trunk.
>>
>>
> right.. the trunk is not for production use. =A0I wasn't suggesting that.
>
> but the trunk is what will eventually become the next release.
>
> Then someone in yahoo will have to decide if they are going to move to
> rebase their production cluster to 0.21, or just continue back-porting wh=
at
> they need to the version they are running on their clusters.
>
> and if yahoo fixes a bug in their version, it would need to be
> forward-ported over to the current trunk. which will get harder and harde=
r
> as the paths diverge.
>
> I'm sure you've seen it happen on other projects when a major branch land=
s
> on the trunk, and the amount of effort it takes to reconcile them.

Hey Ian,

I think we're all in agreement that we need to re-convene on a common
branch that removes most of the deltas against an Apache release that
we have all accumulated (primarily security, append, trunk backports).
The open question is whether we try to come up with a common 20-based
branch or wait for 22.  That's been previously discussed on this list
and there were some concerns, re-iterated on this thread, that we
should invest in 22 rather than the current 20-based branches.

Our current plan is to reconvene with everyone on 22 - a well-tested
release with security and append should get users off the current
20-based branches. However if you and/or the Apache community feels
that there needs to be an Apache 20-based branch and release that
reflects what people are using (security, append, various backports in
YDH/CDH) we are willing to create and maintain this branch on Apache.

Thanks,
Eli




> =A0- Milind
>
> --
> Milind Bhandarkar
> (mailto:milindb@yahoo-inc.com)
> (phone: 408-203-5213 W)
>

From general-return-2245-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 22:04:35 2010
Return-Path: <general-return-2245-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47439 invoked from network); 22 Oct 2010 22:04:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 22:04:35 -0000
Received: (qmail 46748 invoked by uid 500); 22 Oct 2010 22:04:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46688 invoked by uid 500); 22 Oct 2010 22:04:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46680 invoked by uid 99); 22 Oct 2010 22:04:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 22:04:34 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 22:04:27 +0000
Received: by gyg4 with SMTP id 4so1174654gyg.35
        for <general@hadoop.apache.org>; Fri, 22 Oct 2010 15:04:06 -0700 (PDT)
Received: by 10.229.14.143 with SMTP id g15mr2763291qca.208.1287785045302;
        Fri, 22 Oct 2010 15:04:05 -0700 (PDT)
Received: from tp (c-24-4-185-157.hsd1.ca.comcast.net [24.4.185.157])
        by mx.google.com with ESMTPS id m7sm3091442qck.1.2010.10.22.15.04.03
        (version=SSLv3 cipher=RC4-MD5);
        Fri, 22 Oct 2010 15:04:04 -0700 (PDT)
Sender: Konstantin Boudnik <cos@boudnik.org>
Date: Fri, 22 Oct 2010 15:03:59 -0700
From: Konstantin Boudnik <cos@apache.org>
To: general@hadoop.apache.org
Subject: Re: bringing the codebases back in line
Message-ID: <20101022220359.GE17626@tp>
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>
 <4CC0BC80.6050303@apache.org>
 <211EF221-CFC5-42A5-99AD-339F120333D1@apache.org>
 <AANLkTim1ygQZEMhCpGRG0u86kDf0xRr8QWAuo=8TtLqa@mail.gmail.com>
 <E398F71E-867A-4682-A736-A3CB2FE05B05@yahoo-inc.com>
 <AANLkTi=JFzv0gyOg6pzk7be+KEBjNGO+JQZRbqwunSvh@mail.gmail.com>
 <AANLkTi=MMxb=z9K9x-kmUgwixX=5_z5tQpmWBedngvYq@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="oJ71EGRlYNjSvfq7"
Content-Disposition: inline
In-Reply-To: <AANLkTi=MMxb=z9K9x-kmUgwixX=5_z5tQpmWBedngvYq@mail.gmail.com>
X-Organization: It's something of 'Cos
X-PGP-Key: http://www.boudnik.org/~cos/pubkey.asc
User-Agent: Mutt/1.5.20 (2009-06-14)
X-Virus-Checked: Checked by ClamAV on apache.org

--oJ71EGRlYNjSvfq7
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

+1 on moving forward to common 0.22 trunk. 0.20 was dragging on for quite a
long time and, in a sense, create certain imbalance toward 0.20-centric
Hadoop environment

On Fri, Oct 22, 2010 at 02:52PM, Eli Collins wrote:
> On Thu, Oct 21, 2010 at 5:54 PM, Ian Holsman <hadoop@holsman.net> wrote:
> > On Thu, Oct 21, 2010 at 8:42 PM, Milind A Bhandarkar
> > <milindb@yahoo-inc.com>wrote:
> >
> >> > but the other question I have which hopefully you guys can answer is=
 does
> >> > the yahoo distribution have ALL the patches from the trunk on it? be=
cause
> >> if
> >> > it doesn't I think that is problematic as well for other reasons.
> >>
> >> What are these "other" reasons ?
> >
> > yahoo distribution runs on our production clusters, and I do not see wh=
y any
> >> production cluster should run code from trunk.
> >>
> >>
> > right.. the trunk is not for production use. =E2=95=90I wasn't suggesti=
ng that.
> >
> > but the trunk is what will eventually become the next release.
> >
> > Then someone in yahoo will have to decide if they are going to move to
> > rebase their production cluster to 0.21, or just continue back-porting =
what
> > they need to the version they are running on their clusters.
> >
> > and if yahoo fixes a bug in their version, it would need to be
> > forward-ported over to the current trunk. which will get harder and har=
der
> > as the paths diverge.
> >
> > I'm sure you've seen it happen on other projects when a major branch la=
nds
> > on the trunk, and the amount of effort it takes to reconcile them.
>=20
> Hey Ian,
>=20
> I think we're all in agreement that we need to re-convene on a common
> branch that removes most of the deltas against an Apache release that
> we have all accumulated (primarily security, append, trunk backports).
> The open question is whether we try to come up with a common 20-based
> branch or wait for 22.  That's been previously discussed on this list
> and there were some concerns, re-iterated on this thread, that we
> should invest in 22 rather than the current 20-based branches.
>=20
> Our current plan is to reconvene with everyone on 22 - a well-tested
> release with security and append should get users off the current
> 20-based branches. However if you and/or the Apache community feels
> that there needs to be an Apache 20-based branch and release that
> reflects what people are using (security, append, various backports in
> YDH/CDH) we are willing to create and maintain this branch on Apache.
>=20
> Thanks,
> Eli
>=20
>=20
>=20
>=20
> > =E2=95=90- Milind
> >
> > --
> > Milind Bhandarkar
> > (mailto:milindb@yahoo-inc.com)
> > (phone: 408-203-5213 W)
> >

--oJ71EGRlYNjSvfq7
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEARECAAYFAkzCCk8ACgkQIg9pgB8n5iLT2gCfXD4uGlM6Eeq9kHX6F8cTd/qQ
A4AAnRUQy3aV8EMMK30NIjMDoSYk9mTv
=lLfH
-----END PGP SIGNATURE-----

--oJ71EGRlYNjSvfq7--

From general-return-2246-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 22:24:03 2010
Return-Path: <general-return-2246-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55180 invoked from network); 22 Oct 2010 22:24:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 22:24:03 -0000
Received: (qmail 62620 invoked by uid 500); 22 Oct 2010 22:24:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62511 invoked by uid 500); 22 Oct 2010 22:24:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62499 invoked by uid 99); 22 Oct 2010 22:24:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 22:24:01 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 22:23:54 +0000
Received: from SP2-EX07CAS03.ds.corp.yahoo.com (sp2-ex07cas03.corp.sp2.yahoo.com [98.137.59.35])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o9MMMvt7038983
	for <general@hadoop.apache.org>; Fri, 22 Oct 2010 15:22:57 -0700 (PDT)
Received: from SP2-EX07VS04.ds.corp.yahoo.com ([98.137.59.33]) by
 SP2-EX07CAS03.ds.corp.yahoo.com ([98.137.59.35]) with mapi; Fri, 22 Oct 2010
 15:22:57 -0700
From: Mahadev Konar <mahadev@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 22 Oct 2010 15:22:56 -0700
Subject: Re: bringing the codebases back in line
Thread-Topic: bringing the codebases back in line
Thread-Index: ActyNTAFC388RUo6R+SRWpOXnzISIAAAnzko
Message-ID: <C8E75CD0.44AD5%mahadev@yahoo-inc.com>
In-Reply-To: <20101022220359.GE17626@tp>
Accept-Language: en, en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en, en-US
Content-Type: text/plain; charset="big5"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

KzEgZm9yIG1vdmluZyB0byAwLjIyIHRydW5rLg0KDQpUaGFua3MNCm1haGFkZXYNCg0KDQpPbiAx
MC8yMi8xMCAzOjAzIFBNLCAiS29uc3RhbnRpbiBCb3VkbmlrIiA8Y29zQGFwYWNoZS5vcmc+IHdy
b3RlOg0KDQo+ICsxIG9uIG1vdmluZyBmb3J3YXJkIHRvIGNvbW1vbiAwLjIyIHRydW5rLiAwLjIw
IHdhcyBkcmFnZ2luZyBvbiBmb3IgcXVpdGUgYQ0KPiBsb25nIHRpbWUgYW5kLCBpbiBhIHNlbnNl
LCBjcmVhdGUgY2VydGFpbiBpbWJhbGFuY2UgdG93YXJkIDAuMjAtY2VudHJpYw0KPiBIYWRvb3Ag
ZW52aXJvbm1lbnQNCj4gDQo+IE9uIEZyaSwgT2N0IDIyLCAyMDEwIGF0IDAyOjUyUE0sIEVsaSBD
b2xsaW5zIHdyb3RlOg0KPj4gT24gVGh1LCBPY3QgMjEsIDIwMTAgYXQgNTo1NCBQTSwgSWFuIEhv
bHNtYW4gPGhhZG9vcEBob2xzbWFuLm5ldD4gd3JvdGU6DQo+Pj4gT24gVGh1LCBPY3QgMjEsIDIw
MTAgYXQgODo0MiBQTSwgTWlsaW5kIEEgQmhhbmRhcmthcg0KPj4+IDxtaWxpbmRiQHlhaG9vLWlu
Yy5jb20+d3JvdGU6DQo+Pj4gDQo+Pj4+PiBidXQgdGhlIG90aGVyIHF1ZXN0aW9uIEkgaGF2ZSB3
aGljaCBob3BlZnVsbHkgeW91IGd1eXMgY2FuIGFuc3dlciBpcyBkb2VzDQo+Pj4+PiB0aGUgeWFo
b28gZGlzdHJpYnV0aW9uIGhhdmUgQUxMIHRoZSBwYXRjaGVzIGZyb20gdGhlIHRydW5rIG9uIGl0
PyBiZWNhdXNlDQo+Pj4+IGlmDQo+Pj4+PiBpdCBkb2Vzbid0IEkgdGhpbmsgdGhhdCBpcyBwcm9i
bGVtYXRpYyBhcyB3ZWxsIGZvciBvdGhlciByZWFzb25zLg0KPj4+PiANCj4+Pj4gV2hhdCBhcmUg
dGhlc2UgIm90aGVyIiByZWFzb25zID8NCj4+PiANCj4+PiB5YWhvbyBkaXN0cmlidXRpb24gcnVu
cyBvbiBvdXIgcHJvZHVjdGlvbiBjbHVzdGVycywgYW5kIEkgZG8gbm90IHNlZSB3aHkgYW55DQo+
Pj4+IHByb2R1Y3Rpb24gY2x1c3RlciBzaG91bGQgcnVuIGNvZGUgZnJvbSB0cnVuay4NCj4+Pj4g
DQo+Pj4+IA0KPj4+IHJpZ2h0Li4gdGhlIHRydW5rIGlzIG5vdCBmb3IgcHJvZHVjdGlvbiB1c2Uu
IPn5SSB3YXNuJ3Qgc3VnZ2VzdGluZyB0aGF0Lg0KPj4+IA0KPj4+IGJ1dCB0aGUgdHJ1bmsgaXMg
d2hhdCB3aWxsIGV2ZW50dWFsbHkgYmVjb21lIHRoZSBuZXh0IHJlbGVhc2UuDQo+Pj4gDQo+Pj4g
VGhlbiBzb21lb25lIGluIHlhaG9vIHdpbGwgaGF2ZSB0byBkZWNpZGUgaWYgdGhleSBhcmUgZ29p
bmcgdG8gbW92ZSB0bw0KPj4+IHJlYmFzZSB0aGVpciBwcm9kdWN0aW9uIGNsdXN0ZXIgdG8gMC4y
MSwgb3IganVzdCBjb250aW51ZSBiYWNrLXBvcnRpbmcgd2hhdA0KPj4+IHRoZXkgbmVlZCB0byB0
aGUgdmVyc2lvbiB0aGV5IGFyZSBydW5uaW5nIG9uIHRoZWlyIGNsdXN0ZXJzLg0KPj4+IA0KPj4+
IGFuZCBpZiB5YWhvbyBmaXhlcyBhIGJ1ZyBpbiB0aGVpciB2ZXJzaW9uLCBpdCB3b3VsZCBuZWVk
IHRvIGJlDQo+Pj4gZm9yd2FyZC1wb3J0ZWQgb3ZlciB0byB0aGUgY3VycmVudCB0cnVuay4gd2hp
Y2ggd2lsbCBnZXQgaGFyZGVyIGFuZCBoYXJkZXINCj4+PiBhcyB0aGUgcGF0aHMgZGl2ZXJnZS4N
Cj4+PiANCj4+PiBJJ20gc3VyZSB5b3UndmUgc2VlbiBpdCBoYXBwZW4gb24gb3RoZXIgcHJvamVj
dHMgd2hlbiBhIG1ham9yIGJyYW5jaCBsYW5kcw0KPj4+IG9uIHRoZSB0cnVuaywgYW5kIHRoZSBh
bW91bnQgb2YgZWZmb3J0IGl0IHRha2VzIHRvIHJlY29uY2lsZSB0aGVtLg0KPj4gDQo+PiBIZXkg
SWFuLA0KPj4gDQo+PiBJIHRoaW5rIHdlJ3JlIGFsbCBpbiBhZ3JlZW1lbnQgdGhhdCB3ZSBuZWVk
IHRvIHJlLWNvbnZlbmUgb24gYSBjb21tb24NCj4+IGJyYW5jaCB0aGF0IHJlbW92ZXMgbW9zdCBv
ZiB0aGUgZGVsdGFzIGFnYWluc3QgYW4gQXBhY2hlIHJlbGVhc2UgdGhhdA0KPj4gd2UgaGF2ZSBh
bGwgYWNjdW11bGF0ZWQgKHByaW1hcmlseSBzZWN1cml0eSwgYXBwZW5kLCB0cnVuayBiYWNrcG9y
dHMpLg0KPj4gVGhlIG9wZW4gcXVlc3Rpb24gaXMgd2hldGhlciB3ZSB0cnkgdG8gY29tZSB1cCB3
aXRoIGEgY29tbW9uIDIwLWJhc2VkDQo+PiBicmFuY2ggb3Igd2FpdCBmb3IgMjIuICBUaGF0J3Mg
YmVlbiBwcmV2aW91c2x5IGRpc2N1c3NlZCBvbiB0aGlzIGxpc3QNCj4+IGFuZCB0aGVyZSB3ZXJl
IHNvbWUgY29uY2VybnMsIHJlLWl0ZXJhdGVkIG9uIHRoaXMgdGhyZWFkLCB0aGF0IHdlDQo+PiBz
aG91bGQgaW52ZXN0IGluIDIyIHJhdGhlciB0aGFuIHRoZSBjdXJyZW50IDIwLWJhc2VkIGJyYW5j
aGVzLg0KPj4gDQo+PiBPdXIgY3VycmVudCBwbGFuIGlzIHRvIHJlY29udmVuZSB3aXRoIGV2ZXJ5
b25lIG9uIDIyIC0gYSB3ZWxsLXRlc3RlZA0KPj4gcmVsZWFzZSB3aXRoIHNlY3VyaXR5IGFuZCBh
cHBlbmQgc2hvdWxkIGdldCB1c2VycyBvZmYgdGhlIGN1cnJlbnQNCj4+IDIwLWJhc2VkIGJyYW5j
aGVzLiBIb3dldmVyIGlmIHlvdSBhbmQvb3IgdGhlIEFwYWNoZSBjb21tdW5pdHkgZmVlbHMNCj4+
IHRoYXQgdGhlcmUgbmVlZHMgdG8gYmUgYW4gQXBhY2hlIDIwLWJhc2VkIGJyYW5jaCBhbmQgcmVs
ZWFzZSB0aGF0DQo+PiByZWZsZWN0cyB3aGF0IHBlb3BsZSBhcmUgdXNpbmcgKHNlY3VyaXR5LCBh
cHBlbmQsIHZhcmlvdXMgYmFja3BvcnRzIGluDQo+PiBZREgvQ0RIKSB3ZSBhcmUgd2lsbGluZyB0
byBjcmVhdGUgYW5kIG1haW50YWluIHRoaXMgYnJhbmNoIG9uIEFwYWNoZS4NCj4+IA0KPj4gVGhh
bmtzLA0KPj4gRWxpDQo+PiANCj4+IA0KPj4gDQo+PiANCj4+PiD5+S0gTWlsaW5kDQo+Pj4gDQo+
Pj4gLS0NCj4+PiBNaWxpbmQgQmhhbmRhcmthcg0KPj4+IChtYWlsdG86bWlsaW5kYkB5YWhvby1p
bmMuY29tKQ0KPj4+IChwaG9uZTogNDA4LTIwMy01MjEzIFcpDQo+Pj4gDQoNCg==

From general-return-2247-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 23 01:33:57 2010
Return-Path: <general-return-2247-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23377 invoked from network); 23 Oct 2010 01:33:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Oct 2010 01:33:57 -0000
Received: (qmail 70212 invoked by uid 500); 23 Oct 2010 01:33:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70087 invoked by uid 500); 23 Oct 2010 01:33:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70079 invoked by uid 99); 23 Oct 2010 01:33:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 01:33:55 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 01:33:49 +0000
Received: by wyg36 with SMTP id 36so1538298wyg.35
        for <general@hadoop.apache.org>; Fri, 22 Oct 2010 18:33:28 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.227.128.7 with SMTP id i7mr3619430wbs.165.1287797608063; Fri,
 22 Oct 2010 18:33:28 -0700 (PDT)
Received: by 10.216.87.78 with HTTP; Fri, 22 Oct 2010 18:33:27 -0700 (PDT)
In-Reply-To: <C8E75CD0.44AD5%mahadev@yahoo-inc.com>
References: <20101022220359.GE17626@tp>
	<C8E75CD0.44AD5%mahadev@yahoo-inc.com>
Date: Fri, 22 Oct 2010 21:33:27 -0400
Message-ID: <AANLkTik8WhrSLMw5K+C=643ek0_pORYhU+0C+dZe0CcN@mail.gmail.com>
Subject: Re: bringing the codebases back in line
From: Ian Holsman <hadoop@holsman.net>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016367fa506f8a20d04933ebe42
X-Virus-Checked: Checked by ClamAV on apache.org

--0016367fa506f8a20d04933ebe42
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

I think we should push forward to 0.22 as well.
The question then becomes what should be in it.

2010/10/22 Mahadev Konar <mahadev@yahoo-inc.com>

> +1 for moving to 0.22 trunk.
>
> Thanks
> mahadev
>
>
> On 10/22/10 3:03 PM, "Konstantin Boudnik" <cos@apache.org> wrote:
>
> > +1 on moving forward to common 0.22 trunk. 0.20 was dragging on for qui=
te
> a
> > long time and, in a sense, create certain imbalance toward 0.20-centric
> > Hadoop environment
> >
> > On Fri, Oct 22, 2010 at 02:52PM, Eli Collins wrote:
> >> On Thu, Oct 21, 2010 at 5:54 PM, Ian Holsman <hadoop@holsman.net>
> wrote:
> >>> On Thu, Oct 21, 2010 at 8:42 PM, Milind A Bhandarkar
> >>> <milindb@yahoo-inc.com>wrote:
> >>>
> >>>>> but the other question I have which hopefully you guys can answer i=
s
> does
> >>>>> the yahoo distribution have ALL the patches from the trunk on it?
> because
> >>>> if
> >>>>> it doesn't I think that is problematic as well for other reasons.
> >>>>
> >>>> What are these "other" reasons ?
> >>>
> >>> yahoo distribution runs on our production clusters, and I do not see
> why any
> >>>> production cluster should run code from trunk.
> >>>>
> >>>>
> >>> right.. the trunk is not for production use. =E2=95=90I wasn't sugges=
ting that.
> >>>
> >>> but the trunk is what will eventually become the next release.
> >>>
> >>> Then someone in yahoo will have to decide if they are going to move t=
o
> >>> rebase their production cluster to 0.21, or just continue back-portin=
g
> what
> >>> they need to the version they are running on their clusters.
> >>>
> >>> and if yahoo fixes a bug in their version, it would need to be
> >>> forward-ported over to the current trunk. which will get harder and
> harder
> >>> as the paths diverge.
> >>>
> >>> I'm sure you've seen it happen on other projects when a major branch
> lands
> >>> on the trunk, and the amount of effort it takes to reconcile them.
> >>
> >> Hey Ian,
> >>
> >> I think we're all in agreement that we need to re-convene on a common
> >> branch that removes most of the deltas against an Apache release that
> >> we have all accumulated (primarily security, append, trunk backports).
> >> The open question is whether we try to come up with a common 20-based
> >> branch or wait for 22.  That's been previously discussed on this list
> >> and there were some concerns, re-iterated on this thread, that we
> >> should invest in 22 rather than the current 20-based branches.
> >>
> >> Our current plan is to reconvene with everyone on 22 - a well-tested
> >> release with security and append should get users off the current
> >> 20-based branches. However if you and/or the Apache community feels
> >> that there needs to be an Apache 20-based branch and release that
> >> reflects what people are using (security, append, various backports in
> >> YDH/CDH) we are willing to create and maintain this branch on Apache.
> >>
> >> Thanks,
> >> Eli
> >>
> >>
> >>
> >>
> >>> =E2=95=90- Milind
> >>>
> >>> --
> >>> Milind Bhandarkar
> >>> (mailto:milindb@yahoo-inc.com)
> >>> (phone: 408-203-5213 W)
> >>>
>
>

--0016367fa506f8a20d04933ebe42--

From general-return-2248-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 23 03:05:02 2010
Return-Path: <general-return-2248-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43303 invoked from network); 23 Oct 2010 03:05:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Oct 2010 03:05:02 -0000
Received: (qmail 11033 invoked by uid 500); 23 Oct 2010 03:05:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 10603 invoked by uid 500); 23 Oct 2010 03:04:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 10592 invoked by uid 99); 23 Oct 2010 03:04:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 03:04:58 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 03:04:50 +0000
Received: from SP1-EX07CAS02.ds.corp.yahoo.com (sp1-ex07cas02.ds.corp.yahoo.com [216.252.116.138])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id o9N34HZo070213
	for <general@hadoop.apache.org>; Fri, 22 Oct 2010 20:04:17 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1287803057;
	bh=nIaTCs5Ldru6huF73/uUzEPS3tK3p6JpNUsF3EcchNo=;
	h=From:To:Date:Subject:Message-ID:References:In-Reply-To:
	 Content-Type:Content-Transfer-Encoding:MIME-Version;
	b=ACozKiqLJiey6r/qjSLP4ewZLKMag+/yjGMkUhn1iqZ233gbT+2CNbMvyO5nh+COF
	 XKXzVYgD/mH8pcbXeLVSbZaRqxVnXc2fQu+f3L8zOxkDYlVtUxJzpAGSXO8AH7VPeD
	 TvxFldDCfVkS2oCJRbOoquC0S71LdEHu4MFUYkQY=
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 SP1-EX07CAS02.ds.corp.yahoo.com ([216.252.116.167]) with mapi; Fri, 22 Oct
 2010 20:04:17 -0700
From: Milind A Bhandarkar <milindb@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 22 Oct 2010 20:04:15 -0700
Subject: Re: bringing the codebases back in line
Thread-Topic: bringing the codebases back in line
Thread-Index: ActyXvpWvOXCIfb4T7iXFR8nO3fr8g==
Message-ID: <F93FD995-5ADF-4442-A976-857FBBCB91F8@yahoo-inc.com>
References: <20101022220359.GE17626@tp>
 <C8E75CD0.44AD5%mahadev@yahoo-inc.com>
 <AANLkTik8WhrSLMw5K+C=643ek0_pORYhU+0C+dZe0CcN@mail.gmail.com>
In-Reply-To: <AANLkTik8WhrSLMw5K+C=643ek0_pORYhU+0C+dZe0CcN@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


On Oct 22, 2010, at 6:33 PM, Ian Holsman wrote:

> I think we should push forward to 0.22 as well.

"As Well" ? That means there is something else you want to do, right ?

What is it ?

You have said in earlier emails that "Yahoo distribution of hadoop not bein=
g the same as apache hadoop trunk will cause 'other' problems".

Let me ask a yes/no question, based on some of your ambiguous statements in=
 this thread.

Do you want Yahoo! distribution of Hadoop the same as trunk ?

- milind


--
Milind Bhandarkar
(mailto:milindb@yahoo-inc.com)
(phone: 408-203-5213 W)



From general-return-2249-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 23 03:40:44 2010
Return-Path: <general-return-2249-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 52446 invoked from network); 23 Oct 2010 03:40:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Oct 2010 03:40:43 -0000
Received: (qmail 24093 invoked by uid 500); 23 Oct 2010 03:40:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23870 invoked by uid 500); 23 Oct 2010 03:40:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23862 invoked by uid 99); 23 Oct 2010 03:40:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 03:40:41 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.62.80] (HELO qmta08.westchester.pa.mail.comcast.net) (76.96.62.80)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 03:40:31 +0000
Received: from omta16.westchester.pa.mail.comcast.net ([76.96.62.88])
	by qmta08.westchester.pa.mail.comcast.net with comcast
	id N3bW1f0041uE5Es583gBAV; Sat, 23 Oct 2010 03:40:11 +0000
Received: from [10.0.0.13] ([98.234.216.58])
	by omta16.westchester.pa.mail.comcast.net with comcast
	id N3gA1f0011GAoR63c3gBR8; Sat, 23 Oct 2010 03:40:11 +0000
Message-Id: <79F839A8-B258-467A-8B57-B7F4D09A4FC9@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <AANLkTik8WhrSLMw5K+C=643ek0_pORYhU+0C+dZe0CcN@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: bringing the codebases back in line
Date: Fri, 22 Oct 2010 20:40:08 -0700
References: <20101022220359.GE17626@tp> <C8E75CD0.44AD5%mahadev@yahoo-inc.com> <AANLkTik8WhrSLMw5K+C=643ek0_pORYhU+0C+dZe0CcN@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


On Oct 22, 2010, at 6:33 PM, Ian Holsman wrote:

> I think we should push forward to 0.22 as well.
> The question then becomes what should be in it.

I've had better luck with time-based releases rather than feature- 
based releases for open source projects. The current plan of record is  
to make cut a branch next month, stabilize it, and release it.

-- Owen

From general-return-2250-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 23 08:49:58 2010
Return-Path: <general-return-2250-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27784 invoked from network); 23 Oct 2010 08:49:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Oct 2010 08:49:58 -0000
Received: (qmail 32315 invoked by uid 500); 23 Oct 2010 08:49:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32095 invoked by uid 500); 23 Oct 2010 08:49:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32078 invoked by uid 99); 23 Oct 2010 08:49:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 08:49:54 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [81.169.146.161] (HELO mo-p00-ob.rzone.de) (81.169.146.161)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 08:48:47 +0000
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; t=1287823706; l=1367;
	s=domk; d=brainlounge.de;
	h=Content-Transfer-Encoding:Content-Type:In-Reply-To:References:
	Subject:To:MIME-Version:From:Date:X-RZG-CLASS-ID:X-RZG-AUTH;
	bh=VLDpqHQtmmAywYl6UiW0BfrWWmo=;
	b=FOgGkHJSTJRMBntNmv1Q5rBb/Qyk8EuUEknNrS0EwoUuskS1yRrDAVRIqg04jTo5uWA
	wVacgIGUSdVIGqjLhCjwHBz6QlxkYRtRO+IUdoeGf0rz0fDh2Lq5h64MOqcBkNkd9uAGA
	HCTnWa+RZMY71VZeeIfADOnOQiarmccxqNs=
X-RZG-AUTH: :LmkWe0TmffCene0uRkJY6AqDWBjdcSs8EVOKK4t9JJpSMjf2AKg+TcRtFaNcoY+nN7sOdsXpAGK2qzNLPg==
X-RZG-CLASS-ID: mo00
Received: from vitamin-2.local (e180157201.adsl.alicedsl.de [85.180.157.201])
	by post.strato.de (mrclete mo45) (RZmta 23.5)
	with ESMTP id 603997m9N4CTyW for <general@hadoop.apache.org>;
	Sat, 23 Oct 2010 10:48:26 +0200 (MEST)
Message-ID: <4CC2A158.6050202@brainlounge.de>
Date: Sat, 23 Oct 2010 10:48:24 +0200
From: Bernd Fondermann <bf_jak@brainlounge.de>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.5) Gecko/20091204 Thunderbird/3.0
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: bringing the codebases back in line
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>	<4CC0BC80.6050303@apache.org>	<211EF221-CFC5-42A5-99AD-339F120333D1@apache.org>	<AANLkTim1ygQZEMhCpGRG0u86kDf0xRr8QWAuo=8TtLqa@mail.gmail.com>	<E398F71E-867A-4682-A736-A3CB2FE05B05@yahoo-inc.com>	<AANLkTi=JFzv0gyOg6pzk7be+KEBjNGO+JQZRbqwunSvh@mail.gmail.com> <AANLkTi=MMxb=z9K9x-kmUgwixX=5_z5tQpmWBedngvYq@mail.gmail.com>
In-Reply-To: <AANLkTi=MMxb=z9K9x-kmUgwixX=5_z5tQpmWBedngvYq@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 22.10.10 23:52, Eli Collins wrote:
> On Thu, Oct 21, 2010 at 5:54 PM, Ian Holsman<hadoop@holsman.net>  wrote:
> Hey Ian,
>
> I think we're all in agreement that we need to re-convene on a common
> branch that removes most of the deltas against an Apache release that
> we have all accumulated (primarily security, append, trunk backports).
> The open question is whether we try to come up with a common 20-based
> branch or wait for 22.  That's been previously discussed on this list
> and there were some concerns, re-iterated on this thread, that we
> should invest in 22 rather than the current 20-based branches.
>
> Our current plan is to reconvene with everyone on 22 - a well-tested
> release with security and append should get users off the current
> 20-based branches. However if you and/or the Apache community feels
> that there needs to be an Apache 20-based branch and release that
> reflects what people are using (security, append, various backports in
> YDH/CDH) we are willing to create and maintain this branch on Apache.
>
> Thanks,
> Eli

Eli,

You are using "we" and "our" placeholders a lot in the last two 
paragraphs and my brain's parser is failing to resolve them correctly - 
please can you clarify where you speak of "we" the community/the 
PMC/Cloudera/Yahoo/somebody else respectively.

Thank you,

   Bernd

From general-return-2251-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 23 09:25:23 2010
Return-Path: <general-return-2251-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35778 invoked from network); 23 Oct 2010 09:25:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Oct 2010 09:25:23 -0000
Received: (qmail 46667 invoked by uid 500); 23 Oct 2010 09:25:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46232 invoked by uid 500); 23 Oct 2010 09:25:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46224 invoked by uid 99); 23 Oct 2010 09:25:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 09:25:18 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [81.169.146.162] (HELO mo-p00-ob.rzone.de) (81.169.146.162)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 09:25:12 +0000
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; t=1287825889; l=1627;
	s=domk; d=brainlounge.de;
	h=Content-Transfer-Encoding:Content-Type:In-Reply-To:References:
	Subject:To:MIME-Version:From:Date:X-RZG-CLASS-ID:X-RZG-AUTH;
	bh=AbPjpMdRbOsbHJ+ihDXV60cl5RQ=;
	b=FsgiC//9CwfEZZfxYsHBdXs9xbZ1C0FQGsyriJZJ4T4nVdHpThIE1ptEf8qsYfUKSGM
	FF/kD9a5E/mcOalQ3J+Y3URfECfqsTOkx08FxPixVcM0pUCaEFbrfETYZvIgGUI46xbKU
	tPma8h6Xf2wRh8yP1MEwR4tbryUCSsx1zW4=
X-RZG-AUTH: :LmkWe0TmffCene0uRkJY6AqDWBjdcSs8EVOKK4t9JJpSMjf2AKg+TcRtFaNcoY+nN7sOdsXpAGK2qzNLPg==
X-RZG-CLASS-ID: mo00
Received: from vitamin-2.local (e180157201.adsl.alicedsl.de [85.180.157.201])
	by post.strato.de (mrclete mo9) (RZmta 23.5)
	with ESMTP id m04cb1m9N56wae for <general@hadoop.apache.org>;
	Sat, 23 Oct 2010 11:24:49 +0200 (MEST)
Message-ID: <4CC2A9E1.1020909@brainlounge.de>
Date: Sat, 23 Oct 2010 11:24:49 +0200
From: Bernd Fondermann <bf_jak@brainlounge.de>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.5) Gecko/20091204 Thunderbird/3.0
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: bringing the codebases back in line
References: <20101022220359.GE17626@tp> <C8E75CD0.44AD5%mahadev@yahoo-inc.com> <AANLkTik8WhrSLMw5K+C=643ek0_pORYhU+0C+dZe0CcN@mail.gmail.com> <F93FD995-5ADF-4442-A976-857FBBCB91F8@yahoo-inc.com>
In-Reply-To: <F93FD995-5ADF-4442-A976-857FBBCB91F8@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 23.10.10 05:04, Milind A Bhandarkar wrote:
>
> On Oct 22, 2010, at 6:33 PM, Ian Holsman wrote:
>
>> I think we should push forward to 0.22 as well.
>
> "As Well" ? That means there is something else you want to do, right ?
>
> What is it ?
>
> You have said in earlier emails that "Yahoo distribution of hadoop not being the same as apache hadoop trunk will cause 'other' problems".
>
> Let me ask a yes/no question, based on some of your ambiguous statements in this thread.
>
> Do you want Yahoo! distribution of Hadoop the same as trunk ?

(You didn't ask me but FWIW) I don't think the Hadoop community can 
mandadate or even should care what a company will put in their 
downstream distributions. After all, this is Apache-licensed code.

Individuals who work on Hadoop and commercial derivatives of Hadoop at 
the same time might be in a different position, but that's basically 
their problem.

Side note: Here at Apache it is common to prefix a statement with "with 
my PMC hat on...", "with my board hat on...", "with my $EMPLOYER hat on...".

So, with my ASF member hat on, I'd say that the Hadoop project is only 
tasked with working on Apache premises (Apache mailing lists, svn, 
wikis, etc). Currently, this community is by far too much concerned with 
what downstream projects are doing with the code. Reason: Currently 
Hadoop looks like a downstream project from others. This is very very 
bad and has to be turned around.
How to get there?
1. Firmly put your Apache contributor head on
2. Integrate all the pending code committers are willing to contribute
3. Release.

   Bernd

From general-return-2252-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 23 09:28:36 2010
Return-Path: <general-return-2252-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35943 invoked from network); 23 Oct 2010 09:28:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Oct 2010 09:28:36 -0000
Received: (qmail 47906 invoked by uid 500); 23 Oct 2010 09:28:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47837 invoked by uid 500); 23 Oct 2010 09:28:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47829 invoked by uid 99); 23 Oct 2010 09:28:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 09:28:34 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [81.169.146.162] (HELO mo-p00-ob.rzone.de) (81.169.146.162)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 09:28:27 +0000
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; t=1287826086; l=272;
	s=domk; d=brainlounge.de;
	h=Content-Transfer-Encoding:Content-Type:In-Reply-To:References:
	Subject:To:MIME-Version:From:Date:X-RZG-CLASS-ID:X-RZG-AUTH;
	bh=oNpXv2VNHSAFyz06rgM0OJDHBug=;
	b=cyXJqzLee+ACa4/JwjpnT8K6sY/77OUka4KeLehWK8FPxVSvXW7az0Z5iV+elieqzf6
	Q4sn3EygSyLLDewLeN4lHsqRs10x2xIvmwgbv6rS5w0d1S8SMK8Ch6/O772JbbCLLE4rM
	ow31mTC31XQRWGi4EC4vj1tPJtbHNR5Gwsc=
X-RZG-AUTH: :LmkWe0TmffCene0uRkJY6AqDWBjdcSs8EVOKK4t9JJpSMjf2AKg+TcRtFaNcoY+nN7sOdsXpAGK2qzNLPg==
X-RZG-CLASS-ID: mo00
Received: from vitamin-2.local (e180157201.adsl.alicedsl.de [85.180.157.201])
	by post.strato.de (klopstock mo41) (RZmta 23.5)
	with ESMTP id i073f9m9N97Y3O for <general@hadoop.apache.org>;
	Sat, 23 Oct 2010 11:28:06 +0200 (MEST)
Message-ID: <4CC2AAA6.8090605@brainlounge.de>
Date: Sat, 23 Oct 2010 11:28:06 +0200
From: Bernd Fondermann <bf_jak@brainlounge.de>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.5) Gecko/20091204 Thunderbird/3.0
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: bringing the codebases back in line
References: <20101022220359.GE17626@tp> <C8E75CD0.44AD5%mahadev@yahoo-inc.com> <AANLkTik8WhrSLMw5K+C=643ek0_pORYhU+0C+dZe0CcN@mail.gmail.com> <79F839A8-B258-467A-8B57-B7F4D09A4FC9@apache.org>
In-Reply-To: <79F839A8-B258-467A-8B57-B7F4D09A4FC9@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 23.10.10 05:40, Owen O'Malley wrote:
>
>  The current plan of record is to make
> cut a branch next month, stabilize it, and release it.

I'd like to revisit the mailing list thread where this decision was 
made. Can you point me to it?

Thanks a lot,

   Bernd

From general-return-2253-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 23 10:45:24 2010
Return-Path: <general-return-2253-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55379 invoked from network); 23 Oct 2010 10:45:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Oct 2010 10:45:23 -0000
Received: (qmail 83478 invoked by uid 500); 23 Oct 2010 10:45:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83204 invoked by uid 500); 23 Oct 2010 10:45:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83196 invoked by uid 99); 23 Oct 2010 10:45:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 10:45:18 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 10:45:13 +0000
Received: by wwi17 with SMTP id 17so1638018wwi.29
        for <general@hadoop.apache.org>; Sat, 23 Oct 2010 03:44:52 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.166.68 with SMTP id f46mr608324wel.26.1287830692230; Sat,
 23 Oct 2010 03:44:52 -0700 (PDT)
Received: by 10.216.87.78 with HTTP; Sat, 23 Oct 2010 03:44:52 -0700 (PDT)
In-Reply-To: <F93FD995-5ADF-4442-A976-857FBBCB91F8@yahoo-inc.com>
References: <20101022220359.GE17626@tp>
	<C8E75CD0.44AD5%mahadev@yahoo-inc.com>
	<AANLkTik8WhrSLMw5K+C=643ek0_pORYhU+0C+dZe0CcN@mail.gmail.com>
	<F93FD995-5ADF-4442-A976-857FBBCB91F8@yahoo-inc.com>
Date: Sat, 23 Oct 2010 06:44:52 -0400
Message-ID: <AANLkTikrikYT8ztQywKqyrsg6OVU1eW=OtXmWi4G8orY@mail.gmail.com>
Subject: Re: bringing the codebases back in line
From: Ian Holsman <hadoop@holsman.net>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f9481ef0f4e00493467218

--001485f9481ef0f4e00493467218
Content-Type: text/plain; charset=ISO-8859-1

On Fri, Oct 22, 2010 at 11:04 PM, Milind A Bhandarkar <milindb@yahoo-inc.com
> wrote:

>
> On Oct 22, 2010, at 6:33 PM, Ian Holsman wrote:
>
> > I think we should push forward to 0.22 as well.
>
> "As Well" ? That means there is something else you want to do, right ?
>

as well as in I agree with the other people who want to have a 0.22 release,
as opposed to wanting to have another 0.20 release.


>
> What is it ?
>
> You have said in earlier emails that "Yahoo distribution of hadoop not
> being the same as apache hadoop trunk will cause 'other' problems".
>

I'm picking on yahoo here, but the same could be said for cloudera as well.

>
> Let me ask a yes/no question, based on some of your ambiguous statements in
> this thread.
>
> Do you want Yahoo! distribution of Hadoop the same as trunk ?
>


I want the Yahoo & Cloudera distributions of hadoop to be as close as
possible to the released version of apache hadoop.

I want Yahoo (and others) to look at the apache release and be able say we
can use this on our own cluster, and not have to maintain their 500 or so
patches on top of the standard release.

I want to get the 0.22 (and future) apache releases to a point where the
internal Yahoo developers start asking themselves if they should switch, and
if there is a need for them to maintain their github release at all.

and like Bernd says, I don't have the power to dictate what Yahoo runs on
their cluster internally, neither do I want it.

As a user I was quite pleased when Yahoo and Cloudera put their versions out
there. It was tremendously helpful to me getting my shit done, but by them
doing so it told me (by the fact that they had to release it, and how
different they were) that I shouldn't be running on a standard apache
release.

To repeat for those who think I write vaguely.
I want to remove the need for multiple distributions.


which back to the original thread:

one approach suggested to resolve the multiple branches is to do releases
frequently, but in order to do that we have need to things in place to help
test the releases quickly so as to ensure the quality is there.



> - milind
>
>
> --
> Milind Bhandarkar
> (mailto:milindb@yahoo-inc.com)
> (phone: 408-203-5213 W)
>
>
>

--001485f9481ef0f4e00493467218--

From general-return-2254-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 23 15:48:30 2010
Return-Path: <general-return-2254-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30179 invoked from network); 23 Oct 2010 15:48:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Oct 2010 15:48:30 -0000
Received: (qmail 67759 invoked by uid 500); 23 Oct 2010 15:48:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67693 invoked by uid 500); 23 Oct 2010 15:48:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67685 invoked by uid 99); 23 Oct 2010 15:48:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 15:48:27 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 15:48:23 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=kOcbAwdkQA9WBRP5IM+59bDxv2oHtmM1It6e78TQO0hJqPlIMQ50Ku2m
   7v9hk0wiknF7WDGBqax9ZWeivJiomNs/mjmdYZfngTNSbKtwKmsyN6p9s
   EpLP2BoSNfctWqY;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1287848902; x=1319384902;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20bringing=20the=20codebases=20back=20in
   =20line|Date:=20Sat,=2023=20Oct=202010=2015:48:02=20+0000
   |Message-ID:=20<65B0683A-127A-475D-B532-E14582A617F7@link
   edin.com>|To:=20"<general@hadoop.apache.org>"=20<general@
   hadoop.apache.org>|MIME-Version:=201.0
   |Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<935D9AEBCCA251468BD7BEF017FCBAB1@linkedin
   .com>|In-Reply-To:=20<4CC2AAA6.8090605@brainlounge.de>
   |References:=20<20101022220359.GE17626@tp>=0D=0A=20<C8E75
   CD0.44AD5%mahadev@yahoo-inc.com>=0D=0A=20<AANLkTik8WhrSLM
   w5K+C=3D643ek0_pORYhU+0C+dZe0CcN@mail.gmail.com>=0D=0A=20
   <79F839A8-B258-467A-8B57-B7F4D09A4FC9@apache.org>=0D=0A
   =20<4CC2AAA6.8090605@brainlounge.de>;
  bh=3Cabaw/1TCQCw8WTyttq770/Uf9AcYIPk8nrClHyMg4=;
  b=aW3bsDmJGkt8KUU0JdDCcDNx6gRjL7SCO1xUqOpXV1h00HJOCg+7ulfq
   cWjX6MEchFq8+oM8T2vLMr+5e/q0E1ATnf+FyZAR4KakdjHDc8bkR/pGq
   if351dV/EnPUUN+;
X-IronPort-AV: E=Sophos;i="4.58,228,1286175600"; 
   d="scan'208";a="15936993"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi id 14.01.0218.012; Sat,
 23 Oct 2010 08:48:02 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: bringing the codebases back in line
Thread-Topic: bringing the codebases back in line
Thread-Index: AQHLcVQVuqCpAAO/eUm8zw+IHmCpg5NMbtAAgAAGVQCAABMKAIAADmMAgAADdICAAV9hgIAAA1eAgAAFTACAADU7gIAAI2UAgABhOACAAGomgA==
Date: Sat, 23 Oct 2010 15:48:02 +0000
Message-ID: <65B0683A-127A-475D-B532-E14582A617F7@linkedin.com>
References: <20101022220359.GE17626@tp>
 <C8E75CD0.44AD5%mahadev@yahoo-inc.com>
 <AANLkTik8WhrSLMw5K+C=643ek0_pORYhU+0C+dZe0CcN@mail.gmail.com>
 <79F839A8-B258-467A-8B57-B7F4D09A4FC9@apache.org>
 <4CC2AAA6.8090605@brainlounge.de>
In-Reply-To: <4CC2AAA6.8090605@brainlounge.de>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <935D9AEBCCA251468BD7BEF017FCBAB1@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


On Oct 23, 2010, at 2:28 AM, Bernd Fondermann wrote:

> On 23.10.10 05:40, Owen O'Malley wrote:
>>=20
>> The current plan of record is to make
>> cut a branch next month, stabilize it, and release it.
>=20
> I'd like to revisit the mailing list thread where this decision was made.=
 Can you point me to it?


	While you are looking for threads, can you point me to the one where the A=
SF board announced the very heavy handed VP change to the Hadoop community?=
   =

From general-return-2255-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 23 16:11:18 2010
Return-Path: <general-return-2255-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38596 invoked from network); 23 Oct 2010 16:11:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Oct 2010 16:11:17 -0000
Received: (qmail 91061 invoked by uid 500); 23 Oct 2010 16:11:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91006 invoked by uid 500); 23 Oct 2010 16:11:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90998 invoked by uid 99); 23 Oct 2010 16:11:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 16:11:16 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.62.16] (HELO qmta01.westchester.pa.mail.comcast.net) (76.96.62.16)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 16:11:09 +0000
Received: from omta23.westchester.pa.mail.comcast.net ([76.96.62.74])
	by qmta01.westchester.pa.mail.comcast.net with comcast
	id NDD71f0041c6gX851GApBv; Sat, 23 Oct 2010 16:10:49 +0000
Received: from [10.0.0.13] ([98.234.216.58])
	by omta23.westchester.pa.mail.comcast.net with comcast
	id NGAo1f0011GAoR63jGApoa; Sat, 23 Oct 2010 16:10:49 +0000
Message-Id: <BD6D7067-A95C-4DBE-BDA5-987F220032AC@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <4CC2AAA6.8090605@brainlounge.de>
Content-Type: multipart/alternative; boundary=Apple-Mail-1-305401955
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: bringing the codebases back in line
Date: Sat, 23 Oct 2010 09:10:47 -0700
References: <20101022220359.GE17626@tp> <C8E75CD0.44AD5%mahadev@yahoo-inc.com> <AANLkTik8WhrSLMw5K+C=643ek0_pORYhU+0C+dZe0CcN@mail.gmail.com> <79F839A8-B258-467A-8B57-B7F4D09A4FC9@apache.org> <4CC2AAA6.8090605@brainlounge.de>
X-Mailer: Apple Mail (2.936)

--Apple-Mail-1-305401955
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


On Oct 23, 2010, at 2:28 AM, Bernd Fondermann wrote:

> On 23.10.10 05:40, Owen O'Malley wrote:
>>
>> The current plan of record is to make
>> cut a branch next month, stabilize it, and release it.
>
> I'd like to revisit the mailing list thread where this decision was  
> made. Can you point me to it?

Sure.

http://www.mail-archive.com/common-dev@hadoop.apache.org/msg01388.html

-- Owen
--Apple-Mail-1-305401955--

From general-return-2256-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 23 18:23:58 2010
Return-Path: <general-return-2256-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72320 invoked from network); 23 Oct 2010 18:23:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Oct 2010 18:23:57 -0000
Received: (qmail 64665 invoked by uid 500); 23 Oct 2010 18:23:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 64622 invoked by uid 500); 23 Oct 2010 18:23:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 64614 invoked by uid 99); 23 Oct 2010 18:23:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 18:23:56 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 18:23:49 +0000
Received: from sp1-ex07cas01.ds.corp.yahoo.com (sp1-ex07cas01.ds.corp.yahoo.com [216.252.116.137])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o9NIN3a3013840
	for <general@hadoop.apache.org>; Sat, 23 Oct 2010 11:23:03 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1287858183;
	bh=MidFLmk2BcKaTQbZyLTG5dnLmcDuAb/0bsAavJsva88=;
	h=From:To:Date:Subject:Message-ID:References:In-Reply-To:
	 Content-Type:Content-Transfer-Encoding:MIME-Version;
	b=gkrtOjLUH6PF9HPl/fE36IwtMyr2rCK3zwrhEYGt9+Pt/3HdGjEcHbGnnh2hf1gXR
	 p11S7n13flPfHHWU1f6NStWsPYuVuVxDq2MUZM0MRJpFwC+GDTWodkg4UU0xGe0e2n
	 O7lZ1hrN9qrKISsj5+NAU8wsS1oRoFD64z+PCbQk=
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas01.ds.corp.yahoo.com ([216.252.116.137]) with mapi; Sat, 23 Oct
 2010 11:23:03 -0700
From: Milind A Bhandarkar <milindb@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Sat, 23 Oct 2010 11:23:02 -0700
Subject: Re: bringing the codebases back in line
Thread-Topic: bringing the codebases back in line
Thread-Index: Acty31SDppkCVt2JTNGuZrm2jIAQQg==
Message-ID: <A114213E-2272-4BF2-B3AB-723AD3D55AFE@yahoo-inc.com>
References: <20101022220359.GE17626@tp>
 <C8E75CD0.44AD5%mahadev@yahoo-inc.com>
 <AANLkTik8WhrSLMw5K+C=643ek0_pORYhU+0C+dZe0CcN@mail.gmail.com>
 <F93FD995-5ADF-4442-A976-857FBBCB91F8@yahoo-inc.com>
 <AANLkTikrikYT8ztQywKqyrsg6OVU1eW=OtXmWi4G8orY@mail.gmail.com>
In-Reply-To: <AANLkTikrikYT8ztQywKqyrsg6OVU1eW=OtXmWi4G8orY@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

> as well as in I agree with the other people who want to have a 0.22 relea=
se,
> as opposed to wanting to have another 0.20 release.


+1 !

- Milind

--
Milind Bhandarkar
(mailto:milindb@yahoo-inc.com)
(phone: 408-203-5213 W)



From general-return-2257-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 23 18:26:14 2010
Return-Path: <general-return-2257-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72660 invoked from network); 23 Oct 2010 18:26:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Oct 2010 18:26:13 -0000
Received: (qmail 66675 invoked by uid 500); 23 Oct 2010 18:26:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66598 invoked by uid 500); 23 Oct 2010 18:26:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66590 invoked by uid 99); 23 Oct 2010 18:26:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 18:26:12 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 18:26:04 +0000
Received: from sp1-ex07cas01.ds.corp.yahoo.com (sp1-ex07cas01.ds.corp.yahoo.com [216.252.116.137])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id o9NIPHbN072038
	for <general@hadoop.apache.org>; Sat, 23 Oct 2010 11:25:17 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1287858318;
	bh=g+XMG0LiwwtZIvQsotmBxSiEKUe6hWDZCx6i2XXqFzk=;
	h=From:To:Date:Subject:Message-ID:References:In-Reply-To:
	 Content-Type:Content-Transfer-Encoding:MIME-Version;
	b=HupCLLWawU51MssPMziX8uUcK0sPKE3ZS8abhdsqFoNj1jx0iO6tu/1Bl3FVINPSs
	 x+HXOkOMwPDKBk6nbKtc2GlSA8ux9TXt9Rlhihy42wK5dzSZrGFnCLqmd5O3TKrt4Q
	 A2+ynUmINy646hXyIp7PpoQ3Mlor3x+cuBTHLcjA=
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas01.ds.corp.yahoo.com ([216.252.116.137]) with mapi; Sat, 23 Oct
 2010 11:25:17 -0700
From: Milind A Bhandarkar <milindb@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Sat, 23 Oct 2010 11:25:16 -0700
Subject: Re: bringing the codebases back in line
Thread-Topic: bringing the codebases back in line
Thread-Index: Acty36PrgYZsEPNVSxOxMJs5yFfQ0A==
Message-ID: <76E09703-7610-4A87-9F0A-4770F5027B32@yahoo-inc.com>
References: <20101022220359.GE17626@tp>
 <C8E75CD0.44AD5%mahadev@yahoo-inc.com>
 <AANLkTik8WhrSLMw5K+C=643ek0_pORYhU+0C+dZe0CcN@mail.gmail.com>
 <F93FD995-5ADF-4442-A976-857FBBCB91F8@yahoo-inc.com>
 <4CC2A9E1.1020909@brainlounge.de>
In-Reply-To: <4CC2A9E1.1020909@brainlounge.de>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


>=20
> Side note: Here at Apache it is common to prefix a statement with "with=20
> my PMC hat on...", "with my board hat on...", "with my $EMPLOYER hat on..=
.".


I have my "user of yahoo distribution of hadoop" hat on.

- milind

--
Milind Bhandarkar
(mailto:milindb@yahoo-inc.com)
(phone: 408-203-5213 W)



From general-return-2258-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 23 18:51:02 2010
Return-Path: <general-return-2258-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 75825 invoked from network); 23 Oct 2010 18:51:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Oct 2010 18:51:01 -0000
Received: (qmail 75495 invoked by uid 500); 23 Oct 2010 18:51:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75295 invoked by uid 500); 23 Oct 2010 18:51:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75287 invoked by uid 99); 23 Oct 2010 18:51:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 18:51:00 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 18:50:54 +0000
Received: by qyk34 with SMTP id 34so408049qyk.14
        for <general@hadoop.apache.org>; Sat, 23 Oct 2010 11:50:33 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.223.149 with SMTP id ik21mr3818082qcb.224.1287859832895;
 Sat, 23 Oct 2010 11:50:32 -0700 (PDT)
Received: by 10.229.15.70 with HTTP; Sat, 23 Oct 2010 11:50:32 -0700 (PDT)
In-Reply-To: <4CC2A158.6050202@brainlounge.de>
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>
	<4CC0BC80.6050303@apache.org>
	<211EF221-CFC5-42A5-99AD-339F120333D1@apache.org>
	<AANLkTim1ygQZEMhCpGRG0u86kDf0xRr8QWAuo=8TtLqa@mail.gmail.com>
	<E398F71E-867A-4682-A736-A3CB2FE05B05@yahoo-inc.com>
	<AANLkTi=JFzv0gyOg6pzk7be+KEBjNGO+JQZRbqwunSvh@mail.gmail.com>
	<AANLkTi=MMxb=z9K9x-kmUgwixX=5_z5tQpmWBedngvYq@mail.gmail.com>
	<4CC2A158.6050202@brainlounge.de>
Date: Sat, 23 Oct 2010 11:50:32 -0700
Message-ID: <AANLkTi=aUSSyA7BHRHQFxgDaA_Qv+6bgpHcXPMLjaC8w@mail.gmail.com>
Subject: Re: bringing the codebases back in line
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Sat, Oct 23, 2010 at 1:48 AM, Bernd Fondermann <bf_jak@brainlounge.de> w=
rote:
> On 22.10.10 23:52, Eli Collins wrote:
>>
>> On Thu, Oct 21, 2010 at 5:54 PM, Ian Holsman<hadoop@holsman.net> =A0wrot=
e:
>> Hey Ian,
>>
>> I think we're all in agreement that we need to re-convene on a common
>> branch that removes most of the deltas against an Apache release that
>> we have all accumulated (primarily security, append, trunk backports).
>> The open question is whether we try to come up with a common 20-based
>> branch or wait for 22. =A0That's been previously discussed on this list
>> and there were some concerns, re-iterated on this thread, that we
>> should invest in 22 rather than the current 20-based branches.
>>
>> Our current plan is to reconvene with everyone on 22 - a well-tested
>> release with security and append should get users off the current
>> 20-based branches. However if you and/or the Apache community feels
>> that there needs to be an Apache 20-based branch and release that
>> reflects what people are using (security, append, various backports in
>> YDH/CDH) we are willing to create and maintain this branch on Apache.
>>
>> Thanks,
>> Eli
>
> Eli,
>
> You are using "we" and "our" placeholders a lot in the last two paragraph=
s
> and my brain's parser is failing to resolve them correctly - please can y=
ou
> clarify where you speak of "we" the community/the
> PMC/Cloudera/Yahoo/somebody else respectively.
>
> Thank you,
>
> =A0Bernd
>

Hey Bernd,

Apologies for the confusion, I see your point. This is hopefully more clear=
:

I think the *Hadoop contributors* are all in agreement that we need to
re-convene on a common branch that removes most of the deltas against
an Apache release that we have all accumulated (primarily security,
append, trunk backports). The open question is whether *those of us
that maintain 20-based branches (used internally or distributed)* try
to come up with a common 20-based branch or wait for 22.  That's been
previously discussed on this list and there were some concerns,
re-iterated on this thread, that *Hadoop contributors* should invest
in 22 rather than the current 20-based branches.

*Those of us who maintain Cloudera's 20-based branch* would like to
reconvene with everyone on 22 - a well-tested release from trunk with
security and append should get users off the current 20-based
branches. However if you and/or the Apache community feels that there
needs to be an Apache 20-based branch and release that reflects what
people are using (security, append, various backports in YDH/CDH),
*those of us who maintain Cloudera's 20-based branch* are willing to
create and maintain this branch at Apache.

Thanks,
Eli

From general-return-2259-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 23 19:04:18 2010
Return-Path: <general-return-2259-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77121 invoked from network); 23 Oct 2010 19:04:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Oct 2010 19:04:17 -0000
Received: (qmail 86972 invoked by uid 500); 23 Oct 2010 19:04:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86922 invoked by uid 500); 23 Oct 2010 19:04:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86914 invoked by uid 99); 23 Oct 2010 19:04:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 19:04:16 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 19:04:10 +0000
Received: by qyk4 with SMTP id 4so538187qyk.14
        for <general@hadoop.apache.org>; Sat, 23 Oct 2010 12:03:49 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.86.149 with SMTP id s21mr631817qcl.234.1287860628933; Sat,
 23 Oct 2010 12:03:48 -0700 (PDT)
Received: by 10.229.15.70 with HTTP; Sat, 23 Oct 2010 12:03:48 -0700 (PDT)
In-Reply-To: <79F839A8-B258-467A-8B57-B7F4D09A4FC9@apache.org>
References: <20101022220359.GE17626@tp>
	<C8E75CD0.44AD5%mahadev@yahoo-inc.com>
	<AANLkTik8WhrSLMw5K+C=643ek0_pORYhU+0C+dZe0CcN@mail.gmail.com>
	<79F839A8-B258-467A-8B57-B7F4D09A4FC9@apache.org>
Date: Sat, 23 Oct 2010 12:03:48 -0700
Message-ID: <AANLkTimzkM-0WZT5kNTaEm0YWWb+hokEvUbn_x2_eOJC@mail.gmail.com>
Subject: Re: bringing the codebases back in line
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Fri, Oct 22, 2010 at 8:40 PM, Owen O'Malley <omalley@apache.org> wrote:
>
> On Oct 22, 2010, at 6:33 PM, Ian Holsman wrote:
>
>> I think we should push forward to 0.22 as well.
>> The question then becomes what should be in it.
>
> I've had better luck with time-based releases rather than feature-based
> releases for open source projects. The current plan of record is to make cut
> a branch next month, stabilize it, and release it.
>
> -- Owen

Great.  At the contributor's meetup it sounded like the plan of record
was to hold 22 for HDFS-1052.  Speaking of which does someone have
notes to post to the list/wiki?

It would be great to see the project move to periodic time-based
releases.  In previous discussion it seemed like feature-based
releases were favored:
http://www.mail-archive.com/general@hadoop.apache.org/msg01022.html

Thanks,
Eli

From general-return-2260-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 23 19:21:07 2010
Return-Path: <general-return-2260-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83254 invoked from network); 23 Oct 2010 19:21:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Oct 2010 19:21:07 -0000
Received: (qmail 93852 invoked by uid 500); 23 Oct 2010 19:21:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93790 invoked by uid 500); 23 Oct 2010 19:21:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93782 invoked by uid 99); 23 Oct 2010 19:21:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 19:21:05 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [81.169.146.160] (HELO mo-p00-ob.rzone.de) (81.169.146.160)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 19:20:59 +0000
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; t=1287861636; l=1262;
	s=domk; d=brainlounge.de;
	h=Content-Transfer-Encoding:Content-Type:In-Reply-To:References:
	Subject:To:MIME-Version:From:Date:X-RZG-CLASS-ID:X-RZG-AUTH;
	bh=Y/T026B2HseCLFQw2Dq5LqL3CNQ=;
	b=WSMycfLFoWoN7K18gsfyCIal+LcTTRI8tfAirUASuH9jt2JOuPs5MGh+oP+EPiR4oJo
	DdtjO/W3D9GOrY41xRLgSjHEtYNwiHHqRMpwmSQuZwX91zfUEwGJa4Fkg9w2wixRLX9Dq
	r27EZrUsv2TPDTHfpoL6cb/QBJ1NN88b21Y=
X-RZG-AUTH: :LmkWe0TmffCene0uRkJY6AqDWBjdcSs8EVOKK4t9JJpSMjf2AKg+TcRtFaNcoY+nN7sOdsXpA2aQKnqZIA==
X-RZG-CLASS-ID: mo00
Received: from vitamin-2.local (e180163140.adsl.alicedsl.de [85.180.163.140])
	by post.strato.de (fruni mo1) (RZmta 24.1) with ESMTP id e027ddm9NIhvqz
	for <general@hadoop.apache.org>; Sat, 23 Oct 2010 21:20:35 +0200 (MEST)
Message-ID: <4CC33582.6080400@brainlounge.de>
Date: Sat, 23 Oct 2010 21:20:34 +0200
From: Bernd Fondermann <bf_jak@brainlounge.de>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.5) Gecko/20091204 Thunderbird/3.0
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: bringing the codebases back in line
References: <20101022220359.GE17626@tp> <C8E75CD0.44AD5%mahadev@yahoo-inc.com> <AANLkTik8WhrSLMw5K+C=643ek0_pORYhU+0C+dZe0CcN@mail.gmail.com> <79F839A8-B258-467A-8B57-B7F4D09A4FC9@apache.org> <4CC2AAA6.8090605@brainlounge.de> <65B0683A-127A-475D-B532-E14582A617F7@linkedin.com>
In-Reply-To: <65B0683A-127A-475D-B532-E14582A617F7@linkedin.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 23.10.10 17:48, Allen Wittenauer wrote:
>
> On Oct 23, 2010, at 2:28 AM, Bernd Fondermann wrote:
>
>> On 23.10.10 05:40, Owen O'Malley wrote:
>>>
>>> The current plan of record is to make
>>> cut a branch next month, stabilize it, and release it.
>>
>> I'd like to revisit the mailing list thread where this decision was made. Can you point me to it?
>
>
> 	While you are looking for threads, can you point me to the one where the ASF board announced the very heavy handed VP change to the Hadoop community?

Sure, but you'd have to be a committer at the ASF to have access to it:
Message-ID: <4CBF4DE8.8060907@apache.org>
20.10.10 13:15 -0700 - committers@apache.org:
"ASF Board Meeting Summary - October 20, 2010"

PMC chair changes are never announced to a broader audience, AFAICR.
There are other things which currently trouble the ASF community way 
more than this VP change, as every Apache committer can read there and 
which is subject to a press release for everyone to read.

What I learned when talking to users of ASF projects over the years is 
that they mostly don't even know what PMCs are all about, and thus they 
don't care about VPs.

All they care about is features, releases, documentation and support.

   Bernd

From general-return-2261-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 23 19:25:27 2010
Return-Path: <general-return-2261-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83537 invoked from network); 23 Oct 2010 19:25:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Oct 2010 19:25:27 -0000
Received: (qmail 95311 invoked by uid 500); 23 Oct 2010 19:25:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95264 invoked by uid 500); 23 Oct 2010 19:25:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95256 invoked by uid 99); 23 Oct 2010 19:25:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 19:25:25 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [81.169.146.162] (HELO mo-p00-ob.rzone.de) (81.169.146.162)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 19:25:20 +0000
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; t=1287861899; l=538;
	s=domk; d=brainlounge.de;
	h=Content-Transfer-Encoding:Content-Type:In-Reply-To:References:
	Subject:To:MIME-Version:From:Date:X-RZG-CLASS-ID:X-RZG-AUTH;
	bh=H66JhTQEbjVQb28/aX6KPXpZR+Y=;
	b=ljUaB6dN3V5HsW1phVOXsZnIqnkF7uHpCDcwlifoxOOY18lnrTbuDSmKVH6j4ttHElO
	cRYXsCrDWu5l+R0M9zHK3GHSCy/XtsSDDmwD8Flb6ikg7lMkDbjJ6Doc9rNd0Qk9HyUGZ
	IPr1v1CBmH1ak6RITNp0NLTks9aST7Gdcsk=
X-RZG-AUTH: :LmkWe0TmffCene0uRkJY6AqDWBjdcSs8EVOKK4t9JJpSMjf2AKg+TcRtFaNcoY+nN7sOdsXpA2aQKnqZIA==
X-RZG-CLASS-ID: mo00
Received: from vitamin-2.local (e180163140.adsl.alicedsl.de [85.180.163.140])
	by post.strato.de (fruni mo61) (RZmta 23.5)
	with ESMTP id o03673m9NHS0YI for <general@hadoop.apache.org>;
	Sat, 23 Oct 2010 21:24:59 +0200 (MEST)
Message-ID: <4CC3368B.9030200@brainlounge.de>
Date: Sat, 23 Oct 2010 21:24:59 +0200
From: Bernd Fondermann <bf_jak@brainlounge.de>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.5) Gecko/20091204 Thunderbird/3.0
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: bringing the codebases back in line
References: <20101022220359.GE17626@tp> <C8E75CD0.44AD5%mahadev@yahoo-inc.com> <AANLkTik8WhrSLMw5K+C=643ek0_pORYhU+0C+dZe0CcN@mail.gmail.com> <79F839A8-B258-467A-8B57-B7F4D09A4FC9@apache.org> <4CC2AAA6.8090605@brainlounge.de> <BD6D7067-A95C-4DBE-BDA5-987F220032AC@apache.org>
In-Reply-To: <BD6D7067-A95C-4DBE-BDA5-987F220032AC@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 23.10.10 18:10, Owen O'Malley wrote:
>
> On Oct 23, 2010, at 2:28 AM, Bernd Fondermann wrote:
>
>> On 23.10.10 05:40, Owen O'Malley wrote:
>>>
>>> The current plan of record is to make
>>> cut a branch next month, stabilize it, and release it.
>>
>> I'd like to revisit the mailing list thread where this decision was
>> made. Can you point me to it?
>
> Sure.
>
> http://www.mail-archive.com/common-dev@hadoop.apache.org/msg01388.html

Thanks, I wouldn't have been able to track that down myself so easily.

   Bernd

From general-return-2262-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 23 22:28:14 2010
Return-Path: <general-return-2262-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22860 invoked from network); 23 Oct 2010 22:28:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Oct 2010 22:28:14 -0000
Received: (qmail 91361 invoked by uid 500); 23 Oct 2010 22:28:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91311 invoked by uid 500); 23 Oct 2010 22:28:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91303 invoked by uid 99); 23 Oct 2010 22:28:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 22:28:12 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,MIME_BASE64_BLANKS,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of vijaymadhavan23@gmail.com designates 74.125.83.48 as permitted sender)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 23 Oct 2010 22:28:05 +0000
Received: by gwb15 with SMTP id 15so2136896gwb.35
        for <general@hadoop.apache.org>; Sat, 23 Oct 2010 15:27:45 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:date:subject:message-id:from
         :to:mime-version:content-type:content-transfer-encoding;
        bh=nExg71r6l/KXljAe+cLO1YUxeYvaKPFW8D6JgY2dw6w=;
        b=nmXipMqaIzb68eEJERJ9ZRehR1rNzoDn83lQJK9QHPTDS4/5P9clfKM7PYMPJWR0Vy
         6cdgfA+EedTRZyQLAA9xeFtafWjYNEcwwP6/GzlcfXeq+7KXdOmuhg09JGaA3BkpQQKv
         /tscBjib8h1xiNdOD27zKk8yt4Lc6Op3wV0S0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=date:subject:message-id:from:to:mime-version:content-type
         :content-transfer-encoding;
        b=spRHpvGb3v+AtdAbAuVV++IFxUR6IGJ5qlZgBkoZWCh4+m7kB9Rw0xQYnMFji3Taxd
         dMlcEIIGcSkaaD3CAusF8jRYqWKcO0gFKHdusonzzz3Hkuv1rHI0SAgW4W5DGpNg+QQY
         nYVvCwDRC9iPO3qy54JFqd2z8gg1yaGFp/LSI=
Received: by 10.150.159.19 with SMTP id h19mr9333994ybe.340.1287872864642;
        Sat, 23 Oct 2010 15:27:44 -0700 (PDT)
Received: from localhost (134.sub-97-11-209.myvzw.com [97.11.209.134])
        by mx.google.com with ESMTPS id m66sm3864287yha.21.2010.10.23.15.27.00
        (version=SSLv3 cipher=RC4-MD5);
        Sat, 23 Oct 2010 15:27:43 -0700 (PDT)
Date: Sat, 23 Oct 2010 18:26:51 -0400
Subject: Re: bringing the codebases back in line
Message-ID: <w9ms0p8qe8axmp8rgah0nirm.1287872736717@email.android.com>
From: Vijay Madhavan <vijaymadhavan23@gmail.com>
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: base64
X-Virus-Checked: Checked by ClamAV on apache.org

CgpFbGkgQ29sbGlucyA8ZWxpQGNsb3VkZXJhLmNvbT4gd3JvdGU6Cgo+T24gRnJpLCBPY3QgMjIs
IDIwMTAgYXQgODo0MCBQTSwgT3dlbiBPJ01hbGxleSA8b21hbGxleUBhcGFjaGUub3JnPiB3cm90
ZToKPj4KPj4gT24gT2N0IDIyLCAyMDEwLCBhdCA2OjMzIFBNLCBJYW4gSG9sc21hbiB3cm90ZToK
Pj4KPj4+IEkgdGhpbmsgd2Ugc2hvdWxkIHB1c2ggZm9yd2FyZCB0byAwLjIyIGFzIHdlbGwuCj4+
PiBUaGUgcXVlc3Rpb24gdGhlbiBiZWNvbWVzIHdoYXQgc2hvdWxkIGJlIGluIGl0Lgo+Pgo+PiBJ
J3ZlIGhhZCBiZXR0ZXIgbHVjayB3aXRoIHRpbWUtYmFzZWQgcmVsZWFzZXMgcmF0aGVyIHRoYW4g
ZmVhdHVyZS1iYXNlZAo+PiByZWxlYXNlcyBmb3Igb3BlbiBzb3VyY2UgcHJvamVjdHMuIFRoZSBj
dXJyZW50IHBsYW4gb2YgcmVjb3JkIGlzIHRvIG1ha2UgY3V0Cj4+IGEgYnJhbmNoIG5leHQgbW9u
dGgsIHN0YWJpbGl6ZSBpdCwgYW5kIHJlbGVhc2UgaXQuCj4+Cj4+IC0tIE93ZW4KPgo+R3JlYXQu
ICBBdCB0aGUgY29udHJpYnV0b3IncyBtZWV0dXAgaXQgc291bmRlZCBsaWtlIHRoZSBwbGFuIG9m
IHJlY29yZAo+d2FzIHRvIGhvbGQgMjIgZm9yIEhERlMtMTA1Mi4gIFNwZWFraW5nIG9mIHdoaWNo
IGRvZXMgc29tZW9uZSBoYXZlCj5ub3RlcyB0byBwb3N0IHRvIHRoZSBsaXN0L3dpa2k/Cj4KPkl0
IHdvdWxkIGJlIGdyZWF0IHRvIHNlZSB0aGUgcHJvamVjdCBtb3ZlIHRvIHBlcmlvZGljIHRpbWUt
YmFzZWQKPnJlbGVhc2VzLiAgSW4gcHJldmlvdXMgZGlzY3Vzc2lvbiBpdCBzZWVtZWQgbGlrZSBm
ZWF0dXJlLWJhc2VkCj5yZWxlYXNlcyB3ZXJlIGZhdm9yZWQ6Cj5odHRwOi8vd3d3Lm1haWwtYXJj
aGl2ZS5jb20vZ2VuZXJhbEBoYWRvb3AuYXBhY2hlLm9yZy9tc2cwMTAyMi5odG1sCj4KPlRoYW5r
cywKPkVsaQo=


From general-return-2263-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 09:37:25 2010
Return-Path: <general-return-2263-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 40995 invoked from network); 25 Oct 2010 09:37:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 09:37:24 -0000
Received: (qmail 39118 invoked by uid 500); 25 Oct 2010 09:37:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38646 invoked by uid 500); 25 Oct 2010 09:37:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 39433 invoked by uid 99); 25 Oct 2010 04:42:43 -0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ankit.g1290@gmail.com designates 209.85.214.48 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=BRHoCUMNMtpow5dTVhHhlb3SPKUhx4QOowdnZef/w3M=;
        b=XlndyAjZDd6eRSYj/w6xJuAyV/3A2N4vXHrcG58a2OdyppW+unHkVFSXctLSIz4yLy
         hbmNOwWsaMNAe8hybmBMGKYOZEABMT2u3XkN7znNMT41MlYyGl9ZYaP4RN4Zis0KA9pq
         h1GrxOiqyaII9fU/hGkRQFwBfK1+cSmf5AG68=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=stBGvAuTk2wCExPBE9w91mC8HxrXRfybAxqhXV0hNP0lY2y03vSasYARLlLQGBbn1A
         4y+YvCmjJFg9qO5Cm2nm3h0cP2id93fFiXXoor8ZiDVhcMkltyGeZ1ylID7lAJ1uV/eT
         aWSr9t21HOr5smcPhWH4ss7E1LdJJbTEHAjCY=
MIME-Version: 1.0
Date: Mon, 25 Oct 2010 10:12:15 +0530
Message-ID: <AANLkTinsvv8Y18K19+JwoBYkiJ15HciDSNxqCk-K+siq@mail.gmail.com>
Subject: Running jar files inside map task
From: Ankit Gandhi <ankit.g1290@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d64816d6c80e0493699d38

--0016e6d64816d6c80e0493699d38
Content-Type: text/plain; charset=ISO-8859-1

Hey,
I want to know whether can I run a jar file inside a map task or not because
I have to use the output of that file in my map task.
I am able to run it in standalone mode but it fails in psuedo-distributed
mode.
Thanks in advance

-- 
Ankit Gandhi
Undergraduate
Center for Visual Information Technology
Computer Science Engineering & Dual Degree
IIIT-Hyderabad

--0016e6d64816d6c80e0493699d38--

From general-return-2264-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 09:50:05 2010
Return-Path: <general-return-2264-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44132 invoked from network); 25 Oct 2010 09:50:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 09:50:05 -0000
Received: (qmail 51876 invoked by uid 500); 25 Oct 2010 09:50:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51630 invoked by uid 500); 25 Oct 2010 09:50:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51622 invoked by uid 99); 25 Oct 2010 09:50:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 09:50:02 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 09:49:56 +0000
Received: by fxm7 with SMTP id 7so2945545fxm.35
        for <general@hadoop.apache.org>; Mon, 25 Oct 2010 02:49:35 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type;
        bh=H1/ktSNFUTSWfnRVbrAyjX1Jpk6TAK0+uOf2K4AQkB0=;
        b=Gb67+ARegEwjAML7AXasL2PfVCfnEUcA4TPrIZxGVrCZZyw71DLAJ4IxkZNROmbE5/
         y7DFRAjtySHPy8+KPT89mVGUa3EovoUAtbVxTAV1vDqoe1RdR+8zMI1GylQ9Zq1iiALn
         oJzwz7DA6N4uO08HY2zbiCCdmfUsXo1t31spc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=q4Qioht1A0khIVPcs92tSlT2kGMrfX+lLeQ4KWHYSeVisctmIKAJf533U8tlnZvgFA
         9RXBxOuFR5TKiYF/DEgjBt7i0A12ziYAxkHSkXV94MiXQm93Z2EpkS7FJCuTm0yxRt8o
         S1X9OlNzTtBveju7+hT9VToWyNQagQ7dmd8KY=
Received: by 10.102.219.11 with SMTP id r11mr5094165mug.33.1288000175386; Mon,
 25 Oct 2010 02:49:35 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.223.112.66 with HTTP; Mon, 25 Oct 2010 02:49:15 -0700 (PDT)
In-Reply-To: <AANLkTinsvv8Y18K19+JwoBYkiJ15HciDSNxqCk-K+siq@mail.gmail.com>
References: <AANLkTinsvv8Y18K19+JwoBYkiJ15HciDSNxqCk-K+siq@mail.gmail.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Mon, 25 Oct 2010 15:19:15 +0530
Message-ID: <AANLkTi=p5z81Z7Hujtpsj9dupf6-vreb4bjWb8JB0JS6@mail.gmail.com>
Subject: Re: Running jar files inside map task
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Is DistributedCache what you're looking for?
http://hadoop.apache.org/common/docs/r0.20.2/api/org/apache/hadoop/filecache/DistributedCache.html

On Mon, Oct 25, 2010 at 10:12 AM, Ankit Gandhi <ankit.g1290@gmail.com> wrote:
> Hey,
> I want to know whether can I run a jar file inside a map task or not because
> I have to use the output of that file in my map task.
> I am able to run it in standalone mode but it fails in psuedo-distributed
> mode.
> Thanks in advance
>
> --
> Ankit Gandhi
> Undergraduate
> Center for Visual Information Technology
> Computer Science Engineering & Dual Degree
> IIIT-Hyderabad
>



-- 
Harsh J
www.harshj.com

From general-return-2265-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 09:55:48 2010
Return-Path: <general-return-2265-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45415 invoked from network); 25 Oct 2010 09:55:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 09:55:47 -0000
Received: (qmail 63846 invoked by uid 500); 25 Oct 2010 09:55:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63663 invoked by uid 500); 25 Oct 2010 09:55:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63655 invoked by uid 99); 25 Oct 2010 09:55:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 09:55:42 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [64.78.17.142] (HELO EXSMTP012-11.exch012.intermedia.net) (64.78.17.142)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 25 Oct 2010 09:55:36 +0000
Received: from EXVBE012-15.exch012.intermedia.net ([10.254.2.76]) by EXSMTP012-11.exch012.intermedia.net with Microsoft SMTPSVC(6.0.3790.4675);
	 Mon, 25 Oct 2010 02:55:15 -0700
X-MimeOLE: Produced By Microsoft Exchange V6.5
x-cr-hashedpuzzle: BpeG CACD CH4S CeOt D+rb D/Lr Enxd E9Ey GZhj GyLy IPzG IrOJ ItEH JQfs JVOC J4Lo;1;ZwBlAG4AZQByAGEAbABAAGgAYQBkAG8AbwBwAC4AYQBwAGEAYwBoAGUALgBvAHIAZwA=;Sosha1_v1;7;{0CFAD3D1-F059-4109-945F-823C6F86D5E4};YQBuAGkAcgB1AGQAZABoAGEAQABrAHIAaQBsAGwAaQBvAG4ALgBjAG8AbQA=;Mon, 25 Oct 2010 09:55:06 GMT;WABtAGwASQBuAHAAdQB0AEYAbwByAG0AYQB0AA==
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01CB742A.B8FE48FF"
x-cr-puzzleid: {0CFAD3D1-F059-4109-945F-823C6F86D5E4}
Content-class: urn:content-classes:message
Subject: XmlInputFormat
Date: Mon, 25 Oct 2010 02:55:05 -0700
Message-ID: <4917217774F061468FB94129CD619EB602E8EFD2@EXVBE012-15.exch012.intermedia.net>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: XmlInputFormat
Thread-Index: Act0KrOHGJhG1P+ET8OWhqBUMqL2qg==
From: "Aniruddha" <aniruddha@krillion.com>
To: <general@hadoop.apache.org>
X-OriginalArrivalTime: 25 Oct 2010 09:55:15.0886 (UTC) FILETIME=[B97174E0:01CB742A]

------_=_NextPart_001_01CB742A.B8FE48FF
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi,

=20

I am having trouble with using xml text as input, I had tried with
"org.apache.mahout.classifier.bayes. XmlInputFormat", but no luck. I am
not getting any input in mapper. Did anybody have success to use
XmlInputFormat. Please reply me. I am really stuck with this issue

=20

Thanks & Regards,

Aniruddha


------_=_NextPart_001_01CB742A.B8FE48FF--

From general-return-2266-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 04:16:45 2010
Return-Path: <general-return-2266-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85695 invoked from network); 26 Oct 2010 04:16:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 04:16:45 -0000
Received: (qmail 68908 invoked by uid 500); 26 Oct 2010 04:16:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68412 invoked by uid 500); 26 Oct 2010 04:16:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68400 invoked by uid 99); 26 Oct 2010 04:16:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 04:16:39 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 04:16:34 +0000
Received: by fxm7 with SMTP id 7so3994710fxm.35
        for <general@hadoop.apache.org>; Mon, 25 Oct 2010 21:16:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=R6nqxmTONMeix+0iKH4QcM+Ek2uePWrHRseFRpITPSI=;
        b=CO5p9br5sP/Q9cLsoFYNyw4vrAM67Bb4a8PKQZjQHovKhOufMK/RXciwkwlu6J1XfW
         KXHkaARRdNsTOwNkb/IQV7pXITzE8+j7zAB1bdEcZ+V5XKclmrpIMnr4DUhfrTTGQ2xm
         pfTOWANZOtMIPlEnrWHrPEd9uSlhJHXQsO420=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=yAixFB/MYWw7nHqTNNy+AT+qf1niQGMZ09pkRohSX6jTS0bsyMCqOJX72MLxLlufxW
         eYs1vyL1iGpo0h35/O0WtN9YYgSenyNui9v6YZi9IHqSEWg0lXXIFvLnTrGeyTIzde36
         r9hu7ViNy+Pk4REEVNrL5PIT6xSZEYySd4FLY=
MIME-Version: 1.0
Received: by 10.223.93.198 with SMTP id w6mr380523fam.135.1288066572681; Mon,
 25 Oct 2010 21:16:12 -0700 (PDT)
Received: by 10.223.96.193 with HTTP; Mon, 25 Oct 2010 21:16:12 -0700 (PDT)
In-Reply-To: <4917217774F061468FB94129CD619EB602E8EFD2@EXVBE012-15.exch012.intermedia.net>
References: <4917217774F061468FB94129CD619EB602E8EFD2@EXVBE012-15.exch012.intermedia.net>
Date: Mon, 25 Oct 2010 21:16:12 -0700
Message-ID: <AANLkTimFtVSDApu9hBrWgf4YJCw5RHF1XUG4mXhj+e-C@mail.gmail.com>
Subject: Re: XmlInputFormat
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf3054a5578301d804937d5e60

--20cf3054a5578301d804937d5e60
Content-Type: text/plain; charset=ISO-8859-1

Refer to previous discussion entitled 'Hadoop and XML'

On Mon, Oct 25, 2010 at 2:55 AM, Aniruddha <aniruddha@krillion.com> wrote:

> Hi,
>
>
>
> I am having trouble with using xml text as input, I had tried with
> "org.apache.mahout.classifier.bayes. XmlInputFormat", but no luck. I am
> not getting any input in mapper. Did anybody have success to use
> XmlInputFormat. Please reply me. I am really stuck with this issue
>
>
>
> Thanks & Regards,
>
> Aniruddha
>
>

--20cf3054a5578301d804937d5e60--

From general-return-2267-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 07:45:24 2010
Return-Path: <general-return-2267-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 52210 invoked from network); 26 Oct 2010 07:45:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 07:45:24 -0000
Received: (qmail 53878 invoked by uid 500); 26 Oct 2010 07:45:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53529 invoked by uid 500); 26 Oct 2010 07:45:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53288 invoked by uid 99); 26 Oct 2010 07:45:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 07:45:20 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [64.78.17.165] (HELO exsmtp012-1.exch012.intermedia.net) (64.78.17.165)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 26 Oct 2010 07:45:15 +0000
Received: from EXVBE012-15.exch012.intermedia.net ([10.254.2.76]) by exsmtp012-1.exch012.intermedia.net with Microsoft SMTPSVC(6.0.3790.4675);
	 Tue, 26 Oct 2010 00:44:55 -0700
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: RE: XmlInputFormat
Date: Tue, 26 Oct 2010 00:44:48 -0700
Message-ID: <4917217774F061468FB94129CD619EB602E8F26B@EXVBE012-15.exch012.intermedia.net>
In-Reply-To: <AANLkTimFtVSDApu9hBrWgf4YJCw5RHF1XUG4mXhj+e-C@mail.gmail.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: XmlInputFormat
Thread-Index: Act0xJpUxHzFpLyASUuzh3bsub+mTQAHNkCg
References: <4917217774F061468FB94129CD619EB602E8EFD2@EXVBE012-15.exch012.intermedia.net> <AANLkTimFtVSDApu9hBrWgf4YJCw5RHF1XUG4mXhj+e-C@mail.gmail.com>
From: "Aniruddha" <aniruddha@krillion.com>
To: <general@hadoop.apache.org>,
	"Ted Yu" <yuzhihong@gmail.com>
X-OriginalArrivalTime: 26 Oct 2010 07:44:55.0012 (UTC) FILETIME=[AE408A40:01CB74E1]

Please give me some links. I m new to hadoop as well as also new how
this email discussion goes

-----Original Message-----
From: Ted Yu [mailto:yuzhihong@gmail.com]=20
Sent: 26 October 2010 09:46
To: general@hadoop.apache.org
Subject: Re: XmlInputFormat

Refer to previous discussion entitled 'Hadoop and XML'

On Mon, Oct 25, 2010 at 2:55 AM, Aniruddha <aniruddha@krillion.com>
wrote:

> Hi,
>
>
>
> I am having trouble with using xml text as input, I had tried with
> "org.apache.mahout.classifier.bayes. XmlInputFormat", but no luck. I
am
> not getting any input in mapper. Did anybody have success to use
> XmlInputFormat. Please reply me. I am really stuck with this issue
>
>
>
> Thanks & Regards,
>
> Aniruddha
>
>

From general-return-2268-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 09:06:23 2010
Return-Path: <general-return-2268-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97226 invoked from network); 26 Oct 2010 09:06:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 09:06:22 -0000
Received: (qmail 41531 invoked by uid 500); 26 Oct 2010 09:06:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 41100 invoked by uid 500); 26 Oct 2010 09:06:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 41084 invoked by uid 99); 26 Oct 2010 09:06:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 09:06:17 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 09:06:11 +0000
Received: by fxm7 with SMTP id 7so4192351fxm.35
        for <general@hadoop.apache.org>; Tue, 26 Oct 2010 02:05:51 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type;
        bh=GQfkIOWBI43C3fQ4kaeErTE+Alsm3GGK9HZXlUlpQDo=;
        b=d5gRbVvW5rlI87ze8zZwbbWNskM069MynStcH79RpYwB7Uh3s2qhj4MQpJTqgISBmo
         nt1ospjVeRPxsla5RDt4YudOEF76zjpl63tdpT+f4OB/51+/LXtbMtIcc8BStcgXyqvi
         fXoxIv6+9WHaichvPl/xYg810tNL2qvkmwJF8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=WFBDyhIo7xjC2O9z3pcL/Exow1turzLaL+9GObQyM1l0Zvh6heGos/qtMFEdkr4Val
         LwBkaJu8doBOQJTbS/WeA30VAk93il74uNZYxAGm9/94aM7ug9x/bghrxaGpYaoxK07B
         pdc2A2yqmDtcKRY7lltQ06Epl7K+IgmvvZCHg=
Received: by 10.223.75.207 with SMTP id z15mr563639faj.12.1288083951018; Tue,
 26 Oct 2010 02:05:51 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.223.120.80 with HTTP; Tue, 26 Oct 2010 02:05:30 -0700 (PDT)
In-Reply-To: <4917217774F061468FB94129CD619EB602E8F26B@EXVBE012-15.exch012.intermedia.net>
References: <4917217774F061468FB94129CD619EB602E8EFD2@EXVBE012-15.exch012.intermedia.net>
 <AANLkTimFtVSDApu9hBrWgf4YJCw5RHF1XUG4mXhj+e-C@mail.gmail.com> <4917217774F061468FB94129CD619EB602E8F26B@EXVBE012-15.exch012.intermedia.net>
From: Harsh J <qwertymaniac@gmail.com>
Date: Tue, 26 Oct 2010 14:35:30 +0530
Message-ID: <AANLkTinySsCOiToEQb4x9sWOue3g=UdzDLOOhp4SEgz8@mail.gmail.com>
Subject: Re: XmlInputFormat
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Tue, Oct 26, 2010 at 1:14 PM, Aniruddha <aniruddha@krillion.com> wrote:
> Please give me some links. I m new to hadoop as well as also new how
> this email discussion goes
>
> -----Original Message-----
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: 26 October 2010 09:46
> To: general@hadoop.apache.org
> Subject: Re: XmlInputFormat
>
> Refer to previous discussion entitled 'Hadoop and XML'

Here's the thread archive at OS Dir:
http://osdir.com/ml/general-hadoop-apache/2010-07/msg00102.html

>
> On Mon, Oct 25, 2010 at 2:55 AM, Aniruddha <aniruddha@krillion.com>
> wrote:
>
>> Hi,
>>
>>
>>
>> I am having trouble with using xml text as input, I had tried with
>> "org.apache.mahout.classifier.bayes. XmlInputFormat", but no luck. I
> am
>> not getting any input in mapper. Did anybody have success to use
>> XmlInputFormat. Please reply me. I am really stuck with this issue
>>
>>
>>
>> Thanks & Regards,
>>
>> Aniruddha
>>
>>
>



-- 
Harsh J
www.harshj.com

From general-return-2269-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 21:50:31 2010
Return-Path: <general-return-2269-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 79441 invoked from network); 26 Oct 2010 21:50:31 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 21:50:31 -0000
Received: (qmail 4397 invoked by uid 500); 26 Oct 2010 21:50:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4336 invoked by uid 500); 26 Oct 2010 21:50:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4325 invoked by uid 99); 26 Oct 2010 21:50:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 21:50:28 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of dhruba@gmail.com designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 21:50:21 +0000
Received: by ewy3 with SMTP id 3so2738488ewy.35
        for <general@hadoop.apache.org>; Tue, 26 Oct 2010 14:50:01 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=GIG1tmOr0QbQH4rhpF7776hzMu0VeIOzoCJ2u0lMofU=;
        b=PpU4WpSAbtlrYHcNEcuoWMRPO82GGr+BpLNdQzMEkun5O1fqBWUk+HPZJU/ay1cVAd
         21XqvnnKcq/l6DsALkxkyyr2WvnTsotndji0wxKlo50obqOWf/KO8RWQO1KPyeNqKifZ
         xIIhQTrY8DohQq7jkkQpcFrJYuHfoqfcAGR6s=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=WnQy0bEkuAnIvELu97M0laIEjsuhs4NQoPTeGk4fGjJy/pFrCiObQC+Lh1ilnMp59t
         B1KzgEozmbN9G2P8gZ9U59g1rUaPTHC9ZZWnJRIfVn6RQT4wSn2aEtkntDUD7DE2Vx1o
         7XvswON+LeL55aORZYHllmso7MWD8untQFlGc=
MIME-Version: 1.0
Received: by 10.216.11.3 with SMTP id 3mr2302011wew.89.1288129801037; Tue, 26
 Oct 2010 14:50:01 -0700 (PDT)
Received: by 10.216.64.78 with HTTP; Tue, 26 Oct 2010 14:50:00 -0700 (PDT)
Date: Tue, 26 Oct 2010 14:50:00 -0700
Message-ID: <AANLkTikkU7w4BMMD9qz8z4xYYMe832-aJRV-zR89M-_6@mail.gmail.com>
Subject: Hadoop code reviews on ReviewBoard?
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364d206b37271404938c171f
X-Virus-Checked: Checked by ClamAV on apache.org

--0016364d206b37271404938c171f
Content-Type: text/plain; charset=ISO-8859-1

I saw an ASF announcement that there's now a review board instance
available, it will be nice if we can try it out.

https://blogs.apache.org/infra/entry/reviewboard_instance_running_at_the

I filed the following JIRA to see if we can use review board to review
Hadoop JIRAS.

https://issues.apache.org/jira/browse/INFRA-3108

thanks,
dhruba

--0016364d206b37271404938c171f--

From general-return-2270-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 22:39:32 2010
Return-Path: <general-return-2270-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97639 invoked from network); 26 Oct 2010 22:39:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 22:39:32 -0000
Received: (qmail 47727 invoked by uid 500); 26 Oct 2010 22:39:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47643 invoked by uid 500); 26 Oct 2010 22:39:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47635 invoked by uid 99); 26 Oct 2010 22:39:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 22:39:30 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 22:39:24 +0000
Received: by qwb8 with SMTP id 8so4408qwb.35
        for <general@hadoop.apache.org>; Tue, 26 Oct 2010 15:39:03 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.92.20 with SMTP id p20mr8270456qcm.280.1288132743425; Tue,
 26 Oct 2010 15:39:03 -0700 (PDT)
Received: by 10.229.15.70 with HTTP; Tue, 26 Oct 2010 15:38:33 -0700 (PDT)
In-Reply-To: <AANLkTikkU7w4BMMD9qz8z4xYYMe832-aJRV-zR89M-_6@mail.gmail.com>
References: <AANLkTikkU7w4BMMD9qz8z4xYYMe832-aJRV-zR89M-_6@mail.gmail.com>
Date: Tue, 26 Oct 2010 15:38:33 -0700
Message-ID: <AANLkTi=af-dj74z0x4p7e-PRdbWqNeZob026ohxa6=mm@mail.gmail.com>
Subject: Re: Hadoop code reviews on ReviewBoard?
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Oct 26, 2010 at 2:50 PM, Dhruba Borthakur <dhruba@gmail.com> wrote:
> I saw an ASF announcement that there's now a review board instance
> available, it will be nice if we can try it out.

+1  Thanks Dhruba.

Thanks,
Eli

From general-return-2271-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 22:40:24 2010
Return-Path: <general-return-2271-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97901 invoked from network); 26 Oct 2010 22:40:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 22:40:24 -0000
Received: (qmail 49570 invoked by uid 500); 26 Oct 2010 22:40:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49325 invoked by uid 500); 26 Oct 2010 22:40:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49317 invoked by uid 99); 26 Oct 2010 22:40:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 22:40:23 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 22:40:17 +0000
Received: by qyk34 with SMTP id 34so3178263qyk.14
        for <general@hadoop.apache.org>; Tue, 26 Oct 2010 15:39:56 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.224.207.71 with SMTP id fx7mr1898411qab.309.1288132736506;
 Tue, 26 Oct 2010 15:38:56 -0700 (PDT)
Received: by 10.229.15.70 with HTTP; Tue, 26 Oct 2010 15:38:33 -0700 (PDT)
In-Reply-To: <AANLkTikkU7w4BMMD9qz8z4xYYMe832-aJRV-zR89M-_6@mail.gmail.com>
References: <AANLkTikkU7w4BMMD9qz8z4xYYMe832-aJRV-zR89M-_6@mail.gmail.com>
Date: Tue, 26 Oct 2010 15:38:33 -0700
Message-ID: <AANLkTi=af-dj74z0x4p7e-PRdbWqNeZob026ohxa6=mm@mail.gmail.com>
Subject: Re: Hadoop code reviews on ReviewBoard?
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Oct 26, 2010 at 2:50 PM, Dhruba Borthakur <dhruba@gmail.com> wrote:
> I saw an ASF announcement that there's now a review board instance
> available, it will be nice if we can try it out.

+1  Thanks Dhruba.

Thanks,
Eli

From general-return-2272-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 22:50:06 2010
Return-Path: <general-return-2272-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 99519 invoked from network); 26 Oct 2010 22:50:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 22:50:05 -0000
Received: (qmail 55809 invoked by uid 500); 26 Oct 2010 22:50:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55606 invoked by uid 500); 26 Oct 2010 22:50:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55598 invoked by uid 99); 26 Oct 2010 22:50:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 22:50:03 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 22:49:59 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id o9QMnM3t015292
	for <general@hadoop.apache.org>; Tue, 26 Oct 2010 15:49:22 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1288133362;
	bh=4OmSYB8vvOGjSoI/+ssieHwMKowlfwFQysQO1EVtBwg=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=aXBxKgHa91f0vgDUzef/zryMESqhjupVgvIHWGscuzTgseFgTpsmzd5M0iY6KBItL
	 c+eJQE3qiHrDbyMrfaeHMVS6TYQjfTQqxogFZ6G7W5wFnTcKNnQFqdnlarAlT3yGvb
	 TJ4JWJoxJ73xjGRNwuqXVeSfqicWtjFHUkHJ1fzs=
Message-Id: <33C92F80-1C08-48D0-AF52-162FF6BB0379@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <AANLkTikkU7w4BMMD9qz8z4xYYMe832-aJRV-zR89M-_6@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Hadoop code reviews on ReviewBoard?
Date: Tue, 26 Oct 2010 15:49:22 -0700
References: <AANLkTikkU7w4BMMD9qz8z4xYYMe832-aJRV-zR89M-_6@mail.gmail.com>
X-Mailer: Apple Mail (2.936)

+1

On Oct 26, 2010, at 2:50 PM, Dhruba Borthakur wrote:

> I saw an ASF announcement that there's now a review board instance
> available, it will be nice if we can try it out.
>
> https://blogs.apache.org/infra/entry/reviewboard_instance_running_at_the
>
> I filed the following JIRA to see if we can use review board to review
> Hadoop JIRAS.
>
> https://issues.apache.org/jira/browse/INFRA-3108
>
> thanks,
> dhruba


From general-return-2273-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 02:02:42 2010
Return-Path: <general-return-2273-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91031 invoked from network); 27 Oct 2010 02:02:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 02:02:41 -0000
Received: (qmail 2503 invoked by uid 500); 27 Oct 2010 02:02:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2395 invoked by uid 500); 27 Oct 2010 02:02:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2387 invoked by uid 99); 27 Oct 2010 02:02:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 02:02:40 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 02:02:33 +0000
Received: by wyg36 with SMTP id 36so173761wyg.35
        for <general@hadoop.apache.org>; Tue, 26 Oct 2010 19:02:11 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.53.205 with SMTP id g55mr159001wec.112.1288144930878; Tue,
 26 Oct 2010 19:02:10 -0700 (PDT)
Sender: kumar@bike.snu.ac.kr
Received: by 10.216.133.161 with HTTP; Tue, 26 Oct 2010 19:02:10 -0700 (PDT)
In-Reply-To: <AANLkTinsvv8Y18K19+JwoBYkiJ15HciDSNxqCk-K+siq@mail.gmail.com>
References: <AANLkTinsvv8Y18K19+JwoBYkiJ15HciDSNxqCk-K+siq@mail.gmail.com>
Date: Wed, 27 Oct 2010 11:02:10 +0900
X-Google-Sender-Auth: BgpdWzmRroWWlQIRwa5NF1nBUw8
Message-ID: <AANLkTinm2BFevbG16YX1VVamauK26tnpcmdY_JwX6vFE@mail.gmail.com>
Subject: Re: Running jar files inside map task
From: Kumar Harshit <hkumar.arora@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6db5bee0636a904938f9dab

--0016e6db5bee0636a904938f9dab
Content-Type: text/plain; charset=ISO-8859-1

You can create 2nd map reduce job. The input to the mapper of 2nd Map Reduce
job is the output of 1st Map Reduce job. This way you can tackle the issue.

Hope it helps

Kumar

On Mon, Oct 25, 2010 at 1:42 PM, Ankit Gandhi <ankit.g1290@gmail.com> wrote:

> Hey,
> I want to know whether can I run a jar file inside a map task or not
> because
> I have to use the output of that file in my map task.
> I am able to run it in standalone mode but it fails in psuedo-distributed
> mode.
> Thanks in advance
>
> --
> Ankit Gandhi
> Undergraduate
> Center for Visual Information Technology
> Computer Science Engineering & Dual Degree
> IIIT-Hyderabad
>

--0016e6db5bee0636a904938f9dab--

From general-return-2274-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 07:23:03 2010
Return-Path: <general-return-2274-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39855 invoked from network); 27 Oct 2010 07:23:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 07:23:02 -0000
Received: (qmail 44176 invoked by uid 500); 27 Oct 2010 07:23:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44117 invoked by uid 500); 27 Oct 2010 07:22:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44109 invoked by uid 99); 27 Oct 2010 07:22:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 07:22:58 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of gaur.vbagga@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 07:22:52 +0000
Received: by wyg36 with SMTP id 36so377509wyg.35
        for <general@hadoop.apache.org>; Wed, 27 Oct 2010 00:22:31 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type;
        bh=K21V6I46AdpKV6AMAPIBlzSuwpXgTr50EAtPZG0JRAM=;
        b=Q5cPKJ/bAnqRjEL80KfNltdDRMFNE/4I3BEOFEGFWrdcJh3nNt1BUqbItj66343too
         BrmS+mKxEIIuqYZjrBkxllYcJq7jG66SyovEGIOIPPho9g5gLDfCzism9pmrbJJYxurE
         sZJKaL4rMxnHGl0Bw5rJCz3nXMU50nhD7r1LY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=RfWCHJC0BNmpxNt2XWhbD/Qyc7t/pm1Y2kOIcpnnbOuhps5wJhZKtC2fJuN8H6rVpL
         fV5jfRhFRXUW9VF1SlthpTJRGZSHkqH0lf+EcArmWaG9n8XYN+JIMIcbsG6Fd6zkuLDk
         cpZxq6HviexkrUNqk8pWCOg0Ve1b1E7hE4OLU=
Received: by 10.216.168.202 with SMTP id k52mr8560008wel.105.1288164150966;
 Wed, 27 Oct 2010 00:22:30 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.69.142 with HTTP; Wed, 27 Oct 2010 00:22:10 -0700 (PDT)
In-Reply-To: <AANLkTinm2BFevbG16YX1VVamauK26tnpcmdY_JwX6vFE@mail.gmail.com>
References: <AANLkTinsvv8Y18K19+JwoBYkiJ15HciDSNxqCk-K+siq@mail.gmail.com> <AANLkTinm2BFevbG16YX1VVamauK26tnpcmdY_JwX6vFE@mail.gmail.com>
From: gaurav bagga <gaur.vbagga@gmail.com>
Date: Wed, 27 Oct 2010 00:22:10 -0700
Message-ID: <AANLkTi=6ypCxrCNFk=0n92m9OnuS-86vjnQ63+M1aPaT@mail.gmail.com>
Subject: Re: Running jar files inside map task
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f631bea17aa00493941654

--001485f631bea17aa00493941654
Content-Type: text/plain; charset=ISO-8859-1

It would be great if you could tell or point me to an article which uses the
output of first map reduce as input for the 2nd map reduce.

-Gaurav



On Tue, Oct 26, 2010 at 7:02 PM, Kumar Harshit <hkumar.arora@gmail.com>wrote:

> You can create 2nd map reduce job. The input to the mapper of 2nd Map
> Reduce
> job is the output of 1st Map Reduce job. This way you can tackle the issue.
>
> Hope it helps
>
> Kumar
>
> On Mon, Oct 25, 2010 at 1:42 PM, Ankit Gandhi <ankit.g1290@gmail.com>
> wrote:
>
> > Hey,
> > I want to know whether can I run a jar file inside a map task or not
> > because
> > I have to use the output of that file in my map task.
> > I am able to run it in standalone mode but it fails in psuedo-distributed
> > mode.
> > Thanks in advance
> >
> > --
> > Ankit Gandhi
> > Undergraduate
> > Center for Visual Information Technology
> > Computer Science Engineering & Dual Degree
> > IIIT-Hyderabad
> >
>

--001485f631bea17aa00493941654--

From general-return-2275-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 09:16:40 2010
Return-Path: <general-return-2275-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93607 invoked from network); 27 Oct 2010 09:16:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 09:16:40 -0000
Received: (qmail 218 invoked by uid 500); 27 Oct 2010 09:16:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99703 invoked by uid 500); 27 Oct 2010 09:16:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 99695 invoked by uid 99); 27 Oct 2010 09:16:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 09:16:37 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.42] (HELO mail-ww0-f42.google.com) (74.125.82.42)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 09:16:30 +0000
Received: by wwi18 with SMTP id 18so1303848wwi.5
        for <general@hadoop.apache.org>; Wed, 27 Oct 2010 02:16:08 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.48.196 with SMTP id v46mr536149web.28.1288170968546; Wed,
 27 Oct 2010 02:16:08 -0700 (PDT)
Sender: kumar@bike.snu.ac.kr
Received: by 10.216.133.161 with HTTP; Wed, 27 Oct 2010 02:16:08 -0700 (PDT)
In-Reply-To: <AANLkTi=6ypCxrCNFk=0n92m9OnuS-86vjnQ63+M1aPaT@mail.gmail.com>
References: <AANLkTinsvv8Y18K19+JwoBYkiJ15HciDSNxqCk-K+siq@mail.gmail.com>
	<AANLkTinm2BFevbG16YX1VVamauK26tnpcmdY_JwX6vFE@mail.gmail.com>
	<AANLkTi=6ypCxrCNFk=0n92m9OnuS-86vjnQ63+M1aPaT@mail.gmail.com>
Date: Wed, 27 Oct 2010 05:16:08 -0400
X-Google-Sender-Auth: vkAdqKnGx5SPMWAUMtPFVSpjzKQ
Message-ID: <AANLkTi=yL2q7p_Enf1ALtE4gfY+LgaBZXBLE2G3ZfTSu@mail.gmail.com>
Subject: Re: Running jar files inside map task
From: Kumar Harshit <hkumar.arora@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f6ce1efd8a8c049395ac3a

--001485f6ce1efd8a8c049395ac3a
Content-Type: text/plain; charset=ISO-8859-1

I think this is not difficult. Let me give an example here.

There are two Map Reduce job, MR1 and MR2.
MR1 has Map1 and Reduce1 as mapper and reducer.
MR2 has Map2 and Reduce2 as mapper and reducer.

MR1 input directory is Input1 and output directory is Output1
MR2 input directory is Output1 (*this makes sure that Output of MR1 is input
to MR2*) and output directory is MR2.

In the main() function of MR1, specify the paths of input directory and
output directory:
setInputPaths(new Path("Input1"));
setOutputPaths(new Path("Output1"));

In the main() function of MR2, specify the path of input directory and
output directory
*setInputPaths(new Path("Output1")); //Note that, the Output directory of
Previous MR1 is now the input directory o*
setOutputPaths(new Path("Output2"));

Try to search on net for more examples. there are plenty of them.

Best
Kumar
On Wed, Oct 27, 2010 at 3:22 AM, gaurav bagga <gaur.vbagga@gmail.com> wrote:

> It would be great if you could tell or point me to an article which uses
> the
> output of first map reduce as input for the 2nd map reduce.
>
> -Gaurav
>
>
>
> On Tue, Oct 26, 2010 at 7:02 PM, Kumar Harshit <hkumar.arora@gmail.com
> >wrote:
>
> > You can create 2nd map reduce job. The input to the mapper of 2nd Map
> > Reduce
> > job is the output of 1st Map Reduce job. This way you can tackle the
> issue.
> >
> > Hope it helps
> >
> > Kumar
> >
> > On Mon, Oct 25, 2010 at 1:42 PM, Ankit Gandhi <ankit.g1290@gmail.com>
> > wrote:
> >
> > > Hey,
> > > I want to know whether can I run a jar file inside a map task or not
> > > because
> > > I have to use the output of that file in my map task.
> > > I am able to run it in standalone mode but it fails in
> psuedo-distributed
> > > mode.
> > > Thanks in advance
> > >
> > > --
> > > Ankit Gandhi
> > > Undergraduate
> > > Center for Visual Information Technology
> > > Computer Science Engineering & Dual Degree
> > > IIIT-Hyderabad
> > >
> >
>

--001485f6ce1efd8a8c049395ac3a--

From general-return-2276-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 09:52:19 2010
Return-Path: <general-return-2276-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6927 invoked from network); 27 Oct 2010 09:52:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 09:52:18 -0000
Received: (qmail 49851 invoked by uid 500); 27 Oct 2010 09:52:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49515 invoked by uid 500); 27 Oct 2010 09:52:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49507 invoked by uid 99); 27 Oct 2010 09:52:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 09:52:15 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 09:52:09 +0000
Received: by fxm7 with SMTP id 7so454167fxm.35
        for <general@hadoop.apache.org>; Wed, 27 Oct 2010 02:51:48 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type;
        bh=BUxTjQ4gpMZYOp4MDDxZY/GHW7Cv+tKgT7BHL6ph2k8=;
        b=onQGQsWSOBGQa//EOvw0Ef5ED1rwrHg1/0q2yZpRwAebEXYgr5/i25aeOywmQniujM
         AIvHzxatiyjvjQMuoPvxxs7RI0mAhpuwLPztYwzW/WDj7b8pdUEHpq0q/rz0dUPmBHEx
         RAWcDkDpgEO3N0vFF7fZBsMpNG3954+4QY25E=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=Lyo05XS6RvBCPxSZWoE+GQ83sMjuxOTyI3YY0UzXnDY6Yp8nrF5wMM9lPcHj1F0Ifk
         NPCObxoTO9EkT9ZzwuQiYsyx/apmoMkynZf6Fngj+KSyK0L00VKbgVmZryQrC9k0FbbX
         K6LjSL0BZ8DrPFSdSn8pGxVpSm4q+Pt3chaDo=
Received: by 10.223.101.194 with SMTP id d2mr2066095fao.88.1288173108557; Wed,
 27 Oct 2010 02:51:48 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.223.120.80 with HTTP; Wed, 27 Oct 2010 02:51:28 -0700 (PDT)
In-Reply-To: <AANLkTi=6ypCxrCNFk=0n92m9OnuS-86vjnQ63+M1aPaT@mail.gmail.com>
References: <AANLkTinsvv8Y18K19+JwoBYkiJ15HciDSNxqCk-K+siq@mail.gmail.com>
 <AANLkTinm2BFevbG16YX1VVamauK26tnpcmdY_JwX6vFE@mail.gmail.com> <AANLkTi=6ypCxrCNFk=0n92m9OnuS-86vjnQ63+M1aPaT@mail.gmail.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Wed, 27 Oct 2010 15:21:28 +0530
Message-ID: <AANLkTikp5k73L8iFMvAHaP9BgTvcjeDdH7dvkQRNjWBG@mail.gmail.com>
Subject: Re: Running jar files inside map task
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Oct 27, 2010 at 12:52 PM, gaurav bagga <gaur.vbagga@gmail.com> wrote:
> It would be great if you could tell or point me to an article which uses the
> output of first map reduce as input for the 2nd map reduce.

There's ChainMapper and ChainReducer that let you do [MAP+ / REDUCE
MAP*] kind of job configuration (single job).

If you wish to chain jobs (To look like MRMRMR-ish), look at
o.a.h.mapred.jobcontrol.Job at:
http://hadoop.apache.org/common/docs/r0.20.0/api/org/apache/hadoop/mapred/jobcontrol/Job.html

Specifically, look at Job.addDependingJob.

>
> -Gaurav
>
>
>
> On Tue, Oct 26, 2010 at 7:02 PM, Kumar Harshit <hkumar.arora@gmail.com>wrote:
>
>> You can create 2nd map reduce job. The input to the mapper of 2nd Map
>> Reduce
>> job is the output of 1st Map Reduce job. This way you can tackle the issue.
>>
>> Hope it helps
>>
>> Kumar
>>
>> On Mon, Oct 25, 2010 at 1:42 PM, Ankit Gandhi <ankit.g1290@gmail.com>
>> wrote:
>>
>> > Hey,
>> > I want to know whether can I run a jar file inside a map task or not
>> > because
>> > I have to use the output of that file in my map task.
>> > I am able to run it in standalone mode but it fails in psuedo-distributed
>> > mode.
>> > Thanks in advance
>> >
>> > --
>> > Ankit Gandhi
>> > Undergraduate
>> > Center for Visual Information Technology
>> > Computer Science Engineering & Dual Degree
>> > IIIT-Hyderabad
>> >
>>
>



-- 
Harsh J
www.harshj.com

From general-return-2277-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 13:33:58 2010
Return-Path: <general-return-2277-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31874 invoked from network); 27 Oct 2010 13:33:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 13:33:57 -0000
Received: (qmail 45733 invoked by uid 500); 27 Oct 2010 13:33:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45457 invoked by uid 500); 27 Oct 2010 13:33:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45448 invoked by uid 99); 27 Oct 2010 13:33:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 13:33:51 +0000
X-ASF-Spam-Status: No, hits=-1997.8 required=10.0
	tests=ALL_TRUSTED,HTML_MESSAGE,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 27 Oct 2010 13:33:51 +0000
Received: (qmail 31783 invoked by uid 99); 27 Oct 2010 13:33:31 -0000
Received: from localhost.apache.org (HELO mail-bw0-f48.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 13:33:30 +0000
Received: by bwz19 with SMTP id 19so590645bwz.35
        for <general@hadoop.apache.org>; Wed, 27 Oct 2010 06:33:28 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.100.17 with SMTP id w17mr7182151bkn.43.1288186408743; Wed,
 27 Oct 2010 06:33:28 -0700 (PDT)
Received: by 10.204.29.18 with HTTP; Wed, 27 Oct 2010 06:33:28 -0700 (PDT)
Date: Wed, 27 Oct 2010 06:33:28 -0700
Message-ID: <AANLkTik0yFFb=zQ7a=RHVC2EH0+ruR2pyXruEe7x7Cxk@mail.gmail.com>
Subject: ApacheCon US Next Week
From: "Owen O'Malley" <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163646bc984c32490493994521

--00163646bc984c32490493994521
Content-Type: text/plain; charset=UTF-8

All,
   Next week is ApacheCon US in Atlanta. We have a whole day of talks about
Hadoop lined up. Come meet and learn from the people who make Hadoop. The
entire program is here http://bit.ly/aOHIg6, but the Hadoop talks are:

Hadoop Security: Keeping out the Looky Loos by Owen O'Malley
How to make your map-reduce jobs perform as well as Pig by Thejas M Nair
Hive Design by John Sichi
Fighting Spam with Hadoop by Florian Leibert
Getting Apache Hadoop Production Ready by Chris Douglas
Top 10 Lessons Learned from Deploying Hadoop in a Private Cloud by Rod Cope

There is also a NoSQL track:

Introduction to Cassandra by Jonathan Ellis
How We Made CouchDB Scale by Brad Anderson
HBase Today by Jonathan Gray
Cache & Concurrency considerations for a high performance Cassandra
deployment by SriSatish Ambati
Lucandra: Lucene + Cassandra by Jake Luciani
Cassandra + Hadoop: So Happy Together by Jeremy Hanna

Additionally, there is a Hadoop Meetup the previous night (11/2) that is
open to all. http://bit.ly/bzx48N

-- Owen

--00163646bc984c32490493994521--

From general-return-2278-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 14:20:53 2010
Return-Path: <general-return-2278-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50303 invoked from network); 27 Oct 2010 14:20:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 14:20:52 -0000
Received: (qmail 70314 invoked by uid 500); 27 Oct 2010 14:20:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70044 invoked by uid 500); 27 Oct 2010 14:20:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70036 invoked by uid 99); 27 Oct 2010 14:20:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 14:20:49 +0000
X-ASF-Spam-Status: No, hits=4.7 required=10.0
	tests=FREEMAIL_FROM,FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of michael_segel@hotmail.com designates 65.55.34.22 as permitted sender)
Received: from [65.55.34.22] (HELO col0-omc1-s12.col0.hotmail.com) (65.55.34.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 14:20:42 +0000
Received: from COL117-W22 ([65.55.34.7]) by col0-omc1-s12.col0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Wed, 27 Oct 2010 07:20:21 -0700
Message-ID: <COL117-W22567612FBCD730A04A8638F430@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_90806d1d-63e3-4b0e-bf3f-31f141b87526_"
X-Originating-IP: [65.167.11.254]
From: Michael Segel <michael_segel@hotmail.com>
To: <general@hadoop.apache.org>
Subject: RE: Running jar files inside map task
Date: Wed, 27 Oct 2010 09:20:21 -0500
Importance: Normal
In-Reply-To: <AANLkTikp5k73L8iFMvAHaP9BgTvcjeDdH7dvkQRNjWBG@mail.gmail.com>
References:
 <AANLkTinsvv8Y18K19+JwoBYkiJ15HciDSNxqCk-K+siq@mail.gmail.com>,<AANLkTinm2BFevbG16YX1VVamauK26tnpcmdY_JwX6vFE@mail.gmail.com>
 <AANLkTi=6ypCxrCNFk=0n92m9OnuS-86vjnQ63+M1aPaT@mail.gmail.com>,<AANLkTikp5k73L8iFMvAHaP9BgTvcjeDdH7dvkQRNjWBG@mail.gmail.com>
MIME-Version: 1.0
X-OriginalArrivalTime: 27 Oct 2010 14:20:21.0350 (UTC) FILETIME=[16AA2860:01CB75E2]

--_90806d1d-63e3-4b0e-bf3f-31f141b87526_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


Uhm... just to take this in a different direction ...

Yes=2C you can run a jar file within your hadoop m/r job.=20
You may end up with class loader problems though=2C depending on what's in =
the jar.

I think what I'm talking about goes beyond the concept of chaining jobs=2C =
so I'm just tossing it out there as a response to your question ...

HTH

-Mike


> From: qwertymaniac@gmail.com
> Date: Wed=2C 27 Oct 2010 15:21:28 +0530
> Subject: Re: Running jar files inside map task
> To: general@hadoop.apache.org
>=20
> On Wed=2C Oct 27=2C 2010 at 12:52 PM=2C gaurav bagga <gaur.vbagga@gmail.c=
om> wrote:
> > It would be great if you could tell or point me to an article which use=
s the
> > output of first map reduce as input for the 2nd map reduce.
>=20
> There's ChainMapper and ChainReducer that let you do [MAP+ / REDUCE
> MAP*] kind of job configuration (single job).
>=20
> If you wish to chain jobs (To look like MRMRMR-ish)=2C look at
> o.a.h.mapred.jobcontrol.Job at:
> http://hadoop.apache.org/common/docs/r0.20.0/api/org/apache/hadoop/mapred=
/jobcontrol/Job.html
>=20
> Specifically=2C look at Job.addDependingJob.
>=20
> >
> > -Gaurav
> >
> >
> >
> > On Tue=2C Oct 26=2C 2010 at 7:02 PM=2C Kumar Harshit <hkumar.arora@gmai=
l.com>wrote:
> >
> >> You can create 2nd map reduce job. The input to the mapper of 2nd Map
> >> Reduce
> >> job is the output of 1st Map Reduce job. This way you can tackle the i=
ssue.
> >>
> >> Hope it helps
> >>
> >> Kumar
> >>
> >> On Mon=2C Oct 25=2C 2010 at 1:42 PM=2C Ankit Gandhi <ankit.g1290@gmail=
.com>
> >> wrote:
> >>
> >> > Hey=2C
> >> > I want to know whether can I run a jar file inside a map task or not
> >> > because
> >> > I have to use the output of that file in my map task.
> >> > I am able to run it in standalone mode but it fails in psuedo-distri=
buted
> >> > mode.
> >> > Thanks in advance
> >> >
> >> > --
> >> > Ankit Gandhi
> >> > Undergraduate
> >> > Center for Visual Information Technology
> >> > Computer Science Engineering & Dual Degree
> >> > IIIT-Hyderabad
> >> >
> >>
> >
>=20
>=20
>=20
> --=20
> Harsh J
> www.harshj.com
 		 	   		  =

--_90806d1d-63e3-4b0e-bf3f-31f141b87526_--

From general-return-2279-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 16:41:54 2010
Return-Path: <general-return-2279-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 2761 invoked from network); 27 Oct 2010 16:41:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 16:41:18 -0000
Received: (qmail 80009 invoked by uid 500); 27 Oct 2010 16:36:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76700 invoked by uid 500); 27 Oct 2010 16:35:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67254 invoked by uid 99); 27 Oct 2010 16:30:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 16:30:24 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 16:30:18 +0000
Received: by fxm7 with SMTP id 7so847075fxm.35
        for <general@hadoop.apache.org>; Wed, 27 Oct 2010 09:29:58 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=NWHHaRzj5JF6zq8fsrwxuLGWkT54DeEDbS26terTcTU=;
        b=LEbMGQqpvmTxG6qzABalkABxwbqBgMF1Wcjk5Z/bRcHd1DebNPbjb0KJ13g0PevgDQ
         WqFdiGVgLm05t1Km5aDmMg2dh5KawEhcOQnX3F70gOiqr4yJrVkrcpZDxJ4Zbs7TmS+/
         3hDSdhF7b5pEEOv0J8eYoj9IgbxTBcm7FPvrc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=QYnywzf9eiETOLis0izxeV0/zXEXb/rwDpEz/kyzyKS2sICt1NTc85bmqpf/vBsyva
         yIPf8rLgliDlf48TKrWy/9IADypnEqN5buc+YwYHreKTMTF9hojic4iygO761I3OChaX
         NiuK8SNmNfDfdDhF/I3hc7bTp/eqLPlxCVCN4=
MIME-Version: 1.0
Received: by 10.223.95.208 with SMTP id e16mr2609451fan.59.1288196967837; Wed,
 27 Oct 2010 09:29:27 -0700 (PDT)
Received: by 10.223.96.193 with HTTP; Wed, 27 Oct 2010 09:29:27 -0700 (PDT)
In-Reply-To: <AANLkTinySsCOiToEQb4x9sWOue3g=UdzDLOOhp4SEgz8@mail.gmail.com>
References: <4917217774F061468FB94129CD619EB602E8EFD2@EXVBE012-15.exch012.intermedia.net>
	<AANLkTimFtVSDApu9hBrWgf4YJCw5RHF1XUG4mXhj+e-C@mail.gmail.com>
	<4917217774F061468FB94129CD619EB602E8F26B@EXVBE012-15.exch012.intermedia.net>
	<AANLkTinySsCOiToEQb4x9sWOue3g=UdzDLOOhp4SEgz8@mail.gmail.com>
Date: Wed, 27 Oct 2010 09:29:27 -0700
Message-ID: <AANLkTinEE+OUr6T5ncyb8N1fbNGQe26eaj6G5+Zbfjf1@mail.gmail.com>
Subject: Re: XmlInputFormat
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf3054a4a5ab323f04939bbae6
X-Virus-Checked: Checked by ClamAV on apache.org

--20cf3054a4a5ab323f04939bbae6
Content-Type: text/plain; charset=ISO-8859-1

This link has more information:
http://mail-archives.apache.org/mod_mbox/hadoop-general/201007.mbox/%3CAANLkTimScTowrYvQNcC5-ccxwVlwRyWnOdQ4asBWjNUW@mail.gmail.com%3E

I personally store hadoop/hbase discussions in gmail so that I can find them
easily.

On Tue, Oct 26, 2010 at 2:05 AM, Harsh J <qwertymaniac@gmail.com> wrote:

> On Tue, Oct 26, 2010 at 1:14 PM, Aniruddha <aniruddha@krillion.com> wrote:
> > Please give me some links. I m new to hadoop as well as also new how
> > this email discussion goes
> >
> > -----Original Message-----
> > From: Ted Yu [mailto:yuzhihong@gmail.com]
> > Sent: 26 October 2010 09:46
> > To: general@hadoop.apache.org
> > Subject: Re: XmlInputFormat
> >
> > Refer to previous discussion entitled 'Hadoop and XML'
>
> Here's the thread archive at OS Dir:
> http://osdir.com/ml/general-hadoop-apache/2010-07/msg00102.html
>
> >
> > On Mon, Oct 25, 2010 at 2:55 AM, Aniruddha <aniruddha@krillion.com>
> > wrote:
> >
> >> Hi,
> >>
> >>
> >>
> >> I am having trouble with using xml text as input, I had tried with
> >> "org.apache.mahout.classifier.bayes. XmlInputFormat", but no luck. I
> > am
> >> not getting any input in mapper. Did anybody have success to use
> >> XmlInputFormat. Please reply me. I am really stuck with this issue
> >>
> >>
> >>
> >> Thanks & Regards,
> >>
> >> Aniruddha
> >>
> >>
> >
>
>
>
> --
> Harsh J
> www.harshj.com
>

--20cf3054a4a5ab323f04939bbae6--

From general-return-2280-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 18:08:42 2010
Return-Path: <general-return-2280-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69350 invoked from network); 27 Oct 2010 18:08:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 18:08:35 -0000
Received: (qmail 31085 invoked by uid 500); 27 Oct 2010 18:06:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29778 invoked by uid 500); 27 Oct 2010 18:06:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29475 invoked by uid 99); 27 Oct 2010 18:05:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 18:05:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 27 Oct 2010 18:05:17 +0000
Received: (qmail 55662 invoked by uid 99); 27 Oct 2010 18:04:57 -0000
Received: from localhost.apache.org (HELO mail-bw0-f48.google.com) (127.0.0.1)
  (smtp-auth username phunt, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 18:04:57 +0000
Received: by bwz19 with SMTP id 19so869324bwz.35
        for <general@hadoop.apache.org>; Wed, 27 Oct 2010 11:04:55 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.69.65 with SMTP id y1mr2163337bki.23.1288202695621; Wed,
 27 Oct 2010 11:04:55 -0700 (PDT)
Received: by 10.204.114.3 with HTTP; Wed, 27 Oct 2010 11:04:55 -0700 (PDT)
Date: Wed, 27 Oct 2010 11:04:55 -0700
Message-ID: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
Subject: [VOTE] ZooKeeper as TLP?
From: Patrick Hunt <phunt@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Please vote as to whether you think ZooKeeper should become a
top-level Apache project.

The ZooKeeper development community has already voted on and approved
( http://bit.ly/9VCoTi ) the following draft board resolution. It
lists all current active ZooKeeper committers (sans one committer who
has not been active on the project since it's inception
http://bit.ly/9wLkVV) as initial members of the project management
committee (PMC) and myself, Patrick Hunt, as the initial chair.


Do the good people of Hadoop support sending this request on to the
Hadoop PMC?

Regards,

Patrick

------------

    X. Establish the Apache ZooKeeper Project

       WHEREAS, the Board of Directors deems it to be in the best
       interests of the Foundation and consistent with the
       Foundation's purpose to establish a Project Management
       Committee charged with the creation and maintenance of
       open-source software related to distributed system coordination
       for distribution at no charge to the public.

       NOW, THEREFORE, BE IT RESOLVED, that a Project Management
       Committee (PMC), to be known as the "Apache ZooKeeper Project",
       be and hereby is established pursuant to Bylaws of the
       Foundation; and be it further

       RESOLVED, that the Apache ZooKeeper Project be and hereby is
       responsible for the creation and maintenance of software
       related to distributed system coordination; and be it further

       RESOLVED, that the office of "Vice President, Apache ZooKeeper" be
       and hereby is created, the person holding such office to
       serve at the direction of the Board of Directors as the chair
       of the Apache ZooKeeper Project, and to have primary responsibility
       for management of the projects within the scope of
       responsibility of the Apache ZooKeeper Project; and be it further

       RESOLVED, that the persons listed immediately below be and
       hereby are appointed to serve as the initial members of the
       Apache ZooKeeper Project:

         * Patrick Hunt         <phunt@apache.org>
         * Flavio Junqueira     <fpj@apache.org>
         * Mahadev Konar        <mahadev@apache.org>
         * Benjamin Reed        <breed@apache.org>
         * Henry Robinson       <henry@apache.org>

       NOW, THEREFORE, BE IT FURTHER RESOLVED, that Patrick Hunt
       be appointed to the office of Vice President, Apache ZooKeeper, to
       serve in accordance with and subject to the direction of the
       Board of Directors and the Bylaws of the Foundation until
       death, resignation, retirement, removal or disqualification,
       or until a successor is appointed; and be it further

       RESOLVED, that the initial Apache ZooKeeper PMC be and hereby is
       tasked with the creation of a set of bylaws intended to
       encourage open development and increased participation in the
       Apache ZooKeeper Project; and be it further

       RESOLVED, that the Apache ZooKeeper Project be and hereby
       is tasked with the migration and rationalization of the Apache
       Hadoop ZooKeeper sub-project; and be it further

       RESOLVED, that all responsibilities pertaining to the Apache
       Hadoop ZooKeeper sub-project encumbered upon the
       Apache Hadoop Project are hereafter discharged.

From general-return-2281-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 18:11:26 2010
Return-Path: <general-return-2281-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72951 invoked from network); 27 Oct 2010 18:11:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 18:11:25 -0000
Received: (qmail 81180 invoked by uid 500); 27 Oct 2010 18:11:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81114 invoked by uid 500); 27 Oct 2010 18:11:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81106 invoked by uid 99); 27 Oct 2010 18:11:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 18:11:24 +0000
X-ASF-Spam-Status: No, hits=-1997.8 required=10.0
	tests=ALL_TRUSTED,HTML_MESSAGE,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 27 Oct 2010 18:11:21 +0000
Received: (qmail 72911 invoked by uid 99); 27 Oct 2010 18:11:00 -0000
Received: from localhost.apache.org (HELO mail-bw0-f48.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 18:11:00 +0000
Received: by bwz19 with SMTP id 19so875348bwz.35
        for <general@hadoop.apache.org>; Wed, 27 Oct 2010 11:10:58 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.103.133 with SMTP id k5mr7737140bko.68.1288203057897; Wed,
 27 Oct 2010 11:10:57 -0700 (PDT)
Received: by 10.204.29.18 with HTTP; Wed, 27 Oct 2010 11:10:57 -0700 (PDT)
In-Reply-To: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
References: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
Date: Wed, 27 Oct 2010 11:10:57 -0700
Message-ID: <AANLkTikDguEsRE=p8jvzQapUmA1XDXnjaV7stLDr1Vy4@mail.gmail.com>
Subject: Re: [VOTE] ZooKeeper as TLP?
From: "Owen O'Malley" <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d7e4e9aa205804939d250b
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d7e4e9aa205804939d250b
Content-Type: text/plain; charset=UTF-8

On Wed, Oct 27, 2010 at 11:04 AM, Patrick Hunt <phunt@apache.org> wrote:

> Please vote as to whether you think ZooKeeper should become a
> top-level Apache project.


+1

--0016e6d7e4e9aa205804939d250b--

From general-return-2282-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 18:11:38 2010
Return-Path: <general-return-2282-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73075 invoked from network); 27 Oct 2010 18:11:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 18:11:38 -0000
Received: (qmail 82485 invoked by uid 500); 27 Oct 2010 18:11:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82435 invoked by uid 500); 27 Oct 2010 18:11:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82427 invoked by uid 99); 27 Oct 2010 18:11:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 18:11:36 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.161.176] (HELO mail-gx0-f176.google.com) (209.85.161.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 18:11:30 +0000
Received: by gxk24 with SMTP id 24so703278gxk.35
        for <general@hadoop.apache.org>; Wed, 27 Oct 2010 11:11:09 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.24.199 with SMTP id x49mr9381657wex.109.1288203068223;
 Wed, 27 Oct 2010 11:11:08 -0700 (PDT)
Received: by 10.216.161.201 with HTTP; Wed, 27 Oct 2010 11:11:08 -0700 (PDT)
In-Reply-To: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
References: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
Date: Wed, 27 Oct 2010 14:11:08 -0400
Message-ID: <AANLkTi=pZxcBgU=-iRZrnTdCCA7_3YNzmRjMNEtZfa7F@mail.gmail.com>
Subject: Re: [VOTE] ZooKeeper as TLP?
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

+1

On Wed, Oct 27, 2010 at 2:04 PM, Patrick Hunt <phunt@apache.org> wrote:
> Please vote as to whether you think ZooKeeper should become a
> top-level Apache project.
>
> The ZooKeeper development community has already voted on and approved
> ( http://bit.ly/9VCoTi ) the following draft board resolution. It
> lists all current active ZooKeeper committers (sans one committer who
> has not been active on the project since it's inception
> http://bit.ly/9wLkVV) as initial members of the project management
> committee (PMC) and myself, Patrick Hunt, as the initial chair.
>
>
> Do the good people of Hadoop support sending this request on to the
> Hadoop PMC?
>
> Regards,
>
> Patrick
>
> ------------
>
> =A0 =A0X. Establish the Apache ZooKeeper Project
>
> =A0 =A0 =A0 WHEREAS, the Board of Directors deems it to be in the best
> =A0 =A0 =A0 interests of the Foundation and consistent with the
> =A0 =A0 =A0 Foundation's purpose to establish a Project Management
> =A0 =A0 =A0 Committee charged with the creation and maintenance of
> =A0 =A0 =A0 open-source software related to distributed system coordinati=
on
> =A0 =A0 =A0 for distribution at no charge to the public.
>
> =A0 =A0 =A0 NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> =A0 =A0 =A0 Committee (PMC), to be known as the "Apache ZooKeeper Project=
",
> =A0 =A0 =A0 be and hereby is established pursuant to Bylaws of the
> =A0 =A0 =A0 Foundation; and be it further
>
> =A0 =A0 =A0 RESOLVED, that the Apache ZooKeeper Project be and hereby is
> =A0 =A0 =A0 responsible for the creation and maintenance of software
> =A0 =A0 =A0 related to distributed system coordination; and be it further
>
> =A0 =A0 =A0 RESOLVED, that the office of "Vice President, Apache ZooKeepe=
r" be
> =A0 =A0 =A0 and hereby is created, the person holding such office to
> =A0 =A0 =A0 serve at the direction of the Board of Directors as the chair
> =A0 =A0 =A0 of the Apache ZooKeeper Project, and to have primary responsi=
bility
> =A0 =A0 =A0 for management of the projects within the scope of
> =A0 =A0 =A0 responsibility of the Apache ZooKeeper Project; and be it fur=
ther
>
> =A0 =A0 =A0 RESOLVED, that the persons listed immediately below be and
> =A0 =A0 =A0 hereby are appointed to serve as the initial members of the
> =A0 =A0 =A0 Apache ZooKeeper Project:
>
> =A0 =A0 =A0 =A0 * Patrick Hunt =A0 =A0 =A0 =A0 <phunt@apache.org>
> =A0 =A0 =A0 =A0 * Flavio Junqueira =A0 =A0 <fpj@apache.org>
> =A0 =A0 =A0 =A0 * Mahadev Konar =A0 =A0 =A0 =A0<mahadev@apache.org>
> =A0 =A0 =A0 =A0 * Benjamin Reed =A0 =A0 =A0 =A0<breed@apache.org>
> =A0 =A0 =A0 =A0 * Henry Robinson =A0 =A0 =A0 <henry@apache.org>
>
> =A0 =A0 =A0 NOW, THEREFORE, BE IT FURTHER RESOLVED, that Patrick Hunt
> =A0 =A0 =A0 be appointed to the office of Vice President, Apache ZooKeepe=
r, to
> =A0 =A0 =A0 serve in accordance with and subject to the direction of the
> =A0 =A0 =A0 Board of Directors and the Bylaws of the Foundation until
> =A0 =A0 =A0 death, resignation, retirement, removal or disqualification,
> =A0 =A0 =A0 or until a successor is appointed; and be it further
>
> =A0 =A0 =A0 RESOLVED, that the initial Apache ZooKeeper PMC be and hereby=
 is
> =A0 =A0 =A0 tasked with the creation of a set of bylaws intended to
> =A0 =A0 =A0 encourage open development and increased participation in the
> =A0 =A0 =A0 Apache ZooKeeper Project; and be it further
>
> =A0 =A0 =A0 RESOLVED, that the Apache ZooKeeper Project be and hereby
> =A0 =A0 =A0 is tasked with the migration and rationalization of the Apach=
e
> =A0 =A0 =A0 Hadoop ZooKeeper sub-project; and be it further
>
> =A0 =A0 =A0 RESOLVED, that all responsibilities pertaining to the Apache
> =A0 =A0 =A0 Hadoop ZooKeeper sub-project encumbered upon the
> =A0 =A0 =A0 Apache Hadoop Project are hereafter discharged.
>



--=20
Eric Sammer
twitter: esammer
data: www.cloudera.com

From general-return-2283-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 18:12:40 2010
Return-Path: <general-return-2283-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73375 invoked from network); 27 Oct 2010 18:12:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 18:12:39 -0000
Received: (qmail 85167 invoked by uid 500); 27 Oct 2010 18:12:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85066 invoked by uid 500); 27 Oct 2010 18:12:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85058 invoked by uid 99); 27 Oct 2010 18:12:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 18:12:38 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of timrobertson100@gmail.com designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 18:12:33 +0000
Received: by ewy3 with SMTP id 3so552881ewy.35
        for <general@hadoop.apache.org>; Wed, 27 Oct 2010 11:12:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=qMqMwenQ5hfbgzBFBBz1qwTIiuxEy9BSJX0gHYwyzh4=;
        b=mLrmH4uYe72xbvRbipec82EreClo7tfjK7PFo1vE1CYYoK3dllCXuhiVat1r9dVFWR
         RbqPEB+z9GaQkgBcHhpp84Sf3s/vbqLPdETdxFmt3m7gmtCQpMdHN7SMJqpFL2yRo+Gv
         Jqj9yaWcsvM0ZZMJMgLaZvI0EPkmsf7Vdmu6o=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=EJICOdeg/DLB6+rSgZvpcdM8ZZzoJ62670GzIQYycf4vBV8/BB4qPN48sAt5ZQe5Jm
         bg401k+ofgT/Mgux3jDSY9X0oRq/btN5a8aLVdywzEvGBnVlaK2w1wsxmQlvinkwVROx
         uj6vZRooDDTVDxL9pbvM29KOZvGvsKXFWEF9U=
MIME-Version: 1.0
Received: by 10.213.8.65 with SMTP id g1mr3579840ebg.65.1288203131722; Wed, 27
 Oct 2010 11:12:11 -0700 (PDT)
Received: by 10.14.125.201 with HTTP; Wed, 27 Oct 2010 11:12:11 -0700 (PDT)
In-Reply-To: <AANLkTi=pZxcBgU=-iRZrnTdCCA7_3YNzmRjMNEtZfa7F@mail.gmail.com>
References: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
	<AANLkTi=pZxcBgU=-iRZrnTdCCA7_3YNzmRjMNEtZfa7F@mail.gmail.com>
Date: Wed, 27 Oct 2010 20:12:11 +0200
Message-ID: <AANLkTinW9DKVUS6yXbS5uu4GmCiTVid2q7ujPfNXtnhR@mail.gmail.com>
Subject: Re: [VOTE] ZooKeeper as TLP?
From: Tim Robertson <timrobertson100@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

+1

On Wed, Oct 27, 2010 at 8:11 PM, Eric Sammer <esammer@cloudera.com> wrote:
> +1
>
> On Wed, Oct 27, 2010 at 2:04 PM, Patrick Hunt <phunt@apache.org> wrote:
>> Please vote as to whether you think ZooKeeper should become a
>> top-level Apache project.
>>
>> The ZooKeeper development community has already voted on and approved
>> ( http://bit.ly/9VCoTi ) the following draft board resolution. It
>> lists all current active ZooKeeper committers (sans one committer who
>> has not been active on the project since it's inception
>> http://bit.ly/9wLkVV) as initial members of the project management
>> committee (PMC) and myself, Patrick Hunt, as the initial chair.
>>
>>
>> Do the good people of Hadoop support sending this request on to the
>> Hadoop PMC?
>>
>> Regards,
>>
>> Patrick
>>
>> ------------
>>
>> =A0 =A0X. Establish the Apache ZooKeeper Project
>>
>> =A0 =A0 =A0 WHEREAS, the Board of Directors deems it to be in the best
>> =A0 =A0 =A0 interests of the Foundation and consistent with the
>> =A0 =A0 =A0 Foundation's purpose to establish a Project Management
>> =A0 =A0 =A0 Committee charged with the creation and maintenance of
>> =A0 =A0 =A0 open-source software related to distributed system coordinat=
ion
>> =A0 =A0 =A0 for distribution at no charge to the public.
>>
>> =A0 =A0 =A0 NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>> =A0 =A0 =A0 Committee (PMC), to be known as the "Apache ZooKeeper Projec=
t",
>> =A0 =A0 =A0 be and hereby is established pursuant to Bylaws of the
>> =A0 =A0 =A0 Foundation; and be it further
>>
>> =A0 =A0 =A0 RESOLVED, that the Apache ZooKeeper Project be and hereby is
>> =A0 =A0 =A0 responsible for the creation and maintenance of software
>> =A0 =A0 =A0 related to distributed system coordination; and be it furthe=
r
>>
>> =A0 =A0 =A0 RESOLVED, that the office of "Vice President, Apache ZooKeep=
er" be
>> =A0 =A0 =A0 and hereby is created, the person holding such office to
>> =A0 =A0 =A0 serve at the direction of the Board of Directors as the chai=
r
>> =A0 =A0 =A0 of the Apache ZooKeeper Project, and to have primary respons=
ibility
>> =A0 =A0 =A0 for management of the projects within the scope of
>> =A0 =A0 =A0 responsibility of the Apache ZooKeeper Project; and be it fu=
rther
>>
>> =A0 =A0 =A0 RESOLVED, that the persons listed immediately below be and
>> =A0 =A0 =A0 hereby are appointed to serve as the initial members of the
>> =A0 =A0 =A0 Apache ZooKeeper Project:
>>
>> =A0 =A0 =A0 =A0 * Patrick Hunt =A0 =A0 =A0 =A0 <phunt@apache.org>
>> =A0 =A0 =A0 =A0 * Flavio Junqueira =A0 =A0 <fpj@apache.org>
>> =A0 =A0 =A0 =A0 * Mahadev Konar =A0 =A0 =A0 =A0<mahadev@apache.org>
>> =A0 =A0 =A0 =A0 * Benjamin Reed =A0 =A0 =A0 =A0<breed@apache.org>
>> =A0 =A0 =A0 =A0 * Henry Robinson =A0 =A0 =A0 <henry@apache.org>
>>
>> =A0 =A0 =A0 NOW, THEREFORE, BE IT FURTHER RESOLVED, that Patrick Hunt
>> =A0 =A0 =A0 be appointed to the office of Vice President, Apache ZooKeep=
er, to
>> =A0 =A0 =A0 serve in accordance with and subject to the direction of the
>> =A0 =A0 =A0 Board of Directors and the Bylaws of the Foundation until
>> =A0 =A0 =A0 death, resignation, retirement, removal or disqualification,
>> =A0 =A0 =A0 or until a successor is appointed; and be it further
>>
>> =A0 =A0 =A0 RESOLVED, that the initial Apache ZooKeeper PMC be and hereb=
y is
>> =A0 =A0 =A0 tasked with the creation of a set of bylaws intended to
>> =A0 =A0 =A0 encourage open development and increased participation in th=
e
>> =A0 =A0 =A0 Apache ZooKeeper Project; and be it further
>>
>> =A0 =A0 =A0 RESOLVED, that the Apache ZooKeeper Project be and hereby
>> =A0 =A0 =A0 is tasked with the migration and rationalization of the Apac=
he
>> =A0 =A0 =A0 Hadoop ZooKeeper sub-project; and be it further
>>
>> =A0 =A0 =A0 RESOLVED, that all responsibilities pertaining to the Apache
>> =A0 =A0 =A0 Hadoop ZooKeeper sub-project encumbered upon the
>> =A0 =A0 =A0 Apache Hadoop Project are hereafter discharged.
>>
>
>
>
> --
> Eric Sammer
> twitter: esammer
> data: www.cloudera.com
>

From general-return-2284-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 18:14:04 2010
Return-Path: <general-return-2284-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 74681 invoked from network); 27 Oct 2010 18:14:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 18:14:03 -0000
Received: (qmail 87409 invoked by uid 500); 27 Oct 2010 18:14:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87349 invoked by uid 500); 27 Oct 2010 18:14:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87341 invoked by uid 99); 27 Oct 2010 18:14:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 18:14:01 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 18:13:55 +0000
Received: from SP2-EX07CAS05.ds.corp.yahoo.com (sp2-ex07cas05.corp.sp2.yahoo.com [98.137.59.39])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id o9RID5Qj083702
	for <general@hadoop.apache.org>; Wed, 27 Oct 2010 11:13:05 -0700 (PDT)
Received: from SP2-EX07VS04.ds.corp.yahoo.com ([98.137.59.33]) by
 SP2-EX07CAS05.ds.corp.yahoo.com ([98.137.59.39]) with mapi; Wed, 27 Oct 2010
 11:13:04 -0700
From: Mahadev Konar <mahadev@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 27 Oct 2010 11:13:02 -0700
Subject: Re: [VOTE] ZooKeeper as TLP?
Thread-Topic: [VOTE] ZooKeeper as TLP?
Thread-Index: Act2AnmjRoMOcRSaQ+2r4vBevFF4vQAAB47y
Message-ID: <C8EDB9BE.44F4C%mahadev@yahoo-inc.com>
In-Reply-To: <AANLkTi=pZxcBgU=-iRZrnTdCCA7_3YNzmRjMNEtZfa7F@mail.gmail.com>
Accept-Language: en, en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en, en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


+1.

Thanks
mahadev

On 10/27/10 11:11 AM, "Eric Sammer" <esammer@cloudera.com> wrote:

> +1
>=20
> On Wed, Oct 27, 2010 at 2:04 PM, Patrick Hunt <phunt@apache.org> wrote:
>> Please vote as to whether you think ZooKeeper should become a
>> top-level Apache project.
>>=20
>> The ZooKeeper development community has already voted on and approved
>> ( http://bit.ly/9VCoTi ) the following draft board resolution. It
>> lists all current active ZooKeeper committers (sans one committer who
>> has not been active on the project since it's inception
>> http://bit.ly/9wLkVV) as initial members of the project management
>> committee (PMC) and myself, Patrick Hunt, as the initial chair.
>>=20
>>=20
>> Do the good people of Hadoop support sending this request on to the
>> Hadoop PMC?
>>=20
>> Regards,
>>=20
>> Patrick
>>=20
>> ------------
>>=20
>> =A0 =A0X. Establish the Apache ZooKeeper Project
>>=20
>> =A0 =A0 =A0 WHEREAS, the Board of Directors deems it to be in the best
>> =A0 =A0 =A0 interests of the Foundation and consistent with the
>> =A0 =A0 =A0 Foundation's purpose to establish a Project Management
>> =A0 =A0 =A0 Committee charged with the creation and maintenance of
>> =A0 =A0 =A0 open-source software related to distributed system coordinat=
ion
>> =A0 =A0 =A0 for distribution at no charge to the public.
>>=20
>> =A0 =A0 =A0 NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>> =A0 =A0 =A0 Committee (PMC), to be known as the "Apache ZooKeeper Projec=
t",
>> =A0 =A0 =A0 be and hereby is established pursuant to Bylaws of the
>> =A0 =A0 =A0 Foundation; and be it further
>>=20
>> =A0 =A0 =A0 RESOLVED, that the Apache ZooKeeper Project be and hereby is
>> =A0 =A0 =A0 responsible for the creation and maintenance of software
>> =A0 =A0 =A0 related to distributed system coordination; and be it furthe=
r
>>=20
>> =A0 =A0 =A0 RESOLVED, that the office of "Vice President, Apache ZooKeep=
er" be
>> =A0 =A0 =A0 and hereby is created, the person holding such office to
>> =A0 =A0 =A0 serve at the direction of the Board of Directors as the chai=
r
>> =A0 =A0 =A0 of the Apache ZooKeeper Project, and to have primary respons=
ibility
>> =A0 =A0 =A0 for management of the projects within the scope of
>> =A0 =A0 =A0 responsibility of the Apache ZooKeeper Project; and be it fu=
rther
>>=20
>> =A0 =A0 =A0 RESOLVED, that the persons listed immediately below be and
>> =A0 =A0 =A0 hereby are appointed to serve as the initial members of the
>> =A0 =A0 =A0 Apache ZooKeeper Project:
>>=20
>> =A0 =A0 =A0 =A0 * Patrick Hunt =A0 =A0 =A0 =A0 <phunt@apache.org>
>> =A0 =A0 =A0 =A0 * Flavio Junqueira =A0 =A0 <fpj@apache.org>
>> =A0 =A0 =A0 =A0 * Mahadev Konar =A0 =A0 =A0 =A0<mahadev@apache.org>
>> =A0 =A0 =A0 =A0 * Benjamin Reed =A0 =A0 =A0 =A0<breed@apache.org>
>> =A0 =A0 =A0 =A0 * Henry Robinson =A0 =A0 =A0 <henry@apache.org>
>>=20
>> =A0 =A0 =A0 NOW, THEREFORE, BE IT FURTHER RESOLVED, that Patrick Hunt
>> =A0 =A0 =A0 be appointed to the office of Vice President, Apache ZooKeep=
er, to
>> =A0 =A0 =A0 serve in accordance with and subject to the direction of the
>> =A0 =A0 =A0 Board of Directors and the Bylaws of the Foundation until
>> =A0 =A0 =A0 death, resignation, retirement, removal or disqualification,
>> =A0 =A0 =A0 or until a successor is appointed; and be it further
>>=20
>> =A0 =A0 =A0 RESOLVED, that the initial Apache ZooKeeper PMC be and hereb=
y is
>> =A0 =A0 =A0 tasked with the creation of a set of bylaws intended to
>> =A0 =A0 =A0 encourage open development and increased participation in th=
e
>> =A0 =A0 =A0 Apache ZooKeeper Project; and be it further
>>=20
>> =A0 =A0 =A0 RESOLVED, that the Apache ZooKeeper Project be and hereby
>> =A0 =A0 =A0 is tasked with the migration and rationalization of the Apac=
he
>> =A0 =A0 =A0 Hadoop ZooKeeper sub-project; and be it further
>>=20
>> =A0 =A0 =A0 RESOLVED, that all responsibilities pertaining to the Apache
>> =A0 =A0 =A0 Hadoop ZooKeeper sub-project encumbered upon the
>> =A0 =A0 =A0 Apache Hadoop Project are hereafter discharged.
>>=20
>=20
>=20
>=20
> --
> Eric Sammer
> twitter: esammer
> data: www.cloudera.com
>=20


From general-return-2285-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 18:23:52 2010
Return-Path: <general-return-2285-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83573 invoked from network); 27 Oct 2010 18:23:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 18:23:51 -0000
Received: (qmail 4278 invoked by uid 500); 27 Oct 2010 18:23:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4232 invoked by uid 500); 27 Oct 2010 18:23:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4223 invoked by uid 99); 27 Oct 2010 18:23:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 18:23:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 27 Oct 2010 18:23:47 +0000
Received: (qmail 83553 invoked by uid 99); 27 Oct 2010 18:23:25 -0000
Received: from localhost.apache.org (HELO mail-iw0-f176.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 18:23:25 +0000
Received: by iwn7 with SMTP id 7so1199394iwn.35
        for <general@hadoop.apache.org>; Wed, 27 Oct 2010 11:23:24 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.37.73 with SMTP id w9mr1605361ibd.42.1288203804525; Wed,
 27 Oct 2010 11:23:24 -0700 (PDT)
Received: by 10.231.215.234 with HTTP; Wed, 27 Oct 2010 11:23:24 -0700 (PDT)
In-Reply-To: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
References: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
Date: Wed, 27 Oct 2010 11:23:24 -0700
Message-ID: <AANLkTinHsHhyNpwMPBeN60TiLRRuCgppDR9jdG8AEgTy@mail.gmail.com>
Subject: Re: [VOTE] ZooKeeper as TLP?
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Wed, Oct 27, 2010 at 11:04 AM, Patrick Hunt <phunt@apache.org> wrote:
> Please vote as to whether you think ZooKeeper should become a
> top-level Apache project.

+1 -C

From general-return-2286-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 18:24:31 2010
Return-Path: <general-return-2286-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83714 invoked from network); 27 Oct 2010 18:24:31 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 18:24:31 -0000
Received: (qmail 5647 invoked by uid 500); 27 Oct 2010 18:24:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5599 invoked by uid 500); 27 Oct 2010 18:24:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5591 invoked by uid 99); 27 Oct 2010 18:24:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 18:24:29 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=MIME_BASE64_BLANKS,MIME_BASE64_TEXT,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 65.55.88.14 is neither permitted nor denied by domain of jon.creasy@announcemedia.com)
Received: from [65.55.88.14] (HELO TX2EHSOBE008.bigfish.com) (65.55.88.14)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 18:24:24 +0000
Received: from mail105-tx2-R.bigfish.com (10.9.14.254) by
 TX2EHSOBE008.bigfish.com (10.9.40.28) with Microsoft SMTP Server id
 14.1.225.8; Wed, 27 Oct 2010 18:24:03 +0000
Received: from mail105-tx2 (localhost.localdomain [127.0.0.1])	by
 mail105-tx2-R.bigfish.com (Postfix) with ESMTP id 40852DC81C2	for
 <general@hadoop.apache.org>; Wed, 27 Oct 2010 18:24:03 +0000 (UTC)
X-SpamScore: -22
X-BigFish: VS-22(zzbb2dK1432N98dN9371Pzz1202hzz8275bh8275dhz2ei87h2a8h61h)
X-Spam-TCS-SCL: 0:0
X-FB-DOMAIN-IP-MATCH: fail
Received: from mail105-tx2 (localhost.localdomain [127.0.0.1]) by mail105-tx2
 (MessageSwitch) id 1288203842926027_11834; Wed, 27 Oct 2010 18:24:02 +0000
 (UTC)
Received: from TX2EHSMHS014.bigfish.com (unknown [10.9.14.249])	by
 mail105-tx2.bigfish.com (Postfix) with ESMTP id DFE78C80057	for
 <general@hadoop.apache.org>; Wed, 27 Oct 2010 18:24:02 +0000 (UTC)
Received: from VA3DIAHUB040.RED001.local (65.55.171.153) by
 TX2EHSMHS014.bigfish.com (10.9.99.114) with Microsoft SMTP Server (TLS) id
 14.0.482.44; Wed, 27 Oct 2010 18:24:02 +0000
Received: from VA3DIAXVS401.RED001.local ([10.32.57.16]) by
 VA3DIAHUB040.RED001.local ([10.32.21.114]) with mapi; Wed, 27 Oct 2010
 11:24:00 -0700
From: Jonathan Creasy <jon.creasy@Announcemedia.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 27 Oct 2010 11:23:06 -0700
Subject: Re: [VOTE] ZooKeeper as TLP?
Thread-Topic: [VOTE] ZooKeeper as TLP?
Thread-Index: Act2BCCZQhwqXiwuRGaIA7UbuyvF5g==
Message-ID: <07A09366-426C-426F-883C-1C93F0EAEC0E@Announcemedia.com>
References: <C8EDB9BE.44F4C%mahadev@yahoo-inc.com>
In-Reply-To: <C8EDB9BE.44F4C%mahadev@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-Reverse-DNS: smtp801.microsoftonline.com

KzENCg0KDQoNCk9uIE9jdCAyNywgMjAxMCwgYXQgMTE6MjIgQU0sICJNYWhhZGV2IEtvbmFyIiA8
bWFoYWRldkB5YWhvby1pbmMuY29tPiB3cm90ZToNCg0KPiANCj4gKzEuDQo+IA0KPiBUaGFua3MN
Cj4gbWFoYWRldg0KPiANCj4gT24gMTAvMjcvMTAgMTE6MTEgQU0sICJFcmljIFNhbW1lciIgPGVz
YW1tZXJAY2xvdWRlcmEuY29tPiB3cm90ZToNCj4gDQo+PiArMQ0KPj4gDQo+PiBPbiBXZWQsIE9j
dCAyNywgMjAxMCBhdCAyOjA0IFBNLCBQYXRyaWNrIEh1bnQgPHBodW50QGFwYWNoZS5vcmc+IHdy
b3RlOg0KPj4+IFBsZWFzZSB2b3RlIGFzIHRvIHdoZXRoZXIgeW91IHRoaW5rIFpvb0tlZXBlciBz
aG91bGQgYmVjb21lIGENCj4+PiB0b3AtbGV2ZWwgQXBhY2hlIHByb2plY3QuDQo+Pj4gDQo+Pj4g
VGhlIFpvb0tlZXBlciBkZXZlbG9wbWVudCBjb21tdW5pdHkgaGFzIGFscmVhZHkgdm90ZWQgb24g
YW5kIGFwcHJvdmVkDQo+Pj4gKCBodHRwOi8vYml0Lmx5LzlWQ29UaSApIHRoZSBmb2xsb3dpbmcg
ZHJhZnQgYm9hcmQgcmVzb2x1dGlvbi4gSXQNCj4+PiBsaXN0cyBhbGwgY3VycmVudCBhY3RpdmUg
Wm9vS2VlcGVyIGNvbW1pdHRlcnMgKHNhbnMgb25lIGNvbW1pdHRlciB3aG8NCj4+PiBoYXMgbm90
IGJlZW4gYWN0aXZlIG9uIHRoZSBwcm9qZWN0IHNpbmNlIGl0J3MgaW5jZXB0aW9uDQo+Pj4gaHR0
cDovL2JpdC5seS85d0xrVlYpIGFzIGluaXRpYWwgbWVtYmVycyBvZiB0aGUgcHJvamVjdCBtYW5h
Z2VtZW50DQo+Pj4gY29tbWl0dGVlIChQTUMpIGFuZCBteXNlbGYsIFBhdHJpY2sgSHVudCwgYXMg
dGhlIGluaXRpYWwgY2hhaXIuDQo+Pj4gDQo+Pj4gDQo+Pj4gRG8gdGhlIGdvb2QgcGVvcGxlIG9m
IEhhZG9vcCBzdXBwb3J0IHNlbmRpbmcgdGhpcyByZXF1ZXN0IG9uIHRvIHRoZQ0KPj4+IEhhZG9v
cCBQTUM/DQo+Pj4gDQo+Pj4gUmVnYXJkcywNCj4+PiANCj4+PiBQYXRyaWNrDQo+Pj4gDQo+Pj4g
LS0tLS0tLS0tLS0tDQo+Pj4gDQo+Pj4gICAgWC4gRXN0YWJsaXNoIHRoZSBBcGFjaGUgWm9vS2Vl
cGVyIFByb2plY3QNCj4+PiANCj4+PiAgICAgICBXSEVSRUFTLCB0aGUgQm9hcmQgb2YgRGlyZWN0
b3JzIGRlZW1zIGl0IHRvIGJlIGluIHRoZSBiZXN0DQo+Pj4gICAgICAgaW50ZXJlc3RzIG9mIHRo
ZSBGb3VuZGF0aW9uIGFuZCBjb25zaXN0ZW50IHdpdGggdGhlDQo+Pj4gICAgICAgRm91bmRhdGlv
bidzIHB1cnBvc2UgdG8gZXN0YWJsaXNoIGEgUHJvamVjdCBNYW5hZ2VtZW50DQo+Pj4gICAgICAg
Q29tbWl0dGVlIGNoYXJnZWQgd2l0aCB0aGUgY3JlYXRpb24gYW5kIG1haW50ZW5hbmNlIG9mDQo+
Pj4gICAgICAgb3Blbi1zb3VyY2Ugc29mdHdhcmUgcmVsYXRlZCB0byBkaXN0cmlidXRlZCBzeXN0
ZW0gY29vcmRpbmF0aW9uDQo+Pj4gICAgICAgZm9yIGRpc3RyaWJ1dGlvbiBhdCBubyBjaGFyZ2Ug
dG8gdGhlIHB1YmxpYy4NCj4+PiANCj4+PiAgICAgICBOT1csIFRIRVJFRk9SRSwgQkUgSVQgUkVT
T0xWRUQsIHRoYXQgYSBQcm9qZWN0IE1hbmFnZW1lbnQNCj4+PiAgICAgICBDb21taXR0ZWUgKFBN
QyksIHRvIGJlIGtub3duIGFzIHRoZSAiQXBhY2hlIFpvb0tlZXBlciBQcm9qZWN0IiwNCj4+PiAg
ICAgICBiZSBhbmQgaGVyZWJ5IGlzIGVzdGFibGlzaGVkIHB1cnN1YW50IHRvIEJ5bGF3cyBvZiB0
aGUNCj4+PiAgICAgICBGb3VuZGF0aW9uOyBhbmQgYmUgaXQgZnVydGhlcg0KPj4+IA0KPj4+ICAg
ICAgIFJFU09MVkVELCB0aGF0IHRoZSBBcGFjaGUgWm9vS2VlcGVyIFByb2plY3QgYmUgYW5kIGhl
cmVieSBpcw0KPj4+ICAgICAgIHJlc3BvbnNpYmxlIGZvciB0aGUgY3JlYXRpb24gYW5kIG1haW50
ZW5hbmNlIG9mIHNvZnR3YXJlDQo+Pj4gICAgICAgcmVsYXRlZCB0byBkaXN0cmlidXRlZCBzeXN0
ZW0gY29vcmRpbmF0aW9uOyBhbmQgYmUgaXQgZnVydGhlcg0KPj4+IA0KPj4+ICAgICAgIFJFU09M
VkVELCB0aGF0IHRoZSBvZmZpY2Ugb2YgIlZpY2UgUHJlc2lkZW50LCBBcGFjaGUgWm9vS2VlcGVy
IiBiZQ0KPj4+ICAgICAgIGFuZCBoZXJlYnkgaXMgY3JlYXRlZCwgdGhlIHBlcnNvbiBob2xkaW5n
IHN1Y2ggb2ZmaWNlIHRvDQo+Pj4gICAgICAgc2VydmUgYXQgdGhlIGRpcmVjdGlvbiBvZiB0aGUg
Qm9hcmQgb2YgRGlyZWN0b3JzIGFzIHRoZSBjaGFpcg0KPj4+ICAgICAgIG9mIHRoZSBBcGFjaGUg
Wm9vS2VlcGVyIFByb2plY3QsIGFuZCB0byBoYXZlIHByaW1hcnkgcmVzcG9uc2liaWxpdHkNCj4+
PiAgICAgICBmb3IgbWFuYWdlbWVudCBvZiB0aGUgcHJvamVjdHMgd2l0aGluIHRoZSBzY29wZSBv
Zg0KPj4+ICAgICAgIHJlc3BvbnNpYmlsaXR5IG9mIHRoZSBBcGFjaGUgWm9vS2VlcGVyIFByb2pl
Y3Q7IGFuZCBiZSBpdCBmdXJ0aGVyDQo+Pj4gDQo+Pj4gICAgICAgUkVTT0xWRUQsIHRoYXQgdGhl
IHBlcnNvbnMgbGlzdGVkIGltbWVkaWF0ZWx5IGJlbG93IGJlIGFuZA0KPj4+ICAgICAgIGhlcmVi
eSBhcmUgYXBwb2ludGVkIHRvIHNlcnZlIGFzIHRoZSBpbml0aWFsIG1lbWJlcnMgb2YgdGhlDQo+
Pj4gICAgICAgQXBhY2hlIFpvb0tlZXBlciBQcm9qZWN0Og0KPj4+IA0KPj4+ICAgICAgICAgKiBQ
YXRyaWNrIEh1bnQgICAgICAgICA8cGh1bnRAYXBhY2hlLm9yZz4NCj4+PiAgICAgICAgICogRmxh
dmlvIEp1bnF1ZWlyYSAgICAgPGZwakBhcGFjaGUub3JnPg0KPj4+ICAgICAgICAgKiBNYWhhZGV2
IEtvbmFyICAgICAgICA8bWFoYWRldkBhcGFjaGUub3JnPg0KPj4+ICAgICAgICAgKiBCZW5qYW1p
biBSZWVkICAgICAgICA8YnJlZWRAYXBhY2hlLm9yZz4NCj4+PiAgICAgICAgICogSGVucnkgUm9i
aW5zb24gICAgICAgPGhlbnJ5QGFwYWNoZS5vcmc+DQo+Pj4gDQo+Pj4gICAgICAgTk9XLCBUSEVS
RUZPUkUsIEJFIElUIEZVUlRIRVIgUkVTT0xWRUQsIHRoYXQgUGF0cmljayBIdW50DQo+Pj4gICAg
ICAgYmUgYXBwb2ludGVkIHRvIHRoZSBvZmZpY2Ugb2YgVmljZSBQcmVzaWRlbnQsIEFwYWNoZSBa
b29LZWVwZXIsIHRvDQo+Pj4gICAgICAgc2VydmUgaW4gYWNjb3JkYW5jZSB3aXRoIGFuZCBzdWJq
ZWN0IHRvIHRoZSBkaXJlY3Rpb24gb2YgdGhlDQo+Pj4gICAgICAgQm9hcmQgb2YgRGlyZWN0b3Jz
IGFuZCB0aGUgQnlsYXdzIG9mIHRoZSBGb3VuZGF0aW9uIHVudGlsDQo+Pj4gICAgICAgZGVhdGgs
IHJlc2lnbmF0aW9uLCByZXRpcmVtZW50LCByZW1vdmFsIG9yIGRpc3F1YWxpZmljYXRpb24sDQo+
Pj4gICAgICAgb3IgdW50aWwgYSBzdWNjZXNzb3IgaXMgYXBwb2ludGVkOyBhbmQgYmUgaXQgZnVy
dGhlcg0KPj4+IA0KPj4+ICAgICAgIFJFU09MVkVELCB0aGF0IHRoZSBpbml0aWFsIEFwYWNoZSBa
b29LZWVwZXIgUE1DIGJlIGFuZCBoZXJlYnkgaXMNCj4+PiAgICAgICB0YXNrZWQgd2l0aCB0aGUg
Y3JlYXRpb24gb2YgYSBzZXQgb2YgYnlsYXdzIGludGVuZGVkIHRvDQo+Pj4gICAgICAgZW5jb3Vy
YWdlIG9wZW4gZGV2ZWxvcG1lbnQgYW5kIGluY3JlYXNlZCBwYXJ0aWNpcGF0aW9uIGluIHRoZQ0K
Pj4+ICAgICAgIEFwYWNoZSBab29LZWVwZXIgUHJvamVjdDsgYW5kIGJlIGl0IGZ1cnRoZXINCj4+
PiANCj4+PiAgICAgICBSRVNPTFZFRCwgdGhhdCB0aGUgQXBhY2hlIFpvb0tlZXBlciBQcm9qZWN0
IGJlIGFuZCBoZXJlYnkNCj4+PiAgICAgICBpcyB0YXNrZWQgd2l0aCB0aGUgbWlncmF0aW9uIGFu
ZCByYXRpb25hbGl6YXRpb24gb2YgdGhlIEFwYWNoZQ0KPj4+ICAgICAgIEhhZG9vcCBab29LZWVw
ZXIgc3ViLXByb2plY3Q7IGFuZCBiZSBpdCBmdXJ0aGVyDQo+Pj4gDQo+Pj4gICAgICAgUkVTT0xW
RUQsIHRoYXQgYWxsIHJlc3BvbnNpYmlsaXRpZXMgcGVydGFpbmluZyB0byB0aGUgQXBhY2hlDQo+
Pj4gICAgICAgSGFkb29wIFpvb0tlZXBlciBzdWItcHJvamVjdCBlbmN1bWJlcmVkIHVwb24gdGhl
DQo+Pj4gICAgICAgQXBhY2hlIEhhZG9vcCBQcm9qZWN0IGFyZSBoZXJlYWZ0ZXIgZGlzY2hhcmdl
ZC4NCj4+PiANCj4+IA0KPj4gDQo+PiANCj4+IC0tDQo+PiBFcmljIFNhbW1lcg0KPj4gdHdpdHRl
cjogZXNhbW1lcg0KPj4gZGF0YTogd3d3LmNsb3VkZXJhLmNvbQ0KPj4gDQo+IA0KPiANCg==


From general-return-2287-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 18:26:17 2010
Return-Path: <general-return-2287-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84445 invoked from network); 27 Oct 2010 18:26:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 18:26:16 -0000
Received: (qmail 8350 invoked by uid 500); 27 Oct 2010 18:26:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8141 invoked by uid 500); 27 Oct 2010 18:26:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8133 invoked by uid 99); 27 Oct 2010 18:26:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 18:26:15 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 18:26:10 +0000
Received: from [172.21.148.202] (wlanvpn-snve-246-202.corp.yahoo.com [172.21.148.202])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id o9RIPZ7O088156
	for <general@hadoop.apache.org>; Wed, 27 Oct 2010 11:25:35 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1288203935;
	bh=yubnqvOxH5wghrCpYkYGbKsY6Wx8HEC3NSs/FNvj738=;
	h=Message-ID:Date:From:MIME-Version:To:Subject:References:
	 In-Reply-To:Content-Type:Content-Transfer-Encoding;
	b=rt/Z6BiVLwEmPm3d/w9/DxBZD2DGSmKd0Sl6UHOsnVZK4GmlgCM99U/o2gaQ6+V5k
	 tIKpC5J1kUhshL+zlHlMp0QjyQQNk/MQgD1H//39Oo22NvE370yGayMV64kV7DJ47c
	 lCR6Jbaaquno/OfqcDZ8q7eX1AMVO3NJIlDSjqec=
Message-ID: <4CC86E9F.6000109@yahoo-inc.com>
Date: Wed, 27 Oct 2010 11:25:35 -0700
From: Jakob Homan <jhoman@yahoo-inc.com>
User-Agent: Thunderbird 2.0.0.12 (Macintosh/20080213)
MIME-Version: 1.0
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: [VOTE] ZooKeeper as TLP?
References: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com> <AANLkTinHsHhyNpwMPBeN60TiLRRuCgppDR9jdG8AEgTy@mail.gmail.com>
In-Reply-To: <AANLkTinHsHhyNpwMPBeN60TiLRRuCgppDR9jdG8AEgTy@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

+1

From general-return-2288-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 18:40:18 2010
Return-Path: <general-return-2288-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88870 invoked from network); 27 Oct 2010 18:40:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 18:40:18 -0000
Received: (qmail 24649 invoked by uid 500); 27 Oct 2010 18:40:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24577 invoked by uid 500); 27 Oct 2010 18:40:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24569 invoked by uid 99); 27 Oct 2010 18:40:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 18:40:16 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 18:40:10 +0000
Received: from wlanvpn-snve-246-226.corp.yahoo.com (wlanvpn-snve-246-226.corp.yahoo.com [172.21.148.226])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id o9RId3fR082424
	for <general@hadoop.apache.org>; Wed, 27 Oct 2010 11:39:11 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1288204751;
	bh=oFGAovNE2J2J+ZDdrvRvyHLNbTi2pxx+a7zgsdBv8eg=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=EU1BULsjfoCLfy3+HKnf2It7tyEgrU1/KqI/Ocgl1OJ7wiFJuRWUib0OR77UMe/j7
	 g9J5zbDsj37VD7PvwBRYLOMd3u6Bry5zEd8XiWmAEUMFfTAOjPdMH9OvT4LtJXlLwa
	 cReOtpeBUPWGJDOkn9qMb2L1YO5vAmR6c1u3jEGs=
Message-Id: <43E3CD46-4A71-4632-AA2F-DDF43CB5C34E@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] ZooKeeper as TLP?
Date: Wed, 27 Oct 2010 11:39:03 -0700
References: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

On Oct 27, 2010, at 11:04 AM, Patrick Hunt wrote:

> Please vote as to whether you think ZooKeeper should become a
> top-level Apache project.
>

+1

Arun


From general-return-2289-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 18:46:32 2010
Return-Path: <general-return-2289-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91164 invoked from network); 27 Oct 2010 18:46:31 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 18:46:31 -0000
Received: (qmail 36181 invoked by uid 500); 27 Oct 2010 18:46:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36132 invoked by uid 500); 27 Oct 2010 18:46:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36124 invoked by uid 99); 27 Oct 2010 18:46:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 18:46:30 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of marcosluis2186@googlemail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 18:46:26 +0000
Received: by wwi17 with SMTP id 17so981090wwi.29
        for <general@hadoop.apache.org>; Wed, 27 Oct 2010 11:46:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=P7T7DwqdzrCAD2xrMVsACBn5A9SQCqcDBr3aoADs5sg=;
        b=wHZ61kmyevTQl7DEGtQ1RA/BB987kGeKSxriE11w3tjaBT6F/FVXIkvMvtLx90UBn+
         w1ypYXhf4tSiQlKsutnxzr0SWdFPM1rvAfTWUBA14EoW5+l/nA5g/k3PYX7Y1k7eHO7E
         DrguNL0UIQFwOhubgo/tlIGLeX4wjMkDqKv+o=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=vQ6jU3kJow925zTC/okEeQqS6JcNSgV6/rXhjmckPqRxzqs33G7DXdL+Koh9fMaFZ8
         5WxNh2PkVPu4/1E6HbCbBXYjGJNLcx1TQMQQ2e/AnL+ZI8MkLDRHHZBwR+fcj5mW8Ay4
         M2r6ex4trvRzA8iHiND5b00rl4q0bfbf1lHVo=
MIME-Version: 1.0
Received: by 10.227.141.201 with SMTP id n9mr2449691wbu.185.1288205165026;
 Wed, 27 Oct 2010 11:46:05 -0700 (PDT)
Received: by 10.227.147.138 with HTTP; Wed, 27 Oct 2010 11:46:04 -0700 (PDT)
In-Reply-To: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
References: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
Date: Wed, 27 Oct 2010 14:46:04 -0400
Message-ID: <AANLkTinvtR3NXeAdWdiszqRcrnkCLno86qzi_swEs9WH@mail.gmail.com>
Subject: Re: [VOTE] ZooKeeper as TLP?
From: Marcos Luis Ortiz Valmaseda <marcosluis2186@googlemail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636831db24260d904939da3e0

--001636831db24260d904939da3e0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

+1
--=20
Ing. Marcos Lu=EDs Ort=EDz Valmaseda
Data Lover(RDBMS and NOSQL Movement) && System Engineer
Linux User # 418229

http://it.toolbox.com/blogs/sql-apprentice
http://www.linkedin.com/in/marcosluis2186/
http://github.com/marcosluis2186

--001636831db24260d904939da3e0--

From general-return-2290-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 18:51:52 2010
Return-Path: <general-return-2290-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 92138 invoked from network); 27 Oct 2010 18:51:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 18:51:51 -0000
Received: (qmail 44411 invoked by uid 500); 27 Oct 2010 18:51:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44351 invoked by uid 500); 27 Oct 2010 18:51:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44343 invoked by uid 99); 27 Oct 2010 18:51:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 18:51:50 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 18:51:46 +0000
Received: by wyg36 with SMTP id 36so1104704wyg.35
        for <general@hadoop.apache.org>; Wed, 27 Oct 2010 11:51:24 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.227.157.148 with SMTP id b20mr2351312wbx.14.1288205484347;
 Wed, 27 Oct 2010 11:51:24 -0700 (PDT)
Received: by 10.227.43.11 with HTTP; Wed, 27 Oct 2010 11:51:24 -0700 (PDT)
In-Reply-To: <AANLkTinvtR3NXeAdWdiszqRcrnkCLno86qzi_swEs9WH@mail.gmail.com>
References: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
	<AANLkTinvtR3NXeAdWdiszqRcrnkCLno86qzi_swEs9WH@mail.gmail.com>
Date: Wed, 27 Oct 2010 11:51:24 -0700
Message-ID: <AANLkTi=GqbVr528XDHYsZP0MepZub281qH8SWst+Rp6p@mail.gmail.com>
Subject: Re: [VOTE] ZooKeeper as TLP?
From: Henry Robinson <henry@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163683397a4ad32404939db6f4

--00163683397a4ad32404939db6f4
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

+1

On 27 October 2010 11:46, Marcos Luis Ortiz Valmaseda <
marcosluis2186@googlemail.com> wrote:

> +1
> --
> Ing. Marcos Lu=EDs Ort=EDz Valmaseda
> Data Lover(RDBMS and NOSQL Movement) && System Engineer
> Linux User # 418229
>
> http://it.toolbox.com/blogs/sql-apprentice
> http://www.linkedin.com/in/marcosluis2186/
> http://github.com/marcosluis2186
>



--=20
Henry Robinson
Software Engineer
Cloudera
415-994-6679

--00163683397a4ad32404939db6f4--

From general-return-2291-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 18:57:34 2010
Return-Path: <general-return-2291-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93081 invoked from network); 27 Oct 2010 18:57:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 18:57:33 -0000
Received: (qmail 52177 invoked by uid 500); 27 Oct 2010 18:56:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52071 invoked by uid 500); 27 Oct 2010 18:56:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52063 invoked by uid 99); 27 Oct 2010 18:56:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 18:56:49 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.102 as permitted sender)
Received: from [17.148.16.102] (HELO asmtpout027.mac.com) (17.148.16.102)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 18:56:41 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp027.mac.com (Oracle Communications Messaging Exchange Server 7u4-18.01
 64bit (built Jul 15 2010)) with ESMTPSA id <0LAY00MH7PXJG470@asmtp027.mac.com>
 for general@hadoop.apache.org; Wed, 27 Oct 2010 11:56:09 -0700 (PDT)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2010-10-27_09:2010-10-27,2010-10-27,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1004200000 definitions=main-1010270097
Subject: Re: [VOTE] ZooKeeper as TLP?
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
Date: Wed, 27 Oct 2010 11:56:07 -0700
Message-id: <8168FC59-75CC-4FD4-A27A-2F0F49964F3D@mac.com>
References: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1081)


On Oct 27, 2010, at 11:04 AM, Patrick Hunt wrote:

> Please vote as to whether you think ZooKeeper should become a
> top-level Apache project.

+1

From general-return-2292-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 20:43:56 2010
Return-Path: <general-return-2292-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33906 invoked from network); 27 Oct 2010 20:43:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 20:43:55 -0000
Received: (qmail 79795 invoked by uid 500); 27 Oct 2010 20:43:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79719 invoked by uid 500); 27 Oct 2010 20:43:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 79710 invoked by uid 99); 27 Oct 2010 20:43:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 20:43:54 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 27 Oct 2010 20:43:51 +0000
Received: (qmail 33731 invoked by uid 99); 27 Oct 2010 20:43:30 -0000
Received: from localhost.apache.org (HELO [192.168.168.130]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 20:43:30 +0000
Message-ID: <4CC88EF1.4070209@apache.org>
Date: Wed, 27 Oct 2010 13:43:29 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.11) Gecko/20101006 Thunderbird/3.1.5
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] ZooKeeper as TLP?
References: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
In-Reply-To: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

+1

On 10/27/2010 11:04 AM, Patrick Hunt wrote:
> Please vote as to whether you think ZooKeeper should become a
> top-level Apache project.

From general-return-2293-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 21:28:29 2010
Return-Path: <general-return-2293-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 51881 invoked from network); 27 Oct 2010 21:28:29 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 21:28:29 -0000
Received: (qmail 62268 invoked by uid 500); 27 Oct 2010 21:28:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62224 invoked by uid 500); 27 Oct 2010 21:28:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62216 invoked by uid 99); 27 Oct 2010 21:28:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 21:28:27 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 21:28:19 +0000
Received: by iwn7 with SMTP id 7so1382889iwn.35
        for <general@hadoop.apache.org>; Wed, 27 Oct 2010 14:27:58 -0700 (PDT)
Received: by 10.42.220.134 with SMTP id hy6mr1822563icb.61.1288214877196;
        Wed, 27 Oct 2010 14:27:57 -0700 (PDT)
Received: from tp (173-164-176-117-SFBA.hfc.comcastbusiness.net [173.164.176.117])
        by mx.google.com with ESMTPS id c40sm116791vcs.1.2010.10.27.14.27.55
        (version=SSLv3 cipher=RC4-MD5);
        Wed, 27 Oct 2010 14:27:56 -0700 (PDT)
Sender: Konstantin Boudnik <cos@boudnik.org>
Date: Wed, 27 Oct 2010 14:27:51 -0700
From: Konstantin Boudnik <cos@apache.org>
To: general@hadoop.apache.org
Subject: Re: [VOTE] ZooKeeper as TLP?
Message-ID: <20101027212751.GA4207@tp>
References: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="tKW2IUtsqtDRztdT"
Content-Disposition: inline
In-Reply-To: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
X-Organization: It's something of 'Cos
X-PGP-Key: http://www.boudnik.org/~cos/pubkey.asc
User-Agent: Mutt/1.5.20 (2009-06-14)
X-Virus-Checked: Checked by ClamAV on apache.org

--tKW2IUtsqtDRztdT
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

+1

On Wed, Oct 27, 2010 at 11:04AM, Patrick Hunt wrote:
> Please vote as to whether you think ZooKeeper should become a
> top-level Apache project.
>=20
> The ZooKeeper development community has already voted on and approved
> ( http://bit.ly/9VCoTi ) the following draft board resolution. It
> lists all current active ZooKeeper committers (sans one committer who
> has not been active on the project since it's inception
> http://bit.ly/9wLkVV) as initial members of the project management
> committee (PMC) and myself, Patrick Hunt, as the initial chair.
>=20
>=20
> Do the good people of Hadoop support sending this request on to the
> Hadoop PMC?
>=20
> Regards,
>=20
> Patrick
>=20
> ------------
>=20
>     X. Establish the Apache ZooKeeper Project
>=20
>        WHEREAS, the Board of Directors deems it to be in the best
>        interests of the Foundation and consistent with the
>        Foundation's purpose to establish a Project Management
>        Committee charged with the creation and maintenance of
>        open-source software related to distributed system coordination
>        for distribution at no charge to the public.
>=20
>        NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>        Committee (PMC), to be known as the "Apache ZooKeeper Project",
>        be and hereby is established pursuant to Bylaws of the
>        Foundation; and be it further
>=20
>        RESOLVED, that the Apache ZooKeeper Project be and hereby is
>        responsible for the creation and maintenance of software
>        related to distributed system coordination; and be it further
>=20
>        RESOLVED, that the office of "Vice President, Apache ZooKeeper" be
>        and hereby is created, the person holding such office to
>        serve at the direction of the Board of Directors as the chair
>        of the Apache ZooKeeper Project, and to have primary responsibility
>        for management of the projects within the scope of
>        responsibility of the Apache ZooKeeper Project; and be it further
>=20
>        RESOLVED, that the persons listed immediately below be and
>        hereby are appointed to serve as the initial members of the
>        Apache ZooKeeper Project:
>=20
>          * Patrick Hunt         <phunt@apache.org>
>          * Flavio Junqueira     <fpj@apache.org>
>          * Mahadev Konar        <mahadev@apache.org>
>          * Benjamin Reed        <breed@apache.org>
>          * Henry Robinson       <henry@apache.org>
>=20
>        NOW, THEREFORE, BE IT FURTHER RESOLVED, that Patrick Hunt
>        be appointed to the office of Vice President, Apache ZooKeeper, to
>        serve in accordance with and subject to the direction of the
>        Board of Directors and the Bylaws of the Foundation until
>        death, resignation, retirement, removal or disqualification,
>        or until a successor is appointed; and be it further
>=20
>        RESOLVED, that the initial Apache ZooKeeper PMC be and hereby is
>        tasked with the creation of a set of bylaws intended to
>        encourage open development and increased participation in the
>        Apache ZooKeeper Project; and be it further
>=20
>        RESOLVED, that the Apache ZooKeeper Project be and hereby
>        is tasked with the migration and rationalization of the Apache
>        Hadoop ZooKeeper sub-project; and be it further
>=20
>        RESOLVED, that all responsibilities pertaining to the Apache
>        Hadoop ZooKeeper sub-project encumbered upon the
>        Apache Hadoop Project are hereafter discharged.

--tKW2IUtsqtDRztdT
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEARECAAYFAkzImVcACgkQIg9pgB8n5iK69gCeJFZCypkfJhorksW8y/mWonW2
Bf0AoJhh9Z9v3UWznF5+Hj9GStQ88pn3
=SPL8
-----END PGP SIGNATURE-----

--tKW2IUtsqtDRztdT--

From general-return-2294-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 22:16:47 2010
Return-Path: <general-return-2294-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70510 invoked from network); 27 Oct 2010 22:16:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 22:16:47 -0000
Received: (qmail 40733 invoked by uid 500); 27 Oct 2010 22:16:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40674 invoked by uid 500); 27 Oct 2010 22:16:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40666 invoked by uid 99); 27 Oct 2010 22:16:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 22:16:46 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=MIME_QP_LONG_LINE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 22:16:41 +0000
Received: by gyg4 with SMTP id 4so935296gyg.35
        for <general@hadoop.apache.org>; Wed, 27 Oct 2010 15:16:20 -0700 (PDT)
Received: by 10.100.112.20 with SMTP id k20mr8394097anc.60.1288217780056;
        Wed, 27 Oct 2010 15:16:20 -0700 (PDT)
Received: from [10.126.6.178] (mobile-166-137-138-052.mycingular.net [166.137.138.52])
        by mx.google.com with ESMTPS id b25sm260912anb.3.2010.10.27.15.16.17
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Wed, 27 Oct 2010 15:16:19 -0700 (PDT)
Subject: Re: [VOTE] ZooKeeper as TLP?
References: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com> <AANLkTinvtR3NXeAdWdiszqRcrnkCLno86qzi_swEs9WH@mail.gmail.com>
From: Ian Holsman <hadoop@holsman.net>
Content-Type: text/plain;
	charset=utf-8
X-Mailer: iPhone Mail (8B117)
In-Reply-To: <AANLkTinvtR3NXeAdWdiszqRcrnkCLno86qzi_swEs9WH@mail.gmail.com>
Message-Id: <95F1E74D-FE95-446A-99C9-BDB7D7A76466@holsman.net>
Date: Wed, 27 Oct 2010 18:15:13 -0400
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (iPhone Mail 8B117)

+1

---
Ian Holsman - 703 879-3128



On 27/10/2010, at 2:46 PM, Marcos Luis Ortiz Valmaseda <marcosluis2186@googl=
email.com> wrote:

> +1
> --=20
> Ing. Marcos Lu=C3=ADs Ort=C3=ADz Valmaseda
> Data Lover(RDBMS and NOSQL Movement) && System Engineer
> Linux User # 418229
>=20
> http://it.toolbox.com/blogs/sql-apprentice
> http://www.linkedin.com/in/marcosluis2186/
> http://github.com/marcosluis2186

From general-return-2295-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 22:42:57 2010
Return-Path: <general-return-2295-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83145 invoked from network); 27 Oct 2010 22:42:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 22:42:56 -0000
Received: (qmail 77394 invoked by uid 500); 27 Oct 2010 22:42:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77347 invoked by uid 500); 27 Oct 2010 22:42:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77338 invoked by uid 99); 27 Oct 2010 22:42:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 22:42:55 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of msegel@navteq.com designates 65.167.11.240 as permitted sender)
Received: from [65.167.11.240] (HELO xmailchi.navteq.com) (65.167.11.240)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 22:42:49 +0000
Received: from chicago-spamfw.navteq.com (imailchi [10.8.5.161])
	by xmailchi.navteq.com (8.13.8/8.13.6) with ESMTP id o9RMFdW0009537
	for <general@hadoop.apache.org>; Wed, 27 Oct 2010 17:15:39 -0500
Received: from imailchi.hq.navteq.com (localhost [127.0.0.1])
	by chicago-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id EF2BD3A1F6DA
	for <general@hadoop.apache.org>; Wed, 27 Oct 2010 17:42:27 -0500 (CDT)
Received: from imailchi.hq.navteq.com (imailchi.hq.navteq.com [10.8.120.104]) by chicago-spamfw.navteq.com with ESMTP id F3I0HfEZSkb9c0Pq for <general@hadoop.apache.org>; Wed, 27 Oct 2010 17:42:27 -0500 (CDT)
Received: from hq-ex-ht01.ad.navteq.com (hq-ex-ht01.ad.navteq.com [10.8.222.51])
	by imailchi.hq.navteq.com (8.12.9/8.12.9) with ESMTP id o9RMgRtf000843
	for <general@hadoop.apache.org>; Wed, 27 Oct 2010 17:42:27 -0500
Received: from hq-ex-mb02.ad.navteq.com ([fe80::880d:318e:e860:a8db]) by
 hq-ex-ht01.ad.navteq.com ([fe80::55a6:5b94:5c00:fe39%12]) with mapi; Wed, 27
 Oct 2010 17:42:27 -0500
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 27 Oct 2010 17:42:20 -0500
Subject: RE: [VOTE] ZooKeeper as TLP?
Thread-Topic: [VOTE] ZooKeeper as TLP?
Thread-Index: Act2JKYX2UTsZdMaRWeeJxJliLcyQwAA5CsW
Message-ID: <ABC24175AFD3BE4DA15F4CD375ED413D063993F18B@hq-ex-mb02.ad.navteq.com>
References: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
 <AANLkTinvtR3NXeAdWdiszqRcrnkCLno86qzi_swEs9WH@mail.gmail.com>,<95F1E74D-FE95-446A-99C9-BDB7D7A76466@holsman.net>
In-Reply-To: <95F1E74D-FE95-446A-99C9-BDB7D7A76466@holsman.net>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1
X-Virus-Checked: Checked by ClamAV on apache.org

KzENCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18NCkZyb206IElhbiBI
b2xzbWFuIFtoYWRvb3BAaG9sc21hbi5uZXRdDQpTZW50OiBXZWRuZXNkYXksIE9jdG9iZXIgMjcs
IDIwMTAgNToxNSBQTQ0KVG86IGdlbmVyYWxAaGFkb29wLmFwYWNoZS5vcmcNClN1YmplY3Q6IFJl
OiBbVk9URV0gWm9vS2VlcGVyIGFzIFRMUD8NCg0KKzENCg0KLS0tDQpJYW4gSG9sc21hbiAtIDcw
MyA4NzktMzEyOA0KDQoNCg0KT24gMjcvMTAvMjAxMCwgYXQgMjo0NiBQTSwgTWFyY29zIEx1aXMg
T3J0aXogVmFsbWFzZWRhIDxtYXJjb3NsdWlzMjE4NkBnb29nbGVtYWlsLmNvbT4gd3JvdGU6DQoN
Cj4gKzENCj4gLS0NCj4gSW5nLiBNYXJjb3MgTHXtcyBPcnTteiBWYWxtYXNlZGENCj4gRGF0YSBM
b3ZlcihSREJNUyBhbmQgTk9TUUwgTW92ZW1lbnQpICYmIFN5c3RlbSBFbmdpbmVlcg0KPiBMaW51
eCBVc2VyICMgNDE4MjI5DQo+DQo+IGh0dHA6Ly9pdC50b29sYm94LmNvbS9ibG9ncy9zcWwtYXBw
cmVudGljZQ0KPiBodHRwOi8vd3d3LmxpbmtlZGluLmNvbS9pbi9tYXJjb3NsdWlzMjE4Ni8NCj4g
aHR0cDovL2dpdGh1Yi5jb20vbWFyY29zbHVpczIxODYNCg0KDQpUaGUgaW5mb3JtYXRpb24gY29u
dGFpbmVkIGluIHRoaXMgY29tbXVuaWNhdGlvbiBtYXkgYmUgQ09ORklERU5USUFMIGFuZCBpcyBp
bnRlbmRlZCBvbmx5IGZvciB0aGUgdXNlIG9mIHRoZSByZWNpcGllbnQocykgbmFtZWQgYWJvdmUu
ICBJZiB5b3UgYXJlIG5vdCB0aGUgaW50ZW5kZWQgcmVjaXBpZW50LCB5b3UgYXJlIGhlcmVieSBu
b3RpZmllZCB0aGF0IGFueSBkaXNzZW1pbmF0aW9uLCBkaXN0cmlidXRpb24sIG9yIGNvcHlpbmcg
b2YgdGhpcyBjb21tdW5pY2F0aW9uLCBvciBhbnkgb2YgaXRzIGNvbnRlbnRzLCBpcyBzdHJpY3Rs
eSBwcm9oaWJpdGVkLiAgSWYgeW91IGhhdmUgcmVjZWl2ZWQgdGhpcyBjb21tdW5pY2F0aW9uIGlu
IGVycm9yLCBwbGVhc2Ugbm90aWZ5IHRoZSBzZW5kZXIgYW5kIGRlbGV0ZS9kZXN0cm95IHRoZSBv
cmlnaW5hbCBtZXNzYWdlIGFuZCBhbnkgY29weSBvZiBpdCBmcm9tIHlvdXIgY29tcHV0ZXIgb3Ig
cGFwZXIgZmlsZXMuDQo=

From general-return-2296-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 22:55:43 2010
Return-Path: <general-return-2296-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 86110 invoked from network); 27 Oct 2010 22:55:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 22:55:42 -0000
Received: (qmail 90815 invoked by uid 500); 27 Oct 2010 22:55:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90761 invoked by uid 500); 27 Oct 2010 22:55:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90753 invoked by uid 99); 27 Oct 2010 22:55:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 22:55:41 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of tom.e.white@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 22:55:34 +0000
Received: by wyg36 with SMTP id 36so1336450wyg.35
        for <general@hadoop.apache.org>; Wed, 27 Oct 2010 15:55:14 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=FeNd0juL1ZtN2MxWiJ8Gx+XE66iWUsVzx2WApLiyXv8=;
        b=V67yHb3DUvdRGxe9EJnJyW1y8vPJRGMXg4OsNVaw+komety3f+h4z75ym0xJCeyg+U
         gQM0rStzcWZVnGADUe+LUgNEg2MKTRoeJqP0mLQh40gLtmvtuzCowjrIi9XzdpFbrqsR
         wzsQBcz35QLIRzw0ztfhpZZzXF8i+pXgDnoXA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=k4vlachP2H/vC3bX1Cs+wqVAcsipRqrGkNQmMVC/yWyh9pOSv+QNg6rhvp5FKo51l2
         r+j3Gtn6zk7Drl8sb+U+AQlpKeanpS5m9MkbsOkYK523UcJjJSI7X6DwmOvYJPMZbfdc
         2L+b90ApGl4CFUmXAHk86HEs/SIdHlF00dT9c=
Received: by 10.216.7.205 with SMTP id 55mr1494682wep.96.1288220114566; Wed,
 27 Oct 2010 15:55:14 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.59.6 with HTTP; Wed, 27 Oct 2010 15:54:54 -0700 (PDT)
In-Reply-To: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
References: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
From: Tom White <tom.e.white@gmail.com>
Date: Wed, 27 Oct 2010 15:54:54 -0700
Message-ID: <AANLkTinNudLEj1TaPBy=Z7bQrNbfgQeE9VSLgbqRDtb=@mail.gmail.com>
Subject: Re: [VOTE] ZooKeeper as TLP?
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Tom

On Wed, Oct 27, 2010 at 11:04 AM, Patrick Hunt <phunt@apache.org> wrote:
> Please vote as to whether you think ZooKeeper should become a
> top-level Apache project.
>
> The ZooKeeper development community has already voted on and approved
> ( http://bit.ly/9VCoTi ) the following draft board resolution. It
> lists all current active ZooKeeper committers (sans one committer who
> has not been active on the project since it's inception
> http://bit.ly/9wLkVV) as initial members of the project management
> committee (PMC) and myself, Patrick Hunt, as the initial chair.
>
>
> Do the good people of Hadoop support sending this request on to the
> Hadoop PMC?
>
> Regards,
>
> Patrick
>
> ------------
>
> =A0 =A0X. Establish the Apache ZooKeeper Project
>
> =A0 =A0 =A0 WHEREAS, the Board of Directors deems it to be in the best
> =A0 =A0 =A0 interests of the Foundation and consistent with the
> =A0 =A0 =A0 Foundation's purpose to establish a Project Management
> =A0 =A0 =A0 Committee charged with the creation and maintenance of
> =A0 =A0 =A0 open-source software related to distributed system coordinati=
on
> =A0 =A0 =A0 for distribution at no charge to the public.
>
> =A0 =A0 =A0 NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> =A0 =A0 =A0 Committee (PMC), to be known as the "Apache ZooKeeper Project=
",
> =A0 =A0 =A0 be and hereby is established pursuant to Bylaws of the
> =A0 =A0 =A0 Foundation; and be it further
>
> =A0 =A0 =A0 RESOLVED, that the Apache ZooKeeper Project be and hereby is
> =A0 =A0 =A0 responsible for the creation and maintenance of software
> =A0 =A0 =A0 related to distributed system coordination; and be it further
>
> =A0 =A0 =A0 RESOLVED, that the office of "Vice President, Apache ZooKeepe=
r" be
> =A0 =A0 =A0 and hereby is created, the person holding such office to
> =A0 =A0 =A0 serve at the direction of the Board of Directors as the chair
> =A0 =A0 =A0 of the Apache ZooKeeper Project, and to have primary responsi=
bility
> =A0 =A0 =A0 for management of the projects within the scope of
> =A0 =A0 =A0 responsibility of the Apache ZooKeeper Project; and be it fur=
ther
>
> =A0 =A0 =A0 RESOLVED, that the persons listed immediately below be and
> =A0 =A0 =A0 hereby are appointed to serve as the initial members of the
> =A0 =A0 =A0 Apache ZooKeeper Project:
>
> =A0 =A0 =A0 =A0 * Patrick Hunt =A0 =A0 =A0 =A0 <phunt@apache.org>
> =A0 =A0 =A0 =A0 * Flavio Junqueira =A0 =A0 <fpj@apache.org>
> =A0 =A0 =A0 =A0 * Mahadev Konar =A0 =A0 =A0 =A0<mahadev@apache.org>
> =A0 =A0 =A0 =A0 * Benjamin Reed =A0 =A0 =A0 =A0<breed@apache.org>
> =A0 =A0 =A0 =A0 * Henry Robinson =A0 =A0 =A0 <henry@apache.org>
>
> =A0 =A0 =A0 NOW, THEREFORE, BE IT FURTHER RESOLVED, that Patrick Hunt
> =A0 =A0 =A0 be appointed to the office of Vice President, Apache ZooKeepe=
r, to
> =A0 =A0 =A0 serve in accordance with and subject to the direction of the
> =A0 =A0 =A0 Board of Directors and the Bylaws of the Foundation until
> =A0 =A0 =A0 death, resignation, retirement, removal or disqualification,
> =A0 =A0 =A0 or until a successor is appointed; and be it further
>
> =A0 =A0 =A0 RESOLVED, that the initial Apache ZooKeeper PMC be and hereby=
 is
> =A0 =A0 =A0 tasked with the creation of a set of bylaws intended to
> =A0 =A0 =A0 encourage open development and increased participation in the
> =A0 =A0 =A0 Apache ZooKeeper Project; and be it further
>
> =A0 =A0 =A0 RESOLVED, that the Apache ZooKeeper Project be and hereby
> =A0 =A0 =A0 is tasked with the migration and rationalization of the Apach=
e
> =A0 =A0 =A0 Hadoop ZooKeeper sub-project; and be it further
>
> =A0 =A0 =A0 RESOLVED, that all responsibilities pertaining to the Apache
> =A0 =A0 =A0 Hadoop ZooKeeper sub-project encumbered upon the
> =A0 =A0 =A0 Apache Hadoop Project are hereafter discharged.
>

From general-return-2297-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 23:23:51 2010
Return-Path: <general-return-2297-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95545 invoked from network); 27 Oct 2010 23:23:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 23:23:51 -0000
Received: (qmail 12454 invoked by uid 500); 27 Oct 2010 23:23:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12294 invoked by uid 500); 27 Oct 2010 23:23:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12286 invoked by uid 99); 27 Oct 2010 23:23:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 23:23:49 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=MIME_BASE64_BLANKS,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of SRS0=7Aw9rP=R5=2b2collaboration.com=maricela@srs.bis.na.blackberry.com designates 216.9.248.63 as permitted sender)
Received: from [216.9.248.63] (HELO smtp21.bis.na.blackberry.com) (216.9.248.63)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 23:23:44 +0000
Received: from bda834.bisx.prod.on.blackberry (bda834.bisx.prod.on.blackberry [172.20.219.144])
	by srs.bis.na.blackberry.com (8.13.7 TEAMON/8.13.7) with ESMTP id o9RNNKNd004690;
	Wed, 27 Oct 2010 23:23:20 GMT
Received: from bda834.bisx.prod.on.blackberry (localhost.localdomain [127.0.0.1])
	by bda834.bisx.prod.on.blackberry (8.13.7 TEAMON/8.13.7) with ESMTP id o9RNNKMw030888;
	Wed, 27 Oct 2010 23:23:20 GMT
X-rim-org-msg-ref-id: 1897115290
Message-ID: <1897115290-1288221800-cardhu_decombobulator_blackberry.rim.net-404364731-@bda358.bisx.prod.on.blackberry>
Content-Transfer-Encoding: base64
Reply-To: maricela@2b2collaboration.com
X-Priority: Normal
References: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com><AANLkTinNudLEj1TaPBy=Z7bQrNbfgQeE9VSLgbqRDtb=@mail.gmail.com>
In-Reply-To: <AANLkTinNudLEj1TaPBy=Z7bQrNbfgQeE9VSLgbqRDtb=@mail.gmail.com>
Sensitivity: Normal
Importance: Normal
To: general@hadoop.apache.org
Subject: Re: [VOTE] ZooKeeper as TLP?
From: "Maricela Morales" <maricela@2b2collaboration.com>
Date: Wed, 27 Oct 2010 23:25:14 +0000
Content-Type: text/plain; charset="Windows-1252"
MIME-Version: 1.0

KzENCk1lbnNhamUgZW52aWFkbyBkZXNkZSBtaSBCbGFja0JlcnJ5rg0KDQotLS0tLU9yaWdpbmFs
IE1lc3NhZ2UtLS0tLQ0KRnJvbTogVG9tIFdoaXRlIDx0b20uZS53aGl0ZUBnbWFpbC5jb20+DQpE
YXRlOiBXZWQsIDI3IE9jdCAyMDEwIDE1OjU0OjU0IA0KVG86IDxnZW5lcmFsQGhhZG9vcC5hcGFj
aGUub3JnPg0KUmVwbHktVG86IGdlbmVyYWxAaGFkb29wLmFwYWNoZS5vcmcNClN1YmplY3Q6IFJl
OiBbVk9URV0gWm9vS2VlcGVyIGFzIFRMUD8NCg0KKzENCg0KVG9tDQoNCk9uIFdlZCwgT2N0IDI3
LCAyMDEwIGF0IDExOjA0IEFNLCBQYXRyaWNrIEh1bnQgPHBodW50QGFwYWNoZS5vcmc+IHdyb3Rl
Og0KPiBQbGVhc2Ugdm90ZSBhcyB0byB3aGV0aGVyIHlvdSB0aGluayBab29LZWVwZXIgc2hvdWxk
IGJlY29tZSBhDQo+IHRvcC1sZXZlbCBBcGFjaGUgcHJvamVjdC4NCj4NCj4gVGhlIFpvb0tlZXBl
ciBkZXZlbG9wbWVudCBjb21tdW5pdHkgaGFzIGFscmVhZHkgdm90ZWQgb24gYW5kIGFwcHJvdmVk
DQo+ICggaHR0cDovL2JpdC5seS85VkNvVGkgKSB0aGUgZm9sbG93aW5nIGRyYWZ0IGJvYXJkIHJl
c29sdXRpb24uIEl0DQo+IGxpc3RzIGFsbCBjdXJyZW50IGFjdGl2ZSBab29LZWVwZXIgY29tbWl0
dGVycyAoc2FucyBvbmUgY29tbWl0dGVyIHdobw0KPiBoYXMgbm90IGJlZW4gYWN0aXZlIG9uIHRo
ZSBwcm9qZWN0IHNpbmNlIGl0J3MgaW5jZXB0aW9uDQo+IGh0dHA6Ly9iaXQubHkvOXdMa1ZWKSBh
cyBpbml0aWFsIG1lbWJlcnMgb2YgdGhlIHByb2plY3QgbWFuYWdlbWVudA0KPiBjb21taXR0ZWUg
KFBNQykgYW5kIG15c2VsZiwgUGF0cmljayBIdW50LCBhcyB0aGUgaW5pdGlhbCBjaGFpci4NCj4N
Cj4NCj4gRG8gdGhlIGdvb2QgcGVvcGxlIG9mIEhhZG9vcCBzdXBwb3J0IHNlbmRpbmcgdGhpcyBy
ZXF1ZXN0IG9uIHRvIHRoZQ0KPiBIYWRvb3AgUE1DPw0KPg0KPiBSZWdhcmRzLA0KPg0KPiBQYXRy
aWNrDQo+DQo+IC0tLS0tLS0tLS0tLQ0KPg0KPiCgIKBYLiBFc3RhYmxpc2ggdGhlIEFwYWNoZSBa
b29LZWVwZXIgUHJvamVjdA0KPg0KPiCgIKAgoCBXSEVSRUFTLCB0aGUgQm9hcmQgb2YgRGlyZWN0
b3JzIGRlZW1zIGl0IHRvIGJlIGluIHRoZSBiZXN0DQo+IKAgoCCgIGludGVyZXN0cyBvZiB0aGUg
Rm91bmRhdGlvbiBhbmQgY29uc2lzdGVudCB3aXRoIHRoZQ0KPiCgIKAgoCBGb3VuZGF0aW9uJ3Mg
cHVycG9zZSB0byBlc3RhYmxpc2ggYSBQcm9qZWN0IE1hbmFnZW1lbnQNCj4goCCgIKAgQ29tbWl0
dGVlIGNoYXJnZWQgd2l0aCB0aGUgY3JlYXRpb24gYW5kIG1haW50ZW5hbmNlIG9mDQo+IKAgoCCg
IG9wZW4tc291cmNlIHNvZnR3YXJlIHJlbGF0ZWQgdG8gZGlzdHJpYnV0ZWQgc3lzdGVtIGNvb3Jk
aW5hdGlvbg0KPiCgIKAgoCBmb3IgZGlzdHJpYnV0aW9uIGF0IG5vIGNoYXJnZSB0byB0aGUgcHVi
bGljLg0KPg0KPiCgIKAgoCBOT1csIFRIRVJFRk9SRSwgQkUgSVQgUkVTT0xWRUQsIHRoYXQgYSBQ
cm9qZWN0IE1hbmFnZW1lbnQNCj4goCCgIKAgQ29tbWl0dGVlIChQTUMpLCB0byBiZSBrbm93biBh
cyB0aGUgIkFwYWNoZSBab29LZWVwZXIgUHJvamVjdCIsDQo+IKAgoCCgIGJlIGFuZCBoZXJlYnkg
aXMgZXN0YWJsaXNoZWQgcHVyc3VhbnQgdG8gQnlsYXdzIG9mIHRoZQ0KPiCgIKAgoCBGb3VuZGF0
aW9uOyBhbmQgYmUgaXQgZnVydGhlcg0KPg0KPiCgIKAgoCBSRVNPTFZFRCwgdGhhdCB0aGUgQXBh
Y2hlIFpvb0tlZXBlciBQcm9qZWN0IGJlIGFuZCBoZXJlYnkgaXMNCj4goCCgIKAgcmVzcG9uc2li
bGUgZm9yIHRoZSBjcmVhdGlvbiBhbmQgbWFpbnRlbmFuY2Ugb2Ygc29mdHdhcmUNCj4goCCgIKAg
cmVsYXRlZCB0byBkaXN0cmlidXRlZCBzeXN0ZW0gY29vcmRpbmF0aW9uOyBhbmQgYmUgaXQgZnVy
dGhlcg0KPg0KPiCgIKAgoCBSRVNPTFZFRCwgdGhhdCB0aGUgb2ZmaWNlIG9mICJWaWNlIFByZXNp
ZGVudCwgQXBhY2hlIFpvb0tlZXBlciIgYmUNCj4goCCgIKAgYW5kIGhlcmVieSBpcyBjcmVhdGVk
LCB0aGUgcGVyc29uIGhvbGRpbmcgc3VjaCBvZmZpY2UgdG8NCj4goCCgIKAgc2VydmUgYXQgdGhl
IGRpcmVjdGlvbiBvZiB0aGUgQm9hcmQgb2YgRGlyZWN0b3JzIGFzIHRoZSBjaGFpcg0KPiCgIKAg
oCBvZiB0aGUgQXBhY2hlIFpvb0tlZXBlciBQcm9qZWN0LCBhbmQgdG8gaGF2ZSBwcmltYXJ5IHJl
c3BvbnNpYmlsaXR5DQo+IKAgoCCgIGZvciBtYW5hZ2VtZW50IG9mIHRoZSBwcm9qZWN0cyB3aXRo
aW4gdGhlIHNjb3BlIG9mDQo+IKAgoCCgIHJlc3BvbnNpYmlsaXR5IG9mIHRoZSBBcGFjaGUgWm9v
S2VlcGVyIFByb2plY3Q7IGFuZCBiZSBpdCBmdXJ0aGVyDQo+DQo+IKAgoCCgIFJFU09MVkVELCB0
aGF0IHRoZSBwZXJzb25zIGxpc3RlZCBpbW1lZGlhdGVseSBiZWxvdyBiZSBhbmQNCj4goCCgIKAg
aGVyZWJ5IGFyZSBhcHBvaW50ZWQgdG8gc2VydmUgYXMgdGhlIGluaXRpYWwgbWVtYmVycyBvZiB0
aGUNCj4goCCgIKAgQXBhY2hlIFpvb0tlZXBlciBQcm9qZWN0Og0KPg0KPiCgIKAgoCCgICogUGF0
cmljayBIdW50IKAgoCCgIKAgPHBodW50QGFwYWNoZS5vcmc+DQo+IKAgoCCgIKAgKiBGbGF2aW8g
SnVucXVlaXJhIKAgoCA8ZnBqQGFwYWNoZS5vcmc+DQo+IKAgoCCgIKAgKiBNYWhhZGV2IEtvbmFy
IKAgoCCgIKA8bWFoYWRldkBhcGFjaGUub3JnPg0KPiCgIKAgoCCgICogQmVuamFtaW4gUmVlZCCg
IKAgoCCgPGJyZWVkQGFwYWNoZS5vcmc+DQo+IKAgoCCgIKAgKiBIZW5yeSBSb2JpbnNvbiCgIKAg
oCA8aGVucnlAYXBhY2hlLm9yZz4NCj4NCj4goCCgIKAgTk9XLCBUSEVSRUZPUkUsIEJFIElUIEZV
UlRIRVIgUkVTT0xWRUQsIHRoYXQgUGF0cmljayBIdW50DQo+IKAgoCCgIGJlIGFwcG9pbnRlZCB0
byB0aGUgb2ZmaWNlIG9mIFZpY2UgUHJlc2lkZW50LCBBcGFjaGUgWm9vS2VlcGVyLCB0bw0KPiCg
IKAgoCBzZXJ2ZSBpbiBhY2NvcmRhbmNlIHdpdGggYW5kIHN1YmplY3QgdG8gdGhlIGRpcmVjdGlv
biBvZiB0aGUNCj4goCCgIKAgQm9hcmQgb2YgRGlyZWN0b3JzIGFuZCB0aGUgQnlsYXdzIG9mIHRo
ZSBGb3VuZGF0aW9uIHVudGlsDQo+IKAgoCCgIGRlYXRoLCByZXNpZ25hdGlvbiwgcmV0aXJlbWVu
dCwgcmVtb3ZhbCBvciBkaXNxdWFsaWZpY2F0aW9uLA0KPiCgIKAgoCBvciB1bnRpbCBhIHN1Y2Nl
c3NvciBpcyBhcHBvaW50ZWQ7IGFuZCBiZSBpdCBmdXJ0aGVyDQo+DQo+IKAgoCCgIFJFU09MVkVE
LCB0aGF0IHRoZSBpbml0aWFsIEFwYWNoZSBab29LZWVwZXIgUE1DIGJlIGFuZCBoZXJlYnkgaXMN
Cj4goCCgIKAgdGFza2VkIHdpdGggdGhlIGNyZWF0aW9uIG9mIGEgc2V0IG9mIGJ5bGF3cyBpbnRl
bmRlZCB0bw0KPiCgIKAgoCBlbmNvdXJhZ2Ugb3BlbiBkZXZlbG9wbWVudCBhbmQgaW5jcmVhc2Vk
IHBhcnRpY2lwYXRpb24gaW4gdGhlDQo+IKAgoCCgIEFwYWNoZSBab29LZWVwZXIgUHJvamVjdDsg
YW5kIGJlIGl0IGZ1cnRoZXINCj4NCj4goCCgIKAgUkVTT0xWRUQsIHRoYXQgdGhlIEFwYWNoZSBa
b29LZWVwZXIgUHJvamVjdCBiZSBhbmQgaGVyZWJ5DQo+IKAgoCCgIGlzIHRhc2tlZCB3aXRoIHRo
ZSBtaWdyYXRpb24gYW5kIHJhdGlvbmFsaXphdGlvbiBvZiB0aGUgQXBhY2hlDQo+IKAgoCCgIEhh
ZG9vcCBab29LZWVwZXIgc3ViLXByb2plY3Q7IGFuZCBiZSBpdCBmdXJ0aGVyDQo+DQo+IKAgoCCg
IFJFU09MVkVELCB0aGF0IGFsbCByZXNwb25zaWJpbGl0aWVzIHBlcnRhaW5pbmcgdG8gdGhlIEFw
YWNoZQ0KPiCgIKAgoCBIYWRvb3AgWm9vS2VlcGVyIHN1Yi1wcm9qZWN0IGVuY3VtYmVyZWQgdXBv
biB0aGUNCj4goCCgIKAgQXBhY2hlIEhhZG9vcCBQcm9qZWN0IGFyZSBoZXJlYWZ0ZXIgZGlzY2hh
cmdlZC4NCj4NCg==


From general-return-2298-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 28 09:24:53 2010
Return-Path: <general-return-2298-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 61989 invoked from network); 28 Oct 2010 09:24:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Oct 2010 09:24:53 -0000
Received: (qmail 23838 invoked by uid 500); 28 Oct 2010 09:24:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23541 invoked by uid 500); 28 Oct 2010 09:24:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23533 invoked by uid 99); 28 Oct 2010 09:24:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 09:24:50 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 09:24:41 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 09BB1B7DB4
	for <general@hadoop.apache.org>; Thu, 28 Oct 2010 10:24:19 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id tR21yEtqCSer for <general@hadoop.apache.org>;
	Thu, 28 Oct 2010 10:24:19 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 5169BB7DB1
	for <general@hadoop.apache.org>; Thu, 28 Oct 2010 10:24:18 +0100 (BST)
MailScanner-NULL-Check: 1288862637.1821@46H1BIBiFn0Xo8F0WJBO+g
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o9S9Nt9p012045
	for <general@hadoop.apache.org>; Thu, 28 Oct 2010 10:23:55 +0100 (BST)
Message-ID: <4CC9412B.7030805@apache.org>
Date: Thu, 28 Oct 2010 10:23:55 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.12) Gecko/20100915 Thunderbird/3.0.8
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] ZooKeeper as TLP?
References: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
In-Reply-To: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o9S9Nt9p012045
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 27/10/10 19:04, Patrick Hunt wrote:
> Please vote as to whether you think ZooKeeper should become a
> top-level Apache project.

+1

From general-return-2299-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 28 21:14:50 2010
Return-Path: <general-return-2299-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85190 invoked from network); 28 Oct 2010 21:14:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Oct 2010 21:14:50 -0000
Received: (qmail 17461 invoked by uid 500); 28 Oct 2010 21:14:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16796 invoked by uid 500); 28 Oct 2010 21:14:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16612 invoked by uid 99); 28 Oct 2010 21:14:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 21:14:45 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of shv.hadoop@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 21:14:38 +0000
Received: by pwj9 with SMTP id 9so278271pwj.35
        for <general@hadoop.apache.org>; Thu, 28 Oct 2010 14:14:17 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=+0MEBYv2WFnDOUDn2AlUqtbhwnzyMMeXhq48cUdKSU8=;
        b=Vs0Y99GzyPalrH8KStTf4U6iOohcaEItxX9uHQgMnAWHUfxsg7poOIiuWzOziHKmOg
         ULB2RmNXE7P6vegRVhviG1uJw24C3J0mvGLP2ZQdLgOK4ZeuRkOb4noCCUdHcSCOuVel
         t3a4fqkvvVlxGLGCVb76E0bNnXYlAf0zHCrVc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=hzn9/RClmUK6cGtpcAFHX5yl9fQlUMGCYLA9+rpWXVhyK/psL56Bk/z0/bJTNC7Ie+
         dur2w9B36QjfJjwb8C/7EGUdC4dIf6AST0j9jHMB50ze7MUyvhLCzN2T/2kUQPEpUJHF
         EoCD31IcYg7QlX5xdH+LfOlepqDlf7AkdjiCI=
MIME-Version: 1.0
Received: by 10.142.132.6 with SMTP id f6mr756519wfd.63.1288300457102; Thu, 28
 Oct 2010 14:14:17 -0700 (PDT)
Received: by 10.142.105.9 with HTTP; Thu, 28 Oct 2010 14:14:17 -0700 (PDT)
In-Reply-To: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
References: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
Date: Thu, 28 Oct 2010 14:14:17 -0700
Message-ID: <AANLkTikzSjQaU5YBCP-HzYJ0mGA+ne_aSZ2OPyfaYvc4@mail.gmail.com>
Subject: Re: [VOTE] ZooKeeper as TLP?
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd2dd121c0f600493b3d302
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd2dd121c0f600493b3d302
Content-Type: text/plain; charset=ISO-8859-1

+1
Konstantin

> Do the good people of Hadoop support sending this request on to the Hadoop
PMC?

Looks like everybody voted already.


On Wed, Oct 27, 2010 at 11:04 AM, Patrick Hunt <phunt@apache.org> wrote:

> Please vote as to whether you think ZooKeeper should become a
> top-level Apache project.
>
> The ZooKeeper development community has already voted on and approved
> ( http://bit.ly/9VCoTi ) the following draft board resolution. It
> lists all current active ZooKeeper committers (sans one committer who
> has not been active on the project since it's inception
> http://bit.ly/9wLkVV) as initial members of the project management
> committee (PMC) and myself, Patrick Hunt, as the initial chair.
>
>
> Do the good people of Hadoop support sending this request on to the
> Hadoop PMC?
>
> Regards,
>
> Patrick
>
> ------------
>
>    X. Establish the Apache ZooKeeper Project
>
>       WHEREAS, the Board of Directors deems it to be in the best
>       interests of the Foundation and consistent with the
>       Foundation's purpose to establish a Project Management
>       Committee charged with the creation and maintenance of
>       open-source software related to distributed system coordination
>       for distribution at no charge to the public.
>
>       NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>       Committee (PMC), to be known as the "Apache ZooKeeper Project",
>       be and hereby is established pursuant to Bylaws of the
>       Foundation; and be it further
>
>       RESOLVED, that the Apache ZooKeeper Project be and hereby is
>       responsible for the creation and maintenance of software
>       related to distributed system coordination; and be it further
>
>       RESOLVED, that the office of "Vice President, Apache ZooKeeper" be
>       and hereby is created, the person holding such office to
>       serve at the direction of the Board of Directors as the chair
>       of the Apache ZooKeeper Project, and to have primary responsibility
>       for management of the projects within the scope of
>       responsibility of the Apache ZooKeeper Project; and be it further
>
>       RESOLVED, that the persons listed immediately below be and
>       hereby are appointed to serve as the initial members of the
>       Apache ZooKeeper Project:
>
>         * Patrick Hunt         <phunt@apache.org>
>         * Flavio Junqueira     <fpj@apache.org>
>         * Mahadev Konar        <mahadev@apache.org>
>         * Benjamin Reed        <breed@apache.org>
>         * Henry Robinson       <henry@apache.org>
>
>       NOW, THEREFORE, BE IT FURTHER RESOLVED, that Patrick Hunt
>       be appointed to the office of Vice President, Apache ZooKeeper, to
>       serve in accordance with and subject to the direction of the
>       Board of Directors and the Bylaws of the Foundation until
>       death, resignation, retirement, removal or disqualification,
>       or until a successor is appointed; and be it further
>
>       RESOLVED, that the initial Apache ZooKeeper PMC be and hereby is
>       tasked with the creation of a set of bylaws intended to
>       encourage open development and increased participation in the
>       Apache ZooKeeper Project; and be it further
>
>       RESOLVED, that the Apache ZooKeeper Project be and hereby
>       is tasked with the migration and rationalization of the Apache
>       Hadoop ZooKeeper sub-project; and be it further
>
>       RESOLVED, that all responsibilities pertaining to the Apache
>       Hadoop ZooKeeper sub-project encumbered upon the
>       Apache Hadoop Project are hereafter discharged.
>

--000e0cd2dd121c0f600493b3d302--

From general-return-2300-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 28 21:45:13 2010
Return-Path: <general-return-2300-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4078 invoked from network); 28 Oct 2010 21:45:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Oct 2010 21:45:13 -0000
Received: (qmail 86270 invoked by uid 500); 28 Oct 2010 21:45:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86193 invoked by uid 500); 28 Oct 2010 21:45:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86184 invoked by uid 99); 28 Oct 2010 21:45:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 21:45:11 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 21:45:04 +0000
Received: by gyg4 with SMTP id 4so1824883gyg.35
        for <general@hadoop.apache.org>; Thu, 28 Oct 2010 14:44:42 -0700 (PDT)
Received: by 10.150.190.18 with SMTP id n18mr21398722ybf.362.1288302282192;
        Thu, 28 Oct 2010 14:44:42 -0700 (PDT)
Received: from [175.33.26.86] ([175.33.26.86])
        by mx.google.com with ESMTPS id v12sm1119951ybk.23.2010.10.28.14.44.39
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Thu, 28 Oct 2010 14:44:41 -0700 (PDT)
References: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com> <AANLkTikzSjQaU5YBCP-HzYJ0mGA+ne_aSZ2OPyfaYvc4@mail.gmail.com>
In-Reply-To: <AANLkTikzSjQaU5YBCP-HzYJ0mGA+ne_aSZ2OPyfaYvc4@mail.gmail.com>
Mime-Version: 1.0 (iPhone Mail 8B117)
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
	charset=us-ascii
Message-Id: <A7C0E447-7007-4226-A9E8-B422B6259F1A@holsman.net>
Cc: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: iPhone Mail (8B117)
From: Ian Holsman <hadoop@holsman.net>
Subject: Re: [VOTE] ZooKeeper as TLP?
Date: Fri, 29 Oct 2010 08:43:35 +1100
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

The vote needs to be done on the pmc list for it to be official. 


---
Ian Holsman - 703 879-3128

I saw the angel in the marble and carved until I set him free -- Michelangelo

On 29/10/2010, at 8:14 AM, Konstantin Shvachko <shv.hadoop@gmail.com> wrote:

> +1
> Konstantin
> 
>> Do the good people of Hadoop support sending this request on to the Hadoop
> PMC?
> 
> Looks like everybody voted already.
> 
> 
> On Wed, Oct 27, 2010 at 11:04 AM, Patrick Hunt <phunt@apache.org> wrote:
> 
>> Please vote as to whether you think ZooKeeper should become a
>> top-level Apache project.
>> 
>> The ZooKeeper development community has already voted on and approved
>> ( http://bit.ly/9VCoTi ) the following draft board resolution. It
>> lists all current active ZooKeeper committers (sans one committer who
>> has not been active on the project since it's inception
>> http://bit.ly/9wLkVV) as initial members of the project management
>> committee (PMC) and myself, Patrick Hunt, as the initial chair.
>> 
>> 
>> Do the good people of Hadoop support sending this request on to the
>> Hadoop PMC?
>> 
>> Regards,
>> 
>> Patrick
>> 
>> ------------
>> 
>>   X. Establish the Apache ZooKeeper Project
>> 
>>      WHEREAS, the Board of Directors deems it to be in the best
>>      interests of the Foundation and consistent with the
>>      Foundation's purpose to establish a Project Management
>>      Committee charged with the creation and maintenance of
>>      open-source software related to distributed system coordination
>>      for distribution at no charge to the public.
>> 
>>      NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>>      Committee (PMC), to be known as the "Apache ZooKeeper Project",
>>      be and hereby is established pursuant to Bylaws of the
>>      Foundation; and be it further
>> 
>>      RESOLVED, that the Apache ZooKeeper Project be and hereby is
>>      responsible for the creation and maintenance of software
>>      related to distributed system coordination; and be it further
>> 
>>      RESOLVED, that the office of "Vice President, Apache ZooKeeper" be
>>      and hereby is created, the person holding such office to
>>      serve at the direction of the Board of Directors as the chair
>>      of the Apache ZooKeeper Project, and to have primary responsibility
>>      for management of the projects within the scope of
>>      responsibility of the Apache ZooKeeper Project; and be it further
>> 
>>      RESOLVED, that the persons listed immediately below be and
>>      hereby are appointed to serve as the initial members of the
>>      Apache ZooKeeper Project:
>> 
>>        * Patrick Hunt         <phunt@apache.org>
>>        * Flavio Junqueira     <fpj@apache.org>
>>        * Mahadev Konar        <mahadev@apache.org>
>>        * Benjamin Reed        <breed@apache.org>
>>        * Henry Robinson       <henry@apache.org>
>> 
>>      NOW, THEREFORE, BE IT FURTHER RESOLVED, that Patrick Hunt
>>      be appointed to the office of Vice President, Apache ZooKeeper, to
>>      serve in accordance with and subject to the direction of the
>>      Board of Directors and the Bylaws of the Foundation until
>>      death, resignation, retirement, removal or disqualification,
>>      or until a successor is appointed; and be it further
>> 
>>      RESOLVED, that the initial Apache ZooKeeper PMC be and hereby is
>>      tasked with the creation of a set of bylaws intended to
>>      encourage open development and increased participation in the
>>      Apache ZooKeeper Project; and be it further
>> 
>>      RESOLVED, that the Apache ZooKeeper Project be and hereby
>>      is tasked with the migration and rationalization of the Apache
>>      Hadoop ZooKeeper sub-project; and be it further
>> 
>>      RESOLVED, that all responsibilities pertaining to the Apache
>>      Hadoop ZooKeeper sub-project encumbered upon the
>>      Apache Hadoop Project are hereafter discharged.
>> 

From general-return-2301-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 28 22:22:23 2010
Return-Path: <general-return-2301-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42226 invoked from network); 28 Oct 2010 22:22:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Oct 2010 22:22:23 -0000
Received: (qmail 38495 invoked by uid 500); 28 Oct 2010 22:22:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38324 invoked by uid 500); 28 Oct 2010 22:22:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 38316 invoked by uid 99); 28 Oct 2010 22:22:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 22:22:21 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 28 Oct 2010 22:22:19 +0000
Received: (qmail 40939 invoked by uid 99); 28 Oct 2010 22:21:57 -0000
Received: from localhost.apache.org (HELO [172.29.12.106]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 22:21:57 +0000
Message-ID: <4CC9F784.3060004@apache.org>
Date: Thu, 28 Oct 2010 15:21:56 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.11) Gecko/20101006 Thunderbird/3.1.5
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] ZooKeeper as TLP?
References: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com> <AANLkTikzSjQaU5YBCP-HzYJ0mGA+ne_aSZ2OPyfaYvc4@mail.gmail.com> <A7C0E447-7007-4226-A9E8-B422B6259F1A@holsman.net>
In-Reply-To: <A7C0E447-7007-4226-A9E8-B422B6259F1A@holsman.net>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 10/28/2010 02:43 PM, Ian Holsman wrote:
> The vote needs to be done on the pmc list for it to be official.

general@ is the Hadoop PMC's public list, as with the incubator.  We 
hold all votes here that don't need to be held in private.

Doug

From general-return-2302-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 28 23:57:08 2010
Return-Path: <general-return-2302-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 79266 invoked from network); 28 Oct 2010 23:57:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Oct 2010 23:57:08 -0000
Received: (qmail 37171 invoked by uid 500); 28 Oct 2010 23:57:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37131 invoked by uid 500); 28 Oct 2010 23:57:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37123 invoked by uid 99); 28 Oct 2010 23:57:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 23:57:07 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [98.139.53.209] (HELO nm17-vm1.bullet.mail.ac4.yahoo.com) (98.139.53.209)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 28 Oct 2010 23:56:59 +0000
Received: from [98.139.52.194] by nm17.bullet.mail.ac4.yahoo.com with NNFMP; 28 Oct 2010 23:56:38 -0000
Received: from [98.139.52.186] by tm7.bullet.mail.ac4.yahoo.com with NNFMP; 28 Oct 2010 23:56:38 -0000
Received: from [127.0.0.1] by omp1069.mail.ac4.yahoo.com with NNFMP; 28 Oct 2010 23:56:38 -0000
X-Yahoo-Newman-Property: ymail-5
X-Yahoo-Newman-Id: 860186.19127.bm@omp1069.mail.ac4.yahoo.com
Received: (qmail 83768 invoked by uid 60001); 28 Oct 2010 23:56:38 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1288310198; bh=WMnSVf+/dyypg5S+8gumBYqh87x6IuO2OkOLc8NTlEo=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=YvFH+hSN16axFbFqE3B/WwUWsxAntpmHuwgVg83ORkpIXtNf5ft42Qy/wKIyZVNzOxAwlC1G07S/xE88+kc90AtkRRdSh47/zgp+QP1RJGeYQbtFaX8HwFJCFwVNB/0loIeHswEBhTPXhc5IA1CfFXpGXM30cCttBQ/rqUJpszI=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=06RKeGoiqSUx/OfhDhfOOFkAzBbBhKQs5AyqT4agOfxhum8fkH1NIvjWEM4Z7p0TPDIHRG0Ox6UtM6XwgcxImwWT++HBKezZwPepTBTdBVA6nRMsC6l7ZuP30pUwR5J/PwP361oYal4HRMebVRnvHlHt3KTLev0VZLGc0Y8CpbQ=;
Message-ID: <582335.75741.qm@web56208.mail.re3.yahoo.com>
X-YMail-OSG: G_Z5WIAVM1mlpzqTVWtoYwfDswBvaCIwGg.2FXyw3zTpMfw
 539Kj5oyQTdAn83WmN.gOHF.k86xZSDujWpuErR6y7IHcpRT5L8HC4CiJHgY
 mR3LRaLHqG5j6TMTB4EFwVYhGaTLl6SXhbEfIoBAcjE9JcR4jDJS3PBe_XZU
 tltf84YQBmVf9JheA8rnjk.dWxvDM3MXjC2ojsV8LKRMYGVsYJejzyJu7_gw
 XUAgIftgT2Pdu5JxIhl7TiYKzI6g588UvVW.zxV4Lj7yLbLj0DqqEl3dAu8e
 k7ZIeVVhmoiIeuBOuwe3QumlmUCEy.cChcZHivL_Hg_C6SDdkG24qtp6vfw8
 22nbC1diii4sjaa7zlOSdYE0-
Received: from [203.99.254.216] by web56208.mail.re3.yahoo.com via HTTP; Thu, 28 Oct 2010 16:56:38 PDT
X-Mailer: YahooMailRC/504.5 YahooMailWebService/0.8.107.284920
References: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
Date: Thu, 28 Oct 2010 16:56:38 -0700 (PDT)
From: "Tsz Wo \(Nicholas\), Sze" <s29752-hadoopgeneral@yahoo.com>
Subject: Re: [VOTE] ZooKeeper as TLP?
To: general@hadoop.apache.org
In-Reply-To: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-949625388-1288310198=:75741"

--0-949625388-1288310198=:75741
Content-Type: text/plain; charset=us-ascii

+1
Nicholas




________________________________
From: Patrick Hunt <phunt@apache.org>
To: general@hadoop.apache.org
Sent: Thu, October 28, 2010 2:04:55 AM
Subject: [VOTE] ZooKeeper as TLP?

Please vote as to whether you think ZooKeeper should become a
top-level Apache project.

The ZooKeeper development community has already voted on and approved
( http://bit.ly/9VCoTi ) the following draft board resolution. It
lists all current active ZooKeeper committers (sans one committer who
has not been active on the project since it's inception
http://bit.ly/9wLkVV) as initial members of the project management
committee (PMC) and myself, Patrick Hunt, as the initial chair.


Do the good people of Hadoop support sending this request on to the
Hadoop PMC?

Regards,

Patrick

------------

    X. Establish the Apache ZooKeeper Project

       WHEREAS, the Board of Directors deems it to be in the best
       interests of the Foundation and consistent with the
       Foundation's purpose to establish a Project Management
       Committee charged with the creation and maintenance of
       open-source software related to distributed system coordination
       for distribution at no charge to the public.

       NOW, THEREFORE, BE IT RESOLVED, that a Project Management
       Committee (PMC), to be known as the "Apache ZooKeeper Project",
       be and hereby is established pursuant to Bylaws of the
       Foundation; and be it further

       RESOLVED, that the Apache ZooKeeper Project be and hereby is
       responsible for the creation and maintenance of software
       related to distributed system coordination; and be it further

       RESOLVED, that the office of "Vice President, Apache ZooKeeper" be
       and hereby is created, the person holding such office to
       serve at the direction of the Board of Directors as the chair
       of the Apache ZooKeeper Project, and to have primary responsibility
       for management of the projects within the scope of
       responsibility of the Apache ZooKeeper Project; and be it further

       RESOLVED, that the persons listed immediately below be and
       hereby are appointed to serve as the initial members of the
       Apache ZooKeeper Project:

         * Patrick Hunt         <phunt@apache.org>
         * Flavio Junqueira     <fpj@apache.org>
         * Mahadev Konar        <mahadev@apache.org>
         * Benjamin Reed        <breed@apache.org>
         * Henry Robinson       <henry@apache.org>

       NOW, THEREFORE, BE IT FURTHER RESOLVED, that Patrick Hunt
       be appointed to the office of Vice President, Apache ZooKeeper, to
       serve in accordance with and subject to the direction of the
       Board of Directors and the Bylaws of the Foundation until
       death, resignation, retirement, removal or disqualification,
       or until a successor is appointed; and be it further

       RESOLVED, that the initial Apache ZooKeeper PMC be and hereby is
       tasked with the creation of a set of bylaws intended to
       encourage open development and increased participation in the
       Apache ZooKeeper Project; and be it further

       RESOLVED, that the Apache ZooKeeper Project be and hereby
       is tasked with the migration and rationalization of the Apache
       Hadoop ZooKeeper sub-project; and be it further

       RESOLVED, that all responsibilities pertaining to the Apache
       Hadoop ZooKeeper sub-project encumbered upon the
       Apache Hadoop Project are hereafter discharged.

--0-949625388-1288310198=:75741--

From general-return-2303-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 29 00:01:29 2010
Return-Path: <general-return-2303-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81393 invoked from network); 29 Oct 2010 00:01:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Oct 2010 00:01:28 -0000
Received: (qmail 44041 invoked by uid 500); 29 Oct 2010 00:01:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 43931 invoked by uid 500); 29 Oct 2010 00:01:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 43923 invoked by uid 99); 29 Oct 2010 00:01:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 00:01:27 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 00:01:22 +0000
Received: by wwb39 with SMTP id 39so410788wwb.29
        for <general@hadoop.apache.org>; Thu, 28 Oct 2010 17:01:01 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:content-type:content-transfer-encoding;
        bh=01we12Pc3NYJP6Ai3AO8vgC5vH/rVOSGAQnGF+O/dxo=;
        b=Iex4S3JnCcIvhLJHgTHwxWjHbFI7ynBVWphKyFloPSrqCRpqRBXylZtOPLO2R1sRmB
         FTgRf8w/FxmkyL9/iYkBiTwzjjGkdzO0mQLNTgxyKKreot+1H4PlLtxWKbi8XV2OyP3A
         cX/kLQcldg9qpgcyPfcNq6J4rGUU4soof6Wo4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=X9g+XUDm7RQd2KA1B76PXhnZwohVRE4YL1U6r7yjHiF4ARry65e4bskstbsIxUGpUl
         /oAizMHbFOJYhzmU8ULhvH1dxXcH2OAj92tlFdwgvC1MNvK9RIyvl759jQmZvQQcW4oH
         7hehCByU5F/ulpR+D0alGIjwZ4yunwZ8gInao=
MIME-Version: 1.0
Received: by 10.216.154.202 with SMTP id h52mr721580wek.46.1288310460732; Thu,
 28 Oct 2010 17:01:00 -0700 (PDT)
Sender: saint.ack@gmail.com
Received: by 10.216.93.74 with HTTP; Thu, 28 Oct 2010 17:01:00 -0700 (PDT)
In-Reply-To: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
References: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
Date: Thu, 28 Oct 2010 17:01:00 -0700
X-Google-Sender-Auth: dqa_1JJORs8xXdr8EHJrhXxOD1Y
Message-ID: <AANLkTikCRC13-J-VztVNXGq_YF=6B70Ram=4zfbD95J9@mail.gmail.com>
Subject: Re: [VOTE] ZooKeeper as TLP?
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

+1

On Wed, Oct 27, 2010 at 11:04 AM, Patrick Hunt <phunt@apache.org> wrote:
> Please vote as to whether you think ZooKeeper should become a
> top-level Apache project.
>
> The ZooKeeper development community has already voted on and approved
> ( http://bit.ly/9VCoTi ) the following draft board resolution. It
> lists all current active ZooKeeper committers (sans one committer who
> has not been active on the project since it's inception
> http://bit.ly/9wLkVV) as initial members of the project management
> committee (PMC) and myself, Patrick Hunt, as the initial chair.
>
>
> Do the good people of Hadoop support sending this request on to the
> Hadoop PMC?
>
> Regards,
>
> Patrick
>
> ------------
>
> =A0 =A0X. Establish the Apache ZooKeeper Project
>
> =A0 =A0 =A0 WHEREAS, the Board of Directors deems it to be in the best
> =A0 =A0 =A0 interests of the Foundation and consistent with the
> =A0 =A0 =A0 Foundation's purpose to establish a Project Management
> =A0 =A0 =A0 Committee charged with the creation and maintenance of
> =A0 =A0 =A0 open-source software related to distributed system coordinati=
on
> =A0 =A0 =A0 for distribution at no charge to the public.
>
> =A0 =A0 =A0 NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> =A0 =A0 =A0 Committee (PMC), to be known as the "Apache ZooKeeper Project=
",
> =A0 =A0 =A0 be and hereby is established pursuant to Bylaws of the
> =A0 =A0 =A0 Foundation; and be it further
>
> =A0 =A0 =A0 RESOLVED, that the Apache ZooKeeper Project be and hereby is
> =A0 =A0 =A0 responsible for the creation and maintenance of software
> =A0 =A0 =A0 related to distributed system coordination; and be it further
>
> =A0 =A0 =A0 RESOLVED, that the office of "Vice President, Apache ZooKeepe=
r" be
> =A0 =A0 =A0 and hereby is created, the person holding such office to
> =A0 =A0 =A0 serve at the direction of the Board of Directors as the chair
> =A0 =A0 =A0 of the Apache ZooKeeper Project, and to have primary responsi=
bility
> =A0 =A0 =A0 for management of the projects within the scope of
> =A0 =A0 =A0 responsibility of the Apache ZooKeeper Project; and be it fur=
ther
>
> =A0 =A0 =A0 RESOLVED, that the persons listed immediately below be and
> =A0 =A0 =A0 hereby are appointed to serve as the initial members of the
> =A0 =A0 =A0 Apache ZooKeeper Project:
>
> =A0 =A0 =A0 =A0 * Patrick Hunt =A0 =A0 =A0 =A0 <phunt@apache.org>
> =A0 =A0 =A0 =A0 * Flavio Junqueira =A0 =A0 <fpj@apache.org>
> =A0 =A0 =A0 =A0 * Mahadev Konar =A0 =A0 =A0 =A0<mahadev@apache.org>
> =A0 =A0 =A0 =A0 * Benjamin Reed =A0 =A0 =A0 =A0<breed@apache.org>
> =A0 =A0 =A0 =A0 * Henry Robinson =A0 =A0 =A0 <henry@apache.org>
>
> =A0 =A0 =A0 NOW, THEREFORE, BE IT FURTHER RESOLVED, that Patrick Hunt
> =A0 =A0 =A0 be appointed to the office of Vice President, Apache ZooKeepe=
r, to
> =A0 =A0 =A0 serve in accordance with and subject to the direction of the
> =A0 =A0 =A0 Board of Directors and the Bylaws of the Foundation until
> =A0 =A0 =A0 death, resignation, retirement, removal or disqualification,
> =A0 =A0 =A0 or until a successor is appointed; and be it further
>
> =A0 =A0 =A0 RESOLVED, that the initial Apache ZooKeeper PMC be and hereby=
 is
> =A0 =A0 =A0 tasked with the creation of a set of bylaws intended to
> =A0 =A0 =A0 encourage open development and increased participation in the
> =A0 =A0 =A0 Apache ZooKeeper Project; and be it further
>
> =A0 =A0 =A0 RESOLVED, that the Apache ZooKeeper Project be and hereby
> =A0 =A0 =A0 is tasked with the migration and rationalization of the Apach=
e
> =A0 =A0 =A0 Hadoop ZooKeeper sub-project; and be it further
>
> =A0 =A0 =A0 RESOLVED, that all responsibilities pertaining to the Apache
> =A0 =A0 =A0 Hadoop ZooKeeper sub-project encumbered upon the
> =A0 =A0 =A0 Apache Hadoop Project are hereafter discharged.
>

From general-return-2304-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 29 19:18:45 2010
Return-Path: <general-return-2304-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16252 invoked from network); 29 Oct 2010 19:18:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Oct 2010 19:18:45 -0000
Received: (qmail 54834 invoked by uid 500); 29 Oct 2010 19:18:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54678 invoked by uid 500); 29 Oct 2010 19:18:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54664 invoked by uid 99); 29 Oct 2010 19:18:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 19:18:41 +0000
X-ASF-Spam-Status: No, hits=2.5 required=10.0
	tests=HTML_FONT_FACE_BAD,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of shrijeet@rocketfuelinc.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 19:18:35 +0000
Received: by fxm7 with SMTP id 7so3279627fxm.35
        for <multiple recipients>; Fri, 29 Oct 2010 12:18:14 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.223.89.135 with SMTP id e7mr5858254fam.129.1288379894523; Fri,
 29 Oct 2010 12:18:14 -0700 (PDT)
Received: by 10.223.146.1 with HTTP; Fri, 29 Oct 2010 12:18:14 -0700 (PDT)
Date: Fri, 29 Oct 2010 12:18:14 -0700
Message-ID: <AANLkTi=sqaKaVxg2OQZ1kwWavgcg_aYUHVeZWFxwaeGN@mail.gmail.com>
Subject: Loading conf file sitting in HDFS
From: Shrijeet Paliwal <shrijeet@rocketfuel.com>
To: general@hadoop.apache.org, mapreduce-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf3054a477f2f1b20493c65117
X-Virus-Checked: Checked by ClamAV on apache.org

--20cf3054a477f2f1b20493c65117
Content-Type: text/plain; charset=ISO-8859-1

Hello All,
I am trying to load a conf file located in hdfs.
I was hoping following would work:

Path apath = new Path(conf.get("fs.default.name"),
"/home/shrijeet/blah.xml");
conf.addResource(apath);

But I get following exception,

Exception in thread "main" java.lang.RuntimeException:
hdfs://localhost:8020/home/shrijeet/blah.xml not found
        at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1297)
        at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1227)


File does exist however,

[12:20][shrijeet@shrijeet-desktop]~$ hadoop dfs -ls
hdfs://localhost:8020/home/shrijeet/blah.xml
Found 1 items
-rw-r--r--   1 shrijeet shrijeet        488 2010-10-29 12:16
/home/shrijeet/blah.xml

Thoughts?

-Shrijeet

--20cf3054a477f2f1b20493c65117--

From general-return-2305-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 29 19:39:06 2010
Return-Path: <general-return-2305-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23587 invoked from network); 29 Oct 2010 19:39:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Oct 2010 19:39:06 -0000
Received: (qmail 77097 invoked by uid 500); 29 Oct 2010 19:39:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77030 invoked by uid 500); 29 Oct 2010 19:39:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77014 invoked by uid 99); 29 Oct 2010 19:39:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 19:39:03 +0000
X-ASF-Spam-Status: No, hits=2.5 required=10.0
	tests=HTML_FONT_FACE_BAD,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of shrijeet@rocketfuelinc.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 19:38:57 +0000
Received: by fxm7 with SMTP id 7so3297184fxm.35
        for <multiple recipients>; Fri, 29 Oct 2010 12:38:37 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.223.89.135 with SMTP id e7mr5881185fam.129.1288381116892; Fri,
 29 Oct 2010 12:38:36 -0700 (PDT)
Received: by 10.223.146.1 with HTTP; Fri, 29 Oct 2010 12:38:36 -0700 (PDT)
In-Reply-To: <AANLkTi=sqaKaVxg2OQZ1kwWavgcg_aYUHVeZWFxwaeGN@mail.gmail.com>
References: <AANLkTi=sqaKaVxg2OQZ1kwWavgcg_aYUHVeZWFxwaeGN@mail.gmail.com>
Date: Fri, 29 Oct 2010 12:38:36 -0700
Message-ID: <AANLkTik9U=oynmZ3KqtW6yaMEcEaoYYO2SttqnLRQiKN@mail.gmail.com>
Subject: Re: Loading conf file sitting in HDFS
From: Shrijeet Paliwal <shrijeet@rocketfuel.com>
To: general@hadoop.apache.org, mapreduce-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf3054a477cec95b0493c69a35
X-Virus-Checked: Checked by ClamAV on apache.org

--20cf3054a477cec95b0493c69a35
Content-Type: text/plain; charset=ISO-8859-1

I got it working by doing this (passing inputstream):

conf.addResource(hdfs.open(apath));

Would still be interested in knowing insights if any one wants to share.

-Shrijeet

On Fri, Oct 29, 2010 at 12:18 PM, Shrijeet Paliwal
<shrijeet@rocketfuel.com>wrote:

> Hello All,
> I am trying to load a conf file located in hdfs.
> I was hoping following would work:
>
> Path apath = new Path(conf.get("fs.default.name"),
> "/home/shrijeet/blah.xml");
> conf.addResource(apath);
>
> But I get following exception,
>
> Exception in thread "main" java.lang.RuntimeException:
> hdfs://localhost:8020/home/shrijeet/blah.xml not found
>         at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1297)
>         at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1227)
>
>
> File does exist however,
>
> [12:20][shrijeet@shrijeet-desktop]~$ hadoop dfs -ls
> hdfs://localhost:8020/home/shrijeet/blah.xml
> Found 1 items
> -rw-r--r--   1 shrijeet shrijeet        488 2010-10-29 12:16
> /home/shrijeet/blah.xml
>
> Thoughts?
>
> -Shrijeet
>

--20cf3054a477cec95b0493c69a35--

From general-return-2306-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 29 19:55:24 2010
Return-Path: <general-return-2306-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30204 invoked from network); 29 Oct 2010 19:55:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Oct 2010 19:55:23 -0000
Received: (qmail 16900 invoked by uid 500); 29 Oct 2010 19:55:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16807 invoked by uid 500); 29 Oct 2010 19:55:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16799 invoked by uid 99); 29 Oct 2010 19:55:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 19:55:22 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of msegel@navteq.com designates 204.120.70.37 as permitted sender)
Received: from [204.120.70.37] (HELO xmailfargo.navteq.com) (204.120.70.37)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 19:55:15 +0000
Received: from fargo-spamfw.navteq.com (navteq-spamfw.navteq.com [10.6.10.131])
	by xmailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id o9TJTPgs009690
	for <general@hadoop.apache.org>; Fri, 29 Oct 2010 14:29:25 -0500
Received: from imailfargo.navteq.com (localhost [127.0.0.1])
	by fargo-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP
	id 1E94130E4294; Fri, 29 Oct 2010 14:54:54 -0500 (CDT)
Received: from imailfargo.navteq.com (imailfargo.navteq.com [10.6.10.130]) by fargo-spamfw.navteq.com with ESMTP id rEq9Me4ZhSVndFZC; Fri, 29 Oct 2010 14:54:54 -0500 (CDT)
Received: from hq-ex-ht01.ad.navteq.com ([10.8.222.51])
	by imailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id o9TJsrRH009512;
	Fri, 29 Oct 2010 14:54:53 -0500
Received: from hq-ex-mb02.ad.navteq.com ([fe80::880d:318e:e860:a8db]) by
 hq-ex-ht01.ad.navteq.com ([fe80::55a6:5b94:5c00:fe39%12]) with mapi; Fri, 29
 Oct 2010 14:54:53 -0500
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>,
        "mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>
Date: Fri, 29 Oct 2010 14:54:52 -0500
Subject: RE: Loading conf file sitting in HDFS
Thread-Topic: Loading conf file sitting in HDFS
Thread-Index: Act3oPPXy9JjiXX4QhCGemJQKi2VSgAAJWSQ
Message-ID: <ABC24175AFD3BE4DA15F4CD375ED413D0639D3EF29@hq-ex-mb02.ad.navteq.com>
References: <AANLkTi=sqaKaVxg2OQZ1kwWavgcg_aYUHVeZWFxwaeGN@mail.gmail.com>
 <AANLkTik9U=oynmZ3KqtW6yaMEcEaoYYO2SttqnLRQiKN@mail.gmail.com>
In-Reply-To: <AANLkTik9U=oynmZ3KqtW6yaMEcEaoYYO2SttqnLRQiKN@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1
X-Virus-Checked: Checked by ClamAV on apache.org

Insights?

I'm sure someone will correct me...=20

What I saw was the overloaded method addResource().

Looking at the various methods' input:
addResource(String name) :=3D name - resource to be added, the classpath =
is examined for a file with that name.
addResource(URL url) :=3D url - url of the resource to be added, the loca=
l filesystem is examined directly to find the resource, without referring=
 to the classpath.
addResource(Path file) :=3D file - file-path of resource to be added, the=
 local filesystem is examined directly to find the resource, without refe=
rring to the classpath.

And then:

addResource(InputStream in) :=3D in - InputStream to deserialize the obje=
ct from.

The first 3 specifically are looking for a local file. (Unix file system)=
=2E

The last one take a generic InputStream which could come from a Unix file=
 or an HDFS file.

It looks like the initial intention of loading configuration information =
was from Unix and then HDFS was an afterthought. (My Guess).

HTH

-Mike

-----Original Message-----
From: Shrijeet Paliwal [mailto:shrijeet@rocketfuel.com]=20
Sent: Friday, October 29, 2010 2:39 PM
To: general@hadoop.apache.org; mapreduce-user@hadoop.apache.org
Subject: Re: Loading conf file sitting in HDFS

I got it working by doing this (passing inputstream):

conf.addResource(hdfs.open(apath));

Would still be interested in knowing insights if any one wants to share.

-Shrijeet

On Fri, Oct 29, 2010 at 12:18 PM, Shrijeet Paliwal
<shrijeet@rocketfuel.com>wrote:

> Hello All,
> I am trying to load a conf file located in hdfs.
> I was hoping following would work:
>
> Path apath =3D new Path(conf.get("fs.default.name"),
> "/home/shrijeet/blah.xml");
> conf.addResource(apath);
>
> But I get following exception,
>
> Exception in thread "main" java.lang.RuntimeException:
> hdfs://localhost:8020/home/shrijeet/blah.xml not found
>         at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:12=
97)
>         at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1=
227)
>
>
> File does exist however,
>
> [12:20][shrijeet@shrijeet-desktop]~$ hadoop dfs -ls
> hdfs://localhost:8020/home/shrijeet/blah.xml
> Found 1 items
> -rw-r--r--   1 shrijeet shrijeet        488 2010-10-29 12:16
> /home/shrijeet/blah.xml
>
> Thoughts?
>
> -Shrijeet
>


The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.

From general-return-2307-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 29 20:17:16 2010
Return-Path: <general-return-2307-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43314 invoked from network); 29 Oct 2010 20:17:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Oct 2010 20:17:16 -0000
Received: (qmail 35382 invoked by uid 500); 29 Oct 2010 20:17:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 35330 invoked by uid 500); 29 Oct 2010 20:17:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 35315 invoked by uid 99); 29 Oct 2010 20:17:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 20:17:14 +0000
X-ASF-Spam-Status: No, hits=2.5 required=10.0
	tests=HTML_FONT_FACE_BAD,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of shrijeet@rocketfuelinc.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 20:17:10 +0000
Received: by fxm7 with SMTP id 7so3332299fxm.35
        for <multiple recipients>; Fri, 29 Oct 2010 13:16:48 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.223.86.2 with SMTP id q2mr397863fal.53.1288383408541; Fri, 29
 Oct 2010 13:16:48 -0700 (PDT)
Received: by 10.223.146.1 with HTTP; Fri, 29 Oct 2010 13:16:48 -0700 (PDT)
In-Reply-To: <ABC24175AFD3BE4DA15F4CD375ED413D0639D3EF29@hq-ex-mb02.ad.navteq.com>
References: <AANLkTi=sqaKaVxg2OQZ1kwWavgcg_aYUHVeZWFxwaeGN@mail.gmail.com>
	<AANLkTik9U=oynmZ3KqtW6yaMEcEaoYYO2SttqnLRQiKN@mail.gmail.com>
	<ABC24175AFD3BE4DA15F4CD375ED413D0639D3EF29@hq-ex-mb02.ad.navteq.com>
Date: Fri, 29 Oct 2010 13:16:48 -0700
Message-ID: <AANLkTikkmc_xOoC6TecTVL35+XPQgo-kK-uJafQpzwQq@mail.gmail.com>
Subject: Re: Loading conf file sitting in HDFS
From: Shrijeet Paliwal <shrijeet@rocketfuel.com>
To: general@hadoop.apache.org
Cc: "mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=20cf30433ee86691e20493c7239d

--20cf30433ee86691e20493c7239d
Content-Type: text/plain; charset=ISO-8859-1

Yeah addResource with InputStream as input is the friend.

I got trapped by the notion that since its taking Path as an argument, it
will respect the scheme specified in the path.
But if you look into the implementation of loadResource

// Can't use FileSystem API or we get an infinite loop
        // since FileSystem uses Configuration API.  Use java.io.File
instead.
        File file = new
File(((Path)name).toUri().getPath()).getAbsoluteFile();

This guy just looks at path component.

On Fri, Oct 29, 2010 at 12:54 PM, Segel, Mike <msegel@navteq.com> wrote:

> Insights?
>
> I'm sure someone will correct me...
>
> What I saw was the overloaded method addResource().
>
> Looking at the various methods' input:
> addResource(String name) := name - resource to be added, the classpath is
> examined for a file with that name.
> addResource(URL url) := url - url of the resource to be added, the local
> filesystem is examined directly to find the resource, without referring to
> the classpath.
> addResource(Path file) := file - file-path of resource to be added, the
> local filesystem is examined directly to find the resource, without
> referring to the classpath.
>
> And then:
>
> addResource(InputStream in) := in - InputStream to deserialize the object
> from.
>
> The first 3 specifically are looking for a local file. (Unix file system).
>
> The last one take a generic InputStream which could come from a Unix file
> or an HDFS file.
>
> It looks like the initial intention of loading configuration information
> was from Unix and then HDFS was an afterthought. (My Guess).
>
> HTH
>
> -Mike
>
> -----Original Message-----
> From: Shrijeet Paliwal [mailto:shrijeet@rocketfuel.com]
> Sent: Friday, October 29, 2010 2:39 PM
> To: general@hadoop.apache.org; mapreduce-user@hadoop.apache.org
> Subject: Re: Loading conf file sitting in HDFS
>
> I got it working by doing this (passing inputstream):
>
> conf.addResource(hdfs.open(apath));
>
> Would still be interested in knowing insights if any one wants to share.
>
> -Shrijeet
>
> On Fri, Oct 29, 2010 at 12:18 PM, Shrijeet Paliwal
> <shrijeet@rocketfuel.com>wrote:
>
> > Hello All,
> > I am trying to load a conf file located in hdfs.
> > I was hoping following would work:
> >
> > Path apath = new Path(conf.get("fs.default.name"),
> > "/home/shrijeet/blah.xml");
> > conf.addResource(apath);
> >
> > But I get following exception,
> >
> > Exception in thread "main" java.lang.RuntimeException:
> > hdfs://localhost:8020/home/shrijeet/blah.xml not found
> >         at
> >
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1297)
> >         at
> >
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1227)
> >
> >
> > File does exist however,
> >
> > [12:20][shrijeet@shrijeet-desktop]~$ hadoop dfs -ls
> > hdfs://localhost:8020/home/shrijeet/blah.xml
> > Found 1 items
> > -rw-r--r--   1 shrijeet shrijeet        488 2010-10-29 12:16
> > /home/shrijeet/blah.xml
> >
> > Thoughts?
> >
> > -Shrijeet
> >
>
>
> The information contained in this communication may be CONFIDENTIAL and is
> intended only for the use of the recipient(s) named above.  If you are not
> the intended recipient, you are hereby notified that any dissemination,
> distribution, or copying of this communication, or any of its contents, is
> strictly prohibited.  If you have received this communication in error,
> please notify the sender and delete/destroy the original message and any
> copy of it from your computer or paper files.
>

--20cf30433ee86691e20493c7239d--

From general-return-2308-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 30 01:40:14 2010
Return-Path: <general-return-2308-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67830 invoked from network); 30 Oct 2010 01:40:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Oct 2010 01:40:13 -0000
Received: (qmail 63409 invoked by uid 500); 30 Oct 2010 01:40:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63251 invoked by uid 500); 30 Oct 2010 01:40:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63237 invoked by uid 99); 30 Oct 2010 01:40:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 30 Oct 2010 01:40:12 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of michael_segel@hotmail.com designates 65.55.34.17 as permitted sender)
Received: from [65.55.34.17] (HELO col0-omc1-s7.col0.hotmail.com) (65.55.34.17)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 30 Oct 2010 01:40:04 +0000
Received: from COL117-W9 ([65.55.34.7]) by col0-omc1-s7.col0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Fri, 29 Oct 2010 18:39:42 -0700
Message-ID: <COL117-W92FD98FE05774D2AD7D618F460@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_4d38faf5-95c1-401e-911c-d2d52d0359f9_"
X-Originating-IP: [173.15.87.33]
From: Michael Segel <michael_segel@hotmail.com>
To: <general@hadoop.apache.org>
CC: <mapreduce-user@hadoop.apache.org>
Subject: RE: Loading conf file sitting in HDFS
Date: Fri, 29 Oct 2010 20:39:43 -0500
Importance: Normal
In-Reply-To: <AANLkTikkmc_xOoC6TecTVL35+XPQgo-kK-uJafQpzwQq@mail.gmail.com>
References:
 <AANLkTi=sqaKaVxg2OQZ1kwWavgcg_aYUHVeZWFxwaeGN@mail.gmail.com>,<AANLkTik9U=oynmZ3KqtW6yaMEcEaoYYO2SttqnLRQiKN@mail.gmail.com>,<ABC24175AFD3BE4DA15F4CD375ED413D0639D3EF29@hq-ex-mb02.ad.navteq.com>,<AANLkTikkmc_xOoC6TecTVL35+XPQgo-kK-uJafQpzwQq@mail.gmail.com>
MIME-Version: 1.0
X-OriginalArrivalTime: 30 Oct 2010 01:39:42.0785 (UTC) FILETIME=[53330B10:01CB77D3]
X-Virus-Checked: Checked by ClamAV on apache.org

--_4d38faf5-95c1-401e-911c-d2d52d0359f9_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


Well the reason I kind of felt that it was added as an after thought is tha=
t you kind of have a chicken/egg situation.

I mean if you want to read a config file from HDFS=2C your application has =
to already know how to access the cloud so it had to read a=20
configuration file. So the main use case would be to specify the custom con=
figuration file before you actually connect to the cloud.

I think that the input stream is more generic and can be used universally. =
(Another reason why I thought it might have been done as an add on. I mean =
if you did this as your first attempt=2C why then overload the method?)

HTH

-Mike


> Date: Fri=2C 29 Oct 2010 13:16:48 -0700
> Subject: Re: Loading conf file sitting in HDFS
> From: shrijeet@rocketfuel.com
> To: general@hadoop.apache.org
> CC: mapreduce-user@hadoop.apache.org
>=20
> Yeah addResource with InputStream as input is the friend.
>=20
> I got trapped by the notion that since its taking Path as an argument=2C =
it
> will respect the scheme specified in the path.
> But if you look into the implementation of loadResource
>=20
> // Can't use FileSystem API or we get an infinite loop
>         // since FileSystem uses Configuration API.  Use java.io.File
> instead.
>         File file =3D new
> File(((Path)name).toUri().getPath()).getAbsoluteFile()=3B
>=20
> This guy just looks at path component.
>=20
> On Fri=2C Oct 29=2C 2010 at 12:54 PM=2C Segel=2C Mike <msegel@navteq.com>=
 wrote:
>=20
> > Insights?
> >
> > I'm sure someone will correct me...
> >
> > What I saw was the overloaded method addResource().
> >
> > Looking at the various methods' input:
> > addResource(String name) :=3D name - resource to be added=2C the classp=
ath is
> > examined for a file with that name.
> > addResource(URL url) :=3D url - url of the resource to be added=2C the =
local
> > filesystem is examined directly to find the resource=2C without referri=
ng to
> > the classpath.
> > addResource(Path file) :=3D file - file-path of resource to be added=2C=
 the
> > local filesystem is examined directly to find the resource=2C without
> > referring to the classpath.
> >
> > And then:
> >
> > addResource(InputStream in) :=3D in - InputStream to deserialize the ob=
ject
> > from.
> >
> > The first 3 specifically are looking for a local file. (Unix file syste=
m).
> >
> > The last one take a generic InputStream which could come from a Unix fi=
le
> > or an HDFS file.
> >
> > It looks like the initial intention of loading configuration informatio=
n
> > was from Unix and then HDFS was an afterthought. (My Guess).
> >
> > HTH
> >
> > -Mike
> >
> > -----Original Message-----
> > From: Shrijeet Paliwal [mailto:shrijeet@rocketfuel.com]
> > Sent: Friday=2C October 29=2C 2010 2:39 PM
> > To: general@hadoop.apache.org=3B mapreduce-user@hadoop.apache.org
> > Subject: Re: Loading conf file sitting in HDFS
> >
> > I got it working by doing this (passing inputstream):
> >
> > conf.addResource(hdfs.open(apath))=3B
> >
> > Would still be interested in knowing insights if any one wants to share=
.
> >
> > -Shrijeet
> >
> > On Fri=2C Oct 29=2C 2010 at 12:18 PM=2C Shrijeet Paliwal
> > <shrijeet@rocketfuel.com>wrote:
> >
> > > Hello All=2C
> > > I am trying to load a conf file located in hdfs.
> > > I was hoping following would work:
> > >
> > > Path apath =3D new Path(conf.get("fs.default.name")=2C
> > > "/home/shrijeet/blah.xml")=3B
> > > conf.addResource(apath)=3B
> > >
> > > But I get following exception=2C
> > >
> > > Exception in thread "main" java.lang.RuntimeException:
> > > hdfs://localhost:8020/home/shrijeet/blah.xml not found
> > >         at
> > >
> > org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:12=
97)
> > >         at
> > >
> > org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1=
227)
> > >
> > >
> > > File does exist however=2C
> > >
> > > [12:20][shrijeet@shrijeet-desktop]~$ hadoop dfs -ls
> > > hdfs://localhost:8020/home/shrijeet/blah.xml
> > > Found 1 items
> > > -rw-r--r--   1 shrijeet shrijeet        488 2010-10-29 12:16
> > > /home/shrijeet/blah.xml
> > >
> > > Thoughts?
> > >
> > > -Shrijeet
> > >
> >
> >
> > The information contained in this communication may be CONFIDENTIAL and=
 is
> > intended only for the use of the recipient(s) named above.  If you are =
not
> > the intended recipient=2C you are hereby notified that any disseminatio=
n=2C
> > distribution=2C or copying of this communication=2C or any of its conte=
nts=2C is
> > strictly prohibited.  If you have received this communication in error=
=2C
> > please notify the sender and delete/destroy the original message and an=
y
> > copy of it from your computer or paper files.
> >
 		 	   		  =

--_4d38faf5-95c1-401e-911c-d2d52d0359f9_--

From general-return-2309-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 30 07:41:16 2010
Return-Path: <general-return-2309-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 92284 invoked from network); 30 Oct 2010 07:41:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Oct 2010 07:41:16 -0000
Received: (qmail 80799 invoked by uid 500); 30 Oct 2010 07:41:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80320 invoked by uid 500); 30 Oct 2010 07:41:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80305 invoked by uid 99); 30 Oct 2010 07:41:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 30 Oct 2010 07:41:13 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yhemanth@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 30 Oct 2010 07:41:06 +0000
Received: by vws5 with SMTP id 5so1072679vws.35
        for <general@hadoop.apache.org>; Sat, 30 Oct 2010 00:40:45 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=NLxFlyHAdL/yIe/IBfayhDWXPqCDvekmLJ6IXY19/HY=;
        b=ShTZxtCWG7bCBSv/YqVrpOj4GHGD2x6+uWjAdCU5RbbW8RjCfk+KaxueWzP7SMotnp
         UU96GXeLdf3luwSX1TRVhkRlg0+kafw1WIJQOp34meM49oK7nBIm7m1gO7v6tNxlBSnZ
         VImVjuqtHX7LbBkxS2EwN5hWCpMg8O4CxPZWE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=o9NoW7+T+ylNsUlIxmZTbm4abweuAkgWmnMxyZwcrDLsnTQOI5ay0iIMVI2L5dpnJ0
         53wExGC4Qungj2bxIJbZ1HRdWtdLYCPgvWBA3FBxSFLQicEKNZ+lsaEEfcJ8S0sMfyUq
         YSCE7SHt+rF3ETIqwhOXBZXtfE6Dc4lleweRU=
MIME-Version: 1.0
Received: by 10.224.188.2 with SMTP id cy2mr5774774qab.212.1288424445640; Sat,
 30 Oct 2010 00:40:45 -0700 (PDT)
Received: by 10.220.181.138 with HTTP; Sat, 30 Oct 2010 00:40:45 -0700 (PDT)
In-Reply-To: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
References: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
Date: Sat, 30 Oct 2010 13:10:45 +0530
Message-ID: <AANLkTikeJ9TkXHbD7w1a1CmsQ2jdVQrXnX5sX2JaUsUc@mail.gmail.com>
Subject: Re: [VOTE] ZooKeeper as TLP?
From: Hemanth Yamijala <yhemanth@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

+1

On Wed, Oct 27, 2010 at 11:34 PM, Patrick Hunt <phunt@apache.org> wrote:
> Please vote as to whether you think ZooKeeper should become a
> top-level Apache project.
>
> The ZooKeeper development community has already voted on and approved
> ( http://bit.ly/9VCoTi ) the following draft board resolution. It
> lists all current active ZooKeeper committers (sans one committer who
> has not been active on the project since it's inception
> http://bit.ly/9wLkVV) as initial members of the project management
> committee (PMC) and myself, Patrick Hunt, as the initial chair.
>
>
> Do the good people of Hadoop support sending this request on to the
> Hadoop PMC?
>
> Regards,
>
> Patrick
>
> ------------
>
> =A0 =A0X. Establish the Apache ZooKeeper Project
>
> =A0 =A0 =A0 WHEREAS, the Board of Directors deems it to be in the best
> =A0 =A0 =A0 interests of the Foundation and consistent with the
> =A0 =A0 =A0 Foundation's purpose to establish a Project Management
> =A0 =A0 =A0 Committee charged with the creation and maintenance of
> =A0 =A0 =A0 open-source software related to distributed system coordinati=
on
> =A0 =A0 =A0 for distribution at no charge to the public.
>
> =A0 =A0 =A0 NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> =A0 =A0 =A0 Committee (PMC), to be known as the "Apache ZooKeeper Project=
",
> =A0 =A0 =A0 be and hereby is established pursuant to Bylaws of the
> =A0 =A0 =A0 Foundation; and be it further
>
> =A0 =A0 =A0 RESOLVED, that the Apache ZooKeeper Project be and hereby is
> =A0 =A0 =A0 responsible for the creation and maintenance of software
> =A0 =A0 =A0 related to distributed system coordination; and be it further
>
> =A0 =A0 =A0 RESOLVED, that the office of "Vice President, Apache ZooKeepe=
r" be
> =A0 =A0 =A0 and hereby is created, the person holding such office to
> =A0 =A0 =A0 serve at the direction of the Board of Directors as the chair
> =A0 =A0 =A0 of the Apache ZooKeeper Project, and to have primary responsi=
bility
> =A0 =A0 =A0 for management of the projects within the scope of
> =A0 =A0 =A0 responsibility of the Apache ZooKeeper Project; and be it fur=
ther
>
> =A0 =A0 =A0 RESOLVED, that the persons listed immediately below be and
> =A0 =A0 =A0 hereby are appointed to serve as the initial members of the
> =A0 =A0 =A0 Apache ZooKeeper Project:
>
> =A0 =A0 =A0 =A0 * Patrick Hunt =A0 =A0 =A0 =A0 <phunt@apache.org>
> =A0 =A0 =A0 =A0 * Flavio Junqueira =A0 =A0 <fpj@apache.org>
> =A0 =A0 =A0 =A0 * Mahadev Konar =A0 =A0 =A0 =A0<mahadev@apache.org>
> =A0 =A0 =A0 =A0 * Benjamin Reed =A0 =A0 =A0 =A0<breed@apache.org>
> =A0 =A0 =A0 =A0 * Henry Robinson =A0 =A0 =A0 <henry@apache.org>
>
> =A0 =A0 =A0 NOW, THEREFORE, BE IT FURTHER RESOLVED, that Patrick Hunt
> =A0 =A0 =A0 be appointed to the office of Vice President, Apache ZooKeepe=
r, to
> =A0 =A0 =A0 serve in accordance with and subject to the direction of the
> =A0 =A0 =A0 Board of Directors and the Bylaws of the Foundation until
> =A0 =A0 =A0 death, resignation, retirement, removal or disqualification,
> =A0 =A0 =A0 or until a successor is appointed; and be it further
>
> =A0 =A0 =A0 RESOLVED, that the initial Apache ZooKeeper PMC be and hereby=
 is
> =A0 =A0 =A0 tasked with the creation of a set of bylaws intended to
> =A0 =A0 =A0 encourage open development and increased participation in the
> =A0 =A0 =A0 Apache ZooKeeper Project; and be it further
>
> =A0 =A0 =A0 RESOLVED, that the Apache ZooKeeper Project be and hereby
> =A0 =A0 =A0 is tasked with the migration and rationalization of the Apach=
e
> =A0 =A0 =A0 Hadoop ZooKeeper sub-project; and be it further
>
> =A0 =A0 =A0 RESOLVED, that all responsibilities pertaining to the Apache
> =A0 =A0 =A0 Hadoop ZooKeeper sub-project encumbered upon the
> =A0 =A0 =A0 Apache Hadoop Project are hereafter discharged.
>

From general-return-2310-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 30 18:15:16 2010
Return-Path: <general-return-2310-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 64079 invoked from network); 30 Oct 2010 18:15:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Oct 2010 18:15:16 -0000
Received: (qmail 17436 invoked by uid 500); 30 Oct 2010 18:15:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17360 invoked by uid 500); 30 Oct 2010 18:15:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17352 invoked by uid 99); 30 Oct 2010 18:15:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 30 Oct 2010 18:15:13 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 30 Oct 2010 18:15:09 +0000
Received: by vws5 with SMTP id 5so1722831vws.35
        for <general@hadoop.apache.org>; Sat, 30 Oct 2010 11:14:48 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.224.198.5 with SMTP id em5mr3387065qab.335.1288462487691; Sat,
 30 Oct 2010 11:14:47 -0700 (PDT)
Received: by 10.229.15.70 with HTTP; Sat, 30 Oct 2010 11:14:47 -0700 (PDT)
In-Reply-To: <253774F2-2590-4E67-BB5B-86FE5BD5A555@linkedin.com>
References: <AANLkTik_fT+NSCD1JfQFqmob8cLoTbztYEThpsCSTeYT@mail.gmail.com>
	<A2ED1DE3-B131-4D83-AE0A-382F6A64C06F@apache.org>
	<AANLkTimnqhcG_nazMjSjwvXnvw_udDXJkFq3VbNwXZda@mail.gmail.com>
	<4CC0BEED.2020200@yahoo-inc.com>
	<AANLkTi=WF85J47-JcRghD7AzeaA7qZmScGVQt2fn2NHV@mail.gmail.com>
	<06380AD8-CC00-4983-A93D-7DE01EE60753@yahoo-inc.com>
	<AANLkTik79syC6croM4Mw2ir6VJPeJXMTgb9wa-t5QfBw@mail.gmail.com>
	<253774F2-2590-4E67-BB5B-86FE5BD5A555@linkedin.com>
Date: Sat, 30 Oct 2010 11:14:47 -0700
Message-ID: <AANLkTimJTn5HjVoEZ1MK8rZ0_hL6-htGLLVdvWNhLs3O@mail.gmail.com>
Subject: Re: bringing the codebases back in line
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Thu, Oct 21, 2010 at 11:46 PM, Allen Wittenauer
<awittenauer@linkedin.com> wrote:
>
> On Oct 21, 2010, at 5:51 PM, Eli Collins wrote:
>>
>> - The packaging is Linux specific, we've gotten push back when trying
>> to contribute modifications upstream with Linuxisms since Apache
>> supports non-Linux platforms (namely Solaris).
>
> =A0 =A0 =A0 =A0Oh come now Eli. =A0Just say it: I push everyone really ha=
rd on making things multiple platform to the point of being as trollish as =
this Ian fellow that started this thread.
>
> =A0 =A0 =A0 =A0Most of the Linux-specific modifications that I've seen th=
at have been submitted are specifically to avoid configuring the system pro=
perly*, built in such a way that they weren't pluggable for other operating=
 systems, or could be done in a much more POSIX/OS-agnostic way. =A0The fac=
t that there are parts of Hadoop that don't work properly on Mac OS X (desp=
ite the overwhelming number of Mac laptops in use by Hadoop core devs) alwa=
ys struck me as particularly funny when people get frustrated with me when =
I point this problems out.
>
> =A0 =A0 =A0 =A0It is also worth mentioning that, AFAIK, only Linux and AI=
X have OS-specific code in Hadoop. =A0Attempts to get fixes (not even featu=
res!) for specific issues for other operating systems have been fully rejec=
ted with the cry of "we the community don't want to support specific OS iss=
ues in the core", even tho some of them are direct hinderance to the proper=
 operation of Hadoop on those OSes. =A0We have a very big double standard a=
t work here.

Hey Allen,

Sorry for the slow response, Google marked this as spam ;)

Are there outstanding issues building Hadoop on OSX/Solaris?  I
thought we got all those in trunk.

Thanks,
Eli

From general-return-288-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 05:17:39 2009
Return-Path: <general-return-288-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70318 invoked from network); 1 Jul 2009 05:17:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 05:17:39 -0000
Received: (qmail 5162 invoked by uid 500); 1 Jul 2009 05:17:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5081 invoked by uid 500); 1 Jul 2009 05:17:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5071 invoked by uid 99); 1 Jul 2009 05:17:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 05:17:48 +0000
X-ASF-Spam-Status: No, hits=-1.8 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of qiujie@cn.ibm.com designates 202.81.31.148 as permitted sender)
Received: from [202.81.31.148] (HELO e23smtp06.au.ibm.com) (202.81.31.148)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 05:17:38 +0000
Received: from d23relay02.au.ibm.com (d23relay02.au.ibm.com [202.81.31.244])
	by e23smtp06.au.ibm.com (8.13.1/8.13.1) with ESMTP id n615H5O8026571
	for <general@hadoop.apache.org>; Wed, 1 Jul 2009 15:17:05 +1000
Received: from d23av04.au.ibm.com (d23av04.au.ibm.com [9.190.235.139])
	by d23relay02.au.ibm.com (8.13.8/8.13.8/NCO v9.2) with ESMTP id n615HFNe884786
	for <general@hadoop.apache.org>; Wed, 1 Jul 2009 15:17:15 +1000
Received: from d23av04.au.ibm.com (loopback [127.0.0.1])
	by d23av04.au.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id n615HFg4004197
	for <general@hadoop.apache.org>; Wed, 1 Jul 2009 15:17:15 +1000
Received: from d23m0037.cn.ibm.com (d23m0037.cn.ibm.com [9.181.2.105])
	by d23av04.au.ibm.com (8.12.11.20060308/8.12.11) with ESMTP id n615HEES004162;
	Wed, 1 Jul 2009 15:17:15 +1000
In-Reply-To: <4A3BDF80.5070807@yahoo-inc.com>
To: Konstantin Shvachko <shv@yahoo-inc.com>
Cc: general@hadoop.apache.org, Feng WF Wang <wangfwf@cn.ibm.com>,
        dong.bo@mail.xjtu.edu.cn
MIME-Version: 1.0
Subject: Re: HDFS name node HA roadmap
X-Mailer: Lotus Notes Release 7.0 HF277 June 21, 2006
Message-ID: <OFB2B94B9C.9C6BBF27-ON482575E6.001CB8CE-482575E6.001D0AFC@cn.ibm.com>
From: Jie Qiu <qiujie@cn.ibm.com>
Date: Wed, 1 Jul 2009 13:17:20 +0800
X-MIMETrack: Serialize by Router on D23M0037/23/M/IBM(Release 8.0.1|February 07, 2008) at
 07/01/2009 13:17:21,
	Serialize complete at 07/01/2009 13:17:21
Content-Type: multipart/alternative; boundary="=_alternative 001D0AFA482575E6_="
X-Virus-Checked: Checked by ClamAV on apache.org

--=_alternative 001D0AFA482575E6_=
Content-Type: text/plain; charset="US-ASCII"

Dear Shv

I created new item for Hadoop HA in Hadoop. So, we can use it to start our 
discussion and put the feature in the roadmap of hadoop.

https://issues.apache.org/jira/browse/HADOOP-6121

==================================================
True insight comes from within!

Qiu Jie

Cloud Computing Technology
IBM China Research Lab.
Tel: 8610-58748086  Tieline: 905-8086
FAX: 8610-58748330 
Add.: Building 19 Zhongguancun Software Park, 8 Dongbeiwang WestRoad, 
Haidian District, Beijing,P.R.C.100094
Email: qiujie@cn.ibm.com
==================================================


--=_alternative 001D0AFA482575E6_=--

From general-return-289-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 14:35:36 2009
Return-Path: <general-return-289-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 62954 invoked from network); 1 Jul 2009 14:35:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 14:35:36 -0000
Received: (qmail 23558 invoked by uid 500); 1 Jul 2009 14:35:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23513 invoked by uid 500); 1 Jul 2009 14:35:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23496 invoked by uid 99); 1 Jul 2009 14:35:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 14:35:45 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of fredwang222@gmail.com designates 209.85.212.172 as permitted sender)
Received: from [209.85.212.172] (HELO mail-vw0-f172.google.com) (209.85.212.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 14:35:36 +0000
Received: by vwj2 with SMTP id 2so451406vwj.5
        for <general@hadoop.apache.org>; Wed, 01 Jul 2009 07:35:15 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=vHuCNp0JedKXaflJ7/o3kD/sUjIUkuO1gaSNa0mQCkM=;
        b=UoWx6KlscDP2xrbJC5hd+nWvVyadjwZdrqFwmLrt0CfqqbulLF4PkifRGvlWUuDmXG
         84MblYf6nQTXivL3CEP6KIsCfhJpZ2WHi88mI634KToRwcPeQSqHjarDK+kHS733A8Ca
         EUh1MsB9UhZn0uDx6JBmdTX9WSfsFEflblSXo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=bQMuDH57dt0GimZgjupBoeVT9kDAL1tCp/Vq/WVr1+Dva+WAfvnFP8MuoqsiQgZsmZ
         asbLv649ZB5HM4FfHaRabxEAPT6mg6nic3H5Wf1luZGp1So65jlaF+72Tbw2fWbJC44g
         WyePBYe3Kgu2Cb7hIyWfu9KChPK1SOyEF2q+U=
MIME-Version: 1.0
Received: by 10.220.71.20 with SMTP id f20mr9001341vcj.70.1246458915125; Wed, 
	01 Jul 2009 07:35:15 -0700 (PDT)
Date: Wed, 1 Jul 2009 22:35:15 +0800
Message-ID: <702261350907010735u3502197au800281a5a131d963@mail.gmail.com>
Subject: fail to setup passphraseless ssh and need some help
From: fred wang <fredwang222@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6475d18dcfc12046da5d4a5
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6475d18dcfc12046da5d4a5
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi all,

  I failed to setup passphraseless ssh(I mean, I still need to input
password to do ssh localhost) when I tried to configure Hadoop to run on
psuedo-distributed operation,  could anyone help me solve this issue?
Thanks!

(1)I use the Putty0.6 to remote access to Ubuntu by SSH.

(2) execution steps and ouput

$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
Generating public/private dsa key pair.
Your identification has been saved in /home/xxx/.ssh/id_dsa.
Your public key has been saved in /home/xxx/.ssh/id_dsa.pub.
The key fingerprint is:
a9:39:4c:9b:22:f9:a4:77:70:24:fa:bf:12:f5:81:81 xxx


**note: it doesn't have message  'Enter passphrase (empty for no
passphrase):
    Enter same passphrase again: ' which appear in some introductory paper.
"

$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
no output

$ ssh localhost
The authenticity of host 'localhost (127.0.0.1)' can't be established.
RSA key fingerprint is 4f:a1:ff:ed:0c:46:3e:a9:8c:97:bc:b7:46:3e:35:d2.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
xxx@localhost's password:

--0016e6475d18dcfc12046da5d4a5--

From general-return-290-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 15:19:59 2009
Return-Path: <general-return-290-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78111 invoked from network); 1 Jul 2009 15:19:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 15:19:59 -0000
Received: (qmail 3218 invoked by uid 500); 1 Jul 2009 15:20:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3136 invoked by uid 500); 1 Jul 2009 15:20:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3126 invoked by uid 99); 1 Jul 2009 15:20:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 15:20:08 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of xine.jar@googlemail.com designates 74.125.92.26 as permitted sender)
Received: from [74.125.92.26] (HELO qw-out-2122.google.com) (74.125.92.26)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 15:19:57 +0000
Received: by qw-out-2122.google.com with SMTP id 8so379824qwh.35
        for <general@hadoop.apache.org>; Wed, 01 Jul 2009 08:19:36 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=vyGAoU8Qr9Xmda7cpYufjslSa1Ne51VvtxN+U1D+CtU=;
        b=m6PZ2cec3zUfOEF7nukV4OTTpWSkz5jI/L8dRcQlb2WXHDnh96SbUAwb76l0SXiEEt
         zWuuyPKkAYJEZRY9UpV82u8/FFK+1ZEKUPrR8ISptGd8noIxSeoX9bNiPxyocm9pDStz
         Xb1UVxuN7QycAX6qkVDFdXdECrM1Js/UuNEPM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=HwtWqxc3rnxR0DUDRoaTKkk2ZL9u97FIQfVB80+CGA8djeAvRWvMNgt4vVTRi378qu
         8IgQHhvDpDI4HHxg7DqTiQQi+p0u/nLLnoF+VuQCybpGzgdDI2+ZLaP9urJ7d2EujJIf
         uIHbDDvwyD8+HbdB9TzKgZbkfEzSm/MHPMzUs=
MIME-Version: 1.0
Received: by 10.231.14.70 with SMTP id f6mr2457756iba.9.1246461576265; Wed, 01 
	Jul 2009 08:19:36 -0700 (PDT)
Date: Wed, 1 Jul 2009 17:19:36 +0200
Message-ID: <aa41f20e0907010819s3dd4ff11p66e5e7bda2a08c98@mail.gmail.com>
Subject: I have a problem is sending you an email
From: C J <xine.jar@googlemail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00221532cb647ac0da046da6737c
X-Virus-Checked: Checked by ClamAV on apache.org

--00221532cb647ac0da046da6737c
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hallo,
I am trying to send an email as a follow up to my previous one that I have
sent yesterday,
but for some reasons my email is filtered. Do you have any filtering rules
on the content of
the email, title of the email,...

thank you for helping,me

--00221532cb647ac0da046da6737c--

From general-return-291-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 16:16:11 2009
Return-Path: <general-return-291-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 8746 invoked from network); 1 Jul 2009 16:16:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 16:16:11 -0000
Received: (qmail 94014 invoked by uid 500); 1 Jul 2009 16:16:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93959 invoked by uid 500); 1 Jul 2009 16:16:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93948 invoked by uid 99); 1 Jul 2009 16:16:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 16:16:20 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 16:16:09 +0000
Received: from [192.168.1.22] (snvvpn1-10-73-152-c44.hq.corp.yahoo.com [10.73.152.44])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n61GFM2b004992
	for <general@hadoop.apache.org>; Wed, 1 Jul 2009 09:15:23 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=algaMinGqSXM2z2FLw52ejAD57MOWOPzer6Df2EDFDxtD4YVysP3IwZGgRHG9bNy
Message-ID: <4A4B8B99.10303@yahoo-inc.com>
Date: Wed, 01 Jul 2009 09:15:21 -0700
From: Konstantin Boudnik <cos@yahoo-inc.com>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3pre) Gecko/20090223 Thunderbird/3.0b2
MIME-Version: 1.0
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: fail to setup passphraseless ssh and need some help
References: <702261350907010735u3502197au800281a5a131d963@mail.gmail.com>
In-Reply-To: <702261350907010735u3502197au800281a5a131d963@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Make sure that your ~/.ssh/authorized_keys has permissions 600

Cos

On 7/1/09 7:35 AM, fred wang wrote:
> Hi all,
>
>    I failed to setup passphraseless ssh(I mean, I still need to input
> password to do ssh localhost) when I tried to configure Hadoop to run on
> psuedo-distributed operation,  could anyone help me solve this issue?
> Thanks!
>
> (1)I use the Putty0.6 to remote access to Ubuntu by SSH.
>
> (2) execution steps and ouput
>
> $ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
> Generating public/private dsa key pair.
> Your identification has been saved in /home/xxx/.ssh/id_dsa.
> Your public key has been saved in /home/xxx/.ssh/id_dsa.pub.
> The key fingerprint is:
> a9:39:4c:9b:22:f9:a4:77:70:24:fa:bf:12:f5:81:81 xxx
>
>
> **note: it doesn't have message  'Enter passphrase (empty for no
> passphrase):
>      Enter same passphrase again: ' which appear in some introductory paper.
> "
>
> $ cat ~/.ssh/id_dsa.pub>>  ~/.ssh/authorized_keys
> no output
>
> $ ssh localhost
> The authenticity of host 'localhost (127.0.0.1)' can't be established.
> RSA key fingerprint is 4f:a1:ff:ed:0c:46:3e:a9:8c:97:bc:b7:46:3e:35:d2.
> Are you sure you want to continue connecting (yes/no)? yes
> Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
> xxx@localhost's password:

From general-return-292-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 17:16:17 2009
Return-Path: <general-return-292-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35966 invoked from network); 1 Jul 2009 17:16:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 17:16:17 -0000
Received: (qmail 4821 invoked by uid 500); 1 Jul 2009 17:16:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4751 invoked by uid 500); 1 Jul 2009 17:16:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4741 invoked by uid 99); 1 Jul 2009 17:16:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 17:16:27 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bogdan.maryniuk@gmail.com designates 209.85.216.171 as permitted sender)
Received: from [209.85.216.171] (HELO mail-px0-f171.google.com) (209.85.216.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 17:16:19 +0000
Received: by pxi1 with SMTP id 1so900633pxi.5
        for <general@hadoop.apache.org>; Wed, 01 Jul 2009 10:15:59 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type:content-transfer-encoding;
        bh=u/DKF2sN2nBHr6KBPUjPDsCAGsf+F8otNVoecBpHleg=;
        b=RQjnLGfT18Oy+XC+/ldLfUs5/jgL9TfY6beK5MlTAzrjd1S+GaNHXQ2Xc9Rto0vVFY
         tMUenSwW4DoGPU8chID5LbgV9tPxLZJfdkYNR2Q6H70rYvTAigaGCbCFr2xUfdncmUQ3
         AUKlOpX7sOISHdyjlUeDOZKShgwv8485wSZ/0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=fiM4SH061sJDkJi2rx5wF7SfOAnLkFOmkmX+Vw9siJoH1ee4yCgiXJRDbglILBlGEP
         Rh8vv2p34uPKQsNQNSadRG9c+3K+IvTGuDRuetPQGlbHP5oJXao5JeSbNPbAuEJdk1ss
         ww2gCLcDNNzOsDH9QecxaodF3n5YzaI1ZJDNo=
MIME-Version: 1.0
Received: by 10.142.104.19 with SMTP id b19mr1113091wfc.185.1246468559366; 
	Wed, 01 Jul 2009 10:15:59 -0700 (PDT)
Date: Thu, 2 Jul 2009 02:15:59 +0900
Message-ID: <fa561940907011015i52e77dc2gadd0afc092a7e188@mail.gmail.com>
Subject: 0.20: Configured Capacity: 0 (0 KB)
From: "Bogdan M. Maryniuk" <bogdan.maryniuk@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hello!
Not sure I am on right mailing list, because I had a hard time to
subscribe to at least one of those listed... Sorry in advance.

I've just installed pretty much standard way on a 4 nodes hadoop
cluster, redirected all tmp/fsimage stuff to a pretty much solid
space, but I am getting zero capacity and inability to put any kind of
file. Is this is a correct behavior? If no, what I am missing, please?

Version: 0.20, Java 1.6 Update 13. I see no tracebacks, no errors, no
warnings in any logs. Directories are all writable by hadoop user and
no problem with a space. All daemons are running, everything looks OK
so far. Just zero capacity filesystem that are already 100% full.

Thanks!


$ hadoop dfsadmin -report
Configured Capacity: 0 (0 KB)
Present Capacity: 27648 (27 KB)
DFS Remaining: 0 (0 KB)
DFS Used: 27648 (27 KB)
DFS Used%: 100%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0

-------------------------------------------------
Datanodes available: 3 (3 total, 0 dead)

Name: 192.168.1.242:50010
Decommission Status : Normal
Configured Capacity: 0 (0 KB)
DFS Used: 9216 (9 KB)
Non DFS Used: 0 (0 KB)
DFS Remaining: 0(0 KB)
DFS Used%: 100%
DFS Remaining%: 0%
Last contact: Thu Jul 02 02:03:06 JST 2009


Name: 192.168.1.241:50010
Decommission Status : Normal
Configured Capacity: 0 (0 KB)
DFS Used: 9216 (9 KB)
Non DFS Used: 0 (0 KB)
DFS Remaining: 0(0 KB)
DFS Used%: 100%
DFS Remaining%: 0%
Last contact: Thu Jul 02 02:03:04 JST 2009


Name: 192.168.1.243:50010
Decommission Status : Normal
Configured Capacity: 0 (0 KB)
DFS Used: 9216 (9 KB)
Non DFS Used: 0 (0 KB)
DFS Remaining: 0(0 KB)
DFS Used%: 100%
DFS Remaining%: 0%
Last contact: Thu Jul 02 02:03:04 JST 2009



-- 
Kind regards, BM

Things, that are stupid at the beginning, rarely ends up wisely.

From general-return-293-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 18:55:22 2009
Return-Path: <general-return-293-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 2714 invoked from network); 1 Jul 2009 18:55:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 18:55:22 -0000
Received: (qmail 43847 invoked by uid 500); 1 Jul 2009 18:55:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 43793 invoked by uid 500); 1 Jul 2009 18:55:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 43783 invoked by uid 99); 1 Jul 2009 18:55:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 18:55:32 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cutting@gmail.com designates 209.85.222.196 as permitted sender)
Received: from [209.85.222.196] (HELO mail-pz0-f196.google.com) (209.85.222.196)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 18:55:23 +0000
Received: by pzk34 with SMTP id 34so990064pzk.5
        for <general@hadoop.apache.org>; Wed, 01 Jul 2009 11:55:02 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:sender:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=9xItdnLVk0VFGJowT00MxTuwSX23xrR95zubhwOIQOo=;
        b=TWOGWItBSLzSFhlkYWRUaQucy85U9D9CxWWyUX4b+gLOoojqWqGPhti43LFkpPcTDG
         kEYvoFrkmbhtM1Gr0bYooHOWMXaIZz+QWrOi+vt8VT43K48e0J1DHnZHKHfUuYZ06qg3
         QgSWYdUA9VmCGs9c/EaDsUURFdsMQgWxjponU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:date:from:user-agent:mime-version:to:subject
         :references:in-reply-to:content-type:content-transfer-encoding;
        b=tNDku90edpOiz0SyAx6BLA/D5WRnTRTY9eYn15xmllCVIwZsTXi/8sVUL3KJIVrMu2
         8zkNGsPSdGSUWsKOHpmTzSE9BFz13HlA0mHNqHqfjUHk26GuMGKxb41p4/9ZY5Mggjsu
         cKw4rS1oawB+heAtB2C13QBUB6VrQAP/cpBZ4=
Received: by 10.142.154.14 with SMTP id b14mr831413wfe.67.1246474502783;
        Wed, 01 Jul 2009 11:55:02 -0700 (PDT)
Received: from ?192.168.168.16? (c-76-103-176-123.hsd1.ca.comcast.net [76.103.176.123])
        by mx.google.com with ESMTPS id 22sm4833064wfi.32.2009.07.01.11.55.01
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Wed, 01 Jul 2009 11:55:01 -0700 (PDT)
Sender: Doug Cutting <cutting@gmail.com>
Message-ID: <4A4BB103.30906@apache.org>
Date: Wed, 01 Jul 2009 11:54:59 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.22 (X11/20090608)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: jira notifications on common, hdfs, and mapreduce
References: <C1F6B4F2-B91D-49E5-81E1-4F7FBF83BBA6@yahoo-inc.com>
In-Reply-To: <C1F6B4F2-B91D-49E5-81E1-4F7FBF83BBA6@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Owen O'Malley wrote:
> create/resolved/reopen events are sent to {common,mapreduce,hdfs}-dev
> all jira events are sent to {common,mapreduce,hdfs}-issues

I'd like to remove create/resolved/reopen from the -issues lists, so 
that folks who subscribe to both lists do not see such events twice. 
This would change the policy to:

- create/resolved/reopen events are sent to {common,mapreduce,hdfs}-dev
- all *other* jira events are sent to {common,mapreduce,hdfs}-issues

Objections?

Doug

From general-return-294-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 19:04:44 2009
Return-Path: <general-return-294-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 7204 invoked from network); 1 Jul 2009 19:04:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 19:04:44 -0000
Received: (qmail 58690 invoked by uid 500); 1 Jul 2009 19:04:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58596 invoked by uid 500); 1 Jul 2009 19:04:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58586 invoked by uid 99); 1 Jul 2009 19:04:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 19:04:54 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 19:04:43 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n61J2v1X043326
	for <general@hadoop.apache.org>; Wed, 1 Jul 2009 12:02:57 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=sn26EECsmqa2fp3R+JI3M446SEONasm8zJAL6GAdrpLFcN6rFacRinz0qk68KQkn
Message-Id: <361DBFD2-DF1C-4719-9F35-B94D974ECEC1@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <4A4BB103.30906@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v935.3)
Subject: Re: jira notifications on common, hdfs, and mapreduce
Date: Wed, 1 Jul 2009 12:02:57 -0700
References: <C1F6B4F2-B91D-49E5-81E1-4F7FBF83BBA6@yahoo-inc.com> <4A4BB103.30906@apache.org>
X-Mailer: Apple Mail (2.935.3)
X-Virus-Checked: Checked by ClamAV on apache.org


On Jul 1, 2009, at 11:54 AM, Doug Cutting wrote:

> Owen O'Malley wrote:
>> create/resolved/reopen events are sent to {common,mapreduce,hdfs}-dev
>> all jira events are sent to {common,mapreduce,hdfs}-issues
>
> I'd like to remove create/resolved/reopen from the -issues lists, so  
> that folks who subscribe to both lists do not see such events twice.  
> This would change the policy to:
>
> - create/resolved/reopen events are sent to {common,mapreduce,hdfs}- 
> dev
> - all *other* jira events are sent to {common,mapreduce,hdfs}-issues
>
> Objections?
>

-1 from me.
The create/resolve/reopen traffic is very minor compared to the  
comments. I'd prefer to have the whole thread in one place (*-issues)  
rather than fiddle with multiple mailboxes. *smile*

Arun

From general-return-295-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 19:07:12 2009
Return-Path: <general-return-295-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 8279 invoked from network); 1 Jul 2009 19:07:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 19:07:12 -0000
Received: (qmail 62915 invoked by uid 500); 1 Jul 2009 19:07:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62845 invoked by uid 500); 1 Jul 2009 19:07:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62835 invoked by uid 99); 1 Jul 2009 19:07:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 19:07:22 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [216.252.110.214] (HELO web56205.mail.re3.yahoo.com) (216.252.110.214)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 01 Jul 2009 19:07:13 +0000
Received: (qmail 39068 invoked by uid 60001); 1 Jul 2009 19:06:52 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1246475212; bh=54x5iP1gpscCkuej+eMtBXvl/scpGMmvosCbTuVGdF8=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=SX7PaFcOFrmb47cgS5IzLN03PNWX+2P/isVrSvTtESjPLd8TO7jNKtK3hxugq2WIuGcIFIRq+89dAH/egaH0l03w6taLcbxXOGXlaXgKQ3fiHKc92BVbYI30ucdLSQ9oQxTvgjInb+FpwyecScxEG9wc3iAAtCcNPx3fUcCuFXk=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=sEIurb0PHeJfJ0QGoEBM1JpYgCRCKvKrKOBVmDLs/0PNFMURT0JDmQFCXslXK2KmRSmRXinqj7V8Cm5W3k2JyZwQNgdszrFl3qDPRWUX3SozQu/FiSenkmB5nJ/wow4RPshFfUrp/I8u1Nb1/zgB7cyIhGim91C9/KuiEgzUJqI=;
Message-ID: <177936.38438.qm@web56205.mail.re3.yahoo.com>
X-YMail-OSG: 4bGNIRUVM1nbHtVNLCB8XTzd0DSbOF9GqzIEuxuofgttMRrtY5b_Oylid.77Daop4Lpu2KqTVyGDiaSnD.OQk0pAw4la4_iuQNcnLCHnrFKarKgJz.yIBjP5rU.I0C2cMnDESRDFNsJwE21JLI_0DFZhkvlRmH.KG60HybxukcqgOy7hKFjKPU286Eth0BmucukkEIqrnBx9kkQwb8tuMoi5_W_hyuVe0RQXaAx_9xQ_8leOgGDWTQ8Ly6IOZ2B7aBb2XbiOW_dTTLb3CZZ0fsf8k44PgY7MFpDsq0uEH6NEET2SVXUYiS6v1dkkDGBFmlByamDry14klbkr28_bbCbTKbbEOIXbxVprL8Xz
Received: from [216.145.54.15] by web56205.mail.re3.yahoo.com via HTTP; Wed, 01 Jul 2009 12:06:51 PDT
X-Mailer: YahooMailRC/1357.22 YahooMailWebService/0.7.289.15
References: <C1F6B4F2-B91D-49E5-81E1-4F7FBF83BBA6@yahoo-inc.com> <4A4BB103.30906@apache.org>
Date: Wed, 1 Jul 2009 12:06:51 -0700 (PDT)
From: "Tsz Wo \(Nicholas\), Sze" <s29752-hadoopgeneral@yahoo.com>
Subject: Re: jira notifications on common, hdfs, and mapreduce
To: general@hadoop.apache.org
In-Reply-To: <4A4BB103.30906@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org


+1 on Doug's suggestion.
I am getting > 100 emails everyday.  I map the emails to different accounts.  It is good to reduce it.  Otherwise, I probably need to put them on hdfs.  :)

Nicholas Sze




----- Original Message ----
> From: Doug Cutting <cutting@apache.org>
> To: general@hadoop.apache.org
> Sent: Wednesday, July 1, 2009 11:54:59 AM
> Subject: Re: jira notifications on common, hdfs, and mapreduce
> 
> Owen O'Malley wrote:
> > create/resolved/reopen events are sent to {common,mapreduce,hdfs}-dev
> > all jira events are sent to {common,mapreduce,hdfs}-issues
> 
> I'd like to remove create/resolved/reopen from the -issues lists, so that folks 
> who subscribe to both lists do not see such events twice. This would change the 
> policy to:
> 
> - create/resolved/reopen events are sent to {common,mapreduce,hdfs}-dev
> - all *other* jira events are sent to {common,mapreduce,hdfs}-issues
> 
> Objections?
> 
> Doug


From general-return-296-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 19:22:51 2009
Return-Path: <general-return-296-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 20418 invoked from network); 1 Jul 2009 19:22:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 19:22:51 -0000
Received: (qmail 89220 invoked by uid 500); 1 Jul 2009 19:23:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89159 invoked by uid 500); 1 Jul 2009 19:23:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89149 invoked by uid 99); 1 Jul 2009 19:23:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 19:23:01 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cutting@gmail.com designates 209.85.222.196 as permitted sender)
Received: from [209.85.222.196] (HELO mail-pz0-f196.google.com) (209.85.222.196)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 19:22:51 +0000
Received: by pzk34 with SMTP id 34so1004972pzk.5
        for <general@hadoop.apache.org>; Wed, 01 Jul 2009 12:22:30 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:sender:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=QFUpPJKMHDXrgQAU5QUYanr0IPGfZMXduc+L+dctW7s=;
        b=hAb9892AwT5J1Z4v0Xt9r8wRsn90TTn5/ahd/jSbdks4qmchQkYVErLZMJwvkI2Yx4
         H5bfn2+7pSXLECuc/vsjaKTcvemp1XjN3OUYjhELWk/ao3g0hYB8NsqN6vSduWdIv+yJ
         BvO+Qy1wgIAzRWrd9IWsoFjsK0XrsnyP0X6e4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:date:from:user-agent:mime-version:to:subject
         :references:in-reply-to:content-type:content-transfer-encoding;
        b=Vmao7nE4dml6FyyjbWgmRbZQCa5lZLEhIf5x3x4BBd5R5Mt9reL0KXDP8HDLgl4/tV
         zlAxSyNnsh8nBKPo70thPyq2eEUJvi8Z2BrezrzhG+ZggqIiVv0h0OlmaIfVTxl26jCA
         6yshi92gaFgISDfFEPsZFkEfXSpQ9U+p9RHEg=
Received: by 10.142.79.12 with SMTP id c12mr1301132wfb.286.1246476148717;
        Wed, 01 Jul 2009 12:22:28 -0700 (PDT)
Received: from ?192.168.168.16? (c-76-103-176-123.hsd1.ca.comcast.net [76.103.176.123])
        by mx.google.com with ESMTPS id 24sm4643795wff.38.2009.07.01.12.22.27
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Wed, 01 Jul 2009 12:22:28 -0700 (PDT)
Sender: Doug Cutting <cutting@gmail.com>
Message-ID: <4A4BB771.60605@apache.org>
Date: Wed, 01 Jul 2009 12:22:25 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.22 (X11/20090608)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: jira notifications on common, hdfs, and mapreduce
References: <C1F6B4F2-B91D-49E5-81E1-4F7FBF83BBA6@yahoo-inc.com> <4A4BB103.30906@apache.org> <361DBFD2-DF1C-4719-9F35-B94D974ECEC1@yahoo-inc.com>
In-Reply-To: <361DBFD2-DF1C-4719-9F35-B94D974ECEC1@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Arun C Murthy wrote:
> -1 from me.
> The create/resolve/reopen traffic is very minor compared to the 
> comments. I'd prefer to have the whole thread in one place (*-issues) 
> rather than fiddle with multiple mailboxes. *smile*

I assume you mean "thread" loosely, since all messages related to an 
issue are not threaded together by any mail reader that I know of, since 
Jira changes the subject and does not set a "References" or 
"In-Reply-To" header.

You subscribe to both lists and really prefer receiving create/resolve 
messages twice?

I subscribe to both, and have a filter that marks-as-read messages sent 
to -issues, so I have the full set of messages available to searches, 
but only am notified about new and watched issues.

Doug

From general-return-297-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 20:06:45 2009
Return-Path: <general-return-297-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35883 invoked from network); 1 Jul 2009 20:06:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 20:06:45 -0000
Received: (qmail 73152 invoked by uid 500); 1 Jul 2009 20:06:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73072 invoked by uid 500); 1 Jul 2009 20:06:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73062 invoked by uid 99); 1 Jul 2009 20:06:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 20:06:55 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 20:06:44 +0000
Received: from [10.72.104.137] (goodenter-lm.corp.yahoo.com [10.72.104.137])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n61K62Jd073418
	for <general@hadoop.apache.org>; Wed, 1 Jul 2009 13:06:02 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=2CcdXRseXKyTxPW/ZnpWSn7FxxB5t7yH8oOmkltLllK9gyXe4R/5543G+MA6AX41
Message-ID: <4A4BC1AA.3060901@yahoo-inc.com>
Date: Wed, 01 Jul 2009 13:06:02 -0700
From: Konstantin Boudnik <cos@yahoo-inc.com>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3pre) Gecko/20090223 Thunderbird/3.0b2
MIME-Version: 1.0
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: jira notifications on common, hdfs, and mapreduce
References: <C1F6B4F2-B91D-49E5-81E1-4F7FBF83BBA6@yahoo-inc.com> <4A4BB103.30906@apache.org> <361DBFD2-DF1C-4719-9F35-B94D974ECEC1@yahoo-inc.com> <4A4BB771.60605@apache.org>
In-Reply-To: <4A4BB771.60605@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org



On 7/1/09 12:22 PM, Doug Cutting wrote:
> Arun C Murthy wrote:
>> -1 from me.
>> The create/resolve/reopen traffic is very minor compared to the
>> comments. I'd prefer to have the whole thread in one place (*-issues)
>> rather than fiddle with multiple mailboxes. *smile*
>
> I assume you mean "thread" loosely, since all messages related to an
> issue are not threaded together by any mail reader that I know of, since
> Jira changes the subject and does not set a "References" or
> "In-Reply-To" header.

Actually, Thunderbird and Mutt are at least two which does thread JIRA's threads 
properly.

The way they do so is exactly by using Message-ID and In-Reply-To fields of a 
RFC-822 header.

Cos

> You subscribe to both lists and really prefer receiving create/resolve
> messages twice?
>
> I subscribe to both, and have a filter that marks-as-read messages sent
> to -issues, so I have the full set of messages available to searches,
> but only am notified about new and watched issues.
>
> Doug

From general-return-298-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 20:29:25 2009
Return-Path: <general-return-298-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 41930 invoked from network); 1 Jul 2009 20:29:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 20:29:23 -0000
Received: (qmail 9480 invoked by uid 500); 1 Jul 2009 20:29:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 9402 invoked by uid 500); 1 Jul 2009 20:29:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 9392 invoked by uid 99); 1 Jul 2009 20:29:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 20:29:33 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cutting@gmail.com designates 209.85.200.171 as permitted sender)
Received: from [209.85.200.171] (HELO wf-out-1314.google.com) (209.85.200.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 20:29:23 +0000
Received: by wf-out-1314.google.com with SMTP id 23so433761wfg.2
        for <general@hadoop.apache.org>; Wed, 01 Jul 2009 13:29:03 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:sender:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=QiO5E9NlYw/D9pmxZPj3ynYSAnXyDMLMOf2w2Sfx3tw=;
        b=VbANDl3g4hUs3iAdXOo9cbtrp9yEVfMugO31nABKewGSWpVaRvi1Yx4gd90ldIUANu
         ayJF8rr9zZ/qR78WnwamtLxNv2b/fNX8nJ7Y1RL7Sl6EjIZns8SgUO5m4Ul8qlSyD3M2
         HtrVwfUjAHm15Fj+RIb0aatkJ/cxJM00q7VYE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:date:from:user-agent:mime-version:to:subject
         :references:in-reply-to:content-type:content-transfer-encoding;
        b=Z6zgL6WsuFyOUHMswHDVAO7f4s/7h5BEAqKRYPA3POQyvkYBe/8uwbCcUqTcAJZfxR
         BjubEzXmhEQ9FSps/pixFxsP2oGCEWwauu4ILL8oS+u20tt374WURD8WjsCdzGSIEmvO
         3RZZ50pBjRbg3/RYAT07mxmPGcMcE7B5TxK7A=
Received: by 10.142.100.12 with SMTP id x12mr1301253wfb.206.1246480143614;
        Wed, 01 Jul 2009 13:29:03 -0700 (PDT)
Received: from ?192.168.168.16? (c-76-103-176-123.hsd1.ca.comcast.net [76.103.176.123])
        by mx.google.com with ESMTPS id 28sm4794241wfd.24.2009.07.01.13.29.02
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Wed, 01 Jul 2009 13:29:02 -0700 (PDT)
Sender: Doug Cutting <cutting@gmail.com>
Message-ID: <4A4BC70C.9080900@apache.org>
Date: Wed, 01 Jul 2009 13:29:00 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.22 (X11/20090608)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: jira notifications on common, hdfs, and mapreduce
References: <C1F6B4F2-B91D-49E5-81E1-4F7FBF83BBA6@yahoo-inc.com> <4A4BB103.30906@apache.org> <361DBFD2-DF1C-4719-9F35-B94D974ECEC1@yahoo-inc.com> <4A4BB771.60605@apache.org> <4A4BC1AA.3060901@yahoo-inc.com>
In-Reply-To: <4A4BC1AA.3060901@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Konstantin Boudnik wrote:
> Actually, Thunderbird and Mutt are at least two which does thread JIRA's 
> threads properly.
> 
> The way they do so is exactly by using Message-ID and In-Reply-To fields 
> of a RFC-822 header.

The Message-ID is different for each message, and is thus alone not 
useful for threading.  I don't see an In-Reply-To header in Jira-sent 
messages, nor a References header.  These are the preferred means for 
threading according to RFC2822, although some readers thread by subject 
too.  Thus all comments relative to an issue may be threaded, but not 
together with other updates, like assignment, attachment, etc., since 
these use different subjects.

For example, do you see these headers in http://tinyurl.com/lw3cse?

Doug





From general-return-299-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 20:42:22 2009
Return-Path: <general-return-299-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45416 invoked from network); 1 Jul 2009 20:42:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 20:42:22 -0000
Received: (qmail 33274 invoked by uid 500); 1 Jul 2009 20:42:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33196 invoked by uid 500); 1 Jul 2009 20:42:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33186 invoked by uid 99); 1 Jul 2009 20:42:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 20:42:32 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 20:42:20 +0000
Received: from [10.72.104.137] (goodenter-lm.corp.yahoo.com [10.72.104.137])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n61KeUt6080347
	for <general@hadoop.apache.org>; Wed, 1 Jul 2009 13:40:31 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=HmtoJJx1QCWabN2n2HgMhvlmjT/lJxtGBSSKj3zdCAyaI3YnuVrinlXow127F6FF
Message-ID: <4A4BC9BE.5060408@yahoo-inc.com>
Date: Wed, 01 Jul 2009 13:40:30 -0700
From: Konstantin Boudnik <cos@yahoo-inc.com>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3pre) Gecko/20090223 Thunderbird/3.0b2
MIME-Version: 1.0
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: jira notifications on common, hdfs, and mapreduce
References: <C1F6B4F2-B91D-49E5-81E1-4F7FBF83BBA6@yahoo-inc.com> <4A4BB103.30906@apache.org> <361DBFD2-DF1C-4719-9F35-B94D974ECEC1@yahoo-inc.com> <4A4BB771.60605@apache.org> <4A4BC1AA.3060901@yahoo-inc.com> <4A4BC70C.9080900@apache.org>
In-Reply-To: <4A4BC70C.9080900@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Doug,

Message-ID is unique. However, what I can see in my mail is that all subsequent 
emails (after JIRA created message) have its Message-ID in their In-Reply-To field.

Say,
Subject: [jira] Created: (HADOOP-6110) test-patch takes 45min!
Thread-Topic: [jira] Created: (HADOOP-6110) test-patch takes 45min!
Thread-Index: Acn2Jd4/MZg/3jVETQKT5ZEGzVtNCA==
Message-ID: <1892144747.1245997027374.JavaMail.jira@brutus>
List-Help: <mailto:core-dev-help@hadoop.apache.org>

and

Subject: [jira] Updated: (HADOOP-6110) test-patch takes 45min!
Thread-Topic: [jira] Updated: (HADOOP-6110) test-patch takes 45min!
Thread-Index: Acn4rGYcaU6FaceJSyGggrQ+1INmpg==
Message-ID: <1159605925.1246274687243.JavaMail.jira@brutus>
List-Help: <mailto:core-dev-help@hadoop.apache.org>
List-Unsubscribe: <mailto:core-dev-unsubscribe@hadoop.apache.org>
In-Reply-To: <1892144747.1245997027374.JavaMail.jira@brutus>

I don't see In-Reply-To field in the example you've sent to me. Is it because 
some tags are being stripped out by mail archiving software?

Cos

On 7/1/09 1:29 PM, Doug Cutting wrote:
> Konstantin Boudnik wrote:
>> Actually, Thunderbird and Mutt are at least two which does thread JIRA's
>> threads properly.
>>
>> The way they do so is exactly by using Message-ID and In-Reply-To fields
>> of a RFC-822 header.
>
> The Message-ID is different for each message, and is thus alone not
> useful for threading.  I don't see an In-Reply-To header in Jira-sent
> messages, nor a References header.  These are the preferred means for
> threading according to RFC2822, although some readers thread by subject
> too.  Thus all comments relative to an issue may be threaded, but not
> together with other updates, like assignment, attachment, etc., since
> these use different subjects.
>
> For example, do you see these headers in http://tinyurl.com/lw3cse?
>
> Doug
>
>
>
>

From general-return-300-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 02:03:06 2009
Return-Path: <general-return-300-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 388 invoked from network); 2 Jul 2009 02:03:06 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 02:03:06 -0000
Received: (qmail 50108 invoked by uid 500); 2 Jul 2009 02:03:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50042 invoked by uid 500); 2 Jul 2009 02:03:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50032 invoked by uid 99); 2 Jul 2009 02:03:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 02:03:16 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bogdan.maryniuk@gmail.com designates 209.85.222.196 as permitted sender)
Received: from [209.85.222.196] (HELO mail-pz0-f196.google.com) (209.85.222.196)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 02:03:08 +0000
Received: by pzk34 with SMTP id 34so1170453pzk.5
        for <general@hadoop.apache.org>; Wed, 01 Jul 2009 19:02:47 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type:content-transfer-encoding;
        bh=P6TBPMqnhTn/sAqf+CY8Q3FK/u1z7j3/O+iCs+6+UH0=;
        b=U7JogvrbUIWVeBNqKr1JCrZi7jZWuRTqqnfO++PGPo1ZgahBTZShtHuLnZgu78x76T
         YzNt8QFOc29J3lGrfacmoFgrNGUpXbY68yQA+VzJrLqvajCaPUlzoCc9463xpyJRivAo
         UJVUH4zG2ZJdK9+ioeQkcvaiyxIEUFyo+jQQM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=rxOerbsz+xz2A+X1sKWbrNZjFhgFzwQiNUEbHOpbWUL+BsqHz/9zllynkshPhH9qsy
         im7d1hljgW5XqLkRtOoYKOXs9UswYrrNtrelSHy1hI+qniXV7ANnn8vVAQ/L89ACAttR
         dd5zyH4Bk+wXwf06kKjzUL5tXVC/GhSL4vZaM=
MIME-Version: 1.0
Received: by 10.142.246.20 with SMTP id t20mr1285708wfh.181.1246500167886; 
	Wed, 01 Jul 2009 19:02:47 -0700 (PDT)
Date: Thu, 2 Jul 2009 11:02:47 +0900
Message-ID: <fa561940907011902j7035d55fo3a21d7d3c11262a9@mail.gmail.com>
Subject: Mailing lists are broken?
From: "Bogdan M. Maryniuk" <bogdan.maryniuk@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hello.
Tried http://hadoop.apache.org/core/mailing_lists.html
Address "core-user-subscribe@hadoop.apache.org" returns error.

Is that's OK?

-- 
Kind regards, BM

Things, that are stupid at the beginning, rarely ends up wisely.

From general-return-301-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 05:11:31 2009
Return-Path: <general-return-301-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 64200 invoked from network); 2 Jul 2009 05:11:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 05:11:31 -0000
Received: (qmail 78770 invoked by uid 500); 2 Jul 2009 05:11:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78547 invoked by uid 500); 2 Jul 2009 05:11:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78537 invoked by uid 99); 2 Jul 2009 05:11:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 05:11:39 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of fredwang222@gmail.com designates 209.85.212.172 as permitted sender)
Received: from [209.85.212.172] (HELO mail-vw0-f172.google.com) (209.85.212.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 05:11:29 +0000
Received: by vwj2 with SMTP id 2so839381vwj.5
        for <general@hadoop.apache.org>; Wed, 01 Jul 2009 22:11:08 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=8PQ12GXd1XJ6wG/RFgmMpHmwwt36qZBqBqMmRKuw/5g=;
        b=szEuS7QQ78msdNBSlZSaTm2uoO8AKu/EogeD74deGQJKd1cJQo2ljPS0NKkAbYYeTi
         wgDrZlU9tbi/A9ShsRLjjv3YzuzENm61HLqRUxVgMID4RwTEn5MQD6ZM6tphmot3jM/j
         /dhOv31bHpw6FnKiaPsKg1t36hTLuEHb5rgF0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=eOHYtAJcrJADb4m1oBb5ov4WL1aIPEDngPa2PxWOi7GglOg4mvACfnpxcLej1Xc7Wp
         48+bpzeXivc96TJpEn5S80996B/RlabD5ybJSfXb7IhltM0uSfQosSmlDIzk3iMHeKj6
         kQMZQzJbNDrQoemyOV2ggdFuKWW2F0gCnKXWQ=
MIME-Version: 1.0
Received: by 10.220.72.79 with SMTP id l15mr8333999vcj.4.1246511468614; Wed, 
	01 Jul 2009 22:11:08 -0700 (PDT)
In-Reply-To: <4A4B8B99.10303@yahoo-inc.com>
References: <702261350907010735u3502197au800281a5a131d963@mail.gmail.com>
	 <4A4B8B99.10303@yahoo-inc.com>
Date: Thu, 2 Jul 2009 13:11:08 +0800
Message-ID: <702261350907012211r17dec368u17e18052383a8822@mail.gmail.com>
Subject: Re: fail to setup passphraseless ssh and need some help
From: fred wang <fredwang222@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64755aa4b95e1046db2118b
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e64755aa4b95e1046db2118b
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

I have setup ./.ssh/authorized keys has permssion 600, but it didn't work.
Thanks anyway

ls -l .ssh/authorized_keys
-rw------- 1 xxx xxx 1222 2009-07-02 13:08 .ssh/authorized_keys

On Thu, Jul 2, 2009 at 12:15 AM, Konstantin Boudnik <cos@yahoo-inc.com>wrote:

> Make sure that your ~/.ssh/authorized_keys has permissions 600
>
> Cos
>
>
> On 7/1/09 7:35 AM, fred wang wrote:
>
>> Hi all,
>>
>>   I failed to setup passphraseless ssh(I mean, I still need to input
>> password to do ssh localhost) when I tried to configure Hadoop to run on
>> psuedo-distributed operation,  could anyone help me solve this issue?
>> Thanks!
>>
>> (1)I use the Putty0.6 to remote access to Ubuntu by SSH.
>>
>> (2) execution steps and ouput
>>
>> $ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
>> Generating public/private dsa key pair.
>> Your identification has been saved in /home/xxx/.ssh/id_dsa.
>> Your public key has been saved in /home/xxx/.ssh/id_dsa.pub.
>> The key fingerprint is:
>> a9:39:4c:9b:22:f9:a4:77:70:24:fa:bf:12:f5:81:81 xxx
>>
>>
>> **note: it doesn't have message  'Enter passphrase (empty for no
>> passphrase):
>>     Enter same passphrase again: ' which appear in some introductory
>> paper.
>> "
>>
>> $ cat ~/.ssh/id_dsa.pub>>  ~/.ssh/authorized_keys
>> no output
>>
>> $ ssh localhost
>> The authenticity of host 'localhost (127.0.0.1)' can't be established.
>> RSA key fingerprint is 4f:a1:ff:ed:0c:46:3e:a9:8c:97:bc:b7:46:3e:35:d2.
>> Are you sure you want to continue connecting (yes/no)? yes
>> Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
>> xxx@localhost's password:
>>
>

--0016e64755aa4b95e1046db2118b--

From general-return-302-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 05:18:44 2009
Return-Path: <general-return-302-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66751 invoked from network); 2 Jul 2009 05:18:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 05:18:44 -0000
Received: (qmail 84565 invoked by uid 500); 2 Jul 2009 05:18:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84495 invoked by uid 500); 2 Jul 2009 05:18:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84485 invoked by uid 99); 2 Jul 2009 05:18:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 05:18:54 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 05:18:43 +0000
Received: from [192.168.102.128] (snvvpn1-10-73-152-c179.hq.corp.yahoo.com [10.73.152.179])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n625I833078548
	for <general@hadoop.apache.org>; Wed, 1 Jul 2009 22:18:08 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=mnbWGIubH/4ldClQmm+yCwFk+jD6mHH4Ubl1uD/w0YFp6cJQRwlx8NkJJ0CqYDWA
Message-ID: <4A4C430F.7080306@yahoo-inc.com>
Date: Wed, 01 Jul 2009 22:18:07 -0700
From: Konstantin Boudnik <cos@yahoo-inc.com>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3pre) Gecko/20090223 Thunderbird/3.0b2
MIME-Version: 1.0
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: fail to setup passphraseless ssh and need some help
References: <702261350907010735u3502197au800281a5a131d963@mail.gmail.com>	 <4A4B8B99.10303@yahoo-inc.com> <702261350907012211r17dec368u17e18052383a8822@mail.gmail.com>
In-Reply-To: <702261350907012211r17dec368u17e18052383a8822@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Yet another possibility is that your SSH daemon isn't configured to accept 
publickey as a valid authorization mean.

Try to do ssh -v localhost and check if there's something similar to the following:

debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Trying private key: /home/xxx/.ssh/identity
debug1: Trying private key: /home/xxx/.ssh/id_rsa
debug1: Offering public key: /home/xxx/.ssh/id_dsa
debug1: Server accepts key: pkalg ssh-dss blen 435
debug1: read PEM private key done: type DSA
debug1: Authentication succeeded (publickey).

Cos

On 7/1/09 10:11 PM, fred wang wrote:
> I have setup ./.ssh/authorized keys has permssion 600, but it didn't work.
> Thanks anyway
>
> ls -l .ssh/authorized_keys
> -rw------- 1 xxx xxx 1222 2009-07-02 13:08 .ssh/authorized_keys
>
> On Thu, Jul 2, 2009 at 12:15 AM, Konstantin Boudnik<cos@yahoo-inc.com>wrote:
>
>> Make sure that your ~/.ssh/authorized_keys has permissions 600
>>
>> Cos
>>
>>
>> On 7/1/09 7:35 AM, fred wang wrote:
>>
>>> Hi all,
>>>
>>>    I failed to setup passphraseless ssh(I mean, I still need to input
>>> password to do ssh localhost) when I tried to configure Hadoop to run on
>>> psuedo-distributed operation,  could anyone help me solve this issue?
>>> Thanks!
>>>
>>> (1)I use the Putty0.6 to remote access to Ubuntu by SSH.
>>>
>>> (2) execution steps and ouput
>>>
>>> $ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
>>> Generating public/private dsa key pair.
>>> Your identification has been saved in /home/xxx/.ssh/id_dsa.
>>> Your public key has been saved in /home/xxx/.ssh/id_dsa.pub.
>>> The key fingerprint is:
>>> a9:39:4c:9b:22:f9:a4:77:70:24:fa:bf:12:f5:81:81 xxx
>>>
>>>
>>> **note: it doesn't have message  'Enter passphrase (empty for no
>>> passphrase):
>>>      Enter same passphrase again: ' which appear in some introductory
>>> paper.
>>> "
>>>
>>> $ cat ~/.ssh/id_dsa.pub>>   ~/.ssh/authorized_keys
>>> no output
>>>
>>> $ ssh localhost
>>> The authenticity of host 'localhost (127.0.0.1)' can't be established.
>>> RSA key fingerprint is 4f:a1:ff:ed:0c:46:3e:a9:8c:97:bc:b7:46:3e:35:d2.
>>> Are you sure you want to continue connecting (yes/no)? yes
>>> Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
>>> xxx@localhost's password:
>>>

From general-return-303-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 05:51:41 2009
Return-Path: <general-return-303-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87290 invoked from network); 2 Jul 2009 05:51:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 05:51:40 -0000
Received: (qmail 24541 invoked by uid 500); 2 Jul 2009 05:51:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24460 invoked by uid 500); 2 Jul 2009 05:51:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24450 invoked by uid 99); 2 Jul 2009 05:51:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 05:51:50 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of fredwang222@gmail.com designates 209.85.212.172 as permitted sender)
Received: from [209.85.212.172] (HELO mail-vw0-f172.google.com) (209.85.212.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 05:51:39 +0000
Received: by vwj2 with SMTP id 2so849797vwj.5
        for <general@hadoop.apache.org>; Wed, 01 Jul 2009 22:51:18 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=gIYFVxl742VOZs6j1CjQxzM1PMiRvoOnSDlGhjCbpsg=;
        b=iuF7MQ5R3SkvO4HkPBPzmQHt14+JjDPoyyKfdcXL7d6N8hPO5QFOKFs3/x9Zby4BrJ
         cQXi0ULKVug3v0qY7tPIz6c2z8cyREP8NWjoMvnlq6MUY/rEsNwSKnAPOpYqoTTF6LqX
         1dhCdcJFv83I+LSwjT++/X5HDqHWV7gFgcp6s=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=qG8fhGq8oU8tyx30hFyOZel2JCw1F0DRaB6KarF4UCCITlZrHGtEIsgA2wNuZo4dbJ
         oKVK0QBIQ6DeVU/NDIpolsugnlTmikaaKl+GR7UkVUdLXySL1muvx2odvl9V4xQMDoC+
         xOrbW5FyWksQI5fueqnwJdO2HEUaVl+u7vdOo=
MIME-Version: 1.0
Received: by 10.220.71.6 with SMTP id f6mr10710972vcj.16.1246513878067; Wed, 
	01 Jul 2009 22:51:18 -0700 (PDT)
In-Reply-To: <4A4C430F.7080306@yahoo-inc.com>
References: <702261350907010735u3502197au800281a5a131d963@mail.gmail.com>
	 <4A4B8B99.10303@yahoo-inc.com>
	 <702261350907012211r17dec368u17e18052383a8822@mail.gmail.com>
	 <4A4C430F.7080306@yahoo-inc.com>
Date: Thu, 2 Jul 2009 13:51:18 +0800
Message-ID: <702261350907012251s41b18b24m5dc54015007d494e@mail.gmail.com>
Subject: Re: fail to setup passphraseless ssh and need some help
From: fred wang <fredwang222@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485e8d216e8e9c2046db2a032
X-Virus-Checked: Checked by ClamAV on apache.org

--001485e8d216e8e9c2046db2a032
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Here is the output of ssh -v localhost  and the configuration of ssh_config,


xxx@xxx-desktop:~$ ssh -v localhost

OpenSSH_4.7p1 Debian-8ubuntu1.2, OpenSSL 0.9.8g 19 Oct 2007

debug1: Reading configuration data /etc/ssh/ssh_config

debug1: Applying options for *

debug1: Connecting to localhost [127.0.0.1] port 22.

debug1: Connection established.

debug1: identity file /home/xxx/.ssh/identity type -1

debug1: identity file /home/xxx/.ssh/id_rsa type -1

debug1: identity file /home/xxx/.ssh/id_dsa type 2

debug1: Remote protocol version 2.0, remote software version OpenSSH_4.7p1
Debian-8ubuntu1.2

debug1: match: OpenSSH_4.7p1 Debian-8ubuntu1.2 pat OpenSSH*

debug1: Enabling compatibility mode for protocol 2.0

debug1: Local version string SSH-2.0-OpenSSH_4.7p1 Debian-8ubuntu1.2

debug1: SSH2_MSG_KEXINIT sent

debug1: SSH2_MSG_KEXINIT received

debug1: kex: server->client aes128-cbc hmac-md5 none

debug1: kex: client->server aes128-cbc hmac-md5 none

debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent

debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP

debug1: SSH2_MSG_KEX_DH_GEX_INIT sent

debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY

debug1: Host 'localhost' is known and matches the RSA host key.

debug1: Found key in /home/xxx/.ssh/known_hosts:1

debug1: ssh_rsa_verify: signature correct

debug1: SSH2_MSG_NEWKEYS sent

debug1: expecting SSH2_MSG_NEWKEYS

debug1: SSH2_MSG_NEWKEYS received

debug1: SSH2_MSG_SERVICE_REQUEST sent

debug1: SSH2_MSG_SERVICE_ACCEPT received

debug1: Authentications that can continue: publickey,password

debug1: Next authentication method: publickey

debug1: Trying private key: /home/xxx/.ssh/identity

debug1: Trying private key: /home/xxx/.ssh/id_rsa

debug1: Offering public key: /home/xxx/.ssh/id_dsa

debug1: Authentications that can continue: publickey,password

debug1: Next authentication method: password

xxx@localhost's password:







xxx@xxx:~$ vi /etc/ssh/sshd_config

#KerberosOrLocalPasswd yes

#KerberosTicketCleanup yes



# GSSAPI options

#GSSAPIAuthentication no

#GSSAPICleanupCredentials yes



X11Forwarding yes

X11DisplayOffset 10

PrintMotd no

PrintLastLog yes

TCPKeepAlive yes

#UseLogin no



#MaxStartups 10:30:60

#Banner /etc/issue.net



# Allow client to pass locale environment variables

AcceptEnv LANG LC_*



Subsystem sftp /usr/lib/openssh/sftp-server



UsePAM yes



On Thu, Jul 2, 2009 at 1:18 PM, Konstantin Boudnik <cos@yahoo-inc.com>wrote:

> Yet another possibility is that your SSH daemon isn't configured to accept
> publickey as a valid authorization mean.
>
> Try to do ssh -v localhost and check if there's something similar to the
> following:
>
> debug1: Authentications that can continue:
> publickey,password,keyboard-interactive
> debug1: Next authentication method: publickey
> debug1: Trying private key: /home/xxx/.ssh/identity
> debug1: Trying private key: /home/xxx/.ssh/id_rsa
> debug1: Offering public key: /home/xxx/.ssh/id_dsa
> debug1: Server accepts key: pkalg ssh-dss blen 435
> debug1: read PEM private key done: type DSA
> debug1: Authentication succeeded (publickey).
>
> Cos
>
>
> On 7/1/09 10:11 PM, fred wang wrote:
>
>> I have setup ./.ssh/authorized keys has permssion 600, but it didn't work.
>> Thanks anyway
>>
>> ls -l .ssh/authorized_keys
>> -rw------- 1 xxx xxx 1222 2009-07-02 13:08 .ssh/authorized_keys
>>
>> On Thu, Jul 2, 2009 at 12:15 AM, Konstantin Boudnik<cos@yahoo-inc.com
>> >wrote:
>>
>> Make sure that your ~/.ssh/authorized_keys has permissions 600
>>>
>>> Cos
>>>
>>>
>>> On 7/1/09 7:35 AM, fred wang wrote:
>>>
>>> Hi all,
>>>>
>>>>   I failed to setup passphraseless ssh(I mean, I still need to input
>>>> password to do ssh localhost) when I tried to configure Hadoop to run on
>>>> psuedo-distributed operation,  could anyone help me solve this issue?
>>>> Thanks!
>>>>
>>>> (1)I use the Putty0.6 to remote access to Ubuntu by SSH.
>>>>
>>>> (2) execution steps and ouput
>>>>
>>>> $ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
>>>> Generating public/private dsa key pair.
>>>> Your identification has been saved in /home/xxx/.ssh/id_dsa.
>>>> Your public key has been saved in /home/xxx/.ssh/id_dsa.pub.
>>>> The key fingerprint is:
>>>> a9:39:4c:9b:22:f9:a4:77:70:24:fa:bf:12:f5:81:81 xxx
>>>>
>>>>
>>>> **note: it doesn't have message  'Enter passphrase (empty for no
>>>> passphrase):
>>>>     Enter same passphrase again: ' which appear in some introductory
>>>> paper.
>>>> "
>>>>
>>>> $ cat ~/.ssh/id_dsa.pub>>   ~/.ssh/authorized_keys
>>>> no output
>>>>
>>>> $ ssh localhost
>>>> The authenticity of host 'localhost (127.0.0.1)' can't be established.
>>>> RSA key fingerprint is 4f:a1:ff:ed:0c:46:3e:a9:8c:97:bc:b7:46:3e:35:d2.
>>>> Are you sure you want to continue connecting (yes/no)? yes
>>>> Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
>>>> xxx@localhost's password:
>>>>
>>>>

--001485e8d216e8e9c2046db2a032--

From general-return-304-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 06:09:59 2009
Return-Path: <general-return-304-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98107 invoked from network); 2 Jul 2009 06:09:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 06:09:58 -0000
Received: (qmail 36396 invoked by uid 500); 2 Jul 2009 06:10:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36314 invoked by uid 500); 2 Jul 2009 06:10:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36304 invoked by uid 99); 2 Jul 2009 06:10:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 06:10:08 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of fredwang222@gmail.com designates 209.85.212.172 as permitted sender)
Received: from [209.85.212.172] (HELO mail-vw0-f172.google.com) (209.85.212.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 06:09:58 +0000
Received: by vwj2 with SMTP id 2so854680vwj.5
        for <general@hadoop.apache.org>; Wed, 01 Jul 2009 23:09:37 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=aJterOkRMMmKPRay8bi65WyiOhbEjd4dz2nNl3583XA=;
        b=x0hEiDcygINbH+KYuzKCCdPbxsBrNLQz7aRz25Z/aa5BA0C5uOHdkg6/i4q3PXhHzz
         /TPJOXHZBegss5RR4G9Kd2qe07qSuXzB0gG47zjPq22wjVut+sGk1DEwPNHbdS3R6C9l
         6MLKmvoVcdNBEFmU3ygJVbH0su+FrXPaDIN9M=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=CSovY1BHB3A+U8xwd8XsM0HHaPkI46wPysL40nwRpf4mfhjOc52+nRE1AhrkiGIP/t
         m9rLaHmOY5Jq8Hczt68dlq0qD64gx5yCyz85YiOdYmNHesS7R3Djkhi9UDfAZw8R3By1
         tb98mADQk7UG9+Dcu3h/nuvKKDUpeBZOqn3pk=
MIME-Version: 1.0
Received: by 10.220.72.79 with SMTP id l15mr8439952vcj.4.1246514977022; Wed, 
	01 Jul 2009 23:09:37 -0700 (PDT)
In-Reply-To: <702261350907012251s41b18b24m5dc54015007d494e@mail.gmail.com>
References: <702261350907010735u3502197au800281a5a131d963@mail.gmail.com>
	 <4A4B8B99.10303@yahoo-inc.com>
	 <702261350907012211r17dec368u17e18052383a8822@mail.gmail.com>
	 <4A4C430F.7080306@yahoo-inc.com>
	 <702261350907012251s41b18b24m5dc54015007d494e@mail.gmail.com>
Date: Thu, 2 Jul 2009 14:09:36 +0800
Message-ID: <702261350907012309l4c62885bk6df2ebac8e2b7254@mail.gmail.com>
Subject: Re: fail to setup passphraseless ssh and need some help
From: fred wang <fredwang222@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64755aa69abfd046db2e2bd
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e64755aa69abfd046db2e2bd
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

sorry, should incopy ssh_config(instead of sshd_config)


vi /etc/ssh/ssh_config

#  1. command line options

#  2. user-specific file

#  3. system-wide file

# Any configuration value is only changed the first time it is set.

# Thus, host-specific definitions should be at the beginning of the

# configuration file, and defaults at the end.



# Site-wide defaults for some commonly used options.  For a comprehensive

# list of available options, their meanings and defaults, please see the

# ssh_config(5) man page.



Host *

#   ForwardAgent no

#   ForwardX11 no

#   ForwardX11Trusted yes

#   RhostsRSAAuthentication no

#   RSAAuthentication yes

#   PasswordAuthentication yes

#   HostbasedAuthentication no

#   GSSAPIAuthentication no

#   GSSAPIDelegateCredentials no

#   GSSAPIKeyExchange no

#   GSSAPITrustDNS no

#   BatchMode no

#   CheckHostIP yes

#   AddressFamily any

#   ConnectTimeout 0

#   StrictHostKeyChecking ask

#   IdentityFile ~/.ssh/identity

#   IdentityFile ~/.ssh/id_rsa

#   IdentityFile ~/.ssh/id_dsa

#   Port 22

#   Protocol 2,1

#   Cipher 3des

#   Ciphers
aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc

#   MACs hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160

#   EscapeChar ~

#   Tunnel no

#   TunnelDevice any:any

#   PermitLocalCommand no

    SendEnv LANG LC_*

    HashKnownHosts yes

    GSSAPIAuthentication yes

    GSSAPIDelegateCredentials no


On Thu, Jul 2, 2009 at 1:51 PM, fred wang <fredwang222@gmail.com> wrote:

> Here is the output of ssh -v localhost  and the configuration of
> ssh_config,
>
> xxx@xxx-desktop:~$ ssh -v localhost
>
> OpenSSH_4.7p1 Debian-8ubuntu1.2, OpenSSL 0.9.8g 19 Oct 2007
>
> debug1: Reading configuration data /etc/ssh/ssh_config
>
> debug1: Applying options for *
>
> debug1: Connecting to localhost [127.0.0.1] port 22.
>
> debug1: Connection established.
>
> debug1: identity file /home/xxx/.ssh/identity type -1
>
> debug1: identity file /home/xxx/.ssh/id_rsa type -1
>
> debug1: identity file /home/xxx/.ssh/id_dsa type 2
>
> debug1: Remote protocol version 2.0, remote software version OpenSSH_4.7p1
> Debian-8ubuntu1.2
>
> debug1: match: OpenSSH_4.7p1 Debian-8ubuntu1.2 pat OpenSSH*
>
> debug1: Enabling compatibility mode for protocol 2.0
>
> debug1: Local version string SSH-2.0-OpenSSH_4.7p1 Debian-8ubuntu1.2
>
> debug1: SSH2_MSG_KEXINIT sent
>
> debug1: SSH2_MSG_KEXINIT received
>
> debug1: kex: server->client aes128-cbc hmac-md5 none
>
> debug1: kex: client->server aes128-cbc hmac-md5 none
>
> debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
>
> debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
>
> debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
>
> debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
>
> debug1: Host 'localhost' is known and matches the RSA host key.
>
> debug1: Found key in /home/xxx/.ssh/known_hosts:1
>
> debug1: ssh_rsa_verify: signature correct
>
> debug1: SSH2_MSG_NEWKEYS sent
>
> debug1: expecting SSH2_MSG_NEWKEYS
>
> debug1: SSH2_MSG_NEWKEYS received
>
> debug1: SSH2_MSG_SERVICE_REQUEST sent
>
> debug1: SSH2_MSG_SERVICE_ACCEPT received
>
> debug1: Authentications that can continue: publickey,password
>
> debug1: Next authentication method: publickey
>
> debug1: Trying private key: /home/xxx/.ssh/identity
>
> debug1: Trying private key: /home/xxx/.ssh/id_rsa
>
> debug1: Offering public key: /home/xxx/.ssh/id_dsa
>
> debug1: Authentications that can continue: publickey,password
>
> debug1: Next authentication method: password
>
> xxx@localhost's password:
>
>
>
>
>
>
>
> xxx@xxx:~$ vi /etc/ssh/sshd_config
>
> #KerberosOrLocalPasswd yes
>
> #KerberosTicketCleanup yes
>
>
>
> # GSSAPI options
>
> #GSSAPIAuthentication no
>
> #GSSAPICleanupCredentials yes
>
>
>
> X11Forwarding yes
>
> X11DisplayOffset 10
>
> PrintMotd no
>
> PrintLastLog yes
>
> TCPKeepAlive yes
>
> #UseLogin no
>
>
>
> #MaxStartups 10:30:60
>
> #Banner /etc/issue.net
>
>
>
> # Allow client to pass locale environment variables
>
> AcceptEnv LANG LC_*
>
>
>
> Subsystem sftp /usr/lib/openssh/sftp-server
>
>
>
> UsePAM yes
>
>
>
> On Thu, Jul 2, 2009 at 1:18 PM, Konstantin Boudnik <cos@yahoo-inc.com>wrote:
>
>> Yet another possibility is that your SSH daemon isn't configured to accept
>> publickey as a valid authorization mean.
>>
>> Try to do ssh -v localhost and check if there's something similar to the
>> following:
>>
>> debug1: Authentications that can continue:
>> publickey,password,keyboard-interactive
>> debug1: Next authentication method: publickey
>> debug1: Trying private key: /home/xxx/.ssh/identity
>> debug1: Trying private key: /home/xxx/.ssh/id_rsa
>> debug1: Offering public key: /home/xxx/.ssh/id_dsa
>> debug1: Server accepts key: pkalg ssh-dss blen 435
>> debug1: read PEM private key done: type DSA
>> debug1: Authentication succeeded (publickey).
>>
>> Cos
>>
>>
>> On 7/1/09 10:11 PM, fred wang wrote:
>>
>>> I have setup ./.ssh/authorized keys has permssion 600, but it didn't
>>> work.
>>> Thanks anyway
>>>
>>> ls -l .ssh/authorized_keys
>>> -rw------- 1 xxx xxx 1222 2009-07-02 13:08 .ssh/authorized_keys
>>>
>>> On Thu, Jul 2, 2009 at 12:15 AM, Konstantin Boudnik<cos@yahoo-inc.com
>>> >wrote:
>>>
>>> Make sure that your ~/.ssh/authorized_keys has permissions 600
>>>>
>>>> Cos
>>>>
>>>>
>>>> On 7/1/09 7:35 AM, fred wang wrote:
>>>>
>>>> Hi all,
>>>>>
>>>>>   I failed to setup passphraseless ssh(I mean, I still need to input
>>>>> password to do ssh localhost) when I tried to configure Hadoop to run
>>>>> on
>>>>> psuedo-distributed operation,  could anyone help me solve this issue?
>>>>> Thanks!
>>>>>
>>>>> (1)I use the Putty0.6 to remote access to Ubuntu by SSH.
>>>>>
>>>>> (2) execution steps and ouput
>>>>>
>>>>> $ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
>>>>> Generating public/private dsa key pair.
>>>>> Your identification has been saved in /home/xxx/.ssh/id_dsa.
>>>>> Your public key has been saved in /home/xxx/.ssh/id_dsa.pub.
>>>>> The key fingerprint is:
>>>>> a9:39:4c:9b:22:f9:a4:77:70:24:fa:bf:12:f5:81:81 xxx
>>>>>
>>>>>
>>>>> **note: it doesn't have message  'Enter passphrase (empty for no
>>>>> passphrase):
>>>>>     Enter same passphrase again: ' which appear in some introductory
>>>>> paper.
>>>>> "
>>>>>
>>>>> $ cat ~/.ssh/id_dsa.pub>>   ~/.ssh/authorized_keys
>>>>> no output
>>>>>
>>>>> $ ssh localhost
>>>>> The authenticity of host 'localhost (127.0.0.1)' can't be established.
>>>>> RSA key fingerprint is 4f:a1:ff:ed:0c:46:3e:a9:8c:97:bc:b7:46:3e:35:d2.
>>>>> Are you sure you want to continue connecting (yes/no)? yes
>>>>> Warning: Permanently added 'localhost' (RSA) to the list of known
>>>>> hosts.
>>>>> xxx@localhost's password:
>>>>>
>>>>>
>

--0016e64755aa69abfd046db2e2bd--

From general-return-305-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 06:24:14 2009
Return-Path: <general-return-305-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12818 invoked from network); 2 Jul 2009 06:24:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 06:24:14 -0000
Received: (qmail 51231 invoked by uid 500); 2 Jul 2009 06:24:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51152 invoked by uid 500); 2 Jul 2009 06:24:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51136 invoked by uid 99); 2 Jul 2009 06:24:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 06:24:23 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [72.14.220.155] (HELO fg-out-1718.google.com) (72.14.220.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 06:24:14 +0000
Received: by fg-out-1718.google.com with SMTP id l26so1228882fgb.12
        for <general@hadoop.apache.org>; Wed, 01 Jul 2009 23:23:54 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.86.53.11 with SMTP id b11mr3136292fga.12.1246515834070; Wed, 
	01 Jul 2009 23:23:54 -0700 (PDT)
In-Reply-To: <fa561940907011902j7035d55fo3a21d7d3c11262a9@mail.gmail.com>
References: <fa561940907011902j7035d55fo3a21d7d3c11262a9@mail.gmail.com>
From: Philip Zeyliger <philip@cloudera.com>
Date: Wed, 1 Jul 2009 23:23:34 -0700
Message-ID: <15da8a100907012323w49fe44f8rfb45fe43a7deac47@mail.gmail.com>
Subject: Re: Mailing lists are broken?
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd251347f284f046db315a6
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd251347f284f046db315a6
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

That page is currently out of date.  I believe the correct lists are

{common,hdfs,mapred}-{user,dev,issues}@hadoop.apache.org<issues%7D@hadoop.apache.org>

Yes, that's 9 lists.

There's also general@, and lists for the subprojects (Avro, Hive, Pig,
HBase, Zookeeper, ...)

-- Philip

On Wed, Jul 1, 2009 at 7:02 PM, Bogdan M. Maryniuk <
bogdan.maryniuk@gmail.com> wrote:

> Hello.
> Tried http://hadoop.apache.org/core/mailing_lists.html
> Address "core-user-subscribe@hadoop.apache.org" returns error.
>
> Is that's OK?
>
> --
> Kind regards, BM
>
> Things, that are stupid at the beginning, rarely ends up wisely.
>

--000e0cd251347f284f046db315a6--

From general-return-306-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 07:04:28 2009
Return-Path: <general-return-306-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24455 invoked from network); 2 Jul 2009 07:04:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 07:04:27 -0000
Received: (qmail 95056 invoked by uid 500); 2 Jul 2009 07:04:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95010 invoked by uid 500); 2 Jul 2009 07:04:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95000 invoked by uid 99); 2 Jul 2009 07:04:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 07:04:37 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of xine.jar@googlemail.com designates 209.85.210.185 as permitted sender)
Received: from [209.85.210.185] (HELO mail-yx0-f185.google.com) (209.85.210.185)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 07:04:26 +0000
Received: by yxe15 with SMTP id 15so2092976yxe.5
        for <general@hadoop.apache.org>; Thu, 02 Jul 2009 00:04:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=6xI3XVykTTZDSslk4QchAcxsED5m0wPvnB1Uv7T48nY=;
        b=O3yRVa5YrB8QaMjOL54zERP5AuUPHuiCcwO5v4RKq9wNPcMp22I8WG55yZbedrmMfe
         zbHKLCVo+dmf0S9bWOvPTaTnlZnXX/lMsHP9GF6WdpwSIinZQnF/s2WV1SpY1CH6cKCZ
         fjkOXgqxNuJDQeXakgurqEvnQ0gL4BW7T9AUw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=rcW9lnJokTd85PLs5oNbruUP9QNRb3Ibf4QtLspQ0lKqDd5YzzaVTiTy0caZy6ejbS
         cmD7tsKbWYYWLaLwuRFvgRIa5viH6DXwZrEwP5gp8HNFHa6CiRgIpyoVMiJveE2EPSrM
         FfrzmYFdukch/fOZm9Lt6444K1DJStlWrjQHI=
MIME-Version: 1.0
Received: by 10.231.11.136 with SMTP id t8mr3153643ibt.50.1246518244857; Thu, 
	02 Jul 2009 00:04:04 -0700 (PDT)
Date: Thu, 2 Jul 2009 09:04:04 +0200
Message-ID: <aa41f20e0907020004w6482ad49t64dbef5bc206c18a@mail.gmail.com>
Subject: problem in sending you an email
From: C J <xine.jar@googlemail.com>
To: general@hadoop.apache.org, C J <xine.jar@googlemail.com>
Content-Type: multipart/alternative; boundary=000325574c1630d7a1046db3a594
X-Virus-Checked: Checked by ClamAV on apache.org

--000325574c1630d7a1046db3a594
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hallo guys,
since yesterday I am trying to send you an email on
core-user@hadoop.apache.org
and I always got an error. Can someone give me valid email address?

thank you,
Christine

--000325574c1630d7a1046db3a594--

From general-return-307-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 08:11:44 2009
Return-Path: <general-return-307-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46057 invoked from network); 2 Jul 2009 08:11:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 08:11:44 -0000
Received: (qmail 85832 invoked by uid 500); 2 Jul 2009 08:11:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85773 invoked by uid 500); 2 Jul 2009 08:11:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85763 invoked by uid 99); 2 Jul 2009 08:11:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 08:11:53 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of bogdan.maryniuk@gmail.com designates 209.85.216.171 as permitted sender)
Received: from [209.85.216.171] (HELO mail-px0-f171.google.com) (209.85.216.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 08:11:44 +0000
Received: by pxi1 with SMTP id 1so1297078pxi.5
        for <general@hadoop.apache.org>; Thu, 02 Jul 2009 01:11:23 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=hom68bU/HbatsLM5CO4TtIMKQdeeJ1omf0ATp1TdsMI=;
        b=DIf9ChapjCfSS7zhc9mgvUiCmTwKSSPb7YV3zhxsrMxoKj5WxgTeADBtWXABhETkix
         j1fQ+RDub0u5VIlHFiPC5dPFnsmlBESzGn4WyUVkvtgrifBaPqZJxGCELBMDRuSbyEIA
         nVafiqBvBChRDowwxnnAFBuXBfIND2rcNiTPE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=K39nBmfepgJG1EmtK3m9BbvNgnzLOOxN8BUtvORgJkqBJQ2Us0PzzKIxP3D2Q61LgB
         AMxk6rPdle7M/pONPgBaP26g2kClwN4erzXbFLZpUpei1UP0fRlyLrUcXRTIOqBhv2U1
         pJOLIVce3PMT9Q6h8r9AQPShfQD8Pi9e+Ni68=
MIME-Version: 1.0
Received: by 10.142.212.21 with SMTP id k21mr1443687wfg.109.1246522283567; 
	Thu, 02 Jul 2009 01:11:23 -0700 (PDT)
In-Reply-To: <15da8a100907012323w49fe44f8rfb45fe43a7deac47@mail.gmail.com>
References: <fa561940907011902j7035d55fo3a21d7d3c11262a9@mail.gmail.com>
	 <15da8a100907012323w49fe44f8rfb45fe43a7deac47@mail.gmail.com>
Date: Thu, 2 Jul 2009 17:11:23 +0900
Message-ID: <fa561940907020111v7068335dl85382b002e5f9e25@mail.gmail.com>
Subject: Re: Mailing lists are broken?
From: "Bogdan M. Maryniuk" <bogdan.maryniuk@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Thu, Jul 2, 2009 at 3:23 PM, Philip Zeyliger<philip@cloudera.com> wrote:
> That page is currently out of date. =C2=A0I believe the correct lists are
> {common,hdfs,mapred}-{user,dev,issues}@hadoop.apache.org
>
> Yes, that's 9 lists.
>
> There's also general@, and lists for the subprojects (Avro, Hive, Pig,
> HBase, Zookeeper, ...)

Then in this case there must be two updates...:
1. Anyone please update mailing list page to let stop newcomers
cross-post stuff around.
2. Wiki: I can confirm 0.20 works fine on OpenSolaris zones in a true
cluster. :)

...and one question from newbie (me):
By default Hadoop stores stuff in /tmp/.... directory that is "a base
for other temporary directories.", however among that "other
temporary" stuff I see my actual fsimage and VERSION file. Are they
temporary? See, usually /tmp is just 1GB space on a separated
partition and is volatile (e.g. system or node reboot simply wipes it
out). Does that mean that Hadoop moves actual data blocks elsewhere or
actually stores all the stuff actually in /tmp directory?

For some reason, if I set like:

<property>
  <name>hadoop.tmp.dir</name>
  <value>/export/storage/hadoop-data</value>
</property>

...where I have lots of space per a node, somewhat Hadoop says
capacity 0Kb. :-) When I put anything, it crashes with "File
10mb-random.dat could only be replicated to 0 nodes, instead of 1".

09/07/02 16:33:17 WARN hdfs.DFSClient: Error Recovery for block null
bad datanode[0] nodes =3D=3D null
09/07/02 16:33:17 WARN hdfs.DFSClient: Could not get block locations.
Source file "/10mb-random.dat" - Aborting...

After I configure it back to "/tmp/anywhere-I-want", everything OK.
Why it is like this? And how to make sure my data is not in a
temporary place?

--=20
Kind regards, BM

Things, that are stupid at the beginning, rarely ends up wisely.

From general-return-308-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 16:24:34 2009
Return-Path: <general-return-308-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84705 invoked from network); 2 Jul 2009 16:24:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 16:24:34 -0000
Received: (qmail 92458 invoked by uid 500); 2 Jul 2009 16:24:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92385 invoked by uid 500); 2 Jul 2009 16:24:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 92375 invoked by uid 99); 2 Jul 2009 16:24:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 16:24:44 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 16:24:31 +0000
Received: from [192.168.102.128] (snvvpn1-10-73-152-c233.hq.corp.yahoo.com [10.73.152.233])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n62GNiEa086934
	for <general@hadoop.apache.org>; Thu, 2 Jul 2009 09:23:45 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=y2ikerlSS44/2DWnuH4UbrdWRKJy3oT6Hl8/mLc0+Nu+N/Pv+I3J4qNcYDzBpiiU
Message-ID: <4A4CDF0F.2040106@yahoo-inc.com>
Date: Thu, 02 Jul 2009 09:23:43 -0700
From: Konstantin Boudnik <cos@yahoo-inc.com>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b3pre) Gecko/20090223 Thunderbird/3.0b2
MIME-Version: 1.0
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: fail to setup passphraseless ssh and need some help
References: <702261350907010735u3502197au800281a5a131d963@mail.gmail.com>	 <4A4B8B99.10303@yahoo-inc.com>	 <702261350907012211r17dec368u17e18052383a8822@mail.gmail.com>	 <4A4C430F.7080306@yahoo-inc.com>	 <702261350907012251s41b18b24m5dc54015007d494e@mail.gmail.com> <702261350907012309l4c62885bk6df2ebac8e2b7254@mail.gmail.com>
In-Reply-To: <702261350907012309l4c62885bk6df2ebac8e2b7254@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hmm, publickey seems to be allowed. Well, as the last resort, I'd suggest to 
make sure that your public key and the one in the ~/.ssh/authorized_keys are 
actually identical.

On 7/1/09 11:09 PM, fred wang wrote:
> sorry, should incopy ssh_config(instead of sshd_config)
>
>
> vi /etc/ssh/ssh_config
>
> #  1. command line options
>
> #  2. user-specific file
>
> #  3. system-wide file
>
> # Any configuration value is only changed the first time it is set.
>
> # Thus, host-specific definitions should be at the beginning of the
>
> # configuration file, and defaults at the end.
>
>
>
> # Site-wide defaults for some commonly used options.  For a comprehensive
>
> # list of available options, their meanings and defaults, please see the
>
> # ssh_config(5) man page.
>
>
>
> Host *
>
> #   ForwardAgent no
>
> #   ForwardX11 no
>
> #   ForwardX11Trusted yes
>
> #   RhostsRSAAuthentication no
>
> #   RSAAuthentication yes
>
> #   PasswordAuthentication yes
>
> #   HostbasedAuthentication no
>
> #   GSSAPIAuthentication no
>
> #   GSSAPIDelegateCredentials no
>
> #   GSSAPIKeyExchange no
>
> #   GSSAPITrustDNS no
>
> #   BatchMode no
>
> #   CheckHostIP yes
>
> #   AddressFamily any
>
> #   ConnectTimeout 0
>
> #   StrictHostKeyChecking ask
>
> #   IdentityFile ~/.ssh/identity
>
> #   IdentityFile ~/.ssh/id_rsa
>
> #   IdentityFile ~/.ssh/id_dsa
>
> #   Port 22
>
> #   Protocol 2,1
>
> #   Cipher 3des
>
> #   Ciphers
> aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc
>
> #   MACs hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160
>
> #   EscapeChar ~
>
> #   Tunnel no
>
> #   TunnelDevice any:any
>
> #   PermitLocalCommand no
>
>      SendEnv LANG LC_*
>
>      HashKnownHosts yes
>
>      GSSAPIAuthentication yes
>
>      GSSAPIDelegateCredentials no
>
>
> On Thu, Jul 2, 2009 at 1:51 PM, fred wang<fredwang222@gmail.com>  wrote:
>
>> Here is the output of ssh -v localhost  and the configuration of
>> ssh_config,
>>
>> xxx@xxx-desktop:~$ ssh -v localhost
>>
>> OpenSSH_4.7p1 Debian-8ubuntu1.2, OpenSSL 0.9.8g 19 Oct 2007
>>
>> debug1: Reading configuration data /etc/ssh/ssh_config
>>
>> debug1: Applying options for *
>>
>> debug1: Connecting to localhost [127.0.0.1] port 22.
>>
>> debug1: Connection established.
>>
>> debug1: identity file /home/xxx/.ssh/identity type -1
>>
>> debug1: identity file /home/xxx/.ssh/id_rsa type -1
>>
>> debug1: identity file /home/xxx/.ssh/id_dsa type 2
>>
>> debug1: Remote protocol version 2.0, remote software version OpenSSH_4.7p1
>> Debian-8ubuntu1.2
>>
>> debug1: match: OpenSSH_4.7p1 Debian-8ubuntu1.2 pat OpenSSH*
>>
>> debug1: Enabling compatibility mode for protocol 2.0
>>
>> debug1: Local version string SSH-2.0-OpenSSH_4.7p1 Debian-8ubuntu1.2
>>
>> debug1: SSH2_MSG_KEXINIT sent
>>
>> debug1: SSH2_MSG_KEXINIT received
>>
>> debug1: kex: server->client aes128-cbc hmac-md5 none
>>
>> debug1: kex: client->server aes128-cbc hmac-md5 none
>>
>> debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
>>
>> debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
>>
>> debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
>>
>> debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
>>
>> debug1: Host 'localhost' is known and matches the RSA host key.
>>
>> debug1: Found key in /home/xxx/.ssh/known_hosts:1
>>
>> debug1: ssh_rsa_verify: signature correct
>>
>> debug1: SSH2_MSG_NEWKEYS sent
>>
>> debug1: expecting SSH2_MSG_NEWKEYS
>>
>> debug1: SSH2_MSG_NEWKEYS received
>>
>> debug1: SSH2_MSG_SERVICE_REQUEST sent
>>
>> debug1: SSH2_MSG_SERVICE_ACCEPT received
>>
>> debug1: Authentications that can continue: publickey,password
>>
>> debug1: Next authentication method: publickey
>>
>> debug1: Trying private key: /home/xxx/.ssh/identity
>>
>> debug1: Trying private key: /home/xxx/.ssh/id_rsa
>>
>> debug1: Offering public key: /home/xxx/.ssh/id_dsa
>>
>> debug1: Authentications that can continue: publickey,password
>>
>> debug1: Next authentication method: password
>>
>> xxx@localhost's password:
>>
>>
>>
>>
>>
>>
>>
>> xxx@xxx:~$ vi /etc/ssh/sshd_config
>>
>> #KerberosOrLocalPasswd yes
>>
>> #KerberosTicketCleanup yes
>>
>>
>>
>> # GSSAPI options
>>
>> #GSSAPIAuthentication no
>>
>> #GSSAPICleanupCredentials yes
>>
>>
>>
>> X11Forwarding yes
>>
>> X11DisplayOffset 10
>>
>> PrintMotd no
>>
>> PrintLastLog yes
>>
>> TCPKeepAlive yes
>>
>> #UseLogin no
>>
>>
>>
>> #MaxStartups 10:30:60
>>
>> #Banner /etc/issue.net
>>
>>
>>
>> # Allow client to pass locale environment variables
>>
>> AcceptEnv LANG LC_*
>>
>>
>>
>> Subsystem sftp /usr/lib/openssh/sftp-server
>>
>>
>>
>> UsePAM yes
>>
>>
>>
>> On Thu, Jul 2, 2009 at 1:18 PM, Konstantin Boudnik<cos@yahoo-inc.com>wrote:
>>
>>> Yet another possibility is that your SSH daemon isn't configured to accept
>>> publickey as a valid authorization mean.
>>>
>>> Try to do ssh -v localhost and check if there's something similar to the
>>> following:
>>>
>>> debug1: Authentications that can continue:
>>> publickey,password,keyboard-interactive
>>> debug1: Next authentication method: publickey
>>> debug1: Trying private key: /home/xxx/.ssh/identity
>>> debug1: Trying private key: /home/xxx/.ssh/id_rsa
>>> debug1: Offering public key: /home/xxx/.ssh/id_dsa
>>> debug1: Server accepts key: pkalg ssh-dss blen 435
>>> debug1: read PEM private key done: type DSA
>>> debug1: Authentication succeeded (publickey).
>>>
>>> Cos
>>>
>>>
>>> On 7/1/09 10:11 PM, fred wang wrote:
>>>
>>>> I have setup ./.ssh/authorized keys has permssion 600, but it didn't
>>>> work.
>>>> Thanks anyway
>>>>
>>>> ls -l .ssh/authorized_keys
>>>> -rw------- 1 xxx xxx 1222 2009-07-02 13:08 .ssh/authorized_keys
>>>>
>>>> On Thu, Jul 2, 2009 at 12:15 AM, Konstantin Boudnik<cos@yahoo-inc.com
>>>>> wrote:
>>>> Make sure that your ~/.ssh/authorized_keys has permissions 600
>>>>> Cos
>>>>>
>>>>>
>>>>> On 7/1/09 7:35 AM, fred wang wrote:
>>>>>
>>>>> Hi all,
>>>>>>    I failed to setup passphraseless ssh(I mean, I still need to input
>>>>>> password to do ssh localhost) when I tried to configure Hadoop to run
>>>>>> on
>>>>>> psuedo-distributed operation,  could anyone help me solve this issue?
>>>>>> Thanks!
>>>>>>
>>>>>> (1)I use the Putty0.6 to remote access to Ubuntu by SSH.
>>>>>>
>>>>>> (2) execution steps and ouput
>>>>>>
>>>>>> $ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
>>>>>> Generating public/private dsa key pair.
>>>>>> Your identification has been saved in /home/xxx/.ssh/id_dsa.
>>>>>> Your public key has been saved in /home/xxx/.ssh/id_dsa.pub.
>>>>>> The key fingerprint is:
>>>>>> a9:39:4c:9b:22:f9:a4:77:70:24:fa:bf:12:f5:81:81 xxx
>>>>>>
>>>>>>
>>>>>> **note: it doesn't have message  'Enter passphrase (empty for no
>>>>>> passphrase):
>>>>>>      Enter same passphrase again: ' which appear in some introductory
>>>>>> paper.
>>>>>> "
>>>>>>
>>>>>> $ cat ~/.ssh/id_dsa.pub>>    ~/.ssh/authorized_keys
>>>>>> no output
>>>>>>
>>>>>> $ ssh localhost
>>>>>> The authenticity of host 'localhost (127.0.0.1)' can't be established.
>>>>>> RSA key fingerprint is 4f:a1:ff:ed:0c:46:3e:a9:8c:97:bc:b7:46:3e:35:d2.
>>>>>> Are you sure you want to continue connecting (yes/no)? yes
>>>>>> Warning: Permanently added 'localhost' (RSA) to the list of known
>>>>>> hosts.
>>>>>> xxx@localhost's password:
>>>>>>
>>>>>>

From general-return-309-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 16:58:32 2009
Return-Path: <general-return-309-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6804 invoked from network); 2 Jul 2009 16:58:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 16:58:32 -0000
Received: (qmail 66177 invoked by uid 500); 2 Jul 2009 16:58:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66137 invoked by uid 500); 2 Jul 2009 16:58:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66127 invoked by uid 99); 2 Jul 2009 16:58:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 16:58:41 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cutting@gmail.com designates 209.85.200.168 as permitted sender)
Received: from [209.85.200.168] (HELO wf-out-1314.google.com) (209.85.200.168)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 16:58:31 +0000
Received: by wf-out-1314.google.com with SMTP id 23so668965wfg.2
        for <general@hadoop.apache.org>; Thu, 02 Jul 2009 09:58:10 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:sender:message-id:date:from
         :reply-to:user-agent:mime-version:to:subject:content-type
         :content-transfer-encoding;
        bh=PSFJkmzkrozMqWlRhYjR/9ZH1XaOdHhryZjDowcKwHE=;
        b=YYdIcaq3Axao4BOr9FAqKJzinr7k3QeFjJK2jFM2DXIM5PEVGOdQdv4xiQWYoD9GFx
         sCILNVKe35G1iEq7bC7UK//TLsOkp9n8yRGehm5A6S8p835L//Z0Ic+ZNF3nMIzpzEeT
         TRKXIMl3uEonoNWlSYqfRKGeRz+0NE30U7FvU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:date:from:reply-to:user-agent:mime-version:to
         :subject:content-type:content-transfer-encoding;
        b=tXTODeDeRi4LzrfxK+I9AIgV3qg9O1933sX3TWJu5vMaV8jA8A+Qg3HXKNJems+CGQ
         NL7vdOGuPpMuQ/hjVKmzao9FQnmuTsk9k5/EraQnpGO8iCb+wmh5j6XVGNz9q/mrEcxQ
         QMar5/littyuhybR4n1seXZ02QuQWFBiibt9Y=
Received: by 10.142.173.6 with SMTP id v6mr160124wfe.40.1246553890159;
        Thu, 02 Jul 2009 09:58:10 -0700 (PDT)
Received: from ?192.168.168.16? (c-76-103-176-123.hsd1.ca.comcast.net [76.103.176.123])
        by mx.google.com with ESMTPS id 30sm7244955wfd.3.2009.07.02.09.58.08
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Thu, 02 Jul 2009 09:58:09 -0700 (PDT)
Sender: Doug Cutting <cutting@gmail.com>
Message-ID: <4A4CE71F.7000302@apache.org>
Date: Thu, 02 Jul 2009 09:58:07 -0700
From: Doug Cutting <cutting@apache.org>
Reply-To: avro-dev@hadoop.apache.org
User-Agent: Thunderbird 2.0.0.22 (X11/20090608)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: [Fwd: [VOTE] Avro release 1.0.0 (candidate 0)]
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Can folks please download and test this Avro release candidate?

Votes should be cast on avro-dev@.

Thanks,

Doug

-------- Original Message --------
Subject: [VOTE] Avro release 1.0.0 (candidate 0)
Date: Wed, 01 Jul 2009 15:11:22 -0700
From: Doug Cutting <cutting@apache.org>
To: avro-dev@hadoop.apache.org

I have created a candidate build for Avro release 1.0.0.  This would be
the first public release of Avro.  We hope not to change Avro's
specification incompatibly until release 2.0.0, so its adequacy is more
important than bugs or features for this release.

Please download, test and vote by 7 July.

http://people.apache.org/~cutting/avro-1.0.0-candidate-0/

Thanks,

Doug


From general-return-310-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 17:06:48 2009
Return-Path: <general-return-310-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9371 invoked from network); 2 Jul 2009 17:06:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 17:06:48 -0000
Received: (qmail 73872 invoked by uid 500); 2 Jul 2009 17:06:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73747 invoked by uid 500); 2 Jul 2009 17:06:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73723 invoked by uid 99); 2 Jul 2009 17:06:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 17:06:57 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [207.126.228.150] (HELO rsmtp2.corp.yahoo.com) (207.126.228.150)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 17:06:49 +0000
Received: from [172.21.148.64] (wlanvpn-mc2e-246-64.corp.yahoo.com [172.21.148.64])
	(authenticated bits=0)
	by rsmtp2.corp.yahoo.com (8.13.8/8.13.8/y.rout) with ESMTP id n62H6ShC039809
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Thu, 2 Jul 2009 10:06:28 -0700 (PDT)
Message-ID: <4A4CE913.4000900@apache.org>
Date: Thu, 02 Jul 2009 10:06:27 -0700
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.22 (X11/20090608)
MIME-Version: 1.0
To: general@hadoop.apache.org, private@hadoop.apache.org
Subject: [Fwd: [VOTE] Release ZooKeeper 3.2.0 (candidate 1)]
Content-Type: multipart/mixed;
 boundary="------------010808050307060802090503"
X-Virus-Checked: Checked by ClamAV on apache.org

--------------010808050307060802090503
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hadoop PMC,

Please test and vote on this release in zookeeper-dev list.

Thanks,

Patrick

--------------010808050307060802090503
Content-Type: message/rfc822;
 name="[VOTE] Release ZooKeeper 3.2.0 (candidate 1).eml"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename="[VOTE] Release ZooKeeper 3.2.0 (candidate 1).eml"

Delivered-To: phunt1@gmail.com
Received: by 10.100.6.1 with SMTP id 1cs224329anf;
        Wed, 1 Jul 2009 10:40:39 -0700 (PDT)
Received: by 10.141.27.7 with SMTP id e7mr926673rvj.257.1246470038602;
        Wed, 01 Jul 2009 10:40:38 -0700 (PDT)
Return-Path: <phunt@apache.org>
Received: from minotaur.apache.org (minotaur.apache.org [140.211.11.9])
        by mx.google.com with SMTP id 41si340542pzk.125.2009.07.01.10.40.38;
        Wed, 01 Jul 2009 10:40:38 -0700 (PDT)
Received-SPF: pass (google.com: best guess record for domain of phunt@apache.org designates 140.211.11.9 as permitted sender) client-ip=140.211.11.9;
Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of phunt@apache.org designates 140.211.11.9 as permitted sender) smtp.mail=phunt@apache.org
Received: (qmail 52978 invoked by uid 2923); 1 Jul 2009 17:40:27 -0000
Delivered-To: phunt@minotaur.apache.org
Received: (qmail 52973 invoked from network); 1 Jul 2009 17:40:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 17:40:27 -0000
Received: (qmail 45246 invoked by uid 500); 1 Jul 2009 17:40:37 -0000
Delivered-To: apmail-phunt@apache.org
Received: (qmail 45241 invoked by uid 99); 1 Jul 2009 17:40:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 17:40:37 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [207.126.228.150] (HELO rsmtp2.corp.yahoo.com) (207.126.228.150)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 17:40:28 +0000
Received: from [172.21.148.187] (wlanvpn-mc2e-246-187.corp.yahoo.com [172.21.148.187])
	(authenticated bits=0)
	by rsmtp2.corp.yahoo.com (8.13.8/8.13.8/y.rout) with ESMTP id n61HdsJr048210
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Wed, 1 Jul 2009 10:39:54 -0700 (PDT)
Message-ID: <4A4B9F69.4040401@apache.org>
Date: Wed, 01 Jul 2009 10:39:53 -0700
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.22 (X11/20090608)
MIME-Version: 1.0
To: "zookeeper-dev@hadoop.apache.org" <zookeeper-dev@hadoop.apache.org>
Subject: [VOTE] Release ZooKeeper 3.2.0 (candidate 1)
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

I have created a candidate build for ZooKeeper 3.2.0. Over 120 JIRAs are 
addressed in this release, with major new features for; perl/python/rest 
bindings, flexible quorum support, recipe code libraries, chroot support 
in connect string, and much more.

*** Please download, test and VOTE before the
*** vote closes EOD on Monday, July 6.***

http://people.apache.org/~phunt/zookeeper-3.2.0-candidate-1/

Should we release this?

Patrick




--------------010808050307060802090503--

From general-return-311-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 17:22:20 2009
Return-Path: <general-return-311-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17924 invoked from network); 2 Jul 2009 17:22:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 17:22:20 -0000
Received: (qmail 92881 invoked by uid 500); 2 Jul 2009 17:22:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92801 invoked by uid 500); 2 Jul 2009 17:22:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 92791 invoked by uid 99); 2 Jul 2009 17:22:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 17:22:30 +0000
X-ASF-Spam-Status: No, hits=-8.0 required=10.0
	tests=RCVD_IN_DNSWL_HI,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of Jim.Kellerman@microsoft.com designates 131.107.115.215 as permitted sender)
Received: from [131.107.115.215] (HELO smtp.microsoft.com) (131.107.115.215)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 17:22:20 +0000
Received: from TK5EX14MLTC104.redmond.corp.microsoft.com (157.54.79.159) by
 TK5-EXGWY-E802.partners.extranet.microsoft.com (10.251.56.168) with Microsoft
 SMTP Server (TLS) id 8.2.99.4; Thu, 2 Jul 2009 10:21:59 -0700
Received: from TK5EX14MBXC118.redmond.corp.microsoft.com ([169.254.9.116]) by
 TK5EX14MLTC104.redmond.corp.microsoft.com ([157.54.79.159]) with mapi; Thu, 2
 Jul 2009 10:21:58 -0700
From: "Jim Kellerman (POWERSET)" <Jim.Kellerman@microsoft.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: RE: Mailing lists are broken?
Thread-Topic: Mailing lists are broken?
Thread-Index: AQHJ+rlILs2d7A2/8EGuHI/1+O8rypBiOyYAgABBnqA=
Date: Thu, 2 Jul 2009 17:21:34 +0000
Message-ID: <E9C6F5B134FDA3468879CB4E533CF1C804FFD9@TK5EX14MBXC118.redmond.corp.microsoft.com>
References: <fa561940907011902j7035d55fo3a21d7d3c11262a9@mail.gmail.com>
 <15da8a100907012323w49fe44f8rfb45fe43a7deac47@mail.gmail.com>
In-Reply-To: <15da8a100907012323w49fe44f8rfb45fe43a7deac47@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

I tried to subscribe to the mapred-dev list and got:

> Hi. This is the qmail-send program at minotaur.apache.org.
> I'm afraid I wasn't able to deliver your message to the following address=
es.
> This is a permanent error; I've given up. Sorry it didn't work out.
>
> <mapred-dev-subscribe@hadoop.apache.org>:
> 192.87.106.230 does not like recipient.
> Remote host said: 550 mail to mapred-dev-subscribe@hadoop.apache.org not =
accepted here Giving up on 192.87.106.230.
>
> --- Below this line is a copy of the message.
>
> Return-Path: <jimk@apache.org>
> Received: (qmail 9141 invoked by uid 2660); 2 Jul 2009 17:06:00 -0000
> Date: 2 Jul 2009 17:06:00 -0000
> Message-ID: <20090702170600.9138.qmail@minotaur.apache.org>
> From: jimk@apache.org
> To: mapred-dev-subscribe@hadoop.apache.org

I got a similar failure when trying to subscribe to mapred-issues


From general-return-312-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 17:25:09 2009
Return-Path: <general-return-312-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22794 invoked from network); 2 Jul 2009 17:25:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 17:25:09 -0000
Received: (qmail 98967 invoked by uid 500); 2 Jul 2009 17:25:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98909 invoked by uid 500); 2 Jul 2009 17:25:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98899 invoked by uid 99); 2 Jul 2009 17:25:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 17:25:19 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of owen.omalley@gmail.com designates 209.85.198.236 as permitted sender)
Received: from [209.85.198.236] (HELO rv-out-0506.google.com) (209.85.198.236)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 17:25:09 +0000
Received: by rv-out-0506.google.com with SMTP id k40so608617rvb.29
        for <general@hadoop.apache.org>; Thu, 02 Jul 2009 10:24:49 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:from:to
         :in-reply-to:content-type:content-transfer-encoding:x-mailer
         :mime-version:subject:date:references;
        bh=7jg6gXzL8FG58brafvxc0Z+yNJ33HOVjkt0b3C4hOi8=;
        b=AMmznkz1aNylt2A/yDPf+s47HgJz9y8NrM4qjKAo58gQKMu5r3YRJMSxDFiwfX9Dwd
         Z/vq9JDVRKJJl/QaIV/r5EoyQlMW7vDCsDcFNRCJZfeluUuRBK+Cjswbv8d/XlMWlb8Y
         pn9WPg5FAI/L8LFsK0/35XnpTXatTlpi1kQcc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:from:to:in-reply-to:content-type
         :content-transfer-encoding:x-mailer:mime-version:subject:date
         :references;
        b=Iz+bSfIl+bFOssFC8BfbXGaq6spIC+K8pWTLaSpsxexmL7SQJa+/Zfv0dR045YzzTD
         QaqhfY6+kiZzckb8i6iXprrErbU4oY+KSjsxUvQSU4RvJjBDn+FSDPhrtCFFgb2rTwvM
         JrtADhAcAMvyitrWFgCmkil9TLn719KaUW9m8=
Received: by 10.141.1.19 with SMTP id d19mr393665rvi.145.1246555489271;
        Thu, 02 Jul 2009 10:24:49 -0700 (PDT)
Received: from ?172.21.202.57? (lo5.cfw-b-gci.corp.yahoo.com [209.131.62.144])
        by mx.google.com with ESMTPS id k37sm12013597rvb.28.2009.07.02.10.24.47
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Thu, 02 Jul 2009 10:24:48 -0700 (PDT)
Message-Id: <1357BE36-A3A7-4131-B721-8D9EEEEFA31B@gmail.com>
From: Owen O'Malley <owen.omalley@gmail.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <E9C6F5B134FDA3468879CB4E533CF1C804FFD9@TK5EX14MBXC118.redmond.corp.microsoft.com>
Content-Type: text/plain;
	charset=us-ascii;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit
X-Mailer: iPhone Mail (7A341)
Mime-Version: 1.0 (iPhone Mail 7A341)
Subject: Re: Mailing lists are broken?
Date: Thu, 2 Jul 2009 10:24:03 -0700
References: <fa561940907011902j7035d55fo3a21d7d3c11262a9@mail.gmail.com> <15da8a100907012323w49fe44f8rfb45fe43a7deac47@mail.gmail.com> <E9C6F5B134FDA3468879CB4E533CF1C804FFD9@TK5EX14MBXC118.redmond.corp.microsoft.com>
X-Virus-Checked: Checked by ClamAV on apache.org

It is mapreduce-dev...

-- Owen

On Jul 2, 2009, at 10:21, "Jim Kellerman (POWERSET)" <Jim.Kellerman@microsoft.com 
 > wrote:

> I tried to subscribe to the mapred-dev list and got:
>
>> Hi. This is the qmail-send program at minotaur.apache.org.
>> I'm afraid I wasn't able to deliver your message to the following  
>> addresses.
>> This is a permanent error; I've given up. Sorry it didn't work out.
>>
>> <mapred-dev-subscribe@hadoop.apache.org>:
>> 192.87.106.230 does not like recipient.
>> Remote host said: 550 mail to mapred-dev- 
>> subscribe@hadoop.apache.org not accepted here Giving up on 192.87.106.230 
>> .
>>
>> --- Below this line is a copy of the message.
>>
>> Return-Path: <jimk@apache.org>
>> Received: (qmail 9141 invoked by uid 2660); 2 Jul 2009 17:06:00 -0000
>> Date: 2 Jul 2009 17:06:00 -0000
>> Message-ID: <20090702170600.9138.qmail@minotaur.apache.org>
>> From: jimk@apache.org
>> To: mapred-dev-subscribe@hadoop.apache.org
>
> I got a similar failure when trying to subscribe to mapred-issues
>

From general-return-313-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 20:47:37 2009
Return-Path: <general-return-313-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35291 invoked from network); 2 Jul 2009 20:47:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 20:47:37 -0000
Received: (qmail 54349 invoked by uid 500); 2 Jul 2009 20:47:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54305 invoked by uid 500); 2 Jul 2009 20:47:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54295 invoked by uid 99); 2 Jul 2009 20:47:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 20:47:46 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 20:47:34 +0000
Received: from pineapple-lm.corp.yahoo.com (pineapple-lm.corp.yahoo.com [10.72.122.24])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n62Kjuuw091054
	for <general@hadoop.apache.org>; Thu, 2 Jul 2009 13:45:56 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=SeytO+eSfR3OvJnLI8Asckqzc8rM7OJaKjGvhAoB5mfImNZu4WHMLol+jxGRx7Eq
Message-Id: <82F6C99C-17FD-4B38-A397-461043998B65@yahoo-inc.com>
From: Chris Douglas <chrisdo@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <E9C6F5B134FDA3468879CB4E533CF1C804FFD9@TK5EX14MBXC118.redmond.corp.microsoft.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v935.3)
Subject: Re: Mailing lists are broken?
Date: Thu, 2 Jul 2009 13:45:56 -0700
References: <fa561940907011902j7035d55fo3a21d7d3c11262a9@mail.gmail.com> <15da8a100907012323w49fe44f8rfb45fe43a7deac47@mail.gmail.com> <E9C6F5B134FDA3468879CB4E533CF1C804FFD9@TK5EX14MBXC118.redmond.corp.microsoft.com>
X-Mailer: Apple Mail (2.935.3)
X-Virus-Checked: Checked by ClamAV on apache.org

It's mapreduce-dev@ -C

On Jul 2, 2009, at 10:21 AM, Jim Kellerman (POWERSET) wrote:

> I tried to subscribe to the mapred-dev list and got:
>
>> Hi. This is the qmail-send program at minotaur.apache.org.
>> I'm afraid I wasn't able to deliver your message to the following  
>> addresses.
>> This is a permanent error; I've given up. Sorry it didn't work out.
>>
>> <mapred-dev-subscribe@hadoop.apache.org>:
>> 192.87.106.230 does not like recipient.
>> Remote host said: 550 mail to mapred-dev- 
>> subscribe@hadoop.apache.org not accepted here Giving up on  
>> 192.87.106.230.
>>
>> --- Below this line is a copy of the message.
>>
>> Return-Path: <jimk@apache.org>
>> Received: (qmail 9141 invoked by uid 2660); 2 Jul 2009 17:06:00 -0000
>> Date: 2 Jul 2009 17:06:00 -0000
>> Message-ID: <20090702170600.9138.qmail@minotaur.apache.org>
>> From: jimk@apache.org
>> To: mapred-dev-subscribe@hadoop.apache.org
>
> I got a similar failure when trying to subscribe to mapred-issues
>


From general-return-314-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 21:04:51 2009
Return-Path: <general-return-314-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 41053 invoked from network); 2 Jul 2009 21:04:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 21:04:51 -0000
Received: (qmail 80219 invoked by uid 500); 2 Jul 2009 21:04:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80166 invoked by uid 500); 2 Jul 2009 21:04:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80100 invoked by uid 99); 2 Jul 2009 21:04:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 21:04:59 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [72.14.220.154] (HELO fg-out-1718.google.com) (72.14.220.154)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 21:04:49 +0000
Received: by fg-out-1718.google.com with SMTP id 13so673693fge.12
        for <general@hadoop.apache.org>; Thu, 02 Jul 2009 14:04:29 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.86.74.4 with SMTP id w4mr674196fga.65.1246568669613; Thu, 02 
	Jul 2009 14:04:29 -0700 (PDT)
In-Reply-To: <15da8a100907012323w49fe44f8rfb45fe43a7deac47@mail.gmail.com>
References: <fa561940907011902j7035d55fo3a21d7d3c11262a9@mail.gmail.com> 
	<15da8a100907012323w49fe44f8rfb45fe43a7deac47@mail.gmail.com>
From: Philip Zeyliger <philip@cloudera.com>
Date: Thu, 2 Jul 2009 14:04:09 -0700
Message-ID: <15da8a100907021404g31f764f2x6e7445fc0a9db640@mail.gmail.com>
Subject: Re: Mailing lists are broken?
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

As others pointed out, it's actually "mapreduce", and I also left off
the -commits.  So, it's really:

  {common,hdfs,mapreduce}-{user,dev,issues,commits}@hadoop.apache.org

-- Philip

From general-return-315-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 23:37:51 2009
Return-Path: <general-return-315-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 3401 invoked from network); 2 Jul 2009 23:37:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 23:37:51 -0000
Received: (qmail 22549 invoked by uid 500); 2 Jul 2009 23:38:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22484 invoked by uid 500); 2 Jul 2009 23:38:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22474 invoked by uid 99); 2 Jul 2009 23:38:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 23:38:00 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cutting@gmail.com designates 209.85.200.172 as permitted sender)
Received: from [209.85.200.172] (HELO wf-out-1314.google.com) (209.85.200.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 23:37:50 +0000
Received: by wf-out-1314.google.com with SMTP id 23so743384wfg.2
        for <general@hadoop.apache.org>; Thu, 02 Jul 2009 16:37:29 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:sender:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=QXoX9cPhYo0Yg8vzrAqHh1oHdFroOoIMkA2+LJ/0MMg=;
        b=iqdbemu5Z1/xsdSl2XHnInZC8DrsC8hr01CGH+0gaP2xmkCOggXEa5nc5/A9EPa7Hx
         2bZ1eCrkr6EVMQ18i3v1JuTStZRFnc9EbbGnz+kn/XtY1Pyp/OV7ZcxDXK2FhVDQ9gr5
         FSj01c/GB0F169067vEi81mNjzgmfamW1/Dxw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:date:from:user-agent:mime-version:to:subject
         :references:in-reply-to:content-type:content-transfer-encoding;
        b=r8PT8b+wR5TtnI/FqUtduWeIUB8UJ2/6UqH/Ui72f2jYcA4vIfCsnSOKqlpNQ/3wr4
         zl9z3/v1m4DG1PJrdjRZYBHjqx/731s0/VmnlncvpjL/dpDI4CyRlCH65BM1WAHO3m8d
         7BUsl64sO3ghSezpvXdO6biHBZsR5rhNCy5uY=
Received: by 10.142.231.7 with SMTP id d7mr270126wfh.99.1246577849034;
        Thu, 02 Jul 2009 16:37:29 -0700 (PDT)
Received: from ?192.168.168.16? (c-76-103-176-123.hsd1.ca.comcast.net [76.103.176.123])
        by mx.google.com with ESMTPS id 28sm8410197wfd.4.2009.07.02.16.37.27
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Thu, 02 Jul 2009 16:37:27 -0700 (PDT)
Sender: Doug Cutting <cutting@gmail.com>
Message-ID: <4A4D44B5.2060602@apache.org>
Date: Thu, 02 Jul 2009 16:37:25 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.22 (X11/20090608)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: jira notifications on common, hdfs, and mapreduce
References: <C1F6B4F2-B91D-49E5-81E1-4F7FBF83BBA6@yahoo-inc.com> <4A4BB103.30906@apache.org> <361DBFD2-DF1C-4719-9F35-B94D974ECEC1@yahoo-inc.com> <4A4BB771.60605@apache.org>
In-Reply-To: <4A4BB771.60605@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Doug Cutting wrote:
> Arun C Murthy wrote:
>> -1 from me.
> 
> You subscribe to both lists and really prefer receiving create/resolve 
> messages twice?

Arun?  I'm trying to decide whether I must add a filter to automatically 
delete the second copy of these messages, or whether we can eliminate 
the second copy.  Do you really subscribe to both lists and prefer 
receiving both copies?

Doug

From general-return-316-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 04:44:46 2009
Return-Path: <general-return-316-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4399 invoked from network); 3 Jul 2009 04:44:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 04:44:46 -0000
Received: (qmail 91964 invoked by uid 500); 3 Jul 2009 04:44:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91875 invoked by uid 500); 3 Jul 2009 04:44:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91861 invoked by uid 99); 3 Jul 2009 04:44:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 04:44:55 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of qiaomuf@gmail.com designates 209.85.221.188 as permitted sender)
Received: from [209.85.221.188] (HELO mail-qy0-f188.google.com) (209.85.221.188)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 04:44:46 +0000
Received: by qyk26 with SMTP id 26so920615qyk.5
        for <general@hadoop.apache.org>; Thu, 02 Jul 2009 21:44:25 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:from:date:message-id
         :subject:to:content-type;
        bh=osRZg9mtIGUOWFXwFpqQbd7SCKINkTAP1tkzetr2sMk=;
        b=KM/aJNrzvTQhKUYHdi0R5IVIjLp6MABP34aWBqVxBPL3gIA4M3d6zYTEqh3rdrf/h3
         J5KEV8TNfTTg5Hh1SSnyGGIeHauUZsX+150IxZ5dznPiTAz1oOYcqDdcfg20Vj7J0Nzz
         D4oQ1SeC4Aq6qXta57lL0yrvqhQefUQV9Zci4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type;
        b=uUFc5ZrYiPVRJcwkX+Xv6ttrffzKzCB9VBntT+uNfdql1EFIhUA+ptJ9ONmo89Ayev
         UDMawKa3jgjEF1O6Wa/eFgK3QinCwkOGBzUbzF/d50CwpFWF5ZWBxca+8qwjTbyem9jr
         f60Zu71GFH5S0d0HM4XkUGLQlY/NQ+nltJCIs=
MIME-Version: 1.0
Received: by 10.229.85.132 with SMTP id o4mr860020qcl.0.1246596265118; Thu, 02 
	Jul 2009 21:44:25 -0700 (PDT)
From: =?UTF-8?B?5LmU5pyo?= <qiaomuf@gmail.com>
Date: Thu, 2 Jul 2009 21:44:05 -0700
Message-ID: <396ceda90907022144p3b66a345g1b3b52a1ac5b7969@mail.gmail.com>
Subject: Combiner Problem
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364edc328f8e3a046dc5cf22
X-Virus-Checked: Checked by ClamAV on apache.org

--0016364edc328f8e3a046dc5cf22
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi, everyone

I've been learning hadoop recently and I'm confused about the combiner
mechanism.

There is a property min.num.spills.for.combine specifying the minimum numbe=
r
of spills to run combiner when merging. The default value is 3. Why there i=
s
such a restriction? Should it be better that run the combiner no matter how
many spills there are?

The second question is why the combiner could be run at the reduce side.
Can't the reduce function take place of that?

Thanks very much.
--=20
Best wishes,
=E4=B9=94=E6=9C=A8
MOE KLINNS Lab and SKLMS Lab, Xi'an Jiaotong University
Department of Computer Science and Technology, Xi=E2=80=99an Jiaotong Unive=
rsity
TEL: 15991676983
E-mail: qiaomuf@gmail.com

--0016364edc328f8e3a046dc5cf22--

From general-return-317-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 04:50:57 2009
Return-Path: <general-return-317-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5502 invoked from network); 3 Jul 2009 04:50:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 04:50:57 -0000
Received: (qmail 94006 invoked by uid 500); 3 Jul 2009 04:51:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93914 invoked by uid 500); 3 Jul 2009 04:51:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93903 invoked by uid 99); 3 Jul 2009 04:51:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 04:51:07 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cutting@gmail.com designates 209.85.221.188 as permitted sender)
Received: from [209.85.221.188] (HELO mail-qy0-f188.google.com) (209.85.221.188)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 04:50:57 +0000
Received: by qyk26 with SMTP id 26so923038qyk.5
        for <general@hadoop.apache.org>; Thu, 02 Jul 2009 21:50:37 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        bh=YKNPw7wVQJghKygMBxUv1z/GiienyaCu0K9g1WuIE0k=;
        b=FmQPDNd3f5xgyv0C4m09znVKpSIVgAGp9E9rqJPAMjNbwLS1UouptGQMMhgiCHqU+T
         /9CmCdFiLa18wufQLjkKSyCKvF7tYL/KQbt04tbI9U1cw3DZTv8qqcVGrkNNM/yC7M9R
         YvDKPaEqJrQ6yonYW2gItVid/Q6Oj782LBus4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=lqKnnnNESO2PK/avwZcA+xT02uNa9f8mE7gAtJDiWxf3L1uJ/xpJrEsSSBArigTKF9
         uKgX0GlKOhW16k+yLdDOIeqJMvBSWAmc1j+4hmTEKQiCmQybVKciR4aNbf06TqBy3aNr
         Ds7dZ6gUkh2Ur75IfZRObkAHfR3mCW2SGGjfg=
MIME-Version: 1.0
Sender: cutting@gmail.com
Received: by 10.229.74.212 with SMTP id v20mr848020qcj.16.1246596636985; Thu, 
	02 Jul 2009 21:50:36 -0700 (PDT)
In-Reply-To: <1c9416210907022147u3ef8e942see605339a9f618a6@mail.gmail.com>
References: <1c9416210907022147u3ef8e942see605339a9f618a6@mail.gmail.com>
Date: Thu, 2 Jul 2009 21:50:36 -0700
X-Google-Sender-Auth: d09e00f32b5fecb7
Message-ID: <1c9416210907022150g2d42f186q7d3f7db49826b984@mail.gmail.com>
Subject: Fwd: [VOTE] Avro release 1.0.0 (candidate 1)
From: Doug Cutting <cutting@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Can folks please download and test this new Avro release candidate?

Votes should be cast on avro-dev@.

Thanks,

Doug

---------- Forwarded message ----------
From: Doug Cutting <cutting@apache.org>
Date: Thu, Jul 2, 2009 at 9:47 PM
Subject: [VOTE] Avro release 1.0.0 (candidate 1)
To: avro-dev@hadoop.apache.org

I have created a second candidate build for Avro release 1.0.0.
Mostly this adds missing license headers and notices.

Please download, test and vote by 7 July.

http://people.apache.org/~cutting/avro-1.0.0-candidate-1/

Thanks!

Doug

From general-return-318-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 08:41:09 2009
Return-Path: <general-return-318-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85440 invoked from network); 3 Jul 2009 08:41:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 08:41:09 -0000
Received: (qmail 88115 invoked by uid 500); 3 Jul 2009 08:41:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88068 invoked by uid 500); 3 Jul 2009 08:41:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88002 invoked by uid 99); 3 Jul 2009 08:41:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 08:41:14 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of xine.jar@googlemail.com designates 209.85.210.185 as permitted sender)
Received: from [209.85.210.185] (HELO mail-yx0-f185.google.com) (209.85.210.185)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 08:41:04 +0000
Received: by yxe15 with SMTP id 15so946212yxe.5
        for <multiple recipients>; Fri, 03 Jul 2009 01:40:43 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=qDTZpi+iMceEyyT78cQ/AO+5WQ0KWnb7xwHOrP+w1Dw=;
        b=ohu9mUn6quRBX296adKL/fEYyiydznVgiCGeTMJc8Ya/DkIfQrJB7ebCK9auuII0OV
         vONbkOH0hjOoIPxsFPmY+xZrxkf7+eSc90l9Z5tiy0mjvYM+uE2ZHjX3fxEItV2LqAIv
         d0k1rmz1vUsSeHr+33HjPkiqrNUkXX0V8ynHE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=ZudZjAi2PJx+o6BNXzX2FZZ3yW5mh975HAk28M5kN3WHnANI6jvV1LMg2F1EjWfJr1
         a4qJ4FRpFeQTcufBa/+wqNGfxj51yByrQ1AW0ZLTq/5usjkLddFXQQ/GG3VrluypysKq
         mkLddbuR1SJU/NQzC6zU0A2+Hgi9iw8EV4COU=
MIME-Version: 1.0
Received: by 10.231.14.67 with SMTP id f3mr1008553iba.36.1246610443457; Fri, 
	03 Jul 2009 01:40:43 -0700 (PDT)
Date: Fri, 3 Jul 2009 10:40:43 +0200
Message-ID: <aa41f20e0907030140q1cf763ctdcc19cff8a8b7f91@mail.gmail.com>
Subject: problem in sending you an email
From: C J <xine.jar@googlemail.com>
To: common-user@hadoop.apache.org, C J <xine.jar@googlemail.com>, 
	general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00221532cc80a7d3ee046dc91c32
X-Virus-Checked: Checked by ClamAV on apache.org

--00221532cc80a7d3ee046dc91c32
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

hallo, everytime I try sending you and email explaining my problem
in Haddop the email does not reach you and I get the following error

*Technical details of permanent failure:
Google tried to deliver your message, but it was rejected by the recipient
domain.
We recommend contacting the other email provider for further information
about the cause of this error.
The error that the other server returned was: 552 552 spam score (23.5)
exceeded threshold (state 18).

*I hope at least you receive this email and tell me the reason of the error,
is it the length of the email? format?

Thank you for your help

--00221532cc80a7d3ee046dc91c32--

From general-return-319-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 08:46:57 2009
Return-Path: <general-return-319-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87113 invoked from network); 3 Jul 2009 08:46:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 08:46:56 -0000
Received: (qmail 99257 invoked by uid 500); 3 Jul 2009 08:47:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99189 invoked by uid 500); 3 Jul 2009 08:47:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 99178 invoked by uid 99); 3 Jul 2009 08:47:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 08:47:06 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of timrobertson100@gmail.com designates 209.85.219.219 as permitted sender)
Received: from [209.85.219.219] (HELO mail-ew0-f219.google.com) (209.85.219.219)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 08:46:58 +0000
Received: by ewy19 with SMTP id 19so2634938ewy.29
        for <general@hadoop.apache.org>; Fri, 03 Jul 2009 01:46:37 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=913X1WhOV6Jekn51lZImo4PLxjXnb/s6eLku+SbVrCw=;
        b=FdHy2CInW9Wrz7/fSsR5WstFTgWQnAlO3PAkpoRSXYtSN0KEPVGHOCWwSB8iRbEKWW
         n/HON/Ppq/6+nIphN2NVAzH1HF7ccBeCXLnY5c8cHkslfqvtq0OSfJ2JRrFU5xfpjrfE
         xAai03RkK3xh2yC1/v5KBCXNuJsFz8j87gS3Y=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=H7W4A4eNxijZjFa8tkpNFG1zWlmlsj8FPinr/LqxWFivnTa7phkqwrXM3PL/cNdZNA
         bhGoLHYwxFKabgL4LDe/lwGTgRsRkuheyV81Lls9ffg4iJV7v5UYiz4j3/oNQbDR/0Ne
         Pf6QUhE2A8xVOOUyRbY8zYSebLDXsYUTApkzI=
MIME-Version: 1.0
Received: by 10.216.46.79 with SMTP id q57mr296005web.62.1246610797487; Fri, 
	03 Jul 2009 01:46:37 -0700 (PDT)
In-Reply-To: <aa41f20e0907030140q1cf763ctdcc19cff8a8b7f91@mail.gmail.com>
References: <aa41f20e0907030140q1cf763ctdcc19cff8a8b7f91@mail.gmail.com>
Date: Fri, 3 Jul 2009 10:46:37 +0200
Message-ID: <32120a6a0907030146p3ec08b55re05c1cc5cfa58978@mail.gmail.com>
Subject: Re: problem in sending you an email
From: tim robertson <timrobertson100@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hi CJ,

Your recent emails are getting through, but I don't have your original
question anymore.  When I post a question to the list, I don't see my
own question in GMail until someone answers it - presumably the
listserv does not send you back your own question on first post.

Perhaps you could post your question again and hopefully someone will
answer it.  I suggest creating a new subject so the conversation
becomes a new thread.

Cheers,

Tim

On Fri, Jul 3, 2009 at 10:40 AM, C J<xine.jar@googlemail.com> wrote:
> hallo, everytime I try sending you and email explaining my problem
> in Haddop the email does not reach you and I get the following error
>
> *Technical details of permanent failure:
> Google tried to deliver your message, but it was rejected by the recipient
> domain.
> We recommend contacting the other email provider for further information
> about the cause of this error.
> The error that the other server returned was: 552 552 spam score (23.5)
> exceeded threshold (state 18).
>
> *I hope at least you receive this email and tell me the reason of the error,
> is it the length of the email? format?
>
> Thank you for your help
>

From general-return-320-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 09:04:17 2009
Return-Path: <general-return-320-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91808 invoked from network); 3 Jul 2009 09:04:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 09:04:15 -0000
Received: (qmail 18582 invoked by uid 500); 3 Jul 2009 09:04:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18549 invoked by uid 500); 3 Jul 2009 09:04:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18539 invoked by uid 99); 3 Jul 2009 09:04:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 09:04:25 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of xine.jar@googlemail.com designates 209.85.132.244 as permitted sender)
Received: from [209.85.132.244] (HELO an-out-0708.google.com) (209.85.132.244)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 09:04:14 +0000
Received: by an-out-0708.google.com with SMTP id c38so1017990ana.29
        for <general@hadoop.apache.org>; Fri, 03 Jul 2009 02:03:53 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=/ssLxJO7fK+2z3xqHV43Lri5tI7AnZtv0KVF5IsKpM4=;
        b=BrG9liwwmD43NU1E0QdVvy09KEnuEBRdc6ipFWoRYj+gHr6s3qj6ND7z/0yBf0t2xA
         pYNnws8i6QOKtU+6wjhoBO8zauSJ/VBPjLOvIi0Xf0qGzkNdTjrkLNfov1ll/WLNs6Jp
         kfHuHy+V7Y9zdUTrvCIOWSmMH5Mz+mPbPWlZk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=X6fH57ycRfSmEGlCM8XQqH37X/lSVqyU7CycuinlrS15AD37u3qza8eZL3TL+yjplN
         5amLM0Ipibs838Uu7CFwk/uSvPrAOVt0gkyF1b6gYkaWwj16wqgcwHfeEyG05OPpMNGi
         h0AIiZHZz21UbJbcYvHGTb+9eLMidQf2zfsXU=
MIME-Version: 1.0
Received: by 10.231.14.138 with SMTP id g10mr1029082iba.10.1246611832978; Fri, 
	03 Jul 2009 02:03:52 -0700 (PDT)
In-Reply-To: <32120a6a0907030146p3ec08b55re05c1cc5cfa58978@mail.gmail.com>
References: <aa41f20e0907030140q1cf763ctdcc19cff8a8b7f91@mail.gmail.com>
	 <32120a6a0907030146p3ec08b55re05c1cc5cfa58978@mail.gmail.com>
Date: Fri, 3 Jul 2009 11:03:52 +0200
Message-ID: <aa41f20e0907030203o6b9fc927tbf10a349aa64f83e@mail.gmail.com>
Subject: Re: problem in sending you an email
From: C J <xine.jar@googlemail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00221532cbc47a3e3e046dc96f5d
X-Virus-Checked: Checked by ClamAV on apache.org

--00221532cbc47a3e3e046dc96f5d
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Ok thank you,
I will do right now

On Fri, Jul 3, 2009 at 10:46 AM, tim robertson <timrobertson100@gmail.com>wrote:

> Hi CJ,
>
> Your recent emails are getting through, but I don't have your original
> question anymore.  When I post a question to the list, I don't see my
> own question in GMail until someone answers it - presumably the
> listserv does not send you back your own question on first post.
>
> Perhaps you could post your question again and hopefully someone will
> answer it.  I suggest creating a new subject so the conversation
> becomes a new thread.
>
> Cheers,
>
> Tim
>
> On Fri, Jul 3, 2009 at 10:40 AM, C J<xine.jar@googlemail.com> wrote:
> > hallo, everytime I try sending you and email explaining my problem
> > in Haddop the email does not reach you and I get the following error
> >
> > *Technical details of permanent failure:
> > Google tried to deliver your message, but it was rejected by the
> recipient
> > domain.
> > We recommend contacting the other email provider for further information
> > about the cause of this error.
> > The error that the other server returned was: 552 552 spam score (23.5)
> > exceeded threshold (state 18).
> >
> > *I hope at least you receive this email and tell me the reason of the
> error,
> > is it the length of the email? format?
> >
> > Thank you for your help
> >
>

--00221532cbc47a3e3e046dc96f5d--

From general-return-321-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 09:06:56 2009
Return-Path: <general-return-321-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 92525 invoked from network); 3 Jul 2009 09:06:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 09:06:55 -0000
Received: (qmail 25448 invoked by uid 500); 3 Jul 2009 09:07:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25377 invoked by uid 500); 3 Jul 2009 09:07:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25367 invoked by uid 99); 3 Jul 2009 09:07:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 09:07:05 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of timrobertson100@gmail.com designates 209.85.219.219 as permitted sender)
Received: from [209.85.219.219] (HELO mail-ew0-f219.google.com) (209.85.219.219)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 09:06:57 +0000
Received: by ewy19 with SMTP id 19so2645537ewy.29
        for <general@hadoop.apache.org>; Fri, 03 Jul 2009 02:06:36 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type:content-transfer-encoding;
        bh=0NcHNzU5wwm9CykGgjJQw1UyvE2nXON5rlxaAlhgltA=;
        b=l4AHJ2AH4PBIh/0wuOpaqs2ClHp/rK+c2dkjlKS4MaTmC5wrYKKX/EFiwBnDUaVei1
         QWSMbdpx7BXd67A+bjWbqhOOg6LhMkSdlqE1YHmovtyAz7t/uHIzEj8LgV+a+WxawLm4
         0SlO8HG+/Uwmuzv0zW1m+3ZgO8/jwc2OTcPd4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=IBqxVu9ClVa8Nn8wllMHnDeFy5KbJWxLwuO2CaHSNlMtG6VNkPu61EiHR4DVevQh4v
         +OktwLLbOtC8tbt8hjMCgDf9rMISjHmZ5JdizZtVoLOCg3vqSGr9vWKZzaFTvTrxqUzR
         7RF9scOewR9NHsPArqOjQjQxkWzZQAP+W560M=
MIME-Version: 1.0
Received: by 10.216.11.137 with SMTP id 9mr292890wex.180.1246611996410; Fri, 
	03 Jul 2009 02:06:36 -0700 (PDT)
Date: Fri, 3 Jul 2009 11:06:36 +0200
Message-ID: <32120a6a0907030206l77a30d41yef2704e176f6d90f@mail.gmail.com>
Subject: Tools to help administer a cluster?
From: tim robertson <timrobertson100@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hi all,

Can someone please point me at an open source tool that can help with
management of a *nix cluster?
I am looking for something that allows easy ability to copy over a new
Hadoop distribution and sync all the config files etc.  I'm not very
good at system admin, so hoping there is are some tools to help speed
things up.

I am running on a small cluster using Mac OSX right now, but will want
to use the same on a Linux distribution with 16 nodes shortly.

Cheers,

Tim

From general-return-322-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 09:21:43 2009
Return-Path: <general-return-322-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97209 invoked from network); 3 Jul 2009 09:21:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 09:21:43 -0000
Received: (qmail 44901 invoked by uid 500); 3 Jul 2009 09:21:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44820 invoked by uid 500); 3 Jul 2009 09:21:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44802 invoked by uid 99); 3 Jul 2009 09:21:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 09:21:50 +0000
X-ASF-Spam-Status: No, hits=3.7 required=10.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of xine.jar@googlemail.com designates 209.85.132.240 as permitted sender)
Received: from [209.85.132.240] (HELO an-out-0708.google.com) (209.85.132.240)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 09:21:39 +0000
Received: by an-out-0708.google.com with SMTP id c38so1020478ana.29
        for <multiple recipients>; Fri, 03 Jul 2009 02:21:18 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=O6mZ+tL3AfASDUJ8nnpT4aIZhn62Tumj+LFv1A7arI4=;
        b=KzKSMs5BjzAWcntiDdHfUjzNpPIkcCmFdVeyq26H4u6AOiZ1pHeOVOfg2hysModoa0
         793/rbiEBh3WfT6yCZF+GEn6+6BqycL40ZVT3D2AcuF6Sc1fgoe9xcT3z/LKcEHHCnMp
         3YXIb2gs+pTL8cb7cYl3UCMaxuCGmGUPUrgOw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=jlSZOd9K0hfHcTvU1LxlXxXdmPzWnQhv0fyEeRDisrVSZl8pVzSwJhkNciN4AyR86E
         Z3FVvz1S0+5LvcRm3lmTUr2IF/p/MXsXJqN6njzBVkHa3/dRexJimoQmoLoyb7XXi/rX
         Xipspu5aWjTdxYbpopkYnlW7gAa7Qi2eOb2gs=
MIME-Version: 1.0
Received: by 10.231.32.11 with SMTP id a11mr1027993ibd.38.1246612875897; Fri, 
	03 Jul 2009 02:21:15 -0700 (PDT)
Date: Fri, 3 Jul 2009 11:21:15 +0200
Message-ID: <aa41f20e0907030221k6ea5b2ffk3a5aade3b382a9d7@mail.gmail.com>
Subject: My secondary namenode seem not be running, and may be the reason of 
	my problem!!!
From: C J <xine.jar@googlemail.com>
To: common-user@hadoop.apache.org, C J <xine.jar@googlemail.com>, 
	general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0022152d6d75a3ede0046dc9ad60
X-Virus-Checked: Checked by ClamAV on apache.org

--0022152d6d75a3ede0046dc9ad60
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hallo everyone,
I have installed the hadoop 0.18.3 on three linux machines, I am trying to
run the
example of WordCountv1.0 on a cluster. But I guess I have a problem
somewhere.
*
Problem*

*After formating the name node:*
I am getting several STARTUP_MSG and at the end a "SHUTDOWN_MSG: shutting
down the namenode..."
is this normal?

*Afterwards I try starting the dfs:**
*I get a message "starting namenode..."* *afterwards I get another message
"starting secondary namenode"
at this stage the shell is blocked unless I press the enter key. Then the
system tries to start another secondary
namenode and the shell is then not blocked. What is going on?
*
Then I proceed an try starting the mapred:*
I get the two messages "starting jobtracker....." and "starting tasktracker"

*Following the tutorial for runnin WordCode v1.0, If I try to list the files
in the input folder I have created
*I get the famous error "Retrying connect to server:134.130.222.20:9000"
.what am I doing something wrong?


*Steps I have already verified*

*I have already checked the iptables of the three machines and they look**:
**
*Chain INPUT (policy ACCEPT)
target         prot opt source destiantion

Chain FORWARD (policy ACCEPT)
target         prot opt source destination

Chain OUTPUT (policy ACCEPT)
target        prot opt source destination

*
My hadoop-site.xml file looks like this
* configuration>
    <property>
        <name>fs.default.name</name>
        <value>134.130.222.20:9000/</value>
    </property>
    <property>
        <name>mapred.job.tracker</name>
        <value>134.130.222.18:9001</value>
    </property>
    ........
</configuration>

*Can someone help me out?*
*Thank you, CJ*

--0022152d6d75a3ede0046dc9ad60--

From general-return-323-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 09:22:28 2009
Return-Path: <general-return-323-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97682 invoked from network); 3 Jul 2009 09:22:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 09:22:28 -0000
Received: (qmail 48794 invoked by uid 500); 3 Jul 2009 09:22:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48718 invoked by uid 500); 3 Jul 2009 09:22:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48708 invoked by uid 99); 3 Jul 2009 09:22:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 09:22:38 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of anubhav.hanjura@convergys.com designates 167.1.158.29 as permitted sender)
Received: from [167.1.158.29] (HELO cvgmx2.convergys.com) (167.1.158.29)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 09:22:27 +0000
Received: from CDC5NW41.na.convergys.com (CDC5NW41.na.convergys.com [155.90.28.181])
	by cvgmx2.convergys.com (Postfix) with ESMTP id BFC9C154909E
	for <general@hadoop.apache.org>; Fri,  3 Jul 2009 09:20:21 +0000 (UTC)
In-Reply-To: <32120a6a0907030206l77a30d41yef2704e176f6d90f@mail.gmail.com>
To: general@hadoop.apache.org
Cc: general@hadoop.apache.org
Subject: Re: Tools to help administer a cluster?
MIME-Version: 1.0
X-Mailer: Lotus Notes Release 6.5.3 September 14, 2004
Message-ID: <OFAF48B9E5.387EDDAC-ON652575E8.0032DC06-652575E8.00337384@nd.convergys.com>
From: anubhav.hanjura@convergys.com
Date: Fri, 3 Jul 2009 14:51:59 +0530
X-MIMETrack: S/MIME Sign by Notes Client on Anubhav Hanjura/CIMG/CVG(Release 6.5.3|September
 14, 2004) at 07/03/2009 02:51:59 PM,
	Serialize by Notes Client on Anubhav Hanjura/CIMG/CVG(Release 6.5.3|September
 14, 2004) at 07/03/2009 02:51:59 PM,
	Serialize complete at 07/03/2009 02:51:59 PM,
	S/MIME Sign failed at 07/03/2009 02:51:59 PM: The cryptographic key was not
 found,
	Serialize by Router on CVGSMTP1/SRVR/CVG(Release 6.5.5FP3|March 22, 2007) at
 07/03/2009 05:22:33 AM,
	Serialize complete at 07/03/2009 05:22:33 AM
Content-Type: multipart/alternative; boundary="=_alternative 0033737F652575E8_="
X-Virus-Checked: Checked by ClamAV on apache.org

--=_alternative 0033737F652575E8_=
Content-Type: text/plain; charset="US-ASCII"

If you only need to copy files around to multiple hosts, you can use rsync 
utility to do that.
you can write a shell script to do the following 

1. zip the delta distribution preserving the directory structure
2. rsync the zip to a predefined location on all nodes
3. ssh to each of your nodes one after the other
4. execute the unzip and copy overf your hadoop delta distribution on 
remote machines

You can try the above steps on your current host to stage/rehearse your 
installation before you actually do it so that you can verify if things 
are ok.

hope it helps or was i completely unhelpful:)?

-a.r.h





tim robertson <timrobertson100@gmail.com> 
07/03/2009 02:36 PM
Please respond to
general@hadoop.apache.org


To
general@hadoop.apache.org
cc

Subject
Tools to help administer a cluster?






Hi all,

Can someone please point me at an open source tool that can help with
management of a *nix cluster?
I am looking for something that allows easy ability to copy over a new
Hadoop distribution and sync all the config files etc.  I'm not very
good at system admin, so hoping there is are some tools to help speed
things up.

I am running on a small cluster using Mac OSX right now, but will want
to use the same on a Linux distribution with 16 nodes shortly.

Cheers,

Tim



--=_alternative 0033737F652575E8_=--

From general-return-324-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 09:23:43 2009
Return-Path: <general-return-324-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98136 invoked from network); 3 Jul 2009 09:23:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 09:23:43 -0000
Received: (qmail 50163 invoked by uid 500); 3 Jul 2009 09:23:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50092 invoked by uid 500); 3 Jul 2009 09:23:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50082 invoked by uid 99); 3 Jul 2009 09:23:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 09:23:52 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of anubhav.hanjura@convergys.com designates 167.1.158.29 as permitted sender)
Received: from [167.1.158.29] (HELO cvgmx2.convergys.com) (167.1.158.29)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 09:23:43 +0000
Received: from CDC5NW41.na.convergys.com (CDC5NW41.na.convergys.com [155.90.28.181])
	by cvgmx2.convergys.com (Postfix) with ESMTP id D7DE71549051
	for <general@hadoop.apache.org>; Fri,  3 Jul 2009 09:21:38 +0000 (UTC)
To: general@hadoop.apache.org
Subject: how big in size is a hadoop distribution
MIME-Version: 1.0
X-Mailer: Lotus Notes Release 6.5.3 September 14, 2004
Message-ID: <OFF0850ED9.9E7976C0-ON652575E8.00337E24-652575E8.0033912F@nd.convergys.com>
From: anubhav.hanjura@convergys.com
Date: Fri, 3 Jul 2009 14:53:15 +0530
X-MIMETrack: S/MIME Sign by Notes Client on Anubhav Hanjura/CIMG/CVG(Release 6.5.3|September
 14, 2004) at 07/03/2009 02:53:15 PM,
	Serialize by Notes Client on Anubhav Hanjura/CIMG/CVG(Release 6.5.3|September
 14, 2004) at 07/03/2009 02:53:15 PM,
	Serialize complete at 07/03/2009 02:53:15 PM,
	S/MIME Sign failed at 07/03/2009 02:53:15 PM: The cryptographic key was not
 found,
	Serialize by Router on CVGSMTP1/SRVR/CVG(Release 6.5.5FP3|March 22, 2007) at
 07/03/2009 05:23:51 AM,
	Serialize complete at 07/03/2009 05:23:51 AM
Content-Type: multipart/alternative; boundary="=_alternative 0033912E652575E8_="
X-Virus-Checked: Checked by ClamAV on apache.org

--=_alternative 0033912E652575E8_=
Content-Type: text/plain; charset="US-ASCII"

How big is a typical hadoop distribution in megabytes and how is typically 
upgraded?
--=_alternative 0033912E652575E8_=--

From general-return-325-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 09:27:00 2009
Return-Path: <general-return-325-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 99963 invoked from network); 3 Jul 2009 09:27:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 09:27:00 -0000
Received: (qmail 54737 invoked by uid 500); 3 Jul 2009 09:27:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54672 invoked by uid 500); 3 Jul 2009 09:27:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54662 invoked by uid 99); 3 Jul 2009 09:27:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 09:27:10 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of timrobertson100@gmail.com designates 209.85.219.219 as permitted sender)
Received: from [209.85.219.219] (HELO mail-ew0-f219.google.com) (209.85.219.219)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 09:26:59 +0000
Received: by ewy19 with SMTP id 19so2656486ewy.29
        for <general@hadoop.apache.org>; Fri, 03 Jul 2009 02:26:39 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=5VVDRacZmClJeWm7dbk77OrFCZl/e5IkrzGERUwOS0c=;
        b=d+yvNblGhN+8hUH+Jym2IXFUlh8i0yMzEcQ2TZMBlQUM+rxv1Qh0CrMludspknZF61
         xZsjN/AJzsAKgJM/XE4kYacl7vTXDaO8Hijh7fTfE8XdY0Eq5f2C5jJhGk0isP06Hz2k
         Uayzc/9Q8Cn0SHTiiMMKw07X0rH/3F0PHn6kU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=u+lL9Lm/qOLefb9m/5MCHjYD/Eiu1cVQjzAg0fMXbkqJ6dGm8ahP9HYtic+QxBEh5t
         rt2mNDI8g7f/fUdoLJL/OI0gpiFEDLkBf+RW6WgB98o8EAKsohJM6dEp9hQR5LdCNiWJ
         XZfn25SdhFHWYaz5FcU6ozY7QJ4+OrNC1mhXo=
MIME-Version: 1.0
Received: by 10.216.8.78 with SMTP id 56mr280929weq.210.1246613198489; Fri, 03 
	Jul 2009 02:26:38 -0700 (PDT)
In-Reply-To: <aa41f20e0907030221k6ea5b2ffk3a5aade3b382a9d7@mail.gmail.com>
References: <aa41f20e0907030221k6ea5b2ffk3a5aade3b382a9d7@mail.gmail.com>
Date: Fri, 3 Jul 2009 11:26:38 +0200
Message-ID: <32120a6a0907030226m28d6bfc9w1b6af48f8db42347@mail.gmail.com>
Subject: Re: My secondary namenode seem not be running, and may be the reason 
	of my problem!!!
From: tim robertson <timrobertson100@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

"at this stage the shell is blocked unless I press the enter key" - my
cluster did this until I realised passphraseless ssh was not working
between the master and the nodes and the master to itself.
http://fak3r.com/2006/08/10/howto-passwordless-ssh-logins/

Cheers,

Tim



On Fri, Jul 3, 2009 at 11:21 AM, C J<xine.jar@googlemail.com> wrote:
> Hallo everyone,
> I have installed the hadoop 0.18.3 on three linux machines, I am trying t=
o
> run the
> example of WordCountv1.0 on a cluster. But I guess I have a problem
> somewhere.
> *
> Problem*
>
> *After formating the name node:*
> I am getting several STARTUP_MSG and at the end a "SHUTDOWN_MSG: shutting
> down the namenode..."
> is this normal?
>
> *Afterwards I try starting the dfs:**
> *I get a message "starting namenode..."* *afterwards I get another messag=
e
> "starting secondary namenode"
> at this stage the shell is blocked unless I press the enter key. Then the
> system tries to start another secondary
> namenode and the shell is then not blocked. What is going on?
> *
> Then I proceed an try starting the mapred:*
> I get the two messages "starting jobtracker....." and "starting tasktrack=
er"
>
> *Following the tutorial for runnin WordCode v1.0, If I try to list the fi=
les
> in the input folder I have created
> *I get the famous error "Retrying connect to server:134.130.222.20:9000"
> .what am I doing something wrong?
>
>
> *Steps I have already verified*
>
> *I have already checked the iptables of the three machines and they look*=
*:
> **
> *Chain INPUT (policy ACCEPT)
> target =A0 =A0 =A0 =A0 prot opt source destiantion
>
> Chain FORWARD (policy ACCEPT)
> target =A0 =A0 =A0 =A0 prot opt source destination
>
> Chain OUTPUT (policy ACCEPT)
> target =A0 =A0 =A0 =A0prot opt source destination
>
> *
> My hadoop-site.xml file looks like this
> * configuration>
> =A0 =A0<property>
> =A0 =A0 =A0 =A0<name>fs.default.name</name>
> =A0 =A0 =A0 =A0<value>134.130.222.20:9000/</value>
> =A0 =A0</property>
> =A0 =A0<property>
> =A0 =A0 =A0 =A0<name>mapred.job.tracker</name>
> =A0 =A0 =A0 =A0<value>134.130.222.18:9001</value>
> =A0 =A0</property>
> =A0 =A0........
> </configuration>
>
> *Can someone help me out?*
> *Thank you, CJ*
>

From general-return-326-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 12:09:07 2009
Return-Path: <general-return-326-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72256 invoked from network); 3 Jul 2009 12:09:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 12:09:07 -0000
Received: (qmail 61952 invoked by uid 500); 3 Jul 2009 12:09:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61901 invoked by uid 500); 3 Jul 2009 12:09:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61891 invoked by uid 99); 3 Jul 2009 12:09:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 12:09:17 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bogdan.maryniuk@gmail.com designates 209.85.222.196 as permitted sender)
Received: from [209.85.222.196] (HELO mail-pz0-f196.google.com) (209.85.222.196)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 12:09:09 +0000
Received: by pzk34 with SMTP id 34so2130284pzk.5
        for <general@hadoop.apache.org>; Fri, 03 Jul 2009 05:08:49 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=IfCzYwdjjHCHvI7z/Cjac+oH8MDhuPGP3stqGsAFf+4=;
        b=baZSG2L2O5Yrnk9+s0eOg/w02EvSpmNn8U2P5KnJSLcWc/TzSfYc2sybDtui2bdJvw
         oA5Z6/0hF1IBTl1TduO3e24eqtYk+s23OEwG3T52HZTewLQGUxKJKAhhIzpeTdLCcj9b
         yNQyIHffe1T8rzxEB9GJtgEW1S7uvUklLyu4c=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=x+1lhBhhPi4qJvtgcoZMLEs/71KgJtntXZWQ0JZol/+KN3FisA70nNkNsoyj74m45O
         FWzYZXh7/ImQ/Ij9GDeH0g8RqIQMAivkmfQVyCkPBewnDTeZzBOewiMl/1Lwob6VONPG
         qoYx2XuVeWSAFyWvpUDrh1fwpe5SRxizMWkxM=
MIME-Version: 1.0
Received: by 10.142.173.6 with SMTP id v6mr473122wfe.233.1246622929212; Fri, 
	03 Jul 2009 05:08:49 -0700 (PDT)
In-Reply-To: <32120a6a0907030206l77a30d41yef2704e176f6d90f@mail.gmail.com>
References: <32120a6a0907030206l77a30d41yef2704e176f6d90f@mail.gmail.com>
Date: Fri, 3 Jul 2009 21:08:49 +0900
Message-ID: <fa561940907030508h19139e87g4393484bd5e0077e@mail.gmail.com>
Subject: Re: Tools to help administer a cluster?
From: "Bogdan M. Maryniuk" <bogdan.maryniuk@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Fri, Jul 3, 2009 at 6:06 PM, tim robertson<timrobertson100@gmail.com> wr=
ote:
> I am looking for something that allows easy ability to copy over a new
> Hadoop distribution and sync all the config files etc. =C2=A0I'm not very
> good at system admin, so hoping there is are some tools to help speed
> things up.

HINT: Password-less here SSH is not just because Doug wanted so. :)
/bin/bash with SSH/scp are your very opensource friends, e.g.:

for hostname in `cat $HADOOP/slaves`;
do
   echo "Doing stuff on $hostname";
   scp /local/file.txt $hostname:/path/on/remote/host/;
   ssh $hostname cat /path/on/remote/host/file.txt;
done


P.S. Make sure "slaves" does not contains local host's IP. :-)

--=20
HTH, BM

Things, that are stupid at the beginning, rarely ends up wisely.

From general-return-327-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 12:17:46 2009
Return-Path: <general-return-327-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76478 invoked from network); 3 Jul 2009 12:17:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 12:17:46 -0000
Received: (qmail 78450 invoked by uid 500); 3 Jul 2009 12:17:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78394 invoked by uid 500); 3 Jul 2009 12:17:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78384 invoked by uid 99); 3 Jul 2009 12:17:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 12:17:56 +0000
X-ASF-Spam-Status: No, hits=3.7 required=10.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of xine.jar@googlemail.com designates 209.85.132.250 as permitted sender)
Received: from [209.85.132.250] (HELO an-out-0708.google.com) (209.85.132.250)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 12:17:44 +0000
Received: by an-out-0708.google.com with SMTP id c38so1049353ana.29
        for <general@hadoop.apache.org>; Fri, 03 Jul 2009 05:17:23 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=icsHwZPeLCOJaBYN/j44rHikUu0g4fsaLvaJRLwgy4c=;
        b=D5yV5+XU5aWZ0MuEzSTi52vs+a6avnjlFydn3sohRKJ/BdYIYjHru9zPqWzHXdehwl
         91cWtN3pR7KJAksxHlSypHSPQXBAXVWtoUnugMufDrpjHHUQVhyQuV617AGTmiJX3ZpJ
         JOcafN9sS1DRCKlnxvKHHzNTVKJSdmYgXI0wo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=kgn3mZY2aHx2RByfONNNK9ukr8KVB0vQOZ9Y9qjYSSxzFZjz3qJiLQvIUNgPxPoxpP
         pv1ryImhgplQHCjanun6dUaNUqvTQ+R0e2ucbbnFycRAlKUFth1xyfY7f+yLsypn/b9U
         revXt+iA2WmqjxRnQ9GjQ60mX+84CMdercVio=
MIME-Version: 1.0
Received: by 10.231.16.138 with SMTP id o10mr1074016iba.13.1246623440327; Fri, 
	03 Jul 2009 05:17:20 -0700 (PDT)
In-Reply-To: <32120a6a0907030226m28d6bfc9w1b6af48f8db42347@mail.gmail.com>
References: <aa41f20e0907030221k6ea5b2ffk3a5aade3b382a9d7@mail.gmail.com>
	 <32120a6a0907030226m28d6bfc9w1b6af48f8db42347@mail.gmail.com>
Date: Fri, 3 Jul 2009 14:17:20 +0200
Message-ID: <aa41f20e0907030517g3f2889d0u4d1ab2584de15bf8@mail.gmail.com>
Subject: Re: My secondary namenode seem not be running, and may be the reason 
	of my problem!!!
From: C J <xine.jar@googlemail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000325574d2a545725046dcc2354
X-Virus-Checked: Checked by ClamAV on apache.org

--000325574d2a545725046dcc2354
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

*Hallo Tim, and everyone,
I have  fixed the passphraseless, now each of the three machines can ssh the
others without a password. thanks to you I guess I moved one step forward
now the output of my  shell is the following:*

*My current status

**when starting the dfs*
alshain:~/Desktop/hadoop-0.18.3 # bin/start-dfs.sh
starting namenode, logging to
/root/Desktop/hadoop-0.18.3/logs/hadoop-root-namenode-alshain.out

134.130.222.17: starting datanode, logging to
/root/Desktop/hadoop-0.18.3/logs/hadoop-root-datanode-adhil.out

134.130.222.20: starting secondarynamenode, logging to
/root/Desktop/hadoop-0.18.3/logs/hadoop-root-secondarynamenode-alshain.out

134.130.222.18: starting secondarynamenode, logging to
/root/Desktop/hadoop-0.18.3/logs/hadoop-root-secondarynamenode-albali.out

*when starting the mapred*
alshain:~/Desktop/hadoop-0.18.3 # bin/start-mapred.sh
starting jobtracker, logging to
/root/Desktop/hadoop-0.18.3/logs/hadoop-root-jobtracker-alshain.out
1
34.130.222.17: starting tasktracker, logging to
/root/Desktop/hadoop-0.18.3/logs/hadoop-root-tasktracker-adhil.out

*Problem-1*

*when trying to list the content of the input folder*
alshain:~/Desktop/hadoop-0.18.3 # bin/hadoop dfs -ls
usr/tina/wordcount/input
09/07/03 14:00:21 INFO ipc.Client: Retrying connect to server:
alshain.mobnets.rwth-aachen.de/127.0.0.2:14000. Already tried 0 time(s).
09/07/03 14:00:22 INFO ipc.Client: Retrying connect to server:
alshain.mobnets.rwth-aachen.de/127.0.0.2:14000. Already tried 1 time(s).
09/07/03 14:00:23 INFO ipc.Client: Retrying connect to server:
alshain.mobnets.rwth-aachen.de/127.0.0.2:14000. Already tried 2 time(s).
09/07/03 14:00:24 INFO ipc.Client: Retrying connect to server:
alshain.mobnets.rwth-aachen.de/127.0.0.2:14000. Already tried 3 time(s).
09/07/03 14:00:25 INFO ipc.Client: Retrying connect to server:
alshain.mobnets.rwth-aachen.de/127.0.0.2:14000. Already tried 4 time(s).


*Problem 2
**
*alshain:~/Desktop/hadoop-0.18.3 # bin/stop-dfs.sh
no namenode to stop
134.130.222.17: stopping datanode
134.130.222.20: stopping secondarynamenode
134.130.222.18: stopping secondarynamenode

alshain:~/Desktop/hadoop-0.18.3 # bin/stop-mapred.sh
no jobtracker to stop
134.130.222.17: stopping tasktracker

This looks as if the namenode and the jobtracker were never started? any
idea why


*What I have checked

tried to ping the namenode using its name, it is successfull from any
machine

*alshain:~/Desktop/hadoop-0.18.3 # ping alshain.mobnets.rwth-aachen.de
PING alshain.mobnets.rwth-aachen.de (127.0.0.2) 56(84) bytes of data.
64 bytes from alshain.mobnets.rwth-aachen.de (127.0.0.2): icmp_seq=1 ttl=64
time=0.011 ms
64 bytes from alshain.mobnets.rwth-aachen.de (127.0.0.2): icmp_seq=2 ttl=64
time=0.010 ms
64 bytes from alshain.mobnets.rwth-aachen.de (127.0.0.2): icmp_seq=3 ttl=64
time=0.009 ms
--- alshain.mobnets.rwth-aachen.de ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.009/0.010/0.011/0.000 ms

*tried to ping the namenode using its ip address, it is successful from any
machine

*alshain:~/Desktop/hadoop-0.18.3 # ping 134.130.222.20
PING 134.130.222.20 (134.130.222.20) 56(84) bytes of data.
64 bytes from 134.130.222.20: icmp_seq=1 ttl=64 time=0.038 ms
64 bytes from 134.130.222.20: icmp_seq=2 ttl=64 time=0.009 ms
64 bytes from 134.130.222.20: icmp_seq=3 ttl=64 time=0.008 ms
64 bytes from 134.130.222.20: icmp_seq=4 ttl=64 time=0.009 ms
64 bytes from 134.130.222.20: icmp_seq=5 ttl=64 time=0.010 ms

--- 134.130.222.20 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3997ms
rtt min/avg/max/mdev = 0.008/0.014/0.038/0.012 ms



*Can anyone tell me why the namenode and the jobtracker were never started?
I appreciate your help,
CJ
*

On Fri, Jul 3, 2009 at 11:26 AM, tim robertson <timrobertson100@gmail.com>wrote:

> "at this stage the shell is blocked unless I press the enter key" - my
> cluster did this until I realised passphraseless ssh was not working
> between the master and the nodes and the master to itself.
> http://fak3r.com/2006/08/10/howto-passwordless-ssh-logins/
>
> Cheers,
>
> Tim
>
>
>
> On Fri, Jul 3, 2009 at 11:21 AM, C J<xine.jar@googlemail.com> wrote:
> > Hallo everyone,
> > I have installed the hadoop 0.18.3 on three linux machines, I am trying
> to
> > run the
> > example of WordCountv1.0 on a cluster. But I guess I have a problem
> > somewhere.
> > *
> > Problem*
> >
> > *After formating the name node:*
> > I am getting several STARTUP_MSG and at the end a "SHUTDOWN_MSG: shutting
> > down the namenode..."
> > is this normal?
> >
> > *Afterwards I try starting the dfs:**
> > *I get a message "starting namenode..."* *afterwards I get another
> message
> > "starting secondary namenode"
> > at this stage the shell is blocked unless I press the enter key. Then the
> > system tries to start another secondary
> > namenode and the shell is then not blocked. What is going on?
> > *
> > Then I proceed an try starting the mapred:*
> > I get the two messages "starting jobtracker....." and "starting
> tasktracker"
> >
> > *Following the tutorial for runnin WordCode v1.0, If I try to list the
> files
> > in the input folder I have created
> > *I get the famous error "Retrying connect to server:134.130.222.20:9000"
> > .what am I doing something wrong?
> >
> >
> > *Steps I have already verified*
> >
> > *I have already checked the iptables of the three machines and they
> look**:
> > **
> > *Chain INPUT (policy ACCEPT)
> > target         prot opt source destiantion
> >
> > Chain FORWARD (policy ACCEPT)
> > target         prot opt source destination
> >
> > Chain OUTPUT (policy ACCEPT)
> > target        prot opt source destination
> >
> > *
> > My hadoop-site.xml file looks like this
> > * configuration>
> >    <property>
> >        <name>fs.default.name</name>
> >        <value>134.130.222.20:9000/</value>
> >    </property>
> >    <property>
> >        <name>mapred.job.tracker</name>
> >        <value>134.130.222.18:9001</value>
> >    </property>
> >    ........
> > </configuration>
> >
> > *Can someone help me out?*
> > *Thank you, CJ*
> >
>

--000325574d2a545725046dcc2354--

From general-return-328-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 14:00:33 2009
Return-Path: <general-return-328-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27022 invoked from network); 3 Jul 2009 14:00:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 14:00:33 -0000
Received: (qmail 14229 invoked by uid 500); 3 Jul 2009 14:00:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14139 invoked by uid 500); 3 Jul 2009 14:00:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14129 invoked by uid 99); 3 Jul 2009 14:00:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 14:00:43 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of timrobertson100@gmail.com designates 209.85.219.219 as permitted sender)
Received: from [209.85.219.219] (HELO mail-ew0-f219.google.com) (209.85.219.219)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 14:00:35 +0000
Received: by ewy19 with SMTP id 19so2820440ewy.29
        for <general@hadoop.apache.org>; Fri, 03 Jul 2009 07:00:13 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=p+kDVoFeAa52X2zs5fb/+IZua5rrkvD0fNvvq5uk/y0=;
        b=E+ZG+gQdmg7/VvV+4rFqHBgTKotS8xEaRXK6s0Facyvn3QIS4bPnVLx7mpuZ12NwFb
         hMJsmb7Tj7wIGPe9QqpCKFlHIIRH14LnXeXE2S75nB1WIB81Ip349WMOxOiUnLe7CF5v
         etHdFr2ZfzJdyonvknlFUxwCFG7MwKsNbm9fk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=pDGN07cG1cqHt7yRJRqcZHiSLwIg7WxROnEa+DQLK79UdZMY3wJjX4i+HAK1IHeJLk
         GOntf55m8gLKnXoTm1CLveY+NydSwq3JxvgNPQmXF5AP4/PThEc7NfAQRD9YyYNnWaog
         Fw+6MHhOJYbV1gTEOkOsD03OqDG9WL389bvuk=
MIME-Version: 1.0
Received: by 10.216.16.208 with SMTP id h58mr360607weh.60.1246629613821; Fri, 
	03 Jul 2009 07:00:13 -0700 (PDT)
In-Reply-To: <fa561940907030508h19139e87g4393484bd5e0077e@mail.gmail.com>
References: <32120a6a0907030206l77a30d41yef2704e176f6d90f@mail.gmail.com>
	 <fa561940907030508h19139e87g4393484bd5e0077e@mail.gmail.com>
Date: Fri, 3 Jul 2009 16:00:13 +0200
Message-ID: <32120a6a0907030700i12876a6dld5a6b6b3c76b30f9@mail.gmail.com>
Subject: Re: Tools to help administer a cluster?
From: tim robertson <timrobertson100@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks both.  I'll brush up on my scripting skills ;o)


On Fri, Jul 3, 2009 at 2:08 PM, Bogdan M.
Maryniuk<bogdan.maryniuk@gmail.com> wrote:
> On Fri, Jul 3, 2009 at 6:06 PM, tim robertson<timrobertson100@gmail.com> =
wrote:
>> I am looking for something that allows easy ability to copy over a new
>> Hadoop distribution and sync all the config files etc. =A0I'm not very
>> good at system admin, so hoping there is are some tools to help speed
>> things up.
>
> HINT: Password-less here SSH is not just because Doug wanted so. :)
> /bin/bash with SSH/scp are your very opensource friends, e.g.:
>
> for hostname in `cat $HADOOP/slaves`;
> do
> =A0 echo "Doing stuff on $hostname";
> =A0 scp /local/file.txt $hostname:/path/on/remote/host/;
> =A0 ssh $hostname cat /path/on/remote/host/file.txt;
> done
>
>
> P.S. Make sure "slaves" does not contains local host's IP. :-)
>
> --
> HTH, BM
>
> Things, that are stupid at the beginning, rarely ends up wisely.
>

From general-return-329-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 14:32:21 2009
Return-Path: <general-return-329-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45866 invoked from network); 3 Jul 2009 14:32:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 14:32:21 -0000
Received: (qmail 57255 invoked by uid 500); 3 Jul 2009 14:32:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57225 invoked by uid 500); 3 Jul 2009 14:32:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57215 invoked by uid 99); 3 Jul 2009 14:32:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 14:32:31 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of fredwang222@gmail.com designates 209.85.212.172 as permitted sender)
Received: from [209.85.212.172] (HELO mail-vw0-f172.google.com) (209.85.212.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 14:32:20 +0000
Received: by vwj2 with SMTP id 2so1538348vwj.5
        for <general@hadoop.apache.org>; Fri, 03 Jul 2009 07:31:59 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=CFAhDcJ21hqQHtehpR7vT/ez2kGis9x2nlpSyKVLAKc=;
        b=uhhDFw8wPeHCS5Di5WBzrTVjhUtdZ0I83Ojze4QsfV/6AHU2igWuLbisfi0fUn33E3
         DfPHjEqsKMGKL2Cj0JimpVYb75RuxndOceLu3Ax7jL0q/+X/QvfWa38jCcHt9wNmQjGf
         wBKxlGy/u2VNqjhcYaU0JASnfC5rh07LbcDGw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=xCGO0VRxRGj0h7ZS/35Y7dH9AELMFZPVzooHRBibwcunP/OXH9ApokqGAb5FFdxYYR
         L6V1n8HPDvtLZGsY2hJ35CX2iE1FUFaenGD8e/uCKFnCEgdJRjA4Pyaf3hVJoV4c8osU
         F/X06UQ63bG5I3wWqNIG6pre3/9XY9ho/Pq1s=
MIME-Version: 1.0
Received: by 10.220.96.15 with SMTP id f15mr2676615vcn.116.1246631519007; Fri, 
	03 Jul 2009 07:31:59 -0700 (PDT)
In-Reply-To: <4A4CDF0F.2040106@yahoo-inc.com>
References: <702261350907010735u3502197au800281a5a131d963@mail.gmail.com>
	 <4A4B8B99.10303@yahoo-inc.com>
	 <702261350907012211r17dec368u17e18052383a8822@mail.gmail.com>
	 <4A4C430F.7080306@yahoo-inc.com>
	 <702261350907012251s41b18b24m5dc54015007d494e@mail.gmail.com>
	 <702261350907012309l4c62885bk6df2ebac8e2b7254@mail.gmail.com>
	 <4A4CDF0F.2040106@yahoo-inc.com>
Date: Fri, 3 Jul 2009 22:31:58 +0800
Message-ID: <702261350907030731w694cd855safc8b3e5b5048747@mail.gmail.com>
Subject: Re: fail to setup passphraseless ssh and need some help
From: fred wang <fredwang222@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6470efcdb33fe046dce0429
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6470efcdb33fe046dce0429
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

I remove the ~/.ssh and regenerate the key and it seems I still need to
provide password when I ssh localhost. Thank you very much even it couldn't
be fixed finally.

But I found there is some warning information:

ssh localhost

The authenticity of host 'localhost (127.0.0.1)' can't be established.

RSA key fingerprint is 4f:a1:ff:ed:0c:46:3e:a9:8c:97:bc:b7:46:3e:35:d2.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'localhost' (RSA) to the list of known hosts.


On 7/1/09 11:09 PM, fred wang wrote:

>  sorry, should incopy ssh_config(instead of sshd_config)
>>
>>
>> vi /etc/ssh/ssh_config
>>
>> #  1. command line options
>>
>> #  2. user-specific file
>>
>> #  3. system-wide file
>>
>> # Any configuration value is only changed the first time it is set.
>>
>> # Thus, host-specific definitions should be at the beginning of the
>>
>> # configuration file, and defaults at the end.
>>
>>
>>
>> # Site-wide defaults for some commonly used options.  For a comprehensive
>>
>> # list of available options, their meanings and defaults, please see the
>>
>> # ssh_config(5) man page.
>>
>>
>>
>> Host *
>>
>> #   ForwardAgent no
>>
>> #   ForwardX11 no
>>
>> #   ForwardX11Trusted yes
>>
>> #   RhostsRSAAuthentication no
>>
>> #   RSAAuthentication yes
>>
>> #   PasswordAuthentication yes
>>
>> #   HostbasedAuthentication no
>>
>> #   GSSAPIAuthentication no
>>
>> #   GSSAPIDelegateCredentials no
>>
>> #   GSSAPIKeyExchange no
>>
>> #   GSSAPITrustDNS no
>>
>> #   BatchMode no
>>
>> #   CheckHostIP yes
>>
>> #   AddressFamily any
>>
>> #   ConnectTimeout 0
>>
>> #   StrictHostKeyChecking ask
>>
>> #   IdentityFile ~/.ssh/identity
>>
>> #   IdentityFile ~/.ssh/id_rsa
>>
>> #   IdentityFile ~/.ssh/id_dsa
>>
>> #   Port 22
>>
>> #   Protocol 2,1
>>
>> #   Cipher 3des
>>
>> #   Ciphers
>> aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc
>>
>> #   MACs hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160
>>
>> #   EscapeChar ~
>>
>> #   Tunnel no
>>
>> #   TunnelDevice any:any
>>
>> #   PermitLocalCommand no
>>
>>     SendEnv LANG LC_*
>>
>>     HashKnownHosts yes
>>
>>     GSSAPIAuthentication yes
>>
>>     GSSAPIDelegateCredentials no
>>
>>
>> On Thu, Jul 2, 2009 at 1:51 PM, fred wang<fredwang222@gmail.com>  wrote:
>>
>> Here is the output of ssh -v localhost  and the configuration of
>>> ssh_config,
>>>
>>> xxx@xxx-desktop:~$ ssh -v localhost
>>>
>>> OpenSSH_4.7p1 Debian-8ubuntu1.2, OpenSSL 0.9.8g 19 Oct 2007
>>>
>>> debug1: Reading configuration data /etc/ssh/ssh_config
>>>
>>> debug1: Applying options for *
>>>
>>> debug1: Connecting to localhost [127.0.0.1] port 22.
>>>
>>> debug1: Connection established.
>>>
>>> debug1: identity file /home/xxx/.ssh/identity type -1
>>>
>>> debug1: identity file /home/xxx/.ssh/id_rsa type -1
>>>
>>> debug1: identity file /home/xxx/.ssh/id_dsa type 2
>>>
>>> debug1: Remote protocol version 2.0, remote software version
>>> OpenSSH_4.7p1
>>> Debian-8ubuntu1.2
>>>
>>> debug1: match: OpenSSH_4.7p1 Debian-8ubuntu1.2 pat OpenSSH*
>>>
>>> debug1: Enabling compatibility mode for protocol 2.0
>>>
>>> debug1: Local version string SSH-2.0-OpenSSH_4.7p1 Debian-8ubuntu1.2
>>>
>>> debug1: SSH2_MSG_KEXINIT sent
>>>
>>> debug1: SSH2_MSG_KEXINIT received
>>>
>>> debug1: kex: server->client aes128-cbc hmac-md5 none
>>>
>>> debug1: kex: client->server aes128-cbc hmac-md5 none
>>>
>>> debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
>>>
>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
>>>
>>> debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
>>>
>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
>>>
>>> debug1: Host 'localhost' is known and matches the RSA host key.
>>>
>>> debug1: Found key in /home/xxx/.ssh/known_hosts:1
>>>
>>> debug1: ssh_rsa_verify: signature correct
>>>
>>> debug1: SSH2_MSG_NEWKEYS sent
>>>
>>> debug1: expecting SSH2_MSG_NEWKEYS
>>>
>>> debug1: SSH2_MSG_NEWKEYS received
>>>
>>> debug1: SSH2_MSG_SERVICE_REQUEST sent
>>>
>>> debug1: SSH2_MSG_SERVICE_ACCEPT received
>>>
>>> debug1: Authentications that can continue: publickey,password
>>>
>>> debug1: Next authentication method: publickey
>>>
>>> debug1: Trying private key: /home/xxx/.ssh/identity
>>>
>>> debug1: Trying private key: /home/xxx/.ssh/id_rsa
>>>
>>> debug1: Offering public key: /home/xxx/.ssh/id_dsa
>>>
>>> debug1: Authentications that can continue: publickey,password
>>>
>>> debug1: Next authentication method: password
>>>
>>> xxx@localhost's password:
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> xxx@xxx:~$ vi /etc/ssh/sshd_config
>>>
>>> #KerberosOrLocalPasswd yes
>>>
>>> #KerberosTicketCleanup yes
>>>
>>>
>>>
>>> # GSSAPI options
>>>
>>> #GSSAPIAuthentication no
>>>
>>> #GSSAPICleanupCredentials yes
>>>
>>>
>>>
>>> X11Forwarding yes
>>>
>>> X11DisplayOffset 10
>>>
>>> PrintMotd no
>>>
>>> PrintLastLog yes
>>>
>>> TCPKeepAlive yes
>>>
>>> #UseLogin no
>>>
>>>
>>>
>>> #MaxStartups 10:30:60
>>>
>>> #Banner /etc/issue.net
>>>
>>>
>>>
>>> # Allow client to pass locale environment variables
>>>
>>> AcceptEnv LANG LC_*
>>>
>>>
>>>
>>> Subsystem sftp /usr/lib/openssh/sftp-server
>>>
>>>
>>>
>>> UsePAM yes
>>>
>>>
>>>
>>> On Thu, Jul 2, 2009 at 1:18 PM, Konstantin Boudnik<cos@yahoo-inc.com
>>> >wrote:
>>>
>>> Yet another possibility is that your SSH daemon isn't configured to
>>>> accept
>>>> publickey as a valid authorization mean.
>>>>
>>>> Try to do ssh -v localhost and check if there's something similar to the
>>>> following:
>>>>
>>>> debug1: Authentications that can continue:
>>>> publickey,password,keyboard-interactive
>>>> debug1: Next authentication method: publickey
>>>> debug1: Trying private key: /home/xxx/.ssh/identity
>>>> debug1: Trying private key: /home/xxx/.ssh/id_rsa
>>>> debug1: Offering public key: /home/xxx/.ssh/id_dsa
>>>> debug1: Server accepts key: pkalg ssh-dss blen 435
>>>> debug1: read PEM private key done: type DSA
>>>> debug1: Authentication succeeded (publickey).
>>>>
>>>> Cos
>>>>
>>>>
>>>> On 7/1/09 10:11 PM, fred wang wrote:
>>>>
>>>> I have setup ./.ssh/authorized keys has permssion 600, but it didn't
>>>>> work.
>>>>> Thanks anyway
>>>>>
>>>>> ls -l .ssh/authorized_keys
>>>>> -rw------- 1 xxx xxx 1222 2009-07-02 13:08 .ssh/authorized_keys
>>>>>
>>>>> On Thu, Jul 2, 2009 at 12:15 AM, Konstantin Boudnik<cos@yahoo-inc.com
>>>>>
>>>>>> wrote:
>>>>>>
>>>>> Make sure that your ~/.ssh/authorized_keys has permissions 600
>>>>>
>>>>>> Cos
>>>>>>
>>>>>>
>>>>>> On 7/1/09 7:35 AM, fred wang wrote:
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>>>   I failed to setup passphraseless ssh(I mean, I still need to input
>>>>>>> password to do ssh localhost) when I tried to configure Hadoop to run
>>>>>>> on
>>>>>>> psuedo-distributed operation,  could anyone help me solve this issue?
>>>>>>> Thanks!
>>>>>>>
>>>>>>> (1)I use the Putty0.6 to remote access to Ubuntu by SSH.
>>>>>>>
>>>>>>> (2) execution steps and ouput
>>>>>>>
>>>>>>> $ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
>>>>>>> Generating public/private dsa key pair.
>>>>>>> Your identification has been saved in /home/xxx/.ssh/id_dsa.
>>>>>>> Your public key has been saved in /home/xxx/.ssh/id_dsa.pub.
>>>>>>> The key fingerprint is:
>>>>>>> a9:39:4c:9b:22:f9:a4:77:70:24:fa:bf:12:f5:81:81 xxx
>>>>>>>
>>>>>>>
>>>>>>> **note: it doesn't have message  'Enter passphrase (empty for no
>>>>>>> passphrase):
>>>>>>>     Enter same passphrase again: ' which appear in some introductory
>>>>>>> paper.
>>>>>>> "
>>>>>>>
>>>>>>> $ cat ~/.ssh/id_dsa.pub>>    ~/.ssh/authorized_keys
>>>>>>> no output
>>>>>>>
>>>>>>> $ ssh localhost
>>>>>>> The authenticity of host 'localhost (127.0.0.1)' can't be
>>>>>>> established.
>>>>>>> RSA key fingerprint is
>>>>>>> 4f:a1:ff:ed:0c:46:3e:a9:8c:97:bc:b7:46:3e:35:d2.
>>>>>>> Are you sure you want to continue connecting (yes/no)? yes
>>>>>>> Warning: Permanently added 'localhost' (RSA) to the list of known
>>>>>>> hosts.
>>>>>>> xxx@localhost's password:
>>>>>>>
>>>>>>>
>>>>>>>

--0016e6470efcdb33fe046dce0429--

From general-return-330-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 14:45:49 2009
Return-Path: <general-return-330-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 52453 invoked from network); 3 Jul 2009 14:45:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 14:45:49 -0000
Received: (qmail 78107 invoked by uid 500); 3 Jul 2009 14:45:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78016 invoked by uid 500); 3 Jul 2009 14:45:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78004 invoked by uid 99); 3 Jul 2009 14:45:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 14:45:59 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of timrobertson100@gmail.com designates 209.85.219.219 as permitted sender)
Received: from [209.85.219.219] (HELO mail-ew0-f219.google.com) (209.85.219.219)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 14:45:47 +0000
Received: by ewy19 with SMTP id 19so2851592ewy.29
        for <general@hadoop.apache.org>; Fri, 03 Jul 2009 07:45:27 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=FHehYSTnTiMNK/+0FinKd5x1gzAXcz3jwjF6Ck/XtgE=;
        b=kuZ9BaLTqtXzRi4NklweKEGJqzzYT5nBGJW7tOYFAGEtehEdSY270y3wPjbKdAdKD2
         Q+1pLtXiFqiN0NbhbPRj5voF61O72n7uCdt/bZYeaSrpvdcLciUmEut16MvZg5iTxx6/
         ONnOvLtdInrtuYaVkmGEL3KKg3X+MhvLrtaLQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=xE0cwOZHhvDO57PS+wnaURRR2JdZfurjXvaVcPO7dsfDwhAHE6QbuvZhtcQhvHDGta
         I2Z2eVGDTo7cwhfv4906ktbXZkUHeSSb/QtWDU/wbvQUrifiBK9ZvPUQANmzI8WKmRmJ
         GP6Tg0xqFs/+dYskfyVlqxJ+PscI2wjkwQKGw=
MIME-Version: 1.0
Received: by 10.216.2.210 with SMTP id 60mr380216wef.21.1246632327677; Fri, 03 
	Jul 2009 07:45:27 -0700 (PDT)
In-Reply-To: <702261350907030731w694cd855safc8b3e5b5048747@mail.gmail.com>
References: <702261350907010735u3502197au800281a5a131d963@mail.gmail.com>
	 <4A4B8B99.10303@yahoo-inc.com>
	 <702261350907012211r17dec368u17e18052383a8822@mail.gmail.com>
	 <4A4C430F.7080306@yahoo-inc.com>
	 <702261350907012251s41b18b24m5dc54015007d494e@mail.gmail.com>
	 <702261350907012309l4c62885bk6df2ebac8e2b7254@mail.gmail.com>
	 <4A4CDF0F.2040106@yahoo-inc.com>
	 <702261350907030731w694cd855safc8b3e5b5048747@mail.gmail.com>
Date: Fri, 3 Jul 2009 16:45:27 +0200
Message-ID: <32120a6a0907030745g19ef2593o457f48972ef51ea@mail.gmail.com>
Subject: Re: fail to setup passphraseless ssh and need some help
From: tim robertson <timrobertson100@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

This seemed to me to be a good resource (e.g. it worked for me):
http://fak3r.com/2006/08/10/howto-passwordless-ssh-logins/

Tim


On Fri, Jul 3, 2009 at 4:31 PM, fred wang<fredwang222@gmail.com> wrote:
> I remove the ~/.ssh and regenerate the key and it seems I still need to
> provide password when I ssh localhost. Thank you very much even it couldn=
't
> be fixed finally.
>
> But I found there is some warning information:
>
> ssh localhost
>
> The authenticity of host 'localhost (127.0.0.1)' can't be established.
>
> RSA key fingerprint is 4f:a1:ff:ed:0c:46:3e:a9:8c:97:bc:b7:46:3e:35:d2.
>
> Are you sure you want to continue connecting (yes/no)? yes
>
> Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
>
>
> On 7/1/09 11:09 PM, fred wang wrote:
>
>> =A0sorry, should incopy ssh_config(instead of sshd_config)
>>>
>>>
>>> vi /etc/ssh/ssh_config
>>>
>>> # =A01. command line options
>>>
>>> # =A02. user-specific file
>>>
>>> # =A03. system-wide file
>>>
>>> # Any configuration value is only changed the first time it is set.
>>>
>>> # Thus, host-specific definitions should be at the beginning of the
>>>
>>> # configuration file, and defaults at the end.
>>>
>>>
>>>
>>> # Site-wide defaults for some commonly used options. =A0For a comprehen=
sive
>>>
>>> # list of available options, their meanings and defaults, please see th=
e
>>>
>>> # ssh_config(5) man page.
>>>
>>>
>>>
>>> Host *
>>>
>>> # =A0 ForwardAgent no
>>>
>>> # =A0 ForwardX11 no
>>>
>>> # =A0 ForwardX11Trusted yes
>>>
>>> # =A0 RhostsRSAAuthentication no
>>>
>>> # =A0 RSAAuthentication yes
>>>
>>> # =A0 PasswordAuthentication yes
>>>
>>> # =A0 HostbasedAuthentication no
>>>
>>> # =A0 GSSAPIAuthentication no
>>>
>>> # =A0 GSSAPIDelegateCredentials no
>>>
>>> # =A0 GSSAPIKeyExchange no
>>>
>>> # =A0 GSSAPITrustDNS no
>>>
>>> # =A0 BatchMode no
>>>
>>> # =A0 CheckHostIP yes
>>>
>>> # =A0 AddressFamily any
>>>
>>> # =A0 ConnectTimeout 0
>>>
>>> # =A0 StrictHostKeyChecking ask
>>>
>>> # =A0 IdentityFile ~/.ssh/identity
>>>
>>> # =A0 IdentityFile ~/.ssh/id_rsa
>>>
>>> # =A0 IdentityFile ~/.ssh/id_dsa
>>>
>>> # =A0 Port 22
>>>
>>> # =A0 Protocol 2,1
>>>
>>> # =A0 Cipher 3des
>>>
>>> # =A0 Ciphers
>>> aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-=
cbc
>>>
>>> # =A0 MACs hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160
>>>
>>> # =A0 EscapeChar ~
>>>
>>> # =A0 Tunnel no
>>>
>>> # =A0 TunnelDevice any:any
>>>
>>> # =A0 PermitLocalCommand no
>>>
>>> =A0 =A0 SendEnv LANG LC_*
>>>
>>> =A0 =A0 HashKnownHosts yes
>>>
>>> =A0 =A0 GSSAPIAuthentication yes
>>>
>>> =A0 =A0 GSSAPIDelegateCredentials no
>>>
>>>
>>> On Thu, Jul 2, 2009 at 1:51 PM, fred wang<fredwang222@gmail.com> =A0wro=
te:
>>>
>>> Here is the output of ssh -v localhost =A0and the configuration of
>>>> ssh_config,
>>>>
>>>> xxx@xxx-desktop:~$ ssh -v localhost
>>>>
>>>> OpenSSH_4.7p1 Debian-8ubuntu1.2, OpenSSL 0.9.8g 19 Oct 2007
>>>>
>>>> debug1: Reading configuration data /etc/ssh/ssh_config
>>>>
>>>> debug1: Applying options for *
>>>>
>>>> debug1: Connecting to localhost [127.0.0.1] port 22.
>>>>
>>>> debug1: Connection established.
>>>>
>>>> debug1: identity file /home/xxx/.ssh/identity type -1
>>>>
>>>> debug1: identity file /home/xxx/.ssh/id_rsa type -1
>>>>
>>>> debug1: identity file /home/xxx/.ssh/id_dsa type 2
>>>>
>>>> debug1: Remote protocol version 2.0, remote software version
>>>> OpenSSH_4.7p1
>>>> Debian-8ubuntu1.2
>>>>
>>>> debug1: match: OpenSSH_4.7p1 Debian-8ubuntu1.2 pat OpenSSH*
>>>>
>>>> debug1: Enabling compatibility mode for protocol 2.0
>>>>
>>>> debug1: Local version string SSH-2.0-OpenSSH_4.7p1 Debian-8ubuntu1.2
>>>>
>>>> debug1: SSH2_MSG_KEXINIT sent
>>>>
>>>> debug1: SSH2_MSG_KEXINIT received
>>>>
>>>> debug1: kex: server->client aes128-cbc hmac-md5 none
>>>>
>>>> debug1: kex: client->server aes128-cbc hmac-md5 none
>>>>
>>>> debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
>>>>
>>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
>>>>
>>>> debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
>>>>
>>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
>>>>
>>>> debug1: Host 'localhost' is known and matches the RSA host key.
>>>>
>>>> debug1: Found key in /home/xxx/.ssh/known_hosts:1
>>>>
>>>> debug1: ssh_rsa_verify: signature correct
>>>>
>>>> debug1: SSH2_MSG_NEWKEYS sent
>>>>
>>>> debug1: expecting SSH2_MSG_NEWKEYS
>>>>
>>>> debug1: SSH2_MSG_NEWKEYS received
>>>>
>>>> debug1: SSH2_MSG_SERVICE_REQUEST sent
>>>>
>>>> debug1: SSH2_MSG_SERVICE_ACCEPT received
>>>>
>>>> debug1: Authentications that can continue: publickey,password
>>>>
>>>> debug1: Next authentication method: publickey
>>>>
>>>> debug1: Trying private key: /home/xxx/.ssh/identity
>>>>
>>>> debug1: Trying private key: /home/xxx/.ssh/id_rsa
>>>>
>>>> debug1: Offering public key: /home/xxx/.ssh/id_dsa
>>>>
>>>> debug1: Authentications that can continue: publickey,password
>>>>
>>>> debug1: Next authentication method: password
>>>>
>>>> xxx@localhost's password:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> xxx@xxx:~$ vi /etc/ssh/sshd_config
>>>>
>>>> #KerberosOrLocalPasswd yes
>>>>
>>>> #KerberosTicketCleanup yes
>>>>
>>>>
>>>>
>>>> # GSSAPI options
>>>>
>>>> #GSSAPIAuthentication no
>>>>
>>>> #GSSAPICleanupCredentials yes
>>>>
>>>>
>>>>
>>>> X11Forwarding yes
>>>>
>>>> X11DisplayOffset 10
>>>>
>>>> PrintMotd no
>>>>
>>>> PrintLastLog yes
>>>>
>>>> TCPKeepAlive yes
>>>>
>>>> #UseLogin no
>>>>
>>>>
>>>>
>>>> #MaxStartups 10:30:60
>>>>
>>>> #Banner /etc/issue.net
>>>>
>>>>
>>>>
>>>> # Allow client to pass locale environment variables
>>>>
>>>> AcceptEnv LANG LC_*
>>>>
>>>>
>>>>
>>>> Subsystem sftp /usr/lib/openssh/sftp-server
>>>>
>>>>
>>>>
>>>> UsePAM yes
>>>>
>>>>
>>>>
>>>> On Thu, Jul 2, 2009 at 1:18 PM, Konstantin Boudnik<cos@yahoo-inc.com
>>>> >wrote:
>>>>
>>>> Yet another possibility is that your SSH daemon isn't configured to
>>>>> accept
>>>>> publickey as a valid authorization mean.
>>>>>
>>>>> Try to do ssh -v localhost and check if there's something similar to =
the
>>>>> following:
>>>>>
>>>>> debug1: Authentications that can continue:
>>>>> publickey,password,keyboard-interactive
>>>>> debug1: Next authentication method: publickey
>>>>> debug1: Trying private key: /home/xxx/.ssh/identity
>>>>> debug1: Trying private key: /home/xxx/.ssh/id_rsa
>>>>> debug1: Offering public key: /home/xxx/.ssh/id_dsa
>>>>> debug1: Server accepts key: pkalg ssh-dss blen 435
>>>>> debug1: read PEM private key done: type DSA
>>>>> debug1: Authentication succeeded (publickey).
>>>>>
>>>>> Cos
>>>>>
>>>>>
>>>>> On 7/1/09 10:11 PM, fred wang wrote:
>>>>>
>>>>> I have setup ./.ssh/authorized keys has permssion 600, but it didn't
>>>>>> work.
>>>>>> Thanks anyway
>>>>>>
>>>>>> ls -l .ssh/authorized_keys
>>>>>> -rw------- 1 xxx xxx 1222 2009-07-02 13:08 .ssh/authorized_keys
>>>>>>
>>>>>> On Thu, Jul 2, 2009 at 12:15 AM, Konstantin Boudnik<cos@yahoo-inc.co=
m
>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>> Make sure that your ~/.ssh/authorized_keys has permissions 600
>>>>>>
>>>>>>> Cos
>>>>>>>
>>>>>>>
>>>>>>> On 7/1/09 7:35 AM, fred wang wrote:
>>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>>> =A0 I failed to setup passphraseless ssh(I mean, I still need to i=
nput
>>>>>>>> password to do ssh localhost) when I tried to configure Hadoop to =
run
>>>>>>>> on
>>>>>>>> psuedo-distributed operation, =A0could anyone help me solve this i=
ssue?
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> (1)I use the Putty0.6 to remote access to Ubuntu by SSH.
>>>>>>>>
>>>>>>>> (2) execution steps and ouput
>>>>>>>>
>>>>>>>> $ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
>>>>>>>> Generating public/private dsa key pair.
>>>>>>>> Your identification has been saved in /home/xxx/.ssh/id_dsa.
>>>>>>>> Your public key has been saved in /home/xxx/.ssh/id_dsa.pub.
>>>>>>>> The key fingerprint is:
>>>>>>>> a9:39:4c:9b:22:f9:a4:77:70:24:fa:bf:12:f5:81:81 xxx
>>>>>>>>
>>>>>>>>
>>>>>>>> **note: it doesn't have message =A0'Enter passphrase (empty for no
>>>>>>>> passphrase):
>>>>>>>> =A0 =A0 Enter same passphrase again: ' which appear in some introd=
uctory
>>>>>>>> paper.
>>>>>>>> "
>>>>>>>>
>>>>>>>> $ cat ~/.ssh/id_dsa.pub>> =A0 =A0~/.ssh/authorized_keys
>>>>>>>> no output
>>>>>>>>
>>>>>>>> $ ssh localhost
>>>>>>>> The authenticity of host 'localhost (127.0.0.1)' can't be
>>>>>>>> established.
>>>>>>>> RSA key fingerprint is
>>>>>>>> 4f:a1:ff:ed:0c:46:3e:a9:8c:97:bc:b7:46:3e:35:d2.
>>>>>>>> Are you sure you want to continue connecting (yes/no)? yes
>>>>>>>> Warning: Permanently added 'localhost' (RSA) to the list of known
>>>>>>>> hosts.
>>>>>>>> xxx@localhost's password:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>

From general-return-331-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 18:05:29 2009
Return-Path: <general-return-331-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 32745 invoked from network); 3 Jul 2009 18:05:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 18:05:29 -0000
Received: (qmail 4678 invoked by uid 500); 3 Jul 2009 18:05:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4599 invoked by uid 500); 3 Jul 2009 18:05:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4589 invoked by uid 99); 3 Jul 2009 18:05:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 18:05:39 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [72.14.220.159] (HELO fg-out-1718.google.com) (72.14.220.159)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 18:05:29 +0000
Received: by fg-out-1718.google.com with SMTP id 13so893228fge.12
        for <general@hadoop.apache.org>; Fri, 03 Jul 2009 11:05:08 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.86.25.19 with SMTP id 19mr1081486fgy.71.1246644308309; Fri, 03 
	Jul 2009 11:05:08 -0700 (PDT)
In-Reply-To: <32120a6a0907030700i12876a6dld5a6b6b3c76b30f9@mail.gmail.com>
References: <32120a6a0907030206l77a30d41yef2704e176f6d90f@mail.gmail.com> 
	<fa561940907030508h19139e87g4393484bd5e0077e@mail.gmail.com> 
	<32120a6a0907030700i12876a6dld5a6b6b3c76b30f9@mail.gmail.com>
From: Philip Zeyliger <philip@cloudera.com>
Date: Fri, 3 Jul 2009 11:04:48 -0700
Message-ID: <15da8a100907031104s5fc40245tca4ba09cb65e60e@mail.gmail.com>
Subject: Re: Tools to help administer a cluster?
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

We build (and use) RPM & deb packages for Hadoop.  See
http://cloudera.com/hadoop .  You can package up your config files
into an RPM too (see my.cloudera.com) or use rsync for those, and you
can distribute the binaries by pointing your machines to a yum/apt
repository.

Cheers,

-- Philip

On Fri, Jul 3, 2009 at 7:00 AM, tim robertson<timrobertson100@gmail.com> wr=
ote:
> Thanks both. =A0I'll brush up on my scripting skills ;o)
>
>
> On Fri, Jul 3, 2009 at 2:08 PM, Bogdan M.
> Maryniuk<bogdan.maryniuk@gmail.com> wrote:
>> On Fri, Jul 3, 2009 at 6:06 PM, tim robertson<timrobertson100@gmail.com>=
 wrote:
>>> I am looking for something that allows easy ability to copy over a new
>>> Hadoop distribution and sync all the config files etc. =A0I'm not very
>>> good at system admin, so hoping there is are some tools to help speed
>>> things up.
>>
>> HINT: Password-less here SSH is not just because Doug wanted so. :)
>> /bin/bash with SSH/scp are your very opensource friends, e.g.:
>>
>> for hostname in `cat $HADOOP/slaves`;
>> do
>> =A0 echo "Doing stuff on $hostname";
>> =A0 scp /local/file.txt $hostname:/path/on/remote/host/;
>> =A0 ssh $hostname cat /path/on/remote/host/file.txt;
>> done
>>
>>
>> P.S. Make sure "slaves" does not contains local host's IP. :-)
>>
>> --
>> HTH, BM
>>
>> Things, that are stupid at the beginning, rarely ends up wisely.
>>
>

From general-return-332-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 22:01:56 2009
Return-Path: <general-return-332-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 28355 invoked from network); 3 Jul 2009 22:01:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 22:01:56 -0000
Received: (qmail 78651 invoked by uid 500); 3 Jul 2009 22:02:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78574 invoked by uid 500); 3 Jul 2009 22:02:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78564 invoked by uid 99); 3 Jul 2009 22:02:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 22:02:06 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ross.dmochowski@gmail.com designates 209.85.212.172 as permitted sender)
Received: from [209.85.212.172] (HELO mail-vw0-f172.google.com) (209.85.212.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 22:01:58 +0000
Received: by vwj2 with SMTP id 2so1690979vwj.5
        for <general@hadoop.apache.org>; Fri, 03 Jul 2009 15:01:37 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=5FXziWiOlmI5e195RMsgNWoJK0mYMzPu9rrYKB0iDio=;
        b=cfq4HZf69ibNH9Auqdyida4i3oq0vBY0lRO3jFdmF4F+Ui4GQKzceL7llqWWUyeGyu
         zQF4euSAdyo/VGvNhVulkNH3vwWuy7Zc6n5dCpFI1BtNFKhmLnIG3KsgiecWigJJSNfM
         frGUytFWLu9ndV8Uoq1zhIf178zXKhoo7w1K8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=nIK/JARblybLSLYDac1l5LxvJ1NLBocubqIBNhMSCvX0V/3TGMUE+mteTYILvPS9cp
         UzS2Cve09MLEVCx2QRFBG6US3sbKf99NcYI/3jpD280TZDt+lTUEmTBNnvIU+VIVRdbe
         b4LT4bLMy+GP3Af1k2FBPinY/IXmQ2nF4lWOI=
MIME-Version: 1.0
Received: by 10.220.72.78 with SMTP id l14mr3601044vcj.81.1246658497326; Fri, 
	03 Jul 2009 15:01:37 -0700 (PDT)
In-Reply-To: <15da8a100907031104s5fc40245tca4ba09cb65e60e@mail.gmail.com>
References: <32120a6a0907030206l77a30d41yef2704e176f6d90f@mail.gmail.com>
	 <fa561940907030508h19139e87g4393484bd5e0077e@mail.gmail.com>
	 <32120a6a0907030700i12876a6dld5a6b6b3c76b30f9@mail.gmail.com>
	 <15da8a100907031104s5fc40245tca4ba09cb65e60e@mail.gmail.com>
Date: Fri, 3 Jul 2009 15:01:37 -0700
Message-ID: <605941800907031501l756e3db5ud88da9f864f4de6e@mail.gmail.com>
Subject: Re: Tools to help administer a cluster?
From: RSD <ross.dmochowski@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

I am in the Technical Operations Dept at Hi5, and recently given responsibility
for service delivery in regards to our Hadoop installation.

I highly recommend investing the time with CFEngine, as it is a mature
robust configuration management framework, and really what you
want for anything more sophisticated than a simple one or two box
dev instance.

From general-return-333-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 04 03:40:13 2009
Return-Path: <general-return-333-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 32133 invoked from network); 4 Jul 2009 03:40:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Jul 2009 03:40:13 -0000
Received: (qmail 24945 invoked by uid 500); 4 Jul 2009 03:40:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24797 invoked by uid 500); 4 Jul 2009 03:40:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24787 invoked by uid 99); 4 Jul 2009 03:40:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jul 2009 03:40:21 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bogdan.maryniuk@gmail.com designates 209.85.222.196 as permitted sender)
Received: from [209.85.222.196] (HELO mail-pz0-f196.google.com) (209.85.222.196)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jul 2009 03:40:10 +0000
Received: by pzk34 with SMTP id 34so2449533pzk.5
        for <general@hadoop.apache.org>; Fri, 03 Jul 2009 20:39:50 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=6ZpyywOLFlu2oxnXA/uX9HYBqj0tKcQE1itvG1lAC1A=;
        b=Rc9Nv/EGQ8D9SbznBAv9eH4SY7QdKBbunsUHL1n/AubkpPj6/rzCWx5VFHatsUtiuy
         9SZKmqrAJZ7ofNsGRQUmZuC9qsPNTmPDazIyXvYytkkzGj1ueEpUHy3e+ymZaPyKv/8G
         8Z7NTtjCerV40TqFK53cqb1PRjV+VnTun+Dv8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=bu/FEHJe/Pgl7PX0FvyX8OYCxz100cIFZJQNSAS6+/kgotvJcF8Lvu+RFIwpSy/xlz
         13DynrL9/cCwq7XzdJP+PQHmUI3/obzQBp1APM1s6wANPn2bQlvIwxJI0iqFGHW+7y51
         zeE5pKzk4V5H8NV8NBzRVAN9qxCCmha8Azxl8=
MIME-Version: 1.0
Received: by 10.142.238.9 with SMTP id l9mr734011wfh.90.1246678790592; Fri, 03 
	Jul 2009 20:39:50 -0700 (PDT)
In-Reply-To: <605941800907031501l756e3db5ud88da9f864f4de6e@mail.gmail.com>
References: <32120a6a0907030206l77a30d41yef2704e176f6d90f@mail.gmail.com>
	 <fa561940907030508h19139e87g4393484bd5e0077e@mail.gmail.com>
	 <32120a6a0907030700i12876a6dld5a6b6b3c76b30f9@mail.gmail.com>
	 <15da8a100907031104s5fc40245tca4ba09cb65e60e@mail.gmail.com>
	 <605941800907031501l756e3db5ud88da9f864f4de6e@mail.gmail.com>
Date: Sat, 4 Jul 2009 12:39:50 +0900
Message-ID: <fa561940907032039k613d0103v36ffd1f0d2ce03e1@mail.gmail.com>
Subject: Re: Tools to help administer a cluster?
From: "Bogdan M. Maryniuk" <bogdan.maryniuk@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On Sat, Jul 4, 2009 at 7:01 AM, RSD<ross.dmochowski@gmail.com> wrote:
> I highly recommend investing the time with CFEngine

Not sure how CFEngine can be better than simple script in bash for
administering Hadoop in this case. But I am sure how it might be a
disaster when you forget e.g. a semicolon in some certain place and
run it.


On Sat, Jul 4, 2009 at 3:04 AM, Philip Zeyliger<philip@cloudera.com> wrote:
> We build (and use) RPM & deb packages for Hadoop.

In that case Tim has to decide messing up with
http://rpm4darwin.sourceforge.net/ (read: seriously affect his
infrastructure with rpm'ed Darwin ports) or not. I am not sure how
your Linux-only packages will work on "rpmized" OSX either.

> See http://cloudera.com/hadoop .  You can package up your config files
> into an RPM too (see my.cloudera.com) or use rsync for those, and you
> can distribute the binaries by pointing your machines to a yum/apt
> repository.

Any plans to support IPS (http://opensolaris.org/os/project/pkg/)?
C'mon, guys, Java on Solaris is still better and ZFS has much more
advantage over standard POSIX bridges that are available on Linux at
the moment. :-) I've got running Hadoop on Solaris and its zones
finally good: just for some reason Hadoop disliked ZFS rpool within a
zone, but asks for a real device mounted. Haven't figured out it yet
by reading sources, since I am too new to Hadoop, but that's was the
only issue so far (0.20 and 0.19.1 versions tried).

--
BM

From general-return-334-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 04 04:52:47 2009
Return-Path: <general-return-334-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43887 invoked from network); 4 Jul 2009 04:52:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Jul 2009 04:52:47 -0000
Received: (qmail 51729 invoked by uid 500); 4 Jul 2009 04:52:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51629 invoked by uid 500); 4 Jul 2009 04:52:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51619 invoked by uid 99); 4 Jul 2009 04:52:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jul 2009 04:52:55 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of bogdan.maryniuk@gmail.com designates 209.85.216.171 as permitted sender)
Received: from [209.85.216.171] (HELO mail-px0-f171.google.com) (209.85.216.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jul 2009 04:52:46 +0000
Received: by pxi1 with SMTP id 1so2409100pxi.5
        for <general@hadoop.apache.org>; Fri, 03 Jul 2009 21:52:25 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=21bj28juxk3JR6mOZW0RUMIOQrzbHDCe1xzzQ9moWOk=;
        b=LfTcyFoGdPELHvM4fPKwqXjLzPq2pxZQ48zYmMnQUag0egDLYZGHP9wKDBbcMcliRe
         szYgQnXIxSkHjiSJRZLidDTJepFm8R+ZFSE+1gYOquJkgOlSVaFG2LIJGayCwsIuU7tw
         7sR/B8wmN3ZGN8aIll3+/xhS7kM0fNyrCgqx8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=ObDSSEzBloVOBM7GU3nddHc2Ivg+M8crNTcIpg3d9nExrfXLywjYq8/UAt/nZYHs+b
         eC+B9n6vEKoaGxc1yvt5BVlt5pvAhkVhhrGNAohVKxc8slMedk96/jFVc7+U+KqCkr2c
         eu0hm+igaheILHowg2+4QuCC0mVIxOib8YD1o=
MIME-Version: 1.0
Received: by 10.142.125.8 with SMTP id x8mr715675wfc.44.1246683145464; Fri, 03 
	Jul 2009 21:52:25 -0700 (PDT)
In-Reply-To: <aa41f20e0907030517g3f2889d0u4d1ab2584de15bf8@mail.gmail.com>
References: <aa41f20e0907030221k6ea5b2ffk3a5aade3b382a9d7@mail.gmail.com>
	 <32120a6a0907030226m28d6bfc9w1b6af48f8db42347@mail.gmail.com>
	 <aa41f20e0907030517g3f2889d0u4d1ab2584de15bf8@mail.gmail.com>
Date: Sat, 4 Jul 2009 13:52:25 +0900
Message-ID: <fa561940907032152j251791b5u24687d7a053c90f3@mail.gmail.com>
Subject: Re: My secondary namenode seem not be running, and may be the reason 
	of my problem!!!
From: "Bogdan M. Maryniuk" <bogdan.maryniuk@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On Fri, Jul 3, 2009 at 9:17 PM, C J<xine.jar@googlemail.com> wrote:
> *Can anyone tell me why the namenode and the jobtracker were never started?

How about providing here recent information from $HADOOP/logs/ ?..

-- 
Kind regards, BM

Things, that are stupid at the beginning, rarely ends up wisely.

From general-return-335-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 04 17:20:55 2009
Return-Path: <general-return-335-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45667 invoked from network); 4 Jul 2009 17:20:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Jul 2009 17:20:55 -0000
Received: (qmail 73904 invoked by uid 500); 4 Jul 2009 17:21:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73815 invoked by uid 500); 4 Jul 2009 17:21:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73805 invoked by uid 99); 4 Jul 2009 17:21:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jul 2009 17:21:04 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cutting@gmail.com designates 209.85.217.215 as permitted sender)
Received: from [209.85.217.215] (HELO mail-gx0-f215.google.com) (209.85.217.215)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jul 2009 17:20:55 +0000
Received: by gxk11 with SMTP id 11so4180814gxk.5
        for <general@hadoop.apache.org>; Sat, 04 Jul 2009 10:20:34 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:sender:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=QdgDjOmAt8Wor8e8nuBOwYLRyzFJnhp2r1h8gm/eyGI=;
        b=OvIhzuHv5Hu00Fl9xOtmSzM6BWJ7c8rRqjaJvNMdzN7Jwmw2fTg2RmnJVCmcr+GJt1
         7sq9Tv7lb7eHCfxbnUKNWwnp88thb43oPs7kEn4BQvJEq69WU06JX+7haeF0xxTQ1snZ
         TOtID9Dn5rjX+ycxUMLq8cJO9WFyvu+vYmDZo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:date:from:user-agent:mime-version:to:subject
         :references:in-reply-to:content-type:content-transfer-encoding;
        b=EVaZvCCrKQ3r9oaGoziPdCvvAaYlW6GbJ62AWc0soYIeHPKCbeBzyV3a+C00cLZeB2
         Qx8pefLlLXXcmFemPe6qwfFbiS7MjerFLjC9Z/UMM6RWQsev6rlVKjOOXXO0ClN47j7p
         2IuEBE62N4MnBMBiGoVy2ON02OSQcsTNkdLKw=
Received: by 10.90.68.20 with SMTP id q20mr2248640aga.104.1246728034716;
        Sat, 04 Jul 2009 10:20:34 -0700 (PDT)
Received: from ?192.168.1.26? (c-67-175-38-73.hsd1.in.comcast.net [67.175.38.73])
        by mx.google.com with ESMTPS id 38sm8810856aga.1.2009.07.04.10.20.33
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Sat, 04 Jul 2009 10:20:33 -0700 (PDT)
Sender: Doug Cutting <cutting@gmail.com>
Message-ID: <4A4F8F60.9030607@apache.org>
Date: Sat, 04 Jul 2009 10:20:32 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.22 (X11/20090608)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Combiner Problem
References: <396ceda90907022144p3b66a345g1b3b52a1ac5b7969@mail.gmail.com>
In-Reply-To: <396ceda90907022144p3b66a345g1b3b52a1ac5b7969@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
X-Virus-Checked: Checked by ClamAV on apache.org

This question would be more appropriate on the mapreduce-users at 
hadoop.apache.org mailing list.

Unfortunately Hadoop's website is out of date right now and does not 
describe the currently correct mailing lists.  Is someone working to fix 
this?  We do not want general@ to become a place for end-user questions.

Doug

乔木 wrote:
> Hi, everyone
> 
> I've been learning hadoop recently and I'm confused about the combiner
> mechanism.
> 
> There is a property min.num.spills.for.combine specifying the minimum number
> of spills to run combiner when merging. The default value is 3. Why there is
> such a restriction? Should it be better that run the combiner no matter how
> many spills there are?
> 
> The second question is why the combiner could be run at the reduce side.
> Can't the reduce function take place of that?
> 
> Thanks very much.

From general-return-336-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jul 05 18:40:14 2009
Return-Path: <general-return-336-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94183 invoked from network); 5 Jul 2009 18:40:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Jul 2009 18:40:13 -0000
Received: (qmail 65576 invoked by uid 500); 5 Jul 2009 18:40:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65474 invoked by uid 500); 5 Jul 2009 18:40:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65464 invoked by uid 99); 5 Jul 2009 18:40:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jul 2009 18:40:22 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of xine.jar@googlemail.com designates 209.85.210.185 as permitted sender)
Received: from [209.85.210.185] (HELO mail-yx0-f185.google.com) (209.85.210.185)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jul 2009 18:40:11 +0000
Received: by yxe15 with SMTP id 15so2753564yxe.5
        for <general@hadoop.apache.org>; Sun, 05 Jul 2009 11:39:50 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=+00XRZ2D7msAU1Fgz7h+6du9rIeE9idgfIcHexoQTyc=;
        b=jv8wf0ycPoU3s7qD6LKy6DvHMy3ol7fqoYtGHN7hYbQcuB3brD5Kk/UGfI+HIsDOZl
         +rXOBDJ+KNcZQ1YZtN9yV9BsKFxdv/qTo5s5fbhmLiLQDffoUhwo5MIWxUC47D9W4646
         5Da1+M1C/EBNOIlNK5gh3L4z9cb0AetcckezM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=Yeqbr2EOuN+RQ2yfZsz5cf8FFn4q3/xPU6llgtHn9hvWamADpFo9RojMT7ocRr/qW7
         04xAKJFvgIRu2vszN6ivRcAfJRk7smjbzoMDptITpaLt8zlghEe+PppPpK+up41I5IJl
         /URO+ShcnR2QwBls3xYXGwV0UWgAX7dL5z2n0=
MIME-Version: 1.0
Received: by 10.231.10.137 with SMTP id p9mr30518ibp.52.1246819189901; Sun, 05 
	Jul 2009 11:39:49 -0700 (PDT)
Date: Sun, 5 Jul 2009 20:39:49 +0200
Message-ID: <aa41f20e0907051139v6078eea6web824de895bd4a7c@mail.gmail.com>
Subject: sending you an email is incredible!!!
From: C J <xine.jar@googlemail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0022152d60f9e9c88b046df9b6b6
X-Virus-Checked: Checked by ClamAV on apache.org

--0022152d60f9e9c88b046df9b6b6
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

It is really icredible,
If I reply to one of your emails the email cannot be sent, and if create a
new thread
the email cannot be send as well. I am completly blocked, with my hadoop and
with communicating
with people who are trying to help as well.
What shall I do?

--0022152d60f9e9c88b046df9b6b6--

From general-return-337-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jul 05 18:58:48 2009
Return-Path: <general-return-337-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96246 invoked from network); 5 Jul 2009 18:58:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Jul 2009 18:58:48 -0000
Received: (qmail 77172 invoked by uid 500); 5 Jul 2009 18:58:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77100 invoked by uid 500); 5 Jul 2009 18:58:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77090 invoked by uid 99); 5 Jul 2009 18:58:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jul 2009 18:58:58 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of timrobertson100@gmail.com designates 209.85.219.219 as permitted sender)
Received: from [209.85.219.219] (HELO mail-ew0-f219.google.com) (209.85.219.219)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jul 2009 18:58:48 +0000
Received: by ewy19 with SMTP id 19so4023767ewy.29
        for <general@hadoop.apache.org>; Sun, 05 Jul 2009 11:58:28 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=cGi6UlLaAMFYkss3mPG/PG09N3sIso105YITZ8bxWGY=;
        b=OdLcXcHZtY5P4GsxIVsAD5F56bo61nmoRPrUcJb9IzhYPfixk9iUhWo5fxNBSHM1Sp
         iqbdoq03zBy4S3TLCX+9W/EXqmOQzd1qOMAQgbE4wlPx8rWIZonrRmiLs3Si1AvpvL7q
         iAVpoyewEoLpDARLkbpllXXY3Rpe3bPFSzZ9k=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=xgkYVv3IoLknN9e4KSIfgwqIOW4mmC6Aq2an3NBJ72pgIu2ko3cwk2bflKRDdCQHJZ
         q43cHnEGL+D0mQhfc3VHdKfO3yxhh7eO+xrsMo+OjiPUknK7i26Mw08IcWRGoV8xev7+
         HOaKOOjZ0FXHYjVg0zox62cbODsFY2xYJdcgo=
MIME-Version: 1.0
Received: by 10.216.29.72 with SMTP id h50mr930377wea.137.1246820308549; Sun, 
	05 Jul 2009 11:58:28 -0700 (PDT)
In-Reply-To: <aa41f20e0907051139v6078eea6web824de895bd4a7c@mail.gmail.com>
References: <aa41f20e0907051139v6078eea6web824de895bd4a7c@mail.gmail.com>
Date: Sun, 5 Jul 2009 20:58:28 +0200
Message-ID: <32120a6a0907051158k39976c2bi19963fa655ef158a@mail.gmail.com>
Subject: Re: sending you an email is incredible!!!
From: tim robertson <timrobertson100@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hi CJ,

Certainly some of your mails are coming through ok as we got this.

You weren't blocked when you had your recent thread:
http://mail-archives.apache.org/mod_mbox/hadoop-general/200907.mbox/%3Caa41f20e0907030517g3f2889d0u4d1ab2584de15bf8@mail.gmail.com%3E

Perhaps it is you email provider itself?  Try a different account perhaps?
Are you trying to send messages to lists that you aren't a member of
maybe? - you need to join each list seperately (hadoop-core,
hadoop-mapreduce etc)

Cheers,

Tim


On Sun, Jul 5, 2009 at 8:39 PM, C J<xine.jar@googlemail.com> wrote:
> It is really icredible,
> If I reply to one of your emails the email cannot be sent, and if create a
> new thread
> the email cannot be send as well. I am completly blocked, with my hadoop and
> with communicating
> with people who are trying to help as well.
> What shall I do?
>

From general-return-338-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jul 05 19:01:54 2009
Return-Path: <general-return-338-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96811 invoked from network); 5 Jul 2009 19:01:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Jul 2009 19:01:54 -0000
Received: (qmail 77919 invoked by uid 500); 5 Jul 2009 19:02:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77831 invoked by uid 500); 5 Jul 2009 19:02:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77821 invoked by uid 99); 5 Jul 2009 19:02:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jul 2009 19:02:04 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [76.13.9.59] (HELO web65515.mail.ac4.yahoo.com) (76.13.9.59)
    by apache.org (qpsmtpd/0.29) with SMTP; Sun, 05 Jul 2009 19:01:53 +0000
Received: (qmail 66630 invoked by uid 60001); 5 Jul 2009 19:01:30 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1246820490; bh=Q8ET9UoeJIT+9LsgSIDSVvV2JKagX+gXsRboCsoFP+8=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=lK3TX5jp+XBywqVJ9xvf7sWAnS26buIAPvRVMsDC5rG5m59dOiSu/LOOATlU713oQKP82XNdOnfl+7piClQaQhr6bE+gYmlIbi4873KDYsz0Bc8Fbk8jTMU4KZjLySieiGS0zaYD17/5FKUZk7khQwiuV48MzJqAynWT4xsLDH8=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=iJvR39vp23UMDJZajhGobUv182wF5/kjkKSPyiMJLv0UXRjGT7bzccxm+jfmdyLKbHOkkWj3jnwJCUaIOpHq5/PVUCIyYQpl9DmZQ97oNpa54vjdya25dE1Vs69Um1pSCTOivimOobc9IyGgHqvs5Eb6jWJlgzNKFVwd0nBS91E=;
Message-ID: <798259.44995.qm@web65515.mail.ac4.yahoo.com>
X-YMail-OSG: CvfbqpgVM1kJMuZwH2n7B.K3yVy01l.b5hECWMhjb.2JzFG6i4WqK0sUE504hds2.9ESetWD7wnFYrLLEAVXqtsvHx_CEkeCRfByElSiLsVCamgHZ2.rJavDyzSyjdV7bxxeC6jrdC4Un8hitwvNk1S4vDxkMRnK3cyv0.OMly1uro2oyveLxRIwLQ1tNKWG0c9FwQAKh3YU4ftqASwJxOz.Ox_ALinnRt6AxxxL4cscqh2vQYW42J.bP_g_GLHt2XzzT9i38kzI81xab9qQdCqJdFRG980FcRx0Fdbf0tSoyyApGKbC_8xuJuWdulXvlsCmsxflUTzck5PFhg--
Received: from [87.248.121.240] by web65515.mail.ac4.yahoo.com via HTTP; Sun, 05 Jul 2009 12:01:30 PDT
X-Mailer: YahooMailWebService/0.7.289.15
Date: Sun, 5 Jul 2009 12:01:30 -0700 (PDT)
From: Chad Walters <chad_walters@yahoo.com>
Subject: Re: sending you an email is incredible!!!
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org


The problem is likely related to the period in your user name. I've found that the Apache mailing list system has a lot of issues with that. Try using an alternate address without a period in the user name.

Chad

Sent from my iPhone

On Jul 5, 2009, at 2:39 PM, C J <xine.jar@googlemail.com> wrote:

It is really icredible,
If I reply to one of your emails the email cannot be sent, and if create a
new thread
the email cannot be send as well. I am completly blocked, with my hadoop and
with communicating
with people who are trying to help as well.
What shall I do?


From general-return-339-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jul 05 19:09:13 2009
Return-Path: <general-return-339-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98016 invoked from network); 5 Jul 2009 19:09:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Jul 2009 19:09:13 -0000
Received: (qmail 83247 invoked by uid 500); 5 Jul 2009 19:09:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83158 invoked by uid 500); 5 Jul 2009 19:09:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83148 invoked by uid 99); 5 Jul 2009 19:09:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jul 2009 19:09:23 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of xine.jar@googlemail.com designates 209.85.210.185 as permitted sender)
Received: from [209.85.210.185] (HELO mail-yx0-f185.google.com) (209.85.210.185)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jul 2009 19:09:12 +0000
Received: by yxe15 with SMTP id 15so2767790yxe.5
        for <general@hadoop.apache.org>; Sun, 05 Jul 2009 12:08:52 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=JqqXawW4CWEKMu8+duTGuQqU7wCpUrMMQzU5EoQZtVg=;
        b=VkM/paG5DdtBSraBqq8l9yeXefQe8ywecXfx888bNJorsJkqMF0loQul+cCL9HhQa7
         jdFs38sLrpFrT1mDabtIUaZKAGUo+w2W7N8EM7CKO2g2epZl3JQpbFvXS6W6W3Vu9QRf
         xeI+c/d8N+9XXyEy7aGkr2iMv9hkYtqBF+Lpo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=lDqwV+atu52T5ffjvWvPpz+krcTALjywDspklecRrPO/IgDQPFEPWnxZ/ERfRdmO5M
         BxVKld0dNHssu6JMceRZ/o2aJbrDe65lk9ybrLRbT9wj6pFYZHhRs9aqPj42ABRxnpoZ
         GYuHIWroRce4K1LSi7AnEHTf3ask+5fFWaCQ8=
MIME-Version: 1.0
Received: by 10.231.32.11 with SMTP id a11mr1835829ibd.38.1246820931672; Sun, 
	05 Jul 2009 12:08:51 -0700 (PDT)
In-Reply-To: <32120a6a0907051158k39976c2bi19963fa655ef158a@mail.gmail.com>
References: <aa41f20e0907051139v6078eea6web824de895bd4a7c@mail.gmail.com>
	 <32120a6a0907051158k39976c2bi19963fa655ef158a@mail.gmail.com>
Date: Sun, 5 Jul 2009 21:08:51 +0200
Message-ID: <aa41f20e0907051208tb14b964v96e0febb59795be5@mail.gmail.com>
Subject: Re: sending you an email is incredible!!!
From: C J <xine.jar@googlemail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0022152d6d75bb1b18046dfa1e50
X-Virus-Checked: Checked by ClamAV on apache.org

--0022152d6d75bb1b18046dfa1e50
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Basically what I have done, is that I registered myself three weeks ago
to the list you have on the website core-user@hadoop.apache.org.
(which is by the way not working now) I got an automaticd reply for the
registration.
Now what shall I do? there is as far as I know no other list on the website?

Shall I send an email to general@hadoop.apache.org and another one to
common-user@hadoop.apache.org
and ask that they register me?
or could you please register me to theses lists at least and send me a
confirmation email?
I would really appreciate this, then I will try afterwards to anwer your
emails and send you my log files.

Thank zou tim,


On Sun, Jul 5, 2009 at 8:58 PM, tim robertson <timrobertson100@gmail.com>wrote:

> Hi CJ,
>
> Certainly some of your mails are coming through ok as we got this.
>
> You weren't blocked when you had your recent thread:
>
> http://mail-archives.apache.org/mod_mbox/hadoop-general/200907.mbox/%3Caa41f20e0907030517g3f2889d0u4d1ab2584de15bf8@mail.gmail.com%3E
>
> Perhaps it is you email provider itself?  Try a different account perhaps?
> Are you trying to send messages to lists that you aren't a member of
> maybe? - you need to join each list seperately (hadoop-core,
> hadoop-mapreduce etc)
>
> Cheers,
>
> Tim
>
>
> On Sun, Jul 5, 2009 at 8:39 PM, C J<xine.jar@googlemail.com> wrote:
> > It is really icredible,
> > If I reply to one of your emails the email cannot be sent, and if create
> a
> > new thread
> > the email cannot be send as well. I am completly blocked, with my hadoop
> and
> > with communicating
> > with people who are trying to help as well.
> > What shall I do?
> >
>

--0022152d6d75bb1b18046dfa1e50--

From general-return-340-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jul 05 19:11:08 2009
Return-Path: <general-return-340-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98369 invoked from network); 5 Jul 2009 19:11:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Jul 2009 19:11:07 -0000
Received: (qmail 84103 invoked by uid 500); 5 Jul 2009 19:11:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84017 invoked by uid 500); 5 Jul 2009 19:11:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84007 invoked by uid 99); 5 Jul 2009 19:11:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jul 2009 19:11:16 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of xine.jar@googlemail.com designates 209.85.217.215 as permitted sender)
Received: from [209.85.217.215] (HELO mail-gx0-f215.google.com) (209.85.217.215)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jul 2009 19:11:06 +0000
Received: by gxk11 with SMTP id 11so4854999gxk.5
        for <general@hadoop.apache.org>; Sun, 05 Jul 2009 12:10:45 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=bc5vDRpdM4UlO6izjiF2pKcPdOZRSPDJ1jFVCok6auM=;
        b=buky+kHgUFeRW4NLeui1R7w/j/hHha92QuL76LM9WhPlXfOJ7cOug2sX0BvmYTyWUM
         /mmx/JvfU/lt53vTmy4VUff6l637bYjxRP8U337CvHuEhyjhqkXiKNNGemsrxC91BvwJ
         gmyIEJpblw1DOx8DwhTJ1wCMePtvQ2Fs0mNoQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=GGzcbEyDU0oP+gDT0xkRHcgCYZrL8iQmJWvkfUDV+9g2IJS/+anZSqyrzR41XHd0YV
         qFUFvl9um7ByFst+pGzAK2WZ+v4269QGdjw7v0jSkfhbSwm2LbXWbx4biX2dWQzwDvay
         rtimWi/CooYrrKao2vY2GjiDPWvQ69C7NN9G8=
MIME-Version: 1.0
Received: by 10.231.14.4 with SMTP id e4mr1831210iba.33.1246821045567; Sun, 05 
	Jul 2009 12:10:45 -0700 (PDT)
In-Reply-To: <798259.44995.qm@web65515.mail.ac4.yahoo.com>
References: <798259.44995.qm@web65515.mail.ac4.yahoo.com>
Date: Sun, 5 Jul 2009 21:10:45 +0200
Message-ID: <aa41f20e0907051210w4cab8f75s349aff88bf6d1106@mail.gmail.com>
Subject: Re: sending you an email is incredible!!!
From: C J <xine.jar@googlemail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00221532c7a4850065046dfa2509
X-Virus-Checked: Checked by ClamAV on apache.org

--00221532c7a4850065046dfa2509
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Really?
Ok then I will try to create the same account without the period and try
again.
Thank you

On Sun, Jul 5, 2009 at 9:01 PM, Chad Walters <chad_walters@yahoo.com> wrote:

>
> The problem is likely related to the period in your user name. I've found
> that the Apache mailing list system has a lot of issues with that. Try using
> an alternate address without a period in the user name.
>
> Chad
>
> Sent from my iPhone
>
> On Jul 5, 2009, at 2:39 PM, C J <xine.jar@googlemail.com> wrote:
>
> It is really icredible,
> If I reply to one of your emails the email cannot be sent, and if create a
> new thread
> the email cannot be send as well. I am completly blocked, with my hadoop
> and
> with communicating
> with people who are trying to help as well.
> What shall I do?
>
>

--00221532c7a4850065046dfa2509--

From general-return-341-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jul 05 19:25:49 2009
Return-Path: <general-return-341-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1689 invoked from network); 5 Jul 2009 19:25:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Jul 2009 19:25:49 -0000
Received: (qmail 89942 invoked by uid 500); 5 Jul 2009 19:25:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89859 invoked by uid 500); 5 Jul 2009 19:25:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89849 invoked by uid 99); 5 Jul 2009 19:25:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jul 2009 19:25:59 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of timrobertson100@gmail.com designates 209.85.219.219 as permitted sender)
Received: from [209.85.219.219] (HELO mail-ew0-f219.google.com) (209.85.219.219)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jul 2009 19:25:49 +0000
Received: by ewy19 with SMTP id 19so4037004ewy.29
        for <general@hadoop.apache.org>; Sun, 05 Jul 2009 12:25:29 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=mi1v6zXaznuV9eIJzk1SnTA41xUedqtfEjge+CZmUVc=;
        b=MJBmPRvEDhJBFt7lXqDXcjZ64CizWWSZcCmuzy1g1FCPi+x/DIE0FIbw7ok3u28J53
         ii9tdasUCv5dtMWP0BJFXzhOf54pHMa4rQiAlPQa4mi/8b2z8DTnwKZyStrxeC/9f1Bv
         t2lYAIY0bpKosnWU81PkfDsueiWnjWiwj6RIQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=V2LuVscHdhWiRq2GUuImCyxcJcLvSTohqOGr8sx8pAzuFjjyX/o+mpGC/96HZR23KE
         jgfB4jG83uC7bT4OtlM8nIWL64u0CTiC2qauBggQ4F6D75oclfuIB6iJu/NODNzYAsIh
         6vLRVeg/+xLSgbJP11C4+wDvgAua/EVT9UGLw=
MIME-Version: 1.0
Received: by 10.216.29.72 with SMTP id h50mr935805wea.137.1246821929493; Sun, 
	05 Jul 2009 12:25:29 -0700 (PDT)
In-Reply-To: <aa41f20e0907051210w4cab8f75s349aff88bf6d1106@mail.gmail.com>
References: <798259.44995.qm@web65515.mail.ac4.yahoo.com>
	 <aa41f20e0907051210w4cab8f75s349aff88bf6d1106@mail.gmail.com>
Date: Sun, 5 Jul 2009 21:25:29 +0200
Message-ID: <32120a6a0907051225k318f5b4ek15000df6fb06a5aa@mail.gmail.com>
Subject: Re: sending you an email is incredible!!!
From: tim robertson <timrobertson100@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

To subscribe CJ, just send a mail to:

common-user-subscribe@hadoop.apache.org  (this is the old core-user renamed)
mapreduce-user-subscribe@hadoop.apache.org
hdfs-user-subscribe@hadoop.apache.org

general-subscribe@hadoop.apache.org (you must have already done this)

Cheers

Tim


On Sun, Jul 5, 2009 at 9:10 PM, C J<xine.jar@googlemail.com> wrote:
> Really?
> Ok then I will try to create the same account without the period and try
> again.
> Thank you
>
> On Sun, Jul 5, 2009 at 9:01 PM, Chad Walters <chad_walters@yahoo.com> wrote:
>
>>
>> The problem is likely related to the period in your user name. I've found
>> that the Apache mailing list system has a lot of issues with that. Try using
>> an alternate address without a period in the user name.
>>
>> Chad
>>
>> Sent from my iPhone
>>
>> On Jul 5, 2009, at 2:39 PM, C J <xine.jar@googlemail.com> wrote:
>>
>> It is really icredible,
>> If I reply to one of your emails the email cannot be sent, and if create a
>> new thread
>> the email cannot be send as well. I am completly blocked, with my hadoop
>> and
>> with communicating
>> with people who are trying to help as well.
>> What shall I do?
>>
>>
>

From general-return-342-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jul 05 19:37:32 2009
Return-Path: <general-return-342-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 3868 invoked from network); 5 Jul 2009 19:37:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Jul 2009 19:37:32 -0000
Received: (qmail 93770 invoked by uid 500); 5 Jul 2009 19:37:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93700 invoked by uid 500); 5 Jul 2009 19:37:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93690 invoked by uid 99); 5 Jul 2009 19:37:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jul 2009 19:37:42 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of xine.jar@googlemail.com designates 209.85.210.185 as permitted sender)
Received: from [209.85.210.185] (HELO mail-yx0-f185.google.com) (209.85.210.185)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jul 2009 19:37:30 +0000
Received: by yxe15 with SMTP id 15so2781704yxe.5
        for <general@hadoop.apache.org>; Sun, 05 Jul 2009 12:37:09 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=Y61E7hkPluHshKB66ddIhahER/NOyulE1sGK0Er9fb8=;
        b=lZfcP6LaNo12FeTWnmd35q9MrJIoSLuqDdykPoyNfx86665mVvbtde5w/Rsqtqb9Gx
         rFPbHUniLuAblkGMdc2BzSBKyblleG+OClVj9T93cYQTcefousbA/lxIeRds0BwTh5bp
         OFi2E5bO79BjUn//xqDFvF84SolSHUbjr8+tc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=lHJVd5WNWscivluNxOQ/2Jj0UjroW8Fd811Pwu1THNOqfcy76WNzIuer1KVYz00M0k
         TW6WFndyfzKzDV6RYRVBB5Q/Kysw8RGfviICglf+TYfTvDjQBquvkmjg2alqo5Kto7Gn
         phDugbHj90vo2eYDbOK3KXGVeHBYL8IuEgJOc=
MIME-Version: 1.0
Received: by 10.231.13.201 with SMTP id d9mr1822287iba.35.1246822629225; Sun, 
	05 Jul 2009 12:37:09 -0700 (PDT)
In-Reply-To: <32120a6a0907051225k318f5b4ek15000df6fb06a5aa@mail.gmail.com>
References: <798259.44995.qm@web65515.mail.ac4.yahoo.com>
	 <aa41f20e0907051210w4cab8f75s349aff88bf6d1106@mail.gmail.com>
	 <32120a6a0907051225k318f5b4ek15000df6fb06a5aa@mail.gmail.com>
Date: Sun, 5 Jul 2009 21:37:09 +0200
Message-ID: <aa41f20e0907051237l79f48ca7v1d2e87058b301e37@mail.gmail.com>
Subject: Re: sending you an email is incredible!!!
From: C J <xine.jar@googlemail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000325575fdee9b58d046dfa83d8
X-Virus-Checked: Checked by ClamAV on apache.org

--000325575fdee9b58d046dfa83d8
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Ok I have registered to the two missing email lists: mapreduce...and
hdfs....
I shall try to send my email problem to one of those if not, I will try
again from another
email address without a "dot" in the middle of the username.

Many thanks

On Sun, Jul 5, 2009 at 9:25 PM, tim robertson <timrobertson100@gmail.com>wrote:

> To subscribe CJ, just send a mail to:
>
> common-user-subscribe@hadoop.apache.org  (this is the old core-user
> renamed)
> mapreduce-user-subscribe@hadoop.apache.org
> hdfs-user-subscribe@hadoop.apache.org
>
> general-subscribe@hadoop.apache.org (you must have already done this)
>
> Cheers
>
> Tim
>
>
> On Sun, Jul 5, 2009 at 9:10 PM, C J<xine.jar@googlemail.com> wrote:
> > Really?
> > Ok then I will try to create the same account without the period and try
> > again.
> > Thank you
> >
> > On Sun, Jul 5, 2009 at 9:01 PM, Chad Walters <chad_walters@yahoo.com>
> wrote:
> >
> >>
> >> The problem is likely related to the period in your user name. I've
> found
> >> that the Apache mailing list system has a lot of issues with that. Try
> using
> >> an alternate address without a period in the user name.
> >>
> >> Chad
> >>
> >> Sent from my iPhone
> >>
> >> On Jul 5, 2009, at 2:39 PM, C J <xine.jar@googlemail.com> wrote:
> >>
> >> It is really icredible,
> >> If I reply to one of your emails the email cannot be sent, and if create
> a
> >> new thread
> >> the email cannot be send as well. I am completly blocked, with my hadoop
> >> and
> >> with communicating
> >> with people who are trying to help as well.
> >> What shall I do?
> >>
> >>
> >
>

--000325575fdee9b58d046dfa83d8--

From general-return-343-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 06 05:13:30 2009
Return-Path: <general-return-343-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70495 invoked from network); 6 Jul 2009 05:13:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jul 2009 05:13:30 -0000
Received: (qmail 68530 invoked by uid 500); 6 Jul 2009 05:13:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68432 invoked by uid 500); 6 Jul 2009 05:13:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68422 invoked by uid 99); 6 Jul 2009 05:13:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jul 2009 05:13:38 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of fredwang222@gmail.com designates 209.85.212.172 as permitted sender)
Received: from [209.85.212.172] (HELO mail-vw0-f172.google.com) (209.85.212.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jul 2009 05:13:26 +0000
Received: by vwj2 with SMTP id 2so2894512vwj.5
        for <general@hadoop.apache.org>; Sun, 05 Jul 2009 22:13:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=42wwpziNfDjWQ9THlwlADceUGkOAfFKSBOkUi5Si2fw=;
        b=d6gaVkJhVfEh+ToP/ZPDKGaTKGpWIb8GjDNli89rb0/qDpuG5adp1o04xXtiIP7d12
         KaTwWJTPbGXGFwkB+ERkXsOScbT/svPW7Mkn82X7qEe0qnXaa35cDcQZiDodkpeo4VDg
         DzE3l8brkhcz6pw3yWZL3yPrO+AY0AkMqcLeQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=vYcqm7tVJxIR2iwquluLd6a3AXu3mgp++0qmWvyhJhFQKaojUbM+y6jxX7WqhqEz+Z
         e35Gf48l4KEJ2bttkEGT2ces5h1BCS5z37o4izmPDbloEBBg/1l+TWoizgvoujNZCdOv
         thqGFAWYSkPlqmNgz1NDrCL/donyx8VoU3q2M=
MIME-Version: 1.0
Received: by 10.220.46.20 with SMTP id h20mr8705545vcf.55.1246857185013; Sun, 
	05 Jul 2009 22:13:05 -0700 (PDT)
In-Reply-To: <32120a6a0907030745g19ef2593o457f48972ef51ea@mail.gmail.com>
References: <702261350907010735u3502197au800281a5a131d963@mail.gmail.com>
	 <4A4B8B99.10303@yahoo-inc.com>
	 <702261350907012211r17dec368u17e18052383a8822@mail.gmail.com>
	 <4A4C430F.7080306@yahoo-inc.com>
	 <702261350907012251s41b18b24m5dc54015007d494e@mail.gmail.com>
	 <702261350907012309l4c62885bk6df2ebac8e2b7254@mail.gmail.com>
	 <4A4CDF0F.2040106@yahoo-inc.com>
	 <702261350907030731w694cd855safc8b3e5b5048747@mail.gmail.com>
	 <32120a6a0907030745g19ef2593o457f48972ef51ea@mail.gmail.com>
Date: Mon, 6 Jul 2009 13:13:04 +0800
Message-ID: <702261350907052213u6b13190ap35ebc1df2c206d3b@mail.gmail.com>
Subject: Re: fail to setup passphraseless ssh and need some help
From: fred wang <fredwang222@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364ecd80993093046e028f26
X-Virus-Checked: Checked by ClamAV on apache.org

--0016364ecd80993093046e028f26
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

it doesn't work..

On Fri, Jul 3, 2009 at 10:45 PM, tim robertson <timrobertson100@gmail.com>wrote:

> This seemed to me to be a good resource (e.g. it worked for me):
> http://fak3r.com/2006/08/10/howto-passwordless-ssh-logins/
>
> Tim
>
>
> On Fri, Jul 3, 2009 at 4:31 PM, fred wang<fredwang222@gmail.com> wrote:
> > I remove the ~/.ssh and regenerate the key and it seems I still need to
> > provide password when I ssh localhost. Thank you very much even it
> couldn't
> > be fixed finally.
> >
> > But I found there is some warning information:
> >
> > ssh localhost
> >
> > The authenticity of host 'localhost (127.0.0.1)' can't be established.
> >
> > RSA key fingerprint is 4f:a1:ff:ed:0c:46:3e:a9:8c:97:bc:b7:46:3e:35:d2.
> >
> > Are you sure you want to continue connecting (yes/no)? yes
> >
> > Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
> >
> >
> > On 7/1/09 11:09 PM, fred wang wrote:
> >
> >>  sorry, should incopy ssh_config(instead of sshd_config)
> >>>
> >>>
> >>> vi /etc/ssh/ssh_config
> >>>
> >>> #  1. command line options
> >>>
> >>> #  2. user-specific file
> >>>
> >>> #  3. system-wide file
> >>>
> >>> # Any configuration value is only changed the first time it is set.
> >>>
> >>> # Thus, host-specific definitions should be at the beginning of the
> >>>
> >>> # configuration file, and defaults at the end.
> >>>
> >>>
> >>>
> >>> # Site-wide defaults for some commonly used options.  For a
> comprehensive
> >>>
> >>> # list of available options, their meanings and defaults, please see
> the
> >>>
> >>> # ssh_config(5) man page.
> >>>
> >>>
> >>>
> >>> Host *
> >>>
> >>> #   ForwardAgent no
> >>>
> >>> #   ForwardX11 no
> >>>
> >>> #   ForwardX11Trusted yes
> >>>
> >>> #   RhostsRSAAuthentication no
> >>>
> >>> #   RSAAuthentication yes
> >>>
> >>> #   PasswordAuthentication yes
> >>>
> >>> #   HostbasedAuthentication no
> >>>
> >>> #   GSSAPIAuthentication no
> >>>
> >>> #   GSSAPIDelegateCredentials no
> >>>
> >>> #   GSSAPIKeyExchange no
> >>>
> >>> #   GSSAPITrustDNS no
> >>>
> >>> #   BatchMode no
> >>>
> >>> #   CheckHostIP yes
> >>>
> >>> #   AddressFamily any
> >>>
> >>> #   ConnectTimeout 0
> >>>
> >>> #   StrictHostKeyChecking ask
> >>>
> >>> #   IdentityFile ~/.ssh/identity
> >>>
> >>> #   IdentityFile ~/.ssh/id_rsa
> >>>
> >>> #   IdentityFile ~/.ssh/id_dsa
> >>>
> >>> #   Port 22
> >>>
> >>> #   Protocol 2,1
> >>>
> >>> #   Cipher 3des
> >>>
> >>> #   Ciphers
> >>>
> aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc
> >>>
> >>> #   MACs hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160
> >>>
> >>> #   EscapeChar ~
> >>>
> >>> #   Tunnel no
> >>>
> >>> #   TunnelDevice any:any
> >>>
> >>> #   PermitLocalCommand no
> >>>
> >>>     SendEnv LANG LC_*
> >>>
> >>>     HashKnownHosts yes
> >>>
> >>>     GSSAPIAuthentication yes
> >>>
> >>>     GSSAPIDelegateCredentials no
> >>>
> >>>
> >>> On Thu, Jul 2, 2009 at 1:51 PM, fred wang<fredwang222@gmail.com>
>  wrote:
> >>>
> >>> Here is the output of ssh -v localhost  and the configuration of
> >>>> ssh_config,
> >>>>
> >>>> xxx@xxx-desktop:~$ ssh -v localhost
> >>>>
> >>>> OpenSSH_4.7p1 Debian-8ubuntu1.2, OpenSSL 0.9.8g 19 Oct 2007
> >>>>
> >>>> debug1: Reading configuration data /etc/ssh/ssh_config
> >>>>
> >>>> debug1: Applying options for *
> >>>>
> >>>> debug1: Connecting to localhost [127.0.0.1] port 22.
> >>>>
> >>>> debug1: Connection established.
> >>>>
> >>>> debug1: identity file /home/xxx/.ssh/identity type -1
> >>>>
> >>>> debug1: identity file /home/xxx/.ssh/id_rsa type -1
> >>>>
> >>>> debug1: identity file /home/xxx/.ssh/id_dsa type 2
> >>>>
> >>>> debug1: Remote protocol version 2.0, remote software version
> >>>> OpenSSH_4.7p1
> >>>> Debian-8ubuntu1.2
> >>>>
> >>>> debug1: match: OpenSSH_4.7p1 Debian-8ubuntu1.2 pat OpenSSH*
> >>>>
> >>>> debug1: Enabling compatibility mode for protocol 2.0
> >>>>
> >>>> debug1: Local version string SSH-2.0-OpenSSH_4.7p1 Debian-8ubuntu1.2
> >>>>
> >>>> debug1: SSH2_MSG_KEXINIT sent
> >>>>
> >>>> debug1: SSH2_MSG_KEXINIT received
> >>>>
> >>>> debug1: kex: server->client aes128-cbc hmac-md5 none
> >>>>
> >>>> debug1: kex: client->server aes128-cbc hmac-md5 none
> >>>>
> >>>> debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
> >>>>
> >>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
> >>>>
> >>>> debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
> >>>>
> >>>> debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
> >>>>
> >>>> debug1: Host 'localhost' is known and matches the RSA host key.
> >>>>
> >>>> debug1: Found key in /home/xxx/.ssh/known_hosts:1
> >>>>
> >>>> debug1: ssh_rsa_verify: signature correct
> >>>>
> >>>> debug1: SSH2_MSG_NEWKEYS sent
> >>>>
> >>>> debug1: expecting SSH2_MSG_NEWKEYS
> >>>>
> >>>> debug1: SSH2_MSG_NEWKEYS received
> >>>>
> >>>> debug1: SSH2_MSG_SERVICE_REQUEST sent
> >>>>
> >>>> debug1: SSH2_MSG_SERVICE_ACCEPT received
> >>>>
> >>>> debug1: Authentications that can continue: publickey,password
> >>>>
> >>>> debug1: Next authentication method: publickey
> >>>>
> >>>> debug1: Trying private key: /home/xxx/.ssh/identity
> >>>>
> >>>> debug1: Trying private key: /home/xxx/.ssh/id_rsa
> >>>>
> >>>> debug1: Offering public key: /home/xxx/.ssh/id_dsa
> >>>>
> >>>> debug1: Authentications that can continue: publickey,password
> >>>>
> >>>> debug1: Next authentication method: password
> >>>>
> >>>> xxx@localhost's password:
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> xxx@xxx:~$ vi /etc/ssh/sshd_config
> >>>>
> >>>> #KerberosOrLocalPasswd yes
> >>>>
> >>>> #KerberosTicketCleanup yes
> >>>>
> >>>>
> >>>>
> >>>> # GSSAPI options
> >>>>
> >>>> #GSSAPIAuthentication no
> >>>>
> >>>> #GSSAPICleanupCredentials yes
> >>>>
> >>>>
> >>>>
> >>>> X11Forwarding yes
> >>>>
> >>>> X11DisplayOffset 10
> >>>>
> >>>> PrintMotd no
> >>>>
> >>>> PrintLastLog yes
> >>>>
> >>>> TCPKeepAlive yes
> >>>>
> >>>> #UseLogin no
> >>>>
> >>>>
> >>>>
> >>>> #MaxStartups 10:30:60
> >>>>
> >>>> #Banner /etc/issue.net
> >>>>
> >>>>
> >>>>
> >>>> # Allow client to pass locale environment variables
> >>>>
> >>>> AcceptEnv LANG LC_*
> >>>>
> >>>>
> >>>>
> >>>> Subsystem sftp /usr/lib/openssh/sftp-server
> >>>>
> >>>>
> >>>>
> >>>> UsePAM yes
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Jul 2, 2009 at 1:18 PM, Konstantin Boudnik<cos@yahoo-inc.com
> >>>> >wrote:
> >>>>
> >>>> Yet another possibility is that your SSH daemon isn't configured to
> >>>>> accept
> >>>>> publickey as a valid authorization mean.
> >>>>>
> >>>>> Try to do ssh -v localhost and check if there's something similar to
> the
> >>>>> following:
> >>>>>
> >>>>> debug1: Authentications that can continue:
> >>>>> publickey,password,keyboard-interactive
> >>>>> debug1: Next authentication method: publickey
> >>>>> debug1: Trying private key: /home/xxx/.ssh/identity
> >>>>> debug1: Trying private key: /home/xxx/.ssh/id_rsa
> >>>>> debug1: Offering public key: /home/xxx/.ssh/id_dsa
> >>>>> debug1: Server accepts key: pkalg ssh-dss blen 435
> >>>>> debug1: read PEM private key done: type DSA
> >>>>> debug1: Authentication succeeded (publickey).
> >>>>>
> >>>>> Cos
> >>>>>
> >>>>>
> >>>>> On 7/1/09 10:11 PM, fred wang wrote:
> >>>>>
> >>>>> I have setup ./.ssh/authorized keys has permssion 600, but it didn't
> >>>>>> work.
> >>>>>> Thanks anyway
> >>>>>>
> >>>>>> ls -l .ssh/authorized_keys
> >>>>>> -rw------- 1 xxx xxx 1222 2009-07-02 13:08 .ssh/authorized_keys
> >>>>>>
> >>>>>> On Thu, Jul 2, 2009 at 12:15 AM, Konstantin Boudnik<
> cos@yahoo-inc.com
> >>>>>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>> Make sure that your ~/.ssh/authorized_keys has permissions 600
> >>>>>>
> >>>>>>> Cos
> >>>>>>>
> >>>>>>>
> >>>>>>> On 7/1/09 7:35 AM, fred wang wrote:
> >>>>>>>
> >>>>>>> Hi all,
> >>>>>>>
> >>>>>>>>   I failed to setup passphraseless ssh(I mean, I still need to
> input
> >>>>>>>> password to do ssh localhost) when I tried to configure Hadoop to
> run
> >>>>>>>> on
> >>>>>>>> psuedo-distributed operation,  could anyone help me solve this
> issue?
> >>>>>>>> Thanks!
> >>>>>>>>
> >>>>>>>> (1)I use the Putty0.6 to remote access to Ubuntu by SSH.
> >>>>>>>>
> >>>>>>>> (2) execution steps and ouput
> >>>>>>>>
> >>>>>>>> $ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
> >>>>>>>> Generating public/private dsa key pair.
> >>>>>>>> Your identification has been saved in /home/xxx/.ssh/id_dsa.
> >>>>>>>> Your public key has been saved in /home/xxx/.ssh/id_dsa.pub.
> >>>>>>>> The key fingerprint is:
> >>>>>>>> a9:39:4c:9b:22:f9:a4:77:70:24:fa:bf:12:f5:81:81 xxx
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> **note: it doesn't have message  'Enter passphrase (empty for no
> >>>>>>>> passphrase):
> >>>>>>>>     Enter same passphrase again: ' which appear in some
> introductory
> >>>>>>>> paper.
> >>>>>>>> "
> >>>>>>>>
> >>>>>>>> $ cat ~/.ssh/id_dsa.pub>>    ~/.ssh/authorized_keys
> >>>>>>>> no output
> >>>>>>>>
> >>>>>>>> $ ssh localhost
> >>>>>>>> The authenticity of host 'localhost (127.0.0.1)' can't be
> >>>>>>>> established.
> >>>>>>>> RSA key fingerprint is
> >>>>>>>> 4f:a1:ff:ed:0c:46:3e:a9:8c:97:bc:b7:46:3e:35:d2.
> >>>>>>>> Are you sure you want to continue connecting (yes/no)? yes
> >>>>>>>> Warning: Permanently added 'localhost' (RSA) to the list of known
> >>>>>>>> hosts.
> >>>>>>>> xxx@localhost's password:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >
>

--0016364ecd80993093046e028f26--

From general-return-344-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 06 06:28:17 2009
Return-Path: <general-return-344-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 10713 invoked from network); 6 Jul 2009 06:28:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jul 2009 06:28:17 -0000
Received: (qmail 6877 invoked by uid 500); 6 Jul 2009 06:28:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6805 invoked by uid 500); 6 Jul 2009 06:28:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6795 invoked by uid 99); 6 Jul 2009 06:28:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jul 2009 06:28:26 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of bogdan.maryniuk@gmail.com designates 209.85.200.168 as permitted sender)
Received: from [209.85.200.168] (HELO wf-out-1314.google.com) (209.85.200.168)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jul 2009 06:28:15 +0000
Received: by wf-out-1314.google.com with SMTP id 23so1327305wfg.2
        for <general@hadoop.apache.org>; Sun, 05 Jul 2009 23:27:54 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=bXQlFRYeFPCFcxroyH8o9ZkmIkherWjAH+aYmqv7nbs=;
        b=t8XfUcTqU3Xa3tE+GgwZTzZvt3BzGk88D1kYz1ZGRvniKcW29Sb7sUboTlgP6n3ukn
         D5pkGm3VIcsgPC2q6ZoEd+NehwvYvxp4Q0PKbvO1zGoc66Xq3Ka9vTF0QBTI777PuiMF
         puEngqYM/6OyJRaGvSzKudPWstal36nVs8RH4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=mMHFFBJVVLMIZvwrRyFXpgag6w1mRk8r7xabD1tRl5qWjUmUToGzyxVQkkF0Os3xTO
         r6pNZeVdTRHedxjpdgt11BWr6R0Q5/c5E4zh3SFjLzhq8pgQlfDOUzL8jajlIIb1lEtc
         XN6C6yiK2BxF6suCU/KRLEPMM/UkYPEzav3q8=
MIME-Version: 1.0
Received: by 10.142.125.8 with SMTP id x8mr1309406wfc.44.1246861674456; Sun, 
	05 Jul 2009 23:27:54 -0700 (PDT)
In-Reply-To: <702261350907052213u6b13190ap35ebc1df2c206d3b@mail.gmail.com>
References: <702261350907010735u3502197au800281a5a131d963@mail.gmail.com>
	 <4A4B8B99.10303@yahoo-inc.com>
	 <702261350907012211r17dec368u17e18052383a8822@mail.gmail.com>
	 <4A4C430F.7080306@yahoo-inc.com>
	 <702261350907012251s41b18b24m5dc54015007d494e@mail.gmail.com>
	 <702261350907012309l4c62885bk6df2ebac8e2b7254@mail.gmail.com>
	 <4A4CDF0F.2040106@yahoo-inc.com>
	 <702261350907030731w694cd855safc8b3e5b5048747@mail.gmail.com>
	 <32120a6a0907030745g19ef2593o457f48972ef51ea@mail.gmail.com>
	 <702261350907052213u6b13190ap35ebc1df2c206d3b@mail.gmail.com>
Date: Mon, 6 Jul 2009 15:27:54 +0900
Message-ID: <fa561940907052327l759784d3p13fef938f6c0cd28@mail.gmail.com>
Subject: Re: fail to setup passphraseless ssh and need some help
From: "Bogdan M. Maryniuk" <bogdan.maryniuk@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On Mon, Jul 6, 2009 at 2:13 PM, fred wang<fredwang222@gmail.com> wrote:
> it doesn't work..

Yes, it does. 10 years already and for everybody.
http://www.cs.wustl.edu/~mdeters/how-to/ssh/

-- 
Kind regards, BM

Things, that are stupid at the beginning, rarely ends up wisely.

From general-return-345-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 06 07:35:04 2009
Return-Path: <general-return-345-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25292 invoked from network); 6 Jul 2009 07:35:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jul 2009 07:35:03 -0000
Received: (qmail 40994 invoked by uid 500); 6 Jul 2009 07:35:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40925 invoked by uid 500); 6 Jul 2009 07:35:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40915 invoked by uid 99); 6 Jul 2009 07:35:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jul 2009 07:35:13 +0000
X-ASF-Spam-Status: No, hits=3.7 required=10.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of xinejar22@googlemail.com designates 72.14.220.153 as permitted sender)
Received: from [72.14.220.153] (HELO fg-out-1718.google.com) (72.14.220.153)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jul 2009 07:35:02 +0000
Received: by fg-out-1718.google.com with SMTP id l26so652890fgb.12
        for <general@hadoop.apache.org>; Mon, 06 Jul 2009 00:34:40 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=+FhoBqr4ckk5/QUwUAdiMJDmdK3rbofkiR3HxFBkgJ0=;
        b=c5OqJbG+QJlN2jR9HUjo906iGyhnl42mo91Ai+fd3V9mlRyCYTksbNdyW2ynGe3Q/S
         NZaqM6ZYGLVJhLmUaj9/Ycevn3qZVjqLyN7w/+HamzctyZxxwPtZoUyVzyvt4Gd71N8D
         NNilyAvpOqrgXeQBWN9SPRT0rVayPvoEAcph8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=n45BcbRba7/KBS6u/dNcV8jZNd1tCm9hIrG12tMWnyZqdmWXbZlJE2rJqnq97d+vEG
         NtqRSy0JRUxXI7o03RW/L28Zln4caTTNVQvcOQd9rqJyiM7mphCfjhteqsM5NIk0JKvR
         iNh7nZvKiN9++OyI3KTWooybF8dj/4eQodDO4=
MIME-Version: 1.0
Received: by 10.86.68.18 with SMTP id q18mr1950359fga.68.1246865678722; Mon, 
	06 Jul 2009 00:34:38 -0700 (PDT)
Date: Mon, 6 Jul 2009 09:34:38 +0200
Message-ID: <115373b00907060034t170c14afmb3aef33260424a75@mail.gmail.com>
Subject: problem with starting the Jobtracker and thenamenode
From: Xine Jar <xinejar22@googlemail.com>
To: general@hadoop.apache.org, xinejar22@googlemail.com
Content-Type: multipart/alternative; boundary=000e0cd2431cdce6fd046e048930
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd2431cdce6fd046e048930
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

*Hallo, I am using Hadoop 0.18.3 on a cluster of three linux machines.
The jobtracker and the namenode cannot be started. I have copied the log
files and
the configuration file. Can someone help?*


*Jobtracker log file

*2009-07-03 17:34:09,315 INFO org.apache.hadoop.mapred.JobTracker:
STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG:   host = alshain/134.130.222.20
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 0.18.3
STARTUP_MSG:   build =
https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.18 -r 736250;
compiled by 'ndaley' on Thu Jan 22 23:12:08 UTC 2009
************************************************************/
2009-07-03 17:34:09,429 FATAL org.apache.hadoop.mapred.JobTracker:
java.net.BindException: Problem binding to
albali.mobnets.RWTH-Aachen.DE/134.130.222.18:9001 : Cannot assign requested
address
    at org.apache.hadoop.ipc.Server.bind(Server.java:170)
    at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:233)
    at org.apache.hadoop.ipc.Server.<init>(Server.java:953)
    at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:465)
    at org.apache.hadoop.ipc.RPC.getServer(RPC.java:427)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:659)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:131)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:2373)
Caused by: java.net.BindException: Cannot assign requested address
    at sun.nio.ch.Net.bind(Native Method)
    at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
    at org.apache.hadoop.ipc.Server.bind(Server.java:168)
    ... 7 more

2009-07-03 17:34:09,430 INFO org.apache.hadoop.mapred.JobTracker:
SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down JobTracker at alshain/134.130.222.20
************************************************************/

*Namenode log file*

STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = alshain/134.130.222.20
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 0.18.3
STARTUP_MSG:   build =
https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.18 -r 736250;
compiled by 'ndaley' on Thu Jan 22 23:12:08 UTC 2009
************************************************************/
2009-07-03 17:33:23,246 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=NameNode, port=9000
2009-07-03 17:33:23,284 INFO org.apache.hadoop.dfs.NameNode: Namenode up at:
alshain.mobnets.rwth-aachen.de/134.130.222.20:9000
2009-07-03 17:33:23,287 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=NameNode, sessionId=null
2009-07-03 17:33:23,291 INFO org.apache.hadoop.dfs.NameNodeMetrics:
Initializing NameNodeMeterics using context
object:org.apache.hadoop.metrics.spi.NullContext
2009-07-03 17:33:23,355 INFO org.apache.hadoop.fs.FSNamesystem:
fsOwner=root,root
2009-07-03 17:33:23,355 INFO org.apache.hadoop.fs.FSNamesystem:
supergroup=supergroup
2009-07-03 17:33:23,355 INFO org.apache.hadoop.fs.FSNamesystem:
isPermissionEnabled=true
2009-07-03 17:33:23,361 INFO org.apache.hadoop.dfs.FSNamesystemMetrics:
Initializing FSNamesystemMeterics using context
object:org.apache.hadoop.metrics.spi.NullContext
2009-07-03 17:33:23,362 INFO org.apache.hadoop.fs.FSNamesystem: Registered
FSNamesystemStatusMBean
2009-07-03 17:33:23,393 ERROR org.apache.hadoop.fs.FSNamesystem:
FSNamesystem initialization failed.
java.io.IOException: NameNode is not formatted.
    at org.apache.hadoop.dfs.FSImage.recoverTransitionRead(FSImage.java:243)
    at org.apache.hadoop.dfs.FSDirectory.loadFSImage(FSDirectory.java:80)
    at org.apache.hadoop.dfs.FSNamesystem.initialize(FSNamesystem.java:294)
    at org.apache.hadoop.dfs.FSNamesystem.<init>(FSNamesystem.java:273)
    at org.apache.hadoop.dfs.NameNode.initialize(NameNode.java:148)
    at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:193)
    at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:179)
    at org.apache.hadoop.dfs.NameNode.createNameNode(NameNode.java:830)
    at org.apache.hadoop.dfs.NameNode.main(NameNode.java:839)
2009-07-03 17:33:23,394 INFO org.apache.hadoop.ipc.Server: Stopping server
on 9000
2009-07-03 17:33:23,394 ERROR org.apache.hadoop.dfs.NameNode:
java.io.IOException: NameNode is not formatted.
    at org.apache.hadoop.dfs.FSImage.recoverTransitionRead(FSImage.java:243)
    at org.apache.hadoop.dfs.FSDirectory.loadFSImage(FSDirectory.java:80)
    at org.apache.hadoop.dfs.FSNamesystem.initialize(FSNamesystem.java:294)
    at org.apache.hadoop.dfs.FSNamesystem.<init>(FSNamesystem.java:273)
    at org.apache.hadoop.dfs.NameNode.initialize(NameNode.java:148)
    at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:193)
    at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:179)
    at org.apache.hadoop.dfs.NameNode.createNameNode(NameNode.java:830)
    at org.apache.hadoop.dfs.NameNode.main(NameNode.java:839)

2009-07-03 17:33:23,395 INFO org.apache.hadoop.dfs.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at alshain/134.130.222.20
************************************************************/


*Hadoop-site.xml

*<configuration>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://134.130.222.20:9000/</value>
    </property>
    <property>
        <name>mapred.job.tracker</name>
        <value>134.130.222.18:9001</value>
    </property>
    <property>
        <name>dfs.name.dir</name>
        <value>/root/Desktop/hadoop-0.18.3/logstina</value>
    </property>
    <property>
        <name>dfs.data.dir</name>
        <value>/root/Desktop/hadoop-0.18.3/blockstina</value>
    </property>
    <property>
        <name>mapred.system.dir</name>
        <value>systemtina</value>
    </property>
    <property>
        <name>mapred.local.dir</name>
        <value>/root/Desktop/hadoop-0.18.3/tempMapReducetina</value>
    </property>
</configuration>

*Thank you,
CJ*

--000e0cd2431cdce6fd046e048930--

From general-return-346-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 06 08:12:00 2009
Return-Path: <general-return-346-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35979 invoked from network); 6 Jul 2009 08:12:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jul 2009 08:12:00 -0000
Received: (qmail 66529 invoked by uid 500); 6 Jul 2009 08:12:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66455 invoked by uid 500); 6 Jul 2009 08:12:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66445 invoked by uid 99); 6 Jul 2009 08:12:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jul 2009 08:12:09 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bogdan.maryniuk@gmail.com designates 209.85.222.196 as permitted sender)
Received: from [209.85.222.196] (HELO mail-pz0-f196.google.com) (209.85.222.196)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jul 2009 08:12:01 +0000
Received: by pzk34 with SMTP id 34so3333955pzk.5
        for <general@hadoop.apache.org>; Mon, 06 Jul 2009 01:11:41 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:cc:content-type
         :content-transfer-encoding;
        bh=N82nHSrwiR5if1gxMFFom+jEEF4bcYh1AmoXwr8BhBk=;
        b=J1cg0jR8AYBdfOAH/wpWrG7lp2Jt7NXjiTbjoZSwypDwFyTeVo0vhCdpvVO2u8dw53
         f8mFwhAb48CLJhFSxmFdHSuydhiOXmlAKZe/fXWavMI46Zy5bAGwwDOFnUAxp6fl9s+q
         KkOwVsBQGNumNfpM6wXIQu6qiMkTH1je1KCbA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type:content-transfer-encoding;
        b=DMbrKn6fEjtOAWqPq8yf9KdiTR1t5ujdf3uFcXGC53uYPM7MwvpSyMNAzLaus7u0/l
         3uGfrTegrgOi7ZlFxy0yHrbOk4e0FrELKh20/0JmjeeFPDyI7dqe9x+gFXBes7d1OdNO
         +EyUZrCUeaXX9lZ9IhzGsOW6jsqCb+CDKblzQ=
MIME-Version: 1.0
Received: by 10.142.12.7 with SMTP id 7mr1274953wfl.11.1246867901617; Mon, 06 
	Jul 2009 01:11:41 -0700 (PDT)
In-Reply-To: <115373b00907060034t170c14afmb3aef33260424a75@mail.gmail.com>
References: <115373b00907060034t170c14afmb3aef33260424a75@mail.gmail.com>
Date: Mon, 6 Jul 2009 17:11:41 +0900
Message-ID: <fa561940907060111l2d361924g59ee318138a46402@mail.gmail.com>
Subject: Re: problem with starting the Jobtracker and thenamenode
From: "Bogdan M. Maryniuk" <bogdan.maryniuk@gmail.com>
To: general@hadoop.apache.org
Cc: xinejar22@googlemail.com
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Mon, Jul 6, 2009 at 4:34 PM, Xine Jar<xinejar22@googlemail.com> wrote:
> Caused by: java.net.BindException: Cannot assign requested address

See? That's why.

1. Make sure localhost loopback was not screwed (i.e. socket
configuration has not been removed from the IP for the system that is
showing you this error).
2. Make sure you've disabled IPv6 (either
-Djava.net.preferIPv4Stack=3Dtrue or other way) and you're not suffering
from this bug: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=3D6226918
3. Make sure none of instances is running on the same address =E2=80=94 jus=
t
in case... :-)

--=20
Kind regards, BM

Things, that are stupid at the beginning, rarely ends up wisely.

From general-return-347-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 02:47:49 2009
Return-Path: <general-return-347-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 10849 invoked from network); 7 Jul 2009 02:47:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 02:47:49 -0000
Received: (qmail 95819 invoked by uid 500); 7 Jul 2009 02:47:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95735 invoked by uid 500); 7 Jul 2009 02:47:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95725 invoked by uid 99); 7 Jul 2009 02:47:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 02:47:58 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of bogdan.maryniuk@gmail.com designates 209.85.200.168 as permitted sender)
Received: from [209.85.200.168] (HELO wf-out-1314.google.com) (209.85.200.168)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 02:47:48 +0000
Received: by wf-out-1314.google.com with SMTP id 23so1546625wfg.2
        for <general@hadoop.apache.org>; Mon, 06 Jul 2009 19:47:26 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=Xi8M00zvH/45vyuvzDmosviW8t9Bnx5kdNzyn3elyNM=;
        b=JRqmEHO1jHbhlUvrfs1kEvj4qQcrvzP4IVk/3MaTN+EymOF4pQUYf7/tFzVGM7kMFU
         U6iJ8oj900zFm+5GofK9I2pJ3OaTtssWvdK+Y+s8i6bkiI4QkqraP1ucV5w0KbDw25l1
         PrsfhEMz2aSueJFaF+xc+vdOyL4x27bUQDtIk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=aZIN2WZ8oq0mBJar3YYViCCiDziwSpUOyUlkZNP7VyC1cwkjI+xs/xeBevGPE5onHu
         Ca93mBW9MFHKdDw+fSck75FYf/JqGEeKSvhOX1xZq4VS0ksal7ssTIQTk313r59wJEVE
         hcYZHVJdmmNWxSaBwd8iF010EiYLXl5+q8Rz4=
MIME-Version: 1.0
Received: by 10.142.12.7 with SMTP id 7mr1623753wfl.11.1246934846739; Mon, 06 
	Jul 2009 19:47:26 -0700 (PDT)
In-Reply-To: <fa561940907061946g672275c7k9c5adef6230b2f62@mail.gmail.com>
References: <115373b00907060034t170c14afmb3aef33260424a75@mail.gmail.com>
	 <fa561940907060111l2d361924g59ee318138a46402@mail.gmail.com>
	 <115373b00907060524p1fcb996cq2bbb86c2fe85bd5a@mail.gmail.com>
	 <fa561940907061946g672275c7k9c5adef6230b2f62@mail.gmail.com>
Date: Tue, 7 Jul 2009 11:47:26 +0900
Message-ID: <fa561940907061947x5ffe409cg361de4deaa0c358e@mail.gmail.com>
Subject: Re: problem with starting the Jobtracker and thenamenode
From: "Bogdan M. Maryniuk" <bogdan.maryniuk@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Mon, Jul 6, 2009 at 9:24 PM, Xine Jar<xinejar22@googlemail.com> wrote:
> 2.How can I check that ipv6 is really disabled on my JVM?

You've mentioned it is Linux, but how do I know what distribution of
Linux's zoo you use?

On Ubuntu it is sort of like this:
In the file "/etc/modprobe.d/aliases" find "alias net-pf-10 ipv6",
remove it and add the following:

alias net-pf-10 off
alias ipv6 off

Then reboot (welcome to the Linux).

On RedHat and derivatives it is like this:
echo "alias net-pf-10 off" >> =C2=A0/etc/modprobe.conf <ENTER>

Then reboot (insert your ironic joke here). :-)


> and in which file=C2=A0can I insert the flag -Djava.net.preferIPv4Stack=
=3Dtrue?
It is just a JVM parameter for a networking:
1. http://java.sun.com/j2se/1.4.2/docs/guide/net/properties.html
2. http://java.sun.com/j2se/1.4.2/docs/guide/net/ipv6_guide/#ipv6-networkin=
g

See $HADOOP/conf/hadoop-env.sh and look for HADOOP_OPTS that is
commented out by default.

--
Kind regards, BM

Things, that are stupid at the beginning, rarely ends up wisely.

From general-return-348-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 13:33:49 2009
Return-Path: <general-return-348-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5662 invoked from network); 7 Jul 2009 13:33:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 13:33:49 -0000
Received: (qmail 32479 invoked by uid 500); 7 Jul 2009 13:33:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32440 invoked by uid 500); 7 Jul 2009 13:33:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32430 invoked by uid 99); 7 Jul 2009 13:33:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 13:33:58 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of tom.e.white@gmail.com designates 209.85.220.220 as permitted sender)
Received: from [209.85.220.220] (HELO mail-fx0-f220.google.com) (209.85.220.220)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 13:33:49 +0000
Received: by fxm20 with SMTP id 20so5001478fxm.29
        for <general@hadoop.apache.org>; Tue, 07 Jul 2009 06:33:29 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        bh=YAzy8ZNI7pj79DZsnl/OoGN4vrtJI/uMrL14L4E0w8k=;
        b=PcPbGi6BUD2uWSoqpv9Ra2itRzjks6iRiNkfeWTPQKqemVDLcQogAxfSacTKrZl1ge
         D7HY+eo9u4iQ+Cq8N3KswXZO/MUXBIOcXXx4OZRedu3fVxoj1E6upvtVzIJ0SkPd9rMp
         zkyYYReuLeaPEPFl2uzeMeREl+qW5nAL8HlYg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=FbA9agbdY4L+ETQEuEvvRIoqCtHZrXnmHBY3zHH1cZa5T4sQi1fbxcLzLBrRdCZOuz
         8UA5sb9KcwFwUf3Y/YahWLR3bKSlSOCCJKKoT2QGbv39T1T2E7woLkVAQAbyvgcnGUm0
         XOFNi3FKxBgvN3NFuH8yGzoHLMpbMIY4HRXPg=
MIME-Version: 1.0
Sender: tom.e.white@gmail.com
Received: by 10.103.39.17 with SMTP id r17mr3322555muj.75.1246973609299; Tue, 
	07 Jul 2009 06:33:29 -0700 (PDT)
In-Reply-To: <b23e01b50907010244j6885edf4vcc9fd5eb4e21acff@mail.gmail.com>
References: <b23e01b50907010244j6885edf4vcc9fd5eb4e21acff@mail.gmail.com>
Date: Tue, 7 Jul 2009 14:33:29 +0100
X-Google-Sender-Auth: bf1ada7a01b399b1
Message-ID: <b23e01b50907070633j66d7107m3eb7fe63c53b6228@mail.gmail.com>
Subject: Fwd: [VOTE] Release Hadoop 0.19.2 (candidate 0)
From: Tom White <tomwhite@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Please download and test this Hadoop 0.19.2 release candidate, and
vote on common-dev.

Thanks,
Tom

---------- Forwarded message ----------
From: Tom White <tomwhite@apache.org>
Date: Wed, Jul 1, 2009 at 10:44 AM
Subject: [VOTE] Release Hadoop 0.19.2 (candidate 0)
To: common-dev@hadoop.apache.org


I have created a candidate build for Hadoop 0.19.2. This fixes 42
issues in 0.19.1
(http://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&mode=hide&sorter/order=DESC&sorter/field=priority&pid=12310240&fixfor=12313650).

*** Please download, test and vote before the
*** vote closes on Friday, July 3.

http://people.apache.org/~tomwhite/hadoop-0.19.2-candidate-0/

Cheers,
Tom

From general-return-349-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 02:53:07 2009
Return-Path: <general-return-349-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42437 invoked from network); 8 Jul 2009 02:53:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 02:53:07 -0000
Received: (qmail 98691 invoked by uid 500); 8 Jul 2009 02:53:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98602 invoked by uid 500); 8 Jul 2009 02:53:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98592 invoked by uid 99); 8 Jul 2009 02:53:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 02:53:16 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of fredwang222@gmail.com designates 209.85.222.196 as permitted sender)
Received: from [209.85.222.196] (HELO mail-pz0-f196.google.com) (209.85.222.196)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 02:53:03 +0000
Received: by pzk34 with SMTP id 34so4465839pzk.5
        for <general@hadoop.apache.org>; Tue, 07 Jul 2009 19:52:42 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=Z3MVJ1W2yw2TsaLIIOEqzM9TMQO7QznIWPhoz9wOUKc=;
        b=P1fHBgrDW3/6eBZLhtGdqckk+kujdmHsWby3XelleRclb2XHROIBqv47B6Qpg5LUzH
         aYfICVFrssqOmRTZ2cdQTxCXA4LEDp/DuTtiy5UidR5zI7svDd4z2jZJ8r6LLF/aQC9y
         7zTy52H2FFXDHSUrzz4SlxgyWUTc7DMmi5e0Q=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=kr0h/7gmhNSmUhr3k6v9hzyTpiAxAsR+8eI8J5AtD/fqSoysY91X9hSbgRsjJigJCi
         QyOTPRCkrviYCas435KCutfLHztCfz2a6MgmC8RzvURGt9sep+3i+lhOch/3o7t3TKFl
         ww3dCd9VlAReU8ztY8fMqlAF7Agj+11kr60NM=
MIME-Version: 1.0
Received: by 10.142.135.16 with SMTP id i16mr2173263wfd.315.1247021562767; 
	Tue, 07 Jul 2009 19:52:42 -0700 (PDT)
In-Reply-To: <fa561940907052327l759784d3p13fef938f6c0cd28@mail.gmail.com>
References: <702261350907010735u3502197au800281a5a131d963@mail.gmail.com>
	 <702261350907012211r17dec368u17e18052383a8822@mail.gmail.com>
	 <4A4C430F.7080306@yahoo-inc.com>
	 <702261350907012251s41b18b24m5dc54015007d494e@mail.gmail.com>
	 <702261350907012309l4c62885bk6df2ebac8e2b7254@mail.gmail.com>
	 <4A4CDF0F.2040106@yahoo-inc.com>
	 <702261350907030731w694cd855safc8b3e5b5048747@mail.gmail.com>
	 <32120a6a0907030745g19ef2593o457f48972ef51ea@mail.gmail.com>
	 <702261350907052213u6b13190ap35ebc1df2c206d3b@mail.gmail.com>
	 <fa561940907052327l759784d3p13fef938f6c0cd28@mail.gmail.com>
Date: Wed, 8 Jul 2009 10:52:42 +0800
Message-ID: <702261350907071952t541131efyd0bfd8448c0ffb5e@mail.gmail.com>
Subject: Re: fail to setup passphraseless ssh and need some help
From: fred wang <fredwang222@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd3287246a893046e28d5cd
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd3287246a893046e28d5cd
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Maybe  there is something wrong with the authority. I installed a new Ubuntu
and it works. Thanks to those who have helped.

On Mon, Jul 6, 2009 at 2:27 PM, Bogdan M. Maryniuk <
bogdan.maryniuk@gmail.com> wrote:

> On Mon, Jul 6, 2009 at 2:13 PM, fred wang<fredwang222@gmail.com> wrote:
> > it doesn't work..
>
> Yes, it does. 10 years already and for everybody.
> http://www.cs.wustl.edu/~mdeters/how-to/ssh/
>
> --
> Kind regards, BM
>
> Things, that are stupid at the beginning, rarely ends up wisely.
>

--000e0cd3287246a893046e28d5cd--

From general-return-350-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 09:13:01 2009
Return-Path: <general-return-350-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69467 invoked from network); 8 Jul 2009 09:13:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 09:13:01 -0000
Received: (qmail 47637 invoked by uid 500); 8 Jul 2009 09:13:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47562 invoked by uid 500); 8 Jul 2009 09:13:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47552 invoked by uid 99); 8 Jul 2009 09:13:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 09:13:10 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of xinejar22@googlemail.com designates 72.14.220.158 as permitted sender)
Received: from [72.14.220.158] (HELO fg-out-1718.google.com) (72.14.220.158)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 09:12:59 +0000
Received: by fg-out-1718.google.com with SMTP id l26so1095431fgb.12
        for <general@hadoop.apache.org>; Wed, 08 Jul 2009 02:12:38 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=9K5dquAjgF6T7C7iJtB1yLXKMAJSkLMDo3rE81adUo8=;
        b=AoPtmVX5cU/g/sHf17dbUFqt/RAbZ2JhE4RKoOUHbkmGmspEc872n7tD6jBeUpiLCt
         z4TijLnTTIiYSL5YZlKAefRHDiJPekCYwZyzlraoxI5Fr09+iq8ZGc8c1XNH2JsrXY/r
         Q8lMBvGCTFaa1shavtkITxn0S8sSasLxVCrHg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=J5O4rwqYg6genbf+gEoFNfACv/3/hhs7a7NvLVqOqQgrduHcufflBTN93hfC4q96LK
         ezt3cAsp7r9NdEQjVOcoVOImggJgnIBIYy9IRiE2hLrb+ck/g5iNWCYh6MuEKS/1393q
         qhf6LLwopFPNzQXAro00IqB6FiJKQsA4O/kIs=
MIME-Version: 1.0
Received: by 10.86.90.13 with SMTP id n13mr2812305fgb.45.1247044358576; Wed, 
	08 Jul 2009 02:12:38 -0700 (PDT)
In-Reply-To: <fa561940907061947x5ffe409cg361de4deaa0c358e@mail.gmail.com>
References: <115373b00907060034t170c14afmb3aef33260424a75@mail.gmail.com>
	 <fa561940907060111l2d361924g59ee318138a46402@mail.gmail.com>
	 <115373b00907060524p1fcb996cq2bbb86c2fe85bd5a@mail.gmail.com>
	 <fa561940907061946g672275c7k9c5adef6230b2f62@mail.gmail.com>
	 <fa561940907061947x5ffe409cg361de4deaa0c358e@mail.gmail.com>
Date: Wed, 8 Jul 2009 11:12:38 +0200
Message-ID: <115373b00907080212n12ae70d6oc3a9c7a3c2a02fbd@mail.gmail.com>
Subject: Re: problem with starting the Jobtracker and thenamenode
From: Xine Jar <xinejar22@googlemail.com>
To: general@hadoop.apache.org, xinejar22@googlemail.com
Content-Type: multipart/alternative; boundary=000e0cd24a0c031b3a046e2e24f8
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd24a0c031b3a046e2e24f8
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hallo,
I am still struggling with the same problem. In order to be sure that IPv6
is not creating a problem and that the JVM is not suffering from a bug, I
have written a TCP client/server program in Java, I let the server run on
the same machine that was creating the binding problem and using the same
port number. Luckily or unfortunately, the Java server could bind the
address normally to the port, the communication between the client and the
server was successful, and the netstat command shows that the port is used.

I order to be sure that the problem is not a bug for hadoop I have installed
the version 0.20.0,
copied the same configuration and still have the same problem.

*I have few questions: *
As I have previously mentioned, the namenode cannot be started and java
complains that the node has not been formated. The other problem is with the
jobtracker which is giving a binding error on the address:port.

*Q1:* Is it possible that the second problem (jobtracker problem) is
appearing because of the namenode problem?

If it is a yes, how can I solve this? I have actually deleted the /tmp
folder and reformated the node but the formatting error persists!! Do I have
to do something else?

If it is a no,  am I doing something stupid here in any of the configuration
files?!!!

*Q2:  *In order to kill all the instances of the dfs and the mapred, is it
enough to execute the bin/stop-dfs.sh and bin/stop-mapred.sh on the
namenode? Is it possible that the command line is showing that there is no
jobtracker, no namenode, ...... but in fact the port is occupied?

P.S: -Whether hadoop is running or not, the netstat command is showing
always that the port
         is free!!!!!
       - My OS is a openSuse 9*

*I am trying to debug everything, because I am somehow sick of it, It is
certainly something stupid and most probably my fault (or my configuration
or my way of running things) , since others managed to run hadoop on a
cluster. I would appreciate any idea that helps to discover this the
problem.

Thank you,
CJ


On Tue, Jul 7, 2009 at 4:47 AM, Bogdan M. Maryniuk <
bogdan.maryniuk@gmail.com> wrote:

> On Mon, Jul 6, 2009 at 9:24 PM, Xine Jar<xinejar22@googlemail.com> wrote:
> > 2.How can I check that ipv6 is really disabled on my JVM?
>
> You've mentioned it is Linux, but how do I know what distribution of
> Linux's zoo you use?
>
> On Ubuntu it is sort of like this:
> In the file "/etc/modprobe.d/aliases" find "alias net-pf-10 ipv6",
> remove it and add the following:
>
> alias net-pf-10 off
> alias ipv6 off
>
> Then reboot (welcome to the Linux).
>
> On RedHat and derivatives it is like this:
> echo "alias net-pf-10 off" >>  /etc/modprobe.conf <ENTER>
>
> Then reboot (insert your ironic joke here). :-)
>
>
> > and in which file can I insert the flag -Djava.net.preferIPv4Stack=true?
> It is just a JVM parameter for a networking:
> 1. http://java.sun.com/j2se/1.4.2/docs/guide/net/properties.html
> 2.
> http://java.sun.com/j2se/1.4.2/docs/guide/net/ipv6_guide/#ipv6-networking
>
> See $HADOOP/conf/hadoop-env.sh and look for HADOOP_OPTS that is
> commented out by default.
>
> --
> Kind regards, BM
>
> Things, that are stupid at the beginning, rarely ends up wisely.
>

--000e0cd24a0c031b3a046e2e24f8--

From general-return-351-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 12:06:51 2009
Return-Path: <general-return-351-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 10242 invoked from network); 8 Jul 2009 12:06:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 12:06:51 -0000
Received: (qmail 53887 invoked by uid 500); 8 Jul 2009 12:07:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53834 invoked by uid 500); 8 Jul 2009 12:07:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53824 invoked by uid 99); 8 Jul 2009 12:07:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 12:07:00 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saurabhnanda@gmail.com designates 209.85.212.196 as permitted sender)
Received: from [209.85.212.196] (HELO mail-vw0-f196.google.com) (209.85.212.196)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 12:06:52 +0000
Received: by vwj34 with SMTP id 34so503196vwj.5
        for <general@hadoop.apache.org>; Wed, 08 Jul 2009 05:06:31 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=K8IAh/KONyHI1fGgIyq05NbyEHLSH6zg7guDI39Pyzc=;
        b=BVB14axgQKa6pSYvz7NVWDFMQSnw/yQO0odZwwzMZBZvvA6uwkpRAe2ZaUyqsnNERU
         FH8qKDjceL9d5gJ8VN54+R3/CQ1m0Sy6FDE9hfd6Dho2FPQ/IDYAp0gJAtknT6Yc/VUv
         DFlUbMD13Xr8s1t+AnryKnp+RhKQhPkG+TA9I=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=gNoF/7r/lxXPzBZN0QUTuiB8pt4qfuB90uA6Jt/iFHum2i/ICDZSIW3Oifg/ZlTkSn
         x+yU0Ruk9N2WqgHSV34uNIG7hr6FixnaSKhF3/TcH5pstiApoR1/iBpw2KAHiQIyOl6J
         lw9wcDWuwyFEgsArmTLCwL9Y8RK7wSqg7f340=
MIME-Version: 1.0
Received: by 10.220.74.85 with SMTP id t21mr13951979vcj.75.1247054791804; Wed, 
	08 Jul 2009 05:06:31 -0700 (PDT)
Date: Wed, 8 Jul 2009 17:36:31 +0530
Message-ID: <794f042d0907080506h38ff3392tabe85b6d5fb9e243@mail.gmail.com>
Subject: Wrong FS error
From: Saurabh Nanda <saurabhnanda@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364ed584e18b55046e3091ce
X-Virus-Checked: Checked by ClamAV on apache.org

--0016364ed584e18b55046e3091ce
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi,

A lot of my Hadop jobs (actually Cloudbase queries) are showing the
following kind of errors in the job tracker:

java.lang.IllegalArgumentException: Wrong FS:
hdfs://master-hadoop.local/user/ct-admin/cloudbase/index/UID_INDEX/index_new/metadata/1,
expected: hdfs://master-hadoop

My configuration is:

* master-hadoop: a machine running the datanode & job tracker
* slave1-hadoop: a slave (running the task tracker, right?)

All of my map jobs on the slave1-hadoop are failing with the error given
above. Nowhere in my configuration file have I specified master-hadoop or
master-hadoop.local. I have always used external IP addresses.

Any help will be highly appreciated. I'm on Hadoop 0.18.3

Thanks,
Saurabh.

-- 
http://nandz.blogspot.com
http://foodieforlife.blogspot.com

--0016364ed584e18b55046e3091ce--

From general-return-352-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 13:14:44 2009
Return-Path: <general-return-352-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 40036 invoked from network); 8 Jul 2009 13:14:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 13:14:43 -0000
Received: (qmail 51820 invoked by uid 500); 8 Jul 2009 13:14:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51731 invoked by uid 500); 8 Jul 2009 13:14:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51721 invoked by uid 99); 8 Jul 2009 13:14:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 13:14:53 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of xinejar22@googlemail.com designates 72.14.220.157 as permitted sender)
Received: from [72.14.220.157] (HELO fg-out-1718.google.com) (72.14.220.157)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 13:14:43 +0000
Received: by fg-out-1718.google.com with SMTP id l26so1134462fgb.12
        for <general@hadoop.apache.org>; Wed, 08 Jul 2009 06:14:22 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=17lFI+MsWAcxNFI5f1JPFAFGbNJfbnbrwzuzgE7o2tI=;
        b=uMvSVcfOowiP3i1mCMlOKh8lupGVarlZQN9fG2sJ/K8ZSU2lz9LK2qmQhpPU/GNk4g
         W63lPBLjlqk2zS6qlgsxPEIv2swVfC0dEQVqmnMQJ73iXDWKoYLIb40EK520CxEZdFOx
         qdsXzy7kBXVGeztPAHzNFVZQ8xXP1x7WvsSLo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=dVl5Fo+l2WuIXwWY7V0lP8/BErpQwipTpbtRmZm6Bzg+IFODyzm00C6kr1w2Eixa/B
         UBEpu06cV2Y4GC1o175aJh6R4WgRQsbKOa685wfcdLPnUK3DRqUhbQrnHMlHrsZtJDiZ
         +hHwPszoxkAzoLN75rIkWB9yQcWJP3m3rC9m0=
MIME-Version: 1.0
Received: by 10.86.98.16 with SMTP id v16mr2842110fgb.40.1247058862199; Wed, 
	08 Jul 2009 06:14:22 -0700 (PDT)
In-Reply-To: <115373b00907080212n12ae70d6oc3a9c7a3c2a02fbd@mail.gmail.com>
References: <115373b00907060034t170c14afmb3aef33260424a75@mail.gmail.com>
	 <fa561940907060111l2d361924g59ee318138a46402@mail.gmail.com>
	 <115373b00907060524p1fcb996cq2bbb86c2fe85bd5a@mail.gmail.com>
	 <fa561940907061946g672275c7k9c5adef6230b2f62@mail.gmail.com>
	 <fa561940907061947x5ffe409cg361de4deaa0c358e@mail.gmail.com>
	 <115373b00907080212n12ae70d6oc3a9c7a3c2a02fbd@mail.gmail.com>
Date: Wed, 8 Jul 2009 15:14:22 +0200
Message-ID: <115373b00907080614k7c8cb98eq9ed056ef47b64758@mail.gmail.com>
Subject: Re: problem with starting the Jobtracker and thenamenode
From: Xine Jar <xinejar22@googlemail.com>
To: general@hadoop.apache.org, xinejar22@googlemail.com
Content-Type: multipart/alternative; boundary=000e0cd24d747ed667046e318487
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd24d747ed667046e318487
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Thank you all for your great help,

I finally discovered a typing error in one of my configuration files.
Therefore my jobtracker was not able to bind the address to the port.

On the other hand I still have the Namenode which cannot be started. The log
file shows that the namenode is not formatted although I did this. I
appreciate it if someone can point to me the folders I can delete in order
to be able to reformat the node without deleting some critical information.
please note that I have deleted once the /tmp folder and tried to reformat
but this did not help !!

Vielen Dank,
CJ

On Wed, Jul 8, 2009 at 11:12 AM, Xine Jar <xinejar22@googlemail.com> wrote:

> Hallo,
> I am still struggling with the same problem. In order to be sure that IPv6
> is not creating a problem and that the JVM is not suffering from a bug, I
> have written a TCP client/server program in Java, I let the server run on
> the same machine that was creating the binding problem and using the same
> port number. Luckily or unfortunately, the Java server could bind the
> address normally to the port, the communication between the client and the
> server was successful, and the netstat command shows that the port is used.
>
> I order to be sure that the problem is not a bug for hadoop I have
> installed the version 0.20.0,
> copied the same configuration and still have the same problem.
>
> *I have few questions: *
> As I have previously mentioned, the namenode cannot be started and java
> complains that the node has not been formated. The other problem is with the
> jobtracker which is giving a binding error on the address:port.
>
> *Q1:* Is it possible that the second problem (jobtracker problem) is
> appearing because of the namenode problem?
>
> If it is a yes, how can I solve this? I have actually deleted the /tmp
> folder and reformated the node but the formatting error persists!! Do I have
> to do something else?
>
> If it is a no,  am I doing something stupid here in any of the
> configuration files?!!!
>
> *Q2:  *In order to kill all the instances of the dfs and the mapred, is it
> enough to execute the bin/stop-dfs.sh and bin/stop-mapred.sh on the
> namenode? Is it possible that the command line is showing that there is no
> jobtracker, no namenode, ...... but in fact the port is occupied?
>
> P.S: -Whether hadoop is running or not, the netstat command is showing
> always that the port
>          is free!!!!!
>        - My OS is a openSuse 9*
>
> *I am trying to debug everything, because I am somehow sick of it, It is
> certainly something stupid and most probably my fault (or my configuration
> or my way of running things) , since others managed to run hadoop on a
> cluster. I would appreciate any idea that helps to discover this the
> problem.
>
> Thank you,
> CJ
>
>
>
> On Tue, Jul 7, 2009 at 4:47 AM, Bogdan M. Maryniuk <
> bogdan.maryniuk@gmail.com> wrote:
>
>> On Mon, Jul 6, 2009 at 9:24 PM, Xine Jar<xinejar22@googlemail.com> wrote:
>> > 2.How can I check that ipv6 is really disabled on my JVM?
>>
>> You've mentioned it is Linux, but how do I know what distribution of
>> Linux's zoo you use?
>>
>> On Ubuntu it is sort of like this:
>> In the file "/etc/modprobe.d/aliases" find "alias net-pf-10 ipv6",
>> remove it and add the following:
>>
>> alias net-pf-10 off
>> alias ipv6 off
>>
>> Then reboot (welcome to the Linux).
>>
>> On RedHat and derivatives it is like this:
>> echo "alias net-pf-10 off" >>  /etc/modprobe.conf <ENTER>
>>
>> Then reboot (insert your ironic joke here). :-)
>>
>>
>> > and in which file can I insert the flag -Djava.net.preferIPv4Stack=true?
>> It is just a JVM parameter for a networking:
>> 1. http://java.sun.com/j2se/1.4.2/docs/guide/net/properties.html
>> 2.
>> http://java.sun.com/j2se/1.4.2/docs/guide/net/ipv6_guide/#ipv6-networking
>>
>> See $HADOOP/conf/hadoop-env.sh and look for HADOOP_OPTS that is
>> commented out by default.
>>
>> --
>> Kind regards, BM
>>
>> Things, that are stupid at the beginning, rarely ends up wisely.
>>
>
>

--000e0cd24d747ed667046e318487--

From general-return-353-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 13:25:56 2009
Return-Path: <general-return-353-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49214 invoked from network); 8 Jul 2009 13:25:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 13:25:56 -0000
Received: (qmail 77561 invoked by uid 500); 8 Jul 2009 13:26:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77501 invoked by uid 500); 8 Jul 2009 13:26:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77491 invoked by uid 99); 8 Jul 2009 13:26:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 13:26:05 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jason.hadoop@gmail.com designates 209.85.212.196 as permitted sender)
Received: from [209.85.212.196] (HELO mail-vw0-f196.google.com) (209.85.212.196)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 13:25:55 +0000
Received: by vwj34 with SMTP id 34so538832vwj.5
        for <general@hadoop.apache.org>; Wed, 08 Jul 2009 06:25:33 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=84mVGZxC2HUJDzqu8JIZc6v+CydieyTc5wr5TT3mdmU=;
        b=u6bwnNiJiMFRNvZErAA4196yOZZhFQEkjmhSfoOTqcsh3mSgVReCCzVbN6LC1Oe8wJ
         vOKcTUglwjkKwFJUMyPZ/qIV30VC5fJ82CYde+a+YYa2Klbk0dPtfC/d2Km1+d1Ee9yD
         a6iukvHkga3jWjp7Txn72fnpnfI368b0dTpYg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=OesVhUdfZ50FX5oo2nksBUJqYK5FAlg0H93KvKIrn7/QhuBlmibKkJygidh6tuyQfX
         ozDofzXhYSSHS8c+O9M0Tve3kImV680WFnM+5oGdxF05IDCXJTW22A3vpTSf7bHPLsIt
         ocSI7/PowVIbnu6KTMnzNIFlDnZV5zlrCXsio=
MIME-Version: 1.0
Received: by 10.220.77.17 with SMTP id e17mr14175649vck.3.1247059532924; Wed, 
	08 Jul 2009 06:25:32 -0700 (PDT)
In-Reply-To: <115373b00907080614k7c8cb98eq9ed056ef47b64758@mail.gmail.com>
References: <115373b00907060034t170c14afmb3aef33260424a75@mail.gmail.com>
	 <fa561940907060111l2d361924g59ee318138a46402@mail.gmail.com>
	 <115373b00907060524p1fcb996cq2bbb86c2fe85bd5a@mail.gmail.com>
	 <fa561940907061946g672275c7k9c5adef6230b2f62@mail.gmail.com>
	 <fa561940907061947x5ffe409cg361de4deaa0c358e@mail.gmail.com>
	 <115373b00907080212n12ae70d6oc3a9c7a3c2a02fbd@mail.gmail.com>
	 <115373b00907080614k7c8cb98eq9ed056ef47b64758@mail.gmail.com>
Date: Wed, 8 Jul 2009 06:25:32 -0700
Message-ID: <314098690907080625q11360a8fhba59103f00a8aae1@mail.gmail.com>
Subject: Re: problem with starting the Jobtracker and thenamenode
From: jason hadoop <jason.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64eabf4794f08046e31acdc
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e64eabf4794f08046e31acdc
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

You may need to delete the directory that you have configured for dfs
storage on all of the machines in your cluster.
The other issue that may happen is that the user that owns the
Namenode/Datanode processes does not have write permissions on the
filesystem structure to be used for dfs storage.

Does the namenode format command complete successfully?

On Wed, Jul 8, 2009 at 6:14 AM, Xine Jar <xinejar22@googlemail.com> wrote:

> Thank you all for your great help,
>
> I finally discovered a typing error in one of my configuration files.
> Therefore my jobtracker was not able to bind the address to the port.
>
> On the other hand I still have the Namenode which cannot be started. The
> log
> file shows that the namenode is not formatted although I did this. I
> appreciate it if someone can point to me the folders I can delete in order
> to be able to reformat the node without deleting some critical information.
> please note that I have deleted once the /tmp folder and tried to reformat
> but this did not help !!
>
> Vielen Dank,
> CJ
>
> On Wed, Jul 8, 2009 at 11:12 AM, Xine Jar <xinejar22@googlemail.com>
> wrote:
>
> > Hallo,
> > I am still struggling with the same problem. In order to be sure that
> IPv6
> > is not creating a problem and that the JVM is not suffering from a bug, I
> > have written a TCP client/server program in Java, I let the server run on
> > the same machine that was creating the binding problem and using the same
> > port number. Luckily or unfortunately, the Java server could bind the
> > address normally to the port, the communication between the client and
> the
> > server was successful, and the netstat command shows that the port is
> used.
> >
> > I order to be sure that the problem is not a bug for hadoop I have
> > installed the version 0.20.0,
> > copied the same configuration and still have the same problem.
> >
> > *I have few questions: *
> > As I have previously mentioned, the namenode cannot be started and java
> > complains that the node has not been formated. The other problem is with
> the
> > jobtracker which is giving a binding error on the address:port.
> >
> > *Q1:* Is it possible that the second problem (jobtracker problem) is
> > appearing because of the namenode problem?
> >
> > If it is a yes, how can I solve this? I have actually deleted the /tmp
> > folder and reformated the node but the formatting error persists!! Do I
> have
> > to do something else?
> >
> > If it is a no,  am I doing something stupid here in any of the
> > configuration files?!!!
> >
> > *Q2:  *In order to kill all the instances of the dfs and the mapred, is
> it
> > enough to execute the bin/stop-dfs.sh and bin/stop-mapred.sh on the
> > namenode? Is it possible that the command line is showing that there is
> no
> > jobtracker, no namenode, ...... but in fact the port is occupied?
> >
> > P.S: -Whether hadoop is running or not, the netstat command is showing
> > always that the port
> >          is free!!!!!
> >        - My OS is a openSuse 9*
> >
> > *I am trying to debug everything, because I am somehow sick of it, It is
> > certainly something stupid and most probably my fault (or my
> configuration
> > or my way of running things) , since others managed to run hadoop on a
> > cluster. I would appreciate any idea that helps to discover this the
> > problem.
> >
> > Thank you,
> > CJ
> >
> >
> >
> > On Tue, Jul 7, 2009 at 4:47 AM, Bogdan M. Maryniuk <
> > bogdan.maryniuk@gmail.com> wrote:
> >
> >> On Mon, Jul 6, 2009 at 9:24 PM, Xine Jar<xinejar22@googlemail.com>
> wrote:
> >> > 2.How can I check that ipv6 is really disabled on my JVM?
> >>
> >> You've mentioned it is Linux, but how do I know what distribution of
> >> Linux's zoo you use?
> >>
> >> On Ubuntu it is sort of like this:
> >> In the file "/etc/modprobe.d/aliases" find "alias net-pf-10 ipv6",
> >> remove it and add the following:
> >>
> >> alias net-pf-10 off
> >> alias ipv6 off
> >>
> >> Then reboot (welcome to the Linux).
> >>
> >> On RedHat and derivatives it is like this:
> >> echo "alias net-pf-10 off" >>  /etc/modprobe.conf <ENTER>
> >>
> >> Then reboot (insert your ironic joke here). :-)
> >>
> >>
> >> > and in which file can I insert the flag
> -Djava.net.preferIPv4Stack=true?
> >> It is just a JVM parameter for a networking:
> >> 1. http://java.sun.com/j2se/1.4.2/docs/guide/net/properties.html
> >> 2.
> >>
> http://java.sun.com/j2se/1.4.2/docs/guide/net/ipv6_guide/#ipv6-networking
> >>
> >> See $HADOOP/conf/hadoop-env.sh and look for HADOOP_OPTS that is
> >> commented out by default.
> >>
> >> --
> >> Kind regards, BM
> >>
> >> Things, that are stupid at the beginning, rarely ends up wisely.
> >>
> >
> >
>



-- 
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals

--0016e64eabf4794f08046e31acdc--

From general-return-354-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 15:11:40 2009
Return-Path: <general-return-354-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63258 invoked from network); 8 Jul 2009 15:11:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 15:11:39 -0000
Received: (qmail 97876 invoked by uid 500); 8 Jul 2009 15:11:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97829 invoked by uid 500); 8 Jul 2009 15:11:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97819 invoked by uid 99); 8 Jul 2009 15:11:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 15:11:49 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of xinejar22@googlemail.com designates 72.14.220.159 as permitted sender)
Received: from [72.14.220.159] (HELO fg-out-1718.google.com) (72.14.220.159)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 15:11:37 +0000
Received: by fg-out-1718.google.com with SMTP id l26so1158677fgb.12
        for <general@hadoop.apache.org>; Wed, 08 Jul 2009 08:11:17 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=RZgZthw4YbGBHDGzdX1L0z0Dhi2rdFlZND3zjgHKly0=;
        b=qJiItuEhrBXetmr2O6co29TmtZ4wLelYrL1WWRybZ4OpT1IQPgA60E1M+fBpxSRmI0
         HEx+FYUcG1v8drMNujRjVNejn49F5ytNOY4b91/PWiXx2Wgk8KThQc53AR5WmgUpNUSv
         7oaSoTTfyxi7fm2Z5edZvWPMgHeo/g0+hjJrE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=vlwC7bGqBzo8h1St1HHTG96E0pcYuFWMwjC/MwhW/wyRJfcC/08XK3Y3C6yrD33CQa
         BD4NBWXegoWiihcjIMcLblHdhxvUHs1HllbGODCgp3SLuPSt5av3stpnfwBF4xWC/DFB
         dfqtPIVUvoTT7Rk4SW9aycMPttrU5BuI78/vk=
MIME-Version: 1.0
Received: by 10.86.74.1 with SMTP id w1mr2799106fga.76.1247065877399; Wed, 08 
	Jul 2009 08:11:17 -0700 (PDT)
In-Reply-To: <314098690907080625q11360a8fhba59103f00a8aae1@mail.gmail.com>
References: <115373b00907060034t170c14afmb3aef33260424a75@mail.gmail.com>
	 <fa561940907060111l2d361924g59ee318138a46402@mail.gmail.com>
	 <115373b00907060524p1fcb996cq2bbb86c2fe85bd5a@mail.gmail.com>
	 <fa561940907061946g672275c7k9c5adef6230b2f62@mail.gmail.com>
	 <fa561940907061947x5ffe409cg361de4deaa0c358e@mail.gmail.com>
	 <115373b00907080212n12ae70d6oc3a9c7a3c2a02fbd@mail.gmail.com>
	 <115373b00907080614k7c8cb98eq9ed056ef47b64758@mail.gmail.com>
	 <314098690907080625q11360a8fhba59103f00a8aae1@mail.gmail.com>
Date: Wed, 8 Jul 2009 17:11:17 +0200
Message-ID: <115373b00907080811v2c29027cn7867f73137a55147@mail.gmail.com>
Subject: Re: problem with starting the Jobtracker and thenamenode
From: Xine Jar <xinejar22@googlemail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd24e52a24b2d046e332620
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd24e52a24b2d046e332620
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Great thank you,
I can see now that all the nodes started and I shall try to run the
WordCount v1.0 example.Let' s see.

I would also like to know in which case do you advice reformatting the node?
Do I ever need to use this command again?

and concerning the book is there a possibility to know roughly the table of
content for each of the following books: ProHadoop, and Hadoop the
definitive guide? As you see I am starting basically from 0 with hadoop and
would like to reach at point where I can  write my own java  program/my own
search query. Which of the books would you advice ?

Thank you very much for your help


On Wed, Jul 8, 2009 at 3:25 PM, jason hadoop <jason.hadoop@gmail.com> wrote:

> You may need to delete the directory that you have configured for dfs
> storage on all of the machines in your cluster.
> The other issue that may happen is that the user that owns the
> Namenode/Datanode processes does not have write permissions on the
> filesystem structure to be used for dfs storage.
>
> Does the namenode format command complete successfully?
>
> On Wed, Jul 8, 2009 at 6:14 AM, Xine Jar <xinejar22@googlemail.com> wrote:
>
> > Thank you all for your great help,
> >
> > I finally discovered a typing error in one of my configuration files.
> > Therefore my jobtracker was not able to bind the address to the port.
> >
> > On the other hand I still have the Namenode which cannot be started. The
> > log
> > file shows that the namenode is not formatted although I did this. I
> > appreciate it if someone can point to me the folders I can delete in
> order
> > to be able to reformat the node without deleting some critical
> information.
> > please note that I have deleted once the /tmp folder and tried to
> reformat
> > but this did not help !!
> >
> > Vielen Dank,
> > CJ
> >
> > On Wed, Jul 8, 2009 at 11:12 AM, Xine Jar <xinejar22@googlemail.com>
> > wrote:
> >
> > > Hallo,
> > > I am still struggling with the same problem. In order to be sure that
> > IPv6
> > > is not creating a problem and that the JVM is not suffering from a bug,
> I
> > > have written a TCP client/server program in Java, I let the server run
> on
> > > the same machine that was creating the binding problem and using the
> same
> > > port number. Luckily or unfortunately, the Java server could bind the
> > > address normally to the port, the communication between the client and
> > the
> > > server was successful, and the netstat command shows that the port is
> > used.
> > >
> > > I order to be sure that the problem is not a bug for hadoop I have
> > > installed the version 0.20.0,
> > > copied the same configuration and still have the same problem.
> > >
> > > *I have few questions: *
> > > As I have previously mentioned, the namenode cannot be started and java
> > > complains that the node has not been formated. The other problem is
> with
> > the
> > > jobtracker which is giving a binding error on the address:port.
> > >
> > > *Q1:* Is it possible that the second problem (jobtracker problem) is
> > > appearing because of the namenode problem?
> > >
> > > If it is a yes, how can I solve this? I have actually deleted the /tmp
> > > folder and reformated the node but the formatting error persists!! Do I
> > have
> > > to do something else?
> > >
> > > If it is a no,  am I doing something stupid here in any of the
> > > configuration files?!!!
> > >
> > > *Q2:  *In order to kill all the instances of the dfs and the mapred, is
> > it
> > > enough to execute the bin/stop-dfs.sh and bin/stop-mapred.sh on the
> > > namenode? Is it possible that the command line is showing that there is
> > no
> > > jobtracker, no namenode, ...... but in fact the port is occupied?
> > >
> > > P.S: -Whether hadoop is running or not, the netstat command is showing
> > > always that the port
> > >          is free!!!!!
> > >        - My OS is a openSuse 9*
> > >
> > > *I am trying to debug everything, because I am somehow sick of it, It
> is
> > > certainly something stupid and most probably my fault (or my
> > configuration
> > > or my way of running things) , since others managed to run hadoop on a
> > > cluster. I would appreciate any idea that helps to discover this the
> > > problem.
> > >
> > > Thank you,
> > > CJ
> > >
> > >
> > >
> > > On Tue, Jul 7, 2009 at 4:47 AM, Bogdan M. Maryniuk <
> > > bogdan.maryniuk@gmail.com> wrote:
> > >
> > >> On Mon, Jul 6, 2009 at 9:24 PM, Xine Jar<xinejar22@googlemail.com>
> > wrote:
> > >> > 2.How can I check that ipv6 is really disabled on my JVM?
> > >>
> > >> You've mentioned it is Linux, but how do I know what distribution of
> > >> Linux's zoo you use?
> > >>
> > >> On Ubuntu it is sort of like this:
> > >> In the file "/etc/modprobe.d/aliases" find "alias net-pf-10 ipv6",
> > >> remove it and add the following:
> > >>
> > >> alias net-pf-10 off
> > >> alias ipv6 off
> > >>
> > >> Then reboot (welcome to the Linux).
> > >>
> > >> On RedHat and derivatives it is like this:
> > >> echo "alias net-pf-10 off" >>  /etc/modprobe.conf <ENTER>
> > >>
> > >> Then reboot (insert your ironic joke here). :-)
> > >>
> > >>
> > >> > and in which file can I insert the flag
> > -Djava.net.preferIPv4Stack=true?
> > >> It is just a JVM parameter for a networking:
> > >> 1. http://java.sun.com/j2se/1.4.2/docs/guide/net/properties.html
> > >> 2.
> > >>
> >
> http://java.sun.com/j2se/1.4.2/docs/guide/net/ipv6_guide/#ipv6-networking
> > >>
> > >> See $HADOOP/conf/hadoop-env.sh and look for HADOOP_OPTS that is
> > >> commented out by default.
> > >>
> > >> --
> > >> Kind regards, BM
> > >>
> > >> Things, that are stupid at the beginning, rarely ends up wisely.
> > >>
> > >
> > >
> >
>
>
>
> --
> Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> http://www.amazon.com/dp/1430219424?tag=jewlerymall
> www.prohadoopbook.com a community for Hadoop Professionals
>

--000e0cd24e52a24b2d046e332620--

From general-return-355-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 15:46:19 2009
Return-Path: <general-return-355-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39607 invoked from network); 8 Jul 2009 15:46:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 15:46:19 -0000
Received: (qmail 65978 invoked by uid 500); 8 Jul 2009 15:46:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65918 invoked by uid 500); 8 Jul 2009 15:46:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65908 invoked by uid 99); 8 Jul 2009 15:46:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 15:46:28 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bogdan.maryniuk@gmail.com designates 209.85.216.171 as permitted sender)
Received: from [209.85.216.171] (HELO mail-px0-f171.google.com) (209.85.216.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 15:46:20 +0000
Received: by pxi1 with SMTP id 1so376312pxi.5
        for <general@hadoop.apache.org>; Wed, 08 Jul 2009 08:46:00 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=TGOt5dVUif1efWqJWTCIjEPAIxJKluXA8fs6BgtSNak=;
        b=NtupaPZHr0jSJ/2Czwksbj0D7n8dUPgx9o718o6ZKduP+rpnLmB/kg8UHFUD2ztX2Z
         4kOrPwyWNso7P6JXjVxtrgxZiaG0vfF1A9hqpxpGRp5iAFJNWKtK5LGfGAZ7m8imJS/z
         97EFLdlVSCkTO3mo6ptfDfwHAhHTN9lmM4bUA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=sqGoMjrn7Y1CjxCB8fcKQmgvJak7b+8zgMSTh7xX8uQJDy6zKw5PJdNySvhr5LvQMU
         Wz3DqEepsA34u3NUT6Bu3i12I4WUUt7Q7kQxwk4ldjfALxRgx++9D/IvtJQL5i0RGn+m
         tI+xHF8D1iTwDAdLuoIyJ1C+G3t59DcPdRMD0=
MIME-Version: 1.0
Received: by 10.142.76.16 with SMTP id y16mr2373761wfa.346.1247067960434; Wed, 
	08 Jul 2009 08:46:00 -0700 (PDT)
In-Reply-To: <794f042d0907080506h38ff3392tabe85b6d5fb9e243@mail.gmail.com>
References: <794f042d0907080506h38ff3392tabe85b6d5fb9e243@mail.gmail.com>
Date: Thu, 9 Jul 2009 00:46:00 +0900
Message-ID: <fa561940907080846v32444958n59a1a9920feefc1c@mail.gmail.com>
Subject: Re: Wrong FS error
From: "Bogdan M. Maryniuk" <bogdan.maryniuk@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On Wed, Jul 8, 2009 at 9:06 PM, Saurabh Nanda<saurabhnanda@gmail.com> wrote:
> java.lang.IllegalArgumentException: Wrong FS:
> hdfs://master-hadoop.local/user/ct-admin/cloudbase/index/UID_INDEX/index_new/metadata/1,
> expected: hdfs://master-hadoop

Are you sure configs are OK? Looks like you add ".local" suffix on one
node, and not on another...

-- 
Kind regards, BM

Things, that are stupid at the beginning, rarely ends up wisely.

From general-return-356-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 15:48:20 2009
Return-Path: <general-return-356-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 41003 invoked from network); 8 Jul 2009 15:48:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 15:48:19 -0000
Received: (qmail 67762 invoked by uid 500); 8 Jul 2009 15:48:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67672 invoked by uid 500); 8 Jul 2009 15:48:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67662 invoked by uid 99); 8 Jul 2009 15:48:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 15:48:29 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bogdan.maryniuk@gmail.com designates 209.85.200.173 as permitted sender)
Received: from [209.85.200.173] (HELO wf-out-1314.google.com) (209.85.200.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 15:48:21 +0000
Received: by wf-out-1314.google.com with SMTP id 23so1907802wfg.2
        for <general@hadoop.apache.org>; Wed, 08 Jul 2009 08:48:01 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=8nwE70k/JXKOZAmkATL6azffK+eSJ6Nf2BWqu5hPiyA=;
        b=mxKCwsXNmfVrZF2S1rX1mf1lWF4Z5xPGTx1eo8ZzN/jzXUjSX6hp9Yvvzsu5aSTOB5
         ghMEN2rMHgI75kEKRA+pGh/87wyXfrs//YEeIwocFEY22/De7ubzSwuE4JTVGqSWKQJh
         mBR4Vjj91sGyk3UjDpXSnktNnCJn8oxeZIf8A=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=J6X+sfyEjIEhphZ3ajN4TXmQXpmYQU1CQNY48uTdRDDEEVpFM6umXYirbH9tj1/09C
         is3E1oZWg/LacOyeK1lKeZUdztlbtnpdJqwEnwV5JjBMH3Lu8huQavlZm0ddjvd7M6xg
         2o9Xof+tOz72LUqofNG0tvrod7FEHZ+8fPR/4=
MIME-Version: 1.0
Received: by 10.142.51.1 with SMTP id y1mr2577363wfy.237.1247068081224; Wed, 
	08 Jul 2009 08:48:01 -0700 (PDT)
In-Reply-To: <794f042d0907080506h38ff3392tabe85b6d5fb9e243@mail.gmail.com>
References: <794f042d0907080506h38ff3392tabe85b6d5fb9e243@mail.gmail.com>
Date: Thu, 9 Jul 2009 00:48:01 +0900
Message-ID: <fa561940907080848o640fe841h48cf2d4f0b69f745@mail.gmail.com>
Subject: Re: Wrong FS error
From: "Bogdan M. Maryniuk" <bogdan.maryniuk@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On Wed, Jul 8, 2009 at 9:06 PM, Saurabh Nanda<saurabhnanda@gmail.com> wrote:
> Hi,
>
> A lot of my Hadop jobs (actually Cloudbase queries) are showing the
> following kind of errors in the job tracker:
>
> java.lang.IllegalArgumentException: Wrong FS:
> hdfs://master-hadoop.local/user/ct-admin/cloudbase/index/UID_INDEX/index_new/metadata/1,
> expected: hdfs://master-hadoop

P.S. Sounds also like you have no DNS inside but using /etc/hosts that
might be just wrong. Check host names on each and make sure nslookup
shows you identical everywhere.

Hope, this helps.

-- 
Kind regards, BM

Things, that are stupid at the beginning, rarely ends up wisely.

From general-return-357-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 19:57:45 2009
Return-Path: <general-return-357-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21992 invoked from network); 8 Jul 2009 19:57:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 19:57:45 -0000
Received: (qmail 70976 invoked by uid 500); 8 Jul 2009 19:57:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70766 invoked by uid 500); 8 Jul 2009 19:57:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70726 invoked by uid 99); 8 Jul 2009 19:57:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 19:57:53 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [207.126.228.150] (HELO rsmtp2.corp.yahoo.com) (207.126.228.150)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 19:57:44 +0000
Received: from [172.21.148.225] (nat-dip10.cfw-b-gci.corp.yahoo.com [209.131.62.145])
	(authenticated bits=0)
	by rsmtp2.corp.yahoo.com (8.13.8/8.13.8/y.rout) with ESMTP id n68JvAZM090582
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Wed, 8 Jul 2009 12:57:11 -0700 (PDT)
Message-ID: <4A54FA15.6000308@apache.org>
Date: Wed, 08 Jul 2009 12:57:09 -0700
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.22 (X11/20090608)
MIME-Version: 1.0
To: announce@apache.org,
        "zookeeper-dev@hadoop.apache.org" <zookeeper-dev@hadoop.apache.org>,
        "zookeeper-user@hadoop.apache.org" <zookeeper-user@hadoop.apache.org>,
        general@hadoop.apache.org
Subject: [ANNOUNCE] Apache ZooKeeper 3.2.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

The Apache ZooKeeper team is proud to announce Apache ZooKeeper version 
3.2.0.

ZooKeeper is a high-performance coordination service for distributed 
applications. It exposes common services - such as naming, configuration 
management, synchronization, and group services - in a simple interface 
so you don't have to write them from scratch. You can use it 
off-the-shelf to implement consensus, group management, leader election, 
and presence protocols. And you can build on it for your own, specific 
needs.

Key features of the 3.2.0 release:
* client side bindings for; Perl, Python, & REST
* flexible quorum support
* re-usable recipe code libraries
* chroot support in connect string
* many fixes, improvements, improved documentation, etc...

A number of optimizations have gone into this release and our benchmarks 
show that read/write performance of version 3.2.0 is approximately twice 
that of the previous 3.1.0 version!

For ZooKeeper release details and downloads, visit:
http://hadoop.apache.org/zookeeper/releases.html

ZooKeeper 3.2.0 Release Notes are at:
http://hadoop.apache.org/zookeeper/docs/r3.2.0/releasenotes.html

Regards,

The ZooKeeper Team


From general-return-358-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 04:15:43 2009
Return-Path: <general-return-358-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22452 invoked from network); 9 Jul 2009 04:15:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 04:15:43 -0000
Received: (qmail 28090 invoked by uid 500); 9 Jul 2009 04:15:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27984 invoked by uid 500); 9 Jul 2009 04:15:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27974 invoked by uid 99); 9 Jul 2009 04:15:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 04:15:51 +0000
X-ASF-Spam-Status: No, hits=3.7 required=10.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saurabhnanda@gmail.com designates 209.85.212.196 as permitted sender)
Received: from [209.85.212.196] (HELO mail-vw0-f196.google.com) (209.85.212.196)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 04:15:40 +0000
Received: by vwj34 with SMTP id 34so1058026vwj.5
        for <general@hadoop.apache.org>; Wed, 08 Jul 2009 21:15:19 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=DXO1iQFsK50fPa8Vtd2gGvfBHXEaFGc6q2xY7NAtP+I=;
        b=qHz7w7+Lh2v37GlL8v5+FGsAD93HvF5ijXf96aMl+vhxo0QvlTjSRj/44+DHyCJG7b
         Vy8yhMcacmWmnID+doUea2pSpRitese2C4po/ARZRcd6c9TdDMFoap5Tw1hCd1q+I5gr
         yoC2lJtYTKmoDUpgtYyXYZ4gI0JRk5QFcoYhE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=mWA2CJALY5oHQ7YrtTym0+eSyC0NcvuEpxdZ2JilAVDX3bQ9/cdHsUTVmbLJYLk4MR
         e0XhIGsfMsCPVsYHrb2dof4hbuDitjDHjXsrmiBS4DzDhVxZKA510XnpEho7pZMxHLqw
         OMgxB5O+anIGeFwpnpzC1UcqL3QboV69Fro5s=
MIME-Version: 1.0
Received: by 10.220.81.70 with SMTP id w6mr438852vck.32.1247112918102; Wed, 08 
	Jul 2009 21:15:18 -0700 (PDT)
In-Reply-To: <fa561940907080848o640fe841h48cf2d4f0b69f745@mail.gmail.com>
References: <794f042d0907080506h38ff3392tabe85b6d5fb9e243@mail.gmail.com>
	 <fa561940907080848o640fe841h48cf2d4f0b69f745@mail.gmail.com>
Date: Thu, 9 Jul 2009 09:45:18 +0530
Message-ID: <794f042d0907082115q75692e5an527d4284eab61c83@mail.gmail.com>
Subject: Re: Wrong FS error
From: Saurabh Nanda <saurabhnanda@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e647f0267aa6d5046e3e1a51
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e647f0267aa6d5046e3e1a51
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

> >
> > A lot of my Hadop jobs (actually Cloudbase queries) are showing the
> > following kind of errors in the job tracker:
> >
> > java.lang.IllegalArgumentException: Wrong FS:
> >
> hdfs://master-hadoop.local/user/ct-admin/cloudbase/index/UID_INDEX/index_new/metadata/1,
> expected: hdfs://master-hadoop
>
> P.S. Sounds also like you have no DNS inside but using /etc/hosts that
> might be just wrong. Check host names on each and make sure nslookup
> shows you identical everywhere.



You are right. I am using /etc/hosts and my hadoop machines do not have
proper DNS entries. However, why should that matter if I am using IP address
in the configuration files?

Relevant entries from hadoop-site.xml:
fs.default.name=hdfs://172.16.37.56:8020
mapred.job.tracker=172.16.37.56:8021

slaves.xml:
172.16.37.56
172.16.37.72

/etc/hosts on 172.16.37.56:
127.0.0.1    master-hadoop    localhost.localdomain    localhost
172.16.37.72 slave1-hadoop

/etc/hosts on 172.16.37.72:
127.0.0.1    slave1-hadoop    localhost.localdomain    localhost
172.16.37.56 master-hadoop

How should I go about fixing this?

Saurabh.
-- 
http://nandz.blogspot.com
http://foodieforlife.blogspot.com

--0016e647f0267aa6d5046e3e1a51--

From general-return-359-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 09:45:41 2009
Return-Path: <general-return-359-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93123 invoked from network); 9 Jul 2009 09:45:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 09:45:41 -0000
Received: (qmail 73881 invoked by uid 500); 9 Jul 2009 09:45:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73786 invoked by uid 500); 9 Jul 2009 09:45:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73769 invoked by uid 99); 9 Jul 2009 09:45:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 09:45:50 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bogdan.maryniuk@gmail.com designates 74.125.78.24 as permitted sender)
Received: from [74.125.78.24] (HELO ey-out-2122.google.com) (74.125.78.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 09:45:39 +0000
Received: by ey-out-2122.google.com with SMTP id 4so13163eyf.9
        for <general@hadoop.apache.org>; Thu, 09 Jul 2009 02:45:18 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=AqGjcHlJgOifblh74l/falRXNH6D7YqWWSOvV7NmiyM=;
        b=Gyah9NpKxnJArHQ7KP9clsvjY9ZE7G9Dhiwbs9gss/drovgyZY8DyMSMR0YSzxXkJW
         q16uRGEIo4JGX/plYk0oUJP52YZ3qSbcBEBwt7Kn+Ta+VLlPmODvGVIdVN6GBwjZU4gr
         8bJm/HcP5bFC9+wb8BIWxUXkedBZmPOTO1rf8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=NNp71RrJb/Sk31Gvm49aYfgC+Sdfm9r3jTgHW7EUx7YW4DbRuoT0KsGhDmwnLGRHtN
         FLkG0WmXEi4JbAr54cS5DgYKeRlJpbA1/dNHV0xsjEBpN9ThGPO85SA99dFvsVTsWCTW
         s7b5aabPTmixIaxiq0OjSgLzW3zOns6AohtDA=
MIME-Version: 1.0
Received: by 10.210.110.5 with SMTP id i5mr727117ebc.3.1247132718417; Thu, 09 
	Jul 2009 02:45:18 -0700 (PDT)
In-Reply-To: <794f042d0907082115q75692e5an527d4284eab61c83@mail.gmail.com>
References: <794f042d0907080506h38ff3392tabe85b6d5fb9e243@mail.gmail.com>
	 <fa561940907080848o640fe841h48cf2d4f0b69f745@mail.gmail.com>
	 <794f042d0907082115q75692e5an527d4284eab61c83@mail.gmail.com>
Date: Thu, 9 Jul 2009 18:45:17 +0900
Message-ID: <fa561940907090245q5cec3592k37dd7857aba4de6e@mail.gmail.com>
Subject: Re: Wrong FS error
From: "Bogdan M. Maryniuk" <bogdan.maryniuk@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Thu, Jul 9, 2009 at 1:15 PM, Saurabh Nanda<saurabhnanda@gmail.com> wrote=
:
> You are right. I am using /etc/hosts and my hadoop machines do not have
> proper DNS entries. However, why should that matter if I am using IP addr=
ess
> in the configuration files?

Sorry, after I've posted, I found that it was not precisely correct.
:-) What I wanted to say is that your /etc/hosts might be configured
different on machines.

> Relevant entries from hadoop-site.xml:
> fs.default.name=3Dhdfs://172.16.37.56:8020
> mapred.job.tracker=3D172.16.37.56:8021

Well, if you use /etc/hosts, why you bother with IP addresses then?

> slaves.xml:
> 172.16.37.56
> 172.16.37.72

Same here. Use host names instead and make sure they are resolved in
both way correctly. For example, "nslookup master-hadoop.local" should
return you "172.16.37.56", but "nslookup 172.16.37.56" should give you
"master-hadoop.local". Same for slave.

> /etc/hosts on 172.16.37.56:
> 127.0.0.1 =C2=A0 =C2=A0master-hadoop =C2=A0 =C2=A0localhost.localdomain =
=C2=A0 =C2=A0localhost
> 172.16.37.72 slave1-hadoop
>
> /etc/hosts on 172.16.37.72:
> 127.0.0.1 =C2=A0 =C2=A0slave1-hadoop =C2=A0 =C2=A0localhost.localdomain =
=C2=A0 =C2=A0localhost
> 172.16.37.56 master-hadoop
>
> How should I go about fixing this?

Well, they are wrong anyway. For the start, remove "*-hadoop" from
127.0.0.1 on both machines and RTFM about /etc/hosts here:
http://www.geo.arizona.edu/tools/man-cgi?hosts+5 I would suggest you
to go with something like this:

On 172.16.37.72 should be:
--------------------------------------
127.0.0.1	localhost
172.16.37.56 master-hadoop.local master-hadoop


On 172.16.37.56 should be:
--------------------------------------
127.0.0.1	localhost
172.16.37.72 slave1-hadoop.local slave1-hadoop


Are there some more slaves? They have to be managed in the same way.


P.S. It is difficult to manage /etc/hosts and you'd better switch to
local DNS instead.
--=20
Kind regards, BM

Things, that are stupid at the beginning, rarely ends up wisely.

From general-return-360-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 09:49:53 2009
Return-Path: <general-return-360-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95701 invoked from network); 9 Jul 2009 09:49:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 09:49:53 -0000
Received: (qmail 78639 invoked by uid 500); 9 Jul 2009 09:50:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78567 invoked by uid 500); 9 Jul 2009 09:50:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78556 invoked by uid 99); 9 Jul 2009 09:50:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 09:50:03 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of bogdan.maryniuk@gmail.com designates 74.125.78.25 as permitted sender)
Received: from [74.125.78.25] (HELO ey-out-2122.google.com) (74.125.78.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 09:49:53 +0000
Received: by ey-out-2122.google.com with SMTP id 4so13590eyf.9
        for <general@hadoop.apache.org>; Thu, 09 Jul 2009 02:49:33 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=IVaGbJpp1bVl6nqIuGaXpNp8AQpBSZjZthkcf+7mkgw=;
        b=uoHLrsMihhZW5NEJahwMZHV6NZd4/yzMFKXF9caJXfuQB0WUUnK7xu+Fe1lAb+78Ym
         GlSozFbMvpSV+fX5e7Gcek0ZjKO+MgDWYo0lOLIRNuY4WuzRsQJkNDiR+/b2Nv1lJD/c
         IwzljCSGhcCci9pNGJxazbh4KnCykSN6OS1FA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=CLzMMV94ResSlLKdA37M21QHRU5KXc9jEzDREXAPbelm+sHPSRLPtIErKgLI4hGeT+
         y/B4JADBHsEgEdkXGGGFXeU8IF6XddCSOOHWYQLEO6it4N1j7tZ+6FWjfVL5joVkRb7L
         6CwEnnCYw6bS3CtojRWVMXyzskb4Z0OQLnLdc=
MIME-Version: 1.0
Received: by 10.210.17.2 with SMTP id 2mr707512ebq.26.1247132973487; Thu, 09 
	Jul 2009 02:49:33 -0700 (PDT)
In-Reply-To: <115373b00907080811v2c29027cn7867f73137a55147@mail.gmail.com>
References: <115373b00907060034t170c14afmb3aef33260424a75@mail.gmail.com>
	 <fa561940907060111l2d361924g59ee318138a46402@mail.gmail.com>
	 <115373b00907060524p1fcb996cq2bbb86c2fe85bd5a@mail.gmail.com>
	 <fa561940907061946g672275c7k9c5adef6230b2f62@mail.gmail.com>
	 <fa561940907061947x5ffe409cg361de4deaa0c358e@mail.gmail.com>
	 <115373b00907080212n12ae70d6oc3a9c7a3c2a02fbd@mail.gmail.com>
	 <115373b00907080614k7c8cb98eq9ed056ef47b64758@mail.gmail.com>
	 <314098690907080625q11360a8fhba59103f00a8aae1@mail.gmail.com>
	 <115373b00907080811v2c29027cn7867f73137a55147@mail.gmail.com>
Date: Thu, 9 Jul 2009 18:49:33 +0900
Message-ID: <fa561940907090249s762ae3doc8cf1d682133ac4f@mail.gmail.com>
Subject: Re: problem with starting the Jobtracker and thenamenode
From: "Bogdan M. Maryniuk" <bogdan.maryniuk@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On Thu, Jul 9, 2009 at 12:11 AM, Xine Jar<xinejar22@googlemail.com> wrote:
> Which of the books would you advice ?

google...  :-)


-- 
Kind regards, BM

Things, that are stupid at the beginning, rarely ends up wisely.

From general-return-361-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 10:31:22 2009
Return-Path: <general-return-361-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16815 invoked from network); 9 Jul 2009 10:31:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 10:31:22 -0000
Received: (qmail 33423 invoked by uid 500); 9 Jul 2009 10:31:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33364 invoked by uid 500); 9 Jul 2009 10:31:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33354 invoked by uid 99); 9 Jul 2009 10:31:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 10:31:32 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of nicc777@gmail.com designates 209.85.218.206 as permitted sender)
Received: from [209.85.218.206] (HELO mail-bw0-f206.google.com) (209.85.218.206)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 10:31:22 +0000
Received: by bwz2 with SMTP id 2so59369bwz.29
        for <general@hadoop.apache.org>; Thu, 09 Jul 2009 03:31:02 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=hGk+w42xhxpXXabFyZU17kJMno33yWAQebcVGB8PhGk=;
        b=PJoELbzZEHNgz2LqRKkozkWySgPUWsEf4ATGAOrV3vQ6fxoFbiYzV6aGcc7raTbCdN
         dYIo+GOsRDUu973sWcAIvdZjj0haaAFtLWXi5T8zj5s5wffHIfS+Cbw7jgS52fyqjuIM
         nOJKun3n1i4DSEAQZx7304xLdiD1i3qEMiK6U=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=sKb/WgVytiu8eiUrRbR1UEJAwLQE/WN3Crxzz2/yIVqftuxr97vVXNyci5NlHEwq+S
         Gb5oEfkZzg2sF7MpS6udLfn55zv8B4rbFnFUBL4vfMdv5Q+1zocdAdw05zGKH1KRuShC
         2/0l1ecCOadIom7Gvi4O9TkJl22sy2g27rum0=
MIME-Version: 1.0
Received: by 10.204.68.10 with SMTP id t10mr557196bki.182.1247135462287; Thu, 
	09 Jul 2009 03:31:02 -0700 (PDT)
Date: Thu, 9 Jul 2009 12:31:02 +0200
Message-ID: <e21a770d0907090331s5c640c92t1046acb8234b04d1@mail.gmail.com>
Subject: Just to say thanks
From: Nico Coetzee <nicc777@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c5bb93376a18046e435a57
X-Virus-Checked: Checked by ClamAV on apache.org

--001636c5bb93376a18046e435a57
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi,

I found the Hadoop project only over the week-end and I have just moved one
of my logfile parsing jobs onto a 3 node cluster to play around and compare
results from our traditional methods.

Wow !

All I can say is thanks for all the people who contributed toward this
project. This must be the coolest tech I found this year so far.

I have only scratched the surface so far but I am sure I will just like it
more and more as time goes by. My next objective is to really get into Hive.

Cheers all and have fun!

Nico

--001636c5bb93376a18046e435a57--

From general-return-362-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 17:25:50 2009
Return-Path: <general-return-362-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94824 invoked from network); 9 Jul 2009 17:25:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 17:25:50 -0000
Received: (qmail 68692 invoked by uid 500); 9 Jul 2009 17:26:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68621 invoked by uid 500); 9 Jul 2009 17:25:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68532 invoked by uid 99); 9 Jul 2009 17:25:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 17:25:59 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of phil123@gmail.com designates 209.85.210.185 as permitted sender)
Received: from [209.85.210.185] (HELO mail-yx0-f185.google.com) (209.85.210.185)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 17:25:50 +0000
Received: by yxe15 with SMTP id 15so462058yxe.5
        for <general@hadoop.apache.org>; Thu, 09 Jul 2009 10:25:29 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=IE65UG/UUHasaXlDhzQlKdFNMGDiA4pojAQ3XzuAO3Q=;
        b=sBnYtdRkED8XAlWqSS6DOn2DL6Ah1tYgRkwXJrh63V29a8929vNtq/RLzP/J9HEU32
         Fn9Op/myTTRjb6ak+1a4ewXD93u2kCXECxoVqH3dY7S9FFsoD6ZVlJ3ERR/1/nx4lDur
         i5e4cA6YSewFvjmQSsLoVzCysHuMfdvsF4yEA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=DgGV3TcGa8YtDQKBG8KmAYV6JFn8q+wq8EP5kCs068TXhAqSmlw3CCOvMPBP6lLRG/
         lvA/v5VBxciWTc9b9lIymSs0U/HuobimDrIECTEN6MfBOzVuMAfRprGyl7fpn5Kz5ITQ
         STSCJgKYVBq34RCswExv+ZaNK5ecOPEcCPUjk=
MIME-Version: 1.0
Received: by 10.100.138.7 with SMTP id l7mr1383021and.141.1247160329369; Thu, 
	09 Jul 2009 10:25:29 -0700 (PDT)
In-Reply-To: <e21a770d0907090331s5c640c92t1046acb8234b04d1@mail.gmail.com>
References: <e21a770d0907090331s5c640c92t1046acb8234b04d1@mail.gmail.com>
Date: Thu, 9 Jul 2009 10:25:29 -0700
Message-ID: <9cafbc680907091025w3b0c653ao14b26dfcdc9b754b@mail.gmail.com>
Subject: Re: Just to say thanks
From: Phil Whelan <phil123@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Nico,

It sounds like you are just ahead of me. We are looking at doing
exactly the same thing right now. Any tips?

Thanks,
Phil

On Thu, Jul 9, 2009 at 3:31 AM, Nico Coetzee<nicc777@gmail.com> wrote:
> Hi,
>
> I found the Hadoop project only over the week-end and I have just moved one
> of my logfile parsing jobs onto a 3 node cluster to play around and compare
> results from our traditional methods.
>
> Wow !
>
> All I can say is thanks for all the people who contributed toward this
> project. This must be the coolest tech I found this year so far.
>
> I have only scratched the surface so far but I am sure I will just like it
> more and more as time goes by. My next objective is to really get into Hive.
>
> Cheers all and have fun!
>
> Nico
>



-- 
Mobile: +1  778-233-4935
Twitter: philwhln
Email : phil123@gmail.com

From general-return-363-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 21:07:25 2009
Return-Path: <general-return-363-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25333 invoked from network); 9 Jul 2009 21:07:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 21:07:25 -0000
Received: (qmail 72553 invoked by uid 500); 9 Jul 2009 21:07:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72486 invoked by uid 500); 9 Jul 2009 21:07:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72476 invoked by uid 99); 9 Jul 2009 21:07:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 21:07:34 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of nicc777@gmail.com designates 209.85.218.206 as permitted sender)
Received: from [209.85.218.206] (HELO mail-bw0-f206.google.com) (209.85.218.206)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 21:07:23 +0000
Received: by bwz2 with SMTP id 2so449321bwz.29
        for <general@hadoop.apache.org>; Thu, 09 Jul 2009 14:07:02 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=z6051KDFxVPECD5wTiTTsOxLbOi8Xps/PN6R6FRdnmo=;
        b=OFW07+YaO+WiiP75LFQuzXrjC8QNWou4dbkG7hC+ydep/Wn4quYLbdK/vgp+mDbIif
         AB8a6tTxv6qWiLYKdMvi9e4O10aSjG3D+VeBbHdemKOI4Sqr/pmekjzzZ4ln1l/LQ2zy
         /FU1VbWmT1VRRH/BpHDDvqnGc2ZU5p/2sns1A=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=sLJ/FTeGYZoM0bdSbcR0NbdaS/9kvqSIivRK7KZBrsRoaj1Td/71cfu7Pn4FPpYt/P
         sz4+LACkmjJPNY7Vk6vh20KWMB+5JssvUX13aR4Se766RmCjQxh8EAkzdVRiq0nBDXQB
         S6Zdnur8F26iMy0QjsvE2iS2U2UnNNTk6XieI=
MIME-Version: 1.0
Received: by 10.204.70.135 with SMTP id d7mr1091868bkj.194.1247173622352; Thu, 
	09 Jul 2009 14:07:02 -0700 (PDT)
In-Reply-To: <9cafbc680907091025w3b0c653ao14b26dfcdc9b754b@mail.gmail.com>
References: <e21a770d0907090331s5c640c92t1046acb8234b04d1@mail.gmail.com>
	 <9cafbc680907091025w3b0c653ao14b26dfcdc9b754b@mail.gmail.com>
Date: Thu, 9 Jul 2009 23:07:02 +0200
Message-ID: <e21a770d0907091407h7f1ea17fqed82ac43bf54d00b@mail.gmail.com>
Subject: Re: Just to say thanks
From: Nico Coetzee <nicc777@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c5a9e8bc19fe046e4c3cc4
X-Virus-Checked: Checked by ClamAV on apache.org

--001636c5a9e8bc19fe046e4c3cc4
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi

I basically followed the samples on:


   -
   http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Single-Node_Cluster)
   -
   http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Multi-Node_Cluster)

The only difference was that I work on CentOS 5.3 64bit - but that did not
change much. I also used JDK 6 update 14 (
http://java.sun.com/javase/downloads/?intcmp=1281).

My scripts, however, is not yet Java based. In stead I used the streaming
solution and do my scripts in Perl. I used
http://www.michael-noll.com/wiki/Writing_An_Hadoop_MapReduce_Program_In_Pythonas
guide (some modification was required, for example
"contrib/streaming/hadoop-0.19.1-streaming.jar" changed to
"contrib/streaming/hadoop-0.20.0-streaming.jar".

The rest of the config help I got from the documentation.

Lessons learned so far:


   - To make life simple, use identical setup on all nodes (data directory
   location, file locations etc.)
   - Scripts need to be present on all nodes (obvious, but I missed that)

Current benchmarks put my 3 node cluster at 3.2 times faster - and this also
takes into account the additional time it took to upload the data in Hadoop
and download the results again.

Hope that helps.

I am hoping to write a more general Apache log parser and share my
experiences. Just busy with the script fine tuning.

Cheers

Nico

PS: a little off topic, but I just need to ask quickly: do we reply top or
bottom on this list?



On Thu, Jul 9, 2009 at 7:25 PM, Phil Whelan <phil123@gmail.com> wrote:

> Hi Nico,
>
> It sounds like you are just ahead of me. We are looking at doing
> exactly the same thing right now. Any tips?
>
> Thanks,
> Phil
>
> On Thu, Jul 9, 2009 at 3:31 AM, Nico Coetzee<nicc777@gmail.com> wrote:
> > Hi,
> >
> > I found the Hadoop project only over the week-end and I have just moved
> one
> > of my logfile parsing jobs onto a 3 node cluster to play around and
> compare
> > results from our traditional methods.
> >
> > Wow !
> >
> > All I can say is thanks for all the people who contributed toward this
> > project. This must be the coolest tech I found this year so far.
> >
> > I have only scratched the surface so far but I am sure I will just like
> it
> > more and more as time goes by. My next objective is to really get into
> Hive.
> >
> > Cheers all and have fun!
> >
> > Nico
> >
>
>
>
> --
> Mobile: +1  778-233-4935
> Twitter: philwhln
> Email : phil123@gmail.com
>

--001636c5a9e8bc19fe046e4c3cc4--

From general-return-364-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 04:10:14 2009
Return-Path: <general-return-364-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17310 invoked from network); 10 Jul 2009 04:10:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 04:10:14 -0000
Received: (qmail 20474 invoked by uid 500); 10 Jul 2009 04:10:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20361 invoked by uid 500); 10 Jul 2009 04:10:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 20351 invoked by uid 99); 10 Jul 2009 04:10:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 04:10:21 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.146.178] (HELO wa-out-1112.google.com) (209.85.146.178)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 04:10:10 +0000
Received: by wa-out-1112.google.com with SMTP id j32so94621waf.29
        for <general@hadoop.apache.org>; Thu, 09 Jul 2009 21:09:48 -0700 (PDT)
Received: by 10.114.125.15 with SMTP id x15mr2470601wac.217.1247198988422;
        Thu, 09 Jul 2009 21:09:48 -0700 (PDT)
Received: from ?192.168.42.100? (75-101-123-136.dsl.static.sonic.net [75.101.123.136])
        by mx.google.com with ESMTPS id n33sm923995wag.21.2009.07.09.21.09.46
        (version=SSLv3 cipher=RC4-MD5);
        Thu, 09 Jul 2009 21:09:47 -0700 (PDT)
Message-ID: <4A56BEB2.2080503@cloudera.com>
Date: Thu, 09 Jul 2009 21:08:18 -0700
From: Amr Awadallah <aaa@cloudera.com>
Organization: Cloudera, Inc.
User-Agent: Thunderbird 2.0.0.22 (Windows/20090605)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Just to say thanks
References: <e21a770d0907090331s5c640c92t1046acb8234b04d1@mail.gmail.com> <9cafbc680907091025w3b0c653ao14b26dfcdc9b754b@mail.gmail.com>
In-Reply-To: <9cafbc680907091025w3b0c653ao14b26dfcdc9b754b@mail.gmail.com>
Content-Type: multipart/alternative;
 boundary="------------060602070605080605090707"
X-Virus-Checked: Checked by ClamAV on apache.org

--------------060602070605080605090707
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Phil,

  If you prefer working with rpm or deb packages then you should check

Debs: http://www.cloudera.com/hadoop-deb
Rpms: http://www.cloudera.com/hadoop-rpm

you can also try the configurator tool at my.cloudera.com which 
automatically generates the hadoop config files for you based on a 
couple of simple questions about your cluster setup.

-- amr

Phil Whelan wrote:
> Hi Nico,
>
> It sounds like you are just ahead of me. We are looking at doing
> exactly the same thing right now. Any tips?
>
> Thanks,
> Phil
>
> On Thu, Jul 9, 2009 at 3:31 AM, Nico Coetzee<nicc777@gmail.com> wrote:
>   
>> Hi,
>>
>> I found the Hadoop project only over the week-end and I have just moved one
>> of my logfile parsing jobs onto a 3 node cluster to play around and compare
>> results from our traditional methods.
>>
>> Wow !
>>
>> All I can say is thanks for all the people who contributed toward this
>> project. This must be the coolest tech I found this year so far.
>>
>> I have only scratched the surface so far but I am sure I will just like it
>> more and more as time goes by. My next objective is to really get into Hive.
>>
>> Cheers all and have fun!
>>
>> Nico
>>
>>     
>
>
>
>   

--------------060602070605080605090707--

From general-return-365-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 04:38:50 2009
Return-Path: <general-return-365-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 20791 invoked from network); 10 Jul 2009 04:38:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 04:38:48 -0000
Received: (qmail 33584 invoked by uid 500); 10 Jul 2009 04:38:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33525 invoked by uid 500); 10 Jul 2009 04:38:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33511 invoked by uid 99); 10 Jul 2009 04:38:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 04:38:57 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saurabhnanda@gmail.com designates 209.85.212.196 as permitted sender)
Received: from [209.85.212.196] (HELO mail-vw0-f196.google.com) (209.85.212.196)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 04:38:44 +0000
Received: by vwj34 with SMTP id 34so553161vwj.5
        for <general@hadoop.apache.org>; Thu, 09 Jul 2009 21:38:23 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=2YKEXB44OXNeYmNFR8R6HzjiI2mW1RbJT6mu9lz16fw=;
        b=wkHemTzjTTgB78ZJvLMnzApgVUZ31Hed3ExsIKQNKZ4rGFGe+wlut3+nBbdcH40QKE
         o0pPAkMMO7UdPBrsqozClinouR1tBJP/gc3kMq3VtlLWfW6uzUrtAt641iN3kwxlFmkp
         dirupfjdImC3RRjGCgwtZYohIUu25699dubms=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=qsLBY6F51EwjyGgrcE3hEt13+TFb+wfLrVx/QEJmpBzZ119nHBLFTAqEvb6zcq5ffm
         xMZhYOA1kYdNcvwmX+t5RUNvNHVrlmQ5cwLqFX+YfxSOFRluIcXBj6tsBRBgZkO93Zvz
         jkQp+GjGarFdXDVBSRKd/gl8FYmFTuPRTRV4E=
MIME-Version: 1.0
Received: by 10.220.81.70 with SMTP id w6mr2267155vck.32.1247200703372; Thu, 
	09 Jul 2009 21:38:23 -0700 (PDT)
In-Reply-To: <fa561940907090245q5cec3592k37dd7857aba4de6e@mail.gmail.com>
References: <794f042d0907080506h38ff3392tabe85b6d5fb9e243@mail.gmail.com>
	 <fa561940907080848o640fe841h48cf2d4f0b69f745@mail.gmail.com>
	 <794f042d0907082115q75692e5an527d4284eab61c83@mail.gmail.com>
	 <fa561940907090245q5cec3592k37dd7857aba4de6e@mail.gmail.com>
Date: Fri, 10 Jul 2009 10:08:23 +0530
Message-ID: <794f042d0907092138s6a19926u2c9e5f2d340290e1@mail.gmail.com>
Subject: Re: Wrong FS error
From: Saurabh Nanda <saurabhnanda@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e647f026e36e9e046e528a7d
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e647f026e36e9e046e528a7d
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Thanks Bogdan. Your solution given below solved my problem. I tried setting
up tinydns on the master, but the package on Ubuntu Intrepid is broken, so I
gave up and went along with /etc/hosts.

Saurabh.


> Well, if you use /etc/hosts, why you bother with IP addresses then?
> Well, they are wrong anyway. For the start, remove "*-hadoop" from
> 127.0.0.1 on both machines and RTFM about /etc/hosts here:
> http://www.geo.arizona.edu/tools/man-cgi?hosts+5 I would suggest you
> to go with something like this:
>
> On 172.16.37.72 should be:
> --------------------------------------
> 127.0.0.1       localhost
> 172.16.37.56 master-hadoop.local master-hadoop
>
>
> On 172.16.37.56 should be:
> --------------------------------------
> 127.0.0.1       localhost
> 172.16.37.72 slave1-hadoop.local slave1-hadoop
>
>
-- 
http://nandz.blogspot.com
http://foodieforlife.blogspot.com

--0016e647f026e36e9e046e528a7d--

From general-return-366-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 04:45:51 2009
Return-Path: <general-return-366-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21692 invoked from network); 10 Jul 2009 04:45:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 04:45:51 -0000
Received: (qmail 37767 invoked by uid 500); 10 Jul 2009 04:46:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37680 invoked by uid 500); 10 Jul 2009 04:46:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37670 invoked by uid 99); 10 Jul 2009 04:46:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 04:46:00 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saurabhnanda@gmail.com designates 209.85.212.196 as permitted sender)
Received: from [209.85.212.196] (HELO mail-vw0-f196.google.com) (209.85.212.196)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 04:45:52 +0000
Received: by vwj34 with SMTP id 34so555138vwj.5
        for <general@hadoop.apache.org>; Thu, 09 Jul 2009 21:45:31 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=eFkMwsz4kdCjHcJ1hIwuzK2JGvHHikjgb0niRQXSAwU=;
        b=uZSnvFy5cZGXul3nGnVy7dpIkJ3gP98l8xeM/m3tRKX6MzNSB797dW5tqbqxdMZQbQ
         3lP5/LtSU7ggBX16mvSlgFGjtrhTjrggJN9hzOhDlYvqUvFXYoRlaEbpxQm1GQgwvUEy
         fSvegT+QEJBL3kLDslt7EC5TVYkaXePVOy8z8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=TUdnSD7uoAYqPnpWnFZ2mYyVwEjQb47csjaZKBY7r2DUo57hPQGJd5FDFcxsQsgZyL
         FYL+wNAXHGG4rSxlYDey+cwaURiVB/PejglopBFe9UxfNL6KXrwq7eTCoJG6l47tTquJ
         VsM9Uppnojoi+m3TVRsOrp6QCNuJ40LeIomdE=
MIME-Version: 1.0
Received: by 10.220.94.133 with SMTP id z5mr2277689vcm.19.1247201131552; Thu, 
	09 Jul 2009 21:45:31 -0700 (PDT)
Date: Fri, 10 Jul 2009 10:15:31 +0530
Message-ID: <794f042d0907092145m1d622777m4bd4c1dd2dd6af7c@mail.gmail.com>
Subject: Weblog analysis -- Cloudbase vs Hive vs Pig?
From: Saurabh Nanda <saurabhnanda@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64618aa68e1eb046e52a456
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e64618aa68e1eb046e52a456
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi,

Does anyone have any pearls of wisdom around this?

I'm spending part of my work time on developing a scalable weblog analysis
system running on a 4 to 6 node cluster (standard desktop class machines). I
don't have much time to try and benchmark all three tools (Cloudbase, Hive,
and Pig) and would really appreciate if someone can give me a heads-up on
what to spend my time on. Some specifics:

a) Which tool can give me the best performance for this problem? (eg. best
use of indexes, data partitioning, etc.)
b) Which tool has the most efficient data storage so that I can store more
days' worth of data into the cluster for ad-hoc analysis.
c) Which tool is more mature and will not crash (for example, the disclaimer
on the Hive Wiki really scared me --
http://wiki.apache.org/hadoop/Hive/GettingStarted)

Thanks,
Saurabh.
-- 
http://nandz.blogspot.com
http://foodieforlife.blogspot.com

--0016e64618aa68e1eb046e52a456--

From general-return-367-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 04:58:01 2009
Return-Path: <general-return-367-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23117 invoked from network); 10 Jul 2009 04:58:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 04:58:01 -0000
Received: (qmail 44188 invoked by uid 500); 10 Jul 2009 04:58:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44114 invoked by uid 500); 10 Jul 2009 04:58:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44104 invoked by uid 99); 10 Jul 2009 04:58:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 04:58:11 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.85.222.196] (HELO mail-pz0-f196.google.com) (209.85.222.196)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 04:58:00 +0000
Received: by pzk34 with SMTP id 34so447678pzk.5
        for <general@hadoop.apache.org>; Thu, 09 Jul 2009 21:57:39 -0700 (PDT)
Received: by 10.114.121.19 with SMTP id t19mr2486549wac.83.1247201858907;
        Thu, 09 Jul 2009 21:57:38 -0700 (PDT)
Received: from ?192.168.42.100? (75-101-123-136.dsl.static.sonic.net [75.101.123.136])
        by mx.google.com with ESMTPS id c26sm1038082waa.15.2009.07.09.21.57.37
        (version=SSLv3 cipher=RC4-MD5);
        Thu, 09 Jul 2009 21:57:37 -0700 (PDT)
Message-ID: <4A56C9E9.8090907@cloudera.com>
Date: Thu, 09 Jul 2009 21:56:09 -0700
From: Amr Awadallah <aaa@cloudera.com>
Organization: Cloudera, Inc.
User-Agent: Thunderbird 2.0.0.22 (Windows/20090605)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Weblog analysis -- Cloudbase vs Hive vs Pig?
References: <794f042d0907092145m1d622777m4bd4c1dd2dd6af7c@mail.gmail.com>
In-Reply-To: <794f042d0907092145m1d622777m4bd4c1dd2dd6af7c@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

see this thread:

http://markmail.org/thread/wzekarj5vpylj3qc

Also, Hive and Pig are both official Apache Hadoop projects with larger 
user/developer communities than Cloudbase (which is GPL2 license as 
opposed to Apache license).

-- amr

Saurabh Nanda wrote:
> Hi,
>
> Does anyone have any pearls of wisdom around this?
>
> I'm spending part of my work time on developing a scalable weblog analysis
> system running on a 4 to 6 node cluster (standard desktop class machines). I
> don't have much time to try and benchmark all three tools (Cloudbase, Hive,
> and Pig) and would really appreciate if someone can give me a heads-up on
> what to spend my time on. Some specifics:
>
> a) Which tool can give me the best performance for this problem? (eg. best
> use of indexes, data partitioning, etc.)
> b) Which tool has the most efficient data storage so that I can store more
> days' worth of data into the cluster for ad-hoc analysis.
> c) Which tool is more mature and will not crash (for example, the disclaimer
> on the Hive Wiki really scared me --
> http://wiki.apache.org/hadoop/Hive/GettingStarted)
>
> Thanks,
> Saurabh.
>   

From general-return-368-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 05:08:11 2009
Return-Path: <general-return-368-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24964 invoked from network); 10 Jul 2009 05:08:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 05:08:09 -0000
Received: (qmail 50004 invoked by uid 500); 10 Jul 2009 05:08:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49918 invoked by uid 500); 10 Jul 2009 05:08:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49908 invoked by uid 99); 10 Jul 2009 05:08:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 05:08:18 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of nicc777@gmail.com designates 209.85.220.206 as permitted sender)
Received: from [209.85.220.206] (HELO mail-fx0-f206.google.com) (209.85.220.206)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 05:08:09 +0000
Received: by fxm2 with SMTP id 2so163629fxm.29
        for <general@hadoop.apache.org>; Thu, 09 Jul 2009 22:07:47 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=/4DMC1LvYF8U5WicGUc59SWpRxYc14fxvBBoAYX/38A=;
        b=bV8tJpyOKw2r8zgMFY8oN/dhlji3+VupfnUCXU9PAVqbYH1ijBsNpVFqyaa9Kva/K8
         UHg0CVa3Ei71jt48x4qV1Zjcg/aXrNWGwo0EiXhfzmaR6j2AH8ajnVKDQ87akexnQXCZ
         KTjMJkI3fxNInYnPMQ1dPm3pYYPLZaGnraaoo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=E6aZinIYtma6JaKSNYSeNIwYIczPJYhxqdzrY2T5obKOPITQnm56/9qi7mHqzJJmfT
         HG/bkoQBqWdKkMG/I/kWSERvqeqc/hc51n2E587Pqw9VELxWC8qT/RbnwrLtS/DNKrMj
         cUv4CebZuRWuN9+0ytIJhG1RltLEJSL5M+xgg=
MIME-Version: 1.0
Received: by 10.204.55.199 with SMTP id v7mr1486862bkg.141.1247202467594; Thu, 
	09 Jul 2009 22:07:47 -0700 (PDT)
In-Reply-To: <4A56BEB2.2080503@cloudera.com>
References: <e21a770d0907090331s5c640c92t1046acb8234b04d1@mail.gmail.com>
	 <9cafbc680907091025w3b0c653ao14b26dfcdc9b754b@mail.gmail.com>
	 <4A56BEB2.2080503@cloudera.com>
Date: Fri, 10 Jul 2009 07:07:47 +0200
Message-ID: <e21a770d0907092207n71f523f5kb8e5f21b081f415e@mail.gmail.com>
Subject: Re: Just to say thanks
From: Nico Coetzee <nicc777@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c5a90c0b4388046e52f4fb
X-Virus-Checked: Checked by ClamAV on apache.org

--001636c5a90c0b4388046e52f4fb
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Very helpful - thanks!

On Fri, Jul 10, 2009 at 6:08 AM, Amr Awadallah <aaa@cloudera.com> wrote:

> Phil,
>
>  If you prefer working with rpm or deb packages then you should check
>
> Debs: http://www.cloudera.com/hadoop-deb
> Rpms: http://www.cloudera.com/hadoop-rpm
>
> you can also try the configurator tool at my.cloudera.com which
> automatically generates the hadoop config files for you based on a couple of
> simple questions about your cluster setup.
>
> -- amr
>
>
> Phil Whelan wrote:
>
>> Hi Nico,
>>
>> It sounds like you are just ahead of me. We are looking at doing
>> exactly the same thing right now. Any tips?
>>
>> Thanks,
>> Phil
>>
>> On Thu, Jul 9, 2009 at 3:31 AM, Nico Coetzee<nicc777@gmail.com> wrote:
>>
>>
>>> Hi,
>>>
>>> I found the Hadoop project only over the week-end and I have just moved
>>> one
>>> of my logfile parsing jobs onto a 3 node cluster to play around and
>>> compare
>>> results from our traditional methods.
>>>
>>> Wow !
>>>
>>> All I can say is thanks for all the people who contributed toward this
>>> project. This must be the coolest tech I found this year so far.
>>>
>>> I have only scratched the surface so far but I am sure I will just like
>>> it
>>> more and more as time goes by. My next objective is to really get into
>>> Hive.
>>>
>>> Cheers all and have fun!
>>>
>>> Nico
>>>
>>>
>>>
>>
>>
>>
>>
>>
>

--001636c5a90c0b4388046e52f4fb--

From general-return-369-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 12:10:07 2009
Return-Path: <general-return-369-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23575 invoked from network); 10 Jul 2009 12:10:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 12:10:07 -0000
Received: (qmail 23436 invoked by uid 500); 10 Jul 2009 12:10:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23380 invoked by uid 500); 10 Jul 2009 12:10:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23369 invoked by uid 99); 10 Jul 2009 12:10:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 12:10:16 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of qiaomuf@gmail.com designates 209.85.221.188 as permitted sender)
Received: from [209.85.221.188] (HELO mail-qy0-f188.google.com) (209.85.221.188)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 12:10:08 +0000
Received: by qyk26 with SMTP id 26so688874qyk.5
        for <general@hadoop.apache.org>; Fri, 10 Jul 2009 05:09:47 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:from:date:message-id
         :subject:to:content-type;
        bh=IhsDuek+S2o7KouHtyHU2AvmWf2zAmCJMSuADpOuQkA=;
        b=V/IMC/hqXM0V7pX4Q1sTUOQyQkwrTfftovlrwmFKtYk5K+vvoTQDrhybHltvJ7hgDv
         OvTZcz0yb5eK/qt6MDPRE7hIJy0XLqi34YMAaDewQtrKs/Fm3vKnsnVau2d8bPIMpX6Z
         8kZ8vMwQmxsInRuH+PpIeuewQ4w35Dcd2BjDU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type;
        b=Dnqu3o/B8I/dpVgxr8veWN4hT+YDQdRFLH7D1DKmA2Nc6tzMdBU3KUi6KgDzHqhrs0
         /aiSmvTZY87NlPFAdPo04Enl3g1z0HS92bpoZhhVofW2LrqynjhrZSTe3bkefDLEziG9
         5xVWonTZs08IM/JmNplUaQ3cTsGQf9j4lOMiA=
MIME-Version: 1.0
Received: by 10.229.96.16 with SMTP id f16mr328041qcn.85.1247227787073; Fri, 
	10 Jul 2009 05:09:47 -0700 (PDT)
From: Mu Qiao <qiaomuf@gmail.com>
Date: Fri, 10 Jul 2009 20:09:27 +0800
Message-ID: <396ceda90907100509s16cd9d21jd005ee60f4acc18c@mail.gmail.com>
Subject: Why is Spilled Records always equal to Map output records
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364ee31633dea6046e58d9b1
X-Virus-Checked: Checked by ClamAV on apache.org

--0016364ee31633dea6046e58d9b1
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

I notice it from the web console after I've tried to run serveral jobs.
Every one of the jobs has the number of Spilled Records equal to Map output
records.

I know it is better that there are as few Spilled Records as possible. But =
I
don't know why this happened. How could it be?

--=20
Best wishes,
Qiao Mu
MOE KLINNS Lab and SKLMS Lab, Xi'an Jiaotong University
Department of Computer Science and Technology, Xi=92an Jiaotong University
TEL: 15991676983
E-mail: qiaomuf@gmail.com

--0016364ee31633dea6046e58d9b1--

From general-return-370-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 19:06:54 2009
Return-Path: <general-return-370-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 36562 invoked from network); 10 Jul 2009 19:06:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 19:06:54 -0000
Received: (qmail 39872 invoked by uid 500); 10 Jul 2009 19:07:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39793 invoked by uid 500); 10 Jul 2009 19:07:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39783 invoked by uid 99); 10 Jul 2009 19:07:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 19:07:02 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.221.188] (HELO mail-qy0-f188.google.com) (209.85.221.188)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 19:06:51 +0000
Received: by qyk26 with SMTP id 26so947912qyk.5
        for <general@hadoop.apache.org>; Fri, 10 Jul 2009 12:06:30 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.70.141 with SMTP id d13mr447302qcj.70.1247252790434; Fri, 
	10 Jul 2009 12:06:30 -0700 (PDT)
In-Reply-To: <4A56C9E9.8090907@cloudera.com>
References: <794f042d0907092145m1d622777m4bd4c1dd2dd6af7c@mail.gmail.com>
	 <4A56C9E9.8090907@cloudera.com>
Date: Fri, 10 Jul 2009 12:06:30 -0700
Message-ID: <c9118d4c0907101206g37f1d11dw94b940c7ad2ef6af@mail.gmail.com>
Subject: Re: Weblog analysis -- Cloudbase vs Hive vs Pig?
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016360e3b3c84e75c046e5eabe5
X-Virus-Checked: Checked by ClamAV on apache.org

--0016360e3b3c84e75c046e5eabe5
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Also see some benchmarks run by the Hive team at
https://issues.apache.org/jira/browse/HIVE-396.

On Thu, Jul 9, 2009 at 9:56 PM, Amr Awadallah <aaa@cloudera.com> wrote:

> see this thread:
>
> http://markmail.org/thread/wzekarj5vpylj3qc
>
> Also, Hive and Pig are both official Apache Hadoop projects with larger
> user/developer communities than Cloudbase (which is GPL2 license as opposed
> to Apache license).
>
> -- amr
>
>
> Saurabh Nanda wrote:
>
>> Hi,
>>
>> Does anyone have any pearls of wisdom around this?
>>
>> I'm spending part of my work time on developing a scalable weblog analysis
>> system running on a 4 to 6 node cluster (standard desktop class machines).
>> I
>> don't have much time to try and benchmark all three tools (Cloudbase,
>> Hive,
>> and Pig) and would really appreciate if someone can give me a heads-up on
>> what to spend my time on. Some specifics:
>>
>> a) Which tool can give me the best performance for this problem? (eg. best
>> use of indexes, data partitioning, etc.)
>> b) Which tool has the most efficient data storage so that I can store more
>> days' worth of data into the cluster for ad-hoc analysis.
>> c) Which tool is more mature and will not crash (for example, the
>> disclaimer
>> on the Hive Wiki really scared me --
>> http://wiki.apache.org/hadoop/Hive/GettingStarted)
>>
>> Thanks,
>> Saurabh.
>>
>>
>

--0016360e3b3c84e75c046e5eabe5--

From general-return-371-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 11 22:49:27 2009
Return-Path: <general-return-371-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94075 invoked from network); 11 Jul 2009 22:49:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jul 2009 22:49:27 -0000
Received: (qmail 83561 invoked by uid 500); 11 Jul 2009 22:49:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83474 invoked by uid 500); 11 Jul 2009 22:49:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83463 invoked by uid 99); 11 Jul 2009 22:49:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 22:49:35 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of phil123@gmail.com designates 74.125.92.26 as permitted sender)
Received: from [74.125.92.26] (HELO qw-out-2122.google.com) (74.125.92.26)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 22:49:24 +0000
Received: by qw-out-2122.google.com with SMTP id 8so560541qwh.35
        for <general@hadoop.apache.org>; Sat, 11 Jul 2009 15:49:03 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:references:message-id:from:to
         :in-reply-to:content-type:content-transfer-encoding:x-mailer
         :mime-version:subject:date:cc;
        bh=XB51Lpdimqnooav4L+SvTdGGSaDaT+P7RLB8FjeX9IM=;
        b=LfHHf4Y56h59rEmMLFVvq2OQraW93DRKHThWTp2l0H2/fZOAvB6cKAn868dzHwR4ST
         H7n+svMbQiz0en/cUfS3gEtVv0LT1jYogB8FzrI/KcUGGpJJqFAOGSi/b4KmSuBlciBF
         Z6Js5DFNE+r8o7AdUilmt/XZYfs0q5OZD6rEE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=references:message-id:from:to:in-reply-to:content-type
         :content-transfer-encoding:x-mailer:mime-version:subject:date:cc;
        b=eZBRg1vH5FmSB2prlnDhfWv7JRy15n6r4uIPlMy1oquUrQ+qT8SAfjaEEaf4eVcUno
         CiGFyeZF9WzSl/bWFAn3l1I2nUltstu9n5QRbjuzzsU+CtSwhOByJ25uuq4mi2Wg6dbx
         fWE3AKpXezgmP5JgO7nr1pr1TV+TpExn0DxfM=
Received: by 10.224.73.129 with SMTP id q1mr2096493qaj.183.1247352543298;
        Sat, 11 Jul 2009 15:49:03 -0700 (PDT)
Received: from ?10.198.139.18? ([24.114.236.32])
        by mx.google.com with ESMTPS id 26sm3922765qwa.57.2009.07.11.15.49.00
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Sat, 11 Jul 2009 15:49:02 -0700 (PDT)
References: <e21a770d0907090331s5c640c92t1046acb8234b04d1@mail.gmail.com> <9cafbc680907091025w3b0c653ao14b26dfcdc9b754b@mail.gmail.com> <4A56BEB2.2080503@cloudera.com>
Message-Id: <62F58204-08A2-4C11-9BCE-61D76E4A99D1@gmail.com>
From: Phil Whelan <phil123@gmail.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <4A56BEB2.2080503@cloudera.com>
Content-Type: text/plain;
	charset=us-ascii;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit
X-Mailer: iPhone Mail (7C97d)
Mime-Version: 1.0 (iPhone Mail 7C97d)
Subject: Re: Just to say thanks
Date: Sat, 11 Jul 2009 15:48:53 -0700
Cc: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org


Thanks for the advice Nico and Amr!

I'll be sure a send an update on how I get on.

Phil

Sent from my iPhone

On 2009-07-09, at 9:08 PM, Amr Awadallah <aaa@cloudera.com> wrote:

> Phil,
>
> If you prefer working with rpm or deb packages then you should check
>
> Debs: http://www.cloudera.com/hadoop-deb
> Rpms: http://www.cloudera.com/hadoop-rpm
>
> you can also try the configurator tool at my.cloudera.com which  
> automatically generates the hadoop config files for you based on a  
> couple of simple questions about your cluster setup.
>
> -- amr
>
> Phil Whelan wrote:
>> Hi Nico,
>>
>> It sounds like you are just ahead of me. We are looking at doing
>> exactly the same thing right now. Any tips?
>>
>> Thanks,
>> Phil
>>
>> On Thu, Jul 9, 2009 at 3:31 AM, Nico Coetzee<nicc777@gmail.com>  
>> wrote:
>>
>>> Hi,
>>>
>>> I found the Hadoop project only over the week-end and I have just  
>>> moved one
>>> of my logfile parsing jobs onto a 3 node cluster to play around  
>>> and compare
>>> results from our traditional methods.
>>>
>>> Wow !
>>>
>>> All I can say is thanks for all the people who contributed toward  
>>> this
>>> project. This must be the coolest tech I found this year so far.
>>>
>>> I have only scratched the surface so far but I am sure I will just  
>>> like it
>>> more and more as time goes by. My next objective is to really get  
>>> into Hive.
>>>
>>> Cheers all and have fun!
>>>
>>> Nico
>>>
>>>
>>
>>
>>
>>

From general-return-372-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 13 07:15:50 2009
Return-Path: <general-return-372-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43457 invoked from network); 13 Jul 2009 07:15:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jul 2009 07:15:50 -0000
Received: (qmail 38568 invoked by uid 500); 13 Jul 2009 07:15:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38516 invoked by uid 500); 13 Jul 2009 07:15:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 38506 invoked by uid 99); 13 Jul 2009 07:15:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 07:15:58 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of the3rdhr@gmail.com designates 209.85.212.188 as permitted sender)
Received: from [209.85.212.188] (HELO mail-vw0-f188.google.com) (209.85.212.188)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 07:15:50 +0000
Received: by vwj26 with SMTP id 26so722007vwj.5
        for <general@hadoop.apache.org>; Mon, 13 Jul 2009 00:15:29 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=rdW91PZFxbftg5KJLppVyUHRtv6LjLqHnelKv4z1lnM=;
        b=jXhRdiBoHXBEUOl2nvvSsxbYzcCUbAP0p/7wKwCSOGjH3aFHCHZhwKfaZxccyd0mIr
         5rpyc5Fk+5wjfaaGvZRvdi+Bh2Lkx3vHuTinAQbeX7w7KGvQLLsUD2TtP8UU8/LyaPuZ
         YFgWI9kAL3DRZ53uo30TLYG/9Z7K9sLciv/SU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=XiCVBQ1+DR7o6JcNQ/ih0LGQFBTABuNXPT+9+Y75uSC7z5RQTQc+0GOmabC9AJA1Mo
         LcrzLeASzlOGK83UIiyqQSDC/MEd/vWRvhxoVBG3OPoCDaW3hS7+CrVrTdvmq5Wc/YXf
         Y20TXb61Sq2McK00VrkB8f3MrvSILsGwNI6Kw=
MIME-Version: 1.0
Received: by 10.220.83.201 with SMTP id g9mr6683396vcl.42.1247469329505; Mon, 
	13 Jul 2009 00:15:29 -0700 (PDT)
Date: Mon, 13 Jul 2009 16:15:29 +0900
Message-ID: <a38b240f0907130015l6ce34f9bt6554ce3d6d8ba6aa@mail.gmail.com>
Subject: I wonder why split the project
From: the3rdhr <the3rdhr@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364eca22411fce046e911603
X-Virus-Checked: Checked by ClamAV on apache.org

--0016364eca22411fce046e911603
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hello.

I saw the svn last week,
the project is splited to common, hdfs, mapreduce.

I'd like to know the reason why split the project.
I can find the reason in the web page of hadoop.

please someone tell me :)

--0016364eca22411fce046e911603--

From general-return-373-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 13 08:03:22 2009
Return-Path: <general-return-373-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56956 invoked from network); 13 Jul 2009 08:03:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jul 2009 08:03:22 -0000
Received: (qmail 88818 invoked by uid 500); 13 Jul 2009 08:03:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88738 invoked by uid 500); 13 Jul 2009 08:03:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88728 invoked by uid 99); 13 Jul 2009 08:03:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 08:03:29 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [209.85.220.205] (HELO mail-fx0-f205.google.com) (209.85.220.205)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 08:03:20 +0000
Received: by fxm1 with SMTP id 1so373926fxm.29
        for <general@hadoop.apache.org>; Mon, 13 Jul 2009 01:02:59 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.55.140 with SMTP id u12mr4947217bkg.127.1247472179146; 
	Mon, 13 Jul 2009 01:02:59 -0700 (PDT)
In-Reply-To: <a38b240f0907130015l6ce34f9bt6554ce3d6d8ba6aa@mail.gmail.com>
References: <a38b240f0907130015l6ce34f9bt6554ce3d6d8ba6aa@mail.gmail.com>
From: Amr Awadallah <aaa@cloudera.com>
Date: Mon, 13 Jul 2009 01:02:44 -0700
Message-ID: <6e412fb10907130102r67c6da61rbfa25ef9d59b2406@mail.gmail.com>
Subject: Re: I wonder why split the project
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c924d91af0b8046e91c067
X-Virus-Checked: Checked by ClamAV on apache.org

--001636c924d91af0b8046e91c067
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

> please someone tell me :)

to make your life harder :)

Seriously though, the community realized that coupling HDFS and MapReduce
made it hard to evolve them, and even harder to test them. The current split
will create some headaches in the short term, but it sets a very solid
foundation for the future.

-- amr

On Mon, Jul 13, 2009 at 12:15 AM, the3rdhr <the3rdhr@gmail.com> wrote:

> Hello.
>
> I saw the svn last week,
> the project is splited to common, hdfs, mapreduce.
>
> I'd like to know the reason why split the project.
> I can find the reason in the web page of hadoop.
>
> please someone tell me :)
>

--001636c924d91af0b8046e91c067--

From general-return-374-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 13 08:25:01 2009
Return-Path: <general-return-374-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63097 invoked from network); 13 Jul 2009 08:24:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jul 2009 08:24:59 -0000
Received: (qmail 18270 invoked by uid 500); 13 Jul 2009 08:25:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18197 invoked by uid 500); 13 Jul 2009 08:25:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18187 invoked by uid 99); 13 Jul 2009 08:25:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 08:25:08 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of the3rdhr@gmail.com designates 209.85.212.188 as permitted sender)
Received: from [209.85.212.188] (HELO mail-vw0-f188.google.com) (209.85.212.188)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 08:24:58 +0000
Received: by vwj26 with SMTP id 26so740598vwj.5
        for <general@hadoop.apache.org>; Mon, 13 Jul 2009 01:24:37 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=c1IlBWoC6dYH6ercnKBTz+CvkW7N/OoNG0uymNetblw=;
        b=uyDtZ8fpwdKgewum1bIhQV2Tx2dExAlyoC2YIf0GqYuZn5Bv+Evk34dE0xtyFU84k7
         dH83VupgUI1821Hy4t7QJVk6zeUGY8xCrag59iBDRvunCVev6WPmvaUJO7pHxFdQ3GCm
         rJsy9v2E82p3Z5N7ZdNRam/R7i5oGEAVMfGL8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=n9hSP3UjYgCmWUa5LDZ1tn18OwhWl96KAsbZWxJJtr6DLQHAOkaj1tpaQmHS6c49uR
         60P4bTYqLaaoh39V1gp6joimByw7vUQPSJfBvc1CcrW235chJJOXRh5NqmF4i2S+OZRi
         SAdW9PN+MLBDzDnAFiQrGdVEPAlaP0KUGaQlE=
MIME-Version: 1.0
Received: by 10.220.85.198 with SMTP id p6mr6810632vcl.48.1247473477231; Mon, 
	13 Jul 2009 01:24:37 -0700 (PDT)
In-Reply-To: <6e412fb10907130102r67c6da61rbfa25ef9d59b2406@mail.gmail.com>
References: <a38b240f0907130015l6ce34f9bt6554ce3d6d8ba6aa@mail.gmail.com>
	 <6e412fb10907130102r67c6da61rbfa25ef9d59b2406@mail.gmail.com>
Date: Mon, 13 Jul 2009 17:24:37 +0900
Message-ID: <a38b240f0907130124q2013b099ub79cd16e17a60ca1@mail.gmail.com>
Subject: Re: I wonder why split the project
From: the3rdhr <the3rdhr@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6464f6c7a25ab046e920dbb
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6464f6c7a25ab046e920dbb
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Thank you for your reply.

 I'd like to know more information about split project.
 I cannot find the information of that, in hadoop site, mailling list ;-(
Can I see the detail information about that by web page, mailling list or
some documents?

and I don't want to be my life harder :-)

2009/7/13 Amr Awadallah <aaa@cloudera.com>

> > please someone tell me :)
>
> to make your life harder :)
>
> Seriously though, the community realized that coupling HDFS and MapReduce
> made it hard to evolve them, and even harder to test them. The current
> split
> will create some headaches in the short term, but it sets a very solid
> foundation for the future.
>
> -- amr
>
> On Mon, Jul 13, 2009 at 12:15 AM, the3rdhr <the3rdhr@gmail.com> wrote:
>
> > Hello.
> >
> > I saw the svn last week,
> > the project is splited to common, hdfs, mapreduce.
> >
> > I'd like to know the reason why split the project.
> > I can find the reason in the web page of hadoop.
> >
> > please someone tell me :)
> >
>

--0016e6464f6c7a25ab046e920dbb--

From general-return-375-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 13 09:05:01 2009
Return-Path: <general-return-375-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 71954 invoked from network); 13 Jul 2009 09:05:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jul 2009 09:05:00 -0000
Received: (qmail 63506 invoked by uid 500); 13 Jul 2009 09:05:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63411 invoked by uid 500); 13 Jul 2009 09:05:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63401 invoked by uid 99); 13 Jul 2009 09:05:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 09:05:09 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [66.51.199.98] (HELO mail10.dslextreme.com) (66.51.199.98)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 13 Jul 2009 09:05:00 +0000
Received: (qmail 14548 invoked from network); 13 Jul 2009 09:04:38 -0000
Received: from unknown (HELO [192.168.0.100]) (68.183.197.7)
	by mail10.dslextreme.com with (DHE-RSA-AES256-SHA encrypted) SMTP; Mon, 13 Jul 2009 02:04:38 -0700
Message-ID: <4A5AF89E.8060509@cloudera.com>
Date: Mon, 13 Jul 2009 02:04:30 -0700
From: Amr Awadallah <aaa@cloudera.com>
User-Agent: Thunderbird 2.0.0.22 (Windows/20090605)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: I wonder why split the project
References: <a38b240f0907130015l6ce34f9bt6554ce3d6d8ba6aa@mail.gmail.com>	 <6e412fb10907130102r67c6da61rbfa25ef9d59b2406@mail.gmail.com> <a38b240f0907130124q2013b099ub79cd16e17a60ca1@mail.gmail.com>
In-Reply-To: <a38b240f0907130124q2013b099ub79cd16e17a60ca1@mail.gmail.com>
Content-Type: multipart/alternative;
 boundary="------------090901000209070307030602"
X-MagicMail-UUID: 31a08eb4-6f8c-11de-8861-00188bf6df8c
X-Virus-Checked: Checked by ClamAV on apache.org

--------------090901000209070307030602
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

The Apache Hadoop website will be updated shortly to reflect the split.

You can look at these JIRAs to follow progress on that front:

https://issues.apache.org/jira/browse/HADOOP-6136 
<https://issues.apache.org/jira/browse/HADOOP-6136>
https://issues.apache.org/jira/browse/MAPREDUCE-741 
<https://issues.apache.org/jira/browse/MAPREDUCE-741>
https://issues.apache.org/jira/browse/HDFS-477 
<https://issues.apache.org/jira/browse/HDFS-477>

-- amr

> Thank you for your reply.
>
>  I'd like to know more information about split project.
>  I cannot find the information of that, in hadoop site, mailling list ;-(
> Can I see the detail information about that by web page, mailling list or
> some documents?
>
> and I don't want to be my life harder :-)
>
> 2009/7/13 Amr Awadallah <aaa@cloudera.com>
>
>   
>>> please someone tell me :)
>>>       
>> to make your life harder :)
>>
>> Seriously though, the community realized that coupling HDFS and MapReduce
>> made it hard to evolve them, and even harder to test them. The current
>> split
>> will create some headaches in the short term, but it sets a very solid
>> foundation for the future.
>>
>> -- amr
>>
>> On Mon, Jul 13, 2009 at 12:15 AM, the3rdhr <the3rdhr@gmail.com> wrote:
>>
>>     
>>> Hello.
>>>
>>> I saw the svn last week,
>>> the project is splited to common, hdfs, mapreduce.
>>>
>>> I'd like to know the reason why split the project.
>>> I can find the reason in the web page of hadoop.
>>>
>>> please someone tell me :)
>>>
>>>       
>
>   


--------------090901000209070307030602--

From general-return-376-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 13 15:35:40 2009
Return-Path: <general-return-376-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43512 invoked from network); 13 Jul 2009 15:35:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jul 2009 15:35:40 -0000
Received: (qmail 27221 invoked by uid 500); 13 Jul 2009 15:35:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27168 invoked by uid 500); 13 Jul 2009 15:35:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27158 invoked by uid 99); 13 Jul 2009 15:35:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 15:35:48 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of shruti.jain1988@gmail.com designates 209.85.200.173 as permitted sender)
Received: from [209.85.200.173] (HELO wf-out-1314.google.com) (209.85.200.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 15:35:40 +0000
Received: by wf-out-1314.google.com with SMTP id 23so744696wfg.2
        for <general@hadoop.apache.org>; Mon, 13 Jul 2009 08:35:20 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=XzreUijK0HR6kZlL8k8U4NvKY1Wl+/s4mVY0vDY72ag=;
        b=xfu8GAQDBoWgOG0w1ph7LGnSI0qKUzflY2/kfY97LprDTgNtN4di/lhhBmKXyDzr1d
         B5JjDIAH2oMAUUSEi8R9pBm5GZPgbNpWCw5UPiqT6qBgKFz9vNWhKE/7+6J0xDjqvH/d
         cr09liIlffBDM/TY/31h6nE+47lnYMTfc4a/k=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=VPT8Qmf6kCnl8nvwl4HYvwUpt9ZBeEnbJPoHtZPjrkD/3/SZRM1K7uJ0PGmKQ0onY5
         CjoCCMtOOS8OwiEprSkmtjmRYdVb/oFhTLjO+1Bkul14xvbNMuqzhjGXJ4nSksS5jPWr
         WS0AVBGM44bHI1YG+shnh1B8tTiM2uDSmaPfM=
MIME-Version: 1.0
Received: by 10.142.226.1 with SMTP id y1mr1279340wfg.294.1247499320704; Mon, 
	13 Jul 2009 08:35:20 -0700 (PDT)
In-Reply-To: <59714e760907130318l3d13b93ape22750b24b231a7e@mail.gmail.com>
References: <59714e760907130318l3d13b93ape22750b24b231a7e@mail.gmail.com>
Date: Mon, 13 Jul 2009 21:05:20 +0530
Message-ID: <59714e760907130835l4a1e05abld5477e2249ce6080@mail.gmail.com>
Subject: Speculative Processing in Hadoop
From: shruti jain <shruti.jain1988@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hello Everyone,

I am a newbie and need some help. I saw on Hadoop wiki that there can
be projects to improve Hadoop and map-reduce performance on available
benchmarks(sort etc)..

In a distributed file system environment, caching can be followed. In
such systems, whenever a file access is required, the client has to
check the content in the local cache with reference to the server file
system. By the time server responds to this query of the client, the
client can execute the requested operations on the data available in
the cache. If the server responds that the client has the most
recently modified file then the client can proceed with the processing
otherwise it can rollback to a previous state and start with newer
version of the file. This will save processing power, CPU cycles time.

This can be applied to Hadoop as well. Say we are sorting a file. With
map-reduce sorting can be done this way. A client requests the server
about the modification time of the file and starts execution on the
file it has in the cache. When server responds it can check the cached
copy and proceed accordingly.

Could any one please discuss whether this can be done in Hadoop or
not. Is it already implemented or is anyone else working on the same.
If this is not the right place to discuss then can you direct me to
some other source of information.

Thank You.

Shruti

From general-return-377-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 13 18:50:05 2009
Return-Path: <general-return-377-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 51645 invoked from network); 13 Jul 2009 18:50:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jul 2009 18:50:05 -0000
Received: (qmail 59350 invoked by uid 500); 13 Jul 2009 18:50:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59261 invoked by uid 500); 13 Jul 2009 18:50:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59251 invoked by uid 99); 13 Jul 2009 18:50:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 18:50:14 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of andrey.v.kuzmin@gmail.com designates 209.85.212.188 as permitted sender)
Received: from [209.85.212.188] (HELO mail-vw0-f188.google.com) (209.85.212.188)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 18:50:06 +0000
Received: by vwj26 with SMTP id 26so1066745vwj.5
        for <general@hadoop.apache.org>; Mon, 13 Jul 2009 11:49:45 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=PtMpUxQtY5lBl/T8BGFVXR/pC3dfLsiVHAn7eyRPxLY=;
        b=w2mAGzhZHnELR8zyEcEsQ/teIm4lBErS7Nw4h6DtlOpfMBzwchm01Izx9RdquIxmOQ
         Tyl1vripmUaZ12zu9ykcsrXkjKbOq+uXztkvbO3atqbTl5z3S/F6/F5ld+89XitKePj3
         9n8lGXH+pU61C1gYDpLrXRgFdfvlZJ+r5/ctA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=GA346fhoOYf2nh1Ijgy8IvWklzbVu+Y9IHaw8qjO1sR3D1kDUYIIq3VerLXEgV3VkJ
         berJuRiXqG8I1bzKPqEZhoNf0V0OuxDm1tVqcGcozEbCtOdOvvvmdjrCBXViHL3fnrAn
         eIKJ4tAb4lcGyFvSLrWsRZUOHfnRK/pi96q94=
MIME-Version: 1.0
Received: by 10.220.96.15 with SMTP id f15mr7572053vcn.116.1247510985170; Mon, 
	13 Jul 2009 11:49:45 -0700 (PDT)
In-Reply-To: <59714e760907130835l4a1e05abld5477e2249ce6080@mail.gmail.com>
References: <59714e760907130318l3d13b93ape22750b24b231a7e@mail.gmail.com> 
	<59714e760907130835l4a1e05abld5477e2249ce6080@mail.gmail.com>
From: Andrey Kuzmin <andrey.v.kuzmin@gmail.com>
Date: Mon, 13 Jul 2009 22:49:25 +0400
Message-ID: <2a31deca0907131149m52c70e05w85dd9586e91e7ff6@mail.gmail.com>
Subject: Re: Speculative Processing in Hadoop
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

As far as I understand, Hadoop solves the problem you're trying to
tackle differently  - it moves computation close to the data (by
assigning task to a node that possesses local copy), not vice versa
where speculative execution might turn helpful as, e.g., in network
file-systems.

Regards,
Andrey



On Mon, Jul 13, 2009 at 7:35 PM, shruti jain<shruti.jain1988@gmail.com> wrote:
> Hello Everyone,
>
> I am a newbie and need some help. I saw on Hadoop wiki that there can
> be projects to improve Hadoop and map-reduce performance on available
> benchmarks(sort etc)..
>
> In a distributed file system environment, caching can be followed. In
> such systems, whenever a file access is required, the client has to
> check the content in the local cache with reference to the server file
> system. By the time server responds to this query of the client, the
> client can execute the requested operations on the data available in
> the cache. If the server responds that the client has the most
> recently modified file then the client can proceed with the processing
> otherwise it can rollback to a previous state and start with newer
> version of the file. This will save processing power, CPU cycles time.
>
> This can be applied to Hadoop as well. Say we are sorting a file. With
> map-reduce sorting can be done this way. A client requests the server
> about the modification time of the file and starts execution on the
> file it has in the cache. When server responds it can check the cached
> copy and proceed accordingly.
>
> Could any one please discuss whether this can be done in Hadoop or
> not. Is it already implemented or is anyone else working on the same.
> If this is not the right place to discuss then can you direct me to
> some other source of information.
>
> Thank You.
>
> Shruti
>

From general-return-378-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 14 22:11:45 2009
Return-Path: <general-return-378-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60822 invoked from network); 14 Jul 2009 22:11:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jul 2009 22:11:45 -0000
Received: (qmail 97229 invoked by uid 500); 14 Jul 2009 22:11:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97152 invoked by uid 500); 14 Jul 2009 22:11:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97108 invoked by uid 99); 14 Jul 2009 22:11:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 22:11:51 +0000
X-ASF-Spam-Status: No, hits=4.2 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 22:11:38 +0000
Received: from SP1-EX07CAS02.ds.corp.yahoo.com (sp1-ex07cas02.ds.corp.yahoo.com [216.252.116.138])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n6EMAsYa074886
	for <general@hadoop.apache.org>; Tue, 14 Jul 2009 15:10:54 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:accept-language:content-language:x-ms-has-attach:
	x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=MHj9PtkMXTFff5ntSiRL5Bbo+d8e1abhvDquF/+5EC0AFKEAcwF6INmdOgqOWv8g
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 SP1-EX07CAS02.ds.corp.yahoo.com ([216.252.116.167]) with mapi; Tue, 14 Jul
 2009 15:10:54 -0700
From: Dekel Tankel <dekel@yahoo-inc.com>
To: "'general@hadoop.apache.org'" <'general@hadoop.apache.org'>
Date: Tue, 14 Jul 2009 15:10:53 -0700
Subject: Hadoop User Group (Bay Area) - Aug 19th at Yahoo!
Thread-Topic: Hadoop User Group (Bay Area) - Aug 19th at Yahoo!
Thread-Index: AcoEz/Qe1PyDbbbWRXiTzKa8a+fTUg==
Message-ID: <46E2672895E8744A97D7577A6110858B4E143AEC17@SP1-EX07VS01.ds.corp.yahoo.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_46E2672895E8744A97D7577A6110858B4E143AEC17SP1EX07VS01ds_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_46E2672895E8744A97D7577A6110858B4E143AEC17SP1EX07VS01ds_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi all,

I'd like to remind everyone that Yahoo! is excited to continue hosting the =
monthly Bay Area Hadoop user groups.

The next meeting is scheduled for Aug 19th (recurring the 3rd Wednesday of =
every month) at Yahoo!'s Sunnyvale campus.

Agenda and event registration details are coming up shortly.

Looking forward to seeing you there!

Dekel


--_000_46E2672895E8744A97D7577A6110858B4E143AEC17SP1EX07VS01ds_--

From general-return-379-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 11:43:50 2009
Return-Path: <general-return-379-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 90710 invoked from network); 17 Jul 2009 11:43:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 11:43:50 -0000
Received: (qmail 65637 invoked by uid 500); 17 Jul 2009 11:44:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65555 invoked by uid 500); 17 Jul 2009 11:44:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65545 invoked by uid 99); 17 Jul 2009 11:44:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 11:44:54 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of timrobertson100@gmail.com designates 209.85.219.219 as permitted sender)
Received: from [209.85.219.219] (HELO mail-ew0-f219.google.com) (209.85.219.219)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 11:44:45 +0000
Received: by ewy19 with SMTP id 19so755908ewy.29
        for <general@hadoop.apache.org>; Fri, 17 Jul 2009 04:44:25 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type:content-transfer-encoding;
        bh=KwvH8XZZAoRetgmhxPWiThbfOIDx/ajKsH3lNHtY0JM=;
        b=TyjmPAlgkk/dknoAZVInVL8wCtdZJwDHETqVW4voGV130+YRRk2s9PS1/NIZ6mEe+B
         2lFZwhdj/Hkj/3OAPhQOv3MQ11MM2Sf3h607uBspWOaludkIe07CK3e2LcitCQSj5mW/
         s0Eca+Lm48SdXdo1lBPZhMQbhlAzjEb7qdfVU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=RJO6zUoquXBB/BBXpEvKQ2E6rpKI843pYRhQ/k0V9yzP9fan//p8BGR6LjyjNKpVBT
         Ttij/ZjuWLQ+p5N6tL6sIE0DiUI0cVQxThEjVMY52tQYdnakWfi9W74e49hxUTkzRIg+
         XbF0B9nwvQM2KFyJgYP0BDcUTr+WTfZeOe30w=
MIME-Version: 1.0
Received: by 10.216.17.212 with SMTP id j62mr276586wej.132.1247831065402; Fri, 
	17 Jul 2009 04:44:25 -0700 (PDT)
Date: Fri, 17 Jul 2009 13:44:25 +0200
Message-ID: <32120a6a0907170444x61dac5a7s8988012351520459@mail.gmail.com>
Subject: Downloads linking incorrectly or mirrors are not updated?
From: tim robertson <timrobertson100@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hi guys,

I know you are reorganising the sub projects, but just for your info...
The downloads linked from the Hadoop site
(http://www.apache.org/dyn/closer.cgi/hadoop/mapreduce/) are all
incorrect, or the mirrors are not updated (for what I see from Denmark
anyway).
All the links end in .../mapreduce/ but the mirrors have the downloads
available on .../core/

Cheers,

Tim

From general-return-380-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 11:47:12 2009
Return-Path: <general-return-380-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91363 invoked from network); 17 Jul 2009 11:47:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 11:47:12 -0000
Received: (qmail 67850 invoked by uid 500); 17 Jul 2009 11:48:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67768 invoked by uid 500); 17 Jul 2009 11:48:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67755 invoked by uid 99); 17 Jul 2009 11:48:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 11:48:17 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [85.183.249.146] (HELO mail.etracker.de) (85.183.249.146)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 11:48:06 +0000
Received: by mail.etracker.de (Postfix, from userid 1003)
	id 116E8631172; Fri, 17 Jul 2009 13:47:45 +0200 (CEST)
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail.etracker.de
X-Spam-Level: 
Received: from mail.etracker.local (unknown [85.183.249.129])
	by mail.etracker.de (Postfix) with ESMTP id 7DB5663116E
	for <general@hadoop.apache.org>; Fri, 17 Jul 2009 13:47:45 +0200 (CEST)
Subject: Hadoop/HDFS Thrift/PHP connection doesnt work
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Date: Fri, 17 Jul 2009 13:47:43 +0200
Content-class: urn:content-classes:message
X-MimeOLE: Produced By Microsoft Exchange V6.5
Message-ID: <E74E142F203F1A42A9288C280C60F8A5020AE0B6@ebase.etracker.local>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Hadoop/HDFS Thrift/PHP connection doesnt work
Thread-Index: AcoG1GVLEd9JoU0gSvCkOJ13nEYjNg==
From: =?iso-8859-1?Q?J=FCrgen_Kaatz?= <kaatz@etracker.com>
To: <general@hadoop.apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org
X-Old-Spam-Status: No, score=-98.9 required=3.0 tests=BAYES_60,RDNS_NONE,
	USER_IN_WHITELIST autolearn=disabled version=3.2.5

Hi,

I'm trying to use thrift with PHP to handle file system operations on =
HDFS without any success.

I run a thrift server instance started with the script =
'start_thrift_server.sh' located in =
'hadoop/src/contrib/thriftfs/scripts/'. Then I use a little script =
'hdfs.php' to connect to this thrift server and try to create a new file =
'foo' on HDFS. But the file is created on my local filesystem not on =
HDFS.

Can anybody help me?
Thanks Juergen

-- hdfs.php =
------------------------------------------------------------------------
<?php
error_reporting(E_ALL);
ini_set('display_errors', 'on');

$GLOBALS['THRIFT_ROOT'] =3D '/home/hadoop/cc/thrift'; =
define('ETCC_THRIFT_ROOT', $GLOBALS['THRIFT_ROOT']);

require_once(ETCC_THRIFT_ROOT.'/Thrift.php' ); =
require_once(ETCC_THRIFT_ROOT.'/transport/TSocket.php' ); =
require_once(ETCC_THRIFT_ROOT.'/transport/TBufferedTransport.php' ); =
require_once(ETCC_THRIFT_ROOT.'/protocol/TBinaryProtocol.php' );

$socket =3D new TSocket('hadoop-master', 50021); =
$socket->setSendTimeout(10000); $socket->setRecvTimeout(20000); =
$transport =3D new TBufferedTransport($socket); $protocol =3D new =
TBinaryProtocol($transport);

require_once(ETCC_THRIFT_ROOT.'/packages/hadoopfs/ThriftHadoopFileSystem.=
php');
$client =3D new ThriftHadoopFileSystemClient($protocol);

$transport->open();

try
{
    $pathname =3D new Pathname(array('pathname' =3D> 'foo'));
    // $result =3D $client->listStatus($pathname);
    $result =3D $client->create($pathname);
    $client->close($result);
    print_r($result);
}
catch(Exception $e)
{
    print_r($e);
}

$transport->close();


From general-return-381-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 16:17:18 2009
Return-Path: <general-return-381-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23403 invoked from network); 17 Jul 2009 16:17:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 16:17:15 -0000
Received: (qmail 56345 invoked by uid 500); 17 Jul 2009 16:18:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56286 invoked by uid 500); 17 Jul 2009 16:18:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56276 invoked by uid 99); 17 Jul 2009 16:18:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 16:18:19 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cutting@gmail.com designates 209.85.219.219 as permitted sender)
Received: from [209.85.219.219] (HELO mail-ew0-f219.google.com) (209.85.219.219)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 16:18:10 +0000
Received: by ewy19 with SMTP id 19so949198ewy.29
        for <general@hadoop.apache.org>; Fri, 17 Jul 2009 09:17:49 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:sender:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=XJy6ZZOaRt6oujRuqEFiTmVpuLLeh++5QALwp/ryU00=;
        b=P0USzIv6WXsUGIOFY2s90lAzRO00bY+LcykckWmB4oZ5uCkM4Kv/LTXiuYD1QSogOr
         XIKDt71T9YTR8OOR7u9XgCv5nyR0njIRmLlHd8BJULNzZUJUOB/Ns1YPkGcyFinBgbcq
         Fy7dSGWlnzc71DPvhAi9aHGRrFsZIMQxS41JI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:date:from:user-agent:mime-version:to:subject
         :references:in-reply-to:content-type:content-transfer-encoding;
        b=rI4YjUlwysw/J/qbYYuNETaRGg3WrYcI/FPpLDKtC+i/LpQY59TwpGGSVdg7tgGi4U
         k2SzkRZW84mahX15qnnz7EadtQgZ/9/WvAGjUj2xDfqrHHByqj19g2Nz7qlhRe7bF6n3
         rgIPHrEedkWFo+s6ULRUYJaY9AVzldIqpzOWQ=
Received: by 10.216.7.207 with SMTP id 57mr351013wep.104.1247847469467;
        Fri, 17 Jul 2009 09:17:49 -0700 (PDT)
Received: from ?192.168.168.16? (c-76-103-179-105.hsd1.ca.comcast.net [76.103.179.105])
        by mx.google.com with ESMTPS id i35sm3947279gve.26.2009.07.17.09.17.46
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Fri, 17 Jul 2009 09:17:48 -0700 (PDT)
Sender: Doug Cutting <cutting@gmail.com>
Message-ID: <4A60A428.5070606@apache.org>
Date: Fri, 17 Jul 2009 09:17:44 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.22 (X11/20090608)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Downloads linking incorrectly or mirrors are not updated?
References: <32120a6a0907170444x61dac5a7s8988012351520459@mail.gmail.com>
In-Reply-To: <32120a6a0907170444x61dac5a7s8988012351520459@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks for catching this, Tim.  I just committed a fix that should be 
live within an hour.

Doug

tim robertson wrote:
> Hi guys,
> 
> I know you are reorganising the sub projects, but just for your info...
> The downloads linked from the Hadoop site
> (http://www.apache.org/dyn/closer.cgi/hadoop/mapreduce/) are all
> incorrect, or the mirrors are not updated (for what I see from Denmark
> anyway).
> All the links end in .../mapreduce/ but the mirrors have the downloads
> available on .../core/
> 
> Cheers,
> 
> Tim

From general-return-382-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 18 00:27:48 2009
Return-Path: <general-return-382-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46059 invoked from network); 18 Jul 2009 00:27:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jul 2009 00:27:46 -0000
Received: (qmail 38957 invoked by uid 500); 18 Jul 2009 00:28:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38885 invoked by uid 500); 18 Jul 2009 00:28:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 38875 invoked by uid 99); 18 Jul 2009 00:28:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jul 2009 00:28:51 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of dhruba@gmail.com designates 209.85.210.185 as permitted sender)
Received: from [209.85.210.185] (HELO mail-yx0-f185.google.com) (209.85.210.185)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jul 2009 00:28:40 +0000
Received: by yxe15 with SMTP id 15so2035362yxe.5
        for <general@hadoop.apache.org>; Fri, 17 Jul 2009 17:28:19 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=cYUQxNGW2TLCprjX9mwXTF28gKTlNNptTf4FxylLY9Q=;
        b=TgOJNL67ghHq9jR0koYrzqeqSwMXsXS0IGPyiPRLkovJjvNZG0mnkuj/XMdUtssZFj
         oCyevoVuF8IXUOjxAxPNLXpn+3zmKNqrYxAipIcg50MkQjm33JD8hxdCISV/5r5lbHZk
         Ouj5XU8sOAdPv33gBwitU2Ays2/eYnD0Fc3+w=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=opDrrq5b4+YmhBSX5H1mdjNIszkdiYn3imxBSNz4/qU9BA8iJ0eIEFtdKRFDbP1cVD
         9gL1PF24+rd84QIi4Yp0CrHET2azKTXcEU5gbQnALk2P2nS8w+yXBgLP4eEMnz4F2CQA
         iuQtXQg4ufpI6ExTsIBf+QTUuoJ5WoORZrhFY=
MIME-Version: 1.0
Received: by 10.151.10.14 with SMTP id n14mr1701834ybi.106.1247876899682; Fri, 
	17 Jul 2009 17:28:19 -0700 (PDT)
In-Reply-To: <E74E142F203F1A42A9288C280C60F8A5020AE0B6@ebase.etracker.local>
References: <E74E142F203F1A42A9288C280C60F8A5020AE0B6@ebase.etracker.local>
Date: Fri, 17 Jul 2009 17:28:19 -0700
Message-ID: <4aa34eb70907171728o6f089c08l9bdf8024920355e5@mail.gmail.com>
Subject: Re: Hadoop/HDFS Thrift/PHP connection doesnt work
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd6af0a54384f046eeffbdf
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd6af0a54384f046eeffbdf
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I am suspecting that the thrift server is started using the configuration i=
n
your hadoop workspace rather than using the configuration of the HDFS
cluster that you must have setup earlier. Please check the CLASSPATH of the
thrit server....it should include the configuration of your HDFS cluster.

thanks,
dhruba


2009/7/17 J=FCrgen Kaatz <kaatz@etracker.com>

> Hi,
>
> I'm trying to use thrift with PHP to handle file system operations on HDF=
S
> without any success.
>
> I run a thrift server instance started with the script
> 'start_thrift_server.sh' is supposed located in
> 'hadoop/src/contrib/thriftfs/scripts/'. Then I use a little script
> 'hdfs.php' to connect to this thrift server and try to create a new file
> 'foo' on HDFS. But the file is created on my local filesystem not on HDFS=
.
>
> Can anybody help me?
> Thanks Juergen
>
> -- hdfs.php
> ------------------------------------------------------------------------
> <?php
> error_reporting(E_ALL);
> ini_set('display_errors', 'on');
>
> $GLOBALS['THRIFT_ROOT'] =3D '/home/hadoop/cc/thrift';
> define('ETCC_THRIFT_ROOT', $GLOBALS['THRIFT_ROOT']);
>
> require_once(ETCC_THRIFT_ROOT.'/Thrift.php' );
> require_once(ETCC_THRIFT_ROOT.'/transport/TSocket.php' );
> require_once(ETCC_THRIFT_ROOT.'/transport/TBufferedTransport.php' );
> require_once(ETCC_THRIFT_ROOT.'/protocol/TBinaryProtocol.php' );
>
> $socket =3D new TSocket('hadoop-master', 50021);
> $socket->setSendTimeout(10000); $socket->setRecvTimeout(20000); $transpor=
t =3D
> new TBufferedTransport($socket); $protocol =3D new
> TBinaryProtocol($transport);
>
>
> require_once(ETCC_THRIFT_ROOT.'/packages/hadoopfs/ThriftHadoopFileSystem.=
php');
> $client =3D new ThriftHadoopFileSystemClient($protocol);
>
> $transport->open();
>
> try
> {
>    $pathname =3D new Pathname(array('pathname' =3D> 'foo'));
>    // $result =3D $client->listStatus($pathname);
>    $result =3D $client->create($pathname);
>    $client->close($result);
>    print_r($result);
> }
> catch(Exception $e)
> {
>    print_r($e);
> }
>
> $transport->close();
>
>

--000e0cd6af0a54384f046eeffbdf--

From general-return-383-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 20 09:16:05 2009
Return-Path: <general-return-383-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6067 invoked from network); 20 Jul 2009 09:16:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jul 2009 09:16:05 -0000
Received: (qmail 86236 invoked by uid 500); 20 Jul 2009 09:17:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86158 invoked by uid 500); 20 Jul 2009 09:17:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86148 invoked by uid 99); 20 Jul 2009 09:17:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 09:17:09 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [85.183.249.146] (HELO mail.etracker.de) (85.183.249.146)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 09:16:57 +0000
Received: by mail.etracker.de (Postfix, from userid 1003)
	id 71471641710; Mon, 20 Jul 2009 11:16:36 +0200 (CEST)
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail.etracker.de
X-Spam-Level: 
Received: from mail.etracker.local (unknown [85.183.249.129])
	by mail.etracker.de (Postfix) with ESMTP id B588464170D
	for <general@hadoop.apache.org>; Mon, 20 Jul 2009 11:16:35 +0200 (CEST)
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Subject: RE: Hadoop/HDFS Thrift/PHP connection doesnt work
X-MimeOLE: Produced By Microsoft Exchange V6.5
Date: Mon, 20 Jul 2009 11:16:36 +0200
Message-ID: <E74E142F203F1A42A9288C280C60F8A5020AE211@ebase.etracker.local>
In-Reply-To: <4aa34eb70907171728o6f089c08l9bdf8024920355e5@mail.gmail.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Hadoop/HDFS Thrift/PHP connection doesnt work
Thread-Index: AcoHPrxvDBDumW3mTdW1ZiIyInMobAB2mIDA
References: <E74E142F203F1A42A9288C280C60F8A5020AE0B6@ebase.etracker.local> <4aa34eb70907171728o6f089c08l9bdf8024920355e5@mail.gmail.com>
From: =?iso-8859-1?Q?J=FCrgen_Kaatz?= <kaatz@etracker.com>
To: <general@hadoop.apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org
X-Old-Spam-Status: No, score=-97.9 required=3.0 tests=BAYES_80,RDNS_NONE,
	USER_IN_WHITELIST autolearn=disabled version=3.2.5

Hi dhruba,

thank you for your answer. I had the same idea like you before, that the =
thrift server configuration cannot read the hadoop cluster =
configuration. The Hadoop-Cluster is installed in '/usr/local/hadoop' =
and the respective configuration in '/usr/local/hadoop/conf'. You can =
watch the classpath of the thrift server below (and the respective =
script to start the thrift server). For me it seems all is ok...

The Java-Code of the thrift server has a method 'public =
HadoopThriftHandler()', where a configuration is generated. But for me =
it seems, this must be an empty or default configuration, isn't it?

Thanks in advance
Juergen


Method 'HadoopThriftHandler' from =
'hadoop/src/contrib/thriftfs/src/java/org/apache/hadoop/thriftfs/HadoopTh=
riftServer.java'

    /**
     * HadoopThriftServer
     *
     * Constructor for the HadoopThriftServer glue with Thrift Class.
     *
     * @param name - the name of this handler
     */
    public HadoopThriftHandler(String name) {
      conf =3D new Configuration();
      now =3D now();
      try {
        inactivityThread =3D new Daemon(new InactivityMonitor());
        fs =3D FileSystem.get(conf);
      } catch (IOException e) {
        LOG.warn("Unable to open hadoop file system...");
        Runtime.getRuntime().exit(-1);
      }
    }=20

Classpath of the thrift server:
/usr/local/hadoop/hadoop-0.19.1-ant.jar
/usr/local/hadoop/hadoop-0.19.1-core.jar
/usr/local/hadoop/hadoop-0.19.1-examples.jar
/usr/local/hadoop/hadoop-0.19.1-test.jar
/usr/local/hadoop/hadoop-0.19.1-tools.jar
/usr/local/hadoop/lib/commons-cli-2.0-SNAPSHOT.jar
/usr/local/hadoop/lib/commons-codec-1.3.jar
/usr/local/hadoop/lib/commons-httpclient-3.0.1.jar
/usr/local/hadoop/lib/commons-logging-1.0.4.jar
/usr/local/hadoop/lib/commons-logging-api-1.0.4.jar
/usr/local/hadoop/lib/commons-net-1.4.1.jar
/usr/local/hadoop/lib/hsqldb-1.8.0.10.jar
/usr/local/hadoop/lib/jets3t-0.6.1.jar
/usr/local/hadoop/lib/jetty-5.1.4.jar
/usr/local/hadoop/lib/junit-3.8.1.jar
/usr/local/hadoop/lib/kfs-0.2.0.jar
/usr/local/hadoop/lib/log4j-1.2.15.jar
/usr/local/hadoop/lib/oro-2.0.8.jar
/usr/local/hadoop/lib/servlet-api.jar
/usr/local/hadoop/lib/slf4j-api-1.4.3.jar
/usr/local/hadoop/lib/slf4j-log4j12-1.4.3.jar
/usr/local/hadoop/lib/xmlenc-0.52.jar
/usr/local/hadoop/contrib/hive/lib/antlr-runtime-3.0.1.jar
/usr/local/hadoop/contrib/hive/lib/asm-3.1.jar
/usr/local/hadoop/contrib/hive/lib/commons-collections-3.2.1.jar
/usr/local/hadoop/contrib/hive/lib/commons-jexl-1.1.jar
/usr/local/hadoop/contrib/hive/lib/commons-lang-2.4.jar
/usr/local/hadoop/contrib/hive/lib/derby.jar
/usr/local/hadoop/contrib/hive/lib/hive_cli.jar
/usr/local/hadoop/contrib/hive/lib/hive_common.jar
/usr/local/hadoop/contrib/hive/lib/hive_exec.jar
/usr/local/hadoop/contrib/hive/lib/hive_metastore.jar
/usr/local/hadoop/contrib/hive/lib/jdo2-api-2.1.jar
/usr/local/hadoop/contrib/hive/lib/jline-0.9.94.jar
/usr/local/hadoop/contrib/hive/lib/jpox-core-1.2.2.jar
/usr/local/hadoop/contrib/hive/lib/jpox-enhancer-1.2.2.jar
/usr/local/hadoop/contrib/hive/lib/jpox-rdbms-1.2.2.jar
/usr/local/hadoop/contrib/hive/lib/libfb303.jar
/usr/local/hadoop/contrib/hive/lib/libthrift.jar
/usr/local/hadoop/contrib/hive/lib/stringtemplate-3.1b1.jar
/usr/local/hadoop/contrib/hive/lib/TestSerDe.jar
/usr/local/hadoop/contrib/hive/lib/velocity-1.5.jar
/usr/local/hadoop/contrib/thriftfs/hadoop-0.19.1-thriftfs.jar
/usr/local/hadoop/src/contrib/thriftfs/lib/hadoopthriftapi.jar
/usr/local/hadoop/src/contrib/thriftfs/lib/libthrift.jar

Script to start the thrift server:
#!/bin/sh

CLASSPATH=3D
#TOP=3D../../../..
TOP=3D/usr/local/hadoop

# the hadoop libraries
for f in $TOP/*.jar ; do
  CLASSPATH=3D$CLASSPATH:$f
done

# the apache libraries
for f in $TOP/lib/*.jar ; do
  CLASSPATH=3D$CLASSPATH:$f
done

# the thrift libraries
for f in $TOP/contrib/hive/lib/*.jar ; do
  CLASSPATH=3D$CLASSPATH:$f
done

# the thrift server
for f in $TOP/contrib/thriftfs/*.jar ; do
  CLASSPATH=3D$CLASSPATH:$f
done
# the thrift hadoop api
for f in $TOP/src/contrib/thriftfs/lib/*.jar ; do
  CLASSPATH=3D$CLASSPATH:$f
done

echo $CLASSPATH
exit
#-agentlib:jdwp=3Dtransport=3Ddt_socket,address=3D8000,server=3Dy,suspend=
=3Dn
java -Dcom.sun.management.jmxremote -cp $CLASSPATH =
org.apache.hadoop.thriftfs.HadoopThriftServer $*



> -----Original Message-----
> From: Dhruba Borthakur [mailto:dhruba@gmail.com]
> Sent: Saturday, July 18, 2009 2:28 AM
> To: general@hadoop.apache.org
> Subject: Re: Hadoop/HDFS Thrift/PHP connection doesnt work
>=20
> I am suspecting that the thrift server is started using the =
configuration in
> your hadoop workspace rather than using the configuration of the HDFS
> cluster that you must have setup earlier. Please check the CLASSPATH =
of the
> thrit server....it should include the configuration of your HDFS =
cluster.
>=20
> thanks,
> dhruba
>=20
>=20
> 2009/7/17 J=FCrgen Kaatz <kaatz@etracker.com>
>=20
> > Hi,
> >
> > I'm trying to use thrift with PHP to handle file system operations =
on HDFS
> > without any success.
> >
> > I run a thrift server instance started with the script
> > 'start_thrift_server.sh' is supposed located in
> > 'hadoop/src/contrib/thriftfs/scripts/'. Then I use a little script
> > 'hdfs.php' to connect to this thrift server and try to create a new =
file
> > 'foo' on HDFS. But the file is created on my local filesystem not on =
HDFS.
> >
> > Can anybody help me?
> > Thanks Juergen
> >
> > -- hdfs.php
> > =
------------------------------------------------------------------------
> > <?php
> > error_reporting(E_ALL);
> > ini_set('display_errors', 'on');
> >
> > $GLOBALS['THRIFT_ROOT'] =3D '/home/hadoop/cc/thrift';
> > define('ETCC_THRIFT_ROOT', $GLOBALS['THRIFT_ROOT']);
> >
> > require_once(ETCC_THRIFT_ROOT.'/Thrift.php' );
> > require_once(ETCC_THRIFT_ROOT.'/transport/TSocket.php' );
> > require_once(ETCC_THRIFT_ROOT.'/transport/TBufferedTransport.php' );
> > require_once(ETCC_THRIFT_ROOT.'/protocol/TBinaryProtocol.php' );
> >
> > $socket =3D new TSocket('hadoop-master', 50021);
> > $socket->setSendTimeout(10000); $socket->setRecvTimeout(20000); =
$transport =3D
> > new TBufferedTransport($socket); $protocol =3D new
> > TBinaryProtocol($transport);
> >
> >
> >
> =
require_once(ETCC_THRIFT_ROOT.'/packages/hadoopfs/ThriftHadoopFileSystem.=
php')
> ;
> > $client =3D new ThriftHadoopFileSystemClient($protocol);
> >
> > $transport->open();
> >
> > try
> > {
> >    $pathname =3D new Pathname(array('pathname' =3D> 'foo'));
> >    // $result =3D $client->listStatus($pathname);
> >    $result =3D $client->create($pathname);
> >    $client->close($result);
> >    print_r($result);
> > }
> > catch(Exception $e)
> > {
> >    print_r($e);
> > }
> >
> > $transport->close();
> >
> >

From general-return-384-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 20 14:42:21 2009
Return-Path: <general-return-384-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42788 invoked from network); 20 Jul 2009 14:42:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jul 2009 14:42:21 -0000
Received: (qmail 78564 invoked by uid 500); 20 Jul 2009 14:43:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78489 invoked by uid 500); 20 Jul 2009 14:43:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78471 invoked by uid 99); 20 Jul 2009 14:43:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 14:43:25 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of nicc777@gmail.com designates 209.85.218.221 as permitted sender)
Received: from [209.85.218.221] (HELO mail-bw0-f221.google.com) (209.85.218.221)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 14:43:17 +0000
Received: by bwz21 with SMTP id 21so1061960bwz.29
        for <general@hadoop.apache.org>; Mon, 20 Jul 2009 07:42:56 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=WHIvJBDAnq9Z7xBkSZo2wKvkCkf+XOu6nV5ltiGHN0w=;
        b=H5EODEA7FWIhJij733/NyluQeo6dnDLB4INy9a/XwCx7sQPGBCnVkEd+6psQ5uytQx
         SFD46rmnD8UN1jjhK7f7FgSrqYdNxS7PNW194Ze95hfH9khLVQoWpozKmYKCBGBEX+W9
         0ndCxrEteYl/Rv2Ao6YaUurapdRg8hKQOBWY8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=ht4pjgyHqy6PoqslEx1aDZ6dwtjpw4zU5QNBF4CsPqSX1iwEUxjJ6x0b7uvspbyxpC
         nCYhIPwt5BwIYi6EclAOdHVKwbTS3wFFsWcOdvMtR3sEBxVshdbbPJpKoLbkpGODs9aH
         Cy400euDQhrF1t9PBtXgkEhcbyNSHIQFp+ywo=
MIME-Version: 1.0
Received: by 10.204.67.146 with SMTP id r18mr4352978bki.88.1248100976095; Mon, 
	20 Jul 2009 07:42:56 -0700 (PDT)
Date: Mon, 20 Jul 2009 16:42:56 +0200
Message-ID: <e21a770d0907200742v596044a0gf50057ad530ac3c8@mail.gmail.com>
Subject: Job Management - getting practical
From: Nico Coetzee <nicc777@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c5acb052f012046f242767
X-Virus-Checked: Checked by ClamAV on apache.org

--001636c5acb052f012046f242767
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi again,

Our testing is going fantastic and I have a 4 node cluster going to our
production environment mid-August (currently testing on a 3 node cluster).

I am now looking more at the operational side of things and I have a number
of questions around a GUI app to manage jobs (preferably browser based).

Our basic requirements:


   - Define a unique job with parameters:
      - Data source (in our case  a file (or files) on remote systems
      - Method of retrieving above files (most are scp, but some are ftp)
      - which cluster to push the job to (test or production cluster)
      - Which "spool" directory to use on the hadoop master (where I copy
      the raw data before uploading to the HDFS)
      - input and output directories to use for this job in HDFS
      - Where the output must be dumped after processing (export from HDFS)
      - What (if any) post-processing needs to take place (upload or link to
      relevant scripts
      - Of course also upload or link to the map and reduce scripts (I still
      use and prefer the streaming solution)
      - Define the run schedule of the job (almost 90% of our jobs will be
      run at regular intervals - at least once per day. Think along
crontab lines)
      - Additional nice to have requirements:
   - Define mail addresses of people that should get the job report after it
      was run
      - Ability to move a job (with it's scripts etc.) from a "test" cluster
      to a "production" cluster (we test everything first before we
put stuff in
      production)
      - Run certain jobs manually (once off jobs. manually re-run failed
      jobs)


If such a system does not exist in the Open Source community I wonder if
there will be sufficient interest if I start a project like this?

Thanks for your feedback and suggestions

Nico

--001636c5acb052f012046f242767--

From general-return-385-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 20 15:01:40 2009
Return-Path: <general-return-385-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 48764 invoked from network); 20 Jul 2009 15:01:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jul 2009 15:01:40 -0000
Received: (qmail 13208 invoked by uid 500); 20 Jul 2009 15:02:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13160 invoked by uid 500); 20 Jul 2009 15:02:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13139 invoked by uid 99); 20 Jul 2009 15:02:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 15:02:40 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 15:02:28 +0000
Received: from [192.168.1.64] (snvvpn1-10-73-152-c44.hq.corp.yahoo.com [10.73.152.44])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n6KF15Vj099758
	for <general@hadoop.apache.org>; Mon, 20 Jul 2009 08:01:06 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=lOWnbJQKAJcln4OFLIqsXHat7ZGRfZ/cnh/jtUxRrNxMJg1oHBlNW1tJw7cbd6V0
Message-Id: <983BD9F5-EE38-4129-8D23-F513023D96C6@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <e21a770d0907200742v596044a0gf50057ad530ac3c8@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v935.3)
Subject: Re: Job Management - getting practical
Date: Mon, 20 Jul 2009 08:01:05 -0700
References: <e21a770d0907200742v596044a0gf50057ad530ac3c8@mail.gmail.com>
X-Mailer: Apple Mail (2.935.3)
X-Virus-Checked: Checked by ClamAV on apache.org

Nico,

  You might want to take a look at Oozie: http://issues.apache.org/jira/browse/HADOOP-5303 
.

Arun

On Jul 20, 2009, at 7:42 AM, Nico Coetzee wrote:

> Hi again,
>
> Our testing is going fantastic and I have a 4 node cluster going to  
> our
> production environment mid-August (currently testing on a 3 node  
> cluster).
>
> I am now looking more at the operational side of things and I have a  
> number
> of questions around a GUI app to manage jobs (preferably browser  
> based).
>
> Our basic requirements:
>
>
>   - Define a unique job with parameters:
>      - Data source (in our case  a file (or files) on remote systems
>      - Method of retrieving above files (most are scp, but some are  
> ftp)
>      - which cluster to push the job to (test or production cluster)
>      - Which "spool" directory to use on the hadoop master (where I  
> copy
>      the raw data before uploading to the HDFS)
>      - input and output directories to use for this job in HDFS
>      - Where the output must be dumped after processing (export from  
> HDFS)
>      - What (if any) post-processing needs to take place (upload or  
> link to
>      relevant scripts
>      - Of course also upload or link to the map and reduce scripts  
> (I still
>      use and prefer the streaming solution)
>      - Define the run schedule of the job (almost 90% of our jobs  
> will be
>      run at regular intervals - at least once per day. Think along
> crontab lines)
>      - Additional nice to have requirements:
>   - Define mail addresses of people that should get the job report  
> after it
>      was run
>      - Ability to move a job (with it's scripts etc.) from a "test"  
> cluster
>      to a "production" cluster (we test everything first before we
> put stuff in
>      production)
>      - Run certain jobs manually (once off jobs. manually re-run  
> failed
>      jobs)
>
>
> If such a system does not exist in the Open Source community I  
> wonder if
> there will be sufficient interest if I start a project like this?
>
> Thanks for your feedback and suggestions
>
> Nico


From general-return-386-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 20 15:35:11 2009
Return-Path: <general-return-386-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58326 invoked from network); 20 Jul 2009 15:35:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jul 2009 15:35:11 -0000
Received: (qmail 53503 invoked by uid 500); 20 Jul 2009 15:36:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53437 invoked by uid 500); 20 Jul 2009 15:36:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53243 invoked by uid 99); 20 Jul 2009 15:36:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 15:36:15 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of nicc777@gmail.com designates 209.85.218.221 as permitted sender)
Received: from [209.85.218.221] (HELO mail-bw0-f221.google.com) (209.85.218.221)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 15:36:07 +0000
Received: by bwz21 with SMTP id 21so1097872bwz.29
        for <general@hadoop.apache.org>; Mon, 20 Jul 2009 08:35:46 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=gUvrv3lWkY21eSfRLSdA5A4x2PJ8PwKtgGQqvVk+i64=;
        b=M9tfo/OBjXh1vRgIWFAqmnEmlLrV+kiFC/DTkFTFfbU22/7WYAXQebtvGmRCY+OWO1
         qOdCXAkNqP8NYj6E+ABnOt1QulKUR+DxvWqkQRq+66xiNICNB9QdH5mkLiJxkY2Eb4n2
         HpF8xwwdUxSts7tWFOJuCrYJ8wCZM6yE6C/B4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=XAXJVdCuixkwFy5K3RwWrBsTdITRkrh/STBSPRiXNnyyHC4QNtwHYSrFaPooM/3ilp
         VxjQcqL8kH2d77AlfPFdf/tHGj8d1KJZ0K/wzKYNDhVFNS7yNQ5zsPv3u2TOQG+w3Qc6
         O7EJLFukOd0OgA8lz0XTTRKMVtY395EFvAvV8=
MIME-Version: 1.0
Received: by 10.204.68.15 with SMTP id t15mr4360293bki.139.1248104145780; Mon, 
	20 Jul 2009 08:35:45 -0700 (PDT)
In-Reply-To: <983BD9F5-EE38-4129-8D23-F513023D96C6@yahoo-inc.com>
References: <e21a770d0907200742v596044a0gf50057ad530ac3c8@mail.gmail.com>
	 <983BD9F5-EE38-4129-8D23-F513023D96C6@yahoo-inc.com>
Date: Mon, 20 Jul 2009 17:35:44 +0200
Message-ID: <e21a770d0907200835p610b86c7y7c33117f060d3e02@mail.gmail.com>
Subject: Re: Job Management - getting practical
From: Nico Coetzee <nicc777@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c5bcf940950c046f24e44d
X-Virus-Checked: Checked by ClamAV on apache.org

--001636c5bcf940950c046f24e44d
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Thanks - this looks like it has potential. What I'm thinking is then to
maybe just build a UI to basically "populate" or "build" the Oozie XML...



On Mon, Jul 20, 2009 at 5:01 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:

> Nico,
>
>  You might want to take a look at Oozie:
> http://issues.apache.org/jira/browse/HADOOP-5303.
>
> Arun
>
>
> On Jul 20, 2009, at 7:42 AM, Nico Coetzee wrote:
>
>  Hi again,
>>
>> Our testing is going fantastic and I have a 4 node cluster going to our
>> production environment mid-August (currently testing on a 3 node cluster).
>>
>> I am now looking more at the operational side of things and I have a
>> number
>> of questions around a GUI app to manage jobs (preferably browser based).
>>
>> Our basic requirements:
>>
>>
>>  - Define a unique job with parameters:
>>     - Data source (in our case  a file (or files) on remote systems
>>     - Method of retrieving above files (most are scp, but some are ftp)
>>     - which cluster to push the job to (test or production cluster)
>>     - Which "spool" directory to use on the hadoop master (where I copy
>>     the raw data before uploading to the HDFS)
>>     - input and output directories to use for this job in HDFS
>>     - Where the output must be dumped after processing (export from HDFS)
>>     - What (if any) post-processing needs to take place (upload or link to
>>     relevant scripts
>>     - Of course also upload or link to the map and reduce scripts (I still
>>     use and prefer the streaming solution)
>>     - Define the run schedule of the job (almost 90% of our jobs will be
>>     run at regular intervals - at least once per day. Think along
>> crontab lines)
>>     - Additional nice to have requirements:
>>  - Define mail addresses of people that should get the job report after it
>>     was run
>>     - Ability to move a job (with it's scripts etc.) from a "test" cluster
>>     to a "production" cluster (we test everything first before we
>> put stuff in
>>     production)
>>     - Run certain jobs manually (once off jobs. manually re-run failed
>>     jobs)
>>
>>
>> If such a system does not exist in the Open Source community I wonder if
>> there will be sufficient interest if I start a project like this?
>>
>> Thanks for your feedback and suggestions
>>
>> Nico
>>
>
>

--001636c5bcf940950c046f24e44d--

From general-return-387-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 20 15:40:02 2009
Return-Path: <general-return-387-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60324 invoked from network); 20 Jul 2009 15:40:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jul 2009 15:40:02 -0000
Received: (qmail 58516 invoked by uid 500); 20 Jul 2009 15:41:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58436 invoked by uid 500); 20 Jul 2009 15:41:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58426 invoked by uid 99); 20 Jul 2009 15:41:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 15:41:06 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of parbash.project@gmail.com designates 209.85.220.163 as permitted sender)
Received: from [209.85.220.163] (HELO mail-fx0-f163.google.com) (209.85.220.163)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 15:40:56 +0000
Received: by fxm7 with SMTP id 7so359376fxm.5
        for <general@hadoop.apache.org>; Mon, 20 Jul 2009 08:40:36 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from
         :user-agent:mime-version:to:subject:content-type
         :content-transfer-encoding;
        bh=O6K2THrEn7y6BLHAytug/44umWlmBxSfFonN2leDX5Q=;
        b=G+cJOZ01Tyon5dnMByV2KSoGxTM88SM6KfcEl/F7uSpLquebz70G7wQWZ9KfryZkFe
         CZ9W0E9Q8b/jEpxzhlMGXjL+CPG4EAEKEjWf0XxjtOGhC9sPN3egP793OzOenjgEX9yF
         AkbaSPUfkoLmaEWGpnq3Y5G8VLBTKXL+3g50I=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject
         :content-type:content-transfer-encoding;
        b=lywA7SVhvyASTclTboUSFdhU6A3w5/dhwaXBSsaj05A7Us9K5njXvlfG0c5/6fiMyZ
         6qzzu3OCheeMmKl3M4wpYY8YR9ZKnDVdUZFlvsQQNtOVRxi28xe+6p+iunjR6wHfrb/g
         1wbj3eEbHLkD++X+QKZu1UCIw7Ez/Y/mFKzJQ=
Received: by 10.102.220.10 with SMTP id s10mr411222mug.94.1248104436148;
        Mon, 20 Jul 2009 08:40:36 -0700 (PDT)
Received: from W852120CRG3.pwlan-swisscom.mobile.ch ([194.209.131.192])
        by mx.google.com with ESMTPS id j10sm22538067muh.45.2009.07.20.08.40.34
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Mon, 20 Jul 2009 08:40:35 -0700 (PDT)
Message-ID: <4A648FF2.3080306@gmail.com>
Date: Mon, 20 Jul 2009 17:40:34 +0200
From: Milenko Petrovic <parbash.project@gmail.com>
User-Agent: Thunderbird 2.0.0.22 (Macintosh/20090605)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: ANNOUNCE: ParBASH 0.1 release - Hadoop through BASH
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hello,

I'd like to announce the release of the 0.1 version of ParBASH. Using 
ParBASH, it is possible to write bash scripts that are automatically 
translated into Hadoop Streaming Jobs.

Here is an example script to find top 10 references for Barack Obama 
pages on wikipedia using Amazon EC2:

wiki.sh:

cat hdfs:/wikipedia-out/* | grep Obama | \
perl -ne 'while (/<link type="external" href="([^"]+)">/g) { print 
"$1\n"; }' |\perl -ne 'if (/http:\/\/([^\/]+)(\/|$)/) { print "$1\n"; }' |\
perl -ne '
  if (/([^\.]\.)+([^\.]+\.[a-zA-Z]{2,3}\.[^\.]+)$/) { print "$2\n";}
  else if (/([^\.]+\.[a-zA-Z]{2,3}\.[^\.]+)$/) { print "$1\n";}
  else if (/([^\.]\.)*([^\.]+\.[^\.]+)$/) { print "$2\n"; }' |\
sort | uniq -c > hdfs:/out

How and why of wiki.sh and parbash on
http://cloud-dev.blogspot.com/2009/06/introduction-to-parbash.html

Source code and more examples:
http://code.google.com/p/parbash

If anyone is interested in trying it out, I can help you get started.

Thanks,
Milenko


From general-return-388-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 20 16:51:30 2009
Return-Path: <general-return-388-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 92418 invoked from network); 20 Jul 2009 16:51:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jul 2009 16:51:30 -0000
Received: (qmail 64751 invoked by uid 500); 20 Jul 2009 16:52:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 64678 invoked by uid 500); 20 Jul 2009 16:52:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 64668 invoked by uid 99); 20 Jul 2009 16:52:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 16:52:34 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [62.99.145.7] (HELO mx.inode.at) (62.99.145.7)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 16:52:23 +0000
Received: from [213.47.112.79] (port=2372 helo=localhost)
	by smartmx-07.inode.at with esmtpa (Exim 4.69)
	(envelope-from <rho@devc.at>)
	id 1MSw5q-0002v9-DY; Mon, 20 Jul 2009 18:52:02 +0200
Date: Mon, 20 Jul 2009 18:52:01 +0200
From: Robert Barta <rho@devc.at>
To: general@hadoop.apache.org
Subject: Re: ANNOUNCE: ParBASH 0.1 release - Hadoop through BASH
Message-ID: <20090720165201.GF26881@odman.int.devc.at>
Reply-To: rho@devc.at
References: <4A648FF2.3080306@gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <4A648FF2.3080306@gmail.com>
Sender: Robert Barta <rho@devc.at>
User-Agent: Mutt/1.5.18 (2008-05-17)
X-Virus-Checked: Checked by ClamAV on apache.org

On Mon, Jul 20, 2009 at 05:40:34PM +0200, Milenko Petrovic wrote:
> Hello,
>
> I'd like to announce the release of the 0.1 version of ParBASH. Using  
> ParBASH, it is possible to write bash scripts that are automatically  
> translated into Hadoop Streaming Jobs.

Milenko,

How does this related to

   http://md.devc.at/software/mapreduce/bashreduce

?

\rho

From general-return-389-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 20 18:00:30 2009
Return-Path: <general-return-389-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47339 invoked from network); 20 Jul 2009 18:00:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jul 2009 18:00:30 -0000
Received: (qmail 63076 invoked by uid 500); 20 Jul 2009 18:01:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62994 invoked by uid 500); 20 Jul 2009 18:01:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62984 invoked by uid 99); 20 Jul 2009 18:01:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 18:01:34 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [85.183.249.146] (HELO mail.etracker.de) (85.183.249.146)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 18:01:23 +0000
Received: by mail.etracker.de (Postfix, from userid 1003)
	id F328764FC42; Mon, 20 Jul 2009 20:01:01 +0200 (CEST)
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail.etracker.de
X-Spam-Level: 
Received: from mail.etracker.local (unknown [85.183.249.129])
	by mail.etracker.de (Postfix) with ESMTP id 66D8364FC3E
	for <general@hadoop.apache.org>; Mon, 20 Jul 2009 20:01:01 +0200 (CEST)
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Subject: RE: Hadoop/HDFS Thrift/PHP connection doesnt work
X-MimeOLE: Produced By Microsoft Exchange V6.5
Date: Mon, 20 Jul 2009 20:01:00 +0200
Message-ID: <E74E142F203F1A42A9288C280C60F8A5020AE2AB@ebase.etracker.local>
In-Reply-To: <4aa34eb70907171728o6f089c08l9bdf8024920355e5@mail.gmail.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Hadoop/HDFS Thrift/PHP connection doesnt work
Thread-Index: AcoHPrxvDBDumW3mTdW1ZiIyInMobACJNzgg
References: <E74E142F203F1A42A9288C280C60F8A5020AE0B6@ebase.etracker.local> <4aa34eb70907171728o6f089c08l9bdf8024920355e5@mail.gmail.com>
From: =?iso-8859-1?Q?J=FCrgen_Kaatz?= <kaatz@etracker.com>
To: <general@hadoop.apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org
X-Old-Spam-Status: No, score=-96.9 required=3.0 tests=BAYES_95,RDNS_NONE,
	USER_IN_WHITELIST autolearn=disabled version=3.2.5

Hi Dhruba,

I found the missing configuration to make my script working. Like =
mentioned in the API =
(http://hadoop.apache.org/common/docs/r0.19.1/api/index.html) I have to =
add the configuration path '/usr/local/hadoop/conf' to the beginning of =
classpath in the script 'start_thrift_server.sh'. Now it works!

Thank you very much
Juergen

> -----Original Message-----
> From: Dhruba Borthakur [mailto:dhruba@gmail.com]
> Sent: Saturday, July 18, 2009 2:28 AM
> To: general@hadoop.apache.org
> Subject: Re: Hadoop/HDFS Thrift/PHP connection doesnt work
>=20
> I am suspecting that the thrift server is started using the =
configuration in
> your hadoop workspace rather than using the configuration of the HDFS
> cluster that you must have setup earlier. Please check the CLASSPATH =
of the
> thrit server....it should include the configuration of your HDFS =
cluster.
>=20
> thanks,
> dhruba
>=20
>=20
> 2009/7/17 J=FCrgen Kaatz <kaatz@etracker.com>
>=20
> > Hi,
> >
> > I'm trying to use thrift with PHP to handle file system operations =
on HDFS
> > without any success.
> >
> > I run a thrift server instance started with the script
> > 'start_thrift_server.sh' is supposed located in
> > 'hadoop/src/contrib/thriftfs/scripts/'. Then I use a little script
> > 'hdfs.php' to connect to this thrift server and try to create a new =
file
> > 'foo' on HDFS. But the file is created on my local filesystem not on =
HDFS.
> >
> > Can anybody help me?
> > Thanks Juergen
> >
> > -- hdfs.php
> > =
------------------------------------------------------------------------
> > <?php
> > error_reporting(E_ALL);
> > ini_set('display_errors', 'on');
> >
> > $GLOBALS['THRIFT_ROOT'] =3D '/home/hadoop/cc/thrift';
> > define('ETCC_THRIFT_ROOT', $GLOBALS['THRIFT_ROOT']);
> >
> > require_once(ETCC_THRIFT_ROOT.'/Thrift.php' );
> > require_once(ETCC_THRIFT_ROOT.'/transport/TSocket.php' );
> > require_once(ETCC_THRIFT_ROOT.'/transport/TBufferedTransport.php' );
> > require_once(ETCC_THRIFT_ROOT.'/protocol/TBinaryProtocol.php' );
> >
> > $socket =3D new TSocket('hadoop-master', 50021);
> > $socket->setSendTimeout(10000); $socket->setRecvTimeout(20000); =
$transport =3D
> > new TBufferedTransport($socket); $protocol =3D new
> > TBinaryProtocol($transport);
> >
> >
> >
> =
require_once(ETCC_THRIFT_ROOT.'/packages/hadoopfs/ThriftHadoopFileSystem.=
php')
> ;
> > $client =3D new ThriftHadoopFileSystemClient($protocol);
> >
> > $transport->open();
> >
> > try
> > {
> >    $pathname =3D new Pathname(array('pathname' =3D> 'foo'));
> >    // $result =3D $client->listStatus($pathname);
> >    $result =3D $client->create($pathname);
> >    $client->close($result);
> >    print_r($result);
> > }
> > catch(Exception $e)
> > {
> >    print_r($e);
> > }
> >
> > $transport->close();
> >
> >

From general-return-390-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 20 18:12:15 2009
Return-Path: <general-return-390-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54121 invoked from network); 20 Jul 2009 18:12:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jul 2009 18:12:15 -0000
Received: (qmail 83848 invoked by uid 500); 20 Jul 2009 18:13:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83757 invoked by uid 500); 20 Jul 2009 18:13:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83747 invoked by uid 99); 20 Jul 2009 18:13:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 18:13:20 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of dhruba@gmail.com designates 209.85.210.185 as permitted sender)
Received: from [209.85.210.185] (HELO mail-yx0-f185.google.com) (209.85.210.185)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 18:13:09 +0000
Received: by yxe15 with SMTP id 15so3992419yxe.5
        for <general@hadoop.apache.org>; Mon, 20 Jul 2009 11:12:48 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=UzY4XdPxIs0NI2c0D1/48GjAOhgs4h/M2pizGZ7ySmA=;
        b=mX/6Z9nEuJdSe76yhtnfjdsbCH0ONpVpe0EqCweljBvqn+uGJGF/BcxdCFace9NBr1
         GntteZyBFBzfzDNfUfgyiD13k4u2M3srjXUtwsC7+tBDANQjkRLyAggYQdqmqZXuE2NT
         yx5ycXII9knYev9LEZtIC6B+CXO5/G2/SJM1o=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=cnV0QN0HzGPDmMy81hzi/LSro9Gr97LThIaidC9eH58Qn2cXhOGQoueTi6j1ZyuGhk
         sLnL/7PRc3ZjnuSVlaL/bCND2RB7wgzhRUdxNSU4GbmZNRNrXzrnpNgLwqfwRgEU1t18
         CjSlYsXn0OWQECcAKLrmZVAqdHqhpFocPJsAI=
MIME-Version: 1.0
Received: by 10.150.190.5 with SMTP id n5mr7091838ybf.273.1248113568244; Mon, 
	20 Jul 2009 11:12:48 -0700 (PDT)
In-Reply-To: <E74E142F203F1A42A9288C280C60F8A5020AE2AB@ebase.etracker.local>
References: <E74E142F203F1A42A9288C280C60F8A5020AE0B6@ebase.etracker.local>
	 <4aa34eb70907171728o6f089c08l9bdf8024920355e5@mail.gmail.com>
	 <E74E142F203F1A42A9288C280C60F8A5020AE2AB@ebase.etracker.local>
Date: Mon, 20 Jul 2009 11:12:48 -0700
Message-ID: <4aa34eb70907201112x60dbfca6tb03db50faa09c321@mail.gmail.com>
Subject: Re: Hadoop/HDFS Thrift/PHP connection doesnt work
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd340f0dfe5bc046f27154e
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd340f0dfe5bc046f27154e
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Excellent. Good to know that it is working for you.

-dhruba


2009/7/20 J=FCrgen Kaatz <kaatz@etracker.com>

> Hi Dhruba,
>
> I found the missing configuration to make my script working. Like mention=
ed
> in the API (http://hadoop.apache.org/common/docs/r0.19.1/api/index.html) =
I
> have to add the configuration path '/usr/local/hadoop/conf' to the beginn=
ing
> of classpath in the script 'start_thrift_server.sh'. Now it works!
>
> Thank you very much
> Juergen
>
> > -----Original Message-----
> > From: Dhruba Borthakur [mailto:dhruba@gmail.com]
> > Sent: Saturday, July 18, 2009 2:28 AM
> > To: general@hadoop.apache.org
> > Subject: Re: Hadoop/HDFS Thrift/PHP connection doesnt work
> >
> > I am suspecting that the thrift server is started using the configurati=
on
> in
> > your hadoop workspace rather than using the configuration of the HDFS
> > cluster that you must have setup earlier. Please check the CLASSPATH of
> the
> > thrit server....it should include the configuration of your HDFS cluste=
r.
> >
> > thanks,
> > dhruba
> >
> >
> > 2009/7/17 J=FCrgen Kaatz <kaatz@etracker.com>
> >
> > > Hi,
> > >
> > > I'm trying to use thrift with PHP to handle file system operations on
> HDFS
> > > without any success.
> > >
> > > I run a thrift server instance started with the script
> > > 'start_thrift_server.sh' is supposed located in
> > > 'hadoop/src/contrib/thriftfs/scripts/'. Then I use a little script
> > > 'hdfs.php' to connect to this thrift server and try to create a new
> file
> > > 'foo' on HDFS. But the file is created on my local filesystem not on
> HDFS.
> > >
> > > Can anybody help me?
> > > Thanks Juergen
> > >
> > > -- hdfs.php
> > >
> ------------------------------------------------------------------------
> > > <?php
> > > error_reporting(E_ALL);
> > > ini_set('display_errors', 'on');
> > >
> > > $GLOBALS['THRIFT_ROOT'] =3D '/home/hadoop/cc/thrift';
> > > define('ETCC_THRIFT_ROOT', $GLOBALS['THRIFT_ROOT']);
> > >
> > > require_once(ETCC_THRIFT_ROOT.'/Thrift.php' );
> > > require_once(ETCC_THRIFT_ROOT.'/transport/TSocket.php' );
> > > require_once(ETCC_THRIFT_ROOT.'/transport/TBufferedTransport.php' );
> > > require_once(ETCC_THRIFT_ROOT.'/protocol/TBinaryProtocol.php' );
> > >
> > > $socket =3D new TSocket('hadoop-master', 50021);
> > > $socket->setSendTimeout(10000); $socket->setRecvTimeout(20000);
> $transport =3D
> > > new TBufferedTransport($socket); $protocol =3D new
> > > TBinaryProtocol($transport);
> > >
> > >
> > >
> >
> require_once(ETCC_THRIFT_ROOT.'/packages/hadoopfs/ThriftHadoopFileSystem.=
php')
> > ;
> > > $client =3D new ThriftHadoopFileSystemClient($protocol);
> > >
> > > $transport->open();
> > >
> > > try
> > > {
> > >    $pathname =3D new Pathname(array('pathname' =3D> 'foo'));
> > >    // $result =3D $client->listStatus($pathname);
> > >    $result =3D $client->create($pathname);
> > >    $client->close($result);
> > >    print_r($result);
> > > }
> > > catch(Exception $e)
> > > {
> > >    print_r($e);
> > > }
> > >
> > > $transport->close();
> > >
> > >
>

--000e0cd340f0dfe5bc046f27154e--

From general-return-391-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 21 07:45:14 2009
Return-Path: <general-return-391-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98386 invoked from network); 21 Jul 2009 07:45:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Jul 2009 07:45:14 -0000
Received: (qmail 95106 invoked by uid 500); 21 Jul 2009 07:46:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95043 invoked by uid 500); 21 Jul 2009 07:46:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95023 invoked by uid 99); 21 Jul 2009 07:46:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 07:46:19 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.200.172] (HELO wf-out-1314.google.com) (209.85.200.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 07:46:07 +0000
Received: by wf-out-1314.google.com with SMTP id 23so836347wfg.2
        for <general@hadoop.apache.org>; Tue, 21 Jul 2009 00:45:45 -0700 (PDT)
Received: by 10.142.49.4 with SMTP id w4mr1119424wfw.343.1248162344929;
        Tue, 21 Jul 2009 00:45:44 -0700 (PDT)
Received: from ?192.168.42.100? (75-101-123-136.dsl.static.sonic.net [75.101.123.136])
        by mx.google.com with ESMTPS id 30sm15129395wfd.23.2009.07.21.00.45.43
        (version=SSLv3 cipher=RC4-MD5);
        Tue, 21 Jul 2009 00:45:44 -0700 (PDT)
Message-ID: <4A657222.6000606@cloudera.com>
Date: Tue, 21 Jul 2009 00:45:38 -0700
From: Amr Awadallah <aaa@cloudera.com>
Organization: Cloudera, Inc.
User-Agent: Thunderbird 2.0.0.22 (Windows/20090605)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: ANNOUNCE: ParBASH 0.1 release - Hadoop through BASH
References: <4A648FF2.3080306@gmail.com> <20090720165201.GF26881@odman.int.devc.at>
In-Reply-To: <20090720165201.GF26881@odman.int.devc.at>
Content-Type: multipart/alternative;
 boundary="------------020900030008000802050003"
X-Virus-Checked: Checked by ClamAV on apache.org

--------------020900030008000802050003
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

 >  How does this related to [bashreduce]

bashreduce has nothing to do with hadoop, it just implements a simple 
version of the mapreduce framework using bash. It also doesn't have an 
equivalent of HDFS (it relies on all the nodes having local copies or 
access to a shared fs, or pass data using nc).

ParBASH is a extension to bash which allows you to write a job as if it 
was a bash script, and parbash takes care of translating that to a 
hadoop streaming job (very clever)

-- amr

Robert Barta wrote:
> On Mon, Jul 20, 2009 at 05:40:34PM +0200, Milenko Petrovic wrote:
>   
>> Hello,
>>
>> I'd like to announce the release of the 0.1 version of ParBASH. Using  
>> ParBASH, it is possible to write bash scripts that are automatically  
>> translated into Hadoop Streaming Jobs.
>>     
>
> Milenko,
>
> How does this related to
>
>    http://md.devc.at/software/mapreduce/bashreduce
>
> ?
>
> \rho
>   

--------------020900030008000802050003--

From general-return-392-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 21 09:11:23 2009
Return-Path: <general-return-392-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24164 invoked from network); 21 Jul 2009 09:11:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Jul 2009 09:11:22 -0000
Received: (qmail 76335 invoked by uid 500); 21 Jul 2009 09:12:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76264 invoked by uid 500); 21 Jul 2009 09:12:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76254 invoked by uid 99); 21 Jul 2009 09:12:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 09:12:27 +0000
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=RCVD_IN_BL_SPAMCOP_NET,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of parbash.project@gmail.com designates 72.14.220.159 as permitted sender)
Received: from [72.14.220.159] (HELO fg-out-1718.google.com) (72.14.220.159)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 09:12:16 +0000
Received: by fg-out-1718.google.com with SMTP id 13so839377fge.12
        for <general@hadoop.apache.org>; Tue, 21 Jul 2009 02:11:55 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=EiZQ8bOCZLgePSBsJvJRb+d/4q0VbPzPSEnXpn2Jx6E=;
        b=Dv4/WdfyJO9u/eEUKbQMCraCj4qdeGlCoBG4Wsgp4B10ZYERl27hzbeu05GCm0SHxk
         bEUpVnVH+k8ImuxpjGW8Zy8aFTRmfBiEFgbdQrzURc2KafZnyQFbMbDBVqcV5ZYpEa7i
         H7HMgiQ1qcZBy4qprDgYPytov7ICdoLkD806k=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:content-type:content-transfer-encoding;
        b=Nwkrd7SuBr0bK953dKQvdxX99LGGBBBXqOwqSAFCVY4h0C5bq65MS8XjD47Gwd+/2W
         BDFmzdwa4q5Fu7pMxYwm41nj56Q03yZgPDjNGJ0j7EPq2hquZ2ZlBxOL2W3Tuxu9JAEG
         WTPIPOLrWEefFoFQJg5U0tC0/TKf++l9hwxnw=
Received: by 10.86.89.6 with SMTP id m6mr4404250fgb.1.1248167515792;
        Tue, 21 Jul 2009 02:11:55 -0700 (PDT)
Received: from W852120CRG3.pwlan-swisscom.mobile.ch ([194.209.131.192])
        by mx.google.com with ESMTPS id l19sm13718438fgb.26.2009.07.21.02.11.53
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Tue, 21 Jul 2009 02:11:54 -0700 (PDT)
Message-ID: <4A65865A.5090003@gmail.com>
Date: Tue, 21 Jul 2009 11:11:54 +0200
From: Milenko Petrovic <parbash.project@gmail.com>
User-Agent: Thunderbird 2.0.0.22 (Macintosh/20090605)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: ANNOUNCE: ParBASH 0.1 release - Hadoop through BASH
References: <4A648FF2.3080306@gmail.com> <20090720165201.GF26881@odman.int.devc.at> <4A657222.6000606@cloudera.com>
In-Reply-To: <4A657222.6000606@cloudera.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

That's right Amr. ParBASH is a modified bash interpreter which 
translates certain bash constructs into hadoop steaming jobs. With very 
small changes a parbash script can be made to run on hadoop or locally, 
which I find very handy while developing map-reduce jobs.

Frequently, I need to do data cleaning, extraction and transformation on 
my input files before running my map-reduce jobs. I find it convenient 
to use shell tools (grep, awk, perl, etc) for this kind of thing. 
ParBASH tries to imitate unix shell-way of processing files, so I think 
it is a good fit for any scripting-based map-reduce job.

-Milenko

Amr Awadallah wrote:
> >  How does this related to [bashreduce]
>
> bashreduce has nothing to do with hadoop, it just implements a simple 
> version of the mapreduce framework using bash. It also doesn't have an 
> equivalent of HDFS (it relies on all the nodes having local copies or 
> access to a shared fs, or pass data using nc).
>
> ParBASH is a extension to bash which allows you to write a job as if 
> it was a bash script, and parbash takes care of translating that to a 
> hadoop streaming job (very clever)
>
> -- amr
>
> Robert Barta wrote:
>> On Mon, Jul 20, 2009 at 05:40:34PM +0200, Milenko Petrovic wrote:
>>  
>>> Hello,
>>>
>>> I'd like to announce the release of the 0.1 version of ParBASH. 
>>> Using  ParBASH, it is possible to write bash scripts that are 
>>> automatically  translated into Hadoop Streaming Jobs.
>>>     
>>
>> Milenko,
>>
>> How does this related to
>>
>>    http://md.devc.at/software/mapreduce/bashreduce
>>
>> ?
>>
>> \rho
>>   
>

From general-return-393-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 21 12:22:43 2009
Return-Path: <general-return-393-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98658 invoked from network); 21 Jul 2009 12:22:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Jul 2009 12:22:42 -0000
Received: (qmail 98229 invoked by uid 500); 21 Jul 2009 12:23:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98151 invoked by uid 500); 21 Jul 2009 12:23:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98141 invoked by uid 99); 21 Jul 2009 12:23:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 12:23:46 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of anubhav.hanjura@convergys.com designates 167.1.158.29 as permitted sender)
Received: from [167.1.158.29] (HELO cvgmx2.convergys.com) (167.1.158.29)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 12:23:35 +0000
Received: from CDC5NW41.na.convergys.com (CDC5NW41.na.convergys.com [155.90.28.181])
	by cvgmx2.convergys.com (Postfix) with ESMTP id 7D885155D7D9
	for <general@hadoop.apache.org>; Tue, 21 Jul 2009 12:21:08 +0000 (UTC)
To: general@hadoop.apache.org
Subject: hadoop tutorial documentation
MIME-Version: 1.0
X-Mailer: Lotus Notes Release 6.5.3 September 14, 2004
Message-ID: <OF600D9D52.9BFEC117-ON652575FA.0043DE88-652575FA.00440A45@nd.convergys.com>
From: anubhav.hanjura@convergys.com
Date: Tue, 21 Jul 2009 17:53:10 +0530
X-MIMETrack: S/MIME Sign by Notes Client on Anubhav Hanjura/CIMG/CVG(Release 6.5.3|September
 14, 2004) at 07/21/2009 05:53:10 PM,
	Serialize by Notes Client on Anubhav Hanjura/CIMG/CVG(Release 6.5.3|September
 14, 2004) at 07/21/2009 05:53:10 PM,
	Serialize complete at 07/21/2009 05:53:10 PM,
	S/MIME Sign failed at 07/21/2009 05:53:10 PM: The cryptographic key was not
 found,
	Serialize by Router on CVGSMTP1/SRVR/CVG(Release 6.5.5FP3|March 22, 2007) at
 07/21/2009 08:23:46 AM,
	Serialize complete at 07/21/2009 08:23:46 AM
Content-Type: multipart/alternative; boundary="=_alternative 00440A42652575FA_="
X-Virus-Checked: Checked by ClamAV on apache.org

--=_alternative 00440A42652575FA_=
Content-Type: text/plain; charset="US-ASCII"

I have downloaded the hadoop tutorial but am unable to search a term in 
case I need to. Is it possible to have a search box at top right corner to 
search the hadoop tutorial for terms?

I hope it is not too much of an ask ;)

Thanks for your consideration

A.H
--=_alternative 00440A42652575FA_=--

From general-return-394-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 22 03:52:08 2009
Return-Path: <general-return-394-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54383 invoked from network); 22 Jul 2009 03:52:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jul 2009 03:52:08 -0000
Received: (qmail 39680 invoked by uid 500); 22 Jul 2009 03:53:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39609 invoked by uid 500); 22 Jul 2009 03:53:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39599 invoked by uid 99); 22 Jul 2009 03:53:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 03:53:12 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of nicc777@gmail.com designates 209.85.220.205 as permitted sender)
Received: from [209.85.220.205] (HELO mail-fx0-f205.google.com) (209.85.220.205)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 03:53:02 +0000
Received: by fxm1 with SMTP id 1so3064336fxm.29
        for <general@hadoop.apache.org>; Tue, 21 Jul 2009 20:52:40 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=/CTbNjhs4D3/pjz+ibESoa+mcDKQac9h5FmYPwP+f80=;
        b=alAEb6TkIhaSUzowqEJbJqoXutSAoWonQtZiKafYPd/M5UhJUAM5YhTTvC+M1+B8ot
         FETRK1IUZNmf8Nh+bE1nMGNaaN5Gdadvymknt+mu1T6OQ+jFqZGtRtArEJcI+g403Tt7
         EkaHyMbHS/U5DhHfbe8Iof+hj6ekBVQhw27uY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=FbaFc3PAQIIl70TA6VRKtETKhHG65dDgo5luV9z23HVMtbD8IbGoasugrhua7G5hCC
         R9ZcOhMbCn0ytm7dUZi2QzxRQjgYcAaM53dLZ2eelDtbEK46VIV6YE36qKzu1/XRsdTG
         e8oNlBV6m56HH4zyJpW4aVC02YtBTKWxapG7w=
MIME-Version: 1.0
Received: by 10.204.112.16 with SMTP id u16mr414583bkp.29.1248234760652; Tue, 
	21 Jul 2009 20:52:40 -0700 (PDT)
Date: Wed, 22 Jul 2009 05:52:40 +0200
Message-ID: <e21a770d0907212052hf5bc150l6dd8ff75627f772@mail.gmail.com>
Subject: HadoopDB
From: Nico Coetzee <nicc777@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502cfaf81464b046f434d8e
X-Virus-Checked: Checked by ClamAV on apache.org

--00504502cfaf81464b046f434d8e
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi - just saw this project mentioned on slashdot - what is the short version
of the long story? Anybody using it? Any real world advantages/benefits?
Should I consider using it?

Any comment welcome.

Thanks

Nico

Links:

* Slashdot: http://tech.slashdot.org/article.pl?sid=09/07/21/1747241
* Blog:
http://dbmsmusings.blogspot.com/2009/07/announcing-release-of-hadoopdb-longer.html

--00504502cfaf81464b046f434d8e--

From general-return-395-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 22 16:43:27 2009
Return-Path: <general-return-395-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33616 invoked from network); 22 Jul 2009 16:43:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jul 2009 16:43:27 -0000
Received: (qmail 8056 invoked by uid 500); 22 Jul 2009 16:44:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7976 invoked by uid 500); 22 Jul 2009 16:44:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7966 invoked by uid 99); 22 Jul 2009 16:44:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 16:44:31 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [85.183.249.146] (HELO mail.etracker.de) (85.183.249.146)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 16:44:22 +0000
Received: by mail.etracker.de (Postfix, from userid 1003)
	id CD2FE4D8BD8; Wed, 22 Jul 2009 18:43:58 +0200 (CEST)
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail.etracker.de
X-Spam-Level: 
Received: from mail.etracker.local (unknown [85.183.249.129])
	by mail.etracker.de (Postfix) with ESMTP id 8AFD64D8B5C
	for <general@hadoop.apache.org>; Wed, 22 Jul 2009 18:43:55 +0200 (CEST)
x-cr-puzzleid: {78D594D6-C572-4A51-8633-7D51742A1A71}
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Content-class: urn:content-classes:message
X-MimeOLE: Produced By Microsoft Exchange V6.5
Subject: HDFS Thrift method 'write' does not transfer bytes
Date: Wed, 22 Jul 2009 18:43:47 +0200
x-cr-hashedpuzzle: ALxp AXb7 Am3F Behv CUnU DdKz Dp4R D/0k EOZp ElVc Eq27 GWXX HKkM H3+g H40+ J2F8;1;ZwBlAG4AZQByAGEAbABAAGgAYQBkAG8AbwBwAC4AYQBwAGEAYwBoAGUALgBvAHIAZwA=;Sosha1_v1;7;{78D594D6-C572-4A51-8633-7D51742A1A71};awBhAGEAdAB6AEAAZQB0AHIAYQBjAGsAZQByAC4AYwBvAG0A;Wed, 22 Jul 2009 16:43:47 GMT;SABEAEYAUwAgAFQAaAByAGkAZgB0ACAAbQBlAHQAaABvAGQAIAAnAHcAcgBpAHQAZQAnACAAZABvAGUAcwAgAG4AbwB0ACAAdAByAGEAbgBzAGYAZQByACAAYgB5AHQAZQBzAA==
Message-ID: <E74E142F203F1A42A9288C280C60F8A5020AE636@ebase.etracker.local>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: HDFS Thrift method 'write' does not transfer bytes
Thread-Index: AcoK65VlAWGWozl/RzKsmUI6zvrhVg==
From: =?iso-8859-1?Q?J=FCrgen_Kaatz?= <kaatz@etracker.com>
To: <general@hadoop.apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org
X-Old-Spam-Status: No, score=-98.9 required=3.0 tests=BAYES_60,RDNS_NONE,
	USER_IN_WHITELIST autolearn=disabled version=3.2.5

Hi,

I try to copy a local 'gz' file via the HDFS Thrift method 'write' to =
HDFS. Unfortunately the method accepts only a 'String' as data and =
converts this to UTF8. I need t transfer bytes for the gz file.

Can anybody help me?

Juergen


From general-return-396-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 22 17:46:24 2009
Return-Path: <general-return-396-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 61815 invoked from network); 22 Jul 2009 17:46:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jul 2009 17:46:24 -0000
Received: (qmail 30202 invoked by uid 500); 22 Jul 2009 17:47:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30162 invoked by uid 500); 22 Jul 2009 17:47:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30152 invoked by uid 99); 22 Jul 2009 17:47:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 17:47:28 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [208.65.145.78] (HELO p02c12o145.mxlogic.net) (208.65.145.78)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 17:47:18 +0000
Received: from unknown [63.251.237.3] (EHLO mtiexch01.mti.com)
	by p02c12o145.mxlogic.net (mxl_mta-6.3.0-1)
	with ESMTP id f80576a4.0.112938.00-012.235112.p02c12o145.mxlogic.net (envelope-from <satish@mellanox.com>);
	Wed, 22 Jul 2009 11:46:57 -0600 (MDT)
X-MXL-Hash: 4a675091487ef7de-6bfc206df3fa7e2a74b0b50a6d9019ec61cc2368
Content-class: urn:content-classes:message
Subject: RE: Hadoop User Group (Bay Area) May 20th
MIME-Version: 1.0
Content-Type: text/plain;
	charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable
Date: Wed, 22 Jul 2009 10:46:54 -0700
X-MimeOLE: Produced By Microsoft Exchange V6.5
Message-ID: <9FA59C95FFCBB34EA5E42C1A8573784F01EA6508@mtiexch01.mti.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Hadoop User Group (Bay Area) May 20th
Thread-Index: Acnve0D0Y5l5TcTYR9SYn3oB+wIKEgbeRlOw
References: <AD0C16AAE446924586B68FC0EB421C9BAE155806@SNV-EXVS02.ds.corp.yahoo.com> <AD0C16AAE446924586B68FC0EB421C9BAE155864@SNV-EXVS02.ds.corp.yahoo.com> <DD27C9769EA63D43BE3CF4CCCC12DD970262B33A@SNV-EXVS02.ds.corp.yahoo.com> <9FA59C95FFCBB34EA5E42C1A8573784F01DDF27B@mtiexch01.mti.com> <94E98768-027B-464D-A738-5B74069D9291@apache.org>
From: "Satish Kikkeri" <Satish@Mellanox.com>
To: <general@hadoop.apache.org>
Cc: "Dekel Tankel" <dekel@yahoo-inc.com>
X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2009071501)]
X-MAIL-FROM: <satish@mellanox.com>
X-SOURCE-IP: [63.251.237.3]
X-AnalysisOut: [v=1.0 c=1 a=4iyT6hJci8cA:10 a=xupnbh4h0YLOHZnncC45HQ==:17 ]
X-AnalysisOut: [a=CbDCq_QkAAAA:8 a=mV9VRH-2AAAA:8 a=LbXv4wAxBhbhiPxzX6UA:9]
X-AnalysisOut: [ a=0I8il_TIzbEhA2Cv264EMV6NRxUA:4 a=6bbpDATN3xAA:10 a=gQ22]
X-AnalysisOut: [TatrZ4oA:10 a=mqKVMgZcPAkA:10 a=88iI8knYSJUA:10]
X-Virus-Checked: Checked by ClamAV on apache.org

Do we have a meeting today or pushing it to next month

________________________________________________________________________
_______________

=20

Satish Kikkeri

Mellanox Technologies

350 Oakmead Parkway, Sunnyvale, CA 94085

408-916-0045 (Direct)

408-807-9903 (Mobile)

www.mellanox.com

=20

=20

-----Original Message-----
From: Owen O'Malley [mailto:omalley@apache.org]=20
Sent: Wednesday, June 17, 2009 11:39 AM
To: general@hadoop.apache.org
Cc: Dekel Tankel
Subject: Re: Hadoop User Group (Bay Area) May 20th


On Jun 17, 2009, at 11:11 AM, Satish Kikkeri wrote:

> Is the next Hadoop UG meeting today (6/18) or next week (6/25)

We just had the Hadoop Summit last week, so let's skip this month. I =20
expect that the Hadoop User Group meetings will resume next month.

-- Owen

From general-return-397-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 22 18:01:45 2009
Return-Path: <general-return-397-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70593 invoked from network); 22 Jul 2009 18:01:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jul 2009 18:01:45 -0000
Received: (qmail 66335 invoked by uid 500); 22 Jul 2009 18:02:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66291 invoked by uid 500); 22 Jul 2009 18:02:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66281 invoked by uid 99); 22 Jul 2009 18:02:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 18:02:49 +0000
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.30.80] (HELO QMTA08.emeryville.ca.mail.comcast.net) (76.96.30.80)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 18:02:38 +0000
Received: from OMTA21.emeryville.ca.mail.comcast.net ([76.96.30.88])
	by QMTA08.emeryville.ca.mail.comcast.net with comcast
	id K4xZ1c0091u4NiLA862Kgq; Wed, 22 Jul 2009 18:02:19 +0000
Received: from wifi-e-135-237.corp.yahoo.com ([209.131.62.145])
	by OMTA21.emeryville.ca.mail.comcast.net with comcast
	id K6271c00K381T1d8h62AZh; Wed, 22 Jul 2009 18:02:17 +0000
Cc: "Dekel Tankel" <dekel@yahoo-inc.com>
Message-Id: <12905811-764E-4746-97AA-76EBFE4A13E4@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <9FA59C95FFCBB34EA5E42C1A8573784F01EA6508@mtiexch01.mti.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v935.3)
Subject: Re: Hadoop User Group (Bay Area) May 20th
Date: Wed, 22 Jul 2009 11:02:05 -0700
References: <AD0C16AAE446924586B68FC0EB421C9BAE155806@SNV-EXVS02.ds.corp.yahoo.com> <AD0C16AAE446924586B68FC0EB421C9BAE155864@SNV-EXVS02.ds.corp.yahoo.com> <DD27C9769EA63D43BE3CF4CCCC12DD970262B33A@SNV-EXVS02.ds.corp.yahoo.com> <9FA59C95FFCBB34EA5E42C1A8573784F01DDF27B@mtiexch01.mti.com> <94E98768-027B-464D-A738-5B74069D9291@apache.org> <9FA59C95FFCBB34EA5E42C1A8573784F01EA6508@mtiexch01.mti.com>
X-Mailer: Apple Mail (2.935.3)
X-Virus-Checked: Checked by ClamAV on apache.org


On Jul 22, 2009, at 10:46 AM, Satish Kikkeri wrote:

> Do we have a meeting today or pushing it to next month

The next meeting is Aug 19th. Please RSVP at:

http://www.meetup.com/hadoop/calendar/10877366/?action=detail&eventId=10877366

Thanks,
    Owen

From general-return-398-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 15:43:13 2009
Return-Path: <general-return-398-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88171 invoked from network); 23 Jul 2009 15:43:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 15:43:13 -0000
Received: (qmail 65258 invoked by uid 500); 23 Jul 2009 15:44:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65180 invoked by uid 500); 23 Jul 2009 15:44:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65169 invoked by uid 99); 23 Jul 2009 15:44:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 15:44:16 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bernd.fondermann@googlemail.com designates 74.125.78.27 as permitted sender)
Received: from [74.125.78.27] (HELO ey-out-2122.google.com) (74.125.78.27)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 15:44:06 +0000
Received: by ey-out-2122.google.com with SMTP id 22so290409eye.35
        for <general@hadoop.apache.org>; Thu, 23 Jul 2009 08:43:45 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type:content-transfer-encoding;
        bh=5pLnMXDZ088slJ9XrDBTnoYfGLKenoeMen2i6to0dh8=;
        b=Ziu6mbBBaUJ9BU5cVfyLjN90nKAHms5zLHLfwSaMPrSLf+YW8LsC+h8eYP2tbD3ZSL
         oyKmH/g03uoFhTKX6oeTouYqymFoqIicyTeeawEkYK9OP6EipQonGJ+apd41zesoX1UQ
         JD75NKhsMfxcGKgT8kknafS9DFkquZe9Vl6O0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=xWc/euv3cntr/z/9n0Uvr2gur3mcgF98SkjSOpUWXLRk5V/FrX0uuzKr1rg2oSVEjN
         atzqCaHoUj3PZp05NxFxrVsaJxeQnl5UgdaxLGb3xK6NSPfqg3CWxID2JvmHZgxbVFgi
         dlAfJidjK5BKpa+63OgPhIG/xjxxjg1JfayXI=
MIME-Version: 1.0
Received: by 10.216.3.70 with SMTP id 48mr676081weg.74.1248363824894; Thu, 23 
	Jul 2009 08:43:44 -0700 (PDT)
Date: Thu, 23 Jul 2009 17:43:44 +0200
Message-ID: <88f6e29a0907230843k7441c13cn46fb73a7739c361c@mail.gmail.com>
Subject: More info on the "project split"
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

I found this interesting posting:
  http://www.cloudera.com/blog/2009/07/17/the-project-split/
assumedly talking about Apache Hadoop, but I might be wrong on this.

I tried to find more on the project's lists on this topic, but failed.

Anybody can point me to the relevant discussion or proposals?

Thanks,

 Bernd

From general-return-399-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 22:18:14 2009
Return-Path: <general-return-399-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 8335 invoked from network); 23 Jul 2009 22:18:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 22:18:14 -0000
Received: (qmail 17138 invoked by uid 500); 23 Jul 2009 22:19:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17068 invoked by uid 500); 23 Jul 2009 22:19:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17058 invoked by uid 99); 23 Jul 2009 22:19:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 22:19:18 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 22:19:05 +0000
Received: from SNV-EXBH01.ds.corp.yahoo.com (snv-exbh01.ds.corp.yahoo.com [207.126.227.249])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n6NMH2kL069247
	for <general@hadoop.apache.org>; Thu, 23 Jul 2009 15:17:03 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:message-id:
	thread-topic:thread-index:mime-version:content-type:
	content-transfer-encoding:x-originalarrivaltime;
	b=Hbm5dFvyX6L9mq4Nh8iDwLVzrx9QKvXbzUsSkXxq0SFSmziBlXVtmZXOfttK6NNO
Received: from SNV-EXVS09.ds.corp.yahoo.com ([207.126.227.87]) by SNV-EXBH01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Thu, 23 Jul 2009 15:17:02 -0700
Received: from 10.73.146.106 ([10.73.146.106]) by SNV-EXVS09.ds.corp.yahoo.com ([207.126.227.84]) via Exchange Front-End Server snv-webmail.corp.yahoo.com ([207.126.227.59]) with Microsoft Exchange Server HTTP-DAV ;
 Thu, 23 Jul 2009 22:16:47 +0000
User-Agent: Microsoft-Entourage/12.19.0.090515
Date: Thu, 23 Jul 2009 15:16:47 -0700
Subject: Edit link on jira.
From: Mahadev Konar <mahadev@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Message-ID: <C68E2F5F.21065%mahadev@yahoo-inc.com>
Thread-Topic: Edit link on jira.
Thread-Index: AcoL40SbVvjD8x06U0qo/W3bKQF+3g==
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-OriginalArrivalTime: 23 Jul 2009 22:17:02.0516 (UTC) FILETIME=[4DDAFF40:01CA0BE3]
X-Virus-Checked: Checked by ClamAV on apache.org

HI all,
 Looks like we got rid of the edit link on jira. I think it was quite useful
for minor spelling mistake changes like

IF instead of IS..... (now I have to repost the whole comment making
spelling  changes all over the comment)

I am sure there must have been a valid reason for removing it but would it
be possible to reinstate it and use it for editing comments for minor
spelling  changes (some ethics law)?


Thanks
mahadev


From general-return-400-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 23:49:53 2009
Return-Path: <general-return-400-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31917 invoked from network); 23 Jul 2009 23:49:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 23:49:53 -0000
Received: (qmail 98449 invoked by uid 500); 23 Jul 2009 23:50:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98374 invoked by uid 500); 23 Jul 2009 23:50:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98364 invoked by uid 99); 23 Jul 2009 23:50:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 23:50:57 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 23:50:45 +0000
Received: from coatsatfind-lm.corp.yahoo.com (coatsatfind-lm.corp.yahoo.com [10.72.187.241])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n6NNnM70042469
	for <general@hadoop.apache.org>; Thu, 23 Jul 2009 16:49:23 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:mime-version:
	subject:date:references:x-mailer;
	b=o4qoGhHBE6b52tAqv3+0AbDrzlkjdT27kxDKi3/ISy35rhxjja7zefj8hti5AZkT
Message-Id: <3E38BFE2-45B0-4AA6-94B4-E4D305096925@yahoo-inc.com>
From: Hong Tang <htang@yahoo-inc.com>
To: <general@hadoop.apache.org>
In-Reply-To: <C68E2F5F.21065%mahadev@yahoo-inc.com>
Content-Type: multipart/alternative; boundary=Apple-Mail-29--497177324
Mime-Version: 1.0 (Apple Message framework v935.3)
Subject: Re: Edit link on jira.
Date: Thu, 23 Jul 2009 16:49:22 -0700
References: <C68E2F5F.21065%mahadev@yahoo-inc.com>
X-Mailer: Apple Mail (2.935.3)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-29--497177324
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit

+1

On Jul 23, 2009, at 3:16 PM, Mahadev Konar wrote:

> HI all,
>  Looks like we got rid of the edit link on jira. I think it was  
> quite useful
> for minor spelling mistake changes like
>
> IF instead of IS..... (now I have to repost the whole comment making
> spelling  changes all over the comment)
>
> I am sure there must have been a valid reason for removing it but  
> would it
> be possible to reinstate it and use it for editing comments for minor
> spelling  changes (some ethics law)?
>
>
> Thanks
> mahadev
>


--Apple-Mail-29--497177324--

From general-return-401-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 24 06:27:32 2009
Return-Path: <general-return-401-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73952 invoked from network); 24 Jul 2009 06:27:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jul 2009 06:27:29 -0000
Received: (qmail 44246 invoked by uid 500); 24 Jul 2009 06:28:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44152 invoked by uid 500); 24 Jul 2009 06:28:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44142 invoked by uid 99); 24 Jul 2009 06:28:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 06:28:33 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of nicc777@gmail.com designates 209.85.220.205 as permitted sender)
Received: from [209.85.220.205] (HELO mail-fx0-f205.google.com) (209.85.220.205)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 06:28:23 +0000
Received: by fxm1 with SMTP id 1so1280528fxm.29
        for <general@hadoop.apache.org>; Thu, 23 Jul 2009 23:28:02 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=MkUP/Qpcx1RRft0G3wi/8jh3+HIS1R+39kkrTQi5is4=;
        b=tXbVe7pUNpugc0akSqUmRy7OorlzFQDLA+N5iGcwnGCQB7zIAUNbojnJrUw2+ppBoF
         UclpznpaaKjb+z8fnJaFFMhcxaLq7KYDVOxaGbnFTpWzKn14yHLpzQSUV/Xbzoo5i1M4
         uuWRspqmKPPEXA3rgj7R21l3YFAFX8kkniZ6k=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=Du5IRrAYz/n1oQWch6iyGCg2jGh3MTBZhplUilHPsEOxjLZ7A403pqVIdKW9OmdtQ4
         +ApZIZjnMf83/mYVIqJ+ikPfU5X0XmtVLlGLTLwnPOaFHWfd8w3vghBNTO3CckfVsuYr
         X61zJ75ImX0fMx+JXnkeQmuFaXsyZnlU12XSQ=
MIME-Version: 1.0
Received: by 10.204.53.143 with SMTP id m15mr2882251bkg.112.1248416881834; 
	Thu, 23 Jul 2009 23:28:01 -0700 (PDT)
Date: Fri, 24 Jul 2009 08:28:01 +0200
Message-ID: <e21a770d0907232328t4b0439c5s6b749a33998cb389@mail.gmail.com>
Subject: DEPRECATED: hadoop-site.xml found in the classpath.
From: Nico Coetzee <nicc777@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c5ba67c5fccc046f6db4d6
X-Virus-Checked: Checked by ClamAV on apache.org

--001636c5ba67c5fccc046f6db4d6
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi - I keep on getting this message and although everything works I suppose
I have to fix this at some point.

The full error:

09/07/24 07:12:57 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found
in the classpath. Usage of hadoop-site.xml is deprecated. Instead use
core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of
core-default.xml, mapred-default.xml and hdfs-default.xml respectively


My current hadoop-site.xml on my "master" node looks like this (in the
<configuration> section):

<property>
  <name>hadoop.tmp.dir</name>
  <value>/tmp1</value>
</property>

<property>
  <name>fs.default.name</name>
  <value>hdfs://node1:54310</value>
</property>

<property>
  <name>mapred.job.tracker</name>
  <value>node1:54311</value>
</property>

<property>
  <name>dfs.replication</name>
  <value>3</value>
</property>

I think the following:

* hadoop.tmp.dir goes into core-site.xml
* fs.default.name goes into core-site.xml
* mapred.job.tracker goes into mapred-site.xml
* dfs.replication goes into hdfs-site.xml

Is this right? I also assume I can then remove hadoop-site.xml?

Thanks

Nico

--001636c5ba67c5fccc046f6db4d6--

From general-return-402-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 24 08:57:22 2009
Return-Path: <general-return-402-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12452 invoked from network); 24 Jul 2009 08:57:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jul 2009 08:57:22 -0000
Received: (qmail 16165 invoked by uid 500); 24 Jul 2009 08:58:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16099 invoked by uid 500); 24 Jul 2009 08:58:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16089 invoked by uid 99); 24 Jul 2009 08:58:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 08:58:26 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 08:58:13 +0000
Received: from SNV-EXBH01.ds.corp.yahoo.com (snv-exbh01.ds.corp.yahoo.com [207.126.227.249])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n6O8vHxr093865
	for <general@hadoop.apache.org>; Fri, 24 Jul 2009 01:57:17 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:x-mimeole:content-class:mime-version:
	content-type:subject:date:message-id:x-ms-has-attach:
	x-ms-tnef-correlator:thread-topic:thread-index:from:to:x-originalarrivaltime;
	b=AQxA0o/GV8+hgfusvyNEuSEe4v3USVgjMphevKGWcb6MqlrDje72vZCLVKNXwOAs
Received: from SNV-EXVS01.ds.corp.yahoo.com ([216.145.51.186]) by SNV-EXBH01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Fri, 24 Jul 2009 01:57:17 -0700
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01CA0C3C.A085802C"
Subject: Re: DEPRECATED: hadoop-site.xml found in the classpath.
Date: Fri, 24 Jul 2009 01:56:26 -0700
Message-ID: <2C52DBBEC4855C438BB330CB0D3B465903BFADA5@SNV-EXVS01.ds.corp.yahoo.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: DEPRECATED: hadoop-site.xml found in the classpath.
Thread-Index: AcoMPKB0beVRT4BQT7qQFf/vNwzpVQ==
From: "Sharad Agarwal" <sharadag@yahoo-inc.com>
To: <general@hadoop.apache.org>
X-OriginalArrivalTime: 24 Jul 2009 08:57:17.0345 (UTC) FILETIME=[BEE0B910:01CA0C3C]
X-Virus-Checked: Checked by ClamAV on apache.org

------_=_NextPart_001_01CA0C3C.A085802C
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


Nico Coetzee wrote:
> I think the following:
>
> * hadoop.tmp.dir goes into core-site.xml
> * fs.default.name goes into core-site.xml
> * mapred.job.tracker goes into mapred-site.xml
> * dfs.replication goes into hdfs-site.xml
>
> Is this right? I also assume I can then remove hadoop-site.xml?
>

=20
Right.

------_=_NextPart_001_01CA0C3C.A085802C--

From general-return-403-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 24 10:58:26 2009
Return-Path: <general-return-403-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85654 invoked from network); 24 Jul 2009 10:58:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jul 2009 10:58:26 -0000
Received: (qmail 53814 invoked by uid 500); 24 Jul 2009 10:59:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53734 invoked by uid 500); 24 Jul 2009 10:59:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53724 invoked by uid 99); 24 Jul 2009 10:59:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 10:59:30 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of pareekash@gmail.com designates 209.85.216.171 as permitted sender)
Received: from [209.85.216.171] (HELO mail-px0-f171.google.com) (209.85.216.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 10:59:22 +0000
Received: by pxi1 with SMTP id 1so1154844pxi.5
        for <general@hadoop.apache.org>; Fri, 24 Jul 2009 03:59:02 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=UoGINhEqn4zeuRHaZbLjSGiUMOZW/0ovF2J4KKy7D4E=;
        b=dO81hyYqVG3fRbEesFU4yeJcZBaBXrIPHeAhMw7Vq7BhQNz5nKomDSyJoa+FrYB3SJ
         qDOSSHPXPU57Cimx2vYjkmBTRvpYclNtGzZU2g/4OAvpOuS1GprLmU4BX9zNfKjRCxb3
         XavxQUIoa4t5KLgsMFpRZsZZNq+ztoky8dZe8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=VNRajnFid0OjU2E/dD7wz+8h/t3OEyjUTUUaHSSxltXzen8ehH2lVqevQkG/+XFGKG
         xRZ2vDKDH+mMOHbi6UMG0um0RzVss/r/AID68CcXT/t1Dykiz60JEFaWL6ZgHoLijmY2
         IRCIwuGOqVolEiVIWSm0TgaYT3SC87TKJJvsQ=
MIME-Version: 1.0
Received: by 10.114.192.14 with SMTP id p14mr4225768waf.136.1248433142470; 
	Fri, 24 Jul 2009 03:59:02 -0700 (PDT)
In-Reply-To: <45d9159d0907240350l5dcd2920k9b3ecc022781a95e@mail.gmail.com>
References: <45d9159d0907240350l5dcd2920k9b3ecc022781a95e@mail.gmail.com>
Date: Fri, 24 Jul 2009 16:29:02 +0530
Message-ID: <45d9159d0907240359o7d24c140p85c4b728e1d01b8f@mail.gmail.com>
Subject: How Does Hadoop stores Meta Data information
From: ashish pareek <pareekash@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163646c6f8fb9954046f717d3a
X-Virus-Checked: Checked by ClamAV on apache.org

--00163646c6f8fb9954046f717d3a
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi Everybody,

                  From last few weeks I am trying to find out what data
structure does Hadoop uses to store files and directories Meta-data
information. If any one can point out the exact source file where this
source code is, it will help me a lot.

Thank you in advance.

Regards,
Ashish.

--00163646c6f8fb9954046f717d3a--

From general-return-404-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 24 16:48:34 2009
Return-Path: <general-return-404-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17960 invoked from network); 24 Jul 2009 16:48:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jul 2009 16:48:34 -0000
Received: (qmail 29293 invoked by uid 500); 24 Jul 2009 16:49:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29224 invoked by uid 500); 24 Jul 2009 16:49:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29213 invoked by uid 99); 24 Jul 2009 16:49:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 16:49:39 +0000
X-ASF-Spam-Status: No, hits=4.2 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 16:49:27 +0000
Received: from SP1-EX07CAS02.ds.corp.yahoo.com (sp1-ex07cas02.ds.corp.yahoo.com [216.252.116.138])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n6OGlI0I072272
	for <general@hadoop.apache.org>; Fri, 24 Jul 2009 09:47:18 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:in-reply-to:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=sA8kGCLx+7FHG48syGyLAeN1uureTsJOjNwbWp2Xd9tXUPCSD0rv0xd2hGB7Qo37
Received: from SP1-EX07VS02.ds.corp.yahoo.com ([216.252.116.135]) by
 SP1-EX07CAS02.ds.corp.yahoo.com ([216.252.116.167]) with mapi; Fri, 24 Jul
 2009 09:47:18 -0700
From: Jerome Boulon <jboulon@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 24 Jul 2009 09:47:17 -0700
Subject: Re: Edit link on jira.
Thread-Topic: Edit link on jira.
Thread-Index: AcoL8JKL1IMRqAyVT5+t0a2S5g5HRAAjdSi9
Message-ID: <C68F33A5.1F0D6%jboulon@yahoo-inc.com>
In-Reply-To: <3E38BFE2-45B0-4AA6-94B4-E4D305096925@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C68F33A51F0D6jboulonyahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C68F33A51F0D6jboulonyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

+1


On 7/23/09 4:49 PM, "Hong Tang" <htang@yahoo-inc.com> wrote:

+1

On Jul 23, 2009, at 3:16 PM, Mahadev Konar wrote:

> HI all,
>  Looks like we got rid of the edit link on jira. I think it was
> quite useful
> for minor spelling mistake changes like
>
> IF instead of IS..... (now I have to repost the whole comment making
> spelling  changes all over the comment)
>
> I am sure there must have been a valid reason for removing it but
> would it
> be possible to reinstate it and use it for editing comments for minor
> spelling  changes (some ethics law)?
>
>
> Thanks
> mahadev
>



--_000_C68F33A51F0D6jboulonyahooinccom_--

From general-return-405-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 24 17:29:05 2009
Return-Path: <general-return-405-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33186 invoked from network); 24 Jul 2009 17:29:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jul 2009 17:29:05 -0000
Received: (qmail 84046 invoked by uid 500); 24 Jul 2009 17:30:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83996 invoked by uid 500); 24 Jul 2009 17:30:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83986 invoked by uid 99); 24 Jul 2009 17:30:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 17:30:09 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 17:29:57 +0000
Received: from [192.168.1.71] (snvvpn1-10-73-152-c245.hq.corp.yahoo.com [10.73.152.245])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n6OHSie0089752
	for <general@hadoop.apache.org>; Fri, 24 Jul 2009 10:28:45 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:mime-version:
	subject:date:references:x-mailer;
	b=1ScSemCfquVzMo9hLego9au8FfCkQGmioNtWhaUFRsZSTSbmcDKgaN+KVscaPMFk
Message-Id: <4443CF77-8311-4C2B-AB35-C2C2383570A3@yahoo-inc.com>
From: Sanjay Radia <sradia@yahoo-inc.com>
To: <general@hadoop.apache.org>
In-Reply-To: <C68E2F5F.21065%mahadev@yahoo-inc.com>
Content-Type: multipart/alternative; boundary=Apple-Mail-7--433614046
Mime-Version: 1.0 (Apple Message framework v935.3)
Subject: Re: Edit link on jira.
Date: Fri, 24 Jul 2009 10:28:45 -0700
References: <C68E2F5F.21065%mahadev@yahoo-inc.com>
X-Mailer: Apple Mail (2.935.3)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-7--433614046
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


On Jul 23, 2009, at 3:16 PM, Mahadev Konar wrote:

> HI all,
>  Looks like we got rid of the edit link on jira. I think it was  
> quite useful
> for minor spelling mistake changes like
>
> IF instead of IS..... (now I have to repost the whole comment making
> spelling  changes all over the comment)
>
> I am sure there must have been a valid reason for removing it but  
> would it
> be possible to reinstate it and use it for editing comments for minor
> spelling  changes (some ethics law)?
>
>
> Thanks
> mahadev
>

+1.
--Apple-Mail-7--433614046--

From general-return-406-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 24 17:51:39 2009
Return-Path: <general-return-406-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43861 invoked from network); 24 Jul 2009 17:51:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jul 2009 17:51:39 -0000
Received: (qmail 8106 invoked by uid 500); 24 Jul 2009 17:52:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8019 invoked by uid 500); 24 Jul 2009 17:52:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8003 invoked by uid 99); 24 Jul 2009 17:52:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 17:52:43 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [207.126.228.149] (HELO rsmtp1.corp.yahoo.com) (207.126.228.149)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 17:52:34 +0000
Received: from [10.73.152.79] (snvvpn1-10-73-152-c79.hq.corp.yahoo.com [10.73.152.79])
	(authenticated bits=0)
	by rsmtp1.corp.yahoo.com (8.13.8/8.13.8/y.rout) with ESMTP id n6OHq9Oc051152
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Fri, 24 Jul 2009 10:52:10 -0700 (PDT)
Message-ID: <4A69F4C8.9060301@apache.org>
Date: Fri, 24 Jul 2009 10:52:08 -0700
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.22 (X11/20090608)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Edit link on jira.
References: <C68E2F5F.21065%mahadev@yahoo-inc.com> <4443CF77-8311-4C2B-AB35-C2C2383570A3@yahoo-inc.com>
In-Reply-To: <4443CF77-8311-4C2B-AB35-C2C2383570A3@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Sanjay Radia wrote:
> 
> On Jul 23, 2009, at 3:16 PM, Mahadev Konar wrote:
> 
>> HI all,
>>  Looks like we got rid of the edit link on jira. I think it was quite 
>> useful
>> for minor spelling mistake changes like
>>
>> IF instead of IS..... (now I have to repost the whole comment making
>> spelling  changes all over the comment)
>>
>> I am sure there must have been a valid reason for removing it but 
>> would it
>> be possible to reinstate it and use it for editing comments for minor
>> spelling  changes (some ethics law)?
>>
>>
>> Thanks
>> mahadev
>>
> 
> +1.

From general-return-407-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 24 18:05:31 2009
Return-Path: <general-return-407-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 51286 invoked from network); 24 Jul 2009 18:05:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jul 2009 18:05:31 -0000
Received: (qmail 26472 invoked by uid 500); 24 Jul 2009 18:06:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26383 invoked by uid 500); 24 Jul 2009 18:06:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26373 invoked by uid 99); 24 Jul 2009 18:06:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 18:06:35 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 18:06:22 +0000
Received: from SNV-EXBH01.ds.corp.yahoo.com (snv-exbh01.ds.corp.yahoo.com [207.126.227.249])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n6OI5Od7003190
	for <general@hadoop.apache.org>; Fri, 24 Jul 2009 11:05:25 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:x-originalarrivaltime;
	b=UZvPC/dE0fuXNGYa1HeIMLRaSkb7mke1I7iwEIueZiSZZIFWx5Qr3cewGaTNJ9Jx
Received: from SNV-EXVS05.ds.corp.yahoo.com ([207.126.227.225]) by SNV-EXBH01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Fri, 24 Jul 2009 11:05:24 -0700
Received: from 10.72.244.136 ([10.72.244.136]) by SNV-EXVS05.ds.corp.yahoo.com ([207.126.227.45]) via Exchange Front-End Server snv-webmail.corp.yahoo.com ([207.126.227.59]) with Microsoft Exchange Server HTTP-DAV ;
 Fri, 24 Jul 2009 18:05:24 +0000
User-Agent: Microsoft-Entourage/12.19.0.090515
Date: Fri, 24 Jul 2009 11:05:20 -0700
Subject: Re: Edit link on jira.
From: Yiping Han <yhan@yahoo-inc.com>
To: <general@hadoop.apache.org>
Message-ID: <C68F45F0.FC17%yhan@yahoo-inc.com>
Thread-Topic: Edit link on jira.
Thread-Index: AcoL8JKL1IMRqAyVT5+t0a2S5g5HRAAjdSi9AAK50/M=
In-Reply-To: <C68F33A5.1F0D6%jboulon@yahoo-inc.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-OriginalArrivalTime: 24 Jul 2009 18:05:24.0750 (UTC) FILETIME=[514C7AE0:01CA0C89]
X-Virus-Checked: Checked by ClamAV on apache.org

+1


On 7/24/09 9:47 AM, "Jerome Boulon" <jboulon@yahoo-inc.com> wrote:

> +1
> 
> 
> On 7/23/09 4:49 PM, "Hong Tang" <htang@yahoo-inc.com> wrote:
> 
> +1
> 
> On Jul 23, 2009, at 3:16 PM, Mahadev Konar wrote:
> 
>> HI all,
>>  Looks like we got rid of the edit link on jira. I think it was
>> quite useful
>> for minor spelling mistake changes like
>> 
>> IF instead of IS..... (now I have to repost the whole comment making
>> spelling  changes all over the comment)
>> 
>> I am sure there must have been a valid reason for removing it but
>> would it
>> be possible to reinstate it and use it for editing comments for minor
>> spelling  changes (some ethics law)?
>> 
>> 
>> Thanks
>> mahadev
>> 
> 
> 

----------------------
Yiping Han
F-3140 
(408)349-4403
yhan@yahoo-inc.com


From general-return-408-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 24 23:31:35 2009
Return-Path: <general-return-408-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 61056 invoked from network); 24 Jul 2009 23:31:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jul 2009 23:31:35 -0000
Received: (qmail 97997 invoked by uid 500); 24 Jul 2009 23:32:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97920 invoked by uid 500); 24 Jul 2009 23:32:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97910 invoked by uid 99); 24 Jul 2009 23:32:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 23:32:39 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.198.239] (HELO rv-out-0506.google.com) (209.85.198.239)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 23:32:30 +0000
Received: by rv-out-0506.google.com with SMTP id k40so543266rvb.29
        for <general@hadoop.apache.org>; Fri, 24 Jul 2009 16:32:09 -0700 (PDT)
Received: by 10.141.2.18 with SMTP id e18mr2722077rvi.195.1248478329623;
        Fri, 24 Jul 2009 16:32:09 -0700 (PDT)
Received: from ?192.168.42.100? (75-101-123-136.dsl.static.sonic.net [75.101.123.136])
        by mx.google.com with ESMTPS id f21sm7123028rvb.6.2009.07.24.16.32.07
        (version=SSLv3 cipher=RC4-MD5);
        Fri, 24 Jul 2009 16:32:08 -0700 (PDT)
Message-ID: <4A6A4463.60002@cloudera.com>
Date: Fri, 24 Jul 2009 16:31:47 -0700
From: Amr Awadallah <aaa@cloudera.com>
Organization: Cloudera, Inc.
User-Agent: Thunderbird 2.0.0.22 (Windows/20090605)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Edit link on jira.
References: <C68F45F0.FC17%yhan@yahoo-inc.com>
In-Reply-To: <C68F45F0.FC17%yhan@yahoo-inc.com>
Content-Type: multipart/alternative;
 boundary="------------070905030000070203090509"
X-Virus-Checked: Checked by ClamAV on apache.org

--------------070905030000070203090509
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

-1

Comments on JIRA get sent out by email (and it is the email that we 
actually read), we shouldn't let people go back and edit comments in the 
same sense that it is not possible to go back and change an email once 
it is sent (editing comments also leads to many emails sent out further 
clogging our inboxes).

Before we submit a comment, we should read it twice and correct the 
typos, then post it.

Yiping Han wrote:
> +1
>
>
> On 7/24/09 9:47 AM, "Jerome Boulon" <jboulon@yahoo-inc.com> wrote:
>
>   
>> +1
>>
>>
>> On 7/23/09 4:49 PM, "Hong Tang" <htang@yahoo-inc.com> wrote:
>>
>> +1
>>
>> On Jul 23, 2009, at 3:16 PM, Mahadev Konar wrote:
>>
>>     
>>> HI all,
>>>  Looks like we got rid of the edit link on jira. I think it was
>>> quite useful
>>> for minor spelling mistake changes like
>>>
>>> IF instead of IS..... (now I have to repost the whole comment making
>>> spelling  changes all over the comment)
>>>
>>> I am sure there must have been a valid reason for removing it but
>>> would it
>>> be possible to reinstate it and use it for editing comments for minor
>>> spelling  changes (some ethics law)?
>>>
>>>
>>> Thanks
>>> mahadev
>>>
>>>       
>>     
>
> ----------------------
> Yiping Han
> F-3140 
> (408)349-4403
> yhan@yahoo-inc.com
>
>   

--------------070905030000070203090509--

From general-return-409-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 24 23:38:38 2009
Return-Path: <general-return-409-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 62052 invoked from network); 24 Jul 2009 23:38:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jul 2009 23:38:38 -0000
Received: (qmail 2254 invoked by uid 500); 24 Jul 2009 23:39:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2162 invoked by uid 500); 24 Jul 2009 23:39:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2151 invoked by uid 99); 24 Jul 2009 23:39:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 23:39:42 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 23:39:30 +0000
Received: from [10.0.1.6] (UNKNOWN-10-72-168-153.yahoo.com [10.72.168.153] (may be forged))
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n6ONc5bM045587
	for <general@hadoop.apache.org>; Fri, 24 Jul 2009 16:38:05 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=AQA3YgCfTZjTAOZ335JA2bA9WF85Mz9nt+GMrIE8eYM3C/HTMkkybt0a/K002noY
Message-Id: <3B65EFEA-2355-4EC6-B5A7-4CC390B3852D@yahoo-inc.com>
From: Nigel Daley <ndaley@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <4A6A4463.60002@cloudera.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v935.3)
Subject: Re: Edit link on jira.
Date: Fri, 24 Jul 2009 16:38:05 -0700
References: <C68F45F0.FC17%yhan@yahoo-inc.com> <4A6A4463.60002@cloudera.com>
X-Mailer: Apple Mail (2.935.3)
X-Virus-Checked: Checked by ClamAV on apache.org

I agree with Amr.

-1 on editing comments.

n.

On Jul 24, 2009, at 4:31 PM, Amr Awadallah wrote:

> -1
>
> Comments on JIRA get sent out by email (and it is the email that we  
> actually read), we shouldn't let people go back and edit comments in  
> the same sense that it is not possible to go back and change an  
> email once it is sent (editing comments also leads to many emails  
> sent out further clogging our inboxes).
>
> Before we submit a comment, we should read it twice and correct the  
> typos, then post it.
>
> Yiping Han wrote:
>> +1
>>
>>
>> On 7/24/09 9:47 AM, "Jerome Boulon" <jboulon@yahoo-inc.com> wrote:
>>
>>
>>> +1
>>>
>>>
>>> On 7/23/09 4:49 PM, "Hong Tang" <htang@yahoo-inc.com> wrote:
>>>
>>> +1
>>>
>>> On Jul 23, 2009, at 3:16 PM, Mahadev Konar wrote:
>>>
>>>
>>>> HI all,
>>>> Looks like we got rid of the edit link on jira. I think it was
>>>> quite useful
>>>> for minor spelling mistake changes like
>>>>
>>>> IF instead of IS..... (now I have to repost the whole comment  
>>>> making
>>>> spelling  changes all over the comment)
>>>>
>>>> I am sure there must have been a valid reason for removing it but
>>>> would it
>>>> be possible to reinstate it and use it for editing comments for  
>>>> minor
>>>> spelling  changes (some ethics law)?
>>>>
>>>>
>>>> Thanks
>>>> mahadev
>>>>
>>>>
>>>
>>
>> ----------------------
>> Yiping Han
>> F-3140 (408)349-4403
>> yhan@yahoo-inc.com
>>
>>


From general-return-410-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 25 00:47:12 2009
Return-Path: <general-return-410-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85659 invoked from network); 25 Jul 2009 00:47:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Jul 2009 00:47:12 -0000
Received: (qmail 46687 invoked by uid 500); 25 Jul 2009 00:48:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46614 invoked by uid 500); 25 Jul 2009 00:48:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46600 invoked by uid 99); 25 Jul 2009 00:48:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jul 2009 00:48:16 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [207.126.228.150] (HELO rsmtp2.corp.yahoo.com) (207.126.228.150)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jul 2009 00:48:07 +0000
Received: from [10.72.244.181] (snvvpn1-10-72-244-c181.hq.corp.yahoo.com [10.72.244.181])
	(authenticated bits=0)
	by rsmtp2.corp.yahoo.com (8.13.8/8.13.8/y.rout) with ESMTP id n6P0lgN3067307
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Fri, 24 Jul 2009 17:47:43 -0700 (PDT)
Message-ID: <4A6A562E.9030404@apache.org>
Date: Fri, 24 Jul 2009 17:47:42 -0700
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.22 (X11/20090608)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Edit link on jira.
References: <C68F45F0.FC17%yhan@yahoo-inc.com> <4A6A4463.60002@cloudera.com>
In-Reply-To: <4A6A4463.60002@cloudera.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

As I understand it until recently we were able to edit comments, for 
some reason it was changed. People found editing comments useful in 
certain circumstances.

1) has editing of comments been a problem for the hadoop community -- 
were people misusing this?

2) email IS sent on every JIRA comment, including edits to comments. The 
community can therefore review any edit, if there's an issue (misuse) it 
can be rectified.

Given 1 & 2 I don't see why we should make a change to disallow comment 
editing at this time.

Patrick

Amr Awadallah wrote:
> -1
> 
> Comments on JIRA get sent out by email (and it is the email that we 
> actually read), we shouldn't let people go back and edit comments in the 
> same sense that it is not possible to go back and change an email once 
> it is sent (editing comments also leads to many emails sent out further 
> clogging our inboxes).
> 
> Before we submit a comment, we should read it twice and correct the 
> typos, then post it.
> 
> Yiping Han wrote:
>> +1
>>
>>
>> On 7/24/09 9:47 AM, "Jerome Boulon" <jboulon@yahoo-inc.com> wrote:
>>
>>  
>>> +1
>>>
>>>
>>> On 7/23/09 4:49 PM, "Hong Tang" <htang@yahoo-inc.com> wrote:
>>>
>>> +1
>>>
>>> On Jul 23, 2009, at 3:16 PM, Mahadev Konar wrote:
>>>
>>>    
>>>> HI all,
>>>>  Looks like we got rid of the edit link on jira. I think it was
>>>> quite useful
>>>> for minor spelling mistake changes like
>>>>
>>>> IF instead of IS..... (now I have to repost the whole comment making
>>>> spelling  changes all over the comment)
>>>>
>>>> I am sure there must have been a valid reason for removing it but
>>>> would it
>>>> be possible to reinstate it and use it for editing comments for minor
>>>> spelling  changes (some ethics law)?
>>>>
>>>>
>>>> Thanks
>>>> mahadev
>>>>
>>>>       
>>>     
>>
>> ----------------------
>> Yiping Han
>> F-3140 (408)349-4403
>> yhan@yahoo-inc.com
>>
>>   
> 

From general-return-411-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 25 03:34:52 2009
Return-Path: <general-return-411-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30805 invoked from network); 25 Jul 2009 03:34:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Jul 2009 03:34:52 -0000
Received: (qmail 29446 invoked by uid 500); 25 Jul 2009 03:35:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29335 invoked by uid 500); 25 Jul 2009 03:35:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29325 invoked by uid 99); 25 Jul 2009 03:35:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jul 2009 03:35:57 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ctubbsii@gmail.com designates 209.85.221.188 as permitted sender)
Received: from [209.85.221.188] (HELO mail-qy0-f188.google.com) (209.85.221.188)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jul 2009 03:35:47 +0000
Received: by qyk26 with SMTP id 26so2705065qyk.5
        for <general@hadoop.apache.org>; Fri, 24 Jul 2009 20:35:26 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :x-enigmail-version:content-type:content-transfer-encoding;
        bh=LKBsXscXUZrzwhdc4BkXJPmI/oH5SZ4g66rzONGoK/I=;
        b=JtRrFsBLMkESFcQy6BJ+zl8K7A2SRbEq8gucnvOvIjHI1re39Duu1uII3HTDeNfX0S
         0FADIaZFUQrIYqvfSo9LjgQ/Spl0rSUUDJWRWmuCFEm6/1S1ioASqmZNhsjlkX+3BnaX
         zM2SEVrozt8LH7bR440sHFjVLsVqHAhIDB/rA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:x-enigmail-version:content-type
         :content-transfer-encoding;
        b=fLSriZq2sG9mHae0ZCatHvYm80Z1lm7eOdC1ZtaEGKYaWC4o5Qpbmjd0e57Dfx4wfl
         e5kUtpa7houBVNq+Ccl4oOAo2ivgMkuMDMywkgBTCWhlozY/6sqlexdkoj/KycpbR8fa
         hhN4jA0F9hs/0mjJc26midsDhgVlMBhWq30fM=
Received: by 10.224.2.69 with SMTP id 5mr4033786qai.367.1248492926871;
        Fri, 24 Jul 2009 20:35:26 -0700 (PDT)
Received: from ?192.168.90.4? (pool-71-246-212-121.washdc.fios.verizon.net [71.246.212.121])
        by mx.google.com with ESMTPS id 6sm6098205qwd.42.2009.07.24.20.35.26
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Fri, 24 Jul 2009 20:35:26 -0700 (PDT)
Message-ID: <4A6A7D7C.8030202@gmail.com>
Date: Fri, 24 Jul 2009 23:35:24 -0400
From: Christopher L Tubbs II <ctubbsii@gmail.com>
User-Agent: Thunderbird 2.0.0.21 (X11/20090320)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: mapreduce classpath issue
References: <2C52DBBEC4855C438BB330CB0D3B465903BFADA5@SNV-EXVS01.ds.corp.yahoo.com>
In-Reply-To: <2C52DBBEC4855C438BB330CB0D3B465903BFADA5@SNV-EXVS01.ds.corp.yahoo.com>
X-Enigmail-Version: 0.96.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

So, I've been having an issue running on Mac OS, with classpath for
MapReduce jobs on version 0.20.0

The basic idea is that I want to add a jar as a dependency to a
MapReduce job. I know I can add the jar to HADOOP_CLASSPATH, but I do
not have access to hadoop-env.xml (administrator restricted).

I have been using the "hadoop jar" command to execute a class with a
main() contained within my jar. My class extends Configured and
implements Tool. The main() calls ToolRunner.run() to launch the job,
and I specify -libjars on the command line, which I see from the source,
uses the GenericOptionsParser to strip the "-libjars x.jar,y.jar,z.jar"
from the arguments passed to my class. I then get errors that I can't
find classes contained within my library jars.

>From within run(), my debugging code demonstrates that the jar files are
readable, and I can getResource() to verify that the classes are
available in those jars. However, I still get errors, because it doesn't
seem that at any point, the JVM is using the contextClassLoader that the
resources were added with.

Perhaps this is a bug with Mac OS's JVM 1.6, or am I doing something
wrong? Everything works fine in linux with the identical setup.


-
Chris

From general-return-412-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 25 03:36:36 2009
Return-Path: <general-return-412-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30919 invoked from network); 25 Jul 2009 03:36:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Jul 2009 03:36:36 -0000
Received: (qmail 30219 invoked by uid 500); 25 Jul 2009 03:37:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30133 invoked by uid 500); 25 Jul 2009 03:37:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30121 invoked by uid 99); 25 Jul 2009 03:37:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jul 2009 03:37:39 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ctubbsii@gmail.com designates 209.85.221.188 as permitted sender)
Received: from [209.85.221.188] (HELO mail-qy0-f188.google.com) (209.85.221.188)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jul 2009 03:37:30 +0000
Received: by qyk26 with SMTP id 26so2705622qyk.5
        for <general@hadoop.apache.org>; Fri, 24 Jul 2009 20:37:09 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :x-enigmail-version:content-type:content-transfer-encoding;
        bh=/64nImPkfDIFwwubVYdZ8hU2jQVY5hDUWvjssQh5Nhg=;
        b=jEqsSTCKEQSYQDSz9FHKll7mw9wEJXVD2RN5mKK37q3vCQQLC3Z0kHgib9M/l3In7K
         V/irGqKsuGM426qy1cphbeDEb/Rh2NQIYLS5nGnG6NXB72Ue060rcYU47ZomkUlICpHN
         8G0zE8a9IIRYc55U18BHFv+xqiYCWGz9cUy7o=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:x-enigmail-version:content-type
         :content-transfer-encoding;
        b=LmNQ8CnTCeCKSODfK9+9beYOWBbo5eLQAbeicJtiCshhoYaFqIoNCIK3WK300114lx
         81AFkVhNuktQ5EolLKMOJvfWlHIFVYtjwJaGK1uraJnFthpu2UQxkS0Iy2ESCMd6C1xs
         SGJxDL3Bm0DEYVO/8xQ7nzNoMF3FhY9E1Ix0g=
Received: by 10.224.53.196 with SMTP id n4mr4135706qag.43.1248493029446;
        Fri, 24 Jul 2009 20:37:09 -0700 (PDT)
Received: from ?192.168.90.4? (pool-71-246-212-121.washdc.fios.verizon.net [71.246.212.121])
        by mx.google.com with ESMTPS id 6sm24016qwk.34.2009.07.24.20.37.08
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Fri, 24 Jul 2009 20:37:08 -0700 (PDT)
Message-ID: <4A6A7DE3.7020405@gmail.com>
Date: Fri, 24 Jul 2009 23:37:07 -0400
From: Christopher L Tubbs II <ctubbsii@gmail.com>
User-Agent: Thunderbird 2.0.0.21 (X11/20090320)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: DEPRECATED: hadoop-site.xml found in the classpath.
References: <2C52DBBEC4855C438BB330CB0D3B465903BFADA5@SNV-EXVS01.ds.corp.yahoo.com>
In-Reply-To: <2C52DBBEC4855C438BB330CB0D3B465903BFADA5@SNV-EXVS01.ds.corp.yahoo.com>
X-Enigmail-Version: 0.96.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-Virus-Checked: Checked by ClamAV on apache.org

I had similar questions. Where can I find documentation that shows which
0.19.x hadoop-site.xml options go in which of these 3 files, and at what
point are these files accessed (i.e. why are there now 3?). What
happened to the example/default config files for these three?

Sharad Agarwal wrote:
> Nico Coetzee wrote:
>> I think the following:
>>
>> * hadoop.tmp.dir goes into core-site.xml
>> * fs.default.name goes into core-site.xml
>> * mapred.job.tracker goes into mapred-site.xml
>> * dfs.replication goes into hdfs-site.xml
>>
>> Is this right? I also assume I can then remove hadoop-site.xml?
>>
> 
>  
> Right.
> 

From general-return-413-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 25 12:52:18 2009
Return-Path: <general-return-413-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56441 invoked from network); 25 Jul 2009 12:52:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Jul 2009 12:52:18 -0000
Received: (qmail 26018 invoked by uid 500); 25 Jul 2009 12:53:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25942 invoked by uid 500); 25 Jul 2009 12:53:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 61811 invoked by uid 99); 24 Jul 2009 22:42:44 -0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ryanobjc@gmail.com designates 209.85.217.218 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type:content-transfer-encoding;
        bh=dmcA3hgTM6u7ljFiTJAk9GHWt5DGA6jwLgIzgvvg5eY=;
        b=ryZLJRVdR8oBIR7K3sgAM6itOpIh+YcpcT3Puvv3wKnrx4eLpoID6eYn+R5owSS1zZ
         ozj8PttHkinkWLbm1/hlCgEv/wcjuIumO25QEAUK+TIqDiOTdwDkTsC1MFdsoBjhODxD
         IK2Cpu2zxM2/E1Q1QYqO9ecOFWtIcOJPyWO9k=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=tAyVUwzTdI8Z/d5I3fDwFzE09hoZJ+bFIY/w371lsE+e2Ddk0j/QB0m4EMcT6OiugT
         hA/3A/ZP75ghT0zdVWejfU9xrcR2AudYrfTN75SdvU775jpZxN/kDiOSHhV4qC/qLYtP
         oz8w5LYkfrMTQP3Sif5Lys1nt2woNS9+B5z8c=
MIME-Version: 1.0
Date: Fri, 24 Jul 2009 15:42:13 -0700
Message-ID: <78568af10907241542t454a95fdv5abacce56905568@mail.gmail.com>
Subject: August 7-9, San Francisco: Announcing HBase User Group meeting & 
	Hackathon
From: Ryan Rawson <ryanobjc@gmail.com>
To: general@hadoop.apache.org, hbase-dev@hadoop.apache.org, 
	hbase-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Announcing the next HBase User Group meeting, being held on Friday
August the 7th at 4pm.  The location is Stumbleupon's office at 140
Second Street, 3rd Floor, San Francisco.  Come talk and learn about
HBase 0.20.  RSVP at
http://www.meetup.com/hbaseusergroup/calendar/10950511/

Following the user group meeting on Saturday & Sunday (August 8th and
9th) is the HBase 0.21 hackathon! Come talk features, design and code
for the next major version of HBase! RSVP at
http://www.meetup.com/hackathon/calendar/10951718/

Regards,
-ryan

From general-return-414-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jul 26 00:06:59 2009
Return-Path: <general-return-414-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30997 invoked from network); 26 Jul 2009 00:06:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Jul 2009 00:06:59 -0000
Received: (qmail 86920 invoked by uid 500); 26 Jul 2009 00:08:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86856 invoked by uid 500); 26 Jul 2009 00:08:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86846 invoked by uid 99); 26 Jul 2009 00:08:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jul 2009 00:08:03 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of nicc777@gmail.com designates 209.85.218.221 as permitted sender)
Received: from [209.85.218.221] (HELO mail-bw0-f221.google.com) (209.85.218.221)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jul 2009 00:07:54 +0000
Received: by bwz21 with SMTP id 21so2063800bwz.29
        for <general@hadoop.apache.org>; Sat, 25 Jul 2009 17:07:32 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=CZZI2kFi7UIDljg2+OhURrwCyuBDHpkivm3VCOl7M5A=;
        b=D+Ud6TbAd/zHoE82ccayo3KqGZEZFgJz4kFi/1Jn4njMXBzziXz8QT5IBELYdDWiBE
         cgHq5j/dwKx9MlKCM8zFf+pA79wzmcmDBQbtBiFnHrPjvCZJSfHKjeEgHFTxXGX2n5MD
         bAOy0bRDu53skgKgSLBxM06OmyFsdgED3Tqak=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=jPjZgu3RViOD4eikocrbW2amfu6ypr19vam5Tlgq3YADI4nVWCGZKLgOg4euuebPQn
         rxrcp4qXGOKLYlx6LjvxuMb6zaKd/ekS44nNh8Y7kWK1XuMWegAYH1PFJZCeDf975AUk
         9n0OZyZ47wUvmyoF2FxUAiNRxoopPUbiEdUXw=
MIME-Version: 1.0
Received: by 10.204.65.16 with SMTP id g16mr1417894bki.37.1248566852667; Sat, 
	25 Jul 2009 17:07:32 -0700 (PDT)
In-Reply-To: <4A6A7DE3.7020405@gmail.com>
References: <2C52DBBEC4855C438BB330CB0D3B465903BFADA5@SNV-EXVS01.ds.corp.yahoo.com>
	 <4A6A7DE3.7020405@gmail.com>
Date: Sun, 26 Jul 2009 02:07:32 +0200
Message-ID: <e21a770d0907251707o12d04d14kf8a4e890787742ba@mail.gmail.com>
Subject: Re: DEPRECATED: hadoop-site.xml found in the classpath.
From: Nico Coetzee <nicc777@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6ddfff6bb4a80046f909f19
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6ddfff6bb4a80046f909f19
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

I basically looked at the following docs:

* http://hadoop.apache.org/common/docs/current/core-default.html
* http://hadoop.apache.org/common/docs/current/hdfs-default.html
* http://hadoop.apache.org/common/docs/current/mapred-default.html

I searched for the relevant parameter names in each.

Hope that helps.

BTW - I do think we need some better docs in explaining how the ocnfig files
work together, sequence of events etc.

Cheers

Nico

On Sat, Jul 25, 2009 at 5:37 AM, Christopher L Tubbs II
<ctubbsii@gmail.com>wrote:

> I had similar questions. Where can I find documentation that shows which
> 0.19.x hadoop-site.xml options go in which of these 3 files, and at what
> point are these files accessed (i.e. why are there now 3?). What
> happened to the example/default config files for these three?
>
> Sharad Agarwal wrote:
> > Nico Coetzee wrote:
> >> I think the following:
> >>
> >> * hadoop.tmp.dir goes into core-site.xml
> >> * fs.default.name goes into core-site.xml
> >> * mapred.job.tracker goes into mapred-site.xml
> >> * dfs.replication goes into hdfs-site.xml
> >>
> >> Is this right? I also assume I can then remove hadoop-site.xml?
> >>
> >
> >
> > Right.
> >
>

--0016e6ddfff6bb4a80046f909f19--

From general-return-415-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 14:52:11 2009
Return-Path: <general-return-415-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56646 invoked from network); 27 Jul 2009 14:52:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 14:52:11 -0000
Received: (qmail 87321 invoked by uid 500); 27 Jul 2009 14:53:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87260 invoked by uid 500); 27 Jul 2009 14:53:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87250 invoked by uid 99); 27 Jul 2009 14:53:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 14:53:15 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 209.85.210.185 as permitted sender)
Received: from [209.85.210.185] (HELO mail-yx0-f185.google.com) (209.85.210.185)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 14:53:07 +0000
Received: by yxe15 with SMTP id 15so5640865yxe.5
        for <general@hadoop.apache.org>; Mon, 27 Jul 2009 07:52:46 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=2C2Jc8f4o5t3OHFTTHBTp8eYJkoLP+bpHTsed0gk+JY=;
        b=HX9qkPEMc/WNsADG4XnQa96bdEFv2j/ilKXjkaRy1THl0VlpFqUqk3vTrRZR9gbI73
         FyvA2sskLq5VVUc+8/RRGgmWTtsd7NDN1q8rwq1hcArkyBMSkyl3Nlfs3MoO/R87efmB
         uYkevnyA0e96Y3xZl2uqCpJISUKN/M05FuRbY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=STk5diO8zmfZAus2hs381TlCdo+0NeLGxvJ18J9Wmx22YP8P4agzNC59z1eTEDiv41
         lfXoQVJKdxpkXQYX4shl+G91y7idKW8B+f9IdEXSwZMD1T1f4t5X80kjNmwMLBTUDkU8
         WvqcrMPuAm8s4LYZ8po0aOCEXQlgev+i9Ns1I=
MIME-Version: 1.0
Received: by 10.151.157.15 with SMTP id j15mr11272163ybo.329.1248706366440; 
	Mon, 27 Jul 2009 07:52:46 -0700 (PDT)
In-Reply-To: <E74E142F203F1A42A9288C280C60F8A5020AE636@ebase.etracker.local>
References: <E74E142F203F1A42A9288C280C60F8A5020AE636@ebase.etracker.local>
Date: Mon, 27 Jul 2009 07:52:46 -0700
Message-ID: <4aa34eb70907270752i4942bfxb97031c93571cc2d@mail.gmail.com>
Subject: Re: HDFS Thrift method 'write' does not transfer bytes
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015174ff5e86684b6046fb11b83
X-Virus-Checked: Checked by ClamAV on apache.org

--0015174ff5e86684b6046fb11b83
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I believe you are right. It might help to check out the Thrift documentatio=
n
at http://wiki.apache.org/thrift/ and then enhance the hadoop.thriftfs API
to store/retrieve bytes.

thanks
dhruba


2009/7/22 J=FCrgen Kaatz <kaatz@etracker.com>

> Hi,
>
> I try to copy a local 'gz' file via the HDFS Thrift method 'write' to HDF=
S.
> Unfortunately the method accepts only a 'String' as data and converts thi=
s
> to UTF8. I need t transfer bytes for the gz file.
>
> Can anybody help me?
>
> Juergen
>
>

--0015174ff5e86684b6046fb11b83--

From general-return-416-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 18:01:55 2009
Return-Path: <general-return-416-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56669 invoked from network); 27 Jul 2009 18:01:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 18:01:55 -0000
Received: (qmail 50928 invoked by uid 500); 27 Jul 2009 18:02:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50847 invoked by uid 500); 27 Jul 2009 18:02:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50837 invoked by uid 99); 27 Jul 2009 18:02:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 18:02:59 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cutting@gmail.com designates 209.85.218.221 as permitted sender)
Received: from [209.85.218.221] (HELO mail-bw0-f221.google.com) (209.85.218.221)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 18:02:47 +0000
Received: by bwz21 with SMTP id 21so2833576bwz.29
        for <general@hadoop.apache.org>; Mon, 27 Jul 2009 11:02:26 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:sender:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=2ypFKg7Y+r/1XbI7YIv72895ryN2nPcztwES7qorsqE=;
        b=YUKK5ygKXpxTYPA334oi4VWfIDPeJpaWjgWe0DTTAsD/AWxqLc6aOGAW4ODfdA6F+J
         G0YyxT+e/0zg9eRcFpunzbDSYswtzQLECQS2eYvbrdofvARbYRqKJwCZ5ivqVjLBOd+s
         KfQ8X9PXF/NEg7kIgRF5kGZ4M653+l4/zcJpk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:date:from:user-agent:mime-version:to:subject
         :references:in-reply-to:content-type:content-transfer-encoding;
        b=aH4kvK2E7nDS83yAJSHzfOSUsJ09dkEEvPQGKtyQmW0wmAwrjYKloGiFcrn7W/L2Op
         jEphfTrZ4fU/vW/0nJ58IY7pLJnkPxFg5vUBZU/O1ysQLoVKr37tV7ws0W4Bh4yNpLk0
         LMm3ScmUQIJz2Vt3KVi5/ahZK0Vfzlh78kLzo=
Received: by 10.103.182.1 with SMTP id j1mr3373406mup.119.1248717746533;
        Mon, 27 Jul 2009 11:02:26 -0700 (PDT)
Received: from ?192.168.168.16? (c-76-103-183-11.hsd1.ca.comcast.net [76.103.183.11])
        by mx.google.com with ESMTPS id b9sm28773904mug.39.2009.07.27.11.02.23
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Mon, 27 Jul 2009 11:02:25 -0700 (PDT)
Sender: Doug Cutting <cutting@gmail.com>
Message-ID: <4A6DEBAB.1010205@apache.org>
Date: Mon, 27 Jul 2009 11:02:19 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.22 (X11/20090608)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Edit link on jira.
References: <C68F45F0.FC17%yhan@yahoo-inc.com> <4A6A4463.60002@cloudera.com> <4A6A562E.9030404@apache.org>
In-Reply-To: <4A6A562E.9030404@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Patrick Hunt wrote:
> As I understand it until recently we were able to edit comments, for 
> some reason it was changed. 

This change was presented here:

http://www.mail-archive.com/core-dev@hadoop.apache.org/msg42904.html

> 1) has editing of comments been a problem for the hadoop community -- 
> were people misusing this?

For folks who follow issues by email, a small change requires that one 
re-read the entire edited comment, since Jira's diffs are useless. 
Folks were frequently editing long comments making it very difficult to 
determine what change had been made.  It's better to make a new comment 
amending one's prior comment.

I have long argued against editing Jira comments.

http://wiki.apache.org/hadoop/HowToContribute

See the "Jira Guidelines" near the end of the page.  This was added in 2007.

Doug

From general-return-417-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 18:29:31 2009
Return-Path: <general-return-417-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76833 invoked from network); 27 Jul 2009 18:29:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 18:29:31 -0000
Received: (qmail 80328 invoked by uid 500); 27 Jul 2009 18:30:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80253 invoked by uid 500); 27 Jul 2009 18:30:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80239 invoked by uid 99); 27 Jul 2009 18:30:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 18:30:35 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [207.126.228.150] (HELO rsmtp2.corp.yahoo.com) (207.126.228.150)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 18:30:24 +0000
Received: from [10.72.244.16] (snvvpn1-10-72-244-c16.hq.corp.yahoo.com [10.72.244.16])
	(authenticated bits=0)
	by rsmtp2.corp.yahoo.com (8.13.8/8.13.8/y.rout) with ESMTP id n6RITqrl003295
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Mon, 27 Jul 2009 11:29:53 -0700 (PDT)
Message-ID: <4A6DF221.5030408@apache.org>
Date: Mon, 27 Jul 2009 11:29:53 -0700
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.22 (X11/20090608)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Edit link on jira.
References: <C68F45F0.FC17%yhan@yahoo-inc.com> <4A6A4463.60002@cloudera.com> <4A6A562E.9030404@apache.org> <4A6DEBAB.1010205@apache.org>
In-Reply-To: <4A6DEBAB.1010205@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Perhaps I misunderstood the scope of this discussion - is it general 
Hadoop or specific to some subset of Hadoop projects? Regardless, at 
this time the ZooKeeper community would prefer to continue supporting 
edits on comments.

Patrick

Doug Cutting wrote:
> Patrick Hunt wrote:
>> As I understand it until recently we were able to edit comments, for 
>> some reason it was changed. 
> 
> This change was presented here:
> 
> http://www.mail-archive.com/core-dev@hadoop.apache.org/msg42904.html
> 
>> 1) has editing of comments been a problem for the hadoop community -- 
>> were people misusing this?
> 
> For folks who follow issues by email, a small change requires that one 
> re-read the entire edited comment, since Jira's diffs are useless. Folks 
> were frequently editing long comments making it very difficult to 
> determine what change had been made.  It's better to make a new comment 
> amending one's prior comment.
> 
> I have long argued against editing Jira comments.
> 
> http://wiki.apache.org/hadoop/HowToContribute
> 
> See the "Jira Guidelines" near the end of the page.  This was added in 
> 2007.
> 
> Doug

From general-return-418-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 18:47:54 2009
Return-Path: <general-return-418-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85656 invoked from network); 27 Jul 2009 18:47:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 18:47:53 -0000
Received: (qmail 8206 invoked by uid 500); 27 Jul 2009 18:48:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8129 invoked by uid 500); 27 Jul 2009 18:48:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8075 invoked by uid 99); 27 Jul 2009 18:48:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 18:48:50 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cutting@gmail.com designates 209.85.218.221 as permitted sender)
Received: from [209.85.218.221] (HELO mail-bw0-f221.google.com) (209.85.218.221)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 18:48:40 +0000
Received: by bwz21 with SMTP id 21so2863769bwz.29
        for <general@hadoop.apache.org>; Mon, 27 Jul 2009 11:48:19 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:sender:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=j4p9Ne4AHfPEjsoU5Wk6h9ON0MbtnWinAoHoTm97Xs8=;
        b=suuFqqhfatgxNVkL39jdiBhJ+t5VZ6HeaPU1vRvS8kU3okOjTbVElaj2Wm8kXb6dHy
         LfHjdr1z6tSIDE6DTVTIcl6WbnD9eAh85ZF7OmdqSlw3tSOWlOommKqH9HAgztXi3Oz8
         JwvJ+S8ct30st8bJ38hyyJALrHpT7uMO0Y5oA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:date:from:user-agent:mime-version:to:subject
         :references:in-reply-to:content-type:content-transfer-encoding;
        b=mZ7MSRgeK3S9UQh/PIhi9Q2WzD+WF9hu7yUK/+3SgHmyf20W2CoKwNVe24XtEa9JBd
         7qIr/QNoKVcKkk2qjTGc2GDvBfPO1mEffr/nMJsuUvufCpJ8RRNRIcvf13xVbWpCs3xq
         r7hW87Fqb4r9q9QHkl5zkZM7XI/4W6DBXMY6o=
Received: by 10.103.131.8 with SMTP id i8mr3481047mun.124.1248720499219;
        Mon, 27 Jul 2009 11:48:19 -0700 (PDT)
Received: from ?192.168.168.16? (c-76-103-183-11.hsd1.ca.comcast.net [76.103.183.11])
        by mx.google.com with ESMTPS id u26sm29562397mug.22.2009.07.27.11.48.16
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Mon, 27 Jul 2009 11:48:18 -0700 (PDT)
Sender: Doug Cutting <cutting@gmail.com>
Message-ID: <4A6DF66D.1020402@apache.org>
Date: Mon, 27 Jul 2009 11:48:13 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.22 (X11/20090608)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Edit link on jira.
References: <C68F45F0.FC17%yhan@yahoo-inc.com> <4A6A4463.60002@cloudera.com> <4A6A562E.9030404@apache.org> <4A6DEBAB.1010205@apache.org> <4A6DF221.5030408@apache.org>
In-Reply-To: <4A6DF221.5030408@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Patrick Hunt wrote:
> Perhaps I misunderstood the scope of this discussion - is it general 
> Hadoop or specific to some subset of Hadoop projects? Regardless, at 
> this time the ZooKeeper community would prefer to continue supporting 
> edits on comments.

All Hadoop sub-projects currently share a single Jira permission scheme.

Jira permission schemes are only alterable by Jira-wide administrators I 
think, and each TLP has only a few of those.  Owen & I are I believe the 
only Jira-wide admins for the Hadoop TLP.  So it might be impractical to 
use a custom permission scheme per subproject.

If the ZooKeeper community wishes, I suppose we might switch ZooKeeper's 
Jira to the Apache standard permission scheme, which does permit comment 
editing.

Doug

From general-return-419-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 19:00:59 2009
Return-Path: <general-return-419-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 90051 invoked from network); 27 Jul 2009 19:00:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 19:00:59 -0000
Received: (qmail 28269 invoked by uid 500); 27 Jul 2009 19:02:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28201 invoked by uid 500); 27 Jul 2009 19:02:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28187 invoked by uid 99); 27 Jul 2009 19:02:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 19:02:04 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [207.126.228.150] (HELO rsmtp2.corp.yahoo.com) (207.126.228.150)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 19:01:56 +0000
Received: from [10.72.244.16] (snvvpn1-10-72-244-c16.hq.corp.yahoo.com [10.72.244.16])
	(authenticated bits=0)
	by rsmtp2.corp.yahoo.com (8.13.8/8.13.8/y.rout) with ESMTP id n6RJ1UBX005709
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Mon, 27 Jul 2009 12:01:30 -0700 (PDT)
Message-ID: <4A6DF98A.1060400@apache.org>
Date: Mon, 27 Jul 2009 12:01:30 -0700
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.22 (X11/20090608)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Edit link on jira.
References: <C68F45F0.FC17%yhan@yahoo-inc.com> <4A6A4463.60002@cloudera.com> <4A6A562E.9030404@apache.org> <4A6DEBAB.1010205@apache.org> <4A6DF221.5030408@apache.org> <4A6DF66D.1020402@apache.org>
In-Reply-To: <4A6DF66D.1020402@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hrm, thanks for the feedback. I'll investigate the scheme options with 
Owen/yourself offline.

Patrick

Doug Cutting wrote:
> Patrick Hunt wrote:
>> Perhaps I misunderstood the scope of this discussion - is it general 
>> Hadoop or specific to some subset of Hadoop projects? Regardless, at 
>> this time the ZooKeeper community would prefer to continue supporting 
>> edits on comments.
> 
> All Hadoop sub-projects currently share a single Jira permission scheme.
> 
> Jira permission schemes are only alterable by Jira-wide administrators I 
> think, and each TLP has only a few of those.  Owen & I are I believe the 
> only Jira-wide admins for the Hadoop TLP.  So it might be impractical to 
> use a custom permission scheme per subproject.
> 
> If the ZooKeeper community wishes, I suppose we might switch ZooKeeper's 
> Jira to the Apache standard permission scheme, which does permit comment 
> editing.
> 
> Doug

From general-return-420-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 00:27:40 2009
Return-Path: <general-return-420-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53515 invoked from network); 31 Jul 2009 00:27:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 00:27:39 -0000
Received: (qmail 9047 invoked by uid 500); 31 Jul 2009 00:27:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8562 invoked by uid 500); 31 Jul 2009 00:27:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8284 invoked by uid 99); 31 Jul 2009 00:27:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 00:27:36 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.211.200] (HELO mail-yw0-f200.google.com) (209.85.211.200)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 00:27:24 +0000
Received: by ywh38 with SMTP id 38so1320091ywh.20
        for <multiple recipients>; Thu, 30 Jul 2009 17:27:02 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.90.87.5 with SMTP id k5mr1374846agb.46.1249000022184; Thu, 30 
	Jul 2009 17:27:02 -0700 (PDT)
From: Christophe Bisciglia <christophe@cloudera.com>
Date: Thu, 30 Jul 2009 17:26:42 -0700
Message-ID: <69035570907301726t51d298fcjff62162cd309bf7c@mail.gmail.com>
Subject: Extending Submission Deadline for Hadoop Fans on the East Coast
To: general@hadoop.apache.org, common-user@hadoop.apache.org, 
	hive-user@hadoop.apache.org, pig-user@hadoop.apache.org, 
	zookeeper-user@hadoop.apache.org, hbase-user <hbase-user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=0016362837a8a59440046ff57a4a
X-Virus-Checked: Checked by ClamAV on apache.org

--0016362837a8a59440046ff57a4a
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hadoop Fans, we have had a few requests to extend the submission deadline
through the weekend. To be fair, rather than make exceptions, we wanted to
let everyone know we are accepting proposals until August 3rd at 10AM PDT.

We've also been really impressed with the number of international
registrations we've seen. It's a continual reminder that Hadoop is a
world-wide phenomenon, and as such, we'd like to encourage this further with
a more inclusive name for the event, Hadoop World: NYC 2009.

Lastly, we now have a much better idea of how much this will event cost to
put on. NYC is expensive. We're keeping the early registration price at $150
until July 31st, 11PM PDT, but it will be $200 through August 10th, and $299
after we start promoting the event more actively.

You'll find everything you need here:
http://www.cloudera.com/hadoop-world-nyc

Christophe
and the Cloudera Team

-- 
get hadoop: cloudera.com/hadoop
online training: cloudera.com/hadoop-training
blog: cloudera.com/blog
twitter: twitter.com/cloudera

--0016362837a8a59440046ff57a4a--

From general-return-421-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 16:58:03 2009
Return-Path: <general-return-421-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45240 invoked from network); 31 Jul 2009 16:58:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 16:58:02 -0000
Received: (qmail 54890 invoked by uid 500); 31 Jul 2009 16:58:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54843 invoked by uid 500); 31 Jul 2009 16:58:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54829 invoked by uid 99); 31 Jul 2009 16:58:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 16:58:02 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [207.126.228.149] (HELO rsmtp1.corp.yahoo.com) (207.126.228.149)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 16:57:51 +0000
Received: from [172.21.149.156] (nat-dip10.cfw-b-gci.corp.yahoo.com [209.131.62.145])
	(authenticated bits=0)
	by rsmtp1.corp.yahoo.com (8.13.8/8.13.8/y.rout) with ESMTP id n6VGvRKu048037
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Fri, 31 Jul 2009 09:57:27 -0700 (PDT)
Message-ID: <4A732277.4050704@apache.org>
Date: Fri, 31 Jul 2009 09:57:27 -0700
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.22 (X11/20090608)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Hudson issue with m/r patch
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

FYI: I had to kill this process:
http://hudson.zones.apache.org/hudson/view/Mapreduce/job/Mapreduce-Patch-vesta.apache.org/431/

it was running for > 18hrs and showed no sign of stopping any time soon.

Patrick

From general-return-422-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 22:50:01 2009
Return-Path: <general-return-422-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78790 invoked from network); 31 Jul 2009 22:50:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 22:50:01 -0000
Received: (qmail 39224 invoked by uid 500); 31 Jul 2009 22:50:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39137 invoked by uid 500); 31 Jul 2009 22:50:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39119 invoked by uid 99); 31 Jul 2009 22:50:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 22:50:01 +0000
X-ASF-Spam-Status: No, hits=4.2 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 22:49:48 +0000
Received: from SP1-EX07CAS02.ds.corp.yahoo.com (sp1-ex07cas02.ds.corp.yahoo.com [216.252.116.138])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n6VMmt16072979;
	Fri, 31 Jul 2009 15:48:55 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:accept-language:content-language:x-ms-has-attach:
	x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=nrkL+K4is8MTvwCBcw7dku8yDxWxvkgyFymOiyAXM3aGyym/oM8pV3lzqANzGd31
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 SP1-EX07CAS02.ds.corp.yahoo.com ([216.252.116.167]) with mapi; Fri, 31 Jul
 2009 15:48:55 -0700
From: Dekel Tankel <dekel@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>,
        "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Date: Fri, 31 Jul 2009 15:48:53 -0700
Subject: Hadoop User Group (Bay Area) - Aug 19th at Yahoo!
Thread-Topic: Hadoop User Group (Bay Area) - Aug 19th at Yahoo!
Thread-Index: AcoSMRRU8X5auVLKR8ufbpFTQi1hLg==
Message-ID: <46E2672895E8744A97D7577A6110858B4E1AC69A91@SP1-EX07VS01.ds.corp.yahoo.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_46E2672895E8744A97D7577A6110858B4E1AC69A91SP1EX07VS01ds_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_46E2672895E8744A97D7577A6110858B4E1AC69A91SP1EX07VS01ds_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi all,



I'd like to remind everyone that RSVP is open for the next monthly Bay Area=
 Hadoop user group organized by Yahoo!.



Agenda and registration available here

http://www.meetup.com/hadoop/calendar/10877366/?action=3Ddetail&eventId=3D1=
0877366



In light of the recently announced search deal with Microsoft<http://yhoo.c=
lient.shareholder.com/press/releasedetail.cfm?ReleaseID=3D399702>, I would =
like to point you to Eric's blog post around Yahoo's strong and on-going co=
mmitment to Hadoop and its community - http://developer.yahoo.net/blogs/had=
oop/2009/07/news_flash_hadoop_development.html



Looking forward to seeing you at August 19th



Dekel


--_000_46E2672895E8744A97D7577A6110858B4E1AC69A91SP1EX07VS01ds_--

From general-return-3591-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 20:27:59 2011
Return-Path: <general-return-3591-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5F5CA6757
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  1 Jun 2011 20:27:59 +0000 (UTC)
Received: (qmail 99403 invoked by uid 500); 1 Jun 2011 20:27:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99086 invoked by uid 500); 1 Jun 2011 20:27:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98937 invoked by uid 99); 1 Jun 2011 20:27:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 20:27:56 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 20:27:49 +0000
Received: from sp1-ex07cas03.ds.corp.yahoo.com (sp1-ex07cas03.ds.corp.yahoo.com [216.252.116.151])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p51KR8tn063026
	for <general@hadoop.apache.org>; Wed, 1 Jun 2011 13:27:08 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1306960028;
	bh=wE1FmRPeeY8rI7AwEzUeQc92yXKGbCFCeYeCUwmpP0g=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:MIME-Version;
	b=UBb0S/IrtjbQEUp2PVJWtjuUFvJEHbeWd+rbDNCvXjjyKRmrsKNg7c+Zjwt3FnUn5
	 CeHf9jHMNDwETvlX94fCXfe8OjXavPswSGh2Z/QA94YaY9rYxXvB5Z5BhleqbHSNQ0
	 QOf5TtQlLq2EsMwYx+KOfcQbItD0bSZCOhMk40uc=
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas03.ds.corp.yahoo.com ([216.252.116.151]) with mapi; Wed, 1 Jun
 2011 13:27:07 -0700
From: Suresh Srinivas <sureshms@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 1 Jun 2011 13:27:05 -0700
Subject: Re: Update on 0.22
Thread-Topic: Update on 0.22
Thread-Index: AcwbcMvgWMMHX8T7R2SeRcODL6hYngFKXmgj
Message-ID: <CA0BEEA9.1AA9A%sureshms@yahoo-inc.com>
In-Reply-To: <1F63BF19-F939-432C-9B77-8ADC4731A05E@mac.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-Entourage/13.8.0.101117
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Nigel,

I see that there are several non blockers being promoted to 0.22 from trunk=
.
>From my understanding, any non blocker change to 0.22 should be approved by
vote. Is this correct?

Regards,
Suresh


On 5/25/11 11:46 PM, "Nigel Daley" <ndaley@mac.com> wrote:

> Looks like we're down to 12 blockers on 0.22.
>=20
> * Thanks to Cloudera for hosting a couple hack-a-thons over the past coup=
le of
> weeks which helped get this number down.
> * Thanks to Devaraj Das for volunteering to get
> https://issues.apache.org/jira/browse/MAPREDUCE-2178 committed.
> * Thanks to Tom White for getting a CI build running that creates the act=
ual
> release artifact.
>=20
> I'm planning to commit https://issues.apache.org/jira/browse/HADOOP-7106 =
(SVN
> reorg) this Friday at 2pm PDT.  Todd, were you able to test git history b=
ased
> on your svn dump and import?
>=20
> Cheers,
> Nige


From general-return-3592-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 20:46:59 2011
Return-Path: <general-return-3592-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B6673625F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  1 Jun 2011 20:46:59 +0000 (UTC)
Received: (qmail 56900 invoked by uid 500); 1 Jun 2011 20:46:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56839 invoked by uid 500); 1 Jun 2011 20:46:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56831 invoked by uid 99); 1 Jun 2011 20:46:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 20:46:58 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of owen.omalley@gmail.com designates 209.85.210.48 as permitted sender)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 20:46:49 +0000
Received: by pzk10 with SMTP id 10so133207pzk.35
        for <general@hadoop.apache.org>; Wed, 01 Jun 2011 13:46:28 -0700 (PDT)
Received: by 10.142.128.15 with SMTP id a15mr1144196wfd.354.1306961187913;
        Wed, 01 Jun 2011 13:46:27 -0700 (PDT)
Received: from battlerock-lm.corp.yahoo.com (nat-dip4.cfw-a-gci.corp.yahoo.com [209.131.62.113])
        by mx.google.com with ESMTPS id b8sm1371130pbj.94.2011.06.01.13.46.25
        (version=SSLv3 cipher=OTHER);
        Wed, 01 Jun 2011 13:46:26 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: Update on 0.22
From: Owen O'Malley <omalley@apache.org>
In-Reply-To: <CA0BEEA9.1AA9A%sureshms@yahoo-inc.com>
Date: Wed, 1 Jun 2011 13:46:24 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <CE461C76-D5B4-4C47-A96B-919F1FEF658E@apache.org>
References: <CA0BEEA9.1AA9A%sureshms@yahoo-inc.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org


On Jun 1, 2011, at 1:27 PM, Suresh Srinivas wrote:

> I see that there are several non blockers being promoted to 0.22 from =
trunk.
> =46rom my understanding, any non blocker change to 0.22 should be =
approved by
> vote. Is this correct?

No, the Release Manager has full control over what goes into a release. =
The PMC votes on it once there is a release candidate.

-- Owen=

From general-return-3593-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 20:51:18 2011
Return-Path: <general-return-3593-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 815C86D20
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  1 Jun 2011 20:51:18 +0000 (UTC)
Received: (qmail 71228 invoked by uid 500); 1 Jun 2011 20:51:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71186 invoked by uid 500); 1 Jun 2011 20:51:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71177 invoked by uid 99); 1 Jun 2011 20:51:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 20:51:16 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 20:51:12 +0000
Received: from host136.benchmark.com (snvvpn1-10-72-244-c1.hq.corp.yahoo.com [10.72.244.1])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p51KoY57070182
	for <general@hadoop.apache.org>; Wed, 1 Jun 2011 13:50:34 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1306961434;
	bh=3N2cOKeO84G4HeHausqnQfnpNk5RoxGnYbljF1O7lr0=;
	h=Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:
	 Content-Transfer-Encoding:Message-Id:References:To;
	b=PU3uOY+UFbiiMCW2qSci/I+riVsfBD8MuPrbno+eAL450SfeUi8FIvtF9UqpoF93c
	 DI1hIkCCfvKBJ9Ns7zPb/BwsNw0WC9X7CUCdXpx0DEkBtK6pk2l1+l3CmpH0OUMpTV
	 0MzYw/v9bZwUm7GwMxpnjY631mBugvAfjCE/qWSw=
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: Update on 0.22
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <CE461C76-D5B4-4C47-A96B-919F1FEF658E@apache.org>
Date: Wed, 1 Jun 2011 13:50:37 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <5786C837-900C-4371-A3D2-568F1E2F8264@yahoo-inc.com>
References: <CA0BEEA9.1AA9A%sureshms@yahoo-inc.com> <CE461C76-D5B4-4C47-A96B-919F1FEF658E@apache.org>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1084)

makes sense to me, but it might be good to work to make these decisions =
visible so folks can understand what is happening.

On Jun 1, 2011, at 1:46 PM, Owen O'Malley wrote:

>=20
> On Jun 1, 2011, at 1:27 PM, Suresh Srinivas wrote:
>=20
>> I see that there are several non blockers being promoted to 0.22 from =
trunk.
>> =46rom my understanding, any non blocker change to 0.22 should be =
approved by
>> vote. Is this correct?
>=20
> No, the Release Manager has full control over what goes into a =
release. The PMC votes on it once there is a release candidate.
>=20
> -- Owen


From general-return-3594-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 21:17:22 2011
Return-Path: <general-return-3594-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7F1B260A4
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  1 Jun 2011 21:17:22 +0000 (UTC)
Received: (qmail 11998 invoked by uid 500); 1 Jun 2011 21:17:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11941 invoked by uid 500); 1 Jun 2011 21:17:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11933 invoked by uid 99); 1 Jun 2011 21:17:20 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 21:17:20 +0000
Received: from localhost (HELO awittena-mn.linkedin.biz) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 21:17:20 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Update on 0.22
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <5786C837-900C-4371-A3D2-568F1E2F8264@yahoo-inc.com>
Date: Wed, 1 Jun 2011 14:17:18 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <B8C4B16F-93CE-44C4-BBAD-AD52E604805F@apache.org>
References: <CA0BEEA9.1AA9A%sureshms@yahoo-inc.com> <CE461C76-D5B4-4C47-A96B-919F1FEF658E@apache.org> <5786C837-900C-4371-A3D2-568F1E2F8264@yahoo-inc.com>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On Jun 1, 2011, at 1:50 PM, Eric Baldeschwieler wrote:

> makes sense to me, but it might be good to work to make these =
decisions visible so folks can understand what is happening.

lol



From general-return-3595-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 22:02:14 2011
Return-Path: <general-return-3595-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 340DF60A8
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  1 Jun 2011 22:02:14 +0000 (UTC)
Received: (qmail 93761 invoked by uid 500); 1 Jun 2011 22:02:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93720 invoked by uid 500); 1 Jun 2011 22:02:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93712 invoked by uid 99); 1 Jun 2011 22:02:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 22:02:12 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 22:02:06 +0000
Received: by wyb40 with SMTP id 40so369220wyb.35
        for <general@hadoop.apache.org>; Wed, 01 Jun 2011 15:01:46 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.15.137 with SMTP id f9mr7543066wef.62.1306965706031; Wed,
 01 Jun 2011 15:01:46 -0700 (PDT)
Received: by 10.216.28.205 with HTTP; Wed, 1 Jun 2011 15:01:45 -0700 (PDT)
In-Reply-To: <5786C837-900C-4371-A3D2-568F1E2F8264@yahoo-inc.com>
References: <CA0BEEA9.1AA9A%sureshms@yahoo-inc.com>
	<CE461C76-D5B4-4C47-A96B-919F1FEF658E@apache.org>
	<5786C837-900C-4371-A3D2-568F1E2F8264@yahoo-inc.com>
Date: Wed, 1 Jun 2011 15:01:45 -0700
Message-ID: <BANLkTino3VLk7WjWMJzLywy7X0e9onX-uQ@mail.gmail.com>
Subject: Re: Update on 0.22
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Wed, Jun 1, 2011 at 1:50 PM, Eric Baldeschwieler
<eric14@yahoo-inc.com> wrote:
> makes sense to me, but it might be good to work to make these decisions visible so folks can understand what is happening.

Hopefully people are updating the fix version in jira to be 0.22 for
anything they are merging.

Thanks,
Eli

From general-return-3596-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 06:33:04 2011
Return-Path: <general-return-3596-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CB33C4F6C
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  2 Jun 2011 06:33:04 +0000 (UTC)
Received: (qmail 50775 invoked by uid 500); 2 Jun 2011 06:33:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50670 invoked by uid 500); 2 Jun 2011 06:33:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50662 invoked by uid 99); 2 Jun 2011 06:33:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 06:33:02 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of shv.hadoop@gmail.com designates 74.125.83.48 as permitted sender)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 06:32:57 +0000
Received: by gwj22 with SMTP id 22so354074gwj.35
        for <general@hadoop.apache.org>; Wed, 01 Jun 2011 23:32:36 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=1H7BLRKTL1N0v3k+0DxH2zEHMm7fcMrtpC87Icu5oV4=;
        b=CxHDEngA3FRPqDk5HUXSX5ycy4kqOANoX77P7Hn76sNIuyEt6z4D6b3qBwwSjTLfYr
         JsX5dZdyVhyn4YEnK8yyfp4p18TNTNDMXFmHkR7230of4n9YGIhlZJgF2cPe0Q477mwa
         urQ9vomn26QwZbxvQOsIP+hHdMZn30ozlso5M=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=vQnwKMzMOrGqy23AJJHHeXBOzOzQPsp+t8hqK5Yqglj3/+hGrXU4DaWEGrEFoBeGlT
         MCjkOA0RkxU60kRd1WSNufrkCjyAmzlk9A8c2Yadk8epKdDGHjg9An73h8fFQBfzC1zJ
         PUmd7CotTdjib5Z2MteEWo5r17xAPcvM5p1NE=
MIME-Version: 1.0
Received: by 10.150.100.10 with SMTP id x10mr322579ybb.190.1306996356594; Wed,
 01 Jun 2011 23:32:36 -0700 (PDT)
Received: by 10.147.33.13 with HTTP; Wed, 1 Jun 2011 23:32:36 -0700 (PDT)
In-Reply-To: <5786C837-900C-4371-A3D2-568F1E2F8264@yahoo-inc.com>
References: <CA0BEEA9.1AA9A%sureshms@yahoo-inc.com>
	<CE461C76-D5B4-4C47-A96B-919F1FEF658E@apache.org>
	<5786C837-900C-4371-A3D2-568F1E2F8264@yahoo-inc.com>
Date: Wed, 1 Jun 2011 23:32:36 -0700
Message-ID: <BANLkTik-5mHTP++D_A_Q=G5qBM52mnCj3Q@mail.gmail.com>
Subject: Re: Update on 0.22
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd2a0088ebafa04a4b4cdfc

--000e0cd2a0088ebafa04a4b4cdfc
Content-Type: text/plain; charset=ISO-8859-1

I can see them well.
I think Suresh's point is that non-blockers are going into 0.22.
Nigel, do you have full control over it?

Thanks,
--Konstantin

On Wed, Jun 1, 2011 at 1:50 PM, Eric Baldeschwieler <eric14@yahoo-inc.com>wrote:

> makes sense to me, but it might be good to work to make these decisions
> visible so folks can understand what is happening.
>
> On Jun 1, 2011, at 1:46 PM, Owen O'Malley wrote:
>
> >
> > On Jun 1, 2011, at 1:27 PM, Suresh Srinivas wrote:
> >
> >> I see that there are several non blockers being promoted to 0.22 from
> trunk.
> >> From my understanding, any non blocker change to 0.22 should be approved
> by
> >> vote. Is this correct?
> >
> > No, the Release Manager has full control over what goes into a release.
> The PMC votes on it once there is a release candidate.
> >
> > -- Owen
>
>

--000e0cd2a0088ebafa04a4b4cdfc--

From general-return-3597-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 17:12:30 2011
Return-Path: <general-return-3597-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AF3424DF1
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  2 Jun 2011 17:12:30 +0000 (UTC)
Received: (qmail 72176 invoked by uid 500); 2 Jun 2011 17:12:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72092 invoked by uid 500); 2 Jun 2011 17:12:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72078 invoked by uid 99); 2 Jun 2011 17:12:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 17:12:28 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of eli@cloudera.com designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 17:12:22 +0000
Received: by ewy22 with SMTP id 22so589150ewy.35
        for <general@hadoop.apache.org>; Thu, 02 Jun 2011 10:12:00 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.28.200 with SMTP id g50mr905133wea.92.1307034720472; Thu,
 02 Jun 2011 10:12:00 -0700 (PDT)
Received: by 10.216.28.205 with HTTP; Thu, 2 Jun 2011 10:12:00 -0700 (PDT)
Date: Thu, 2 Jun 2011 10:12:00 -0700
Message-ID: <BANLkTikt0dw2wQCZBaMKemf1G2Sd3uMqnQ@mail.gmail.com>
Subject: Please join me in welcoming Aaron Myers as the newest Apache Hadoop committer
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On behalf of the PMC, I am pleased to announce that Aaron Myers has
been elected a committer in the Apache Hadoop Common and HDFS
projects.  We appreciate all the work Aaron has put into the project
so far, and are looking forward to his future contributions.  Welcome
aboard atm!

Thanks,
Eli

From general-return-3598-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 17:40:35 2011
Return-Path: <general-return-3598-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 746F04655
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  2 Jun 2011 17:40:35 +0000 (UTC)
Received: (qmail 22815 invoked by uid 500); 2 Jun 2011 17:40:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22716 invoked by uid 500); 2 Jun 2011 17:40:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22708 invoked by uid 99); 2 Jun 2011 17:40:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 17:40:33 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 17:40:27 +0000
Received: by bwz8 with SMTP id 8so1979498bwz.35
        for <general@hadoop.apache.org>; Thu, 02 Jun 2011 10:40:06 -0700 (PDT)
Received: by 10.204.82.224 with SMTP id c32mr971593bkl.161.1307036406324; Thu,
 02 Jun 2011 10:40:06 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.75 with HTTP; Thu, 2 Jun 2011 10:39:46 -0700 (PDT)
In-Reply-To: <BANLkTik-5mHTP++D_A_Q=G5qBM52mnCj3Q@mail.gmail.com>
References: <CA0BEEA9.1AA9A%sureshms@yahoo-inc.com> <CE461C76-D5B4-4C47-A96B-919F1FEF658E@apache.org>
 <5786C837-900C-4371-A3D2-568F1E2F8264@yahoo-inc.com> <BANLkTik-5mHTP++D_A_Q=G5qBM52mnCj3Q@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Thu, 2 Jun 2011 10:39:46 -0700
Message-ID: <BANLkTim0UJoc_r6VCE5dOjosFw=C79ppkw@mail.gmail.com>
Subject: Re: Update on 0.22
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6dd8eb8b51eb004a4be20ca
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6dd8eb8b51eb004a4be20ca
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Jun 1, 2011 at 11:32 PM, Konstantin Shvachko
<shv.hadoop@gmail.com>wrote:

> I can see them well.
> I think Suresh's point is that non-blockers are going into 0.22.
> Nigel, do you have full control over it?
>

Of course it's up to Nigel to decide, but here's my personal opinion:

One of the reasons we had a lot of divergence (read: external
branches/forks/whatever) off of 0.20 is that the commit rules on the branch
were held pretty strictly. So, if you wanted a non-critical bug fix or a
small improvement, the only option was to do such things on an external
fork. 0.20 was branched in December '08 and not released until mid April
'09. In 4 months a fair number of bug fixes and small improvements go in.
0.22 has been around even longer. If we were to keep it to *only* blockers,
then again it would be a fairly useless release due to the number of
non-blocker bugs.

Clearly there's a balance and a judgment call when moving things back to a
branch. But at this point I'd consider small improvements and pretty much
any bug fix to be reasonable, so long as it doesn't involve major reworking
of components. Nigel: if this assumption doesn't jive (ha ha, get it?) with
what you're thinking, please let me know :)

-Todd


> On Wed, Jun 1, 2011 at 1:50 PM, Eric Baldeschwieler <eric14@yahoo-inc.com
> >wrote:
>
> > makes sense to me, but it might be good to work to make these decisions
> > visible so folks can understand what is happening.
> >
> > On Jun 1, 2011, at 1:46 PM, Owen O'Malley wrote:
> >
> > >
> > > On Jun 1, 2011, at 1:27 PM, Suresh Srinivas wrote:
> > >
> > >> I see that there are several non blockers being promoted to 0.22 from
> > trunk.
> > >> From my understanding, any non blocker change to 0.22 should be
> approved
> > by
> > >> vote. Is this correct?
> > >
> > > No, the Release Manager has full control over what goes into a release.
> > The PMC votes on it once there is a release candidate.
> > >
> > > -- Owen
> >
> >
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--0016e6dd8eb8b51eb004a4be20ca--

From general-return-3599-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 17:51:21 2011
Return-Path: <general-return-3599-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C421E44D7
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  2 Jun 2011 17:51:21 +0000 (UTC)
Received: (qmail 45339 invoked by uid 500); 2 Jun 2011 17:51:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45279 invoked by uid 500); 2 Jun 2011 17:51:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45271 invoked by uid 99); 2 Jun 2011 17:51:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 17:51:20 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of harsh@cloudera.com designates 74.125.83.176 as permitted sender)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 17:51:13 +0000
Received: by pve37 with SMTP id 37so705588pve.35
        for <general@hadoop.apache.org>; Thu, 02 Jun 2011 10:50:52 -0700 (PDT)
Received: by 10.142.156.17 with SMTP id d17mr153436wfe.11.1307037052122; Thu,
 02 Jun 2011 10:50:52 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.142.185.10 with HTTP; Thu, 2 Jun 2011 10:50:32 -0700 (PDT)
In-Reply-To: <BANLkTikt0dw2wQCZBaMKemf1G2Sd3uMqnQ@mail.gmail.com>
References: <BANLkTikt0dw2wQCZBaMKemf1G2Sd3uMqnQ@mail.gmail.com>
From: Harsh J <harsh@cloudera.com>
Date: Thu, 2 Jun 2011 23:20:32 +0530
Message-ID: <BANLkTikdtAiyS0bURMvy1_EUPK+RMAN9fQ@mail.gmail.com>
Subject: Re: Please join me in welcoming Aaron Myers as the newest Apache
 Hadoop committer
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Congratulations Aaron

On Thu, Jun 2, 2011 at 10:42 PM, Eli Collins <eli@cloudera.com> wrote:
> On behalf of the PMC, I am pleased to announce that Aaron Myers has
> been elected a committer in the Apache Hadoop Common and HDFS
> projects. =A0We appreciate all the work Aaron has put into the project
> so far, and are looking forward to his future contributions. =A0Welcome
> aboard atm!
>
> Thanks,
> Eli
>



--=20
Harsh J

From general-return-3600-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 18:03:36 2011
Return-Path: <general-return-3600-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7047A499F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  2 Jun 2011 18:03:36 +0000 (UTC)
Received: (qmail 73699 invoked by uid 500); 2 Jun 2011 18:03:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73638 invoked by uid 500); 2 Jun 2011 18:03:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73630 invoked by uid 99); 2 Jun 2011 18:03:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 18:03:34 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 18:03:28 +0000
Received: by qyk2 with SMTP id 2so3143921qyk.14
        for <general@hadoop.apache.org>; Thu, 02 Jun 2011 11:03:06 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=gtNOB63gjCLF4v+gO7k6FDTzKjNCFJ7+ti9Yj0naum4=;
        b=QkoeGdRiUtuJ//1U5yxvWANAZObkBH0rec9Jo1b/EoFXxZGsx8/VLnquXdNOYYZy40
         48pZ2UFBzA+HLlyCDoEV8RhhqLkaI41jZ5UTSwwFd77NFR5FkDQiyiXRaAFX603SN35y
         j84gXgunMApBd3Qo1Bqr+qIlHi2H+dd4oe0vA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=VI5onBDboQ5Qm/UY3UFsjNktSbdEqy6n+lurI1eEFtyhSrchgT97pj/NmM+7dmnBTn
         oGO42HJLdNrSsZ3iCWxY/Lgh/keU2sGbc5xIbCvPtl25u35oG21pYAU0yAyHKf4TXLFz
         /5hDF+hyjSxabCS3D9HRMdOFdi22U8s2VI5Yg=
Received: by 10.224.9.2 with SMTP id j2mr772851qaj.57.1307037786390;
        Thu, 02 Jun 2011 11:03:06 -0700 (PDT)
Received: from [10.180.150.116] (h-64-236-128-40.nat.aol.com [64.236.128.40])
        by mx.google.com with ESMTPS id i34sm498795qck.31.2011.06.02.11.03.02
        (version=TLSv1/SSLv3 cipher=OTHER);
        Thu, 02 Jun 2011 11:03:03 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: Update on 0.22
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <BANLkTim0UJoc_r6VCE5dOjosFw=C79ppkw@mail.gmail.com>
Date: Thu, 2 Jun 2011 14:03:01 -0400
Content-Transfer-Encoding: quoted-printable
Message-Id: <DFD61246-7389-4F31-8768-C589AC2444CF@Holsman.NET>
References: <CA0BEEA9.1AA9A%sureshms@yahoo-inc.com> <CE461C76-D5B4-4C47-A96B-919F1FEF658E@apache.org> <5786C837-900C-4371-A3D2-568F1E2F8264@yahoo-inc.com> <BANLkTik-5mHTP++D_A_Q=G5qBM52mnCj3Q@mail.gmail.com> <BANLkTim0UJoc_r6VCE5dOjosFw=C79ppkw@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

would it make sense to start a 0.22.1 for those kind of features when =
the 0.22.0 release gets closer?

On Jun 2, 2011, at 1:39 PM, Todd Lipcon wrote:

> On Wed, Jun 1, 2011 at 11:32 PM, Konstantin Shvachko
> <shv.hadoop@gmail.com>wrote:
>=20
>> I can see them well.
>> I think Suresh's point is that non-blockers are going into 0.22.
>> Nigel, do you have full control over it?
>>=20
>=20
> Of course it's up to Nigel to decide, but here's my personal opinion:
>=20
> One of the reasons we had a lot of divergence (read: external
> branches/forks/whatever) off of 0.20 is that the commit rules on the =
branch
> were held pretty strictly. So, if you wanted a non-critical bug fix or =
a
> small improvement, the only option was to do such things on an =
external
> fork. 0.20 was branched in December '08 and not released until mid =
April
> '09. In 4 months a fair number of bug fixes and small improvements go =
in.
> 0.22 has been around even longer. If we were to keep it to *only* =
blockers,
> then again it would be a fairly useless release due to the number of
> non-blocker bugs.
>=20
> Clearly there's a balance and a judgment call when moving things back =
to a
> branch. But at this point I'd consider small improvements and pretty =
much
> any bug fix to be reasonable, so long as it doesn't involve major =
reworking
> of components. Nigel: if this assumption doesn't jive (ha ha, get it?) =
with
> what you're thinking, please let me know :)
>=20
> -Todd
>=20
>=20
>> On Wed, Jun 1, 2011 at 1:50 PM, Eric Baldeschwieler =
<eric14@yahoo-inc.com
>>> wrote:
>>=20
>>> makes sense to me, but it might be good to work to make these =
decisions
>>> visible so folks can understand what is happening.
>>>=20
>>> On Jun 1, 2011, at 1:46 PM, Owen O'Malley wrote:
>>>=20
>>>>=20
>>>> On Jun 1, 2011, at 1:27 PM, Suresh Srinivas wrote:
>>>>=20
>>>>> I see that there are several non blockers being promoted to 0.22 =
from
>>> trunk.
>>>>> =46rom my understanding, any non blocker change to 0.22 should be
>> approved
>>> by
>>>>> vote. Is this correct?
>>>>=20
>>>> No, the Release Manager has full control over what goes into a =
release.
>>> The PMC votes on it once there is a release candidate.
>>>>=20
>>>> -- Owen
>>>=20
>>>=20
>>=20
>=20
>=20
>=20
> --=20
> Todd Lipcon
> Software Engineer, Cloudera


From general-return-3601-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 18:07:05 2011
Return-Path: <general-return-3601-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 503884F4A
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  2 Jun 2011 18:07:05 +0000 (UTC)
Received: (qmail 80984 invoked by uid 500); 2 Jun 2011 18:07:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80819 invoked by uid 500); 2 Jun 2011 18:07:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80811 invoked by uid 99); 2 Jun 2011 18:07:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 18:07:03 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of shv.hadoop@gmail.com designates 74.125.83.48 as permitted sender)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 18:06:59 +0000
Received: by gwj22 with SMTP id 22so686583gwj.35
        for <general@hadoop.apache.org>; Thu, 02 Jun 2011 11:06:38 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=ckNt8q/PcVX+7HSzOQbRNsxLv2AEP1kGdb8S7ngl1s0=;
        b=Ydf0bDEl8DKuX8aIVWXoowC3VvVf6WXvlU6y6Qota23t5W1hKuQEAVC7XF9UdjNDbX
         9IJiOPaG9hWVgOZwRBxS3W0706PhpyoZQSkG0oFXrgH0vrWmoTCvFJqex91EKJ9rhWhk
         2xZgQxu2u6PW7GxelGEkFD0vd03nCBZfgazvc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=nRlDJsvU0oW2jkyZKEQpqDVZ77ydM+gaKGrJ42tqny4TLHM66qaB20G63vuxIwulCi
         rHo5Jc5kNh4OVaUgcensF9w1djEoQB4ytYB86PUBHppE39W1LWZQfsAJxLXT+xATE+b6
         WxjXP+U9X8CUgm1Z25oogGefs0yiIuvXMHrJA=
MIME-Version: 1.0
Received: by 10.150.187.18 with SMTP id k18mr1121049ybf.19.1307037998343; Thu,
 02 Jun 2011 11:06:38 -0700 (PDT)
Received: by 10.147.33.13 with HTTP; Thu, 2 Jun 2011 11:06:38 -0700 (PDT)
In-Reply-To: <BANLkTim0UJoc_r6VCE5dOjosFw=C79ppkw@mail.gmail.com>
References: <CA0BEEA9.1AA9A%sureshms@yahoo-inc.com>
	<CE461C76-D5B4-4C47-A96B-919F1FEF658E@apache.org>
	<5786C837-900C-4371-A3D2-568F1E2F8264@yahoo-inc.com>
	<BANLkTik-5mHTP++D_A_Q=G5qBM52mnCj3Q@mail.gmail.com>
	<BANLkTim0UJoc_r6VCE5dOjosFw=C79ppkw@mail.gmail.com>
Date: Thu, 2 Jun 2011 11:06:38 -0700
Message-ID: <BANLkTimirnAWcd2SrMvEAbHBptzo33nbDg@mail.gmail.com>
Subject: Re: Update on 0.22
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd6a84099682f04a4be7fdc

--000e0cd6a84099682f04a4be7fdc
Content-Type: text/plain; charset=ISO-8859-1

I propose just to make them blockers before committing to attract attention
of the release manager and get his approval. Imho, even small changes, like
HDFS-1954 are blockers, because a vague UI message is bug and bugs are
blockers.
Thanks,
--Konstantin


On Thu, Jun 2, 2011 at 10:39 AM, Todd Lipcon <todd@cloudera.com> wrote:

> On Wed, Jun 1, 2011 at 11:32 PM, Konstantin Shvachko
> <shv.hadoop@gmail.com>wrote:
>
> > I can see them well.
> > I think Suresh's point is that non-blockers are going into 0.22.
> > Nigel, do you have full control over it?
> >
>
> Of course it's up to Nigel to decide, but here's my personal opinion:
>
> One of the reasons we had a lot of divergence (read: external
> branches/forks/whatever) off of 0.20 is that the commit rules on the branch
> were held pretty strictly. So, if you wanted a non-critical bug fix or a
> small improvement, the only option was to do such things on an external
> fork. 0.20 was branched in December '08 and not released until mid April
> '09. In 4 months a fair number of bug fixes and small improvements go in.
> 0.22 has been around even longer. If we were to keep it to *only* blockers,
> then again it would be a fairly useless release due to the number of
> non-blocker bugs.
>
> Clearly there's a balance and a judgment call when moving things back to a
> branch. But at this point I'd consider small improvements and pretty much
> any bug fix to be reasonable, so long as it doesn't involve major reworking
> of components. Nigel: if this assumption doesn't jive (ha ha, get it?) with
> what you're thinking, please let me know :)
>
> -Todd
>
>
> > On Wed, Jun 1, 2011 at 1:50 PM, Eric Baldeschwieler <
> eric14@yahoo-inc.com
> > >wrote:
> >
> > > makes sense to me, but it might be good to work to make these decisions
> > > visible so folks can understand what is happening.
> > >
> > > On Jun 1, 2011, at 1:46 PM, Owen O'Malley wrote:
> > >
> > > >
> > > > On Jun 1, 2011, at 1:27 PM, Suresh Srinivas wrote:
> > > >
> > > >> I see that there are several non blockers being promoted to 0.22
> from
> > > trunk.
> > > >> From my understanding, any non blocker change to 0.22 should be
> > approved
> > > by
> > > >> vote. Is this correct?
> > > >
> > > > No, the Release Manager has full control over what goes into a
> release.
> > > The PMC votes on it once there is a release candidate.
> > > >
> > > > -- Owen
> > >
> > >
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

--000e0cd6a84099682f04a4be7fdc--

From general-return-3602-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 18:08:53 2011
Return-Path: <general-return-3602-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3991940E7
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  2 Jun 2011 18:08:53 +0000 (UTC)
Received: (qmail 84055 invoked by uid 500); 2 Jun 2011 18:08:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84010 invoked by uid 500); 2 Jun 2011 18:08:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84002 invoked by uid 99); 2 Jun 2011 18:08:51 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 18:08:51 +0000
Received: from localhost (HELO gkoo.linkedin.biz) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 18:08:51 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Update on 0.22
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <BANLkTimirnAWcd2SrMvEAbHBptzo33nbDg@mail.gmail.com>
Date: Thu, 2 Jun 2011 11:08:49 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <418585F1-50C4-4BCE-93C3-AB4F12FF10A5@apache.org>
References: <CA0BEEA9.1AA9A%sureshms@yahoo-inc.com> <CE461C76-D5B4-4C47-A96B-919F1FEF658E@apache.org> <5786C837-900C-4371-A3D2-568F1E2F8264@yahoo-inc.com> <BANLkTik-5mHTP++D_A_Q=G5qBM52mnCj3Q@mail.gmail.com> <BANLkTim0UJoc_r6VCE5dOjosFw=C79ppkw@mail.gmail.com> <BANLkTimirnAWcd2SrMvEAbHBptzo33nbDg@mail.gmail.com>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On Jun 2, 2011, at 11:06 AM, Konstantin Shvachko wrote:

> I propose just to make them blockers before committing to attract =
attention
> of the release manager and get his approval.

The traditional response has almost always been that they get changed to =
non-blockers before release.  One person's blocker is another person's =
issue to ignore.=

From general-return-3603-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 18:38:35 2011
Return-Path: <general-return-3603-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B093B43C8
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  2 Jun 2011 18:38:35 +0000 (UTC)
Received: (qmail 38649 invoked by uid 500); 2 Jun 2011 18:38:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38510 invoked by uid 500); 2 Jun 2011 18:38:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 38501 invoked by uid 99); 2 Jun 2011 18:38:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 18:38:34 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 18:38:29 +0000
Received: by bwz8 with SMTP id 8so2064785bwz.35
        for <general@hadoop.apache.org>; Thu, 02 Jun 2011 11:38:08 -0700 (PDT)
Received: by 10.204.31.231 with SMTP id z39mr971212bkc.134.1307039487195; Thu,
 02 Jun 2011 11:31:27 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.75 with HTTP; Thu, 2 Jun 2011 11:31:07 -0700 (PDT)
In-Reply-To: <BANLkTimirnAWcd2SrMvEAbHBptzo33nbDg@mail.gmail.com>
References: <CA0BEEA9.1AA9A%sureshms@yahoo-inc.com> <CE461C76-D5B4-4C47-A96B-919F1FEF658E@apache.org>
 <5786C837-900C-4371-A3D2-568F1E2F8264@yahoo-inc.com> <BANLkTik-5mHTP++D_A_Q=G5qBM52mnCj3Q@mail.gmail.com>
 <BANLkTim0UJoc_r6VCE5dOjosFw=C79ppkw@mail.gmail.com> <BANLkTimirnAWcd2SrMvEAbHBptzo33nbDg@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Thu, 2 Jun 2011 11:31:07 -0700
Message-ID: <BANLkTikNG76RDdAHkO3K0qZm3vnw=aXc4g@mail.gmail.com>
Subject: Re: Update on 0.22
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f7949a577b8404a4bed8fc

--001485f7949a577b8404a4bed8fc
Content-Type: text/plain; charset=ISO-8859-1

On Thu, Jun 2, 2011 at 11:06 AM, Konstantin Shvachko
<shv.hadoop@gmail.com>wrote:

> I propose just to make them blockers before committing to attract attention
> of the release manager and get his approval. Imho, even small changes, like
> HDFS-1954 are blockers, because a vague UI message is bug and bugs are
> blockers.
>

Bugs are blockers? Then we'll never release!

Let's hear from Nigel what he thinks. It's his branch, if he's upset about
the way it's being handled, he can deal with it as he sees fit.

-Todd


> On Thu, Jun 2, 2011 at 10:39 AM, Todd Lipcon <todd@cloudera.com> wrote:
>
> > On Wed, Jun 1, 2011 at 11:32 PM, Konstantin Shvachko
> > <shv.hadoop@gmail.com>wrote:
> >
> > > I can see them well.
> > > I think Suresh's point is that non-blockers are going into 0.22.
> > > Nigel, do you have full control over it?
> > >
> >
> > Of course it's up to Nigel to decide, but here's my personal opinion:
> >
> > One of the reasons we had a lot of divergence (read: external
> > branches/forks/whatever) off of 0.20 is that the commit rules on the
> branch
> > were held pretty strictly. So, if you wanted a non-critical bug fix or a
> > small improvement, the only option was to do such things on an external
> > fork. 0.20 was branched in December '08 and not released until mid April
> > '09. In 4 months a fair number of bug fixes and small improvements go in.
> > 0.22 has been around even longer. If we were to keep it to *only*
> blockers,
> > then again it would be a fairly useless release due to the number of
> > non-blocker bugs.
> >
> > Clearly there's a balance and a judgment call when moving things back to
> a
> > branch. But at this point I'd consider small improvements and pretty much
> > any bug fix to be reasonable, so long as it doesn't involve major
> reworking
> > of components. Nigel: if this assumption doesn't jive (ha ha, get it?)
> with
> > what you're thinking, please let me know :)
> >
> > -Todd
> >
> >
> > > On Wed, Jun 1, 2011 at 1:50 PM, Eric Baldeschwieler <
> > eric14@yahoo-inc.com
> > > >wrote:
> > >
> > > > makes sense to me, but it might be good to work to make these
> decisions
> > > > visible so folks can understand what is happening.
> > > >
> > > > On Jun 1, 2011, at 1:46 PM, Owen O'Malley wrote:
> > > >
> > > > >
> > > > > On Jun 1, 2011, at 1:27 PM, Suresh Srinivas wrote:
> > > > >
> > > > >> I see that there are several non blockers being promoted to 0.22
> > from
> > > > trunk.
> > > > >> From my understanding, any non blocker change to 0.22 should be
> > > approved
> > > > by
> > > > >> vote. Is this correct?
> > > > >
> > > > > No, the Release Manager has full control over what goes into a
> > release.
> > > > The PMC votes on it once there is a release candidate.
> > > > >
> > > > > -- Owen
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--001485f7949a577b8404a4bed8fc--

From general-return-3604-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 02:04:59 2011
Return-Path: <general-return-3604-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3D58A6C4E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri,  3 Jun 2011 02:04:59 +0000 (UTC)
Received: (qmail 41170 invoked by uid 500); 3 Jun 2011 02:04:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 41113 invoked by uid 500); 3 Jun 2011 02:04:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 41105 invoked by uid 99); 3 Jun 2011 02:04:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 02:04:57 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of tearsend@gmail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 02:04:52 +0000
Received: by iwr19 with SMTP id 19so1870469iwr.35
        for <general@hadoop.apache.org>; Thu, 02 Jun 2011 19:04:32 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=4MwNKnUulX0PzJWRxRE2snIRvbHqXbeMlfntv8D8SZA=;
        b=vcVrHDoK3oxS1Ndgh6bo6aOzX/PIFGpcSzlW7Ycbo/Q9BS67TPdiy2LsMRvB+6R1PN
         b5JeQmXioK7fPhvv9mc1kIJDepTMOv3plc+NIqrNAodgaVfpodnHCTAw8HFBqMRWXYHl
         b85FOTShSf9r07PDb9xZdv2sKqXOjxZRwXBsM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=jBK8x7Z8n9HIQxcl65F7ZmqNlLq/mgY5xvkPURuNCJ4Z8H2/bsjLIfILz7rEueSODp
         H+uoPHSPohBaIXmzlszJPD5ArRJ9KxswDuwy8BeoN2qPRv4KPn8QsipfDXYklp96N6n9
         YhN7o5hLDnacRPpDhRmOVYyuByt6mLa7G80OY=
MIME-Version: 1.0
Received: by 10.231.82.197 with SMTP id c5mr1766367ibl.131.1307066671843; Thu,
 02 Jun 2011 19:04:31 -0700 (PDT)
Received: by 10.231.148.3 with HTTP; Thu, 2 Jun 2011 19:04:31 -0700 (PDT)
In-Reply-To: <BANLkTikdtAiyS0bURMvy1_EUPK+RMAN9fQ@mail.gmail.com>
References: <BANLkTikt0dw2wQCZBaMKemf1G2Sd3uMqnQ@mail.gmail.com>
	<BANLkTikdtAiyS0bURMvy1_EUPK+RMAN9fQ@mail.gmail.com>
Date: Fri, 3 Jun 2011 11:04:31 +0900
Message-ID: <BANLkTik8bt9033ue787evT_ip2gfbUY-9Q@mail.gmail.com>
Subject: Re: Please join me in welcoming Aaron Myers as the newest Apache
 Hadoop committer
From: Kum seong Moon <tearsend@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cdf13deac4ac804a4c52cd2

--000e0cdf13deac4ac804a4c52cd2
Content-Type: text/plain; charset=ISO-8859-1

Congratulations, Mr.Myers!

When, how can I be a person like you?

Cheers,
Jamie Moon

2011/6/3 Harsh J <harsh@cloudera.com>

> Congratulations Aaron!
>
> On Thu, Jun 2, 2011 at 10:42 PM, Eli Collins <eli@cloudera.com> wrote:
> > On behalf of the PMC, I am pleased to announce that Aaron Myers has
> > been elected a committer in the Apache Hadoop Common and HDFS
> > projects.  We appreciate all the work Aaron has put into the project
> > so far, and are looking forward to his future contributions.  Welcome
> > aboard atm!
> >
> > Thanks,
> > Eli
> >
>
>
>
> --
> Harsh J
>

--000e0cdf13deac4ac804a4c52cd2--

From general-return-3605-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 04:58:48 2011
Return-Path: <general-return-3605-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 24F716D2F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri,  3 Jun 2011 04:58:48 +0000 (UTC)
Received: (qmail 69228 invoked by uid 500); 3 Jun 2011 04:58:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69179 invoked by uid 500); 3 Jun 2011 04:58:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69171 invoked by uid 99); 3 Jun 2011 04:58:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 04:58:43 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 04:58:36 +0000
Received: by bwz8 with SMTP id 8so2660273bwz.35
        for <general@hadoop.apache.org>; Thu, 02 Jun 2011 21:58:15 -0700 (PDT)
Received: by 10.204.79.196 with SMTP id q4mr1478578bkk.107.1307077095138; Thu,
 02 Jun 2011 21:58:15 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.75 with HTTP; Thu, 2 Jun 2011 21:57:55 -0700 (PDT)
From: Todd Lipcon <todd@cloudera.com>
Date: Thu, 2 Jun 2011 21:57:55 -0700
Message-ID: <BANLkTi=jQfSVdcGAUa2=SVnS-zpLL1yaQQ@mail.gmail.com>
Subject: Spam on wiki
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6dd9788f3259904a4c79955

--0016e6dd9788f3259904a4c79955
Content-Type: text/plain; charset=ISO-8859-1

FYI, I've filed the following ticket with ASF Infrastructure to see if we
can get a CAPTCHA set up on our wiki:
https://issues.apache.org/jira/browse/INFRA-3670

In the meantime, we've been doing a decent job of policing, let's keep it up

<https://issues.apache.org/jira/browse/INFRA-3670>-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

--0016e6dd9788f3259904a4c79955--

From general-return-3606-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 05:19:41 2011
Return-Path: <general-return-3606-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CFA076851
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri,  3 Jun 2011 05:19:41 +0000 (UTC)
Received: (qmail 90704 invoked by uid 500); 3 Jun 2011 05:19:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90641 invoked by uid 500); 3 Jun 2011 05:19:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90633 invoked by uid 99); 3 Jun 2011 05:19:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 05:19:38 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 05:19:31 +0000
Received: by bwz8 with SMTP id 8so2673456bwz.35
        for <general@hadoop.apache.org>; Thu, 02 Jun 2011 22:19:10 -0700 (PDT)
Received: by 10.204.187.129 with SMTP id cw1mr1512283bkb.80.1307078350185;
 Thu, 02 Jun 2011 22:19:10 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.75 with HTTP; Thu, 2 Jun 2011 22:18:50 -0700 (PDT)
In-Reply-To: <BANLkTik8bt9033ue787evT_ip2gfbUY-9Q@mail.gmail.com>
References: <BANLkTikt0dw2wQCZBaMKemf1G2Sd3uMqnQ@mail.gmail.com>
 <BANLkTikdtAiyS0bURMvy1_EUPK+RMAN9fQ@mail.gmail.com> <BANLkTik8bt9033ue787evT_ip2gfbUY-9Q@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Thu, 2 Jun 2011 22:18:50 -0700
Message-ID: <BANLkTi=pn2nk11EVcxq_jKqJCyZUWD7YWA@mail.gmail.com>
Subject: Re: Please join me in welcoming Aaron Myers as the newest Apache
 Hadoop committer
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf3024505fc1a34004a4c7e406
X-Virus-Checked: Checked by ClamAV on apache.org

--20cf3024505fc1a34004a4c7e406
Content-Type: text/plain; charset=ISO-8859-1

To Aaron: congrats!

To Jamie, and anyone else wondering how to become a committer, there are
some brief guidelines to how many people in the Apache community see the
role:
http://community.apache.org/newcommitter.html

In the end, though, it is a vote by the project's management committee (PMC)
where each member can vote based on the criteria they think will best
benefit the project.

I won't speak for other PMC members, but my top criteria are:
- Do they have good technical skills in general, and also demonstrate good
understanding of some part of our specific codebase? (in other words, can
they effectively review and commit new contributor's work without
introducing bad code or bugs)
- Do they know their limitations? (eg knowing when it's better to let
another committer review a patch because they don't know the possible
ramifications of the change well enough)
- Are they a team player who can compromise even if it means some extra work
for them? (eg adding additional unit tests when asked, changing a design
based on some feedback even if they have to throw away some code, etc)

So, if you or anyone else would like to become a committer, the best way is
to start by showing the qualities above. Here are some concrete steps you
might take as a new contributor:
- subscribe to the "dev" mailing lists and watch for broken Hudson builds.
If you see a test start newly failing, try to investigate which commit broke
it and do your best to debug what might have caused it.
- subscribe to the "user" mailing list and answer other users' questions
(this helps demonstrate that you want to help others and that you know the
product/code well)
- find a bug and submit a patch to fix it. Here's a link to a bunch of
tickets that have been deemed easy enough for new contributors:
https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+in+(%22HADOOP%22,+%22MAPREDUCE%22,+%22HDFS%22)+and+labels+%3D+%22newbie%22+and+status+%3D+open

If you or anyone else would like any more advice about how best to
contribute, feel free to reach out to me or the dev lists (specific
questions are best!)

Thanks
-Todd


On Thu, Jun 2, 2011 at 7:04 PM, Kum seong Moon <tearsend@gmail.com> wrote:

> Congratulations, Mr.Myers!
>
> When, how can I be a person like you?
>
> Cheers,
> Jamie Moon
>
> 2011/6/3 Harsh J <harsh@cloudera.com>
>
> > Congratulations Aaron!
> >
> > On Thu, Jun 2, 2011 at 10:42 PM, Eli Collins <eli@cloudera.com> wrote:
> > > On behalf of the PMC, I am pleased to announce that Aaron Myers has
> > > been elected a committer in the Apache Hadoop Common and HDFS
> > > projects.  We appreciate all the work Aaron has put into the project
> > > so far, and are looking forward to his future contributions.  Welcome
> > > aboard atm!
> > >
> > > Thanks,
> > > Eli
> > >
> >
> >
> >
> > --
> > Harsh J
> >
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--20cf3024505fc1a34004a4c7e406--

From general-return-3607-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 11:21:52 2011
Return-Path: <general-return-3607-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 58ADF64A8
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri,  3 Jun 2011 11:21:52 +0000 (UTC)
Received: (qmail 27484 invoked by uid 500); 3 Jun 2011 11:21:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27412 invoked by uid 500); 3 Jun 2011 11:21:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27404 invoked by uid 99); 3 Jun 2011 11:21:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 11:21:50 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [98.139.52.232] (HELO nm13-vm0.bullet.mail.ac4.yahoo.com) (98.139.52.232)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 03 Jun 2011 11:21:41 +0000
Received: from [98.139.52.192] by nm13.bullet.mail.ac4.yahoo.com with NNFMP; 03 Jun 2011 11:21:20 -0000
Received: from [98.139.52.177] by tm5.bullet.mail.ac4.yahoo.com with NNFMP; 03 Jun 2011 11:21:20 -0000
Received: from [127.0.0.1] by omp1060.mail.ac4.yahoo.com with NNFMP; 03 Jun 2011 11:21:20 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 221587.32032.bm@omp1060.mail.ac4.yahoo.com
Received: (qmail 84819 invoked by uid 60001); 3 Jun 2011 11:21:20 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1307100080; bh=iB8cu/y9rds4U1rYe2XLAaYEmkslAAAngc0sdQVqez4=; h=Message-ID:X-YMail-OSG:Received:X-RocketYMMF:X-Mailer:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=As+MJChixVUpYs1yH32eaJsNBA4PduxcNwkJ5pV/0gLyz6iBnCNgnB1bhw3m6uEKa1cxmilFMMa0qFx+IyYMk1hJiAguzURFgtqvubTEF2AAnOlBVUa98ze1aAgG1dL+S3ZkZOMFJjJf3TqXb5iphcE0o2NUTU6VysoLttXGPTw=
Message-ID: <897227.67593.qm@web65506.mail.ac4.yahoo.com>
X-YMail-OSG: mMZbNxkVM1m6rvO0jUYyJPMQnAJn0iXMd5BXvhgauoJ36RX
 _Ur.5X77AU8EOqv2hkVJWuOGDY5Vk5DybSRi7UHnirZnezBM6jvo4D0ys1nu
 2z1YFXWIqSRztB6vKggRA6_wpOZJMaI5cLHQA0E.7bguxeOt7OsiFOCINQrr
 M8NcGDKPTa0jWeeMOKF.6WZykSMHkEi4r2NPq6_ZRha7s4aYyW.TbvqqjrUt
 gBIrjuxeex5KgMKMQH8_b0yHSWqcEtlXKvr5OIaibGPDx7Qxwyw5slv37OUd
 RhpquzYEL8FotZ49Yel2LHiwA.8dN.skOgTkQ8398uSqUq1tGAJKii9VQUHI
 hnt0zJ39aTj2NnrxHPPnYB3CFWg6uetch5WTqLwhsyQCvxzSLBig3NuSEb3V
 qteC3LCd07FkK4A--
Received: from [213.61.227.42] by web65506.mail.ac4.yahoo.com via HTTP; Fri, 03 Jun 2011 04:21:19 PDT
X-RocketYMMF: apurtell
X-Mailer: YahooMailClassic/14.0.1 YahooMailWebService/0.8.111.303096
Date: Fri, 3 Jun 2011 04:21:19 -0700 (PDT)
From: Andrew Purtell <apurtell@apache.org>
Reply-To: apurtell@apache.org
Subject: Re: Please join me in welcoming Aaron Myers as the newest Apache Hadoop committer
To: general@hadoop.apache.org
In-Reply-To: <BANLkTikt0dw2wQCZBaMKemf1G2Sd3uMqnQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Congratulations Aaron.=0A=0AIt's great to see new committers on HDFS, and knowing a bit about what you guys are up to, this is exciting.=0A=0A  - Andy=
=0A=0A> From: Eli Collins <eli@cloudera.com>=0A> Subject: Please join me in=
 welcoming Aaron Myers as the newest Apache Hadoop committer=0A> To: genera=
l@hadoop.apache.org=0A> Date: Thursday, June 2, 2011, 10:12 AM=0A> On behal=
f of the PMC, I am pleased to announce that Aaron Myers has=0A> been electe=
d a committer in the Apache Hadoop Common and HDFS=0A> projects.=A0 We appr=
eciate all the work Aaron has put into the project=0A> so far, and are look=
ing forward to his future contributions.=A0 Welcome=0A> aboard atm!=0A> =0A=
> Thanks,=0A> Eli=0A

From general-return-3608-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 12:46:54 2011
Return-Path: <general-return-3608-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3D2AF6EDC
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri,  3 Jun 2011 12:46:54 +0000 (UTC)
Received: (qmail 74202 invoked by uid 500); 3 Jun 2011 12:46:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74149 invoked by uid 500); 3 Jun 2011 12:46:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74140 invoked by uid 99); 3 Jun 2011 12:46:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 12:46:52 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mlortiz@uci.cu designates 200.55.140.180 as permitted sender)
Received: from [200.55.140.180] (HELO mx3.uci.cu) (200.55.140.180)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 03 Jun 2011 12:46:47 +0000
Received: (qmail 26708 invoked by uid 507); 3 Jun 2011 12:46:20 -0000
Received: from 10.0.0.185 by ns3.uci.cu (envelope-from <mlortiz@uci.cu>, uid 501) with qmail-scanner-2.01st 
 (avp: 5.0.2.0. spamassassin: 3.0.6. perlscan: 2.01st.  
 Clear:RC:1(10.0.0.185):. 
 Processed in 0.637393 secs); 03 Jun 2011 12:46:20 -0000
Received: from unknown (HELO ucimail4.uci.cu) (10.0.0.185)
  by 0 with SMTP; 3 Jun 2011 12:46:19 -0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by ucimail4.uci.cu (Postfix) with ESMTP id 97E76132461C;
	Fri,  3 Jun 2011 08:46:19 -0400 (CDT)
X-Virus-Scanned: amavisd-new at uci.cu
Received: from ucimail4.uci.cu ([127.0.0.1])
	by localhost (ucimail4.uci.cu [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id J09YaFjC7TLg; Fri,  3 Jun 2011 08:46:18 -0400 (CDT)
Received: from [10.36.18.57] (skynet2010.uci.cu [10.36.18.57])
	by ucimail4.uci.cu (Postfix) with ESMTP id A50C813243A7;
	Fri,  3 Jun 2011 08:46:18 -0400 (CDT)
Message-ID: <4DE8D7F9.3020501@uci.cu>
Date: Fri, 03 Jun 2011 08:47:53 -0400
From: Marcos Ortiz <mlortiz@uci.cu>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; es-ES; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: general@hadoop.apache.org
CC: Andrew Purtell <apurtell@apache.org>
Subject: Re: Please join me in welcoming Aaron Myers as the newest Apache
 Hadoop committer
References: <897227.67593.qm@web65506.mail.ac4.yahoo.com>
In-Reply-To: <897227.67593.qm@web65506.mail.ac4.yahoo.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit

El 6/3/2011 7:21 AM, Andrew Purtell escribió:
> Congratulations Aaron.
>
> It's great to see new committers on HDFS, and knowing a bit about what you guys are up to, this is exciting.
>
>    - Andy
>
>    
>> From: Eli Collins<eli@cloudera.com>
>> Subject: Please join me in welcoming Aaron Myers as the newest Apache Hadoop committer
>> To: general@hadoop.apache.org
>> Date: Thursday, June 2, 2011, 10:12 AM
>> On behalf of the PMC, I am pleased to announce that Aaron Myers has
>> been elected a committer in the Apache Hadoop Common and HDFS
>> projects.  We appreciate all the work Aaron has put into the project
>> so far, and are looking forward to his future contributions.  Welcome
>> aboard atm!
>>
>> Thanks,
>> Eli
>>      
>    
Congratulations, Aaron, good for you.
Regards

-- 
Marcos Luís Ortíz Valmaseda
  Software Engineer (Large-Scaled Distributed Systems)
  University of Information Sciences,
  La Habana, Cuba
  Linux User # 418229
  http://marcosluis2186.posterous.com


From general-return-3609-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 13:07:21 2011
Return-Path: <general-return-3609-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 10590657E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri,  3 Jun 2011 13:07:21 +0000 (UTC)
Received: (qmail 5876 invoked by uid 500); 3 Jun 2011 13:07:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5819 invoked by uid 500); 3 Jun 2011 13:07:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5801 invoked by uid 99); 3 Jun 2011 13:07:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 13:07:19 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mlortiz@uci.cu designates 200.55.140.180 as permitted sender)
Received: from [200.55.140.180] (HELO mx3.uci.cu) (200.55.140.180)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 03 Jun 2011 13:07:14 +0000
Received: (qmail 4346 invoked by uid 507); 3 Jun 2011 13:06:49 -0000
Received: from 10.0.0.185 by ns3.uci.cu (envelope-from <mlortiz@uci.cu>, uid 501) with qmail-scanner-2.01st 
 (avp: 5.0.2.0. spamassassin: 3.0.6. perlscan: 2.01st.  
 Clear:RC:1(10.0.0.185):. 
 Processed in 0.646783 secs); 03 Jun 2011 13:06:49 -0000
Received: from unknown (HELO ucimail4.uci.cu) (10.0.0.185)
  by 0 with SMTP; 3 Jun 2011 13:06:49 -0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by ucimail4.uci.cu (Postfix) with ESMTP id 3A474132461C;
	Fri,  3 Jun 2011 09:06:49 -0400 (CDT)
X-Virus-Scanned: amavisd-new at uci.cu
Received: from ucimail4.uci.cu ([127.0.0.1])
	by localhost (ucimail4.uci.cu [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id kMkhjpiPoKMV; Fri,  3 Jun 2011 09:06:48 -0400 (CDT)
Received: from [10.36.18.57] (skynet2010.uci.cu [10.36.18.57])
	by ucimail4.uci.cu (Postfix) with ESMTP id 1FC8113243A7;
	Fri,  3 Jun 2011 09:06:48 -0400 (CDT)
Message-ID: <4DE8DCD0.3090605@uci.cu>
Date: Fri, 03 Jun 2011 09:08:32 -0400
From: Marcos Ortiz <mlortiz@uci.cu>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; es-ES; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: general@hadoop.apache.org
CC: Todd Lipcon <todd@cloudera.com>
Subject: Re: Please join me in welcoming Aaron Myers as the newest Apache
 Hadoop committer
References: <BANLkTikt0dw2wQCZBaMKemf1G2Sd3uMqnQ@mail.gmail.com> <BANLkTikdtAiyS0bURMvy1_EUPK+RMAN9fQ@mail.gmail.com> <BANLkTik8bt9033ue787evT_ip2gfbUY-9Q@mail.gmail.com> <BANLkTi=pn2nk11EVcxq_jKqJCyZUWD7YWA@mail.gmail.com>
In-Reply-To: <BANLkTi=pn2nk11EVcxq_jKqJCyZUWD7YWA@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit

El 6/3/2011 1:18 AM, Todd Lipcon escribió:
> To Aaron: congrats!
>
> To Jamie, and anyone else wondering how to become a committer, there are
> some brief guidelines to how many people in the Apache community see the
> role:
> http://community.apache.org/newcommitter.html
>
> In the end, though, it is a vote by the project's management committee (PMC)
> where each member can vote based on the criteria they think will best
> benefit the project.
>
> I won't speak for other PMC members, but my top criteria are:
> - Do they have good technical skills in general, and also demonstrate good
> understanding of some part of our specific codebase? (in other words, can
> they effectively review and commit new contributor's work without
> introducing bad code or bugs)
> - Do they know their limitations? (eg knowing when it's better to let
> another committer review a patch because they don't know the possible
> ramifications of the change well enough)
> - Are they a team player who can compromise even if it means some extra work
> for them? (eg adding additional unit tests when asked, changing a design
> based on some feedback even if they have to throw away some code, etc)
>
> So, if you or anyone else would like to become a committer, the best way is
> to start by showing the qualities above. Here are some concrete steps you
> might take as a new contributor:
> - subscribe to the "dev" mailing lists and watch for broken Hudson builds.
> If you see a test start newly failing, try to investigate which commit broke
> it and do your best to debug what might have caused it.
> - subscribe to the "user" mailing list and answer other users' questions
> (this helps demonstrate that you want to help others and that you know the
> product/code well)
> - find a bug and submit a patch to fix it. Here's a link to a bunch of
> tickets that have been deemed easy enough for new contributors:
> https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+in+(%22HADOOP%22,+%22MAPREDUCE%22,+%22HDFS%22)+and+labels+%3D+%22newbie%22+and+status+%3D+open
>
> If you or anyone else would like any more advice about how best to
> contribute, feel free to reach out to me or the dev lists (specific
> questions are best!)
>
> Thanks
> -Todd
>
>
> On Thu, Jun 2, 2011 at 7:04 PM, Kum seong Moon<tearsend@gmail.com>  wrote:
>
>    
>> Congratulations, Mr.Myers!
>>
>> When, how can I be a person like you?
>>
>> Cheers,
>> Jamie Moon
>>
>> 2011/6/3 Harsh J<harsh@cloudera.com>
>>
>>      
>>> Congratulations Aaron!
>>>
>>> On Thu, Jun 2, 2011 at 10:42 PM, Eli Collins<eli@cloudera.com>  wrote:
>>>        
>>>> On behalf of the PMC, I am pleased to announce that Aaron Myers has
>>>> been elected a committer in the Apache Hadoop Common and HDFS
>>>> projects.  We appreciate all the work Aaron has put into the project
>>>> so far, and are looking forward to his future contributions.  Welcome
>>>> aboard atm!
>>>>
>>>> Thanks,
>>>> Eli
>>>>
>>>>          
>>>
>>>
>>> --
>>> Harsh J
>>>
>>>        
>>      
>
>
>    
Good points, Todd.

-- 
Marcos Luís Ortíz Valmaseda
  Software Engineer (Distributed Systems)
  Universidad de las Ciencias Informaticas
  La Habana, Cuba
  http://marcosluis2186.posterous.com
  http://twitter.com/marcosluis2186
   


From general-return-3610-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun  5 10:01:23 2011
Return-Path: <general-return-3610-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 92E96674F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sun,  5 Jun 2011 10:01:23 +0000 (UTC)
Received: (qmail 36875 invoked by uid 500); 5 Jun 2011 10:01:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36730 invoked by uid 500); 5 Jun 2011 10:01:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36722 invoked by uid 99); 5 Jun 2011 10:01:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jun 2011 10:01:16 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jun 2011 10:01:06 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 212081BA0A4
	for <general@hadoop.apache.org>; Sun,  5 Jun 2011 11:00:45 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id Ea453i7yhZPu for <general@hadoop.apache.org>;
	Sun,  5 Jun 2011 11:00:41 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 4BCBF1BA6AD
	for <general@hadoop.apache.org>; Sun,  5 Jun 2011 11:00:40 +0100 (BST)
MailScanner-NULL-Check: 1307872829.48937@omzdN2Scki71S7Avu/IGzw
Received: from [16.24.224.148] ([16.24.224.148])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p55A0SP0018370
	for <general@hadoop.apache.org>; Sun, 5 Jun 2011 11:00:28 +0100 (BST)
Message-ID: <4DEB53B7.6070301@apache.org>
Date: Sun, 05 Jun 2011 11:00:23 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.2.17) Gecko/20110414 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Spam on wiki
References: <BANLkTi=jQfSVdcGAUa2=SVnS-zpLL1yaQQ@mail.gmail.com>
In-Reply-To: <BANLkTi=jQfSVdcGAUa2=SVnS-zpLL1yaQQ@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p55A0SP0018370
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 03/06/2011 05:57, Todd Lipcon wrote:
> FYI, I've filed the following ticket with ASF Infrastructure to see if we
> can get a CAPTCHA set up on our wiki:
> https://issues.apache.org/jira/browse/INFRA-3670
>
> In the meantime, we've been doing a decent job of policing, let's keep it up
>
> <https://issues.apache.org/jira/browse/INFRA-3670>-Todd

There's some way of setting up patterns to block. I think it's just 
adding the relevant regexp to

http://wiki.apache.org/hadoop/BadContent

is that right?

From general-return-3611-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 16:45:24 2011
Return-Path: <general-return-3611-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 980B74F2F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  6 Jun 2011 16:45:24 +0000 (UTC)
Received: (qmail 62117 invoked by uid 500); 6 Jun 2011 16:45:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62035 invoked by uid 500); 6 Jun 2011 16:45:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61864 invoked by uid 99); 6 Jun 2011 16:45:22 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 16:45:22 +0000
Received: from localhost (HELO dhcp-02.private.iobm.com) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 16:45:21 +0000
From: Allen Wittenauer <aw@apache.org>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: LimitedPrivate and HBase
Date: Mon, 6 Jun 2011 09:45:21 -0700
Message-Id: <41B787A2-3FF6-42D6-87E2-38DCBC7174F4@apache.org>
To: general@hadoop.apache.org
Mime-Version: 1.0 (Apple Message framework v1082)
X-Mailer: Apple Mail (2.1082)



	I have some concerns over the recent usage of LimitedPrivate =
being opened up to HBase.  Shouldn't HBase really be sticking to public =
APIs rather than poking through some holes?  If HBase needs an API, =
wouldn't other clients as well?




From general-return-3612-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 17:01:37 2011
Return-Path: <general-return-3612-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6DBB34890
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  6 Jun 2011 17:01:37 +0000 (UTC)
Received: (qmail 13189 invoked by uid 500); 6 Jun 2011 17:01:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13133 invoked by uid 500); 6 Jun 2011 17:01:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13125 invoked by uid 99); 6 Jun 2011 17:01:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 17:01:35 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 17:01:27 +0000
Received: by bwz8 with SMTP id 8so6087477bwz.35
        for <general@hadoop.apache.org>; Mon, 06 Jun 2011 10:01:07 -0700 (PDT)
Received: by 10.204.145.18 with SMTP id b18mr5229225bkv.26.1307379667508; Mon,
 06 Jun 2011 10:01:07 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.75 with HTTP; Mon, 6 Jun 2011 10:00:47 -0700 (PDT)
In-Reply-To: <41B787A2-3FF6-42D6-87E2-38DCBC7174F4@apache.org>
References: <41B787A2-3FF6-42D6-87E2-38DCBC7174F4@apache.org>
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 6 Jun 2011 10:00:47 -0700
Message-ID: <BANLkTi=X8qD1Ypa8qKA1AF7284sb7ick2Q@mail.gmail.com>
Subject: Re: LimitedPrivate and HBase
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001517476246ab1e2b04a50e0c13
X-Virus-Checked: Checked by ClamAV on apache.org

--001517476246ab1e2b04a50e0c13
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Jun 6, 2011 at 9:45 AM, Allen Wittenauer <aw@apache.org> wrote:

>
>
>        I have some concerns over the recent usage of LimitedPrivate being
> opened up to HBase.  Shouldn't HBase really be sticking to public APIs
> rather than poking through some holes?  If HBase needs an API, wouldn't
> other clients as well?
>

IMO LimitedPrivate can be used to open an API for a specific project when
it's not clear that the API is generally useful, and/or we anticipate the
API might be pretty unstable. Marking it LimitedPrivate to HBase gives us
the opportunity to talk to the HBase team and say "hey, we want to rename
this without @Deprecation" or "hey, we're going to kill this, is that OK?"
Making it true public, even if we call it Unstable, is a bit harder to move.

I agree that most of these things in the long run would be determined
generally useful and made public.

Do you have a specific thing in mind?

-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

--001517476246ab1e2b04a50e0c13--

From general-return-3613-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 17:36:53 2011
Return-Path: <general-return-3613-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0B31F449C
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  6 Jun 2011 17:36:52 +0000 (UTC)
Received: (qmail 96265 invoked by uid 500); 6 Jun 2011 17:36:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96217 invoked by uid 500); 6 Jun 2011 17:36:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96209 invoked by uid 99); 6 Jun 2011 17:36:51 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 17:36:51 +0000
Received: from localhost (HELO awittena-md.linkedin.biz) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 17:36:51 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: LimitedPrivate and HBase
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <BANLkTi=X8qD1Ypa8qKA1AF7284sb7ick2Q@mail.gmail.com>
Date: Mon, 6 Jun 2011 10:36:49 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <1C10ECAD-DBDC-4ED7-BF45-4B922F72306A@apache.org>
References: <41B787A2-3FF6-42D6-87E2-38DCBC7174F4@apache.org> <BANLkTi=X8qD1Ypa8qKA1AF7284sb7ick2Q@mail.gmail.com>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On Jun 6, 2011, at 10:00 AM, Todd Lipcon wrote:

> On Mon, Jun 6, 2011 at 9:45 AM, Allen Wittenauer <aw@apache.org> =
wrote:
>=20
>>=20
>>=20
>>       I have some concerns over the recent usage of LimitedPrivate =
being
>> opened up to HBase.  Shouldn't HBase really be sticking to public =
APIs
>> rather than poking through some holes?  If HBase needs an API, =
wouldn't
>> other clients as well?
>>=20
>=20
> IMO LimitedPrivate can be used to open an API for a specific project =
when
> it's not clear that the API is generally useful, and/or we anticipate =
the
> API might be pretty unstable. Marking it LimitedPrivate to HBase gives =
us
> the opportunity to talk to the HBase team and say "hey, we want to =
rename
> this without @Deprecation" or "hey, we're going to kill this, is that =
OK?"
> Making it true public, even if we call it Unstable, is a bit harder to =
move.

	True, but what makes HBase a unique snowflake?  What is the =
criteria by which an API gets opened to something outside of the Hadoop =
umbrella?  Why shouldn't something be private to Hadoop just because =
HBase or someone else wants to use it? =20

> I agree that most of these things in the long run would be determined
> generally useful and made public.
>=20
> Do you have a specific thing in mind?

	Not really.  I just wanted to take a temperature to see what =
other people are thinking and whether we actually have any guidelines.  =
Doing this wild west style seems fraught with danger.  ("You opened this =
up for them, why can't you up this up for us?")

	(Although I must admit that the first usage of this--the =
HttpServer code--sort of left with a "Seriously? You must be kidding." =
kind of feeling about it.)=

From general-return-3614-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 18:35:16 2011
Return-Path: <general-return-3614-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C4DD74F85
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  6 Jun 2011 18:35:16 +0000 (UTC)
Received: (qmail 4273 invoked by uid 500); 6 Jun 2011 18:35:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4223 invoked by uid 500); 6 Jun 2011 18:35:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4215 invoked by uid 99); 6 Jun 2011 18:35:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 18:35:15 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 209.85.216.176 as permitted sender)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 18:35:08 +0000
Received: by qyk30 with SMTP id 30so2605168qyk.14
        for <general@hadoop.apache.org>; Mon, 06 Jun 2011 11:34:47 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=L0N8kLlYZ2UssbW9/snzRm9euy33M67WIs/QgB6fn8s=;
        b=OVc82242HHIjVNvot3WhHygOtCMpW12TcKJ/8u9zpNpnYPUcOYAY43Mim7gXAMAZ8s
         9otEBLksc5nBqZftY70nLZLGWr1jiEBSkC/GC2WV7n01QxlgGhXIGavIf/gvrd6tqiZG
         /QqFL6d0fB15qVJTGjktw8ki2pv75WsFjikzA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=cLVncWrwu/rPp4U+zQRh3xelzYYNouqyHPe0KKAR7ADmhogv7Fj5LodWuQyji9H+FK
         ksrng5q5rJu4ZV6rUYJCsLemrf2YFSxq+bo0zZQ7lKqfHSGb9KWN3cuY3/a/BqIE+2Mr
         jDzp7rhGs4cUG9VF9+H35tDpM0W/ynPOcpZ24=
MIME-Version: 1.0
Received: by 10.224.192.69 with SMTP id dp5mr2262646qab.267.1307385287611;
 Mon, 06 Jun 2011 11:34:47 -0700 (PDT)
Sender: saint.ack@gmail.com
Received: by 10.224.6.146 with HTTP; Mon, 6 Jun 2011 11:34:47 -0700 (PDT)
In-Reply-To: <41B787A2-3FF6-42D6-87E2-38DCBC7174F4@apache.org>
References: <41B787A2-3FF6-42D6-87E2-38DCBC7174F4@apache.org>
Date: Mon, 6 Jun 2011 11:34:47 -0700
X-Google-Sender-Auth: YFdwPuOD5PbtU7-oHgYp5fX2sso
Message-ID: <BANLkTineoX-pPK+Z3DzEb3qNC7FjxbvjVw@mail.gmail.com>
Subject: Re: LimitedPrivate and HBase
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Mon, Jun 6, 2011 at 9:45 AM, Allen Wittenauer <aw@apache.org> wrote:
>
>
> =A0 =A0 =A0 =A0I have some concerns over the recent usage of LimitedPriva=
te being opened up to HBase. =A0Shouldn't HBase really be sticking to publi=
c APIs rather than poking through some holes? =A0If HBase needs an API, wou=
ldn't other clients as well?
>

HBase uses public APIs.  A method we relied on went from protected to
private.  Fixing this brought on the flagging of HttpServer with
LimitedPrivate (HttpServer, the class flagged, is mostly internal to
Hadoop but HBase, because, in part, of its subproject provenance,
extends HttpServer providing its UI reusing Hadoop's log level, thread
dumping, etc., servlets)

The LimitedPrivate aid as I see it is to give folks pause the next
time they mess with access/signatures or, if change is necessary,
they'll know who to give the head-up to that change is coming --
should they choose to do so.

St.Ack

From general-return-3615-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 20:45:26 2011
Return-Path: <general-return-3615-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 58CB3447A
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  6 Jun 2011 20:45:26 +0000 (UTC)
Received: (qmail 24294 invoked by uid 500); 6 Jun 2011 20:45:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24243 invoked by uid 500); 6 Jun 2011 20:45:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24235 invoked by uid 99); 6 Jun 2011 20:45:24 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 20:45:24 +0000
Received: from localhost (HELO awittena-md.linkedin.biz) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 20:45:24 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: LimitedPrivate and HBase
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <BANLkTineoX-pPK+Z3DzEb3qNC7FjxbvjVw@mail.gmail.com>
Date: Mon, 6 Jun 2011 13:45:21 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <08E3D059-20B7-49F4-AA53-C1012C9480A1@apache.org>
References: <41B787A2-3FF6-42D6-87E2-38DCBC7174F4@apache.org> <BANLkTineoX-pPK+Z3DzEb3qNC7FjxbvjVw@mail.gmail.com>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On Jun 6, 2011, at 11:34 AM, Stack wrote:

> On Mon, Jun 6, 2011 at 9:45 AM, Allen Wittenauer <aw@apache.org> =
wrote:
>>=20
>>=20
>>        I have some concerns over the recent usage of LimitedPrivate =
being opened up to HBase.  Shouldn't HBase really be sticking to public =
APIs rather than poking through some holes?  If HBase needs an API, =
wouldn't other clients as well?
>>=20
>=20
> HBase uses public APIs.  A method we relied on went from protected to
> private.  Fixing this brought on the flagging of HttpServer with
> LimitedPrivate (HttpServer, the class flagged, is mostly internal to
> Hadoop but HBase, because, in part, of its subproject provenance,
> extends HttpServer providing its UI reusing Hadoop's log level, thread
> dumping, etc., servlets)

	So, no, HBase does not actually use public APIs....


From general-return-3616-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 23:22:53 2011
Return-Path: <general-return-3616-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A0AF94B4C
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  6 Jun 2011 23:22:53 +0000 (UTC)
Received: (qmail 30640 invoked by uid 500); 6 Jun 2011 23:22:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30581 invoked by uid 500); 6 Jun 2011 23:22:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30571 invoked by uid 99); 6 Jun 2011 23:22:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 23:22:52 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [98.139.52.220] (HELO nm23.bullet.mail.ac4.yahoo.com) (98.139.52.220)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 06 Jun 2011 23:22:41 +0000
Received: from [98.139.52.189] by nm23.bullet.mail.ac4.yahoo.com with NNFMP; 06 Jun 2011 23:22:20 -0000
Received: from [98.139.52.186] by tm2.bullet.mail.ac4.yahoo.com with NNFMP; 06 Jun 2011 23:22:20 -0000
Received: from [127.0.0.1] by omp1069.mail.ac4.yahoo.com with NNFMP; 06 Jun 2011 23:22:20 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 836124.38278.bm@omp1069.mail.ac4.yahoo.com
Received: (qmail 9172 invoked by uid 60001); 6 Jun 2011 23:22:20 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1307402540; bh=v5340qFXkxivifsRX6Bq/dM5Lb4wKKPDpX6n5+SvJAI=; h=Message-ID:X-YMail-OSG:Received:X-RocketYMMF:X-Mailer:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=oFYAPuzE0SSd5BNgOiI3Zwu89EpW90nxVgybBXqqvGeOKSyqTOMxxuqn4kkrvgYsxlQL33+Ep5bRj0WaiNEsF7c4ZhLgloE8UqrfiOpJChOMdq/D6SNy2ZO6fcwVAjBgE3zbmPuRjM+25/CBannow5vErNteHT7PpjyLd3G2ZQg=
Message-ID: <580675.9102.qm@web65506.mail.ac4.yahoo.com>
X-YMail-OSG: KvyWBfQVM1n4ZTUgMQvcPRMMDj1Cg4YDCLzifmsea6IAPjr
 oFEl6KTE2Mus0JmUvQhGafpIOOAPxkc7YzHQNc0K8SdC.gctPD5c77Sx5hSC
 i50lLq5ZMUpib_SivohibWd3NAZnuI41GK6BY4_ZPv3RMjIYNc5Z_3DQuGLy
 qyCrO6AVcaTtDvXx28x6r7nrweLGH2Xzl6utlBlGLVEjP01gP3dIdcwkOuGR
 ASDWxSATM7QTdT.rwuwAGrI_kfxcsK9742TrDkY_HLvHVnilWynLHePZl_PG
 5xDGi9v6wd0_POXdJra56p_.9DgKfLmmM4gxE3g.RXbxUUaF8Wy4JHNiB9cv
 KoXgeoJtM2D9nFYOPeI59XBlrP.AlQ4c3ul.0GzB4BF_JUS.bRPEmI0MJd7R
 4.l07m3R_Ifi4Tg--
Received: from [213.61.227.39] by web65506.mail.ac4.yahoo.com via HTTP; Mon, 06 Jun 2011 16:22:19 PDT
X-RocketYMMF: apurtell
X-Mailer: YahooMailClassic/14.0.1 YahooMailWebService/0.8.111.303096
Date: Mon, 6 Jun 2011 16:22:19 -0700 (PDT)
From: Andrew Purtell <apurtell@apache.org>
Reply-To: apurtell@apache.org
Subject: Re: LimitedPrivate and HBase
To: general@hadoop.apache.org
In-Reply-To: <08E3D059-20B7-49F4-AA53-C1012C9480A1@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org

On Mon, 6/6/11, Allen Wittenauer <aw@apache.org> wrote:
>     So, no, HBase does not actually use public APIs....

We could instead cut and paste the code in question here from Hadoop and modify it for our purposes. (This was done for RPC, earlier.) 

However other Hadoop-ish daemons may well want to extend HttpServer in this same way so the servlets for thread dumps and adjusting logging levels can be reused rather than cloned or reimplemented (or dropped). How HttpServer as implemented configures the container requires one to reach in a bit to set these up plus our own webapps.

Perhaps opening a jira for a cleaner framework for HttpServer extension could be useful?

Best regards,

    - Andy


From general-return-3617-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 23:34:14 2011
Return-Path: <general-return-3617-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1B62C41DB
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  6 Jun 2011 23:34:14 +0000 (UTC)
Received: (qmail 42597 invoked by uid 500); 6 Jun 2011 23:34:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42542 invoked by uid 500); 6 Jun 2011 23:34:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42534 invoked by uid 99); 6 Jun 2011 23:34:12 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 23:34:12 +0000
Received: from localhost (HELO awittena-md.linkedin.biz) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 23:34:12 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: LimitedPrivate and HBase
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <580675.9102.qm@web65506.mail.ac4.yahoo.com>
Date: Mon, 6 Jun 2011 16:34:10 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <4AFD3D6A-9556-4343-91B0-6CA794A0192F@apache.org>
References: <580675.9102.qm@web65506.mail.ac4.yahoo.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)


On Jun 6, 2011, at 4:22 PM, Andrew Purtell wrote:
>=20
> Perhaps opening a jira for a cleaner framework for HttpServer =
extension could be useful?

	Sure.  That's probably what should have happened to begin with =
rather than the quickly changing the API to a different classification.  =
I was a bit surprised that the JIRA went through so quickly without much =
comment, but I assumed that's because it was HBase requesting or because =
so many people were on their way to Berlin.

	That said, big picture issue: I still think there should be some =
official guidance on when LimitedPrivate is the right thing to do.  I'd =
hate to see random groups requesting an API change "just this once" with =
no real leg to stand on whether it was appropriate or not.=

From general-return-3618-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 00:57:16 2011
Return-Path: <general-return-3618-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6681B4F57
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue,  7 Jun 2011 00:57:16 +0000 (UTC)
Received: (qmail 18558 invoked by uid 500); 7 Jun 2011 00:57:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18489 invoked by uid 500); 7 Jun 2011 00:57:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18481 invoked by uid 99); 7 Jun 2011 00:57:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 00:57:14 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 00:57:06 +0000
Received: by bwz8 with SMTP id 8so6649453bwz.35
        for <general@hadoop.apache.org>; Mon, 06 Jun 2011 17:56:46 -0700 (PDT)
Received: by 10.204.41.206 with SMTP id p14mr5267178bke.53.1307408206173; Mon,
 06 Jun 2011 17:56:46 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.75 with HTTP; Mon, 6 Jun 2011 17:56:25 -0700 (PDT)
In-Reply-To: <4AFD3D6A-9556-4343-91B0-6CA794A0192F@apache.org>
References: <580675.9102.qm@web65506.mail.ac4.yahoo.com> <4AFD3D6A-9556-4343-91B0-6CA794A0192F@apache.org>
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 6 Jun 2011 17:56:25 -0700
Message-ID: <BANLkTimvOLS-EnwAeExQKxfVUMf74dq+dg@mail.gmail.com>
Subject: Re: LimitedPrivate and HBase
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec555517ab4972504a514b1f0
X-Virus-Checked: Checked by ClamAV on apache.org

--bcaec555517ab4972504a514b1f0
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Jun 6, 2011 at 4:34 PM, Allen Wittenauer <aw@apache.org> wrote:

>
> On Jun 6, 2011, at 4:22 PM, Andrew Purtell wrote:
> >
> > Perhaps opening a jira for a cleaner framework for HttpServer extension
> could be useful?
>
>         Sure.  That's probably what should have happened to begin with
> rather than the quickly changing the API to a different classification.  I
> was a bit surprised that the JIRA went through so quickly without much
> comment, but I assumed that's because it was HBase requesting or because so
> many people were on their way to Berlin.
>

Or because this is the sort of thing that could take weeks of discussion or
just 5 minutes to unblock HBase from moving on to trunk. I'd rather have the
weeks of discussion *after* the 5 minute patch, so people can continue to
make progress. We've moved too slowly for too long.


>
>        That said, big picture issue: I still think there should be some
> official guidance on when LimitedPrivate is the right thing to do.  I'd hate
> to see random groups requesting an API change "just this once" with no real
> leg to stand on whether it was appropriate or not.




-- 
Todd Lipcon
Software Engineer, Cloudera

--bcaec555517ab4972504a514b1f0--

From general-return-3619-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 01:05:19 2011
Return-Path: <general-return-3619-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2045646AA
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue,  7 Jun 2011 01:05:19 +0000 (UTC)
Received: (qmail 23600 invoked by uid 500); 7 Jun 2011 01:05:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23465 invoked by uid 500); 7 Jun 2011 01:05:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23457 invoked by uid 99); 7 Jun 2011 01:05:17 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 01:05:17 +0000
Received: from localhost (HELO awittena-md.linkedin.biz) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 01:05:17 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: LimitedPrivate and HBase
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <BANLkTimvOLS-EnwAeExQKxfVUMf74dq+dg@mail.gmail.com>
Date: Mon, 6 Jun 2011 18:05:15 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <742D84CF-B599-463A-A6E0-2B5732FD27B6@apache.org>
References: <580675.9102.qm@web65506.mail.ac4.yahoo.com> <4AFD3D6A-9556-4343-91B0-6CA794A0192F@apache.org> <BANLkTimvOLS-EnwAeExQKxfVUMf74dq+dg@mail.gmail.com>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On Jun 6, 2011, at 5:56 PM, Todd Lipcon wrote:

> Or because this is the sort of thing that could take weeks of =
discussion or
> just 5 minutes to unblock HBase from moving on to trunk. I'd rather =
have the
> weeks of discussion *after* the 5 minute patch, so people can continue =
to
> make progress. We've moved too slowly for too long.


	I didn't realize trunk was coming out as a release next month.

	Let's face it: this happened because it was HBase.  If it was =
almost anyone else, it would have sat there.... and *that's* the point =
where I'm mainly concerned.=

From general-return-3620-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 01:09:09 2011
Return-Path: <general-return-3620-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B173F46F6
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue,  7 Jun 2011 01:09:09 +0000 (UTC)
Received: (qmail 28543 invoked by uid 500); 7 Jun 2011 01:09:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28486 invoked by uid 500); 7 Jun 2011 01:09:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28476 invoked by uid 99); 7 Jun 2011 01:09:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 01:09:08 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 01:09:01 +0000
Received: by bwz8 with SMTP id 8so6659397bwz.35
        for <general@hadoop.apache.org>; Mon, 06 Jun 2011 18:08:40 -0700 (PDT)
Received: by 10.204.145.18 with SMTP id b18mr341807bkv.26.1307408920229; Mon,
 06 Jun 2011 18:08:40 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.75 with HTTP; Mon, 6 Jun 2011 18:08:20 -0700 (PDT)
In-Reply-To: <742D84CF-B599-463A-A6E0-2B5732FD27B6@apache.org>
References: <580675.9102.qm@web65506.mail.ac4.yahoo.com> <4AFD3D6A-9556-4343-91B0-6CA794A0192F@apache.org>
 <BANLkTimvOLS-EnwAeExQKxfVUMf74dq+dg@mail.gmail.com> <742D84CF-B599-463A-A6E0-2B5732FD27B6@apache.org>
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 6 Jun 2011 18:08:20 -0700
Message-ID: <BANLkTin12Cwt2juVRS63rbBdcMQ7bRd_Qw@mail.gmail.com>
Subject: Re: LimitedPrivate and HBase
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015174762464436aa04a514dca6

--0015174762464436aa04a514dca6
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Jun 6, 2011 at 6:05 PM, Allen Wittenauer <aw@apache.org> wrote:

>
> On Jun 6, 2011, at 5:56 PM, Todd Lipcon wrote:
>
> > Or because this is the sort of thing that could take weeks of discussion
> or
> > just 5 minutes to unblock HBase from moving on to trunk. I'd rather have
> the
> > weeks of discussion *after* the 5 minute patch, so people can continue to
> > make progress. We've moved too slowly for too long.
>
>
>         I didn't realize trunk was coming out as a release next month.
>

If all goes well, 0.22 will come out as a release some time in that
timeframe. Stack has been getting HBase running on it. This patch was to fix
0.22.


>
>        Let's face it: this happened because it was HBase.  If it was almost
> anyone else, it would have sat there.... and *that's* the point where I'm
> mainly concerned.


If you want to feel better, take a look at HDFS-941, HDFS-347, and HDFS-918
- these are patches that HBase has been asking for for nearly 2 years in
some cases and haven't gone in. Satisfied?

-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

--0015174762464436aa04a514dca6--

From general-return-3621-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 01:18:09 2011
Return-Path: <general-return-3621-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5C73A4310
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue,  7 Jun 2011 01:18:09 +0000 (UTC)
Received: (qmail 35934 invoked by uid 500); 7 Jun 2011 01:18:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 35880 invoked by uid 500); 7 Jun 2011 01:18:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 35872 invoked by uid 99); 7 Jun 2011 01:18:07 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 01:18:07 +0000
Received: from localhost (HELO awittena-md.linkedin.biz) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 01:18:07 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: LimitedPrivate and HBase
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <BANLkTin12Cwt2juVRS63rbBdcMQ7bRd_Qw@mail.gmail.com>
Date: Mon, 6 Jun 2011 18:18:06 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <361B71C9-1216-4EBF-98DA-732AEB0C21E8@apache.org>
References: <580675.9102.qm@web65506.mail.ac4.yahoo.com> <4AFD3D6A-9556-4343-91B0-6CA794A0192F@apache.org> <BANLkTimvOLS-EnwAeExQKxfVUMf74dq+dg@mail.gmail.com> <742D84CF-B599-463A-A6E0-2B5732FD27B6@apache.org> <BANLkTin12Cwt2juVRS63rbBdcMQ7bRd_Qw@mail.gmail.com>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On Jun 6, 2011, at 6:08 PM, Todd Lipcon wrote:
>=20
>>=20
>>       Let's face it: this happened because it was HBase.  If it was =
almost
>> anyone else, it would have sat there.... and *that's* the point where =
I'm
>> mainly concerned.
>=20
>=20
> If you want to feel better, take a look at HDFS-941, HDFS-347, and =
HDFS-918
> - these are patches that HBase has been asking for for nearly 2 years =
in
> some cases and haven't gone in. Satisfied?

	These cases don't appear to be about re-classification of an API =
from private to semi-public.  So no, I'm not.  None of these appear to =
answer the base set of question:

	- What is the real criteria for changing an API from private to =
limited?
	- How "closely related" does a project need to be to get this =
privilege?

	(Yes, I've read the classification docs.  That's too vague.)

	I can tell you feel I'm picking on HBase, especially in light of =
my flat out rejection of the "we want to mmap() blocks" case.  But if =
this reclassification had been with anything else outside of the Hadoop =
project, I would have asked the same thing.  It raises important =
questions that we as a project need to answer.=

From general-return-3622-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 01:25:44 2011
Return-Path: <general-return-3622-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 70DA34F17
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue,  7 Jun 2011 01:25:44 +0000 (UTC)
Received: (qmail 45540 invoked by uid 500); 7 Jun 2011 01:25:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45488 invoked by uid 500); 7 Jun 2011 01:25:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45480 invoked by uid 99); 7 Jun 2011 01:25:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 01:25:42 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 01:25:37 +0000
Received: by bwz8 with SMTP id 8so6673452bwz.35
        for <general@hadoop.apache.org>; Mon, 06 Jun 2011 18:25:16 -0700 (PDT)
Received: by 10.204.41.206 with SMTP id p14mr5286622bke.53.1307409916155; Mon,
 06 Jun 2011 18:25:16 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.75 with HTTP; Mon, 6 Jun 2011 18:23:48 -0700 (PDT)
In-Reply-To: <361B71C9-1216-4EBF-98DA-732AEB0C21E8@apache.org>
References: <580675.9102.qm@web65506.mail.ac4.yahoo.com> <4AFD3D6A-9556-4343-91B0-6CA794A0192F@apache.org>
 <BANLkTimvOLS-EnwAeExQKxfVUMf74dq+dg@mail.gmail.com> <742D84CF-B599-463A-A6E0-2B5732FD27B6@apache.org>
 <BANLkTin12Cwt2juVRS63rbBdcMQ7bRd_Qw@mail.gmail.com> <361B71C9-1216-4EBF-98DA-732AEB0C21E8@apache.org>
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 6 Jun 2011 18:23:48 -0700
Message-ID: <BANLkTinh-9M2USuGRMy9-K2WreuuRbqeVA@mail.gmail.com>
Subject: Re: LimitedPrivate and HBase
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec555517aa0d6d204a51517cd

--bcaec555517aa0d6d204a51517cd
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Jun 6, 2011 at 6:18 PM, Allen Wittenauer <aw@apache.org> wrote:

>        These cases don't appear to be about re-classification of an API
> from private to semi-public.  So no, I'm not.  None of these appear to
> answer the base set of question:
>
>
In the specific case of HttpServer, this API *used* to be semi-public, then
it was made private as a side effect of another change. (I should know, I'm
the one who made the change that accidentally privatized it... it wasn't for
any thought-out reason)


>        - What is the real criteria for changing an API from private to
> limited?
>        - How "closely related" does a project need to be to get this
> privilege?
>
>        (Yes, I've read the classification docs.  That's too vague.)
>
>        I can tell you feel I'm picking on HBase, especially in light of my
> flat out rejection of the "we want to mmap() blocks" case.  But if this
> reclassification had been with anything else outside of the Hadoop project,
> I would have asked the same thing.  It raises important questions that we as
> a project need to answer.


Nah, I just think these "meta discussions" waste an awful lot of time that's
better spent making real progress on the code, or reviewing the complex
changes where extra eyes really make a big difference.

http://www.bikeshed.com/

-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

--bcaec555517aa0d6d204a51517cd--

From general-return-3623-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 01:33:14 2011
Return-Path: <general-return-3623-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5B395408D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue,  7 Jun 2011 01:33:14 +0000 (UTC)
Received: (qmail 55055 invoked by uid 500); 7 Jun 2011 01:33:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54997 invoked by uid 500); 7 Jun 2011 01:33:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54989 invoked by uid 99); 7 Jun 2011 01:33:12 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 01:33:12 +0000
Received: from localhost (HELO awittena-md.linkedin.biz) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 01:33:12 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: LimitedPrivate and HBase
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <BANLkTinh-9M2USuGRMy9-K2WreuuRbqeVA@mail.gmail.com>
Date: Mon, 6 Jun 2011 18:33:10 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <8A170F27-BA48-474A-905B-1F5B38D5F4FF@apache.org>
References: <580675.9102.qm@web65506.mail.ac4.yahoo.com> <4AFD3D6A-9556-4343-91B0-6CA794A0192F@apache.org> <BANLkTimvOLS-EnwAeExQKxfVUMf74dq+dg@mail.gmail.com> <742D84CF-B599-463A-A6E0-2B5732FD27B6@apache.org> <BANLkTin12Cwt2juVRS63rbBdcMQ7bRd_Qw@mail.gmail.com> <361B71C9-1216-4EBF-98DA-732AEB0C21E8@apache.org> <BANLkTinh-9M2USuGRMy9-K2WreuuRbqeVA@mail.gmail.com>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On Jun 6, 2011, at 6:23 PM, Todd Lipcon wrote:
>=20
> Nah, I just think these "meta discussions" waste an awful lot of time =
that's
> better spent making real progress on the code, or reviewing the =
complex
> changes where extra eyes really make a big difference.


	OK.  That's make it easier to just -1 changes like this with =
reasoning such as "HBase is not a related project." Then we can go back =
working on core Hadoop.





From general-return-3624-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 01:50:57 2011
Return-Path: <general-return-3624-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7FF8A4F7E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue,  7 Jun 2011 01:50:57 +0000 (UTC)
Received: (qmail 71691 invoked by uid 500); 7 Jun 2011 01:50:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71623 invoked by uid 500); 7 Jun 2011 01:50:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71615 invoked by uid 99); 7 Jun 2011 01:50:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 01:50:55 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 01:50:49 +0000
Received: by vws7 with SMTP id 7so5480632vws.35
        for <general@hadoop.apache.org>; Mon, 06 Jun 2011 18:50:28 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=NxmEbPKX4rxNpc3ml+KPv3WNeJZfz8VUJMkOwLs6irE=;
        b=GSXLWpRU329N9Ias3mfxjE/deTLgeAFEEi/qQx2RnhUSyY01vFquvrBE0bi2F2zxY6
         Fl19+638fQyAG5pEIjYKc8o0qE8pU4cNajtWj2TqNBeGbaU314snCuc0eee4sj9UTVxj
         LagpfiYeRwE9KdalCLucUy0ghRGmAfZHCSDeE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=i3aRrTyFvDkzuboohAd9tHIWsn2DTUu1KPIe2Zq+n/A38+hgl4g7a/nnG3KSTe9KOS
         XAy4knlRFZ7iyyZDGw4NYbOkZzHcOTxVQV7l497zn2Xc+ZX3cw21dTijsUoxwjMYTMnk
         rXvU0RMIW7VqcoRlh37bJBUj9kfvbUcdzD0oc=
Received: by 10.52.97.197 with SMTP id ec5mr4431204vdb.259.1307411428488;
        Mon, 06 Jun 2011 18:50:28 -0700 (PDT)
Received: from [192.168.10.200] (static-76-160-17-94.dsl.cavtel.net [76.160.17.94])
        by mx.google.com with ESMTPS id b9sm796783vdk.13.2011.06.06.18.50.27
        (version=TLSv1/SSLv3 cipher=OTHER);
        Mon, 06 Jun 2011 18:50:27 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: LimitedPrivate and HBase
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <361B71C9-1216-4EBF-98DA-732AEB0C21E8@apache.org>
Date: Mon, 6 Jun 2011 21:50:26 -0400
Content-Transfer-Encoding: quoted-printable
Message-Id: <17CBDF8F-CDE2-4D8D-AC5D-EB222CF24BD1@Holsman.NET>
References: <580675.9102.qm@web65506.mail.ac4.yahoo.com> <4AFD3D6A-9556-4343-91B0-6CA794A0192F@apache.org> <BANLkTimvOLS-EnwAeExQKxfVUMf74dq+dg@mail.gmail.com> <742D84CF-B599-463A-A6E0-2B5732FD27B6@apache.org> <BANLkTin12Cwt2juVRS63rbBdcMQ7bRd_Qw@mail.gmail.com> <361B71C9-1216-4EBF-98DA-732AEB0C21E8@apache.org>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)


On Jun 6, 2011, at 9:18 PM, Allen Wittenauer wrote:

>=20
> 	 But if this reclassification had been with anything else =
outside of the Hadoop project, I would have asked the same thing.  It =
raises important questions that we as a project need to answer.

"Life's not fair".:-) (couldn't resist.. sorry)

But in all seriousness, the project should treat every request on it's =
own merits, regardless of which project/person requests it.
In actuality if you've been working closely with another project or know =
the person who is asking for it, you'll naturally put more weight on the =
request, as you have more trust that that person knows what they are =
doing, and aren't being lazy/poorly informed about requesting it.

I was wondering if there are other examples of interfaces you (or =
others) think should un-deprecated/made public.=20=

From general-return-3625-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 09:10:13 2011
Return-Path: <general-return-3625-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 725AC6893
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue,  7 Jun 2011 09:10:13 +0000 (UTC)
Received: (qmail 45247 invoked by uid 500); 7 Jun 2011 09:10:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45054 invoked by uid 500); 7 Jun 2011 09:10:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45046 invoked by uid 99); 7 Jun 2011 09:10:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 09:10:11 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 09:10:02 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 3B8B5B7D49
	for <general@hadoop.apache.org>; Tue,  7 Jun 2011 10:09:41 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id 4WlnaIa+ravN for <general@hadoop.apache.org>;
	Tue,  7 Jun 2011 10:09:40 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 7220BB7D44
	for <general@hadoop.apache.org>; Tue,  7 Jun 2011 10:09:40 +0100 (BST)
MailScanner-NULL-Check: 1308042566.36247@xS+uW9JIAF7ENmWpvD31ag
Received: from [16.25.175.86] (wildhaus.hpl.hp.com [16.25.175.86])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p5799O0k020122
	for <general@hadoop.apache.org>; Tue, 7 Jun 2011 10:09:24 +0100 (BST)
Message-ID: <4DEDEAC4.8000408@apache.org>
Date: Tue, 07 Jun 2011 10:09:24 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110419 Red Hat/3.1.10-1.el6_0 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: LimitedPrivate and HBase
References: <41B787A2-3FF6-42D6-87E2-38DCBC7174F4@apache.org>
In-Reply-To: <41B787A2-3FF6-42D6-87E2-38DCBC7174F4@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p5799O0k020122
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 06/06/2011 05:45 PM, Allen Wittenauer wrote:
>
>
> 	I have some concerns over the recent usage of LimitedPrivate being opened up to HBase.  Shouldn't HBase really be sticking to public APIs rather than poking through some holes?  If HBase needs an API, wouldn't other clients as well?
>
>

Hey, isn't HBase part of the official Hadoop stack?

From general-return-3626-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 10:11:31 2011
Return-Path: <general-return-3626-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AD6E96A2A
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue,  7 Jun 2011 10:11:31 +0000 (UTC)
Received: (qmail 51559 invoked by uid 500); 7 Jun 2011 10:11:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51505 invoked by uid 500); 7 Jun 2011 10:11:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51497 invoked by uid 99); 7 Jun 2011 10:11:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 10:11:30 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 10:11:22 +0000
Received: by vws7 with SMTP id 7so5797577vws.35
        for <general@hadoop.apache.org>; Tue, 07 Jun 2011 03:11:01 -0700 (PDT)
Received: by 10.52.177.233 with SMTP id ct9mr1165362vdc.112.1307441461068;
 Tue, 07 Jun 2011 03:11:01 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.52.110.134 with HTTP; Tue, 7 Jun 2011 03:10:41 -0700 (PDT)
X-Originating-IP: [79.194.77.34]
In-Reply-To: <4DEDEAC4.8000408@apache.org>
References: <41B787A2-3FF6-42D6-87E2-38DCBC7174F4@apache.org> <4DEDEAC4.8000408@apache.org>
From: Ted Dunning <tdunning@maprtech.com>
Date: Tue, 7 Jun 2011 03:10:41 -0700
Message-ID: <BANLkTikYPt+qQ3spxRstau+51mjOcL2Exw@mail.gmail.com>
Subject: Re: LimitedPrivate and HBase
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf307abde9da048104a51c6fde

--20cf307abde9da048104a51c6fde
Content-Type: text/plain; charset=ISO-8859-1

Not really.  It isn't part of any Hadoop release.  And no official Hadoop
release will run hbase reliably.

On Tue, Jun 7, 2011 at 2:09 AM, Steve Loughran <stevel@apache.org> wrote:

> On 06/06/2011 05:45 PM, Allen Wittenauer wrote:
>
>>
>>
>>        I have some concerns over the recent usage of LimitedPrivate being
>> opened up to HBase.  Shouldn't HBase really be sticking to public APIs
>> rather than poking through some holes?  If HBase needs an API, wouldn't
>> other clients as well?
>>
>>
>>
> Hey, isn't HBase part of the official Hadoop stack?
>

--20cf307abde9da048104a51c6fde--

From general-return-3627-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 12:25:11 2011
Return-Path: <general-return-3627-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E4B8C60A3
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue,  7 Jun 2011 12:25:10 +0000 (UTC)
Received: (qmail 47562 invoked by uid 500); 7 Jun 2011 12:25:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47460 invoked by uid 500); 7 Jun 2011 12:25:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47381 invoked by uid 99); 7 Jun 2011 12:25:08 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 12:25:07 +0000
Received: from localhost (HELO [10.0.0.77]) (127.0.0.1)
  (smtp-auth username gsingers, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 12:25:07 +0000
From: Grant Ingersoll <gsingers@apache.org>
Content-Type: multipart/alternative; boundary=Apple-Mail-17-577307994
Subject: [OT] 11 Days to Scale-A-Thon
Date: Tue, 7 Jun 2011 08:25:05 -0400
Message-Id: <B6D25042-55C0-426A-9322-4E575ABEBE2C@apache.org>
To: general@hadoop.apache.org,
 user@mahout.apache.org
Mime-Version: 1.0 (Apple Message framework v1084)
X-Mailer: Apple Mail (2.1084)

--Apple-Mail-17-577307994
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

Just a friendly reminder to everyone of the upcoming Scale-A-Thon event =
at Bronto Software.  We've got a great day planned of hands on =
programming with Apache Hadoop and other related tools like Apache =
HBase, Mahout, Hive, etc.  The event is June 18th at Bronto Software in =
Durham, NC. =20

Thanks in no small part to the generosity of several sponsors, the =
event, including lunch, dinner, drinks and compute time on Amazon EC2 =
will be completely free.  Our sponsors so far are Bronto Software, Booz =
Allen Hamilton, iContact, Open Source Integrators and Cloudera.  If you =
want to find out more, check out http://www.scaleathon.com or register =
at http://scaleathon-rtp.eventbrite.com/

Cheers,
Grant=

--Apple-Mail-17-577307994--

From general-return-3628-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 13:52:04 2011
Return-Path: <general-return-3628-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5CBB34F87
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  8 Jun 2011 13:52:04 +0000 (UTC)
Received: (qmail 32980 invoked by uid 500); 8 Jun 2011 13:52:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32890 invoked by uid 500); 8 Jun 2011 13:52:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32878 invoked by uid 99); 8 Jun 2011 13:52:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 13:52:02 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.18.222.48] (HELO smtp2.4emm.com) (69.18.222.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 13:51:55 +0000
Received: from EX2K7VS03.4emm.local ([192.168.160.203]) by HUB02.4emm.local
 ([192.168.161.133]) with mapi; Wed, 8 Jun 2011 09:51:33 -0400
From: Doug Meil <doug.meil@explorysmedical.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 8 Jun 2011 09:53:05 -0400
Subject: RE: LimitedPrivate and HBase (thoughts from an observer)
Thread-Topic: LimitedPrivate and HBase (thoughts from an observer)
Thread-Index: Acwl4Iw6RCuOjurXTV64oHfuXGbYYQ==
Message-ID: <67680900F79B1D4F99C844EE386FC5952823BED1E9@EX2K7VS03.4emm.local>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi there-

The following are some thoughts on questions raised in this thread that are=
 more on the Hadoop-core development process than this particular issue.  D=
isclosure:  I'm active on the HBase dist-list, so Hadoop-core folks can tak=
e my comments with a pinch or two of salt if required.=20

Re:  "What is the real criteria for changing an API from private to limited=
?"

I don't know, but from the perspective of an observer my request to Hadoop-=
core developers is to not to over-think this.=20

Re: "How "closely related" does a project need to be to get this privilege?=
" / " What is the criteria by which an API gets opened to something outside=
 of the Hadoop umbrella"

Given the context of the original question, is this debate really necessary=
?  Everybody knows that although HBase is a TLP now it grew out of Hadoop (=
e.g, there's a chapter about HBase in the Hadoop book, etc.)  It's not like=
 somebody from Hypertable was strong-arming for feature requests.

Re:  "If it was almost anyone else, it would have sat there.... and *that's=
* the point where I'm mainly concerned."

Hadoop-core development has been slow/stalled over the past 2 years, but re=
cent events such as Yahoo now backing the Apache distro are great signs tha=
t velocity will pick up and push forward.  Forward progress, even with item=
s as small as this, is good. =20

Re:  "Then we can go back working on core Hadoop."

Hadoop-core is critical to many frameworks in the Hadoop family (Hive, Pig,=
 and yes, HBase), but software frameworks are only good when utilized and s=
erving the needs of those who use them.  My request to Hadoop-core develope=
rs is to not assume that Core exists as an end to itself.

Thanks, and keep up the good work!

-----Original Message-----
From: Allen Wittenauer [mailto:aw@apache.org]=20
Sent: Monday, June 06, 2011 9:33 PM
To: general@hadoop.apache.org
Subject: Re: LimitedPrivate and HBase


On Jun 6, 2011, at 6:23 PM, Todd Lipcon wrote:
>=20
> Nah, I just think these "meta discussions" waste an awful lot of time=20
> that's better spent making real progress on the code, or reviewing the=20
> complex changes where extra eyes really make a big difference.


	OK.  That's make it easier to just -1 changes like this with reasoning suc=
h as "HBase is not a related project." Then we can go back working on core =
Hadoop.





From general-return-3629-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 16:14:08 2011
Return-Path: <general-return-3629-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C8AF74D1F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  8 Jun 2011 16:14:08 +0000 (UTC)
Received: (qmail 88001 invoked by uid 500); 8 Jun 2011 16:14:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87931 invoked by uid 500); 8 Jun 2011 16:14:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87923 invoked by uid 99); 8 Jun 2011 16:14:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 16:14:07 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [98.139.53.216] (HELO nm21-vm0.bullet.mail.ac4.yahoo.com) (98.139.53.216)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 08 Jun 2011 16:13:56 +0000
Received: from [98.139.52.194] by nm21.bullet.mail.ac4.yahoo.com with NNFMP; 08 Jun 2011 16:13:35 -0000
Received: from [98.139.52.182] by tm7.bullet.mail.ac4.yahoo.com with NNFMP; 08 Jun 2011 16:13:35 -0000
Received: from [127.0.0.1] by omp1065.mail.ac4.yahoo.com with NNFMP; 08 Jun 2011 16:13:35 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 422563.37668.bm@omp1065.mail.ac4.yahoo.com
Received: (qmail 26281 invoked by uid 60001); 8 Jun 2011 16:13:35 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1307549615; bh=NE1jwwWVw1pz7M0OHp+DI9CAETClFUoVu3LQfHegSB8=; h=Message-ID:X-YMail-OSG:Received:X-RocketYMMF:X-Mailer:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=vlOuYPYFkB+RqM3BtqMF/itiFkVLFnQiNBBoBG7UhtEXZxoKvFZmv217Y6UwRHXyCsq93x9QVmxDampuPHvfFejbPIGXSWalfh8GU1LPykfzg+LjUVMxsIw34zyI4TxpYYWg8C8F7WSYfb5GKRhlsFDlniNj7rASNACwKS2GDOw=
Message-ID: <206764.24240.qm@web65511.mail.ac4.yahoo.com>
X-YMail-OSG: CNJRXRsVM1lEPdcZp9GbDAfxhUo4pRTqEpRcYOfCU3YzAwD
 A5OBmps2.9FB_3ROEp5t9vltuQNLMP2BbEe7fDRneT2ImEFShQQH1M7WUAVV
 WYSZV5iHoC36IOpyegfKBG_u2zsuTmaov9s7QJf9p3CS8fPKVBTCrY9HV5v3
 sLsnYD0J8zLT1LxDF7CkP6zHPzwhuz5dLlTcXwGkgQisrVgQexuNCM7dIreb
 LGCQ6eBnCK2i4zdBJOBCpYro.dg3JSp8pFtY.GikzI7MfJrnoLp7TFOGStSe
 mn3.Wqci3HAjbW_JZ0blc.6XtuRu2U1CCbk3hm8J2FLL9aL9.s2.en8IBEoe
 AEOMgJXUmdrFUkAtHTZsiD7RorGaddlPmVlO0U1vOKHijHhJje8LuSYVe1uW
 zhFcYwztry6Iq_Q--
Received: from [213.61.227.39] by web65511.mail.ac4.yahoo.com via HTTP; Wed, 08 Jun 2011 09:13:34 PDT
X-RocketYMMF: apurtell
X-Mailer: YahooMailClassic/14.0.1 YahooMailWebService/0.8.111.303096
Date: Wed, 8 Jun 2011 09:13:34 -0700 (PDT)
From: Andrew Purtell <apurtell@apache.org>
Reply-To: apurtell@apache.org
Subject: Re: LimitedPrivate and HBase
To: general@hadoop.apache.org
In-Reply-To: <361B71C9-1216-4EBF-98DA-732AEB0C21E8@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

> I can tell you feel I'm picking on HBase, especially in light of my=0A> f=
lat out rejection of the "we want to mmap() blocks" case.=0A=0AI for one un=
derstand the objection there.=0A=0AAlthough it does negatively impact the w=
ork of a recent promising new contributor. As a project, HBase suffers for =
it. Of course that is no concern of HDFS. =0A=0AOn the other hand I do beli=
eve Todd has a point. MapReduce is perhaps the only constituency that HDFS =
really cares about. Any reasonable person would come to that conclusion aft=
er surveying submitted JIRAs and their resolution times (or not). Historica=
lly with HDFS the local itch, the concern of the big MapReduce shops, gets =
the scratch and others are of not much concern. Therefore there is unfortun=
ate business that lingers today -- Facebook, StumbleUpon, Trend Micro, and =
others have effectively forked HDFS (0.20) in house for use with HBase, and=
 nobody I know is seriously considering using HDFS 0.22 or TRUNK due to a l=
ack of evidence that anyone with a stake in it is running it in production =
at scale. Past discussion to mend the breach with an HBase-friendly release=
 of HDFS 0.20 ended with what I would describe as an inflexible and legalis=
tic air. =0A=0ABest regards,=0A=0A    - Andy=0A=0AProblems worthy of attack=
 prove their worth by hitting back. - Piet Hein (via Tom White)=0A=0A=0A---=
 On Mon, 6/6/11, Allen Wittenauer <aw@apache.org> wrote:=0A=0A> From: Allen=
 Wittenauer <aw@apache.org>=0A> Subject: Re: LimitedPrivate and HBase=0A> T=
o: general@hadoop.apache.org=0A> Date: Monday, June 6, 2011, 6:18 PM=0A> =
=0A> On Jun 6, 2011, at 6:08 PM, Todd Lipcon wrote:=0A> > =0A> >> =0A> >>=
=A0 =A0 =A0=A0=A0Let's face it: this=0A> happened because it was HBase.=A0 =
If it was almost=0A> >> anyone else, it would have sat there.... and=0A> *t=
hat's* the point where I'm=0A> >> mainly concerned.=0A> > =0A> > =0A> > If =
you want to feel better, take a look at HDFS-941,=0A> HDFS-347, and HDFS-91=
8=0A> > - these are patches that HBase has been asking for for=0A> nearly 2=
 years in=0A> > some cases and haven't gone in. Satisfied?=0A> =0A> =A0=A0=
=A0 These cases don't appear to be about=0A> re-classification of an API fr=
om private to=0A> semi-public.=A0 So no, I'm not.=A0 None of these=0A> appe=
ar to answer the base set of question:=0A> =0A> =A0=A0=A0 - What is the rea=
l criteria for changing=0A> an API from private to limited?=0A> =A0=A0=A0 -=
 How "closely related" does a project=0A> need to be to get this privilege?=
=0A> =0A> =A0=A0=A0 (Yes, I've read the classification=0A> docs.=A0 That's =
too vague.)=0A> =0A> =A0=A0=A0 I can tell you feel I'm picking on=0A> HBase=
, especially in light of my flat out rejection of the=0A> "we want to mmap(=
) blocks" case.=A0 But if this=0A> reclassification had been with anything =
else outside of the=0A> Hadoop project, I would have asked the same thing.=
=A0 It=0A> raises important questions that we as a project need to=0A> answ=
er.

From general-return-3630-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 16:17:40 2011
Return-Path: <general-return-3630-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7F8E04912
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  8 Jun 2011 16:17:40 +0000 (UTC)
Received: (qmail 94654 invoked by uid 500); 8 Jun 2011 16:17:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94596 invoked by uid 500); 8 Jun 2011 16:17:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94588 invoked by uid 99); 8 Jun 2011 16:17:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 16:17:38 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [98.139.53.210] (HELO nm18-vm0.bullet.mail.ac4.yahoo.com) (98.139.53.210)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 08 Jun 2011 16:17:28 +0000
Received: from [98.139.52.196] by nm18.bullet.mail.ac4.yahoo.com with NNFMP; 08 Jun 2011 16:17:07 -0000
Received: from [98.139.52.160] by tm9.bullet.mail.ac4.yahoo.com with NNFMP; 08 Jun 2011 16:17:07 -0000
Received: from [127.0.0.1] by omp1043.mail.ac4.yahoo.com with NNFMP; 08 Jun 2011 16:17:07 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 958764.61337.bm@omp1043.mail.ac4.yahoo.com
Received: (qmail 15251 invoked by uid 60001); 8 Jun 2011 16:17:07 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1307549827; bh=ShwbcuOZldccVS+Qn2fHK2qpIJ2H49CnwiHaGzwqfcA=; h=Message-ID:X-YMail-OSG:Received:X-RocketYMMF:X-Mailer:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=Ho5AHkpecKvoyLAcnbjj+t140rORWXY/LFRuotrQ8xvQlXocleM7LB6bc5gC1eaGsy6N0YA5J7qbN23Pk3hvFgHOsetSya+YiQVHZFoLqXSw33yb1EqvyxOOPFkY5AlLFVKghwTdXayzobrJntfZpnj+ZsMsVaWsh6Q8xZuQ73I=
Message-ID: <457011.690.qm@web65508.mail.ac4.yahoo.com>
X-YMail-OSG: Yg3B44IVM1nCZBvHWa1OTFXJG81ZDquQ6qhdISeF1nWIdJg
 Ms5yPs3jbFf76kePc2tmud8G995QLCkzGhitfiUzUnjcvx3vw25Ttp2p.vNl
 oUvXpgAhG.8HBjkHirgr0c6xHzldy.FyAvY2ZNKZEG4wgHK7_pZ0UV4pAAZh
 2VeEamxlMRobhrQ2bv5cJjx0Lf0_afc.x5dgQP85Z75jYg2MqSGkfsLBvoyO
 zg31teb.8KjAmpCgM_3QLLmWZ7vcWMNGad5T2x22nSrUCkyNerQKZQt6X_TS
 aHugzm3ZWoyvpeoeWG.apun_536vHZNrh5eY_.oo1KRWNx2T1XT2G3XHqn9n
 6wHh3I_jKcZiboIWt0q1GJZnQ9XpvAky_hxPOAdSLxf4HJUuLEQhzmPMhr3W
 LA2ZWQbQLmM_bVQ--
Received: from [213.61.227.39] by web65508.mail.ac4.yahoo.com via HTTP; Wed, 08 Jun 2011 09:17:07 PDT
X-RocketYMMF: apurtell
X-Mailer: YahooMailClassic/14.0.1 YahooMailWebService/0.8.111.303096
Date: Wed, 8 Jun 2011 09:17:07 -0700 (PDT)
From: Andrew Purtell <apurtell@apache.org>
Reply-To: apurtell@apache.org
Subject: Re: LimitedPrivate and HBase
To: general@hadoop.apache.org
In-Reply-To: <8A170F27-BA48-474A-905B-1F5B38D5F4FF@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

> From: Allen Wittenauer <aw@apache.org>=0A> =A0=A0=A0 OK.=A0 That's make i=
t easier to just=0A> -1 changes like this with reasoning such as "HBase is =
not a=0A> related project." Then we can go back working on core=0A> Hadoop.=
=0A=0ASeriously? =0A=0AForget what I said about filing a JIRA (and working =
on it) to give HttpServer an extensibility that possibly would past muster =
with you.=0A=0A  - Andy=0A=0A

From general-return-3631-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 16:37:23 2011
Return-Path: <general-return-3631-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 110604449
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  8 Jun 2011 16:37:23 +0000 (UTC)
Received: (qmail 42500 invoked by uid 500); 8 Jun 2011 16:37:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42442 invoked by uid 500); 8 Jun 2011 16:37:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42434 invoked by uid 99); 8 Jun 2011 16:37:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 16:37:21 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 16:37:11 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 52582B7E18
	for <general@hadoop.apache.org>; Wed,  8 Jun 2011 17:36:51 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id qXZHJwMORU-B for <general@hadoop.apache.org>;
	Wed,  8 Jun 2011 17:36:50 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 7D56CB7DE0
	for <general@hadoop.apache.org>; Wed,  8 Jun 2011 17:36:50 +0100 (BST)
MailScanner-NULL-Check: 1308155796.58054@sDmmwJz3646ibi7VH8TSkw
Received: from [16.25.175.86] (wildhaus.hpl.hp.com [16.25.175.86])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p58GaZRH010026
	for <general@hadoop.apache.org>; Wed, 8 Jun 2011 17:36:35 +0100 (BST)
Message-ID: <4DEFA513.1050807@apache.org>
Date: Wed, 08 Jun 2011 17:36:35 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110419 Red Hat/3.1.10-1.el6_0 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: LimitedPrivate and HBase
References: <206764.24240.qm@web65511.mail.ac4.yahoo.com>
In-Reply-To: <206764.24240.qm@web65511.mail.ac4.yahoo.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p58GaZRH010026
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 06/08/2011 05:13 PM, Andrew Purtell wrote:
>> I can tell you feel I'm picking on HBase, especially in light of my
>> flat out rejection of the "we want to mmap() blocks" case.
>
> I for one understand the objection there.
>
> Although it does negatively impact the work of a recent promising new contributor. As a project, HBase suffers for it. Of course that is no concern of HDFS.
>
> On the other hand I do believe Todd has a point. MapReduce is perhaps the only constituency that HDFS really cares about. Any reasonable person would come to that conclusion after surveying submitted JIRAs and their resolution times (or not). Historically with HDFS the local itch, the concern of the big MapReduce shops, gets the scratch and others are of not much concern. Therefore there is unfortunate business that lingers today -- Facebook, StumbleUpon, Trend Micro, and others have effectively forked HDFS (0.20) in house for use with HBase, and nobody I know is seriously considering using HDFS 0.22 or TRUNK due to a lack of evidence that anyone with a stake in it is running it in production at scale. Past discussion to mend the breach with an HBase-friendly release of HDFS 0.20 ended with what I would describe as an inflexible and legalistic air.
>

well, today MR is the primary constituency, but to be a stack you do 
have to make the otyher layers work. MR, with Hive and Pig on top, 
HBase, mahout.

These extra layers can form part of the regression tests for the 
underlying code: if a change breaks HBase or Hive, that's something to 
catch early, and say "this change to hadoop-common broke it".

yes, it's extra hassle dealing with changes that break things, but you 
find the problems so end users don't have to. And Jenkins can be set up 
to do much of the work, you just tweak the dependencies of the 
downstream projects to use the svn.trunk or -SNAPSHOT version of your 
code, run the builds in the right order to generate the artifacts, and 
wait for the emails to come in.

-steve

From general-return-3632-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 16:40:15 2011
Return-Path: <general-return-3632-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AC1124693
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  8 Jun 2011 16:40:15 +0000 (UTC)
Received: (qmail 53712 invoked by uid 500); 8 Jun 2011 16:40:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53659 invoked by uid 500); 8 Jun 2011 16:40:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53651 invoked by uid 99); 8 Jun 2011 16:40:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 16:40:14 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 16:40:06 +0000
Received: by bwz8 with SMTP id 8so916504bwz.35
        for <general@hadoop.apache.org>; Wed, 08 Jun 2011 09:39:46 -0700 (PDT)
Received: by 10.204.31.231 with SMTP id z39mr724259bkc.134.1307551186159; Wed,
 08 Jun 2011 09:39:46 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.75 with HTTP; Wed, 8 Jun 2011 09:39:26 -0700 (PDT)
In-Reply-To: <457011.690.qm@web65508.mail.ac4.yahoo.com>
References: <8A170F27-BA48-474A-905B-1F5B38D5F4FF@apache.org> <457011.690.qm@web65508.mail.ac4.yahoo.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Wed, 8 Jun 2011 09:39:26 -0700
Message-ID: <BANLkTi=dLr236bbupPJ+_Df1Ra3feo1f7Q@mail.gmail.com>
Subject: Re: LimitedPrivate and HBase
To: general@hadoop.apache.org, apurtell@apache.org
Content-Type: multipart/alternative; boundary=001485f7949afa09c704a535fb6f
X-Virus-Checked: Checked by ClamAV on apache.org

--001485f7949afa09c704a535fb6f
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Jun 8, 2011 at 9:17 AM, Andrew Purtell <apurtell@apache.org> wrote:

> > From: Allen Wittenauer <aw@apache.org>
> >     OK.  That's make it easier to just
> > -1 changes like this with reasoning such as "HBase is not a
> > related project." Then we can go back working on core
> > Hadoop.
>
> Seriously?
>
> Forget what I said about filing a JIRA (and working on it) to give
> HttpServer an extensibility that possibly would past muster with you.
>

Please know that Allen's opinions are not representative of the whole
community, and -1s without technical basis can be overridden.

I'll leave it at that to try to short circuit a potential flame war.

-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

--001485f7949afa09c704a535fb6f--

From general-return-3633-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 16:40:38 2011
Return-Path: <general-return-3633-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5F2C546DF
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  8 Jun 2011 16:40:38 +0000 (UTC)
Received: (qmail 56289 invoked by uid 500); 8 Jun 2011 16:40:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56204 invoked by uid 500); 8 Jun 2011 16:40:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56166 invoked by uid 99); 8 Jun 2011 16:40:32 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 16:40:32 +0000
Received: from localhost (HELO dhcp-02.private.iobm.com) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 16:40:32 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: LimitedPrivate and HBase (thoughts from an observer)
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <67680900F79B1D4F99C844EE386FC5952823BED1E9@EX2K7VS03.4emm.local>
Date: Wed, 8 Jun 2011 09:40:31 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <97A5F450-4DE0-4634-B1FF-0CAEBF6952E6@apache.org>
References: <67680900F79B1D4F99C844EE386FC5952823BED1E9@EX2K7VS03.4emm.local>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On Jun 8, 2011, at 6:53 AM, Doug Meil wrote:
>=20
> Re: "How "closely related" does a project need to be to get this =
privilege?" / " What is the criteria by which an API gets opened to =
something outside of the Hadoop umbrella"
>=20
> Given the context of the original question, is this debate really =
necessary?  Everybody knows that although HBase is a TLP now it grew out =
of Hadoop (e.g, there's a chapter about HBase in the Hadoop book, etc.)  =
It's not like somebody from Hypertable was strong-arming for feature =
requests.

	If HBase needs an API, why wouldn't something else?  Why should =
something be marked LimitedPrivate to HBase instead of just making it =
Public and being done with it?=20=

From general-return-3634-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 16:45:39 2011
Return-Path: <general-return-3634-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 64F714F54
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  8 Jun 2011 16:45:39 +0000 (UTC)
Received: (qmail 72457 invoked by uid 500); 8 Jun 2011 16:45:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72234 invoked by uid 500); 8 Jun 2011 16:45:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71302 invoked by uid 99); 8 Jun 2011 16:45:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 16:45:35 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of eli@cloudera.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 16:45:30 +0000
Received: by wyb40 with SMTP id 40so752997wyb.35
        for <general@hadoop.apache.org>; Wed, 08 Jun 2011 09:45:09 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.221.32 with SMTP id q32mr5065988wep.77.1307551509013; Wed,
 08 Jun 2011 09:45:09 -0700 (PDT)
Received: by 10.216.28.205 with HTTP; Wed, 8 Jun 2011 09:45:08 -0700 (PDT)
Date: Wed, 8 Jun 2011 09:45:08 -0700
Message-ID: <BANLkTimzn3xosAr-vwM9E5POPWex3W+vBg@mail.gmail.com>
Subject: Apache Hadoop Hackathon in LA June 22nd
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hey,

Thanks to everyone who came out for the Apache Hadoop Hackathons in
Palo Alto and San Francisco last month.  There have been a lot of
requests for a similar event in LA, so we're doing it!  The Hadoop
Hackathon in LA will take place on Wednesday June 22nd, right before
the LA HUG, hosted by Shopzilla. Cloudera's Aaron Myers (Hadoop
committer) and Eric Sammer (Solutions architect) will be on hand to
facilitate. Whether you're a sysadmin or devops and want to help build
and test Apache Hadoop or a developer looking to get their first patch
into Hadoop, this is a great chance to learn how it's done.

Sign up here: http://www.meetup.com/LA-HUG/events/21380931 and we'll
see you there!

Thanks,
Eli

From general-return-3635-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 17:42:13 2011
Return-Path: <general-return-3635-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A5D804B59
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  8 Jun 2011 17:42:13 +0000 (UTC)
Received: (qmail 21272 invoked by uid 500); 8 Jun 2011 17:42:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21219 invoked by uid 500); 8 Jun 2011 17:42:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21208 invoked by uid 99); 8 Jun 2011 17:42:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 17:42:11 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 17:42:05 +0000
Received: from SP1-EX07CAS02.ds.corp.yahoo.com (sp1-ex07cas02.ds.corp.yahoo.com [216.252.116.138])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p58HfPdK069074
	for <general@hadoop.apache.org>; Wed, 8 Jun 2011 10:41:25 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1307554885;
	bh=S3agYwXtrVKfOTdqgtPD808wbXNmHvrYf6LCRf4Ns7k=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:MIME-Version;
	b=TqNmwD6rG+/mRniPrqz+gHMt7VFMCvQpSLZevG2XzjvEAmTqHCtMZdbq8ZRhUuGXm
	 Ln9xPS14spOSBOZbWAt9xhZUDko4o3759mjzwPffxRPF/2VONlxz7oZxBipfpf8O4s
	 SzoL7OFn9ncf/fEUD+T7BHmrDJtLlxUbghvadej4=
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 SP1-EX07CAS02.ds.corp.yahoo.com ([216.252.116.167]) with mapi; Wed, 8 Jun
 2011 10:41:25 -0700
From: Suresh Srinivas <sureshms@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 8 Jun 2011 10:41:23 -0700
Subject: Re: LimitedPrivate and HBase (thoughts from an observer)
Thread-Topic: LimitedPrivate and HBase (thoughts from an observer)
Thread-Index: Acwl+uBXxu572p+PR2aE/V3+3LIDaAACGgpe
Message-ID: <CA150253.1ADC9%sureshms@yahoo-inc.com>
In-Reply-To: <97A5F450-4DE0-4634-B1FF-0CAEBF6952E6@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-Entourage/13.8.0.101117
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

I do not see any issue with the change that Todd has made. We have done
similar changes in HDFS-1586 in the past.

Making APIs public comes with a cost. That is what we are avoiding with
LimitedPrivate. The intention was to include the following projects that ar=
e
closely tied to Hadoop as projects eligible for LimitedPrivate.
{"HBase", "HDFS", "Hive", "MapReduce", "Pig"}. This list could grow in the
future.

When such projects break because of API change, we can co-ordinate as
community and fix the issues. This is not true for some application that we
do not know of breaks!

If others, outside the umbrella of these projects need an API, they could
open a jira and we could address it.


On 6/8/11 9:40 AM, "Allen Wittenauer" <aw@apache.org> wrote:

>=20
> On Jun 8, 2011, at 6:53 AM, Doug Meil wrote:
>>=20
>> Re: "How "closely related" does a project need to be to get this privile=
ge?"
>> / " What is the criteria by which an API gets opened to something outsid=
e of
>> the Hadoop umbrella"
>>=20
>> Given the context of the original question, is this debate really necess=
ary?
>> Everybody knows that although HBase is a TLP now it grew out of Hadoop (=
e.g,
>> there's a chapter about HBase in the Hadoop book, etc.)  It's not like
>> somebody from Hypertable was strong-arming for feature requests.
>=20
> If HBase needs an API, why wouldn't something else?  Why should something=
 be
> marked LimitedPrivate to HBase instead of just making it Public and being=
 done
> with it?=20


From general-return-3636-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 17:47:58 2011
Return-Path: <general-return-3636-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C55A14316
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  8 Jun 2011 17:47:58 +0000 (UTC)
Received: (qmail 36918 invoked by uid 500); 8 Jun 2011 17:47:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36866 invoked by uid 500); 8 Jun 2011 17:47:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36849 invoked by uid 99); 8 Jun 2011 17:47:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 17:47:57 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 17:47:52 +0000
Received: by fxm7 with SMTP id 7so779623fxm.35
        for <general@hadoop.apache.org>; Wed, 08 Jun 2011 10:47:31 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=jTI57tHLGjP8xv1+Ea9Y6aje42xFlTu0V5MXoIposu8=;
        b=Rlx1f5b6cVY3FIUWhdhbvcmJVzbRmh0yXK6H1r47+D8UtMUlW2Qfk1qYXbOAdTNGG8
         PuXNk+v4pSqwbOUOFHGb5PW+XR2CMFWg6bQ0ALOR2erIaZivX7zk/UTZr9Owm1+6Mv/d
         vlqy6LGn4rlrgjLKV+Ks9q/PkfBJZIIg32DJs=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=U/Ul+7l/cR1WhANa907y80iODp/GbaXRQ3PupbYDIKH8vuWPPyGyoah1BZDYKvHaBT
         owyTWrejuGvGexsxdyyQWr56xNqjeHtiJ5gpqlFR7bxRMPw4akT3U7PUGSG8g5ZM1DFl
         bpy2KdwzmBE9u+Mwzo1y6u6kf4wVRn3zH69Yc=
MIME-Version: 1.0
Received: by 10.223.62.146 with SMTP id x18mr1426843fah.54.1307555251282; Wed,
 08 Jun 2011 10:47:31 -0700 (PDT)
Received: by 10.223.113.19 with HTTP; Wed, 8 Jun 2011 10:47:31 -0700 (PDT)
In-Reply-To: <CA150253.1ADC9%sureshms@yahoo-inc.com>
References: <97A5F450-4DE0-4634-B1FF-0CAEBF6952E6@apache.org>
	<CA150253.1ADC9%sureshms@yahoo-inc.com>
Date: Wed, 8 Jun 2011 10:47:31 -0700
Message-ID: <BANLkTi=6Np5=1UWZqMbceKVdic8yUpjhww@mail.gmail.com>
Subject: Re: LimitedPrivate and HBase (thoughts from an observer)
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015173fe6c646e4e904a536eeaf

--0015173fe6c646e4e904a536eeaf
Content-Type: text/plain; charset=ISO-8859-1

I too think that LimitedPrivate is a good idea for projects that work
closely with the Hadoop ecosystem (Hive, HBase, MR, etc) It allows us to
experiment with an API, that if proved useful in the longer run, can
graduate to be a public API in future.

Some people may rightly claim that this gives unfair advantage to projects
in the Hadoop ecosystem vs projects that are outside of this system, but I
see no harm in that. One reason for this is that there are many developers
who work on multiple of these projects, and it is easier to coordinate
changes among these projects.

thanks,
-dhruba

On Wed, Jun 8, 2011 at 10:41 AM, Suresh Srinivas <sureshms@yahoo-inc.com>wrote:

> I do not see any issue with the change that Todd has made. We have done
> similar changes in HDFS-1586 in the past.
>
> Making APIs public comes with a cost. That is what we are avoiding with
> LimitedPrivate. The intention was to include the following projects that
> are
> closely tied to Hadoop as projects eligible for LimitedPrivate.
> {"HBase", "HDFS", "Hive", "MapReduce", "Pig"}. This list could grow in the
> future.
>
> When such projects break because of API change, we can co-ordinate as
> community and fix the issues. This is not true for some application that we
> do not know of breaks!
>
> If others, outside the umbrella of these projects need an API, they could
> open a jira and we could address it.
>
>
> On 6/8/11 9:40 AM, "Allen Wittenauer" <aw@apache.org> wrote:
>
> >
> > On Jun 8, 2011, at 6:53 AM, Doug Meil wrote:
> >>
> >> Re: "How "closely related" does a project need to be to get this
> privilege?"
> >> / " What is the criteria by which an API gets opened to something
> outside of
> >> the Hadoop umbrella"
> >>
> >> Given the context of the original question, is this debate really
> necessary?
> >> Everybody knows that although HBase is a TLP now it grew out of Hadoop
> (e.g,
> >> there's a chapter about HBase in the Hadoop book, etc.)  It's not like
> >> somebody from Hypertable was strong-arming for feature requests.
> >
> > If HBase needs an API, why wouldn't something else?  Why should something
> be
> > marked LimitedPrivate to HBase instead of just making it Public and being
> done
> > with it?
>
>


-- 
Connect to me at http://www.facebook.com/dhruba

--0015173fe6c646e4e904a536eeaf--

From general-return-3637-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 17:52:49 2011
Return-Path: <general-return-3637-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F3DF84AA8
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  8 Jun 2011 17:52:48 +0000 (UTC)
Received: (qmail 57410 invoked by uid 500); 8 Jun 2011 17:52:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57364 invoked by uid 500); 8 Jun 2011 17:52:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57356 invoked by uid 99); 8 Jun 2011 17:52:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 17:52:47 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 17:52:39 +0000
Received: from sp1-ex07cas03.ds.corp.yahoo.com (sp1-ex07cas03.ds.corp.yahoo.com [216.252.116.151])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p58HpXMY025837
	for <general@hadoop.apache.org>; Wed, 8 Jun 2011 10:51:33 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1307555493;
	bh=42yVFyODW+S1ixaRaqDoFm5Zram9Mu/XHxxr93JRT1M=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:MIME-Version;
	b=FA0Fdo5015WVbAqjRXGpV/+XeoOsfJ3BFAys0qGMs3i3SNk2j9YabSGb0eMdJcJx8
	 hUSM7YyHhJSBH7JWzKrHVnm7Sp12jPrSkhjQl3MJPcEZHMsAWl0GtPUt/pYAy7u+5E
	 ISfoHuRQhnmzN3QKWq5UQNzVDr8Udn3+N73G7to0=
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas03.ds.corp.yahoo.com ([216.252.116.151]) with mapi; Wed, 8 Jun
 2011 10:51:32 -0700
From: Suresh Srinivas <sureshms@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 8 Jun 2011 10:51:30 -0700
Subject: Re: LimitedPrivate and HBase (thoughts from an observer)
Thread-Topic: LimitedPrivate and HBase (thoughts from an observer)
Thread-Index: AcwmBEoahMXSBHxuTNqSe8Li/9AJZwAAGgyc
Message-ID: <CA1504B2.1ADD0%sureshms@yahoo-inc.com>
In-Reply-To: <BANLkTi=6Np5=1UWZqMbceKVdic8yUpjhww@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
user-agent: Microsoft-Entourage/13.8.0.101117
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

BTW, thank you Todd and Stack for all your effort in making changes to HDFS
to make it work well with HBase. These changes are very important in making
HDFS better!


On 6/8/11 10:47 AM, "Dhruba Borthakur" <dhruba@gmail.com> wrote:

> I too think that LimitedPrivate is a good idea for projects that work
> closely with the Hadoop ecosystem (Hive, HBase, MR, etc) It allows us to
> experiment with an API, that if proved useful in the longer run, can
> graduate to be a public API in future.
>=20
> Some people may rightly claim that this gives unfair advantage to project=
s
> in the Hadoop ecosystem vs projects that are outside of this system, but =
I
> see no harm in that. One reason for this is that there are many developer=
s
> who work on multiple of these projects, and it is easier to coordinate
> changes among these projects.
>=20
> thanks,
> -dhruba
>=20
> On Wed, Jun 8, 2011 at 10:41 AM, Suresh Srinivas
> <sureshms@yahoo-inc.com>wrote:
>=20
>> I do not see any issue with the change that Todd has made. We have done
>> similar changes in HDFS-1586 in the past.
>>=20
>> Making APIs public comes with a cost. That is what we are avoiding with
>> LimitedPrivate. The intention was to include the following projects that
>> are
>> closely tied to Hadoop as projects eligible for LimitedPrivate.
>> {"HBase", "HDFS", "Hive", "MapReduce", "Pig"}. This list could grow in t=
he
>> future.
>>=20
>> When such projects break because of API change, we can co-ordinate as
>> community and fix the issues. This is not true for some application that=
 we
>> do not know of breaks!
>>=20
>> If others, outside the umbrella of these projects need an API, they coul=
d
>> open a jira and we could address it.
>>=20
>>=20
>> On 6/8/11 9:40 AM, "Allen Wittenauer" <aw@apache.org> wrote:
>>=20
>>>=20
>>> On Jun 8, 2011, at 6:53 AM, Doug Meil wrote:
>>>>=20
>>>> Re: "How "closely related" does a project need to be to get this
>> privilege?"
>>>> / " What is the criteria by which an API gets opened to something
>> outside of
>>>> the Hadoop umbrella"
>>>>=20
>>>> Given the context of the original question, is this debate really
>> necessary?
>>>> Everybody knows that although HBase is a TLP now it grew out of Hadoop
>> (e.g,
>>>> there's a chapter about HBase in the Hadoop book, etc.)  It's not like
>>>> somebody from Hypertable was strong-arming for feature requests.
>>>=20
>>> If HBase needs an API, why wouldn't something else?  Why should somethi=
ng
>> be
>>> marked LimitedPrivate to HBase instead of just making it Public and bei=
ng
>> done
>>> with it?
>>=20
>>=20
>=20


From general-return-3638-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 17:53:27 2011
Return-Path: <general-return-3638-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E1D174B2A
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  8 Jun 2011 17:53:27 +0000 (UTC)
Received: (qmail 63686 invoked by uid 500); 8 Jun 2011 17:53:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63634 invoked by uid 500); 8 Jun 2011 17:53:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63626 invoked by uid 99); 8 Jun 2011 17:53:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 17:53:26 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of eli@cloudera.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 17:53:21 +0000
Received: by wyb40 with SMTP id 40so829934wyb.35
        for <general@hadoop.apache.org>; Wed, 08 Jun 2011 10:53:00 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.79.11 with SMTP id h11mr7381194wee.77.1307555580312; Wed,
 08 Jun 2011 10:53:00 -0700 (PDT)
Received: by 10.216.28.205 with HTTP; Wed, 8 Jun 2011 10:53:00 -0700 (PDT)
In-Reply-To: <BANLkTi=6Np5=1UWZqMbceKVdic8yUpjhww@mail.gmail.com>
References: <97A5F450-4DE0-4634-B1FF-0CAEBF6952E6@apache.org>
	<CA150253.1ADC9%sureshms@yahoo-inc.com>
	<BANLkTi=6Np5=1UWZqMbceKVdic8yUpjhww@mail.gmail.com>
Date: Wed, 8 Jun 2011 10:53:00 -0700
Message-ID: <BANLkTimHCKOjNbwse9brxKLVHk=r9prhVw@mail.gmail.com>
Subject: Re: LimitedPrivate and HBase (thoughts from an observer)
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Agree.  HDFS is not a general purpose file system. Its API was, and
continues to be, co-designed with it's directly adjacent projects (MR,
HBase, Hive, Pig etc) in mind. IMO it's one of it's biggest strengths.

This is of course does not mean we get to ignore existing users,
compatibility concerns, etc.

On Wed, Jun 8, 2011 at 10:47 AM, Dhruba Borthakur <dhruba@gmail.com> wrote:
> I too think that LimitedPrivate is a good idea for projects that work
> closely with the Hadoop ecosystem (Hive, HBase, MR, etc) It allows us to
> experiment with an API, that if proved useful in the longer run, can
> graduate to be a public API in future.
>
> Some people may rightly claim that this gives unfair advantage to project=
s
> in the Hadoop ecosystem vs projects that are outside of this system, but =
I
> see no harm in that. One reason for this is that there are many developer=
s
> who work on multiple of these projects, and it is easier to coordinate
> changes among these projects.
>
> thanks,
> -dhruba
>
> On Wed, Jun 8, 2011 at 10:41 AM, Suresh Srinivas <sureshms@yahoo-inc.com>=
wrote:
>
>> I do not see any issue with the change that Todd has made. We have done
>> similar changes in HDFS-1586 in the past.
>>
>> Making APIs public comes with a cost. That is what we are avoiding with
>> LimitedPrivate. The intention was to include the following projects that
>> are
>> closely tied to Hadoop as projects eligible for LimitedPrivate.
>> {"HBase", "HDFS", "Hive", "MapReduce", "Pig"}. This list could grow in t=
he
>> future.
>>
>> When such projects break because of API change, we can co-ordinate as
>> community and fix the issues. This is not true for some application that=
 we
>> do not know of breaks!
>>
>> If others, outside the umbrella of these projects need an API, they coul=
d
>> open a jira and we could address it.
>>
>>
>> On 6/8/11 9:40 AM, "Allen Wittenauer" <aw@apache.org> wrote:
>>
>> >
>> > On Jun 8, 2011, at 6:53 AM, Doug Meil wrote:
>> >>
>> >> Re: "How "closely related" does a project need to be to get this
>> privilege?"
>> >> / " What is the criteria by which an API gets opened to something
>> outside of
>> >> the Hadoop umbrella"
>> >>
>> >> Given the context of the original question, is this debate really
>> necessary?
>> >> Everybody knows that although HBase is a TLP now it grew out of Hadoo=
p
>> (e.g,
>> >> there's a chapter about HBase in the Hadoop book, etc.) =A0It's not l=
ike
>> >> somebody from Hypertable was strong-arming for feature requests.
>> >
>> > If HBase needs an API, why wouldn't something else? =A0Why should some=
thing
>> be
>> > marked LimitedPrivate to HBase instead of just making it Public and be=
ing
>> done
>> > with it?
>>
>>
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
>

From general-return-3639-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 11:43:11 2011
Return-Path: <general-return-3639-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8BA83630F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  9 Jun 2011 11:43:11 +0000 (UTC)
Received: (qmail 55013 invoked by uid 500); 9 Jun 2011 11:43:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54950 invoked by uid 500); 9 Jun 2011 11:43:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54942 invoked by uid 99); 9 Jun 2011 11:43:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 11:43:09 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 11:43:00 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 654941BA955
	for <general@hadoop.apache.org>; Thu,  9 Jun 2011 12:42:39 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id W8zJ6Y3HfWKN for <general@hadoop.apache.org>;
	Thu,  9 Jun 2011 12:42:38 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 3787E1BA943
	for <general@hadoop.apache.org>; Thu,  9 Jun 2011 12:42:37 +0100 (BST)
MailScanner-NULL-Check: 1308224544.0206@wkDPCuSw5Ri/jnqUajibrQ
Received: from [16.25.175.86] (wildhaus.hpl.hp.com [16.25.175.86])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p59BgLIX006595
	for <general@hadoop.apache.org>; Thu, 9 Jun 2011 12:42:21 +0100 (BST)
Message-ID: <4DF0B19D.6050203@apache.org>
Date: Thu, 09 Jun 2011 12:42:21 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110419 Red Hat/3.1.10-1.el6_0 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: LimitedPrivate and HBase (thoughts from the build and test world)
References: <CA150253.1ADC9%sureshms@yahoo-inc.com>
In-Reply-To: <CA150253.1ADC9%sureshms@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p59BgLIX006595
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 06/08/2011 06:41 PM, Suresh Srinivas wrote:
> I do not see any issue with the change that Todd has made. We have done
> similar changes in HDFS-1586 in the past.
>
> Making APIs public comes with a cost. That is what we are avoiding with
> LimitedPrivate. The intention was to include the following projects that are
> closely tied to Hadoop as projects eligible for LimitedPrivate.
> {"HBase", "HDFS", "Hive", "MapReduce", "Pig"}. This list could grow in the
> future.

I'm going to talk about my experience on the Ant team.

One of the lessons of that project is that in the open source world, you 
can't predict how your code gets used, or control it. If someone wants 
to take your app and use it as a library -they can. If someone wants to 
do something completely unexpected with that library -they can. And this 
is a good thing, because your code gets used. Yes, you get new bugreps, 
but every person using your code is someone not using somebody elses 
code. You win.

The other lesson from that is the following: in open source, there is no 
such thing as private code.

* If you mark something as package scoped, they just inject their 
classes into your package (and who hasn't done that with their Hadoop 
extensions?).
* If you mark something as protected, they subclass and open up its 
privacy.
* If you mark something as private, they edit your source and create a 
new JAR with the relaxed permission

for any of these actions, you end up fielding the bugreps, as the stack 
trace points to you. And it increases maintenance costs for everyone.


Alternatively they cut and paste your code into their codebase, possibly 
-but not always- retaining the apache credits.

That
  * complicates copyright and lawsuits:
  http://www.theserverside.com/news/thread.tss?thread_id=29958

  * increases maintenance costs for everyone, especially if there are 
security issues with the original code.

> When such projects break because of API change, we can co-ordinate as
> community and fix the issues. This is not true for some application that we
> do not know of breaks!

The way Ant handled this with Gump, the nightly clean build of all the 
OSS Java projects built with Ant
http://vmgump.apache.org/gump/public/

For all the projects, they thought they were getting a free CI build 
run, but what it really was was a regression test of Ant and every 
single OSS project. If a change in Ant broke anyone's build: we noticed. 
If a change in Log4J broke a build, someone noticed. It became a 
rapid-response regression test for the entire OSS suite.

Sadly, it doesn't work so well. I'd blame Maven, but the move to ivy 
dependencies doesn't help either, it complicates classpaths no end.

Even so, the idea is great: build and test your downstream applications, 
and the things you depend on, so you find problems within 24 hours of 
the change being committed -regardless of which project committed the 
change.

The way to do it now would be with Jenkins, not just building and 
testing Hadooop-{core, hdfs, mapreduce}, but
  -building and publishing every upstream dependency.
  -test against the trunk versions build locally.
  -build and test against the ivy-versioned artifacts that are 
controlled by the version.properties

Together this flags up when something works against the old artifacts, 
but doesn't work against the trunk versions: that's their regressions, 
caught early.

Downstream
  -build and test the OSS projects that work with Hadoop.
  That's the apache ones: HBase, Mahout, Pig, Hive, Hama etc, and the 
other ones, such as Cascading.

That can be offered as a service to these projects "we will build and 
test your code against our trunk", a service designed to benefit 
everyone. They find their bugs, we find regressions.

This is a pretty complex project, especially when you think about the 
challenge of testing your RPM generation code will install the RPMs (I 
bring up clean CentOS VMs for such a purpose), but without it you don't 
get everything working together, which is the state things appear to be 
in today.

Ignoring the RPM install & test problems, if people are interested in 
working on this, we should be able to do a lot of it on Jenkins. Who is 
willing to get involved?

-Steve

From general-return-3640-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 14:17:44 2011
Return-Path: <general-return-3640-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 41B8269B7
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  9 Jun 2011 14:17:44 +0000 (UTC)
Received: (qmail 23068 invoked by uid 500); 9 Jun 2011 14:17:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22969 invoked by uid 500); 9 Jun 2011 14:17:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22961 invoked by uid 99); 9 Jun 2011 14:17:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 14:17:42 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mlortiz@uci.cu designates 200.55.140.180 as permitted sender)
Received: from [200.55.140.180] (HELO mx3.uci.cu) (200.55.140.180)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 09 Jun 2011 14:17:36 +0000
Received: (qmail 12284 invoked by uid 507); 9 Jun 2011 14:17:08 -0000
Received: from 10.0.0.185 by ns3.uci.cu (envelope-from <mlortiz@uci.cu>, uid 501) with qmail-scanner-2.01st 
 (avp: 5.0.2.0. spamassassin: 3.0.6. perlscan: 2.01st.  
 Clear:RC:1(10.0.0.185):. 
 Processed in 0.655884 secs); 09 Jun 2011 14:17:08 -0000
Received: from unknown (HELO ucimail4.uci.cu) (10.0.0.185)
  by 0 with SMTP; 9 Jun 2011 14:17:07 -0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by ucimail4.uci.cu (Postfix) with ESMTP id E43541324569
	for <general@hadoop.apache.org>; Thu,  9 Jun 2011 10:17:07 -0400 (CDT)
X-Virus-Scanned: amavisd-new at uci.cu
Received: from ucimail4.uci.cu ([127.0.0.1])
	by localhost (ucimail4.uci.cu [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id 2n71AYSstLvb; Thu,  9 Jun 2011 10:17:06 -0400 (CDT)
Received: from [10.36.18.57] (marcosluis-aspire-5251.uci.cu [10.36.18.57])
	by ucimail4.uci.cu (Postfix) with ESMTP id CB78813245B3
	for <general@hadoop.apache.org>; Thu,  9 Jun 2011 10:17:06 -0400 (CDT)
Message-ID: <4DF0DD55.4000201@uci.cu>
Date: Thu, 09 Jun 2011 10:18:53 -0430
From: Marcos Ortiz <mlortiz@uci.cu>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110424 Lightning/1.0b2 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Process to release files signature
References: <BANLkTikt0dw2wQCZBaMKemf1G2Sd3uMqnQ@mail.gmail.com> <BANLkTikdtAiyS0bURMvy1_EUPK+RMAN9fQ@mail.gmail.com> <BANLkTik8bt9033ue787evT_ip2gfbUY-9Q@mail.gmail.com> <BANLkTi=pn2nk11EVcxq_jKqJCyZUWD7YWA@mail.gmail.com>
In-Reply-To: <BANLkTi=pn2nk11EVcxq_jKqJCyZUWD7YWA@mail.gmail.com>
Content-Type: multipart/alternative;
 boundary="------------080803030401040702050300"
X-Virus-Checked: Checked by ClamAV on apache.org

--------------080803030401040702050300
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit

Regards to all the list
Which is the process that you follow to sign the releases files with Hadoop?
md5, sha1?

Thanks a lot for your time

-- 
Marcos Luís Ortíz Valmaseda
  Software Engineer (UCI)
  http://marcosluis2186.posterous.com
  http://twitter.com/marcosluis2186



--------------080803030401040702050300--

From general-return-3641-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 16:12:59 2011
Return-Path: <general-return-3641-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7196C6FCD
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  9 Jun 2011 16:12:59 +0000 (UTC)
Received: (qmail 82949 invoked by uid 500); 9 Jun 2011 16:12:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82807 invoked by uid 500); 9 Jun 2011 16:12:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82799 invoked by uid 99); 9 Jun 2011 16:12:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 16:12:57 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of owen.omalley@gmail.com designates 209.85.210.48 as permitted sender)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 16:12:50 +0000
Received: by pzk10 with SMTP id 10so1113391pzk.35
        for <general@hadoop.apache.org>; Thu, 09 Jun 2011 09:12:30 -0700 (PDT)
Received: by 10.68.27.71 with SMTP id r7mr346060pbg.385.1307635950009;
        Thu, 09 Jun 2011 09:12:30 -0700 (PDT)
Received: from battlerock-lm.corp.yahoo.com (nat-dip4.cfw-a-gci.corp.yahoo.com [209.131.62.113])
        by mx.google.com with ESMTPS id y2sm1512065pbi.83.2011.06.09.09.12.28
        (version=SSLv3 cipher=OTHER);
        Thu, 09 Jun 2011 09:12:28 -0700 (PDT)
From: Owen O'Malley <omalley@apache.org>
Mime-Version: 1.0 (Apple Message framework v1084)
Content-Type: multipart/alternative; boundary=Apple-Mail-15-763749744
Subject: Re: Process to release files signature
Date: Thu, 9 Jun 2011 09:12:27 -0700
In-Reply-To: <4DF0DD55.4000201@uci.cu>
To: general@hadoop.apache.org
References: <BANLkTikt0dw2wQCZBaMKemf1G2Sd3uMqnQ@mail.gmail.com> <BANLkTikdtAiyS0bURMvy1_EUPK+RMAN9fQ@mail.gmail.com> <BANLkTik8bt9033ue787evT_ip2gfbUY-9Q@mail.gmail.com> <BANLkTi=pn2nk11EVcxq_jKqJCyZUWD7YWA@mail.gmail.com> <4DF0DD55.4000201@uci.cu>
Message-Id: <B4776F69-9CAB-493A-AB11-E02EFAFA2FF9@apache.org>
X-Mailer: Apple Mail (2.1084)

--Apple-Mail-15-763749744
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii


On Jun 9, 2011, at 7:48 AM, Marcos Ortiz wrote:

> Regards to all the list
> Which is the process that you follow to sign the releases files with =
Hadoop?
> md5, sha1?

Both. *grin*

As documented in: http://wiki.apache.org/hadoop/HowToRelease

we use "gpg --print-mds"

-- Owen=

--Apple-Mail-15-763749744--

From general-return-3642-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 16:29:56 2011
Return-Path: <general-return-3642-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2A8DA6DCE
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  9 Jun 2011 16:29:56 +0000 (UTC)
Received: (qmail 40557 invoked by uid 500); 9 Jun 2011 16:29:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40501 invoked by uid 500); 9 Jun 2011 16:29:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40489 invoked by uid 99); 9 Jun 2011 16:29:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 16:29:54 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 16:29:48 +0000
Received: from [10.0.1.3] (snvvpn4-10-72-169-c11.hq.corp.yahoo.com [10.72.169.11])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p59GSPf7057414;
	Thu, 9 Jun 2011 09:28:26 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1307636907;
	bh=CBsF6Q0Q/1SMP7laBe9w6lFbLJgr81igl2AZKbHHUAw=;
	h=Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:
	 Content-Transfer-Encoding:Message-Id:References:To;
	b=CCmuOp8WMIAncfhOtDZgop5/HAtndAEjvNvCMSkTkrj7D8vPV0Cdks9RD6IkmaHnM
	 UDkEfzdVJk7b0xGF3Ze5EDGguoFf/nIbJiiFzsyYQiNgYAGbR/rSMuBTxmscZI7qf9
	 TDq/0o8AhdrGKkshQbUDpDK3tdmanByYcuttOoJU=
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: LimitedPrivate and HBase
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <457011.690.qm@web65508.mail.ac4.yahoo.com>
Date: Thu, 9 Jun 2011 09:28:25 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <F0CA2186-6E63-459D-ABFF-21863EE018E5@yahoo-inc.com>
References: <457011.690.qm@web65508.mail.ac4.yahoo.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>,
        "apurtell@apache.org" <apurtell@apache.org>
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

I'd like to see a proposal circulated to handle this concern in a =
maintainable way. =20

On Jun 8, 2011, at 9:17 AM, Andrew Purtell wrote:

>> From: Allen Wittenauer <aw@apache.org>
>>     OK.  That's make it easier to just
>> -1 changes like this with reasoning such as "HBase is not a
>> related project." Then we can go back working on core
>> Hadoop.
>=20
> Seriously?=20
>=20
> Forget what I said about filing a JIRA (and working on it) to give =
HttpServer an extensibility that possibly would past muster with you.
>=20
>  - Andy
>=20
>=20


From general-return-3643-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 16:45:32 2011
Return-Path: <general-return-3643-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9F8EE6DD5
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  9 Jun 2011 16:45:32 +0000 (UTC)
Received: (qmail 96320 invoked by uid 500); 9 Jun 2011 16:45:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96157 invoked by uid 500); 9 Jun 2011 16:45:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96148 invoked by uid 99); 9 Jun 2011 16:45:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 16:45:31 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of owen.omalley@gmail.com designates 209.85.210.48 as permitted sender)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 16:45:23 +0000
Received: by pzk10 with SMTP id 10so1135052pzk.35
        for <general@hadoop.apache.org>; Thu, 09 Jun 2011 09:45:03 -0700 (PDT)
Received: by 10.68.65.75 with SMTP id v11mr481934pbs.56.1307637903140;
        Thu, 09 Jun 2011 09:45:03 -0700 (PDT)
Received: from battlerock-lm.corp.yahoo.com (nat-dip4.cfw-a-gci.corp.yahoo.com [209.131.62.113])
        by mx.google.com with ESMTPS id p5sm1539288pbk.36.2011.06.09.09.45.01
        (version=SSLv3 cipher=OTHER);
        Thu, 09 Jun 2011 09:45:01 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: Process to release files signature
From: Owen O'Malley <omalley@apache.org>
In-Reply-To: <4DF0DD55.4000201@uci.cu>
Date: Thu, 9 Jun 2011 09:45:00 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <876CA43B-0C59-420E-BAF0-E79C75301FDF@apache.org>
References: <BANLkTikt0dw2wQCZBaMKemf1G2Sd3uMqnQ@mail.gmail.com> <BANLkTikdtAiyS0bURMvy1_EUPK+RMAN9fQ@mail.gmail.com> <BANLkTik8bt9033ue787evT_ip2gfbUY-9Q@mail.gmail.com> <BANLkTi=pn2nk11EVcxq_jKqJCyZUWD7YWA@mail.gmail.com> <4DF0DD55.4000201@uci.cu>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)


On Jun 9, 2011, at 7:48 AM, Marcos Ortiz wrote:

> Regards to all the list
> Which is the process that you follow to sign the releases files with =
Hadoop?
> md5, sha1?

I should also say that we sign our releases with gpg using the private =
key of the release manager.

-- Owen=

From general-return-3644-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 17:28:04 2011
Return-Path: <general-return-3644-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D4B4D6DEC
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  9 Jun 2011 17:28:04 +0000 (UTC)
Received: (qmail 87156 invoked by uid 500); 9 Jun 2011 17:28:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87093 invoked by uid 500); 9 Jun 2011 17:28:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87085 invoked by uid 99); 9 Jun 2011 17:28:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 17:28:03 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of tom.e.white@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 17:27:58 +0000
Received: by wyb40 with SMTP id 40so1979681wyb.35
        for <general@hadoop.apache.org>; Thu, 09 Jun 2011 10:27:37 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:cc:content-type:content-transfer-encoding;
        bh=Cwiz6Ud506SsaqyBtbHzVDeffJsIC84On4aKIngCQaM=;
        b=BylvJdDJAIWPHWK34WKwNk1em4f3ZflAoc3h/cfbjK8Xwndc40+l3WstbCt09CHbzM
         KglpTbJ/t7AknF5c8NwI/8HE7HWJmg382YkjJKxyN7jSfoMjfnhVQaFh0hdqNLVKKagV
         LGTg24c1mQEyN2BMI/tRJO9aM2Cm6uWpjAEYk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :cc:content-type:content-transfer-encoding;
        b=juP85ZCwqq0cCw2O0fmbRddN5Lx76lOQNK94w7l8LNr++cKp1/OOuzxD6acpS4Y52y
         5a0OoVFeByGToUQWWoeHBfwJyFJ/TmZpWYu07eg5rWE2RYf3UK1zW0ePkg4oxcx7JKN6
         ycTU49ccYo1Jijr13NXk8WGqj1y0rLIJOmekg=
Received: by 10.216.229.3 with SMTP id g3mr6639375weq.91.1307640457186; Thu,
 09 Jun 2011 10:27:37 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.3.21 with HTTP; Thu, 9 Jun 2011 10:27:14 -0700 (PDT)
In-Reply-To: <F0CA2186-6E63-459D-ABFF-21863EE018E5@yahoo-inc.com>
References: <457011.690.qm@web65508.mail.ac4.yahoo.com> <F0CA2186-6E63-459D-ABFF-21863EE018E5@yahoo-inc.com>
From: Tom White <tom.e.white@gmail.com>
Date: Thu, 9 Jun 2011 10:27:14 -0700
Message-ID: <BANLkTikYuO0nRHz3UWZFZU245dMw82Y08Q@mail.gmail.com>
Subject: Re: LimitedPrivate and HBase
To: general@hadoop.apache.org
Cc: "apurtell@apache.org" <apurtell@apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Looking at current usage in Hadoop, there are only 4 LimitedPrivate
references to HBase (the http, io.retry, ipc, and metrics packages in
Common), and 2 references to Pig (the two LineRecordReader classes in
MapReduce). The other LimitedPrivate references are all to HDFS or
MapReduce. Given that Private means "Intended for use only within
Hadoop itself" (according to the javadoc), we can replace these
references with Private.

We could also change the remaining 6 cases of LimitedPrivate to Public
(note that they are already annotated Evolving or Unstable), and
deprecate LimitedPrivate. Would this allay people's concerns?

Cheers,
Tom

On Thu, Jun 9, 2011 at 9:28 AM, Eric Baldeschwieler
<eric14@yahoo-inc.com> wrote:
> I'd like to see a proposal circulated to handle this concern in a maintai=
nable way.
>
> On Jun 8, 2011, at 9:17 AM, Andrew Purtell wrote:
>
>>> From: Allen Wittenauer <aw@apache.org>
>>> =A0 =A0 OK. =A0That's make it easier to just
>>> -1 changes like this with reasoning such as "HBase is not a
>>> related project." Then we can go back working on core
>>> Hadoop.
>>
>> Seriously?
>>
>> Forget what I said about filing a JIRA (and working on it) to give HttpS=
erver an extensibility that possibly would past muster with you.
>>
>> =A0- Andy
>>
>>
>
>

From general-return-3645-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 17:34:21 2011
Return-Path: <general-return-3645-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 784826318
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  9 Jun 2011 17:34:21 +0000 (UTC)
Received: (qmail 98127 invoked by uid 500); 9 Jun 2011 17:34:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97920 invoked by uid 500); 9 Jun 2011 17:34:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97912 invoked by uid 99); 9 Jun 2011 17:34:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 17:34:19 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 17:34:11 +0000
Received: from sp1-ex07cas03.ds.corp.yahoo.com (sp1-ex07cas03.ds.corp.yahoo.com [216.252.116.151])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p59HXIc4082312
	for <general@hadoop.apache.org>; Thu, 9 Jun 2011 10:33:18 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1307640799;
	bh=NZ4ClEuGqv6yHYB7TOdeVsrlse0akQNFmeqwDNKhceg=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:MIME-Version;
	b=e3XiVfgrG+h+bsW6eP80T6Xd+156XMrKIKt+Icrv4lUABurszViO8D8A3S4qKrQhK
	 SubNeER0KH1pLenqw7eQHFTzDe4+24UTWcN7+DvcsgOOwBCU9SjonKoSiCmpmjjGtz
	 1AJ3fd61a/1F/qDXqnj3ZvgreNCnwniRAuvkpEpo=
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas03.ds.corp.yahoo.com ([216.252.116.151]) with mapi; Thu, 9 Jun
 2011 10:33:18 -0700
From: Suresh Srinivas <sureshms@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 9 Jun 2011 10:33:15 -0700
Subject: Re: LimitedPrivate and HBase (thoughts from the build and test
 world)
Thread-Topic: LimitedPrivate and HBase (thoughts from the build and test
 world)
Thread-Index: AcwmmoMt68A03sMaSpiXE+dWTj8GTQAMMzeQ
Message-ID: <CA1651EB.1BDAE%sureshms@yahoo-inc.com>
In-Reply-To: <4DF0B19D.6050203@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-Entourage/13.8.0.101117
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


> The other lesson from that is the following: in open source, there is no
> such thing as private code.

The goal of InterfaceAudiencme and InterfaceStability is not to prevent som=
e
one from using the code. It merely suggests who the interface is intended
for and its stability.

An interface marked Public and Stable guarantees backward compatibility.
These are intended for every one to use. Changes to these interfaces must b=
e
done extra carefully to ensure this.

One can still use LimitedPrivate/Private or Unstable/Evolving interfaces
outside. But these interfaces can change freely, in non backward compatible
way. The interface might even be deleted in future releases. Any one using
it, do it at their own risk of seeing their code break and having to change
their code as the interface evolves.

Regards,
Suresh


From general-return-3646-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 17:38:40 2011
Return-Path: <general-return-3646-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 61D69654B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  9 Jun 2011 17:38:40 +0000 (UTC)
Received: (qmail 1720 invoked by uid 500); 9 Jun 2011 17:38:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1652 invoked by uid 500); 9 Jun 2011 17:38:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1644 invoked by uid 99); 9 Jun 2011 17:38:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 17:38:38 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 17:38:32 +0000
Received: by pzk10 with SMTP id 10so1167851pzk.35
        for <general@hadoop.apache.org>; Thu, 09 Jun 2011 10:38:12 -0700 (PDT)
Received: by 10.68.54.71 with SMTP id h7mr496657pbp.39.1307641092544;
        Thu, 09 Jun 2011 10:38:12 -0700 (PDT)
Received: from tp (c-24-4-185-157.hsd1.ca.comcast.net [24.4.185.157])
        by mx.google.com with ESMTPS id i7sm1568168pbj.74.2011.06.09.10.38.11
        (version=SSLv3 cipher=OTHER);
        Thu, 09 Jun 2011 10:38:11 -0700 (PDT)
Sender: Konstantin Boudnik <cos@boudnik.org>
Date: Thu, 9 Jun 2011 10:38:08 -0700
From: Konstantin Boudnik <cos@apache.org>
To: general@hadoop.apache.org
Cc: "apurtell@apache.org" <apurtell@apache.org>
Subject: Re: LimitedPrivate and HBase
Message-ID: <20110609173808.GE3079@tp>
References: <457011.690.qm@web65508.mail.ac4.yahoo.com>
 <F0CA2186-6E63-459D-ABFF-21863EE018E5@yahoo-inc.com>
 <BANLkTikYuO0nRHz3UWZFZU245dMw82Y08Q@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature"; boundary="YToU2i3Vx8H2dn7O"
Content-Disposition: inline
In-Reply-To: <BANLkTikYuO0nRHz3UWZFZU245dMw82Y08Q@mail.gmail.com>
X-Organization: It's something of 'Cos
X-PGP-Key: http://www.boudnik.org/~cos/pubkey.asc
User-Agent: Mutt/1.5.20 (2009-06-14)

--YToU2i3Vx8H2dn7O
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

+1 on this. I think LimitedPrivate is somewhat moot. It seems
to be a better way to just get rid of it. $0.02

Cos

On Thu, Jun 09, 2011 at 10:27AM, Tom White wrote:
> Looking at current usage in Hadoop, there are only 4 LimitedPrivate
> references to HBase (the http, io.retry, ipc, and metrics packages in
> Common), and 2 references to Pig (the two LineRecordReader classes in
> MapReduce). The other LimitedPrivate references are all to HDFS or
> MapReduce. Given that Private means "Intended for use only within
> Hadoop itself" (according to the javadoc), we can replace these
> references with Private.
>=20
> We could also change the remaining 6 cases of LimitedPrivate to Public
> (note that they are already annotated Evolving or Unstable), and
> deprecate LimitedPrivate. Would this allay people's concerns?
>=20
> Cheers,
> Tom
>=20
> On Thu, Jun 9, 2011 at 9:28 AM, Eric Baldeschwieler
> <eric14@yahoo-inc.com> wrote:
> > I'd like to see a proposal circulated to handle this concern in a maint=
ainable way.
> >
> > On Jun 8, 2011, at 9:17 AM, Andrew Purtell wrote:
> >
> >>> From: Allen Wittenauer <aw@apache.org>
> >>> =E2=95=90 =E2=95=90 OK. =E2=95=90That's make it easier to just
> >>> -1 changes like this with reasoning such as "HBase is not a
> >>> related project." Then we can go back working on core
> >>> Hadoop.
> >>
> >> Seriously?
> >>
> >> Forget what I said about filing a JIRA (and working on it) to give Htt=
pServer an extensibility that possibly would past muster with you.
> >>
> >> =E2=95=90- Andy
> >>
> >>
> >
> >

--YToU2i3Vx8H2dn7O
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iF4EAREIAAYFAk3xBQAACgkQenyFlstYjhL79gD/T9ZC88aHiuK79dENswLWak24
slZDDansJl+IH7/tQYgBAK+0puL7Yw66jib/lPg3lZg/z436HgCjX7qCKW7Z0qJF
=MeXN
-----END PGP SIGNATURE-----

--YToU2i3Vx8H2dn7O--

From general-return-3647-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 17:57:12 2011
Return-Path: <general-return-3647-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 224046E8B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  9 Jun 2011 17:57:12 +0000 (UTC)
Received: (qmail 28510 invoked by uid 500); 9 Jun 2011 17:57:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28463 invoked by uid 500); 9 Jun 2011 17:57:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28455 invoked by uid 99); 9 Jun 2011 17:57:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 17:57:10 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 17:57:02 +0000
Received: from sp1-ex07cas01.ds.corp.yahoo.com (sp1-ex07cas01.ds.corp.yahoo.com [216.252.116.137])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p59HuNXg033094
	for <general@hadoop.apache.org>; Thu, 9 Jun 2011 10:56:23 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1307642183;
	bh=3W6g6KOb65dkFmttcieE9vZ6cI72jTmifR0lHKsQk9w=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:MIME-Version;
	b=I10zhrjxD9zHuiUb2CWjhqS2xhJCrA77TsxsrRB4NYOPWdXWDXH+7BjZVpCGKfC9i
	 hREi/PMpx1h3PB2hNUyZdnY/BFDr1ExlDlxIkEQ3QAviFAOq9ZrNHQN3n+0uPe+2r+
	 W0CLLo1WwLEKzU8z+evRQxJMQzul9BSXp8H+WBQM=
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas01.ds.corp.yahoo.com ([216.252.116.137]) with mapi; Thu, 9 Jun
 2011 10:56:22 -0700
From: Suresh Srinivas <sureshms@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 9 Jun 2011 10:56:20 -0700
Subject: Re: LimitedPrivate and HBase
Thread-Topic: LimitedPrivate and HBase
Thread-Index: AcwmyqrTmLVTcbKMTFyrOvkSsDRCBAAA968S
Message-ID: <CA165754.1BDBC%sureshms@yahoo-inc.com>
In-Reply-To: <BANLkTikYuO0nRHz3UWZFZU245dMw82Y08Q@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-Entourage/13.8.0.101117
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Javadoc did not capture the intent well. Please see HADOOP-5073. We should
fix the Javadoc to avoid confusion.

Even though there are only few instances of LimitedPrivate, I prefer
retaining. I prefer to keep MiniDFSCluster LimitedPrivate and not support a=
s
as public interface.


On 6/9/11 10:27 AM, "Tom White" <tom.e.white@gmail.com> wrote:

> Looking at current usage in Hadoop, there are only 4 LimitedPrivate
> references to HBase (the http, io.retry, ipc, and metrics packages in
> Common), and 2 references to Pig (the two LineRecordReader classes in
> MapReduce). The other LimitedPrivate references are all to HDFS or
> MapReduce. Given that Private means "Intended for use only within
> Hadoop itself" (according to the javadoc), we can replace these
> references with Private.
>=20
> We could also change the remaining 6 cases of LimitedPrivate to Public
> (note that they are already annotated Evolving or Unstable), and
> deprecate LimitedPrivate. Would this allay people's concerns?
>=20
> Cheers,
> Tom
>=20
> On Thu, Jun 9, 2011 at 9:28 AM, Eric Baldeschwieler
> <eric14@yahoo-inc.com> wrote:
>> I'd like to see a proposal circulated to handle this concern in a
>> maintainable way.
>>=20
>> On Jun 8, 2011, at 9:17 AM, Andrew Purtell wrote:
>>=20
>>>> From: Allen Wittenauer <aw@apache.org>
>>>> =A0 =A0 OK. =A0That's make it easier to just
>>>> -1 changes like this with reasoning such as "HBase is not a
>>>> related project." Then we can go back working on core
>>>> Hadoop.
>>>=20
>>> Seriously?
>>>=20
>>> Forget what I said about filing a JIRA (and working on it) to give
>>> HttpServer an extensibility that possibly would past muster with you.
>>>=20
>>> =A0- Andy
>>>=20
>>>=20
>>=20
>>=20


From general-return-3648-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 18:03:14 2011
Return-Path: <general-return-3648-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 03E736F4E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  9 Jun 2011 18:03:14 +0000 (UTC)
Received: (qmail 45798 invoked by uid 500); 9 Jun 2011 18:03:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45744 invoked by uid 500); 9 Jun 2011 18:03:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45736 invoked by uid 99); 9 Jun 2011 18:03:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 18:03:11 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 18:03:04 +0000
Received: by pwi16 with SMTP id 16so1186698pwi.35
        for <general@hadoop.apache.org>; Thu, 09 Jun 2011 11:02:43 -0700 (PDT)
Received: by 10.68.29.130 with SMTP id k2mr447329pbh.514.1307642562943;
        Thu, 09 Jun 2011 11:02:42 -0700 (PDT)
Received: from tp (c-24-4-185-157.hsd1.ca.comcast.net [24.4.185.157])
        by mx.google.com with ESMTPS id o2sm1585033pbj.65.2011.06.09.11.02.41
        (version=SSLv3 cipher=OTHER);
        Thu, 09 Jun 2011 11:02:42 -0700 (PDT)
Sender: Konstantin Boudnik <cos@boudnik.org>
Date: Thu, 9 Jun 2011 11:02:39 -0700
From: Konstantin Boudnik <cos@apache.org>
To: general@hadoop.apache.org
Subject: Re: LimitedPrivate and HBase
Message-ID: <20110609180239.GH3079@tp>
References: <BANLkTikYuO0nRHz3UWZFZU245dMw82Y08Q@mail.gmail.com>
 <CA165754.1BDBC%sureshms@yahoo-inc.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature"; boundary="reI/iBAAp9kzkmX4"
Content-Disposition: inline
In-Reply-To: <CA165754.1BDBC%sureshms@yahoo-inc.com>
X-Organization: It's something of 'Cos
X-PGP-Key: http://www.boudnik.org/~cos/pubkey.asc
User-Agent: Mutt/1.5.20 (2009-06-14)
X-Virus-Checked: Checked by ClamAV on apache.org

--reI/iBAAp9kzkmX4
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Thu, Jun 09, 2011 at 10:56AM, Suresh Srinivas wrote:
> Javadoc did not capture the intent well. Please see HADOOP-5073. We should
> fix the Javadoc to avoid confusion.
>=20
> Even though there are only few instances of LimitedPrivate, I prefer
> retaining. I prefer to keep MiniDFSCluster LimitedPrivate and not support=
 as
> as public interface.

Suresh, but MiniDFSCluster is a test facility with a somewhat limited
functionality. It isn't that important really if it has LimitedPrivate or
Private annotation.

Cos

> On 6/9/11 10:27 AM, "Tom White" <tom.e.white@gmail.com> wrote:
>=20
> > Looking at current usage in Hadoop, there are only 4 LimitedPrivate
> > references to HBase (the http, io.retry, ipc, and metrics packages in
> > Common), and 2 references to Pig (the two LineRecordReader classes in
> > MapReduce). The other LimitedPrivate references are all to HDFS or
> > MapReduce. Given that Private means "Intended for use only within
> > Hadoop itself" (according to the javadoc), we can replace these
> > references with Private.
> >=20
> > We could also change the remaining 6 cases of LimitedPrivate to Public
> > (note that they are already annotated Evolving or Unstable), and
> > deprecate LimitedPrivate. Would this allay people's concerns?
> >=20
> > Cheers,
> > Tom
> >=20
> > On Thu, Jun 9, 2011 at 9:28 AM, Eric Baldeschwieler
> > <eric14@yahoo-inc.com> wrote:
> >> I'd like to see a proposal circulated to handle this concern in a
> >> maintainable way.
> >>=20
> >> On Jun 8, 2011, at 9:17 AM, Andrew Purtell wrote:
> >>=20
> >>>> From: Allen Wittenauer <aw@apache.org>
> >>>> =E2=95=90 =E2=95=90 OK. =E2=95=90That's make it easier to just
> >>>> -1 changes like this with reasoning such as "HBase is not a
> >>>> related project." Then we can go back working on core
> >>>> Hadoop.
> >>>=20
> >>> Seriously?
> >>>=20
> >>> Forget what I said about filing a JIRA (and working on it) to give
> >>> HttpServer an extensibility that possibly would past muster with you.
> >>>=20
> >>> =E2=95=90- Andy
> >>>=20
> >>>=20
> >>=20
> >>=20
>=20

--reI/iBAAp9kzkmX4
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iF4EAREIAAYFAk3xCr4ACgkQenyFlstYjhIpOQEAumim7q3tFcn2Bp7v15oI3nth
7hychA0ZbG5N9cUYr/sA/2p39HMYchYT/eCgF4EA1Kw8G5Ey8yP9Kg9nWC06mNR1
=QLfC
-----END PGP SIGNATURE-----

--reI/iBAAp9kzkmX4--

From general-return-3649-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 18:20:48 2011
Return-Path: <general-return-3649-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D97E66113
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  9 Jun 2011 18:20:48 +0000 (UTC)
Received: (qmail 73842 invoked by uid 500); 9 Jun 2011 18:20:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73776 invoked by uid 500); 9 Jun 2011 18:20:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73768 invoked by uid 99); 9 Jun 2011 18:20:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 18:20:47 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 209.85.216.169 as permitted sender)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 18:20:40 +0000
Received: by qyk2 with SMTP id 2so3219941qyk.14
        for <general@hadoop.apache.org>; Thu, 09 Jun 2011 11:20:20 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=fcRF/10J3DEOxI/UiN6cak0q8O8hJHQb1FjHSmwKluw=;
        b=D3NuSNFB2yNXNVYkH22LSKKfzUQjKl6lkOm5cMLINvwDZNBbtL0a0QCmGelrnstXxZ
         FfcTq3UN0GGmYgLkaqyUmSsGCdEkESN7vMiIWreYzYJa1tCtze3+zIl0BhtvD6PmKwRk
         gsoS31KzCz4jcJbSbsMkQXLx13vwp6BVFA09M=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=Kj61tk38eBNOVMc/Lm5mlfWf6DKsmLXcbp8wKMpfmBxNBJEFw5Ql+q2j/Sk4xZtMDs
         cX0MwGRr+ujjcVHBYApruT4JxwEZYCjlx69Xtl4kJxu423CRGGlWFAg7faFRxik0u5wD
         M8huhFvvBMZYyGB2VzNJ0kY1ibMkXJtMbtvKA=
MIME-Version: 1.0
Received: by 10.224.183.204 with SMTP id ch12mr765474qab.341.1307643620169;
 Thu, 09 Jun 2011 11:20:20 -0700 (PDT)
Sender: saint.ack@gmail.com
Received: by 10.224.19.135 with HTTP; Thu, 9 Jun 2011 11:20:20 -0700 (PDT)
In-Reply-To: <08E3D059-20B7-49F4-AA53-C1012C9480A1@apache.org>
References: <41B787A2-3FF6-42D6-87E2-38DCBC7174F4@apache.org>
	<BANLkTineoX-pPK+Z3DzEb3qNC7FjxbvjVw@mail.gmail.com>
	<08E3D059-20B7-49F4-AA53-C1012C9480A1@apache.org>
Date: Thu, 9 Jun 2011 11:20:20 -0700
X-Google-Sender-Auth: IOqY9ji8PXifwOgniXxTTlZvCIc
Message-ID: <BANLkTi=LKvtXM30J-NezWhE+jZkXabhONg@mail.gmail.com>
Subject: Re: LimitedPrivate and HBase
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Mon, Jun 6, 2011 at 1:45 PM, Allen Wittenauer <aw@apache.org> wrote:
>
> On Jun 6, 2011, at 11:34 AM, Stack wrote:
>
>> On Mon, Jun 6, 2011 at 9:45 AM, Allen Wittenauer <aw@apache.org> wrote:
>>>
>>>
>>> =A0 =A0 =A0 =A0I have some concerns over the recent usage of LimitedPri=
vate being opened up to HBase. =A0Shouldn't HBase really be sticking to pub=
lic APIs rather than poking through some holes? =A0If HBase needs an API, w=
ouldn't other clients as well?
>>>
>>
>> HBase uses public APIs. =A0A method we relied on went from protected to
>> private. =A0Fixing this brought on the flagging of HttpServer with
>> LimitedPrivate (HttpServer, the class flagged, is mostly internal to
>> Hadoop but HBase, because, in part, of its subproject provenance,
>> extends HttpServer providing its UI reusing Hadoop's log level, thread
>> dumping, etc., servlets)
>
> =A0 =A0 =A0 =A0So, no, HBase does not actually use public APIs....
>

The move from protected to private was an error rectified by HADOOP-7351.
St.Ack

From general-return-3650-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 18:31:05 2011
Return-Path: <general-return-3650-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 91E966B13
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  9 Jun 2011 18:31:05 +0000 (UTC)
Received: (qmail 96064 invoked by uid 500); 9 Jun 2011 18:31:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96015 invoked by uid 500); 9 Jun 2011 18:31:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96007 invoked by uid 99); 9 Jun 2011 18:31:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 18:31:04 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 209.85.216.176 as permitted sender)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 18:30:56 +0000
Received: by qyk30 with SMTP id 30so1221615qyk.14
        for <general@hadoop.apache.org>; Thu, 09 Jun 2011 11:30:35 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=uwahV+K6UoAB1bKXKPyp6LC9avMlRAs9FRL54InE0AA=;
        b=mzX83ZVtyUahxv3p9o+2hx9my27zf41OUqpfq7Rn6x36dum8lhVnrQ0fNcWiKYbwNE
         hlCE29lqiJIbHgNb6NfO3s3BppJJGRuz3ZX/vYVh4JlMPOOTiNeb6AF5Twg/rMX34sFz
         NqufRf4PYLommZep5Ii344BkIV1kJtv+K+1pQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=THffPYkyAskr2WcNHNnvlhE7ImhY1n3/U4KBDNw4ud8swry7loQMEhjN6H6s1kW6ej
         CzTydZK0DfjzIJYnySNP+gii9mDdzsm35lYXnzJYnPXqfxRTCj+EDd9Jp+NVv2eN3PQL
         7TwTBjZWhzKQRtr+ZB6Nfzw695Hq3TLFTLcj0=
MIME-Version: 1.0
Received: by 10.224.209.198 with SMTP id gh6mr782227qab.391.1307644235501;
 Thu, 09 Jun 2011 11:30:35 -0700 (PDT)
Sender: saint.ack@gmail.com
Received: by 10.224.19.135 with HTTP; Thu, 9 Jun 2011 11:30:35 -0700 (PDT)
In-Reply-To: <4AFD3D6A-9556-4343-91B0-6CA794A0192F@apache.org>
References: <580675.9102.qm@web65506.mail.ac4.yahoo.com>
	<4AFD3D6A-9556-4343-91B0-6CA794A0192F@apache.org>
Date: Thu, 9 Jun 2011 11:30:35 -0700
X-Google-Sender-Auth: CAt9QNpRmUq5EFJrs7hXb9JdQzk
Message-ID: <BANLkTiktEcW82V9WAdTTO=4pkFFVc5-4+w@mail.gmail.com>
Subject: Re: LimitedPrivate and HBase
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Mon, Jun 6, 2011 at 4:34 PM, Allen Wittenauer <aw@apache.org> wrote:
> On Jun 6, 2011, at 4:22 PM, Andrew Purtell wrote:
>>
>> Perhaps opening a jira for a cleaner framework for HttpServer extension =
could be useful?
>
> =A0 =A0 =A0 =A0Sure. =A0That's probably what should have happened to begi=
n with rather than the quickly changing the API to a different classificati=
on.

It was done in HADOOP-2024 back in 2007.

St.Ack

From general-return-3651-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 19:23:55 2011
Return-Path: <general-return-3651-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0F5836FD8
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  9 Jun 2011 19:23:55 +0000 (UTC)
Received: (qmail 91251 invoked by uid 500); 9 Jun 2011 19:23:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91182 invoked by uid 500); 9 Jun 2011 19:23:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91172 invoked by uid 99); 9 Jun 2011 19:23:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 19:23:53 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 19:23:48 +0000
Received: by qwj9 with SMTP id 9so1245365qwj.35
        for <general@hadoop.apache.org>; Thu, 09 Jun 2011 12:23:27 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=oOdRUul+urVVzDXceTvRACXE264ZnhwefigCD+wDZ80=;
        b=Xv1Hi5nkp32iYynM31a/Zeos/vE1MUehIlEHbN5F73QMqJS+Itqn2tE3NkdALeryJP
         binXRPMIWH/oPvdQx6YKxc3T1g7B2T7OXVyUjQye/AkOUWYFN5X6RtmiJQYWdlfO3EwP
         yls4xl4o9tdtPG/dfDU9aE2bkV5r8qfJnQukc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=a5y+q8jduSdmGZtHrAZtABlCK/TTi06tSEHy8Mo9rQvIaT7NBm7oqNgzZq6oDqqKNz
         qUpCtntX3W19QLbuhy8wmpXJr+HXo2Z51/0lDpIpnDqsGJa2BP1OHPCW/gwlMpVxfLPU
         HucUp+NNMIGX0lidvYWpXvPSX5U0QRXQiCKnc=
MIME-Version: 1.0
Received: by 10.224.183.204 with SMTP id ch12mr821298qab.341.1307647407154;
 Thu, 09 Jun 2011 12:23:27 -0700 (PDT)
Sender: saint.ack@gmail.com
Received: by 10.224.19.135 with HTTP; Thu, 9 Jun 2011 12:23:27 -0700 (PDT)
In-Reply-To: <4DF0B19D.6050203@apache.org>
References: <CA150253.1ADC9%sureshms@yahoo-inc.com>
	<4DF0B19D.6050203@apache.org>
Date: Thu, 9 Jun 2011 12:23:27 -0700
X-Google-Sender-Auth: jgR5Px8hMY5GVMgqJjm2OmricvI
Message-ID: <BANLkTinyMTJyB8-N+sEHEKNbziB5Y7C9bQ@mail.gmail.com>
Subject: Re: LimitedPrivate and HBase (thoughts from the build and test world)
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Nice reality check and thanks for the how it was addressed elsewhere Steve.

As you say, it sounds like a large undertaking but it would be a sweet
service for the downstreamers.

St.Ack

On Thu, Jun 9, 2011 at 4:42 AM, Steve Loughran <stevel@apache.org> wrote:
> On 06/08/2011 06:41 PM, Suresh Srinivas wrote:
>>
>> I do not see any issue with the change that Todd has made. We have done
>> similar changes in HDFS-1586 in the past.
>>
>> Making APIs public comes with a cost. That is what we are avoiding with
>> LimitedPrivate. The intention was to include the following projects that
>> are
>> closely tied to Hadoop as projects eligible for LimitedPrivate.
>> {"HBase", "HDFS", "Hive", "MapReduce", "Pig"}. This list could grow in t=
he
>> future.
>
> I'm going to talk about my experience on the Ant team.
>
> One of the lessons of that project is that in the open source world, you
> can't predict how your code gets used, or control it. If someone wants to
> take your app and use it as a library -they can. If someone wants to do
> something completely unexpected with that library -they can. And this is =
a
> good thing, because your code gets used. Yes, you get new bugreps, but ev=
ery
> person using your code is someone not using somebody elses code. You win.
>
> The other lesson from that is the following: in open source, there is no
> such thing as private code.
>
> * If you mark something as package scoped, they just inject their classes
> into your package (and who hasn't done that with their Hadoop extensions?=
).
> * If you mark something as protected, they subclass and open up its priva=
cy.
> * If you mark something as private, they edit your source and create a ne=
w
> JAR with the relaxed permission
>
> for any of these actions, you end up fielding the bugreps, as the stack
> trace points to you. And it increases maintenance costs for everyone.
>
>
> Alternatively they cut and paste your code into their codebase, possibly
> -but not always- retaining the apache credits.
>
> That
> =A0* complicates copyright and lawsuits:
> =A0http://www.theserverside.com/news/thread.tss?thread_id=3D29958
>
> =A0* increases maintenance costs for everyone, especially if there are
> security issues with the original code.
>
>> When such projects break because of API change, we can co-ordinate as
>> community and fix the issues. This is not true for some application that
>> we
>> do not know of breaks!
>
> The way Ant handled this with Gump, the nightly clean build of all the OS=
S
> Java projects built with Ant
> http://vmgump.apache.org/gump/public/
>
> For all the projects, they thought they were getting a free CI build run,
> but what it really was was a regression test of Ant and every single OSS
> project. If a change in Ant broke anyone's build: we noticed. If a change=
 in
> Log4J broke a build, someone noticed. It became a rapid-response regressi=
on
> test for the entire OSS suite.
>
> Sadly, it doesn't work so well. I'd blame Maven, but the move to ivy
> dependencies doesn't help either, it complicates classpaths no end.
>
> Even so, the idea is great: build and test your downstream applications, =
and
> the things you depend on, so you find problems within 24 hours of the cha=
nge
> being committed -regardless of which project committed the change.
>
> The way to do it now would be with Jenkins, not just building and testing
> Hadooop-{core, hdfs, mapreduce}, but
> =A0-building and publishing every upstream dependency.
> =A0-test against the trunk versions build locally.
> =A0-build and test against the ivy-versioned artifacts that are controlle=
d by
> the version.properties
>
> Together this flags up when something works against the old artifacts, bu=
t
> doesn't work against the trunk versions: that's their regressions, caught
> early.
>
> Downstream
> =A0-build and test the OSS projects that work with Hadoop.
> =A0That's the apache ones: HBase, Mahout, Pig, Hive, Hama etc, and the ot=
her
> ones, such as Cascading.
>
> That can be offered as a service to these projects "we will build and tes=
t
> your code against our trunk", a service designed to benefit everyone. The=
y
> find their bugs, we find regressions.
>
> This is a pretty complex project, especially when you think about the
> challenge of testing your RPM generation code will install the RPMs (I br=
ing
> up clean CentOS VMs for such a purpose), but without it you don't get
> everything working together, which is the state things appear to be in
> today.
>
> Ignoring the RPM install & test problems, if people are interested in
> working on this, we should be able to do a lot of it on Jenkins. Who is
> willing to get involved?
>
> -Steve
>

From general-return-3652-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 02:22:04 2011
Return-Path: <general-return-3652-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5FF024AFE
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 10 Jun 2011 02:22:04 +0000 (UTC)
Received: (qmail 88595 invoked by uid 500); 10 Jun 2011 02:22:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88534 invoked by uid 500); 10 Jun 2011 02:22:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88524 invoked by uid 99); 10 Jun 2011 02:22:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 02:22:02 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mbhandarkar@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 02:21:58 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:In-Reply-To:Accept-Language:
   Content-Language:X-MS-Has-Attach:X-MS-TNEF-Correlator:
   x-originating-ip:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version;
  b=qfSYkOsauICQMnSVmvkbJW/Ib0GgGunM043uz+02M9qHWndu8ADkRuyU
   ded1wa6fRrtPWxnNNvtZvokHCHKH60AsIq+ZHKprxzi12Ryf75dMaK0kf
   dorFgX7BeaoXmk+;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=mbhandarkar@linkedin.com; q=dns/txt;
  s=proddkim; t=1307672518; x=1339208518;
  h=from:to:subject:date:message-id:in-reply-to:content-id:
   content-transfer-encoding:mime-version;
  bh=TderFz0i7Hhu04tHEVhsnq9vQf0hOize4JDlplEk170=;
  b=sD7PHYHoqzisXpmqyU0Y8nNfefhL5AyHnwdAgcXxDF1Gw6rBAlX80vgi
   p89S7uRg7O7ffk5c1JnzP6VJl64wBAomk2nYjnOqBZlCIHet5ZeQYTDrG
   AmUquyYTNTV08WU;
X-IronPort-AV: E=Sophos;i="4.65,344,1304319600"; 
   d="scan'208";a="24109196"
Received: from ESV4-EXC02.linkedin.biz ([fe80::4d74:48bd:e0bd:13ee]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi id 14.01.0251.000; Thu, 9
 Jun 2011 19:21:37 -0700
From: Milind Bhandarkar <mbhandarkar@linkedin.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: LimitedPrivate and HBase (thoughts from the build and test
 world)
Thread-Topic: LimitedPrivate and HBase (thoughts from the build and test
 world)
Thread-Index: AQHMJppoPUQKvSUkokSrJXKzSN4afJS13QiA
Date: Fri, 10 Jun 2011 02:21:37 +0000
Message-ID: <CA16CCDF.B531%mbhandarkar@linkedin.com>
In-Reply-To: <4DF0B19D.6050203@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <794563460E14C8419F9FC6DE32BF01BD@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

[Just wondering if one of the criteria for graduating to a top-level
project should be "no dependency on the LimitedPrivate APIs of the parent
project".]

Steve,

I agree with your suggestion for a downstream-project-build-and-test
instance.

All I can say is, "stay tuned".

- milind

--=20
Milind Bhandarkar
mbhandarkar@linkedin.com
+1-650-776-3167






On 6/9/11 4:42 AM, "Steve Loughran" <stevel@apache.org> wrote:

>On 06/08/2011 06:41 PM, Suresh Srinivas wrote:
>> I do not see any issue with the change that Todd has made. We have done
>> similar changes in HDFS-1586 in the past.
>>
>> Making APIs public comes with a cost. That is what we are avoiding with
>> LimitedPrivate. The intention was to include the following projects
>>that are
>> closely tied to Hadoop as projects eligible for LimitedPrivate.
>> {"HBase", "HDFS", "Hive", "MapReduce", "Pig"}. This list could grow in
>>the
>> future.
>
>I'm going to talk about my experience on the Ant team.
>
>One of the lessons of that project is that in the open source world, you
>can't predict how your code gets used, or control it. If someone wants
>to take your app and use it as a library -they can. If someone wants to
>do something completely unexpected with that library -they can. And this
>is a good thing, because your code gets used. Yes, you get new bugreps,
>but every person using your code is someone not using somebody elses
>code. You win.
>
>The other lesson from that is the following: in open source, there is no
>such thing as private code.
>
>* If you mark something as package scoped, they just inject their
>classes into your package (and who hasn't done that with their Hadoop
>extensions?).
>* If you mark something as protected, they subclass and open up its
>privacy.
>* If you mark something as private, they edit your source and create a
>new JAR with the relaxed permission
>
>for any of these actions, you end up fielding the bugreps, as the stack
>trace points to you. And it increases maintenance costs for everyone.
>
>
>Alternatively they cut and paste your code into their codebase, possibly
>-but not always- retaining the apache credits.
>
>That
>  * complicates copyright and lawsuits:
>  http://www.theserverside.com/news/thread.tss?thread_id=3D29958
>
>  * increases maintenance costs for everyone, especially if there are
>security issues with the original code.
>
>> When such projects break because of API change, we can co-ordinate as
>> community and fix the issues. This is not true for some application
>>that we
>> do not know of breaks!
>
>The way Ant handled this with Gump, the nightly clean build of all the
>OSS Java projects built with Ant
>http://vmgump.apache.org/gump/public/
>
>For all the projects, they thought they were getting a free CI build
>run, but what it really was was a regression test of Ant and every
>single OSS project. If a change in Ant broke anyone's build: we noticed.
>If a change in Log4J broke a build, someone noticed. It became a
>rapid-response regression test for the entire OSS suite.
>
>Sadly, it doesn't work so well. I'd blame Maven, but the move to ivy
>dependencies doesn't help either, it complicates classpaths no end.
>
>Even so, the idea is great: build and test your downstream applications,
>and the things you depend on, so you find problems within 24 hours of
>the change being committed -regardless of which project committed the
>change.
>
>The way to do it now would be with Jenkins, not just building and
>testing Hadooop-{core, hdfs, mapreduce}, but
>  -building and publishing every upstream dependency.
>  -test against the trunk versions build locally.
>  -build and test against the ivy-versioned artifacts that are
>controlled by the version.properties
>
>Together this flags up when something works against the old artifacts,
>but doesn't work against the trunk versions: that's their regressions,
>caught early.
>
>Downstream
>  -build and test the OSS projects that work with Hadoop.
>  That's the apache ones: HBase, Mahout, Pig, Hive, Hama etc, and the
>other ones, such as Cascading.
>
>That can be offered as a service to these projects "we will build and
>test your code against our trunk", a service designed to benefit
>everyone. They find their bugs, we find regressions.
>
>This is a pretty complex project, especially when you think about the
>challenge of testing your RPM generation code will install the RPMs (I
>bring up clean CentOS VMs for such a purpose), but without it you don't
>get everything working together, which is the state things appear to be
>in today.
>
>Ignoring the RPM install & test problems, if people are interested in
>working on this, we should be able to do a lot of it on Jenkins. Who is
>willing to get involved?
>
>-Steve


From general-return-3653-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 05:29:38 2011
Return-Path: <general-return-3653-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0D89A49FE
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 10 Jun 2011 05:29:38 +0000 (UTC)
Received: (qmail 40189 invoked by uid 500); 10 Jun 2011 05:29:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40145 invoked by uid 500); 10 Jun 2011 05:29:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40135 invoked by uid 99); 10 Jun 2011 05:29:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 05:29:33 +0000
X-ASF-Spam-Status: No, hits=3.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of hailong.yang1115@gmail.com designates 209.85.210.48 as permitted sender)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 05:29:26 +0000
Received: by pzk10 with SMTP id 10so1486842pzk.35
        for <general@hadoop.apache.org>; Thu, 09 Jun 2011 22:29:04 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:date:from:to:subject:message-id:x-mailer
         :mime-version:content-type;
        bh=LqBgO9KNufLhVhqaMoaeFzenq7ijFMkNL8tfuhHKGqs=;
        b=KM9MjNRL+5CVI2VndmFiXPr6eRuacX7EK099+o1wquvXEAhcYx9t/659qJTKR4NBFw
         w2hbhBHg/8ctXoAgUBZGEdDT2WhpH7xS2Vi8Ukdu6PGIYgdRWgGRARjJNMGMlQ0qDshV
         zDzKuJl4GQRj+6RgbF2ka/RqAndEogBE5uMNA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=date:from:to:subject:message-id:x-mailer:mime-version:content-type;
        b=W6UHoLTnz1qT5coOxjhLberdS4aq61w9nGmI01v8J6GlVSGFkiWox+rh553uqoVj+n
         GUTLkFp/Ilyohp7QAHdAfnP6cP1Z13F29tOZ7IwWr56xvwXkMI4WbFXxwCoEyuggBr/5
         TkR8NxVaLlVvETELykZLu3O/jeyw7goqlL9gI=
Received: by 10.143.20.14 with SMTP id x14mr282691wfi.105.1307683744462;
        Thu, 09 Jun 2011 22:29:04 -0700 (PDT)
Received: from HailongYangLen ([124.205.18.226])
        by mx.google.com with ESMTPS id n1sm2029381pbi.31.2011.06.09.22.28.51
        (version=SSLv3 cipher=OTHER);
        Thu, 09 Jun 2011 22:29:03 -0700 (PDT)
Date: Fri, 10 Jun 2011 13:28:54 +0800
From: "hailong.yang1115" <hailong.yang1115@gmail.com>
To: "general" <general@hadoop.apache.org>
Subject: Problems about the job counters
Message-ID: <201106101328462304641@gmail.com>
X-mailer: Foxmail 6, 15, 201, 26 [cn]
Mime-Version: 1.0
Content-Type: multipart/alternative;
	boundary="=====003_Dragon731067674787_====="

--=====003_Dragon731067674787_=====
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit

Dear all,

I am trying to the built-in example wordcount with nearly 15GB input. When the Hadoop job finished, I got the following counters.


CounterMapReduceTotal
Job CountersLaunched reduce tasks001
Rack-local map tasks0035
Launched map tasks002,318
Data-local map tasks002,283
FileSystemCountersFILE_BYTES_READ22,863,580,65617,654,943,34140,518,523,997
HDFS_BYTES_READ154,400,997,4590154,400,997,459
FILE_BYTES_WRITTEN33,490,829,40317,654,943,34151,145,772,744
HDFS_BYTES_WRITTEN02,747,356,7042,747,356,704


My question is what does the FILE_BYTES_READ counter mean? And what is the difference between FILE_BYTES_READ and HDFS_BYTES_READ? In my opinion, all the input is located in HDFS, so where does FILE_BYTES_READ come from during the map phase?


Any help will be appreciated!

Hailong

2011-06-10 



***********************************************
* Hailong Yang, PhD. Candidate 
* Sino-German Joint Software Institute, 
* School of Computer Science&Engineering, Beihang University
* Phone: (86-010)82315908
* Email: hailong.yang1115@gmail.com
* Address: G413, New Main Building in Beihang University, 
*              No.37 XueYuan Road,HaiDian District, 
*              Beijing,P.R.China,100191
***********************************************

--=====003_Dragon731067674787_=====--


From general-return-3654-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 09:01:33 2011
Return-Path: <general-return-3654-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D3CF34E4A
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 10 Jun 2011 09:01:33 +0000 (UTC)
Received: (qmail 91275 invoked by uid 500); 10 Jun 2011 09:01:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91210 invoked by uid 500); 10 Jun 2011 09:01:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91202 invoked by uid 99); 10 Jun 2011 09:01:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 09:01:31 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 09:01:23 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 7BD41B7DBF
	for <general@hadoop.apache.org>; Fri, 10 Jun 2011 10:01:01 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id KklxEgSQmEgz for <general@hadoop.apache.org>;
	Fri, 10 Jun 2011 10:01:00 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by tobor.hpl.hp.com (Postfix) with ESMTP id AF608B7DBB
	for <general@hadoop.apache.org>; Fri, 10 Jun 2011 10:01:00 +0100 (BST)
MailScanner-NULL-Check: 1308301246.81056@Z6W18QAqAz/2t2hNotlZbQ
Received: from [16.25.175.86] (wildhaus.hpl.hp.com [16.25.175.86])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p5A90kcG006863
	for <general@hadoop.apache.org>; Fri, 10 Jun 2011 10:00:46 +0100 (BST)
Message-ID: <4DF1DD3E.5070500@apache.org>
Date: Fri, 10 Jun 2011 10:00:46 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110419 Red Hat/3.1.10-1.el6_0 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: LimitedPrivate and HBase (thoughts from the build and test world)
References: <CA150253.1ADC9%sureshms@yahoo-inc.com>	<4DF0B19D.6050203@apache.org> <BANLkTinyMTJyB8-N+sEHEKNbziB5Y7C9bQ@mail.gmail.com>
In-Reply-To: <BANLkTinyMTJyB8-N+sEHEKNbziB5Y7C9bQ@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p5A90kcG006863
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 06/09/2011 08:23 PM, Stack wrote:
> Nice reality check and thanks for the how it was addressed elsewhere Steve.
>
> As you say, it sounds like a large undertaking but it would be a sweet
> service for the downstreamers.

as an aside, the thing that usually prevents you using Java apps as 
libraries is random calls to System.exit(). Hadoop does that; when I 
brought up nodes in-VM I'd catch those calls in a security manager, 
convert to RuntimeExceptions and throw them up the stack. IMO it'd be 
better for the whole hadoop stack to do this rather than have random 
threads take down the VMs, which is a debugging nightmare


From general-return-3655-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 18:20:51 2011
Return-Path: <general-return-3655-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2A28F47FF
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 10 Jun 2011 18:20:51 +0000 (UTC)
Received: (qmail 47042 invoked by uid 500); 10 Jun 2011 18:20:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46983 invoked by uid 500); 10 Jun 2011 18:20:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46975 invoked by uid 99); 10 Jun 2011 18:20:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 18:20:49 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 18:20:43 +0000
Received: by bwz8 with SMTP id 8so1996635bwz.35
        for <general@hadoop.apache.org>; Fri, 10 Jun 2011 11:20:21 -0700 (PDT)
Received: by 10.204.19.74 with SMTP id z10mr2217055bka.183.1307730021323; Fri,
 10 Jun 2011 11:20:21 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.79 with HTTP; Fri, 10 Jun 2011 11:20:01 -0700 (PDT)
From: Todd Lipcon <todd@cloudera.com>
Date: Fri, 10 Jun 2011 11:20:01 -0700
Message-ID: <BANLkTikSmUOd_4BqePmNH1fcPAYS8tbQtw@mail.gmail.com>
Subject: HADOOP-7106 (project unsplit) this weekend
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00032555a47e62147604a55f9ffd

--00032555a47e62147604a55f9ffd
Content-Type: text/plain; charset=ISO-8859-1

Hi all,

Pending any unforeseen issues, I am planning on committing HADOOP-7106 this
weekend. I have the credentials from Jukka to take care of the git trees as
well, and have done a "practice" move several times on a local mirror of the
svn.

I'll send out an announcement of the exact time in advance of when I
actually do the commit.

Thanks
-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

--00032555a47e62147604a55f9ffd--

From general-return-3656-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 18:34:59 2011
Return-Path: <general-return-3656-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3AAAD4394
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 10 Jun 2011 18:34:59 +0000 (UTC)
Received: (qmail 71051 invoked by uid 500); 10 Jun 2011 18:34:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70988 invoked by uid 500); 10 Jun 2011 18:34:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70980 invoked by uid 99); 10 Jun 2011 18:34:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 18:34:57 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of eli@cloudera.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 18:34:52 +0000
Received: by wyb40 with SMTP id 40so3179943wyb.35
        for <general@hadoop.apache.org>; Fri, 10 Jun 2011 11:34:31 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.62.137 with SMTP id y9mr3632099wec.107.1307730871151; Fri,
 10 Jun 2011 11:34:31 -0700 (PDT)
Received: by 10.216.28.205 with HTTP; Fri, 10 Jun 2011 11:34:31 -0700 (PDT)
In-Reply-To: <BANLkTikSmUOd_4BqePmNH1fcPAYS8tbQtw@mail.gmail.com>
References: <BANLkTikSmUOd_4BqePmNH1fcPAYS8tbQtw@mail.gmail.com>
Date: Fri, 10 Jun 2011 11:34:31 -0700
Message-ID: <BANLkTikkT1C2dJKMNnrKzs54jeRE6YuzYQ@mail.gmail.com>
Subject: Re: HADOOP-7106 (project unsplit) this weekend
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Spectacular.  Thanks Todd!

On Fri, Jun 10, 2011 at 11:20 AM, Todd Lipcon <todd@cloudera.com> wrote:
> Hi all,
>
> Pending any unforeseen issues, I am planning on committing HADOOP-7106 this
> weekend. I have the credentials from Jukka to take care of the git trees as
> well, and have done a "practice" move several times on a local mirror of the
> svn.
>
> I'll send out an announcement of the exact time in advance of when I
> actually do the commit.
>
> Thanks
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

From general-return-3657-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 18:40:33 2011
Return-Path: <general-return-3657-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AF233480B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 10 Jun 2011 18:40:33 +0000 (UTC)
Received: (qmail 83481 invoked by uid 500); 10 Jun 2011 18:40:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83433 invoked by uid 500); 10 Jun 2011 18:40:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83425 invoked by uid 99); 10 Jun 2011 18:40:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 18:40:32 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of owen.omalley@gmail.com designates 209.85.210.48 as permitted sender)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 18:40:24 +0000
Received: by pzk10 with SMTP id 10so1883332pzk.35
        for <general@hadoop.apache.org>; Fri, 10 Jun 2011 11:40:04 -0700 (PDT)
Received: by 10.142.191.6 with SMTP id o6mr403996wff.312.1307731204038;
        Fri, 10 Jun 2011 11:40:04 -0700 (PDT)
Received: from battlerock-lm.corp.yahoo.com (nat-dip4.cfw-a-gci.corp.yahoo.com [209.131.62.113])
        by mx.google.com with ESMTPS id c5sm2527544pbj.41.2011.06.10.11.40.02
        (version=SSLv3 cipher=OTHER);
        Fri, 10 Jun 2011 11:40:02 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: HADOOP-7106 (project unsplit) this weekend
From: Owen O'Malley <omalley@apache.org>
In-Reply-To: <BANLkTikSmUOd_4BqePmNH1fcPAYS8tbQtw@mail.gmail.com>
Date: Fri, 10 Jun 2011 11:40:01 -0700
Content-Transfer-Encoding: 7bit
Message-Id: <2A5D0706-BAAD-4C3A-9463-ADE78877F33C@apache.org>
References: <BANLkTikSmUOd_4BqePmNH1fcPAYS8tbQtw@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)


On Jun 10, 2011, at 11:20 AM, Todd Lipcon wrote:

> Pending any unforeseen issues, I am planning on committing HADOOP-7106 this
> weekend.

Thanks Todd for all of your work getting the details worked out and tested.

-- Owen

From general-return-3658-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 03:26:44 2011
Return-Path: <general-return-3658-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2237845AF
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sun, 12 Jun 2011 03:26:44 +0000 (UTC)
Received: (qmail 88956 invoked by uid 500); 12 Jun 2011 03:26:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88319 invoked by uid 500); 12 Jun 2011 03:26:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88311 invoked by uid 99); 12 Jun 2011 03:26:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 03:26:35 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 03:26:28 +0000
Received: by bwz8 with SMTP id 8so3113628bwz.35
        for <general@hadoop.apache.org>; Sat, 11 Jun 2011 20:26:08 -0700 (PDT)
Received: by 10.204.74.70 with SMTP id t6mr545733bkj.21.1307849168173; Sat, 11
 Jun 2011 20:26:08 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.79 with HTTP; Sat, 11 Jun 2011 20:25:48 -0700 (PDT)
In-Reply-To: <BANLkTikSmUOd_4BqePmNH1fcPAYS8tbQtw@mail.gmail.com>
References: <BANLkTikSmUOd_4BqePmNH1fcPAYS8tbQtw@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Sat, 11 Jun 2011 20:25:48 -0700
Message-ID: <BANLkTink5G6L55LVR+uKyYXb_ifRTTSkcA@mail.gmail.com>
Subject: Re: HADOOP-7106 (project unsplit) this weekend
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d7847916bc4c04a57b5db4
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d7847916bc4c04a57b5db4
Content-Type: text/plain; charset=ISO-8859-1

Hi all,

I'm figuring out one more small nit I noticed in my testing this evening.
Hopefully I will figure out what's going wrong and be ready to press the big
button tomorrow.

Assuming I don't have to "abort mission", my hope is to do this at around
3PM PST tomorrow (Sunday). I'll send out a message asking folks to please
hold commits to all branches while the move is in progress.

Thanks
-Todd

On Fri, Jun 10, 2011 at 11:20 AM, Todd Lipcon <todd@cloudera.com> wrote:

> Hi all,
>
> Pending any unforeseen issues, I am planning on committing HADOOP-7106 this
> weekend. I have the credentials from Jukka to take care of the git trees as
> well, and have done a "practice" move several times on a local mirror of the
> svn.
>
> I'll send out an announcement of the exact time in advance of when I
> actually do the commit.
>
> Thanks
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--0016e6d7847916bc4c04a57b5db4--

From general-return-3659-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 18:32:06 2011
Return-Path: <general-return-3659-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A24F441D8
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sun, 12 Jun 2011 18:32:06 +0000 (UTC)
Received: (qmail 59505 invoked by uid 500); 12 Jun 2011 18:32:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59436 invoked by uid 500); 12 Jun 2011 18:32:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59428 invoked by uid 99); 12 Jun 2011 18:32:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 18:32:03 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 18:31:57 +0000
Received: by wwi18 with SMTP id 18so4400771wwi.29
        for <general@hadoop.apache.org>; Sun, 12 Jun 2011 11:31:37 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.15.137 with SMTP id f9mr4192946wef.62.1307903496898; Sun,
 12 Jun 2011 11:31:36 -0700 (PDT)
Received: by 10.216.28.205 with HTTP; Sun, 12 Jun 2011 11:31:36 -0700 (PDT)
In-Reply-To: <31722744-6C23-4AB0-BB5C-86CE3D2A3522@mac.com>
References: <D97CF74A-147E-4573-B188-2F9C48CFC482@mac.com>
	<AANLkTi=F2eBsACBfjYSVava+VwFcsRpqZdRRN+OHXNRJ@mail.gmail.com>
	<823262F0-EE37-4A0A-8F8C-D9422541D67B@mac.com>
	<0CBAFDBB-AC2A-495D-8936-4DF4D6731B69@Holsman.NET>
	<AANLkTimUhDbTnwn6Hk2oYXFJMts+A=FnL-q-veXqX1m0@mail.gmail.com>
	<F2EC1B1B-A2BB-4FC8-9842-3904A25338EE@gbiv.com>
	<AANLkTikVcGcKHnh9EgUDQr2JGLPvkBvBRgn4my3uJfOc@mail.gmail.com>
	<864B55E7-8A46-458F-8B16-8252A2011835@Holsman.NET>
	<AANLkTikhe6YjUBGY8LMc3OG=b7VXWz7WhyNKGkJRu_7K@mail.gmail.com>
	<FB7E6664-F292-480F-8C4F-2B3CCAB9FFB7@yahoo-inc.com>
	<D3254862-5A94-4121-9B0B-9AE276D0955C@mac.com>
	<BANLkTinND1e0rDZrPeke1WqCqDG_BPUhZg@mail.gmail.com>
	<31722744-6C23-4AB0-BB5C-86CE3D2A3522@mac.com>
Date: Sun, 12 Jun 2011 11:31:36 -0700
Message-ID: <BANLkTim3r46UT5e5PDiqDCZ+qUw7rL5mkA@mail.gmail.com>
Subject: Re: [VOTE] Abandon hdfsproxy HDFS contrib
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Sounds like hdfs proxy isn't being maintained. The hdfs proxy tests
were disabled as part of HDFS-1666 because no one would maintain it.
If it's not going to be maintained we should remove it, currently
HDFS-1666 is a blocker for 0.22 because we don't want to release
bitrot'd code to users as part of 22.

Let's remove the current hdfsproxy in contrib and then either add the
features to HDFS directly (Sanjay suggested above might be an option)
or we can add a new non-contrib one (Alejandro built a nice working
one that just needs docs and some more tests that he could
contribute).

Owen - is that an acceptable solution to you?  HDSF-1666 is the last
blocker for 0.22 and we'd like to make progress on the release.

Thanks,
Eli

On Sat, Apr 9, 2011 at 10:01 PM, Nigel Daley <ndaley@mac.com> wrote:
> AFAICT, Owen was the one to -1 removal of HDFS Proxy. =A0Owen, are you gu=
ys maintaining this?
>
> Cheers,
> Nige
>
> On Apr 4, 2011, at 12:19 PM, Todd Lipcon wrote:
>
>> Could those of you who -1ed the removal of HDFS Proxy please look into t=
he
>> test that has been failing our Hudson build for the last several months:
>> https://issues.apache.org/jira/browse/HDFS-1666
>>
>> <https://issues.apache.org/jira/browse/HDFS-1666>It is one thing to say =
that
>> we "should" maintain a piece of code, but it's another to actually maint=
ain
>> it. In my mind, part of maintaining a project involves addressing consis=
tent
>> test failures as high priority items.
>>
>> -Todd
>>
>> On Tue, Feb 22, 2011 at 9:27 PM, Nigel Daley <ndaley@mac.com> wrote:
>>
>>> For closure, this vote fails due to a couple binding -1 votes.
>>>
>>> Nige
>>>
>>> On Feb 18, 2011, at 4:46 AM, Eric Baldeschwieler wrote:
>>>
>>>> Hi Bernd,
>>>>
>>>> Apache Hadoop is about scale. Most clusters will always be small, but
>>> Hadoop is going mainstream precisely because it scales to huge data and
>>> cluster sizes.
>>>>
>>>> There are lots of systems that work well on 10 node clusters. People
>>> select =A0 Hadoop because they are confident that as their business / p=
roblem
>>> grows, Hadoop can grow with it.
>>>>
>>>> ---
>>>> E14 - via iPhone
>>>>
>>>> On Feb 17, 2011, at 7:25 AM, "Bernd Fondermann" <
>>> bernd.fondermann@googlemail.com> wrote:
>>>>
>>>>> On Thu, Feb 17, 2011 at 14:58, Ian Holsman <hadoop@holsman.net> wrote=
:
>>>>>> Hi Bernd.
>>>>>>
>>>>>> On Feb 17, 2011, at 7:43 AM, Bernd Fondermann wrote:
>>>>>>>
>>>>>>> We have the very unfortunate situation here at Hadoop where Apache
>>>>>>> Hadoop is not the primary and foremost place of Hadoop development.
>>>>>>> Instead, code is developed internally at Yahoo and then contributed=
 in
>>>>>>> (smaller or larger) chunks to Hadoop.
>>>>>>
>>>>>> This has been the situation in the past,
>>>>>> but as you can see in the last month, this has changed.
>>>>>>
>>>>>> Yahoo! has publicly committed to move their development into the mai=
n
>>> code base, and you can see they have started doing this with the 20.100
>>> branch,
>>>>>> and their recent commits to trunk.
>>>>>> Combine this with Nige taking on the 0.22 release branch, (and
>>> sheperding it into a stable release) and I think we have are addressing=
 your
>>> concerns.
>>>>>>
>>>>>> They have also started bringing the discussions back on the list, se=
e
>>> the recent discussion about Jobtracker-nextgen Arun has re-started in
>>> MAPREDUCE-279.
>>>>>>
>>>>>> I'm not saying it's perfect, but I think the major players understan=
d
>>> there is an issue, and they are *ALL* moving in the right direction.
>>>>>
>>>>> I enthusiastically would like to see your optimism be verified.
>>>>> Maybe I'm misreading the statements issued publicly, but I don't thin=
k
>>>>> that this is fully understood. I agree though that it's a move into
>>>>> the right direction.
>>>>>
>>>>>>> This is open source development upside down.
>>>>>>> It is not ok for people to diff ASF svn against their internal code
>>>>>>> and provide the diff as a patch without reviewing IP first for ever=
y
>>>>>>> line of code changed.
>>>>>>> For larger chunks I'd suggest to even go via the Incubator IP
>>> clearance process.
>>>>>>> Only then will we force committers to primarily work here in the op=
en
>>>>>>> and return to what I'd consider a healthy project.
>>>>>>>
>>>>>>> To be honest: Hadoop is in the process of falling apart.
>>>>>>> Contrib Code gets moved out of Apache instead of being maintained
>>> here.
>>>>>>> Discussions are seldom consense-driven.
>>>>>>> Release branches stagnate.
>>>>>>
>>>>>> True. releases do take a long time. This is mainly due to it being
>>> extremely hard to test and verify that a release is stable.
>>>>>> It's not enough to just run the thing on 4 machines, you need at lea=
st
>>> 50 to test some of the major problems. This requires some serious $ for
>>> someone to verify.
>>>>>
>>>>> It has been proposed on the list before, IIRC. Don't know how to get
>>>>> there, but the project seriously needs access to a cluster of this
>>>>> size.
>>>>>
>>>>>>> Downstream projects like HBase don't get proper support.
>>>>>>> Production setups are made from 3rd party distributions.
>>>>>>> Development is not happening here, but elsewhere behind corporate
>>> doors.
>>>>>>> Discussion about future developments are started on corporate blogs=
 (
>>>>>>>
>>> http://developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextgen=
/
>>>>>>> ) instead of on the proper mailing list.
>>>>>>> Hurdles for committing are way too high.
>>>>>>> On the bright side, new committers and PMC members are added, this =
is
>>>>>>> an improvement.
>>>>>>>
>>>>>>> I'd suggest to move away from relying on large code dumps from
>>>>>>> corporations, and move back to the ASF-proven "individual committer
>>>>>>> commits on trunk"-model where more committers can get involved.
>>>>>>> If that means not to support high end cluster sizes for some months=
,
>>>>>>> well, so be it.
>>>>>>
>>>>>>> Average committers cannot run - e.g. test - on high
>>>>>>> end cluster sizes. If that would mean they cannot participate, then
>>>>>>> the open source project better concentrate on small and medium size=
d
>>>>>>> cluster instead.
>>>>>>
>>>>>>
>>>>>> Well.. that's one approach.. but there are several companies out the=
re
>>> who rely on apache's hadoop to power their large clusters, so I'd hate =
to
>>> see hadoop become something that only runs well on
>>>>>> 10-nodes.. as I don't think that will help anyone either.
>>>>>
>>>>> But only looking at high-end scale doesn't help either.
>>>>>
>>>>> Lets face the fact that Hadoop is now moving from early adaptors phas=
e
>>>>> into a much broader market. I predict that small to medium sized
>>>>> clusters will be the majority of Hadoop deployments in a few month
>>>>> time. 4000, or even 500 machines is the high-end range. If the open
>>>>> source project Hadoop cannot support those users adequately (without
>>>>> becoming defunct), the committership might be better off to focus on
>>>>> the low-end and medium sized users.
>>>>>
>>>>> I'm not suggesting to turn away from the handfull (?) of high-end
>>>>> users. They certainly have most valuable input. But also, *they*
>>>>> obviously have the resources in terms of larger clusters and
>>>>> developers to deal with their specific setups. Obviously, they don't
>>>>> need to rely on the open source project to make releases. In fact,
>>>>> they *do* work on their own Hadoop derivatives.
>>>>> All the other users, the hundreds of boring small cluster users, don'=
t
>>>>> have that choice. They *depend* on the open source releases.
>>>>>
>>>>> Hadoop is an Apache project, to provide HDFS and MR free of charge to
>>>>> the general public. Not only to me - nor to only one or two big
>>>>> companies either.
>>>>> Focus on all the users.
>>>>>
>>>>> Bernd
>>>
>>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>
>

From general-return-3660-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 21:51:19 2011
Return-Path: <general-return-3660-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 00ADD4CA2
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sun, 12 Jun 2011 21:51:18 +0000 (UTC)
Received: (qmail 19920 invoked by uid 500); 12 Jun 2011 21:51:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19854 invoked by uid 500); 12 Jun 2011 21:51:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19843 invoked by uid 99); 12 Jun 2011 21:51:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 21:51:16 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 21:51:09 +0000
Received: by bwz8 with SMTP id 8so3616090bwz.35
        for <general@hadoop.apache.org>; Sun, 12 Jun 2011 14:50:49 -0700 (PDT)
Received: by 10.205.35.1 with SMTP id su1mr3989405bkb.129.1307915448875; Sun,
 12 Jun 2011 14:50:48 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.79 with HTTP; Sun, 12 Jun 2011 14:50:24 -0700 (PDT)
In-Reply-To: <BANLkTink5G6L55LVR+uKyYXb_ifRTTSkcA@mail.gmail.com>
References: <BANLkTikSmUOd_4BqePmNH1fcPAYS8tbQtw@mail.gmail.com> <BANLkTink5G6L55LVR+uKyYXb_ifRTTSkcA@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Sun, 12 Jun 2011 14:50:24 -0700
Message-ID: <BANLkTiktkoY=Hg7Pc2mvYv5ydAbr-t6MJQ@mail.gmail.com>
Subject: Re: HADOOP-7106 (project unsplit) this weekend
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec52c64dfb9fbee04a58acb20
X-Virus-Checked: Checked by ClamAV on apache.org

--bcaec52c64dfb9fbee04a58acb20
Content-Type: text/plain; charset=ISO-8859-1

All of the nits I ran into should be resolved and we should be good to go. I
will start this in just about 10 minutes (3pm PST).

***Please hold all commits until further notice!*** I anticipate that this
should take under an hour, but if there are any bumps along the way it might
stretch into the evening. I'll send out an "all clear" email when things are
ready to go on the new layout.

I've disabled all of the Hudson builds for now and will be re-enabling them
one by one after reconfiguring their SVN URLs.

-Todd

On Sat, Jun 11, 2011 at 8:25 PM, Todd Lipcon <todd@cloudera.com> wrote:

> Hi all,
>
> I'm figuring out one more small nit I noticed in my testing this evening.
> Hopefully I will figure out what's going wrong and be ready to press the big
> button tomorrow.
>
> Assuming I don't have to "abort mission", my hope is to do this at around
> 3PM PST tomorrow (Sunday). I'll send out a message asking folks to please
> hold commits to all branches while the move is in progress.
>
> Thanks
> -Todd
>
>
> On Fri, Jun 10, 2011 at 11:20 AM, Todd Lipcon <todd@cloudera.com> wrote:
>
>> Hi all,
>>
>> Pending any unforeseen issues, I am planning on committing HADOOP-7106
>> this weekend. I have the credentials from Jukka to take care of the git
>> trees as well, and have done a "practice" move several times on a local
>> mirror of the svn.
>>
>> I'll send out an announcement of the exact time in advance of when I
>> actually do the commit.
>>
>> Thanks
>> -Todd
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--bcaec52c64dfb9fbee04a58acb20--

From general-return-3661-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 22:14:44 2011
Return-Path: <general-return-3661-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DEEF44647
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sun, 12 Jun 2011 22:14:44 +0000 (UTC)
Received: (qmail 36951 invoked by uid 500); 12 Jun 2011 22:14:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36895 invoked by uid 500); 12 Jun 2011 22:14:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36887 invoked by uid 99); 12 Jun 2011 22:14:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 22:14:43 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.88 as permitted sender)
Received: from [17.148.16.88] (HELO asmtpout013.mac.com) (17.148.16.88)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 22:14:36 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.17]
 (c-76-103-137-188.hsd1.ca.comcast.net [76.103.137.188])
 by asmtp013.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LMP004KS73PBS20@asmtp013.mac.com>
 for general@hadoop.apache.org; Sun, 12 Jun 2011 15:14:15 -0700 (PDT)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.4.6813,1.0.148,0.0.0000
 definitions=2011-06-12_08:2011-06-10,2011-06-12,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1106120145
References: <CA0BEEA9.1AA9A%sureshms@yahoo-inc.com>
 <CE461C76-D5B4-4C47-A96B-919F1FEF658E@apache.org>
 <5786C837-900C-4371-A3D2-568F1E2F8264@yahoo-inc.com>
 <BANLkTik-5mHTP++D_A_Q=G5qBM52mnCj3Q@mail.gmail.com>
 <BANLkTim0UJoc_r6VCE5dOjosFw=C79ppkw@mail.gmail.com>
 <BANLkTimirnAWcd2SrMvEAbHBptzo33nbDg@mail.gmail.com>
 <BANLkTikNG76RDdAHkO3K0qZm3vnw=aXc4g@mail.gmail.com>
In-reply-to: <BANLkTikNG76RDdAHkO3K0qZm3vnw=aXc4g@mail.gmail.com>
Message-id: <79D95D0B-2CE7-4BDE-9857-C4D359D23A97@mac.com>
Cc: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: iPad Mail (8J2)
From: Nigel Daley <ndaley@mac.com>
Subject: Re: Update on 0.22
Date: Sun, 12 Jun 2011 15:14:19 -0700
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

Sorry I missed this thread earlier. 

I'm not going to worry about the water under the bridge at this point, but going forward I would like to only include those issues marked as blocker. If a new issue crops up I will be taking a closer look at it and may push back. 

We've got less than 10 issues left to go :-)

Cheers,
Nige

Sent from my iPad

On Jun 2, 2011, at 11:31 AM, Todd Lipcon <todd@cloudera.com> wrote:

> On Thu, Jun 2, 2011 at 11:06 AM, Konstantin Shvachko
> <shv.hadoop@gmail.com>wrote:
> 
>> I propose just to make them blockers before committing to attract attention
>> of the release manager and get his approval. Imho, even small changes, like
>> HDFS-1954 are blockers, because a vague UI message is bug and bugs are
>> blockers.
>> 
> 
> Bugs are blockers? Then we'll never release!
> 
> Let's hear from Nigel what he thinks. It's his branch, if he's upset about
> the way it's being handled, he can deal with it as he sees fit.
> 
> -Todd
> 
> 
>> On Thu, Jun 2, 2011 at 10:39 AM, Todd Lipcon <todd@cloudera.com> wrote:
>> 
>>> On Wed, Jun 1, 2011 at 11:32 PM, Konstantin Shvachko
>>> <shv.hadoop@gmail.com>wrote:
>>> 
>>>> I can see them well.
>>>> I think Suresh's point is that non-blockers are going into 0.22.
>>>> Nigel, do you have full control over it?
>>>> 
>>> 
>>> Of course it's up to Nigel to decide, but here's my personal opinion:
>>> 
>>> One of the reasons we had a lot of divergence (read: external
>>> branches/forks/whatever) off of 0.20 is that the commit rules on the
>> branch
>>> were held pretty strictly. So, if you wanted a non-critical bug fix or a
>>> small improvement, the only option was to do such things on an external
>>> fork. 0.20 was branched in December '08 and not released until mid April
>>> '09. In 4 months a fair number of bug fixes and small improvements go in.
>>> 0.22 has been around even longer. If we were to keep it to *only*
>> blockers,
>>> then again it would be a fairly useless release due to the number of
>>> non-blocker bugs.
>>> 
>>> Clearly there's a balance and a judgment call when moving things back to
>> a
>>> branch. But at this point I'd consider small improvements and pretty much
>>> any bug fix to be reasonable, so long as it doesn't involve major
>> reworking
>>> of components. Nigel: if this assumption doesn't jive (ha ha, get it?)
>> with
>>> what you're thinking, please let me know :)
>>> 
>>> -Todd
>>> 
>>> 
>>>> On Wed, Jun 1, 2011 at 1:50 PM, Eric Baldeschwieler <
>>> eric14@yahoo-inc.com
>>>>> wrote:
>>>> 
>>>>> makes sense to me, but it might be good to work to make these
>> decisions
>>>>> visible so folks can understand what is happening.
>>>>> 
>>>>> On Jun 1, 2011, at 1:46 PM, Owen O'Malley wrote:
>>>>> 
>>>>>> 
>>>>>> On Jun 1, 2011, at 1:27 PM, Suresh Srinivas wrote:
>>>>>> 
>>>>>>> I see that there are several non blockers being promoted to 0.22
>>> from
>>>>> trunk.
>>>>>>> From my understanding, any non blocker change to 0.22 should be
>>>> approved
>>>>> by
>>>>>>> vote. Is this correct?
>>>>>> 
>>>>>> No, the Release Manager has full control over what goes into a
>>> release.
>>>>> The PMC votes on it once there is a release candidate.
>>>>>> 
>>>>>> -- Owen
>>>>> 
>>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>> 
>> 
> 
> 
> 
> -- 
> Todd Lipcon
> Software Engineer, Cloudera

From general-return-3662-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 23:39:47 2011
Return-Path: <general-return-3662-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ECD8F49F2
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sun, 12 Jun 2011 23:39:47 +0000 (UTC)
Received: (qmail 80852 invoked by uid 500); 12 Jun 2011 23:39:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80791 invoked by uid 500); 12 Jun 2011 23:39:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80783 invoked by uid 99); 12 Jun 2011 23:39:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 23:39:46 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 23:39:39 +0000
Received: by bwz8 with SMTP id 8so3665483bwz.35
        for <general@hadoop.apache.org>; Sun, 12 Jun 2011 16:39:18 -0700 (PDT)
Received: by 10.204.71.193 with SMTP id i1mr4265886bkj.102.1307921958181; Sun,
 12 Jun 2011 16:39:18 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.79 with HTTP; Sun, 12 Jun 2011 16:38:58 -0700 (PDT)
In-Reply-To: <BANLkTiktkoY=Hg7Pc2mvYv5ydAbr-t6MJQ@mail.gmail.com>
References: <BANLkTikSmUOd_4BqePmNH1fcPAYS8tbQtw@mail.gmail.com>
 <BANLkTink5G6L55LVR+uKyYXb_ifRTTSkcA@mail.gmail.com> <BANLkTiktkoY=Hg7Pc2mvYv5ydAbr-t6MJQ@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Sun, 12 Jun 2011 16:38:58 -0700
Message-ID: <BANLkTikU1LxFc6aF_dV6Fbu4E-fxf3c7ww@mail.gmail.com>
Subject: Re: HADOOP-7106 (project unsplit) this weekend
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c5b6c9b61caa04a58c4fc6
X-Virus-Checked: Checked by ClamAV on apache.org

--001636c5b6c9b61caa04a58c4fc6
Content-Type: text/plain; charset=ISO-8859-1

OK, this seems to have succeeded without any big problems!

I've re-enabled the git mirrors and the hudson builds. Feel free to commit
to the new trees.

Here are some instructions for the migration:

=== SVN users ===

Next time you "svn up" in your "common" working directory you'll end up
seeing the combined tree - ie a mapreduce/, hdfs/, and common/ subdirectory.
This is probably the easiest place from which to work, now. The URLs for the
combined SVN trees are:

trunk: https://svn.apache.org/repos/asf/hadoop/common/trunk/
branch-0.22:
http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22
branch-0.21:
http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21
yahoo-merge:
http://svn.apache.org/repos/asf/hadoop/common/branches/yahoo-merge
  (this one has the yahoo-merge branches from common, hdfs, and mapred)
MR-279: http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
  (this one has the yahoo-merge common and hdfs, and the MR-279 mapred)

The same kind of thing happened for HDFS-1073 and branch-0.21-old.
Pre-project-split branches like branch-0.20 should have remained untouched.

You can proceed to delete your checkouts of the individual mapred and hdfs
trees, since they exist within the combined trees above. If for some reason
you prefer to 'svn switch' an old MR or HDFS-specific checkout to point to
its new location, you can use the following incantation:
svn sw $(svn info | grep URL | awk '{print $2}' | sed
's,\(hdfs\|mapreduce\|common\)/\(.*\),common/\2/\1,')

=== Git Users ===
The git mirrors of the above 7 branches should now have a set of 4 commits
near the top that look like this:

Merge: 928d485 cd66945 77f628f
Author: Todd Lipcon <todd@apache.org>
Date:   Sun Jun 12 22:53:28 2011 +0000

    HADOOP-7106. Reorganize SVN layout to combine HDFS, Common, and MR in a
single tree (project unsplit)

    git-svn-id:
https://svn.apache.org/repos/asf/hadoop/common/trunk@113499413f79535-47bb-0310-9956-ffa450edef68

commit 77f628ff5925c25ba2ee4ce14590789eb2e7b85b
Author: Todd Lipcon <todd@apache.org>
Date:   Sun Jun 12 22:53:27 2011 +0000

    Relocate mapreduce into mapreduce/

commit cd66945f62635f589ff93468e94c0039684a8b6d
Author: Todd Lipcon <todd@apache.org>
Date:   Sun Jun 12 22:53:26 2011 +0000

    Relocate hdfs into hdfs/

commit 928d485e2743115fe37f9d123ce9a635c5afb91a
Author: Todd Lipcon <todd@apache.org>
Date:   Sun Jun 12 22:53:25 2011 +0000

    Relocate common into common/

The first of these 4 is a 3-parent "octopus" merge commit of the
pre-project-unsplit branches. In theory, git is smart enough to track
changes through this merge, so long as you pass the right flags (eg
--follow). For example:

todd@todd-w510:~/git/hadoop-common$ git log --pretty=oneline --abbrev-commit
--follow mapreduce/src/java/org/apache/hadoop/mapred/JobTracker.java | head
-10
77f628f Relocate mapreduce into mapreduce/
90df0cb MAPREDUCE-2455. Remove deprecated JobTracker.State in favour of
JobTrackerStatus.
ca2aba0 MAPREDUCE-2490. Add logging to graylist and blacklist activity to
aid diagnosis of related issues. Contributed by Jonathan Eagles
32aaa2a MAPREDUCE-2515. MapReduce code references some deprecated options.
Contributed by Ari Rabkin.

If you want to be able to have git follow renames all the way through the
project split back to the beginning of time, put the following in
hadoop-common/.git/info/grafts:
5128a9a453d64bfe1ed978cf9ffed27985eeef36
6c16dc8cf2b28818c852e95302920a278d07ad0c
6a3ac690e493c7da45bbf2ae2054768c427fd0e1
6c16dc8cf2b28818c852e95302920a278d07ad0c
546d96754ffee3142bcbbf4563c624c053d0ed0d
6c16dc8cf2b28818c852e95302920a278d07ad0c

In terms of rebasing git branches, git is actually pretty smart. For
example, I have a local "HDFS-1073" branch in my hdfs repo. To transition it
to the new combined repo, I did the following:

# Add my project-split hdfs git repo as a remote:
git remote add splithdfs /home/todd/git/hadoop-hdfs/
git fetch splithdfs

# Checkout a branch in my combined repo
git checkout -b HDFS-1073 splithdfs/HDFS-1073

# Rebase it on the combined 1073 branch
git rebase origin/HDFS-1073

...and it actually applies my patches inside the appropriate subdirectory (I
was surprised and impressed by this!)
If the branch you're rebasing has added or moved files, it might not be
smart enough and you'll have to manually rename them in your branch inside
of the appropriate subtree.. but for simple patches this seems to work. For
less simple things, the best bet may be to use "git filter-branch" on the
patch series to relocate it inside a subdirectory, and then try to rebase.
Let me know if you need a hand with any git cleanup, happy to help.


== Outstanding issues ==

The one outstanding issue I'm aware of is that the test-patch builds should
be smart enough to be able to deal with patches that are relative to the
combined root instead of the original project. Right now, if you export a
diff from git, it will include "hdfs/" or "mapreduce/" in the changed file
names, and the QA bot won't know how to apply it. The workaround for this is
to change directory into the relative subproject dir, and then pass
"--relative" to "git diff" or "git show", for example:

todd@todd-w510:~/git/hadoop-common/mapreduce$ git diff --relative
--no-prefix
diff --git CHANGES.txt CHANGES.txt
...


I imagine there are probably some other things that fell through the cracks.
Please get in touch if there's anything that seems amiss.

-Todd


On Sun, Jun 12, 2011 at 2:50 PM, Todd Lipcon <todd@cloudera.com> wrote:

> All of the nits I ran into should be resolved and we should be good to go.
> I will start this in just about 10 minutes (3pm PST).
>
> ***Please hold all commits until further notice!*** I anticipate that this
> should take under an hour, but if there are any bumps along the way it might
> stretch into the evening. I'll send out an "all clear" email when things are
> ready to go on the new layout.
>
> I've disabled all of the Hudson builds for now and will be re-enabling them
> one by one after reconfiguring their SVN URLs.
>
> -Todd
>
> On Sat, Jun 11, 2011 at 8:25 PM, Todd Lipcon <todd@cloudera.com> wrote:
>
>> Hi all,
>>
>> I'm figuring out one more small nit I noticed in my testing this evening.
>> Hopefully I will figure out what's going wrong and be ready to press the big
>> button tomorrow.
>>
>> Assuming I don't have to "abort mission", my hope is to do this at around
>> 3PM PST tomorrow (Sunday). I'll send out a message asking folks to please
>> hold commits to all branches while the move is in progress.
>>
>> Thanks
>> -Todd
>>
>>
>> On Fri, Jun 10, 2011 at 11:20 AM, Todd Lipcon <todd@cloudera.com> wrote:
>>
>>> Hi all,
>>>
>>> Pending any unforeseen issues, I am planning on committing HADOOP-7106
>>> this weekend. I have the credentials from Jukka to take care of the git
>>> trees as well, and have done a "practice" move several times on a local
>>> mirror of the svn.
>>>
>>> I'll send out an announcement of the exact time in advance of when I
>>> actually do the commit.
>>>
>>> Thanks
>>> -Todd
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>>
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--001636c5b6c9b61caa04a58c4fc6--

From general-return-3663-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 07:40:20 2011
Return-Path: <general-return-3663-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D4A21604D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 07:40:20 +0000 (UTC)
Received: (qmail 29439 invoked by uid 500); 13 Jun 2011 07:40:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29369 invoked by uid 500); 13 Jun 2011 07:40:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29361 invoked by uid 99); 13 Jun 2011 07:40:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 07:40:18 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of eli@cloudera.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 07:40:12 +0000
Received: by wyb40 with SMTP id 40so4671327wyb.35
        for <general@hadoop.apache.org>; Mon, 13 Jun 2011 00:39:51 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.231.89 with SMTP id k67mr4554381weq.113.1307950791051;
 Mon, 13 Jun 2011 00:39:51 -0700 (PDT)
Received: by 10.216.30.212 with HTTP; Mon, 13 Jun 2011 00:39:51 -0700 (PDT)
In-Reply-To: <BANLkTikU1LxFc6aF_dV6Fbu4E-fxf3c7ww@mail.gmail.com>
References: <BANLkTikSmUOd_4BqePmNH1fcPAYS8tbQtw@mail.gmail.com>
	<BANLkTink5G6L55LVR+uKyYXb_ifRTTSkcA@mail.gmail.com>
	<BANLkTiktkoY=Hg7Pc2mvYv5ydAbr-t6MJQ@mail.gmail.com>
	<BANLkTikU1LxFc6aF_dV6Fbu4E-fxf3c7ww@mail.gmail.com>
Date: Mon, 13 Jun 2011 00:39:51 -0700
Message-ID: <BANLkTi=uD-WUkSDNRJq7B3dh2QBKwq_8Ww@mail.gmail.com>
Subject: Re: HADOOP-7106 (project unsplit) this weekend
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Nice work Todd.   Thank you for making this happen!


On Sun, Jun 12, 2011 at 4:38 PM, Todd Lipcon <todd@cloudera.com> wrote:
> OK, this seems to have succeeded without any big problems!
>
> I've re-enabled the git mirrors and the hudson builds. Feel free to commi=
t
> to the new trees.
>
> Here are some instructions for the migration:
>
> =3D=3D=3D SVN users =3D=3D=3D
>
> Next time you "svn up" in your "common" working directory you'll end up
> seeing the combined tree - ie a mapreduce/, hdfs/, and common/ subdirecto=
ry.
> This is probably the easiest place from which to work, now. The URLs for =
the
> combined SVN trees are:
>
> trunk: https://svn.apache.org/repos/asf/hadoop/common/trunk/
> branch-0.22:
> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22
> branch-0.21:
> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21
> yahoo-merge:
> http://svn.apache.org/repos/asf/hadoop/common/branches/yahoo-merge
> =A0(this one has the yahoo-merge branches from common, hdfs, and mapred)
> MR-279: http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
> =A0(this one has the yahoo-merge common and hdfs, and the MR-279 mapred)
>
> The same kind of thing happened for HDFS-1073 and branch-0.21-old.
> Pre-project-split branches like branch-0.20 should have remained untouche=
d.
>
> You can proceed to delete your checkouts of the individual mapred and hdf=
s
> trees, since they exist within the combined trees above. If for some reas=
on
> you prefer to 'svn switch' an old MR or HDFS-specific checkout to point t=
o
> its new location, you can use the following incantation:
> svn sw $(svn info | grep URL | awk '{print $2}' | sed
> 's,\(hdfs\|mapreduce\|common\)/\(.*\),common/\2/\1,')
>
> =3D=3D=3D Git Users =3D=3D=3D
> The git mirrors of the above 7 branches should now have a set of 4 commit=
s
> near the top that look like this:
>
> Merge: 928d485 cd66945 77f628f
> Author: Todd Lipcon <todd@apache.org>
> Date: =A0 Sun Jun 12 22:53:28 2011 +0000
>
> =A0 =A0HADOOP-7106. Reorganize SVN layout to combine HDFS, Common, and MR=
 in a
> single tree (project unsplit)
>
> =A0 =A0git-svn-id:
> https://svn.apache.org/repos/asf/hadoop/common/trunk@113499413f79535-47bb=
-0310-9956-ffa450edef68
>
> commit 77f628ff5925c25ba2ee4ce14590789eb2e7b85b
> Author: Todd Lipcon <todd@apache.org>
> Date: =A0 Sun Jun 12 22:53:27 2011 +0000
>
> =A0 =A0Relocate mapreduce into mapreduce/
>
> commit cd66945f62635f589ff93468e94c0039684a8b6d
> Author: Todd Lipcon <todd@apache.org>
> Date: =A0 Sun Jun 12 22:53:26 2011 +0000
>
> =A0 =A0Relocate hdfs into hdfs/
>
> commit 928d485e2743115fe37f9d123ce9a635c5afb91a
> Author: Todd Lipcon <todd@apache.org>
> Date: =A0 Sun Jun 12 22:53:25 2011 +0000
>
> =A0 =A0Relocate common into common/
>
> The first of these 4 is a 3-parent "octopus" merge commit of the
> pre-project-unsplit branches. In theory, git is smart enough to track
> changes through this merge, so long as you pass the right flags (eg
> --follow). For example:
>
> todd@todd-w510:~/git/hadoop-common$ git log --pretty=3Doneline --abbrev-c=
ommit
> --follow mapreduce/src/java/org/apache/hadoop/mapred/JobTracker.java | he=
ad
> -10
> 77f628f Relocate mapreduce into mapreduce/
> 90df0cb MAPREDUCE-2455. Remove deprecated JobTracker.State in favour of
> JobTrackerStatus.
> ca2aba0 MAPREDUCE-2490. Add logging to graylist and blacklist activity to
> aid diagnosis of related issues. Contributed by Jonathan Eagles
> 32aaa2a MAPREDUCE-2515. MapReduce code references some deprecated options=
.
> Contributed by Ari Rabkin.
>
> If you want to be able to have git follow renames all the way through the
> project split back to the beginning of time, put the following in
> hadoop-common/.git/info/grafts:
> 5128a9a453d64bfe1ed978cf9ffed27985eeef36
> 6c16dc8cf2b28818c852e95302920a278d07ad0c
> 6a3ac690e493c7da45bbf2ae2054768c427fd0e1
> 6c16dc8cf2b28818c852e95302920a278d07ad0c
> 546d96754ffee3142bcbbf4563c624c053d0ed0d
> 6c16dc8cf2b28818c852e95302920a278d07ad0c
>
> In terms of rebasing git branches, git is actually pretty smart. For
> example, I have a local "HDFS-1073" branch in my hdfs repo. To transition=
 it
> to the new combined repo, I did the following:
>
> # Add my project-split hdfs git repo as a remote:
> git remote add splithdfs /home/todd/git/hadoop-hdfs/
> git fetch splithdfs
>
> # Checkout a branch in my combined repo
> git checkout -b HDFS-1073 splithdfs/HDFS-1073
>
> # Rebase it on the combined 1073 branch
> git rebase origin/HDFS-1073
>
> ...and it actually applies my patches inside the appropriate subdirectory=
 (I
> was surprised and impressed by this!)
> If the branch you're rebasing has added or moved files, it might not be
> smart enough and you'll have to manually rename them in your branch insid=
e
> of the appropriate subtree.. but for simple patches this seems to work. F=
or
> less simple things, the best bet may be to use "git filter-branch" on the
> patch series to relocate it inside a subdirectory, and then try to rebase=
.
> Let me know if you need a hand with any git cleanup, happy to help.
>
>
> =3D=3D Outstanding issues =3D=3D
>
> The one outstanding issue I'm aware of is that the test-patch builds shou=
ld
> be smart enough to be able to deal with patches that are relative to the
> combined root instead of the original project. Right now, if you export a
> diff from git, it will include "hdfs/" or "mapreduce/" in the changed fil=
e
> names, and the QA bot won't know how to apply it. The workaround for this=
 is
> to change directory into the relative subproject dir, and then pass
> "--relative" to "git diff" or "git show", for example:
>
> todd@todd-w510:~/git/hadoop-common/mapreduce$ git diff --relative
> --no-prefix
> diff --git CHANGES.txt CHANGES.txt
> ...
>
>
> I imagine there are probably some other things that fell through the crac=
ks.
> Please get in touch if there's anything that seems amiss.
>
> -Todd
>
>
> On Sun, Jun 12, 2011 at 2:50 PM, Todd Lipcon <todd@cloudera.com> wrote:
>
>> All of the nits I ran into should be resolved and we should be good to g=
o.
>> I will start this in just about 10 minutes (3pm PST).
>>
>> ***Please hold all commits until further notice!*** I anticipate that th=
is
>> should take under an hour, but if there are any bumps along the way it m=
ight
>> stretch into the evening. I'll send out an "all clear" email when things=
 are
>> ready to go on the new layout.
>>
>> I've disabled all of the Hudson builds for now and will be re-enabling t=
hem
>> one by one after reconfiguring their SVN URLs.
>>
>> -Todd
>>
>> On Sat, Jun 11, 2011 at 8:25 PM, Todd Lipcon <todd@cloudera.com> wrote:
>>
>>> Hi all,
>>>
>>> I'm figuring out one more small nit I noticed in my testing this evenin=
g.
>>> Hopefully I will figure out what's going wrong and be ready to press th=
e big
>>> button tomorrow.
>>>
>>> Assuming I don't have to "abort mission", my hope is to do this at arou=
nd
>>> 3PM PST tomorrow (Sunday). I'll send out a message asking folks to plea=
se
>>> hold commits to all branches while the move is in progress.
>>>
>>> Thanks
>>> -Todd
>>>
>>>
>>> On Fri, Jun 10, 2011 at 11:20 AM, Todd Lipcon <todd@cloudera.com> wrote=
:
>>>
>>>> Hi all,
>>>>
>>>> Pending any unforeseen issues, I am planning on committing HADOOP-7106
>>>> this weekend. I have the credentials from Jukka to take care of the gi=
t
>>>> trees as well, and have done a "practice" move several times on a loca=
l
>>>> mirror of the svn.
>>>>
>>>> I'll send out an announcement of the exact time in advance of when I
>>>> actually do the commit.
>>>>
>>>> Thanks
>>>> -Todd
>>>> --
>>>> Todd Lipcon
>>>> Software Engineer, Cloudera
>>>>
>>>
>>>
>>>
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>>
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

From general-return-3664-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 13:25:05 2011
Return-Path: <general-return-3664-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 70E806BAF
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 13:25:05 +0000 (UTC)
Received: (qmail 36882 invoked by uid 500); 13 Jun 2011 13:25:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36807 invoked by uid 500); 13 Jun 2011 13:25:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36799 invoked by uid 99); 13 Jun 2011 13:25:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 13:25:03 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mlortiz@uci.cu designates 200.55.140.180 as permitted sender)
Received: from [200.55.140.180] (HELO mx3.uci.cu) (200.55.140.180)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 13 Jun 2011 13:24:57 +0000
Received: (qmail 20140 invoked by uid 507); 13 Jun 2011 13:24:31 -0000
Received: from 10.0.0.183 by ns3.uci.cu (envelope-from <mlortiz@uci.cu>, uid 501) with qmail-scanner-2.01st 
 (avp: 5.0.2.0. spamassassin: 3.0.6. perlscan: 2.01st.  
 Clear:RC:1(10.0.0.183):. 
 Processed in 0.642131 secs); 13 Jun 2011 13:24:31 -0000
Received: from unknown (HELO ucimail2.uci.cu) (10.0.0.183)
  by 0 with SMTP; 13 Jun 2011 13:24:30 -0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by ucimail2.uci.cu (Postfix) with ESMTP id 92DE23DC009
	for <general@hadoop.apache.org>; Mon, 13 Jun 2011 09:26:18 -0400 (CDT)
X-Virus-Scanned: amavisd-new at uci.cu
Received: from ucimail2.uci.cu ([127.0.0.1])
	by localhost (ucimail2.uci.cu [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id IxuQDelPbfoZ; Mon, 13 Jun 2011 09:26:17 -0400 (CDT)
Received: from [10.36.18.57] (marcosluis-aspire-5251.uci.cu [10.36.18.57])
	by ucimail2.uci.cu (Postfix) with ESMTP id A185C3DC007
	for <general@hadoop.apache.org>; Mon, 13 Jun 2011 09:26:17 -0400 (CDT)
Message-ID: <4DF61707.4070902@uci.cu>
Date: Mon, 13 Jun 2011 09:26:23 -0430
From: Marcos Ortiz <mlortiz@uci.cu>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110424 Lightning/1.0b2 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Development Branch for the Hadoop Next Generation
References: <BANLkTikSmUOd_4BqePmNH1fcPAYS8tbQtw@mail.gmail.com>	<BANLkTink5G6L55LVR+uKyYXb_ifRTTSkcA@mail.gmail.com>	<BANLkTiktkoY=Hg7Pc2mvYv5ydAbr-t6MJQ@mail.gmail.com>	<BANLkTikU1LxFc6aF_dV6Fbu4E-fxf3c7ww@mail.gmail.com> <BANLkTi=uD-WUkSDNRJq7B3dh2QBKwq_8Ww@mail.gmail.com>
In-Reply-To: <BANLkTi=uD-WUkSDNRJq7B3dh2QBKwq_8Ww@mail.gmail.com>
Content-Type: multipart/alternative;
 boundary="------------010902030407040008010602"
X-Virus-Checked: Checked by ClamAV on apache.org

--------------010902030407040008010602
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit

I was reading the Hadoop Next Generation post on the YDN by Arun 
(http://developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextgen/), 
and
I was wondering if this development is public and where I can find the 
sources for this?
Regards and thanks a lot for reading


-- 
Marcos Luís Ortíz Valmaseda
  Software Engineer (UCI)
  http://marcosluis2186.posterous.com
  http://twitter.com/marcosluis2186



--------------010902030407040008010602--

From general-return-3665-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 13:32:39 2011
Return-Path: <general-return-3665-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 443D46A19
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 13:32:39 +0000 (UTC)
Received: (qmail 48254 invoked by uid 500); 13 Jun 2011 13:32:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48196 invoked by uid 500); 13 Jun 2011 13:32:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48188 invoked by uid 99); 13 Jun 2011 13:32:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 13:32:37 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of harsh@cloudera.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 13:32:32 +0000
Received: by iwr19 with SMTP id 19so5975177iwr.35
        for <general@hadoop.apache.org>; Mon, 13 Jun 2011 06:32:11 -0700 (PDT)
Received: by 10.42.219.65 with SMTP id ht1mr7085606icb.14.1307971931119; Mon,
 13 Jun 2011 06:32:11 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.42.218.72 with HTTP; Mon, 13 Jun 2011 06:31:51 -0700 (PDT)
In-Reply-To: <4DF61707.4070902@uci.cu>
References: <BANLkTikSmUOd_4BqePmNH1fcPAYS8tbQtw@mail.gmail.com>
 <BANLkTink5G6L55LVR+uKyYXb_ifRTTSkcA@mail.gmail.com> <BANLkTiktkoY=Hg7Pc2mvYv5ydAbr-t6MJQ@mail.gmail.com>
 <BANLkTikU1LxFc6aF_dV6Fbu4E-fxf3c7ww@mail.gmail.com> <BANLkTi=uD-WUkSDNRJq7B3dh2QBKwq_8Ww@mail.gmail.com>
 <4DF61707.4070902@uci.cu>
From: Harsh J <harsh@cloudera.com>
Date: Mon, 13 Jun 2011 19:01:51 +0530
Message-ID: <BANLkTi=vAe+JTvuC6QYtcuwdT81DMpZGrQ@mail.gmail.com>
Subject: Re: Development Branch for the Hadoop Next Generation
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Right now, its in the public SVN branch of MapReduce named "MR-279".

See https://issues.apache.org/jira/browse/MAPREDUCE-279, which is its
giant umbrella ticket. It may be merging into trunk (0.23) sometime
soon in the future, though.

On Mon, Jun 13, 2011 at 7:26 PM, Marcos Ortiz <mlortiz@uci.cu> wrote:
> I was reading the Hadoop Next Generation post on the YDN by Arun
> (http://developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextgen/=
),
> and
> I was wondering if this development is public and where I can find the
> sources for this?
> Regards and thanks a lot for reading
>
>
> --
> Marcos Lu=EDs Ort=EDz Valmaseda
> =A0Software Engineer (UCI)
> =A0http://marcosluis2186.posterous.com
> =A0http://twitter.com/marcosluis2186
>
>
>



--=20
Harsh J

From general-return-3666-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 13:43:23 2011
Return-Path: <general-return-3666-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C1BC36809
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 13:43:23 +0000 (UTC)
Received: (qmail 70764 invoked by uid 500); 13 Jun 2011 13:43:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70709 invoked by uid 500); 13 Jun 2011 13:43:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70701 invoked by uid 99); 13 Jun 2011 13:43:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 13:43:22 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mlortiz@uci.cu designates 200.55.140.180 as permitted sender)
Received: from [200.55.140.180] (HELO mx3.uci.cu) (200.55.140.180)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 13 Jun 2011 13:43:16 +0000
Received: (qmail 28595 invoked by uid 507); 13 Jun 2011 13:42:51 -0000
Received: from 10.0.0.185 by ns3.uci.cu (envelope-from <mlortiz@uci.cu>, uid 501) with qmail-scanner-2.01st 
 (avp: 5.0.2.0. spamassassin: 3.0.6. perlscan: 2.01st.  
 Clear:RC:1(10.0.0.185):. 
 Processed in 0.652667 secs); 13 Jun 2011 13:42:51 -0000
Received: from unknown (HELO ucimail4.uci.cu) (10.0.0.185)
  by 0 with SMTP; 13 Jun 2011 13:42:50 -0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by ucimail4.uci.cu (Postfix) with ESMTP id 7A18814C4029;
	Mon, 13 Jun 2011 09:42:50 -0400 (CDT)
X-Virus-Scanned: amavisd-new at uci.cu
Received: from ucimail4.uci.cu ([127.0.0.1])
	by localhost (ucimail4.uci.cu [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id s5au-jCJuGL4; Mon, 13 Jun 2011 09:42:49 -0400 (CDT)
Received: from [10.36.18.57] (marcosluis-aspire-5251.uci.cu [10.36.18.57])
	by ucimail4.uci.cu (Postfix) with ESMTP id E7AE114C400C;
	Mon, 13 Jun 2011 09:42:48 -0400 (CDT)
Message-ID: <4DF61B52.9000102@uci.cu>
Date: Mon, 13 Jun 2011 09:44:42 -0430
From: Marcos Ortiz <mlortiz@uci.cu>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110424 Lightning/1.0b2 Thunderbird/3.1.10
MIME-Version: 1.0
To: Harsh J <harsh@cloudera.com>
CC: general@hadoop.apache.org
Subject: Re: Development Branch for the Hadoop Next Generation
References: <BANLkTikSmUOd_4BqePmNH1fcPAYS8tbQtw@mail.gmail.com> <BANLkTink5G6L55LVR+uKyYXb_ifRTTSkcA@mail.gmail.com> <BANLkTiktkoY=Hg7Pc2mvYv5ydAbr-t6MJQ@mail.gmail.com> <BANLkTikU1LxFc6aF_dV6Fbu4E-fxf3c7ww@mail.gmail.com> <BANLkTi=uD-WUkSDNRJq7B3dh2QBKwq_8Ww@mail.gmail.com> <4DF61707.4070902@uci.cu> <BANLkTi=vAe+JTvuC6QYtcuwdT81DMpZGrQ@mail.gmail.com>
In-Reply-To: <BANLkTi=vAe+JTvuC6QYtcuwdT81DMpZGrQ@mail.gmail.com>
Content-Type: multipart/alternative;
 boundary="------------090601060406090607000502"
X-Virus-Checked: Checked by ClamAV on apache.org

--------------090601060406090607000502
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit

On 06/13/2011 09:01 AM, Harsh J wrote:
> Right now, its in the public SVN branch of MapReduce named "MR-279".
>
> See https://issues.apache.org/jira/browse/MAPREDUCE-279, which is its
> giant umbrella ticket. It may be merging into trunk (0.23) sometime
> soon in the future, though.
>
> On Mon, Jun 13, 2011 at 7:26 PM, Marcos Ortiz<mlortiz@uci.cu>  wrote:
>> I was reading the Hadoop Next Generation post on the YDN by Arun
>> (http://developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextgen/),
>> and
>> I was wondering if this development is public and where I can find the
>> sources for this?
>> Regards and thanks a lot for reading
>>
>>
>> --
>> Marcos Luís Ortíz Valmaseda
>>   Software Engineer (UCI)
>>   http://marcosluis2186.posterous.com
>>   http://twitter.com/marcosluis2186
>>
>>
>>
>
>
  Good.
  Thanks a lot, Harsh
   It's a very interesting work.
   Why is been merged to the 0.23 version?


-- 
Marcos Luís Ortíz Valmaseda
  Software Engineer (UCI)
  http://marcosluis2186.posterous.com
  http://twitter.com/marcosluis2186



--------------090601060406090607000502--

From general-return-3667-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 13:59:19 2011
Return-Path: <general-return-3667-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2009461A5
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 13:59:19 +0000 (UTC)
Received: (qmail 97901 invoked by uid 500); 13 Jun 2011 13:59:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97711 invoked by uid 500); 13 Jun 2011 13:59:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97569 invoked by uid 99); 13 Jun 2011 13:59:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 13:59:15 +0000
X-ASF-Spam-Status: No, hits=3.3 required=5.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 13:59:08 +0000
Received: from SP1-EX07CAS02.ds.corp.yahoo.com (sp1-ex07cas02.ds.corp.yahoo.com [216.252.116.138])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5DDwSK3007009;
	Mon, 13 Jun 2011 06:58:28 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1307973509;
	bh=A46zXJhEItleDyOTgGOyoEA3cK7HobsrU/S33GgUaaY=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 MIME-Version;
	b=FDk4P9YytpTE/z+A7d9ed+PegR1VP0kRKF2jaTDFNyihRSK8+bEMBrWg8EyQ2neNu
	 /QtKAvp6c+rx6U/VhqcaZwRD8i+ZxnO6JUiELEW8piuV/rTgojfknZDj4Ng2QTsZpD
	 OmwCosQHIAThgNoHIlACZTfqwSzW6fFghFna3yIU=
Received: from SP1-EX07VS02.ds.corp.yahoo.com ([216.252.116.135]) by
 SP1-EX07CAS02.ds.corp.yahoo.com ([216.252.116.167]) with mapi; Mon, 13 Jun
 2011 06:58:28 -0700
From: Robert Evans <evans@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>,
        Harsh J
	<harsh@cloudera.com>
Date: Mon, 13 Jun 2011 06:58:28 -0700
Subject: Re: Development Branch for the Hadoop Next Generation
Thread-Topic: Development Branch for the Hadoop Next Generation
Thread-Index: Acwpz+6PDKl02YjfTkO7ji/ILVVEJQAAgngv
Message-ID: <CA1B81B4.24E0D%evans@yahoo-inc.com>
In-Reply-To: <4DF61B52.9000102@uci.cu>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_CA1B81B424E0Devansyahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_CA1B81B424E0Devansyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

0.23 is trunk right now. 0.22 was branched for stabilization before release=
.

--Bobby

On 6/13/11 9:14 AM, "Marcos Ortiz" <mlortiz@uci.cu> wrote:

On 06/13/2011 09:01 AM, Harsh J wrote:
> Right now, its in the public SVN branch of MapReduce named "MR-279".
>
> See https://issues.apache.org/jira/browse/MAPREDUCE-279, which is its
> giant umbrella ticket. It may be merging into trunk (0.23) sometime
> soon in the future, though.
>
> On Mon, Jun 13, 2011 at 7:26 PM, Marcos Ortiz<mlortiz@uci.cu>  wrote:
>> I was reading the Hadoop Next Generation post on the YDN by Arun
>> (http://developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextgen=
/),
>> and
>> I was wondering if this development is public and where I can find the
>> sources for this?
>> Regards and thanks a lot for reading
>>
>>
>> --
>> Marcos Lu=EDs Ort=EDz Valmaseda
>>   Software Engineer (UCI)
>>   http://marcosluis2186.posterous.com
>>   http://twitter.com/marcosluis2186
>>
>>
>>
>
>
  Good.
  Thanks a lot, Harsh
   It's a very interesting work.
   Why is been merged to the 0.23 version?


--
Marcos Lu=EDs Ort=EDz Valmaseda
  Software Engineer (UCI)
  http://marcosluis2186.posterous.com
  http://twitter.com/marcosluis2186




--_000_CA1B81B424E0Devansyahooinccom_--

From general-return-3668-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 14:05:53 2011
Return-Path: <general-return-3668-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 298EA6978
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 14:05:53 +0000 (UTC)
Received: (qmail 10054 invoked by uid 500); 13 Jun 2011 14:05:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 9991 invoked by uid 500); 13 Jun 2011 14:05:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 9981 invoked by uid 99); 13 Jun 2011 14:05:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 14:05:51 +0000
X-ASF-Spam-Status: No, hits=3.3 required=5.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 14:05:43 +0000
Received: from SP1-EX07CAS02.ds.corp.yahoo.com (sp1-ex07cas02.ds.corp.yahoo.com [216.252.116.138])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p5DE5395084255
	for <general@hadoop.apache.org>; Mon, 13 Jun 2011 07:05:03 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1307973903;
	bh=gv4sxLB+dUGYgWM8i6PADiXFs6JumMaFrt5jHzmiL8Q=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 MIME-Version;
	b=GQ5tjmzUU+uVz4+PiK1M+HcGmJBKBNjP3lpmC6/GPANzTqVSXVOIAdagGcV0a3NF2
	 CzBD43hkQTVtIg/olUBFpB2szRzBkI+8HEQDimMqYlz6a2yu1naltwAFBl3askEduO
	 7GAEjQGV7cWcqS8YqJzXVm/tmLOM9AFZRtTHWMWI=
Received: from SP1-EX07VS02.ds.corp.yahoo.com ([216.252.116.135]) by
 SP1-EX07CAS02.ds.corp.yahoo.com ([216.252.116.167]) with mapi; Mon, 13 Jun
 2011 07:05:03 -0700
From: Robert Evans <evans@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 13 Jun 2011 07:05:01 -0700
Subject: Re: HADOOP-7106 (project unsplit) this weekend
Thread-Topic: HADOOP-7106 (project unsplit) this weekend
Thread-Index: AcwpWiyLtLgAt/gQSPmqE/fuBKRFlAAeLYm0
Message-ID: <CA1B833D.24E16%evans@yahoo-inc.com>
In-Reply-To: <BANLkTikU1LxFc6aF_dV6Fbu4E-fxf3c7ww@mail.gmail.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_CA1B833D24E16evansyahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_CA1B833D24E16evansyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Could someone unlock some of these branches for anonymous read only checkou=
t? At least with MR-279 I get a 403 forbidden error when I try to check out=
.

--Bobby

On 6/12/11 6:38 PM, "Todd Lipcon" <todd@cloudera.com> wrote:

OK, this seems to have succeeded without any big problems!

I've re-enabled the git mirrors and the hudson builds. Feel free to commit
to the new trees.

Here are some instructions for the migration:

=3D=3D=3D SVN users =3D=3D=3D

Next time you "svn up" in your "common" working directory you'll end up
seeing the combined tree - ie a mapreduce/, hdfs/, and common/ subdirectory=
.
This is probably the easiest place from which to work, now. The URLs for th=
e
combined SVN trees are:

trunk: https://svn.apache.org/repos/asf/hadoop/common/trunk/
branch-0.22:
http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22
branch-0.21:
http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21
yahoo-merge:
http://svn.apache.org/repos/asf/hadoop/common/branches/yahoo-merge
  (this one has the yahoo-merge branches from common, hdfs, and mapred)
MR-279: http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
  (this one has the yahoo-merge common and hdfs, and the MR-279 mapred)

The same kind of thing happened for HDFS-1073 and branch-0.21-old.
Pre-project-split branches like branch-0.20 should have remained untouched.

You can proceed to delete your checkouts of the individual mapred and hdfs
trees, since they exist within the combined trees above. If for some reason
you prefer to 'svn switch' an old MR or HDFS-specific checkout to point to
its new location, you can use the following incantation:
svn sw $(svn info | grep URL | awk '{print $2}' | sed
's,\(hdfs\|mapreduce\|common\)/\(.*\),common/\2/\1,')

=3D=3D=3D Git Users =3D=3D=3D
The git mirrors of the above 7 branches should now have a set of 4 commits
near the top that look like this:

Merge: 928d485 cd66945 77f628f
Author: Todd Lipcon <todd@apache.org>
Date:   Sun Jun 12 22:53:28 2011 +0000

    HADOOP-7106. Reorganize SVN layout to combine HDFS, Common, and MR in a
single tree (project unsplit)

    git-svn-id:
https://svn.apache.org/repos/asf/hadoop/common/trunk@113499413f79535-47bb-0=
310-9956-ffa450edef68

commit 77f628ff5925c25ba2ee4ce14590789eb2e7b85b
Author: Todd Lipcon <todd@apache.org>
Date:   Sun Jun 12 22:53:27 2011 +0000

    Relocate mapreduce into mapreduce/

commit cd66945f62635f589ff93468e94c0039684a8b6d
Author: Todd Lipcon <todd@apache.org>
Date:   Sun Jun 12 22:53:26 2011 +0000

    Relocate hdfs into hdfs/

commit 928d485e2743115fe37f9d123ce9a635c5afb91a
Author: Todd Lipcon <todd@apache.org>
Date:   Sun Jun 12 22:53:25 2011 +0000

    Relocate common into common/

The first of these 4 is a 3-parent "octopus" merge commit of the
pre-project-unsplit branches. In theory, git is smart enough to track
changes through this merge, so long as you pass the right flags (eg
--follow). For example:

todd@todd-w510:~/git/hadoop-common$ git log --pretty=3Doneline --abbrev-com=
mit
--follow mapreduce/src/java/org/apache/hadoop/mapred/JobTracker.java | head
-10
77f628f Relocate mapreduce into mapreduce/
90df0cb MAPREDUCE-2455. Remove deprecated JobTracker.State in favour of
JobTrackerStatus.
ca2aba0 MAPREDUCE-2490. Add logging to graylist and blacklist activity to
aid diagnosis of related issues. Contributed by Jonathan Eagles
32aaa2a MAPREDUCE-2515. MapReduce code references some deprecated options.
Contributed by Ari Rabkin.

If you want to be able to have git follow renames all the way through the
project split back to the beginning of time, put the following in
hadoop-common/.git/info/grafts:
5128a9a453d64bfe1ed978cf9ffed27985eeef36
6c16dc8cf2b28818c852e95302920a278d07ad0c
6a3ac690e493c7da45bbf2ae2054768c427fd0e1
6c16dc8cf2b28818c852e95302920a278d07ad0c
546d96754ffee3142bcbbf4563c624c053d0ed0d
6c16dc8cf2b28818c852e95302920a278d07ad0c

In terms of rebasing git branches, git is actually pretty smart. For
example, I have a local "HDFS-1073" branch in my hdfs repo. To transition i=
t
to the new combined repo, I did the following:

# Add my project-split hdfs git repo as a remote:
git remote add splithdfs /home/todd/git/hadoop-hdfs/
git fetch splithdfs

# Checkout a branch in my combined repo
git checkout -b HDFS-1073 splithdfs/HDFS-1073

# Rebase it on the combined 1073 branch
git rebase origin/HDFS-1073

...and it actually applies my patches inside the appropriate subdirectory (=
I
was surprised and impressed by this!)
If the branch you're rebasing has added or moved files, it might not be
smart enough and you'll have to manually rename them in your branch inside
of the appropriate subtree.. but for simple patches this seems to work. For
less simple things, the best bet may be to use "git filter-branch" on the
patch series to relocate it inside a subdirectory, and then try to rebase.
Let me know if you need a hand with any git cleanup, happy to help.


=3D=3D Outstanding issues =3D=3D

The one outstanding issue I'm aware of is that the test-patch builds should
be smart enough to be able to deal with patches that are relative to the
combined root instead of the original project. Right now, if you export a
diff from git, it will include "hdfs/" or "mapreduce/" in the changed file
names, and the QA bot won't know how to apply it. The workaround for this i=
s
to change directory into the relative subproject dir, and then pass
"--relative" to "git diff" or "git show", for example:

todd@todd-w510:~/git/hadoop-common/mapreduce$ git diff --relative
--no-prefix
diff --git CHANGES.txt CHANGES.txt
...


I imagine there are probably some other things that fell through the cracks=
.
Please get in touch if there's anything that seems amiss.

-Todd


On Sun, Jun 12, 2011 at 2:50 PM, Todd Lipcon <todd@cloudera.com> wrote:

> All of the nits I ran into should be resolved and we should be good to go=
.
> I will start this in just about 10 minutes (3pm PST).
>
> ***Please hold all commits until further notice!*** I anticipate that thi=
s
> should take under an hour, but if there are any bumps along the way it mi=
ght
> stretch into the evening. I'll send out an "all clear" email when things =
are
> ready to go on the new layout.
>
> I've disabled all of the Hudson builds for now and will be re-enabling th=
em
> one by one after reconfiguring their SVN URLs.
>
> -Todd
>
> On Sat, Jun 11, 2011 at 8:25 PM, Todd Lipcon <todd@cloudera.com> wrote:
>
>> Hi all,
>>
>> I'm figuring out one more small nit I noticed in my testing this evening=
.
>> Hopefully I will figure out what's going wrong and be ready to press the=
 big
>> button tomorrow.
>>
>> Assuming I don't have to "abort mission", my hope is to do this at aroun=
d
>> 3PM PST tomorrow (Sunday). I'll send out a message asking folks to pleas=
e
>> hold commits to all branches while the move is in progress.
>>
>> Thanks
>> -Todd
>>
>>
>> On Fri, Jun 10, 2011 at 11:20 AM, Todd Lipcon <todd@cloudera.com> wrote:
>>
>>> Hi all,
>>>
>>> Pending any unforeseen issues, I am planning on committing HADOOP-7106
>>> this weekend. I have the credentials from Jukka to take care of the git
>>> trees as well, and have done a "practice" move several times on a local
>>> mirror of the svn.
>>>
>>> I'll send out an announcement of the exact time in advance of when I
>>> actually do the commit.
>>>
>>> Thanks
>>> -Todd
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>>
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>



--
Todd Lipcon
Software Engineer, Cloudera


--_000_CA1B833D24E16evansyahooinccom_--

From general-return-3669-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 14:52:49 2011
Return-Path: <general-return-3669-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 874656685
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 14:52:49 +0000 (UTC)
Received: (qmail 83365 invoked by uid 500); 13 Jun 2011 14:52:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83314 invoked by uid 500); 13 Jun 2011 14:52:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83306 invoked by uid 99); 13 Jun 2011 14:52:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 14:52:47 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 14:52:43 +0000
Received: by bwz8 with SMTP id 8so4352079bwz.35
        for <general@hadoop.apache.org>; Mon, 13 Jun 2011 07:52:21 -0700 (PDT)
Received: by 10.204.74.70 with SMTP id t6mr1979167bkj.21.1307976741204; Mon,
 13 Jun 2011 07:52:21 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.79 with HTTP; Mon, 13 Jun 2011 07:52:01 -0700 (PDT)
In-Reply-To: <CA1B833D.24E16%evans@yahoo-inc.com>
References: <BANLkTikU1LxFc6aF_dV6Fbu4E-fxf3c7ww@mail.gmail.com> <CA1B833D.24E16%evans@yahoo-inc.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 13 Jun 2011 07:52:01 -0700
Message-ID: <BANLkTimEPjp=MG1v-+0Qkr8b2UormUPTtw@mail.gmail.com>
Subject: Re: HADOOP-7106 (project unsplit) this weekend
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d7847908b55304a599117b

--0016e6d7847908b55304a599117b
Content-Type: text/plain; charset=ISO-8859-1

Hey Robert,

It seems to be working for me... what URL are you trying to check out?
I moved aside my ~/.subversion dir, and then did:
$ svn co http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/

Thanks
-Todd

On Mon, Jun 13, 2011 at 7:05 AM, Robert Evans <evans@yahoo-inc.com> wrote:

> Could someone unlock some of these branches for anonymous read only
> checkout? At least with MR-279 I get a 403 forbidden error when I try to
> check out.
>
> --Bobby
>
> On 6/12/11 6:38 PM, "Todd Lipcon" <todd@cloudera.com> wrote:
>
> OK, this seems to have succeeded without any big problems!
>
> I've re-enabled the git mirrors and the hudson builds. Feel free to commit
> to the new trees.
>
> Here are some instructions for the migration:
>
> === SVN users ===
>
> Next time you "svn up" in your "common" working directory you'll end up
> seeing the combined tree - ie a mapreduce/, hdfs/, and common/
> subdirectory.
> This is probably the easiest place from which to work, now. The URLs for
> the
> combined SVN trees are:
>
> trunk: https://svn.apache.org/repos/asf/hadoop/common/trunk/
> branch-0.22:
> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22
> branch-0.21:
> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21
> yahoo-merge:
> http://svn.apache.org/repos/asf/hadoop/common/branches/yahoo-merge
>  (this one has the yahoo-merge branches from common, hdfs, and mapred)
> MR-279: http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
>  (this one has the yahoo-merge common and hdfs, and the MR-279 mapred)
>
> The same kind of thing happened for HDFS-1073 and branch-0.21-old.
> Pre-project-split branches like branch-0.20 should have remained untouched.
>
> You can proceed to delete your checkouts of the individual mapred and hdfs
> trees, since they exist within the combined trees above. If for some reason
> you prefer to 'svn switch' an old MR or HDFS-specific checkout to point to
> its new location, you can use the following incantation:
> svn sw $(svn info | grep URL | awk '{print $2}' | sed
> 's,\(hdfs\|mapreduce\|common\)/\(.*\),common/\2/\1,')
>
> === Git Users ===
> The git mirrors of the above 7 branches should now have a set of 4 commits
> near the top that look like this:
>
> Merge: 928d485 cd66945 77f628f
> Author: Todd Lipcon <todd@apache.org>
> Date:   Sun Jun 12 22:53:28 2011 +0000
>
>    HADOOP-7106. Reorganize SVN layout to combine HDFS, Common, and MR in a
> single tree (project unsplit)
>
>    git-svn-id:
> https://svn.apache.org/repos/asf/hadoop/common/trunk@113499413f79535-47bb-<https://svn.apache.org/repos/asf/hadoop/common/trunk@113499413f79535-47bb-0310-9956-ffa450edef68>
> 0310-9956-ffa450edef68
>
> commit 77f628ff5925c25ba2ee4ce14590789eb2e7b85b
> Author: Todd Lipcon <todd@apache.org>
> Date:   Sun Jun 12 22:53:27 2011 +0000
>
>    Relocate mapreduce into mapreduce/
>
> commit cd66945f62635f589ff93468e94c0039684a8b6d
> Author: Todd Lipcon <todd@apache.org>
> Date:   Sun Jun 12 22:53:26 2011 +0000
>
>    Relocate hdfs into hdfs/
>
> commit 928d485e2743115fe37f9d123ce9a635c5afb91a
> Author: Todd Lipcon <todd@apache.org>
> Date:   Sun Jun 12 22:53:25 2011 +0000
>
>    Relocate common into common/
>
> The first of these 4 is a 3-parent "octopus" merge commit of the
> pre-project-unsplit branches. In theory, git is smart enough to track
> changes through this merge, so long as you pass the right flags (eg
> --follow). For example:
>
> todd@todd-w510:~/git/hadoop-common$ git log --pretty=oneline
> --abbrev-commit
> --follow mapreduce/src/java/org/apache/hadoop/mapred/JobTracker.java | head
> -10
> 77f628f Relocate mapreduce into mapreduce/
> 90df0cb MAPREDUCE-2455. Remove deprecated JobTracker.State in favour of
> JobTrackerStatus.
> ca2aba0 MAPREDUCE-2490. Add logging to graylist and blacklist activity to
> aid diagnosis of related issues. Contributed by Jonathan Eagles
> 32aaa2a MAPREDUCE-2515. MapReduce code references some deprecated options.
> Contributed by Ari Rabkin.
>
> If you want to be able to have git follow renames all the way through the
> project split back to the beginning of time, put the following in
> hadoop-common/.git/info/grafts:
> 5128a9a453d64bfe1ed978cf9ffed27985eeef36
> 6c16dc8cf2b28818c852e95302920a278d07ad0c
> 6a3ac690e493c7da45bbf2ae2054768c427fd0e1
> 6c16dc8cf2b28818c852e95302920a278d07ad0c
> 546d96754ffee3142bcbbf4563c624c053d0ed0d
> 6c16dc8cf2b28818c852e95302920a278d07ad0c
>
> In terms of rebasing git branches, git is actually pretty smart. For
> example, I have a local "HDFS-1073" branch in my hdfs repo. To transition
> it
> to the new combined repo, I did the following:
>
> # Add my project-split hdfs git repo as a remote:
> git remote add splithdfs /home/todd/git/hadoop-hdfs/
> git fetch splithdfs
>
> # Checkout a branch in my combined repo
> git checkout -b HDFS-1073 splithdfs/HDFS-1073
>
> # Rebase it on the combined 1073 branch
> git rebase origin/HDFS-1073
>
> ...and it actually applies my patches inside the appropriate subdirectory
> (I
> was surprised and impressed by this!)
> If the branch you're rebasing has added or moved files, it might not be
> smart enough and you'll have to manually rename them in your branch inside
> of the appropriate subtree.. but for simple patches this seems to work. For
> less simple things, the best bet may be to use "git filter-branch" on the
> patch series to relocate it inside a subdirectory, and then try to rebase.
> Let me know if you need a hand with any git cleanup, happy to help.
>
>
> == Outstanding issues ==
>
> The one outstanding issue I'm aware of is that the test-patch builds should
> be smart enough to be able to deal with patches that are relative to the
> combined root instead of the original project. Right now, if you export a
> diff from git, it will include "hdfs/" or "mapreduce/" in the changed file
> names, and the QA bot won't know how to apply it. The workaround for this
> is
> to change directory into the relative subproject dir, and then pass
> "--relative" to "git diff" or "git show", for example:
>
> todd@todd-w510:~/git/hadoop-common/mapreduce$ git diff --relative
> --no-prefix
> diff --git CHANGES.txt CHANGES.txt
> ...
>
>
> I imagine there are probably some other things that fell through the
> cracks.
> Please get in touch if there's anything that seems amiss.
>
> -Todd
>
>
> On Sun, Jun 12, 2011 at 2:50 PM, Todd Lipcon <todd@cloudera.com> wrote:
>
> > All of the nits I ran into should be resolved and we should be good to
> go.
> > I will start this in just about 10 minutes (3pm PST).
> >
> > ***Please hold all commits until further notice!*** I anticipate that
> this
> > should take under an hour, but if there are any bumps along the way it
> might
> > stretch into the evening. I'll send out an "all clear" email when things
> are
> > ready to go on the new layout.
> >
> > I've disabled all of the Hudson builds for now and will be re-enabling
> them
> > one by one after reconfiguring their SVN URLs.
> >
> > -Todd
> >
> > On Sat, Jun 11, 2011 at 8:25 PM, Todd Lipcon <todd@cloudera.com> wrote:
> >
> >> Hi all,
> >>
> >> I'm figuring out one more small nit I noticed in my testing this
> evening.
> >> Hopefully I will figure out what's going wrong and be ready to press the
> big
> >> button tomorrow.
> >>
> >> Assuming I don't have to "abort mission", my hope is to do this at
> around
> >> 3PM PST tomorrow (Sunday). I'll send out a message asking folks to
> please
> >> hold commits to all branches while the move is in progress.
> >>
> >> Thanks
> >> -Todd
> >>
> >>
> >> On Fri, Jun 10, 2011 at 11:20 AM, Todd Lipcon <todd@cloudera.com>
> wrote:
> >>
> >>> Hi all,
> >>>
> >>> Pending any unforeseen issues, I am planning on committing HADOOP-7106
> >>> this weekend. I have the credentials from Jukka to take care of the git
> >>> trees as well, and have done a "practice" move several times on a local
> >>> mirror of the svn.
> >>>
> >>> I'll send out an announcement of the exact time in advance of when I
> >>> actually do the commit.
> >>>
> >>> Thanks
> >>> -Todd
> >>> --
> >>> Todd Lipcon
> >>> Software Engineer, Cloudera
> >>>
> >>
> >>
> >>
> >> --
> >> Todd Lipcon
> >> Software Engineer, Cloudera
> >>
> >
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

--0016e6d7847908b55304a599117b--

From general-return-3670-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 14:58:18 2011
Return-Path: <general-return-3670-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 18D2467B4
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 14:58:18 +0000 (UTC)
Received: (qmail 94940 invoked by uid 500); 13 Jun 2011 14:58:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94885 invoked by uid 500); 13 Jun 2011 14:58:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94877 invoked by uid 99); 13 Jun 2011 14:58:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 14:58:16 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 14:58:09 +0000
Received: by bwz8 with SMTP id 8so4360872bwz.35
        for <general@hadoop.apache.org>; Mon, 13 Jun 2011 07:57:49 -0700 (PDT)
Received: by 10.205.35.1 with SMTP id su1mr4688732bkb.129.1307977069151; Mon,
 13 Jun 2011 07:57:49 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.79 with HTTP; Mon, 13 Jun 2011 07:57:29 -0700 (PDT)
In-Reply-To: <BANLkTimEPjp=MG1v-+0Qkr8b2UormUPTtw@mail.gmail.com>
References: <BANLkTikU1LxFc6aF_dV6Fbu4E-fxf3c7ww@mail.gmail.com>
 <CA1B833D.24E16%evans@yahoo-inc.com> <BANLkTimEPjp=MG1v-+0Qkr8b2UormUPTtw@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 13 Jun 2011 07:57:29 -0700
Message-ID: <BANLkTikLBpvXRdhKu-ukc-m-K1K1FNXrXg@mail.gmail.com>
Subject: Re: HADOOP-7106 (project unsplit) this weekend
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec52c64df94c4f704a599246d
X-Virus-Checked: Checked by ClamAV on apache.org

--bcaec52c64df94c4f704a599246d
Content-Type: text/plain; charset=ISO-8859-1

Hmm, as I got farther down my email this morning, I saw some complaints that
HBase's SVN was giving 403s as well, this morning. Perhaps there is some
ASF-wide issue going on, completely unrelated to the Hadoop changes over the
weekend?

-Todd

On Mon, Jun 13, 2011 at 7:52 AM, Todd Lipcon <todd@cloudera.com> wrote:

> Hey Robert,
>
> It seems to be working for me... what URL are you trying to check out?
> I moved aside my ~/.subversion dir, and then did:
> $ svn co http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/
>
> Thanks
> -Todd
>
> On Mon, Jun 13, 2011 at 7:05 AM, Robert Evans <evans@yahoo-inc.com> wrote:
>
>> Could someone unlock some of these branches for anonymous read only
>> checkout? At least with MR-279 I get a 403 forbidden error when I try to
>> check out.
>>
>> --Bobby
>>
>> On 6/12/11 6:38 PM, "Todd Lipcon" <todd@cloudera.com> wrote:
>>
>> OK, this seems to have succeeded without any big problems!
>>
>> I've re-enabled the git mirrors and the hudson builds. Feel free to commit
>> to the new trees.
>>
>> Here are some instructions for the migration:
>>
>> === SVN users ===
>>
>> Next time you "svn up" in your "common" working directory you'll end up
>> seeing the combined tree - ie a mapreduce/, hdfs/, and common/
>> subdirectory.
>> This is probably the easiest place from which to work, now. The URLs for
>> the
>> combined SVN trees are:
>>
>> trunk: https://svn.apache.org/repos/asf/hadoop/common/trunk/
>> branch-0.22:
>> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22
>> branch-0.21<http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22branch-0.21>
>> :
>> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21
>> yahoo-merge<http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21yahoo-merge>
>> :
>> http://svn.apache.org/repos/asf/hadoop/common/branches/yahoo-merge
>>  (this one has the yahoo-merge branches from common, hdfs, and mapred)
>> MR-279: http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
>>  (this one has the yahoo-merge common and hdfs, and the MR-279 mapred)
>>
>> The same kind of thing happened for HDFS-1073 and branch-0.21-old.
>> Pre-project-split branches like branch-0.20 should have remained
>> untouched.
>>
>> You can proceed to delete your checkouts of the individual mapred and hdfs
>> trees, since they exist within the combined trees above. If for some
>> reason
>> you prefer to 'svn switch' an old MR or HDFS-specific checkout to point to
>> its new location, you can use the following incantation:
>> svn sw $(svn info | grep URL | awk '{print $2}' | sed
>> 's,\(hdfs\|mapreduce\|common\)/\(.*\),common/\2/\1,')
>>
>> === Git Users ===
>> The git mirrors of the above 7 branches should now have a set of 4 commits
>> near the top that look like this:
>>
>> Merge: 928d485 cd66945 77f628f
>> Author: Todd Lipcon <todd@apache.org>
>> Date:   Sun Jun 12 22:53:28 2011 +0000
>>
>>    HADOOP-7106. Reorganize SVN layout to combine HDFS, Common, and MR in a
>> single tree (project unsplit)
>>
>>    git-svn-id:
>> https://svn.apache.org/repos/asf/hadoop/common/trunk@113499413f79535-47bb-<https://svn.apache.org/repos/asf/hadoop/common/trunk@113499413f79535-47bb-0310-9956-ffa450edef68>
>> 0310-9956-ffa450edef68
>>
>> commit 77f628ff5925c25ba2ee4ce14590789eb2e7b85b
>> Author: Todd Lipcon <todd@apache.org>
>> Date:   Sun Jun 12 22:53:27 2011 +0000
>>
>>    Relocate mapreduce into mapreduce/
>>
>> commit cd66945f62635f589ff93468e94c0039684a8b6d
>> Author: Todd Lipcon <todd@apache.org>
>> Date:   Sun Jun 12 22:53:26 2011 +0000
>>
>>    Relocate hdfs into hdfs/
>>
>> commit 928d485e2743115fe37f9d123ce9a635c5afb91a
>> Author: Todd Lipcon <todd@apache.org>
>> Date:   Sun Jun 12 22:53:25 2011 +0000
>>
>>    Relocate common into common/
>>
>> The first of these 4 is a 3-parent "octopus" merge commit of the
>> pre-project-unsplit branches. In theory, git is smart enough to track
>> changes through this merge, so long as you pass the right flags (eg
>> --follow). For example:
>>
>> todd@todd-w510:~/git/hadoop-common$ git log --pretty=oneline
>> --abbrev-commit
>> --follow mapreduce/src/java/org/apache/hadoop/mapred/JobTracker.java |
>> head
>> -10
>> 77f628f Relocate mapreduce into mapreduce/
>> 90df0cb MAPREDUCE-2455. Remove deprecated JobTracker.State in favour of
>> JobTrackerStatus.
>> ca2aba0 MAPREDUCE-2490. Add logging to graylist and blacklist activity to
>> aid diagnosis of related issues. Contributed by Jonathan Eagles
>> 32aaa2a MAPREDUCE-2515. MapReduce code references some deprecated options.
>> Contributed by Ari Rabkin.
>>
>> If you want to be able to have git follow renames all the way through the
>> project split back to the beginning of time, put the following in
>> hadoop-common/.git/info/grafts:
>> 5128a9a453d64bfe1ed978cf9ffed27985eeef36
>> 6c16dc8cf2b28818c852e95302920a278d07ad0c
>> 6a3ac690e493c7da45bbf2ae2054768c427fd0e1
>> 6c16dc8cf2b28818c852e95302920a278d07ad0c
>> 546d96754ffee3142bcbbf4563c624c053d0ed0d
>> 6c16dc8cf2b28818c852e95302920a278d07ad0c
>>
>> In terms of rebasing git branches, git is actually pretty smart. For
>> example, I have a local "HDFS-1073" branch in my hdfs repo. To transition
>> it
>> to the new combined repo, I did the following:
>>
>> # Add my project-split hdfs git repo as a remote:
>> git remote add splithdfs /home/todd/git/hadoop-hdfs/
>> git fetch splithdfs
>>
>> # Checkout a branch in my combined repo
>> git checkout -b HDFS-1073 splithdfs/HDFS-1073
>>
>> # Rebase it on the combined 1073 branch
>> git rebase origin/HDFS-1073
>>
>> ...and it actually applies my patches inside the appropriate subdirectory
>> (I
>> was surprised and impressed by this!)
>> If the branch you're rebasing has added or moved files, it might not be
>> smart enough and you'll have to manually rename them in your branch inside
>> of the appropriate subtree.. but for simple patches this seems to work.
>> For
>> less simple things, the best bet may be to use "git filter-branch" on the
>> patch series to relocate it inside a subdirectory, and then try to rebase.
>> Let me know if you need a hand with any git cleanup, happy to help.
>>
>>
>> == Outstanding issues ==
>>
>> The one outstanding issue I'm aware of is that the test-patch builds
>> should
>> be smart enough to be able to deal with patches that are relative to the
>> combined root instead of the original project. Right now, if you export a
>> diff from git, it will include "hdfs/" or "mapreduce/" in the changed file
>> names, and the QA bot won't know how to apply it. The workaround for this
>> is
>> to change directory into the relative subproject dir, and then pass
>> "--relative" to "git diff" or "git show", for example:
>>
>> todd@todd-w510:~/git/hadoop-common/mapreduce$ git diff --relative
>> --no-prefix
>> diff --git CHANGES.txt CHANGES.txt
>> ...
>>
>>
>> I imagine there are probably some other things that fell through the
>> cracks.
>> Please get in touch if there's anything that seems amiss.
>>
>> -Todd
>>
>>
>> On Sun, Jun 12, 2011 at 2:50 PM, Todd Lipcon <todd@cloudera.com> wrote:
>>
>> > All of the nits I ran into should be resolved and we should be good to
>> go.
>> > I will start this in just about 10 minutes (3pm PST).
>> >
>> > ***Please hold all commits until further notice!*** I anticipate that
>> this
>> > should take under an hour, but if there are any bumps along the way it
>> might
>> > stretch into the evening. I'll send out an "all clear" email when things
>> are
>> > ready to go on the new layout.
>> >
>> > I've disabled all of the Hudson builds for now and will be re-enabling
>> them
>> > one by one after reconfiguring their SVN URLs.
>> >
>> > -Todd
>> >
>> > On Sat, Jun 11, 2011 at 8:25 PM, Todd Lipcon <todd@cloudera.com> wrote:
>> >
>> >> Hi all,
>> >>
>> >> I'm figuring out one more small nit I noticed in my testing this
>> evening.
>> >> Hopefully I will figure out what's going wrong and be ready to press
>> the big
>> >> button tomorrow.
>> >>
>> >> Assuming I don't have to "abort mission", my hope is to do this at
>> around
>> >> 3PM PST tomorrow (Sunday). I'll send out a message asking folks to
>> please
>> >> hold commits to all branches while the move is in progress.
>> >>
>> >> Thanks
>> >> -Todd
>> >>
>> >>
>> >> On Fri, Jun 10, 2011 at 11:20 AM, Todd Lipcon <todd@cloudera.com>
>> wrote:
>> >>
>> >>> Hi all,
>> >>>
>> >>> Pending any unforeseen issues, I am planning on committing HADOOP-7106
>> >>> this weekend. I have the credentials from Jukka to take care of the
>> git
>> >>> trees as well, and have done a "practice" move several times on a
>> local
>> >>> mirror of the svn.
>> >>>
>> >>> I'll send out an announcement of the exact time in advance of when I
>> >>> actually do the commit.
>> >>>
>> >>> Thanks
>> >>> -Todd
>> >>> --
>> >>> Todd Lipcon
>> >>> Software Engineer, Cloudera
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Todd Lipcon
>> >> Software Engineer, Cloudera
>> >>
>> >
>> >
>> >
>> > --
>> > Todd Lipcon
>> > Software Engineer, Cloudera
>> >
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--bcaec52c64df94c4f704a599246d--

From general-return-3671-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 15:06:27 2011
Return-Path: <general-return-3671-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 47B57610E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 15:06:27 +0000 (UTC)
Received: (qmail 13636 invoked by uid 500); 13 Jun 2011 15:06:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13533 invoked by uid 500); 13 Jun 2011 15:06:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13525 invoked by uid 99); 13 Jun 2011 15:06:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 15:06:25 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 15:06:16 +0000
Received: from SP2-EX07CAS02.ds.corp.yahoo.com (sp2-ex07cas02.corp.sp2.yahoo.com [98.137.59.38])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5DF5YiX024771
	for <general@hadoop.apache.org>; Mon, 13 Jun 2011 08:05:34 -0700 (PDT)
Received: from SP2-EX07VS04.ds.corp.yahoo.com ([98.137.59.33]) by
 SP2-EX07CAS02.ds.corp.yahoo.com ([98.137.59.38]) with mapi; Mon, 13 Jun 2011
 08:05:34 -0700
From: Jeffrey Naisbitt <jnaisbit@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 13 Jun 2011 08:05:32 -0700
Subject: Re: HADOOP-7106 (project unsplit) this weekend
Thread-Topic: HADOOP-7106 (project unsplit) this weekend
Thread-Index: AcwpWg7IMsdIjZLTQm2/+zQZPXnG9AAgUgkz
Message-ID: <CA1B916C.78CF%jnaisbit@yahoo-inc.com>
In-Reply-To: <BANLkTikU1LxFc6aF_dV6Fbu4E-fxf3c7ww@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-Entourage/13.9.0.110114
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

When I checkout the yahoo-merge branch, I see these svn externals warnings:
svn: warning: Error handling externals definition for
'yahoo-merge/hdfs/src/test/bin':
svn: warning: URL=20
'https://svn.apache.org/repos/asf/hadoop/common/trunk/src/test/bin' at
revision 1135120 doesn't exist
svn: warning: Error handling externals definition for
'yahoo-merge/mapreduce/src/test/bin':
svn: warning: URL=20
'https://svn.apache.org/repos/asf/hadoop/common/trunk/src/test/bin' at
revision 1135120 doesn't exist

Also, the ant eclipse targets seem to be broken now.  It seems like various
parts of the eclipse target need to be commonized now (the
.eclipse-templates stuff and .classpath, .launches, etc.)

-Jeff



On 6/12/11 6:38 PM, "Todd Lipcon" <todd@cloudera.com> wrote:

> OK, this seems to have succeeded without any big problems!
>=20
> I've re-enabled the git mirrors and the hudson builds. Feel free to commi=
t
> to the new trees.
>=20
> Here are some instructions for the migration:
>=20
> =3D=3D=3D SVN users =3D=3D=3D
>=20
> Next time you "svn up" in your "common" working directory you'll end up
> seeing the combined tree - ie a mapreduce/, hdfs/, and common/ subdirecto=
ry.
> This is probably the easiest place from which to work, now. The URLs for =
the
> combined SVN trees are:
>=20
> trunk: https://svn.apache.org/repos/asf/hadoop/common/trunk/
> branch-0.22:
> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22
> branch-0.21:
> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21
> yahoo-merge:
> http://svn.apache.org/repos/asf/hadoop/common/branches/yahoo-merge
>   (this one has the yahoo-merge branches from common, hdfs, and mapred)
> MR-279: http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
>   (this one has the yahoo-merge common and hdfs, and the MR-279 mapred)
>=20
> The same kind of thing happened for HDFS-1073 and branch-0.21-old.
> Pre-project-split branches like branch-0.20 should have remained untouche=
d.
>=20
> You can proceed to delete your checkouts of the individual mapred and hdf=
s
> trees, since they exist within the combined trees above. If for some reas=
on
> you prefer to 'svn switch' an old MR or HDFS-specific checkout to point t=
o
> its new location, you can use the following incantation:
> svn sw $(svn info | grep URL | awk '{print $2}' | sed
> 's,\(hdfs\|mapreduce\|common\)/\(.*\),common/\2/\1,')
>=20
> =3D=3D=3D Git Users =3D=3D=3D
> The git mirrors of the above 7 branches should now have a set of 4 commit=
s
> near the top that look like this:
>=20
> Merge: 928d485 cd66945 77f628f
> Author: Todd Lipcon <todd@apache.org>
> Date:   Sun Jun 12 22:53:28 2011 +0000
>=20
>     HADOOP-7106. Reorganize SVN layout to combine HDFS, Common, and MR in=
 a
> single tree (project unsplit)
>=20
>     git-svn-id:
> https://svn.apache.org/repos/asf/hadoop/common/trunk@113499413f79535-47bb=
-0310
> -9956-ffa450edef68
>=20
> commit 77f628ff5925c25ba2ee4ce14590789eb2e7b85b
> Author: Todd Lipcon <todd@apache.org>
> Date:   Sun Jun 12 22:53:27 2011 +0000
>=20
>     Relocate mapreduce into mapreduce/
>=20
> commit cd66945f62635f589ff93468e94c0039684a8b6d
> Author: Todd Lipcon <todd@apache.org>
> Date:   Sun Jun 12 22:53:26 2011 +0000
>=20
>     Relocate hdfs into hdfs/
>=20
> commit 928d485e2743115fe37f9d123ce9a635c5afb91a
> Author: Todd Lipcon <todd@apache.org>
> Date:   Sun Jun 12 22:53:25 2011 +0000
>=20
>     Relocate common into common/
>=20
> The first of these 4 is a 3-parent "octopus" merge commit of the
> pre-project-unsplit branches. In theory, git is smart enough to track
> changes through this merge, so long as you pass the right flags (eg
> --follow). For example:
>=20
> todd@todd-w510:~/git/hadoop-common$ git log --pretty=3Doneline --abbrev-c=
ommit
> --follow mapreduce/src/java/org/apache/hadoop/mapred/JobTracker.java | he=
ad
> -10
> 77f628f Relocate mapreduce into mapreduce/
> 90df0cb MAPREDUCE-2455. Remove deprecated JobTracker.State in favour of
> JobTrackerStatus.
> ca2aba0 MAPREDUCE-2490. Add logging to graylist and blacklist activity to
> aid diagnosis of related issues. Contributed by Jonathan Eagles
> 32aaa2a MAPREDUCE-2515. MapReduce code references some deprecated options=
.
> Contributed by Ari Rabkin.
>=20
> If you want to be able to have git follow renames all the way through the
> project split back to the beginning of time, put the following in
> hadoop-common/.git/info/grafts:
> 5128a9a453d64bfe1ed978cf9ffed27985eeef36
> 6c16dc8cf2b28818c852e95302920a278d07ad0c
> 6a3ac690e493c7da45bbf2ae2054768c427fd0e1
> 6c16dc8cf2b28818c852e95302920a278d07ad0c
> 546d96754ffee3142bcbbf4563c624c053d0ed0d
> 6c16dc8cf2b28818c852e95302920a278d07ad0c
>=20
> In terms of rebasing git branches, git is actually pretty smart. For
> example, I have a local "HDFS-1073" branch in my hdfs repo. To transition=
 it
> to the new combined repo, I did the following:
>=20
> # Add my project-split hdfs git repo as a remote:
> git remote add splithdfs /home/todd/git/hadoop-hdfs/
> git fetch splithdfs
>=20
> # Checkout a branch in my combined repo
> git checkout -b HDFS-1073 splithdfs/HDFS-1073
>=20
> # Rebase it on the combined 1073 branch
> git rebase origin/HDFS-1073
>=20
> ...and it actually applies my patches inside the appropriate subdirectory=
 (I
> was surprised and impressed by this!)
> If the branch you're rebasing has added or moved files, it might not be
> smart enough and you'll have to manually rename them in your branch insid=
e
> of the appropriate subtree.. but for simple patches this seems to work. F=
or
> less simple things, the best bet may be to use "git filter-branch" on the
> patch series to relocate it inside a subdirectory, and then try to rebase=
.
> Let me know if you need a hand with any git cleanup, happy to help.
>=20
>=20
> =3D=3D Outstanding issues =3D=3D
>=20
> The one outstanding issue I'm aware of is that the test-patch builds shou=
ld
> be smart enough to be able to deal with patches that are relative to the
> combined root instead of the original project. Right now, if you export a
> diff from git, it will include "hdfs/" or "mapreduce/" in the changed fil=
e
> names, and the QA bot won't know how to apply it. The workaround for this=
 is
> to change directory into the relative subproject dir, and then pass
> "--relative" to "git diff" or "git show", for example:
>=20
> todd@todd-w510:~/git/hadoop-common/mapreduce$ git diff --relative
> --no-prefix
> diff --git CHANGES.txt CHANGES.txt
> ...
>=20
>=20
> I imagine there are probably some other things that fell through the crac=
ks.
> Please get in touch if there's anything that seems amiss.
>=20
> -Todd
>=20
>=20
> On Sun, Jun 12, 2011 at 2:50 PM, Todd Lipcon <todd@cloudera.com> wrote:
>=20
>> All of the nits I ran into should be resolved and we should be good to g=
o.
>> I will start this in just about 10 minutes (3pm PST).
>>=20
>> ***Please hold all commits until further notice!*** I anticipate that th=
is
>> should take under an hour, but if there are any bumps along the way it m=
ight
>> stretch into the evening. I'll send out an "all clear" email when things=
 are
>> ready to go on the new layout.
>>=20
>> I've disabled all of the Hudson builds for now and will be re-enabling t=
hem
>> one by one after reconfiguring their SVN URLs.
>>=20
>> -Todd
>>=20
>> On Sat, Jun 11, 2011 at 8:25 PM, Todd Lipcon <todd@cloudera.com> wrote:
>>=20
>>> Hi all,
>>>=20
>>> I'm figuring out one more small nit I noticed in my testing this evenin=
g.
>>> Hopefully I will figure out what's going wrong and be ready to press th=
e big
>>> button tomorrow.
>>>=20
>>> Assuming I don't have to "abort mission", my hope is to do this at arou=
nd
>>> 3PM PST tomorrow (Sunday). I'll send out a message asking folks to plea=
se
>>> hold commits to all branches while the move is in progress.
>>>=20
>>> Thanks
>>> -Todd
>>>=20
>>>=20
>>> On Fri, Jun 10, 2011 at 11:20 AM, Todd Lipcon <todd@cloudera.com> wrote=
:
>>>=20
>>>> Hi all,
>>>>=20
>>>> Pending any unforeseen issues, I am planning on committing HADOOP-7106
>>>> this weekend. I have the credentials from Jukka to take care of the gi=
t
>>>> trees as well, and have done a "practice" move several times on a loca=
l
>>>> mirror of the svn.
>>>>=20
>>>> I'll send out an announcement of the exact time in advance of when I
>>>> actually do the commit.
>>>>=20
>>>> Thanks
>>>> -Todd
>>>> --
>>>> Todd Lipcon
>>>> Software Engineer, Cloudera
>>>>=20
>>>=20
>>>=20
>>>=20
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>>=20
>>=20
>>=20
>>=20
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>=20
>=20
>=20


From general-return-3672-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 15:12:17 2011
Return-Path: <general-return-3672-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0E91C63EB
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 15:12:17 +0000 (UTC)
Received: (qmail 26911 invoked by uid 500); 13 Jun 2011 15:12:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26828 invoked by uid 500); 13 Jun 2011 15:12:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26820 invoked by uid 99); 13 Jun 2011 15:12:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 15:12:15 +0000
X-ASF-Spam-Status: No, hits=3.3 required=5.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 15:12:09 +0000
Received: from SP1-EX07CAS02.ds.corp.yahoo.com (sp1-ex07cas02.ds.corp.yahoo.com [216.252.116.138])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5DFAthp026465
	for <general@hadoop.apache.org>; Mon, 13 Jun 2011 08:10:55 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1307977855;
	bh=VKyYiNhKxDPB31gHinWA1rm46E+n5vzTFg8OTpxqjeo=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 MIME-Version;
	b=thzTKUAOrz7qNeuCXSMpKbJ2cHhFLtgBGAxHATywpu7lRv4CxJ5RIjippo1eqG7UE
	 V7OjZK2ZVruoxRg852mWn7OEI2K+pAXxsnLEG8ecNVrLn0QtCyAL0jcoTbE+5S6lKq
	 Ac9Yzh77cF/53rMbfAhWSRbhMtXk/s1k+x7STRQs=
Received: from SP1-EX07VS02.ds.corp.yahoo.com ([216.252.116.135]) by
 SP1-EX07CAS02.ds.corp.yahoo.com ([216.252.116.167]) with mapi; Mon, 13 Jun
 2011 08:10:54 -0700
From: Robert Evans <evans@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 13 Jun 2011 08:10:52 -0700
Subject: Re: HADOOP-7106 (project unsplit) this weekend
Thread-Topic: HADOOP-7106 (project unsplit) this weekend
Thread-Index: Acwp2mQs9RKc727US1axwzGEdZpNqAAAbF/l
Message-ID: <CA1B92AD.24F23%evans@yahoo-inc.com>
In-Reply-To: <BANLkTikLBpvXRdhKu-ukc-m-K1K1FNXrXg@mail.gmail.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_CA1B92AD24F23evansyahooinccom_"
MIME-Version: 1.0

--_000_CA1B92AD24F23evansyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Todd,

It seems to be working now, so whatever it was it is now gone.

--Bobby

On 6/13/11 9:57 AM, "Todd Lipcon" <todd@cloudera.com> wrote:

Hmm, as I got farther down my email this morning, I saw some complaints tha=
t
HBase's SVN was giving 403s as well, this morning. Perhaps there is some
ASF-wide issue going on, completely unrelated to the Hadoop changes over th=
e
weekend?

-Todd

On Mon, Jun 13, 2011 at 7:52 AM, Todd Lipcon <todd@cloudera.com> wrote:

> Hey Robert,
>
> It seems to be working for me... what URL are you trying to check out?
> I moved aside my ~/.subversion dir, and then did:
> $ svn co http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/
>
> Thanks
> -Todd
>
> On Mon, Jun 13, 2011 at 7:05 AM, Robert Evans <evans@yahoo-inc.com> wrote=
:
>
>> Could someone unlock some of these branches for anonymous read only
>> checkout? At least with MR-279 I get a 403 forbidden error when I try to
>> check out.
>>
>> --Bobby
>>
>> On 6/12/11 6:38 PM, "Todd Lipcon" <todd@cloudera.com> wrote:
>>
>> OK, this seems to have succeeded without any big problems!
>>
>> I've re-enabled the git mirrors and the hudson builds. Feel free to comm=
it
>> to the new trees.
>>
>> Here are some instructions for the migration:
>>
>> =3D=3D=3D SVN users =3D=3D=3D
>>
>> Next time you "svn up" in your "common" working directory you'll end up
>> seeing the combined tree - ie a mapreduce/, hdfs/, and common/
>> subdirectory.
>> This is probably the easiest place from which to work, now. The URLs for
>> the
>> combined SVN trees are:
>>
>> trunk: https://svn.apache.org/repos/asf/hadoop/common/trunk/
>> branch-0.22:
>> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22
>> branch-0.21<http://svn.apache.org/repos/asf/hadoop/common/branches/branc=
h-0.22branch-0.21>
>> :
>> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21
>> yahoo-merge<http://svn.apache.org/repos/asf/hadoop/common/branches/branc=
h-0.21yahoo-merge>
>> :
>> http://svn.apache.org/repos/asf/hadoop/common/branches/yahoo-merge
>>  (this one has the yahoo-merge branches from common, hdfs, and mapred)
>> MR-279: http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
>>  (this one has the yahoo-merge common and hdfs, and the MR-279 mapred)
>>
>> The same kind of thing happened for HDFS-1073 and branch-0.21-old.
>> Pre-project-split branches like branch-0.20 should have remained
>> untouched.
>>
>> You can proceed to delete your checkouts of the individual mapred and hd=
fs
>> trees, since they exist within the combined trees above. If for some
>> reason
>> you prefer to 'svn switch' an old MR or HDFS-specific checkout to point =
to
>> its new location, you can use the following incantation:
>> svn sw $(svn info | grep URL | awk '{print $2}' | sed
>> 's,\(hdfs\|mapreduce\|common\)/\(.*\),common/\2/\1,')
>>
>> =3D=3D=3D Git Users =3D=3D=3D
>> The git mirrors of the above 7 branches should now have a set of 4 commi=
ts
>> near the top that look like this:
>>
>> Merge: 928d485 cd66945 77f628f
>> Author: Todd Lipcon <todd@apache.org>
>> Date:   Sun Jun 12 22:53:28 2011 +0000
>>
>>    HADOOP-7106. Reorganize SVN layout to combine HDFS, Common, and MR in=
 a
>> single tree (project unsplit)
>>
>>    git-svn-id:
>> https://svn.apache.org/repos/asf/hadoop/common/trunk@113499413f79535-47b=
b-<https://svn.apache.org/repos/asf/hadoop/common/trunk@113499413f79535-47b=
b-0310-9956-ffa450edef68>
>> 0310-9956-ffa450edef68
>>
>> commit 77f628ff5925c25ba2ee4ce14590789eb2e7b85b
>> Author: Todd Lipcon <todd@apache.org>
>> Date:   Sun Jun 12 22:53:27 2011 +0000
>>
>>    Relocate mapreduce into mapreduce/
>>
>> commit cd66945f62635f589ff93468e94c0039684a8b6d
>> Author: Todd Lipcon <todd@apache.org>
>> Date:   Sun Jun 12 22:53:26 2011 +0000
>>
>>    Relocate hdfs into hdfs/
>>
>> commit 928d485e2743115fe37f9d123ce9a635c5afb91a
>> Author: Todd Lipcon <todd@apache.org>
>> Date:   Sun Jun 12 22:53:25 2011 +0000
>>
>>    Relocate common into common/
>>
>> The first of these 4 is a 3-parent "octopus" merge commit of the
>> pre-project-unsplit branches. In theory, git is smart enough to track
>> changes through this merge, so long as you pass the right flags (eg
>> --follow). For example:
>>
>> todd@todd-w510:~/git/hadoop-common$ git log --pretty=3Doneline
>> --abbrev-commit
>> --follow mapreduce/src/java/org/apache/hadoop/mapred/JobTracker.java |
>> head
>> -10
>> 77f628f Relocate mapreduce into mapreduce/
>> 90df0cb MAPREDUCE-2455. Remove deprecated JobTracker.State in favour of
>> JobTrackerStatus.
>> ca2aba0 MAPREDUCE-2490. Add logging to graylist and blacklist activity t=
o
>> aid diagnosis of related issues. Contributed by Jonathan Eagles
>> 32aaa2a MAPREDUCE-2515. MapReduce code references some deprecated option=
s.
>> Contributed by Ari Rabkin.
>>
>> If you want to be able to have git follow renames all the way through th=
e
>> project split back to the beginning of time, put the following in
>> hadoop-common/.git/info/grafts:
>> 5128a9a453d64bfe1ed978cf9ffed27985eeef36
>> 6c16dc8cf2b28818c852e95302920a278d07ad0c
>> 6a3ac690e493c7da45bbf2ae2054768c427fd0e1
>> 6c16dc8cf2b28818c852e95302920a278d07ad0c
>> 546d96754ffee3142bcbbf4563c624c053d0ed0d
>> 6c16dc8cf2b28818c852e95302920a278d07ad0c
>>
>> In terms of rebasing git branches, git is actually pretty smart. For
>> example, I have a local "HDFS-1073" branch in my hdfs repo. To transitio=
n
>> it
>> to the new combined repo, I did the following:
>>
>> # Add my project-split hdfs git repo as a remote:
>> git remote add splithdfs /home/todd/git/hadoop-hdfs/
>> git fetch splithdfs
>>
>> # Checkout a branch in my combined repo
>> git checkout -b HDFS-1073 splithdfs/HDFS-1073
>>
>> # Rebase it on the combined 1073 branch
>> git rebase origin/HDFS-1073
>>
>> ...and it actually applies my patches inside the appropriate subdirector=
y
>> (I
>> was surprised and impressed by this!)
>> If the branch you're rebasing has added or moved files, it might not be
>> smart enough and you'll have to manually rename them in your branch insi=
de
>> of the appropriate subtree.. but for simple patches this seems to work.
>> For
>> less simple things, the best bet may be to use "git filter-branch" on th=
e
>> patch series to relocate it inside a subdirectory, and then try to rebas=
e.
>> Let me know if you need a hand with any git cleanup, happy to help.
>>
>>
>> =3D=3D Outstanding issues =3D=3D
>>
>> The one outstanding issue I'm aware of is that the test-patch builds
>> should
>> be smart enough to be able to deal with patches that are relative to the
>> combined root instead of the original project. Right now, if you export =
a
>> diff from git, it will include "hdfs/" or "mapreduce/" in the changed fi=
le
>> names, and the QA bot won't know how to apply it. The workaround for thi=
s
>> is
>> to change directory into the relative subproject dir, and then pass
>> "--relative" to "git diff" or "git show", for example:
>>
>> todd@todd-w510:~/git/hadoop-common/mapreduce$ git diff --relative
>> --no-prefix
>> diff --git CHANGES.txt CHANGES.txt
>> ...
>>
>>
>> I imagine there are probably some other things that fell through the
>> cracks.
>> Please get in touch if there's anything that seems amiss.
>>
>> -Todd
>>
>>
>> On Sun, Jun 12, 2011 at 2:50 PM, Todd Lipcon <todd@cloudera.com> wrote:
>>
>> > All of the nits I ran into should be resolved and we should be good to
>> go.
>> > I will start this in just about 10 minutes (3pm PST).
>> >
>> > ***Please hold all commits until further notice!*** I anticipate that
>> this
>> > should take under an hour, but if there are any bumps along the way it
>> might
>> > stretch into the evening. I'll send out an "all clear" email when thin=
gs
>> are
>> > ready to go on the new layout.
>> >
>> > I've disabled all of the Hudson builds for now and will be re-enabling
>> them
>> > one by one after reconfiguring their SVN URLs.
>> >
>> > -Todd
>> >
>> > On Sat, Jun 11, 2011 at 8:25 PM, Todd Lipcon <todd@cloudera.com> wrote=
:
>> >
>> >> Hi all,
>> >>
>> >> I'm figuring out one more small nit I noticed in my testing this
>> evening.
>> >> Hopefully I will figure out what's going wrong and be ready to press
>> the big
>> >> button tomorrow.
>> >>
>> >> Assuming I don't have to "abort mission", my hope is to do this at
>> around
>> >> 3PM PST tomorrow (Sunday). I'll send out a message asking folks to
>> please
>> >> hold commits to all branches while the move is in progress.
>> >>
>> >> Thanks
>> >> -Todd
>> >>
>> >>
>> >> On Fri, Jun 10, 2011 at 11:20 AM, Todd Lipcon <todd@cloudera.com>
>> wrote:
>> >>
>> >>> Hi all,
>> >>>
>> >>> Pending any unforeseen issues, I am planning on committing HADOOP-71=
06
>> >>> this weekend. I have the credentials from Jukka to take care of the
>> git
>> >>> trees as well, and have done a "practice" move several times on a
>> local
>> >>> mirror of the svn.
>> >>>
>> >>> I'll send out an announcement of the exact time in advance of when I
>> >>> actually do the commit.
>> >>>
>> >>> Thanks
>> >>> -Todd
>> >>> --
>> >>> Todd Lipcon
>> >>> Software Engineer, Cloudera
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Todd Lipcon
>> >> Software Engineer, Cloudera
>> >>
>> >
>> >
>> >
>> > --
>> > Todd Lipcon
>> > Software Engineer, Cloudera
>> >
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>



--
Todd Lipcon
Software Engineer, Cloudera


--_000_CA1B92AD24F23evansyahooinccom_--

From general-return-3673-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 15:14:52 2011
Return-Path: <general-return-3673-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1DF9D65B4
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 15:14:52 +0000 (UTC)
Received: (qmail 32161 invoked by uid 500); 13 Jun 2011 15:14:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32093 invoked by uid 500); 13 Jun 2011 15:14:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32085 invoked by uid 99); 13 Jun 2011 15:14:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 15:14:50 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 15:14:43 +0000
Received: by bwz8 with SMTP id 8so4387025bwz.35
        for <general@hadoop.apache.org>; Mon, 13 Jun 2011 08:14:23 -0700 (PDT)
Received: by 10.204.74.70 with SMTP id t6mr2001597bkj.21.1307978063231; Mon,
 13 Jun 2011 08:14:23 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.79 with HTTP; Mon, 13 Jun 2011 08:14:03 -0700 (PDT)
In-Reply-To: <CA1B916C.78CF%jnaisbit@yahoo-inc.com>
References: <BANLkTikU1LxFc6aF_dV6Fbu4E-fxf3c7ww@mail.gmail.com> <CA1B916C.78CF%jnaisbit@yahoo-inc.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 13 Jun 2011 08:14:03 -0700
Message-ID: <BANLkTincXUjoeiXhp8tHn4+qLs642WkbMA@mail.gmail.com>
Subject: Re: HADOOP-7106 (project unsplit) this weekend
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d78479d53ade04a5995fc9
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d78479d53ade04a5995fc9
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Jun 13, 2011 at 8:05 AM, Jeffrey Naisbitt <jnaisbit@yahoo-inc.com>wrote:

> When I checkout the yahoo-merge branch, I see these svn externals warnings:
> svn: warning: Error handling externals definition for
> 'yahoo-merge/hdfs/src/test/bin':
> svn: warning: URL
> 'https://svn.apache.org/repos/asf/hadoop/common/trunk/src/test/bin' at
> revision 1135120 doesn't exist
> svn: warning: Error handling externals definition for
> 'yahoo-merge/mapreduce/src/test/bin':
> svn: warning: URL
> 'https://svn.apache.org/repos/asf/hadoop/common/trunk/src/test/bin' at
> revision 1135120 doesn't exist
>

Oops, sorry about that one. I will take care of that in about 30 minutes
(just headed out the door now to catch a train). If someone else with commit
access wants to, you just need to propset the externals to point to th new
common/trunk/common/src/test/bin instead of the old location.


> Also, the ant eclipse targets seem to be broken now.  It seems like various
> parts of the eclipse target need to be commonized now (the
> .eclipse-templates stuff and .classpath, .launches, etc.)
>
>
Will look into this as well.

-Todd


>
> On 6/12/11 6:38 PM, "Todd Lipcon" <todd@cloudera.com> wrote:
>
> > OK, this seems to have succeeded without any big problems!
> >
> > I've re-enabled the git mirrors and the hudson builds. Feel free to
> commit
> > to the new trees.
> >
> > Here are some instructions for the migration:
> >
> > === SVN users ===
> >
> > Next time you "svn up" in your "common" working directory you'll end up
> > seeing the combined tree - ie a mapreduce/, hdfs/, and common/
> subdirectory.
> > This is probably the easiest place from which to work, now. The URLs for
> the
> > combined SVN trees are:
> >
> > trunk: https://svn.apache.org/repos/asf/hadoop/common/trunk/
> > branch-0.22:
> > http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22
> > branch-0.21:
> > http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21
> > yahoo-merge:
> > http://svn.apache.org/repos/asf/hadoop/common/branches/yahoo-merge
> >   (this one has the yahoo-merge branches from common, hdfs, and mapred)
> > MR-279: http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
> >   (this one has the yahoo-merge common and hdfs, and the MR-279 mapred)
> >
> > The same kind of thing happened for HDFS-1073 and branch-0.21-old.
> > Pre-project-split branches like branch-0.20 should have remained
> untouched.
> >
> > You can proceed to delete your checkouts of the individual mapred and
> hdfs
> > trees, since they exist within the combined trees above. If for some
> reason
> > you prefer to 'svn switch' an old MR or HDFS-specific checkout to point
> to
> > its new location, you can use the following incantation:
> > svn sw $(svn info | grep URL | awk '{print $2}' | sed
> > 's,\(hdfs\|mapreduce\|common\)/\(.*\),common/\2/\1,')
> >
> > === Git Users ===
> > The git mirrors of the above 7 branches should now have a set of 4
> commits
> > near the top that look like this:
> >
> > Merge: 928d485 cd66945 77f628f
> > Author: Todd Lipcon <todd@apache.org>
> > Date:   Sun Jun 12 22:53:28 2011 +0000
> >
> >     HADOOP-7106. Reorganize SVN layout to combine HDFS, Common, and MR in
> a
> > single tree (project unsplit)
> >
> >     git-svn-id:
> >
> https://svn.apache.org/repos/asf/hadoop/common/trunk@113499413f79535-47bb-0310
> > -9956-ffa450edef68
> >
> > commit 77f628ff5925c25ba2ee4ce14590789eb2e7b85b
> > Author: Todd Lipcon <todd@apache.org>
> > Date:   Sun Jun 12 22:53:27 2011 +0000
> >
> >     Relocate mapreduce into mapreduce/
> >
> > commit cd66945f62635f589ff93468e94c0039684a8b6d
> > Author: Todd Lipcon <todd@apache.org>
> > Date:   Sun Jun 12 22:53:26 2011 +0000
> >
> >     Relocate hdfs into hdfs/
> >
> > commit 928d485e2743115fe37f9d123ce9a635c5afb91a
> > Author: Todd Lipcon <todd@apache.org>
> > Date:   Sun Jun 12 22:53:25 2011 +0000
> >
> >     Relocate common into common/
> >
> > The first of these 4 is a 3-parent "octopus" merge commit of the
> > pre-project-unsplit branches. In theory, git is smart enough to track
> > changes through this merge, so long as you pass the right flags (eg
> > --follow). For example:
> >
> > todd@todd-w510:~/git/hadoop-common$ git log --pretty=oneline
> --abbrev-commit
> > --follow mapreduce/src/java/org/apache/hadoop/mapred/JobTracker.java |
> head
> > -10
> > 77f628f Relocate mapreduce into mapreduce/
> > 90df0cb MAPREDUCE-2455. Remove deprecated JobTracker.State in favour of
> > JobTrackerStatus.
> > ca2aba0 MAPREDUCE-2490. Add logging to graylist and blacklist activity to
> > aid diagnosis of related issues. Contributed by Jonathan Eagles
> > 32aaa2a MAPREDUCE-2515. MapReduce code references some deprecated
> options.
> > Contributed by Ari Rabkin.
> >
> > If you want to be able to have git follow renames all the way through the
> > project split back to the beginning of time, put the following in
> > hadoop-common/.git/info/grafts:
> > 5128a9a453d64bfe1ed978cf9ffed27985eeef36
> > 6c16dc8cf2b28818c852e95302920a278d07ad0c
> > 6a3ac690e493c7da45bbf2ae2054768c427fd0e1
> > 6c16dc8cf2b28818c852e95302920a278d07ad0c
> > 546d96754ffee3142bcbbf4563c624c053d0ed0d
> > 6c16dc8cf2b28818c852e95302920a278d07ad0c
> >
> > In terms of rebasing git branches, git is actually pretty smart. For
> > example, I have a local "HDFS-1073" branch in my hdfs repo. To transition
> it
> > to the new combined repo, I did the following:
> >
> > # Add my project-split hdfs git repo as a remote:
> > git remote add splithdfs /home/todd/git/hadoop-hdfs/
> > git fetch splithdfs
> >
> > # Checkout a branch in my combined repo
> > git checkout -b HDFS-1073 splithdfs/HDFS-1073
> >
> > # Rebase it on the combined 1073 branch
> > git rebase origin/HDFS-1073
> >
> > ...and it actually applies my patches inside the appropriate subdirectory
> (I
> > was surprised and impressed by this!)
> > If the branch you're rebasing has added or moved files, it might not be
> > smart enough and you'll have to manually rename them in your branch
> inside
> > of the appropriate subtree.. but for simple patches this seems to work.
> For
> > less simple things, the best bet may be to use "git filter-branch" on the
> > patch series to relocate it inside a subdirectory, and then try to
> rebase.
> > Let me know if you need a hand with any git cleanup, happy to help.
> >
> >
> > == Outstanding issues ==
> >
> > The one outstanding issue I'm aware of is that the test-patch builds
> should
> > be smart enough to be able to deal with patches that are relative to the
> > combined root instead of the original project. Right now, if you export a
> > diff from git, it will include "hdfs/" or "mapreduce/" in the changed
> file
> > names, and the QA bot won't know how to apply it. The workaround for this
> is
> > to change directory into the relative subproject dir, and then pass
> > "--relative" to "git diff" or "git show", for example:
> >
> > todd@todd-w510:~/git/hadoop-common/mapreduce$ git diff --relative
> > --no-prefix
> > diff --git CHANGES.txt CHANGES.txt
> > ...
> >
> >
> > I imagine there are probably some other things that fell through the
> cracks.
> > Please get in touch if there's anything that seems amiss.
> >
> > -Todd
> >
> >
> > On Sun, Jun 12, 2011 at 2:50 PM, Todd Lipcon <todd@cloudera.com> wrote:
> >
> >> All of the nits I ran into should be resolved and we should be good to
> go.
> >> I will start this in just about 10 minutes (3pm PST).
> >>
> >> ***Please hold all commits until further notice!*** I anticipate that
> this
> >> should take under an hour, but if there are any bumps along the way it
> might
> >> stretch into the evening. I'll send out an "all clear" email when things
> are
> >> ready to go on the new layout.
> >>
> >> I've disabled all of the Hudson builds for now and will be re-enabling
> them
> >> one by one after reconfiguring their SVN URLs.
> >>
> >> -Todd
> >>
> >> On Sat, Jun 11, 2011 at 8:25 PM, Todd Lipcon <todd@cloudera.com> wrote:
> >>
> >>> Hi all,
> >>>
> >>> I'm figuring out one more small nit I noticed in my testing this
> evening.
> >>> Hopefully I will figure out what's going wrong and be ready to press
> the big
> >>> button tomorrow.
> >>>
> >>> Assuming I don't have to "abort mission", my hope is to do this at
> around
> >>> 3PM PST tomorrow (Sunday). I'll send out a message asking folks to
> please
> >>> hold commits to all branches while the move is in progress.
> >>>
> >>> Thanks
> >>> -Todd
> >>>
> >>>
> >>> On Fri, Jun 10, 2011 at 11:20 AM, Todd Lipcon <todd@cloudera.com>
> wrote:
> >>>
> >>>> Hi all,
> >>>>
> >>>> Pending any unforeseen issues, I am planning on committing HADOOP-7106
> >>>> this weekend. I have the credentials from Jukka to take care of the
> git
> >>>> trees as well, and have done a "practice" move several times on a
> local
> >>>> mirror of the svn.
> >>>>
> >>>> I'll send out an announcement of the exact time in advance of when I
> >>>> actually do the commit.
> >>>>
> >>>> Thanks
> >>>> -Todd
> >>>> --
> >>>> Todd Lipcon
> >>>> Software Engineer, Cloudera
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Todd Lipcon
> >>> Software Engineer, Cloudera
> >>>
> >>
> >>
> >>
> >> --
> >> Todd Lipcon
> >> Software Engineer, Cloudera
> >>
> >
> >
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

--0016e6d78479d53ade04a5995fc9--

From general-return-3674-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 15:36:32 2011
Return-Path: <general-return-3674-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C5E43666A
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 15:36:32 +0000 (UTC)
Received: (qmail 70346 invoked by uid 500); 13 Jun 2011 15:36:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70304 invoked by uid 500); 13 Jun 2011 15:36:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70296 invoked by uid 99); 13 Jun 2011 15:36:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 15:36:31 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.102 as permitted sender)
Received: from [17.148.16.102] (HELO asmtpout027.mac.com) (17.148.16.102)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 15:36:22 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.32.222.113]
 (166-205-139-215.mobile.mymmode.com [166.205.139.215])
 by asmtp027.mac.com (Oracle Communications Messaging Exchange Server 7u4-18.01
 64bit (built Jul 15 2010)) with ESMTPSA id <0LMQ00HVEJBXC350@asmtp027.mac.com>
 for general@hadoop.apache.org; Mon, 13 Jun 2011 08:36:01 -0700 (PDT)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.4.6813,1.0.148,0.0.0000
 definitions=2011-06-13_04:2011-06-13,2011-06-13,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1106130085
Subject: Re: HADOOP-7106 (project unsplit) this weekend
References: <BANLkTikSmUOd_4BqePmNH1fcPAYS8tbQtw@mail.gmail.com>
 <BANLkTink5G6L55LVR+uKyYXb_ifRTTSkcA@mail.gmail.com>
 <BANLkTiktkoY=Hg7Pc2mvYv5ydAbr-t6MJQ@mail.gmail.com>
 <BANLkTikU1LxFc6aF_dV6Fbu4E-fxf3c7ww@mail.gmail.com>
 <BANLkTi=uD-WUkSDNRJq7B3dh2QBKwq_8Ww@mail.gmail.com>
From: Nigel Daley <ndaley@mac.com>
X-Mailer: iPhone Mail (8J2)
In-reply-to: <BANLkTi=uD-WUkSDNRJq7B3dh2QBKwq_8Ww@mail.gmail.com>
Message-id: <4D158B60-98CF-4BF2-B43A-724B4509596B@mac.com>
Date: Mon, 13 Jun 2011 08:35:52 -0700
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

Yes, many thanks for carrying this over the finish line!

Cheers,
Nige

Sent from my iPhone4

On Jun 13, 2011, at 12:39 AM, Eli Collins <eli@cloudera.com> wrote:

> Nice work Todd.   Thank you for making this happen!
> 
> 
> On Sun, Jun 12, 2011 at 4:38 PM, Todd Lipcon <todd@cloudera.com> wrote:
>> OK, this seems to have succeeded without any big problems!
>> 
>> I've re-enabled the git mirrors and the hudson builds. Feel free to commit
>> to the new trees.
>> 
>> Here are some instructions for the migration:
>> 
>> === SVN users ===
>> 
>> Next time you "svn up" in your "common" working directory you'll end up
>> seeing the combined tree - ie a mapreduce/, hdfs/, and common/ subdirectory.
>> This is probably the easiest place from which to work, now. The URLs for the
>> combined SVN trees are:
>> 
>> trunk: https://svn.apache.org/repos/asf/hadoop/common/trunk/
>> branch-0.22:
>> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22
>> branch-0.21:
>> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21
>> yahoo-merge:
>> http://svn.apache.org/repos/asf/hadoop/common/branches/yahoo-merge
>>  (this one has the yahoo-merge branches from common, hdfs, and mapred)
>> MR-279: http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
>>  (this one has the yahoo-merge common and hdfs, and the MR-279 mapred)
>> 
>> The same kind of thing happened for HDFS-1073 and branch-0.21-old.
>> Pre-project-split branches like branch-0.20 should have remained untouched.
>> 
>> You can proceed to delete your checkouts of the individual mapred and hdfs
>> trees, since they exist within the combined trees above. If for some reason
>> you prefer to 'svn switch' an old MR or HDFS-specific checkout to point to
>> its new location, you can use the following incantation:
>> svn sw $(svn info | grep URL | awk '{print $2}' | sed
>> 's,\(hdfs\|mapreduce\|common\)/\(.*\),common/\2/\1,')
>> 
>> === Git Users ===
>> The git mirrors of the above 7 branches should now have a set of 4 commits
>> near the top that look like this:
>> 
>> Merge: 928d485 cd66945 77f628f
>> Author: Todd Lipcon <todd@apache.org>
>> Date:   Sun Jun 12 22:53:28 2011 +0000
>> 
>>    HADOOP-7106. Reorganize SVN layout to combine HDFS, Common, and MR in a
>> single tree (project unsplit)
>> 
>>    git-svn-id:
>> https://svn.apache.org/repos/asf/hadoop/common/trunk@113499413f79535-47bb-0310-9956-ffa450edef68
>> 
>> commit 77f628ff5925c25ba2ee4ce14590789eb2e7b85b
>> Author: Todd Lipcon <todd@apache.org>
>> Date:   Sun Jun 12 22:53:27 2011 +0000
>> 
>>    Relocate mapreduce into mapreduce/
>> 
>> commit cd66945f62635f589ff93468e94c0039684a8b6d
>> Author: Todd Lipcon <todd@apache.org>
>> Date:   Sun Jun 12 22:53:26 2011 +0000
>> 
>>    Relocate hdfs into hdfs/
>> 
>> commit 928d485e2743115fe37f9d123ce9a635c5afb91a
>> Author: Todd Lipcon <todd@apache.org>
>> Date:   Sun Jun 12 22:53:25 2011 +0000
>> 
>>    Relocate common into common/
>> 
>> The first of these 4 is a 3-parent "octopus" merge commit of the
>> pre-project-unsplit branches. In theory, git is smart enough to track
>> changes through this merge, so long as you pass the right flags (eg
>> --follow). For example:
>> 
>> todd@todd-w510:~/git/hadoop-common$ git log --pretty=oneline --abbrev-commit
>> --follow mapreduce/src/java/org/apache/hadoop/mapred/JobTracker.java | head
>> -10
>> 77f628f Relocate mapreduce into mapreduce/
>> 90df0cb MAPREDUCE-2455. Remove deprecated JobTracker.State in favour of
>> JobTrackerStatus.
>> ca2aba0 MAPREDUCE-2490. Add logging to graylist and blacklist activity to
>> aid diagnosis of related issues. Contributed by Jonathan Eagles
>> 32aaa2a MAPREDUCE-2515. MapReduce code references some deprecated options.
>> Contributed by Ari Rabkin.
>> 
>> If you want to be able to have git follow renames all the way through the
>> project split back to the beginning of time, put the following in
>> hadoop-common/.git/info/grafts:
>> 5128a9a453d64bfe1ed978cf9ffed27985eeef36
>> 6c16dc8cf2b28818c852e95302920a278d07ad0c
>> 6a3ac690e493c7da45bbf2ae2054768c427fd0e1
>> 6c16dc8cf2b28818c852e95302920a278d07ad0c
>> 546d96754ffee3142bcbbf4563c624c053d0ed0d
>> 6c16dc8cf2b28818c852e95302920a278d07ad0c
>> 
>> In terms of rebasing git branches, git is actually pretty smart. For
>> example, I have a local "HDFS-1073" branch in my hdfs repo. To transition it
>> to the new combined repo, I did the following:
>> 
>> # Add my project-split hdfs git repo as a remote:
>> git remote add splithdfs /home/todd/git/hadoop-hdfs/
>> git fetch splithdfs
>> 
>> # Checkout a branch in my combined repo
>> git checkout -b HDFS-1073 splithdfs/HDFS-1073
>> 
>> # Rebase it on the combined 1073 branch
>> git rebase origin/HDFS-1073
>> 
>> ...and it actually applies my patches inside the appropriate subdirectory (I
>> was surprised and impressed by this!)
>> If the branch you're rebasing has added or moved files, it might not be
>> smart enough and you'll have to manually rename them in your branch inside
>> of the appropriate subtree.. but for simple patches this seems to work. For
>> less simple things, the best bet may be to use "git filter-branch" on the
>> patch series to relocate it inside a subdirectory, and then try to rebase.
>> Let me know if you need a hand with any git cleanup, happy to help.
>> 
>> 
>> == Outstanding issues ==
>> 
>> The one outstanding issue I'm aware of is that the test-patch builds should
>> be smart enough to be able to deal with patches that are relative to the
>> combined root instead of the original project. Right now, if you export a
>> diff from git, it will include "hdfs/" or "mapreduce/" in the changed file
>> names, and the QA bot won't know how to apply it. The workaround for this is
>> to change directory into the relative subproject dir, and then pass
>> "--relative" to "git diff" or "git show", for example:
>> 
>> todd@todd-w510:~/git/hadoop-common/mapreduce$ git diff --relative
>> --no-prefix
>> diff --git CHANGES.txt CHANGES.txt
>> ...
>> 
>> 
>> I imagine there are probably some other things that fell through the cracks.
>> Please get in touch if there's anything that seems amiss.
>> 
>> -Todd
>> 
>> 
>> On Sun, Jun 12, 2011 at 2:50 PM, Todd Lipcon <todd@cloudera.com> wrote:
>> 
>>> All of the nits I ran into should be resolved and we should be good to go.
>>> I will start this in just about 10 minutes (3pm PST).
>>> 
>>> ***Please hold all commits until further notice!*** I anticipate that this
>>> should take under an hour, but if there are any bumps along the way it might
>>> stretch into the evening. I'll send out an "all clear" email when things are
>>> ready to go on the new layout.
>>> 
>>> I've disabled all of the Hudson builds for now and will be re-enabling them
>>> one by one after reconfiguring their SVN URLs.
>>> 
>>> -Todd
>>> 
>>> On Sat, Jun 11, 2011 at 8:25 PM, Todd Lipcon <todd@cloudera.com> wrote:
>>> 
>>>> Hi all,
>>>> 
>>>> I'm figuring out one more small nit I noticed in my testing this evening.
>>>> Hopefully I will figure out what's going wrong and be ready to press the big
>>>> button tomorrow.
>>>> 
>>>> Assuming I don't have to "abort mission", my hope is to do this at around
>>>> 3PM PST tomorrow (Sunday). I'll send out a message asking folks to please
>>>> hold commits to all branches while the move is in progress.
>>>> 
>>>> Thanks
>>>> -Todd
>>>> 
>>>> 
>>>> On Fri, Jun 10, 2011 at 11:20 AM, Todd Lipcon <todd@cloudera.com> wrote:
>>>> 
>>>>> Hi all,
>>>>> 
>>>>> Pending any unforeseen issues, I am planning on committing HADOOP-7106
>>>>> this weekend. I have the credentials from Jukka to take care of the git
>>>>> trees as well, and have done a "practice" move several times on a local
>>>>> mirror of the svn.
>>>>> 
>>>>> I'll send out an announcement of the exact time in advance of when I
>>>>> actually do the commit.
>>>>> 
>>>>> Thanks
>>>>> -Todd
>>>>> --
>>>>> Todd Lipcon
>>>>> Software Engineer, Cloudera
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Todd Lipcon
>>>> Software Engineer, Cloudera
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>> 
>> 
>> 
>> 
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>> 

From general-return-3675-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 16:04:44 2011
Return-Path: <general-return-3675-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B9C2B6DF7
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 16:04:44 +0000 (UTC)
Received: (qmail 41040 invoked by uid 500); 13 Jun 2011 16:04:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40969 invoked by uid 500); 13 Jun 2011 16:04:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40957 invoked by uid 99); 13 Jun 2011 16:04:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 16:04:43 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 16:04:37 +0000
Received: by bwz8 with SMTP id 8so4462531bwz.35
        for <general@hadoop.apache.org>; Mon, 13 Jun 2011 09:04:16 -0700 (PDT)
Received: by 10.204.61.197 with SMTP id u5mr2879915bkh.210.1307981056351; Mon,
 13 Jun 2011 09:04:16 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.79 with HTTP; Mon, 13 Jun 2011 09:03:56 -0700 (PDT)
In-Reply-To: <BANLkTincXUjoeiXhp8tHn4+qLs642WkbMA@mail.gmail.com>
References: <BANLkTikU1LxFc6aF_dV6Fbu4E-fxf3c7ww@mail.gmail.com>
 <CA1B916C.78CF%jnaisbit@yahoo-inc.com> <BANLkTincXUjoeiXhp8tHn4+qLs642WkbMA@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 13 Jun 2011 09:03:56 -0700
Message-ID: <BANLkTinVKj9DE+UoLRfhjVNA=mBR0k5yFg@mail.gmail.com>
Subject: Re: HADOOP-7106 (project unsplit) this weekend
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c5a9d03c9fd604a59a12b1

--001636c5a9d03c9fd604a59a12b1
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Jun 13, 2011 at 8:14 AM, Todd Lipcon <todd@cloudera.com> wrote:

>
> Oops, sorry about that one. I will take care of that in about 30 minutes
> (just headed out the door now to catch a train). If someone else with commit
> access wants to, you just need to propset the externals to point to th new
> common/trunk/common/src/test/bin instead of the old location.
>
>
Fixed the svn:externals


>
>> Also, the ant eclipse targets seem to be broken now.  It seems like
>> various
>> parts of the eclipse target need to be commonized now (the
>> .eclipse-templates stuff and .classpath, .launches, etc.)
>>
>>
> Will look into this as well.
>
> Can you explain further what's broken? Are you trying to make a project
that's rooted in the directory that contains common/, mapreduce/, and hdfs/?
I can imagine that wouldn't work, but I'm not sure why it wouldn't work to
continue having three separate projects. Are you using some kind of SVN
integration with Eclipse?

-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

--001636c5a9d03c9fd604a59a12b1--

From general-return-3676-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 16:25:33 2011
Return-Path: <general-return-3676-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A1D3D69C3
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 16:25:33 +0000 (UTC)
Received: (qmail 91997 invoked by uid 500); 13 Jun 2011 16:25:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91940 invoked by uid 500); 13 Jun 2011 16:25:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91932 invoked by uid 99); 13 Jun 2011 16:25:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 16:25:32 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 16:25:23 +0000
Received: from SP2-EX07CAS05.ds.corp.yahoo.com (sp2-ex07cas05.corp.sp2.yahoo.com [98.137.59.39])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p5DGOdX2026059
	for <general@hadoop.apache.org>; Mon, 13 Jun 2011 09:24:39 -0700 (PDT)
Received: from SP2-EX07VS04.ds.corp.yahoo.com ([98.137.59.33]) by
 SP2-EX07CAS05.ds.corp.yahoo.com ([98.137.59.39]) with mapi; Mon, 13 Jun 2011
 09:24:39 -0700
From: Jeffrey Naisbitt <jnaisbit@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 13 Jun 2011 09:24:38 -0700
Subject: Re: HADOOP-7106 (project unsplit) this weekend
Thread-Topic: HADOOP-7106 (project unsplit) this weekend
Thread-Index: Acwp46325uX3cxbWRWG79YHTeQ0sAgAArXNp
Message-ID: <CA1BA3F6.78F3%jnaisbit@yahoo-inc.com>
In-Reply-To: <BANLkTinVKj9DE+UoLRfhjVNA=mBR0k5yFg@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
user-agent: Microsoft-Entourage/13.9.0.110114
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

On 6/13/11 11:03 AM, "Todd Lipcon" <todd@cloudera.com> wrote:

> On Mon, Jun 13, 2011 at 8:14 AM, Todd Lipcon <todd@cloudera.com> wrote:
> Fixed the svn:externals

Thanks! :)

>>=20
>>> Also, the ant eclipse targets seem to be broken now.  It seems like
>>> various
>>> parts of the eclipse target need to be commonized now (the
>>> .eclipse-templates stuff and .classpath, .launches, etc.)
>>>=20
>>>=20
>> Will look into this as well.
>>=20
>> Can you explain further what's broken? Are you trying to make a project
> that's rooted in the directory that contains common/, mapreduce/, and hdf=
s/?
> I can imagine that wouldn't work, but I'm not sure why it wouldn't work t=
o
> continue having three separate projects. Are you using some kind of SVN
> integration with Eclipse?

>>> Also, the ant eclipse targets seem to be broken now.  It seems like
>>> various
>>> parts of the eclipse target need to be commonized now (the
>>> .eclipse-templates stuff and .classpath, .launches, etc.)
>>>=20
>>>=20
>> Will look into this as well.
>>=20
>> Can you explain further what's broken? Are you trying to make a project
> that's rooted in the directory that contains common/, mapreduce/, and hdf=
s/?
> I can imagine that wouldn't work, but I'm not sure why it wouldn't work t=
o
> continue having three separate projects. Are you using some kind of SVN
> integration with Eclipse?

I am using subversive for svn integration.  I was actually trying to use a
single project for the whole thing
(https://svn.apache.org/repos/asf/hadoop/common/trunk)
As you say though, I would think it should still work with three separate
projects, so I'll just go back to that for now.  It would be nice to be abl=
e
to build the whole thing as a single project though (since it's a single
repository in svn now), but that would probably take some extra work - and
would probably make sense in a separate Jira.

Thanks.
-Jeff


From general-return-3677-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 16:51:24 2011
Return-Path: <general-return-3677-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B02596A4A
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 16:51:24 +0000 (UTC)
Received: (qmail 62105 invoked by uid 500); 13 Jun 2011 16:51:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62052 invoked by uid 500); 13 Jun 2011 16:51:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62044 invoked by uid 99); 13 Jun 2011 16:51:23 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 16:51:23 +0000
Received: from localhost (HELO dhcp-02.private.iobm.com) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 16:51:23 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: LimitedPrivate and HBase
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <BANLkTikYuO0nRHz3UWZFZU245dMw82Y08Q@mail.gmail.com>
Date: Mon, 13 Jun 2011 09:51:22 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <AA158FFB-1067-48FF-BDD4-D3FC0B5721C6@apache.org>
References: <457011.690.qm@web65508.mail.ac4.yahoo.com> <F0CA2186-6E63-459D-ABFF-21863EE018E5@yahoo-inc.com> <BANLkTikYuO0nRHz3UWZFZU245dMw82Y08Q@mail.gmail.com>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On Jun 9, 2011, at 10:27 AM, Tom White wrote:
>=20
> We could also change the remaining 6 cases of LimitedPrivate to Public
> (note that they are already annotated Evolving or Unstable), and
> deprecate LimitedPrivate. Would this allay people's concerns?


Thanks for doing the search Tom.

Rather than just flip them all to public, I'd like to propose the =
following:

a) LimitedPrivate APIs to non-Hadoop projects are allowed to exist for 1 =
minor release.
b) When a LimitedPrivate to non-Hadoop project is created, a blocker =
JIRA for the next release is created.
c) That JIRA should be used to determine whether the API should go =
private or public, and if the latter, what interface changes are =
required.

This gives us a roadmap to make determinations on whether the =
LimitedPrivate API was a success or not and gives us a timeline as to =
when they go away, rather than just hanging around forever in a limbo =
state.

I suspect that most of these will go public anyway, but it would be good =
to have some documentation on the how's, why's, etc.  This could also be =
used to allay the fears of a "bad" public interface.=

From general-return-3678-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 17:02:32 2011
Return-Path: <general-return-3678-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0FC3C6932
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 17:02:32 +0000 (UTC)
Received: (qmail 78560 invoked by uid 500); 13 Jun 2011 17:02:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78319 invoked by uid 500); 13 Jun 2011 17:02:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78311 invoked by uid 99); 13 Jun 2011 17:02:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 17:02:30 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 17:02:25 +0000
Received: by bwz8 with SMTP id 8so4553653bwz.35
        for <general@hadoop.apache.org>; Mon, 13 Jun 2011 10:02:04 -0700 (PDT)
Received: by 10.204.71.193 with SMTP id i1mr5074107bkj.102.1307984524177; Mon,
 13 Jun 2011 10:02:04 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.79 with HTTP; Mon, 13 Jun 2011 10:01:44 -0700 (PDT)
In-Reply-To: <CA1BA3F6.78F3%jnaisbit@yahoo-inc.com>
References: <BANLkTinVKj9DE+UoLRfhjVNA=mBR0k5yFg@mail.gmail.com> <CA1BA3F6.78F3%jnaisbit@yahoo-inc.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 13 Jun 2011 10:01:44 -0700
Message-ID: <BANLkTi=R4XV4-0formszUMymD_7h1QO4OQ@mail.gmail.com>
Subject: Re: HADOOP-7106 (project unsplit) this weekend
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c5b6c9ef734c04a59ae0de

--001636c5b6c9ef734c04a59ae0de
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Jun 13, 2011 at 9:24 AM, Jeffrey Naisbitt <jnaisbit@yahoo-inc.com>wrote:

> As you say though, I would think it should still work with three separate
> projects, so I'll just go back to that for now.  It would be nice to be
> able
> to build the whole thing as a single project though (since it's a single
> repository in svn now), but that would probably take some extra work - and
> would probably make sense in a separate Jira.
>

Yep - I think the idea is this will happen once we mavenize everything.
Maven apparently supports the idea of "modules" - basically a recursive
structure in which a top level pom file can build sub-poms, but the subpoms
could also continue to be built independently if necessary. I think the
top-level pom would live in the new root.

-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

--001636c5b6c9ef734c04a59ae0de--

From general-return-3679-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 17:05:33 2011
Return-Path: <general-return-3679-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DAFB863C6
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 17:05:32 +0000 (UTC)
Received: (qmail 83173 invoked by uid 500); 13 Jun 2011 17:05:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83126 invoked by uid 500); 13 Jun 2011 17:05:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83118 invoked by uid 99); 13 Jun 2011 17:05:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 17:05:31 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of tucu@cloudera.com designates 209.85.210.48 as permitted sender)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 17:05:26 +0000
Received: by pzk10 with SMTP id 10so3014702pzk.35
        for <general@hadoop.apache.org>; Mon, 13 Jun 2011 10:05:06 -0700 (PDT)
Received: by 10.143.63.10 with SMTP id q10mr869739wfk.291.1307984706053; Mon,
 13 Jun 2011 10:05:06 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.143.83.16 with HTTP; Mon, 13 Jun 2011 10:04:36 -0700 (PDT)
In-Reply-To: <BANLkTi=R4XV4-0formszUMymD_7h1QO4OQ@mail.gmail.com>
References: <BANLkTinVKj9DE+UoLRfhjVNA=mBR0k5yFg@mail.gmail.com>
 <CA1BA3F6.78F3%jnaisbit@yahoo-inc.com> <BANLkTi=R4XV4-0formszUMymD_7h1QO4OQ@mail.gmail.com>
From: Alejandro Abdelnur <tucu@cloudera.com>
Date: Mon, 13 Jun 2011 10:04:36 -0700
Message-ID: <BANLkTi=yNLZbN86ff3rXc9Y=bg4ib48S1g@mail.gmail.com>
Subject: Re: HADOOP-7106 (project unsplit) this weekend
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015173ffa7cc6a80504a59aebf6

--0015173ffa7cc6a80504a59aebf6
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Jun 13, 2011 at 10:01 AM, Todd Lipcon <todd@cloudera.com> wrote:

> On Mon, Jun 13, 2011 at 9:24 AM, Jeffrey Naisbitt <jnaisbit@yahoo-inc.com
> >wrote:
>
> > As you say though, I would think it should still work with three separate
> > projects, so I'll just go back to that for now.  It would be nice to be
> > able
> > to build the whole thing as a single project though (since it's a single
> > repository in svn now), but that would probably take some extra work -
> and
> > would probably make sense in a separate Jira.
> >
>
> Yep - I think the idea is this will happen once we mavenize everything.
> Maven apparently supports the idea of "modules" - basically a recursive
> structure in which a top level pom file can build sub-poms, but the subpoms
> could also continue to be built independently if necessary. I think the
> top-level pom would live in the new root.
>
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>


That sounds just right!

Thxs.

Alejandro

--0015173ffa7cc6a80504a59aebf6--

From general-return-3680-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 17:09:47 2011
Return-Path: <general-return-3680-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5956866AE
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 17:09:47 +0000 (UTC)
Received: (qmail 97997 invoked by uid 500); 13 Jun 2011 17:09:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97952 invoked by uid 500); 13 Jun 2011 17:09:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97944 invoked by uid 99); 13 Jun 2011 17:09:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 17:09:45 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 17:09:38 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5DH90wE067459
	for <general@hadoop.apache.org>; Mon, 13 Jun 2011 10:09:00 -0700 (PDT)
Received: from SP2-EX07VS04.ds.corp.yahoo.com ([98.137.59.33]) by
 SP2-EX07CAS01.ds.corp.yahoo.com ([98.137.59.37]) with mapi; Mon, 13 Jun 2011
 10:08:59 -0700
From: Jeffrey Naisbitt <jnaisbit@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 13 Jun 2011 10:08:58 -0700
Subject: Re: HADOOP-7106 (project unsplit) this weekend
Thread-Topic: HADOOP-7106 (project unsplit) this weekend
Thread-Index: Acwp67qow1N5lwaxSHmgr5zz32CIZwAANqZ7
Message-ID: <CA1BAE5A.790B%jnaisbit@yahoo-inc.com>
In-Reply-To: <BANLkTi=R4XV4-0formszUMymD_7h1QO4OQ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-Entourage/13.9.0.110114
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

On 6/13/11 12:01 PM, "Todd Lipcon" <todd@cloudera.com> wrote:

> On Mon, Jun 13, 2011 at 9:24 AM, Jeffrey Naisbitt
> <jnaisbit@yahoo-inc.com>wrote:
>=20
>> As you say though, I would think it should still work with three separat=
e
>> projects, so I'll just go back to that for now.  It would be nice to be
>> able
>> to build the whole thing as a single project though (since it's a single
>> repository in svn now), but that would probably take some extra work - a=
nd
>> would probably make sense in a separate Jira.
>>=20
>=20
> Yep - I think the idea is this will happen once we mavenize everything.
> Maven apparently supports the idea of "modules" - basically a recursive
> structure in which a top level pom file can build sub-poms, but the subpo=
ms
> could also continue to be built independently if necessary. I think the
> top-level pom would live in the new root.
>=20
> -Todd

I look forward to the mavenization! :)
I verified the eclipse builds are working fine for me in separate projects
and that the svn:externals work fine on yahoo-merge now.
Thanks for the quick responses and for all the other work on the "unsplit"!
-Jeff


From general-return-3681-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 17:11:04 2011
Return-Path: <general-return-3681-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C245E677C
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 17:11:04 +0000 (UTC)
Received: (qmail 1219 invoked by uid 500); 13 Jun 2011 17:11:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1136 invoked by uid 500); 13 Jun 2011 17:11:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1128 invoked by uid 99); 13 Jun 2011 17:11:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 17:11:03 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.30.16] (HELO qmta01.emeryville.ca.mail.comcast.net) (76.96.30.16)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 17:10:54 +0000
Received: from omta11.emeryville.ca.mail.comcast.net ([76.96.30.36])
	by qmta01.emeryville.ca.mail.comcast.net with comcast
	id vUdF1g00A0mlR8UA1VAYaZ; Mon, 13 Jun 2011 17:10:32 +0000
Received: from fs ([24.4.185.157])
	by omta11.emeryville.ca.mail.comcast.net with comcast
	id vVAZ1g00J3QAh8g8XVAaRP; Mon, 13 Jun 2011 17:10:35 +0000
Received: from mail.boudnik.org (localhost [127.0.0.1])
	by fs (8.14.3/8.14.3/Debian-5+lenny1) with ESMTP id p5DHAVjg031665
	for <general@hadoop.apache.org>; Mon, 13 Jun 2011 10:10:31 -0700
Received: (from cos@localhost)
	by mail.boudnik.org (8.14.3/8.14.3/Submit) id p5DHAUg1031664
	for general@hadoop.apache.org; Mon, 13 Jun 2011 10:10:30 -0700
X-Authentication-Warning: mail.boudnik.org: cos set sender to cos@apache.org using -f
Date: Mon, 13 Jun 2011 10:10:30 -0700
From: Konstantin Boudnik <cos@apache.org>
To: general@hadoop.apache.org
Subject: Re: LimitedPrivate and HBase
Message-ID: <20110613171030.GA31569@linspire.com>
References: <457011.690.qm@web65508.mail.ac4.yahoo.com> <F0CA2186-6E63-459D-ABFF-21863EE018E5@yahoo-inc.com> <BANLkTikYuO0nRHz3UWZFZU245dMw82Y08Q@mail.gmail.com> <AA158FFB-1067-48FF-BDD4-D3FC0B5721C6@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <AA158FFB-1067-48FF-BDD4-D3FC0B5721C6@apache.org>
X-Organization: It's something of 'Cos
User-Agent: Mutt/1.5.18 (2008-05-17)

On Mon, Jun 13, 2011 at 09:51AM, Allen Wittenauer wrote:
> 
> On Jun 9, 2011, at 10:27 AM, Tom White wrote:
> > 
> > We could also change the remaining 6 cases of LimitedPrivate to Public
> > (note that they are already annotated Evolving or Unstable), and
> > deprecate LimitedPrivate. Would this allay people's concerns?
> 
> 
> Thanks for doing the search Tom.
> 
> Rather than just flip them all to public, I'd like to propose the following:
> 
> a) LimitedPrivate APIs to non-Hadoop projects are allowed to exist for 1 minor release.
> b) When a LimitedPrivate to non-Hadoop project is created, a blocker JIRA for the next release is created.
> c) That JIRA should be used to determine whether the API should go private
> or public, and if the latter, what interface changes are required.

Allen, the idea looks reasonable yet an over-complicated for the dev. process.
It seems rather better to stick with original Tom's proposal: e.g. eliminate
LimitedPrivate and use just Private instead: lesser interfaces to
track/decisions to make, more transparent the process will be, IMO.

> This gives us a roadmap to make determinations on whether the LimitedPrivate
> API was a success or not and gives us a timeline as to when they go away,
> rather than just hanging around forever in a limbo state.
> 
> I suspect that most of these will go public anyway, but it would be good to
> have some documentation on the how's, why's, etc.  This could also be used
> to allay the fears of a "bad" public interface.


From general-return-3682-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 18:29:02 2011
Return-Path: <general-return-3682-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 18B5B6490
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 18:29:02 +0000 (UTC)
Received: (qmail 39914 invoked by uid 500); 13 Jun 2011 18:29:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39837 invoked by uid 500); 13 Jun 2011 18:29:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39829 invoked by uid 99); 13 Jun 2011 18:29:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 18:29:00 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 18:28:52 +0000
Received: by bwz8 with SMTP id 8so4682756bwz.35
        for <general@hadoop.apache.org>; Mon, 13 Jun 2011 11:28:32 -0700 (PDT)
Received: by 10.204.233.15 with SMTP id jw15mr4928172bkb.48.1307989712165;
 Mon, 13 Jun 2011 11:28:32 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.79 with HTTP; Mon, 13 Jun 2011 11:28:12 -0700 (PDT)
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 13 Jun 2011 11:28:12 -0700
Message-ID: <BANLkTikGFdEhGz+t+nVAnF8FX-2u-cajCg@mail.gmail.com>
Subject: JIRAs post-"unsplit"
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=485b3979db4629da0b04a59c1665
X-Virus-Checked: Checked by ClamAV on apache.org

--485b3979db4629da0b04a59c1665
Content-Type: text/plain; charset=ISO-8859-1

After the "project unsplit" this weekend, we're now back to a place where we
have a single SVN/git tree that encompasses all of the subprojects. This
opens up the next question: should we merge the JIRAs and allow a single
issue to have a patch which spans projects?

My thoughts are:
- the biggest pain point with the project split is dealing with
cross-project patches
- one of the biggest reasons we did the project split was that the combined
traffic from the HADOOP JIRA was hard to follow for people who really care
about certain subprojects.
- the jira split is a coarse-grained way of allowing people to watch just
the sub-areas they care about.

So, I was thinking the following... what if there were a way to watch JIRAs
based on subtrees? I'm imagining a web page where any community user could
have an account and manage a "watch list" of subtrees. If you want to watch
all MR jiras, you could simply watch mapreduce/*. If you care only about
libhdfs, you could watch hdfs/src/c++/libhdfs, etc. Then a bot would watch
all patches attached to JIRA, and any time a patch is uploaded that touches
something on your watch list, it automatically adds you as a watcher on the
ticket and sends you a notification via email. It would also be easy to set
up a watch based on patch size, for example.

I think even if we don't recombine the JIRAs, this might be a handy way to
cut down on mailing list traffic for contributors who have a more narrow
focus on certain areas of the code.

Does this sound useful? I don't know if/when I'd have time to build such a
thing, but if the community thinks it would be really helpful, I might
become inspired.

-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

--485b3979db4629da0b04a59c1665--

From general-return-3683-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 18:42:55 2011
Return-Path: <general-return-3683-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A89696C7F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 18:42:55 +0000 (UTC)
Received: (qmail 64565 invoked by uid 500); 13 Jun 2011 18:42:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 64462 invoked by uid 500); 13 Jun 2011 18:42:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 64454 invoked by uid 99); 13 Jun 2011 18:42:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 18:42:54 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [98.139.52.230] (HELO nm8-vm0.bullet.mail.ac4.yahoo.com) (98.139.52.230)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 13 Jun 2011 18:42:43 +0000
Received: from [98.139.52.191] by nm8.bullet.mail.ac4.yahoo.com with NNFMP; 13 Jun 2011 18:42:20 -0000
Received: from [98.139.52.148] by tm4.bullet.mail.ac4.yahoo.com with NNFMP; 13 Jun 2011 18:42:20 -0000
Received: from [127.0.0.1] by omp1031.mail.ac4.yahoo.com with NNFMP; 13 Jun 2011 18:42:20 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 658667.55828.bm@omp1031.mail.ac4.yahoo.com
Received: (qmail 53102 invoked by uid 60001); 13 Jun 2011 18:42:20 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1307990540; bh=BeaCJmwBbMfnV26Ht6B4SBbYvGnmXyqSKwQ8LHDy0qM=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=1WoMkaa2iaWYmEBDkr9RW9Ry2YJZDk47ZiZCGAKu/GhNLWfb5jivNtikgp5/8zlHopoJWvSv8ETf6jf+NTmZ4nMFe0ttW2HrNXr0LLcf+k5AuFY0O1TJwGaB61LqWQwEl7ywD4NOzmOF7Nahc3DibpXEFsTqPTspBn0etmSlbEk=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=BXyhiPv9sHNZIc6R/fDXffS+Oi85RkcLu58rhxblkSAlk94ynPr8dlt9QRFsdzNzVMZMqzzU3YhA5/Y+7BOhnnGMTWlPHN8HAcBJshdyjPglVsoiyEtxI3eng3YwJZbzm8mNR1eWb+eq3/k/Rk7anbwgKqK/9dS/UQQOfFSXGNw=;
Message-ID: <550632.48430.qm@web65913.mail.ac4.yahoo.com>
X-YMail-OSG: wToyzyQVM1kzpnw3nh07Nsjkox54g0ptieI5_8APTAxbqRv
 g4w1B9AHbBBbqADlvurF5RaVShrseyf26GtfzICduE.0jyeD7xLtMU.HiBM6
 8PTTn5XRJAG7k6Rx8taPlapHwu7krQu_syaxUZGDSTKPbfcAz8asFfX1oLNX
 PjvDYpGomLAiJRD0Ja0zPCE8uC5FyvQCLgOFc6XNCImOjfVs.zHtC1ov2g7f
 qJJNcV5j8uQ0QsKx_2ajO41lmf_vvpxIILxs88baR2g7tQcCa2ou9rXaBMVg
 CJzkKhoShhFanzrDMkYvi5Sc8tPOaeUjXAcyiNg43ywehAxKJ0PmoZSTCuny
 vKUUkBalDcTbSrWVClnN0dOoIqenb1K33XlNTI__nDnchmObRTCgplq_dk6X
 2kzVL5w3j.oB7twUu82BAcV801Der55VZmA--
Received: from [209.131.62.113] by web65913.mail.ac4.yahoo.com via HTTP; Mon, 13 Jun 2011 11:42:20 PDT
X-Mailer: YahooMailRC/570 YahooMailWebService/0.8.111.304355
References: <BANLkTinVKj9DE+UoLRfhjVNA=mBR0k5yFg@mail.gmail.com> <CA1BA3F6.78F3%jnaisbit@yahoo-inc.com> <BANLkTi=R4XV4-0formszUMymD_7h1QO4OQ@mail.gmail.com>
Date: Mon, 13 Jun 2011 11:42:20 -0700 (PDT)
From: "Tsz Wo \(Nicholas\), Sze" <s29752-hadoopgeneral@yahoo.com>
Subject: Re: HADOOP-7106 (project unsplit) this weekend
To: general@hadoop.apache.org
In-Reply-To: <BANLkTi=R4XV4-0formszUMymD_7h1QO4OQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-816861540-1307990540=:48430"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-816861540-1307990540=:48430
Content-Type: text/plain; charset=us-ascii

Todd,

Great work!


A few minor problems:
(1) I had committed MAPREDUCE-2588, however, an commit email was sent to both 
common-commits@ and mapreduce-commits@.
(2) hadoop/site becomes an empty directory.
(3) There are svn properties in hadoop/common/trunk/

(2) and (3) are simple.  I will remove hadoop/site and the svn properties for 
hadoop/common/trunk/.

Does anyone know how to fix (1)?


A separated question:
Strictly speaking, this is not "project unsplit" since we are going to submit 
patches for individual sub-projects as before, i.e. we keep generating and 
committing patches to common/trunk/[common/hdfs/mapreduce] but not 
common/trunk;  hudson will pick up patches from individual sub-projects, etc.  
Am I correct?  Just want to make sure that everyone is on the same page.  :)


Regards,
Tsz-Wo

--0-816861540-1307990540=:48430--

From general-return-3684-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 18:52:18 2011
Return-Path: <general-return-3684-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0C515672B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 18:52:18 +0000 (UTC)
Received: (qmail 87582 invoked by uid 500); 13 Jun 2011 18:52:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87517 invoked by uid 500); 13 Jun 2011 18:52:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87509 invoked by uid 99); 13 Jun 2011 18:52:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 18:52:16 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.27.228] (HELO qmta15.emeryville.ca.mail.comcast.net) (76.96.27.228)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 18:52:08 +0000
Received: from omta19.emeryville.ca.mail.comcast.net ([76.96.30.76])
	by qmta15.emeryville.ca.mail.comcast.net with comcast
	id vWPD1g0041eYJf8AFWrmU0; Mon, 13 Jun 2011 18:51:46 +0000
Received: from fs ([24.4.185.157])
	by omta19.emeryville.ca.mail.comcast.net with comcast
	id vWrm1g0083QAh8g01WrmSb; Mon, 13 Jun 2011 18:51:47 +0000
Received: from mail.boudnik.org (localhost [127.0.0.1])
	by fs (8.14.3/8.14.3/Debian-5+lenny1) with ESMTP id p5DIpihL032068
	for <general@hadoop.apache.org>; Mon, 13 Jun 2011 11:51:44 -0700
Received: (from cos@localhost)
	by mail.boudnik.org (8.14.3/8.14.3/Submit) id p5DIpiGx032067
	for general@hadoop.apache.org; Mon, 13 Jun 2011 11:51:44 -0700
X-Authentication-Warning: mail.boudnik.org: cos set sender to cos@apache.org using -f
Date: Mon, 13 Jun 2011 11:51:44 -0700
From: Konstantin Boudnik <cos@apache.org>
To: general@hadoop.apache.org
Subject: Re: JIRAs post-"unsplit"
Message-ID: <20110613185144.GA32026@linspire.com>
References: <BANLkTikGFdEhGz+t+nVAnF8FX-2u-cajCg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <BANLkTikGFdEhGz+t+nVAnF8FX-2u-cajCg@mail.gmail.com>
X-Organization: It's something of 'Cos
User-Agent: Mutt/1.5.18 (2008-05-17)

I tend to agree: JIRA separation was the benefit of the split.

I'd rather keep the current JIRA split in effect (e.g. separate JIRA projects
for separate Hadoop components; don't recombine them) and file patches in the
same way (for common, hdfs, mapreduce). If a cross component patch is needed
then HADOOP project JIRA can be used for tracking, patches, etc.

Tree-based watch-list seems like a great idea, but won't it narrow the scope
somehow? Are you saying that if I am interested in say hdfs/src/c++/libhdfs,
but a JIRA is open which affects libhdfs and something else (e.g. NameNode) I
will still get the notification?

Cos

On Mon, Jun 13, 2011 at 11:28AM, Todd Lipcon wrote:
> After the "project unsplit" this weekend, we're now back to a place where we
> have a single SVN/git tree that encompasses all of the subprojects. This
> opens up the next question: should we merge the JIRAs and allow a single
> issue to have a patch which spans projects?
> 
> My thoughts are:
> - the biggest pain point with the project split is dealing with
> cross-project patches
> - one of the biggest reasons we did the project split was that the combined
> traffic from the HADOOP JIRA was hard to follow for people who really care
> about certain subprojects.
> - the jira split is a coarse-grained way of allowing people to watch just
> the sub-areas they care about.
> 
> So, I was thinking the following... what if there were a way to watch JIRAs
> based on subtrees? I'm imagining a web page where any community user could
> have an account and manage a "watch list" of subtrees. If you want to watch
> all MR jiras, you could simply watch mapreduce/*. If you care only about
> libhdfs, you could watch hdfs/src/c++/libhdfs, etc. Then a bot would watch
> all patches attached to JIRA, and any time a patch is uploaded that touches
> something on your watch list, it automatically adds you as a watcher on the
> ticket and sends you a notification via email. It would also be easy to set
> up a watch based on patch size, for example.
> 
> I think even if we don't recombine the JIRAs, this might be a handy way to
> cut down on mailing list traffic for contributors who have a more narrow
> focus on certain areas of the code.
> 
> Does this sound useful? I don't know if/when I'd have time to build such a
> thing, but if the community thinks it would be really helpful, I might
> become inspired.
> 
> -Todd
> -- 
> Todd Lipcon
> Software Engineer, Cloudera

From general-return-3685-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 19:52:24 2011
Return-Path: <general-return-3685-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1D4B16860
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 19:52:24 +0000 (UTC)
Received: (qmail 17575 invoked by uid 500); 13 Jun 2011 19:52:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17519 invoked by uid 500); 13 Jun 2011 19:52:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17509 invoked by uid 99); 13 Jun 2011 19:52:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 19:52:22 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 19:52:15 +0000
Received: from SP2-EX07CAS02.ds.corp.yahoo.com (sp2-ex07cas02.corp.sp2.yahoo.com [98.137.59.38])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5DJpjin017796
	for <general@hadoop.apache.org>; Mon, 13 Jun 2011 12:51:45 -0700 (PDT)
Received: from SP2-EX07VS03.ds.corp.yahoo.com ([98.137.59.32]) by
 SP2-EX07CAS02.ds.corp.yahoo.com ([98.137.59.38]) with mapi; Mon, 13 Jun 2011
 12:51:45 -0700
From: Matthew Foley <mattf@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: Matthew Foley <mattf@yahoo-inc.com>
Date: Mon, 13 Jun 2011 12:51:44 -0700
Subject: Re: HADOOP-7106 (project unsplit) this weekend
Thread-Topic: HADOOP-7106 (project unsplit) this weekend
Thread-Index: AcwqA1LS/t67qfo0Rnq8qP2Va7TSjQ==
Message-ID: <77E52CF7-D482-42F7-A871-677E2D29A5AE@yahoo-inc.com>
References: <CA1BAE5A.790B%jnaisbit@yahoo-inc.com>
In-Reply-To: <CA1BAE5A.790B%jnaisbit@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Todd, it's so great to have this done.  Thanks for the huge amount of hard =
work!
--Matt

On Jun 13, 2011, at 10:08 AM, Jeffrey Naisbitt wrote:

On 6/13/11 12:01 PM, "Todd Lipcon" <todd@cloudera.com> wrote:

> On Mon, Jun 13, 2011 at 9:24 AM, Jeffrey Naisbitt
> <jnaisbit@yahoo-inc.com>wrote:
>=20
>> As you say though, I would think it should still work with three separat=
e
>> projects, so I'll just go back to that for now.  It would be nice to be
>> able
>> to build the whole thing as a single project though (since it's a single
>> repository in svn now), but that would probably take some extra work - a=
nd
>> would probably make sense in a separate Jira.
>>=20
>=20
> Yep - I think the idea is this will happen once we mavenize everything.
> Maven apparently supports the idea of "modules" - basically a recursive
> structure in which a top level pom file can build sub-poms, but the subpo=
ms
> could also continue to be built independently if necessary. I think the
> top-level pom would live in the new root.
>=20
> -Todd

I look forward to the mavenization! :)
I verified the eclipse builds are working fine for me in separate projects
and that the svn:externals work fine on yahoo-merge now.
Thanks for the quick responses and for all the other work on the "unsplit"!
-Jeff



From general-return-3686-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 20:36:22 2011
Return-Path: <general-return-3686-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7FA336A23
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 20:36:22 +0000 (UTC)
Received: (qmail 91879 invoked by uid 500); 13 Jun 2011 20:36:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91823 invoked by uid 500); 13 Jun 2011 20:36:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91815 invoked by uid 99); 13 Jun 2011 20:36:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 20:36:20 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 20:36:12 +0000
Received: by bwz8 with SMTP id 8so4873978bwz.35
        for <general@hadoop.apache.org>; Mon, 13 Jun 2011 13:35:52 -0700 (PDT)
Received: by 10.204.71.193 with SMTP id i1mr5275622bkj.102.1307997352171; Mon,
 13 Jun 2011 13:35:52 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.79 with HTTP; Mon, 13 Jun 2011 13:35:32 -0700 (PDT)
In-Reply-To: <550632.48430.qm@web65913.mail.ac4.yahoo.com>
References: <BANLkTinVKj9DE+UoLRfhjVNA=mBR0k5yFg@mail.gmail.com>
 <CA1BA3F6.78F3%jnaisbit@yahoo-inc.com> <BANLkTi=R4XV4-0formszUMymD_7h1QO4OQ@mail.gmail.com>
 <550632.48430.qm@web65913.mail.ac4.yahoo.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 13 Jun 2011 13:35:32 -0700
Message-ID: <BANLkTi=1VoUN-0_9CRRK4i=LiUCD9rNM=g@mail.gmail.com>
Subject: Re: HADOOP-7106 (project unsplit) this weekend
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c5b6c98b19c104a59ddda8
X-Virus-Checked: Checked by ClamAV on apache.org

--001636c5b6c98b19c104a59ddda8
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Jun 13, 2011 at 11:42 AM, Tsz Wo (Nicholas), Sze <
s29752-hadoopgeneral@yahoo.com> wrote:

> Todd,
>
> Great work!
>
>
> A few minor problems:
> (1) I had committed MAPREDUCE-2588, however, an commit email was sent to
> both
> common-commits@ and mapreduce-commits@.
> (2) hadoop/site becomes an empty directory.
> (3) There are svn properties in hadoop/common/trunk/
>
> (2) and (3) are simple.  I will remove hadoop/site and the svn properties
> for
> hadoop/common/trunk/.
>
> Does anyone know how to fix (1)?
>

Ah, we need to update the mailer config. This is the remaining task I
mentioned on the HADOOP-7106 JIRA. We had updated it to include both the old
and new spots for the sake of the transition. I'll ping Ian about this - I
think he's the only one with access.


>
>
> A separated question:
> Strictly speaking, this is not "project unsplit" since we are going to
> submit
> patches for individual sub-projects as before, i.e. we keep generating and
> committing patches to common/trunk/[common/hdfs/mapreduce] but not
> common/trunk;  hudson will pick up patches from individual sub-projects,
> etc.
> Am I correct?  Just want to make sure that everyone is on the same page.
>  :)
>
>
Correct, that's where we at for today. I opened HADOOP-7384 which will allow
patches generated against the "full repo" to apply to an individual project,
to make life easier for git users, but right now we still need separate
JIRAs and patches. See my other thread from this morning about some ideas
how we might be able to do cross-project patches in a reasonable way.

-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

--001636c5b6c98b19c104a59ddda8--

From general-return-3687-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 20:41:53 2011
Return-Path: <general-return-3687-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A12966D96
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 20:41:53 +0000 (UTC)
Received: (qmail 5656 invoked by uid 500); 13 Jun 2011 20:41:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5593 invoked by uid 500); 13 Jun 2011 20:41:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5579 invoked by uid 99); 13 Jun 2011 20:41:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 20:41:52 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 20:41:45 +0000
Received: by bwz8 with SMTP id 8so4881868bwz.35
        for <general@hadoop.apache.org>; Mon, 13 Jun 2011 13:41:24 -0700 (PDT)
Received: by 10.204.189.209 with SMTP id df17mr5187786bkb.75.1307997684086;
 Mon, 13 Jun 2011 13:41:24 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.79 with HTTP; Mon, 13 Jun 2011 13:37:08 -0700 (PDT)
In-Reply-To: <20110613185144.GA32026@linspire.com>
References: <BANLkTikGFdEhGz+t+nVAnF8FX-2u-cajCg@mail.gmail.com> <20110613185144.GA32026@linspire.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 13 Jun 2011 13:37:08 -0700
Message-ID: <BANLkTintDEnXgUbvh2a7gOAKtQMzj5mt3g@mail.gmail.com>
Subject: Re: JIRAs post-"unsplit"
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=485b397dd4e353b73a04a59df1f2

--485b397dd4e353b73a04a59df1f2
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Jun 13, 2011 at 11:51 AM, Konstantin Boudnik <cos@apache.org> wrote:

> I tend to agree: JIRA separation was the benefit of the split.
>
> I'd rather keep the current JIRA split in effect (e.g. separate JIRA
> projects
> for separate Hadoop components; don't recombine them) and file patches in
> the
> same way (for common, hdfs, mapreduce). If a cross component patch is
> needed
> then HADOOP project JIRA can be used for tracking, patches, etc.
>

Yea, perhaps we just need the QA bot to be smart enough that it could handle
a cross-project patch attached to HADOOP? Maybe we do something crazy and
make a new HADOOPCROSS jira for patches that affect multiple projects? (just
brainstorming here...)


> Tree-based watch-list seems like a great idea, but won't it narrow the
> scope
> somehow? Are you saying that if I am interested in say
> hdfs/src/c++/libhdfs,
> but a JIRA is open which affects libhdfs and something else (e.g. NameNode)
> I
> will still get the notification?
>

Right, that's the idea. You'd be added as a watcher (and get notified) for
any patch that touches the area you care about, regardless of whether it
also touches some other areas.

-Todd

On Mon, Jun 13, 2011 at 11:28AM, Todd Lipcon wrote:
> > After the "project unsplit" this weekend, we're now back to a place where
> we
> > have a single SVN/git tree that encompasses all of the subprojects. This
> > opens up the next question: should we merge the JIRAs and allow a single
> > issue to have a patch which spans projects?
> >
> > My thoughts are:
> > - the biggest pain point with the project split is dealing with
> > cross-project patches
> > - one of the biggest reasons we did the project split was that the
> combined
> > traffic from the HADOOP JIRA was hard to follow for people who really
> care
> > about certain subprojects.
> > - the jira split is a coarse-grained way of allowing people to watch just
> > the sub-areas they care about.
> >
> > So, I was thinking the following... what if there were a way to watch
> JIRAs
> > based on subtrees? I'm imagining a web page where any community user
> could
> > have an account and manage a "watch list" of subtrees. If you want to
> watch
> > all MR jiras, you could simply watch mapreduce/*. If you care only about
> > libhdfs, you could watch hdfs/src/c++/libhdfs, etc. Then a bot would
> watch
> > all patches attached to JIRA, and any time a patch is uploaded that
> touches
> > something on your watch list, it automatically adds you as a watcher on
> the
> > ticket and sends you a notification via email. It would also be easy to
> set
> > up a watch based on patch size, for example.
> >
> > I think even if we don't recombine the JIRAs, this might be a handy way
> to
> > cut down on mailing list traffic for contributors who have a more narrow
> > focus on certain areas of the code.
> >
> > Does this sound useful? I don't know if/when I'd have time to build such
> a
> > thing, but if the community thinks it would be really helpful, I might
> > become inspired.
> >
> > -Todd
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--485b397dd4e353b73a04a59df1f2--

From general-return-3688-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 20:55:08 2011
Return-Path: <general-return-3688-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2337B64CD
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 20:55:08 +0000 (UTC)
Received: (qmail 25432 invoked by uid 500); 13 Jun 2011 20:55:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25346 invoked by uid 500); 13 Jun 2011 20:55:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25338 invoked by uid 99); 13 Jun 2011 20:55:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 20:55:06 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 20:55:00 +0000
Received: by bwz8 with SMTP id 8so4898138bwz.35
        for <general@hadoop.apache.org>; Mon, 13 Jun 2011 13:54:38 -0700 (PDT)
Received: by 10.204.189.209 with SMTP id df17mr5199027bkb.75.1307998478428;
 Mon, 13 Jun 2011 13:54:38 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.79 with HTTP; Mon, 13 Jun 2011 13:52:50 -0700 (PDT)
In-Reply-To: <BANLkTikU1LxFc6aF_dV6Fbu4E-fxf3c7ww@mail.gmail.com>
References: <BANLkTikSmUOd_4BqePmNH1fcPAYS8tbQtw@mail.gmail.com>
 <BANLkTink5G6L55LVR+uKyYXb_ifRTTSkcA@mail.gmail.com> <BANLkTiktkoY=Hg7Pc2mvYv5ydAbr-t6MJQ@mail.gmail.com>
 <BANLkTikU1LxFc6aF_dV6Fbu4E-fxf3c7ww@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 13 Jun 2011 13:52:50 -0700
Message-ID: <BANLkTikT=Hr9FhXDeayep9K6XbRUCjExBA@mail.gmail.com>
Subject: Re: HADOOP-7106 (project unsplit) this weekend
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=485b397dd4e3ac6b8004a59e20df

--485b397dd4e3ac6b8004a59e20df
Content-Type: text/plain; charset=ISO-8859-1

On Sun, Jun 12, 2011 at 4:38 PM, Todd Lipcon <todd@cloudera.com> wrote:

> If you want to be able to have git follow renames all the way through the
> project split back to the beginning of time, put the following in
> hadoop-common/.git/info/grafts:
> 5128a9a453d64bfe1ed978cf9ffed27985eeef36
> 6c16dc8cf2b28818c852e95302920a278d07ad0c
> 6a3ac690e493c7da45bbf2ae2054768c427fd0e1
> 6c16dc8cf2b28818c852e95302920a278d07ad0c
> 546d96754ffee3142bcbbf4563c624c053d0ed0d
> 6c16dc8cf2b28818c852e95302920a278d07ad0c
>
>
ATM has pointed out that the above contents line-wrapped in some people's
email inboxes. I've put the correct information on the GitAndHadoop wiki
page here:
http://wiki.apache.org/hadoop/GitAndHadoop

-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

--485b397dd4e3ac6b8004a59e20df--

From general-return-3689-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 21:06:12 2011
Return-Path: <general-return-3689-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 030E860DD
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 21:06:12 +0000 (UTC)
Received: (qmail 54660 invoked by uid 500); 13 Jun 2011 21:06:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54614 invoked by uid 500); 13 Jun 2011 21:06:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54606 invoked by uid 99); 13 Jun 2011 21:06:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 21:06:10 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of vicaya@gmail.com designates 209.85.213.176 as permitted sender)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 21:06:03 +0000
Received: by yxd5 with SMTP id 5so3010335yxd.35
        for <general@hadoop.apache.org>; Mon, 13 Jun 2011 14:05:43 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        bh=1/7tEtLMubyvXq+bDHW9ByFVz0X1rrGoxEvpum9ykGk=;
        b=FdsX79x6pXAFvX4nlwTZmEc/YgbSCdzcFLthoYAIn1Dl+jC5lM5ffa2A/eiKflsG+0
         Q3E+zypSHaCAwOCIdd5rdLaZckTaTQceMBaf6xn8P1FHktOvBbxc9FxxEZYQoU7ASBm3
         Nwex/Ry5Yw5Xmu06D89mNZnVRl5tq+LQZ/dZs=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=c+F5Gvwd0UX7Kq9zBEryfbk0WGIUS82SGtZuZZh0VwyUqIsjAH6U/xcntZomNDNXPq
         2AoaEDL6aZHNxqubwkVH3nUqi9DP0ze4ZV2Oci7lG3eMkMfjcPwmdW3w846Y5cd83pCj
         ri24hKKs/dmGumR9kz6r2onmuOS4mBr+J/SlM=
MIME-Version: 1.0
Received: by 10.236.43.50 with SMTP id k38mr3146627yhb.186.1307999141576; Mon,
 13 Jun 2011 14:05:41 -0700 (PDT)
Sender: vicaya@gmail.com
Received: by 10.236.111.1 with HTTP; Mon, 13 Jun 2011 14:05:41 -0700 (PDT)
In-Reply-To: <BANLkTintDEnXgUbvh2a7gOAKtQMzj5mt3g@mail.gmail.com>
References: <BANLkTikGFdEhGz+t+nVAnF8FX-2u-cajCg@mail.gmail.com>
	<20110613185144.GA32026@linspire.com>
	<BANLkTintDEnXgUbvh2a7gOAKtQMzj5mt3g@mail.gmail.com>
Date: Mon, 13 Jun 2011 14:05:41 -0700
X-Google-Sender-Auth: iPKgNNT6LXbhB5oWoaZ8nyMh0Ng
Message-ID: <BANLkTikP6eqKgORfA82kV517R2DYDoT_Uw@mail.gmail.com>
Subject: Re: JIRAs post-"unsplit"
From: Luke Lu <llu@vicaya.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Mon, Jun 13, 2011 at 1:37 PM, Todd Lipcon <todd@cloudera.com> wrote:
> Maybe we do something crazy and  make a new HADOOPCROSS jira for patches that affect multiple projects? (just
> brainstorming here...)

Sounds like a good idea. Having separate JIRA components for common,
hdfs and mapreduce is a way to assert code locality. Having a HADOOPX
jira component make the cross-componentness explicit (and making CI
easier), which is a good thing, IMO.

__Luke

From general-return-3690-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 22:03:55 2011
Return-Path: <general-return-3690-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6527E67B2
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 22:03:55 +0000 (UTC)
Received: (qmail 70380 invoked by uid 500); 13 Jun 2011 22:03:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70326 invoked by uid 500); 13 Jun 2011 22:03:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70318 invoked by uid 99); 13 Jun 2011 22:03:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 22:03:53 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [156.148.72.33] (HELO raffaello.crs4.it) (156.148.72.33)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 22:03:46 +0000
Received: from [172.16.106.57] (vpn-dc-nat.crs4.it [156.148.160.116])
	by raffaello.crs4.it (Postfix) with ESMTP id DAA52790102
	for <general@hadoop.apache.org>; Tue, 14 Jun 2011 00:03:23 +0200 (CEST)
Message-ID: <4DF696F3.4010609@crs4.it>
Date: Tue, 14 Jun 2011 01:02:11 +0200
From: Simone Leo <simone.leo@crs4.it>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.17) Gecko/20110510 Lightning/1.0b3pre Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: docs on setting up security?
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hello everyone,

is there any documentation on configuring kerberos security for Hadoop
0.20.203.0? The current documentation still refers to this feature as a
"future improvement". I remember seeing a guide on this on the Yahoo!
dev network before Yahoo!'s distribution was discontinued, but it looks
like it's not accessible anymore.

Thanks in advance

Simone

-- 
Simone Leo
Data Fusion - Distributed Computing
CRS4
POLARIS - Building #1
Piscina Manna
I-09010 Pula (CA) - Italy
e-mail: simone.leo@crs4.it
http://www.crs4.it

From general-return-3691-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 22:04:13 2011
Return-Path: <general-return-3691-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 432B2694F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 22:04:13 +0000 (UTC)
Received: (qmail 72079 invoked by uid 500); 13 Jun 2011 22:04:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72024 invoked by uid 500); 13 Jun 2011 22:04:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72016 invoked by uid 99); 13 Jun 2011 22:04:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 22:04:11 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [156.148.72.33] (HELO raffaello.crs4.it) (156.148.72.33)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 22:04:05 +0000
Received: from [172.16.106.57] (vpn-dc-nat.crs4.it [156.148.160.116])
	by raffaello.crs4.it (Postfix) with ESMTP id B0D75790106
	for <general@hadoop.apache.org>; Tue, 14 Jun 2011 00:03:41 +0200 (CEST)
Message-ID: <4DF69706.6000702@crs4.it>
Date: Tue, 14 Jun 2011 01:02:30 +0200
From: Simone Leo <simone.leo@crs4.it>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.17) Gecko/20110510 Lightning/1.0b3pre Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: docs on setting up security?
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hello everyone,

is there any documentation on configuring kerberos security for Hadoop
0.20.203.0? The current documentation still refers to this feature as a
"future improvement". I remember seeing a guide on this on the Yahoo!
dev network before Yahoo!'s distribution was discontinued, but it looks
like it's not accessible anymore.

Thanks in advance

Simone

-- 
Simone Leo
Data Fusion - Distributed Computing
CRS4
POLARIS - Building #1
Piscina Manna
I-09010 Pula (CA) - Italy
e-mail: simone.leo@crs4.it
http://www.crs4.it

From general-return-3692-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 22:12:15 2011
Return-Path: <general-return-3692-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 921386E7D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 22:12:15 +0000 (UTC)
Received: (qmail 86188 invoked by uid 500); 13 Jun 2011 22:12:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86115 invoked by uid 500); 13 Jun 2011 22:12:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86107 invoked by uid 99); 13 Jun 2011 22:12:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 22:12:13 +0000
X-ASF-Spam-Status: No, hits=3.3 required=5.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 22:12:07 +0000
Received: from sp1-ex07cas01.ds.corp.yahoo.com (sp1-ex07cas01.ds.corp.yahoo.com [216.252.116.137])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5DMBWFs065974
	for <general@hadoop.apache.org>; Mon, 13 Jun 2011 15:11:32 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1308003092;
	bh=rchLnEYjMzn+kVc9f7NMeUEqeXXkPRlWWo5mL9IXU1Y=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 MIME-Version;
	b=hFPAeHTGMBN59XWhZV1EjxqZ2DMh3sINCs+vldVHUVRsm18crATUOlJC1KeB6iaJ8
	 9aANvxZpsj9I8gqi0kpdaWQKNjLwvviNZZieyYrtUYrDNESa1TecqaS+3pU1Cqic7d
	 kxIMpQHOhZ0xSan7fkKeA/HQVr8YBRU+0ROfGv/4=
Received: from SP1-EX07VS02.ds.corp.yahoo.com ([216.252.116.135]) by
 sp1-ex07cas01.ds.corp.yahoo.com ([216.252.116.137]) with mapi; Mon, 13 Jun
 2011 15:11:32 -0700
From: Kihwal Lee <kihwal@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 13 Jun 2011 15:11:30 -0700
Subject: Re: docs on setting up security?
Thread-Topic: docs on setting up security?
Thread-Index: AcwqFems0bN5SL+wT9e8zljhp1H4UwAAO8Hq
Message-ID: <CA1BF542.7120%kihwal@yahoo-inc.com>
In-Reply-To: <4DF69706.6000702@crs4.it>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_CA1BF5427120kihwalyahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_CA1BF5427120kihwalyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Take a look at these:

http://yahoo.github.com/hadoop-common/installing.html
https://ccp.cloudera.com/display/CDHDOC/CDH3+Security+Guide

Kihwal

On 6/13/11 6:02 PM, "Simone Leo" <simone.leo@crs4.it> wrote:

Hello everyone,

is there any documentation on configuring kerberos security for Hadoop
0.20.203.0? The current documentation still refers to this feature as a
"future improvement". I remember seeing a guide on this on the Yahoo!
dev network before Yahoo!'s distribution was discontinued, but it looks
like it's not accessible anymore.

Thanks in advance

Simone

--
Simone Leo
Data Fusion - Distributed Computing
CRS4
POLARIS - Building #1
Piscina Manna
I-09010 Pula (CA) - Italy
e-mail: simone.leo@crs4.it
http://www.crs4.it


--_000_CA1BF5427120kihwalyahooinccom_--

From general-return-3693-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 23:55:04 2011
Return-Path: <general-return-3693-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8D8DB6BF2
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 13 Jun 2011 23:55:04 +0000 (UTC)
Received: (qmail 31312 invoked by uid 500); 13 Jun 2011 23:55:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31260 invoked by uid 500); 13 Jun 2011 23:55:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31252 invoked by uid 99); 13 Jun 2011 23:55:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:55:03 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.30.32] (HELO qmta03.emeryville.ca.mail.comcast.net) (76.96.30.32)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:54:54 +0000
Received: from omta07.emeryville.ca.mail.comcast.net ([76.96.30.59])
	by qmta03.emeryville.ca.mail.comcast.net with comcast
	id vbBn1g0051GXsucA3buYPt; Mon, 13 Jun 2011 23:54:32 +0000
Received: from fs ([24.4.185.157])
	by omta07.emeryville.ca.mail.comcast.net with comcast
	id vbuU1g00J3QAh8g8UbuZNZ; Mon, 13 Jun 2011 23:54:36 +0000
Received: from mail.boudnik.org (localhost [127.0.0.1])
	by fs (8.14.3/8.14.3/Debian-5+lenny1) with ESMTP id p5DNsOQB000674
	for <general@hadoop.apache.org>; Mon, 13 Jun 2011 16:54:24 -0700
Received: (from cos@localhost)
	by mail.boudnik.org (8.14.3/8.14.3/Submit) id p5DNsN8i000673
	for general@hadoop.apache.org; Mon, 13 Jun 2011 16:54:23 -0700
X-Authentication-Warning: mail.boudnik.org: cos set sender to cos@apache.org using -f
Date: Mon, 13 Jun 2011 16:54:23 -0700
From: Konstantin Boudnik <cos@apache.org>
To: general@hadoop.apache.org
Subject: Re: JIRAs post-"unsplit"
Message-ID: <20110613235423.GA636@linspire.com>
References: <BANLkTikGFdEhGz+t+nVAnF8FX-2u-cajCg@mail.gmail.com> <20110613185144.GA32026@linspire.com> <BANLkTintDEnXgUbvh2a7gOAKtQMzj5mt3g@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <BANLkTintDEnXgUbvh2a7gOAKtQMzj5mt3g@mail.gmail.com>
X-Organization: It's something of 'Cos
User-Agent: Mutt/1.5.18 (2008-05-17)

On Mon, Jun 13, 2011 at 01:37PM, Todd Lipcon wrote:
> On Mon, Jun 13, 2011 at 11:51 AM, Konstantin Boudnik <cos@apache.org> wrote:
> 
> > I tend to agree: JIRA separation was the benefit of the split.
> >
> > I'd rather keep the current JIRA split in effect (e.g. separate JIRA
> > projects
> > for separate Hadoop components; don't recombine them) and file patches in
> > the
> > same way (for common, hdfs, mapreduce). If a cross component patch is
> > needed
> > then HADOOP project JIRA can be used for tracking, patches, etc.
> >
> 
> Yea, perhaps we just need the QA bot to be smart enough that it could handle
> a cross-project patch attached to HADOOP? Maybe we do something crazy and
> make a new HADOOPCROSS jira for patches that affect multiple projects? (just
> brainstorming here...)

Correct me if I'm wrong but in the new structure cross-component patch differs
from a component one by a patch level (i.e. p0 vs p1 if looked from
common/trunk), right? I guess the bot can be hacked to use this distinction
thus saving us an extra JIRA project which will merely serve the purpose of
meta-project.

cos

> > Tree-based watch-list seems like a great idea, but won't it narrow the
> > scope
> > somehow? Are you saying that if I am interested in say
> > hdfs/src/c++/libhdfs,
> > but a JIRA is open which affects libhdfs and something else (e.g. NameNode)
> > I
> > will still get the notification?
> >
> 
> Right, that's the idea. You'd be added as a watcher (and get notified) for
> any patch that touches the area you care about, regardless of whether it
> also touches some other areas.
> 
> -Todd
> 
> On Mon, Jun 13, 2011 at 11:28AM, Todd Lipcon wrote:
> > > After the "project unsplit" this weekend, we're now back to a place where
> > we
> > > have a single SVN/git tree that encompasses all of the subprojects. This
> > > opens up the next question: should we merge the JIRAs and allow a single
> > > issue to have a patch which spans projects?
> > >
> > > My thoughts are:
> > > - the biggest pain point with the project split is dealing with
> > > cross-project patches
> > > - one of the biggest reasons we did the project split was that the
> > combined
> > > traffic from the HADOOP JIRA was hard to follow for people who really
> > care
> > > about certain subprojects.
> > > - the jira split is a coarse-grained way of allowing people to watch just
> > > the sub-areas they care about.
> > >
> > > So, I was thinking the following... what if there were a way to watch
> > JIRAs
> > > based on subtrees? I'm imagining a web page where any community user
> > could
> > > have an account and manage a "watch list" of subtrees. If you want to
> > watch
> > > all MR jiras, you could simply watch mapreduce/*. If you care only about
> > > libhdfs, you could watch hdfs/src/c++/libhdfs, etc. Then a bot would
> > watch
> > > all patches attached to JIRA, and any time a patch is uploaded that
> > touches
> > > something on your watch list, it automatically adds you as a watcher on
> > the
> > > ticket and sends you a notification via email. It would also be easy to
> > set
> > > up a watch based on patch size, for example.
> > >
> > > I think even if we don't recombine the JIRAs, this might be a handy way
> > to
> > > cut down on mailing list traffic for contributors who have a more narrow
> > > focus on certain areas of the code.
> > >
> > > Does this sound useful? I don't know if/when I'd have time to build such
> > a
> > > thing, but if the community thinks it would be really helpful, I might
> > > become inspired.
> > >
> > > -Todd
> > > --
> > > Todd Lipcon
> > > Software Engineer, Cloudera
> >
> 
> 
> 
> -- 
> Todd Lipcon
> Software Engineer, Cloudera

From general-return-3694-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 01:38:54 2011
Return-Path: <general-return-3694-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 304736D63
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 14 Jun 2011 01:38:54 +0000 (UTC)
Received: (qmail 39856 invoked by uid 500); 14 Jun 2011 01:38:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39797 invoked by uid 500); 14 Jun 2011 01:38:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39789 invoked by uid 99); 14 Jun 2011 01:38:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 01:38:52 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.161.51 as permitted sender)
Received: from [209.85.161.51] (HELO mail-fx0-f51.google.com) (209.85.161.51)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 01:38:44 +0000
Received: by fxm5 with SMTP id 5so3621821fxm.38
        for <general@hadoop.apache.org>; Mon, 13 Jun 2011 18:38:24 -0700 (PDT)
Received: by 10.223.161.194 with SMTP id s2mr512034fax.143.1308015504167; Mon,
 13 Jun 2011 18:38:24 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.79 with HTTP; Mon, 13 Jun 2011 18:38:04 -0700 (PDT)
In-Reply-To: <20110613235423.GA636@linspire.com>
References: <BANLkTikGFdEhGz+t+nVAnF8FX-2u-cajCg@mail.gmail.com>
 <20110613185144.GA32026@linspire.com> <BANLkTintDEnXgUbvh2a7gOAKtQMzj5mt3g@mail.gmail.com>
 <20110613235423.GA636@linspire.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 13 Jun 2011 18:38:04 -0700
Message-ID: <BANLkTin6JwyQLNf-jjOS=8Ca6diuGgKe5w@mail.gmail.com>
Subject: Re: JIRAs post-"unsplit"
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0023545309787c951904a5a2170d
X-Virus-Checked: Checked by ClamAV on apache.org

--0023545309787c951904a5a2170d
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Jun 13, 2011 at 4:54 PM, Konstantin Boudnik <cos@apache.org> wrote:

> On Mon, Jun 13, 2011 at 01:37PM, Todd Lipcon wrote:
> > On Mon, Jun 13, 2011 at 11:51 AM, Konstantin Boudnik <cos@apache.org>
> wrote:
> >
> > > I tend to agree: JIRA separation was the benefit of the split.
> > >
> > > I'd rather keep the current JIRA split in effect (e.g. separate JIRA
> > > projects
> > > for separate Hadoop components; don't recombine them) and file patches
> in
> > > the
> > > same way (for common, hdfs, mapreduce). If a cross component patch is
> > > needed
> > > then HADOOP project JIRA can be used for tracking, patches, etc.
> > >
> >
> > Yea, perhaps we just need the QA bot to be smart enough that it could
> handle
> > a cross-project patch attached to HADOOP? Maybe we do something crazy and
> > make a new HADOOPCROSS jira for patches that affect multiple projects?
> (just
> > brainstorming here...)
>
> Correct me if I'm wrong but in the new structure cross-component patch
> differs
> from a component one by a patch level (i.e. p0 vs p1 if looked from
> common/trunk), right? I guess the bot can be hacked to use this distinction
> thus saving us an extra JIRA project which will merely serve the purpose of
> meta-project.
>
>
Yes, I am about to commit HADOOP-7384 which can at least deal with patches
relative to either trunk/ or trunk/<project>. But, it will also detect a
cross-project patch and barf.

It could certainly be extended to apply and test a cross-project patch,
though it would be substantially more work.

The advantage of a separate HADOOPX jira would be to allow people to notice
cross-project patches. For example, a dev who primarily works on HDFS may
not subscribe to mapreduce-dev or mapreduce-issues, but if an MR issue is
going to modify something in the HDFS codebase, he or she will certainly
want to be aware of it.

-Todd


> > > Tree-based watch-list seems like a great idea, but won't it narrow the
> > > scope
> > > somehow? Are you saying that if I am interested in say
> > > hdfs/src/c++/libhdfs,
> > > but a JIRA is open which affects libhdfs and something else (e.g.
> NameNode)
> > > I
> > > will still get the notification?
> > >
> >
> > Right, that's the idea. You'd be added as a watcher (and get notified)
> for
> > any patch that touches the area you care about, regardless of whether it
> > also touches some other areas.
> >
> > -Todd
> >
> > On Mon, Jun 13, 2011 at 11:28AM, Todd Lipcon wrote:
> > > > After the "project unsplit" this weekend, we're now back to a place
> where
> > > we
> > > > have a single SVN/git tree that encompasses all of the subprojects.
> This
> > > > opens up the next question: should we merge the JIRAs and allow a
> single
> > > > issue to have a patch which spans projects?
> > > >
> > > > My thoughts are:
> > > > - the biggest pain point with the project split is dealing with
> > > > cross-project patches
> > > > - one of the biggest reasons we did the project split was that the
> > > combined
> > > > traffic from the HADOOP JIRA was hard to follow for people who really
> > > care
> > > > about certain subprojects.
> > > > - the jira split is a coarse-grained way of allowing people to watch
> just
> > > > the sub-areas they care about.
> > > >
> > > > So, I was thinking the following... what if there were a way to watch
> > > JIRAs
> > > > based on subtrees? I'm imagining a web page where any community user
> > > could
> > > > have an account and manage a "watch list" of subtrees. If you want to
> > > watch
> > > > all MR jiras, you could simply watch mapreduce/*. If you care only
> about
> > > > libhdfs, you could watch hdfs/src/c++/libhdfs, etc. Then a bot would
> > > watch
> > > > all patches attached to JIRA, and any time a patch is uploaded that
> > > touches
> > > > something on your watch list, it automatically adds you as a watcher
> on
> > > the
> > > > ticket and sends you a notification via email. It would also be easy
> to
> > > set
> > > > up a watch based on patch size, for example.
> > > >
> > > > I think even if we don't recombine the JIRAs, this might be a handy
> way
> > > to
> > > > cut down on mailing list traffic for contributors who have a more
> narrow
> > > > focus on certain areas of the code.
> > > >
> > > > Does this sound useful? I don't know if/when I'd have time to build
> such
> > > a
> > > > thing, but if the community thinks it would be really helpful, I
> might
> > > > become inspired.
> > > >
> > > > -Todd
> > > > --
> > > > Todd Lipcon
> > > > Software Engineer, Cloudera
> > >
> >
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--0023545309787c951904a5a2170d--

From general-return-3695-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 01:50:04 2011
Return-Path: <general-return-3695-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 210C86415
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 14 Jun 2011 01:50:04 +0000 (UTC)
Received: (qmail 48001 invoked by uid 500); 14 Jun 2011 01:50:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47939 invoked by uid 500); 14 Jun 2011 01:50:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47931 invoked by uid 99); 14 Jun 2011 01:50:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 01:50:02 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.161.51 as permitted sender)
Received: from [209.85.161.51] (HELO mail-fx0-f51.google.com) (209.85.161.51)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 01:49:54 +0000
Received: by fxm5 with SMTP id 5so3635890fxm.38
        for <general@hadoop.apache.org>; Mon, 13 Jun 2011 18:49:34 -0700 (PDT)
Received: by 10.223.127.210 with SMTP id h18mr633534fas.67.1308016174178; Mon,
 13 Jun 2011 18:49:34 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.79 with HTTP; Mon, 13 Jun 2011 18:49:14 -0700 (PDT)
In-Reply-To: <550632.48430.qm@web65913.mail.ac4.yahoo.com>
References: <BANLkTinVKj9DE+UoLRfhjVNA=mBR0k5yFg@mail.gmail.com>
 <CA1BA3F6.78F3%jnaisbit@yahoo-inc.com> <BANLkTi=R4XV4-0formszUMymD_7h1QO4OQ@mail.gmail.com>
 <550632.48430.qm@web65913.mail.ac4.yahoo.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 13 Jun 2011 18:49:14 -0700
Message-ID: <BANLkTimJ5Nr9MqdLHoqmCyo0rGgouCtEDA@mail.gmail.com>
Subject: Re: HADOOP-7106 (project unsplit) this weekend
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0023545309706c22e604a5a23f12
X-Virus-Checked: Checked by ClamAV on apache.org

--0023545309706c22e604a5a23f12
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Jun 13, 2011 at 11:42 AM, Tsz Wo (Nicholas), Sze <
s29752-hadoopgeneral@yahoo.com> wrote:

> A few minor problems:
> (1) I had committed MAPREDUCE-2588, however, an commit email was sent to
> both
> common-commits@ and mapreduce-commits@.
>

I put up a patch to fix this on HADOOP-7106. Unfortunately I think Doug and
Ian are the only ones who can commit it, and they're both traveling at the
moment. So we may have to endure dup emails for a day or two. Sorry for the
mailbox noise

-Todd

-- 
Todd Lipcon
Software Engineer, Cloudera

--0023545309706c22e604a5a23f12--

From general-return-3696-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 12:34:42 2011
Return-Path: <general-return-3696-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 642546118
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 14 Jun 2011 12:34:42 +0000 (UTC)
Received: (qmail 64284 invoked by uid 500); 14 Jun 2011 12:34:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 64233 invoked by uid 500); 14 Jun 2011 12:34:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 64225 invoked by uid 99); 14 Jun 2011 12:34:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 12:34:39 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 12:34:29 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 31323B7DEA
	for <general@hadoop.apache.org>; Tue, 14 Jun 2011 13:34:09 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id p79crocGQt1s for <general@hadoop.apache.org>;
	Tue, 14 Jun 2011 13:34:08 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 765FFB7DE9
	for <general@hadoop.apache.org>; Tue, 14 Jun 2011 13:34:07 +0100 (BST)
MailScanner-NULL-Check: 1308659634.31789@Xn0xCKdZYpTyN/dr9xnSvw
Received: from [16.25.175.86] (wildhaus.hpl.hp.com [16.25.175.86])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p5ECXsXw028108
	for <general@hadoop.apache.org>; Tue, 14 Jun 2011 13:33:54 +0100 (BST)
Message-ID: <4DF75531.9050302@apache.org>
Date: Tue, 14 Jun 2011 13:33:53 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Bigtop Proposal
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p5ECXsXw028108
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

For those people who aren't on the Apache Incubator mailing lists, Tom 
White has proposed "BigTop", which is the tooling to integrate the build 
and testing of the Apache Hadoop technologies, perhaps eventually to 
have coordinated releases.

http://wiki.apache.org/incubator/BigtopProposal


For this to work, it needs support from all the hadoop-related projects 
in the ASF, and, in an ideal world, integration with those bits from 
outside (JUnit, Jetty) as well as downstream code (Cascading, etc). What 
is key is that the projects it builds and tests will need to care about 
the test failure reports, because BigTop should not be applying any 
patches to private branches of the specific projects (the way Cloudera 
and Ubuntu have done with the Hadoop and Linux source trees, 
respectively). Some kind of synchronized release schedule would be nice 
too, but I view that as a feature creep once the basic set up "build and 
test everything" is done.

This could also be the place to put the functional tests against live 
clusters, tests which could eventually become the definition of 
compatibility that we've discussed before.

For these reasons
  -I think everyone in the Hadoop-* projects dev teams ought to get 
involved, initially in reviewing the proposal, then hopefully on the 
mailing lists.

  -It might make sense for the hadoop, pig, hive, hama, mahout, hbase &c 
committers to all have access to this, so that they can help evolve the 
packaging and functional tests.

thoughts?

-steve

From general-return-3697-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 16:35:47 2011
Return-Path: <general-return-3697-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E05AF6BC2
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 14 Jun 2011 16:35:47 +0000 (UTC)
Received: (qmail 71204 invoked by uid 500); 14 Jun 2011 16:35:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71129 invoked by uid 500); 14 Jun 2011 16:35:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71121 invoked by uid 99); 14 Jun 2011 16:35:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 16:35:46 +0000
X-ASF-Spam-Status: No, hits=-7.0 required=5.0
	tests=RCVD_IN_DNSWL_HI,RCVD_IN_RP_SAFE
X-Spam-Check-By: apache.org
Received-SPF: unknown (nike.apache.org: error in processing during lookup of jrottinghuis@ebay.com)
Received: from [216.33.244.6] (HELO rhv-mipot-001.corp.ebay.com) (216.33.244.6)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 16:35:39 +0000
DomainKey-Signature: s=corp; d=ebay.com; c=nofws; q=dns;
  h=X-EBay-Corp:X-IronPort-AV:Received:Received:Received:
   From:To:Date:Subject:Thread-Topic:Thread-Index:Message-ID:
   References:In-Reply-To:Accept-Language:Content-Language:
   X-MS-Has-Attach:X-MS-TNEF-Correlator:acceptlanguage:
   Content-Type:Content-Transfer-Encoding:MIME-Version:
   Return-Path:X-EMS-Proccessed:X-EMS-STAMP:X-CFilter;
  b=WHALS379FQBjG1YAZemSTiCg08IwawkN7itjkqCoxLvHuSo/yMqXCH4s
   57YB474cmx5aZdLQyPuJzqEOqs+UokD+Dkn45hCEMAxvqG81Dp7m2nSKX
   L30ETUB3Eq5Z1sB;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=ebay.com; i=jrottinghuis@ebay.com; q=dns/txt; s=corp;
  t=1308069339; x=1339605339;
  h=from:to:date:subject:message-id:references:in-reply-to:
   content-transfer-encoding:mime-version;
  bh=vxz0eguL3rC9cfndMc3uFGJhWx/Pq+u82bQrMQ5C4FE=;
  b=TbHXA6ZyYyllAV9u05B5JPEXYFnck52pB17EI6QzfJDmlrenkvtmxKJP
   Y2YhNDknomzk6a+FJznNy9A4CPhxEpWNZ6OoSI0j7ebRQjr10wvXg8nyH
   A7llX5y5ICDsWMT;
X-EBay-Corp: Yes
X-IronPort-AV: E=Sophos;i="4.65,365,1304319600"; 
   d="scan'208";a="21052336"
Received: from rhv-vtenf-002.corp.ebay.com (HELO RHV-MEXHT-001.corp.ebay.com) ([10.112.113.53])
  by rhv-mipot-001.corp.ebay.com with ESMTP; 14 Jun 2011 09:35:15 -0700
Received: from RHV-MEXHT-004.corp.ebay.com (10.245.24.106) by
 RHV-MEXHT-001.corp.ebay.com (10.245.24.100) with Microsoft SMTP Server (TLS)
 id 8.3.137.0; Tue, 14 Jun 2011 09:35:14 -0700
Received: from RHV-MEXMS-002.corp.ebay.com ([10.245.17.115]) by
 RHV-MEXHT-004.corp.ebay.com ([10.245.24.106]) with mapi; Tue, 14 Jun 2011
 09:35:14 -0700
From: "Rottinghuis, Joep" <jrottinghuis@ebay.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 14 Jun 2011 09:35:13 -0700
Subject: RE: JIRAs post-"unsplit"
Thread-Topic: JIRAs post-"unsplit"
Thread-Index: AcwqClSfLGqatEuPRmW4gLM7GsChOgApY4c2
Message-ID: <F4FCE59AE46B2243922F224426BC869B01EC996721@RHV-MEXMS-002.corp.ebay.com>
References: <BANLkTikGFdEhGz+t+nVAnF8FX-2u-cajCg@mail.gmail.com>
 <20110613185144.GA32026@linspire.com>,<BANLkTintDEnXgUbvh2a7gOAKtQMzj5mt3g@mail.gmail.com>
In-Reply-To: <BANLkTintDEnXgUbvh2a7gOAKtQMzj5mt3g@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-EMS-Proccessed: 10SqDH0iR7ekR7SRpKqm5A==
X-EMS-STAMP: GCOvcmsWGtwPjEQvCS4Oew==
X-CFilter: Scanned
X-Virus-Checked: Checked by ClamAV on apache.org

Project un-split definitely simplifies things.

Todd, if people add a watch based on patches, would they not miss notificti=
ons for those entries in an earlier phase of their lifecycle?
For example when issues are just reported, discussed and assigned, but no p=
atch has been attached yet?

A separate HADOOPX Jira project would eliminate such issues.

It does raise another question though: What happens if an issue starts out =
in one area, and then turns out to require changes in other areas?
Would one then first create a HADOOP-x, a HDFS-y, or MAPREDUCE-z and then w=
hen it turns out other components are involved a new HADOOPX- referring to =
such earlier Jira?

Cheers,

Joep

________________________________________
From: Todd Lipcon [todd@cloudera.com]
Sent: Monday, June 13, 2011 1:37 PM
To: general@hadoop.apache.org
Subject: Re: JIRAs post-"unsplit"

On Mon, Jun 13, 2011 at 11:51 AM, Konstantin Boudnik <cos@apache.org> wrote=
:

> I tend to agree: JIRA separation was the benefit of the split.
>
> I'd rather keep the current JIRA split in effect (e.g. separate JIRA
> projects
> for separate Hadoop components; don't recombine them) and file patches in
> the
> same way (for common, hdfs, mapreduce). If a cross component patch is
> needed
> then HADOOP project JIRA can be used for tracking, patches, etc.
>

Yea, perhaps we just need the QA bot to be smart enough that it could handl=
e
a cross-project patch attached to HADOOP? Maybe we do something crazy and
make a new HADOOPCROSS jira for patches that affect multiple projects? (jus=
t
brainstorming here...)


> Tree-based watch-list seems like a great idea, but won't it narrow the
> scope
> somehow? Are you saying that if I am interested in say
> hdfs/src/c++/libhdfs,
> but a JIRA is open which affects libhdfs and something else (e.g. NameNod=
e)
> I
> will still get the notification?
>

Right, that's the idea. You'd be added as a watcher (and get notified) for
any patch that touches the area you care about, regardless of whether it
also touches some other areas.

-Todd=

From general-return-3698-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 17:00:23 2011
Return-Path: <general-return-3698-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BD4396A9D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 14 Jun 2011 17:00:23 +0000 (UTC)
Received: (qmail 13253 invoked by uid 500); 14 Jun 2011 17:00:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12960 invoked by uid 500); 14 Jun 2011 17:00:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12952 invoked by uid 99); 14 Jun 2011 17:00:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 17:00:21 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 17:00:14 +0000
Received: by bwz8 with SMTP id 8so5110081bwz.35
        for <general@hadoop.apache.org>; Tue, 14 Jun 2011 09:59:54 -0700 (PDT)
Received: by 10.204.61.197 with SMTP id u5mr3274732bkh.210.1308070794223; Tue,
 14 Jun 2011 09:59:54 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.79 with HTTP; Tue, 14 Jun 2011 09:59:34 -0700 (PDT)
In-Reply-To: <F4FCE59AE46B2243922F224426BC869B01EC996721@RHV-MEXMS-002.corp.ebay.com>
References: <BANLkTikGFdEhGz+t+nVAnF8FX-2u-cajCg@mail.gmail.com>
 <20110613185144.GA32026@linspire.com> <BANLkTintDEnXgUbvh2a7gOAKtQMzj5mt3g@mail.gmail.com>
 <F4FCE59AE46B2243922F224426BC869B01EC996721@RHV-MEXMS-002.corp.ebay.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Tue, 14 Jun 2011 09:59:34 -0700
Message-ID: <BANLkTikv2OLxz39=qeSKq_3dgUCWpGQNfg@mail.gmail.com>
Subject: Re: JIRAs post-"unsplit"
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c5a9d007e3ef04a5aef76a
X-Virus-Checked: Checked by ClamAV on apache.org

--001636c5a9d007e3ef04a5aef76a
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Jun 14, 2011 at 9:35 AM, Rottinghuis, Joep <jrottinghuis@ebay.com>wrote:

> Project un-split definitely simplifies things.
>
> Todd, if people add a watch based on patches, would they not miss
> notifictions for those entries in an earlier phase of their lifecycle?
> For example when issues are just reported, discussed and assigned, but no
> patch has been attached yet?
>
>
Another thought that Alejandro just suggested offline is to use JIRA
components rather than just the file paths. So, assuming there is a bot that
watches the JIRA, it would be easy enough to allow you to permawatch a
component (JIRA itself doesn't give this option).

Then, assuming the patch is assigned the right components, it will be seen
by people who care early on. If it's not given the right components, then it
will be seen once you upload a patch.



> A separate HADOOPX Jira project would eliminate such issues.
>
> It does raise another question though: What happens if an issue starts out
> in one area, and then turns out to require changes in other areas?
> Would one then first create a HADOOP-x, a HDFS-y, or MAPREDUCE-z and then
> when it turns out other components are involved a new HADOOPX- referring to
> such earlier Jira?
>
> Cheers,
>
> Joep
>
> ________________________________________
> From: Todd Lipcon [todd@cloudera.com]
> Sent: Monday, June 13, 2011 1:37 PM
> To: general@hadoop.apache.org
> Subject: Re: JIRAs post-"unsplit"
>
> On Mon, Jun 13, 2011 at 11:51 AM, Konstantin Boudnik <cos@apache.org>
> wrote:
>
> > I tend to agree: JIRA separation was the benefit of the split.
> >
> > I'd rather keep the current JIRA split in effect (e.g. separate JIRA
> > projects
> > for separate Hadoop components; don't recombine them) and file patches in
> > the
> > same way (for common, hdfs, mapreduce). If a cross component patch is
> > needed
> > then HADOOP project JIRA can be used for tracking, patches, etc.
> >
>
> Yea, perhaps we just need the QA bot to be smart enough that it could
> handle
> a cross-project patch attached to HADOOP? Maybe we do something crazy and
> make a new HADOOPCROSS jira for patches that affect multiple projects?
> (just
> brainstorming here...)
>
>
> > Tree-based watch-list seems like a great idea, but won't it narrow the
> > scope
> > somehow? Are you saying that if I am interested in say
> > hdfs/src/c++/libhdfs,
> > but a JIRA is open which affects libhdfs and something else (e.g.
> NameNode)
> > I
> > will still get the notification?
> >
>
> Right, that's the idea. You'd be added as a watcher (and get notified) for
> any patch that touches the area you care about, regardless of whether it
> also touches some other areas.
>
> -Todd
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--001636c5a9d007e3ef04a5aef76a--

From general-return-3699-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 18:38:48 2011
Return-Path: <general-return-3699-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DFB5F6B45
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 14 Jun 2011 18:38:48 +0000 (UTC)
Received: (qmail 11119 invoked by uid 500); 14 Jun 2011 18:38:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11057 invoked by uid 500); 14 Jun 2011 18:38:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11049 invoked by uid 99); 14 Jun 2011 18:38:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 18:38:47 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.30.32] (HELO qmta03.emeryville.ca.mail.comcast.net) (76.96.30.32)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 18:38:39 +0000
Received: from omta11.emeryville.ca.mail.comcast.net ([76.96.30.36])
	by qmta03.emeryville.ca.mail.comcast.net with comcast
	id vtVx1g0010mlR8UA3ueH3t; Tue, 14 Jun 2011 18:38:17 +0000
Received: from fs ([24.4.185.157])
	by omta11.emeryville.ca.mail.comcast.net with comcast
	id vueH1g01s3QAh8g8XueKpb; Tue, 14 Jun 2011 18:38:19 +0000
Received: from mail.boudnik.org (localhost [127.0.0.1])
	by fs (8.14.3/8.14.3/Debian-5+lenny1) with ESMTP id p5EIcF51005388
	for <general@hadoop.apache.org>; Tue, 14 Jun 2011 11:38:15 -0700
Received: (from cos@localhost)
	by mail.boudnik.org (8.14.3/8.14.3/Submit) id p5EIcFn5005387
	for general@hadoop.apache.org; Tue, 14 Jun 2011 11:38:15 -0700
X-Authentication-Warning: mail.boudnik.org: cos set sender to cos@apache.org using -f
Date: Tue, 14 Jun 2011 11:38:15 -0700
From: Konstantin Boudnik <cos@apache.org>
To: general@hadoop.apache.org
Subject: Re: Bigtop Proposal
Message-ID: <20110614183814.GA5307@linspire.com>
References: <4DF75531.9050302@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <4DF75531.9050302@apache.org>
X-Organization: It's something of 'Cos
User-Agent: Mutt/1.5.18 (2008-05-17)

Steve, 'functional' or rather system & integration tests are the core of this
idea. If you're interested in a little history of this project feel free to
check

 https://docs.google.com/present/edit?id=0AVSlqgtwzvr9ZGdtNGM0OTJfMTRoYmRkMzhndA&hl=en_US&authkey=CLjMt_IN

that's basically the original idea/initial implementation (based on the work
done by Yahoo and known as HIT) which essentially lead to this proposal, I
believe (anyone please correct me if you found this suggestion wrong).

With regards,
  Cos

On Tue, Jun 14, 2011 at 01:33PM, Steve Loughran wrote:
> For those people who aren't on the Apache Incubator mailing lists, Tom  
> White has proposed "BigTop", which is the tooling to integrate the build  
> and testing of the Apache Hadoop technologies, perhaps eventually to  
> have coordinated releases.
>
> http://wiki.apache.org/incubator/BigtopProposal
>
> For this to work, it needs support from all the hadoop-related projects  
> in the ASF, and, in an ideal world, integration with those bits from  
> outside (JUnit, Jetty) as well as downstream code (Cascading, etc). What  
> is key is that the projects it builds and tests will need to care about  
> the test failure reports, because BigTop should not be applying any  
> patches to private branches of the specific projects (the way Cloudera  
> and Ubuntu have done with the Hadoop and Linux source trees,  
> respectively). Some kind of synchronized release schedule would be nice  
> too, but I view that as a feature creep once the basic set up "build and  
> test everything" is done.
>
> This could also be the place to put the functional tests against live  
> clusters, tests which could eventually become the definition of  
> compatibility that we've discussed before.
>
> For these reasons
>  -I think everyone in the Hadoop-* projects dev teams ought to get  
> involved, initially in reviewing the proposal, then hopefully on the  
> mailing lists.
>
>  -It might make sense for the hadoop, pig, hive, hama, mahout, hbase &c  
> committers to all have access to this, so that they can help evolve the  
> packaging and functional tests.
>
> thoughts?
>
> -steve

From general-return-3700-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 19:58:54 2011
Return-Path: <general-return-3700-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D4B28688E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 14 Jun 2011 19:58:54 +0000 (UTC)
Received: (qmail 75383 invoked by uid 500); 14 Jun 2011 19:58:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75223 invoked by uid 500); 14 Jun 2011 19:58:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75211 invoked by uid 99); 14 Jun 2011 19:58:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 19:58:53 +0000
X-ASF-Spam-Status: No, hits=2.9 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 216.145.54.171 is neither permitted nor denied by domain of sradia@yahoo-inc.com)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 19:58:48 +0000
Received: from [192.168.1.71] (snvvpn4-10-72-168-c30.hq.corp.yahoo.com [10.72.168.30])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5EJw52t013534;
	Tue, 14 Jun 2011 12:58:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1308081486;
	bh=FaseLUIwTCzZgH/yyZp8hUcrug6rfFtbONo3wKy6Ft4=;
	h=Cc:Message-Id:From:To:In-Reply-To:Content-Type:Mime-Version:
	 Subject:Date:References;
	b=egodKK8DQSQMsBJHV/h1AE0mskkMN/LnOZVCYDPNzn5ZWhsx0LgDLglv+DM+PqeIO
	 d/qms1HfoQJYBB3aOhwCvQKAlJRjt4dikylyzcqCA++nihCruWC8rCykniGUN5oZR5
	 TFsNjAGj0e8aE70I0izsrAQ5rqQNfM66MBu5DZW0=
Cc: "apurtell@apache.org" <apurtell@apache.org>
Message-Id: <8B7097C4-597D-4BA4-8C69-C48F5A571AEF@yahoo-inc.com>
From: Sanjay Radia <sradia@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <BANLkTikYuO0nRHz3UWZFZU245dMw82Y08Q@mail.gmail.com>
Content-Type: multipart/alternative; boundary=Apple-Mail-10--938195917
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: LimitedPrivate and HBase
Date: Tue, 14 Jun 2011 12:58:05 -0700
References: <457011.690.qm@web65508.mail.ac4.yahoo.com> <F0CA2186-6E63-459D-ABFF-21863EE018E5@yahoo-inc.com> <BANLkTikYuO0nRHz3UWZFZU245dMw82Y08Q@mail.gmail.com>
X-Mailer: Apple Mail (2.936)

--Apple-Mail-10--938195917
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


On Jun 9, 2011, at 10:27 AM, Tom White wrote:

> Looking at current usage in Hadoop, there are only 4 LimitedPrivate
> references to HBase (the http, io.retry, ipc, and metrics packages in
> Common), and 2 references to Pig (the two LineRecordReader classes in
> MapReduce). The other LimitedPrivate references are all to HDFS or
> MapReduce. Given that Private means "Intended for use only within
> Hadoop itself" (according to the javadoc), we can replace these
> references with Private.


Okay so that was incorrectly stated in the Javadoc - if you read
>
> We could also change the remaining 6 cases of LimitedPrivate to Public
> (note that they are already annotated Evolving or Unstable), and
> deprecate LimitedPrivate. Would this allay people's concerns?


-1
I disagree with the proposed changes.

Most folks are missing the point of limited private.
The Jira (HADOOP-5073)  that created this classification gives a very  
detailed explanation of the motivation and purpose of the  
classification.
Unfortunately most of the explanation in Jira were not copied to the  
Javadoc. My mistake here.
I will file a jira to copy the classification documentation from the  
Jira to a  Javadoc.


BTW the the javadoc is incorrect
>  Private means "Intended for use only within
> Hadoop itself" (according to the javadoc)

The definition in the Jira (Hadoop-5073) explains that private means  
project private.  It is private to HDFS or private to MR etc. Private  
does not mean private to Hadoop - otherwise MR can use any internal  
private class inside HDFS. We don't want that.
When we did the actual annotation tags the words project-private was  
simplified to private since folks felt it was too verbose.

The Jira states:
 >>>
project-private
the interface is for internal use within the project and should not be  
used by applications. It is subject to change at anytime without  
notice. Most interfaces of a project are project private.
<<<<

My mistake for not checking the actual javadoc carefully when Jacob's   
annotations patch was committed.
I will file a jira to copy the document from the Jira to the Javadoc.

I will post a longer email explaining my position and my -1 more  
clearly after I have had a chance to read all the emails carefully.


sanjay


--Apple-Mail-10--938195917--

From general-return-3701-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 21:21:30 2011
Return-Path: <general-return-3701-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A78F56AFD
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 14 Jun 2011 21:21:30 +0000 (UTC)
Received: (qmail 24521 invoked by uid 500); 14 Jun 2011 21:21:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24459 invoked by uid 500); 14 Jun 2011 21:21:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24447 invoked by uid 99); 14 Jun 2011 21:21:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 21:21:28 +0000
X-ASF-Spam-Status: No, hits=2.9 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 216.145.54.173 is neither permitted nor denied by domain of sradia@yahoo-inc.com)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 21:21:22 +0000
Received: from [192.168.1.71] (snvvpn4-10-72-168-c30.hq.corp.yahoo.com [10.72.168.30])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p5ELK6jQ072116;
	Tue, 14 Jun 2011 14:20:06 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1308086407;
	bh=xOJLlpEKDIPReEzUr7cT/q/szN7EquDuDkDadZP1qdM=;
	h=Cc:Message-Id:From:To:In-Reply-To:Content-Type:Mime-Version:
	 Subject:Date:References;
	b=ZyRORXWKXMtYzR0FtqN/hdH9/lqzUygORyxxP1YBQGKYpVdOYbuaXFJ3aPVmLUKNS
	 qT4RSAt+zIIqnEkpwTthVqltcpHipsSQzYb6mypnhHwS+jwPXmGU2EXX0qt00maMbZ
	 Aqnc94PePMi0eqOCrscpF0S3kVlQ6wh/iNjqP+H4=
Cc: "apurtell@apache.org" <apurtell@apache.org>
Message-Id: <E4194561-D1E3-4D1E-91EB-449427805669@yahoo-inc.com>
From: Sanjay Radia <sradia@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <8B7097C4-597D-4BA4-8C69-C48F5A571AEF@yahoo-inc.com>
Content-Type: multipart/alternative; boundary=Apple-Mail-12--933275672
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: LimitedPrivate and HBase
Date: Tue, 14 Jun 2011 14:20:05 -0700
References: <457011.690.qm@web65508.mail.ac4.yahoo.com> <F0CA2186-6E63-459D-ABFF-21863EE018E5@yahoo-inc.com> <BANLkTikYuO0nRHz3UWZFZU245dMw82Y08Q@mail.gmail.com> <8B7097C4-597D-4BA4-8C69-C48F5A571AEF@yahoo-inc.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-12--933275672
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


On Jun 14, 2011, at 12:58 PM, Sanjay Radia wrote:

>
>
>
> -1
> I disagree with the proposed changes.
> .....
> I will post a longer email explaining my position and my -1 more
> clearly after I have had a chance to read all the emails carefully.
>
>
> sanjay

Please don't take my -1 too strongly.
It was NOT meant to be offensive. I saw a lot of +1s and wanted to  
make sure that this doesn't turn into a jira and a commit in  few days.

sanjay
--Apple-Mail-12--933275672--

From general-return-3702-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 22:57:05 2011
Return-Path: <general-return-3702-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CB8026853
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 14 Jun 2011 22:57:05 +0000 (UTC)
Received: (qmail 21107 invoked by uid 500); 14 Jun 2011 22:57:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21039 invoked by uid 500); 14 Jun 2011 22:57:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21031 invoked by uid 99); 14 Jun 2011 22:57:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 22:57:04 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of owen.omalley@gmail.com designates 209.85.210.48 as permitted sender)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 22:56:55 +0000
Received: by pzk10 with SMTP id 10so3966775pzk.35
        for <general@hadoop.apache.org>; Tue, 14 Jun 2011 15:56:34 -0700 (PDT)
Received: by 10.68.13.228 with SMTP id k4mr3473638pbc.40.1308092194050;
        Tue, 14 Jun 2011 15:56:34 -0700 (PDT)
Received: from battlerock-lm.corp.yahoo.com (nat-dip4.cfw-a-gci.corp.yahoo.com [209.131.62.113])
        by mx.google.com with ESMTPS id p5sm5923398pbk.36.2011.06.14.15.56.32
        (version=SSLv3 cipher=OTHER);
        Tue, 14 Jun 2011 15:56:33 -0700 (PDT)
From: Owen O'Malley <omalley@apache.org>
Content-Type: multipart/alternative; boundary=Apple-Mail-5--927489155
Subject: [VOTE] Shall we adopt the "Defining Hadoop" page
Date: Tue, 14 Jun 2011 15:56:32 -0700
Message-Id: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
Cc: Apache Brand Management <trademarks@apache.org>
To: general@hadoop.apache.org
Mime-Version: 1.0 (Apple Message framework v1084)
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-5--927489155
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

All,
   Steve Loughran has done some great work on defining what can be =
called Hadoop at http://wiki.apache.org/hadoop/Defining%20Hadoop. After =
some cleanup from Noirin and Shane, I think we've got a really good =
base. I'd like a vote to approve the content (at the current revision =
12) and put the content on our web site.

Clearly, I'm +1.

-- Owen=

--Apple-Mail-5--927489155--

From general-return-3703-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 23:32:38 2011
Return-Path: <general-return-3703-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B0984672D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 14 Jun 2011 23:32:38 +0000 (UTC)
Received: (qmail 58007 invoked by uid 500); 14 Jun 2011 23:32:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57950 invoked by uid 500); 14 Jun 2011 23:32:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57939 invoked by uid 99); 14 Jun 2011 23:32:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 23:32:37 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 23:32:31 +0000
Received: by qyk2 with SMTP id 2so2072646qyk.14
        for <general@hadoop.apache.org>; Tue, 14 Jun 2011 16:32:10 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=0XtclB361Ux4f0dM82ZvLINOIb334VgEJpB23nWjRMo=;
        b=ftb2NaQh5uYrvyhLNQMqfUbZxyWXtX9Z0fSNE0Dsm8L6pelRUC0O/PBXmbEjquuWPH
         L0PZJ8EtVm18qaajuNi+iGQtVJ0n9MO+ikd/+16LTZeC6UbH2nMvSANBzKWV1Gg0VYc2
         YUhQPMFsGCbrDoTmZTttCUGry03jbkSTDupOg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=F7eY7O6DWV8878hFwuDTsOliFEoZf1acBZhfL2KaJTr0noIRuYYemJhg1HWarUrxdN
         /HA/2UVkJUVrvq1ZNQzVsb+5YRRjobiQ09oPn/+48DDIPeQftbxoiuffytdQKC8EJL3q
         cczDshWTAqB/2IkS7o700QmjwzhSeL5KpT2l4=
Received: by 10.224.200.6 with SMTP id eu6mr1946669qab.220.1308094329243;
        Tue, 14 Jun 2011 16:32:09 -0700 (PDT)
Received: from [10.172.1.74] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id p10sm5783712qcu.37.2011.06.14.16.32.07
        (version=TLSv1/SSLv3 cipher=OTHER);
        Tue, 14 Jun 2011 16:32:08 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: HADOOP-7106 (project unsplit) this weekend
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <BANLkTimJ5Nr9MqdLHoqmCyo0rGgouCtEDA@mail.gmail.com>
Date: Wed, 15 Jun 2011 09:32:03 +1000
Content-Transfer-Encoding: 7bit
Message-Id: <F6E12C16-CE83-4275-AFAB-64D16AC34501@Holsman.NET>
References: <BANLkTinVKj9DE+UoLRfhjVNA=mBR0k5yFg@mail.gmail.com> <CA1BA3F6.78F3%jnaisbit@yahoo-inc.com> <BANLkTi=R4XV4-0formszUMymD_7h1QO4OQ@mail.gmail.com> <550632.48430.qm@web65913.mail.ac4.yahoo.com> <BANLkTimJ5Nr9MqdLHoqmCyo0rGgouCtEDA@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)


On Jun 14, 2011, at 11:49 AM, Todd Lipcon wrote:

> On Mon, Jun 13, 2011 at 11:42 AM, Tsz Wo (Nicholas), Sze <
> s29752-hadoopgeneral@yahoo.com> wrote:
> 
>> A few minor problems:
>> (1) I had committed MAPREDUCE-2588, however, an commit email was sent to
>> both
>> common-commits@ and mapreduce-commits@.
>> 
> 
> I put up a patch to fix this on HADOOP-7106. Unfortunately I think Doug and
> Ian are the only ones who can commit it, and they're both traveling at the
> moment. So we may have to endure dup emails for a day or two. Sorry for the
> mailbox noise
> 

should be fixed now.
(back home, nothing quite like sleeping in your own bed)

> -Todd
> 
> -- 
> Todd Lipcon
> Software Engineer, Cloudera


From general-return-3704-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 23:35:11 2011
Return-Path: <general-return-3704-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E081160B3
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 14 Jun 2011 23:35:10 +0000 (UTC)
Received: (qmail 63503 invoked by uid 500); 14 Jun 2011 23:35:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63448 invoked by uid 500); 14 Jun 2011 23:35:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63433 invoked by uid 99); 14 Jun 2011 23:35:09 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 23:35:09 +0000
Received: from localhost (HELO awittena-md.linkedin.biz) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 23:35:09 +0000
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
Mime-Version: 1.0 (Apple Message framework v1082)
Content-Type: text/plain; charset=us-ascii
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
Date: Tue, 14 Jun 2011 16:35:04 -0700
Cc: Apache Brand Management <trademarks@apache.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <4F34B183-F0B2-4C42-B132-8D19DBCE9219@apache.org>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On Jun 14, 2011, at 3:56 PM, Owen O'Malley wrote:

> All,
>   Steve Loughran has done some great work on defining what can be =
called Hadoop at http://wiki.apache.org/hadoop/Defining%20Hadoop. After =
some cleanup from Noirin and Shane, I think we've got a really good =
base. I'd like a vote to approve the content (at the current revision =
12) and put the content on our web site.
>=20
> Clearly, I'm +1.


	This is awesome.  Good job everyone!

	A minor nit: I'd like to see some cleanup between the first =
paragraph and the fourth paragraph in compatibility.  Or was the =
re-iteration of our "not a standards committee" intentional?  It is sort =
of awkward as it is currently written.

	Also, where can I download Camshaft?



From general-return-3705-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 23:39:25 2011
Return-Path: <general-return-3705-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A0B646155
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 14 Jun 2011 23:39:25 +0000 (UTC)
Received: (qmail 70346 invoked by uid 500); 14 Jun 2011 23:39:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70295 invoked by uid 500); 14 Jun 2011 23:39:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70287 invoked by uid 99); 14 Jun 2011 23:39:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 23:39:23 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.220.176] (HELO mail-vx0-f176.google.com) (209.85.220.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 23:39:17 +0000
Received: by vxa37 with SMTP id 37so7968572vxa.35
        for <general@hadoop.apache.org>; Tue, 14 Jun 2011 16:38:56 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:subject:mime-version:content-type:from
         :in-reply-to:date:cc:message-id:references:to:x-mailer;
        bh=LuhDH8RqdnoBHVfmtKgG8HQeJ1fglDS7euN1aihrUqA=;
        b=NCj9sfMMWxMd3aMCfeKJ/FWDmiLZuWzapZz2W9ykqGNMXeiirZCmfS4//b/1Ujc12X
         89sPJmYrmD24Q2dIeVqk7GlUFtpe9+oSBs2Xtn4uof448IIr1aTrCWBJJLcEYLrDbnwG
         R8lhUnVZP1ksXlq3uvkYWA02o19I42h8JNwg4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=subject:mime-version:content-type:from:in-reply-to:date:cc
         :message-id:references:to:x-mailer;
        b=FgwFlABMqFxijnYFgOImMy91yuQ11nRjTlNGNJ6CPfvMXF+2cIlQjdLctrJj1upX2K
         nWUL7Y/Ygmz/aJiaGSay+hc88YCLj6r2nx42TpaPJtyyVb2PHDTfkwdDMq0JzuhIym5Z
         uosozoPqpSj9CBRnAuLa+OM/RsOVwiNh+UjeI=
Received: by 10.52.76.198 with SMTP id m6mr658061vdw.0.1308094736740;
        Tue, 14 Jun 2011 16:38:56 -0700 (PDT)
Received: from [10.172.1.74] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id j4sm2504037vdu.31.2011.06.14.16.38.54
        (version=SSLv3 cipher=OTHER);
        Tue, 14 Jun 2011 16:38:56 -0700 (PDT)
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
Mime-Version: 1.0 (Apple Message framework v1084)
Content-Type: multipart/alternative; boundary=Apple-Mail-24--924950421
From: Ian Holsman <ian@holsman.net>
In-Reply-To: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
Date: Wed, 15 Jun 2011 09:38:51 +1000
Cc: general@hadoop.apache.org
Message-Id: <A81C6B1F-4A2F-4852-A1AB-AB2C7FA39184@Holsman.net>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
To: trademarks@apache.org
X-Mailer: Apple Mail (2.1084)

--Apple-Mail-24--924950421
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

+1.
great job Steve!

On Jun 15, 2011, at 8:56 AM, Owen O'Malley wrote:

> All,
>    Steve Loughran has done some great work on defining what can be =
called Hadoop at http://wiki.apache.org/hadoop/Defining%20Hadoop. After =
some cleanup from Noirin and Shane, I think we've got a really good =
base. I'd like a vote to approve the content (at the current revision =
12) and put the content on our web site.
>=20
> Clearly, I'm +1.
>=20
> -- Owen

--
Ian Holsman
Ian@Holsman.net
PH: +1-703 879-3128 AOLIM: ianholsman Skype:iholsman

Never explain - your friends do not need it and your enemies will not =
believe you anyway. Elbert Hubbard





--Apple-Mail-24--924950421--

From general-return-3706-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 00:16:58 2011
Return-Path: <general-return-3706-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8C09467D6
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 00:16:58 +0000 (UTC)
Received: (qmail 5789 invoked by uid 500); 15 Jun 2011 00:16:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5737 invoked by uid 500); 15 Jun 2011 00:16:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5729 invoked by uid 99); 15 Jun 2011 00:16:56 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 00:16:56 +0000
Received: from localhost (HELO awittena-md.linkedin.biz) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 00:16:56 +0000
From: Allen Wittenauer <aw@apache.org>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: Hadoop Java Versions
Date: Tue, 14 Jun 2011 17:16:54 -0700
Message-Id: <1F501BC2-7671-45B5-8D31-599C25068191@apache.org>
To: general@hadoop.apache.org
Mime-Version: 1.0 (Apple Message framework v1082)
X-Mailer: Apple Mail (2.1082)


While we're looking at the wiki, could folks update =
http://wiki.apache.org/hadoop/HadoopJavaVersions with whatever versions =
of Hadoop they are using successfully?

Thanks.

P.S., yes, I'm thinking about upgrading ours. :p=

From general-return-3707-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 00:17:33 2011
Return-Path: <general-return-3707-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2C8AA6B17
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 00:17:33 +0000 (UTC)
Received: (qmail 8964 invoked by uid 500); 15 Jun 2011 00:17:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8907 invoked by uid 500); 15 Jun 2011 00:17:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8899 invoked by uid 99); 15 Jun 2011 00:17:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 00:17:31 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.213.48] (HELO mail-yw0-f48.google.com) (209.85.213.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 00:17:23 +0000
Received: by ywk9 with SMTP id 9so4480903ywk.35
        for <general@hadoop.apache.org>; Tue, 14 Jun 2011 17:17:02 -0700 (PDT)
Received: by 10.150.61.20 with SMTP id j20mr88593yba.175.1308097020430; Tue,
 14 Jun 2011 17:17:00 -0700 (PDT)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.150.205.10 with HTTP; Tue, 14 Jun 2011 17:16:40 -0700 (PDT)
In-Reply-To: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
From: Konstantin Boudnik <cos@apache.org>
Date: Tue, 14 Jun 2011 17:16:40 -0700
X-Google-Sender-Auth: ISh2Fc4pdTHWyeMFqx6286tIZKs
Message-ID: <BANLkTimyYuLSYwL4AOoSnSu8qGXerfnMkg@mail.gmail.com>
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

+1 - makes sense!

--
=A0 Take care,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616 =A06115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.



On Tue, Jun 14, 2011 at 15:56, Owen O'Malley <omalley@apache.org> wrote:
> All,
> =A0 Steve Loughran has done some great work on defining what can be calle=
d Hadoop at http://wiki.apache.org/hadoop/Defining%20Hadoop. After some cle=
anup from Noirin and Shane, I think we've got a really good base. I'd like =
a vote to approve the content (at the current revision 12) and put the cont=
ent on our web site.
>
> Clearly, I'm +1.
>
> -- Owen

From general-return-3708-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 00:48:31 2011
Return-Path: <general-return-3708-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8DA6D6CE6
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 00:48:31 +0000 (UTC)
Received: (qmail 46465 invoked by uid 500); 15 Jun 2011 00:48:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46396 invoked by uid 500); 15 Jun 2011 00:48:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46382 invoked by uid 99); 15 Jun 2011 00:48:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 00:48:29 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 00:48:22 +0000
Received: by wwi18 with SMTP id 18so6794440wwi.29
        for <general@hadoop.apache.org>; Tue, 14 Jun 2011 17:48:02 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.46.21 with SMTP id q21mr1393038web.113.1308098881800; Tue,
 14 Jun 2011 17:48:01 -0700 (PDT)
Received: by 10.216.30.212 with HTTP; Tue, 14 Jun 2011 17:48:01 -0700 (PDT)
In-Reply-To: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
Date: Tue, 14 Jun 2011 17:48:01 -0700
Message-ID: <BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com>
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Cc: Apache Brand Management <trademarks@apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Tue, Jun 14, 2011 at 3:56 PM, Owen O'Malley <omalley@apache.org> wrote:
> All,
> =A0 Steve Loughran has done some great work on defining what can be calle=
d Hadoop at http://wiki.apache.org/hadoop/Defining%20Hadoop. After some cle=
anup from Noirin and Shane, I think we've got a really good base. I'd like =
a vote to approve the content (at the current revision 12) and put the cont=
ent on our web site.
>
> Clearly, I'm +1.
>
> -- Owen

Thanks for putting this together Steve, good stuff!

Wrt derivative works, it's not clear from the document, but I think we
should explicitly adopt the policy of HTTPD and Subversion that
backported patches from trunk and security fixes are permitted.
Specifically, that cherry-picking changes from trunk or release
branches and, in general, any code that's been subject to lazy
consensus approval by the PMC does not make you a derivative work. For
example, RedHat backports [1] to Apache HTTP and of course still calls
it Apache HTTP.

In short, an Apache Hadoop release with a backport of PMC approved
code or critical security fix is not powered by Hadoop, it is Hadoop,
while a new product that contains or runs atop Hadoop is powered by
Hadoop.

Reasonable?

Thanks,
Eli

1. https://access.redhat.com/security/updates/backporting/?sc_cid=3D3093

From general-return-3709-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 01:15:09 2011
Return-Path: <general-return-3709-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 92AD8690B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 01:15:09 +0000 (UTC)
Received: (qmail 64932 invoked by uid 500); 15 Jun 2011 01:15:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 64789 invoked by uid 500); 15 Jun 2011 01:15:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 64769 invoked by uid 99); 15 Jun 2011 01:15:07 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 01:15:07 +0000
Received: from localhost (HELO awittena-md.linkedin.biz) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 01:15:06 +0000
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
Mime-Version: 1.0 (Apple Message framework v1082)
Content-Type: text/plain; charset=us-ascii
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com>
Date: Tue, 14 Jun 2011 18:15:04 -0700
Cc: Apache Brand Management <trademarks@apache.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <FBF55C01-D66A-44C0-9252-D54E7CAB103D@apache.org>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org> <BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On Jun 14, 2011, at 5:48 PM, Eli Collins wrote:
> In short, an Apache Hadoop release with a backport of PMC approved
> code or critical security fix is not powered by Hadoop, it is Hadoop,
> while a new product that contains or runs atop Hadoop is powered by
> Hadoop.
>=20
> Reasonable?

	I'd say: Security, yes.  Features, no.

	The reason I say this is because there have been many, many, =
many posts in the -user mailing lists where people are confused as to =
what versions have what features because their local branch has a back =
ported fix.  [I think I run out of fingers if I count how many times =
just the mapred.map.child.java.opts was said to be "in 20" prior to the =
0.20.203 release...]

	This also adds pressure to do timely releases. :)


From general-return-3710-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 01:16:23 2011
Return-Path: <general-return-3710-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 67D3C6CE4
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 01:16:23 +0000 (UTC)
Received: (qmail 67597 invoked by uid 500); 15 Jun 2011 01:16:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67481 invoked by uid 500); 15 Jun 2011 01:16:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67473 invoked by uid 99); 15 Jun 2011 01:16:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 01:16:21 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of eli@cloudera.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 01:16:15 +0000
Received: by wwi18 with SMTP id 18so6811301wwi.29
        for <general@hadoop.apache.org>; Tue, 14 Jun 2011 18:15:54 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.221.103 with SMTP id q81mr6905438wep.83.1308100554090;
 Tue, 14 Jun 2011 18:15:54 -0700 (PDT)
Received: by 10.216.30.212 with HTTP; Tue, 14 Jun 2011 18:15:54 -0700 (PDT)
In-Reply-To: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
Date: Tue, 14 Jun 2011 18:15:54 -0700
Message-ID: <BANLkTindvA4g1JpTG2b2Csz6tbRAd5_Tgg@mail.gmail.com>
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Cc: Apache Brand Management <trademarks@apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Tue, Jun 14, 2011 at 3:56 PM, Owen O'Malley <omalley@apache.org> wrote:
> All,
> =A0 Steve Loughran has done some great work on defining what can be calle=
d Hadoop at http://wiki.apache.org/hadoop/Defining%20Hadoop. After some cle=
anup from Noirin and Shane, I think we've got a really good base. I'd like =
a vote to approve the content (at the current revision 12) and put the cont=
ent on our web site.
>
> Clearly, I'm +1.
>
> -- Owen

I'd like to make another suggestion. Currently we call two types of
things powered by Apache Hadoop:

1. Something that runs on Hadoop (eg HBase or Karmasphere)
2. Something that includes Hadoop artifacts/source code

Shouldn't we distinguish between these two, such that the 2nd is not
powered by Hadoop? Eg tc server is not powered by Apache Tomcat right?

Apologies for having discussion on a vote thread but this is the first
time I've seen the current revision and it seems reasonable to have an
opportunity to discuss a specific revision before voting on it.

Thanks,
Eli

From general-return-3711-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 01:45:53 2011
Return-Path: <general-return-3711-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 42E166CE1
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 01:45:53 +0000 (UTC)
Received: (qmail 86848 invoked by uid 500); 15 Jun 2011 01:45:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86789 invoked by uid 500); 15 Jun 2011 01:45:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86781 invoked by uid 99); 15 Jun 2011 01:45:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 01:45:51 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 01:45:44 +0000
Received: by wwi18 with SMTP id 18so6828304wwi.29
        for <general@hadoop.apache.org>; Tue, 14 Jun 2011 18:45:23 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.245.4 with SMTP id n4mr1478588wer.83.1308102323390; Tue,
 14 Jun 2011 18:45:23 -0700 (PDT)
Received: by 10.216.30.212 with HTTP; Tue, 14 Jun 2011 18:45:23 -0700 (PDT)
In-Reply-To: <FBF55C01-D66A-44C0-9252-D54E7CAB103D@apache.org>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
	<BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com>
	<FBF55C01-D66A-44C0-9252-D54E7CAB103D@apache.org>
Date: Tue, 14 Jun 2011 18:45:23 -0700
Message-ID: <BANLkTi=BWPUEoqD3L98A9AXLkbafZs15Zw@mail.gmail.com>
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Cc: Apache Brand Management <trademarks@apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Tue, Jun 14, 2011 at 6:15 PM, Allen Wittenauer <aw@apache.org> wrote:
>
> On Jun 14, 2011, at 5:48 PM, Eli Collins wrote:
>> In short, an Apache Hadoop release with a backport of PMC approved
>> code or critical security fix is not powered by Hadoop, it is Hadoop,
>> while a new product that contains or runs atop Hadoop is powered by
>> Hadoop.
>>
>> Reasonable?
>
> =A0 =A0 =A0 =A0I'd say: Security, yes. =A0Features, no.
>
> =A0 =A0 =A0 =A0The reason I say this is because there have been many, man=
y, many posts in the -user mailing lists where people are confused as to wh=
at versions have what features because their local branch has a back ported=
 fix. =A0[I think I run out of fingers if I count how many times just the m=
apred.map.child.java.opts was said to be "in 20" prior to the 0.20.203 rele=
ase...]
>
> =A0 =A0 =A0 =A0This also adds pressure to do timely releases. :)
>

I agree this is a problem, I don't think this is an effective means of
solving it.

Are we really going to go after all the web companies that patch in an
enhancement to their current Hadoop build and tell them to stop saying
that they are using Hadoop?  You've patched Hadoop many times, should
your employer not be able to say they use Hadoop?  I'm -1 on a
proposal that does this.

Thanks,
Eli

From general-return-3712-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 02:16:53 2011
Return-Path: <general-return-3712-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9E1D7497A
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 02:16:53 +0000 (UTC)
Received: (qmail 26608 invoked by uid 500); 15 Jun 2011 02:16:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26557 invoked by uid 500); 15 Jun 2011 02:16:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26547 invoked by uid 99); 15 Jun 2011 02:16:52 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 02:16:52 +0000
Received: from localhost (HELO mail-iy0-f176.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 02:16:51 +0000
Received: by iym1 with SMTP id 1so8071024iym.35
        for <general@hadoop.apache.org>; Tue, 14 Jun 2011 19:16:51 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.82.139 with SMTP id b11mr6945938ibl.134.1308104211156;
 Tue, 14 Jun 2011 19:16:51 -0700 (PDT)
Received: by 10.231.212.149 with HTTP; Tue, 14 Jun 2011 19:16:51 -0700 (PDT)
In-Reply-To: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
Date: Tue, 14 Jun 2011 19:16:51 -0700
Message-ID: <BANLkTi=sZygosNJzM3-F+1c855TDud2bSQ@mail.gmail.com>
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

+1 on revision 12. Thanks for all your work on this, Steve. -C

On Tue, Jun 14, 2011 at 3:56 PM, Owen O'Malley <omalley@apache.org> wrote:
> All,
> =A0=A0 Steve Loughran has done some great work on defining what can be ca=
lled
> Hadoop at=A0http://wiki.apache.org/hadoop/Defining%20Hadoop. After some
> cleanup from Noirin and Shane, I think we've got a really good base. I'd
> like a vote to approve the content (at the current revision 12) and put t=
he
> content on our web site.
> Clearly, I'm +1.
> -- Owen

From general-return-3713-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 02:33:26 2011
Return-Path: <general-return-3713-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5CFC64C59
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 02:33:26 +0000 (UTC)
Received: (qmail 36622 invoked by uid 500); 15 Jun 2011 02:33:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36504 invoked by uid 500); 15 Jun 2011 02:33:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36496 invoked by uid 99); 15 Jun 2011 02:33:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 02:33:24 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.213.48] (HELO mail-yw0-f48.google.com) (209.85.213.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 02:33:18 +0000
Received: by ywk9 with SMTP id 9so4553362ywk.35
        for <general@hadoop.apache.org>; Tue, 14 Jun 2011 19:32:57 -0700 (PDT)
Received: by 10.150.235.20 with SMTP id i20mr184200ybh.232.1308105177144; Tue,
 14 Jun 2011 19:32:57 -0700 (PDT)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.150.205.10 with HTTP; Tue, 14 Jun 2011 19:32:36 -0700 (PDT)
In-Reply-To: <BANLkTindvA4g1JpTG2b2Csz6tbRAd5_Tgg@mail.gmail.com>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org> <BANLkTindvA4g1JpTG2b2Csz6tbRAd5_Tgg@mail.gmail.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Tue, 14 Jun 2011 19:32:36 -0700
X-Google-Sender-Auth: SD-vUthQjY0fs8FVe64Ppr59-yU
Message-ID: <BANLkTim8Ov+Vex_ZtA7Sac8GZXmXMy54vw@mail.gmail.com>
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Tue, Jun 14, 2011 at 18:15, Eli Collins <eli@cloudera.com> wrote:
> On Tue, Jun 14, 2011 at 3:56 PM, Owen O'Malley <omalley@apache.org> wrote=
:
>> All,
>> =A0 Steve Loughran has done some great work on defining what can be call=
ed Hadoop at http://wiki.apache.org/hadoop/Defining%20Hadoop. After some cl=
eanup from Noirin and Shane, I think we've got a really good base. I'd like=
 a vote to approve the content (at the current revision 12) and put the con=
tent on our web site.
>>
>> Clearly, I'm +1.
>>
>> -- Owen
>
> I'd like to make another suggestion. Currently we call two types of
> things powered by Apache Hadoop:
>
> 1. Something that runs on Hadoop (eg HBase or Karmasphere)

To be completely precise Karmasphere doesn't 'run on Hadoop'. Their
products "integrate with a variety of Hadoop distributions and related
technologies..." as you can see here
http://karmasphere.com/Miscellaneous/overview.html. Although in case
of HBase you right ;)

Cos

> 2. Something that includes Hadoop artifacts/source code
>
> Shouldn't we distinguish between these two, such that the 2nd is not
> powered by Hadoop? Eg tc server is not powered by Apache Tomcat right?
>
> Apologies for having discussion on a vote thread but this is the first
> time I've seen the current revision and it seems reasonable to have an
> opportunity to discuss a specific revision before voting on it.
>
> Thanks,
> Eli
>

From general-return-3714-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 02:46:34 2011
Return-Path: <general-return-3714-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A1ABA40D3
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 02:46:34 +0000 (UTC)
Received: (qmail 45153 invoked by uid 500); 15 Jun 2011 02:46:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44977 invoked by uid 500); 15 Jun 2011 02:46:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44967 invoked by uid 99); 15 Jun 2011 02:46:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 02:46:31 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of owen.omalley@gmail.com designates 209.85.210.176 as permitted sender)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 02:46:24 +0000
Received: by iym1 with SMTP id 1so8092765iym.35
        for <general@hadoop.apache.org>; Tue, 14 Jun 2011 19:46:03 -0700 (PDT)
Received: by 10.231.111.228 with SMTP id t36mr7369180ibp.59.1308105963677;
        Tue, 14 Jun 2011 19:46:03 -0700 (PDT)
Received: from [10.0.0.24] (c-98-234-216-58.hsd1.ca.comcast.net [98.234.216.58])
        by mx.google.com with ESMTPS id w8sm3592908ibh.1.2011.06.14.19.45.59
        (version=SSLv3 cipher=OTHER);
        Tue, 14 Jun 2011 19:46:02 -0700 (PDT)
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
Mime-Version: 1.0 (Apple Message framework v1084)
Content-Type: text/plain; charset=us-ascii
From: Owen O'Malley <omalley@apache.org>
In-Reply-To: <BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com>
Date: Tue, 14 Jun 2011 19:45:58 -0700
Cc: general@hadoop.apache.org
Content-Transfer-Encoding: quoted-printable
Message-Id: <090F2993-B9AA-410F-AD99-7AC047FFC974@apache.org>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org> <BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com>
To: trademarks@apache.org
X-Mailer: Apple Mail (2.1084)


On Jun 14, 2011, at 5:48 PM, Eli Collins wrote:

> Wrt derivative works, it's not clear from the document, but I think we
> should explicitly adopt the policy of HTTPD and Subversion that
> backported patches from trunk and security fixes are permitted.

Actually, the document is extremely clear that only Apache releases may =
be called Hadoop.

There was a very long thread about why the rapidly expanding =
Hadoop-ecosystem is leading to at lot of customer confusion about the =
different "versions" of Hadoop. We as the Hadoop project don't have the =
resources or the necessary compatibility test suite to test =
compatibility between the different sets of cherry picked patches. We =
also don't have time to ensure that all of the 1,000's of patches =
applied to 0.20.2 in each of the many (10? 15?) different versions have =
been committed to trunk. Futhermore, under the Apache license, a company =
Foo could claim that it is a cherry pick version of Hadoop without =
releasing their source code that would enable verification.

In summary,
  1. Hadoop is very successful.
  2. There are many different commercial products that are trying to use =
the Hadoop name.
  3. We can't check or enforce that the cherry pick versions are =
following the rules.
  4. We don't have a TCK like Java does to validate new versions are =
compatible.
  5. By far the most fair way to ensure compatibility and fairness =
between companies is that only Apache Hadoop releases may be called =
Hadoop.

That said, a package that includes a small number (< 3) of security =
patches that haven't been released yet doesn't seem unreasonable.

-- Owen


From general-return-3715-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 02:46:55 2011
Return-Path: <general-return-3715-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 849A94422
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 02:46:55 +0000 (UTC)
Received: (qmail 46716 invoked by uid 500); 15 Jun 2011 02:46:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46605 invoked by uid 500); 15 Jun 2011 02:46:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46590 invoked by uid 99); 15 Jun 2011 02:46:53 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 02:46:53 +0000
Received: from localhost (HELO dhcp-02.private.iobm.com) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 02:46:53 +0000
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
Mime-Version: 1.0 (Apple Message framework v1082)
Content-Type: text/plain; charset=us-ascii
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <BANLkTi=BWPUEoqD3L98A9AXLkbafZs15Zw@mail.gmail.com>
Date: Tue, 14 Jun 2011 19:46:52 -0700
Cc: Apache Brand Management <trademarks@apache.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <92D32E6E-3B65-464F-B91B-4C30EAB2642C@apache.org>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org> <BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com> <FBF55C01-D66A-44C0-9252-D54E7CAB103D@apache.org> <BANLkTi=BWPUEoqD3L98A9AXLkbafZs15Zw@mail.gmail.com>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On Jun 14, 2011, at 6:45 PM, Eli Collins wrote:
> Are we really going to go after all the web companies that patch in an
> enhancement to their current Hadoop build and tell them to stop saying
> that they are using Hadoop?  You've patched Hadoop many times, should
> your employer not be able to say they use Hadoop?  I'm -1 on a
> proposal that does this.

	I think there is a big difference between some company that uses =
Hadoop with some patches internally and a company that puts out a =
distribution for others to use, usually for-profit.=

From general-return-3716-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 02:52:42 2011
Return-Path: <general-return-3716-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 40F274BE0
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 02:52:42 +0000 (UTC)
Received: (qmail 50543 invoked by uid 500); 15 Jun 2011 02:52:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50274 invoked by uid 500); 15 Jun 2011 02:52:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50266 invoked by uid 99); 15 Jun 2011 02:52:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 02:52:39 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.161.176] (HELO mail-gx0-f176.google.com) (209.85.161.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 02:52:33 +0000
Received: by gxk7 with SMTP id 7so5919985gxk.35
        for <general@hadoop.apache.org>; Tue, 14 Jun 2011 19:52:12 -0700 (PDT)
Received: by 10.150.165.4 with SMTP id n4mr195456ybe.289.1308106332085; Tue,
 14 Jun 2011 19:52:12 -0700 (PDT)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.150.205.10 with HTTP; Tue, 14 Jun 2011 19:51:52 -0700 (PDT)
In-Reply-To: <92D32E6E-3B65-464F-B91B-4C30EAB2642C@apache.org>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
 <BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com> <FBF55C01-D66A-44C0-9252-D54E7CAB103D@apache.org>
 <BANLkTi=BWPUEoqD3L98A9AXLkbafZs15Zw@mail.gmail.com> <92D32E6E-3B65-464F-B91B-4C30EAB2642C@apache.org>
From: Konstantin Boudnik <cos@apache.org>
Date: Tue, 14 Jun 2011 19:51:52 -0700
X-Google-Sender-Auth: usmVmnA-ybW2BsoOdtGWs8T6opY
Message-ID: <BANLkTi=1JGe8kVortj1Act3jdt9L2RCrhw@mail.gmail.com>
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
To: general@hadoop.apache.org
Cc: Apache Brand Management <trademarks@apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Tue, Jun 14, 2011 at 19:46, Allen Wittenauer <aw@apache.org> wrote:
>
> On Jun 14, 2011, at 6:45 PM, Eli Collins wrote:
>> Are we really going to go after all the web companies that patch in an
>> enhancement to their current Hadoop build and tell them to stop saying
>> that they are using Hadoop? =A0You've patched Hadoop many times, should
>> your employer not be able to say they use Hadoop? =A0I'm -1 on a
>> proposal that does this.
>
> =A0 =A0 =A0 =A0I think there is a big difference between some company tha=
t uses Hadoop with some patches internally and a company that puts out a di=
stribution for others to use, usually for-profit.

Just as the reminder: this whole conversation has started as a result
of EMC announcement of 100% compatible version of Apache Hadoop. So,
Allen's point is right on target here: the above example is simply
incorrect.

Cos

From general-return-3717-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 03:00:46 2011
Return-Path: <general-return-3717-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3A847421A
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 03:00:46 +0000 (UTC)
Received: (qmail 58028 invoked by uid 500); 15 Jun 2011 03:00:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57947 invoked by uid 500); 15 Jun 2011 03:00:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57923 invoked by uid 99); 15 Jun 2011 03:00:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 03:00:43 +0000
X-ASF-Spam-Status: No, hits=1.7 required=5.0
	tests=ASF_LIST_OPS,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of hailong.yang1115@gmail.com designates 74.125.83.176 as permitted sender)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 03:00:33 +0000
Received: by pve37 with SMTP id 37so4105906pve.35
        for <multiple recipients>; Tue, 14 Jun 2011 20:00:11 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:date:from:to:references:subject:message-id
         :x-mailer:mime-version:content-type;
        bh=cmo9IaFfqAHHiGDLFg2R26fPuXI2iPG5Wws6ya2Hw+8=;
        b=i7EKWyvZ1xd+TN42QHvEDcExE/ftZMiWK/uofiIBLuEL7GOd94VWndZV3bhSw7/Y4e
         LNOGhRKgaGO62Sk6Kzg6/s3iahRRwKnPQ4h2lbDGps39PuLBlsOFNkv14yUMXi6eCae/
         kHlEZUrWednizAXsEP04SmG3TENLhS4fjyPCg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=date:from:to:references:subject:message-id:x-mailer:mime-version
         :content-type;
        b=vqRotgOCEp5GwwotIFbBSHO4WWimxwFaPCcZG/Gqs8jOnjmi1Llj26ge5uE8Hwfb9i
         Agmn73CJfDType1zHeYZwOKRS0R8Ho93iHbQLzAvuGfaqLW/tk/bNamZ1lYLpunfdZHz
         Ah7MWYJSaOavvdxE7fSOIVuQUimtR4ye/AQLE=
Received: by 10.68.10.42 with SMTP id f10mr65196pbb.259.1308106811574;
        Tue, 14 Jun 2011 20:00:11 -0700 (PDT)
Received: from HailongYangLen (tu200023.ip.tsinghua.edu.cn [166.111.200.23])
        by mx.google.com with ESMTPS id g10sm1081091pbi.9.2011.06.14.20.00.07
        (version=SSLv3 cipher=OTHER);
        Tue, 14 Jun 2011 20:00:10 -0700 (PDT)
Date: Wed, 15 Jun 2011 11:00:08 +0800
From: "hailong.yang1115" <hailong.yang1115@gmail.com>
To: "general" <general@hadoop.apache.org>,
 "general-help" <general-help@hadoop.apache.org>,
 "hdfs-user" <hdfs-user@hadoop.apache.org>,
 "hdfs-user-help" <hdfs-user-help@hadoop.apache.org>
References: <201106101328462304641@gmail.com>
Subject: Fw: Problems about the job counters
Message-ID: <201106151100061050081@gmail.com>
X-mailer: Foxmail 6, 15, 201, 26 [cn]
Mime-Version: 1.0
Content-Type: multipart/alternative;
	boundary="=====003_Dragon735314350340_====="
X-Virus-Checked: Checked by ClamAV on apache.org

--=====003_Dragon735314350340_=====
Content-Type: text/plain;
	charset="gb2312"
Content-Transfer-Encoding: base64

DQpTb3JyeSBmb3Igc2VuZGluZyB0aGlzIGVtYWlsIGFnYWluIGJ1dCBJIGdvdCBubyBhbnN3ZXJz
IGZyb20gdGhlIGZpcnN0IG9uZS4gQW55b25lIHBsZWFzZSBoZWxwIG9yIGZvcndhcmQgaXQgdG8g
bWFpbC1saXN0IHRoYXQgd291bGQgaGVscC4NCg0KMjAxMS0wNi0xNQ0KDQoNCg0KKioqKioqKioq
KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioNCiogSGFpbG9uZyBZYW5nLCBQ
aEQuIENhbmRpZGF0ZSANCiogU2luby1HZXJtYW4gSm9pbnQgU29mdHdhcmUgSW5zdGl0dXRlLCAN
CiogU2Nob29sIG9mIENvbXB1dGVyIFNjaWVuY2UmRW5naW5lZXJpbmcsIEJlaWhhbmcgVW5pdmVy
c2l0eQ0KKiBQaG9uZTogKDg2LTAxMCk4MjMxNTkwOA0KKiBFbWFpbDogaGFpbG9uZy55YW5nMTEx
NUBnbWFpbC5jb20NCiogQWRkcmVzczogRzQxMywgTmV3IE1haW4gQnVpbGRpbmcgaW4gQmVpaGFu
ZyBVbml2ZXJzaXR5LCANCiogICAgICAgICAgICAgIE5vLjM3IFh1ZVl1YW4gUm9hZCxIYWlEaWFu
IERpc3RyaWN0LCANCiogICAgICAgICAgICAgIEJlaWppbmcsUC5SLkNoaW5hLDEwMDE5MQ0KKioq
KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioNCg0KDQoNCreivP7I
y6O6IGhhaWxvbmcueWFuZzExMTUNCreiy83Ksbzko7ogMjAxMS0wNi0xMCAxMzoyODo0Ng0KytW8
/sjLo7ogZ2VuZXJhbA0Ks63LzaO6IA0K1vfM4qO6IFByb2JsZW1zIGFib3V0IHRoZSBqb2IgY291
bnRlcnMNCg0KRGVhciBhbGwsDQoNCkkgYW0gdHJ5aW5nIHRvIHRoZSBidWlsdC1pbiBleGFtcGxl
IHdvcmRjb3VudCB3aXRoIG5lYXJseSAxNUdCIGlucHV0LiBXaGVuIHRoZSBIYWRvb3Agam9iIGZp
bmlzaGVkLCBJIGdvdCB0aGUgZm9sbG93aW5nIGNvdW50ZXJzLg0KDQoNCkNvdW50ZXJNYXBSZWR1
Y2VUb3RhbA0KSm9iIENvdW50ZXJzTGF1bmNoZWQgcmVkdWNlIHRhc2tzMDAxDQpSYWNrLWxvY2Fs
IG1hcCB0YXNrczAwMzUNCkxhdW5jaGVkIG1hcCB0YXNrczAwMiwzMTgNCkRhdGEtbG9jYWwgbWFw
IHRhc2tzMDAyLDI4Mw0KRmlsZVN5c3RlbUNvdW50ZXJzRklMRV9CWVRFU19SRUFEMjIsODYzLDU4
MCw2NTYxNyw2NTQsOTQzLDM0MTQwLDUxOCw1MjMsOTk3DQpIREZTX0JZVEVTX1JFQUQxNTQsNDAw
LDk5Nyw0NTkwMTU0LDQwMCw5OTcsNDU5DQpGSUxFX0JZVEVTX1dSSVRURU4zMyw0OTAsODI5LDQw
MzE3LDY1NCw5NDMsMzQxNTEsMTQ1LDc3Miw3NDQNCkhERlNfQllURVNfV1JJVFRFTjAyLDc0Nywz
NTYsNzA0Miw3NDcsMzU2LDcwNA0KDQoNCk15IHF1ZXN0aW9uIGlzIHdoYXQgZG9lcyB0aGUgRklM
RV9CWVRFU19SRUFEIGNvdW50ZXIgbWVhbj8gQW5kIHdoYXQgaXMgdGhlIGRpZmZlcmVuY2UgYmV0
d2VlbiBGSUxFX0JZVEVTX1JFQUQgYW5kIEhERlNfQllURVNfUkVBRD8gSW4gbXkgb3Bpbmlvbiwg
YWxsIHRoZSBpbnB1dCBpcyBsb2NhdGVkIGluIEhERlMsIHNvIHdoZXJlIGRvZXMgRklMRV9CWVRF
U19SRUFEIGNvbWUgZnJvbSBkdXJpbmcgdGhlIG1hcCBwaGFzZT8NCg0KDQpBbnkgaGVscCB3aWxs
IGJlIGFwcHJlY2lhdGVkIQ0KDQpIYWlsb25nDQoNCjIwMTEtMDYtMTAgDQoNCg0KDQoqKioqKioq
KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKg0KKiBIYWlsb25nIFlhbmcs
IFBoRC4gQ2FuZGlkYXRlIA0KKiBTaW5vLUdlcm1hbiBKb2ludCBTb2Z0d2FyZSBJbnN0aXR1dGUs
IA0KKiBTY2hvb2wgb2YgQ29tcHV0ZXIgU2NpZW5jZSZFbmdpbmVlcmluZywgQmVpaGFuZyBVbml2
ZXJzaXR5DQoqIFBob25lOiAoODYtMDEwKTgyMzE1OTA4DQoqIEVtYWlsOiBoYWlsb25nLnlhbmcx
MTE1QGdtYWlsLmNvbQ0KKiBBZGRyZXNzOiBHNDEzLCBOZXcgTWFpbiBCdWlsZGluZyBpbiBCZWlo
YW5nIFVuaXZlcnNpdHksIA0KKiAgICAgICAgICAgICAgTm8uMzcgWHVlWXVhbiBSb2FkLEhhaURp
YW4gRGlzdHJpY3QsIA0KKiAgICAgICAgICAgICAgQmVpamluZyxQLlIuQ2hpbmEsMTAwMTkxDQoq
KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKg0K

--=====003_Dragon735314350340_=====--


From general-return-3718-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 03:19:53 2011
Return-Path: <general-return-3718-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 758864DFF
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 03:19:53 +0000 (UTC)
Received: (qmail 73165 invoked by uid 500); 15 Jun 2011 03:19:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72384 invoked by uid 500); 15 Jun 2011 03:19:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72348 invoked by uid 99); 15 Jun 2011 03:19:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 03:19:48 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of owen.omalley@gmail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 03:19:42 +0000
Received: by iwr19 with SMTP id 19so8153878iwr.35
        for <general@hadoop.apache.org>; Tue, 14 Jun 2011 20:19:21 -0700 (PDT)
Received: by 10.231.187.232 with SMTP id cx40mr7682525ibb.73.1308107961744;
        Tue, 14 Jun 2011 20:19:21 -0700 (PDT)
Received: from [10.0.0.24] (c-98-234-216-58.hsd1.ca.comcast.net [98.234.216.58])
        by mx.google.com with ESMTPS id w11sm455ibw.24.2011.06.14.20.19.20
        (version=SSLv3 cipher=OTHER);
        Tue, 14 Jun 2011 20:19:21 -0700 (PDT)
From: Owen O'Malley <omalley@apache.org>
Content-Type: multipart/alternative; boundary=Apple-Mail-7--911722727
Subject: [VOTE] Powered by Logo
Date: Tue, 14 Jun 2011 20:19:18 -0700
Message-Id: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
To: general@hadoop.apache.org
Mime-Version: 1.0 (Apple Message framework v1084)
X-Mailer: Apple Mail (2.1084)

--Apple-Mail-7--911722727
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

All,
   We've had a wide range of entries for a powered by logo. I've put =
them all on a page, here:

http://people.apache.org/~omalley/hadoop-powered-by/

Since there are a lot of contenders and we only want a single round of =
voting, let's use single transferable vote ( STV =
http://en.wikipedia.org/wiki/Single_transferable_vote). The important =
thing is to pick the images *IN ORDER* that you would like them.

My vote (in order of course): 4, 1, 2, 3, 5, 6.

In other words, I want option 4 most and option 6 least. With STV, you =
don't need to worry about voting for an unpopular choice since your vote =
will automatically roll over to your next choice.

-- Owen



--Apple-Mail-7--911722727--

From general-return-3719-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 03:45:29 2011
Return-Path: <general-return-3719-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AA0B64AB8
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 03:45:29 +0000 (UTC)
Received: (qmail 85838 invoked by uid 500); 15 Jun 2011 03:45:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85562 invoked by uid 500); 15 Jun 2011 03:45:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85540 invoked by uid 99); 15 Jun 2011 03:45:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 03:45:26 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of atm@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 03:45:19 +0000
Received: by bwz8 with SMTP id 8so261955bwz.35
        for <general@hadoop.apache.org>; Tue, 14 Jun 2011 20:44:59 -0700 (PDT)
Received: by 10.204.7.156 with SMTP id d28mr14746bkd.28.1308109499209; Tue, 14
 Jun 2011 20:44:59 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.72.195 with HTTP; Tue, 14 Jun 2011 20:44:29 -0700 (PDT)
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
From: "Aaron T. Myers" <atm@cloudera.com>
Date: Tue, 14 Jun 2011 20:44:29 -0700
Message-ID: <BANLkTimDk13Oh0pmqwOzGayCdPhGf+oOng@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001517588b5a07199904a5b7fa10
X-Virus-Checked: Checked by ClamAV on apache.org

--001517588b5a07199904a5b7fa10
Content-Type: text/plain; charset=ISO-8859-1

My vote: 5 1 6 2 3 4

tl;dr regarding STV - since we're only selecting one winner, STV reduces to
instant-runoff voting.

--
Aaron T. Myers
Software Engineer, Cloudera



On Tue, Jun 14, 2011 at 8:19 PM, Owen O'Malley <omalley@apache.org> wrote:

> All,
>   We've had a wide range of entries for a powered by logo. I've put them
> all on a page, here:
>
> http://people.apache.org/~omalley/hadoop-powered-by/
>
> Since there are a lot of contenders and we only want a single round of
> voting, let's use single transferable vote ( STV
> http://en.wikipedia.org/wiki/Single_transferable_vote). The important
> thing is to pick the images *IN ORDER* that you would like them.
>
> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>
> In other words, I want option 4 most and option 6 least. With STV, you
> don't need to worry about voting for an unpopular choice since your vote
> will automatically roll over to your next choice.
>
> -- Owen
>
>
>

--001517588b5a07199904a5b7fa10--

From general-return-3720-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 04:07:15 2011
Return-Path: <general-return-3720-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3F80146FD
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 04:07:15 +0000 (UTC)
Received: (qmail 440 invoked by uid 500); 15 Jun 2011 04:07:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 392 invoked by uid 500); 15 Jun 2011 04:07:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 383 invoked by uid 99); 15 Jun 2011 04:07:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 04:07:11 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.213.48] (HELO mail-yw0-f48.google.com) (209.85.213.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 04:07:04 +0000
Received: by ywk9 with SMTP id 9so21565ywk.35
        for <general@hadoop.apache.org>; Tue, 14 Jun 2011 21:06:43 -0700 (PDT)
Received: by 10.150.229.17 with SMTP id b17mr263237ybh.61.1308110803143; Tue,
 14 Jun 2011 21:06:43 -0700 (PDT)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.150.205.10 with HTTP; Tue, 14 Jun 2011 21:06:23 -0700 (PDT)
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
From: Konstantin Boudnik <cos@apache.org>
Date: Tue, 14 Jun 2011 21:06:23 -0700
X-Google-Sender-Auth: RVbPV8i2DLJmpjoO9nC5Zr6YuaA
Message-ID: <BANLkTikpASwROUv0MJf9Sn7cS6rPnOa75Q@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

4,2,6,1,3,5
--
=A0 Take care,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616 =A06115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.

On Tue, Jun 14, 2011 at 20:19, Owen O'Malley <omalley@apache.org> wrote:
> All,
> =A0 We've had a wide range of entries for a powered by logo. I've put the=
m all on a page, here:
>
> http://people.apache.org/~omalley/hadoop-powered-by/
>
> Since there are a lot of contenders and we only want a single round of vo=
ting, let's use single transferable vote ( STV http://en.wikipedia.org/wiki=
/Single_transferable_vote). The important thing is to pick the images *IN O=
RDER* that you would like them.
>
> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>
> In other words, I want option 4 most and option 6 least. With STV, you do=
n't need to worry about voting for an unpopular choice since your vote will=
 automatically roll over to your next choice.
>
> -- Owen
>
>
>

From general-return-3721-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 04:28:31 2011
Return-Path: <general-return-3721-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 64E8C4DE4
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 04:28:31 +0000 (UTC)
Received: (qmail 11231 invoked by uid 500); 15 Jun 2011 04:28:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11172 invoked by uid 500); 15 Jun 2011 04:28:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11164 invoked by uid 99); 15 Jun 2011 04:28:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 04:28:28 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 04:28:21 +0000
Received: by bwz8 with SMTP id 8so288086bwz.35
        for <general@hadoop.apache.org>; Tue, 14 Jun 2011 21:28:01 -0700 (PDT)
Received: by 10.204.233.15 with SMTP id jw15mr38038bkb.48.1308112081200; Tue,
 14 Jun 2011 21:28:01 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.79 with HTTP; Tue, 14 Jun 2011 21:27:40 -0700 (PDT)
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
From: Todd Lipcon <todd@cloudera.com>
Date: Tue, 14 Jun 2011 21:27:40 -0700
Message-ID: <BANLkTimNexYAgugzWLwtoOzC0VYMtoLtSQ@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=485b3979db46ed266204a5b8931e
X-Virus-Checked: Checked by ClamAV on apache.org

--485b3979db46ed266204a5b8931e
Content-Type: text/plain; charset=ISO-8859-1

Who is allowed to vote in this? Committers? PMC? Everyone?

My vote: 5, 2, 6, 3, 1, 4

On Tue, Jun 14, 2011 at 8:19 PM, Owen O'Malley <omalley@apache.org> wrote:

> All,
>   We've had a wide range of entries for a powered by logo. I've put them
> all on a page, here:
>
> http://people.apache.org/~omalley/hadoop-powered-by/
>
> Since there are a lot of contenders and we only want a single round of
> voting, let's use single transferable vote ( STV
> http://en.wikipedia.org/wiki/Single_transferable_vote). The important
> thing is to pick the images *IN ORDER* that you would like them.
>
> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>
> In other words, I want option 4 most and option 6 least. With STV, you
> don't need to worry about voting for an unpopular choice since your vote
> will automatically roll over to your next choice.
>
> -- Owen
>
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

--485b3979db46ed266204a5b8931e--

From general-return-3722-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 04:40:57 2011
Return-Path: <general-return-3722-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8F6EC4540
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 04:40:57 +0000 (UTC)
Received: (qmail 19784 invoked by uid 500); 15 Jun 2011 04:40:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19724 invoked by uid 500); 15 Jun 2011 04:40:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19715 invoked by uid 99); 15 Jun 2011 04:40:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 04:40:54 +0000
X-ASF-Spam-Status: No, hits=3.3 required=5.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 69.147.107.21 is neither permitted nor denied by domain of amarsri@yahoo-inc.com)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 04:40:47 +0000
Received: from EGL-EX07CAS03.ds.corp.yahoo.com (egl-ex07cas03.eglbp.corp.yahoo.com [203.83.248.219])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5F4e4TG069198
	for <general@hadoop.apache.org>; Tue, 14 Jun 2011 21:40:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1308112805;
	bh=HcLg0dR4hRES+in8MT5sSitIoHzzDxM9P/s7RDICA80=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 MIME-Version;
	b=fOQIK94XQo00APXjHJ+Ya3PDnXipzWu7ZZuJFqFkgvenez8Y8gIs5GePBXEuCi0Af
	 KJcmFNwQ5WZwfOFhYuRjNUQEDYpdj5QkXsg1SWiaf/Xlrjdi5YvfMCBk4NPt/xfkx3
	 ZhLqcp4jJVGfGwPYsG4w0XbTWfxQ/onWjlXciJRw=
Received: from EGL-EX07VS01.ds.corp.yahoo.com ([203.83.248.205]) by
 EGL-EX07CAS03.ds.corp.yahoo.com ([203.83.248.219]) with mapi; Wed, 15 Jun
 2011 10:10:03 +0530
From: Amareshwari Sri Ramadasu <amarsri@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 15 Jun 2011 10:10:02 +0530
Subject: Re: [VOTE] Powered by Logo
Thread-Topic: [VOTE] Powered by Logo
Thread-Index: AcwrFLk163gjScLrTYmEbeZ55yC1fAAAZDve
Message-ID: <CA1E357A.2E2C5%amarsri@yahoo-inc.com>
In-Reply-To: <BANLkTimNexYAgugzWLwtoOzC0VYMtoLtSQ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_CA1E357A2E2C5amarsriyahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_CA1E357A2E2C5amarsriyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

My vote 4, 5, 2, 3, 1, 6

Thanks
Amareshwari

On 6/15/11 9:57 AM, "Todd Lipcon" <todd@cloudera.com> wrote:

Who is allowed to vote in this? Committers? PMC? Everyone?

My vote: 5, 2, 6, 3, 1, 4

On Tue, Jun 14, 2011 at 8:19 PM, Owen O'Malley <omalley@apache.org> wrote:

> All,
>   We've had a wide range of entries for a powered by logo. I've put them
> all on a page, here:
>
> http://people.apache.org/~omalley/hadoop-powered-by/
>
> Since there are a lot of contenders and we only want a single round of
> voting, let's use single transferable vote ( STV
> http://en.wikipedia.org/wiki/Single_transferable_vote). The important
> thing is to pick the images *IN ORDER* that you would like them.
>
> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>
> In other words, I want option 4 most and option 6 least. With STV, you
> don't need to worry about voting for an unpopular choice since your vote
> will automatically roll over to your next choice.
>
> -- Owen
>
>
>


--
Todd Lipcon
Software Engineer, Cloudera


--_000_CA1E357A2E2C5amarsriyahooinccom_--

From general-return-3723-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 05:08:09 2011
Return-Path: <general-return-3723-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CDDD443E3
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 05:08:09 +0000 (UTC)
Received: (qmail 35383 invoked by uid 500); 15 Jun 2011 05:08:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 35332 invoked by uid 500); 15 Jun 2011 05:08:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 35324 invoked by uid 99); 15 Jun 2011 05:08:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 05:08:07 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 05:07:59 +0000
Received: by qyk2 with SMTP id 2so2175595qyk.14
        for <general@hadoop.apache.org>; Tue, 14 Jun 2011 22:07:38 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=HBotSOGfQ9SjvDf8hvlpkaEt1z+hLKEdFL542yNsivw=;
        b=tDj2oMCI+36t+T3dz+oUVYpFDpZtDg3hAT/3QtvJwkrFeqo/gdvlS9K75VfOmDbXSD
         cJ2KvFDurU0FJi6rVLSUk04orDVOaR+XvjyQXNEfl/Tuz00RxyV7jV+MlHozcuBG4X8b
         BGMbmJpgB/fOQyhr5961V8Fqhtaz8jwIaWw5M=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=MzHmFZitRTpe1hda5PRX/6dyuArN+BoIh8PBI9z1qxTUDn649kBwkBskFgIR633vJb
         L8PSBMjPegMG5NSRNqU6NBcZXsO5xHG2aeY/3GrSCgCKPcczJajqi8wxEnRQNCCB8miZ
         fwRosWMYsrKPZ0sXH9DBoaWcVxxIgwZIeKYNY=
Received: by 10.224.195.201 with SMTP id ed9mr53009qab.20.1308114458397;
        Tue, 14 Jun 2011 22:07:38 -0700 (PDT)
Received: from [10.172.1.74] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id r32sm54733qcs.14.2011.06.14.22.07.36
        (version=TLSv1/SSLv3 cipher=OTHER);
        Tue, 14 Jun 2011 22:07:37 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: [VOTE] Powered by Logo
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
Date: Wed, 15 Jun 2011 15:07:32 +1000
Content-Transfer-Encoding: 7bit
Message-Id: <F219A8B2-FAED-4837-8C42-6DC37E3BC1E4@Holsman.NET>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

5,4,2,3,6,1

On Jun 15, 2011, at 1:19 PM, Owen O'Malley wrote:

> http://people.apache.org/~omalley/hadoop-powered-by/


From general-return-3724-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 06:37:34 2011
Return-Path: <general-return-3724-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1DC0442D4
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 06:37:34 +0000 (UTC)
Received: (qmail 122 invoked by uid 500); 15 Jun 2011 06:37:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99965 invoked by uid 500); 15 Jun 2011 06:37:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 99148 invoked by uid 99); 15 Jun 2011 06:37:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 06:37:25 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of atm@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 06:37:18 +0000
Received: by bwz8 with SMTP id 8so360402bwz.35
        for <general@hadoop.apache.org>; Tue, 14 Jun 2011 23:36:58 -0700 (PDT)
Received: by 10.204.7.156 with SMTP id d28mr142593bkd.28.1308119818267; Tue,
 14 Jun 2011 23:36:58 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.72.195 with HTTP; Tue, 14 Jun 2011 23:36:28 -0700 (PDT)
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
From: "Aaron T. Myers" <atm@cloudera.com>
Date: Tue, 14 Jun 2011 23:36:28 -0700
Message-ID: <BANLkTi=JoDUqY5h5PSyggwRDqxqqhfFMJw@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001517588b5a176ddf04a5ba61ec
X-Virus-Checked: Checked by ClamAV on apache.org

--001517588b5a176ddf04a5ba61ec
Content-Type: text/plain; charset=ISO-8859-1

When casting your vote, please keep in mind that the vast majority of the
time the "Powered By" logo will be displayed in a size much smaller than
what is displayed on Owen's link. Here's a similar page which shows all the
logos at a few different sizes:

http://people.apache.org/~atm/hadoop-powered-by/

(This is the reason I prefer choice #5 to all others. I believe it'll be
more visually distinct from the project logo at small sizes, which was the
point of developing a "Powered By" logo in the first place.)

--
Aaron T. Myers
Software Engineer, Cloudera



On Tue, Jun 14, 2011 at 8:19 PM, Owen O'Malley <omalley@apache.org> wrote:

> All,
>   We've had a wide range of entries for a powered by logo. I've put them
> all on a page, here:
>
> http://people.apache.org/~omalley/hadoop-powered-by/
>
> Since there are a lot of contenders and we only want a single round of
> voting, let's use single transferable vote ( STV
> http://en.wikipedia.org/wiki/Single_transferable_vote). The important
> thing is to pick the images *IN ORDER* that you would like them.
>
> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>
> In other words, I want option 4 most and option 6 least. With STV, you
> don't need to worry about voting for an unpopular choice since your vote
> will automatically roll over to your next choice.
>
> -- Owen
>
>
>

--001517588b5a176ddf04a5ba61ec--

From general-return-3725-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 06:44:17 2011
Return-Path: <general-return-3725-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7D2E3443A
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 06:44:17 +0000 (UTC)
Received: (qmail 5553 invoked by uid 500); 15 Jun 2011 06:44:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5509 invoked by uid 500); 15 Jun 2011 06:44:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5501 invoked by uid 99); 15 Jun 2011 06:44:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 06:44:15 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jithendranath.j@huawei.com designates 119.145.14.67 as permitted sender)
Received: from [119.145.14.67] (HELO szxga04-in.huawei.com) (119.145.14.67)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 06:44:09 +0000
Received: from huawei.com (szxga04-in [172.24.2.12])
 by szxga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug
 8 2006)) with ESMTP id <0LMT00IG6JWV78@szxga04-in.huawei.com> for
 general@hadoop.apache.org; Wed, 15 Jun 2011 14:41:19 +0800 (CST)
Received: from szxeml207-edg.china.huawei.com ([172.24.2.119])
 by szxga04-in.huawei.com
 (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug  8 2006))
 with ESMTP id <0LMT004I2JWVTJ@szxga04-in.huawei.com> for
 general@hadoop.apache.org; Wed, 15 Jun 2011 14:41:19 +0800 (CST)
Received: from SZXEML401-HUB.china.huawei.com (10.82.67.31)
 by szxeml207-edg.china.huawei.com (172.24.2.59) with Microsoft SMTP Server
 (TLS) id 14.1.270.1; Wed, 15 Jun 2011 14:41:17 +0800
Received: from blrnshtipl5nc (10.18.1.35) by SZXEML401-HUB.china.huawei.com
 (10.82.67.31) with Microsoft SMTP Server id 14.1.270.1; Wed,
 15 Jun 2011 14:41:18 +0800
Date: Wed, 15 Jun 2011 12:11:17 +0530
From: Jithendranath <jithendranath.j@huawei.com>
Subject: RE: [VOTE] Powered by Logo
In-reply-to: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
X-Originating-IP: [10.18.1.35]
To: general@hadoop.apache.org
Reply-to: jithendranath.j@huawei.com
Message-id: <DEC5F4CA57D742BD8467E54523CE6CF3@china.huawei.com>
Organization: Htipl
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.3790.4841
X-Mailer: Microsoft Office Outlook 11
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
Thread-index: AcwrCxtTJAsk34pzTn+PMo2w+Z+2dAAHA5Eg
X-Virus-Checked: Checked by ClamAV on apache.org


4,2,6,1,5,3 

:)

-- Joijoide

****************************************************************************
***********
This e-mail and attachments contain confidential information from HUAWEI,
which is intended only for the person or entity whose address is listed
above. Any use of the information contained herein in any way (including,
but not limited to, total or partial disclosure, reproduction, or
dissemination) by persons other than the intended recipient's) is
prohibited. If you receive this e-mail in error, please notify the sender by
phone or email immediately and delete it!

________________________________

-----Original Message-----
From: Owen O'Malley [mailto:omalley@apache.org] 
Sent: Wednesday, June 15, 2011 8:49 AM
To: general@hadoop.apache.org
Subject: [VOTE] Powered by Logo

All,
   We've had a wide range of entries for a powered by logo. I've put them
all on a page, here:

http://people.apache.org/~omalley/hadoop-powered-by/

Since there are a lot of contenders and we only want a single round of
voting, let's use single transferable vote ( STV
http://en.wikipedia.org/wiki/Single_transferable_vote). The important thing
is to pick the images *IN ORDER* that you would like them.

My vote (in order of course): 4, 1, 2, 3, 5, 6.

In other words, I want option 4 most and option 6 least. With STV, you don't
need to worry about voting for an unpopular choice since your vote will
automatically roll over to your next choice.

-- Owen




From general-return-3726-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 06:46:45 2011
Return-Path: <general-return-3726-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 203DA41A0
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 06:46:45 +0000 (UTC)
Received: (qmail 8486 invoked by uid 500); 15 Jun 2011 06:46:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8429 invoked by uid 500); 15 Jun 2011 06:46:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8421 invoked by uid 99); 15 Jun 2011 06:46:43 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 06:46:43 +0000
Received: from localhost (HELO mail-iw0-f176.google.com) (127.0.0.1)
  (smtp-auth username edwardyoon, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 06:46:43 +0000
Received: by iwr19 with SMTP id 19so54897iwr.35
        for <general@hadoop.apache.org>; Tue, 14 Jun 2011 23:46:42 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.197.10 with SMTP id ei10mr119567ibb.176.1308120402634;
 Tue, 14 Jun 2011 23:46:42 -0700 (PDT)
Received: by 10.231.37.201 with HTTP; Tue, 14 Jun 2011 23:46:42 -0700 (PDT)
In-Reply-To: <DEC5F4CA57D742BD8467E54523CE6CF3@china.huawei.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
	<DEC5F4CA57D742BD8467E54523CE6CF3@china.huawei.com>
Date: Wed, 15 Jun 2011 15:46:42 +0900
Message-ID: <BANLkTi=wJR-fhn0snNnFXcG_bmD=AUAVog@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: "Edward J. Yoon" <edwardyoon@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

4,3,5,6,2,1

On Wed, Jun 15, 2011 at 3:41 PM, Jithendranath
<jithendranath.j@huawei.com> wrote:
>
> 4,2,6,1,5,3
>
> :)
>
> -- Joijoide
>
> *************************************************************************=
***
> ***********
> This e-mail and attachments contain confidential information from HUAWEI,
> which is intended only for the person or entity whose address is listed
> above. Any use of the information contained herein in any way (including,
> but not limited to, total or partial disclosure, reproduction, or
> dissemination) by persons other than the intended recipient's) is
> prohibited. If you receive this e-mail in error, please notify the sender=
 by
> phone or email immediately and delete it!
>
> ________________________________
>
> -----Original Message-----
> From: Owen O'Malley [mailto:omalley@apache.org]
> Sent: Wednesday, June 15, 2011 8:49 AM
> To: general@hadoop.apache.org
> Subject: [VOTE] Powered by Logo
>
> All,
> =C2=A0 We've had a wide range of entries for a powered by logo. I've put =
them
> all on a page, here:
>
> http://people.apache.org/~omalley/hadoop-powered-by/
>
> Since there are a lot of contenders and we only want a single round of
> voting, let's use single transferable vote ( STV
> http://en.wikipedia.org/wiki/Single_transferable_vote). The important thi=
ng
> is to pick the images *IN ORDER* that you would like them.
>
> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>
> In other words, I want option 4 most and option 6 least. With STV, you do=
n't
> need to worry about voting for an unpopular choice since your vote will
> automatically roll over to your next choice.
>
> -- Owen
>
>
>
>



--=20
Best Regards, Edward J. Yoon
@eddieyoon

From general-return-3727-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 06:54:57 2011
Return-Path: <general-return-3727-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 122114C6B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 06:54:57 +0000 (UTC)
Received: (qmail 18671 invoked by uid 500); 15 Jun 2011 06:54:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18620 invoked by uid 500); 15 Jun 2011 06:54:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18609 invoked by uid 99); 15 Jun 2011 06:54:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 06:54:54 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of bijeetsingh@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 06:54:48 +0000
Received: by fxm7 with SMTP id 7so292435fxm.35
        for <general@hadoop.apache.org>; Tue, 14 Jun 2011 23:54:28 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=7s5CInTba40hrIbDJ1PHdwNCa4/yPTs7GNaJrriu+6c=;
        b=Q2AS4zr0l88vlPp3SdxVe38mqBas44PXUm8rgJDjoEKkGpPgT94Rlrqn+V9g2rXdwl
         tEcv2EA1akHrRCBRPzqKlLdAZ3SLE/VZNre+DsIKsTY2MHGsN3pu3CtcogZtd8lB8Z0e
         B/op2VsosQ81TBXJVHJSmARr8fsrk3Itl2VMI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=cEl7/9Q4zrNPPG53esreYXt4mLKLt+P6m0TsjcUdFP9JjqJrp4XotfzQ5Bez1m+8a6
         dNCoxJIqmqY+bg1kurG9NNS58T/m9En2fiXU7c8vxWt4mQ9FoNYLqhMDVLVcYvff2dsb
         lEqSljzrLsggMQLH5cZi3iYvUJshi0lbcqJ6U=
MIME-Version: 1.0
Received: by 10.223.30.82 with SMTP id t18mr146496fac.106.1308120868042; Tue,
 14 Jun 2011 23:54:28 -0700 (PDT)
Received: by 10.223.148.8 with HTTP; Tue, 14 Jun 2011 23:54:28 -0700 (PDT)
In-Reply-To: <BANLkTi=wJR-fhn0snNnFXcG_bmD=AUAVog@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
	<DEC5F4CA57D742BD8467E54523CE6CF3@china.huawei.com>
	<BANLkTi=wJR-fhn0snNnFXcG_bmD=AUAVog@mail.gmail.com>
Date: Wed, 15 Jun 2011 12:24:28 +0530
Message-ID: <BANLkTi=zufzSkaVfCEzYVEk2TOB=vA+h8w@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: Bijeet Singh <bijeetsingh@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00151744866ca9b94004a5ba9f58
X-Virus-Checked: Checked by ClamAV on apache.org

--00151744866ca9b94004a5ba9f58
Content-Type: text/plain; charset=ISO-8859-1

5, 4, 6, 2, 3, 1

Cheers,

Bijeet





> > -----Original Message-----
> > From: Owen O'Malley [mailto:omalley@apache.org]
> > Sent: Wednesday, June 15, 2011 8:49 AM
> > To: general@hadoop.apache.org
> > Subject: [VOTE] Powered by Logo
> >
> > All,
> >   We've had a wide range of entries for a powered by logo. I've put them
> > all on a page, here:
> >
> > http://people.apache.org/~omalley/hadoop-powered-by/
> >
> > Since there are a lot of contenders and we only want a single round of
> > voting, let's use single transferable vote ( STV
> > http://en.wikipedia.org/wiki/Single_transferable_vote). The important
> thing
> > is to pick the images *IN ORDER* that you would like them.
> >
> > My vote (in order of course): 4, 1, 2, 3, 5, 6.
> >
> > In other words, I want option 4 most and option 6 least. With STV, you
> don't
> > need to worry about voting for an unpopular choice since your vote will
> > automatically roll over to your next choice.
> >
> > -- Owen
> >
> >
> >
> >
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>

--00151744866ca9b94004a5ba9f58--

From general-return-3728-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 06:58:07 2011
Return-Path: <general-return-3728-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E23B84CEB
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 06:58:06 +0000 (UTC)
Received: (qmail 23960 invoked by uid 500); 15 Jun 2011 06:58:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23899 invoked by uid 500); 15 Jun 2011 06:58:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23891 invoked by uid 99); 15 Jun 2011 06:58:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 06:58:04 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of teamphy6@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 06:57:57 +0000
Received: by bwz8 with SMTP id 8so372292bwz.35
        for <general@hadoop.apache.org>; Tue, 14 Jun 2011 23:57:37 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=hB8sc5CxUhVvQyTr3S+rrUvL7AcasZDxTeqthJi1hdQ=;
        b=nHZXEuHyngwEOjTP2dsficMDDwj3KLYeN4rwPQmqvECk6fUL9WxycwOzjjVFxdv4K9
         2RVbkA4Z2wrn4zacjw36qN7OwcRLaXJMLG8bbSxFrIog/fUHlQ4Xtu5CZTl1t7KpaLkI
         YI934h+L8iB3VztlPJsZTpI1j5J3Nm1+hUefU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=O0crcXKV6c2aG8wTWwhmp67wWH6FhG7f2FpD7/m/NKv8wzawB0TXHw9p8DKSlZ3XRF
         iqpgk8CC5NEQcmBVGfHjjXfX1+VvuGUickGMX9+VQiJWRrmSFN/qfjFc4ilZPj+0kXXZ
         w4rZ4lwNLgC7QsLFB1eRjOLMU6gvkn+QefK+M=
MIME-Version: 1.0
Received: by 10.204.84.144 with SMTP id j16mr130887bkl.180.1308121057138; Tue,
 14 Jun 2011 23:57:37 -0700 (PDT)
Received: by 10.204.20.3 with HTTP; Tue, 14 Jun 2011 23:57:37 -0700 (PDT)
In-Reply-To: <BANLkTi=wJR-fhn0snNnFXcG_bmD=AUAVog@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
	<DEC5F4CA57D742BD8467E54523CE6CF3@china.huawei.com>
	<BANLkTi=wJR-fhn0snNnFXcG_bmD=AUAVog@mail.gmail.com>
Date: Wed, 15 Jun 2011 02:57:37 -0400
Message-ID: <BANLkTinOVG-CcVLL-xr_S9OWne_b4D5WYg@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: Daniel Jue <teamphy6@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

My vote:

5,4,3,2,6,1

IMO, 5 is much more polished than the rest, but I like the whole elephant in 4.

From general-return-3729-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 07:34:08 2011
Return-Path: <general-return-3729-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5C9A244D7
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 07:34:08 +0000 (UTC)
Received: (qmail 52057 invoked by uid 500); 15 Jun 2011 07:34:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51991 invoked by uid 500); 15 Jun 2011 07:34:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51983 invoked by uid 99); 15 Jun 2011 07:34:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 07:34:06 +0000
X-ASF-Spam-Status: No, hits=3.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of akimball83@gmail.com designates 209.85.218.48 as permitted sender)
Received: from [209.85.218.48] (HELO mail-yi0-f48.google.com) (209.85.218.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 07:33:59 +0000
Received: by yic24 with SMTP id 24so109839yic.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 00:33:38 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type;
        bh=EwH1OWVIv7WOTPO2l1l8RaGZzuYyTNAmSu7+ZeIw8VA=;
        b=PaaA38POhmhsiN0bQDgW/pYjoAV8Av/KgUKtiE8huH0spl5esvG+onC1Tt6Ls68C5g
         uSTegtxhn8kCaHu/A8T3swtDkeVIWeuUzsA69Byej+Cf68DJju+/iSTEIUWS+kuvhP6G
         eQkeBZrsvd0gyhxke56LqDy126OOCqef5LcXU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=frAJ+UZdn/+A2+X2mP54ANBViwRlGdvIDubbpOkg302/fAKS0WKb6PuM/AKdzbxaiV
         1NdvGuAG/5JDJNBa9Npz/bFQ3phn3QK8PuSI2uXSIxnqBO/qHZEj8H9XRBFIZRFjP+Li
         n2OrUFi8tKHm9ZMGK1S8jXrZkg77S+I1x5HdA=
Received: by 10.150.132.6 with SMTP id f6mr429460ybd.425.1308123218231; Wed,
 15 Jun 2011 00:33:38 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.150.220.6 with HTTP; Wed, 15 Jun 2011 00:33:18 -0700 (PDT)
In-Reply-To: <BANLkTinOVG-CcVLL-xr_S9OWne_b4D5WYg@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
 <DEC5F4CA57D742BD8467E54523CE6CF3@china.huawei.com> <BANLkTi=wJR-fhn0snNnFXcG_bmD=AUAVog@mail.gmail.com>
 <BANLkTinOVG-CcVLL-xr_S9OWne_b4D5WYg@mail.gmail.com>
From: Aaron Kimball <akimball83@gmail.com>
Date: Wed, 15 Jun 2011 00:33:18 -0700
Message-ID: <BANLkTimrn6oEv4jPXk5itcfEKkkrE3mpGA@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd403a0bec08c04a5bb2b5f
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd403a0bec08c04a5bb2b5f
Content-Type: text/plain; charset=ISO-8859-1

5 4 2 6 3 1
- Aaron

On Tue, Jun 14, 2011 at 11:57 PM, Daniel Jue <teamphy6@gmail.com> wrote:

> My vote:
>
> 5,4,3,2,6,1
>
> IMO, 5 is much more polished than the rest, but I like the whole elephant
> in 4.
>

--000e0cd403a0bec08c04a5bb2b5f--

From general-return-3730-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 08:45:03 2011
Return-Path: <general-return-3730-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 019844379
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 08:45:03 +0000 (UTC)
Received: (qmail 39300 invoked by uid 500); 15 Jun 2011 08:45:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39239 invoked by uid 500); 15 Jun 2011 08:45:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39230 invoked by uid 99); 15 Jun 2011 08:45:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 08:45:01 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.220.176] (HELO mail-vx0-f176.google.com) (209.85.220.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 08:44:54 +0000
Received: by vxa37 with SMTP id 37so167713vxa.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 01:44:33 -0700 (PDT)
Received: by 10.52.115.130 with SMTP id jo2mr378278vdb.43.1308127473064; Wed,
 15 Jun 2011 01:44:33 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.52.111.166 with HTTP; Wed, 15 Jun 2011 01:44:13 -0700 (PDT)
X-Originating-IP: [151.76.103.169]
In-Reply-To: <CA1E357A.2E2C5%amarsri@yahoo-inc.com>
References: <BANLkTimNexYAgugzWLwtoOzC0VYMtoLtSQ@mail.gmail.com> <CA1E357A.2E2C5%amarsri@yahoo-inc.com>
From: Ted Dunning <tdunning@maprtech.com>
Date: Wed, 15 Jun 2011 10:44:13 +0200
Message-ID: <BANLkTimYxUzKASqX1D6haktHdNXWBfY2LA@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec548a95d5a5b7e04a5bc29fe
X-Virus-Checked: Checked by ClamAV on apache.org

--bcaec548a95d5a5b7e04a5bc29fe
Content-Type: text/plain; charset=ISO-8859-1

4, 2, 6

Yes.  That isn't enough votes, but I think that the other logos don't cut
the mustard for various reasons.  5 is a recycled product logo and I don't
think the others make the required visual case.

On Wed, Jun 15, 2011 at 6:40 AM, Amareshwari Sri Ramadasu <
amarsri@yahoo-inc.com> wrote:

> My vote 4, 5, 2, 3, 1, 6
>
> Thanks
> Amareshwari
>
> On 6/15/11 9:57 AM, "Todd Lipcon" <todd@cloudera.com> wrote:
>
> Who is allowed to vote in this? Committers? PMC? Everyone?
>
> My vote: 5, 2, 6, 3, 1, 4
>
> On Tue, Jun 14, 2011 at 8:19 PM, Owen O'Malley <omalley@apache.org> wrote:
>
> > All,
> >   We've had a wide range of entries for a powered by logo. I've put them
> > all on a page, here:
> >
> > http://people.apache.org/~omalley/hadoop-powered-by/
> >
> > Since there are a lot of contenders and we only want a single round of
> > voting, let's use single transferable vote ( STV
> > http://en.wikipedia.org/wiki/Single_transferable_vote). The important
> > thing is to pick the images *IN ORDER* that you would like them.
> >
> > My vote (in order of course): 4, 1, 2, 3, 5, 6.
> >
> > In other words, I want option 4 most and option 6 least. With STV, you
> > don't need to worry about voting for an unpopular choice since your vote
> > will automatically roll over to your next choice.
> >
> > -- Owen
> >
> >
> >
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
>

--bcaec548a95d5a5b7e04a5bc29fe--

From general-return-3731-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 09:27:15 2011
Return-Path: <general-return-3731-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AAB884323
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 09:27:15 +0000 (UTC)
Received: (qmail 91021 invoked by uid 500); 15 Jun 2011 09:27:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90935 invoked by uid 500); 15 Jun 2011 09:27:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90926 invoked by uid 99); 15 Jun 2011 09:27:14 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 09:27:14 +0000
Received: from localhost (HELO [10.65.8.159]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 09:27:13 +0000
Message-ID: <4DF87AEF.6080506@apache.org>
Date: Wed, 15 Jun 2011 11:27:11 +0200
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Powered by Logo
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
X-Enigmail-Version: 1.1.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

5, 2, 4, 6, 1, 3

Doug

On 06/15/2011 05:19 AM, Owen O'Malley wrote:
> All,
>    We've had a wide range of entries for a powered by logo. I've put them all on a page, here:
> 
> http://people.apache.org/~omalley/hadoop-powered-by/
> 
> Since there are a lot of contenders and we only want a single round of voting, let's use single transferable vote ( STV http://en.wikipedia.org/wiki/Single_transferable_vote). The important thing is to pick the images *IN ORDER* that you would like them.
> 
> My vote (in order of course): 4, 1, 2, 3, 5, 6.
> 
> In other words, I want option 4 most and option 6 least. With STV, you don't need to worry about voting for an unpopular choice since your vote will automatically roll over to your next choice.
> 
> -- Owen
> 
> 
> 

From general-return-3732-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 09:50:23 2011
Return-Path: <general-return-3732-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DEDBE4684
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 09:50:23 +0000 (UTC)
Received: (qmail 26066 invoked by uid 500); 15 Jun 2011 09:50:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26000 invoked by uid 500); 15 Jun 2011 09:50:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25992 invoked by uid 99); 15 Jun 2011 09:50:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 09:50:22 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 09:50:12 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 5AE6E1BA94C
	for <general@hadoop.apache.org>; Wed, 15 Jun 2011 10:49:52 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id 6ToA3cbnAvek for <general@hadoop.apache.org>;
	Wed, 15 Jun 2011 10:49:52 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by colossus.hpl.hp.com (Postfix) with ESMTP id C13D41BA93A
	for <general@hadoop.apache.org>; Wed, 15 Jun 2011 10:49:51 +0100 (BST)
MailScanner-NULL-Check: 1308736177.36536@qgUvT/O9GYhWZbV9uvuNpw
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p5F9naNC029054
	for <general@hadoop.apache.org>; Wed, 15 Jun 2011 10:49:36 +0100 (BST)
Message-ID: <4DF88030.9020600@apache.org>
Date: Wed, 15 Jun 2011 10:49:36 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org> <BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com> <FBF55C01-D66A-44C0-9252-D54E7CAB103D@apache.org>
In-Reply-To: <FBF55C01-D66A-44C0-9252-D54E7CAB103D@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p5F9naNC029054
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 15/06/11 02:15, Allen Wittenauer wrote:
>
>  I run out of fingers if I count how many times just the mapred.map.child.java.opts was said to be "in 20" prior to the 0.20.203 release...]

yeah, that incident involving Camshaft 3.02 beta and your left hand 
really reduced your counting ability.


From general-return-3733-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 09:53:14 2011
Return-Path: <general-return-3733-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 74C644719
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 09:53:14 +0000 (UTC)
Received: (qmail 30426 invoked by uid 500); 15 Jun 2011 09:53:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30366 invoked by uid 500); 15 Jun 2011 09:53:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30358 invoked by uid 99); 15 Jun 2011 09:53:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 09:53:12 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 09:53:03 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 2843EB7DC2
	for <general@hadoop.apache.org>; Wed, 15 Jun 2011 10:52:41 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id FP3xt1l8OSjj for <general@hadoop.apache.org>;
	Wed, 15 Jun 2011 10:52:41 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 643ADB7DBF
	for <general@hadoop.apache.org>; Wed, 15 Jun 2011 10:52:41 +0100 (BST)
MailScanner-NULL-Check: 1308736347.66403@CEDbIim4uLbqoeq2DrU9rg
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p5F9qQPj029148
	for <general@hadoop.apache.org>; Wed, 15 Jun 2011 10:52:27 +0100 (BST)
Message-ID: <4DF880DA.7060105@apache.org>
Date: Wed, 15 Jun 2011 10:52:26 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org> <BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com> <FBF55C01-D66A-44C0-9252-D54E7CAB103D@apache.org> <BANLkTi=BWPUEoqD3L98A9AXLkbafZs15Zw@mail.gmail.com> <92D32E6E-3B65-464F-B91B-4C30EAB2642C@apache.org> <BANLkTi=1JGe8kVortj1Act3jdt9L2RCrhw@mail.gmail.com>
In-Reply-To: <BANLkTi=1JGe8kVortj1Act3jdt9L2RCrhw@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p5F9qQPj029148
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 15/06/11 03:51, Konstantin Boudnik wrote:
> On Tue, Jun 14, 2011 at 19:46, Allen Wittenauer<aw@apache.org>  wrote:
>>
>> On Jun 14, 2011, at 6:45 PM, Eli Collins wrote:
>>> Are we really going to go after all the web companies that patch in an
>>> enhancement to their current Hadoop build and tell them to stop saying
>>> that they are using Hadoop?  You've patched Hadoop many times, should
>>> your employer not be able to say they use Hadoop?  I'm -1 on a
>>> proposal that does this.
>>
>>         I think there is a big difference between some company that uses Hadoop with some patches internally and a company that puts out a distribution for others to use, usually for-profit.
>
> Just as the reminder: this whole conversation has started as a result
> of EMC announcement of 100% compatible version of Apache Hadoop. So,
> Allen's point is right on target here: the above example is simply
> incorrect.

I seem to recall this dicussion starting a bit earlier, with the whole 
notion of compatibility, before EMC got involved.

Regarding the vote, I think the discussion here is interesting and 
should be finalised before the vote. It's worth resolving the issues.

also: banners, stickers and clothing? Can I have T-shirts saying "I 
broke the hadoop build" with the logo on, or should it be "I broke the 
Apache Hadoop build"?

From general-return-3734-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 09:59:15 2011
Return-Path: <general-return-3734-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 04FF24871
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 09:59:15 +0000 (UTC)
Received: (qmail 37999 invoked by uid 500); 15 Jun 2011 09:59:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37957 invoked by uid 500); 15 Jun 2011 09:59:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37949 invoked by uid 99); 15 Jun 2011 09:59:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 09:59:13 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 09:59:03 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 7EA7A1BA355
	for <general@hadoop.apache.org>; Wed, 15 Jun 2011 10:58:43 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id uoxE7qHCXUIq for <general@hadoop.apache.org>;
	Wed, 15 Jun 2011 10:58:42 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 6DDE91BA318
	for <general@hadoop.apache.org>; Wed, 15 Jun 2011 10:58:42 +0100 (BST)
MailScanner-NULL-Check: 1308736708.75957@FQrK4Uf514jLjxYS4B8sHg
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p5F9wRYX029288
	for <general@hadoop.apache.org>; Wed, 15 Jun 2011 10:58:27 +0100 (BST)
Message-ID: <4DF88243.4050307@apache.org>
Date: Wed, 15 Jun 2011 10:58:27 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org> <4F34B183-F0B2-4C42-B132-8D19DBCE9219@apache.org>
In-Reply-To: <4F34B183-F0B2-4C42-B132-8D19DBCE9219@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p5F9wRYX029288
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 15/06/11 00:35, Allen Wittenauer wrote:

> 	A minor nit: I'd like to see some cleanup between the first paragraph and the fourth paragraph in compatibility.  Or was the re-iteration of our "not a standards committee" intentional?  It is sort of awkward as it is currently written.

well it is a wiki...

>
> 	Also, where can I download Camshaft?

It's a fork of Hadoop 0.15 optimised for Windows ME and FAT32 that 
requires a human to fetch blocks from remote machines using a floppy - a 
process that limits blocksize to 1.44MB and kills your latency. You 
don't really want it.

What you saw on the page was marketing's spin on the harsh truth.

From general-return-3735-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 12:04:40 2011
Return-Path: <general-return-3735-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B6D5B4E69
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 12:04:40 +0000 (UTC)
Received: (qmail 98657 invoked by uid 500); 15 Jun 2011 12:04:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98609 invoked by uid 500); 15 Jun 2011 12:04:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98587 invoked by uid 99); 15 Jun 2011 12:04:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 12:04:38 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 12:04:30 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id EF3341BA969
	for <general@hadoop.apache.org>; Wed, 15 Jun 2011 13:04:08 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id 2yqz4JG7uFZq for <general@hadoop.apache.org>;
	Wed, 15 Jun 2011 13:04:08 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 874521BA963
	for <general@hadoop.apache.org>; Wed, 15 Jun 2011 13:04:08 +0100 (BST)
MailScanner-NULL-Check: 1308744234.6353@1aOCQPGbksWRjc4jTEIWJg
Received: from [16.25.175.86] (wildhaus.hpl.hp.com [16.25.175.86])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p5FC3qjR003293
	for <general@hadoop.apache.org>; Wed, 15 Jun 2011 13:03:52 +0100 (BST)
Message-ID: <4DF89FA8.6090902@apache.org>
Date: Wed, 15 Jun 2011 13:03:52 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Powered by Logo
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p5FC3qjR003293
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 15/06/11 04:19, Owen O'Malley wrote:
> All,
>     We've had a wide range of entries for a powered by logo. I've put them all on a page, here:
>
> http://people.apache.org/~omalley/hadoop-powered-by/
>
>
my vote: 4, 5, 2

From general-return-3736-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 12:15:31 2011
Return-Path: <general-return-3736-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 058B24562
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 12:15:31 +0000 (UTC)
Received: (qmail 23495 invoked by uid 500); 15 Jun 2011 12:15:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23450 invoked by uid 500); 15 Jun 2011 12:15:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23442 invoked by uid 99); 15 Jun 2011 12:15:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 12:15:29 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of harsh@cloudera.com designates 209.85.210.176 as permitted sender)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 12:15:22 +0000
Received: by iym1 with SMTP id 1so313013iym.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 05:15:02 -0700 (PDT)
Received: by 10.42.74.72 with SMTP id v8mr433371icj.282.1308140101795; Wed, 15
 Jun 2011 05:15:01 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.42.218.72 with HTTP; Wed, 15 Jun 2011 05:14:40 -0700 (PDT)
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
From: Harsh J <harsh@cloudera.com>
Date: Wed, 15 Jun 2011 17:44:40 +0530
Message-ID: <BANLkTimUiELv8hBwav9JhV7=xH7_O9RMuQ@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

5 6 4 2 3 1

On Wed, Jun 15, 2011 at 8:49 AM, Owen O'Malley <omalley@apache.org> wrote:
> All,
> =A0 We've had a wide range of entries for a powered by logo. I've put the=
m all on a page, here:
>
> http://people.apache.org/~omalley/hadoop-powered-by/
>
> Since there are a lot of contenders and we only want a single round of vo=
ting, let's use single transferable vote ( STV http://en.wikipedia.org/wiki=
/Single_transferable_vote). The important thing is to pick the images *IN O=
RDER* that you would like them.
>
> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>
> In other words, I want option 4 most and option 6 least. With STV, you do=
n't need to worry about voting for an unpopular choice since your vote will=
 automatically roll over to your next choice.
>
> -- Owen
>
>
>



--=20
Harsh J

From general-return-3737-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 12:19:51 2011
Return-Path: <general-return-3737-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AD0EB497F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 12:19:51 +0000 (UTC)
Received: (qmail 34203 invoked by uid 500); 15 Jun 2011 12:19:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34159 invoked by uid 500); 15 Jun 2011 12:19:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34151 invoked by uid 99); 15 Jun 2011 12:19:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 12:19:50 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of dhruba@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 12:19:44 +0000
Received: by fxm7 with SMTP id 7so525748fxm.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 05:19:24 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=/fWwv61/JYqO9mitFTbRuIXUqRn+j3LaKVlrbd4oFCc=;
        b=UqcaStcaKMabkxEShv8At8J8Ic0gRTt+yY7h99IM7MV++Hrqc+0l75+CFfZgf0Kcah
         FqxqzAdOZMDoxZyqAJtVWkk9OzGb0PzCDQIM8HV2vpy9ohdf0j1DhBmx7vPnQ8EPN/Ru
         pqZf+BpEhTZwLmGPppwlBNmnhDERzm3gTLyJ0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=USjuPx/Q20BP1ELEhzCKbumXdFG5kXyJqfhJuGifs4pAtBnsW8mQJfhjy3rsvht8Wc
         gkwqt+IoXoFEm6XlVvBFTPKN80528unpy6LNsdFfwlveqG4fl1jQedL6HorYr8F9hBRV
         MY4G/S+FrbDaCQ8TOGzHyp/9MrJ2u//SwclpU=
MIME-Version: 1.0
Received: by 10.223.4.136 with SMTP id 8mr537580far.16.1308140364226; Wed, 15
 Jun 2011 05:19:24 -0700 (PDT)
Received: by 10.223.113.19 with HTTP; Wed, 15 Jun 2011 05:19:24 -0700 (PDT)
In-Reply-To: <BANLkTimUiELv8hBwav9JhV7=xH7_O9RMuQ@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
	<BANLkTimUiELv8hBwav9JhV7=xH7_O9RMuQ@mail.gmail.com>
Date: Wed, 15 Jun 2011 15:19:24 +0300
Message-ID: <BANLkTi=mn=7qeaenBBecPgOFLqP7SDf8dQ@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0ce00688b9e00504a5bf2990
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0ce00688b9e00504a5bf2990
Content-Type: text/plain; charset=ISO-8859-1

5 only

-dhruba

On Wed, Jun 15, 2011 at 3:14 PM, Harsh J <harsh@cloudera.com> wrote:

> 5 6 4 2 3 1
>
> On Wed, Jun 15, 2011 at 8:49 AM, Owen O'Malley <omalley@apache.org> wrote:
> > All,
> >   We've had a wide range of entries for a powered by logo. I've put them
> all on a page, here:
> >
> > http://people.apache.org/~omalley/hadoop-powered-by/
> >
> > Since there are a lot of contenders and we only want a single round of
> voting, let's use single transferable vote ( STV
> http://en.wikipedia.org/wiki/Single_transferable_vote). The important
> thing is to pick the images *IN ORDER* that you would like them.
> >
> > My vote (in order of course): 4, 1, 2, 3, 5, 6.
> >
> > In other words, I want option 4 most and option 6 least. With STV, you
> don't need to worry about voting for an unpopular choice since your vote
> will automatically roll over to your next choice.
> >
> > -- Owen
> >
> >
> >
>
>
>
> --
> Harsh J
>



-- 
Connect to me at http://www.facebook.com/dhruba

--000e0ce00688b9e00504a5bf2990--

From general-return-3738-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 12:29:53 2011
Return-Path: <general-return-3738-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BF188407C
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 12:29:53 +0000 (UTC)
Received: (qmail 59373 invoked by uid 500); 15 Jun 2011 12:29:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59318 invoked by uid 500); 15 Jun 2011 12:29:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59309 invoked by uid 99); 15 Jun 2011 12:29:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 12:29:52 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of hadoop.li@gmail.com designates 209.85.210.176 as permitted sender)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 12:29:46 +0000
Received: by iym1 with SMTP id 1so328283iym.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 05:29:25 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=KAVFrNFKuKDgLoPJI8Ome3GJ/Nym52pj+UKTKAAHXNM=;
        b=Tol8llfKurKOHswR806Cao3XcHHucWscs9uZQob49IiTpg8Y7id4z6jWnrC0BatRkv
         8d5XuznWRVY8ue7+fAWP4CDCBVCDSyC18ij0o6HDrn2fWcg3GmJ5S6A7e+QCmZedNGqz
         b8iAAvdZuTNxjunqQthYwg91nBr9rncK4b8Ps=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=UKZRWQEEQ7xxrzK0u2B8An6t9y0oN9eVidh3n0CCQRE+OZCvgpB0ok5IXPY3z7+kEr
         vHtErxzmYVHiIToH4EEUX0DzGdEw3HiwGYFfAJNb3oryU1r5rpHDvLrXnA4mewJCBv3P
         UqzatUs21lPaWzYfSnBzmpe3rWtvdxj99YyjA=
MIME-Version: 1.0
Received: by 10.42.163.130 with SMTP id c2mr422203icy.522.1308140964844; Wed,
 15 Jun 2011 05:29:24 -0700 (PDT)
Received: by 10.42.222.65 with HTTP; Wed, 15 Jun 2011 05:29:24 -0700 (PDT)
In-Reply-To: <BANLkTi=mn=7qeaenBBecPgOFLqP7SDf8dQ@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
	<BANLkTimUiELv8hBwav9JhV7=xH7_O9RMuQ@mail.gmail.com>
	<BANLkTi=mn=7qeaenBBecPgOFLqP7SDf8dQ@mail.gmail.com>
Date: Wed, 15 Jun 2011 20:29:24 +0800
Message-ID: <BANLkTim==0S+sJL5AnUP5_J2AERO0=1rvjFTMs2nMKLkpUuCZA@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: Weiwei Li <hadoop.li@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba61395c8693d104a5bf4d00
X-Virus-Checked: Checked by ClamAV on apache.org

--90e6ba61395c8693d104a5bf4d00
Content-Type: text/plain; charset=ISO-8859-1

5 6 2 1 4 3

2011/6/15 Dhruba Borthakur <dhruba@gmail.com>

> 5 only
>
> -dhruba
>
> On Wed, Jun 15, 2011 at 3:14 PM, Harsh J <harsh@cloudera.com> wrote:
>
> > 5 6 4 2 3 1
> >
> > On Wed, Jun 15, 2011 at 8:49 AM, Owen O'Malley <omalley@apache.org>
> wrote:
> > > All,
> > >   We've had a wide range of entries for a powered by logo. I've put
> them
> > all on a page, here:
> > >
> > > http://people.apache.org/~omalley/hadoop-powered-by/
> > >
> > > Since there are a lot of contenders and we only want a single round of
> > voting, let's use single transferable vote ( STV
> > http://en.wikipedia.org/wiki/Single_transferable_vote). The important
> > thing is to pick the images *IN ORDER* that you would like them.
> > >
> > > My vote (in order of course): 4, 1, 2, 3, 5, 6.
> > >
> > > In other words, I want option 4 most and option 6 least. With STV, you
> > don't need to worry about voting for an unpopular choice since your vote
> > will automatically roll over to your next choice.
> > >
> > > -- Owen
> > >
> > >
> > >
> >
> >
> >
> > --
> > Harsh J
> >
>
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
>

--90e6ba61395c8693d104a5bf4d00--

From general-return-3739-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 14:34:56 2011
Return-Path: <general-return-3739-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EA0694330
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 14:34:56 +0000 (UTC)
Received: (qmail 88235 invoked by uid 500); 15 Jun 2011 14:34:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88077 invoked by uid 500); 15 Jun 2011 14:34:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88064 invoked by uid 99); 15 Jun 2011 14:34:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 14:34:55 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of owen.omalley@gmail.com designates 209.85.161.176 as permitted sender)
Received: from [209.85.161.176] (HELO mail-gx0-f176.google.com) (209.85.161.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 14:34:47 +0000
Received: by gxk7 with SMTP id 7so396887gxk.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 07:34:26 -0700 (PDT)
Received: by 10.42.131.71 with SMTP id y7mr579426ics.315.1308148465163;
        Wed, 15 Jun 2011 07:34:25 -0700 (PDT)
Received: from [10.0.0.24] (c-98-234-216-58.hsd1.ca.comcast.net [98.234.216.58])
        by mx.google.com with ESMTPS id hp8sm454495icc.11.2011.06.15.07.34.20
        (version=SSLv3 cipher=OTHER);
        Wed, 15 Jun 2011 07:34:23 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: [VOTE] Powered by Logo
From: Owen O'Malley <omalley@apache.org>
In-Reply-To: <BANLkTimNexYAgugzWLwtoOzC0VYMtoLtSQ@mail.gmail.com>
Date: Wed, 15 Jun 2011 07:34:19 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <F59FA6CC-CE95-4073-920A-08A55037D14D@apache.org>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org> <BANLkTimNexYAgugzWLwtoOzC0VYMtoLtSQ@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)


On Jun 14, 2011, at 9:27 PM, Todd Lipcon wrote:

> Who is allowed to vote in this? Committers? PMC? Everyone?

Sorry, I should have been clearer. Everyone is allowed and encouraged to =
vote. Only the PMC votes are binding, however.=20

-- Owen=20=

From general-return-3740-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 14:36:24 2011
Return-Path: <general-return-3740-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DD9F443A3
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 14:36:24 +0000 (UTC)
Received: (qmail 94457 invoked by uid 500); 15 Jun 2011 14:36:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94396 invoked by uid 500); 15 Jun 2011 14:36:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94385 invoked by uid 99); 15 Jun 2011 14:36:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 14:36:23 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of owen.omalley@gmail.com designates 209.85.210.176 as permitted sender)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 14:36:14 +0000
Received: by iym1 with SMTP id 1so483673iym.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 07:35:53 -0700 (PDT)
Received: by 10.231.66.135 with SMTP id n7mr459108ibi.189.1308148553146;
        Wed, 15 Jun 2011 07:35:53 -0700 (PDT)
Received: from [10.0.0.24] (c-98-234-216-58.hsd1.ca.comcast.net [98.234.216.58])
        by mx.google.com with ESMTPS id d10sm260859ibb.32.2011.06.15.07.35.51
        (version=SSLv3 cipher=OTHER);
        Wed, 15 Jun 2011 07:35:52 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: [VOTE] Powered by Logo
From: Owen O'Malley <omalley@apache.org>
In-Reply-To: <BANLkTinOVG-CcVLL-xr_S9OWne_b4D5WYg@mail.gmail.com>
Date: Wed, 15 Jun 2011 07:35:50 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <AECC1E7D-F1D9-41DB-A2A2-7A5CEA3BB9B6@apache.org>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org> <DEC5F4CA57D742BD8467E54523CE6CF3@china.huawei.com> <BANLkTi=wJR-fhn0snNnFXcG_bmD=AUAVog@mail.gmail.com> <BANLkTinOVG-CcVLL-xr_S9OWne_b4D5WYg@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org


On Jun 14, 2011, at 11:57 PM, Daniel Jue wrote:

> IMO, 5 is much more polished than the rest, but I like the whole =
elephant in 4.

Ted has offered to make a more polished version of 4, if there is =
interest in adopting it.

-- Owen=

From general-return-3741-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 14:46:14 2011
Return-Path: <general-return-3741-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A01B84DF8
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 14:46:14 +0000 (UTC)
Received: (qmail 16123 invoked by uid 500); 15 Jun 2011 14:46:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16061 invoked by uid 500); 15 Jun 2011 14:46:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16053 invoked by uid 99); 15 Jun 2011 14:46:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 14:46:13 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of owen.omalley@gmail.com designates 209.85.210.176 as permitted sender)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 14:46:05 +0000
Received: by iym1 with SMTP id 1so493363iym.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 07:45:45 -0700 (PDT)
Received: by 10.42.75.193 with SMTP id b1mr522521ick.238.1308148880717;
        Wed, 15 Jun 2011 07:41:20 -0700 (PDT)
Received: from [10.0.0.24] (c-98-234-216-58.hsd1.ca.comcast.net [98.234.216.58])
        by mx.google.com with ESMTPS id 4sm264572ibc.25.2011.06.15.07.41.17
        (version=SSLv3 cipher=OTHER);
        Wed, 15 Jun 2011 07:41:19 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: [VOTE] Powered by Logo
From: Owen O'Malley <omalley@apache.org>
In-Reply-To: <BANLkTi=JoDUqY5h5PSyggwRDqxqqhfFMJw@mail.gmail.com>
Date: Wed, 15 Jun 2011 07:41:16 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <B2A77B38-00AD-44CF-9C84-550B391A5AD3@apache.org>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org> <BANLkTi=JoDUqY5h5PSyggwRDqxqqhfFMJw@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)


On Jun 14, 2011, at 11:36 PM, Aaron T. Myers wrote:

> When casting your vote, please keep in mind that the vast majority of =
the
> time the "Powered By" logo will be displayed in a size much smaller =
than
> what is displayed on Owen's link.

That's a really good point Aaron. The fonts in all of the examples are =
probably smaller than they should be to be very readable in the 4th (100 =
pixel width) column.

-- Owen=

From general-return-3742-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 15:15:24 2011
Return-Path: <general-return-3742-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 673CC4F75
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 15:15:24 +0000 (UTC)
Received: (qmail 75831 invoked by uid 500); 15 Jun 2011 15:15:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75782 invoked by uid 500); 15 Jun 2011 15:15:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75774 invoked by uid 99); 15 Jun 2011 15:15:22 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 15:15:22 +0000
Received: from localhost (HELO dhcp-02.private.iobm.com) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 15:15:22 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [VOTE] Powered by Logo
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <BANLkTimYxUzKASqX1D6haktHdNXWBfY2LA@mail.gmail.com>
Date: Wed, 15 Jun 2011 08:15:21 -0700
Content-Transfer-Encoding: 7bit
Message-Id: <1793305B-F27D-485A-BBA6-55156DF72E7F@apache.org>
References: <BANLkTimNexYAgugzWLwtoOzC0VYMtoLtSQ@mail.gmail.com> <CA1E357A.2E2C5%amarsri@yahoo-inc.com> <BANLkTimYxUzKASqX1D6haktHdNXWBfY2LA@mail.gmail.com>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On Jun 15, 2011, at 1:44 AM, Ted Dunning wrote:

> 4, 2, 6
> 
> Yes.  That isn't enough votes, but I think that the other logos don't cut
> the mustard for various reasons.  5 is a recycled product logo and I don't
> think the others make the required visual case.


+1 

4,2,6

From general-return-3743-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 15:21:04 2011
Return-Path: <general-return-3743-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 805A54742
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 15:21:04 +0000 (UTC)
Received: (qmail 86042 invoked by uid 500); 15 Jun 2011 15:21:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85981 invoked by uid 500); 15 Jun 2011 15:21:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85973 invoked by uid 99); 15 Jun 2011 15:21:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 15:21:02 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of devaraj.k@huawei.com designates 119.145.14.67 as permitted sender)
Received: from [119.145.14.67] (HELO szxga04-in.huawei.com) (119.145.14.67)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 15:20:56 +0000
Received: from huawei.com (szxga04-in [172.24.2.12])
 by szxga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug
 8 2006)) with ESMTP id <0LMU00IER7YAAC@szxga04-in.huawei.com> for
 general@hadoop.apache.org; Wed, 15 Jun 2011 23:20:34 +0800 (CST)
Received: from szxeml206-edg.china.huawei.com ([172.24.2.119])
 by szxga04-in.huawei.com
 (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug  8 2006))
 with ESMTP id <0LMU00GQQ7YAT7@szxga04-in.huawei.com> for
 general@hadoop.apache.org; Wed, 15 Jun 2011 23:20:34 +0800 (CST)
Received: from SZXEML401-HUB.china.huawei.com (10.82.67.31)
 by szxeml206-edg.china.huawei.com (172.24.2.58) with Microsoft SMTP Server
 (TLS) id 14.1.270.1; Wed, 15 Jun 2011 23:20:32 +0800
Received: from blrnshtipl5nc (10.18.1.35) by SZXEML401-HUB.china.huawei.com
 (10.82.67.31) with Microsoft SMTP Server id 14.1.270.1; Wed,
 15 Jun 2011 23:20:33 +0800
Date: Wed, 15 Jun 2011 20:50:31 +0530
From: Devaraj K <devaraj.k@huawei.com>
Subject: RE: [VOTE] Powered by Logo
In-reply-to: <F59FA6CC-CE95-4073-920A-08A55037D14D@apache.org>
X-Originating-IP: [10.18.1.35]
To: general@hadoop.apache.org
Reply-to: devaraj.k@huawei.com
Message-id: <B27A273AD2164777881B00C1D393159D@china.huawei.com>
Organization: Htipl
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.3790.4841
X-Mailer: Microsoft Office Outlook 11
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
Thread-index: AcwraWhmYdZo7qUIRQSFPHGKWevmrgABYIYg


4,5,1,2,6,3



Devaraj K 

-----Original Message-----
From: Owen O'Malley [mailto:omalley@apache.org] 
Sent: Wednesday, June 15, 2011 8:04 PM
To: general@hadoop.apache.org
Subject: Re: [VOTE] Powered by Logo


On Jun 14, 2011, at 9:27 PM, Todd Lipcon wrote:

> Who is allowed to vote in this? Committers? PMC? Everyone?

Sorry, I should have been clearer. Everyone is allowed and encouraged to
vote. Only the PMC votes are binding, however. 

-- Owen =


From general-return-3744-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 15:24:12 2011
Return-Path: <general-return-3744-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 44785479C
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 15:24:12 +0000 (UTC)
Received: (qmail 89495 invoked by uid 500); 15 Jun 2011 15:24:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89436 invoked by uid 500); 15 Jun 2011 15:24:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89428 invoked by uid 99); 15 Jun 2011 15:24:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 15:24:10 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mlortiz@uci.cu designates 200.55.140.180 as permitted sender)
Received: from [200.55.140.180] (HELO mx3.uci.cu) (200.55.140.180)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 15 Jun 2011 15:24:06 +0000
Received: (qmail 12020 invoked by uid 507); 15 Jun 2011 15:23:38 -0000
Received: from 10.0.0.183 by ns3.uci.cu (envelope-from <mlortiz@uci.cu>, uid 501) with qmail-scanner-2.01st 
 (avp: 5.0.2.0. spamassassin: 3.0.6. perlscan: 2.01st.  
 Clear:RC:1(10.0.0.183):. 
 Processed in 0.650014 secs); 15 Jun 2011 15:23:38 -0000
Received: from unknown (HELO ucimail2.uci.cu) (10.0.0.183)
  by 0 with SMTP; 15 Jun 2011 15:23:37 -0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by ucimail2.uci.cu (Postfix) with ESMTP id 7F3FA3DC573;
	Wed, 15 Jun 2011 11:25:25 -0400 (CDT)
X-Virus-Scanned: amavisd-new at uci.cu
Received: from ucimail2.uci.cu ([127.0.0.1])
	by localhost (ucimail2.uci.cu [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id icb+dNQdXUU9; Wed, 15 Jun 2011 11:25:24 -0400 (CDT)
Received: from [10.36.18.57] (unknown [10.36.18.57])
	by ucimail2.uci.cu (Postfix) with ESMTP id A2FC13DC4BE;
	Wed, 15 Jun 2011 11:25:24 -0400 (CDT)
Message-ID: <4DF8D5F4.8090405@uci.cu>
Date: Wed, 15 Jun 2011 11:25:32 -0430
From: Marcos Ortiz <mlortiz@uci.cu>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110424 Lightning/1.0b2 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
CC: Harsh J <harsh@cloudera.com>
Subject: Re: [VOTE] Powered by Logo
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org> <BANLkTimUiELv8hBwav9JhV7=xH7_O9RMuQ@mail.gmail.com>
In-Reply-To: <BANLkTimUiELv8hBwav9JhV7=xH7_O9RMuQ@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit

On 06/15/2011 07:44 AM, Harsh J wrote:
> 5 6 4 2 3 1
My vote
5, 4, 6, 3, 1, 2

-- 

Marcos Luís Ortíz Valmaseda
  Software Engineer (UCI)
  http://marcosluis2186.posterous.com
  http://twitter.com/marcosluis2186



From general-return-3745-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 15:29:57 2011
Return-Path: <general-return-3745-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DDF4A4A0C
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 15:29:56 +0000 (UTC)
Received: (qmail 2641 invoked by uid 500); 15 Jun 2011 15:29:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2578 invoked by uid 500); 15 Jun 2011 15:29:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2570 invoked by uid 99); 15 Jun 2011 15:29:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 15:29:55 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 69.147.107.20 is neither permitted nor denied by domain of tgraves@yahoo-inc.com)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 15:29:49 +0000
Received: from sp1-ex07cas01.ds.corp.yahoo.com (sp1-ex07cas01.ds.corp.yahoo.com [216.252.116.137])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5FFT9EA099163
	for <general@hadoop.apache.org>; Wed, 15 Jun 2011 08:29:10 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1308151750;
	bh=NNxMgL+ePv5LvK/tC6jRUGUVoI04TEU0JdW+1k7fygg=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:MIME-Version;
	b=aK+CpDJYZ3cC+epHkI87DlzIV7/9k73xhgMxoeR4QiEp/I40NoCUm9vxZJZMsi0cz
	 x1qEgYgi5m6xs2+ghZBm7JvKwwqnN+x2w8dpjMGRyRr9/dZuqxR3Y5T8nP3HhBz92Y
	 vLZ4KCR50eQ6Xb9CdCdz/uF+pyfnzI4z9d3xneh0=
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas01.ds.corp.yahoo.com ([216.252.116.137]) with mapi; Wed, 15 Jun
 2011 08:29:10 -0700
From: Thomas Graves <tgraves@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 15 Jun 2011 08:29:06 -0700
Subject: Re: [VOTE] Powered by Logo
Thread-Topic: [VOTE] Powered by Logo
Thread-Index: AcwrCyJZLQPXI5UcQ/KYyVlvXAmo7QAZdQ6N
Message-ID: <CA1E39F2.BDEF%tgraves@yahoo-inc.com>
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-Entourage/13.9.0.110114
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

My vote: 4,5,1,2,3,6

Tom


On 6/14/11 10:19 PM, "Owen O'Malley" <omalley@apache.org> wrote:

> All,
>    We've had a wide range of entries for a powered by logo. I've put them=
 all
> on a page, here:
>=20
> http://people.apache.org/~omalley/hadoop-powered-by/
>=20
> Since there are a lot of contenders and we only want a single round of vo=
ting,
> let's use single transferable vote ( STV
> http://en.wikipedia.org/wiki/Single_transferable_vote). The important thi=
ng is
> to pick the images *IN ORDER* that you would like them.
>=20
> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>=20
> In other words, I want option 4 most and option 6 least. With STV, you do=
n't
> need to worry about voting for an unpopular choice since your vote will
> automatically roll over to your next choice.
>=20
> -- Owen
>=20
>=20


From general-return-3746-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 15:53:30 2011
Return-Path: <general-return-3746-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BEDDF4E1C
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 15:53:30 +0000 (UTC)
Received: (qmail 69873 invoked by uid 500); 15 Jun 2011 15:53:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69745 invoked by uid 500); 15 Jun 2011 15:53:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69737 invoked by uid 99); 15 Jun 2011 15:53:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 15:53:28 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [63.245.208.150] (HELO dm-mail01.mozilla.org) (63.245.208.150)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 15:53:22 +0000
X-Virus-Scanned: amavisd-new at mozilla.org
Received: from joubert.local (v74-nslb.mozilla.org [10.2.74.4])
	(Authenticated sender: xstevens@mozilla.com)
	by dm-mail01.mozilla.org (Postfix) with ESMTP id 546C1B803B
	for <general@hadoop.apache.org>; Wed, 15 Jun 2011 08:53:01 -0700 (PDT)
Message-ID: <4DF8D55D.80705@mozilla.com>
Date: Wed, 15 Jun 2011 08:53:01 -0700
From: Xavier Stevens <xstevens@mozilla.com>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:5.0) Gecko/20110528 Thunderbird/5.0b1
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Powered by Logo
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org> <BANLkTimUiELv8hBwav9JhV7=xH7_O9RMuQ@mail.gmail.com> <4DF8D5F4.8090405@uci.cu>
In-Reply-To: <4DF8D5F4.8090405@uci.cu>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

My vote: 5, 4, 2, 6, 3, 1


On 6/15/11 8:55 AM, Marcos Ortiz wrote:
> On 06/15/2011 07:44 AM, Harsh J wrote:
>> 5 6 4 2 3 1
> My vote
> 5, 4, 6, 3, 1, 2
>

From general-return-3747-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 15:55:22 2011
Return-Path: <general-return-3747-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E4BCA4EF9
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 15:55:22 +0000 (UTC)
Received: (qmail 77446 invoked by uid 500); 15 Jun 2011 15:55:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77386 invoked by uid 500); 15 Jun 2011 15:55:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77378 invoked by uid 99); 15 Jun 2011 15:55:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 15:55:21 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of eli@cloudera.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 15:55:16 +0000
Received: by wyb40 with SMTP id 40so603286wyb.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 08:54:55 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.217.7.3 with SMTP id z3mr733071wes.68.1308153295247; Wed, 15
 Jun 2011 08:54:55 -0700 (PDT)
Received: by 10.216.30.212 with HTTP; Wed, 15 Jun 2011 08:54:55 -0700 (PDT)
In-Reply-To: <CA1E39F2.BDEF%tgraves@yahoo-inc.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
	<CA1E39F2.BDEF%tgraves@yahoo-inc.com>
Date: Wed, 15 Jun 2011 08:54:55 -0700
Message-ID: <BANLkTikZxGvEMGANiN+VQ6T_dUoQAeGqLA@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

My vote 5 3 6 1 2 4

On Wed, Jun 15, 2011 at 8:29 AM, Thomas Graves <tgraves@yahoo-inc.com> wrot=
e:
> My vote: 4,5,1,2,3,6
>
> Tom
>
>
> On 6/14/11 10:19 PM, "Owen O'Malley" <omalley@apache.org> wrote:
>
>> All,
>> =A0 =A0We've had a wide range of entries for a powered by logo. I've put=
 them all
>> on a page, here:
>>
>> http://people.apache.org/~omalley/hadoop-powered-by/
>>
>> Since there are a lot of contenders and we only want a single round of v=
oting,
>> let's use single transferable vote ( STV
>> http://en.wikipedia.org/wiki/Single_transferable_vote). The important th=
ing is
>> to pick the images *IN ORDER* that you would like them.
>>
>> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>>
>> In other words, I want option 4 most and option 6 least. With STV, you d=
on't
>> need to worry about voting for an unpopular choice since your vote will
>> automatically roll over to your next choice.
>>
>> -- Owen
>>
>>
>
>

From general-return-3748-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 15:58:15 2011
Return-Path: <general-return-3748-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F3F174011
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 15:58:14 +0000 (UTC)
Received: (qmail 88038 invoked by uid 500); 15 Jun 2011 15:58:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87972 invoked by uid 500); 15 Jun 2011 15:58:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87964 invoked by uid 99); 15 Jun 2011 15:58:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 15:58:13 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of henry@cloudera.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 15:58:07 +0000
Received: by iwr19 with SMTP id 19so598227iwr.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 08:57:46 -0700 (PDT)
Received: by 10.231.61.83 with SMTP id s19mr587377ibh.19.1308153466072; Wed,
 15 Jun 2011 08:57:46 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.42.243.130 with HTTP; Wed, 15 Jun 2011 08:57:26 -0700 (PDT)
In-Reply-To: <BANLkTikZxGvEMGANiN+VQ6T_dUoQAeGqLA@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
 <CA1E39F2.BDEF%tgraves@yahoo-inc.com> <BANLkTikZxGvEMGANiN+VQ6T_dUoQAeGqLA@mail.gmail.com>
From: Henry Robinson <henry@cloudera.com>
Date: Wed, 15 Jun 2011 08:57:26 -0700
Message-ID: <BANLkTimc4JDRHodNKSb-VGQF0OYmJOHYsg@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001517740b62a8313504a5c23604
X-Virus-Checked: Checked by ClamAV on apache.org

--001517740b62a8313504a5c23604
Content-Type: text/plain; charset=ISO-8859-1

5 2 1 3 6 4

On 15 June 2011 08:54, Eli Collins <eli@cloudera.com> wrote:

> My vote 5 3 6 1 2 4
>
> On Wed, Jun 15, 2011 at 8:29 AM, Thomas Graves <tgraves@yahoo-inc.com>
> wrote:
> > My vote: 4,5,1,2,3,6
> >
> > Tom
> >
> >
> > On 6/14/11 10:19 PM, "Owen O'Malley" <omalley@apache.org> wrote:
> >
> >> All,
> >>    We've had a wide range of entries for a powered by logo. I've put
> them all
> >> on a page, here:
> >>
> >> http://people.apache.org/~omalley/hadoop-powered-by/
> >>
> >> Since there are a lot of contenders and we only want a single round of
> voting,
> >> let's use single transferable vote ( STV
> >> http://en.wikipedia.org/wiki/Single_transferable_vote). The important
> thing is
> >> to pick the images *IN ORDER* that you would like them.
> >>
> >> My vote (in order of course): 4, 1, 2, 3, 5, 6.
> >>
> >> In other words, I want option 4 most and option 6 least. With STV, you
> don't
> >> need to worry about voting for an unpopular choice since your vote will
> >> automatically roll over to your next choice.
> >>
> >> -- Owen
> >>
> >>
> >
> >
>



-- 
Henry Robinson
Software Engineer
Cloudera
415-994-6679

--001517740b62a8313504a5c23604--

From general-return-3749-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 15:59:45 2011
Return-Path: <general-return-3749-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 88E8F408E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 15:59:45 +0000 (UTC)
Received: (qmail 92021 invoked by uid 500); 15 Jun 2011 15:59:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91913 invoked by uid 500); 15 Jun 2011 15:59:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91905 invoked by uid 99); 15 Jun 2011 15:59:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 15:59:43 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.218.48] (HELO mail-yi0-f48.google.com) (209.85.218.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 15:59:36 +0000
Received: by yic24 with SMTP id 24so493382yic.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 08:59:15 -0700 (PDT)
Received: by 10.150.47.5 with SMTP id u5mr994825ybu.118.1308153555084; Wed, 15
 Jun 2011 08:59:15 -0700 (PDT)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.150.205.10 with HTTP; Wed, 15 Jun 2011 08:58:55 -0700 (PDT)
In-Reply-To: <4DF880DA.7060105@apache.org>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
 <BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com> <FBF55C01-D66A-44C0-9252-D54E7CAB103D@apache.org>
 <BANLkTi=BWPUEoqD3L98A9AXLkbafZs15Zw@mail.gmail.com> <92D32E6E-3B65-464F-B91B-4C30EAB2642C@apache.org>
 <BANLkTi=1JGe8kVortj1Act3jdt9L2RCrhw@mail.gmail.com> <4DF880DA.7060105@apache.org>
From: Konstantin Boudnik <cos@apache.org>
Date: Wed, 15 Jun 2011 08:58:55 -0700
X-Google-Sender-Auth: X7jWo89HaxukrKBgtdRRsbp8zI0
Message-ID: <BANLkTim6DrVsb5NpwnvAvQtLqa5hq0eOrg@mail.gmail.com>
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Wed, Jun 15, 2011 at 02:52, Steve Loughran <stevel@apache.org> wrote:
> On 15/06/11 03:51, Konstantin Boudnik wrote:
>>
>> On Tue, Jun 14, 2011 at 19:46, Allen Wittenauer<aw@apache.org> =A0wrote:
>>>
>>> On Jun 14, 2011, at 6:45 PM, Eli Collins wrote:
>>>>
>>>> Are we really going to go after all the web companies that patch in an
>>>> enhancement to their current Hadoop build and tell them to stop saying
>>>> that they are using Hadoop? =A0You've patched Hadoop many times, shoul=
d
>>>> your employer not be able to say they use Hadoop? =A0I'm -1 on a
>>>> proposal that does this.
>>>
>>> =A0 =A0 =A0 =A0I think there is a big difference between some company t=
hat uses
>>> Hadoop with some patches internally and a company that puts out a
>>> distribution for others to use, usually for-profit.
>>
>> Just as the reminder: this whole conversation has started as a result
>> of EMC announcement of 100% compatible version of Apache Hadoop. So,
>> Allen's point is right on target here: the above example is simply
>> incorrect.
>
> I seem to recall this dicussion starting a bit earlier, with the whole
> notion of compatibility, before EMC got involved.
>
> Regarding the vote, I think the discussion here is interesting and should=
 be
> finalised before the vote. It's worth resolving the issues.
>
> also: banners, stickers and clothing? Can I have T-shirts saying "I broke
> the hadoop build" with the logo on, or should it be "I broke the Apache
> Hadoop build"?

I think such a T-shirt should be forcefully worn on any person who did
just that.

From general-return-3750-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 16:04:19 2011
Return-Path: <general-return-3750-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 57ED6478F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 16:04:19 +0000 (UTC)
Received: (qmail 4409 invoked by uid 500); 15 Jun 2011 16:04:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4347 invoked by uid 500); 15 Jun 2011 16:04:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4339 invoked by uid 99); 15 Jun 2011 16:04:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 16:04:17 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 16:04:11 +0000
Received: by wwi18 with SMTP id 18so640815wwi.29
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 09:03:51 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.220.213 with SMTP id o63mr2292851wep.98.1308153831501;
 Wed, 15 Jun 2011 09:03:51 -0700 (PDT)
Received: by 10.216.30.212 with HTTP; Wed, 15 Jun 2011 09:03:51 -0700 (PDT)
In-Reply-To: <BANLkTimYxUzKASqX1D6haktHdNXWBfY2LA@mail.gmail.com>
References: <BANLkTimNexYAgugzWLwtoOzC0VYMtoLtSQ@mail.gmail.com>
	<CA1E357A.2E2C5%amarsri@yahoo-inc.com>
	<BANLkTimYxUzKASqX1D6haktHdNXWBfY2LA@mail.gmail.com>
Date: Wed, 15 Jun 2011 09:03:51 -0700
Message-ID: <BANLkTinCQHpGpFkdOOsyvh8o+fhjUZuF4A@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Wed, Jun 15, 2011 at 1:44 AM, Ted Dunning <tdunning@maprtech.com> wrote:
> 4, 2, 6
>
> Yes. =A0That isn't enough votes, but I think that the other logos don't c=
ut
> the mustard for various reasons. =A05 is a recycled product logo and I do=
n't
> think the others make the required visual case.

5 is not a product logo, nor was it. Aaron Nuton contributed this and
the Hive logo (which was subsequently adopted) to Apache.
See MAPREDUCE-484 and
http://www.cloudera.com/blog/2009/04/hive-and-jobtracker-needed-logos.
This logo was used as an icon in Hue (Hadoop user interface) which is
an Apache licensed open source project.

Thanks,
Eli

From general-return-3751-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 16:04:56 2011
Return-Path: <general-return-3751-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D44134942
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 16:04:56 +0000 (UTC)
Received: (qmail 7053 invoked by uid 500); 15 Jun 2011 16:04:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6923 invoked by uid 500); 15 Jun 2011 16:04:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6914 invoked by uid 99); 15 Jun 2011 16:04:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 16:04:55 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [98.138.90.72] (HELO nm9.bullet.mail.ne1.yahoo.com) (98.138.90.72)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 15 Jun 2011 16:04:47 +0000
Received: from [98.138.90.54] by nm9.bullet.mail.ne1.yahoo.com with NNFMP; 15 Jun 2011 16:04:25 -0000
Received: from [98.138.88.234] by tm7.bullet.mail.ne1.yahoo.com with NNFMP; 15 Jun 2011 16:04:25 -0000
Received: from [127.0.0.1] by omp1034.mail.ne1.yahoo.com with NNFMP; 15 Jun 2011 16:04:19 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 674980.3147.bm@omp1034.mail.ne1.yahoo.com
Received: (qmail 47160 invoked by uid 60001); 15 Jun 2011 16:04:19 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1308153859; bh=SrsjQV1FPdUl+tPmCv4gxVdPcT/91T8aTxTJ5Tm4+iw=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=u6YEViWMMmw8WiTY7ZTsPzuLyTDb6Uf4KG1JsdZuE/0BXqx2thfjayAH132YJ3MrLFpjuHC4wqQncRZKa5oDIAN5JfZ79NPekvzhcOJvpNjy2jI06S+vGPbdsGyj/XDMn4aGn1RdjpD7eDEj07qO7vozGsXinvy3GHglUEgDExY=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=5t0C7p3u8gzhQigtXpXsfyrmWatFzOQ/I+jl66zcrlseiGqkKWcJQK6+3xslGsHJxFIPoae/NMZAEaN7neZ2VhjPTGCjS8c9YTREXNTSwI0tc83RMMu74PpVRIisUzQxcBhw6nlxyaWLywK3pyPNZzeRr2bJrYq7Ugyw/tKrsnA=;
Message-ID: <548612.46727.qm@web125905.mail.ne1.yahoo.com>
X-YMail-OSG: IWyme9IVM1m7D7hFXL_VZCotTqytBGRiphw7BVaFrjJHcT.
 3rrdwGhjTvZ94HFLtklRh6eRC4E3VmwEQ5LUdLIrfC1_rXa8TGcygXKTQi5j
 3nnM460GgOXGb4h_x7HnUQtuIICPp8cWSimMYfcIqrqWIfVHwi9muQGmKbaw
 PhqkCIdWwXxrGyDUUi7v1OEWYJxCaqLqL.QoNg87rofXfNUy6RjHcaU6Gl5s
 8EVIFroIKbZCN2.uxT6u25SQLaO9LwPf3T4PmoyGQ2eGgeQWi6iDoe3Gr6W0
 Ocv7kgX8HP51Ii4W8ig1TQJZHk48QjLIapQPjztzp.n9bPZgBNRkI5pmrnOQ
 4J8Vvkka4_uPrVeycvB.caSr3OiMyOWM6E4hW5t1bD.KTxAwE6puLs94Ib7z
 7IVs5HhXejyRpPhrTDzv9._nspN2k7b4OrsVfHo9eislRwH8CyAQnjjbSq2J
 ejqROdwmbJKRXcCZ7cKyjbyN85Unq.vt1sCzMWWTh.iDJNN4NXcWD22Z_.qZ
 S
Received: from [98.138.83.207] by web125905.mail.ne1.yahoo.com via HTTP; Wed, 15 Jun 2011 09:04:19 PDT
X-Mailer: YahooMailRC/570 YahooMailWebService/0.8.111.304355
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
Date: Wed, 15 Jun 2011 09:04:19 -0700 (PDT)
From: lohit <lohit_bv@yahoo.com>
Subject: Re: [VOTE] Powered by Logo
To: general@hadoop.apache.org
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii

4,5,6,3,2,1

PS: Powered logo would be advertised by users. Their vote should count :)

----- Original Message ----
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
Sent: Tue, June 14, 2011 8:19:18 PM
Subject: [VOTE] Powered by Logo

All,
   We've had a wide range of entries for a powered by logo. I've put them all on 
a page, here:

http://people.apache.org/~omalley/hadoop-powered-by/

Since there are a lot of contenders and we only want a single round of voting, 
let's use single transferable vote ( STV 
http://en.wikipedia.org/wiki/Single_transferable_vote). The important thing is 
to pick the images *IN ORDER* that you would like them.

My vote (in order of course): 4, 1, 2, 3, 5, 6.

In other words, I want option 4 most and option 6 least. With STV, you don't 
need to worry about voting for an unpopular choice since your vote will 
automatically roll over to your next choice.

-- Owen

From general-return-3752-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 16:20:38 2011
Return-Path: <general-return-3752-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 745D84984
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 16:20:38 +0000 (UTC)
Received: (qmail 46240 invoked by uid 500); 15 Jun 2011 16:20:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46182 invoked by uid 500); 15 Jun 2011 16:20:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46174 invoked by uid 99); 15 Jun 2011 16:20:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 16:20:36 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.161.176] (HELO mail-gx0-f176.google.com) (209.85.161.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 16:20:31 +0000
Received: by gxk7 with SMTP id 7so511079gxk.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 09:20:10 -0700 (PDT)
Received: by 10.151.67.41 with SMTP id u41mr1056067ybk.403.1308154810266; Wed,
 15 Jun 2011 09:20:10 -0700 (PDT)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.150.205.10 with HTTP; Wed, 15 Jun 2011 09:19:50 -0700 (PDT)
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
From: Konstantin Boudnik <cos@apache.org>
Date: Wed, 15 Jun 2011 09:19:50 -0700
X-Google-Sender-Auth: KZtkRzURIAEGio5aEy7ioD9vjMw
Message-ID: <BANLkTikWpmBwCLiKgc0yhe=6nxHbjBrWhw@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I have put the votes together in a form of a spreadsheet
  http://tinyurl.com/3j6629h
for the convenience of watching the race.

Statistics fascinates me usually. Stay tuned.
--
=A0 Take care,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616 =A06115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.



On Tue, Jun 14, 2011 at 20:19, Owen O'Malley <omalley@apache.org> wrote:
> All,
> =A0 We've had a wide range of entries for a powered by logo. I've put the=
m all on a page, here:
>
> http://people.apache.org/~omalley/hadoop-powered-by/
>
> Since there are a lot of contenders and we only want a single round of vo=
ting, let's use single transferable vote ( STV http://en.wikipedia.org/wiki=
/Single_transferable_vote). The important thing is to pick the images *IN O=
RDER* that you would like them.
>
> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>
> In other words, I want option 4 most and option 6 least. With STV, you do=
n't need to worry about voting for an unpopular choice since your vote will=
 automatically roll over to your next choice.
>
> -- Owen
>
>
>

From general-return-3753-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 16:23:40 2011
Return-Path: <general-return-3753-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CE0A84C66
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 16:23:40 +0000 (UTC)
Received: (qmail 57596 invoked by uid 500); 15 Jun 2011 16:23:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57531 invoked by uid 500); 15 Jun 2011 16:23:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57496 invoked by uid 99); 15 Jun 2011 16:23:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 16:23:38 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 16:23:31 +0000
Received: by wwi18 with SMTP id 18so662124wwi.29
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 09:23:11 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.46.21 with SMTP id q21mr2283929web.113.1308154990822; Wed,
 15 Jun 2011 09:23:10 -0700 (PDT)
Received: by 10.216.30.212 with HTTP; Wed, 15 Jun 2011 09:23:10 -0700 (PDT)
In-Reply-To: <92D32E6E-3B65-464F-B91B-4C30EAB2642C@apache.org>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
	<BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com>
	<FBF55C01-D66A-44C0-9252-D54E7CAB103D@apache.org>
	<BANLkTi=BWPUEoqD3L98A9AXLkbafZs15Zw@mail.gmail.com>
	<92D32E6E-3B65-464F-B91B-4C30EAB2642C@apache.org>
Date: Wed, 15 Jun 2011 09:23:10 -0700
Message-ID: <BANLkTi=Gj-XvME7EyHYqPQaUwGNJusr17g@mail.gmail.com>
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Cc: Apache Brand Management <trademarks@apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Tue, Jun 14, 2011 at 7:46 PM, Allen Wittenauer <aw@apache.org> wrote:
>
> On Jun 14, 2011, at 6:45 PM, Eli Collins wrote:
>> Are we really going to go after all the web companies that patch in an
>> enhancement to their current Hadoop build and tell them to stop saying
>> that they are using Hadoop? =A0You've patched Hadoop many times, should
>> your employer not be able to say they use Hadoop? =A0I'm -1 on a
>> proposal that does this.
>
> =A0 =A0 =A0 =A0I think there is a big difference between some company tha=
t uses Hadoop with some patches internally and a company that puts out a di=
stribution for others to use, usually for-profit.

The wiki makes no such distinction. The PMC will apply the rules
equally to all parties.

According to Owen's email if you are using a release of Apache Hadoop
and have applied more than 2 security patches or any backports you are
not using Hadoop.

Thanks,
Eli

From general-return-3754-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 16:39:48 2011
Return-Path: <general-return-3754-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 513FB4AFD
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 16:39:48 +0000 (UTC)
Received: (qmail 626 invoked by uid 500); 15 Jun 2011 16:39:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 514 invoked by uid 500); 15 Jun 2011 16:39:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 506 invoked by uid 99); 15 Jun 2011 16:39:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 16:39:46 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of tom.e.white@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 16:39:40 +0000
Received: by wyb40 with SMTP id 40so657886wyb.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 09:39:20 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type:content-transfer-encoding;
        bh=uhZVFSx/hIevn2AsysL18XnxWH80FnZUs8Sz5LVAy3I=;
        b=eOioNSr2CjrjjaVrXX8FdZbKZ6IKPXr+vj2uqj9Z2R8+kxo+CKcpQi/ftoSb2ToKRr
         aRRE5G8kedlX33aoqB3omd+r4b2Z/zOStx+t25NJrsj3RVEYfMyZTNPMUjakZoSgrKw2
         ZrEfanXrakTFCTlse8NCmL7wU0B2d9LH1K5f0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=dRbsxCW93qYUoqhIFxnFg1jhyCW+jin/Nczvuile2+ewkfZE4oeYIF5PBjjvawF/xS
         iPdj6OYQclkik15SZb7lDXsnwiUcaOAHtWwfeN7eG/4pytssDwq9fagwg0pzkcxlUwvJ
         c4RhPwjOg9q8qTVkkWlJCDSXrSTAlZ6QnYET8=
Received: by 10.216.80.32 with SMTP id j32mr784478wee.91.1308155960098; Wed,
 15 Jun 2011 09:39:20 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.3.21 with HTTP; Wed, 15 Jun 2011 09:39:00 -0700 (PDT)
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
From: Tom White <tom.e.white@gmail.com>
Date: Wed, 15 Jun 2011 09:39:00 -0700
Message-ID: <BANLkTinx8Z16+1nZBVZv_xETX9jfzDXfRA@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

4, 2, 1, 5, 6, 3

Tom

On Tue, Jun 14, 2011 at 8:19 PM, Owen O'Malley <omalley@apache.org> wrote:
> All,
> =A0 We've had a wide range of entries for a powered by logo. I've put the=
m all on a page, here:
>
> http://people.apache.org/~omalley/hadoop-powered-by/
>
> Since there are a lot of contenders and we only want a single round of vo=
ting, let's use single transferable vote ( STV http://en.wikipedia.org/wiki=
/Single_transferable_vote). The important thing is to pick the images *IN O=
RDER* that you would like them.
>
> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>
> In other words, I want option 4 most and option 6 least. With STV, you do=
n't need to worry about voting for an unpopular choice since your vote will=
 automatically roll over to your next choice.
>
> -- Owen
>
>
>

From general-return-3755-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 16:40:49 2011
Return-Path: <general-return-3755-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 53F8A4B94
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 16:40:49 +0000 (UTC)
Received: (qmail 3655 invoked by uid 500); 15 Jun 2011 16:40:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3562 invoked by uid 500); 15 Jun 2011 16:40:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3497 invoked by uid 99); 15 Jun 2011 16:40:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 16:40:47 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of eli@cloudera.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 16:40:40 +0000
Received: by wyb40 with SMTP id 40so659065wyb.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 09:40:19 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.46.21 with SMTP id q21mr2302174web.113.1308156019457; Wed,
 15 Jun 2011 09:40:19 -0700 (PDT)
Received: by 10.216.30.212 with HTTP; Wed, 15 Jun 2011 09:40:19 -0700 (PDT)
In-Reply-To: <090F2993-B9AA-410F-AD99-7AC047FFC974@apache.org>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
	<BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com>
	<090F2993-B9AA-410F-AD99-7AC047FFC974@apache.org>
Date: Wed, 15 Jun 2011 09:40:19 -0700
Message-ID: <BANLkTi=fkguAtbfi9Wk9CPm8JmFa2Hti3Q@mail.gmail.com>
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Cc: trademarks@apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Tue, Jun 14, 2011 at 7:45 PM, Owen O'Malley <omalley@apache.org> wrote:
>
> On Jun 14, 2011, at 5:48 PM, Eli Collins wrote:
>
>> Wrt derivative works, it's not clear from the document, but I think we
>> should explicitly adopt the policy of HTTPD and Subversion that
>> backported patches from trunk and security fixes are permitted.
>
> Actually, the document is extremely clear that only Apache releases may b=
e called Hadoop.
>
> There was a very long thread about why the rapidly expanding Hadoop-ecosy=
stem is leading to at lot of customer confusion about the different "versio=
ns" of Hadoop. We as the Hadoop project don't have the resources or the nec=
essary compatibility test suite to test compatibility between the different=
 sets of cherry picked patches. We also don't have time to ensure that all =
of the 1,000's of patches applied to 0.20.2 in each of the many (10? 15?) d=
ifferent versions have been committed to trunk. Futhermore, under the Apach=
e license, a company Foo could claim that it is a cherry pick version of Ha=
doop without releasing their source code that would enable verification.
>
> In summary,
> =A01. Hadoop is very successful.
> =A02. There are many different commercial products that are trying to use=
 the Hadoop name.
> =A03. We can't check or enforce that the cherry pick versions are followi=
ng the rules.
> =A04. We don't have a TCK like Java does to validate new versions are com=
patible.
> =A05. By far the most fair way to ensure compatibility and fairness betwe=
en companies is that only Apache Hadoop releases may be called Hadoop.
>
> That said, a package that includes a small number (< 3) of security patch=
es that haven't been released yet doesn't seem unreasonable.
>

I've spoken with ops teams at many companies,  I am not aware of
anyone who runs an official release (with just 2 security patches). By
this definition many of the most valuable contributors to Hadoop,
including Yahoo!, Cloudera, Facebook, etc are not using Hadoop.  Is
that really the message we want to send? We expect the PMC to enforce
this equally across all parties?

It's a fact of life that companies and ops teams that support Hadoop
need to patch the software before the PMC has time and/or will to vote
on new releases. This is why HTTP and Subversion allow this. Putting a
build of Hadoop that has 4 security patches applied into the same
category as a product that has entirely re-worked the code and not
gotten it checked into trunk does a major disservice to the people who
contribute to and invest in the project.

Thanks,
Eli

From general-return-3756-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 16:44:30 2011
Return-Path: <general-return-3756-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6825F4D78
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 16:44:30 +0000 (UTC)
Received: (qmail 14958 invoked by uid 500); 15 Jun 2011 16:44:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14910 invoked by uid 500); 15 Jun 2011 16:44:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14902 invoked by uid 99); 15 Jun 2011 16:44:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 16:44:28 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 216.145.54.173 is neither permitted nor denied by domain of jnaisbit@yahoo-inc.com)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 16:44:21 +0000
Received: from SP2-EX07CAS05.ds.corp.yahoo.com (sp2-ex07cas05.corp.sp2.yahoo.com [98.137.59.39])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p5FGhZLp033189
	for <general@hadoop.apache.org>; Wed, 15 Jun 2011 09:43:35 -0700 (PDT)
Received: from SP2-EX07VS04.ds.corp.yahoo.com ([98.137.59.33]) by
 SP2-EX07CAS05.ds.corp.yahoo.com ([98.137.59.39]) with mapi; Wed, 15 Jun 2011
 09:43:35 -0700
From: Jeffrey Naisbitt <jnaisbit@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 15 Jun 2011 09:43:34 -0700
Subject: Re: [VOTE] Powered by Logo
Thread-Topic: [VOTE] Powered by Logo
Thread-Index: AcwrCzFiLf5EThxpTO6I5LsX1VpvSwAcCxRY
Message-ID: <CA1E4B66.7D4B%jnaisbit@yahoo-inc.com>
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
user-agent: Microsoft-Entourage/13.10.0.110428
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

My vote: 4,5,1,2,6,3
--Jeff


On 6/14/11 10:19 PM, "Owen O'Malley" <omalley@apache.org> wrote:

> All,
>    We've had a wide range of entries for a powered by logo. I've put them=
 all
> on a page, here:
>=20
> http://people.apache.org/~omalley/hadoop-powered-by/
>=20
> Since there are a lot of contenders and we only want a single round of vo=
ting,
> let's use single transferable vote ( STV
> http://en.wikipedia.org/wiki/Single_transferable_vote). The important thi=
ng is
> to pick the images *IN ORDER* that you would like them.
>=20
> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>=20
> In other words, I want option 4 most and option 6 least. With STV, you do=
n't
> need to worry about voting for an unpopular choice since your vote will
> automatically roll over to your next choice.
>=20
> -- Owen
>=20
>=20


From general-return-3757-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 16:45:01 2011
Return-Path: <general-return-3757-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A18864DBE
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 16:45:01 +0000 (UTC)
Received: (qmail 16953 invoked by uid 500); 15 Jun 2011 16:44:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16877 invoked by uid 500); 15 Jun 2011 16:44:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16869 invoked by uid 99); 15 Jun 2011 16:44:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 16:44:59 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 16:44:50 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 8B5E1B7E6B
	for <general@hadoop.apache.org>; Wed, 15 Jun 2011 17:44:28 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id Ne9Rg29SIlqs for <general@hadoop.apache.org>;
	Wed, 15 Jun 2011 17:44:28 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 44570B7E65
	for <general@hadoop.apache.org>; Wed, 15 Jun 2011 17:44:27 +0100 (BST)
MailScanner-NULL-Check: 1308761054.138@y3ivNxf5U45N+ZSIWN9sTg
Received: from [16.25.175.86] (wildhaus.hpl.hp.com [16.25.175.86])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p5FGiDqr015469
	for <general@hadoop.apache.org>; Wed, 15 Jun 2011 17:44:13 +0100 (BST)
Message-ID: <4DF8E15C.3030802@apache.org>
Date: Wed, 15 Jun 2011 17:44:12 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>	<BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com>	<FBF55C01-D66A-44C0-9252-D54E7CAB103D@apache.org>	<BANLkTi=BWPUEoqD3L98A9AXLkbafZs15Zw@mail.gmail.com>	<92D32E6E-3B65-464F-B91B-4C30EAB2642C@apache.org> <BANLkTi=Gj-XvME7EyHYqPQaUwGNJusr17g@mail.gmail.com>
In-Reply-To: <BANLkTi=Gj-XvME7EyHYqPQaUwGNJusr17g@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p5FGiDqr015469
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 15/06/11 17:23, Eli Collins wrote:
> On Tue, Jun 14, 2011 at 7:46 PM, Allen Wittenauer<aw@apache.org>  wrote:
>>
>> On Jun 14, 2011, at 6:45 PM, Eli Collins wrote:
>>> Are we really going to go after all the web companies that patch in an
>>> enhancement to their current Hadoop build and tell them to stop saying
>>> that they are using Hadoop?  You've patched Hadoop many times, should
>>> your employer not be able to say they use Hadoop?  I'm -1 on a
>>> proposal that does this.
>>
>>         I think there is a big difference between some company that uses Hadoop with some patches internally and a company that puts out a distribution for others to use, usually for-profit.
>
> The wiki makes no such distinction. The PMC will apply the rules
> equally to all parties.
>
> According to Owen's email if you are using a release of Apache Hadoop
> and have applied more than 2 security patches or any backports you are
> not using Hadoop.
>
> Thanks,
> Eli

What you do in house is of no concern to the trademarks and PMC people, 
but naming of public redistributables is -and that's where the confusion 
of what "a distribution of Apache Hadoop" is, because it's gone from 
weakly defined to very vague recently, and that needs to be corrected 
before people are left in a world of confusion.


It's been complicated enough with people posting issues related to the 
Cloudera Distribution including Apache Hadoop, what happens when people 
start posting EMC-enterprise-hadoopish issues, file bugreps against 
Brisk's "Hadoop built on other things" product on the apache JIRA?

From general-return-3758-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 16:48:31 2011
Return-Path: <general-return-3758-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 197124509
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 16:48:31 +0000 (UTC)
Received: (qmail 23785 invoked by uid 500); 15 Jun 2011 16:48:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23737 invoked by uid 500); 15 Jun 2011 16:48:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23729 invoked by uid 99); 15 Jun 2011 16:48:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 16:48:29 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jeagles@gmail.com designates 74.125.83.176 as permitted sender)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 16:48:24 +0000
Received: by pve37 with SMTP id 37so624673pve.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 09:48:04 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=s7bBDb0yARppO3NUGTwO9q4YnmJ74zgns0J/5O5AFQo=;
        b=xnmbLqc8yEDHghtfuni3aOlHf/ghpZLG3F3SA70CzfSdjJzRO/Ck700Ou9KzWyJusI
         yLPJfWW/umsvLj1Cr+R+4VW2tzMs+pKCcYoQpulgCmpqCy+UboJG2qv4ag2oV45T5515
         CG9UiyJ3QWukDgeXIToawfc4fe26oTjoeDJkI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Yol9YsVg7+kACRsfvyEAwLd3T+j9SYIFND3Vocl+QTsHoLgRNK56ZuT8Ly9XSf+Ilm
         d0YSRT5SVmAE4hHQmUGAZcdh+Dj+o7OV9O0G5aALl1IXvzribc8YahP0zAQ/WDSqs1uK
         iYybJFvnw9CIkyfdWvAQ1xcmHfCh2NONw5a8k=
MIME-Version: 1.0
Received: by 10.142.245.16 with SMTP id s16mr196632wfh.139.1308156483865; Wed,
 15 Jun 2011 09:48:03 -0700 (PDT)
Received: by 10.142.234.4 with HTTP; Wed, 15 Jun 2011 09:48:03 -0700 (PDT)
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
Date: Wed, 15 Jun 2011 11:48:03 -0500
Message-ID: <BANLkTimE8Cx5hkEW2rTZ6F624umq8U=4nQ@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: Jonathan Eagles <jeagles@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd2def2880cb904a5c2ea39

--000e0cd2def2880cb904a5c2ea39
Content-Type: text/plain; charset=ISO-8859-1

My vote: 2, 4, 5, 6, 3, 1.

On Tue, Jun 14, 2011 at 10:19 PM, Owen O'Malley <omalley@apache.org> wrote:

> All,
>   We've had a wide range of entries for a powered by logo. I've put them
> all on a page, here:
>
> http://people.apache.org/~omalley/hadoop-powered-by/
>
> Since there are a lot of contenders and we only want a single round of
> voting, let's use single transferable vote ( STV
> http://en.wikipedia.org/wiki/Single_transferable_vote). The important
> thing is to pick the images *IN ORDER* that you would like them.
>
> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>
> In other words, I want option 4 most and option 6 least. With STV, you
> don't need to worry about voting for an unpopular choice since your vote
> will automatically roll over to your next choice.
>
> -- Owen
>
>
>

--000e0cd2def2880cb904a5c2ea39--

From general-return-3759-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 16:48:36 2011
Return-Path: <general-return-3759-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 91D9C452F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 16:48:36 +0000 (UTC)
Received: (qmail 25262 invoked by uid 500); 15 Jun 2011 16:48:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25195 invoked by uid 500); 15 Jun 2011 16:48:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25186 invoked by uid 99); 15 Jun 2011 16:48:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 16:48:34 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 69.147.107.21 is neither permitted nor denied by domain of anupams@yahoo-inc.com)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 16:48:26 +0000
Received: from SP2-EX07CAS02.ds.corp.yahoo.com (sp2-ex07cas02.corp.sp2.yahoo.com [98.137.59.38])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5FGling094278
	for <general@hadoop.apache.org>; Wed, 15 Jun 2011 09:47:44 -0700 (PDT)
Received: from SP2-EX07VS08.ds.corp.yahoo.com ([98.137.59.27]) by
 SP2-EX07CAS02.ds.corp.yahoo.com ([98.137.59.38]) with mapi; Wed, 15 Jun 2011
 09:47:44 -0700
From: Anupam Seth <anupams@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 15 Jun 2011 09:47:43 -0700
Subject: RE: [VOTE] Powered by Logo
Thread-Topic: [VOTE] Powered by Logo
Thread-Index: AcwrCzFiLf5EThxpTO6I5LsX1VpvSwAcCxRYAAAYPOA=
Message-ID: <E528CD053982584BB880486DAC73E10826BD1C651C@SP2-EX07VS08.ds.corp.yahoo.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
 <CA1E4B66.7D4B%jnaisbit@yahoo-inc.com>
In-Reply-To: <CA1E4B66.7D4B%jnaisbit@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

My vote:

5, 4, 1, 6, 2, 3

-----Original Message-----
From: Jeffrey Naisbitt [mailto:jnaisbit@yahoo-inc.com]=20
Sent: Wednesday, June 15, 2011 11:44 AM
To: general@hadoop.apache.org
Subject: Re: [VOTE] Powered by Logo

My vote: 4,5,1,2,6,3
--Jeff


On 6/14/11 10:19 PM, "Owen O'Malley" <omalley@apache.org> wrote:

> All,
>    We've had a wide range of entries for a powered by logo. I've put them=
 all
> on a page, here:
>=20
> http://people.apache.org/~omalley/hadoop-powered-by/
>=20
> Since there are a lot of contenders and we only want a single round of vo=
ting,
> let's use single transferable vote ( STV
> http://en.wikipedia.org/wiki/Single_transferable_vote). The important thi=
ng is
> to pick the images *IN ORDER* that you would like them.
>=20
> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>=20
> In other words, I want option 4 most and option 6 least. With STV, you do=
n't
> need to worry about voting for an unpopular choice since your vote will
> automatically roll over to your next choice.
>=20
> -- Owen
>=20
>=20


From general-return-3760-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 16:53:51 2011
Return-Path: <general-return-3760-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 842D44713
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 16:53:51 +0000 (UTC)
Received: (qmail 42272 invoked by uid 500); 15 Jun 2011 16:53:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42222 invoked by uid 500); 15 Jun 2011 16:53:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42214 invoked by uid 99); 15 Jun 2011 16:53:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 16:53:49 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of dvryaboy@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 16:53:44 +0000
Received: by wwi18 with SMTP id 18so693961wwi.29
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 09:53:24 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type:content-transfer-encoding;
        bh=+2e75iHGtSruaswMXtm6+QhYi2PgzrgGuaoly+D8k8k=;
        b=XB/1UyXLARksk1GyNj9eV9rw6r4X8ZDSALZKXQrf0z2DHJHtCpwU5zoq3NxPjRWbDZ
         D4YhMutUleIg9JrZQHvk2RISA2n/oMkkoYY5ASSOAsQDHNjD3sFZm+or5LIo0rG6o5tA
         izwvEqIIaVCWI4tL8m0TdbRw3kOn2lBabuqdw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=UCTJs+/jiEAsJ4JRy7EdULslgyOU8DDZdmM/yovM9QcEM3/JCDqpoE49wh+Pac9H5m
         gZ68cZi5efwf/yLq6O3VngTTxOaQjvMBD7DBfstk81KhNXSY2s4WGnZiD4eJR19/aTFD
         ufS2Pedk2xiyU2Dv5FyQDHrIpO0PFk9CxaLN8=
MIME-Version: 1.0
Received: by 10.216.221.22 with SMTP id q22mr838061wep.37.1308156803769; Wed,
 15 Jun 2011 09:53:23 -0700 (PDT)
Received: by 10.216.166.140 with HTTP; Wed, 15 Jun 2011 09:53:23 -0700 (PDT)
In-Reply-To: <E528CD053982584BB880486DAC73E10826BD1C651C@SP2-EX07VS08.ds.corp.yahoo.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
	<CA1E4B66.7D4B%jnaisbit@yahoo-inc.com>
	<E528CD053982584BB880486DAC73E10826BD1C651C@SP2-EX07VS08.ds.corp.yahoo.com>
Date: Wed, 15 Jun 2011 09:53:23 -0700
Message-ID: <BANLkTikw1DJWOePjZ+ayPS7M6_b9vtRSgw@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: Dmitriy Ryaboy <dvryaboy@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

2 6 5 1 4 3

On Wed, Jun 15, 2011 at 9:47 AM, Anupam Seth <anupams@yahoo-inc.com> wrote:
> My vote:
>
> 5, 4, 1, 6, 2, 3
>
> -----Original Message-----
> From: Jeffrey Naisbitt [mailto:jnaisbit@yahoo-inc.com]
> Sent: Wednesday, June 15, 2011 11:44 AM
> To: general@hadoop.apache.org
> Subject: Re: [VOTE] Powered by Logo
>
> My vote: 4,5,1,2,6,3
> --Jeff
>
>
> On 6/14/11 10:19 PM, "Owen O'Malley" <omalley@apache.org> wrote:
>
>> All,
>> =A0 =A0We've had a wide range of entries for a powered by logo. I've put=
 them all
>> on a page, here:
>>
>> http://people.apache.org/~omalley/hadoop-powered-by/
>>
>> Since there are a lot of contenders and we only want a single round of v=
oting,
>> let's use single transferable vote ( STV
>> http://en.wikipedia.org/wiki/Single_transferable_vote). The important th=
ing is
>> to pick the images *IN ORDER* that you would like them.
>>
>> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>>
>> In other words, I want option 4 most and option 6 least. With STV, you d=
on't
>> need to worry about voting for an unpopular choice since your vote will
>> automatically roll over to your next choice.
>>
>> -- Owen
>>
>>
>
>

From general-return-3761-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 16:58:27 2011
Return-Path: <general-return-3761-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EA39848EC
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 16:58:27 +0000 (UTC)
Received: (qmail 56887 invoked by uid 500); 15 Jun 2011 16:58:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56826 invoked by uid 500); 15 Jun 2011 16:58:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56818 invoked by uid 99); 15 Jun 2011 16:58:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 16:58:26 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of eli@cloudera.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 16:58:20 +0000
Received: by wyb40 with SMTP id 40so679690wyb.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 09:57:58 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.159.75 with SMTP id r53mr802095wek.98.1308157078642; Wed,
 15 Jun 2011 09:57:58 -0700 (PDT)
Received: by 10.216.30.212 with HTTP; Wed, 15 Jun 2011 09:57:58 -0700 (PDT)
In-Reply-To: <4DF8E15C.3030802@apache.org>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
	<BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com>
	<FBF55C01-D66A-44C0-9252-D54E7CAB103D@apache.org>
	<BANLkTi=BWPUEoqD3L98A9AXLkbafZs15Zw@mail.gmail.com>
	<92D32E6E-3B65-464F-B91B-4C30EAB2642C@apache.org>
	<BANLkTi=Gj-XvME7EyHYqPQaUwGNJusr17g@mail.gmail.com>
	<4DF8E15C.3030802@apache.org>
Date: Wed, 15 Jun 2011 09:57:58 -0700
Message-ID: <BANLkTi=s61XzH7WssRaBfOb+G8SVZUfVig@mail.gmail.com>
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Wed, Jun 15, 2011 at 9:44 AM, Steve Loughran <stevel@apache.org> wrote:
> On 15/06/11 17:23, Eli Collins wrote:
>>
>> On Tue, Jun 14, 2011 at 7:46 PM, Allen Wittenauer<aw@apache.org> =A0wrot=
e:
>>>
>>> On Jun 14, 2011, at 6:45 PM, Eli Collins wrote:
>>>>
>>>> Are we really going to go after all the web companies that patch in an
>>>> enhancement to their current Hadoop build and tell them to stop saying
>>>> that they are using Hadoop? =A0You've patched Hadoop many times, shoul=
d
>>>> your employer not be able to say they use Hadoop? =A0I'm -1 on a
>>>> proposal that does this.
>>>
>>> =A0 =A0 =A0 =A0I think there is a big difference between some company t=
hat uses
>>> Hadoop with some patches internally and a company that puts out a
>>> distribution for others to use, usually for-profit.
>>
>> The wiki makes no such distinction. The PMC will apply the rules
>> equally to all parties.
>>
>> According to Owen's email if you are using a release of Apache Hadoop
>> and have applied more than 2 security patches or any backports you are
>> not using Hadoop.
>>
>> Thanks,
>> Eli
>
> What you do in house is of no concern to the trademarks and PMC people, b=
ut
> naming of public redistributables is -and that's where the confusion of w=
hat
> "a distribution of Apache Hadoop" is, because it's gone from weakly defin=
ed
> to very vague recently, and that needs to be corrected before people are
> left in a world of confusion.
>

Steve, I'm on the PMC and it is a concern.  What happens in house
often gets released on github, documented, blogged about, etc. All
this stuff creates confusion about the product and is therefore a
concern of the PMC.

>
> It's been complicated enough with people posting issues related to the
> Cloudera Distribution including Apache Hadoop, what happens when people
> start posting EMC-enterprise-hadoopish issues, file bugreps against Brisk=
's
> "Hadoop built on other things" product on the apache JIRA?
>

The same thing we do today. We point them to another more appropriate forum=
.

Isn't your proposal w/ the HTTPD/Subversion policy wrt backporting
effective?  Note that it's pretty strict, you have to get your code
committed to a branch that will be released subject to approval by the
PMC.  It's not saying that anyone can do whatever they want to the
Hadoop source and call it Hadoop.

Thanks,
Eli

From general-return-3762-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 16:59:47 2011
Return-Path: <general-return-3762-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2208E493C
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 16:59:47 +0000 (UTC)
Received: (qmail 59590 invoked by uid 500); 15 Jun 2011 16:59:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59535 invoked by uid 500); 15 Jun 2011 16:59:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59527 invoked by uid 99); 15 Jun 2011 16:59:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 16:59:45 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of atm@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 16:59:38 +0000
Received: by bwz8 with SMTP id 8so1044476bwz.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 09:59:17 -0700 (PDT)
Received: by 10.204.7.156 with SMTP id d28mr749144bkd.28.1308157157505; Wed,
 15 Jun 2011 09:59:17 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.72.195 with HTTP; Wed, 15 Jun 2011 09:58:19 -0700 (PDT)
In-Reply-To: <AECC1E7D-F1D9-41DB-A2A2-7A5CEA3BB9B6@apache.org>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
 <DEC5F4CA57D742BD8467E54523CE6CF3@china.huawei.com> <BANLkTi=wJR-fhn0snNnFXcG_bmD=AUAVog@mail.gmail.com>
 <BANLkTinOVG-CcVLL-xr_S9OWne_b4D5WYg@mail.gmail.com> <AECC1E7D-F1D9-41DB-A2A2-7A5CEA3BB9B6@apache.org>
From: "Aaron T. Myers" <atm@cloudera.com>
Date: Wed, 15 Jun 2011 09:58:19 -0700
Message-ID: <BANLkTi=3A7y2tpazNy1Tf5VeA_gMMm13Jw@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001517588b5aaefbc704a5c3121e
X-Virus-Checked: Checked by ClamAV on apache.org

--001517588b5aaefbc704a5c3121e
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Jun 15, 2011 at 7:35 AM, Owen O'Malley <omalley@apache.org> wrote:

> Ted has offered to make a more polished version of 4, if there is interest
> in adopting it.
>

On that note, is it the case that the outcome of this vote is to adopt the
"style" of the winning option? Or the precise pixels? I assume the former.

I ask because when posting option #5 on the original JIRA, Aaron Newton also
posted a variant optimized for very small treatments:

https://issues.apache.org/jira/secure/attachment/12458850/powered-by-hadoop-small.png

--
Aaron T. Myers
Software Engineer, Cloudera

--001517588b5aaefbc704a5c3121e--

From general-return-3763-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 17:22:38 2011
Return-Path: <general-return-3763-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EDC0F4DAA
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 17:22:37 +0000 (UTC)
Received: (qmail 10225 invoked by uid 500); 15 Jun 2011 17:22:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 10178 invoked by uid 500); 15 Jun 2011 17:22:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 10170 invoked by uid 99); 15 Jun 2011 17:22:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:22:36 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of paliwalashish@gmail.com designates 209.85.220.176 as permitted sender)
Received: from [209.85.220.176] (HELO mail-vx0-f176.google.com) (209.85.220.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:22:30 +0000
Received: by vxa37 with SMTP id 37so742281vxa.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 10:22:09 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=v9m3vWnU9swpFc9f0YcIU5xNYpM4e93EMOEJ9Wkfaa4=;
        b=hexTngFG5raUlnso9QMpYt+lZr6LY0U+dZomhqr5h2lao000zHiRlg6t5Psbq2U/Ts
         roCh51C6PgJP2Z9qs6456TPyODa64ZdY4ldNKU3w4SUJq39f1h2ICT1wKXxPDEtRQBH1
         +1uQqDuElA/jHI04qtmv1Aq2+0MoMfCjqfP0g=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=JWJvk0YrEU0ehsX1xzTHhnm5ztFNZd/pL3aaW/gnQTTUFlgjo6aTW2KCTUTTdS/ssZ
         IfnXu8O/LoYyquqy/cCp52Aypxzkipl6KtIX+ALMJ846dqSk/oSklmwfIerapVllxTLw
         pyLsTiYE4+a5l1l88dufS1s4dSx6D54NhIS9Q=
MIME-Version: 1.0
Received: by 10.52.185.8 with SMTP id ey8mr1185649vdc.68.1308158529569; Wed,
 15 Jun 2011 10:22:09 -0700 (PDT)
Received: by 10.52.106.137 with HTTP; Wed, 15 Jun 2011 10:22:09 -0700 (PDT)
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
Date: Wed, 15 Jun 2011 10:22:09 -0700
Message-ID: <BANLkTi=SCP-iF9Rn5J7kn7=fnyuHkuUKaQ@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: Ashish <paliwalashish@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec54866e277040604a5c3640e
X-Virus-Checked: Checked by ClamAV on apache.org

--bcaec54866e277040604a5c3640e
Content-Type: text/plain; charset=ISO-8859-1

5, 4, 2, 6

On Tue, Jun 14, 2011 at 8:19 PM, Owen O'Malley <omalley@apache.org> wrote:

> All,
>   We've had a wide range of entries for a powered by logo. I've put them
> all on a page, here:
>
> http://people.apache.org/~omalley/hadoop-powered-by/
>
> Since there are a lot of contenders and we only want a single round of
> voting, let's use single transferable vote ( STV
> http://en.wikipedia.org/wiki/Single_transferable_vote). The important
> thing is to pick the images *IN ORDER* that you would like them.
>
> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>
> In other words, I want option 4 most and option 6 least. With STV, you
> don't need to worry about voting for an unpopular choice since your vote
> will automatically roll over to your next choice.
>
> -- Owen
>
>
>


-- 
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal

--bcaec54866e277040604a5c3640e--

From general-return-3764-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 17:24:14 2011
Return-Path: <general-return-3764-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 000D24E0A
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 17:24:13 +0000 (UTC)
Received: (qmail 13488 invoked by uid 500); 15 Jun 2011 17:24:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13425 invoked by uid 500); 15 Jun 2011 17:24:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13417 invoked by uid 99); 15 Jun 2011 17:24:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:24:12 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 216.145.54.171 is neither permitted nor denied by domain of sureshms@yahoo-inc.com)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:24:06 +0000
Received: from sp1-ex07cas03.ds.corp.yahoo.com (sp1-ex07cas03.ds.corp.yahoo.com [216.252.116.151])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5FHNMOo017195
	for <general@hadoop.apache.org>; Wed, 15 Jun 2011 10:23:22 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1308158602;
	bh=0dIeAsD0Tl1b948ZwXOkjevHMfQoUksOAvqreHaw9Mw=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:MIME-Version;
	b=XyGRA6NKoQ9bEI6RDiEwzASw1UGW/JsdXilT8ExvrJQWaM10u2aQXneNEmjRavJkX
	 Gv/Lq/uuPukOIzLcsnDiQva8isiz21lMpqAOnB9baszRJ6GMqO6iiLTL/abpLwsh4N
	 FpRcYvsIDEMsbujx2lEQW4fJyXDSJvMsROIsRgCs=
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas03.ds.corp.yahoo.com ([216.252.116.151]) with mapi; Wed, 15 Jun
 2011 10:23:21 -0700
From: Suresh Srinivas <sureshms@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 15 Jun 2011 10:23:20 -0700
Subject: Re: [VOTE] Powered by Logo
Thread-Topic: [VOTE] Powered by Logo
Thread-Index: AcwrCyRCefBbe/cXTJeQuFAwnKvIdwAdcedb
Message-ID: <CA1E3898.1C0A1%sureshms@yahoo-inc.com>
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-Entourage/13.8.0.101117
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

2, 4


On 6/14/11 8:19 PM, "Owen O'Malley" <omalley@apache.org> wrote:

> All,
>    We've had a wide range of entries for a powered by logo. I've put them=
 all
> on a page, here:
>=20
> http://people.apache.org/~omalley/hadoop-powered-by/
>=20
> Since there are a lot of contenders and we only want a single round of vo=
ting,
> let's use single transferable vote ( STV
> http://en.wikipedia.org/wiki/Single_transferable_vote). The important thi=
ng is
> to pick the images *IN ORDER* that you would like them.
>=20
> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>=20
> In other words, I want option 4 most and option 6 least. With STV, you do=
n't
> need to worry about voting for an unpopular choice since your vote will
> automatically roll over to your next choice.
>=20
> -- Owen
>=20
>=20


From general-return-3765-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 17:25:01 2011
Return-Path: <general-return-3765-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2D6014E44
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 17:25:01 +0000 (UTC)
Received: (qmail 15014 invoked by uid 500); 15 Jun 2011 17:24:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14958 invoked by uid 500); 15 Jun 2011 17:24:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14950 invoked by uid 99); 15 Jun 2011 17:24:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:24:59 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of owen.omalley@gmail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:24:50 +0000
Received: by iwr19 with SMTP id 19so702313iwr.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 10:24:29 -0700 (PDT)
Received: by 10.231.82.197 with SMTP id c5mr620502ibl.131.1308158669428;
        Wed, 15 Jun 2011 10:24:29 -0700 (PDT)
Received: from battlerock-lm.corp.yahoo.com (nat-dip4.cfw-a-gci.corp.yahoo.com [209.131.62.113])
        by mx.google.com with ESMTPS id ly7sm573055icb.12.2011.06.15.10.24.24
        (version=SSLv3 cipher=OTHER);
        Wed, 15 Jun 2011 10:24:27 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: [VOTE] Powered by Logo
From: Owen O'Malley <omalley@apache.org>
In-Reply-To: <BANLkTi=3A7y2tpazNy1Tf5VeA_gMMm13Jw@mail.gmail.com>
Date: Wed, 15 Jun 2011 10:24:23 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <068A4109-EFA8-467E-B7B9-9F3F56225E74@apache.org>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org> <DEC5F4CA57D742BD8467E54523CE6CF3@china.huawei.com> <BANLkTi=wJR-fhn0snNnFXcG_bmD=AUAVog@mail.gmail.com> <BANLkTinOVG-CcVLL-xr_S9OWne_b4D5WYg@mail.gmail.com> <AECC1E7D-F1D9-41DB-A2A2-7A5CEA3BB9B6@apache.org> <BANLkTi=3A7y2tpazNy1Tf5VeA_gMMm13Jw@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org


On Jun 15, 2011, at 9:58 AM, Aaron T. Myers wrote:

> On that note, is it the case that the outcome of this vote is to adopt =
the
> "style" of the winning option? Or the precise pixels? I assume the =
former.

It is about the style. Technically, it would require a vote to replace =
the precise pixels, but usually two alternatives in the same style one =
is clearly better and we can make the decision without an explicit vote.

-- Owen


From general-return-3766-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 17:31:25 2011
Return-Path: <general-return-3766-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D434B46A4
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 17:31:25 +0000 (UTC)
Received: (qmail 24563 invoked by uid 500); 15 Jun 2011 17:31:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24504 invoked by uid 500); 15 Jun 2011 17:31:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24496 invoked by uid 99); 15 Jun 2011 17:31:24 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:31:24 +0000
Received: from localhost (HELO mail-iw0-f176.google.com) (127.0.0.1)
  (smtp-auth username phunt, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:31:22 +0000
Received: by iwr19 with SMTP id 19so710617iwr.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 10:31:22 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.202.144 with SMTP id fe16mr617587ibb.133.1308159082216;
 Wed, 15 Jun 2011 10:31:22 -0700 (PDT)
Received: by 10.42.239.7 with HTTP; Wed, 15 Jun 2011 10:31:21 -0700 (PDT)
In-Reply-To: <CA1E3898.1C0A1%sureshms@yahoo-inc.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
	<CA1E3898.1C0A1%sureshms@yahoo-inc.com>
Date: Wed, 15 Jun 2011 10:31:21 -0700
Message-ID: <BANLkTin_UqgRE_P2rdRBupG9j1DJ_d3TTw@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: Patrick Hunt <phunt@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

5, 6, 1, 3, 4, 2

On Wed, Jun 15, 2011 at 10:23 AM, Suresh Srinivas
<sureshms@yahoo-inc.com> wrote:
> 2, 4
>
>
> On 6/14/11 8:19 PM, "Owen O'Malley" <omalley@apache.org> wrote:
>
>> All,
>> =A0 =A0We've had a wide range of entries for a powered by logo. I've put=
 them all
>> on a page, here:
>>
>> http://people.apache.org/~omalley/hadoop-powered-by/
>>
>> Since there are a lot of contenders and we only want a single round of v=
oting,
>> let's use single transferable vote ( STV
>> http://en.wikipedia.org/wiki/Single_transferable_vote). The important th=
ing is
>> to pick the images *IN ORDER* that you would like them.
>>
>> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>>
>> In other words, I want option 4 most and option 6 least. With STV, you d=
on't
>> need to worry about voting for an unpopular choice since your vote will
>> automatically roll over to your next choice.
>>
>> -- Owen
>>
>>
>
>

From general-return-3767-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 17:44:33 2011
Return-Path: <general-return-3767-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3E89C4BA8
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 17:44:33 +0000 (UTC)
Received: (qmail 53981 invoked by uid 500); 15 Jun 2011 17:44:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53866 invoked by uid 500); 15 Jun 2011 17:44:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53858 invoked by uid 99); 15 Jun 2011 17:44:31 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:44:31 +0000
Received: from localhost (HELO mail-fx0-f48.google.com) (127.0.0.1)
  (smtp-auth username mahadev, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:44:31 +0000
Received: by fxm7 with SMTP id 7so933366fxm.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 10:44:29 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.223.92.154 with SMTP id r26mr18819fam.35.1308159869624; Wed,
 15 Jun 2011 10:44:29 -0700 (PDT)
Received: by 10.223.100.7 with HTTP; Wed, 15 Jun 2011 10:44:29 -0700 (PDT)
In-Reply-To: <BANLkTin_UqgRE_P2rdRBupG9j1DJ_d3TTw@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
	<CA1E3898.1C0A1%sureshms@yahoo-inc.com>
	<BANLkTin_UqgRE_P2rdRBupG9j1DJ_d3TTw@mail.gmail.com>
Date: Wed, 15 Jun 2011 10:44:29 -0700
Message-ID: <BANLkTik2X1xs=MtXMNeiXmX8GW2T4jXKYg@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: Mahadev Konar <mahadev@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

2, 4, 6.


mahadev

On Wed, Jun 15, 2011 at 10:31 AM, Patrick Hunt <phunt@apache.org> wrote:
> 5, 6, 1, 3, 4, 2
>
> On Wed, Jun 15, 2011 at 10:23 AM, Suresh Srinivas
> <sureshms@yahoo-inc.com> wrote:
>> 2, 4
>>
>>
>> On 6/14/11 8:19 PM, "Owen O'Malley" <omalley@apache.org> wrote:
>>
>>> All,
>>> =A0 =A0We've had a wide range of entries for a powered by logo. I've pu=
t them all
>>> on a page, here:
>>>
>>> http://people.apache.org/~omalley/hadoop-powered-by/
>>>
>>> Since there are a lot of contenders and we only want a single round of =
voting,
>>> let's use single transferable vote ( STV
>>> http://en.wikipedia.org/wiki/Single_transferable_vote). The important t=
hing is
>>> to pick the images *IN ORDER* that you would like them.
>>>
>>> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>>>
>>> In other words, I want option 4 most and option 6 least. With STV, you =
don't
>>> need to worry about voting for an unpopular choice since your vote will
>>> automatically roll over to your next choice.
>>>
>>> -- Owen
>>>
>>>
>>
>>
>



--=20
thanks
mahadev
@mahadevkonar

From general-return-3768-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 17:45:27 2011
Return-Path: <general-return-3768-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B7FD24F0D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 17:45:27 +0000 (UTC)
Received: (qmail 57393 invoked by uid 500); 15 Jun 2011 17:45:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57315 invoked by uid 500); 15 Jun 2011 17:45:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57300 invoked by uid 99); 15 Jun 2011 17:45:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:45:25 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 216.145.54.173 is neither permitted nor denied by domain of mattf@yahoo-inc.com)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:45:17 +0000
Received: from SP2-EX07CAS04.ds.corp.yahoo.com (sp2-ex07cas04.corp.sp2.yahoo.com [98.137.59.5])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p5FHiYmk056545;
	Wed, 15 Jun 2011 10:44:34 -0700 (PDT)
Received: from SP2-EX07VS03.ds.corp.yahoo.com ([98.137.59.32]) by
 SP2-EX07CAS04.ds.corp.yahoo.com ([98.137.59.5]) with mapi; Wed, 15 Jun 2011
 10:44:34 -0700
From: Matthew Foley <mattf@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: Matthew Foley <mattf@yahoo-inc.com>,
        "trademarks@apache.org"
	<trademarks@apache.org>
Date: Wed, 15 Jun 2011 10:44:33 -0700
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
Thread-Topic: [VOTE] Shall we adopt the "Defining Hadoop" page
Thread-Index: Acwrg+KNtkiQeg+MSTGAUSS6eqE8/w==
Message-ID: <54668CDE-443D-4AF1-B17A-0497D0CDCC8D@yahoo-inc.com>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
 <BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com>
 <090F2993-B9AA-410F-AD99-7AC047FFC974@apache.org>
 <BANLkTi=fkguAtbfi9Wk9CPm8JmFa2Hti3Q@mail.gmail.com>
In-Reply-To: <BANLkTi=fkguAtbfi9Wk9CPm8JmFa2Hti3Q@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Eli, you said:
> Putting a build of Hadoop that has 4 security patches applied into the sa=
me
> category as a product that has entirely re-worked the code and not
> gotten it checked into trunk does a major disservice to the people who
> contribute to and invest in the project.

How would you phrase the distinction, so that it is clear and reasonably un=
ambiguous
for people who are not Hadoop developers?  Do the HTTP and Subversion polic=
ies
draw this distinction, and if so could you please point us at the specific =
text, or=20
copy that text to this thread?

Thanks,
--Matt


On Jun 15, 2011, at 9:40 AM, Eli Collins wrote:

On Tue, Jun 14, 2011 at 7:45 PM, Owen O'Malley <omalley@apache.org> wrote:
>=20
> On Jun 14, 2011, at 5:48 PM, Eli Collins wrote:
>=20
>> Wrt derivative works, it's not clear from the document, but I think we
>> should explicitly adopt the policy of HTTPD and Subversion that
>> backported patches from trunk and security fixes are permitted.
>=20
> Actually, the document is extremely clear that only Apache releases may b=
e called Hadoop.
>=20
> There was a very long thread about why the rapidly expanding Hadoop-ecosy=
stem is leading to at lot of customer confusion about the different "versio=
ns" of Hadoop. We as the Hadoop project don't have the resources or the nec=
essary compatibility test suite to test compatibility between the different=
 sets of cherry picked patches. We also don't have time to ensure that all =
of the 1,000's of patches applied to 0.20.2 in each of the many (10? 15?) d=
ifferent versions have been committed to trunk. Futhermore, under the Apach=
e license, a company Foo could claim that it is a cherry pick version of Ha=
doop without releasing their source code that would enable verification.
>=20
> In summary,
>  1. Hadoop is very successful.
>  2. There are many different commercial products that are trying to use t=
he Hadoop name.
>  3. We can't check or enforce that the cherry pick versions are following=
 the rules.
>  4. We don't have a TCK like Java does to validate new versions are compa=
tible.
>  5. By far the most fair way to ensure compatibility and fairness between=
 companies is that only Apache Hadoop releases may be called Hadoop.
>=20
> That said, a package that includes a small number (< 3) of security patch=
es that haven't been released yet doesn't seem unreasonable.
>=20

I've spoken with ops teams at many companies,  I am not aware of
anyone who runs an official release (with just 2 security patches). By
this definition many of the most valuable contributors to Hadoop,
including Yahoo!, Cloudera, Facebook, etc are not using Hadoop.  Is
that really the message we want to send? We expect the PMC to enforce
this equally across all parties?

It's a fact of life that companies and ops teams that support Hadoop
need to patch the software before the PMC has time and/or will to vote
on new releases. This is why HTTP and Subversion allow this. Putting a
build of Hadoop that has 4 security patches applied into the same
category as a product that has entirely re-worked the code and not
gotten it checked into trunk does a major disservice to the people who
contribute to and invest in the project.

Thanks,
Eli


From general-return-3769-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 17:50:30 2011
Return-Path: <general-return-3769-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 037CA429F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 17:50:30 +0000 (UTC)
Received: (qmail 62992 invoked by uid 500); 15 Jun 2011 17:50:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62837 invoked by uid 500); 15 Jun 2011 17:50:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62829 invoked by uid 99); 15 Jun 2011 17:50:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:50:28 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [98.139.52.205] (HELO nm8.bullet.mail.ac4.yahoo.com) (98.139.52.205)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 15 Jun 2011 17:50:19 +0000
Received: from [98.139.52.190] by nm8.bullet.mail.ac4.yahoo.com with NNFMP; 15 Jun 2011 17:49:57 -0000
Received: from [98.139.52.168] by tm3.bullet.mail.ac4.yahoo.com with NNFMP; 15 Jun 2011 17:49:57 -0000
Received: from [127.0.0.1] by omp1051.mail.ac4.yahoo.com with NNFMP; 15 Jun 2011 17:49:57 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 743388.17113.bm@omp1051.mail.ac4.yahoo.com
Received: (qmail 61383 invoked by uid 60001); 15 Jun 2011 17:49:57 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1308160197; bh=poikW3J4cHwpfXC743SeO2Y125Vttwq2e+GEAD8pTNc=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=i0wCOrrgaSUg5IPNHkZ1iIJkEI3wBh1y4hAf8qm9d3+p5UKgkCOKaggdNRBGSzx9UQOX8m6/IIsrCbW4SEuzYK0Fj4QF55i1Z7vDp/G4t3J9k8bF9yohFT8Jc+yftqgnJTJBJ83V1mjSHUSN/csszK0EZKPsgSovMjLZjdbpX0k=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=OT3dZ/hvyoeVXQjmh7FTOIzlnoUN2Hdx+YYCmfRiDSxoJslpUff0toZTLwfPcEwEVd+I8VopZUznNhgkbnU6Gw/p7bb3zuCW4mizYwD2ZbhYt5WyQALV+5Br/5sKjuX8wXktWuG+wvVhPvh45JVCui425VMbBoAZGP5nP24a5QU=;
Message-ID: <635634.55935.qm@web65904.mail.ac4.yahoo.com>
X-YMail-OSG: o1wt7bMVM1n1_3jooPT_jnYwCsAJLOy7EejOMCWIey9yEJS
 xMiug90254WeSSMKoSmEod5dSQ_qsCQNJfCOnDfkU15C3sG4RmTTOvyXIIq7
 sUMPDrdFcADPkAmKCON0D9ylkkt4q4FFe_5d8aGjlnfQ6Ua_5lCSfW76hm6N
 GncNgs1lmM9nWL9DYUVGoQ6Q.Lve4aVJp0jud8jfz76CtNtx86Q0VhH4Zntt
 xK_FsZYShc088dWoer78bd65ih6IiAO7tnZL2isqox_VvEsTKxpb2S87hXFR
 qR0ktHMuhU1D2A_YP.Dc..W3XkHSDkS74Yj.1viobHLzyhdfKHezNSvCz39C
 7quCYaus44zbvINV8dTlCUhTg.cusO0V1jmz4fY7WronTEptFsEUYHklDTcv
 qCTAfLkwcqMRWMUFcE8lH8Z0Ig_ZMPvx10dw67KUQ2ljRG3WtgRpwPyhBjpY
 1FGy3AUSkWxlHrDkZ0DuvfHZmQ4Pc55GqncmiQW7No.NFOMeyRNuwHCxEDaR
 0DXXA7arE9zRM4ioOiA--
Received: from [209.131.62.113] by web65904.mail.ac4.yahoo.com via HTTP; Wed, 15 Jun 2011 10:49:57 PDT
X-Mailer: YahooMailRC/570 YahooMailWebService/0.8.111.304355
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
Date: Wed, 15 Jun 2011 10:49:57 -0700 (PDT)
From: "Tsz Wo \(Nicholas\), Sze" <s29752-hadoopgeneral@yahoo.com>
Subject: Re: [VOTE] Powered by Logo
To: general@hadoop.apache.org
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1523813513-1308160197=:55935"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-1523813513-1308160197=:55935
Content-Type: text/plain; charset=us-ascii

4, 2

Regards,
Tsz-Wo





________________________________
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
Sent: Tue, June 14, 2011 8:19:18 PM
Subject: [VOTE] Powered by Logo

All,
   We've had a wide range of entries for a powered by logo. I've put them all on 
a page, here:

http://people.apache.org/~omalley/hadoop-powered-by/

Since there are a lot of contenders and we only want a single round of voting, 
let's use single transferable vote ( STV 
http://en.wikipedia.org/wiki/Single_transferable_vote). The important thing is 
to pick the images *IN ORDER* that you would like them.

My vote (in order of course): 4, 1, 2, 3, 5, 6.

In other words, I want option 4 most and option 6 least. With STV, you don't 
need to worry about voting for an unpopular choice since your vote will 
automatically roll over to your next choice.

-- Owen
--0-1523813513-1308160197=:55935--

From general-return-3770-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 18:01:17 2011
Return-Path: <general-return-3770-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B56AC4D72
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 18:01:17 +0000 (UTC)
Received: (qmail 86779 invoked by uid 500); 15 Jun 2011 18:01:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86720 invoked by uid 500); 15 Jun 2011 18:01:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86705 invoked by uid 99); 15 Jun 2011 18:01:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 18:01:15 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 69.147.107.21 is neither permitted nor denied by domain of mattf@yahoo-inc.com)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 18:01:07 +0000
Received: from SP2-EX07CAS05.ds.corp.yahoo.com (sp2-ex07cas05.corp.sp2.yahoo.com [98.137.59.39])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5FI0LtQ023748;
	Wed, 15 Jun 2011 11:00:21 -0700 (PDT)
Received: from SP2-EX07VS03.ds.corp.yahoo.com ([98.137.59.32]) by
 SP2-EX07CAS05.ds.corp.yahoo.com ([98.137.59.39]) with mapi; Wed, 15 Jun 2011
 11:00:21 -0700
From: Matthew Foley <mattf@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "trademarks@apache.org" <trademarks@apache.org>,
        Matthew Foley
	<mattf@yahoo-inc.com>
Date: Wed, 15 Jun 2011 11:00:20 -0700
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
Thread-Topic: [VOTE] Shall we adopt the "Defining Hadoop" page
Thread-Index: AcwrhhbgxJxKg/kTR0aiVqmVkFsFZw==
Message-ID: <EE59E34D-6618-4A58-8D09-95669FF2EC08@yahoo-inc.com>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
 <BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com>
 <090F2993-B9AA-410F-AD99-7AC047FFC974@apache.org>
 <BANLkTi=fkguAtbfi9Wk9CPm8JmFa2Hti3Q@mail.gmail.com>
 <54668CDE-443D-4AF1-B17A-0497D0CDCC8D@yahoo-inc.com>
In-Reply-To: <54668CDE-443D-4AF1-B17A-0497D0CDCC8D@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Oh, and while I can't officially vote, I think this page is extremely well =
done and
I strongly support it. =20

As an editorial note, however, I would remove the last paragraph in the "Co=
mpatibility"=20
section, referencing the email thread (that I contributed to at length :-) =
). =20
That thread went all over the place, and would be misinforming to the typic=
al
reader.  The distillation on this twiki page IS normative and not confusing=
, and=20
we should leave it at that.

Best,
--Matt

On Jun 15, 2011, at 10:44 AM, Matthew Foley wrote:

Eli, you said:
> Putting a build of Hadoop that has 4 security patches applied into the sa=
me
> category as a product that has entirely re-worked the code and not
> gotten it checked into trunk does a major disservice to the people who
> contribute to and invest in the project.

How would you phrase the distinction, so that it is clear and reasonably un=
ambiguous
for people who are not Hadoop developers?  Do the HTTP and Subversion polic=
ies
draw this distinction, and if so could you please point us at the specific =
text, or=20
copy that text to this thread?

Thanks,
--Matt


On Jun 15, 2011, at 9:40 AM, Eli Collins wrote:

On Tue, Jun 14, 2011 at 7:45 PM, Owen O'Malley <omalley@apache.org> wrote:
>=20
> On Jun 14, 2011, at 5:48 PM, Eli Collins wrote:
>=20
>> Wrt derivative works, it's not clear from the document, but I think we
>> should explicitly adopt the policy of HTTPD and Subversion that
>> backported patches from trunk and security fixes are permitted.
>=20
> Actually, the document is extremely clear that only Apache releases may b=
e called Hadoop.
>=20
> There was a very long thread about why the rapidly expanding Hadoop-ecosy=
stem is leading to at lot of customer confusion about the different "versio=
ns" of Hadoop. We as the Hadoop project don't have the resources or the nec=
essary compatibility test suite to test compatibility between the different=
 sets of cherry picked patches. We also don't have time to ensure that all =
of the 1,000's of patches applied to 0.20.2 in each of the many (10? 15?) d=
ifferent versions have been committed to trunk. Futhermore, under the Apach=
e license, a company Foo could claim that it is a cherry pick version of Ha=
doop without releasing their source code that would enable verification.
>=20
> In summary,
> 1. Hadoop is very successful.
> 2. There are many different commercial products that are trying to use th=
e Hadoop name.
> 3. We can't check or enforce that the cherry pick versions are following =
the rules.
> 4. We don't have a TCK like Java does to validate new versions are compat=
ible.
> 5. By far the most fair way to ensure compatibility and fairness between =
companies is that only Apache Hadoop releases may be called Hadoop.
>=20
> That said, a package that includes a small number (< 3) of security patch=
es that haven't been released yet doesn't seem unreasonable.
>=20

I've spoken with ops teams at many companies,  I am not aware of
anyone who runs an official release (with just 2 security patches). By
this definition many of the most valuable contributors to Hadoop,
including Yahoo!, Cloudera, Facebook, etc are not using Hadoop.  Is
that really the message we want to send? We expect the PMC to enforce
this equally across all parties?

It's a fact of life that companies and ops teams that support Hadoop
need to patch the software before the PMC has time and/or will to vote
on new releases. This is why HTTP and Subversion allow this. Putting a
build of Hadoop that has 4 security patches applied into the same
category as a product that has entirely re-worked the code and not
gotten it checked into trunk does a major disservice to the people who
contribute to and invest in the project.

Thanks,
Eli



From general-return-3771-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 18:14:16 2011
Return-Path: <general-return-3771-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A37F040A6
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 18:14:16 +0000 (UTC)
Received: (qmail 8916 invoked by uid 500); 15 Jun 2011 18:14:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8867 invoked by uid 500); 15 Jun 2011 18:14:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8859 invoked by uid 99); 15 Jun 2011 18:14:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 18:14:14 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ted.dunning@gmail.com designates 209.85.220.176 as permitted sender)
Received: from [209.85.220.176] (HELO mail-vx0-f176.google.com) (209.85.220.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 18:14:08 +0000
Received: by vxa37 with SMTP id 37so813123vxa.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 11:13:47 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:cc:content-type;
        bh=CIuIYSDkCo7Xigkt0HY9ZUNDJiaLU4zbncqwkOQlWN4=;
        b=ZRGcQ2dtTwAJDE4FVE9dgtfa7C0iDyticLvZKR4o0/TDUf4W9dEol+grkiLgp2gspU
         sG2BIxEGfzVhb4fCkrylxWX2vwkJ9SkO2TK0ZLY2MCF4PFtUTdCxhQp6k2Cp/DwDFhUW
         oBl07v3GwELvblvEfg8d+kkLiXqOGM6KG34rY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :cc:content-type;
        b=W6H+Pes4Ni5xtJA/UK8ZNd9VwQ93rP3WAHr9NYrW+avRy5oDFKBs0NfEvjyHACARuQ
         IdDKdd9UXPE0yzfI2xQsHTw6FQ5FKcoY2obrEuZCRQ+3y6OfOBlQ2x9FKpwAPpgeRe0r
         PaMKbELzyGbGeR3G55lClQVUdbxW9Q28mPGcE=
Received: by 10.52.98.39 with SMTP id ef7mr66829vdb.88.1308161627201; Wed, 15
 Jun 2011 11:13:47 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.52.187.66 with HTTP; Wed, 15 Jun 2011 11:13:27 -0700 (PDT)
In-Reply-To: <BANLkTi=fkguAtbfi9Wk9CPm8JmFa2Hti3Q@mail.gmail.com>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
 <BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com> <090F2993-B9AA-410F-AD99-7AC047FFC974@apache.org>
 <BANLkTi=fkguAtbfi9Wk9CPm8JmFa2Hti3Q@mail.gmail.com>
From: Ted Dunning <ted.dunning@gmail.com>
Date: Wed, 15 Jun 2011 20:13:27 +0200
Message-ID: <BANLkTi=BcQUEWBN1WPhXnZm8so_SDJp7Ow@mail.gmail.com>
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
To: trademarks@apache.org
Cc: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf307f37ce1921da04a5c41ddc
X-Virus-Checked: Checked by ClamAV on apache.org

--20cf307f37ce1921da04a5c41ddc
Content-Type: text/plain; charset=UTF-8

+1 to what Eli says.  If nobody is running official Hadoop according to this
definition, but everybody thinks that they are running hadoop, then this
definition is a bit out of whack.  The source of the dissonance is related
to the fact that release just don't happen often enough in Hadoop.

In addition, I think that the limitations on usage are too strict.  For
instance, if "QuickBooks for Windows" [1] doesn't cause Microsoft to sue
Intuit, then "Joe's Foo for Apache Hadoop" really shouldn't cause any more
grief.

So I would give a (non-binding) -1 to the policy as stated.

[1]
http://quickbooks.intuit.com/product/accounting_software/windows_financial_management_software.jsp

On Wed, Jun 15, 2011 at 6:40 PM, Eli Collins <eli@cloudera.com> wrote:

> On Tue, Jun 14, 2011 at 7:45 PM, Owen O'Malley <omalley@apache.org> wrote:
> >
> > On Jun 14, 2011, at 5:48 PM, Eli Collins wrote:
> >
> >> Wrt derivative works, it's not clear from the document, but I think we
> >> should explicitly adopt the policy of HTTPD and Subversion that
> >> backported patches from trunk and security fixes are permitted.
> >
> > Actually, the document is extremely clear that only Apache releases may
> be called Hadoop.
> >
> > There was a very long thread about why the rapidly expanding
> Hadoop-ecosystem is leading to at lot of customer confusion about the
> different "versions" of Hadoop. We as the Hadoop project don't have the
> resources or the necessary compatibility test suite to test compatibility
> between the different sets of cherry picked patches. We also don't have time
> to ensure that all of the 1,000's of patches applied to 0.20.2 in each of
> the many (10? 15?) different versions have been committed to trunk.
> Futhermore, under the Apache license, a company Foo could claim that it is a
> cherry pick version of Hadoop without releasing their source code that would
> enable verification.
> >
> > In summary,
> >  1. Hadoop is very successful.
> >  2. There are many different commercial products that are trying to use
> the Hadoop name.
> >  3. We can't check or enforce that the cherry pick versions are following
> the rules.
> >  4. We don't have a TCK like Java does to validate new versions are
> compatible.
> >  5. By far the most fair way to ensure compatibility and fairness between
> companies is that only Apache Hadoop releases may be called Hadoop.
> >
> > That said, a package that includes a small number (< 3) of security
> patches that haven't been released yet doesn't seem unreasonable.
> >
>
> I've spoken with ops teams at many companies,  I am not aware of
> anyone who runs an official release (with just 2 security patches). By
> this definition many of the most valuable contributors to Hadoop,
> including Yahoo!, Cloudera, Facebook, etc are not using Hadoop.  Is
> that really the message we want to send? We expect the PMC to enforce
> this equally across all parties?
>
> It's a fact of life that companies and ops teams that support Hadoop
> need to patch the software before the PMC has time and/or will to vote
> on new releases. This is why HTTP and Subversion allow this. Putting a
> build of Hadoop that has 4 security patches applied into the same
> category as a product that has entirely re-worked the code and not
> gotten it checked into trunk does a major disservice to the people who
> contribute to and invest in the project.
>
> Thanks,
> Eli
>

--20cf307f37ce1921da04a5c41ddc--

From general-return-3772-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 18:22:19 2011
Return-Path: <general-return-3772-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AA8B248FA
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 18:22:19 +0000 (UTC)
Received: (qmail 34410 invoked by uid 500); 15 Jun 2011 18:22:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34341 invoked by uid 500); 15 Jun 2011 18:22:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34329 invoked by uid 99); 15 Jun 2011 18:22:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 18:22:17 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of tom.e.white@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 18:22:11 +0000
Received: by wwi18 with SMTP id 18so782091wwi.29
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 11:21:51 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:cc:content-type:content-transfer-encoding;
        bh=mChKf+1vJq5ndVXFvVLB4XkwY8ETHOUTDOon9ngDu+8=;
        b=PhADrl9mq0jvlNSLvOgngORqbKMtChaXL15GlRIFsC0K4EGkXx3/FbeZLoIf7s+uMu
         NGA46SBYHYXB0qpnbwu8XFRPmC1FnqcGtcv3YD9WOMt3Q7wb/jt5B5HpX4trLdpcXUHD
         CMnsig5Lmg4pzNrvXkazBsIHw84CkTxxt3Pt0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :cc:content-type:content-transfer-encoding;
        b=QF0/oz8sVumuX1H5zkI/Ete9rS8ZsV8HLpI1efM8WJ11GqDq7DIpc7OGGkSgKLgxN5
         Rv2IQrcwgsTA1q2I1Gbuw93YbOj5XMdKbDWiC94vjaa+y1RDWgtu6QmULYxc1Lew61K2
         syJzCxPKEsPwYqA6ZFohGEgC7kh17s1qO9c/U=
Received: by 10.217.5.130 with SMTP id w2mr2441239wes.61.1308162111217; Wed,
 15 Jun 2011 11:21:51 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.3.21 with HTTP; Wed, 15 Jun 2011 11:21:31 -0700 (PDT)
In-Reply-To: <E4194561-D1E3-4D1E-91EB-449427805669@yahoo-inc.com>
References: <457011.690.qm@web65508.mail.ac4.yahoo.com> <F0CA2186-6E63-459D-ABFF-21863EE018E5@yahoo-inc.com>
 <BANLkTikYuO0nRHz3UWZFZU245dMw82Y08Q@mail.gmail.com> <8B7097C4-597D-4BA4-8C69-C48F5A571AEF@yahoo-inc.com>
 <E4194561-D1E3-4D1E-91EB-449427805669@yahoo-inc.com>
From: Tom White <tom.e.white@gmail.com>
Date: Wed, 15 Jun 2011 11:21:31 -0700
Message-ID: <BANLkTi=290oMBaBxq4EBEdPbfTQq5ez++A@mail.gmail.com>
Subject: Re: LimitedPrivate and HBase
To: general@hadoop.apache.org
Cc: "apurtell@apache.org" <apurtell@apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

No offense taken.

Sanjay and I chatted offline. We disagree on whether Private should be
scoped to the whole project (Hadoop), or to subprojects (Common, HDFS,
MapReduce). The intent of HADOOP-5073 was for the latter, but I'm not
convinced it buys us anything really, which is why I've been arguing
for the former. However, what we have at the moment in the javadoc is
misleading, so that at least needs clearing up via HADOOP-7391. Thanks
for doing this Sanjay.

I do think (and I think Sanjay agrees with me here) we should move
away from LimitedPrivate access for external projects by creating
public APIs that we are prepared to support (note again that they may
be Evolving or even Unstable) - either by marking existing
(Limited)Private APIs as public, or by creating a new API. We should
do this for the 6 cases I highlighted earlier in the thread.

Cheers,
Tom

On Tue, Jun 14, 2011 at 2:20 PM, Sanjay Radia <sradia@yahoo-inc.com> wrote:
>
> On Jun 14, 2011, at 12:58 PM, Sanjay Radia wrote:
>
>>
>>
>>
>> -1
>> I disagree with the proposed changes.
>> .....
>> I will post a longer email explaining my position and my -1 more
>> clearly after I have had a chance to read all the emails carefully.
>>
>>
>> sanjay
>
> Please don't take my -1 too strongly.
> It was NOT meant to be offensive. I saw a lot of +1s and wanted to make s=
ure
> that this doesn't turn into a jira and a commit in =A0few days.
>
> sanjay

From general-return-3773-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 18:23:19 2011
Return-Path: <general-return-3773-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3ECE74962
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 18:23:19 +0000 (UTC)
Received: (qmail 37608 invoked by uid 500); 15 Jun 2011 18:23:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37547 invoked by uid 500); 15 Jun 2011 18:23:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37539 invoked by uid 99); 15 Jun 2011 18:23:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 18:23:17 +0000
X-ASF-Spam-Status: No, hits=3.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jameswarren3@gmail.com designates 209.85.161.176 as permitted sender)
Received: from [209.85.161.176] (HELO mail-gx0-f176.google.com) (209.85.161.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 18:23:10 +0000
Received: by gxk7 with SMTP id 7so625210gxk.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 11:22:49 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:reply-to:sender:in-reply-to
         :references:from:date:x-google-sender-auth:message-id:subject:to
         :content-type;
        bh=lz+qzbOzs2uCBdu28wJE5fFQg5oPoQCSa9J8vjx572k=;
        b=I9X/igIhjgl4BD9wOwlLpGdsTEzHB50Ft+pwouS4xJa46zgXyN1neGVy5JR78QnBVf
         e5skk0/3YQy6nrmo6bd5kqz8b6zaVCphowqf298kdvnSD0wEuQ2wVq6bFctw6FsnKjUU
         UYwiHIJ+J0xET9ytmq2EMq7EqYezhT9LcLEYY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:reply-to:sender:in-reply-to:references:from:date
         :x-google-sender-auth:message-id:subject:to:content-type;
        b=KuTPyQgS8CLZFKelAuFzzdC/6A+0geXLwzG2pD1/kyDXNUNS7xp57P2Gx/ZfqGijbk
         SdSxiEnKbAA9pgOrLBxBlJHzAXMUqsX60K+vP/pFSvHYK+cpQtXj9hbCzgrsBsgwHomr
         rTd2pP8+rK2OOjnwpi/EAT5qIhGfBbKqd/Ykw=
Received: by 10.236.154.233 with SMTP id h69mr107691yhk.84.1308162169393; Wed,
 15 Jun 2011 11:22:49 -0700 (PDT)
MIME-Version: 1.0
Reply-To: james.warren@stanfordalumni.org
Sender: jameswarren3@gmail.com
Received: by 10.236.153.228 with HTTP; Wed, 15 Jun 2011 11:22:09 -0700 (PDT)
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
From: James Warren <james.warren@stanfordalumni.org>
Date: Wed, 15 Jun 2011 11:22:09 -0700
X-Google-Sender-Auth: Royv0_AqDARcRAD0Q1i_xoXLxyI
Message-ID: <BANLkTik8=X3KjaZvBe-U+h8eYay33KoohQ@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf303f69c26a52b804a5c43d22
X-Virus-Checked: Checked by ClamAV on apache.org

--20cf303f69c26a52b804a5c43d22
Content-Type: text/plain; charset=ISO-8859-1

5, 4, 2, 3, 1, 6

On Tue, Jun 14, 2011 at 8:19 PM, Owen O'Malley <omalley@apache.org> wrote:

> All,
>   We've had a wide range of entries for a powered by logo. I've put them
> all on a page, here:
>
> http://people.apache.org/~omalley/hadoop-powered-by/
>
> Since there are a lot of contenders and we only want a single round of
> voting, let's use single transferable vote ( STV
> http://en.wikipedia.org/wiki/Single_transferable_vote). The important
> thing is to pick the images *IN ORDER* that you would like them.
>
> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>
> In other words, I want option 4 most and option 6 least. With STV, you
> don't need to worry about voting for an unpopular choice since your vote
> will automatically roll over to your next choice.
>
> -- Owen
>
>
>

--20cf303f69c26a52b804a5c43d22--

From general-return-3774-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 18:24:13 2011
Return-Path: <general-return-3774-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 672AE499F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 18:24:13 +0000 (UTC)
Received: (qmail 40447 invoked by uid 500); 15 Jun 2011 18:24:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40299 invoked by uid 500); 15 Jun 2011 18:24:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40290 invoked by uid 99); 15 Jun 2011 18:24:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 18:24:11 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.220.176] (HELO mail-vx0-f176.google.com) (209.85.220.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 18:24:02 +0000
Received: by vxa37 with SMTP id 37so826063vxa.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 11:23:41 -0700 (PDT)
Received: by 10.52.72.230 with SMTP id g6mr61771vdv.163.1308162221051; Wed, 15
 Jun 2011 11:23:41 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.52.111.166 with HTTP; Wed, 15 Jun 2011 11:23:21 -0700 (PDT)
X-Originating-IP: [82.105.90.90]
In-Reply-To: <4DF75531.9050302@apache.org>
References: <4DF75531.9050302@apache.org>
From: Ted Dunning <tdunning@maprtech.com>
Date: Wed, 15 Jun 2011 20:23:21 +0200
Message-ID: <BANLkTikChz5mH=NFdjd_8r7UMNPJTAHVuQ@mail.gmail.com>
Subject: Re: Bigtop Proposal
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec50160117e8f6404a5c4400f
X-Virus-Checked: Checked by ClamAV on apache.org

--bcaec50160117e8f6404a5c4400f
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Jun 14, 2011 at 2:33 PM, Steve Loughran <stevel@apache.org> wrote:

> For those people who aren't on the Apache Incubator mailing lists, Tom
> White has proposed "BigTop", which is the tooling to integrate the build and
> testing of the Apache Hadoop technologies, perhaps eventually to have
> coordinated releases.
>

I will comment on the proposal directly as well, but coordinated releases
would be a disaster.  Mahout, for one, should not have to wait for Hadoop to
have a release.  We are already unhappy that we only get two releases a year
and want to up that to four.

Much better is to have a directed acyclic dependency graph with coherent
versioning.  That allows projects to stay independent, but still express
version requirements.



>  -It might make sense for the hadoop, pig, hive, hama, mahout, hbase &c
> committers to all have access to this, so that they can help evolve the
> packaging and functional tests.
>
>
Multi-project access sounds right.

--bcaec50160117e8f6404a5c4400f--

From general-return-3775-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 18:38:36 2011
Return-Path: <general-return-3775-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CA86F439E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 18:38:35 +0000 (UTC)
Received: (qmail 71607 invoked by uid 500); 15 Jun 2011 18:38:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71527 invoked by uid 500); 15 Jun 2011 18:38:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71514 invoked by uid 99); 15 Jun 2011 18:38:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 18:38:34 +0000
X-ASF-Spam-Status: No, hits=2.9 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 216.145.54.173 is neither permitted nor denied by domain of acm@yahoo-inc.com)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 18:38:29 +0000
Received: from [192.168.1.9] (vpn-client-104-2.eglbp.corp.yahoo.com [10.66.104.2])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p5FIbrLE078494;
	Wed, 15 Jun 2011 11:37:54 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1308163075;
	bh=Do/D17BehTXvh7KMgE2zEKOP+nf5EsBnQHxRB+iU8ZM=;
	h=Cc:Message-Id:From:To:In-Reply-To:Content-Type:Mime-Version:
	 Subject:Date:References;
	b=q6/I80Deppk3EZ6PV3M5rtWiu/ZG1U7sFkBJaoixURSuQu7fdKKkRc8OIHjt+vZEa
	 GrcjK5qii+lNjnLXtal+P+TkOTQsiAzE5zbBQ2yGHIN8IbcM1oD3oSSuU0zHfL0qJZ
	 C6Q6EAGhFDBUBhzuRGFK//02j83yoNGosPCh9m9c=
Cc: "trademarks@apache.org" <trademarks@apache.org>
Message-Id: <3E913938-C23D-42DA-8A85-B89298AB09BA@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <BANLkTi=fkguAtbfi9Wk9CPm8JmFa2Hti3Q@mail.gmail.com>
Content-Type: multipart/alternative; boundary=Apple-Mail-27--856608964
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
Date: Thu, 16 Jun 2011 00:07:52 +0530
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org> <BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com> <090F2993-B9AA-410F-AD99-7AC047FFC974@apache.org> <BANLkTi=fkguAtbfi9Wk9CPm8JmFa2Hti3Q@mail.gmail.com>
X-Mailer: Apple Mail (2.936)

--Apple-Mail-27--856608964
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


On Jun 15, 2011, at 10:10 PM, Eli Collins wrote:

> I've spoken with ops teams at many companies,  I am not aware of
> anyone who runs an official release (with just 2 security patches). By
> this definition many of the most valuable contributors to Hadoop,
> including Yahoo!, Cloudera, Facebook, etc are not using Hadoop.  Is
> that really the message we want to send? We expect the PMC to enforce
> this equally across all parties?

This is only a recent (less than 2 yrs) phenomenon with hadoop-0.20  
onwards.

I've been on the project for over 5 years now and I've run official  
Apache Hadoop releases at Y! for the majority of that time. From  
hadoop-0.1 to hadoop-0.18.

IAC, getting everyone to run an official release isn't an anti-goal.  
And, as Steve points out, this really doesn't concern internal  
deployments - public redistributables is something I worry about as a  
PMC member with my Apache hat on.

+1 for the current version (Defining Hadoop (last edited 2011-06-09  
02:56:39 by OwenOMalley)

thanks,
Arun



--Apple-Mail-27--856608964--

From general-return-3776-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 18:45:19 2011
Return-Path: <general-return-3776-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6980F481F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 18:45:19 +0000 (UTC)
Received: (qmail 89864 invoked by uid 500); 15 Jun 2011 18:45:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89786 invoked by uid 500); 15 Jun 2011 18:45:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89778 invoked by uid 99); 15 Jun 2011 18:45:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 18:45:17 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mcsrivas@gmail.com designates 74.125.83.176 as permitted sender)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 18:45:12 +0000
Received: by pve37 with SMTP id 37so770622pve.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 11:44:52 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=BAVSLC7AEy8kk6rM1A1mkZWQe+MINGx3NrpdwmmB7nU=;
        b=mq8FGW8wEW+9yOQucNzpQymXy/58ECSDghfIouc6Ix+b8Yed5DfnmIbe6KbgaDEo+K
         NKaCDuVykZ+dI/3dgFmzBEtYI4ssn1uGBcCwyxOo2K3aHqNFX9LjnMxSwZm2dKJe1H45
         kNYwVPe5wGpHbDDIWK0NVk1vwSvbu8hp5mCLs=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=cj9NOmXSB/p4hpKEM/a3Nz/FZ73pHY5lRZcUn85AbxHodeMSgsqbN5tH5nLFV/0R5j
         ziCQDhsixG2R9W4dW4p/Ta8ZmjMaz0GTsTIrWqz0R2BDeE8RPZjE5JHiJL8Ka21jqCND
         H3NWgLGmzAET7Sctfc2mtzEjmSHSlTaMs0GEo=
MIME-Version: 1.0
Received: by 10.68.2.1 with SMTP id 1mr94571pbq.102.1308163492368; Wed, 15 Jun
 2011 11:44:52 -0700 (PDT)
Received: by 10.68.48.201 with HTTP; Wed, 15 Jun 2011 11:44:52 -0700 (PDT)
In-Reply-To: <BANLkTik8=X3KjaZvBe-U+h8eYay33KoohQ@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
	<BANLkTik8=X3KjaZvBe-U+h8eYay33KoohQ@mail.gmail.com>
Date: Wed, 15 Jun 2011 11:44:52 -0700
Message-ID: <BANLkTikRhg-HHm=4m+22bUgUJLFr7miBvg@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: "M. C. Srivas" <mcsrivas@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec52158d345528404a5c48c1a

--bcaec52158d345528404a5c48c1a
Content-Type: text/plain; charset=ISO-8859-1

4, 2, 6, 1

On Wed, Jun 15, 2011 at 11:22 AM, James Warren <
james.warren@stanfordalumni.org> wrote:

> 5, 4, 2, 3, 1, 6
>
> On Tue, Jun 14, 2011 at 8:19 PM, Owen O'Malley <omalley@apache.org> wrote:
>
> > All,
> >   We've had a wide range of entries for a powered by logo. I've put them
> > all on a page, here:
> >
> > http://people.apache.org/~omalley/hadoop-powered-by/
> >
> > Since there are a lot of contenders and we only want a single round of
> > voting, let's use single transferable vote ( STV
> > http://en.wikipedia.org/wiki/Single_transferable_vote). The important
> > thing is to pick the images *IN ORDER* that you would like them.
> >
> > My vote (in order of course): 4, 1, 2, 3, 5, 6.
> >
> > In other words, I want option 4 most and option 6 least. With STV, you
> > don't need to worry about voting for an unpopular choice since your vote
> > will automatically roll over to your next choice.
> >
> > -- Owen
> >
> >
> >
>

--bcaec52158d345528404a5c48c1a--

From general-return-3777-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 18:47:44 2011
Return-Path: <general-return-3777-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 71D734344
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 18:47:44 +0000 (UTC)
Received: (qmail 92469 invoked by uid 500); 15 Jun 2011 18:47:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92414 invoked by uid 500); 15 Jun 2011 18:47:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 57610 invoked by uid 99); 15 Jun 2011 07:37:15 -0000
X-ASF-Spam-Status: No, hits=1.3 required=5.0
	tests=RCVD_IN_BL_SPAMCOP_NET,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of bbc@bbcn.name designates 209.85.160.66 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <BANLkTinOVG-CcVLL-xr_S9OWne_b4D5WYg@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
	<DEC5F4CA57D742BD8467E54523CE6CF3@china.huawei.com>
	<BANLkTi=wJR-fhn0snNnFXcG_bmD=AUAVog@mail.gmail.com>
	<BANLkTinOVG-CcVLL-xr_S9OWne_b4D5WYg@mail.gmail.com>
Date: Wed, 15 Jun 2011 15:36:45 +0800
Message-ID: <BANLkTi=QxMBeXg6DGmpTp83UnAR6B+dE5g@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: Bochun Bai <bbc@bbcn.name>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

6, 4, 5, 2, 1, 3


On Wed, Jun 15, 2011 at 2:57 PM, Daniel Jue <teamphy6@gmail.com> wrote:
> My vote:
>
> 5,4,3,2,6,1
>
> IMO, 5 is much more polished than the rest, but I like the whole elephant in 4.
>

From general-return-3778-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 18:54:39 2011
Return-Path: <general-return-3778-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 09D024F37
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 18:54:38 +0000 (UTC)
Received: (qmail 5983 invoked by uid 500); 15 Jun 2011 18:54:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5864 invoked by uid 500); 15 Jun 2011 18:54:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5799 invoked by uid 99); 15 Jun 2011 18:54:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 18:54:30 +0000
X-ASF-Spam-Status: No, hits=3.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of akimball83@gmail.com designates 209.85.218.48 as permitted sender)
Received: from [209.85.218.48] (HELO mail-yi0-f48.google.com) (209.85.218.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 18:54:24 +0000
Received: by yic24 with SMTP id 24so655244yic.35
        for <multiple recipients>; Wed, 15 Jun 2011 11:54:03 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:from:date:message-id:subject:to
         :content-type;
        bh=GWunHiBxoU3GkYH/8YhdVcg04uiJG5E+zbdyZ69+KL0=;
        b=kEx8iqm7ZsbJmBR0rgrSj5gKy7VvYeKBshP8feV/UyCoIlBJfPwQn6+HfH61na4qc7
         oS/MPlbl2tf2h/t7CPm1dXv+OPRby8l4ZRWJ0fV0X/ywzdFsXVEVYJEzKPZcEc8FLli4
         LwPeLBJ40TJwt/aTmCB/7C2ibtHjbreLNXRwI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type;
        b=qqbrNihSjsAjperIPgppS0p9zjfakU9LPRVvehfOLeL3dvKMFT0xUh9bgSscGcWkkV
         9lsI9uZe7TL74W/pD4ohgZmP2soxIWmqPuMQ3VAinZKfq8r/YgModtHkyIuxb/W04X5Z
         mF1/6A6KdjKhKVt/dAZjPRFEMce7BH4YGfO3w=
Received: by 10.150.210.6 with SMTP id i6mr86318ybg.311.1308164043099; Wed, 15
 Jun 2011 11:54:03 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.150.220.6 with HTTP; Wed, 15 Jun 2011 11:53:43 -0700 (PDT)
From: Aaron Kimball <akimball83@gmail.com>
Date: Wed, 15 Jun 2011 11:53:43 -0700
Message-ID: <BANLkTik9DnVJGt0j__q_anFcJku6M1miWw@mail.gmail.com>
Subject: Meetup Announcement: July 2011 SF HUG (7/13/2011)
To: general@hadoop.apache.org, common-user@hadoop.apache.org, 
	user@pig.apache.org, user@hive.apache.org, user@hbase.apache.org
Content-Type: multipart/alternative; boundary=000e0cd3561818ce0004a5c4adca
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd3561818ce0004a5c4adca
Content-Type: text/plain; charset=ISO-8859-1

Hadoop fans,

Last week we held another successful San Francisco Hadoop "unconference"
meetup at RichRelevance's office.

Discussion topics included:
* Working with multiple Hadoop clusters
* HBase
* Hive SerDes and evolving data formats
* Zookeeper and Leader Election
* Timeseries databases / statistics
* Hadoop internals
* Hadoop Pipes
* Hive performance with Joins
* Pig + Java
* Avro

We will hold next month's meetup on Wednesday July 13, from 6--8pm.
This meetup will be hosted once again by our friends at CBSi. Their office
is at 235 Second Street.

As usual, we will use the discussion-based "unconference" format. At the
beginning of the meetup we will collaboratively construct an agenda
consisting of several discussion breakout groups. All participants may
propose a topic and volunteer to facilitate a discussion. All Hadoop-related
topics are encouraged, and all members of the Hadoop community are welcome.
It's never too early to start brainstorming topics if you'd like to lead a
session -- send me an email off-list if you want to discuss an idea.

Event schedule:

* 6pm - Welcome
* 6:30pm - Introductions; start creating agenda
* Breakout sessions begin as soon as we're ready
* 8pm - Conclusion

Food and refreshments will be provided, courtesy of CBSi.

I hope to see you there! Please RSVP at http://bit.ly/kLpLQR so we can get
an accurate count for food and beverages.
Cheers,
- Aaron Kimball

--000e0cd3561818ce0004a5c4adca--

From general-return-3779-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 18:55:45 2011
Return-Path: <general-return-3779-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7B3E44FBF
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 18:55:45 +0000 (UTC)
Received: (qmail 15048 invoked by uid 500); 15 Jun 2011 18:55:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14996 invoked by uid 500); 15 Jun 2011 18:55:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14988 invoked by uid 99); 15 Jun 2011 18:55:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 18:55:43 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 18:55:36 +0000
Received: by bwz8 with SMTP id 8so1211118bwz.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 11:55:14 -0700 (PDT)
Received: by 10.204.114.144 with SMTP id e16mr52369bkq.119.1308164114399; Wed,
 15 Jun 2011 11:55:14 -0700 (PDT)
MIME-Version: 1.0
Sender: bsolowiej@maprtech.com
Received: by 10.204.66.200 with HTTP; Wed, 15 Jun 2011 11:54:43 -0700 (PDT)
X-Originating-IP: [64.105.168.204]
In-Reply-To: <BANLkTi=QxMBeXg6DGmpTp83UnAR6B+dE5g@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
 <DEC5F4CA57D742BD8467E54523CE6CF3@china.huawei.com> <BANLkTi=wJR-fhn0snNnFXcG_bmD=AUAVog@mail.gmail.com>
 <BANLkTinOVG-CcVLL-xr_S9OWne_b4D5WYg@mail.gmail.com> <BANLkTi=QxMBeXg6DGmpTp83UnAR6B+dE5g@mail.gmail.com>
From: Bartosz Solowiej <toszter@maprtech.com>
Date: Wed, 15 Jun 2011 11:54:43 -0700
X-Google-Sender-Auth: nAs9FOLpyPdvLwC0gA-6kvFNFu4
Message-ID: <BANLkTimkxowVxHX+GsmLn02uSvczChtr5A@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016368e2ae858c01704a5c4b1bd

--0016368e2ae858c01704a5c4b1bd
Content-Type: text/plain; charset=ISO-8859-1

2, 4

On Wed, Jun 15, 2011 at 12:36 AM, Bochun Bai <bbc@bbcn.name> wrote:

> 6, 4, 5, 2, 1, 3
>
>
> On Wed, Jun 15, 2011 at 2:57 PM, Daniel Jue <teamphy6@gmail.com> wrote:
> > My vote:
> >
> > 5,4,3,2,6,1
> >
> > IMO, 5 is much more polished than the rest, but I like the whole elephant
> in 4.
> >
>

--0016368e2ae858c01704a5c4b1bd--

From general-return-3780-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 18:58:24 2011
Return-Path: <general-return-3780-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A71794099
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 18:58:24 +0000 (UTC)
Received: (qmail 23764 invoked by uid 500); 15 Jun 2011 18:58:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23712 invoked by uid 500); 15 Jun 2011 18:58:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23704 invoked by uid 99); 15 Jun 2011 18:58:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 18:58:22 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 18:58:15 +0000
Received: by iym1 with SMTP id 1so815966iym.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 11:57:53 -0700 (PDT)
Received: by 10.231.81.83 with SMTP id w19mr66756ibk.34.1308164273485;
        Wed, 15 Jun 2011 11:57:53 -0700 (PDT)
Received: from [192.168.100.146] (h-64-105-168-204.snvacaid.static.covad.net [64.105.168.204])
        by mx.google.com with ESMTPS id s9sm367707ibe.27.2011.06.15.11.57.51
        (version=TLSv1/SSLv3 cipher=OTHER);
        Wed, 15 Jun 2011 11:57:52 -0700 (PDT)
Message-ID: <4DF900AC.5070506@maprtech.com>
Date: Wed, 15 Jun 2011 11:57:48 -0700
From: Peter Conrad <pconrad@maprtech.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.4) Gecko/20100608 Lightning/1.0b2 Thunderbird/3.1
MIME-Version: 1.0
To: general@hadoop.apache.org
CC: Bartosz Solowiej <toszter@maprtech.com>
Subject: Re: [VOTE] Powered by Logo
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org> <DEC5F4CA57D742BD8467E54523CE6CF3@china.huawei.com> <BANLkTi=wJR-fhn0snNnFXcG_bmD=AUAVog@mail.gmail.com> <BANLkTinOVG-CcVLL-xr_S9OWne_b4D5WYg@mail.gmail.com> <BANLkTi=QxMBeXg6DGmpTp83UnAR6B+dE5g@mail.gmail.com> <BANLkTimkxowVxHX+GsmLn02uSvczChtr5A@mail.gmail.com>
In-Reply-To: <BANLkTimkxowVxHX+GsmLn02uSvczChtr5A@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

  4, 2, 6, 1, 3, 5

On 06/15/2011 11:54 AM, Bartosz Solowiej wrote:
> 2, 4
>
> On Wed, Jun 15, 2011 at 12:36 AM, Bochun Bai<bbc@bbcn.name>  wrote:
>
>> 6, 4, 5, 2, 1, 3
>>
>>
>> On Wed, Jun 15, 2011 at 2:57 PM, Daniel Jue<teamphy6@gmail.com>  wrote:
>>> My vote:
>>>
>>> 5,4,3,2,6,1
>>>
>>> IMO, 5 is much more polished than the rest, but I like the whole elephant
>> in 4.

From general-return-3781-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 18:59:23 2011
Return-Path: <general-return-3781-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 341B840F8
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 18:59:23 +0000 (UTC)
Received: (qmail 28201 invoked by uid 500); 15 Jun 2011 18:59:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28111 invoked by uid 500); 15 Jun 2011 18:59:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28103 invoked by uid 99); 15 Jun 2011 18:59:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 18:59:21 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 18:59:12 +0000
Received: by bwz8 with SMTP id 8so1215486bwz.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 11:58:52 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.169.130 with SMTP id z2mr48682bky.137.1308164331654; Wed,
 15 Jun 2011 11:58:51 -0700 (PDT)
Received: by 10.204.53.68 with HTTP; Wed, 15 Jun 2011 11:58:51 -0700 (PDT)
X-Originating-IP: [64.105.168.204]
In-Reply-To: <BANLkTi=QxMBeXg6DGmpTp83UnAR6B+dE5g@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
	<DEC5F4CA57D742BD8467E54523CE6CF3@china.huawei.com>
	<BANLkTi=wJR-fhn0snNnFXcG_bmD=AUAVog@mail.gmail.com>
	<BANLkTinOVG-CcVLL-xr_S9OWne_b4D5WYg@mail.gmail.com>
	<BANLkTi=QxMBeXg6DGmpTp83UnAR6B+dE5g@mail.gmail.com>
Date: Wed, 15 Jun 2011 11:58:51 -0700
Message-ID: <BANLkTi=S6bcAze=r6XuYEf=XuQ=JEkfDvw@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: Subhash Gopinath <sgopinath@maprtech.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

4, 2, 6

From general-return-3782-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 19:02:02 2011
Return-Path: <general-return-3782-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A30764BCD
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 19:02:02 +0000 (UTC)
Received: (qmail 37469 invoked by uid 500); 15 Jun 2011 19:02:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37396 invoked by uid 500); 15 Jun 2011 19:02:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37382 invoked by uid 99); 15 Jun 2011 19:02:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 19:02:00 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of tom.e.white@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 19:01:53 +0000
Received: by wyb40 with SMTP id 40so823982wyb.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 12:01:33 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type:content-transfer-encoding;
        bh=7t9lejXA6uafHGgtudY8x8iFUqmkUp540JqEHXRkfvE=;
        b=wQlfXR9t2OvW52gvh4Nnclkpel8VL+Loz/b0nCrRy9PuOBkYEKS+ejlINoGK2NNQdb
         HL3EP0vWTnj+TjKEsRJb9RJ2UvrezJ7KIJDrWFPgjTliDJfxVg4UpRNAiH8NYUHJuKpa
         xf6RP5B/xPlch/Smi1rGe5zMP/SkJngCMIvQo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=MveMYdnTM4vWTiekzEsiozmF5yMCSHELeIjYPmGLvwVRm6DaBUgd6yU63ZjTRP3k/X
         zMkYMFvxPzcFekFWeovTxK3zYxYgfWdm6qOEBjaQrTgf2DIkuYT1Y+FfHCfi61S0cj03
         DUifaje3u8fNZxDJF1Qgze44UmrriqOOzipzc=
Received: by 10.216.229.3 with SMTP id g3mr14890weq.91.1308164493155; Wed, 15
 Jun 2011 12:01:33 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.3.21 with HTTP; Wed, 15 Jun 2011 12:01:13 -0700 (PDT)
In-Reply-To: <BANLkTikChz5mH=NFdjd_8r7UMNPJTAHVuQ@mail.gmail.com>
References: <4DF75531.9050302@apache.org> <BANLkTikChz5mH=NFdjd_8r7UMNPJTAHVuQ@mail.gmail.com>
From: Tom White <tom.e.white@gmail.com>
Date: Wed, 15 Jun 2011 12:01:13 -0700
Message-ID: <BANLkTikoS5HpcDQ6dA-NOMD4gXX2jx3t3w@mail.gmail.com>
Subject: Re: Bigtop Proposal
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Wed, Jun 15, 2011 at 11:23 AM, Ted Dunning <tdunning@maprtech.com> wrote=
:
> On Tue, Jun 14, 2011 at 2:33 PM, Steve Loughran <stevel@apache.org> wrote=
:
>
>> For those people who aren't on the Apache Incubator mailing lists, Tom
>> White has proposed "BigTop", which is the tooling to integrate the build=
 and
>> testing of the Apache Hadoop technologies, perhaps eventually to have
>> coordinated releases.
>>
>
> I will comment on the proposal directly as well, but coordinated releases
> would be a disaster. =A0Mahout, for one, should not have to wait for Hado=
op to
> have a release. =A0We are already unhappy that we only get two releases a=
 year
> and want to up that to four.
>
> Much better is to have a directed acyclic dependency graph with coherent
> versioning. =A0That allows projects to stay independent, but still expres=
s
> version requirements.

I agree. Bigtop would use the releases that the upstream projects have
made - it cannot impose new requirements on them like coordinated
releases. Any bugs that are uncovered would be submitted to the
upstream project, for release there.

I'm going to clarify the text in the proposal on this. Thanks for
pointing it out Ted.

Cheers,
Tom

>
>
>
>> =A0-It might make sense for the hadoop, pig, hive, hama, mahout, hbase &=
c
>> committers to all have access to this, so that they can help evolve the
>> packaging and functional tests.
>>
>>
> Multi-project access sounds right.
>

From general-return-3783-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 19:12:56 2011
Return-Path: <general-return-3783-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7E5B04C4A
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 19:12:56 +0000 (UTC)
Received: (qmail 70885 invoked by uid 500); 15 Jun 2011 19:12:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70808 invoked by uid 500); 15 Jun 2011 19:12:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 40269 invoked by uid 99); 15 Jun 2011 19:02:32 -0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
X-Originating-IP: [64.105.168.204]
Date: Wed, 15 Jun 2011 12:02:03 -0700
Message-ID: <BANLkTinYxrmgcXWNKyaimwt8eb27OCh=cA@mail.gmail.com>
Subject: Powered by Logo Vote
From: Richa Khandelwal <rkhandelwal@maprtech.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e687b6d2baf85804a5c4c996

--0016e687b6d2baf85804a5c4c996
Content-Type: text/plain; charset=ISO-8859-1

4, 2, 6, 1, 3, 5

Thanks,
Richa

--0016e687b6d2baf85804a5c4c996--

From general-return-3784-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 19:19:37 2011
Return-Path: <general-return-3784-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E0C6D4219
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 19:19:36 +0000 (UTC)
Received: (qmail 84394 invoked by uid 500); 15 Jun 2011 19:19:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84337 invoked by uid 500); 15 Jun 2011 19:19:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84329 invoked by uid 99); 15 Jun 2011 19:19:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 19:19:34 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.213.48] (HELO mail-yw0-f48.google.com) (209.85.213.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 19:19:27 +0000
Received: by ywk9 with SMTP id 9so674779ywk.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 12:19:06 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.236.192.228 with SMTP id i64mr217905yhn.111.1308165546185;
 Wed, 15 Jun 2011 12:19:06 -0700 (PDT)
Received: by 10.236.208.105 with HTTP; Wed, 15 Jun 2011 12:19:06 -0700 (PDT)
X-Originating-IP: [64.105.168.204]
In-Reply-To: <BANLkTikRhg-HHm=4m+22bUgUJLFr7miBvg@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
	<BANLkTik8=X3KjaZvBe-U+h8eYay33KoohQ@mail.gmail.com>
	<BANLkTikRhg-HHm=4m+22bUgUJLFr7miBvg@mail.gmail.com>
Date: Wed, 15 Jun 2011 12:19:06 -0700
Message-ID: <BANLkTimC=acgJJvdcFi1=cHg25t2Wh9HZQ@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: Chun Chang <cchang@maprtech.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf30563ed7b0139b04a5c5062b
X-Virus-Checked: Checked by ClamAV on apache.org

--20cf30563ed7b0139b04a5c5062b
Content-Type: text/plain; charset=ISO-8859-1

2,4,6,1,3,5

On Wed, Jun 15, 2011 at 11:44 AM, M. C. Srivas <mcsrivas@gmail.com> wrote:

> 4, 2, 6, 1
>
> On Wed, Jun 15, 2011 at 11:22 AM, James Warren <
> james.warren@stanfordalumni.org> wrote:
>
> > 5, 4, 2, 3, 1, 6
> >
> > On Tue, Jun 14, 2011 at 8:19 PM, Owen O'Malley <omalley@apache.org>
> wrote:
> >
> > > All,
> > >   We've had a wide range of entries for a powered by logo. I've put
> them
> > > all on a page, here:
> > >
> > > http://people.apache.org/~omalley/hadoop-powered-by/
> > >
> > > Since there are a lot of contenders and we only want a single round of
> > > voting, let's use single transferable vote ( STV
> > > http://en.wikipedia.org/wiki/Single_transferable_vote). The important
> > > thing is to pick the images *IN ORDER* that you would like them.
> > >
> > > My vote (in order of course): 4, 1, 2, 3, 5, 6.
> > >
> > > In other words, I want option 4 most and option 6 least. With STV, you
> > > don't need to worry about voting for an unpopular choice since your
> vote
> > > will automatically roll over to your next choice.
> > >
> > > -- Owen
> > >
> > >
> > >
> >
>

--20cf30563ed7b0139b04a5c5062b--

From general-return-3785-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 20:26:55 2011
Return-Path: <general-return-3785-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D5E1B41BF
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 20:26:55 +0000 (UTC)
Received: (qmail 26884 invoked by uid 500); 15 Jun 2011 20:26:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26842 invoked by uid 500); 15 Jun 2011 20:26:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 15360 invoked by uid 99); 15 Jun 2011 20:23:43 -0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
X-Originating-IP: [64.105.168.204]
Date: Wed, 15 Jun 2011 13:23:13 -0700
Message-ID: <BANLkTikc5_sFK0wuwfR9iwLBsexouJyPpg@mail.gmail.com>
Subject: 2 4 6 5 3 1
From: Dave Jespersen <djespersen@maprtech.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636af00acfdab9a04a5c5eb5a
X-Virus-Checked: Checked by ClamAV on apache.org

--001636af00acfdab9a04a5c5eb5a
Content-Type: text/plain; charset=ISO-8859-1

2 4 6 5 3 1

--001636af00acfdab9a04a5c5eb5a--

From general-return-3786-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 20:56:04 2011
Return-Path: <general-return-3786-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4756B4952
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 20:56:04 +0000 (UTC)
Received: (qmail 74867 invoked by uid 500); 15 Jun 2011 20:56:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74698 invoked by uid 500); 15 Jun 2011 20:56:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74681 invoked by uid 99); 15 Jun 2011 20:56:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 20:56:01 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [156.148.72.33] (HELO raffaello.crs4.it) (156.148.72.33)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 20:55:54 +0000
Received: from [172.16.106.57] (vpn-dc-nat.crs4.it [156.148.160.116])
	by raffaello.crs4.it (Postfix) with ESMTP id EE00879011A
	for <general@hadoop.apache.org>; Wed, 15 Jun 2011 22:55:31 +0200 (CEST)
Message-ID: <4DF92A08.8010106@crs4.it>
Date: Wed, 15 Jun 2011 23:54:16 +0200
From: Simone Leo <simone.leo@crs4.it>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.17) Gecko/20110510 Lightning/1.0b3pre Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Powered by Logo
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org> <DEC5F4CA57D742BD8467E54523CE6CF3@china.huawei.com> <BANLkTi=wJR-fhn0snNnFXcG_bmD=AUAVog@mail.gmail.com> <BANLkTinOVG-CcVLL-xr_S9OWne_b4D5WYg@mail.gmail.com> <AECC1E7D-F1D9-41DB-A2A2-7A5CEA3BB9B6@apache.org>
In-Reply-To: <AECC1E7D-F1D9-41DB-A2A2-7A5CEA3BB9B6@apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

I only like 4. A more polished version would be great. Maybe with
"Hadoop" instead of "Apache Hadoop". And maybe with a different font :)

On 06/15/11 16:35, Owen O'Malley wrote:
> 
> On Jun 14, 2011, at 11:57 PM, Daniel Jue wrote:
> 
>> IMO, 5 is much more polished than the rest, but I like the whole elephant in 4.
> 
> Ted has offered to make a more polished version of 4, if there is interest in adopting it.
> 
> -- Owen


-- 
Simone Leo
Data Fusion - Distributed Computing
CRS4
POLARIS - Building #1
Piscina Manna
I-09010 Pula (CA) - Italy
e-mail: simone.leo@crs4.it
http://www.crs4.it

From general-return-3787-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 22:26:26 2011
Return-Path: <general-return-3787-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2506E4A51
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 22:26:26 +0000 (UTC)
Received: (qmail 96120 invoked by uid 500); 15 Jun 2011 22:26:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95933 invoked by uid 500); 15 Jun 2011 22:26:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95925 invoked by uid 99); 15 Jun 2011 22:26:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 22:26:24 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 22:26:18 +0000
Received: by wwi18 with SMTP id 18so987048wwi.29
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 15:25:58 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.159.75 with SMTP id r53mr115919wek.98.1308176757854; Wed,
 15 Jun 2011 15:25:57 -0700 (PDT)
Received: by 10.216.30.212 with HTTP; Wed, 15 Jun 2011 15:25:57 -0700 (PDT)
In-Reply-To: <3E913938-C23D-42DA-8A85-B89298AB09BA@yahoo-inc.com>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
	<BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com>
	<090F2993-B9AA-410F-AD99-7AC047FFC974@apache.org>
	<BANLkTi=fkguAtbfi9Wk9CPm8JmFa2Hti3Q@mail.gmail.com>
	<3E913938-C23D-42DA-8A85-B89298AB09BA@yahoo-inc.com>
Date: Wed, 15 Jun 2011 15:25:57 -0700
Message-ID: <BANLkTi=YJKfPXc1qSUzLmi=rpJSR-ccpWg@mail.gmail.com>
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Cc: "trademarks@apache.org" <trademarks@apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Wed, Jun 15, 2011 at 11:37 AM, Arun C Murthy <acm@yahoo-inc.com> wrote:
>
> On Jun 15, 2011, at 10:10 PM, Eli Collins wrote:
>
>> I've spoken with ops teams at many companies, =A0I am not aware of
>> anyone who runs an official release (with just 2 security patches). By
>> this definition many of the most valuable contributors to Hadoop,
>> including Yahoo!, Cloudera, Facebook, etc are not using Hadoop. =A0Is
>> that really the message we want to send? We expect the PMC to enforce
>> this equally across all parties?
>
> This is only a recent (less than 2 yrs) phenomenon with hadoop-0.20 onwar=
ds.
>
> I've been on the project for over 5 years now and I've run official Apach=
e
> Hadoop releases at Y! for the majority of that time. From hadoop-0.1 to
> hadoop-0.18.

But Yahoo! hasn't.  According to this wiki YDH (0.20.100) would *not*
be considered Apache Hadoop. For example see HADOOP-6962 which refers
to 0.20.9, an internal Yahoo! release, not an official Apache release.
Are you really comfortable saying Yahoo! doesn't run Hadoop?

Thanks,
Eli

From general-return-3788-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 22:42:39 2011
Return-Path: <general-return-3788-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CAE56442C
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 22:42:39 +0000 (UTC)
Received: (qmail 26878 invoked by uid 500); 15 Jun 2011 22:42:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26720 invoked by uid 500); 15 Jun 2011 22:42:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26705 invoked by uid 99); 15 Jun 2011 22:42:37 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 22:42:37 +0000
Received: from localhost (HELO mail-iy0-f176.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 22:42:37 +0000
Received: by iym1 with SMTP id 1so1055627iym.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 15:42:36 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.84.3 with SMTP id h3mr134859ibl.109.1308177756831; Wed, 15
 Jun 2011 15:42:36 -0700 (PDT)
Received: by 10.231.79.133 with HTTP; Wed, 15 Jun 2011 15:42:36 -0700 (PDT)
In-Reply-To: <BANLkTi=YJKfPXc1qSUzLmi=rpJSR-ccpWg@mail.gmail.com>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
	<BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com>
	<090F2993-B9AA-410F-AD99-7AC047FFC974@apache.org>
	<BANLkTi=fkguAtbfi9Wk9CPm8JmFa2Hti3Q@mail.gmail.com>
	<3E913938-C23D-42DA-8A85-B89298AB09BA@yahoo-inc.com>
	<BANLkTi=YJKfPXc1qSUzLmi=rpJSR-ccpWg@mail.gmail.com>
Date: Wed, 15 Jun 2011 15:42:36 -0700
Message-ID: <BANLkTik_rr4LiiimQ5AhY=syBprXK_C0nA@mail.gmail.com>
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
From: Chris Douglas <cdouglas@apache.org>
To: trademarks@apache.org
Cc: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Wed, Jun 15, 2011 at 3:25 PM, Eli Collins <eli@cloudera.com> wrote:
> But Yahoo! hasn't. =A0According to this wiki YDH (0.20.100) would *not*
> be considered Apache Hadoop. For example see HADOOP-6962 which refers
> to 0.20.9, an internal Yahoo! release, not an official Apache release.
> Are you really comfortable saying Yahoo! doesn't run Hadoop?

This conclusion does not follow. The guidelines prohibit companies
from distributing that software as "Hadoop". It takes no position on
whether a company that modifies a release of Apache Hadoop for its
deployment credits the project.

On Wed, Jun 15, 2011 at 11:13 AM, Ted Dunning <ted.dunning@gmail.com> wrote=
:
> In addition, I think that the limitations on usage are too strict. =A0For
> instance, if "QuickBooks for Windows" [1] doesn't cause Microsoft to sue
> Intuit, then "Joe's Foo for Apache Hadoop" really shouldn't cause any mor=
e
> grief.

This analogy is also inexact. QuickBooks is an application running on
Windows, not a replacement for it. -C

From general-return-3789-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 23:11:35 2011
Return-Path: <general-return-3789-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 04DF44B31
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 15 Jun 2011 23:11:35 +0000 (UTC)
Received: (qmail 80563 invoked by uid 500); 15 Jun 2011 23:11:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80360 invoked by uid 500); 15 Jun 2011 23:11:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80251 invoked by uid 99); 15 Jun 2011 23:11:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 23:11:33 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of eli@cloudera.com designates 74.125.82.42 as permitted sender)
Received: from [74.125.82.42] (HELO mail-ww0-f42.google.com) (74.125.82.42)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 23:11:28 +0000
Received: by wwk4 with SMTP id 4so1183234wwk.5
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 16:11:06 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.217.7.3 with SMTP id z3mr160487wes.68.1308179466730; Wed, 15
 Jun 2011 16:11:06 -0700 (PDT)
Received: by 10.216.30.212 with HTTP; Wed, 15 Jun 2011 16:11:06 -0700 (PDT)
In-Reply-To: <BANLkTik_rr4LiiimQ5AhY=syBprXK_C0nA@mail.gmail.com>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
	<BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com>
	<090F2993-B9AA-410F-AD99-7AC047FFC974@apache.org>
	<BANLkTi=fkguAtbfi9Wk9CPm8JmFa2Hti3Q@mail.gmail.com>
	<3E913938-C23D-42DA-8A85-B89298AB09BA@yahoo-inc.com>
	<BANLkTi=YJKfPXc1qSUzLmi=rpJSR-ccpWg@mail.gmail.com>
	<BANLkTik_rr4LiiimQ5AhY=syBprXK_C0nA@mail.gmail.com>
Date: Wed, 15 Jun 2011 16:11:06 -0700
Message-ID: <BANLkTino1_EwDjnR+4DO_DztEOuH0B+GnQ@mail.gmail.com>
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Cc: trademarks@apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Wed, Jun 15, 2011 at 3:42 PM, Chris Douglas <cdouglas@apache.org> wrote:
> On Wed, Jun 15, 2011 at 3:25 PM, Eli Collins <eli@cloudera.com> wrote:
>> But Yahoo! hasn't. =A0According to this wiki YDH (0.20.100) would *not*
>> be considered Apache Hadoop. For example see HADOOP-6962 which refers
>> to 0.20.9, an internal Yahoo! release, not an official Apache release.
>> Are you really comfortable saying Yahoo! doesn't run Hadoop?
>
> This conclusion does not follow. The guidelines prohibit companies
> from distributing that software as "Hadoop". It takes no position on
> whether a company that modifies a release of Apache Hadoop for its
> deployment credits the project.
>

This is independent of distribution. The guideline clearly defines
such an artifact as a "derivative work" and states that "Products that
are derivative works of Apache Hadoop are not Apache Hadoop".
Therefore it's false for the company to claim they are using Apache
Hadoop.

Thanks,
Eli

From general-return-3790-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 00:35:42 2011
Return-Path: <general-return-3790-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D5A544BD2
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 00:35:42 +0000 (UTC)
Received: (qmail 65688 invoked by uid 500); 16 Jun 2011 00:35:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65635 invoked by uid 500); 16 Jun 2011 00:35:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65627 invoked by uid 99); 16 Jun 2011 00:35:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 00:35:41 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 69.147.107.21 is neither permitted nor denied by domain of toddp@yahoo-inc.com)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 00:35:33 +0000
Received: from SP2-EX07CAS02.ds.corp.yahoo.com (sp2-ex07cas02.corp.sp2.yahoo.com [98.137.59.38])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5G0YxMu054739
	for <general@hadoop.apache.org>; Wed, 15 Jun 2011 17:34:59 -0700 (PDT)
Received: from SP2-EX07VS07.ds.corp.yahoo.com ([98.137.59.26]) by
 SP2-EX07CAS02.ds.corp.yahoo.com ([98.137.59.38]) with mapi; Wed, 15 Jun 2011
 17:34:59 -0700
From: Todd Papaioannou <toddp@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 15 Jun 2011 17:34:57 -0700
Subject: Re: [VOTE] Powered by Logo
Thread-Topic: [VOTE] Powered by Logo
Thread-Index: AcwrvThT+GJ3rC4KQtSgW1zj+stQJw==
Message-ID: <CA1E9D81.ABE7%toddp@yahoo-inc.com>
In-Reply-To: <BANLkTimkxowVxHX+GsmLn02uSvczChtr5A@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-MacOutlook/14.10.0.110310
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

2, 4, 5, 3, 6, 1


From general-return-3791-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 00:40:00 2011
Return-Path: <general-return-3791-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 617EF4C68
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 00:40:00 +0000 (UTC)
Received: (qmail 72834 invoked by uid 500); 16 Jun 2011 00:39:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72787 invoked by uid 500); 16 Jun 2011 00:39:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72779 invoked by uid 99); 16 Jun 2011 00:39:58 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 00:39:58 +0000
Received: from localhost (HELO mail-iy0-f176.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 00:39:58 +0000
Received: by iym1 with SMTP id 1so1152048iym.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 17:39:58 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.82.139 with SMTP id b11mr206048ibl.134.1308184797994; Wed,
 15 Jun 2011 17:39:57 -0700 (PDT)
Received: by 10.231.79.133 with HTTP; Wed, 15 Jun 2011 17:39:57 -0700 (PDT)
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
Date: Wed, 15 Jun 2011 17:39:57 -0700
Message-ID: <BANLkTi=j4tQGmbApTcnKTiZsKkhdpV=6cg@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

My vote: 2, 4, 5

-C

On Tue, Jun 14, 2011 at 8:19 PM, Owen O'Malley <omalley@apache.org> wrote:
> All,
> =A0 We've had a wide range of entries for a powered by logo. I've put the=
m all on a page, here:
>
> http://people.apache.org/~omalley/hadoop-powered-by/
>
> Since there are a lot of contenders and we only want a single round of vo=
ting, let's use single transferable vote ( STV http://en.wikipedia.org/wiki=
/Single_transferable_vote). The important thing is to pick the images *IN O=
RDER* that you would like them.
>
> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>
> In other words, I want option 4 most and option 6 least. With STV, you do=
n't need to worry about voting for an unpopular choice since your vote will=
 automatically roll over to your next choice.
>
> -- Owen
>
>
>

From general-return-3792-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 00:48:28 2011
Return-Path: <general-return-3792-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BAC5443CF
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 00:48:28 +0000 (UTC)
Received: (qmail 84602 invoked by uid 500); 16 Jun 2011 00:48:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84550 invoked by uid 500); 16 Jun 2011 00:48:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84542 invoked by uid 99); 16 Jun 2011 00:48:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 00:48:27 +0000
X-ASF-Spam-Status: No, hits=-1.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_SOFTFAIL
X-Spam-Check-By: apache.org
Received-SPF: softfail (nike.apache.org: transitioning domain of matei@eecs.berkeley.edu does not designate 169.229.218.147 as permitted sender)
Received: from [169.229.218.147] (HELO cm06fe.IST.Berkeley.EDU) (169.229.218.147)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 00:48:19 +0000
Received: from [12.201.122.245] (helo=[131.106.56.249])
	by cm06fe.ist.berkeley.edu with esmtpsa (TLSv1:AES128-SHA:128)
	(Exim 4.76)
	(auth plain:matei@eecs.berkeley.edu)
	(envelope-from <matei@eecs.berkeley.edu>)
	id 1QX0kX-0006KZ-Kl
	for general@hadoop.apache.org; Wed, 15 Jun 2011 17:47:58 -0700
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: [VOTE] Powered by Logo
From: Matei Zaharia <matei@eecs.berkeley.edu>
In-Reply-To: <BANLkTi=j4tQGmbApTcnKTiZsKkhdpV=6cg@mail.gmail.com>
Date: Wed, 15 Jun 2011 17:47:46 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <5249CC1F-242E-450A-BA7B-79233D62D300@eecs.berkeley.edu>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org> <BANLkTi=j4tQGmbApTcnKTiZsKkhdpV=6cg@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

5, 3, 2, 4, 6, 1

On Jun 15, 2011, at 5:39 PM, Chris Douglas wrote:

> My vote: 2, 4, 5
>=20
> -C
>=20
> On Tue, Jun 14, 2011 at 8:19 PM, Owen O'Malley <omalley@apache.org> =
wrote:
>> All,
>>   We've had a wide range of entries for a powered by logo. I've put =
them all on a page, here:
>>=20
>> http://people.apache.org/~omalley/hadoop-powered-by/
>>=20
>> Since there are a lot of contenders and we only want a single round =
of voting, let's use single transferable vote ( STV =
http://en.wikipedia.org/wiki/Single_transferable_vote). The important =
thing is to pick the images *IN ORDER* that you would like them.
>>=20
>> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>>=20
>> In other words, I want option 4 most and option 6 least. With STV, =
you don't need to worry about voting for an unpopular choice since your =
vote will automatically roll over to your next choice.
>>=20
>> -- Owen
>>=20
>>=20
>>=20


From general-return-3793-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 00:58:19 2011
Return-Path: <general-return-3793-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B2ACF462C
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 00:58:19 +0000 (UTC)
Received: (qmail 1183 invoked by uid 500); 16 Jun 2011 00:58:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1116 invoked by uid 500); 16 Jun 2011 00:58:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1108 invoked by uid 99); 16 Jun 2011 00:58:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 00:58:18 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of shv.hadoop@gmail.com designates 209.85.213.176 as permitted sender)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 00:58:11 +0000
Received: by yxd5 with SMTP id 5so870112yxd.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 17:57:50 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=WJqOd/G0S6Xr/yaTOOld60Fj3upSk4p9xPaQHlgsydY=;
        b=JkTv+sx4S0AZwxJZokzdsI4TyhgNOn1qUFkck3hrS1UAemQ6CaGG6UvJn+W6Zwvb1t
         HskcvyI/h/asO/cfZ+jUXwUQxcCJF1A7cygpLVlTjrB8wyWXDK58KGxfAleQWztcBNCy
         TFcRlGDq8jm7kJqDKx05TuC1JdKYLGH/Vpv1o=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=i1bF+fe1O2ivbNB3n6Ol9PwqWJXmPs623DzwU1DtTMMV608ty7OhVpV9vvrFk6oGAy
         BGDvxdeuGp5ZiXpTPGo6KZl++RgkZCBGd8LxrGVcbaeL4/r1djCasMTjaJvDZYs0QiSr
         DWfLzIyAEjuatRlCNvRgb0/Cb5dLCGSUtXyLA=
MIME-Version: 1.0
Received: by 10.151.5.13 with SMTP id h13mr335799ybi.144.1308185870573; Wed,
 15 Jun 2011 17:57:50 -0700 (PDT)
Received: by 10.147.113.16 with HTTP; Wed, 15 Jun 2011 17:57:50 -0700 (PDT)
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
Date: Wed, 15 Jun 2011 17:57:50 -0700
Message-ID: <BANLkTi=5HhcyxaHSmqqdV4hCgSHSBkHyoA@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd4b3c01d9f1c04a5c9c27f
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd4b3c01d9f1c04a5c9c27f
Content-Type: text/plain; charset=ISO-8859-1

It should look something like this:
http://api.ning.com/files/6tMSBWmZ67YKGzfS0NBp6Xt67YUi5L4ft6OZRsIrq-3EO40OP0mqSJs4b3*5QV0V7shmKnK44LzEGzNSvmDUFqCv6hirrSWX/1105100WC001.jpeg

But given the choices:  6 2 4 5 1 3

I put 6 first because it's the only one having Apache logo on it.
I think whatever choice wins the feather should be put on it.

Thanks,
--Konstantin


On Tue, Jun 14, 2011 at 8:19 PM, Owen O'Malley <omalley@apache.org> wrote:

> All,
>   We've had a wide range of entries for a powered by logo. I've put them
> all on a page, here:
>
> http://people.apache.org/~omalley/hadoop-powered-by/
>
> Since there are a lot of contenders and we only want a single round of
> voting, let's use single transferable vote ( STV
> http://en.wikipedia.org/wiki/Single_transferable_vote). The important
> thing is to pick the images *IN ORDER* that you would like them.
>
> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>
> In other words, I want option 4 most and option 6 least. With STV, you
> don't need to worry about voting for an unpopular choice since your vote
> will automatically roll over to your next choice.
>
> -- Owen
>
>
>

--000e0cd4b3c01d9f1c04a5c9c27f--

From general-return-3794-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 01:03:21 2011
Return-Path: <general-return-3794-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A9B5F47D7
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 01:03:21 +0000 (UTC)
Received: (qmail 4740 invoked by uid 500); 16 Jun 2011 01:03:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4681 invoked by uid 500); 16 Jun 2011 01:03:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4673 invoked by uid 99); 16 Jun 2011 01:03:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 01:03:19 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of eli@cloudera.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 01:03:14 +0000
Received: by wyb40 with SMTP id 40so1106738wyb.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 18:02:53 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.231.89 with SMTP id k67mr207866weq.113.1308186173253; Wed,
 15 Jun 2011 18:02:53 -0700 (PDT)
Received: by 10.216.30.212 with HTTP; Wed, 15 Jun 2011 18:02:53 -0700 (PDT)
In-Reply-To: <54668CDE-443D-4AF1-B17A-0497D0CDCC8D@yahoo-inc.com>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
	<BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com>
	<090F2993-B9AA-410F-AD99-7AC047FFC974@apache.org>
	<BANLkTi=fkguAtbfi9Wk9CPm8JmFa2Hti3Q@mail.gmail.com>
	<54668CDE-443D-4AF1-B17A-0497D0CDCC8D@yahoo-inc.com>
Date: Wed, 15 Jun 2011 18:02:53 -0700
Message-ID: <BANLkTimTGcEU29e_uLB52uc9CeuSkK__6w@mail.gmail.com>
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Cc: Matthew Foley <mattf@yahoo-inc.com>, "trademarks@apache.org" <trademarks@apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Wed, Jun 15, 2011 at 10:44 AM, Matthew Foley <mattf@yahoo-inc.com> wrote=
:
> Eli, you said:
>> Putting a build of Hadoop that has 4 security patches applied into the s=
ame
>> category as a product that has entirely re-worked the code and not
>> gotten it checked into trunk does a major disservice to the people who
>> contribute to and invest in the project.
>
> How would you phrase the distinction, so that it is clear and reasonably =
unambiguous
> for people who are not Hadoop developers? =A0Do the HTTP and Subversion p=
olicies
> draw this distinction, and if so could you please point us at the specifi=
c text, or
> copy that text to this thread?
>

I'll try to find it, this was told to me verbally a while back. Maybe
Roy can chime in.

Since there seems to be some confusion around distribution we should
make this explicit.  Some people are currently interpreting the
guidelines to say that if you patch an Apache Hadoop release yourself
then you're still running Apache Hadoop.  But if a vendor patches
Apache Hadoop for you then you're not running Apache Hadoop. How about
if a subcontractor patches Apache Hadoop for you, then is it Apache
Hadoop? This isn't sustainable.

Thanks,
Eli
> Thanks,
> --Matt
>
>
> On Jun 15, 2011, at 9:40 AM, Eli Collins wrote:
>
> On Tue, Jun 14, 2011 at 7:45 PM, Owen O'Malley <omalley@apache.org> wrote=
:
>>
>> On Jun 14, 2011, at 5:48 PM, Eli Collins wrote:
>>
>>> Wrt derivative works, it's not clear from the document, but I think we
>>> should explicitly adopt the policy of HTTPD and Subversion that
>>> backported patches from trunk and security fixes are permitted.
>>
>> Actually, the document is extremely clear that only Apache releases may =
be called Hadoop.
>>
>> There was a very long thread about why the rapidly expanding Hadoop-ecos=
ystem is leading to at lot of customer confusion about the different "versi=
ons" of Hadoop. We as the Hadoop project don't have the resources or the ne=
cessary compatibility test suite to test compatibility between the differen=
t sets of cherry picked patches. We also don't have time to ensure that all=
 of the 1,000's of patches applied to 0.20.2 in each of the many (10? 15?) =
different versions have been committed to trunk. Futhermore, under the Apac=
he license, a company Foo could claim that it is a cherry pick version of H=
adoop without releasing their source code that would enable verification.
>>
>> In summary,
>> =A01. Hadoop is very successful.
>> =A02. There are many different commercial products that are trying to us=
e the Hadoop name.
>> =A03. We can't check or enforce that the cherry pick versions are follow=
ing the rules.
>> =A04. We don't have a TCK like Java does to validate new versions are co=
mpatible.
>> =A05. By far the most fair way to ensure compatibility and fairness betw=
een companies is that only Apache Hadoop releases may be called Hadoop.
>>
>> That said, a package that includes a small number (< 3) of security patc=
hes that haven't been released yet doesn't seem unreasonable.
>>
>
> I've spoken with ops teams at many companies, =A0I am not aware of
> anyone who runs an official release (with just 2 security patches). By
> this definition many of the most valuable contributors to Hadoop,
> including Yahoo!, Cloudera, Facebook, etc are not using Hadoop. =A0Is
> that really the message we want to send? We expect the PMC to enforce
> this equally across all parties?
>
> It's a fact of life that companies and ops teams that support Hadoop
> need to patch the software before the PMC has time and/or will to vote
> on new releases. This is why HTTP and Subversion allow this. Putting a
> build of Hadoop that has 4 security patches applied into the same
> category as a product that has entirely re-worked the code and not
> gotten it checked into trunk does a major disservice to the people who
> contribute to and invest in the project.
>
> Thanks,
> Eli
>
>

From general-return-3795-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 01:09:10 2011
Return-Path: <general-return-3795-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5C56B4D9A
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 01:09:10 +0000 (UTC)
Received: (qmail 12460 invoked by uid 500); 16 Jun 2011 01:09:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12399 invoked by uid 500); 16 Jun 2011 01:09:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12391 invoked by uid 99); 16 Jun 2011 01:09:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 01:09:08 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 01:09:02 +0000
Received: by yxd5 with SMTP id 5so875250yxd.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 18:08:41 -0700 (PDT)
Received: by 10.150.207.17 with SMTP id e17mr324137ybg.243.1308186521132; Wed,
 15 Jun 2011 18:08:41 -0700 (PDT)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.150.158.15 with HTTP; Wed, 15 Jun 2011 18:08:21 -0700 (PDT)
In-Reply-To: <BANLkTi=5HhcyxaHSmqqdV4hCgSHSBkHyoA@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org> <BANLkTi=5HhcyxaHSmqqdV4hCgSHSBkHyoA@mail.gmail.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Wed, 15 Jun 2011 18:08:21 -0700
X-Google-Sender-Auth: yS1gGgFi_3w5aLBj4Dh8wawXXMY
Message-ID: <BANLkTinZvQfzQB5aF6UAzafXtw2FD5Xw2Q@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

What are you trying to say? Let's promote a brutal muscling macho
power vs. a stuffed yet adorable animal? Think of the children, man!

It's been an interesting race: so far 49 people have casted their
votes. I think this is by far most alive discussion here!
--
=A0 Take care,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616 =A06115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.

On Wed, Jun 15, 2011 at 17:57, Konstantin Shvachko <shv.hadoop@gmail.com> w=
rote:
> It should look something like this:
> http://api.ning.com/files/6tMSBWmZ67YKGzfS0NBp6Xt67YUi5L4ft6OZRsIrq-3EO40=
OP0mqSJs4b3*5QV0V7shmKnK44LzEGzNSvmDUFqCv6hirrSWX/1105100WC001.jpeg
>
> But given the choices: =A06 2 4 5 1 3
>
> I put 6 first because it's the only one having Apache logo on it.
> I think whatever choice wins the feather should be put on it.
>
> Thanks,
> --Konstantin
>
>
> On Tue, Jun 14, 2011 at 8:19 PM, Owen O'Malley <omalley@apache.org> wrote=
:
>
>> All,
>> =A0 We've had a wide range of entries for a powered by logo. I've put th=
em
>> all on a page, here:
>>
>> http://people.apache.org/~omalley/hadoop-powered-by/
>>
>> Since there are a lot of contenders and we only want a single round of
>> voting, let's use single transferable vote ( STV
>> http://en.wikipedia.org/wiki/Single_transferable_vote). The important
>> thing is to pick the images *IN ORDER* that you would like them.
>>
>> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>>
>> In other words, I want option 4 most and option 6 least. With STV, you
>> don't need to worry about voting for an unpopular choice since your vote
>> will automatically roll over to your next choice.
>>
>> -- Owen
>>
>>
>>
>

From general-return-3796-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 01:16:28 2011
Return-Path: <general-return-3796-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4EB4A4259
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 01:16:28 +0000 (UTC)
Received: (qmail 16957 invoked by uid 500); 16 Jun 2011 01:16:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16832 invoked by uid 500); 16 Jun 2011 01:16:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 13917 invoked by uid 99); 16 Jun 2011 01:09:42 -0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 69.147.107.21 is neither permitted nor denied by domain of anew@yahoo-inc.com)
From: Andreas Neumann <anew@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 15 Jun 2011 18:08:27 -0700
Subject: Re: [VOTE] Powered by Logo
Thread-Topic: [VOTE] Powered by Logo
Thread-Index: Acwrwedb5iIVAdakRXG1zmWhSuDRXg==
Message-ID: <CA1EA531.6787%anew@yahoo-inc.com>
In-Reply-To: <BANLkTi=5HhcyxaHSmqqdV4hCgSHSBkHyoA@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-MacOutlook/14.10.0.110310
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

2 4 5 6 1 3


From general-return-3797-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 01:19:34 2011
Return-Path: <general-return-3797-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E65B040A9
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 01:19:34 +0000 (UTC)
Received: (qmail 18822 invoked by uid 500); 16 Jun 2011 01:19:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18770 invoked by uid 500); 16 Jun 2011 01:19:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18762 invoked by uid 99); 16 Jun 2011 01:19:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 01:19:33 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 216.145.54.171 is neither permitted nor denied by domain of mattf@yahoo-inc.com)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 01:19:25 +0000
Received: from SP2-EX07CAS04.ds.corp.yahoo.com (sp2-ex07cas04.corp.sp2.yahoo.com [98.137.59.5])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5G1HcUe068922;
	Wed, 15 Jun 2011 18:17:38 -0700 (PDT)
Received: from SP2-EX07VS03.ds.corp.yahoo.com ([98.137.59.32]) by
 SP2-EX07CAS04.ds.corp.yahoo.com ([98.137.59.5]) with mapi; Wed, 15 Jun 2011
 18:17:37 -0700
From: Matthew Foley <mattf@yahoo-inc.com>
To: Eli Collins <eli@cloudera.com>,
        "general@hadoop.apache.org"
	<general@hadoop.apache.org>
CC: Matthew Foley <mattf@yahoo-inc.com>,
        "trademarks@apache.org"
	<trademarks@apache.org>
Date: Wed, 15 Jun 2011 18:17:36 -0700
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
Thread-Topic: [VOTE] Shall we adopt the "Defining Hadoop" page
Thread-Index: Acwrwy2suOVxJO+0S22r6Lb08blubA==
Message-ID: <EFB94DDE-9E2C-42D9-99C2-2FEB742535D1@yahoo-inc.com>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
 <BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com>
 <090F2993-B9AA-410F-AD99-7AC047FFC974@apache.org>
 <BANLkTi=fkguAtbfi9Wk9CPm8JmFa2Hti3Q@mail.gmail.com>
 <54668CDE-443D-4AF1-B17A-0497D0CDCC8D@yahoo-inc.com>
 <BANLkTimTGcEU29e_uLB52uc9CeuSkK__6w@mail.gmail.com>
In-Reply-To: <BANLkTimTGcEU29e_uLB52uc9CeuSkK__6w@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

I tend to agree with what I think you are saying, that=20
	* applying a small-number-of-patches that are
	* for high-severity-bug-fixes, and
	* have been Apache-Hadoop-committed=20
to an Apache Hadoop release should not demote the result to a "derived work=
".=20
However, if so many patches are applied that the result cannot be meaningfu=
lly=20
correlated with a specific Apache Hadoop release, then it probably has
become a derived work.

But how do we draw a meaningful line across that big gray area?  That's why=
 I'd like to
see specific text from one of the other projects you cited as an example.

Thanks,
--Matt


On Jun 15, 2011, at 6:02 PM, Eli Collins wrote:

On Wed, Jun 15, 2011 at 10:44 AM, Matthew Foley <mattf@yahoo-inc.com> wrote=
:
> Eli, you said:
>> Putting a build of Hadoop that has 4 security patches applied into the s=
ame
>> category as a product that has entirely re-worked the code and not
>> gotten it checked into trunk does a major disservice to the people who
>> contribute to and invest in the project.
>=20
> How would you phrase the distinction, so that it is clear and reasonably =
unambiguous
> for people who are not Hadoop developers?  Do the HTTP and Subversion pol=
icies
> draw this distinction, and if so could you please point us at the specifi=
c text, or
> copy that text to this thread?
>=20

I'll try to find it, this was told to me verbally a while back. Maybe
Roy can chime in.

Since there seems to be some confusion around distribution we should
make this explicit.  Some people are currently interpreting the
guidelines to say that if you patch an Apache Hadoop release yourself
then you're still running Apache Hadoop.  But if a vendor patches
Apache Hadoop for you then you're not running Apache Hadoop. How about
if a subcontractor patches Apache Hadoop for you, then is it Apache
Hadoop? This isn't sustainable.

Thanks,
Eli
> Thanks,
> --Matt
>=20
>=20
> On Jun 15, 2011, at 9:40 AM, Eli Collins wrote:
>=20
> On Tue, Jun 14, 2011 at 7:45 PM, Owen O'Malley <omalley@apache.org> wrote=
:
>>=20
>> On Jun 14, 2011, at 5:48 PM, Eli Collins wrote:
>>=20
>>> Wrt derivative works, it's not clear from the document, but I think we
>>> should explicitly adopt the policy of HTTPD and Subversion that
>>> backported patches from trunk and security fixes are permitted.
>>=20
>> Actually, the document is extremely clear that only Apache releases may =
be called Hadoop.
>>=20
>> There was a very long thread about why the rapidly expanding Hadoop-ecos=
ystem is leading to at lot of customer confusion about the different "versi=
ons" of Hadoop. We as the Hadoop project don't have the resources or the ne=
cessary compatibility test suite to test compatibility between the differen=
t sets of cherry picked patches. We also don't have time to ensure that all=
 of the 1,000's of patches applied to 0.20.2 in each of the many (10? 15?) =
different versions have been committed to trunk. Futhermore, under the Apac=
he license, a company Foo could claim that it is a cherry pick version of H=
adoop without releasing their source code that would enable verification.
>>=20
>> In summary,
>>  1. Hadoop is very successful.
>>  2. There are many different commercial products that are trying to use =
the Hadoop name.
>>  3. We can't check or enforce that the cherry pick versions are followin=
g the rules.
>>  4. We don't have a TCK like Java does to validate new versions are comp=
atible.
>>  5. By far the most fair way to ensure compatibility and fairness betwee=
n companies is that only Apache Hadoop releases may be called Hadoop.
>>=20
>> That said, a package that includes a small number (< 3) of security patc=
hes that haven't been released yet doesn't seem unreasonable.
>>=20
>=20
> I've spoken with ops teams at many companies,  I am not aware of
> anyone who runs an official release (with just 2 security patches). By
> this definition many of the most valuable contributors to Hadoop,
> including Yahoo!, Cloudera, Facebook, etc are not using Hadoop.  Is
> that really the message we want to send? We expect the PMC to enforce
> this equally across all parties?
>=20
> It's a fact of life that companies and ops teams that support Hadoop
> need to patch the software before the PMC has time and/or will to vote
> on new releases. This is why HTTP and Subversion allow this. Putting a
> build of Hadoop that has 4 security patches applied into the same
> category as a product that has entirely re-worked the code and not
> gotten it checked into trunk does a major disservice to the people who
> contribute to and invest in the project.
>=20
> Thanks,
> Eli
>=20
>=20


From general-return-3798-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 01:30:52 2011
Return-Path: <general-return-3798-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A78F44997
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 01:30:52 +0000 (UTC)
Received: (qmail 22421 invoked by uid 500); 16 Jun 2011 01:30:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22369 invoked by uid 500); 16 Jun 2011 01:30:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22359 invoked by uid 99); 16 Jun 2011 01:30:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 01:30:51 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of shv.hadoop@gmail.com designates 209.85.218.48 as permitted sender)
Received: from [209.85.218.48] (HELO mail-yi0-f48.google.com) (209.85.218.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 01:30:46 +0000
Received: by yic24 with SMTP id 24so887883yic.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 18:30:25 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=Lm0f4kUFo80BOjRLnV335BDD+lgBlf6NsHbuH+QfULg=;
        b=Co1fbKYTM5T4/BZlyom38Ims5D5qIVG5rZUPSNskyZgfAeS8U/yYS/uhnOwRyKibGB
         pdh+D63/98Jgr4l6p49sz1Gc1rRX18mcsIyk594Zl2UeGcPX9i0djj5PIFvxYKis3mZ0
         uVcb3OQST8cKKyRm47g2nVN7gWK96eoCVBpRA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=B2eiAD8M6F4zfR+u0giyFQjhQfjxsAm++hxq5ET3kr5DBgQd+jIpVkVvQuumaZqnTt
         fQVp3AYK2tDh0UH2vH/ax1mIpfHifBUZSh+fYCedRlOkwEGQT2U5TL/vBhqEmz5n8lbz
         nio6rriBjh2P0dXsGpMk+6a1dcFZOsqEhOSD4=
MIME-Version: 1.0
Received: by 10.236.184.33 with SMTP id r21mr342799yhm.11.1308187825433; Wed,
 15 Jun 2011 18:30:25 -0700 (PDT)
Received: by 10.147.113.16 with HTTP; Wed, 15 Jun 2011 18:30:25 -0700 (PDT)
In-Reply-To: <BANLkTinZvQfzQB5aF6UAzafXtw2FD5Xw2Q@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
	<BANLkTi=5HhcyxaHSmqqdV4hCgSHSBkHyoA@mail.gmail.com>
	<BANLkTinZvQfzQB5aF6UAzafXtw2FD5Xw2Q@mail.gmail.com>
Date: Wed, 15 Jun 2011 18:30:25 -0700
Message-ID: <BANLkTi=nP=XZ70TsJd+Dk=oDtcvgazsrYQ@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf305b0a5ea26bfd04a5ca3688

--20cf305b0a5ea26bfd04a5ca3688
Content-Type: text/plain; charset=ISO-8859-1

It's not macho, it's the power coming from Hadoop.
The feather would classy on that background too.
And children need exercise, especially those dealing with Hadoop.
--Konstantin

On Wed, Jun 15, 2011 at 6:08 PM, Konstantin Boudnik <cos@apache.org> wrote:

> What are you trying to say? Let's promote a brutal muscling macho
> power vs. a stuffed yet adorable animal? Think of the children, man!
>
> It's been an interesting race: so far 49 people have casted their
> votes. I think this is by far most alive discussion here!
> --
>   Take care,
> Konstantin (Cos) Boudnik
> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>
> Disclaimer: Opinions expressed in this email are those of the author,
> and do not necessarily represent the views of any company the author
> might be affiliated with at the moment of writing.
>
> On Wed, Jun 15, 2011 at 17:57, Konstantin Shvachko <shv.hadoop@gmail.com>
> wrote:
> > It should look something like this:
> >
> http://api.ning.com/files/6tMSBWmZ67YKGzfS0NBp6Xt67YUi5L4ft6OZRsIrq-3EO40OP0mqSJs4b3*5QV0V7shmKnK44LzEGzNSvmDUFqCv6hirrSWX/1105100WC001.jpeg
> >
> > But given the choices:  6 2 4 5 1 3
> >
> > I put 6 first because it's the only one having Apache logo on it.
> > I think whatever choice wins the feather should be put on it.
> >
> > Thanks,
> > --Konstantin
> >
> >
> > On Tue, Jun 14, 2011 at 8:19 PM, Owen O'Malley <omalley@apache.org>
> wrote:
> >
> >> All,
> >>   We've had a wide range of entries for a powered by logo. I've put them
> >> all on a page, here:
> >>
> >> http://people.apache.org/~omalley/hadoop-powered-by/
> >>
> >> Since there are a lot of contenders and we only want a single round of
> >> voting, let's use single transferable vote ( STV
> >> http://en.wikipedia.org/wiki/Single_transferable_vote). The important
> >> thing is to pick the images *IN ORDER* that you would like them.
> >>
> >> My vote (in order of course): 4, 1, 2, 3, 5, 6.
> >>
> >> In other words, I want option 4 most and option 6 least. With STV, you
> >> don't need to worry about voting for an unpopular choice since your vote
> >> will automatically roll over to your next choice.
> >>
> >> -- Owen
> >>
> >>
> >>
> >
>

--20cf305b0a5ea26bfd04a5ca3688--

From general-return-3799-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 01:48:26 2011
Return-Path: <general-return-3799-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BCD6E445B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 01:48:26 +0000 (UTC)
Received: (qmail 41022 invoked by uid 500); 16 Jun 2011 01:48:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40945 invoked by uid 500); 16 Jun 2011 01:48:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40937 invoked by uid 99); 16 Jun 2011 01:48:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 01:48:25 +0000
X-ASF-Spam-Status: No, hits=4.0 required=5.0
	tests=FREEMAIL_FROM,FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of kengoodhope@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 01:48:20 +0000
Received: by vws7 with SMTP id 7so1246690vws.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 18:47:59 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=X6GDRqwvnulNLZvH4XgefUI1LRr4vnv3HjZj6YAMhF0=;
        b=WkzenW77OiRkI6VO7wzBVLvDYFCi5MjHf32dLPA5KKkHybRnULHQk7RkPSZnKK1R5h
         zLQ0FNBaXLiTwedJvcY2m88/R8JJsEobXPVSfOQwvEa+bYaeHtz9jdbz46iEmb8dKhmK
         zgdl9iynLoMeQIHjImJMZJlt7LiURerjuGm+I=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=FPfFxN8WeEQ+B6JaEEb1SIlS7T0Kbu+Y1zesz4LwerwDhOj/Ms01XYdMkT86MocD02
         lVJIsguCQ8cR+vo3Ow2vpJzMNr4BXIp2oqmu5yYJODHgGO1vIF9HgkkPgPXFvwId2W5k
         iGw1mkRkMnK4mYb8dPR6f+xpsg1H9gO++88bk=
MIME-Version: 1.0
Received: by 10.52.92.132 with SMTP id cm4mr438359vdb.266.1308188879047; Wed,
 15 Jun 2011 18:47:59 -0700 (PDT)
Received: by 10.52.185.102 with HTTP; Wed, 15 Jun 2011 18:47:58 -0700 (PDT)
In-Reply-To: <BANLkTi=nP=XZ70TsJd+Dk=oDtcvgazsrYQ@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
	<BANLkTi=5HhcyxaHSmqqdV4hCgSHSBkHyoA@mail.gmail.com>
	<BANLkTinZvQfzQB5aF6UAzafXtw2FD5Xw2Q@mail.gmail.com>
	<BANLkTi=nP=XZ70TsJd+Dk=oDtcvgazsrYQ@mail.gmail.com>
Date: Wed, 15 Jun 2011 18:47:58 -0700
Message-ID: <BANLkTi=43_0VCb2zAqQ5ru7UdoCw6VnxkQ@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: Ken Goodhope <kengoodhope@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf307f33586f4a9304a5ca759b

--20cf307f33586f4a9304a5ca759b
Content-Type: text/plain; charset=ISO-8859-1

6,2,4

On Wed, Jun 15, 2011 at 6:30 PM, Konstantin Shvachko
<shv.hadoop@gmail.com>wrote:

> It's not macho, it's the power coming from Hadoop.
> The feather would classy on that background too.
> And children need exercise, especially those dealing with Hadoop.
> --Konstantin
>
> On Wed, Jun 15, 2011 at 6:08 PM, Konstantin Boudnik <cos@apache.org>
> wrote:
>
> > What are you trying to say? Let's promote a brutal muscling macho
> > power vs. a stuffed yet adorable animal? Think of the children, man!
> >
> > It's been an interesting race: so far 49 people have casted their
> > votes. I think this is by far most alive discussion here!
> > --
> >   Take care,
> > Konstantin (Cos) Boudnik
> > 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> >
> > Disclaimer: Opinions expressed in this email are those of the author,
> > and do not necessarily represent the views of any company the author
> > might be affiliated with at the moment of writing.
> >
> > On Wed, Jun 15, 2011 at 17:57, Konstantin Shvachko <shv.hadoop@gmail.com
> >
> > wrote:
> > > It should look something like this:
> > >
> >
> http://api.ning.com/files/6tMSBWmZ67YKGzfS0NBp6Xt67YUi5L4ft6OZRsIrq-3EO40OP0mqSJs4b3*5QV0V7shmKnK44LzEGzNSvmDUFqCv6hirrSWX/1105100WC001.jpeg
> > >
> > > But given the choices:  6 2 4 5 1 3
> > >
> > > I put 6 first because it's the only one having Apache logo on it.
> > > I think whatever choice wins the feather should be put on it.
> > >
> > > Thanks,
> > > --Konstantin
> > >
> > >
> > > On Tue, Jun 14, 2011 at 8:19 PM, Owen O'Malley <omalley@apache.org>
> > wrote:
> > >
> > >> All,
> > >>   We've had a wide range of entries for a powered by logo. I've put
> them
> > >> all on a page, here:
> > >>
> > >> http://people.apache.org/~omalley/hadoop-powered-by/
> > >>
> > >> Since there are a lot of contenders and we only want a single round of
> > >> voting, let's use single transferable vote ( STV
> > >> http://en.wikipedia.org/wiki/Single_transferable_vote). The important
> > >> thing is to pick the images *IN ORDER* that you would like them.
> > >>
> > >> My vote (in order of course): 4, 1, 2, 3, 5, 6.
> > >>
> > >> In other words, I want option 4 most and option 6 least. With STV, you
> > >> don't need to worry about voting for an unpopular choice since your
> vote
> > >> will automatically roll over to your next choice.
> > >>
> > >> -- Owen
> > >>
> > >>
> > >>
> > >
> >
>

--20cf307f33586f4a9304a5ca759b--

From general-return-3800-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 02:25:26 2011
Return-Path: <general-return-3800-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C1E8B62F1
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 02:25:26 +0000 (UTC)
Received: (qmail 69358 invoked by uid 500); 16 Jun 2011 02:25:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69300 invoked by uid 500); 16 Jun 2011 02:25:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 65111 invoked by uid 99); 16 Jun 2011 02:20:22 -0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of craig.russell@oracle.com designates 148.87.113.121 as permitted sender)
From: Craig L Russell <craig.russell@oracle.com>
To: trademarks@apache.org
In-Reply-To: <EFB94DDE-9E2C-42D9-99C2-2FEB742535D1@yahoo-inc.com>
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org> <BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com> <090F2993-B9AA-410F-AD99-7AC047FFC974@apache.org> <BANLkTi=fkguAtbfi9Wk9CPm8JmFa2Hti3Q@mail.gmail.com> <54668CDE-443D-4AF1-B17A-0497D0CDCC8D@yahoo-inc.com> <BANLkTimTGcEU29e_uLB52uc9CeuSkK__6w@mail.gmail.com> <EFB94DDE-9E2C-42D9-99C2-2FEB742535D1@yahoo-inc.com>
Message-Id: <24C673BE-B252-4643-BA96-EF6E4A9D978D@oracle.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Date: Wed, 15 Jun 2011 19:19:08 -0700
Cc: Eli Collins <eli@cloudera.com>,
        "general@hadoop.apache.org" <general@hadoop.apache.org>,
        Matthew Foley <mattf@yahoo-inc.com>
X-Mailer: Apple Mail (2.936)
X-Source-IP: rtcsinet22.oracle.com [66.248.204.30]
X-CT-RefId: str=0001.0A090207.4DF96828.0076:SCFSTAT5015188,ss=1,fgs=0
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Matthew,

I'm sorry I have to disagree.

If you change a bit in a work, it becomes a derived work. There's no  
"demotion" involved. Just a definition of derived work.

There's no ambiguity. Either you ship the bits that the Apache PMC has  
voted on as a release, or you change it (one bit) and it is no longer  
what the PMC has voted on. It's a derived work.

The rules for voting in Apache require that if you change a bit in an  
artifact, you can no longer count votes for the previous artifact.  
Because the new work is different. A new vote is required.

Not gray. Black and white.

Simple as that.

Craig

P.S. for the anthropologists, look at the history of Apache Derby and  
Sun JavaDB. Meaningful, specific example.

On Jun 15, 2011, at 6:17 PM, Matthew Foley wrote:

> I tend to agree with what I think you are saying, that
> 	* applying a small-number-of-patches that are
> 	* for high-severity-bug-fixes, and
> 	* have been Apache-Hadoop-committed
> to an Apache Hadoop release should not demote the result to a  
> "derived work".
> However, if so many patches are applied that the result cannot be  
> meaningfully
> correlated with a specific Apache Hadoop release, then it probably has
> become a derived work.
>
> But how do we draw a meaningful line across that big gray area?   
> That's why I'd like to
> see specific text from one of the other projects you cited as an  
> example.
>
> Thanks,
> --Matt
>
>
> On Jun 15, 2011, at 6:02 PM, Eli Collins wrote:
>
> On Wed, Jun 15, 2011 at 10:44 AM, Matthew Foley <mattf@yahoo- 
> inc.com> wrote:
>> Eli, you said:
>>> Putting a build of Hadoop that has 4 security patches applied into  
>>> the same
>>> category as a product that has entirely re-worked the code and not
>>> gotten it checked into trunk does a major disservice to the people  
>>> who
>>> contribute to and invest in the project.
>>
>> How would you phrase the distinction, so that it is clear and  
>> reasonably unambiguous
>> for people who are not Hadoop developers?  Do the HTTP and  
>> Subversion policies
>> draw this distinction, and if so could you please point us at the  
>> specific text, or
>> copy that text to this thread?
>>
>
> I'll try to find it, this was told to me verbally a while back. Maybe
> Roy can chime in.
>
> Since there seems to be some confusion around distribution we should
> make this explicit.  Some people are currently interpreting the
> guidelines to say that if you patch an Apache Hadoop release yourself
> then you're still running Apache Hadoop.  But if a vendor patches
> Apache Hadoop for you then you're not running Apache Hadoop. How about
> if a subcontractor patches Apache Hadoop for you, then is it Apache
> Hadoop? This isn't sustainable.
>
> Thanks,
> Eli
>> Thanks,
>> --Matt
>>
>>
>> On Jun 15, 2011, at 9:40 AM, Eli Collins wrote:
>>
>> On Tue, Jun 14, 2011 at 7:45 PM, Owen O'Malley <omalley@apache.org>  
>> wrote:
>>>
>>> On Jun 14, 2011, at 5:48 PM, Eli Collins wrote:
>>>
>>>> Wrt derivative works, it's not clear from the document, but I  
>>>> think we
>>>> should explicitly adopt the policy of HTTPD and Subversion that
>>>> backported patches from trunk and security fixes are permitted.
>>>
>>> Actually, the document is extremely clear that only Apache  
>>> releases may be called Hadoop.
>>>
>>> There was a very long thread about why the rapidly expanding  
>>> Hadoop-ecosystem is leading to at lot of customer confusion about  
>>> the different "versions" of Hadoop. We as the Hadoop project don't  
>>> have the resources or the necessary compatibility test suite to  
>>> test compatibility between the different sets of cherry picked  
>>> patches. We also don't have time to ensure that all of the 1,000's  
>>> of patches applied to 0.20.2 in each of the many (10? 15?)  
>>> different versions have been committed to trunk. Futhermore, under  
>>> the Apache license, a company Foo could claim that it is a cherry  
>>> pick version of Hadoop without releasing their source code that  
>>> would enable verification.
>>>
>>> In summary,
>>> 1. Hadoop is very successful.
>>> 2. There are many different commercial products that are trying to  
>>> use the Hadoop name.
>>> 3. We can't check or enforce that the cherry pick versions are  
>>> following the rules.
>>> 4. We don't have a TCK like Java does to validate new versions are  
>>> compatible.
>>> 5. By far the most fair way to ensure compatibility and fairness  
>>> between companies is that only Apache Hadoop releases may be  
>>> called Hadoop.
>>>
>>> That said, a package that includes a small number (< 3) of  
>>> security patches that haven't been released yet doesn't seem  
>>> unreasonable.
>>>
>>
>> I've spoken with ops teams at many companies,  I am not aware of
>> anyone who runs an official release (with just 2 security patches).  
>> By
>> this definition many of the most valuable contributors to Hadoop,
>> including Yahoo!, Cloudera, Facebook, etc are not using Hadoop.  Is
>> that really the message we want to send? We expect the PMC to enforce
>> this equally across all parties?
>>
>> It's a fact of life that companies and ops teams that support Hadoop
>> need to patch the software before the PMC has time and/or will to  
>> vote
>> on new releases. This is why HTTP and Subversion allow this.  
>> Putting a
>> build of Hadoop that has 4 security patches applied into the same
>> category as a product that has entirely re-worked the code and not
>> gotten it checked into trunk does a major disservice to the people  
>> who
>> contribute to and invest in the project.
>>
>> Thanks,
>> Eli
>>
>>
>

Craig L Russell
Secretary, Apache Software Foundation
Chair, OpenJPA PMC
clr@apache.org http://db.apache.org/jdo










From general-return-3801-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 02:43:15 2011
Return-Path: <general-return-3801-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6380C6A4F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 02:43:15 +0000 (UTC)
Received: (qmail 83077 invoked by uid 500); 16 Jun 2011 02:43:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82995 invoked by uid 500); 16 Jun 2011 02:43:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82984 invoked by uid 99); 16 Jun 2011 02:43:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 02:43:12 +0000
X-ASF-Spam-Status: No, hits=4.0 required=5.0
	tests=FREEMAIL_FROM,FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of wbsrvc@gmail.com designates 209.85.220.176 as permitted sender)
Received: from [209.85.220.176] (HELO mail-vx0-f176.google.com) (209.85.220.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 02:43:07 +0000
Received: by vxa37 with SMTP id 37so1300054vxa.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 19:42:46 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=f5N31Pg6jmWqc6hIHX/D9wLZDu+I4YPITUQB51NiFAw=;
        b=vydTt+libV+cNL2Orz3wy7nrJDBv+FTM8BzlXCvSBw1d05isdEizpNRi7tdR9+nyBs
         rmJOE33tFlhuLvhAStU81KRHCn652kJzlMWY82JaJ7ApWiyp2OLqSsmoF8/gKDTeFgNp
         EZ6NDxHKsM7CltT6WmdedTKdSjkhiKqvFv4XQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=uw51sSSDz5YclIPJ+A/Icz/rPrKZqnPmT7hMUeDa3+XdAW2pNkhxfRKRkXr+MUGlYF
         AUPo9cLDYOneGaM4jPKyx7l6FS580aki14jufmPSNBJVYru5xv4VFRnPzohB8BcAPO9Z
         ls4iVBkHCdADeByM7Nq9hUnAIY9nrGwjPCDUs=
MIME-Version: 1.0
Received: by 10.220.86.8 with SMTP id q8mr130426vcl.174.1308192166327; Wed, 15
 Jun 2011 19:42:46 -0700 (PDT)
Received: by 10.220.80.149 with HTTP; Wed, 15 Jun 2011 19:42:46 -0700 (PDT)
In-Reply-To: <BANLkTi=43_0VCb2zAqQ5ru7UdoCw6VnxkQ@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
	<BANLkTi=5HhcyxaHSmqqdV4hCgSHSBkHyoA@mail.gmail.com>
	<BANLkTinZvQfzQB5aF6UAzafXtw2FD5Xw2Q@mail.gmail.com>
	<BANLkTi=nP=XZ70TsJd+Dk=oDtcvgazsrYQ@mail.gmail.com>
	<BANLkTi=43_0VCb2zAqQ5ru7UdoCw6VnxkQ@mail.gmail.com>
Date: Wed, 15 Jun 2011 19:42:46 -0700
Message-ID: <BANLkTikRVLgK+p4cbCZHyuLdzEinfA-fcQ@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: web service <wbsrvc@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64764165f34d404a5cb3937

--0016e64764165f34d404a5cb3937
Content-Type: text/plain; charset=ISO-8859-1

2.

Yep, the feather is bit ....arghhhhh.

On Wed, Jun 15, 2011 at 6:47 PM, Ken Goodhope <kengoodhope@gmail.com> wrote:

> 6,2,4
>
> On Wed, Jun 15, 2011 at 6:30 PM, Konstantin Shvachko
> <shv.hadoop@gmail.com>wrote:
>
> > It's not macho, it's the power coming from Hadoop.
> > The feather would classy on that background too.
> > And children need exercise, especially those dealing with Hadoop.
> > --Konstantin
> >
> > On Wed, Jun 15, 2011 at 6:08 PM, Konstantin Boudnik <cos@apache.org>
> > wrote:
> >
> > > What are you trying to say? Let's promote a brutal muscling macho
> > > power vs. a stuffed yet adorable animal? Think of the children, man!
> > >
> > > It's been an interesting race: so far 49 people have casted their
> > > votes. I think this is by far most alive discussion here!
> > > --
> > >   Take care,
> > > Konstantin (Cos) Boudnik
> > > 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> > >
> > > Disclaimer: Opinions expressed in this email are those of the author,
> > > and do not necessarily represent the views of any company the author
> > > might be affiliated with at the moment of writing.
> > >
> > > On Wed, Jun 15, 2011 at 17:57, Konstantin Shvachko <
> shv.hadoop@gmail.com
> > >
> > > wrote:
> > > > It should look something like this:
> > > >
> > >
> >
> http://api.ning.com/files/6tMSBWmZ67YKGzfS0NBp6Xt67YUi5L4ft6OZRsIrq-3EO40OP0mqSJs4b3*5QV0V7shmKnK44LzEGzNSvmDUFqCv6hirrSWX/1105100WC001.jpeg
> > > >
> > > > But given the choices:  6 2 4 5 1 3
> > > >
> > > > I put 6 first because it's the only one having Apache logo on it.
> > > > I think whatever choice wins the feather should be put on it.
> > > >
> > > > Thanks,
> > > > --Konstantin
> > > >
> > > >
> > > > On Tue, Jun 14, 2011 at 8:19 PM, Owen O'Malley <omalley@apache.org>
> > > wrote:
> > > >
> > > >> All,
> > > >>   We've had a wide range of entries for a powered by logo. I've put
> > them
> > > >> all on a page, here:
> > > >>
> > > >> http://people.apache.org/~omalley/hadoop-powered-by/
> > > >>
> > > >> Since there are a lot of contenders and we only want a single round
> of
> > > >> voting, let's use single transferable vote ( STV
> > > >> http://en.wikipedia.org/wiki/Single_transferable_vote). The
> important
> > > >> thing is to pick the images *IN ORDER* that you would like them.
> > > >>
> > > >> My vote (in order of course): 4, 1, 2, 3, 5, 6.
> > > >>
> > > >> In other words, I want option 4 most and option 6 least. With STV,
> you
> > > >> don't need to worry about voting for an unpopular choice since your
> > vote
> > > >> will automatically roll over to your next choice.
> > > >>
> > > >> -- Owen
> > > >>
> > > >>
> > > >>
> > > >
> > >
> >
>

--0016e64764165f34d404a5cb3937--

From general-return-3802-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 02:53:01 2011
Return-Path: <general-return-3802-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A72486010
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 02:53:01 +0000 (UTC)
Received: (qmail 88412 invoked by uid 500); 16 Jun 2011 02:52:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88334 invoked by uid 500); 16 Jun 2011 02:52:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88196 invoked by uid 99); 16 Jun 2011 02:52:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 02:52:57 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 02:52:52 +0000
Received: by vws7 with SMTP id 7so1287922vws.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 19:52:31 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:subject:mime-version:content-type:from
         :in-reply-to:date:cc:content-transfer-encoding:message-id:references
         :to:x-mailer;
        bh=uDPiaxooo03EhmoQ2ACRZ6OXBUj6xfQPhpFBQPiv21w=;
        b=sW+Mn/YZJsCD/uL+9Z3xkqE7bkLXiAsfYLmd6d+3MdLrs3cuxbrjc50B05rVcw2ltF
         eK22mWdCNlKW+ZZnGFzHfdAWW05KVh9F6ipRyPSafDFLJrz+oXfHbpS8SmjwkpG6T7yu
         DuAFLLHwL44GexBZbsJiEztO1ul7xNKG+FHxU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=subject:mime-version:content-type:from:in-reply-to:date:cc
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=jyWEvTRYddtE5UlbaMEezQwU/MdjItHfT8Ro3jj6i2UumPG9FFHdrKnAUNB8W9R0mE
         yTfgmIAx+m7xjmnc1qeM77p3asJOvNfjYMzRJXWZkUzDC17cvpR2oqiyvtJQo+WvvUbE
         wF3l8MlINYFFWAb9p/Ie6cO8C1KTvarmm5Rm0=
Received: by 10.52.113.138 with SMTP id iy10mr519769vdb.55.1308192751385;
        Wed, 15 Jun 2011 19:52:31 -0700 (PDT)
Received: from [10.172.2.121] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id e14sm659792vby.6.2011.06.15.19.52.27
        (version=SSLv3 cipher=OTHER);
        Wed, 15 Jun 2011 19:52:30 -0700 (PDT)
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
Mime-Version: 1.0 (Apple Message framework v1084)
Content-Type: text/plain; charset=us-ascii
From: Ian Holsman <ian@holsman.net>
In-Reply-To: <24C673BE-B252-4643-BA96-EF6E4A9D978D@oracle.com>
Date: Thu, 16 Jun 2011 12:52:24 +1000
Cc: Eli Collins <eli@cloudera.com>,
 "general@hadoop.apache.org" <general@hadoop.apache.org>,
 Matthew Foley <mattf@yahoo-inc.com>
Content-Transfer-Encoding: quoted-printable
Message-Id: <57051FE9-4CC3-4997-B7B7-D8B62DBC377C@Holsman.net>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org> <BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com> <090F2993-B9AA-410F-AD99-7AC047FFC974@apache.org> <BANLkTi=fkguAtbfi9Wk9CPm8JmFa2Hti3Q@mail.gmail.com> <54668CDE-443D-4AF1-B17A-0497D0CDCC8D@yahoo-inc.com> <BANLkTimTGcEU29e_uLB52uc9CeuSkK__6w@mail.gmail.com> <EFB94DDE-9E2C-42D9-99C2-2FEB742535D1@yahoo-inc.com> <24C673BE-B252-4643-BA96-EF6E4A9D978D@oracle.com>
To: trademarks@apache.org
X-Mailer: Apple Mail (2.1084)

So to second a point here.

We are not saying you can't patch your distribution, add your own =
features, share it with your friends, or do whatever you want to the =
code.=20
all we're saying is that you can't call that 'Apache Hadoop'.


On Jun 16, 2011, at 12:19 PM, Craig L Russell wrote:

> Hi Matthew,
>=20
> I'm sorry I have to disagree.
>=20
> If you change a bit in a work, it becomes a derived work. There's no =
"demotion" involved. Just a definition of derived work.
>=20
> There's no ambiguity. Either you ship the bits that the Apache PMC has =
voted on as a release, or you change it (one bit) and it is no longer =
what the PMC has voted on. It's a derived work.
>=20
> The rules for voting in Apache require that if you change a bit in an =
artifact, you can no longer count votes for the previous artifact. =
Because the new work is different. A new vote is required.
>=20
> Not gray. Black and white.
>=20
> Simple as that.
>=20
> Craig
>=20
> P.S. for the anthropologists, look at the history of Apache Derby and =
Sun JavaDB. Meaningful, specific example.
>=20
> On Jun 15, 2011, at 6:17 PM, Matthew Foley wrote:
>=20
>> I tend to agree with what I think you are saying, that
>> 	* applying a small-number-of-patches that are
>> 	* for high-severity-bug-fixes, and
>> 	* have been Apache-Hadoop-committed
>> to an Apache Hadoop release should not demote the result to a =
"derived work".
>> However, if so many patches are applied that the result cannot be =
meaningfully
>> correlated with a specific Apache Hadoop release, then it probably =
has
>> become a derived work.
>>=20
>> But how do we draw a meaningful line across that big gray area?  =
That's why I'd like to
>> see specific text from one of the other projects you cited as an =
example.
>>=20
>> Thanks,
>> --Matt
>>=20
>>=20
>> On Jun 15, 2011, at 6:02 PM, Eli Collins wrote:
>>=20
>> On Wed, Jun 15, 2011 at 10:44 AM, Matthew Foley <mattf@yahoo-inc.com> =
wrote:
>>> Eli, you said:
>>>> Putting a build of Hadoop that has 4 security patches applied into =
the same
>>>> category as a product that has entirely re-worked the code and not
>>>> gotten it checked into trunk does a major disservice to the people =
who
>>>> contribute to and invest in the project.
>>>=20
>>> How would you phrase the distinction, so that it is clear and =
reasonably unambiguous
>>> for people who are not Hadoop developers?  Do the HTTP and =
Subversion policies
>>> draw this distinction, and if so could you please point us at the =
specific text, or
>>> copy that text to this thread?
>>>=20
>>=20
>> I'll try to find it, this was told to me verbally a while back. Maybe
>> Roy can chime in.
>>=20
>> Since there seems to be some confusion around distribution we should
>> make this explicit.  Some people are currently interpreting the
>> guidelines to say that if you patch an Apache Hadoop release yourself
>> then you're still running Apache Hadoop.  But if a vendor patches
>> Apache Hadoop for you then you're not running Apache Hadoop. How =
about
>> if a subcontractor patches Apache Hadoop for you, then is it Apache
>> Hadoop? This isn't sustainable.
>>=20
>> Thanks,
>> Eli
>>> Thanks,
>>> --Matt
>>>=20
>>>=20
>>> On Jun 15, 2011, at 9:40 AM, Eli Collins wrote:
>>>=20
>>> On Tue, Jun 14, 2011 at 7:45 PM, Owen O'Malley <omalley@apache.org> =
wrote:
>>>>=20
>>>> On Jun 14, 2011, at 5:48 PM, Eli Collins wrote:
>>>>=20
>>>>> Wrt derivative works, it's not clear from the document, but I =
think we
>>>>> should explicitly adopt the policy of HTTPD and Subversion that
>>>>> backported patches from trunk and security fixes are permitted.
>>>>=20
>>>> Actually, the document is extremely clear that only Apache releases =
may be called Hadoop.
>>>>=20
>>>> There was a very long thread about why the rapidly expanding =
Hadoop-ecosystem is leading to at lot of customer confusion about the =
different "versions" of Hadoop. We as the Hadoop project don't have the =
resources or the necessary compatibility test suite to test =
compatibility between the different sets of cherry picked patches. We =
also don't have time to ensure that all of the 1,000's of patches =
applied to 0.20.2 in each of the many (10? 15?) different versions have =
been committed to trunk. Futhermore, under the Apache license, a company =
Foo could claim that it is a cherry pick version of Hadoop without =
releasing their source code that would enable verification.
>>>>=20
>>>> In summary,
>>>> 1. Hadoop is very successful.
>>>> 2. There are many different commercial products that are trying to =
use the Hadoop name.
>>>> 3. We can't check or enforce that the cherry pick versions are =
following the rules.
>>>> 4. We don't have a TCK like Java does to validate new versions are =
compatible.
>>>> 5. By far the most fair way to ensure compatibility and fairness =
between companies is that only Apache Hadoop releases may be called =
Hadoop.
>>>>=20
>>>> That said, a package that includes a small number (< 3) of security =
patches that haven't been released yet doesn't seem unreasonable.
>>>>=20
>>>=20
>>> I've spoken with ops teams at many companies,  I am not aware of
>>> anyone who runs an official release (with just 2 security patches). =
By
>>> this definition many of the most valuable contributors to Hadoop,
>>> including Yahoo!, Cloudera, Facebook, etc are not using Hadoop.  Is
>>> that really the message we want to send? We expect the PMC to =
enforce
>>> this equally across all parties?
>>>=20
>>> It's a fact of life that companies and ops teams that support Hadoop
>>> need to patch the software before the PMC has time and/or will to =
vote
>>> on new releases. This is why HTTP and Subversion allow this. Putting =
a
>>> build of Hadoop that has 4 security patches applied into the same
>>> category as a product that has entirely re-worked the code and not
>>> gotten it checked into trunk does a major disservice to the people =
who
>>> contribute to and invest in the project.
>>>=20
>>> Thanks,
>>> Eli
>>>=20
>>>=20
>>=20
>=20
> Craig L Russell
> Secretary, Apache Software Foundation
> Chair, OpenJPA PMC
> clr@apache.org http://db.apache.org/jdo
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20
>=20

--
Ian Holsman
Ian@Holsman.net
PH: +1-703 879-3128 AOLIM: ianholsman Skype:iholsman

If you can believe in your power to do great things, you will. -- =
Michael Berg


From general-return-3803-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 03:04:56 2011
Return-Path: <general-return-3803-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B54E368ED
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 03:04:56 +0000 (UTC)
Received: (qmail 95321 invoked by uid 500); 16 Jun 2011 03:04:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95266 invoked by uid 500); 16 Jun 2011 03:04:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95257 invoked by uid 99); 16 Jun 2011 03:04:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 03:04:54 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.218.48] (HELO mail-yi0-f48.google.com) (209.85.218.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 03:04:49 +0000
Received: by yic24 with SMTP id 24so928488yic.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 20:04:28 -0700 (PDT)
Received: by 10.150.229.11 with SMTP id b11mr420302ybh.300.1308193467217; Wed,
 15 Jun 2011 20:04:27 -0700 (PDT)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.150.158.15 with HTTP; Wed, 15 Jun 2011 20:04:07 -0700 (PDT)
In-Reply-To: <BANLkTi=nP=XZ70TsJd+Dk=oDtcvgazsrYQ@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
 <BANLkTi=5HhcyxaHSmqqdV4hCgSHSBkHyoA@mail.gmail.com> <BANLkTinZvQfzQB5aF6UAzafXtw2FD5Xw2Q@mail.gmail.com>
 <BANLkTi=nP=XZ70TsJd+Dk=oDtcvgazsrYQ@mail.gmail.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Wed, 15 Jun 2011 20:04:07 -0700
X-Google-Sender-Auth: kM0Bmz25vTFIgz3_PGuij8XzwAM
Message-ID: <BANLkTinu9aRakCVvPDuHSJ3nC7+nSdveQg@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Wed, Jun 15, 2011 at 18:30, Konstantin Shvachko <shv.hadoop@gmail.com> w=
rote:
> It's not macho, it's the power coming from Hadoop.
> The feather would classy on that background too.
> And children need exercise, especially those dealing with Hadoop.

Yeah, lotta heavy lifting here for sure...

> --Konstantin
>
> On Wed, Jun 15, 2011 at 6:08 PM, Konstantin Boudnik <cos@apache.org> wrot=
e:
>
>> What are you trying to say? Let's promote a brutal muscling macho
>> power vs. a stuffed yet adorable animal? Think of the children, man!
>>
>> It's been an interesting race: so far 49 people have casted their
>> votes. I think this is by far most alive discussion here!
>> --
>> =A0 Take care,
>> Konstantin (Cos) Boudnik
>> 2CAC 8312 4870 D885 8616 =A06115 220F 6980 1F27 E622
>>
>> Disclaimer: Opinions expressed in this email are those of the author,
>> and do not necessarily represent the views of any company the author
>> might be affiliated with at the moment of writing.
>>
>> On Wed, Jun 15, 2011 at 17:57, Konstantin Shvachko <shv.hadoop@gmail.com=
>
>> wrote:
>> > It should look something like this:
>> >
>> http://api.ning.com/files/6tMSBWmZ67YKGzfS0NBp6Xt67YUi5L4ft6OZRsIrq-3EO4=
0OP0mqSJs4b3*5QV0V7shmKnK44LzEGzNSvmDUFqCv6hirrSWX/1105100WC001.jpeg
>> >
>> > But given the choices: =A06 2 4 5 1 3
>> >
>> > I put 6 first because it's the only one having Apache logo on it.
>> > I think whatever choice wins the feather should be put on it.
>> >
>> > Thanks,
>> > --Konstantin
>> >
>> >
>> > On Tue, Jun 14, 2011 at 8:19 PM, Owen O'Malley <omalley@apache.org>
>> wrote:
>> >
>> >> All,
>> >> =A0 We've had a wide range of entries for a powered by logo. I've put=
 them
>> >> all on a page, here:
>> >>
>> >> http://people.apache.org/~omalley/hadoop-powered-by/
>> >>
>> >> Since there are a lot of contenders and we only want a single round o=
f
>> >> voting, let's use single transferable vote ( STV
>> >> http://en.wikipedia.org/wiki/Single_transferable_vote). The important
>> >> thing is to pick the images *IN ORDER* that you would like them.
>> >>
>> >> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>> >>
>> >> In other words, I want option 4 most and option 6 least. With STV, yo=
u
>> >> don't need to worry about voting for an unpopular choice since your v=
ote
>> >> will automatically roll over to your next choice.
>> >>
>> >> -- Owen
>> >>
>> >>
>> >>
>> >
>>
>

From general-return-3804-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 03:40:23 2011
Return-Path: <general-return-3804-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5E837672D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 03:40:23 +0000 (UTC)
Received: (qmail 19372 invoked by uid 500); 16 Jun 2011 03:40:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18836 invoked by uid 500); 16 Jun 2011 03:40:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 13545 invoked by uid 99); 16 Jun 2011 03:32:48 -0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mpdm.kyoji@googlemail.com designates 209.85.215.176 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=rAx0iI+Rd46Xpwu46Y7A/vFAiPBWa3v+M3EVdzos+vQ=;
        b=pRn2V+1S5quEdCshx+DGbqFRiyOZqm49KTujoPGBBLLxBCp0y5eEYLQDP/WaYpWt1I
         0fEbXY9fiWZGZK2JpruLAawFv5CHQld79rvN783A484FjHFvZ99WTEYKhx9kj0Ws+0/D
         uNvMJh32EN6UO1cXc9YFCWzX6rqCD/Aop6z6g=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=iVIDLYAkJEO70QeTiOXOAEqMDAiUG/GmF5DvyauH0dOqhepR9kFWKgscC+3h05+/12
         AEZ3T8CT5aoVA2OEU3jMx0YwHXNeCl04LgiLKUPPw1/4VFpxBc2XjS6ezjZChE51bKTk
         /Np5GtNyGamXy3XQni9BozzJktTfEibLb4e9s=
MIME-Version: 1.0
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
Date: Thu, 16 Jun 2011 12:32:21 +0900
Message-ID: <BANLkTinT5bF8eSdyoFOW633dM5b19dPRvg@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: Kyoji KATO <mpdm.kyoji@googlemail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015174c45beb990ab04a5cbea73

--0015174c45beb990ab04a5cbea73
Content-Type: text/plain; charset=ISO-8859-1

Hey all,

My vote.. 6,3,5,4,2,1

Thanks,
Kyoji KATO

On 15 June 2011 12:19, Owen O'Malley <omalley@apache.org> wrote:

> All,
>   We've had a wide range of entries for a powered by logo. I've put them
> all on a page, here:
>
> http://people.apache.org/~omalley/hadoop-powered-by/
>
> Since there are a lot of contenders and we only want a single round of
> voting, let's use single transferable vote ( STV
> http://en.wikipedia.org/wiki/Single_transferable_vote). The important
> thing is to pick the images *IN ORDER* that you would like them.
>
> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>
> In other words, I want option 4 most and option 6 least. With STV, you
> don't need to worry about voting for an unpopular choice since your vote
> will automatically roll over to your next choice.
>
> -- Owen
>
>
>

--0015174c45beb990ab04a5cbea73--

From general-return-3805-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 04:25:06 2011
Return-Path: <general-return-3805-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 59B4769F3
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 04:25:06 +0000 (UTC)
Received: (qmail 40631 invoked by uid 500); 16 Jun 2011 04:25:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40428 invoked by uid 500); 16 Jun 2011 04:25:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40419 invoked by uid 99); 16 Jun 2011 04:25:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 04:25:03 +0000
X-ASF-Spam-Status: No, hits=-7.0 required=5.0
	tests=RCVD_IN_DNSWL_HI,RCVD_IN_RP_SAFE
X-Spam-Check-By: apache.org
Received-SPF: unknown (athena.apache.org: error in processing during lookup of jrottinghuis@ebay.com)
Received: from [216.33.244.6] (HELO rhv-mipot-001.corp.ebay.com) (216.33.244.6)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 04:24:58 +0000
DomainKey-Signature: s=corp; d=ebay.com; c=nofws; q=dns;
  h=X-EBay-Corp:X-IronPort-AV:Received:Received:From:To:CC:
   Date:Subject:Thread-Topic:Thread-Index:Message-ID:
   References:In-Reply-To:Accept-Language:Content-Language:
   X-MS-Has-Attach:X-MS-TNEF-Correlator:acceptlanguage:
   x-ems-proccessed:x-ems-stamp:Content-Type:
   Content-Transfer-Encoding:MIME-Version:X-CFilter;
  b=NH8qeEcxcUrwDJpX2CppzcRuW+PUe1VUeej6TFjXA7tXluCDUU6DK4dm
   qlVFn+/D8a061x/TKynFAfvfzDVkwrXeTlHABpzb5vdRlmkEm9wN601xb
   NaToqT8MFAe2Id6;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=ebay.com; i=jrottinghuis@ebay.com; q=dns/txt; s=corp;
  t=1308198298; x=1339734298;
  h=from:to:cc:date:subject:message-id:references:
   in-reply-to:content-transfer-encoding:mime-version;
  bh=jAD3vEEk9nusIGR7BIFifk+wgKVWC73O3AmZ3MNHXuU=;
  b=VIhQRgqi+DjVd1btvYnIsPAkYSMD03xVEe534ELmLUZovTncynWTKQ5T
   Fhmj/Wxhmlt0z7Fg4rf7Z0NV1vMa2YvC+fiNTK4dd52wmDRTvNQVmmM/k
   yV+62zStvch4Wux;
X-EBay-Corp: Yes
X-IronPort-AV: E=Sophos;i="4.65,373,1304319600"; 
   d="scan'208";a="21223833"
Received: from rhv-vtenf-002.corp.ebay.com (HELO RHV-MEXHT-002.corp.ebay.com) ([10.112.113.53])
  by rhv-mipot-001.corp.ebay.com with ESMTP; 15 Jun 2011 21:24:35 -0700
Received: from RHV-MEXMS-002.corp.ebay.com ([10.245.17.115]) by
 RHV-MEXHT-002.corp.ebay.com ([10.245.24.101]) with mapi; Wed, 15 Jun 2011
 21:24:35 -0700
From: "Rottinghuis, Joep" <jrottinghuis@ebay.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: Apache Brand Management <trademarks@apache.org>
Date: Wed, 15 Jun 2011 21:24:34 -0700
Subject: RE: [VOTE] Shall we adopt the "Defining Hadoop" page
Thread-Topic: [VOTE] Shall we adopt the "Defining Hadoop" page
Thread-Index: AcwreJbFqZl9y8X4TmqfGatLTE6gBgAYkcpg
Message-ID: <F4FCE59AE46B2243922F224426BC869B01EC996726@RHV-MEXMS-002.corp.ebay.com>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
	<BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com>
	<FBF55C01-D66A-44C0-9252-D54E7CAB103D@apache.org>
	<BANLkTi=BWPUEoqD3L98A9AXLkbafZs15Zw@mail.gmail.com>
	<92D32E6E-3B65-464F-B91B-4C30EAB2642C@apache.org>,<BANLkTi=Gj-XvME7EyHYqPQaUwGNJusr17g@mail.gmail.com>
In-Reply-To: <BANLkTi=Gj-XvME7EyHYqPQaUwGNJusr17g@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
x-ems-proccessed: 10SqDH0iR7ekR7SRpKqm5A==
x-ems-stamp: iqa3cp3DvxV5uxeg0v2OYA==
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-CFilter: Scanned

It does make sense to me to distinguish between the case when a company see=
ks to benefit from using the Hadoop name for their product and the case whe=
n a company uses Hadoop internally with some minor patches.

For example: large company creates a game-show playing appliance and explai=
ns that they have used Hadoop for some of the learning tasks. Not allowed i=
f they applied more than 3 patches?

Or: company claims they have a large Hadoop deployment and are looking for =
developers to help them with their Hadoop development work is not allowed? =
What's the alternative? Wanted: Powered by Apache=99 Hadoop=99 developers?

Also, if thousands of changes are packaged together into one giant patch, i=
s that allowed?

Perhaps a similarity index (such as used by Git to determine if two files a=
re similar enough to be considered a rename) would make sense? If 98% of th=
e code is the same, would it be Hadoop if used internally and not sold/mark=
etted as a product?

Cheers,

Joep

________________________________________
From: Eli Collins [eli@cloudera.com]
Sent: Wednesday, June 15, 2011 9:23 AM
To: general@hadoop.apache.org
Cc: Apache Brand Management
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page

On Tue, Jun 14, 2011 at 7:46 PM, Allen Wittenauer <aw@apache.org> wrote:
>
> On Jun 14, 2011, at 6:45 PM, Eli Collins wrote:
>> Are we really going to go after all the web companies that patch in an
>> enhancement to their current Hadoop build and tell them to stop saying
>> that they are using Hadoop?  You've patched Hadoop many times, should
>> your employer not be able to say they use Hadoop?  I'm -1 on a
>> proposal that does this.
>
>        I think there is a big difference between some company that uses H=
adoop with some patches internally and a company that puts out a distributi=
on for others to use, usually for-profit.

The wiki makes no such distinction. The PMC will apply the rules
equally to all parties.

According to Owen's email if you are using a release of Apache Hadoop
and have applied more than 2 security patches or any backports you are
not using Hadoop.

Thanks,
Eli=

From general-return-3806-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 04:31:10 2011
Return-Path: <general-return-3806-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2161E6F70
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 04:31:10 +0000 (UTC)
Received: (qmail 42923 invoked by uid 500); 16 Jun 2011 04:31:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42846 invoked by uid 500); 16 Jun 2011 04:31:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42837 invoked by uid 99); 16 Jun 2011 04:31:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 04:31:06 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 04:31:01 +0000
Received: by bwz8 with SMTP id 8so1725363bwz.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 21:30:40 -0700 (PDT)
Received: by 10.204.19.74 with SMTP id z10mr298670bka.183.1308198640178; Wed,
 15 Jun 2011 21:30:40 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.79 with HTTP; Wed, 15 Jun 2011 21:30:20 -0700 (PDT)
In-Reply-To: <24C673BE-B252-4643-BA96-EF6E4A9D978D@oracle.com>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
 <BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com> <090F2993-B9AA-410F-AD99-7AC047FFC974@apache.org>
 <BANLkTi=fkguAtbfi9Wk9CPm8JmFa2Hti3Q@mail.gmail.com> <54668CDE-443D-4AF1-B17A-0497D0CDCC8D@yahoo-inc.com>
 <BANLkTimTGcEU29e_uLB52uc9CeuSkK__6w@mail.gmail.com> <EFB94DDE-9E2C-42D9-99C2-2FEB742535D1@yahoo-inc.com>
 <24C673BE-B252-4643-BA96-EF6E4A9D978D@oracle.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Wed, 15 Jun 2011 21:30:20 -0700
Message-ID: <BANLkTikqc18JbPS1XTiysXtH5XtpsBu51w@mail.gmail.com>
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
To: general@hadoop.apache.org
Cc: trademarks@apache.org, Eli Collins <eli@cloudera.com>, 
	Matthew Foley <mattf@yahoo-inc.com>
Content-Type: multipart/alternative; boundary=00032555a47e3e57ab04a5ccbbd1

--00032555a47e3e57ab04a5ccbbd1
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Jun 15, 2011 at 7:19 PM, Craig L Russell
<craig.russell@oracle.com>wrote:

> There's no ambiguity. Either you ship the bits that the Apache PMC has
> voted on as a release, or you change it (one bit) and it is no longer what
> the PMC has voted on. It's a derived work.
>
> The rules for voting in Apache require that if you change a bit in an
> artifact, you can no longer count votes for the previous artifact. Because
> the new work is different. A new vote is required.
>

Sorry, but this is just silly. Are you telling me that the httpd package in
Ubuntu isn't Apache httpd? It has 43 patches applied. Tomcat6 has 17. I'm
sure every other commonly used piece of software bundled with ubuntu has
been patched, too. I don't see them calling their packages "Ubuntu HTTP
server powered by Apache HTTPD". It's just httpd.

The httpd in RHEL 5 is the same way. In fact they even provide some nice
metadata in their patches, for example:
httpd-2.0.48-release.patch:Upstream-Status: vendor-specific change
httpd-2.1.10-apctl.patch:Upstream-Status: Vendor-specific changes for better
initscript integration

To me, this is a good thing: allowing vendors to redistribute the software
with some modifications makes it much more accessible to users and
businesses alike, and that's part of why Hadoop has had so much success. So
long as we require the vendors to upstream those modifications back to the
ASF, we get the benefits of these contributions back in the community and
everyone should be happy.

-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

--00032555a47e3e57ab04a5ccbbd1--

From general-return-3807-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 04:47:37 2011
Return-Path: <general-return-3807-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B95EC61EF
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 04:47:37 +0000 (UTC)
Received: (qmail 54872 invoked by uid 500); 16 Jun 2011 04:47:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54804 invoked by uid 500); 16 Jun 2011 04:47:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54788 invoked by uid 99); 16 Jun 2011 04:47:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 04:47:34 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 04:47:29 +0000
Received: by vws7 with SMTP id 7so1348622vws.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 21:47:08 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:subject:mime-version:content-type:from
         :in-reply-to:date:cc:content-transfer-encoding:message-id:references
         :to:x-mailer;
        bh=SrWbuX7UU81WvF60WW5bbEEZELJTZwTR4tn5h+vmmGo=;
        b=VeAthUr8EoYU1JMmnerkqoZeEuI1UdN17IZ15ZOC69lGfT1yqFMF04dJ8bDzMTCMvj
         ZW93X97i+ovjeci8KcEfNfOhSgyvaufnMehU7bkinlSYc1JmnAsjbWiAwClWh8VgeO+p
         kRP9qMxpaIHhKIVw8IRoFj35JePZ+2vindU8w=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=subject:mime-version:content-type:from:in-reply-to:date:cc
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=RDmZjX1m1p8wZ/7bGXWcFhK8EL5dymOoUusb26AiRUCe/MXJnmoGbJDw8xJ9rGjMvj
         qR+FyQON2UMZufdb0M6/qXh5KHKj5GAZ/DFmLPRy5epmCXJW+uxcXW127IIRkWSpRwVf
         Lrls13vJJJYMbUzZVWk0StdKUL1x4NwJ4sERw=
Received: by 10.52.109.101 with SMTP id hr5mr583030vdb.251.1308199628361;
        Wed, 15 Jun 2011 21:47:08 -0700 (PDT)
Received: from [10.172.2.121] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id l15sm450639vdt.22.2011.06.15.21.47.05
        (version=TLSv1/SSLv3 cipher=OTHER);
        Wed, 15 Jun 2011 21:47:07 -0700 (PDT)
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
Mime-Version: 1.0 (Apple Message framework v1084)
Content-Type: text/plain; charset=us-ascii
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <BANLkTikqc18JbPS1XTiysXtH5XtpsBu51w@mail.gmail.com>
Date: Thu, 16 Jun 2011 14:47:01 +1000
Cc: trademarks@apache.org,
 Eli Collins <eli@cloudera.com>,
 Matthew Foley <mattf@yahoo-inc.com>
Content-Transfer-Encoding: quoted-printable
Message-Id: <61D08301-F6A9-4B9B-9B78-60D7D3B731A4@Holsman.NET>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org> <BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com> <090F2993-B9AA-410F-AD99-7AC047FFC974@apache.org> <BANLkTi=fkguAtbfi9Wk9CPm8JmFa2Hti3Q@mail.gmail.com> <54668CDE-443D-4AF1-B17A-0497D0CDCC8D@yahoo-inc.com> <BANLkTimTGcEU29e_uLB52uc9CeuSkK__6w@mail.gmail.com> <EFB94DDE-9E2C-42D9-99C2-2FEB742535D1@yahoo-inc.com> <24C673BE-B252-4643-BA96-EF6E4A9D978D@oracle.com> <BANLkTikqc18JbPS1XTiysXtH5XtpsBu51w@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)


On Jun 16, 2011, at 2:30 PM, Todd Lipcon wrote:

> On Wed, Jun 15, 2011 at 7:19 PM, Craig L Russell
> <craig.russell@oracle.com>wrote:
>=20
>> There's no ambiguity. Either you ship the bits that the Apache PMC =
has
>> voted on as a release, or you change it (one bit) and it is no longer =
what
>> the PMC has voted on. It's a derived work.
>>=20
>> The rules for voting in Apache require that if you change a bit in an
>> artifact, you can no longer count votes for the previous artifact. =
Because
>> the new work is different. A new vote is required.
>>=20
>=20
> Sorry, but this is just silly. Are you telling me that the httpd =
package in
> Ubuntu isn't Apache httpd? It has 43 patches applied. Tomcat6 has 17. =
I'm
> sure every other commonly used piece of software bundled with ubuntu =
has
> been patched, too. I don't see them calling their packages "Ubuntu =
HTTP
> server powered by Apache HTTPD". It's just httpd.
>=20

well.. for RHEL in the early days of httpd, a configuration that ran on =
RHEL
would not work on the 'vanilla' httpd.

(they implemented a feature called include which could take a wildcard, =
which wasn't in the released version of httpd at the time)

even today.. I can't build redis on my mac as I am using GNU's libtool =
instead of the one packaged on the mac.=20
http://code.google.com/p/redis/issues/detail?id=3D443

so yes .. even a simple patch makes it derived, because it is different.


> -Todd
> --=20
> Todd Lipcon
> Software Engineer, Cloudera


From general-return-3808-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 04:47:48 2011
Return-Path: <general-return-3808-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2A4FB631C
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 04:47:48 +0000 (UTC)
Received: (qmail 56373 invoked by uid 500); 16 Jun 2011 04:47:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56320 invoked by uid 500); 16 Jun 2011 04:47:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 44456 invoked by uid 99); 16 Jun 2011 04:32:58 -0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of uvsaradhi@gmail.com designates 209.85.213.176 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=Xwput261sJ25nqLnajjVf5EF9nuyhbJl0rAs6DBmu7c=;
        b=n7Hzyn+LvFhLdTJS0yphzle/Uit89S6OxPyzaO76KGsDVThvpHT1JcAAYnf83iYFyn
         Y+wqJWaiGm2cvJIBqMiwSIZm/LuYyemDnZVNo9wXuBrTrCNGGPwzBp9SzeEn217PEn7Z
         7tAwxZxE3s08INrZYkLO9NElnsX2c8l8X5IfM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=YMEI3moq/Kt4fFt+GOEz+F+GkFVKXdbn4iz95Dw2hNVjbwe6p0JmPD5tFzEh2l89eE
         5YlVRcWW3xLmLEfWJDx69kY9WOzvTlAFTX+34rAnuk0JQnUxI4HArYaMHehH8l52A4IZ
         RXalh3jp1ENRjEjFR0lBK4IXkb4mrYUmad0JE=
MIME-Version: 1.0
Date: Thu, 16 Jun 2011 10:02:29 +0530
Message-ID: <BANLkTink1XyruQD-URYx647ttWSbL=vdzw@mail.gmail.com>
Subject: powered by hadoop logo:
From: UV Saradhi <uvsaradhi@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

4, 2, 6, 1, 3, 5

From general-return-3809-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 04:51:37 2011
Return-Path: <general-return-3809-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F334B66ED
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 04:51:36 +0000 (UTC)
Received: (qmail 61193 invoked by uid 500); 16 Jun 2011 04:51:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61149 invoked by uid 500); 16 Jun 2011 04:51:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61141 invoked by uid 99); 16 Jun 2011 04:51:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 04:51:34 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of esteban@cloudera.com designates 209.85.161.176 as permitted sender)
Received: from [209.85.161.176] (HELO mail-gx0-f176.google.com) (209.85.161.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 04:51:28 +0000
Received: by gxk7 with SMTP id 7so3737gxk.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 21:51:08 -0700 (PDT)
Received: by 10.91.82.7 with SMTP id j7mr613396agl.99.1308199867146; Wed, 15
 Jun 2011 21:51:07 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.90.84.6 with HTTP; Wed, 15 Jun 2011 21:50:47 -0700 (PDT)
In-Reply-To: <BANLkTinT5bF8eSdyoFOW633dM5b19dPRvg@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org> <BANLkTinT5bF8eSdyoFOW633dM5b19dPRvg@mail.gmail.com>
From: Esteban Gutierrez <esteban@cloudera.com>
Date: Wed, 15 Jun 2011 21:50:47 -0700
Message-ID: <BANLkTimJOe3SMkssUjHjOznWwkrZbWzO+A@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

5, 6, 4, 2, 3, 1

esteban.


On Wed, Jun 15, 2011 at 8:32 PM, Kyoji KATO <mpdm.kyoji@googlemail.com> wro=
te:
>
> Hey all,
>
> My vote.. 6,3,5,4,2,1
>
> Thanks,
> Kyoji KATO
>
> On 15 June 2011 12:19, Owen O'Malley <omalley@apache.org> wrote:
>
> > All,
> > =A0 We've had a wide range of entries for a powered by logo. I've put t=
hem
> > all on a page, here:
> >
> > http://people.apache.org/~omalley/hadoop-powered-by/
> >
> > Since there are a lot of contenders and we only want a single round of
> > voting, let's use single transferable vote ( STV
> > http://en.wikipedia.org/wiki/Single_transferable_vote). The important
> > thing is to pick the images *IN ORDER* that you would like them.
> >
> > My vote (in order of course): 4, 1, 2, 3, 5, 6.
> >
> > In other words, I want option 4 most and option 6 least. With STV, you
> > don't need to worry about voting for an unpopular choice since your vot=
e
> > will automatically roll over to your next choice.
> >
> > -- Owen
> >
> >
> >

From general-return-3810-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 05:06:28 2011
Return-Path: <general-return-3810-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9D4DA6E3B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 05:06:28 +0000 (UTC)
Received: (qmail 72128 invoked by uid 500); 16 Jun 2011 05:06:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72075 invoked by uid 500); 16 Jun 2011 05:06:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72066 invoked by uid 99); 16 Jun 2011 05:06:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 05:06:25 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 05:06:18 +0000
Received: by qwj9 with SMTP id 9so771013qwj.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 22:05:57 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=0HFqoVQ8Y0z3OwFl+sFn1a0lpGBY6lWXDXtLTDm4sQc=;
        b=mttjBeLNgt21nsBhv1Xb4DEn5BXq70bpdFnZXogPfPowl0fENnKnNkUd/bj8rim3qK
         +kgT1DGGX1Xf7wzh1a4fqExan5D8LDS3nw80ZyQuW2HCv+zVdn/b/RAyFGYdbsIPwYYZ
         4DjsVOAh64aa9omGBi0ViyTG1EGT+Yf1fM1/I=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=tKifIZvPTUq8D2J2wkwmD/EWa3yfah5PvGget+V0IAH0hTSaQHCyLaq/qs8XiAX1QL
         OPltJQCXWTQG9GT3FAJpeJzRNONaAgYGqStpT/NrhSaf8WhE/9OOtzXup780HAOgTT5X
         KQLcj1OOGjDv53Dkga0GxnaE4HuZCSt63/lQk=
MIME-Version: 1.0
Received: by 10.224.218.66 with SMTP id hp2mr333802qab.291.1308200757553; Wed,
 15 Jun 2011 22:05:57 -0700 (PDT)
Sender: saint.ack@gmail.com
Received: by 10.224.19.135 with HTTP; Wed, 15 Jun 2011 22:05:57 -0700 (PDT)
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
Date: Wed, 15 Jun 2011 22:05:57 -0700
X-Google-Sender-Auth: 0dDOitHFDhV0dgW7dbcyUU0-Ybc
Message-ID: <BANLkTikcv5F+gHVk-sOKdBiRdOsT5kKftQ@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

2
St.Ack

On Tue, Jun 14, 2011 at 8:19 PM, Owen O'Malley <omalley@apache.org> wrote:
> All,
> =A0 We've had a wide range of entries for a powered by logo. I've put the=
m all on a page, here:
>
> http://people.apache.org/~omalley/hadoop-powered-by/
>
> Since there are a lot of contenders and we only want a single round of vo=
ting, let's use single transferable vote ( STV http://en.wikipedia.org/wiki=
/Single_transferable_vote). The important thing is to pick the images *IN O=
RDER* that you would like them.
>
> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>
> In other words, I want option 4 most and option 6 least. With STV, you do=
n't need to worry about voting for an unpopular choice since your vote will=
 automatically roll over to your next choice.
>
> -- Owen
>
>
>

From general-return-3811-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 06:36:41 2011
Return-Path: <general-return-3811-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 47B6E67A9
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 06:36:41 +0000 (UTC)
Received: (qmail 63867 invoked by uid 500); 16 Jun 2011 06:36:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63664 invoked by uid 500); 16 Jun 2011 06:36:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63628 invoked by uid 99); 16 Jun 2011 06:36:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 06:36:17 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of esammer@cloudera.com designates 74.125.83.176 as permitted sender)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 06:36:13 +0000
Received: by pve37 with SMTP id 37so1320884pve.35
        for <general@hadoop.apache.org>; Wed, 15 Jun 2011 23:35:52 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.68.5.234 with SMTP id v10mr288781pbv.132.1308206152425; Wed,
 15 Jun 2011 23:35:52 -0700 (PDT)
Received: by 10.68.47.165 with HTTP; Wed, 15 Jun 2011 23:35:52 -0700 (PDT)
In-Reply-To: <61D08301-F6A9-4B9B-9B78-60D7D3B731A4@Holsman.NET>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
	<BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com>
	<090F2993-B9AA-410F-AD99-7AC047FFC974@apache.org>
	<BANLkTi=fkguAtbfi9Wk9CPm8JmFa2Hti3Q@mail.gmail.com>
	<54668CDE-443D-4AF1-B17A-0497D0CDCC8D@yahoo-inc.com>
	<BANLkTimTGcEU29e_uLB52uc9CeuSkK__6w@mail.gmail.com>
	<EFB94DDE-9E2C-42D9-99C2-2FEB742535D1@yahoo-inc.com>
	<24C673BE-B252-4643-BA96-EF6E4A9D978D@oracle.com>
	<BANLkTikqc18JbPS1XTiysXtH5XtpsBu51w@mail.gmail.com>
	<61D08301-F6A9-4B9B-9B78-60D7D3B731A4@Holsman.NET>
Date: Wed, 15 Jun 2011 23:35:52 -0700
Message-ID: <BANLkTim46ifZus_ZoKEJ_DC8SQvvkDR0iQ@mail.gmail.com>
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Cc: trademarks@apache.org, Eli Collins <eli@cloudera.com>, 
	Matthew Foley <mattf@yahoo-inc.com>
Content-Type: multipart/alternative; boundary=bcaec52158290221a304a5ce7b31

--bcaec52158290221a304a5ce7b31
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Jun 15, 2011 at 9:47 PM, Ian Holsman <hadoop@holsman.net> wrote:

>
> so yes .. even a simple patch makes it derived, because it is different.
>

...and a "dervied work" is fine. Nothing inherently wrong with the term
derived. I think the question is can one call it "Hadoop?" Note I'm *not*
saying "Apache Hadoop," just "Hadoop" when the derived work is actually
derived (to any degree, as Craig R pointed out). Apache Hadoop always and
forever means the bits voted on by the PMC - no vendor can claim that - but
there does appear to be plenty of prior examples of "reasonable" use of ASF
(and other OSS organization) project names in clearly derived works. I do
agree there should be a policy and it needs to be universally applied to be
fair to all involved.

Not to kick up the compatibility dust storm again, but people will always
claim crazy stuff that may or may not be true. We should just ignore it. Any
day of the week someone is claiming XYZ compatible either explicitly or
implicitly (as in client libraries for Foo Project). For cases where a
vendor makes a claim that isn't true, users will ask, we'll clarify that
Apache makes no guarantees of derived work compatibility and doesn't certify
anything (and specifically does the opposite - *NO* guarantees or
warranties).

Example uses I think should be fine / acceptable:

YDH (even though it no longer exists, it's a good example) and Y!'s use of
Hadoop
Facebook Hadoop
Hadoop at eBay
Hadoop at LinkedIn
IBM's use of Hadoop
and yes, CDH*

Even if some / all of the above modify at least a single bit (and may
*technically* be derived works) everyone understands what they mean. As for
the confusion, the OSS community has always just said "oh, they patch some
stuff, you should probably ask them" when confronted with vendor modified
versions of upstream projects; I've been involved in many of those upstream
projects, including a Linux distro (downstream). We should always be polite
to downstream users in redirecting them, but I think redirecting them is
fine. It's not confusing to users in my experience (we can make it a FAQ or
something and just point people there) as RedHat, Novell, Oracle, IBM, and
many other vendors have been happily[1] coexisting with their upstream
counterparts for a long time.

I believe we (the collective Apache Hadoop community including those that
redistribute Hadoop bits in various forms) should focus on producing
regular, quality releases in a cooperative and constructive environment, and
continue to require vendors to provide the proper attribution and license
information. This is in everyone's interest, vendors and direct users alike.

*Disclosure: I work for Cloudera and I think this should apply to anyone and
everyone, including my employer (with whom I obviously do not clear emails.
:))

[1] OK, maybe not always "happily" but mostly so. You know what I mean.

Thanks to Steve L and others for their hard work on this one.
(Sorry for the long email.)

-- 
Eric Sammer
twitter: esammer
data: www.cloudera.com

--bcaec52158290221a304a5ce7b31--

From general-return-3812-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 07:24:13 2011
Return-Path: <general-return-3812-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DCD9662BC
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 07:24:13 +0000 (UTC)
Received: (qmail 28283 invoked by uid 500); 16 Jun 2011 07:24:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28214 invoked by uid 500); 16 Jun 2011 07:24:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28206 invoked by uid 99); 16 Jun 2011 07:24:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 07:24:12 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of pedro.figueiredo@playfish.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 07:24:04 +0000
Received: by wyb40 with SMTP id 40so1289312wyb.35
        for <general@hadoop.apache.org>; Thu, 16 Jun 2011 00:23:44 -0700 (PDT)
Received: by 10.227.178.72 with SMTP id bl8mr545028wbb.6.1308209024088;
        Thu, 16 Jun 2011 00:23:44 -0700 (PDT)
Received: from [10.21.98.155] ([80.169.35.118])
        by mx.google.com with ESMTPS id o38sm938317wba.3.2011.06.16.00.23.42
        (version=TLSv1/SSLv3 cipher=OTHER);
        Thu, 16 Jun 2011 00:23:43 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: [VOTE] Powered by Logo
From: Pedro Figueiredo <pedro.figueiredo@playfish.com>
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
Date: Thu, 16 Jun 2011 08:23:41 +0100
Content-Transfer-Encoding: quoted-printable
Message-Id: <515495E0-B5AF-450B-9BBF-F4A5633E6F7B@playfish.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

5, 4, 2, 6, 1, 3

Cheers,

Pedro Figueiredo
Systems Engineer, Playfish
EA Interactive

Skype: pfiguk
Web: http://www.playfish.com
----
Electronic Arts Ltd, Registered office address Onslow House, Onslow =
Street, Guildford, Surrey, GU1 4TN, UK.  Reg. in England & Wales, Reg. =
No. 2057591




From general-return-3813-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 08:44:56 2011
Return-Path: <general-return-3813-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6508F652B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 08:44:56 +0000 (UTC)
Received: (qmail 55408 invoked by uid 500); 16 Jun 2011 08:44:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55360 invoked by uid 500); 16 Jun 2011 08:44:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55347 invoked by uid 99); 16 Jun 2011 08:44:54 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 08:44:54 +0000
Received: from localhost (HELO mail-vw0-f48.google.com) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 08:44:54 +0000
Received: by vws7 with SMTP id 7so1472793vws.35
        for <general@hadoop.apache.org>; Thu, 16 Jun 2011 01:44:53 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.52.37.129 with SMTP id y1mr834511vdj.199.1308213893419; Thu,
 16 Jun 2011 01:44:53 -0700 (PDT)
Received: by 10.52.183.8 with HTTP; Thu, 16 Jun 2011 01:44:53 -0700 (PDT)
In-Reply-To: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
Date: Thu, 16 Jun 2011 10:44:53 +0200
Message-ID: <BANLkTikY006VsFnS+XPkSYBy-Kj7MLP_AQ@mail.gmail.com>
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
From: Doug Cutting <cutting@apache.org>
To: general@hadoop.apache.org
Cc: Apache Brand Management <trademarks@apache.org>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

-1 if patches that have been committed to trunk are not permitted to
be applied to distributions that are still called "Apache Hadoop".
That's the rule we agreed on some time ago at Roy's suggestion.  Let's
first document the status quo, then, separately, discuss and vote on
changes to it.  Also, such a branding rule should probably be uniform
across Apache projects, not Hadoop-specific.

Doug

On Wed, Jun 15, 2011 at 12:56 AM, Owen O'Malley <omalley@apache.org> wrote:
> All,
> =C2=A0=C2=A0 Steve Loughran has done some great work on defining what can=
 be called
> Hadoop at=C2=A0http://wiki.apache.org/hadoop/Defining%20Hadoop. After som=
e
> cleanup from Noirin and Shane, I think we've got a really good base. I'd
> like a vote to approve the content (at the current revision 12) and put t=
he
> content on our web site.
> Clearly, I'm +1.
> -- Owen

From general-return-3814-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 09:28:19 2011
Return-Path: <general-return-3814-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0657F6BF3
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 09:28:19 +0000 (UTC)
Received: (qmail 3194 invoked by uid 500); 16 Jun 2011 09:28:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3144 invoked by uid 500); 16 Jun 2011 09:28:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3136 invoked by uid 99); 16 Jun 2011 09:28:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 09:28:16 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 69.147.107.21 is neither permitted nor denied by domain of ranjit@yahoo-inc.com)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 09:28:11 +0000
Received: from dishnailfind-dx.eglbp.corp.yahoo.com (dishnailfind-dx.eglbp.corp.yahoo.com [10.66.77.50])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5G9R8XF008288
	for <general@hadoop.apache.org>; Thu, 16 Jun 2011 02:27:09 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1308216430;
	bh=/PZ6r4Inc/J+hyQej2SqWBwFvcvxy7WHYAQJj8yYNdw=;
	h=Message-ID:Date:From:MIME-Version:To:Subject:References:
	 In-Reply-To:Content-Type:Content-Transfer-Encoding;
	b=LDelv2F3Su8Rk9FB2mYMrwAEla0pPvED8DXyh3kl/ctF55A6bSbKef0tMA1rbDx6m
	 aYqj9TiCnugc+opAGHlnNSb8exe/AFVpa5eI1AQcynKixEBj7XBhw+4/DQJASfZ9+k
	 9GOiZ4W3ZZhNwG/g0iqrb8YktYjWpZyGxCMD6LjE=
Message-ID: <4DF9CC6C.4050306@yahoo-inc.com>
Date: Thu, 16 Jun 2011 14:57:08 +0530
From: Ranjit Mathew <ranjit@yahoo-inc.com>
Organization: Yahoo
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.17) Gecko/20110414 Lightning/1.0b2 Thunderbird/3.1.10
MIME-Version: 1.0
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: HADOOP-7106 (project unsplit) this weekend
References: <BANLkTikU1LxFc6aF_dV6Fbu4E-fxf3c7ww@mail.gmail.com> <CA1B916C.78CF%jnaisbit@yahoo-inc.com> <BANLkTincXUjoeiXhp8tHn4+qLs642WkbMA@mail.gmail.com> <BANLkTinVKj9DE+UoLRfhjVNA=mBR0k5yFg@mail.gmail.com>
In-Reply-To: <BANLkTinVKj9DE+UoLRfhjVNA=mBR0k5yFg@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 06/13/2011 09:33 PM, Todd Lipcon wrote:
>> Oops, sorry about that one. I will take care of that in about 30 minutes
>> (just headed out the door now to catch a train). If someone else with commit
>> access wants to, you just need to propset the externals to point to th new
>> common/trunk/common/src/test/bin instead of the old location.
>>
>>
> Fixed the svn:externals

Does it make sense for "test-patch.sh" and "smart-apply-patch.sh" to
remain "external" now that they're within the same project?

Currently on every "svn up" for trunk I get:
-------------------------- 8< --------------------------
$ svn up

Fetching external item into 'hdfs/src/test/bin'
External at revision 1136333.


Fetching external item into 'mapreduce/src/test/bin'
External at revision 1136333.

At revision 1136333.
-------------------------- 8< --------------------------

after a little pause.

Ranjit

PS: I'm using "svn, version 1.6.16 (r1073529)" on Fedora 14/x86-32.

From general-return-3815-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 10:45:30 2011
Return-Path: <general-return-3815-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9BE1866B9
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 10:45:30 +0000 (UTC)
Received: (qmail 82760 invoked by uid 500); 16 Jun 2011 10:45:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82696 invoked by uid 500); 16 Jun 2011 10:45:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82688 invoked by uid 99); 16 Jun 2011 10:45:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 10:45:28 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 69.147.107.21 is neither permitted nor denied by domain of svenkat@yahoo-inc.com)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 10:45:21 +0000
Received: from EGL-EX07CAS01.ds.corp.yahoo.com (egl-ex07cas01.eglbp.corp.yahoo.com [203.83.248.208])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5GAijqd032810
	for <general@hadoop.apache.org>; Thu, 16 Jun 2011 03:44:45 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1308221086;
	bh=TpOD2X7ziI/iYycgJniXx+myxfrzyJC76jE8699rd5k=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:MIME-Version;
	b=ovbNwAxe0IBkZGZmsSgprKr+1Z9FZuj41Qkwd2lF3hG3+z6U+xqFN8y8rutN8vXbX
	 sDNfRiKMRfZwXCBo+ZoolXXrf+MJfrl44qUjq+1teGwP6Gv2pQbdDZ9eA6evZs7jIJ
	 djGsELPjDt6r0EJnennYJAtThuIm9xItfzy8/2TA=
Received: from EGL-EX07VS02.ds.corp.yahoo.com ([203.83.248.206]) by
 EGL-EX07CAS01.ds.corp.yahoo.com ([203.83.248.215]) with mapi; Thu, 16 Jun
 2011 16:14:45 +0530
From: Venkatesh S <svenkat@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 16 Jun 2011 16:14:44 +0530
Subject: Re: [VOTE] Powered by Logo
Thread-Topic: [VOTE] Powered by Logo
Thread-Index: AcwrCyYHVipAuqPyRaKXOV7kHDLtQQAH01Gr
Message-ID: <CA1E574D.5394C%svenkat@yahoo-inc.com>
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

My vote: 4, 3, 2, 6, 5, 1

Thanks,
Venkatesh

On 6/15/11 8:49 AM, "Owen O'Malley" <omalley@apache.org> wrote:

> All,
>    We've had a wide range of entries for a powered by logo. I've put them=
 all
> on a page, here:
>=20
> http://people.apache.org/~omalley/hadoop-powered-by/
>=20
> Since there are a lot of contenders and we only want a single round of vo=
ting,
> let's use single transferable vote ( STV
> http://en.wikipedia.org/wiki/Single_transferable_vote). The important thi=
ng is
> to pick the images *IN ORDER* that you would like them.
>=20
> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>=20
> In other words, I want option 4 most and option 6 least. With STV, you do=
n't
> need to worry about voting for an unpopular choice since your vote will
> automatically roll over to your next choice.
>=20
> -- Owen
>=20
>=20
>=20


From general-return-3816-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 11:00:53 2011
Return-Path: <general-return-3816-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D3639630F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 11:00:53 +0000 (UTC)
Received: (qmail 21274 invoked by uid 500); 16 Jun 2011 11:00:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21216 invoked by uid 500); 16 Jun 2011 11:00:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21208 invoked by uid 99); 16 Jun 2011 11:00:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 11:00:51 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 11:00:43 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 91821B7DDB
	for <general@hadoop.apache.org>; Thu, 16 Jun 2011 12:00:21 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id bCAfPVE5QjR5 for <general@hadoop.apache.org>;
	Thu, 16 Jun 2011 12:00:21 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by tobor.hpl.hp.com (Postfix) with ESMTP id C8064B7DD3
	for <general@hadoop.apache.org>; Thu, 16 Jun 2011 12:00:20 +0100 (BST)
MailScanner-NULL-Check: 1308826806.68105@1G+p/SSlMaE2iJgMSCTHdw
Received: from [16.25.175.86] (wildhaus.hpl.hp.com [16.25.175.86])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p5GB06Mk013119
	for <general@hadoop.apache.org>; Thu, 16 Jun 2011 12:00:06 +0100 (BST)
Message-ID: <4DF9E236.5050500@apache.org>
Date: Thu, 16 Jun 2011 12:00:06 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Bigtop Proposal
References: <4DF75531.9050302@apache.org> <BANLkTikChz5mH=NFdjd_8r7UMNPJTAHVuQ@mail.gmail.com>
In-Reply-To: <BANLkTikChz5mH=NFdjd_8r7UMNPJTAHVuQ@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p5GB06Mk013119
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 15/06/11 19:23, Ted Dunning wrote:
> On Tue, Jun 14, 2011 at 2:33 PM, Steve Loughran<stevel@apache.org>  wrote:
>
>> For those people who aren't on the Apache Incubator mailing lists, Tom
>> White has proposed "BigTop", which is the tooling to integrate the build and
>> testing of the Apache Hadoop technologies, perhaps eventually to have
>> coordinated releases.
>>
>
> I will comment on the proposal directly as well, but coordinated releases
> would be a disaster.  Mahout, for one, should not have to wait for Hadoop to
> have a release.  We are already unhappy that we only get two releases a year
> and want to up that to four.

>
> Much better is to have a directed acyclic dependency graph with coherent
> versioning.  That allows projects to stay independent, but still express


you could do synchronised factored releases: if Hadoop ships every 12 
months, other things could come out every 4 months against the last 
major release, things like Hama could come out monthly while in the 
rapidly evolving state.

From general-return-3817-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 11:47:32 2011
Return-Path: <general-return-3817-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 12B6B67DE
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 11:47:32 +0000 (UTC)
Received: (qmail 77422 invoked by uid 500); 16 Jun 2011 11:47:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77349 invoked by uid 500); 16 Jun 2011 11:47:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77341 invoked by uid 99); 16 Jun 2011 11:47:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 11:47:30 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 11:47:22 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id DFB8DB7D5D
	for <general@hadoop.apache.org>; Thu, 16 Jun 2011 12:47:01 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id CPxDDY0FbEHn for <general@hadoop.apache.org>;
	Thu, 16 Jun 2011 12:47:00 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 80452B7CFF
	for <general@hadoop.apache.org>; Thu, 16 Jun 2011 12:47:00 +0100 (BST)
MailScanner-NULL-Check: 1308829606.68752@WxITeuoRSA6XcOfAAW/81Q
Received: from [16.25.175.86] (wildhaus.hpl.hp.com [16.25.175.86])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p5GBkkIO014640
	for <general@hadoop.apache.org>; Thu, 16 Jun 2011 12:46:46 +0100 (BST)
Message-ID: <4DF9ED26.50108@apache.org>
Date: Thu, 16 Jun 2011 12:46:46 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>	<BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com>	<090F2993-B9AA-410F-AD99-7AC047FFC974@apache.org>	<BANLkTi=fkguAtbfi9Wk9CPm8JmFa2Hti3Q@mail.gmail.com>	<54668CDE-443D-4AF1-B17A-0497D0CDCC8D@yahoo-inc.com>	<BANLkTimTGcEU29e_uLB52uc9CeuSkK__6w@mail.gmail.com>	<EFB94DDE-9E2C-42D9-99C2-2FEB742535D1@yahoo-inc.com>	<24C673BE-B252-4643-BA96-EF6E4A9D978D@oracle.com>	<BANLkTikqc18JbPS1XTiysXtH5XtpsBu51w@mail.gmail.com>	<61D08301-F6A9-4B9B-9B78-60D7D3B731A4@Holsman.NET> <BANLkTim46ifZus_ZoKEJ_DC8SQvvkDR0iQ@mail.gmail.com>
In-Reply-To: <BANLkTim46ifZus_ZoKEJ_DC8SQvvkDR0iQ@mail.gmail.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 8bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p5GBkkIO014640
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 16/06/11 07:35, Eric Sammer wrote:
> On Wed, Jun 15, 2011 at 9:47 PM, Ian Holsman<hadoop@holsman.net>  wrote:
>
>>
>> so yes .. even a simple patch makes it derived, because it is different.
>>
>
> ...and a "dervied work" is fine. Nothing inherently wrong with the term
> derived. I think the question is can one call it "Hadoop?" Note I'm *not*
> saying "Apache Hadoop," just "Hadoop" when the derived work is actually
> derived (to any degree, as Craig R pointed out). Apache Hadoop always and
> forever means the bits voted on by the PMC - no vendor can claim that - but
> there does appear to be plenty of prior examples of "reasonable" use of ASF
> (and other OSS organization) project names in clearly derived works. I do
> agree there should be a policy and it needs to be universally applied to be
> fair to all involved.
>
> Not to kick up the compatibility dust storm again, but people will always
> claim crazy stuff that may or may not be true. We should just ignore it.

The issue is branding and trademarks, eventually things get downgraded 
to become meaningless. If I code an MR engine in erlang (I have one 
somewhere), can I call it "Hadoop for Erlang"?

 > Any
> day of the week someone is claiming XYZ compatible either explicitly or
> implicitly (as in client libraries for Foo Project). For cases where a
> vendor makes a claim that isn't true, users will ask, we'll clarify that
> Apache makes no guarantees of derived work compatibility and doesn't certify
> anything (and specifically does the opposite - *NO* guarantees or
> warranties).

-BigTop could provide that defensible compatibility statement. 
"Automotive Joe's Crankshaft platform passed the Apache BigTop DFS, MR, 
Mahout and HBase test suites"

>
> Example uses I think should be fine / acceptable:
>
> YDH (even though it no longer exists, it's a good example) and Y!'s use of
> Hadoop

-creates confusion and encourages the notion that anything is a 
distribution of hadoop, which is the situation that the trademarks 
people are trying to crack down

> Facebook Hadoop

-depends on internal vs external

> Hadoop at eBay
> Hadoop at LinkedIn

details of internal use, as valid as "Hadoop in Steve's house", which, 
given my known network state, is always something to cherish. And while 
I have built my branch up and published it, it's no longer something I 
distribute (though it is in an open SVN repository somewhere). I'm 
working directly with Apache Hadoop 0.20.203 these days.


> IBM's use of Hadoop

not sure about IBM distribution of Apache Hadoop, as I presume it has 
the uncommitted patch to work on IBM JVMs (though were someone to commit 
it..)

http://www.alphaworks.ibm.com/tech/idah

The biginsights product is more explicit and, to me, a good example of 
terminology. Their own brand, description of the benefits, and details 
on what's in there:

"IBM InfoSphere BigInsights Enterprise Edition
For turning complex, internet-scale information into insight, cost 
effectively

IBM® InfoSphere™ BigInsights Enterprise Edition enables new solutions 
that turn large, complex volumes of data into insight, cost effectively. 
InfoSphere BigInsights delivers an enterprise-ready big data solution by 
combining Apache Hadoop, including the MapReduce framework and the 
Hadoop Distributed File Systems (HDFS), with unique technologies and 
capabilities from across IBM."

That gives them the flexibility to swap things around in future (switch 
to GPFS, MapR, Brisk) without having to change their branding.

> and yes, CDH*

If you look a the CDH site its now "Cloudera's Distribution including 
Apache Hadoop". After all it's Cloudera's data analysis stack including 
Apache Hadoop,



> Even if some / all of the above modify at least a single bit (and may
> *technically* be derived works) everyone understands what they mean. As for
> the confusion, the OSS community has always just said "oh, they patch some
> stuff, you should probably ask them" when confronted with vendor modified
> versions of upstream projects; I've been involved in many of those upstream
> projects, including a Linux distro (downstream). We should always be polite
> to downstream users in redirecting them, but I think redirecting them is
> fine. It's not confusing to users in my experience (we can make it a FAQ or
> something and just point people there) as RedHat, Novell, Oracle, IBM, and
> many other vendors have been happily[1] coexisting with their upstream
> counterparts for a long time.

co-existence yes; happiness, not always:

http://www.jonobacon.org/2010/07/30/red-hat-canonical-and-gnome-contributions/
http://lwn.net/Articles/374737/
http://gburt.blogspot.com/2011/02/banshee-supporting-gnome-on-ubuntu.html
http://bazaar.launchpad.net/~mozillateam/firefox/firefox-4.0.head/view/head:/debian/patches/ubuntu-codes-amazon.patch

Where ubuntu are good is that launchpad is a good entry point for filing 
and tracking any ubuntu-related problem, and helping to push that 
upstream, so the local issue can be linked to the source issue, letting 
me deal with problems like getting sound to work:
https://bugs.launchpad.net/ubuntu/+source/amarok/+bug/523269
https://bugs.launchpad.net/ubuntu/+source/alsa-driver/+bug/584844

JIRA doesn't do that cross-instance tracking, which is painful for me at 
work, where I do deal with multiple JIRA instances. You can put remote 
URLs in, but they don't get synchronised. I can't say SFOS-780 depends 
on apache.org/MAPREDUCE-279, for example.

>
> I believe we (the collective Apache Hadoop community including those that
> redistribute Hadoop bits in various forms) should focus on producing
> regular, quality releases in a cooperative and constructive environment, and
> continue to require vendors to provide the proper attribution and license
> information. This is in everyone's interest, vendors and direct users alike.

+1

> *Disclosure: I work for Cloudera and I think this should apply to anyone and
> everyone, including my employer (with whom I obviously do not clear emails.
> :))

I understand -it may ultimately affect my employer too. Which is why a 
consistent approach matters, then nobody will feel they are being 
discriminated against.




From general-return-3818-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 13:49:33 2011
Return-Path: <general-return-3818-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BA8486EE7
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 13:49:33 +0000 (UTC)
Received: (qmail 3431 invoked by uid 500); 16 Jun 2011 13:49:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3382 invoked by uid 500); 16 Jun 2011 13:49:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3374 invoked by uid 99); 16 Jun 2011 13:49:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 13:49:32 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of david.scott.williams.nc@gmail.com designates 209.85.216.169 as permitted sender)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 13:49:27 +0000
Received: by qyk2 with SMTP id 2so3123928qyk.14
        for <general@hadoop.apache.org>; Thu, 16 Jun 2011 06:49:06 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=MyYSiUanthQaLHwHgm5Dh2KPfJRHQ6b8jgO7jUuVrzU=;
        b=cxufKPhjQ/W0SHiGFcGtbwpSI0zcIH3JHbXmGXr/Ee6vZIINXa8YyfjEUr4ln74quH
         aEAhk4He/+mG9nbI9RhG3bD0aFCCLjevszu6/xfmDDp6SkVZkwrDTZbFlYEbRPX1pmdQ
         N1pHfj3DmhEOfXzoIIOIojSmpjZQxJZt63LT0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=CHLr/TEzk3fRG0/bPmBgfEeb5YyZzMayOolihlxyEXLOXK5akstuBaYdY7+nRnPeZC
         KFvjZI/1agq7Yw8pn1+Ji98mheVKaTLlTeDUpdgpFX6hvD/PjXyC09Hvhejcc1M1M/63
         tdV+kqw+VL7NVti8I2nYGppXAjCJe7a0iq5S8=
MIME-Version: 1.0
Received: by 10.229.48.200 with SMTP id s8mr762731qcf.118.1308232142246; Thu,
 16 Jun 2011 06:49:02 -0700 (PDT)
Received: by 10.229.184.147 with HTTP; Thu, 16 Jun 2011 06:49:02 -0700 (PDT)
Date: Thu, 16 Jun 2011 09:49:02 -0400
Message-ID: <BANLkTik1_g4Pt7xgWmz8GvwC=wEvngn++g@mail.gmail.com>
Subject: Is this just bluster?
From: David Scott Williams <david.scott.williams.nc@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

http://gigaom.com/cloud/lexisnexis-open-sources-its-hadoop-killer/

-- 
===========================
David Scott Williams
828-245-3866 (home)
973-981-1997 (cell)
===========================

I write my blog at http://deadpenguinsociety.org
Twitter @dscott_williams

From general-return-3819-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 14:23:14 2011
Return-Path: <general-return-3819-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EBA566A14
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 14:23:14 +0000 (UTC)
Received: (qmail 84626 invoked by uid 500); 16 Jun 2011 14:23:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84580 invoked by uid 500); 16 Jun 2011 14:23:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84572 invoked by uid 99); 16 Jun 2011 14:23:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 14:23:13 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of msegel@navteq.com designates 204.120.70.37 as permitted sender)
Received: from [204.120.70.37] (HELO xmailfargo.navteq.com) (204.120.70.37)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 14:23:08 +0000
Received: from fargo-spamfw.navteq.com (navteq-spamfw.navteq.com [10.6.10.131])
	by xmailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id p5GDlG4f002239
	for <general@hadoop.apache.org>; Thu, 16 Jun 2011 08:47:16 -0500
Received: from imailfargo.navteq.com (localhost [127.0.0.1])
	by fargo-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id 2C799490BF37
	for <general@hadoop.apache.org>; Thu, 16 Jun 2011 09:22:47 -0500 (CDT)
Received: from imailfargo.navteq.com (imailfargo.navteq.com [10.6.10.130]) by fargo-spamfw.navteq.com with ESMTP id xn6JTa3RgSjK4trh for <general@hadoop.apache.org>; Thu, 16 Jun 2011 09:22:47 -0500 (CDT)
Received: from hq-ex-ht02.ad.navteq.com ([10.8.222.172])
	by imailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id p5GEMk4V013824;
	Thu, 16 Jun 2011 09:22:46 -0500
Received: from hq-ex-mb03.ad.navteq.com ([fe80::c4dd:7b21:5c22:cfe4]) by
 hq-ex-ht02.ad.navteq.com ([fe80::24f6:ca50:734d:df6e%13]) with mapi; Thu, 16
 Jun 2011 09:22:46 -0500
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 16 Jun 2011 09:23:25 -0500
Subject: Re: Is this just bluster?
Thread-Topic: Is this just bluster?
Thread-Index: AcwsMNyn6bAhAU3wQqeep3H088i7HQ==
Message-ID: <3FCB6229-BB89-4D05-BA98-6ADA1510EF90@navteq.com>
References: <BANLkTik1_g4Pt7xgWmz8GvwC=wEvngn++g@mail.gmail.com>
In-Reply-To: <BANLkTik1_g4Pt7xgWmz8GvwC=wEvngn++g@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1

It depends on what you mean by bluster.

There isn't anything really new about the concept of distributed file sys=
tems or parallel processing frameworks. =20

My take is that LexisNexus wants to expand the market for their existing =
solution.
Not so much bluster. However, definitely some marketing spin and their in=
terpretation of things.

It's interesting in that some of their comments about some of the issues =
with Hadoop echo the comments made by others like maprtech, EMC, IBM etc =
=2E.. =20

I just don't think that there is going to be much interest in their offer=
ings beyond their existing customer base. (IMHO) The market is big and as=
 we transition from leading edge/bleeding edge we will see more fragmenta=
tion then in a couple of years consolidation.

Sent from a remote device. Please excuse any typos...

Mike Segel

On Jun 16, 2011, at 8:49 AM, "David Scott Williams" <david.scott.williams=
=2Enc@gmail.com> wrote:

> http://gigaom.com/cloud/lexisnexis-open-sources-its-hadoop-killer/
>=20
> --=20
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D
> David Scott Williams
> 828-245-3866 (home)
> 973-981-1997 (cell)
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D
>=20
> I write my blog at http://deadpenguinsociety.org
> Twitter @dscott_williams


The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.

From general-return-3820-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 14:48:51 2011
Return-Path: <general-return-3820-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D14626F6C
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 14:48:51 +0000 (UTC)
Received: (qmail 34291 invoked by uid 500); 16 Jun 2011 14:48:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34174 invoked by uid 500); 16 Jun 2011 14:48:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34161 invoked by uid 99); 16 Jun 2011 14:48:50 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 14:48:50 +0000
Received: from localhost (HELO mail-qw0-f48.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 14:48:49 +0000
Received: by qwj9 with SMTP id 9so1032335qwj.35
        for <general@hadoop.apache.org>; Thu, 16 Jun 2011 07:48:48 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.10.209 with SMTP id q17mr816727qcq.106.1308235728687; Thu,
 16 Jun 2011 07:48:48 -0700 (PDT)
Received: by 10.229.215.206 with HTTP; Thu, 16 Jun 2011 07:48:48 -0700 (PDT)
In-Reply-To: <F4FCE59AE46B2243922F224426BC869B01EC996726@RHV-MEXMS-002.corp.ebay.com>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
	<BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com>
	<FBF55C01-D66A-44C0-9252-D54E7CAB103D@apache.org>
	<BANLkTi=BWPUEoqD3L98A9AXLkbafZs15Zw@mail.gmail.com>
	<92D32E6E-3B65-464F-B91B-4C30EAB2642C@apache.org>
	<BANLkTi=Gj-XvME7EyHYqPQaUwGNJusr17g@mail.gmail.com>
	<F4FCE59AE46B2243922F224426BC869B01EC996726@RHV-MEXMS-002.corp.ebay.com>
Date: Thu, 16 Jun 2011 07:48:48 -0700
Message-ID: <BANLkTinRgz-Lnm0WzB=oGyxr8ii2WK6hRQ@mail.gmail.com>
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
From: "Owen O'Malley" <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364ecb74e4119904a5d55db6

--0016364ecb74e4119904a5d55db6
Content-Type: text/plain; charset=UTF-8

On Wed, Jun 15, 2011 at 9:24 PM, Rottinghuis, Joep <jrottinghuis@ebay.com>wrote:

It does make sense to me to distinguish between the case when a company
> seeks to benefit from using the Hadoop name for their product and the case
> when a company uses Hadoop internally with some minor patches.
>

If they aren't distributing the version that they use, no one will know or
care if they have patches applied. Eli is just trying to cloud the real
issue, which is about distributors and what they call
their derivative works.

For example: large company creates a game-show playing appliance and
> explains that they have used Hadoop for some of the learning tasks. Not
> allowed if they applied more than 3 patches?
>

Of course it is allowed. It is only a question of whether you can distribute
it to others and call it Hadoop.


> Also, if thousands of changes are packaged together into one giant patch,
> is that allowed?
>

No, the exception is strictly for critical security fixes and I would
sincerely hope that those would be released by Apache in very short order.

-- Owen

--0016364ecb74e4119904a5d55db6--

From general-return-3821-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 15:02:22 2011
Return-Path: <general-return-3821-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C4D55606A
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 15:02:22 +0000 (UTC)
Received: (qmail 65515 invoked by uid 500); 16 Jun 2011 15:02:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65436 invoked by uid 500); 16 Jun 2011 15:02:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65419 invoked by uid 99); 16 Jun 2011 15:02:17 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 15:02:17 +0000
Received: from localhost (HELO mail-qy0-f176.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 15:02:17 +0000
Received: by qyk30 with SMTP id 30so1029661qyk.14
        for <general@hadoop.apache.org>; Thu, 16 Jun 2011 08:02:16 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.10.209 with SMTP id q17mr831401qcq.106.1308236536034; Thu,
 16 Jun 2011 08:02:16 -0700 (PDT)
Received: by 10.229.215.206 with HTTP; Thu, 16 Jun 2011 08:02:15 -0700 (PDT)
In-Reply-To: <BANLkTim46ifZus_ZoKEJ_DC8SQvvkDR0iQ@mail.gmail.com>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
	<BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com>
	<090F2993-B9AA-410F-AD99-7AC047FFC974@apache.org>
	<BANLkTi=fkguAtbfi9Wk9CPm8JmFa2Hti3Q@mail.gmail.com>
	<54668CDE-443D-4AF1-B17A-0497D0CDCC8D@yahoo-inc.com>
	<BANLkTimTGcEU29e_uLB52uc9CeuSkK__6w@mail.gmail.com>
	<EFB94DDE-9E2C-42D9-99C2-2FEB742535D1@yahoo-inc.com>
	<24C673BE-B252-4643-BA96-EF6E4A9D978D@oracle.com>
	<BANLkTikqc18JbPS1XTiysXtH5XtpsBu51w@mail.gmail.com>
	<61D08301-F6A9-4B9B-9B78-60D7D3B731A4@Holsman.NET>
	<BANLkTim46ifZus_ZoKEJ_DC8SQvvkDR0iQ@mail.gmail.com>
Date: Thu, 16 Jun 2011 08:02:15 -0700
Message-ID: <BANLkTin2YDAqSG3Uk_hLSQziRJsW1QRFeA@mail.gmail.com>
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
From: "Owen O'Malley" <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364ecb7403360304a5d58ebe

--0016364ecb7403360304a5d58ebe
Content-Type: text/plain; charset=UTF-8

On Wed, Jun 15, 2011 at 11:35 PM, Eric Sammer <esammer@cloudera.com> wrote:

I think the question is can one call it "Hadoop?" Note I'm *not*
> saying "Apache Hadoop," just "Hadoop" when the derived work is actually
> derived (to any degree, as Craig R pointed out). Apache Hadoop always and
> forever means the bits voted on by the PMC - no vendor can claim that - but
> there does appear to be plenty of prior examples of "reasonable" use of ASF
> (and other OSS organization) project names in clearly derived works.
>

Thank you, Eric, for demonstrating why we are fixing it. Apache owns the
Hadoop trademark. Hadoop is PRECISELY the same as Apache Hadoop. They are
two names for the same thing. If the Hadoop PMC were to fail to enforce
that, the Apache board would remove us en masse from the PMC.

-- Owen

--0016364ecb7403360304a5d58ebe--

From general-return-3822-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 15:32:05 2011
Return-Path: <general-return-3822-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6982464B6
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 15:32:05 +0000 (UTC)
Received: (qmail 44710 invoked by uid 500); 16 Jun 2011 15:32:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44664 invoked by uid 500); 16 Jun 2011 15:32:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44656 invoked by uid 99); 16 Jun 2011 15:32:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 15:32:03 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 15:31:55 +0000
Received: by wyb40 with SMTP id 40so1739159wyb.35
        for <general@hadoop.apache.org>; Thu, 16 Jun 2011 08:31:35 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.159.75 with SMTP id r53mr996279wek.98.1308238295574; Thu,
 16 Jun 2011 08:31:35 -0700 (PDT)
Received: by 10.216.30.212 with HTTP; Thu, 16 Jun 2011 08:31:35 -0700 (PDT)
In-Reply-To: <BANLkTinRgz-Lnm0WzB=oGyxr8ii2WK6hRQ@mail.gmail.com>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
	<BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com>
	<FBF55C01-D66A-44C0-9252-D54E7CAB103D@apache.org>
	<BANLkTi=BWPUEoqD3L98A9AXLkbafZs15Zw@mail.gmail.com>
	<92D32E6E-3B65-464F-B91B-4C30EAB2642C@apache.org>
	<BANLkTi=Gj-XvME7EyHYqPQaUwGNJusr17g@mail.gmail.com>
	<F4FCE59AE46B2243922F224426BC869B01EC996726@RHV-MEXMS-002.corp.ebay.com>
	<BANLkTinRgz-Lnm0WzB=oGyxr8ii2WK6hRQ@mail.gmail.com>
Date: Thu, 16 Jun 2011 08:31:35 -0700
Message-ID: <BANLkTimVZGONtNp2Ynz5qhevGiAqXa3Knw@mail.gmail.com>
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Thu, Jun 16, 2011 at 7:48 AM, Owen O'Malley <omalley@apache.org> wrote:
> On Wed, Jun 15, 2011 at 9:24 PM, Rottinghuis, Joep <jrottinghuis@ebay.com>wrote:
>
> It does make sense to me to distinguish between the case when a company
>> seeks to benefit from using the Hadoop name for their product and the case
>> when a company uses Hadoop internally with some minor patches.
>>
>
> If they aren't distributing the version that they use, no one will know or
> care if they have patches applied. Eli is just trying to cloud the real
> issue, which is about distributors and what they call
> their derivative works.
>

I truly don't see distribution as the relevant issue, in particular I
don't see why the definition of what Hadoop should change on whether
or not you distribute it.

> For example: large company creates a game-show playing appliance and
>> explains that they have used Hadoop for some of the learning tasks. Not
>> allowed if they applied more than 3 patches?
>>
>
> Of course it is allowed. It is only a question of whether you can distribute
> it to others and call it Hadoop.
>

So you want IBM to call what they run Hadoop, unless they put it up on
a website in which case they can no longer call it Hadoop. What is the
rationale?

Thanks,
Eli

From general-return-3823-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 15:38:49 2011
Return-Path: <general-return-3823-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7E271611F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 15:38:49 +0000 (UTC)
Received: (qmail 67472 invoked by uid 500); 16 Jun 2011 15:38:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67407 invoked by uid 500); 16 Jun 2011 15:38:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67399 invoked by uid 99); 16 Jun 2011 15:38:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 15:38:47 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 15:38:40 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id EED721BAA27
	for <general@hadoop.apache.org>; Thu, 16 Jun 2011 16:38:19 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id 9mWLl0TxUUzn for <general@hadoop.apache.org>;
	Thu, 16 Jun 2011 16:38:19 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 3241C1BAA1D
	for <general@hadoop.apache.org>; Thu, 16 Jun 2011 16:38:18 +0100 (BST)
MailScanner-NULL-Check: 1308843485.35787@NYUnXM77F+TWZQp71VEb2A
Received: from [16.25.175.86] (wildhaus.hpl.hp.com [16.25.175.86])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p5GFc3Qb021151
	for <general@hadoop.apache.org>; Thu, 16 Jun 2011 16:38:03 +0100 (BST)
Message-ID: <4DFA235A.1060301@apache.org>
Date: Thu, 16 Jun 2011 16:38:02 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Is this just bluster?
References: <BANLkTik1_g4Pt7xgWmz8GvwC=wEvngn++g@mail.gmail.com>
In-Reply-To: <BANLkTik1_g4Pt7xgWmz8GvwC=wEvngn++g@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p5GFc3Qb021151
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 16/06/11 14:49, David Scott Williams wrote:
> http://gigaom.com/cloud/lexisnexis-open-sources-its-hadoop-killer/
>

It's interesting that they decided they had to become open source to 
survive. It's the Linux effect: it's not that it's better than Solaris 
was, it just got the momentum up.

A strength of Hadoop is that it does have layers on it, and a lot of the 
interesting stuff is above the basic layer -Mahout, Pig, Hive, Hama, 
etc, and you can plug in things: filesystems, schedulers, HDFS placers. 
While we can debate what "compatible" means, by implementing the APIs 
that the higher layers use, MapR and hence EMC's products can run those 
higher layers. HPCC looks to be a completely new ecosystem.

Oh, and the license is AGPL, which complicates any external-facing web 
app way more than even GPL does. Good for business models (you can pay 
for the alternate license), but not ideal for takeup.

HPCC do a good comparision page here, seems quite unbiased

http://hpccsystems.com/why-HPCC/HPCC-vs-hadoop

Regarding performance, I haven't seen any new terasort numbers for a 
while. Whoever next brings up a 1000+ node cluster should publish them.

As HPCC say: "In practice, HPCC configurations require significantly 
fewer nodes to provide the same processing performance as a Hadoop 
cluster. Sizing of clusters may depend however on the overall storage 
requirements for the distributed file system."

That means if your cluster is driven by storage demands, that fixes node 
size more than CPU issues (though if you need less CPUs, that's capex 
and opex savings or the opportunity to do other things with CPU time)

From general-return-3824-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 15:41:59 2011
Return-Path: <general-return-3824-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7994861AC
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 15:41:59 +0000 (UTC)
Received: (qmail 75608 invoked by uid 500); 16 Jun 2011 15:41:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75557 invoked by uid 500); 16 Jun 2011 15:41:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75549 invoked by uid 99); 16 Jun 2011 15:41:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 15:41:57 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 15:41:51 +0000
Received: by wyb40 with SMTP id 40so1751346wyb.35
        for <general@hadoop.apache.org>; Thu, 16 Jun 2011 08:41:30 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.221.103 with SMTP id q81mr1014220wep.83.1308238890738;
 Thu, 16 Jun 2011 08:41:30 -0700 (PDT)
Received: by 10.216.30.212 with HTTP; Thu, 16 Jun 2011 08:41:30 -0700 (PDT)
In-Reply-To: <BANLkTin2YDAqSG3Uk_hLSQziRJsW1QRFeA@mail.gmail.com>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
	<BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com>
	<090F2993-B9AA-410F-AD99-7AC047FFC974@apache.org>
	<BANLkTi=fkguAtbfi9Wk9CPm8JmFa2Hti3Q@mail.gmail.com>
	<54668CDE-443D-4AF1-B17A-0497D0CDCC8D@yahoo-inc.com>
	<BANLkTimTGcEU29e_uLB52uc9CeuSkK__6w@mail.gmail.com>
	<EFB94DDE-9E2C-42D9-99C2-2FEB742535D1@yahoo-inc.com>
	<24C673BE-B252-4643-BA96-EF6E4A9D978D@oracle.com>
	<BANLkTikqc18JbPS1XTiysXtH5XtpsBu51w@mail.gmail.com>
	<61D08301-F6A9-4B9B-9B78-60D7D3B731A4@Holsman.NET>
	<BANLkTim46ifZus_ZoKEJ_DC8SQvvkDR0iQ@mail.gmail.com>
	<BANLkTin2YDAqSG3Uk_hLSQziRJsW1QRFeA@mail.gmail.com>
Date: Thu, 16 Jun 2011 08:41:30 -0700
Message-ID: <BANLkTikOZOYX25s6xJGe8Sn2ebc88aAbYA@mail.gmail.com>
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Thu, Jun 16, 2011 at 8:02 AM, Owen O'Malley <omalley@apache.org> wrote:
> On Wed, Jun 15, 2011 at 11:35 PM, Eric Sammer <esammer@cloudera.com> wrote:
>
> I think the question is can one call it "Hadoop?" Note I'm *not*
>> saying "Apache Hadoop," just "Hadoop" when the derived work is actually
>> derived (to any degree, as Craig R pointed out). Apache Hadoop always and
>> forever means the bits voted on by the PMC - no vendor can claim that - but
>> there does appear to be plenty of prior examples of "reasonable" use of ASF
>> (and other OSS organization) project names in clearly derived works.
>>
>
> Thank you, Eric, for demonstrating why we are fixing it. Apache owns the
> Hadoop trademark. Hadoop is PRECISELY the same as Apache Hadoop. They are
> two names for the same thing. If the Hadoop PMC were to fail to enforce
> that, the Apache board would remove us en masse from the PMC.
>

By this logic the Apache board should en masse remove the PMC from the
HTTP Server, Subversion and Tomcat because they've failed to enforce
Red Hat, Novell, Ubuntu and others to stop calling them Apache X.
Clearly that hasn't happened. Let's let the Apache board speak for
itself.

Thanks,
Eli

From general-return-3825-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 15:45:57 2011
Return-Path: <general-return-3825-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 860EB6924
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 15:45:57 +0000 (UTC)
Received: (qmail 83836 invoked by uid 500); 16 Jun 2011 15:45:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83781 invoked by uid 500); 16 Jun 2011 15:45:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83769 invoked by uid 99); 16 Jun 2011 15:45:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 15:45:55 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 216.145.54.173 is neither permitted nor denied by domain of sradia@yahoo-inc.com)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 15:45:51 +0000
Received: from oceanfarearth-lm.corp.yahoo.com (oceanfarearth-lm.corp.yahoo.com [10.72.113.156])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p5GFis3X052946;
	Thu, 16 Jun 2011 08:44:54 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1308239094;
	bh=dtkMdDflQNJB7rDsxDk0m/OGobSMPVs1TnR/CqNDF84=;
	h=Cc:Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=S0B6dya6FtORCbyXnkDEOwiGlX39c/3PQJObCFrKYN/XlzhiyuJzRFFehmdxq5L9D
	 oa/LjlkrHobIQnM28rWW5+4Deu3cWJGO8wtZDiinyUv/yGJCyPIuPh8ejuwT93Nx/s
	 FO56Ld0o41u3KT73UoQV1bwE/km15AUlR9DZFYNk=
Cc: "apurtell@apache.org" <apurtell@apache.org>
Message-Id: <06A09687-D5EC-4585-BC41-9F8DB2B8994E@yahoo-inc.com>
From: Sanjay Radia <sradia@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <8B7097C4-597D-4BA4-8C69-C48F5A571AEF@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: LimitedPrivate and HBase
Date: Thu, 16 Jun 2011 08:44:51 -0700
References: <457011.690.qm@web65508.mail.ac4.yahoo.com> <F0CA2186-6E63-459D-ABFF-21863EE018E5@yahoo-inc.com> <BANLkTikYuO0nRHz3UWZFZU245dMw82Y08Q@mail.gmail.com> <8B7097C4-597D-4BA4-8C69-C48F5A571AEF@yahoo-inc.com>
X-Mailer: Apple Mail (2.936)

I have updated
  https://issues.apache.org/jira/browse/HADOOP-7391
with a html file that is for adding as an overview.html to the  
classification package.
The text is mostly what was in HADOOP-5073; i have added 3 FAQs.


From general-return-3826-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 16:02:38 2011
Return-Path: <general-return-3826-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CC4416ADD
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 16:02:38 +0000 (UTC)
Received: (qmail 37629 invoked by uid 500); 16 Jun 2011 16:02:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37572 invoked by uid 500); 16 Jun 2011 16:02:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37564 invoked by uid 99); 16 Jun 2011 16:02:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 16:02:36 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 16:02:30 +0000
Received: by bwz8 with SMTP id 8so2336290bwz.35
        for <general@hadoop.apache.org>; Thu, 16 Jun 2011 09:02:10 -0700 (PDT)
Received: by 10.204.74.70 with SMTP id t6mr856138bkj.21.1308240130216; Thu, 16
 Jun 2011 09:02:10 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.79 with HTTP; Thu, 16 Jun 2011 09:01:50 -0700 (PDT)
In-Reply-To: <4DF9CC6C.4050306@yahoo-inc.com>
References: <BANLkTikU1LxFc6aF_dV6Fbu4E-fxf3c7ww@mail.gmail.com>
 <CA1B916C.78CF%jnaisbit@yahoo-inc.com> <BANLkTincXUjoeiXhp8tHn4+qLs642WkbMA@mail.gmail.com>
 <BANLkTinVKj9DE+UoLRfhjVNA=mBR0k5yFg@mail.gmail.com> <4DF9CC6C.4050306@yahoo-inc.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Thu, 16 Jun 2011 09:01:50 -0700
Message-ID: <BANLkTinu2wzxcqyjWE64ExrGS92PTUGgng@mail.gmail.com>
Subject: Re: HADOOP-7106 (project unsplit) this weekend
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d784793e150d04a5d66460
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d784793e150d04a5d66460
Content-Type: text/plain; charset=ISO-8859-1

On Thu, Jun 16, 2011 at 2:27 AM, Ranjit Mathew <ranjit@yahoo-inc.com> wrote:

> On 06/13/2011 09:33 PM, Todd Lipcon wrote:
>
>> Oops, sorry about that one. I will take care of that in about 30 minutes
>>> (just headed out the door now to catch a train). If someone else with
>>> commit
>>> access wants to, you just need to propset the externals to point to th
>>> new
>>> common/trunk/common/src/test/**bin instead of the old location.
>>>
>>>
>>>  Fixed the svn:externals
>>
>
> Does it make sense for "test-patch.sh" and "smart-apply-patch.sh" to
> remain "external" now that they're within the same project?
>

Probably not long-term... but right now the various subproject hudson builds
only check out the hdfs/ or mapreduce/ subtree, so rooting the test scripts
above that level would require a bit of reconfiguration.


>
> Currently on every "svn up" for trunk I get:
> -------------------------- 8< --------------------------
> $ svn up
>
> Fetching external item into 'hdfs/src/test/bin'
> External at revision 1136333.
>
>
> Fetching external item into 'mapreduce/src/test/bin'
> External at revision 1136333.
>
> At revision 1136333.
> -------------------------- 8< --------------------------
>
> after a little pause.
>
> Ranjit
>
> PS: I'm using "svn, version 1.6.16 (r1073529)" on Fedora 14/x86-32.
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--0016e6d784793e150d04a5d66460--

From general-return-3827-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 16:05:50 2011
Return-Path: <general-return-3827-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2D1BF6599
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 16:05:50 +0000 (UTC)
Received: (qmail 51121 invoked by uid 500); 16 Jun 2011 16:05:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51058 invoked by uid 500); 16 Jun 2011 16:05:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51026 invoked by uid 99); 16 Jun 2011 16:05:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 16:05:49 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 16:05:43 +0000
Received: by wwi18 with SMTP id 18so1803482wwi.29
        for <general@hadoop.apache.org>; Thu, 16 Jun 2011 09:05:23 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.229.150 with SMTP id h22mr1124466weq.68.1308240323274;
 Thu, 16 Jun 2011 09:05:23 -0700 (PDT)
Received: by 10.216.30.212 with HTTP; Thu, 16 Jun 2011 09:05:23 -0700 (PDT)
In-Reply-To: <EFB94DDE-9E2C-42D9-99C2-2FEB742535D1@yahoo-inc.com>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
	<BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com>
	<090F2993-B9AA-410F-AD99-7AC047FFC974@apache.org>
	<BANLkTi=fkguAtbfi9Wk9CPm8JmFa2Hti3Q@mail.gmail.com>
	<54668CDE-443D-4AF1-B17A-0497D0CDCC8D@yahoo-inc.com>
	<BANLkTimTGcEU29e_uLB52uc9CeuSkK__6w@mail.gmail.com>
	<EFB94DDE-9E2C-42D9-99C2-2FEB742535D1@yahoo-inc.com>
Date: Thu, 16 Jun 2011 09:05:23 -0700
Message-ID: <BANLkTinrqU8UUgNkNt6Wke8z=e_h10JaQA@mail.gmail.com>
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
From: Eli Collins <eli@cloudera.com>
To: Matthew Foley <mattf@yahoo-inc.com>
Cc: "general@hadoop.apache.org" <general@hadoop.apache.org>, 
	"trademarks@apache.org" <trademarks@apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Wed, Jun 15, 2011 at 6:17 PM, Matthew Foley <mattf@yahoo-inc.com> wrote:
> I tend to agree with what I think you are saying, that
> =A0 =A0 =A0 =A0* applying a small-number-of-patches that are
> =A0 =A0 =A0 =A0* for high-severity-bug-fixes, and
> =A0 =A0 =A0 =A0* have been Apache-Hadoop-committed
> to an Apache Hadoop release should not demote the result to a "derived wo=
rk".
> However, if so many patches are applied that the result cannot be meaning=
fully
> correlated with a specific Apache Hadoop release, then it probably has
> become a derived work.
>

This is one reason why I think the definition of derived work in the
draft of the wiki is way too broad. Something that's nothing like
Hadoop at all but includes a Hadoop jar is given the same label as
something with a single security patch. I think we can come up with a
more useful definition of derived work. If we do that would help us
draw the distinction between:
1. An Apache Hadoop release voted on the PMC, bit-for-bit identical
2. An Apache Hadoop release + backports (eg say per the above
definition of backport)
3. Something that is powered by Hadoop (eg HBase)
4. Something that is not Hadoop nor powered by Hadoop (eg the way tc
Server is not powered by Apache Tomcat)

Note that the current document does not make an exception for security
patches. I and Owen made this suggestion on this thread but the
writeup we are voting on makes no such exception.

> But how do we draw a meaningful line across that big gray area? =A0That's=
 why I'd like to
> see specific text from one of the other projects you cited as an example.
>

Googling didn't turn up anything in their public archives. This was in
an email exchange I had with Shane several years ago. Hopefully their
PMC can chime in.

Thanks,
Eli

From general-return-3828-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 17:18:26 2011
Return-Path: <general-return-3828-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9BDC26124
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 17:18:26 +0000 (UTC)
Received: (qmail 10321 invoked by uid 500); 16 Jun 2011 17:18:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 10266 invoked by uid 500); 16 Jun 2011 17:18:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 10258 invoked by uid 99); 16 Jun 2011 17:18:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 17:18:24 +0000
X-ASF-Spam-Status: No, hits=3.3 required=5.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 216.145.54.173 is neither permitted nor denied by domain of mattf@yahoo-inc.com)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 17:18:18 +0000
Received: from SP2-EX07CAS02.ds.corp.yahoo.com (sp2-ex07cas02.corp.sp2.yahoo.com [98.137.59.38])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p5GHH4M9088420;
	Thu, 16 Jun 2011 10:17:04 -0700 (PDT)
Received: from SP2-EX07VS03.ds.corp.yahoo.com ([98.137.59.32]) by
 SP2-EX07CAS02.ds.corp.yahoo.com ([98.137.59.38]) with mapi; Thu, 16 Jun 2011
 10:17:04 -0700
From: Matthew Foley <mattf@yahoo-inc.com>
To: Eric Sammer <esammer@cloudera.com>,
        "general@hadoop.apache.org"
	<general@hadoop.apache.org>
CC: Matthew Foley <mattf@yahoo-inc.com>,
        "trademarks@apache.org"
	<trademarks@apache.org>
Date: Thu, 16 Jun 2011 10:17:03 -0700
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
Thread-Topic: [VOTE] Shall we adopt the "Defining Hadoop" page
Thread-Index: AcwsSTWTIYQJQZ7XTQmfTy/j+GTvVg==
Message-ID: <A9F14549-E299-4964-8B60-7DB52B65F216@yahoo-inc.com>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
 <BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com>
 <090F2993-B9AA-410F-AD99-7AC047FFC974@apache.org>
 <BANLkTi=fkguAtbfi9Wk9CPm8JmFa2Hti3Q@mail.gmail.com>
 <54668CDE-443D-4AF1-B17A-0497D0CDCC8D@yahoo-inc.com>
 <BANLkTimTGcEU29e_uLB52uc9CeuSkK__6w@mail.gmail.com>
 <EFB94DDE-9E2C-42D9-99C2-2FEB742535D1@yahoo-inc.com>
 <24C673BE-B252-4643-BA96-EF6E4A9D978D@oracle.com>
 <BANLkTikqc18JbPS1XTiysXtH5XtpsBu51w@mail.gmail.com>
 <61D08301-F6A9-4B9B-9B78-60D7D3B731A4@Holsman.NET>
 <BANLkTim46ifZus_ZoKEJ_DC8SQvvkDR0iQ@mail.gmail.com>
In-Reply-To: <BANLkTim46ifZus_ZoKEJ_DC8SQvvkDR0iQ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_A9F14549E29949648B607DB52B65F216yahooinccom_"
MIME-Version: 1.0

--_000_A9F14549E29949648B607DB52B65F216yahooinccom_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi Eric,
sorry, but drawing a distinction between "Hadoop" and "Apache Hadoop" canno=
t be done, under general trademark usage nor the Apache Trademark Policy.  =
Trademark usage is a specialized language just like a programming language,=
 and that usage violates the intended semantics of the trademark.

--Matt


On Jun 15, 2011, at 11:35 PM, Eric Sammer wrote:

On Wed, Jun 15, 2011 at 9:47 PM, Ian Holsman <hadoop@holsman.net<mailto:had=
oop@holsman.net>> wrote:

so yes .. even a simple patch makes it derived, because it is different.

...and a "dervied work" is fine. Nothing inherently wrong with the term der=
ived. I think the question is can one call it "Hadoop?" Note I'm *not* sayi=
ng "Apache Hadoop," just "Hadoop" when the derived work is actually derived=
 (to any degree, as Craig R pointed out). Apache Hadoop always and forever =
means the bits voted on by the PMC - no vendor can claim that - but there d=
oes appear to be plenty of prior examples of "reasonable" use of ASF (and o=
ther OSS organization) project names in clearly derived works. I do agree t=
here should be a policy and it needs to be universally applied to be fair t=
o all involved.

Not to kick up the compatibility dust storm again, but people will always c=
laim crazy stuff that may or may not be true. We should just ignore it. Any=
 day of the week someone is claiming XYZ compatible either explicitly or im=
plicitly (as in client libraries for Foo Project). For cases where a vendor=
 makes a claim that isn't true, users will ask, we'll clarify that Apache m=
akes no guarantees of derived work compatibility and doesn't certify anythi=
ng (and specifically does the opposite - *NO* guarantees or warranties).

Example uses I think should be fine / acceptable:

YDH (even though it no longer exists, it's a good example) and Y!'s use of =
Hadoop
Facebook Hadoop
Hadoop at eBay
Hadoop at LinkedIn
IBM's use of Hadoop
and yes, CDH*

Even if some / all of the above modify at least a single bit (and may *tech=
nically* be derived works) everyone understands what they mean. As for the =
confusion, the OSS community has always just said "oh, they patch some stuf=
f, you should probably ask them" when confronted with vendor modified versi=
ons of upstream projects; I've been involved in many of those upstream proj=
ects, including a Linux distro (downstream). We should always be polite to =
downstream users in redirecting them, but I think redirecting them is fine.=
 It's not confusing to users in my experience (we can make it a FAQ or some=
thing and just point people there) as RedHat, Novell, Oracle, IBM, and many=
 other vendors have been happily[1] coexisting with their upstream counterp=
arts for a long time.

I believe we (the collective Apache Hadoop community including those that r=
edistribute Hadoop bits in various forms) should focus on producing regular=
, quality releases in a cooperative and constructive environment, and conti=
nue to require vendors to provide the proper attribution and license inform=
ation. This is in everyone's interest, vendors and direct users alike.

*Disclosure: I work for Cloudera and I think this should apply to anyone an=
d everyone, including my employer (with whom I obviously do not clear email=
s. :))

[1] OK, maybe not always "happily" but mostly so. You know what I mean.

Thanks to Steve L and others for their hard work on this one.
(Sorry for the long email.)

--
Eric Sammer
twitter: esammer
data: www.cloudera.com<http://www.cloudera.com/>


--_000_A9F14549E29949648B607DB52B65F216yahooinccom_--

From general-return-3829-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 17:40:15 2011
Return-Path: <general-return-3829-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8A47363A0
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 17:40:15 +0000 (UTC)
Received: (qmail 61751 invoked by uid 500); 16 Jun 2011 17:40:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61650 invoked by uid 500); 16 Jun 2011 17:40:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61635 invoked by uid 99); 16 Jun 2011 17:40:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 17:40:13 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 216.145.54.173 is neither permitted nor denied by domain of mattf@yahoo-inc.com)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 17:40:05 +0000
Received: from SP2-EX07CAS04.ds.corp.yahoo.com (sp2-ex07cas04.corp.sp2.yahoo.com [98.137.59.5])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p5GHcmQZ095372;
	Thu, 16 Jun 2011 10:38:48 -0700 (PDT)
Received: from SP2-EX07VS03.ds.corp.yahoo.com ([98.137.59.32]) by
 SP2-EX07CAS04.ds.corp.yahoo.com ([98.137.59.5]) with mapi; Thu, 16 Jun 2011
 10:38:48 -0700
From: Matthew Foley <mattf@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: Matthew Foley <mattf@yahoo-inc.com>,
        "trademarks@apache.org"
	<trademarks@apache.org>
Date: Thu, 16 Jun 2011 10:38:48 -0700
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
Thread-Topic: [VOTE] Shall we adopt the "Defining Hadoop" page
Thread-Index: AcwsTD9f4DRl9DlqSru4cXZqF8JmUA==
Message-ID: <26BCE485-DD1A-45C5-A328-C03CEC5BC297@yahoo-inc.com>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
 <BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com>
 <090F2993-B9AA-410F-AD99-7AC047FFC974@apache.org>
 <BANLkTi=fkguAtbfi9Wk9CPm8JmFa2Hti3Q@mail.gmail.com>
 <54668CDE-443D-4AF1-B17A-0497D0CDCC8D@yahoo-inc.com>
 <BANLkTimTGcEU29e_uLB52uc9CeuSkK__6w@mail.gmail.com>
 <EFB94DDE-9E2C-42D9-99C2-2FEB742535D1@yahoo-inc.com>
 <BANLkTinrqU8UUgNkNt6Wke8z=e_h10JaQA@mail.gmail.com>
In-Reply-To: <BANLkTinrqU8UUgNkNt6Wke8z=e_h10JaQA@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

After writing my note to Eric, I realize that Eli and I are guilty of the s=
ame attempt
to use legal terminology in an engineering context.  Craig Russell is absol=
utely right.
If you change one bit, it is a "derived work".

However, we can still allow the trademark to be applied to that work, if it=
=20
meets licensing criteria.  So what we are arguing about is, "Where is the b=
oundary
line between something we are willing to call 'Apache Hadoop' and something
that must be called 'Product XYZ Powered by Apache Hadoop'?"

I'm in favor of a very strict definition.  It needs to be really, really cl=
ose to a
PMC-approved release.  But I'm open to the argument that a small number
of security patches could be necessary for a viable commercial product,
and that shouldn't necessarily prevent it from using the trademark.

But I suggest we stop focusing on the term "derived work".  Note that the=20
"Defining Apache Hadoop" draft document we are voting on doesn't use=20
that term.

--Matt


On Jun 16, 2011, at 9:05 AM, Eli Collins wrote:

On Wed, Jun 15, 2011 at 6:17 PM, Matthew Foley <mattf@yahoo-inc.com> wrote:
> I tend to agree with what I think you are saying, that
>        * applying a small-number-of-patches that are
>        * for high-severity-bug-fixes, and
>        * have been Apache-Hadoop-committed
> to an Apache Hadoop release should not demote the result to a "derived wo=
rk".
> However, if so many patches are applied that the result cannot be meaning=
fully
> correlated with a specific Apache Hadoop release, then it probably has
> become a derived work.
>=20

This is one reason why I think the definition of derived work in the
draft of the wiki is way too broad. Something that's nothing like
Hadoop at all but includes a Hadoop jar is given the same label as
something with a single security patch. I think we can come up with a
more useful definition of derived work. If we do that would help us
draw the distinction between:
1. An Apache Hadoop release voted on the PMC, bit-for-bit identical
2. An Apache Hadoop release + backports (eg say per the above
definition of backport)
3. Something that is powered by Hadoop (eg HBase)
4. Something that is not Hadoop nor powered by Hadoop (eg the way tc
Server is not powered by Apache Tomcat)

Note that the current document does not make an exception for security
patches. I and Owen made this suggestion on this thread but the
writeup we are voting on makes no such exception.

> But how do we draw a meaningful line across that big gray area?  That's w=
hy I'd like to
> see specific text from one of the other projects you cited as an example.
>=20

Googling didn't turn up anything in their public archives. This was in
an email exchange I had with Shane several years ago. Hopefully their
PMC can chime in.

Thanks,
Eli


From general-return-3830-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 17:58:50 2011
Return-Path: <general-return-3830-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C03C66083
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 17:58:50 +0000 (UTC)
Received: (qmail 98459 invoked by uid 500); 16 Jun 2011 17:58:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98408 invoked by uid 500); 16 Jun 2011 17:58:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 55266 invoked by uid 99); 16 Jun 2011 06:30:12 -0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 345493.4040.bm@omp1051.mail.bf1.yahoo.com
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1308205781; bh=mEsbP2niN6ecdD5ewfyPQxNB3HFWYu+RnlMk8w2pSnU=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=6sC+WwgySIszpRFNzT4jnP1JuVIOvRHvtEOCwvy8EwlJ0GD2WuPmnVmNn5AsT11riJdRh4Wlay2Bq+bAIIsViOtKMG6a4WkH75Ry4dG4MM6Frs9wgMQSl7X9hlyt9/Ei/na2n0kjVaTdBsVwL8mN2xKQ1Xkbbkm2hGNCGfOYXaQ=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=cv/oi4rc2gMocav3X4V4Bt9pkscRI9rKTmJ4x4KvkHQ9RKShvEve5gXt584ByqDrpCHyP/5qIih60Gvb9qJXNwffxnzgdT0MByQ6J47XFY2C/ZFIh8kUp4WoSHWP9S47b+kXJEGKfAYdQHNx8N8q4Fdd8skFE3cyPpAqqTQ9wws=;
Message-ID: <144508.16148.qm@web161714.mail.bf1.yahoo.com>
X-YMail-OSG: 0Ib52xoVM1kZ5R7w4X41cEF1a9zxsrH8k1YXZJNx.5r1Wlo
 UJi0bt_qhd.3KkKF2SUIA8LXUHamCVs4TCDysX_zuSQIxoLQWq3WN1sQPUgA
 y._U1CVs_PorqLoFBTcX5rhTc6qlP1f73h_QHr77amexf23panG53nl8ide8
 CGo632EoMwjhS7Jn_wh6e9Ht_S6gVhDmV2dvsO2ARP.Yefl7xmJmbzlxClNX
 xUQpzjt0WrIA.y6pJ0bwWPQeIDo_F.kO7BJ8YyXaW99t3fgDTP_VBDhGpeqW
 qe2kUVQdBhuJ8pgJY4CRyB6xncsMKAq3X_eKGndgkytm2WnJ5KkWbmnq_b9n
 pn4pYbEDxVsAU7ymuk88JZsdR0zGNrruwDp6AAuzBHAc-
X-Mailer: YahooMailRC/570 YahooMailWebService/0.8.111.304355
Date: Wed, 15 Jun 2011 23:29:41 -0700 (PDT)
From: Vivekanand Vellanki <ganjuga@yahoo.com>
Subject: [VOTE] Powered  by Logo
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-611399123-1308205781=:16148"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-611399123-1308205781=:16148
Content-Type: text/plain; charset=us-ascii

4, 2, 6, 1, 3, 5

--0-611399123-1308205781=:16148--

From general-return-3831-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 17:58:54 2011
Return-Path: <general-return-3831-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EE10A6096
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 17:58:53 +0000 (UTC)
Received: (qmail 99939 invoked by uid 500); 16 Jun 2011 17:58:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99864 invoked by uid 500); 16 Jun 2011 17:58:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 82521 invoked by uid 99); 16 Jun 2011 12:43:09 -0000
X-ASF-Spam-Status: No, hits=3.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of anuragc1978@gmail.com designates 209.85.210.176 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=PEPdiGr3YllfjjiwdFXaQNuoRUbFowykKEekunaqyF4=;
        b=oODlqad6dqA7EMKyX6o0cDsKS1u1++e+xr0DwOZt8fHcZz6bdUr2x16dkdn8MzDHnU
         QW6inPc7Jp/sDA4Hlc/jsAix/S9NKANyMBjzgFN1Z57/VOWrxwLcty3e+aH3n4SLW3g6
         1eTpDYreW5X3pwyRlaIINCnENhnbgvOreWlu8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=cxzEBHatJONl6g2GTMb4mMqJ4HbIsFUHZIB26vxao5ZfIk+/5P2eYVOpLgZT9L97nu
         ujB9GAE6i7QpW/qQD/kGfdWS71G5OQERBuAvllR4jZoVsM+S6q3W3f5vUFKOI1fr7OLy
         tUVtJx0XMlX8X0CP0gDGG6PAh0tLFZlInyRNo=
MIME-Version: 1.0
Date: Thu, 16 Jun 2011 18:12:41 +0530
Message-ID: <BANLkTikti0pSnq7LUr7WoOrp+PE86MVw3g@mail.gmail.com>
Subject: [VOTE] Powered by Logo X
From: anuragg choudhary <anuragc1978@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f6cee6e0ea0904a5d39a42

--001485f6cee6e0ea0904a5d39a42
Content-Type: text/plain; charset=ISO-8859-1

4, 2, 6, 1, 3, 5

--001485f6cee6e0ea0904a5d39a42--

From general-return-3832-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 17:59:01 2011
Return-Path: <general-return-3832-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C26B360AF
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 17:59:01 +0000 (UTC)
Received: (qmail 1498 invoked by uid 500); 16 Jun 2011 17:58:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1439 invoked by uid 500); 16 Jun 2011 17:58:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 24820 invoked by uid 99); 16 Jun 2011 17:27:28 -0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
From: "Lawrence Rosen" <lrosen@rosenlaw.com>
To: <trademarks@apache.org>,
	"'Matthew Foley'" <mattf@yahoo-inc.com>
Cc: <general@hadoop.apache.org>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>	<BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com>	<090F2993-B9AA-410F-AD99-7AC047FFC974@apache.org>	<BANLkTi=fkguAtbfi9Wk9CPm8JmFa2Hti3Q@mail.gmail.com>	<54668CDE-443D-4AF1-B17A-0497D0CDCC8D@yahoo-inc.com>	<BANLkTimTGcEU29e_uLB52uc9CeuSkK__6w@mail.gmail.com>	<EFB94DDE-9E2C-42D9-99C2-2FEB742535D1@yahoo-inc.com> <BANLkTinrqU8UUgNkNt6Wke8z=e_h10JaQA@mail.gmail.com>
In-Reply-To: <BANLkTinrqU8UUgNkNt6Wke8z=e_h10JaQA@mail.gmail.com>
Subject: RE: [VOTE] Shall we adopt the "Defining Hadoop" page
Date: Thu, 16 Jun 2011 10:27:05 -0700
Message-ID: <04e601cc2c4a$9d293fb0$d77bbf10$@com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
X-Mailer: Microsoft Office Outlook 12.0
Thread-Index: AcwsP0Kr/5gwYMLAS2OUV8YY3JwSXwACvm8g
Content-Language: en-us

I'm very confused by this thread. What does trademark law have to do =
with
derivative work analysis under copyright law? Is there something =
specific in
our FAQ or trademark policy that confuses these concepts and that we =
should
clean up?

/Larry

Please cc: trademarks@ because I'm not on the other lists.


> -----Original Message-----
> From: Eli Collins [mailto:eli@cloudera.com]
> Sent: Thursday, June 16, 2011 9:05 AM
> To: Matthew Foley
> Cc: general@hadoop.apache.org; trademarks@apache.org
> Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
>=20
> On Wed, Jun 15, 2011 at 6:17 PM, Matthew Foley <mattf@yahoo-inc.com>
> wrote:
> > I tend to agree with what I think you are saying, that
> > =A0 =A0 =A0 =A0* applying a small-number-of-patches that are
> > =A0 =A0 =A0 =A0* for high-severity-bug-fixes, and
> > =A0 =A0 =A0 =A0* have been Apache-Hadoop-committed
> > to an Apache Hadoop release should not demote the result to a
> "derived work".
> > However, if so many patches are applied that the result cannot be
> meaningfully
> > correlated with a specific Apache Hadoop release, then it probably
> has
> > become a derived work.
> >
>=20
> This is one reason why I think the definition of derived work in the
> draft of the wiki is way too broad. Something that's nothing like
> Hadoop at all but includes a Hadoop jar is given the same label as
> something with a single security patch. I think we can come up with a
> more useful definition of derived work. If we do that would help us
> draw the distinction between:
> 1. An Apache Hadoop release voted on the PMC, bit-for-bit identical
> 2. An Apache Hadoop release + backports (eg say per the above
> definition of backport)
> 3. Something that is powered by Hadoop (eg HBase)
> 4. Something that is not Hadoop nor powered by Hadoop (eg the way tc
> Server is not powered by Apache Tomcat)
>=20
> Note that the current document does not make an exception for security
> patches. I and Owen made this suggestion on this thread but the
> writeup we are voting on makes no such exception.
>=20
> > But how do we draw a meaningful line across that big gray area?
> =A0That's why I'd like to
> > see specific text from one of the other projects you cited as an
> example.
> >
>=20
> Googling didn't turn up anything in their public archives. This was in
> an email exchange I had with Shane several years ago. Hopefully their
> PMC can chime in.
>=20
> Thanks,
> Eli



From general-return-3833-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 17:59:34 2011
Return-Path: <general-return-3833-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 548D360EB
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 17:59:34 +0000 (UTC)
Received: (qmail 3249 invoked by uid 500); 16 Jun 2011 17:59:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3129 invoked by uid 500); 16 Jun 2011 17:59:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3119 invoked by uid 99); 16 Jun 2011 17:59:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 17:59:32 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [192.220.47.202] (HELO rosenlaw.com) (192.220.47.202)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 17:59:26 +0000
Received: (qmail 4681 invoked by uid 12234); 16 Jun 2011 17:59:05 -0000
Received: from unknown (HELO Lawrencei) ([208.106.45.202])
          (envelope-sender <lrosen@rosenlaw.com>)
          by 192.220.47.202 (qmail-ldap-1.03) with SMTP
          for <general@hadoop.apache.org>; 16 Jun 2011 17:59:05 -0000
From: "Lawrence Rosen" <lrosen@rosenlaw.com>
To: <trademarks@apache.org>
Cc: <general@hadoop.apache.org>
Subject: Trademarks and Derivative Works
Date: Thu, 16 Jun 2011 10:59:09 -0700
Message-ID: <04f001cc2c4f$1815c460$48414d20$@com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_NextPart_000_04F1_01CC2C14.6BB6EC60"
X-Mailer: Microsoft Office Outlook 12.0
Thread-Index: AcwsTvz8+66DbjpfTs2sh+mV1G6vzQ==
Content-Language: en-us

------=_NextPart_000_04F1_01CC2C14.6BB6EC60
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit

Apache's trademarks, including the HADOOP and APACHE HADOOP trademarks, can
be applied to software distributed by Apache Software Foundation. Once
distributed by us, we permit that software to be redistributed by others.
(E.g., Best Buy can put APACHE HADOOP on their store shelves if they want
to.) Under default trademark law, those others can distribute HADOOP and
APACHE HADOOP only if it is a redistribution of *our* HADOOP or APACHE
HADOOP software. (That's why you can buy Jello Brand gelatin at Safeway.)
Those trademarks are our names for our software. ASF is the source and
origin of those software goods. Nobody else can apply those trademarks to
their own software.

 

That doesn't mean that the entire third party software distribution must be
*our* software, although to claim that such a larger distribution *is*
HADOOP or APACHE HADOOP would be trademark infringement. So the real
question we should be asking is:

 

How can we encourage third parties use our HADOOP and APACHE HADOOP
trademarks properly to indicate that their unique third party software
*contains, or works with, or includes, or is a plug-in for,  or is a larger
distribution of, or supports* APACHE HADOOP?

 

Companies use lots of marketing techniques to protect their trademarks and
simultaneously to let third parties help to advertise those trademarked
goods and generate goodwill for them. Apache should consider doing some of
the same things. It will be to our advantage to have HADOOP and APACHE
HADOOP software better known and widely used throughout the world. For that
purpose, we should be defining the rules we want to *encourage* third
parties to follow, not arguing about derivative work analysis or voting on
whether or not something is a trademark.

 

As you look around the web for the software products you buy, which
trademark marketing techniques do you like or dislike? Maybe those examples
will help us focus the discussion here about our HADOOP and APACHE HADOOP
trademarks.

 

Best,

 

/Larry

 

 

Lawrence Rosen

Rosenlaw & Einschlag, a technology law firm (www.rosenlaw.com) 

3001 King Ranch Road, Ukiah, CA 95482

Cell: 707-478-8932

Apache Software Foundation, member and counsel (www.apache.org) 

Open Web Foundation, board member (www.openwebfoundation.org) 

Stanford University, Instructor in Law

Author, Open Source Licensing: Software Freedom and Intellectual Property
Law (Prentice Hall 2004)

 


------=_NextPart_000_04F1_01CC2C14.6BB6EC60--


From general-return-3834-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 18:11:31 2011
Return-Path: <general-return-3834-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8E7ED6AC1
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 18:11:31 +0000 (UTC)
Received: (qmail 29917 invoked by uid 500); 16 Jun 2011 18:11:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29832 invoked by uid 500); 16 Jun 2011 18:11:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29824 invoked by uid 99); 16 Jun 2011 18:11:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 18:11:29 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 18:11:23 +0000
Received: by wyb40 with SMTP id 40so1921061wyb.35
        for <general@hadoop.apache.org>; Thu, 16 Jun 2011 11:11:03 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.245.4 with SMTP id n4mr1276277wer.83.1308247863519; Thu,
 16 Jun 2011 11:11:03 -0700 (PDT)
Received: by 10.216.30.212 with HTTP; Thu, 16 Jun 2011 11:11:03 -0700 (PDT)
In-Reply-To: <26BCE485-DD1A-45C5-A328-C03CEC5BC297@yahoo-inc.com>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
	<BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com>
	<090F2993-B9AA-410F-AD99-7AC047FFC974@apache.org>
	<BANLkTi=fkguAtbfi9Wk9CPm8JmFa2Hti3Q@mail.gmail.com>
	<54668CDE-443D-4AF1-B17A-0497D0CDCC8D@yahoo-inc.com>
	<BANLkTimTGcEU29e_uLB52uc9CeuSkK__6w@mail.gmail.com>
	<EFB94DDE-9E2C-42D9-99C2-2FEB742535D1@yahoo-inc.com>
	<BANLkTinrqU8UUgNkNt6Wke8z=e_h10JaQA@mail.gmail.com>
	<26BCE485-DD1A-45C5-A328-C03CEC5BC297@yahoo-inc.com>
Date: Thu, 16 Jun 2011 11:11:03 -0700
Message-ID: <BANLkTi=pR3WXyNGyRPamne9o5A7XZLzo=A@mail.gmail.com>
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Cc: Matthew Foley <mattf@yahoo-inc.com>, "trademarks@apache.org" <trademarks@apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Thu, Jun 16, 2011 at 10:38 AM, Matthew Foley <mattf@yahoo-inc.com> wrote=
:
> After writing my note to Eric, I realize that Eli and I are guilty of the=
 same attempt
> to use legal terminology in an engineering context. =A0Craig Russell is a=
bsolutely right.
> If you change one bit, it is a "derived work".
>
> However, we can still allow the trademark to be applied to that work, if =
it
> meets licensing criteria. =A0So what we are arguing about is, "Where is t=
he boundary
> line between something we are willing to call 'Apache Hadoop' and somethi=
ng
> that must be called 'Product XYZ Powered by Apache Hadoop'?"
>
> I'm in favor of a very strict definition. =A0It needs to be really, reall=
y close to a
> PMC-approved release. =A0But I'm open to the argument that a small number
> of security patches could be necessary for a viable commercial product,
> and that shouldn't necessarily prevent it from using the trademark.
>
> But I suggest we stop focusing on the term "derived work". =A0Note that t=
he
> "Defining Apache Hadoop" draft document we are voting on doesn't use
> that term.

See the section titled "Derivative Works". The term "derivative work"
is used throughout the document.  I think you're right that the key
point here is not what is and is not a derivative work, but what can
be called Hadoop.

Seems like the board should have an ASF-wide stance on what can be
called Apache X instead of doing this per-project.

Thanks,
Eli

From general-return-3835-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 18:32:32 2011
Return-Path: <general-return-3835-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A5B046378
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 18:32:32 +0000 (UTC)
Received: (qmail 81872 invoked by uid 500); 16 Jun 2011 18:32:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81735 invoked by uid 500); 16 Jun 2011 18:32:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81715 invoked by uid 99); 16 Jun 2011 18:32:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 18:32:30 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of owen.omalley@gmail.com designates 74.125.83.176 as permitted sender)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 18:32:23 +0000
Received: by pve37 with SMTP id 37so59485pve.35
        for <general@hadoop.apache.org>; Thu, 16 Jun 2011 11:32:02 -0700 (PDT)
Received: by 10.142.59.3 with SMTP id h3mr221572wfa.387.1308249121890;
        Thu, 16 Jun 2011 11:32:01 -0700 (PDT)
Received: from battlerock-lm.corp.yahoo.com (nat-dip4.cfw-a-gci.corp.yahoo.com [209.131.62.113])
        by mx.google.com with ESMTPS id g10sm1069624pbi.73.2011.06.16.11.31.59
        (version=SSLv3 cipher=OTHER);
        Thu, 16 Jun 2011 11:32:00 -0700 (PDT)
Subject: Re: Trademarks and Derivative Works
Mime-Version: 1.0 (Apple Message framework v1084)
Content-Type: text/plain; charset=us-ascii
From: Owen O'Malley <omalley@apache.org>
In-Reply-To: <04f001cc2c4f$1815c460$48414d20$@com>
Date: Thu, 16 Jun 2011 11:31:58 -0700
Cc: <trademarks@apache.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <B71A7B6B-02EB-4049-A8DD-2E8321F788EE@apache.org>
References: <04f001cc2c4f$1815c460$48414d20$@com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)


On Jun 16, 2011, at 10:59 AM, Lawrence Rosen wrote:

> Under default trademark law, those others can distribute HADOOP and
> APACHE HADOOP only if it is a redistribution of *our* HADOOP or APACHE
> HADOOP software. (That's why you can buy Jello Brand gelatin at =
Safeway.)
> Those trademarks are our names for our software. ASF is the source and
> origin of those software goods. Nobody else can apply those trademarks =
to
> their own software.

The problem is that a rapidly growing set of companies are distributing =
products that have never been released by Apache and calling them =
Hadoop. The rules from HTTPD, as I understand them, are that they allow =
artifacts to be called HTTPD that are releases plus patches that have =
been committed. With HTTPD that has a formal specification and a very =
large compatibility test suite, that works. For Hadoop without a formal =
specification or test suite, we simply can't handle companies calling =
things Hadoop that are thousands of patches away from our releases. We =
are trying to assert that only the Apache releases can be called Hadoop. =
That seems to be the best way to help the project ensure compatibility =
and prevent user confusion.

> It will be to our advantage to have HADOOP and APACHE
> HADOOP software better known and widely used throughout the world. For =
that
> purpose, we should be defining the rules we want to *encourage* third
> parties to follow, not arguing about derivative work analysis or =
voting on
> whether or not something is a trademark.

I want Hadoop to be used in as many products as possible. Having a FooCo =
product that is called "FooCo HugeInsights powered by Hadoop" is =
absolutely great. The question is just whether they can call something =
Hadoop if it isn't an Apache release.

-- Owen


From general-return-3836-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 19:23:08 2011
Return-Path: <general-return-3836-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C85116E52
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 19:23:08 +0000 (UTC)
Received: (qmail 83734 invoked by uid 500); 16 Jun 2011 19:23:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83680 invoked by uid 500); 16 Jun 2011 19:23:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83672 invoked by uid 99); 16 Jun 2011 19:23:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 19:23:07 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of msegel@navteq.com designates 204.120.70.37 as permitted sender)
Received: from [204.120.70.37] (HELO xmailfargo.navteq.com) (204.120.70.37)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 19:23:00 +0000
Received: from fargo-spamfw.navteq.com (navteq-spamfw.navteq.com [10.6.10.131])
	by xmailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id p5GIl82g017588
	for <general@hadoop.apache.org>; Thu, 16 Jun 2011 13:47:08 -0500
Received: from imailfargo.navteq.com (localhost [127.0.0.1])
	by fargo-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP
	id 3647F43601B9; Thu, 16 Jun 2011 14:22:39 -0500 (CDT)
Received: from imailfargo.navteq.com (imailfargo.navteq.com [10.6.10.130]) by fargo-spamfw.navteq.com with ESMTP id tfnbKdKAOertGcYD; Thu, 16 Jun 2011 14:22:39 -0500 (CDT)
Received: from hq-ex-ht01.ad.navteq.com ([10.8.222.51])
	by imailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id p5GJMcNX020923;
	Thu, 16 Jun 2011 14:22:38 -0500
Received: from hq-ex-mb03.ad.navteq.com ([fe80::c4dd:7b21:5c22:cfe4]) by
 hq-ex-ht01.ad.navteq.com ([fe80::55a6:5b94:5c00:fe39%12]) with mapi; Thu, 16
 Jun 2011 14:22:38 -0500
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "trademarks@apache.org" <trademarks@apache.org>
Date: Thu, 16 Jun 2011 14:22:37 -0500
Subject: RE: Trademarks and Derivative Works
Thread-Topic: Trademarks and Derivative Works
Thread-Index: AcwsU8Iu8sHEl+9VT4yYe59PrjLaqgAASklQ
Message-ID: <3798688F8784154192FDA3388F4C208101086D60@hq-ex-mb03.ad.navteq.com>
References: <04f001cc2c4f$1815c460$48414d20$@com>
 <B71A7B6B-02EB-4049-A8DD-2E8321F788EE@apache.org>
In-Reply-To: <B71A7B6B-02EB-4049-A8DD-2E8321F788EE@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1

Owen,=20
=46rom your response below you say the following:
"We are trying to assert that only the Apache releases can be called Hado=
op. That seems to be the best way to help the project ensure compatibilit=
y and prevent user confusion."
and
" I want Hadoop to be used in as many products as possible. Having a FooC=
o product that is called "FooCo HugeInsights powered by Hadoop" is absolu=
tely great. The question is just whether they can call something Hadoop i=
f it isn't an Apache release.

-- Owen"

Owen,=20
Unfortunately what you're saying is that you would only approve of compan=
ies that build their products on top of Apache's release and doesn't modi=
fy the Apache release.=20
To give you an example... if  Acme Risk Management Company sold a product=
 using Hadoop to do risk analysis on a bank's portfolio, they can only sa=
y "powered by Hadoop" if they build their application on top of Apache's =
release. But the minute they build their solution on top of anyone else, =
they would lose that right? So using Cloudera's release, which contains t=
hings outside of the official Apache release would disallow them?  Or if =
they make their own modifications to the underlying release which isn't p=
art of the official release, they could no longer make that claim?

This interpretation of  "powered by Hadoop" would unfortunately lead to a=
s many problems as it attempts to solve.
First, many choose Cloudera's release because they sell commercial suppor=
t. So in choosing Cloudera's release, they would lose the ability to say =
"powered by Hadoop".
This diminishes the branding message.

The Apache License allows for broad reuse and relicensing as long as the =
company complies with Apache's T's & C's.  Limiting the ability to say "p=
owered by Hadoop" means that they will say that their solution uses a com=
mercially supported  'derivative of Hadoop'.=20
In terms of legalese, good luck in trying to get them on a misuse of your=
 trademark.  Cloudera, EMC, MapRTech, Datastax all offer derivatives of H=
adoop. (I'm not forgetting about Yahoo!, but are they releasing their own=
 version as well?) The term Hadoop is used as a reference to Apache's Had=
oop release.

I hope that you start to see the dangers on taking a narrow approach in h=
ow you define Hadoop.=20

Just IMHO.

-Mike

-----Original Message-----
From: Owen O'Malley [mailto:omalley@apache.org]=20
Sent: Thursday, June 16, 2011 1:32 PM
To: general@hadoop.apache.org
Cc: trademarks@apache.org
Subject: Re: Trademarks and Derivative Works


On Jun 16, 2011, at 10:59 AM, Lawrence Rosen wrote:

> Under default trademark law, those others can distribute HADOOP and=20
> APACHE HADOOP only if it is a redistribution of *our* HADOOP or APACHE=20
> HADOOP software. (That's why you can buy Jello Brand gelatin at=20
> Safeway.) Those trademarks are our names for our software. ASF is the=20
> source and origin of those software goods. Nobody else can apply those=20
> trademarks to their own software.

The problem is that a rapidly growing set of companies are distributing p=
roducts that have never been released by Apache and calling them Hadoop. =
The rules from HTTPD, as I understand them, are that they allow artifacts=
 to be called HTTPD that are releases plus patches that have been committ=
ed. With HTTPD that has a formal specification and a very large compatibi=
lity test suite, that works. For Hadoop without a formal specification or=
 test suite, we simply can't handle companies calling things Hadoop that =
are thousands of patches away from our releases. We are trying to assert =
that only the Apache releases can be called Hadoop. That seems to be the =
best way to help the project ensure compatibility and prevent user confus=
ion.

> It will be to our advantage to have HADOOP and APACHE HADOOP software=20
> better known and widely used throughout the world. For that purpose,=20
> we should be defining the rules we want to *encourage* third parties=20
> to follow, not arguing about derivative work analysis or voting on=20
> whether or not something is a trademark.

I want Hadoop to be used in as many products as possible. Having a FooCo =
product that is called "FooCo HugeInsights powered by Hadoop" is absolute=
ly great. The question is just whether they can call something Hadoop if =
it isn't an Apache release.

-- Owen



The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.

From general-return-3837-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 20:28:18 2011
Return-Path: <general-return-3837-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 707F36781
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 20:28:18 +0000 (UTC)
Received: (qmail 8171 invoked by uid 500); 16 Jun 2011 20:28:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8116 invoked by uid 500); 16 Jun 2011 20:28:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8108 invoked by uid 99); 16 Jun 2011 20:28:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:28:16 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 216.145.54.173 is neither permitted nor denied by domain of eric14@yahoo-inc.com)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:28:11 +0000
Received: from [10.0.1.3] (snvvpn2-10-73-152-c94.hq.corp.yahoo.com [10.73.152.94])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p5GKRGaM046737
	for <general@hadoop.apache.org>; Thu, 16 Jun 2011 13:27:17 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1308256037;
	bh=ZXvTCco2yH7CdM+bXfFczIK1NSmfzkBsbqxJ6CRP52M=;
	h=Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:
	 Content-Transfer-Encoding:Message-Id:References:To;
	b=GtERTbXaSrvqxhSocUbXqoCSTOb7T8zxdK4oB61ovHkYvHRN6dpgDq7X2SC8//2kZ
	 l1H9z8Yvas+rNaUcDUH44XCB0+tjQjmsJ+TG28v4ze0B1ez8i8SVVJL8nAui3w3xYc
	 SemfFEoyaIUsaToOjEUNjJo5O2qc/vnnzoCOMExo=
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: [VOTE] Powered by Logo
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <CA1E574D.5394C%svenkat@yahoo-inc.com>
Date: Thu, 16 Jun 2011 13:27:16 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <B3143E6E-858A-4C77-AEBD-604021E29CE4@yahoo-inc.com>
References: <CA1E574D.5394C%svenkat@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

5,2

> On 6/15/11 8:49 AM, "Owen O'Malley" <omalley@apache.org> wrote:
>=20
>> All,
>>   We've had a wide range of entries for a powered by logo. I've put =
them all
>> on a page, here:
>>=20
>> http://people.apache.org/~omalley/hadoop-powered-by/
>>=20
>> Since there are a lot of contenders and we only want a single round =
of voting,
>> let's use single transferable vote ( STV
>> http://en.wikipedia.org/wiki/Single_transferable_vote). The important =
thing is
>> to pick the images *IN ORDER* that you would like them.
>>=20
>> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>>=20
>> In other words, I want option 4 most and option 6 least. With STV, =
you don't
>> need to worry about voting for an unpopular choice since your vote =
will
>> automatically roll over to your next choice.
>>=20
>> -- Owen
>>=20
>>=20
>>=20
>=20


From general-return-3838-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 21:29:18 2011
Return-Path: <general-return-3838-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E8D3D6282
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 21:29:17 +0000 (UTC)
Received: (qmail 77318 invoked by uid 500); 16 Jun 2011 21:13:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77272 invoked by uid 500); 16 Jun 2011 21:13:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77263 invoked by uid 99); 16 Jun 2011 21:13:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 21:13:21 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jwfbean@cloudera.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 21:13:16 +0000
Received: by wyb40 with SMTP id 40so2104318wyb.35
        for <general@hadoop.apache.org>; Thu, 16 Jun 2011 14:12:55 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.217.6.137 with SMTP id y9mr1417652wes.47.1308258774923; Thu,
 16 Jun 2011 14:12:54 -0700 (PDT)
Received: by 10.216.29.7 with HTTP; Thu, 16 Jun 2011 14:12:54 -0700 (PDT)
In-Reply-To: <B3143E6E-858A-4C77-AEBD-604021E29CE4@yahoo-inc.com>
References: <CA1E574D.5394C%svenkat@yahoo-inc.com>
	<B3143E6E-858A-4C77-AEBD-604021E29CE4@yahoo-inc.com>
Date: Thu, 16 Jun 2011 14:12:54 -0700
Message-ID: <BANLkTinVACaGghJcMLBcgmPhDme98URE4g@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: Jeff Bean <jwfbean@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf302ad6428dbaf504a5dabb50

--20cf302ad6428dbaf504a5dabb50
Content-Type: text/plain; charset=ISO-8859-1

5,6,2,4,3,1

On Thu, Jun 16, 2011 at 1:27 PM, Eric Baldeschwieler
<eric14@yahoo-inc.com>wrote:

> 5,2
>
> > On 6/15/11 8:49 AM, "Owen O'Malley" <omalley@apache.org> wrote:
> >
> >> All,
> >>   We've had a wide range of entries for a powered by logo. I've put them
> all
> >> on a page, here:
> >>
> >> http://people.apache.org/~omalley/hadoop-powered-by/
> >>
> >> Since there are a lot of contenders and we only want a single round of
> voting,
> >> let's use single transferable vote ( STV
> >> http://en.wikipedia.org/wiki/Single_transferable_vote). The important
> thing is
> >> to pick the images *IN ORDER* that you would like them.
> >>
> >> My vote (in order of course): 4, 1, 2, 3, 5, 6.
> >>
> >> In other words, I want option 4 most and option 6 least. With STV, you
> don't
> >> need to worry about voting for an unpopular choice since your vote will
> >> automatically roll over to your next choice.
> >>
> >> -- Owen
> >>
> >>
> >>
> >
>
>

--20cf302ad6428dbaf504a5dabb50--

From general-return-3839-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 23:12:50 2011
Return-Path: <general-return-3839-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6D8066070
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 16 Jun 2011 23:12:50 +0000 (UTC)
Received: (qmail 52891 invoked by uid 500); 16 Jun 2011 23:12:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52814 invoked by uid 500); 16 Jun 2011 23:12:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52806 invoked by uid 99); 16 Jun 2011 23:12:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 23:12:48 +0000
X-ASF-Spam-Status: No, hits=1.3 required=5.0
	tests=RCVD_IN_BL_SPAMCOP_NET,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of chad@cloudera.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 23:12:39 +0000
Received: by pwi16 with SMTP id 16so1370334pwi.35
        for <general@hadoop.apache.org>; Thu, 16 Jun 2011 16:12:18 -0700 (PDT)
Received: by 10.142.245.16 with SMTP id s16mr298508wfh.139.1308265938034;
        Thu, 16 Jun 2011 16:12:18 -0700 (PDT)
Received: from [172.29.12.55] (173-167-109-121-sfba.hfc.comcastbusiness.net [173.167.109.121])
        by mx.google.com with ESMTPS id x16sm2095858wfc.10.2011.06.16.16.12.15
        (version=SSLv3 cipher=OTHER);
        Thu, 16 Jun 2011 16:12:15 -0700 (PDT)
Message-ID: <4DFA8DCE.7060508@cloudera.com>
Date: Thu, 16 Jun 2011 16:12:14 -0700
From: Chad Metcalf <chad@cloudera.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:5.0) Gecko/20110603 Thunderbird/5.0b1
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Powered by Logo
References: <CA1E574D.5394C%svenkat@yahoo-inc.com> <B3143E6E-858A-4C77-AEBD-604021E29CE4@yahoo-inc.com>
In-Reply-To: <B3143E6E-858A-4C77-AEBD-604021E29CE4@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

5,2

Cheers
Chad

From general-return-3840-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 00:36:21 2011
Return-Path: <general-return-3840-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DDF6B6F7E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 17 Jun 2011 00:36:21 +0000 (UTC)
Received: (qmail 48032 invoked by uid 500); 17 Jun 2011 00:36:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47921 invoked by uid 500); 17 Jun 2011 00:36:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47913 invoked by uid 99); 17 Jun 2011 00:36:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 00:36:20 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 216.145.54.172 is neither permitted nor denied by domain of eric14@yahoo-inc.com)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 00:36:15 +0000
Received: from [10.0.1.3] (snvvpn2-10-73-152-c94.hq.corp.yahoo.com [10.73.152.94])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5H0ZOZX038544;
	Thu, 16 Jun 2011 17:35:25 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1308270925;
	bh=2uppDKpLS6/r3DqBeHWEUjLYhjY4Nu2+o3eHzPJrRJo=;
	h=Subject:Mime-Version:Content-Type:From:In-Reply-To:Date:Cc:
	 Content-Transfer-Encoding:Message-Id:References:To;
	b=j//3kRlGJsEo25PYkWvHMQcswOXVdSmIfqSTJdOjhQV8lmZ/jHyWvBtlPzce14pXw
	 TcYNFb/C4nhgPjKT+I4lfgSd6ZSAvSM1whr/hAt6ei06ZROndEmhSnG2vJQ6NLFh5L
	 ne2daD/Gwl2+q4Hh3C9BcllUrIMGsjFYAqISX2Vs=
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
Mime-Version: 1.0 (Apple Message framework v1084)
Content-Type: text/plain; charset=us-ascii
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <BANLkTi=pR3WXyNGyRPamne9o5A7XZLzo=A@mail.gmail.com>
Date: Thu, 16 Jun 2011 17:35:24 -0700
Cc: Matthew Foley <mattf@yahoo-inc.com>,
        "trademarks@apache.org" <trademarks@apache.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <5069B9B8-4255-4810-9A22-4D7081FA3A3B@yahoo-inc.com>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org> <BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com> <090F2993-B9AA-410F-AD99-7AC047FFC974@apache.org> <BANLkTi=fkguAtbfi9Wk9CPm8JmFa2Hti3Q@mail.gmail.com> <54668CDE-443D-4AF1-B17A-0497D0CDCC8D@yahoo-inc.com> <BANLkTimTGcEU29e_uLB52uc9CeuSkK__6w@mail.gmail.com> <EFB94DDE-9E2C-42D9-99C2-2FEB742535D1@yahoo-inc.com> <BANLkTinrqU8UUgNkNt6Wke8z=e_h10JaQA@mail.gmail.com> <26BCE485-DD1A-45C5-A328-C03CEC5BC297@yahoo-inc.com> <BANLkTi=pR3WXyNGyRPamne9o5A7XZLzo=A@mail.gmail.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1084)

If the board does have a stance, I'd love to hear it.  That could =
usefully end this discussion.

Absent that, it seems reasonable for the PMC to make a decision in this =
area.  Each project has different use cases and ecosystems, so it may =
not be reasonable to expect a one size fits all solution.  I see no =
reason not to make a local proposal, the board can always clarify.

On Jun 16, 2011, at 11:11 AM, Eli Collins wrote:

> On Thu, Jun 16, 2011 at 10:38 AM, Matthew Foley <mattf@yahoo-inc.com> =
wrote:
>> After writing my note to Eric, I realize that Eli and I are guilty of =
the same attempt
>> to use legal terminology in an engineering context.  Craig Russell is =
absolutely right.
>> If you change one bit, it is a "derived work".
>>=20
>> However, we can still allow the trademark to be applied to that work, =
if it
>> meets licensing criteria.  So what we are arguing about is, "Where is =
the boundary
>> line between something we are willing to call 'Apache Hadoop' and =
something
>> that must be called 'Product XYZ Powered by Apache Hadoop'?"
>>=20
>> I'm in favor of a very strict definition.  It needs to be really, =
really close to a
>> PMC-approved release.  But I'm open to the argument that a small =
number
>> of security patches could be necessary for a viable commercial =
product,
>> and that shouldn't necessarily prevent it from using the trademark.
>>=20
>> But I suggest we stop focusing on the term "derived work".  Note that =
the
>> "Defining Apache Hadoop" draft document we are voting on doesn't use
>> that term.
>=20
> See the section titled "Derivative Works". The term "derivative work"
> is used throughout the document.  I think you're right that the key
> point here is not what is and is not a derivative work, but what can
> be called Hadoop.
>=20
> Seems like the board should have an ASF-wide stance on what can be
> called Apache X instead of doing this per-project.
>=20
> Thanks,
> Eli


From general-return-3841-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 04:26:09 2011
Return-Path: <general-return-3841-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7058A4702
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 17 Jun 2011 04:26:09 +0000 (UTC)
Received: (qmail 17509 invoked by uid 500); 17 Jun 2011 04:26:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17169 invoked by uid 500); 17 Jun 2011 04:26:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 13671 invoked by uid 99); 17 Jun 2011 04:17:51 -0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of d.giriprasad@gmail.com designates 209.85.212.48 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=Y9kbLUW+wsriZwnkkY4uNS4ivA4OHaRakceuIdpVMNE=;
        b=HJZEEguwyNnGl4VIop7NKe67Z4fXg9PzUI9QmsuDpWGZWF3K4LaCMmbkaeH1+4mmON
         OxKhQ0NfU51sRH9bgfyJTUPnXB9JFKTviB/8YQFiSvoSGJE1yig+SGkAmx8r91lAhw9O
         UpSsJgTqJUAvLrPcsKoP64YxRq5DuHmFQ3fZY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=hTlk4Ol21cQCNKIxsA1HzZI0cPaFDpxTHxPYfEBWUpAmk/4GilsSijeJKaA5j1DqJP
         fl6ir/hO07te0IuDx2s3IPzI3qzQ5hJuO2M/nb7B/RiAD3yMbyEYZsrcug9s5131Ou2v
         xDfuh4aPO2/pCp9QaW9J/AX3eWCNzHJCCSsEo=
MIME-Version: 1.0
Date: Fri, 17 Jun 2011 09:47:22 +0530
Message-ID: <BANLkTik5dT+L13nB6v4abX_Ga8SpnE=47A@mail.gmail.com>
Subject: [VOTE] Powered by logo
From: Giri Prasad Reddy <d.giriprasad@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

I liked 4 a lot. My preference is: 4, 2, 6, 1, 3, 5

Regards,
giri

From general-return-3842-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 06:12:27 2011
Return-Path: <general-return-3842-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2EEE24BA1
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 17 Jun 2011 06:12:27 +0000 (UTC)
Received: (qmail 93992 invoked by uid 500); 17 Jun 2011 06:12:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93940 invoked by uid 500); 17 Jun 2011 06:12:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93932 invoked by uid 99); 17 Jun 2011 06:12:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 06:12:22 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of guosxu@163.com designates 220.181.13.49 as permitted sender)
Received: from [220.181.13.49] (HELO m13-49.163.com) (220.181.13.49)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 06:12:14 +0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=163.com;
	s=s110527; h=Received:Date:From:To:Message-ID:In-Reply-To:
	References:Subject:MIME-Version:Content-Type:
	Content-Transfer-Encoding; bh=DYquZ5tfTy3dhNOnkdgsWPgBXlDo3FPXa7
	fbcZhQuEY=; b=nENwGxePBqxS6TqetVS+U/42p+lF9Lp3QgRTXKcHctQ6mK717e
	9phmO9nPBdLOh4ZZTMTxLviNyLAjLjGznRdQOMeW138ThW2ERKdKe1AeYLTLbfVy
	lhMdaGvevm9s7LoqlA6TMa+Ey3ThhmTL0kORX/Ptkzpomi/IjNkYSs86I=
Received: from guosxu ( [218.242.218.34] ) by ajax-webmail-wmsvr49
 (Coremail) ; Fri, 17 Jun 2011 14:11:46 +0800 (CST)
Date: Fri, 17 Jun 2011 14:11:46 +0800 (CST)
From: =?GBK?B?ufnLs9Dx?= <guosxu@163.com>
To: general@hadoop.apache.org
Message-ID: <6fe0d887.7743.1309c3a0722.Coremail.guosxu@163.com>
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
Subject: Re:[VOTE] Powered by Logo
MIME-Version: 1.0
Content-Type: text/plain; charset=GBK
Content-Transfer-Encoding: quoted-printable
X-Originating-IP: [218.242.218.34]
X-Priority: 3
X-Mailer: Coremail Webmail Server Version SP_ntes V3.5 build
 110420(13372.3725.3724) Copyright (c) 2002-2011 www.mailtech.cn 163com
X-CM-TRANSID:McGowKBLvdcj8PpNAQwSAA--.3141W
X-CM-SenderInfo: pjxr25rx6rljoofrz/1tbiFANhd00vIYIoQgABsr
X-Coremail-Antispam: 1U5529EdanIXcx71UUUUU7vcSsGvfC2KfnxnUU==
X-Virus-Checked: Checked by ClamAV on apache.org

My vote (in order of course): 2, 6, 4, 3, 5, 1.
At 2011-06-15 11:19:18=A3=AC"Owen OMalley" <omalley@apache.org> wrote:
>All,
>   We've had a wide range of entries for a powered by logo. I've put them =
all on a page, here:
>
>http://people.apache.org/~omalley/hadoop-powered-by/
>
>Since there are a lot of contenders and we only want a single round of vot=
ing, let's use single transferable vote ( STV http://en.wikipedia.org/wiki/=
Single_transferable_vote). The important thing is to pick the images *IN OR=
DER* that you would like them.
>
>My vote (in order of course): 4, 1, 2, 3, 5, 6.
>
>In other words, I want option 4 most and option 6 least. With STV, you don=
't need to worry about voting for an unpopular choice since your vote will =
automatically roll over to your next choice.
>
>-- Owen
>
>


From general-return-3843-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 07:18:32 2011
Return-Path: <general-return-3843-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 69A4A4AE9
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 17 Jun 2011 07:18:32 +0000 (UTC)
Received: (qmail 51175 invoked by uid 500); 17 Jun 2011 07:18:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51096 invoked by uid 500); 17 Jun 2011 07:18:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51085 invoked by uid 99); 17 Jun 2011 07:18:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 07:18:28 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 216.145.54.171 is neither permitted nor denied by domain of eric14@yahoo-inc.com)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 07:18:23 +0000
Received: from [10.0.1.3] (snvvpn2-10-73-152-c94.hq.corp.yahoo.com [10.73.152.94])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5H7Huvo081359
	for <general@hadoop.apache.org>; Fri, 17 Jun 2011 00:17:57 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1308295077;
	bh=5CldhLlutTelewrHTm+aeiwWCUl6MdMbGdNJjmceChI=;
	h=From:Content-Type:Content-Transfer-Encoding:Subject:Date:
	 Message-Id:To:Mime-Version;
	b=YGZ8mzBNcgTp8lf8N022HUMg5LvL+Oh2Nf4oD/HGAxClAZqHBQ0LwhbM7fzYMeTQV
	 JkSMuKkfqtZRtS4mMW3ibNKmIGFxXLXbiiiRZsleCOeXsteVEM/Zl9Inyy2l36Hy/L
	 TxFWrqEAprZYklfHcSS0gplEavC9+Jwwf3G7Rnpw=
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: Thinking about the next hadoop mainline release
Date: Fri, 17 Jun 2011 00:17:56 -0700
Message-Id: <EFF679DA-26AB-4250-8F59-A2330D0E3E5C@yahoo-inc.com>
To: general@hadoop.apache.org
Mime-Version: 1.0 (Apple Message framework v1084)
X-Mailer: Apple Mail (2.1084)

Hi Folks,

I'd like to start a conversation on mainline planning and the next =
release of Apache Hadoop beyond 0.22.

The Yahoo! Hadoop team has been working hard to complete several big =
Hadoop projects, including:

- HDFS Federation [HDFS-1052]
 - Already merged into trunk

- Next Generation Map-Reduce [MR-279]
 - Passing most tests now and discussing merging into trunk

- The merging of our previous work on Hadoop with security into mainline =
[http://yhoo.it/i9Ww8W]
 - This is mostly done, but owen and others are doing a scrub to close =
out the remaining issues

All of these projects are now reaching a place where we would like to =
combine them with the good work already in 0.22 and put out a new apache =
release, perhaps 0.23.  We think the best way to accomplish that is to =
finish the merge in the next few weeks and then cut a release from =
trunk.

Yahoo stands ready to help us (the Apache Hadoop Community) turn this =
new release into a stable release by running it through its 9 month test =
and burn in process.  The result of that will be another stable release =
such as 0.18, 0.20 or 0.20.203 (hadoop with security).  We have Yahoo!s =
support for this substantial investment because this new release will =
have a great combination of new features for small and very large sites =
alike:
 - New Write Pipeline - HBase support [also in 0.21 & 0.22]
 - Federation - Scale up to larger clusters and the ability to =
experiment with new namenode approaches
 - Next Gen MapReduce - Scaleup, performance improvements, ability to =
experiment with new processing frameworks

I think this effort will produce a great new Apache Hadoop release for =
the community.  I'm starting this thread to collect feedback and =
hopefully folks' endorsement for merging in MR-279 and putting together =
this new release.  Feedback please?

Thanks,

E14


From general-return-3844-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 07:37:03 2011
Return-Path: <general-return-3844-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DB1154CA7
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 17 Jun 2011 07:37:03 +0000 (UTC)
Received: (qmail 84877 invoked by uid 500); 17 Jun 2011 07:37:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84530 invoked by uid 500); 17 Jun 2011 07:36:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83845 invoked by uid 99); 17 Jun 2011 07:36:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 07:36:48 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ryanobjc@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 07:36:41 +0000
Received: by fxm7 with SMTP id 7so386221fxm.35
        for <general@hadoop.apache.org>; Fri, 17 Jun 2011 00:36:21 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type:content-transfer-encoding;
        bh=g1e0TJaxszoOUTkPwfpsl/uOp3RuOXqhKjmaqeiz2ks=;
        b=wV90O/zdiBqHsfYL6aM9+nja58nCAen3xU7a0EHxWATXXtzz+o7IjSjJHxgqtfVxwk
         PSQyZtNCpLYh5LWmGG+ag+edvER4CiAdU9z7DJ8B3P78fKM8EUmUmWKR5Hips72KShlQ
         2x8Iq2SQRw+HU86O2Ho4+c2VzAt4BeDDrJMJM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=p8XXCKAxIeDnuPK6eQLqu+IiysfTujNbFj6JU6VL/JtuIrDjP59MMzj7Wbyep9IEwF
         Wxi6oVPR2YqAWMPHScHmxR8nEd4dNExVI1YBSgb+z8U00yzTRmKsxeWxA4lXfayyNM7S
         e+saihia74n9DWIx3TOBhMeiN0pUjb036bfXM=
MIME-Version: 1.0
Received: by 10.223.81.80 with SMTP id w16mr2118420fak.65.1308296181736; Fri,
 17 Jun 2011 00:36:21 -0700 (PDT)
Received: by 10.223.86.77 with HTTP; Fri, 17 Jun 2011 00:36:21 -0700 (PDT)
In-Reply-To: <EFF679DA-26AB-4250-8F59-A2330D0E3E5C@yahoo-inc.com>
References: <EFF679DA-26AB-4250-8F59-A2330D0E3E5C@yahoo-inc.com>
Date: Fri, 17 Jun 2011 00:36:21 -0700
Message-ID: <BANLkTimX8gSQRFbVZaVDQBOSN9c4iRi=Nw@mail.gmail.com>
Subject: Re: Thinking about the next hadoop mainline release
From: Ryan Rawson <ryanobjc@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

HDFS-918 and HDFS-347 are absolutely critical for random read
performance.  The smarter sites are already running HDFS-347 (I guess
they aren't running "Hadoop" then?), and soon they will be testing and
running HDFS-918 as well.  Opening 1 socket for every read just isn't
really scalable.

-ryan

On Fri, Jun 17, 2011 at 12:17 AM, Eric Baldeschwieler
<eric14@yahoo-inc.com> wrote:
> Hi Folks,
>
> I'd like to start a conversation on mainline planning and the next releas=
e of Apache Hadoop beyond 0.22.
>
> The Yahoo! Hadoop team has been working hard to complete several big Hado=
op projects, including:
>
> - HDFS Federation [HDFS-1052]
> =A0- Already merged into trunk
>
> - Next Generation Map-Reduce [MR-279]
> =A0- Passing most tests now and discussing merging into trunk
>
> - The merging of our previous work on Hadoop with security into mainline =
[http://yhoo.it/i9Ww8W]
> =A0- This is mostly done, but owen and others are doing a scrub to close =
out the remaining issues
>
> All of these projects are now reaching a place where we would like to com=
bine them with the good work already in 0.22 and put out a new apache relea=
se, perhaps 0.23. =A0We think the best way to accomplish that is to finish =
the merge in the next few weeks and then cut a release from trunk.
>
> Yahoo stands ready to help us (the Apache Hadoop Community) turn this new=
 release into a stable release by running it through its 9 month test and b=
urn in process. =A0The result of that will be another stable release such a=
s 0.18, 0.20 or 0.20.203 (hadoop with security). =A0We have Yahoo!s support=
 for this substantial investment because this new release will have a great=
 combination of new features for small and very large sites alike:
> =A0- New Write Pipeline - HBase support [also in 0.21 & 0.22]
> =A0- Federation - Scale up to larger clusters and the ability to experime=
nt with new namenode approaches
> =A0- Next Gen MapReduce - Scaleup, performance improvements, ability to e=
xperiment with new processing frameworks
>
> I think this effort will produce a great new Apache Hadoop release for th=
e community. =A0I'm starting this thread to collect feedback and hopefully =
folks' endorsement for merging in MR-279 and putting together this new rele=
ase. =A0Feedback please?
>
> Thanks,
>
> E14
>
>

From general-return-3845-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 08:14:11 2011
Return-Path: <general-return-3845-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 70C2641E0
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 17 Jun 2011 08:14:11 +0000 (UTC)
Received: (qmail 36626 invoked by uid 500); 17 Jun 2011 08:14:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36571 invoked by uid 500); 17 Jun 2011 08:14:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36563 invoked by uid 99); 17 Jun 2011 08:14:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 08:14:07 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.220.176] (HELO mail-vx0-f176.google.com) (209.85.220.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 08:13:59 +0000
Received: by vxd2 with SMTP id 2so643889vxd.35
        for <general@hadoop.apache.org>; Fri, 17 Jun 2011 01:13:38 -0700 (PDT)
Received: by 10.52.100.74 with SMTP id ew10mr2638554vdb.283.1308298418069;
 Fri, 17 Jun 2011 01:13:38 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.52.111.166 with HTTP; Fri, 17 Jun 2011 01:13:18 -0700 (PDT)
X-Originating-IP: [89.96.138.154]
In-Reply-To: <BANLkTimX8gSQRFbVZaVDQBOSN9c4iRi=Nw@mail.gmail.com>
References: <EFF679DA-26AB-4250-8F59-A2330D0E3E5C@yahoo-inc.com> <BANLkTimX8gSQRFbVZaVDQBOSN9c4iRi=Nw@mail.gmail.com>
From: Ted Dunning <tdunning@maprtech.com>
Date: Fri, 17 Jun 2011 10:13:18 +0200
Message-ID: <BANLkTi=igCLuZ7v2+L9_nObs8wfPP4_F8g@mail.gmail.com>
Subject: Re: Thinking about the next hadoop mainline release
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf3071c69478220104a5e3f623

--20cf3071c69478220104a5e3f623
Content-Type: text/plain; charset=ISO-8859-1

NG map reduce is a huge deal both in terms of making things better for
users, but also in terms of unblocking the Hadoop development process.

On Fri, Jun 17, 2011 at 9:36 AM, Ryan Rawson <ryanobjc@gmail.com> wrote:

> > - Next Generation Map-Reduce [MR-279]
> >  - Passing most tests now and discussing merging into trunk
>

--20cf3071c69478220104a5e3f623--

From general-return-3846-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 11:02:51 2011
Return-Path: <general-return-3846-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EC0A94849
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 17 Jun 2011 11:02:50 +0000 (UTC)
Received: (qmail 60997 invoked by uid 500); 17 Jun 2011 11:02:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60927 invoked by uid 500); 17 Jun 2011 11:02:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60908 invoked by uid 99); 17 Jun 2011 11:02:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 11:02:46 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 11:02:37 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id AF1B31BA9A7
	for <general@hadoop.apache.org>; Fri, 17 Jun 2011 12:02:16 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id DJSbGZMYL2v8 for <general@hadoop.apache.org>;
	Fri, 17 Jun 2011 12:02:16 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by colossus.hpl.hp.com (Postfix) with ESMTP id D51391BA97A
	for <general@hadoop.apache.org>; Fri, 17 Jun 2011 12:02:15 +0100 (BST)
MailScanner-NULL-Check: 1308913318.01421@rATPq0mSPaq/VZhMIChwyQ
Received: from [16.25.175.86] (wildhaus.hpl.hp.com [16.25.175.86])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p5HB1vOV015292
	for <general@hadoop.apache.org>; Fri, 17 Jun 2011 12:01:57 +0100 (BST)
Message-ID: <4DFB3421.2010902@apache.org>
Date: Fri, 17 Jun 2011 12:01:53 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org> <BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com> <FBF55C01-D66A-44C0-9252-D54E7CAB103D@apache.org> <BANLkTi=BWPUEoqD3L98A9AXLkbafZs15Zw@mail.gmail.com> <92D32E6E-3B65-464F-B91B-4C30EAB2642C@apache.org> <BANLkTi=1JGe8kVortj1Act3jdt9L2RCrhw@mail.gmail.com> <4DF880DA.7060105@apache.org> <BANLkTim6DrVsb5NpwnvAvQtLqa5hq0eOrg@mail.gmail.com>
In-Reply-To: <BANLkTim6DrVsb5NpwnvAvQtLqa5hq0eOrg@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p5HB1vOV015292
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 15/06/11 16:58, Konstantin Boudnik wrote:
> On Wed, Jun 15, 2011 at 02:52, Steve Loughran<stevel@apache.org>  wrote:

>>
>> Regarding the vote, I think the discussion here is interesting and should be
>> finalised before the vote. It's worth resolving the issues.
>>
>> also: banners, stickers and clothing? Can I have T-shirts saying "I broke
>> the hadoop build" with the logo on, or should it be "I broke the Apache
>> Hadoop build"?
>
> I think such a T-shirt should be forcefully worn on any person who did
> just that.

Here you go with the poster:
http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/hadoop-components/hadoop-ops/doc/breaking_the_hadoop_build.odp?revision=8630

I can add it to hadoop-common SVN for people to work on...

From general-return-3847-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 13:20:55 2011
Return-Path: <general-return-3847-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 232C5421F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 17 Jun 2011 13:20:55 +0000 (UTC)
Received: (qmail 72046 invoked by uid 500); 17 Jun 2011 13:20:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71992 invoked by uid 500); 17 Jun 2011 13:20:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71984 invoked by uid 99); 17 Jun 2011 13:20:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 13:20:53 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.18.222.49] (HELO smtp3.4emm.com) (69.18.222.49)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 13:20:47 +0000
Received: from EX2K7VS03.4emm.local ([192.168.160.203]) by HUB03.4emm.local
 ([192.168.161.134]) with mapi; Fri, 17 Jun 2011 09:20:25 -0400
From: Doug Meil <doug.meil@explorysmedical.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 17 Jun 2011 09:21:38 -0400
Subject: RE: Thinking about the next hadoop mainline release
Thread-Topic: Thinking about the next hadoop mainline release
Thread-Index: AcwswQPbKF7N4UQqTHqHbmTupD/U6wAMHJQA
Message-ID: <67680900F79B1D4F99C844EE386FC5952823BEDB40@EX2K7VS03.4emm.local>
References: <EFF679DA-26AB-4250-8F59-A2330D0E3E5C@yahoo-inc.com>
 <BANLkTimX8gSQRFbVZaVDQBOSN9c4iRi=Nw@mail.gmail.com>
In-Reply-To: <BANLkTimX8gSQRFbVZaVDQBOSN9c4iRi=Nw@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

+1 on what Ryan said.

-----Original Message-----
From: Ryan Rawson [mailto:ryanobjc@gmail.com]=20
Sent: Friday, June 17, 2011 3:36 AM
To: general@hadoop.apache.org
Subject: Re: Thinking about the next hadoop mainline release

HDFS-918 and HDFS-347 are absolutely critical for random read performance. =
 The smarter sites are already running HDFS-347 (I guess they aren't runnin=
g "Hadoop" then?), and soon they will be testing and running HDFS-918 as we=
ll.  Opening 1 socket for every read just isn't really scalable.

-ryan

On Fri, Jun 17, 2011 at 12:17 AM, Eric Baldeschwieler <eric14@yahoo-inc.com=
> wrote:
> Hi Folks,
>
> I'd like to start a conversation on mainline planning and the next releas=
e of Apache Hadoop beyond 0.22.
>
> The Yahoo! Hadoop team has been working hard to complete several big Hado=
op projects, including:
>
> - HDFS Federation [HDFS-1052]
> =A0- Already merged into trunk
>
> - Next Generation Map-Reduce [MR-279]
> =A0- Passing most tests now and discussing merging into trunk
>
> - The merging of our previous work on Hadoop with security into=20
> mainline [http://yhoo.it/i9Ww8W]
> =A0- This is mostly done, but owen and others are doing a scrub to close=
=20
> out the remaining issues
>
> All of these projects are now reaching a place where we would like to com=
bine them with the good work already in 0.22 and put out a new apache relea=
se, perhaps 0.23. =A0We think the best way to accomplish that is to finish =
the merge in the next few weeks and then cut a release from trunk.
>
> Yahoo stands ready to help us (the Apache Hadoop Community) turn this new=
 release into a stable release by running it through its 9 month test and b=
urn in process. =A0The result of that will be another stable release such a=
s 0.18, 0.20 or 0.20.203 (hadoop with security). =A0We have Yahoo!s support=
 for this substantial investment because this new release will have a great=
 combination of new features for small and very large sites alike:
> =A0- New Write Pipeline - HBase support [also in 0.21 & 0.22]
> =A0- Federation - Scale up to larger clusters and the ability to=20
> experiment with new namenode approaches
> =A0- Next Gen MapReduce - Scaleup, performance improvements, ability to=20
> experiment with new processing frameworks
>
> I think this effort will produce a great new Apache Hadoop release for th=
e community. =A0I'm starting this thread to collect feedback and hopefully =
folks' endorsement for merging in MR-279 and putting together this new rele=
ase. =A0Feedback please?
>
> Thanks,
>
> E14
>
>

From general-return-3848-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 14:17:47 2011
Return-Path: <general-return-3848-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EFB224AA8
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 17 Jun 2011 14:17:46 +0000 (UTC)
Received: (qmail 67587 invoked by uid 500); 17 Jun 2011 14:17:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67537 invoked by uid 500); 17 Jun 2011 14:17:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67529 invoked by uid 99); 17 Jun 2011 14:17:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 14:17:45 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 216.145.54.172 is neither permitted nor denied by domain of arunc@yahoo-inc.com)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 14:17:35 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5HEFwsX057871;
	Fri, 17 Jun 2011 07:15:58 -0700 (PDT)
Received: from SP2-EX07VS04.ds.corp.yahoo.com ([98.137.59.33]) by
 SP2-EX07CAS01.ds.corp.yahoo.com ([98.137.59.37]) with mapi; Fri, 17 Jun 2011
 07:15:58 -0700
From: Arun C Murthy <arunc@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 17 Jun 2011 07:15:48 -0700
Subject: Re: Thinking about the next hadoop mainline release
Thread-Topic: Thinking about the next hadoop mainline release
Thread-Index: Acws+RPlMNysOuyuR6ehLH41N0fBKg==
Message-ID: <E9E17CC3-013C-46E5-AD48-BF76DEE8CE02@yahoo-inc.com>
References: <EFF679DA-26AB-4250-8F59-A2330D0E3E5C@yahoo-inc.com>
 <BANLkTimX8gSQRFbVZaVDQBOSN9c4iRi=Nw@mail.gmail.com>
 <BANLkTi=igCLuZ7v2+L9_nObs8wfPP4_F8g@mail.gmail.com>
In-Reply-To: <BANLkTi=igCLuZ7v2+L9_nObs8wfPP4_F8g@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

I volunteer to be the RM for the release since I've been leading the NG NR =
effort.

Are folks ok with this?

thanks,
Arun

Sent from my iPhone

On Jun 17, 2011, at 1:45 PM, "Ted Dunning" <tdunning@maprtech.com> wrote:

> NG map reduce is a huge deal both in terms of making things better for
> users, but also in terms of unblocking the Hadoop development process.
>=20
> On Fri, Jun 17, 2011 at 9:36 AM, Ryan Rawson <ryanobjc@gmail.com> wrote:
>=20
>>> - Next Generation Map-Reduce [MR-279]
>>> - Passing most tests now and discussing merging into trunk
>>=20

From general-return-3849-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 14:30:49 2011
Return-Path: <general-return-3849-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6811C4E9D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 17 Jun 2011 14:30:49 +0000 (UTC)
Received: (qmail 85580 invoked by uid 500); 17 Jun 2011 14:30:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85509 invoked by uid 500); 17 Jun 2011 14:30:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85501 invoked by uid 99); 17 Jun 2011 14:30:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 14:30:47 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [129.93.165.11] (HELO cse-mail.unl.edu) (129.93.165.11)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 14:30:41 +0000
Received: from cse-barracuda.cse.unl.edu (cse-barracuda.unl.edu [129.93.164.185])
	(authenticated bits=0)
	by cse-mail.unl.edu (8.14.3/8.14.3) with ESMTP id p5HEUENI018382
	for <general@hadoop.apache.org>; Fri, 17 Jun 2011 09:30:19 -0500 (CDT)
X-ASG-Debug-ID: 1308321014-034b5c117d287600001-IjwG86
Received: from cse.unl.edu (cse.unl.edu [129.93.165.2]) by cse-barracuda.cse.unl.edu with ESMTP id e7gzXJD69NvRGCkc for <general@hadoop.apache.org>; Fri, 17 Jun 2011 09:30:14 -0500 (CDT)
X-Barracuda-Envelope-From: bbockelm@cse.unl.edu
X-Barracuda-RBL-Trusted-Forwarder: 129.93.165.2
Received: from pcp088901pcs.unl.edu (pcp088901pcs.unl.edu [129.93.158.16])
	(authenticated bits=0)
	by cse.unl.edu (8.14.4/8.14.3) with ESMTP id p5HEUEVL020908
	(version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO)
	for <general@hadoop.apache.org>; Fri, 17 Jun 2011 09:30:14 -0500
Content-Type: text/plain; charset=us-ascii
X-Barracuda-Apparent-Source-IP: 129.93.158.16
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: Thinking about the next hadoop mainline release
From: Brian Bockelman <bbockelm@cse.unl.edu>
X-ASG-Orig-Subj: Re: Thinking about the next hadoop mainline release
In-Reply-To: <BANLkTimX8gSQRFbVZaVDQBOSN9c4iRi=Nw@mail.gmail.com>
Date: Fri, 17 Jun 2011 09:30:13 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <39C6561C-9CB6-4BA5-BE31-C269590B7842@cse.unl.edu>
References: <EFF679DA-26AB-4250-8F59-A2330D0E3E5C@yahoo-inc.com> <BANLkTimX8gSQRFbVZaVDQBOSN9c4iRi=Nw@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Barracuda-Connect: cse.unl.edu[129.93.165.2]
X-Barracuda-Start-Time: 1308321014
X-Barracuda-URL: http://cse-barracuda.unl.edu:8000/cgi-mod/mark.cgi
X-Virus-Scanned: clamav-milter 0.97.1 at cse-mail
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using global scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=9.0 tests=MAILTO_TO_SPAM_ADDR
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.2.66338
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 MAILTO_TO_SPAM_ADDR    URI: Includes a link to a likely spammer email
X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0.1 (cse-mail.unl.edu [129.93.165.11]); Fri, 17 Jun 2011 09:30:20 -0500 (CDT)
X-Virus-Status: Clean

Hi Ryan, Eric,

Just looked at those two for the first time in awhile.
- HDFS-918 (now 1323?) doesn't seem like it's too controversial, but =
does seem like there's a bit of validation left.
- HDFS-347 has a long, contentious history.  However, it seems that most =
of the strong objections have been cleared up.  Is there anyone left who =
objects to it, now that it doesn't appear to bypass security?

Finally, I see Todd has posted HDFS-2080 claiming some sizable =
performance improvements.  Would it be possible that could finish in =
time for release?

As a site which heavily uses random reads and high-throughput reads, I'm =
very excited for this release!

Brian

On Jun 17, 2011, at 2:36 AM, Ryan Rawson wrote:

> HDFS-918 and HDFS-347 are absolutely critical for random read
> performance.  The smarter sites are already running HDFS-347 (I guess
> they aren't running "Hadoop" then?), and soon they will be testing and
> running HDFS-918 as well.  Opening 1 socket for every read just isn't
> really scalable.
>=20
> -ryan
>=20
> On Fri, Jun 17, 2011 at 12:17 AM, Eric Baldeschwieler
> <eric14@yahoo-inc.com> wrote:
>> Hi Folks,
>>=20
>> I'd like to start a conversation on mainline planning and the next =
release of Apache Hadoop beyond 0.22.
>>=20
>> The Yahoo! Hadoop team has been working hard to complete several big =
Hadoop projects, including:
>>=20
>> - HDFS Federation [HDFS-1052]
>>  - Already merged into trunk
>>=20
>> - Next Generation Map-Reduce [MR-279]
>>  - Passing most tests now and discussing merging into trunk
>>=20
>> - The merging of our previous work on Hadoop with security into =
mainline [http://yhoo.it/i9Ww8W]
>>  - This is mostly done, but owen and others are doing a scrub to =
close out the remaining issues
>>=20
>> All of these projects are now reaching a place where we would like to =
combine them with the good work already in 0.22 and put out a new apache =
release, perhaps 0.23.  We think the best way to accomplish that is to =
finish the merge in the next few weeks and then cut a release from =
trunk.
>>=20
>> Yahoo stands ready to help us (the Apache Hadoop Community) turn this =
new release into a stable release by running it through its 9 month test =
and burn in process.  The result of that will be another stable release =
such as 0.18, 0.20 or 0.20.203 (hadoop with security).  We have Yahoo!s =
support for this substantial investment because this new release will =
have a great combination of new features for small and very large sites =
alike:
>>  - New Write Pipeline - HBase support [also in 0.21 & 0.22]
>>  - Federation - Scale up to larger clusters and the ability to =
experiment with new namenode approaches
>>  - Next Gen MapReduce - Scaleup, performance improvements, ability to =
experiment with new processing frameworks
>>=20
>> I think this effort will produce a great new Apache Hadoop release =
for the community.  I'm starting this thread to collect feedback and =
hopefully folks' endorsement for merging in MR-279 and putting together =
this new release.  Feedback please?
>>=20
>> Thanks,
>>=20
>> E14
>>=20
>>=20


From general-return-3850-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 14:38:15 2011
Return-Path: <general-return-3850-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5A67F42DF
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 17 Jun 2011 14:38:15 +0000 (UTC)
Received: (qmail 97243 invoked by uid 500); 17 Jun 2011 14:38:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97185 invoked by uid 500); 17 Jun 2011 14:38:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97177 invoked by uid 99); 17 Jun 2011 14:38:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 14:38:13 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jaybooth@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 14:38:07 +0000
Received: by qwj9 with SMTP id 9so1694448qwj.35
        for <general@hadoop.apache.org>; Fri, 17 Jun 2011 07:37:47 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type:content-transfer-encoding;
        bh=4nGgpYuR1VkQ9FqG/3/kjIwjB4AsG3gYDtses1ewg3A=;
        b=LMOgbxqEo/ZJPQpD59cEUS5pLWDCUaqMQFAIvy5YrjAoTOzHkvsbdFdm6RSi+wDh+z
         Q4FKX24k6FZEi69f+nlp5xlmMoXVK+2GlgYnH3rwz6TFz0+Q+sPoPYQi49fWd9CWSCow
         FtqIKWU0fmToIRjkvBKvmwODRD6wbUZpH6DEs=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=ug06oIufYewQ0mP8ekI1Pc189A4gERvzvPsDBdbWwFggYuA1Wann/PwVoTYoIOUY6r
         uSPdXA9/uVlw2t9CYlA5bCtyg+rAKIVNo+wgoW6tlpZZpj8Wa80dK5Q3k4mXUO3cFFpx
         F4jqWa9BHm/woy8AOwq9toJSo2HFA7lIoF93o=
MIME-Version: 1.0
Received: by 10.224.32.94 with SMTP id b30mr619278qad.302.1308321466932; Fri,
 17 Jun 2011 07:37:46 -0700 (PDT)
Received: by 10.224.60.73 with HTTP; Fri, 17 Jun 2011 07:37:46 -0700 (PDT)
In-Reply-To: <39C6561C-9CB6-4BA5-BE31-C269590B7842@cse.unl.edu>
References: <EFF679DA-26AB-4250-8F59-A2330D0E3E5C@yahoo-inc.com>
	<BANLkTimX8gSQRFbVZaVDQBOSN9c4iRi=Nw@mail.gmail.com>
	<39C6561C-9CB6-4BA5-BE31-C269590B7842@cse.unl.edu>
Date: Fri, 17 Jun 2011 10:37:46 -0400
Message-ID: <BANLkTintrCcUBreDcGBws14X1Cy9prGeJA@mail.gmail.com>
Subject: Re: Thinking about the next hadoop mainline release
From: Jay Booth <jaybooth@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

I can look at 1323 (hdfs-918's successor) next week/weekend and clear
the test problems, thanks Todd for updating the patch to current
trunk.  1323 is only filechannel-pooling, which is much less
disruptive than refactoring everything in the DN to be event-driven.

On Fri, Jun 17, 2011 at 10:30 AM, Brian Bockelman <bbockelm@cse.unl.edu> wr=
ote:
> Hi Ryan, Eric,
>
> Just looked at those two for the first time in awhile.
> - HDFS-918 (now 1323?) doesn't seem like it's too controversial, but does=
 seem like there's a bit of validation left.
> - HDFS-347 has a long, contentious history. =A0However, it seems that mos=
t of the strong objections have been cleared up. =A0Is there anyone left wh=
o objects to it, now that it doesn't appear to bypass security?
>
> Finally, I see Todd has posted HDFS-2080 claiming some sizable performanc=
e improvements. =A0Would it be possible that could finish in time for relea=
se?
>
> As a site which heavily uses random reads and high-throughput reads, I'm =
very excited for this release!
>
> Brian
>
> On Jun 17, 2011, at 2:36 AM, Ryan Rawson wrote:
>
>> HDFS-918 and HDFS-347 are absolutely critical for random read
>> performance. =A0The smarter sites are already running HDFS-347 (I guess
>> they aren't running "Hadoop" then?), and soon they will be testing and
>> running HDFS-918 as well. =A0Opening 1 socket for every read just isn't
>> really scalable.
>>
>> -ryan
>>
>> On Fri, Jun 17, 2011 at 12:17 AM, Eric Baldeschwieler
>> <eric14@yahoo-inc.com> wrote:
>>> Hi Folks,
>>>
>>> I'd like to start a conversation on mainline planning and the next rele=
ase of Apache Hadoop beyond 0.22.
>>>
>>> The Yahoo! Hadoop team has been working hard to complete several big Ha=
doop projects, including:
>>>
>>> - HDFS Federation [HDFS-1052]
>>> =A0- Already merged into trunk
>>>
>>> - Next Generation Map-Reduce [MR-279]
>>> =A0- Passing most tests now and discussing merging into trunk
>>>
>>> - The merging of our previous work on Hadoop with security into mainlin=
e [http://yhoo.it/i9Ww8W]
>>> =A0- This is mostly done, but owen and others are doing a scrub to clos=
e out the remaining issues
>>>
>>> All of these projects are now reaching a place where we would like to c=
ombine them with the good work already in 0.22 and put out a new apache rel=
ease, perhaps 0.23. =A0We think the best way to accomplish that is to finis=
h the merge in the next few weeks and then cut a release from trunk.
>>>
>>> Yahoo stands ready to help us (the Apache Hadoop Community) turn this n=
ew release into a stable release by running it through its 9 month test and=
 burn in process. =A0The result of that will be another stable release such=
 as 0.18, 0.20 or 0.20.203 (hadoop with security). =A0We have Yahoo!s suppo=
rt for this substantial investment because this new release will have a gre=
at combination of new features for small and very large sites alike:
>>> =A0- New Write Pipeline - HBase support [also in 0.21 & 0.22]
>>> =A0- Federation - Scale up to larger clusters and the ability to experi=
ment with new namenode approaches
>>> =A0- Next Gen MapReduce - Scaleup, performance improvements, ability to=
 experiment with new processing frameworks
>>>
>>> I think this effort will produce a great new Apache Hadoop release for =
the community. =A0I'm starting this thread to collect feedback and hopefull=
y folks' endorsement for merging in MR-279 and putting together this new re=
lease. =A0Feedback please?
>>>
>>> Thanks,
>>>
>>> E14
>>>
>>>
>
>

From general-return-3851-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 14:43:39 2011
Return-Path: <general-return-3851-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 18C7E4608
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 17 Jun 2011 14:43:39 +0000 (UTC)
Received: (qmail 13711 invoked by uid 500); 17 Jun 2011 14:43:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13619 invoked by uid 500); 17 Jun 2011 14:43:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13608 invoked by uid 99); 17 Jun 2011 14:43:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 14:43:36 +0000
X-ASF-Spam-Status: No, hits=4.6 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL,URIBL_BLACK
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 69.147.107.21 is neither permitted nor denied by domain of arunc@yahoo-inc.com)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 14:43:29 +0000
Received: from SP2-EX07CAS02.ds.corp.yahoo.com (sp2-ex07cas02.corp.sp2.yahoo.com [98.137.59.38])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5HEgm1T015588;
	Fri, 17 Jun 2011 07:42:48 -0700 (PDT)
Received: from SP2-EX07VS04.ds.corp.yahoo.com ([98.137.59.33]) by
 SP2-EX07CAS02.ds.corp.yahoo.com ([98.137.59.38]) with mapi; Fri, 17 Jun 2011
 07:42:48 -0700
From: Arun C Murthy <arunc@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 17 Jun 2011 07:42:43 -0700
Subject: Re: Thinking about the next hadoop mainline release
Thread-Topic: Thinking about the next hadoop mainline release
Thread-Index: Acws/NOUhs6xCR1yS/u5cgfszrw8bg==
Message-ID: <BBB6229A-6025-47ED-9E01-B43834ED47DA@yahoo-inc.com>
References: <EFF679DA-26AB-4250-8F59-A2330D0E3E5C@yahoo-inc.com>
 <BANLkTimX8gSQRFbVZaVDQBOSN9c4iRi=Nw@mail.gmail.com>
 <39C6561C-9CB6-4BA5-BE31-C269590B7842@cse.unl.edu>
In-Reply-To: <39C6561C-9CB6-4BA5-BE31-C269590B7842@cse.unl.edu>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Ryan & Brian,


All that needs to be included in a release branch after the branch is cut i=
s that someone needs to convince the RM to include it. It should be fairly =
straight-forward.

OTOH, if it's in trunk when the branch is made this discussion is moot.

So, please provide necessary feedback to the RM when the branch is made and=
 let's focus on a high-level goal for a next release off trunk in this thre=
ad. Makes sense?

thanks,
Arun=20

Sent from my iPhone

On Jun 17, 2011, at 8:01 PM, "Brian Bockelman" <bbockelm@cse.unl.edu> wrote=
:

> Hi Ryan, Eric,
>=20
> Just looked at those two for the first time in awhile.
> - HDFS-918 (now 1323?) doesn't seem like it's too controversial, but does=
 seem like there's a bit of validation left.
> - HDFS-347 has a long, contentious history.  However, it seems that most =
of the strong objections have been cleared up.  Is there anyone left who ob=
jects to it, now that it doesn't appear to bypass security?
>=20
> Finally, I see Todd has posted HDFS-2080 claiming some sizable performanc=
e improvements.  Would it be possible that could finish in time for release=
?
>=20
> As a site which heavily uses random reads and high-throughput reads, I'm =
very excited for this release!
>=20
> Brian
>=20
> On Jun 17, 2011, at 2:36 AM, Ryan Rawson wrote:
>=20
>> HDFS-918 and HDFS-347 are absolutely critical for random read
>> performance.  The smarter sites are already running HDFS-347 (I guess
>> they aren't running "Hadoop" then?), and soon they will be testing and
>> running HDFS-918 as well.  Opening 1 socket for every read just isn't
>> really scalable.
>>=20
>> -ryan
>>=20
>> On Fri, Jun 17, 2011 at 12:17 AM, Eric Baldeschwieler
>> <eric14@yahoo-inc.com> wrote:
>>> Hi Folks,
>>>=20
>>> I'd like to start a conversation on mainline planning and the next rele=
ase of Apache Hadoop beyond 0.22.
>>>=20
>>> The Yahoo! Hadoop team has been working hard to complete several big Ha=
doop projects, including:
>>>=20
>>> - HDFS Federation [HDFS-1052]
>>> - Already merged into trunk
>>>=20
>>> - Next Generation Map-Reduce [MR-279]
>>> - Passing most tests now and discussing merging into trunk
>>>=20
>>> - The merging of our previous work on Hadoop with security into mainlin=
e [http://yhoo.it/i9Ww8W]
>>> - This is mostly done, but owen and others are doing a scrub to close o=
ut the remaining issues
>>>=20
>>> All of these projects are now reaching a place where we would like to c=
ombine them with the good work already in 0.22 and put out a new apache rel=
ease, perhaps 0.23.  We think the best way to accomplish that is to finish =
the merge in the next few weeks and then cut a release from trunk.
>>>=20
>>> Yahoo stands ready to help us (the Apache Hadoop Community) turn this n=
ew release into a stable release by running it through its 9 month test and=
 burn in process.  The result of that will be another stable release such a=
s 0.18, 0.20 or 0.20.203 (hadoop with security).  We have Yahoo!s support f=
or this substantial investment because this new release will have a great c=
ombination of new features for small and very large sites alike:
>>> - New Write Pipeline - HBase support [also in 0.21 & 0.22]
>>> - Federation - Scale up to larger clusters and the ability to experimen=
t with new namenode approaches
>>> - Next Gen MapReduce - Scaleup, performance improvements, ability to ex=
periment with new processing frameworks
>>>=20
>>> I think this effort will produce a great new Apache Hadoop release for =
the community.  I'm starting this thread to collect feedback and hopefully =
folks' endorsement for merging in MR-279 and putting together this new rele=
ase.  Feedback please?
>>>=20
>>> Thanks,
>>>=20
>>> E14
>>>=20
>>>=20
>=20

From general-return-3852-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 17:31:51 2011
Return-Path: <general-return-3852-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2880E4D1B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 17 Jun 2011 17:31:51 +0000 (UTC)
Received: (qmail 61476 invoked by uid 500); 17 Jun 2011 17:31:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61410 invoked by uid 500); 17 Jun 2011 17:31:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61402 invoked by uid 99); 17 Jun 2011 17:31:49 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 17:31:49 +0000
Received: from localhost (HELO dhcp-02.private.iobm.com) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 17:31:49 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Thinking about the next hadoop mainline release
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <EFF679DA-26AB-4250-8F59-A2330D0E3E5C@yahoo-inc.com>
Date: Fri, 17 Jun 2011 10:31:48 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <F77C0A6C-92AC-4486-8326-589D1D297F9A@apache.org>
References: <EFF679DA-26AB-4250-8F59-A2330D0E3E5C@yahoo-inc.com>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On Jun 17, 2011, at 12:17 AM, Eric Baldeschwieler wrote:
> Yahoo stands ready to help us (the Apache Hadoop Community) turn this =
new release into a stable release by running it through its 9 month test =
and burn in process.  The result of that will be another stable release =
such as 0.18, 0.20 or 0.20.203 (hadoop with security). =20


	I'd consider 0.20.203 pseudo-stable.  It has some significant =
regressions on non-Linux platforms due to libhadoop.so being a dumping =
ground for all compiled code.  On OS X, hadoop actually lies about =
things now. Nine months of testing is useless for those folks if the =
outcome is another 0.20.203.


> I think this effort will produce a great new Apache Hadoop release for =
the community.  I'm starting this thread to collect feedback and =
hopefully folks' endorsement for merging in MR-279 and putting together =
this new release.  Feedback please?


	For .23, we desperately need to have libhadoop tell the Java =
code what it supports rather than just assuming that all the =
functionality is present if the library loads.  I pretty much consider =
this a blocking issue.=

From general-return-3853-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 17:33:20 2011
Return-Path: <general-return-3853-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 86E7A44B9
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 17 Jun 2011 17:33:20 +0000 (UTC)
Received: (qmail 63483 invoked by uid 500); 17 Jun 2011 17:33:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63430 invoked by uid 500); 17 Jun 2011 17:33:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63418 invoked by uid 99); 17 Jun 2011 17:33:18 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 17:33:18 +0000
Received: from localhost (HELO dhcp-02.private.iobm.com) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 17:33:18 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Thinking about the next hadoop mainline release
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <BANLkTimX8gSQRFbVZaVDQBOSN9c4iRi=Nw@mail.gmail.com>
Date: Fri, 17 Jun 2011 10:33:17 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <43D4BB64-1B14-45F6-941B-B65F31636266@apache.org>
References: <EFF679DA-26AB-4250-8F59-A2330D0E3E5C@yahoo-inc.com> <BANLkTimX8gSQRFbVZaVDQBOSN9c4iRi=Nw@mail.gmail.com>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On Jun 17, 2011, at 12:36 AM, Ryan Rawson wrote:

> HDFS-918 and HDFS-347 are absolutely critical for random read
> performance.  The smarter sites are already running HDFS-347 (I guess
> they aren't running "Hadoop" then?), and soon they will be testing and
> running HDFS-918 as well.  Opening 1 socket for every read just isn't
> really scalable.

	Isn't "random read [on HDFS]" and "smarter sites" in the same =
breath an oxymoron?



From general-return-3854-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 17:34:47 2011
Return-Path: <general-return-3854-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B4CC84C17
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 17 Jun 2011 17:34:47 +0000 (UTC)
Received: (qmail 66290 invoked by uid 500); 17 Jun 2011 17:34:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66228 invoked by uid 500); 17 Jun 2011 17:34:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66220 invoked by uid 99); 17 Jun 2011 17:34:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 17:34:46 +0000
X-ASF-Spam-Status: No, hits=2.8 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS,URIBL_BLACK
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 17:34:40 +0000
Received: by bwz8 with SMTP id 8so1291839bwz.35
        for <general@hadoop.apache.org>; Fri, 17 Jun 2011 10:34:19 -0700 (PDT)
Received: by 10.204.233.15 with SMTP id jw15mr1871172bkb.48.1308332059330;
 Fri, 17 Jun 2011 10:34:19 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.79 with HTTP; Fri, 17 Jun 2011 10:33:59 -0700 (PDT)
In-Reply-To: <39C6561C-9CB6-4BA5-BE31-C269590B7842@cse.unl.edu>
References: <EFF679DA-26AB-4250-8F59-A2330D0E3E5C@yahoo-inc.com>
 <BANLkTimX8gSQRFbVZaVDQBOSN9c4iRi=Nw@mail.gmail.com> <39C6561C-9CB6-4BA5-BE31-C269590B7842@cse.unl.edu>
From: Todd Lipcon <todd@cloudera.com>
Date: Fri, 17 Jun 2011 10:33:59 -0700
Message-ID: <BANLkTi=PxobBe9QBjc=rMF+mbYc5yCokwA@mail.gmail.com>
Subject: Re: Thinking about the next hadoop mainline release
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Fri, Jun 17, 2011 at 7:30 AM, Brian Bockelman <bbockelm@cse.unl.edu> wro=
te:
>
> Hi Ryan, Eric,
>
> Just looked at those two for the first time in awhile.
> - HDFS-918 (now 1323?) doesn't seem like it's too controversial, but does=
 seem like there's a bit of validation left.

Yes, 1323 and also 1148 would be "nice to haves", but neither is ready
to go, yet. Though I really want to improve HBase performance, I also
tend to be fairly conservative on how much testing these things should
need before getting checked in (unless they can be completely
pluggable).

The good news is we did get 941 in last week, and that's a real nice
improvement.

> - HDFS-347 has a long, contentious history. =A0However, it seems that mos=
t of the strong objections have been cleared up. =A0Is there anyone left wh=
o objects to it, now that it doesn't appear to bypass security?

It still has a way to go to be pushed over the finish line. I don't
foresee it happening for this release.

> Finally, I see Todd has posted HDFS-2080 claiming some sizable performanc=
e improvements. =A0Would it be possible that could finish in time for relea=
se?

HDFS-2080 has very good bang-for-the-buck in the gains-per-complexity
ratio, especially compared to 347. It could also be made completely
pluggable, since it's just a new implementation of BlockReader. So it
might be feasible to include but not enabled by default.

But, I wouldn't block the 0.23 (or any other) release on including
these things. If they're done and look low-risk at an early enough
date, I'll do my best to convince the RM to include them, but if they
haven't had enough testing, then off to the next release with em.

-Todd

>
> On Jun 17, 2011, at 2:36 AM, Ryan Rawson wrote:
>
> > HDFS-918 and HDFS-347 are absolutely critical for random read
> > performance. =A0The smarter sites are already running HDFS-347 (I guess
> > they aren't running "Hadoop" then?), and soon they will be testing and
> > running HDFS-918 as well. =A0Opening 1 socket for every read just isn't
> > really scalable.
> >
> > -ryan
> >
> > On Fri, Jun 17, 2011 at 12:17 AM, Eric Baldeschwieler
> > <eric14@yahoo-inc.com> wrote:
> >> Hi Folks,
> >>
> >> I'd like to start a conversation on mainline planning and the next rel=
ease of Apache Hadoop beyond 0.22.
> >>
> >> The Yahoo! Hadoop team has been working hard to complete several big H=
adoop projects, including:
> >>
> >> - HDFS Federation [HDFS-1052]
> >> =A0- Already merged into trunk
> >>
> >> - Next Generation Map-Reduce [MR-279]
> >> =A0- Passing most tests now and discussing merging into trunk
> >>
> >> - The merging of our previous work on Hadoop with security into mainli=
ne [http://yhoo.it/i9Ww8W]
> >> =A0- This is mostly done, but owen and others are doing a scrub to clo=
se out the remaining issues
> >>
> >> All of these projects are now reaching a place where we would like to =
combine them with the good work already in 0.22 and put out a new apache re=
lease, perhaps 0.23. =A0We think the best way to accomplish that is to fini=
sh the merge in the next few weeks and then cut a release from trunk.
> >>
> >> Yahoo stands ready to help us (the Apache Hadoop Community) turn this =
new release into a stable release by running it through its 9 month test an=
d burn in process. =A0The result of that will be another stable release suc=
h as 0.18, 0.20 or 0.20.203 (hadoop with security). =A0We have Yahoo!s supp=
ort for this substantial investment because this new release will have a gr=
eat combination of new features for small and very large sites alike:
> >> =A0- New Write Pipeline - HBase support [also in 0.21 & 0.22]
> >> =A0- Federation - Scale up to larger clusters and the ability to exper=
iment with new namenode approaches
> >> =A0- Next Gen MapReduce - Scaleup, performance improvements, ability t=
o experiment with new processing frameworks
> >>
> >> I think this effort will produce a great new Apache Hadoop release for=
 the community. =A0I'm starting this thread to collect feedback and hopeful=
ly folks' endorsement for merging in MR-279 and putting together this new r=
elease. =A0Feedback please?
> >>
> >> Thanks,
> >>
> >> E14
> >>
> >>
>



--
Todd Lipcon
Software Engineer, Cloudera

From general-return-3855-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 18:17:59 2011
Return-Path: <general-return-3855-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 895CD4110
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 17 Jun 2011 18:17:59 +0000 (UTC)
Received: (qmail 43241 invoked by uid 500); 17 Jun 2011 18:17:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 43155 invoked by uid 500); 17 Jun 2011 18:17:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 43147 invoked by uid 99); 17 Jun 2011 18:17:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 18:17:57 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.62.32] (HELO qmta03.westchester.pa.mail.comcast.net) (76.96.62.32)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 18:17:49 +0000
Received: from omta17.westchester.pa.mail.comcast.net ([76.96.62.89])
	by qmta03.westchester.pa.mail.comcast.net with comcast
	id x66G1g00A1vXlb8536HVH7; Fri, 17 Jun 2011 18:17:29 +0000
Received: from fs ([24.4.185.157])
	by omta17.westchester.pa.mail.comcast.net with comcast
	id x6HT1g00N3QAh8g3d6HUJj; Fri, 17 Jun 2011 18:17:29 +0000
Received: from mail.boudnik.org (localhost [127.0.0.1])
	by fs (8.14.3/8.14.3/Debian-5+lenny1) with ESMTP id p5HIHQi8022461
	for <general@hadoop.apache.org>; Fri, 17 Jun 2011 11:17:26 -0700
Received: (from cos@localhost)
	by mail.boudnik.org (8.14.3/8.14.3/Submit) id p5HIHP0X022460
	for general@hadoop.apache.org; Fri, 17 Jun 2011 11:17:25 -0700
X-Authentication-Warning: mail.boudnik.org: cos set sender to cos@apache.org using -f
Date: Fri, 17 Jun 2011 11:17:25 -0700
From: Konstantin Boudnik <cos@apache.org>
To: general@hadoop.apache.org
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
Message-ID: <20110617181725.GB22372@linspire.com>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org> <BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com> <FBF55C01-D66A-44C0-9252-D54E7CAB103D@apache.org> <BANLkTi=BWPUEoqD3L98A9AXLkbafZs15Zw@mail.gmail.com> <92D32E6E-3B65-464F-B91B-4C30EAB2642C@apache.org> <BANLkTi=1JGe8kVortj1Act3jdt9L2RCrhw@mail.gmail.com> <4DF880DA.7060105@apache.org> <BANLkTim6DrVsb5NpwnvAvQtLqa5hq0eOrg@mail.gmail.com> <4DFB3421.2010902@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <4DFB3421.2010902@apache.org>
X-Organization: It's something of 'Cos
User-Agent: Mutt/1.5.18 (2008-05-17)

On Fri, Jun 17, 2011 at 12:01PM, Steve Loughran wrote:
> On 15/06/11 16:58, Konstantin Boudnik wrote:
>> On Wed, Jun 15, 2011 at 02:52, Steve Loughran<stevel@apache.org>  wrote:
>
>>>
>>> Regarding the vote, I think the discussion here is interesting and should be
>>> finalised before the vote. It's worth resolving the issues.
>>>
>>> also: banners, stickers and clothing? Can I have T-shirts saying "I broke
>>> the hadoop build" with the logo on, or should it be "I broke the Apache
>>> Hadoop build"?
>>
>> I think such a T-shirt should be forcefully worn on any person who did
>> just that.
>
> Here you go with the poster:
> http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/hadoop-components/hadoop-ops/doc/breaking_the_hadoop_build.odp?revision=8630
>
> I can add it to hadoop-common SVN for people to work on...

Please do, by all means :) !

From general-return-3856-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 18:22:09 2011
Return-Path: <general-return-3856-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7D1294CB7
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 17 Jun 2011 18:22:09 +0000 (UTC)
Received: (qmail 52995 invoked by uid 500); 17 Jun 2011 18:22:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52854 invoked by uid 500); 17 Jun 2011 18:22:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52846 invoked by uid 99); 17 Jun 2011 18:22:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 18:22:07 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 18:22:02 +0000
Received: by bwz8 with SMTP id 8so1349137bwz.35
        for <general@hadoop.apache.org>; Fri, 17 Jun 2011 11:21:41 -0700 (PDT)
Received: by 10.204.19.74 with SMTP id z10mr1938735bka.183.1308334901638; Fri,
 17 Jun 2011 11:21:41 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.37.79 with HTTP; Fri, 17 Jun 2011 11:21:21 -0700 (PDT)
In-Reply-To: <E9E17CC3-013C-46E5-AD48-BF76DEE8CE02@yahoo-inc.com>
References: <EFF679DA-26AB-4250-8F59-A2330D0E3E5C@yahoo-inc.com>
 <BANLkTimX8gSQRFbVZaVDQBOSN9c4iRi=Nw@mail.gmail.com> <BANLkTi=igCLuZ7v2+L9_nObs8wfPP4_F8g@mail.gmail.com>
 <E9E17CC3-013C-46E5-AD48-BF76DEE8CE02@yahoo-inc.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Fri, 17 Jun 2011 11:21:21 -0700
Message-ID: <BANLkTik6PoHHVB89dUA2C8EF29FbyGcyHg@mail.gmail.com>
Subject: Re: Thinking about the next hadoop mainline release
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Fri, Jun 17, 2011 at 7:15 AM, Arun C Murthy <arunc@yahoo-inc.com> wrote:
> I volunteer to be the RM for the release since I've been leading the NG NR effort.
>
> Are folks ok with this?

+1. It would be an honor to fix bugs for you, Arun.

-Todd


> Sent from my iPhone
>
> On Jun 17, 2011, at 1:45 PM, "Ted Dunning" <tdunning@maprtech.com> wrote:
>
>> NG map reduce is a huge deal both in terms of making things better for
>> users, but also in terms of unblocking the Hadoop development process.
>>
>> On Fri, Jun 17, 2011 at 9:36 AM, Ryan Rawson <ryanobjc@gmail.com> wrote:
>>
>>>> - Next Generation Map-Reduce [MR-279]
>>>> - Passing most tests now and discussing merging into trunk
>>>
>



-- 
Todd Lipcon
Software Engineer, Cloudera

From general-return-3857-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 19:00:20 2011
Return-Path: <general-return-3857-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2EC0448D9
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 17 Jun 2011 19:00:20 +0000 (UTC)
Received: (qmail 10641 invoked by uid 500); 17 Jun 2011 19:00:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 10590 invoked by uid 500); 17 Jun 2011 19:00:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 29238 invoked by uid 99); 17 Jun 2011 18:13:06 -0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
X-Authentication-Warning: mail.boudnik.org: cos set sender to Konstantin@Boudnik.org using -f
Date: Fri, 17 Jun 2011 11:12:35 -0700
From: Konstantin Boudnik <Konstantin@Boudnik.org>
To: general@hadoop.apache.org
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
Message-ID: <20110617181235.GA22372@linspire.com>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org> <BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com> <FBF55C01-D66A-44C0-9252-D54E7CAB103D@apache.org> <BANLkTi=BWPUEoqD3L98A9AXLkbafZs15Zw@mail.gmail.com> <92D32E6E-3B65-464F-B91B-4C30EAB2642C@apache.org> <BANLkTi=1JGe8kVortj1Act3jdt9L2RCrhw@mail.gmail.com> <4DF880DA.7060105@apache.org> <BANLkTim6DrVsb5NpwnvAvQtLqa5hq0eOrg@mail.gmail.com> <4DFB3421.2010902@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <4DFB3421.2010902@apache.org>
X-Organization: It's something of 'Cos
User-Agent: Mutt/1.5.18 (2008-05-17)

On Fri, Jun 17, 2011 at 12:01PM, Steve Loughran wrote:
> On 15/06/11 16:58, Konstantin Boudnik wrote:
>> On Wed, Jun 15, 2011 at 02:52, Steve Loughran<stevel@apache.org>  wrote:
>
>>>
>>> Regarding the vote, I think the discussion here is interesting and should be
>>> finalised before the vote. It's worth resolving the issues.
>>>
>>> also: banners, stickers and clothing? Can I have T-shirts saying "I broke
>>> the hadoop build" with the logo on, or should it be "I broke the Apache
>>> Hadoop build"?
>>
>> I think such a T-shirt should be forcefully worn on any person who did
>> just that.
>
> Here you go with the poster:
> http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/hadoop-components/hadoop-ops/doc/breaking_the_hadoop_build.odp?revision=8630
>
> I can add it to hadoop-common SVN for people to work on...

Please do :)!


From general-return-3858-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 20:27:49 2011
Return-Path: <general-return-3858-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 553204A30
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 17 Jun 2011 20:27:49 +0000 (UTC)
Received: (qmail 46043 invoked by uid 500); 17 Jun 2011 20:27:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45972 invoked by uid 500); 17 Jun 2011 20:27:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45964 invoked by uid 99); 17 Jun 2011 20:27:47 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 20:27:47 +0000
Received: from localhost (HELO awittena-md.linkedin.biz) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 20:27:47 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Thinking about the next hadoop mainline release
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <F77C0A6C-92AC-4486-8326-589D1D297F9A@apache.org>
Date: Fri, 17 Jun 2011 13:27:43 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <4C60EEFE-1183-47DD-9E2E-EB6EFF589727@apache.org>
References: <EFF679DA-26AB-4250-8F59-A2330D0E3E5C@yahoo-inc.com> <F77C0A6C-92AC-4486-8326-589D1D297F9A@apache.org>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On Jun 17, 2011, at 10:31 AM, Allen Wittenauer wrote:

>=20
> On Jun 17, 2011, at 12:17 AM, Eric Baldeschwieler wrote:
>> Yahoo stands ready to help us (the Apache Hadoop Community) turn this =
new release into a stable release by running it through its 9 month test =
and burn in process.  The result of that will be another stable release =
such as 0.18, 0.20 or 0.20.203 (hadoop with security). =20
>=20
>=20
> 	I'd consider 0.20.203 pseudo-stable.=20

	Actually, I was just reminded about the complete disaster that =
is metrics.  So while it may be pseudo-stable, it  isn't actually usable =
for anyone but Yahoo!.=

From general-return-3859-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 01:14:17 2011
Return-Path: <general-return-3859-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 208A94859
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat, 18 Jun 2011 01:14:17 +0000 (UTC)
Received: (qmail 42218 invoked by uid 500); 18 Jun 2011 01:14:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42162 invoked by uid 500); 18 Jun 2011 01:14:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42153 invoked by uid 99); 18 Jun 2011 01:14:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 01:14:15 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bcwalrus@cloudera.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 01:14:09 +0000
Received: by gyb11 with SMTP id 11so204638gyb.35
        for <general@hadoop.apache.org>; Fri, 17 Jun 2011 18:13:48 -0700 (PDT)
Received: by 10.101.108.14 with SMTP id k14mr3091410anm.89.1308359628290; Fri,
 17 Jun 2011 18:13:48 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.100.146.12 with HTTP; Fri, 17 Jun 2011 18:13:28 -0700 (PDT)
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
From: bc Wong <bcwalrus@cloudera.com>
Date: Fri, 17 Jun 2011 18:13:28 -0700
Message-ID: <BANLkTikaL7Yqor4ThLiRCvnCGEqa_6xHDA@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8

5, 6, 2.

Cheers,
bc

From general-return-3860-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 02:03:43 2011
Return-Path: <general-return-3860-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F1788682F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat, 18 Jun 2011 02:03:42 +0000 (UTC)
Received: (qmail 74497 invoked by uid 500); 18 Jun 2011 02:03:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74429 invoked by uid 500); 18 Jun 2011 02:03:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74420 invoked by uid 99); 18 Jun 2011 02:03:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 02:03:41 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 216.145.54.173 is neither permitted nor denied by domain of rajive@yahoo-inc.com)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 02:03:35 +0000
Received: from bringlake.corp.yahoo.com (bringlake.corp.yahoo.com [10.72.115.56])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p5I22pfu040438
	for <general@hadoop.apache.org>; Fri, 17 Jun 2011 19:02:51 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1308362571;
	bh=ScavgmsmK31vv+EL6/irtONHPh21ZzuHOfpjZjTlcFA=;
	h=Date:From:To:Subject:Message-ID:Reply-To:References:MIME-Version:
	 Content-Type:In-Reply-To;
	b=oyvnMCP6eW7ScC4Rn1yReQxl98Ixjnh1igXY3VBfEMxWV/LDTe7sVm08cT2JEoPV1
	 bPpDEoicpyql8ZCaCryVbkdyy4AJZnZZ8aCpQoSQZnYCHshyuxVPBXsz2oVdBw5lZZ
	 8sAoKq0Cl2GdX5ssMXkAcBa+El3ND79l52KH33jw=
Received: from bringlake.corp.yahoo.com (localhost.localdomain [127.0.0.1])
	by bringlake.corp.yahoo.com (8.13.8/8.13.8) with ESMTP id p5I22pHE020964
	for <general@hadoop.apache.org>; Fri, 17 Jun 2011 19:02:51 -0700
Received: (from rajive@localhost)
	by bringlake.corp.yahoo.com (8.13.8/8.13.8/Submit) id p5I22prA020963
	for general@hadoop.apache.org; Fri, 17 Jun 2011 19:02:51 -0700
X-Authentication-Warning: bringlake.corp.yahoo.com: rajive set sender to rajive@yahoo-inc.com using -f
Date: Fri, 17 Jun 2011 19:02:51 -0700
From: Rajiv Chittajallu <rajive@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: Thinking about the next hadoop mainline release
Message-ID: <20110618020251.GB16097@bringlake.corp.yahoo.com>
Reply-To: Rajiv Chittajallu <rajive@yahoo-inc.com>
References: <EFF679DA-26AB-4250-8F59-A2330D0E3E5C@yahoo-inc.com>
 <F77C0A6C-92AC-4486-8326-589D1D297F9A@apache.org>
 <4C60EEFE-1183-47DD-9E2E-EB6EFF589727@apache.org>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="FkmkrVfFsRoUs1wW"
Content-Disposition: inline
In-Reply-To: <4C60EEFE-1183-47DD-9E2E-EB6EFF589727@apache.org>
Organization: Y!
User-Agent: Mutt/1.5.20 (2009-06-14)
X-Virus-Checked: Checked by ClamAV on apache.org

--FkmkrVfFsRoUs1wW
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Allen Wittenauer wrote on 06/17/11 at 13:27:43 -0700:
>
>	Actually, I was just reminded about the complete disaster that is metrics=
=2E  So while it may be pseudo-stable, it  isn't actually usable for anyone=
 but Yahoo!.

Did you try to use the new metrics framework? Are you complaining
cause there is no port of ganglia metrics module to metrics V2?=20

Its pluggable and you can choose to disable it if you want to.=20
All metrics and status information is available via jmx.=20
Multiple sinks and filters, more importantly you can refresh
metrics configs without restarting the process.=20

-rajive

--FkmkrVfFsRoUs1wW
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)

iD8DBQFN/AdLobJMer3p0EoRAuFsAJ0WlgTkfeqYp6r6NplTtRmb/jd+QACcDIZ3
nkd1O0YlY+iMbaKRF1dDULg=
=lZa/
-----END PGP SIGNATURE-----

--FkmkrVfFsRoUs1wW--

From general-return-3861-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 02:15:04 2011
Return-Path: <general-return-3861-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B0BDD6BE0
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat, 18 Jun 2011 02:15:04 +0000 (UTC)
Received: (qmail 81289 invoked by uid 500); 18 Jun 2011 02:15:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81220 invoked by uid 500); 18 Jun 2011 02:15:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81211 invoked by uid 99); 18 Jun 2011 02:15:03 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 02:15:03 +0000
Received: from localhost (HELO [192.168.100.72]) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 02:15:03 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Thinking about the next hadoop mainline release
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <20110618020251.GB16097@bringlake.corp.yahoo.com>
Date: Fri, 17 Jun 2011 19:15:02 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <1EB86860-E788-40E8-893D-9543C20F72E8@apache.org>
References: <EFF679DA-26AB-4250-8F59-A2330D0E3E5C@yahoo-inc.com> <F77C0A6C-92AC-4486-8326-589D1D297F9A@apache.org> <4C60EEFE-1183-47DD-9E2E-EB6EFF589727@apache.org> <20110618020251.GB16097@bringlake.corp.yahoo.com>
To: <general@hadoop.apache.org>,
 Rajiv Chittajallu <rajive@yahoo-inc.com>
X-Mailer: Apple Mail (2.1082)


On Jun 17, 2011, at 7:02 PM, Rajiv Chittajallu wrote:

> Allen Wittenauer wrote on 06/17/11 at 13:27:43 -0700:
>>=20
>> 	Actually, I was just reminded about the complete disaster that =
is metrics.  So while it may be pseudo-stable, it  isn't actually usable =
for anyone but Yahoo!.
>=20
> Did you try to use the new metrics framework? Are you complaining
> cause there is no port of ganglia metrics module to metrics V2?=20

	Like 80-90%+ of the Hadoop sites that gather metrics, yes, we =
use Ganglia.=20

> Its pluggable and you can choose to disable it if you want to.=20
> All metrics and status information is available via jmx.=20
> Multiple sinks and filters, more importantly you can refresh
> metrics configs without restarting the process.=20

	But first we need to switch to a new metrics infrastructure.  =
This wouldn't have been so bad if the release notes actually had said =
"You know your current metrics system? Yeah, toss it because we just =
broke everything."

	I'm certainly not a fan of Ganglia, but this sort of unexpected =
breakage is very, very bad.=

From general-return-3862-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 04:42:53 2011
Return-Path: <general-return-3862-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 37EA26D4D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat, 18 Jun 2011 04:42:53 +0000 (UTC)
Received: (qmail 38136 invoked by uid 500); 18 Jun 2011 04:42:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37411 invoked by uid 500); 18 Jun 2011 04:42:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37402 invoked by uid 99); 18 Jun 2011 04:42:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 04:42:49 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 216.145.54.172 is neither permitted nor denied by domain of arunc@yahoo-inc.com)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 04:42:40 +0000
Received: from SP2-EX07CAS03.ds.corp.yahoo.com (sp2-ex07cas03.corp.sp2.yahoo.com [98.137.59.35])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5I4gC9v094597
	for <general@hadoop.apache.org>; Fri, 17 Jun 2011 21:42:12 -0700 (PDT)
Received: from SP2-EX07VS04.ds.corp.yahoo.com ([98.137.59.33]) by
 SP2-EX07CAS03.ds.corp.yahoo.com ([98.137.59.35]) with mapi; Fri, 17 Jun 2011
 21:42:13 -0700
From: Arun C Murthy <arunc@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 17 Jun 2011 21:42:04 -0700
Subject: Re: Thinking about the next hadoop mainline release
Thread-Topic: Thinking about the next hadoop mainline release
Thread-Index: AcwtchbSroOV+iaXSzO254S3IsLb+Q==
Message-ID: <03019E1C-61F7-410E-9978-140C00D91E45@yahoo-inc.com>
References: <EFF679DA-26AB-4250-8F59-A2330D0E3E5C@yahoo-inc.com>
 <F77C0A6C-92AC-4486-8326-589D1D297F9A@apache.org>
 <4C60EEFE-1183-47DD-9E2E-EB6EFF589727@apache.org>
 <20110618020251.GB16097@bringlake.corp.yahoo.com>
 <1EB86860-E788-40E8-893D-9543C20F72E8@apache.org>
In-Reply-To: <1EB86860-E788-40E8-893D-9543C20F72E8@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Allen,

Can we pls focus on the next release here, on this thread?

Please provide your valuable f/b to the RM abt stuff you'd like to see in t=
he next release.

thanks,
Arun

Sent from my iPhone

On Jun 18, 2011, at 7:45 AM, "Allen Wittenauer" <aw@apache.org> wrote:

>=20
> On Jun 17, 2011, at 7:02 PM, Rajiv Chittajallu wrote:
>=20
>> Allen Wittenauer wrote on 06/17/11 at 13:27:43 -0700:
>>>=20
>>>    Actually, I was just reminded about the complete disaster that is me=
trics.  So while it may be pseudo-stable, it  isn't actually usable for any=
one but Yahoo!.
>>=20
>> Did you try to use the new metrics framework? Are you complaining
>> cause there is no port of ganglia metrics module to metrics V2?=20
>=20
>    Like 80-90%+ of the Hadoop sites that gather metrics, yes, we use Gang=
lia.=20
>=20
>> Its pluggable and you can choose to disable it if you want to.=20
>> All metrics and status information is available via jmx.=20
>> Multiple sinks and filters, more importantly you can refresh
>> metrics configs without restarting the process.=20
>=20
>    But first we need to switch to a new metrics infrastructure.  This wou=
ldn't have been so bad if the release notes actually had said "You know you=
r current metrics system? Yeah, toss it because we just broke everything."
>=20
>    I'm certainly not a fan of Ganglia, but this sort of unexpected breaka=
ge is very, very bad.

From general-return-3863-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 06:11:56 2011
Return-Path: <general-return-3863-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F2E576FF1
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat, 18 Jun 2011 06:11:55 +0000 (UTC)
Received: (qmail 64851 invoked by uid 500); 18 Jun 2011 06:11:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 64738 invoked by uid 500); 18 Jun 2011 06:11:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 64728 invoked by uid 99); 18 Jun 2011 06:11:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 06:11:52 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 216.145.54.173 is neither permitted nor denied by domain of eric14@yahoo-inc.com)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 06:11:39 +0000
Received: from [10.0.1.3] (snvvpn4-10-72-168-c87.hq.corp.yahoo.com [10.72.168.87])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p5I6AY7Q094629
	for <general@hadoop.apache.org>; Fri, 17 Jun 2011 23:10:35 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1308377435;
	bh=+yRlYQBEJCdtrlVNFzr+WnSgXwffZieS1ybdpGPEa/U=;
	h=Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:
	 Content-Transfer-Encoding:Message-Id:References:To;
	b=qOUi30pM2POHjje6KlcDZvNpFLNY/c8HHXDVUQqlAGpPk9Qx9COBzAtqyg1b9iR3U
	 y6JlDBf6Zc7OPYaHcUluPZT7BIRmWrAVzWdblvpFc2vxyIoFIUUgyAMSSuENa7eFiQ
	 g0YgHE93Uxr4UBPdSxgEoFDbYCgMMkpbB10k2YlM=
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: Thinking about the next hadoop mainline release
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <43D4BB64-1B14-45F6-941B-B65F31636266@apache.org>
Date: Fri, 17 Jun 2011 23:10:33 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <76BC80C1-7A3B-4B80-93C2-DBF1EF3152F0@yahoo-inc.com>
References: <EFF679DA-26AB-4250-8F59-A2330D0E3E5C@yahoo-inc.com> <BANLkTimX8gSQRFbVZaVDQBOSN9c4iRi=Nw@mail.gmail.com> <43D4BB64-1B14-45F6-941B-B65F31636266@apache.org>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1084)

Hey Allen,

I agree with you that we should avoid future regressions like this.  I =
think the tradeoff that got security working well on one heavily used =
platform was the right one for hadoop-with-security, but I'll be sure to =
raise such issues for discussion in the future as soon as I become aware =
of them.

I agree that we should think about how to generalize the security work =
for other platforms.  That work is just waiting for someone to jump =
in....

On ganglia and the metrics framework, it would be great if you could put =
your head together with rajive and come up with a joint proposal on what =
docs / code is needed to fix the regression in a clean way.  That sounds =
like something we should be thinking about fixing in the 20 line and all =
future releases.

thanks,

E14




On Jun 17, 2011, at 10:33 AM, Allen Wittenauer wrote:

>=20
> On Jun 17, 2011, at 12:36 AM, Ryan Rawson wrote:
>=20
>> HDFS-918 and HDFS-347 are absolutely critical for random read
>> performance.  The smarter sites are already running HDFS-347 (I guess
>> they aren't running "Hadoop" then?), and soon they will be testing =
and
>> running HDFS-918 as well.  Opening 1 socket for every read just isn't
>> really scalable.
>=20
> 	Isn't "random read [on HDFS]" and "smarter sites" in the same =
breath an oxymoron?
>=20
>=20


From general-return-3864-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 06:51:22 2011
Return-Path: <general-return-3864-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1A7FA64D1
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat, 18 Jun 2011 06:51:22 +0000 (UTC)
Received: (qmail 81557 invoked by uid 500); 18 Jun 2011 06:51:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81329 invoked by uid 500); 18 Jun 2011 06:51:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81321 invoked by uid 99); 18 Jun 2011 06:51:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 06:51:13 +0000
X-ASF-Spam-Status: No, hits=4.2 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL,URIBL_BLACK
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 216.145.54.171 is neither permitted nor denied by domain of eric14@yahoo-inc.com)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 06:51:09 +0000
Received: from [10.0.1.3] (snvvpn4-10-72-168-c87.hq.corp.yahoo.com [10.72.168.87])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5I6ofbv057275
	for <general@hadoop.apache.org>; Fri, 17 Jun 2011 23:50:42 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1308379842;
	bh=5e3fxNbGO695IKal4JVtPwM6ONxxEf128L0JgfJpNVY=;
	h=From:Content-Type:Content-Transfer-Encoding:Subject:Date:
	 Message-Id:To:Mime-Version;
	b=AxcbalspvFtLntvEV8h/IhAOjSOA5crzJwLiT3Vs3PKV/2jn74lMiyQqJH+WQJIqS
	 Ge5hkMwFMGNSCIpatwk0HCPLgvVUgqLL7tJZu8FpM+woFQLy3mB3o1f3boYHWGz4c4
	 Zg9S27RL919VUu3hgKJkluLLpz28BU7dMcpHV1os=
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: [DISCUSSION] Thinking about 20.204 and beyond 
Date: Fri, 17 Jun 2011 23:50:40 -0700
Message-Id: <4E18CE25-6863-4318-8A95-17896B0B87D1@yahoo-inc.com>
To: general@hadoop.apache.org
Mime-Version: 1.0 (Apple Message framework v1084)
X-Mailer: Apple Mail (2.1084)

Hi Folks,

Along with starting a new release off the mainline (see previous mail), =
the Yahoo! team plans to continue producing sustaining releases off the =
Hadoop with security branch, such as 0.20.203 .  I'm writing this email =
to outline our plans, explain Yahoo's motivation for supporting this =
work and request feedback and hopefully your endorsement.  This =
initiative stems from Yahoo's commitment to do its hadoop work in Apache =
and discontinue the Yahoo Distribution of Hadoop =
(http://yhoo.it/i9Ww8W).

We hope to produce a new 0.20.204 release in Apache in the next few =
weeks.  Owen O'Malley is planning to act as release master for this =
release.  This will be based on work in the hadoop-with-security branch, =
just as 0.20.203 but will include bugfixes and enhancements beyond those =
in 0.20.203.  This is one in a series of releases we hope to do in the =
next 6-9 months as hadoop 0.23 (or whatever the community chooses to =
call it) goes through the various stages of stability testing and =
burn-in.

CONTENTS OF THE RELEASE:

Some highlights:

- RPM & .deb packaging to ease deployment (back ported from trunk)
 - I am excited to see hadoop released with .deb & RPM packaging from =
Apache for the first time.
 - This will greatly ease deployment
- Disk fail in place (merged with trunk, except for some MR changes =
conflict with MR-279, these will be reimplemented in MR-279)
 - This change has been motivated by operational problems we observed =
with our new 12 disk machines.
 - This work should greatly improve Hadoop availability by keeping nodes =
working when one of their disks fails

- Lots of of additional fixes (I've included the change log below)

WHY THIS PROCESS:

Producing a stable release of Hadoop is a long, hard and expensive =
process.  Historically Y! has produced all such releases.  Other =
releases of Hadoop have either not been stable (Hadoop 0.19 and Hadoop =
0.21) or have been based on a stable Apache release driven by Yahoo (CDH =
and Facebook).  Once we've paid the price of making a stable release, it =
makes a lot of sense to accept safe improvements as well as bug fixes.  =
Doing so allows one to get customer impacting improvements into =
production in days, rather than years, which is what would happen if one =
waited for changes to come in the next stable release off the Hadoop =
mainline.  Given that it takes many months to stabilize trunk, there is =
no way to get new easy fixes into users hands quickly via a new mainline =
release.

For the last few years Yahoo has done sustaining engineering in open =
source via Github.  These patches have been contributed to Apache Hadoop =
mainline and backported to the sustaining branch on github (for yahoo =
0.20 for example).  We've then cut Yahoo releases from Github.  Cloudera =
and Facebook have also taken these patches from Github and incorporated =
these improvements into their releases, so the community has benefitted =
from this process for years.  What we are planning to do now is simply =
move this process into Apache, so that Apache releases themselves are =
timely and relevant, not always a year or two behind what users need.

How do I propose making these decisions?  Deciding what is a safe patch =
is a judgement call.  Apache process suggests that the release manager =
makes these calls (http://bit.ly/mJcBjc).  For releases Y! champions, =
such as 0.20 (arun & owen), we are ready to do the sustaining =
engineering, make these calls and stand our reputation behind the =
quality of the result.  Other release masters are championing other =
Apache Hadoop releases currently (Nigel and Tom) and I think they should =
be free to do the same.  For hadoop 20.204, I propose pushing what is =
currently in the hadoop-with-security branch.  Part of the reason for =
this thread is to socialize this process, so that community members can =
champion stable patches for inclusion in 20.205.  In the future I =
propose that a branch's release master request suggestions for future =
releases on this list, but is free to use their judgement on what is =
accepted (pretty much what nigel is doing on 0.22 today).

----

Conclusion:

The vote on 0.20.203 was acrimonious, but I believe that 0.20.203 was a =
useful step forward for Apache Hadoop.  0.20.204 will again be the best =
stable release of Apache Hadoop ever.  I hope folks can support the =
effort.  With your contribution 0.20.205 can be even better, fixing =
issues that plague your Hadoop clusters.  This email is part of a wider =
effort from the Yahoo team to co-plan our work with the community.

Thanks,

Eric14
---
eric14 a.k.a. Eric Baldeschwieler
VP Hadoop Software Development @Yahoo!


=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

=46rom CHANGES.txt:

Release 0.20.204.0 - unreleased

 NEW FEATURES

   HADOOP-6255. Create RPM and Debian packages for common. Changes =
deployment
   layout to be consistent across the binary tgz, rpm, and deb. Adds =
setup
   scripts for easy one node cluster configuration and user creation.
   (Eric Yang via omalley)

 BUG FIXES

   MAPREDUCE-2495. exit() the TaskTracker when the distributed cache =
cleanup
   thread dies. (Robert Joseph Evans via cdouglas)

   HDFS-1878. TestHDFSServerPorts unit test failure - race condition=20
   in FSNamesystem.close() causes NullPointerException without serious
   consequence. (mattf)

   MAPREDUCE-2452. Moves the cancellation of delegation tokens to a =
separate
   thread. (ddas)

   MAPREDUCE-2555. Avoid sprious logging from completedtasks. (Thomas =
Graves
   via cdouglas)

   MAPREDUCE-2451. Log the details from health check script at the
   JobTracker. (Thomas Graves via cdouglas)

   MAPREDUCE-2535. Fix NPE in JobClient caused by retirement. (Robert =
Joseph
   Evans via cdouglas)

   MAPREDUCE-2456. Log the reduce taskID and associated TaskTrackers =
with
   failed fetch notifications in the JobTracker log.
   (Jeffrey Naisbitt via cdouglas)

   HDFS-2044. TestQueueProcessingStatistics failing automatic test due =
to=20
   timing issues. (mattf)

   HADOOP-7248. Update eclipse target to generate .classpath from ivy =
config.
   (Thomas Graves and Tom White via cdouglas)

   MAPREDUCE-2558. Add queue-level metrics 0.20-security branch - test =
fix
   (jeffrey nasbit via mahadev)

   HADOOP-7364. TestMiniMRDFSCaching fails if test.build.dir is set to=20=

   something other than build/test. (Thomas Graves via mahadev)

   HADOOP-7277. Add generation of run configurations to eclipse target.
   (Jeffrey Naisbitt and Philip Zeyliger via cdouglas)

   HADOOP-7373. Fix {start,stop}-{dfs,mapred} and hadoop-daemons.sh from
   trying to use the wrong bin directory. (omalley)

   HADOOP-7274. Fix typos in IOUtils. (Jonathan Eagles via cdouglas)

   HADOOP-7369. Fix permissions in tarball for sbin/* and libexec/* =
(omalley)

   MAPREDUCE-2479. Move distributed cache cleanup to a background task,
   backporting MAPREDUCE-1568. (Robert Joseph Evans via cdouglas)

   HADOOP-7356. Fix bin/hadoop scripts (eyang via omalley)

   HADOOP-7272. Remove unnecessary security related info logs. (suresh)

   MAPREDUCE-2514. Fix typo in TaskTracker ReinitTrackerAction log =
message.
   (Jonathan Eagles via cdouglas)

   HDFS-1906. Remove logging exception stack trace in client logs when =
one of
   the datanode targets to read from is not reachable. (suresh)

   MAPREDUCE-2490. Add logging to graylist and blacklist activity to aid
   diagnosis of related issues. (Jonathan Eagles via cdouglas)

   MAPREDUCE-2447. Fix Child.java to set Task.jvmContext sooner to avoid
   corner cases in error handling. (Siddharth Seth via acmurthy)=20

   MAPREDUCE-2429. Validate JVM in TaskUmbilicalProtocol. (Siddharth =
Seth via
   acmurthy)=20

   MAPREDUCE-2418. Show job errors in JobHistory page. (Siddharth Seth =
via
   acmurthy)=20

   HDFS-1592. At Startup, Valid volumes required in FSDataset doesn't
   handle consistently with volumes tolerated. (Bharath Mundlapudi)

   HDFS-1598. Directory listing on hftp:// does not show
   .*.crc files.  (szetszwo)

   HDFS-1750. ListPathsServlet should not use =
HdfsFileStatus.getLocalName()
   to get file name since it may return an empty string.  (szetszwo)

   HDFS-1758. Make Web UI JSP pages thread safe. (Tanping via suresh)

   HDFS-1773. Do not show decommissioned datanodes, which are not in =
both
   include and exclude lists, on web and JMX interfaces.
   (Tanping Wang via szetszwo)

   MAPREDUCE-2409. Distinguish distributed cache artifacts localized as
   files, archives. (Siddharth Seth via cdouglas)

   MAPREDUCE-118. Fix Job.getJobID() to get the new ID as soon as it's=20=

   assigned. (Amareshwari Sriramadasu and Dick King via cdouglas)

   MAPREDUCE-2411. Force an exception when the queue has an invalid name =
or
   its ACLs are misconfigured. (Dick King via cdouglas)

   HDFS-1258. Clearing namespace quota on "/" corrupts fs image. =20
   (Aaron T. Myers via szetszwo)

   HDFS-1189. Quota counts missed between clear quota and set quota.
   (John George via szetszwo)

   HDFS-1692. In secure mode, Datanode process doesn't exit when disks=20=

   fail. (bharathm via boryas)

   MAPREDUCE-2420. JobTracker should be able to renew delegation token=20=

   over HTTP (boryas)

   MAPREDUCE-2443. Fix TaskAspect for TaskUmbilicalProtocol.ping(..).
   (Siddharth Seth via szetszwo)

   HDFS-1842. Handle editlog opcode conflict with 0.20.203 during =
upgrade,
   by throwing an error to indicate the editlog needs to be empty.
   (suresh)

   HDFS-1377. Quota bug for partial blocks allows quotas to be violated. =
(eli)

   HDFS-2057. Wait time to terminate the threads causes unit tests to
   take longer time. (Bharath Mundlapudi via suresh)

 IMPROVEMENTS

   HADOOP-7144. Expose JMX metrics via JSON servlet. (Robert Joseph =
Evans via
   cdouglas)

   MAPREDUCE-2524. Port reduce failure reporting semantics from trunk, =
to
   fail faulty maps more aggressively. (Thomas Graves via cdouglas)
   MAPREDUCE-2529. Add support for regex-based shuffle metric counting
   exceptions. (Thomas Graves via cdouglas)

   HADOOP-7398. Suppress warnings about use of HADOOP_HOME. (omalley)

   MAPREDUCE-2415. Distribute the user task logs on to multiple disks.
   (Bharath Mundlapudi via omalley)

   MAPREDUCE-2413. TaskTracker should handle disk failures by =
reinitializing=20
   itself. (Ravi Gummadi and Jagane Sundar via omalley)

   HDFS-1541. Not marking datanodes dead when namenode in safemode.
   (hairong)

   HDFS-1767. Namenode ignores non-initial block report from datanodes
   when in safemode during startup. (Matt Foley via suresh)

   MAPREDUCE-1251. c++ utils doesn't compile. (Eli Collins via shv)

  HADOOP-7330. Fix MetricsSourceAdapter to use the value instead of the=20=

   object. (Luke Lu via omalley)


From general-return-3865-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 08:18:57 2011
Return-Path: <general-return-3865-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 28DB164A5
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat, 18 Jun 2011 08:18:57 +0000 (UTC)
Received: (qmail 21084 invoked by uid 500); 18 Jun 2011 08:18:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21031 invoked by uid 500); 18 Jun 2011 08:18:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21023 invoked by uid 99); 18 Jun 2011 08:18:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 08:18:55 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of szhaoyu@gmail.com designates 209.85.213.48 as permitted sender)
Received: from [209.85.213.48] (HELO mail-yw0-f48.google.com) (209.85.213.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 08:18:49 +0000
Received: by ywn13 with SMTP id 13so1541533ywn.35
        for <general@hadoop.apache.org>; Sat, 18 Jun 2011 01:18:28 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=pZFOTQUK9mYpMIYqLS6k131QbVpydjQDrWWUKuxwiK0=;
        b=FB+D3Rz975lN6JriKKTvtgMfr+hitolRgnZI2xiZiep5UAzpjKF1diysE0Df3ZGZ+N
         6EAP0h0YZFwBawyjP5M+9wUJLyt2onk8Y5Q0Wdy4lvI9JdrgW3A2ZIk429aePp6WBKcS
         BTn9Wup6ww9j7bU34hJgIlsDY1HMqgk0t4A2o=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=VwgvZYC5PtzQ+kWpWZaLrnGU+p19LYy5pH95BR88J882lLt4ZYFHaLDq2D9MG59X0X
         Kg79AHEvyHaSlanY2SAN6h9QyYWuhDnikcn+w+IxHY1R1LiFdM4jughAKmqq+F41oi9H
         rV0urRc54PRizgFRQO81INuKOeT4RwCR++iNo=
MIME-Version: 1.0
Received: by 10.150.32.17 with SMTP id f17mr3290244ybf.267.1308385108323; Sat,
 18 Jun 2011 01:18:28 -0700 (PDT)
Received: by 10.151.108.5 with HTTP; Sat, 18 Jun 2011 01:18:28 -0700 (PDT)
In-Reply-To: <BANLkTikaL7Yqor4ThLiRCvnCGEqa_6xHDA@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
	<BANLkTikaL7Yqor4ThLiRCvnCGEqa_6xHDA@mail.gmail.com>
Date: Sat, 18 Jun 2011 16:18:28 +0800
Message-ID: <BANLkTi=2qVYW5qShB42+X4ZrUmry7kAOCQ@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: =?UTF-8?B?5a2Z5YWG546J?= <szhaoyu@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd2a0249c6d7704a5f8259a

--000e0cd2a0249c6d7704a5f8259a
Content-Type: text/plain; charset=ISO-8859-1

2 5 6

--000e0cd2a0249c6d7704a5f8259a--

From general-return-3866-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 20:22:45 2011
Return-Path: <general-return-3866-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 59EB66C6E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat, 18 Jun 2011 20:22:45 +0000 (UTC)
Received: (qmail 32534 invoked by uid 500); 18 Jun 2011 20:22:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32491 invoked by uid 500); 18 Jun 2011 20:22:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32483 invoked by uid 99); 18 Jun 2011 20:22:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 20:22:43 +0000
X-ASF-Spam-Status: No, hits=3.5 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS,URIBL_BLACK
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [208.97.132.66] (HELO homiemail-a28.g.dreamhost.com) (208.97.132.66)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 20:22:36 +0000
Received: from homiemail-a28.g.dreamhost.com (localhost [127.0.0.1])
	by homiemail-a28.g.dreamhost.com (Postfix) with ESMTP id 152501B405F
	for <general@hadoop.apache.org>; Sat, 18 Jun 2011 13:22:15 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; q=dns; s=
	gbiv.com; b=oLdsd6DY1cYOHGlSBXMdpEgoDA0sXjLc4DskLKYsLSi4IHKAoeb4
	zbgFtDB0Wh+KHPH/Gs9p9imgjExOWo/sQcuAarSFjrAsp1RB+5hFZIr6sCKSjvr4
	2ksVw9EJS4vXUrQevo9kncTC559CVrWPDAVOP+0khllo/nf5L0j54sg=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; s=gbiv.com;
	 bh=0GhER5n6CRIBo2P416OrXnhJwsE=; b=1FPdRIphEfb+gS5gQyaIkrcllNwZ
	lkicoYoYayT4VYY3RPl4o5Ehh27pslnJvFm4rccMhuUikow95e+d7G7WI60UddgS
	+0E/pZwYPw5MLCX/7KceYIrC22x2sXwYSMeMfjGtFk/LdZYeM2u9XDicasxPBaYd
	d7jz22hmIlZbUro=
Received: from [192.168.1.84] (99-21-208-82.lightspeed.irvnca.sbcglobal.net [99.21.208.82])
	(using TLSv1 with cipher AES128-SHA (128/128 bits))
	(No client certificate requested)
	(Authenticated sender: fielding@gbiv.com)
	by homiemail-a28.g.dreamhost.com (Postfix) with ESMTPSA id DCA521B4058
	for <general@hadoop.apache.org>; Sat, 18 Jun 2011 13:22:14 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: [DISCUSSION] Thinking about 20.204 and beyond 
From: "Roy T. Fielding" <fielding@gbiv.com>
In-Reply-To: <4E18CE25-6863-4318-8A95-17896B0B87D1@yahoo-inc.com>
Date: Sat, 18 Jun 2011 13:22:14 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <0C4895F3-95E8-45ED-B2B0-C3BF4B5436B1@gbiv.com>
References: <4E18CE25-6863-4318-8A95-17896B0B87D1@yahoo-inc.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

On Jun 17, 2011, at 11:50 PM, Eric Baldeschwieler wrote:

> Along with starting a new release off the mainline (see previous =
mail), the Yahoo! team plans to continue producing sustaining releases =
off the Hadoop with security branch, such as 0.20.203 .  I'm writing =
this email to outline our plans, explain Yahoo's motivation for =
supporting this work and request feedback and hopefully your =
endorsement.  This initiative stems from Yahoo's commitment to do its =
hadoop work in Apache and discontinue the Yahoo Distribution of Hadoop =
(http://yhoo.it/i9Ww8W).

Eric,

Please make an effort to understand that we act at Apache as =
individuals.
That includes you and your team.  Yes, everyone greatly appreciates the
fact that Yahoo! pays your salaries.  Nevertheless, here you are called
an individual named Eric.

Companies do not have motivations.  You do.  There is nothing wrong with
each and every individual here being motivated, at least partly, by what
their boss(es) consider the right path forward.  But you don't have to
sell it as something Yahoo! is doing, and you don't need to send =
marketing
messaging to the development lists at Apache.  Just be yourself and
represent yourself.

Anyone on the PMC can be an RM.  A package can be rolled at any time, =
and
a release vote on that package can be called at any time.  While it =
makes
sense for the individual planning to RM a release to announce their =
plans,
doing so does not in any way form the direction the product is headed,
nor does it exclude anyone else from being RM of their own release =
process.

There is absolutely no reason that trunk cannot be packaged for release
tomorrow as 0.23.  There may be many reasons why it won't pass a release
vote, but we probably aren't going to find them until somebody tries.

....Roy=

From general-return-3867-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 22:50:06 2011
Return-Path: <general-return-3867-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 62C1E6C3C
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat, 18 Jun 2011 22:50:06 +0000 (UTC)
Received: (qmail 92677 invoked by uid 500); 18 Jun 2011 22:50:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92608 invoked by uid 500); 18 Jun 2011 22:50:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 92600 invoked by uid 99); 18 Jun 2011 22:50:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 22:50:04 +0000
X-ASF-Spam-Status: No, hits=-3.5 required=5.0
	tests=RCVD_IN_DNSWL_HI,RCVD_IN_RP_SAFE,URIBL_BLACK
X-Spam-Check-By: apache.org
Received-SPF: unknown (athena.apache.org: error in processing during lookup of jrottinghuis@ebay.com)
Received: from [216.33.244.6] (HELO rhv-mipot-001.corp.ebay.com) (216.33.244.6)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 22:50:00 +0000
DomainKey-Signature: s=corp; d=ebay.com; c=nofws; q=dns;
  h=X-EBay-Corp:X-IronPort-AV:Received:Received:From:To:Date:
   Subject:Thread-Topic:Thread-Index:Message-ID:References:
   In-Reply-To:Accept-Language:Content-Language:
   X-MS-Has-Attach:X-MS-TNEF-Correlator:acceptlanguage:
   x-ems-proccessed:x-ems-stamp:Content-Type:
   Content-Transfer-Encoding:MIME-Version:X-CFilter;
  b=yWEUEClNgHc7+5OFhJfQiEdNoVg8j+Vzcckw4rEAeHzCDroqtbnHTgvx
   ik+qoboWcuqK2pNFLiBrGXTzBhUtNifNRPedE0g60tZwNj5bMZXrz1cSC
   F/9xjkd7uWzhvPr;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=ebay.com; i=jrottinghuis@ebay.com; q=dns/txt; s=corp;
  t=1308437400; x=1339973400;
  h=from:to:date:subject:message-id:references:in-reply-to:
   content-transfer-encoding:mime-version;
  bh=bUX8seb9VTEX9w7t4SMppONAq3fPJV52AwFYqRSHiEg=;
  b=l5SftAUJTvoLqHECMpyViKfZwBovZF0QEcFyd7A55Wob7mPcdTPfMpuY
   bbcTND7LxPObjN/DJHwCOFO7SV8J6ZPtLGfnu8NA0gu/MFQBdLcsTodoo
   9qb6OsmjeQUN1/y;
X-EBay-Corp: Yes
X-IronPort-AV: E=Sophos;i="4.65,387,1304319600"; 
   d="scan'208";a="21455353"
Received: from rhv-vtenf-002.corp.ebay.com (HELO RHV-MEXHT-003.corp.ebay.com) ([10.112.113.53])
  by rhv-mipot-001.corp.ebay.com with ESMTP; 18 Jun 2011 15:49:38 -0700
Received: from RHV-MEXMS-002.corp.ebay.com ([10.245.17.115]) by
 RHV-MEXHT-003.corp.ebay.com ([10.245.24.102]) with mapi; Sat, 18 Jun 2011
 15:49:36 -0700
From: "Rottinghuis, Joep" <jrottinghuis@ebay.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Sat, 18 Jun 2011 15:47:16 -0700
Subject: RE: [DISCUSSION] Thinking about 20.204 and beyond 
Thread-Topic: [DISCUSSION] Thinking about 20.204 and beyond 
Thread-Index: AcwthDfXw9WrxdpeR4Oj0J1JWIclygAhXPiQ
Message-ID: <F4FCE59AE46B2243922F224426BC869B01EC996739@RHV-MEXMS-002.corp.ebay.com>
References: <4E18CE25-6863-4318-8A95-17896B0B87D1@yahoo-inc.com>
In-Reply-To: <4E18CE25-6863-4318-8A95-17896B0B87D1@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
x-ems-proccessed: 10SqDH0iR7ekR7SRpKqm5A==
x-ems-stamp: 9tpq8CilICIOIdZtjvkQqA==
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-CFilter: Scanned

Thanks for clarifying your position and being open with your plans going fo=
rward.

Joep
________________________________________
From: Eric Baldeschwieler [eric14@yahoo-inc.com]
Sent: Friday, June 17, 2011 11:50 PM
To: general@hadoop.apache.org
Subject: [DISCUSSION] Thinking about 20.204 and beyond

Hi Folks,

Along with starting a new release off the mainline (see previous mail), the=
 Yahoo! team plans to continue producing sustaining releases off the Hadoop=
 with security branch, such as 0.20.203 .  I'm writing this email to outlin=
e our plans, explain Yahoo's motivation for supporting this work and reques=
t feedback and hopefully your endorsement.  This initiative stems from Yaho=
o's commitment to do its hadoop work in Apache and discontinue the Yahoo Di=
stribution of Hadoop (http://yhoo.it/i9Ww8W).

We hope to produce a new 0.20.204 release in Apache in the next few weeks. =
 Owen O'Malley is planning to act as release master for this release.  This=
 will be based on work in the hadoop-with-security branch, just as 0.20.203=
 but will include bugfixes and enhancements beyond those in 0.20.203.  This=
 is one in a series of releases we hope to do in the next 6-9 months as had=
oop 0.23 (or whatever the community chooses to call it) goes through the va=
rious stages of stability testing and burn-in.

CONTENTS OF THE RELEASE:

Some highlights:

- RPM & .deb packaging to ease deployment (back ported from trunk)
 - I am excited to see hadoop released with .deb & RPM packaging from Apach=
e for the first time.
 - This will greatly ease deployment
- Disk fail in place (merged with trunk, except for some MR changes conflic=
t with MR-279, these will be reimplemented in MR-279)
 - This change has been motivated by operational problems we observed with =
our new 12 disk machines.
 - This work should greatly improve Hadoop availability by keeping nodes wo=
rking when one of their disks fails

- Lots of of additional fixes (I've included the change log below)

WHY THIS PROCESS:

Producing a stable release of Hadoop is a long, hard and expensive process.=
  Historically Y! has produced all such releases.  Other releases of Hadoop=
 have either not been stable (Hadoop 0.19 and Hadoop 0.21) or have been bas=
ed on a stable Apache release driven by Yahoo (CDH and Facebook).  Once we'=
ve paid the price of making a stable release, it makes a lot of sense to ac=
cept safe improvements as well as bug fixes.  Doing so allows one to get cu=
stomer impacting improvements into production in days, rather than years, w=
hich is what would happen if one waited for changes to come in the next sta=
ble release off the Hadoop mainline.  Given that it takes many months to st=
abilize trunk, there is no way to get new easy fixes into users hands quick=
ly via a new mainline release.

For the last few years Yahoo has done sustaining engineering in open source=
 via Github.  These patches have been contributed to Apache Hadoop mainline=
 and backported to the sustaining branch on github (for yahoo 0.20 for exam=
ple).  We've then cut Yahoo releases from Github.  Cloudera and Facebook ha=
ve also taken these patches from Github and incorporated these improvements=
 into their releases, so the community has benefitted from this process for=
 years.  What we are planning to do now is simply move this process into Ap=
ache, so that Apache releases themselves are timely and relevant, not alway=
s a year or two behind what users need.

How do I propose making these decisions?  Deciding what is a safe patch is =
a judgement call.  Apache process suggests that the release manager makes t=
hese calls (http://bit.ly/mJcBjc).  For releases Y! champions, such as 0.20=
 (arun & owen), we are ready to do the sustaining engineering, make these c=
alls and stand our reputation behind the quality of the result.  Other rele=
ase masters are championing other Apache Hadoop releases currently (Nigel a=
nd Tom) and I think they should be free to do the same.  For hadoop 20.204,=
 I propose pushing what is currently in the hadoop-with-security branch.  P=
art of the reason for this thread is to socialize this process, so that com=
munity members can champion stable patches for inclusion in 20.205.  In the=
 future I propose that a branch's release master request suggestions for fu=
ture releases on this list, but is free to use their judgement on what is a=
ccepted (pretty much what nigel is doing on 0.22 today).

----

Conclusion:

The vote on 0.20.203 was acrimonious, but I believe that 0.20.203 was a use=
ful step forward for Apache Hadoop.  0.20.204 will again be the best stable=
 release of Apache Hadoop ever.  I hope folks can support the effort.  With=
 your contribution 0.20.205 can be even better, fixing issues that plague y=
our Hadoop clusters.  This email is part of a wider effort from the Yahoo t=
eam to co-plan our work with the community.

Thanks,

Eric14
---
eric14 a.k.a. Eric Baldeschwieler
VP Hadoop Software Development @Yahoo!


=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

>From CHANGES.txt:

Release 0.20.204.0 - unreleased

 NEW FEATURES

   HADOOP-6255. Create RPM and Debian packages for common. Changes deployme=
nt
   layout to be consistent across the binary tgz, rpm, and deb. Adds setup
   scripts for easy one node cluster configuration and user creation.
   (Eric Yang via omalley)

 BUG FIXES

   MAPREDUCE-2495. exit() the TaskTracker when the distributed cache cleanu=
p
   thread dies. (Robert Joseph Evans via cdouglas)

   HDFS-1878. TestHDFSServerPorts unit test failure - race condition
   in FSNamesystem.close() causes NullPointerException without serious
   consequence. (mattf)

   MAPREDUCE-2452. Moves the cancellation of delegation tokens to a separat=
e
   thread. (ddas)

   MAPREDUCE-2555. Avoid sprious logging from completedtasks. (Thomas Grave=
s
   via cdouglas)

   MAPREDUCE-2451. Log the details from health check script at the
   JobTracker. (Thomas Graves via cdouglas)

   MAPREDUCE-2535. Fix NPE in JobClient caused by retirement. (Robert Josep=
h
   Evans via cdouglas)

   MAPREDUCE-2456. Log the reduce taskID and associated TaskTrackers with
   failed fetch notifications in the JobTracker log.
   (Jeffrey Naisbitt via cdouglas)

   HDFS-2044. TestQueueProcessingStatistics failing automatic test due to
   timing issues. (mattf)

   HADOOP-7248. Update eclipse target to generate .classpath from ivy confi=
g.
   (Thomas Graves and Tom White via cdouglas)

   MAPREDUCE-2558. Add queue-level metrics 0.20-security branch - test fix
   (jeffrey nasbit via mahadev)

   HADOOP-7364. TestMiniMRDFSCaching fails if test.build.dir is set to
   something other than build/test. (Thomas Graves via mahadev)

   HADOOP-7277. Add generation of run configurations to eclipse target.
   (Jeffrey Naisbitt and Philip Zeyliger via cdouglas)

   HADOOP-7373. Fix {start,stop}-{dfs,mapred} and hadoop-daemons.sh from
   trying to use the wrong bin directory. (omalley)

   HADOOP-7274. Fix typos in IOUtils. (Jonathan Eagles via cdouglas)

   HADOOP-7369. Fix permissions in tarball for sbin/* and libexec/* (omalle=
y)

   MAPREDUCE-2479. Move distributed cache cleanup to a background task,
   backporting MAPREDUCE-1568. (Robert Joseph Evans via cdouglas)

   HADOOP-7356. Fix bin/hadoop scripts (eyang via omalley)

   HADOOP-7272. Remove unnecessary security related info logs. (suresh)

   MAPREDUCE-2514. Fix typo in TaskTracker ReinitTrackerAction log message.
   (Jonathan Eagles via cdouglas)

   HDFS-1906. Remove logging exception stack trace in client logs when one =
of
   the datanode targets to read from is not reachable. (suresh)

   MAPREDUCE-2490. Add logging to graylist and blacklist activity to aid
   diagnosis of related issues. (Jonathan Eagles via cdouglas)

   MAPREDUCE-2447. Fix Child.java to set Task.jvmContext sooner to avoid
   corner cases in error handling. (Siddharth Seth via acmurthy)

   MAPREDUCE-2429. Validate JVM in TaskUmbilicalProtocol. (Siddharth Seth v=
ia
   acmurthy)

   MAPREDUCE-2418. Show job errors in JobHistory page. (Siddharth Seth via
   acmurthy)

   HDFS-1592. At Startup, Valid volumes required in FSDataset doesn't
   handle consistently with volumes tolerated. (Bharath Mundlapudi)

   HDFS-1598. Directory listing on hftp:// does not show
   .*.crc files.  (szetszwo)

   HDFS-1750. ListPathsServlet should not use HdfsFileStatus.getLocalName()
   to get file name since it may return an empty string.  (szetszwo)

   HDFS-1758. Make Web UI JSP pages thread safe. (Tanping via suresh)

   HDFS-1773. Do not show decommissioned datanodes, which are not in both
   include and exclude lists, on web and JMX interfaces.
   (Tanping Wang via szetszwo)

   MAPREDUCE-2409. Distinguish distributed cache artifacts localized as
   files, archives. (Siddharth Seth via cdouglas)

   MAPREDUCE-118. Fix Job.getJobID() to get the new ID as soon as it's
   assigned. (Amareshwari Sriramadasu and Dick King via cdouglas)

   MAPREDUCE-2411. Force an exception when the queue has an invalid name or
   its ACLs are misconfigured. (Dick King via cdouglas)

   HDFS-1258. Clearing namespace quota on "/" corrupts fs image.
   (Aaron T. Myers via szetszwo)

   HDFS-1189. Quota counts missed between clear quota and set quota.
   (John George via szetszwo)

   HDFS-1692. In secure mode, Datanode process doesn't exit when disks
   fail. (bharathm via boryas)

   MAPREDUCE-2420. JobTracker should be able to renew delegation token
   over HTTP (boryas)

   MAPREDUCE-2443. Fix TaskAspect for TaskUmbilicalProtocol.ping(..).
   (Siddharth Seth via szetszwo)

   HDFS-1842. Handle editlog opcode conflict with 0.20.203 during upgrade,
   by throwing an error to indicate the editlog needs to be empty.
   (suresh)

   HDFS-1377. Quota bug for partial blocks allows quotas to be violated. (e=
li)

   HDFS-2057. Wait time to terminate the threads causes unit tests to
   take longer time. (Bharath Mundlapudi via suresh)

 IMPROVEMENTS

   HADOOP-7144. Expose JMX metrics via JSON servlet. (Robert Joseph Evans v=
ia
   cdouglas)

   MAPREDUCE-2524. Port reduce failure reporting semantics from trunk, to
   fail faulty maps more aggressively. (Thomas Graves via cdouglas)
   MAPREDUCE-2529. Add support for regex-based shuffle metric counting
   exceptions. (Thomas Graves via cdouglas)

   HADOOP-7398. Suppress warnings about use of HADOOP_HOME. (omalley)

   MAPREDUCE-2415. Distribute the user task logs on to multiple disks.
   (Bharath Mundlapudi via omalley)

   MAPREDUCE-2413. TaskTracker should handle disk failures by reinitializin=
g
   itself. (Ravi Gummadi and Jagane Sundar via omalley)

   HDFS-1541. Not marking datanodes dead when namenode in safemode.
   (hairong)

   HDFS-1767. Namenode ignores non-initial block report from datanodes
   when in safemode during startup. (Matt Foley via suresh)

   MAPREDUCE-1251. c++ utils doesn't compile. (Eli Collins via shv)

  HADOOP-7330. Fix MetricsSourceAdapter to use the value instead of the
   object. (Luke Lu via omalley)=

From general-return-3868-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 22:56:31 2011
Return-Path: <general-return-3868-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8831D6DBD
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat, 18 Jun 2011 22:56:31 +0000 (UTC)
Received: (qmail 96795 invoked by uid 500); 18 Jun 2011 22:56:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96740 invoked by uid 500); 18 Jun 2011 22:56:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96732 invoked by uid 99); 18 Jun 2011 22:56:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 22:56:29 +0000
X-ASF-Spam-Status: No, hits=4.6 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL,URIBL_BLACK
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 69.147.107.20 is neither permitted nor denied by domain of eric14@yahoo-inc.com)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 22:56:22 +0000
Received: from SP2-EX07CAS04.ds.corp.yahoo.com (sp2-ex07cas04.corp.sp2.yahoo.com [98.137.59.5])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5IMtp9D014518;
	Sat, 18 Jun 2011 15:55:51 -0700 (PDT)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by
 SP2-EX07CAS04.ds.corp.yahoo.com ([98.137.59.5]) with mapi; Sat, 18 Jun 2011
 15:55:51 -0700
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Sat, 18 Jun 2011 15:55:47 -0700
Subject: Re: [DISCUSSION] Thinking about 20.204 and beyond 
Thread-Topic: [DISCUSSION] Thinking about 20.204 and beyond 
Thread-Index: AcwuCt5+1RQYs1wNR/mtFYMej1Y8iQ==
Message-ID: <3F4E1CA0-6A80-4AF1-AAC7-0A43996E4EA5@yahoo-inc.com>
References: <4E18CE25-6863-4318-8A95-17896B0B87D1@yahoo-inc.com>
 <0C4895F3-95E8-45ED-B2B0-C3BF4B5436B1@gbiv.com>
In-Reply-To: <0C4895F3-95E8-45ED-B2B0-C3BF4B5436B1@gbiv.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Roy,

Thanks for the clarification. I will endeavor to keep marketing to a minimu=
m. I always try to keep my actions as a manager and project contributor con=
sistent with our community norms.  When one is speaking as the manager of a=
 team of contributors pronouns get complicated.=20

I've gotten lots of feedback that folks would like a better understanding o=
f what I plan to ask team members to invest in. And lots of questions about=
 why we don't do various things others would like to see done.  My goal is =
simply to be transparent. That should allow all of us to make better decisi=
ons as a community.=20

Given the number of questions I have gotten on why "yahoo" chooses to do su=
staining I think the background is needed for an informed debate.=20

I expect yahoo team members will continue to make branches and call release=
 votes. I hope that sharing my thinking, intentions and motivations up fron=
t reduces conflict.=20

---
E14 - typing on glass

On Jun 18, 2011, at 1:23 PM, "Roy T. Fielding" <fielding@gbiv.com> wrote:

> On Jun 17, 2011, at 11:50 PM, Eric Baldeschwieler wrote:
>=20
>> Along with starting a new release off the mainline (see previous mail), =
the Yahoo! team plans to continue producing sustaining releases off the Had=
oop with security branch, such as 0.20.203 .  I'm writing this email to out=
line our plans, explain Yahoo's motivation for supporting this work and req=
uest feedback and hopefully your endorsement.  This initiative stems from Y=
ahoo's commitment to do its hadoop work in Apache and discontinue the Yahoo=
 Distribution of Hadoop (http://yhoo.it/i9Ww8W).
>=20
> Eric,
>=20
> Please make an effort to understand that we act at Apache as individuals.
> That includes you and your team.  Yes, everyone greatly appreciates the
> fact that Yahoo! pays your salaries.  Nevertheless, here you are called
> an individual named Eric.
>=20
> Companies do not have motivations.  You do.  There is nothing wrong with
> each and every individual here being motivated, at least partly, by what
> their boss(es) consider the right path forward.  But you don't have to
> sell it as something Yahoo! is doing, and you don't need to send marketin=
g
> messaging to the development lists at Apache.  Just be yourself and
> represent yourself.
>=20
> Anyone on the PMC can be an RM.  A package can be rolled at any time, and
> a release vote on that package can be called at any time.  While it makes
> sense for the individual planning to RM a release to announce their plans=
,
> doing so does not in any way form the direction the product is headed,
> nor does it exclude anyone else from being RM of their own release proces=
s.
>=20
> There is absolutely no reason that trunk cannot be packaged for release
> tomorrow as 0.23.  There may be many reasons why it won't pass a release
> vote, but we probably aren't going to find them until somebody tries.
>=20
> ....Roy

From general-return-3869-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 19 00:56:37 2011
Return-Path: <general-return-3869-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6821B6082
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sun, 19 Jun 2011 00:56:37 +0000 (UTC)
Received: (qmail 33644 invoked by uid 500); 19 Jun 2011 00:56:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33577 invoked by uid 500); 19 Jun 2011 00:56:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 88257 invoked by uid 99); 18 Jun 2011 14:45:41 -0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_HELO_PASS,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Message-ID: <4DFCB9F1.2010603@shanecurcuru.org>
Date: Sat, 18 Jun 2011 10:45:05 -0400
From: Shane Curcuru <asf@shanecurcuru.org>
User-Agent: Thunderbird 2.0.0.24 (Windows/20100228)
MIME-Version: 1.0
To: trademarks@apache.org
CC: general@hadoop.apache.org
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
In-Reply-To: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Provags-ID: V02:K0:8idlbUD7/+41kjzJS14KUVDJH0PkIwIYioCNcgbJtM6
 CyzuWn4BwI3J4FaFMEfu7eEidImJB3sfWIz3fHQaseQFdQ47tj
 Df0hEFE5aEOLwpGqTZYmN1T0jnj/uu8KYR53rfh7fKc6DXq4tu
 krW2E08aY1y9uVBYATHm9mLKi4MtcDXIIBiXjxub+Zv/v13wWz
 eYd1VSt0VFNklUhy2PiOGHiQl1d6zupz/FTyRW2Sc4=
X-Virus-Checked: Checked by ClamAV on apache.org

One clarification: I've only had time to review the wiki document for 
some terminology updates, and not for the overall content yet.  So from 
the trademarks@ point of view, more review is needed before we work on 
making this official.

 From the significant amount of discussion in this vote thread, I think 
it might be good to have the Hadoop PMC and trademarks@ work on getting 
a more organized consensus first, before voting on an updated proposed 
Hadoop policy.

- Shane

Owen O'Malley wrote:
> All,
>    Steve Loughran has done some great work on defining what can be 
> called Hadoop at http://wiki.apache.org/hadoop/Defining%20Hadoop 
> <http://wiki.apache.org/hadoop/Defining Hadoop>. After some cleanup from 
> Noirin and Shane, I think we've got a really good base. I'd like a vote 
> to approve the content (at the current revision 12) and put the content 
> on our web site.
> 
> Clearly, I'm +1.
> 
> -- Owen


From general-return-3870-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 19 00:56:39 2011
Return-Path: <general-return-3870-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9998E60A7
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sun, 19 Jun 2011 00:56:39 +0000 (UTC)
Received: (qmail 34063 invoked by uid 500); 19 Jun 2011 00:56:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33943 invoked by uid 500); 19 Jun 2011 00:56:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 82667 invoked by uid 99); 18 Jun 2011 14:30:49 -0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_HELO_PASS,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Message-ID: <4DFCB676.2060308@shanecurcuru.org>
Date: Sat, 18 Jun 2011 10:30:14 -0400
From: Shane Curcuru <asf@shanecurcuru.org>
User-Agent: Thunderbird 2.0.0.24 (Windows/20100228)
MIME-Version: 1.0
To: trademarks@apache.org, 
 "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: Trademarks and Derivative Works
References: <04f001cc2c4f$1815c460$48414d20$@com> <B71A7B6B-02EB-4049-A8DD-2E8321F788EE@apache.org> <3798688F8784154192FDA3388F4C208101086D60@hq-ex-mb03.ad.navteq.com>
In-Reply-To: <3798688F8784154192FDA3388F4C208101086D60@hq-ex-mb03.ad.navteq.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Provags-ID: V02:K0:YMDhRvt2MPvgKCYfaqceK3QWqsUFr6HftKzfRMr6q7I
 8hK9Uz/3eoW9L+KDYsZZOESGi3BXZ84gIRpLWgK69ls31VmAci
 S33oLYvQlfgKzX/v0O7s7tc4DB7PA8Kws4KVcARPuEhWIe1D8r
 Tz5N3SQ8b4zmcWokrMYNA/EwM+X9ribPKFjQ1AdctZol5cfRUx
 OZFDJIY4KsO6qYtu9NaxaH7vOiOj96KrPQ8q3lWt74=

(Apologies; I'm on the way to a family memorial service, so I don't have 
time for a complete answer here, although this is an important issue for 
trademarks@)

I certainly appreciate the work that the Hadoop PMC has been doing in 
terms of better defining the kinds of use cases for Hadoop derivatives 
or related kinds of software!  The pre-existing "Powered By" metaphor 
from httpd is a great model, and one that allows for strong independent 
branding by third parties while still allowing for use of Apache marks 
within third party names, with a clear separation.

However the phrase "Powered by" does not fit many uses of software like 
Hadoop, so we need both some alternate naming styles, as well as some 
better definitions of  models of third party uses; derivatives with 
additional code, integrations with other products, uses of Apache 
software as services, etc.  I've always wanted to create a generic list 
of other phrases to use instead of "Powered By" that we would list as a 
policy of pre-approved uses for these kind of cases.

I'd like to see more discussion from the PMC, because Hadoop in 
particular is a key technology that's being used in a wide variety of 
ways - really defining a new computing model.  However I think we really 
need to take this input and then have trademarks@ work on an Apache-wide 
policy for approved third party product and services naming styles for 
all Apache products.  Obviously, some of the specific types of uses of 
Hadoop may not be applicable to other Apache products, but in general I 
think working on a larger set of guidelines that apply would be the best 
way to approach this situation for all Apache projects.

Overall, trademarks@ and VP, Brand Management set branding policy for 
all Apache projects.  But we definitely need the combined input of both 
folks on Hadoop - with so many new and innovative uses of our software - 
plus the input of some of our longer-running projects like httpd and 
Tomcat, with experience in *nix-style packaging and the like, to come up 
with the best policy for these kind of third party uses of our product 
marks.

- Shane Curcuru
   VP, Brand Management, The Apache Software Foundation

Segel, Mike wrote:
> Owen, From your response below you say the following: "We are trying
> to assert that only the Apache releases can be called Hadoop. That
> seems to be the best way to help the project ensure compatibility and
> prevent user confusion." and " I want Hadoop to be used in as many
> products as possible. Having a FooCo product that is called "FooCo
> HugeInsights powered by Hadoop" is absolutely great. The question is
> just whether they can call something Hadoop if it isn't an Apache
> release.
> 
> -- Owen"
> 
> Owen, Unfortunately what you're saying is that you would only approve
> of companies that build their products on top of Apache's release and
> doesn't modify the Apache release. To give you an example... if  Acme
> Risk Management Company sold a product using Hadoop to do risk
> analysis on a bank's portfolio, they can only say "powered by Hadoop"
> if they build their application on top of Apache's release. But the
> minute they build their solution on top of anyone else, they would
> lose that right? So using Cloudera's release, which contains things
> outside of the official Apache release would disallow them?  Or if
> they make their own modifications to the underlying release which
> isn't part of the official release, they could no longer make that
> claim?
> 
> This interpretation of  "powered by Hadoop" would unfortunately lead
> to as many problems as it attempts to solve. First, many choose
> Cloudera's release because they sell commercial support. So in
> choosing Cloudera's release, they would lose the ability to say
> "powered by Hadoop". This diminishes the branding message.
> 
> The Apache License allows for broad reuse and relicensing as long as
> the company complies with Apache's T's & C's.  Limiting the ability
> to say "powered by Hadoop" means that they will say that their
> solution uses a commercially supported  'derivative of Hadoop'. In
> terms of legalese, good luck in trying to get them on a misuse of
> your trademark.  Cloudera, EMC, MapRTech, Datastax all offer
> derivatives of Hadoop. (I'm not forgetting about Yahoo!, but are they
> releasing their own version as well?) The term Hadoop is used as a
> reference to Apache's Hadoop release.
> 
> I hope that you start to see the dangers on taking a narrow approach
> in how you define Hadoop.
> 
> Just IMHO.
> 
> -Mike
> 
> -----Original Message----- From: Owen O'Malley
> [mailto:omalley@apache.org] Sent: Thursday, June 16, 2011 1:32 PM To:
> general@hadoop.apache.org Cc: trademarks@apache.org Subject: Re:
> Trademarks and Derivative Works
> 
> 
> On Jun 16, 2011, at 10:59 AM, Lawrence Rosen wrote:
> 
>> Under default trademark law, those others can distribute HADOOP and
>>  APACHE HADOOP only if it is a redistribution of *our* HADOOP or
>> APACHE HADOOP software. (That's why you can buy Jello Brand gelatin
>> at Safeway.) Those trademarks are our names for our software. ASF
>> is the source and origin of those software goods. Nobody else can
>> apply those trademarks to their own software.
> 
> The problem is that a rapidly growing set of companies are
> distributing products that have never been released by Apache and
> calling them Hadoop. The rules from HTTPD, as I understand them, are
> that they allow artifacts to be called HTTPD that are releases plus
> patches that have been committed. With HTTPD that has a formal
> specification and a very large compatibility test suite, that works.
> For Hadoop without a formal specification or test suite, we simply
> can't handle companies calling things Hadoop that are thousands of
> patches away from our releases. We are trying to assert that only the
> Apache releases can be called Hadoop. That seems to be the best way
> to help the project ensure compatibility and prevent user confusion.
> 
>> It will be to our advantage to have HADOOP and APACHE HADOOP
>> software better known and widely used throughout the world. For
>> that purpose, we should be defining the rules we want to
>> *encourage* third parties to follow, not arguing about derivative
>> work analysis or voting on whether or not something is a trademark.
>> 
> 
> I want Hadoop to be used in as many products as possible. Having a
> FooCo product that is called "FooCo HugeInsights powered by Hadoop"
> is absolutely great. The question is just whether they can call
> something Hadoop if it isn't an Apache release.
> 
> -- Owen
> 
> 
> 
> The information contained in this communication may be CONFIDENTIAL
> and is intended only for the use of the recipient(s) named above.  If
> you are not the intended recipient, you are hereby notified that any
> dissemination, distribution, or copying of this communication, or any
> of its contents, is strictly prohibited.  If you have received this
> communication in error, please notify the sender and delete/destroy
> the original message and any copy of it from your computer or paper
> files.


From general-return-3871-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 05:56:17 2011
Return-Path: <general-return-3871-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 225EC6975
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 05:56:17 +0000 (UTC)
Received: (qmail 49958 invoked by uid 500); 20 Jun 2011 05:56:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49620 invoked by uid 500); 20 Jun 2011 05:56:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49603 invoked by uid 99); 20 Jun 2011 05:56:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 05:56:08 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.30.80] (HELO qmta08.emeryville.ca.mail.comcast.net) (76.96.30.80)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 05:56:01 +0000
Received: from omta18.emeryville.ca.mail.comcast.net ([76.96.30.74])
	by qmta08.emeryville.ca.mail.comcast.net with comcast
	id y5e31g0011bwxycA85vfTj; Mon, 20 Jun 2011 05:55:39 +0000
Received: from fs ([24.4.185.157])
	by omta18.emeryville.ca.mail.comcast.net with comcast
	id y5wA1g0043QAh8g8e5wAg2; Mon, 20 Jun 2011 05:56:10 +0000
Received: from mail.boudnik.org (localhost [127.0.0.1])
	by fs (8.14.3/8.14.3/Debian-5+lenny1) with ESMTP id p5K5tdKL018230
	for <general@hadoop.apache.org>; Sun, 19 Jun 2011 22:55:39 -0700
Received: (from cos@localhost)
	by mail.boudnik.org (8.14.3/8.14.3/Submit) id p5K5td7a018229
	for general@hadoop.apache.org; Sun, 19 Jun 2011 22:55:39 -0700
X-Authentication-Warning: mail.boudnik.org: cos set sender to cos@apache.org using -f
Date: Sun, 19 Jun 2011 22:55:39 -0700
From: Konstantin Boudnik <cos@apache.org>
To: general@hadoop.apache.org
Subject: Re: [VOTE] Powered by Logo
Message-ID: <20110620055539.GA18201@linspire.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
X-Organization: It's something of 'Cos
User-Agent: Mutt/1.5.18 (2008-05-17)

How the long the vote is suppose to be running? Usual 72 hours? Or longer?

The up-to date vote results are here http://tinyurl.com/3j6629h

Cos

On Tue, Jun 14, 2011 at 08:19PM, Owen O'Malley wrote:
> All,
>    We've had a wide range of entries for a powered by logo. I've put them
>    all on a page, here:
> 
> http://people.apache.org/~omalley/hadoop-powered-by/
> 
> Since there are a lot of contenders and we only want a single round of
> voting, let's use single transferable vote ( STV
> http://en.wikipedia.org/wiki/Single_transferable_vote). The important thing
> is to pick the images *IN ORDER* that you would like them.
> 
> My vote (in order of course): 4, 1, 2, 3, 5, 6.
> 
> In other words, I want option 4 most and option 6 least. With STV, you don't
> need to worry about voting for an unpopular choice since your vote will
> automatically roll over to your next choice.
> 
> -- Owen
> 
> 

From general-return-3872-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 06:22:44 2011
Return-Path: <general-return-3872-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 403FA6C14
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 06:22:44 +0000 (UTC)
Received: (qmail 76220 invoked by uid 500); 20 Jun 2011 06:22:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75915 invoked by uid 500); 20 Jun 2011 06:22:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 67112 invoked by uid 99); 20 Jun 2011 06:14:36 -0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of scgkiran@gmail.com designates 209.85.210.48 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=MCyEdkAVfOxWZIzm6Z0ZmlS3355GLgh02W4VZrPXeTc=;
        b=aZ27sdCM+rVDXKhNDn/BDypUPnjAAu2bKAKjePthpXivW5brFyhHfz6vkkwsyz3tiL
         ytkTj2ZhakKv3ODUk9DWPdvAmLrKZYrGZtLgCbfQTC/F6nStmcPe968H7Z3b0IKasytY
         8liysXO4jKHQdRgx1X1ifcFaJtNbeIoOrLQMc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=DMZ/W7pDz4PwDC2HNrq0W7tXc99/nf0CABxn+QNXLxySKM7zytoviZlCMRaUED3kU6
         Z+uIKM7bZzwUJ+V+Y/kD2Q0GfreV6pJPtOf3XYijNb2QXVP/q9kIx+MoM1Tc4CAlr49H
         d2LYREK+rNeXFqV9d0TR/GGzJ3DUF4PwlwitA=
MIME-Version: 1.0
Date: Mon, 20 Jun 2011 11:44:06 +0530
Message-ID: <BANLkTikyG8FZuvrscn455_J5gLffVhKOAA@mail.gmail.com>
Subject: Powered by Hadoop logo:
From: Chandra Guru Kiran Babu Sanapala <scgkiran@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec520f22b8d39d904a61ea44e
X-Virus-Checked: Checked by ClamAV on apache.org

--bcaec520f22b8d39d904a61ea44e
Content-Type: text/plain; charset=ISO-8859-1

4, 2, 6, 1, 3, 5

--bcaec520f22b8d39d904a61ea44e--

From general-return-3873-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 06:22:55 2011
Return-Path: <general-return-3873-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CA8416C3A
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 06:22:55 +0000 (UTC)
Received: (qmail 77735 invoked by uid 500); 20 Jun 2011 06:22:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76437 invoked by uid 500); 20 Jun 2011 06:22:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 69733 invoked by uid 99); 20 Jun 2011 06:17:34 -0000
X-ASF-Spam-Status: No, hits=3.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ramesh.416@gmail.com designates 209.85.218.48 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=b59+w15I639tVLUerQPWmglpcuWltVmxCPNHAPN8DuA=;
        b=A+QzOJeZxeU04GBL81IZMLF1G/IjFyrk4/5Eu6scz1AF//IVTALXHujtSwhrXGGLMC
         gU9+h3Gqe32Rvy8+pm1Mf5OGlxCWnEKjsCXPdat7CIIiSeUDekVURYJOr958xKprUjZE
         ZbHyihhh7AtkEkq0IHSvJxT+fCGbDoiCKJg8k=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=lJQ4+cvT3sYvomFbOSwvF/Ws+b3WBg9PvWEnS6fLLz5QQvjCxvrIqI1CHYBX1lx7o0
         SI1F0yMbN4fdVBxTX3nwFJKFCYZ8yDce7rEcdIxY8yekMq5PCD6fXo+acWtF9qRnS/Jm
         G2aUXInQL0EUM8jdsp0Cdq4yjAJpvZOTM3MCQ=
MIME-Version: 1.0
Date: Mon, 20 Jun 2011 11:47:06 +0530
Message-ID: <BANLkTimJTegZyozz3EobStZMJXbKcLZD1Q@mail.gmail.com>
Subject: powered by hadoop logo
From: Ramesh Muppidi <ramesh.416@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636ef0d6d3eef6204a61eafa7
X-Virus-Checked: Checked by ClamAV on apache.org

--001636ef0d6d3eef6204a61eafa7
Content-Type: text/plain; charset=ISO-8859-1

4,5,6,1,3,2

--001636ef0d6d3eef6204a61eafa7--

From general-return-3874-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 06:23:06 2011
Return-Path: <general-return-3874-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 46C626C6F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 06:23:06 +0000 (UTC)
Received: (qmail 79289 invoked by uid 500); 20 Jun 2011 06:23:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78637 invoked by uid 500); 20 Jun 2011 06:22:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 68639 invoked by uid 99); 20 Jun 2011 06:16:29 -0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of marella@gmail.com designates 209.85.212.48 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=c+Uu8Nzhvvt0lHTlRxGeF0pBu2RaWzvXJRr6wGDWtic=;
        b=TGgxydwRdwWEjXOJ9EV6iVkSHv+dp7wcItRkrcXxGJM8XmOCppRSklicuxrF2I6Luf
         m6d0YIDjQ3H5YiRxO+xyGNqO2p7+LJVneEMn6Xbj0uGdBx9AkWWVLpk6qsJpZTNZmEgb
         Jmd4lQnLuTHaHmDSmjvO2zFo1HXHP7bw41X5Y=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=E8JBhp9S0pk4wcxjk/flQpjekhDejsanACMjW6rNG5PqHQyYrltEsmXtyzRPO8q8TS
         koqyrOQYiZem8//u1Ui46jjR83O+c0UOkiYzBFiE/AxP1crZMs284nv8pQmpOxnTL2rA
         RlBX17cgBMByVNq6e32V4Ut0xzohqKo1gZwII=
MIME-Version: 1.0
Date: Mon, 20 Jun 2011 11:46:02 +0530
Message-ID: <BANLkTikepCCEYwAu+-KChZbsUU=gRgLhmQ@mail.gmail.com>
Subject: powered by hadoop logo:
From: Santosh Marella <marella@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64f466e7154f204a61eabed

--0016e64f466e7154f204a61eabed
Content-Type: text/plain; charset=ISO-8859-1

 4, 2, 6, 1, 3, 5

--0016e64f466e7154f204a61eabed--

From general-return-3875-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 06:26:11 2011
Return-Path: <general-return-3875-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 185136D35
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 06:26:11 +0000 (UTC)
Received: (qmail 84988 invoked by uid 500); 20 Jun 2011 06:26:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84720 invoked by uid 500); 20 Jun 2011 06:26:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 71741 invoked by uid 99); 20 Jun 2011 06:20:21 -0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ravindrap@gmail.com designates 209.85.214.48 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=CqOTEQnXkKR7qMLv384YLt8gRpEqxnlg3dbjp+SXYv4=;
        b=taTqOzVHuzZwgR1WUm9R6ilm4yDigLZp+hmpz4NVW+M5eAIv6xIDrermVMpEAKr2tl
         0bwm+Ao/ZgxzHi5I8g7PRzzpSGb34xpTtm8T7adBnvUjYBPgLqIocCc8o0Mwi1mrn0p+
         RYrTiKYuiXF0jprWVljCoAfitC+/xyizDs30o=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=V6YzE94U7qA2c90rHwFKuE4s7NhaIuq0UWz8sSMd/7zJixZlMhnJ7oOwRzu9NsPBGZ
         LvugHAhDmgVZZ/pJ3T8sR6l+o82XYM8czy4jWvD6TVwFmo+VlOWWoCJCjs88Gsjvehkh
         nGIzM1ssahQFt40Zyc8WfeDTicFTolqhgfxYs=
MIME-Version: 1.0
Date: Mon, 20 Jun 2011 11:49:53 +0530
Message-ID: <BANLkTi=7bpEL_fjVyMvMvtW_D7ZrJvFh+Q@mail.gmail.com>
Subject: powered by hadoop logo
From: Pindikura Ravindra <ravindrap@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00032555b34e31ba5304a61eb99e
X-Virus-Checked: Checked by ClamAV on apache.org

--00032555b34e31ba5304a61eb99e
Content-Type: text/plain; charset=ISO-8859-1

4, 2, 6, 1, 3, 5

--00032555b34e31ba5304a61eb99e--

From general-return-3876-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 06:29:31 2011
Return-Path: <general-return-3876-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1022F6DD1
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 06:29:31 +0000 (UTC)
Received: (qmail 92038 invoked by uid 500); 20 Jun 2011 06:29:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90930 invoked by uid 500); 20 Jun 2011 06:29:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 80877 invoked by uid 99); 20 Jun 2011 06:23:54 -0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of arvind.pande@gmail.com designates 209.85.220.176 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=2tlht4HgDFkK3R/Zw72pHFslrpaiHA5HnNaILujZ0pI=;
        b=MBASDBv1s9+HgpHtgJTMmXufmy21cVhBqybBVSHG8Hpoc3gBwO+ASbxegj7gWp5zN1
         cgR+cynfR6KFimMPbq0U0ZERJtfWiTHk6DJBhLXtiTHQh5craUJGDxx2R8rJOHwnH4z4
         CVmMjmEKj1DX2UU1QwwwQTGlc3mdYuLL7TZYk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=ZgnGxQzfb+H0FyC3aNhI/AwbwG85R8mLHvA38W1Cf58QaiZVmnATad8yWs72BY5Xmt
         ewAPbxMvTWnTIsGioDiWlbWkljr19jHp1jmff68pYVAmVAMQY15Ntk8JFLPIParqwha/
         pAW3s6yNQpZyHbPvohj0ARvk8QK/7M2v3F7fo=
MIME-Version: 1.0
Date: Mon, 20 Jun 2011 11:53:27 +0530
Message-ID: <BANLkTi=PATQXDyeWEZcsCf590j7m7xHOuw@mail.gmail.com>
Subject: powered by hadoop logo
From: Arvind Pande <arvind.pande@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec5016643f9e7c604a61ec51f

--bcaec5016643f9e7c604a61ec51f
Content-Type: text/plain; charset=ISO-8859-1

4, 2, 6, 1, 3, 5

--bcaec5016643f9e7c604a61ec51f--

From general-return-3877-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 06:46:06 2011
Return-Path: <general-return-3877-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6136E6E5A
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 06:46:06 +0000 (UTC)
Received: (qmail 6887 invoked by uid 500); 20 Jun 2011 06:46:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6709 invoked by uid 500); 20 Jun 2011 06:45:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6701 invoked by uid 99); 20 Jun 2011 06:45:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 06:45:54 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of teamphy6@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 06:45:48 +0000
Received: by bwz8 with SMTP id 8so3163020bwz.35
        for <general@hadoop.apache.org>; Sun, 19 Jun 2011 23:45:28 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type:content-transfer-encoding;
        bh=zXXqB/8KLYtSiVZzAVYaXEKVkpo5XsD2oAh+4gGi3ps=;
        b=FyyXAUt/2A70m2IuNB/nndeabpHG1t9xycYq8/1149fKm1ZztcBIwSWVsmsaNHI14r
         jl/2NAXUEZMDNSHhRzwCSfHbHei1UD3gvDmraswug7Red6+nQmSvnG2ozaMZkIdvPuEc
         wdVMl2nVRxHynp2JN1iRNRzj0aRUwRSb1WaSg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=SAsUzjLwCPQ6ZNA0jwBHFPZb0ZTB0mwB6OgTDPNFllgLoev73+YODXKV+JZvgf3nUd
         gi/Cren03NChpBA0h1yclskVp4X9hRlDObJYfAxvQC6Z2qSEMU5JdjK93r0QGPIy5V/0
         2YqJB8+p+Hu0PmiOhd4jl9ECa01yxGnMYzUjU=
MIME-Version: 1.0
Received: by 10.204.25.66 with SMTP id y2mr1348420bkb.153.1308552327851; Sun,
 19 Jun 2011 23:45:27 -0700 (PDT)
Received: by 10.204.20.3 with HTTP; Sun, 19 Jun 2011 23:45:27 -0700 (PDT)
In-Reply-To: <20110620055539.GA18201@linspire.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
	<20110620055539.GA18201@linspire.com>
Date: Mon, 20 Jun 2011 02:45:27 -0400
Message-ID: <BANLkTikbmBb7Qyw2fbhegg0B6uFEgwZamA@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: Daniel Jue <teamphy6@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Lots of the same bitstring:

4,2,6,1,3,5

out of 6 choose 6, 720 permutations.
Just Sayin.

On Mon, Jun 20, 2011 at 1:55 AM, Konstantin Boudnik <cos@apache.org> wrote:
> How the long the vote is suppose to be running? Usual 72 hours? Or longer=
?
>
> The up-to date vote results are here http://tinyurl.com/3j6629h
>
> Cos
>
> On Tue, Jun 14, 2011 at 08:19PM, Owen O'Malley wrote:
>> All,
>> =A0 =A0We've had a wide range of entries for a powered by logo. I've put=
 them
>> =A0 =A0all on a page, here:
>>
>> http://people.apache.org/~omalley/hadoop-powered-by/
>>
>> Since there are a lot of contenders and we only want a single round of
>> voting, let's use single transferable vote ( STV
>> http://en.wikipedia.org/wiki/Single_transferable_vote). The important th=
ing
>> is to pick the images *IN ORDER* that you would like them.
>>
>> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>>
>> In other words, I want option 4 most and option 6 least. With STV, you d=
on't
>> need to worry about voting for an unpopular choice since your vote will
>> automatically roll over to your next choice.
>>
>> -- Owen
>>
>>
>

From general-return-3878-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 07:15:43 2011
Return-Path: <general-return-3878-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ECD2D69A8
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 07:15:43 +0000 (UTC)
Received: (qmail 37148 invoked by uid 500); 20 Jun 2011 07:15:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36467 invoked by uid 500); 20 Jun 2011 07:15:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36459 invoked by uid 99); 20 Jun 2011 07:15:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 07:15:34 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of arvind.pande@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 07:15:26 +0000
Received: by vws7 with SMTP id 7so1357645vws.35
        for <general@hadoop.apache.org>; Mon, 20 Jun 2011 00:15:06 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=zLuEt2HUh8K53087e14NCYcxIlz1AQ6WkOVmXK318MQ=;
        b=cWMWXRPqzn7bI8csTYRzYCX9YczQkKyccr8TvP80OP8RhniSLhcsko7akmpmYoXfWK
         mH6+F//eejZ6CAQdsN1z4HTITNouP9veS0c+1vrpmB0N30RLtOHj7jMSrrXMsZ5+giwI
         WlE4dT/YCpFM9mYQ4LM95jkukkr83fya/iNWw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=tUhMvRc315uHPDLB4S1K/dWQ4x4656Xqb+v2R/MTY4LK620A3zGM91Z4y2NFVlJUEI
         t9Zg3nOi+yq7ZqbqyzR7zRTWspADqgNrFaMP/A0vanLUiJxkoNLHduXfrKqgZrJi94oQ
         i8jK075qCSsLN/lN3d4BNkNrbiWyTcHjSndeo=
MIME-Version: 1.0
Received: by 10.220.48.71 with SMTP id q7mr1900507vcf.8.1308554106102; Mon, 20
 Jun 2011 00:15:06 -0700 (PDT)
Received: by 10.220.201.13 with HTTP; Mon, 20 Jun 2011 00:15:06 -0700 (PDT)
Date: Mon, 20 Jun 2011 12:45:06 +0530
Message-ID: <BANLkTinTnZ91McBJCR+AumLaQ7ZD8nYAyQ@mail.gmail.com>
Subject: powered by hadoop logo:
From: Arvind Pande <arvind.pande@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64616c4a9e2c704a61f7e83
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e64616c4a9e2c704a61f7e83
Content-Type: text/plain; charset=ISO-8859-1

4, 2, 6, 1, 3, 5

--0016e64616c4a9e2c704a61f7e83--

From general-return-3879-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 07:52:16 2011
Return-Path: <general-return-3879-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DAF6E6EDE
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 07:52:16 +0000 (UTC)
Received: (qmail 78649 invoked by uid 500); 20 Jun 2011 07:52:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78357 invoked by uid 500); 20 Jun 2011 07:52:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78337 invoked by uid 99); 20 Jun 2011 07:52:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 07:52:14 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of atm@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 07:52:08 +0000
Received: by bwz8 with SMTP id 8so3205936bwz.35
        for <general@hadoop.apache.org>; Mon, 20 Jun 2011 00:51:46 -0700 (PDT)
Received: by 10.204.7.6 with SMTP id b6mr3838543bkb.177.1308556306176; Mon, 20
 Jun 2011 00:51:46 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.118.210 with HTTP; Mon, 20 Jun 2011 00:51:15 -0700 (PDT)
In-Reply-To: <20110620055539.GA18201@linspire.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org> <20110620055539.GA18201@linspire.com>
From: "Aaron T. Myers" <atm@cloudera.com>
Date: Mon, 20 Jun 2011 00:51:15 -0700
Message-ID: <BANLkTi=LKRj_berM5uikgtbZniYU+X-dbA@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000325555866cc5a4e04a62001a4

--000325555866cc5a4e04a62001a4
Content-Type: text/plain; charset=ISO-8859-1

On Sun, Jun 19, 2011 at 10:55 PM, Konstantin Boudnik <cos@apache.org> wrote:

> How the long the vote is suppose to be running? Usual 72 hours? Or longer?


http://hadoop.apache.org/bylaws.html#Decision+Making

>From that page I assume that the time frame is 7 days, since the Hadoop
bylaws mention no other length of time a vote may stay open.

--
Aaron T. Myers
Software Engineer, Cloudera

--000325555866cc5a4e04a62001a4--

From general-return-3880-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 08:07:16 2011
Return-Path: <general-return-3880-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 58B686738
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 08:07:16 +0000 (UTC)
Received: (qmail 130 invoked by uid 500); 20 Jun 2011 08:07:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99899 invoked by uid 500); 20 Jun 2011 08:07:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 99879 invoked by uid 99); 20 Jun 2011 08:07:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 08:07:13 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of hammer@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 07:58:06 +0000
Received: by bwz8 with SMTP id 8so3209803bwz.35
        for <general@hadoop.apache.org>; Mon, 20 Jun 2011 00:57:46 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.205.24.9 with SMTP id rc9mr1953996bkb.92.1308556665856; Mon,
 20 Jun 2011 00:57:45 -0700 (PDT)
Received: by 10.204.40.195 with HTTP; Mon, 20 Jun 2011 00:57:45 -0700 (PDT)
In-Reply-To: <BANLkTi=LKRj_berM5uikgtbZniYU+X-dbA@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
	<20110620055539.GA18201@linspire.com>
	<BANLkTi=LKRj_berM5uikgtbZniYU+X-dbA@mail.gmail.com>
Date: Mon, 20 Jun 2011 00:57:45 -0700
Message-ID: <BANLkTiny_h_xcBzb4QknJaitHvF0VD3peg@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf303640133ca2a604a620175e
X-Virus-Checked: Checked by ClamAV on apache.org

--20cf303640133ca2a604a620175e
Content-Type: text/plain; charset=ISO-8859-1

5, 2.

On Mon, Jun 20, 2011 at 12:51 AM, Aaron T. Myers <atm@cloudera.com> wrote:

> On Sun, Jun 19, 2011 at 10:55 PM, Konstantin Boudnik <cos@apache.org>
> wrote:
>
> > How the long the vote is suppose to be running? Usual 72 hours? Or
> longer?
>
>
> http://hadoop.apache.org/bylaws.html#Decision+Making
>
> From that page I assume that the time frame is 7 days, since the Hadoop
> bylaws mention no other length of time a vote may stay open.
>
> --
> Aaron T. Myers
> Software Engineer, Cloudera
>

--20cf303640133ca2a604a620175e--

From general-return-3881-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 08:14:56 2011
Return-Path: <general-return-3881-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4F3406ACE
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 08:14:56 +0000 (UTC)
Received: (qmail 11435 invoked by uid 500); 20 Jun 2011 08:14:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11241 invoked by uid 500); 20 Jun 2011 08:14:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11232 invoked by uid 99); 20 Jun 2011 08:14:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 08:14:54 +0000
X-ASF-Spam-Status: No, hits=2.3 required=5.0
	tests=FREEMAIL_FROM,FROM_EXCESS_BASE64,HTML_MESSAGE,HTML_NONELEMENT_30_40,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of nowplay@qq.com designates 64.71.138.44 as permitted sender)
Received: from [64.71.138.44] (HELO smtpbg55.qq.com) (64.71.138.44)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 20 Jun 2011 08:14:48 +0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=qq.com; s=s0907;
	t=1308557657; bh=zC99YMUmKb5xShX7wvmr+2Ncqb3ruSgsBu2OONAKGrA=;
	h=X-QQ-SSF:X-QQ-BUSINESS-ORIGIN:X-Originating-IP:X-QQ-STYLE:
	 X-QQ-mid:From:To:Subject:Mime-Version:Content-Type:
	 Content-Transfer-Encoding:Date:X-Priority:Message-ID:X-QQ-MIME:
	 X-Mailer:X-QQ-Mailer:X-QQ-ReplyHash;
	b=GBfzZGtm/zohXuZCNWVUxR9zr1JZ7DkoIGCEmnAaRD8iOzT0yLWFTISfon0ztfonF
	 YVLlGIJyjcQ+CQgp2EaN7jl7yVV9u6CQnAdIO4PVobEDcaulc3hGgD/mM10UaRS
X-QQ-SSF:0000000000000050
X-QQ-BUSINESS-ORIGIN:2
X-Originating-IP: 64.233.99.51
X-QQ-STYLE:
X-QQ-mid:webmail368t1308557654t445523
From: "=?ISO-8859-1?B?bm93cGxheQ==?=" <nowplay@qq.com>
To: "=?ISO-8859-1?B?Z2VuZXJhbA==?=" <general@hadoop.apache.org>
Subject: Re:[VOTE] Powered by Logo
Mime-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_NextPart_4DFF0155_DEBF6CF8_1CD762BE"
Content-Transfer-Encoding: 8Bit
Date: Mon, 20 Jun 2011 16:14:13 +0800
X-Priority: 3
Message-ID: <tencent_48DCE33525C479B93AFBCB73@qq.com>
X-QQ-MIME: TCMime 1.0 by Tencent
X-Mailer: QQMail 2.x
X-QQ-Mailer: QQMail 2.x
X-QQ-ReplyHash: 2327217350

------=_NextPart_4DFF0155_DEBF6CF8_1CD762BE
Content-Type: text/plain;
	charset="ISO-8859-1"
Content-Transfer-Encoding: base64

MSw2LDUsMiw0LDM=

------=_NextPart_4DFF0155_DEBF6CF8_1CD762BE--


From general-return-3882-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 08:29:19 2011
Return-Path: <general-return-3882-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AFFBB62C2
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 08:29:19 +0000 (UTC)
Received: (qmail 34498 invoked by uid 500); 20 Jun 2011 08:29:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34426 invoked by uid 500); 20 Jun 2011 08:29:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34418 invoked by uid 99); 20 Jun 2011 08:29:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 08:29:18 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of hammer@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 08:29:13 +0000
Received: by bwz8 with SMTP id 8so3231514bwz.35
        for <general@hadoop.apache.org>; Mon, 20 Jun 2011 01:28:52 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.75.23 with SMTP id w23mr3943483bkj.200.1308558531716; Mon,
 20 Jun 2011 01:28:51 -0700 (PDT)
Received: by 10.204.40.195 with HTTP; Mon, 20 Jun 2011 01:28:51 -0700 (PDT)
In-Reply-To: <4DFCB9F1.2010603@shanecurcuru.org>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
	<4DFCB9F1.2010603@shanecurcuru.org>
Date: Mon, 20 Jun 2011 01:28:51 -0700
Message-ID: <BANLkTi=wZomy0sMGDrg3+VGjoYyHbjBxkw@mail.gmail.com>
Subject: Fwd: [VOTE] Shall we adopt the "Defining Hadoop" page
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d77df67364e604a6208609

--0016e6d77df67364e604a6208609
Content-Type: text/plain; charset=ISO-8859-1

Hey,

I'm inclined towards freedom with respect to the use of the trademark. I
personally was drawn towards the Apache 2.0 license because of its more
liberal stance on source code modifications; I worked hard to submit many of
our internal projects at Facebook to the ASF against significant internal
resistance because I felt that the ASF would uphold this ecumenical stance
in the face of corporate interests for control. I'm surprised by the
consensus for complex controls versus promiscuity.

I'm skeptical of any argument that relies on the ignorance of the customer.
When a press release from Company X comes out using the Apache Hadoop name,
that raises awareness for the project. Customers are smart enough to ask
whether Company X make significant contributions to the Apache Hadoop
project, and to discern in what ways the project deviates form the Apache
Hadoop releases.  We need to consider the benefit as well as the harm to the
project when these Apache Hadoop-related projects use the Apache Hadoop
trademark: in this particular case, I believe the benefit exceeds the harm.
If we were shipping pharmaceuticals or food, I'd feel differently. As it
stands, we're shipping software used by highly informed and technical
customers. Let's give them the right to make up their mind, and provide
further encouragement for other companies to pile on the Apache Hadoop name.

Further, enforcing these regulations will require significant manpower. In
my experience, the ASF is run by a small and dedicated staff. I'd prefer if
their energies were directed towards the maintenance of the infrastructure
for hosting and developing projects, as well as the mechanisms for making
projects develop more rapidly. While there is great enthusiasm for drafting
these rules right now, I suspect that sustained enthusiasm for the
enforcement of these rules will prove more difficult to muster.

Lastly, I'd love to learn more about how other prominent open source
projects have approached this issue. If you have any knowledge about how
Linux handled the use of its trademark, please add your thoughts to
http://www.quora.com/What-are-the-rules-for-using-the-Linux-trademark-in-a-product-name.
Because Apache Hadoop is a kernel technology, similar to Linux, I suspect
there are many useful lessons to learn. Or at least crazy email threads to
read.

Anyways, that's my version of the "the more, the merrier" argument. I'm
pretty excited to see EMC and IBM show up on our doorstep. I'd be even more
excited to see them contribute code; I suspect that will happen in time, if
we create a welcoming community.

Later,
Jeff

---------- Forwarded message ----------
From: Shane Curcuru <asf@shanecurcuru.org>
Date: Sat, Jun 18, 2011 at 7:45 AM
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
To: trademarks@apache.org
Cc: general@hadoop.apache.org


One clarification: I've only had time to review the wiki document for some
terminology updates, and not for the overall content yet.  So from the
trademarks@ point of view, more review is needed before we work on making
this official.

>From the significant amount of discussion in this vote thread, I think it
might be good to have the Hadoop PMC and trademarks@ work on getting a more
organized consensus first, before voting on an updated proposed Hadoop
policy.

- Shane

Owen O'Malley wrote:

> All,
>   Steve Loughran has done some great work on defining what can be called
> Hadoop at http://wiki.apache.org/hadoop/**Defining%20Hadoop<http://wiki.apache.org/hadoop/Defining%20Hadoop><
> http://wiki.apache.org/**hadoop/Defining<http://wiki.apache.org/hadoop/Defining>Hadoop>. After some cleanup from Noirin and Shane, I think we've got a
> really good base. I'd like a vote to approve the content (at the current
> revision 12) and put the content on our web site.
>
>
> Clearly, I'm +1.
>
> -- Owen
>

--0016e6d77df67364e604a6208609--

From general-return-3883-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 08:45:04 2011
Return-Path: <general-return-3883-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B7FD96DF1
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 08:45:04 +0000 (UTC)
Received: (qmail 48420 invoked by uid 500); 20 Jun 2011 08:45:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48183 invoked by uid 500); 20 Jun 2011 08:45:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 53869 invoked by uid 99); 20 Jun 2011 07:25:53 -0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of knsathya@hotmail.com designates 65.55.90.48 as permitted sender)
Message-ID: <SNT105-W48F20F430ACC2F2EBF1CA3DC6E0@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_85af0a5f-4719-4f3d-81d9-357e284362f0_"
X-Originating-IP: [24.130.198.228]
From: Sathya Kavacheri <knsathya@hotmail.com>
To: <general@hadoop.apache.org>
Subject: vote
Date: Mon, 20 Jun 2011 00:25:24 -0700
Importance: Normal
MIME-Version: 1.0
X-OriginalArrivalTime: 20 Jun 2011 07:25:25.0367 (UTC) FILETIME=[38FDA070:01CC2F1B]

--_85af0a5f-4719-4f3d-81d9-357e284362f0_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


4=2C 6=2C 2=2C 1=2C 5=2C 3
thx 		 	   		  =

--_85af0a5f-4719-4f3d-81d9-357e284362f0_--

From general-return-3884-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 09:26:05 2011
Return-Path: <general-return-3884-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C9258668D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 09:26:05 +0000 (UTC)
Received: (qmail 10871 invoked by uid 500); 20 Jun 2011 09:26:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 10828 invoked by uid 500); 20 Jun 2011 09:26:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 10820 invoked by uid 99); 20 Jun 2011 09:26:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 09:26:03 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 69.147.107.20 is neither permitted nor denied by domain of ranjit@yahoo-inc.com)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 09:25:57 +0000
Received: from dishnailfind-dx.eglbp.corp.yahoo.com (dishnailfind-dx.eglbp.corp.yahoo.com [10.66.77.50])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5K9PHTV067228
	for <general@hadoop.apache.org>; Mon, 20 Jun 2011 02:25:18 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1308561919;
	bh=RfLR4mJA+Mq8SEwn8zwMiGEb1tjTXrGiJ/I3jaivCNA=;
	h=Message-ID:Date:From:MIME-Version:To:Subject:References:
	 In-Reply-To:Content-Type:Content-Transfer-Encoding;
	b=r8tua+69XdcgaUHT98faKg08eOiUYOBpr/H0+NdTP0+NQIIu6P3YjCFd8L5N4voU1
	 Q+SnoMvwhhHOIySAXxwp+ZVXTal+izTAQ8zFuU6w2czTR9SNgPgOOR7csCWhtznIPU
	 ido6ZgU+iZ0I+oGLqw8gYzoYeXn91ujikxYDPIYo=
Message-ID: <4DFF11FD.8020507@yahoo-inc.com>
Date: Mon, 20 Jun 2011 14:55:17 +0530
From: Ranjit Mathew <ranjit@yahoo-inc.com>
Organization: Yahoo
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.17) Gecko/20110414 Lightning/1.0b2 Thunderbird/3.1.10
MIME-Version: 1.0
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: [VOTE] Powered by Logo
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 06/15/2011 08:49 AM, Owen O'Malley wrote:
>     We've had a wide range of entries for a powered by logo. I've put them all on a page, here:
>
> http://people.apache.org/~omalley/hadoop-powered-by/

I think #4 modified slightly to make the elephant run on a wheel
or mill like the usual hamster-on-a-wheel thing would be perfect.

Ranjit

From general-return-3885-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 12:44:11 2011
Return-Path: <general-return-3885-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 65E9461C9
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 12:44:11 +0000 (UTC)
Received: (qmail 45852 invoked by uid 500); 20 Jun 2011 12:44:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45685 invoked by uid 500); 20 Jun 2011 12:44:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45677 invoked by uid 99); 20 Jun 2011 12:44:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 12:44:09 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 12:44:00 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 92BC9B8091
	for <general@hadoop.apache.org>; Mon, 20 Jun 2011 13:43:37 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id PbLMJBPp9QcC for <general@hadoop.apache.org>;
	Mon, 20 Jun 2011 13:43:34 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by tobor.hpl.hp.com (Postfix) with ESMTP id D70C3B7E2B
	for <general@hadoop.apache.org>; Mon, 20 Jun 2011 13:43:33 +0100 (BST)
MailScanner-NULL-Check: 1309178599.73771@mqKNf4IdrSu+96Dnqf9oYw
Received: from [16.24.203.78] ([16.24.203.78])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p5KChJL7020320
	for <general@hadoop.apache.org>; Mon, 20 Jun 2011 13:43:19 +0100 (BST)
Message-ID: <4DFF4061.9060800@apache.org>
Date: Mon, 20 Jun 2011 13:43:13 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.2.17) Gecko/20110414 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org> <BANLkTimRX0N7pyrnQrfiwcN=AqHE=FJWzg@mail.gmail.com> <FBF55C01-D66A-44C0-9252-D54E7CAB103D@apache.org> <BANLkTi=BWPUEoqD3L98A9AXLkbafZs15Zw@mail.gmail.com> <92D32E6E-3B65-464F-B91B-4C30EAB2642C@apache.org> <BANLkTi=1JGe8kVortj1Act3jdt9L2RCrhw@mail.gmail.com> <4DF880DA.7060105@apache.org> <BANLkTim6DrVsb5NpwnvAvQtLqa5hq0eOrg@mail.gmail.com> <4DFB3421.2010902@apache.org> <20110617181725.GB22372@linspire.com>
In-Reply-To: <20110617181725.GB22372@linspire.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p5KChJL7020320
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 17/06/2011 19:17, Konstantin Boudnik wrote:
> On Fri, Jun 17, 2011 at 12:01PM, Steve Loughran wrote:
>> On 15/06/11 16:58, Konstantin Boudnik wrote:
>>> On Wed, Jun 15, 2011 at 02:52, Steve Loughran<stevel@apache.org>   wrote:
>>>> also: banners, stickers and clothing? Can I have T-shirts saying "I broke
>>>> the hadoop build" with the logo on, or should it be "I broke the Apache
>>>> Hadoop build"?
>>>
>>> I think such a T-shirt should be forcefully worn on any person who did
>>> just that.
>>
>> Here you go with the poster:
>> http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/hadoop-components/hadoop-ops/doc/breaking_the_hadoop_build.odp?revision=8630
>>
>> I can add it to hadoop-common SVN for people to work on...
>
> Please do, by all means :) !

https://issues.apache.org/jira/browse/HADOOP-7406

now, what happens if it gets checked in in a way that breaks the build? 
That would be too much.

From general-return-3886-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 16:39:58 2011
Return-Path: <general-return-3886-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 294A763C8
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 16:39:58 +0000 (UTC)
Received: (qmail 1142 invoked by uid 500); 20 Jun 2011 16:39:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 970 invoked by uid 500); 20 Jun 2011 16:39:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 962 invoked by uid 99); 20 Jun 2011 16:39:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 16:39:56 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [98.139.52.243] (HELO nm26-vm1.bullet.mail.ac4.yahoo.com) (98.139.52.243)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 20 Jun 2011 16:39:47 +0000
Received: from [98.139.52.196] by nm26.bullet.mail.ac4.yahoo.com with NNFMP; 20 Jun 2011 16:39:26 -0000
Received: from [98.139.52.173] by tm9.bullet.mail.ac4.yahoo.com with NNFMP; 20 Jun 2011 16:39:26 -0000
Received: from [127.0.0.1] by omp1056.mail.ac4.yahoo.com with NNFMP; 20 Jun 2011 16:39:26 -0000
X-Yahoo-Newman-Property: ymail-5
X-Yahoo-Newman-Id: 490711.15938.bm@omp1056.mail.ac4.yahoo.com
Received: (qmail 70324 invoked by uid 60001); 20 Jun 2011 16:39:26 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1308587966; bh=zf1DrDmLwFdpPJZcWu96XFmQ4uT3d5tk4FUye91k2mI=; h=Message-ID:X-YMail-OSG:Received:X-RocketYMMF:X-Mailer:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=noNH0ow/cL7ECq4ZXqlF2vEwiPrUbFZoGgME7iLimeHiMeW2Vp9HTsNBzzMoWbnlDzKCoeDVQpYkK0DdvyrxLBMbSlMKyaXGZm0q+d1yK8qmMq24QE5nzkNbpH8PLdKmFgxxeBn8AOwFucyl1z4eaaolWHxISwYPJV73/+m9ciY=
Message-ID: <374085.46450.qm@web65510.mail.ac4.yahoo.com>
X-YMail-OSG: syoCJ2EVM1lquKvTfb6Av5QislD0eEhk3CECCkpbmzEbHoO
 e.VZz2oNC4KbwD4Lxyx8pCjamf3di1MyywLWccG5Vbw9d8OQQqtUd6b2831m
 iyBWojMsZ_AdvOJq_AymjO_RPs1QwWuNYD_tYvFJZy3EqkvRjjzclKRukiTe
 Bcvi5Aqs.BIup9vzHPCqGPPi2x6wYSRFV4cT.QrQjHQseanIqO7fhZbkYOUE
 1x.ohZil444yYdYtNxbanwKPItj0Ryqk_jQIZibnA4GSLSdcRl33iGr26Ua1
 76KT1lIJBRI.0ZXNpjeTjnuHf4iBY34k_Ud_0QdQc6jMPI_blpRtqxatQDuT
 5H8nG_ADNenfoKK9uAIKHHOHJLdQymvU1EKlNIDqBlYEe0kjEBiNbrcg7pUz
 _30UBQTnRFoGL3X.N.UlOcr7.gZQ6lvEP339gF8KD36f2udNnDhrtBPgJ05B
 HjRJzdKrQekBVi0ECt2ZDIKk_fUWXkGB2lRXCJcwhWEclX9tENtysPujfwX1
 b5T7TpCCIB0SChgaiZOhd_Alzm1gAzGJ5ln5t5q0-
Received: from [69.231.27.174] by web65510.mail.ac4.yahoo.com via HTTP; Mon, 20 Jun 2011 09:39:26 PDT
X-RocketYMMF: apurtell
X-Mailer: YahooMailClassic/14.0.3 YahooMailWebService/0.8.111.304355
Date: Mon, 20 Jun 2011 09:39:26 -0700 (PDT)
From: Andrew Purtell <apurtell@apache.org>
Reply-To: apurtell@apache.org
Subject: Re: Fwd: [VOTE] Shall we adopt the "Defining Hadoop" page
To: general@hadoop.apache.org
In-Reply-To: <BANLkTi=wZomy0sMGDrg3+VGjoYyHbjBxkw@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Jeff,

First, apologies for removing most of your argument for clarity. Readers can find it in the general@ archives I am sure.

> Lastly, I'd love to learn more about how other prominent open source
> projects have approached this issue. If you have any knowledge about
> how Linux handled the use of its trademark, please add your
> thoughts to
> http://www.quora.com/What-are-the-rules-for-using-the-Linux-trademark-in-a-product-name.
> Because Apache Hadoop is a kernel technology, similar to Linux, I
> suspect there are many useful lessons to learn. Or at least crazy
> email threads to read.

I would argue the concern about trademark has an additional dimension here, and perhaps a fairly core additional motivation to protect, because these are open source projects. The mention of Linux helps to illustrate it.

The obvious difference between Hadoop and Linux is Linux has a universally recognized clear hierarchy with a single -- and exceptional, and quickly and forcefully opinionated -- authority at the top. For Linux, the power to define Linux rests obviously with Linus. Regarding Hadoop, the power to do anything, including define what is Hadoop, is diffuse.

For would-be open source participants who want to contribute to the Linux kernel, the canonical source of the Linux kernel is clearly Linus' tree and you want your contribution to end up there. He is the authority. Linux will always be defined by Linus until he is gone. (That is a long term problem for Linux of course.) It is a benevolent dictatorship that perhaps uniquely works, allowing enough contributors to see the fruits of their labor to sustain it while simultaneously maintaining a strong identity. 

Hadoop has no equivalent.

Linux, for now at least, can be quite liberal in how the Linux mark is used because of how its identity as a project is defined, therefore its ability to attract contributions.

Hadoop I think needs to be more careful. What triggered this discussion is the arrival of new players releasing products they call Hadoop but containing severe changes the community, by way of the ASF umbrella we all work under, had nothing to do with designing or developing. And some of these are being open sourced as a Hadoop. There is no Linus here. Which of these is _the_ Hadoop? As a would-be contributor, which should I select?

Already we have some issues. In some cases I'd rather contribute to Cloudera sources because at least I know my contribution to CDH will see a timely release.

Furthermore, I believe the extent to which users see value in ASF Hadoop, and have a clear definition of what ASF Hadoop is, will be correlated with the extent to which the ASF can attract enough contributions to Hadoop to sustain innovation against competing technologies.

The open source value proposition "I contribute to Hadoop" impacts the long term survival of the project. Individuals and organizations are both motivated by this, for various reasons.

   - Andy


From general-return-3887-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 17:10:47 2011
Return-Path: <general-return-3887-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3AA126BF2
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 17:10:47 +0000 (UTC)
Received: (qmail 60248 invoked by uid 500); 20 Jun 2011 17:10:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60051 invoked by uid 500); 20 Jun 2011 17:10:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60029 invoked by uid 99); 20 Jun 2011 17:10:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 17:10:45 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.220.176] (HELO mail-vx0-f176.google.com) (209.85.220.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 17:10:37 +0000
Received: by vxd2 with SMTP id 2so3318823vxd.35
        for <general@hadoop.apache.org>; Mon, 20 Jun 2011 10:10:16 -0700 (PDT)
Received: by 10.52.174.49 with SMTP id bp17mr826933vdc.243.1308589816438; Mon,
 20 Jun 2011 10:10:16 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.52.111.166 with HTTP; Mon, 20 Jun 2011 10:09:56 -0700 (PDT)
X-Originating-IP: [64.105.168.204]
In-Reply-To: <374085.46450.qm@web65510.mail.ac4.yahoo.com>
References: <BANLkTi=wZomy0sMGDrg3+VGjoYyHbjBxkw@mail.gmail.com> <374085.46450.qm@web65510.mail.ac4.yahoo.com>
From: Ted Dunning <tdunning@maprtech.com>
Date: Mon, 20 Jun 2011 17:09:56 +0000
Message-ID: <BANLkTikusmVmKVVMbTqCgLprPrC0ibB3Gw@mail.gmail.com>
Subject: Re: Fwd: [VOTE] Shall we adopt the "Defining Hadoop" page
To: general@hadoop.apache.org, apurtell@apache.org
Content-Type: multipart/alternative; boundary=bcaec51b17232a5dd304a627cf32

--bcaec51b17232a5dd304a627cf32
Content-Type: text/plain; charset=ISO-8859-1

Great summary Andrew.

I would add one more precipitating factor here.  That is the arrival of a
number of products which are very close to the Apache version of Hadoop but
for which there is no good and widely accepted terminology that gives proper
credit to their lineage while making clear the distinction from bit-for-bit
copies of official Apache releases.

Some products are analogous to hive, pig or hbase in that they are
independent systems that run ON hadoop (or close equivalents).  These have
no terminology problem because these products aren't hadoop, but rather use
hadoop.

Other products contain Hadoop internally as a critical component but do not
necessarily expose Hadoop capabilities to the end user (I can't name these
products, but they exist).  These products have little nomenclatural
difficulty because the powerd-by-Hadoop description fits very well.

The products with the terminology problem are the ones that are add either
curation and packaging (Cloudera) or substantial additional performance
enhancing components (MapR).  These products are upwardly compatible with
Apache Hadoop in that programs that run on Hadoop will very probably run on
these Hadoop-like systems.  The problem is that there is no good term for
these products.  They may even contain components that are bit-for-bit
identical to the same components for Apache releases.  It is fair to say
that these are not Apache released software, but it is also fair to say that
there ought to be a better name for the class of these products.

On Mon, Jun 20, 2011 at 4:39 PM, Andrew Purtell <apurtell@apache.org> wrote:

> Hadoop I think needs to be more careful. What triggered this discussion is
> the arrival of new players releasing products they call Hadoop but
> containing severe changes the community, by way of the ASF umbrella we all
> work under, had nothing to do with designing or developing. And some of
> these are being open sourced as a Hadoop. There is no Linus here. Which of
> these is _the_ Hadoop? As a would-be contributor, which should I select?
>

--bcaec51b17232a5dd304a627cf32--

From general-return-3888-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 18:45:57 2011
Return-Path: <general-return-3888-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DAFCE605D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 18:45:56 +0000 (UTC)
Received: (qmail 33580 invoked by uid 500); 20 Jun 2011 18:45:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33508 invoked by uid 500); 20 Jun 2011 18:45:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 53618 invoked by uid 99); 20 Jun 2011 08:49:25 -0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
X-Originating-IP: [166.205.139.225]
Date: Mon, 20 Jun 2011 01:48:55 -0700
Message-ID: <BANLkTi=N2iBr63Hf+jFvtz6=2Zsv=6iN6A@mail.gmail.com>
Subject: Powered by Hadoop logo
From: Steven Phillips <sphillips@maprtech.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00151773e9ae2fb52904a620ce2d
X-Virus-Checked: Checked by ClamAV on apache.org

--00151773e9ae2fb52904a620ce2d
Content-Type: text/plain; charset=ISO-8859-1

4,2,6,1,3,5

--00151773e9ae2fb52904a620ce2d--

From general-return-3889-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 18:45:58 2011
Return-Path: <general-return-3889-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D006B60B0
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 18:45:58 +0000 (UTC)
Received: (qmail 34606 invoked by uid 500); 20 Jun 2011 18:45:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34524 invoked by uid 500); 20 Jun 2011 18:45:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 98277 invoked by uid 99); 20 Jun 2011 09:14:47 -0000
X-ASF-Spam-Status: No, hits=2.1 required=5.0
	tests=FREEMAIL_FROM,HK_RANDOM_ENVFROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of narendra.aaaa@gmail.com designates 74.125.83.48 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=QkPIUajBhOc+OAj1WlvRrzWcNhApEfgNp5VOmFwglwg=;
        b=ijdBZGE5L+XFCFywPn4JCVXjmNupSdjviMUIvEcJ2GABAQgBeBQQCYXryC0PG76Glk
         QDLuHa/r2eaDFmBSeex2cvugqDUIAcVcFu59WErsWU6hmtDhAKdl0eJwtUU6QTSgS5Dc
         pOWkSzI2Vvql/I39aA92s1eEJ6KB/24K/T6l4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=jgH5A6fSOcpUbxrfiFF3rsSYmeKRWjQPiBGabKn2/nb/zM3+CxXABmfjMtsfW7e4Gt
         3tPVaw23R0y1omZuE9n3Y2xMHCF7IDxgstJlxTgPf98WScMeoRHDw2GZhINQjBY1da6R
         BQ8whfziePTFD93G54prSgFUU9VfW1MhnYcSI=
MIME-Version: 1.0
In-Reply-To: <BANLkTi=RYUgnGbqRtzN9WHkRBwMkd73Bdw@mail.gmail.com>
References: <BANLkTink1XyruQD-URYx647ttWSbL=vdzw@mail.gmail.com>
	<BANLkTik=zyXtAJLpfpJ+4c-z+awabXEzZw@mail.gmail.com>
	<BANLkTi=sChb8cxs0t0T50SNZ3AqVfonxCA@mail.gmail.com>
	<BANLkTi=RYUgnGbqRtzN9WHkRBwMkd73Bdw@mail.gmail.com>
Date: Mon, 20 Jun 2011 14:44:20 +0530
Message-ID: <BANLkTinLV9gHdZLh8RE382n6vf7V3+vUdA@mail.gmail.com>
Subject: powered by hadoop logo:
From: narendra agrawal <narendra.aaaa@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636283af213f49004a6212983

--001636283af213f49004a6212983
Content-Type: text/plain; charset=ISO-8859-1

4, 2, 6, 3, 1, 5

--001636283af213f49004a6212983--

From general-return-3890-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 19:58:06 2011
Return-Path: <general-return-3890-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1DDE5637F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 19:58:06 +0000 (UTC)
Received: (qmail 87576 invoked by uid 500); 20 Jun 2011 19:58:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87520 invoked by uid 500); 20 Jun 2011 19:58:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87511 invoked by uid 99); 20 Jun 2011 19:58:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 19:58:04 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of vicaya@gmail.com designates 74.125.83.48 as permitted sender)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 19:57:58 +0000
Received: by gwj22 with SMTP id 22so1180619gwj.35
        for <general@hadoop.apache.org>; Mon, 20 Jun 2011 12:57:37 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=XpJkrMDFJdWdfav3zWZkXzeDkDP0qF48SANlJ8bA6RA=;
        b=wZynFhocfVrEsoVk6N23tIuns+GfvyUfMeDd2vO7nSHoAt4Vo4kUYq652ePcQrqb9d
         93f0BOqHuiUSfY5gCbJPMGDeMJhZRLZXz9R4kwRAXLEpmtaI/9Oo8dj9HPCmjG2S4Kuz
         wGTRsyp/JQaTWOgh02I2qB1peme/qdQhs6mtc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=gqhK4UaQvMXEShPUAQ8nUKQxzmdiAlVAOD8crf6MsoICmarLqO/FqI/srP3/Q+Gl3o
         4VQaCuhgACyvEX5tndcAvv9+0kO4JRbB7HuKoMyVECguCbzbvpQj3bAKLmES+VjniBhS
         0p17mlNPLZ88qoeUN16MR2aMimXepcRKSmgUg=
MIME-Version: 1.0
Received: by 10.236.66.65 with SMTP id g41mr5903649yhd.320.1308599857045; Mon,
 20 Jun 2011 12:57:37 -0700 (PDT)
Sender: vicaya@gmail.com
Received: by 10.236.111.1 with HTTP; Mon, 20 Jun 2011 12:57:36 -0700 (PDT)
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
Date: Mon, 20 Jun 2011 12:57:36 -0700
X-Google-Sender-Auth: C0Wv2rnx08rbqWDBHiH3mImTI8M
Message-ID: <BANLkTinwpw70j_ZvBkahShguZJOGKoR62g@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: Luke Lu <llu@vicaya.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Just got a few minutes to look at these. I must say that I'm very
disappointed. Some of the choices are nice clip art but none of them
are appropriate for powered-by logos at typical logo size: <=3D64 pixels
in height, where they are completely illegible.

I present choice 0, which IMHO is a better candidate than the rest of
the choices:

https://issues.apache.org/jira/secure/attachment/12483197/pbh-64-logos.png

Please find the larger version on
https://issues.apache.org/jira/browse/HADOOP-7020

__Luke

On Tue, Jun 14, 2011 at 8:19 PM, Owen O'Malley <omalley@apache.org> wrote:
> All,
> =A0 We've had a wide range of entries for a powered by logo. I've put the=
m all on a page, here:
>
> http://people.apache.org/~omalley/hadoop-powered-by/
>
> Since there are a lot of contenders and we only want a single round of vo=
ting, let's use single transferable vote ( STV http://en.wikipedia.org/wiki=
/Single_transferable_vote). The important thing is to pick the images *IN O=
RDER* that you would like them.
>
> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>
> In other words, I want option 4 most and option 6 least. With STV, you do=
n't need to worry about voting for an unpopular choice since your vote will=
 automatically roll over to your next choice.
>
> -- Owen
>
>
>

From general-return-3891-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 20:13:40 2011
Return-Path: <general-return-3891-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CB31A6CE3
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 20:13:40 +0000 (UTC)
Received: (qmail 18321 invoked by uid 500); 20 Jun 2011 20:13:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18255 invoked by uid 500); 20 Jun 2011 20:13:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18225 invoked by uid 99); 20 Jun 2011 20:13:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 20:13:35 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.220.176] (HELO mail-vx0-f176.google.com) (209.85.220.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 20:13:28 +0000
Received: by vxd2 with SMTP id 2so3538365vxd.35
        for <general@hadoop.apache.org>; Mon, 20 Jun 2011 13:13:07 -0700 (PDT)
Received: by 10.52.72.230 with SMTP id g6mr622385vdv.163.1308600787129; Mon,
 20 Jun 2011 13:13:07 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.52.111.166 with HTTP; Mon, 20 Jun 2011 13:12:47 -0700 (PDT)
X-Originating-IP: [64.105.168.204]
In-Reply-To: <BANLkTinwpw70j_ZvBkahShguZJOGKoR62g@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org> <BANLkTinwpw70j_ZvBkahShguZJOGKoR62g@mail.gmail.com>
From: Ted Dunning <tdunning@maprtech.com>
Date: Mon, 20 Jun 2011 13:12:47 -0700
Message-ID: <BANLkTik77fTkaQKLY=6Gm-FjQc4YigFOfQ@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec501601111d45104a62a5d69
X-Virus-Checked: Checked by ClamAV on apache.org

--bcaec501601111d45104a62a5d69
Content-Type: text/plain; charset=ISO-8859-1

Keep in mind that it is thematic elements, not specific implementation that
is being judged here.

Many of these suggestions can be adapted to different font sizes depending
on the desired final rendition
size.

On Mon, Jun 20, 2011 at 12:57 PM, Luke Lu <llu@vicaya.com> wrote:

> Just got a few minutes to look at these. I must say that I'm very
> disappointed. Some of the choices are nice clip art but none of them
> are appropriate for powered-by logos at typical logo size: <=64 pixels
> in height, where they are completely illegible.
>
> I present choice 0, which IMHO is a better candidate than the rest of
> the choices:
>
> https://issues.apache.org/jira/secure/attachment/12483197/pbh-64-logos.png
>
> Please find the larger version on
> https://issues.apache.org/jira/browse/HADOOP-7020
>
> __Luke
>
> On Tue, Jun 14, 2011 at 8:19 PM, Owen O'Malley <omalley@apache.org> wrote:
> > All,
> >   We've had a wide range of entries for a powered by logo. I've put them
> all on a page, here:
> >
> > http://people.apache.org/~omalley/hadoop-powered-by/
> >
> > Since there are a lot of contenders and we only want a single round of
> voting, let's use single transferable vote ( STV
> http://en.wikipedia.org/wiki/Single_transferable_vote). The important
> thing is to pick the images *IN ORDER* that you would like them.
> >
> > My vote (in order of course): 4, 1, 2, 3, 5, 6.
> >
> > In other words, I want option 4 most and option 6 least. With STV, you
> don't need to worry about voting for an unpopular choice since your vote
> will automatically roll over to your next choice.
> >
> > -- Owen
> >
> >
> >
>

--bcaec501601111d45104a62a5d69--

From general-return-3892-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 20:16:26 2011
Return-Path: <general-return-3892-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DC3D8616B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 20:16:26 +0000 (UTC)
Received: (qmail 25186 invoked by uid 500); 20 Jun 2011 20:16:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25128 invoked by uid 500); 20 Jun 2011 20:16:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25120 invoked by uid 99); 20 Jun 2011 20:16:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 20:16:25 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of kntreadway@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 20:16:19 +0000
Received: by bwz8 with SMTP id 8so4060157bwz.35
        for <general@hadoop.apache.org>; Mon, 20 Jun 2011 13:15:59 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type;
        bh=KqkIIE9fDxeAVwMPqDOMpyge4yrJ8Ev76iXZbIgcHhY=;
        b=UWuZ9tv/lxAMcags3EO8cO63fbXbl3kP3Ral4iM1a43J7g3Sm9VAeY/yNMxFAAXYFQ
         wAqgMDkXIpihr7iY9cNmiWd/tIR/k7iyU5JuJ7sDKHL0O0q2YUgWWsgmyTeAzsxLsej+
         ROAcqElsWim7dAJxjiAGSzN8/rxi5dIG3iw9I=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=WGUNkW3gDKp410yKBA5cPPrFXebB+5cOHQvmLadxlFVYVNW8ICqCAx89L1emvOTDDJ
         sM6WXALdk0sh7XkzfIZG94jGo+/gIrvL63xeEjgvDQEf5xQTKGFNWoA6X8eeegPi3/eH
         rbk7j59zB1TUVkAwOyQ0+54QSIvDqSsS9Xdoc=
Received: by 10.204.136.144 with SMTP id r16mr1483789bkt.8.1308600959107; Mon,
 20 Jun 2011 13:15:59 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.42.70 with HTTP; Mon, 20 Jun 2011 13:15:39 -0700 (PDT)
In-Reply-To: <BANLkTik77fTkaQKLY=6Gm-FjQc4YigFOfQ@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
 <BANLkTinwpw70j_ZvBkahShguZJOGKoR62g@mail.gmail.com> <BANLkTik77fTkaQKLY=6Gm-FjQc4YigFOfQ@mail.gmail.com>
From: Nichole Treadway <kntreadway@gmail.com>
Date: Mon, 20 Jun 2011 16:15:39 -0400
Message-ID: <BANLkTi=nH0cF3x4+yhbYjuqmag88MY-70Q@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015174c13ea5200cd04a62a6733
X-Virus-Checked: Checked by ClamAV on apache.org

--0015174c13ea5200cd04a62a6733
Content-Type: text/plain; charset=ISO-8859-1

5, 6, 2, 4, 1, 3

On Mon, Jun 20, 2011 at 4:12 PM, Ted Dunning <tdunning@maprtech.com> wrote:

> Keep in mind that it is thematic elements, not specific implementation that
> is being judged here.
>
> Many of these suggestions can be adapted to different font sizes depending
> on the desired final rendition
> size.
>
> On Mon, Jun 20, 2011 at 12:57 PM, Luke Lu <llu@vicaya.com> wrote:
>
> > Just got a few minutes to look at these. I must say that I'm very
> > disappointed. Some of the choices are nice clip art but none of them
> > are appropriate for powered-by logos at typical logo size: <=64 pixels
> > in height, where they are completely illegible.
> >
> > I present choice 0, which IMHO is a better candidate than the rest of
> > the choices:
> >
> >
> https://issues.apache.org/jira/secure/attachment/12483197/pbh-64-logos.png
> >
> > Please find the larger version on
> > https://issues.apache.org/jira/browse/HADOOP-7020
> >
> > __Luke
> >
> > On Tue, Jun 14, 2011 at 8:19 PM, Owen O'Malley <omalley@apache.org>
> wrote:
> > > All,
> > >   We've had a wide range of entries for a powered by logo. I've put
> them
> > all on a page, here:
> > >
> > > http://people.apache.org/~omalley/hadoop-powered-by/
> > >
> > > Since there are a lot of contenders and we only want a single round of
> > voting, let's use single transferable vote ( STV
> > http://en.wikipedia.org/wiki/Single_transferable_vote). The important
> > thing is to pick the images *IN ORDER* that you would like them.
> > >
> > > My vote (in order of course): 4, 1, 2, 3, 5, 6.
> > >
> > > In other words, I want option 4 most and option 6 least. With STV, you
> > don't need to worry about voting for an unpopular choice since your vote
> > will automatically roll over to your next choice.
> > >
> > > -- Owen
> > >
> > >
> > >
> >
>

--0015174c13ea5200cd04a62a6733--

From general-return-3893-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 20:26:42 2011
Return-Path: <general-return-3893-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D203464AC
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 20:26:42 +0000 (UTC)
Received: (qmail 40209 invoked by uid 500); 20 Jun 2011 20:26:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40147 invoked by uid 500); 20 Jun 2011 20:26:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40139 invoked by uid 99); 20 Jun 2011 20:26:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 20:26:41 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of vicaya@gmail.com designates 209.85.161.176 as permitted sender)
Received: from [209.85.161.176] (HELO mail-gx0-f176.google.com) (209.85.161.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 20:26:35 +0000
Received: by gxk7 with SMTP id 7so2540780gxk.35
        for <general@hadoop.apache.org>; Mon, 20 Jun 2011 13:26:14 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=ov1inoUFVCy0lWkM7YtcNs5Q/kiDJyl3pG+2tOkQrc8=;
        b=eiI8+PMKlN1OOFrbOrNxXww6DWF7gFwW02eXXCfeYcqSkERnJrYhPtgxGGs3PqObQh
         yfSJoDQu+N35RbSJrXvECshyRTJ0HKB7SSw2tgE3GVuAIOAOKF1x6Nk6X61iKOqt3knR
         wZRuk1/cq1c30xr+biWH3lVHPB3A+IU0oHIoQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=COCdvlq01ZfRvabcekrzjcWqfWChfNm55LXV3jE4BTLXNQyi39E7sp7Ud4GXL+Qz2Y
         UMO+fbwDah30HIw823OVp53EFuemGCqMuSdK23toNNgZZQCHwcST9suKdKjvginAh2V2
         cHaRWfJS8eupg6SvafJ1L6FgyRJeYU4QDW2uk=
MIME-Version: 1.0
Received: by 10.236.174.37 with SMTP id w25mr8788205yhl.186.1308601574629;
 Mon, 20 Jun 2011 13:26:14 -0700 (PDT)
Sender: vicaya@gmail.com
Received: by 10.236.111.1 with HTTP; Mon, 20 Jun 2011 13:26:14 -0700 (PDT)
In-Reply-To: <BANLkTik77fTkaQKLY=6Gm-FjQc4YigFOfQ@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
	<BANLkTinwpw70j_ZvBkahShguZJOGKoR62g@mail.gmail.com>
	<BANLkTik77fTkaQKLY=6Gm-FjQc4YigFOfQ@mail.gmail.com>
Date: Mon, 20 Jun 2011 13:26:14 -0700
X-Google-Sender-Auth: 0tWvqCleEYaBn-7O30AIsfOCAY4
Message-ID: <BANLkTi=KmpiUZYEc0N5OCtXt+dbvPcb2Vg@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: Luke Lu <llu@vicaya.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Agreed. My suggestion is essentially thematic as well, as the Apache
theme is mostly missing from the rest of the entries. My suggestion
emphasizes more on Apache and Hadoop rather than the proverbial
elephant.

On Mon, Jun 20, 2011 at 1:12 PM, Ted Dunning <tdunning@maprtech.com> wrote:
> Keep in mind that it is thematic elements, not specific implementation th=
at
> is being judged here.
>
> Many of these suggestions can be adapted to different font sizes dependin=
g
> on the desired final rendition
> size.
>
> On Mon, Jun 20, 2011 at 12:57 PM, Luke Lu <llu@vicaya.com> wrote:
>
>> Just got a few minutes to look at these. I must say that I'm very
>> disappointed. Some of the choices are nice clip art but none of them
>> are appropriate for powered-by logos at typical logo size: <=3D64 pixels
>> in height, where they are completely illegible.
>>
>> I present choice 0, which IMHO is a better candidate than the rest of
>> the choices:
>>
>> https://issues.apache.org/jira/secure/attachment/12483197/pbh-64-logos.p=
ng
>>
>> Please find the larger version on
>> https://issues.apache.org/jira/browse/HADOOP-7020
>>
>> __Luke
>>
>> On Tue, Jun 14, 2011 at 8:19 PM, Owen O'Malley <omalley@apache.org> wrot=
e:
>> > All,
>> > =A0 We've had a wide range of entries for a powered by logo. I've put =
them
>> all on a page, here:
>> >
>> > http://people.apache.org/~omalley/hadoop-powered-by/
>> >
>> > Since there are a lot of contenders and we only want a single round of
>> voting, let's use single transferable vote ( STV
>> http://en.wikipedia.org/wiki/Single_transferable_vote). The important
>> thing is to pick the images *IN ORDER* that you would like them.
>> >
>> > My vote (in order of course): 4, 1, 2, 3, 5, 6.
>> >
>> > In other words, I want option 4 most and option 6 least. With STV, you
>> don't need to worry about voting for an unpopular choice since your vote
>> will automatically roll over to your next choice.
>> >
>> > -- Owen
>> >
>> >
>> >
>>
>

From general-return-3894-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 20:43:36 2011
Return-Path: <general-return-3894-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B0AA06EF6
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 20:43:36 +0000 (UTC)
Received: (qmail 64122 invoked by uid 500); 20 Jun 2011 20:43:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 64065 invoked by uid 500); 20 Jun 2011 20:43:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 64057 invoked by uid 99); 20 Jun 2011 20:43:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 20:43:35 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of wbsrvc@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 20:43:29 +0000
Received: by vws7 with SMTP id 7so2160958vws.35
        for <general@hadoop.apache.org>; Mon, 20 Jun 2011 13:43:08 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=QwIrDQ03OahJJiTmx8LuiltHd8OLPOSkUqR+AbOElBU=;
        b=BwD2NbYilpbdIIkBx6xOUKOxp9Suq+6ms7Yw89T5UrVGXKQfTGTL5qUqP+hisWv7Rb
         ZY8a1n3JoNO8Gqit90gVK5GGO831uWaWiGE6yqQmx2aTgfrbpqvPOdIOfnDi0LmLo0n3
         guhOlnbqsjdZN2CeCfHzCyZeTi93Fdqt2vSbw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=yHe8yH+vTAPY9WIqZ+Tir+JRMbXXWodvUzzCdc3xYWkLldMWG0+ZNxqdjReq9IeUyo
         mOo9orE01HI0iXTPVC7Pd5YsYMvjJk7BdHFErLD3PDdsaDpx7phAOl+gtuglB8VtHAMf
         cwQhS1eeBqsUAKkiv2od0oJx3SbsTbA8+qXkA=
MIME-Version: 1.0
Received: by 10.220.39.2 with SMTP id d2mr1927601vce.274.1308602587912; Mon,
 20 Jun 2011 13:43:07 -0700 (PDT)
Received: by 10.220.96.14 with HTTP; Mon, 20 Jun 2011 13:43:07 -0700 (PDT)
In-Reply-To: <BANLkTinwpw70j_ZvBkahShguZJOGKoR62g@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
	<BANLkTinwpw70j_ZvBkahShguZJOGKoR62g@mail.gmail.com>
Date: Mon, 20 Jun 2011 13:43:07 -0700
Message-ID: <BANLkTi=49ReNeyYeZfR3YUQaT=difjGHzA@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: web service <wbsrvc@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec54ee9e467984d04a62ac876
X-Virus-Checked: Checked by ClamAV on apache.org

--bcaec54ee9e467984d04a62ac876
Content-Type: text/plain; charset=ISO-8859-1

Can we still submit logos ?

On Mon, Jun 20, 2011 at 12:57 PM, Luke Lu <llu@vicaya.com> wrote:

> Just got a few minutes to look at these. I must say that I'm very
> disappointed. Some of the choices are nice clip art but none of them
> are appropriate for powered-by logos at typical logo size: <=64 pixels
> in height, where they are completely illegible.
>
> I present choice 0, which IMHO is a better candidate than the rest of
> the choices:
>
> https://issues.apache.org/jira/secure/attachment/12483197/pbh-64-logos.png
>
> Please find the larger version on
> https://issues.apache.org/jira/browse/HADOOP-7020
>
> __Luke
>
> On Tue, Jun 14, 2011 at 8:19 PM, Owen O'Malley <omalley@apache.org> wrote:
> > All,
> >   We've had a wide range of entries for a powered by logo. I've put them
> all on a page, here:
> >
> > http://people.apache.org/~omalley/hadoop-powered-by/
> >
> > Since there are a lot of contenders and we only want a single round of
> voting, let's use single transferable vote ( STV
> http://en.wikipedia.org/wiki/Single_transferable_vote). The important
> thing is to pick the images *IN ORDER* that you would like them.
> >
> > My vote (in order of course): 4, 1, 2, 3, 5, 6.
> >
> > In other words, I want option 4 most and option 6 least. With STV, you
> don't need to worry about voting for an unpopular choice since your vote
> will automatically roll over to your next choice.
> >
> > -- Owen
> >
> >
> >
>

--bcaec54ee9e467984d04a62ac876--

From general-return-3895-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 20:45:28 2011
Return-Path: <general-return-3895-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 65D296248
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 20:45:28 +0000 (UTC)
Received: (qmail 67419 invoked by uid 500); 20 Jun 2011 20:45:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67325 invoked by uid 500); 20 Jun 2011 20:45:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67317 invoked by uid 99); 20 Jun 2011 20:45:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 20:45:26 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 216.145.54.171 is neither permitted nor denied by domain of sradia@yahoo-inc.com)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 20:45:22 +0000
Received: from oceanfarearth-lm.corp.yahoo.com (oceanfarearth-lm.corp.yahoo.com [10.72.113.156])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5KKikbd085856
	for <general@hadoop.apache.org>; Mon, 20 Jun 2011 13:44:46 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1308602686;
	bh=35UMOcY00mVkGQ3qQsKvaobRLGN79OQeAbZYYGkkilU=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=B6A01vJVf0mw5IuTIs/OPA7GgKaGRTGvzJ9lZv0V2GzwMMCH6zWsZe6eqfQLfLZ5/
	 o8VPBnopKivw89+LApOJDfWQzHeRcyAXqLcQcZN/Xwf4colR9G/MxBJ8w24KhLWiFd
	 bu26rRoGUZdAl3G0a8KR40RUkKlUgrB5U7ii7PRE=
Message-Id: <8186AF6A-8B9A-4BC2-A374-B819B81F4831@yahoo-inc.com>
From: Sanjay Radia <sradia@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <E9E17CC3-013C-46E5-AD48-BF76DEE8CE02@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Thinking about the next hadoop mainline release
Date: Mon, 20 Jun 2011 13:44:46 -0700
References: <EFF679DA-26AB-4250-8F59-A2330D0E3E5C@yahoo-inc.com> <BANLkTimX8gSQRFbVZaVDQBOSN9c4iRi=Nw@mail.gmail.com> <BANLkTi=igCLuZ7v2+L9_nObs8wfPP4_F8g@mail.gmail.com> <E9E17CC3-013C-46E5-AD48-BF76DEE8CE02@yahoo-inc.com>
X-Mailer: Apple Mail (2.936)


On Jun 17, 2011, at 7:15 AM, Arun C Murthy wrote:

> I volunteer to be the RM for the release since I've been leading the  
> NG NR effort.
>
> Are folks ok with this?


+1
sanjay

>
> thanks,
> Arun
>
> Sent from my iPhone
>
> On Jun 17, 2011, at 1:45 PM, "Ted Dunning" <tdunning@maprtech.com>  
> wrote:
>
>> NG map reduce is a huge deal both in terms of making things better  
>> for
>> users, but also in terms of unblocking the Hadoop development  
>> process.
>>
>> On Fri, Jun 17, 2011 at 9:36 AM, Ryan Rawson <ryanobjc@gmail.com>  
>> wrote:
>>
>>>> - Next Generation Map-Reduce [MR-279]
>>>> - Passing most tests now and discussing merging into trunk
>>>


From general-return-3896-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 20:57:35 2011
Return-Path: <general-return-3896-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BB7076A44
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 20:57:35 +0000 (UTC)
Received: (qmail 87396 invoked by uid 500); 20 Jun 2011 20:57:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87337 invoked by uid 500); 20 Jun 2011 20:57:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87329 invoked by uid 99); 20 Jun 2011 20:57:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 20:57:34 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of vicaya@gmail.com designates 209.85.161.176 as permitted sender)
Received: from [209.85.161.176] (HELO mail-gx0-f176.google.com) (209.85.161.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 20:57:29 +0000
Received: by gxk7 with SMTP id 7so2558308gxk.35
        for <general@hadoop.apache.org>; Mon, 20 Jun 2011 13:57:08 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=Gw7QO7906E6OYmtuXEgKlR+iX6ApfiFAuZKcFLpmgXc=;
        b=Ked3sGLU9J/fJ9Z7pRn6a+ftBdcpi7rtDIyxrfA5tlroYmm0tTrzl7PChGlABuS6Qq
         2PJzDqwlSZLCT9Gv/NC2LQq35ScWdepe4+at5J3StV5SiykZrWKxt80e5IqNwTdeRF/m
         Ml3px8+Yi7BA9QmvcsIuo3NMRcQhhRATru4R8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=hzDHCknAHCabMR3zPIZ6KtV1LUQuxAYLFRSnlwKQmWv7CK9DPJYlC7uzthOu/DbBP0
         fv5/FnwMQUU8nKKDugUlaJpqw0JcSmHQgkFw5IsGofV8yJw4ygIe2YTTUts8sMzgBmvK
         4SF1mdItOJJ3tIf/jsreEuWCZ7SaDPTOwFq4U=
MIME-Version: 1.0
Received: by 10.236.154.201 with SMTP id h49mr8862110yhk.119.1308603428553;
 Mon, 20 Jun 2011 13:57:08 -0700 (PDT)
Sender: vicaya@gmail.com
Received: by 10.236.111.1 with HTTP; Mon, 20 Jun 2011 13:57:08 -0700 (PDT)
In-Reply-To: <BANLkTi=49ReNeyYeZfR3YUQaT=difjGHzA@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
	<BANLkTinwpw70j_ZvBkahShguZJOGKoR62g@mail.gmail.com>
	<BANLkTi=49ReNeyYeZfR3YUQaT=difjGHzA@mail.gmail.com>
Date: Mon, 20 Jun 2011 13:57:08 -0700
X-Google-Sender-Auth: oDwzRiq62ZjHMBF57WYjkbtKk_w
Message-ID: <BANLkTik2X4TBHs7Nh_F8H8J-Usvqj+k7jw@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: Luke Lu <llu@vicaya.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Mon, Jun 20, 2011 at 1:43 PM, web service <wbsrvc@gmail.com> wrote:
> Can we still submit logos ?

I encourage everyone who care about the logo to attach submissions to
HADOOP-7020 (make sure you click grant rights to apache in the attach
files form). As Ted said, the logos here are mostly thematic. The
final version would be tweaked by a professional designer with
input/ideas from all the submissions.

> On Mon, Jun 20, 2011 at 12:57 PM, Luke Lu <llu@vicaya.com> wrote:
>
>> Just got a few minutes to look at these. I must say that I'm very
>> disappointed. Some of the choices are nice clip art but none of them
>> are appropriate for powered-by logos at typical logo size: <=3D64 pixels
>> in height, where they are completely illegible.
>>
>> I present choice 0, which IMHO is a better candidate than the rest of
>> the choices:
>>
>> https://issues.apache.org/jira/secure/attachment/12483197/pbh-64-logos.p=
ng
>>
>> Please find the larger version on
>> https://issues.apache.org/jira/browse/HADOOP-7020
>>
>> __Luke
>>
>> On Tue, Jun 14, 2011 at 8:19 PM, Owen O'Malley <omalley@apache.org> wrot=
e:
>> > All,
>> > =A0 We've had a wide range of entries for a powered by logo. I've put =
them
>> all on a page, here:
>> >
>> > http://people.apache.org/~omalley/hadoop-powered-by/
>> >
>> > Since there are a lot of contenders and we only want a single round of
>> voting, let's use single transferable vote ( STV
>> http://en.wikipedia.org/wiki/Single_transferable_vote). The important
>> thing is to pick the images *IN ORDER* that you would like them.
>> >
>> > My vote (in order of course): 4, 1, 2, 3, 5, 6.
>> >
>> > In other words, I want option 4 most and option 6 least. With STV, you
>> don't need to worry about voting for an unpopular choice since your vote
>> will automatically roll over to your next choice.
>> >
>> > -- Owen
>> >
>> >
>> >
>>
>

From general-return-3897-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 21:13:06 2011
Return-Path: <general-return-3897-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4407E6574
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 21:13:06 +0000 (UTC)
Received: (qmail 19464 invoked by uid 500); 20 Jun 2011 21:13:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19407 invoked by uid 500); 20 Jun 2011 21:13:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19399 invoked by uid 99); 20 Jun 2011 21:13:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 21:13:04 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.220.176] (HELO mail-vx0-f176.google.com) (209.85.220.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 21:12:56 +0000
Received: by vxd2 with SMTP id 2so3603756vxd.35
        for <general@hadoop.apache.org>; Mon, 20 Jun 2011 14:12:35 -0700 (PDT)
Received: by 10.52.179.130 with SMTP id dg2mr4599540vdc.203.1308604355065;
 Mon, 20 Jun 2011 14:12:35 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.52.111.166 with HTTP; Mon, 20 Jun 2011 14:12:15 -0700 (PDT)
X-Originating-IP: [64.105.168.204]
In-Reply-To: <BANLkTik2X4TBHs7Nh_F8H8J-Usvqj+k7jw@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
 <BANLkTinwpw70j_ZvBkahShguZJOGKoR62g@mail.gmail.com> <BANLkTi=49ReNeyYeZfR3YUQaT=difjGHzA@mail.gmail.com>
 <BANLkTik2X4TBHs7Nh_F8H8J-Usvqj+k7jw@mail.gmail.com>
From: Ted Dunning <tdunning@maprtech.com>
Date: Mon, 20 Jun 2011 21:12:15 +0000
Message-ID: <BANLkTikTJQ8mv_4YcLZ4rDwHL5c-q-W+cQ@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec5171ecdbc369904a62b3173
X-Virus-Checked: Checked by ClamAV on apache.org

--bcaec5171ecdbc369904a62b3173
Content-Type: text/plain; charset=ISO-8859-1

But keep in mind that creating new logos forever will tend to prevent
consensus being formed.  That will result in the same result as no decision
at all (i.e. everybody will just use their own).

Owen already asked for votes on the original 6.  The only substantive
difference in the new logo is the addition of the feature (and the
esthetic/political sub-text issue of putting the feather in the elephants
nose).  Font size can and will be adjusted.  That adjustment will be done by
our art team in MapR if the circle logo concept is the accepted consensus.

My own view here is that consensus is at least as important as the precise
content.  Adding new options tends to value small details in content over
consensus.  I would prefer to not muddy the issue further.

On Mon, Jun 20, 2011 at 8:57 PM, Luke Lu <llu@vicaya.com> wrote:

> On Mon, Jun 20, 2011 at 1:43 PM, web service <wbsrvc@gmail.com> wrote:
> > Can we still submit logos ?
>
> I encourage everyone who care about the logo to attach submissions to
> HADOOP-7020 (make sure you click grant rights to apache in the attach
> files form). As Ted said, the logos here are mostly thematic. The
> final version would be tweaked by a professional designer with
> input/ideas from all the submissions.
>
> > On Mon, Jun 20, 2011 at 12:57 PM, Luke Lu <llu@vicaya.com> wrote:
> >
> >> Just got a few minutes to look at these. I must say that I'm very
> >> disappointed. Some of the choices are nice clip art but none of them
> >> are appropriate for powered-by logos at typical logo size: <=64 pixels
> >> in height, where they are completely illegible.
> >>
> >> I present choice 0, which IMHO is a better candidate than the rest of
> >> the choices:
> >>
> >>
> https://issues.apache.org/jira/secure/attachment/12483197/pbh-64-logos.png
> >>
> >> Please find the larger version on
> >> https://issues.apache.org/jira/browse/HADOOP-7020
> >>
> >> __Luke
> >>
> >> On Tue, Jun 14, 2011 at 8:19 PM, Owen O'Malley <omalley@apache.org>
> wrote:
> >> > All,
> >> >   We've had a wide range of entries for a powered by logo. I've put
> them
> >> all on a page, here:
> >> >
> >> > http://people.apache.org/~omalley/hadoop-powered-by/
> >> >
> >> > Since there are a lot of contenders and we only want a single round of
> >> voting, let's use single transferable vote ( STV
> >> http://en.wikipedia.org/wiki/Single_transferable_vote). The important
> >> thing is to pick the images *IN ORDER* that you would like them.
> >> >
> >> > My vote (in order of course): 4, 1, 2, 3, 5, 6.
> >> >
> >> > In other words, I want option 4 most and option 6 least. With STV, you
> >> don't need to worry about voting for an unpopular choice since your vote
> >> will automatically roll over to your next choice.
> >> >
> >> > -- Owen
> >> >
> >> >
> >> >
> >>
> >
>

--bcaec5171ecdbc369904a62b3173--

From general-return-3898-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 21:19:15 2011
Return-Path: <general-return-3898-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C8CA4671B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 21:19:15 +0000 (UTC)
Received: (qmail 40070 invoked by uid 500); 20 Jun 2011 21:19:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40012 invoked by uid 500); 20 Jun 2011 21:19:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40004 invoked by uid 99); 20 Jun 2011 21:19:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 21:19:14 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 21:19:08 +0000
Received: by wyb36 with SMTP id 36so1392094wyb.35
        for <general@hadoop.apache.org>; Mon, 20 Jun 2011 14:18:47 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.217.7.3 with SMTP id z3mr5174722wes.68.1308604727310; Mon, 20
 Jun 2011 14:18:47 -0700 (PDT)
Received: by 10.216.30.212 with HTTP; Mon, 20 Jun 2011 14:18:47 -0700 (PDT)
In-Reply-To: <BANLkTikTJQ8mv_4YcLZ4rDwHL5c-q-W+cQ@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
	<BANLkTinwpw70j_ZvBkahShguZJOGKoR62g@mail.gmail.com>
	<BANLkTi=49ReNeyYeZfR3YUQaT=difjGHzA@mail.gmail.com>
	<BANLkTik2X4TBHs7Nh_F8H8J-Usvqj+k7jw@mail.gmail.com>
	<BANLkTikTJQ8mv_4YcLZ4rDwHL5c-q-W+cQ@mail.gmail.com>
Date: Mon, 20 Jun 2011 14:18:47 -0700
Message-ID: <BANLkTimk3ESmhRptnt=qy_Zw84V1aFkdpg@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Mon, Jun 20, 2011 at 2:12 PM, Ted Dunning <tdunning@maprtech.com> wrote:
> But keep in mind that creating new logos forever will tend to prevent
> consensus being formed. =A0That will result in the same result as no deci=
sion
> at all (i.e. everybody will just use their own).
>
> Owen already asked for votes on the original 6. =A0The only substantive
> difference in the new logo is the addition of the feature (and the
> esthetic/political sub-text issue of putting the feather in the elephants
> nose). =A0Font size can and will be adjusted. =A0That adjustment will be =
done by
> our art team in MapR if the circle logo concept is the accepted consensus=
.
>

If the circle logo concept is accepted it would be good to adjust the
circle itself, I noticed that the circle in your logo submission is
the same design element used throughout MapR.com.

Thanks,
Eli

From general-return-3899-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 21:31:12 2011
Return-Path: <general-return-3899-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7D87A6535
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 21:31:12 +0000 (UTC)
Received: (qmail 63135 invoked by uid 500); 20 Jun 2011 21:31:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63078 invoked by uid 500); 20 Jun 2011 21:31:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63070 invoked by uid 99); 20 Jun 2011 21:31:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 21:31:10 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jghoman@gmail.com designates 74.125.83.176 as permitted sender)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 21:31:05 +0000
Received: by pve37 with SMTP id 37so3826007pve.35
        for <general@hadoop.apache.org>; Mon, 20 Jun 2011 14:30:43 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type:content-transfer-encoding;
        bh=pqLDIMVmUxb056EoVAXDreDGH3bNlq7iwfX0X4F4uWg=;
        b=QJ6VJZMIX2hgysllkTR82600CHMI6LOsvjsTG79AA7GsshiIwjB4z0Op9Xm0OFos0l
         TGCd4KDx8cRZ/WQhRC7I3kimR26TengoY3o4Rv0S3S5o0PY1sWK7Mt5RvuTffUZ9pmAn
         DnfL8XsIgCqozqSndOsImqkRCSy5brPk8BWwI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=ar6Wu+AUVgpFIaMj2Mq4Qb1zIS+a9H7NIdAbVVjuTGWfILJTpAyyvtHy3nvt511IWu
         VtlTX2IurXzc/HriOZQRONyoODWh5AFJwSjUZuujxTngqti2gcrjfCMfFzkiuXFveJ6p
         j1rVfdqMGpc+1Vmqu6M//fc5V6AziZ8E8jcH4=
Received: by 10.142.4.10 with SMTP id 10mr482357wfd.361.1308605443508; Mon, 20
 Jun 2011 14:30:43 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.143.92.11 with HTTP; Mon, 20 Jun 2011 14:30:13 -0700 (PDT)
In-Reply-To: <BANLkTimk3ESmhRptnt=qy_Zw84V1aFkdpg@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
 <BANLkTinwpw70j_ZvBkahShguZJOGKoR62g@mail.gmail.com> <BANLkTi=49ReNeyYeZfR3YUQaT=difjGHzA@mail.gmail.com>
 <BANLkTik2X4TBHs7Nh_F8H8J-Usvqj+k7jw@mail.gmail.com> <BANLkTikTJQ8mv_4YcLZ4rDwHL5c-q-W+cQ@mail.gmail.com>
 <BANLkTimk3ESmhRptnt=qy_Zw84V1aFkdpg@mail.gmail.com>
From: Jakob Homan <jghoman@gmail.com>
Date: Mon, 20 Jun 2011 14:30:13 -0700
Message-ID: <BANLkTin7KRS4+b6Xnh7Ng4k6PznwSYTzhg@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Whatever gets picked, I volunteer to have some stickers printed up so
we can all put them on our bikesheds.  Seems appropriate.

On Mon, Jun 20, 2011 at 2:18 PM, Eli Collins <eli@cloudera.com> wrote:
> On Mon, Jun 20, 2011 at 2:12 PM, Ted Dunning <tdunning@maprtech.com> wrot=
e:
>> But keep in mind that creating new logos forever will tend to prevent
>> consensus being formed. =A0That will result in the same result as no dec=
ision
>> at all (i.e. everybody will just use their own).
>>
>> Owen already asked for votes on the original 6. =A0The only substantive
>> difference in the new logo is the addition of the feature (and the
>> esthetic/political sub-text issue of putting the feather in the elephant=
s
>> nose). =A0Font size can and will be adjusted. =A0That adjustment will be=
 done by
>> our art team in MapR if the circle logo concept is the accepted consensu=
s.
>>
>
> If the circle logo concept is accepted it would be good to adjust the
> circle itself, I noticed that the circle in your logo submission is
> the same design element used throughout MapR.com.
>
> Thanks,
> Eli
>

From general-return-3900-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 21:43:08 2011
Return-Path: <general-return-3900-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 716AE66CB
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 21:43:08 +0000 (UTC)
Received: (qmail 79418 invoked by uid 500); 20 Jun 2011 21:43:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79351 invoked by uid 500); 20 Jun 2011 21:43:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 79343 invoked by uid 99); 20 Jun 2011 21:43:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 21:43:06 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of vicaya@gmail.com designates 209.85.213.176 as permitted sender)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 21:43:00 +0000
Received: by yxd5 with SMTP id 5so3552032yxd.35
        for <general@hadoop.apache.org>; Mon, 20 Jun 2011 14:42:40 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=/wZMfD7rAEP0L4IGtgEpJjTObNorYBFOdSfnZmTKrgU=;
        b=xFhsIDNGBZb8n0Rir8iJs0GepW8HBPfmy+5D33u7MwSLkAESMnziS1+GUw2iglEk7z
         zwzIoMQDsn6raj77rRI9WuYiaBKI9SBEthY1+ch1GgJ460w/E0H8Tq7M5oK7F7YnDDfJ
         pvdgpxwd9udeL+xli5SV9o4XCw1QvkojvlX3I=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=cTZo+ox4ATiXLsmRCTvRoqnmpcTmTtB3yUmarpCjXybMsn3SQDuChqSHf4703/0/qZ
         vlBREQf1FFBeMk0PPetHUwAKLt8Cen+pHHqx+ikvgFrWhkVOlzZutLY56KoRvpNl+OcU
         OjjzK89mnvDfbdSVRfBiA9mhrrRtS2pWP7UyI=
MIME-Version: 1.0
Received: by 10.236.28.2 with SMTP id f2mr6570257yha.387.1308606159725; Mon,
 20 Jun 2011 14:42:39 -0700 (PDT)
Sender: vicaya@gmail.com
Received: by 10.236.111.1 with HTTP; Mon, 20 Jun 2011 14:42:39 -0700 (PDT)
In-Reply-To: <BANLkTikTJQ8mv_4YcLZ4rDwHL5c-q-W+cQ@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
	<BANLkTinwpw70j_ZvBkahShguZJOGKoR62g@mail.gmail.com>
	<BANLkTi=49ReNeyYeZfR3YUQaT=difjGHzA@mail.gmail.com>
	<BANLkTik2X4TBHs7Nh_F8H8J-Usvqj+k7jw@mail.gmail.com>
	<BANLkTikTJQ8mv_4YcLZ4rDwHL5c-q-W+cQ@mail.gmail.com>
Date: Mon, 20 Jun 2011 14:42:39 -0700
X-Google-Sender-Auth: AQzNQrMkswW8NShqOmDfTxUMMTM
Message-ID: <BANLkTinXguuoTrV1ego5vMAbXYMLn9krBA@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: Luke Lu <llu@vicaya.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Mon, Jun 20, 2011 at 2:12 PM, Ted Dunning <tdunning@maprtech.com> wrote:
> But keep in mind that creating new logos forever will tend to prevent
> consensus being formed. =A0That will result in the same result as no deci=
sion
> at all (i.e. everybody will just use their own).

The the binding votes are from limited number of PMC members. I see no
obstacle reaching consensus even if we have much larger number of
selections.

> Owen already asked for votes on the original 6. =A0The only substantive
> difference in the new logo is the addition of the feature (and the
> esthetic/political sub-text issue of putting the feather in the elephants
> nose).

Elephant's nose is a versatile hand. What political sub-text are you
talking about? I see no problem with an elephant upholding an apache
feather. OTOH, feather in hand/nose is not the most substantive
element rather than playful variation. The key element is the theme of
apache and hadoop bigger than the proverbial elephant.

> =A0Font size can and will be adjusted. =A0That adjustment will be done by
> our art team in MapR if the circle logo concept is the accepted consensus=
.

Since you eluded aesthetic and political sub-text, the MapR circle is
problematic as well, which IMO looks like a plastic button or a
deformed mouse wheel confining the elephant.

> My own view here is that consensus is at least as important as the precis=
e
> content. =A0Adding new options tends to value small details in content ov=
er
> consensus. =A0I would prefer to not muddy the issue further.

Again, I see no problem reaching a consensus here, given the limited
number of binding votes.

> On Mon, Jun 20, 2011 at 8:57 PM, Luke Lu <llu@vicaya.com> wrote:
>
>> On Mon, Jun 20, 2011 at 1:43 PM, web service <wbsrvc@gmail.com> wrote:
>> > Can we still submit logos ?
>>
>> I encourage everyone who care about the logo to attach submissions to
>> HADOOP-7020 (make sure you click grant rights to apache in the attach
>> files form). As Ted said, the logos here are mostly thematic. The
>> final version would be tweaked by a professional designer with
>> input/ideas from all the submissions.
>>
>> > On Mon, Jun 20, 2011 at 12:57 PM, Luke Lu <llu@vicaya.com> wrote:
>> >
>> >> Just got a few minutes to look at these. I must say that I'm very
>> >> disappointed. Some of the choices are nice clip art but none of them
>> >> are appropriate for powered-by logos at typical logo size: <=3D64 pix=
els
>> >> in height, where they are completely illegible.
>> >>
>> >> I present choice 0, which IMHO is a better candidate than the rest of
>> >> the choices:
>> >>
>> >>
>> https://issues.apache.org/jira/secure/attachment/12483197/pbh-64-logos.p=
ng
>> >>
>> >> Please find the larger version on
>> >> https://issues.apache.org/jira/browse/HADOOP-7020
>> >>
>> >> __Luke
>> >>
>> >> On Tue, Jun 14, 2011 at 8:19 PM, Owen O'Malley <omalley@apache.org>
>> wrote:
>> >> > All,
>> >> > =A0 We've had a wide range of entries for a powered by logo. I've p=
ut
>> them
>> >> all on a page, here:
>> >> >
>> >> > http://people.apache.org/~omalley/hadoop-powered-by/
>> >> >
>> >> > Since there are a lot of contenders and we only want a single round=
 of
>> >> voting, let's use single transferable vote ( STV
>> >> http://en.wikipedia.org/wiki/Single_transferable_vote). The important
>> >> thing is to pick the images *IN ORDER* that you would like them.
>> >> >
>> >> > My vote (in order of course): 4, 1, 2, 3, 5, 6.
>> >> >
>> >> > In other words, I want option 4 most and option 6 least. With STV, =
you
>> >> don't need to worry about voting for an unpopular choice since your v=
ote
>> >> will automatically roll over to your next choice.
>> >> >
>> >> > -- Owen
>> >> >
>> >> >
>> >> >
>> >>
>> >
>>
>

From general-return-3901-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 22:20:22 2011
Return-Path: <general-return-3901-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A559061B9
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 22:20:22 +0000 (UTC)
Received: (qmail 37872 invoked by uid 500); 20 Jun 2011 22:20:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37811 invoked by uid 500); 20 Jun 2011 22:20:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37803 invoked by uid 99); 20 Jun 2011 22:20:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 22:20:21 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 22:20:14 +0000
Received: by vws7 with SMTP id 7so2255720vws.35
        for <general@hadoop.apache.org>; Mon, 20 Jun 2011 15:19:53 -0700 (PDT)
Received: by 10.52.113.170 with SMTP id iz10mr2301456vdb.123.1308608393040;
 Mon, 20 Jun 2011 15:19:53 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.52.111.166 with HTTP; Mon, 20 Jun 2011 15:19:33 -0700 (PDT)
X-Originating-IP: [64.105.168.204]
In-Reply-To: <BANLkTimk3ESmhRptnt=qy_Zw84V1aFkdpg@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
 <BANLkTinwpw70j_ZvBkahShguZJOGKoR62g@mail.gmail.com> <BANLkTi=49ReNeyYeZfR3YUQaT=difjGHzA@mail.gmail.com>
 <BANLkTik2X4TBHs7Nh_F8H8J-Usvqj+k7jw@mail.gmail.com> <BANLkTikTJQ8mv_4YcLZ4rDwHL5c-q-W+cQ@mail.gmail.com>
 <BANLkTimk3ESmhRptnt=qy_Zw84V1aFkdpg@mail.gmail.com>
From: Ted Dunning <tdunning@maprtech.com>
Date: Mon, 20 Jun 2011 22:19:33 +0000
Message-ID: <BANLkTimeFSM-n31=H+6WHoTB1HTa0krXXQ@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec54867446ad4b604a62c2299
X-Virus-Checked: Checked by ClamAV on apache.org

--bcaec54867446ad4b604a62c2299
Content-Type: text/plain; charset=ISO-8859-1

?

Are you talking about color palette choices?  We use a pretty ordinary
corporate blue based palette which inevitably makes gray tones important.

Or the fact that we put circular borders around badge-like design element
like this: http://mapr.com/images/products/dependable/btn_ha.png ?

I don't think that we have any isolated rings such as the proposed logo on
mapr.com at this time.  We would like to (i.e. if this logo gets approved).

How would you suggest adjusting the circle?

On Mon, Jun 20, 2011 at 9:18 PM, Eli Collins <eli@cloudera.com> wrote:

> On Mon, Jun 20, 2011 at 2:12 PM, Ted Dunning <tdunning@maprtech.com>
> wrote:
> > But keep in mind that creating new logos forever will tend to prevent
> > consensus being formed.  That will result in the same result as no
> decision
> > at all (i.e. everybody will just use their own).
> >
> > Owen already asked for votes on the original 6.  The only substantive
> > difference in the new logo is the addition of the feature (and the
> > esthetic/political sub-text issue of putting the feather in the elephants
> > nose).  Font size can and will be adjusted.  That adjustment will be done
> by
> > our art team in MapR if the circle logo concept is the accepted
> consensus.
> >
>
> If the circle logo concept is accepted it would be good to adjust the
> circle itself, I noticed that the circle in your logo submission is
> the same design element used throughout MapR.com.
>
> Thanks,
> Eli
>

--bcaec54867446ad4b604a62c2299--

From general-return-3902-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 22:44:25 2011
Return-Path: <general-return-3902-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 545996EB4
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 20 Jun 2011 22:44:25 +0000 (UTC)
Received: (qmail 85412 invoked by uid 500); 20 Jun 2011 22:44:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85357 invoked by uid 500); 20 Jun 2011 22:44:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85349 invoked by uid 99); 20 Jun 2011 22:44:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 22:44:23 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of eli@cloudera.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 22:44:19 +0000
Received: by wwi18 with SMTP id 18so5406699wwi.29
        for <general@hadoop.apache.org>; Mon, 20 Jun 2011 15:43:58 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.61.8 with SMTP id v8mr1494751wec.83.1308609838074; Mon, 20
 Jun 2011 15:43:58 -0700 (PDT)
Received: by 10.216.30.212 with HTTP; Mon, 20 Jun 2011 15:43:57 -0700 (PDT)
In-Reply-To: <BANLkTimeFSM-n31=H+6WHoTB1HTa0krXXQ@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
	<BANLkTinwpw70j_ZvBkahShguZJOGKoR62g@mail.gmail.com>
	<BANLkTi=49ReNeyYeZfR3YUQaT=difjGHzA@mail.gmail.com>
	<BANLkTik2X4TBHs7Nh_F8H8J-Usvqj+k7jw@mail.gmail.com>
	<BANLkTikTJQ8mv_4YcLZ4rDwHL5c-q-W+cQ@mail.gmail.com>
	<BANLkTimk3ESmhRptnt=qy_Zw84V1aFkdpg@mail.gmail.com>
	<BANLkTimeFSM-n31=H+6WHoTB1HTa0krXXQ@mail.gmail.com>
Date: Mon, 20 Jun 2011 15:43:57 -0700
Message-ID: <BANLkTimErD6vNEiDbOmpnwSdi+q7+7hesA@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Mon, Jun 20, 2011 at 3:19 PM, Ted Dunning <tdunning@maprtech.com> wrote:
> ?
>
> Are you talking about color palette choices? =A0We use a pretty ordinary
> corporate blue based palette which inevitably makes gray tones important.
>
> Or the fact that we put circular borders around badge-like design element
> like this: http://mapr.com/images/products/dependable/btn_ha.png ?
>

Yup.

> I don't think that we have any isolated rings such as the proposed logo o=
n
> mapr.com at this time. =A0We would like to (i.e. if this logo gets approv=
ed).
>
> How would you suggest adjusting the circle?
>

Would be good for it not to re-use the same circular border, ie the
design elements should be visually distinct. Assume the artist just
used the existing design element they had and wrapped it around the
Hadoop logo vs creating a new one.

Thanks,
Eli


> On Mon, Jun 20, 2011 at 9:18 PM, Eli Collins <eli@cloudera.com> wrote:
>
>> On Mon, Jun 20, 2011 at 2:12 PM, Ted Dunning <tdunning@maprtech.com>
>> wrote:
>> > But keep in mind that creating new logos forever will tend to prevent
>> > consensus being formed. =A0That will result in the same result as no
>> decision
>> > at all (i.e. everybody will just use their own).
>> >
>> > Owen already asked for votes on the original 6. =A0The only substantiv=
e
>> > difference in the new logo is the addition of the feature (and the
>> > esthetic/political sub-text issue of putting the feather in the elepha=
nts
>> > nose). =A0Font size can and will be adjusted. =A0That adjustment will =
be done
>> by
>> > our art team in MapR if the circle logo concept is the accepted
>> consensus.
>> >
>>
>> If the circle logo concept is accepted it would be good to adjust the
>> circle itself, I noticed that the circle in your logo submission is
>> the same design element used throughout MapR.com.
>>
>> Thanks,
>> Eli
>>
>

From general-return-3903-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 11:28:14 2011
Return-Path: <general-return-3903-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 145BD48D5
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 21 Jun 2011 11:28:14 +0000 (UTC)
Received: (qmail 40382 invoked by uid 500); 21 Jun 2011 11:28:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40317 invoked by uid 500); 21 Jun 2011 11:28:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40309 invoked by uid 99); 21 Jun 2011 11:28:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 11:28:11 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 11:28:03 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id BF2F7B809C
	for <general@hadoop.apache.org>; Tue, 21 Jun 2011 12:27:41 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id QMvY+yJjFGBv for <general@hadoop.apache.org>;
	Tue, 21 Jun 2011 12:27:41 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 1E469B80B3
	for <general@hadoop.apache.org>; Tue, 21 Jun 2011 12:27:37 +0100 (BST)
MailScanner-NULL-Check: 1309260443.91675@5t0u0jyjgKeAV5smAO2fxw
Received: from [16.24.213.88] ([16.24.213.88])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p5LBRNED019846
	for <general@hadoop.apache.org>; Tue, 21 Jun 2011 12:27:23 +0100 (BST)
Message-ID: <4E008016.9010009@apache.org>
Date: Tue, 21 Jun 2011 12:27:18 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.2.17) Gecko/20110414 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Thinking about the next hadoop mainline release
References: <EFF679DA-26AB-4250-8F59-A2330D0E3E5C@yahoo-inc.com> <F77C0A6C-92AC-4486-8326-589D1D297F9A@apache.org> <4C60EEFE-1183-47DD-9E2E-EB6EFF589727@apache.org>
In-Reply-To: <4C60EEFE-1183-47DD-9E2E-EB6EFF589727@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p5LBRNED019846
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 17/06/2011 21:27, Allen Wittenauer wrote:
>
> On Jun 17, 2011, at 10:31 AM, Allen Wittenauer wrote:
>
>>
>> On Jun 17, 2011, at 12:17 AM, Eric Baldeschwieler wrote:
>>> Yahoo stands ready to help us (the Apache Hadoop Community) turn this new release into a stable release by running it through its 9 month test and burn in process.  The result of that will be another stable release such as 0.18, 0.20 or 0.20.203 (hadoop with security).
>>
>>
>> 	I'd consider 0.20.203 pseudo-stable.
>
> 	Actually, I was just reminded about the complete disaster that is metrics.  So while it may be pseudo-stable, it  isn't actually usable for anyone but Yahoo!.

ooh, I have work on that that I should catch up with and get in.

I'd also like to put in the IBM JVM patch, not out of personal need, but 
because I don't like Oracle-JVM dependencies, and it'd be nice to get 
IBM's donation into the source tree.

From general-return-3904-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 11:39:58 2011
Return-Path: <general-return-3904-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E44BB4259
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 21 Jun 2011 11:39:57 +0000 (UTC)
Received: (qmail 74528 invoked by uid 500); 21 Jun 2011 11:39:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74477 invoked by uid 500); 21 Jun 2011 11:39:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74448 invoked by uid 99); 21 Jun 2011 11:39:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 11:39:56 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 11:39:46 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 9C03B1BA914
	for <general@hadoop.apache.org>; Tue, 21 Jun 2011 12:39:25 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id 5TST5NRzmptG for <general@hadoop.apache.org>;
	Tue, 21 Jun 2011 12:39:25 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 1E1951BA8D6
	for <general@hadoop.apache.org>; Tue, 21 Jun 2011 12:39:24 +0100 (BST)
MailScanner-NULL-Check: 1309261150.7834@rpS+vPppXzKfT0lFPRn7Ww
Received: from [16.24.213.88] ([16.24.213.88])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p5LBdANS020209
	for <general@hadoop.apache.org>; Tue, 21 Jun 2011 12:39:10 +0100 (BST)
Message-ID: <4E0082D9.5090607@apache.org>
Date: Tue, 21 Jun 2011 12:39:05 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.2.17) Gecko/20110414 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSSION] Thinking about 20.204 and beyond
References: <4E18CE25-6863-4318-8A95-17896B0B87D1@yahoo-inc.com> <0C4895F3-95E8-45ED-B2B0-C3BF4B5436B1@gbiv.com>
In-Reply-To: <0C4895F3-95E8-45ED-B2B0-C3BF4B5436B1@gbiv.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p5LBdANS020209
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 18/06/2011 21:22, Roy T. Fielding wrote:
olutely no reason that trunk cannot be packaged for release
> tomorrow as 0.23.  There may be many reasons why it won't pass a release
> vote, but we probably aren't going to find them until somebody tries.
>

One limitation with releases has always been size of cluster testing 
-where Yahoo!s contributions have been invaluable. That said, we 
shouldn't make them an SPOF in the release process; we should all set up 
to do some more release testing. Looking round my house I see that I 
have 1 linux desktop, 4 linux laptops (All Ubuntu 10.04) , 2 OS/X 
machines and an OLPC, plus three WebOS phones that can take a JVM, all 
connected via a netgear N600 router, (ignoring CentOS virtual machine 
images). If I can actually bring up a heterogenous cluster here then it 
would be a contribution, and I'm sure others have equal messes of home 
and work systems that could be used to test bad system configurations. 
If successful we can add a netgear 802.11 {a, b, g, n} router to the 
list of switches known to work once you get DNS right.

What would be good here is more documentation on how the beta testers 
can generate realistic loads to stress our clusters, using the stress 
test tools that are already in the codebase. I guess I may start on that 
if/when I start playing with the tools.

-Steve

From general-return-3905-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 18:40:46 2011
Return-Path: <general-return-3905-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 67CB64D70
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 21 Jun 2011 18:40:46 +0000 (UTC)
Received: (qmail 13719 invoked by uid 500); 21 Jun 2011 18:40:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13671 invoked by uid 500); 21 Jun 2011 18:40:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13663 invoked by uid 99); 21 Jun 2011 18:40:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 18:40:44 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [208.97.132.145] (HELO homiemail-a73.g.dreamhost.com) (208.97.132.145)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 18:40:37 +0000
Received: from homiemail-a73.g.dreamhost.com (localhost [127.0.0.1])
	by homiemail-a73.g.dreamhost.com (Postfix) with ESMTP id 365E51F008B
	for <general@hadoop.apache.org>; Tue, 21 Jun 2011 11:40:15 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; q=dns; s=
	gbiv.com; b=OrUctrTVve1bdoExAngmbunXPegJbciDjor9EoDNVHNe/OpwhCWh
	KPDEXDvTyyjTf3aInvKl2oDGG5ynhpuU8pTuNAMrco38imbpgLn25ZSKCkgF0Zam
	m7QRV7zVm1yBtlmItzhP2BpAF9fJ5VvA/N6Ze7MjJx18jY2JoT1v1lE=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; s=gbiv.com;
	 bh=QLXsnauWYv5ltnr7YzXJUMpG0Fk=; b=G3y3ij8DOGaQzQNtE9U6DJDv5y0P
	9rbHPliNKicUb1feOXw3wc69OrU/alo7YhDG/yv5VHU6oFrTtw9kMVngAOG+eBg6
	RdeNh9p3B74DQtkteRu3NPVcMds1jvu2ohEtEFn4UdRphOe9IPQT2if0fXYa2FML
	BN9qot+JDvZmO4o=
Received: from [192.168.1.84] (99-21-208-82.lightspeed.irvnca.sbcglobal.net [99.21.208.82])
	(using TLSv1 with cipher AES128-SHA (128/128 bits))
	(No client certificate requested)
	(Authenticated sender: fielding@gbiv.com)
	by homiemail-a73.g.dreamhost.com (Postfix) with ESMTPSA id D35471F0084
	for <general@hadoop.apache.org>; Tue, 21 Jun 2011 11:40:14 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: [DISCUSSION] Thinking about 20.204 and beyond
From: "Roy T. Fielding" <fielding@gbiv.com>
In-Reply-To: <4E0082D9.5090607@apache.org>
Date: Tue, 21 Jun 2011 11:40:14 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <D32D716F-308A-41C1-844B-08B99D2443D0@gbiv.com>
References: <4E18CE25-6863-4318-8A95-17896B0B87D1@yahoo-inc.com> <0C4895F3-95E8-45ED-B2B0-C3BF4B5436B1@gbiv.com> <4E0082D9.5090607@apache.org>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

On Jun 21, 2011, at 4:39 AM, Steve Loughran wrote:

> On 18/06/2011 21:22, Roy T. Fielding wrote:
> olutely no reason that trunk cannot be packaged for release
>> tomorrow as 0.23.  There may be many reasons why it won't pass a =
release
>> vote, but we probably aren't going to find them until somebody tries.
>=20
> One limitation with releases has always been size of cluster testing =
-where Yahoo!s contributions have been invaluable. That said, we =
shouldn't make them an SPOF in the release process; we should all set up =
to do some more release testing.

Yes, more testing is better, but if it can't be tested by the dev team
in 72 hours then it doesn't belong in our release process.

Please note that one of the main advantages of open source development
is that the bulk of testing/QA occurs *after* the release.  That's why
labels like alpha/beta/GA are best applied/updated after the version =
number
has been cut and the software has been proven in real deployments.
If testing on 5000 nodes is important to our customers, then add a
scale-tested metric to the download site so that the customers know
which release package has been tested at what scale -- they will
understand the difference between frequent releases and those fully =
tested
at scale.  Let them decide which version is best to use for their own =
needs.

....Roy


From general-return-3906-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 18:43:47 2011
Return-Path: <general-return-3906-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1CC454EBD
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 21 Jun 2011 18:43:47 +0000 (UTC)
Received: (qmail 19348 invoked by uid 500); 21 Jun 2011 18:43:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19295 invoked by uid 500); 21 Jun 2011 18:43:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19287 invoked by uid 99); 21 Jun 2011 18:43:45 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 18:43:45 +0000
Received: from localhost (HELO awittena-md.linkedin.biz) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 18:43:45 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [DISCUSSION] Thinking about 20.204 and beyond
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <4E0082D9.5090607@apache.org>
Date: Tue, 21 Jun 2011 11:43:42 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <82A248C7-D17B-4AD3-A8B2-75F91E89F600@apache.org>
References: <4E18CE25-6863-4318-8A95-17896B0B87D1@yahoo-inc.com> <0C4895F3-95E8-45ED-B2B0-C3BF4B5436B1@gbiv.com> <4E0082D9.5090607@apache.org>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On Jun 21, 2011, at 4:39 AM, Steve Loughran wrote:
>  If I can actually bring up a heterogenous cluster here

 	I believe there was a post in one of the mailing lists in the =
past 6 months where someone tried a mixed endian grid.  It blew up big =
time.=

From general-return-3907-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 19:05:05 2011
Return-Path: <general-return-3907-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B7E594D1F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 21 Jun 2011 19:05:05 +0000 (UTC)
Received: (qmail 66307 invoked by uid 500); 21 Jun 2011 19:05:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66261 invoked by uid 500); 21 Jun 2011 19:05:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66252 invoked by uid 99); 21 Jun 2011 19:05:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 19:05:04 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.220.176] (HELO mail-vx0-f176.google.com) (209.85.220.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 19:04:56 +0000
Received: by vxd2 with SMTP id 2so91024vxd.35
        for <general@hadoop.apache.org>; Tue, 21 Jun 2011 12:04:35 -0700 (PDT)
Received: by 10.52.115.130 with SMTP id jo2mr222131vdb.43.1308683075080; Tue,
 21 Jun 2011 12:04:35 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.52.111.166 with HTTP; Tue, 21 Jun 2011 12:04:15 -0700 (PDT)
X-Originating-IP: [64.105.168.204]
In-Reply-To: <82A248C7-D17B-4AD3-A8B2-75F91E89F600@apache.org>
References: <4E18CE25-6863-4318-8A95-17896B0B87D1@yahoo-inc.com>
 <0C4895F3-95E8-45ED-B2B0-C3BF4B5436B1@gbiv.com> <4E0082D9.5090607@apache.org> <82A248C7-D17B-4AD3-A8B2-75F91E89F600@apache.org>
From: Ted Dunning <tdunning@maprtech.com>
Date: Tue, 21 Jun 2011 12:04:15 -0700
Message-ID: <BANLkTi=rk=8nQJofmupt-KCWEpQYTb4JNA@mail.gmail.com>
Subject: Re: [DISCUSSION] Thinking about 20.204 and beyond
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec548a95dd0519904a63d85c0

--bcaec548a95dd0519904a63d85c0
Content-Type: text/plain; charset=ISO-8859-1

For the pain in doing this, it is probably better to just drop $10 and bring
up a nice EC2 cluster with 10 m1.large instances using spot pricing for 5
hours.

On Tue, Jun 21, 2011 at 11:43 AM, Allen Wittenauer <aw@apache.org> wrote:

>
> On Jun 21, 2011, at 4:39 AM, Steve Loughran wrote:
> >  If I can actually bring up a heterogenous cluster here
>
>        I believe there was a post in one of the mailing lists in the past 6
> months where someone tried a mixed endian grid.  It blew up big time.

--bcaec548a95dd0519904a63d85c0--

From general-return-3908-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 19:09:15 2011
Return-Path: <general-return-3908-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6C4154DEE
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 21 Jun 2011 19:09:15 +0000 (UTC)
Received: (qmail 69568 invoked by uid 500); 21 Jun 2011 19:09:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69516 invoked by uid 500); 21 Jun 2011 19:09:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69508 invoked by uid 99); 21 Jun 2011 19:09:13 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 19:09:13 +0000
Received: from localhost (HELO awittena-md.linkedin.biz) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 19:09:13 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [DISCUSSION] Thinking about 20.204 and beyond
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <BANLkTi=rk=8nQJofmupt-KCWEpQYTb4JNA@mail.gmail.com>
Date: Tue, 21 Jun 2011 12:09:12 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <314AE6A6-9EFF-4B8A-BA70-A223045BEEC9@apache.org>
References: <4E18CE25-6863-4318-8A95-17896B0B87D1@yahoo-inc.com> <0C4895F3-95E8-45ED-B2B0-C3BF4B5436B1@gbiv.com> <4E0082D9.5090607@apache.org> <82A248C7-D17B-4AD3-A8B2-75F91E89F600@apache.org> <BANLkTi=rk=8nQJofmupt-KCWEpQYTb4JNA@mail.gmail.com>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On Jun 21, 2011, at 12:04 PM, Ted Dunning wrote:

> For the pain in doing this, it is probably better to just drop $10 and =
bring
> up a nice EC2 cluster with 10 m1.large instances using spot pricing =
for 5
> hours.


	Testing on non-Intel, non-Linux is something we need to do more =
of.   Amazon won't help us there, AFAIK.=

From general-return-3909-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 19:11:51 2011
Return-Path: <general-return-3909-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2CED34017
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 21 Jun 2011 19:11:51 +0000 (UTC)
Received: (qmail 73481 invoked by uid 500); 21 Jun 2011 19:11:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73345 invoked by uid 500); 21 Jun 2011 19:11:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73337 invoked by uid 99); 21 Jun 2011 19:11:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 19:11:49 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mlortiz@uci.cu designates 200.55.140.180 as permitted sender)
Received: from [200.55.140.180] (HELO mx3.uci.cu) (200.55.140.180)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 21 Jun 2011 19:11:43 +0000
Received: (qmail 25923 invoked by uid 507); 21 Jun 2011 19:11:16 -0000
Received: from 10.0.0.183 by ns3.uci.cu (envelope-from <mlortiz@uci.cu>, uid 501) with qmail-scanner-2.01st 
 (avp: 5.0.2.0. spamassassin: 3.0.6. perlscan: 2.01st.  
 Clear:RC:1(10.0.0.183):. 
 Processed in 0.862733 secs); 21 Jun 2011 19:11:16 -0000
Received: from unknown (HELO ucimail2.uci.cu) (10.0.0.183)
  by 0 with SMTP; 21 Jun 2011 19:11:15 -0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by ucimail2.uci.cu (Postfix) with ESMTP id 7ABD93DC00A;
	Tue, 21 Jun 2011 15:13:04 -0400 (CDT)
X-Virus-Scanned: amavisd-new at uci.cu
Received: from ucimail2.uci.cu ([127.0.0.1])
	by localhost (ucimail2.uci.cu [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id MMOodzAS43xw; Tue, 21 Jun 2011 15:13:03 -0400 (CDT)
Received: from [10.36.18.57] (marcosluis-aspire-5251.uci.cu [10.36.18.57])
	by ucimail2.uci.cu (Postfix) with ESMTP id 017FD3DC009;
	Tue, 21 Jun 2011 15:13:02 -0400 (CDT)
Message-ID: <4E00F451.1070406@uci.cu>
Date: Tue, 21 Jun 2011 15:13:13 -0430
From: Marcos Ortiz <mlortiz@uci.cu>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110424 Lightning/1.0b2 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
CC: Allen Wittenauer <aw@apache.org>
Subject: Re: [DISCUSSION] Thinking about 20.204 and beyond
References: <4E18CE25-6863-4318-8A95-17896B0B87D1@yahoo-inc.com> <0C4895F3-95E8-45ED-B2B0-C3BF4B5436B1@gbiv.com> <4E0082D9.5090607@apache.org> <82A248C7-D17B-4AD3-A8B2-75F91E89F600@apache.org> <BANLkTi=rk=8nQJofmupt-KCWEpQYTb4JNA@mail.gmail.com> <314AE6A6-9EFF-4B8A-BA70-A223045BEEC9@apache.org>
In-Reply-To: <314AE6A6-9EFF-4B8A-BA70-A223045BEEC9@apache.org>
Content-Type: multipart/alternative;
 boundary="------------010509090409040004040608"
X-Virus-Checked: Checked by ClamAV on apache.org

--------------010509090409040004040608
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit

For example in Solaris and BSD´s systems Right?
regards

On 06/21/2011 02:39 PM, Allen Wittenauer wrote:
> On Jun 21, 2011, at 12:04 PM, Ted Dunning wrote:
>
>> For the pain in doing this, it is probably better to just drop $10 and bring
>> up a nice EC2 cluster with 10 m1.large instances using spot pricing for 5
>> hours.
>
> 	Testing on non-Intel, non-Linux is something we need to do more of.   Amazon won't help us there, AFAIK.

-- 
Marcos Luís Ortíz Valmaseda
  Software Engineer (UCI)
  http://marcosluis2186.posterous.com
  http://twitter.com/marcosluis2186



--------------010509090409040004040608--

From general-return-3910-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 19:12:19 2011
Return-Path: <general-return-3910-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1D2784041
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 21 Jun 2011 19:12:19 +0000 (UTC)
Received: (qmail 74900 invoked by uid 500); 21 Jun 2011 19:12:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74851 invoked by uid 500); 21 Jun 2011 19:12:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74843 invoked by uid 99); 21 Jun 2011 19:12:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 19:12:17 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 19:12:08 +0000
Received: by vws7 with SMTP id 7so99136vws.35
        for <general@hadoop.apache.org>; Tue, 21 Jun 2011 12:11:47 -0700 (PDT)
Received: by 10.52.179.130 with SMTP id dg2mr6308538vdc.203.1308683507038;
 Tue, 21 Jun 2011 12:11:47 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.52.111.166 with HTTP; Tue, 21 Jun 2011 12:11:27 -0700 (PDT)
X-Originating-IP: [64.105.168.204]
In-Reply-To: <314AE6A6-9EFF-4B8A-BA70-A223045BEEC9@apache.org>
References: <4E18CE25-6863-4318-8A95-17896B0B87D1@yahoo-inc.com>
 <0C4895F3-95E8-45ED-B2B0-C3BF4B5436B1@gbiv.com> <4E0082D9.5090607@apache.org>
 <82A248C7-D17B-4AD3-A8B2-75F91E89F600@apache.org> <BANLkTi=rk=8nQJofmupt-KCWEpQYTb4JNA@mail.gmail.com>
 <314AE6A6-9EFF-4B8A-BA70-A223045BEEC9@apache.org>
From: Ted Dunning <tdunning@maprtech.com>
Date: Tue, 21 Jun 2011 12:11:27 -0700
Message-ID: <BANLkTimq8zX2y8_U2Ns10gJ-veAE_rL+GA@mail.gmail.com>
Subject: Re: [DISCUSSION] Thinking about 20.204 and beyond
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec5171ecd8f7b1104a63d9fa3
X-Virus-Checked: Checked by ClamAV on apache.org

--bcaec5171ecd8f7b1104a63d9fa3
Content-Type: text/plain; charset=ISO-8859-1

Amazon can help with the non-linux, but not with the non-Intel.

On Tue, Jun 21, 2011 at 12:09 PM, Allen Wittenauer <aw@apache.org> wrote:

>
> On Jun 21, 2011, at 12:04 PM, Ted Dunning wrote:
>
> > For the pain in doing this, it is probably better to just drop $10 and
> bring
> > up a nice EC2 cluster with 10 m1.large instances using spot pricing for 5
> > hours.
>
>
>        Testing on non-Intel, non-Linux is something we need to do more of.
>   Amazon won't help us there, AFAIK.

--bcaec5171ecd8f7b1104a63d9fa3--

From general-return-3911-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 20:28:18 2011
Return-Path: <general-return-3911-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 941CA4785
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 21 Jun 2011 20:28:18 +0000 (UTC)
Received: (qmail 2383 invoked by uid 500); 21 Jun 2011 20:28:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2331 invoked by uid 500); 21 Jun 2011 20:28:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2323 invoked by uid 99); 21 Jun 2011 20:28:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 20:28:16 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of makrai.list@gmail.com designates 209.85.220.176 as permitted sender)
Received: from [209.85.220.176] (HELO mail-vx0-f176.google.com) (209.85.220.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 20:28:10 +0000
Received: by vxd2 with SMTP id 2so190763vxd.35
        for <general@hadoop.apache.org>; Tue, 21 Jun 2011 13:27:49 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=BFuYboshZd2xJnQ55POK+n1qwBRBb621TPih/r0tExE=;
        b=HRfNcFaYvx+5AjIl3wvOlcoGyu/4ELyGBopL8ROU5dPYFLcukayCOPVjSn/VLXBEsl
         NFkHi/h825E1Jerf2kQO1B5T9E9aB8uAO6lxdxPfA0zcJqGLW4OKzj6SmiaXUgWQHBJh
         CD8sD9iCjNXShcQXP3/1zLozm2Fg6sf1zZctk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=w2dOPIwcxiRErWEoxAuTsKj25PHow+O+hu2fphp23SyofeUkDSDKzyTOttSlUIgdcP
         a7IXkEz2bp/PCMyZ9d//zWWtEifSDcwd3c0tEiUaqjopyrUuvKaTAfycuXpWYo1nFvW4
         Gh7HmpOy/d82s+34xVDYG4zXY/MAg69bAw74s=
MIME-Version: 1.0
Received: by 10.52.98.34 with SMTP id ef2mr268429vdb.293.1308688069716; Tue,
 21 Jun 2011 13:27:49 -0700 (PDT)
Received: by 10.52.109.69 with HTTP; Tue, 21 Jun 2011 13:27:49 -0700 (PDT)
Date: Tue, 21 Jun 2011 22:27:49 +0200
Message-ID: <BANLkTik91dkWnrWyVGp3HnpnZLFHTQjhKQ@mail.gmail.com>
Subject: Large startup time in remote MapReduce job
From: Gabor Makrai <makrai.list@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf307f3528846c4504a63eafcd

--20cf307f3528846c4504a63eafcd
Content-Type: text/plain; charset=ISO-8859-1

Hi everyone,

I have a little problem with running MapReduce jobs.
I have a pretty large Java program (my jar size is more than 20MB) , where I
implemented a MapReduce job. I tested it in my local cluster, and it worked
fine. But I tried it with low-bandwith Internet access and I experienced
very-very slow job starting time :( I guess my whole JAR file was uploaded,
because I experienced unusual upgoing network traffic.
Could anyone tell me how can I solve this problem?

Thanks,
Gabor

--20cf307f3528846c4504a63eafcd--

From general-return-3912-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 20:32:11 2011
Return-Path: <general-return-3912-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9FF0D428F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 21 Jun 2011 20:32:11 +0000 (UTC)
Received: (qmail 16265 invoked by uid 500); 21 Jun 2011 20:32:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16210 invoked by uid 500); 21 Jun 2011 20:32:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16202 invoked by uid 99); 21 Jun 2011 20:32:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 20:32:10 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of harsh@cloudera.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 20:32:03 +0000
Received: by qwj9 with SMTP id 9so112199qwj.35
        for <general@hadoop.apache.org>; Tue, 21 Jun 2011 13:31:43 -0700 (PDT)
Received: by 10.229.49.133 with SMTP id v5mr5563068qcf.165.1308688303105; Tue,
 21 Jun 2011 13:31:43 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.105.7 with HTTP; Tue, 21 Jun 2011 13:31:23 -0700 (PDT)
In-Reply-To: <BANLkTik91dkWnrWyVGp3HnpnZLFHTQjhKQ@mail.gmail.com>
References: <BANLkTik91dkWnrWyVGp3HnpnZLFHTQjhKQ@mail.gmail.com>
From: Harsh J <harsh@cloudera.com>
Date: Wed, 22 Jun 2011 02:01:23 +0530
Message-ID: <BANLkTimSm==X+=9VnHyX+oU-zknzjKRzDA@mail.gmail.com>
Subject: Re: Large startup time in remote MapReduce job
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Gabor,

If your jar does not contain code changes that need to get transmitted
every time, you can consider placing them on the JT/TT classpaths and
not do any jar registration in your job submission code. You'll see a
related WARN but it should be OK to ignore that.

If not, work on other ways to get your jar size reduced. Does it
really contain 20 MB worth of user code or is that with libraries?

On Wed, Jun 22, 2011 at 1:57 AM, Gabor Makrai <makrai.list@gmail.com> wrote:
> Hi everyone,
>
> I have a little problem with running MapReduce jobs.
> I have a pretty large Java program (my jar size is more than 20MB) , where I
> implemented a MapReduce job. I tested it in my local cluster, and it worked
> fine. But I tried it with low-bandwith Internet access and I experienced
> very-very slow job starting time :( I guess my whole JAR file was uploaded,
> because I experienced unusual upgoing network traffic.
> Could anyone tell me how can I solve this problem?
>
> Thanks,
> Gabor
>



-- 
Harsh J

From general-return-3913-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 20:58:42 2011
Return-Path: <general-return-3913-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1AB144EA8
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 21 Jun 2011 20:58:42 +0000 (UTC)
Received: (qmail 62135 invoked by uid 500); 21 Jun 2011 20:58:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62079 invoked by uid 500); 21 Jun 2011 20:58:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62071 invoked by uid 99); 21 Jun 2011 20:58:40 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 20:58:40 +0000
Received: from localhost (HELO awittena-md.linkedin.biz) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 20:58:40 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Large startup time in remote MapReduce job
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <BANLkTimSm==X+=9VnHyX+oU-zknzjKRzDA@mail.gmail.com>
Date: Tue, 21 Jun 2011 13:58:38 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <12015E99-0D45-4837-9C68-69148A8C32E4@apache.org>
References: <BANLkTik91dkWnrWyVGp3HnpnZLFHTQjhKQ@mail.gmail.com> <BANLkTimSm==X+=9VnHyX+oU-zknzjKRzDA@mail.gmail.com>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On Jun 21, 2011, at 1:31 PM, Harsh J wrote:

> Gabor,
>=20
> If your jar does not contain code changes that need to get transmitted
> every time, you can consider placing them on the JT/TT classpaths

	... which means you get to bounce your system every time you =
change code.


> and
> not do any jar registration in your job submission code. You'll see a
> related WARN but it should be OK to ignore that.
>=20
> If not, work on other ways to get your jar size reduced. Does it
> really contain 20 MB worth of user code or is that with libraries?

	Harsh is on the right track.

	Break your jar up into multiple chunks, putting the fairly =
static pieces into a distributed cache.  See =
http://wiki.apache.org/hadoop/FAQ#How_do_I_submit_extra_content_.28jars.2C=
_static_files.2C_etc.29_for_my_job_to_use_during_runtime.3F for more =
info.


From general-return-3914-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 21:03:42 2011
Return-Path: <general-return-3914-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3CFB944E0
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 21 Jun 2011 21:03:42 +0000 (UTC)
Received: (qmail 71336 invoked by uid 500); 21 Jun 2011 21:03:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71277 invoked by uid 500); 21 Jun 2011 21:03:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71253 invoked by uid 99); 21 Jun 2011 21:03:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 21:03:40 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of harsh@cloudera.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 21:03:33 +0000
Received: by qwj9 with SMTP id 9so132260qwj.35
        for <general@hadoop.apache.org>; Tue, 21 Jun 2011 14:03:13 -0700 (PDT)
Received: by 10.224.138.16 with SMTP id y16mr5408859qat.3.1308690193065; Tue,
 21 Jun 2011 14:03:13 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.105.7 with HTTP; Tue, 21 Jun 2011 14:02:53 -0700 (PDT)
In-Reply-To: <12015E99-0D45-4837-9C68-69148A8C32E4@apache.org>
References: <BANLkTik91dkWnrWyVGp3HnpnZLFHTQjhKQ@mail.gmail.com>
 <BANLkTimSm==X+=9VnHyX+oU-zknzjKRzDA@mail.gmail.com> <12015E99-0D45-4837-9C68-69148A8C32E4@apache.org>
From: Harsh J <harsh@cloudera.com>
Date: Wed, 22 Jun 2011 02:32:53 +0530
Message-ID: <BANLkTini-1Gi4xWr=WRv+2TebcjDxcu9BQ@mail.gmail.com>
Subject: Re: Large startup time in remote MapReduce job
To: mapreduce-user@hadoop.apache.org
Cc: makrai.list@gmail.com
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Allen,

On Wed, Jun 22, 2011 at 2:28 AM, Allen Wittenauer <aw@apache.org> wrote:
>
> On Jun 21, 2011, at 1:31 PM, Harsh J wrote:
>
>> Gabor,
>>
>> If your jar does not contain code changes that need to get transmitted
>> every time, you can consider placing them on the JT/TT classpaths
>
> =A0 =A0 =A0 =A0... which means you get to bounce your system every time y=
ou change code.

Its ugly, but if the jar filename remains the same there shouldn't
need to be any bouncing. Doable if there's no activity at replacement
point of time?

P.s. Gabor: Moving the discussion to mapreduce-user@hadoop.apache.org
Please use the general@ list only for general project-wide discussions
on Hadoop.

--=20
Harsh J

From general-return-3915-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 21:57:47 2011
Return-Path: <general-return-3915-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4BE1B44D9
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 21 Jun 2011 21:57:47 +0000 (UTC)
Received: (qmail 72833 invoked by uid 500); 21 Jun 2011 21:57:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72790 invoked by uid 500); 21 Jun 2011 21:57:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72782 invoked by uid 99); 21 Jun 2011 21:57:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 21:57:45 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of scott@richrelevance.com designates 64.78.17.17 as permitted sender)
Received: from [64.78.17.17] (HELO EXHUB018-2.exch018.msoutlookonline.net) (64.78.17.17)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 21:57:39 +0000
Received: from EXVMBX018-1.exch018.msoutlookonline.net ([64.78.17.47]) by
 EXHUB018-2.exch018.msoutlookonline.net ([64.78.17.17]) with mapi; Tue, 21 Jun
 2011 14:57:19 -0700
From: Scott Carey <scott@richrelevance.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 21 Jun 2011 14:58:06 -0700
Subject: Re: [DISCUSSION] Thinking about 20.204 and beyond 
Thread-Topic: [DISCUSSION] Thinking about 20.204 and beyond 
Thread-Index: AcwwXjFkHwqoUfRaSWK3n2KUovcGMA==
Message-ID: <CA265FFF.3F1AB%scott@richrelevance.com>
In-Reply-To: <4E18CE25-6863-4318-8A95-17896B0B87D1@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-MacOutlook/14.10.0.110310
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0



On 6/17/11 11:50 PM, "Eric Baldeschwieler" <eric14@yahoo-inc.com> wrote:

>Producing a stable release of Hadoop is a long, hard and expensive
>process.  Historically Y! has produced all such releases.  Other releases
>of Hadoop have either not been stable (Hadoop 0.19 and Hadoop 0.21)

You lost me there.  Enough Y! Chest pounding.  Your contributions are
large and important, but not every release not tested by Y! is destined
for failure.

0.19.x eventually was stable and an improvement over 0.18.x in several
critical ways (I used it in production for a year and relied on several
new features).
0.19.0 was broken, append needed to be disabled, etc.  But by the end
after testing in the community, (by research institutions and smaller
corporations) and some bug fix releases it became stable.


From general-return-3916-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 22 09:18:20 2011
Return-Path: <general-return-3916-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4F3156B85
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 22 Jun 2011 09:18:20 +0000 (UTC)
Received: (qmail 96385 invoked by uid 500); 22 Jun 2011 09:18:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96188 invoked by uid 500); 22 Jun 2011 09:18:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96180 invoked by uid 99); 22 Jun 2011 09:18:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 09:18:17 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 09:18:09 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 8881EB8125
	for <general@hadoop.apache.org>; Wed, 22 Jun 2011 10:17:47 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id MoR47i6xvfIH for <general@hadoop.apache.org>;
	Wed, 22 Jun 2011 10:17:47 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by tobor.hpl.hp.com (Postfix) with ESMTP id C23B3B8123
	for <general@hadoop.apache.org>; Wed, 22 Jun 2011 10:17:46 +0100 (BST)
MailScanner-NULL-Check: 1309338976.73388@/+4FyQxpho9PzCHibl444Q
Received: from [16.24.212.58] ([16.24.212.58])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p5M9GGwI020691
	for <general@hadoop.apache.org>; Wed, 22 Jun 2011 10:16:16 +0100 (BST)
Message-ID: <4E01B2DB.3050503@apache.org>
Date: Wed, 22 Jun 2011 10:16:11 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.2.17) Gecko/20110414 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSSION] Thinking about 20.204 and beyond
References: <4E18CE25-6863-4318-8A95-17896B0B87D1@yahoo-inc.com> <0C4895F3-95E8-45ED-B2B0-C3BF4B5436B1@gbiv.com> <4E0082D9.5090607@apache.org> <82A248C7-D17B-4AD3-A8B2-75F91E89F600@apache.org> <BANLkTi=rk=8nQJofmupt-KCWEpQYTb4JNA@mail.gmail.com>
In-Reply-To: <BANLkTi=rk=8nQJofmupt-KCWEpQYTb4JNA@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p5M9GGwI020691
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 21/06/2011 20:04, Ted Dunning wrote:
> For the pain in doing this, it is probably better to just drop $10 and bring
> up a nice EC2 cluster with 10 m1.large instances using spot pricing for 5
> hours.

this lacks the mess of networking that is common in the outside world

From general-return-3917-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 22 09:31:09 2011
Return-Path: <general-return-3917-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6DC146BCD
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 22 Jun 2011 09:31:09 +0000 (UTC)
Received: (qmail 16436 invoked by uid 500); 22 Jun 2011 09:31:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16354 invoked by uid 500); 22 Jun 2011 09:31:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16313 invoked by uid 99); 22 Jun 2011 09:31:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 09:31:06 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 09:30:56 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id C709DB8125
	for <general@hadoop.apache.org>; Wed, 22 Jun 2011 10:30:35 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id ks3SOSGOmGpo for <general@hadoop.apache.org>;
	Wed, 22 Jun 2011 10:30:35 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by tobor.hpl.hp.com (Postfix) with ESMTP id F37CCB8123
	for <general@hadoop.apache.org>; Wed, 22 Jun 2011 10:30:34 +0100 (BST)
MailScanner-NULL-Check: 1309339821.13194@MzEi+1wpKu4BRcJdeL/xdQ
Received: from [16.24.212.58] ([16.24.212.58])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p5M9UKYi021101
	for <general@hadoop.apache.org>; Wed, 22 Jun 2011 10:30:20 +0100 (BST)
Message-ID: <4E01B627.6000304@apache.org>
Date: Wed, 22 Jun 2011 10:30:15 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.2.17) Gecko/20110414 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSSION] Thinking about 20.204 and beyond
References: <4E18CE25-6863-4318-8A95-17896B0B87D1@yahoo-inc.com> <0C4895F3-95E8-45ED-B2B0-C3BF4B5436B1@gbiv.com> <4E0082D9.5090607@apache.org> <D32D716F-308A-41C1-844B-08B99D2443D0@gbiv.com>
In-Reply-To: <D32D716F-308A-41C1-844B-08B99D2443D0@gbiv.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p5M9UKYi021101
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 21/06/2011 19:40, Roy T. Fielding wrote:
> On Jun 21, 2011, at 4:39 AM, Steve Loughran wrote:
>
>> On 18/06/2011 21:22, Roy T. Fielding wrote:
>> olutely no reason that trunk cannot be packaged for release
>>> tomorrow as 0.23.  There may be many reasons why it won't pass a release
>>> vote, but we probably aren't going to find them until somebody tries.
>>
>> One limitation with releases has always been size of cluster testing -where Yahoo!s contributions have been invaluable. That said, we shouldn't make them an SPOF in the release process; we should all set up to do some more release testing.
>
> Yes, more testing is better, but if it can't be tested by the dev team
> in 72 hours then it doesn't belong in our release process.
>
> Please note that one of the main advantages of open source development
> is that the bulk of testing/QA occurs *after* the release.  That's why
> labels like alpha/beta/GA are best applied/updated after the version number
> has been cut and the software has been proven in real deployments.
> If testing on 5000 nodes is important to our customers, then add a
> scale-tested metric to the download site so that the customers know
> which release package has been tested at what scale -- they will
> understand the difference between frequent releases and those fully tested
> at scale.  Let them decide which version is best to use for their own needs.

I agree with in-field testing; the big issue there is that people who do 
have 1+PB of data are nervous about Hadoop upgrades, JVM upgrades. I 
haven't even heard of anyone who owns up to moving to ext4 fs 
underneath. It's the cost of loss of data that raises concerns, not the 
loss of time in testing and rolling back [1]. And very few people have 
500+ node clusters sitting around idle.

I like the point about mixed endian; Sparc may be dead but there are 
other architectures out there, and Arm looks up and coming. Then there 
are bits of the config space like a block replication factor of 2 that 
can be tested at small scale.

-steve

[1] Steve reverted his work desktop from RHEL6 to Ubuntu 10.04 last week 
and, after discussion with his 9 year old sun, is going to switch back 
to Firefox 3.6 as it does better flash games)


From general-return-3918-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 22 15:42:30 2011
Return-Path: <general-return-3918-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 36D146482
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 22 Jun 2011 15:42:30 +0000 (UTC)
Received: (qmail 3087 invoked by uid 500); 22 Jun 2011 15:42:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3014 invoked by uid 500); 22 Jun 2011 15:42:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3001 invoked by uid 99); 22 Jun 2011 15:42:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 15:42:28 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 216.145.54.171 is neither permitted nor denied by domain of eric14@yahoo-inc.com)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 15:42:23 +0000
Received: from [10.0.1.3] (snvvpn2-10-73-152-c147.hq.corp.yahoo.com [10.73.152.147])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5MFfXRY069919;
	Wed, 22 Jun 2011 08:41:34 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1308757294;
	bh=HO8U8E3uEvSlDTYGn7S8b3HGnJLTWDMtM81AGWs2krA=;
	h=Subject:Mime-Version:Content-Type:From:In-Reply-To:Date:Cc:
	 Content-Transfer-Encoding:Message-Id:References:To;
	b=K600aoLqOeKl2u0LnyNoczSSBHbWT37QTnDXd9olOHKpPJjjCZF2KrD/F53Q/nieC
	 tVhd9hK3r1LKCw9rWVNpuf6+6UCa7ygpay0HLJLyzP1itpMbCH4PQvdJ/cRHiK4vcY
	 +EUJqjukbclP3RQl3HuoCm5PQOMUs06uWN9+n7Gw=
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
Mime-Version: 1.0 (Apple Message framework v1084)
Content-Type: text/plain; charset=us-ascii
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <BANLkTikusmVmKVVMbTqCgLprPrC0ibB3Gw@mail.gmail.com>
Date: Wed, 22 Jun 2011 08:41:33 -0700
Cc: "apurtell@apache.org" <apurtell@apache.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <0A3DB5BA-925A-4A67-ADC9-B984CBF1D004@yahoo-inc.com>
References: <BANLkTi=wZomy0sMGDrg3+VGjoYyHbjBxkw@mail.gmail.com> <374085.46450.qm@web65510.mail.ac4.yahoo.com> <BANLkTikusmVmKVVMbTqCgLprPrC0ibB3Gw@mail.gmail.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1084)

I agree with this.

We need to find a middle ground that achieves three aims:

1) Makes it clear that an ASF release of Hadoop is THE APACHE HADOOP.  =
Jeff's manpower argument actually reinforces this.  We need a very =
testable definition of what is an Apache Hadoop Release or enforcement =
will be impossible because each test of the policy might require a visit =
to the supreme court.  It's MD5 matches the MD5 of an apache release is =
a clear definition.

2) We need a proposal for derived products that vendors feel are =
branding friendly.  These should be clear enough that users understand =
the difference between a product that packages Apache Hadoop (MD5 test), =
one that is completely open source under the Apache license (easy to =
test) and one that simply uses some subset of the code under a more =
restrictive license or closed source.

3) Compatibility: I think it would be great to harness all this energy =
around compatibility to start a compatibility suite inside the Apache =
Hadoop project.  Then we could define compatible with Apache Hadoop in a =
clear way controlled by the Apache Hadoop PMC.  With luck vendors on =
both sides of the debate will be incentivized to contribute to the =
project this way.  Such a suite would also prove useful to the =
developers of Apache Hadoop.

E14

On Jun 20, 2011, at 10:09 AM, Ted Dunning wrote:

> Great summary Andrew.
>=20
> I would add one more precipitating factor here.  That is the arrival =
of a
> number of products which are very close to the Apache version of =
Hadoop but
> for which there is no good and widely accepted terminology that gives =
proper
> credit to their lineage while making clear the distinction from =
bit-for-bit
> copies of official Apache releases.
>=20
> Some products are analogous to hive, pig or hbase in that they are
> independent systems that run ON hadoop (or close equivalents).  These =
have
> no terminology problem because these products aren't hadoop, but =
rather use
> hadoop.
>=20
> Other products contain Hadoop internally as a critical component but =
do not
> necessarily expose Hadoop capabilities to the end user (I can't name =
these
> products, but they exist).  These products have little nomenclatural
> difficulty because the powerd-by-Hadoop description fits very well.
>=20
> The products with the terminology problem are the ones that are add =
either
> curation and packaging (Cloudera) or substantial additional =
performance
> enhancing components (MapR).  These products are upwardly compatible =
with
> Apache Hadoop in that programs that run on Hadoop will very probably =
run on
> these Hadoop-like systems.  The problem is that there is no good term =
for
> these products.  They may even contain components that are bit-for-bit
> identical to the same components for Apache releases.  It is fair to =
say
> that these are not Apache released software, but it is also fair to =
say that
> there ought to be a better name for the class of these products.
>=20
> On Mon, Jun 20, 2011 at 4:39 PM, Andrew Purtell <apurtell@apache.org> =
wrote:
>=20
>> Hadoop I think needs to be more careful. What triggered this =
discussion is
>> the arrival of new players releasing products they call Hadoop but
>> containing severe changes the community, by way of the ASF umbrella =
we all
>> work under, had nothing to do with designing or developing. And some =
of
>> these are being open sourced as a Hadoop. There is no Linus here. =
Which of
>> these is _the_ Hadoop? As a would-be contributor, which should I =
select?
>>=20


From general-return-3919-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 22 16:27:02 2011
Return-Path: <general-return-3919-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 800AE6419
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 22 Jun 2011 16:27:02 +0000 (UTC)
Received: (qmail 526 invoked by uid 500); 22 Jun 2011 16:27:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 456 invoked by uid 500); 22 Jun 2011 16:27:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 448 invoked by uid 99); 22 Jun 2011 16:27:00 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 16:27:00 +0000
Received: from localhost (HELO dhcp-02.private.iobm.com) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 16:27:00 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [DISCUSSION] Thinking about 20.204 and beyond
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <4E01B627.6000304@apache.org>
Date: Wed, 22 Jun 2011 09:27:01 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <C2CC515C-27E5-435C-9E80-639D56A4A44B@apache.org>
References: <4E18CE25-6863-4318-8A95-17896B0B87D1@yahoo-inc.com> <0C4895F3-95E8-45ED-B2B0-C3BF4B5436B1@gbiv.com> <4E0082D9.5090607@apache.org> <D32D716F-308A-41C1-844B-08B99D2443D0@gbiv.com> <4E01B627.6000304@apache.org>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On Jun 22, 2011, at 2:30 AM, Steve Loughran wrote:
>=20
>  I haven't even heard of anyone who owns up to moving to ext4 fs =
underneath.

	Yes you do.

	:D


From general-return-3920-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 22 20:27:12 2011
Return-Path: <general-return-3920-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3BB966B58
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 22 Jun 2011 20:27:12 +0000 (UTC)
Received: (qmail 16278 invoked by uid 500); 22 Jun 2011 20:27:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16223 invoked by uid 500); 22 Jun 2011 20:27:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16215 invoked by uid 99); 22 Jun 2011 20:27:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 20:27:10 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of scott@richrelevance.com designates 64.78.17.16 as permitted sender)
Received: from [64.78.17.16] (HELO EXHUB018-1.exch018.msoutlookonline.net) (64.78.17.16)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 20:27:02 +0000
Received: from EXVMBX018-1.exch018.msoutlookonline.net ([64.78.17.47]) by
 EXHUB018-1.exch018.msoutlookonline.net ([64.78.17.16]) with mapi; Wed, 22 Jun
 2011 13:26:42 -0700
From: Scott Carey <scott@richrelevance.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 22 Jun 2011 13:27:29 -0700
Subject: Re: Hadoop Java Versions
Thread-Topic: Hadoop Java Versions
Thread-Index: AcwxGrJdy2N367J0RJiueePq8AbC2A==
Message-ID: <CA279A5D.3F341%scott@richrelevance.com>
In-Reply-To: <1F501BC2-7671-45B5-8D31-599C25068191@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-MacOutlook/14.10.0.110310
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

"Problems have been reported with Hadoop, the 64-bit JVM and Compressed
Object References (the -XX:+UseCompressedOops option), so use of that
option is discouraged."

I think the above is dated.  It also lacks critical information. What JVM
and OS version was the problem seen?

CompressedOops had several issues prior to Jre 6u20, and a few minor ones
were fixed in u21.  FWIW, I now exclusively use 64 bit w/ CompressedOops
for all Hadoop and non-Hadoop apps and have seen no issues.  It is the
default in 6u24 and 6u25 on a 64 bit JVM.

On 6/14/11 5:16 PM, "Allen Wittenauer" <aw@apache.org> wrote:

>
>While we're looking at the wiki, could folks update
>http://wiki.apache.org/hadoop/HadoopJavaVersions with whatever versions
>of Hadoop they are using successfully?
>
>Thanks.
>
>P.S., yes, I'm thinking about upgrading ours. :p


From general-return-3921-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 22 20:50:03 2011
Return-Path: <general-return-3921-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0DE346BF0
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 22 Jun 2011 20:50:03 +0000 (UTC)
Received: (qmail 45687 invoked by uid 500); 22 Jun 2011 20:50:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45477 invoked by uid 500); 22 Jun 2011 20:50:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45469 invoked by uid 99); 22 Jun 2011 20:50:01 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 20:50:01 +0000
Received: from localhost (HELO awittena-md.linkedin.biz) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 20:50:01 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Hadoop Java Versions
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <CA279A5D.3F341%scott@richrelevance.com>
Date: Wed, 22 Jun 2011 13:49:58 -0700
Content-Transfer-Encoding: 7bit
Message-Id: <8460D6B5-E8F3-4FCC-8765-655A247ED6EB@apache.org>
References: <CA279A5D.3F341%scott@richrelevance.com>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On Jun 22, 2011, at 1:27 PM, Scott Carey wrote:

> "Problems have been reported with Hadoop, the 64-bit JVM and Compressed
> Object References (the -XX:+UseCompressedOops option), so use of that
> option is discouraged."
> 
> I think the above is dated.  It also lacks critical information. What JVM
> and OS version was the problem seen?

That is definitely dated.  Those were problems from 18 and 19.

> 
> CompressedOops had several issues prior to Jre 6u20, and a few minor ones
> were fixed in u21.  FWIW, I now exclusively use 64 bit w/ CompressedOops
> for all Hadoop and non-Hadoop apps and have seen no issues.  It is the
> default in 6u24 and 6u25 on a 64 bit JVM.

	We use compressedoops on 21 every-so-often as well.  

	Feel free to edit that whole section. :)

	Thanks!


From general-return-3922-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 02:42:34 2011
Return-Path: <general-return-3922-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B49074FA1
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 23 Jun 2011 02:42:34 +0000 (UTC)
Received: (qmail 43563 invoked by uid 500); 23 Jun 2011 02:42:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 43247 invoked by uid 500); 23 Jun 2011 02:42:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 43239 invoked by uid 99); 23 Jun 2011 02:42:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 02:42:31 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of scott@richrelevance.com designates 64.78.17.17 as permitted sender)
Received: from [64.78.17.17] (HELO EXHUB018-2.exch018.msoutlookonline.net) (64.78.17.17)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 02:42:22 +0000
Received: from EXVMBX018-1.exch018.msoutlookonline.net ([64.78.17.47]) by
 EXHUB018-2.exch018.msoutlookonline.net ([64.78.17.17]) with mapi; Wed, 22 Jun
 2011 19:42:01 -0700
From: Scott Carey <scott@richrelevance.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 22 Jun 2011 19:42:48 -0700
Subject: Re: Hadoop Java Versions
Thread-Topic: Hadoop Java Versions
Thread-Index: AcwxTyDBwhQDXYqyTSqXa0mC/LF0vg==
Message-ID: <CA27F51C.3F440%scott@richrelevance.com>
In-Reply-To: <8460D6B5-E8F3-4FCC-8765-655A247ED6EB@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-MacOutlook/14.10.0.110310
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org



On 6/22/11 1:49 PM, "Allen Wittenauer" <aw@apache.org> wrote:

>
>On Jun 22, 2011, at 1:27 PM, Scott Carey wrote:
>
>> "Problems have been reported with Hadoop, the 64-bit JVM and Compressed
>> Object References (the -XX:+UseCompressedOops option), so use of that
>> option is discouraged."
>>=20
>> I think the above is dated.  It also lacks critical information. What
>>JVM
>> and OS version was the problem seen?
>
>That is definitely dated.  Those were problems from 18 and 19.
>
>>=20
>> CompressedOops had several issues prior to Jre 6u20, and a few minor
>>ones
>> were fixed in u21.  FWIW, I now exclusively use 64 bit w/ CompressedOops
>> for all Hadoop and non-Hadoop apps and have seen no issues.  It is the
>> default in 6u24 and 6u25 on a 64 bit JVM.
>
>	We use compressedoops on 21 every-so-often as well.
>
>	Feel free to edit that whole section. :)
>
>	Thanks!

Updated the CompressedOops section.

I am using 6u25 in production and have used 6u24 as well.  (64 bit
version, CentOS 5.5)
We usually roll out JVM's to a subset of nodes, then after a little
burn-in to the whole thing.  First on development / qa clusters (which are
just as busy as production, but smaller), then production.

>


From general-return-3923-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 06:22:19 2011
Return-Path: <general-return-3923-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7462849BB
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 23 Jun 2011 06:22:19 +0000 (UTC)
Received: (qmail 49437 invoked by uid 500); 23 Jun 2011 06:22:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47452 invoked by uid 500); 23 Jun 2011 06:22:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47443 invoked by uid 99); 23 Jun 2011 06:22:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 06:22:01 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 216.145.54.173 is neither permitted nor denied by domain of eric14@yahoo-inc.com)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 06:21:56 +0000
Received: from [10.0.1.3] (snvvpn4-10-72-168-c88.hq.corp.yahoo.com [10.72.168.88])
	by mrout3.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5N6LBgv015566
	for <general@hadoop.apache.org>; Wed, 22 Jun 2011 23:21:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1308810073;
	bh=5+RqFgfrxedFNuLLjlEjQ/ZNovQimedHEjJ1DjpGOBo=;
	h=Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:
	 Content-Transfer-Encoding:Message-Id:References:To;
	b=Qbs6I9Q0n/U5w3azv+bRPs5/exnFhGAezbnO3BCdbFXzZtaLHCoUPGhOOEwGIf+52
	 jVBNzmrA28/gLrmyihCjIjeeShemyHHOr6JvwEHqsA64zfIc4DOpsldV/abI0Fb7rq
	 xXW24+ryIAnSXxcZbLOCVtX/ArWA7k2Cs/4y1IPw=
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: [DISCUSSION] Thinking about 20.204 and beyond
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <D32D716F-308A-41C1-844B-08B99D2443D0@gbiv.com>
Date: Wed, 22 Jun 2011 23:21:13 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <DE6A6192-7B6E-4A49-AA61-E535B8D888CC@yahoo-inc.com>
References: <4E18CE25-6863-4318-8A95-17896B0B87D1@yahoo-inc.com> <0C4895F3-95E8-45ED-B2B0-C3BF4B5436B1@gbiv.com> <4E0082D9.5090607@apache.org> <D32D716F-308A-41C1-844B-08B99D2443D0@gbiv.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1084)

Yup.  Frequent releases, some focused on getting to stable on older code =
lines, some pushing new code out for people to try.

On Jun 21, 2011, at 11:40 AM, Roy T. Fielding wrote:

> On Jun 21, 2011, at 4:39 AM, Steve Loughran wrote:
>=20
>> On 18/06/2011 21:22, Roy T. Fielding wrote:
>> olutely no reason that trunk cannot be packaged for release
>>> tomorrow as 0.23.  There may be many reasons why it won't pass a =
release
>>> vote, but we probably aren't going to find them until somebody =
tries.
>>=20
>> One limitation with releases has always been size of cluster testing =
-where Yahoo!s contributions have been invaluable. That said, we =
shouldn't make them an SPOF in the release process; we should all set up =
to do some more release testing.
>=20
> Yes, more testing is better, but if it can't be tested by the dev team
> in 72 hours then it doesn't belong in our release process.
>=20
> Please note that one of the main advantages of open source development
> is that the bulk of testing/QA occurs *after* the release.  That's why
> labels like alpha/beta/GA are best applied/updated after the version =
number
> has been cut and the software has been proven in real deployments.
> If testing on 5000 nodes is important to our customers, then add a
> scale-tested metric to the download site so that the customers know
> which release package has been tested at what scale -- they will
> understand the difference between frequent releases and those fully =
tested
> at scale.  Let them decide which version is best to use for their own =
needs.
>=20
> ....Roy
>=20


From general-return-3924-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 12:48:26 2011
Return-Path: <general-return-3924-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ABA384C1B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 23 Jun 2011 12:48:26 +0000 (UTC)
Received: (qmail 63942 invoked by uid 500); 23 Jun 2011 12:48:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63821 invoked by uid 500); 23 Jun 2011 12:48:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63812 invoked by uid 99); 23 Jun 2011 12:48:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 12:48:24 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 12:48:14 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 2600F1BA969
	for <general@hadoop.apache.org>; Thu, 23 Jun 2011 13:47:53 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id ov5AnUaevmrO for <general@hadoop.apache.org>;
	Thu, 23 Jun 2011 13:47:52 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by colossus.hpl.hp.com (Postfix) with ESMTP id B3FB91BA937
	for <general@hadoop.apache.org>; Thu, 23 Jun 2011 13:47:52 +0100 (BST)
MailScanner-NULL-Check: 1309438056.67389@DBsYd2iZiQJP1WqwXTpnQQ
Received: from [16.24.224.182] ([16.24.224.182])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p5NClZBp029022
	for <general@hadoop.apache.org>; Thu, 23 Jun 2011 13:47:35 +0100 (BST)
Message-ID: <4E0335E1.9010109@apache.org>
Date: Thu, 23 Jun 2011 13:47:29 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.2.17) Gecko/20110414 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSSION] Thinking about 20.204 and beyond
References: <4E18CE25-6863-4318-8A95-17896B0B87D1@yahoo-inc.com> <0C4895F3-95E8-45ED-B2B0-C3BF4B5436B1@gbiv.com> <4E0082D9.5090607@apache.org> <D32D716F-308A-41C1-844B-08B99D2443D0@gbiv.com> <4E01B627.6000304@apache.org> <C2CC515C-27E5-435C-9E80-639D56A4A44B@apache.org>
In-Reply-To: <C2CC515C-27E5-435C-9E80-639D56A4A44B@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p5NClZBp029022
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 22/06/2011 17:27, Allen Wittenauer wrote:
>
> On Jun 22, 2011, at 2:30 AM, Steve Loughran wrote:
>>
>>   I haven't even heard of anyone who owns up to moving to ext4 fs underneath.
>
> 	Yes you do.
>
> 	:D
>

Did it work?
and RHEL6.0?

From general-return-3925-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 12:50:51 2011
Return-Path: <general-return-3925-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 854C14D54
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 23 Jun 2011 12:50:51 +0000 (UTC)
Received: (qmail 68436 invoked by uid 500); 23 Jun 2011 12:50:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68384 invoked by uid 500); 23 Jun 2011 12:50:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68376 invoked by uid 99); 23 Jun 2011 12:50:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 12:50:49 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 12:50:41 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id F33DB1BA96D
	for <general@hadoop.apache.org>; Thu, 23 Jun 2011 13:50:19 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id Oe9dBYFKHqF1 for <general@hadoop.apache.org>;
	Thu, 23 Jun 2011 13:50:19 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 613181BA969
	for <general@hadoop.apache.org>; Thu, 23 Jun 2011 13:50:19 +0100 (BST)
MailScanner-NULL-Check: 1309438205.08489@qhPTSmp7EBdqQ+vbJb4j1Q
Received: from [16.24.224.182] ([16.24.224.182])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p5NCo3n7029163
	for <general@hadoop.apache.org>; Thu, 23 Jun 2011 13:50:03 +0100 (BST)
Message-ID: <4E033676.8030205@apache.org>
Date: Thu, 23 Jun 2011 13:49:58 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.2.17) Gecko/20110414 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Hadoop Java Versions
References: <CA279A5D.3F341%scott@richrelevance.com>
In-Reply-To: <CA279A5D.3F341%scott@richrelevance.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p5NCo3n7029163
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 22/06/2011 21:27, Scott Carey wrote:
> "Problems have been reported with Hadoop, the 64-bit JVM and Compressed
> Object References (the -XX:+UseCompressedOops option), so use of that
> option is discouraged."
>
> I think the above is dated.  It also lacks critical information. What JVM
> and OS version was the problem seen?

A colleague saw it, intermittent JVM crashes. Unless he's updated I 
can't say the problem has gone away.

>
> CompressedOops had several issues prior to Jre 6u20, and a few minor ones
> were fixed in u21.  FWIW, I now exclusively use 64 bit w/ CompressedOops
> for all Hadoop and non-Hadoop apps and have seen no issues.  It is the
> default in 6u24 and 6u25 on a 64 bit  JVM.


what's your HW setup? #cores/server, #servers, underlying OS?

From general-return-3926-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 18:23:43 2011
Return-Path: <general-return-3926-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 89AC54697
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 23 Jun 2011 18:23:43 +0000 (UTC)
Received: (qmail 93432 invoked by uid 500); 23 Jun 2011 18:23:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93372 invoked by uid 500); 23 Jun 2011 18:23:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93359 invoked by uid 99); 23 Jun 2011 18:23:41 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 18:23:41 +0000
Received: from localhost (HELO awittena-md.linkedin.biz) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 18:23:41 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [DISCUSSION] Thinking about 20.204 and beyond
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <4E0335E1.9010109@apache.org>
Date: Thu, 23 Jun 2011 11:23:40 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <BC812FA4-4551-4575-BE49-551084CC2A16@apache.org>
References: <4E18CE25-6863-4318-8A95-17896B0B87D1@yahoo-inc.com> <0C4895F3-95E8-45ED-B2B0-C3BF4B5436B1@gbiv.com> <4E0082D9.5090607@apache.org> <D32D716F-308A-41C1-844B-08B99D2443D0@gbiv.com> <4E01B627.6000304@apache.org> <C2CC515C-27E5-435C-9E80-639D56A4A44B@apache.org> <4E0335E1.9010109@apache.org>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On Jun 23, 2011, at 5:47 AM, Steve Loughran wrote:

> On 22/06/2011 17:27, Allen Wittenauer wrote:
>>=20
>> On Jun 22, 2011, at 2:30 AM, Steve Loughran wrote:
>>>=20
>>>  I haven't even heard of anyone who owns up to moving to ext4 fs =
underneath.
>>=20
>> 	Yes you do.
>>=20
>> 	:D
>>=20
>=20
> Did it work?
> and RHEL6.0?


	We've been using ext4 for HDFS and MR spill space (the rest are =
ext3) on CentOS 5.5 in some form or another for almost a year now.  No =
issues to report, other than it isn't ZFS (so we lost some functionality =
that we greatly miss).

	At this point, we're waiting for 6.1 or 6.2 before changing our =
Linux version.   RH (historically) has too much shift between 0->1->2 =
for it to be considered stable until the .2 release.  But we might jump =
on .1 anyway.=

From general-return-3927-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 00:16:29 2011
Return-Path: <general-return-3927-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 50E484D49
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 24 Jun 2011 00:16:29 +0000 (UTC)
Received: (qmail 71094 invoked by uid 500); 24 Jun 2011 00:16:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71013 invoked by uid 500); 24 Jun 2011 00:16:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71005 invoked by uid 99); 24 Jun 2011 00:16:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 00:16:27 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of dujunping@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 00:16:19 +0000
Received: by wyb36 with SMTP id 36so2530687wyb.35
        for <general@hadoop.apache.org>; Thu, 23 Jun 2011 17:15:59 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=dN5v3//TlGA2vtR5tMCdRTl2KxfS8SOkn2dK10adahM=;
        b=oGC5AAnfWHiF03RP4dgxsJevtoUTLBJtfMofSYiD0Ci4guQW2gh5qBPIT3HQ6wFG9n
         YlX0ZvRo65Lk6a1B7SWYq2VlA+ulMkq97y13JJgMcTAryLWejfF+zI4TSt3qj4vPfqQt
         92RcIAZHr8bFo9Nzy6pXoAALBmfLfVgisW8no=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=d+Ow0Nh+UqsZJ8ZhLA0ywYuhKsTw6hpm1DiiLtxKLO5N4AF7/ETKm30YkisOPRW/8N
         Ry6rpjATdohUVygaG0/qrovgDx9ISSm8FKfhlKCVrES5BWVOLN8/erK3TqMNMW4soPIe
         v5PnSysD5BZh7PdiVUMgxnUsl4FjcDPxzjVcw=
MIME-Version: 1.0
Received: by 10.216.197.96 with SMTP id s74mr2326899wen.21.1308874559552; Thu,
 23 Jun 2011 17:15:59 -0700 (PDT)
Received: by 10.216.20.147 with HTTP; Thu, 23 Jun 2011 17:15:59 -0700 (PDT)
Date: Fri, 24 Jun 2011 08:15:59 +0800
Message-ID: <BANLkTi=V7pqOtAFbn=09wD2hJePWgmWV7A@mail.gmail.com>
Subject: How to configure a multi-tenancy hadoop cluster?
From: =?GB2312?B?v6HGvbbC?= <dujunping@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6dd863c2d7bcd04a66a1b1c
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6dd863c2d7bcd04a66a1b1c
Content-Type: text/plain; charset=ISO-8859-1

Hello all,
         I setup a hadoop cluster (0.20.2 distribution), and start it with
one user on master node. When I switch to another user, I even cannot see
any processes for hadoop (use jps command) which means I cannot submit job
into this cluster user another user account. I see a lot of articles saying
that hadoop support multi-tenancy, can anyone tell me how to configure it?
Thanks a lot!

Best Regards,

J.P

--0016e6dd863c2d7bcd04a66a1b1c--

From general-return-3928-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 05:09:29 2011
Return-Path: <general-return-3928-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4725662E0
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 24 Jun 2011 05:09:29 +0000 (UTC)
Received: (qmail 65991 invoked by uid 500); 24 Jun 2011 05:09:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 64925 invoked by uid 500); 24 Jun 2011 05:09:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 64904 invoked by uid 99); 24 Jun 2011 05:08:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 05:08:58 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of harsh@cloudera.com designates 209.85.216.176 as permitted sender)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 05:08:51 +0000
Received: by qyk4 with SMTP id 4so1742010qyk.14
        for <general@hadoop.apache.org>; Thu, 23 Jun 2011 22:08:30 -0700 (PDT)
Received: by 10.229.49.133 with SMTP id v5mr2186787qcf.165.1308892110104; Thu,
 23 Jun 2011 22:08:30 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.105.7 with HTTP; Thu, 23 Jun 2011 22:08:10 -0700 (PDT)
In-Reply-To: <BANLkTi=V7pqOtAFbn=09wD2hJePWgmWV7A@mail.gmail.com>
References: <BANLkTi=V7pqOtAFbn=09wD2hJePWgmWV7A@mail.gmail.com>
From: Harsh J <harsh@cloudera.com>
Date: Fri, 24 Jun 2011 10:38:10 +0530
Message-ID: <BANLkTi=11t++3jCEz35mygTp=tCv9Jd6Vw@mail.gmail.com>
Subject: Re: How to configure a multi-tenancy hadoop cluster?
To: common-user@hadoop.apache.org
Cc: dujunping@gmail.com
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

(-general@, +common-user@ -- Please use general@ only for project wide
discussions)

User jobs do not need visibility of the java processes to submit jobs.

Specifically, are you facing any issues trying to run a job as another user=
?

On Fri, Jun 24, 2011 at 5:45 AM, =E4=BF=8A=E5=B9=B3=E5=A0=B5 <dujunping@gma=
il.com> wrote:
> Hello all,
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 I setup a hadoop cluster (0.20.2 distribution=
), and start it with
> one user on master node. When I switch to another user, I even cannot see
> any processes for hadoop (use jps command) which means I cannot submit jo=
b
> into this cluster user another user account. I see a lot of articles sayi=
ng
> that hadoop support multi-tenancy, can anyone tell me how to configure it=
?
> Thanks a lot!
>
> Best Regards,
>
> J.P
>



--=20
Harsh J

From general-return-3929-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 07:36:58 2011
Return-Path: <general-return-3929-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 64CCE6E2C
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 24 Jun 2011 07:36:58 +0000 (UTC)
Received: (qmail 16806 invoked by uid 500); 24 Jun 2011 07:36:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16005 invoked by uid 500); 24 Jun 2011 07:36:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15830 invoked by uid 99); 24 Jun 2011 07:36:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 07:36:38 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_FREEMAIL_DOC_PDF,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of owen.omalley@gmail.com designates 74.125.83.176 as permitted sender)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 07:36:27 +0000
Received: by pve37 with SMTP id 37so2280151pve.35
        for <general@hadoop.apache.org>; Fri, 24 Jun 2011 00:36:05 -0700 (PDT)
Received: by 10.68.1.170 with SMTP id 10mr1469880pbn.189.1308900965169;
        Fri, 24 Jun 2011 00:36:05 -0700 (PDT)
Received: from [192.168.1.103] (cpe-70-95-55-11.hawaii.res.rr.com [70.95.55.11])
        by mx.google.com with ESMTPS id b8sm1852741pbj.94.2011.06.24.00.36.00
        (version=SSLv3 cipher=OTHER);
        Fri, 24 Jun 2011 00:36:02 -0700 (PDT)
From: Owen O'Malley <omalley@apache.org>
Mime-Version: 1.0 (Apple Message framework v1084)
Content-Type: multipart/alternative; boundary=Apple-Mail-12--118723366
Subject: [RESULT] Powered by Logo
Date: Fri, 24 Jun 2011 00:35:58 -0700
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
To: general@hadoop.apache.org
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
Message-Id: <BAB91A74-E0CC-4A65-A0AC-4186014E473F@apache.org>
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-12--118723366
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

Ok,
   I collated the votes filtering on the subject line. I know that =
method missed some of the non-binding votes, but I don't believe it =
changed the outcome.

With the binding votes from the PMC:



First Round=09
2	4
4	4
5	6
6	1
Second Round (Drop 6)
2	5
4	4
5	6
Third Round (Drop 4)
2	8
5	7

=20
So in the very close race, #2 wins. My spreadsheets, which are attached =
below, show the counted votes.

With all of the binding and non-binding votes:



First Round=09
1	1
2	13
4	23
5	27
6	4
Second Round(Drop 1)
2	13
4	23
5	27
6	5
Third Round (Drop 6)
2	15
4	24
5	29
Forth Round (Drop 2)
4	35
5	31


-- Owen


--Apple-Mail-12--118723366
Content-Type: multipart/mixed;
	boundary=Apple-Mail-13--118723366


--Apple-Mail-13--118723366
Content-Transfer-Encoding: 7bit
Content-Type: text/html;
	charset=us-ascii

<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Ok,<div>&nbsp;&nbsp; I collated the votes filtering on the subject line. I know that method missed some of the non-binding votes, but I don't believe it changed the outcome.</div><div><br></div><div>With the binding votes from the PMC:</div><div><br></div><div><table border="0" cellpadding="0" cellspacing="0" width="150" style="border-collapse: collapse; position: static; z-index: auto; ">
<!--StartFragment-->
 <col width="75" span="2">
 <tbody><tr height="13">
  <td height="13" width="75"><br></td><td width="75"><br></td></tr></tbody></table><table border="0" cellpadding="0" cellspacing="0" width="150" style="border-collapse:
 collapse">
<!--StartFragment-->
 <col width="75" span="2">
 <tbody><tr height="13">
  <td height="13" width="75">First Round</td>
  <td width="75"></td>
 </tr>
 <tr height="13">
  <td height="13" align="right" x:num="2.0">2</td>
  <td align="right" x:num="4.0">4</td>
 </tr>
 <tr height="13">
  <td height="13" align="right" x:num="4.0">4</td>
  <td align="right" x:num="4.0">4</td>
 </tr>
 <tr height="13">
  <td height="13" align="right" x:num="5.0">5</td>
  <td align="right" x:num="6.0">6</td>
 </tr>
 <tr height="13">
  <td height="13" align="right" x:num="6.0">6</td>
  <td align="right" x:num="1.0">1</td>
 </tr>
 <tr height="13">
  <td height="13"></td>
  <td></td>
 </tr>
 <tr height="13">
  <td height="13" colspan="2">Second Round (Drop 6)</td>
 </tr>
 <tr height="13">
  <td height="13" align="right" x:num="2.0">2</td>
  <td align="right" x:num="5.0">5</td>
 </tr>
 <tr height="13">
  <td height="13" align="right" x:num="4.0">4</td>
  <td align="right" x:num="4.0">4</td>
 </tr>
 <tr height="13">
  <td height="13" align="right" x:num="5.0">5</td>
  <td align="right" x:num="6.0">6</td>
 </tr>
 <tr height="13">
  <td height="13"></td>
  <td></td>
 </tr>
 <tr height="13">
  <td height="13" colspan="2">Third Round (Drop 4)</td>
 </tr>
 <tr height="13">
  <td height="13" align="right" x:num="2.0">2</td>
  <td align="right" x:num="8.0">8</td>
 </tr>
 <tr height="13">
  <td height="13" align="right" x:num="5.0">5</td>
  <td align="right" x:num="7.0">7</td>
 </tr>
<!--EndFragment-->
</tbody></table>



<br>&nbsp;<br>



</div><div>So in the very close race, #2 wins. My spreadsheets, which are attached below, show the counted votes.</div><div><br></div><div>With all of the binding and non-binding votes:</div><div><br></div><div><table border="0" cellpadding="0" cellspacing="0" width="150" style="border-collapse: collapse; position: static; z-index: auto; ">
<!--StartFragment-->
 <col width="75" span="2">
 <tbody><tr height="13">
  <td height="13" width="75"><br></td><td width="75"><br></td></tr></tbody></table><table border="0" cellpadding="0" cellspacing="0" width="150" style="border-collapse:
 collapse">
<!--StartFragment-->
 <col width="75" span="2">
 <tbody><tr height="13">
  <td height="13" width="75">First Round</td>
  <td width="75"></td>
 </tr>
 <tr height="13">
  <td height="13" align="right" x:num="1.0">1</td>
  <td align="right" x:num="1.0">1</td>
 </tr>
 <tr height="13">
  <td height="13" align="right" x:num="2.0">2</td>
  <td align="right" x:num="13.0">13</td>
 </tr>
 <tr height="13">
  <td height="13" align="right" x:num="4.0">4</td>
  <td align="right" x:num="23.0">23</td>
 </tr>
 <tr height="13">
  <td height="13" align="right" x:num="5.0">5</td>
  <td align="right" x:num="27.0">27</td>
 </tr>
 <tr height="13">
  <td height="13" align="right" x:num="6.0">6</td>
  <td align="right" x:num="4.0">4</td>
 </tr>
 <tr height="13">
  <td height="13"></td>
  <td></td>
 </tr>
 <tr height="13">
  <td height="13" colspan="2">Second Round(Drop 1)</td>
 </tr>
 <tr height="15">
  <td height="15" align="right" x:num="2.0">2</td>
  <td align="right" x:num="13.0">13</td>
 </tr>
 <tr height="15">
  <td height="15" align="right" x:num="4.0">4</td>
  <td align="right" x:num="23.0">23</td>
 </tr>
 <tr height="13">
  <td height="13" align="right" x:num="5.0">5</td>
  <td align="right" x:num="27.0">27</td>
 </tr>
 <tr height="13">
  <td height="13" align="right" x:num="6.0">6</td>
  <td align="right" x:num="5.0">5</td>
 </tr>
 <tr height="13">
  <td height="13"></td>
  <td></td>
 </tr>
 <tr height="13">
  <td height="13" colspan="2">Third Round (Drop 6)</td>
 </tr>
 <tr height="13">
  <td height="13" align="right" x:num="2.0">2</td>
  <td align="right" x:num="15.0">15</td>
 </tr>
 <tr height="13">
  <td height="13" align="right" x:num="4.0">4</td>
  <td align="right" x:num="24.0">24</td>
 </tr>
 <tr height="13">
  <td height="13" align="right" x:num="5.0">5</td>
  <td align="right" x:num="29.0">29</td>
 </tr>
 <tr height="13">
  <td height="13"></td>
  <td></td>
 </tr>
 <tr height="13">
  <td height="13" colspan="2">Forth Round (Drop 2)</td>
 </tr>
 <tr height="13">
  <td height="13" align="right" x:num="4.0">4</td>
  <td align="right" x:num="35.0">35</td>
 </tr>
 <tr height="13">
  <td height="13" align="right" x:num="5.0">5</td>
  <td align="right" x:num="31.0">31</td>
 </tr>
<!--EndFragment-->
</tbody></table>



<br><br>



</div><div>-- Owen</div><div><br></div><div></div></body></html>
--Apple-Mail-13--118723366
Content-Disposition: inline;
	filename=Powered-By-All.pdf
Content-Type: application/pdf;
	x-mac-hide-extension=yes;
	x-unix-mode=0644;
	name="Powered-By-All.pdf"
Content-Transfer-Encoding: base64

JVBERi0xLjMKJcTl8uXrp/Og0MTGCjQgMCBvYmoKPDwgL0xlbmd0aCA1IDAgUiAvRmlsdGVyIC9G
bGF0ZURlY29kZSA+PgpzdHJlYW0KeAHVXWtv7Mhx/c5fwW8ZAbnjITlPwDC0dq7j2Lt+7L2w83AQ
WCNbiZNda3e010l+fYrsPqc5XV1SUaQSB4YxdzXkYU+zzqlHF5vf1L+qv6l327rZbNftcdPUu02z
7up9s1sf2vrb39e/qb+uv/ejS1OfL/Vm+N/lLKds1s1+tw9/Cf9sjvV2s960m82xOn9V//BjvW2H
7+Wj7eSbQ7uTI3bH9XZ3rD9+VX/vx816Uzf1xz/U/1Svvr4RzLZe/an/bOrVn2/qd5v1tlo9/kf4
V7363U197A/5r3ho3X9u69X3cYCBUROjUhi333yzjnBnXPqr/g9dvfrBTf3P9cef1u8/DrM07Sc3
+2a/3u8P1k9eNTf1xz++Evuw264PjQm9nwF9PO3Xh+5kYu9ejy2mcVqLNRhGsGpnQDdiYYeTPdvb
Gdjtfrvenjpz3N0M7EQMIWDTCBtyYvw0WjmsMxj5YMmD9T/9a7RfocfwB7AA9Hg/0EPsGQc8kFG/
Hyy9Wl1wLKn0x+EkIVs4xD65Xl1uH8CY391UwxD+DYzkld6cZMXpm2NSDUlWhp5hUU0iWRl7EZKV
oWdoQ5tIVsaeQ4REsjL2DLls9+12ve96/1TEFu/z44EI9YqG++1AKmHGE6jx5U0tMOKZQMTvhkOS
57qf4y7oIbvT9rQ+7DstBB9vxEsfxj7y/r6OhPvlMDjxheD4I/7BX4Rh4wvIBD7/1EPJz8PPwu+m
JDwB4f7+8VZ8cCPjTBeECoWDKs7Su0EKBJcDwQXPLk2oXog16jzWoONVM1kNscYETdDYURMAXV2r
9cqvCRoammBh+zVBYcPxWtB+3mpoaIKF7dcbjQ1NsLD9mqCxjyeJczfb+g2wu1ZkRsKzN4BOMnE8
7dbbZpdZoEjZZ70siRKAZvdByrQnp9P/OcIAUD+582soQCppyDXl972WSMQRg/h69ZJeJFUt6YWA
SSTi04vJuUnSizilmfJO0AuV9yCG6Axov15oaOqFge3XC4VNvTCg/ZzW0NQLA9vPaY1NvTCw/Tqn
sakXBvbk5EWy281mI37+PEqUu+ZwWO+6fW1ZTDP+Db3DtrNwrXlINrrjVlJvyU2T+6pCFv4jhP1w
5xAPem+qxt/0OrNLxIV4SHZR9Zl7Cg14CkOJ8z1iidFZQ74/Ous2qIpUACAvZ4wKSQkiCKBhtA/L
pO6FeeqrFcsogroFA/QyilDGnqMIyfzL2JPH/ZL5ly/TLmL+h6PUOJqTw/xhnrnnUzyBiX7dez5x
Vg/KR55pvelYYVB/7G3ucB+DIVerJ1g6T4bJv1G1qtNzs5jJG9CTTSfVFZlIW8Oe46litcqCnuOo
6ASNKRlbudjIMyKvHRWdoIG9iAoshf2CChiXaQ+jiturneCha9edhOnKCX4RE3vwFF4H/30PUn6K
B4LrP7upu20f9yKXhmzgTPgnnECHmAvA9SWrlbpkdI5S2gY2FQKjg1JcO8dqNck5qiAihctx/rIM
ZIJz1NhIrw1ov1JoaIbLBrZfKRR2S+e4PDZzSQN6xpSkXHJ/OMrCTZfdScklfxhjPNgY7JeloMt/
o9ILk/7QnyLLNrA6VoDxB+aEDClhsKw5AyvSQ9wgTk7Xo2fEmCwnWl850eCdQY23cqJxQkfSspgT
NaD9dqA9BtOCfdut94d9Nmyxg3BTU20Q4Qnu3AV6hTuHE3BzeK+hiPyDaGhfSEzrGTpbuCAKA5q6
LOTzInWG6rm6ZEpX3rnqkr1zqVeAn5JVKJVIwhmnOaPbIsJpQPutQw2bxfOujC3WgUwQtwdkfYQ1
SKiUFlLDgvP6tK/37XG9b4c1Z6kWbWtZHGxOp2NXvXLZWY9dLnA67nZ1t5MlvNMxzHgrMfdQCf5+
s99sf+BdhdXoB0mim22G3onBBPTN3d1psz/PuEK32/XL5lKkHI9/vGqOGWYgkQQSlIR2y1059Fl5
WD2vWOcfr9710j2q1FO7n9XLyrlEriYwMSL8PNGdatQUsAghihO38sfAetAMJMrQi8QRZeh2FPU+
nxXEUafouko5UyoxcdbHrRircYVpxNVGWi+2b8nVrlnv2uYgtr4Rhp5OgawjOu3u2jlk7da73SaH
T1pwON3vXg/fSb/Mru8/MEe/PcwSgsN6vznm8M8qwQO8JuT48p8xXcEXt6LLIrzVSmx2KNdL2jl8
Pkv3+R0xmKMszpjAdxXDsNAejSeH9rNSQyfCB8PMsf3OVWGz0G4MW27Ia3uEuFhvzbZfA/WwWWMw
xj2nNsJkysCePN1JBYuFduMy40rjr2QR+n+vR67bbjsJjwphOAItxKMxiJb8CKEXw+q8R+5LaRDY
9+V6tXB/lwcGKqa4lxh9KFTiKi9hyHqftARJ7DxqOuLIfEHFAipTnsZFVMaAXkRlDOw5dEUl04Ce
w1ZWMg3syWxNgUpLlTGw56gjVcbAnjzdL6iMcZllKpmyWLjeS/+tqmSy1oKM+Q7kvuAv4DQ4Tp6e
wxHVCqcwxyDq3SU/+/xsNhHXBd86vNDTMbf6kip8BvYM5rPCZ0BPtsQRg1jZ6brduh2v9/YzMirs
PCFLPP/7oPXprqOSkx9Ra7u46GNu77+LC2K9mxmKKdGu6hXM4OvoXWCRT1PKLCo2SUll8TeP5X+z
lgZ2KQb0Se/lPLnRLNVPpXF+fWpb8i80msn0/gJFJlIGPxI/GuTikX/Vz4Yk4V/E+4AbQ7fJfwAq
d/XkKWY8hAupwkYE/iOHenF1oeLdw0WgBA/jWs9uL9WQ3bHtOx2kLCILLodjP+UnKfPIv6TO84fK
+1CBzsTRYd/F+c8Lan7fo7GxEmFA+12mhmY+YWC37qBfYTOfMKD9HlND09Mb2H6d0tj09Aa2X141
Npp/DSvx1C/l8ql++UI/qhKkkUz0EfhhS5ngsy8fVWcvKAt5+E3USBT16KW5DIIjyX6umQALeguy
47+pTTZ47tWr1M1LDQHcW62pSDhYmL0JLbfqzrBoYED7iaihE8nLw/YLiMJOJC9D+4mooRPJy9h+
ImrsRPIytl+cNDbD+aVu5QvhvHGZZjz3r25MaA5SlryK1mJ33qi/QLyoLEwhikiRF/7yRIXAX4JA
VFzTheP+LkrLPQ7kmSEULAR6keLydMSLgUIK83A9BgrjMG9y8JXCvHy2Ypf/Es4/QudxhV8XlDdK
DUsGtp9gCpu6YED7JUdDUxcMbD93NTZ1wcAec6o3+ykdr9QFA3uylSRdKC6pxMtkLn7V+q+jtY1J
XNMc1sfDuG0xZnF4RJDMjc8EVnx6D9zuHwboUVKbLb5BnsHHCbOAolqxPSO/THq0AGAvXUYqlgg2
1lF+iH6LbyBMMrLd1cMECGDiOFIuMiv0EMOsrh5aHklMceIXCT3K93ScpU6NN0cSUx62X2K0LaKS
aAzbLzEaOklMedh+GdDYSWLK2H750thJYsrYk8edJKa4XmFMfXsapY2vDj02e1km7Qo9kSlFAfvk
2cNQqcCqBPgKEcAn5cLujX6SMlFfF0gRTeHkcACapVM5IzZLX/d5hYMhLs8Kw4zYI5+uBWOPCP0m
sYeB7RcG7cTppHqFaKSHJ9Wio5P6LJoLE0b+AxaFew7DCknvKNd8YraLQ+G1YD6AQpAJqJgVD0Zz
7UxeHcEu10hbnrIJ7kXdDma2BvQiEayBPceKWKspY3tqNfLTFqjVtCfZYePY7bJATgYQ6rIprQkW
Vq34RAg0B6b3Ido9TJJy+Cl/hDp2PlZpQ5NQupFHH7nQ8szZeZGmb+SKKy6kGgb3rCC+/hFKY9om
mLLyrzBlC9pvyhoaRRoL22/KChvJmAU9J1JCxGFhT042no84rMt048jmtRFHe9rIcod0NiV/EYsd
3NIAmk7xB5PAMFIPJQZ4h2+lGfgq2WHMwirpI5mQTsoypPvYWf6XEHGo6Vou4gB05rknpCLaDSWC
hbucY/sJprATwcrQfoJpaKQi1pT40wWNjVTEwh6TSoLYV1U7FsNOwlCqduAyI+4OC+2df+61biKQ
bI9bWTyUjaiSMGRr1oj57kLBINFTPY3wt3hIBQLxSDcK1iOQZOUE4kKluDy4z5b+J3hu7Ig0N0mZ
4ZPLE7mITzagF/HJBrZfMp6xrYNs/nE4pQV7rsT9OgZrtI9PN/Wpr2+FZjZUxGEtufXgv+lnBE+2
85G8Fk6MD8czJMuxcA0OQVniQzpFhiXgrJ2BEjzCte/Gqa8HrmOfBv0hLHhcpn/JCpXcsYbWxinP
PYA/VtHYcY3egvZboYam4zKG7bdChU3HZUD7xVND03EZ2Is4LgN7juOi6B86eUZlu82sRNKuZPKh
sgNLl1jwOsR74JP38YHY9CgizrlHhRmgCCjNTppwYMX1NiDpq5/l9g19r1KOGz4P8fMYP5GlJXdF
olMS1v2xwutZXFTyN+JimOXMtS7iEeINzKH9XNTDTlwsD9vPRYWduFiG9nNRQyculrH9XNTYDCKN
6fZzUWOTi/tjt97IY3oqAGM4RcsNBBpV6uCzfqkYcgHp6CO/xMZ68JH39/IU50AAwDAIuwc1HrJr
j55a60OtsSbIMjWvCfYlQSGOh32Vu7SktDmxL85rpnET2Kex4QkN6Ha0NCDKMiWn4GpSa2D72aeG
TfYZ0H72aWiyz8D2s09jk30Gtp99GjuxT55dPWzjI6vjB72w3AKbRsxJQwaNaNCBRdUKy8TwXjhQ
sYyVFlwDseCjeM5mU3EHZrn5g5ObQatq0jbLasJGtCpO2DK0KkMvQ6sy9iK0KkMvQ6sy9jK0KmOP
aJWeS93upbG9fy613ch221vZIHzTnU6ht3i0dfnru4zZvtlK66Y0Zl8/oyyRqayMHg/9Tqjp0WsQ
ShawJPuTPi1UDvDFb1ehMbxa/Zy05f4tOJj8BQsvv71BP/lfxwgRgB96hyld47IdhzyImtJNfE8H
ehGjLcanMrvD39uY6r7DpfJgmVu3YZyPDxASSBH+W9Vf+ZOm5KUyrhALy0+0NmOYuIu9mMPLu9g/
ozUwhfDg9II12AE4jwxm6MzIelvZb3DXZthiv7/OSxS4iyhEwPzoFWJ9TT8joq3tUyyfBNsykf8/
bNvR7oozuIiDMaBnqGmK2wzsOUbVr87KU0bWjMzxXYzbjGGPnMALjVGKvtKzG/bxt8Y9xzEybusd
0mFT2Jf3/SCo8kTsvTTcy/ylFOTbKObMiCRQawbRWw/fJKL9w00tDTqihtBefIJZioGQ4njRVHXg
RaXoIRVOAOETgGZ7ROotDINMEECetM+YykNTYBdnNMtDJ/BOYyNfMqD9vNPQrFYY2H4DVtjMlwxo
P+80NHlnYPvlQmOTdwa2n3caO/Gukf28d+N15LhchHwJTBCXdL3hlNjruFwwKqeDTT/raSihB/6b
0VTclq9aRdArThugoAvca9bvOjD0ukUJI2eoCIgX6oN9vDShWXrEODWX/Zr8BMYp7WUXxzZAZ2HI
IhvytAa237zUsBPjysP2s0JDJ8aVsf1s1tiJcWVsv8BpbDJO2CbPdcZ9esYVCsmIZOFYvBRSBzgV
lBEQUXKrt1hRrFZjLg7vehJiDetvIYDkCWTg00PxlOSd+i3hnn1VRXJcTHlItOAFq4W2npe5NDrM
2ziXb+HjDOgZfijFlga2n3Fa0BFbGtB+xmloMs7A9rNCY5NxBrafzRo7MU42INrvx93Z0cehsgdP
8YfeYckmnOGz7mk1EBLfCyWG/4ZD+zkYScu/3PGfT/IKnFBdIOu44BwIktweTxqfnzOwUsIAgbAY
aL/8YbHyYRfmNndMS6xPG9B+BioRHjGwPGw/AxU2fZ4xbD8DNXRiYHnYfpZo7MTAMraf3RqbDGy3
x3XXFLrbPwyMGz2DcO24QLTP42EweOyO8QBXOfZmQ0UPDAMCOXhhPQaXEpJKeplqbCD11C2iZUf6
H9wsstRVnq5FAkkDehFSGdhzLJ8W1JyGzb0ynZE63GfRNPIgKdfuvvq267ORVEYI6owDqcEwGdim
ecCXQ4QkARu61fIx3MOUuBUvsGmOOCKdej0q3cGeuwUdL4Il74Y+dvnJ/GlgxVu/E6iNt+stAjMD
2m/BKlJIbsHAbt0LwQqbbsGA9surhqZbMLD93kxj0y0Y2H6Xo7ETqbt8A64YmFGMYcpB50edEny/
IJlknDOy/qd/6bVCtP7uU2yUcHEp9vmF0K9QO1hE8qVlt9+LLNO3RSTfgPYTRjn2EWHKw/Zbh8JO
hClD+wmjoRNhyth+nmvsRJgytp+MGpuE2Rz7/cgyIxEnCFdFoYc7AX8g/Mhk4Ic+jykK/o7j8T3Z
dXFAh0Bq3ZNMnCL5mGPfnuFRL5ImDITEObyIbGa2CK2KEzaHVWllcCN3RXYv0zcDEQlcO34t5hb3
AnP8RZww6d4a1Wxwlq7dAPcJN/n2zLnGNRAg8wUoAnddWMWUFwRtxpOAqRgapyePAPyqo/wGi6EG
tJ9hGprLDwa2X9AUNgXNgG5nBBcUNAPbL5Z62BQ0A9uf4mtsCFpzOpR2pxJF+8lQDB21ZATWDKXO
QV7yWsyXaIcEBVIlBoS74CtggYFUOaSTkCycCTLioopx6b0AsxknKvqq5YfCXA7LD37GKccDxgE6
07pFnnGzsP1sVsMG4yxoPys0NBhnYfuFQmODcRa2n3EaOzGu7XfJ2me3UhiHBT+Q5LvolvhgAJ8X
xTdP2C0rdfjzYFLqrI5Ou1U8FJfJhQASPoCC6zgMDAtUXGiZvBEX3s/HG/gpC3qG4DPwtrD9JqJF
OS4hWNB+y9bQiTXl2faTXWMn1pSx/WzX2GTNcW/sIfVR4irr7fKf96Y7fjGMfmpaeSBuP3l/TydT
eBQH7BA2TAvrqvcf677r77VOJk3E1TtnlnAyETpTpmWcjIHtt2mtqKCLAe03Ow1NuhjYfrpobNLF
wPYriMZOdGmG/ZCyWylO5j26uSDjzPjOoEL+8ryY24xe5njpe35Ry5Ul8zHFQAxmRPgDLqgW5fID
0ohkmofsVKZk+CxVi4KzgnPC57vhZ06tvC6R8TbH4tTP6hSJbV8WtN+fPWMxh52x/xGjDsTmSH/Z
9of8GTfy6RwOqVIJAgu6sLEUnShUqm5qcPg/T6YlDh9m5y2CFAPaf1Of8ZqHjbEb0EsL77hPP0EW
iJuOkgc+cc/B7jvcej4WkB8BaJpAbgGADp+pIwaXKBqICITErC/lftIRA4XAVZZ5TLyJM50J7iKs
N6D9BvIM6/c7Ue9joaD5RYz6cd+5aEt1xn38xyHvZ1twapLAqbhvPBNfAIGWgHsCG9IXvYU14fN8
QX5yhz/hetKo2GyQx4y0CMdJLeEkaxE4H3/mln5TLENxkHU4KSUMU/wW0mFA+yMfPWzU4axh+61O
YbMqYAzbH/hoaAZsBrY/ztTYDNgMbH8wqLEZsO1OsmtMYYv994jXGE9FGqXmejpoSBsP5T9IPdTT
8qJ2Ull9zkS9lYBMxACsmqW3ar4Sq4rzNUFuNTSCrDLyIpwqQ8+xTeRAZeQ5lklGlaHn6AAJVYae
owOJT1tjp5zPYk0ATAjVsSq9jx5OCN7pQzwB/oE7QOMAui+4rWvo0avuAe3u7a1W716VyCyxgNTs
ilO4DMfK0DPufCrMGcOewwWyrDzsOQRONCtjL8OzMvYMTeMqbCPRY7/zTRbeSKUBRAusQC9vKqWB
JaARHlahrwJbYkoxWtCOPiq5wIyC9QoPqUwBw043Mi2h2iDvggz/wk8gGp2k4eaW6u7F7L5FWhFv
XA7t56BKKxIHDWy/MStsBo8GtJ+DGpocNLD9PNHY9HUGtl+WNDad3bYbdrzJbqVw8O9HuRkeSxkS
ZVoyvNq3valLBh3YWLGhhSnYp6sHWkDePI6sV3hZM9p8h2fMRm0VPJNbDSJslT0AsloiaUbCg4kG
7+pJm3LoGBBvZmvijGaqtojvM6D9vNPDZtJmYPt5p7AT74KN5TPi552GTrwrY/t5p7ET78rYft5p
bPKuOwx73WRzMvJ93NEz8k22dEK1C26Provd7rR6kjR1TAtJJblSdCAqT+FffFvchCr6rGRN6VNK
1uI8Zfo0gU0aG9maAe1nk4YmmwxsP5sUNtlkQPvZpKGTVbayXUA212KTKPlCQhHcwANQyrn7Szgy
dfzAR+AMSLkyZT5mdX2tVGRg0S63bFwCn/mlbvHGsNTxgKUFuBGcyt5/8gmNEPjlYfyjnmZ4lRxq
qVaHrr8zmVpMYIFSIrYHFYH9HNDAiQOlIbczOuaQSxWHPEfx6U2KyHP0nr6kiOznrJrmlEW1h2GX
m8w2hLVqN/n+FVUSqaU8Cuz7cUxW6AIeYO8gEb8BRcIXsrALgwdWcko4N152WJO76scH/SK4vF3E
eVlZrcFlbx/JUVGMIVDFTooZvOxWBQcKss5yWuqWJKdVviWL0NWAXoSwBvYSlDWg5xCApDWwF6Gt
gT1HbOhs22bYNkcT9+96OkoDAlwgnA0oxpVVkIAUuMA/XZ+K3KzGa1b0ShsJBcxHbtKBv1hXSf4U
vAoDxvqZPCG60HpYecaW4VXxZkx42kDLAR2hMexFeFUe9hz7TLwqY8/hLN2hMSVzOEteiQ4/vy3O
NTlGQRw8Fi0djEMAnPsnfB+dnrzjERC4BggZthGt/duIjladQau3clflCVuEVgb0Iu7KwJ5j+4gw
Deg5jCWtDOxFaGVgL0MraZWV+dHuCk1/jBFjDpdyNpCCD35+6D3cuPUPHoObAihmpUdCmCICFp9P
6UE3hTeuZYRlZg6XbH9znhVncBmelaGX4VkZe47Bkmdl6GV4VsaeIw90X33buObCahGeyY6l5R1v
vhiytxQWfjv896gvjWWYX2DlNxwiARiNnZXyjGD16qu4Cy7pgLjvGZBbOMQzLxBIJKPCV1NiP1Wa
EkGTPY73B2nHCtOSFacmkEdjoxBoQPvJo6EZ+xnYfvIobBYCDWi/gWtoOikD22/gGpvkMbD9pNfY
iP1Op2GvmkIJk8UQBGZPMGta7hkrv6BGzLPS84HwKvRjqdiB0gKsnoeOH5MKjke1t8Lv5IRbqHaI
Sck89wTq6NwmUseC9pu3hgZ1LGy/DSpsUMeC9lNHQ4M6FrZfTTQ2qGNh+6mjsUmdbb9Jzy5/mE7q
iCzqI86KjKn67XjCsmv2Tao3fJ4HeuAWmROIkzbp1YUIWT3u95oHS+ifSNxxbNfHlaOMjgfj7DfK
oU7l2VuEYwb0IhwzsOcYVIztrBmZQ19yzBj2IhwzsOdoAzh2PPYbxcibGtT7hVBgAJFC7TrtZgja
INfCcfBmcDc4jvRiLkauoizOpvcc4/mmXfFh0vkuznPas4u159lFpU4M/4yZW4JfFvQS/LKwF+CX
BT2HA+CXhT2HA/BhFvYcXSC/OtliZ38q1Cju+IgivVmItlLxWj/+fncW0lxtW093IvY/9Dwh3Lv0
pIlvHkcYh+9eerKo+NK7JZ4nlJe5D9PxBjmTBe0njQri2QJoYftJo7C5Wno4Dhv/NNmUSJTjeSsN
VBcq+1HMQ1pYU5sDDoCkjt97ENcmg90AgTodco9kjUxP3JB/AVEQZjfzb0uotAXtNzjtXJBpWNh+
g1PYyDQs6CVU2sKeo6RQaQt7jgeASh/aYd+gLrMS4SAXNsGfILOF1jfES4pHyMOvEbj2X62ej26E
nc7oJsRByxS3MCOZKk3gjVI89PdY0H7eaGjyJt7IfNh+41bY5I0B7aekhkZ0Y02J37Y1NnljjNvP
SY0N3uwPw243heLW+yEPF8NF57RskwkC5JkB32QawhNZsmQeDVcTWIfHLtIyvl07DmcKT7n+qa4/
kXehVPbGWTtmNFOiCbxTwg/eWdB+3mlo8M7C9vNOYYN3FrSfdxoavLOw/bzT2OCdhe3nncYm7/pM
93B85j07cDpwV4j8SQK4KaTvSBHAND4YCNYiHAQSDgQQ40a+2BtH4AxeG3FhKrGR8Exn3ppn5Rmc
wzNG9Dt5O9iu2xRqKj+7qbtt786ha0EK06NflELcDnxyUvGHR0Bk81/1T4ulnQJGu6PhTHzakH8B
98eYw9UcsYorBBZ0+/o+YCaKFvYcHQTpd/K+v+ZYiFFpVbizsCq0lycjSJ41HJPK4h9iWVyEo7ee
T/Ix1LgD/WXxFIoSzQvXyswvhLUR4XjsewbB5ACQOK/L6rC7tKD8rC6IQxZ4tDMtE/fGSc7jR7/Z
qWAJ7tdA9ludRob3NaD9Rqeg4XwN5J2bKhoZvteA9rtHDQ3Xa0D7vbqGBgm3u2EPokK1pmfhqX/o
sN+JJe32xNYgmjIYIycMBvzZTbWTEz4OTzfK+b9AqwQOpGeFqT/e41/glpByKP6FTZ/BTX3t2wd8
h88Helw4FGDjNRQcOI/EVd9o5QpznMfAfhKqyAkktKD9pqGhwUIL208WhQ0aWtB+hmto8NDC9guT
xgYRLWw/yTU2mbiRddXjoeAP85A2ui7dNAGC4QSEurR4HEAG3t2db+X/X8cXPoAIX0f6AQAUAqWm
+Cb1i7nitC3/4iViIgt6hoExJrKwl+CFMSNzbJe8MLDn2C55YWCPZOhX/wMJYvx/CmVuZHN0cmVh
bQplbmRvYmoKNSAwIG9iago3ODkzCmVuZG9iagoyIDAgb2JqCjw8IC9UeXBlIC9QYWdlIC9QYXJl
bnQgMyAwIFIgL1Jlc291cmNlcyA2IDAgUiAvQ29udGVudHMgNCAwIFIgL01lZGlhQm94IFswIDAg
NjEyIDc5Ml0KPj4KZW5kb2JqCjYgMCBvYmoKPDwgL1Byb2NTZXQgWyAvUERGIC9UZXh0IF0gL0Nv
bG9yU3BhY2UgPDwgL0NzMSA3IDAgUiA+PiAvRm9udCA8PCAvRjEuMCA4IDAgUgovRjMuMSAxMSAw
IFIgL0YyLjIgMTAgMCBSID4+ID4+CmVuZG9iagoxMyAwIG9iago8PCAvTGVuZ3RoIDE0IDAgUiAv
TiAzIC9BbHRlcm5hdGUgL0RldmljZVJHQiAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0K
eAGdlndUU9kWh8+9N73QEiIgJfQaegkg0jtIFQRRiUmAUAKGhCZ2RAVGFBEpVmRUwAFHhyJjRRQL
g4Ji1wnyEFDGwVFEReXdjGsJ7601896a/cdZ39nnt9fZZ+9917oAUPyCBMJ0WAGANKFYFO7rwVwS
E8vE9wIYEAEOWAHA4WZmBEf4RALU/L09mZmoSMaz9u4ugGS72yy/UCZz1v9/kSI3QyQGAApF1TY8
fiYX5QKUU7PFGTL/BMr0lSkyhjEyFqEJoqwi48SvbPan5iu7yZiXJuShGlnOGbw0noy7UN6aJeGj
jAShXJgl4GejfAdlvVRJmgDl9yjT0/icTAAwFJlfzOcmoWyJMkUUGe6J8gIACJTEObxyDov5OWie
AHimZ+SKBIlJYqYR15hp5ejIZvrxs1P5YjErlMNN4Yh4TM/0tAyOMBeAr2+WRQElWW2ZaJHtrRzt
7VnW5mj5v9nfHn5T/T3IevtV8Sbsz55BjJ5Z32zsrC+9FgD2JFqbHbO+lVUAtG0GQOXhrE/vIADy
BQC03pzzHoZsXpLE4gwnC4vs7GxzAZ9rLivoN/ufgm/Kv4Y595nL7vtWO6YXP4EjSRUzZUXlpqem
S0TMzAwOl89k/fcQ/+PAOWnNycMsnJ/AF/GF6FVR6JQJhIlou4U8gViQLmQKhH/V4X8YNicHGX6d
axRodV8AfYU5ULhJB8hvPQBDIwMkbj96An3rWxAxCsi+vGitka9zjzJ6/uf6Hwtcim7hTEEiU+b2
DI9kciWiLBmj34RswQISkAd0oAo0gS4wAixgDRyAM3AD3iAAhIBIEAOWAy5IAmlABLJBPtgACkEx
2AF2g2pwANSBetAEToI2cAZcBFfADXALDIBHQAqGwUswAd6BaQiC8BAVokGqkBakD5lC1hAbWgh5
Q0FQOBQDxUOJkBCSQPnQJqgYKoOqoUNQPfQjdBq6CF2D+qAH0CA0Bv0BfYQRmALTYQ3YALaA2bA7
HAhHwsvgRHgVnAcXwNvhSrgWPg63whfhG/AALIVfwpMIQMgIA9FGWAgb8URCkFgkAREha5EipAKp
RZqQDqQbuY1IkXHkAwaHoWGYGBbGGeOHWYzhYlZh1mJKMNWYY5hWTBfmNmYQM4H5gqVi1bGmWCes
P3YJNhGbjS3EVmCPYFuwl7ED2GHsOxwOx8AZ4hxwfrgYXDJuNa4Etw/XjLuA68MN4SbxeLwq3hTv
gg/Bc/BifCG+Cn8cfx7fjx/GvyeQCVoEa4IPIZYgJGwkVBAaCOcI/YQRwjRRgahPdCKGEHnEXGIp
sY7YQbxJHCZOkxRJhiQXUiQpmbSBVElqIl0mPSa9IZPJOmRHchhZQF5PriSfIF8lD5I/UJQoJhRP
ShxFQtlOOUq5QHlAeUOlUg2obtRYqpi6nVpPvUR9Sn0vR5Mzl/OX48mtk6uRa5Xrl3slT5TXl3eX
Xy6fJ18hf0r+pvy4AlHBQMFTgaOwVqFG4bTCPYVJRZqilWKIYppiiWKD4jXFUSW8koGStxJPqUDp
sNIlpSEaQtOledK4tE20Otpl2jAdRzek+9OT6cX0H+i99AllJWVb5SjlHOUa5bPKUgbCMGD4M1IZ
pYyTjLuMj/M05rnP48/bNq9pXv+8KZX5Km4qfJUilWaVAZWPqkxVb9UU1Z2qbapP1DBqJmphatlq
+9Uuq43Pp893ns+dXzT/5PyH6rC6iXq4+mr1w+o96pMamhq+GhkaVRqXNMY1GZpumsma5ZrnNMe0
aFoLtQRa5VrntV4wlZnuzFRmJbOLOaGtru2nLdE+pN2rPa1jqLNYZ6NOs84TXZIuWzdBt1y3U3dC
T0svWC9fr1HvoT5Rn62fpL9Hv1t/ysDQINpgi0GbwaihiqG/YZ5ho+FjI6qRq9Eqo1qjO8Y4Y7Zx
ivE+41smsImdSZJJjclNU9jU3lRgus+0zwxr5mgmNKs1u8eisNxZWaxG1qA5wzzIfKN5m/krCz2L
WIudFt0WXyztLFMt6ywfWSlZBVhttOqw+sPaxJprXWN9x4Zq42Ozzqbd5rWtqS3fdr/tfTuaXbDd
FrtOu8/2DvYi+yb7MQc9h3iHvQ732HR2KLuEfdUR6+jhuM7xjOMHJ3snsdNJp9+dWc4pzg3OowsM
F/AX1C0YctFx4bgccpEuZC6MX3hwodRV25XjWuv6zE3Xjed2xG3E3dg92f24+ysPSw+RR4vHlKeT
5xrPC16Il69XkVevt5L3Yu9q76c+Oj6JPo0+E752vqt9L/hh/QL9dvrd89fw5/rX+08EOASsCegK
pARGBFYHPgsyCRIFdQTDwQHBu4IfL9JfJFzUFgJC/EN2hTwJNQxdFfpzGC4sNKwm7Hm4VXh+eHcE
LWJFREPEu0iPyNLIR4uNFksWd0bJR8VF1UdNRXtFl0VLl1gsWbPkRoxajCCmPRYfGxV7JHZyqffS
3UuH4+ziCuPuLjNclrPs2nK15anLz66QX8FZcSoeGx8d3xD/iRPCqeVMrvRfuXflBNeTu4f7kufG
K+eN8V34ZfyRBJeEsoTRRJfEXYljSa5JFUnjAk9BteB1sl/ygeSplJCUoykzqdGpzWmEtPi000Il
YYqwK10zPSe9L8M0ozBDuspp1e5VE6JA0ZFMKHNZZruYjv5M9UiMJJslg1kLs2qy3mdHZZ/KUcwR
5vTkmuRuyx3J88n7fjVmNXd1Z752/ob8wTXuaw6thdauXNu5Tnddwbrh9b7rj20gbUjZ8MtGy41l
G99uit7UUaBRsL5gaLPv5sZCuUJR4b0tzlsObMVsFWzt3WazrWrblyJe0fViy+KK4k8l3JLr31l9
V/ndzPaE7b2l9qX7d+B2CHfc3em681iZYlle2dCu4F2t5czyovK3u1fsvlZhW3FgD2mPZI+0Mqiy
vUqvakfVp+qk6oEaj5rmvep7t+2d2sfb17/fbX/TAY0DxQc+HhQcvH/I91BrrUFtxWHc4azDz+ui
6rq/Z39ff0TtSPGRz0eFR6XHwo911TvU1zeoN5Q2wo2SxrHjccdv/eD1Q3sTq+lQM6O5+AQ4ITnx
4sf4H++eDDzZeYp9qukn/Z/2ttBailqh1tzWibakNml7THvf6YDTnR3OHS0/m/989Iz2mZqzymdL
z5HOFZybOZ93fvJCxoXxi4kXhzpXdD66tOTSna6wrt7LgZevXvG5cqnbvfv8VZerZ645XTt9nX29
7Yb9jdYeu56WX+x+aem172296XCz/ZbjrY6+BX3n+l37L972un3ljv+dGwOLBvruLr57/17cPel9
3v3RB6kPXj/Mejj9aP1j7OOiJwpPKp6qP6391fjXZqm99Oyg12DPs4hnj4a4Qy//lfmvT8MFz6nP
K0a0RupHrUfPjPmM3Xqx9MXwy4yX0+OFvyn+tveV0auffnf7vWdiycTwa9HrmT9K3qi+OfrW9m3n
ZOjk03dp76anit6rvj/2gf2h+2P0x5Hp7E/4T5WfjT93fAn88ngmbWbm3/eE8/sKZW5kc3RyZWFt
CmVuZG9iagoxNCAwIG9iagoyNjEyCmVuZG9iago3IDAgb2JqClsgL0lDQ0Jhc2VkIDEzIDAgUiBd
CmVuZG9iagozIDAgb2JqCjw8IC9UeXBlIC9QYWdlcyAvTWVkaWFCb3ggWzAgMCA2MTIgNzkyXSAv
Q291bnQgMSAvS2lkcyBbIDIgMCBSIF0gPj4KZW5kb2JqCjE1IDAgb2JqCjw8IC9UeXBlIC9DYXRh
bG9nIC9QYWdlcyAzIDAgUiA+PgplbmRvYmoKMTYgMCBvYmoKPDwgL0xlbmd0aCAxNyAwIFIgL0xl
bmd0aDEgMjE4NzIgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCngBrbwJYFTV2Td+zt3m
zr5kZpJMlpnJZJ+ECVkJBOaSBRICJiyBBIhJgLBEkACBstVQFUFURG1darVura1tNQIiaFopWkR9
baVa29pq1apVK2ottb6QmXy/c+4korXv1//7/Se59z7nruc859mf596BjZt7iYXsIiLRlq/r6Sf8
Z6zH5pfLtwwE9La0gBB668r+Vev0tvwBIcLkVWu3rdTbpvsIaRpZ3duzQm+TEWwrV2OH3qbl2Gav
XjewVW+r+bhf+tr1yxPHTW7sP7CuZ2vi+eSPaAcu7VnXq5/f1sra/es3DejtBXdhO9S/sTdxPm1H
f9gzL/jZ8Ag0Z5JVxEAuITIRiINEyEKcaaKnMF7Kj8vC9QfJj1/ustf8g6Sp/PoHLqsvYMCJJ3/0
7og5dpP5JpXdyYg76D+0VHN8HiGWohHzqMF8E79T4iDfzDxGFowe1zIPFpRWOg4GDmoHWw/2H9x1
8K6DQwdfOPj6QdPxgx8fFI7ilP5HklMq/fXUvtC/UGhp62oT1i+g313w0AJh7vxkad58rzR/nkea
1TRPmtFUJc1sKpUasTRVVEs10VJpanSqNC0alOqiGVJtdJ40HYuGJVpRKpWWrZDKKsqlivIFUnlF
pvRC+evlH5eLR0c/PHQ4p7Hy6Ojrhw47Qth+qFkPG+2Vh32N0pZDVx1Ctz4+dIifcU4bPWTMrjzk
bpSu3psk9a/t3yrYv/OnOwXtDm9qpfYdb1qldksyoJuT0yqv2p3kt19p323fb7/efsB/pX+///rI
/l27d+29/oYDuw/sObDXrl1udFTaN/o3CtoGo6XSvo4GTtHA0zR68qOTQuAX2i8EsoySZY5lgtZz
V49gX0KL3U6pyJ0jhd3VUqE7SSpweyS/O1MKBuqkgLtGesbXIPnSZkppvhrJ5y6VPDgvCd11uX2S
E0u/m2ru6XWVdluhnyjU+mSz33Ki2W863uw3YpGHm/3ST5v94rFmv/BYs58eafaTR5v9T54o9B9/
otD/U23hcND/2LGg/9EjQf+JJ5+yPnH859bhn/7Mcuyxxy1HHj1qcQzvGha0Y7uOCfYj0SMtRwaP
SPYjEYDrAT5x5FdHRo+oJmOVZLEKsiSIgkCJ0CrTo3SUDrmaSfOC2qEkiu382oeNpeHmoRXzandf
d13G0M3N89qHdmV0HFVxTvsQHaL7O4bU5vkJkITZb9PApk0c+NJqSGwYUhpW9wwpofpNrGFjDVuo
HsCQncH2UH2YDrkbVg+5Af3LTTaN/XBIP6g/iMNk85cex5usLwPoUTisZCpu+WP5tLRT6hRfBdeR
0b+MvhHfGl8R7xC/SfzgkJvJA+QYOUl+Oc40w+QEh7eQg+Q4eW58PwO+Qb5Jvk/+i7xCPhrffyu5
k/yIDI23GXCAsL33kR+SB8kh8hh5Evv2khuw93vkxxecuZ7sIdeT28ld5EWakdj/pOCmeg/eIxbh
NN1E9xMfKSL1ZCnZRC4jV6Ffp+hs7JuKfa3Yu5FsJTdi7zFy6oJ7j4FTIWk6SR+5lDyMM37Odxfi
2gVkBfayffpvA9lOriZ3k/vJ42Q94D3o77fHbnLB9htCUAiSAfI2rnyWfks4iRHdT3YrbmIiRD7N
sCp1ctyS0TcIia8Y/Qch4jLhrHCPcAN5SOgjszVP24KqytLIhOKi3By/O8nltFpAk0WBITGnIdQQ
6lm9L9CwOrAvVN9dX1wE+muoTwsGO4qLAthdHxii3YGGoRlbVqfsa2AnDLnCQ0JOA1v6hrRrugGE
6oPBII4kfX4EMu7aCw4F1gxpPUPkmsDDRcf3XXvUQZZ1hy0rQit6lrYPiT141sMEnVkNiseGLd2r
A0MSbsxXadiT6CI7trob61A9rvrK/dhtrGvfEzyeNuTCtmHIGR6aiTvN3P5WmrivIWVNgDX37dsT
GLprbvuFR4PsnI6OjpQvoGFGaEb3vn0zQoEZ+7r39Rwd3bUsFHCE9j3c3Lyvv6E7MERawZzY/9g1
aUMzru0YcnSvppOBPTaOGfPao2lBJ7oaDLLxXnNUI8vQGNo1t11vB8iytINEi4Q7hoRuduT42BFP
Gzuya+zI+OXdIY7runYxTcCNm+eHmucubg807OtOzFtizyR9Fh8WSO3DIbp37sMa3Tt/cfsxMGZg
74L2gwIV6rprOxgahboF7RdehSsZBUDfQcUJL0B35pL7xHvII1huEt4jN2F7G7Yx2Uu9spfcjeV+
LHOwPI7laizfxrIdywblWhJXrqUWaNkheSvxyTvJKflmskkpxNZOTkm3kVNKOdoiOSVeTK4W7yVF
8jXkeWkA+3+Gc0awnU02SS8CPoBlK7lK+hDUj3sZ3iNL5R2kXvqAKNheJntHR6RfktXSCDkmhcka
bNdKT5E+jKGewbKDHBNKwRc/Hz0hDZOfA37ccIocY/ulP5A+fh3OE/v49ZeKeWQajj2Mc6diHAux
rWGwVA4uZ5YGsxBgEBAF0gt4JUsSe9he9hNgcUgcknGOgUMqbAoTMeMqK7ERO5OU+DmxuLAkETfx
EC9JJikkFXInjaSTDJIJHg+QIMkiIZyTTXJILrZ5JJ8UkEISBqz/irApJhNg8ZSQiaSUlJFyUkEq
SRWZRKrJZDKF1EAaTSNRopHppJbUQao1kBmJq/14yiRyLXmPJtGJdA59UJgnKuI3paB0QorJv1XW
K4eVc4Y/qZ3GKuPDpjrTvWaD+UeWMssOy++s663HbDvsefanHWuc6c5/uupddyXlJt3vLnSvc1/h
2e2t976YvDclK+XHqft9c3wH0xrS/pielzE7U8580D8QKA78OHhPVl7W70MVoV9lX5w9nB0n0JkE
KFMg0LBxP6oIEmFL5PlXn+eriSVBZ9CZgxXFWed2yeQ82xIAbG7uE0/TyyEpReLRTIIgUrtMSZEY
OdNJomWRiSVUDIn08vDesOKO/VSowxWPxFeIdlzhIQu1OiM1GlJpqiFfzJdbaKPYKLcYumiXYT1d
bxikW4WtyqDBZaDUsl2iagmbd7sF6r7NbjEasfZLe72Os2fC4bJIZw2eGMUDO2koV3A6XFVlHkUx
KILH7Ur2epNF+9sPP/XUw2/PvSla09w0rebbc+IrnqOv02L8vf6cqemJwR3x3933o/hbu3Y83cBG
dlN8hXCG97NPq1ZEJckjepJyaa6Ym5TrmUk1UUua6WkVW5O6xe6kbWSL0C/2J21xe1xUsmwm1BWV
qCSZj46ePcw6zADNzjpt9hOL3S60kRuTHZ+Gv9x3h2AIVVRWVlW6KsqFvNzcvIoyr0s4g47PuX3K
tKZZU6M3zcVAhJr4i/HAc6aGp3fsouk/uo/m7xh8osn0XDzAe75fiEL7iqRCy/XRQhoWKki10EAa
8dQOYQVU3S9ECyZrIbhGEnyCIEQ6yyLEcbaUzVinkYaShGj8nRsfpBmxDcL1DBu3CRNFo/AO7hnQ
3LTWDsFml1tIi9xFumSQghDBFJBI5xncIFgRFI2xA0K/MPERdm0Mqw94fwJHhIVUJcX06Oi7molh
IUKjVKCRzvAZEsW1wZCzjH7w0Uc4m1Ivelcjvwzu3atVyDMUxSLaxEaq2p1+pyALfju12y02i8Ui
tNmsFovSZgsIUXG92C+KosXhENpgkL+umdljRC9DPdpntQyzGVAmu0pUrJga0WG1KlizO4iRMfur
swxTU10a6YyUgbBipaBm1j+MzRmsKMUEVVaVOYNSzcgrtDL+bPRAzoQK6XZacqv4zl6PO3XO9HMn
MPK7MYIb5I8hVd7Q5rb6u/2CLCpOr+hxZjunyJOsFbZoRjSz2t8sN1obbC0ZLZlN/i6xU+qUlxgX
OrtSL07rTO/K6MrsE1covc5lnvWZ/cKAc9A3mD6YmYPRvHuYdRpexbtalEHE7rAXq5H0ErtmV+wa
GyPWGJ3dbp6VJAj+WVT1C2rQyynQy5nIKzGEeBlqUtkFXi+7k9cbuDPLnuXPEoDI24KOT4EJtuLo
OeOq5ig5U4Z258QSrGgnUMPotrKiPDeUpXAqLmNOA3iQ/welG0Yca15ccnz/bVcveanXNPPM+rep
FC7MW9N8yVvLxeDpxYc7HvvD4MAVWu2vQ5Nf/WnbTbXTtjat+cUC4PH+0TekncDjVHJU2282yxGf
2RMpMOdGCmpqzBXuiVnlkVnmBnddVl1kIe2QO8xtkT7zykhfzVbzlshAxY4aX/nk+snClMnALy12
FgvFxQWz/MaJgt3qtwpWq3OW0RQKVnFSqpIYUVQpDAtVmRO8QXFC5mQ4GaKPkwzOAZncFbVH/VHB
8u1pjnc6He+Ew87kaseZSIThh1EymCHqqmabSKy6GijqpF6vjotQFmdqbxknImALXB4KVaCZgMeR
54XcYtd4vF7JVjJtVl3zc9t2fjzH3vbOJdH9RROKy4qLd81aPOPWRyYUhJdN63q5i+F03ffrGmc9
9LWSncLz4ctXrXwgOqNuSuj0pFmFBUV9c1vXZPqTvz+4vXKuz+eun3Y6NCW/qGTvkp3HUmxqGfTo
HNDrIVi8JmjN17WgZHVbs63l1nprv1WxpLCRW6yNmE6LYjBam6jMvGwQjdImy6JBFNWoqcUkgK/9
dsFgkbjsA0UB+LsGPlXapIDJaFVaFAoB8JZmZoRHTQytaJ87zPAO4EOtlJ1LqV3xK1FI3lzGy4rA
+FhJYXysWNiVmCFcqfBuKQm+hRhjFNrpqi7j/FvdCakUxVREwjWxUld1NcTTHkdMOh5mesIZAi/T
MmdZ0EmlQ68ej1UJp4++Gl8ee4LeG++k974jNo5sFO6KdTMp9jhocDdwU0hu0aaaVJ8aVqeqFc6p
3ma13rlYXVDQp25XLRkZviZ7jj9HyxFzgrNylEzBbvIDJybbLMUUyAq0ZNCMo6MfaBPYCDLAaljb
2MAzuPzJcAdJIMNI+AHy3SJ7kb9IMN4e1qnMWc2IjA/wTORzKot0xpjg7qSc176SvkBTzqAn6Kys
LCtlZCXtnl3b+PQV29+8yDbvj30zd5cXFVdEyr+1tP3eKeKu2PTw4uC2I7Nb2+nvV/9s+ozmsuwX
y5vyS8NbW+b0BXL9KRZh9KH4gCQVlFc9CKq5Gpi5Xz4Du6mKfEubpFi91uqcsollVU05tRPrqrro
QmtroDXYG9w80eYTC5oykpKSZ2WIdqGiSTT5iiKuUJC4jAQzf5grRiaKOG0QPsM48IFmZzgi36m2
V/urhUjQiHOOsJONt05iWpTzILiQ4QezDNxgzoGVCEQVxwxhaMkVKspdVZXZDAeeENguixjGMGLQ
GRGIUhSPm+EIHCndH//N7y893LCws62znXqPTWktMKVvmPLbUeJZcO8lXTfMbu94rio6oX9q241z
BGF69YRLojd8n/75z/E36uvmU9fPT9LSr20YNFmfsKfF//52WUWoYupj13VuLw648wu9Bf47H60o
KngYtAWvVPoWaEshLVqpkZqEbNpIm4V2YRvYCgZwAMoaOlacJYuCalf96g5BFIkgSHZcTCSme0Hi
rmpG6zFnNdPiexxn9hwHkdMQ08fSt2IXvyw8PzIknpc+OW+Tsx6CJt8++pr0HfnvsIDzSRVVj5Fc
GCtWIDWHcS4DsseA0BiQBUDbwo4Vh8s9ZVnleeVl9Z7pWfV5DWWtniWpi9MW+xdkdYU7iromLihb
UNWtLrMtcy1L7Q51522xbXHtKLrKlaEIP8z9fkTI9Zoikpgx0yFUNIIQAiSJJiWRiMlaECTe3ECC
Ce7Q5zwQtLJHc21utZYGlVsw8UwBYe7fwsRj5rFylkU2MPkbPeNKru6Ex6VldhRdXSQUFJWKFZGC
SCV88YWhFaHbchVfICTmZjjZeXzVATphrBWmbhBLdkV5ZVVFbm5FOaiFazFIYZGrNp06kisrkzjJ
5HF6YcTynfiLb30Sf+PAFVs3Ufdv/kRNl22/9ptnvrfrsrvnzsu5pnb5bP/cLZH+zsXrHrv+xofo
d38+Ss49ufOZKYp268YfvP7y93qfrFJqhoSWSwa3rmxcU+CanFS7P7Zp6fpJ3tysiT/o2zN0M3ht
w+ifpevljzivXalVq1KqVCDV5NSEKybMzpkdrpvQLnUld6bMS+unO3LszozSJndBk1vJSEigCqcR
zGb0gafOaiHObZA74Kw7dSwXBX0OJmF9Etvru5kxV4K7OG8x1gJbMd02zlqCQZE+FzquKkXhvAbU
ES5sXOOsNc5XVZWV0vUdi5fEPzxWvjTblNE3/dXz7s77epZ+q7m9gxb9bu3Rhralz2qTImujB+6v
1IrX1l501wwqirVPxk/0b9xptoChqPG9SSXZ5VOHr3iLZtbVzY+fv+/24fLivMP3dm0t9nsK8z0F
YI44/OVd0nPgrK7Dm2RahIG/qeWaUqrETLbyw4kTRMZFmkpLVE0VJPgvRKOt3CotY8Kkk9m1ZZ2Q
uxNLECTUzLK0QBREhRkm0CX4l4930M4cWkE9lMq7zl8mXT6yUdw/R6A7BXo4vjm+GfqDWtCT/bwn
PYd3irQImu5NLR+dkFlPZD+NUEG2U0rHeiKynkRoCdWoCAv5q/siSgtkWREW0C/0haIv+Jf3n79M
3D+yUbpcuCg2epheQ685HBsFFQ2NZhouAhXVkhbysnYPdKtklk1Ge5ot3R61a17BL6XL/jR/ujvL
neeP+qfnCEVSkRxJi6RnZwXyItHI9JlafePCxgyTLOd1NF1q7LWu8a0O9uatjK6cPuDdkdafN1A9
MMXukp2qa+Y8m1vzpFW5JWnOfLW4uHCuTZ06MXPuxKmCHe6X7KxzFbtnuaJmarbPDcwVUgrc5io7
CZBBCCmSXVB1SStndci46jNQ6GciZbFS2KPwMLm0hxSIYHk+jDVO4aoQylAwBMGboEUIfUaIFVD0
zDyFW5UdypLGXEOJGWJwFcHzFygIKdnrok6mIpg2wCmGi+Y2xxvTZu1t/8Ezn/6oqX/6d/9eGF7S
3h4f+d5d8X92da9b3bWcmu5Y+OiCnh90PBY/sXHTrqsGBui0R56i5X19G2LXR1dUf+PGgR11Vwm3
XBMfuWSgRou/9WdqCwZLRo40v9FxH7V0d68aWLYs/tG3vxf/qKd3lTdlv8c+uHETrX3yGI1u3nzV
zv7++C/imqBkpB7+/r33T2ORDx9iNBdDcxhgsz11SDMym+wz7hUoY4CBi2xmUs1SGgyC0WhShT2g
MzcuMprEPfBT3aChncomgyCWmzRmbZk0pm9LTJqp3ySajCZFpNtlKqt2C4XpJ8oWhEeqQT6LEXAd
wH3IpRYcMslhuVKeI7fJvfIO2SCvMMNugdsA7QQpS6KdNVFMD6Qs8xI7wTidx48f1zcqdBVkL+kM
hsSgCJ2VBFa6+MUbYztvfEbIpOrO+Pn4OfrdeI98emSr8IdYDkgDsWE88DTxoDNl1Kq5ZIvHkmdp
ExZ6BlMVl7OoPJMZqG4m5jIzDRnlqlhcblC9HleRnesSZoPZA0zSof2Jls7GbM9l3iDby7wmQ46H
cFcScuOdMeuEeV0QmbiEm6wA3gIA74v5Ydx7Ihsq7BVahZBZ5DbYGD4xByOH2eMAvKIrMYPKBC3a
H/LbAXif3w7A3/jtGHCEdcOwtpyJYf0XKwVCdRC2jW72QhrrOg/tM9zGHbMF4axyey/hRtCEt4G9
zFXTFViIncSacnjBzAXP3B77mB67955Z82atXXzLg/FD2fmRq5Z/QEnnpZFI3mDlzJKrl8Wfocrl
36+YVE6fXf9AVe0k+XRKbnjPxX3fKlb9zwlS5azkNGt8XlJmZlfs24v7clLtsZfTsvOQwKRk0+jb
8gzkOAvJem2hTK1Gxe2laUa3J8dT6alzL1HbTe22JY4l+d1ij7tf2GLvdyd5vb5yl1BYmFuumBDr
hFFMmV0cKYoWrS+SAx6Lm2HWks7waVkd1lUWsFTDTWTmjDFyy+Fu6Be5fMwM/ILRV1Umz6jqaJx6
/cJ74v9c1r129bIuar1v60c32nd8sm/DIzMb5rTVzXh89fXn1tnWphQmJ6Ut6emiOSeO0qwVPSsn
N/111cVNc5rfvvmON2fOmrlsGXiU0elB0KkNMcZhLVTtanKtEVZbJS8IMhkEuYVQu4eYGJ0QhQ0F
hPTmGLGdPcyGhT3vj1Ed/HRObQN+uz/i1xBVkJL//yGzzDHK4jp/jMrAuLCnGAqZTIWgZPTC5GNC
ryNko5PPwYdvWnn++fheOvAKpR23PvCr7dvaT+577LHrf9Oxfr3wl+fiR5ZEQSvRqq74Uy8/9HFD
ad75KwqrZ77L6AI4ku4AjszkwKPGCqI4FAEC7LiWzZhXUahcIYimCqpKREUAaZPVbqWK0U05c0GT
6swFIMFclDMXcyY51gDozAXgPc5cDODMRddaPh81Z66aTkY4sCDBTRt4zIe7Tsx9Ckp3jITF34z8
TbSzRT49FF89FPttov+D6L+R3H4QfWVd97CuC4KBVqiiQSVii5kZ7UdHX9J8fP5WmO1mqHzKWFz+
X03g62Ny4oOEnDBdMJQwBMXZMFeT+ljYFGIIGAbcXmkwZhf2xLadFB+Vg/GlQ7EydJ7z55/le8Cf
2eRZbYqBGhXFlqEk2YK2ClsTnW6ba+tVes3LbQO2gXR7VoUWoqGQRXQ4ksstQka5aNpipFmOLKMj
eHQ0riUxyg2uJRKnbEeCsj8do+y3/oWyWQiAi9HzWohhhWzOtedquYLPY3Sxq42cxY0WdpZxdQ4f
LMz/Mdl4Jgz1H2F2AWavLOJEg8cjwfowULl2dxBGusz3Y1GXLOJkOyqrEAYIyvdsj/9pz4Px11au
6qd307WD1Hiby7+luuGh9efir8IMU7qfaIxvEOZfOml+d3cPDT1Je+kdU5v+mnKRz18QfyL+YfxP
8SdyM+m6B3V6kKdwen7zoFihMnrwcnNbdaiCqsom+HWyahTc8GqOY9icxc8d5piCbuK6ArTyjq4r
CCdnJgS0bH6ui2sgOMVAUhJH1VawhGZttYqq6IYN8Gt+KwA6ZwBIcAYeillBW2cIADqLcIDdjwGc
nOR/4YwxHRSOlZJoTbQGCN7AdDmitoyoEA91lslTTsZST54U/nJS+H0sTz4dOyo0Ah9Xw1j5DcdH
rxYySqWKaBJLqWpdZ1LNi01uURYWJ2K2PLaD2OZrnE4A6HQCIM55GcCHRxgSxHXjnHu21BHD8g6P
SsZKGaEHYe6FKoIedEv4Tezwk08Ks5988lbp7ltvPd+F/hSN/lV4H/3xkA2au4/upIKrzCMaDOZy
0ZiU5HID12+Ozca5MYr9cIxidZThnFeYLMbcufhMCHwmvpZsT6ZKr5fFbMZJE+F17jgBW0w3f+4v
McVbwehPeP+jZ0ruqjQXbI0uXedLs8d/IVB6xVMvOS3HbJmFefkDs8XeO9Hz56FNNnNM/rc2mGpc
SJfCHjPmGycZZxlXG/cZf2c02KnJmElTBUT+jdW02lhhbqJNxgbzUtpr3ki2qQ6EE/bSU3BxDkHg
qsZDghlW3pUmqgoJwoSoMpns1gAyXBqRWvHYS0FdVBU59YDCuOAFqfxb8hqnqnE6Y9YSJ69PvoK8
mLT6tLOzVI/ecNpisToYicePb4+lSMcR0Nge60xhduKGjcEgNXCCo2VU3hwfjR36JkjuxY9jq4Rb
7owbYCP+ExmIenSb6ZUtwJZMNh8RRAkaBLP2rqbzDDHYDTB+/5+0yPsJLaKMzzX8ZYQhwkwKsYQB
Ywtpy8i8k8K78ulzf9L7JH+KPllok7ZkoYlOEibJlab1Qre4Xu42DSKRNCj3m8xtxoWmxWZxhTgg
bsYUmwTRqAhEkLiBLmmMjSVupkPASvXSAgk/g9koUliWJjM4CjFVzcqVkJtkjkkWbQ6Xw3oKivv6
hEdgCZQSCDmFEzIPcpOtNrvNb2u1wdznMoNH42REBjCLSQYHuyNs1P8Pdi0jBZizYwLGsNZ6AdIg
XD7nF+TFEFEfEzLMM0As53C51CcJnR0AD66QaGcHvApGIxtJ50aIIES6mGajNCh/ejK+bEu89xi1
0evoLpokiyO3iGvOxUAYT4pTE/aGDFuEGGlY0ywGv6Hc0GCYa+gxbDAYtijUTgXFTz1KuVKvzFcu
od3KIO1XzBYqKcJiimA01LoKp0hSFSpgQJ9xyQAA2T74BgDO6So5CeR2dkyQjIv1N7UGXRkwvBJd
mMPvAMx1G5sRCPatMBA0syBwnoNswY0FnpYQkqBSgX0E1nXsA0iId4mfjLbOfwB0/gOg8x87xLsm
rb3AWkCu60Lsx9BENC2Bf4bkjRsQBoFzpqMYYn7SP2LTjtEy4cpjcvm5/5JPn9ek47B1N42+Ib+C
TE0ycug/07IkIgFrZlcySVZSLamuRXSRPN/QZW63tju7kuYnOzyoiNNS2GCMfEhbjNs8Qlq5RwiW
G00pGB/va4oHUl8fKoB3OLIBvDmmED7WyrlG2JRjz6Es/B7NETM5oWZ67Nx0sPMEoB3og4vHj9hX
Z+vUdybMfFSMGETHTQeQGyMr6nV5dIP3C5EBb5ID8UFmOZSVEjlzSc/yjqXn774jPrp4cU/30nYq
f/uu0ZnxkTf+HI9R9bXXqEHOXRF/7ejR+Ks9vStXL19OA8eO0OCqZavXxHpoFp0Cl/61+B/gUlUx
f57Jq5tBlw5UCfxRK5nsnprR7G7OaLUtsPfaDanlxOAwCAaDMaXcJBpVe9AfFJweXUz3owjiAtPh
M83MqWsshv7xmJXxLudCnPp+wlldH7QHo0Eh1eA2crGO4LqOawAJsjJyskJbJysAOlkB0G8H4G0+
Vca1gc+ZmtnTZxlmQV2sIoAborpNlkhW8BwMd1G/5E1INzdMm/Pru06epN+86rHGts5fVlaV7Lj4
qfu33gw3VLIv/+G0OXNisCiKS6of2DNnY7Y/LfaTcKSkL+Fv3ZrA4XltvkozkOOfTKszGuyN7saM
xXShvcO9nq4Ruk295svoZrOTKUIHcRh85QKXqGyttGlIcgtySjk31RiqtaDo9MDzgM32mZbBsGtN
ZwRl5Vxr5Zkoq9URgIzn2OWyNFU0Mm4W3eR/ozT1CIB8AVJLYzDqw50IpIyjtUbHKwiYZTu4pkyB
noSWZGbZ5xGAhL8/7rDdGh+N2+LvnaR37zncOHfJPft7isvDW1rfO3XxtROLw0JrbEg+HSouu/1r
d/++it6rLc/KSI79MlhcuI5J0KvgyQvwFEpA0CSSEHLFY9JuAgDtRsbXKZymkvnay9ceC9vv5g4v
9JWfhNJUt79AzU/J9mdHqtVKx6SkCn9l4Sy1wdGU1OCflVdf2C60pbX524ovSV2Z1utfGe6O7PD2
+/sDA4UDxVe5QkbN5qhS2Qoq0unLlzKUYDCnnIe9ETUI5nt8XAD4mLjBbCG27fSQoI9lnXRxAuBj
DcUlkMEDpfbS/lLB2DdxLLmbyLslkm66c5FczdILnnbnovzVzlX525xb8q92XpV/i/O2fFNnB0+K
c8LHaiwxl82qOVi8Uc/0wgXR0wvMIUFIMpF3QupJFuY2tf7m5rvjo7ttG2j+5Uef71ne/NCyk0/Q
mr/fAdvS1hb/6w3f/Xn3Nu2Ded//Af3hogemaI01Uz67eOW+Tcsv9rl97sLn7n38o5qi9xu7rlzd
2Zduy/cUHWSzhp/0IXjDQDZpqVSqUJC1tRv9xhajSJZQBAeFNuqGjvhUMzEsSUtaEPZjxGxmcyar
zHhgbS6CAXzCRTDfw84HMIo9MBD0yFS482w4Fn6Lsb5uljN6RDrqw9gHJ2MfoCfBc3+Sg0OsZ2OR
TIVcq62cSRvBepJsUBYpVyEFjCJ2KhukRdJVkii5EX1XaT1PkG1CjF0hsrBZRHpAUBvILFYHLEpw
YCcnopMKuRQ5M4p/sxgWK8Q2sVfcgZKrFQYWnWRmJ1Qd6yGLLzPBzyOSbMWMTZgTMDIRi4y9Gf8s
9uZL9EX6Iuy4CJY35Uz0eynqp64DRi3kGe2BmeIqcZsoWqlZkCRBllWLOZmmiilyqppqLhAL1ALz
FKFaLJXK1RpjmWmyuVmol+rV2cY6U7O5jS4GxhfLiwwdxjZTL+0TeqU+uc/Yy2xCaZO607jRtNM8
weLGUw1uRcbcUZHbgUa+RkjUKGCKIHAQPQE2ppBypZnUK9vJZkUhG2HMRW1dtkGbpKyyOj5EzpSN
moVmIZ653GDJDAo9z7KG+Me48W+4Lv71P8Wfjv/ylfiW52g1LYeqolUMB9JL54ug+Qull89nSm+i
V/XwTZhVZSYntFs20+0GwSTJJp/kMRVJIVOVcY5Ua2oXu6R2eZGx1bTIvFpcJ62WVxm7TavMO6RN
pmQzG5vRrRrgxUL6ym4E7WTJQE1mRVBZ7hNhH8Er5AqVwkxBNqqpaoFarTaqsqAaTAhkK4IVZX25
KMabiUJihay0qkYlVSlQqpVGpUtRlJXwGTtL2QL7ErECWDUcBXz4Z1jSlP8DBZ1GgSOBz/6kWFyg
b8f74t2/Ewxx+S16A/02fNsAgii9sduFd4X3YvcKneg7Cumkd4ABlbykLS2iRVK+ocKgUU3SDK2G
1VK/weRVUtU8JV9dqHSovUqfqqpszCjrlmSBWLFQ6HGDKMGMBwWxMaOAoMU0aJLAoHBXxllU90+Z
Pcd5EECcO6oA3uXsCUBX+gD00AoDUBUE/mSMPQj5PabIwa9gVf13Jlzt4CzLSENnCIYWnTDAwAwj
0juxsydjn/yB3kJvh1U9HNskbBU7YisFpLIparqJrAILRtKhVSL4pahJaq5aqc5QF6or1S0Y8yG4
oAaDCPkD+5bF9TC7BoMkGDfxqNgaExe9YEzmECZ8Z17lwLTbmc6UPcfh4bPMtgfmvnp+i3jDyA3S
vJGV4tBRac3QofM3Ejo6En9J9o3OBls4j9BWAX0SIvDMCCvkYpfKPsbC8Ze2wmZYjRqGDGkrSKeM
rtE6LCYplGryhKQwjHOlrYivi/m6wzY3c2nRGlt3xvriHabt7v6MHUUmQc2fWuLUnILTGVBb0ml6
eko0IE2crsKntqPkw5lXwVkUPpke0OUAt+iZovSyhwg+kmFG5BNylnCblekmzckdBV72knDRmHXG
9VSiug2eAu8YTj+reYxGtPXA0A1IQvgrohUilPFn3PrjWhkuodI2QWWPnJCGusRzWhV7Jsx0yHaz
ysS4mQfazFxRmrnNbIb6VNoYjDVUA9a7L8hKsDpGOBAohBr7IV3MvIiwnqjTI3JItLOQHC/JwCFY
KKGKRFQemhCRkOyqr6yBEp38LD0RJ2U8ntqWH9k+75Zfr+tdSTPvKy7M758660iPqeqF3i0PadHa
xxe+Vz93xcDXlt/3NedUV7L/1O2DdxQXB9QMbUFKsiMv5wl7dl5kwo1r4xkQY+6k5J627p45oIFj
oIEDUDVJJEBdWkG5UGGf4ikJ1AsN9maPFljkWuUaVHekW2xGJbnWKVlopqaYzKqbzSnDCwNgA8O0
SWPWxFjU6OOxYB6qFxhyCUqhsEZVC7gUkzV2OYC/awXcZjyAwrtolmBLM3LPGwYkrmA2OOKenH2N
PguLkLF5A/CmHiW08JPR/pBPNoBPNTO70qKwK9H+mPcTwEeIoGHPXhT2jU1YYrbGDXY2pZCNZ7jR
Dr04llJCHY2BxfpY4YxLr4gwOPVqmQMtdTMfWNm1v8EyNNxycP3Jt09cedO8HzS2bmr6zsNC1bWv
z25pKUYSxx17afr8+Avxd079auak2K7sdIS0KFkz+hfx79LXSJA8os22h1pCQphm2Qq92SmTaYVt
srcipYm2mOptLd7pKR20zbaG9tq20022JIfDHbVIwaAvKhrtIR4dCUHIAcVjrs9rY4h+TZvA8Xtd
KJlTdXKakdO7EdwABHMOMHJD0chRBq/mM44p4+6sMUyd4QkmYO3C9BLUBfJLiboH5hmOZ5Z0okWQ
r6pM/PvFP+za9mxjUyst/mf3sTmmhY8uuuvYI/dVb4kUNHpMM4pLZzY2/vEm6qKTKvNO1zX+9oVn
f5+Z4ok4QZtrQZt1CdoUtJwaX0n6pECLrza9MdCurFb6HUYXFZxyynQbCqUza2WT0/1vZA2vlBV8
WRoXLIxGQlzkcNOOOPheTkikkCOR24GgzQ+1YiY4WAH2uIS5UafTNI7oNF7BlpampjBBgpB3XAuz
u6n8biqPNQHPQLTKSywZrLSpmACsd19AiSC8CyJBYRiQTIww15z5OdWIqumCI5QlOJnU4DVtzjLx
wnS9VDc8d2jVqb/Obah/pKd9b/Pw8OytM+8c2ntz632bZ1xEy6lz/2sXzW7NyaNvnRsVvpHl++Oz
T/9qJvPA+0bfkbqlnXgzwE9OaXm5UthaIk2x1mTWSc3W5szF1lZvn7U7eat1e6aN1vj99vSpHlbn
/a5eCWk2G6J2MGmQy/sgJ8RUhmXgXWhL9ZEAxyqT1fWcFq+H9+2H/y36kQnCKawgEtWPaS6ORhdH
m4vTp4ujzcWPu1C1qrS5do8720CS7mwzScyizciAhLlfyEPOwbGSNl4WHwro8QuXh6dBFIPUPfLM
tMry6xdu/MtEU9fJdfH346do+Oyb/3iU3nTzLYcsQtqq2yaWlCwpej6/Es60BzRaG//s74XfvOfg
leBd2H2iS8kEzp7WVvk4Zfm4I626q92bZVRxSFEPMdumqU7ZilczNKNgtBtteNWT0waEFAZu5izH
YCgXyoZn9tmdxKZZHVU2L6NRW4DRk41fYxuXbjbwNPYyCkV1GKrAXUzaoX2e4ZxVhbN72a5OHeNg
JvFKS+FHcwAGBs8ZRVFUwkL0zOCBbkKKiHlkHj1fhvpcKCjG1qLL5F+Ru20dnR8/NDw4ePLxaG+h
fLEx6ZJrc+8cmS4+cWfO07/Bazjg2HiHVAc6CsFDjmjFU5OmFZYWTS6pNzYnzS6sLWouWUI75cXe
PrpW7vPulPsDzizZFfTka5kSwod6TogBWhobFDITmmidMN1jsCtUCWaXciS7mO5gBiEHGBIZoOkU
4iNKCvj2U23uf8Dfvn/l7VJ/abRUCHPSC/NZCael8Cwe4nJxLYfNRQoXlgh84NEpvJqewQqHsd49
cSz8w7QJK9Qb+33RPOD5ujAnV87eOUjLQc/AW2alNyxDAjvhC+wOafpFdo/H42c7fjjPNOHUiu7L
QqHMttu3gvtnTH9sac8VTdBHzd/Qbj945W3zvjcYfyv+aWrycVfFhIK8S+tX1tfBwzMcOD17Zkte
fsnIy0JPVsYLJ4dPREHXeJNQ6oLU9dIyLUn0eD2bPXhbQK1NkmyUWtWvlLAsFMxr8pktwK06ZmPz
KYDlpjl5+F26QMwye47NIeYqnrAQshg+0X4dNgS3EM5p5eyuhBM04fYXuSHFn9KdIji+wEPcRkjw
kA9xKt1GAJB4EcLKmQdt3UYA8Ck3JgHAx2dPg8eEJ6H9N24sMICrQOte9r7K2I8zD8vZjO3Alkke
mAw1XExzIe0cL6qF26CbDF6P1DXsSkm9uHnOD+YMD7cPL3/kZ8LOOXtyCwtmTxn5GYyD55vmvQKL
QCAPwSy4Qv4D7HcDuVdLovUC0FOFyjv41YN4c+hGYGhEK+K46pb4wBDZBeYkLkkYDEfHhzdVQKm0
e5BHMnSEwO9JICSR2xhDCA7oCAHApoUnrEb5TMp71fHhdr7FtVL4rTD3oaO8oolCSogYJ73ipZcs
w8NyypPncvDiKBFGT8Q7BA8fSSo5rYVMcrosjA/HOy1VNpvtmsHakkyTB03URLvZ6xicKADoIRYA
en/YId4fL6viHNHAcKCVblNyP16k4vG1VG7HQOm8ptuFqTx4izZK7hmhpSKXgPVY/RKAD1iuXmlL
NTCMpcpMXqbemDY+WkxrZ2np2JDxTgOcZlYFeoaPG4X0GPe/5jSRfRU8wMSpH+RfUWRK7ypubPd6
re/R7zHEmJ465bAcMqfn5+dvmCteeSezAH8ObnsI3GYm57T6fOF39A9G0UjxigbNEPzWYhqxliAf
ssC8RthO2etL1Mezl4d59pKlLmW8KYb4B+K6/azkHug5q+VzDiLWAPKXCCFwKhE5lYhwpRGX5VQi
+jDf46SRsKfHSWOcInSmwak8Lw/a4LdA+5+cVxjAeUW+ejw1DdwleIUlBVGoradVUEHLYk08vfkP
Pb0Z47lNpFjGU5vSQ5/Fte3Dw4L/TOy/6XsD8WsU94hPiMRGgK3HgbKvAVsi+bqWh1Sxyuj7RsIH
mLB8OSczGJLDN4iSv7ExAkiQP+Xno60PDUDCZ2CBBpghzADmwF5pnBw6+YA44TNHmlH814aH0RVd
YhqSofXC9JTWLGaLBUnZSQX1gfrcRwsNR3Jojj8jXU2uzc+SMmTqSFe1YuovLinWiluL+4vlf9/5
YqYIkxkfF3OLiqYwgUq5zYo+vs99LQBnNSePj5QwCqe8oA97P+FiFMDHWpgPhhtPtMeRY05PvEnI
KcLOEQYDFwiz+xwcF+w5Dv4ctH+tu+OOXHZ7B7eMsfczfnsOsLsDOM/JAcCoFmTM5PD7OJ7xChEu
9PHHMFhp8/nSxyYFQKLcI52fjLZOeAD02WGALqTTTexJaLNQgA5oeI0EUI/foTl2OURHZDyeM0aB
Diaf+Y9PIIIr49qZ7eXim8XDaiDB8bYMnHX+5hImeFyM44UkSPLxJpPqHt3+5muPIXnY6kleOLfl
zhZR0sE5tzMx/9Dyjd/N2zh8ydGHhJ2NV+WHi1qmJk/NjFUIO2ftzg+HmeiXOnc2zetu6277E5Jg
Cd0LSvLSgi/rXtQN/q90b/IFupfLRDKmaONjybHX2Qx/SdFihrV8hu8LVC6bx4Ti5bP5lSqXUyUU
6Oe6VvfHx5Xw/6vK/b9pXH1u/meNy9EOhcs8nzekDcC4Gdnbc5pviq3cUe6e4m221Tvq3c1e1R41
Sp6oaEL8QM93A9Dz3QA+0Y1tS1oqZ1CmmsbiHhA13PzRNRVyipxfcAarpwFWdTrnwDltKqNiciDV
nupPjaauT5VcPOKJCBcwztbwetIUL5vJxMtg3EBVuG+EksG4lsomUIHfhDXP9LJjgHenjEswTusX
GC8Mjzz1y4x/ekF+d/yND5ih0ob4u389E3+PJp/5K0058cAtt/3wgVtv/pEwIf5R/ClaQ534mxp/
Mv7R71988fe//v1vgdFj8RXSAWCURZSytJxSodpTGqgTmjy1gYWIJl2m7kw3jUWTZESTjGZLIpqE
IBLQwvV8IprEaffo6AtjpuTHunJPVA+MKfOvxOqn/2NYCcLmPwsr6dIIc63TMwN0afQfxZcSavDz
jDCXOP8mwjQuWL4cYbpoZu2hFYuuaxoebn6879k3Tuy7fu59zQgw3TEk1Ox946JZc3Pz40Xyf2+O
tsV/Ff/g2VMzqmN7sn0vcX9sBffH2FwYtPAUcSpiKJMDs8Vm34z0WQEWQZEFp5SiIYJiQQTF6HTr
MZL/2Mr/TyMp57QOTvf/90gKL2xWYc0iZgKvlsVSGBsk4ic2dhcVu/59FGUspJ9QAV8Oo9AQIqv/
s181vOjHK54+M7++9uDyxdc0wpG6aOuMex+4+qZ59+H9eF9zE5IBtgOvNje15ueVjDwhbA2lv3ri
qRcRSeESXNwIA9iFqmc3sTpgg8H+ssPKqzPZkbwBrY7X1H+s5fPYKHFr7n63YDFwFQh7FMM1cHOe
wfCDfYjJ6YYagIQRk4iSjulLHPhQc7LbGVE+DNrmypLF8riyBPDfelBvb9LnEuFCK43FPaEAWQUp
i3vqoQBulSWUnbjRVNhSuegehJX6f9QxsahIPGAyzpk68hep83uLm2W8D4GCudG3xd8io1BG52uL
FMGY5hFS03KNhdmlxprsWuPs7IvlTu/84MLIgtL18lpvd2BFpLfUvV0edA4EtuUPhPfRvdbdvj35
36TfTjMTW0qBlCnuyoIYYYIgKyt3mh4n0Lixj/DANNEYRNDjHDNxhLYCjjm2VtoK0iq4TE7hsaQU
RAfgrHOtBUf+00fYKSm4UhflDGBaChGTNBJMMWCS3h8T5ONuLM5gDyFudqsLRY6OYez5TMvjgnx/
IgfRVTFYIRu42DbwVIIB9g+m86pynjT4PHXArBKIYz19zOTyeCgL5jN/U49NyZdqKCvK81hBL0+o
jwUE9Rc8kVNP9ibj+w/4E38b+8POX80wdbyyYue1ublr879RcdOO6smTfnLJiufrTY2/XL5qf7jw
4vJvhK+YOZPW3vbUlNCLdS2tC2uzslKMKba8Wy5t2F4SqZoYeqaiqeWihlDIa0kxZTbNwlxPG31f
iMl34jsiB7Vai+zDaymi2WGYZjWb5LS0ZASmWzIGMwQbuTZDtTo4aSNYCtw5uJXNYBiJPpNqYMEu
RH1e05xMZRp4wCvBC2PkjcMJ8jakM/I28Htg7990FWxIZpg37E2/gLjB+LoXEsFL/AnvrQwBL0S7
EoHrClTL4gUpXh/+eaxQiFV8feKPDg4ODtMr4zvVFO+clgkrvHib2XX0OWHenXR6/Ik742L78nB+
TpqRUf3DsCEWgefxUofmNiuphu0GUZA9RtmJmDRVvzoo/Sm3A5ic/Txk8qGWwY0GlUdKLrTWYLxz
omNGv249aGVcnurBEW4Gkxv14Mi/j40gPq0nUAEkzO5EoHoMzzigv+AD4G+6olM517A2lyQA/sEl
ibrnwtgIt8ousC4SooWrOtRM88gIj2flVgDh48h2lkmLhrseXDP01LDDl7ZwXtNPmod3Nrf+9gXU
L1/Zti1clD97ilgLHKOAURoEjhVynTY9VypQKqVqZYbUpCgFcrWsyXPlbrzP5cOZkg+lEvkkT5xE
qsRZZKaInLygJsolkDNHpQR4FW9ZGB1VFnwDpw9fyZLIdaxcQhSTUCexWZTEdCZwxMtRLcEK5c52
6pWBF1ZLoGqAJcwTJQPSYLzmp/Hof9HFFJRw/l6pc2SPuA29wfcB0dVOVEp8coyoQC6TISjj++Tz
SjWYaUz+7GH8YBUzxCJaKBSIOVKunK2GzeV0ioyyD3kRbZc65EXmdcIyaYXaZ1xhusS8jX5d2CgN
qDuMm0zbzZkWNnyDD0USxOhAtHmsTMIEp32sQgIIYJ8YiOCjSD5uXcGtA0HttzlQJ7HeJhL05XUu
+wDoVUIAUFPLqC3x3YErEiWsMGuQ/EwUEwDgvr1eTgHTMkzDYyUVSVCYSUpl/NUH42/E//yT+CtP
PU+Tb6eZJxiqxM4Rhq7vij1sYfxUg7neBZyZ8QWRa43mNOoW3YY0Y56YZ6ghU2i5WC6VK+WGKcap
ptmkmdaLKCdR6g3Nxjkm1KmKi+U2w2Jjm3k97RbXyN2G9caV5pBdIGpUKFFbBE39utAPovaZzCaO
LB5SEX2SLKH4wozaCWm7tBnFJDLqShF3FqwUSDNLEoutHNfwokeVgk5ex17HYZ/S0KxdVtRgoBCD
S3rpchYG6SzlRTZhVnGhf46BvQ7IEfTFgguEPljdCa86kXadgSn98z/g/dnWM3QKrfljvIn+JD5f
KBZK4ovp92OvMOxMhX3NOMGAmqHpEq/WbVW6lX5FMYoGOVVMlmfQJrGdLKLb8PUaA6MJ2YcKoiYy
Q8JXo/Cyr2ARViMwgjdzWPGkPiTGCbM4L8jkOuQpUE6UJDVIvdJm4OVyFbURYG6wARsOmFqP5LCq
IfyjYoiXz7BppnjJZuCZF+J1/0UX0cVS5zkD/bWUN/KUiEnF+3TImfwZfTfiO0iTPepkvCUyS2xQ
l4oL1G51UOxXTaiPmKbIqI+YdkF9xH6z3xw1d5nXmwfNsnAlr5N466vqJNgHWRIFEuKfR7YL18Su
EFfFNgrfvUasuOOqEfjYqIhgP1RIbNWhL61hxCG6pH9Fi31By/kVX83yI18bQrEV+1bW51/KKvrS
97H0r2N9/m2sWmSM2BexZpJG0gRcN5PZ+PbIRXgbuZXMJfPIfHzLrw3yYhFpJx0o41qCKivgC1hz
YWE/BW/TkZnzFy5onR9e2LtxRc+lPf8HDWgfwwplbmRzdHJlYW0KZW5kb2JqCjE3IDAgb2JqCjE0
OTE5CmVuZG9iagoxOCAwIG9iago8PCAvVHlwZSAvRm9udERlc2NyaXB0b3IgL0FzY2VudCAxMDA1
IC9DYXBIZWlnaHQgNzI3IC9EZXNjZW50IC0yMTAgL0ZsYWdzCjMyIC9Gb250QkJveCBbLTQ5NSAt
MzAzIDE0NDYgMTAwMV0gL0ZvbnROYW1lIC9HU1ZUUFMrVmVyZGFuYSAvSXRhbGljQW5nbGUKMCAv
U3RlbVYgMCAvQXZnV2lkdGggNTA4IC9NYXhXaWR0aCAxNTIxIC9YSGVpZ2h0IDU0NSAvRm9udEZp
bGUyIDE2IDAgUiA+PgplbmRvYmoKMTkgMCBvYmoKWyAzNTIgMCAwIDAgMCAwIDAgMjY5IDQ1NCA0
NTQgMCAwIDM2NCA0NTQgMzY0IDAgMCA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNgo2MzYgNjM2IDYz
NiAwIDAgODE4IDAgODE4IDAgMTAwMCA2ODQgNjg2IDY5OCA3NzEgNjMyIDU3NSA3NzUgNzUxIDQy
MSA0NTUgNjkzCjU1NyA4NDMgNzQ4IDc4NyA2MDMgMCA2OTUgNjg0IDYxNiAwIDY4NCA5ODkgNjg1
IDYxNSA2ODUgMCAwIDAgMCA2MzYgMCA2MDEKNjIzIDUyMSA2MjMgNTk2IDM1MiA2MjMgNjMzIDI3
NCAzNDQgNTkyIDI3NCA5NzMgNjMzIDYwNyA2MjMgNjIzIDQyNyA1MjEgMzk0CjYzMyA1OTIgODE4
IDU5MiA1OTIgNTI1IF0KZW5kb2JqCjggMCBvYmoKPDwgL1R5cGUgL0ZvbnQgL1N1YnR5cGUgL1Ry
dWVUeXBlIC9CYXNlRm9udCAvR1NWVFBTK1ZlcmRhbmEgL0ZvbnREZXNjcmlwdG9yCjE4IDAgUiAv
V2lkdGhzIDE5IDAgUiAvRmlyc3RDaGFyIDMyIC9MYXN0Q2hhciAxMjIgL0VuY29kaW5nIC9NYWNS
b21hbkVuY29kaW5nCj4+CmVuZG9iagoyMiAwIG9iago8PCAvTGVuZ3RoIDIzIDAgUiAvU3VidHlw
ZSAvQ0lERm9udFR5cGUwQyAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeAGUvAd4FEe2
BjpK3QMNIgwDUjdMiwwCASJnEAIJCUkooIQQyjnnnMPMdBiNcs45IBQQOSeDSTa2sdcJ2zitc9ot
cVvc+04L7LX33v3ee8zH9Jyq6qruqlMn/OeUdCT6uhIdHZ25B/cfdLWyXnUwLN7vkF9EkktioJn7
VrHGY5wan6+ZphhfoDOu0GWn6bHT9IUzE2fnG1TM1y83WGAqkTxYYPrarMzJL8OyafQCSX7Xp9MW
zyfOT1u0QCKhmqctgcvcN2aJ369PWypers2QYDo62DS2dvCGRWCMf5CtX6xftLllTGxafFhIaKLJ
ioCVJubbtm0wW79u3XqT/X5h0WGxsTHRJi4B8UFB0Sb2wSFrTCxj1qw2sUsMXGNiERlp4izel2Di
HJQQFJ8cFLhGfJWQsOgYk7AEEz+TxHi/wKAov/gIk5jg/9jdX9/eQvx34N+mZLLQ7t8KJWdgEiVz
JMslByUekkhJhqRSclZyX/KTzgIdcx0PnVCdDJ0anUc6L3Tn6i7R3aJrq+utm6Z7S/dvuj/qSfWs
9QL0CvVq9Xr1rujd13tX7wv9KfqR+iP6V/V/M5Aa0AbbDfwMUg2KDYYMHhp8aPAjthdzxHyxWCwL
Y7BT2HXsQ+yfuD6+FN+D2+NRuBqvxQfws/ht/G38S3xcOlO6XnpEmix9LP1S+nyK6ZRNU/ZPcZni
M6ViyjtTvpyqP3X+1C1T908NmNoxdYyYT0QSlcQ54jbxDvEl8ds0Ytr8aRumeU1rmXZt2hvTPp1u
MH3u9MPTy6ffm/69oa7hHENLw2rDJ4bfzsBmKGasnGExI3hGwoziGQ0zemdcn/HWzCkzjWfazfSa
GTozY2btzJGZj2bpzZo5a/6sdbPsZh2flTlLNatz1sisW7Mez94y22G2z+yI2Rmz2dmjs2/Ofmf2
r7P/RzZNRsqWySxlTjI/WYysTNYlG5Hdkj2WfSL7x5wpx/SFacI0NA3z/v2H4cdzAz4exZ/+6RuK
hGkfo2lQ+Nfr7+WGyAqfiBGiJ2JRtIFghRuiI7iQK2w2mEjAhFy02UA4ghuKvT7Pn+z2Rd5cQvb1
iezmCH/S4WjgLq+Auu5YRUZDUW3RgNRQ9otHQNwRK9Ku5chooMLxtV/CkYxCs37uu3WHPj3advN1
krDUb21MS0pKE/83trY2NrbShsH6SIWOCkcFlaCG71dXdBSpxXLxShuG6n//Y5DNY/rZjnNLhCmk
EC/YwCdeSBBskA1K+HHom4fvKD57/9fzaApl+BBjyxrZEio/v6hQWSxsFpyNiG8xdRtT1Uve7ser
y1VF5XRVgaaoNqg6UpvOZ0nfxcIDj6a7U8S+PfK4aJYroPkMLiWOJI7F4kHempIYWtiJa0/08ecp
Qy5ejgy/XykYCjNWvvz+Hs2AEvimDdFx/M3TlwZvUL2t+am1dHVGSVoAKfjhDmq/Ii8FCsSKx1SX
bpGoBH8//WLAqGIo+EiTFUUgP/wOP1Z6QSEEYlo/jacTKZTgOxs8RwMUwcPXUh9Sn9xsv/uAZgrZ
zGKSQP74o7aR4bNUW1N2UhldksWnR5JCBE5A/0UX1QNvkOhW3yT1b6MRKAgruqoefo0c78If8adK
xxSGUFR8WnXtGonu4G/yo6WnoCha7GYMWrXgD/gx7VWFMiCb3U8JMoxAMZi2Q1vfcBrtHLcwqhko
b2o+KxXiMXVSUWxu8BrhdSNGxSjVxUwxXNWMkilW5Rc3sJyqUq1lShh+DbpnlHOquEXdLjVEGVgH
8jiJpn2AzC5+0z362gMpgbKxuv6G/vr2138wuruqMaYpvCbK2PD5Yf3vrpgLewQrT/OVKzy+QtZo
y/Vv/k4bPvee+xnaMoQWo/0kmp+IpnkgQ0XmycKGmCsBGUadxyvT6kKkFwSsXZguGJGCUYJg6CUY
KDKCc5MT/aWGgg6Wl80yhXR5uaakvALtQK5GjS09vcOjUGetH5Udm5iiSIw9nulPuUe1nOxp7ThZ
TZ+qOa39nIcGxzG0DFl1DdQ1N3cbK33VLsdJQvDFbQNdAo9Q0UmVjVk0IfhhcYGp8V4uQpRQZuRx
mGWPe0oJIQRzCOC1nrRgiRNCFFYTX5J/ejl0GYUVheZHx1itF1KMCCERy4vLiksKP7rXyPNJcl9B
XcaIsZCMjXq0RWkL0WLhGLS5gnnlsbwfzQQxCerkWOG+kSpGlaCMQg5pRhdbT3W8Tj295r5jh4vb
Fku/vgfptOAdWedeHdMWYbzj1o++aBaFQtFshKEjT+Pve16mr7nt615P2bpnhh+n+QauUksSLwzn
JpzKbckcknoh/Xg0E80j0dx2NPMiMlDUnaxo7DwbUG8U616Qnhki9RSmwxQLFClQMOWXBENFfVBl
Sru31NAbueHFZ9Un75CoDSc44QuD0NSU8PDE5jOpCsHmQOfRhuTeJOOws3fTnlAfPmw5dZY+OdBw
5h6JvsLf4s+UDgJbnjyKHVX5FYcpkD9WNKzuuQFbAUeh8m2ul9979+aVjz++5b53t6P7JpoYX/R8
SC6YIh9kigv+Qry89Djvb0sKd3BiXPX8R7nQKEQI4fDdiOCKGuETDh+4AgXlsG5SIUL+6ZVju/c4
um3d4Xj1nfduXPyMJrbX7O7cr6gL6AgaCJcSNreehf5KIU80H01HFmivgCGJYOPuW5AWRMfnZuUl
JJS+VtZbf0JKON98L+Z76ocPem7cpLu769rargj7LhoVxBXmZmZJCZfrH0V9SyHpl6feuUQTR5q8
ToYrgkdvpr1LfXSj4fQNmnC//V7cN9TZi+U1o/Tjtns3B9+SDr3b/P0/SKSb9I3/fcW3h5ddEWQU
yM75wOgHhJ1oiqCDDj672XL5Gl1knR+SGiE9Hns83oUKiKhpT6KJolH1GIiJO/hdEDvnFYalx/hw
G3KiDt+gDij0VRClAbyflThf9iC4jikMWwItay0oU4vUIFeaGOhvvHCVfCvyom2/gjjReqL+AnXv
dISXd3LcJid6qVVQdFDoAVuvPXFbcwKLo5RHpZqhNu4LinjNYfOAKSXMXuyy1IEm/iPDEP+R2Qhl
YnFufrY0JiUkI5Ai/l93939sQAy/3oF0fyM/yn4v7rGCWL3RU9AXZpNCMfIUPFExfDzgo0Q6Ab/u
+0BBeAqGr5iabBcMRaauC65M7vAebTQikn0zo2LSg1jjuFRtQzhN/G9pRISfSu5K6RY2omQjwlId
xQQqzQsOJnuESIn8lvy23K6kK0XlcWVS4gTyF3aBustBXsIeFCD7jhjsjgkNjRH/dw8OdncP0oZv
nQ1zdfEJs9znPXj92rkTb9GyXzjhWznxB0M3AENH/J8MLbJ8hNAwyfL/atH0L5aXmSAnXLa32CNA
7UQJCNvKeXIZnFTYg5+IYdl3LUHBXx3fN07LBQthp70DywZHGgv7cP7SIHePGl+APRxRF5+mkTNe
fMxf7UAJuzHmSAyrprQlDfU93cgVBRgJVsgqsza1MbXRmK3VsBqKaWpgKym0BSOW6F8YCfTyChT/
j1y4MDJyAbRpLFY0ph56gxwfwN/mT0zqphrQP5UnGi6i0PEfjDreqGir7ZUKOZgqrDAm21tYOpFk
VJyqLsrxyR1j2YIuUDKNWMe5lsGuMZCu2VhDYGV8sxMKm/gBRGc1NuTTG9h3HB0W6o0Ilmd5jtTA
RUHIZqLDuKyl+FiY2poS3sJ2xXC8P116lA+1JydO41bq4KIABX+6m3uXQo+wT0+KL24oM3EEme5K
CzCLV7XDF/kHFDqIVbWwXCOdnKBSFhYIZ4VfjPzc4+MSEqSCBc6fG+IeUOPzsLdH1cVj0EEoCkMr
8bdGQqys/AMPQMF727JYjQdd6sX7upETZ/AD6rAiXxi4BwYmZO8dCuH4YzTs3eOu5MQw7qg+XuSj
INBs/Kef0FqB6/TrcauNMEY1WNhYdqPjM5iMHbjaJoZxoYSvQJUfwF/nB0qvKIRQDEYIdCAnBvDD
8GoeChQK1fZ40eEExoYSItFy6OMhvOU5Gubl/+Os/K8OCOSJgdwZeZccb8QfgvUB5k4jVuRXEJF4
UFg5YWkkmI5bJj4o6C88bSw0YFpfjdthcqIRtwYRdFwh3jymPgNCKwq/qvyo6HWFNotlq0I0Bb1r
uOKvhCajrj1VwdpQeMkwrOj0ZMtE/KL6evFZRbdbQ1RXALIX+o1OuLcFNwZJwYwB2+jcHXK8G78P
IvCsQgjDSr35ABtSSMVdeO8Sb0XCmdT+6NPQXwRWdFY9fJ9EbugLwW1y9f0UYulF9Yn7Im++yQ9N
8mY2ph0q66sZ7frECA1jXZsrw0rjyr2Ml6MXcsER+xpdNTAUIrBSXz7ESpxrQqjFiqPzo9KOCpEi
Q2qQQv6g4XT/Y+qN02GHamnDYl8f9R5KqEI7USRGMJVsaQmpuqkcK+iXopWhgs4nwjqKGLvSf7K9
XfrH0hIyEyt3Xuv/ig9PD/NvUvA0hGyLcHNiujwopfvqufrb7Wfp8x2DHcPUYFuMv3eGQ8Ix2iMu
NDqYMpTtrTyWwkdQ6w7HHjpKy1reF7xweOTQfaI6sFWHFHkqUABWdEE9DMp5HV5ypqylqV0q23uy
Z6TxMtXXUZx1DjjX7uCkxSsbGC9A9+ULnfZv3GTz5Ifv7r/55bOr9ksVhjKPrfEcH0CXefPh1iQB
rG+FzPG7/dEuziEhjtBBpGA+flj+5b0DprRsYIWtxfrgkNr2BEVUT8a5m+Rv1+9/Bn1wwnGkKxgi
26eXG689hG170fX+xfSPqWePOq9coW/d6fv0nyQ6uhBJBduthxLdHRVcLVtRRz4FYXJSC9znI/J9
yD4Y/hFavvNTYf4h15TgIEVAQLLDXnJ3i/1IkCJo5FrK6xSHJuSXOv22b4z3dfeIO/Pph11jl+mJ
74Wn8isBz02ef+Y2agCb1TqY433EzRoMUmIQt1BHFXlP8sk59dAjEm39Ge/vKEhtoTuSqhP7N/Zt
rfYvjZASK3D3eL4kjBbm4iUjrdwlSn1MZZsVEhJqnJ6eWxiVJzXcimex+Vym4qqNU8t2atuBWA8P
mtiBHUjgtMG0cAAv6WvgLlOGftixaL40ghYovHS0mbtAIQw74WKQIpjixyL50iixQtvfy5+jkKkv
RoRgoTBuNC3swEt6Ovl+ali4jVnXhg1HKxL6z+c+oq6NlNaepTPR6vjs1OI41jg1o6Qsl86uKK7t
JlHFGH6yU63spdF2XBmZyPhQIS0YMYo/eetdYWaPV6tPnZ8xKsdiTihLHZ4JO5DKCC1FJteuVleN
nDAWmrDU6KQ4b29BV9AxQidAGpwWN9A17OpZtfIsjebhyoBgtQ3l+gNG/AN/zJ8sPaMQArDSQN4X
dk7X71ZJdAd2pan54nBvQXqfog8tP1LnX+rCGhOF1nLBCXmDOjMcvGGADLCzPeriARpReFFwovoY
5Q/SbTdW0l5zsfbMHXTMSFvL8mVtfH5Jtjb9DljVjU7lUdpo2Pkg7IKPiSrgDtzhiL0LcvoMDYKx
2CdIvQvcNpBJF9WnHpLjdfAKF8CdAnOfQEexoa7J8ZzBCfPBH4xeG75Jdbfmp4FDmP7KISRQJnbv
nLr4Ki0zQpa47GsCfYe/A/7XkEKIxEqDeB8wv77BBf3ejbfsFXav/RSKZlKoCC1AcpRKQ9u3YFJO
KwwFDI8/6pvrRtkdbRqMALt1GaZMZtlYMxfB1kidz7I5gUy5sr643RdZG+WeLzqpHgTLFmvyqPOp
CkX6wv+AHlyORQjeBoIu1o+8DQhhNa7pb2M/oJAGIwR73EXlXQxsDD7hFfWZK6QowdQRhX457n7C
AaOiWJZVZSqri1rVZU7ICTqLFveU7wFyogM/qPYpClXww6e4NyhDoRQrjM6KjPcW9ITjRoIe8ok/
k9uZO2gMFcTERtwqMvzgoajuvyUrBPeDbYdr43sijO1vfBSJdKjvP+y5cpk+f7719Y/I8U34Z1xv
2RmFIbCBjTpA1AzAzJMvYCxQSA8jTqNl+JkOdfHw5IKHxaiPU8SjCX15qHtQti9l6zN0o7uqo7WZ
rq2pqTjRLDX8+OBWfF2Oa6KHp9TZJXzDSpIYTwYOfOtCRuggPRLYZL+Z3Jp00N9PERLinW5LGY4X
vFDLX7/ePDQE0106PiJPaU5qy+5i24xP9XdevtifEBSRmOAUrSD2vJIgV1PuUYR9u8NIgMLrwgex
31J3blU2XaWJo8cbByLoY+cTnnxLEqFXv0j+gnryYV37FXqoYbC7tVfaOVh7/jJ5K+GK37AitDe8
1qFCKvsJ4V03H/yDvJl6IXhIcd3ZvFfQoQTDtWE2DnREaEZUqj2KjDS6V3ey+zH1YDjS9dCxiNWB
4ZPyZS9OxKSEguW8w+NNJEG6F+59eLI7J7GZJjJSEvJSqfT8ivoKrlJTBe7BsTBmMyUUjxdiRNE5
pv8xifRBTw5ohxVEsbufaN2lYvHpLJNPE+p6pqqaPFfXOdal+OxaZVl7q3RspPk8+HkKAf9MsBPw
9WE2TgqitqA8O5OMDEs/fDCs7UqewrAzpz42knTxDtvt4dfQm6Ag+poG6s9Srw3GHAvPSD4aTe/3
LShKiJFGBOYEWJHEwC9o3y96KAGdkhNDwa7Nhyg7UVrTxMhQ8w14wCVmT4XNCuJi3YmWe9SDniQv
mnjrraqWR/T52vbqa/0Oz4y8ouNzAnOkWd5HCgDqsXz8QyjgU++8Xd16nyY0aVyxtkAqe5RQ1585
QhH3PLYNCHrUku1h9sdoAs29c+3BdySavuhvACftXLRTmO3YEHTJSdEW5xAZlxoVYUz4jvqMJXTs
fc/o+q2Bi5d+FVZ/YUSUnijtaLyF9BHssp+2v+bwZBWSIbnR1m9s76x6BPbojmX7hGlhFWH1scBJ
nsJMU2HN9hMHX/eBAWlhxjPBTTBcEWzrpogNyzruThLakJK00tRGr56gU7GXwu6mfV4kJST6iQYj
2E3inEQEq3UkBiISa68TordW7yP9a8uvPP/uis6PV57bXdH78flP8v8+8vzUfx3BN7vLhYcT84RH
4/MMnpu+uAnU+DwE/w1erHxxXQ6/5qIHE3MNXpiOyJ834ujRhLGBoTUy+QzEzxw0MxMtykfzZ49X
oiVoClo2J1r25PnbaIO82L0oM7NQWpDpUGxNpTlzXBDtyRVxOSXSXI2yvI4sG/fkH7Cs5rqxMj6C
caAOYByawtZq+6WCOaaJ5oM9yWCcVbOqYjIKC8nkNHE06C5NWys3QHVgxcLiHLOEDcm7Uv39nc6G
DgWe9Lj7+eA3Hcik7mn551o0VzqK5U/EGKhYJatYMpwBXtz8Ma5kWHGvtLeipXmwv+Z6GTKK1ebz
ceVSppWt6yfHl+PvcB2aXoUsWnwi8HocMZcoriSO1kRzEe7khCt+HFCVFAWSaAyQDyZsHS+Rg2s9
VZiCCTP14dcc+GW4CU1j0covkbT768GvO9EMcL0XzX4frahDK4rQ4lFk9h1aN2eXr+xSLsp+3iM3
xTzTuBJQx0a4ZqCZO0uhn7GeWkbZQCMjXJWWzURTK2uwZq62pFEhbMVK0risVBLlT5jJVem5bAAl
VGL+kZwmnhZm4pq+Fujga4xt5NirVHpORl5WoTQW47NZJZclVW7xAeGXXchyaXSNQDeXlHPdvHEn
38SNldRyTVwbLw3H0wuVkXGZXKmjosmrLEZzVMoMdTAd1JPh0ppWur2+qvM0+RH+QRnLvlEmRdPx
w8X5jEPxqY4zfW3NdTUsW1EqnfBHO+TRTKoqSQFOGvsahSqx2xWsuoeWXYpV17PsFRIw6EZWy9cp
1FnhzDEKXC+vBI6PpjUpXFoUiVbEyJF5KHCBzDL2uSV6JFcmZDB+lGCPhSdxGmiWxCVGki+ccdVh
g0g8l8lRxSnQdkzZr758m3w+9WM8WtDNjfDfJH19U//eQ2RoXoFvbh7HpykavUvD+P1SpqaSLaMu
jmorO+grTZ3nH5NfRL9n8ZNi6IOet3uelrVqyrRVUsNjYKLko4XIHi3RGV+H1uvN+WR83XNTuWAv
LMBtAieFqBmuHRvk7lJogZAvxGK2vnwJWNrrce3V0/x16rkEuzamLgYLZj1eHBjD2FGuGJI+8b3t
f9HtuvE5pp0ZU0qRGa70DlcfpYStMJQX9uAkIzpW63BlkK/akprowvZBr6G0AMb16Bn+IYU+F96X
o82YYGBz064n+JylsWCDGW5Hi3XGm9BqPfTG+Hm5YICv8N0cvyJzUNhi9FXf5ydvdGfuj/J09fcI
9EvZHDvmXZ06suuH0fujV6999fUQMuRQGI8GjLibXLumUypswUri+YQAEvZOsZJUC9bHP/C75fDY
WJi58a7FkMulFcbjSYJE/oHvTZ939n2JjI22vO4ytOXcruVGAOv9uune62dvPPoR9O/TcFt/V0sL
Y7QaP8+1lIwohM1YSTCX6ky+sMVXn0xEq9Hss2Xt/YqRnrpLZWiOQ2VU1ZEeKZvD5heTLkWBeXmK
vIJkZSh1OEvbTBtKdHV0jM3H6WnL2Ol/CrqdEcNq78weN/k/ymlmtqBh5xtYsZhiql5t+YqYaVOu
TL1CXJk27cq06dy0WYqpBrXVrs3TZv9X5pwzMsm6l6ErE8kWyT7JYYm3JEKSLlFJqiVdktOSO5L3
JV9L/qlD6JjorNXZp+OiE6yToFOoU63ToTOic1Xnvs4znZ91dXRn6prortO10PXSjdDN1FXrdupe
031H9++6z/Wm6Mn1lv7vYA0h8x0dtxudsBtPxA1lvgETdgHjdhOAXwPqMG43bgc/CFnLKC7TQjlm
iCZGdcYTR/XY8SVy4ndiPPG/9F9R0P5PdSI1bjdZ+Taa9upGO2j6O/Xqzkny5a1/1P713h2js2FY
me+/bnhV0vLXErhL5jvZv1gvUi2EeNkxCq8jPuYEvJwA7/Sq8Ork40GhnVjorR8gTB/Fjr+8GJ6H
YBWCD37u9x+GH+ujaR8L07Cnr67/q+CpPvFUf/TjAOzjyW/D5/mT1xcvL1D74tjL3p93/z4YEYCm
j+Jn505eiJCPP/74KXzEbx2CffVLpPXEtULOo4Lzq7USnAOQ8x9rhZzFtULO4ksmQhUs4J8oQMPE
qVCJa4bSXy3gv0iULq7SKxo6ermIf6In+33ZYLY4kjihkzfBWOkBMJ+qyUV92TUs4p/oP/oWl1m8
V+9V7b91ZCekT67MZE+/1119+UJ2gvO/6mBpYRyZ7587flX28ple9S+WwXDAEC+f5ne6hQB+B65H
6RC2hLq/Tvn/n01rsEzctEaSNZK9EhdJsCRekifRSFolZyQPJd9JnusY6azS2a1jp+OpE6aTpqOG
7dqpM6ZzHSLQn+n8qDOhO0N3vu5K3U2wYe11j+qGwpZV6dbo9uie1b2j+57u17rjEJU21lupt03P
Rs8NotOxerl6JXrNeoN6lyBC/ZHed3oTEKOeq79Yf53+Ln17/WMQr07TL9av0u/SP61/W/+J/jP9
H/VBJBvIDEwM8g00Bv0Grxs8MZjApmIKiF/bY/5YClaAabE2bAj7APsa+w1fgpvju/FIPBMi2A14
H0Sw38A/wb/HBeksKS1dLd0jtZN6SoOlCVK1tEbaJR2R/m2KZMr0KSTEtLdATNtpiv+UmCmZU9RT
qqb0Tjk35faUJ1OeTfl1qt7U2VMXTjWfajHVaarf1NipWVP5qY1TL099OPXp1F+IOdMLp49Mv2Mo
M1xqaDvjwow3Znw44+eZB2a6yAxkrrJEWR5EnN+UfTJn1xynOffmvCsvlA/Kr8rvy/8m/0KO5lJz
V8+1mOsyN25u7dxrc5/M/WLuL/P058nmmc6znec9r3pe+7w78z6Z972RntFMo+VGbkZhRplGJUaN
Rn1Gp41uGD0y+sDoB+PpxrTxWuOdxoeNjxmHGacZFxtXGbcbnza+a/w34y+NfzH+H3IaaUyuI3eR
tqQb6U9Gk+lkDdlJniNvk++RP1N61DxqHWVJOVK+VBSVRWmpJuoqdZ/6G/UF9fN83fmG8xfMr5rf
OX9k/vX5787/aj5agC+Yt2D5gl0L7Bd4L4hakLWAWVC9oHvBmQUPFry/4KsFvyn0FTMVJgozxXaF
tcJdEaJIURQqKhSdijHFLcXbis8VP9M69Cx6IW1OW9BOtB8dS+fQGrqJHqRv0m/Tn9O/meibzDFZ
arLJ5ICJq0mQSZJJkUmlSafJmMktk3dMvjD550KDhfKFSxduWnhgoetC/4XxC3MWahY2LxxYeHHh
3YVPFj5b+NPCF4umLyIXLV+0ZdH+RUcWBS5KXFSwqHxR+6LRRTcWvbHok0U/LBIWE4vnLV66eOPi
1MXqxV8voZd4L6ld0rnkwpK7S95f8sOS/1qKLV24dN3SXUsPLz2+NGpp5lLV0uqlXUuvLL239IOl
Py/TWTZn2cJlG5ftX+a8zGdZzDJ2Wf2y/mWXlz1c9vGyH5YJy6XL5yxfuHzj8r3L7Zd7LQ9ZnrQ8
bzm/vG559/JTy68tf7j8/eVfLf9thc4KwxWKFWYrtq+wXuG+ImRF4orCFaUrWlcMr7i84t6K91Z8
seKXlborZ6xcunL9yn0rD688tjJsZfLKgpXlK9tXjq68sfKtlc9W/mqqZzrT1MR0nekeU0dTP9NY
0zxTrWmz6aDpZdOHph+b/mD636umryJXrVy1dZX1KrdVwauSVxWsKlvVumpo1aVVD1a9v+qrVf9c
rbd65mp69ZrVu1cfWu2xOmh1/Oq81drVjav7Vp9ZfXP1m6s/Xv3t6nEzqZncbJHZWrO9Zo5mx8xC
zVRmTWZDZrfNHpt9bPaz2Ys1s9csWbNuzc41tmvc1oSuSV9TvKZmzek1t9e8teaTNT+tebF26tp5
a5es3bh2/1qntX5r49bmreXX1q3tXntq7bW1D9d+uPbrtb+t01lnuI5at2Ld1nUH1jmv81kXuS5t
Hbeudl3XutF1t9c9Wfds3c/rXphPMZebLzQ3M99ubm3uYu5nnmieb15i3mDeb37e/Lb5Y/On5j+Y
//d6Yj21ftX6resPrHde778+Zn3O+pL1zeuH1l9d/8b6j9b/ukGyYfqGBRtMN2zfcHiDz4aYDRkb
VBtqNvRsGNtwfcPjDU83/LhhYuO0jfM3rty4eaPVRpeNARsTNuZu1G5s2Xhy48WN9zZ+sPHbjcIm
YhO1adWmHZsObfLaFLEpYxO7qX5T/6YLm+5t+mDTt5vGN+Ob52xesnnDZsvNLpsDNsduzt2s3dyy
eWjz1c0PNn+0+ZvNaIv+FtkWky1mW7Zvsd3iuSVsS9oW9ZbaLb1bTm+5veXJlmdbft2qt1W2dfHW
dVt3bbXfenRryNbErQVby7a2bh3aenHrw63vb/1xq7Bt6ra525ZuM99muc15W+y2om1127q3ndp2
ddv9bX/b9uW2X7f9z3Ziu/H2ZdvXb9+z3W67z/bI7Tnbue0N23u3n95+ffuj7R9u/3r7P3bo7Zi9
Y+GONTt273DYcXxH9I7sHeyOmh09O87suL3jnR2f7/htp/7OOTuX7Ny403Kn807/nTE7c3aW7Gze
ObDz4s77Oz/Y+e1OYdfUXeSu5bs27rLc5bjLe1f4rvRdyl0Vuzp2ndp1c9fbuz7f9fNuye4Zu6nd
Zru37bba7bU7ZHfS7vzdZbs7dw/vvrT79d0f7f51j86e6XuM9yzds3GPxR7HPd57wvek71HuqdjT
umdwz+U99/a8t+frPf/Ya7B39l567+q92/Ye3OuxN3Rvyt7CvRV7O/ee3ntj71t7n+391ULPYqbF
Agszi+0WNhaeFsEWCRYFFlqLZouTFhct7lo8sfjS4td9evtk+0z2rdu3d5/jPu99kfuy9zH7GvYN
7ju/7+6+J/ue7ftp3wvLqZbzLJdYmlvutjxk6WEZZJlkmW9ZZtlsOWB5zvK25VuWn1h+b/lf+/H9
c/Yv379xv8V+h/1e+0P3Z+xX7i/f37x/YP+5/bf3v7X/6f7v9j8/gB+Yc2DhAfMDFgecDvgdiDuQ
d6D0QOuB4QPXDrx54NMDP1vpWM2yWmhlbmVh5WTlaxVrlWXFW9VZdVudsrpp9abVp1Y/W+tYz7Je
aG1ubWHtYO1lHWGdYc1a11v3W1+wvmf9gfW31uMH9Q/OPDj/4JqDuw7aHPQ4GHUw7aDqYMXB1oPD
By8ffP3guwc/t5HYTLdZbLPOxsrmmE28jcqm0qbN5qTNRZs7Nh/Y/GAj2OK2MlvadpXtFltL28O2
x2xDbRNtC2y1to22J2wv2T6w/cj2G1t0SP/QnEMLD605tOPQoUNeh0IPpRwqPlR1qOvQ6UO3Dz05
9OWhf9rp2c20o+1W2+20s7M7Zhdhl2HH2tXYddidtnts99TuWzvBXmovt19hv9/+iL2ffbR9un2x
fZl9q/2w/WX7e/bv2X9tP+5g4DDbgXZY67DTwcbBzSHAIc2h2KHcod1h1OGqwwOH9x2+cvjnYeyw
/PCSwxsP7z/seNj3cMzhzMPc4YbDA4fPHb57+G+Hvzz8q6Oeo8zRxNHMcbujtaObY7BjsmO+Y7lj
s+OI4xXHe47vOj5z/KeTntMMJ8ppudMGp71Oh52OO0U5pTmpnWqdep3OOd11euL0pdOvzhLn6c6k
s6nzFmdrZ3fnEOdk5yLnSuc25xHn686PnT9z/tlFx8XQhXJZ5bLVxcbFwyXMJdVF5VLh0uJywuWS
y12XJy7PXH50geywI3OPLD2y/si+I85HfI7EHMk5UnKk+cjJI1eOPDry9Mh3RyZcp7kauy5z3exq
5ermGuCa4Jrryrk2uQ67Xna95/qB67duMreFbmvcdrjZunm6hbkluxW4ad1a3E64XXJ73e19t2/c
kDvmPtd9mftmdyt3N/dg9wT3XHfevdH9hPsl9wfuT92/cx/3MPCQe6zy2OJh7eHmEeSR5FHkUeHR
7jHqccXjnsd7Hl97/MPTwFPuucRzo6eFp4Pncc8ozyxPtWeNZ6fnac/bno89P5f9ZtaPZmAXhxn1
SRrhuCooWO1BEaZYYAYnJn+twks6a7lW6rTghBFmmEcEXxJCC9PxkoEe7iZF7MUSMzgum3bAudoG
roV6ghFW/1b0BkZ4YzHpXEkqLSzFiW+wxnKWqaI5nmU0tOGhSDwya7JuBU74RuHHw/iSWFpYiJd0
D/BnKCK4DhviW0vaFcJa0VFPCiEzg8HpkGDnBxjlKI0W4srwMLUbJUjKIf5ggKszCthwCh42Mpfl
02k0dyJGfup6LJoLN72sjRDBlT8Rzq+aCuX6XEM1101BuINAU3BldAEL/ZphroUsBIsINBVXRWcx
3pQwB/POErEZAinw4tBYxosS+rVwzxKsuJ9puUmi2zBY2Nyn8VswFN4h/yR+M1Qq8dNcP9+pUCUm
Mf4QQHmE9bcz6lZa3cI09UH2lAnEovr4s1SPGBWYfP9+/hQ1JlQBtQILS+L4GFoTz8X7QNsVuKa3
l++ikB1UrsdKovhoP9I6EsKqG/FIdZgqXIFIFCe/be+IGQoumEcoL4Jua3BiYip+kElWRinQBox4
sRICumHKCJFQDqgH70PC05+KiBerfq/fhBHhVgM4Ojb3cfh+jIg61Isjq7lvRh3ACI0QJ/fEDxQb
XMeJi6eP4oLpRflR/4swAVgjfq6BUXbQYHcSwqoL2Em+raRJxPU0yXxCOEkkC4da8EauVlMHADha
FokLy7bJ0bJwCOQQ//5gIg1PvR0rHmAgy49wUQeI9FpM1cp0tJGEa/wkrrgcLznRwQGILb4y4EjA
RqMDEA8k/GI4TQKtieKivMgXcpz4N+4mglK5kgR6QOiR8w1VbB9FRAD7JNB8PAdAvTCME1FZrLgA
sVwa0OuBzuE4SCkr1+eby9keiogvYvlsmk9jCyF3MoHNUWcq+KoytoMisphoJlcB8BKrIglVYqTI
LmHY8XBOkwghhKQ0JoJahfmmcpoUIFMzGIA572IhyRwMT6gzi9lEajWWWMzymUBmF7Hx1BosTiWS
hurcQiC3YvGTJMEqi9kC6qRg3llaqxnSGhN8dRnbTn2MtZex6moA+jubuJMU5Aicr2HUTTRR08Ay
JXQpK+5DormSYepodSNb0U0S7Y0sU0ZXQE0JTXQDHNutUMdGi2zbwzXwjQp1ZiaEPYke6LeZFrdV
ej4bBXSDyNA/48qkAjaIIga4RhHHXYeVpHCpkSQxyLXyHQpVfCITRhGjbBXfBB1ls5DPdKqJAcj0
jw1GjPG9JT2KPzibGGtj1MBHU3BVTDoDMa1rTYyyj0bLYY+mQkoHcb2PUQ7BXseVwcGALhI3uH5N
D+y0cLGxxe3wtr1npIMTbXKi82DV8cod0nF5n5y4jhSXYZvUIVKH+MB6zOfudiRFU4yInWiZsBQt
Hf8FLZ5NrMY0mWxeJFl0HE9J47hMOqQnsT6s8YiN0UBqd8pgxLlbRvtxvr6Na6LOHT+MpYAwzALR
FtydUB/caOxhcyK1K204wnj4usFeaNYKgrQVywkuTE/wkrKqNFZJEWY439zMdVDoEEZsxrOVquy8
AhhIcaotqye/PrPVWAUrpNJIDXfgXE0N10xVY44RrdHVSTURxo2VpVnaFD7J+Ou5hA8erI4pClGg
XRjRibWCbKmjvwJxmMgkURZn72DdVeKiP8PVaRlMAuUAQuYDXJ1fxKRTJy4IkJf4OdSkMfGU8BBL
TAEAmQYm/xI/xZZp2hWquEwGRGEZdjxORN/5ZC4qiURLo+SETRgOrFqSQgsrccImGvdP4EqSRULT
3s33wssSthF4JAh/2Csr8ZLmWq6Pgmh5QIfBx9gh1CS/hhG5I9gtAExhvZdgRHEL1sU1AmesxfgU
LiOGJM5+Iz9+5iz2d/3jp89ixOW/4co2pqlHjMqd4Jo1zQpVQjwTSEHWdmCy+GyaZC45ggQAgkAk
rgz0A2U28RmIXwXWWs0wtXQPVCzDVRmZTCyVj2Zit7kW7YBC2IBpw7kkSD6emYPHJfCwNUUh29bJ
t1OGaD2mPKk+BVD+cbj5D8IfiAK8Hvi4RsHkZjEZkOCHZU5qHS6dy8oiBTWeyqQymQquSsM2UMS4
E34BGBO2QHIc40oREHYOgnDI5FAE3HsskRNV33IYt4UbhvrFoGWH+CvUKTQP8tyW4pqWWq4TskvN
QOovE6NDfCdVDYlE7qkJhSnqHoC/Cdht3rAEkXRJAJfsB6rCHNOGcvEe5AsbEMJARHDxLqQwKxuo
9WJVgjv5wg6SW4KVkFy6HjreiGnDuCRXEtJZoU0S7qn2UQa/rCoCIkwZpECbsOJBpucCdK/EPdT+
ysCX9erfG6+EfkrxY0xiMcSoRU1jAZrETwlqyRyIfbgjE6METt1wclzAbp1QK3sg4S2gAu+sY1RN
9M/nPHHijrDDAROM9e8Iex1g9YyFA3J0oA8yPgzHlS+zWMP/lMXaAGl/EX/KYgXh9q9k198z//7P
ZFd3JlYVpuC7erlTFOGdx0LgDbg7B9KZA0GjJNHCEpj1VjAKiKAEMSoHCiA7liSC8lk+FbQDmx9C
EmHpk5sikgvzJ4kIIKCPSC40gERGIfJ/FUzWQmgvVqwViRgmQ52q4M/UsJ0UkcykqNMUXF8zGFGE
6hbL1lP3R6o6uun+nqYrt0lCXcd09pLoY7yDqwMJTKivsmwN9fcLTZd7aYI/zuWFkRPBuC8TB6Eo
gs/jcjNJk9Ttwa4KQhPGRR0jhSzckwlTQWy+JIpLciEDM/HA6EkjphEEQwv9Na7OygHBQHS1Mup6
+u+4OjuPiaOIPr65pHkyGJfMp0SBEG/lS6rptoqKphESZeLEmU5GBaIbx5UR0Wpvirg6rFZCvsli
SI0JUTtTxF2uCYyeRZgmlIs7RhL3uT5Nl0IZkcjYUsRbXGtJv0KYipVEcHGQi/Q211wCe5EQ6XgL
8j8nyv6eWb25jk2DT5HxtsJD6fsUhpknC+pjr06mv1ek1oVKISd1679lyBNrETUeiYizaIEO5H4v
vYEWoRVI8QZarjcn/fmW8Wo5sQT3UoeKPA8JJX1M320SeeKjbK2qS1GVVJPUkFw7VtpQ0ys9jPM1
pWwbdQsbKGfVNbS6iq3sJk+0wmES2byZZgndWA9Xzlcr1Ll5YJEewhLBQsihbXC+spJtoUYwmVRi
luiIy0wkZgnATrlQJTsoMeNryqHT24k+WGKhWGqN81UVcMMbvw9TzZR3kzKtxOxE+yZ8Q0seOobk
Z0pb2xUtjTV9FVelMl9bM/fKhIrIZimPXallmWZQ60xLJ4lUeBfXLGrdxETRyqAxNFt/GIf2Dk+C
MMFiWC7TwRwE23hki5vNjXLpxW+Yy9fiUKhNwmSSmdp2vIut5KsUMkmLVp2Xz4ZS7lhCAcvniW8F
D9lK9SR6YwmQlJ9HB+IyyWxtUltmc8pAF+TvNaukwBSq9Exgql/wU9UQGm3KzIEshwypzMTaviGL
V7fGS9k8likiffBUNlcVrOA7K9jL1OsQYtVqOb6iSqPRaEtLpYjE0uNZtrBQiqInQuVOZnhTe2lF
VYO2UFtQnSo9eVWeW8eyGY1VLWV1FS1SYXae/NIHeFGKMj8/OadWWZJRL432kldnVeVW5ORnFRZk
pEpli3IXCA+RsTwxFAfjhc+n+VJeU1otZSs1rJZCE4KzfAivzi7Pr8yUfov2ymWSvQtCugxOYrJZ
kgVCYrZcsIqHE0UdQqU8H09jC9TJip8xWYtkAdMAVhTkKM7G1EGq3LxiaVFOhHo1tTlP00rLZkoW
XMENTfWROlRQY8L7kAUZKeimuAfsSA/LTc9MQ0YCZqQp5/iymnNfnfj8JDKUGk7aiqLCXYhrmutA
H5wRpmPEGjy9MbsmsUka90/5zfcd8PT6wrLkRqnwMeqXD08QQdjzXShRXuwaZuJremZjx54BYSUi
bhpF/hT3yO9Z37t1p1rvVrTzWk2p1HAdXseXsJ2cNA8PZqLUiQrCHOdba8GKHojPxSISOT6R3obz
be2gyqvifDDnLI5PpsF2kTka5vLNTVw/hXQTN2JRmRyfIrZsbhSLZmD99QyYi+oGpraLPDNsj8vK
LufuHs1GhcjzhralT9HbWnei6q2E0iI+pVTK1jKNzSTi8P5XLBvH+FKC7xkw+ccaGXU/LSsblqib
mKZe8k28D6zSZoU6IQnsPVnZ6dzOiUi5jaDA25uqKhsbK/LL8hoTpWrkIb8qUFhiO8tm1VXXl9fU
90kFmyj5AyTF85OL8vIyU+rVfHK91OuIvD21NlerzM5RqWPTpNb6l/Ha7Oqc6lSO4VW8SjphND4o
b6otL62qlMKJjLzs4uLcHGnNTrmwzRdtw08LVXLB2xd5w+bGcseEM3ImBbggTwrRH5kWy2WqGW0N
iUIwZZoyr6hImp2doBY2UEsieG0WXVCi1taRMpOuXEg/MR/Xyk9BL90SRPhjE/QpYLwQSbwwrQM3
3AIZGonF0Qq0DlN2MM3nSWSCE9sw30TR0xFmwG9QBwlKaLASU3YybRdI0RHbBevRIDq5+rBE0Wmi
obfnX0vUB1PbJIqLmm5yaFgIDRoGkVEjigywGsMoQYqBsoNV3Q0cUc91UR8mQhGxRx9J4+DHRD1a
LCcscR7slGbqI8zbLyzVIr4rtbyoKUUahGcwWeAVyXQc1/ANFewZ6ok1BsSCmEqsha3gKxWgcdgE
6uik3Mujj+IaEK9NVB8mwyQLkgPweNiXubQbSF1RQI4mR2IpkxLVC0q0IHsuYq01DFNNqyvYyhZS
ZixZ0Nct0DHVeDMIrUow0UT1dhRLAlmbT3tOCu/J3k0kC1L8cVGO5dBHgIklCzTV5eA3nUr6fQSP
f40AuR31r0bo6wYWrsld4NqXA4ewVrSVjrYpStK5Ah7E2ADLllJXO7WQR9JdX9n1Ovl8E/4m16o9
qRBWY2CaxduRL0zxNKUyJS+D53MVyWXFfFaptAbrAxeqcVJet5NoDd4NtjC4Wxnp4CQWC1vk3gJ+
EsSQtqgiq75WW1Jbpy0sUVZlSk+MyovKCioKKsqqSkqrmqRl+qGdeWmqovwc48KywtKMuphoo1FQ
0qJEihfC5MdUDLuMMi2G/HxRCHX34DlMrjpf8XcM1NECppwprydls+YvEKY+kCO6R6DxaOQtR/N6
hHn4syXyqrzS4sq8/PyCorxCqZCKPpLXV5WWVJRLkQzLzVWpsvOlgsRLnoBDH9Gu7aA8Vv2AqmPk
3TdjcBv0rTwlFs/J4rhc2h7n68rA1OlMTsCiclg+iwZvqqGSBf7cjvVUsuoG+jGuzsljI6nolg6s
upFhKum7uGhu51LRze1YVxULfg60Sc8Cgx7yOeMyRJbeD3BUDfhkE5UfyrVBXNwh8sVq3IqJKfZX
oNVY8SjT9YR8vgJ/wLVDlpShJa7tbedOU+NLhb3yCQPs8smOtuFzUiaHLUwlQZwzecpscZupapny
FrIKh+lh2hUX3DuSrgVLhXcn1siHEBuMywSJwKDrcgJGr6sFO649JQ77/T25ykaummpLTsQIUJYt
TeDxfZi0ENwjPKtImVCQzfGZijrXkkSNp5StKAOdw2rhQAY11lVSUUt31lf3XCFtx8fl3T0VVW0D
naGV2R3xUkE5AXniOHEIiwQfDNwugNDs8HgmVTQre+thTxCHsZrK0pLa6obaEk1FadAJI3VmKhNK
CbMwbSnPl5bxPMvxnJTVcHAkBaYxqwCmOralB6tq+H2qM2Gqo5p7sIY6ESF4OdVxFGBShCseyKSL
+UgLMVU7U3+WvIoTHlhVZWVlVXV1jUZToikF31Wrdb1nJCZ3AfA2C+M1LFuiLauEDVIuLG8ySvYJ
On4scB+abXQ8alGMMCu97Vh1eK37bbEBJwVDqskqBI8DNwlkmugBNfNtVD8WtSTWw9m+x6EmqMbh
xKXartZbUsIH7K0qLQAOqzDCH3Rjvah1TLD+ZkbVTF/Az4cNpL8VVX+rdqTr5tZW+36X84EDcQMZ
J6VMXhhgLKBEvSHPCWCyObimswZW7wRm63w8Zl+09A38HivqF1VqgshjKzDIGdOA9R/NRfmTExx+
VH1UBd7M7K+G3ZAUA/FTNUSOe+FtYY3hNWH5HJginBSApIQ4csIU92JiVIC4TcNUvUwviGQpTgRg
XBtXytdKhcUYn8UVcMlSEKZQVq6plgqLIKUNypIgt40IxaLAR4T0N11c0yHqXyIM55oquFoKGWNE
LO530q/b/sQSYaoRkYF5Z7NaAHAVOJEjSm1I7ZucwCbRwVDjUS+5pLOTG4VZWhLUhXVytaKsScsU
+QOkegPeUy26Al/i6swCEbIZwU4D7gI4DBj3iamgam1fw4gLmMwlV9jghQuk/gVhtxdmeB8f4wZE
dx6M/ig+3J+0TcIjM7mSNBocWuIBfuWlsw9wbiSX5kbapOGxSZOOL4AG3Y2wL9pEAPBtbLhHdAzR
UlwZG60Gc9UGGwSHv0UhZkzGc2lcuFSYjxEf4eqkBGCthI5R7BIYGO3wvMSn+BA0bVWoIiJEeOjv
uCopC344drZjcBz6eg+jHBEzDpVBYSLW/T1+lq0RW0fHMYBq/YxdrGXVbTQYFw0d5BNwuOph8QlZ
zHgNOp2LZxcpk5ISNaWRirpQbbomXKq+xbGNcCbpi6uq+BSAIsDE8AE0D+BhEAhsJqvOJ4VP8AA4
cQALt3QS5QVRU4GdBscXJhPeLz4FMvPRp7hsIpctZyuaSGKOo0R2+BMkex4qJ/xqsW5wwloVwHkl
CTyczc5yx8U8SsAK5gMQLOZREgOf4WdBFIJuqmVqG8hbeB0n4hLEsyIsm8lRZynQYnCjJZPvfuIl
ah6q9gTUHEp1fp8RY0A0MPwO16kFrwwQkTAuzpEUsFwoBv8uMplxpoQerJmv0jRMwrgZXDaXKhXq
MRfgStg5OCxgF3cdwG05mGTtgHQICzFNDJcAfiMyxpUhkYAiTazGjoRzcOpDLFInpwKqKDhjwSni
DifAQlGGe0LmplBcBk9mgikb2SqwbPThERbBesVCpioITeDAdAA74xjpI6gww+ClK+tIODGB1k5C
gDaUE/zehY8A4tgL6FUSwC1w8JhA1vgQ4DWiRs0VudwXC08VzVngJxGgHJdgPYBENtEf4urUdCYS
IJtp+A2uF6ZDFRIDQxPjGwDCadd0i2qceL4Mv882lIwC0JjMRFEwY8TzHdgZwCIh/57Gief2cPqC
UQ6KwCTx3B8H9uVbAK9KA0uWECT4MXWEEvTRWgBL9DCfNE4TSfNRXCR4yYI+1s1VaAAsB1kGWBbf
WAkYM6D9A6Us+OqTRS2lLMg3B6wHnE/AOyAoEQnvkkxrIrloL+hCjgWBQoWYCwYwjrH+XWGJPSZr
/wR+g3t7TER7AIuBczF8Rws3SMGpMAJERUlvPXeRuhEBBI0BipWZRqa4NuGQL8tViIsJxbg3AEKh
L++mIThUw92mbqZDDQQwICZ0g4LzdSJy1VjP9VKN8Bsw+Q5R6IzAb1O4oQ7Kq+H3GlHYZWWQhYJb
Gd7AVUMYAGI8mnQ2Lx0ef40IVaW6k9YZk7iVBtQGrykr15aWVQh72wHo+j3sYRMIDTbgEepQFQQE
ZCMXYtFU6H0TFpPDaZJoPowL9xaP97qpXVV+CkLYDBGSUFEELxq7HIfmQlNINAa8LplMEVLw40d5
jT/MpjUWk8pxEDEAzogihU1wUzgARYRwGGvlKiGbF0A4Pp3LT4ZHjce8IyePE0E23MR0/Ig6EmIq
msFucSdM1OM+QIMRbgpnPYZxByYKHNvfvj/h9XdAwsAmeRmA2Sia8ENvwXEsSa1ElZLPulMCHEDY
1PMphohoec8n0bjg+0/5X+hg9L2c8Ay/hCNybr8nLJmP5xkcmfrIz1z2AaViBymU2x0/Ef4uJ4K9
h/Bzv8i9+88BwCpU4A7xIq9pErh0wFcRXiFHZcIOQHcNowXzHhxdm4tsokHcAlCKNuuhO2CwoaNX
UM6XyPobQYYWGkVEBscdiVnlKSzwFOZESpEjDrCOaNvuw8DYioAjYF/hvpqo0nRFRUxjcnuuNPn0
JSUKoVDH45pbI3TncNWlKmRkXxFUY9cvXVvkkZuiiM3wLhR2UUJwruYigHfT0Sz8XDcjIkmAK//l
eY8ET0Ygl0Gk5yR/mSLc1H4qYMc5X495I0NM3cE2QBzLXR3zKlzUw3ReB5qJBxWt6aphr1MA6RGv
wpjA8C/DmJ5wTgTCmsZ4SX8fsDDh+TLMCefSJlma8ASBHkPzoVz0UXJiCGJok8xwVH1c3Angf51Q
n7tIEl5MrBhNW42pupnOa/AnNOBRI8W4mwjjnqOIY5NRQ2Eq7Lk2bgzwRkCso8Re44+ShHfKqyFE
mMwbeAa4ebcIQLVfIYnjIB1A3E/B+c52UNiEL9wJzxMhbvWJVtyO8VdFKgi/aK4knOYTuYgAkvCD
V45WaLpbxXCYf7YYDOVT2MxgkgiejHWJuqO9GQJChAjgJ9LoPvgXRDiwBgD8caI8fHEU/wtNhDPx
6ngF19Yoapy/SJtICBYDwr8Ctn0DB9EwUd+D6FkyKQbATPlD/2saGmD7E1EJ4uaCvzSSl0QSUX+E
ByZlBoTa/iXHkAEEGaLg6SETfvKhIMJJRIO9AOagDkDgHWJv0bkcl0cfnETWgIyddAFg40f4kL+F
yYmMfA4CN3CgOyaFJJRhUWp36oUL5hjPaeHUmDIiEqLHEzLsKLAAnGIX0+9BI0mwiMmZIFRRMeog
CiyNoFhRxxGq2AQG9GYM5g5+dpxIJ4shYR3MA/DhSJHOYCDLfyZ2LENcMkIFobIEygQLhRWEeJ+q
jqluI9vwFq4RlCihambLR8gxfACs/XYgO5n2SyQ6iA/AAYVWoAeYzlFy3AQ/A2c1egD0TRa1kuB2
CtljjU0M00bLfpUQ6vQ0GALivD4AGAISo84Q/SUhCIsCV1cMKr4Mia/GIgGZS/+DtnkV956sB9vc
FosEH1hsn5nPRlNu7VewrpfqhulnS98Rjy6/y5ZxXQqCq68REYQgrBcc7nqAnye11kQ2fogJAclK
APvEh5NwqjOciRHxI+DIOIiaGuFhTIIqHmgIsfZSoLdfeniTdB+FgrGeSROG4JvLxPpVWP+kB0jw
L3WdLdYLug7+QoUmgotyJ4UE3AXgbhDK/+bhEa8iHra4q/oobFBCG8PH+ZCZwDCOEAaGRWri+vh2
hTo9FQ6cQDy7GWwlkOirMT6Ty0kliRauRrRw1mCaDC5TpOtZdR8tc9MCMNLYQxKtXJ0G/Iw9YmAz
Cxi4HXT1S+WVymWFkUQHmB0t0H0KWAUQPq0QO1uBwTmrfAgddEMlYFWTK0mAucG00D/iquRcNhjA
dq5ejLYBix2ihM5SjOiHdYeekuPBWoH4aT1Xp1DB6oLpEon9hSZOAkP9YfAOAegO/kp8EpijxF9M
4WGwi9vAYIkEk5n4d/PozzQx2s2oemlVF9N2liTO8H2iRW8uphrEgHw6DwHZHlrdyjSPkMSFClbV
Sv9hzREXWhlVtxj1V8dnikbTpWpWzHNoY+sfkJAGQVyGngfoXyYPz4J1LvoDvZPpHVFcihtJXG3+
T8HdwHDRXIdzG6/s08lYFoQUTmiGFcrIMAbOpT5gm0qGwOJLZCAJRFRhs0LA6ZxwWS1HVkApEiCV
3g/ECQTThX1P4bD/WO9kGokZTtRZlnlqt0rHZzbLiWamnG1QS1UdTOsYqEjZ6JU4NAVjGpnmNpKw
QsZD3w1+dxJOEu5FxrPRF2iJsAQO0KjRUrgumTORO0dPIpuQjNs+J+X2R50jdsRUZZYUl2cfH/bt
cRpYLUwzIuDvpix6Exl/iohP0Pw5a8eJLWjm98gIItC6fUivDFE6Z9D0YQTnUvXfRxK9Objk+Xx0
QN5xqN6ySZiDpo0ZRSDjpIdBT5tG68ca77c+Aa+6SjAE/94l40DsdinY8echUARMnpIKWD+x/hfP
d9ve3/lF9eOON9u+5l5Lh0MuHe/3fjkbxf9jGMHBUbAz0Cx7pDcn+opEtjY797nFc2s5seGfqnvc
p/vft7zbdrf1btOHu++6fZp31xtRa99zf633vf4vZiMFoj772efdfU9a/97x/ROk6/jtHAeTwTmS
Q7OIXR8mnW+95Pmg5FzzmZaLx++63lWeKbwbOtJ+oe1my8OWW80PZl9HJjeeBd7vftbz7OYPl9HC
OSZ6786ZNV9CrAdQsg0QRgQZRdswP4iqQTDOBNc01QPk3SP+PSZLfWQYKRhi40cRJSccAUkqBcTj
S6xNI8JGRD6ezeQBpIUIcMmZilqyEieK8DBQweCIUjhRgalPsNoBEp3FifNg70eLYjsQC4nmONDJ
SVxcGIRAceICQEs5LGiE/S0IRMUNnMmChA5qbRuYuXdwVVw8A6eZ/DFQvWK+xtsg2Os4iAeD+5BE
EX/DR1/GqNOSGNjeT7GGapbR/p5y8Q0mm1cIog28NuJbvANQSvGQWIGYSfEjrk7JZkJA73SBE0QM
umBoxtzwGwNwpEo//PoAJsyYeyQcMm6QLlZXJcrgc/B7ytzB8FBoMSAfCg8WWR8bHVKLeRGrYKtF
wNYBrSSmLI1w3WIeRmIEvLFIj3FdvCgTRBr+IAYB4V1lj/rUPfL5On1CmIdFAkoMxkMsF3yEfAFJ
RKAMo14WxXMhsGHBb9B0DPw/bZ15eFzlefavtMo5/vKmtAXPhMwQDQmrMZjEkGKWshmDTQwY27It
b/Iia58ZjTTaLMs2hkiaM2ckJ6y2tVnWbm1e5UUGO8QUSCAQ0kKbXv2u5mvakO5N2mOuY7f9Pe+Z
kUT7/TV6dGbOnDnzLvdz3/fzTGo42OvmAqfvMlq32PEnA1eAw26BVLtJGVupXb6CpxYb6yngI+Pi
bDmBK7dmADC505QpRo7JaSPGKjBRJJSK2tEi4hEzYsXB8Pb+9uR4UGqf/8e/lPtbKG8t6ML+IKq6
/2WWWrGmiuzUoR57BDReTaeQAkFW80Djtrk1UQhGgh8Hpxs5+amWotDXTQUzOwXL1ZX7zaW8RNRq
o2nUGnuH+V/4genc6B9/nJu8fOCHhvNAxPfMAFzIxo2nzFOf+sSo8GmWyssBjj/ifytvGdA8b8Kc
+LUvb2JCjpSsw0t1i3+yZC1WjdyY8atzvr5c6B1B8U7WOd9Qrqzuq/ku/+ANnzyq0TUE/+d1nzyq
sVWFzNbzPnlUR3PzDef6cz55VEeP55vuHf4tR49TCx7eZDj+Qd+p8EZDnS7ZYDjXDvvOlqzj+509
6RtZW2y4s2/EVmDO8Y2sKTYc83VfxnWFxcGzgy3HhVQsOVDdlsB6U6ULREvtKF/P576bVcxPz7U3
0CdK+mroCforzNJSAPFG9Lwa9LwO+RY2R1IyB2XUDKYkBknpERFDTN/i5XFzxNjVPRxQW6mCLJEO
ES2DAwJW89Ef0Nfjdi0+K1xbKeT0aruKjbWA0zBY4nYNqXwRPoyGEPs5BaSqGNxTnwlwGwJyUpV2
IePvLsDkjDGlyhnX1ZwjFYkE3LgZSUQEv1Tw3x2hVG1yF5u7GL12hdybs1L7X0x2YvzyiLQhTbdW
WlWJTSDkV5A3FIKvvSdkJZJ2IqTiVo1Vmm13vJrkdsStBqshO7VPYI6q4oJ2CnmS2ieksWo8Yg2k
2eluygivauq3OgCFN5mn7D7AAIi0Oo1A9exQzZG4rEERYzW3KgrErCyXeHNmTfJsUGSp64vtFuCh
bP5M9JiRG5ZFVTVXVQjibe4HgKnmOh3sMIqBv4IVK2vgxJiamzAgAncTNVWsfpkFedqfxk1pQJvB
ofYCZOpcreHsDK2QfzTiUZtrVOBg28nrEcOr5LhWvOX47ucRfW7XOvJujj+3m+P3GN5dngq9pyvs
aa+cFzZoHBDVDR7usNoxRcwxewE8HcQHrb7DASeOaNgn+DlxyOoeCzg5WIW62AuV1bDdqgu6c406
fFpAAGu/tb8r8Bd60caEmPxuU7I5mGxOJK2g8raTXxtdmNz2g3H17vL3GS+dF/5HZrMRFaMr+Pfp
8CpS63hlwF1j8q3j6lAtcbs8N3D5K+ZqKwJfrr6f/J7Vmg3XtzcVUB37xDX3b9qYVR1UHfvTIVtI
TVB53r13Tc/rp3p4MiQfraNGwZn9CeAwI6N5R4MVpS1Yl2UdDEHF92Io7T0o0JKgj6BfA+l/xQ72
gmwv/fuTuH0TB5MvnwmogeT3YQoTOJfYogbsDnHOldSw+TxppB19/wI+fU42sMEu4dLFZxeptoBi
h7mt3dmJqhrQlsKOJdsIo7EwqMbYzUC5bJ0cGdPvnng1+SLgebxfvHz/bjbHZaSp8QEJf2s2V1RK
eITEJ8PYqWPA4YPZiZICazOGQd6rVwyDsrGesHtbCPAxcOSk3QZqTj/tZI8FVm/utDpYPITIPSQw
tRuYet7ub4HI07NFXei1pIPLbeyMlYJRf2B340tpjkZhKAlGofi8eaLebk1ah0OJ3uSBt0GC8ysg
npzJHp9zf4V7PzvQjQPYfp2vwdE4cyKm+rqxBdaN1Q0D2teN1nK7ckvA3WuqW4zNzFZSYg/C9ATP
IjhsLNtSXbT7LfePrv3/H//f/53+D+o/UEjOo+CJAEv7YBQu0QPkQRHe8ZIK7/eEud6uT1a1zhJT
WU+nmMqWmhvJ1fGc3WKCmTaQFcdbZmk3WQ8L9wc0xzHzaPmF1bXArlwTcLH4HDZ/iDFnRPN/Y+ZF
XJ2j+u93xIHXCiF/q3HNB+pDSVolgs/8kCPdYhv6ptESSwn9+1NzONWNW2/6Px/JpJTn8Py/MYYZ
WeQMfB8V2yVp+TVTuEvLEZV8xW6vRhws29V6dS/YbW4DRsBhldjRNQG1q9DcCIkAJ4DCvXscP2GH
XO9N7HW/Y44lXxNgtbtWsjFQfNpM+iUQ0zxhavp/ELgEWwuV6vliSWSk481hnYtibiUwRzlHezr1
UjTXOUJ8ILuZRLmYc241plIg5RSJ9sDiI85exj6mm3eT7TJL8FAuI24130p2sBg180FXAlFgdzte
Sx4N/jODaZbRGrEr1gXcJNDp92T4VG8M4KQTdlfGFeOH63EDsKid9kRweBuvkUA6IeiA0aU9IQO8
QDsGew6zw/4lf8OXIa6MB3v5+w4jnx2RISInu8fIZ69Mn3mBsZUj6eC+mU+73yhsSrLZsrTJ9wnV
5xns9TkWpqmDdBDxpsCLoMdnDc9ZDduk3BVQ3psESgnzvGqaSG4eSZyAPqPN0YwnF3A8X+g1iFO3
aGaww1zFMloGnGiHexbjqWsZK/BTMx7ABQIVsWUDrOEQlvHil8365kZrp9Xu3natujxg5LH418lm
lVsMeer3D+eWGGrbM0dMZ5n/nW3UBkTWw4Le4X89ssZQseW9prPY/3YMG2Ksr98Y1Hj9Z6aK9fcb
/fARB0IEFTk9pnOX/0JFjqHa89cZa5hjCCJobwPO+3iDyvAGOV/1D7qPh4F54jLvbhX6fqGhxo8U
mNucL/pGjxSZ6p3VTxnOQ2/45FH9eMWThpuV9ePlPJhZsLHur3yolv1Lj9x1AxnkNy9P+k7ZvbIP
urPJNFN9XUIqPgtACoew25UD0pbTaYy7Ui10Xg4fuMxupW/WDWZqdBieX+V6nGOV/kpFUtJ6YeuQ
iAAqlxlFHKTTRrsNe5lrVTeX0+bi5eQPg2ot7GI8O3V4f/ICPKYH8RmttGrZADnHy+YjdImmrjbw
TF42MCDBRnhLdMjMABZe05u1LagQghO9WQw1SesTiSHRoqEybpg8YTSo8ngrZDTvBLCQm3BpspiF
zJa+HiFGZZaggcSkt4YO6tKBe68ZtUqFLdqix75H7KstADY+x8D+JFeX790OPP4rAy6MQD6sL/AS
3nElflBNoTLCU52vkn6ofCAPoLLW3lkVeJaNvkE2+vT45Z5x/aDVqfHcor336dj5WKhWwatwoNvt
XSBZT/8STtZ74XTsvbAA+BSXHKwqElCFYDOUDXJUQbqaWCRVFVMBb1kEiVcqH5rOyNqayqr4ZQ62
MflVCZ8dUcObw6UYhbl34iQeERxeypkwEVyFt+c1OVNp5kyijZVR/8B76noIVcZnJx1nlmlhKkPQ
6uIIFc6sIKn2VzByKI+F5MYJbShsLTdSl7IgrKj0UiES10tMCIXSL3vX77G7ddkHvbIKbZrE9EMY
t/agZaY69sEMAHwbALJfNzxaXXm1Ae5XDY8sJ64XZtYwNuhBRLwjXbzD/ig4+aDVNR6geeARuwuY
oRLlBQhzl181NtHMDuY2UfdCElr0AcO7RpXotTpOBJxHJadPwYJa+GVR/TZmaNbUZruGzf52TBHR
RAEsZ4Vdwzi6HR90JBH1WE9YzpwplpPbwwq10eiHHxMWFaTJ8eWGgDVYTo/qd2YbJ3R1hEoN0LxG
Otud7LISvbCghXZMs6ArsRQwrtOsaDzDinojXFcEid6l9mOywsilG+kosUp1hN4DhYnnSnUnk4mT
IavXaj8sdObLacZyu707DpLEFNYV+guE9jpxBHuXJ1Uf3CA00ukYZjLsxR26KgRIKccHBZUCwtoB
YYMYqOj2I5VY+skSA2El3sPdxm4s0vv08SGOeyfT34Ya4rr36+NenEYJ9eUCHqe2bO+rU5/btUf1
tyx0PfBijFQKhBerEfw49pq8ZzQrgfOTcBy9Hya1psKiVdYRcGxHZuP3dv1Ew27Z2I8CNNM1KfV4
IdSxbjFCw2rGtkkJynH2CdwLfLCqPUkOp0tcpIRFP11MjBwnRjDANz2FStIlLjNiffyUVzxWLv0G
1SmMDqBXefF2IXGmgIt8RdyIs/aAsKDh4gSgfQqbZA7CeB4OObfy4iKkEPXG4UQTpX+3m42FYSlL
OM8Hobjmy2ZTuArFn2KaNnBzU3lU4Eo6iOlnvgnxy2UESAfqSUlhZvtbAO9c47qgumhLzzRpkPhM
UAkEEmJ7V/JpIM/CQvr+OTf5x/iL/XlRGf1Yna/4sfeXUa+2KCyhn81yEZuluygq4dX+XndRzLhq
+ZnCnsXHZt1I/0n18y1jVe+spcn/tU9O5PU8NjxL3Wys8YQ7JL55QIZyEV6/ZTT1WQcnAyLJ36lr
4SgMY2eWDaGKDlepTu2keUzYIMEbKPtvG9ZIMmnjwaJnVxA4myDBwvoDvfWJadXsIDFznzNqkJ0a
QjhQBlgQSHZqhXRWnxoXD2sCO2g2FUtl4h17DaxfqX0a6e6+01y1TZem4QQZGU+dDeJaaPyWuZqy
Ry4riAJ/GFDzS5Dr7xpnhxNNmDVuAoJOBcB25zrM9mW4NxbwLLw4vMsK7Yqg/nDY6sCvfpEn3STF
iAff/B/BWxy532jss7qOkfeKGcB7Q0pk3DlGS1Uqujng1sH1iL5dZz9XG3BXc4hgh727LnALf6PR
19sN2wMuhLr7XWMTuhhUz8285CUcqZFGGsTONRonrF5yS/ckindNI4Uf4h2YYGmqliBTKKncU+ZK
qyp9/LKuihPCiRoemJYrd5urE/lyPtjB9SfOGI6x2Xfi/GZTxdd0C9x6PQ5AOx2B2vragO9MZL2h
LlTSdSr/kK9y2SFTXbz0O77cfzxnuNuz1KfxO6jF7Pb9U/xWbtqSRky4O6BguFIVdef0m87rfmdu
FGZAVbjfAstN+p0FFTgD1ApK+cqz7dHO5Bm0Zk97lipD+eoyVYbXaZn4rakYrI1s/Pl40L44JSt7
x0VmzgAO2QrzvPoltLNNBBBq7IM8E7UVIMReDa65D9v1i0lgUPkuGxLqNnypr5EDqkrMlA2YKb0a
3HSYKclVVcnvJupFphabTBsSaHhTYlnwyj3GMpptaUmzSjbGAiOXlstawmxI4vTJM1bv1DamRLiY
2X1llpFbLNBSWJjXjgX+itqw11JomHZbK1vzO8ahl5KYMxWjZmdt4IW5Zs5WaRumUrvshj0BjP1S
/igY9Vzw0n8ak1SZ9LGHFQsx4taD6KNUrao2lDVPSdth19cHlNc5TdpTt2Sj2k3LdLuh9egTJ/uU
Ho+eTGdZbaH3maD1VjXMCZYSDt8sbeT0Noaq152NMaRlu11fwbYGcf5y6BMKXRrk6f3ssnAbX6FG
QazOagjuQKzQ8lnZL4RvSsS3swqoo21Wc1/oF+ydu2Sh825DoqpaJLUTCHbU+NZUy7J8EjIFRqRq
V5KnnezWjMNsszkcFwPQaXJUtqCKclkfz7BgD2U3lYRldT9zJNHMgpypbFbpZYCYmc4CnY6pfiwp
kvjCiDYZsRLkF7MqqDfpJItx4hb6sG1NsGa763uc9UZfn2UdCqm7zdTAkH0BL0Yl37CEYwM4BJyc
CrfbUO/i7IsLO5hlqH/ANScCadUYxso2MnU1+z/PX3PV+UvvfobH9dE8MzpFJCzcbEqBN8CcqUE1
Y29wlFk2y+hik+sO/Zo152qjeczqJNPehgxDht1Ush3ahWIz+j2S6u/jPiVgYti7MP6e9Xagm82m
aFj2FTlrl9RIclb5u+cA8PQsf3/byAHDkmixKl5ezuoSbtyajWAChTp0MaAu76Wpd0wqeW47celv
jQtjCapG1WPOWl/rYLt9PuiYVCJuOmY61/lPbkVOiCwk71vm/zjyEIz72iLjNx5Prk7ftwGyPOv0
vfrBW3GuH/CdjUCmX3g4R45deEg/qD+LPmw4T/f7/jxKGebHuQ8ZzuZzPnlUa/DIkRdx+cPdksms
ga0FilclG0ia1pPWkGkAoL9DoHncTEAOgi/MOwIz01qqKxMHNdbfipxNyWOFXi7zYamrJCjjhPkU
X3AkKr0cr5ikMmx1YfKcfkmSSC6gqOSZqwJXKBSYGatSOH+Afm1yZ35AUW9svxBKWhbN7FVTp9U2
ELjmCw/395iKKsyD/YG/hvTpEFK6id+3OBn4zDBP233UcKnm0nLgwuVTxspyrbgniqy6wsDN5jP6
KwJr11mPBN1wt/OsoRINtVZULBZYrmCRVeL5PXDIazRH/LyUocIc7j0IFTXuc4xKAL1KtiZbWgKv
pNrw9auWkuSeFYHLp81n+X0B1H5SH9wBFHkvwo7DmtIyeIRGAp8tN06NJQA2V3V7/i3Pa0vq1Ysj
cJDde6fM7UywiyuC3pRhD2WFzi+1b3xWFrzETk3sSNFXl6wPLUMBdtUxhL5eKVnmngsFcRSbIIQi
3lTAnQQAo6IqgZRHRzTVRRFqRVgq5o7Zgy2AtWhxglUB2AgP9pGZqIwz9dQJlhqw2K1SXScnOg3v
2UehS7WsIGdTYwJiy8vlrJNYWCFBq2tkDr2OTxdwyQpRXiuEJtSnGBLidYJ9z3fphYiDXvH0m+2W
oIxQek6qi8kDcrHhGpl6znWVfDGX7mRLdW9gT+WXZnTc7Yu7N1A17NxYQdnupUU9vgp3LsPCyYu7
9QbV9iIWLDF3JRsT1L/ihm9DmzeP2yMzjcH8ZyI1KOI/uOIn2hjM39809mIMzgnwn/OskUNixKBm
UbgF+U8Xq6ZwhsIQCj0scBN8vbg8U2aMyXhJtSmOJNYjgqXhzBHs8dsiJpvuXgALQf2PjVO8/4C8
pwihTQWbExTntxLcmSn5jfGZ6DIwmjiNWRtDlOf2msCTWkIAttTWL0p9ZgSbCKbKhOU1cZNypjPf
C1id9O3HZzrfeGPQagaE/yHG0GfMMUhvvp6aOmHOyX1zqZVNl9xePdOseQ1HNH92B2vfNfjO84RM
m0cwOxPwQVzfzNd8hSNplxpabuYI9w87TvoE8pprM0ck+Grmffh4uvD5c//ygs2N6SsMTD9ZyMj0
uwnD9zUQ+BTdJ77RdMAt4/5Bnc8wiYKGtTOsk9eJi+yAPRSUv+WrT0XyAk/UgC89+8mGwJPS6QL/
Z0TaXizKJ/ge2CSFnQsiSBU+OWY6K/3vFWL/OL92FcvxpO/CWoi5j3IXGk7OOd+f5j7CN7zoRfMY
RpVDIecqU60sFZUR9rAK3LkKRAvRGbbjW3AO5qVk2aYNyvjJFITTasRgKqAXGE3jiVGchelvBA/g
cOL0OYn1d8f44VO+PhVPeQTTxxGTPc/gWsqndeU18XFI0LQ3d+r50/Fo4hh3LR0zvLzXr+f9NKMq
78fx9ZSukvR8m/qCxOhpOLCNKc850do7RKt8VcgaC797EzTWAfuQjr2C/dbuAzQbzjj3+A4gkmFr
wswj5AzvOxH2BsYuHbeL0y/mGSogng+1ib7Q3JZ85Ujgt+Yg9P0+SJWqcrExzDYKPBLGqrT21ASS
ND0FDVIAULsqsLjerMCRyHpBwwih/KktgEYYA690hHDgiP9onOyXHhTxmKxux9BqevRSx5mR0I+z
uoK5sJKzDqRbR+ihwuhQJ70Jrt3CYYlxlbKi4rov3CgL7sk3Ek0nQuIbD+cnHgvOcDpFU5RNZ2JM
DYw2HR+WBWjKCXVKX0xzSaVAwVNcCc6KTJBeWsQztYFGBhjJWWr0lejYu7LMoFanPI8VvvsyO4aD
+7T3fD0DxHN1euqTeFcyFe8N23L8PGIMy6deLMVHdWHMajoeooisqTCftuVk570sps1F0ptXPX2+
qOfBU7P63YOYnWq+//yh6Mjxa2naLzUbdhmG6SZT3ZD5O2VuTKxrxjy9zNxuPZ94LtvufRHFQu0x
SvB5ovuT/H2MnNMmy3FpWHacvzJprCA6Eat5oR1fEaAFgPpXs4eKEqxiZLT/jkwitFu6ROOaL3VR
5/il82f5gYLVf26MpHpb2U3ZFj671nzfHmaDbCout+5hefxC5qq2M++RtQ522p3BF1kqbpeChQra
7T7MAYJSOwa8eYrgzsw1PGo+4eX4TM9h6/BPWIQ4hv8WY8uj3hM5hb7YJ5iXZZ45X3+ESu8jzPw8
j3vnkhWTp2TOQmd+aXQw/V96LyymwcLM/3pvIlezVd5Enssw0LveYm+hS1sYn+Cipjo4LPaCYruS
D8ntvOyDaiwhbfoNVvLFnExfMSvHkDX2HunzYn4loDCzOFgj7wbUk70/xjkS8/W+x7a2Lues6Tzk
v7COpbHgmXHTyfG/U4DsUHxhxHC3ZhW/OULXnwt4jbZmifNIleUOmc58/2QZUy78FB2AFvt/FP5O
Wi2Z7z8VwWcSu3SP8Tbfaw8V/+rI2W3mujm+s9uAycff22K6q/1PHX+HOiX32j7TGfY7t5YjHapn
wL6svZ4bndIQgbvxlCzEYEcZXX+EdNUPXFdrEhFZ22gChKsbf3VuokQIG8zzo9Yh1jpRF0gGIJP7
uyVpzgN7bpYzV5UGRCnwIHYZLpOtyQZq5KThehNtBuCDYbMZS11S06rKwcrbsQyJ/Qx7R43IF1Qt
QJ9XeYQRpXXtwOGSArKrK1cbqzUroJoR7CmRvcnAvwwqx6xbJ/E1fyf1CDQHwILWbr6M5YCynFR1
MhENfMeMkppvB6J6c26PudAqwUGeiZvZM/W8Ywp5488bvrTD11MqM5zF8BpjCD8iPwshkFcmAhD4
Kd3Xg7iDGl74py8Li7NzO5kzuS2adoe2M6aDg1Yb3kYq7snLf0kh4B5pzIPlBHjUXKbpvknyJ4Di
PHpeRYQofJNCIdAZS2NZKsKC85bduRfVVbdS4XLVu8Dl/uxbDdpd1C4LqD9DEmb9M2Wgxx5BSw/G
KJW6NK/XF3Nv6KWKkSrau7ETXu984wvSf4ylBZg2F/gUkmoiXYS8I7kqKJs3ZGVeqbUkePnnxpIS
e+/WEDT73okRfo1LiU9rLDFxLnDpb3lpjXEMrlT6WSinHcSolVm8t9hpkbbTvU3IQt3naWSiG52I
3LgXzr6yMU+Ip8aTVg+NTi67xsPYZRBWsJXnSjVpmpq6h7n3ECRXRBDbnUOffRFj4uRHhnPDWt/k
X4L9Fn/yPrzVg75Pfvqgqdbfe8Z06vy/WM875m84ajp/7J/MZzoNTZaZa//NNzgZ5ocq3Ouho45m
6KglxfrjzTP3Hu23fwYFpfOp1miqcBtIRWe/aaSypkgb7crtGB7/9IcRFGqd0FMl3cWFqTJxPqDW
looSSCWylBut5ZxUE1Ta9SQua2nxEw3ZXrCtINVS7CWcAzYEWCHuOioARseorVRFpKaYpSJ2IT3J
iuDJ6rNTbVqZKUGDJLclaSVrQlYSSU6XJV0+xG1eRynG5/4rPVLEdhXRbVHKyE61HYwXU3iDuJTu
tGXTnQ3Egdtf1K+w10OFgFeyA0KalYNtauQHDCrYbmOcZiqo1P2zVtKCaS+14+KzP3g+QHfPMzqf
wmaUxKeCSQldaxeUFc96GWL2ANYPfhKEE1ZS9jHfXIt9HSGn9cR46q3gZzcZF7FXngyp/fulT9ZL
1hf/xFTdNMvCA8EmjDW8MqC698opbDtppUIfmqrPPii+FeghBqC43ae0EIQSkS/S8R6xzYzSfALB
JlxB5ne5CUoi+eq0FnESasWTAKJ1gnvOvpIUaw4ldWkJ4KhkmcCnoUl8MJ7js6Zc5I5z9hDiQFNh
KS6xK5Ddf2J3Swekb8sWFn9WWrQMklbyywiCEz4gOWaTrq6WfPiD5D5xFJNGkpp+CPsH1oH24lKd
q8toYPRolnN1KY/8xFABv4F3qXHcV+AuGWfbmtOPydLJ9jt3RVkf1LwMDY92/3DWpXtjl+81Mo9q
wrg4pHvKgBKiHxqv0yTGg00Ru5pVRiQVT5kr5I3+wGzaVIreciOzlv5SXV0MzEH+1oT1zngAQhfE
ld5Nl2YKAlx7RnOip5e8bTqb/H/6NAtLzsYLpvOo/1QOe9nA31F0Veu/ewBufOQHxebK3/iGf0DS
99P1jxlO3hmfPKqIezOc0YTfmR/hfZi8ASavbG1R2doS7uIUHe5e9aoLlHOL+5jPeeyw+xgVSF6D
OgheTfirtHBBwzqvImnNtBrfptV4T9iQllBSZIcAXyX1btQaweUzoTcUpmQX3ZIqZPanU607ZCU8
xvc/tRFWlxB4pDN4hS5HeUzo0uxU97Bsb1IyFIMkOiRy/3SVEMahUaRjqzwRyU71jcC9qTJdKaIF
YE/j5Wo1+9eCF4WsYlrW1aUbqpZ6nz3SC2RflzABtbqV2oJM2L7PakYE/AbU6y46kKlDWNd0Y8RO
PFzTCuCO50QhPAw9QfEFrG+1Xc/yAgsgXQX5sQV+UIN9/nDqkEBHYHZ5qoLjEzjm2ln/OXcSTnZa
DaM4gWRiEub1WHo2sIN1twCRoUCZjm8h4zDDYCFpMMl2xzxJHyQ78Ygo53uH+FGUSn4JRTlfLmZ/
fYAu/fKonAUx1APnR70+54EYGrJy6iMQIw9neY8UGE44TxjOXKr1nGV5ppprbN2pmUz2u7nGNvY+
gAx/P15pSsal761yqtnB2sRkF5ffCNEjvrOLrK2HUf5aZkizt9Dsyyt9EWhZvGjEdLb5f1b8KH8v
5e91/veLF4PtHsGFXOj/uARyMuaGsLkc8ztzYshAaCB3oYFc9Dv3Vbj3AcjZWUiIvZ1lXfN61Gbh
N9JFb5liUsbaCRLezdqh0bLWlrE27X/wbA2Fnk/4G1gMXpFss1RXLLkvZaU69DiiU5z4Q2i61dsm
wka6boyZ7ZWGwRCmx1l/l3hl0bzLMKlJJZDYvjPSNUqWJ12LPEwhTOw53Y9wWnuWHqEYFnXCUlgB
U3kFqWcEvAINXRET7n6cehqKdSvq5EaPd1jNqNIYLUkygUWeAq/7EWpxdoKuHVNiLi+GBMVF2VRU
IDDp3IuZlRlAuFWEUSEDE7StI3AXlooWGnQX+tSPKJipx4TJz6Bwn2wgpPRjhSnFgHuDLhDGlSBN
ErWC7H6NQfX7WgNArJFq6SUUN89Qq52o/5OiBw0nNur7eRFWxc+uo3Z5erBfud4QEFCoQcCGwJXr
MuBRuA7d8vU47i0R9dJ93BhX686dxVOe6zt3NtdUD7rf+cR0ful3nnmQqlh+w/XRYtP9yO/eOSIl
zUupRQS+/aHZOjaaeiOoBNiDnjyGJQe6BZmPAMAxHQD5p6iv9GiaeYkzRlNfpxjJsb54jLcYVEo1
5+hmYXTcl4SokWnDUALV90mzDRXxGnjqel5VWyXdRx6gZqRT1qNEd7KN2qRGbdXDJyylhoAPKRUu
FutEev/IzCrV7vlgkdBraxNxdnK4YM+l4DXCnHY5bBcPifrc95KuxyoukTxZ6rHYX71gek8Pi7NF
XYD+JKMXxsCuhRB1bi8hsb20athX4n5zGET7bBFf0aWxUV+RmzdqqhwjJya9/6Q11HCfsD13D/+C
6oAS3/CnbFyDPwmb7kb/0sF3mdLJHXhSpBXQBHvJ1DhYH7gSMNXGGQ4pXfpcJNWlZbqa0AlcLqci
M+PwS3W+JN1Pq7QUfgv3U9t0vOUetXJ/h02m1Nyf3Ad7OhsbZSfQQknL25EANkpvxqn92tTx/0xr
1y6rNih3l3ZqVVnWrhpxQNN/VHS8hzK6nVRWzf+sy+cCiSKr2X0X+s9HVhnqV+vmGU71Wd+n66BH
p2kzj8DcglLCiMPTyvY45VxMm46ayspEjdhvLEVf4SfIaFHVPhwQg+p+Vg81SP9hui/Mlf7D1VRM
i1gnJLzgMvYPjaYKykBT6gIgrE98GdLgjtl9aUHs8gJBNgvKeVTOJnoA94gsEBNFVRXNHzWdav9f
F9FfN+4uQ9P+v35naxwfpBK7PSAXOyNDuzd1QHoB0Pei1t5lh2dlagpvy9QUuo8Xi8HwNvdxn7L7
UQI6ZkFJ4E1rsMtnqf8GMOQUQgplbmRzdHJlYW0KZW5kb2JqCjIzIDAgb2JqCjI2NTkwCmVuZG9i
agoyMSAwIG9iago8PCAvVHlwZSAvRm9udERlc2NyaXB0b3IgL0FzY2VudCA4ODAgL0NhcEhlaWdo
dCA4MDEgL0Rlc2NlbnQgLTEyMCAvRmxhZ3MgMzIKL0ZvbnRCQm94IFstMzIzIC01ODAgMTQ2MCAx
MjAzXSAvRm9udE5hbWUgL0hESFVGRytIaXJhS2FrdVN0ZC1XOCAvSXRhbGljQW5nbGUKMCAvU3Rl
bVYgNTIgL0F2Z1dpZHRoIDEwMDAgL0xlYWRpbmcgNTAwIC9NYXhXaWR0aCAxNzE5IC9TdGVtSCA1
MiAvWEhlaWdodAo1NzkgL0ZvbnRGaWxlMyAyMiAwIFIgPj4KZW5kb2JqCjI0IDAgb2JqClsgXQpl
bmRvYmoKMjAgMCBvYmoKPDwgL1R5cGUgL0ZvbnQgL1N1YnR5cGUgL0NJREZvbnRUeXBlMCAvQmFz
ZUZvbnQgL0hESFVGRytIaXJhS2FrdVN0ZC1XOCAvQ0lEU3lzdGVtSW5mbwo8PCAvUmVnaXN0cnkg
KEFkb2JlKSAvT3JkZXJpbmcgKEphcGFuMSkgL1N1cHBsZW1lbnQgMyA+PiAvRm9udERlc2NyaXB0
b3IgMjEgMCBSCi9XIDI0IDAgUiAvRFcgMTAwMCA+PgplbmRvYmoKMTEgMCBvYmoKPDwgL1R5cGUg
L0ZvbnQgL1N1YnR5cGUgL1R5cGUwIC9FbmNvZGluZyAvSWRlbnRpdHktSCAvQmFzZUZvbnQgL0hE
SFVGRytIaXJhS2FrdVN0ZC1XOAovRGVzY2VuZGFudEZvbnRzIFsgMjAgMCBSIF0gPj4KZW5kb2Jq
CjI3IDAgb2JqCjw8IC9MZW5ndGggMjggMCBSIC9MZW5ndGgxIDI2NDggL0ZpbHRlciAvRmxhdGVE
ZWNvZGUgPj4Kc3RyZWFtCngBxVVdaFv3FT//+79Xsr6uLFlftqzPa1myJMeSZUl1cWS7kZ04jhel
/qg0Vid27MRa4sV0TpfSrDWDwibY2o1CWsbG3sagBZWOoYpBQ8fotpf1OZixl5WyPexhbFCo5f3u
1Y1Y+zD6MNiVzj2/c/7nS+eec3X43N1dstARccpc3986IO0Sq2BD158/DHdldg1cvHFwc1+Xd4iE
j2/efuFGVxbvE0m/3Nvdgl67PsO9sAdFV2RT4CN7+4f3urJ4Cdx6+851/Vw8hNy3v3VPz0/HkMPf
2Nrf7dqbM+CJg+d29XOm1tdHszRH8/QUnaMyLdBi1/bxXSZiwFy4Tqp3VyJyoXKy/K2ngc3S8ca7
V+0z/6RB/omq/813a9rxh+986j95u/Mn8afiH6AwwbF7wYd/7fSQgmLi9FenTdGsZdIPNcZbZE61
iFLI5wJPL7fIVKm+w9gPai12+kqLyoH3EJFf3RxvEUuHwwv1cpNdgyCkoUhGgHg6vNjkscWnq0ot
3Ag3lnYa4cXw3tZOU4xpHAe7jdpEuEmr1Trua9VIc67m78HdWu1JxBHVOHCBeaOGCF/XI4BrqokT
GEnp5XCTj1aqV6rNo7K/OVeu+SOR8ELzYaXafFj2R2o1WBl6laLib9d9es1G1GxI4ryvG2W12pzz
N6nWaKgxV6tKpHnUaPgb+B263KKHX1Aw+qJiTlegE4iBTiy02FEFwcCUiF9VKBElgjprZeQ2pZdX
qwuoNKJWav7vLbf0fghsrSjforXc9j9qufxlWm7/Ui3v71X6uZY7UHO/2nLn/7HlA//RcuzEJ7QB
ioGi2AKGj7oxVjKA1Evdx+41xTaER3jnbGAb17gZyIi3UK4NhjipNpYbywEukARObTBBUzBEUw8s
MOGpTHYg4ogUQQNKPrfBOh3hArvdef2tl17i5s/+tSLcO3n3iESKnf6Mf8AD5CA/FegO/aNNRbw2
1EAhsmvcCjnkcE63KaGjFoX6WzR+3KYsbNSzFp2DZuWjNl3u2RSh2YXmRs+/riNqUXaiRVOgGGhm
ok01qujJZiirJbPTY5TooVIPlXvooo5aVEHC1Y9AiFkDXgNe0/Em8Cby3Oj51XWUybIgM8ra9wyL
l1gxyNgZHgfzcpm5HS5ZUHAQZCWWd0yNAhQLHqZqC3FPSTjL4Nt1zBeVqMGYDwqTuSdhXXTL3M7c
AAaPF0bCit1oNzkN8qTSH/FOlM2uKJs3B0cS7kL6K6E/m0xyfHFasFjSxaDFILlG3f2GzoxVmc0x
XzwzZFJWMka7bXJ74uXvZ88HPPFUfiSZj0c9BqvNabsWmx/mBlvivCKYBacojIUW37hokXnpdy/2
mTrP5+/c2o77tv+4I9yWfPaTMV9s0OqKe82iddgtbOYu+yRBEKwnPJgcto3FDIIjOvx3d2bYE3E6
xgdf/TnnJYEJ+KYls+xxfGvHk3LJfoc36SFMXfT0Df5bHsKIRmmNDugV+jFbatM2rWuP9C7d13iD
fqjxRbqocQsNaXwWU8HxnzCLRzaNxzSNx+SG7zTGYF6iRzD7K0h4dt6P2R+iQdAYaBq0BKqB6qAX
QN8DvQn6Beg90O9BtmdbNITYAcQOYBxUrAAryFPAuUWb34vQVqCtQHukZ2/TAx11q/vJMSb3AUIM
wChOilZ+Af+w6rq4aYAeaJMb0BFSIGgOQZPHLYoDT2AdKr3TAjRPQ/PVnuetHjpC9DiiZbISZuvx
LLoMRpkbR+MlXiyog5ib9MS1iQMqCUX3GaZgziYLRVlwBZjL41UcEYdSYmfZDEMYj7fEpKkzQtwD
R4y8Ntva7PL8gCbCozflXgz655RwVFPyH43XcvHiTvJRX58oib7MctEZe2Il6+MGweBWsBhrZ28p
yfvnpy7kE66RqysWj3X5m26mTlCp85cZgWPWQnuy++SYCZyPBfq5bJSMzoDHHnQNRl19vzYaJWkR
oSVvUHbyAyPQkiipxu6gK7L+xOVoZnQ9yT6WXJbCXmbiyqWl0ahz+Gx5OXnl5azELZ+K4ixX7YVZ
UQzMj0rc4fcHXe+PzAXZdzp3w6XYh76o32szsnA54k+Ehuxmm0lyODqvLd3d3khZbGJiefPFZ9ic
5O3vrKa3UqNXc2w7V/FIImPyOVietOfXR0bW5wU2GA/57QZRfXszcupvcgNhMZYXnnpm6XJqde3C
bv2wvnZu/FL95t7hvwGitCJxCmVuZHN0cmVhbQplbmRvYmoKMjggMCBvYmoKMTY1NQplbmRvYmoK
MjYgMCBvYmoKPDwgL1R5cGUgL0ZvbnREZXNjcmlwdG9yIC9Bc2NlbnQgODYwIC9DYXBIZWlnaHQg
Nzc4IC9EZXNjZW50IC0xNDAgL0ZsYWdzIDMyCi9Gb250QkJveCBbLTU5NSAtMjkwIDExODIgMTIy
Nl0gL0ZvbnROYW1lIC9LRUJXSU8rU1RIZWl0aVRDLUxpZ2h0IC9JdGFsaWNBbmdsZQowIC9TdGVt
ViA3OCAvQXZnV2lkdGggOTk3IC9MZWFkaW5nIDMwIC9NYXhXaWR0aCAxMDY0IC9TdGVtSCAyOSAv
WEhlaWdodCA1OTEKL0ZvbnRGaWxlMiAyNyAwIFIgPj4KZW5kb2JqCjI5IDAgb2JqClsgXQplbmRv
YmoKMzAgMCBvYmoKPDwgL0xlbmd0aCAzMSAwIFIgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3Ry
ZWFtCngB7dCBCQAACMOw6f9H+4UwSC8oSUSAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAEC
BAgQIECAAAECBAgQIECAQJ3A1B0bJkCAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQ
IECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECA
AAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAEC
BAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQIECAAAECBAgQ
IECAAAECBAgQIECAAIEfgT27YAAECmVuZHN0cmVhbQplbmRvYmoKMzEgMCBvYmoKMzAyCmVuZG9i
agoyNSAwIG9iago8PCAvVHlwZSAvRm9udCAvU3VidHlwZSAvQ0lERm9udFR5cGUyIC9CYXNlRm9u
dCAvS0VCV0lPK1NUSGVpdGlUQy1MaWdodCAvQ0lEU3lzdGVtSW5mbwo8PCAvUmVnaXN0cnkgKEFk
b2JlKSAvT3JkZXJpbmcgKElkZW50aXR5KSAvU3VwcGxlbWVudCAwID4+IC9Gb250RGVzY3JpcHRv
cgoyNiAwIFIgL1cgMjkgMCBSIC9EVyAxMDAwIC9DSURUb0dJRE1hcCAzMCAwIFIgPj4KZW5kb2Jq
CjMyIDAgb2JqCjw8IC9MZW5ndGggMzMgMCBSIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVh
bQp4AV2QTWrEMAyF9z6FltPFkGRoOg0YwzClkEV/aNoDOLYSDI1tHGeR21d20ilUYIElfY8nFdf2
qbUmQvEenOowwmCsDji7JSiEHkdjWXUCbVTcf7mmJulZQXC3zhGn1g4OOGcAxQchcwwrHC7a9XiX
am9BYzB2hMPXtcuVbvH+Gye0EUomBGgcSO5F+lc5IRQZPbaa+iauR6L+Jj5Xj0COiKg2S8ppnL1U
GKQdkfGSQvBnCsHQ6n/t0wb1wz5dPZT3gm+57utGMH5udC323DyeZZb5BZJi2v7mVi0hkNF8orxD
8mYs3q7onU8+8vsBqpl2pQplbmRzdHJlYW0KZW5kb2JqCjMzIDAgb2JqCjI0MgplbmRvYmoKMTAg
MCBvYmoKPDwgL1R5cGUgL0ZvbnQgL1N1YnR5cGUgL1R5cGUwIC9FbmNvZGluZyAvSWRlbnRpdHkt
SCAvVG9Vbmljb2RlIDMyIDAgUiAvQmFzZUZvbnQKL0tFQldJTytTVEhlaXRpVEMtTGlnaHQgL0Rl
c2NlbmRhbnRGb250cyBbIDI1IDAgUiBdID4+CmVuZG9iagozNCAwIG9iagooV29ya2Jvb2sxKQpl
bmRvYmoKMzUgMCBvYmoKKE1hYyBPUyBYIDEwLjYuNyBRdWFydHogUERGQ29udGV4dCkKZW5kb2Jq
CjM2IDAgb2JqCihPd2VuIE8nTWFsbGV5KQplbmRvYmoKMzcgMCBvYmoKKCkKZW5kb2JqCjM4IDAg
b2JqCihNaWNyb3NvZnQgRXhjZWwpCmVuZG9iagozOSAwIG9iagooRDoyMDExMDYyNDA3MTg0Mlow
MCcwMCcpCmVuZG9iago0MCAwIG9iagooKQplbmRvYmoKNDEgMCBvYmoKWyAoKSBdCmVuZG9iagox
IDAgb2JqCjw8IC9UaXRsZSAzNCAwIFIgL0F1dGhvciAzNiAwIFIgL1N1YmplY3QgMzcgMCBSIC9Q
cm9kdWNlciAzNSAwIFIgL0NyZWF0b3IKMzggMCBSIC9DcmVhdGlvbkRhdGUgMzkgMCBSIC9Nb2RE
YXRlIDM5IDAgUiAvS2V5d29yZHMgNDAgMCBSIC9BQVBMOktleXdvcmRzCjQxIDAgUiA+PgplbmRv
YmoKeHJlZgowIDQyCjAwMDAwMDAwMDAgNjU1MzUgZiAKMDAwMDA1NzY4MyAwMDAwMCBuIAowMDAw
MDA4MDA5IDAwMDAwIG4gCjAwMDAwMTEwMDkgMDAwMDAgbiAKMDAwMDAwMDAyMiAwMDAwMCBuIAow
MDAwMDA3OTg5IDAwMDAwIG4gCjAwMDAwMDgxMTMgMDAwMDAgbiAKMDAwMDAxMDk3MyAwMDAwMCBu
IAowMDAwMDI2NzU1IDAwMDAwIG4gCjAwMDAwMDAwMDAgMDAwMDAgbiAKMDAwMDA1NzI4MiAwMDAw
MCBuIAowMDAwMDU0MTMyIDAwMDAwIG4gCjAwMDAwMDAwMDAgMDAwMDAgbiAKMDAwMDAwODIzNyAw
MDAwMCBuIAowMDAwMDEwOTUyIDAwMDAwIG4gCjAwMDAwMTEwOTIgMDAwMDAgbiAKMDAwMDAxMTE0
MiAwMDAwMCBuIAowMDAwMDI2MTUyIDAwMDAwIG4gCjAwMDAwMjYxNzQgMDAwMDAgbiAKMDAwMDAy
NjQxMiAwMDAwMCBuIAowMDAwMDUzOTI4IDAwMDAwIG4gCjAwMDAwNTM2MzkgMDAwMDAgbiAKMDAw
MDAyNjkyNyAwMDAwMCBuIAowMDAwMDUzNjE3IDAwMDAwIG4gCjAwMDAwNTM5MDggMDAwMDAgbiAK
MDAwMDA1NjcxNyAwMDAwMCBuIAowMDAwMDU2MDMxIDAwMDAwIG4gCjAwMDAwNTQyNjUgMDAwMDAg
biAKMDAwMDA1NjAxMCAwMDAwMCBuIAowMDAwMDU2Mjk5IDAwMDAwIG4gCjAwMDAwNTYzMTkgMDAw
MDAgbiAKMDAwMDA1NjY5NyAwMDAwMCBuIAowMDAwMDU2OTQ0IDAwMDAwIG4gCjAwMDAwNTcyNjIg
MDAwMDAgbiAKMDAwMDA1NzQzNCAwMDAwMCBuIAowMDAwMDU3NDYyIDAwMDAwIG4gCjAwMDAwNTc1
MTQgMDAwMDAgbiAKMDAwMDA1NzU0NiAwMDAwMCBuIAowMDAwMDU3NTY1IDAwMDAwIG4gCjAwMDAw
NTc1OTkgMDAwMDAgbiAKMDAwMDA1NzY0MSAwMDAwMCBuIAowMDAwMDU3NjYwIDAwMDAwIG4gCnRy
YWlsZXIKPDwgL1NpemUgNDIgL1Jvb3QgMTUgMCBSIC9JbmZvIDEgMCBSIC9JRCBbIDxkZTY0MDEw
NDgwM2NmZWQ5MGEwY2Q0OTIxZGQyNTBkMz4KPGRlNjQwMTA0ODAzY2ZlZDkwYTBjZDQ5MjFkZDI1
MGQzPiBdID4+CnN0YXJ0eHJlZgo1Nzg1OAolJUVPRgo=

--Apple-Mail-13--118723366
Content-Transfer-Encoding: 7bit
Content-Type: text/html;
	charset=us-ascii

<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div></div></body></html>
--Apple-Mail-13--118723366
Content-Disposition: inline;
	filename=Powered-By-PMC.pdf
Content-Type: application/pdf;
	x-mac-hide-extension=yes;
	x-unix-mode=0644;
	name="Powered-By-PMC.pdf"
Content-Transfer-Encoding: base64

JVBERi0xLjMKJcTl8uXrp/Og0MTGCjQgMCBvYmoKPDwgL0xlbmd0aCA1IDAgUiAvRmlsdGVyIC9G
bGF0ZURlY29kZSA+PgpzdHJlYW0KeAHNXNuO1EYQffdX9Fs8Utb4fpGiaAMBJSHkAivxEPIAM7CQ
BNgwA1Hy9Sm7+1R7y7a2p+0do33wXLrbNXU5daq6vX+rX9XfqshVUedRTJc4jeJUJVkRNaX68FI9
Ve/UnXv7RG33Ku7+9luaEUdJWZT6E/0yqVUe09w4roPtW3X3QuVp9z1d0oy+qdKCRhR1lBe1unir
7jxIolgl6uKV+k2F9zbqLI5SFb7e0OJ0/dBdg/BN90Wuwr1qP6EX37bXQoXv22uiwo/dSBVe/oWh
z81InvKV/iYItzuHWeeYf4UXW0j1sl05U2G0CTopsZqWlmT4eqN+Vxc/qPsXpFlH3QZH6FZJ3SZl
nEdZ3mpf6za4rtsw3aiLP46Sh20d3GzroTxV1URVwuKwqYPO1GHuL466WZyB6yV1U0ZV1gzl0a4X
Fk7yBJ6hMFBPLxTIbkmSCHNRKDwy/gzng+/h/Q5O+MkMRGA83AQZ+YKNjHdiJXgpJpiwUOFb49a4
xWvj3ng/uKVTiFCoUCC/b9eiMMXNvULEuGQQO/jAQOe9ENE6J58MevAzJ0RInuAGOBzKwyHC4mg0
XD1EhDwmRMoTh0iZ1lFTlxSyUyHywPgrJwc41/4A3368USkFmA0GnSaCEEGx80BqRkYfs3PoZ01O
EFlmAhop9J+0v4tSHFIafhYifg8swK/EBAx8swkojdIK+JWsIQKLhO6qQsS0TI6UYa/fNggHtwVO
7M//lctBMmREXM+61Jv0EjlE20YGnrYYi+Vdk+hiqMwIAdMIVJ6DEHOSKMQRnnLqJJrWTRGVRaoG
8hiESE+LEFkaNxHRHZZHmmsGyfAxl43sVlF5Ugh79SL7gPDb/mm8X4by1Igg5HS9H4453yF4XyCa
tghfxBkC72WbkYkmHFzjbLFCwMaZVNQ6jmQd28izsiNZx5b6mUcNIirViIu1mcGDr/TcO6earayG
7v2zRnkV/oMaDM4Hp4Ob88gvTK571MVBwHmJqzh+gaUo53SMEktxPMDjdf44ZqkbeWxApZ6+K26C
VHvZ5w9FSVSjrHJK63FC5XMaVXU7rWmaml5R9fAqcKwG+xzj6Eq7F2DCVCbA3JBxucRqqG5Wa3Fk
fCVOiWMUf3yAmqtByCM8eU6ed6hMBtVpGqdlFNeVGshjzJU56Wcxc6VJXkV5mbM80l6FkzyL2StN
yzoi12F5pL3cSpNRebzs1cJyR4SMP7M8xl5u+lnMXr18MR5fM/TjE182U1Q1tV2ShvWj8xcRoQsq
HOJKNwYIIalAA+sHrD81gIuKgkuYg04EQYiRnAAOAGesBdBG6sB7Tk968V5VYhZX4fklFgFVYwE4
I2G5XuWiKdWKjEpqfGXAN+JIAHGrFEYD1schLeBL9RhCNSMBeQEIA76UZx0AsYA/Ya9TAwgDvtSP
sZdbQhz1Hy97MeBLeYy93PRzC4A/Ya+BP7f7OGVRRXFMxDSNaV+nVnUUZ02jeWlvg4e6q+3fzV32
QUezh/tEfbNqpKtNuF9Xbf9p/x9KBMA4AT41xHqtOnzxLETz6CfGYN6KAeQzKAOu9882qEa+FBXD
k7bype4YiVC0W0hID7ghJ5Q94VRXIzfmWpkrJfju89TU0Ge4FbIVpOCNJsh5dYm7vWvXoM443n/Y
BEkV1bY1xz/Jqc3WtFN7GUm325HEZmUkL8jFhlRWaV8QHODUzasEJYgUx0SwW0ZaLIJtqJQVxWGa
iQxJFOmb1j/IS2FCuBQqT/hN2w0u2rghVtO0E54b98ZADhr497Bd3O660kwMeNy1dck3sZSUYQdh
uFmMqRw7GGGn6qCDVHu8YPGc3Lzjimddf5t+Ms9FMH2W3eRxE8+JAK+IRAQYcWRAuhUti+VUy8mk
PCtFJHOyCXPN4Bw+5rKcTOpnJc7KnEzKsw4HSrE/mI3bixAUR0SANJyDAVZE25yOadwG7KdZVFbl
EPYhNIT9aGjCJYTmMzL45nAwMGh3NXkww/F2MNruDl6O9j8JrnvsBMwCYkGnzvv4i+nQNjfLUR06
nuUYBTKfQGVmYcSRuJr6N8scSPigmdjDVa0elsfE6Yl7v7a5OWEut9ppMXP1cFXoZ3VcFfIYe7nl
ncXCy26GTdhrhj/7hJdtbk7E14l5i2XuRdlEWTpyinHY3NztlCHlPxpWz8z1iruH4LAAesZu7mvu
dudbUxUGg4OPfDqLoPl6IQns5juBk5uqMDjmmOJCG1PjynPE7sWcnbHbiCPS8ZyNIB9ft9gt1WOw
aQZW+uQSi91SnpWwiTemJuw16Hu5n2j2sZfdmJL6MfZyy7Wjuc3LXtynlPKsk/stdk/Ya4Z+vOyF
E+hZkeRtCxSHpLW5qGS43zXy6FQMN98YqAHLzL9BhXkoQzPPAaQPT9mhe8OTeQ5DvD00C+7uD/F0
8ptOHpz27IHQsXFBt3R9CwjfWVwCvBu5Gg1QHwfsAfyYA4br4ft1cYyt3OB0MVtZaq5dRxrrxMyz
h+7XtbM6uF8XZ53A6mF7J4401gxX9oksS8vzMo7KLB1i+/ct9aamMTrTAGcA+3cd9vceKWJs3mvC
bI+UyalM1W/e/znXQyx/l3cZCgj+jrsid+Dk5hr8XWp5HSdk/m7EkU54Yn5h4V2qx0DGDATz4oPc
05byGHO5pb9bAPgJe50aNLinLfVj7OWWAEfpgZe9mJ/mcRmldc4gpu1FBPWX7oGWbpdPbw8eqNyn
Tm3vDNPgTDkDG3gkgIQfk2H8usIzX4Ohoz3i9s52Bxs4BWJMO9rd98494sUUaXvE44pcrc9gxJE4
dWK/7+GU8LOV/N7ilJBndZzS8kh7nTivWCYq3XmlvMJ7b1IexilsY2l0wvN3QXgFtkUqdNp7m8KE
o5+3tAyRqv+oyUaeX4DQYHEQHmD4AtQRP+Jui709xgjEO2AFTMBjPlgJK2MhBuAdZmIpzOB7e5xM
DfgQyaxzQJTSjta6ReJxrTsi8ZQXHP1ohN3ekPIYqEmdtusWo0S2jDLySKipneQZ1c+8Oiptqqio
R85uP9yo8QfM7QNpTDC4q4VPtMMH/IwruXsSq/BTd+Ko91wqH/CjyGkHgFMMAoaOPwVmhbpuyzuw
EAQSQnB4FG8sksyDs1yUYbWpvZMb/6XCYp5iI8lYRnrKqUkEzhONO8qcvRMf7mw5jZTHBLZbzl7M
XHbvZMJcxYkDm/dOpH5W4nxce0l5VuJ8vHfyedjLJgapH2MvOovM/9HlfzslxYAKZW5kc3RyZWFt
CmVuZG9iago1IDAgb2JqCjI0MjAKZW5kb2JqCjIgMCBvYmoKPDwgL1R5cGUgL1BhZ2UgL1BhcmVu
dCAzIDAgUiAvUmVzb3VyY2VzIDYgMCBSIC9Db250ZW50cyA0IDAgUiAvTWVkaWFCb3ggWzAgMCA2
MTIgNzkyXQo+PgplbmRvYmoKNiAwIG9iago8PCAvUHJvY1NldCBbIC9QREYgL1RleHQgXSAvQ29s
b3JTcGFjZSA8PCAvQ3MxIDcgMCBSID4+IC9Gb250IDw8IC9GMS4wIDggMCBSCj4+ID4+CmVuZG9i
ago5IDAgb2JqCjw8IC9MZW5ndGggMTAgMCBSIC9OIDMgL0FsdGVybmF0ZSAvRGV2aWNlUkdCIC9G
aWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4AZ2Wd1RT2RaHz703vdASIiAl9Bp6CSDSO0gV
BFGJSYBQAoaEJnZEBUYUESlWZFTAAUeHImNFFAuDgmLXCfIQUMbBUURF5d2MawnvrTXz3pr9x1nf
2ee319ln733XugBQ/IIEwnRYAYA0oVgU7uvBXBITy8T3AhgQAQ5YAcDhZmYER/hEAtT8vT2ZmahI
xrP27i6AZLvbLL9QJnPW/3+RIjdDJAYACkXVNjx+JhflApRTs8UZMv8EyvSVKTKGMTIWoQmirCLj
xK9s9qfmK7vJmJcm5KEaWc4ZvDSejLtQ3pol4aOMBKFcmCXgZ6N8B2W9VEmaAOX3KNPT+JxMADAU
mV/M5yahbIkyRRQZ7onyAgAIlMQ5vHIOi/k5aJ4AeKZn5IoEiUliphHXmGnl6Mhm+vGzU/liMSuU
w03hiHhMz/S0DI4wF4Cvb5ZFASVZbZloke2tHO3tWdbmaPm/2d8eflP9Pch6+1XxJuzPnkGMnlnf
bOysL70WAPYkWpsds76VVQC0bQZA5eGsT+8gAPIFALTenPMehmxeksTiDCcLi+zsbHMBn2suK+g3
+5+Cb8q/hjn3mcvu+1Y7phc/gSNJFTNlReWmp6ZLRMzMDA6Xz2T99xD/48A5ac3Jwyycn8AX8YXo
VVHolAmEiWi7hTyBWJAuZAqEf9Xhfxg2JwcZfp1rFGh1XwB9hTlQuEkHyG89AEMjAyRuP3oCfetb
EDEKyL68aK2Rr3OPMnr+5/ofC1yKbuFMQSJT5vYMj2RyJaIsGaPfhGzBAhKQB3SgCjSBLjACLGAN
HIAzcAPeIACEgEgQA5YDLkgCaUAEskE+2AAKQTHYAXaDanAA1IF60AROgjZwBlwEV8ANcAsMgEdA
CobBSzAB3oFpCILwEBWiQaqQFqQPmULWEBtaCHlDQVA4FAPFQ4mQEJJA+dAmqBgqg6qhQ1A99CN0
GroIXYP6oAfQIDQG/QF9hBGYAtNhDdgAtoDZsDscCEfCy+BEeBWcBxfA2+FKuBY+DrfCF+Eb8AAs
hV/CkwhAyAgD0UZYCBvxREKQWCQBESFrkSKkAqlFmpAOpBu5jUiRceQDBoehYZgYFsYZ44dZjOFi
VmHWYkow1ZhjmFZMF+Y2ZhAzgfmCpWLVsaZYJ6w/dgk2EZuNLcRWYI9gW7CXsQPYYew7HA7HwBni
HHB+uBhcMm41rgS3D9eMu4Drww3hJvF4vCreFO+CD8Fz8GJ8Ib4Kfxx/Ht+PH8a/J5AJWgRrgg8h
liAkbCRUEBoI5wj9hBHCNFGBqE90IoYQecRcYimxjthBvEkcJk6TFEmGJBdSJCmZtIFUSWoiXSY9
Jr0hk8k6ZEdyGFlAXk+uJJ8gXyUPkj9QlCgmFE9KHEVC2U45SrlAeUB5Q6VSDahu1FiqmLqdWk+9
RH1KfS9HkzOX85fjya2Tq5FrleuXeyVPlNeXd5dfLp8nXyF/Sv6m/LgCUcFAwVOBo7BWoUbhtMI9
hUlFmqKVYohimmKJYoPiNcVRJbySgZK3Ek+pQOmw0iWlIRpC06V50ri0TbQ62mXaMB1HN6T705Pp
xfQf6L30CWUlZVvlKOUc5Rrls8pSBsIwYPgzUhmljJOMu4yP8zTmuc/jz9s2r2le/7wplfkqbip8
lSKVZpUBlY+qTFVv1RTVnaptqk/UMGomamFq2Wr71S6rjc+nz3eez51fNP/k/IfqsLqJerj6avXD
6j3qkxqaGr4aGRpVGpc0xjUZmm6ayZrlmuc0x7RoWgu1BFrlWue1XjCVme7MVGYls4s5oa2u7act
0T6k3as9rWOos1hno06zzhNdki5bN0G3XLdTd0JPSy9YL1+vUe+hPlGfrZ+kv0e/W3/KwNAg2mCL
QZvBqKGKob9hnmGj4WMjqpGr0SqjWqM7xjhjtnGK8T7jWyawiZ1JkkmNyU1T2NTeVGC6z7TPDGvm
aCY0qzW7x6Kw3FlZrEbWoDnDPMh8o3mb+SsLPYtYi50W3RZfLO0sUy3rLB9ZKVkFWG206rD6w9rE
mmtdY33HhmrjY7POpt3mta2pLd92v+19O5pdsN0Wu067z/YO9iL7JvsxBz2HeIe9DvfYdHYou4R9
1RHr6OG4zvGM4wcneyex00mn351ZzinODc6jCwwX8BfULRhy0XHhuBxykS5kLoxfeHCh1FXbleNa
6/rMTdeN53bEbcTd2D3Z/bj7Kw9LD5FHi8eUp5PnGs8LXoiXr1eRV6+3kvdi72rvpz46Pok+jT4T
vna+q30v+GH9Av12+t3z1/Dn+tf7TwQ4BKwJ6AqkBEYEVgc+CzIJEgV1BMPBAcG7gh8v0l8kXNQW
AkL8Q3aFPAk1DF0V+nMYLiw0rCbsebhVeH54dwQtYkVEQ8S7SI/I0shHi40WSxZ3RslHxUXVR01F
e0WXRUuXWCxZs+RGjFqMIKY9Fh8bFXskdnKp99LdS4fj7OIK4+4uM1yWs+zacrXlqcvPrpBfwVlx
Kh4bHx3fEP+JE8Kp5Uyu9F+5d+UE15O7h/uS58Yr543xXfhl/JEEl4SyhNFEl8RdiWNJrkkVSeMC
T0G14HWyX/KB5KmUkJSjKTOp0anNaYS0+LTTQiVhirArXTM9J70vwzSjMEO6ymnV7lUTokDRkUwo
c1lmu5iO/kz1SIwkmyWDWQuzarLeZ0dln8pRzBHm9OSa5G7LHcnzyft+NWY1d3Vnvnb+hvzBNe5r
Dq2F1q5c27lOd13BuuH1vuuPbSBtSNnwy0bLjWUb326K3tRRoFGwvmBos+/mxkK5QlHhvS3OWw5s
xWwVbO3dZrOtatuXIl7R9WLL4oriTyXckuvfWX1X+d3M9oTtvaX2pft34HYId9zd6brzWJliWV7Z
0K7gXa3lzPKi8re7V+y+VmFbcWAPaY9kj7QyqLK9Sq9qR9Wn6qTqgRqPmua96nu37Z3ax9vXv99t
f9MBjQPFBz4eFBy8f8j3UGutQW3FYdzhrMPP66Lqur9nf19/RO1I8ZHPR4VHpcfCj3XVO9TXN6g3
lDbCjZLGseNxx2/94PVDexOr6VAzo7n4BDghOfHix/gf754MPNl5in2q6Sf9n/a20FqKWqHW3NaJ
tqQ2aXtMe9/pgNOdHc4dLT+b/3z0jPaZmrPKZ0vPkc4VnJs5n3d+8kLGhfGLiReHOld0Prq05NKd
rrCu3suBl69e8blyqdu9+/xVl6tnrjldO32dfb3thv2N1h67npZf7H5p6bXvbb3pcLP9luOtjr4F
fef6Xfsv3va6feWO/50bA4sG+u4uvnv/Xtw96X3e/dEHqQ9eP8x6OP1o/WPs46InCk8qnqo/rf3V
+Ndmqb307KDXYM+ziGePhrhDL/+V+a9PwwXPqc8rRrRG6ketR8+M+YzderH0xfDLjJfT44W/Kf62
95XRq59+d/u9Z2LJxPBr0euZP0reqL45+tb2bedk6OTTd2nvpqeK3qu+P/aB/aH7Y/THkensT/hP
lZ+NP3d8CfzyeCZtZubf94Tz+wplbmRzdHJlYW0KZW5kb2JqCjEwIDAgb2JqCjI2MTIKZW5kb2Jq
CjcgMCBvYmoKWyAvSUNDQmFzZWQgOSAwIFIgXQplbmRvYmoKMyAwIG9iago8PCAvVHlwZSAvUGFn
ZXMgL01lZGlhQm94IFswIDAgNjEyIDc5Ml0gL0NvdW50IDEgL0tpZHMgWyAyIDAgUiBdID4+CmVu
ZG9iagoxMSAwIG9iago8PCAvVHlwZSAvQ2F0YWxvZyAvUGFnZXMgMyAwIFIgPj4KZW5kb2JqCjEy
IDAgb2JqCjw8IC9MZW5ndGggMTMgMCBSIC9MZW5ndGgxIDE5MTY4IC9GaWx0ZXIgL0ZsYXRlRGVj
b2RlID4+CnN0cmVhbQp4Aa18CXhU1dnwOecuc+/sM5mZJJNlZjLZJ2FCMpMQCMwlC2QBE5aEBBiS
QIAQQQIEZKvBhUVUQLSudd9a/VvCIoKmlYKlqNVKbbXauqNVK0qVUguZyf+ec2dCbO33fP/3/QP3
znvues67b5O+1WsXIz3agjikLFrR2YvYR06Cr1cXretzq2N+NkL4riW9S1eoY+ELhMj4pcs3LFHH
2i6EQr/oXtwJ3+wzBPvSbjigDnEAvjO7V/StV8dSLnyfW75yUey8/CmMN67oXB97P/ozjN1Xda5Y
rF5ff4yOe1eu6YuNM+H7R72rF8eux60wH/rOUR8jTBmGFWgp0qArkYAIMiM/aoErtfgUrBez8wLZ
fQD9nzfaTRV/RykSu//Ja6rzKHD8xFOfDukit+luk+iTZHiC+oGRpIvOREhfMKQb1uhuY0+KnWRf
FUfR7OFjSvqBvOJS8wH3AeVA04HeA1sOPHhg4MBrB94/oD124NwBcgQu6X06ManUVY1NLa4W0tjc
3kxWzsYPzN43m8yYlcjPnOXgZ8208/V1M/kpdWX81Lpivha2umA5XxEq5ieGJvKTQh6+KpTGV4Zm
8pNhU2ALBYv54pIuviQY4IOB2XwgmM6/Fng/cC7AHRn+8uChrNrSI8PvHzxk9sL3l4rhkGwqPeSs
5dcd3HYQpnXu4EF2xUVl+KCcWXrQVsvfuCOB713eu56YfvTe/US5z5FcqvzIkVKq3JkI0B2JKaXb
tia4TDeYtpp2mXab9rhucO1y7fbv2rJ1y47dt+7Zumf7nh0m5TrZXGpa7VpNlFWyvtS0ArtPYfev
cejkVyeJ+1fKrwhaiNFC80KidD7YSUzzcKHNwhfYsnifrZzPtyXweTY777Kl8x53Fe+2VfAvOmt4
Z8pUPsVZwTttxbwdrkuA6VptTt4CW68NK7bJVaUmY74LidhwosGlP97g0h5rcMmwCYMNLv7nDS7u
aIOLPNvgwocbXOiZBteJ4/muY8/nu36utAx6XM8e9bieOexxHT/xguH5Y780DP78F/qjzz6nP/zM
Eb15cMsgUY5uOUpMh0OHGw/3H+ZNh/0ArgTw+cO/PTx8WNLKZbzeQASecIRgRJoEfAQP4wFrA2qY
XTmQgOF7VuV+udjXMNA1s3LrLbekDdzRMLN1YEta2xEJrmkdwAN4V9uA1DArBiIf/azpW7OGAf+y
G+BqBsSa7s4B0Vu9hg6MdGD0VgMwYKKwyVvtwwO2mu4BG0D/9pA18Q+cUk+qL2IwWvsvr2NDOpc+
mJHPJ6aLNuGccJrfzIe5d0Dq0PBfhj+Iro92Rdu425ELJOQO9CQ6ik6iV0eEZhAdZ/A6dAAdQy+P
HKfAteh29Dj6DXobfTVy/C50P3oKDYyMKbAH0aOPop+gn6GD6Fl0Ao7tQLfC0cfQ/xl15Uq0He1G
96IH0es4LXb8BLFhdQafIT05jdfgXciJClA1mo/WoGvQNpjXKTwNjk2EY01wdDVaj/bC0aPo1Khn
x8GJoGnCqAddhfbDFb9kh/Ph3tmoC47SY+pnFdqIbkQPoSfQc2glwNthvvfEHzLq+1riIR7Uhz6G
O1/CPyQnYUVPoK2iDWkREk5TrPJhhls0/AFC0a7hvyPELSTnycPkVrSP9KBpir15dllpsX9MYUF2
lsuWYLUY9MCTBe4BLqvGW+Pt7N7prul27/RWd1QXFgD/1VSneDxthQVuOFztHsAd7pqBKeu6k3bW
0AsGrL4BklVDt54B5aYOALzVHo8HziRcPgM67uZRp9zLBpTOAXSTe3/BsZ03HzGjhR0+fZe3q3N+
6wDXCe/aj2Ay3cDx8EW3jm73AA8PZrsUOBKbIj3X3QF7bzXc9b3H4bBc1brdcyxlwArfNQMW38BU
eNLUjWdSuJ01ScvcdLhz53b3wIMzWkef9dBr2trakr6DhineKR07d07xuqfs7NjZeWR4y0Kv2+zd
ub+hYWdvTYd7ADWBcMLxZ29KGZhyc9uAuaMbjwfs0XVMmdkaSvFYYKoeD13vTUcUtBAGA1tmtKpj
N1qYcgApfl/bAOmgZ47Fz9ib6Zkt8TMjt3d4Ga6rWrkUAg9umOVtmDG31V2zsyNGt9iRcSoV9xNU
ud+Ld8zYr+Ads+a2HgXBdO+Y3XqAYFLVUdlG0UiqZreOvgvupBwA9g5MHHkNbGc2epR7GD0N223k
M3QbfN8N3xHBgR2CAz0E2xOwTYftOdhuhO0e2DbCtkq8GUXFm7EerOyAsB45hc3olHAHWiPmw7cJ
neLvRqdEDp3iFsB9N6FX+D4Y/wLODcH3NLSGfx3gPbCtR9v4LxHSfIbm86+ibn4IHeV9aBl8L+df
QD30WzCjo6QY7eMH0S/h+znNKXSUHuP/xM4f5XrYPVdxOWgSHN8P102E+bXQbz4Akku9B2r1wcgj
ETQS4ArNix2hR+mHgBfBM0iAazQMksBP0CId3GVARmSi2g8+FtissCUgG7IjB0pESSgZxk6UAvtU
lIbSQXbdyAOjDORF1L9BKIvtv7vLRjkoF+WhfDjsA/0EtEFjwK8pQmNRMSqBUQAFUSkqQ+NQOYzG
owmwpx8X/BuHbkaf4QQ8Fk/HPyMzOZG7nffwx/mI8Ka4UjwkXtS8J4XlMnm/tkr7iE6j2617U3+V
/ohhozHbeNLUbU4xX7BUWV60vpyw05Zse9Xe7ahPlBOvShKSmpLXOsc430y5LzU1zZp2Z3pd+mOu
kOs+13vwZrB5QC1AEeBLg2zPiIRHdPO/8s4rbDe2yGPxWLJgh+Gqi1sEdIl+IwAoHR7lTuPrQNNx
yK5oCeGwScCogPOfDaNQiX9sEea8HL7Ot8Mn2iI/J1Vwx9PRLs4Ed9hRi1IlY1mTjJM1uVyu0Ihr
uVqhUdOO2zUr8UpNP15P1ov9GqsGY/1GHktFlMYmPZjrZpNelmHv4nc4zOfP+nwl/nAFvDEELwxj
bzaxmK1lJXZR1IjEbrMmOhyJnOnj/S+8sP/jGbeFKhrqJlXcMz3a9TJ+HxfCv/df1tY9378p+sdH
n4qe2bLp1zV0ZbdFu8hZNs8epVzkxAQ7Z0/IxtlcdkK2fSpWOCVhqr2Ja0ro4DoSNqB1pJfrTVhn
s1sxr1+LsDXEY57XHRk+f4hOmAKKiU5a50J6k4k0o72J5gu+f527mWi8wdLSslJrMEBysrNzgiUO
KzkLE59+74RJdfUTQ7fNgIWQiujrUffL2ppfb9qCU596FOdu6n++Tvty1M1mvouEwHpyKKhkO3E+
9pEgKic1qBbe2ka6wFT9itMDsVpAQnjiJIT4wyV+ZD5fTCkWlrE3gYSin+z9GU6LrCK7KTbuJmM5
mXwCz3QrNlxpAsVkEhpRo9CO2gVgBeIHEiB/+Cw8wBP0cHJkD+klY5+m90Zg9wWbj/swacESKsRH
hj9VtBQLfhzCBPvDvrMoBPd6vJYS/MVXX8HVGDtgdhXCGyCpO5SgMEUU9ZyRq8WSyeKyEIG4TNhk
0hv1ej1pNhr0erHZ6CYhbiXXy3Gc3mwmzeBQv6/o6Gs4B0U9jM8raTodQOn0Lk40AGk4s8Egwp4+
gfPH/adwCZCmvNgf9pcAY0WKgZvp/GBtFk+wGAhUWlZi8fAVQ2/j0uhLoT1ZY4L8vbjoLu6THXZb
8vTJF4/Dyh+CFdwqnENu9IEyo8nV4SICJ1ocnN2SaZkgjDMEjaG0UHq5q0GoNdQYG9Ma0+tc7VyY
Dwvz5BZLe/KClHBqe1p7eg/XJS62LLSvTO8lfZZ+Z39qf3oWrObTQ3TSEBV8qoQohExmU6HkTy0y
KSbRpNA1wh5WZzLp6hMIcdVjyUUkj4NxoIMJkYOnCHFQ1CTTGxwO+iSHw31/hinDlUEAkXd7zBcA
E3TH0HPWWs5QcrYExuGxRbDDYUAN5dvSYCDbmyEyLi6hTj/IIPvv4W8dMi97fd6xXXffOO/3i7VT
z678GPO+/JxlDVeeWcR5Ts891Pbsn/r7rlcqf+cd/87Pm2+rnLS+btmvZgMenxj+gN8MeJyIjii7
dDrB79TZ/Xm6bH9eRYUuaBubEfDX62psVRlV/hbcJrTpmv09uiX+nor1unX+vuCmCmdgfPV4MmE8
4BcXWgpJYWFevUseS0wGl4EYDJZ6Wev1lDFWKuMpU5SJFAtl6WMcHm5M+ngIEjgnYxm4BtjkwZAp
5AoR/T2TzJ+EzZ/4fJbEcvNZv5/ih3IyCEPIWk6//JHyckBRGDscKi68GUyoHSWMiQBbIOVebxCG
MXgEeQ7QW/Qeu8PBG4sm1Vc1vLxh87nppuZPrgztKhhTWFJYuKV+7pS7nh6T51s4qf2NdorTFY9X
1dbvu7poM3nFd93SJU+GplRN8J4eV5+fV9Azo2lZuivx8f6NpTOcTlv1pNPeCbkFRTvmbT6aZJRK
wGZOB349CB6rFizk+4qHN9gMmYaAodrQaxD1SXTlekMtkFMvamRDHRZolAxMIzYLAqfhOCmkbdQS
kGuXiWj0PNN9wFEAfKOAnIrNvFsrG8RGEYMCOKPoKONhLUUrjC8eongH4EulmF6LsUl0iSHQvNlU
lkVC5VhMonIs6umdQCG4U2TTEmNyC2qMcmjYWl7C5Lc8DFopBKTw+yoixdbyclBP280R/piP2gmL
F2QZl1hKPBbMH3znWKSMnD7yTnRR5Hn8SDSMH/mEqx1aTR6MdFAt9hzw4FbATT66U5molZyST5oo
BS0THQ1StWWuNDuvR9oo6dPSnHWmLFeWksVleeqzxHRi0roAJ1pjvah1Z7gb03DakeEvlDF0BWkg
arA30oWnMf2TZvMgd5qM2An0QIGpwFVA5Ht9KpdZyimTsQWe9V/mMn84QhV3GDNZ+17+Ap6yeOwe
S2lpSTFlK37rtMraX1+/8cMrjDP/3DN1a6CgMOgP/HB+6yMTuC2Ryb65ng2HpzW14re6fzF5SkNJ
5uuButxi3/rG6T3ubFeSngzvi/bxfF6g7GfANTcCZp4QzoJ3VIZ+qIwTDQ5DeVbJ2JKyuqzKsVVl
7bjF0ORu8iz2rB1rdHJ5dWkJCYn1aZyJBOs4rbPAb/V6kFVGQPlDzDBSVcR4AzEKw4kvFBPFEfpR
uancVU78HhmuOUwvlu8aR60ok0GQQoofoDLgBmgOWPGDqmKYQRQt2SQYsJaVZlIc2L0gdhlIE8eI
RhVEQJQo2m0URyCR/BPRP7x11aGalnBzuBU7jk5oytOmrprw5jCyz37kyvZbp7W2vVwWGtM7sXnv
dEIml4+5MnTr4/ijj6IfVFfNwtZfnsTFV6/q1xqeN6VEv/m4JOgNTnz2lvDGQrctN9+R57r/mWBB
3n7gLYgq+R8Cb4moUSmWsZZk4lrcQFrJBhArcHbdYKzBxnL1Akckk+SSNhGOQ4TwJrgZ8dT2Aotb
yymvRyzl1IpvN5/dfgyYHHupPeZ/GFnwBnllaIC7xH99yShk7ANLvnH4Xf5Hwjfg7eaiMiwdRdng
rBgAqVlUcimQGQe8cSADAGUdPVfoC9hLMgI5gZJq++SM6pyakib7vOS5KXNdszPafW0F7WNnl8wu
65AWGhdaFyZ3eDty1hnXWTcVbLOmieQn2Y/7SbZD6+e5tKlmEqwFRnCjBJyQgPxaQ54HObLdMSG4
T6W522Ogr2bW3GAo9oh3AuGpAQLanwHCA+VhZynxr6L6N3TWmlgehohJSW8ruLGA5BUUc0F/nr8U
YukWb5f37mzR6fZy2WkWeh3btQGfUNHyYRswS2YwUFoWzM4OBoBbmBUDLcwx06ZyR2JpaQJjmRzG
L5RZfhR9/czX0Q/2XL9+Dbb94T2svWbjzbeffWzLNQ/NmJl1U+Wiaa4Z6/y94bkrnt29dx9+4JfD
6OKJzS9OEJW7Vv/4/TceW3yiTKwYII1X9q9fUrsszzo+oXJXZM38leMc2Rljf9yzfeAOkLVVwx/x
u4WvmKzdoJRLfDKfx1dkVfiCY6ZlTfNVjWnl2xPDSTNTevGmLJMlrbjOlldnE9NiGihokUHYZCfI
1HnFy6QN9A5I1v0qlgs8TjPVsE6eHnXeQYUrJl1MtqhogVhR2zYiWkQj8peVjrVMFJmsAeoQUzbW
EdEakauy0lJ+d9vcedEvjwbmZ2rTeia/c8kWfrRz/g8bWttwwR+XH6lpnv+SMs6/PLTniVKlcHnl
FQ9OwRxXeSJ6vHf1Zp0eBArLn40rygxMHLz+DE6vqpoVvfTovYOBwpxDj7SvL3TZ83PteSAcUYh3
t/Avg2S1H1oj4AJY+IdKtjapjEunOxeEZ4SjUqRIuEhSJMJD/IIU3MS80hKqTMLUry0Jg94dWwRJ
PkUn8LM5wonUMQFbAv+FY204nIWD2I6xsOXSNfx1Q6u5XdMJ3kzwoeja6FqwH1gPM9nFZtJ5aDOH
C8DSfajkwiQEOhPBhf2YCCaMcXwmHJ2JHxdhBXPgIX//XDh+tiCIZDb+zlwwzAX+C7suXcPtGlrN
X0euiAwfwjfhmw5FhoGLBobTNVcAF1WiRvSG8jDYVl4naGVTijHVFDIpDuLiUwVXiivVlmHLgVhx
chYp4AsEf4o/NTPDneMP+SdPVaprW2rTtIKQ01Z3lbzYsMzZ7VmcsyS0ZHKfY1NKb05fed8Ek1Ww
SNapM402xZ5SZuP56bOkwsL8GUZp4tj0GWMnEhOEX4Klylpoq7eGdFhnmuGeQZLybLoyE3KjflBS
KDOv7MomJuqg48rPgkE/6y+JFIM/ChEm0/agBfywveKDPVzCTCEYQ6LxgGwCL4LSp4wYBENP3VMI
qzK9GXw8NOSpIwahIsj8KAPBJzqs2EJNBLUGcInmihkN0dqU+h2tP37xwlN1vZMf+CbfN6+1NTr0
2IPRf7R3rOhuX4S197U8M7vzx23PRo+vXrNlW18fnvT0CzjQ07MqsjvUVX7t3r5NVdvInTdFh67s
q1CiZz7CRo+naOhwwwdtj2J9R8fSvoULo1/d81j0q87FSx1Ju+ym/tVrcOWJozi0du22zb290V9F
FSKmJR96/JEnJoFFgCwFEhaA5dCAz/bCQUWmPtm3LCoQ44CGqWzqUtWLNRoiy1qJbAc+s8FNspbb
DnGqDXhos7hGQ7iAVqHellah9rZIq2h7tZxW1ooc3ihgQTLpMbh+nKCHNEg5sM9cSJj2wXPQVXo4
pRV8QqkwXWgWFgubBI3QpQO/BcIGsE6gZVEoXBEC8oCWpVFiGAQnfOzYMfVLAlsFuheFPV7Ow4HN
SgBRWvD63sjmvS+SdCxtjl6KXsQPRDuF00PryZ8iWcAakNuFF55GdphMCTYoVkFv1+fom0mLvT9Z
tFoKAunUQbVRNZeerkkLSFxhQCM57NYCE7Ml1Aczuammg/HXSipdsymbRoP0KI2aNFl2xEJJ0Buf
xL0TGnWByoRbmMsKwBkAIPqicRiLntCqoCmoBEl6gU1jpPgEGgwdoq8D4G3ViGkkqmhh/CV7HACf
s8cB8Df2OAocptPQLA9QNax+IsWAUBUE30Z1e0EbqzYPxmeZjxv3BSFYZf5eLIzAsWgDjtJQTTVg
XnoRHQq+2VNnv3hv5Bw++sjD9TPrl8+982fRg5m5/m2LvsAofJXfn9NfOrXoxoXRF7F43ePBcQH8
0sonyyrHCaeTsn3bF/T8sFByvUz40vrEFEN0ZkJ6envknrk9WcmmyBspmTld1IteM/yxMAVqlPlo
pdIiYIMs2hw4RbbZs+yl9irbPKlV22qcZ56X28F12nrJOlOvLcHhcAasJD8/OyBqIVcJTjGmfrG/
IFSwskBw2/U2ill9KsWnvtunmizAUgVzkWkwRtkti4Wh35XyuBv4HaevrESYUtZWO3F3y8PRfyzs
WN69sB0bHl3/1V7Tpq93rnp6as305qopz3XvvrjCuDwpPzEhZV5nO846fgRndHUuGV/316UL6qY3
fHzHfR9OrZ+6cCHIKOXTA8CnRsglDirecmuddRnpNvAOYMhEYMh1CJvsSEv5BIl0KcBIH8aZ7fwh
uiw48nmc6yBOZ9zW5zK5/C4Fsgp84v8fNkuPcxaz+XEuA8EFf4qikOpUUJSUX6h+jNl1SNmo7HNg
/21LLr0S3YH73sa47a4nf7txQ+vJnc8+u/sPbStXkr+8HD08LwS8Eiprj77wxr5zNcU5l67PL5/6
KeULwBF/H+BIh/Y8IweRaBYJKLBjSiYVXlHEQpBw2iCWeCRBAmmNwWTAomzDTLjAkqrCBUBMuDAT
LhpMMqwBoAoXAJ8x4aIAEy68XH951Uy4KsKUccCDBGlaxXI+LHSi4ZOHv2/Ix/1h6G+ciW7C6YFo
90Dkzdj8+2H+Mrr3AMyVTt1Op06IBgclTiMhrlFHnfYjw79XnIx+XTqTDkw+piIu/I8I+H5cT3wR
0xPaUUvxgaI472NmUl0LJSEsAZYBYS/fHzGR7ZENJ7lnBE90/kCkBCavrkOYwOjw4QEuKNF1OJib
KJklIkmCFuIRQZKJDbzxY0xfwYouHmK8CzqV6Tg48omq4xAjA2VeJZMFcFamOSGYA22ZQBeO1gMp
FUOTgZM4G9iu37FHAaBSFIAYReGlIAcwVgkJgEpaBtDnUYChQfg3isZ1py9SjEIVoQqg7CpqgyDb
SJEBeTxLiTDhZCT55Enyl5PkrUiOcDpyhNQCPm4EI/sHho/Filfmi0VOyxVjybBCK+nmam2cQObG
co0sJwE5uXeZ5AJwhkkuAFHGgwB8eZgigVsxwnHni80R2D5h2bRIMSWQB9wUb9Bjh2mRP0QOnThB
pp04cRf/0F13XWqH+bwCumQtm88/lf5kuQXPB2ss58rj5Hq5W94p/1HWmLBWTsfJBPK+cjkul4O6
Olwn1+jm48W61WiDZIZgcgc+BQ7uQRA3ST5IdGDjb9BiicTIC4yq1ZoMbqhcKIhvgtdeBTTCEsdo
AHRiYgcI/49EGqHNCLWorWRE+vp7iER59UI4XKzG7oxCNFMDLsKxYxsjSfwxCGc3RsJJ1EtYtdrj
wRpGNlyChbXR4cjB24Fwr5+LLCV33h/VgIfwD8g/V6vczK8DbAlo7WHC8aA/gBc/VVTOQxqTBlyf
/5UO+TymQ8TLgnc2DEGoj2oQmi6mzMWvG5p5knwqnL74njon4QLMSY/rlHktWjyOjBNKtStJB7dS
6ND2QxmhX+jV6prlFu1cHdfF9XFrgcRawskiQYRn7hmvUGHgmZOGeDNfzc/m4aPRyRwGv0KrA74E
sVQMTAXZUHpcPpXpzL6oBQgW6SGWf0OgksDKJDHBZClOtN5oMrqMTUZw9pjksVyMAHEhUDFBY/5/
92ooK4AzExdTzXLDKKSBiI74NWB9aD41LqrUL4RI/lCA7+FJuA3AA108DreBT0l5ZDUKrwZBhjwH
1WsYe4QLJ6ML10UXH8VGfAveghMEbuhObtnFCDDGCW5iXMuBJUIy9imKXuPSBDQ1mhmaTs0qjWad
iE2YiC5sFwNitThLvBJ3iP24V9TpMS+SuRhSkaDUJXCJeUnEBBb0LZN0AKDWA54hABdVhZwA7Hb+
35Tjh0qNqlKZZVdVInidgH89wz9QhCpGMA+KjhAmc4Rhn7CkNEngGfYhrarKHwAxJcmzi2Gsyh8A
qvwBoMofPcWmxi8fZSug0jEa+xEYQi4lhn+K5NWrIAgG11xFMSjLcX+PTDqKS8gNR4XAxd8Ipy8p
/DHwdNYMfyC8DXn6RKiT/kLJ4BEPWNNZE1GimKxPts7Bc4RZmnZdq6HV0p4wK9Fsh34mJYmyksyW
tE7eYCcpATvxBGRtEqyPzTXJDrpTXSoAn8TV6odxtXpOCTC9uibLlIVp8jWUxaUzRk23m6yU4U2s
/GMC9AHMzpi6M1XuO+ujEQqsGJiOmX1gN8pW2GG1q+7Od+JCR4IZskPIYobcBhLS53Uuapt/6aH7
osNz53Z2zG/Fwj0PDk+NDn3wUTSCpXffxRohuyv67pEj0Xc6Fy/pXrQIu48exp6lC7uXRTpxBp4A
Ad270T+BQ11GoznqBd0BfGmGevCflaLxtolpDbaGtCbjbNNikyY5gDRmDdFo5KSAlpMlk8flIRa7
qqZ7odw9ygB/q+gYd8UzqOfitvpTJoXMJDsYn630mDwhD0nW2GSm1iG1quIagBhbyYytYKyyFQAq
WwGgPg6Ajxmp5OXuy0JNvanzFLPAXbQezNwQViGMhycsA88ClH/xJfk7aiZN/92DJ0/i27c9W9sc
frW0rGjTgheeWH8HBCG8adFPJk2fHgG7XFhU/uT26aszXSmRn/r8RT0xb/uuGA4vKbMknAYV3vG4
PK3GVGurTZuLW0xttpV4GenQLtZdg9fqLNQQmpFZ4wwQplHpXmxWoMRJhKQAc3goqhUPZ7GD3wme
z7dKGsWuIZUylIFJrYHVIQwGsxt0PHV4HEyWkzmZYpmzof+J0VTjP2EUUosj4NL5whBGj6C1QsUr
MDDNdTNLmQR2EqwkdW4ux3+xaG/EXb8rOhw1Rj87iR/afqh2xryHd3UWBnzrmj47teDmsYU+0hQZ
EE57C0vuvfqht8rwI8qijLTEyKuewvwVVINugziOQBxXBAyN/DElVxjXdmMAUPZSuU5iPJXI9g62
t+vpcRsLd8BeuZA3RbK58qTcpExXpr9cKjWPSwi6SvPrpRpzXUKNqz6nOr+VNKc0u5oLr0xekrLY
tcTX4d/k6HX1uvvy+wq3Wb2yYjSXSXQHJtLizOXTRI8nK8CSnhAzenLtTqYAnFTdALUgs2mxI4+T
1hxUdQLAOQVaC0AH9xWbinuLidwzNl7ai1VdYiUXGk/ShDFNLttbLXNyuy1LczdY1uXeaNmWe6fl
7lxtuI2VRBnjwy5elsmktXyabVLrfFD2U5PLtBQBCalY1QEKDwKZUdf0hzseig5vNa7CudcdeaVz
UcO+hSefxxXf3IfFxcbm6F9vfeCXHRuUL2Y+/mP8kzlPTlBqKyZ8u2DJzjWLFjhtTlv+y48891VF
wee17Td0h3tSjbn2ggOUavDhvwTZ0KA1SjLmgyLU7EyyS26UOTQPQ2qINGMb2IgLipZiiZ/XCEkf
ysw6SjNBorqUjpkKBuBrpoLZEXo9AMNwBBwENS/hC5/3RXxnqOirzi3lRyhGfBn54mTkC5iJ5+J7
gmeAzmw+dKHcAjPToxeVJ6dyS7kNHGfAOsLzRBAkvS4RJ3NJQrKUrMvj8qQ83QRSzhXzAalCLtGO
1zWQar5amiZXaRt0zXguzHyuMEfTJjdrF+MespjvEXrkxdS34tdIm+XV2s26MXobvFVjEwXAAeaY
PyWzPSSWZAJLBcGFGBTqLRNQQGxA1eJGtFYU0WpwikLGdmO/kReXGsxfQuWJro8muEDNMfmjKWEM
9pLWXuB/Cab/NbdEf/Be9NfRV9+OrnsZl+MAqHxcBj6hn//9pQKwoPn8G5fS+Q9Bg3VDPS2NX4+y
Iau1TGnTa3lvstbu5X3gKojNBWxfyPZtxhnp8wuWGTvSVhZu0m609aZtKtASKXdikUWxEIvFLTWm
4tTUpJCbHztZAg/fBOVHS06QLRQ8RDW5wADK+xRQE1jEidJ0EIUD1RGzoFRSFAtzW1gJNuYwUlvB
pCbWaQF+C5sYXH5escsyjNVg71ZIiLmCoSAHquFbZouYjgAHVWweI9F1jUmBHpmLShl9JzgNwGk6
iTKVjuV1dExsdcyC60CYxWYKwx4YFfZbR2XIaE8NuDNQlI9/oHRBfRqfmjSmsT0UzCwQBYbV8iCc
An3pDcYyRCCXkA7LLPveejxnYVepSWE+7bnk5lz/xpl3/m7F4iU4/dHC/NzeifWHO7Vlry1et08J
VT7X8ln1jK6+qxc9erVlojXRdere/vsKC91SmjI7KdGck/W8KTPHP2bv8mgaMIMtIbGzuaNzOvDA
UeCBPdB9lIDc2KrkBUjQNMFe5K4mNaYGu+KeY11q7Zc2peqNsphYaeH1OF0RtTrJRmlK8UIBsMig
aFOobotnk87FA3SopFHkIijLwx4qrCDSQKz47QB8o+QxC7YHmkBCGcSYIrM4AMwZ3EE9ArGZwrB3
6mnUS+kGwIdq5K9nF8P4S0ZsAC4oOnqnXqR3wvgcmycAX0FUDEd2QJNJnGAxao24D5Sk1H9gLgQY
ubj/ADVdDY3faRHXqlbnNBa1crunsWrqk0vad9XoBwYbD6w8+fHxG26b+ePapjV1P9pPym5+f1pj
YyEkFG2R30+eFX0t+smp304dF9mSmQoBNkbLhv/CfcNfjTzoaWWaydvoJT6cYcx3ZCaNx0HjeEcw
qQ43aquNjY7JSW242bgMLzZuxGuMCWazLaTnPR5niJNNXhareVkbzkgp+904ot9VxjD83uJNZFyd
mCIzfpdBGgDBTAJkZrZkhjLwsb5lmJK3ZsQxdZYlOwFro1Od0IsDuc5YDY76qSNZTpVpIdNbVsJ9
s+An7Rteqq1rwoX/6Dg6XdvyzJwHjz79aPk6f16tXTulsHhqbe2fb8NWPK4053RV7ZuvvfRWepLd
bwHeXA68WRXjTaJkVTiLUse5G52VqbXuVrFb7DXLVkwsQtJkIzTtpVcKWovtP+ga1rVFnBkKUyyU
R7xM5TBDg8zsKGMklM/4lFkl4M0vlUKqOGgz4IiG2avyaQrjzxTWTZGSIiVRRQJprKjio0+T2NMk
FvkCngHREmv3obDYLAEBYL91FCcC442KS31gzqgaoYEC9brKIcZXFYc3g1io1mD9FZYSbnTpiK8a
nDGw9NRfZ9RUP93ZuqNhcHDa+qn3D+y4o+nRtVOuwAFs2fXuFdOasnLwmYvD5NoM559f+vVvp9J4
oGf4E76D3wwdqS50SsnJ5n2GIn6CoSK9im8wNKTPNTQ5egwdiesNG9ONuMLlMqVOtNOew0/Vrhyd
ThMygZB6mL73MEZMplgGvJPmZCeCKgeT/vNKNePF3RALuCAa4FyQlYRTtDkHOnFSrAyNVoY2K+NP
K0OblZ23QgeV2GzdOuL6A5JU159qYmgspPrWx7xUSLqFsSfeXsFaNL1uNZqy2mkjFPRf8B1DL04q
DexuWf2Xsdr2kyuin0dPYd/5D//+DL7tjjsP6knK0rvHFhXNK3gltxRcezvwaGX022/yb3/4wA2U
O6NtfBXgzAu+qV8pnJgwKb+4YHxRtdyQMC2/sqChaB4OC3MdPXi50OPYLPS6LRmC1WPPVdJ5CNzV
nCYFlBTKYhqNTuEMYybbNSYRi57MYsaqVqonqbfEAIopCigqNpxITAIevaDM+G/wsvPf+bjYVRwq
Jj6GZh/jS19KkpnyMUTEUSWL8nESUwwQcsCrk1gXI4VFBsN+69h44EU1J22QiH++awqZJfQx0jBW
zjIjz3fb08Amfoe1QXN8l7Wj0ej5tp/M1I451dVxjdeb3nzveuD0KZOfnd95fR3o3oZrlXsP3HD3
zMf6o2eiF5ITj1mDY/JyrqpeUl2FIY+35/S0qY05uUVDb5DOjLTXTg4eD4H+PQp+ajtoGAcuURI4
u8O+1g5dmlJlAm/E2CB9rzahSRjWC0ntHgC0x+HzeO5lSLFQBCJ+lEqhvguLkSlWVWuYEZOH98Fe
Mmt4UQkwu8gUFWK+Bro1yZXUkUTMOqZLwKgBDXTMHlIYnBEnRIiqPQQg1oBqYBfDWLWHAFxgjhMA
4F3TtxnYI2D8N2YYKcDUvWEH7ROOf1TL+B2VxPwdMI8VTCUxhWQZaWYCj1s1jw473z5oTUpe0DD9
x9MHB1sHFz39C7J5+vbs/LxpE4Z+AYbwlbqZb4P1I2gfmMDrhT+BI6xBjygJuJoAesqg4wE88X7o
2N4LmB1SChiuOni2MMipABZ4ZqgoDK1+TugQBk7FHf0shlARAhFCDCGxrGIcIXBCRQgAlCwsVTzM
KCnskEYQED7DNLDvjI953SFWScbQPcnBOvH1v/+9fnBQSDpxMQt+cAN89Evgo33ARzp0UanOJX/E
f5I5GUPTJ04jLkMh9huKIMc2W7eMbMS0IRo7WUb8EMuI03S4AL3nEAtArqCXNvHBws8ruYw3kMEN
OXEicWz9HFs/x9OJUxj2TljJyKJjXtHIokfWqrIDXMoqJrBq9ggY/4NxAQUYFwg3jhQNgBViXEAT
zeDJqqk66MlRw39Imf9dTZlHWL4c0nYj6XJ+37dRZePgIHGdjfwTf9YXvUm0DTmJPzIE2HoOUHY1
YItDP1ByCMYSpdxexBYY818Yj1IYpMPZD00E8TUCECMsZtfDWF0aADHPD7OlwfjbwxRNeAd/maxs
QYykEEGxJvCrBwdhKqou0CSCPvfhU0oDl8nlJWQm5FW7q7OfydcczsJZrrRUKbEyN4NPE7A5VVIK
sauwqFApbCrsLRT+8+QLqYpPpBxayOwiTqLaGjPPA+b4OfOYATivWFhYXEQZHLMWATj6NVMQAJxT
fGwxzATiTnOWLjX22wTGESaGMHBTAGEmp5nhgr7HzN4D49+pQZU5mz7ezPwbOPotezwD6NMBuMTY
AYBhxUOtrdnlZHiGpmS40cleQ2Gx2elMjRMFgFghLpVdDGOV8QBQqUMBVf2kaumbYEwDOhVQoDEV
oE6XWTFvMXNmP4Tz8Y/Kgeb4UOVI2oV3+RKaUYUPZI5DFaCboP8WQi7WCw0EHlFQ0OIMOmpkSPWV
XfWi2N6uSRw02BNbZjTe38jxKjj9XqrA9i1a/UDO6sErj+wjm2u35foKGicmTkyPBMnm+q25Ph9V
anx4c93MjuaO5vcgsRqzKsBJDpz3r1YFOhH+R1YlcZRVARqAhYmbkGg84fo+pTCIy2gTAhRWcim+
RxkTSseYSWHU/F5jwrgSTMNlK6JGVSPm5X9rTBjJ/gtbotLmv7YlDO1gSqj/+gG/CjCug4rARcU5
wRgwB2wTHA3GanO1rcEhmUIybw9xWogC1RoKAGoNBYCvqZsKgWFKMhNQanTi0SuoGuZbgdPD/Ne3
mbzAFV8y0wEA5fOYBZ/IXNs9yaZkV3IoeWUybwVFBB4bkxy6B981RXRQSsbay5nrJTIPF5oQokoy
JaAI3i/sWfWAngN4a9KIBmO8PsosUzyycgKrJYyqGYz0kEIikF8V/fSvZ6Of4cSzf8VJx5+88+6f
PHnXHU+RMdGvoi/gCmyBfxOjJ6JfvfX662/97q03AaNHo138HsAozQtkKFnFpNxe7K4idfZKdwvk
BK6RNqdq4zkBAXICsk4fywlAKgDQwixqLCfAePfI8GtxJ+lcrGgPOGFYHWlv+nesXvgvkwOgbP57
yQFVGwGtVX6mgKqN/ltZgpgZvFxlYBrnP+QJRhTLv+YJrphaebBrzi11g4MNz/W89MHxnbtnPNoA
aYL7BkjFjg+uqJ+RnRstEP65NtQc/W30i5dOTSmPbM90/l7VJ9xqcDSs0NVjQwYzeATgDZjA56jS
mgSZ1ppHesbOKbks34Jsiq3XRvQappA1jAc1zG2iMMQbTojzVbcBgJhJjWVe4tobTnypWOjjZGiP
AUwz1U3zA4zpAfinmijYkXCZP0f7DDSXAuoY0tgsl8I64eEHApdVL7dam99YOudhCFV7n2obW1DA
7dHK0ycO/YUPPza3QYB+P2gJGP6YexOylCV4ljJHJHKKnSSnZMv5mcVyRWalPC1zgRB2zPK0+GcX
rxSWOzrcXf7FxbaNQr+lz70ht8+3E+8wbHVuz70d35OiQ8akPD6d25IBTE3ZMiMje5IajyksEQJh
2CRO9hhphhAMLmnOY5ije7E5LyXINEQSi0+TIAqDoIjpUAiYLjxNL0mCO1XFQgGqM+HXYinIk6QB
In0eVysj4QJcwZS5LSYJIwKgYhju+VbJYWplVyyv2R7sDwoapkQ0LD2pAWsM5NwWYInIy+lI1qnu
86kJcqolRsJjcOZYJzoNkWmCi2ZwoGJAm0+DgZxsiI5ZySCeZFB/wABVg0QH1Azojnsz8qfNv52i
bXu7a/PN2dnLc68N3rapfPy4n17Z9Uq1tvbVRUt3+fIXBK71XT91Kq68+4UJ3terGptaKjMykuQk
Y86dV9VsLPKXjfW+GKxrvKLG63Xok7TpdfVA60nDn5OIcD/8IvaAUqkXnNB2yenMmkkGnVZISUmE
ZFdjWn8aMaKb0ySDmbE2JGCADGbm81EYXBanVtIoBnMZRNfvKhaqwDVu6oDFZCHO3nA6xt6aVMre
GvYMOPo31SBoEinmNTtSRzE35GRUj8QPP1IDh4P2qJdAl7AfkKkmw4LQVQMNwKz/6XL+gUSCPxj7
1IH+/kF8Q3SzlOSY3jimywG/1rEeeZnMvB9Pjj5/f5RrXeTLzUqRKdfvB4s2B2QemhYVm05M1mzU
cESwy4IF8lxY+v5E1wVmlWjUejk0/VJJYyZMYhHpaN8BXEmmfqkLykzYBaWE+QqqB8GcMrRXDUL/
cwwKOa8PmWIHIOYExpJfcTzDCVXDA/A3Ve1KTGromGkSAP7ONIm0fXQMqrp1o7Cvop4pXuitYhEo
yxtkBwHhI8i2lPBzBtt/tmzghUGzM6VlZt1PGwY3NzS9+Rr0Od3QvMFXkDttAlcJOIYWDb4fcCyi
W5TJ2XyeWMqXi1P4OlHME8oFRZghdEC/shOu5J3QiJ+LcrhxqIyrR1O5tXgjkZBA1kJNRyASwbTh
DroIZXOZHn7L3QN/xYFHt8BPaODHBAncYm4tx3OpVOFw12lAUKEV4HxY7X0I0W5zaryhmAP/oZrD
6jlQyeH7oxU/j4Z+g+di4IRLj/Dhoe3cBpgN/P0amGoYalhfH0USIJfqEOhY+PpyLR6cBqp/tlN5
MHBpXAHOJ3lcFp8tZEo+XQBPEKpxgzAHt/JtwhzdCrKQ75J65C7tlboN+AdkNd8nbZLXaDfq0vV0
+RonlK+QbJaJWrqCvRZCyHjtChBAf0Lnhx/uO5mthyADGGqX0QwVrJVGDsFc3me6DwC1DgoAdA0x
R4fVs8XrY006YGShoBKrdAEwqtAFjo4P++LFrgQodiWIpdF3fhb9IPrRT6Nvv/AKTrwXpx+nqOLC
QxRdD3CddKPyNBH8GUprDbpZmcyzjpsmsUPsFUWZ0wjJXKIwBddxrWgO3gC/P9bQVQtOnuPr0BQe
fuMPP9cgetINgSj0VtIGiGNKRozW9YzaArpFNsmY4xP4Gn4xv5YX+Osk8xna8wGEhqIdqN9Y5Byj
NJTOWemOLgRoHel78bVo1W/wHDyXD1/U4N/xOUMvcBV07pBi4D6CucvwS/bxdmk89EvWczXSfG62
1CH1c72SVqPhJokC9PBMgs5ViYPWDZ7Iu3QuXUjXrlup69cJ5AYtLTOfAbNAm99gNlR1McXFflJL
f7xlxx7uo6GN5KbI9dzSyGrywE1c8L5tQxDT0GwN+wxPg7948n0fmCjE8+pfPaB/8cDynb9yQP/G
QSrklj2QLc2Ev2kQ/wsG9K8XjP7LBepfLRiPqlENmoKmolpUjxrQNHQF/H6kCc1AM9Es1Az8Pwe1
ojb4UcB89lcaKI6ssNGPCN3PqHr23PqWyb6Wxau7Oq/q/L+Lv6J6CmVuZHN0cmVhbQplbmRvYmoK
MTMgMCBvYmoKMTMwNzgKZW5kb2JqCjE0IDAgb2JqCjw8IC9UeXBlIC9Gb250RGVzY3JpcHRvciAv
QXNjZW50IDEwMDUgL0NhcEhlaWdodCA3MjcgL0Rlc2NlbnQgLTIxMCAvRmxhZ3MKMzIgL0ZvbnRC
Qm94IFstNDk1IC0zMDMgMTQ0NiAxMDAxXSAvRm9udE5hbWUgL0RUWkpWQStWZXJkYW5hIC9JdGFs
aWNBbmdsZQowIC9TdGVtViAwIC9BdmdXaWR0aCA1MDggL01heFdpZHRoIDE1MjEgL1hIZWlnaHQg
NTQ1IC9Gb250RmlsZTIgMTIgMCBSID4+CmVuZG9iagoxNSAwIG9iagpbIDM1MiAwIDAgMCAwIDAg
MCAyNjkgNDU0IDQ1NCAwIDAgMzY0IDQ1NCAzNjQgMCAwIDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2
CjYzNiA2MzYgNjM2IDAgMCA4MTggMCA4MTggMCAxMDAwIDY4NCA2ODYgNjk4IDc3MSA2MzIgNTc1
IDAgNzUxIDQyMSAwIDY5MyA1NTcKODQzIDc0OCA3ODcgNjAzIDAgNjk1IDY4NCA2MTYgMCAwIDk4
OSAwIDAgMCAwIDAgMCAwIDAgMCA2MDEgNjIzIDUyMSA2MjMgNTk2CjAgNjIzIDYzMyAyNzQgMCA1
OTIgMjc0IDk3MyA2MzMgNjA3IDYyMyAwIDQyNyA1MjEgMzk0IDYzMyA1OTIgODE4IDAgNTkyIDUy
NQpdCmVuZG9iago4IDAgb2JqCjw8IC9UeXBlIC9Gb250IC9TdWJ0eXBlIC9UcnVlVHlwZSAvQmFz
ZUZvbnQgL0RUWkpWQStWZXJkYW5hIC9Gb250RGVzY3JpcHRvcgoxNCAwIFIgL1dpZHRocyAxNSAw
IFIgL0ZpcnN0Q2hhciAzMiAvTGFzdENoYXIgMTIyIC9FbmNvZGluZyAvTWFjUm9tYW5FbmNvZGlu
Zwo+PgplbmRvYmoKMTYgMCBvYmoKKFdvcmtib29rMSkKZW5kb2JqCjE3IDAgb2JqCihNYWMgT1Mg
WCAxMC42LjcgUXVhcnR6IFBERkNvbnRleHQpCmVuZG9iagoxOCAwIG9iagooT3dlbiBPJ01hbGxl
eSkKZW5kb2JqCjE5IDAgb2JqCigpCmVuZG9iagoyMCAwIG9iagooTWljcm9zb2Z0IEV4Y2VsKQpl
bmRvYmoKMjEgMCBvYmoKKEQ6MjAxMTA2MjQwNzI1MzRaMDAnMDAnKQplbmRvYmoKMjIgMCBvYmoK
KCkKZW5kb2JqCjIzIDAgb2JqClsgKCkgXQplbmRvYmoKMSAwIG9iago8PCAvVGl0bGUgMTYgMCBS
IC9BdXRob3IgMTggMCBSIC9TdWJqZWN0IDE5IDAgUiAvUHJvZHVjZXIgMTcgMCBSIC9DcmVhdG9y
CjIwIDAgUiAvQ3JlYXRpb25EYXRlIDIxIDAgUiAvTW9kRGF0ZSAyMSAwIFIgL0tleXdvcmRzIDIy
IDAgUiAvQUFQTDpLZXl3b3JkcwoyMyAwIFIgPj4KZW5kb2JqCnhyZWYKMCAyNAowMDAwMDAwMDAw
IDY1NTM1IGYgCjAwMDAwMTk4MTIgMDAwMDAgbiAKMDAwMDAwMjUzNiAwMDAwMCBuIAowMDAwMDA1
NTA4IDAwMDAwIG4gCjAwMDAwMDAwMjIgMDAwMDAgbiAKMDAwMDAwMjUxNiAwMDAwMCBuIAowMDAw
MDAyNjQwIDAwMDAwIG4gCjAwMDAwMDU0NzMgMDAwMDAgbiAKMDAwMDAxOTM5MSAwMDAwMCBuIAow
MDAwMDAyNzM4IDAwMDAwIG4gCjAwMDAwMDU0NTIgMDAwMDAgbiAKMDAwMDAwNTU5MSAwMDAwMCBu
IAowMDAwMDA1NjQxIDAwMDAwIG4gCjAwMDAwMTg4MTAgMDAwMDAgbiAKMDAwMDAxODgzMiAwMDAw
MCBuIAowMDAwMDE5MDcwIDAwMDAwIG4gCjAwMDAwMTk1NjMgMDAwMDAgbiAKMDAwMDAxOTU5MSAw
MDAwMCBuIAowMDAwMDE5NjQzIDAwMDAwIG4gCjAwMDAwMTk2NzUgMDAwMDAgbiAKMDAwMDAxOTY5
NCAwMDAwMCBuIAowMDAwMDE5NzI4IDAwMDAwIG4gCjAwMDAwMTk3NzAgMDAwMDAgbiAKMDAwMDAx
OTc4OSAwMDAwMCBuIAp0cmFpbGVyCjw8IC9TaXplIDI0IC9Sb290IDExIDAgUiAvSW5mbyAxIDAg
UiAvSUQgWyA8NzhhZjI2MjU5OGE2MjFiM2U0YmUxNTdmNmIyOTNmNWI+Cjw3OGFmMjYyNTk4YTYy
MWIzZTRiZTE1N2Y2YjI5M2Y1Yj4gXSA+PgpzdGFydHhyZWYKMTk5ODcKJSVFT0YK

--Apple-Mail-13--118723366
Content-Transfer-Encoding: 7bit
Content-Type: text/html;
	charset=us-ascii

<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div></div></body></html>
--Apple-Mail-13--118723366--

--Apple-Mail-12--118723366--

From general-return-3930-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 08:01:47 2011
Return-Path: <general-return-3930-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1FD0462C4
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 24 Jun 2011 08:01:47 +0000 (UTC)
Received: (qmail 56403 invoked by uid 500); 24 Jun 2011 08:01:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54870 invoked by uid 500); 24 Jun 2011 08:01:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54577 invoked by uid 99); 24 Jun 2011 08:01:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 08:01:25 +0000
X-ASF-Spam-Status: No, hits=3.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_BL_SPAMCOP_NET,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of owen.omalley@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 08:01:16 +0000
Received: by pwj9 with SMTP id 9so2444119pwj.35
        for <general@hadoop.apache.org>; Fri, 24 Jun 2011 01:00:55 -0700 (PDT)
Received: by 10.143.79.1 with SMTP id g1mr620763wfl.121.1308902455100;
        Fri, 24 Jun 2011 01:00:55 -0700 (PDT)
Received: from [192.168.1.103] (cpe-70-95-55-11.hawaii.res.rr.com [70.95.55.11])
        by mx.google.com with ESMTPS id l10sm1788801wfk.21.2011.06.24.01.00.52
        (version=SSLv3 cipher=OTHER);
        Fri, 24 Jun 2011 01:00:54 -0700 (PDT)
Subject: Re: Trademarks and Derivative Works
Mime-Version: 1.0 (Apple Message framework v1084)
Content-Type: multipart/alternative; boundary=Apple-Mail-14--117230071
From: Owen O'Malley <omalley@apache.org>
In-Reply-To: <3798688F8784154192FDA3388F4C208101086D60@hq-ex-mb03.ad.navteq.com>
Date: Fri, 24 Jun 2011 01:00:51 -0700
Cc: "trademarks@apache.org" <trademarks@apache.org>
Message-Id: <1B066F6B-4EC8-43E2-9927-6A048B0D9D83@apache.org>
References: <04f001cc2c4f$1815c460$48414d20$@com> <B71A7B6B-02EB-4049-A8DD-2E8321F788EE@apache.org> <3798688F8784154192FDA3388F4C208101086D60@hq-ex-mb03.ad.navteq.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-14--117230071
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

On Jun 16, 2011, at 12:22 PM, Segel, Mike wrote:

> Unfortunately what you're saying is that you would only approve of =
companies that build their products on top of Apache's release and =
doesn't modify the Apache release.=20

No, that isn't what I'm saying. I'm saying that if they want to call =
what they are shipping Hadoop, it has to be an Apache release. Derived =
works may be "powered by Hadoop." Please see the original proposal here: =
 http://wiki.apache.org/hadoop/Defining%20Hadoop

> This interpretation of  "powered by Hadoop" would unfortunately lead =
to as many problems as it attempts to solve.
> First, many choose Cloudera's release because they sell commercial =
support. So in choosing Cloudera's release, they would lose the ability =
to say "powered by Hadoop".
> This diminishes the branding message.

No, you misread the proposal. They *would* be able to say their product =
was "powered by Hadoop". What would be prohibited would be calling a =
derivative work "Hadoop."

>  (I'm not forgetting about Yahoo!, but are they releasing their own =
version as well?)

Not for the last year.

> I hope that you start to see the dangers on taking a narrow approach =
in how you define Hadoop.=20

I'm trying to learn.

-- Owen=

--Apple-Mail-14--117230071--

From general-return-3931-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 08:05:08 2011
Return-Path: <general-return-3931-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5CDC66030
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 24 Jun 2011 08:05:08 +0000 (UTC)
Received: (qmail 59053 invoked by uid 500); 24 Jun 2011 08:05:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58827 invoked by uid 500); 24 Jun 2011 08:05:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58818 invoked by uid 99); 24 Jun 2011 08:05:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 08:05:02 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of atm@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 08:04:54 +0000
Received: by bwz8 with SMTP id 8so3646688bwz.35
        for <general@hadoop.apache.org>; Fri, 24 Jun 2011 01:04:34 -0700 (PDT)
Received: by 10.204.133.27 with SMTP id d27mr1754289bkt.69.1308902674217; Fri,
 24 Jun 2011 01:04:34 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.118.210 with HTTP; Fri, 24 Jun 2011 01:04:04 -0700 (PDT)
In-Reply-To: <BAB91A74-E0CC-4A65-A0AC-4186014E473F@apache.org>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org> <BAB91A74-E0CC-4A65-A0AC-4186014E473F@apache.org>
From: "Aaron T. Myers" <atm@cloudera.com>
Date: Fri, 24 Jun 2011 01:04:04 -0700
Message-ID: <BANLkTi=DXqy=n6kQeHBCnSSYjAzDdCKEwA@mail.gmail.com>
Subject: Re: [RESULT] Powered by Logo
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001517477f86f13cfc04a670a691
X-Virus-Checked: Checked by ClamAV on apache.org

--001517477f86f13cfc04a670a691
Content-Type: text/plain; charset=ISO-8859-1

I independently counted all of the votes as well and came to the same
conclusions that Owen did. Even considering those who voted with a different
subject line, the outcome was indeed the same.

Thanks to all who participated!

--
Aaron T. Myers
Software Engineer, Cloudera



On Fri, Jun 24, 2011 at 12:35 AM, Owen O'Malley <omalley@apache.org> wrote:

> Ok,
>    I collated the votes filtering on the subject line. I know that method
> missed some of the non-binding votes, but I don't believe it changed the
> outcome.
>
> With the binding votes from the PMC:
>
>
>
>  First Round  2 4  4 4  5 6  6 1    Second Round (Drop 6)  2 5  4 4  5 6    Third
> Round (Drop 4)  2 8  5 7
>
> So in the very close race, #2 wins. My spreadsheets, which are attached
> below, show the counted votes.
>
> With all of the binding and non-binding votes:
>
>
>
>  First Round  1 1  2 13  4 23  5 27  6 4    Second Round(Drop 1)  2 13  4
> 23  5 27  6 5    Third Round (Drop 6)  2 15  4 24  5 29    Forth Round
> (Drop 2)  4 35  5 31
>
> -- Owen
>
>
>
>
>

--001517477f86f13cfc04a670a691--

From general-return-3932-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 08:26:43 2011
Return-Path: <general-return-3932-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2B3A369CD
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 24 Jun 2011 08:26:43 +0000 (UTC)
Received: (qmail 91413 invoked by uid 500); 24 Jun 2011 08:26:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90574 invoked by uid 500); 24 Jun 2011 08:26:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90556 invoked by uid 99); 24 Jun 2011 08:26:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 08:26:34 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of owen.omalley@gmail.com designates 209.85.210.48 as permitted sender)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 08:26:27 +0000
Received: by pzk10 with SMTP id 10so2189113pzk.35
        for <general@hadoop.apache.org>; Fri, 24 Jun 2011 01:26:06 -0700 (PDT)
Received: by 10.68.44.130 with SMTP id e2mr1585717pbm.515.1308903966673;
        Fri, 24 Jun 2011 01:26:06 -0700 (PDT)
Received: from [192.168.1.103] (cpe-70-95-55-11.hawaii.res.rr.com [70.95.55.11])
        by mx.google.com with ESMTPS id p5sm1889725pbd.44.2011.06.24.01.26.04
        (version=SSLv3 cipher=OTHER);
        Fri, 24 Jun 2011 01:26:05 -0700 (PDT)
From: Owen O'Malley <omalley@apache.org>
Mime-Version: 1.0 (Apple Message framework v1084)
Content-Type: multipart/alternative; boundary=Apple-Mail-15--115718501
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
Date: Fri, 24 Jun 2011 01:26:03 -0700
In-Reply-To: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
To: general@hadoop.apache.org
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org>
Message-Id: <C62A9D62-B5DA-42C7-8D00-57C2126B51C2@apache.org>
X-Mailer: Apple Mail (2.1084)

--Apple-Mail-15--115718501
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii


On Jun 14, 2011, at 3:56 PM, Owen O'Malley wrote:

> All,
>    Steve Loughran has done some great work on defining what can be =
called Hadoop at http://wiki.apache.org/hadoop/Defining%20Hadoop. After =
some cleanup from Noirin and Shane, I think we've got a really good =
base. I'd like a vote to approve the content (at the current revision =
12) and put the content on our web site.

Binding +1:  Arun, Chris, Ian, Owen
Binding -1: Doug, Eli, Todd

Non-binding +1: Allen, Cos, Matt, Steve
Non-binding -1: Ted, Jeff

Well, technically this passed, but we've been encouraged to discuss it =
more. Personally, I'd love to get more feedback from Larry about how we =
should accomplish the goal of getting packagers to either use Apache =
Hadoop releases or only use "powered by Hadoop." Clearly there is a =
significant difference of opinion about the value of that goal that is =
unlikely to be resolved by debate.

Having a clearly stated trademark statement on the website will help =
significantly with contacting organizations that are mis-using the =
trademark, so I don't want to postpone this too long. Let's discuss it =
for a week and then call a new vote if we think that is merited.

-- Owen=

--Apple-Mail-15--115718501--

From general-return-3933-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 13:43:45 2011
Return-Path: <general-return-3933-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DE3296694
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 24 Jun 2011 13:43:44 +0000 (UTC)
Received: (qmail 13971 invoked by uid 500); 24 Jun 2011 13:43:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13845 invoked by uid 500); 24 Jun 2011 13:43:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13837 invoked by uid 99); 24 Jun 2011 13:43:42 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 13:43:42 +0000
Received: from localhost (HELO [10.65.8.158]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 13:43:41 +0000
Message-ID: <4E04948B.4050802@apache.org>
Date: Fri, 24 Jun 2011 15:43:39 +0200
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org> <C62A9D62-B5DA-42C7-8D00-57C2126B51C2@apache.org>
In-Reply-To: <C62A9D62-B5DA-42C7-8D00-57C2126B51C2@apache.org>
X-Enigmail-Version: 1.1.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On 06/24/2011 10:26 AM, Owen O'Malley wrote:
> Having a clearly stated trademark statement on the website will help
> significantly with contacting organizations that are mis-using the
> trademark, so I don't want to postpone this too long. Let's discuss
> it for a week and then call a new vote if we think that is merited.

Might it be better to improve the existing Apache trademark policy page?

http://www.apache.org/foundation/marks/

This way all projects can benefit, e.g., Pig, Hive, Zookeeper, etc.

We might, e.g., propose more examples of acceptable and not-acceptable
uses of Apache marks there, etc.  We can work with trademarks@ to build
a library of boilerplate letters to be sent to folks whose use Apache
marks in objectionable ways.

Doug

From general-return-3934-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 15:48:03 2011
Return-Path: <general-return-3934-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 151866EB1
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 24 Jun 2011 15:48:03 +0000 (UTC)
Received: (qmail 28833 invoked by uid 500); 24 Jun 2011 15:48:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28740 invoked by uid 500); 24 Jun 2011 15:48:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28723 invoked by uid 99); 24 Jun 2011 15:48:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 15:48:00 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 15:47:51 +0000
Received: by vws7 with SMTP id 7so3364152vws.35
        for <general@hadoop.apache.org>; Fri, 24 Jun 2011 08:47:29 -0700 (PDT)
Received: by 10.52.72.230 with SMTP id g6mr4762847vdv.163.1308930449059; Fri,
 24 Jun 2011 08:47:29 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.52.111.166 with HTTP; Fri, 24 Jun 2011 08:47:08 -0700 (PDT)
X-Originating-IP: [67.160.196.149]
In-Reply-To: <BAB91A74-E0CC-4A65-A0AC-4186014E473F@apache.org>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org> <BAB91A74-E0CC-4A65-A0AC-4186014E473F@apache.org>
From: Ted Dunning <tdunning@maprtech.com>
Date: Fri, 24 Jun 2011 08:47:08 -0700
Message-ID: <BANLkTin_-0_wPuUY1Br9Ac=QsFEiYeSBkw@mail.gmail.com>
Subject: Re: [RESULT] Powered by Logo
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec501601173b1a004a6771e96
X-Virus-Checked: Checked by ClamAV on apache.org

--bcaec501601173b1a004a6771e96
Content-Type: text/plain; charset=ISO-8859-1

So there is a significant difference between the community of developers and
the PMC here and we definitely don't really have a consensus.

One good suggestion that I had was from Eli.  It sounds to me like his
objection (and possibly related objections) to the logo that I proposed was
based on the fact that the logo that I proposed matched the color scheme on
our web site.  In a side conversation (that obviously didn't officially
happen, being off-list), it seemed that he would be willing to push for
support for a logo with more flexibility in the way that core logo is
encircled.

In our conversation, it seems that a significant portion of the divergence
of opinion may simply be that different groups are attuned to their own
company or local community color palettes which makes logos tuned to other
color combinations much less attractive.

How about we have a logo family where the yellow elephant and "Powered by"
text is the core and the circle or its graphical equivalent is optional?

The intent would be to allow designers the freedom to help the logo conform
to their local look while maintaining a core identity?

Can we get consensus around that?

On Fri, Jun 24, 2011 at 12:35 AM, Owen O'Malley <omalley@apache.org> wrote:

> Ok,
>    I collated the votes filtering on the subject line. I know that method
> missed some of the non-binding votes, but I don't believe it changed the
> outcome.
>
> With the binding votes from the PMC:
>
>
>
>  First Round  2 4  4 4  5 6  6 1    Second Round (Drop 6)  2 5  4 4  5 6    Third
> Round (Drop 4)  2 8  5 7
>
> So in the very close race, #2 wins. My spreadsheets, which are attached
> below, show the counted votes.
>
> With all of the binding and non-binding votes:
>
>
>
>  First Round  1 1  2 13  4 23  5 27  6 4    Second Round(Drop 1)  2 13  4
> 23  5 27  6 5    Third Round (Drop 6)  2 15  4 24  5 29    Forth Round
> (Drop 2)  4 35  5 31
>
> -- Owen
>
>
>
>
>

--bcaec501601173b1a004a6771e96--

From general-return-3935-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 16:47:58 2011
Return-Path: <general-return-3935-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6C782675A
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 24 Jun 2011 16:47:58 +0000 (UTC)
Received: (qmail 7637 invoked by uid 500); 24 Jun 2011 16:47:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7575 invoked by uid 500); 24 Jun 2011 16:47:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7566 invoked by uid 99); 24 Jun 2011 16:47:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 16:47:55 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 16:47:49 +0000
Received: by wwi14 with SMTP id 14so2858411wwi.29
        for <general@hadoop.apache.org>; Fri, 24 Jun 2011 09:47:29 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.229.7 with SMTP id g7mr3026082weq.98.1308934048771; Fri,
 24 Jun 2011 09:47:28 -0700 (PDT)
Received: by 10.216.30.212 with HTTP; Fri, 24 Jun 2011 09:47:28 -0700 (PDT)
In-Reply-To: <BANLkTin_-0_wPuUY1Br9Ac=QsFEiYeSBkw@mail.gmail.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
	<BAB91A74-E0CC-4A65-A0AC-4186014E473F@apache.org>
	<BANLkTin_-0_wPuUY1Br9Ac=QsFEiYeSBkw@mail.gmail.com>
Date: Fri, 24 Jun 2011 09:47:28 -0700
Message-ID: <BANLkTik2Gezh3y1vMQD7x=tGheuet=4MRQ@mail.gmail.com>
Subject: Re: [RESULT] Powered by Logo
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Fri, Jun 24, 2011 at 8:47 AM, Ted Dunning <tdunning@maprtech.com> wrote:
> So there is a significant difference between the community of developers =
and
> the PMC here and we definitely don't really have a consensus.
>
> One good suggestion that I had was from Eli. =A0It sounds to me like his
> objection (and possibly related objections) to the logo that I proposed w=
as
> based on the fact that the logo that I proposed matched the color scheme =
on
> our web site. =A0In a side conversation (that obviously didn't officially
> happen, being off-list), it seemed that he would be willing to push for
> support for a logo with more flexibility in the way that core logo is
> encircled.
>

Hey Ted,

Off list I said I thought any of the logos were acceptable choices
modulo some design tweaks, and that if the circle logo concept is
accepted it would be good to adjust the circle itself.  But the PMC
did not vote for the circle logo, so it's a mute point.

Thanks,
Eli

From general-return-3936-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 16:54:53 2011
Return-Path: <general-return-3936-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 26B736851
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 24 Jun 2011 16:54:53 +0000 (UTC)
Received: (qmail 16412 invoked by uid 500); 24 Jun 2011 16:54:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16365 invoked by uid 500); 24 Jun 2011 16:54:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16355 invoked by uid 99); 24 Jun 2011 16:54:51 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 16:54:51 +0000
Received: from localhost (HELO awittena-md.linkedin.biz) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 16:54:51 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [RESULT] Powered by Logo
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <BANLkTik2Gezh3y1vMQD7x=tGheuet=4MRQ@mail.gmail.com>
Date: Fri, 24 Jun 2011 09:54:49 -0700
Content-Transfer-Encoding: 7bit
Message-Id: <420C1431-457C-4D2E-B3DB-618C9B69D8CC@apache.org>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org> <BAB91A74-E0CC-4A65-A0AC-4186014E473F@apache.org> <BANLkTin_-0_wPuUY1Br9Ac=QsFEiYeSBkw@mail.gmail.com> <BANLkTik2Gezh3y1vMQD7x=tGheuet=4MRQ@mail.gmail.com>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On Jun 24, 2011, at 9:47 AM, Eli Collins wrote:
> 
> Off list I said I thought any of the logos were acceptable choices
> modulo some design tweaks, and that if the circle logo concept is
> accepted it would be good to adjust the circle itself.  But the PMC
> did not vote for the circle logo, so it's a mute point.

Clearly the PMC is full of circle-phobes.


From general-return-3937-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 17:08:04 2011
Return-Path: <general-return-3937-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 45CBE60F9
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 24 Jun 2011 17:08:04 +0000 (UTC)
Received: (qmail 40252 invoked by uid 500); 24 Jun 2011 17:08:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39892 invoked by uid 500); 24 Jun 2011 17:08:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39629 invoked by uid 99); 24 Jun 2011 17:08:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 17:08:01 +0000
X-ASF-Spam-Status: No, hits=1.3 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_BL_SPAMCOP_NET,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of owen.omalley@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 17:07:54 +0000
Received: by pwj9 with SMTP id 9so2801463pwj.35
        for <general@hadoop.apache.org>; Fri, 24 Jun 2011 10:07:33 -0700 (PDT)
Received: by 10.68.48.129 with SMTP id l1mr1820656pbn.112.1308935253926;
        Fri, 24 Jun 2011 10:07:33 -0700 (PDT)
Received: from [192.168.1.103] (cpe-70-95-55-11.hawaii.res.rr.com [70.95.55.11])
        by mx.google.com with ESMTPS id v6sm2188574pbh.22.2011.06.24.10.07.30
        (version=SSLv3 cipher=OTHER);
        Fri, 24 Jun 2011 10:07:32 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
From: Owen O'Malley <omalley@apache.org>
In-Reply-To: <4E04948B.4050802@apache.org>
Date: Fri, 24 Jun 2011 10:07:27 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <C3BF2AA2-B17C-4E9E-8BDE-D0776B8B8435@apache.org>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org> <C62A9D62-B5DA-42C7-8D00-57C2126B51C2@apache.org> <4E04948B.4050802@apache.org>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)


On Jun 24, 2011, at 6:43 AM, Doug Cutting wrote:

> Might it be better to improve the existing Apache trademark policy =
page?

When the project is having trouble agreeing, reaching agreement at the =
foundation level seems unrealistic. Let's reach a workable solution for =
Hadoop, see how it functions in practice, iterate and improve, and then =
we can consider pushing it to the entire foundation.

-- Owen=

From general-return-3938-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 21:08:42 2011
Return-Path: <general-return-3938-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5EC7167E9
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 24 Jun 2011 21:08:42 +0000 (UTC)
Received: (qmail 13437 invoked by uid 500); 24 Jun 2011 21:08:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13343 invoked by uid 500); 24 Jun 2011 21:08:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13335 invoked by uid 99); 24 Jun 2011 21:08:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 21:08:39 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 216.145.54.171 is neither permitted nor denied by domain of sureshms@yahoo-inc.com)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 21:08:32 +0000
Received: from sp1-ex07cas03.ds.corp.yahoo.com (sp1-ex07cas03.ds.corp.yahoo.com [216.252.116.151])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5OL81m3026057
	for <general@hadoop.apache.org>; Fri, 24 Jun 2011 14:08:01 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yahoo-inc.com;
	s=cobra; t=1308949681;
	bh=OIhBzCELR4pdao+G6gUwCcuXG1AMeOfbIr8k+ZbFDuw=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:MIME-Version;
	b=t2gZZuuMfcuAN0++PkZrErwG1pvQysp4saZz9sr/9Nin3a2GlcY/YJxtm1LecB7BG
	 Syel1EHoFVQi1pRYMNHZOLEPDc0cgY4oy7XhwybF8FxHfVWDUI5m7RISe2Wa5M8nP1
	 gQrvHD18vdZBOfHFo4EwYcAIOfijeIzK+knopDgc=
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas03.ds.corp.yahoo.com ([216.252.116.151]) with mapi; Fri, 24 Jun
 2011 14:08:00 -0700
From: Suresh Srinivas <sureshms@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 24 Jun 2011 14:07:57 -0700
Subject: Re: Thinking about the next hadoop mainline release
Thread-Topic: Thinking about the next hadoop mainline release
Thread-Index: Acws+RPlMNysOuyuR6ehLH41N0fBKgFubad4
Message-ID: <CA2A4ABD.1C55C%sureshms@yahoo-inc.com>
In-Reply-To: <E9E17CC3-013C-46E5-AD48-BF76DEE8CE02@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-Entourage/13.8.0.101117
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

+1. Arun, I can also help you with managing the release for HDFS.


On 6/17/11 7:15 AM, "Arun C Murthy" <arunc@yahoo-inc.com> wrote:

> I volunteer to be the RM for the release since I've been leading the NG N=
R
> effort.
>=20
> Are folks ok with this?
>=20
> thanks,
> Arun
>=20
> Sent from my iPhone
>=20
> On Jun 17, 2011, at 1:45 PM, "Ted Dunning" <tdunning@maprtech.com> wrote:
>=20
>> NG map reduce is a huge deal both in terms of making things better for
>> users, but also in terms of unblocking the Hadoop development process.
>>=20
>> On Fri, Jun 17, 2011 at 9:36 AM, Ryan Rawson <ryanobjc@gmail.com> wrote:
>>=20
>>>> - Next Generation Map-Reduce [MR-279]
>>>> - Passing most tests now and discussing merging into trunk
>>>=20


From general-return-3939-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 22:05:22 2011
Return-Path: <general-return-3939-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7675D66F8
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 24 Jun 2011 22:05:22 +0000 (UTC)
Received: (qmail 83074 invoked by uid 500); 24 Jun 2011 22:05:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83011 invoked by uid 500); 24 Jun 2011 22:05:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83003 invoked by uid 99); 24 Jun 2011 22:05:20 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 22:05:20 +0000
Received: from localhost (HELO mail-fx0-f48.google.com) (127.0.0.1)
  (smtp-auth username mahadev, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 22:05:19 +0000
Received: by fxg7 with SMTP id 7so273653fxg.35
        for <general@hadoop.apache.org>; Fri, 24 Jun 2011 15:05:18 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.223.4.136 with SMTP id 8mr5100245far.16.1308953118154; Fri, 24
 Jun 2011 15:05:18 -0700 (PDT)
Received: by 10.223.100.7 with HTTP; Fri, 24 Jun 2011 15:05:17 -0700 (PDT)
In-Reply-To: <CA2A4ABD.1C55C%sureshms@yahoo-inc.com>
References: <E9E17CC3-013C-46E5-AD48-BF76DEE8CE02@yahoo-inc.com>
	<CA2A4ABD.1C55C%sureshms@yahoo-inc.com>
Date: Fri, 24 Jun 2011 15:05:17 -0700
Message-ID: <BANLkTin+FJ4ryb3Hn9khjVE66LCnpcRzzQ@mail.gmail.com>
Subject: Re: Thinking about the next hadoop mainline release
From: Mahadev Konar <mahadev@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

+1. I'd be happy to help in any way possible.

thanks
mahadev

On Fri, Jun 24, 2011 at 2:07 PM, Suresh Srinivas <sureshms@yahoo-inc.com> wrote:
> +1. Arun, I can also help you with managing the release for HDFS.
>
>
> On 6/17/11 7:15 AM, "Arun C Murthy" <arunc@yahoo-inc.com> wrote:
>
>> I volunteer to be the RM for the release since I've been leading the NG NR
>> effort.
>>
>> Are folks ok with this?
>>
>> thanks,
>> Arun
>>
>> Sent from my iPhone
>>
>> On Jun 17, 2011, at 1:45 PM, "Ted Dunning" <tdunning@maprtech.com> wrote:
>>
>>> NG map reduce is a huge deal both in terms of making things better for
>>> users, but also in terms of unblocking the Hadoop development process.
>>>
>>> On Fri, Jun 17, 2011 at 9:36 AM, Ryan Rawson <ryanobjc@gmail.com> wrote:
>>>
>>>>> - Next Generation Map-Reduce [MR-279]
>>>>> - Passing most tests now and discussing merging into trunk
>>>>
>
>



-- 
thanks
mahadev
@mahadevkonar

From general-return-3940-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 00:29:10 2011
Return-Path: <general-return-3940-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6EC15600E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat, 25 Jun 2011 00:29:10 +0000 (UTC)
Received: (qmail 97691 invoked by uid 500); 25 Jun 2011 00:29:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97615 invoked by uid 500); 25 Jun 2011 00:29:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97604 invoked by uid 99); 25 Jun 2011 00:29:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 00:29:07 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 216.145.54.173 is neither permitted nor denied by domain of arunc@yahoo-inc.com)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 00:29:01 +0000
Received: from SP2-EX07CAS05.ds.corp.yahoo.com (sp2-ex07cas05.corp.sp2.yahoo.com [98.137.59.39])
	by mrout3.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5P0ScAT056459
	for <general@hadoop.apache.org>; Fri, 24 Jun 2011 17:28:38 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yahoo-inc.com;
	s=cobra; t=1308961718;
	bh=YhGKVnDHShkvdKLE9H+3+jr0rBYGd1C2OMg0tgP6SXY=;
	h=From:To:Date:Subject:Message-ID:References:In-Reply-To:
	 Content-Type:Content-Transfer-Encoding:MIME-Version;
	b=BmrFP/tTiLXp5GfxRajsTgvngF6UKfdigDQJJfph+LMzyOD1sikkgR32Ei6HZXKPh
	 4LInP2eRwGvqTfIFbrOlsyQ6xdlB5joNLYPLf1pw95fheyeCqWqhoCRNYiQtNhX7Hf
	 9fvyYwJU+Gg9Nob0YD+KYvT+3mu6M+4GPViIh8tU=
Received: from SP2-EX07VS04.ds.corp.yahoo.com ([98.137.59.33]) by
 SP2-EX07CAS05.ds.corp.yahoo.com ([98.137.59.39]) with mapi; Fri, 24 Jun 2011
 17:28:37 -0700
From: Arun C Murthy <arunc@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 24 Jun 2011 17:28:33 -0700
Subject: Re: Thinking about the next hadoop mainline release
Thread-Topic: Thinking about the next hadoop mainline release
Thread-Index: AcwyztMW3IYuYrP1ReWRoElgPrT3/Q==
Message-ID: <41D837D5-97C3-48BB-9F95-01720DFB769C@yahoo-inc.com>
References: <CA2A4ABD.1C55C%sureshms@yahoo-inc.com>
In-Reply-To: <CA2A4ABD.1C55C%sureshms@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Thanks Suresh!

Todd - I'd appreciate if you could help on some of the HBase/Performance ji=
ras... thanks!

Sent from my iPhone

On Jun 24, 2011, at 2:09 PM, "Suresh Srinivas" <sureshms@yahoo-inc.com> wro=
te:

> +1. Arun, I can also help you with managing the release for HDFS.
>=20
>=20
> On 6/17/11 7:15 AM, "Arun C Murthy" <arunc@yahoo-inc.com> wrote:
>=20
>> I volunteer to be the RM for the release since I've been leading the NG =
NR
>> effort.
>>=20
>> Are folks ok with this?
>>=20
>> thanks,
>> Arun
>>=20
>> Sent from my iPhone
>>=20
>> On Jun 17, 2011, at 1:45 PM, "Ted Dunning" <tdunning@maprtech.com> wrote=
:
>>=20
>>> NG map reduce is a huge deal both in terms of making things better for
>>> users, but also in terms of unblocking the Hadoop development process.
>>>=20
>>> On Fri, Jun 17, 2011 at 9:36 AM, Ryan Rawson <ryanobjc@gmail.com> wrote=
:
>>>=20
>>>>> - Next Generation Map-Reduce [MR-279]
>>>>> - Passing most tests now and discussing merging into trunk
>>>>=20
>=20

From general-return-3941-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 02:54:11 2011
Return-Path: <general-return-3941-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 581154AD0
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat, 25 Jun 2011 02:54:11 +0000 (UTC)
Received: (qmail 81393 invoked by uid 500); 25 Jun 2011 02:54:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80947 invoked by uid 500); 25 Jun 2011 02:54:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80937 invoked by uid 99); 25 Jun 2011 02:53:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 02:53:57 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mlortiz@uci.cu designates 200.55.140.180 as permitted sender)
Received: from [200.55.140.180] (HELO mx3.uci.cu) (200.55.140.180)
    by apache.org (qpsmtpd/0.29) with SMTP; Sat, 25 Jun 2011 02:53:51 +0000
Received: (qmail 27820 invoked by uid 507); 25 Jun 2011 02:53:24 -0000
Received: from 10.0.0.183 by ns3.uci.cu (envelope-from <mlortiz@uci.cu>, uid 501) with qmail-scanner-2.01st 
 (avp: 5.0.2.0. spamassassin: 3.0.6. perlscan: 2.01st.  
 Clear:RC:1(10.0.0.183):. 
 Processed in 0.65361 secs); 25 Jun 2011 02:53:24 -0000
Received: from unknown (HELO ucimail2.uci.cu) (10.0.0.183)
  by 0 with SMTP; 25 Jun 2011 02:53:23 -0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by ucimail2.uci.cu (Postfix) with ESMTP id B9CE63DC009
	for <general@hadoop.apache.org>; Fri, 24 Jun 2011 22:55:13 -0400 (CDT)
X-Virus-Scanned: amavisd-new at uci.cu
Received: from ucimail2.uci.cu ([127.0.0.1])
	by localhost (ucimail2.uci.cu [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id t6YCvTfJa48E; Fri, 24 Jun 2011 22:55:12 -0400 (CDT)
Received: from [10.8.46.25] (unknown [10.8.46.25])
	by ucimail2.uci.cu (Postfix) with ESMTP id 345AA3DC007
	for <general@hadoop.apache.org>; Fri, 24 Jun 2011 22:55:11 -0400 (CDT)
Message-ID: <4E054E1A.2090809@uci.cu>
Date: Fri, 24 Jun 2011 22:55:22 -0400
From: Marcos Ortiz <mlortiz@uci.cu>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; es-ES; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: =?ISO-8859-1?Q?Hadoop=B4s_Internationalization?=
References: <CA2A4ABD.1C55C%sureshms@yahoo-inc.com> <41D837D5-97C3-48BB-9F95-01720DFB769C@yahoo-inc.com>
In-Reply-To: <41D837D5-97C3-48BB-9F95-01720DFB769C@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit
X-Virus-Checked: Checked by ClamAV on apache.org

Regards to all the list.
I´m looking for a proper way to work on the internationalization of 
Hadoop, but I don´t know if this is a good project
or if this is useful for the community, at least, I think that it would 
be very useful for many people that want to see the messages
of the project in another language, for example, in Spanish.

I would like to work on this, but I don´t know how to start on this. I 
think that a first step is to translate the message of the commands more 
used by administrators and developers of Hadoop: (The Hadoop fs shell, 
for example).

It would be very useful to have on the source code, a directory for 
earch language´s messages, but I don´t know if this could carry out
many changes on the main API.

What do you think about this?

Thanls for your time

-- 
Marcos Luís Ortíz Valmaseda
  Software Engineer (UCI)
  http://marcosluis2186.posterous.com
  http://twitter.com/marcosluis2186
   


From general-return-3942-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 03:42:11 2011
Return-Path: <general-return-3942-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4729743BA
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat, 25 Jun 2011 03:42:11 +0000 (UTC)
Received: (qmail 5514 invoked by uid 500); 25 Jun 2011 03:42:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4686 invoked by uid 500); 25 Jun 2011 03:41:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4669 invoked by uid 99); 25 Jun 2011 03:41:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 03:41:45 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.30.32] (HELO qmta03.emeryville.ca.mail.comcast.net) (76.96.30.32)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 03:41:35 +0000
Received: from omta23.emeryville.ca.mail.comcast.net ([76.96.30.90])
	by qmta03.emeryville.ca.mail.comcast.net with comcast
	id 03hB1h0031wfjNsA33hD25; Sat, 25 Jun 2011 03:41:13 +0000
Received: from fs ([24.4.185.157])
	by omta23.emeryville.ca.mail.comcast.net with comcast
	id 03hV1h00d3QAh8g8j3hVSB; Sat, 25 Jun 2011 03:41:30 +0000
Received: from mail.boudnik.org (localhost [127.0.0.1])
	by fs (8.14.3/8.14.3/Debian-5+lenny1) with ESMTP id p5P3fD7R014546
	for <general@hadoop.apache.org>; Fri, 24 Jun 2011 20:41:13 -0700
Received: (from cos@localhost)
	by mail.boudnik.org (8.14.3/8.14.3/Submit) id p5P3fCdY014545
	for general@hadoop.apache.org; Fri, 24 Jun 2011 20:41:12 -0700
X-Authentication-Warning: mail.boudnik.org: cos set sender to cos@apache.org using -f
Date: Fri, 24 Jun 2011 20:41:12 -0700
From: Konstantin Boudnik <cos@apache.org>
To: general@hadoop.apache.org
Subject: Re: [RESULT] Powered by Logo
Message-ID: <20110625034112.GA14500@linspire.com>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org> <BAB91A74-E0CC-4A65-A0AC-4186014E473F@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <BAB91A74-E0CC-4A65-A0AC-4186014E473F@apache.org>
X-Organization: It's something of 'Cos
User-Agent: Mutt/1.5.18 (2008-05-17)

Owen,

looks like your non-PMC list doesn't account for about 13 or so user votes
(which we cast under different subjects and making total number of #4 equals
37) as you can see here http://tinyurl.com/3j6629h

Hope it helps
  Cos

On Fri, Jun 24, 2011 at 12:35AM, Owen O'Malley wrote:
> Ok,
>    I collated the votes filtering on the subject line. I know that method missed some of the non-binding votes, but I don't believe it changed the outcome.
> 
> With the binding votes from the PMC:
> 
> 
> 
> First Round	
> 2	4
> 4	4
> 5	6
> 6	1
> Second Round (Drop 6)
> 2	5
> 4	4
> 5	6
> Third Round (Drop 4)
> 2	8
> 5	7
> 
>  
> So in the very close race, #2 wins. My spreadsheets, which are attached below, show the counted votes.
> 
> With all of the binding and non-binding votes:
> 
> 
> 
> First Round	
> 1	1
> 2	13
> 4	23
> 5	27
> 6	4
> Second Round(Drop 1)
> 2	13
> 4	23
> 5	27
> 6	5
> Third Round (Drop 6)
> 2	15
> 4	24
> 5	29
> Forth Round (Drop 2)
> 4	35
> 5	31
> 
> 
> -- Owen
> 

From general-return-3943-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 05:08:27 2011
Return-Path: <general-return-3943-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 39816403D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat, 25 Jun 2011 05:08:27 +0000 (UTC)
Received: (qmail 33079 invoked by uid 500); 25 Jun 2011 05:08:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32562 invoked by uid 500); 25 Jun 2011 05:08:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32553 invoked by uid 99); 25 Jun 2011 05:08:16 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 05:08:16 +0000
Received: from localhost (HELO [10.65.8.159]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 05:08:15 +0000
Message-ID: <4E056D3B.9030008@apache.org>
Date: Sat, 25 Jun 2011 07:08:11 +0200
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org> <C62A9D62-B5DA-42C7-8D00-57C2126B51C2@apache.org> <4E04948B.4050802@apache.org> <C3BF2AA2-B17C-4E9E-8BDE-D0776B8B8435@apache.org>
In-Reply-To: <C3BF2AA2-B17C-4E9E-8BDE-D0776B8B8435@apache.org>
X-Enigmail-Version: 1.1.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On 06/24/2011 07:07 PM, Owen O'Malley wrote:
> On Jun 24, 2011, at 6:43 AM, Doug Cutting wrote:
> 
>> Might it be better to improve the existing Apache trademark policy
>> page?
> 
> When the project is having trouble agreeing, reaching agreement at
> the foundation level seems unrealistic.

ASF trademark policy is set by Shane, VP Trademark, not by a committee.

Doug

From general-return-3944-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 05:40:03 2011
Return-Path: <general-return-3944-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2D04A4087
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat, 25 Jun 2011 05:40:03 +0000 (UTC)
Received: (qmail 46282 invoked by uid 500); 25 Jun 2011 05:40:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45438 invoked by uid 500); 25 Jun 2011 05:39:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45226 invoked by uid 99); 25 Jun 2011 05:39:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 05:39:35 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates 209.85.213.176 as permitted sender)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 05:39:30 +0000
Received: by yxd5 with SMTP id 5so2086340yxd.35
        for <general@hadoop.apache.org>; Fri, 24 Jun 2011 22:39:09 -0700 (PDT)
Received: by 10.100.83.19 with SMTP id g19mr4502700anb.47.1308980349157; Fri,
 24 Jun 2011 22:39:09 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.100.141.12 with HTTP; Fri, 24 Jun 2011 22:38:49 -0700 (PDT)
In-Reply-To: <41D837D5-97C3-48BB-9F95-01720DFB769C@yahoo-inc.com>
References: <CA2A4ABD.1C55C%sureshms@yahoo-inc.com> <41D837D5-97C3-48BB-9F95-01720DFB769C@yahoo-inc.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Fri, 24 Jun 2011 22:38:49 -0700
Message-ID: <BANLkTi==6SNFyUkxm+QMdqk0v04tZmC+ow@mail.gmail.com>
Subject: Re: Thinking about the next hadoop mainline release
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504501710ebac01f04a682bc4c

--00504501710ebac01f04a682bc4c
Content-Type: text/plain; charset=ISO-8859-1

On Fri, Jun 24, 2011 at 5:28 PM, Arun C Murthy <arunc@yahoo-inc.com> wrote:

> Thanks Suresh!
>
> Todd - I'd appreciate if you could help on some of the HBase/Performance
> jiras... thanks!
>
>
Sure thing.

-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

--00504501710ebac01f04a682bc4c--

From general-return-3945-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 08:38:38 2011
Return-Path: <general-return-3945-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 007654D54
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat, 25 Jun 2011 08:38:38 +0000 (UTC)
Received: (qmail 23322 invoked by uid 500); 25 Jun 2011 08:38:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22507 invoked by uid 500); 25 Jun 2011 08:38:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22492 invoked by uid 99); 25 Jun 2011 08:38:15 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 08:38:15 +0000
Received: from localhost (HELO mail-qy0-f169.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 08:38:13 +0000
Received: by qyk32 with SMTP id 32so787470qyk.14
        for <general@hadoop.apache.org>; Sat, 25 Jun 2011 01:38:12 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.51.67 with SMTP id c3mr973590qcg.12.1308991092578; Sat, 25
 Jun 2011 01:38:12 -0700 (PDT)
Received: by 10.229.37.13 with HTTP; Sat, 25 Jun 2011 01:38:12 -0700 (PDT)
In-Reply-To: <4E054E1A.2090809@uci.cu>
References: <CA2A4ABD.1C55C%sureshms@yahoo-inc.com>
	<41D837D5-97C3-48BB-9F95-01720DFB769C@yahoo-inc.com>
	<4E054E1A.2090809@uci.cu>
Date: Sat, 25 Jun 2011 01:38:12 -0700
Message-ID: <BANLkTik5iE_y8FM8nUVJqdcS8tnBpx+Bvg@mail.gmail.com>
Subject: =?UTF-8?Q?Re=3A_Hadoop=C2=B4s_Internationalization?=
From: "Owen O'Malley" <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636832358165a3504a6853d03

--001636832358165a3504a6853d03
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On Fri, Jun 24, 2011 at 7:55 PM, Marcos Ortiz <mlortiz@uci.cu> wrote:

> Regards to all the list.
> I=C2=B4m looking for a proper way to work on the internationalization of =
Hadoop,
> but I don=C2=B4t know if this is a good project
> or if this is useful for the community, at least, I think that it would b=
e
> very useful for many people that want to see the messages
> of the project in another language, for example, in Spanish.


I think it would be very useful, but very time consuming project. I'd
suggest that you start by translating the documentation for the release int=
o
another language first.

-- Owen

--001636832358165a3504a6853d03--

From general-return-3946-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 15:16:07 2011
Return-Path: <general-return-3946-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 955FB45E6
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat, 25 Jun 2011 15:16:07 +0000 (UTC)
Received: (qmail 53297 invoked by uid 500); 25 Jun 2011 15:16:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53174 invoked by uid 500); 25 Jun 2011 15:16:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53164 invoked by uid 99); 25 Jun 2011 15:16:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 15:16:04 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mlortiz@uci.cu designates 200.55.140.180 as permitted sender)
Received: from [200.55.140.180] (HELO mx3.uci.cu) (200.55.140.180)
    by apache.org (qpsmtpd/0.29) with SMTP; Sat, 25 Jun 2011 15:15:58 +0000
Received: (qmail 16391 invoked by uid 507); 25 Jun 2011 15:15:32 -0000
Received: from 10.0.0.183 by ns3.uci.cu (envelope-from <mlortiz@uci.cu>, uid 501) with qmail-scanner-2.01st 
 (avp: 5.0.2.0. spamassassin: 3.0.6. perlscan: 2.01st.  
 Clear:RC:1(10.0.0.183):. 
 Processed in 0.6502 secs); 25 Jun 2011 15:15:32 -0000
Received: from unknown (HELO ucimail2.uci.cu) (10.0.0.183)
  by 0 with SMTP; 25 Jun 2011 15:15:31 -0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by ucimail2.uci.cu (Postfix) with ESMTP id 1B9DD3DC009;
	Sat, 25 Jun 2011 11:17:22 -0400 (CDT)
X-Virus-Scanned: amavisd-new at uci.cu
Received: from ucimail2.uci.cu ([127.0.0.1])
	by localhost (ucimail2.uci.cu [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id T9WRO8UNmBzE; Sat, 25 Jun 2011 11:17:21 -0400 (CDT)
Received: from [10.7.3.1] (skynet2010.uci.cu [10.7.3.1])
	by ucimail2.uci.cu (Postfix) with ESMTP id 3CE223DC007;
	Sat, 25 Jun 2011 11:17:21 -0400 (CDT)
Message-ID: <4E05FC0B.2070901@uci.cu>
Date: Sat, 25 Jun 2011 11:17:31 -0400
From: Marcos Ortiz <mlortiz@uci.cu>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; es-ES; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: general@hadoop.apache.org
CC: Owen O'Malley <omalley@apache.org>
Subject: Re: =?UTF-8?B?SGFkb29wwrRzIEludGVybmF0aW9uYWxpemF0aW9u?=
References: <CA2A4ABD.1C55C%sureshms@yahoo-inc.com>	<41D837D5-97C3-48BB-9F95-01720DFB769C@yahoo-inc.com>	<4E054E1A.2090809@uci.cu> <BANLkTik5iE_y8FM8nUVJqdcS8tnBpx+Bvg@mail.gmail.com>
In-Reply-To: <BANLkTik5iE_y8FM8nUVJqdcS8tnBpx+Bvg@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
X-Virus-Checked: Checked by ClamAV on apache.org

OK, Owen, Where I can find the sources of the docs?
Which is the format for the docs? DocBook, ReST, etc?

El 6/25/2011 4:38 AM, Owen O'Malley escribió:
> On Fri, Jun 24, 2011 at 7:55 PM, Marcos Ortiz<mlortiz@uci.cu>  wrote:
>
>    
>> Regards to all the list.
>> I´m looking for a proper way to work on the internationalization of Hadoop,
>> but I don´t know if this is a good project
>> or if this is useful for the community, at least, I think that it would be
>> very useful for many people that want to see the messages
>> of the project in another language, for example, in Spanish.
>>      
>
> I think it would be very useful, but very time consuming project. I'd
> suggest that you start by translating the documentation for the release into
> another language first.
>
> -- Owen
>
>    

-- 
Marcos Luís Ortíz Valmaseda
  Software Engineer (UCI)
  http://marcosluis2186.posterous.com
  http://twitter.com/marcosluis2186
   


From general-return-3947-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 16:09:13 2011
Return-Path: <general-return-3947-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2EF2346FC
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat, 25 Jun 2011 16:09:13 +0000 (UTC)
Received: (qmail 84322 invoked by uid 500); 25 Jun 2011 16:09:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84255 invoked by uid 500); 25 Jun 2011 16:09:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84247 invoked by uid 99); 25 Jun 2011 16:09:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 16:09:10 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com designates 74.125.83.176 as permitted sender)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 16:09:06 +0000
Received: by pve37 with SMTP id 37so3187456pve.35
        for <general@hadoop.apache.org>; Sat, 25 Jun 2011 09:08:46 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=VWYQJQhuAhhKwSo+v/BkBi/hcEsLDaDo6vyAdNhfXPg=;
        b=izUM/wDvbnoMRHAQM1NGCKwFhkPO9feZIiRg9ofFmyr6gOLciRdm/uMEqzZgjG7jJA
         UANevHQMyfKsy57pxoHRHaKYDzYI8eqymbY0z1MIVZE5nDSEGExRcNph33ZNJ+3cRYbR
         f9ZANOfIHijPJ0mPTjiwlyXuikWQ7kFJEWZnw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=jBgtU0ml8JjzSxhP2PmC2kt/4kYgMaQIUFTRvZ0Bj1shVUqrUGqxJvVqppepXV1r1h
         7xeoXmvSv5i+/UuoIOtkCBaWK/VxwfwSNALt7Mt9OT52dCl0VHfRWf8Bnzh6FbrlWh4c
         aCJ/pVM47dQWVcZWqMxVNYWt0sdsIOWMV/lwA=
MIME-Version: 1.0
Received: by 10.68.47.39 with SMTP id a7mr2021006pbn.217.1309018126395; Sat,
 25 Jun 2011 09:08:46 -0700 (PDT)
Received: by 10.68.66.168 with HTTP; Sat, 25 Jun 2011 09:08:46 -0700 (PDT)
In-Reply-To: <4E05FC0B.2070901@uci.cu>
References: <CA2A4ABD.1C55C%sureshms@yahoo-inc.com>
	<41D837D5-97C3-48BB-9F95-01720DFB769C@yahoo-inc.com>
	<4E054E1A.2090809@uci.cu>
	<BANLkTik5iE_y8FM8nUVJqdcS8tnBpx+Bvg@mail.gmail.com>
	<4E05FC0B.2070901@uci.cu>
Date: Sat, 25 Jun 2011 09:08:46 -0700
Message-ID: <BANLkTikwzrmNvHhBpqtDc-Dj4tGbBLaWLQ@mail.gmail.com>
Subject: =?ISO-8859-1?Q?Re=3A_Hadoop=B4s_Internationalization?=
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec52e66d96daa0304a68b8879

--bcaec52e66d96daa0304a68b8879
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Marcos:
Which hadoop version(s) do you plan to work on ?

>> Where I can find the sources of the docs?

In the latest TRUNK, you would find these directories:

./common/src/docs
./hdfs/src/c++/libhdfs/docs
./hdfs/src/docs
./mapreduce/src/docs

Cheers

On Sat, Jun 25, 2011 at 8:17 AM, Marcos Ortiz <mlortiz@uci.cu> wrote:

> OK, Owen, Where I can find the sources of the docs?
> Which is the format for the docs? DocBook, ReST, etc?
>
> El 6/25/2011 4:38 AM, Owen O'Malley escribi=F3:
>
>  On Fri, Jun 24, 2011 at 7:55 PM, Marcos Ortiz<mlortiz@uci.cu>  wrote:
>>
>>
>>
>>> Regards to all the list.
>>> I=B4m looking for a proper way to work on the internationalization of
>>> Hadoop,
>>> but I don=B4t know if this is a good project
>>> or if this is useful for the community, at least, I think that it would
>>> be
>>> very useful for many people that want to see the messages
>>> of the project in another language, for example, in Spanish.
>>>
>>>
>>
>> I think it would be very useful, but very time consuming project. I'd
>> suggest that you start by translating the documentation for the release
>> into
>> another language first.
>>
>> -- Owen
>>
>>
>>
>
> --
> Marcos Lu=EDs Ort=EDz Valmaseda
>  Software Engineer (UCI)
>  http://marcosluis2186.**posterous.com<http://marcosluis2186.posterous.co=
m>
>  http://twitter.com/**marcosluis2186 <http://twitter.com/marcosluis2186>
>
>

--bcaec52e66d96daa0304a68b8879--

From general-return-3948-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 26 07:07:23 2011
Return-Path: <general-return-3948-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4F7A265DE
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sun, 26 Jun 2011 07:07:23 +0000 (UTC)
Received: (qmail 36108 invoked by uid 500); 26 Jun 2011 07:07:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34943 invoked by uid 500); 26 Jun 2011 07:06:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34923 invoked by uid 99); 26 Jun 2011 07:06:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 07:06:44 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 07:06:38 +0000
Received: by yxd5 with SMTP id 5so2419964yxd.35
        for <general@hadoop.apache.org>; Sun, 26 Jun 2011 00:06:16 -0700 (PDT)
Received: by 10.236.207.166 with SMTP id n26mr7585848yho.513.1309071976166;
 Sun, 26 Jun 2011 00:06:16 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.147.35.11 with HTTP; Sun, 26 Jun 2011 00:05:56 -0700 (PDT)
In-Reply-To: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
From: Eran Kampf <eran@ekampf.com>
Date: Sun, 26 Jun 2011 10:05:56 +0300
Message-ID: <BANLkTinjiJ=X3Bs0croPigTw9=fGVxABmQ@mail.gmail.com>
Subject: Re: [VOTE] Powered by Logo
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd60afa1ff5a404a69812e1
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd60afa1ff5a404a69812e1
Content-Type: text/plain; charset=ISO-8859-1

My vote: 5 6 4 2 1 3

- Eran

On Wed, Jun 15, 2011 at 6:19 AM, Owen O'Malley <omalley@apache.org> wrote:

> All,
>   We've had a wide range of entries for a powered by logo. I've put them
> all on a page, here:
>
> http://people.apache.org/~omalley/hadoop-powered-by/
>
> Since there are a lot of contenders and we only want a single round of
> voting, let's use single transferable vote ( STV
> http://en.wikipedia.org/wiki/Single_transferable_vote). The important
> thing is to pick the images *IN ORDER* that you would like them.
>
> My vote (in order of course): 4, 1, 2, 3, 5, 6.
>
> In other words, I want option 4 most and option 6 least. With STV, you
> don't need to worry about voting for an unpopular choice since your vote
> will automatically roll over to your next choice.
>
> -- Owen
>
>
>

--000e0cd60afa1ff5a404a69812e1--

From general-return-3949-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 26 19:11:49 2011
Return-Path: <general-return-3949-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5D68A6EB1
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sun, 26 Jun 2011 19:11:49 +0000 (UTC)
Received: (qmail 45860 invoked by uid 500); 26 Jun 2011 19:11:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45758 invoked by uid 500); 26 Jun 2011 19:11:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45750 invoked by uid 99); 26 Jun 2011 19:11:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 19:11:46 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [208.97.132.81] (HELO homiemail-a63.g.dreamhost.com) (208.97.132.81)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 19:11:40 +0000
Received: from homiemail-a63.g.dreamhost.com (localhost [127.0.0.1])
	by homiemail-a63.g.dreamhost.com (Postfix) with ESMTP id 242AE2F4059
	for <general@hadoop.apache.org>; Sun, 26 Jun 2011 12:11:20 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; q=dns; s=
	gbiv.com; b=Q78zWU9H6sFEPh+zey3Ksriaz24GuCJSc4GN9XYXOYWYHrnxX+3n
	2LNyYIjNCqhSSV491ih68t1ruflt33dfSK6lf9ssdE73Y3y/01aciJr4lPhMBJip
	o9yuc82PXJbNiZJONNA3jOusHcV7ewObVJQjJffGIMFtWsUXK7pI7GA=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; s=gbiv.com;
	 bh=TeclYzF9l/P4a1mW7N8phQXYZeQ=; b=qNrvjQUPIz12mkSr3QGE29RaxoxY
	b+Q93PYuhrOBrBIIoXzKH2cUxNdiVia7Nfp8Kvwl3HSr2SPq+0Oq4LTyVeQuw9/j
	XGdPguMJfL9vW/4DMjRdTZ0e4eChnJAtZTuoETrKyLfNFSUgdeEtDTZhuCAhsc4H
	gg2+3Pv1rSQV9Ek=
Received: from [192.168.1.84] (99-21-208-82.lightspeed.irvnca.sbcglobal.net [99.21.208.82])
	(using TLSv1 with cipher AES128-SHA (128/128 bits))
	(No client certificate requested)
	(Authenticated sender: fielding@gbiv.com)
	by homiemail-a63.g.dreamhost.com (Postfix) with ESMTPSA id E988B2F4057
	for <general@hadoop.apache.org>; Sun, 26 Jun 2011 12:11:19 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: [VOTE] Shall we adopt the "Defining Hadoop" page
From: "Roy T. Fielding" <fielding@gbiv.com>
In-Reply-To: <4E056D3B.9030008@apache.org>
Date: Sun, 26 Jun 2011 12:11:19 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <16935C45-FE78-418E-9999-EDD1ECC3257D@gbiv.com>
References: <A3383C10-E615-434D-A9E1-8C444827596D@apache.org> <C62A9D62-B5DA-42C7-8D00-57C2126B51C2@apache.org> <4E04948B.4050802@apache.org> <C3BF2AA2-B17C-4E9E-8BDE-D0776B8B8435@apache.org> <4E056D3B.9030008@apache.org>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)

On Jun 24, 2011, at 10:08 PM, Doug Cutting wrote:

> On 06/24/2011 07:07 PM, Owen O'Malley wrote:
>> On Jun 24, 2011, at 6:43 AM, Doug Cutting wrote:
>>=20
>>> Might it be better to improve the existing Apache trademark policy
>>> page?
>>=20
>> When the project is having trouble agreeing, reaching agreement at
>> the foundation level seems unrealistic.
>=20
> ASF trademark policy is set by Shane, VP Trademark, not by a =
committee.

If we apply trademark policy to this discussion, then the only answer =
possible
is that only releases made by the Apache Hadoop PMC can be called =
Hadoop.
That is, after all, the essence of board delegation to PMCs and the =
meaning
of trademarks.

Traditionally, we have also allowed distributions that apply
released security patches, for example as found in=20

   http://www.apache.org/dist/httpd/patches/

and turned a blind eye toward changes that are purely to port to a new =
platform.
I did not write those exceptions down because I don't know what (if any) =
impact
they might have on enforcement.

I said before that we typically don't argue about distributions that =
include
revisions that are on a release branch, but that assumed the project is
actually working toward a release of that branch.  I have a hard time =
believing
that Hadoop's trunk is a release branch.  In any case, this very =
specific
exception should be entirely decided by the project -- the VP of =
Trademarks
has no role in deciding what is the purview of each PMC, namely the =
decision
on what is or is not released in the name of that project.

....Roy


From general-return-3950-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 26 19:23:28 2011
Return-Path: <general-return-3950-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 38C2265A0
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sun, 26 Jun 2011 19:23:28 +0000 (UTC)
Received: (qmail 55813 invoked by uid 500); 26 Jun 2011 19:23:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55757 invoked by uid 500); 26 Jun 2011 19:23:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55747 invoked by uid 99); 26 Jun 2011 19:23:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 19:23:25 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of scott@richrelevance.com designates 64.78.17.17 as permitted sender)
Received: from [64.78.17.17] (HELO EXHUB018-2.exch018.msoutlookonline.net) (64.78.17.17)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 19:23:16 +0000
Received: from EXVMBX018-1.exch018.msoutlookonline.net ([64.78.17.47]) by
 EXHUB018-2.exch018.msoutlookonline.net ([64.78.17.17]) with mapi; Sun, 26 Jun
 2011 12:22:55 -0700
From: Scott Carey <scott@richrelevance.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Sun, 26 Jun 2011 12:23:45 -0700
Subject: Re: Hadoop Java Versions
Thread-Topic: Hadoop Java Versions
Thread-Index: Acw0NnNxOcpMJme1Sdi1qjHjpJzJZg==
Message-ID: <CA2CD1F8.3F91C%scott@richrelevance.com>
In-Reply-To: <4E033676.8030205@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-MacOutlook/14.10.0.110310
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org



On 6/23/11 5:49 AM, "Steve Loughran" <stevel@apache.org> wrote:

>On 22/06/2011 21:27, Scott Carey wrote:
>> "Problems have been reported with Hadoop, the 64-bit JVM and Compressed
>> Object References (the -XX:+UseCompressedOops option), so use of that
>> option is discouraged."
>>
>> I think the above is dated.  It also lacks critical information. What
>>JVM
>> and OS version was the problem seen?
>
>A colleague saw it, intermittent JVM crashes. Unless he's updated I
>can't say the problem has gone away.
>
>>
>> CompressedOops had several issues prior to Jre 6u20, and a few minor
>>ones
>> were fixed in u21.  FWIW, I now exclusively use 64 bit w/ CompressedOops
>> for all Hadoop and non-Hadoop apps and have seen no issues.  It is the
>> default in 6u24 and 6u25 on a 64 bit  JVM.
>
>
>what's your HW setup? #cores/server, #servers, underlying OS?

CentOS 5.6. =20
4 cores / 8 threads a server (Nehalem generation Intel processor).

Also run a smaller cluster with 2x quad core Core 2 generation Xeons.

Off topic:
The single proc Nehalem is faster than the dual core 2's for most use
cases -- and much lower power.  Looking forward to single proc 4 or 6 core
Sandy Bridge based systems for the next expansion -- testing 4 core vs 4
core has these 30% faster than the Nehalem generation systems in CPU bound
tasks and lower power.  Intel prices single socket Xeons so much lower
than the Dual socket ones that the best value for us is to get more single
socket servers rather than fewer dual socket ones (with similar processor
to hard drive ratio).  We are power constrained per rack either way.


From general-return-3951-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 26 23:15:33 2011
Return-Path: <general-return-3951-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5216967C0
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sun, 26 Jun 2011 23:15:33 +0000 (UTC)
Received: (qmail 95155 invoked by uid 500); 26 Jun 2011 23:15:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95080 invoked by uid 500); 26 Jun 2011 23:15:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95072 invoked by uid 99); 26 Jun 2011 23:15:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 23:15:30 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.220.176] (HELO mail-vx0-f176.google.com) (209.85.220.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 23:15:24 +0000
Received: by vxd2 with SMTP id 2so4935305vxd.35
        for <general@hadoop.apache.org>; Sun, 26 Jun 2011 16:15:03 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=qa3xAPu5mPIqboULBAeFUns6v6zZcTt+nc9OEEtTm4o=;
        b=D1n44csPIFBbRvB5A8z47BZfsplmt4A2D/B29w5BglS2tRvsOgBzXWT4R7cw31ZdtW
         XwBZuTTtPD4zgU9LMaN4Buoxq1gVWiys0oETSHPawBaoqbsehKue1Q1Ilq5SYPt8TxnO
         nSmvXqjYwdmkJa6sTyltSJ23WZr7PFz+ZOYxE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=QhLcbYQN/+j6RPNuGtNBY+A6rkTG+g05TRc4D0GwhAYjH1D/43uZuj1CoquvvHPOvb
         2zouf81R6Q+M0qd7RlGUSEEvVnGr2cjXXPJ75yFfEg8Pt7MjNZMrNDhT6qhoXFFzilMt
         gQgBretZvPnDrvrOtuoZbf94rwf81oZ5XWTtI=
Received: by 10.52.98.39 with SMTP id ef7mr7640515vdb.88.1309130103467;
        Sun, 26 Jun 2011 16:15:03 -0700 (PDT)
Received: from [10.172.0.74] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id ck16sm1370968vdb.44.2011.06.26.16.15.01
        (version=TLSv1/SSLv3 cipher=OTHER);
        Sun, 26 Jun 2011 16:15:02 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: [RESULT] Powered by Logo
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <20110625034112.GA14500@linspire.com>
Date: Mon, 27 Jun 2011 09:14:57 +1000
Content-Transfer-Encoding: quoted-printable
Message-Id: <F4A7BBFA-B3DA-43F6-86C0-E863964FFE00@Holsman.NET>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org> <BAB91A74-E0CC-4A65-A0AC-4186014E473F@apache.org> <20110625034112.GA14500@linspire.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)

Is there a reason we can't have 2 powered by logo's that vendors etc can =
choose from?

On Jun 25, 2011, at 1:41 PM, Konstantin Boudnik wrote:

> Owen,
>=20
> looks like your non-PMC list doesn't account for about 13 or so user =
votes
> (which we cast under different subjects and making total number of #4 =
equals
> 37) as you can see here http://tinyurl.com/3j6629h
>=20
> Hope it helps
>  Cos
>=20
> On Fri, Jun 24, 2011 at 12:35AM, Owen O'Malley wrote:
>> Ok,
>>   I collated the votes filtering on the subject line. I know that =
method missed some of the non-binding votes, but I don't believe it =
changed the outcome.
>>=20
>> With the binding votes from the PMC:
>>=20
>>=20
>>=20
>> First Round=09
>> 2	4
>> 4	4
>> 5	6
>> 6	1
>> Second Round (Drop 6)
>> 2	5
>> 4	4
>> 5	6
>> Third Round (Drop 4)
>> 2	8
>> 5	7
>>=20
>>=20
>> So in the very close race, #2 wins. My spreadsheets, which are =
attached below, show the counted votes.
>>=20
>> With all of the binding and non-binding votes:
>>=20
>>=20
>>=20
>> First Round=09
>> 1	1
>> 2	13
>> 4	23
>> 5	27
>> 6	4
>> Second Round(Drop 1)
>> 2	13
>> 4	23
>> 5	27
>> 6	5
>> Third Round (Drop 6)
>> 2	15
>> 4	24
>> 5	29
>> Forth Round (Drop 2)
>> 4	35
>> 5	31
>>=20
>>=20
>> -- Owen
>>=20


From general-return-3952-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 26 23:24:21 2011
Return-Path: <general-return-3952-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7C2046CED
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sun, 26 Jun 2011 23:24:21 +0000 (UTC)
Received: (qmail 3833 invoked by uid 500); 26 Jun 2011 23:24:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3712 invoked by uid 500); 26 Jun 2011 23:24:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3704 invoked by uid 99); 26 Jun 2011 23:24:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 23:24:19 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.220.176] (HELO mail-vx0-f176.google.com) (209.85.220.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 23:24:13 +0000
Received: by vxd2 with SMTP id 2so4938818vxd.35
        for <general@hadoop.apache.org>; Sun, 26 Jun 2011 16:23:52 -0700 (PDT)
Received: by 10.52.175.133 with SMTP id ca5mr7740478vdc.82.1309130632055; Sun,
 26 Jun 2011 16:23:52 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.52.109.8 with HTTP; Sun, 26 Jun 2011 16:23:32 -0700 (PDT)
X-Originating-IP: [67.160.196.149]
In-Reply-To: <F4A7BBFA-B3DA-43F6-86C0-E863964FFE00@Holsman.NET>
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org>
 <BAB91A74-E0CC-4A65-A0AC-4186014E473F@apache.org> <20110625034112.GA14500@linspire.com>
 <F4A7BBFA-B3DA-43F6-86C0-E863964FFE00@Holsman.NET>
From: Ted Dunning <tdunning@maprtech.com>
Date: Sun, 26 Jun 2011 16:23:32 -0700
Message-ID: <BANLkTinJgzmsyJjYTHMgzwHg5kSD2pq2Ng@mail.gmail.com>
Subject: Re: [RESULT] Powered by Logo
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec5014c6b49cb4004a6a5ba52

--bcaec5014c6b49cb4004a6a5ba52
Content-Type: text/plain; charset=ISO-8859-1

I would propose that the core be specified with some look and feel
flexibility.

Logo 2, for instance, is basically the same logo as log 4 except for font
changes and the circle.  If we consider the circle to be totally optional
and intended just to make the logo fit in with the rest of the site, then
the basic idea of *that* yellow elephant with *those* words can be the
definition.

How does that sound?  I am happy to have an artist produce some variants on
logo 2 that have

a) larger font

b) various sizes

Would that be useful?

On Sun, Jun 26, 2011 at 4:14 PM, Ian Holsman <hadoop@holsman.net> wrote:

> Is there a reason we can't have 2 powered by logo's that vendors etc can
> choose from?
>
> On Jun 25, 2011, at 1:41 PM, Konstantin Boudnik wrote:
>
> > Owen,
> >
> > looks like your non-PMC list doesn't account for about 13 or so user
> votes
> > (which we cast under different subjects and making total number of #4
> equals
> > 37) as you can see here http://tinyurl.com/3j6629h
> >
> > Hope it helps
> >  Cos
> >
> > On Fri, Jun 24, 2011 at 12:35AM, Owen O'Malley wrote:
> >> Ok,
> >>   I collated the votes filtering on the subject line. I know that method
> missed some of the non-binding votes, but I don't believe it changed the
> outcome.
> >>
> >> With the binding votes from the PMC:
> >>
> >>
> >>
> >> First Round
> >> 2    4
> >> 4    4
> >> 5    6
> >> 6    1
> >> Second Round (Drop 6)
> >> 2    5
> >> 4    4
> >> 5    6
> >> Third Round (Drop 4)
> >> 2    8
> >> 5    7
> >>
> >>
> >> So in the very close race, #2 wins. My spreadsheets, which are attached
> below, show the counted votes.
> >>
> >> With all of the binding and non-binding votes:
> >>
> >>
> >>
> >> First Round
> >> 1    1
> >> 2    13
> >> 4    23
> >> 5    27
> >> 6    4
> >> Second Round(Drop 1)
> >> 2    13
> >> 4    23
> >> 5    27
> >> 6    5
> >> Third Round (Drop 6)
> >> 2    15
> >> 4    24
> >> 5    29
> >> Forth Round (Drop 2)
> >> 4    35
> >> 5    31
> >>
> >>
> >> -- Owen
> >>
>
>

--bcaec5014c6b49cb4004a6a5ba52--

From general-return-3953-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 27 11:31:36 2011
Return-Path: <general-return-3953-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1E6BF406E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 27 Jun 2011 11:31:36 +0000 (UTC)
Received: (qmail 79488 invoked by uid 500); 27 Jun 2011 11:31:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79372 invoked by uid 500); 27 Jun 2011 11:31:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 79363 invoked by uid 99); 27 Jun 2011 11:31:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 11:31:32 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 11:31:24 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 15582B7E87
	for <general@hadoop.apache.org>; Mon, 27 Jun 2011 12:31:02 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id GhyLr+P-slqR for <general@hadoop.apache.org>;
	Mon, 27 Jun 2011 12:31:02 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 1EB7DB7E82
	for <general@hadoop.apache.org>; Mon, 27 Jun 2011 12:31:01 +0100 (BST)
MailScanner-NULL-Check: 1309779048.01447@zXb4FLMRFxkHGAkm55Fx7g
Received: from [16.25.175.86] (wildhaus.hpl.hp.com [16.25.175.86])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p5RBUkcw029373
	for <general@hadoop.apache.org>; Mon, 27 Jun 2011 12:30:46 +0100 (BST)
Message-ID: <4E0869E6.1000505@apache.org>
Date: Mon, 27 Jun 2011 12:30:46 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [RESULT] Powered by Logo
References: <8F5720E1-00D0-4352-B9D4-76C51C48F184@apache.org> <BAB91A74-E0CC-4A65-A0AC-4186014E473F@apache.org> <BANLkTin_-0_wPuUY1Br9Ac=QsFEiYeSBkw@mail.gmail.com> <BANLkTik2Gezh3y1vMQD7x=tGheuet=4MRQ@mail.gmail.com> <420C1431-457C-4D2E-B3DB-618C9B69D8CC@apache.org>
In-Reply-To: <420C1431-457C-4D2E-B3DB-618C9B69D8CC@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p5RBUkcw029373
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 24/06/11 17:54, Allen Wittenauer wrote:
>
> On Jun 24, 2011, at 9:47 AM, Eli Collins wrote:
>>
>> Off list I said I thought any of the logos were acceptable choices
>> modulo some design tweaks, and that if the circle logo concept is
>> accepted it would be good to adjust the circle itself.  But the PMC
>> did not vote for the circle logo, so it's a mute point.
>
> Clearly the PMC is full of circle-phobes.
>

it's why I work in a cube

From general-return-3954-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 27 11:39:36 2011
Return-Path: <general-return-3954-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5990B40B9
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 27 Jun 2011 11:39:36 +0000 (UTC)
Received: (qmail 93750 invoked by uid 500); 27 Jun 2011 11:39:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93684 invoked by uid 500); 27 Jun 2011 11:39:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93676 invoked by uid 99); 27 Jun 2011 11:39:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 11:39:33 +0000
X-ASF-Spam-Status: No, hits=-1.4 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL,TO_NO_BRKTS_PCNT
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 11:39:24 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id C163BB7EE1
	for <general@hadoop.apache.org>; Mon, 27 Jun 2011 12:39:03 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id Rh2bhkh+FE+S for <general@hadoop.apache.org>;
	Mon, 27 Jun 2011 12:39:03 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by tobor.hpl.hp.com (Postfix) with ESMTP id CC2CFB7ECA
	for <general@hadoop.apache.org>; Mon, 27 Jun 2011 12:39:02 +0100 (BST)
MailScanner-NULL-Check: 1309779528.15627@kiBsdPPPAEWJuKctw/bh/A
Received: from [16.25.175.86] (wildhaus.hpl.hp.com [16.25.175.86])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p5RBclCU029793
	for <general@hadoop.apache.org>; Mon, 27 Jun 2011 12:38:47 +0100 (BST)
Message-ID: <4E086BC7.5000208@apache.org>
Date: Mon, 27 Jun 2011 12:38:47 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Hadoop Java Versions
References: <CA2CD1F8.3F91C%scott@richrelevance.com>
In-Reply-To: <CA2CD1F8.3F91C%scott@richrelevance.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p5RBclCU029793
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 26/06/11 20:23, Scott Carey wrote:
>
>
> On 6/23/11 5:49 AM, "Steve Loughran"<stevel@apache.org>  wrote:
>

>> what's your HW setup? #cores/server, #servers, underlying OS?
>
> CentOS 5.6.
> 4 cores / 8 threads a server (Nehalem generation Intel processor).


that should be enough to find problems. I've just moved up to a 6-core 
12 thread desktop and that found problems on some non-Hadoop code, which 
shows that the more threads you have, and the faster the machines are, 
the more your race conditions show up. With Hadoop the fact that you can 
have 10-1000 servers means that in a large cluster the probability of 
that race condition showing up scales well.

> Also run a smaller cluster with 2x quad core Core 2 generation Xeons.
>
> Off topic:
> The single proc Nehalem is faster than the dual core 2's for most use
> cases -- and much lower power.  Looking forward to single proc 4 or 6 core
> Sandy Bridge based systems for the next expansion -- testing 4 core vs 4
> core has these 30% faster than the Nehalem generation systems in CPU bound
> tasks and lower power.  Intel prices single socket Xeons so much lower
> than the Dual socket ones that the best value for us is to get more single
> socket servers rather than fewer dual socket ones (with similar processor
> to hard drive ratio).

Yes, in a large cluster the price of filling the second socket can 
compare to a lot of storage, and TB of storage is more tangible. I guess 
it depends on your application.

Regarding Sandy Bridge, I've no experience of those, but I worry that 10 
Gbps is still bleeding edge, and shouldn't be needed for code with good 
locality anyway; it is probably more cost effective to stay at 
1Gbps/server, though the issue there is the #of HDD/s server generates 
lots of replication traffic when a single server fails...

From general-return-3955-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 00:11:15 2011
Return-Path: <general-return-3955-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6E2B240CD
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 28 Jun 2011 00:11:15 +0000 (UTC)
Received: (qmail 81875 invoked by uid 500); 28 Jun 2011 00:11:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81812 invoked by uid 500); 28 Jun 2011 00:11:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81804 invoked by uid 99); 28 Jun 2011 00:11:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 00:11:12 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ryanobjc@gmail.com designates 209.85.161.51 as permitted sender)
Received: from [209.85.161.51] (HELO mail-fx0-f51.google.com) (209.85.161.51)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 00:11:05 +0000
Received: by fxh10 with SMTP id 10so2460176fxh.38
        for <general@hadoop.apache.org>; Mon, 27 Jun 2011 17:10:44 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type:content-transfer-encoding;
        bh=GvJjmTJVAZw74GyHNXhyZ+QT+e5kAT0nNQfajKmRThM=;
        b=QMEHXa/s6IeCr8EHiDqGkzvkH78mkCFj8Y/NiMS/2Zmu4i7XUfLYu6oGvRaaAv4jPe
         Re6q5FwyHsM7HPM+OwbpeE4YgcJxUcTTfQkN6YG0oPaFVNgZwxI1rsHCTRiGn58BpHI+
         mYPN87EF0XMB0ipN0XSIFjQ6RfVZqd+AmvEs0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=MnufmwaWIDdeyugGg5Z8KUniJF28+vfGiLf6HJqoOBx4cW9SkgAmCb+IQOTqGWR12q
         1ssupgoidfPJgkyUrczp7H1q6VEDyUgtJIORkBN3it/XZlrYFO6ym548OYzGAQ3bIAtM
         SsRmwXia0YDBubni3O65dPluBvY+Hj2lIkH9E=
MIME-Version: 1.0
Received: by 10.223.100.197 with SMTP id z5mr5308979fan.46.1309219844537; Mon,
 27 Jun 2011 17:10:44 -0700 (PDT)
Received: by 10.223.81.78 with HTTP; Mon, 27 Jun 2011 17:10:44 -0700 (PDT)
In-Reply-To: <4E086BC7.5000208@apache.org>
References: <CA2CD1F8.3F91C%scott@richrelevance.com>
	<4E086BC7.5000208@apache.org>
Date: Mon, 27 Jun 2011 17:10:44 -0700
Message-ID: <BANLkTimZ4SpW6wEGCGT=g6SfMXq4g2UaWw@mail.gmail.com>
Subject: Re: Hadoop Java Versions
From: Ryan Rawson <ryanobjc@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On the subject of gige vs 10-gige, I think that we will very shortly
be seeing interest in 10gig, since gige is only 120MB/sec - 1 hard
drive of streaming data.  Nodes with 4+ disks are throttled by the
network.  On a small cluster (20 nodes), the replication traffic can
choke a cluster to death.  The only way to fix quickly it is to bring
that node back up.  Perhaps the HortonWorks guys can work on that.

-ryan

On Mon, Jun 27, 2011 at 4:38 AM, Steve Loughran <stevel@apache.org> wrote:
> On 26/06/11 20:23, Scott Carey wrote:
>>
>>
>> On 6/23/11 5:49 AM, "Steve Loughran"<stevel@apache.org> =A0wrote:
>>
>
>>> what's your HW setup? #cores/server, #servers, underlying OS?
>>
>> CentOS 5.6.
>> 4 cores / 8 threads a server (Nehalem generation Intel processor).
>
>
> that should be enough to find problems. I've just moved up to a 6-core 12
> thread desktop and that found problems on some non-Hadoop code, which sho=
ws
> that the more threads you have, and the faster the machines are, the more
> your race conditions show up. With Hadoop the fact that you can have 10-1=
000
> servers means that in a large cluster the probability of that race condit=
ion
> showing up scales well.
>
>> Also run a smaller cluster with 2x quad core Core 2 generation Xeons.
>>
>> Off topic:
>> The single proc Nehalem is faster than the dual core 2's for most use
>> cases -- and much lower power. =A0Looking forward to single proc 4 or 6 =
core
>> Sandy Bridge based systems for the next expansion -- testing 4 core vs 4
>> core has these 30% faster than the Nehalem generation systems in CPU bou=
nd
>> tasks and lower power. =A0Intel prices single socket Xeons so much lower
>> than the Dual socket ones that the best value for us is to get more sing=
le
>> socket servers rather than fewer dual socket ones (with similar processo=
r
>> to hard drive ratio).
>
> Yes, in a large cluster the price of filling the second socket can compar=
e
> to a lot of storage, and TB of storage is more tangible. I guess it depen=
ds
> on your application.
>
> Regarding Sandy Bridge, I've no experience of those, but I worry that 10
> Gbps is still bleeding edge, and shouldn't be needed for code with good
> locality anyway; it is probably more cost effective to stay at 1Gbps/serv=
er,
> though the issue there is the #of HDD/s server generates lots of replicat=
ion
> traffic when a single server fails...
>

From general-return-3956-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 00:13:09 2011
Return-Path: <general-return-3956-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D7C2A4106
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 28 Jun 2011 00:13:09 +0000 (UTC)
Received: (qmail 83757 invoked by uid 500); 28 Jun 2011 00:13:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83689 invoked by uid 500); 28 Jun 2011 00:13:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83681 invoked by uid 99); 28 Jun 2011 00:13:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 00:13:07 +0000
X-ASF-Spam-Status: No, hits=2.4 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL,TO_NO_BRKTS_PCNT
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 00:12:59 +0000
Received: by vws7 with SMTP id 7so6097499vws.35
        for <general@hadoop.apache.org>; Mon, 27 Jun 2011 17:12:38 -0700 (PDT)
Received: by 10.52.176.74 with SMTP id cg10mr9498775vdc.242.1309219958095;
 Mon, 27 Jun 2011 17:12:38 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.52.109.8 with HTTP; Mon, 27 Jun 2011 17:12:18 -0700 (PDT)
X-Originating-IP: [64.105.168.204]
In-Reply-To: <BANLkTimZ4SpW6wEGCGT=g6SfMXq4g2UaWw@mail.gmail.com>
References: <CA2CD1F8.3F91C%scott@richrelevance.com> <4E086BC7.5000208@apache.org>
 <BANLkTimZ4SpW6wEGCGT=g6SfMXq4g2UaWw@mail.gmail.com>
From: Ted Dunning <tdunning@maprtech.com>
Date: Mon, 27 Jun 2011 17:12:18 -0700
Message-ID: <BANLkTik_5mYOaKA08-G6uRM9e3w+S8+_Tg@mail.gmail.com>
Subject: Re: Hadoop Java Versions
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf307f34c28900e904a6ba863d

--20cf307f34c28900e904a6ba863d
Content-Type: text/plain; charset=ISO-8859-1

Come to Srivas talk at the Summit.

On Mon, Jun 27, 2011 at 5:10 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:

> On the subject of gige vs 10-gige, I think that we will very shortly
> be seeing interest in 10gig, since gige is only 120MB/sec - 1 hard
> drive of streaming data.  Nodes with 4+ disks are throttled by the
> network.  On a small cluster (20 nodes), the replication traffic can
> choke a cluster to death.  The only way to fix quickly it is to bring
> that node back up.  Perhaps the HortonWorks guys can work on that.
>
> -ryan
>
> On Mon, Jun 27, 2011 at 4:38 AM, Steve Loughran <stevel@apache.org> wrote:
> > On 26/06/11 20:23, Scott Carey wrote:
> >>
> >>
> >> On 6/23/11 5:49 AM, "Steve Loughran"<stevel@apache.org>  wrote:
> >>
> >
> >>> what's your HW setup? #cores/server, #servers, underlying OS?
> >>
> >> CentOS 5.6.
> >> 4 cores / 8 threads a server (Nehalem generation Intel processor).
> >
> >
> > that should be enough to find problems. I've just moved up to a 6-core 12
> > thread desktop and that found problems on some non-Hadoop code, which
> shows
> > that the more threads you have, and the faster the machines are, the more
> > your race conditions show up. With Hadoop the fact that you can have
> 10-1000
> > servers means that in a large cluster the probability of that race
> condition
> > showing up scales well.
> >
> >> Also run a smaller cluster with 2x quad core Core 2 generation Xeons.
> >>
> >> Off topic:
> >> The single proc Nehalem is faster than the dual core 2's for most use
> >> cases -- and much lower power.  Looking forward to single proc 4 or 6
> core
> >> Sandy Bridge based systems for the next expansion -- testing 4 core vs 4
> >> core has these 30% faster than the Nehalem generation systems in CPU
> bound
> >> tasks and lower power.  Intel prices single socket Xeons so much lower
> >> than the Dual socket ones that the best value for us is to get more
> single
> >> socket servers rather than fewer dual socket ones (with similar
> processor
> >> to hard drive ratio).
> >
> > Yes, in a large cluster the price of filling the second socket can
> compare
> > to a lot of storage, and TB of storage is more tangible. I guess it
> depends
> > on your application.
> >
> > Regarding Sandy Bridge, I've no experience of those, but I worry that 10
> > Gbps is still bleeding edge, and shouldn't be needed for code with good
> > locality anyway; it is probably more cost effective to stay at
> 1Gbps/server,
> > though the issue there is the #of HDD/s server generates lots of
> replication
> > traffic when a single server fails...
> >
>

--20cf307f34c28900e904a6ba863d--

From general-return-3957-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 01:53:57 2011
Return-Path: <general-return-3957-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 426414F6F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 28 Jun 2011 01:53:57 +0000 (UTC)
Received: (qmail 61014 invoked by uid 500); 28 Jun 2011 01:53:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60950 invoked by uid 500); 28 Jun 2011 01:53:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60942 invoked by uid 99); 28 Jun 2011 01:53:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 01:53:54 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of msegel@navteq.com designates 65.167.11.240 as permitted sender)
Received: from [65.167.11.240] (HELO xmailchi.navteq.com) (65.167.11.240)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 01:53:48 +0000
Received: from chicago-spamfw.navteq.com (imailchi [10.8.5.161])
	by xmailchi.navteq.com (8.13.8/8.13.6) with ESMTP id p5S1rREo002124
	for <general@hadoop.apache.org>; Mon, 27 Jun 2011 20:53:27 -0500
Received: from imailchi.hq.navteq.com (localhost [127.0.0.1])
	by chicago-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id 8CBEA3EED8DB
	for <general@hadoop.apache.org>; Mon, 27 Jun 2011 20:53:27 -0500 (CDT)
Received: from imailchi.hq.navteq.com (imailchi.hq.navteq.com [10.8.120.104]) by chicago-spamfw.navteq.com with ESMTP id uVEuJup1nwnFcXaE for <general@hadoop.apache.org>; Mon, 27 Jun 2011 20:53:27 -0500 (CDT)
Received: from hq-ex-ht02.ad.navteq.com (hq-ex-ht02.ad.navteq.com [10.8.222.172])
	by imailchi.hq.navteq.com (8.12.9/8.12.9) with ESMTP id p5S1rRNe006096;
	Mon, 27 Jun 2011 20:53:27 -0500
Received: from hq-ex-mb03.ad.navteq.com ([fe80::c4dd:7b21:5c22:cfe4]) by
 hq-ex-ht02.ad.navteq.com ([fe80::24f6:ca50:734d:df6e%13]) with mapi; Mon, 27
 Jun 2011 20:53:26 -0500
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 27 Jun 2011 20:54:13 -0500
Subject: Re: Hadoop Java Versions
Thread-Topic: Hadoop Java Versions
Thread-Index: Acw1NitudJPPhQAbQm6NvvTRwMCcPA==
Message-ID: <9068C5A7-6090-42A3-8891-F60EF3B660BD@navteq.com>
References: <CA2CD1F8.3F91C%scott@richrelevance.com>
 <4E086BC7.5000208@apache.org>
 <BANLkTimZ4SpW6wEGCGT=g6SfMXq4g2UaWw@mail.gmail.com>
In-Reply-To: <BANLkTimZ4SpW6wEGCGT=g6SfMXq4g2UaWw@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1

That doesn't seem right.
In one of our test clusters (19 data nodes) we found that under heavy loa=
ds we were disk I/O bound and not network bound. Of course YMMV depending=
 on your ToR switch. If we had more than 4 disks per node, we would proba=
bly see the network being the bottleneck. What did you set your bandwidth=
 settings in the hdfs-site.xml? ( going from memory not sure of the exact=
 setting...)

But the good news... Newer hardware will start to have 10GBe on the mothe=
rboard.

Sent from a remote device. Please excuse any typos...

Mike Segel

On Jun 27, 2011, at 7:11 PM, "Ryan Rawson" <ryanobjc@gmail.com> wrote:

> On the subject of gige vs 10-gige, I think that we will very shortly
> be seeing interest in 10gig, since gige is only 120MB/sec - 1 hard
> drive of streaming data.  Nodes with 4+ disks are throttled by the
> network.  On a small cluster (20 nodes), the replication traffic can
> choke a cluster to death.  The only way to fix quickly it is to bring
> that node back up.  Perhaps the HortonWorks guys can work on that.
>=20
> -ryan
>=20
> On Mon, Jun 27, 2011 at 4:38 AM, Steve Loughran <stevel@apache.org> wro=
te:
>> On 26/06/11 20:23, Scott Carey wrote:
>>>=20
>>>=20
>>> On 6/23/11 5:49 AM, "Steve Loughran"<stevel@apache.org>  wrote:
>>>=20
>>=20
>>>> what's your HW setup? #cores/server, #servers, underlying OS?
>>>=20
>>> CentOS 5.6.
>>> 4 cores / 8 threads a server (Nehalem generation Intel processor).
>>=20
>>=20
>> that should be enough to find problems. I've just moved up to a 6-core=
 12
>> thread desktop and that found problems on some non-Hadoop code, which =
shows
>> that the more threads you have, and the faster the machines are, the m=
ore
>> your race conditions show up. With Hadoop the fact that you can have 1=
0-1000
>> servers means that in a large cluster the probability of that race con=
dition
>> showing up scales well.
>>=20
>>> Also run a smaller cluster with 2x quad core Core 2 generation Xeons.
>>>=20
>>> Off topic:
>>> The single proc Nehalem is faster than the dual core 2's for most use
>>> cases -- and much lower power.  Looking forward to single proc 4 or 6=
 core
>>> Sandy Bridge based systems for the next expansion -- testing 4 core v=
s 4
>>> core has these 30% faster than the Nehalem generation systems in CPU =
bound
>>> tasks and lower power.  Intel prices single socket Xeons so much lowe=
r
>>> than the Dual socket ones that the best value for us is to get more s=
ingle
>>> socket servers rather than fewer dual socket ones (with similar proce=
ssor
>>> to hard drive ratio).
>>=20
>> Yes, in a large cluster the price of filling the second socket can com=
pare
>> to a lot of storage, and TB of storage is more tangible. I guess it de=
pends
>> on your application.
>>=20
>> Regarding Sandy Bridge, I've no experience of those, but I worry that =
10
>> Gbps is still bleeding edge, and shouldn't be needed for code with goo=
d
>> locality anyway; it is probably more cost effective to stay at 1Gbps/s=
erver,
>> though the issue there is the #of HDD/s server generates lots of repli=
cation
>> traffic when a single server fails...
>>=20


The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.

From general-return-3958-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 02:34:09 2011
Return-Path: <general-return-3958-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D0DB76D2F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 28 Jun 2011 02:34:09 +0000 (UTC)
Received: (qmail 98493 invoked by uid 500); 28 Jun 2011 02:34:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98043 invoked by uid 500); 28 Jun 2011 02:34:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98023 invoked by uid 99); 28 Jun 2011 02:33:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 02:33:56 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ryanobjc@gmail.com designates 209.85.161.51 as permitted sender)
Received: from [209.85.161.51] (HELO mail-fx0-f51.google.com) (209.85.161.51)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 02:33:50 +0000
Received: by fxh10 with SMTP id 10so2546409fxh.38
        for <general@hadoop.apache.org>; Mon, 27 Jun 2011 19:33:29 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type:content-transfer-encoding;
        bh=tN1DknEwVkHHcplke29kfpZdsFZDCeA5TR2TifIidUg=;
        b=vm+qFLaBXBpDxKTDKDPOVuHcm2MVSVsCjaw/qIEhaag/HHm/jZDkNwHeI/zYb3slKw
         XzS82PR//VKfPDmFFfey6P1CgxkemwghZeTJTM1FHA56cgSvOaDFQOhLy8i5RtMV9qTJ
         afRY8WITeCR2sCtoMOGahBwGHgzCS7ANpnRuw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=ITrebfKElufB4aWIGOwMr/KZLCkdglCB4rXZG+MaYMlD3f0GfZfsspwBc518Gs3dUO
         CTKBvoKvXZv1TCs93wDfPLLJHQBs3vz8kFMSo3y8p47BF9bUDx4UUdQ1i0drYWAVtpZI
         u1KNv3K3n8K+zdxzuo9qWQgs+ryw/7g21Eva0=
MIME-Version: 1.0
Received: by 10.223.59.92 with SMTP id k28mr9860758fah.27.1309228408894; Mon,
 27 Jun 2011 19:33:28 -0700 (PDT)
Received: by 10.223.81.78 with HTTP; Mon, 27 Jun 2011 19:33:28 -0700 (PDT)
In-Reply-To: <9068C5A7-6090-42A3-8891-F60EF3B660BD@navteq.com>
References: <CA2CD1F8.3F91C%scott@richrelevance.com>
	<4E086BC7.5000208@apache.org>
	<BANLkTimZ4SpW6wEGCGT=g6SfMXq4g2UaWw@mail.gmail.com>
	<9068C5A7-6090-42A3-8891-F60EF3B660BD@navteq.com>
Date: Mon, 27 Jun 2011 19:33:28 -0700
Message-ID: <BANLkTimi1XDyaV7P0meXV=v5z60izzNc6A@mail.gmail.com>
Subject: Re: Hadoop Java Versions
From: Ryan Rawson <ryanobjc@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

There are no bandwidth limitations in 0.20.x.  None that I saw at
least.  It was basically bandwidth-management-by-pwm.  You could
adjust the frequency of how many files-per-node were copied.

In my case, the load was HBase real time serving, so it was servicing
more smaller random reads, not a map-reduce. Everyone has their own
use case :-)

-ryan

On Mon, Jun 27, 2011 at 6:54 PM, Segel, Mike <msegel@navteq.com> wrote:
> That doesn't seem right.
> In one of our test clusters (19 data nodes) we found that under heavy loa=
ds we were disk I/O bound and not network bound. Of course YMMV depending o=
n your ToR switch. If we had more than 4 disks per node, we would probably =
see the network being the bottleneck. What did you set your bandwidth setti=
ngs in the hdfs-site.xml? ( going from memory not sure of the exact setting=
...)
>
> But the good news... Newer hardware will start to have 10GBe on the mothe=
rboard.
>
> Sent from a remote device. Please excuse any typos...
>
> Mike Segel
>
> On Jun 27, 2011, at 7:11 PM, "Ryan Rawson" <ryanobjc@gmail.com> wrote:
>
>> On the subject of gige vs 10-gige, I think that we will very shortly
>> be seeing interest in 10gig, since gige is only 120MB/sec - 1 hard
>> drive of streaming data. =A0Nodes with 4+ disks are throttled by the
>> network. =A0On a small cluster (20 nodes), the replication traffic can
>> choke a cluster to death. =A0The only way to fix quickly it is to bring
>> that node back up. =A0Perhaps the HortonWorks guys can work on that.
>>
>> -ryan
>>
>> On Mon, Jun 27, 2011 at 4:38 AM, Steve Loughran <stevel@apache.org> wrot=
e:
>>> On 26/06/11 20:23, Scott Carey wrote:
>>>>
>>>>
>>>> On 6/23/11 5:49 AM, "Steve Loughran"<stevel@apache.org> =A0wrote:
>>>>
>>>
>>>>> what's your HW setup? #cores/server, #servers, underlying OS?
>>>>
>>>> CentOS 5.6.
>>>> 4 cores / 8 threads a server (Nehalem generation Intel processor).
>>>
>>>
>>> that should be enough to find problems. I've just moved up to a 6-core =
12
>>> thread desktop and that found problems on some non-Hadoop code, which s=
hows
>>> that the more threads you have, and the faster the machines are, the mo=
re
>>> your race conditions show up. With Hadoop the fact that you can have 10=
-1000
>>> servers means that in a large cluster the probability of that race cond=
ition
>>> showing up scales well.
>>>
>>>> Also run a smaller cluster with 2x quad core Core 2 generation Xeons.
>>>>
>>>> Off topic:
>>>> The single proc Nehalem is faster than the dual core 2's for most use
>>>> cases -- and much lower power. =A0Looking forward to single proc 4 or =
6 core
>>>> Sandy Bridge based systems for the next expansion -- testing 4 core vs=
 4
>>>> core has these 30% faster than the Nehalem generation systems in CPU b=
ound
>>>> tasks and lower power. =A0Intel prices single socket Xeons so much low=
er
>>>> than the Dual socket ones that the best value for us is to get more si=
ngle
>>>> socket servers rather than fewer dual socket ones (with similar proces=
sor
>>>> to hard drive ratio).
>>>
>>> Yes, in a large cluster the price of filling the second socket can comp=
are
>>> to a lot of storage, and TB of storage is more tangible. I guess it dep=
ends
>>> on your application.
>>>
>>> Regarding Sandy Bridge, I've no experience of those, but I worry that 1=
0
>>> Gbps is still bleeding edge, and shouldn't be needed for code with good
>>> locality anyway; it is probably more cost effective to stay at 1Gbps/se=
rver,
>>> though the issue there is the #of HDD/s server generates lots of replic=
ation
>>> traffic when a single server fails...
>>>
>
>
> The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above. =A0If you are =
not the intended recipient, you are hereby notified that any dissemination,=
 distribution, or copying of this communication, or any of its contents, is=
 strictly prohibited. =A0If you have received this communication in error, =
please notify the sender and delete/destroy the original message and any co=
py of it from your computer or paper files.
>

From general-return-3959-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 02:58:26 2011
Return-Path: <general-return-3959-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E4782678B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 28 Jun 2011 02:58:26 +0000 (UTC)
Received: (qmail 14127 invoked by uid 500); 28 Jun 2011 02:58:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13911 invoked by uid 500); 28 Jun 2011 02:58:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13892 invoked by uid 99); 28 Jun 2011 02:58:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 02:58:19 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of scott@richrelevance.com designates 64.78.17.18 as permitted sender)
Received: from [64.78.17.18] (HELO EXHUB018-3.exch018.msoutlookonline.net) (64.78.17.18)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 02:58:12 +0000
Received: from EXVMBX018-1.exch018.msoutlookonline.net ([64.78.17.47]) by
 EXHUB018-3.exch018.msoutlookonline.net ([64.78.17.18]) with mapi; Mon, 27 Jun
 2011 19:57:51 -0700
From: Scott Carey <scott@richrelevance.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 27 Jun 2011 19:58:43 -0700
Subject: Re: Hadoop Java Versions
Thread-Topic: Hadoop Java Versions
Thread-Index: Acw1PysZESKZW5H9Toauz1SW5fWmsA==
Message-ID: <CA2E8F93.3FB21%scott@richrelevance.com>
In-Reply-To: <BANLkTimZ4SpW6wEGCGT=g6SfMXq4g2UaWw@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-MacOutlook/14.10.0.110310
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

For cost reasons, we just bonded two 1G network ports together.  A single
stream won't go past 1Gbps, but concurrent ones do -- this is with the
Linux built-in bonding.  The network is only stressed during 'sort-like'
jobs or big replication events.
We also removed some disk bottlenecks by tuning the file systems
aggressively -- using a separate partition for the M/R temp and the
location that jars may unpack into helps tremendously.  Ext4 can be
configured to delay flushing to disk for this temp space, which for small
jobs decreases the I/O tremendously as many files are deleted before they
get pushed to disk.

On 6/27/11 5:10 PM, "Ryan Rawson" <ryanobjc@gmail.com> wrote:

>On the subject of gige vs 10-gige, I think that we will very shortly
>be seeing interest in 10gig, since gige is only 120MB/sec - 1 hard
>drive of streaming data.  Nodes with 4+ disks are throttled by the
>network.  On a small cluster (20 nodes), the replication traffic can
>choke a cluster to death.  The only way to fix quickly it is to bring
>that node back up.  Perhaps the HortonWorks guys can work on that.
>
>-ryan
>
>On Mon, Jun 27, 2011 at 4:38 AM, Steve Loughran <stevel@apache.org> wrote:
>> On 26/06/11 20:23, Scott Carey wrote:
>>>
>>>
>>> On 6/23/11 5:49 AM, "Steve Loughran"<stevel@apache.org>  wrote:
>>>
>>
>>>> what's your HW setup? #cores/server, #servers, underlying OS?
>>>
>>> CentOS 5.6.
>>> 4 cores / 8 threads a server (Nehalem generation Intel processor).
>>
>>
>> that should be enough to find problems. I've just moved up to a 6-core
>>12
>> thread desktop and that found problems on some non-Hadoop code, which
>>shows
>> that the more threads you have, and the faster the machines are, the
>>more
>> your race conditions show up. With Hadoop the fact that you can have
>>10-1000
>> servers means that in a large cluster the probability of that race
>>condition
>> showing up scales well.
>>
>>> Also run a smaller cluster with 2x quad core Core 2 generation Xeons.
>>>
>>> Off topic:
>>> The single proc Nehalem is faster than the dual core 2's for most use
>>> cases -- and much lower power.  Looking forward to single proc 4 or 6
>>>core
>>> Sandy Bridge based systems for the next expansion -- testing 4 core vs
>>>4
>>> core has these 30% faster than the Nehalem generation systems in CPU
>>>bound
>>> tasks and lower power.  Intel prices single socket Xeons so much lower
>>> than the Dual socket ones that the best value for us is to get more
>>>single
>>> socket servers rather than fewer dual socket ones (with similar
>>>processor
>>> to hard drive ratio).
>>
>> Yes, in a large cluster the price of filling the second socket can
>>compare
>> to a lot of storage, and TB of storage is more tangible. I guess it
>>depends
>> on your application.
>>
>> Regarding Sandy Bridge, I've no experience of those, but I worry that 10
>> Gbps is still bleeding edge, and shouldn't be needed for code with good
>> locality anyway; it is probably more cost effective to stay at
>>1Gbps/server,
>> though the issue there is the #of HDD/s server generates lots of
>>replication
>> traffic when a single server fails...
>>


From general-return-3960-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 03:49:33 2011
Return-Path: <general-return-3960-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B8F836771
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 28 Jun 2011 03:49:33 +0000 (UTC)
Received: (qmail 50509 invoked by uid 500); 28 Jun 2011 03:49:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50329 invoked by uid 500); 28 Jun 2011 03:49:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50321 invoked by uid 99); 28 Jun 2011 03:49:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 03:49:13 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of msegel@navteq.com designates 204.120.70.37 as permitted sender)
Received: from [204.120.70.37] (HELO xmailfargo.navteq.com) (204.120.70.37)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 03:49:06 +0000
Received: from fargo-spamfw.navteq.com (navteq-spamfw.navteq.com [10.6.10.131])
	by xmailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id p5S3Chi0011185
	for <general@hadoop.apache.org>; Mon, 27 Jun 2011 22:12:43 -0500
Received: from imailfargo.navteq.com (localhost [127.0.0.1])
	by fargo-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id 90682488FF25
	for <general@hadoop.apache.org>; Mon, 27 Jun 2011 22:48:44 -0500 (CDT)
Received: from imailfargo.navteq.com (imailfargo.navteq.com [10.6.10.130]) by fargo-spamfw.navteq.com with ESMTP id pi7z0jqipuIoDXVe for <general@hadoop.apache.org>; Mon, 27 Jun 2011 22:48:44 -0500 (CDT)
Received: from hq-ex-ht02.ad.navteq.com ([10.8.222.172])
	by imailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id p5S3mhPP011138
	for <general@hadoop.apache.org>; Mon, 27 Jun 2011 22:48:44 -0500
Received: from hq-ex-mb03.ad.navteq.com ([fe80::c4dd:7b21:5c22:cfe4]) by
 hq-ex-ht02.ad.navteq.com ([fe80::24f6:ca50:734d:df6e%13]) with mapi; Mon, 27
 Jun 2011 22:48:43 -0500
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 27 Jun 2011 22:49:31 -0500
Subject: Re: Hadoop Java Versions
Thread-Topic: Hadoop Java Versions
Thread-Index: Acw1RkZxBLZ+20d6SSimJ/e80DXtjw==
Message-ID: <918BB6E5-EF4B-4EB2-9B55-8BCBAF6DBA7B@navteq.com>
References: <CA2CD1F8.3F91C%scott@richrelevance.com>
 <4E086BC7.5000208@apache.org>
 <BANLkTimZ4SpW6wEGCGT=g6SfMXq4g2UaWw@mail.gmail.com>
 <9068C5A7-6090-42A3-8891-F60EF3B660BD@navteq.com>
 <BANLkTimi1XDyaV7P0meXV=v5z60izzNc6A@mail.gmail.com>
In-Reply-To: <BANLkTimi1XDyaV7P0meXV=v5z60izzNc6A@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1
X-Virus-Checked: Checked by ClamAV on apache.org

Hmmm. I could have sworn there was a background balancing bandwidth limit=
er.

Haven't tested random reads... The last test we did ended up hitting the =
cache, but we didn't push it hard enough to hit network bandwidth limitat=
ions... Not to say they don't exist. Like I said in the other post, if we=
 had more disks... we would hit it.=20

We'll have to do more random testing.

Sent from a remote device. Please excuse any typos...

Mike Segel

On Jun 27, 2011, at 9:34 PM, "Ryan Rawson" <ryanobjc@gmail.com> wrote:

> There are no bandwidth limitations in 0.20.x.  None that I saw at
> least.  It was basically bandwidth-management-by-pwm.  You could
> adjust the frequency of how many files-per-node were copied.
>=20
> In my case, the load was HBase real time serving, so it was servicing
> more smaller random reads, not a map-reduce. Everyone has their own
> use case :-)
>=20
> -ryan
>=20
> On Mon, Jun 27, 2011 at 6:54 PM, Segel, Mike <msegel@navteq.com> wrote:
>> That doesn't seem right.
>> In one of our test clusters (19 data nodes) we found that under heavy =
loads we were disk I/O bound and not network bound. Of course YMMV depend=
ing on your ToR switch. If we had more than 4 disks per node, we would pr=
obably see the network being the bottleneck. What did you set your bandwi=
dth settings in the hdfs-site.xml? ( going from memory not sure of the ex=
act setting...)
>>=20
>> But the good news... Newer hardware will start to have 10GBe on the mo=
therboard.
>>=20
>> Sent from a remote device. Please excuse any typos...
>>=20
>> Mike Segel
>>=20
>> On Jun 27, 2011, at 7:11 PM, "Ryan Rawson" <ryanobjc@gmail.com> wrote:
>>=20
>>> On the subject of gige vs 10-gige, I think that we will very shortly
>>> be seeing interest in 10gig, since gige is only 120MB/sec - 1 hard
>>> drive of streaming data.  Nodes with 4+ disks are throttled by the
>>> network.  On a small cluster (20 nodes), the replication traffic can
>>> choke a cluster to death.  The only way to fix quickly it is to bring
>>> that node back up.  Perhaps the HortonWorks guys can work on that.
>>>=20
>>> -ryan
>>>=20
>>> On Mon, Jun 27, 2011 at 4:38 AM, Steve Loughran <stevel@apache.org> w=
rote:
>>>> On 26/06/11 20:23, Scott Carey wrote:
>>>>>=20
>>>>>=20
>>>>> On 6/23/11 5:49 AM, "Steve Loughran"<stevel@apache.org>  wrote:
>>>>>=20
>>>>=20
>>>>>> what's your HW setup? #cores/server, #servers, underlying OS?
>>>>>=20
>>>>> CentOS 5.6.
>>>>> 4 cores / 8 threads a server (Nehalem generation Intel processor).
>>>>=20
>>>>=20
>>>> that should be enough to find problems. I've just moved up to a 6-co=
re 12
>>>> thread desktop and that found problems on some non-Hadoop code, whic=
h shows
>>>> that the more threads you have, and the faster the machines are, the=
 more
>>>> your race conditions show up. With Hadoop the fact that you can have=
 10-1000
>>>> servers means that in a large cluster the probability of that race c=
ondition
>>>> showing up scales well.
>>>>=20
>>>>> Also run a smaller cluster with 2x quad core Core 2 generation Xeon=
s.
>>>>>=20
>>>>> Off topic:
>>>>> The single proc Nehalem is faster than the dual core 2's for most u=
se
>>>>> cases -- and much lower power.  Looking forward to single proc 4 or=
 6 core
>>>>> Sandy Bridge based systems for the next expansion -- testing 4 core=
 vs 4
>>>>> core has these 30% faster than the Nehalem generation systems in CP=
U bound
>>>>> tasks and lower power.  Intel prices single socket Xeons so much lo=
wer
>>>>> than the Dual socket ones that the best value for us is to get more=
 single
>>>>> socket servers rather than fewer dual socket ones (with similar pro=
cessor
>>>>> to hard drive ratio).
>>>>=20
>>>> Yes, in a large cluster the price of filling the second socket can c=
ompare
>>>> to a lot of storage, and TB of storage is more tangible. I guess it =
depends
>>>> on your application.
>>>>=20
>>>> Regarding Sandy Bridge, I've no experience of those, but I worry tha=
t 10
>>>> Gbps is still bleeding edge, and shouldn't be needed for code with g=
ood
>>>> locality anyway; it is probably more cost effective to stay at 1Gbps=
/server,
>>>> though the issue there is the #of HDD/s server generates lots of rep=
lication
>>>> traffic when a single server fails...
>>>>=20
>>=20
>>=20
>> The information contained in this communication may be CONFIDENTIAL an=
d is intended only for the use of the recipient(s) named above.  If you a=
re not the intended recipient, you are hereby notified that any dissemina=
tion, distribution, or copying of this communication, or any of its conte=
nts, is strictly prohibited.  If you have received this communication in =
error, please notify the sender and delete/destroy the original message a=
nd any copy of it from your computer or paper files.
>>=20


The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.

From general-return-3961-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 10:00:43 2011
Return-Path: <general-return-3961-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 923BB6C5B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 28 Jun 2011 10:00:43 +0000 (UTC)
Received: (qmail 23005 invoked by uid 500); 28 Jun 2011 10:00:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22244 invoked by uid 500); 28 Jun 2011 10:00:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22230 invoked by uid 99); 28 Jun 2011 10:00:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 10:00:27 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 10:00:19 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 179361BA2E9
	for <general@hadoop.apache.org>; Tue, 28 Jun 2011 10:59:58 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id a6bhNdoI-0ir for <general@hadoop.apache.org>;
	Tue, 28 Jun 2011 10:59:57 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 91F7F1BA0FB
	for <general@hadoop.apache.org>; Tue, 28 Jun 2011 10:59:57 +0100 (BST)
MailScanner-NULL-Check: 1309859983.49478@+UAzzVM+FNcKlRvcHfdRDQ
Received: from [16.25.175.86] (wildhaus.hpl.hp.com [16.25.175.86])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p5S9xhh5002415
	for <general@hadoop.apache.org>; Tue, 28 Jun 2011 10:59:43 +0100 (BST)
Message-ID: <4E09A60C.1090605@apache.org>
Date: Tue, 28 Jun 2011 10:59:40 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Hadoop Java Versions
References: <CA2CD1F8.3F91C%scott@richrelevance.com> <4E086BC7.5000208@apache.org> <BANLkTimZ4SpW6wEGCGT=g6SfMXq4g2UaWw@mail.gmail.com> <9068C5A7-6090-42A3-8891-F60EF3B660BD@navteq.com> <BANLkTimi1XDyaV7P0meXV=v5z60izzNc6A@mail.gmail.com> <918BB6E5-EF4B-4EB2-9B55-8BCBAF6DBA7B@navteq.com>
In-Reply-To: <918BB6E5-EF4B-4EB2-9B55-8BCBAF6DBA7B@navteq.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p5S9xhh5002415
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 28/06/11 04:49, Segel, Mike wrote:
> Hmmm. I could have sworn there was a background balancing bandwidth limiter.

There is, for the rebalancer, node outages are taken more seriously, 
though there have been problems in past 0.20.x where there was a risk of 
a cascade failure on a big switch/rack failure. The risk has been 
reduced, though we all await field reports to confirm this :)

You can get 12-24 TB in a server today, which means the loss of a server 
generates a lot of traffic -which argues for 10 Gbe.

But
  -big increase in switch cost, especially if you (CoI warning) go with 
Cisco
  -there have been problems with things like BIOS PXE and lights out 
management on 10 Gbe -probably due to the NICs being things the BIOS 
wasn't expecting and off the mainboard. This should improve.
  -I don't know how well linux works with ether that fast (field reports 
useful)
  -the big threat is still ToR switch failure, as that will trigger a 
re-replication of every block in the rack.

2x1 Gbe lets you have redundant switches, albeit at the price of more 
wiring, more things to go wrong with the wiring, etc.

The other thing to consider is how well the "enterprise" switches work 
in this world -with a Hadoop cluster you can really test those claims 
how well the switches handle every port lighting up at full rate. 
Indeed, I recommend that as part of your acceptance tests for the switch.



From general-return-3962-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 12:27:00 2011
Return-Path: <general-return-3962-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 678C064E2
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 28 Jun 2011 12:27:00 +0000 (UTC)
Received: (qmail 32379 invoked by uid 500); 28 Jun 2011 12:26:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32295 invoked by uid 500); 28 Jun 2011 12:26:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32278 invoked by uid 99); 28 Jun 2011 12:26:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 12:26:57 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,LOTS_OF_MONEY,MIME_QP_LONG_LINE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of im_gumby@hotmail.com designates 65.55.116.29 as permitted sender)
Received: from [65.55.116.29] (HELO blu0-omc1-s18.blu0.hotmail.com) (65.55.116.29)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 12:26:47 +0000
Received: from BLU0-SMTP155 ([65.55.116.9]) by blu0-omc1-s18.blu0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Tue, 28 Jun 2011 05:26:26 -0700
X-Originating-IP: [173.15.87.37]
X-Originating-Email: [im_gumby@hotmail.com]
Message-ID: <BLU0-SMTP155159CDC315FC41B2E9B58F0560@phx.gbl>
Received: from [192.168.1.100] ([173.15.87.37]) by BLU0-SMTP155.blu0.hotmail.com over TLS secured channel with Microsoft SMTPSVC(6.0.3790.4675);
	 Tue, 28 Jun 2011 05:26:26 -0700
Subject: Re: Hadoop Java Versions
References: <CA2CD1F8.3F91C%scott@richrelevance.com> <4E086BC7.5000208@apache.org> <BANLkTimZ4SpW6wEGCGT=g6SfMXq4g2UaWw@mail.gmail.com> <9068C5A7-6090-42A3-8891-F60EF3B660BD@navteq.com> <BANLkTimi1XDyaV7P0meXV=v5z60izzNc6A@mail.gmail.com> <918BB6E5-EF4B-4EB2-9B55-8BCBAF6DBA7B@navteq.com> <4E09A60C.1090605@apache.org>
From: Michel Segel <michael_segel@hotmail.com>
Content-Type: text/plain; charset="us-ascii"
X-Mailer: iPad Mail (8J2)
In-Reply-To: <4E09A60C.1090605@apache.org>
Date: Tue, 28 Jun 2011 07:27:13 -0500
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0 (iPad Mail 8J2)
X-OriginalArrivalTime: 28 Jun 2011 12:26:26.0771 (UTC) FILETIME=[99BB2A30:01CC358E]
Sender: <im_gumby@hotmail.com>
X-Virus-Checked: Checked by ClamAV on apache.org

You're preaching to the choir. :-)

With Sandybridge, you're going to start seeing 10 GBe on the motherboard.
We built our clusters using 1U boxes where you're stuck w 4 3.5" drives. Wit=
h larger chassis,
You can fit an additional controller card and more drives.

More drives reduces the bottleneck and means  your performance will  be thro=
ttled by your network and the amount of memory.

I priced out a couple of vendors and when you build out your boxes, the magi=
c number per data node is $10,000.00 USD. (budget this amount per data node.=
) Moore's Law doesn't drop the price, but it gets you more bang for your buc=
k. Note that this magic number is pre-discount and YMMV. [This also gets in t=
o what is meant by commodity hardware.]

I agree that 10GBe is a necessity and I have been looking at it for the past=
 2 years, only to be shot down by my client's IT group. I agree that Cisco's=
 ToR switches are expensive, however there are Arista and Blade networks swi=
tches that claim to be Cisco friendly and aren't too pricey. Somewhere aroun=
d 10K a box. (Again YMMV).

If you want to upgrade existing boxes, you will probably want to look at Sol=
arflare cards.=20

jMHO...


Sent from a remote device. Please excuse any typos...

Mike Segel

On Jun 28, 2011, at 4:59 AM, Steve Loughran <stevel@apache.org> wrote:

> On 28/06/11 04:49, Segel, Mike wrote:
>> Hmmm. I could have sworn there was a background balancing bandwidth limit=
er.
>=20
> There is, for the rebalancer, node outages are taken more seriously, thoug=
h there have been problems in past 0.20.x where there was a risk of a cascad=
e failure on a big switch/rack failure. The risk has been reduced, though we=
 all await field reports to confirm this :)
>=20
> You can get 12-24 TB in a server today, which means the loss of a server g=
enerates a lot of traffic -which argues for 10 Gbe.
>=20
> But
> -big increase in switch cost, especially if you (CoI warning) go with Cisc=
o
> -there have been problems with things like BIOS PXE and lights out managem=
ent on 10 Gbe -probably due to the NICs being things the BIOS wasn't expecti=
ng and off the mainboard. This should improve.
> -I don't know how well linux works with ether that fast (field reports use=
ful)
> -the big threat is still ToR switch failure, as that will trigger a re-rep=
lication of every block in the rack.
>=20
> 2x1 Gbe lets you have redundant switches, albeit at the price of more wiri=
ng, more things to go wrong with the wiring, etc.
>=20
> The other thing to consider is how well the "enterprise" switches work in t=
his world -with a Hadoop cluster you can really test those claims how well t=
he switches handle every port lighting up at full rate. Indeed, I recommend t=
hat as part of your acceptance tests for the switch.
>=20
>=20
>=20

From general-return-3963-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 17:26:09 2011
Return-Path: <general-return-3963-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EB2696C5D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 28 Jun 2011 17:26:09 +0000 (UTC)
Received: (qmail 31699 invoked by uid 500); 28 Jun 2011 17:26:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31634 invoked by uid 500); 28 Jun 2011 17:26:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31626 invoked by uid 99); 28 Jun 2011 17:26:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 17:26:07 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 216.145.54.171 is neither permitted nor denied by domain of arunc@yahoo-inc.com)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 17:25:59 +0000
Received: from SP2-EX07CAS04.ds.corp.yahoo.com (sp2-ex07cas04.corp.sp2.yahoo.com [98.137.59.5])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5SHPAV3092265;
	Tue, 28 Jun 2011 10:25:10 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yahoo-inc.com;
	s=cobra; t=1309281910;
	bh=1fynlfPBgRaGCoZhzcRIhlkd9H05BQARSuBxR14W57Y=;
	h=From:To:CC:Date:Subject:Message-ID:References:In-Reply-To:
	 Content-Type:Content-Transfer-Encoding:MIME-Version;
	b=idKsLZQ8piqj/BXvs4BMcEKws5hUec2nxi/nxk8TeDRku16yIEysXDCqTbcJjBq7D
	 lLkbjIGgu08fS/PmxH4MjLlzOqlW8/Y4DbkfSuwtgrPokzD7MOqpo5pMUnRwXCdzXW
	 Foz+2v6BqET/BAbe/tW8whVAK6LiWLCfT8JknSuI=
Received: from SP2-EX07VS04.ds.corp.yahoo.com ([98.137.59.33]) by
 SP2-EX07CAS04.ds.corp.yahoo.com ([98.137.59.5]) with mapi; Tue, 28 Jun 2011
 10:25:10 -0700
From: Arun C Murthy <arunc@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 28 Jun 2011 10:25:05 -0700
Subject: Re: Hadoop Java Versions
Thread-Topic: Hadoop Java Versions
Thread-Index: Acw1uFVaG7l7VCOvTwyMrfWlJCgZTA==
Message-ID: <F41867EF-1F87-4421-AABE-D344D8BFA2EA@yahoo-inc.com>
References: <CA2CD1F8.3F91C%scott@richrelevance.com>
 <4E086BC7.5000208@apache.org>
 <BANLkTimZ4SpW6wEGCGT=g6SfMXq4g2UaWw@mail.gmail.com>
 <9068C5A7-6090-42A3-8891-F60EF3B660BD@navteq.com>
 <BANLkTimi1XDyaV7P0meXV=v5z60izzNc6A@mail.gmail.com>
 <918BB6E5-EF4B-4EB2-9B55-8BCBAF6DBA7B@navteq.com>
 <4E09A60C.1090605@apache.org>
In-Reply-To: <4E09A60C.1090605@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

We at Yahoo are about to deploy code to ensure a disk failure on a datanode=
 is just that - a disk failure. Not a node failure. This really helps avoid=
 replication storms.

It's in the 0.20.204 branch for the curious.

Arun

Sent from my iPhone

On Jun 28, 2011, at 3:01 AM, "Steve Loughran" <stevel@apache.org> wrote:

> On 28/06/11 04:49, Segel, Mike wrote:
>> Hmmm. I could have sworn there was a background balancing bandwidth limi=
ter.
>=20
> There is, for the rebalancer, node outages are taken more seriously,=20
> though there have been problems in past 0.20.x where there was a risk of=
=20
> a cascade failure on a big switch/rack failure. The risk has been=20
> reduced, though we all await field reports to confirm this :)
>=20
> You can get 12-24 TB in a server today, which means the loss of a server=
=20
> generates a lot of traffic -which argues for 10 Gbe.
>=20
> But
>  -big increase in switch cost, especially if you (CoI warning) go with=20
> Cisco
>  -there have been problems with things like BIOS PXE and lights out=20
> management on 10 Gbe -probably due to the NICs being things the BIOS=20
> wasn't expecting and off the mainboard. This should improve.
>  -I don't know how well linux works with ether that fast (field reports=20
> useful)
>  -the big threat is still ToR switch failure, as that will trigger a=20
> re-replication of every block in the rack.
>=20
> 2x1 Gbe lets you have redundant switches, albeit at the price of more=20
> wiring, more things to go wrong with the wiring, etc.
>=20
> The other thing to consider is how well the "enterprise" switches work=20
> in this world -with a Hadoop cluster you can really test those claims=20
> how well the switches handle every port lighting up at full rate.=20
> Indeed, I recommend that as part of your acceptance tests for the switch.
>=20
>=20

From general-return-3964-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 07:13:01 2011
Return-Path: <general-return-3964-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CE8D96B35
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 30 Jun 2011 07:13:01 +0000 (UTC)
Received: (qmail 61301 invoked by uid 500); 30 Jun 2011 07:12:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60909 invoked by uid 500); 30 Jun 2011 07:12:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60900 invoked by uid 99); 30 Jun 2011 07:12:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 07:12:48 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.161.176 as permitted sender)
Received: from [209.85.161.176] (HELO mail-gx0-f176.google.com) (209.85.161.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 07:12:40 +0000
Received: by gxk7 with SMTP id 7so1114315gxk.35
        for <general@hadoop.apache.org>; Thu, 30 Jun 2011 00:12:19 -0700 (PDT)
Received: by 10.101.165.19 with SMTP id s19mr1475711ano.53.1309417939721; Thu,
 30 Jun 2011 00:12:19 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.100.164.19 with HTTP; Thu, 30 Jun 2011 00:11:59 -0700 (PDT)
From: Todd Lipcon <todd@cloudera.com>
Date: Thu, 30 Jun 2011 00:11:59 -0700
Message-ID: <BANLkTimMRWjfD9TgCqB=c67YoNHNhdyzng@mail.gmail.com>
Subject: Hoping to merge HDFS-1073 branch soon
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c92a8d28dd5004a6e89fa4
X-Virus-Checked: Checked by ClamAV on apache.org

--001636c92a8d28dd5004a6e89fa4
Content-Type: text/plain; charset=ISO-8859-1

Hey all,

Work on the HDFS-1073 branch has been progressing steadily, and I believe
we're coming close to the point where it can be merged. To briefly summarize
the status:
- NameNode and SecondaryNameNode are both fully working and have undergone
some stress/fault testing in addition to a over 3000 lines worth of new unit
tests.
- Most of the existing unit tests have been updated, though a few more need
some small tweaks (HDFS-2101)
- The BackupNode and CheckpointNode are not currently working, though I am
working on it locally and making good progress (HDFS-1979)
- There are a few various and sundry small improvements that should probably
be done before release, but I think could be done either before or after
merge (eg HDFS-2104)

Given this, I am expecting that we can merge this into trunk by the end of
July if not earlier, as soon as the BN/CN work is complete. If you are
hoping to review the code or tests before merge time, this is your early
warning! Please do so now!

Thanks!

-Todd
P.S. I will also be giving a short talk about the motivations and current
status of this project at Friday's contributor meeting, for those who are
able to attend. If we're lucky, maybe even a demo!
-- 
Todd Lipcon
Software Engineer, Cloudera

--001636c92a8d28dd5004a6e89fa4--

From general-return-3965-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 16:07:03 2011
Return-Path: <general-return-3965-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4C4616EC7
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 30 Jun 2011 16:07:03 +0000 (UTC)
Received: (qmail 137 invoked by uid 500); 30 Jun 2011 16:07:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99810 invoked by uid 500); 30 Jun 2011 16:07:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 99801 invoked by uid 99); 30 Jun 2011 16:06:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 16:06:59 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 69.147.107.21 is neither permitted nor denied by domain of eric14@yahoo-inc.com)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 16:06:54 +0000
Received: from [10.0.1.3] (snvvpn1-10-72-244-c110.hq.corp.yahoo.com [10.72.244.110])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p5UG69FN070513
	for <general@hadoop.apache.org>; Thu, 30 Jun 2011 09:06:11 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yahoo-inc.com;
	s=cobra; t=1309449971;
	bh=uDOCAQxshJsKghnWqPo+spYzI9RwgY+PbSViIWCoV2c=;
	h=Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:
	 Content-Transfer-Encoding:Message-Id:References:To;
	b=Pb4SaqXxuytEXiCS9D1+JV1a0Mw37zx8kQXJZ9FZ7qNdIe0u+HfUaUpzpm0Iw2Cv0
	 xHEPee5l8yINpJMcbTEC9eykASiheRcioq10rkcUpXJAg/VThwz4Xx8qq46mKk2s3D
	 Zph9zWQ5wCDHZBkVBLKY86cSY8tHPzAVvN++HNV8=
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: =?iso-8859-1?Q?Re=3A_Hadoop=B4s_Internationalization?=
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <BANLkTik5iE_y8FM8nUVJqdcS8tnBpx+Bvg@mail.gmail.com>
Date: Thu, 30 Jun 2011 09:06:08 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <67CCDC04-7B42-4B1F-96A5-DD617B3905CF@yahoo-inc.com>
References: <CA2A4ABD.1C55C%sureshms@yahoo-inc.com> <41D837D5-97C3-48BB-9F95-01720DFB769C@yahoo-inc.com> <4E054E1A.2090809@uci.cu> <BANLkTik5iE_y8FM8nUVJqdcS8tnBpx+Bvg@mail.gmail.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

Do other apache projects have a good localization framework for error =
messages? =20

I'd think there would be interest in starting this, although since we'd =
need to add the framework, this would be an investment.

On Jun 25, 2011, at 1:38 AM, Owen O'Malley wrote:

> On Fri, Jun 24, 2011 at 7:55 PM, Marcos Ortiz <mlortiz@uci.cu> wrote:
>=20
>> Regards to all the list.
>> I=B4m looking for a proper way to work on the internationalization of =
Hadoop,
>> but I don=B4t know if this is a good project
>> or if this is useful for the community, at least, I think that it =
would be
>> very useful for many people that want to see the messages
>> of the project in another language, for example, in Spanish.
>=20
>=20
> I think it would be very useful, but very time consuming project. I'd
> suggest that you start by translating the documentation for the =
release into
> another language first.
>=20
> -- Owen


From general-return-3966-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 21:27:53 2011
Return-Path: <general-return-3966-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D69966AD6
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 30 Jun 2011 21:27:53 +0000 (UTC)
Received: (qmail 53749 invoked by uid 500); 30 Jun 2011 21:27:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53324 invoked by uid 500); 30 Jun 2011 21:27:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53064 invoked by uid 99); 30 Jun 2011 21:27:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 21:27:48 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jghoman@gmail.com designates 74.125.83.176 as permitted sender)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 21:27:42 +0000
Received: by pve37 with SMTP id 37so3211249pve.35
        for <general@hadoop.apache.org>; Thu, 30 Jun 2011 14:27:21 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        bh=tgBPB98Men4x7xF5o9+C2lIPZ1753BjGXH/XBzAdvBU=;
        b=QgKGdoloVH8fS6m5SKk82AAhJH7p1Dj5dxPQMAFScC4LTFxdLWTM4NJizpfkpcE3bg
         xZT38ExkRH1Aa1cIwIw4PrmQONpZeydwy3THornIpd00B8/vsG7vf9gDykWqTw23U6Or
         Fq7hqosMdKDeCjTSqW8LAlHMq5LnIRSsNm838=
Received: by 10.142.207.2 with SMTP id e2mr1198916wfg.415.1309469241216; Thu,
 30 Jun 2011 14:27:21 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.143.92.11 with HTTP; Thu, 30 Jun 2011 14:26:51 -0700 (PDT)
In-Reply-To: <67CCDC04-7B42-4B1F-96A5-DD617B3905CF@yahoo-inc.com>
References: <CA2A4ABD.1C55C%sureshms@yahoo-inc.com> <41D837D5-97C3-48BB-9F95-01720DFB769C@yahoo-inc.com>
 <4E054E1A.2090809@uci.cu> <BANLkTik5iE_y8FM8nUVJqdcS8tnBpx+Bvg@mail.gmail.com>
 <67CCDC04-7B42-4B1F-96A5-DD617B3905CF@yahoo-inc.com>
From: Jakob Homan <jghoman@gmail.com>
Date: Thu, 30 Jun 2011 14:26:51 -0700
Message-ID: <BANLkTimci+CrAqQFtJMbsdAFBiK26EL64g@mail.gmail.com>
Subject: =?ISO-8859-1?Q?Re=3A_Hadoop=B4s_Internationalization?=
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

I'd be hesitant about this given our experience with the Chinese
version of the Hadoop documents.  Picking the Chinese version of
hdfs_quota_admin_guide.xml, as an example, shows that it has not
actually been updated (beyond svn moves and copyright years) since its
original commit back on 12/10/08 (svn rev: 736174).  I'm actually
going to open a JIRA to remove it since it's now woefully out of data
(and even worse than no documentation is wrong documentation).  We'd
need to make sure that there would be contributors willing and able to
keep these documents up-to-date, which hasn't happened with the
Chinese, even though we have quite a few Chinese-speaking
contributors.
-jakob

On Thu, Jun 30, 2011 at 9:06 AM, Eric Baldeschwieler
<eric14@yahoo-inc.com> wrote:
> Do other apache projects have a good localization framework for error mes=
sages?
>
> I'd think there would be interest in starting this, although since we'd n=
eed to add the framework, this would be an investment.
>
> On Jun 25, 2011, at 1:38 AM, Owen O'Malley wrote:
>
>> On Fri, Jun 24, 2011 at 7:55 PM, Marcos Ortiz <mlortiz@uci.cu> wrote:
>>
>>> Regards to all the list.
>>> I=B4m looking for a proper way to work on the internationalization of H=
adoop,
>>> but I don=B4t know if this is a good project
>>> or if this is useful for the community, at least, I think that it would=
 be
>>> very useful for many people that want to see the messages
>>> of the project in another language, for example, in Spanish.
>>
>>
>> I think it would be very useful, but very time consuming project. I'd
>> suggest that you start by translating the documentation for the release =
into
>> another language first.
>>
>> -- Owen
>
>

From general-return-3967-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 21:36:29 2011
Return-Path: <general-return-3967-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 23FA3643F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 30 Jun 2011 21:36:29 +0000 (UTC)
Received: (qmail 72754 invoked by uid 500); 30 Jun 2011 21:36:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72695 invoked by uid 500); 30 Jun 2011 21:36:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72685 invoked by uid 99); 30 Jun 2011 21:36:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 21:36:26 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [145.100.8.49] (HELO smtp.sara.nl) (145.100.8.49)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 21:36:18 +0000
Received: from planck.ka.sara.nl (145.100.8.32) by sara-exch-fe1.ka.sara.nl
 (145.100.8.49) with Microsoft SMTP Server (TLS) id 14.0.702.0; Thu, 30 Jun
 2011 23:35:55 +0200
Received: from planck.ka.sara.nl ([145.100.8.32]) by planck.ka.sara.nl
 ([145.100.8.32]) with mapi; Thu, 30 Jun 2011 23:35:51 +0200
From: Evert Lammerts <Evert.Lammerts@sara.nl>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 30 Jun 2011 23:31:26 +0200
Subject: RE: Hadoop Java Versions
Thread-Topic: Hadoop Java Versions
Thread-Index: Acw1ekRqZgpWnTB4T6m5N4iD7AhplgB8sz+f
Message-ID: <ADF94D8555C7A246B86A633685E0178AB90A35CA96@planck.ka.sara.nl>
References: <CA2CD1F8.3F91C%scott@richrelevance.com>
 <4E086BC7.5000208@apache.org>
 <BANLkTimZ4SpW6wEGCGT=g6SfMXq4g2UaWw@mail.gmail.com>
 <9068C5A7-6090-42A3-8891-F60EF3B660BD@navteq.com>
 <BANLkTimi1XDyaV7P0meXV=v5z60izzNc6A@mail.gmail.com>
 <918BB6E5-EF4B-4EB2-9B55-8BCBAF6DBA7B@navteq.com>,<4E09A60C.1090605@apache.org>
In-Reply-To: <4E09A60C.1090605@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

> You can get 12-24 TB in a server today, which means the loss of a server
> generates a lot of traffic -which argues for 10 Gbe.
>=20
> But
>   -big increase in switch cost, especially if you (CoI warning) go with
> Cisco
>   -there have been problems with things like BIOS PXE and lights out
> management on 10 Gbe -probably due to the NICs being things the BIOS
> wasn't expecting and off the mainboard. This should improve.
>   -I don't know how well linux works with ether that fast (field reports
> useful)
>   -the big threat is still ToR switch failure, as that will trigger a
> re-replication of every block in the rack.

Keeping the amount of disks per node low and the amount of nodes high shoul=
d keep the impact of dead nodes in control. A ToR switch failing is differe=
nt - missing 30 nodes (~120TB) at once cannot be fixed by adding more nodes=
; that actually increases ToR switch failure. Although such failure is quit=
e rare to begin with, I guess. The back-of-the-envelope-calculation I made =
suggests that ~150 (1U) nodes should be fine with 1Gb ethernet. (e.g., when=
 6 nodes fail in a cluster with 150 nodes with four 2TB disks each, with HD=
FS 60% full, it takes around ~32 minutes to recover. 2 nodes failing should=
 take around 640 seconds. Also see the attached spreadsheet.) This doesn't =
take ToR switch failure in account though. On the other hand - 150 nodes is=
 only ~5 racks - in such a scenario you might rather want to shut the syste=
m down completely rather than letting it replicate 20% of all data.

Cheers,
Evert=

From general-return-3968-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 21:39:25 2011
Return-Path: <general-return-3968-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0410265A4
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 30 Jun 2011 21:39:24 +0000 (UTC)
Received: (qmail 77084 invoked by uid 500); 30 Jun 2011 21:39:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76999 invoked by uid 500); 30 Jun 2011 21:39:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76990 invoked by uid 99); 30 Jun 2011 21:39:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 21:39:22 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [145.100.8.49] (HELO smtp.sara.nl) (145.100.8.49)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 21:39:16 +0000
Received: from planck.ka.sara.nl (145.100.8.32) by sara-exch-fe1.ka.sara.nl
 (145.100.8.49) with Microsoft SMTP Server (TLS) id 14.0.702.0; Thu, 30 Jun
 2011 23:38:53 +0200
Received: from planck.ka.sara.nl ([145.100.8.32]) by planck.ka.sara.nl
 ([145.100.8.32]) with mapi; Thu, 30 Jun 2011 23:38:32 +0200
From: Evert Lammerts <Evert.Lammerts@sara.nl>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 30 Jun 2011 23:37:11 +0200
Subject: RE: Hadoop Java Versions
Thread-Topic: Hadoop Java Versions
Thread-Index: Acw1ekRqZgpWnTB4T6m5N4iD7AhplgB8sz+fAAAzYYM=
Message-ID: <ADF94D8555C7A246B86A633685E0178AB90A35CA97@planck.ka.sara.nl>
References: <CA2CD1F8.3F91C%scott@richrelevance.com>
 <4E086BC7.5000208@apache.org>
 <BANLkTimZ4SpW6wEGCGT=g6SfMXq4g2UaWw@mail.gmail.com>
 <9068C5A7-6090-42A3-8891-F60EF3B660BD@navteq.com>
 <BANLkTimi1XDyaV7P0meXV=v5z60izzNc6A@mail.gmail.com>
 <918BB6E5-EF4B-4EB2-9B55-8BCBAF6DBA7B@navteq.com>,<4E09A60C.1090605@apache.org>,<ADF94D8555C7A246B86A633685E0178AB90A35CA96@planck.ka.sara.nl>
In-Reply-To: <ADF94D8555C7A246B86A633685E0178AB90A35CA96@planck.ka.sara.nl>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/mixed;
	boundary="_002_ADF94D8555C7A246B86A633685E0178AB90A35CA97planckkasaran_"
MIME-Version: 1.0

--_002_ADF94D8555C7A246B86A633685E0178AB90A35CA97planckkasaran_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Forgot to attach the spreadsheet - here it is. All three tabs contain some =
data. The speeds are per second.

Evert
________________________________________
From: Evert Lammerts [Evert.Lammerts@sara.nl]
Sent: Thursday, June 30, 2011 11:31 PM
To: general@hadoop.apache.org
Subject: RE: Hadoop Java Versions

> You can get 12-24 TB in a server today, which means the loss of a server
> generates a lot of traffic -which argues for 10 Gbe.
>
> But
>   -big increase in switch cost, especially if you (CoI warning) go with
> Cisco
>   -there have been problems with things like BIOS PXE and lights out
> management on 10 Gbe -probably due to the NICs being things the BIOS
> wasn't expecting and off the mainboard. This should improve.
>   -I don't know how well linux works with ether that fast (field reports
> useful)
>   -the big threat is still ToR switch failure, as that will trigger a
> re-replication of every block in the rack.

Keeping the amount of disks per node low and the amount of nodes high shoul=
d keep the impact of dead nodes in control. A ToR switch failing is differe=
nt - missing 30 nodes (~120TB) at once cannot be fixed by adding more nodes=
; that actually increases ToR switch failure. Although such failure is quit=
e rare to begin with, I guess. The back-of-the-envelope-calculation I made =
suggests that ~150 (1U) nodes should be fine with 1Gb ethernet. (e.g., when=
 6 nodes fail in a cluster with 150 nodes with four 2TB disks each, with HD=
FS 60% full, it takes around ~32 minutes to recover. 2 nodes failing should=
 take around 640 seconds. Also see the attached spreadsheet.) This doesn't =
take ToR switch failure in account though. On the other hand - 150 nodes is=
 only ~5 racks - in such a scenario you might rather want to shut the syste=
m down completely rather than letting it replicate 20% of all data.

Cheers,
Evert

--_002_ADF94D8555C7A246B86A633685E0178AB90A35CA97planckkasaran_
Content-Type: application/vnd.oasis.opendocument.spreadsheet;
	name="recovery_speed.ods"
Content-Description: recovery_speed.ods
Content-Disposition: attachment; filename="recovery_speed.ods"; size=86421;
	creation-date="Thu, 30 Jun 2011 23:37:45 GMT";
	modification-date="Thu, 30 Jun 2011 23:37:45 GMT"
Content-Transfer-Encoding: base64

UEsDBBQAAAgAADar3j6FbDmKLgAAAC4AAAAIAAAAbWltZXR5cGVhcHBsaWNhdGlvbi92bmQub2Fz
aXMub3BlbmRvY3VtZW50LnNwcmVhZHNoZWV0UEsDBBQACAgIADar3j4AAAAAAAAAAAAAAAALAAAA
Y29udGVudC54bWzs3e2PW0eW5/n3+1cIWmBRPYBCceLhRIS3XIMmi9VvqjGz1duLBQaDRVpK2dmt
J2SmyuX56/eS8bsnT9y2y44uyy5eHMlm6OZJiZkkv8mnDy9/+1//8u7tsz/f3j/cfXj/5XNy/vmz
2/evPry+e//1l8//9f/+w4v6/L/+7n/77Yc3b+5e3X7x+sOrT+9u3z++ePXh/eOyPlv+9vuHL/r0
y+ef7t9/8eHm4e7hi/c3724fvnh89cWHj7fv17/1hf7sLy7H1T/y8Pjd25/81y+frP/24+1fHn/q
Xz5/7vB3b7766cd8+WT9t1/f33z7U//y+XOXE1X/9Tcffupf/svD2xdvPiyn+ruPN493m6/iL2/v
3v/7l8+/eXz8+MXLl99++637NroP91+/pNbay8tUvuBX8nkfP92/vXzW61cvb9/eno/s4SU5erl+
7rvbx5uf+vWdP1d/Se8/vfvq9v4nnzQ3jzf/4Vz9eH/7sHzK8u2eL5g/7R/Sf2e4fP3565986frz
1z9wMr/65ub+J1/OLp88XlTi659+UYmv9d99d/P4zQ+cv/XlPy/Dy8E///HpcnX/7qce1/lzh5Pq
1f3dx5/8bfbP1n//w4cP8qWe/0KP/fLlBu/Ty76tPvvbv/rp397fPd7eq09/9Vc//dXN21dyin94
930n2vJ59HL5jBe3fz5f5CWi8wnx8AN/IbzsY/nkh9c/+E//v//8x3959c3tu5unT7778U9+cff+
4fHm/dMpc38+E37wO80v728/frh/lBPmzU//4bucW0G+tm8e37394R8d5+n6qV/fv379vZ+6fDnx
5fJjZIn4xZ/vbr/934efrX/98tBeXj5JLrh3t2/XSuRz8e3c/uXj7f3d+Tu5eXu+ILx497CcaMuF
48PHL9TfHn++3r/7y0/7584XiA+v32z/RYkDf1ddUS6n4eUk+eLx/ub9w/kz8aMK3+xyuj+4y/nx
4nzkl++4/yvLT9nlAvXw9vHlcozh/vUbt2w8/916HdubengpH3izXNe+eHPz6vbF69tXbx9+99v+
s1I+/Kxvn7+vL5//8W75wXv5Sp79y/J1PX+2/EBbP/Xd3dvvvnz+f9x8/PDwf24+r3/w+bPhnz5/
/ouvb98vJ9SS3cO3dw8Pw2d8vHt8tfxg+vPN/d3lXHz517+039/+283/8+mvf1nqc37Kl/Tdw+Pt
ux/7ml7+0OmIj998evxwPvNevbj8O3ICXw6H7+DVB5Ijw5d9ufwu1xhvP717/3z9m/qDLz4ul6fb
+8e724dnbz588dX97c2/v/jqdrnALP/g+ajXfxGf/u3d6/NP++Uy0lK+e3/5BtTX89e+uPDLfXGJ
As99cfEXPOVqnDzl0i/3xVErYe6Ly7/gF5dimfvi+Jf64ryrLUyereUXvMwlP3m21l/wbI1h6my9
/6Gfc/cfvt18ZctH9JfVR+cPfnN79/U3j+fzjUpdjvyvf8WfHm6Xq8nHu3c3b1/ov/14/+n2p3/d
jzff/3WvH3y33M24vX/x8ebr2xfrddKbm09vHzfflPqG+n2+13cPH9/efIevB//a+bbpco/uxbsP
r5d/6e39i8evfvqX+vX9f/hSv76/+fjN3Sv5WrCtv5rzncgvPixf4fkPL24ePt6+Wk4kmjmF/r/l
HvD97ZvvP51+6snw5ubtw/ecLy9/8BoVg68+vP7u6YbOcnft5vXDN7e3j7/7bT+CyyGOrH/B/3Ie
LydV/9jln3uhzuz+4Y/3y+229asa/i2U8z1//XJdjm+rXwhevLp9+/aF/pz10vHyJ/+b4TP8m/Ez
/JvpSr7O/Bn+zc9xvvN/9t9cfth9zz94/gG8Oe7l35N7ITdvP92+ePzu4/KZD4/LZf/r8yefH9n6
+LvT4ze39+9vH58tPxtuXz/7zekffvsSo9++3P6D/7mj+Md3Hz69f1w+89n75Sffw7Pf/OPPfxzH
m483r+4ev3u2XDEs38W//vzH8Pu7h39/9mo9mt/8/vMcw8Oz5Sfo5XR69pv//vMfxR9u7t4upw/O
hz/8/Efwp9tXH5Y7vd89W66al2/hT8/mjuLl5rOWS/vPe/F/8/bDzePzYbJcH3r/9B0sG3/zqfK9
xxLUkYTPdBze8dORLBuf6zu5/NLfzuXX5zm29HQ86TN9P+o7mTuG/oHzgzmf3t58+fzDmy++/M3/
cH8I//O//A/3+8vh6XJ4DP/zH14uk3+8bB2Wrec//fuv+qQ+b019jb/77eVG4Jv7pRR8vbfvX/fr
m5vXr+9vHx7WW03uX5LcHjl/zl/6rXF/vhelPvzd5cMhn++/9FuY/+vF3fvXt+dPxwd0nP2W65+/
/gJ3L5rLJV7+7vmD6w33styX5LZ+9HLMsXBaP3A5Ts/x7nxPp9+q/erflhuz/ejef3i8e/Pdiw/v
X3z6+Prmcblr8ObF/c37r2+fvrVD+GL9U/b+Gf78T/LRf1o++vzZ5cmPL75Zbu9++dy9/G/9KGj9
OH7k3b37eHkI8vKxh2/Oj0jfvvvq9vX6oZtXj5+WL2K5OLz/44eb1+dr8MtXudxZ+fr2e4/iT7fL
zeRXeE7l5zvWl0/n/PdeVPb7o5ZU0WQ/anf8ozZeftReDk+Xw2PEj9rL1iFO/ajNhVwKdVkuh+rk
2EzsVs16hupzdPIstdSuKrV0Se1yeLocHhNSu2wd0lRqkaPjuP7WqW0mltp6hkZ1jkZLbcep5Utq
l8PT5fCYkdpl65CnUqMSXaPoU0ylBnUGbyeW2nqGqstM+EwXGkvt7yI1vqR2OTxdDo+M1C5bB55K
bTip0+zpvOuksjrnsiW146TKJanL4elyeCxI6rJ1KDNJxZrUSX3esqTWM05d2ievvC2p60qqXpK6
HJ4uh8eKpC5bhzqVFLfgoi/LcjlUeW0mltp6hqo7qGHuLqmldl2ptUtql8PT5fDYkNpl69CmUss5
uyy/1Bm8nVhq6xla1TlaLbUdp0b+0lpfTn05LkvPrW8fztsTwaVQ3fooPSX1kNl2YsGtZ6u6ARDm
rvItuCsLjnpw1IOjHhytwVEPjqaCi+RdTKmGkivrZ163EwtuPcn0HVqTQLsOrlOgvpz6cqRVA/Xt
A015oBj0CX7esrDWk0b/9DH3seuwOvzoy6kvR1rtR98+0JT+iL6xKyVRiznVqB8W2UwsOPlZpH8S
WXB7Dq7zj76c+nKkVYD07QNNGZA4nODRni3Tt6r17WgLa89hdezRl1NfjrR6j759oCnxEZpvzrf1
t9awm4kFJ48b6UeKLLg9B9fJR19OfTnSqj769oGm3EeoIbocWiIqnIoObjOx4OSZEf1ciAW35+A6
COnLqS9HWk1I3z7QlAoJJQVXnx7N18/OjhMLbg1OP/loWmTXwXUu0pdTX460ipG+faApMxKY2bH8
0hxiM7Hg1uDUiTT5MgYL7sqC62ikL6e+HGl1I337QFNyJOSW3PJ//09Lv3Fgua25KTcSzY3sObfQ
3UhfTn05htWN9O1DmHIjIQd2kXKpLZXI2iBtJhbcGpx+hsTcyK6D626kL6e+HMPqRvr2IUy5kZCY
XI51WS6H+gVX48SCWy84w2uLLLg9B4ddyGAfMtiJjOxFBruRmXIjIQ0neLJn254uIOoB22RuZNdh
dTfSl1NfjmF1I337EKbcSIiJXOLl8HKgX+e4mVhw6wVHna/J3Miug+tupC+nvhzD6kb69iFMuZEQ
anbyAP/A1zcTC2694ChPMvnSBwvuyoLrnqQvp74cw+pJ+vYhzHmSEIPLuVJsOaaqHxzZTCy49YKj
LznmSXYdXPckfTn15RhWT9K3D2HOk1AlV0l+qy95M7Hg1guOut2dzJPsOrjuSfpy6ssxrJ6kbx/C
nCehGF18+qWDGycW3HrB0TsSM0+y6+C6J+nLqS/HsHqSvn0Ic57EV3Ytcw5UYiMd3GZiwa0XHL2T
TPMkuw6ue5K+nPpyDKsn6duHMOdJfDrfU4vUKNWmMeB2YsGtFxx1uzuZKNlzcLGLkr6c+nKMqyjp
24c4J0r+wwluYa0XEPWQbTI5suuwuhzpy6kvx7jKkb59iFNyhFpujmrkEhM3rZC2EwsOx5LVkWST
I7sOrsuRvpz6coyrHOnbhzglR6jp9646b1lYa1hKjmSTI7sOC281g/eawZvNyLvN4O1mpuQI1Rpc
fHp1qLrAbiYW3BqcOl+zyZFdB9flSF9OfTnGVY707UOckiNUEzsSW5x1cOPEgluDUw/NZpMjuw6u
y5G+nPpyjKsc6duHOCVHqBK5GHzJqRC3qoMbJxbcGpy65GSTI7sOrsuRvpz6coyrHOnbhzglR6iU
4or80sFtJhbcGpy6GZBNjuw6uC5H+nLqyzGucqRvH+KUHKGSskt5/a1vUm4mFtwanGph8o0PLLgr
C67Lkb6c+nKMqxzp24c4JUeoUHJP78RLOrhxYsGtwalHlibfu9iCu7Lguhzpy6kvx7jKkb59iFNy
hLguWZHPgc+7QdDPt20mFtwanLrdnU2O7Dm41OVIX059OaZVjvTtQ5qSI8Q5OyohUSxN79N+M7Dc
1tzUz6RsnmTXuXVP0pdTX45p9SR9+5DmPAmH4ogqp+R5eMRkHFhuOBZWR8KmSXadW9ckfTn15ZhW
TdK3D2lOk/BwIWVzkU9hqet6Nk2y67C6JunLqS/HtGqSvn1Ic5okl+hK8alR9VG/jeZ2YsGtwanz
lU2T7Dq4rkn6curLMa2apG8f0pwmyam6WItvfHlzKB3cOLHg1uCUJmHTJLsOrmuSvpz6ckyrJunb
hzSnSXKIrvq2LJdDHdw4seDW4NQlh02T7Dq4rkn6curLMa2apG8f0pwmycOFNNt9taew1DP9bGpk
12F1NdKXU1+OaVUjffuQ5tRIKuxaiL7w+VDfV9tMLLg1ONUCmxrZdXBdjfTl1JdjWtVI3z6kOTWS
cnL5SWPp4MaJBbcGp9TI5Nv6WHBXFlxXI3059eWYVjXStw9pTo2kGFyNuQZfWx7uq20mFtwanLoZ
MPk+IxbcdQWXuxrpy6kvx7yqkb59yHNqJBG582tFfa41xOEabpxYcGtw6iFbNjey6+C6G+nLqS/H
vLqRvn3Ic24kNnLRp5hKDez1w/+biQWHYynqSIrJkV0H1+VIX059OeZVjvTtQ56TI7GQS0/AXwc3
Tiy4NTglSiZfEGHBXVlwXZT05dSXY15FSd8+5DlREnNwRIEL+8wx6eDGiQW3BqfO12KiZNfBdVHS
l1NfjnkVJX37kOdESfyh3ZBvJxbcGpw6kYqJkl0H10VJX059OeZVlPTtQ54TJZGy81xTW/7gebgP
N04suDU4dckpJkp2HVwXJX059eWYV1HStw95TpSEVtzyf/9PX8GNA8ttzU09kFvMmew6t+5M+nLq
yzGvzqRvH/KcMwlVX0iXLQtrDUtd5ot5kl2H1T1JX059OebVk/TtQ57zJIGjo1xqSyUutxJ1ZOPE
gluDUw/YFvMkuw6ue5K+nPpyzKsn6duHPOdJQmJXclyW86F+enszseDW4JQnmXz5ugV3XcFx9yR9
OfXlyKsn6dsHnvMkIXrXEfLlUAc3Tiy4NTj1+NHk3pAsuCsLrnuSvpz6cuTVk/TtA895kkDZUW21
xERFP5G0nVhwOBZ953byrq0Fd2XBdU/Sl1Nfjrx6kr594DlPEoYLabBXtz2FpdxINTey67C6G+nL
qS9HXt1I3z7wnBuher6+yvhPP6u2mVhwa3DqfK3mRnYdXHcjfTn15cirG+nbB55zI1SWe2RxOewH
OrhxYsGtwak7tNXcyK6D626kL6e+HHl1I337wHNuhDI7Dpw9+5D0LlrHgeW25qYuN5PPiVhuV5Zb
VyN9OfXlyKsa6dsHnlMjlIJ72rG/frZtM7Hg1uDUMySTb2dnwV1ZcN2N9OXUlyOvbqRvH3jOjVBo
Du89en7VqA5unFhwa3CqhWqeZNfBdU/Sl1Nfjrx6kr594DlPQsQulOa5FS5puIYbJxbcGpy6m1vN
k+w6uO5J+nLqy5FXT9K3DzznSchHl1INJVcO40Mm48SCW4NTNwMmr/gtuOsKrnRP0pdTX45l9SR9
+1DmPIlv3jUvv/UFdpxYcGtw6kSq5kl2HVz3JH059eVYVk/Stw9lzpP4Uh1n4ppji4H1BXacWHA4
lqaOpJkn2XVw3ZP05dSXY1k9Sd8+lDlP4pkdyy99k3IzseDW4NRTJ82cya6D686kL6e+HMvqTPr2
ocw5E5+Taykvy/lwuIYbJxbcGpw6X5s5k10H151JX059OZbVmfTtQ5lzJj5F13dqV7kNe+DaTCy4
NTjlTJo5k10H151JX059OZbVmfTtQ5lzJj4GF3L1nJa7a6RfhLOZWHBrcOqS00ya7Dq4Lk36curL
sazSpG8fypw08YFcKJyX67CU9B65txMLbg1OSZNm0mTXwXVp0pdTX45llSZ9+1DmpIkn73LgiF0m
6AvsOLHg1uBUC82kya6D69KkL6e+HMsqTfr2ocxJE7+5kNqr3iQs9dOnmSjZdVhdlPTl1JdjWUVJ
3z6UKVHSanPcfAnEqSXtb7cTC05OMn0iWXA7Dq52UdKXU1+OdRUlfftQp0RJK83lRjVyiYmH+/2b
iQW3nmRKlDQTJbsOrouSvpz6cqyrKOnbhzolShqfr8fW3/rG0jj4FXPrH3j/6d1Xt/cvXn14++nd
+4cX97cfb28eb1+fT+W/syAtlr+PWLoG6cupL8e6apC+fahTGqSxxkRs97LUIz7DgzwW1p7D6uqj
L6e+HOuqPvr2oU6pj5a9S8sB/teCaDOx5J6e1RieyLDk9pxcdx99OfXlWFf30bcPdcp9tESOnt5B
VD/+vJlYck/P3A9P1ltye06uy4++nPpyrKv86NuHOiU/WgzOx1rj+YosZZ3cZmLJybHol816sx+7
Tq7bj76c+nKsq/3o24c6ZT9aiM5/7x6PtxNLTo5F74jdm/7YdXJdf/Tl1JdjXfVH3z7UKf3RKLnw
vTv22U4sOTmW4XUO5j92nVz3H3059eVYV//Rtw91yn80nx2zj6XEXNPwAqzNxJKTYxlYmsmQXSfX
ZUhfTn051lWG9O1DnZIhtRVHjQMT1xCS3vPoZmLJybEM9NpsyJ6Ta92G9OXUl2NbbUjfPrQpG1Jr
dfXpl96RxmZiycmxjPtkseT2nFzXIX059eXYVh3Stw9tSofUet7TeGwptZwoDMmNE0tuPRbSx0Jm
THadXDcmfTn15dhWY9K3D23KmNQSXAnyW9+w3EwsOUlu2Fu76ZNdJ9f1SV9OfTm2VZ/07UOb0ieV
k1v+X//Te9neTCw5SU7rEzJ9suvkuj7py6kvx7bqk759aFP6pObi1qcBzocquc3EkpPktD6Z3DuE
JXdlyXV90pdTX45t1Sd9+9Cm9ElNzeVccggUYtUvrdlOLDlJbnxbEktuz8l1fdKXU1+ObdUnffvQ
pvRJTcGRl/0d6BuWm4klJ8lpfUKmT3adXNcnfTn15dhWfdK3D21Kn9SYnOyyjrN+C87NxJKT5LQ+
IdMnu06u65O+nPpybKs+6duHNqVPaiju8rYaXBvnIbnNxJKT5Mb3c7Xk9pxc1yd9OfXl2FZ90rcP
bU6fBO8yhWW5HOp3dt9MLDlJbnjHO9Mne05uuRRdmsN6wno8rz07fORw+chEeBRdzi2G4GOsel83
24mFJ+Fpg0JmUPYdHiE8QniE8EjCI4Q3J1E8uxLCckPSl8z6WfFxYNmtxxL0sQRzKPvOLiC7gOwC
sguSXUB2cxpFn+rV9iup8tLmZPJ2tuV1bXlF5BWRV0ReUfKKyGtKnpQWXaw1hVZy8KzO7O3EwpPw
tDwJJk/2HV5CeAnhJYSXJLyE8Kb8Sansas0h5eaJo7qHsp1YeBKe9ieTL5i38K4tvIzwMsLLCC9L
eBnhTSmUUr1LdTnsB0N448TCk/C0QgmmUPYdHiM8RniM8FjCY4Q3ZVFKSY5azKnGWrze0+t2YuFJ
eNqiBLMo+w6vILyC8ArCKxJeQXhTIqXod5wq9iZTKi/tTibfstzyura8KvKqyKsiryp5VeQ1pU8K
k2s+LcvlMOjWxomFJ+FpfRJMn+w7vIbwGsJrCK9JeA3hTRmUkvOSV85Elcjr+yjbiYUn4WmDEsyg
7Do8gkEhGBSCQSExKASDQlMGpWSvW7Pn5J7y0tJkcp8Ulte15QVpQpAmBGlCIk0I0oSmpElJyVFl
n3i5AmP9E3s7sfDWY4n6WKJZk32HB2tCsCYEa0JiTQjWhKasSYnVJaYc6+VQPzWwmVh4Ep5WKNEU
yr7Dg0IhKBSCQiFRKASFQnMKJQZXQ0qhpOqbfjXYdmLhSXhaoURTKPsODwqFoFAICoVEoRAUCs0p
lFBcULv0UuFtJhaehKcVSjSFsu/woFAICoWgUEgUCkGh0JxCCeSqT8vduFAqNX2Nt5lYeBKeVijR
FMq+w4NCISgUgkIhUSgEhUJzCoXYJSqNfEqlsLr3sp1YeBKeVijRFMq+w4NCISgUgkIhUSgEhUJz
CoXIkfqlwxsnFp6Ep33K5PtgWnjXFh58CsGnEHwKiU8h+BSa8yk+u/PbqObQ0tKsvo+3mVh4Ep72
KdF8yr7Dg08h+BSCTyHxKQSfQnM+xXtXQmth+cN51eGNEwtPwtM+JZpP2XV4AT4lwKcE+JQgPiXA
p4Qpn8ItO86BSmwUl3tzT1/4dmLhSXharkw+wWnhXVt4kCsBciVArgSRKwFyJUzJFW7LmZpKqsFT
ivoNsrYTC289lqSPJZlc2Xd4kCsBciVArgSRKwFyJUzJFa7Zlac3LVBn9nZi4Ul4Wq4kkyv7Dg9y
JUCuBMiVIHIlQK6EKbnC1buayXOsoZVadHjjxMKT8LRcSSZX9h0e5EqAXAmQK0HkSoBcCVNyhQs7
nznG6kMl1jc1NxMLT8LTcmXyfaEtvGsLD3IlQK4EyJUgciVAroQpucKFXFxuSBa6HAYd3jix8CQ8
LVcmH3Wy8K4tPMiVALkSIFeCyJUAuRKm5AozO376pW9qbiYWnoSn5UoyubLv8CBXAuRKgFwJIlcC
5EqYkivMweENRKgEfe9lO7HwJDwtV5LJlX2HB7kSIFcC5EoQuRIgV8KUXOFcXI4ppOUPy6ruvWwn
Fp6Ep+VKMrmy7/AgVwLkSoBcCSJXAuRKmJIrnONyvUYhpFRb0a9O2E4sPAlPy5VkcmXX4UXIlQi5
EiFXosiVCLkS5+RKqm75f/1PP6q5mVh4Ep6WK5MnjIV3beFBrkTIlQi5EkWuRMiVOCdXUnKh5eiD
p+V2pb6puZlYeOux6L1A0eReoCy8awsPciVCrkTIlShyJUKuxDm5ov0TT/qnfeelfUo2n7LvvOBT
InxKhE+J4lMifEqc8ykxu8KUa0s+tqIp9GZi4Ul42qdMvme0hXdt4cGnRPiUCJ8SxadE+JQ451Mi
uVyWuErkmvR7tW0nFp6Ep31KNp+y7/DgUyJ8SoRPieJTInxKnPMpobiUfavLtVqgrJ8m30wsPAlP
+5RsPmXf4cGnRPiUCJ8SxadE+JQ451NCdLHwslwO9U3NzcTCk/C0T8nmU/YdHnxKhE+J8ClRfEqE
T4lzPoWakzfP4qKeh9pOLDwJT/uUbD5l3+HBp0T4lAifEsWnRPiUOOdTKDvsEfp8qJ8m30wsPAlP
+5TJ2+AW3rWFB58S4VMifEoUnxLhU+KcTyFyiVPjSjlUHsIbJxaehKd9SjafsuvwEnxKgk9J8ClJ
fEqCT0lzPsUXl1tqkYuvVe89czux8CQ87VOy+ZR9hwefkuBTEnxKEp+S4FPSnE/x0ZUSc03NZx7D
GycW3nosrI+FzafsOzz4lASfkuBTkviUBJ+S5nyKPtV58hTfd17ap0z+3LG8ri0v+JQEn5LgU5L4
lASfkqZ8Sm7sQomhUQs56Xty24mFJ+FpnzL5rtIW3rWFB5+S4FMSfEoSn5LgU9KUT8ktLPfXwvpf
1OGNEwtPwtM+hc2n7Ds8+JQEn5LgU5L4lASfkqZ8Sq7VtVx8zoFS0U+TbycWnoSnfQqbT9l3ePAp
CT4lwack8SkJPiVN+ZRcs4u8HPaDosMbJxaehKd9CptP2Xd48CkJPiXBpyTxKQk+JU35lFzJVZLf
QYc3Tiw8CU/7lMk7vxbetYUHn5LgUxJ8ShKfkuBT0pRPyaW6SDHk6Cl4vYOC7cTCk/C0T5nclZqF
d23hwack+JQEn5LEpyT4lDTlU3JJrmYfWkueqA3hjRMLT8LTPoXNp+w6vAyfkuFTMnxKFp+S4VPy
lE/JhZy810HWO6fdTiw8CU/7lMndZVt41xYefEqGT8nwKVl8SoZPyVM+JXN1PpUYy/ldy71+cGUz
sfDWYyn6WIr5lH2HB5+S4VMyfEoWn5LhU/KUT8mcXPFZ3sZVhzdOLDwJT8uVYnJl3+FBrmTIlQy5
kkWuZMiVPCdXeLlB6WP2tRQanj8fB5adZKfdyuQtcMvu2rKDW8lwKxluJYtbyXArec6t5OoothxT
rctNSn19t5lYeBKedivF3Mq+w4NbyXArGW4li1vJcCt5zq3k5BrF5ZZkaYWrfkxzM7HwJDztViZ3
sWbhXVt4cCsZbiXDrWRxKxluJc+5lUyuhESxtEjL7Uod3jix8CQ87VaKuZV9hwe3kuFWMtxKFreS
4VbynFtJ1eWn905WD5NvJxaehKfdSjG3su/w4FYy3EqGW8niVjLcSp5zKym7lOW3flHQZmLhSXja
rRRzK/sOD24lw61kuJUsbiXDreQ5t5KCiyXUSjHnFvXT55uJhSfhabcyuYs1C+/KwmO4FYZbYbgV
FrfCcCs851ZiczHkmgI35uGm5mZi4Ul42q0Ucyv7Dg9uheFWGG6Fxa0w3ArPuZXILlKoqREHT/rp
hM3EwluPpepjqeZW9h0e3ArDrTDcCotbYbgVnnMrMbr49Eu/DHYzsfAkPO1WqrmVfYcHt8JwKwy3
wuJWGG6F59xK9C5WLsnHSKnp+3ibiYUn4Wm5MvkyRQvv2sKDXGHIFYZcYZErDLnCc3Ll/PYjJSzL
5VA/gb6ZWHgSnpYr1eTKvsODXGHIFYZcYZErDLnCc3IlJJebX64nW0wp6KcTNhMLT8LTcqWaXNl3
eJArDLnCkCsscoUhV3hOrgRyJTaKPsVU9M/y7cTCk/C0XKkmV/YdHuQKQ64w5AqLXGHIFZ6TK1Rd
o7r+p59O2EwsPAlPy5VqcmXf4UGuMOQKQ66wyBWGXOE5uULsKDQfQ8jV671nbScWnoSn5Uo1ubLv
8CBXGHKFIVdY5ApDrvCcXKHoItdWY8sljNd448TCk/C0XJl82YaFd2XhFciVArlSIFeKyJUCuVLm
5AotZ2qsoZVaQhxef76ZWHgSnpYr1eTKvsODXCmQKwVypYhcKZArZU6u+OJa5GW5HOqnEzYTC289
lqaPpZlc2Xd4kCsFcqVArhSRKwVypczJFZ9d4Ei51JbKsB/pzcTCk/C0XGkmV/YdHuRKgVwpkCtF
5EqBXClzcsUHx1SqJ58KDbt+2EwsPAlPy5VJ0mPhXVt4kCsFcqVArhSRKwVypczJFX2qZ3sPPJWX
9imT+3qyvK4tL/iUAp9S4FOK+JQCn1KmfEpqxSXfYuAUUtU7p9tOLDwJT/uUZj5l3+HBpxT4lAKf
UsSnFPiUMuVTUkuuphzZ15BKCjq8cWLhSXjapzTzKfsODz6lwKcU+JQiPqXAp5Qpn5JacNEXbqEf
6vDGiYUn4Wmf0syn7Ds8+JQCn1LgU4r4lAKfUqZ8SqrNlZYb1cglJvXY5XZi4Ul42qc08yn7Dg8+
pcCnFPiUIj6lwKeUKZ+y3Htbrtcap8TF+6zv420mFp6Ep33K5FslWXhXFl6FT6nwKRU+pYpPqfAp
dcqnpHq+Jye/sw5vnFh4Ep72Kc18yr7Dg0+p8CkVPqWKT6nwKXXKp6QaXKLg2UdPg0/ZTiy8pwvr
cEG18HYdHnxKhU+p8ClVfEqFT6lTPiXp/fOkyf3z7Dsv0hdHUyj7zgsKpUKhVCiUKgqlQqHUKYWS
SnFM1JIPrY7Xa5uJhScXVn32elMo+w4PCqVCoVQolCoKpUKh1CmFkkp2YTlY/9fhjRMLTy6sUV9Q
zafsOzz4lAqfUuFTqviUCp9S53xKCa55Tj4uV21Vv1xzO7Hw5MKa9AXVfMq+w4NPqfApFT6lik+p
8Cl1zqcU757ev5WGm5rjxMKTC2vWF1TzKfsODz6lwqdU+JQqPqXCp9Q5n8LVhRa44kCdNJuJhScX
VtYXVPMp+w4PPqXCp1T4lCo+pcKn1Dmfwuw8LVdpZbkbV/Q+MrcTC08urEVfUM2n7Ds8+JQKn1Lh
U6r4lAqfUud8CkdXuBLlloIfr/HGiYUnF9aqL6jmU3YdXoNPafApDT6liU9p8CltzqcwuRzrslwO
NYXeTCw8ubA2fUE1n7Lv8OBTGnxKg09p4lMafEqb8ym5uRg9pZiTZ/2g5jiw7NazlvTFlEyn7Ds7
6JQGndKgU5rolAad0uZ0Si5O3jh5OdTdjRMLT8LTbmXyncgsvGsLD26lwa00uJUmbqXBrbQ5t5KT
a4Xj+c19So6swxsnFp6Ep89eMrey7/DgVhrcSoNbaeJWGtxKm3MrObi63J6slUsLSbuVzcTCk/C0
WyFzK/sOD26lwa00uJUmbqXBrbQ5t5K9Kz6FkGNebjZFHd44sfAkPO1WyNzKvsODW2lwKw1upYlb
aXArbc6tpOpyqxR9TURBP6a5mVh4Ep52K2RuZd/hwa00uJUGt9LErTS4lTbnVhK7TMShNM+t6AdX
NhMLT8LTboXMrew7PLiVBrfS4FaauJUGt9Lm3EpKLj390vfxNhMLT8LTboXMrew7PLiVBrfS4Faa
uJUGt9Lm3EoKLjaOFCMnn4fwxomFJ+FptzJpxy286wpvOT0u4WE9YT2e1x4ePnK4fGQmPO8iLzcl
QyrFsz5pNhMLT8LTbmXyncgsvGsLjxAeITxCeCThEcKbcyuxupiZePnDsuoHVzYTC289c4O+oAaT
K/sOLyC8gPACwgsSXkB4c3Il8nK99vRbhzdOLDwJT8uVyfdlsfCuLbyI8CLCiwgvSngR4c3JlZhc
rI2SD4WSfsuN7cTCk/D02Tv5PIuFd23hJYSXEF5CeEnCSwhvTq7E4Jb/1/+G8MaJhSfhabkSTK7s
O7yM8DLCywgvS3gZ4c3JlejdcnUWSswpsN5t3XZi4Ul4Wq4Ekyv7Do8RHiM8Rngs4THCm5Mrobr1
9QfLoX4Z7GZi4Ul4Wq4Ekyv7Dq8gvILwCsIrEl5BeHNyJbDjp19armwmFp6Ep+VKMLmy7/AqwqsI
ryK8KuFVhDcnV0JypdTC5Jna8I5Am4mFJ+FpuRJMruw7vIbwGsJrCK9JeA3hzcmVcN53Zkm1tsql
6pNmM7HwJDwtVyb3AWXhXVl4BLlCkCsEuUIiVwhyhebkSiDnc+BIudTNNd44sfAkPC1XJu/8WnjX
Fh7kCkGuEOQKiVwhyBWakyvUXKDYSiytkb4RtZ1YeOuZG/UFNZpc2Xd4kCsEuUKQKyRyhSBXaE6u
UHGxUfQpplL1wwbbiYUn4Wm5Ek2u7Ds8yBWCXCHIFRK5QpArNCdXKLtc07JcDofwxomFJ+Hpszea
XNl3eJArBLlCkCskcoUgV2hOrlB0pUXyMaUayhDeOLHwJDwtVyZJj4V3beFBrhDkCkGukMgVglyh
OblCwXkqxCVWakWfNJuJhSfhabkSTa7sOzzIFYJcIcgVErlCkCs0J1fIu5B5WS6HurthYNlJdtqt
RHMr+84OboXgVghuhcStENwKzbkVX10mz7GGVmrRRHozsfAkPO1WJl8tZeFdW3hwKwS3QnArJG6F
4FZozq14dqU0ZvLELWmpuZlYeBKedivR3Mq+w4NbIbgVglshcSsEt0JzbsVnt1yrhUYUWxhuaW4m
Fp6Ep91KNLey6/AC3EqAWwlwK0HcSoBbCXNuxUcXmWJKOZaq9xe5nVh4Ep52K9Hcyr7Dg1sJcCsB
biWIWwlwK2HOrXhyXM57EaPLosMbJxbeeuYmfUFN5lb2HR7cSoBbCXArQdxKgFsJc25Fn+pp8hTf
d15ap0ze0ra8ri0v6JQAnRKgU4LolACdEqZ0SmzVxeRDqo0DZ3VmbycWnoSnz95kOmXf4UGnBOiU
AJ0SRKcE6JQwpVNiY8ctBUq1sNfuYjux8CQ8rVOS6ZR9hwedEqBTAnRKEJ0SoFPClE6J7fwIJZVQ
lz95ve/M7cTCk/C0TkmmU/YdHnRKgE4J0ClBdEqATglTOiW26FIKHMJSVx26GwaWnWSndcrk20ZY
dteWHXRKgE4J0ClBdEqATglTOiU2cjVyiYlbrKXq7saJhSfhaZ2STKfsOzzolACdEqBTguiUAJ0S
pnRKbN4t91Kax4EOb5xYeBKe1imTL7638K4tPOiUAJ0SoFOC6JQAnRKmdEqs1THHxoXp/M4/KrzN
xMKT8LROSaZTdh1ehE6J0CkROiWKTonQKXFKpyw3Ip1f4iqJWsz6IrWdWHgSntYpk1DVwru28KBT
InRKhE6JolMidEqc0imxZpdTIK5cljtzTYc3Tiy89czN+oKaTafsOzzolAidEqFTouiUCJ0Sp3RK
1Pvsi7abPpWX1inZdMq+84JOidApETolik6J0ClxTqfU4FLxRC0HaqSfM9hMLDwJT5+92XTKvsOD
TonQKRE6JYpOidApcU6nVO+Wspblckg6vHFi4Ul4Wqdk0yn7Dg86JUKnROiUKDolQqfEOZ1SmkuJ
OflIHLx+7HIzsfAkPK1TsumUfYcHnRKhUyJ0ShSdEqFT4pxOKeV8vVZybj7n4abmZmLhSXjap2Tz
KfsODz4lwqdE+JQoPiXCp8Q5n1L4fE8u17pcVYamH0/ZTCw8CU/7lMmXZlh41xYefEqET4nwKVF8
SoRPiXM+pWTdmr1i/CkvrVCyKZR95wWFEqFQIhRKFIUSoVDinEIp0eXka+blDhvpd63ZTiw8CU8r
lGwKZdfhJSiUBIWSoFCSKJQEhZLmFEoJzrfoQ1zuk1QablCOEwtPwtMKJZtC2Xd4UCgJCiVBoSRR
KAkKJc0plLKcqZmX5XKonzTYTCy89cxlfUFlUyj7Dg8KJUGhJCiUJAolQaGkOYXCzYXoC7d+qMLb
TCw8CU/7FDafsu/w4FMSfEqCT0niUxJ8SprzKVxcJfaN2nKHLuidOGwmFp6Ep8/eyRfZW3jXFh58
SoJPSfApSXxKgk9Jcz6F2SUKgWJJPumHDbYTC0/C0z6FzafsOzz4lASfkuBTkviUBJ+S5nwKZ+ep
eaLkuer9gmwnFp6Ep33K5A5CLbxrCw8+JcGnJPiUJD4lwaekOZ/C0XGU30N448TCk/C0T2HzKfsO
Dz4lwack+JQkPiXBp6Q5n8LBLT+1KfuWYmxaZG4mFp6Ep30Km0/Zd3jwKQk+JcGnJPEpCT4lzfkU
9q75EHLm2Hwb7uONEwtPwtNyhU2u7Ds8yJUEuZIgV5LIlQS5kubkSm4uZ5+ZSitFv6xlO7HwJDwt
V9jkyq7Dy5ArGXIlQ65kkSsZciXPyZVcXfC15RxCrMOr7jYTC0/C03Jl8glOC+/awoNcyZArGXIl
i1zJkCt5Tq5kdrWUGGJaTnP9juXbiYW3nrlFX1CLyZV9hwe5kiFXMuRKFrmSIVfynFzJeblBKb/0
gyubiYUn4Wm5MmnpLLxrCw9yJUOuZMiVLHIlQ67kObmS03kXtDnhQIc3Tiw8CU+fvcXkyr7Dg1zJ
kCsZciWLXMmQK3lOruTgWiIq3F/aqsMbJxaehKflSjG5su/wIFcy5EqGXMkiVzLkSp6TK5kcp7os
l0O9g4fNxMKT8LRcmbzza+FdW3iQKxlyJUOuZJErGXIlz8mV7F3k1Hj5Q9QkYzOw7CQ77VYmX5hv
2V1bdnArGW4lw61kcSsZbiXPuZXUnG9Pv1V3m4mFJ+Fpt1LMrew7PLiVDLeS4VayuJUMt5Ln3Eoq
rgb2lVvmHPQNzc3EwpPwtFsp5lb2HR7cSoZbyXArWdxKhlvJc24lscvFx1aIqJF2K5uJhSfhabcy
+RaAFt6VhcdwKwy3wnArLG6F4VZ4zq2k7GKIJS2X00LDLvw2EwtPwtNupZhb2Xd4cCsMt8JwKyxu
heFWeM6tpOSWW5M+UPY56nsv24mFt565VV9Qq7mVfYcHt8JwKwy3wuJWGG6F59xKCq7mQilcDoeb
muPEwpPwtFuZJKwW3rWFB7fCcCsMt8LiVhhuhefcSiJ33o8Y+fN7uOodtW4nFp6Ep8/eam5l3+HB
rTDcCsOtsLgVhlvhObeSvEsh50iNUh3v440TC0/C026lmlvZd3hwKwy3wnArLG6F4VZ4zq3E5kJY
7himmnyr+nm8zcTCk/C0W5l8V04L79rCg1thuBWGW2FxKwy3wnNuJVbnlzsq3Dyd99auwxsnFp6E
p+XK5Alj4V1beJArDLnCkCsscoUhV3hOrkR2NQXyeblzSMPO/TYTC0/C03KlmlzZd3iQKwy5wpAr
LHKFIVd4Tq7E7JjTslwO9dMJm4mFJ+FpuVJNruw7PMgVhlxhyBUWucKQKzwnV2JyqaVSYqZGw879
NhMLT8LTcmXy1VIW3pWFVyBXCuRKgVwpIlcK5EqZkysxuqh+6fDGiYUn4Wm5Mvmok4V3beFBrhTI
lQK5UkSuFMiVMidXYnBUWou+tFzTcB9vnFh465nb9AW1mVzZd3iQKwVypUCuFJErBXKlzMmVSM7H
lGoouXLQ+xjbTCw8CU/LlWZyZd/hQa4UyJUCuVJErhTIlTInV0JztXHzJSz345p+OmEzsfAkPH32
NpMr+w4PcqVArhTIlSJypUCulDm5EqorXCgsfzivOrxxYuFJeFquNJMr+w4PcqVArhTIlSJypUCu
lDm5EorjlH0mailHLVc2EwtPwtNyZdLSWXjXFh7kSoFcKZArReRKgVwpc3IlsMvRM4Ww3J0b3jlh
M7HwJDwtVyZvg1t41xYe5EqBXCmQK0XkSoFcKXNyJWSXQiyhVoo5D9d448TCk/C0XJl8EyUL79rC
g1wpkCsFcqWIXCmQK2VOroTklv/lPx3eOLHwJDwtV5rJlX2HB7lSIFcK5EoRuVIgV8qcXAnRhfj0
W4c3Tiw8CU/Llcnb4BbelYVXIVcq5EqFXKkiVyrkSp2TKyE4SoXzcjnlQsOjmuPEwpPwtFxpJlf2
HR7kSoVcqZArVeRKhVypc3IlkPPFh5iIYiG9e7/NxMJbTzJ9LPFzHYuF93cSHuRKhVypkCtV5EqF
XKlzciXoS1GYO8X3nZe+9vfmU/adF3xKhU+p8ClVfEqFT6lzPoWqa5FrKb5wGq7XNhMLT8LTp5I3
n7Lv8OBTKnxKhU+p4lMqfEqd8ylUXK2+tFQ8Nf16su3EwpPw9ANN3nzKvsODT6nwKRU+pYpPqfAp
dc6nELu6HOD/IbxxYuFJePqpFW8+Zd/hwadU+JQKn1LFp1T4lDrnUyi7UpdrteVGZQrDY5ebiYUn
4Wkw7s2n7Ds8+JQKn1LhU6r4lAqfUud8CiVX8nlnYZE9D2+3vJlYeBKePpW8+ZR9hwefUuFTKnxK
FZ9S4VPqnE+h6ErIqS3349jrff9vJxaehKdvkHvzKfsODz6lwqdU+JQqPqXCp9Q5n0LBFR9zPaPn
Ouw9czOx8CQ8Dca9+ZRdh9fgUxp8SoNPaeJTGnxKm/MpRI5rXZbL4fB0wjix8CQ8/eNp8n05Lbxr
Cw8+pcGnNPiUJj6lwae0OZ9Cy5laGi95ZZ+G3dZuJhbe00k2nEoW3q7Dg09p8CkNPqWJT2nwKW3O
p/i25FUStZhTjePP8mFi4T3dSBhuF1h4uw4PcqVBrjTIlSZypUGutDm54utyg5K4BB9yGV6DsJlY
eE93i4d7whbersODXGmQKw1ypYlcaZArbU6u+OK4hWW5HA4PG4wTC+/pgeDhsV8Lb9fhQa40yJUG
udJErjTIlTYnVzy74lvgTCUXGm5qjhML7+mpz+HZTgtv1+FBrjTIlQa50kSuNMiVNidXfHYlhvMb
bpXzGyEMJmOYWHgS3vAQlMmVfYcHudIgVxrkShO50iBX2pxc8ckVpubT5XDAUOPEwpPwBt9jcmXf
4UGuNMiVBrnSRK40yJU2J1d8dKWVxDXmlEZ2rweWnWQ3voLDstt1dnArDW6lwa00cSsNbqXNuRUf
XI2NiJc7cWHsbpxYeBKefgCKzK3sObzou1vBesJ6PK89PHzkcPnITHjkauWypJWHN/DeDCw7yW5A
daZW9p0dITtCdoTsSLIjZDenVrx3LfkUay2UytDdOLHw1pPsb9kfhoV3beEFhBcQXkB4QcILCG9S
rfwN++bZd17DPtXMpuw7r4i8IvKKyCtKXhF5TdmU5W6f85xT8Jl9ScOeH8eJhSfhDftUM5uy7/AS
wksILyG8JOElhDdlU0KrjiJzyo1DicP72YwTC0/CG3ffa+HtOryM8DLCywgvS3gZ4U3ZlNCKC0tf
IYRGXv8s304sPAlv3GG9hbfr8BjhMcJjhMcSHiO8KZsSGrvlf/lPhzdOLDwJb3gbTrMp+w6vILyC
8ArCKxJeQXhTNiW07GJNTLkf6vDGiYUn4WmbEsym7Du8ivAqwqsIr0p4FeFN2ZTQkkul5lDYl+V2
pQ5vnFh4Ep7WKcF0yr7DawivIbyG8JqE1xDelE4JLbpcSqvEmWIZbmqOEwtPwhvehtN0yq7DI+gU
gk4h6BQSnULQKTSlU0ILjutyiAMd3jix8CS84W04zafsOzz4FIJPIfgUEp9C8Ck05VNCI1dajBQ9
t+aHN5IaJxbeGl7UsiCaT9l3ePApBJ9C8CkkPoXgU2jKp4Tm3VLV+pt0eOPEwpPwtFyJJlf2HR7k
CkGuEOQKiVwhyBWalCve+egDUao+chzD0xMLT8LTciWaXNl3eJArBLlCkCskcoUgV2hOrtTmKDMH
zp69fmp4O7HwJDwtVyZJj4V3beFBrhDkCkGukMgVglyhOblSq1v+l/90eOPEwpPwtFyJJlf2HR7k
CkGuEOQKiVwhyBWakyu1uBQypVa8z8O7dG8mFp6Ep+VKNLmy7/AgVwhyhSBXSOQKQa7QnFyp7DIX
4kSlkR9uao4TC0/C03IlmlzZd3iQKwS5QpArJHKFIFdoTq7U7AqlZbkc6ifQNxMLT8LTciWaXNl3
eJArBLlCkCskcoUgV2hOrtTkKnPiWlPQLzPfDCw7yU67lWhuZdfZBbiVALcS4FaCuJUAtxLm3MpS
lw+RuXJsPm+60xMLT8LTbmXyTSMsvGsLD24lwK0EuJUgbiXArYQ5t1Kjo5rPe6X1pbF+bcJmYuGt
4SXtVpK5lX2HB7cS4FYC3EoQtxLgVsKcW6nBxRxaIiqcinYrm4mFJ+Fpt5LMrew7PLiVALcS4FaC
uJUAtxLm3Eoll0Nq4XyTMuidsW4nFp6Ep91KMrey7/DgVgLcSoBbCeJWAtxKmHQr3hVPJadY8+am
5jix8CQ87VYm9+Vr4V1beHArAW4lwK0EcSsBbiXMuZXSXK0x1vP+oKlpt7KZWHgSnnYrydzKvsOD
WwlwKwFuJYhbCXArYc6tlPMu/FrhkhJVpjE8PbHwJDztVpK5lX2HB7cS4FYC3EoQtxLgVsKcWynV
BfaVW+Y83sfbTCw8CU+7lWRuZd/hwa0EuJUAtxLErQS4lTDnVkpxKTOnElut+kbUdmLhSXjarSRz
K/sOD24lwK0EuJUgbiXArYQ5t1LYcS6efaZa9OvMthMLT8LTcmVyB9sW3pWFFyFXIuRKhFyJIlci
5Eqckyslu8rBc22c83hTc5xYeBKelivJ5Mq+w4NciZArEXIlilyJkCtxTq4seflCHGOIpQ77kd5M
LLw1vKzlSja5su/wIFci5EqEXIkiVyLkSpyTKyW59eUH50Md3jix8CQ8LVeyyZV9hwe5EiFXIuRK
FLkSIVfinFwp0WV/fvXB5XAIb5xYeBKelivZ5Mq+w4NciZArEXIlilyJkCtxTq6U4EqQ3/rVCZuJ
hSfhabky+bINC+/awoNciZArEXIlilyJkCtxUq6QazlxSSH4NuxjbDOx8CQ8LVeyyZV9hwe5EiFX
IuRKFLkSIVfipFwhR5VT8lxKYx7D0xMLT8LTciWbXNl3eJArEXIlQq5EkSsRciVOyhXvlqu0vNxX
IRp34L6ZWHgSnpYr2eTKvsODXImQKxFyJYpciZArcU6ucHPMoQWKqcbhTbo2EwtPwtNyJZtc2Xd4
kCsRciVCrkSRKxFyJc7JFa6u+ZI5+rBctenuhoFlJ9lpt5LNrew6uwS3kuBWEtxKEreS4FbSnFtZ
6qLMKaRcl4sqj93piYUn4Wm3ks2t7Ds8uJUEt5LgVpK4lQS3kubcCheXfEtluWoLvuqd2W4mFt4a
Hmu3wuZW9h0e3EqCW0lwK0ncSoJbSXNuhXm5Hye/tNTcTCw8CU+7lcmXKVp41xYe3EqCW0lwK0nc
SoJbSXNuhbNrobRSamzDjsm3EwtPwtNuZfL9qS28awsPbiXBrSS4lSRuJcGtpDm3suRFS1fFE+eS
eAxPTyw8CU+7lUlXYOFdW3hwKwluJcGtJHErCW4lzbkVTi5xqL41KsMTw9uJhSfhabfC5lb2HR7c
SoJbSXArSdxKgltJc26FoyvL7SWOl0NNpDcTC0/C025lcudrFt61hQe3kuBWEtxKEreS4FbSnFtZ
8vIUuVWOy1rG8PTEwpPwtFuZlHQW3rWFB7eS4FYS3EoSt5LgVtKkWwkutOwrNS6BNJHeTCw8CU+7
lcmHey28awsPbiXBrSS4lSRuJcGtpEm3Qi5XHyulQnF4Gm8YWHaSnXYrk4DVsruy7DLcSoZbyXAr
WdxKhlvJk27FO9l7X2yDFxsnFp6Ep90Km1vZd3hwKxluJcOtZHErGW4lT7oV74iXQxyM4emJhbeG
V7RbKeZW9h0e3EqGW8lwK1ncSoZbyXNuJTeXcsN/urtxYNlJdlqtTL6BkmV3bdlBrWSolQy1kkWt
ZKiVPKdWcnVluTrzpRIX0l/4ZmLhSXharRRTK/sOD2olQ61kqJUsaiVDreQ5tbLk5TllYgrNx014
emLhSXharUzuhsbCu7bwoFYy1EqGWsmiVjLUSp5TK7m45ac250Q5pzbc0hwnFp6Ep9XK5I7XLLxr
Cw9qJUOtZKiVLGolQ63kObWS2XFNqVEr49sEjQPLTrLTZqWYWdl3djArGWYlw6xkMSsZZiXPmZU8
XMXZtdpTXlqmFJMp+84LMiVDpmTIlCwyJUOm5DmZkrOL1CjVFnl4Y4RxYNlJdtqlFHMp+84OLiXD
pWS4lCwuJcOl5DmXkpPjRM23SikGTaA3EwtPwtMypZhM2XV4DJnCkCkMmcIiUxgyhedkSo6ucV2W
y+EQ3jix8CQ8LVOKyZR9hweZwpApDJnCIlMYMoXnZMqSV1iu03L0MUVPY3h6YuGt4VUtU6rJlH2H
B5nCkCkMmcIiUxgyhSdlSnAcKZfaUhl473Zi4Ul42qZUsyn7Dg82hWFTGDaFxaYwbApP2hQ636DM
PrZYvB/CGycWnoSnbUo1m7Lv8GBTGDaFYVNYbArDpvCkTSEXfW3e5xDHpw+GgWUn2WmZUk2m7Ds7
yBSGTGHIFBaZwpApPClTljM1h5IoU4pDdsPAspPstEup5lL2nR1cCsOlMFwKi0thuBSedCl+SM3y
kry0P6nmT/adF/wJw58w/AmLP2H4E57zJ6m5mD2n1g/VF76ZWHgSnpYp1WTKvsODTGHIFIZMYZEp
DJnCczIlVVd8Yo6BQhp2SLuZWHgSnrYp1WzKvsODTWHYFIZNYbEpDJvCczYlnV/AEzzl7JcLEY/h
6YmFJ+Fpm1LNpuw6vAKbUmBTCmxKEZtSYFPKnE1J5fzmkVx9ojC8Pno7sfAkPG1TJt+NxcK7tvBg
UwpsSoFNKWJTCmxKmbMpiV2p59uTsflxF+ybiYW3hte0TWlmU/YdHmxKgU0psClFbEqBTSlzNmXJ
i3KsifvhGJ6eWHgSnrYpkztQs/CuLTzYlAKbUmBTitiUAptS5mxKyi6HFBpzydT0F76ZWHgSnrYp
kyeMhXdt4cGmFNiUAptSxKYU2JQyZ1NScrWVlhvVyHrHBNuJhSfhaZ3STKfsOzzolAKdUqBTiuiU
Ap1S5nTKklcoqfpG7KPPY3h6YuFJeNqnNPMp+w4PPqXApxT4lCI+pcCnlDmfkqLjnGoIMWYfhvDG
iYUn4Wm50kyu7Ds8yJUCuVIgV4rIlQK5UiblSnQ+llYS15hTGsPTEwtPwtNyZXIXahbetYUHuVIg
VwrkShG5UiBXyqRcCS6Fp986vHFi4Ul4Wq40kyv7Dg9ypUCuFMiVInKlQK6USblCrlJkfyYqNLzN
1mZi4Ul4Wq40kyu7Dq9CrlTIlQq5UkWuVMiVOilXyAV/fvSEsi/DXlU2EwtPwtNypZlc2Xd4kCsV
cqVCrlSRKxVypU7KleVMPacVS27DRWo7sfDWi48+lvS5jsXC+zsJD3KlQq5UyJUqcqVCrtRJueKH
1iwvyYt0XuZT9p0XfEqFT6nwKVV8SoVPqXM+JTa3XHhS47Bce2X9hW8mFp6EF3R45lP2HR58SoVP
qfApVXxKhU+pcz4lVld9buWc2Lh/zM3EwpPwog7PfMq+w4NPqfApFT6lik+p8Cl1zqcseQWKyz21
8122EMbw9MTCk/D0BcibT9l3ePApFT6lwqdU8SkVPqXO+ZRYHC8H+F/vpm8zsfAkvKzDM5+y7/Dg
Uyp8SoVPqeJTKnxKnfMpS17LFZr8N4anJxaehMc6PPMp+w4PPqXCp1T4lCo+pcKn1DmfEtmlHIOn
5c5cSvpp8s3EwpPwig7PfMq+w4NPqfApFT6lik+p8Cl1zqfE7OpyDy7my6F+uetmYuFJeFWHZz5l
1+E1+JQGn9LgU5r4lAaf0uZ8ypJXaImocCo+tzE8PbHwJLymwzOfsu/w4FMafEqDT2niUxp8Spvz
KTG5QrUGTj7X4c1HNhMLb734kPYpZD5l3+HBpzT4lAaf0sSnNPiUNudTlrwocUz9II/h6YmFJ+Fp
uUImV/YdHuRKg1xpkCtN5EqDXGmTciW6XJjT+UWty6rDGycWnoSn5QqZXNl3eJArDXKlQa40kSsN
cqVNypXovG/F1xJL5TSGpycWnoSn5crkq6IsvGsLD3KlQa40yJUmcqVBrrRJuRJcSjmH5X4cEQ3h
jRMLT8LTF6DJE8bCu7bwIFca5EqDXGkiVxrkSpuUK+RqTT6XVsiP9/HGiYUn4Wm5MnnCWHjXFh7k
SoNcaZArTeRKg1xpk3KFXAw5ego+VWpjeHpi4Ul4Wq6QyZV9hwe50iBXGuRKE7nSIFfapFzxrnBb
lsuhfoeEzcTCk/C0XCGTK/sOD3KlQa40yJUmcqVBrrRJueJdoFxDibEOO1YZB5adZKfdCplb2XN2
yXe3gvWE9Xhee3b4yOHykYnsQnPM2RcqxcfhPfA2EwtPwtNuZfIWuIV3beERwiOERwiPJDxCeHNu
ZcmLiFvK5Mu4X5XNxMJbLz5Bu5VgbmXf4QWEFxBeQHhBwgsIb86thOpyoRTq5XAIb5xYeBKedivB
3Mq+w4sILyK8iPCihBcR3pxbWfLyS1s+Zg7LtdsYnp5YeBKediuTL8y38K4tvITwEsJLCC9JeAnh
zbmVUFyqlTJxrmUg0puJhSfhabcSzK3sO7yM8DLCywgvS3gZ4c25lcDuvEuVnNi3Nt7UHCcWnoSn
L0DB3Mq+w2OExwiPER5LeIzw5tzKklei7FuKsbH+Wb6dWHgSnnYrk2+VZOFdW3gF4RWEVxBekfAK
wptzKyG7WkPM/RWveo8rm4mFJ+FptxLMrew7vIrwKsKrCK9KeBXhzbmVJa+lLaLAhX3mMTw9sfAk
PO1WgrmVfYfXEF5DeA3hNQmvIbw5txKSqyHFnFuh2oZrvHFi4Ul4Wq4Ekyu7Do8gVwhyhSBXSOQK
Qa7QpFxJLrTmOaewXK3RGJ6eWHgSnpYrk486WXjXFh7kCkGuEOQKiVwhyBWalCvRlcLLcjnUr8bb
TCy89eITtVyJJlf2HR7kCkGuEOQKiVwhyBWalCvRhZwrxZZjqnUMT08sPAlPy5XJF21YeNcWHuQK
Qa4Q5AqJXCHIFZqUK8Gdd+xAzS+VDe+csJlYeBKelivR5Mq+w4NcIcgVglwhkSsEuUKTciW4oH6N
4emJhSfhabkSTa7sOzzIFYJcIcgVErlCkCs0KVfIFd+85xbqsGvy7cTCk/D0BSiaXNl3eJArBLlC
kCskcoUgV2hSrpCjVikz1+BDHcPTEwtPwtNyZfKFihbetYUHuUKQKwS5QiJXCHKFJuXKcqbWxjlT
CcPes7YTC0/C03Jl8i0lLLxrCw9yhSBXCHKFRK4Q5ApNyhXvqMZUz2+Al2MYw9MTC0/C03Jl8t3L
LLxrCw9yhSBXCHKFRK4Q5ArNyRVqjkvjGCuXMFjNzcTCk/C0XJl8v04L78rCC5ArAXIlQK4EkSsB
ciXMyZUlLyrV1+UPy0pjeHpi4Ul4Wq5Ekyv7Dg9yJUCuBMiVIHIlQK6EOblCdbleO7/eNdVWkn49
3mZi4a0Xn6TlSjK5su/wIFcC5EqAXAkiVwLkSpiTK0teVJ9+j+HpiYUn4Wm5kkyu7Ds8yJUAuRIg
V4LIlQC5EubkChXHtfiSamhteHBlM7HwJDwtV5LJlX2HB7kSIFcC5EoQuRIgV8KcXFnyopZy8RRz
SWUMT08sPAlPy5XJt4a38K4tPMiVALkSIFeCyJUAuRLm5AqxKz6FWltsPg/XeOPEwpPw9AUomVzZ
d3iQKwFyJUCuBJErAXIlzMmVJa/lf/lvDE9PLDwJT8uVyf0eWnjXFh7kSoBcCZArQeRKgFwJc3KF
sivRx7jcmas8XuONEwtPwtNyJZlc2Xd4kCsBciVArgSRKwFyJczJlSWvkJgjlzK8Y8nwcYtOotNq
ZfKOr0V3bdFBrQSolQC1EkStBKiVMKlWkiucl0spF6I0PHk+Tiw8CU+rlWRqZdfhRaiVCLUSoVai
qJUItRIn1UpyeGeEy+EYnp5YeBKeViuTqsDCu7bwoFYi1EqEWomiViLUSpxUK9FVXxPHkImHZxKG
gWW3XniyNivZzMq+s4NZiTArEWYlilmJMCtx0qxEF9WvsTs9sfAkPG1WspmVfYcHsxJhViLMShSz
EmFW4qRZCa6yj+TT8sc6PJEwTiw8CU+blckHei28awsPZiXCrESYlShmJcKsxEmzElysLVafo28l
juHpiYUn4Wmzks2s7Ds8mJUIsxJhVqKYlQizEifNCrkWfA6teor6jTi2EwtPwtMXoGxmZd/hwaxE
mJUIsxLFrESYlThpVsgtP7VLSqnF2MIYnp5YeBKeNivZzMq+w4NZiTArEWYlilmJMCtx0qx412rz
5C+Hw7N448TCk/C0WZl8HxcL79rCg1mJMCsRZiWKWYkwK3HSrHiXA0fKpbZU0hienlh4Ep52K9nc
yr7Dg1uJcCsRbiWKW4lwK3HSrXjnOfNyk4lSGN6ScjOx8CQ87VayuZVdh5fgVhLcSoJbSeJWEtxK
mnMrvjn23CrX1jLr8DYTC0/C024lm1vZd3hwKwluJcGtJHErCW4lzbmVJS/KPgYurcThebzNxMJb
Lz6s5QqbXNl3eJArCXIlQa4kkSsJciXNyRVfHS9XaDlQiW14ZcJmYuFJeFquTL5loIV3beFBriTI
lQS5kkSuJMiVNCdXlrxCCi17OrPMsTs1sOwkO+1W2NzKvrODW0lwKwluJYlbSXArac6t+OJKC8ty
OdR7F9tMLDwJT7uVybu+Ft61hQe3kuBWEtxKEreS4FbSnFtZ8oopJgo5cGub8PTEwpPw9AWIza3s
Ozy4lQS3kuBWkriVBLeS5tyKZ/f0jiRxCG+cWHgSnnYrbG5l3+HBrSS4lQS3ksStJLiVNOdWlrxS
JqbQfAyBx/D0xMKT8LRbYXMr+w4PbiXBrSS4lSRuJcGtpDm3suTlfc01cvOtljE8PbHwJDztViZf
H2zhXVt4cCsJbiXBrSRxKwluJc25FZ9d5rJctdXglys3fRaMEwtPwtNuhc2t7Dq8DLeS4VYy3EoW
t5LhVvKkW8mOwpJW7odjeHpi4Ul42q2wuZV9hwe3kuFWMtxKFreS4VbypFtJbrkpWSPlzNkPT5+P
EwtvvfgU7VaKuZV9hwe3kuFWMtxKFreS4VbypFtJLuQcqVGqLfIYnp5YeBKedivF3Mq+w4NbyXAr
GW4li1vJcCt50q1EVwNxZp99y8NNzXFi4Ul4Wq4Ukyv7Dg9yJUOuZMiVLHIlQ67kSbkSz/tVqXGp
q6ayCU9PLDwJT8uVyV3RWHjXFh7kSoZcyZArWeRKhlzJk3IluJbPeVXmOpKxcWLhSXj6AjT5+mAL
79rCg1zJkCsZciWLXMmQK3lSrgS33I87v9iVm9a/m4FlJ9lptzJ5Q8Cyu7bs4FYy3EqGW8niVjLc
Sp50K8GRl9075LE7PbHwJDztVoq5lX2HB7eS4VYy3EoWt5LhVvKkWyG33IzkEgLXNt7QHCcWnoSn
3Uoxt7Lv8OBWMtxKhlvJ4lYy3EqedCvkwnKNxsQxhzB2pwaWnWSn1UoxtbLr7BhqhaFWGGqFRa0w
1ApPqhXvauS6rJdlOAuGiYUn4Wm1Ukyt7Ds8qBWGWmGoFRa1wlArPKlWvEtUcuQc0/BWi9uJhbde
fKpWK9XUyr7Dg1phqBWGWmFRKwy1wpNqZTzVLS/JS9uUajZl33nBpjBsCsOmsNgUhk3hKZtCrblc
07Ish+rLHj9u0Ul02qVM3sq26K4tOrgUhkthuBQWl8JwKTzlUs5xUfEU0vJzO4Y6ZqcnFp6Ep11K
NZey7/DgUhguheFSWFwKw6XwlEuhVl3JpXBsJZOWmNuJhSfh6QtQNZey7/DgUhguheFSWFwKw6Xw
lEs55xVTIV9SrJ7KGJ6eWHgSnpYp1WTKvsODTGHIFIZMYZEpDJnCUzKFWnEtVkophlbbEN44sfAk
PC1TJnefbeFdW3iQKQyZwpApLDKFIVN4Sqac88pn5ZyWP+VIY3h6YuFJeFqmTO5qxsK7tvAgUxgy
hSFTWGQKQ6bwlEw553XeL2ZmLjHTJjw9sfAkPG1TqtmUXYdXYFMKbEqBTSliUwpsSpmyKdTYlUCR
KJWchu6GgWUn2WmZUk2m7Ds7yJQCmVIgU4rIlAKZUqZkyrmuSM1TTT4NBHo7sfDWi0/TMqWZTNl3
eJApBTKlQKYUkSkFMqVMyRRq2TWqcbkxudxEjU2HN04sPAlPm5VmZmXf4cGsFJiVArNSxKwUmJUy
aVayy9RiqoFr3YanJxaehKfdSjO3su/w4FYK3EqBWyniVgrcSpl0K+c9Yy6HOBjD0xMLT8LTbqWZ
W9l3eHArBW6lwK0UcSsFbqVMupXkSmBijkE/OzV+3KKT6PSFp5lZ2Xd0MCsFZqXArBQxKwVmpUya
leRipFZj9Zm5jtnpiYUn4WmzMrm3Cwvv2sKDWSkwKwVmpYhZKTArZdKsRNei/E46vHFi4Ul42qw0
Myv7Dg9mpcCsFJiVImalwKyUSbMSXU7LtVrxjUvZhKcnFp6Ep83KpKKz8K4tPJiVArNSYFaKmJUC
s1ImzUp0lGvLsYQQ4yY8PbHwJDxtVibfAtfCu7LwKsxKhVmpMCtVzEqFWamTZiW4UsjXFGOJaXgq
YZxYeBKeVivN1Mq+w4NaqVArFWqlilqpUCt1Uq0EF2sqvFyzNV/qGJ6eWHg4lqyPJX+uY7Hw/k7C
g1qpUCsVaqWKWqlQK3VSrQxP29kzdU95kc7LbMq+84JNqbApFTalik2psCl10qaQY+LCnLh4Hp6r
GycWnoQXdHhmU/YdHmxKhU2psClVbEqFTamTNoVciNnnVti3vAlPTyw8CS/q8Mym7Ds82JQKm1Jh
U6rYlAqbUidtinc1p1p8KTmm4Z7cOLHwJLykwzOfsu/w4FMqfEqFT6niUyp8Sp30Kd6lwstyOaQx
PD2x8CS8rMMzn7Lv8OBTKnxKhU+p4lMqfEqd9Cne+dY8/h+7UwPLTrJjnZ3plH1nB51SoVMqdEoV
nVKhU+qcTqnNlRBj8JQr6fsu24mFJ+EVHZ7plH2HB51SoVMqdEoVnVKhU+qcTlnyiqkRhRTYRx7D
0xMLT8KrOjzTKbsOr0GnNOiUBp3SRKc06JQ2p1Nqda1wLKXlUIbHNDcTC0/Cazo80yn7Dg86pUGn
NOiUJjqlQae0OZ2y5MU+k68+D88LbwaW3ZodaZtCZlP2nR1sSoNNabApTWxKg01pczZlqSvEHFoi
KqzfZGs7sfAkPK1WJvdpaOFdW3hQKw1qpUGtNFErDWqlzamVWlxlbstNyRba8L4Im4mFJ+FptTK5
xwsL79rCg1ppUCsNaqWJWmlQK21OrSx5bWqzxJ4S0z6FzKfsOzH4lAaf0uBTmviUBp/S5nzK0hTF
lH1oIbZSxtr0xMKT8LRPIfMp+w4PPqXBpzT4lCY+pcGntDmfUtmV4lOonnJm/aLWzcTCk/C0TyHz
KfsODz6lwac0+JQmPqXBp7Q5n7LklXwpy922tly30Rienlh4Ep4WKmRCZd/hQag0CJUGodJEqDQI
lTYpVNj5xJmIQ2m+jOHpiYUn4WmhMvn2fxbetYUHodIgVBqEShOh0iBU2qRQyY4rc46lUBmeJx8G
lp1kp30KmU/Zc3bZd5+C9YT1eF57dvjI4fKRuexiqMvSD8fu9ORvD09/ZPgr+K57fP/yze3tY3j+
PVk+3tD64Y/3d+8fl2/p5u3D7TbVD28/vXv/PX/91Qf5669v39x8evt4+VJf6M/5fR88f/mT/83w
Gf7N8hn+zfQZ/s34Gf7N+hn+zc9xvvN/9t/8W69yHh6Xy/7XT52eHr+5vX9/+/js4ePt7etnvzn9
w9/8M217FP/47sOn94/LZz57/+H17cOz3/zjz38cx5uPN6/uHr979unh/F38689/DL+/e/j3Z6/W
o/nN7z/PMTw8+3h7fzmdnv3mv//8R/GHm7u3y+mD8+EPP/8R/On21Yc/395/9+zx7t3yLfzp2dxR
7PkWF6sj4c/mo/Tr285bn+d4aHuLi+wW13CLq/uorqO6jVplVHdRUyqK2bH6pe4vbydTX/vvfvv6
/ubbL97cLw3h+7h9/7pfE928fn1/+/Cw3p5y/1eSa6zz5/zlfFErgcrde/3h784fplDPH7382//r
xd3717fnz8YHdLVf3y/XrA9//vqLb+9eP37z5fPquLR8/rvnD35ze/f1N8uNtOJqJfno5Yh9inH9
wOUofWzn48Q39OGrf7t99diP7/2Hx7s337348P7Fp4+vbx5vX3x48+L+5v3Xt0/f2jF8sf6plWf4
4z/JB/+pLben/vL27v2/f/HN/e2bL5+7l/+tH0FYP44fhXfvPr69XT/28M2Hb798fvvuq9vX64du
Xj1+Wr6E5ULy/o8fbl6fr9kvX+Pdu5uvb7/3KP50+/Htzavbd7fvHx9+xmN9+XS+f+8FxX4E/20/
gof7vJ/rxY72I/hHfgR3KdedXFdyq5HrQm7Kx1V1Qte509hu0/zNQZUhqM/0ELEF9WMP3vaHbvsD
t/1h2/VB2/6Q7UxQLbqofz19sduJxfbLxlaH2D7TA8MW2489ZNsfsO0P1/YHa9eHavsDtVMP0/rh
foJ+WnIzsdh+2djaENtnerGixfYjsXX/1vVbt2+rfOvubU69BX1KB7ux+MsmRbqoz/RqKAvqR4Lq
rq2rtm7aVtHWPducZovjLUJ1sm8mltovm9rYmsX268TWLVuXbN2xrYqtG7Y5wZZ+8KZispuKv2Zs
YYjtM73c0GL7kdi6X+t6rdu1Va51tzan1vRFhyYvO5bU35xUHJL6TC8vtKR+JCnqOK0vp74caZVp
fftAcy6t/OBNxmI3GX/N5NKQ3Ge6iFpyP5Zc321SX059OdK6z6S+faDJPSb94A3Hajccf83kBhH0
uV5SaMn9WHKdBPXl1JcjrSqobx9oygWFjXm3sH7ZsAbn8bleMmhh/VhYHXr05dSXI63Wo28faEp7
BPqhm4/biSX3yyY3SJDP9WJBS+7HkusUpC+nvhxp1SB9+0BTHiSEH7r5uJ1Ycr9scoMH+VwvFLTk
fiy5DkL6curLkVYT0rcPNKVCQtI3H5PdfPyFwxrsB5n9+JXC6vijL6e+HGn1H337QFMCJOQfvPmY
7ebjr5jc8BSaPYP2KwXXcUhfTn050upD+vaBpoRI4B+88ch24/HXDG4QIsGEyK+UXCcifTn15Uir
EunbB5pyIkG/IibYS2J+6bDGqzK7LvuVwuocpC+nvhxpFSF9+0BTJiT8/+2d3W4cxxGFX4XwlXJh
hNPd83cTwN1qXwcO8gAMtY6FSKQgrgLo7UP5bElTE7fpsjhdEHCuDootcc3Rfu7h9nd21+bN48qb
R0/klC0SaIv4IBdgiyAqogSxRTDnYLJFYrMes18hcn2RU7ZIoC3ihBxsEURFlCC2COYcTLZI3JZk
IksyvcFSTkigE+IE1uVtYi7vE3N5o5jP7xRzeasYkxMSm2WZ/QqR64ucskUCbREn5GCLICqiBLFF
MOdgskWiLsZs/p33K0SuL3LKFgm0RZyQgy2CqIgSxBbBnIPJFonbJ1BkcaY3WMoJCXRCnMCCE4Ko
iBLECcGcg8kJic3izH6FyPVFTtkigbaIE3KwRRAVUYLYIphzMNkisVmc2a8Qua7Iqdf7+XK/E3Cw
RRAVUYLYIphzMNkiaXu9E2szvbFSTkikE+IEFpwQREWUIE4I5hxMTkhq1mb2K0SuL3LKFom0RZyQ
gy2CqIgSxBbBnIPJFknN2sx+hcj1RU7fPfL20Qe5CFsEURElii2COUeTLZK2tZnE2kxvsJQTEumE
OIEFJwRRESWKE4I5R5MTkpq1mf0KkeuLnLJFIm0RJ+RgiyAqokSxRTDnaLJFUrM4s18hcn2RU7ZI
pC3ihNzlQ2QunyJz+RiZz58jc/kgGZMtkrbFmcTiTG+wlBMS6YQ4gQUnBFERJYoTgjlHkxOSmsWZ
/QqR64ucskUibREn5GCLICqiRLFFMOdoskXGZnFmv0Lk+iKnbJFIW8QJOdgiiIooUWwRzDmabJFx
W5wZWZzpDJZ68ZGvPTphBScEURElihOCOUeTEzI2azP7FQLXFzhliyTaIk7IwRZBVESJYotgztFk
i4zNT5rZrxC5vsgpWyTRFnFCDrYIoiJKFFsEc44mW2TcPoFG1mZ6g6WckEQnxAesBCcEUREliROC
OSeTEzI2azP7FSLXFzn9Cxt/Y3NCDrYIoiJKElsEc04mW2Rs1mb2K0SuL3LKFkm0RZyQgy2CqIiS
xBbBnJPJFpmu1ROIt4+dwVJOSKIT4gQWnBBERZQkTgjmnExOyNQszuxXiFxf5JQtkmiLOCEHWwRR
ESWJLYI5J5MtMjWLM/sVItcXOWWLJNoiTsjBFkFURElii2DOyWSLTNvizMTiTG+wlBOS6IQ4gQUn
BFERJYkTgjknkxMyNYsz+xUi1xU59UoIXwhxAg62CKIiShJbBHNOJltkatZm9isEri9wyhYZaYs4
IQdbBFERJYktgjknky0ybWszE2szvcFSTshIJ8QJLDghiIooSZwQzDmZnJCpWZvZrxC5vsgpW2Sk
LeKD3AhbBFERZRRbBHMeTbbI3KzN7FeIXF/klC0y0hZxQg62CKIiyii2COY8mmyReVubmVmb6Q2W
fiWEL4U4gQUnBFERZRQnBHMeTU7I3CzO7FeIXF/klC0y0hZxQg62CKIiyii2COY8mmyRuVmc2a8Q
ub7IKVtkpC3ihBxsEURFlFFsEcx5NNki8/YJNLM40xss5YSMdEKcwIITgqiIMooTgjmPJidk1vWY
zUXZrxC5vsgpW2SkLeKEHGwRREWUUWwRzHk02SJzszizXyFyXZFTv7Dx9zUn4GCLICqijGKLYM6j
yRZZttd7YW2mN1bKCZnohDiBBScEURFlFCcEcx5NTsjSrM3sV4hcX+SULTLRFnFCDrYIoiLKKLYI
5jyabJGlWZvZrxC5vsgpW2SiLeKD3ARbBFERZRJbBHOeTLbIsq3NLKzN9AZLOSETnRAnsOCEICqi
TOKEYM6TyQlZxtZrj/sVItcXOWWLTLRFnJCDLYKoiDKJLYI5TyZbZGkWZ/YrRK4vcvrFR7766IQc
bBFERZRJbBHMeTLZIsu2OLOwONMbLOWEGP+/RrCeDSw4IYiKKJM4IZjzZHJClrV5+7jy9tETOWWL
TLRFnJCDLYKoiDKJLYI5TyZbZG0WZ/YrRK4vcsoWmWiLOCEHWwRREWUSWwRznky2yLotzqwsznQG
S9098ubRCSs4IYiKKJM4IZjzZHJC1ti6edyvELi+wClbZKYt4oQcbBFERZRJbBHMeTLZImuzNrNf
IXJ9kVO2yExbxAk52CKIiiiT2CKY82SyRdbtE2hlbaY3WMoJmemE+IA1wwlBVESZxQnBnGeTE7I2
azP7FSLXFzlli8y0RZyQgy2CqIgyiy2COc8mW2Rt1mb2K0SuL3LKFplpizghB1sEURFlFlsEc55N
tsjwfxecaPVFS1khM60QJ7RghSAqosxihWDOs8kKGa51Q2Z78fUKkeuLnH7Fny/5OyEHXwRREWUW
XwRznk2+yHCtGjLz9uLrFSLXFznli8z0RZyQgy+CqIgyiy+COc8mX2S4TuqCszzTGy3lhcz0QpzQ
gheCqIgyixeCOc8mL2S4Hps3kCNvIP2QU5sZ9zIn4GCMICqizGKMYM6zyRgZrqfm7ePE20dH4JQx
stAYcUIOxgiiIsosxgjmPJuMkeF6URec5ZneaCkzZKEZ4oQWzBBERZRZzBDMeTaZIcP12rx9XHn7
6IicckYWOiM+yC1wRhAVURZxRjDnxeSMDMN16wZyt0Lk+iKnnJGFzogTcnBGEBVRFnFGMOfF5IwM
w7Y982kiWn3RUm7IQjfECS24IYiKKIu4IZjzYnNDhti6gdytELm+yClnZKEz4oQcnBFERZRFnBHM
ebE5I0Nq3kAm3kA6IqeckYXOiBNycEYQFVEWcUYw58XmjAyTuoFkhaY3Wvo4jedpTmjBDUFURFnE
DcGcF5sbMszNG8iZN5COyClnZKEz4oQcnBFERZRFnBHMebE5I8PSvIFceAPph5wijsA5AQdnBFER
ZRFnBHNebM5IUBc8sEDTGyzlhqx0Q5zQghuCqIiyiBuCOS82NyQ0CzSBBRpP5JQzstIZcUIOzgii
IsoizgjmvNickdAs0AQWaDyRU87ISmfEB7kVzgiiIsoqzgjmvNqckaAKNIEFmt5oKTdkpRvihBbc
EERFlFXcEMx5tbkhoVmgCSzQeCKnnJGVzogTcnBGEBVRVnFGMOfV5oyEZoUmsELjiZxyRlY6I07I
wRlBVERZxRnBnFebMxJUhSawQtMbLeWGrHRDnNCCG4KoiLKKG4I5rzY3JDQrNIEVGk/klDOy0hlx
Qg7OCKIiyirOCOa82pyR2KzQRFZoPJHTJ9g8wnZCDs4IoiLKKs4I5rzanJGoKjSRFZquaG3Or3l4
7QQVvBBERZRVvBDMebV5IbFZnonPXZ7ZfkX9lcvPC+D+8cvpdI7f/QaK55tBvvzu/eu78+OPdPPm
4bTH8/7Nh7d3v/HXb+8///VXp59vPrw5//qf+v32z7zEwnd//cPfMxzwPecDvmc64HvGA77ncsD3
/NP/7l+7JTycH5+n//7CVD3/cnp/dzpfPbw7nV5dvah/+er/8+wf4oe39x/uzo9/8uru/tXp4erF
D8//GOXm3c3t6/PHqw8Pn36Kfz7/I7x8/fCfq1t5mBcvj3mEh6t3p/e/XqerF39//of48eb1m8fr
c/l3+PH5H+Cn0+39f0/vP16dX799/BF+uvr9h+Dtz1f9ZqEEvoNugcL+FigceQu02e2PMjX+9E1j
4xYIR0E4CcJBkJwD4RjI+q7yf/zZSXq+jh51ljocdJZKfp7gB+c6ONbBqY4c6uBMx3akM26fnyP5
OZAf5bIepLKSnifowdENTm5wcCPnNji2MZ3ahN2VJj0H0qN2n8Ddx4cfnMPgGAanMHIIgzMY0xFM
2O4+gbvPkfworfugZxvpeYIeHKngRAUHKnKeguMU02lK3F7pyN3nUHrU7hO5+/jwc3kndrx1Ld65
Vt64Fu9ba+Jnu/tE7j5H8qOaD0cd1pGe36fnUpxFiQ8dPqnwocFnoSdtr3Ti7nMoPWr3Sdx9fPhB
CxYlWHRgpQKLBqypAJu2u0/i7nMkPwof0uNDz4BGK6IiyiCNVsx5MDVax+31HrkHHcqQhogUOVGE
8iqiIsog5VXMeTCVV8ftTjRyJzqSItWNO6gaR4aeYggKAqIiyiAWAuY8mDyE6Vo9fcjQkQypnWji
TuREEUQEREWUQVwEzHkw2QjTdieauBMd+mm+W4gOapKSoacYgo6AqIgyiJGAOQ8mJ2HeXu+ZO9Gh
DKmdaOZO5EQRpARERZRBvATMeTCZCfN2J5q5E33rH3JNhp5iCGoCoiLKIHYC5jyY/IRle70X7kTf
/mejkaKnKIKggKiIMoijgDkPJkth2e5EC3eib/2t88nQUwxBU0BURBnEVMCcB5OrsG6v98qd6Nt/
x0VS9BRFkBUQFVEG8RUw58FkLKzbnWjlTvQtv1EH+XmCnwBbAVERJYitgDkH2/tvX++eoFaCtl95
nC7f/eHd+9PNq4dP77nx5Yv/un/18cv06v72w9vT3fn72/u782P+7X9QSwcILWdoW6ZlAAAaaggA
UEsDBBQACAgIADar3j4AAAAAAAAAAAAAAAAbAAAAT2JqZWN0UmVwbGFjZW1lbnRzL09iamVjdCAx
7L0J3NdVmf7/dQEFVEBZlE1ABZRF2UEBlUXBBVDUVBzF/kq5lEsp/lzKZXIptXEptVwqtVwmtQkt
l3IptVzKZXIZl8alFHMpl1JM+F/n+Z7rnG9H8X/P/LzPfeJ/nJdeeD/A+309n+tj8TjpHtN22nm3
7VZpjGw0f1ul0bvR+tu/zG80PtgX9+0bjVe2azT/xH1j2qhG42v47qvitnz5SfhWl5Yf9nd8ryXb
xe/Rv3EyvjUA31q1DbBq29X95r7XKuHH4PoN/Onq/oPf9D+8Q+Mr+Fa7NtTyxir/6v+k+duqqzR+
iVivsefcbXebud2snXbab96MXfabOmP7WbPdx3/QVuu7bX8c2/bH9o0vb+7rNGI2fDumu8fvd+Ly
tt/4HVcJP+TPWzc+9rfk4yd+6Fvx0jgMbcf9rxXdZ8J9crsmn4kZs6fzB3+l9TP3D5/Gz/gn+CZK
ut877ddovLud5Ec8gc/Fv+D3H8xrNDaaHr/Tj3/8Y/f4VmmcjT/p3nSat9uuc3ac8Y/P57C2wge1
/XE0/ti+0WNYo3HJNo3GPu1idsIOHto1prvz+/mVxPzwb6v9w5/9g+I/NPwcvuec/5XEqo0ejdZd
/ONv+DT0bYSRtnwaWh7Ohz5xn//4T9ycNt6stj+u5xhtThd714tln5h/FP74T8zmHwkppfiIcU2n
EeMUi7dCSil+x9Cmk0u14q2QUorPG9B0cqlWvBVSSvE/9Gg6/aGHYvFWSCnFF63VdHKpVrwVUkrx
jqs2nVyqFW+F/N8X///+bw6zcfk7EAcjr9nmk/hUOftR2yr/B2AKKWEj/78tfop3OkWzeAqpxQ2L
X+udrtUsnkJqccPij3qnRzWLp5Ba3LD4+97pfc3iKaQWNyy+Ufumk0u14imkFjcsvoN32kGzeAqp
xQ2LH+mdjtQsnkJqccPi3/JO39IsnkJqccPid3mnuzSLp5Ba3LD4K97pFc3iKaQWNyzedY2mk0u1
4imkFjcsPtE7TdQsnkJqccPi+3un/TWLp5Ba3LD4Gd7pDM3iKaQWNyz+I+/0I83iKaQWNyz+pHd6
UrN4CqnFDYs31mw6uVQrnkJqccPiQ7zTEM3iKaQWNyw+1zvN1SyeQmpxw+LHeKdjNIunkFrcsPh3
vNN3NIunkFrcsPi93ulezeIppBY3LP6Gd3pDs3gKqcUNi/fs0HRyqVY8hdTihsW39k5baxZPIbW4
YfHPeKfPaBZPIbW4YfGve6evaxZPIbW4YfGbvNNNmsVTSC1uWPxZ7/SsZvEUUosbFm/fsenkUq14
CqnFDYuP8E4jNIunkFrcsPge3mkPzeIppBY3LH6CdzpBs3gKqcUNi1/pna7ULJ5CanHD4g96pwc1
i6eQWtyw+Dve6R3N4imkFjcs3rdT08mlWvEUUosbFp/unaZrFk8htbhh8UO906GaxVNILW5Y/Hzv
dL5m8RRSixsWv8073aZZPIXU4obFX/ROL2oWTyG1uGHxtdZqOrlUK55CanHD4qO902jN4imkFjcs
Pt87zdcsnkJqccPip3inUzSLp5Ba3LD4td7pWs3iKaQWNyz+qHd6VLN4CqnFDYsv9U5LNYunkFrc
sPjAtZtOLtWKp5Ba3LD4Dt5pB83iKaQWNyx+pHc6UrN4CqnFDYt/yzt9S7N4CqnFDYvf5Z3u0iye
Qmpxw+JLvNMSzeIppBY3LN51naaTS7XiKaQWNyw+0TtN1CyeQmpxw+L7e6f9NYunkFrcsPgZ3ukM
zeIppBY3LH6Dd7pBs3gKqcUNiz/hnZ7QLJ5CanHD4o3OTSeXasVTSC1uWHyIdxqiWTyF1OKGxed6
p7maxVNILW5YfJF3WqRZPIXU4obFL/NOl2kWTyG1uGHxe73TvZrFU0gtblj8De/0hmbxFFKLGxbv
2aXp5FKteAqpxQ2Lb+2dttYsnkJqccPiC73TQs3iKaQWNyx+tnc6W7N4CqnFDYvf5J1u0iyeQmpx
w+LPeqdnNYunkFrcsHj7rk0nl2rFU0gtblh8uHcarlk8hdTihsV39067axZPIbW4YfETvNMJmsVT
SC1uWPxK73SlZvEUUosbFn/QOz2oWTyF1OKGxd/xTu9oFk8htbhh8T7rNp1cqhVPIbW4YfHp3mm6
ZvEUUosbFj/UOx2qWTyF1OKGxc/3TudrFk8htbhh8du8022axVNILW5Y/AXv9IJm8RRSixsW77Re
08mlWvEUUosbFh/tnUZrFk8htbhh8fneab5m8RRSixsWP8U7naJZPIXU4obFr/VO12oWTyG1uGHx
R7zTI5rFU0gtblh8qXdaqlk8hdTihsUHdms6uVQrnkJqccPiO3inHTSLp5Ba3LD4kd7pSM3iKaQW
Nyx+kXe6SLN4CqnFDYvf6Z3u1CyeQmpxw+JLvNMSzeIppBY3LN61e9PJpVrxFFKLGxaf6J0mahZP
IbW4YfEF3mmBZvEUUosbFj/dO52uWTyF1OKGxW/wTjdoFk8htbhh8Se80xOaxVNILW5Y3Ck4J5dq
xVNILW5YfIh3GqJZPIXU4obF53inOZrFU0gtblh8kXdapFk8hdTihsUv806XaRZPIbW4YfF7vdO9
msVTSC1uWPwN7/SGZvEUUosbFu/Rs+nkUq14CqnFDYtP8U5TNIunkFrcsPhC77RQs3gKqcUNi5/t
nc7WLJ5CanHD4jd5p5s0i6eQWtyw+LPe6VnN4imkFjcs3m79ppNLteIppBY3LD7cOw3XLJ5CanHD
4rt7p901i6eQWtyw+Ane6QTN4imkFjcsfqV3ulKzeAqpxQ2LP+CdHtAsnkJqccPib3untzWLp5Ba
3LB4nw2aTi7ViqeQWtyw+HTvNF2zeAqpxQ2LH+qdDtUsnkJqccPi53mn8zSLp5Ba3LD4rd7pVs3i
KaQWNyz+gnd6QbN4CqnFDYt36tV0cqlWPIXU4obFR3un0ZrFU0gtblh8vnear1k8hdTihsVP9k4n
axZPIbW4YfFrvNM1msVTSC1uWPwR7/SIZvEUUosbFl/qnZZqFk8htbhh8YG9m04u1YqnkFrcsPgs
7zRLs3gKqcUNix/hnY7QLJ5CanHD4hd5p4s0i6eQWtyw+J3e6U7N4imkFjcsvsQ7LdEsnkJqccPi
Xfs0nVyqFU8htbhh8QneaYJm8RRSixsWX+CdFmgWTyG1uGHx073T6ZrFU0gtblj8Bu90g2bxFFKL
GxZ/wjs9oVk8hdTihsWXe6flmsVTSC1uWHxw36aTS7XiKaQWNyw+xzvN0SyeQmpxw+KLvNMizeIp
pBY3LH6Zd7pMs3gKqcUNi9/jne7RLJ5CanHD4q97p9c1i6eQWtyweI9+TSeXasVTSC1uWHyKd5qi
WTyF1OKGxRd6p4WaxVNILW5Y/GzvdLZm8RRSixsWv9E73ahZPIXU4obFn/FOz2gWTyG1uGHxdhs2
nVyqFU8htbhh8eHeabhm8RRSixsW39077a5ZPIXU4obFj/dOx2sWTyG1uGHxK7zTFZrFU0gtblj8
Ae/0gGbxFFKLGxZ/2zu9rVk8hdTihsX79G86uVQrnkJqccPi073TdM3iKaQWNyx+iHc6RLN4CqnF
DYuf553O0yyeQmpxw+K3eqdbNYunkFrcsPgL3ukFzeIppBY3LN5pQNPJpVrxFFKLGxYf5Z1GaRZP
IbW4YfG9vdPemsVTSC1uWPxk73SyZvEUUosbFr/GO12jWTyF1OKGxR/xTo9oFk8htbhh8aXeaalm
8RRSixsWHzCw6eRSrXgKqcUNi8/yTrM0i6eQWtyw+BHe6QjN4imkFjcsfpF3ukizeAqpxQ2L3+md
7tQsnkJqccPiL3unlzWLp5Ba3LB4l42aTi7ViqeQWtyw+ATvNEGzeAqpxQ2LL/BOCzSLp5Ba3LD4
6d7pdM3iKaQWNyx+vXe6XrN4CqnFDYs/7p0e1yyeQmpxw+LLvdNyzeIppBY3LD5446aTS7XiKaQW
Nyw+xzvN0SyeQmpxw+KLvNMizeIppBY3LH6pd7pUs3gKqcUNi9/jne7RLJ5CanHD4q97p9c1i6eQ
WtyweI9Nmk4u1YqnkFrcsPgU7zRFs3gKqcUNix/onQ7ULJ5CanHD4md5p7M0i6eQWtyw+I3e6UbN
4imkFjcs/ox3ekazeAqpxQ2LtxvUdHKpVjyF1OKGxYd7p+GaxVNILW5YfDfvtJtm8RRSixsWP947
Ha9ZPIXU4obFr/BOV2gWTyG1uGHxB7zTA5rFU0gtblj8be/0tmbxFFKLGxbvPbjp5FKteAqpxQ2L
T/NO0zSLp5Ba3LD4Id7pEM3iKaQWNyx+nnc6T7N4CqnFDYvf6p1u1SyeQmpxw+LPe6fnNYunkFrc
sHjHIU0nl2rFU0gtblh8lHcapVk8hdTihsX39k57axZPIbW4YfGTvdPJmsVTSC1uWPwa73SNZvEU
UosbFn/YOz2sWTyF1OKGxd/zTu9pFk8htbhh8QGbNp1cqhVPIbW4YfFZ3mmWZvEUUosbFj/COx2h
WTyF1OKGxS/0ThdqFk8htbhh8Tu80x2axVNILW5Y/GXv9LJm8RRSixsW77JZ08mlWvEUUosbFp/g
nSZoFk8htbhh8QXeaYFm8RRSixsWP807naZZPIXU4obFr/dO12sWTyG1uGHxx73T45rFU0gtblh8
uXdarlk8hdTihsUHD206uVQrnkJqccPis73TbM3iKaQWNyx+tHc6WrN4CqnFDYtf6p0u1SyeQmpx
w+L3eKd7NIunkFrcsPjr3ul1zeIppBY3LN59WNPJpVrxFFKLGxaf7J0maxZPIbW4YfEDvdOBmsVT
SC1uWPws73SWZvEUUosbFr/RO92oWTyF1OKGxZ/xTs9oFk8htbhh8dWHN51cqhVPIbW4YfFh3mmY
ZvEUUosbFt/NO+2mWTyF1OKGxY/3TsdrFk8htbhh8Su80xWaxVNILW5Y/H7vdL9m8RRSixsWf8s7
vaVZPIXU4obFe49oOrlUK55CanHD4tO80zTN4imkFjcsfoh3OkSzeAqpxQ2Ln+edztMsnkJqccPi
t3inWzSLp5Ba3LD4897pec3iKaQWNyzecfOmk0u14imkFjcsPso7jdIsnkJqccPie3unvTWLp5Ba
3LD4Sd7pJM3iKaQWNyx+tXe6WrN4CqnFDYs/7J0e1iyeQmpxw+Lveaf3NIunkFrcsPiALZpOLtWK
p5Ba3LD4LO80S7N4CqnFDYsf7p0O1yyeQmpxw+IXeqcLNYunkFrcsPgd3ukOzeIppBY3LP6yd3pZ
s3gKqcUNi3cZ2XRyqVY8hdTihsXHe6fxmsVTSC1uWHw/77SfZvEUUosbFj/NO52mWTyF1OKGxa/3
TtdrFk8htbhh8ce90+OaxVNILW5YfJl3WqZZPIXU4obFB41qOrlUK55CanHD4rO902zN4imkFjcs
frR3OlqzeAqpxQ2LX+qdLtUsnkJqccPi93inezSLp5Ba3LD4a97pNc3iKaQWNyzefXTTyaVa8RRS
ixsWn+ydJmsWTyG1uGHxA73TgZrFU0gtblj8LO90lmbxFFKLGxZf7J0WaxZPIbW4YfGnvdPTmsVT
SC1uWHz1MU0nl2rFU0gtblh8mHcaplk8hdTihsV38067aRZPIbW4YfHjvdPxmsVTSC1uWPxy73S5
ZvEUUosbFr/fO92vWTyF1OKGxd/yTm9pFk8htbhh8d5jm04u1YqnkFrcsPg07zRNs3gKqcUNix/s
nQ7WLJ5CanHD4ud6p3M1i6eQWtyw+C3e6RbN4imkFjcs/rx3el6zeAqpxQ2LdxzXdHKpVjyF1OKG
xUd6p5GaxVNILW5YfC/vtJdm8RRSixsWP8k7naRZPIXU4obFr/ZOV2sWTyG1uGHxh73Tw5rFU0gt
blj8Pe/0nmbxFFKLGxbvP77p5FKteAqpxQ2Lz/ROMzWLp5Ba3LD44d7pcM3iKaQWNyx+oXe6ULN4
CqnFDYvf4Z3u0CyeQmpxw+IveaeXNIunkFrcsHjnCU0nl2rFU0gtblh8vHcar1k8hdTihsX38077
aRZPIbW4YfHTvNNpmsVTSC1uWPx673S9ZvEUUosbFn/MOz2mWTyF1OKGxZd5p2WaxVNILW5YfNDE
ppNLteIppBY3LD7bO83WLJ5CanHD4kd7p6M1i6eQWtyw+CXe6RLN4imkFjcsfrd3uluzeAqpxQ2L
v+adXtMsnkJqccPi3bdsOrlUK55CanHD4pO902TN4imkFjcsfqB3OlCzeAqpxQ2Ln+mdztQsnkJq
ccPii73TYs3iKaQWNyz+tHd6WrN4CqnFDYuvvlXTyaVa8RRSixsWH+adhmkWTyG1uGHxed5pnmbx
FFKLGxY/zjsdp1k8hdTihsUv906XaxZPIbW4YfH7vdP9msVTSC1uWPwt7/SWZvEUUosbFu81qenk
Uq14CqnFDYtP9U5TNYunkFrcsPjB3ulgzeIppBY3LH6udzpXs3gKqcUNi9/inW7RLJ5CanHD4s97
p+c1i6eQWtyweIfJTSeXasVTSC1uWHykdxqpWTyF1OKGxffyTntpFk8htbhh8ZO800maxVNILW5Y
/GrvdLVm8RRSixsWf8g7PaRZPIXU4obF3/VO72oWTyG1uGHx/lOaTi7ViqeQWtyw+EzvNFOzeAqp
xQ2LH+6dDtcsnkJqccPiF3qnCzWLp5Ba3LD47d7pds3iKaQWNyz+knd6SbN4CqnFDYt33rrp5FKt
eAqpxQ2Lj/dO4zWLp5Ba3LD4ft5pP83iKaQWNyx+qnc6VbN4CqnFDYtf552u0yyeQmpxw+KPeafH
NIunkFrcsPgy77RMs3gKqcUNiw/apunkUq14CqnFDYvv7J121iyeQmpxw+JHeaejNIunkFrcsPgl
3ukSzeIppBY3LH63d7pbs3gKqcUNi7/mnV7TLJ5CanHD4t23bTq5VCueQmpxw+KTvNMkzeIppBY3
LH6AdzpAs3gKqcUNi5/pnc7ULJ5CanHD4ou902LN4imkFjcs/rR3elqzeAqpxQ2Lrza16eRSrXgK
qcUNiw/1TkM1i6eQWtyw+DzvNE+zeAqpxQ2LH+edjtMsnkJqccPil3unyzWLp5Ba3LD4/d7pfs3i
KaQWNyz+pnd6U7N4CqnFDYv3mtZ0cqlWPIXU4obFp3qnqZrFU0gtblj8YO90sGbxFFKLGxY/1zud
q1k8hdTihsVv9k43axZPIbW4YfHnvNNzmsVTSC1uWLzD9KaTS7XiKaQWNyw+0juN1CyeQmpxw+J7
eae9NIunkFrcsPiJ3ulEzeIppBY3LH6Vd7pKs3gKqcUNiz/knR7SLJ5CanHD4u96p3c1i6eQWtyw
eP8ZTSeXasVTSC1uWHymd5qpWTyF1OKGxQ/zTodpFk8htbhh8Qu80wWaxVNILW5Y/HbvdLtm8RRS
ixsWf8k7vaRZPIXU4obFO2/XdHKpVjyF1OKGxcd5p3GaxVNILW5YfF/vtK9m8RRSixsWP9U7napZ
PIXU4obFr/NO12kWTyG1uGHxx7zTY5rFU0gtblh8mXdaplk8hdTihsU32b7p5FKteAqpxQ2L7+yd
dtYsnkJqccPiR3mnozSLp5Ba3LD4Jd7pEs3iKaQWNyx+t3e6W7N4CqnFDYu/6p1e1SyeQmpxw+Ld
ZjadXKoVTyG1uGHxSd5pkmbxFFKLGxY/wDsdoFk8hdTihsXP9E5nahZPIbW4YfHF3mmxZvEUUosb
Fn/KOz2lWTyF1OKGxVeb1XRyqVY8hdTihsWHeqehmsVTSC1uWHyed5qnWTyF1OKGxY/zTsdpFk8h
tbhh8e95p+9pFk8htbhh8fu8032axVNILW5Y/E3v9KZm8RRSixsW77VD08mlWvEUUosbFp/qnaZq
Fk8htbhh8YO800GaxVNILW5Y/BzvdI5m8RRSixsWv9k73axZPIXU4obFn/NOz2kWTyG1uGHxDjs2
nVyqFU8htbhh8ZHeaaRm8RRSixsW39M77alZPIXU4obFT/ROJ2oWTyG1uGHxq7zTVZrFU0gtblj8
Ie/0kGbxFFKLGxZ/1zu9q1k8hdTihsU33Knp5FKteAqpxQ2Lb++dttcsnkJqccPih3mnwzSLp5Ba
3LD4Bd7pAs3iKaQWNyx+u3e6XbN4CqnFDYu/5J1e0iyeQmpxw+Lr7Nx0cqlWPIXU4obFx3mncZrF
U0gtblh8X++0r2bxFFKLGxY/1Tudqlk8hdTihsWv807XaRZPIbW4YfHfeaffaRZPIbW4YfEPvNMH
msVTSC1uWHyT2U0nl2rFU0gtblh8Z++0s2bxFFKLGxY/yjsdpVk8hdTihsUv9k4XaxZPIbW4YfFf
eqdfahZPIbW4YfFXvdOrmsVTSC1uWLzbnKaTS7XiKaQWNyw+yTtN0iyeQmpxw+IHeKcDNIunkFrc
sPjXvNPXNIunkFrcsPiPvdOPNYunkFrcsPhT3ukpzeIppBY3LL7a3KaTS7XiKaQWNyw+1DsN1Sye
Qmpxw+K7eqddNYunkFrcsPix3ulYzeIppBY3LP497/Q9zeIppBY3LH6fd7pPs3gKqcUNi7/pnd7U
LJ5CanHD4r12aTq5VCueQmpxw+LbeqdtNYunkFrcsPhB3ukgzeIppBY3LH6OdzpHs3gKqcUNi9/s
nW7WLJ5CanHD4s95p+c0i6eQWtyw+Jq7Np1cqhVPIbW4YfEtvNMWmsVTSC1uWHxP77SnZvEUUosb
Fj/RO52oWTyF1OKGxa/yTldpFk8htbhh8d96p99qFk8htbhh8X3aNV0e0iyeQkoofs/qTRe6qRRP
IbW4cfER45pOLtWKt0JqcePidwxtOrlUK94KqcWNi88b0HRyqVa8FVKLGxf/Q4+m0x96KBZvhdTi
xsUXrdV0cqlWvBVSixsX77hq08mlWvFWSC1uWJy/blAtnkL+74v/A+hfW//kM/iTLsjZuHx5nUbj
WfxS+NYp8UecOgPkf/gRDnhbBG43a6ed/vGzdlmb7EVtfxze9sd2jd/jWXbBT/9ku5gdOzfa7kx3
bzRObDRmzDj11A9/9v689Qo+AR/98RM/9K14aRyGz+mI/7Gaa94b0TVp/hELa6rj+7siH7Owz7R9
v/3b/uj++3DUIZ461As6/9vltX3kQ8vb/n8E/+QW+Qntqz1++JFux+1jHtq10XZntt3z70uulmVf
xBFPHeoFHY19SeHF7WsEfvgl6zYaQ9rHvAD/keDuTHc32JdcLcu+iCOeOtQLOhr7ksKL29ce+OEX
dms05raPeW73Rtud6e4G+5KrZdkXccRTh3pBR2NfUnhx+/oSfvhn4XRM+5if7tlouzPd3WBfcrUs
+yKOeOpQL+ho7EsKL25f38cPX2/9RuM77WOuvUGj7c50d4N9ydWy7Is44qlDvaCjsS8pvLh9Pej+
gxui97aPeUGvRtud6e4G+5KrZdkXccRTh3pBR2NfUnhx+3rH/UIXz+aN9jFX79NouzPd3WBfcrUs
+yKOeOpQL+ho7EsKL25ffSH4LxDtuUbMT0HC3ZnubrAvuVqWfRFHPHWoF3Q09iWFF7evGRC7CNCt
14h5Xr9G253p7gb7kqtl2RdxxFOHekFHY19SeHH7+hzE7oDoZ9aIecuGjbY7090N9iVXy7Iv4oin
DvWCjsa+pPDi9nU+xB6F6NlrxHywf6PtznR3g33J1bLsizjiqUO9oKOxLym8uH3dBrHHIHrTGjEf
GtBouzPd3WBfcrUs+yKOeOpQL+ho7EsKL25fL0LsVxB9do2Ydw1stN2Z7m6wL7laln0RRzx1qBd0
NPYlhRe3r7Ug+EOItl8z5g82arTdme5usC+5WpZ9EUc8dagXdDT2JYUXt68xEPtXiI5YM+aXNm60
3ZnubrAvuVqWfRFHPHWoF3Q09iWFF7evfSA2B6J7rBlz1iaNtjvT3Q32JVfLsi/iiKcO9YKOxr6k
8OL2dQrEukD0hDVjdhzUaLsz3d1gX3K1LPsijnjqUC/oaOxLCi9uX9dC7B6IXblmzDsg7u7Me2z2
JVfLsi/iiKcO9YKOxr6k8OL29SjEvgDRB9eM+bnBjbY7090N9iVXy7Iv4oinDvWCjsa+pPDi9vU+
xHpC9J01Y3Yd0mi7M93dYF9ytSz7Io546lAv6GjsSwovbl8bQXAxxPp2iHkdxN2dudhmX3K1LPsi
jnjqUC/oaOxLCi9uXztAbCZEp3eIOXXTRtud6e4G+5KrZdkXccRTh3pBR2NfUnhx+zoSYo9B7NAO
MR+CuLszH7PZl1wty76II5461As6GvuSwovb17cgti9Ez+8Qc8/NGm13prsb7EuulmVfxBFPHeoF
HY19SeHF7esuiP0eYrd1iPkkxN2d+XubfcnVsuyLOOKpQ72go7EvKby4fb0CsfkQfbFDzN3xCXV3
prsb7EuulmVfxBFPHeoFHY19SeHF7WtdCD4CsbU6xnwA4u7OfMRmX3K1LPsijnjqUC/oaOxLCi9u
XxMhNg2iozvGnDys0XZnurvBvuRqWfZFHPHUoV7Q0diXFF7cvvaH2A0Qm98x5jUQd3fmDTb7kqtl
2RdxxFOHekFHY19SeHH7OgNiG0D0lI4x18MH3Z3p7gb7kqtl2RdxxFOHekFHY19SeHH7+hHEToDY
tR1jLsIH3Z15gs2+5GpZ9kUc8dShXtDR2JcUXty+noTYsxB7tGPMJ/BBd2c+a7MvuVqWfRFHPHWo
F3Q09iWFF7evBgS3wp8sbcmx7gm3pLsb7EuulmVfAefxS1uS962an9ZPfl9SeHH7GgKx8/EnAzvF
PBvi7s4832ZfcrUs+yKOeOpQL+ho7EsKL25fcyH2J/zJDp1i/gHi7s78k82+5GpZ9kUc8dShXtDR
2JcUXty+joHYFIge2SnmhM0bbXemuxvsS66WZV/EEU8d6gUdjX1J4cXt6zv+L7jf6hTzdIi7O/Ns
m33J1bLsizjiqUO9oKOxLym8uH3dC7GnIXZnp5iPQdzdmU/b7EuulmVfxBFPHeoFHY19SeHF7esN
iA2G6JJOMQds0Wi7M93dYF9ytSz7Io546lAv6GjsSwovbl89IXgYxLquFfMgiLs78zCbfcnVsuyL
OOKpQ72go7EvKby4fW0NsRshNnGtmNdD3N2ZN9rsS66WZV/EEU8d6gUdjX1J4cXt6zMQew9i+68V
8y2IuzvzPZt9ydWy7Is44qlDvaCjsS8pvLh9fR1iW0L0jLVijhnZaLsz3d1gX3K1LPsijnjqUC/o
aOxLCi9uXzdB7FiI3bBWzC9C3N2Zx9rsS66WZV/EEU8d6gUdjX1J4cXt61mI/RRiT6wV8z8g7u7M
n9rsS66WZV/EEU8d6gUdjX1J4cXtq73/D/LG2jFfg7i7M9+y2ZdcLcu+iCOeOtQLOhr7ksKL29cI
iA2H6JC1Yw4e1Wi7M93dYF9ytSz7Io546lAv6GjsSwovbl97QGwhxOauHXMBxN2dudBmX3K1LPsi
jnjqUC/oaOxLCi9uXydA7GKILVo75jch7u7Mi232JVfLsi/iiKcO9YKOxr6k8OL2dSXEHobYZWvH
vB/i7s582GZfcrUs+yKOeOpQL+ho7EsKL25fD0KsHUTvXTvmcqS7M93dYF9ytSz7Io546lAv6Gjs
Swovbl/vQGw8xN5YO+bI0Y22O3O8zb7kaln2RRzx1KFe0NHYlxRe3L76QvCzEOu5TsxPQ9zdmZ+1
2ZdcLcu+iCOeOtQLOhr7ksKL29cMiF0Asa3XiXkOxN2deYHNvuRqWfZFHPHUoV7Q0diXFF7cvg6F
2N0QW7hOzNsh7u7Mu232JVfLsi/iiKcO9YKOxr6k8OL2dT7E3oTY2evEfBXi7s5802ZfcrUs+yKO
eOpQL+ho7EsKL25ft0GsH0RvWifm+mMabXemuxvsS66WZV/EEU8d6gUdjX1J4cXt60WIzYLYs+vE
nAZxd2fOstmXXC3LvogjnjrUCzoa+5LCi9vXWhA8EmLtO8c8FOLuzjzSZl9ytSz7Io546lAv6Gjs
Swovbl+jIfZtiA3vHPMbEHd35rdt9iVXy7Iv4oinDvWCjsa+pPDi9jUfYndBbPfOMW+DuLsz77LZ
l1wty76II5461As6GvuSwovb1ykQWwKxEzrHfAHi7s5cYrMvuVqWfRFHPHWoF3Q09iWFF7evayHW
GaJXdo7ZYWyj7c50d4N9ydWy7Is44qlDvaCjsS8pvLh9PQqx0RB7sHPM4RB3d+Zom33J1bLsizji
qUO9oKOxLym8uH0thdinIPZ255i7QNzdmZ+y2ZdcLcu+iCOeOtQLOhr7ksKL29dACB4LsT5dYn4R
4u7OPNZmX3K1LPsijnjqUC/oaOxLCi9uXztA7BKITe8S8wKIuzvzEpt9ydWy7Is44qlDvaCjsS8p
vLh9HQmxn0Ps0C4xfwpxd2f+3GZfcrUs+yKOeOpQL+ho7EsKL25f34LY7yF2fpeYT0Lc3Zm/t9mX
XC3LvogjnjrUCzoa+5LCi9vXXRBbDrHbusR8D+Luzlxusy+5WpZ9EUc8dagXdDT2JYUXt68lEOsH
0Re6xFx/XKPtznR3g33J1bLsizjiqUO9oKOxLym8uH11heBkiHXqGnM8xN2dOdlmX3K1LPsijnjq
UC/oaOxLCi9uXxMhNh9io7vG3B3i7s6cb7MvuVqWfRFHPHWoF3Q09iWFF7ev/SF2DMTmd415JMTd
nXmMzb7kaln2RRzx1KFe0NHYlxRe3L7OgNg3IHZK15hfh7i7M79hsy+5WpZ9EUc8dagXdDT2JYUX
t68bIPYjiF3TNea1EHd35o9s9iVXy7Iv4oinDvWCjsa+pPDi9vUExB6A2CNdY94DcXdnPmCzL7la
ln0RRzx1qBd0NPYlhRe3LzyTxksQW9qSz41re1YhX7LZl1wty74CzuOXtiTvL2ntSwovbl9DILYq
RAeuG/PvSHdnurvBvuRqWfZFHPHUoV7Q0diXFF7cvuZCrA/Edlg3Zo/xjbY7s4/NvuRqWfZFHPHU
oV7Q0diXFF7cvo6B2FiIHbluzM0h7u7MsTb7kqtl2RdxxFOHekFHY19SeHH7ugxisyF20boxZ0Lc
3ZmzbfYlV8uyL+KIpw71go7GvqTw4vZ1L8QWQuzOdWMugLi7Mxfa7EuulmVfxBFPHeoFHY19SeHF
7esNiH0JYkvWjXkMxN2d+SWbfcnVsuyLOOKpQ72go7EvKby4ffWE4Dch1nW9mP8GcXdnftNmX3K1
LPsijnjqUC/oaOxLCi9uX1tD7IcQm7hezB9A3N2ZP7TZl1wty76II5461As6GvuSwovb10KI/QJi
C9aL+TOIuzvzFzb7kqtl2RdxxFOHekFHY19SeHH7OhtiT0Ls9PViPgpxd2c+abMvuVqWfRFHPHWo
F3Q09iWFF7evmyD2BsRuWC/mEoi7O/MNm33J1bLsizjiqUO9oKOxLym8uH09C7F2EH1ivZjLke7O
dHeDfcnVsuyLOOKpQ72go7EvKby4fbWHYC/3WekWs9uERtud2ctmX3K1LPsijnjqUC/oaOxLCi9u
XyMgNgJiQ7rFHAJxd2eOsNmXXC3LvogjnjrUCzoa+5LCi9vX7hCbCrE53WJOgri7M6fa7EuulmVf
xBFPHeoFHY19SeHF7esEiO0BsUXdYs6FuLsz97DZl1wty76II5461As6GvuSwovb15UQOxhil3WL
eQDE3Z15sM2+5GpZ9kUc8dShXtDR2JcUXty+HoTYCRC7t1vMRRB3d+YJNvuSq2XZF3HEU4d6QUdj
X1J4cft6x/+NrDe6xfwqxN2d+W82+5KrZdkXccRTh3pBR2NfUnhx++oDwcsh1qN7zEsg7u7My232
JVfLsi/iiKcO9YKOxr6k8OL2NR1iN0JsSveY10Pc3Zk32uxLrpZlX8QRTx3qBR2NfUnhxe3rUIjd
C7GF3WPeCXF3Z95rsy+5WpZ9EUc8dagXdDT2JYUXt6/zIfYExM7uHvMRiLs78wmbfcnVsuyLOOKp
Q72go7EvKby4fd0GsVcgdlP3mC9C3N2Zr9jsS66WZV/EEU8d6gUdjX1J4cXt6wWIvQ+xZ7rHfAfi
7s5832ZfcrUs+yKOeOpQL+ho7EsKL25fnSC4FkTb9YjZfmKj7c50d4N9ydWy7Is44qlDvaCjsS8p
vLh9jYZYH4gN7xGzB8TdndnHZl9ytSz7Io546lAv6GjsSwovbl/zITbM/c3iHjEHQdzdmcNs9iVX
y7Iv4oinDvWCjsa+pPDi9nVK9+bfiD+hR8xxEHd35iSbfcnVsuyLOOKpQ72go7EvKby4fV0LsZ3c
3yzuEXM7iLs7cyebfcnVsuyLOOKpQ72go7EvKby4fT0CsX0g9kCPmHtA3N2Z+9jsS66WZV/EEU8d
6gUdjX1J4cXtaynEDoHY2z1iHghxd2ceYrMvuVqWfRFHPHWoF3Q09iWFF7evgfA5FmJ9esb8IsTd
nXmszb7kaln2RRzx1KFe0NHYlxRe3L52gM8ZEJveM+YpEHd35hk2+5KrZdkXccRTh3pBR2NfUnhx
+zoSPhdC7NCeMc+FuLszL7TZl1wty76II5461As6GvuSwovb10Xw+T7EzusZ8zsQd3fm9232JVfL
si/iiKcO9YKOxr6k8OL2dSd8FkPs1p4xr4O4uzMX2+xLrpZlX8QRTx3qBR2NfUnhxe1rCXzugtgL
PWPeBnF3Z95lsy+5WpZ9EUc8dagXdDT2JYUXt6+uEHwIYp3Wj3kfxN2d+ZDNvuRqWfZFHPHUoV7Q
0diXFF7cviZC7BmIjV4/5uMQd3fmMzb7kqtl2RdxxFOHekFHY19SeHH72h9if4LY/PVj/gHi7s78
k82+5GpZ9kUc8dShXtDR2JcUXty+TofYuxA7ef2Yb0Lc3Znv2uxLrpZlX8QRTx3qBR2NfUnhxe3r
Boi1g+g168dcjnR3prsb7EuulmVfxBFPHeoFHY19SeHF7esJiK3r/mbx+jHX2rLRdmeua7MvuVqW
fRFHPHWoF3Q09iWFF7evBgT7QWxpS64P8UZL9rPZl1wty74CzuOXtiTv/bT2JYUXt68hEBsKsYEb
xNwE4u7OHGqzL7laln0RRzx1qBd0NPYlhRe3rzkQGw+xWRvEHAlxd2eOt9mXXC3LvogjnjrUCzoa
+5LCi9vXIohNg9gRG8ScDHF3Z06z2ZdcLcu+iCOeOtQLOhr7ksKL29dlEJsDsYs2iDkL4u7OnGOz
L7laln0RRzx1qBd0NPYlhRe3r3shNh9id24Qc3eIuztzvs2+5GpZ9kUc8dShXtDR2JcUXty+3oDY
Qogt2SDmAoi7O3Ohzb7kaln2RRzx1KFe0NHYlxRe3L56QPBIiHXpFfNQiLs780ibfcnVsuyLOOKp
Q72go7EvKby4fU2B2AkQm9Ar5iKIuzvzBJt9ydWy7Is44qlDvaCjsS8pvLh9LYTY6RBb0CvmyRB3
d+bpNvuSq2XZF3HEU4d6QUdjX1J4cfs6G2LnQez0XjHPgri7M8+z2ZdcLcu+iCOeOtQLOhr7ksKL
29dNELsUYjf0inkhxN2deanNvuRqWfZFHPHUoV7Q0diXFF7cvp6F2NUQe6JXzMsh7u7Mq232JVfL
si/iiKcO9YKOxr6k8OL21Q6CiyG2vCWvg3i7llxssy+5WpZ9EUf88pbkfbHWvqTw4vY1HGI/h9jg
3jF/CnF3Z/7cZl9ytSz7Io546lAv6GjsSwovbl+7Q+xXEJvTO+ZdEHd35q9s9iVXy7Iv4oinDvWC
jsa+pPDi9nUCxB6F2KLeMR+EuLszH7XZl1wty76II5461As6GvuSwovb15UQewZil/WO+TjE3Z35
jM2+5GpZ9kUc8dShXtDR2JcUXty+HoDYSxC7p3fM5yDu7syXbPYlV8uyL+KIpw71go7GvqTw4vb1
NsT+DLHXe8d8BeLuzvyzzb7kaln2RRzx1KFe0NHYlxRe3L76QHApxHr0ifk2xN2dudRmX3K1LPsi
jnjqUC/oaOxLCi9uX9MhtjpEp/SJuQzp7kx3N9iXXC3LvogjnjrUCzoa+5LCi9vXoRBbG2IL+8Rc
Y6tG2525ts2+5GpZ9kUc8dShXtDR2JcUXty+zodYT4id3SdmV4i7O7Onzb7kaln2RRzx1KFe0NHY
lxRe3L5uhVh/iN3YJ2YviLs7s7/NvuRqWfZFHPHUoV7Q0diXFF7cvl6A2KYQe6ZPzI0g7u7MTW32
JVfLsi/iiKcO9YKOxr6k8OL21QmCoyDWrm/MYRB3d+Yom33J1bLsizjiqUO9oKOxLym8uH2NhthW
EBveN+ZYiLs7cyubfcnVsuyLOOKpQ72go7EvKby4fc2H2HSI7d435hSIuztzus2+5GpZ9kUc8dSh
XtDR2JcUXty+TobYThA7vm/M7SDu7sydbPYlV8uyL+KIpw71go7GvqTw4vZ1TZ/m/1Dzir4x50Dc
3Zm72+xLrpZlX8QRTx3qBR2NfUnhxe3rEYjtC7EH+sbcE+LuztzXZl9ytSz7Io546lAv6GjsSwov
bl9LIbYQYm/3jbkA4u7OXGizL7laln0RRzx1qBd0NPYlhRe3r4EAfh5iffrF/CzE3Z35eZt9ydWy
7Is44qlDvaCjsS8pvLh9zerb/B8CT+sX8wiIuztzkc2+5GpZ9kUc8dShXtDR2JcUXty+jgDwyxA7
pF/M/wNxd2d+2WZfcrUs+yKOeOpQL+ho7EsKL25fFwF4GsTO6xfzJIi7O/M0m33J1bLsizjiqUO9
oKOxLym8uH3dCeDXIXZrv5hnQNzdmV+32ZdcLcu+iCOeOtQLOhr7ksKL29cSAC+A2Av9Yp4DcXdn
XmCzL7laln0RRzx1qBd0NPYlhRe3r64QvBRinTaMeSHE3Z15qc2+5GpZ9kUc8dShXtDR2JcUXty+
JkDs+xAbtWHM70Dc3Znft9mXXC3LvogjnjrUCzoa+5LCi9vXgn7Nf5DB3hvGvAri7s68zmZfcrUs
+yKOeOpQL+ho7EsKL25fp0PsRoidvGHM6yHu7swbbfYlV8uyL+KIpw71go7GvqTw4vZ1A8R+BrFr
Noz5E4i7O/NnNvuSq2XZF3HEU4d6QUdjX1J4cft6AmK/hNgjG8b8OcTdnflLm33J1bLsizjiqUO9
oKOxLym8uH0th9j9EHtvw5h3Q9zdmffb7EuulmVfxBFPHeoFHY19SeHF7WswBB+B2ID+MR+AuLsz
H7HZl1wty76II5461As6GvuSwovb1xyI/RfEZvWP+Z8Qd3fmf9nsS66WZV/EEU8d6gUdjX1J4cXt
a9GGzX8QyxH9Yz4FcXdnPmezL7laln0RRzx1qBd0NPYlhRe3r8sg9jLELuof83mIuzvzZZt9ydWy
7Is44qlDvaCjsS8pvLh93QuxNyB2Z/+YSyDu7sw3bPYlV8uyL+KIpw71go7GvqTw4vb1OsTegdjL
/WO+AXF3Z75jsy+5WpZ9EUc8dagXdDT2JYUXt68eEPw7xLoMiPlXiLs78+82+5KrZdkXccRTh3pB
R2NfUnhx+5oCsdXc3yweEPMDpLsz3d1gX3K1LPsijnjqUC/oaOxLCi9uXwsh1tH9zeIBMVef1Gi7
Mzva7EuulmVfxBFPHeoFHY19SeHF7etsiHWB2OkDYnaEuLszu9jsS66WZV/EEU8d6gUdjX1J4cXt
60aI9YTY9QNidoW4uzN72uxLrpZlX8QRTx3qBR2NfUnhxe3rGYj1hdjjA2L2hLi7M/va7EuulmVf
xBFPHeoFHY19SeHF7avdgOY/qGx5S/aFeLuW3MhmX3K1LPsijvjlLcn7Rlr7ksKL29dwiG0GscED
Y24McXdnbmazL7laln0RRzx1qBd0NPYlhRe3r90htgXE5gyMuRnE3Z25hc2+5GpZ9kUc8dShXtDR
2JcUXty+ToDYOIgtGhhzC4i7O3Oczb7kaln2RRzx1KFe0NHYlxRe3L6ugNgkiF06MOY4iLs7c5LN
vuRqWfZFHPHUoV7Q0diXFF7cvh6A2FSI3TMw5iSIuztzqs2+5GpZ9kUc8dShXtDR2JcUXty+3obY
TIi9PjDmVIi7O3Omzb7kaln2RRzx1KFe0NHYlxRe3L76QHA2xHpsFHMmxN2dOdtmX3K1LPsijnjq
UC/oaOxLCi9uX9MhthvEpmwUczbE3Z25m82+5GpZ9kUc8dShXtDR2JcUXty+DoHY3hA7cKOYu0Hc
3Zl72+xLrpZlX8QRTx3qBR2NfUnhxe3rvIHNf5DnWRvF3Bvi7s5cYLMvuVqWfRFHPHWoF3Q09iWF
F7evWyG2EGI3bhRzAcTdnbnQZl9ytSz7Io546lAv6GjsSwovbl8vQOwQiD2zUcwDIe7uzENs9iVX
y7Iv4oinDvWCjsa+pPDi9tVpo+Y/KLbdxjEPgbi7M4+w2ZdcLcu+iCOeOtQLOhr7ksKL29coiB0N
sWEbxzwc4u7OPNpmX3K1LPsijnjqUC/oaOxLCi9uX3tD7HiI7bZxzKMh7u7M4232JVfLsi/iiKcO
9YKOxr6k8OL2dfJGzX8Q8fEbxzwO4u7OPMlmX3K1LPsijnjqUC/oaOxLCi9uX9dA7DSIXbFxzJMg
7u7M02z2JVfLsi/iiKcO9YKOxr6k8OL29QjEzoTYAxvHPBXi7s4802ZfcrUs+yKOeOpQL+ho7EsK
L25fSyF2LsTe3jjmmRB3d+a5NvuSq2XZF3HEU4d6QUdjX1J4cfsaAMELINZ7k5jnQNzdmRfY7Euu
lmVfxBFPHeoFHY19SeHF7WsWxC6G2LRNYn4T4u7OvNhmX3K1LPsijnjqUC/oaOxLCi9uX0dA7LsQ
O2STmN+GuLszv2uzL7laln0RRzx1qBd0NPYlhRe3r4sg9n2InbdJzO9A3N2Z37fZl1wty76II546
1As6GvuSwovb150QuxZit24S80qIuzvzWpt9ydWy7Is44qlDvaCjsS8pvLh9vQyxGyD2/CYxr4G4
uzNvsNmXXC3LvogjnjrUCzoa+5LCi9tXFwjeCLGOg2JeD3F3Z95osy+5WpZ9EUc8dagXdDT2JYUX
t68JELsFYqMGxVwMcXdn3mKzL7laln0RRzx1qBd0NPYlhRe3rwUQux1iew+KeTPE3Z15u82+5GpZ
9kUc8dShXtDR2JcUXty+TofYLyF28qCYP4e4uzN/abMvuVqWfRFHPHWoF3Q09iWFF7evGyD2a4hd
MyjmLyDu7sxf2+xLrpZlX8QRTx3qBR2NfUnhxe3rcYj9BmIPD4r5K4i7O/M3NvuSq2XZF3HEU4d6
QUdjX1J4cftaDrFHIfbeoJgPQtzdmY/a7EuulmVfxBFPHeoFHY19SeHF7WswBB+H2IDBMR+GuLsz
H7fZl1wty76II5461As6GvuSwovb1xyIPQ2xWYNjPgZxd2c+bbMvuVqWfRFHPHWoF3Q09iWFF7ev
RRB7DmJHDI75FMTdnfmczb7kaln2RRzx1KFe0NHYlxRe3L4uhdgfIHbh4Ji/h7i7M/9gsy+5WpZ9
EUc8dagXdDT2JYUXt697IPYKxO4YHPNFiLs78xWbfcnVsuyLOOKpQ72go7EvKby4fb0Osdch9vLg
mC9D3N2Zr9vsS66WZV/EEU8d6gUdjX1J4cXtqwcE34JYlyExX4O4uzPfstmXXC3LvogjnjrUCzoa
+5LCi9vXFIj9DWIThsT8C8Tdnfk3m33J1bLsizjiqUO9oKOxLym8uH0dCLH3IbbfkJjvQNzdme/b
7EuulmVfxBFPHeoFHY19SeHF7essiDUgetqQmEuR7s50d4N9ydWy7Is44qlDvaCjsS8pvLh93Ti4
+S96u35IzGVId2eubrMvuVqWfRFHPHWoF3Q09iWFF7evZyC2JsQeHxJz1cmNtjtzTZt9ydWy7Is4
4qlDvaCjsS8pvLh9tYPg2hBb3pJrQLxdS65tsy+5WpZ9EUf88pbkfW2tfUnhxe1r+JDmv0hw8KYx
O0Hc3ZldbfYlV8uyL+KIpw71go7GvqTw4va1G8S6Q2z2pjE7Q9zdmd1t9iVXy7Iv4oinDvWCjsa+
pPDi9nU8xDaA2NGbxlwP4u7O3MBmX3K1LPsijnjqUC/oaOxLCi9uX1cMaf6LKi/dNGZPiLs7s6/N
vuRqWfZFHPHUoV7Q0diXFF7cvh6A2ACI3bNpzN4Qd3fmAJt9ydWy7Is44qlDvaCjsS8pvLh9vQ2x
TSD2+qYxN4S4uzM3sdmXXC3LvogjnjrUCzoa+5LCi9tX702b/yLU7pvF3Bji7s7czGZfcrUs+yKO
eOpQL+ho7EsKL25f0yA2HGKTN4s5GOLuzhxusy+5WpZ9EUc8dagXdDT2JYUXt69DIDYSYgduFnMo
xN2dOdJmX3K1LPsijnjqUC/oaOxLCi9uX+dBbCzEztos5uYQd3fmWJt9ydWy7Is44qlDvaCjsS8p
vLh93QqxiRC7cbOYoyHu7syJNvuSq2XZF3HEU4d6QUdjX1J4cft6AWKTIfbMZjHHQ9zdmZNt9iVX
y7Iv4oinDvWCjsa+pPDi9tURgttCbPWhMbeCuLszt7XZl1wty76II5461As6GvuSwovb1yiIzYDY
sKExt4a4uzNn2OxLrpZlX8QRTx3qBR2NfUnhxe1rb4jNgthuQ2NOg7i7M2fZ7EuulmVfxBFPHeoF
HY19SeHF7etkiO0EseOHxtwO4u7O3MlmX3K1LPsijnjqUC/oaOxLCi9uX9dAbC7ErhgacweIuztz
rs2+5GpZ9kUc8dShXtDR2JcUXty+Ht6s+S+iv39ozNkQd3fmbjb7kqtl2RdxxFOHekFHY19SeHH7
eg9in4LYW0Nj7gJxd2d+ymZfcrUs+yKOeOpQL+ho7EsKL25fAyA4H2K9h8XcHeLuzpxvsy+5WpZ9
EUc8dagXdDT2JYUXt69ZENsPYtOGxdwL4u7O3M9mX3K1LPsijnjqUC/oaOxLCi9uX0dA7NMQO2RY
zH0g7u7MT9vsS66WZV/EEU8d6gUdjX1J4cXt6yKILYTYecNiLoC4uzMX2uxLrpZlX8QRTx3qBR2N
fUnhxe3rDogdBLFbhsX8fyDu7syDbPYlV8uyL+KIpw71go7GvqTw4vb1MsQ+B7Hnh8X8DMTdnfk5
m33J1bLsizjiqUO9oKOxLym8uH11geDhEOs4PObBEHd35uE2+5KrZdkXccRTh3pBR2NfUnhx+5oA
sS9CbNTwmJ+HuLszv2izL7laln0RRzx1qBd0NPYlhRe3rwUQWwSxvYfHPALi7s5cZLMvuVqWfRFH
PHWoF3Q09iWFF7ev0yB2HMROGh7zKIi7O/M4m33J1bLsizjiqUO9oKOxLym8uH1dD7EvQezq4TGP
gbi7M79ksy+5WpZ9EUc8dagXdDT2JYUXt6/HIXYSxB4eHvM4iLs78ySbfcnVsuyLOOKpQ72go7Ev
Kby4fS2H2Fcg9t7wmF+GuLszv2KzL7laln0RRzx1qBd0NPYlhRe3r8H4xukQGzAi5skQd3fm6Tb7
kqtl2RdxxFOHekFHY19SeHH7mo1vfA1iM0fE/ArE3Z35NZt9ydWy7Is44qlDvaCjsS8pvLh9HY1v
nA2xw0fEPB3i7s4822ZfcrUs+yKOeOpQL+ho7EsKL25fl+Ib50LswhExz4S4uzPPtdmXXC3Lvogj
njrUCzoa+5LCi9vXPfjGNyB2x4iYX4e4uzO/YbMvuVqWfRFHPHWoF3Q09iWFF7ev1/GNCyH28oiY
50Lc3ZkX2uxLrpZlX8QRTx3qBR2NfUnhxe2rBwS/DbEum8f8BsTdnfltm33J1bLsizjiqUO9oKOx
Lym8uH1NhtilEBu/ecwLIe7uzEtt9iVXy7Iv4oinDvWCjsa+pPDi9nUgxL4Hsf02j3kxxN2d+T2b
fcnVsuyLOOKpQ72go7EvKby4fZ0FsSshdtrmMS+DuLszr7TZl1wty76II5461As6GvuSwovb140Q
uwpi128e83sQd3fmVTb7kqtl2RdxxFOHekFHY19SeHH7egZi10Ls8c1jXglxd2dea7MvuVqWfRFH
PHWoF3Q09iWFF7ev1SF4HcSWteRVEF+9Ja+z2ZdcLcu+iCN+WUvyfp3WvqTw4vY1DGI/gtigLWJe
C3F3Z/7IZl9ytSz7Io546lAv6GjsSwovbl+7QWwxxGZvEfM6iLs7c7HNvuRqWfZFHPHUoV7Q0diX
FF7cvo6H2E8gdvQWMX8EcXdn/sRmX3K1LPsijnjqUC/oaOxLCi9uX1dA7BaIXbpFzMUQd3fmLTb7
kqtl2RdxxFOHekFHY19SeHH7egBiP4PYPVvE/AnE3Z35M5t9ydWy7Is44qlDvaCjsS8pvLh9vQWx
OyD22hYxb4G4uzPvsNmXXC3LvogjnjrUCzoa+5LCi9tXbwjeBbHuI2PeBnF3Z95lsy+5WpZ9EUc8
dagXdDT2JYUXt69pELsbYpNHxrwd4u7OvNtmX3K1LPsijnjqUC/oaOxLCi9uX4dA7FcQO3BkzLsg
7u7MX9nsS66WZV/EEU8d6gUdjX1J4cXt6zyI3Q+xs0bGvBvi7s6832ZfcrUs+yKOeOpQL+ho7EsK
L25ft0DsNxBbPDLmryDu7szf2OxLrpZlX8QRTx3qBR2NfUnhxe3reYg9DLGnR8a8H+LuznzYZl9y
tSz7Io546lAv6GjsSwovbl8dIfgoxFYfFfNBiLs781GbfcnVsuyLOOKpQ72go7EvKby4fY2C2GMQ
GzYq5kMQd3fmYzb7kqtl2RdxxFOHekFHY19SeHH72htiT0Jst1ExH4W4uzOftNmXXC3LvogjnjrU
Czoa+5LCi9vXSRB7GmLHjYr5GMTdnfm0zb7kaln2RRzx1KFe0NHYlxRe3L6uhtizELt8VMwnIO7u
zGdt9iVXy7Iv4oinDvWCjsa+pPDi9vUwxJ6D2P2jYj4FcXdnPmezL7laln0RRzx1qBd0NPYlhRe3
r/cg9iLE3hoV81mIuzvzRZt9ydWy7Is44qlDvaCjsS8pvLh9DYDgHyHWe3TM/4a4uzP/aLMvuVqW
fRFHPHWoF3Q09iWFF7evWRBbArFpo2O+AHF3Zy6x2ZdcLcu+iCOeOtQLOhr7ksKL29fhEHsVYgeP
jvlHiLs781WbfcnVsuyLOOKpQ72go7EvKby4fV0Isdchdu7omC9D3N2Zr9vsS66WZV/EEU8d6gUd
jX1J4cXt6w6I/QVit4yO+SeIuzvzLzb7kqtl2RdxxFOHekFHY19SeHH7ehlib0Hs+dExX4O4uzPf
stmXXC3LvogjnjrUCzoa+5LCi9tXFwj+FWIdx8T8M8TdnflXm33J1bLsizjiqUO9oKOxLym8uH2N
h9h7EBs5JuZbEHd35ns2+5KrZdkXccRTh3pBR2NfUnhx+9oPYu9DbK8xMd+BuLsz37fZl1wty76I
I5461As6GvuSwovb12kQWwaxk8bEfBfi7s5cZrMvuVqWfRFHPHWoF3Q09iWFF7ev6yHWcH+zeEzM
pUh3Z7q7wb7kaln2RRzx1KFe0NHYlxRe3L4eh9hq7m8Wj4n5AdLdmavZ7EuulmVfxBFPHeoFHY19
SeHF7Ws5xNpB7L0xMZcj3Z3ZzmZfcrUs+yKOeOpQL+ho7EsKL25fgyC4BsT6j425ypRG2525hs2+
5GpZ9kUc8dShXtDR2JcUXty+ZkOsI8Rmjo25OsTdndnRZl9ytSz7Io546lAv6GjsSwovbl9HQ2wt
iB0+NmZ7iLs7cy2bfcnVsuyLOOKpQ72go7EvKby4fV0Ksc4Qu3BszA4Qd3dmZ5t9ydWy7Is44qlD
vaCjsS8pvLh93QOxrhC7Y2zMThB3d2ZXm33J1bLsizjiqUO9oKOxLym8uH29BrFuEHtpbMx1IO7u
zG42+5KrZdkXccRTh3pBR2NfUnhx++oOwR4Q6zwuZheIuzuzh82+5GpZ9kUc8dShXtDR2JcUXty+
JkNsfYiNHxdzXYi7O3N9m33J1bLsizjiqUO9oKOxLym8uH0dCLHeENtvXMzuEHd3Zm+bfcnVsuyL
OOKpQ72go7EvKby4fZ0Fsb4QO21czJ4Qd3dmX5t9ydWy7Is44qlDvaCjsS8pvLh9LYbYhhC7blzM
DSDu7swNbfYlV8uyL+KIpw71go7GvqTw4vb1NMQGQOyxcTF7Q9zdmQNs9iVXy7Iv4oinDvWCjsa+
pPDi9rU6BDeG2LKW7Afx1VtyY5t9ydWy7Is44pe1JO8ba+1LCi9uX8MgNghig8bH7A9xd2cOstmX
XC3LvogjnjrUCzoa+5LCi9vXbhAbArHZ42MOhLi7M4fY7EuulmVfxBFPHeoFHY19SeHF7et4iG0G
saPHx9wY4u7O3MxmX3K1LPsijnjqUC/oaOxLCi9uX5dDbDjELhkfczDE3Z053GZfcrUs+yKOeOpQ
L+ho7EsKL25f90Nsc4jdPT7mphB3d+bmNvuSq2XZF3HEU4d6QUdjX1J4cft6C2IjIfba+JhDIe7u
zJE2+5KrZdkXccRTh3pBR2NfUnhx++oNwdEQ6z4h5nCIuztztM2+5GpZ9kUc8dShXtDR2JcUXty+
pkFsLMQmT4i5OcTdnTnWZl9ytSz7Io546lAv6GjsSwovbl8HQ2wCxA6YEHMUxN2dOcFmX3K1LPsi
jnjqUC/oaOxLCi9uX+dCbEuInTkh5hiIuztzS5t9ydWy7Is44qlDvaCjsS8pvLh93QKxSRBbPCHm
OIi7O3OSzb7kaln2RRzx1KFe0NHYlxRe3L6eh9gUiD09IeYEiLs7c4rNvuRqWfZFHPHUoV7Q0diX
FF7cvjpCcBuIrT4x5pYQd3fmNjb7kqtl2RdxxFOHekFHY19SeHH7GgWxqRAbNjHmJIi7O3Oqzb7k
aln2RRzx1KFe0NHYlxRe3L72gth0iM2bGHMKxN2dOd1mX3K1LPsijnjqUC/oaOxLCi9uXydBbDuI
HTcx5jYQd3fmdjb7kqtl2RdxxFOHekFHY19SeHH7uhpiMyF2+cSYUyHu7syZNvuSq2XZF3HEU4d6
QUdjX1J4cft6GGI7Quz+iTFnQNzdmTva7EuulmVfxBFPHeoFHY19SeHF7es9iO0Msbcmxtwe4u7O
3NlmX3K1LPsijnjqUC/oaOxLCi9uX/0hOAdivbaMOQvi7s6cY7MvuVqWfRFHPHWoF3Q09iWFF7ev
mRDbBWJTt4y5I8TdnbmLzb7kaln2RRzx1KFe0NHYlxRe3L4Oh9g8iB28ZcydIe7uzHk2+5KrZdkX
ccRTh3pBR2NfUnhx+7oQYrtD7NwtY86BuLszd7fZl1wty76II5461As6GvuSwovb1x0Q+xTEbtky
5i4Qd3fmp2z2JVfLsi/iiKcO9YKOxr6k8OL29TLE9oLY81vGnAdxd2fuZbMvuVqWfRFHPHWoF3Q0
9iWFF7evzhCcD7EOW8XcHeLuzpxvsy+5WpZ9EUc8dagXdDT2JYUXt6/xENsHYiO3irkHxN2duY/N
vuRqWfZFHPHUoV7Q0diXFF7cvvaD2L4Q22urmHtC3N2Z+9rsS66WZV/EEU8d6gUdjX1J4cXt6zSI
LYDYSVvF3Bvi7s5cYLMvuVqWfRFHPHWoF3Q09iWFF7ev6yH2aYhdvVXMfSDu7sxP2+xLrpZlX8QR
Tx3qBR2NfUnhxe3rMYgdALGHtoq5L8TdnXmAzb7kaln2RRzx1KFe0NHYlxRe3L6WQWwhxN7dKuYC
iLs7c6HNvuRqWfZFHPHUoV7Q0diXFF7cvgZB8LMQ6z8p5qch7u7Mz9rsS66WZV/EEU8d6gUdjX1J
4cXtazbEDobYzEkxD4C4uzMPttmXXC3LvogjnjrUCzoa+5LCi9vX0RA7FGKHT4q5EOLuzjzUZl9y
tSz7Io546lAv6GjsSwovbl+XQOzzELtgUszPQtzdmZ+32ZdcLcu+iCOeOtQLOhr7ksKL29fdEDsM
YrdPinkQxN2deZjNvuRqWfZFHPHUoV7Q0diXFF7cvl6D2BEQe2lSzEMg7u7MI2z2JVfLsi/iiKcO
9YKOxr6k8OL21R2CX5jc/JcHMj8HcXdnfsFmX3K1LPsijnjqUC/oaOxLCi9uX5MhdhTExk+OeRjE
3Z15lM2+5GpZ9kUc8dShXtDR2JcUXty+DoTYIojtNznmERB3d+Yim33J1bLsizjiqUO9oKOxLym8
uH2dCbFjIHbq5JhHQtzdmcfY7EuulmVfxBFPHeoFHY19SeHF7WsxxI6F2HWTY34R4u7OPNZmX3K1
LPsijnjqUC/oaOxLCi9uX09D7HiIPTY55tEQd3fm8Tb7kqtl2RdxxFOHekFHY19SeHH7Wh1SX8Lv
y1ryGIiv3pJfstmXXC3LvogjfllL8v4lrX1J4cXtaxikTsTvg6bEPBa/uzvzRJt9ydWy7Is44qlD
vaCjsS8pvLh9uf9R5kmTm/9jTeZxU5r/Y03mSTb7kqtl2RdxxFOHekFHY19SeHH7Og5Sp+D3o6bE
PMHJT455is2+5GpZ9kUc8dShXtDR2JcUXty+LofUV/D7JVNifhm/uzvzKzb7kqtl2RdxxFOHekFH
Y19SeHH7uh9Sp+H3u6fEPAm/uzvzNJt9ydWy7Is44qlDvaCjsS8pvLh9vQWp0/H7a1Ninozf3Z15
us2+5GpZ9kUc8dShXtDR2JcUXty+3L8k/KsQ6751zH+d0vyXhzO/arMvuVqWfRFHPHWoF3Q09iWF
F7cv9w/xPBNik7aOeeqU5j/ck3mmzb7kaln2RRzx1KFe0NHYlxRe3L4OhthZEDtg65in4ebuzLNs
9iVXy7Iv4oinDvWCjsa+pPDi9nUuxL4OsTO3jnkGbu7O/LrNvuRqWfZFHPHUoV7Q0diXFF7cvm6B
2DkQW7x1zK/h5u7Mc2z2JVfLsi/iiKcO9YKOxr6k8OL29bz7CwHEnt465pm4uTvzXJt9ydWy7Is4
4qlDvaCjsS8pvLh9dYDg+RBbbZuYZ0PW3Znn2+xLrpZlX8QRTx3qBR2NfUnhxe1rJMS+CbGh28T8
N4i7O/ObNvuSq2XZF3HEU4d6QUdjX1J4cfvaC2IXQGzeNjHPgbi7My+w2ZdcLcu+iCOeOtQLOhr7
ksKL29dJELsIYsdtE/M8iLs78yKbfcnVsuyLOOKpQ72go7EvKby4fV0NsW9D7PJtYn4D4u7O/LbN
vuRqWfZFHPHUoV7Q0diXFF7cvh6C2MUQu2+bmN+EuLszL7bZl1wty76II5461As6GvuSwovb17sQ
uxRib24T80KIuzvzUpt9ydWy7Is44qlDvaCjsS8pvLh99YfgZRDrtW3MiyDu7szLbPYlV8uyL+KI
pw71go7GvqTw4vY1E2LfhdjUbWN+G+Luzvyuzb7kaln2RRzx1KFe0NHYlxRe3L4O36b5//x48LYx
3f9TpLszL7fZl1wty76II5461As6GvuSwovb14UQuwJi524b81KIuzvzCpt9ydWy7Is44qlDvaCj
sS8pvLh93Q6x70Ps5m1jfgfi7s78vs2+5GpZ9kUc8dShXtDR2JcUXty+XoLYDyD23LYxvwtxd2f+
wGZfcrUs+yKOeOpQL+ho7EsKL25fnSF4NcQ6TI15OcTdnXm1zb7kaln2RRzx1KFe0NHYlxRe3L7G
Q+waiI2cGvMKiLs78xqbfcnVsuyLOOKpQ72go7EvKby4fe0HsX+H2F5TY34f4u7O/HebfcnVsuyL
OOKpQ72go7EvKby4fZ0KsR9C7MSpMX8AcXdn/tBmX3K1LPsijnjqUC/oaOxLCi9uX9dB7HqIXTU1
5tUQd3fm9Tb7kqtl2RdxxFOHekFHY19SeHH7egxiN0Dsoakxr4G4uzNvsNmXXC3LvogjnjrUCzoa
+5LCi9vXMoj9B8TenRrz3yHu7sz/sNmXXC3LvogjnjrUCzoa+5LCi9vXIAj+GGL9p8X8IcTdnflj
m33J1bLsizjiqUO9oKOxLym8uH3NhtiNEJs5Leb1EHd35o02+5KrZdkXccRTh3pBR2NfUnhx+zoK
YjdB7LBpMW+AuLszb7LZl1wty76II5461As6GvuSwovb1yUQ+ynELpgW8z8g7u7Mn9rsS66WZV/E
EU8d6gUdjX1J4cXt626I3Qyx26fF/DHE3Z15s82+5GpZ9kUc8dShXtDR2JcUXty+XoPYrRB7aVrM
GyHu7sxbbfYlV8uyL+KIpw71go7GvqTw4vbVHYK3Qazz9Jg3QdzdmbfZ7EuulmVfxBFPHeoFHY19
SeHF7WsSxH4OsXHTY/4U4u7O/LnNvuRqWfZFHPHUoV7Q0diXFF7cvg6A2O0Q23d6zJsh7u7M2232
JVfLsi/iiKcO9YKOxr6k8OL2dSbE7oTYqdNj3gpxd2feabMvuVqWfRFHPHWoF3Q09iWFF7evxRC7
C2LXTY95G8TdnXmXzb7kaln2RRzx1KFe0NHYlxRe3L6ehtgvIPbY9Jg/g7i7M39hsy+5WpZ9EUc8
dagXdDT2JYUXt6/V8BPfDbEPWvJ2iK/Wknfb7EuulmVfxBH/QUvyfrfWvqTw4vY1FD/xPRDbZEbM
OyDu7sx7bPYlV8uyL+KIpw71go7GvqTw4vY1Dz/xryC284yYd0Hc3Zm/stmXXC3LvogjnjrUCzoa
+5LCi9vXcfiJfw2xo2bE/AXE3Z35a5t9ydWy7Is44qlDvaCjsS8pvLh9XY6f+D6IXTIj5i8h7u7M
+2z2JVfLsi/iiKcO9YKOxr6k8OL2dT9+4gcgdveMmPdA3N2ZD9jsS66WZV/EEU8d6gUdjX1J4cXt
6038xA9C7NUZMe+FuLszH7TZl1wty76II5461As6GvuSwovbVy/8NL+FWLftYv4a4u7O/K3NvuRq
WfZFHPHUoV7Q0diXFF7cvqbip3kIYpO2i3kfxN2d+ZDNvuRqWfZFHPHUoV7Q0diXFF7cvg7GT/Mw
xA7YLub9EHd35sM2+5KrZdkXccRTh3pBR2NfUnhx+zoXP82jEDtzu5gPQtzdmY/a7EuulmVfxBFP
HeoFHY19SeHF7etm/DT/CbEfbxfzNxB3d+Z/2uxLrpZlX8QRTx3qBR2NfUnhxe3rOfw0v4PYU9vF
/C3E3Z35O5t9ydWy7Is44qlDvaCjsS8pvLh9dYDg4xBbbfuYD0Pc3ZmP2+xLrpZlX8QRTx3qBR2N
fUnhxe1rJMSegNjQ7WM+AnF3Zz5hsy+5WpZ9EUc8dagXdDT2JYUXt6+9IPYkxOZtH/NRiLs780mb
fcnVsuyLOOKpQ72go7EvKby4fZ0Esacgdtz2MX8HcXdnPmWzL7laln0RRzx1qBd0NPYlhRe3r6sg
9jTEvrd9zMcg7u7Mp232JVfLsi/iiKcO9YKOxr6k8OL29RDEnoHYfdvHfBzi7s58xmZfcrUs+yKO
eOpQL+ho7EsKL25f70Ls9xB7c/uYT0Lc3Zm/t9mXXC3LvogjnjrUCzoa+5LCi9tXfwj+N8R6zYz5
XxB3d+Z/2+xLrpZlX8QRTx3qBR2NfUnhxe1rJsSeg9jUmTGfgri7M5+z2ZdcLcu+iCOeOtQLOhr7
ksKL29dhEHseYgfNjPk0xN2d+bzNvuRqWfZFHPHUoV7Q0diXFF7cvi6A2IsQO2dmzGch7u7MF232
JVfLsi/iiKcO9YKOxr6k8OL2dTvE/gCxm2fG/D3E3Z35B5t9ydWy7Is44qlDvaCjsS8pvLh9vQSx
P0LsuZkx/xvi7s78o82+5GpZ9kUc8dShXtDR2JcUXty+OkPwJYh1mBXzOYi7O/Mlm33J1bLsizji
qUO9oKOxLym8uH2Nh9gSiI2cFfMFiLs7c4nNvuRqWfZFHPHUoV7Q0diXFF7cvvaF2CsQ23NWzBch
7u7MV2z2JVfLsi/iiKcO9YKOxr6k8OL2dSrE/gSxE2fF/APE3Z35J5t9ydWy7Is44qlDvaCjsS8p
vLh9XQexVyF21ayYf4S4uzNftdmXXC3LvogjnjrUCzoa+5LCi9vXYxB7HWIPzYr5MsTdnfm6zb7k
aln2RRzx1KFe0NHYlxRe3L6WQewNiL07K+YSiLs78w2bfcnVsuyLOOKpQ72go7EvKby4fW0CwT9D
bMMdYr4CcXdn/tlmX3K1LPsijnjqUC/oaOxLCi9uXztD7C8Q236HmH+CuLsz/2KzL7laln0RRzx1
qBd0NPYlhRe3r6Mg9hbEDtsh5msQd3fmWzb7kqtl2RdxxFOHekFHY19SeHH7ugRib0Psgh1ivg5x
d2e+bbMvuVqWfRFHPHWoF3Q09iWFF7evuyH2DsRu3yHmGxB3d+Y7NvuSq2XZF3HEU4d6QUdjX1J4
cft6FWJ/hdgfd4j5Z4i7O/OvNvuSq2XZF3HEU4d6QUdjX1J4cfvqBsG/QWydHWP+BeLuzvybzb7k
aln2RRzx1KFe0NHYlxRe3L4mQew9iI3bMeZbEHd35ns2+5KrZdkXccRTh3pBR2NfUnhx+zoAYksh
tu+OMd+GuLszl9rsS66WZV/EEU8d6gUdjX1J4cXt60yIvQ+xU3eM+Q7E3Z35vs2+5GpZ9kUc8dSh
XtDR2JcUXty+FkPs7xC7bseYf4W4uzP/brMvuVqWfRFHPHWoF3Q09iWFF7evpyD2AcR+t2PMv0Hc
3Zkf2OxLrpZlX8QRTx3qBR2NfUnhxe1rNQgug9gHLfkuxFdryWU2+5KrZdkXccR/0JK8L9PalxRe
3L6GQqzh/mbxTjGXIt2d6e4G+5KrZdkXccRTh3pBR2NfUnhx+5oHsVXc3yzeKeb7SHdnrmKzL7la
ln0RRzx1qBd0NPYlhRe3r+Mgtqr7m8U7xfw70t2Zq9rsS66WZV/EEU8d6gUdjX1J4cXt63sQWw1i
F+8U8wOkuzNXs9mXXC3LvogjnjrUCzoa+5LCi9vXfRBbHWK/3CnmMqS7M1e32ZdcLcu+iCOeOtQL
Ohr7ksKL29ebEGsHsVd3irkc6e7Mdjb7kqtl2RdxxFOHekFHY19SeHH76gXB9hDrtnPMBgbh7sz2
NvuSq2XZF3HEU4d6QUdjX1J4cfuaCrE1IDZp55irQNzdmWvY7EuulmVfxBFPHeoFHY19SeHF7etg
iHWA2AE7x1wN4u7O7GCzL7laln0RRzx1qBd0NPYlhRe3r3Mg1hFiX9s55uoQd3dmR5t9ydWy7Is4
4qlDvaCjsS8pvLh93QyxThD78c4x20Hc3ZmdbPYlV8uyL+KIpw71go7GvqTw4vb1HMTWgthTO8ds
D3F3Z65lsy+5WpZ9EUc8dagXdDT2JYUXt68OEFwbYqvNjrkGxN2dubbNvuRqWfZFHPHUoV7Q0diX
FF7cvkZCbB2IDZ0dc02IuztzHZt9ydWy7Is44qlDvaCjsS8pvLh97QmxzhDbdXbMDhB3d2Znm33J
1bLsizjiqUO9oKOxLym8uH2dCLEuEDt2dsyOEHd3ZhebfcnVsuyLOOKpQ72go7EvKby4fV0Fsa4Q
+97smJ0g7u7Mrjb7kqtl2RdxxFOHekFHY19SeHH7eghi60Lsvtkx14K4uzPXtdmXXC3LvogjnjrU
Czoa+5LCi9vXuxBbD2Jvzo65NsTdnbmezb7kaln2RRzx1KFe0NHYlxRe3L42xEe6Q2yDOTE7Q9zd
md1t9iVXy7Iv4oinDvWCjsa+pPDi9rU9PtIDYtvOidkF4u7O7GGzL7laln0RRzx1qBd0NPYlhRe3
r8PwkZ4QO2hOzK4Qd3dmT5t9ydWy7Is44qlDvaCjsS8pvLh9XYCPrA+xc+bEXBfi7s5c32ZfcrUs
+yKOeOpQL+ho7EsKL25ft+MjG0Ds5jkx14O4uzM3sNmXXC3LvogjnjrUCzoa+5LCi9vXS/hIL4g9
NydmN4i7O7OXzb7kaln2RRzx1KFe0NHYlxRe3L7WgWBviK05N2Z3iLs7s7fNvuRqWfZFHPHUoV7Q
0diXFF7cvsZBrA/EtpgbswfE3Z3Zx2ZfcrUs+yKOeOpQL+ho7EsKL25f+0KsL8T2nBuzJ8TdndnX
Zl9ytSz7Io546lAv6GjsSwovbl+nQqwfxE6cG3N9iLs7s5/NvuRqWfZFHPHUoV7Q0diXFF7cvq6D
2IYQu2puzA0g7u7MDW32JVfLsi/iiKcO9YKOxr6k8OL29TuI9YfYb+fG7AVxd2f2t9mXXC3Lvogj
njrUCzoa+5LCi9vXBxAbALG/zY3ZG+Luzhxgsy+5WpZ9EUc8dagXdDT2JYUXt69NIDgQYhvuErMP
xN2dOdBmX3K1LPsijnjqUC/oaOxLCi9uXztDbCOIbb9LzL4Qd3fmRjb7kqtl2RdxxFOHekFHY19S
eHH7OgpiG0PssF1i9oO4uzM3ttmXXC3LvogjnjrUCzoa+5LCi9vXJRDbBGIX7BJzQ4i7O3MTm33J
1bLsizjiqUO9oKOxLym8uH39EmKDIPbzXWL2h7i7MwfZ7EuulmVfxBFPHeoFHY19SeHF7etViA2G
2B93iTkA4u7OHGyzL7laln0RRzx1qBd0NPYlhRe3r24QHAKxdXaNORDi7s4cYrMvuVqWfRFHPHWo
F3Q09iWFF7evSRDbFGLjdo25EcTdnbmpzb7kaln2RRzx1KFe0NHYlxRe3L4OgNhmENt315gbQ9zd
mZvZ7EuulmVfxBFPHeoFHY19SeHF7etrEBsKsa/sGnMTiLs7c6jNvuRqWfZFHPHUoV7Q0diXFF7c
vn4MsWEQ++GuMQdB3N2Zw2z2JVfLsi/iiKcO9YKOxr6k8OL29RTEhkPsd7vGHAxxd2cOt9mXXC3L
vogjnjrUCzoa+5LCi9vXahAcAbEPWnIIxFdryRE2+5KrZdkXccR/0JK8j9DalxRe3L6GQmxziG0y
L+amEHd35uY2+5KrZdkXccRTh3pBR2NfUnhx+5oHsS0gtvO8mJtB3N2ZW9jsS66WZV/EEU8d6gUd
jX1J4Z/cvj6khk/JKo0uyJ+tjv9cnoq/duIj3aY3f9r+iB6NPXebsedu+83ddtbs3ebN3HbujPjp
dL99FT9+W+Rqja3wx86NnQ769IFH7v/Fgw77fP95+3/+C83v1Avuy5c32kp/6Lflq7XF6f5Tu0rj
NO+3HL/hia7SgFpj7MQtG0fgOwzCt3+Ow7PTGm3ffVRjCyfb9m33KkxZxf1Z2xUN1sOlg28wY860
1s9Z8qFVVvihnVb8o+a2fqgfotuHP1v/o6cwo1PzKXxlrfKfwnad4lMYh8/4h59C2/Wf8Cl8qWvz
KeyzXtlPwf019ctdm0/B/cCRWP2YNv5qLc/h6VXcxX/sE3oa01Zd4YdyPqhHcfkmPuPrrdF8YKU+
KPe6/Cc+etlU/kVrfOOj/qI1/p/ydXmlffOzv9maZb8u7in8qX38i9botpclfQqjP8HXJOdTWGXN
5ruwbYfy34VV14zvwhj8penDT6Ht+k/4FHr6z/6nOpb/LqzfIb4LYxoTPvIpTPinfAojOjbfhc93
Kv9d2LxjfBfG4v8+/BTarv+ET2H+Ws2n8O21y38K+6wVn8K4xsSPeApt13/Cp3Ck/+z/xzrl/xXp
C2vHvyKNx2f8w0+h7fpP+BTOWKf5Lvy6c/nvwlfXie/CBPz3oQ8/hbbrP+FT+I7/7P93l/Lfhe92
ju/CxI/8IsfEf9IvctzUpfku/LVr+e/CT7rEd2HiR/6qbeI/6a/avrFu8yl8oVvZT8F9keOb6zaf
QvOLHCPbfsXw0V/kGPkJ/rqhkC9yXLte8wGd0b3sv2i5B/Xv67V+NWpk2y8qVvSgPrlfWhTyoO7q
1nyjvtOj/DfqF91a36hRbb/u+OgHNeoT/NVHIQ/qye7NB/STnuW/Uf/VvfWNGt32dfaPflCjP8Gv
thfyoP7co/lG/Wb98t+ov/RofaNGt/0ackUP6pP7lWQhD6q9f0AvblD+G7XG+q1v1Ji2X2Z+9IMa
8wn+YrOQB9V3g+Yb9X6v8t+ofhu0vlFj234l+tEPauwn+OvRQh7UGP+A1u1T/hs1tlfrGzWu7Rer
H/2gxn2Cv2Qt5EHt0Lv5Rg3pW/4btWPv1jdqXNuvZ1f0oD65X9UW8qD279N8QFv3K/+N+nSf1jdq
/Mf8Xf3xK9/f1T+mb/ON2mPD8t+o/9O39Y2a8DFfmZiw8n1l4ux+zQd0aP/y36iv92t9oyZ8zFcm
Jqx8X5m4csPmG3XKgPLfqO9v2PpGTfyYr0xMXPm+MnFb/+YD+tbA8t+on/WPb5T7/7dc0Vcm/MdW
rgf16IDmG/Wjjcp/o/5zQHyj3MNY0Vcm/MdWrge1ZGDzAd27cflv1CsDW9+okSv8yoT/2Mr1oBob
N9+oZzcp/41aZePWN2rUCr8y4T+2cj2onv4BvTOo/Ddq/U1a36jRK/zKhP/YyvWghg9qvlGdhpT/
Ro0Y1PpGjV7hVyb8x1auBzV9cPMBDdy0/DdqxuDWN2rMCr8y4T+2cj2o+UOab9TEzcp/o/YZ0vpG
jV3hVyb8x1auB3Xkps0HNHdo+W/UFzZtfaPGrvArE/5jK9eDOn2z5hu1cFj5b9QZm7W+UeNW+JUJ
/7GV60FdNrT5gE4YXv4b9Z2hrW/U+I/5ysT4le8rEzcNa75R548o/436ybDWN2r8x3xlYvzK95WJ
B4Y3H9A1m5f/Rj04vPWNmvAxX5mYsPJ9ZeKFEc036s4tyn+jXhzR+kZN/JivTExc+b4ysXTz5gN6
YmT5b9T7m8c3yn3tYUVfmfAfW7keVJeRzTfq9VHlv1FdR8Y3yj2MFX1lwn9s5XpQg/0Dajem/Ddq
yKjWN2rkCr8y4T+2cj2oKaObb1SfseW/UVuPbn2jRq3wKxP+YyvXg9p9TPMBjR5X/hu1x5jWN2rU
Cr8y4T+2cj2oQ8Y236hZ48t/ow4d2/pGjV7hVyb8x1auB3XyuOYDWjCh/DfqlHGtb9SYFX5lwn9s
5XpQF41vvlGLJpb/Rn1rfOsbNWaFX5nwH1u5HtT1E5oP6Kwty3+jbpjQ+kaNXeFXJvzHVq4Hdc/E
5ht1xVblv1H3Tmx9o8at8CsT/mMr14N6ZsvmA7p1Uvlv1LNbtr5R4z/mKxPjV76vTLy9VfONemRy
+W/UO1u1vlHjP+YrE+NXvq9MdPQP6OUp5b9RnSa3vlETPuYrExNWvq9MuH/BtHujlm9d/hvl/mXF
8Y2a+DFfmZi48n1lYoJ/QD22Lf+Nmrh16xs18WO+MjFx5fvKxOz/t7xzCW0iCAPwJDa2oKIi2kMV
+lAvrWCS3TxAadPtJk2TNEkbW5FqK6KCj1aE6kGFKlgPVgRFsAe9KIg968EHovUo9KJgDp56UDwo
giAKCv6z/2x20jJJAz6GSSCZzf47O49vvu3mhyYdaFRbp/xGpTscozS4XRBlJlhMLVAHIwjIMOQ3
6lDEMUqzvvZDBMqrXmbibCcatadLfqPOdfJGeYWZCRZTC9Q1AwGdMOU36rrBG+UTZiZYTC1Q97vQ
qEtR+Y2a6eKN8gszEyymFqjnJgK6HZPfqBcmb5QmzEywmFqg3kbRqIfd8huVj/JGacLMBIupBepT
DAG9istv1OcYb5QuzEywmFqgauJo1HyP/EZ54rxRAWFmgsXUAtXAAP1IyG/Uxh7eqIAwM8FiaoHy
JtCo1Sn5jfIleKOCJTITQfUyE91JBLS1V36j4kneqFCJzERIvczEcAqN2pmW36iRFG9UqERmIqRe
ZuJULwLKZeQ36nQvb1S4RGYirF5mYiqNRh3Jym/UlbRjlA4faUWZCRZTC9SdDAI63ye/UXczjlG6
9Vt7IlBe9TITj7No1M1++Y16kuWN8gozEyymFqhZGOevdkIOwKzsiMgLio71JdSojeCv3m63qriI
/WPHFRr0L6f4PfTyqo5TPBeQd4pp1z7A6aZhg+73Wv9mRie6xnrP23DGTfcWjvj7ThjLpGHZ1Ios
R9rkZ9nahix9ZVn6qpLlg0Zk+bVJfpbPmpClvyxLf1WyjG5Alhfr5WeZqUeWWlmWWlWyzK9AlvWr
5Gc5vxJZ6mVZ6lXJ8qgLWc645Wc57kaWgbIsA1XHcmwXIalmQrbsJeTW5qXUeAc15qBGYBD+erUs
tUYQjtShRo6rMWkCwqIadFRPoVhHdmciue5oPJkc7jezzjLCzwXT1us269VDwgOEfIMO5bnyBjQT
5kq6n5AJQkxzctJurMCNfGknJR8L4hOLtpw9ZMz+7tCKukZH3gDF2gUj52DyM+Wix9OBrMfj+3N9
6YRZPFeHreP2W69wd8t1x24+z5X2fqs7qJdTLn64it4Vda1oQRwH22MVNe6G68jC8zsPGPYmUlgg
3LArWvVDsBIfQVujQ4R8byHSXsEGWF9noY91sG3A6jpGxmFWR6G/dF7rrP1v4HkSmr1H75jgvBq0
8hqezXCyNR561KKa//UKZ9SIQx5xaLkwNCgOGbXC0J+70E6xBXuZcW0kF2DL+vxhae0uiPQxai/u
n/vo/kJVu/wNUEsHCKqvay8FagAA6A0JAFBLAwQUAAgICAA2q94+AAAAAAAAAAAAAAAAGwAAAE9i
amVjdFJlcGxhY2VtZW50cy9PYmplY3QgMu2dCZjdRZXFqzv7vnUWkibpbrInnfS+pJNOJ51OOiTp
RBMCyCjLIBAUENBPRdQIBBUBHZFNQIRhBMFRAXVQUFxg/FRUXDAuIKAyGHEYNhkG0cy9r97513sV
klcot6o+X7Vf+nTf916f33l1LhTCl2zv3bhp29oK1aj0R4WaoQo/Hj1KqVnblZozWKk9g5T+pmVA
qboxSn2Inl5Js71730dfjS942Ux6VvOAeUaNOoe+qqWvKnMGlbkpf/CzKrLX0PRS+nZw/sHL8i8f
oXbRV0NyVntVxXn5b/RHZYW6h2SSOmLLqm39a9dv3Hj01r7XHL26b936AX78xlys63KfW3Ofh6rx
i3QK/oCqfDooz83zdu7NfeCJFdlLnlqpDvhhPb5zn6/MRJ1Gadv+ZkR+J/jNnWC9E30Da/DiXYXv
XNHbeGL+BJ+lkPxrMf34VQMur7iF3otz6dfsDdSKDeZJt99+Ox9fhbqYvpmsmbZue+3mDX3F53Na
LvCO3Odm+jxUPTdPqWvWKbVkiNEG6sGXDzXKczwv3xKj+34MKvquCLEo4Sn0zM1/E0SlmqIKe1H8
QW/DwSoracHbUHA4+7xxpx74jduc81uf+zyJPdQdxHR1nvVqtzemGPjAb8zSlzWJJfiT3ZqJVSx4
oUkswXvaNFNPm2DwQpNYgl9Sr5lYxYIXmsQS/LHZmolVLHihSSzB26s1E6tY8EKTWIJfMEkzsYoF
LzSJJfiDIzUTq1jwQpNYgi+t1EysYsELTf7+4KXvirU0qSOvLtJPr3s13iqmb+oXvvLYJjF0pGyD
78kz7ZEMbpuk4AGD3zBUM7GKBbdNUvCAwY8ZpplYxYLbJil4wOAzhmsmVrHgtkkKHjD47jzTbsng
tkkKHjD4R0doJlax4LZJCh4w+KaRmolVLLhtkoIHDD5ilGZiFQtum6TgAYPfk2e6RzK4bZKCBwz+
3tGaiVUsuG2SggcM3jVGM7GKBbdNUvCAwZ/PMz0vGdw2ScEDBr9trGZiFQtum6TgAYOfMk4zsYoF
t01S8IDB54/XTKxiwW2TFDxg8N/lmX4nGdw2ScEDBr92gmZiFQtum6TgAYMfNVEzsYoFt01S8IDB
p0zSTKxiwW2TFDxg8B/lmX4kGdw2ScEDBr+oSjOxigW3TVLwgMH7J2smVrHgtkkKHjD44CmaiVUs
uG2SggcMfnee6W7J4LZJCh4w+FlTNROrWHDbJAUPGLxtmmZiFQtum6TgAYM/nWd6WjK4bZKCBwz+
7wdpJlax4LZJCh4w+EnTNROrWHDbJAUPGPyQGZqJVSy4bZKCBwz+cJ7pYcngtkkKHjD4x6s1E6tY
cNskBQ8YfPvBmolVLLhtkoIHDD5hpmZiFQtum6TgAYPfl2e6TzK4bZKCBwz+gVmaiVUsuG2SggcM
3lujmVjFgtsmKXjA4HvzTHslg9smKXjA4F+p1UysYsFtkxQ8YPC31WkmVrHgtkkKHjB44yGaiVUs
uG2SggcM/sc80x8lg9smKXjA4DfN1kysYsFtkxQ8YPB/nqOZWMWC2yYpeMDgM+dqJlax4LZJCh4w
+C/zTL+UDG6bpOABg186TzOxigW3TVLwgMFfM18zsYoFt01S8IDBRy/QTKxiwW2TFDxg8G/nmb4t
Gdw2ScEDBj93oWZiFQtum6TgAYN3L9JMrGLBbZMUPGDwF/NML0oGt01S8IDBv7hYM7GKBbdNUvCA
wd9Sr5lYxYLbJil4wOCLl2gmVrHgtkkKHjD443mmxyWD2yYpeMDg/7pUM7GKBbdNUvCAwd/QoJlY
xYLbJil4wOAHNWomVrHgtkkKHjD4A3mmBySD2yYpeMDgH2nSTKxiwW2TFDxg8I3NmolVLLhtkoIH
DD6sRTOxigW3TVLwgMG/mWf6pmRw2yQFDxj8Pa2aiVUsuG2SggcM3tmmmVjFgtsmKXjA4H/KM/1J
MrhtkoIHDP75ds3EKhbcNknBAwZ/U4dmYhULbpuk4AGDz+vUTKxiwW2TFDxg8N/kmX4jGdw2ScED
Bv/EMs3EKhbcNknBAwY/skszsYoFt01S8IDBq5ZrJlax4LZJCh4w+P15pvslg9smKXjA4B9aoZlY
xYLbJil4wODrujUTq1hw2yQFDxi8cqVmYhULbpuk4AGDfzXP9FXJ4LZJCh4w+Dt7NBOrWHDbJAUP
GLxllWZiFQtum6TgAYM/lWd6SjK4bZKCBwx+y2rNxCoW3DZJwQMGP6FXM7GKBbdNUvCAwevWaCZW
seC2SQoeMPhDeaaHJIPbJil4wOBX9mkmVrHgtkkKHjD4trWaiVUsuG2SggcMPm6dZmIVC26bpOAB
g38vz/Q9yeC2SQoeMPj5/ZqJVSy4bZKCBwy+er1mYhULbpuk4AGD/yXP9BfJ4LZJCh4w+B2HaiZW
seC2SQoeMPiSIZpFNLhtEkPwKwZrFrCJBLdNUvDAwZ/s1kysYsELTVLwwMF72jQTq1jwQpMUPHDw
S+o1E6tY8EKTFDxw8MdmayZWseCFJil44ODt1ZqJVSx4oUkKHjj4BZM0E6tY8EKTFDxw8AdHaiZW
seCFJil44OBLKzUTq1jwQpMUPGBw/H8DosFtk78/eJHReYXfnEjfjCetpckXxih176FK3bPGvOL8
PnIuegUb3mUM167fuLH4Xbs2B3tl7rO+1Q9Rt9JZLqMfe8sQoy30o3kOXZbD3alUX9/55+/77j21
cj9vwMs/vnOfr8xEncbv8itG4+QzSCZYyV+mYRqdns9BDtCwE3PPOzb3eRF9NjiwBw7wMpy/tXm5
R/Zp3rpXZP7qNfJV6teb6eWf6lXqhKFGP0lmPIfyPEC/3NG89At2sAcO8DIciX65mkfXr3kEOJHA
aocZHU1mPIdODNMvdzQv/YId7IEDvAxHol+u5tH167cE9p7VSj00zOg7CJTnUJ4H6Jc7mpd+wQ72
wAFehiPRL1fz6Pr1CQL8wyqlrhhu9HcEynMozwP0yx3NS79gB3vgAC/DkeiXq3l0/XodAR5GYNtG
GN1M4DyHHhamX+5oXvoFO9gDB3gZjkS/XM2j69dkAvxGj1LjRhq9k0B5DuV5gH65o3npF+xgDxzg
ZTgS/XI1j65f9xNYPYF9d6TReQTOc2h97hB3+u6XO5qXfsEO9sABXoYj0S9X8+j6dSEBXkkFOH+U
0UsIlOfQK3MF2em7X+5oXvoFO9gDB3gZjkS/XM2j69c6AhxCYKtGG91LynPokDD9ckfz0i/YwR44
wMtwJPrlah5dvyoJ8PRupV4q0DcxaIHyPEC/3NG89At2sH+pQDE//e/5Dxpzj+ynX67m0fXrawT2
0Aql7hhjdDeB8hzK8wD9ckfz0i/YwR44wMtwJPrlah5dv95JgOsJ7MyxRnsJnOfQ9WH65Y7mpV+w
gz1wgJfhSPTL1Ty6frUS4BeXK7V0nNHPESjPoTwP0C93NC/9gh3sgQO8DEeiX67m0fXrKQKrIbA/
jDM6ncB5Dq0J0y93NC/9gh3sgQO8DEeiX67m0fXrMwR4YZdSnxpv9HwC5TmU5wH65Y7mpV+wgz1w
gJfhSPTL1Ty6fp1IgC8sU+rYCUafIVCeQ3keoF/uaF76BTvYAwd4GY5Ev1zNo+tXHQEeT2DVE42+
gcB5Dj0+TL/c0bz0C3awBw7wMhyJfrmaR9evXxPY/Z1K/Xyi0e8SKM+hPA/QL3c0L/2CHeyBA7wM
R6JfrubR9etKAlxOYJdMMtpG4DyHLg/TL3c0L/2CHeyBA7wMR6JfrubR9eswArypQ6nNVUavJ1Ce
Q3keoF/uaF76BTvYAwd4GY5Ev1zNo+vXeAKsIrCRk42OJXCeQ6vC9MsdzUu/YAd74AAvw5Hol6t5
dP36HoHtbFfq3slGzyJQnkN5HqBf7mhe+gU72AMHeBmORL9czaPr1/sJ8Ik2pd43xehjBMpzKM8D
9MsdzUu/YAd74AAvw5Hol6t5dP1aTTyHE9jyqUZfQ+A8hx4epl/uaF76BTvYAwd4GY5Ev1zNo+vX
X4nnW61KvTDV6FcJlOdQngfolzual37BDvbAAV6GI9EvV/Po+vVlAlxKYLdPM7qAwHkOXRqmX+5o
XvoFO9gDB3gZjkS/XM2j69dbCfCqFqVOPcjopQTKcyjPA/TLHc1Lv2AHe+AAL8OR6JereXT9aiDA
YQS2cLrRCgLnOXRYmH65o3npF+xgDxzgZTgS/XI1j65fTxDYmc30DyPTjZ5CoDyH8jxAv9zRvPQL
drAHDvAyHIl+uZpH168bCfDhJqWum2H0FwTKcyjPA/TLHc1Lv2AHe+AAL8OR6JereXT9Oo4ANxDY
P1Ub7SNwnkM3hOmXO5qXfsEO9sABXoYj0S9X8+j6dTABfqlRqakHG/08gfIcyvMA/XJH89Iv2MEe
OMDLcCT65WoeXb9+QWB1BPaTg41WEzjPoXVh+uWO5qVfsIM9cICX4Uj0y9U8un59jAwvbqBfM41+
gEB5DuV5gH65o3npF+xgDxzgZTgS/XI1j65fWwjwxaVKHTrL6HMEynMozwP0yx3NS79gB3vgAC/D
keiXq3l0/RpFgCcQ2JAao8cQOM+hJ4Tplzual37BDvbAAV6GI9EvV/Po+vWfBPZjOtGv1xi9j0B5
DuV5gH65o3npF+xgDxzgZTgS/XI1j65f5xBgN4GdXWu0g8B5Du0O0y93NC/9gh3sgQO8DEeiX67m
0fVrBQHeTN+01xm9gUB5DuV5gH65o3npF+xgDxzgZTgS/XI1j65fLxDYFPrmmTqj4wmc59ApYfrl
jualX7CDPXCAl+FI9MvVPLp+fYEAz1ms1GcPMXo2PchzKM8D9MsdzUu/YAd74AAvw5Hol6t5dP06
jQD/m97AHbONPk6gPIfyPEC/3NG89At2sAcO8DIciX65mkfXr0UEeCSBzZljdCuB8xx6ZJh+uaN5
6RfsYA8c4GU4Ev1yNY+uX/9FYPcuVOqROUbvJlCeQ3keoF/uaF76BTvYAwd4GY5Ev1zNo+vX9QTY
SGBXzzW6iMB5Dm0M0y93NC/9gh3sgQO8DEeiX67m0fXr9QR4zQKlDp9n9HIC5TmU5wH65Y7mpV+w
gz1wgJfhSPTL1Ty6fk0jwBEENnG+0UEEznPoiDD9ckfz0i/YwR44wMtwJPrlah5dv35KYG8jyB/M
N3oagfIcyvMA/XJH89Iv2MEeOMDLcCT65WoeXb8+TFCPEtwHFxj9Fc14DuV5gH65o3npF+xgDxzg
ZTgS/XI1j65fGwhwE4H1LTS6jmB5Dt0Upl/uaF76BTvYAwd4GY5Ev1zNo+vXUAK8gy6H/CZCbyNQ
nkNzc//9ckfz0i/YwR44wMtwJPrlah5dv75BYLMJ7M5FRmcSOM+hs8P0yx3NS79gB3vgAC/DkeiX
q3l0/Xo3AX5kjlJvX2z0AgLlOZTnAfrljualX7CDPXCAl+FI9MvVPLp+dRDgS7OVaqo3+jyB8hzK
8wD9ckfz0i/YwR44wMtwJPrlah5dv54lsJP4X5LWGz2OwHkOPSlMv9zRvPQLdrAHDvAyHIl+uZpH
16/P0Rc/PUSpTy8x+gMC5TmU5wH65Y7mpV+wgz1wgJfhSPTL1Ty6fp1MgD0EdvxSo8sInOfQnjD9
ckfz0i/YwR44wMtwJPrlah5dv+YS4GfqlKppMPopAuU5lOcB+uWO5qVfsIM9cICX4Uj0y9U8un49
SmDTCOxXDUYnEjjPodPC9MsdzUu/YAd74AAvw5Hol6t5dP26mgDPrVXqskaj7yZQnkN5HqBf7mhe
+gU72AMHeBmORL9czaPr1xEE+D81Sr22yegeAuU5lOcB+uWO5qVfsIM9cICX4Uj0y9U8un5NIsCj
CGxMs9HDCJzn0KNyh7jTd7/c0bz0C3awBw7wMhyJfrmaR9evHxLYt2cp9Z1mo98gUJ5DeR6gX+5o
XvoFO9gDB3gZjkS/XM2j69cFBNhMYOe1GK0ncJ5Dm8P0yx3NS79gB3vgAC/DkeiXq3l0/eojwGtn
KrWy1eiVBMpzKM8D9MsdzUu/YAd74AAvw5Hol6t5dP3i34R4FIH9uUCHEHhFgY4K0y93NC/9gh3s
/1ygmI+S6pereXT9uovA3s6/yWKb0dMJlOdQngfolzual37BDvbAAV6GI9EvV/Po+vUOAvxttVJn
tBt9iCB4DuV5gH65o3npF+xgDxzgZTgS/XI1j65fzQQ4QGD1HUb7CYLn0IEw/XJH89Iv2MEeOMDL
cCT65WoeXb+eJLCv0Nn8vsPoFwiU51CeB+iXO5qXfsEO9sABXoYj0S9X8+j6dTMBziWwGzqN1hA4
z6Fzw/TLHc1Lv2AHe+AAL8OR6JereXT9eiMBfnS6UkcvM3ohgfIcyvMA/XJH89Iv2MEeOMDLcCT6
5WoeXb9qCfCvByk1o8voCwTKcyjPA/TLHc1Lv2AHe+AAL8OR6JereXT9epDATiawn3UZPZ7AeQ49
OUy/3NG89At2sAcO8DIciX65mkfXr8sJ8GfTlPqX5UbvJ1CeQ3keoF/uaF76BTvYAwd4GY5Ev1zN
o+vXVgJcTWCbVhhdTuA8h64O0y93NC/9gh3sgQO8DEeiX67m0fVrLAF+dqpSw7uN3kSgPIfyPEC/
3NG89At2sAcO8DIciX65mkfXr+8S2HQCu6fbaBWB8xw6PUy/3NG89At2sAcO8DIciX65mkfXr10E
uIuY3rvS6HsJlOdQngfolzual37BDvbAAV6GI9EvV/Po+tVDgE9PVqqrx+gTxMhzKM8D9MsdzUu/
YAd74AAvw5Hol6t5dP16icBeT2DP9xg9nBh5Dn19mH65o3npF+xgDxzgZTgS/XI1j65f/0GA36lS
6tZVRr9FoDyH8jxAv9zRvPQLdrAHDvAyHIl+uZpH168zCbCVwE5ZbXQpgfMc2hqmX+5oXvoFO9gD
B3gZjkS/XM2j69cSArxuklLze41eRaA8h/I8QL/c0bz0C3awBw7wMhyJfrmaR9evPQQ2hsB+22t0
GIHzHDomTL/c0bz0C3awBw7wMhyJfrmaR9evfyPAd06kx9YYPZNAeQ7leYB+uaN56RfsYA8c4GU4
Ev1yNY+uX8fQD35sglKv6zP6MIHyHMrzAP1yR/PSL9jBHjjAy3Ak+uVqHl2/qunHbCGwKWuNHkrg
PIduCdMvdzQv/YId7IEDvAxHol+u5tH1azf9mLvGK/WjtUa/RKA8h/I8QL/c0bz0C3awBw7wMhyJ
frmaR9evjxLgfAK7cJ3ROgLnOXR+mH65o3npF+xgDxzgZTgS/XI1j65fAwT4sXFK9fcbvZhAeQ7l
eYB+uaN56RfsYA8c4GU4Ev1yNY+uXyMIUBHYoPVGXyTlOVSF6Zc7mpd+wQ72wAFehiPRL1fz6Pp1
L4G9eaxSd683egKB8hzK8wD9ckfz0i/YwR44wMtwJPrlah5dv3YS4M/HKHXWoUZ/TKA8h/I8QL/c
0bz0C3awBw7wMhyJfrmaR9evLgJcQ2CtG4x2EzjPoWvC9MsdzUu/YAd74AAvw5Hol6v5q9evfdDo
LalQ/M8v0wYr9QD1ee1Qpao26B9bQzJFHbGt74htR29ZtX5g29b+VVv6zNvJHxfQ61eRDlJdiv9m
vnHHcW8849i37jjt1Jqtx556pn7SdGLfu1flQu/zsXdQTj6Qf2sr1PvzfHvpg060QhGaau1cpk6n
J7TR1wfR4Nf03vC8gZrVoFoVf8ff8zo8Sb8uIsvFlTzNnkGJ+N/EjMgn6tvcW/geWg9V7P+hyv0/
NGi/D23cv9eWwof499Kp2vc9f0VneTFNLqO/9901TJ9pzGf5YXr02vWFZ9le8izby+gsh+TPsHlE
/Hs5dJi9l50lz7KzjM7y7OF6L28eGf9evnt48V420v8OfJa5Z5TNWT47Qp9h7ej49/K5EcV72aia
S55lcxmd5Y5Rei8vHxP/Xp48yt7LUnefxrK6+zwyWp/h2HHx7+Wjo+29LHX3aSyru8/hY/Ve7hof
/14eMdbey1J3n8ayuvv8YJw+w5cmxL+XPxxXvJdNJe8+TWV19+mboPeS/1uw2Pdy7YTivWwqefdp
Kqu7z50T9RnuqYp/L++aaO9lqbtPU1ndfZqq9F4eMyX+vWyusvey1N2nqazuPp+erM9w99T49/Lm
yfZelrr7NJXV3admqt7LgYPi38vaqcV72Vzy7tNcVnefy6fpM7x3evx7ecW04r1sLnn3aS6ru8+Y
6Xovu6rj38ux0+29LHX3aS6ru895M/QZ3nZw/Hu5a4a9l6XuPs1ldff5c7XeywWz4t/Ll6rtvSx1
92kuq7vPGTP1GX6yJv69PHNm8V62lLz7tJTV3WfPLL2XU+vi38s/zCrey5aSd5+Wsrr7HF2rz/Ci
Q+Lfy2Nq7b0sdfdpKau7z8/q9F4OnhP/Xu6us/ey1N2npazuPptm6zN819z493Jgtr2Xpe4+LWV1
97lnjt7LZ+bFv5f3ziney9aSd5/Wsrr7dOXPcMeC+Pdy+bzivWwtefdpLau7z63z9V4+vDD+vbxt
vr2Xpe4+rWV195mfP8Pti+PfywUL7b0sdfdpLau7z7WL9F5+vz7+vfzkInsvS919Wsvq7jMlf4Zr
lsa/l1Pri/eyreTdp62s7j4XLdF7eWdD/Ht58ZLivWwrefdpK6u7z6D8GTY2xb+XgxvsvSx192kr
q7vPWY16L29qjn8v39Vo72Wpu09bWd19nm7SZzirNf69fKbJ3stSd5+2srr7nNSi9/Kytvj3ckdL
8V62l7z7tJfV3efhVn2GYzri38tHWov3sr3k3ae9rO4+h7XrvTy3M/693N5u72Wpu097Wd197uvQ
Z/jisvj38vsd9l6Wuvu0l9Xdp3eZ3svTl8e/l2uW2XtZ6u7TXlZ3n6906TP8/Yr49/LOruK97Ch5
9+koq7tP4wq9l0evjH8vm1YU72VHybtPR1ndfW7s1mf4QE/8e3lTt72Xpe4+HWV195nZo/dy4+r4
93JWj72Xpe4+HWV197l0lT7Db/XGv5eXrbL3stTdp6Os7j6je/VeLuuLfy/H9BbvZWfJu09nWd19
zl2jz/DWtfHv5Xlriveys+Tdp7Os7j7/16f3cl5//Hv5Yp+9l6XuPp1ldfd5yzp9hp9YH/9enr7O
3stSd5/Osrr7PN6v93Lyhvj38vf99l6Wuvt0ltXd52OE8Rf6e2UrvSHL++M9S856Kb1iWL/+HXUb
ci+pUPiNlBte2Zl5/e2wKN7mbv0WX7wy3reYf6vnI+h5R65kT6Wa6E1tyPkPyr/N3WT2YAVP8o/J
r4nvg7quVR/UI23xH9QtbfqgWg5wUC3/qAf13GJ9UM1L4j+ov9brg2o7wEG1/aMe1NrZ+qB2zYn/
oLbM0QfVcYCD6vhHPKg7iPCKGfqgdlfHe1D8V+Uv04+7vlpf6Rpzh8FHZa50OKp35a502TPK5krH
Z/nERH2WC6viP8s/TcJZNpU8S2/3jojOcsVIfZZnj4r/LNeNwlm2lDxLb1eTiM7yIxX6LH9YGf9Z
frwSZ9lW8iy93V4iOcuvk9mzdFl46zalVs13eUX9Jnr2XHomnetx81xf8Wt6xXn0iv+d+6r/YUMf
pB9/Ff3Y+QNGlxEYz6E8V2qn7z9syB2Nk4v/YUOwgz1wgJfhSPxhQ67mPv6woc+Q5xvI62tblbpm
nor2r2Db86wnEuNw+rqX2vVm9TZ6V08lXn5fh+fmP6VfZ5DtjfRrLf3cFnL5Cf2qox82fgg/a59X
Bv0rXO/g/T80ZP8PDd3vQ4fv/6HeYft96NX7C+2F+cJ+KH+uNeoc+qqWX55b68pskZoHUO6Z23me
vRT6/1BLBwhzGJDnpBwAABAqAgBQSwMEFAAICAgANqvePgAAAAAAAAAAAAAAAAoAAABzdHlsZXMu
eG1s3Vldb9s2FH3frxBUYNiA0ZTsJI212HlYMaxAGwztundaomSukiiQVOz01++SFGXJshSt3dBt
KZBA5LmXl+d+Sr27Pxa590iFZLzc+OEi8D1axjxhZbbxP/z2M7r177ff3PE0ZTGNEh7XBS0Vkuop
p9ID4VJGdnPj16KMOJFMRiUpqIxUHPGKlk4o6qIjc5RdMcrmihtwV1rRo5orrLE9WbKbf7IBd6UT
QQ5zhTUWOO2Kp3yu8FHmKOUo5kVFFDuz4piz8uPG3ytVRRgfDofFYbXgIsPher3GZrc1OG5xVS1y
g0piTHOqD5M4XITYYQuqyFz7NLZrUlkXOypmU0MUGXi1ElQCBK6r43Keoq5ML74es9nR9ZiN0Bzv
iZgdZwbcD5VVMj9UVklXtiBqP+LfW/wWNs2vt29OcSWKuWdpbI+qWLBq9jUtuivPOW9N1QI22Y25
yyC4wva5gz5Mwg+CKSo68HgSHpM8bhnnxSXSABdiQCD6qEPeoYW+9KjmayxoxYVqDUnnFztgZ9mm
6l4V+Xiq6l0HzUSSXISCOSsMaQtJgx4ZPbzo1bJp/tfYgHyvqcGdug82miMjJUgpdVg0qdcog3vJ
hbkv0nqNRqsFqsYSQ4VSmCfpUiTpAh78rWsZKYd2kZKYooTGudze2VRvlz37rEnc+G8Y1A1zsPce
zPA9yEcHLVj+tPG/JRWXP57h7KLv9VRrPMpoSQWDqJEHJmUPUTEVQ149EsEMKXjatFf0D/J7PW1W
BzPHpCepaPGcTXiMx2bddmJne0JTUudNf3aaGxuN71FMc+0du3VaQpUAvwrFoKs7VTErCGzkcKjc
+Cg8UVQRQTJBqv1QDFTC6bxCCZOKlHouCBbXrDwJ6yY8lDP3G4mClEc5KbOaZLBLS7MQ87pUAq71
4b1/rgJBnJLy3GcG4/Q4yKe922kUuo2fHoZqdWfI6XFacQvas3PV7dbrB+PaCz6b40jDPItbLzbP
XU71uBHJPUn4AUGgSKrQUTsiDG9D8MWl/afO/ixHazeSWnFZEe1lllBuoSSv9sTdvarLWNXGm+gA
2xD3TLPQ8sZKinaCEphhpILEUG5HF38YmVDBE1BfAbc9j7Ayobrg6nHUKNF2mLE1Jbmk3Rg3ASlP
jF+61XR01pIimN80XebwmOccBhwlamqi8ZkaBQmfzqgIgheknCwI7VmSfQJSwmWlZqdHThU0U/SR
itLQZI3vHKZ1uvBfXlXqS3PG6Gtj/qLGL00WO2k2A2cvWWwpeQj8M5DXPBWshBBSNAO5hGVMQYkz
FQ5f0OmCY3jAK2uPP15r8bjwOyqnZb02XtuXLtQ/dyxs2zix71ZMkRwqRjd1a0gdYRPHYiTPWTIC
ObBED6E63UcQTUacsuMUqwfKsj3MVzueJx1XPsOsJWf5Wew0xE5Q/wsl+h337+F+oovaKpmzrESS
10IXypQdnXIYKynRfEEglspVrunqC6yelG78GARhRB7trmcF46YpGJeC48v81TAafhalzh1zKBXc
vuIhKCLa/HUwNBKfTUfNo45fPdjGqD826eaCcvLEa9W709uqCP0LoKFR/WaVC6R2J5/s4XbnpaRZ
SznXRbnvMF2b9o0TgsVyff2S2apeEJHBXk5TvdNfFA2+v7rjSuk3oWARrG+vbGvH41Y15nwNS6FF
XzSzbxIe+GKGD5f/Mx/qvR0Xif7GAourqxtWeqZ8ey8C82MgFUnsdzzABC9DJ0jij5mAJpu4mv0i
DvS/lqQOAt4AMjoImcGF/xsx9e+l7fkYx6MVrNkoiGxVtHWtWdSapqaWblZcKHzW/O2d+XpaNX/l
nlKL3t7f39/h88VmpToj4SwAtC/d+yaT8Kr5NGiClpv29F/1XZoHbbYd07ahO6+zNjDBqeqRPmkC
HvD4HLXvmi9FE8wuB8y6aSDTbU0b9BfJ9r6zOMVU3oXY5+8HRPRO6i2ZPDo7PSHK3dF8duo27gcY
0FoQeiR5DYvLIAxRcINWkBzBDV4FWK80Vmjg9gfPGQzWL1fR8jq6um6NvhQ+ffu+Wkx5uAs07yrb
9boLtGv/UOzhy9mOL///zPZPUEsHCJ3z0ZISBgAA3xkAAFBLAwQUAAgICAA2q94+AAAAAAAAAAAA
AAAAFAAAAE9iamVjdCAxL2NvbnRlbnQueG1szZ3fcxs5csff81e4dM+C0A2gAbh2fVVJVe7lLi9J
qvJKS7StHCUqlLy289enMfzVjZVs7e51dURpRuR3SM5nBgN8gQEaP/35693mzS/r3ePt9v7nCwjx
4s36/np7c3v/8eeL//yPf71sF39+908/bT98uL1ev73ZXn++W98/XV5v7594/Ybfff/4dq/+fPF5
d/92u3q8fXx7v7pbP759un67fVjfH9/1Vm79dvmu/SuPT982r377srF899P669Nr3zy2Ve9dvX/9
Ny8by3ff7FZfXvvmsS0fVPn2D9vXvvnr4+byw5aP+t3D6ul22ouvm9v7v/988enp6eHt1dWXL1/C
lxS2u49X0Hu/WtTTDl+ftnv4vNssW91cX6036/Flj1cQ4Oq47d36afXa/Rvbyl26/3z3fr179aFZ
Pa1+dVYff/n46hTxy8cXDs31p9Xu1Wlj2Vif3nTz+tObbuR771ZPn144J+3qbywui7/99ZwWdnev
/a6xrTpU17vbh1dj7reW799ut6ddHW/YX6DL7mKM+Wr/XGz95bubf9ndPq13YvPr725+vdpcn474
9u65g8bbwRVvcbn+ZSTTU8IfB+LxhTfg1V4+bfx48+JH/9ff/vrv15/Wd6vzxrc/3vjy9v7xaXV/
PjK7cRJeJC1Xu/XDdvd0OjAfXp9h8tnC0759errbvHy5D/W46cfdzc2zm/LupCu+9PnCu/zldv3l
Tyo//H566FfLRqeEe7veHK+S07YHnPXXh/XudpCsNiMhXN498kHjxLF9eCverfPE3d3X133cSBDb
mw/zJ54ujsN7ReHGx3A5JG+fdqv7x7ElZxlDOsDycX8My/m4HF++EO8/hXNGTlCPm6cr/kbc3XwI
/OTi3bFcXH1+2o7Pur5c8rHHdz/tc8BDRrh/9c0+jxssP1/8W7yYNnpzeHZ3e78gfeT33dx+vH3i
FA4XV+9+unrmM9/9tP/QX3/B9Se4OLzwYXV3u/k2XuIM7uL4lo+71cMn3uMHPnbr3dPt+vHNKKP4
o3bbv/MH3G/v18vXim/43tfh979ueSa/bJ/djiN3+bB9vN2fiKfd5+VLf8MuvuHs//AKlwKbLRc7
f3qfxuNiv/WH283muO3phdOmaxqP83cOhyC/8MOWkxQ7nsfb/+VvbA9PJ8zjq5ecYFb3L2ijXNqs
v+7V33A00+86mo/f7t5vN5dP3x74Q06J8uKg3t5fbz7frC8/3d7crO8vr9ebDaetD6vN4/riO6dD
SvujIF9+2q1XT5fru4enb8dP3KxXv6wvP64ejts8PrAJWV9udzfDFKTjy7vbj5+eLlf3Hzfrm8vV
1/Xj+fy/9jDl5w/T4cUlf1v+vRRX3XeP4M0t7+zq2+Vm9X690aSb7ccVF22f7m6vp6O2W49MZs2X
6259vT9yB33xmwv8ez5Of5+P9lfOac5HO3435b+czP+/pd3ymzOe77K99mvpH50W1NU03nRzuX/p
Qm+w/7zH//m82q0n6cvtzfCBMWC5vpu0T+uR/k/i78/1Tl8SWxvf8szRZKNVGj2X/x2UV6Yh+k4a
elY7pSH6bWmo/rHC63G7ub35xxUNr93r9g9M+c/s0PXy8+sdunrRhhyE99ubb6cnyz69+2mfFpfl
sheHVIQ5JKjHZHRMo9AClDxeXeqTbz/t1mxfQzgm6evN6vHxgLvkd6e0Lq64YUmO37thf3N/80Y+
ETkhP9t/PycchNBaOe4QH9MWapdXk/oGHEdnLzxstly6cLb77IZ8gPf1+VFoXbIf/MgJ+eZmtx4c
//5pvX6C8M/49vhfifHN4f+/nF79C7963I19/rL9vLvmAnb1uC9AxiHZbj7f3Z9oYsiNBEwMidLx
+fEUxNAh/+oU1BBpcB/5RulxKrXYFe9d7tfjDu1BH9gwr3bfLr8+f7jy6dOuV2w5t7uRJn/jcVnS
43mXvr97357fvW8/2r2Pu9ubZzcpUxK8W/33dvfCLj2udyJ3Vx9Dx4/5ZbX5vH68/A798+f/11fA
8VuXlPGwZU9/sgsP7JnWXGvKvYtd3e/e8W1fVpvNs7taz0n8w2a73T27UROfe7oQ3v20P7XL8nCa
92/YbLkafrmv2KmtuJRasWu73CfjR63tXxxf9Nve8aqPeXbTw2fvtl8mgV+ZPpRPIL8y/NfDrz56
r02vnCqMIwUcSn3Omm/vP14cP+jdvyw79eYv/Hn7F5774Ktf7dhzx+dZhtdQvWI/977z9ualK/Yd
xu8SvOYrOe2tni6Uwgm6jU9/aS+mK+fd2Po3Hsl/xNF5h2BEXyqEjI1Xy/L8jbPiQo1G1FyCBUrH
h6SeFBfqZEQNNXExnWJOuTYkQT0pLtTZiDqK6/vdeOZCV2zoUsuCbjxzoft+ivn9dNQxpFh5tSwF
6aS4UH8/d/j91KWUUE4/4jqdFRfqZkTNpUw4FjaQk6CeFBfq76ew30+dgOtTOTespRGCoJ4UD+pk
5LkSyjx5PHOhM/JUKXYKtWboqeSWZK41KS7URp4qqXI2OZWzycg7YY89xH58gMwZtOJCbeSdsGEK
BXsGqJSrpJ4UF2ojT4U1Y2jnAkeW+FpxoTbyWkhEgU4/Vbo7rbhQG3ktLD0H/tv/kjTsSnBhNnJa
WJBCglJbzzVRk9BacaE2clqYCULhGhLBspQ1T614UP+g3vYHqGWpPJ650Bk5LUwZQiZeLosi21C0
4kJt5LSQC59wKoNQ+MtZcaG2cmCYkOu+DVIvKTeZa02KC7WVA4MGocHpIVuEteJCbeXAIKWQzj+S
Wisu1FYOLDYKvVBBqKmDpJ4UF2orBxbzuHoTdMitJ1kuT4oLtZUHU7VldKotZyOvBb30AC1RTZm6
bAGZFQ/qYuS1oGOUpD+4Y2lFZ+S1oDUM6VzlF6ST4kJt5LWA6wMBTpWDIqm14kJt5LXYT0BIGGvJ
Fag3Sa0VF2ojrwW11lBPP5J6UlyojbwW1FxCLseHTOGT4kJt5LWA60fh3LkBJLVWXKiNvBZQYzaI
BWm08shyeVJcqI28FlApASpmSLUneaq14MJs5cAIawBolHMklZVpwYOZrPwXSU89nrnQWfmvUhOX
QTF3aDGhPKWT4kJt5b9KbiG1Gjst900ltVZcqK38V8EUWuy8WpaSWisu1Fb+q6jrtzhdv1Y+K1cK
h95KvJTX76S4UFv5LHaOoZxNpKTWigu1lc/KCUNLpWFsvajrd1JcqK18Ftf/w2gAiKU1TOpca8WF
2spppc414mPP2ShLqEnxoK5WXiu91G98VlyorTxYKsjmGalSLJSypNaKC7WVB0sv3Y2ZFRdqKw/G
9eAQuVrMNcMSSV3XWnGhtvJg2Gvgv/2vPNVacGG2cmbYpO/kZy50Vg4MKYVzlx6ZZ02KC7WVA8NM
oZbEq7GUXmRSXKitHBhXH8JpTASqy1crLtRWDgw5P4bWW00ZapVue1I8qH+Qm/wBalVbdupD36yc
FrRx5srhV5a+k+JCbeW0oPJVmni5X0hqrbhQWzktKBQIib1UxCzb57Xgwmzls+ClPuaz4kJt5bTY
RYZDL4HRFCCpteJCbeXAAChg7VxVqFSzOtdacaG2cmAQUzgNX9N5mVZcqK0cWOwx9PNwIEE9KS7U
Vg4s1haoALWSepIjy2fFg7pbObD40oiZWXGhtnJmseTQc+HVWKpzrRUXaitnFjPnWUs7bqOu2jsn
xYXaypnFhAFLi5T5EpY9kmfFhdrKm0UELpWp8NnMWY6PmhUXaitvFiGGQ/ef0SIkqbXiQm3lzVQ/
bHDqh92NPFhvPVCPFYE4S5a9V2fFhdrIg/XaQ+mnTuZZHmetuFAbebBO44weH1VeOkrwYIYfXFZ/
ADpKUJeLl7NHI7gSQ+bF4Q9kGaAVH2wjq9UzBDh3FiBZ4GvFB9vIa3HtKMTUWhonNMuoP7Pig21k
tjqmEJ+9ZTErPthGbqvrTuayjJoUH2wju9Vj4XpxTLWm0rIMozIrPthGPqz1GqATElBDlLGtZsUH
28iItdZCO/+IxrBZ8cE2cmKtjVs1qWfOrzOgwtaKCzYYebFWMVQ8PWQinxQfbCOX1igH/jv+ijrW
rPhgG7k0LpsCPNsBcFZ8sI1cGhdNoZRaEAFTk3XLWfHBNnJpLWM4DQ5rXSbySfHBNnJpbLnDqWWX
ZD/uWfHBNnJpDWtYbs1R66S6r8+KD7aVS8MYCiCvliVIbK34YFu5NEiccXElE2PijEs0d8+KD7aV
S4vEpgQ5IcdaSLoVLbhAWwW7l7Hu3zmFsgerWPa1p5AaF0idS+FIogVlVnywjZxYbTSmm8FcegSS
wVZmxQfbyInVNiam4eV+obC14oNt5MRqzeEQPniMfRW+e1Z8sI2cWJW3YavPnVewCnNfCUKPmVfL
EiWpVnywjfxWLYXhSoERVCaiqEvNig+2kd+qJUpSpzLYyFXVzJlQo5iJTx7JKAWz4oJtFdC+phaO
4WN5KQujSfHBtnJcYzAz5ow1t9hBOq5J8cG2clxcC0bRoCmHYGrFB9vKcY15DWPmyxdrgy7P9qT4
YFs5LqCQoXaIOVcuieUwY634YFs5LoAA4kdia8UH28qLxRKWGAX7cIby2p4UH2wrLxZjqNg78j9j
LbG14oNt5MWol3CKqhvl0IxZ8cE2cmnUY6DMRTNGyEnemJ0VF2yrYPikoqM3UYDNig+2kUujFkMr
ECk17LVVia0VH2wjl0aVwojFkVrEBjKAw6z4YBu5NKojQGvj1bJEia0VH2wjl6bnqJGDcWbFB9vI
pRFhONyMG4Es5dmeFB9sI5dGpYaSMmb+h9fi7s2s+GAbuTQqic8pIFesW6+yBjYrPthWLi23cO5H
I5v6Z8UH28ql5RywlxTZi3GKlol8UlywrcLok5zRlnwmtAWrKPqU2GgSlNZzTL3KCsek+GBbebEE
4RxoSHatmBUfbCsvhjXkEnvjM4ogO8nNig+2lRfDFFJlA5KWpUzkk+KDbeXFoIfT7VcVdHtWfLCt
vBiUcJ48T94EmBUfbCsvBmOCudypca7ZSGFrxQfbyovFOqbF7IlqbE22hs+KD7aVF4sjxPoyNCUW
0thaccG2CqmvIuo7BdQHq4j6pY/gOwk7dCxZXsGz4oNt5MVKR75O8fibJLZWfLCNvFhpLfRSYynI
9XNpSmbFB9vIi5XGZTDxcr+oElsrPthGXqzoyTFRYmvFB9vIi3FNKiRIWFIEjFlY0FnxwTbyYqXm
0ErEznVngK6wteKDbeTFio60LhP5pPhgG3mxQi3EXLkKOXqbyMhys+KCbRVyv1AOxwkURhcEia0V
H2wrl0aclGMqsdUKyq1owQfayqOVFo7TWXNilud6UnywrTzaCCDHhVTPtVeS8UFnxQfbyqMVCKdZ
6KKcOGVWfLCtPFpuoTw7j/2s+GBbeTQ1q2aW1c5J8cG28mgZA2dYrUFaxhRKbK34YFt5tNQDu+6W
kTqRSuST4oNt5dEScXUDRyQXwih7j86KC7ZVUP6i58ORjQyT4oNt5dFSDKlRZTuWIHd5bU+KD7aV
Sxs38SryallKuzIpPthWLg1zOIwoTTnLaM+z4oNt5dKQvdi+N/CY0U3WRCbFB9vKpUFj592Ov7IA
mxQfbCuXBhQAe0y4RDmWnnxSfLCtXBpwMUWtN65icu1SnW2t+GBbuTSI4dgJGpNqW5kUH2wrlxZr
6Il4tSxlATYpLthWgftLLOEczlrdE5kUH2wrlxYxEFd0IsRcQTUqTYoPtpVLk/ewnSaV/lH75O+G
y52NZhwza2TMrYpzOis+2EZeLPccWuaqZGyYq+z2Pis+2EZejCtQIYmYuBJbKz7YRl4stx5qPwVu
F/n1rPhgG3kxvmr5nHbKmWqMMv73rPhgG3mx3MYVfHoUia0VH2wjL5YbhgwYKaYIyovNigf2j6aX
/APYUZK6lM5oFcE/18q+ikvBiL3pczopPthGjitXdtG8OP5JbK34YFt5sYqhRxrDN3JscpztrPhg
W3mxGsO57wGoRK4VH2wrL0YtYEdqh4XAnhQfbCsvRhQi51tQ+fKtss17VnywrbwYpVCpAZSeOd9U
Z1srPthWXowglPMQDlnhmBQfbCsvVnpIaUTAKDnKseST4AJtFb8/vxjRfVZ8sK082uhZVCmNG5S1
yOmGZsUH28qjFQyNU3JrVDtm6dEmxQfbyqOVGGrMiIVNCcgZLWfFB9vKo40OVb1Bio2NGMp8fFJ8
sK08WqawRJvcT7Ets7RJ8cG28mg5h3z+kdf2pPhgW3m00Y+sU4KUuKpVFLZWfLCtPFqOIREnYq5Q
V65kSmyt+GBbebTUQioExP/wWmZpk+KCbRXBP4/uc3R+SGyt+GBbubSUQ2odckQ2oXIYwKz4YFu5
tISB/46/ClsrPthWLi2N+U0TVq5mIUV552NSfLCtXBq+1CF+VnywrVwavhS6bVZ8sK1cGuZQa6sE
kUBNuD4rPthWLg1HW3jNrXWuYDZpVybFB9vKpSGE8zRp+mxrxQfbyqVBD1yZ7zXV3kGGSp8VF2yr
+QAy1HDuECy7SM+KD7aVS4MSSsu8WpYKWys+2FYuDVLYD9DLuWFV2FrxwbZyaYAhQgWqqYGKZTcr
PthWLg1iwEK8WpaSWgk+0FYeLbIBPQeKlhWRSfHBtvJoYy7A2omdGFCXkUdnxQfbyqPFEviMYgcu
oFGl8UnxwbbyaHEMcQDOq0uqDWQD4qT4YFt5tAiBc2ri9bKS2FpxwbaaDSDLfuHZp1/4j66g3w2X
egspR8ytE1IRlmRWfLCNnFjqFDg3Rshce46yvWxWfLCNnFjqI1eGimNwVpRt4bPig23kxFJPYQwz
5dpriU1RK8EH2siJceUx7Hu7U09NRpmdFR9sIyeWOpvrzMvDQmJrxQfbyIml1gIRZ16VYNy9FNiT
4oNt5MQ4+YYRKbnm/Yy2ElsrPthGTiy1EgqXUTQiJpCchWtWXLCtIv4n2QqcnBp+rSL+pzF6o47g
gwWhgyylJsUH28qJtRiYi1fLEiS2VnywrZxY7ew8aHT1H+FtZH49KT7YVk6s1nFOaynLtM0ykU+K
D7aVF6s0ruDTGAeJrRUfbCsvJof0JKdRPFZx/VNNXNLGVriWCCAHOMyKD7aV46rIvipFTHy1NFBJ
WSs+2FaOq8ZAhXi1LGUxNSku2FZx/RP1gGK8uMCeFB9sKy9GNTSg2Nly5Shn1JoVH2wrL0ZjnmpE
SDXHXGSWNik+2FZejMoYkMYGO0dqcnLfWfHBtvJilILoBqqwteKDbeXFCAMfUyix55Tk3Iiz4oNt
5cWIa1MRuWim1GNX17ZWfLCtXFrpoZRYCGqvtcicfFJ8sK1cWmkBY+ulIFedVa16UnywrVxaodBq
TZgyxih7msyKC7ZVxP9UCifl04/M0ibFB9vKpZU8GvRLPiwktlZ8sK1cWsEwwiRU2jccSGyt+GBb
ubQCYwpfXi1L2XQ0KT7YVi6tjLFZuRP/k2R40UnwgbbyaLmH2M8PQT0pPthWHi1z5ZLr8o16oSJH
386KD7aVRxtjbOuYzBgAuhxrPSs+2FYeLZfAfqRmYEMCqlF4UnywrTxazoHTcUSuZ5UkO0HPigu2
VcR/9p2hnYdmqUSuFR9sK4+WuWxONUIc/Q9ksJBZ8cG28mg5hsw16sS5Vm762taKD7aVR0s9ICIx
WY69yXJ7UnywrTxaaiHyJUQ9wrjvI7G14oNt5dIS16kzwmFQmqyITIoPtpVLSyUQZV4tS1mATYoP
tpVLSznknscU1px1qebiSfHBtnJpej4eWW5Pig+2lUtLGKD2nmLtpamOdZPigm0V8T8lCIcxeKWR
DH0/Kz7YVi4Ne2idC6mKfP12WYBNig+2lUvDFipVQP5nrCW2VnywrVwa1kCZ61gjrnBJ0qVNig+2
lUtDCiVFAkS+jNUdsEnxwbZyaVi4wnGeN1Fia8UH28qlYQ7nsDfq5sCk+GBbuTRMgTOt00Nia8UH
28qlIZuSXKlAjJxdq5xcKz7YVi5thASpccw3Bamq7rST4oGdrOYCSBglqcuIzGQ1F0Aas+UlarXG
Slmd00nxwbbyYlBDa1ynGCMdugyKPys+2FZeDCg0Xhz+FLZWfLCtvBiUsJ86jUY7kcyvJ8UH28qL
QQ61jAbQRFGPtp4UH2wrLzai22DJna9f4gtZYmvFB9vKiwGGGlNpo2rRVGv4pPhgW3kxgECNsyxY
lqoA04oPtpUXGxOA1k4MV6KajmpWXLCtZgPg4xnOo2xVY8Kk+GBbubTYOCkDVYxYqqpnTYoPtpVL
izWc5g5McqKqWfHBtnJpI4hT7Mi1yFoqqESuFR9sK5cW2YslHLdsK6mowrPig23l0iJ7MYIe87Is
yg8rxQfbyqXFEYOwZq5tlKw6qGjBB9rKo8UxiUnnopkvXtTUWvHBtvJoEcJx3hZsiloJPtBWDi3G
0HPMqbUKuSpqrbhgW80EkGT8sh+1yZnBGfkw7D1wfTljLDRCQqtGV6X4YBv5MOwtQCLKpRNWWTLP
ig+2kQ/DXkcsaELEDlF2kJ4VH2wjH4adAv+dftXtM6X4YBv5MOwlnGcOlGE2Z8UH28iHYc8h11aw
cr6VSCVyrfhgGzkx7CmUWjtXnQuofuGz4oNt5MSwYzjNAMsL1e1FKT7YRl6Mc+kR8Jxr7JG6GuEz
Ky7YVvH+sY+x1qcHSGyt+GCbubQYIleoAHKLiZLGlooPtpVLaz1AoRE+N5KKRTkrPthWLq21wH+n
X9VLVyk+2FYurdUxsgNyrzEWGZRxVnywrVxao1CoAmWobL1VIteKD7aVS2slnCfaatKuTIoPtpVL
azk0rl1SaxllE8ok+EBbeTRmi8iV6jYipJSJWio+2FYeraUArYw2/li7HHs+Ky7YVvH+sWFYYjHu
AylIjzYpPthWHq1BKJi5pkFjfEOV2FrxwTbzaGM6b6glp1amRK4VH2wrj1Z7aC2NaZcwg+wMPys+
2FYerY5G4V6p5gyNQGNLxQfbyqNV9t2n6BHq2p4UH2wrj1bZd4+xmDX1pgL7zYoPtpVHqxSoVK5Y
Fmg1yQbESfHBtnJptbADxUiNM+uiE7lWfLCtXBrDRa5kpRExpKl7IpPigm01FwDWHFCEzZDYWvHB
tnJpI7Z0HDWsZamwteKDbeXSKoaKp4esgU2KD7aZS4PQS2ZPghi7ajedFB9sM5cGXL2knCMtUydq
bKn4YJu5tBEaBgtfRQD6VtCk+GBbuTTqgQg7AluDpG7zTooPtpVLozZmMS+UIhYZaWASfKCtPBqz
QaGMubQxIFNTS8UH28qjEVc34hjAVssIviuxteKCbTV7ABJXN84/0pFPig+2lUejEjqOqNIt9ZRU
hqYVH2wrj8ZwwFQ1AtcxM2lsqfhgW3k0yiETttg7sPdWZ1srPthWHo1SGH2QKC1LWRGZFB9sK4/G
cBES9cbFM6SqsaXig23m0TAsc6RCp4oyROus+GCbeTQIpcXUIFdIqthWgg+0mUeL4dQenFQfpUnx
wTbzaDEA51fxsNDYUnHBtpo7AEsPufTDr6TWgg+0lUMrI7xXhFjHQE05RHlWfLCtHBrDRcoFCLDH
NGFLxQfbyqGVGviYUslQSlbd5yfFB9vKoRWuY7WcO/Sqb3VqwQfayp8VdXqdzqiVCyslHOJGc+VV
ek8t+EBbebCSA2UYM1OBDuQ2Kz7YVi6spNCp8WpZKmyt+GBbuTCGQz6fJcWUuR6lsaXigm01OwAW
DCNS9jHykcTWig+2mQ+DkZRLTJ0NblTYWvHBNvNhEFJsPcaCSRdYSvCBNnNhY55crGy1OLdW0Erw
gTbzYFGB+sBZea3cQypcbej7pSCdFB9sKxeWuYoYM1FCGPPLSWyt+GBb+bA8qogYuaYU+UtIY0vF
B9vKh+UxdgepxQyoJv6eFR9sKx+WKXDlkFNy6lHfzJkUF2yr+P8D7jiofCw1tlR8sK18WC6jhz92
olqgy4ahSfHBtvJhOYfWay8dWiLVmW5SfLCtnBjDsd1qsQPFFIvGlooPtpUXy4l9Zm6IiV0JKmyt
+GCbubQUYjrH79LYUvHBNnNpyKXz+SGxteKDbebSIDRIFIcdA3WjdlJ8sM1cGgSMI8+CEqtqLZsU
H2wzl8ZV5wGWaulJDTmfFA/sbBX/n+EUqQ+clRdLPfCH507IZ07Ocz0rPthWXiy10GLpdQDq9u5J
8cG28mIMh5D4Ch2Xqhq9Myk+2FZeLNVAvDj8yYbfSfHBtvJiDMcn8/SrsaXig23lxRKFXBIbD76I
VWjZWfHBtvJiqYTGV24qy1I2JkyKD7aVF2O4U4yEKCccmxUfbCsvlnKo0BoSZx9N3cKbFBdsq/j/
Aw4ypbxfFI0tFR9sM5c2YhIS5dFkUNUQtUnxwTZzaSnEyHl1q6k21et7UnywzVwahpxLQb5+AdRg
40nxwTZzaSMYOGdatVeI+trWig+2mUuDkLCkCBhzg66xpeKDbebSYqjnKTzkna5J8cE2c2mR61ml
jWDZTTWYacEH2sqj4RhZW2KFyrUNdQ97UnywrTwawwFQzwVi1e1lk+KCbTUDAGIL53gYqLC14oNt
5dEYLuLoQ1aIqx1FY0vFB9vKo2ENuTUoQFy9VBWRSfHBtvJoSGE0lZVMsXedyLXig23l0RiOnVDs
OaVOciKyWfHBtvJoWNiA4jEao2xJmxQfbCuPxnCHICCVouo6Oik+2FYeDXNoo0dd4epG6+psa8UH
28yl5YC9H2exAY0tFR9sM5eWQq3Eq2Upa9uT4oJtNjcAw2EpXNXoJWUVNn1SfLDNXBqG0WQEPTKj
ugM2KT7YZi4NA4ofjS0VH2wzlwZjvswYqeMIHy2xteKDbebSIMAYvETUkGseGlsqPthmLi2GY9BN
zKrddFJ8sM1cWgzQOKceN7CLGmc9KT7YVi4N+jLPdUqNKipPPik+2FYujeGgttj4H16DxpaKD7aV
S4PG53S0JmT23Wr0z6S4YJvNDsBw0M4PjS0VH2wrlwaV8+s6JtLkeofK0ibFB9vKpTEc9FxqhFRq
rhpbKj7YVi4NxqzmXK9tfUxros62VnywrVwaw/Hf6VdjS8UH28qlwZjMPY74fLWRPtta8cG2cmkM
h2OwJtWq7vup132QzRzamL++jPiqFSArq6IVH2wzh/ZirHjwjxWfzWYGgBTYfGZKWEBFX9WCC7TZ
vADMlsSPppaKD7aZP8PQKCbgshn0dHST4oNt5s8wJLYiLZYUu5oXYFJ8sM38GYSObL96Y/uphjJO
ig+2mT+D0Vu8sgHsKalhypPig23mz2LorUeIy1KV2lrxwTbzZzEUPMU+yhpbKj7YZh4thkiFODmx
K1E39CfFB9vKo8UeKI7oya33oqY6mRQfbCuPFsdE0DEhVS6kVLk9KS7YZjMDxBZEIGF1trXig23l
0hhuBNMonGPr6CJa8IG28mixhtqRV8tStphOig+2lUdjuJRTBuSiqvcJWyo+2FYeLVI439dTIzsn
xQfbyqPFMZDxGD4ZSWNLxQfbyqMxXIyttEQjDGvV2FLxwbbyaLGMme35tDaMoPqaTooPtplHKwGQ
wcp+qbGl4oNt5tEyW5I4ZiBjXxKVWdGKC7bZzAAMh6Wc4kprbKn4YJt5tBQaAtezYoldzeczKT7Y
Zi4tjfaylpit5TphS8UH28ylYehlwDWips2pVnywzVwaBr5+R1MCdUJNLQQfaDOPhgHiqeGoaGqp
+GCbeTQInICpjlilXSdxrfhgm3k04EJqTHBBSXcr1oIPtJlDi4EdN1uwuKwktVZ8sM0cWgwZaklU
Us6qm86kuGCbzRoQoyL1gTPyYdB7KC3zipciIevXfZCNPNhAg8rVxsxHNcle4rPig23kwaC3UEut
lHotIB33rPhgG3mwAZe4GhNrTi3K6XxmxQfbyIVBr6GPiRNzwq7G8s2KD7aRCxtwZdQlOOfi4gg0
tlR8sI1c2IAb7dyFqKYCE7ZUfLCNfBh0CmM2VACuzWRFrQQfaCMXNtgS9Agtx6wqGrPigm01ZwD0
EvZh4lOmLmeJnRUfbDN/VkKBntgNUWsztlR8sM082mjp5uVhobGl4oNt5tEy51tcZaakehfq132Q
zfxZDolzrTb6EBI1DS0VH2wzf5bYhZ0eWWJrxQfbzJ+lULiK02rsVOuELRUfbDN/lgKU1ssIMZ3S
hC0VH2wzf4ah1tHXJiUunFXhpRUfbDOHNjpF50p8VnusTWNLxQO7WM0XoGY0cprEqFjNFwAdApe/
I+Is1UiqbNaKD7aZD4OAqYx5ASj2MmFLxQfbzIfF0Lg0qrHWEQtGYmvFB9vMi8XAmROvliVobKn4
YJt5sRhi7/Hwp6mF4ANt5cRa5ypFGpMClAayu82s+GBbOTGGY78BgBkpyu42s+KDbeXEWgucUada
e8Gq8vFJ8cG2cmIMR7Gwz4ylyGnOJ8EF2mqugMF2CsxI8jbtrPhgWzm0VkMj6pyIO6oeZbPig23l
0BhuYnUCtPJiTAQps8XtmHqtmlUqPthWXqwR15BHDJAx47OcA2JWfLCtvBjDZfbWfLl2Pq+gsaXi
g23mxijETJw5EtYeq8aWig+2mRsrgThrLmw/oCpXogQfaDMvVkLCBiMy3Vhqaqn8Ruz5lUf9Ej+7
/rTaPb1dlvzssFvT0/fbm2/nZzfb68936/uny+vt/ROv3/0fUEsHCOFDg/T/JAAA8fMBAFBLAwQU
AAgICAA2q94+AAAAAAAAAAAAAAAAEwAAAE9iamVjdCAxL3N0eWxlcy54bWyNlEtugzAQhvc9RUTX
4EBaqbHy2PUE7QEc2xCr4EFjA+nta5uCSJtEXoK/f94zu+OlqVe9RKNA75M8WycrqTkIpat98vnx
nr4lx8PTDspScUkF8K6R2qbGftfSrJxYGzo+7pMONQVmlKGaNdJQyym0Uk8iuqRpcDX+CcZi5QFe
qq282FixZ6+07BTvOcBLtUA2xIo962q6lJcQK76YOi0h5dC0zKo/UVxqpb/2ydnalhIyDEM2bDLA
iuTb7ZaE1zlgPnNth3WgBCeylt6ZIXmWk4nVXXOSGJ0es+xfZ0xfRXe1r+6kx88Mo/sb4OsWbUR8
izbiKnyOqo12PdJLPQDM5faCcfBD0Yv1+oWM3wt6eIgPqKzEBc4f4pzVfK4CNLcGxHE5cUQqe9/+
iUaf9F3LrwRlC2jnQMr4xXfVKeaxPdumvj+2/nVCKxTiJurC2RA3wm740l7J4flqrx/Xf0sClKx+
79HiBroYg0tqkWlTAjZuLP3TrzGXl8lCvqm3GyyOVtwGFcRtqyUgygJFmbmP5DCdz/FqksOO3L6n
hx9QSwcIxJ7dQp4BAACPBQAAUEsDBBQACAgIADar3j4AAAAAAAAAAAAAAAARAAAAT2JqZWN0IDEv
bWV0YS54bWyNkj1zgyAYx/d+Co9mlUcxQ+TU3HXomqWZe5ygpVXwAKMfv4gxTXodOsLz4/8CFMe5
76KLMFZqVaIUJygSqtZcqrZE57fX+ICO1VOhm0bWgnJdj71QLu6FY5E/qixdRyUajaKaWWmpYr2w
1NVUD0JtR+g9TYPRujN3Un2V6MO5gQJM04SnDGvTQprnOYTphvL6xg2j6QLFaxCdWBwspDiFjV0S
/jfUwt5H0lrfjBZ8DR3sSJLsYV1vdGs47/4q4NkMfELmWHyRYnpG0bX+3YUTFAUB6gxTttGmZy6M
rno+qsWhQrxECcKriq9LYLadA80bYniD/QJV21stpaoiVGuFEoY5baqTlzj9tMkw2Z2VnKPH/ffB
6E9RO8hI0qdk9zLKjsf5/pAV8EuwgAc7+OujVN9QSwcI9J0lTDIBAABmAgAAUEsDBBQACAgIADar
3j4AAAAAAAAAAAAAAAAUAAAAT2JqZWN0IDIvY29udGVudC54bWzNXcuS2zYW3c9XqJQ1RRJ8q9rK
IlXxJp5NZqpmyyYhiWO+AlItab5+AL4EoAmZ3fEt0K5yTNyLx7k8BM6ppJGXX29FvnnDpMmq8svW
3lnbDS6TKs3K05ftv//1uxFufz3846U6HrME79MquRS4bI2kKlv6zw3tXTb7PvpleyHlvoqbrNmX
cYGbfZvsqxqXY689n73v5upbmvaeL+7eJfO9W3xrl3ZmuULf+HX5zF0y3zsl8XVpZ5ZLi8p3P1ZL
O9+a3DhWtOpFHbeZtIpbnpXfv2zPbVvvTfN6ve6uzq4iJ9OOosjsotOCkymvvpC8y0oTE+eYTdaY
9s42x9wCt/HS9bFcfknlpXjFZHFp4jZ+91abt9NiRrydFKVJzjFZzI0uWXy9Trr89Top37eI27Pi
nYTmNxrs/vj2x4MLpFg6F8sVSpWQrF4Ms8/m+1dVNS2Vdeg/0G65yLJcs3/msq9P068kazHh0pOn
6UmcJ1PFq2KuaDTPNmmGgd8YTSfis0I0ig7I7MNTcpMqh/7Ptz/+TM64iB/J2Y+Tjaxs2rh8VIaw
l6BE6pkE1xVpp8Icl2+Y9G2haW3ntsjVnzuLjqknkqazqXQ5jkk/ffrhGW8Zvv4i7IfP+RCZXdJE
3Azn41cy5Q5w8K3GJGNI4pwRwSgaWjRKjqrec73FPZEUt2XDMUJU6VEecfo4hr7c4UZr2JVk35K4
bFgm3TJYaABL697suvdhsMk7xP0odGekhGry1qQzIpIed/RhexjPxfjSVmysxOj2sebw0u+Aw0bY
t276PY5h+bL9p7WVkjbDU5GVHaQT7Zdmp6ylDLe35uHFnBnz8NIP+n6C5Gxvh4ZjXGT5nTXRDW47
djmRuD7TFde0dpi0GW427IyiQ5HqOx2grErcTcvN8Gw69Hy67omfrN9uWeWMumqy/kW05NJN+oEl
buj2P7TQUyCv6LHzy6vDfm/77GOW52Pu1DClYp/9fszJFAI/4bGilKKKp8n+R2cM63aCObYalDBx
qYixcynHtz76gWo6n6pmcy9eq9xo7zUdZCLldohmZZJfUmycszTFpZHgPKfcOsZ5g7dPXgcf6qvA
N7cEx62Bi7q9jyPmOH7Dximux5ympiIEGxVJmShwxmaSnc6tEZenHKdGfMPN4/0vLZM7X6ahsdvf
ur8a3Ff3tIJpRhcb3408fsW5iDSvTjE92s5FlkhVI5htMph+rgQnfeWGeKc3O/CvtE7f5Wrf6E7z
qLb1lPlqmq+Nu96HN56n2JZO6/9sLghfE+uUGn3TVkzox2v+usQES6FrljIdaO2QlxRS7IwZ/6fg
53e9aRIrDNksM9WkQssL/bn9b4gs5JD/hEOzsYlD/sc4FPy9w6up8iz9eUfD0lWHP5H5MwtKul/v
F2QqZcgQeK3S+/TQrenw0nOx+7NbxcAihHaWG4w0GjmKKLkQa+zs5P5MMFWvu93I6CSPm2ZA2213
E9W5D44pknHanMqbMt3wD9xGSJ/66Slv7Gjn+/a4HlrSaBegkPuYhBkQK04fqPOKHi50151NpPXt
7Tw7swwqB0+Ux2lKMMPx5xnjFu1+Q/vxb1GwGf76dWr8GgXjIvrNpbqQhJ6ucdOfHqwgVX4pygmL
tXNdDgl9HJ+G4tvhLgiRXHza2jWOyNixMR1XVA738vY2LqaHWFOlHJO7cZsvlDuNlsRUa1aEkfFj
Fel4+FjR89Xd51d3/9HqTiRLZ1M8iXtF/N+KKJbUYMLt6sIw/jjMW5xfcGM8AT/75t8zf5y040Rd
USk/qYSaSiVMzVLkcwvtFzf2usZ5PrvQ4MHsY15VZDYp5Mad+H946d9r9+fwjvsOeUXNt9HbOSGL
nk0x1WpGz99GjPWNbKKP9Vg0zGzqMDaprlKAtkiD0tdHW5jqqt8N3ceklskmsvc/nPV0Q87K03Yc
6PBbt6jNVzpe3zA3sPluYXP1mcWwBNWCdfZqM0sVn+uB7uHeUwhL5qTki9utEKGHu0836cevQL0k
8SM6yB0/WOGfUTVWFh+mLKH1mCW0NGF7XtNPY4ucncP/eswpRzThDmFw25ZA2QfVD3JEE+4ICDfi
yEwf9KCzgcA5Imc5pFJEE2wo3K6SzO4ayGwjINw+T2ZfF5mfs+nz6AIlm4NVsNkFwh0q2Ryugs1A
6gtZHJvpgyZ0QCIK2So2yxFNuIEEFkIqNssRTbiBBBZyeTa7utgMJKOQp2SztwY2Ax24yFeS2V8D
mRGQwEK8BUS6PCCCequRkszRKsgMJLAcpQeUI5pwAwksh/eAji4PiIBklKM0gXJEE24ggeWIVs/n
cEsRTbiBBJbDm0BHlwlEQDLKUZpAOaIJN5DAcpQmUI7owQ10JLm8B3R1eUAHSEa5Sg8oRzThBhJY
rtIDyhFNuKHYzHtAV5cHdIBklKv0gHJEE24ggeUqTaAc0YQbSGC5vAl0dZlAB0hGuUoTKEc04QYS
WJ7SBMoRTbiBBJbHm0BPlwkE2po9pQeUI5pgAwksT/kvAuWIJtxAAsvjPaCnywO6QDLKU3pAOaIJ
N9RXrPSAckQTbqj/DIs3gb4uE+gCyShfaQLliCbcQALLV5pAOaIJN5DA8nkT6OsygS6QjPKVJlCO
6MEN9l+KKsm8Cg/oAQksn/eAvi4P6AHJKF/pAeWIJtxAAitQekA5ogk3kMAKeA8Y6PKAHtAeFShN
oBzRhBtIYAVKEyhHNOEGElgBbwIDXSbQA5JRgWj1XA6qFNGEG0hgBUoTKEf04Ab7mRPhh040kdkH
klGh0gPKEU24gQRWqPSAckQTbiCBFfIeMNTlAX0gGRV6qq1ZjmjCDSSwQqUJlCOacEPtzcIPBOoy
gT+o6ufRRUo2R6tgM5DAipQmUI5owg0ksCLeBEa6TCDYj7eqyCxHNMEGEliR0gPKEU24gQRWxHvA
SJcHDIBkVKT0gHJEE24ggRUpPaAc0YQbSGDZFu8C2ZMmfEBCyrZEt8djFSOacAOdSrYluD3h8gEx
ogk32KULrsBnXUYwgLpcwfKUfPZWwGe4uzSUdPZXQOcQ6toFKxTorMsJhlDXK1iRks7RGugMdfGC
eAUMz2cpogk31MUL4uUw2m6HCaHklHQLDI/VWQOfoWSWeAuMwGd3DXyGklni/TDaLogJoc5d6R4Y
HmuwBj5DySzxHhiBz+EK+Ax3dZd4d5cmOkdQcgop3SBagxuMoGQWUrpBtAY3GEHJLOGKGFvbHTER
lJxCSjeI1uAGIyiZhZR2EK3BDkZQMku4JcbWdk1MBCWnkNIOojXYwQhKZjlKO+iswQ5GUHpDuCfG
1nRRDNxVoSoyO3/LC8otjdh0GO/1Hq7vH+/8lx77m/7HJ/n/13f4P1BLBwjJQzHnzgkAAPBvAABQ
SwMEFAAICAgANqvePgAAAAAAAAAAAAAAABMAAABPYmplY3QgMi9zdHlsZXMueG1sjZRLboMwEIb3
PUVE1+BAWqmx8tj1BO0BHNsQq+BBYwPp7WubgkibRF6Cv3/eM7vjpalXvUSjQO+TPFsnK6k5CKWr
ffL58Z6+JcfD0w7KUnFJBfCukdqmxn7X0qycWBs6Pu6TDjUFZpShmjXSUMsptFJPIrqkaXA1/gnG
YuUBXqqtvNhYsWevtOwU7znAS7VANsSKPetqupSXECu+mDotIeXQtMyqP1FcaqW/9snZ2pYSMgxD
NmwywIrk2+2WhNc5YD5zbYd1oAQnspbemSF5lpOJ1V1zkhidHrPsX2dMX0V3ta/upMfPDKP7G+Dr
Fm1EfIs24ip8jqqNdj3SSz0AzOX2gnHwQ9GL9fqFjN8LeniID6isxAXOH+Kc1XyuAjS3BsRxOXFE
Knvf/olGn/Rdy68EZQto50DK+MV31SnmsT3bpr4/tv51QisU4ibqwtkQN8Ju+NJeyeH5aq8f139L
ApSsfu/R4ga6GINLapFpUwI2biz9068xl5fJQr6ptxssjlbcBhXEbaslIMoCRZm5j+Qwnc/xapLD
jty+p4cfUEsHCMSe3UKeAQAAjwUAAFBLAwQUAAgICAA2q94+AAAAAAAAAAAAAAAAEQAAAE9iamVj
dCAyL21ldGEueG1sjZI9c4MgGMf3fgqPZpVHMUPk1Nx16JqlmXucoKVV8ACjH7+IMU16HTrC8+P/
AhTHue+iizBWalWiFCcoEqrWXKq2ROe31/iAjtVToZtG1oJyXY+9UC7uhWORP6osXUclGo2imllp
qWK9sNTVVA9CbUfoPU2D0bozd1J9lejDuYECTNOEpwxr00Ka5zmE6Yby+sYNo+kCxWsQnVgcLKQ4
hY1dEv431MLeR9Ja34wWfA0d7EiS7GFdb3RrOO/+KuDZDHxC5lh8kWJ6RtG1/t2FExQFAeoMU7bR
pmcujK56PqrFoUK8RAnCq4qvS2C2nQPNG2J4g/0CVdtbLaWqIlRrhRKGOW2qk5c4/bTJMNmdlZyj
x/33wehPUTvISNKnZPcyyo7H+f6QFfBLsIAHO/jro1TfUEsHCPSdJUwyAQAAZgIAAFBLAwQUAAAI
AAA2q94+9/opwtwDAADcAwAACAAAAG1ldGEueG1sPD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGlu
Zz0iVVRGLTgiPz4KPG9mZmljZTpkb2N1bWVudC1tZXRhIHhtbG5zOm9mZmljZT0idXJuOm9hc2lz
Om5hbWVzOnRjOm9wZW5kb2N1bWVudDp4bWxuczpvZmZpY2U6MS4wIiB4bWxuczp4bGluaz0iaHR0
cDovL3d3dy53My5vcmcvMTk5OS94bGluayIgeG1sbnM6ZGM9Imh0dHA6Ly9wdXJsLm9yZy9kYy9l
bGVtZW50cy8xLjEvIiB4bWxuczptZXRhPSJ1cm46b2FzaXM6bmFtZXM6dGM6b3BlbmRvY3VtZW50
OnhtbG5zOm1ldGE6MS4wIiB4bWxuczpvb289Imh0dHA6Ly9vcGVub2ZmaWNlLm9yZy8yMDA0L29m
ZmljZSIgeG1sbnM6Z3JkZGw9Imh0dHA6Ly93d3cudzMub3JnLzIwMDMvZy9kYXRhLXZpZXcjIiBv
ZmZpY2U6dmVyc2lvbj0iMS4yIiBncmRkbDp0cmFuc2Zvcm1hdGlvbj0iaHR0cDovL2RvY3Mub2Fz
aXMtb3Blbi5vcmcvb2ZmaWNlLzEuMi94c2x0L29kZjJyZGYueHNsIj48b2ZmaWNlOm1ldGE+PG1l
dGE6aW5pdGlhbC1jcmVhdG9yPkV2ZXJ0IDwvbWV0YTppbml0aWFsLWNyZWF0b3I+PG1ldGE6Y3Jl
YXRpb24tZGF0ZT4yMDExLTA2LTMwVDIxOjU1OjQxPC9tZXRhOmNyZWF0aW9uLWRhdGU+PGRjOmRh
dGU+MjAxMS0wNi0zMFQyMzoyNTo0NTwvZGM6ZGF0ZT48ZGM6Y3JlYXRvcj5FdmVydCA8L2RjOmNy
ZWF0b3I+PG1ldGE6ZWRpdGluZy1kdXJhdGlvbj5QVDAwSDEzTTMyUzwvbWV0YTplZGl0aW5nLWR1
cmF0aW9uPjxtZXRhOmVkaXRpbmctY3ljbGVzPjE8L21ldGE6ZWRpdGluZy1jeWNsZXM+PG1ldGE6
ZG9jdW1lbnQtc3RhdGlzdGljIG1ldGE6dGFibGUtY291bnQ9IjMiIG1ldGE6Y2VsbC1jb3VudD0i
NDMxOSIgbWV0YTpvYmplY3QtY291bnQ9IjIiLz48bWV0YTpnZW5lcmF0b3I+T3Blbk9mZmljZS5v
cmcvMy4yJFVuaXggT3Blbk9mZmljZS5vcmdfcHJvamVjdC8zMjBtMTIkQnVpbGQtOTQ4MzwvbWV0
YTpnZW5lcmF0b3I+PC9vZmZpY2U6bWV0YT48L29mZmljZTpkb2N1bWVudC1tZXRhPlBLAwQUAAgI
CAA2q94+AAAAAAAAAAAAAAAAGAAAAFRodW1ibmFpbHMvdGh1bWJuYWlsLnBuZ22WdVyTfffHB4K0
oMQUJLxVFIShdEqLiKQ0DKRzIyWkkQGCgoCCtIADBlJSo3WUjC5BwjmJ0Tm6fhs89/N7Pfft/rn2
uq7v63xPfM77nJdaGip01KzUAACATvWhkg7h2QMAkAAoSQn/POd+TgEAQHJVJXldn7TldD+57p2T
9l54Z4c2jyoox5JSG0plUsN3UHtBKN5QR1VMt2Ju9lIyw6WUibE+5xxIlaitaAek4xvEEtTfkKOa
3rPsfT99cD4gxnDjKKAHtyVT0s65j5fI7J3f2q/PxHn0+Prfv88pk/y4TeHNSRC+AgzIvvMth/tY
yhf/kbNOynvIyo3ftDm73fT1qAub1MbDL7DYHVCpw1/RG5NcdS/z4t9xmHzGeIHzGyw6HyH2rqNx
9KsNjd+HMplk0Ya1cNQNuwNOmbcwfqq7mAYzhlKLB7ikRumCXn63jB3/xT12UBv/7oFAoTn3rtsx
3qpYKkEaQmeWB+PO2h+DbnQseFOj3sI3Oj8nlgQqmw+gl7bkU4vXx1zJej8FVLcgjdxTx+Pmfho4
0INx4+lUByUxajoXV806Ks4VUrAOgzVdTybcYvA+y3faleWPmjdCTKtsOikDA3zptwUO5iPur24h
+LseRdUYvd2ZsJAr1HmJ3bRk7BWBYmpL9uzCAmbmpSZvjcv0FO/5aCI9cegc0oX+5GjJvE6DtLkf
fm9vGvgISTxtnqyqf9TQsXhdfaSBs/eK7If9FJ+2RIZtu8L8mczIIqkoNWvpBQfZLd4oakmoZbuE
+exc0e1F35UYF7+9loc0xYXUCdXzgx4r2xMv1xWNRX5EPX7TcFIWAVz8Mr/hfasCfWStkB1i0u7U
JLuIc3wmG8Hm/6XJGTIXN1b1bGdlkjEG4z2UYw/rFSn51XQoduVVzvN1Q5sRw6aKJaXHR6jW5YC/
Pgn1yM40tes7b2tyO2vxnaxn9KXQoW1YoOmbXIJaP0INWlcXpxAcnNLFl1gNf7Bqs1kM+7Fc1TP3
4ysIfKn9zvHevl7A8ORo2UrHIf5cca4z9gjjAWuVVVFmLxRB+DoC1xHNPVQFdGw3Jpjku0sqrk5s
t8EdX+DCE0Tbnu1O7TRdnThO3BmenMWqh4xk5YNuHaJa1xEJ/KOfcDhUQwk53I8J1jSV6bK68CVv
4RWbj1WOu5z5fVCmbsmugh5Xu2WFt2aTwato46EbNk3cZAfsfVZXtsCXV8VGEZicsmZ+oe1GIYG2
Jsey1jcQK+De1q8yahBL1NbYBRpKTo+1bReOr4F6zJugCYkAwSm3pHtVGyqbLVPp+8xog49mnp/H
NnSyEJ18mA4QS5ppQ5Uijo8Kt+qBSfP4HWK5V3B0MSKcn7osJ2N7P3lTb1MnHnxCm5JQkv/IZ3UI
fPAyHa8RByPfWsVDm5piT8BiH72f+AEdOnoP0Le4P/XRmKfpCUN5TTCoDqv+V9E1ZezAsEWZtbqt
NefZBvpq+Eme+wv1asj1d3fKFoFP8KhkuH1aQROvXawldnll/I7s816LURNGCK1wYtOmX5Vj8hIo
Vo9f18ZFH3+7sxc5IWMdr3L0gOvg99ur1x+lD2P1Aqg4J++6botmhUJHt9jt5gej2bvoONzIazBm
hV7OE6V7v42i9vL+UvQLMthdNvxxZHlu22rPeF9/69z++vRqQMwkiG1fyHsgqmrB5VHU14VF1eG9
pd3ZcJKAVqQ1wm7u3VQ2h4h7CXfQFR4O7OUKwV+l4ldFCzspLq5nQHijHekynaeT18T5qu5rZnZb
eky7wAWPQE/++m2nE8tKpaOn2VkeK44s+7WE3fIA8f2Uqj2mM+BOQI86J7rv+VWJKbehmeE4D+uh
J6o3dWYMmDoGdTQiRJ8vwDoGh3LP5eER71sds2+u53SBfUOLFxlHQxe9Y++/n275LjZkVy6xBf8s
zsej1qbvfy4eKK+uuuioNKRCkg3c2BsVRFhaxOwj33Qydn5LdXLOskgle7DAlgjOx1BGXN9nFcW3
6ghsXT8Blcx1rhX9rFTZJBOa6BoVqRYK4vXzWtvBjg9G2Mwc8KRoQ0fz8i4xb5aL+d+M+8YZLx2k
ryJqvbv9UX7vSjsvfCoREd9mO+xCYVRiQyFWe19aYbimXHMquNHZlPbVY7gN6cVomxsFPn9Bm3EL
1YG+R6/zV4cNpaUykfRFLbd0mhYHjcuCO9bo/YIYWd1VSwqePP19uXtYPVxgjp2JNV5RprY1J1C5
t8XBZV6m24itfgdxU9NWauBVZ7zutH+VZPZSeLr5wuV+oyBH35B++/E6NHOCdG6F8eX6OeBK0djX
qAxk7vJHZdKy5xaFikvkmh+Xz4vsxX1Y0Wirlqm2dWa3QR/jPCNV2sxjBBIXJiGqSqMu8RtG/asZ
qqYG3bVbExlXaYtfFgQGOGKVh1ChFvIZRcViNfP5Nqh3nT7IicM28x+hZf6pZGiTqfv1Bl8r/ce/
Pyy7qBvwbmLo9rCU0v5Ao+H5BhalH2Rd6es1MG32Qs2nh4Yq2BQkUnaPW2/mFo0JQnRpu9FuQP/J
ptTOsifPPXuL3Fdibs3oGGtzp/bxE+nFXvgt1nhbe5kKdcvftTo/YAkY073pEnn/Ag0h33WaYxtT
a/Zb+8Zkez7swgXFjrKyiLrGudnORRelY2TGsy/zsucifV80HtpUHUZQiEDKuRfVNrLwIRtBUwMU
WvRFkg+PPks261kpS+Ret9x+rdVTVJOPmT+C9dNh7kbOsGV6GczSbngrpsaJGpRHAwuB3I1+s5pO
SR+4zSXPN6AEq3m4X4PjBcHTsoMROsq5LrmzEI28Y3BN0HS0Uy49dElZT7bd/0aZa1BrwjcsndZi
umm4cxaCQy8ALKzROZGr8pa3sW4THtCh5iDkOY9Lft1EfyKpKKHncW9j9+A+HVuVRaCndJm7f4B3
OXMA1Xk854QT0jGzrTIlZWNGd69l41tRC2/6NxKIuym1zX1oIX0NYrbR0ZTyMq+UMd1apuvtGev5
grJlcbTRwOcEhBg3uU/Tfvmt5n5jwihNU7c4qN1drbB/xCGcKobIDM+/QhcjK777zGBg8bC/TcTR
Q5zFagQa2DHguuuyvSXJlmg1e7V2vEY56MFrsfzMLD+R6SB6gDYjMLBa0r61xsGoXYYvc+bTOuY4
UiCBNmpMstxSsRkvisngxyd0yRkHCkq6avI0FQ94GRk/nbS1D+goYG1XNiqfW31xcd0uzFkj8wtZ
kN/CJLPckOEDrwwa8MpbdN6xRHXe0wc6Y0j0Wwk/jzFgYW6JbzGr/cdlQQwgecj5cEzXrrO+L1aA
Ja2DfAQoAYpKcSrJDTCDfu+tD3cZ1NgLIJEbwnfVGa0vyHahHXkdxzxFrReTva2ZWRIjx0Z5cjkN
IivB06lfIWJfftQv4fcnpVdFlnnL2Z6HDRCsoVMWH0FHRkuiaD6p4R93c14ZxjAoDbFasFVC9Iz0
7zC+wyQ1DmMK5sLcazel9sdAy3RxTOyxUbW/q6DwQuWt3gKOLLxyTPRnW5lBDW7nK9FO1qhA0Cfg
JCCbm+eY55qt0/dku9DLDe9bN5cVQgME4RNlhUCkfpp5fto2zqEOLZYjeUtyqT3KYqbzhWYM1XWC
U0nrfXteUWWu7jyRUoMey5SNOTtzbx+ay8rMl86YRfUCumJZqxtMpi9Ufws3T2Lfrbf4zHaPju04
EjziwRZEyNC4T6ltd/3PkIPY/G6LwKE3JxfvOlKqPx1iSAEwiLA/QfolGqc7zlXL4DAjNqT5rl/l
eCcurasUBGYFUwNpw1CvV1FCON4DGCmvqRwDdfWPE1WYxAcU7BzCNWk3mcOsaElM9HmtSJjBRV7T
u/a3TT+bM3xAKb8ne0CXRfWKVYERkJ1rRe1dncklD/e4TrFxEgBZfkGw+xHFX8ymMDcRog4Whrl9
QJET7EEkDb2ShOdMAc3NCjAaah0sarAm3xVL8AeWE6ivHpGDgjNOPkEOqqbTkOoiKRO+5F8TDtFW
tCeVG/J8jA0UXe1Av2nufQUL5YhzUGOXM/WhYgZdHmIq9lxTSfWivycBlF7VSW7JoiHDhpx6ULls
z8PzkH5dlSfF/Qu1Qo8WNjS0r+p3hITBrlBCjOTNmtX6pbhfZE9dXlMDL3xEgYVV3leJ3+x+rPtz
0k/0jeYHQq4+uLGsxFwvrzRQU5ODRd89jfiySBg3QEm9nQn0bLZouuMi/cjQvITRYFcl2LhcDkZz
fouGR/yyLVRyvJkqUjeH+EYH68V3/Nw3Yzz7nmyLvn3dp9lQaiDDa+oR/NfGhJvvclWlNGceOm9O
ku4DxCmjTp2hi/1ZFgQqlDQxNVq4cM2OiehLlitCoplxsEL3Mc2/nVkpdsjL3ZvZnnOCDshy9bRf
aAFy6CKfcjGvTMW1Vp6XLn7Ysup83sLzNiELIbpIl/GRFTUXTPd0Gzq5w0Cl2J2s1EQRRkO5wTdx
cIuztKg2tVJ0sISxowJsSAyKQgcLyuiVqRpdpLGKs9ZNE3ijSX7mUkJjSkGDsTFBPWGqRM0Ri0bC
5SiZ67oTVt099CbzFQWhSg9hB782rNT3OnzOrtdn/pqykTBP0AVZlitMQuCbZi5BNRcc7bCgAPlA
QaLnfVpIwpH1f4tNzAY9s9XhO5yd7WHAvp1u3E+hYHUm6wI4zaUq1ehY2E0No6cU1+z4dJHBXMyg
ZzRCFsc/XT41eBu5EuLZJMRDuWXyfti0G/l7mpl6ppSJIP1884cWrKvfIQlmiV2crMzE4Hyvkcc2
S1Xo/6n2K1BMVNhaTnYVKv4CSUdWjA8LKH0uERXgcxD4ta/0XEZrnBkuqjVGlOEB5Wn4483HMVvo
myQ42M2el+wktPQ0zKDblDCTdzW7WFQF3i30rVFHTSNbGkn9i2wU7MX+ctexVRU37+HIWHvVoCzZ
APC0+yaK/Na9bCkHLP+3wxEeK8WGGw0taZBflw4QfgYMmCLt0zQHjdsldY162vtjC2queOfCxuXF
nejO/Gn3LVV7vDRQ/13zEYzU1xyQHU6QZBjKhgZ9nrxkx8Wbx3qlchvg3kboVCodbDh5pRQraxIq
oWJLYwV9iEUQiyrMBD0cKTa+A1f+Z5HOURvo4Cc//92tdcAh59DvxBC+ibPfH6vwi+yo+eWbtGJj
Dhghvp0zuuB5+6xRP2Pw8w507GiCTlDyu3dx1EG2A9p5N3bAYjj7Ba02okiuqsqP/X4rLKTt6+1C
DxyckHsakIUif9G4tcSuwtpbT040+U8tDiZWHMCzMNtaILaX1EBqAp0SHutt0ZATO4RpheSOosc5
myYtotAVsCtEKnxcfm7GqIt05WIurvuVtytu01TWP1h0rT6KoF95RWz8USiAGYS68b91yPRNKnFC
5qMyvoX4Q0a+Xz3TOtzjfD6oQOsvjsgRFPwi4Vj6J2fIjZaibkAwUfLU3rO/N3bbTFLlCRavPoRJ
3RQbK1z/bpvv6qC0w7RyufLsy//f9XdUXhVcz3pyxIc7MGrsgIkdilgWgMKyfUjedbeNakUgJZQu
liU4RNq4r+YFxI5wiME9XIWGS56DX3xnhUKa8fe0C8DPSdoYQLFh/5W2IXRUbgbvzJUpt25PcllY
Y5c7Tz5YhYaB+5S/OeF9ZMH58+9R8Io/As/Ja3SRfdyzKoKRAfHgP+y9vS8p2Qvpf89peacCrO8h
f5paaCeTrABCWH3Ml1fFMnWVpY/ii7zWaZZXLmDuCUl8WnB2s0FIhaVpE8jwIhz1rpkz431ukaMZ
57/nwUqav2WBOTS2cZb76Nxv/yU5f6z8KWtZGkxMnb5MRSkd9tgyPIggPdN7Zcl4I3578jH0U1qy
/7NpXhVAnQjx5l5Uw12U+gS6oM7ISehIMu7x+DsRmjOXur5rTinWN95lUX3Quke6p0ucl6zUYvOF
CGO+XPI/gAnilmrFesos4rjjnJHpe/s3/pzMOBA8sskXeE3F31DvLKCA07cG5U+52sMx5oY3e3aG
RoK0HNGaRJlbXCKyhlhzQGRKDM/EMqPAzRs4veioHvCzUW9HZU7K11mn12TCejRHufnvsySK24Sw
h6WVaWFJQ3WRQ3DJsD2/prLtnzZf+70sus8RxDW0koaL6xOQnYpKMLj9tlkn35UkK9eVDq7pMik6
b9NxmKVNLF4KUGdNvF7Rr7UC8KcpXOIJ3tXWNrFQjQie1lKRpknS8FTDsjQ+0borvBRs5/ZxpMay
LIcu7KLCaTGMd5Yh7NU3ftWXtkiUXQ/GWmShYGQIV80rss+Rd/kD0rhs9N2LV8/bAVYJ4xb7wUuq
/KsypuQV9Unldtkynj4EDztNuveiWX36ldCKy/+YeUZB/OZ/Ah/980xQctl83y5Kz0No5TFN9QvX
76wE0hByv+zNDra0TJqtDNThZfp2QMpwKgNm6MpuzYvQkYPqOibBHWM/gITVmacytI9Ez7fA125A
GhLqBIN8Fogio4OxnVQWbjs6UOW7XvyfHglGwd9IS/zGpkjK4Np3oA5nzGKIb28XTV/GWrdCFCIE
CWJ+ygAMbLj3n/5I6cs6VYkSvWYa39/d4TcCfhJxNgirk+usTkz1n1UhPw8FzOVfa5JfJEyPu7uE
XSA/c/bXvsK/phP0+zSrXtNylqnoJULZWZlBPwSYiDDSQlqfUu68CIzYfSrj5A3cxEmX6+p3tPnE
X/H/IUeaI48t2PrTHnb867nE31v/jzPSDRNIR5qtjUV/tJ0ayZU0mYUF+pcpSTOUwNSuE+Bbs3xe
PDUp3awpFceOIf+yVko4rQDGxmR4BUGGmKuf3QHzI8eKP/kSez4FVdl46d7RfxdkFZjardo8V29w
77AXTwnVv8keSy9bWAuCjulQAMSsCLWhVsBeGmnlTxD3DOGSPgPw9DO3Iq5pWmYQ1xATtHgSUIbY
qEUmyVxKAywR5SKvjh1j46Cn1jUDep+tfSH9H1yHi6P/FL2tupBBajiAsDC8Aepw5VkEn9JOry6f
lPY/i9XR7nTumDIz4CqZ8mkH9CSvheyRJergFL5dIec1fSp827TbYfYPHb66n9uwmjSeU1KjehIs
zinFYO5zmtbxqMIoJzof07hIzA9a1JzWKV0gdMuh2/DqVVt3q5mu6cLrBhKko0XEMThmhPSV8aPa
1Z/sNk6dH3lpPQYqMvpw9uHocRAd3cY+yvDGFWYczfQTEqI79uMIVXP1YS/+Eop/Z7ecW8/hjEeE
dbWa6r/7m/rZwkbrSL2T2EtztskRascuXW+Pv3/GMqKCUpz+CTOi+V3ZD3o4TeaQ0l0A4aeqrKFU
rPA09P8AUEsHCOod2T44FQAAPhcAAFBLAwQUAAgICAA2q94+AAAAAAAAAAAAAAAAJwAAAENvbmZp
Z3VyYXRpb25zMi9hY2NlbGVyYXRvci9jdXJyZW50LnhtbAMAUEsHCAAAAAACAAAAAAAAAFBLAwQU
AAAIAAA2q94+AAAAAAAAAAAAAAAAHAAAAENvbmZpZ3VyYXRpb25zMi9wcm9ncmVzc2Jhci9QSwME
FAAACAAANqvePgAAAAAAAAAAAAAAABgAAABDb25maWd1cmF0aW9uczIvZmxvYXRlci9QSwMEFAAA
CAAANqvePgAAAAAAAAAAAAAAABoAAABDb25maWd1cmF0aW9uczIvcG9wdXBtZW51L1BLAwQUAAAI
AAA2q94+AAAAAAAAAAAAAAAAGAAAAENvbmZpZ3VyYXRpb25zMi9tZW51YmFyL1BLAwQUAAAIAAA2
q94+AAAAAAAAAAAAAAAAGAAAAENvbmZpZ3VyYXRpb25zMi90b29sYmFyL1BLAwQUAAAIAAA2q94+
AAAAAAAAAAAAAAAAHwAAAENvbmZpZ3VyYXRpb25zMi9pbWFnZXMvQml0bWFwcy9QSwMEFAAACAAA
NqvePgAAAAAAAAAAAAAAABoAAABDb25maWd1cmF0aW9uczIvc3RhdHVzYmFyL1BLAwQUAAgICAA2
q94+AAAAAAAAAAAAAAAADAAAAHNldHRpbmdzLnhtbO1a33PaOBB+v7+C8TsBDMcVT6BD6OWSa67N
AEl/vAl7AU1krUeSIfSvP9mGTOLYiWOstml5YrCkb9er3U+fZB2/vfVZbQVCUuR9q3XUtGrAXfQo
X/Stq+lp/Y31dvDHMc7n1AXHQzf0gau6BKV0F1nTw7l0kua+FQruIJFUOpz4IB3lOhgA3w1z7vd2
YmPJk1tG+U3fWioVOI3Ger0+WrePUCwarV6v14hbd11d5HO6KGoq6X3fFCLeGYoGJM7Exuxms9NI
/lu1rZP3QmNbg10cdq8/ON4aSH7qVIEfxaa2fRy51re0SWdFYX0XNStr3MMx11TSGYOhADLFwNo1
qk2gGylX1qB53HgM8iLgC5grM8ifqKeWWdCdXrfT2hv+DOhimem6bXd67U5RA3WfBHXKPbgFL20M
1tnTFI/RCSY2RVyG9bmX8lMqoXPAGkSNhUMRW41AU35OiQ7Jc44+HDJZAqhWgRQchUKiuERJla6A
z5nTWW4qHyJ/yUJutXrlsM9Q0G/IFWGTgFH1H3qQnoAlij2yHISirin0lPe7EFVZpff9N4A/dBVd
QYw+JnyREx67HPjO34qZawc7zqOVPXGrZfAd6gkqhX6FwF8R/alGqTSjI9BrwsI06rbKy8aALCAi
0CfRuyXBJ0tc/yNomrhniAwItwZKhJCNnPHw6fXiSZK2qyHpVtsYS3feHEj6QNIHkj6Q9G9L0u1q
SLprTkkfKPpA0QeK/rUpuvlbUXRec3xAUrS04qOTnLOZ5Ixkb+pxBTI2IyL3OMz+6/Wk2E+rAiID
JwLIzaWA6Jg1L93mhMmcfCti5isIjP2XL8/nIvgfUJmCLleEzyFHqCNkKFLQDKMSatndtm3/WVLX
PJhXA1E5I1K7Hvp8jOszIB4IM0ZiItFMYwD9XH4MFaMcJht/hkxOIL3kVmJkwkkwxTGRCtITXUVl
JcDncnu6b8zCGKSe7/wzktJMnIbP1uB7wk/CmUdXVOa6XxF4lRuIc5nAD2+pnGy4uxTI6Tf4fmpg
+2Enu4MEVfzjXfIgFCSa4Jd8xRsypnlMawr1L85GhLvATNToexB8KCnhlyF3VUgytiJVVNIF5TdX
gUcU5O/XSp41xjHSqQgqTEv8GZHQ7ZxQTsTGavzgijEtBIxW5DAI2OZKgnhHFHlty51pRfC6lwmT
WuwCiTfWGgk52xjS1oa0r3lxYVwfbYnxg/6TvVctwIjmtgCjJRHE1f6N0A8EyIixKt+WGtLq+Zc/
TlHMqOcBv3u9/a+CXBC+CMnjY7zCkzjCMLJUevw1EVogpLcIxcefwIJyrQDKv8Hf3Hty/L4y7/la
/S4bpnfbG2gTnTz5ancvzqlKWD9naRgqHBHmhkxrPgNLOlnBdXK/7iMfMZQmGDoRrKcC/Sn4QbkX
yd1ANB7dB2zk3ZQc/A9QSwcI2Cck4SoEAABrKQAAUEsDBBQACAgIADar3j4AAAAAAAAAAAAAAAAV
AAAATUVUQS1JTkYvbWFuaWZlc3QueG1s1VfNbtQwEL73KaIcektMc0LpZivRqogDQkLljGadSdbI
sV173O6+PXbLdgMssBtiIW62Nf5+PDNOvLjaDDJ7QOuEVk1+Ub7KM1Rct0L1Tf7p7rZ4nV8tzxYD
KNGho3o3yMI+5V6mTe6tqjU44WoFA7qaeK0NqlZzP6Ci+vv4OjItz7I9cCckFiHQbrM9GbYCCtoa
bHIwRgoOFHSyB9WWT1zlmKJ0xiK0bo1I+R5kZK4aLXdeysIArZuc5ewkKYQbYsHPYTSuFUU1MeA0
3LHFTRGt6a4THIu+FQMSxP2XjyIYfnSfO20HoHjWzfm913T59ubd+xB0G4KeFw7L+7D6gpw+opHA
MR6bY89L2cX/Jbc6Ue7R+LNWg6OtRDehGH4Pu0sZm15tRxKkNhAr5S975RfXAV+DpT+wp/BUpU5K
lTgp1T9NSjVvUiZaOQx2t/bDSoGQjtFuWBrVzw0+C961Vp3ovX3Kh6sYcI4Sw1Rbxr210yr0dK4k
BMbq3qJzK0hE0EkNhKnUa+NN6AefBj4iJzsY0lomAxcD9OjYG0EDmER98I0jCbYjID+lKH+8Op1X
sTlLL0o+5jhOxry/MEgUngMvn5sF++k1sPwKUEsHCONAAG/LAQAASAwAAFBLAQIUABQAAAgAADar
3j6FbDmKLgAAAC4AAAAIAAAAAAAAAAAAAAAAAAAAAABtaW1ldHlwZVBLAQIUABQACAgIADar3j4t
Z2hbpmUAABpqCAALAAAAAAAAAAAAAAAAAFQAAABjb250ZW50LnhtbFBLAQIUABQACAgIADar3j6q
r2svBWoAAOgNCQAbAAAAAAAAAAAAAAAAADNmAABPYmplY3RSZXBsYWNlbWVudHMvT2JqZWN0IDFQ
SwECFAAUAAgICAA2q94+cxiQ56QcAAAQKgIAGwAAAAAAAAAAAAAAAACB0AAAT2JqZWN0UmVwbGFj
ZW1lbnRzL09iamVjdCAyUEsBAhQAFAAICAgANqvePp3z0ZISBgAA3xkAAAoAAAAAAAAAAAAAAAAA
bu0AAHN0eWxlcy54bWxQSwECFAAUAAgICAA2q94+4UOD9P8kAADx8wEAFAAAAAAAAAAAAAAAAAC4
8wAAT2JqZWN0IDEvY29udGVudC54bWxQSwECFAAUAAgICAA2q94+xJ7dQp4BAACPBQAAEwAAAAAA
AAAAAAAAAAD5GAEAT2JqZWN0IDEvc3R5bGVzLnhtbFBLAQIUABQACAgIADar3j70nSVMMgEAAGYC
AAARAAAAAAAAAAAAAAAAANgaAQBPYmplY3QgMS9tZXRhLnhtbFBLAQIUABQACAgIADar3j7JQzHn
zgkAAPBvAAAUAAAAAAAAAAAAAAAAAEkcAQBPYmplY3QgMi9jb250ZW50LnhtbFBLAQIUABQACAgI
ADar3j7Ent1CngEAAI8FAAATAAAAAAAAAAAAAAAAAFkmAQBPYmplY3QgMi9zdHlsZXMueG1sUEsB
AhQAFAAICAgANqvePvSdJUwyAQAAZgIAABEAAAAAAAAAAAAAAAAAOCgBAE9iamVjdCAyL21ldGEu
eG1sUEsBAhQAFAAACAAANqvePvf6KcLcAwAA3AMAAAgAAAAAAAAAAAAAAAAAqSkBAG1ldGEueG1s
UEsBAhQAFAAICAgANqvePuod2T44FQAAPhcAABgAAAAAAAAAAAAAAAAAqy0BAFRodW1ibmFpbHMv
dGh1bWJuYWlsLnBuZ1BLAQIUABQACAgIADar3j4AAAAAAgAAAAAAAAAnAAAAAAAAAAAAAAAAAClD
AQBDb25maWd1cmF0aW9uczIvYWNjZWxlcmF0b3IvY3VycmVudC54bWxQSwECFAAUAAAIAAA2q94+
AAAAAAAAAAAAAAAAHAAAAAAAAAAAAAAAAACAQwEAQ29uZmlndXJhdGlvbnMyL3Byb2dyZXNzYmFy
L1BLAQIUABQAAAgAADar3j4AAAAAAAAAAAAAAAAYAAAAAAAAAAAAAAAAALpDAQBDb25maWd1cmF0
aW9uczIvZmxvYXRlci9QSwECFAAUAAAIAAA2q94+AAAAAAAAAAAAAAAAGgAAAAAAAAAAAAAAAADw
QwEAQ29uZmlndXJhdGlvbnMyL3BvcHVwbWVudS9QSwECFAAUAAAIAAA2q94+AAAAAAAAAAAAAAAA
GAAAAAAAAAAAAAAAAAAoRAEAQ29uZmlndXJhdGlvbnMyL21lbnViYXIvUEsBAhQAFAAACAAANqve
PgAAAAAAAAAAAAAAABgAAAAAAAAAAAAAAAAAXkQBAENvbmZpZ3VyYXRpb25zMi90b29sYmFyL1BL
AQIUABQAAAgAADar3j4AAAAAAAAAAAAAAAAfAAAAAAAAAAAAAAAAAJREAQBDb25maWd1cmF0aW9u
czIvaW1hZ2VzL0JpdG1hcHMvUEsBAhQAFAAACAAANqvePgAAAAAAAAAAAAAAABoAAAAAAAAAAAAA
AAAA0UQBAENvbmZpZ3VyYXRpb25zMi9zdGF0dXNiYXIvUEsBAhQAFAAICAgANqvePtgnJOEqBAAA
aykAAAwAAAAAAAAAAAAAAAAACUUBAHNldHRpbmdzLnhtbFBLAQIUABQACAgIADar3j7jQABvywEA
AEgMAAAVAAAAAAAAAAAAAAAAAG1JAQBNRVRBLUlORi9tYW5pZmVzdC54bWxQSwUGAAAAABcAFwAE
BgAAe0sBAAAA

--_002_ADF94D8555C7A246B86A633685E0178AB90A35CA97planckkasaran_--

From general-return-3969-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 21:43:02 2011
Return-Path: <general-return-3969-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B6E3A66E4
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 30 Jun 2011 21:43:02 +0000 (UTC)
Received: (qmail 85488 invoked by uid 500); 30 Jun 2011 21:43:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85434 invoked by uid 500); 30 Jun 2011 21:43:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85426 invoked by uid 99); 30 Jun 2011 21:43:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 21:43:00 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [145.100.8.49] (HELO smtp.sara.nl) (145.100.8.49)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 21:42:55 +0000
Received: from planck.ka.sara.nl (145.100.8.32) by sara-exch-fe1.ka.sara.nl
 (145.100.8.49) with Microsoft SMTP Server (TLS) id 14.0.702.0; Thu, 30 Jun
 2011 23:42:33 +0200
Received: from planck.ka.sara.nl ([145.100.8.32]) by planck.ka.sara.nl
 ([145.100.8.32]) with mapi; Thu, 30 Jun 2011 23:42:33 +0200
From: Evert Lammerts <Evert.Lammerts@sara.nl>
To: Abhishek Mehta <abhishek@tresata.com>, "general@hadoop.apache.org"
	<general@hadoop.apache.org>
Date: Thu, 30 Jun 2011 23:40:15 +0200
Subject: RE: Hadoop Java Versions
Thread-Topic: Hadoop Java Versions
Thread-Index: Acw3biTCPQz0yvRMRFOO80O3HF/alAAACfbg
Message-ID: <ADF94D8555C7A246B86A633685E0178AB90A35CA98@planck.ka.sara.nl>
References: <ADF94D8555C7A246B86A633685E0178AB90A35CA96@planck.ka.sara.nl>,<93245D75-B596-4AC0-9733-BB2FF4E458E5@tresata.com>
In-Reply-To: <93245D75-B596-4AC0-9733-BB2FF4E458E5@tresata.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

That's not a question I'm qualified to answer. I do know we're now buying a=
n Arista for a different cluster, but there's probably loads others out the=
re.

*forwarded to general@...*

________________________________________
From: Abhishek Mehta [abhishek@tresata.com]
Sent: Thursday, June 30, 2011 11:38 PM
To: Evert Lammerts
Subject: Fwd: Hadoop Java Versions

what are the other switch options (other than cisco that is?)?

cheers


Abhishek Mehta
(e) abhishek@tresata.com<mailto:abhishek@tresata.com>
(v) 980.355.9855

Begin forwarded message:

From: Evert Lammerts <Evert.Lammerts@sara.nl<mailto:Evert.Lammerts@sara.nl>=
>
Date: June 30, 2011 5:31:26 PM EDT
To: "general@hadoop.apache.org<mailto:general@hadoop.apache.org>" <general@=
hadoop.apache.org<mailto:general@hadoop.apache.org>>
Subject: RE: Hadoop Java Versions
Reply-To: general@hadoop.apache.org<mailto:general@hadoop.apache.org>

You can get 12-24 TB in a server today, which means the loss of a server
generates a lot of traffic -which argues for 10 Gbe.

But
 -big increase in switch cost, especially if you (CoI warning) go with
Cisco
 -there have been problems with things like BIOS PXE and lights out
management on 10 Gbe -probably due to the NICs being things the BIOS
wasn't expecting and off the mainboard. This should improve.
 -I don't know how well linux works with ether that fast (field reports
useful)
 -the big threat is still ToR switch failure, as that will trigger a
re-replication of every block in the rack.

Keeping the amount of disks per node low and the amount of nodes high shoul=
d keep the impact of dead nodes in control. A ToR switch failing is differe=
nt - missing 30 nodes (~120TB) at once cannot be fixed by adding more nodes=
; that actually increases ToR switch failure. Although such failure is quit=
e rare to begin with, I guess. The back-of-the-envelope-calculation I made =
suggests that ~150 (1U) nodes should be fine with 1Gb ethernet. (e.g., when=
 6 nodes fail in a cluster with 150 nodes with four 2TB disks each, with HD=
FS 60% full, it takes around ~32 minutes to recover. 2 nodes failing should=
 take around 640 seconds. Also see the attached spreadsheet.) This doesn't =
take ToR switch failure in account though. On the other hand - 150 nodes is=
 only ~5 racks - in such a scenario you might rather want to shut the syste=
m down completely rather than letting it replicate 20% of all data.

Cheers,
Evert


From general-return-3970-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 23:18:33 2011
Return-Path: <general-return-3970-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3FEC06E7B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 30 Jun 2011 23:18:33 +0000 (UTC)
Received: (qmail 97594 invoked by uid 500); 30 Jun 2011 23:18:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97324 invoked by uid 500); 30 Jun 2011 23:18:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97316 invoked by uid 99); 30 Jun 2011 23:18:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 23:18:30 +0000
X-ASF-Spam-Status: No, hits=2.4 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL,TO_NO_BRKTS_PCNT
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 23:18:22 +0000
Received: by wwi14 with SMTP id 14so2879053wwi.29
        for <general@hadoop.apache.org>; Thu, 30 Jun 2011 16:18:02 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.237.136 with SMTP id y8mr526602weq.76.1309475882246; Thu,
 30 Jun 2011 16:18:02 -0700 (PDT)
Received: by 10.217.1.79 with HTTP; Thu, 30 Jun 2011 16:18:02 -0700 (PDT)
X-Originating-IP: [76.126.69.120]
In-Reply-To: <ADF94D8555C7A246B86A633685E0178AB90A35CA96@planck.ka.sara.nl>
References: <CA2CD1F8.3F91C%scott@richrelevance.com>
	<4E086BC7.5000208@apache.org>
	<BANLkTimZ4SpW6wEGCGT=g6SfMXq4g2UaWw@mail.gmail.com>
	<9068C5A7-6090-42A3-8891-F60EF3B660BD@navteq.com>
	<BANLkTimi1XDyaV7P0meXV=v5z60izzNc6A@mail.gmail.com>
	<918BB6E5-EF4B-4EB2-9B55-8BCBAF6DBA7B@navteq.com>
	<4E09A60C.1090605@apache.org>
	<ADF94D8555C7A246B86A633685E0178AB90A35CA96@planck.ka.sara.nl>
Date: Thu, 30 Jun 2011 16:18:02 -0700
Message-ID: <BANLkTi=wqMG3cYpN3R_acBjD3bKApqSwwA@mail.gmail.com>
Subject: Re: Hadoop Java Versions
From: Aaron Eng <aeng@maprtech.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd519f0cda22904a6f61c30
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd519f0cda22904a6f61c30
Content-Type: text/plain; charset=ISO-8859-1

>Keeping the amount of disks per node low and the amount of nodes high
should keep the impact of dead nodes in control.

It keeps the impact of dead nodes in control but I don't think thats
long-term cost efficient.  As prices of 10GbE go down, the "keep the node
small" arguement seems less fitting.  And on another note, most servers
manufactured in the last 10 years have dual 1GbE network interfaces.  If one
were to go by these calcs:

>150 nodes with four 2TB disks each, with HDFS 60% full, it takes around ~32
minutes to recover

It seems like that assumes a single 1GbE interface, why  not leverage the
second?

On Thu, Jun 30, 2011 at 2:31 PM, Evert Lammerts <Evert.Lammerts@sara.nl>wrote:

> > You can get 12-24 TB in a server today, which means the loss of a server
> > generates a lot of traffic -which argues for 10 Gbe.
> >
> > But
> >   -big increase in switch cost, especially if you (CoI warning) go with
> > Cisco
> >   -there have been problems with things like BIOS PXE and lights out
> > management on 10 Gbe -probably due to the NICs being things the BIOS
> > wasn't expecting and off the mainboard. This should improve.
> >   -I don't know how well linux works with ether that fast (field reports
> > useful)
> >   -the big threat is still ToR switch failure, as that will trigger a
> > re-replication of every block in the rack.
>
> Keeping the amount of disks per node low and the amount of nodes high
> should keep the impact of dead nodes in control. A ToR switch failing is
> different - missing 30 nodes (~120TB) at once cannot be fixed by adding more
> nodes; that actually increases ToR switch failure. Although such failure is
> quite rare to begin with, I guess. The back-of-the-envelope-calculation I
> made suggests that ~150 (1U) nodes should be fine with 1Gb ethernet. (e.g.,
> when 6 nodes fail in a cluster with 150 nodes with four 2TB disks each, with
> HDFS 60% full, it takes around ~32 minutes to recover. 2 nodes failing
> should take around 640 seconds. Also see the attached spreadsheet.) This
> doesn't take ToR switch failure in account though. On the other hand - 150
> nodes is only ~5 racks - in such a scenario you might rather want to shut
> the system down completely rather than letting it replicate 20% of all data.
>
> Cheers,
> Evert

--000e0cd519f0cda22904a6f61c30--

From general-return-55-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 02 16:06:10 2008
Return-Path: <general-return-55-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 10736 invoked from network); 2 Oct 2008 16:06:10 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 2 Oct 2008 16:06:10 -0000
Received: (qmail 53281 invoked by uid 500); 2 Oct 2008 16:06:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52942 invoked by uid 500); 2 Oct 2008 16:06:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52847 invoked by uid 99); 2 Oct 2008 16:06:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Oct 2008 09:06:02 -0700
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Oct 2008 16:04:57 +0000
Received: from battlerock-lm.corp.yahoo.com (battlerock-lm.corp.yahoo.com [10.72.112.37])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id m92G3o8i068717;
	Thu, 2 Oct 2008 09:03:50 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:content-type:
	content-transfer-encoding:mime-version:subject:date:cc:x-mailer;
	b=cB8S+Ltt26xWkMJFqFOqxtQ5C3ac1kL/1FoU6sqlBmAqJpoayAW6q6BCt7jGtpDz
Message-Id: <57D7A94E-E310-4F65-82E6-858EBB1D14EE@yahoo-inc.com>
From: "Owen O'Malley" <oom@yahoo-inc.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v929.2)
Subject: Hadoop Camp next month
Date: Thu, 2 Oct 2008 09:03:50 -0700
Cc: core-user@hadoop.apache.org, zookeeper-user@hadoop.apache.org,
        hbase-user@hadoop.apache.org, pig-user@incubator.apache.org
X-Mailer: Apple Mail (2.929.2)
X-Virus-Checked: Checked by ClamAV on apache.org

Hi all,
   I'd like to remind everyone that the Hadoop Camp & ApacheCon US is  
coming up in New Orleans next month. http://tinyurl.com/hadoop-camp

It will be the largest gathering of Hadoop developers outside of  
California. We'll have:

Core: Doug Cutting, Dhruba Borthakur, Arun Murthy, Owen O'Malley,  
Sameer Paranjpye,
            Sanjay Radia, Tom White
Zookeeper: Ben Reed

There will also be a training session on Practical Problem Solving  
with Hadoop by Milind Bhandarkar on Monday.

So if you'd like to meet the developers or find out more about Hadoop,  
come join us!

-- Owen

From general-return-56-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 02 16:30:15 2008
Return-Path: <general-return-56-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 29843 invoked from network); 2 Oct 2008 16:30:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 2 Oct 2008 16:30:15 -0000
Received: (qmail 97382 invoked by uid 500); 2 Oct 2008 16:30:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97365 invoked by uid 500); 2 Oct 2008 16:30:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97354 invoked by uid 99); 2 Oct 2008 16:30:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Oct 2008 09:30:13 -0700
X-ASF-Spam-Status: No, hits=4.0 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_BL_SPAMCOP_NET,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of owen.omalley@gmail.com designates 209.85.200.173 as permitted sender)
Received: from [209.85.200.173] (HELO wf-out-1314.google.com) (209.85.200.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Oct 2008 16:29:08 +0000
Received: by wf-out-1314.google.com with SMTP id 24so1177456wfg.2
        for <general@hadoop.apache.org>; Thu, 02 Oct 2008 09:29:34 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:resent-to:from:to:resent-from
         :subject:references:message-id:content-type:resent-date:mime-version
         :date:cc:x-mailer:sender;
        bh=BNAPIMLdyEQarWnSgm4lrfTxUJW1YKgvifPZ2GnEQNQ=;
        b=Hq04usDVjSSxR9CgdTCGZmxhKJ5qdnB1WXquaOc2u0NMvg/KPLgNxbGiDa9TsYqSQb
         4H2V5lGr8SXG4n+iri40dFJu1wJZfjWCiZDzA5wvOBZqs+78jg/RXzhx7llBEQMOmvMo
         zZ/aPzA7s58CeBJS1vMW035jYlL70evLygJoc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=resent-to:from:to:resent-from:subject:references:message-id
         :content-type:resent-date:mime-version:date:cc:x-mailer:sender;
        b=JLSuyX8D9Uqxg22if2UEzfJJSg62ZsYWVraz3bziLASx/33khMTNTBcY9/Qlq6vdPx
         32wtbzDdnxMXwOUTbL4CUiJE/xvU8/SpvXNUOmbmfPE9qszJ19tNV5owangU3YbPQl9+
         JbtBUYxtY/MftYfWtBDAxdk43Mgugl+xKkzZ0=
Received: by 10.141.88.3 with SMTP id q3mr5484456rvl.3.1222964973116;
        Thu, 02 Oct 2008 09:29:33 -0700 (PDT)
Received: from battlerock-lm.corp.yahoo.com (nat-dip4.fw.corp.yahoo.com [209.131.62.113])
        by mx.google.com with ESMTPS id f42sm8372497rvb.6.2008.10.02.09.29.31
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Thu, 02 Oct 2008 09:29:32 -0700 (PDT)
Resent-To: general@hadoop.apache.org
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
Resent-From: Owen O'Malley <omalley@apache.org>
Subject: Fwd: CFP open for ApacheCon Europe 2009
References: <20081002082206.GA8370@infiltrator.gizzard.com>
Message-Id: <A07E8938-BCC7-4710-B04D-06FF1A23BFC5@apache.org>
Content-Type: multipart/alternative; boundary=Apple-Mail-6--155364869
Resent-Date: Thu, 2 Oct 2008 09:29:30 -0700
Mime-Version: 1.0 (Apple Message framework v929.2)
Date: Thu, 2 Oct 2008 08:59:39 -0700
Cc: core-user@hadoop.apache.org,
 zookeeper-user@hadoop.apache.org,
 hbase-user@hadoop.apache.org,
 pig-user@incubator.apache.org
X-Mailer: Apple Mail (2.929.2)
Sender: Owen O'Malley <owen.omalley@gmail.com>
X-Virus-Checked: Checked by ClamAV on apache.org
Resent-Message-Id: <20081002162919.A8A46724956@athena.apache.org>

--Apple-Mail-6--155364869
Content-Type: text/plain;
	charset=WINDOWS-1252;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: quoted-printable

All,
    The call for proposals for ApacheCon Europe 2009 is out. Note that =20=

no paper is called for, just a presentation. The description of the =20
presentation is due on 24 October.

-- Owen

Begin forwarded message:

> If you only have thirty seconds:
>
> The Call for Papers for ApacheCon Europe 2009, to be held in =20
> Amsterdam, from 23rd to 27th March, is now open! Submit your =20
> proposals at http://eu.apachecon.com/c/aceu2009/cfp/ before 24th =20
> October.
>
> Remember that early bird prices for ApacheCon US 2008, to be held in =20=

> New Orleans, from 3rd to 7th November, will go up this Friday, at =20
> midnight Eastern time!
>
> Sponsorship opportunities for ApacheCon US 2008 and ApacheCon EU =20
> 2009 are still available. If you or your company are interested in =20
> becoming a sponsor, please contact Delia Frees at =20
> delia@apachecon.com for details.
>
> *******
>
> If you want all the details:
>
> ApacheCon Europe 2009 - Leading the Wave of Open Source
> Amsterdam, The Netherlands
> 23rd to 27th March, 2009
>
> Call for Papers Opens for ApacheCon Europe 2009
>
> The Apache Software Foundation (ASF) invites submissions to its =20
> official conference, ApacheCon Europe 2009. To be held 23rd to 27th =20=

> March, 2009 at the M=F6venpick Hotel Amsterdam City Centre, ApacheCon =20=

> serves as a forum for showcasing the ASF's latest developments, =20
> including its projects, membership, and community. ApacheCon offers =20=

> unparalleled educational opportunities, with dedicated =20
> presentations, hands-on trainings, and sessions that address core =20
> technology, development, business/marketing, and licensing issues in =20=

> Open Source.
>
> ApacheCon's wide range of activities are designed to promote the =20
> exchange of ideas amongst ASF Members, innovators, developers, =20
> vendors, and users interested in the future of Open Source =20
> technology. The conference program includes competitively selected =20
> presentations, trainings/workshops, and a small number of invited =20
> speakers. All sessions undergo a peer review process by the =20
> ApacheCon Conference Planning team. The following information =20
> provides presentation category descriptions, and information about =20
> how to submit your
> proposal.
>
> Conference Themes and Topics
>
> APACHECON 2009 - LEADING THE WAVE OF OPEN SOURCE
>
> Building on the success of the last two years, we are excited to =20
> return to Amsterdam in 2009. We'll be continuing to offer our very =20
> popular two-day trainings, including certifications of completion =20
> for those who fulfill all the requirements of these trainings.
>
> The ASF comprises some of the most active and recognized developers =20=

> in the Open Source community. By bringing together the pioneers, =20
> developers, and users of flagship Open Source technologies, =20
> ApacheCon provides an influential platform for dialogue, between the =20=

> speaker and the audience, between project contributors and the =20
> community at large, traversing a wide range of ideas, expertise, and =20=

> personalities.
>
> ApacheCon welcomes submissions from like-minded delegates across =20
> many fields, geographic locations, and areas of development. The =20
> breadth and loosely-structured nature of the Apache community lends =20=

> itself to conference content that is also somewhat loosely-=20
> structured. Common themes of interest address groundbreaking =20
> technologies and emerging trends, successful practices (from =20
> development to deployment), and lessons learned (tips, tools, and =20
> tricks). In addition to technical content, ApacheCon invites =20
> Business Track submissions that address Open Source business, =20
> marketing, and legal/licensing issues.
>
> Topics appropriate for submission to this conference are manifold, =20
> and may include but are not restricted to:
>
> - Apache HTTP server topics such as installation, configuration, and =20=

> migration
> - ASF-wide projects such as Lucene, SpamAssassin, Jackrabbit, and =20
> Maven
> - Scripting languages and dynamic content such as Java, Perl, =20
> Python, Ruby, XSL, and PHP
> - Security and e-commerce
> - Performance tuning, load balancing and high availability
> - New technologies and broader initiatives such as Web Services and =20=

> Web 2.0
> - ASF-Incubated projects such as Sling, UIMA, and Shindig
>
>
> Submission Guidelines
> Submissions must include
> - Title
> - Speaker name, with affiliation and email address
> - Speaker bio (100 words or less)
> - Short description (50 words or less)
> - Full description including abstract and objectives (200 words or
> less)
> - Expertise level (beginner to advanced)
> - Format and duration (trainings vs. general presentation; half-, =20
> full- or two-day workshop, etc.)
> - Intended audience and maximum number of participants (trainings =20
> only)
> - Background knowledge expected of the participants (trainings only)
>
>
> Types of Presentations
>
> - Trainings/Workshops
> - General Sessions
> - Case Studies/Industry Profiles
> - Invited Keynotes/Panels/Speakers
> - Corporate Showcases & Demonstrations
>
> BoF sessions and Fast Feather Track talks will be selected separately
>
> Pre Conference Trainings/Workshops
> Held on the first and second day of the conference =96 2008-03-23 and =20=

> 2008-03-24, Trainings require a registration fee beyond the regular =20=

> conference fee. Proposals may be submitted for half-day (3 hours), =20
> full-day (6 hours), or two-day (12 hours) training sessions, aimed =20
> at providing in-depth, hands-on development experience or related =20
> continuing education. Training submissions are welcome at beginner, =20=

> intermediate, and expert levels.
>
> General Sessions
> General Sessions include presentations on practical development =20
> applications, insight into high-interest projects, best practices =20
> and key advances, overcoming implementation challenges, and industry =20=

> innovations. Especially welcome are submissions that extend =20
> participants' understanding the role of ASF projects and their =20
> influence on the Open Source community at large. General Sessions =20
> are scheduled for 50 minutes and are accessible to all conference =20
> delegates.
>
> Case Study/Industry Profile
> Practitioners are invited to submit presentations that focus on how =20=

> implementing particular ASF technologies led to improved products/=20
> solutions, service offerings, changes in work practices, among other =20=

> successes. Proposals that highlight overcoming interesting =20
> challenges in application design and developing innovative =20
> frameworks using multiple ASF projects are particularly encouraged. =20=

> NOTE: Marketing-oriented submissions aimed at promoting specific =20
> organizations or products will not be accepted.
>
> Invited Keynotes/Panels/Speakers
> Each conference the ApacheCon Planning team invites select =20
> presenters dealing with engaging, dialectical, and challenging =20
> subjects to present in keynote and/or panel formats. Topics include =20=

> cutting-edge technology development, industry leadership, hot or =20
> emerging trends, opinions on controversial issues, insight on =20
> technology paradigms, and contrasting viewpoints in complementary =20
> professional areas. Those interested in suggesting a candidate for =20
> an invited speaker opportunity should submit a brief proposal with =20
> the speaker's name, affiliation, background/bio, overview of topics =20=

> of interest, and contact information.
>
> Important Dates
> Please submit your completed proposal, at =
http://eu.apachecon.com/c/aceu2009/cfp/=20
>  before Friday, 24th October, 2008 Midnight GMT.
>
> About ApacheCon 2009
> ApacheCon is co-produced by the Apache Software Foundation and Stone =20=

> Circle Productions. The ApacheCon Planning team comprises ASF =20
> Members from all over the world working on a wholly-volunteer basis. =20=

> For more information, visit
> http://eu.apachecon.com/c/aceu2009/
>
>
> ----- End forwarded message -----


--Apple-Mail-6--155364869--

From general-return-57-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 06 23:35:21 2008
Return-Path: <general-return-57-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 74743 invoked from network); 6 Oct 2008 23:35:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 6 Oct 2008 23:35:21 -0000
Received: (qmail 39033 invoked by uid 500); 6 Oct 2008 23:35:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39012 invoked by uid 500); 6 Oct 2008 23:35:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 79145 invoked by uid 99); 6 Oct 2008 22:34:05 -0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=DNS_FROM_SECURITYSAGE,HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:x-mimeole:content-class:mime-version:
	content-type:subject:date:message-id:x-ms-has-attach:
	x-ms-tnef-correlator:thread-topic:thread-index:from:to:cc:return-path:x-originalarrivaltime;
	b=Z2ExU8eDUh7QyfRAQU/bMTgDXDzYbEHL7sHVbRw94ljIoxUJi7HLmUOERcyK4K8e
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C92803.7328F78D"
Subject: [VOTE] Pig graduation to hadoop subproject
Date: Mon, 6 Oct 2008 15:32:43 -0700
Message-ID: <236D5828CF7F5749AF1BCB7F87596DE1029AF8FA@SNV-EXVS09.ds.corp.yahoo.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: [VOTE] Pig graduation to hadoop subproject
Thread-Index: AckoA3LHD+ivlQt4Ro+OiDvaGryK2A==
From: "Olga Natkovich" <olgan@yahoo-inc.com>
To: <pig-dev@incubator.apache.org>
Cc: <general@incubator.apache.org>, <general@hadoop.apache.org>
X-OriginalArrivalTime: 06 Oct 2008 22:32:44.0741 (UTC) FILETIME=[73AB9F50:01C92803]
X-Virus-Checked: Checked by ClamAV on apache.org

------_=_NextPart_001_01C92803.7328F78D
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Pig Developers and Mentors,
=20
Pig has been incubating for over a year now. In this period of time, we
had extended our community with 2 new committers, had a release,
resolved 300 issues. We have made some significant code improvements
including pipeline redesign, addition of streaming and limit
functionality, grunt shell improvements and significant performance
speedup. We have a constant traffic and lively discussions on both
pig-dev and pig-user mailing lists and we conduct our business in the
open by publishing proposals and discussing them in the mailing lists.
=20
As of now, we have completed graduation requirements as described in
http://incubator.apache.org/guides/graduation.html.
=20
I would like to call for a graduation vote at this time.
=20
I would also propose that we graduate as a subproject of Hadoop. There
are several advantages to this approach. First, this would allow us to
extend both our user and developer base. Second, it would bring benefits
to the Hadoop community by providing a higher level interface and easier
entry point for new users. Third, having an established project to
provide guidance, would help Pig to become a mature participant in the
open source community.
=20
Please, vote by the end of the day on Thursday, 10/9.
=20
Thanks,
=20
Olga
=20
PS: I am ccing  hadoop and incubator general mailing lists; however, no
action is required from them at this time. This step is for Pig
community only.

------_=_NextPart_001_01C92803.7328F78D--

From general-return-58-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 06 23:35:31 2008
Return-Path: <general-return-58-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 74794 invoked from network); 6 Oct 2008 23:35:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 6 Oct 2008 23:35:30 -0000
Received: (qmail 39173 invoked by uid 500); 6 Oct 2008 23:35:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39156 invoked by uid 500); 6 Oct 2008 23:35:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 86885 invoked by uid 99); 6 Oct 2008 22:42:15 -0000
X-ASF-Spam-Status: No, hits=1.3 required=10.0
	tests=DNS_FROM_SECURITYSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:x-mimeole:content-class:mime-version:
	content-type:content-transfer-encoding:subject:date:message-id:
	in-reply-to:x-ms-has-attach:x-ms-tnef-correlator:thread-topic:
	thread-index:references:from:to:cc:return-path:x-originalarrivaltime;
	b=N7Xmlz5Klrg0/uEdAYmnPaYxHrJzwK7JXcrkyPYtmWqN64WLftF0/W8rjwXYs0ME
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable
Subject: RE: [VOTE] Pig graduation to hadoop subproject
Date: Mon, 6 Oct 2008 15:39:21 -0700
Message-ID: <088A0B616C8C1D4787DD686C6922A72AD86118@SNV-EXVS10.ds.corp.yahoo.com>
In-Reply-To: <236D5828CF7F5749AF1BCB7F87596DE1029AF8FA@SNV-EXVS09.ds.corp.yahoo.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: [VOTE] Pig graduation to hadoop subproject
Thread-Index: AckoA3LHD+ivlQt4Ro+OiDvaGryK2AAAOSeQ
References: <236D5828CF7F5749AF1BCB7F87596DE1029AF8FA@SNV-EXVS09.ds.corp.yahoo.com>
From: "Santhosh Srinivasan" <sms@yahoo-inc.com>
To: <pig-dev@incubator.apache.org>
Cc: <general@incubator.apache.org>, <general@hadoop.apache.org>
X-OriginalArrivalTime: 06 Oct 2008 22:40:19.0493 (UTC) FILETIME=[82B94550:01C92804]
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Santhosh=20

-----Original Message-----
From: Olga Natkovich [mailto:olgan@yahoo-inc.com]=20
Sent: Monday, October 06, 2008 3:33 PM
To: pig-dev@incubator.apache.org
Cc: general@incubator.apache.org; general@hadoop.apache.org
Subject: [VOTE] Pig graduation to hadoop subproject

Pig Developers and Mentors,
=20
Pig has been incubating for over a year now. In this period of time, we
had extended our community with 2 new committers, had a release,
resolved 300 issues. We have made some significant code improvements
including pipeline redesign, addition of streaming and limit
functionality, grunt shell improvements and significant performance
speedup. We have a constant traffic and lively discussions on both
pig-dev and pig-user mailing lists and we conduct our business in the
open by publishing proposals and discussing them in the mailing lists.
=20
As of now, we have completed graduation requirements as described in
http://incubator.apache.org/guides/graduation.html.
=20
I would like to call for a graduation vote at this time.
=20
I would also propose that we graduate as a subproject of Hadoop. There
are several advantages to this approach. First, this would allow us to
extend both our user and developer base. Second, it would bring benefits
to the Hadoop community by providing a higher level interface and easier
entry point for new users. Third, having an established project to
provide guidance, would help Pig to become a mature participant in the
open source community.
=20
Please, vote by the end of the day on Thursday, 10/9.
=20
Thanks,
=20
Olga
=20
PS: I am ccing  hadoop and incubator general mailing lists; however, no
action is required from them at this time. This step is for Pig
community only.

From general-return-59-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 06 23:35:41 2008
Return-Path: <general-return-59-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 74839 invoked from network); 6 Oct 2008 23:35:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 6 Oct 2008 23:35:41 -0000
Received: (qmail 39440 invoked by uid 500); 6 Oct 2008 23:35:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39325 invoked by uid 500); 6 Oct 2008 23:35:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 20556 invoked by uid 99); 6 Oct 2008 23:11:38 -0000
X-ASF-Spam-Status: No, hits=0.1 required=10.0
	tests=DNS_FROM_SECURITYSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of luckbr1975@gmail.com designates 72.14.220.152 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:to
         :subject:cc:mime-version:content-type:content-transfer-encoding
         :content-disposition;
        bh=fEVt6DsHwR+qEcw9g+umAuekU5ZqfOYUchZ6zOogQEA=;
        b=jwSpTgGalBYOCsJ2YtmK+1AwrqzaGOMB8+l3mwEHy2uXTB4eE52H0gL0QTWrW5UFuJ
         g11nQw1mxzvI1F08/SS6EP0K01hKTcN4U52QUYi5XdtnkH10IzYz8/K7NxsBqz/oZ7E5
         j6fQivKPosgLfw8PV4ZZhQ4GLpL1sqEkaDxBI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:cc:mime-version:content-type
         :content-transfer-encoding:content-disposition;
        b=vbeEiU7ynttBZtWIJ44mFg7kSJaWqets1Nr2anotVQp4gEmDzekbCFkNTOcctyc9Jc
         RVKdpYiAjGQrqWr/gZwrCD3yi/TwetOOGD3H9OeSBj5y6qErkd5SYj4cEDlsLpxDrRVK
         gUnNCa/3+x6WRB+VaJXrG5N8Y5IKsy1UGA0TE=
Message-ID: <5a75db780810061611x2651cbf6p41bf8e41d73675e8@mail.gmail.com>
Date: Mon, 6 Oct 2008 16:11:11 -0700
From: "Luciano Resende" <luckbr1975@gmail.com>
To: general@incubator.apache.org
Subject: Graduation Checklist , was Re: [VOTE] Pig graduation to hadoop subproject
Cc: pig-dev@incubator.apache.org, general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
X-Virus-Checked: Checked by ClamAV on apache.org

On Mon, Oct 6, 2008 at 3:32 PM, Olga Natkovich <olgan@yahoo-inc.com> wrote:
> Pig Developers and Mentors,
>
> Pig has been incubating for over a year now. In this period of time, we
> had extended our community with 2 new committers,

Has this list been updated recently ? Does it reflect the current list
of committers ?
[1] http://incubator.apache.org/pig/whoweare.html

>
> As of now, we have completed graduation requirements as described in
> http://incubator.apache.org/guides/graduation.html.
>

Have you guys looked at the "Creating an Open and Diverse community"
section on the graduation checklist ?



-- 
Luciano Resende
Apache Tuscany, Apache PhotArk
http://people.apache.org/~lresende
http://lresende.blogspot.com/

From general-return-60-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 06 23:36:04 2008
Return-Path: <general-return-60-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 74935 invoked from network); 6 Oct 2008 23:36:04 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 6 Oct 2008 23:36:04 -0000
Received: (qmail 39518 invoked by uid 500); 6 Oct 2008 23:36:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39503 invoked by uid 500); 6 Oct 2008 23:36:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 28375 invoked by uid 99); 6 Oct 2008 23:25:52 -0000
X-ASF-Spam-Status: No, hits=1.3 required=10.0
	tests=DNS_FROM_SECURITYSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:x-mimeole:content-class:mime-version:
	content-type:content-transfer-encoding:subject:date:message-id:
	in-reply-to:x-ms-has-attach:x-ms-tnef-correlator:thread-topic:
	thread-index:references:from:to:cc:return-path:x-originalarrivaltime;
	b=Yy1fd0xvq+uBnP5elRarrIcqSvXYzP3cc+ZnVJaIzHCA6Ft0Lf6zaiqz3OOFqjgR
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: RE: Graduation Checklist , was Re: [VOTE] Pig graduation to hadoop subproject
Date: Mon, 6 Oct 2008 16:23:48 -0700
Message-ID: <236D5828CF7F5749AF1BCB7F87596DE1029AF90B@SNV-EXVS09.ds.corp.yahoo.com>
In-Reply-To: <5a75db780810061611x2651cbf6p41bf8e41d73675e8@mail.gmail.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Graduation Checklist , was Re: [VOTE] Pig graduation to hadoop subproject
Thread-Index: AckoCRzumdo8rIGfSZGY0mAZEymE4QAARdKw
References: <5a75db780810061611x2651cbf6p41bf8e41d73675e8@mail.gmail.com>
From: "Olga Natkovich" <olgan@yahoo-inc.com>
To: <general@incubator.apache.org>
Cc: <pig-dev@incubator.apache.org>, <general@hadoop.apache.org>
X-OriginalArrivalTime: 06 Oct 2008 23:23:50.0040 (UTC) FILETIME=[96BB2180:01C9280A]
X-Virus-Checked: Checked by ClamAV on apache.org

=20

> -----Original Message-----
> From: Luciano Resende [mailto:luckbr1975@gmail.com]=20
> Sent: Monday, October 06, 2008 4:11 PM
> To: general@incubator.apache.org
> Cc: pig-dev@incubator.apache.org; general@hadoop.apache.org
> Subject: Graduation Checklist , was Re: [VOTE] Pig graduation=20
> to hadoop subproject
>=20
> On Mon, Oct 6, 2008 at 3:32 PM, Olga Natkovich=20
> <olgan@yahoo-inc.com> wrote:
> > Pig Developers and Mentors,
> >
> > Pig has been incubating for over a year now. In this period=20
> of time,=20
> > we had extended our community with 2 new committers,
>=20
> Has this list been updated recently ? Does it reflect the=20
> current list of committers ?
> [1] http://incubator.apache.org/pig/whoweare.html

Yes, the document is up-to-date.

>=20
> >
> > As of now, we have completed graduation requirements as=20
> described in=20
> > http://incubator.apache.org/guides/graduation.html.
> >
>=20
> Have you guys looked at the "Creating an Open and Diverse community"
> section on the graduation checklist ?

Yes, and I think we satisfied these requirements.

>=20
>=20
>=20
> --
> Luciano Resende
> Apache Tuscany, Apache PhotArk
> http://people.apache.org/~lresende
> http://lresende.blogspot.com/
>=20
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>=20
>=20

From general-return-61-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 07 01:18:15 2008
Return-Path: <general-return-61-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 29951 invoked from network); 7 Oct 2008 01:18:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 7 Oct 2008 01:18:15 -0000
Received: (qmail 43710 invoked by uid 500); 7 Oct 2008 01:18:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 43592 invoked by uid 500); 7 Oct 2008 01:18:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 43581 invoked by uid 99); 7 Oct 2008 01:18:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Oct 2008 18:18:13 -0700
X-ASF-Spam-Status: No, hits=2.1 required=10.0
	tests=DNS_FROM_SECURITYSAGE,HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of tang9527@gmail.com designates 74.125.46.30 as permitted sender)
Received: from [74.125.46.30] (HELO yw-out-2324.google.com) (74.125.46.30)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Oct 2008 01:17:11 +0000
Received: by yw-out-2324.google.com with SMTP id 9so526821ywe.29
        for <general@hadoop.apache.org>; Mon, 06 Oct 2008 18:17:29 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:to
         :subject:cc:in-reply-to:mime-version:content-type:references;
        bh=nU2HUMztGDqYFrq0YKj00IUAn3Aoa02S5UN2OxUgBCc=;
        b=n5b/5d4k8hCUetjr9AqH4Pwr3WIaMp9x5Xte4qD+J/YYA6+QRd2BGx/8e5E4YQeQRf
         Sl1txS9G1niFuu1lEFDy/NnOU7kVFxUe6KXsnyK0cH5Fs46TJ+A/LcQ3F7fJF+ARABgK
         Pk1s1px3COnjhdXneriQtVknNdHK7bWmNjBAs=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:cc:in-reply-to:mime-version
         :content-type:references;
        b=ex40uPzZvQW3//Up57tK/6au13SZDBbjf+5Ym2hCLGp8wVgzkCmhstV1sAad4IFRrv
         9Yaw2OM/198ADeZ5ieu/DEhtZGUr8u2WFecQdmWDZzFZ7daLxQzxyhxo+sHjuYa+VLpT
         wHqznMpIxJFdn5YE81QuIy1pZbCoOmM/MSebE=
Received: by 10.110.26.20 with SMTP id 20mr8237026tiz.29.1223342248227;
        Mon, 06 Oct 2008 18:17:28 -0700 (PDT)
Received: by 10.110.17.1 with HTTP; Mon, 6 Oct 2008 18:17:28 -0700 (PDT)
Message-ID: <98fa49780810061817r2a471964n8bf43d9d88c64651@mail.gmail.com>
Date: Tue, 7 Oct 2008 09:17:28 +0800
From: tang9527@gmail.com
To: general@hadoop.apache.org
Subject: Re: [VOTE] Pig graduation to hadoop subproject
Cc: pig-dev@incubator.apache.org, general@incubator.apache.org
In-Reply-To: <088A0B616C8C1D4787DD686C6922A72AD86118@SNV-EXVS10.ds.corp.yahoo.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; 
	boundary="----=_Part_66805_29733018.1223342248224"
References: <236D5828CF7F5749AF1BCB7F87596DE1029AF8FA@SNV-EXVS09.ds.corp.yahoo.com>
	 <088A0B616C8C1D4787DD686C6922A72AD86118@SNV-EXVS10.ds.corp.yahoo.com>
X-Virus-Checked: Checked by ClamAV on apache.org

------=_Part_66805_29733018.1223342248224
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

+1

2008/10/7 Santhosh Srinivasan <sms@yahoo-inc.com>

> +1
>
> Santhosh
>
> -----Original Message-----
> From: Olga Natkovich [mailto:olgan@yahoo-inc.com]
> Sent: Monday, October 06, 2008 3:33 PM
> To: pig-dev@incubator.apache.org
> Cc: general@incubator.apache.org; general@hadoop.apache.org
> Subject: [VOTE] Pig graduation to hadoop subproject
>
> Pig Developers and Mentors,
>
> Pig has been incubating for over a year now. In this period of time, we
> had extended our community with 2 new committers, had a release,
> resolved 300 issues. We have made some significant code improvements
> including pipeline redesign, addition of streaming and limit
> functionality, grunt shell improvements and significant performance
> speedup. We have a constant traffic and lively discussions on both
> pig-dev and pig-user mailing lists and we conduct our business in the
> open by publishing proposals and discussing them in the mailing lists.
>
> As of now, we have completed graduation requirements as described in
> http://incubator.apache.org/guides/graduation.html.
>
> I would like to call for a graduation vote at this time.
>
> I would also propose that we graduate as a subproject of Hadoop. There
> are several advantages to this approach. First, this would allow us to
> extend both our user and developer base. Second, it would bring benefits
> to the Hadoop community by providing a higher level interface and easier
> entry point for new users. Third, having an established project to
> provide guidance, would help Pig to become a mature participant in the
> open source community.
>
> Please, vote by the end of the day on Thursday, 10/9.
>
> Thanks,
>
> Olga
>
> PS: I am ccing  hadoop and incubator general mailing lists; however, no
> action is required from them at this time. This step is for Pig
> community only.
>



-- 
jim

------=_Part_66805_29733018.1223342248224--

From general-return-62-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 07 01:18:51 2008
Return-Path: <general-return-62-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 30309 invoked from network); 7 Oct 2008 01:18:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 7 Oct 2008 01:18:51 -0000
Received: (qmail 43783 invoked by uid 500); 7 Oct 2008 01:18:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 43749 invoked by uid 500); 7 Oct 2008 01:18:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 43736 invoked by uid 99); 7 Oct 2008 01:18:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Oct 2008 18:18:50 -0700
X-ASF-Spam-Status: No, hits=2.1 required=10.0
	tests=DNS_FROM_SECURITYSAGE,HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of tang9527@gmail.com designates 74.125.46.30 as permitted sender)
Received: from [74.125.46.30] (HELO yw-out-2324.google.com) (74.125.46.30)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Oct 2008 01:17:48 +0000
Received: by yw-out-2324.google.com with SMTP id 9so526821ywe.29
        for <general@hadoop.apache.org>; Mon, 06 Oct 2008 18:18:23 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:to
         :subject:cc:in-reply-to:mime-version:content-type:references;
        bh=nU2HUMztGDqYFrq0YKj00IUAn3Aoa02S5UN2OxUgBCc=;
        b=n5b/5d4k8hCUetjr9AqH4Pwr3WIaMp9x5Xte4qD+J/YYA6+QRd2BGx/8e5E4YQeQRf
         Sl1txS9G1niFuu1lEFDy/NnOU7kVFxUe6KXsnyK0cH5Fs46TJ+A/LcQ3F7fJF+ARABgK
         Pk1s1px3COnjhdXneriQtVknNdHK7bWmNjBAs=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:cc:in-reply-to:mime-version
         :content-type:references;
        b=ex40uPzZvQW3//Up57tK/6au13SZDBbjf+5Ym2hCLGp8wVgzkCmhstV1sAad4IFRrv
         9Yaw2OM/198ADeZ5ieu/DEhtZGUr8u2WFecQdmWDZzFZ7daLxQzxyhxo+sHjuYa+VLpT
         wHqznMpIxJFdn5YE81QuIy1pZbCoOmM/MSebE=
Received: by 10.110.26.20 with SMTP id 20mr8237026tiz.29.1223342248227;
        Mon, 06 Oct 2008 18:17:28 -0700 (PDT)
Received: by 10.110.17.1 with HTTP; Mon, 6 Oct 2008 18:17:28 -0700 (PDT)
Message-ID: <98fa49780810061817r2a471964n8bf43d9d88c64651@mail.gmail.com>
Date: Tue, 7 Oct 2008 09:17:28 +0800
From: tang9527@gmail.com
To: general@hadoop.apache.org
Subject: Re: [VOTE] Pig graduation to hadoop subproject
Cc: pig-dev@incubator.apache.org, general@incubator.apache.org
In-Reply-To: <088A0B616C8C1D4787DD686C6922A72AD86118@SNV-EXVS10.ds.corp.yahoo.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; 
	boundary="----=_Part_66805_29733018.1223342248224"
References: <236D5828CF7F5749AF1BCB7F87596DE1029AF8FA@SNV-EXVS09.ds.corp.yahoo.com>
	 <088A0B616C8C1D4787DD686C6922A72AD86118@SNV-EXVS10.ds.corp.yahoo.com>
X-Virus-Checked: Checked by ClamAV on apache.org

------=_Part_66805_29733018.1223342248224
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

+1

2008/10/7 Santhosh Srinivasan <sms@yahoo-inc.com>

> +1
>
> Santhosh
>
> -----Original Message-----
> From: Olga Natkovich [mailto:olgan@yahoo-inc.com]
> Sent: Monday, October 06, 2008 3:33 PM
> To: pig-dev@incubator.apache.org
> Cc: general@incubator.apache.org; general@hadoop.apache.org
> Subject: [VOTE] Pig graduation to hadoop subproject
>
> Pig Developers and Mentors,
>
> Pig has been incubating for over a year now. In this period of time, we
> had extended our community with 2 new committers, had a release,
> resolved 300 issues. We have made some significant code improvements
> including pipeline redesign, addition of streaming and limit
> functionality, grunt shell improvements and significant performance
> speedup. We have a constant traffic and lively discussions on both
> pig-dev and pig-user mailing lists and we conduct our business in the
> open by publishing proposals and discussing them in the mailing lists.
>
> As of now, we have completed graduation requirements as described in
> http://incubator.apache.org/guides/graduation.html.
>
> I would like to call for a graduation vote at this time.
>
> I would also propose that we graduate as a subproject of Hadoop. There
> are several advantages to this approach. First, this would allow us to
> extend both our user and developer base. Second, it would bring benefits
> to the Hadoop community by providing a higher level interface and easier
> entry point for new users. Third, having an established project to
> provide guidance, would help Pig to become a mature participant in the
> open source community.
>
> Please, vote by the end of the day on Thursday, 10/9.
>
> Thanks,
>
> Olga
>
> PS: I am ccing  hadoop and incubator general mailing lists; however, no
> action is required from them at this time. This step is for Pig
> community only.
>



-- 
jim

------=_Part_66805_29733018.1223342248224--

From general-return-63-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 07 16:32:32 2008
Return-Path: <general-return-63-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 29137 invoked from network); 7 Oct 2008 16:32:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 7 Oct 2008 16:32:32 -0000
Received: (qmail 79809 invoked by uid 500); 7 Oct 2008 16:32:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79793 invoked by uid 500); 7 Oct 2008 16:32:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 52501 invoked by uid 99); 7 Oct 2008 16:12:21 -0000
X-ASF-Spam-Status: No, hits=1.3 required=10.0
	tests=DNS_FROM_SECURITYSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:x-mimeole:content-class:mime-version:
	content-type:content-transfer-encoding:subject:date:message-id:
	in-reply-to:x-ms-has-attach:x-ms-tnef-correlator:thread-topic:
	thread-index:references:from:to:cc:return-path:x-originalarrivaltime;
	b=vXwK6sBV0blb3oj2SpwJdEp6Q1YClo0mutppczUsYesOgRi5FStbLqSATEgkhOeW
x-mimeole: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: RE: [VOTE] Pig graduation to hadoop subproject
Date: Tue, 7 Oct 2008 09:11:48 -0700
Message-ID: <11ED50FC7C760F4EA9F44109C531617D019501BA@SNV-EXVS06.ds.corp.yahoo.com>
In-Reply-To: <98fa49780810061817r2a471964n8bf43d9d88c64651@mail.gmail.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: [VOTE] Pig graduation to hadoop subproject
Thread-Index: AckokWODheRuxH8QRnWAs2HP+eM8fwABf+2w
References: <236D5828CF7F5749AF1BCB7F87596DE1029AF8FA@SNV-EXVS09.ds.corp.yahoo.com> <088A0B616C8C1D4787DD686C6922A72AD86118@SNV-EXVS10.ds.corp.yahoo.com> <98fa49780810061817r2a471964n8bf43d9d88c64651@mail.gmail.com>
From: "Pradeep Kamath" <pradeepk@yahoo-inc.com>
To: <pig-dev@incubator.apache.org>, <general@hadoop.apache.org>
Cc: <general@incubator.apache.org>
X-OriginalArrivalTime: 07 Oct 2008 16:11:15.0121 (UTC) FILETIME=[52CE8A10:01C92897]
X-Virus-Checked: Checked by ClamAV on apache.org

+1

-----Original Message-----
From: tang9527@gmail.com [mailto:tang9527@gmail.com]=20
Sent: Monday, October 06, 2008 6:17 PM
To: general@hadoop.apache.org
Cc: pig-dev@incubator.apache.org; general@incubator.apache.org
Subject: Re: [VOTE] Pig graduation to hadoop subproject

+1

2008/10/7 Santhosh Srinivasan <sms@yahoo-inc.com>

> +1
>
> Santhosh
>
> -----Original Message-----
> From: Olga Natkovich [mailto:olgan@yahoo-inc.com]
> Sent: Monday, October 06, 2008 3:33 PM
> To: pig-dev@incubator.apache.org
> Cc: general@incubator.apache.org; general@hadoop.apache.org
> Subject: [VOTE] Pig graduation to hadoop subproject
>
> Pig Developers and Mentors,
>
> Pig has been incubating for over a year now. In this period of time,
we
> had extended our community with 2 new committers, had a release,
> resolved 300 issues. We have made some significant code improvements
> including pipeline redesign, addition of streaming and limit
> functionality, grunt shell improvements and significant performance
> speedup. We have a constant traffic and lively discussions on both
> pig-dev and pig-user mailing lists and we conduct our business in the
> open by publishing proposals and discussing them in the mailing lists.
>
> As of now, we have completed graduation requirements as described in
> http://incubator.apache.org/guides/graduation.html.
>
> I would like to call for a graduation vote at this time.
>
> I would also propose that we graduate as a subproject of Hadoop. There
> are several advantages to this approach. First, this would allow us to
> extend both our user and developer base. Second, it would bring
benefits
> to the Hadoop community by providing a higher level interface and
easier
> entry point for new users. Third, having an established project to
> provide guidance, would help Pig to become a mature participant in the
> open source community.
>
> Please, vote by the end of the day on Thursday, 10/9.
>
> Thanks,
>
> Olga
>
> PS: I am ccing  hadoop and incubator general mailing lists; however,
no
> action is required from them at this time. This step is for Pig
> community only.
>



--=20
jim

From general-return-64-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 07 16:33:28 2008
Return-Path: <general-return-64-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 29400 invoked from network); 7 Oct 2008 16:33:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 7 Oct 2008 16:33:28 -0000
Received: (qmail 80764 invoked by uid 500); 7 Oct 2008 16:33:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80744 invoked by uid 500); 7 Oct 2008 16:33:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 95932 invoked by uid 99); 7 Oct 2008 04:14:53 -0000
X-ASF-Spam-Status: No, hits=0.1 required=10.0
	tests=DNS_FROM_SECURITYSAGE,FORGED_MUA_OUTLOOK,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of daijyc@gmail.com designates 74.125.44.28 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:from:to:cc
         :references:subject:date:mime-version:content-type
         :content-transfer-encoding:x-priority:x-msmail-priority:x-mailer
         :x-mimeole;
        bh=qKI4H0tukL5kbkxGXcdH/TJeX5hM5p/oyU3kFx9YY/4=;
        b=eUBs6hzqB3lXTdJFXM+4pwOxbPhUlAfm40VVLPpWJa3FeA0lU605cfcyqQUZFMUVnR
         KAE4MjEKW4l3NPGowAIjiXlMIAWOvGePoqrF0AeLsi+sMfmVXfHREgG9q2fvmwIF3uUL
         6VZQW95twQ6s8tilCyn/qzFLv9bjACEY6+Jlk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:from:to:cc:references:subject:date:mime-version
         :content-type:content-transfer-encoding:x-priority:x-msmail-priority
         :x-mailer:x-mimeole;
        b=Drz11jtJor0EIaTYrBivheGmOkKP/XObnl015uX9MTaD1k4LP/QREhT5t7b9qE5/VI
         4dA11dqvNBWWHTheUzSoAU4nfmuPhG1w6PiU0y2eqQCAvZELoX08Flxe8VTyDv/NBIOx
         pGYvej6Pmq6iL5e2xiM3AEtbDdMh4XfDd9oB0=
Message-ID: <A9FE3A5B5C22495E9EAD6F93EA58FDA5@daijypc>
From: "Daniel" <daijyc@gmail.com>
To: <pig-dev@incubator.apache.org>
Cc: <general@incubator.apache.org>,
	<general@hadoop.apache.org>
References: <236D5828CF7F5749AF1BCB7F87596DE1029AF8FA@SNV-EXVS09.ds.corp.yahoo.com>
Subject: Re: [VOTE] Pig graduation to hadoop subproject
Date: Tue, 7 Oct 2008 00:14:03 -0400
MIME-Version: 1.0
Content-Type: text/plain;
	format=flowed;
	charset="iso-8859-1";
	reply-type=original
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2900.5512
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5579
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Daniel

----- Original Message ----- 
From: "Olga Natkovich" <olgan@yahoo-inc.com>
To: <pig-dev@incubator.apache.org>
Cc: <general@incubator.apache.org>; <general@hadoop.apache.org>
Sent: Monday, October 06, 2008 6:32 PM
Subject: [VOTE] Pig graduation to hadoop subproject


Pig Developers and Mentors,
 
Pig has been incubating for over a year now. In this period of time, we
had extended our community with 2 new committers, had a release,
resolved 300 issues. We have made some significant code improvements
including pipeline redesign, addition of streaming and limit
functionality, grunt shell improvements and significant performance
speedup. We have a constant traffic and lively discussions on both
pig-dev and pig-user mailing lists and we conduct our business in the
open by publishing proposals and discussing them in the mailing lists.
 
As of now, we have completed graduation requirements as described in
http://incubator.apache.org/guides/graduation.html.
 
I would like to call for a graduation vote at this time.
 
I would also propose that we graduate as a subproject of Hadoop. There
are several advantages to this approach. First, this would allow us to
extend both our user and developer base. Second, it would bring benefits
to the Hadoop community by providing a higher level interface and easier
entry point for new users. Third, having an established project to
provide guidance, would help Pig to become a mature participant in the
open source community.
 
Please, vote by the end of the day on Thursday, 10/9.
 
Thanks,
 
Olga
 
PS: I am ccing  hadoop and incubator general mailing lists; however, no
action is required from them at this time. This step is for Pig
community only.


From general-return-65-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 07 22:33:12 2008
Return-Path: <general-return-65-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 30423 invoked from network); 7 Oct 2008 22:33:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 7 Oct 2008 22:33:07 -0000
Received: (qmail 35548 invoked by uid 500); 7 Oct 2008 22:33:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 35523 invoked by uid 500); 7 Oct 2008 22:33:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 35293 invoked by uid 99); 7 Oct 2008 22:33:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Oct 2008 15:33:03 -0700
X-ASF-Spam-Status: No, hits=1.3 required=10.0
	tests=DNS_FROM_SECURITYSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Oct 2008 22:31:57 +0000
Received: from wlan-c-214-208.corp.yahoo.com (snvvpn1-10-72-72-c89.corp.yahoo.com [10.72.72.89])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id m97MV1Lq042120;
	Tue, 7 Oct 2008 15:31:01 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=cc:message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=D3xOLQc3dXo1Zpyt9KE6A0QudwjdDtlxCnr3k7y5Y4K+TT9Ab0weGBS4M3ix/KQT
Cc: general@hadoop.apache.org, general@incubator.apache.org
Message-Id: <C094649F-6900-40A3-A066-325CF83AE5F0@yahoo-inc.com>
From: Nigel Daley <ndaley@yahoo-inc.com>
To: pig-dev@incubator.apache.org
In-Reply-To: <236D5828CF7F5749AF1BCB7F87596DE1029AF8FA@SNV-EXVS09.ds.corp.yahoo.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v929.2)
Subject: Re: [VOTE] Pig graduation to hadoop subproject
Date: Tue, 7 Oct 2008 15:31:00 -0700
References: <236D5828CF7F5749AF1BCB7F87596DE1029AF8FA@SNV-EXVS09.ds.corp.yahoo.com>
X-Mailer: Apple Mail (2.929.2)
X-Virus-Checked: Checked by ClamAV on apache.org

+1

On Oct 6, 2008, at 3:32 PM, Olga Natkovich wrote:

> Pig Developers and Mentors,
>
> Pig has been incubating for over a year now. In this period of time,  
> we
> had extended our community with 2 new committers, had a release,
> resolved 300 issues. We have made some significant code improvements
> including pipeline redesign, addition of streaming and limit
> functionality, grunt shell improvements and significant performance
> speedup. We have a constant traffic and lively discussions on both
> pig-dev and pig-user mailing lists and we conduct our business in the
> open by publishing proposals and discussing them in the mailing lists.
>
> As of now, we have completed graduation requirements as described in
> http://incubator.apache.org/guides/graduation.html.
>
> I would like to call for a graduation vote at this time.
>
> I would also propose that we graduate as a subproject of Hadoop. There
> are several advantages to this approach. First, this would allow us to
> extend both our user and developer base. Second, it would bring  
> benefits
> to the Hadoop community by providing a higher level interface and  
> easier
> entry point for new users. Third, having an established project to
> provide guidance, would help Pig to become a mature participant in the
> open source community.
>
> Please, vote by the end of the day on Thursday, 10/9.
>
> Thanks,
>
> Olga
>
> PS: I am ccing  hadoop and incubator general mailing lists; however,  
> no
> action is required from them at this time. This step is for Pig
> community only.


From general-return-66-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 09 18:48:58 2008
Return-Path: <general-return-66-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 88654 invoked from network); 9 Oct 2008 18:48:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 9 Oct 2008 18:48:58 -0000
Received: (qmail 46306 invoked by uid 500); 9 Oct 2008 18:48:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46163 invoked by uid 500); 9 Oct 2008 18:48:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46107 invoked by uid 99); 9 Oct 2008 18:48:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Oct 2008 11:48:57 -0700
X-ASF-Spam-Status: No, hits=1.3 required=10.0
	tests=DNS_FROM_SECURITYSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Oct 2008 18:47:50 +0000
Received: from [192.168.0.198] (snvvpn2-10-72-76-c52.corp.yahoo.com [10.72.76.52])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id m99IjT9Z064599;
	Thu, 9 Oct 2008 11:45:29 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=in-reply-to:references:mime-version:content-type:message-id:cc:
	content-transfer-encoding:from:subject:date:to:x-mailer;
	b=NVO0ssXzyrR1PF8FuAxcx7y1rvE13rJ+n+F0XWaj5BTX5X1/+Ye88pOqz9XTzvUR
In-Reply-To: <236D5828CF7F5749AF1BCB7F87596DE1029AF8FA@SNV-EXVS09.ds.corp.yahoo.com>
References: <236D5828CF7F5749AF1BCB7F87596DE1029AF8FA@SNV-EXVS09.ds.corp.yahoo.com>
Mime-Version: 1.0 (Apple Message framework v753.1)
Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed
Message-Id: <390EBE2A-F32B-4DBC-AA8C-2AB63ED89935@yahoo-inc.com>
Cc: <pig-dev@incubator.apache.org>, <general@incubator.apache.org>
Content-Transfer-Encoding: 7bit
From: Alan Gates <gates@yahoo-inc.com>
Subject: Re: [VOTE] Pig graduation to hadoop subproject
Date: Thu, 9 Oct 2008 11:44:48 -0700
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.753.1)
X-Virus-Checked: Checked by ClamAV on apache.org

+1.

Alan.
On Oct 6, 2008, at 3:32 PM, Olga Natkovich wrote:

> Pig Developers and Mentors,
>
> Pig has been incubating for over a year now. In this period of  
> time, we
> had extended our community with 2 new committers, had a release,
> resolved 300 issues. We have made some significant code improvements
> including pipeline redesign, addition of streaming and limit
> functionality, grunt shell improvements and significant performance
> speedup. We have a constant traffic and lively discussions on both
> pig-dev and pig-user mailing lists and we conduct our business in the
> open by publishing proposals and discussing them in the mailing lists.
>
> As of now, we have completed graduation requirements as described in
> http://incubator.apache.org/guides/graduation.html.
>
> I would like to call for a graduation vote at this time.
>
> I would also propose that we graduate as a subproject of Hadoop. There
> are several advantages to this approach. First, this would allow us to
> extend both our user and developer base. Second, it would bring  
> benefits
> to the Hadoop community by providing a higher level interface and  
> easier
> entry point for new users. Third, having an established project to
> provide guidance, would help Pig to become a mature participant in the
> open source community.
>
> Please, vote by the end of the day on Thursday, 10/9.
>
> Thanks,
>
> Olga
>
> PS: I am ccing  hadoop and incubator general mailing lists;  
> however, no
> action is required from them at this time. This step is for Pig
> community only.


From general-return-67-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 09 20:20:16 2008
Return-Path: <general-return-67-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 22764 invoked from network); 9 Oct 2008 20:20:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 9 Oct 2008 20:20:16 -0000
Received: (qmail 47769 invoked by uid 500); 9 Oct 2008 20:20:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47711 invoked by uid 500); 9 Oct 2008 20:20:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47668 invoked by uid 99); 9 Oct 2008 20:20:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Oct 2008 13:20:14 -0700
X-ASF-Spam-Status: No, hits=1.3 required=10.0
	tests=DNS_FROM_SECURITYSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Oct 2008 20:19:07 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id m99KHAtc035568;
	Thu, 9 Oct 2008 13:17:10 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=cc:message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=llxuGBUWLVvlLrG9G9H6pG2sTVOP5FjNvCDZ1dc+5Z/ODdeh4EQol30IurwVY3yT
Cc: <general@incubator.apache.org>, <general@hadoop.apache.org>
Message-Id: <D52C5A69-1DAD-4302-A89D-D2298798F020@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: pig-dev@incubator.apache.org
In-Reply-To: <236D5828CF7F5749AF1BCB7F87596DE1029AF8FA@SNV-EXVS09.ds.corp.yahoo.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v929.2)
Subject: Re: [VOTE] Pig graduation to hadoop subproject
Date: Thu, 9 Oct 2008 13:17:10 -0700
References: <236D5828CF7F5749AF1BCB7F87596DE1029AF8FA@SNV-EXVS09.ds.corp.yahoo.com>
X-Mailer: Apple Mail (2.929.2)
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Arun

On Oct 6, 2008, at 3:32 PM, Olga Natkovich wrote:

> Pig Developers and Mentors,
>
> Pig has been incubating for over a year now. In this period of time,  
> we
> had extended our community with 2 new committers, had a release,
> resolved 300 issues. We have made some significant code improvements
> including pipeline redesign, addition of streaming and limit
> functionality, grunt shell improvements and significant performance
> speedup. We have a constant traffic and lively discussions on both
> pig-dev and pig-user mailing lists and we conduct our business in the
> open by publishing proposals and discussing them in the mailing lists.
>
> As of now, we have completed graduation requirements as described in
> http://incubator.apache.org/guides/graduation.html.
>
> I would like to call for a graduation vote at this time.
>
> I would also propose that we graduate as a subproject of Hadoop. There
> are several advantages to this approach. First, this would allow us to
> extend both our user and developer base. Second, it would bring  
> benefits
> to the Hadoop community by providing a higher level interface and  
> easier
> entry point for new users. Third, having an established project to
> provide guidance, would help Pig to become a mature participant in the
> open source community.
>
> Please, vote by the end of the day on Thursday, 10/9.
>
> Thanks,
>
> Olga
>
> PS: I am ccing  hadoop and incubator general mailing lists; however,  
> no
> action is required from them at this time. This step is for Pig
> community only.


From general-return-68-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 09 21:04:49 2008
Return-Path: <general-return-68-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 37044 invoked from network); 9 Oct 2008 21:04:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 9 Oct 2008 21:04:49 -0000
Received: (qmail 8693 invoked by uid 500); 9 Oct 2008 21:04:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8571 invoked by uid 500); 9 Oct 2008 21:04:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 37540 invoked by uid 99); 9 Oct 2008 18:43:54 -0000
X-ASF-Spam-Status: No, hits=1.3 required=10.0
	tests=DNS_FROM_SECURITYSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:x-mimeole:content-class:mime-version:
	content-type:content-transfer-encoding:subject:date:message-id:
	in-reply-to:x-ms-has-attach:x-ms-tnef-correlator:thread-topic:
	thread-index:references:from:to:return-path:x-originalarrivaltime;
	b=U1FOGzuIZtQ+ZKzkDNhCgO/hjQ39l1eFmisDrnfoDAESKrvS0nYd1lG3ETpHYxU3
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable
Subject: Hadoop User Group (Bay Area) Oct 15th
Date: Thu, 9 Oct 2008 11:35:56 -0700
Message-ID: <DD27C9769EA63D43BE3CF4CCCC12DD9701AA84EB@SNV-EXVS02.ds.corp.yahoo.com>
In-Reply-To: <57D7A94E-E310-4F65-82E6-858EBB1D14EE@yahoo-inc.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Hadoop User Group (Bay Area) Oct 15th
Thread-Index: AckkqOob9DI4jbMCQEeTE1RcN+ztSQFk7y8w
References: <57D7A94E-E310-4F65-82E6-858EBB1D14EE@yahoo-inc.com>
From: "Ajay Anand" <aanand@yahoo-inc.com>
To: <core-user@hadoop.apache.org>, <general@hadoop.apache.org>,
        <zookeeper-user@hadoop.apache.org>, <hbase-user@hadoop.apache.org>,
        <pig-user@incubator.apache.org>
X-OriginalArrivalTime: 09 Oct 2008 18:36:27.0751 (UTC) FILETIME=[F0C3DF70:01C92A3D]
X-Virus-Checked: Checked by ClamAV on apache.org

The next Bay Area User Group meeting is scheduled for October 15th at
Yahoo! 2821 Mission College Blvd, Santa Clara, Building 1, Training
Rooms 3 & 4 from 6:00-7:30 pm.

Agenda:
- Exploiting database join techniques for analytics with Hadoop: Jun
Rao, IBM
- Jaql Update: Kevin Beyer, IBM
- Experiences moving a Petabyte Data Center: Sriram Rao, Quantcast

Look forward to seeing you there!
Ajay

From general-return-69-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 10 17:20:15 2008
Return-Path: <general-return-69-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 98788 invoked from network); 10 Oct 2008 17:20:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 10 Oct 2008 17:20:13 -0000
Received: (qmail 43683 invoked by uid 500); 10 Oct 2008 17:20:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 43437 invoked by uid 500); 10 Oct 2008 17:20:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 43345 invoked by uid 99); 10 Oct 2008 17:20:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Oct 2008 10:20:08 -0700
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Oct 2008 17:19:01 +0000
Received: from SNV-EXBH01.ds.corp.yahoo.com (snv-exbh01.ds.corp.yahoo.com [207.126.227.249])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id m9AHIrHf080713;
	Fri, 10 Oct 2008 10:18:55 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:x-mimeole:content-class:mime-version:
	content-type:content-transfer-encoding:subject:date:message-id:
	in-reply-to:x-ms-has-attach:x-ms-tnef-correlator:thread-topic:
	thread-index:references:from:to:cc:return-path:x-originalarrivaltime;
	b=r+5bbOgmj/+4sGgf1D8UCM1IWoPAeE8g7j3zodoNxmUNdEbRqz8TL2ACxmyXZSW8
Received: from SNV-EXVS09.ds.corp.yahoo.com ([207.126.227.86]) by SNV-EXBH01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Fri, 10 Oct 2008 10:18:53 -0700
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable
Subject: RE: [VOTE] Pig graduation to hadoop subproject
Date: Fri, 10 Oct 2008 10:18:24 -0700
Message-ID: <236D5828CF7F5749AF1BCB7F87596DE1029AFE69@SNV-EXVS09.ds.corp.yahoo.com>
In-Reply-To: <236D5828CF7F5749AF1BCB7F87596DE1029AF8FA@SNV-EXVS09.ds.corp.yahoo.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: [VOTE] Pig graduation to hadoop subproject
Thread-Index: AckoA3LHD+ivlQt4Ro+OiDvaGryK2AC+K+Gg
References: <236D5828CF7F5749AF1BCB7F87596DE1029AF8FA@SNV-EXVS09.ds.corp.yahoo.com>
From: "Olga Natkovich" <olgan@yahoo-inc.com>
To: <pig-dev@incubator.apache.org>
Cc: <general@incubator.apache.org>, <general@hadoop.apache.org>
X-OriginalArrivalTime: 10 Oct 2008 17:18:53.0353 (UTC) FILETIME=[44F0D990:01C92AFC]
X-Virus-Checked: Checked by ClamAV on apache.org

The vote is closed now. Thanks for everybody who voted!

Total: 		9 +1	0 -1
Pig Committers: 	6 +1	0 -1
Pig Mentors:	1 +1	0 -1

Thanks,

Olga=20

> -----Original Message-----
> From: Olga Natkovich [mailto:olgan@yahoo-inc.com]=20
> Sent: Monday, October 06, 2008 3:33 PM
> To: pig-dev@incubator.apache.org
> Cc: general@incubator.apache.org; general@hadoop.apache.org
> Subject: [VOTE] Pig graduation to hadoop subproject
>=20
> Pig Developers and Mentors,
> =20
> Pig has been incubating for over a year now. In this period=20
> of time, we had extended our community with 2 new committers,=20
> had a release, resolved 300 issues. We have made some=20
> significant code improvements including pipeline redesign,=20
> addition of streaming and limit functionality, grunt shell=20
> improvements and significant performance speedup. We have a=20
> constant traffic and lively discussions on both pig-dev and=20
> pig-user mailing lists and we conduct our business in the=20
> open by publishing proposals and discussing them in the mailing lists.
> =20
> As of now, we have completed graduation requirements as=20
> described in http://incubator.apache.org/guides/graduation.html.
> =20
> I would like to call for a graduation vote at this time.
> =20
> I would also propose that we graduate as a subproject of=20
> Hadoop. There are several advantages to this approach. First,=20
> this would allow us to extend both our user and developer=20
> base. Second, it would bring benefits to the Hadoop community=20
> by providing a higher level interface and easier entry point=20
> for new users. Third, having an established project to=20
> provide guidance, would help Pig to become a mature=20
> participant in the open source community.
> =20
> Please, vote by the end of the day on Thursday, 10/9.
> =20
> Thanks,
> =20
> Olga
> =20
> PS: I am ccing  hadoop and incubator general mailing lists;=20
> however, no action is required from them at this time. This=20
> step is for Pig community only.
>=20

From general-return-70-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 13 05:08:17 2008
Return-Path: <general-return-70-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 90120 invoked from network); 13 Oct 2008 05:08:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 13 Oct 2008 05:08:17 -0000
Received: (qmail 67675 invoked by uid 500); 13 Oct 2008 05:08:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67628 invoked by uid 500); 13 Oct 2008 05:08:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67609 invoked by uid 99); 13 Oct 2008 05:08:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Oct 2008 22:08:13 -0700
X-ASF-Spam-Status: No, hits=3.2 required=10.0
	tests=RCVD_IN_BL_SPAMCOP_NET,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.62.16] (HELO QMTA01.westchester.pa.mail.comcast.net) (76.96.62.16)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Oct 2008 05:07:05 +0000
Received: from OMTA11.westchester.pa.mail.comcast.net ([76.96.62.36])
	by QMTA01.westchester.pa.mail.comcast.net with comcast
	id S4ww1a0030mv7h05157Z8s; Mon, 13 Oct 2008 05:07:33 +0000
Received: from [12.7.212.84] ([209.131.62.115])
	by OMTA11.westchester.pa.mail.comcast.net with comcast
	id S56f1a0022VBGtd3X56l4d; Mon, 13 Oct 2008 05:06:55 +0000
X-Authority-Analysis: v=1.0 c=1 a=SodKoXcs9TsA:10 a=feAe3NOjdhMA:10
 a=yzoYlIRGkuBQZggdGLgA:9 a=-JAYgQfiNDxMrd8BeML3K0sTzAsA:4 a=AXjVNIk09o8A:10
Message-Id: <A2880476-79A5-4D72-9DD9-B07366F12801@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v929.2)
Subject: [VOTE] Should we accept Pig as a Hadoop subproject?
Date: Sun, 12 Oct 2008 22:07:11 -0700
X-Mailer: Apple Mail (2.929.2)
X-Virus-Checked: Checked by ClamAV on apache.org

Should we accept Pig as a subproject? I think it has done well getting  
through the incubator, both with attracting new committers and making  
its first release. Within Yahoo, on our research clusters, the jobs  
are roughly broken down into 1/3 streaming, 1/3 java map/reduce, and  
1/3 pig, so it is a very relevant project.

+1

-- Owen

From general-return-71-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 13 05:14:41 2008
Return-Path: <general-return-71-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 91389 invoked from network); 13 Oct 2008 05:14:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 13 Oct 2008 05:14:41 -0000
Received: (qmail 72749 invoked by uid 500); 13 Oct 2008 05:14:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72627 invoked by uid 500); 13 Oct 2008 05:14:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72612 invoked by uid 99); 13 Oct 2008 05:14:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Oct 2008 22:14:41 -0700
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Oct 2008 05:13:32 +0000
Received: from [10.0.1.5] (snvvpn2-10-72-78-c178.corp.yahoo.com [10.72.78.178])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id m9D5Bj3V077222;
	Sun, 12 Oct 2008 22:12:38 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=XT8/iBW/PAIIimqLpfb8lOnH7Y1frbUnYl6a+hJ0FV/LogETjDwloG7MN2Gqgt1S
Message-Id: <5AFD70A1-0E3C-4403-9F57-BBC921E54693@yahoo-inc.com>
From: Nigel Daley <ndaley@yahoo-inc.com>
To: general@hadoop.apache.org, "Owen O'Malley" <omalley@apache.org>
In-Reply-To: <A2880476-79A5-4D72-9DD9-B07366F12801@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v929.2)
Subject: Re: [VOTE] Should we accept Pig as a Hadoop subproject?
Date: Sun, 12 Oct 2008 22:12:39 -0700
References: <A2880476-79A5-4D72-9DD9-B07366F12801@apache.org>
X-Mailer: Apple Mail (2.929.2)
X-Virus-Checked: Checked by ClamAV on apache.org

+1

On Oct 12, 2008, at 10:07 PM, Owen O'Malley wrote:

> Should we accept Pig as a subproject? I think it has done well  
> getting through the incubator, both with attracting new committers  
> and making its first release. Within Yahoo, on our research  
> clusters, the jobs are roughly broken down into 1/3 streaming, 1/3  
> java map/reduce, and 1/3 pig, so it is a very relevant project.
>
> +1
>
> -- Owen


From general-return-72-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 13 09:17:34 2008
Return-Path: <general-return-72-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 97810 invoked from network); 13 Oct 2008 09:17:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 13 Oct 2008 09:17:34 -0000
Received: (qmail 6524 invoked by uid 500); 13 Oct 2008 09:17:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6501 invoked by uid 500); 13 Oct 2008 09:17:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6490 invoked by uid 99); 13 Oct 2008 09:17:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Oct 2008 02:17:34 -0700
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 66.206.198.176 is neither permitted nor denied by domain of enis.soz.nutch@gmail.com)
Received: from [66.206.198.176] (HELO Generic33.powerdnn.com) (66.206.198.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Oct 2008 09:16:28 +0000
Received: from [192.168.2.15] ([85.105.135.220]) by Generic33.powerdnn.com with MailEnable ESMTP; Mon, 13 Oct 2008 02:20:23 -0700
Message-ID: <48F31205.3090300@gmail.com>
Date: Mon, 13 Oct 2008 12:16:53 +0300
From: Enis Soztutar <enis.soz.nutch@gmail.com>
User-Agent: Thunderbird 2.0.0.17 (X11/20080925)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Should we accept Pig as a Hadoop subproject?
References: <A2880476-79A5-4D72-9DD9-B07366F12801@apache.org>
In-Reply-To: <A2880476-79A5-4D72-9DD9-B07366F12801@apache.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-ME-Bayesian: 0.000000
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Owen O'Malley wrote:
> Should we accept Pig as a subproject? I think it has done well getting 
> through the incubator, both with attracting new committers and making 
> its first release. Within Yahoo, on our research clusters, the jobs 
> are roughly broken down into 1/3 streaming, 1/3 java map/reduce, and 
> 1/3 pig, so it is a very relevant project.
>
> +1
>
> -- Owen
>


From general-return-73-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 13 16:09:51 2008
Return-Path: <general-return-73-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 57612 invoked from network); 13 Oct 2008 16:09:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 13 Oct 2008 16:09:51 -0000
Received: (qmail 47301 invoked by uid 500); 13 Oct 2008 16:09:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47282 invoked by uid 500); 13 Oct 2008 16:09:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47271 invoked by uid 99); 13 Oct 2008 16:09:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Oct 2008 09:09:50 -0700
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Oct 2008 16:08:41 +0000
Received: from [192.168.1.64] (snvvpn1-10-72-72-c169.corp.yahoo.com [10.72.72.169])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id m9DG8pm8020563
	for <general@hadoop.apache.org>; Mon, 13 Oct 2008 09:08:51 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=Pi0Xt7vxFNW/XTiq6rHIGdylJjeJKf1kYe8w66FRnWWBFz5dJlVaBKZDT18e1iAH
Message-Id: <A29C1FE9-DD86-4C1F-8068-0CB30F40F077@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <A2880476-79A5-4D72-9DD9-B07366F12801@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v929.2)
Subject: Re: [VOTE] Should we accept Pig as a Hadoop subproject?
Date: Mon, 13 Oct 2008 09:08:53 -0700
References: <A2880476-79A5-4D72-9DD9-B07366F12801@apache.org>
X-Mailer: Apple Mail (2.929.2)
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Arun

On Oct 12, 2008, at 10:07 PM, Owen O'Malley wrote:

> Should we accept Pig as a subproject? I think it has done well  
> getting through the incubator, both with attracting new committers  
> and making its first release. Within Yahoo, on our research  
> clusters, the jobs are roughly broken down into 1/3 streaming, 1/3  
> java map/reduce, and 1/3 pig, so it is a very relevant project.
>
> +1
>
> -- Owen


From general-return-74-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 13 17:11:57 2008
Return-Path: <general-return-74-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 84652 invoked from network); 13 Oct 2008 17:11:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 13 Oct 2008 17:11:57 -0000
Received: (qmail 63367 invoked by uid 500); 13 Oct 2008 17:11:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63345 invoked by uid 500); 13 Oct 2008 17:11:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63334 invoked by uid 99); 13 Oct 2008 17:11:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Oct 2008 10:11:57 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 72.14.220.158 as permitted sender)
Received: from [72.14.220.158] (HELO fg-out-1718.google.com) (72.14.220.158)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Oct 2008 17:10:52 +0000
Received: by fg-out-1718.google.com with SMTP id l26so1275131fgb.35
        for <general@hadoop.apache.org>; Mon, 13 Oct 2008 10:11:11 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:to
         :subject:in-reply-to:mime-version:content-type
         :content-transfer-encoding:content-disposition:references;
        bh=hnu96zMA4vnq6+JZ/xd8dF6yS2NhbhhTX5OEZ/+KY2M=;
        b=GhuyEMxLyPPLCNXQaktPFYlMtrLGa4pE04oqBB89UZAGP3enu9a9b4bVWE8cUsbcFt
         qf9pQHednTsgnvcnLftrTxCBKFq3YSGYPlr8hls4VsE/ScwPph9i8lXNvVvBAUFp7M2J
         SoR8UoZhleg/mA6Nyq0clitlTher2QB/a/XH0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version
         :content-type:content-transfer-encoding:content-disposition
         :references;
        b=ta1r6zRg/N6hr0nQfvUBjyEdWn0R3BeX04Pq1UQRMf8NtWMjJ1JOXarG9f+kvm1xM/
         lgH3bk+UAjlmlLkQrdMzu9qlT4FdkKybEC7bv08Lxa4WgKKYp9UDJyyfcaD1rOBy2Lua
         ofZqb2DfFLAKxeEVlrMDaCTLtx+Nhlj4GZtfY=
Received: by 10.187.159.15 with SMTP id l15mr1090570fao.95.1223917871088;
        Mon, 13 Oct 2008 10:11:11 -0700 (PDT)
Received: by 10.187.224.13 with HTTP; Mon, 13 Oct 2008 10:11:10 -0700 (PDT)
Message-ID: <4aa34eb70810131011v5280ae8t8d5dd978b8c165fe@mail.gmail.com>
Date: Mon, 13 Oct 2008 10:11:10 -0700
From: "Dhruba Borthakur" <dhruba@gmail.com>
To: general@hadoop.apache.org
Subject: Re: [VOTE] Should we accept Pig as a Hadoop subproject?
In-Reply-To: <A29C1FE9-DD86-4C1F-8068-0CB30F40F077@yahoo-inc.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <A2880476-79A5-4D72-9DD9-B07366F12801@apache.org>
	 <A29C1FE9-DD86-4C1F-8068-0CB30F40F077@yahoo-inc.com>
X-Virus-Checked: Checked by ClamAV on apache.org

+1.

dhruba

On Mon, Oct 13, 2008 at 9:08 AM, Arun C Murthy <acm@yahoo-inc.com> wrote:
> +1
>
> Arun
>
> On Oct 12, 2008, at 10:07 PM, Owen O'Malley wrote:
>
>> Should we accept Pig as a subproject? I think it has done well getting
>> through the incubator, both with attracting new committers and making its
>> first release. Within Yahoo, on our research clusters, the jobs are roughly
>> broken down into 1/3 streaming, 1/3 java map/reduce, and 1/3 pig, so it is a
>> very relevant project.
>>
>> +1
>>
>> -- Owen
>
>

From general-return-75-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 13 19:01:22 2008
Return-Path: <general-return-75-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 57067 invoked from network); 13 Oct 2008 19:01:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 13 Oct 2008 19:01:22 -0000
Received: (qmail 13136 invoked by uid 500); 13 Oct 2008 19:01:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12907 invoked by uid 500); 13 Oct 2008 19:01:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12845 invoked by uid 99); 13 Oct 2008 19:01:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Oct 2008 12:01:18 -0700
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Oct 2008 19:00:09 +0000
Received: from SNV-EXBH01.ds.corp.yahoo.com (snv-exbh01.ds.corp.yahoo.com [207.126.227.249])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id m9DIwGaY035426;
	Mon, 13 Oct 2008 11:58:17 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:x-mimeole:content-class:mime-version:
	content-type:content-transfer-encoding:subject:date:message-id:
	in-reply-to:x-ms-has-attach:x-ms-tnef-correlator:thread-topic:
	thread-index:references:from:to:cc:return-path:x-originalarrivaltime;
	b=ZHMSUTsbMZhxrvmn7oh1tyMykqRmpN0nvninA2zqK7qgN7Bj+wi9Izf/UaDNME3L
Received: from SNV-EXVS09.ds.corp.yahoo.com ([207.126.227.86]) by SNV-EXBH01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Mon, 13 Oct 2008 11:58:16 -0700
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable
Subject: RE: [VOTE] Pig graduation to hadoop subproject
Date: Mon, 13 Oct 2008 11:58:15 -0700
Message-ID: <236D5828CF7F5749AF1BCB7F87596DE1029B0027@SNV-EXVS09.ds.corp.yahoo.com>
In-Reply-To: <NBBBJGEAGJAKLIDBKJOPAEEKHHAD.noel@devtech.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: [VOTE] Pig graduation to hadoop subproject
Thread-Index: AcktXm6U9TzLIFY0Q8yocLvuT8myYgABVN7Q
References: <236D5828CF7F5749AF1BCB7F87596DE1029AF8FA@SNV-EXVS09.ds.corp.yahoo.com> <NBBBJGEAGJAKLIDBKJOPAEEKHHAD.noel@devtech.com>
From: "Olga Natkovich" <olgan@yahoo-inc.com>
To: <general@incubator.apache.org>, <pig-dev@incubator.apache.org>
Cc: <general@hadoop.apache.org>
X-OriginalArrivalTime: 13 Oct 2008 18:58:16.0476 (UTC) FILETIME=[A67A75C0:01C92D65]
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Noel,

My understanding is that graduation into a subproject is a 3 step
process:

(1) Community vote (optional but desired)
(2) Project that will accept the subproject (in this case Hadoop) votes
(3) Incubator PMC votes.

I think we completed step (1) and need to move to step (2).

Is this correct?

Olga

> -----Original Message-----
> From: Noel J. Bergman [mailto:noel@devtech.com]=20
> Sent: Monday, October 13, 2008 11:04 AM
> To: general@incubator.apache.org; pig-dev@incubator.apache.org
> Cc: general@hadoop.apache.org
> Subject: RE: [VOTE] Pig graduation to hadoop subproject
>=20
> Olga,
>=20
> I saw your:
>=20
> Total: 		9 +1	0 -1
> Pig Committers: 	6 +1	0 -1
> Pig Mentors:	1 +1	0 -1
>=20
> But please note that only Incubator PMC votes actually count,=20
> and you need at least 3.  Whom do you have besides the one=20
> Mentor and mine (herein granted)?
>=20
> Also, since you intend to graduate into Hadoop, that PMC must=20
> vote to accept Pig -- we don't force it on them.  ;-)  Has=20
> that happened?
>=20
> 	--- Noel
>=20
>=20
>=20
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>=20
>=20

From general-return-76-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 14 06:58:10 2008
Return-Path: <general-return-76-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 59918 invoked from network); 14 Oct 2008 06:58:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 14 Oct 2008 06:58:09 -0000
Received: (qmail 69384 invoked by uid 500); 14 Oct 2008 06:58:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69368 invoked by uid 500); 14 Oct 2008 06:58:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 35638 invoked by uid 99); 13 Oct 2008 18:05:33 -0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FORGED_MUA_OIMO,MSGID_FROM_MTA_HEADER,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Message-ID: <NBBBJGEAGJAKLIDBKJOPAEEKHHAD.noel@devtech.com>
MIME-Version: 1.0
X-MessageIsInfected: false
From: "Noel J. Bergman" <noel@devtech.com>
To: <general@incubator.apache.org>,
	<pig-dev@incubator.apache.org>
Cc: <general@hadoop.apache.org>
Subject: RE: [VOTE] Pig graduation to hadoop subproject
Date: Mon, 13 Oct 2008 14:04:05 -0400
Content-Type: text/plain;
	charset="windows-1250"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook IMO, Build 9.0.6604 (9.0.2911.0)
In-Reply-To: <236D5828CF7F5749AF1BCB7F87596DE1029AF8FA@SNV-EXVS09.ds.corp.yahoo.com>
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3350
Importance: Normal
X-Virus-Checked: Checked by ClamAV on apache.org

Olga,

I saw your:

Total: 		9 +1	0 -1
Pig Committers: 	6 +1	0 -1
Pig Mentors:	1 +1	0 -1

But please note that only Incubator PMC votes actually count, and you need
at least 3.  Whom do you have besides the one Mentor and mine (herein
granted)?

Also, since you intend to graduate into Hadoop, that PMC must vote to accept
Pig -- we don't force it on them.  ;-)  Has that happened?

	--- Noel



From general-return-77-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 14 06:58:26 2008
Return-Path: <general-return-77-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 59971 invoked from network); 14 Oct 2008 06:58:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 14 Oct 2008 06:58:26 -0000
Received: (qmail 69535 invoked by uid 500); 14 Oct 2008 06:58:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69513 invoked by uid 500); 14 Oct 2008 06:58:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 66030 invoked by uid 99); 13 Oct 2008 18:25:37 -0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Date: Mon, 13 Oct 2008 11:24:57 -0700
From: Craig L Russell <Craig.Russell@Sun.COM>
Subject: Re: [VOTE] Pig graduation to hadoop subproject
In-reply-to: <NBBBJGEAGJAKLIDBKJOPAEEKHHAD.noel@devtech.com>
Sender: Craig.Russell@Sun.COM
To: general@incubator.apache.org
Cc: pig-dev@incubator.apache.org, general@hadoop.apache.org
Message-id: <4262C847-D56F-42C2-81B4-C5C4EC2970DD@SUN.com>
MIME-version: 1.0
X-Mailer: Apple Mail (2.929.2)
Content-type: multipart/signed; protocol="application/pkcs7-signature";
 micalg=sha1; boundary=Apple-Mail-368-801961841
X-Priority: 3 (Normal)
References: <NBBBJGEAGJAKLIDBKJOPAEEKHHAD.noel@devtech.com>
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-368-801961841
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit

Hi Noel,

On Oct 13, 2008, at 11:04 AM, Noel J. Bergman wrote:

> Olga,
>
> I saw your:
>
> Total: 		9 +1	0 -1
> Pig Committers: 	6 +1	0 -1
> Pig Mentors:	1 +1	0 -1
>
> But please note that only Incubator PMC votes actually count, and  
> you need
> at least 3.  Whom do you have besides the one Mentor and mine (herein
> granted)?
>
> Also, since you intend to graduate into Hadoop, that PMC must vote  
> to accept
> Pig -- we don't force it on them.  ;-)  Has that happened?
>
> 	--- Noel

Olga's [VOTE] was for Pig community members (and mentors, of course).

 From Olga's message:


> Please, vote by the end of the day on Thursday, 10/9.
>
> Thanks,
>
> Olga
>
> PS: I am ccing  hadoop and incubator general mailing lists; however,  
> no
> action is required from them at this time. This step is for Pig
> community only.


Craig


>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>

Craig L Russell
Architect, Sun Java Enterprise System http://db.apache.org/jdo
408 276-5638 mailto:Craig.Russell@sun.com
P.S. A good JDO? O, Gasp!


--Apple-Mail-368-801961841
Content-Disposition: attachment;
	filename=smime.p7s
Content-Type: application/pkcs7-signature;
	name=smime.p7s
Content-Transfer-Encoding: base64

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIGUDCCAwkw
ggJyoAMCAQICECvOQSuIjHMvOZRC95BRg/wwDQYJKoZIhvcNAQEFBQAwYjELMAkGA1UEBhMCWkEx
JTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQ
ZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA3MTIxMDE1MjM1MVoXDTA4MTIwOTE1MjM1
MVowbDEQMA4GA1UEBBMHUnVzc2VsbDEUMBIGA1UEKhMLQ3JhaWcgTGFpcmQxHDAaBgNVBAMTE0Ny
YWlnIExhaXJkIFJ1c3NlbGwxJDAiBgkqhkiG9w0BCQEWFUNyYWlnLlJ1c3NlbGxAU3VuLkNPTTCC
ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKzqGlLUastboCRqc0iBoTz2ODcqpMpEyPUo
nYtluchkSIoWzOW63AuoTczRt9sKfhwoK5mope+62B6Li06WJabm2UHqKAaNSuMHLsmyqvOdwbSt
enY7/HxOSCMqVoyVBTRJc2M8feCSVgi7ptGq9cM+Maa64R1/p9zqaQNucceU/1uper90bWplsjAT
rHgicgr9XJIQb6uYjhjlgxxnY/aispnCvLxMX+CiA2FWeeJTI7AiFlLwibTXYF4v12ToByvXtTiJ
knuND8qpwhK3Wp0tL4ae8mZ0nlKjCuNnqh99ZyEyTFHZBfVx8WSWRXkY4qxCG/IDQUo7WUaefOQT
1mECAwEAAaMyMDAwIAYDVR0RBBkwF4EVQ3JhaWcuUnVzc2VsbEBTdW4uQ09NMAwGA1UdEwEB/wQC
MAAwDQYJKoZIhvcNAQEFBQADgYEAEqfFNFoch0QPVKWJ4maAZl3MJD10yMeWt5xb+WNSkhYKHD8I
42E8tpdE3kmc5wp2cZrz9JqJF/KCQ/gI4pmDk1qpTs5pvXzFNiD5Lu5eLza4iyxSlTHUXcCnyNC6
4m0qC8p4m/51NEql5hyacj/+vdlEe5dygpyNGUCiyA/SdAswggM/MIICqKADAgECAgENMA0GCSqG
SIb3DQEBBQUAMIHRMQswCQYDVQQGEwJaQTEVMBMGA1UECBMMV2VzdGVybiBDYXBlMRIwEAYDVQQH
EwlDYXBlIFRvd24xGjAYBgNVBAoTEVRoYXd0ZSBDb25zdWx0aW5nMSgwJgYDVQQLEx9DZXJ0aWZp
Y2F0aW9uIFNlcnZpY2VzIERpdmlzaW9uMSQwIgYDVQQDExtUaGF3dGUgUGVyc29uYWwgRnJlZW1h
aWwgQ0ExKzApBgkqhkiG9w0BCQEWHHBlcnNvbmFsLWZyZWVtYWlsQHRoYXd0ZS5jb20wHhcNMDMw
NzE3MDAwMDAwWhcNMTMwNzE2MjM1OTU5WjBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3Rl
IENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWls
IElzc3VpbmcgQ0EwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAMSmPFVzVftOucqZWh5owHUE
cJ3f6f+jHuy9zfVb8hp2vX8MOmHyv1HOAdTlUAow1wJjWiyJFXCO3cnwK4Vaqj9xVsuvPAsH5/Ef
kTYkKhPPK9Xzgnc9A74r/rsYPge/QIACZNenprufZdHFKlSFD0gEf6e20TxhBEAeZBlyYLf7AgMB
AAGjgZQwgZEwEgYDVR0TAQH/BAgwBgEB/wIBADBDBgNVHR8EPDA6MDigNqA0hjJodHRwOi8vY3Js
LnRoYXd0ZS5jb20vVGhhd3RlUGVyc29uYWxGcmVlbWFpbENBLmNybDALBgNVHQ8EBAMCAQYwKQYD
VR0RBCIwIKQeMBwxGjAYBgNVBAMTEVByaXZhdGVMYWJlbDItMTM4MA0GCSqGSIb3DQEBBQUAA4GB
AEiM0VCD6gsuzA2jZqxnD3+vrL7CF6FDlpSdf0whuPg2H6otnzYvwPQcUCCTcDz9reFhYsPZOhl+
hLGZGwDFGguCdJ4lUJRix9sncVcljd2pnDmOjCBPZV+V2vf3h9bGCE6u9uo05RAaWzVNd+NWIXiC
3CEZNd4ksdMdRv9dX2VPMYIDEDCCAwwCAQEwdjBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhh
d3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVt
YWlsIElzc3VpbmcgQ0ECECvOQSuIjHMvOZRC95BRg/wwCQYFKw4DAhoFAKCCAW8wGAYJKoZIhvcN
AQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMDgxMDEzMTgyNDU3WjAjBgkqhkiG9w0B
CQQxFgQUVrbFvkdta8J/9cuSVeJvdSw+QAEwgYUGCSsGAQQBgjcQBDF4MHYwYjELMAkGA1UEBhMC
WkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0
ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBAhArzkEriIxzLzmUQveQUYP8MIGHBgsqhkiG
9w0BCRACCzF4oHYwYjELMAkGA1UEBhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQ
dHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBAhAr
zkEriIxzLzmUQveQUYP8MA0GCSqGSIb3DQEBAQUABIIBAHJfoON8Z/TBOW9vs+7Pwg1C2RFjFX+Q
25FCRr3o5D9KDGbc8sDi8nUc7sLaO7QvK+GO+TURwt1PYuRqsmnbunpnD0I2eHTbS2t+ORd6Xl3S
lvzkbzJb2RSxAdzLB3k6Rkyij6Eoccntjqgo7RLJl3JSG8MBjDgrVm5JOu/lypJx2PLE5DbOgjVs
seSULYUZP8wy+WrOJJMtE+alFXP1sC2dVhwAd3/+xEM2TXdyXrutNkRsLWDIPewsZ6UImJjvQxfY
p/t4j/FBRkEsKtCfOBD2DY/WKQcOwHXMhq5hStFHT8tANkTDLCPFujknjRFNvp/7p1Bw9spR5HVL
x5BOU/gAAAAAAAA=

--Apple-Mail-368-801961841--

From general-return-78-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 14 16:25:28 2008
Return-Path: <general-return-78-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 86065 invoked from network); 14 Oct 2008 16:25:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 14 Oct 2008 16:25:28 -0000
Received: (qmail 36750 invoked by uid 500); 14 Oct 2008 16:25:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36731 invoked by uid 500); 14 Oct 2008 16:25:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 23998 invoked by uid 99); 14 Oct 2008 16:20:36 -0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yoavshapira@gmail.com designates 209.85.217.13 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:sender
         :to:subject:cc:in-reply-to:mime-version:content-type
         :content-transfer-encoding:content-disposition:references
         :x-google-sender-auth;
        bh=1VPB8OEJ45rj3qfa1qA3HyLb/u81WOrkjgDfm6/Gwzg=;
        b=WxCgC/Ypwp++2Se+ZTSMkdU9WI6PNjzA9nPm4AaIiaX4j4YVzzn5SiGKUajo/IvbxN
         fvAFp1v2/zXkgpuudxPtwpQ5J11KGSraZDVHpTyZqhBxKdotYvX22akKJutAypjAm14p
         CNuXIAnH1TInR6cTIVrB02rN5sODc3V6cwzPY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version
         :content-type:content-transfer-encoding:content-disposition
         :references:x-google-sender-auth;
        b=q3TrLFq+BkUHMvn03W3FegYPRarXrcCO1hLws9ObMC/jatYVIl0u4A6JxlxB9gzy9d
         Yvl2CYqvXdwkzLyN0PX0pn5NyzOioIuS83tRPOtx3sT+wHlfBGw4GPjSFIbu4nwUH4OF
         KMns/+AtKoUbrUSAFZRoqK+k2o7oW7ADJM6eI=
Message-ID: <db94f1a0810140920hcd2c74doadd6a966f6041ec7@mail.gmail.com>
Date: Tue, 14 Oct 2008 12:20:06 -0400
From: "Yoav Shapira" <yoavs@apache.org>
Sender: yoavshapira@gmail.com
To: pig-dev@incubator.apache.org
Subject: Re: [VOTE] Pig graduation to hadoop subproject
Cc: general@incubator.apache.org, general@hadoop.apache.org
In-Reply-To: <D52C5A69-1DAD-4302-A89D-D2298798F020@yahoo-inc.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <236D5828CF7F5749AF1BCB7F87596DE1029AF8FA@SNV-EXVS09.ds.corp.yahoo.com>
	 <D52C5A69-1DAD-4302-A89D-D2298798F020@yahoo-inc.com>
X-Google-Sender-Auth: 499d7ada11fcda47
X-Virus-Checked: Checked by ClamAV on apache.org

> On Oct 6, 2008, at 3:32 PM, Olga Natkovich wrote:
> As of now, we have completed graduation requirements as described in
> http://incubator.apache.org/guides/graduation.html.
>
> I would like to call for a graduation vote at this time.

+1

Late, I know, was traveling.

Yoav

From general-return-79-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 16 17:21:15 2008
Return-Path: <general-return-79-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 62452 invoked from network); 16 Oct 2008 17:21:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 16 Oct 2008 17:21:15 -0000
Received: (qmail 19710 invoked by uid 500); 16 Oct 2008 17:21:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19687 invoked by uid 500); 16 Oct 2008 17:21:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19674 invoked by uid 99); 16 Oct 2008 17:21:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Oct 2008 10:21:15 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cutting@gmail.com designates 209.85.217.13 as permitted sender)
Received: from [209.85.217.13] (HELO mail-gx0-f13.google.com) (209.85.217.13)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Oct 2008 17:20:08 +0000
Received: by gxk6 with SMTP id 6so7689340gxk.5
        for <general@hadoop.apache.org>; Thu, 16 Oct 2008 10:19:45 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding:sender;
        bh=KKBu/3Tia+mqjYlvPmBdiQOKpPTV07GDXAvNTbah790=;
        b=N4+Qsf6M/o2rr8v1CmUM2WPjEgUpq79xPRxh+nAoZY8Aarewwk+njcNFwx6ca2Uex3
         233jQdKnZTCUeShScw/klIYsYDeSM3o9NaIr/AAyifD3ubYm9sjYyQEz7SilAE+6tYgp
         zSkXFSMCA4lPIDsHiMdkgXyhknO0idp5UWVk4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:content-type:content-transfer-encoding:sender;
        b=m8Yz+qugtDrJ00Rh7bBGeV3JH8PN5T1xW345UpBLaimvg64JTNzitadCtW7wKClMsZ
         pHA/rfqCzXQRcQdls0i/cgCWEcrkkzvFL1b42G6dJTJya0qYtFEf5oxQp1zaRH3uSpOT
         uFSPcu33seCjlok/pYmSIt+ySUyXuFyHlbuXk=
Received: by 10.142.212.19 with SMTP id k19mr834151wfg.155.1224177584372;
        Thu, 16 Oct 2008 10:19:44 -0700 (PDT)
Received: from ?192.168.168.16? (c-76-103-191-253.hsd1.ca.comcast.net [76.103.191.253])
        by mx.google.com with ESMTPS id 20sm783558wfi.11.2008.10.16.10.19.41
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Thu, 16 Oct 2008 10:19:42 -0700 (PDT)
Message-ID: <48F777AC.80509@apache.org>
Date: Thu, 16 Oct 2008 10:19:40 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.17 (X11/20080925)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Should we accept Pig as a Hadoop subproject?
References: <A2880476-79A5-4D72-9DD9-B07366F12801@apache.org>
In-Reply-To: <A2880476-79A5-4D72-9DD9-B07366F12801@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Sender: Doug Cutting <cutting@gmail.com>
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Doug

Owen O'Malley wrote:
> Should we accept Pig as a subproject? I think it has done well getting 
> through the incubator, both with attracting new committers and making 
> its first release. Within Yahoo, on our research clusters, the jobs are 
> roughly broken down into 1/3 streaming, 1/3 java map/reduce, and 1/3 
> pig, so it is a very relevant project.
> 
> +1
> 
> -- Owen

From general-return-80-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 17 17:24:40 2008
Return-Path: <general-return-80-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 17863 invoked from network); 17 Oct 2008 17:24:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 17 Oct 2008 17:24:40 -0000
Received: (qmail 1615 invoked by uid 500); 17 Oct 2008 17:24:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1467 invoked by uid 500); 17 Oct 2008 17:24:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1453 invoked by uid 99); 17 Oct 2008 17:24:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Oct 2008 10:24:41 -0700
X-ASF-Spam-Status: No, hits=4.0 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_BL_SPAMCOP_NET,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.62.40] (HELO QMTA04.westchester.pa.mail.comcast.net) (76.96.62.40)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Oct 2008 17:23:32 +0000
Received: from OMTA03.westchester.pa.mail.comcast.net ([76.96.62.27])
	by QMTA04.westchester.pa.mail.comcast.net with comcast
	id TriK1a0780bG4ec54tQ9V6; Fri, 17 Oct 2008 17:24:09 +0000
Received: from battlerock-lm.corp.yahoo.com ([209.131.62.113])
	by OMTA03.westchester.pa.mail.comcast.net with comcast
	id TtPd1a00T2SbwD53PtPjJB; Fri, 17 Oct 2008 17:24:04 +0000
X-Authority-Analysis: v=1.0 c=1 a=Bx4iz1Cc0xIA:10 a=QT7C_Xz5BxwA:10
 a=skSSq9RChP7sE-xtKCUA:9 a=i3VmUkf1QgXPTj67B-dv1xUJnKMA:4 a=bdDRB4-mPNEA:10
Cc: pig-dev@incubator.apache.org,
 general@incubator.apache.org
Message-Id: <357828F7-5CAC-40DE-96AA-CC14FBF18508@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <48F777AC.80509@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v929.2)
Subject: Re: [VOTE] Should we accept Pig as a Hadoop subproject?
Date: Fri, 17 Oct 2008 10:23:37 -0700
References: <A2880476-79A5-4D72-9DD9-B07366F12801@apache.org> <48F777AC.80509@apache.org>
X-Mailer: Apple Mail (2.929.2)
X-Virus-Checked: Checked by ClamAV on apache.org

Owen O'Malley wrote:

> Should we accept Pig as a subproject?

With six +1's from the Hadoop PMC, the vote passes. Hadoop will have a  
new subproject. *smile*

-- Owen

From general-return-81-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 17 17:40:27 2008
Return-Path: <general-return-81-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 30571 invoked from network); 17 Oct 2008 17:40:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 17 Oct 2008 17:40:25 -0000
Received: (qmail 24869 invoked by uid 500); 17 Oct 2008 17:40:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24615 invoked by uid 500); 17 Oct 2008 17:40:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24577 invoked by uid 99); 17 Oct 2008 17:40:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Oct 2008 10:40:23 -0700
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Oct 2008 17:39:12 +0000
Received: from SNV-EXPF01.ds.corp.yahoo.com (snv-expf01.ds.corp.yahoo.com [207.126.227.250])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id m9HHcUfZ020847;
	Fri, 17 Oct 2008 10:38:30 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:x-mimeole:content-class:mime-version:
	content-type:content-transfer-encoding:subject:date:message-id:
	in-reply-to:x-ms-has-attach:x-ms-tnef-correlator:thread-topic:
	thread-index:references:from:to:cc:return-path:x-originalarrivaltime;
	b=SGtCLiBBQcuQgMwpZnCyFSygTe4iGs1tqhpgi9zY/XAXlUaxicBJEN4xhfsuou75
Received: from SNV-EXVS09.ds.corp.yahoo.com ([207.126.227.86]) by SNV-EXPF01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Fri, 17 Oct 2008 10:38:30 -0700
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable
Subject: RE: [VOTE] Should we accept Pig as a Hadoop subproject?
Date: Fri, 17 Oct 2008 10:38:24 -0700
Message-ID: <236D5828CF7F5749AF1BCB7F87596DE1029B0611@SNV-EXVS09.ds.corp.yahoo.com>
In-Reply-To: <f767f0600810171028j25cc8c6dsf1106b2f0cf6beb9@mail.gmail.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: [VOTE] Should we accept Pig as a Hadoop subproject?
Thread-Index: Ackwff3XXalbnsaFTl23jXd8QuP5UwAAP1cw
References: <A2880476-79A5-4D72-9DD9-B07366F12801@apache.org> <48F777AC.80509@apache.org> <357828F7-5CAC-40DE-96AA-CC14FBF18508@apache.org> <f767f0600810171028j25cc8c6dsf1106b2f0cf6beb9@mail.gmail.com>
From: "Olga Natkovich" <olgan@yahoo-inc.com>
To: <general@incubator.apache.org>, <pig-dev@incubator.apache.org>
Cc: <general@hadoop.apache.org>
X-OriginalArrivalTime: 17 Oct 2008 17:38:30.0432 (UTC) FILETIME=[2B6D1A00:01C9307F]
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Bertrand,

Since you are one of our mentors, could you please start that vote.

Thanks,

Olga=20

> -----Original Message-----
> From: bdelacretaz@gmail.com [mailto:bdelacretaz@gmail.com] On=20
> Behalf Of Bertrand Delacretaz
> Sent: Friday, October 17, 2008 10:29 AM
> To: pig-dev@incubator.apache.org
> Cc: general@hadoop.apache.org; general@incubator.apache.org
> Subject: Re: [VOTE] Should we accept Pig as a Hadoop subproject?
>=20
> On Fri, Oct 17, 2008 at 7:23 PM, Owen O'Malley=20
> <omalley@apache.org> wrote:
> > Owen O'Malley wrote:
> >
> >> Should we accept Pig as a subproject?
> >
> > With six +1's from the Hadoop PMC, the vote passes. Hadoop=20
> will have a=20
> > new subproject. *smile*
>=20
> Note that according to [1] this requires a final "Graduation=20
> Approval Vote" by the incubator PMC.
>=20
> -Bertrand (*smile* as well, don't get me wrong ;-)
>=20
> [1] http://incubator.apache.org/guides/graduation.html#subproject
>=20
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>=20
>=20

From general-return-82-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 17 17:47:29 2008
Return-Path: <general-return-82-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 36344 invoked from network); 17 Oct 2008 17:47:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 17 Oct 2008 17:47:29 -0000
Received: (qmail 34803 invoked by uid 500); 17 Oct 2008 17:47:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34776 invoked by uid 500); 17 Oct 2008 17:47:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 11449 invoked by uid 99); 17 Oct 2008 17:29:28 -0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bdelacretaz@gmail.com designates 74.125.44.30 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:sender
         :to:subject:cc:in-reply-to:mime-version:content-type
         :content-transfer-encoding:content-disposition:references
         :x-google-sender-auth;
        bh=1MqWIovlQGd0PY/5A7Lspjl6UhoANpFvdjRKSPml5nk=;
        b=S29QZPpRzfxTJG0/YC0Bs6/o0q44tT17NZi0uvHMsxgkoQIp+gRZYcVRM7hf1d9Wur
         fMUBY9UcDEYa4sjUrYlGM/c5xkpy2s+oG/MxRNoioQ005ukziUsIxVwJ6ZzSJaYeVXty
         3S3+fFuevVvyGLlS+lwB9ToVrwCzsv0OMpjPc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version
         :content-type:content-transfer-encoding:content-disposition
         :references:x-google-sender-auth;
        b=cej1Oag0YFSxj1vv27R/Q2BhssInf+Dkp41fM4/L0fAmxPUK0B/IffiaU3+xRoRQmq
         HRPE2lfPCqt5hbecxU0/YxgmNLdkfLLZOT34nuyFc8SWpz3ax3ipPISo8BptfJEYXXBF
         YqXA2GZHCFhEjMHcyTnZwUaQf8Ej+2e0xSfyw=
Message-ID: <f767f0600810171028j25cc8c6dsf1106b2f0cf6beb9@mail.gmail.com>
Date: Fri, 17 Oct 2008 19:28:48 +0200
From: "Bertrand Delacretaz" <bdelacretaz@apache.org>
Sender: bdelacretaz@gmail.com
To: pig-dev@incubator.apache.org
Subject: Re: [VOTE] Should we accept Pig as a Hadoop subproject?
Cc: general@hadoop.apache.org, general@incubator.apache.org
In-Reply-To: <357828F7-5CAC-40DE-96AA-CC14FBF18508@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <A2880476-79A5-4D72-9DD9-B07366F12801@apache.org>
	 <48F777AC.80509@apache.org>
	 <357828F7-5CAC-40DE-96AA-CC14FBF18508@apache.org>
X-Google-Sender-Auth: 80a47cb4fbb8853c
X-Virus-Checked: Checked by ClamAV on apache.org

On Fri, Oct 17, 2008 at 7:23 PM, Owen O'Malley <omalley@apache.org> wrote:
> Owen O'Malley wrote:
>
>> Should we accept Pig as a subproject?
>
> With six +1's from the Hadoop PMC, the vote passes. Hadoop will have a new
> subproject. *smile*

Note that according to [1] this requires a final "Graduation Approval
Vote" by the incubator PMC.

-Bertrand (*smile* as well, don't get me wrong ;-)

[1] http://incubator.apache.org/guides/graduation.html#subproject

From general-return-83-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 22 01:30:23 2008
Return-Path: <general-return-83-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 59576 invoked from network); 22 Oct 2008 01:30:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 22 Oct 2008 01:30:23 -0000
Received: (qmail 72414 invoked by uid 500); 22 Oct 2008 01:30:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72388 invoked by uid 500); 22 Oct 2008 01:30:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 43565 invoked by uid 99); 21 Oct 2008 22:39:50 -0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:cc:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:return-path:x-originalarrivaltime;
	b=XwIwaJ64xdUW9icCdxpvKYZHgG6SEQqDTKA4xyb1Z00MVMuYSE/EUWNM9sybetA+
User-Agent: Microsoft-Entourage/12.13.0.080930
Date: Tue, 21 Oct 2008 15:37:29 -0700
Subject: Re: Hadoop Camp next month
From: Milind Bhandarkar <milindb@yahoo-inc.com>
To: <core-user@hadoop.apache.org>, <general@hadoop.apache.org>
CC: <zookeeper-user@hadoop.apache.org>, <hbase-user@hadoop.apache.org>,
        <pig-user@incubator.apache.org>
Message-ID: <C523A7B9.5DD95%milindb@yahoo-inc.com>
Thread-Topic: Hadoop Camp next month
Thread-Index: AckkqOl125MZ2Ah2T/iPg34L8dZJSQPJK/VK
In-Reply-To: <57D7A94E-E310-4F65-82E6-858EBB1D14EE@yahoo-inc.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-OriginalArrivalTime: 21 Oct 2008 22:38:19.0875 (UTC) FILETIME=[B79F4330:01C933CD]
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

Just received an email from ApacheCon US organizers that they are giving 50%
discount for Hadoop Training
(http://us.apachecon.com/c/acus2008/sessions/93).

---- 
>We just created a discount code for you to give people 50% off of the
>cost of the training. The code is <  Hadoop50  >  The discount would
>apply only to the Training.
----

Hope to see you there !

- Milind


On 10/2/08 9:03 AM, "Owen O'Malley" <oom@yahoo-inc.com> wrote:

> Hi all,
>    I'd like to remind everyone that the Hadoop Camp & ApacheCon US is
> coming up in New Orleans next month. http://tinyurl.com/hadoop-camp
> 
> It will be the largest gathering of Hadoop developers outside of
> California. We'll have:
> 
> Core: Doug Cutting, Dhruba Borthakur, Arun Murthy, Owen O'Malley,
> Sameer Paranjpye,
>             Sanjay Radia, Tom White
> Zookeeper: Ben Reed
> 
> There will also be a training session on Practical Problem Solving
> with Hadoop by Milind Bhandarkar on Monday.
> 
> So if you'd like to meet the developers or find out more about Hadoop,
> come join us!
> 
> -- Owen


-- 
Milind Bhandarkar
Y!IM: GridSolutions
408-349-2136 
(milindb@yahoo-inc.com)


From general-return-84-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 22 01:30:46 2008
Return-Path: <general-return-84-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 59874 invoked from network); 22 Oct 2008 01:30:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 22 Oct 2008 01:30:46 -0000
Received: (qmail 72590 invoked by uid 500); 22 Oct 2008 01:30:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72568 invoked by uid 500); 22 Oct 2008 01:30:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 69054 invoked by uid 99); 22 Oct 2008 01:26:51 -0000
X-ASF-Spam-Status: No, hits=-8.0 required=10.0
	tests=RCVD_IN_DNSWL_HI,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of Jim.Kellerman@microsoft.com designates 131.107.115.215 as permitted sender)
From: "Jim Kellerman (POWERSET)" <Jim.Kellerman@microsoft.com>
To: Milind Bhandarkar <milindb@yahoo-inc.com>, "core-user@hadoop.apache.org"
	<core-user@hadoop.apache.org>, "general@hadoop.apache.org"
	<general@hadoop.apache.org>
CC: "zookeeper-user@hadoop.apache.org" <zookeeper-user@hadoop.apache.org>,
	"hbase-user@hadoop.apache.org" <hbase-user@hadoop.apache.org>,
	"pig-user@incubator.apache.org" <pig-user@incubator.apache.org>
Date: Tue, 21 Oct 2008 18:25:57 -0700
Subject: RE: Hadoop Camp next month
Thread-Topic: Hadoop Camp next month
Thread-Index: AckkqOl125MZ2Ah2T/iPg34L8dZJSQPJK/VKAAXiAMA=
Message-ID: <EAFDEC03CDA5D644878904F4A8F0158A5FE8612ED6@NA-EXMSG-C103.redmond.corp.microsoft.com>
References: <57D7A94E-E310-4F65-82E6-858EBB1D14EE@yahoo-inc.com>
 <C523A7B9.5DD95%milindb@yahoo-inc.com>
In-Reply-To: <C523A7B9.5DD95%milindb@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org



---
Jim Kellerman, Powerset (Live Search, Microsoft Corporation)


> -----Original Message-----
> From: Milind Bhandarkar [mailto:milindb@yahoo-inc.com]
> Sent: Tuesday, October 21, 2008 3:37 PM
> To: core-user@hadoop.apache.org; general@hadoop.apache.org
> Cc: zookeeper-user@hadoop.apache.org; hbase-user@hadoop.apache.org; pig-
> user@incubator.apache.org
> Subject: Re: Hadoop Camp next month
>
> Hi,
>
> Just received an email from ApacheCon US organizers that they are giving
> 50%
> discount for Hadoop Training
> (http://us.apachecon.com/c/acus2008/sessions/93).
>
> ----
> >We just created a discount code for you to give people 50% off of the
> >cost of the training. The code is <  Hadoop50  >  The discount would
> >apply only to the Training.
> ----
>
> Hope to see you there !
>
> - Milind
>
>
> On 10/2/08 9:03 AM, "Owen O'Malley" <oom@yahoo-inc.com> wrote:
>
> > Hi all,
> >    I'd like to remind everyone that the Hadoop Camp & ApacheCon US is
> > coming up in New Orleans next month. http://tinyurl.com/hadoop-camp
> >
> > It will be the largest gathering of Hadoop developers outside of
> > California. We'll have:
> >
> > Core: Doug Cutting, Dhruba Borthakur, Arun Murthy, Owen O'Malley,
> > Sameer Paranjpye,
> >             Sanjay Radia, Tom White
> > Zookeeper: Ben Reed
> >
> > There will also be a training session on Practical Problem Solving
> > with Hadoop by Milind Bhandarkar on Monday.
> >
> > So if you'd like to meet the developers or find out more about Hadoop,
> > come join us!
> >
> > -- Owen
>
>
> --
> Milind Bhandarkar
> Y!IM: GridSolutions
> 408-349-2136
> (milindb@yahoo-inc.com)


From general-return-85-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 22 07:52:49 2008
Return-Path: <general-return-85-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 18887 invoked from network); 22 Oct 2008 07:52:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 22 Oct 2008 07:52:49 -0000
Received: (qmail 51234 invoked by uid 500); 22 Oct 2008 07:52:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51183 invoked by uid 500); 22 Oct 2008 07:52:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51126 invoked by uid 99); 22 Oct 2008 07:52:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Oct 2008 00:52:51 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bdelacretaz@gmail.com designates 74.125.44.29 as permitted sender)
Received: from [74.125.44.29] (HELO yx-out-2324.google.com) (74.125.44.29)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Oct 2008 07:51:39 +0000
Received: by yx-out-2324.google.com with SMTP id 31so545529yxl.29
        for <general@hadoop.apache.org>; Wed, 22 Oct 2008 00:52:17 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:sender
         :to:subject:cc:in-reply-to:mime-version:content-type
         :content-transfer-encoding:content-disposition:references
         :x-google-sender-auth;
        bh=xXyfvL84L5MChJBeEFdLEYQHnuN8cCBqPqtbKVwvF8I=;
        b=mcDlwnHeTg7vbBccm8JkbhNQFallbeWY5vLLE51Sm6wKHgbOM8asr1O8e+Lf2plXtO
         JAwasqXB+M0x8Hzq5tOz4Tb3QYrhz6JxkAc4lf1Rvbf+WlHF/Zr3bdBCwcDjwJMBeTOE
         v8bhWdHXP1vi44bcQjxBnz8VfNt6qNvXq2mQo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version
         :content-type:content-transfer-encoding:content-disposition
         :references:x-google-sender-auth;
        b=HLJ3wQIMwVCmKnJcfOqQDI2MDyqdreLbM3MhvaFTABeWnU8Lunr8bDTM1qQ+akmgwc
         rBZhey2AKCbgwszntzqw0fzYvowccKTfAv9xt670+rcf9Ie1qyWPwNO33AnXpqQZTiXK
         waX5SsDC63bvB9REuw2Qc1CHkxPRccJehjWEw=
Received: by 10.150.95.15 with SMTP id s15mr1168257ybb.219.1224661937159;
        Wed, 22 Oct 2008 00:52:17 -0700 (PDT)
Received: by 10.150.227.21 with HTTP; Wed, 22 Oct 2008 00:52:17 -0700 (PDT)
Message-ID: <f767f0600810220052y31e412ber246dca03044f0392@mail.gmail.com>
Date: Wed, 22 Oct 2008 09:52:17 +0200
From: "Bertrand Delacretaz" <bdelacretaz@apache.org>
Sender: bdelacretaz@gmail.com
To: general@incubator.apache.org
Subject: [VOTE][RESULT] Accept Pig graduation as a Hadoop subproject
Cc: pig-dev@incubator.apache.org, general@hadoop.apache.org
In-Reply-To: <f767f0600810171144w3aaff127p65766a0a2a1a9662@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <f767f0600810171144w3aaff127p65766a0a2a1a9662@mail.gmail.com>
X-Google-Sender-Auth: 90569f5287dcac85
X-Virus-Checked: Checked by ClamAV on apache.org

On Fri, Oct 17, 2008 at 8:44 PM, Bertrand Delacretaz
<bdelacretaz@apache.org> wrote:
> ...The Pig [1] and Hadoop [2] communities have voted to graduate Pig as a
> Hadoop subproject.
>
> According to [3| the Incubator PMC needs to accept this graduation.
>
> Please cast your votes - as the weekend is starting, I'll tally the
> vote open until Wednesday October 22n morning UTC....

The vote passes with +1s from the following 9 incubator PMC members,
and no other votes:

Yoav Shapira (mentor)
Upayavira	
Doug Cutting (mentor)
Craig L Russell	
Paul Fremantle	
Jukka Zitting	
Martijn Dashorst
Matt Hogstrom
Bertrand Delacretaz (mentor)

Thanks everybody, and congratulations to the Pig team!

I will update the incubator website info accordingly, in the next few days.

-Bertrand

> [1] http://mail-archives.apache.org/mod_mbox/incubator-pig-dev/200810.mbox/%3C236D5828CF7F5749AF1BCB7F87596DE1029AF8FA@SNV-EXVS09.ds.corp.yahoo.com%3E
>
> [2] http://mail-archives.apache.org/mod_mbox/hadoop-general/200810.mbox/%3CA2880476-79A5-4D72-9DD9-B07366F12801@apache.org%3E
>
> [3] http://incubator.apache.org/guides/graduation.html#subproject
> [4] http://en.wikipedia.org/wiki/Pig

From general-return-86-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 22 14:26:13 2008
Return-Path: <general-return-86-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 36481 invoked from network); 22 Oct 2008 14:26:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 22 Oct 2008 14:26:13 -0000
Received: (qmail 47304 invoked by uid 500); 22 Oct 2008 14:26:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47233 invoked by uid 500); 22 Oct 2008 14:26:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47198 invoked by uid 99); 22 Oct 2008 14:26:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Oct 2008 07:26:14 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bdelacretaz@gmail.com designates 209.85.217.13 as permitted sender)
Received: from [209.85.217.13] (HELO mail-gx0-f13.google.com) (209.85.217.13)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Oct 2008 14:25:04 +0000
Received: by gxk6 with SMTP id 6so7383288gxk.5
        for <general@hadoop.apache.org>; Wed, 22 Oct 2008 07:24:41 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:sender
         :to:subject:cc:in-reply-to:mime-version:content-type
         :content-transfer-encoding:content-disposition:references
         :x-google-sender-auth;
        bh=J3tKJ3i6C+cDBu5eRAR/xumNWgo1A6A0sHhKh6SeHSU=;
        b=Qtic/FZahxgYveUNBONzX6p1xMnZ4KMdGLYjFT3fl/ZnObaoUR+tw7uXHlFmz/QKjg
         vHHNn9rFfDJta3Sst6RO3Bwv3rDgy+Fbg4c53K15l11jWIdqkA5/HCzUsscrf4fTW1gL
         GR51Vf3e43zpZV71JBsrhcCs61ihBoPI92rSs=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version
         :content-type:content-transfer-encoding:content-disposition
         :references:x-google-sender-auth;
        b=KsLvxN2/uqcAR699HxhaIQRsvGLAhvKyH2Hp6o7XAQ3RNy+ZH2gAtvIHvfnS5GIfS6
         qW7kkEziIoBrWWo5KuT0qP/6UCIdCETmgzmEawtSv3RgZTr8bKPa0xiwB9rqd41+MexG
         JOV5uajBjEr7NSnI9UYX1jdQrnVl+0Ie1X2k4=
Received: by 10.151.42.18 with SMTP id u18mr1753894ybj.186.1224685481373;
        Wed, 22 Oct 2008 07:24:41 -0700 (PDT)
Received: by 10.150.227.21 with HTTP; Wed, 22 Oct 2008 07:24:41 -0700 (PDT)
Message-ID: <f767f0600810220724s11c0a58axa82ddc4d1352c9db@mail.gmail.com>
Date: Wed, 22 Oct 2008 16:24:41 +0200
From: "Bertrand Delacretaz" <bdelacretaz@apache.org>
Sender: bdelacretaz@gmail.com
To: general@incubator.apache.org
Subject: Re: [VOTE][RESULT] Accept Pig graduation as a Hadoop subproject
Cc: pig-dev@incubator.apache.org, general@hadoop.apache.org
In-Reply-To: <f767f0600810220052y31e412ber246dca03044f0392@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <f767f0600810171144w3aaff127p65766a0a2a1a9662@mail.gmail.com>
	 <f767f0600810220052y31e412ber246dca03044f0392@mail.gmail.com>
X-Google-Sender-Auth: 14d7a7397cf519a7
X-Virus-Checked: Checked by ClamAV on apache.org

On Wed, Oct 22, 2008 at 9:52 AM, Bertrand Delacretaz
<bdelacretaz@apache.org> wrote:

> ...I will update the incubator website info accordingly, in the next few days...

Done, the following pages reflect pig's new status:

http://incubator.apache.org/projects/index.html
http://incubator.apache.org/clutch.html
http://incubator.apache.org/projects/pig.html

And I have also removed pig from
http://wiki.apache.org/incubator/ReportingSchedule

-Bertrand

From general-return-87-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 24 09:10:01 2008
Return-Path: <general-return-87-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 37387 invoked from network); 24 Oct 2008 09:10:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 24 Oct 2008 09:10:01 -0000
Received: (qmail 70540 invoked by uid 500); 24 Oct 2008 09:10:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70514 invoked by uid 500); 24 Oct 2008 09:10:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70502 invoked by uid 99); 24 Oct 2008 09:10:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Oct 2008 02:10:03 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of pinaki.poddar@gmail.com designates 64.233.182.189 as permitted sender)
Received: from [64.233.182.189] (HELO nf-out-0910.google.com) (64.233.182.189)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Oct 2008 09:08:49 +0000
Received: by nf-out-0910.google.com with SMTP id e27so392251nfd.9
        for <general@hadoop.apache.org>; Fri, 24 Oct 2008 02:09:18 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:to
         :subject:mime-version:content-type:content-transfer-encoding
         :content-disposition;
        bh=w4CRVQjPJVjbxOGa+3hR3RLrHYOVhsw4wpj0Wp8UQ84=;
        b=uWuy34rQz6n82VTonR8/3EeUNWsj9o82m07tbCAnp50tdMRWTNaQ2MwitgDFFy+Mv4
         ekk59pX6RQdVS6TIOTpHy+gkHTKEtVgs+D6OfIam+zXcLgI00ZDzvK9bBxyXgKl7nnkO
         P+RUdyV5BMWd6gfVkTJrekYWlmpB0+n7D/2hg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:mime-version:content-type
         :content-transfer-encoding:content-disposition;
        b=aiwdsWk9aZgN8K+Sv5Qhymj57QMV7miHU09bF2ZY8vGm8a+l7SYqdhRKKaaWvK3KzD
         oBeMwiFDv+X4p6+C4eheN+nmEgyoX66rxZYxgEJZn3kijfamf/AkfHLi9lxA6J5zXSQf
         sGZUA8Sri/zxfW5cPuLouVuCKN46kOfnJE6IM=
Received: by 10.210.128.5 with SMTP id a5mr1997147ebd.151.1224839358106;
        Fri, 24 Oct 2008 02:09:18 -0700 (PDT)
Received: by 10.210.10.16 with HTTP; Fri, 24 Oct 2008 02:09:18 -0700 (PDT)
Message-ID: <e24e2ec50810240209t3996b40avea2dcee787f4e92f@mail.gmail.com>
Date: Fri, 24 Oct 2008 04:09:18 -0500
From: "Pinaki Poddar" <pinaki.poddar@gmail.com>
To: general@hadoop.apache.org
Subject: broken links to Hadoop at ApacheCon presentations
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,
  Please note that one or more URL [1] under 'Hadoop at ApacheCon"  in
the front page [2] of Hadoop website is broken.

[1] http://eu.apachecon.com/eu2008/program/talk/2403
[2] http://hadoop.apache.org/index.html


Pinaki

From general-return-88-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 27 03:01:32 2008
Return-Path: <general-return-88-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 79008 invoked from network); 27 Oct 2008 03:01:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 27 Oct 2008 03:01:29 -0000
Received: (qmail 53735 invoked by uid 500); 27 Oct 2008 03:01:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53536 invoked by uid 500); 27 Oct 2008 03:01:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53403 invoked by uid 99); 27 Oct 2008 03:01:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Oct 2008 20:01:27 -0700
X-ASF-Spam-Status: No, hits=0.2 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [207.126.228.150] (HELO rsmtp2.corp.yahoo.com) (207.126.228.150)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Oct 2008 03:00:16 +0000
Received: from [10.72.78.125] (snvvpn2-10-72-78-c125.corp.yahoo.com [10.72.78.125])
	(authenticated bits=0)
	by rsmtp2.corp.yahoo.com (8.13.8/8.13.8/y.rout) with ESMTP id m9R30sHk062091
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Sun, 26 Oct 2008 20:00:54 -0700 (PDT)
Message-ID: <49052EE6.30809@apache.org>
Date: Sun, 26 Oct 2008 20:00:54 -0700
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.17 (X11/20080925)
MIME-Version: 1.0
To: announce@apache.org, zookeeper-dev@hadoop.apache.org,
        zookeeper-user@hadoop.apache.org, core-user@hadoop.apache.org,
        general@hadoop.apache.org
Subject: [ANNOUNCE] Apache ZooKeeper 3.0.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

The Apache ZooKeeper team is proud to announce our first official Apache 
release, version 3.0.0 of ZooKeeper.

ZooKeeper is a high-performance coordination service for distributed 
applications. It exposes common services - such as naming, configuration 
management, synchronization, and group services - in a simple interface 
so you don't have to write them from scratch. You can use it 
off-the-shelf to implement consensus, group management, leader election, 
and presence protocols. And you can build on it for your own, specific 
needs.

Version 3.0.0 is a major version upgrade from our previous 2.2.1 release 
on SourceForge, it addresses over 100 issues. If you are upgrading be 
sure to review the 3.0.0 release notes for migration instructions.

For ZooKeeper release details and downloads, visit:
http://hadoop.apache.org/zookeeper/releases.html

ZooKeeper 3.0.0 Release Notes and Migration Instructions are at:
http://hadoop.apache.org/zookeeper/docs/r3.0.0/releasenotes.html

Regards,

The ZooKeeper Team

From general-return-89-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 28 15:42:31 2008
Return-Path: <general-return-89-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 61429 invoked from network); 28 Oct 2008 15:42:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 28 Oct 2008 15:42:31 -0000
Received: (qmail 29885 invoked by uid 500); 28 Oct 2008 15:42:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29857 invoked by uid 500); 28 Oct 2008 15:42:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29846 invoked by uid 99); 28 Oct 2008 15:42:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Oct 2008 08:42:35 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [212.162.86.230] (HELO relay4.rai.it) (212.162.86.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Oct 2008 15:41:20 +0000
Received: from unknown (HELO smtp705.servizi.rai.it) ([10.24.1.135])
  by relay10.rai.it with ESMTP; 28 Oct 2008 16:40:58 +0100
Importance: normal
Received: from VRMTEULEXC710A.ict.corp.rai.it ([10.24.49.136]) by smtp705.servizi.rai.it with Microsoft SMTPSVC(6.0.3790.3959); Tue, 28 Oct 2008 16:40:58 +0100
Priority: normal
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.3790.4325
Content-Class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: Connection problems
Date: Tue, 28 Oct 2008 16:40:59 +0100
Message-ID: <163906E8B916DB4E8D15AE5225965EFEF2ED16@VRMTEULEXC710A.ict.corp.rai.it>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Connection problems
thread-index: Ack5E5Mq8cBRxZNoSHKI7Or3sJiDVA==
From: "Messina Alberto" <a.messina@rai.it>
To: <general@hadoop.apache.org>
X-OriginalArrivalTime: 28 Oct 2008 15:40:58.0301 (UTC) FILETIME=[92928ED0:01C93913]
X-Virus-Checked: Checked by ClamAV on apache.org

Hello,

I've just installed a minimal fully distributed hadoop pool with two
servers: a master (democrito) and a slave (pascal). The master runs on
port 24000. All firewalls are down, ssh passphraseless is ok, nothing in
the hosts.allow/hosts.deny. Configuration files are the following (resp.
on the master and on the slave):

<?xml version=3D"1.0"?>
<?xml-stylesheet type=3D"text/xsl" href=3D"configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <property>
    <name>fs.default.name</name>
    <value>democrito:24000</value>
  </property>
  <property>
    <name>mapred.job.tracker</name>
    <value>democrito:24001</value>
  </property>
  <property>
    <name>dfs.replication</name>
    <value>2</value>
  </property>
</configuration>

<?xml version=3D"1.0"?>
<?xml-stylesheet type=3D"text/xsl" href=3D"configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <property>
        <name>fs.default.name</name>
        <value>hdfs://democrito:24000</value>
  </property>
  <property>
    <name>mapred.job.tracker</name>
    <value>democrito:24001</value>
  </property>
  <property>
    <name>dfs.replication</name>
    <value>2</value>
  </property>

</configuration>


I've got "connection refused " problems when the slave tries to connect
to the master in the bootstrap phase:

2008-10-28 16:18:35,228 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: democrito/10.2.6.187:24000. Already tried 0 time(s).
2008-10-28 16:18:36,231 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: democrito/10.2.6.187:24000. Already tried 1 time(s).
2008-10-28 16:18:37,233 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: democrito/10.2.6.187:24000. Already tried 2 time(s).
2008-10-28 16:18:38,235 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: democrito/10.2.6.187:24000. Already tried 3 time(s).
2008-10-28 16:18:39,238 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: democrito/10.2.6.187:24000. Already tried 4 time(s).
2008-10-28 16:18:40,241 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: democrito/10.2.6.187:24000. Already tried 5 time(s).
2008-10-28 16:18:41,244 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: democrito/10.2.6.187:24000. Already tried 6 time(s).
2008-10-28 16:18:42,246 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: democrito/10.2.6.187:24000. Already tried 7 time(s).
2008-10-28 16:18:43,248 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: democrito/10.2.6.187:24000. Already tried 8 time(s).
2008-10-28 16:18:44,250 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: democrito/10.2.6.187:24000. Already tried 9 time(s).
2008-10-28 16:18:44,254 ERROR org.apache.hadoop.dfs.DataNode:
java.io.IOException: Call failed on local exception
        at org.apache.hadoop.ipc.Client.call(Client.java:718)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
        at org.apache.hadoop.dfs.$Proxy4.getProtocolVersion(Unknown
Source)
        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:319)
        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:306)
        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:343)
        at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:288)
        at
org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:244)
        at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:190)
        at
org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:2987)
        at
org.apache.hadoop.dfs.DataNode.instantiateDataNode(DataNode.java:2942)
        at
org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:2950)
        at org.apache.hadoop.dfs.DataNode.main(DataNode.java:3072)
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
        at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
        at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:300)
        at
org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:177)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:789)
        at org.apache.hadoop.ipc.Client.call(Client.java:704)
        ... 12 more

Can anyone help please?


Thank you

Alberto=20
--------------------------------------------------------

Questa e-mail, ed i suoi eventuali allegati, contengono informazioni =
confidenziali e riservate.=20
Se avete ricevuto questa comunicazione per errore non  utilizzatene il =
contenuto e non portatelo a conoscenza di alcuno. Siete inoltre pregati =
di eliminarla dalla vostra casella e avvisare il mittente.=20
E' da rilevare inoltre che l'attuale infrastruttura tecnologica non puo' =
garantire l'autenticita' del mittente, ne' tantomeno l'integrita' dei =
contenuti.=20
Opinioni, conclusioni ed altre informazioni contenute nel messaggio =
possono rappresentare punti di vista personali a meno di diversa =
esplicita indicazione autorizzata.
--------------------------------------------------------

From general-return-90-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 30 18:12:04 2008
Return-Path: <general-return-90-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 97195 invoked from network); 30 Oct 2008 18:12:04 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 30 Oct 2008 18:12:04 -0000
Received: (qmail 39119 invoked by uid 500); 30 Oct 2008 18:12:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38984 invoked by uid 500); 30 Oct 2008 18:12:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 38973 invoked by uid 99); 30 Oct 2008 18:12:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Oct 2008 11:12:09 -0700
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=SPF_HELO_PASS,SPF_PASS,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of gmackey@cs.ucf.edu designates 132.170.108.1 as permitted sender)
Received: from [132.170.108.1] (HELO longwood.cs.ucf.edu) (132.170.108.1)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Oct 2008 18:10:51 +0000
Received: from longwood.cs.ucf.edu (localhost [127.0.0.1])
	by longwood.cs.ucf.edu (8.14.1/8.14.1) with ESMTP id m9UI3msm012270
	for <general@hadoop.apache.org>; Thu, 30 Oct 2008 14:03:51 -0400 (EDT)
Received: (from prayer@localhost)
	by longwood.cs.ucf.edu (8.14.1/8.14.1/Submit) id m9UI3loA012268
	for general@hadoop.apache.org; Thu, 30 Oct 2008 14:03:47 -0400 (EDT)
X-Authentication-Warning: longwood.cs.ucf.edu: prayer set sender to gmackey@cs.ucf.edu using -f
From: gmackey@cs.ucf.edu
To: general@hadoop.apache.org
Subject: Re: Connection problems
Date: 30 Oct 2008 14:03:47 -0400
X-Mailer: Prayer v1.0.16
X-Originating-IP: [70.119.152.61]
In-Reply-To: <163906E8B916DB4E8D15AE5225965EFEF2ED16@VRMTEULEXC710A.ict.corp.rai.it>
Message-ID: <Prayer.1.0.16.0810301403470.8487@longwood.cs.ucf.edu>
References: <163906E8B916DB4E8D15AE5225965EFEF2ED16@VRMTEULEXC710A.ict.corp.rai.it>
Mime-Version: 1.0
Content-Type: text/plain; format=flowed; charset=ISO-8859-1
X-Spam-Level: ****
X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on gondor.cs.ucf.edu
X-Virus-Checked: Checked by ClamAV on apache.org
X-Old-Spam-Status: No, score=4.7 required=5.0 tests=FH_RELAY_NODNS,
	RCVD_IN_SORBS_DUL,RDNS_NONE,WEIRD_PORT autolearn=no version=3.2.4

Alberto,

Drop the 'hdfs,' hadoop will complain but it won't keep it from working 
(you can add it later). make sure that the full address of democrito is 
written out. So, for instance my cluster is scerola, so the headnode is 
<value>scerola.cs.ucf.edu:9000</value> (can also just use the ip address). 
Also the hadoop-site.xml needs to be identical for all nodes whether master 
or slave

regards

 - Grant

On Oct 28 2008, Messina Alberto wrote:

>Hello,
>
>I've just installed a minimal fully distributed hadoop pool with two
>servers: a master (democrito) and a slave (pascal). The master runs on
>port 24000. All firewalls are down, ssh passphraseless is ok, nothing in
>the hosts.allow/hosts.deny. Configuration files are the following (resp.
>on the master and on the slave):
>
><?xml version="1.0"?>
><?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
><!-- Put site-specific property overrides in this file. -->
>
><configuration>
>  <property>
>    <name>fs.default.name</name>
>    <value>democrito:24000</value>
>  </property>
>  <property>
>    <name>mapred.job.tracker</name>
>    <value>democrito:24001</value>
>  </property>
>  <property>
>    <name>dfs.replication</name>
>    <value>2</value>
>  </property>
></configuration>
>
><?xml version="1.0"?>
><?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
><!-- Put site-specific property overrides in this file. -->
>
><configuration>
>  <property>
>        <name>fs.default.name</name>
>        <value>hdfs://democrito:24000</value>
>  </property>
>  <property>
>    <name>mapred.job.tracker</name>
>    <value>democrito:24001</value>
>  </property>
>  <property>
>    <name>dfs.replication</name>
>    <value>2</value>
>  </property>
>
></configuration>
>
>
>I've got "connection refused " problems when the slave tries to connect
>to the master in the bootstrap phase:
>
>2008-10-28 16:18:35,228 INFO org.apache.hadoop.ipc.Client: Retrying
>connect to server: democrito/10.2.6.187:24000. Already tried 0 time(s).
>2008-10-28 16:18:36,231 INFO org.apache.hadoop.ipc.Client: Retrying
>connect to server: democrito/10.2.6.187:24000. Already tried 1 time(s).
>2008-10-28 16:18:37,233 INFO org.apache.hadoop.ipc.Client: Retrying
>connect to server: democrito/10.2.6.187:24000. Already tried 2 time(s).
>2008-10-28 16:18:38,235 INFO org.apache.hadoop.ipc.Client: Retrying
>connect to server: democrito/10.2.6.187:24000. Already tried 3 time(s).
>2008-10-28 16:18:39,238 INFO org.apache.hadoop.ipc.Client: Retrying
>connect to server: democrito/10.2.6.187:24000. Already tried 4 time(s).
>2008-10-28 16:18:40,241 INFO org.apache.hadoop.ipc.Client: Retrying
>connect to server: democrito/10.2.6.187:24000. Already tried 5 time(s).
>2008-10-28 16:18:41,244 INFO org.apache.hadoop.ipc.Client: Retrying
>connect to server: democrito/10.2.6.187:24000. Already tried 6 time(s).
>2008-10-28 16:18:42,246 INFO org.apache.hadoop.ipc.Client: Retrying
>connect to server: democrito/10.2.6.187:24000. Already tried 7 time(s).
>2008-10-28 16:18:43,248 INFO org.apache.hadoop.ipc.Client: Retrying
>connect to server: democrito/10.2.6.187:24000. Already tried 8 time(s).
>2008-10-28 16:18:44,250 INFO org.apache.hadoop.ipc.Client: Retrying
>connect to server: democrito/10.2.6.187:24000. Already tried 9 time(s).
>2008-10-28 16:18:44,254 ERROR org.apache.hadoop.dfs.DataNode:
>java.io.IOException: Call failed on local exception
>        at org.apache.hadoop.ipc.Client.call(Client.java:718)
>        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
>        at org.apache.hadoop.dfs.$Proxy4.getProtocolVersion(Unknown
>Source)
>        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:319)
>        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:306)
>        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:343)
>        at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:288)
>        at
>org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:244)
>        at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:190)
>        at
>org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:2987)
>        at
>org.apache.hadoop.dfs.DataNode.instantiateDataNode(DataNode.java:2942)
>        at
>org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:2950)
>        at org.apache.hadoop.dfs.DataNode.main(DataNode.java:3072)
>Caused by: java.net.ConnectException: Connection refused
>        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>        at
>sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>        at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
>        at
>org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:300)
>        at
>org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:177)
>        at org.apache.hadoop.ipc.Client.getConnection(Client.java:789)
>        at org.apache.hadoop.ipc.Client.call(Client.java:704)
>        ... 12 more
>
>Can anyone help please?
>
>
>Thank you
>
>Alberto 
>--------------------------------------------------------
>
> Questa e-mail, ed i suoi eventuali allegati, contengono informazioni 
> confidenziali e riservate. Se avete ricevuto questa comunicazione per 
> errore non utilizzatene il contenuto e non portatelo a conoscenza di 
> alcuno. Siete inoltre pregati di eliminarla dalla vostra casella e 
> avvisare il mittente. E' da rilevare inoltre che l'attuale infrastruttura 
> tecnologica non puo' garantire l'autenticita' del mittente, ne' tantomeno 
> l'integrita' dei contenuti. Opinioni, conclusioni ed altre informazioni 
> contenute nel messaggio possono rappresentare punti di vista personali a 
> meno di diversa esplicita indicazione autorizzata. 
> --------------------------------------------------------
>

-- 
Grant Mackey
UCF Researcher
Eng. III Rm238


From general-return-91-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 31 17:49:17 2008
Return-Path: <general-return-91-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 32617 invoked from network); 31 Oct 2008 17:49:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 31 Oct 2008 17:49:17 -0000
Received: (qmail 94412 invoked by uid 500); 31 Oct 2008 17:49:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94264 invoked by uid 500); 31 Oct 2008 17:49:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93784 invoked by uid 99); 31 Oct 2008 17:49:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Oct 2008 10:49:16 -0700
X-ASF-Spam-Status: No, hits=3.2 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Oct 2008 17:47:56 +0000
Received: from battlerock-lm.corp.yahoo.com (battlerock-lm.corp.yahoo.com [10.72.112.37])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id m9VHkvHS083782;
	Fri, 31 Oct 2008 10:46:57 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:content-type:mime-version:subject:date:cc:x-mailer;
	b=tDy6v01/y6NaaFCxBZM4y0NV0IEeQGH5wM1el04/TQoASXPjDjf9dXHtbA0yugzr
Message-Id: <EE75491E-4AEB-4554-9D53-99DAF8BB5236@yahoo-inc.com>
From: "Owen O'Malley" <oom@yahoo-inc.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=Apple-Mail-5-207398310
Mime-Version: 1.0 (Apple Message framework v929.2)
Subject: ApacheCon US 2008
Date: Fri, 31 Oct 2008 10:46:57 -0700
Cc: core-user@hadoop.apache.org
X-Mailer: Apple Mail (2.929.2)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-5-207398310
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit

Just a reminder that ApacheCon US is next week in New Orleans. There  
will be a lot of Hadoop developers and talks. (I'm CC'ing core-user  
because it has the widest coverage. Please join the low traffic  
general@hadoop list for cross sub-project announcements.)

     * Hadoop Camp with lots of talks about Hadoop
           o Introduction to Hadoop by Owen O'Malley
           o A Tour of Apache Hadoop by Tom White
           o Programming Hadoop Map/Reduce by Arun Murthy
           o Hadoop at Yahoo! by Eric Baldeschwieler
           o Hadoop Futures Panel
           o Using Hadoop for an Intranet Seach Engine by Shivakumar  
Vaithyanthan
           o Cloud Computing Testbed by Thomas Sandholm
           o Improving Virtualization and Performance Tracing of  
Hadoop with Open Solaris
               by George Porter
           o An Insight into Hadoop Usage at Facebook by Dhruba  
Borthakur
           o Pig by Alan Gates
           o Zookeeper, Coordinating the Distributed Application by  
Ben Reed
           o Querying JSON Data on Hadoop using Jaql by Kevin Beyer
           o HBase by Michael Stack
     * Hadoop training on Practical Problem Solving in Hadoop
     * Cloudera is providing a test Hadoop cluster and a Hadoop  
hacking contest.

There is also a new Hadoop tutorial available.

-- Owen
--Apple-Mail-5-207398310--

From general-return-92-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 31 18:25:47 2008
Return-Path: <general-return-92-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 61558 invoked from network); 31 Oct 2008 18:25:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 31 Oct 2008 18:25:46 -0000
Received: (qmail 42399 invoked by uid 500); 31 Oct 2008 18:25:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42325 invoked by uid 500); 31 Oct 2008 18:25:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42301 invoked by uid 99); 31 Oct 2008 18:25:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Oct 2008 11:25:46 -0700
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [208.97.132.83] (HELO spunkymail-a7.g.dreamhost.com) (208.97.132.83)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Oct 2008 18:24:31 +0000
Received: from [192.168.0.3] (adsl-074-229-189-244.sip.rmo.bellsouth.net [74.229.189.244])
	by spunkymail-a7.g.dreamhost.com (Postfix) with ESMTP id 0C5BA5B61B;
	Fri, 31 Oct 2008 11:24:40 -0700 (PDT)
Cc: general@hadoop.apache.org
Message-Id: <0A9C2702-EA2F-4F69-A225-36C82EC147D2@apache.org>
From: Grant Ingersoll <gsingers@apache.org>
To: core-user@hadoop.apache.org
In-Reply-To: <EE75491E-4AEB-4554-9D53-99DAF8BB5236@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v929.2)
Subject: Re: ApacheCon US 2008
Date: Fri, 31 Oct 2008 14:24:40 -0400
References: <EE75491E-4AEB-4554-9D53-99DAF8BB5236@yahoo-inc.com>
X-Mailer: Apple Mail (2.929.2)
X-Virus-Checked: Checked by ClamAV on apache.org

I will also be presenting on Mahout (machine learning) on Wednesday at  
3:30 (I think).  It will have some Hadoop flavor in it.

-Grant

On Oct 31, 2008, at 1:46 PM, Owen O'Malley wrote:

> Just a reminder that ApacheCon US is next week in New Orleans. There  
> will be a lot of Hadoop developers and talks. (I'm CC'ing core-user  
> because it has the widest coverage. Please join the low traffic  
> general@hadoop list for cross sub-project announcements.)
>
>    * Hadoop Camp with lots of talks about Hadoop
>          o Introduction to Hadoop by Owen O'Malley
>          o A Tour of Apache Hadoop by Tom White
>          o Programming Hadoop Map/Reduce by Arun Murthy
>          o Hadoop at Yahoo! by Eric Baldeschwieler
>          o Hadoop Futures Panel
>          o Using Hadoop for an Intranet Seach Engine by Shivakumar  
> Vaithyanthan
>          o Cloud Computing Testbed by Thomas Sandholm
>          o Improving Virtualization and Performance Tracing of  
> Hadoop with Open Solaris
>              by George Porter
>          o An Insight into Hadoop Usage at Facebook by Dhruba  
> Borthakur
>          o Pig by Alan Gates
>          o Zookeeper, Coordinating the Distributed Application by  
> Ben Reed
>          o Querying JSON Data on Hadoop using Jaql by Kevin Beyer
>          o HBase by Michael Stack
>    * Hadoop training on Practical Problem Solving in Hadoop
>    * Cloudera is providing a test Hadoop cluster and a Hadoop  
> hacking contest.
>
> There is also a new Hadoop tutorial available.
>
> -- Owen


From general-return-93-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 04 18:23:06 2008
Return-Path: <general-return-93-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 72695 invoked from network); 4 Nov 2008 18:23:06 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 4 Nov 2008 18:23:06 -0000
Received: (qmail 37534 invoked by uid 500); 4 Nov 2008 18:23:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37371 invoked by uid 500); 4 Nov 2008 18:23:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37324 invoked by uid 99); 4 Nov 2008 18:23:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Nov 2008 10:23:05 -0800
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Nov 2008 18:21:45 +0000
Received: from thickbeside-lm.corp.yahoo.com (thickbeside-lm.corp.yahoo.com [10.72.109.129])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id mA4ILxKS042274;
	Tue, 4 Nov 2008 10:21:59 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:content-type:
	content-transfer-encoding:mime-version:subject:date:x-mailer;
	b=k0F0BRu127Yc5QiYIN9z5sOfQnqI6dp8ZkxQgGkzrPPLPXQ5Tr6iPeOprAls3SaU
Message-Id: <A2FA9873-03BD-4994-BD1D-FF71851E60A1@yahoo-inc.com>
From: Nigel Daley <ndaley@yahoo-inc.com>
To: core-user@hadoop.apache.org, general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v929.2)
Subject: [ANNOUNCE] Hadoop release 0.18.2 available
Date: Tue, 4 Nov 2008 10:21:58 -0800
X-Mailer: Apple Mail (2.929.2)
X-Virus-Checked: Checked by ClamAV on apache.org

Release 0.18.2 fixes many critical bugs in 0.18.1.

For Hadoop release details and downloads, visit:
http://hadoop.apache.org/core/releases.html

Hadoop 0.18.2 Release Notes are at
http://hadoop.apache.org/core/docs/r0.18.2/releasenotes.html

Thanks to all who contributed to this release!

Nigel

From general-return-94-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Nov 15 00:49:20 2008
Return-Path: <general-return-94-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 13129 invoked from network); 15 Nov 2008 00:49:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 15 Nov 2008 00:49:20 -0000
Received: (qmail 46202 invoked by uid 500); 15 Nov 2008 00:49:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46031 invoked by uid 500); 15 Nov 2008 00:49:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45917 invoked by uid 99); 15 Nov 2008 00:49:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Nov 2008 16:49:19 -0800
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Nov 2008 00:47:56 +0000
Received: from SNV-EXBH01.ds.corp.yahoo.com (snv-exbh01.ds.corp.yahoo.com [207.126.227.249])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id mAF0kjHT003925;
	Fri, 14 Nov 2008 16:46:45 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:x-mimeole:content-class:mime-version:
	content-type:content-transfer-encoding:subject:date:message-id:
	in-reply-to:x-ms-has-attach:x-ms-tnef-correlator:thread-topic:
	thread-index:references:from:to:return-path:x-originalarrivaltime;
	b=B4GD7iyJ52kljX6qv8eY9DoG7NEn4EGsi0btBxxbUXLJvord7PwId064mkpjz8G0
Received: from SNV-EXVS02.ds.corp.yahoo.com ([216.145.51.196]) by SNV-EXBH01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Fri, 14 Nov 2008 16:46:44 -0800
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: Hadoop User Group (Bay Area) Nov 19th
Date: Fri, 14 Nov 2008 16:46:44 -0800
Message-ID: <DD27C9769EA63D43BE3CF4CCCC12DD9701D2460A@SNV-EXVS02.ds.corp.yahoo.com>
In-Reply-To: <AD0C16AAE446924586B68FC0EB421C9BAE153F34@SNV-EXVS02.ds.corp.yahoo.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Hadoop User Group (Bay Area) Nov 19th
Thread-Index: AckkqOob9DI4jbMCQEeTE1RcN+ztSQFk7y8wBx86AbA=
References: <57D7A94E-E310-4F65-82E6-858EBB1D14EE@yahoo-inc.com> <AD0C16AAE446924586B68FC0EB421C9BAE153F34@SNV-EXVS02.ds.corp.yahoo.com>
From: "Ajay Anand" <aanand@yahoo-inc.com>
To: <core-user@hadoop.apache.org>, <general@hadoop.apache.org>,
        <zookeeper-user@hadoop.apache.org>, <hbase-user@hadoop.apache.org>,
        <pig-user@incubator.apache.org>
X-OriginalArrivalTime: 15 Nov 2008 00:46:45.0006 (UTC) FILETIME=[A226F2E0:01C946BB]
X-Virus-Checked: Checked by ClamAV on apache.org

The next Bay Area Hadoop User Group meeting is scheduled for Wednesday,
November 19th at Yahoo! 2821 Mission College Blvd, Santa Clara, Building
1, Training Rooms 3 & 4 from 6:00-7:30 pm.

Join us for a talk on Chukwa, a Hadoop based large scale performance
monitoring system.

Please register at: http://upcoming.yahoo.com/event/1363930/

Look forward to seeing you there!
Ajay

From general-return-95-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 18 23:00:59 2008
Return-Path: <general-return-95-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 96053 invoked from network); 18 Nov 2008 23:00:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 18 Nov 2008 23:00:58 -0000
Received: (qmail 37607 invoked by uid 500); 18 Nov 2008 23:00:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37506 invoked by uid 500); 18 Nov 2008 23:00:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37398 invoked by uid 99); 18 Nov 2008 23:00:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Nov 2008 15:00:56 -0800
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Nov 2008 22:59:32 +0000
Received: from SNV-EXBH01.ds.corp.yahoo.com (snv-exbh01.ds.corp.yahoo.com [207.126.227.249])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id mAIMwn9P042621;
	Tue, 18 Nov 2008 14:58:49 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:x-mimeole:content-class:mime-version:
	content-type:content-transfer-encoding:subject:date:message-id:
	in-reply-to:x-ms-has-attach:x-ms-tnef-correlator:thread-topic:
	thread-index:references:from:to:return-path:x-originalarrivaltime;
	b=RWyg9xVgj4kcxM8M1hp2MDU1IAAgNLCLS+IKbTxQ1/wtqEuXZOwvOKi1WOTpLnwX
Received: from SNV-EXVS02.ds.corp.yahoo.com ([216.145.51.196]) by SNV-EXBH01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Tue, 18 Nov 2008 14:58:48 -0800
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: RE: Hadoop User Group (Bay Area) Nov 19th
Date: Tue, 18 Nov 2008 14:58:47 -0800
Message-ID: <DD27C9769EA63D43BE3CF4CCCC12DD9701D2479F@SNV-EXVS02.ds.corp.yahoo.com>
In-Reply-To: <AD0C16AAE446924586B68FC0EB421C9BAE154283@SNV-EXVS02.ds.corp.yahoo.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Hadoop User Group (Bay Area) Nov 19th
Thread-Index: AckkqOob9DI4jbMCQEeTE1RcN+ztSQFk7y8wBx86AbAAxZK/kA==
References: <57D7A94E-E310-4F65-82E6-858EBB1D14EE@yahoo-inc.com> <AD0C16AAE446924586B68FC0EB421C9BAE153F34@SNV-EXVS02.ds.corp.yahoo.com> <AD0C16AAE446924586B68FC0EB421C9BAE154283@SNV-EXVS02.ds.corp.yahoo.com>
From: "Ajay Anand" <aanand@yahoo-inc.com>
To: <core-user@hadoop.apache.org>, <general@hadoop.apache.org>,
        <zookeeper-user@hadoop.apache.org>, <hbase-user@hadoop.apache.org>,
        <pig-user@incubator.apache.org>
X-OriginalArrivalTime: 18 Nov 2008 22:58:48.0778 (UTC) FILETIME=[37ABFAA0:01C949D1]
X-Virus-Checked: Checked by ClamAV on apache.org

Please note that the room for this has been changed to Yahoo! Mission
College Building 2, Training Rooms 5 & 6.
Thanks
Ajay

-----Original Message-----
From: Ajay Anand=20
Sent: Friday, November 14, 2008 4:47 PM
To: 'core-user@hadoop.apache.org'; 'general@hadoop.apache.org';
'zookeeper-user@hadoop.apache.org'; 'hbase-user@hadoop.apache.org';
'pig-user@incubator.apache.org'
Subject: Hadoop User Group (Bay Area) Nov 19th

The next Bay Area Hadoop User Group meeting is scheduled for Wednesday,
November 19th at Yahoo! 2821 Mission College Blvd, Santa Clara, Building
1, Training Rooms 3 & 4 from 6:00-7:30 pm.

Join us for a talk on Chukwa, a Hadoop based large scale performance
monitoring system.

Please register at: http://upcoming.yahoo.com/event/1363930/

Look forward to seeing you there!
Ajay

From general-return-96-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 21 16:44:47 2008
Return-Path: <general-return-96-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 40024 invoked from network); 21 Nov 2008 16:44:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 21 Nov 2008 16:44:47 -0000
Received: (qmail 38249 invoked by uid 500); 21 Nov 2008 16:44:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38094 invoked by uid 500); 21 Nov 2008 16:44:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 38083 invoked by uid 99); 21 Nov 2008 16:44:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Nov 2008 08:44:55 -0800
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of alloe130@gmail.com designates 209.85.146.178 as permitted sender)
Received: from [209.85.146.178] (HELO wa-out-1112.google.com) (209.85.146.178)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Nov 2008 16:43:31 +0000
Received: by wa-out-1112.google.com with SMTP id n7so559735wag.29
        for <general@hadoop.apache.org>; Fri, 21 Nov 2008 08:44:16 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:to
         :subject:in-reply-to:mime-version:content-type:references;
        bh=I+HCrO6DcEMkM6IdZgZ1YVQXMdEtjY4S8ifgmrHoIHU=;
        b=gg8tefUx/d/6DAbjWrqG6I6nwwnxyWVn6SbOIRLSrICdd2akC0SQL0JRi458j+1YrG
         MTcvxqAIYVsDE2boXH0C8sOnXYYIszjiZ29L5MFsysop0YRZlBanfOi6bq7dB3Ko2qrl
         ugjA7eP/1tyoS0/dhCnIaNoGb4f4xj098T5Es=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version
         :content-type:references;
        b=F0DUFTbDrJ+LV9yUuR/k5ll9RaTJeA7OvWRT35Lbim64kmu63jIpq1G6KRxC0MBMJH
         4p0pkfZ97kLTq/aFBwx73c3qHYDPvVIX/ZPTu9PPJ2/nQmnJ6lekVxvdkyggkZH6rfwj
         W2bGA/ww9PaYI4PYTO3N3ux1aEuoWh1uD8LE4=
Received: by 10.114.133.1 with SMTP id g1mr417487wad.21.1227285856573;
        Fri, 21 Nov 2008 08:44:16 -0800 (PST)
Received: by 10.114.176.16 with HTTP; Fri, 21 Nov 2008 08:44:16 -0800 (PST)
Message-ID: <92adf0e80811210844r4bbd3eeob988a668f3cb7ca@mail.gmail.com>
Date: Sat, 22 Nov 2008 01:44:16 +0900
From: "Donguk Choi" <alloe130@gmail.com>
To: general@hadoop.apache.org
Subject: Re: Hadoop User Group (Bay Area) Nov 19th
In-Reply-To: <DD27C9769EA63D43BE3CF4CCCC12DD9701D2479F@SNV-EXVS02.ds.corp.yahoo.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; 
	boundary="----=_Part_24656_26313052.1227285856563"
References: <57D7A94E-E310-4F65-82E6-858EBB1D14EE@yahoo-inc.com>
	 <AD0C16AAE446924586B68FC0EB421C9BAE153F34@SNV-EXVS02.ds.corp.yahoo.com>
	 <AD0C16AAE446924586B68FC0EB421C9BAE154283@SNV-EXVS02.ds.corp.yahoo.com>
	 <DD27C9769EA63D43BE3CF4CCCC12DD9701D2479F@SNV-EXVS02.ds.corp.yahoo.com>
X-Virus-Checked: Checked by ClamAV on apache.org

------=_Part_24656_26313052.1227285856563
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Thank you for your help :)

On 11/19/08, Ajay Anand <aanand@yahoo-inc.com> wrote:
>
> Please note that the room for this has been changed to Yahoo! Mission
> College Building 2, Training Rooms 5 & 6.
> Thanks
> Ajay
>
> -----Original Message-----
> From: Ajay Anand
> Sent: Friday, November 14, 2008 4:47 PM
> To: 'core-user@hadoop.apache.org'; 'general@hadoop.apache.org';
> 'zookeeper-user@hadoop.apache.org'; 'hbase-user@hadoop.apache.org';
> 'pig-user@incubator.apache.org'
> Subject: Hadoop User Group (Bay Area) Nov 19th
>
> The next Bay Area Hadoop User Group meeting is scheduled for Wednesday,
> November 19th at Yahoo! 2821 Mission College Blvd, Santa Clara, Building
> 1, Training Rooms 3 & 4 from 6:00-7:30 pm.
>
> Join us for a talk on Chukwa, a Hadoop based large scale performance
> monitoring system.
>
> Please register at: http://upcoming.yahoo.com/event/1363930/
>
> Look forward to seeing you there!
> Ajay
>



-- 
Best. Regards,
Donguk Choi
http://www.alloes.com

------=_Part_24656_26313052.1227285856563--

From general-return-97-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 24 23:38:12 2008
Return-Path: <general-return-97-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 35076 invoked from network); 24 Nov 2008 23:38:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 24 Nov 2008 23:38:12 -0000
Received: (qmail 15850 invoked by uid 500); 24 Nov 2008 23:38:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15772 invoked by uid 500); 24 Nov 2008 23:38:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15304 invoked by uid 99); 24 Nov 2008 23:38:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Nov 2008 15:38:15 -0800
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Nov 2008 23:36:48 +0000
Received: from wlan-snve-153-138.corp.yahoo.com (snvvpn1-10-72-72-c126.hq.corp.yahoo.com [10.72.72.126])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id mAONbAVn072966;
	Mon, 24 Nov 2008 15:37:10 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:content-type:
	content-transfer-encoding:mime-version:subject:date:x-mailer;
	b=wwd8M0JaWsn4ErU6DB4QmmLgIoSgUQ5/4x4O3/KDN8uYGZmLI2gbiJH4j+eTztos
Message-Id: <43A4FA57-5660-4212-BD17-34996ED59600@yahoo-inc.com>
From: Nigel Daley <ndaley@yahoo-inc.com>
To: core-user@hadoop.apache.org, general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v929.2)
Subject: [ANNOUNCE] Hadoop release 0.19.0 available
Date: Mon, 24 Nov 2008 15:37:10 -0800
X-Mailer: Apple Mail (2.929.2)
X-Virus-Checked: Checked by ClamAV on apache.org

Release 0.19.0 contains many improvements, new features, bug fixes and  
optimizations.

For release details and downloads, visit:

  http://hadoop.apache.org/core/releases.html

Thanks to all who contributed to this release!

Nigel

From general-return-14-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 06 05:30:55 2008
Return-Path: <general-return-14-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 72384 invoked from network); 6 Aug 2008 05:30:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 6 Aug 2008 05:30:55 -0000
Received: (qmail 68946 invoked by uid 500); 6 Aug 2008 05:30:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68927 invoked by uid 500); 6 Aug 2008 05:30:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 65434 invoked by uid 99); 6 Aug 2008 05:23:19 -0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:content-type:
	content-transfer-encoding:mime-version:subject:date:x-mailer;
	b=cFrC3KyBonB0TFA0ckXVqfE28RRLSH5CZouUJLUFuqulDVqoykE+5QJ2LDSOMJgQ
Message-Id: <688CEB2A-13EF-4C38-855A-3D6D4C3485B3@yahoo-inc.com>
From: "Owen O'Malley" <oom@yahoo-inc.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v926)
Subject: [VOTE] Should we create sub-projects for HDFS and Map/Reduce?
Date: Tue, 5 Aug 2008 22:18:14 -0700
X-Mailer: Apple Mail (2.926)
X-Virus-Checked: Checked by ClamAV on apache.org

I think the time has come to split Hadoop Core into three pieces:

   1. Core (src/core)
   2. HDFS (src/hdfs)
   3. Map/Reduce (src/mapred)

There will be lots of details to work out, such as what we do with  
tools and contrib, but I think it is a good idea. This will create  
separate jiras and mailing lists for HDFS and map/reduce, which will  
make the community much more approachable. I would propose that we  
wait until 0.19.0 is released to give us time to plan the split.

-- Owen

From general-return-15-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 06 05:39:16 2008
Return-Path: <general-return-15-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 76413 invoked from network); 6 Aug 2008 05:39:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 6 Aug 2008 05:39:15 -0000
Received: (qmail 77046 invoked by uid 500); 6 Aug 2008 05:39:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76987 invoked by uid 500); 6 Aug 2008 05:39:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76976 invoked by uid 99); 6 Aug 2008 05:39:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Aug 2008 22:39:15 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 209.85.128.190 as permitted sender)
Received: from [209.85.128.190] (HELO fk-out-0910.google.com) (209.85.128.190)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Aug 2008 05:38:19 +0000
Received: by fk-out-0910.google.com with SMTP id 26so2559383fkx.13
        for <general@hadoop.apache.org>; Tue, 05 Aug 2008 22:38:45 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:to
         :subject:in-reply-to:mime-version:content-type
         :content-transfer-encoding:content-disposition:references;
        bh=5YYa7WGd4hbkN5lpq6Tcmbvc/j1069+Jjl6Eiw4rXV0=;
        b=BHPwzA8uRSbbaFSUzenrHZdFpu8rRAP+BXaZvK2nmktiWp3e6svNk8c3ZMxYwhQQy4
         ywB+oBK9wO2mL7r9OiXt098+GZ7D19RcwGgRaLXNgGqZupIslh8rOfpzMwS+r3bVc8oQ
         QFFt/z5dgmsJO4zbbxJEvUR1sLfD7KeVRFCro=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version
         :content-type:content-transfer-encoding:content-disposition
         :references;
        b=krKX9E3bNIFOlbyM+3L4anpegme9zPCzOXss1dAtbox/vLJFok2Kd7NZk3NhmenjJE
         BHonO7X0rhqCsE/Hiu6tOfGOlESOElzK0J/2bKs8fxDjpukN44xRMNisrugYbKjaba5H
         Q/4CuG0syCz2WyHx85cjXsWE4aEZrs931C2IA=
Received: by 10.125.87.8 with SMTP id p8mr101867mkl.80.1218001125082;
        Tue, 05 Aug 2008 22:38:45 -0700 (PDT)
Received: by 10.125.145.7 with HTTP; Tue, 5 Aug 2008 22:38:45 -0700 (PDT)
Message-ID: <4aa34eb70808052238sb272613v6766a69b76537c33@mail.gmail.com>
Date: Tue, 5 Aug 2008 21:38:45 -0800
From: "Dhruba Borthakur" <dhruba@gmail.com>
To: general@hadoop.apache.org
Subject: Re: [VOTE] Should we create sub-projects for HDFS and Map/Reduce?
In-Reply-To: <688CEB2A-13EF-4C38-855A-3D6D4C3485B3@yahoo-inc.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <688CEB2A-13EF-4C38-855A-3D6D4C3485B3@yahoo-inc.com>
X-Virus-Checked: Checked by ClamAV on apache.org

Are you talking about sub-projects for core, hdfs and mapreduce? Or is
there another way to allow for having separate mailing lists/jiras for
these components?

I had liked the fact that these pieces are together. It makes the code
compile together, keeps API upgradation simple and enhances developer
community building across all these three pieces of code. Actually,
JIRAs have their own components and we can always filter them using
their component, can't we?

thanks,
dhruba

On Tue, Aug 5, 2008 at 9:18 PM, Owen O'Malley <oom@yahoo-inc.com> wrote:
> I think the time has come to split Hadoop Core into three pieces:
>
>  1. Core (src/core)
>  2. HDFS (src/hdfs)
>  3. Map/Reduce (src/mapred)
>
> There will be lots of details to work out, such as what we do with tools and
> contrib, but I think it is a good idea. This will create separate jiras and
> mailing lists for HDFS and map/reduce, which will make the community much
> more approachable. I would propose that we wait until 0.19.0 is released to
> give us time to plan the split.
>
> -- Owen
>

From general-return-16-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 06 06:14:15 2008
Return-Path: <general-return-16-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 94323 invoked from network); 6 Aug 2008 06:14:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 6 Aug 2008 06:14:15 -0000
Received: (qmail 30164 invoked by uid 500); 6 Aug 2008 06:14:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30148 invoked by uid 500); 6 Aug 2008 06:14:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30136 invoked by uid 99); 6 Aug 2008 06:14:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Aug 2008 23:14:14 -0700
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Aug 2008 06:13:16 +0000
Received: from [10.0.0.200] (snvvpn2-10-72-78-c47.corp.yahoo.com [10.72.78.47])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id m766CE8d052301
	for <general@hadoop.apache.org>; Tue, 5 Aug 2008 23:12:14 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=GBIL8IiId+KMaammv1yoirJkpAab716ODWkD/pSBpY+h6OC/Ukb/3hVei8QOVgkw
Message-Id: <46A3BDE2-E00E-4632-B30F-26304E5E24F9@yahoo-inc.com>
From: "Owen O'Malley" <oom@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <4aa34eb70808052238sb272613v6766a69b76537c33@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v926)
Subject: Re: [VOTE] Should we create sub-projects for HDFS and Map/Reduce?
Date: Tue, 5 Aug 2008 23:12:13 -0700
References: <688CEB2A-13EF-4C38-855A-3D6D4C3485B3@yahoo-inc.com> <4aa34eb70808052238sb272613v6766a69b76537c33@mail.gmail.com>
X-Mailer: Apple Mail (2.926)
X-Virus-Checked: Checked by ClamAV on apache.org


On Aug 5, 2008, at 10:38 PM, Dhruba Borthakur wrote:

> Are you talking about sub-projects for core, hdfs and mapreduce?

Sorry, I wasn't clear. Yes, I mean creating new sub-projects. There  
will clearly be overlap between the sub-projects in terms of  
committers. In fact, all current Hadoop committers would be  on all 3.  
It just feels like getting 100+ emails a day is overwhelming and most  
of the developers specialize on one side or the other. So the division  
seems pretty natural to me.

-- Owen

From general-return-17-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 06 09:46:31 2008
Return-Path: <general-return-17-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 33216 invoked from network); 6 Aug 2008 09:46:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 6 Aug 2008 09:46:31 -0000
Received: (qmail 84995 invoked by uid 500); 6 Aug 2008 09:46:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84969 invoked by uid 500); 6 Aug 2008 09:46:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84958 invoked by uid 99); 6 Aug 2008 09:46:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Aug 2008 02:46:30 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of guosijie@gmail.com designates 209.85.142.188 as permitted sender)
Received: from [209.85.142.188] (HELO ti-out-0910.google.com) (209.85.142.188)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Aug 2008 09:45:33 +0000
Received: by ti-out-0910.google.com with SMTP id d27so1049728tid.9
        for <general@hadoop.apache.org>; Wed, 06 Aug 2008 02:45:42 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=INwQcSfPeQcYLdcRrv33FbzhYMLUG0dAwa0Am/OUAqw=;
        b=veV/lsIHytSvjpLGAg1abzUEb6MPVfOom0GpQA23J9er12XbnlfFj7c6hVGyxj0oOl
         0orfJ3bF8MoZSJHFavjuawSMQ4ypCAc2OZiC/ZpU9MwKelTFv+8ON1uBqwwLRlvXAr3p
         hwd6+ow5Nsj79qEiQ6H0MDKgK/rEzuhAUdBIw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:content-type:content-transfer-encoding;
        b=FHkJsh/oH6eJ19rtm6L7+eYdP6z0Zcl3v2HqG7vDZfeHxUyfD5NZAFMU61b3XgM8aJ
         ftCs0HGRER9wVgp6nbrT6fy8OCpH8m0yGbbBSESq4Q8+nDo8Xcb5gW7HAw/Bvr8tIvca
         unri5ilL8+KiW0iVj2zeLuvnAuq9c3hKyVAD4=
Received: by 10.110.7.18 with SMTP id 18mr2236559tig.39.1218015942654;
        Wed, 06 Aug 2008 02:45:42 -0700 (PDT)
Received: from ?10.10.11.253? ( [159.226.41.129])
        by mx.google.com with ESMTPS id j5sm959045tid.12.2008.08.06.02.45.40
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Wed, 06 Aug 2008 02:45:41 -0700 (PDT)
Message-ID: <489972C7.3060803@gmail.com>
Date: Wed, 06 Aug 2008 17:45:43 +0800
From: Samuel Guo <guosijie@gmail.com>
User-Agent: Thunderbird 2.0.0.12 (Windows/20080213)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Should we create sub-projects for HDFS and Map/Reduce?
References: <688CEB2A-13EF-4C38-855A-3D6D4C3485B3@yahoo-inc.com> <4aa34eb70808052238sb272613v6766a69b76537c33@mail.gmail.com> <46A3BDE2-E00E-4632-B30F-26304E5E24F9@yahoo-inc.com>
In-Reply-To: <46A3BDE2-E00E-4632-B30F-26304E5E24F9@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Owen O'Malley wrote:
>
> On Aug 5, 2008, at 10:38 PM, Dhruba Borthakur wrote:
>
>> Are you talking about sub-projects for core, hdfs and mapreduce?
>
> Sorry, I wasn't clear. Yes, I mean creating new sub-projects. There 
> will clearly be overlap between the sub-projects in terms of 
> committers. In fact, all current Hadoop committers would be  on all 3. 
> It just feels like getting 100+ emails a day is overwhelming and most 
> of the developers specialize on one side or the other. So the division 
> seems pretty natural to me.
>
> -- Owen
It sounds a good idea.

From general-return-18-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 06 17:12:30 2008
Return-Path: <general-return-18-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 65244 invoked from network); 6 Aug 2008 17:12:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 6 Aug 2008 17:12:28 -0000
Received: (qmail 74462 invoked by uid 500); 6 Aug 2008 17:12:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74441 invoked by uid 500); 6 Aug 2008 17:12:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 69098 invoked by uid 99); 6 Aug 2008 17:08:00 -0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cutting@gmail.com designates 209.85.198.239 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding:sender;
        bh=A+qx4lL3GpPrHgRQBMcYu4VGYM1Dh4a/ymY7phiqbKQ=;
        b=Dzc9WbwDphIaqvU2F3CdDFA/g5NAQNf7VK35QTvw7jCE7qHlMH5TgiExbnsiIRKq8C
         QnxjF0Fe1mhqlJPQNWU4Im6bqecpCr80vPi5SnBxFoZ/D+mt/Jt6UZlx/OU02YN5KAwu
         8rWUi+UzKXVjCnfroGNbampTrHZOmE6fHTo/k=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:content-type:content-transfer-encoding:sender;
        b=D768kQBmuYICRDDNLB6uLWXeOWSAdlgbmehQWpNvSKFJ0RYfTz7yr4tkva3TaveXPD
         JZcpMOx3PvCQ770nB0dnZfTFwZ5hT+d8XO0FJR62z3N8KjMKOb6x/IDSaPsxaTjYbUS8
         vEQ7ob/L7WcY3Jh0Bw6g6wUyU8+aeFSiMv4tE=
Message-ID: <4899DA52.4@apache.org>
Date: Wed, 06 Aug 2008 10:07:30 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.16 (X11/20080724)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Should we create sub-projects for HDFS and Map/Reduce?
References: <688CEB2A-13EF-4C38-855A-3D6D4C3485B3@yahoo-inc.com>
In-Reply-To: <688CEB2A-13EF-4C38-855A-3D6D4C3485B3@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Sender: Doug Cutting <cutting@gmail.com>
X-Virus-Checked: Checked by ClamAV on apache.org

+1

I agree that it is time to do this.  Should we start using Ivy, so that 
the inter-dependencies are easier to manage?

Doug

Owen O'Malley wrote:
> I think the time has come to split Hadoop Core into three pieces:
> 
>   1. Core (src/core)
>   2. HDFS (src/hdfs)
>   3. Map/Reduce (src/mapred)
> 
> There will be lots of details to work out, such as what we do with tools 
> and contrib, but I think it is a good idea. This will create separate 
> jiras and mailing lists for HDFS and map/reduce, which will make the 
> community much more approachable. I would propose that we wait until 
> 0.19.0 is released to give us time to plan the split.
> 
> -- Owen

From general-return-19-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 06 17:19:55 2008
Return-Path: <general-return-19-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 68447 invoked from network); 6 Aug 2008 17:19:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 6 Aug 2008 17:19:51 -0000
Received: (qmail 92679 invoked by uid 500); 6 Aug 2008 17:19:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92665 invoked by uid 500); 6 Aug 2008 17:19:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 92654 invoked by uid 99); 6 Aug 2008 17:19:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Aug 2008 10:19:51 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 74.125.92.27 as permitted sender)
Received: from [74.125.92.27] (HELO qw-out-2122.google.com) (74.125.92.27)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Aug 2008 17:18:55 +0000
Received: by qw-out-2122.google.com with SMTP id 3so208762qwe.35
        for <general@hadoop.apache.org>; Wed, 06 Aug 2008 10:19:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:to
         :subject:in-reply-to:mime-version:content-type
         :content-transfer-encoding:content-disposition:references;
        bh=dbPWfAUF+EHNobo0WldEwLtKzgl+XSvmrfsEC9tZuUs=;
        b=lz5eeuBIs7VhSDZMdcpBF/Uz48uWq6Kizppppu6RJHEt/671fsQTWYL4COa0/r0syf
         WC3rvEyEX+ZcEKD7kfJHqgzs+eYn3EP6t4kIaZW5dxUeFHuyCvPkIb38G4mcPiB9Nwt2
         w/Q5laj+anqf3OP3CCmF5qBSJg1axP9GIHGVA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version
         :content-type:content-transfer-encoding:content-disposition
         :references;
        b=YjiXDDMospne0ZlAYVqAXxGmsG+Fiyg54yKfKkKH2qoL2t8XpE19Sr0fe2nQSAc/3s
         420oiLow7jSmoR2XyPZIIFjdfXzn8TrdZ5i1q6VWkW+75SYpdzxjEaQZ0fb4hsLoWINS
         mTT/ruRsTeQ+Fq9ipwGqn2i7Ju1SOZBok0Lxw=
Received: by 10.150.97.19 with SMTP id u19mr3711798ybb.165.1218043145126;
        Wed, 06 Aug 2008 10:19:05 -0700 (PDT)
Received: by 10.150.220.4 with HTTP; Wed, 6 Aug 2008 10:19:05 -0700 (PDT)
Message-ID: <4aa34eb70808061019o77dc85cbg6b9ec52d694992bb@mail.gmail.com>
Date: Wed, 6 Aug 2008 10:19:05 -0700
From: "Dhruba Borthakur" <dhruba@gmail.com>
To: general@hadoop.apache.org
Subject: Re: [VOTE] Should we create sub-projects for HDFS and Map/Reduce?
In-Reply-To: <4899DA52.4@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <688CEB2A-13EF-4C38-855A-3D6D4C3485B3@yahoo-inc.com>
	 <4899DA52.4@apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

+1.

-dhruba

On Wed, Aug 6, 2008 at 10:07 AM, Doug Cutting <cutting@apache.org> wrote:
> +1
>
> I agree that it is time to do this.  Should we start using Ivy, so that the
> inter-dependencies are easier to manage?
>
> Doug
>
> Owen O'Malley wrote:
>>
>> I think the time has come to split Hadoop Core into three pieces:
>>
>>  1. Core (src/core)
>>  2. HDFS (src/hdfs)
>>  3. Map/Reduce (src/mapred)
>>
>> There will be lots of details to work out, such as what we do with tools
>> and contrib, but I think it is a good idea. This will create separate jiras
>> and mailing lists for HDFS and map/reduce, which will make the community
>> much more approachable. I would propose that we wait until 0.19.0 is
>> released to give us time to plan the split.
>>
>> -- Owen
>

From general-return-20-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 06 20:30:17 2008
Return-Path: <general-return-20-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 13690 invoked from network); 6 Aug 2008 20:30:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 6 Aug 2008 20:30:17 -0000
Received: (qmail 8101 invoked by uid 500); 6 Aug 2008 20:30:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8083 invoked by uid 500); 6 Aug 2008 20:30:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8072 invoked by uid 99); 6 Aug 2008 20:30:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Aug 2008 13:30:16 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 74.125.92.26 as permitted sender)
Received: from [74.125.92.26] (HELO qw-out-2122.google.com) (74.125.92.26)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Aug 2008 20:29:21 +0000
Received: by qw-out-2122.google.com with SMTP id 3so217903qwe.35
        for <general@hadoop.apache.org>; Wed, 06 Aug 2008 13:29:47 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:to
         :subject:in-reply-to:mime-version:content-type
         :content-transfer-encoding:content-disposition:references;
        bh=ecs6VB5dUW1pNtqZFE6KHdEQ9A2HEbVIlLsDbQqVDH8=;
        b=dD+xcxITUt4RiHPX6ulRF7+x3Y1+CwgxXHx5oL0ehS4eGO1I3woIDyheCTydBI4YAV
         gdXGkCDGUlrkslVEgAIgMUz1qvaRw06ouuoLFmlUQ+NGtaS7JQOL5ssvI172R1eFgf8E
         hs3N3+3k6oMYFU+g4DCCXpG5rk1uYWXz2FxeQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version
         :content-type:content-transfer-encoding:content-disposition
         :references;
        b=DNXdhvqxP5GxvI+k3j0xADNDJ1fQcCbNfGACKdlDpEfbmevodJyMls26/BTrW2Mi3G
         ABoyGYOYx+jd2UY+HgjLzbyLcW4xYAVDeVlZUnZXEqEOL6Cxct3cBYK+uVx68Hx9vJ7M
         q1s7VP4nBXAiM2H5zh8LEb4lcrypag9rpSlgA=
Received: by 10.150.139.15 with SMTP id m15mr4005706ybd.135.1218054587709;
        Wed, 06 Aug 2008 13:29:47 -0700 (PDT)
Received: by 10.150.220.4 with HTTP; Wed, 6 Aug 2008 13:29:47 -0700 (PDT)
Message-ID: <4aa34eb70808061329h35964613i5a75cc33780749bb@mail.gmail.com>
Date: Wed, 6 Aug 2008 13:29:47 -0700
From: "Dhruba Borthakur" <dhruba@gmail.com>
To: general@hadoop.apache.org
Subject: Re: [VOTE] Should we create sub-projects for HDFS and Map/Reduce?
In-Reply-To: <4aa34eb70808061019o77dc85cbg6b9ec52d694992bb@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <688CEB2A-13EF-4C38-855A-3D6D4C3485B3@yahoo-inc.com>
	 <4899DA52.4@apache.org>
	 <4aa34eb70808061019o77dc85cbg6b9ec52d694992bb@mail.gmail.com>
X-Virus-Checked: Checked by ClamAV on apache.org

What about releases? Does this mean that each sub-project will be
released separately?  If so, then the life of an administrator becomes
even more harder :-). he has to pick and choose each package, verify
whether they are compatible with one another, run various installation
utilities to install them, etc.etc.

-dhruba

On Wed, Aug 6, 2008 at 10:19 AM, Dhruba Borthakur <dhruba@gmail.com> wrote:
> +1.
>
> -dhruba
>
> On Wed, Aug 6, 2008 at 10:07 AM, Doug Cutting <cutting@apache.org> wrote:
>> +1
>>
>> I agree that it is time to do this.  Should we start using Ivy, so that the
>> inter-dependencies are easier to manage?
>>
>> Doug
>>
>> Owen O'Malley wrote:
>>>
>>> I think the time has come to split Hadoop Core into three pieces:
>>>
>>>  1. Core (src/core)
>>>  2. HDFS (src/hdfs)
>>>  3. Map/Reduce (src/mapred)
>>>
>>> There will be lots of details to work out, such as what we do with tools
>>> and contrib, but I think it is a good idea. This will create separate jiras
>>> and mailing lists for HDFS and map/reduce, which will make the community
>>> much more approachable. I would propose that we wait until 0.19.0 is
>>> released to give us time to plan the split.
>>>
>>> -- Owen
>>
>

From general-return-21-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 06 20:37:00 2008
Return-Path: <general-return-21-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 17702 invoked from network); 6 Aug 2008 20:37:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 6 Aug 2008 20:37:00 -0000
Received: (qmail 13953 invoked by uid 500); 6 Aug 2008 20:36:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13934 invoked by uid 500); 6 Aug 2008 20:36:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13923 invoked by uid 99); 6 Aug 2008 20:36:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Aug 2008 13:36:59 -0700
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Aug 2008 20:36:01 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id m76KaFLZ063922
	for <general@hadoop.apache.org>; Wed, 6 Aug 2008 13:36:15 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=0IhEodC1G6qFQX31UXTQt7+UvF/rYr/BuaVO2f0ily4pP8EUeEQzonyO3MqpOKz3
Message-Id: <787AD7EE-369C-42B5-A99E-A75CEB6673E5@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <4aa34eb70808061329h35964613i5a75cc33780749bb@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v928.1)
Subject: Re: [VOTE] Should we create sub-projects for HDFS and Map/Reduce?
Date: Wed, 6 Aug 2008 13:36:15 -0700
References: <688CEB2A-13EF-4C38-855A-3D6D4C3485B3@yahoo-inc.com> <4899DA52.4@apache.org> <4aa34eb70808061019o77dc85cbg6b9ec52d694992bb@mail.gmail.com> <4aa34eb70808061329h35964613i5a75cc33780749bb@mail.gmail.com>
X-Mailer: Apple Mail (2.928.1)
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Arun

>>> Owen O'Malley wrote:
>>>>
>>>> I think the time has come to split Hadoop Core into three pieces:
>>>>
>>>> 1. Core (src/core)
>>>> 2. HDFS (src/hdfs)
>>>> 3. Map/Reduce (src/mapred)
>>>>
>>>> There will be lots of details to work out, such as what we do  
>>>> with tools
>>>> and contrib, but I think it is a good idea. This will create  
>>>> separate jiras
>>>> and mailing lists for HDFS and map/reduce, which will make the  
>>>> community
>>>> much more approachable. I would propose that we wait until 0.19.0  
>>>> is
>>>> released to give us time to plan the split.
>>>>
>>>> -- Owen
>>>
>>


From general-return-22-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 06 20:55:31 2008
Return-Path: <general-return-22-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 29061 invoked from network); 6 Aug 2008 20:55:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 6 Aug 2008 20:55:30 -0000
Received: (qmail 32983 invoked by uid 500); 6 Aug 2008 20:55:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32964 invoked by uid 500); 6 Aug 2008 20:55:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 27494 invoked by uid 99); 6 Aug 2008 20:49:57 -0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type:Message-ID;
  b=nCmTGYZjyez7IJpNhYCiAgzAeg0/A7xEjnSVKgE4lHvFXAgceCjyZs84St1cXsxnrf7fbfB2RMZtOWDDustefT1g/Z4R6LPpOjhwB3/xdBy+LOW7KgolNoWrFSbwqFawmYJmBXXcYLrDEObN3Zw9Eu7+3BC5rSRTo2jNPFPydJU=;
X-Mailer: YahooMailRC/1042.48 YahooMailWebService/0.7.218
Date: Wed, 6 Aug 2008 13:48:27 -0700 (PDT)
From: lohit <lohit_bv@yahoo.com>
Subject: Re: [VOTE] Should we create sub-projects for HDFS and Map/Reduce?
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Message-ID: <394209.75538.qm@web53604.mail.re2.yahoo.com>
X-Virus-Checked: Checked by ClamAV on apache.org

On similar note, would Core continue to have things not part of HDFS and Map/Reduce? Would it still be called core. Map/reduce and HDFS are supposed to be 'core' of hadoop, right :)
-Lohit



----- Original Message ----
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
Sent: Wednesday, August 6, 2008 1:36:15 PM
Subject: Re: [VOTE] Should we create sub-projects for HDFS and Map/Reduce?

+1

Arun

>>> Owen O'Malley wrote:
>>>>
>>>> I think the time has come to split Hadoop Core into three pieces:
>>>>
>>>> 1. Core (src/core)
>>>> 2. HDFS (src/hdfs)
>>>> 3. Map/Reduce (src/mapred)
>>>>
>>>> There will be lots of details to work out, such as what we do  
>>>> with tools
>>>> and contrib, but I think it is a good idea. This will create  
>>>> separate jiras
>>>> and mailing lists for HDFS and map/reduce, which will make the  
>>>> community
>>>> much more approachable. I would propose that we wait until 0.19.0  
>>>> is
>>>> released to give us time to plan the split.
>>>>
>>>> -- Owen
>>>
>>

From general-return-23-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 07 21:30:53 2008
Return-Path: <general-return-23-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 55580 invoked from network); 7 Aug 2008 21:30:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 7 Aug 2008 21:30:53 -0000
Received: (qmail 72186 invoked by uid 500); 7 Aug 2008 21:30:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72169 invoked by uid 500); 7 Aug 2008 21:30:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72158 invoked by uid 99); 7 Aug 2008 21:30:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Aug 2008 14:30:52 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cutting@gmail.com designates 209.85.198.231 as permitted sender)
Received: from [209.85.198.231] (HELO rv-out-0506.google.com) (209.85.198.231)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Aug 2008 21:29:56 +0000
Received: by rv-out-0506.google.com with SMTP id k40so1070962rvb.29
        for <general@hadoop.apache.org>; Thu, 07 Aug 2008 14:30:24 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding:sender;
        bh=xLG2BUalTgO2VVhjnhcwZehXmreEinoX/sjs+GJhAuw=;
        b=bHIcNnBKoSLs7G+FeKxHb4T5SCOINM0WoX4GZk2hvxUqdc+/c2jjvEk1uDIEDnz/17
         EWdB8Z4ptu3pxgnz3/3Mmq01AUCw1EAF8w4IN5OVRzJuIvN1YGEzzgY+myvI0Ezqf5oc
         0VhSsiyHT42jiUTkUoDCslRS0iwzpsOhgYe8Y=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:content-type:content-transfer-encoding:sender;
        b=jw0ba3T9oQ2Fla0W+k4S92JD0GORlEj8n9h5e5owj3ZczEIqzzAgwwmAH+sk6SH43P
         T7/PYrMNMgSL2ikTgBoSSjo27AY1i4EASRvOeEN7L8UTmpYkgO5hhO65dtCnsVxxvaVH
         cVWftBteWS4eck+HQwp0HwnTb+WSJBe0PrDts=
Received: by 10.140.166.21 with SMTP id o21mr934012rve.254.1218144624386;
        Thu, 07 Aug 2008 14:30:24 -0700 (PDT)
Received: from ?192.168.168.16? ( [76.103.181.218])
        by mx.google.com with ESMTPS id k2sm348086rvb.4.2008.08.07.14.30.23
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Thu, 07 Aug 2008 14:30:23 -0700 (PDT)
Message-ID: <489B696E.8090504@apache.org>
Date: Thu, 07 Aug 2008 14:30:22 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.16 (X11/20080724)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Should we create sub-projects for HDFS and Map/Reduce?
References: <688CEB2A-13EF-4C38-855A-3D6D4C3485B3@yahoo-inc.com>	 <4899DA52.4@apache.org>	 <4aa34eb70808061019o77dc85cbg6b9ec52d694992bb@mail.gmail.com> <4aa34eb70808061329h35964613i5a75cc33780749bb@mail.gmail.com>
In-Reply-To: <4aa34eb70808061329h35964613i5a75cc33780749bb@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Sender: Doug Cutting <cutting@gmail.com>
X-Virus-Checked: Checked by ClamAV on apache.org

Dhruba Borthakur wrote:
> What about releases? Does this mean that each sub-project will be
> released separately?

Yes.  Although we might coordinate and release in waves.  It would 
increase the importance of back-compatibility.

> If so, then the life of an administrator becomes
> even more harder :-). he has to pick and choose each package, verify
> whether they are compatible with one another, run various installation
> utilities to install them, etc.etc.

We intend to use Ivy assist with compatibility.  It would be silly 
release a new X version A that requires Y version B if Y version B has 
not yet been released.  So upgrading should not be any harder than it is 
today: to upgrade mapreduce you might need to upgrade hdfs to a 
compatible version.  Today you always have to upgrade both.  So, in some 
cases, upgrades should get easier, since not everything need be upgraded 
at once.

Doug

From general-return-24-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 07 23:24:45 2008
Return-Path: <general-return-24-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 16310 invoked from network); 7 Aug 2008 23:24:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 7 Aug 2008 23:24:44 -0000
Received: (qmail 16000 invoked by uid 500); 7 Aug 2008 23:24:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15983 invoked by uid 500); 7 Aug 2008 23:24:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15972 invoked by uid 99); 7 Aug 2008 23:24:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Aug 2008 16:24:43 -0700
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Aug 2008 23:23:45 +0000
Received: from thickbeside-lm.corp.yahoo.com (thickbeside-lm.corp.yahoo.com [10.72.109.129])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id m77NMLjp091830
	for <general@hadoop.apache.org>; Thu, 7 Aug 2008 16:22:21 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=u7ysTfUwVlOKMhjoxaw17/Yjl2wbfOKEwS/Hz/mbMngQVegqZqWe7+Fu29YVbqTL
Message-Id: <2DA5A7B1-0CC0-4C4F-BE80-AE1E3C7B47A3@yahoo-inc.com>
From: Nigel Daley <ndaley@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <688CEB2A-13EF-4C38-855A-3D6D4C3485B3@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v928.1)
Subject: Re: [VOTE] Should we create sub-projects for HDFS and Map/Reduce?
Date: Thu, 7 Aug 2008 16:22:20 -0700
References: <688CEB2A-13EF-4C38-855A-3D6D4C3485B3@yahoo-inc.com>
X-Mailer: Apple Mail (2.928.1)
X-Virus-Checked: Checked by ClamAV on apache.org

So we'll need to create and maintain 3 patch processes, one for each  
component?  Not a trivial amount of work given the way the patch  
process is currently structured.

How will unit tests be divided?  For instance, will all three have to  
have MiniDFSCluster and other shared test infrastructure?

We can use Ivy now to manage dependencies on outside libraries.
We can build separate jars for mapred, hdfs, and core right now.
We can use email filters to reduce inbox emails.
We can use TestNG to categorize our tests and narrow the number of  
unit tests run for each component.

-1 until I better understand the benefit of making the split.

Nige

On Aug 5, 2008, at 10:18 PM, Owen O'Malley wrote:

> I think the time has come to split Hadoop Core into three pieces:
>
>  1. Core (src/core)
>  2. HDFS (src/hdfs)
>  3. Map/Reduce (src/mapred)
>
> There will be lots of details to work out, such as what we do with  
> tools and contrib, but I think it is a good idea. This will create  
> separate jiras and mailing lists for HDFS and map/reduce, which will  
> make the community much more approachable. I would propose that we  
> wait until 0.19.0 is released to give us time to plan the split.
>
> -- Owen


From general-return-25-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 08 05:24:54 2008
Return-Path: <general-return-25-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 832 invoked from network); 8 Aug 2008 05:24:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 8 Aug 2008 05:24:54 -0000
Received: (qmail 81073 invoked by uid 500); 8 Aug 2008 05:24:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81050 invoked by uid 500); 8 Aug 2008 05:24:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81039 invoked by uid 99); 8 Aug 2008 05:24:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Aug 2008 22:24:53 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 209.85.128.184 as permitted sender)
Received: from [209.85.128.184] (HELO fk-out-0910.google.com) (209.85.128.184)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Aug 2008 05:23:58 +0000
Received: by fk-out-0910.google.com with SMTP id 26so561089fkx.13
        for <general@hadoop.apache.org>; Thu, 07 Aug 2008 22:24:08 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:to
         :subject:in-reply-to:mime-version:content-type
         :content-transfer-encoding:content-disposition:references;
        bh=q/ebHjCQuz4QxTEjlGjbS7VWowUtrFbL+v5yzvOimBQ=;
        b=Kxa0BbKtpKHw+YrJ2l1wyvb9kZV8yIITOCnVAYHQSffSY1djjqzLqhbpcnQNLw9QWa
         CHUSqZIwanSIbjTIEdaBHfAwObkN6WBuoEqyDGsPjd8+8Bs+21b76s+vSRMv/uRuoekZ
         UTtpGtJfXxqDnRRIP+BQ7T0SaNVLZetzFVwR4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version
         :content-type:content-transfer-encoding:content-disposition
         :references;
        b=x4pgAGSRqM/FJChBME06nEzQi+4lhjLZCk4+jmtxGkVbyjGpqVsp/Jojx5uNwQxsd7
         /aREsF7ZyAikLhNnO1FP8/OSJlBgMDIXi8+vhYf5vHtLl2rQbpV9o4w8YT3HG4KTVHSO
         eT80y8UhirUKnwMZ1dGohNvtmGc8lIIn7YuMQ=
Received: by 10.180.213.14 with SMTP id l14mr1938371bkg.55.1218173048177;
        Thu, 07 Aug 2008 22:24:08 -0700 (PDT)
Received: by 10.125.155.7 with HTTP; Thu, 7 Aug 2008 22:24:08 -0700 (PDT)
Message-ID: <4aa34eb70808072224h1c77aab2vd87c1d475a435171@mail.gmail.com>
Date: Thu, 7 Aug 2008 21:24:08 -0800
From: "Dhruba Borthakur" <dhruba@gmail.com>
To: general@hadoop.apache.org
Subject: Re: [VOTE] Should we create sub-projects for HDFS and Map/Reduce?
In-Reply-To: <2DA5A7B1-0CC0-4C4F-BE80-AE1E3C7B47A3@yahoo-inc.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <688CEB2A-13EF-4C38-855A-3D6D4C3485B3@yahoo-inc.com>
	 <2DA5A7B1-0CC0-4C4F-BE80-AE1E3C7B47A3@yahoo-inc.com>
X-Virus-Checked: Checked by ClamAV on apache.org

I too am "-1" on this one. I think we should try to see if some of the
pieces in contrib deserve being a subproject but we can keep hdfs and
map-reduce together. I think it reduces complexity from a deployment
perspective too.

If we can use the "component" of the JIRA in the subject of the email,
then it leads itself to very easy email filtering. Does anyone know
(or advice me) on how to make the "component" be part of the email
subject?

thanks,
dhruba

On Thu, Aug 7, 2008 at 3:22 PM, Nigel Daley <ndaley@yahoo-inc.com> wrote:
> So we'll need to create and maintain 3 patch processes, one for each
> component?  Not a trivial amount of work given the way the patch process is
> currently structured.
>
> How will unit tests be divided?  For instance, will all three have to have
> MiniDFSCluster and other shared test infrastructure?
>
> We can use Ivy now to manage dependencies on outside libraries.
> We can build separate jars for mapred, hdfs, and core right now.
> We can use email filters to reduce inbox emails.
> We can use TestNG to categorize our tests and narrow the number of unit
> tests run for each component.
>
> -1 until I better understand the benefit of making the split.
>
> Nige
>
> On Aug 5, 2008, at 10:18 PM, Owen O'Malley wrote:
>
>> I think the time has come to split Hadoop Core into three pieces:
>>
>>  1. Core (src/core)
>>  2. HDFS (src/hdfs)
>>  3. Map/Reduce (src/mapred)
>>
>> There will be lots of details to work out, such as what we do with tools
>> and contrib, but I think it is a good idea. This will create separate jiras
>> and mailing lists for HDFS and map/reduce, which will make the community
>> much more approachable. I would propose that we wait until 0.19.0 is
>> released to give us time to plan the split.
>>
>> -- Owen
>
>

From general-return-26-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 08 09:08:42 2008
Return-Path: <general-return-26-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 78788 invoked from network); 8 Aug 2008 09:08:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 8 Aug 2008 09:08:42 -0000
Received: (qmail 15607 invoked by uid 500); 8 Aug 2008 09:08:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15591 invoked by uid 500); 8 Aug 2008 09:08:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15580 invoked by uid 99); 8 Aug 2008 09:08:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Aug 2008 02:08:41 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of tom.e.white@gmail.com designates 209.85.198.227 as permitted sender)
Received: from [209.85.198.227] (HELO rv-out-0506.google.com) (209.85.198.227)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Aug 2008 09:07:46 +0000
Received: by rv-out-0506.google.com with SMTP id k40so1437819rvb.29
        for <general@hadoop.apache.org>; Fri, 08 Aug 2008 02:08:13 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:to
         :subject:in-reply-to:mime-version:content-type
         :content-transfer-encoding:content-disposition:references;
        bh=FkExk3uPKrttl8lyqvOhHNdaJbpgUEexNbDYNo8NClo=;
        b=RrD170B6CPZnJm3wQEiiNn9L7wtlxrDsDYSAp+lu+o5N2D3iZ6gOmGu+weXegBGouM
         gmvHzxWPre9LwO0mFBp7aYVwxgfScX0hC8uEb7b0we+lvMfDvv0z7jiGSRHHNwhCvT05
         En3MgO6iiE7XjOEHkJvUmcH4k5wTu2Cpi+fYU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version
         :content-type:content-transfer-encoding:content-disposition
         :references;
        b=qCHi7LJ3jXUSqNm9S4DXbhx30Gciv34nyEKBFBulg/BYzzsCZmDan6boo0jT05J8t/
         8ygy+a/RNg/4y/18rwBKQU1FE+wbIKlJXWKf22yQy1H7DBi0pttiXstqTw7WIYVnBS/l
         T2NfJIrNIxa2oheKxx3uY5+Gsn8aefzE31JvI=
Received: by 10.141.141.3 with SMTP id t3mr1296534rvn.124.1218186493515;
        Fri, 08 Aug 2008 02:08:13 -0700 (PDT)
Received: by 10.141.132.6 with HTTP; Fri, 8 Aug 2008 02:08:13 -0700 (PDT)
Message-ID: <b23e01b50808080208k355953b8l76baf9ffc5bafbc3@mail.gmail.com>
Date: Fri, 8 Aug 2008 10:08:13 +0100
From: "Tom White" <tom.e.white@gmail.com>
To: general@hadoop.apache.org
Subject: Re: [VOTE] Should we create sub-projects for HDFS and Map/Reduce?
In-Reply-To: <2DA5A7B1-0CC0-4C4F-BE80-AE1E3C7B47A3@yahoo-inc.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <688CEB2A-13EF-4C38-855A-3D6D4C3485B3@yahoo-inc.com>
	 <2DA5A7B1-0CC0-4C4F-BE80-AE1E3C7B47A3@yahoo-inc.com>
X-Virus-Checked: Checked by ClamAV on apache.org

+1

I was initially concerned about the overhead of having to install
separate packages for each component, but in some ways it will make
things clearer. Folks on the user list are often asking how to use
HDFS by itself for instance - or even if it is possible. By splitting
it up it would make it clear that HDFS and MapReduce can be used
without the other (although of course, they are best used together).
Also, I can see some benefit from having separate configuration for
HDFS and MapReduce, since it will make the configuration files smaller
and more manageable (something like hdfs-(default|site).xml,
mapreduce-(default|site).xml).

It's not totally clear to me how Core fits into this. It's just a jar
file and doesn't have daemons, so it should be bundled with the
MapReduce and HDFS releases, shouldn't it?

Nigel Daley <ndaley@yahoo-inc.com> wrote:
> How will unit tests be divided?  For instance, will all three have to have
> MiniDFSCluster and other shared test infrastructure?

Today the tests for core, hdfs and mapred are under one source tree
because they are so tightly intertwined. I think the goal should be to
have independent unit tests for each module, as well as integration
tests that test that MapReduce works with HDFS. We should do this even
if we don't split the projects.

Tom

From general-return-27-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 08 20:07:08 2008
Return-Path: <general-return-27-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 31900 invoked from network); 8 Aug 2008 20:07:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 8 Aug 2008 20:07:08 -0000
Received: (qmail 40581 invoked by uid 500); 8 Aug 2008 20:07:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40554 invoked by uid 500); 8 Aug 2008 20:07:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40540 invoked by uid 99); 8 Aug 2008 20:07:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Aug 2008 13:07:07 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cutting@gmail.com designates 209.85.198.224 as permitted sender)
Received: from [209.85.198.224] (HELO rv-out-0506.google.com) (209.85.198.224)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Aug 2008 20:06:11 +0000
Received: by rv-out-0506.google.com with SMTP id k40so1902465rvb.29
        for <general@hadoop.apache.org>; Fri, 08 Aug 2008 13:06:39 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding:sender;
        bh=0U9Y6sanHJvFdamwbNtY3bii2QHl01rJaKoyMPf5Yq0=;
        b=k6OdpeVTGHSSBmEpVAOumUF0NTehogRsL2uv/7o4Lv3zLHx+IwtEpSzE1BIxMbHqw4
         aMBUwKJMLU03azIWs3CVW6et3cD9Q1xMEXiO8dRD2CPB7YgtmJbaWnnRIMrDsus6PjUU
         FwCXI/O0tFfaDb1RoeP8TpNSoaXXqlrvyuYoQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:content-type:content-transfer-encoding:sender;
        b=U3l2F71QsbEYmtvwT6m9Sbi4fl8vGuxI7jy9tXonCTmud3O8fs2oTRYFrR9CCaVDn8
         XdsBEPKSAaikrK0yvo/pSo2+CPPJ/tccbr1ccbK27zUDiw5JMZ3/HqC6u5KkcHRmcKJx
         U7BV6iSaljgINlleKwDowBvymSnfZCgyq9TgM=
Received: by 10.141.88.3 with SMTP id q3mr1674688rvl.3.1218225999073;
        Fri, 08 Aug 2008 13:06:39 -0700 (PDT)
Received: from ?192.168.168.16? ( [76.103.181.218])
        by mx.google.com with ESMTPS id b8sm2568874rvf.8.2008.08.08.13.06.38
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Fri, 08 Aug 2008 13:06:38 -0700 (PDT)
Message-ID: <489CA74F.8000901@apache.org>
Date: Fri, 08 Aug 2008 13:06:39 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.16 (X11/20080724)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Should we create sub-projects for HDFS and Map/Reduce?
References: <688CEB2A-13EF-4C38-855A-3D6D4C3485B3@yahoo-inc.com> <2DA5A7B1-0CC0-4C4F-BE80-AE1E3C7B47A3@yahoo-inc.com>
In-Reply-To: <2DA5A7B1-0CC0-4C4F-BE80-AE1E3C7B47A3@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Sender: Doug Cutting <cutting@gmail.com>
X-Virus-Checked: Checked by ClamAV on apache.org

Nigel Daley wrote:
> How will unit tests be divided?  For instance, will all three have to 
> have MiniDFSCluster and other shared test infrastructure?

The HDFS project can release an hdfs-test.jar file that contains 
MiniDFSCluster.  This will be used by mapred tests.  Similarly, mapred 
will release a mapred-test.jar that contains MiniMRCluster, which can be 
used by hdfs tests.  There is a circular dependency, but only in the 
test code, not in the mapred or hdfs code itself.  This is easy to 
enforce, since test code is not on the classpath when we compile 
non-test code.

> -1 until I better understand the benefit of making the split.

One benefit is that developers would spend less time reading messages 
about areas they're not interested in.  The core-dev mailing list 
traffic is becoming unmanageable.  Splitting these without splitting the 
project would mean that a split developer community would attempt to 
build a coherent product, which sounds dangerous.

Another benefit is that it would increase the separation of these 
technologies, so that, e.g., folks could more easily run different 
versions of mapreduce on top of different versions of HDFS.  Currently 
we make no such guarantees.  Folks would be able to upgrade to, e.g., 
the next release of mapreduce on a subset of their cluster without 
upgrading their HDFS.  That's not currently supported.  As we move 
towards splitting mapreduce into a scheduler and runtime, where folks 
can specify a different runtime per job, this will be even more critical.

We need to make this split eventually.  Why not now?

Doug

From general-return-28-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 12 06:31:30 2008
Return-Path: <general-return-28-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 79878 invoked from network); 12 Aug 2008 06:31:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 12 Aug 2008 06:31:30 -0000
Received: (qmail 68434 invoked by uid 500); 12 Aug 2008 06:31:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68415 invoked by uid 500); 12 Aug 2008 06:31:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68404 invoked by uid 99); 12 Aug 2008 06:31:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Aug 2008 23:31:28 -0700
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of alloe130@gmail.com designates 64.233.184.233 as permitted sender)
Received: from [64.233.184.233] (HELO wr-out-0506.google.com) (64.233.184.233)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Aug 2008 06:30:33 +0000
Received: by wr-out-0506.google.com with SMTP id c30so2191365wra.21
        for <general@hadoop.apache.org>; Mon, 11 Aug 2008 23:31:00 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:to
         :subject:mime-version:content-type;
        bh=3JkkHWCQFi78vAeFrvEK3gWHPizsaA8zgW6ZHfBnP0M=;
        b=xa74sXKxbYrR4BKN6sMKK7DEAO7gjvfu+SzcEyZV6E2OjNBJmGbNEJ6YDVBLCycEw8
         b8frObERUC2Fmk9r6lt4/BJifAxQMvboyax8tNOeuszhkZHeTBV2QPVCwaYNAzfRLzcA
         Iabz0R6QP6wRfgSkxdnFrZ2wpgaKWENs7yo/k=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:mime-version:content-type;
        b=f0by//4nSVRylzerVYLj0lFckzZ8stHQYidW5Hu073BtGt39L68zt0Tz7DGdskwEOx
         Qhk+oafE3BJ1gxc5cdKPnJexl/aFO3Q6ODsYYn4L6xdAnphFOlxgbXEcuwpu5NFeUKg0
         xUTdentNSOuiwwuB/xG6vW2ncOGa9kD52nfJY=
Received: by 10.90.55.9 with SMTP id d9mr9828599aga.23.1218522660116;
        Mon, 11 Aug 2008 23:31:00 -0700 (PDT)
Received: by 10.90.113.12 with HTTP; Mon, 11 Aug 2008 23:31:00 -0700 (PDT)
Message-ID: <92adf0e80808112331q546ccaecrbe69a64edb942ed3@mail.gmail.com>
Date: Tue, 12 Aug 2008 15:31:00 +0900
From: "Donguk Choi" <alloe130@gmail.com>
To: general@hadoop.apache.org
Subject: join
MIME-Version: 1.0
Content-Type: multipart/alternative; 
	boundary="----=_Part_95802_30869858.1218522660122"
X-Virus-Checked: Checked by ClamAV on apache.org

------=_Part_95802_30869858.1218522660122
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

short mail
-- 
Best. Regards,
Donguk Choi
http://www.alloes.com

------=_Part_95802_30869858.1218522660122--

From general-return-29-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 15 21:00:52 2008
Return-Path: <general-return-29-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 14884 invoked from network); 15 Aug 2008 21:00:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 15 Aug 2008 21:00:52 -0000
Received: (qmail 99605 invoked by uid 500); 15 Aug 2008 21:00:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99586 invoked by uid 500); 15 Aug 2008 21:00:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 99575 invoked by uid 99); 15 Aug 2008 21:00:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Aug 2008 14:00:50 -0700
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.198.233] (HELO rv-out-0506.google.com) (209.85.198.233)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Aug 2008 20:59:54 +0000
Received: by rv-out-0506.google.com with SMTP id k40so1713249rvb.29
        for <general@hadoop.apache.org>; Fri, 15 Aug 2008 14:00:22 -0700 (PDT)
Received: by 10.140.199.3 with SMTP id w3mr1772948rvf.67.1218834022802;
        Fri, 15 Aug 2008 14:00:22 -0700 (PDT)
Received: by 10.140.174.4 with HTTP; Fri, 15 Aug 2008 14:00:22 -0700 (PDT)
Message-ID: <ef38762f0808151400q574f1f88h2fefac5df86936b1@mail.gmail.com>
Date: Fri, 15 Aug 2008 14:00:22 -0700
From: "Brendan Miller" <catphive@catphive.net>
To: general@hadoop.apache.org
Subject: is it possible to commit if you have previously worked at google?
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
X-Virus-Checked: Checked by ClamAV on apache.org

Is it possible to commit to hadoop given I have worked at google in
the past and used the original mapreduce and bigtable to write
applications? Would I need to seek google's permission, or have they
already given blanket permission? Are there legal entanglements?

Thanks

From general-return-30-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 16 00:06:16 2008
Return-Path: <general-return-30-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 22331 invoked from network); 16 Aug 2008 00:06:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 16 Aug 2008 00:06:16 -0000
Received: (qmail 47235 invoked by uid 500); 16 Aug 2008 00:06:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47207 invoked by uid 500); 16 Aug 2008 00:06:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47196 invoked by uid 99); 16 Aug 2008 00:06:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Aug 2008 17:06:14 -0700
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 16 Aug 2008 00:05:14 +0000
Received: from battlerock-lm.corp.yahoo.com (battlerock-lm.corp.yahoo.com [10.72.112.37])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id m7G03qKO067119
	for <general@hadoop.apache.org>; Fri, 15 Aug 2008 17:03:52 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=Ne2FCgl5FPNiG/3wEfpy/OrfEW78LwDyVE6mTeLIMtPWcWCJFDHdexKusK4E0iP5
Message-Id: <319EC963-8D5E-407F-B64B-EE3E6D208A85@yahoo-inc.com>
From: "Owen O'Malley" <oom@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <ef38762f0808151400q574f1f88h2fefac5df86936b1@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v926)
Subject: Re: is it possible to commit if you have previously worked at google?
Date: Fri, 15 Aug 2008 17:03:52 -0700
References: <ef38762f0808151400q574f1f88h2fefac5df86936b1@mail.gmail.com>
X-Mailer: Apple Mail (2.926)
X-Virus-Checked: Checked by ClamAV on apache.org

On Aug 15, 2008, at 2:00 PM, Brendan Miller wrote:

> Is it possible to commit to hadoop given I have worked at google in
> the past and used the original mapreduce and bigtable to write
> applications? Would I need to seek google's permission, or have they
> already given blanket permission? Are there legal entanglements?

I am not a lawyer. In general, Apache doesn't care who your previous  
employers are. It is your responsibility to make sure that you don't  
contribute code that you aren't authorized to license to the ASF. A  
more appropriate forum for questions like this is legal-discuss@apache.org 
.

-- Owen

From general-return-31-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 16 19:28:48 2008
Return-Path: <general-return-31-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 65227 invoked from network); 16 Aug 2008 19:28:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 16 Aug 2008 19:28:47 -0000
Received: (qmail 48456 invoked by uid 500); 16 Aug 2008 19:28:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48429 invoked by uid 500); 16 Aug 2008 19:28:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48418 invoked by uid 99); 16 Aug 2008 19:28:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 16 Aug 2008 12:28:46 -0700
X-ASF-Spam-Status: No, hits=3.0 required=10.0
	tests=DATE_IN_PAST_12_24,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 16 Aug 2008 19:27:47 +0000
Received: from [192.168.1.165] (snvvpn2-10-72-76-c108.corp.yahoo.com [10.72.76.108])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id m7GJQbwF044552
	for <general@hadoop.apache.org>; Sat, 16 Aug 2008 12:26:37 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=AbpEuZyXo18R0NoXhrN5QDlw6tk/3Gy0aNleBXNXYzW1+t0QfWfPHLTalHQcVKaA
Message-Id: <175CD9D8-34A3-4A66-87B0-359EA5E22169@yahoo-inc.com>
From: Nigel Daley <ndaley@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <489CA74F.8000901@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v928.1)
Subject: Re: [VOTE] Should we create sub-projects for HDFS and Map/Reduce?
Date: Fri, 15 Aug 2008 22:03:10 -0700
References: <688CEB2A-13EF-4C38-855A-3D6D4C3485B3@yahoo-inc.com> <2DA5A7B1-0CC0-4C4F-BE80-AE1E3C7B47A3@yahoo-inc.com> <489CA74F.8000901@apache.org>
X-Mailer: Apple Mail (2.928.1)
X-Virus-Checked: Checked by ClamAV on apache.org

> Another benefit is that it would increase the separation of these  
> technologies, so that, e.g., folks could more easily run different  
> versions of mapreduce on top of different versions of HDFS.   
> Currently we make no such guarantees.  Folks would be able to  
> upgrade to, e.g., the next release of mapreduce on a subset of their  
> cluster without upgrading their HDFS.  That's not currently  
> supported.  As we move towards splitting mapreduce into a scheduler  
> and runtime, where folks can specify a different runtime per job,  
> this will be even more critical.

Sounds like we simply need to create separate jar files for these  
different components.  This can be done in the current project.

Wouldn't the amount of effort to make this split and get it right be  
better spent on getting all components of Hadoop to 1.0 (API  
stability)?  The proposal feels like a distraction to me at this point  
in the project.

Nige


From general-return-3137-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 01 11:25:56 2011
Return-Path: <general-return-3137-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56954 invoked from network); 1 Mar 2011 11:25:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Mar 2011 11:25:56 -0000
Received: (qmail 2407 invoked by uid 500); 1 Mar 2011 11:25:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2015 invoked by uid 500); 1 Mar 2011 11:25:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1997 invoked by uid 99); 1 Mar 2011 11:25:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 11:25:44 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 11:25:36 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 87D57B7DAE
	for <general@hadoop.apache.org>; Tue,  1 Mar 2011 11:25:14 +0000 (GMT)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id 4z--BdOIIvkN for <general@hadoop.apache.org>;
	Tue,  1 Mar 2011 11:25:03 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 7DCA0B7D76
	for <general@hadoop.apache.org>; Tue,  1 Mar 2011 11:25:02 +0000 (GMT)
MailScanner-NULL-Check: 1299583490.87877@U17CPSQI4zf4BvpVRuE1jA
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p21BOodP011534
	for <general@hadoop.apache.org>; Tue, 1 Mar 2011 11:24:50 GMT
Message-ID: <4D6CD782.5090507@apache.org>
Date: Tue, 01 Mar 2011 11:24:50 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Build/test infrastructure
References: <C98EEB2E.FFBA%eyang@yahoo-inc.com>
In-Reply-To: <C98EEB2E.FFBA%eyang@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p21BOodP011534
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 27/02/11 01:38, Eric Yang wrote:
>    Instead of ant +
> puppet combination, the patch test build structure could be simplified by
> using maven + shell scripts.


The trouble with any scripted approach vs state driven CM tooling is 
their inability to cope with a starting point that doesn't match the 
scripts assumptions.


This isn't me picking on maven -different topic- more the CM problem. 
Lots of people do use shell scripts, and if your starting point is known 
-such as a golden VM image- it mostly works. Mostly. Even RPM 
installation (remember, it's scripts underneath) ensure that if you 
kickstart install and then update a thousand servers, they will end up 
executing their scripts in different orders, and so end up in different 
final states.

Of course, the weakness of CM tools is that they often end up being 
given goals that can't be consistently satisfied, so they cycle between 
the set of sub-optimal configurations.

From general-return-3138-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 02 17:06:12 2011
Return-Path: <general-return-3138-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 13551 invoked from network); 2 Mar 2011 17:06:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Mar 2011 17:06:12 -0000
Received: (qmail 78223 invoked by uid 500); 2 Mar 2011 17:06:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77781 invoked by uid 500); 2 Mar 2011 17:06:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77773 invoked by uid 99); 2 Mar 2011 17:06:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 17:06:07 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mlortiz@uci.cu designates 200.55.140.180 as permitted sender)
Received: from [200.55.140.180] (HELO mx3.uci.cu) (200.55.140.180)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 02 Mar 2011 17:06:02 +0000
Received: (qmail 7778 invoked by uid 507); 2 Mar 2011 17:05:34 -0000
Received: from 10.0.0.185 by ns3.uci.cu (envelope-from <mlortiz@uci.cu>, uid 501) with qmail-scanner-2.01st 
 (avp: 5.0.2.0. spamassassin: 3.0.6. perlscan: 2.01st.  
 Clear:RC:1(10.0.0.185):. 
 Processed in 0.628193 secs); 02 Mar 2011 17:05:34 -0000
Received: from unknown (HELO ucimail4.uci.cu) (10.0.0.185)
  by 0 with SMTP; 2 Mar 2011 17:05:33 -0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by ucimail4.uci.cu (Postfix) with ESMTP id 47BDB13245F8
	for <general@hadoop.apache.org>; Wed,  2 Mar 2011 12:05:33 -0500 (CST)
X-Virus-Scanned: amavisd-new at uci.cu
Received: from ucimail4.uci.cu ([127.0.0.1])
	by localhost (ucimail4.uci.cu [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id ZG3sxt6j7B6a; Wed,  2 Mar 2011 12:05:32 -0500 (CST)
Received: from ucimail4.uci.cu (ucimail4.uci.cu [10.0.0.185])
	by ucimail4.uci.cu (Postfix) with ESMTP id 43D2B13245A1
	for <general@hadoop.apache.org>; Wed,  2 Mar 2011 12:05:32 -0500 (CST)
Date: Wed, 2 Mar 2011 12:05:32 -0500 (CST)
From:  Marcos Ortiz Valmaseda <mlortiz@uci.cu>
To: general@hadoop.apache.org
Message-ID: <2052738279.11140861299085532253.JavaMail.root@ucimail4.uci.cu>
Subject: Hadoop Development and the new Oracle's Plans
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Originating-IP: [10.0.0.186]
X-Mailer: Zimbra 5.0.16_GA_2921.RHEL5_64 (ZimbraWebClient - FF3.0 (Win)/5.0.16_GA_2921.RHEL5_64)

Regards to all the list.
I have a doubt in my mind since Oracle announced its new plans for Java.=20
1- How the new restrictions can affect to the Hadoop Development?
2- Will Hadoop support the new OpenJDK platform?

Thanks a lot for your time

--=20
Marcos Lu=C3=ADs Ort=C3=ADz Valmaseda
 Software Engineer=20
 Universidad de las Ciencias Inform=C3=A1ticas
 Linux User # 418229

http://uncubanitolinuxero.blogspot.com
http://www.linkedin.com/in/marcosluis2186


From general-return-3139-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 02 23:41:37 2011
Return-Path: <general-return-3139-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 15692 invoked from network); 2 Mar 2011 23:41:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Mar 2011 23:41:37 -0000
Received: (qmail 6452 invoked by uid 500); 2 Mar 2011 23:41:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5733 invoked by uid 500); 2 Mar 2011 23:41:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5562 invoked by uid 99); 2 Mar 2011 23:41:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 23:41:34 +0000
X-ASF-Spam-Status: No, hits=3.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of akimball83@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 23:41:28 +0000
Received: by bwz8 with SMTP id 8so946853bwz.35
        for <multiple recipients>; Wed, 02 Mar 2011 15:41:08 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:from:date:message-id:subject:to
         :content-type;
        bh=DoeEj2tbSqfr8AJMvwLoKdSfNs9QSZdBPz1s83cWdH8=;
        b=dnfRiXPuNSqsPmpTiKrgH923Y0sqc84sTHKx1Gu6LFfBYn6IrzRndvxXgJ0dkErOj0
         /QaZVJy5KWgBho8ecRqCZkzamP6NlYi7dcw9M7RR+nddJtktrz6uMOsCC5dbee/91NzH
         oecCwVQvG88A5dtMtvm6vxj3lnATRDKLdT+J0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type;
        b=dY9qFGvywI5ie9nl+sV3m+LTeeY4/WjyFUJxguFHruSgjuOIP6Fr+4X5h1+3320ZFC
         YrklFWDaLjw/8nylzkgq3fJfwvWNHZjFYaWRff/N0E2njInP+Jr2UReoQOVZT7OxYGIR
         rlQRP+Mfwj81kjU3xCPDskmAU2gTRqUP0grV4=
Received: by 10.204.153.14 with SMTP id i14mr669742bkw.173.1299109268103; Wed,
 02 Mar 2011 15:41:08 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.151.215 with HTTP; Wed, 2 Mar 2011 15:40:48 -0800 (PST)
From: Aaron Kimball <akimball83@gmail.com>
Date: Wed, 2 Mar 2011 15:40:48 -0800
Message-ID: <AANLkTi=PYXQpcG=aCag4B5Tz7Q_Wd2adCoDuhG7qNHqP@mail.gmail.com>
Subject: Reminder: SF Hadoop meetup in 1 week
To: general@hadoop.apache.org, common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175ccf8a732156049d887209
X-Virus-Checked: Checked by ClamAV on apache.org

--0015175ccf8a732156049d887209
Content-Type: text/plain; charset=ISO-8859-1

Hadoop fans,

As a reminder -- the third SF Hadoop meetup is one week away! We will meet
on March 9th, from 6pm to 8pm. (We will hopefully continue using the 2nd
Wednesday of the month for successive meetups).

This meetup will be hosted by Yelp, at their office at 706 Mission St, San
Francisco (3rd and Mission). Yelp has graciously offered to sponsor food &
drinks for the event as well.

Space is still available for more attendees! Yelp has asked that all
attendees RSVP in advance, to comply with their security policy. Please join
the meetup group and RSVP at http://www.meetup.com/hadoopsf/events/16678757/

As per our emerging tradition, we will continue to use the discussion-based
"unconference" format. At the beginning of the meetup we will
collaboratively construct an agenda consisting of several discussion
breakout groups. All participants may propose a topic and volunteer to
facilitate a discussion. All members of the Hadoop community are welcome to
attend. While all Hadoop-related subjects are on topic, this month's
discussion theme is "integration."

Regards,
- Aaron Kimball

--0015175ccf8a732156049d887209--

From general-return-3140-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 03 10:43:58 2011
Return-Path: <general-return-3140-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53284 invoked from network); 3 Mar 2011 10:43:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Mar 2011 10:43:58 -0000
Received: (qmail 58526 invoked by uid 500); 3 Mar 2011 10:43:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58436 invoked by uid 500); 3 Mar 2011 10:43:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58427 invoked by uid 99); 3 Mar 2011 10:43:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 10:43:53 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 10:43:48 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 260E9B7DA0
	for <general@hadoop.apache.org>; Thu,  3 Mar 2011 10:43:25 +0000 (GMT)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id AcxpAlLlNoYk for <general@hadoop.apache.org>;
	Thu,  3 Mar 2011 10:43:14 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 37F20B7D93
	for <general@hadoop.apache.org>; Thu,  3 Mar 2011 10:43:13 +0000 (GMT)
MailScanner-NULL-Check: 1299753781.60345@AHFYjg8fu3a+DTi1R7aZZQ
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p23Ah1Fr013847
	for <general@hadoop.apache.org>; Thu, 3 Mar 2011 10:43:01 GMT
Message-ID: <4D6F70B5.3020302@apache.org>
Date: Thu, 03 Mar 2011 10:43:01 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Hadoop Development and the new Oracle's Plans
References: <2052738279.11140861299085532253.JavaMail.root@ucimail4.uci.cu>
In-Reply-To: <2052738279.11140861299085532253.JavaMail.root@ucimail4.uci.cu>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p23Ah1Fr013847
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 02/03/11 17:05, Marcos Ortiz Valmaseda wrote:
> Regards to all the list.
> I have a doubt in my mind since Oracle announced its new plans for Java.

The ASF has left the JCP due to Sun and then Oracles unwillingness to 
release the Java test kit without imposing freedom of use restrictions 
on Apache source code, despite the JCP rules not permitting this. that 
doesn't mean the ASF is backing off Java development, merely backing off 
attempting to collaborate with others by way of a Standards body that 
has more in common with the peoples congress of the union of soviet 
socialist republics than with a functional democracy. We will work as 
open source projects, with all discussion, code and tests in the open.



> 1- How the new restrictions can affect to the Hadoop Development?

none, Apache code.

Sun/Oracle are free to participate; some of the SunGrid people have been 
involved in the past. Their contributions are still welcome, if 
Hudson/Jenkins is happy with the code and the number of tests they 
provide.

> 2- Will Hadoop support the new OpenJDK platform?


OpenJDK 6 is effectively Sun JDK with a different rendering engine, the 
closed source one has C/C++ source from things like OSF/Motif and the 
like in there so it's copyright is dirty.

I don't see anyone rushing to move to Java7 for Hadoop in production, 
either in source code or binary. Everyone with large clusters likes 
stable versions, and tends to be trailing edge with Java 6 versions, 
avoiding new features like compressed oops as you can predict that with 
500+ 12-core servers, if there is a race condition in the JDK, you will 
find it.

The other issue is that a lot of people out there use Mac laptops for 
their development, and until java7 ships for the mac, nobody who has a 
mac will be able to develop any Java 7 code. That's another reason for 
Hadoop staying on Java6

What could be interesting would be for Hadoop to have more scala 
integration. I think there are some good arguments for us moving some of 
the high level code to that language, within the JVM, rather than follow 
the oracle roadmap.

Speaking of Oracle Roadmaps, they keep talking about Java EE moving to 
be cloud computing. Well, the Java Cloud Computing API is the ASF stack, 
from libraries like Whirr to talk to the infrastructure, the Hadoop DFS 
and MR APIs to talk to the filesystem and MR engine, and the layers on 
top. They can lay down some standards in the JCP, but out here you get 
to build things that are useful today.

-steve

From general-return-3141-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 03 13:59:35 2011
Return-Path: <general-return-3141-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95561 invoked from network); 3 Mar 2011 13:59:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Mar 2011 13:59:35 -0000
Received: (qmail 49946 invoked by uid 500); 3 Mar 2011 13:59:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49885 invoked by uid 500); 3 Mar 2011 13:59:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49877 invoked by uid 99); 3 Mar 2011 13:59:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 13:59:33 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 13:59:27 +0000
Received: by vws20 with SMTP id 20so1230367vws.35
        for <general@hadoop.apache.org>; Thu, 03 Mar 2011 05:59:06 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:from:content-type:content-transfer-encoding
         :subject:date:message-id:to:mime-version:x-mailer;
        bh=1SRKOZXmf7/ldZf53Mdu1pgLKti/DVP4Ek0yVY67lqs=;
        b=PZNpDOURZ3YqfMWYGIcdxjKLsBANIoCQYFEI72A/cgRWP33KUQfmuXhkHNrIUpBolR
         ZcUrGpqtfsha/bk6rG39xi/PtDOsliEsZpoIWspCSxUdI1nDhy5s+uBTTXmpa11qYMDv
         HgLe8RzMLl5su0cyUc0dCjRSFzWeg5kAFA4VA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=from:content-type:content-transfer-encoding:subject:date:message-id
         :to:mime-version:x-mailer;
        b=phHC6z4aLBEjfjC8lLebcOZ1Ou+KSqDw66v16lj14MvXOZ/DBCKU7tLEuP1pL9dpke
         T0QGJvS7xYrq4tHlFxrx6opS9RJNiOd066crztpAaFwZkMFIP3E4jt1xZjpHoR/4jttk
         VV0MbtoXTcPzsOGMqZOuuVnIQRbTa9jeM9sHM=
Received: by 10.52.161.67 with SMTP id xq3mr1832289vdb.57.1299160746585;
        Thu, 03 Mar 2011 05:59:06 -0800 (PST)
Received: from [10.180.150.79] (h-64-236-128-43.nat.aol.com [64.236.128.43])
        by mx.google.com with ESMTPS id b18sm483560vcm.16.2011.03.03.05.59.05
        (version=SSLv3 cipher=OTHER);
        Thu, 03 Mar 2011 05:59:05 -0800 (PST)
From: Ian Holsman <ian@holsman.net>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: Apache Sonar - http://analysis.apache.org/ does anyone want hadoop in there?
Date: Thu, 3 Mar 2011 08:59:04 -0500
Message-Id: <FF258184-01BE-419F-B5A0-69051472BBDB@Holsman.net>
To: general@hadoop.apache.org
Mime-Version: 1.0 (Apple Message framework v1082)
X-Mailer: Apple Mail (2.1082)

I just discovered this Apache site,=20
Apache Sonar performs source code analysis on the java code, and it =
looks pretty, and I'm sure some people would find it useful.

If people are interested I can start putting finding out how to put the =
hadoop code in there, so we can get stats on our own code.

Regards
Ian
--
Ian Holsman
Ian@Holsman.net
PH: +1-703 879-3128 AOLIM: ianholsman Skype:iholsman

To know recursion, you must first know recursion.




From general-return-3142-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 03 16:40:03 2011
Return-Path: <general-return-3142-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42914 invoked from network); 3 Mar 2011 16:40:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Mar 2011 16:40:03 -0000
Received: (qmail 38379 invoked by uid 500); 3 Mar 2011 16:40:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38282 invoked by uid 500); 3 Mar 2011 16:40:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 38125 invoked by uid 99); 3 Mar 2011 16:40:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 16:40:00 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mlortiz@uci.cu designates 200.55.140.180 as permitted sender)
Received: from [200.55.140.180] (HELO mx3.uci.cu) (200.55.140.180)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 03 Mar 2011 16:39:56 +0000
Received: (qmail 17402 invoked by uid 507); 3 Mar 2011 16:39:29 -0000
Received: from 10.0.0.184 by ns3.uci.cu (envelope-from <mlortiz@uci.cu>, uid 501) with qmail-scanner-2.01st 
 (avp: 5.0.2.0. spamassassin: 3.0.6. perlscan: 2.01st.  
 Clear:RC:1(10.0.0.184):. 
 Processed in 0.625812 secs); 03 Mar 2011 16:39:29 -0000
Received: from unknown (HELO ucimail3.uci.cu) (10.0.0.184)
  by 0 with SMTP; 3 Mar 2011 16:39:29 -0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by ucimail3.uci.cu (Postfix) with ESMTP id D94D51E8C021
	for <general@hadoop.apache.org>; Thu,  3 Mar 2011 11:39:28 -0500 (CST)
X-Virus-Scanned: amavisd-new at uci.cu
Received: from ucimail3.uci.cu ([127.0.0.1])
	by localhost (ucimail3.uci.cu [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id wZueyPYirLQT; Thu,  3 Mar 2011 11:39:25 -0500 (CST)
Received: from [10.7.2.15] (marcosluis-aspire-5251.uci.cu [10.7.2.15])
	by ucimail3.uci.cu (Postfix) with ESMTP id D409E1E8C007
	for <general@hadoop.apache.org>; Thu,  3 Mar 2011 11:39:24 -0500 (CST)
Subject: Re: Hadoop Development and the new Oracle's Plans
From: Marcos Ortiz <mlortiz@uci.cu>
To: general@hadoop.apache.org
In-Reply-To: <4D6F70B5.3020302@apache.org>
References: <2052738279.11140861299085532253.JavaMail.root@ucimail4.uci.cu>
	 <4D6F70B5.3020302@apache.org>
Content-Type: text/plain; charset="UTF-8"
Organization: UCI
Date: Thu, 03 Mar 2011 11:40:26 -0430
Message-ID: <1299168626.3413.1.camel@marcosluis-Aspire-5251>
Mime-Version: 1.0
X-Mailer: Evolution 2.30.3 
Content-Transfer-Encoding: 8bit

Thanks a lot for the excellent response.
Regards.
On Thu, 2011-03-03 at 10:43 +0000, Steve Loughran wrote: 
> On 02/03/11 17:05, Marcos Ortiz Valmaseda wrote:
> > Regards to all the list.
> > I have a doubt in my mind since Oracle announced its new plans for Java.
> 
> The ASF has left the JCP due to Sun and then Oracles unwillingness to 
> release the Java test kit without imposing freedom of use restrictions 
> on Apache source code, despite the JCP rules not permitting this. that 
> doesn't mean the ASF is backing off Java development, merely backing off 
> attempting to collaborate with others by way of a Standards body that 
> has more in common with the peoples congress of the union of soviet 
> socialist republics than with a functional democracy. We will work as 
> open source projects, with all discussion, code and tests in the open.
> 
> 
> 
> > 1- How the new restrictions can affect to the Hadoop Development?
> 
> none, Apache code.
> 
> Sun/Oracle are free to participate; some of the SunGrid people have been 
> involved in the past. Their contributions are still welcome, if 
> Hudson/Jenkins is happy with the code and the number of tests they 
> provide.
> 
> > 2- Will Hadoop support the new OpenJDK platform?
> 
> 
> OpenJDK 6 is effectively Sun JDK with a different rendering engine, the 
> closed source one has C/C++ source from things like OSF/Motif and the 
> like in there so it's copyright is dirty.
> 
> I don't see anyone rushing to move to Java7 for Hadoop in production, 
> either in source code or binary. Everyone with large clusters likes 
> stable versions, and tends to be trailing edge with Java 6 versions, 
> avoiding new features like compressed oops as you can predict that with 
> 500+ 12-core servers, if there is a race condition in the JDK, you will 
> find it.
> 
> The other issue is that a lot of people out there use Mac laptops for 
> their development, and until java7 ships for the mac, nobody who has a 
> mac will be able to develop any Java 7 code. That's another reason for 
> Hadoop staying on Java6
> 
> What could be interesting would be for Hadoop to have more scala 
> integration. I think there are some good arguments for us moving some of 
> the high level code to that language, within the JVM, rather than follow 
> the oracle roadmap.
> 
> Speaking of Oracle Roadmaps, they keep talking about Java EE moving to 
> be cloud computing. Well, the Java Cloud Computing API is the ASF stack, 
> from libraries like Whirr to talk to the infrastructure, the Hadoop DFS 
> and MR APIs to talk to the filesystem and MR engine, and the layers on 
> top. They can lay down some standards in the JCP, but out here you get 
> to build things that are useful today.
> 
> -steve

-- 
 Marcos Luís Ortíz Valmaseda
 Software Engineer
 Centro de Tecnologías de Gestión de Datos (DATEC)
 Universidad de las Ciencias Informáticas
 http://uncubanitolinuxero.blogspot.com
 http://www.linkedin.com/in/marcosluis2186



From general-return-3143-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 03 16:42:53 2011
Return-Path: <general-return-3143-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44999 invoked from network); 3 Mar 2011 16:42:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Mar 2011 16:42:53 -0000
Received: (qmail 50658 invoked by uid 500); 3 Mar 2011 16:42:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50606 invoked by uid 500); 3 Mar 2011 16:42:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50593 invoked by uid 99); 3 Mar 2011 16:42:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 16:42:51 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mlortiz@uci.cu designates 200.55.140.180 as permitted sender)
Received: from [200.55.140.180] (HELO mx3.uci.cu) (200.55.140.180)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 03 Mar 2011 16:42:47 +0000
Received: (qmail 18723 invoked by uid 507); 3 Mar 2011 16:42:22 -0000
Received: from 10.0.0.184 by ns3.uci.cu (envelope-from <mlortiz@uci.cu>, uid 501) with qmail-scanner-2.01st 
 (avp: 5.0.2.0. spamassassin: 3.0.6. perlscan: 2.01st.  
 Clear:RC:1(10.0.0.184):. 
 Processed in 0.679263 secs); 03 Mar 2011 16:42:22 -0000
Received: from unknown (HELO ucimail3.uci.cu) (10.0.0.184)
  by 0 with SMTP; 3 Mar 2011 16:42:21 -0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by ucimail3.uci.cu (Postfix) with ESMTP id 269B61E8C037;
	Thu,  3 Mar 2011 11:42:21 -0500 (CST)
X-Virus-Scanned: amavisd-new at uci.cu
Received: from ucimail3.uci.cu ([127.0.0.1])
	by localhost (ucimail3.uci.cu [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id WcsLKIktQUD7; Thu,  3 Mar 2011 11:42:20 -0500 (CST)
Received: from [10.7.2.15] (marcosluis-aspire-5251.uci.cu [10.7.2.15])
	by ucimail3.uci.cu (Postfix) with ESMTP id 21CBD1E8C033;
	Thu,  3 Mar 2011 11:42:20 -0500 (CST)
Subject: Re: Hadoop Development and the new Oracle's Plans
From: Marcos Ortiz <mlortiz@uci.cu>
To: general@hadoop.apache.org
Cc: Steve Loughran <stevel@apache.org>
In-Reply-To: <4D6F70B5.3020302@apache.org>
References: <2052738279.11140861299085532253.JavaMail.root@ucimail4.uci.cu>
	 <4D6F70B5.3020302@apache.org>
Content-Type: text/plain; charset="UTF-8"
Organization: UCI
Date: Thu, 03 Mar 2011 11:43:21 -0430
Message-ID: <1299168801.3413.3.camel@marcosluis-Aspire-5251>
Mime-Version: 1.0
X-Mailer: Evolution 2.30.3 
Content-Transfer-Encoding: 8bit


> What could be interesting would be for Hadoop to have more scala 
> integration. I think there are some good arguments for us moving some of 
> the high level code to that language, within the JVM, rather than follow 
> the oracle roadmap.

About this topic, Steve, Is there any development focused on this theme?
It´s a interesting project to work.

Regards

-- 
 Marcos Luís Ortíz Valmaseda
 Software Engineer
 Centro de Tecnologías de Gestión de Datos (DATEC)
 Universidad de las Ciencias Informáticas
 http://uncubanitolinuxero.blogspot.com
 http://www.linkedin.com/in/marcosluis2186



From general-return-3144-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 03 17:35:16 2011
Return-Path: <general-return-3144-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34607 invoked from network); 3 Mar 2011 17:35:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Mar 2011 17:35:15 -0000
Received: (qmail 10186 invoked by uid 500); 3 Mar 2011 17:35:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 9953 invoked by uid 500); 3 Mar 2011 17:35:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 9945 invoked by uid 99); 3 Mar 2011 17:35:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 17:35:14 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 17:35:09 +0000
Received: by wwi14 with SMTP id 14so1711413wwi.29
        for <general@hadoop.apache.org>; Thu, 03 Mar 2011 09:34:47 -0800 (PST)
Received: by 10.216.245.195 with SMTP id o45mr1347223wer.15.1299173687169;
 Thu, 03 Mar 2011 09:34:47 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.180.70 with HTTP; Thu, 3 Mar 2011 09:34:26 -0800 (PST)
In-Reply-To: <FF258184-01BE-419F-B5A0-69051472BBDB@Holsman.net>
References: <FF258184-01BE-419F-B5A0-69051472BBDB@Holsman.net>
From: Konstantin Boudnik <cos@apache.org>
Date: Thu, 3 Mar 2011 09:34:26 -0800
X-Google-Sender-Auth: GYIjRsW83TMniB96nP9aMMO6wWc
Message-ID: <AANLkTimsma0e9cGAN0o_zz4dBDNRnkDXEgKcVx00zbEV@mail.gmail.com>
Subject: Re: Apache Sonar - http://analysis.apache.org/ does anyone want
 hadoop in there?
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Ian,

Sonar was around for a while and Apache has this setup since late
2009, I believe.  I had asked  INFRA folks on a few occasions about
adding Hadoop repos to their installation, but never heard anything
back.

I'd be awesome if you can talk to the right people so it'd happen eventuall=
y.
--
=A0 Thanks in advance,
Konstantin (Cos) Boudnik


On Thu, Mar 3, 2011 at 05:59, Ian Holsman <ian@holsman.net> wrote:
> I just discovered this Apache site,
> Apache Sonar performs source code analysis on the java code, and it looks=
 pretty, and I'm sure some people would find it useful.
>
> If people are interested I can start putting finding out how to put the h=
adoop code in there, so we can get stats on our own code.
>
> Regards
> Ian
> --
> Ian Holsman
> Ian@Holsman.net
> PH: +1-703 879-3128 AOLIM: ianholsman Skype:iholsman
>
> To know recursion, you must first know recursion.
>
>
>
>

From general-return-3145-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 03 18:12:31 2011
Return-Path: <general-return-3145-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49304 invoked from network); 3 Mar 2011 18:12:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Mar 2011 18:12:30 -0000
Received: (qmail 36192 invoked by uid 500); 3 Mar 2011 18:12:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36090 invoked by uid 500); 3 Mar 2011 18:12:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36082 invoked by uid 99); 3 Mar 2011 18:12:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 18:12:28 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 18:12:23 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=JZGOjQ79uqUsXJg299cdtlPuMFhrWKemuhgwV3n9mNqpCSzTXOfiHB5w
   fJSxZAm9vYJpo6h2RkLk457nTZiwNM2UUTwwHmvfrW7dDNDMozMiQbYKI
   1PgmeCKvgz+k2Qm;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1299175937; x=1330711937;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20Apache=20Sonar=20-=20http://analysis.ap
   ache.org/=20does=20anyone=20want=0D=0A=20hadoop=20in=20th
   ere?|Date:=20Thu,=203=20Mar=202011=2018:13:52=20+0000
   |Message-ID:=20<A3B0B1D4-6B59-4F06-9801-D9DA2C303ECE@link
   edin.com>|To:=20"<general@hadoop.apache.org>"=20<general@
   hadoop.apache.org>|MIME-Version:=201.0
   |Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<68222B35484EE04F9365A532D454284D@linkedin
   .com>|In-Reply-To:=20<FF258184-01BE-419F-B5A0-69051472BBD
   B@Holsman.net>|References:=20<FF258184-01BE-419F-B5A0-690
   51472BBDB@Holsman.net>;
  bh=I7wNsr4drbEZLTU97dbE8yxTZM08E4PpzuIBdXNofzc=;
  b=aWFon0FUMMpdE4RHXHlRGt0/EKSjRy+sPMuuaKb56wteYW83X6t2vL5P
   jiUcAg/D3XH2AIN1GkOAnfVN1jumdTPtcAirlUN990S0Sh8uhY/dsSJsY
   mhcXLXeHAo8oT/v;
X-IronPort-AV: E=Sophos;i="4.62,259,1297065600"; 
   d="scan'208";a="21308623"
Received: from ESV4-EXC03.linkedin.biz ([fe80::985a:a6b4:f1e1:23b0]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi id 14.01.0251.000; Thu, 3
 Mar 2011 10:13:52 -0800
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: Apache Sonar - http://analysis.apache.org/ does anyone want
 hadoop in there?
Thread-Topic: Apache Sonar - http://analysis.apache.org/ does anyone want
 hadoop in there?
Thread-Index: AQHL2at+eyfgXZDy0EeiMzwlXMfL7JQcb7yA
Date: Thu, 3 Mar 2011 18:13:52 +0000
Message-ID: <A3B0B1D4-6B59-4F06-9801-D9DA2C303ECE@linkedin.com>
References: <FF258184-01BE-419F-B5A0-69051472BBDB@Holsman.net>
In-Reply-To: <FF258184-01BE-419F-B5A0-69051472BBDB@Holsman.net>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <68222B35484EE04F9365A532D454284D@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


On Mar 3, 2011, at 5:59 AM, Ian Holsman wrote:

> I just discovered this Apache site,=20
> Apache Sonar performs source code analysis on the java code, and it looks=
 pretty, and I'm sure some people would find it useful.

	Is it actually returning for anyone else?  I get a time out.


From general-return-3146-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 03 18:17:08 2011
Return-Path: <general-return-3146-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55106 invoked from network); 3 Mar 2011 18:17:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Mar 2011 18:17:08 -0000
Received: (qmail 54421 invoked by uid 500); 3 Mar 2011 18:17:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54367 invoked by uid 500); 3 Mar 2011 18:17:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54358 invoked by uid 99); 3 Mar 2011 18:17:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 18:17:06 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 18:17:00 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=I7gIVzLXh25CrHtxFngpMw5VTQcBY51fjuUwUAcLlrQUqN5JBTiCEk4Y
   Q8Oxw0i7rAHzvsiJDaFWSbWofXasyq4tGhI++9BWKgLpg0Te4IZ+3EzVg
   BfmDHRjg6bv9Tdv;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1299176214; x=1330712214;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20Apache=20Sonar=20-=20http://analysis.ap
   ache.org/=20does=20anyone=20want=0D=0A=20hadoop=20in=20th
   ere?|Date:=20Thu,=203=20Mar=202011=2018:18:31=20+0000
   |Message-ID:=20<A85D4470-9631-4C1D-860A-06F247C6E5A4@link
   edin.com>|To:=20"<general@hadoop.apache.org>"=20<general@
   hadoop.apache.org>|MIME-Version:=201.0
   |Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<2756B26FF85689488C1A00B8272AD03E@linkedin
   .com>|In-Reply-To:=20<A3B0B1D4-6B59-4F06-9801-D9DA2C303EC
   E@linkedin.com>|References:=20<FF258184-01BE-419F-B5A0-69
   051472BBDB@Holsman.net>=0D=0A=20<A3B0B1D4-6B59-4F06-9801-
   D9DA2C303ECE@linkedin.com>;
  bh=gJ5X10pxa6LVn/BhqHY29Ggun8kPT8zFEM2ybY3zb3g=;
  b=aWwVSF+QCYHBZEWJP42DUzBLX8pJfPlkM60iFpAlYhCon7lak6fNjLlu
   cUo3UB8PcdfG5TN2JHW74FtYFr+Sz9KzZXera427TS+XvX5aXWK26mFXg
   /3aHTS1P80AvV6A;
X-IronPort-AV: E=Sophos;i="4.62,259,1297065600"; 
   d="scan'208";a="21308817"
Received: from ESV4-EXC03.linkedin.biz ([fe80::985a:a6b4:f1e1:23b0]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi id 14.01.0251.000; Thu, 3
 Mar 2011 10:18:32 -0800
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: Apache Sonar - http://analysis.apache.org/ does anyone want
 hadoop in there?
Thread-Topic: Apache Sonar - http://analysis.apache.org/ does anyone want
 hadoop in there?
Thread-Index: AQHL2at+eyfgXZDy0EeiMzwlXMfL7JQcb7yAgAABToA=
Date: Thu, 3 Mar 2011 18:18:31 +0000
Message-ID: <A85D4470-9631-4C1D-860A-06F247C6E5A4@linkedin.com>
References: <FF258184-01BE-419F-B5A0-69051472BBDB@Holsman.net>
 <A3B0B1D4-6B59-4F06-9801-D9DA2C303ECE@linkedin.com>
In-Reply-To: <A3B0B1D4-6B59-4F06-9801-D9DA2C303ECE@linkedin.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <2756B26FF85689488C1A00B8272AD03E@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


On Mar 3, 2011, at 10:13 AM, Allen Wittenauer wrote:

>=20
> On Mar 3, 2011, at 5:59 AM, Ian Holsman wrote:
>=20
>> I just discovered this Apache site,=20
>> Apache Sonar performs source code analysis on the java code, and it look=
s pretty, and I'm sure some people would find it useful.
>=20
> 	Is it actually returning for anyone else?  I get a time out.
>=20

It took a few tries, but it finally came back.


From general-return-3147-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 03 21:05:22 2011
Return-Path: <general-return-3147-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 52578 invoked from network); 3 Mar 2011 21:05:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Mar 2011 21:05:21 -0000
Received: (qmail 89503 invoked by uid 500); 3 Mar 2011 21:05:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89461 invoked by uid 500); 3 Mar 2011 21:05:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89452 invoked by uid 99); 3 Mar 2011 21:05:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 21:05:20 +0000
X-ASF-Spam-Status: No, hits=2.9 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 21:05:14 +0000
Received: from [10.72.184.42] (mayvarious-lm.corp.yahoo.com [10.72.184.42])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p23L4EmA008347
	for <general@hadoop.apache.org>; Thu, 3 Mar 2011 13:04:14 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1299186254;
	bh=nvpZt47a5krUwhSC2TBNx9Xw+1bFU7c+VRGxMtecL+w=;
	h=Message-Id:From:To:Content-Type:Mime-Version:Subject:Date:
	 References;
	b=Mg9MIcNDI1WEoGUWGYOCrh0IK7NoaWyvcsVvA/7CogPyndj3yQjN0TcLbe5rzwPQX
	 kIbaEDmXTsdosjTg+2WseVM0LpQSGTDkjVzsUNUfZtAj77DUhHF2X3uueKqd1lmwp0
	 oO23g68L3IKzVgDaLFocDXTg4vy1J9j3uPGEKLy4=
Message-Id: <A781321C-37B2-4D95-8B82-4171C1D5F7E7@yahoo-inc.com>
From: "Owen O'Malley" <oom@yahoo-inc.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=Apple-Mail-8-903990662
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Fwd: [Announce] Now Open: Call for Participation for ApacheCon North America
Date: Thu, 3 Mar 2011 13:04:14 -0800
References: <959692.86880.qm@web30802.mail.mud.yahoo.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-8-903990662
Content-Type: text/plain;
	charset=WINDOWS-1252;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: quoted-printable

All,
    Here's the call for presentations at ApacheCon 2011.

-- Owen

Begin forwarded message:

> From: Sally Khudairi <sk@apache.org>
> Date: March 3, 2011 12:10:17 PM PST
> To: "announce@apachecon.com" <announce@apachecon.com>
> Subject: [Announce] Now Open: Call for Participation for ApacheCon =20
> North America
> Reply-To: "sk@apache.org" <sk@apache.org>
>
> Call for Participation
> ApacheCon North America 2011
> 7-11 November 2011
> Westin Bayshore, Vancouver, Canada
>
> All submissions must be received by Friday, 29 April 2011 at =20
> midnight Pacific Time.
>
> ApacheCon, the official conference, trainings, and expo of The =20
> Apache Software Foundation (ASF), heads to Vancouver, Canada, this =20
> November, with dozens of technical, business, and community-focused =20=

> sessions for beginner, intermediate, and expert audiences.
>
> Now in its 11th year, the ASF develops and shepherds nearly 150 Top-=20=

> Level Projects and new initiatives in the Apache Incubator and Labs. =20=

> With hundreds of thousands of applications deploying ASF products =20
> and code contributions by more than 2,500 Committers from around the =20=

> world, the Apache community is recognized as among the most robust, =20=

> successful, and respected in Open Source.
>
> This year's ApacheCon focuses on highly-relevant, professionally-=20
> directed presentations that demonstrate specific problems and real-=20
> world solutions. We welcome proposals --from developers and users =20
> alike-- in the areas of "Apache and ...":
>
> ... Enterprise Solutions (from ActiveMQ to Axis2 to ServiceMix, =20
> OFBiz to Chemistry, the gang's all here!)
>
> ... Cloud Computing (Hadoop, Cassandra, HBase, CouchDB, and friends)
>
> ... Emerging Technologies + Innovation (Incubating projects such as =20=

> Libcloud, Stonehenge, and Wookie)
>
> ... Community Leadership (mentoring and meritocracy, GSoC and =20
> related initiatives)
>
> ... Data Handling, Search + Analytics (Lucene, Solr, Mahout, OODT, =20
> Hive and friends)
>
> ... Pervasive Computing (Felix/OSGi, Tomcat, MyFaces Trinidad, and =20
> friends)
>
> ... Servers, Infrastructure + Tools (HTTP Server, SpamAssassin, =20
> Geronimo, Sling, Wicket and friends)
>
>
> Submissions are open to anyone with relevant expertise: ASF =20
> affiliation is not required to present at, attend, or otherwise =20
> participate in ApacheCon.
>
> Whilst we encourage submissions that the highlight the use of =20
> specific Apache solutions, we are unable to accept marketing/=20
> commercially-oriented presentations.
>
> Other proposals, such as panels, have been considered in the past; =20
> you are welcome to submit an alternate presentation, however, such =20
> sessions are accepted under exceptional circumstances. Please be as =20=

> descriptive as possible, including names/bios of proposed panelists =20=

> and any related details.
>
> Accepted speakers (not co-presenters) qualify for general conference =20=

> admission and a minimum of two nights lodging at the conference =20
> hotel. Additional hotel nights and travel assistance are possible, =20
> depending on the number of presentations given and type of =20
> assistance needed.
>
> To submit a presentation proposal, please complete our ONLINE =20
> SUBMISSION FORM at http://na11.apachecon.com/proposals/new
>
> To be considered, proposals must be received by Friday, 29 April =20
> 2011 at midnight Pacific Time. Please email any questions regarding =20=

> proposal submissions to cfp AT apachecon DOT com.
>
> Key Dates:
>
> 3 March 2011 - CFP Opens
> 29 April 2011 - CFP Closes
> 20 May-30 June 2011 - Speaker Notifications and Confirmations
> 7-11 November 2011 - ApacheCon NA 2011
>
>
> We look forward to seeing you in Vancouver!
>
> =96 The ApacheCon Planning team
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: announce-unsubscribe@apachecon.com
> For additional commands, e-mail: announce-help@apachecon.com
>


--Apple-Mail-8-903990662--

From general-return-3148-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 13:00:26 2011
Return-Path: <general-return-3148-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77230 invoked from network); 4 Mar 2011 13:00:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 13:00:25 -0000
Received: (qmail 5169 invoked by uid 500); 4 Mar 2011 13:00:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5087 invoked by uid 500); 4 Mar 2011 13:00:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5074 invoked by uid 99); 4 Mar 2011 13:00:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 13:00:23 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of binhara@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 13:00:15 +0000
Received: by wya21 with SMTP id 21so2582412wya.35
        for <general@hadoop.apache.org>; Fri, 04 Mar 2011 04:59:55 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=TQwIGDoqeQWkAzBHJcwM/vyLU1ZPj7kAoslcBYe+EdY=;
        b=jPq0cU4wWMfz8scWWClz3NMN3UBC5yv0lQNdYqXWX3eHoYhwqU7nqFVaMwvshqIyVH
         bjClydJxsaozjtuT/NBjHwDuF0jGKdYOk4uJruGROqRlChevCISA+OTVvUbURq4iFuig
         a/FDZZd90Hv8B6nQ6qtViysfvlq6M3lKWPxn4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=FqT1U+CR8wu5ucU8tlrjN0Ajp4avHTfrYp3wEcUTDNLVzE7rvACV63jmwoRH8m5Xwc
         iguFtTg7IsjytFVJHoVpg3k1ZKHhYXfgs3IPLr1QSWf3gwyjXijm353BHvMoLN/UBAmt
         RR9sVwy4Zd4i2JSmKtNvpyr2iwa5DXuoky3Jg=
MIME-Version: 1.0
Received: by 10.216.171.19 with SMTP id q19mr546515wel.53.1299243595221; Fri,
 04 Mar 2011 04:59:55 -0800 (PST)
Received: by 10.216.172.201 with HTTP; Fri, 4 Mar 2011 04:59:55 -0800 (PST)
Date: Fri, 4 Mar 2011 09:59:55 -0300
Message-ID: <AANLkTik-OHbe_6Ykp-iJxtksxGMwDavmQDotc-R15Z3x@mail.gmail.com>
Subject: How i can test a Job finished and sucess in java
From: Alessandro Binhara <binhara@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e65b60fef84b0a049da7b8a1
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e65b60fef84b0a049da7b8a1
Content-Type: text/plain; charset=ISO-8859-1

hello all ..

i writte mapreduce in java . It works very well.
How i know  a Job finished and sucess in java ?


 JobConf conf = new JobConf(AverageJob.class);
    conf.setJobName("AverageProductTime");

    FileInputFormat.addInputPath(conf, new Path(inputDir));
    FileOutputFormat.setOutputPath(conf, new Path(outputDir));

    conf.setMapperClass(MapAverage.class);
    conf.setReducerClass(RedAverage.class);
      conf.setOutputKeyClass(Text.class);
      conf.setOutputValueClass(IntWritable.class);

    JobClient.runJob(conf);


    if (JobClient.SUCCESS == 0)
    {
                ... do someting ...
    }

????

Thanks

--0016e65b60fef84b0a049da7b8a1--

From general-return-3149-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 13:52:20 2011
Return-Path: <general-return-3149-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70223 invoked from network); 4 Mar 2011 13:52:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 13:52:19 -0000
Received: (qmail 68541 invoked by uid 500); 4 Mar 2011 13:52:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68489 invoked by uid 500); 4 Mar 2011 13:52:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68481 invoked by uid 99); 4 Mar 2011 13:52:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 13:52:18 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 13:52:12 +0000
Received: by fxm2 with SMTP id 2so2805060fxm.35
        for <general@hadoop.apache.org>; Fri, 04 Mar 2011 05:51:52 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type;
        bh=0PJEevogRaosjm/3H6rL9ANLubAmhpc9w1z1rAH9vzY=;
        b=k4dMInydm2qnKTFTtY5EVrH6zmygZmxa1/0VKH2PmMeIYQOYi0xGPP5/cbDALwcMgb
         EUdHWgqLjczRS/P6AaIQ6X4oNKGIDdGkkk+5dM1tjVkYy4iAjtNgUE+RqWDL9GL4q77i
         Ikwvi5ZtPaWO1SS2qM6xM+tpxX398nRlM4Hpo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=lAvKBA63eiPKzk+MZLcvt9mXaQp0cZOnjDZ2warCcVCGj9fAHp3Od8sN1DFiKCXypk
         g3H/Q7hM4AA91N2gLHzoFxZCpFdqLQNXijEUlelW1hDWUw6iACugDXiEwh+uNRvk71bS
         regW9sm/pNgCz3tIgv2W5cNYjaMPDPdW5OuF8=
Received: by 10.223.7.26 with SMTP id b26mr816245fab.119.1299246710760; Fri,
 04 Mar 2011 05:51:50 -0800 (PST)
MIME-Version: 1.0
Received: by 10.223.122.83 with HTTP; Fri, 4 Mar 2011 05:49:35 -0800 (PST)
In-Reply-To: <AANLkTik-OHbe_6Ykp-iJxtksxGMwDavmQDotc-R15Z3x@mail.gmail.com>
References: <AANLkTik-OHbe_6Ykp-iJxtksxGMwDavmQDotc-R15Z3x@mail.gmail.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Fri, 4 Mar 2011 19:19:35 +0530
Message-ID: <AANLkTimB5O1aDn_nDKV+EpbG8zO=cfC3-jqx94iUi3eL@mail.gmail.com>
Subject: Re: How i can test a Job finished and sucess in java
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Fri, Mar 4, 2011 at 6:29 PM, Alessandro Binhara <binhara@gmail.com> wrote:
> ????
>
> Thanks
>

JobClient.runJob returns a RunningJob object at. Use that to determine
the result.
Please see: http://hadoop.apache.org/common/docs/r0.20.0/api/org/apache/hadoop/mapred/RunningJob.html#isSuccessful()

-- 
Harsh J
www.harshj.com

From general-return-3150-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 15:42:21 2011
Return-Path: <general-return-3150-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 20846 invoked from network); 4 Mar 2011 15:42:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 15:42:21 -0000
Received: (qmail 57251 invoked by uid 500); 4 Mar 2011 15:42:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57162 invoked by uid 500); 4 Mar 2011 15:42:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56998 invoked by uid 99); 4 Mar 2011 15:42:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 15:42:18 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mlortiz@uci.cu designates 200.55.140.180 as permitted sender)
Received: from [200.55.140.180] (HELO mx3.uci.cu) (200.55.140.180)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 04 Mar 2011 15:42:11 +0000
Received: (qmail 23948 invoked by uid 507); 4 Mar 2011 15:41:45 -0000
Received: from 10.0.0.184 by ns3.uci.cu (envelope-from <mlortiz@uci.cu>, uid 501) with qmail-scanner-2.01st 
 (avp: 5.0.2.0. spamassassin: 3.0.6. perlscan: 2.01st.  
 Clear:RC:1(10.0.0.184):. 
 Processed in 0.023453 secs); 04 Mar 2011 15:41:45 -0000
Received: from unknown (HELO ucimail3.uci.cu) (10.0.0.184)
  by 0 with SMTP; 4 Mar 2011 15:41:45 -0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by ucimail3.uci.cu (Postfix) with ESMTP id 586D61E8C047;
	Fri,  4 Mar 2011 10:41:45 -0500 (CST)
X-Virus-Scanned: amavisd-new at uci.cu
Received: from ucimail3.uci.cu ([127.0.0.1])
	by localhost (ucimail3.uci.cu [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id PuLKmVwBVAAg; Fri,  4 Mar 2011 10:41:43 -0500 (CST)
Received: from [10.36.18.44] (skynet2010.uci.cu [10.36.18.44])
	by ucimail3.uci.cu (Postfix) with ESMTP id 1C6CEB49065;
	Fri,  4 Mar 2011 10:41:43 -0500 (CST)
Message-ID: <4D710873.4080003@uci.cu>
Date: Fri, 04 Mar 2011 10:42:43 -0500
From: Marcos Ortiz <mlortiz@uci.cu>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; es-ES; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: general@hadoop.apache.org
CC: Alessandro Binhara <binhara@gmail.com>
Subject: Re: How i can test a Job finished and sucess in java
References: <AANLkTik-OHbe_6Ykp-iJxtksxGMwDavmQDotc-R15Z3x@mail.gmail.com>
In-Reply-To: <AANLkTik-OHbe_6Ykp-iJxtksxGMwDavmQDotc-R15Z3x@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit
X-Virus-Checked: Checked by ClamAV on apache.org

El 3/4/2011 7:59 AM, Alessandro Binhara escribió:
> hello all ..
>
> i writte mapreduce in java . It works very well.
> How i know  a Job finished and sucess in java ?
>
>
>   JobConf conf = new JobConf(AverageJob.class);
>      conf.setJobName("AverageProductTime");
>
>      FileInputFormat.addInputPath(conf, new Path(inputDir));
>      FileOutputFormat.setOutputPath(conf, new Path(outputDir));
>
>      conf.setMapperClass(MapAverage.class);
>      conf.setReducerClass(RedAverage.class);
>        conf.setOutputKeyClass(Text.class);
>        conf.setOutputValueClass(IntWritable.class);
>
>      JobClient.runJob(conf);
>
>
>      if (JobClient.SUCCESS == 0)
>      {
>                  ... do someting ...
>      }
>
> ????
>
> Thanks
>
>    
Regards, Alessandro.
I 'm going to use a example of the Pro Hadoop's book. On Chapter 2, 
Jason Venner explains the different ways to see the success of a certain 
job. The RunningJobs class has the right methods to do this. For example:
  logger.info("Launching the job...");
/** Send the job configuration to the framework
* and request that the job be run.
*/
final RunningJob job = JobClient.runJob(conf);
logger.info("The job has completed.");

Jason says that one of the more useful methods is job.isSuccessful(). I 
recommend to you that you look on the Hadoop MapReduce API for this 
class.(RunningJob)



From general-return-3151-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 23:05:23 2011
Return-Path: <general-return-3151-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24394 invoked from network); 4 Mar 2011 23:05:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 23:05:22 -0000
Received: (qmail 62710 invoked by uid 500); 4 Mar 2011 23:05:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62613 invoked by uid 500); 4 Mar 2011 23:05:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62471 invoked by uid 99); 4 Mar 2011 23:05:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 23:05:20 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 23:05:13 +0000
Received: by wya21 with SMTP id 21so3282400wya.35
        for <general@hadoop.apache.org>; Fri, 04 Mar 2011 15:04:53 -0800 (PST)
Received: by 10.216.141.142 with SMTP id g14mr1114441wej.15.1299279893117;
 Fri, 04 Mar 2011 15:04:53 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.180.70 with HTTP; Fri, 4 Mar 2011 15:04:33 -0800 (PST)
In-Reply-To: <AANLkTimNZUnHqPAeRgynNbnpP7rPiKc6spHsxmFEs-D-@mail.gmail.com>
References: <1013602343.131299102932362.JavaMail.hudson@aegis>
 <661564799.1071299188134602.JavaMail.hudson@aegis> <AANLkTimd4YGLHcztbcDd4-b9cj8VYJMZMYCDyp8v2jNb@mail.gmail.com>
 <AANLkTimNZUnHqPAeRgynNbnpP7rPiKc6spHsxmFEs-D-@mail.gmail.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Fri, 4 Mar 2011 15:04:33 -0800
X-Google-Sender-Auth: LD93KX9wvd9fSlGJAYgDFwvTsMY
Message-ID: <AANLkTikoL=x1YehCCkbX08MPKwDdyfhK_9ynOtMMyYxA@mail.gmail.com>
Subject: Re: Hadoop-Mapreduce-trunk-Commit - Build # 621 - Still Failing
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Looks like the same symptoms are affecting Common trunk builds
  https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/516/
Have our build env. has been changed recently?
--
=A0 Take care,
Konstantin (Cos) Boudnik

On Thu, Mar 3, 2011 at 21:36, Konstantin Boudnik <cos@apache.org> wrote:
> That seems like settings.xml has been modified, but then it'd affect
> other builds as well. Unless we are using different machines/users for
> different component's builds.
>
> On Thu, Mar 3, 2011 at 13:41, Todd Lipcon <todd@cloudera.com> wrote:
>> Anyone understand why we can't seem to publish to apache maven anymore?
>>
>> -Todd
>>
>> On Thu, Mar 3, 2011 at 1:35 PM, Apache Hudson Server
>> <hudson@hudson.apache.org> wrote:
>>> See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/=
621/
>>>
>>> #######################################################################=
############
>>> ########################## LAST 60 LINES OF THE CONSOLE ###############=
############
>>> [...truncated 4464 lines...]
>>> =A0 =A0[javac] Compiling 11 source files to /grid/0/hudson/hudson-slave=
/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/build-fi/system/test/classes
>>> =A0 =A0[javac] Note: Some input files use or override a deprecated API.
>>> =A0 =A0[javac] Note: Recompile with -Xlint:deprecation for details.
>>>
>>> jar-test-system:
>>>
>>> -do-jar-test:
>>> =A0 =A0 =A0[jar] Building jar: /grid/0/hudson/hudson-slave/workspace/Ha=
doop-Mapreduce-trunk-Commit/trunk/build-fi/system/hadoop-mapred-instrumente=
d-test-0.23.0-SNAPSHOT.jar
>>> =A0 =A0 =A0[jar] Building jar: /grid/0/hudson/hudson-slave/workspace/Ha=
doop-Mapreduce-trunk-Commit/trunk/build-fi/system/hadoop-mapred-instrumente=
d-test-0.23.0-SNAPSHOT-sources.jar
>>>
>>> set-version:
>>> =A0 =A0 [copy] Copying 1 file to /grid/0/hudson/hudson-slave/workspace/=
Hadoop-Mapreduce-trunk-Commit/trunk/ivy
>>> =A0 =A0 [copy] Copying 1 file to /grid/0/hudson/hudson-slave/workspace/=
Hadoop-Mapreduce-trunk-Commit/trunk/ivy
>>> =A0 =A0 [copy] Copying 1 file to /grid/0/hudson/hudson-slave/workspace/=
Hadoop-Mapreduce-trunk-Commit/trunk/ivy
>>> =A0 =A0 [copy] Copying 1 file to /grid/0/hudson/hudson-slave/workspace/=
Hadoop-Mapreduce-trunk-Commit/trunk/ivy
>>>
>>> clean-sign:
>>>
>>> sign:
>>>
>>> signanddeploy:
>>>
>>> simpledeploy:
>>> [artifact:install-provider] Installing provider: org.apache.maven.wagon=
:wagon-http:jar:1.0-beta-2:runtime
>>> [artifact:deploy] Deploying to https://repository.apache.org/content/re=
positories/snapshots
>>> [artifact:deploy] [INFO] Retrieving previous build number from apache.s=
napshots.https
>>> [artifact:deploy] Uploading: org/apache/hadoop/hadoop-mapred/0.23.0-SNA=
PSHOT/hadoop-mapred-0.23.0-20110303.213523-60.jar to apache.snapshots.https
>>> [artifact:deploy] Uploaded 1741K
>>> [artifact:deploy] An error has occurred while processing the Maven arti=
fact tasks.
>>> [artifact:deploy] =A0Diagnosis:
>>> [artifact:deploy]
>>> [artifact:deploy] Error deploying artifact 'org.apache.hadoop:hadoop-ma=
pred:jar': Error deploying artifact: Failed to transfer file: https://repos=
itory.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-ma=
pred/0.23.0-SNAPSHOT/hadoop-mapred-0.23.0-20110303.213523-60.jar. Return co=
de is: 401
>>>
>>> BUILD FAILED
>>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/tru=
nk/build.xml:1528: Error deploying artifact 'org.apache.hadoop:hadoop-mapre=
d:jar': Error deploying artifact: Failed to transfer file: https://reposito=
ry.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-mapre=
d/0.23.0-SNAPSHOT/hadoop-mapred-0.23.0-20110303.213523-60.jar. Return code =
is: 401
>>>
>>> Total time: 3 minutes 20 seconds
>>>
>>>
>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>> STORE: saving artifacts
>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>>
>>>
>>> mv: cannot stat `build/*.tar.gz': No such file or directory
>>> mv: cannot stat `build/test/findbugs': No such file or directory
>>> mv: cannot stat `build/docs/api': No such file or directory
>>> Build Failed
>>> [FINDBUGS] Skipping publisher since build result is FAILURE
>>> Recording fingerprints
>>> Archiving artifacts
>>> Recording test results
>>> Publishing Javadoc
>>> Publishing Clover coverage report...
>>> No Clover report will be published due to a Build Failure
>>> Email was triggered for: Failure
>>> Sending email for trigger: Failure
>>>
>>>
>>>
>>> #######################################################################=
############
>>> ############################## FAILED TESTS (if any) ##################=
############
>>> No tests ran.
>>>
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>

From general-return-3152-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 05 00:22:49 2011
Return-Path: <general-return-3152-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59307 invoked from network); 5 Mar 2011 00:22:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Mar 2011 00:22:48 -0000
Received: (qmail 39605 invoked by uid 500); 5 Mar 2011 00:22:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39504 invoked by uid 500); 5 Mar 2011 00:22:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39487 invoked by uid 99); 5 Mar 2011 00:22:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Mar 2011 00:22:46 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Mar 2011 00:22:38 +0000
Received: from SP2-EX07CAS05.ds.corp.yahoo.com (sp2-ex07cas05.corp.sp2.yahoo.com [98.137.59.39])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p250Lvga051061;
	Fri, 4 Mar 2011 16:21:57 -0800 (PST)
Received: from SP2-EX07VS02.ds.corp.yahoo.com ([98.137.59.30]) by
 SP2-EX07CAS05.ds.corp.yahoo.com ([98.137.59.39]) with mapi; Fri, 4 Mar 2011
 16:21:57 -0800
From: "Giridharan  Kesavan" <gkesavan@yahoo-inc.com>
To: "mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>,
        "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 4 Mar 2011 16:21:55 -0800
Subject: Re: Hadoop-Mapreduce-trunk-Commit - Build # 621 - Still Failing
Thread-Topic: Hadoop-Mapreduce-trunk-Commit - Build # 621 - Still Failing
Thread-Index: AcvayRnCmrdJOscJTrCDvSrF6QdZHwAAjtEg
Message-ID: <C996C223.199FC%gkesavan@yahoo-inc.com>
In-Reply-To: <AANLkTikoL=x1YehCCkbX08MPKwDdyfhK_9ynOtMMyYxA@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-Entourage/13.8.0.101117
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

I'm lookin into it..
-Giri

On 3/4/11 3:04 PM, "Konstantin Boudnik" <cos@apache.org> wrote:

> Looks like the same symptoms are affecting Common trunk builds
>   https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/516/
> Have our build env. has been changed recently?
> --
> =A0 Take care,
> Konstantin (Cos) Boudnik
>=20
> On Thu, Mar 3, 2011 at 21:36, Konstantin Boudnik <cos@apache.org> wrote:
>> That seems like settings.xml has been modified, but then it'd affect
>> other builds as well. Unless we are using different machines/users for
>> different component's builds.
>>=20
>> On Thu, Mar 3, 2011 at 13:41, Todd Lipcon <todd@cloudera.com> wrote:
>>> Anyone understand why we can't seem to publish to apache maven anymore?
>>>=20
>>> -Todd
>>>=20
>>> On Thu, Mar 3, 2011 at 1:35 PM, Apache Hudson Server
>>> <hudson@hudson.apache.org> wrote:
>>>> See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit=
/621/
>>>>=20
>>>> ######################################################################=
#####
>>>> ########
>>>> ########################## LAST 60 LINES OF THE CONSOLE
>>>> ###########################
>>>> [...truncated 4464 lines...]
>>>> =A0 =A0[javac] Compiling 11 source files to
>>>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/tr=
unk/b
>>>> uild-fi/system/test/classes
>>>> =A0 =A0[javac] Note: Some input files use or override a deprecated API=
.
>>>> =A0 =A0[javac] Note: Recompile with -Xlint:deprecation for details.
>>>>=20
>>>> jar-test-system:
>>>>=20
>>>> -do-jar-test:
>>>> =A0 =A0 =A0[jar] Building jar:
>>>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/tr=
unk/b
>>>> uild-fi/system/hadoop-mapred-instrumented-test-0.23.0-SNAPSHOT.jar
>>>> =A0 =A0 =A0[jar] Building jar:
>>>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/tr=
unk/b
>>>> uild-fi/system/hadoop-mapred-instrumented-test-0.23.0-SNAPSHOT-sources=
.jar
>>>>=20
>>>> set-version:
>>>> =A0 =A0 [copy] Copying 1 file to
>>>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/tr=
unk/i
>>>> vy
>>>> =A0 =A0 [copy] Copying 1 file to
>>>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/tr=
unk/i
>>>> vy
>>>> =A0 =A0 [copy] Copying 1 file to
>>>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/tr=
unk/i
>>>> vy
>>>> =A0 =A0 [copy] Copying 1 file to
>>>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/tr=
unk/i
>>>> vy
>>>>=20
>>>> clean-sign:
>>>>=20
>>>> sign:
>>>>=20
>>>> signanddeploy:
>>>>=20
>>>> simpledeploy:
>>>> [artifact:install-provider] Installing provider:
>>>> org.apache.maven.wagon:wagon-http:jar:1.0-beta-2:runtime
>>>> [artifact:deploy] Deploying to
>>>> https://repository.apache.org/content/repositories/snapshots
>>>> [artifact:deploy] [INFO] Retrieving previous build number from
>>>> apache.snapshots.https
>>>> [artifact:deploy] Uploading:
>>>> org/apache/hadoop/hadoop-mapred/0.23.0-SNAPSHOT/hadoop-mapred-0.23.0-2=
01103
>>>> 03.213523-60.jar to apache.snapshots.https
>>>> [artifact:deploy] Uploaded 1741K
>>>> [artifact:deploy] An error has occurred while processing the Maven art=
ifact
>>>> tasks.
>>>> [artifact:deploy] =A0Diagnosis:
>>>> [artifact:deploy]
>>>> [artifact:deploy] Error deploying artifact
>>>> 'org.apache.hadoop:hadoop-mapred:jar': Error deploying artifact: Faile=
d to
>>>> transfer file:
>>>> https://repository.apache.org/content/repositories/snapshots/org/apach=
e/had
>>>> oop/hadoop-mapred/0.23.0-SNAPSHOT/hadoop-mapred-0.23.0-20110303.213523=
-60.j
>>>> ar. Return code is: 401
>>>>=20
>>>> BUILD FAILED
>>>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/tr=
unk/b
>>>> uild.xml:1528: Error deploying artifact
>>>> 'org.apache.hadoop:hadoop-mapred:jar': Error deploying artifact: Faile=
d to
>>>> transfer file:
>>>> https://repository.apache.org/content/repositories/snapshots/org/apach=
e/had
>>>> oop/hadoop-mapred/0.23.0-SNAPSHOT/hadoop-mapred-0.23.0-20110303.213523=
-60.j
>>>> ar. Return code is: 401
>>>>=20
>>>> Total time: 3 minutes 20 seconds
>>>>=20
>>>>=20
>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>>> STORE: saving artifacts
>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>>>=20
>>>>=20
>>>> mv: cannot stat `build/*.tar.gz': No such file or directory
>>>> mv: cannot stat `build/test/findbugs': No such file or directory
>>>> mv: cannot stat `build/docs/api': No such file or directory
>>>> Build Failed
>>>> [FINDBUGS] Skipping publisher since build result is FAILURE
>>>> Recording fingerprints
>>>> Archiving artifacts
>>>> Recording test results
>>>> Publishing Javadoc
>>>> Publishing Clover coverage report...
>>>> No Clover report will be published due to a Build Failure
>>>> Email was triggered for: Failure
>>>> Sending email for trigger: Failure
>>>>=20
>>>>=20
>>>>=20
>>>> ######################################################################=
#####
>>>> ########
>>>> ############################## FAILED TESTS (if any)
>>>> ##############################
>>>> No tests ran.
>>>>=20
>>>=20
>>>=20
>>>=20
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>>=20
>>=20


From general-return-3153-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 05 00:31:13 2011
Return-Path: <general-return-3153-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94390 invoked from network); 5 Mar 2011 00:31:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Mar 2011 00:31:12 -0000
Received: (qmail 54012 invoked by uid 500); 5 Mar 2011 00:31:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53877 invoked by uid 500); 5 Mar 2011 00:31:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53861 invoked by uid 99); 5 Mar 2011 00:31:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Mar 2011 00:31:10 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Mar 2011 00:31:04 +0000
Received: from SP2-EX07CAS03.ds.corp.yahoo.com (sp2-ex07cas03.corp.sp2.yahoo.com [98.137.59.35])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p250UUfW066528;
	Fri, 4 Mar 2011 16:30:30 -0800 (PST)
Received: from SP2-EX07VS02.ds.corp.yahoo.com ([98.137.59.30]) by
 SP2-EX07CAS03.ds.corp.yahoo.com ([98.137.59.35]) with mapi; Fri, 4 Mar 2011
 16:30:30 -0800
From: "Giridharan  Kesavan" <gkesavan@yahoo-inc.com>
To: "mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>,
        "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 4 Mar 2011 16:30:28 -0800
Subject: Re: Hadoop-Mapreduce-trunk-Commit - Build # 621 - Still Failing
Thread-Topic: Hadoop-Mapreduce-trunk-Commit - Build # 621 - Still Failing
Thread-Index: AcvayRnCmrdJOscJTrCDvSrF6QdZHwAAjtEgAABMcYY=
Message-ID: <C996C424.19A03%gkesavan@yahoo-inc.com>
In-Reply-To: <C996C223.199FC%gkesavan@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-Entourage/13.8.0.101117
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

It looks like settings.xml file is deleted by someone while cleaning the m2
repository (my guess). I ve copied it back.
I'm triggering a build to verify this.

-Giri


On 3/4/11 4:21 PM, "Giridharan Kesavan" <gkesavan@yahoo-inc.com> wrote:

> I'm lookin into it..
> -Giri
>=20
> On 3/4/11 3:04 PM, "Konstantin Boudnik" <cos@apache.org> wrote:
>=20
>> Looks like the same symptoms are affecting Common trunk builds
>>   https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/516/
>> Have our build env. has been changed recently?
>> --
>> =A0 Take care,
>> Konstantin (Cos) Boudnik
>>=20
>> On Thu, Mar 3, 2011 at 21:36, Konstantin Boudnik <cos@apache.org> wrote:
>>> That seems like settings.xml has been modified, but then it'd affect
>>> other builds as well. Unless we are using different machines/users for
>>> different component's builds.
>>>=20
>>> On Thu, Mar 3, 2011 at 13:41, Todd Lipcon <todd@cloudera.com> wrote:
>>>> Anyone understand why we can't seem to publish to apache maven anymore=
?
>>>>=20
>>>> -Todd
>>>>=20
>>>> On Thu, Mar 3, 2011 at 1:35 PM, Apache Hudson Server
>>>> <hudson@hudson.apache.org> wrote:
>>>>> See=20
>>>>> https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/62=
1/
>>>>>=20
>>>>>=20
##########################################################################>=
>>>>
#
>>>>> ########
>>>>> ########################## LAST 60 LINES OF THE CONSOLE
>>>>> ###########################
>>>>> [...truncated 4464 lines...]
>>>>> =A0 =A0[javac] Compiling 11 source files to
>>>>>=20
/grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/>=
>>>>
b
>>>>> uild-fi/system/test/classes
>>>>> =A0 =A0[javac] Note: Some input files use or override a deprecated AP=
I.
>>>>> =A0 =A0[javac] Note: Recompile with -Xlint:deprecation for details.
>>>>>=20
>>>>> jar-test-system:
>>>>>=20
>>>>> -do-jar-test:
>>>>> =A0 =A0 =A0[jar] Building jar:
>>>>>=20
/grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/>=
>>>>
b
>>>>> uild-fi/system/hadoop-mapred-instrumented-test-0.23.0-SNAPSHOT.jar
>>>>> =A0 =A0 =A0[jar] Building jar:
>>>>>=20
/grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/>=
>>>>
b
>>>>> uild-fi/system/hadoop-mapred-instrumented-test-0.23.0-SNAPSHOT-source=
s.jar
>>>>>=20
>>>>> set-version:
>>>>> =A0 =A0 [copy] Copying 1 file to
>>>>>=20
/grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/>=
>>>>
i
>>>>> vy
>>>>> =A0 =A0 [copy] Copying 1 file to
>>>>>=20
/grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/>=
>>>>
i
>>>>> vy
>>>>> =A0 =A0 [copy] Copying 1 file to
>>>>>=20
/grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/>=
>>>>
i
>>>>> vy
>>>>> =A0 =A0 [copy] Copying 1 file to
>>>>>=20
/grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/>=
>>>>
i
>>>>> vy
>>>>>=20
>>>>> clean-sign:
>>>>>=20
>>>>> sign:
>>>>>=20
>>>>> signanddeploy:
>>>>>=20
>>>>> simpledeploy:
>>>>> [artifact:install-provider] Installing provider:
>>>>> org.apache.maven.wagon:wagon-http:jar:1.0-beta-2:runtime
>>>>> [artifact:deploy] Deploying to
>>>>> https://repository.apache.org/content/repositories/snapshots
>>>>> [artifact:deploy] [INFO] Retrieving previous build number from
>>>>> apache.snapshots.https
>>>>> [artifact:deploy] Uploading:
>>>>>=20
org/apache/hadoop/hadoop-mapred/0.23.0-SNAPSHOT/hadoop-mapred-0.23.0-20110>=
>>>>
3
>>>>> 03.213523-60.jar to apache.snapshots.https
>>>>> [artifact:deploy] Uploaded 1741K
>>>>> [artifact:deploy] An error has occurred while processing the Maven
>>>>> artifact
>>>>> tasks.
>>>>> [artifact:deploy] =A0Diagnosis:
>>>>> [artifact:deploy]
>>>>> [artifact:deploy] Error deploying artifact
>>>>> 'org.apache.hadoop:hadoop-mapred:jar': Error deploying artifact: Fail=
ed to
>>>>> transfer file:
>>>>>=20
https://repository.apache.org/content/repositories/snapshots/org/apache/ha>=
>>>>
d
>>>>>=20
oop/hadoop-mapred/0.23.0-SNAPSHOT/hadoop-mapred-0.23.0-20110303.213523-60.>=
>>>>
j
>>>>> ar. Return code is: 401
>>>>>=20
>>>>> BUILD FAILED
>>>>>=20
/grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/>=
>>>>
b
>>>>> uild.xml:1528: Error deploying artifact
>>>>> 'org.apache.hadoop:hadoop-mapred:jar': Error deploying artifact: Fail=
ed to
>>>>> transfer file:
>>>>>=20
https://repository.apache.org/content/repositories/snapshots/org/apache/ha>=
>>>>
d
>>>>>=20
oop/hadoop-mapred/0.23.0-SNAPSHOT/hadoop-mapred-0.23.0-20110303.213523-60.>=
>>>>
j
>>>>> ar. Return code is: 401
>>>>>=20
>>>>> Total time: 3 minutes 20 seconds
>>>>>=20
>>>>>=20
>>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>>>> STORE: saving artifacts
>>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>>>>=20
>>>>>=20
>>>>> mv: cannot stat `build/*.tar.gz': No such file or directory
>>>>> mv: cannot stat `build/test/findbugs': No such file or directory
>>>>> mv: cannot stat `build/docs/api': No such file or directory
>>>>> Build Failed
>>>>> [FINDBUGS] Skipping publisher since build result is FAILURE
>>>>> Recording fingerprints
>>>>> Archiving artifacts
>>>>> Recording test results
>>>>> Publishing Javadoc
>>>>> Publishing Clover coverage report...
>>>>> No Clover report will be published due to a Build Failure
>>>>> Email was triggered for: Failure
>>>>> Sending email for trigger: Failure
>>>>>=20
>>>>>=20
>>>>>=20
>>>>>=20
##########################################################################>=
>>>>
#
>>>>> ########
>>>>> ############################## FAILED TESTS (if any)
>>>>> ##############################
>>>>> No tests ran.
>>>>>=20
>>>>=20
>>>>=20
>>>>=20
>>>> --
>>>> Todd Lipcon
>>>> Software Engineer, Cloudera
>>>>=20
>>>=20
>=20


From general-return-3154-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 05 00:40:46 2011
Return-Path: <general-return-3154-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33570 invoked from network); 5 Mar 2011 00:40:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Mar 2011 00:40:45 -0000
Received: (qmail 70824 invoked by uid 500); 5 Mar 2011 00:40:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70645 invoked by uid 500); 5 Mar 2011 00:40:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 62289 invoked by uid 99); 5 Mar 2011 00:35:05 -0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cos@cloudera.com designates 209.85.161.176 as permitted sender)
MIME-Version: 1.0
Sender: cos@cloudera.com
In-Reply-To: <C996C424.19A03%gkesavan@yahoo-inc.com>
References: <C996C223.199FC%gkesavan@yahoo-inc.com> <C996C424.19A03%gkesavan@yahoo-inc.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Fri, 4 Mar 2011 16:34:17 -0800
X-Google-Sender-Auth: z_oWUpOsHHV_PDIrTC2dFHhRz2c
Message-ID: <AANLkTi=Nvkay8kcsB2JLZO3S=qKdzpWKLFLuuLYDQZ90@mail.gmail.com>
Subject: Re: Hadoop-Mapreduce-trunk-Commit - Build # 621 - Still Failing
To: mapreduce-dev@hadoop.apache.org
Cc: Giridharan Kesavan <gkesavan@yahoo-inc.com>, 
	"general@hadoop.apache.org" <general@hadoop.apache.org>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks a lot Giri!
I guess this is another reason to have CM in place for the build machines.

On Fri, Mar 4, 2011 at 16:30, Giridharan  Kesavan
<gkesavan@yahoo-inc.com> wrote:
> It looks like settings.xml file is deleted by someone while cleaning the =
m2
> repository (my guess). I ve copied it back.
> I'm triggering a build to verify this.
>
> -Giri
>
>
> On 3/4/11 4:21 PM, "Giridharan Kesavan" <gkesavan@yahoo-inc.com> wrote:
>
>> I'm lookin into it..
>> -Giri
>>
>> On 3/4/11 3:04 PM, "Konstantin Boudnik" <cos@apache.org> wrote:
>>
>>> Looks like the same symptoms are affecting Common trunk builds
>>> =C2=A0 https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/=
516/
>>> Have our build env. has been changed recently?
>>> --
>>> =C2=A0 Take care,
>>> Konstantin (Cos) Boudnik
>>>
>>> On Thu, Mar 3, 2011 at 21:36, Konstantin Boudnik <cos@apache.org> wrote=
:
>>>> That seems like settings.xml has been modified, but then it'd affect
>>>> other builds as well. Unless we are using different machines/users for
>>>> different component's builds.
>>>>
>>>> On Thu, Mar 3, 2011 at 13:41, Todd Lipcon <todd@cloudera.com> wrote:
>>>>> Anyone understand why we can't seem to publish to apache maven anymor=
e?
>>>>>
>>>>> -Todd
>>>>>
>>>>> On Thu, Mar 3, 2011 at 1:35 PM, Apache Hudson Server
>>>>> <hudson@hudson.apache.org> wrote:
>>>>>> See
>>>>>> https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/6=
21/
>>>>>>
>>>>>>
> #########################################################################=
#>>>>>
> #
>>>>>> ########
>>>>>> ########################## LAST 60 LINES OF THE CONSOLE
>>>>>> ###########################
>>>>>> [...truncated 4464 lines...]
>>>>>> =C2=A0 =C2=A0[javac] Compiling 11 source files to
>>>>>>
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk=
/>>>>>
> b
>>>>>> uild-fi/system/test/classes
>>>>>> =C2=A0 =C2=A0[javac] Note: Some input files use or override a deprec=
ated API.
>>>>>> =C2=A0 =C2=A0[javac] Note: Recompile with -Xlint:deprecation for det=
ails.
>>>>>>
>>>>>> jar-test-system:
>>>>>>
>>>>>> -do-jar-test:
>>>>>> =C2=A0 =C2=A0 =C2=A0[jar] Building jar:
>>>>>>
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk=
/>>>>>
> b
>>>>>> uild-fi/system/hadoop-mapred-instrumented-test-0.23.0-SNAPSHOT.jar
>>>>>> =C2=A0 =C2=A0 =C2=A0[jar] Building jar:
>>>>>>
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk=
/>>>>>
> b
>>>>>> uild-fi/system/hadoop-mapred-instrumented-test-0.23.0-SNAPSHOT-sourc=
es.jar
>>>>>>
>>>>>> set-version:
>>>>>> =C2=A0 =C2=A0 [copy] Copying 1 file to
>>>>>>
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk=
/>>>>>
> i
>>>>>> vy
>>>>>> =C2=A0 =C2=A0 [copy] Copying 1 file to
>>>>>>
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk=
/>>>>>
> i
>>>>>> vy
>>>>>> =C2=A0 =C2=A0 [copy] Copying 1 file to
>>>>>>
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk=
/>>>>>
> i
>>>>>> vy
>>>>>> =C2=A0 =C2=A0 [copy] Copying 1 file to
>>>>>>
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk=
/>>>>>
> i
>>>>>> vy
>>>>>>
>>>>>> clean-sign:
>>>>>>
>>>>>> sign:
>>>>>>
>>>>>> signanddeploy:
>>>>>>
>>>>>> simpledeploy:
>>>>>> [artifact:install-provider] Installing provider:
>>>>>> org.apache.maven.wagon:wagon-http:jar:1.0-beta-2:runtime
>>>>>> [artifact:deploy] Deploying to
>>>>>> https://repository.apache.org/content/repositories/snapshots
>>>>>> [artifact:deploy] [INFO] Retrieving previous build number from
>>>>>> apache.snapshots.https
>>>>>> [artifact:deploy] Uploading:
>>>>>>
> org/apache/hadoop/hadoop-mapred/0.23.0-SNAPSHOT/hadoop-mapred-0.23.0-2011=
0>>>>>
> 3
>>>>>> 03.213523-60.jar to apache.snapshots.https
>>>>>> [artifact:deploy] Uploaded 1741K
>>>>>> [artifact:deploy] An error has occurred while processing the Maven
>>>>>> artifact
>>>>>> tasks.
>>>>>> [artifact:deploy] =C2=A0Diagnosis:
>>>>>> [artifact:deploy]
>>>>>> [artifact:deploy] Error deploying artifact
>>>>>> 'org.apache.hadoop:hadoop-mapred:jar': Error deploying artifact: Fai=
led to
>>>>>> transfer file:
>>>>>>
> https://repository.apache.org/content/repositories/snapshots/org/apache/h=
a>>>>>
> d
>>>>>>
> oop/hadoop-mapred/0.23.0-SNAPSHOT/hadoop-mapred-0.23.0-20110303.213523-60=
.>>>>>
> j
>>>>>> ar. Return code is: 401
>>>>>>
>>>>>> BUILD FAILED
>>>>>>
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk=
/>>>>>
> b
>>>>>> uild.xml:1528: Error deploying artifact
>>>>>> 'org.apache.hadoop:hadoop-mapred:jar': Error deploying artifact: Fai=
led to
>>>>>> transfer file:
>>>>>>
> https://repository.apache.org/content/repositories/snapshots/org/apache/h=
a>>>>>
> d
>>>>>>
> oop/hadoop-mapred/0.23.0-SNAPSHOT/hadoop-mapred-0.23.0-20110303.213523-60=
.>>>>>
> j
>>>>>> ar. Return code is: 401
>>>>>>
>>>>>> Total time: 3 minutes 20 seconds
>>>>>>
>>>>>>
>>>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>>>>> STORE: saving artifacts
>>>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>>>>>
>>>>>>
>>>>>> mv: cannot stat `build/*.tar.gz': No such file or directory
>>>>>> mv: cannot stat `build/test/findbugs': No such file or directory
>>>>>> mv: cannot stat `build/docs/api': No such file or directory
>>>>>> Build Failed
>>>>>> [FINDBUGS] Skipping publisher since build result is FAILURE
>>>>>> Recording fingerprints
>>>>>> Archiving artifacts
>>>>>> Recording test results
>>>>>> Publishing Javadoc
>>>>>> Publishing Clover coverage report...
>>>>>> No Clover report will be published due to a Build Failure
>>>>>> Email was triggered for: Failure
>>>>>> Sending email for trigger: Failure
>>>>>>
>>>>>>
>>>>>>
>>>>>>
> #########################################################################=
#>>>>>
> #
>>>>>> ########
>>>>>> ############################## FAILED TESTS (if any)
>>>>>> ##############################
>>>>>> No tests ran.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Todd Lipcon
>>>>> Software Engineer, Cloudera
>>>>>
>>>>
>>
>
>

From general-return-3155-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 05 05:49:39 2011
Return-Path: <general-return-3155-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27213 invoked from network); 5 Mar 2011 05:49:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Mar 2011 05:49:39 -0000
Received: (qmail 11633 invoked by uid 500); 5 Mar 2011 05:49:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 10418 invoked by uid 500); 5 Mar 2011 05:49:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 10294 invoked by uid 99); 5 Mar 2011 05:49:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Mar 2011 05:49:32 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Mar 2011 05:49:24 +0000
Received: by iwr19 with SMTP id 19so3433508iwr.35
        for <multiple recipients>; Fri, 04 Mar 2011 21:49:04 -0800 (PST)
Received: by 10.42.29.71 with SMTP id q7mr1743760icc.152.1299304144092; Fri,
 04 Mar 2011 21:49:04 -0800 (PST)
MIME-Version: 1.0
Received: by 10.42.229.1 with HTTP; Fri, 4 Mar 2011 21:48:44 -0800 (PST)
X-Originating-IP: [67.160.196.149]
In-Reply-To: <AANLkTimA0FGsXqw0bKqZfz+yv-swG_MV798YGjky5dPe@mail.gmail.com>
References: <AANLkTimA0FGsXqw0bKqZfz+yv-swG_MV798YGjky5dPe@mail.gmail.com>
From: Ted Dunning <tdunning@maprtech.com>
Date: Fri, 4 Mar 2011 21:48:44 -0800
Message-ID: <AANLkTimJFeGSwyJAXCuZ2sDKJwSQMf_a+QFeGYvER=4K@mail.gmail.com>
Subject: Re: Digital Signal Processing Library + Hadoop
To: common-user@hadoop.apache.org
Cc: Roger Smith <rogersmith1711@gmail.com>, general@hadoop.apache.org, 
	Mahout Dev List <dev@mahout.apache.org>
Content-Type: multipart/alternative; boundary=20cf303f6c32f6bd80049db5d137

--20cf303f6c32f6bd80049db5d137
Content-Type: text/plain; charset=ISO-8859-1

Come on over to the Apache Mahout mailing list for a warm welcome at least.

We don't have a lot of time series stuff but would be very interested in
hearing more about what you need and would like to see if there are some
common issues that we might work on together.

On Fri, Mar 4, 2011 at 9:05 PM, Roger Smith <rogersmith1711@gmail.com>wrote:

> All -
> I wonder if any of you have integrated a DSP library with Hadoop.
> We are considering using Hadoop to processing time series data, but don't
> want to write standard DSP functions.
>
> Roger.
>

--20cf303f6c32f6bd80049db5d137--

From general-return-3156-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 05 16:05:01 2011
Return-Path: <general-return-3156-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 65690 invoked from network); 5 Mar 2011 16:05:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Mar 2011 16:05:01 -0000
Received: (qmail 71172 invoked by uid 500); 5 Mar 2011 16:04:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71111 invoked by uid 500); 5 Mar 2011 16:04:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 15265 invoked by uid 99); 5 Mar 2011 05:05:49 -0000
X-ASF-Spam-Status: No, hits=3.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of rogersmith1711@gmail.com designates 74.125.82.176 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=JytTLjb0APXr9cb801/sQ507zcgdDkjPy23zD2HNUOY=;
        b=xVwnWihxtkR9hgt+pByytBRfvaz32KDJMyaGXPWa8daW/i4hnq/ecw0F+1OYxO8dTf
         F2zR/R9drNqlrj9YWsoMv62fgTBjCWdSKQffClxXxrqHq9KlG9+PiO/gbJXmm1ytdAZC
         I43RKPuA1vnkDBGCUpJW23RiRyInJrvKOgmms=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=Wjr3Z9XGtiDneYkD7GR2MpbsFrghNm1XflCE8ubbNe4N9nVcKWEe7TjPzR1nvY7jU0
         RvbMHxFrRYg6jntvV2DqAZBaOZbnNX1e/izzh9IybeHUdhusJv/d7NtajRmP6vP8GDRO
         VdsGMPD7PikH5e9KJHi7zBvl8DnVxhxNcIpno=
MIME-Version: 1.0
Date: Fri, 4 Mar 2011 21:05:19 -0800
Message-ID: <AANLkTimA0FGsXqw0bKqZfz+yv-swG_MV798YGjky5dPe@mail.gmail.com>
Subject: Digital Signal Processing Library + Hadoop
From: Roger Smith <rogersmith1711@gmail.com>
To: common-user <common-user@hadoop.apache.org>, general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=485b3973ed7f866dd5049db535d6
X-Virus-Checked: Checked by ClamAV on apache.org

--485b3973ed7f866dd5049db535d6
Content-Type: text/plain; charset=ISO-8859-1

All -
I wonder if any of you have integrated a DSP library with Hadoop.
We are considering using Hadoop to processing time series data, but don't
want to write standard DSP functions.

Roger.

--485b3973ed7f866dd5049db535d6--

From general-return-3157-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 05 21:30:26 2011
Return-Path: <general-return-3157-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77416 invoked from network); 5 Mar 2011 21:30:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Mar 2011 21:30:25 -0000
Received: (qmail 76338 invoked by uid 500); 5 Mar 2011 21:30:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76279 invoked by uid 500); 5 Mar 2011 21:30:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76268 invoked by uid 99); 5 Mar 2011 21:30:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Mar 2011 21:30:24 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Mar 2011 21:30:18 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p25LTrYw028581
	for <general@hadoop.apache.org>; Sat, 5 Mar 2011 13:29:53 -0800 (PST)
Received: from [10.0.1.2] (10.72.168.26) by SP2-EX07CAS01.ds.corp.yahoo.com
 (98.137.59.4) with Microsoft SMTP Server (TLS) id 8.3.137.0; Sat, 5 Mar 2011
 13:29:52 -0800
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [DISCUSS] Move common, hdfs, mapreduce contrib components to apache-extras.org or elsewhere
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <5E398D0F-ABCC-497B-939A-9213A5B3C7C4@mac.com>
Date: Sat, 5 Mar 2011 13:29:50 -0800
Content-Transfer-Encoding: quoted-printable
Message-ID: <7F080B69-E0AE-4A38-9928-048377E73E72@yahoo-inc.com>
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com> <AANLkTi=75e6oyBuGVivXP3is+76gOODSc1NcLKfg=VxF@mail.gmail.com> <AANLkTi=HQ1cn++JO6g+vp-cJAtAjB00Ok-FW2VrfTiSO@mail.gmail.com> <5E398D0F-ABCC-497B-939A-9213A5B3C7C4@mac.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)

Hi Nigel,

I liked your previous plan.  Could you summarize your current plan?

Are we now planning to leave inactive projects in place?  Ship them as =
source? =20

Seems like this still sends a confusing message.  I'm all for removing =
them from trunk in SVN, leaving a wiki link to their most recent state =
and letting their contributors make a case to revive them in whatever =
place they choose.

Keeping projects with small constituencies in our tree seems counter =
productive.

E14

On Feb 11, 2011, at 10:01 PM, Nigel Daley wrote:

> +1.  And only include them as source in releases.
>=20
> Nige=20
>=20
> On Feb 11, 2011, at 10:12 AM, Tom White wrote:
>=20
>> For contrib components that we elect not to remove, I suggest that we
>> remove them from the main build, so that failures in a contrib
>> component don't hinder the main build and release. This is what
>> ZooKeeper does, for example.
>>=20
>> Tom
>>=20
>> On Fri, Feb 11, 2011 at 2:03 AM, Bernd Fondermann
>> <bernd.fondermann@googlemail.com> wrote:
>>> -1.
>>>=20
>>> Move it away from TRUNK so it doesn't affect builds is a much better
>>> (and simpler) solution. If someone wants to revive it, he can within
>>> the bounds of Apache Hadoop and will become a part of the Hadoop
>>> community, which would be good.
>>> If you'd move it off-site, if the code ever gets new contributors,
>>> it's hard to integrate them (code and contributors) into Hadoop =
again.
>>>=20
>>> AFAIUI, apache-extras is for placing non-Apache code closer to the
>>> related Apache projects, not for moving our code away from our own
>>> premises.
>>>=20
>>> Bernd
>>>=20
>>> On Mon, Jan 31, 2011 at 04:42, Nigel Daley <ndaley@mac.com> wrote:
>>>> Folks,
>>>>=20
>>>> Now that http://apache-extras.org is launched =
(https://blogs.apache.org/foundation/entry/the_apache_software_foundation_=
launches) I'd like to start a discussion on moving contrib components =
out of common, mapreduce, and hdfs.
>>>>=20
>>>> These contrib components complicate the builds, cause test failures =
that nobody seems to care about, have releases that are tied to Hadoop's =
long release cycles, etc.  Most folks I've talked with agree that these =
contrib components would be better served by being pulled out of Hadoop =
and hosted elsewhere. The new apache-extras code hosting site seems like =
a natural *default* location for migrating these contrib projects.  =
Perhaps some should graduate from contrib to src (ie from contrib to =
core of the project they're included in).  If folks agree, we'll need to =
come up with a mapping of contrib component to it's final destination =
and file a jira.
>>>>=20
>>>> Here are the contrib components by project (hopefully I didn't miss =
any).
>>>>=20
>>>> Common Contrib:
>>>> failmon
>>>> hod
>>>> test
>>>>=20
>>>>=20
>>>> MapReduce Contrib:
>>>> capacity-scheduler -- move to MR core?
>>>> data_join
>>>> dynamic-scheduler
>>>> eclipse-plugin
>>>> fairscheduler -- move to MR core?
>>>> gridmix
>>>> index
>>>> mrunit
>>>> mumak
>>>> raid
>>>> sqoop
>>>> streaming -- move to MR core?
>>>> vaidya
>>>> vertica
>>>>=20
>>>>=20
>>>> HDFS Contrib:
>>>> fuse-dfs
>>>> hdfsproxy
>>>> thriftfs
>>>>=20
>>>>=20
>>>> Cheers,
>>>> Nige
>>>>=20
>>>=20
>=20


From general-return-3158-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 22:13:34 2011
Return-Path: <general-return-3158-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59113 invoked from network); 7 Mar 2011 22:13:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 22:13:33 -0000
Received: (qmail 30857 invoked by uid 500); 7 Mar 2011 22:13:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30648 invoked by uid 500); 7 Mar 2011 22:13:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30355 invoked by uid 99); 7 Mar 2011 22:13:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 22:13:30 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 22:13:24 +0000
Received: by bwz8 with SMTP id 8so5746337bwz.35
        for <multiple recipients>; Mon, 07 Mar 2011 14:13:04 -0800 (PST)
Received: by 10.204.227.193 with SMTP id jb1mr3761248bkb.157.1299535981171;
 Mon, 07 Mar 2011 14:13:01 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.154.208 with HTTP; Mon, 7 Mar 2011 14:12:41 -0800 (PST)
In-Reply-To: <AANLkTi=Nvkay8kcsB2JLZO3S=qKdzpWKLFLuuLYDQZ90@mail.gmail.com>
References: <C996C223.199FC%gkesavan@yahoo-inc.com> <C996C424.19A03%gkesavan@yahoo-inc.com>
 <AANLkTi=Nvkay8kcsB2JLZO3S=qKdzpWKLFLuuLYDQZ90@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 7 Mar 2011 14:12:41 -0800
Message-ID: <AANLkTi=yoi5pua_VDmWUfem3CtArT6rTMgZ7G6oUkbFh@mail.gmail.com>
Subject: Re: Hadoop-Mapreduce-trunk-Commit - Build # 621 - Still Failing
To: mapreduce-dev@hadoop.apache.org
Cc: Konstantin Boudnik <cos@apache.org>, Giridharan Kesavan <gkesavan@yahoo-inc.com>, 
	"general@hadoop.apache.org" <general@hadoop.apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

It looks like something automated must be rming the settings.xml file.

Maybe one of our builds has an "rm -Rf ~/.m2" in it or something?

According to the docs I've found, we could also put this setting in
$ANT_HOME/etc. Maybe that's a safer spot?

Or put it in the spot in ~/.m2 but with permissions set to make it
harder to remove?

-Todd

On Fri, Mar 4, 2011 at 4:34 PM, Konstantin Boudnik <cos@apache.org> wrote:
> Thanks a lot Giri!
> I guess this is another reason to have CM in place for the build machines=
.
>
> On Fri, Mar 4, 2011 at 16:30, Giridharan =A0Kesavan
> <gkesavan@yahoo-inc.com> wrote:
>> It looks like settings.xml file is deleted by someone while cleaning the=
 m2
>> repository (my guess). I ve copied it back.
>> I'm triggering a build to verify this.
>>
>> -Giri
>>
>>
>> On 3/4/11 4:21 PM, "Giridharan Kesavan" <gkesavan@yahoo-inc.com> wrote:
>>
>>> I'm lookin into it..
>>> -Giri
>>>
>>> On 3/4/11 3:04 PM, "Konstantin Boudnik" <cos@apache.org> wrote:
>>>
>>>> Looks like the same symptoms are affecting Common trunk builds
>>>> =A0 https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/51=
6/
>>>> Have our build env. has been changed recently?
>>>> --
>>>> =A0 Take care,
>>>> Konstantin (Cos) Boudnik
>>>>
>>>> On Thu, Mar 3, 2011 at 21:36, Konstantin Boudnik <cos@apache.org> wrot=
e:
>>>>> That seems like settings.xml has been modified, but then it'd affect
>>>>> other builds as well. Unless we are using different machines/users fo=
r
>>>>> different component's builds.
>>>>>
>>>>> On Thu, Mar 3, 2011 at 13:41, Todd Lipcon <todd@cloudera.com> wrote:
>>>>>> Anyone understand why we can't seem to publish to apache maven anymo=
re?
>>>>>>
>>>>>> -Todd
>>>>>>
>>>>>> On Thu, Mar 3, 2011 at 1:35 PM, Apache Hudson Server
>>>>>> <hudson@hudson.apache.org> wrote:
>>>>>>> See
>>>>>>> https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/=
621/
>>>>>>>
>>>>>>>
>> ########################################################################=
##>>>>>
>> #
>>>>>>> ########
>>>>>>> ########################## LAST 60 LINES OF THE CONSOLE
>>>>>>> ###########################
>>>>>>> [...truncated 4464 lines...]
>>>>>>> =A0 =A0[javac] Compiling 11 source files to
>>>>>>>
>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trun=
k/>>>>>
>> b
>>>>>>> uild-fi/system/test/classes
>>>>>>> =A0 =A0[javac] Note: Some input files use or override a deprecated =
API.
>>>>>>> =A0 =A0[javac] Note: Recompile with -Xlint:deprecation for details.
>>>>>>>
>>>>>>> jar-test-system:
>>>>>>>
>>>>>>> -do-jar-test:
>>>>>>> =A0 =A0 =A0[jar] Building jar:
>>>>>>>
>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trun=
k/>>>>>
>> b
>>>>>>> uild-fi/system/hadoop-mapred-instrumented-test-0.23.0-SNAPSHOT.jar
>>>>>>> =A0 =A0 =A0[jar] Building jar:
>>>>>>>
>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trun=
k/>>>>>
>> b
>>>>>>> uild-fi/system/hadoop-mapred-instrumented-test-0.23.0-SNAPSHOT-sour=
ces.jar
>>>>>>>
>>>>>>> set-version:
>>>>>>> =A0 =A0 [copy] Copying 1 file to
>>>>>>>
>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trun=
k/>>>>>
>> i
>>>>>>> vy
>>>>>>> =A0 =A0 [copy] Copying 1 file to
>>>>>>>
>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trun=
k/>>>>>
>> i
>>>>>>> vy
>>>>>>> =A0 =A0 [copy] Copying 1 file to
>>>>>>>
>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trun=
k/>>>>>
>> i
>>>>>>> vy
>>>>>>> =A0 =A0 [copy] Copying 1 file to
>>>>>>>
>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trun=
k/>>>>>
>> i
>>>>>>> vy
>>>>>>>
>>>>>>> clean-sign:
>>>>>>>
>>>>>>> sign:
>>>>>>>
>>>>>>> signanddeploy:
>>>>>>>
>>>>>>> simpledeploy:
>>>>>>> [artifact:install-provider] Installing provider:
>>>>>>> org.apache.maven.wagon:wagon-http:jar:1.0-beta-2:runtime
>>>>>>> [artifact:deploy] Deploying to
>>>>>>> https://repository.apache.org/content/repositories/snapshots
>>>>>>> [artifact:deploy] [INFO] Retrieving previous build number from
>>>>>>> apache.snapshots.https
>>>>>>> [artifact:deploy] Uploading:
>>>>>>>
>> org/apache/hadoop/hadoop-mapred/0.23.0-SNAPSHOT/hadoop-mapred-0.23.0-201=
10>>>>>
>> 3
>>>>>>> 03.213523-60.jar to apache.snapshots.https
>>>>>>> [artifact:deploy] Uploaded 1741K
>>>>>>> [artifact:deploy] An error has occurred while processing the Maven
>>>>>>> artifact
>>>>>>> tasks.
>>>>>>> [artifact:deploy] =A0Diagnosis:
>>>>>>> [artifact:deploy]
>>>>>>> [artifact:deploy] Error deploying artifact
>>>>>>> 'org.apache.hadoop:hadoop-mapred:jar': Error deploying artifact: Fa=
iled to
>>>>>>> transfer file:
>>>>>>>
>> https://repository.apache.org/content/repositories/snapshots/org/apache/=
ha>>>>>
>> d
>>>>>>>
>> oop/hadoop-mapred/0.23.0-SNAPSHOT/hadoop-mapred-0.23.0-20110303.213523-6=
0.>>>>>
>> j
>>>>>>> ar. Return code is: 401
>>>>>>>
>>>>>>> BUILD FAILED
>>>>>>>
>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trun=
k/>>>>>
>> b
>>>>>>> uild.xml:1528: Error deploying artifact
>>>>>>> 'org.apache.hadoop:hadoop-mapred:jar': Error deploying artifact: Fa=
iled to
>>>>>>> transfer file:
>>>>>>>
>> https://repository.apache.org/content/repositories/snapshots/org/apache/=
ha>>>>>
>> d
>>>>>>>
>> oop/hadoop-mapred/0.23.0-SNAPSHOT/hadoop-mapred-0.23.0-20110303.213523-6=
0.>>>>>
>> j
>>>>>>> ar. Return code is: 401
>>>>>>>
>>>>>>> Total time: 3 minutes 20 seconds
>>>>>>>
>>>>>>>
>>>>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>>>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>>>>>> STORE: saving artifacts
>>>>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>>>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>>>>>>
>>>>>>>
>>>>>>> mv: cannot stat `build/*.tar.gz': No such file or directory
>>>>>>> mv: cannot stat `build/test/findbugs': No such file or directory
>>>>>>> mv: cannot stat `build/docs/api': No such file or directory
>>>>>>> Build Failed
>>>>>>> [FINDBUGS] Skipping publisher since build result is FAILURE
>>>>>>> Recording fingerprints
>>>>>>> Archiving artifacts
>>>>>>> Recording test results
>>>>>>> Publishing Javadoc
>>>>>>> Publishing Clover coverage report...
>>>>>>> No Clover report will be published due to a Build Failure
>>>>>>> Email was triggered for: Failure
>>>>>>> Sending email for trigger: Failure
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>> ########################################################################=
##>>>>>
>> #
>>>>>>> ########
>>>>>>> ############################## FAILED TESTS (if any)
>>>>>>> ##############################
>>>>>>> No tests ran.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Todd Lipcon
>>>>>> Software Engineer, Cloudera
>>>>>>
>>>>>
>>>
>>
>>
>



--=20
Todd Lipcon
Software Engineer, Cloudera

From general-return-3159-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 01:07:23 2011
Return-Path: <general-return-3159-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 52881 invoked from network); 8 Mar 2011 01:07:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 01:07:21 -0000
Received: (qmail 60849 invoked by uid 500); 8 Mar 2011 01:07:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60753 invoked by uid 500); 8 Mar 2011 01:07:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60738 invoked by uid 99); 8 Mar 2011 01:07:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 01:07:19 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 01:07:12 +0000
Received: from pbug.corp.yahoo.com (pbug.corp.yahoo.com [207.126.231.185])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p2816GlN080691;
	Mon, 7 Mar 2011 17:06:16 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1299546376;
	bh=WaixdO1kWF61Ax1MM//BmBD5tPDvUdGCSyDjVBUJnm0=;
	h=Date:Message-Id:From:Reply-To:To:Subject:Cc:In-Reply-To;
	b=UcMN9jcKJImY+fqMZhMkuETkVcih9QCV469uJRmay5JpZM3zzA8/9/eRx6lTHsYKE
	 C7tpUlHUw6fqeJDVMUNEDr5M9w2ykh+2rPyUyaESbuOTmcI+t6iODH9pzpZzwjh/e7
	 CNF1yYmty03erinch7pzWR86ZdY73tPKrWPABjyc=
Received: (from roelofs@localhost)
	by pbug.corp.yahoo.com (8.12.11/8.12.11/Submit) id p2816G7n003542;
	Mon, 7 Mar 2011 17:06:16 -0800
Date: Mon, 7 Mar 2011 17:06:16 -0800
Message-Id: <201103080106.p2816G7n003542@pbug.corp.yahoo.com>
From: Greg Roelofs <roelofs@yahoo-inc.com>
Reply-To: Greg Roelofs <roelofs@yahoo-inc.com>
To: mapreduce-dev@hadoop.apache.org
Subject: Re: Hadoop-Mapreduce-trunk-Commit - Build # 621 - Still Failing
Cc: cos@apache.org, gkesavan@yahoo-inc.com, todd@cloudera.com
In-Reply-To: <AANLkTi=yoi5pua_VDmWUfem3CtArT6rTMgZ7G6oUkbFh@mail.gmail.com>
X-Virus-Checked: Checked by ClamAV on apache.org

[general@ moved to Bcc; this seems like a pure dev issue now.]

Todd Lipcon wrote:

> It looks like something automated must be rming the settings.xml file.
> Maybe one of our builds has an "rm -Rf ~/.m2" in it or something?

(probably "rm -rf")

> Or put it in the spot in ~/.m2 but with permissions set to make it
> harder to remove?

I don't believe there's any combination of owner/perms that would protect
it against "rm -rf" while still allowing the creation of the repository/
dir in there.  Possibly the sticky bit or something similar (I think one
of them prevents deletion of another user's files even if the directory
has appropriate write-permissions), but I'm not sure even that would work.

If the hypothetical "rm -rf" can be found, perhaps simply changing it to
nuke only ~/.m2/repository would suffice?  Or does settings.xml live below
that?

Restoring it as one of the first steps of a Hudson build is probably the
best approach, though (especially if it's restored from svn or somewhere
with change-history).

Greg

From general-return-3160-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 14:46:26 2011
Return-Path: <general-return-3160-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17868 invoked from network); 8 Mar 2011 14:46:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 14:46:25 -0000
Received: (qmail 2597 invoked by uid 500); 8 Mar 2011 14:46:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1823 invoked by uid 500); 8 Mar 2011 14:46:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1570 invoked by uid 99); 8 Mar 2011 14:46:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 14:46:22 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of josh@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 14:46:17 +0000
Received: by bwz8 with SMTP id 8so6463861bwz.35
        for <multiple recipients>; Tue, 08 Mar 2011 06:45:57 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.80.29 with SMTP id r29mr4380299bkk.195.1299594252500; Tue,
 08 Mar 2011 06:24:12 -0800 (PST)
Received: by 10.204.53.202 with HTTP; Tue, 8 Mar 2011 06:24:12 -0800 (PST)
In-Reply-To: <AANLkTimA0FGsXqw0bKqZfz+yv-swG_MV798YGjky5dPe@mail.gmail.com>
References: <AANLkTimA0FGsXqw0bKqZfz+yv-swG_MV798YGjky5dPe@mail.gmail.com>
Date: Tue, 8 Mar 2011 09:24:12 -0500
Message-ID: <AANLkTin7Ywwm8wc2NXxgsm=svyBbgzNSAV4v13-mUZht@mail.gmail.com>
Subject: Re: Digital Signal Processing Library + Hadoop
From: Josh Patterson <josh@cloudera.com>
To: common-user@hadoop.apache.org
Cc: Roger Smith <rogersmith1711@gmail.com>, general@hadoop.apache.org, 
	dev@mahout.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Roger,
A basic time series construct is the "sliding" window in conjunction
with sorted time/value data; A sample implementation is at my github:

https://github.com/jpatanooga/Caduceus/tree/master/src/tv/floe/caduceus/hadoop/movingaverage

There are two jobs in there, one that uses the shuffle and one that
does not --- to illustrate the difference. I have a blog draft coming
that accompanies this code, I'll follow up and send you a copy draft
of it.

>From that code you should be able to build out a more complex time
series / DSP process (using it as base code), something along the
lines of a 1NN classifier:

https://openpdc.svn.codeplex.com/svn/Hadoop/Current%20Version/
https://openpdc.svn.codeplex.com/svn/Hadoop/Current%20Version/docs/openPDC%20Datamining%20Tools%20Guide.pdf
https://openpdc.svn.codeplex.com/svn/Hadoop/Current%20Version/src/TVA/Hadoop/MapReduce/Datamining/SAX/SlidingTSClassifier_kNN.java

I'm in the process of updating that older openPDC code to be more
modern and modular for general data sources.

Josh




On Sat, Mar 5, 2011 at 12:05 AM, Roger Smith <rogersmith1711@gmail.com> wrote:
> All -
> I wonder if any of you have integrated a DSP library with Hadoop.
> We are considering using Hadoop to processing time series data, but don't
> want to write standard DSP functions.
>
> Roger.
>



-- 
Twitter: @jpatanooga
Solution Architect @ Cloudera
hadoop: http://www.cloudera.com
blog: http://jpatterson.floe.tv

From general-return-3161-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 09 14:11:17 2011
Return-Path: <general-return-3161-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93075 invoked from network); 9 Mar 2011 14:11:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Mar 2011 14:11:16 -0000
Received: (qmail 92312 invoked by uid 500); 9 Mar 2011 14:11:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92249 invoked by uid 500); 9 Mar 2011 14:11:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 92241 invoked by uid 99); 9 Mar 2011 14:11:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 14:11:14 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 14:11:07 +0000
Received: by qwd6 with SMTP id 6so458562qwd.35
        for <general@hadoop.apache.org>; Wed, 09 Mar 2011 06:10:45 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=3tOYKe5RrFXVhz8XkRzS0JYKieBIeW8GPDiEpEPafbI=;
        b=Im/So0lZwhtkrWnn8e3tgiW3u8xDIWxHlAMbyYHxgYExjuT2FGu1/GRsCJXcO+ci1Y
         FwzP9Agc+fSkuLSdjVPvAzYIYf9JETqYvFw2dauR9ydL38vL3WhZvCYv5kWoHwTkpoHg
         U0Ky7DxvavVQkTchLxrxD4gnCcT7oHCo1VWVY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=eQjhE6wUB0var3oS0hWMr/liRSpVPSE3tfqXfu3VAKgminVWnP9fvSJpVl2PKd+dI6
         6HZRLwDbQzPJY5PwB+1s4POIY0mxyvxoLh2aDAxuKopRBCjFxWOtBtVaj+zihOkAf6ba
         3Ge56PEwwEC4sHvJbKTCtz501OEuaUScAZef8=
Received: by 10.224.63.199 with SMTP id c7mr5672132qai.146.1299679845510;
        Wed, 09 Mar 2011 06:10:45 -0800 (PST)
Received: from [10.167.202.101] (h-64-236-128-41.nat.aol.com [64.236.128.41])
        by mx.google.com with ESMTPS id e29sm1421277qck.39.2011.03.09.06.10.44
        (version=TLSv1/SSLv3 cipher=OTHER);
        Wed, 09 Mar 2011 06:10:44 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Apache Sonar - http://analysis.apache.org/ does anyone want hadoop in there?
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <AANLkTimsma0e9cGAN0o_zz4dBDNRnkDXEgKcVx00zbEV@mail.gmail.com>
Date: Wed, 9 Mar 2011 09:10:43 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <597116E4-1991-43A6-B35B-BAA9F577897E@Holsman.NET>
References: <FF258184-01BE-419F-B5A0-69051472BBDB@Holsman.net> <AANLkTimsma0e9cGAN0o_zz4dBDNRnkDXEgKcVx00zbEV@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Cos.
apparently they can't build from ant in the current version.
I'll need to have a look at it.. they sent me some info.

On Mar 3, 2011, at 12:34 PM, Konstantin Boudnik wrote:

> Ian,
>=20
> Sonar was around for a while and Apache has this setup since late
> 2009, I believe.  I had asked  INFRA folks on a few occasions about
> adding Hadoop repos to their installation, but never heard anything
> back.
>=20
> I'd be awesome if you can talk to the right people so it'd happen =
eventually.
> --
>   Thanks in advance,
> Konstantin (Cos) Boudnik
>=20
>=20
> On Thu, Mar 3, 2011 at 05:59, Ian Holsman <ian@holsman.net> wrote:
>> I just discovered this Apache site,
>> Apache Sonar performs source code analysis on the java code, and it =
looks pretty, and I'm sure some people would find it useful.
>>=20
>> If people are interested I can start putting finding out how to put =
the hadoop code in there, so we can get stats on our own code.
>>=20
>> Regards
>> Ian
>> --
>> Ian Holsman
>> Ian@Holsman.net
>> PH: +1-703 879-3128 AOLIM: ianholsman Skype:iholsman
>>=20
>> To know recursion, you must first know recursion.
>>=20
>>=20
>>=20
>>=20


From general-return-3162-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 09 16:07:12 2011
Return-Path: <general-return-3162-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88702 invoked from network); 9 Mar 2011 16:07:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Mar 2011 16:07:12 -0000
Received: (qmail 13185 invoked by uid 500); 9 Mar 2011 16:07:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13129 invoked by uid 500); 9 Mar 2011 16:07:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13121 invoked by uid 99); 9 Mar 2011 16:07:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 16:07:10 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.213.48] (HELO mail-yw0-f48.google.com) (209.85.213.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 16:07:05 +0000
Received: by ywo32 with SMTP id 32so413634ywo.35
        for <general@hadoop.apache.org>; Wed, 09 Mar 2011 08:06:43 -0800 (PST)
Received: by 10.236.75.133 with SMTP id z5mr63846yhd.310.1299686803298; Wed,
 09 Mar 2011 08:06:43 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.236.110.1 with HTTP; Wed, 9 Mar 2011 08:06:23 -0800 (PST)
In-Reply-To: <597116E4-1991-43A6-B35B-BAA9F577897E@Holsman.NET>
References: <FF258184-01BE-419F-B5A0-69051472BBDB@Holsman.net>
 <AANLkTimsma0e9cGAN0o_zz4dBDNRnkDXEgKcVx00zbEV@mail.gmail.com> <597116E4-1991-43A6-B35B-BAA9F577897E@Holsman.NET>
From: Konstantin Boudnik <cos@apache.org>
Date: Wed, 9 Mar 2011 08:06:23 -0800
X-Google-Sender-Auth: sha7Mp0jAud6zdxQ7zVFmSwcd9E
Message-ID: <AANLkTi=cPCdv1ipMSN41Cpabhg_jbSZvDRh8_BzyUFR_@mail.gmail.com>
Subject: Re: Apache Sonar - http://analysis.apache.org/ does anyone want
 hadoop in there?
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Ian.

yes, Sonar needs to have a pom'ed project. In case of an ant build
such a pom can be created to 'wrap' around build.xml and is usually
quite simple to manufacture. I have done it in the past for some HDFS
experiments without much of a hassle.

Here's the example:
  http://docs.codehaus.org/display/SONAR/Analyse+with+Maven
in "Analyze projects which are not built with Maven" section.

There's one more caveat though: handling of unit tests reports. It can
be done with some new features Sonar has. See here:
  http://www.sonarsource.org/tag/ant/

I can quickly provide a working fixture to add say HDFS into Sonar as
a starter. Pl. let me know where it needs to be put (perhaps a JIRA
with a patch would do to?).

Thanks for looking into this,
  Cos

On Wed, Mar 9, 2011 at 06:10, Ian Holsman <hadoop@holsman.net> wrote:
> Hi Cos.
> apparently they can't build from ant in the current version.
> I'll need to have a look at it.. they sent me some info.
>
> On Mar 3, 2011, at 12:34 PM, Konstantin Boudnik wrote:
>
>> Ian,
>>
>> Sonar was around for a while and Apache has this setup since late
>> 2009, I believe. =A0I had asked =A0INFRA folks on a few occasions about
>> adding Hadoop repos to their installation, but never heard anything
>> back.
>>
>> I'd be awesome if you can talk to the right people so it'd happen eventu=
ally.
>> --
>> =A0 Thanks in advance,
>> Konstantin (Cos) Boudnik
>>
>>
>> On Thu, Mar 3, 2011 at 05:59, Ian Holsman <ian@holsman.net> wrote:
>>> I just discovered this Apache site,
>>> Apache Sonar performs source code analysis on the java code, and it loo=
ks pretty, and I'm sure some people would find it useful.
>>>
>>> If people are interested I can start putting finding out how to put the=
 hadoop code in there, so we can get stats on our own code.
>>>
>>> Regards
>>> Ian
>>> --
>>> Ian Holsman
>>> Ian@Holsman.net
>>> PH: +1-703 879-3128 AOLIM: ianholsman Skype:iholsman
>>>
>>> To know recursion, you must first know recursion.
>>>
>>>
>>>
>>>
>
>

From general-return-3163-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 09 16:31:00 2011
Return-Path: <general-return-3163-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46577 invoked from network); 9 Mar 2011 16:30:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Mar 2011 16:30:59 -0000
Received: (qmail 58922 invoked by uid 500); 9 Mar 2011 16:30:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58855 invoked by uid 500); 9 Mar 2011 16:30:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58847 invoked by uid 99); 9 Mar 2011 16:30:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 16:30:57 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.213.48] (HELO mail-yw0-f48.google.com) (209.85.213.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 16:30:53 +0000
Received: by ywo32 with SMTP id 32so428469ywo.35
        for <general@hadoop.apache.org>; Wed, 09 Mar 2011 08:30:32 -0800 (PST)
Received: by 10.236.184.4 with SMTP id r4mr91956yhm.160.1299688232140; Wed, 09
 Mar 2011 08:30:32 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.236.110.1 with HTTP; Wed, 9 Mar 2011 08:30:12 -0800 (PST)
In-Reply-To: <AANLkTi=cPCdv1ipMSN41Cpabhg_jbSZvDRh8_BzyUFR_@mail.gmail.com>
References: <FF258184-01BE-419F-B5A0-69051472BBDB@Holsman.net>
 <AANLkTimsma0e9cGAN0o_zz4dBDNRnkDXEgKcVx00zbEV@mail.gmail.com>
 <597116E4-1991-43A6-B35B-BAA9F577897E@Holsman.NET> <AANLkTi=cPCdv1ipMSN41Cpabhg_jbSZvDRh8_BzyUFR_@mail.gmail.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Wed, 9 Mar 2011 08:30:12 -0800
X-Google-Sender-Auth: -ZVAT4Y_6g-6I46n0nrp7kL0JvQ
Message-ID: <AANLkTika3WVVhJKXNQ2O4Kfd16jamiT5k2RJSy=Yi8ao@mail.gmail.com>
Subject: Re: Apache Sonar - http://analysis.apache.org/ does anyone want
 hadoop in there?
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I have looked around Apache's Sonar server and it seems to be
sufficient to just have a correct pom.xml in place. I have opened
  https://issues.apache.org/jira/browse/HDFS-1741
Will attend to it shortly.
--
=A0 Take care,
Konstantin (Cos) Boudnik

On Wed, Mar 9, 2011 at 08:06, Konstantin Boudnik <cos@apache.org> wrote:
> Hi Ian.
>
> yes, Sonar needs to have a pom'ed project. In case of an ant build
> such a pom can be created to 'wrap' around build.xml and is usually
> quite simple to manufacture. I have done it in the past for some HDFS
> experiments without much of a hassle.
>
> Here's the example:
> =A0http://docs.codehaus.org/display/SONAR/Analyse+with+Maven
> in "Analyze projects which are not built with Maven" section.
>
> There's one more caveat though: handling of unit tests reports. It can
> be done with some new features Sonar has. See here:
> =A0http://www.sonarsource.org/tag/ant/
>
> I can quickly provide a working fixture to add say HDFS into Sonar as
> a starter. Pl. let me know where it needs to be put (perhaps a JIRA
> with a patch would do to?).
>
> Thanks for looking into this,
> =A0Cos
>
> On Wed, Mar 9, 2011 at 06:10, Ian Holsman <hadoop@holsman.net> wrote:
>> Hi Cos.
>> apparently they can't build from ant in the current version.
>> I'll need to have a look at it.. they sent me some info.
>>
>> On Mar 3, 2011, at 12:34 PM, Konstantin Boudnik wrote:
>>
>>> Ian,
>>>
>>> Sonar was around for a while and Apache has this setup since late
>>> 2009, I believe. =A0I had asked =A0INFRA folks on a few occasions about
>>> adding Hadoop repos to their installation, but never heard anything
>>> back.
>>>
>>> I'd be awesome if you can talk to the right people so it'd happen event=
ually.
>>> --
>>> =A0 Thanks in advance,
>>> Konstantin (Cos) Boudnik
>>>
>>>
>>> On Thu, Mar 3, 2011 at 05:59, Ian Holsman <ian@holsman.net> wrote:
>>>> I just discovered this Apache site,
>>>> Apache Sonar performs source code analysis on the java code, and it lo=
oks pretty, and I'm sure some people would find it useful.
>>>>
>>>> If people are interested I can start putting finding out how to put th=
e hadoop code in there, so we can get stats on our own code.
>>>>
>>>> Regards
>>>> Ian
>>>> --
>>>> Ian Holsman
>>>> Ian@Holsman.net
>>>> PH: +1-703 879-3128 AOLIM: ianholsman Skype:iholsman
>>>>
>>>> To know recursion, you must first know recursion.
>>>>
>>>>
>>>>
>>>>
>>
>>
>

From general-return-3164-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 09 17:07:04 2011
Return-Path: <general-return-3164-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87714 invoked from network); 9 Mar 2011 17:07:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Mar 2011 17:07:03 -0000
Received: (qmail 73001 invoked by uid 500); 9 Mar 2011 17:07:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72953 invoked by uid 500); 9 Mar 2011 17:07:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72945 invoked by uid 99); 9 Mar 2011 17:07:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 17:07:01 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mlortiz@uci.cu designates 200.55.140.180 as permitted sender)
Received: from [200.55.140.180] (HELO mx3.uci.cu) (200.55.140.180)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 09 Mar 2011 17:06:56 +0000
Received: (qmail 26909 invoked by uid 507); 9 Mar 2011 17:06:31 -0000
Received: from 10.0.0.183 by ns3.uci.cu (envelope-from <mlortiz@uci.cu>, uid 501) with qmail-scanner-2.01st 
 (avp: 5.0.2.0. spamassassin: 3.0.6. perlscan: 2.01st.  
 Clear:RC:1(10.0.0.183):. 
 Processed in 0.651023 secs); 09 Mar 2011 17:06:31 -0000
Received: from unknown (HELO ucimail2.uci.cu) (10.0.0.183)
  by 0 with SMTP; 9 Mar 2011 17:06:30 -0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by ucimail2.uci.cu (Postfix) with ESMTP id 49B451808C3;
	Wed,  9 Mar 2011 12:07:59 -0500 (CST)
X-Virus-Scanned: amavisd-new at uci.cu
Received: from ucimail2.uci.cu ([127.0.0.1])
	by localhost (ucimail2.uci.cu [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id PgZX98JslhPF; Wed,  9 Mar 2011 12:07:59 -0500 (CST)
Received: from [10.36.18.44] (skynet2010.uci.cu [10.36.18.44])
	by ucimail2.uci.cu (Postfix) with ESMTP id 09F291808A5;
	Wed,  9 Mar 2011 12:07:59 -0500 (CST)
Message-ID: <4D77B3DF.1040800@uci.cu>
Date: Wed, 09 Mar 2011 12:07:43 -0500
From: Marcos Ortiz <mlortiz@uci.cu>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; es-ES; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: general@hadoop.apache.org, Steve Loughran <stevel@apache.org>
Subject: Re: Hadoop Development and the new Oracle's Plans
References: <2052738279.11140861299085532253.JavaMail.root@ucimail4.uci.cu> <4D6F70B5.3020302@apache.org>
In-Reply-To: <4D6F70B5.3020302@apache.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
X-Virus-Checked: Checked by ClamAV on apache.org


>
> The other issue is that a lot of people out there use Mac laptops for 
> their development, and until java7 ships for the mac, nobody who has a 
> mac will be able to develop any Java 7 code. That's another reason for 
> Hadoop staying on Java6
>
> What could be interesting would be for Hadoop to have more scala 
> integration. I think there are some good arguments for us moving some 
> of the high level code to that language, within the JVM, rather than 
> follow the oracle roadmap.

Regards, Steve and all the list.
I'm very interested on the Scala integration for the high level code of 
Hadoop. Is there any discussion about it?
Any JIRA issue?
Ideas?

I would like to work on this, so I will keep on mind this topic.
Thanks

-- 
Marcos Luís Ortíz Valmaseda
  Software Engineer
  Universidad de las Ciencias Informáticas
  Linux User # 418229

http://uncubanitolinuxero.blogspot.com
http://www.linkedin.com/in/marcosluis2186


From general-return-3165-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 00:10:25 2011
Return-Path: <general-return-3165-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50581 invoked from network); 10 Mar 2011 00:10:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 00:10:25 -0000
Received: (qmail 52688 invoked by uid 500); 10 Mar 2011 00:10:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52647 invoked by uid 500); 10 Mar 2011 00:10:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52639 invoked by uid 99); 10 Mar 2011 00:10:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 00:10:23 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of michael_segel@hotmail.com designates 65.55.34.11 as permitted sender)
Received: from [65.55.34.11] (HELO col0-omc1-s1.col0.hotmail.com) (65.55.34.11)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 00:10:15 +0000
Received: from COL117-W56 ([65.55.34.7]) by col0-omc1-s1.col0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Wed, 9 Mar 2011 16:09:54 -0800
Message-ID: <COL117-W56798CFE5F230A06C8FDF88FC80@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_f7a1d1a5-f63b-4058-b525-c9d2a0039480_"
X-Originating-IP: [65.167.11.254]
From: Michael Segel <michael_segel@hotmail.com>
To: <general@hadoop.apache.org>
Subject: Running m/r jobs from within Eclipse and NetBeans on Windows?
Date: Wed, 9 Mar 2011 18:09:55 -0600
Importance: Normal
MIME-Version: 1.0
X-OriginalArrivalTime: 10 Mar 2011 00:09:54.0536 (UTC) FILETIME=[7BAABE80:01CBDEB7]

--_f7a1d1a5-f63b-4058-b525-c9d2a0039480_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


I just got this from a developer using Eclipse=20

"

11/03/09 12:20:14 INFO
zookeeper.ClientCnxn: Socket connection established to 10.8.120.61:2181=2C =
initiating session

11/03/09 12:20:14 INFO
zookeeper.ClientCnxn: Session establishment complete on server 10.8.120.61:=
2181=2C sessionid =3D 0x22df2ab81010592=2C negotiated timeout
=3D 40000

11/03/09 12:20:16 INFO jvm.JvmMetrics:
Initializing JVM Metrics with processName=3DJobTracker=2C sessionId=3D

Exception in thread "main" java.io.IOException: Failed to set
permissions of path:
file:/tmp/hadoop-boo/mapred/staging/boo1251500722/.staging to 0700

"

I had the same issue using NetBeans=2C but I couldn't find the solution and=
 I gave up... (Too many other things to worry about and work on...)

The directory exists=2C the file exists and when looking at it through cygw=
in=2C I can see that the file permissions are 0700.

This appears to be the last step blocking us from being able to launch jobs=
 directly out of Eclipse or NetBeans.

So I have to ask... what am I missing and if anyone else has seen this or c=
ame up with a solution?

Thx

-Mike

 		 	   		  =

--_f7a1d1a5-f63b-4058-b525-c9d2a0039480_--

From general-return-3166-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 21:59:43 2011
Return-Path: <general-return-3166-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1094 invoked from network); 11 Mar 2011 21:59:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 21:59:43 -0000
Received: (qmail 70788 invoked by uid 500); 11 Mar 2011 21:59:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70082 invoked by uid 500); 11 Mar 2011 21:59:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69701 invoked by uid 99); 11 Mar 2011 21:59:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 21:59:38 +0000
X-ASF-Spam-Status: No, hits=3.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of akimball83@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 21:59:32 +0000
Received: by bwz8 with SMTP id 8so4335648bwz.35
        for <multiple recipients>; Fri, 11 Mar 2011 13:59:12 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:from:date:message-id:subject:to
         :content-type;
        bh=zbXKOntp8w07lJ8PiF3VAoflr7EKUbqOAJ5fp1Jdo24=;
        b=gexlOjbxT6kgmjtLqvsnL5zIaJR14Rb7qMVXxsT8RuYDL35pQeOU2MCetta9NmnHMT
         zTmpfIkJzxFO1HzDZI3m6Xfk/faQtbmDX4gswjotlCVyF6BAaZxoQ++i+fYoF1j6lpt1
         7dUh5lbLASawrNOrF+IZrE6HumrIH0v8044UY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type;
        b=FlfKfBaYmGJrU7YfT8iM5ftI3Nb1ewqxprBHCtK82NRk6p9GVv7MwzX5NUJKTc6eDH
         yGF6wV6gApdIbWGhu83EzeoMOx/vdJIxB+IvuFgOXV8wB/FcMHPvsUsb78Df0qRtunTW
         SfH+NLE1ppbr9cg6s+TJf8JZmutGcBwPrZYwU=
Received: by 10.204.16.140 with SMTP id o12mr212506bka.125.1299880752075; Fri,
 11 Mar 2011 13:59:12 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.14.1 with HTTP; Fri, 11 Mar 2011 13:58:52 -0800 (PST)
From: Aaron Kimball <akimball83@gmail.com>
Date: Fri, 11 Mar 2011 13:58:52 -0800
Message-ID: <AANLkTik2W2My=1BefoeOdrG1-zg2+cGbn_pE2DtsMHdi@mail.gmail.com>
Subject: SF Hadoop Meetup - March review and April announcement (April 13)
To: general@hadoop.apache.org, core-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00032555a16e7a56da049e3c12b3
X-Virus-Checked: Checked by ClamAV on apache.org

--00032555a16e7a56da049e3c12b3
Content-Type: text/plain; charset=ISO-8859-1

Hadoop fans,

This week we had a great SF Hadoop meetup hosted by Yelp. As usual, we used
an unconference format to construct agenda topics, and broke out into
separate discussion groups. About 75 people attended, and we had a dozen
great discussions; some were business-focused, others very technical, and
some highly theoretical:

* Learning thrift
* Cluster capacity planning
* Selling Hadoop to your organization
* Alternate JVM languages and Hadoop
* Schema / metadata management
* Integration and Migration: Data warehousing, BI, and Hadoop
* Abstract game theory problems and massive data sets
* Hadoop monitoring / management best practices
* "Oinkers anonymous"; for Pig users
* Yahoo S4 and Cloudera Flume
* Configuration tuning and performance
* Production-level workflow tools

Notes for some topics will be available at our meetup site at
http://meetup.com/hadoopsf

If you haven't yet come to one of our meetups, please consider trying it out
next month! We need your voice to help contribute to the discussions. All
members of the Hadoop community, novice or experienced, are welcome.

Next month's meetup will be held Wednesday, April 13, from 6pm to 8pm. The
meetup will be hosted by our friends at Twitter. Their office is at 795
Folsom St. (4th and Folsom)

We will use the discussion-based "unconference" format. At the beginning of
the meetup we will collaboratively construct an agenda consisting of several
discussion breakout groups. All participants may propose a topic and
volunteer to facilitate a discussion. All Hadoop-related topics are
encouraged, and all members of the Hadoop community are welcome.

Please RSVP at http://bit.ly/gkIvqH

Event schedule:
* 6pm - Welcome
* 6:30pm - Introductions; start creating agenda
* Breakout sessions begin as soon as we're ready
* 8pm - Conclusion

Regards,
- Aaron Kimball

--00032555a16e7a56da049e3c12b3--

From general-return-3167-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 12 03:02:39 2011
Return-Path: <general-return-3167-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 90581 invoked from network); 12 Mar 2011 03:02:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Mar 2011 03:02:39 -0000
Received: (qmail 43871 invoked by uid 500); 12 Mar 2011 03:02:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 43788 invoked by uid 500); 12 Mar 2011 03:02:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 43773 invoked by uid 99); 12 Mar 2011 03:02:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 03:02:33 +0000
X-ASF-Spam-Status: No, hits=-0.1 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of Ken_Walker@ca.ibm.com designates 32.97.182.142 as permitted sender)
Received: from [32.97.182.142] (HELO e2.ny.us.ibm.com) (32.97.182.142)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 03:02:24 +0000
Received: from d01dlp02.pok.ibm.com (d01dlp02.pok.ibm.com [9.56.224.85])
	by e2.ny.us.ibm.com (8.14.4/8.13.1) with ESMTP id p2C2hYGk000353
	for <general@hadoop.apache.org>; Fri, 11 Mar 2011 21:43:34 -0500
Received: from d01relay05.pok.ibm.com (d01relay05.pok.ibm.com [9.56.227.237])
	by d01dlp02.pok.ibm.com (Postfix) with ESMTP id B273C6E8036
	for <general@hadoop.apache.org>; Fri, 11 Mar 2011 22:02:02 -0500 (EST)
Received: from d01av04.pok.ibm.com (d01av04.pok.ibm.com [9.56.224.64])
	by d01relay05.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id p2C3229F226960
	for <general@hadoop.apache.org>; Fri, 11 Mar 2011 22:02:02 -0500
Received: from d01av04.pok.ibm.com (loopback [127.0.0.1])
	by d01av04.pok.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id p2C3229R018734
	for <general@hadoop.apache.org>; Fri, 11 Mar 2011 22:02:02 -0500
Received: from d25ml04.torolab.ibm.com (d25ml04.torolab.ibm.com [9.26.6.105])
	by d01av04.pok.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id p2C3228R018722
	for <general@hadoop.apache.org>; Fri, 11 Mar 2011 22:02:02 -0500
Subject: AUTO: Ken Walker is on vacation (returning 2011/03/22)
Auto-Submitted: auto-generated
From: Ken Walker <Ken_Walker@ca.ibm.com>
To: general@hadoop.apache.org
Message-ID: <OF5BB068D6.0941420A-ON85257851.0010A9EB-85257851.0010A9EB@ca.ibm.com>
Date: Fri, 11 Mar 2011 22:02:00 -0500
X-MIMETrack: Serialize by Router on D25ML04/25/M/IBM(Release 8.5.1FP4|July 25, 2010) at
 03/11/2011 22:02:01
MIME-Version: 1.0
Content-type: multipart/alternative; 
	Boundary="0__=0ABBF2C2DF832F7B8f9e8a93df938690918c0ABBF2C2DF832F7B"
Content-Disposition: inline
X-Content-Scanned: Fidelis XPS MAILER

--0__=0ABBF2C2DF832F7B8f9e8a93df938690918c0ABBF2C2DF832F7B
Content-type: text/plain; charset=US-ASCII
Content-transfer-encoding: quoted-printable



I am out of the office until 2011/03/22.

I am travelling on vacation and do not have access to email.  I will ch=
eck
and respond to emails upon my return.


Note: This is an automated response to your message  "SF Hadoop Meetup =
-
March review and April announcement (April 13)" sent on 3/11/11 16:58:5=
2.

This is the only notification you will receive while this person is awa=
y.=

--0__=0ABBF2C2DF832F7B8f9e8a93df938690918c0ABBF2C2DF832F7B--


From general-return-3168-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 12 19:00:43 2011
Return-Path: <general-return-3168-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14022 invoked from network); 12 Mar 2011 19:00:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Mar 2011 19:00:42 -0000
Received: (qmail 61994 invoked by uid 500); 12 Mar 2011 19:00:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61918 invoked by uid 500); 12 Mar 2011 19:00:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61910 invoked by uid 99); 12 Mar 2011 19:00:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 19:00:40 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mlortiz@uci.cu designates 200.55.140.180 as permitted sender)
Received: from [200.55.140.180] (HELO mx3.uci.cu) (200.55.140.180)
    by apache.org (qpsmtpd/0.29) with SMTP; Sat, 12 Mar 2011 19:00:33 +0000
Received: (qmail 25111 invoked by uid 507); 12 Mar 2011 19:00:06 -0000
Received: from 10.0.0.185 by ns3.uci.cu (envelope-from <mlortiz@uci.cu>, uid 501) with qmail-scanner-2.01st 
 (avp: 5.0.2.0. spamassassin: 3.0.6. perlscan: 2.01st.  
 Clear:RC:1(10.0.0.185):. 
 Processed in 0.629389 secs); 12 Mar 2011 19:00:06 -0000
Received: from unknown (HELO ucimail4.uci.cu) (10.0.0.185)
  by 0 with SMTP; 12 Mar 2011 19:00:05 -0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by ucimail4.uci.cu (Postfix) with ESMTP id 20924132450F;
	Sat, 12 Mar 2011 14:00:05 -0500 (CST)
X-Virus-Scanned: amavisd-new at uci.cu
Received: from ucimail4.uci.cu ([127.0.0.1])
	by localhost (ucimail4.uci.cu [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id f5pmGF5I8KuZ; Sat, 12 Mar 2011 14:00:00 -0500 (CST)
Received: from ucimail4.uci.cu (ucimail4.uci.cu [10.0.0.185])
	by ucimail4.uci.cu (Postfix) with ESMTP id 4D8151324371;
	Sat, 12 Mar 2011 14:00:00 -0500 (CST)
Date: Sat, 12 Mar 2011 14:00:00 -0500 (CST)
From:  Marcos Ortiz Valmaseda <mlortiz@uci.cu>
To: general@hadoop.apache.org
Cc: akimball83@gmail.com
Message-ID: <514379149.69241299956400283.JavaMail.root@ucimail4.uci.cu>
In-Reply-To: <1346003106.68851299956284464.JavaMail.root@ucimail4.uci.cu>
Subject: Re: SF Hadoop Meetup - March review and April announcement (April
 13)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Originating-IP: [10.0.0.186]
X-Mailer: Zimbra 5.0.16_GA_2921.RHEL5_64 (ZimbraWebClient - FF3.0 (Win)/5.0.16_GA_2921.RHEL5_64)
X-Virus-Checked: Checked by ClamAV on apache.org

Regards, Aaron for this excellent meetup.
There are many topics very interesting on this meetup.
Are available all talks slides?

Thanks a lot, =20
----- Mensaje original -----
De: "Aaron Kimball" <akimball83@gmail.com>
Para: general@hadoop.apache.org, core-user@hadoop.apache.org
Enviados: Viernes, 11 de Marzo 2011 16:58:52 (GMT-0500) Auto-Detected
Asunto: SF Hadoop Meetup - March review and April announcement (April 13)

Hadoop fans,

This week we had a great SF Hadoop meetup hosted by Yelp. As usual, we used
an unconference format to construct agenda topics, and broke out into
separate discussion groups. About 75 people attended, and we had a dozen
great discussions; some were business-focused, others very technical, and
some highly theoretical:

* Learning thrift
* Cluster capacity planning
* Selling Hadoop to your organization
* Alternate JVM languages and Hadoop
* Schema / metadata management
* Integration and Migration: Data warehousing, BI, and Hadoop
* Abstract game theory problems and massive data sets
* Hadoop monitoring / management best practices
* "Oinkers anonymous"; for Pig users
* Yahoo S4 and Cloudera Flume
* Configuration tuning and performance
* Production-level workflow tools

Notes for some topics will be available at our meetup site at
http://meetup.com/hadoopsf

If you haven't yet come to one of our meetups, please consider trying it ou=
t
next month! We need your voice to help contribute to the discussions. All
members of the Hadoop community, novice or experienced, are welcome.

Next month's meetup will be held Wednesday, April 13, from 6pm to 8pm. The
meetup will be hosted by our friends at Twitter. Their office is at 795
Folsom St. (4th and Folsom)

We will use the discussion-based "unconference" format. At the beginning of
the meetup we will collaboratively construct an agenda consisting of severa=
l
discussion breakout groups. All participants may propose a topic and
volunteer to facilitate a discussion. All Hadoop-related topics are
encouraged, and all members of the Hadoop community are welcome.

Please RSVP at http://bit.ly/gkIvqH

Event schedule:
* 6pm - Welcome
* 6:30pm - Introductions; start creating agenda
* Breakout sessions begin as soon as we're ready
* 8pm - Conclusion

Regards,
- Aaron Kimball

--=20
Marcos Lu=C3=ADs Ort=C3=ADz Valmaseda
 Software Engineer=20
 Universidad de las Ciencias Inform=C3=A1ticas
 Linux User # 418229

http://uncubanitolinuxero.blogspot.com
http://www.linkedin.com/in/marcosluis2186


From general-return-3169-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 12 21:37:45 2011
Return-Path: <general-return-3169-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 26729 invoked from network); 12 Mar 2011 21:37:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Mar 2011 21:37:44 -0000
Received: (qmail 42161 invoked by uid 500); 12 Mar 2011 21:37:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42083 invoked by uid 500); 12 Mar 2011 21:37:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42075 invoked by uid 99); 12 Mar 2011 21:37:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 21:37:43 +0000
X-ASF-Spam-Status: No, hits=3.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of akimball83@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 21:37:37 +0000
Received: by bwz8 with SMTP id 8so5088957bwz.35
        for <general@hadoop.apache.org>; Sat, 12 Mar 2011 13:37:16 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:cc:content-type;
        bh=YeWqg1gE1qvQPWlHb20/c/2GbI8YkGIrggjKFXoEBjc=;
        b=FJLic/0AxNMMPvmrUTKwBCGmWqTKx+QrFD9Yxkhr+7epp455ao4h9jmfizTm/YtAUc
         k2qYomMVg7vGoGCga5e4Sk0sakr1SkZvOYousDmvONX98o6aOZqdesSoJ0ewkbP47a6a
         M2qHjDF6wjrzRZDiRmMaqopYRIzWQjUN91j+0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :cc:content-type;
        b=jKd9UG/umJdSnWY8GAy74JyQKuAZx5zwxlqEJhhvTKOmR4MhvDFsUlV0lJGAzra5Og
         rDniaqKaBUvABOAoYXnat9dK8kRbp1EgM8bnqissxoPer9c2UPvlKZLf3QdeyMVqOJ9S
         V6sETfE2Sm5VFppFpe4oOJGRqrWWGb7tJtM5s=
Received: by 10.204.189.76 with SMTP id dd12mr1294898bkb.206.1299965836113;
 Sat, 12 Mar 2011 13:37:16 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.14.1 with HTTP; Sat, 12 Mar 2011 13:36:56 -0800 (PST)
In-Reply-To: <514379149.69241299956400283.JavaMail.root@ucimail4.uci.cu>
References: <1346003106.68851299956284464.JavaMail.root@ucimail4.uci.cu> <514379149.69241299956400283.JavaMail.root@ucimail4.uci.cu>
From: Aaron Kimball <akimball83@gmail.com>
Date: Sat, 12 Mar 2011 13:36:56 -0800
Message-ID: <AANLkTineLD12mCMy=qoMOdkvnNcQvmvrcLhRH6tWkLGQ@mail.gmail.com>
Subject: Re: SF Hadoop Meetup - March review and April announcement (April 13)
To: Marcos Ortiz Valmaseda <mlortiz@uci.cu>
Cc: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf30334581e1b5f7049e4fe170

--20cf30334581e1b5f7049e4fe170
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Marcos,

There were no slides used at this meetup; the topics listed below were used
to seed round-table discussions between the attendees. Some notes from
various discussions have been transcribed and put on the meetup group's
message board: http://www.meetup.com/hadoopsf/messages/boards/

- Aaron

On Sat, Mar 12, 2011 at 11:00 AM, Marcos Ortiz Valmaseda <mlortiz@uci.cu>wr=
ote:

> Regards, Aaron for this excellent meetup.
> There are many topics very interesting on this meetup.
> Are available all talks slides?
>
> Thanks a lot,
> ----- Mensaje original -----
> De: "Aaron Kimball" <akimball83@gmail.com>
> Para: general@hadoop.apache.org, core-user@hadoop.apache.org
> Enviados: Viernes, 11 de Marzo 2011 16:58:52 (GMT-0500) Auto-Detected
> Asunto: SF Hadoop Meetup - March review and April announcement (April 13)
>
> Hadoop fans,
>
> This week we had a great SF Hadoop meetup hosted by Yelp. As usual, we us=
ed
> an unconference format to construct agenda topics, and broke out into
> separate discussion groups. About 75 people attended, and we had a dozen
> great discussions; some were business-focused, others very technical, and
> some highly theoretical:
>
> * Learning thrift
> * Cluster capacity planning
> * Selling Hadoop to your organization
> * Alternate JVM languages and Hadoop
> * Schema / metadata management
> * Integration and Migration: Data warehousing, BI, and Hadoop
> * Abstract game theory problems and massive data sets
> * Hadoop monitoring / management best practices
> * "Oinkers anonymous"; for Pig users
> * Yahoo S4 and Cloudera Flume
> * Configuration tuning and performance
> * Production-level workflow tools
>
> Notes for some topics will be available at our meetup site at
> http://meetup.com/hadoopsf
>
> If you haven't yet come to one of our meetups, please consider trying it
> out
> next month! We need your voice to help contribute to the discussions. All
> members of the Hadoop community, novice or experienced, are welcome.
>
> Next month's meetup will be held Wednesday, April 13, from 6pm to 8pm. Th=
e
> meetup will be hosted by our friends at Twitter. Their office is at 795
> Folsom St. (4th and Folsom)
>
> We will use the discussion-based "unconference" format. At the beginning =
of
> the meetup we will collaboratively construct an agenda consisting of
> several
> discussion breakout groups. All participants may propose a topic and
> volunteer to facilitate a discussion. All Hadoop-related topics are
> encouraged, and all members of the Hadoop community are welcome.
>
> Please RSVP at http://bit.ly/gkIvqH
>
> Event schedule:
> * 6pm - Welcome
> * 6:30pm - Introductions; start creating agenda
> * Breakout sessions begin as soon as we're ready
> * 8pm - Conclusion
>
> Regards,
> - Aaron Kimball
>
> --
> Marcos Lu=EDs Ort=EDz Valmaseda
>  Software Engineer
>  Universidad de las Ciencias Inform=E1ticas
>  Linux User # 418229
>
> http://uncubanitolinuxero.blogspot.com
> http://www.linkedin.com/in/marcosluis2186
>
>

--20cf30334581e1b5f7049e4fe170--

From general-return-3170-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 14 17:49:26 2011
Return-Path: <general-return-3170-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 10719 invoked from network); 14 Mar 2011 17:49:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Mar 2011 17:49:25 -0000
Received: (qmail 146 invoked by uid 500); 14 Mar 2011 17:49:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99522 invoked by uid 500); 14 Mar 2011 17:49:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 99338 invoked by uid 99); 14 Mar 2011 17:49:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 17:49:22 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of binhara@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 17:49:16 +0000
Received: by wyb40 with SMTP id 40so5886625wyb.35
        for <multiple recipients>; Mon, 14 Mar 2011 10:48:54 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=cvGqQxIBeZZcvkFdsR/gx+ZxVd4ZYjqXF6R829Hibds=;
        b=k/XLrnguAbEvSUR8CyDNgmi7LL+A45J8LPLaIjhGbcRS0sgE6HAY0iES4u2+RlvLfM
         /dlAEBbsVgmQLApTlzE0unGiGXSJGdKfcpDJfznkMnZlDJbVEfQKEAj++EGrNlBMhLr7
         q+qfrbj1wmEayC7mYSwL1JeHydQSMhr+WLc0A=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=AwyP7nPBRhWt8YrKeic+JOZnTi8z62cwAo7G5rA2f3A2tjDhWoP3T/yvsYIFBh/OqX
         x/zhHvyNlIy9TuabLNiDG2IMkXUSdX2Q7aGw8T+GFv3nFB52HQPu8a0xQG8ymh3nkF4L
         o3b99JDRcaQk11Dx+knPKiGi0ki5vDnAMwd1Y=
MIME-Version: 1.0
Received: by 10.216.239.200 with SMTP id c50mr5881861wer.8.1300124931661; Mon,
 14 Mar 2011 10:48:51 -0700 (PDT)
Received: by 10.216.172.201 with HTTP; Mon, 14 Mar 2011 10:48:51 -0700 (PDT)
Date: Mon, 14 Mar 2011 14:48:51 -0300
Message-ID: <AANLkTikFebGD0-cmabwg64fpiGGOS-TSKq9TE+Qs+VCU@mail.gmail.com>
Subject: problem to write on HDFS
From: Alessandro Binhara <binhara@gmail.com>
To: hdfs-user@hadoop.apache.org, general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=e0cb4e385666b71e9b049e74eccc

--e0cb4e385666b71e9b049e74eccc
Content-Type: text/plain; charset=ISO-8859-1

Hello ...

I have a servlet on tomcat.. and it open a hdfs and write simple file with a
content of post information.
Well , in first test we had a 14.000 request per second.
My servet start many trads to write on filesystem.

i got this message on tomcat:

Mar 11, 2011 6:00:20 PM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
SEVERE: Socket accept failed
java.net.SocketException: Too many open files


HDFS is slow to write a file?
How is a better strategy to write on HFDS...

In real aplication we will have a 100.000 request per second to salve in
hdfs.

thanks..

--e0cb4e385666b71e9b049e74eccc--

From general-return-3171-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 14 18:16:40 2011
Return-Path: <general-return-3171-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76002 invoked from network); 14 Mar 2011 18:16:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Mar 2011 18:16:39 -0000
Received: (qmail 63722 invoked by uid 500); 14 Mar 2011 18:16:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63647 invoked by uid 500); 14 Mar 2011 18:16:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63626 invoked by uid 99); 14 Mar 2011 18:16:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 18:16:38 +0000
X-ASF-Spam-Status: No, hits=3.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of akimball83@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 18:16:31 +0000
Received: by bwz8 with SMTP id 8so6721779bwz.35
        for <multiple recipients>; Mon, 14 Mar 2011 11:16:10 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:cc:content-type;
        bh=32/3hY4vJu60OxbjyOyl7xW2thIqfRc5codLZm35A/A=;
        b=MVsOyj2KRyebo0oahP9PKE/x4zVQ5F/y1h9uICXnHiF4REnRoaLv0UVlq0luWkNj84
         eoaI9ajeX2tx6dpMfEpZx1hxjqQao5ajXLbbIsysB3PgPZizTuwS/Ubmu183TTVa+FQ8
         YSjsdEyPE7IGyIfAv5jUlRS1qskQ2eFOsgjJM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :cc:content-type;
        b=Y8+pkrblcdcYESSphoTv6DXO0HPqSp8KHq8GrihK13exZn3vpyQQuNwVWPv0pJe39i
         3OrC9rquz0ueQ6uGE14dY4lyun1VdMAI/gw1eS84V9Clg0MJA6F4lm4yhhTSBzQovNb0
         4fah/ff4nXaiqESEY1AAi7i0xQygJr1WiEeJA=
Received: by 10.204.16.140 with SMTP id o12mr2830531bka.125.1300126547140;
 Mon, 14 Mar 2011 11:15:47 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.14.1 with HTTP; Mon, 14 Mar 2011 11:15:27 -0700 (PDT)
In-Reply-To: <AANLkTikFebGD0-cmabwg64fpiGGOS-TSKq9TE+Qs+VCU@mail.gmail.com>
References: <AANLkTikFebGD0-cmabwg64fpiGGOS-TSKq9TE+Qs+VCU@mail.gmail.com>
From: Aaron Kimball <akimball83@gmail.com>
Date: Mon, 14 Mar 2011 11:15:27 -0700
Message-ID: <AANLkTi=qLRw0NDjsSjWLTgQnUwGOTiqBZXePoiBQrauL@mail.gmail.com>
Subject: Re: problem to write on HDFS
To: general@hadoop.apache.org
Cc: Alessandro Binhara <binhara@gmail.com>, hdfs-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00032555a16e017328049e754d03

--00032555a16e017328049e754d03
Content-Type: text/plain; charset=ISO-8859-1

Alessandro,

I think your requirements are outside the operating envelope for HDFS'
design. HDFS is not particularly well-suited for interactive operation --
it's designed for batch workloads like those performed by MapReduce.
Opening, writing, and closing 100,000 files/second is unlikely to work on
HDFS.

When users head to your site, why do they need to write files in HDFS? What
do those files represent?

If each servlet is just trying to log its operations to HDFS, then you
should use a system like Flume which will aggregate the log records together
into a smaller number of streams and then write those to a more reasonable
number of files which are kept open for a longer amount of time.

If instead each servlet is saving some sort of per-user state that you
expect to retrieve in an online fashion, you should look at HBase or another
distributed (key, value) store (there are several options), which will
enable faster retrieval of this information.

Good luck,
- Aaron


On Mon, Mar 14, 2011 at 10:48 AM, Alessandro Binhara <binhara@gmail.com>wrote:

> Hello ...
>
> I have a servlet on tomcat.. and it open a hdfs and write simple file with
> a
> content of post information.
> Well , in first test we had a 14.000 request per second.
> My servet start many trads to write on filesystem.
>
> i got this message on tomcat:
>
> Mar 11, 2011 6:00:20 PM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
> SEVERE: Socket accept failed
> java.net.SocketException: Too many open files
>
>
> HDFS is slow to write a file?
> How is a better strategy to write on HFDS...
>
> In real aplication we will have a 100.000 request per second to salve in
> hdfs.
>
> thanks..
>

--00032555a16e017328049e754d03--

From general-return-3172-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 14 21:10:25 2011
Return-Path: <general-return-3172-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 64063 invoked from network); 14 Mar 2011 21:10:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Mar 2011 21:10:25 -0000
Received: (qmail 83792 invoked by uid 500); 14 Mar 2011 21:10:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83747 invoked by uid 500); 14 Mar 2011 21:10:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83739 invoked by uid 99); 14 Mar 2011 21:10:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 21:10:24 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [98.139.52.242] (HELO nm26-vm0.bullet.mail.ac4.yahoo.com) (98.139.52.242)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 14 Mar 2011 21:10:15 +0000
Received: from [98.139.52.188] by nm26.bullet.mail.ac4.yahoo.com with NNFMP; 14 Mar 2011 21:09:53 -0000
Received: from [98.139.52.178] by tm1.bullet.mail.ac4.yahoo.com with NNFMP; 14 Mar 2011 21:09:53 -0000
Received: from [127.0.0.1] by omp1061.mail.ac4.yahoo.com with NNFMP; 14 Mar 2011 21:09:53 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 775481.809.bm@omp1061.mail.ac4.yahoo.com
Received: (qmail 78122 invoked by uid 60001); 14 Mar 2011 21:09:53 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1300136993; bh=YmljpKEi+NpgAih0nbuDTi01+FTUWFKYjwhwZLD0STE=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=cqmahPERZtyQsMCGlxxQlW3cymZZBttSH5S1+R9LDIa5m/CwjIOorjLxcQitIFs0WxOsAYVSwNTY4P66s9OiXOkNW+OO7X8tyVJxGs9UmZS9Yk1Ss27AWr/xAnySh6oDdoknp9eqqTCFPQLXNNKQltHc1qiHYrM3HW+FJ4Bp1A8=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=Eq0sygschytWJLTZj0IDkQmL82kIa60JmyUyMVd7DsWQdHUep23KOnhJgQTYTJnzr7rX/iucldOj/QLASK8ZVcmg6+9QIp0Tfu2Ri4xZWgnKHQfjlEIK/me/gTd1rVk62/8hy7bQH6Mep1VwiAHXxE7W8kObCFBjaFinDwmbrBg=;
Message-ID: <675580.77045.qm@web56206.mail.re3.yahoo.com>
X-YMail-OSG: 6cvmwygVM1n_KECoQ8K.slAs8.XhOe68QPu0IEcxCqLy5.V
 U7SdJvYAeiCb50Jiq90ecs6Qf7EcFyHGkE6QqtjgI0hru2jiHt.WGeze4H.y
 lJlLUagsULpYLB6DsPplgcbCrPSGmcT_7NwYjGuSoT7sPqx6AKc9ibvgAWXo
 7S58YP8MbYVL8WjCqA7aGr7zX4SBiQlT.u1kZnlqsC7FVXiocdhkATvG6rcl
 roYVniVzMReS9P3gYQ4PCKFJK5iAB9Zlifnq2dMcVto4K9D79u48HZif23sv
 20.d8B6280uEDlyyLY4Ut4g0iCTdEMdd9Ye.AkqWAW07ZioWhYtSP5uyi6dj
 skL8gepma7Wos9G_ENS6USg0paLCb7lY5.ZENpBLiWTQXjRFDmMo44YxS2QH
 PhdOOQVgFqFbp99nHbvu7G.4ZYZhGeoBG
Received: from [209.131.62.113] by web56206.mail.re3.yahoo.com via HTTP; Mon, 14 Mar 2011 14:09:53 PDT
X-Mailer: YahooMailRC/559 YahooMailWebService/0.8.109.295617
References: <5125873.329641294877388152.JavaMail.jira@thor> <D570B102-7A4F-42E2-BB9A-B3CFC72428CC@mac.com> <AANLkTik14AD5XC+K-5=XMp-v9bdo=vmSG-9o7x=-qbP-@mail.gmail.com> <627F5E4D-DBDC-49BF-A94A-D30BB4FE7F9A@mac.com>
Date: Mon, 14 Mar 2011 14:09:53 -0700 (PDT)
From: "Tsz Wo \(Nicholas\), Sze" <s29752-hadoopgeneral@yahoo.com>
Subject: Re: triggering automated precommit testing
To: general@hadoop.apache.org
In-Reply-To: <627F5E4D-DBDC-49BF-A94A-D30BB4FE7F9A@mac.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-865380475-1300136993=:77045"

--0-865380475-1300136993=:77045
Content-Type: text/plain; charset=us-ascii

Hi Ian,

Could you add me to the  hudson-jobadmin group?  It seems that Hudson sometimes 
does not run automatically.

Nicholas Sze





________________________________
From: Nigel Daley <ndaley@mac.com>
To: general@hadoop.apache.org
Sent: Wed, January 12, 2011 5:12:57 PM
Subject: Re: triggering automated precommit testing

I believe ommitters can gain access following this:
http://wiki.apache.org/general/Hudson#How_do_I_get_an_account
Ian would have to run a command on people.apache.org.

Nige

--0-865380475-1300136993=:77045--

From general-return-3173-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 14 22:11:44 2011
Return-Path: <general-return-3173-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60682 invoked from network); 14 Mar 2011 22:11:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Mar 2011 22:11:44 -0000
Received: (qmail 74046 invoked by uid 500); 14 Mar 2011 22:11:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73988 invoked by uid 500); 14 Mar 2011 22:11:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73980 invoked by uid 99); 14 Mar 2011 22:11:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 22:11:42 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 22:11:35 +0000
Received: by qyk2 with SMTP id 2so2065081qyk.14
        for <general@hadoop.apache.org>; Mon, 14 Mar 2011 15:11:13 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=nb+Uk+EKPIjxt2XQLoPRROAy2YZfVrhbgOUa4//+6qY=;
        b=Uym1q6rcP8KxQMoiAubikhdpwnIBRPSaA3OaVtVPts0/nDnOgP4p8urrHDaJSE4fUd
         ajBGbR4k6W2oX/iJTR4qV4eEDwQX+NJkrJq6nYLgAMBiok7tYYp4EvIxWHMgteuYpp8e
         w59x8+wOw5ndjh0LuhlKGyDXkMlEigW1PRj2Q=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=GpHs4SdNfFM6Rs+PQTFHVBmnB363+j0ozwC5LeJFvImgvrk+04i1JlPHSSfKOfVFSZ
         /N3c2FiCPCwT85wMHL1R0buBFz0F7hQqtEBVCmtYpnsu4k8Mjy5dyCy0i7DZP9MKSI1g
         Q9mqwKfQ/uB1HEf2BAQzSm5sGxWBjCTDVv1Dc=
Received: by 10.229.51.142 with SMTP id d14mr10412571qcg.47.1300140673794;
        Mon, 14 Mar 2011 15:11:13 -0700 (PDT)
Received: from [10.172.0.122] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id s16sm4667708qco.13.2011.03.14.15.11.11
        (version=TLSv1/SSLv3 cipher=OTHER);
        Mon, 14 Mar 2011 15:11:12 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: triggering automated precommit testing
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <675580.77045.qm@web56206.mail.re3.yahoo.com>
Date: Tue, 15 Mar 2011 09:11:07 +1100
Content-Transfer-Encoding: quoted-printable
Message-Id: <2AFA74C9-8D5C-4F30-B776-CD75450113A2@Holsman.NET>
References: <5125873.329641294877388152.JavaMail.jira@thor> <D570B102-7A4F-42E2-BB9A-B3CFC72428CC@mac.com> <AANLkTik14AD5XC+K-5=XMp-v9bdo=vmSG-9o7x=-qbP-@mail.gmail.com> <627F5E4D-DBDC-49BF-A94A-D30BB4FE7F9A@mac.com> <675580.77045.qm@web56206.mail.re3.yahoo.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

done.
give it a shot and let me know how it goes.

On Mar 15, 2011, at 8:09 AM, Tsz Wo (Nicholas), Sze wrote:

> Hi Ian,
>=20
> Could you add me to the  hudson-jobadmin group?  It seems that Hudson =
sometimes=20
> does not run automatically.
>=20
> Nicholas Sze
>=20
>=20
>=20
>=20
>=20
> ________________________________
> From: Nigel Daley <ndaley@mac.com>
> To: general@hadoop.apache.org
> Sent: Wed, January 12, 2011 5:12:57 PM
> Subject: Re: triggering automated precommit testing
>=20
> I believe ommitters can gain access following this:
> http://wiki.apache.org/general/Hudson#How_do_I_get_an_account
> Ian would have to run a command on people.apache.org.
>=20
> Nige


From general-return-3174-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 14 22:14:44 2011
Return-Path: <general-return-3174-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 61665 invoked from network); 14 Mar 2011 22:14:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Mar 2011 22:14:43 -0000
Received: (qmail 81501 invoked by uid 500); 14 Mar 2011 22:14:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81445 invoked by uid 500); 14 Mar 2011 22:14:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81437 invoked by uid 99); 14 Mar 2011 22:14:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 22:14:42 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [98.139.52.213] (HELO nm16.bullet.mail.ac4.yahoo.com) (98.139.52.213)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 14 Mar 2011 22:14:33 +0000
Received: from [98.139.52.191] by nm16.bullet.mail.ac4.yahoo.com with NNFMP; 14 Mar 2011 22:14:12 -0000
Received: from [98.139.52.147] by tm4.bullet.mail.ac4.yahoo.com with NNFMP; 14 Mar 2011 22:14:12 -0000
Received: from [127.0.0.1] by omp1030.mail.ac4.yahoo.com with NNFMP; 14 Mar 2011 22:14:12 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 367370.98200.bm@omp1030.mail.ac4.yahoo.com
Received: (qmail 73881 invoked by uid 60001); 14 Mar 2011 22:14:12 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1300140852; bh=NUL/G8owVOzZCffUzPabaj5a8Jk5NPGG37f2+ayAIzA=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=5lqJmEa+4iVBXINCMXVsEnAS7Av1wUOy6NrQIKxOpPNezOc/1hyup19VwuY8gFRD3lkjCuEJo9nRdgykbBEfBy56JbIe65Jp3NhHjA4x3eKirLvVJ9C8tAHtlJDR0dHNW9LfxcFM42oGXWbTJMacIG9VwP6xWd/dkfp6HEcEe5M=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=TkbNBZ7SWrQmORQI9A7v6eosS9EEoV0+Cxe3i5eyfFTpswQhIfvUMYV+7KOx6d0YSP97o5Kko/8GUjD4eFRITR5tVax32lT+W3Qn2IugjAGSU6VNceN+C3KwkyuEcAr69VJvyNDek179MKCq53akp0NrVLb2ZuWwPbxI+BW8GMA=;
Message-ID: <229091.73636.qm@web56201.mail.re3.yahoo.com>
X-YMail-OSG: OsnZOskVM1nVH8sX7puc7KWlH1PwXRogtH8JcJ5gN4C27g0
 7_qGQCTJEYELIYkbBz7sNK2VKWAzyRToni4sWME1E4CvpTIpq5BbZ8pbFXSw
 QPBAbqKOb1rvrtyWUv.IVCRpP.33Db10bY5Qow_kCRkWsgNHR_cV1sbBSNjn
 PFPH7TgTio2hZhQyctqj7NWBI1C87ELS.inq7BYX3AjUUFvBkDcehskowkl5
 xKaQ8WgLFP_JAprTH1tF4KOmhlQjSVBpUVwsPFotPhOSjv6gI72SH06TL954
 T1IhmykNKiiYzhCy96IfGpNaTG47Rw5J8Ym3Dqq3GBzuB4ji2TAnq8zjqPwM
 lhm_UP1QZo4mDM4wvAt_th5D6Wpad7GIgtlII_1UECTaOnCO4.LxK7Q--
Received: from [209.131.62.113] by web56201.mail.re3.yahoo.com via HTTP; Mon, 14 Mar 2011 15:14:11 PDT
X-Mailer: YahooMailRC/559 YahooMailWebService/0.8.109.295617
References: <5125873.329641294877388152.JavaMail.jira@thor> <D570B102-7A4F-42E2-BB9A-B3CFC72428CC@mac.com> <AANLkTik14AD5XC+K-5=XMp-v9bdo=vmSG-9o7x=-qbP-@mail.gmail.com> <627F5E4D-DBDC-49BF-A94A-D30BB4FE7F9A@mac.com> <675580.77045.qm@web56206.mail.re3.yahoo.com> <2AFA74C9-8D5C-4F30-B776-CD75450113A2@Holsman.NET>
Date: Mon, 14 Mar 2011 15:14:11 -0700 (PDT)
From: "Tsz Wo \(Nicholas\), Sze" <s29752-hadoopgeneral@yahoo.com>
Subject: Re: triggering automated precommit testing
To: general@hadoop.apache.org
In-Reply-To: <2AFA74C9-8D5C-4F30-B776-CD75450113A2@Holsman.NET>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1973338577-1300140851=:73636"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-1973338577-1300140851=:73636
Content-Type: text/plain; charset=us-ascii

Hi Ian,
Thanks for the fast response!  It works. I just have started a build.
Nicholas




________________________________
From: Ian Holsman <hadoop@holsman.net>
To: general@hadoop.apache.org
Sent: Mon, March 14, 2011 3:11:07 PM
Subject: Re: triggering automated precommit testing

done.
give it a shot and let me know how it goes.

On Mar 15, 2011, at 8:09 AM, Tsz Wo (Nicholas), Sze wrote:

> Hi Ian,
> 
> Could you add me to the  hudson-jobadmin group?  It seems that Hudson sometimes 
>
> does not run automatically.
> 
> Nicholas Sze
> 
> 
> 
> 
> 
> ________________________________
> From: Nigel Daley <ndaley@mac.com>
> To: general@hadoop.apache.org
> Sent: Wed, January 12, 2011 5:12:57 PM
> Subject: Re: triggering automated precommit testing
> 
> I believe ommitters can gain access following this:
> http://wiki.apache.org/general/Hudson#How_do_I_get_an_account
> Ian would have to run a command on people.apache.org.
> 
> Nige
--0-1973338577-1300140851=:73636--

From general-return-3175-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 15 05:03:36 2011
Return-Path: <general-return-3175-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 74963 invoked from network); 15 Mar 2011 05:03:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Mar 2011 05:03:36 -0000
Received: (qmail 59656 invoked by uid 500); 15 Mar 2011 05:03:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59464 invoked by uid 500); 15 Mar 2011 05:03:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59451 invoked by uid 99); 15 Mar 2011 05:03:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 05:03:22 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.101 as permitted sender)
Received: from [17.148.16.101] (HELO asmtpout026.mac.com) (17.148.16.101)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 05:03:13 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.5]
 (c-71-198-192-174.hsd1.ca.comcast.net [71.198.192.174])
 by asmtp026.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LI300C3M20RBZ20@asmtp026.mac.com>
 for general@hadoop.apache.org; Mon, 14 Mar 2011 22:02:53 -0700 (PDT)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-03-14_05:2011-03-14,2011-03-14,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1103140223
Subject: Re: triggering automated precommit testing
References: <5125873.329641294877388152.JavaMail.jira@thor>
 <D570B102-7A4F-42E2-BB9A-B3CFC72428CC@mac.com>
 <AANLkTik14AD5XC+K-5=XMp-v9bdo=vmSG-9o7x=-qbP-@mail.gmail.com>
 <627F5E4D-DBDC-49BF-A94A-D30BB4FE7F9A@mac.com>
 <675580.77045.qm@web56206.mail.re3.yahoo.com>
From: Nigel Daley <ndaley@mac.com>
X-Mailer: iPhone Mail (8F190)
In-reply-to: <675580.77045.qm@web56206.mail.re3.yahoo.com>
Message-id: <54E2AC29-E45F-458F-951B-D1F39B1B4DBC@mac.com>
Date: Mon, 14 Mar 2011 22:02:48 -0700
To: "general@hadoop.apache.org" <general@hadoop.apache.org>

Nicholas, what do u mean it doesn't always start automatically?

Cheers,
Nige

Sent from my iPhone4

On Mar 14, 2011, at 2:09 PM, "Tsz Wo (Nicholas), Sze" <s29752-hadoopgeneral@yahoo.com> wrote:

> Hi Ian,
> 
> Could you add me to the  hudson-jobadmin group?  It seems that Hudson sometimes 
> does not run automatically.
> 
> Nicholas Sze
> 
> 
> 
> 
> 
> ________________________________
> From: Nigel Daley <ndaley@mac.com>
> To: general@hadoop.apache.org
> Sent: Wed, January 12, 2011 5:12:57 PM
> Subject: Re: triggering automated precommit testing
> 
> I believe ommitters can gain access following this:
> http://wiki.apache.org/general/Hudson#How_do_I_get_an_account
> Ian would have to run a command on people.apache.org.
> 
> Nige

From general-return-3176-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 15 09:56:00 2011
Return-Path: <general-return-3176-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16941 invoked from network); 15 Mar 2011 09:55:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Mar 2011 09:55:59 -0000
Received: (qmail 56053 invoked by uid 500); 15 Mar 2011 09:55:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55975 invoked by uid 500); 15 Mar 2011 09:55:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55967 invoked by uid 99); 15 Mar 2011 09:55:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 09:55:57 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of wlangiewicz@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 09:55:48 +0000
Received: by fxm7 with SMTP id 7so455157fxm.35
        for <general@hadoop.apache.org>; Tue, 15 Mar 2011 02:55:28 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:message-id:date:from:user-agent:mime-version:to
         :subject:content-type:content-transfer-encoding;
        bh=Fy8nppy4AZTYWbHd78BI406RyEepWUJ0cyMbazycqR4=;
        b=Fo/YBRLfTm92yxIUU14q3CmMBeHDydSYcnBYuVbOmDudZG6EFIO4Rsp7j+d9KCxu5b
         iNs3UPNW1iABpr4BGOIn3s6O0wgduRJaYERHG16i5xhYnupscCMCStlJQi9bQ9z1h7Px
         jw19vnHR3rOOCED3G7G54tgXeXFQXmkS3wigU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject
         :content-type:content-transfer-encoding;
        b=VvPUtjfw3PxgHnyJWwQ7pbsQqY6GIe8PU9fYk97t55YO19wHWG58RHD8ZPNFtB4u03
         CLtfUyTeuiZOh65cFHbvTIj+LDL0wV1nvH3pvRe/paY4GHhG39HrMdM6cvBxIsjhHZWk
         SItimY/6/Tw/ne2UXs7ZLFIvQ42/6eiDr/LFw=
Received: by 10.223.160.80 with SMTP id m16mr1375045fax.72.1300182928319;
        Tue, 15 Mar 2011 02:55:28 -0700 (PDT)
Received: from [172.19.4.124] (static.nk-net.pl [195.88.186.3])
        by mx.google.com with ESMTPS id f15sm3381029fax.34.2011.03.15.02.55.26
        (version=SSLv3 cipher=OTHER);
        Tue, 15 Mar 2011 02:55:27 -0700 (PDT)
Message-ID: <4D7F378E.30402@gmail.com>
Date: Tue, 15 Mar 2011 10:55:26 +0100
From: Wojciech Langiewicz <wlangiewicz@gmail.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.14) Gecko/20110223 Thunderbird/3.1.8
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: java.io.IOException: Split metadata size exceeded 10000000
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hello,
I'm having this problem running mapreduce jobs over about 10TB of data 
(smaller jobs are ok):
2011-03-15 07:48:22,031 ERROR org.apache.hadoop.mapred.JobTracker: Job 
initialization failed:
java.io.IOException: Split metadata size exceeded 10000000. Aborting job 
job_201103141436_0058
         at 
org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(SplitMetaInfoReader.java:48)
         at 
org.apache.hadoop.mapred.JobInProgress.createSplits(JobInProgress.java:732)
         at 
org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:633)
         at 
org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:3965)
         at 
org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:79)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
         at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
         at java.lang.Thread.run(Thread.java:619)

2011-03-15 07:48:22,031 INFO org.apache.hadoop.mapred.JobTracker: 
Failing job job_201103141436_0058

What settings should I change to run this job?
I'm using CDH3b3.
Thanks for all answers.

--
Wojciech Langiewcz

From general-return-3177-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 15 10:33:42 2011
Return-Path: <general-return-3177-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42354 invoked from network); 15 Mar 2011 10:33:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Mar 2011 10:33:41 -0000
Received: (qmail 91985 invoked by uid 500); 15 Mar 2011 10:33:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91931 invoked by uid 500); 15 Mar 2011 10:33:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91922 invoked by uid 99); 15 Mar 2011 10:33:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 10:33:39 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 10:33:35 +0000
Received: by fxm7 with SMTP id 7so485747fxm.35
        for <general@hadoop.apache.org>; Tue, 15 Mar 2011 03:33:14 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:cc:content-type:content-transfer-encoding;
        bh=Eopyq5LACkbhk3f0VEEIs+zfoevZb5PmSeAW9uYGYvE=;
        b=Uo55mpEUIcueDWmm4Zw/M42UHKpLsAb5ycgkseFWielnNiI5bDKz7ZB0/4iIjFGpCR
         2JBE9yOAQjorQkBwjUBNi7e2IJAx4HKYWeKOY6a3dboco8E1yP01NXliT56/vgcuM4eW
         MBApH7ZhxZf4IyIly/ug5yf0TsTesW3bG2L2k=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=bcc:mime-version:in-reply-to:references:from:date:message-id
         :subject:to:cc:content-type:content-transfer-encoding;
        b=jM827pAJnqHt0S9ibAsoTuNlPWQDsed+UhD7J9nHfDmDQOV60uheYSZArqEyYlS/H9
         3quhbPcay0ghW4Ze3heatJFOcEmaJwPhtic9NSeyR/gOQMLICYmdwBin2YEKIX1LYT6V
         0JhWF0VgzJb6DqRYpxZzdZl05jCRpZZXg62dg=
Received: by 10.223.127.14 with SMTP id e14mr2480258fas.97.1300185194166; Tue,
 15 Mar 2011 03:33:14 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.223.123.139 with HTTP; Tue, 15 Mar 2011 03:32:54 -0700 (PDT)
In-Reply-To: <4D7F378E.30402@gmail.com>
References: <4D7F378E.30402@gmail.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Tue, 15 Mar 2011 16:02:54 +0530
Message-ID: <AANLkTikHAG-0YYsHNpKTT_dMjQQ2Y07oXngYqZNHfZZY@mail.gmail.com>
Subject: Re: java.io.IOException: Split metadata size exceeded 10000000
To: CDH Users <cdh-user@cloudera.org>
Cc: wlangiewicz@gmail.com
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Moving this discussion to the CDH users list at cdh-user [at]
cloudera.org since it could be a CDH specific issue.

[Bcc: general]

On Tue, Mar 15, 2011 at 3:25 PM, Wojciech Langiewicz
<wlangiewicz@gmail.com> wrote:
> Hello,
> I'm having this problem running mapreduce jobs over about 10TB of data
> (smaller jobs are ok):
> 2011-03-15 07:48:22,031 ERROR org.apache.hadoop.mapred.JobTracker: Job
> initialization failed:
> java.io.IOException: Split metadata size exceeded 10000000. Aborting job
> job_201103141436_0058
> =A0 =A0 =A0 =A0at
> org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(S=
plitMetaInfoReader.java:48)
> =A0 =A0 =A0 =A0at
> org.apache.hadoop.mapred.JobInProgress.createSplits(JobInProgress.java:73=
2)
> =A0 =A0 =A0 =A0at
> org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:633)
> =A0 =A0 =A0 =A0at org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.=
java:3965)
> =A0 =A0 =A0 =A0at
> org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(Eage=
rTaskInitializationListener.java:79)
> =A0 =A0 =A0 =A0at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor=
.java:886)
> =A0 =A0 =A0 =A0at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav=
a:908)
> =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:619)
>
> 2011-03-15 07:48:22,031 INFO org.apache.hadoop.mapred.JobTracker: Failing
> job job_201103141436_0058
>
> What settings should I change to run this job?
> I'm using CDH3b3.
> Thanks for all answers.
>
> --
> Wojciech Langiewcz
>



--=20
Harsh J
http://harshj.com

From general-return-3178-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 15 14:20:09 2011
Return-Path: <general-return-3178-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72176 invoked from network); 15 Mar 2011 14:20:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Mar 2011 14:20:09 -0000
Received: (qmail 4035 invoked by uid 500); 15 Mar 2011 14:20:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3958 invoked by uid 500); 15 Mar 2011 14:20:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3950 invoked by uid 99); 15 Mar 2011 14:20:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 14:20:07 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 14:19:59 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 19D531BA799
	for <general@hadoop.apache.org>; Tue, 15 Mar 2011 14:19:38 +0000 (GMT)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id gWmC965brMCY for <general@hadoop.apache.org>;
	Tue, 15 Mar 2011 14:19:26 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by colossus.hpl.hp.com (Postfix) with ESMTPS id 622E11BA787
	for <general@hadoop.apache.org>; Tue, 15 Mar 2011 14:19:26 +0000 (GMT)
MailScanner-NULL-Check: 1300803554.4064@4KSnyYfCnIksmA8vznGbdg
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p2FEJDTs007571
	for <general@hadoop.apache.org>; Tue, 15 Mar 2011 14:19:14 GMT
Message-ID: <4D7F7561.30709@apache.org>
Date: Tue, 15 Mar 2011 14:19:13 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.14) Gecko/20110223 Thunderbird/3.1.8
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: problem to write on HDFS
References: <AANLkTikFebGD0-cmabwg64fpiGGOS-TSKq9TE+Qs+VCU@mail.gmail.com>
In-Reply-To: <AANLkTikFebGD0-cmabwg64fpiGGOS-TSKq9TE+Qs+VCU@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p2FEJDTs007571
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 14/03/11 17:48, Alessandro Binhara wrote:
> Hello ...
>
> I have a servlet on tomcat.. and it open a hdfs and write simple file with a
> content of post information.
> Well , in first test we had a 14.000 request per second.
> My servet start many trads to write on filesystem.
>
> i got this message on tomcat:
>
> Mar 11, 2011 6:00:20 PM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
> SEVERE: Socket accept failed
> java.net.SocketException: Too many open files
>
>
> HDFS is slow to write a file?
> How is a better strategy to write on HFDS...
>
> In real aplication we will have a 100.000 request per second to salve in
> hdfs.
>
> thanks..
>

Aaron is right, your expectations of Hadoop HDFS are probably wrong, but 
you haven't got that far yet, you are running out of socket handles, 
which is because you need to increase your ulimits so that more sockets 
can be open. Search for the error string you've seen and you'll find 
details on the problem and various solutions -a problem that has nothing 
to do with Hadoop at all.


From general-return-3179-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 17 19:02:06 2011
Return-Path: <general-return-3179-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 51838 invoked from network); 17 Mar 2011 19:02:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Mar 2011 19:02:05 -0000
Received: (qmail 52606 invoked by uid 500); 17 Mar 2011 19:02:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52545 invoked by uid 500); 17 Mar 2011 19:02:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52537 invoked by uid 99); 17 Mar 2011 19:02:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 19:02:03 +0000
X-ASF-Spam-Status: No, hits=-5.0 required=5.0
	tests=RCVD_IN_DNSWL_HI
X-Spam-Check-By: apache.org
Received-SPF: unknown (athena.apache.org: error in processing during lookup of jrottinghuis@ebay.com)
Received: from [216.33.244.7] (HELO rhv-mipot-002.corp.ebay.com) (216.33.244.7)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 19:01:58 +0000
DomainKey-Signature: s=corp; d=ebay.com; c=nofws; q=dns;
  h=X-EBay-Corp:X-IronPort-AV:Received:Received:From:To:CC:
   Date:Subject:Thread-Topic:Thread-Index:Message-ID:
   References:In-Reply-To:Accept-Language:Content-Language:
   X-MS-Has-Attach:X-MS-TNEF-Correlator:acceptlanguage:
   x-ems-proccessed:x-ems-stamp:Content-Type:
   Content-Transfer-Encoding:MIME-Version:X-CFilter;
  b=pSgxhKuWmBvzHbIJm50Sbnqu5ynwg/mIMWjhDKpA7Ruul28n1pET0WXz
   t3NDnORd1eefpSreAzXc+ZXjy3S/pD9PHW95JeRQ/zU9o7ohChQyAAW+8
   aSsigtNPACRyBw+;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=ebay.com; i=jrottinghuis@ebay.com; q=dns/txt; s=corp;
  t=1300388518; x=1331924518;
  h=from:to:cc:date:subject:message-id:references:
   in-reply-to:content-transfer-encoding:mime-version;
  bh=fa0ZVHjma0bjdtITRgVsILGCbvTGlJDxbfEvHUUt/jI=;
  b=16oOZz3cfC0eNPucn54tsE2HfkLM4xwOSYQAvSPXNEUhoRrfsKQXFgZf
   hGeE8FgwWEcxFbkRP2pYVJXx1SX3FuHwUruJSQyyREe1V83xxUpDt4GEM
   Z726JTaQBO3pvAZ;
X-EBay-Corp: Yes
X-IronPort-AV: E=Sophos;i="4.63,200,1299484800"; 
   d="scan'208";a="11431482"
Received: from rhv-vtenf-002.corp.ebay.com (HELO RHV-MEXHT-002.corp.ebay.com) ([10.112.113.53])
  by rhv-mipot-002.corp.ebay.com with ESMTP; 17 Mar 2011 12:01:36 -0700
Received: from RHV-MEXMS-002.corp.ebay.com ([10.245.17.114]) by
 RHV-MEXHT-002.corp.ebay.com ([10.245.24.101]) with mapi; Thu, 17 Mar 2011
 12:01:36 -0700
From: "Rottinghuis, Joep" <jrottinghuis@ebay.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>, CDH Users
	<cdh-user@cloudera.org>
CC: "wlangiewicz@gmail.com" <wlangiewicz@gmail.com>
Date: Thu, 17 Mar 2011 12:01:34 -0700
Subject: RE: java.io.IOException: Split metadata size exceeded 10000000
Thread-Topic: java.io.IOException: Split metadata size exceeded 10000000
Thread-Index: Acvi/HgegM3CCCS3TXyDOASoYHwwSwBzYtww
Message-ID: <F4FCE59AE46B2243922F224426BC869B01BD745BBA@RHV-MEXMS-002.corp.ebay.com>
References: <4D7F378E.30402@gmail.com>
 <AANLkTikHAG-0YYsHNpKTT_dMjQQ2Y07oXngYqZNHfZZY@mail.gmail.com>
In-Reply-To: <AANLkTikHAG-0YYsHNpKTT_dMjQQ2Y07oXngYqZNHfZZY@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
x-ems-proccessed: 10SqDH0iR7ekR7SRpKqm5A==
x-ems-stamp: AltoNIPk0e6ZbVQynXM4kA==
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-CFilter: Scanned

Doubt this is a CDH3 issue.
We saw the same with a large job using the 0.20-security branch.

There is a property (mapreduce.jobtracker.split.metainfo.maxsize) that can =
be used to override the default of 10^6.
We found that passing this along with the job has no effect, this worked on=
ly when setting this property on the jobtracker node. Not sure if this is a=
 feature or a bug.

Cheers,

Joep

-----Original Message-----
From: Harsh J [mailto:qwertymaniac@gmail.com]=20
Sent: Tuesday, March 15, 2011 3:33 AM
To: CDH Users
Cc: wlangiewicz@gmail.com
Subject: Re: java.io.IOException: Split metadata size exceeded 10000000

Moving this discussion to the CDH users list at cdh-user [at]
cloudera.org since it could be a CDH specific issue.

[Bcc: general]

On Tue, Mar 15, 2011 at 3:25 PM, Wojciech Langiewicz
<wlangiewicz@gmail.com> wrote:
> Hello,
> I'm having this problem running mapreduce jobs over about 10TB of data
> (smaller jobs are ok):
> 2011-03-15 07:48:22,031 ERROR org.apache.hadoop.mapred.JobTracker: Job
> initialization failed:
> java.io.IOException: Split metadata size exceeded 10000000. Aborting job
> job_201103141436_0058
> =A0 =A0 =A0 =A0at
> org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(S=
plitMetaInfoReader.java:48)
> =A0 =A0 =A0 =A0at
> org.apache.hadoop.mapred.JobInProgress.createSplits(JobInProgress.java:73=
2)
> =A0 =A0 =A0 =A0at
> org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:633)
> =A0 =A0 =A0 =A0at org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.=
java:3965)
> =A0 =A0 =A0 =A0at
> org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(Eage=
rTaskInitializationListener.java:79)
> =A0 =A0 =A0 =A0at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor=
.java:886)
> =A0 =A0 =A0 =A0at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav=
a:908)
> =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:619)
>
> 2011-03-15 07:48:22,031 INFO org.apache.hadoop.mapred.JobTracker: Failing
> job job_201103141436_0058
>
> What settings should I change to run this job?
> I'm using CDH3b3.
> Thanks for all answers.
>
> --
> Wojciech Langiewcz
>



--=20
Harsh J
http://harshj.com

From general-return-3180-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 17 20:16:14 2011
Return-Path: <general-return-3180-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84898 invoked from network); 17 Mar 2011 20:16:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Mar 2011 20:16:13 -0000
Received: (qmail 54082 invoked by uid 500); 17 Mar 2011 20:16:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54038 invoked by uid 500); 17 Mar 2011 20:16:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54030 invoked by uid 99); 17 Mar 2011 20:16:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 20:16:12 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of binhara@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 20:16:06 +0000
Received: by wyb40 with SMTP id 40so3863308wyb.35
        for <general@hadoop.apache.org>; Thu, 17 Mar 2011 13:15:44 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=lqHbJFoDziPUaFkZECwrcS7Xb6LuSZnnAxk7uL+w8oY=;
        b=CJY+lSC7POSTqMAkT35tM5+uku4pssiI7JS3ZC+JU0LH9HfEvEHQg9B3DFJuo2HHkp
         6ZjeX5X2PlTMlZ4oWGhHdS+DiZdJFWuuGue/1A5e/LWkxvCO/6Xt8mMxUzPOVi42ttPW
         HCsQvtkPRSbtERgYDLyTp+AMAz8cXR1G+T5gA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=k9sZKXQaAK8uhxc5+d+57dxj6zl2eOyoBnMfWLmESQr2Ygn0jhDVRdiUmvjALMpgnF
         CdLgo9rEM0+DEnOtDGutGQPu57NVI6PVrsLV4wSTVcDnWxldiEXQ3lBXk150uDpmKX4Q
         wnu8ye6HeDZ+sg6eBbVTL0HszrxmR8175thEM=
MIME-Version: 1.0
Received: by 10.216.87.8 with SMTP id x8mr206128wee.46.1300392944451; Thu, 17
 Mar 2011 13:15:44 -0700 (PDT)
Received: by 10.216.21.82 with HTTP; Thu, 17 Mar 2011 13:15:44 -0700 (PDT)
Date: Thu, 17 Mar 2011 17:15:44 -0300
Message-ID: <AANLkTikohj-11t7hF5bQ=wh6A0nzJDVtD4mpWveQC6Q1@mail.gmail.com>
Subject: How to pass a parameter to map ?
From: Alessandro Binhara <binhara@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6db297885c030049eb353dd

--0016e6db297885c030049eb353dd
Content-Type: text/plain; charset=ISO-8859-1

>
>
>
I need select a data on map...
I can pass paramenter to map when i configure a job?

Other questions :
I need sort a data and save in many files. the name of file is a sort Key
..???

thanks ..

--0016e6db297885c030049eb353dd--

From general-return-3181-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 17 20:58:20 2011
Return-Path: <general-return-3181-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21081 invoked from network); 17 Mar 2011 20:58:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Mar 2011 20:58:20 -0000
Received: (qmail 27662 invoked by uid 500); 17 Mar 2011 20:58:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27479 invoked by uid 500); 17 Mar 2011 20:58:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27471 invoked by uid 99); 17 Mar 2011 20:58:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 20:58:18 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 20:58:11 +0000
Received: by fxm7 with SMTP id 7so4025149fxm.35
        for <general@hadoop.apache.org>; Thu, 17 Mar 2011 13:57:50 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=cPAjiOkw4kFX5vvRDcLxdG1q8xuvmSDndB94N579OFY=;
        b=eo6NJ2leMUV7voxsQZX3JqgOwb4/XvOEKNCmH26juICuECalWfpnw3XgG11ht+H2pC
         aIeSNgeWgHHN50+uKkGNStbYmpFTe0Ucu+kgEGIAJ4SCST8QrIHO4X4qcBsruS75EBm+
         j6PbpxqjAwiHebQmmZ0i3vV9JBMvLPQOvPRP8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=OVI6IRzDZhfNqaSkSH0deSYIeraOqLTJGVlU+bAQFt8P9lDiio4cjXqeDoMC3SF8PM
         UZtASbOmFCdAuUruczn3cGAxDqmJ6BZwDI9juUAdf0BMsc/+8ldOe4glfYHxnI0zqdhq
         VU80HPxHZZQfCP1ISGlnXpmsMUinEyHPsjbz8=
MIME-Version: 1.0
Received: by 10.223.16.82 with SMTP id n18mr293001faa.48.1300395438948; Thu,
 17 Mar 2011 13:57:18 -0700 (PDT)
Received: by 10.223.83.196 with HTTP; Thu, 17 Mar 2011 13:57:18 -0700 (PDT)
In-Reply-To: <AANLkTikohj-11t7hF5bQ=wh6A0nzJDVtD4mpWveQC6Q1@mail.gmail.com>
References: <AANLkTikohj-11t7hF5bQ=wh6A0nzJDVtD4mpWveQC6Q1@mail.gmail.com>
Date: Thu, 17 Mar 2011 13:57:18 -0700
Message-ID: <AANLkTinXMFtfGLOUFMWco68q-BJeiqbOd8ftAZRDOFuk@mail.gmail.com>
Subject: Re: How to pass a parameter to map ?
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00151747862e34c1ef049eb3e84a
X-Virus-Checked: Checked by ClamAV on apache.org

--00151747862e34c1ef049eb3e84a
Content-Type: text/plain; charset=ISO-8859-1

>> I can pass paramenter to map when i configure a job?
You can utilize hadoop Configuration (JobConf).

On Thu, Mar 17, 2011 at 1:15 PM, Alessandro Binhara <binhara@gmail.com>wrote:

> >
> >
> >
> I need select a data on map...
> I can pass paramenter to map when i configure a job?
>
> Other questions :
> I need sort a data and save in many files. the name of file is a sort Key
> ..???
>
> thanks ..
>

--00151747862e34c1ef049eb3e84a--

From general-return-3182-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 17 21:37:21 2011
Return-Path: <general-return-3182-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67362 invoked from network); 17 Mar 2011 21:37:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Mar 2011 21:37:21 -0000
Received: (qmail 92198 invoked by uid 500); 17 Mar 2011 21:37:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92135 invoked by uid 500); 17 Mar 2011 21:37:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 92127 invoked by uid 99); 17 Mar 2011 21:37:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 21:37:19 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mlortiz@uci.cu designates 200.55.140.180 as permitted sender)
Received: from [200.55.140.180] (HELO mx3.uci.cu) (200.55.140.180)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 17 Mar 2011 21:37:15 +0000
Received: (qmail 20315 invoked by uid 507); 17 Mar 2011 21:36:46 -0000
Received: from 10.0.0.183 by ns3.uci.cu (envelope-from <mlortiz@uci.cu>, uid 501) with qmail-scanner-2.01st 
 (avp: 5.0.2.0. spamassassin: 3.0.6. perlscan: 2.01st.  
 Clear:RC:1(10.0.0.183):. 
 Processed in 0.792996 secs); 17 Mar 2011 21:36:46 -0000
Received: from unknown (HELO ucimail2.uci.cu) (10.0.0.183)
  by 0 with SMTP; 17 Mar 2011 21:36:45 -0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by ucimail2.uci.cu (Postfix) with ESMTP id 307F6C481B1;
	Thu, 17 Mar 2011 17:38:16 -0400 (CDT)
X-Virus-Scanned: amavisd-new at uci.cu
Received: from ucimail2.uci.cu ([127.0.0.1])
	by localhost (ucimail2.uci.cu [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id GHwFUW8MnEDy; Thu, 17 Mar 2011 17:38:15 -0400 (CDT)
Received: from [10.36.18.44] (marcosluis-aspire-5251.uci.cu [10.36.18.44])
	by ucimail2.uci.cu (Postfix) with ESMTP id 3999DC480A1;
	Thu, 17 Mar 2011 17:38:15 -0400 (CDT)
Subject: Re: How to pass a parameter to map ?
From: Marcos Ortiz <mlortiz@uci.cu>
To: Ted Yu <yuzhihong@gmail.com>
Cc: general@hadoop.apache.org
In-Reply-To: <AANLkTinXMFtfGLOUFMWco68q-BJeiqbOd8ftAZRDOFuk@mail.gmail.com>
References: <AANLkTikohj-11t7hF5bQ=wh6A0nzJDVtD4mpWveQC6Q1@mail.gmail.com>
	 <AANLkTinXMFtfGLOUFMWco68q-BJeiqbOd8ftAZRDOFuk@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Organization: UCI
Date: Thu, 17 Mar 2011 17:37:59 -0430
Message-ID: <1300399679.1308.28.camel@marcosluis-Aspire-5251>
Mime-Version: 1.0
X-Mailer: Evolution 2.30.3 
Content-Transfer-Encoding: 8bit

On Thu, 2011-03-17 at 13:57 -0700, Ted Yu wrote: 
> >> I can pass paramenter to map when i configure a job?
> You can utilize hadoop Configuration (JobConf).
> 
> On Thu, Mar 17, 2011 at 1:15 PM, Alessandro Binhara <binhara@gmail.com>wrote:
> 
> > >
> > >
> > >
> > I need select a data on map...
> > I can pass paramenter to map when i configure a job?
> >
> > Other questions :
> > I need sort a data and save in many files. the name of file is a sort Key
> > ..???
> >
> > thanks ..
> >
Take a deep look to JobConf object because there are many tuneable
parameters divided by sections:
- HDFS Writing
- Job submission
- Per-Job
- Map task Sumission and Execution
etc

Regards 
-- 
 Marcos Luís Ortíz Valmaseda
 Software Engineer
 Centro de Tecnologías de Gestión de Datos (DATEC)
 Universidad de las Ciencias Informáticas
 http://uncubanitolinuxero.blogspot.com
 http://www.linkedin.com/in/marcosluis2186



From general-return-3183-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 18 19:45:51 2011
Return-Path: <general-return-3183-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 40112 invoked from network); 18 Mar 2011 19:45:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Mar 2011 19:45:50 -0000
Received: (qmail 76448 invoked by uid 500); 18 Mar 2011 19:45:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76354 invoked by uid 500); 18 Mar 2011 19:45:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 32646 invoked by uid 99); 18 Mar 2011 19:07:20 -0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of krjeschke@omniti.com designates 74.125.82.48 as permitted sender)
MIME-Version: 1.0
Date: Fri, 18 Mar 2011 15:06:52 -0400
Message-ID: <AANLkTinJUVyjy73CQ5Vi0Sx_RrrHr9UfYaCrLsMHhHLA@mail.gmail.com>
Subject: Surge 2011 Conference CFP
From: Katherine Jeschke <krjeschke@omniti.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364162a9133449049ec67b96
X-Virus-Checked: Checked by ClamAV on apache.org

--0016364162a9133449049ec67b96
Content-Type: text/plain; charset=ISO-8859-1

We are excited to announce Surge 2011, the Scalability and Performance
Conference, to be held in Baltimore on Sept 28-30, 2011. The event focuses
on case studies that demonstrate successes (and failures) in Web
applications and Internet architectures. This year, we're adding Hack Day on
September 28th.

The inaugural, 2010 conference (http://omniti.com/surge/2010) was a smashing
success and we are currently accepting submissions for papers through April
3rd. You can find more information about topics online:

http://omniti.com/surge/2011

2010 attendees compared Surge to the early days of Velocity, and our
speakers received 3.5-4 out of 4 stars for quality of presentation and
quality of content! Nearly 90% of first-year attendees are planning to come
again in 2011.

For more information about the CFP or sponsorship of the event, please
contact us at surge (AT) omniti (DOT) com.

-- 
Katherine Jeschke
Marketing Director
OmniTI Computer Consulting, Inc.
7070 Samuel Morse Drive, Ste.150
Columbia, MD 21046
O: 410/872-4910, 222
C: 443/643-6140
omniti.com
circonus.com

--0016364162a9133449049ec67b96--

From general-return-3184-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 22 16:19:41 2011
Return-Path: <general-return-3184-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56642 invoked from network); 22 Mar 2011 16:19:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Mar 2011 16:19:41 -0000
Received: (qmail 11092 invoked by uid 500); 22 Mar 2011 16:19:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11001 invoked by uid 500); 22 Mar 2011 16:19:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 10978 invoked by uid 99); 22 Mar 2011 16:19:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 16:19:39 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of samplise@gmail.com designates 209.85.218.48 as permitted sender)
Received: from [209.85.218.48] (HELO mail-yi0-f48.google.com) (209.85.218.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 16:19:33 +0000
Received: by yia25 with SMTP id 25so3845140yia.35
        for <general@hadoop.apache.org>; Tue, 22 Mar 2011 09:19:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=WLC2u7LsYX9UOlEK7iZ2jm2BpsA9U/VvSmXtvCHT0rM=;
        b=M4u6U+H69soCNmdUiUelAgxjcnL4Xc7bQQRifP7XHd3zmB1yDMwytBKtcqN6MiOcV5
         je9HtlL4Ebcp4aUVx1GHnfE3/CLlhGGBTKwZLWJI/shXrUD2TfkKflt5ijHzfDWidWXB
         ksvjPgdzZTlFmhJ7DwePTHd4B8eE+8h7qEzJU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=pPx3G+7aByVu7hJRlADUl+UHD1CsSEm5AdYBSVi2QXssnra4dgAi5+2h4mGD5xU7Nn
         YMqnmyP/slm95vkALoo7XMXBzAkiTSxTI5D4IM+nGe20ZO1fiYn/DMzar7PUSyCigicD
         O8Yrky/k3Oj5oXgJQDkihLz46g1WLx55RmFNk=
MIME-Version: 1.0
Received: by 10.236.67.1 with SMTP id i1mr7093890yhd.341.1300810752326; Tue,
 22 Mar 2011 09:19:12 -0700 (PDT)
Received: by 10.236.110.19 with HTTP; Tue, 22 Mar 2011 09:19:12 -0700 (PDT)
Date: Tue, 22 Mar 2011 12:19:12 -0400
Message-ID: <AANLkTink=8yg=5yLr9ws+AGFSRYkXY1WNEiNxXHdoB+Q@mail.gmail.com>
Subject: How to insert some print codes into Hadoop?
From: Bo Sang <samplise@gmail.com>
To: Hadoop General Mail List <general@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=00248c710507d0014a049f149a8f

--00248c710507d0014a049f149a8f
Content-Type: text/plain; charset=ISO-8859-1

Hi, guys:

I would like to do some minor modification of Hadoop (just to insert some
print codes into particular places). And I have the following questions:

1. It seems there are three parts of Hadoop: common, hdfs, mapred. And they
are packed as three independent jar packages. Could I only modify one part
(eg, common) and pack a new jar package without modifying the rest two.

2. I have try to import the folder hadoop-0.21.0/common into eclipse as a
project. But eclipse fails to recognize it as an existing project. But if I
import folder hadoop-0.21.0 as a existing project, it works. However, I only
want to modify common part. How could I only modify  common part and export
a new common jar package without modifying the rest two parts.

-- 
Best Regards!

Sincerely
Bo Sang

--00248c710507d0014a049f149a8f--

From general-return-3185-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 22 20:03:49 2011
Return-Path: <general-return-3185-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12838 invoked from network); 22 Mar 2011 20:03:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Mar 2011 20:03:49 -0000
Received: (qmail 69474 invoked by uid 500); 22 Mar 2011 20:03:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69411 invoked by uid 500); 22 Mar 2011 20:03:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69403 invoked by uid 99); 22 Mar 2011 20:03:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 20:03:47 +0000
X-ASF-Spam-Status: No, hits=3.9 required=5.0
	tests=DCC_CHECK,HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of chenqibj@cn.ibm.com designates 202.81.31.145 as permitted sender)
Received: from [202.81.31.145] (HELO e23smtp03.au.ibm.com) (202.81.31.145)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 20:03:37 +0000
Received: from d23relay03.au.ibm.com (d23relay03.au.ibm.com [202.81.31.245])
	by e23smtp03.au.ibm.com (8.14.4/8.13.1) with ESMTP id p2MJwRKE029226
	for <general@hadoop.apache.org>; Wed, 23 Mar 2011 06:58:27 +1100
Received: from d23av04.au.ibm.com (d23av04.au.ibm.com [9.190.235.139])
	by d23relay03.au.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id p2MK37qq2134220
	for <general@hadoop.apache.org>; Wed, 23 Mar 2011 07:03:07 +1100
Received: from d23av04.au.ibm.com (loopback [127.0.0.1])
	by d23av04.au.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id p2MK36tR002479
	for <general@hadoop.apache.org>; Wed, 23 Mar 2011 07:03:06 +1100
Received: from d23m0037.cn.ibm.com (d23m0037.cn.ibm.com [9.181.2.105])
	by d23av04.au.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id p2MK36L2002475
	for <general@hadoop.apache.org>; Wed, 23 Mar 2011 07:03:06 +1100
Subject: Keith Chen is out of office for vacation
Auto-Submitted: auto-generated
From: Qi BJ Chen <chenqibj@cn.ibm.com>
To: general@hadoop.apache.org
Message-ID: <OF9315E288.1FA32857-ON4825785B.006E24E3-4825785B.006E24E3@cn.ibm.com>
Date: Wed, 23 Mar 2011 04:03:04 +0800
X-MIMETrack: Serialize by Router on D23M0037/23/M/IBM(Release 8.5.1FP4|July 25, 2010) at
 03/23/2011 04:03:04
MIME-Version: 1.0
Content-type: multipart/alternative; 
	Boundary="0__=C7BBF2C8DFFDA2738f9e8a93df938690918cC7BBF2C8DFFDA273"
Content-Disposition: inline
X-Virus-Checked: Checked by ClamAV on apache.org

--0__=C7BBF2C8DFFDA2738f9e8a93df938690918cC7BBF2C8DFFDA273
Content-type: text/plain; charset=US-ASCII


I will be out of the office starting  2011-03-22 and will not return until
2011-03-31.

--0__=C7BBF2C8DFFDA2738f9e8a93df938690918cC7BBF2C8DFFDA273--


From general-return-3186-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 23 08:13:15 2011
Return-Path: <general-return-3186-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56797 invoked from network); 23 Mar 2011 08:13:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Mar 2011 08:13:15 -0000
Received: (qmail 61656 invoked by uid 500); 23 Mar 2011 08:13:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61578 invoked by uid 500); 23 Mar 2011 08:13:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61570 invoked by uid 99); 23 Mar 2011 08:13:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 08:13:13 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of zly928@gmail.com designates 209.85.220.176 as permitted sender)
Received: from [209.85.220.176] (HELO mail-vx0-f176.google.com) (209.85.220.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 08:13:06 +0000
Received: by vxi40 with SMTP id 40so8279100vxi.35
        for <general@hadoop.apache.org>; Wed, 23 Mar 2011 01:12:46 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type:content-transfer-encoding;
        bh=D7XH9iRCs46a/7xNOG4AyZTp7UMb27GzgIZCwVbL8m0=;
        b=xKdq3pbGQSeFOVHlL74cHFPtshMsOV5vQf65buGwDR9gAvZKf/oHAnpl28e7Awmosn
         kHOea0+RwpFrmyGVssJDGx61ZUiDa3XnhYG/FHyExDEo/++qCQ8lLdK5PwTLjOR/oXwu
         BQBiOcn/WRSzSfJDP8KkCeAuP7lzgYSdTInyc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=V8pE/IXeUDBPgCwq/iQYOKoio7LTpE8QE2pXWD/aTgUL7n8JzIOm1U0TjRcfvXxLq/
         OXEHk+PJ9ZUxQHrIs4w0y/NbDZz3orHk6W68Y4llWVxJMMw+k8/qoWlQ64nNMsFRTcvG
         8F35IHHg9JoWZEaaPD5eXWyvLB9D7x+KHfhzw=
MIME-Version: 1.0
Received: by 10.52.98.97 with SMTP id eh1mr4230184vdb.148.1300867965992; Wed,
 23 Mar 2011 01:12:45 -0700 (PDT)
Received: by 10.220.96.77 with HTTP; Wed, 23 Mar 2011 01:12:45 -0700 (PDT)
In-Reply-To: <OF9315E288.1FA32857-ON4825785B.006E24E3-4825785B.006E24E3@cn.ibm.com>
References: <OF9315E288.1FA32857-ON4825785B.006E24E3-4825785B.006E24E3@cn.ibm.com>
Date: Wed, 23 Mar 2011 16:12:45 +0800
Message-ID: <AANLkTing-GBTP2ZONTA82xKLrjhW5vttZ+TcPOg+k-xE@mail.gmail.com>
Subject: Re: Keith Chen is out of office for vacation
From: =?GB2312?B?1tzB4dHF?= <zly928@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=GB2312
Content-Transfer-Encoding: quoted-printable

Oh! It's wonderful! And I hope your vacation happy!

2011/3/23, Qi BJ Chen <chenqibj@cn.ibm.com>:
>
> I will be out of the office starting  2011-03-22 and will not return unti=
l
> 2011-03-31.
>

--=20
=B4=D3=CE=D2=B5=C4=D2=C6=B6=AF=C9=E8=B1=B8=B7=A2=CB=CD

From general-return-3190-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 23 19:13:46 2011
Return-Path: <general-return-3190-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 888 invoked from network); 23 Mar 2011 19:13:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Mar 2011 19:13:33 -0000
Received: (qmail 10479 invoked by uid 500); 23 Mar 2011 18:45:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 10433 invoked by uid 500); 23 Mar 2011 18:45:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 60283 invoked by uid 99); 23 Mar 2011 17:12:10 -0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Subject: Re: google snappy
From: Tim Wintle <tim.wintle@teamrubber.com>
Reply-To: tim.wintle@teamrubber.com
To: common-user@hadoop.apache.org
Cc: general@hadoop.apache.org, Weishung Chung <weishung@gmail.com>
In-Reply-To: <AANLkTina=Mcpuzsy6pwqz5VkPrrfJ6YoTqAALsyX-wGZ@mail.gmail.com>
References: <AANLkTimJavGPFwc89uWp5GHjWa2j2vvDyRyYb2m3YUOa@mail.gmail.com>
	 <AANLkTina=Mcpuzsy6pwqz5VkPrrfJ6YoTqAALsyX-wGZ@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Organization: Team Rubber
Date: Wed, 23 Mar 2011 17:11:39 +0000
Message-ID: <1300900299.8811.3.camel@tim-laptop>
Mime-Version: 1.0
X-Mailer: Evolution 2.28.3 
Content-Transfer-Encoding: 7bit
X-SA-Exim-Connect-IP: 87.194.110.230
X-SA-Exim-Mail-From: tim.wintle@teamrubber.com
X-SA-Exim-Scanned: No (on megamail.netsight.co.uk); SAEximRunCond expanded to false

On Wed, 2011-03-23 at 10:03 -0700, Jean-Daniel Cryans wrote:
> Somebody obviously needs to publish some benchmarks, but knowing
> Snappy's origin I can believe that claim. 

There were some benchmarks in the original Bigtable presentation

Results from compressing bigtable blocks:

Algorithm  % remaining	Encoding	Decoding
Gzip       13.4%        21MB/s          118MB/s
LZO        20.5%        135MB/s         410MB/s
Zippy      22.2%        172MB/s         409MB/s

(Zippy is apparently what's now been renamed snappy)

Tim Wintle


From general-return-3191-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 23 19:14:06 2011
Return-Path: <general-return-3191-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1064 invoked from network); 23 Mar 2011 19:13:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Mar 2011 19:13:39 -0000
Received: (qmail 11905 invoked by uid 500); 23 Mar 2011 18:45:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11842 invoked by uid 500); 23 Mar 2011 18:45:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 65089 invoked by uid 99); 23 Mar 2011 18:20:59 -0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of weishung@gmail.com designates 209.85.214.48 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:cc:content-type;
        bh=h1h9SVX+1ft8ZNNMWF/j+rglYfNubsviNW0t9HlG57g=;
        b=Hg45bQ29Vp05osqTCpp2DmNYm6RRK0/fKgQhNsHW3vwJSfELduECGCeHR/x2icshxU
         EBHR1yv5z5oFuzi/ynBkln7P3TEusFV8E97Bpk3bt8qe+1wWu3Z7KqBWN7qpwQihP/U9
         c62EnzdV2m8I86vBGL/GFz3jAETDlAffpZciY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=d6PwrRT3BHQRA3eNlHWXpdGkLr9r2FpxDjCnztF/EIjnOb0ZluqLKhUlf2xsLx5KEi
         bgatfEE8X3DrSyOY5q+uOWQ/vej2c7kBRYdfZ3qikarJN1Crx+6EIYZRFAGrUJbgMsDZ
         ecJ+1ts12veFiEWTm8v/kINVyKOVep7T+xWE8=
MIME-Version: 1.0
In-Reply-To: <1300900299.8811.3.camel@tim-laptop>
References: <AANLkTimJavGPFwc89uWp5GHjWa2j2vvDyRyYb2m3YUOa@mail.gmail.com>
	<AANLkTina=Mcpuzsy6pwqz5VkPrrfJ6YoTqAALsyX-wGZ@mail.gmail.com>
	<1300900299.8811.3.camel@tim-laptop>
Date: Wed, 23 Mar 2011 13:20:32 -0500
Message-ID: <AANLkTim2=B+GZiv29o1b2EoNY_u_aeUVC1JfqQWOxvZO@mail.gmail.com>
Subject: Re: google snappy
From: Weishung Chung <weishung@gmail.com>
To: general@hadoop.apache.org
Cc: common-user@hadoop.apache.org, tim.wintle@teamrubber.com
Content-Type: multipart/alternative; boundary=00032555947694a57d049f2a6a90

--00032555947694a57d049f2a6a90
Content-Type: text/plain; charset=ISO-8859-1

Great to know that hadoop/hbase are integrating it :D

On Wed, Mar 23, 2011 at 12:11 PM, Tim Wintle <tim.wintle@teamrubber.com>wrote:

> On Wed, 2011-03-23 at 10:03 -0700, Jean-Daniel Cryans wrote:
> > Somebody obviously needs to publish some benchmarks, but knowing
> > Snappy's origin I can believe that claim.
>
> There were some benchmarks in the original Bigtable presentation
>
> Results from compressing bigtable blocks:
>
> Algorithm  % remaining  Encoding        Decoding
> Gzip       13.4%        21MB/s          118MB/s
> LZO        20.5%        135MB/s         410MB/s
> Zippy      22.2%        172MB/s         409MB/s
>
> (Zippy is apparently what's now been renamed snappy)
>
> Tim Wintle
>
>

--00032555947694a57d049f2a6a90--

From general-return-3188-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 23 19:31:04 2011
Return-Path: <general-return-3188-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63138 invoked from network); 23 Mar 2011 19:31:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Mar 2011 19:31:03 -0000
Received: (qmail 2776 invoked by uid 500); 23 Mar 2011 17:41:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2728 invoked by uid 500); 23 Mar 2011 17:41:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2720 invoked by uid 99); 23 Mar 2011 17:41:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 17:41:42 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 17:41:35 +0000
Received: by wwc33 with SMTP id 33so11097973wwc.29
        for <general@hadoop.apache.org>; Wed, 23 Mar 2011 10:41:14 -0700 (PDT)
Received: by 10.227.177.199 with SMTP id bj7mr297214wbb.140.1300902074121;
 Wed, 23 Mar 2011 10:41:14 -0700 (PDT)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.227.133.135 with HTTP; Wed, 23 Mar 2011 10:40:54 -0700 (PDT)
In-Reply-To: <AANLkTink=8yg=5yLr9ws+AGFSRYkXY1WNEiNxXHdoB+Q@mail.gmail.com>
References: <AANLkTink=8yg=5yLr9ws+AGFSRYkXY1WNEiNxXHdoB+Q@mail.gmail.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Wed, 23 Mar 2011 10:40:54 -0700
X-Google-Sender-Auth: zRZtxH0MGjvf5DvtwXJ1nRHmcE8
Message-ID: <AANLkTinXui3rK=mEFmGSJDjP12DsXQZfA26LotLwVsJc@mail.gmail.com>
Subject: Re: How to insert some print codes into Hadoop?
To: common-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

[Moving to common-user@, Bcc'ing general@]

If you know where you need to have your print statements you can use
AspectJ to do runtime injection of needed java code into desirable
spots. You don't need to even touch the source code for that - just
instrument (weave) the jar file
--
=A0 Take care,
Konstantin (Cos) Boudnik

On Tue, Mar 22, 2011 at 09:19, Bo Sang <samplise@gmail.com> wrote:
> Hi, guys:
>
> I would like to do some minor modification of Hadoop (just to insert some
> print codes into particular places). And I have the following questions:
>
> 1. It seems there are three parts of Hadoop: common, hdfs, mapred. And th=
ey
> are packed as three independent jar packages. Could I only modify one par=
t
> (eg, common) and pack a new jar package without modifying the rest two.
>
> 2. I have try to import the folder hadoop-0.21.0/common into eclipse as a
> project. But eclipse fails to recognize it as an existing project. But if=
 I
> import folder hadoop-0.21.0 as a existing project, it works. However, I o=
nly
> want to modify common part. How could I only modify =A0common part and ex=
port
> a new common jar package without modifying the rest two parts.
>
> --
> Best Regards!
>
> Sincerely
> Bo Sang
>

From general-return-3189-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 23 19:50:10 2011
Return-Path: <general-return-3189-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 52218 invoked from network); 23 Mar 2011 19:50:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Mar 2011 19:50:09 -0000
Received: (qmail 29163 invoked by uid 500); 23 Mar 2011 18:00:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29109 invoked by uid 500); 23 Mar 2011 18:00:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29099 invoked by uid 99); 23 Mar 2011 18:00:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 18:00:48 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of samplise@gmail.com designates 74.125.83.48 as permitted sender)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 18:00:42 +0000
Received: by gwj22 with SMTP id 22so4603280gwj.35
        for <general@hadoop.apache.org>; Wed, 23 Mar 2011 11:00:21 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=c6LILXuUknXOZ8tx4FDRNVdVo5O/pYo1QLwHEL+Q+LQ=;
        b=ttlU1zzhvMrWp9jiK4awpwoDFFrGSeRFKgw6snk2pK/vVnR0Wy2t8FrTSgZbyykqy0
         HQttmoURJPuxLCZMryw/HRUCLAiQHKclSfbtMcLCMpv/9VXunNyppmkSjRwG5x0UdGK1
         mGLx8p4g/dOz05e5vhCBtXRjsZl3rWul9WH3Q=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=YuK+Wwfcky4/ATaBdpbeKhm7nw+t/Npr9nzjOdz2yWzEJi4aoXhGFYPnxqnU3Tl6nX
         OzvSXn4SvATKG4pf095cfQEO9ha2PooOZjN+NopCXaG0j1n3eyIQmwiycVUHuISyIeiB
         3NCB6awqNHF/0T0XutNDB3M4R5vTki1MQcx30=
MIME-Version: 1.0
Received: by 10.236.154.97 with SMTP id g61mr7258919yhk.91.1300903221331; Wed,
 23 Mar 2011 11:00:21 -0700 (PDT)
Received: by 10.236.110.19 with HTTP; Wed, 23 Mar 2011 11:00:21 -0700 (PDT)
In-Reply-To: <AANLkTinXui3rK=mEFmGSJDjP12DsXQZfA26LotLwVsJc@mail.gmail.com>
References: <AANLkTink=8yg=5yLr9ws+AGFSRYkXY1WNEiNxXHdoB+Q@mail.gmail.com>
	<AANLkTinXui3rK=mEFmGSJDjP12DsXQZfA26LotLwVsJc@mail.gmail.com>
Date: Wed, 23 Mar 2011 14:00:21 -0400
Message-ID: <AANLkTi=c9MpYhxq_ybYLs2KtuE+xmq+-oZ=kwBJfKJX8@mail.gmail.com>
Subject: Re: How to insert some print codes into Hadoop?
From: Bo Sang <samplise@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf303dd75865083b049f2a22c1

--20cf303dd75865083b049f2a22c1
Content-Type: text/plain; charset=ISO-8859-1

Thanks very much for your reply. But if I would like to modify Hadoop source
code, how could I achieve this goal?

On Wed, Mar 23, 2011 at 1:40 PM, Konstantin Boudnik <cos@apache.org> wrote:

> [Moving to common-user@, Bcc'ing general@]
>
> If you know where you need to have your print statements you can use
> AspectJ to do runtime injection of needed java code into desirable
> spots. You don't need to even touch the source code for that - just
> instrument (weave) the jar file
> --
>   Take care,
> Konstantin (Cos) Boudnik
>
> On Tue, Mar 22, 2011 at 09:19, Bo Sang <samplise@gmail.com> wrote:
> > Hi, guys:
> >
> > I would like to do some minor modification of Hadoop (just to insert some
> > print codes into particular places). And I have the following questions:
> >
> > 1. It seems there are three parts of Hadoop: common, hdfs, mapred. And
> they
> > are packed as three independent jar packages. Could I only modify one
> part
> > (eg, common) and pack a new jar package without modifying the rest two.
> >
> > 2. I have try to import the folder hadoop-0.21.0/common into eclipse as a
> > project. But eclipse fails to recognize it as an existing project. But if
> I
> > import folder hadoop-0.21.0 as a existing project, it works. However, I
> only
> > want to modify common part. How could I only modify  common part and
> export
> > a new common jar package without modifying the rest two parts.
> >
> > --
> > Best Regards!
> >
> > Sincerely
> > Bo Sang
> >
>



-- 
Best Regards!

Sincerely
Bo Sang

--20cf303dd75865083b049f2a22c1--

From general-return-3187-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 23 19:57:13 2011
Return-Path: <general-return-3187-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56350 invoked from network); 23 Mar 2011 19:57:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Mar 2011 19:57:13 -0000
Received: (qmail 35468 invoked by uid 500); 23 Mar 2011 17:03:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 35392 invoked by uid 500); 23 Mar 2011 17:03:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 35335 invoked by uid 99); 23 Mar 2011 17:03:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 17:03:49 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jdcryans@gmail.com designates 209.85.218.48 as permitted sender)
Received: from [209.85.218.48] (HELO mail-yi0-f48.google.com) (209.85.218.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 17:03:43 +0000
Received: by yia25 with SMTP id 25so4569320yia.35
        for <general@hadoop.apache.org>; Wed, 23 Mar 2011 10:03:22 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:cc:content-type;
        bh=22/8P2Q/E70qAO0+oRsCASfu0ifq2ACkbSoI90JYtJA=;
        b=V46jEAtVIg3cQ9XeG/SUT/LqI3VAHsJW72IxZSQS36YU9MweygJ9rCd5MjPweFqIys
         Ww4/Egvxwm2oRmdtcz8CukuGZENkOZZePj159N5+eKXz0YGtSL7Hky5gxrpevcayIlG9
         7wX5Jd0fJqrE0gxmNvaxUCyo047eNevxQPCn4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:cc:content-type;
        b=lflwKhCwWKjMQH3FpTOWrnRD6Ge6zrPjUS/YjDccvvLJ5pPs/eshw8NtLomvWV8KQQ
         so/krghFev/DL9ozj2o1TMmFEmositj+GP+aKdS4lFJ8icAJm/sRL5zyLGhU0VrUF3JF
         trcz9aH+BpPFPdtzCPTyG0S/FnXJL43H5rQV8=
MIME-Version: 1.0
Received: by 10.42.170.74 with SMTP id e10mr4119022icz.56.1300899802044; Wed,
 23 Mar 2011 10:03:22 -0700 (PDT)
Sender: jdcryans@gmail.com
Received: by 10.42.162.197 with HTTP; Wed, 23 Mar 2011 10:03:21 -0700 (PDT)
In-Reply-To: <AANLkTimJavGPFwc89uWp5GHjWa2j2vvDyRyYb2m3YUOa@mail.gmail.com>
References: <AANLkTimJavGPFwc89uWp5GHjWa2j2vvDyRyYb2m3YUOa@mail.gmail.com>
Date: Wed, 23 Mar 2011 10:03:21 -0700
X-Google-Sender-Auth: teMvtYYDgQJCDyIY_SZEEJzIILM
Message-ID: <AANLkTina=Mcpuzsy6pwqz5VkPrrfJ6YoTqAALsyX-wGZ@mail.gmail.com>
Subject: Re: google snappy
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: general@hadoop.apache.org
Cc: Weishung Chung <weishung@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1

(Please don't cross-post like that, it only adds confusion. I put
everything in bcc and posted to general instead)

Their README says the following:

Snappy usually is faster than algorithms in the same class (e.g. LZO,
LZF, FastLZ, QuickLZ, etc.) while achieving comparable compression
ratios.

Somebody obviously needs to publish some benchmarks, but knowing
Snappy's origin I can believe that claim.

Relevant jiras:

HADOOP-7206 Integrate Snappy compression
HBASE-3691   Add compressor support for 'snappy', google's compressor

J-D

On Wed, Mar 23, 2011 at 9:52 AM, Weishung Chung <weishung@gmail.com> wrote:
> Hey my fellow hadoop/hbase developers,
>
> I just came across this google compression/decompression package yesterday,
> could we make a good use of this compression scheme in hadoop? It's written
> in C++ though.
>
> http://code.google.com/p/snappy/
>
> <http://code.google.com/p/snappy/>I haven't looked close into this snappy
> package yet but i would love to know about the differences compared to LZO.
>
> Thank you,
> Wei Shung
>

From general-return-3192-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 08:43:40 2011
Return-Path: <general-return-3192-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 15241 invoked from network); 24 Mar 2011 08:43:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 08:43:39 -0000
Received: (qmail 98357 invoked by uid 500); 24 Mar 2011 08:43:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98136 invoked by uid 500); 24 Mar 2011 08:43:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98128 invoked by uid 99); 24 Mar 2011 08:43:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 08:43:37 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of wlangiewicz@gmail.com designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 08:43:30 +0000
Received: by ewy22 with SMTP id 22so3025590ewy.35
        for <general@hadoop.apache.org>; Thu, 24 Mar 2011 01:43:08 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:message-id:date:from:user-agent:mime-version:to
         :subject:content-type:content-transfer-encoding;
        bh=GbtlBDD7TNLK3r59L2ra62e/A7t4xcmxBz4VtadzY1w=;
        b=ZBJO7m+n9Rtoxn+n5tEvj5O2IDC8NLMkBLUlM9KGwZzyuDgiuiRDYzPlnOE1UEMZ8d
         DiYuyEKWAWG82iEThUu4Gsam/cKyG7R0Caaaby0L/DZRRJ8XjfWgMApH5IIW2rQfWHZT
         GBgsAPGo28ISFYP/uewNamutIAdP0Lz+qE/UQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject
         :content-type:content-transfer-encoding;
        b=C3NZFriSsv/gapV8scghUdk/hDd9ZGR67U4uJ8GZJDwG/Mpe0HnbStigqyrPSGbzhG
         N7hPdE/5ZA4F2RebVyHaZ6Kx6O3F66Q8PY+evrRG0aNLLvvNLMW0ttnO3RP6j4zXw5kJ
         OkdTcWVF2mU4PyLzJoWlQ01Wn5x37tMylqHvE=
Received: by 10.213.113.205 with SMTP id b13mr162754ebq.62.1300956188621;
        Thu, 24 Mar 2011 01:43:08 -0700 (PDT)
Received: from [172.19.4.124] (static.nk-net.pl [195.88.186.3])
        by mx.google.com with ESMTPS id x54sm4298924eeh.5.2011.03.24.01.43.07
        (version=SSLv3 cipher=OTHER);
        Thu, 24 Mar 2011 01:43:07 -0700 (PDT)
Message-ID: <4D8B041A.7020007@gmail.com>
Date: Thu, 24 Mar 2011 09:43:06 +0100
From: Wojciech Langiewicz <wlangiewicz@gmail.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.14) Gecko/20110223 Thunderbird/3.1.8
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Problem with NameNode Storage: IMAGE_AND_EDITS Failed
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

Hello,
Right now I'm having this issue on my cluster:
On NameNode web interface at the bottom of the page is table:
NameNode Storage
Storage Directory	Type	State
/srv/dfs/name	IMAGE_AND_EDITS	Failed

I have had this issue before, and I rebooted Hadoop, but lost many files.
What can I do now that I won't lose files since last backup, and what 
this warning actually means?
Any help would be greatly appreciated.
--
Wojciech Langiewicz

From general-return-3193-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 08:51:50 2011
Return-Path: <general-return-3193-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60165 invoked from network); 24 Mar 2011 08:51:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 08:51:49 -0000
Received: (qmail 9654 invoked by uid 500); 24 Mar 2011 08:51:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 9591 invoked by uid 500); 24 Mar 2011 08:51:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 9583 invoked by uid 99); 24 Mar 2011 08:51:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 08:51:48 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 08:51:42 +0000
Received: by fxm7 with SMTP id 7so11451105fxm.35
        for <general@hadoop.apache.org>; Thu, 24 Mar 2011 01:51:22 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type:content-transfer-encoding;
        bh=ZVc2rH68hoyyTSB2Kqxe6XT9tDCyzST+jqLN4EPePiE=;
        b=hjGMN0f3/l1iZjJ0qLq5PdIrk2ysRSwmH91vgDugCbzV5MsntTWNhe6oLUJ9yxcyQe
         odSvUteK4zZYAyXMwMdeXSnjbwDT4T/nwYxRE8N9/ZztoO+yJH8e/LYmHoYtbZG6WAeD
         1+oUQcu3Xbh7XTZqLwFTzJgUIRXFyvcwa7Kxg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=Bsw82wIbqmawx0ijnO1sfD1uUQLhU9oKkUh1lbFmuAlvCkDOBA0Oxg6OXKelmtSiKR
         lYOTj1sroaI6DMF3qX21B/UpW1/xI8fSjGL8Ai83tl6G9fSUxKHv6PEXd8rCBjUJv6G+
         F/sxJ+46FSPCOWZRwp3cad9uqI2dQ7qbjNEdY=
Received: by 10.223.101.87 with SMTP id b23mr2217413fao.97.1300956682101; Thu,
 24 Mar 2011 01:51:22 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.223.123.139 with HTTP; Thu, 24 Mar 2011 01:51:02 -0700 (PDT)
In-Reply-To: <4D8B041A.7020007@gmail.com>
References: <4D8B041A.7020007@gmail.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Thu, 24 Mar 2011 14:21:02 +0530
Message-ID: <AANLkTik-165whbDrGD1Znes3FZ2fyOGcYZDy=COTxrVH@mail.gmail.com>
Subject: Re: Problem with NameNode Storage: IMAGE_AND_EDITS Failed
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Hello,

Can you verify and confirm if that location (is it the only one?) is a
valid one (as in, has free space left, is not corrupt, etc.)?

Take a backup of whatever exists at your dfs.name.dir before you
proceed doing anything.

On Thu, Mar 24, 2011 at 2:13 PM, Wojciech Langiewicz
<wlangiewicz@gmail.com> wrote:
> Hello,
> Right now I'm having this issue on my cluster:
> On NameNode web interface at the bottom of the page is table:
> NameNode Storage
> Storage Directory =A0 =A0 =A0 Type =A0 =A0State
> /srv/dfs/name =A0 IMAGE_AND_EDITS Failed
>
> I have had this issue before, and I rebooted Hadoop, but lost many files.
> What can I do now that I won't lose files since last backup, and what thi=
s
> warning actually means?
> Any help would be greatly appreciated.
> --
> Wojciech Langiewicz
>



--=20
Harsh J
http://harshj.com

From general-return-3194-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 09:02:09 2011
Return-Path: <general-return-3194-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 79814 invoked from network); 24 Mar 2011 09:02:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 09:02:08 -0000
Received: (qmail 23152 invoked by uid 500); 24 Mar 2011 09:02:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23097 invoked by uid 500); 24 Mar 2011 09:02:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23089 invoked by uid 99); 24 Mar 2011 09:02:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 09:02:07 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of wlangiewicz@gmail.com designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 09:01:58 +0000
Received: by ewy22 with SMTP id 22so3029833ewy.35
        for <general@hadoop.apache.org>; Thu, 24 Mar 2011 02:01:38 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:message-id:date:from:user-agent:mime-version:to
         :subject:references:in-reply-to:content-type
         :content-transfer-encoding;
        bh=aCOrFItOjYPByRMaaoPZkC27PQKZa1d9SV336lBhEKs=;
        b=HD7xOu8mFz2lm2/u08Xdv/todUXjqpMxmR2clL8bHUK6DzU7Kx6fbsmw3QZG1qzG5O
         CKLiE5oMFpgDsMUX/rEwJH2Eb4M4vXoKzFDDIHBqlffnQP6cLSPJ2m8b46iWSQCAEx/8
         +gTmpVDCNAVxLmqbC8BO0kxrCvZkAp1GppsLA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:content-type:content-transfer-encoding;
        b=GgVCSBdaCN13oE3PjWqkXmrw7XD6j1KSRCY/9XiK4NVJUgTPeUsKfUio645ktz0M5h
         +w5dCDGeGWVtech4go19vWYHJVDbT3cAjJKO4wZHVCrPZerZAw9S/UAZWK6aNkv1sdqP
         /exD3aRZslIP0iCTjI31+uUq61OKE2ge93P4Y=
Received: by 10.213.16.199 with SMTP id p7mr171789eba.13.1300957297812;
        Thu, 24 Mar 2011 02:01:37 -0700 (PDT)
Received: from [172.19.4.124] (static.nk-net.pl [195.88.186.3])
        by mx.google.com with ESMTPS id b52sm4312347eei.22.2011.03.24.02.01.36
        (version=SSLv3 cipher=OTHER);
        Thu, 24 Mar 2011 02:01:37 -0700 (PDT)
Message-ID: <4D8B086F.9030708@gmail.com>
Date: Thu, 24 Mar 2011 10:01:35 +0100
From: Wojciech Langiewicz <wlangiewicz@gmail.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.14) Gecko/20110223 Thunderbird/3.1.8
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Problem with NameNode Storage: IMAGE_AND_EDITS Failed
References: <4D8B041A.7020007@gmail.com> <AANLkTik-165whbDrGD1Znes3FZ2fyOGcYZDy=COTxrVH@mail.gmail.com>
In-Reply-To: <AANLkTik-165whbDrGD1Znes3FZ2fyOGcYZDy=COTxrVH@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hello,
Yes, it's the only one, it has free space, and is not corrupted locally. 
I have found this exception in namenode logs:
2011-03-24 09:46:51,531 WARN org.mortbay.log: /getimage: 
java.io.IOException: GetImage failed. java.lang.NullPointerException
         at 
org.apache.hadoop.hdfs.server.namenode.FSImage.getImageFile(FSImage.java:219)
         at 
org.apache.hadoop.hdfs.server.namenode.FSImage.getFsImageName(FSImage.java:1584)
         at 
org.apache.hadoop.hdfs.server.namenode.GetImageServlet$1.run(GetImageServlet.java:75)
         at 
org.apache.hadoop.hdfs.server.namenode.GetImageServlet$1.run(GetImageServlet.java:70)
         at java.security.AccessController.doPrivileged(Native Method)
         at javax.security.auth.Subject.doAs(Subject.java:396)
         at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
         at 
org.apache.hadoop.hdfs.server.namenode.GetImageServlet.doGet(GetImageServlet.java:70)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
         at 
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
         at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1124)
         at 
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:826)
         at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1115)
         at 
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:361)
         at 
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
         at 
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
         at 
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
         at 
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
         at 
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
         at 
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
         at org.mortbay.jetty.Server.handle(Server.java:324)
         at 
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
         at 
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
         at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
         at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
         at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
         at 
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
         at 
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)

It appears there every 5 minutes.

On 24.03.2011 09:51, Harsh J wrote:
> Hello,
>
> Can you verify and confirm if that location (is it the only one?) is a
> valid one (as in, has free space left, is not corrupt, etc.)?
>
> Take a backup of whatever exists at your dfs.name.dir before you
> proceed doing anything.
>
> On Thu, Mar 24, 2011 at 2:13 PM, Wojciech Langiewicz
> <wlangiewicz@gmail.com>  wrote:
>> Hello,
>> Right now I'm having this issue on my cluster:
>> On NameNode web interface at the bottom of the page is table:
>> NameNode Storage
>> Storage Directory       Type    State
>> /srv/dfs/name   IMAGE_AND_EDITS Failed
>>
>> I have had this issue before, and I rebooted Hadoop, but lost many files.
>> What can I do now that I won't lose files since last backup, and what this
>> warning actually means?
>> Any help would be greatly appreciated.
>> --
>> Wojciech Langiewicz
>>
>
>
>


From general-return-3195-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 09:26:31 2011
Return-Path: <general-return-3195-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59506 invoked from network); 24 Mar 2011 09:26:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 09:26:31 -0000
Received: (qmail 45959 invoked by uid 500); 24 Mar 2011 09:26:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45912 invoked by uid 500); 24 Mar 2011 09:26:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45904 invoked by uid 99); 24 Mar 2011 09:26:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 09:26:29 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of wlangiewicz@gmail.com designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 09:26:22 +0000
Received: by ewy22 with SMTP id 22so3036643ewy.35
        for <general@hadoop.apache.org>; Thu, 24 Mar 2011 02:26:01 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:message-id:date:from:user-agent:mime-version:to
         :subject:references:in-reply-to:content-type
         :content-transfer-encoding;
        bh=wctRIIpik535e5Izw5UIbJg70Fb6I1IxOV5nKlgk+6w=;
        b=IAIrb12MCcVldgzsxfEHeG4512EfaJa6w4cZyMg0CtuvL5iRXdS2qaZL+SQgWZa566
         2o42gau5vzI21jP6Bn/0v6AUIQWVFJ+rBt/Kak5ZJ4/81OcWIbLTyvtwT5yxqKnFqee0
         97elOoXU1ql2GAuiDrSyXIp7AfEoqYBUEub8Q=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:content-type:content-transfer-encoding;
        b=ew+b/zoZIBFXDzuJ70a5yfGsd54MkYQFrMe8HRj0PVqiWDcI09rADDtWnr0TcQmIEY
         As2eQY2Zjgi8fPFhb8Nvfdx/XPeN+o4vC3sijMgAkzqN7sAql9z/pWv0V1wnQ/cBLw7U
         i2+hDeGAs+f7s6FGGCTY5JEQx6PxqD/4TeYZU=
Received: by 10.213.111.11 with SMTP id q11mr3827392ebp.19.1300958760794;
        Thu, 24 Mar 2011 02:26:00 -0700 (PDT)
Received: from [172.19.4.124] (static.nk-net.pl [195.88.186.3])
        by mx.google.com with ESMTPS id b52sm4329520eei.1.2011.03.24.02.25.59
        (version=SSLv3 cipher=OTHER);
        Thu, 24 Mar 2011 02:26:00 -0700 (PDT)
Message-ID: <4D8B0E27.6080909@gmail.com>
Date: Thu, 24 Mar 2011 10:25:59 +0100
From: Wojciech Langiewicz <wlangiewicz@gmail.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.14) Gecko/20110223 Thunderbird/3.1.8
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Problem with NameNode Storage: IMAGE_AND_EDITS Failed
References: <4D8B041A.7020007@gmail.com> <AANLkTik-165whbDrGD1Znes3FZ2fyOGcYZDy=COTxrVH@mail.gmail.com> <4D8B086F.9030708@gmail.com>
In-Reply-To: <4D8B086F.9030708@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Before that there's another exception which shows that someone must have 
changed setting from 64k  back to 1024.

So probably I would like to know if there is a command to manually 
trigger saving metadata from NN that after rebooting everything should 
be ok.

java.io.IOException: Too many open files
         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
         at 
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
         at 
org.mortbay.jetty.nio.SelectChannelConnector$1.acceptChannel(SelectChannelConnector.java:75)
         at 
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
         at 
org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:185)
         at 
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
         at 
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:707)
         at 
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)

On 24.03.2011 10:01, Wojciech Langiewicz wrote:
> Hello,
> Yes, it's the only one, it has free space, and is not corrupted locally.
> I have found this exception in namenode logs:
> 2011-03-24 09:46:51,531 WARN org.mortbay.log: /getimage:
> java.io.IOException: GetImage failed. java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.getImageFile(FSImage.java:219)
>
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.getFsImageName(FSImage.java:1584)
>
> at
> org.apache.hadoop.hdfs.server.namenode.GetImageServlet$1.run(GetImageServlet.java:75)
>
> at
> org.apache.hadoop.hdfs.server.namenode.GetImageServlet$1.run(GetImageServlet.java:70)
>
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>
> at
> org.apache.hadoop.hdfs.server.namenode.GetImageServlet.doGet(GetImageServlet.java:70)
>
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1124)
>
> at
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:826)
>
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1115)
>
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:361)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
>
> It appears there every 5 minutes.
>
> On 24.03.2011 09:51, Harsh J wrote:
>> Hello,
>>
>> Can you verify and confirm if that location (is it the only one?) is a
>> valid one (as in, has free space left, is not corrupt, etc.)?
>>
>> Take a backup of whatever exists at your dfs.name.dir before you
>> proceed doing anything.
>>
>> On Thu, Mar 24, 2011 at 2:13 PM, Wojciech Langiewicz
>> <wlangiewicz@gmail.com> wrote:
>>> Hello,
>>> Right now I'm having this issue on my cluster:
>>> On NameNode web interface at the bottom of the page is table:
>>> NameNode Storage
>>> Storage Directory Type State
>>> /srv/dfs/name IMAGE_AND_EDITS Failed
>>>
>>> I have had this issue before, and I rebooted Hadoop, but lost many
>>> files.
>>> What can I do now that I won't lose files since last backup, and what
>>> this
>>> warning actually means?
>>> Any help would be greatly appreciated.
>>> --
>>> Wojciech Langiewicz
>>>
>>
>>
>>
>


From general-return-3196-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 23:30:00 2011
Return-Path: <general-return-3196-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 65380 invoked from network); 24 Mar 2011 23:30:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 23:30:00 -0000
Received: (qmail 41105 invoked by uid 500); 24 Mar 2011 23:29:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 41058 invoked by uid 500); 24 Mar 2011 23:29:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 41050 invoked by uid 99); 24 Mar 2011 23:29:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 23:29:58 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of owen.omalley@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 23:29:51 +0000
Received: by wyb40 with SMTP id 40so638394wyb.35
        for <general@hadoop.apache.org>; Thu, 24 Mar 2011 16:29:31 -0700 (PDT)
Received: by 10.216.61.65 with SMTP id v43mr1185683wec.84.1301009371360;
        Thu, 24 Mar 2011 16:29:31 -0700 (PDT)
Received: from [10.253.240.139] ([84.233.227.242])
        by mx.google.com with ESMTPS id c54sm146098wer.30.2011.03.24.16.29.29
        (version=SSLv3 cipher=OTHER);
        Thu, 24 Mar 2011 16:29:30 -0700 (PDT)
Message-Id: <8A6FDBD1-0B3D-4EEA-9792-058FEDC37575@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=Apple-Mail-2-579622931
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Hadoop project wins MediaGuardian Innovation Award
Date: Thu, 24 Mar 2011 23:29:29 +0000
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-2-579622931
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit

The Hadoop project won the top MediaGuardian Innovation award. The  
article is below:

Groundbreaking open source project wins top MediaGuardian Innovation  
Award.

All of the contributors should be very proud of this award. Three  
members of the Hadoop Project Management Committee were there to  
receive the award on behalf of the project.

I've been working on Hadoop full time since the beginning and it has  
been a pleasure working with such bright and dedicated engineers. It  
takes a village to raise an elephant from a prototype that runs on a  
few nodes to the project that is disrupting the big data industry.

Congratulations Hadoop team!

-- Owen


--Apple-Mail-2-579622931--

From general-return-3197-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 25 06:03:32 2011
Return-Path: <general-return-3197-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42158 invoked from network); 25 Mar 2011 06:03:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Mar 2011 06:03:32 -0000
Received: (qmail 7126 invoked by uid 500); 25 Mar 2011 06:03:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7052 invoked by uid 500); 25 Mar 2011 06:03:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7042 invoked by uid 99); 25 Mar 2011 06:03:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 06:03:29 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of weishung@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 06:03:23 +0000
Received: by bwz8 with SMTP id 8so1092618bwz.35
        for <general@hadoop.apache.org>; Thu, 24 Mar 2011 23:03:02 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to:cc
         :content-type;
        bh=U1/DyoKlIoqBueDEbkoH//QtodZVaLPsOQzYg0PXNU4=;
        b=fbN0ooHkFsSz1Rd7K4W1uXK0EtRiFjpaOUi08dZEZrx+obNNuOGeifYlymuTEZkF8J
         Pcg05qNJOk5vhI6LCuLlsxzwNVcbRNI4r+rWmNrpPVWv9ghJxzRmUSgyk6QdxCGDQHfE
         DejK7BC02CweZFN2m3P8j0LVbHPWrhDscNYoI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:cc:content-type;
        b=bHjwXl02b3UPjjp+NGFUpOnc7F8/c27TKB/NqHCSY3qW/L052PE5nDcal4WX5QvJyQ
         ZFthkSMI9YsNpp/sQvxSNtG86VzQUNTbOO9Ix/n9WrNQiw0Xru1SLpv8dY2+F7UihIZr
         tM9rqMzp/8lJL/iyxlnGzz5lCPtVrwGM3g0vA=
MIME-Version: 1.0
Received: by 10.204.127.29 with SMTP id e29mr322376bks.52.1301032982245; Thu,
 24 Mar 2011 23:03:02 -0700 (PDT)
Received: by 10.204.20.1 with HTTP; Thu, 24 Mar 2011 23:03:02 -0700 (PDT)
Date: Fri, 25 Mar 2011 01:03:02 -0500
Message-ID: <AANLkTim95vVNKVXyF7r4-CEX3HGWEfi6H3L1ZP6+oLex@mail.gmail.com>
Subject: congrats !!!
From: Weishung Chung <weishung@gmail.com>
To: general@hadoop.apache.org
Cc: Doug Cutting <cutting@apache.org>, omalley@apache.org, Stack <saint.ack@gmail.com>, 
	jdcryans@apache.org
Content-Type: multipart/alternative; boundary=0016e6dab507bf718b049f48585f

--0016e6dab507bf718b049f48585f
Content-Type: text/plain; charset=ISO-8859-1

My fellow Apache Hadoop developers,

Congratulations for winning the top prize at the 2011 MediaGuardian
Innovation Awards !

I don't know how you all do this, but I am truly impressed with your works
and having so fun browsing and learning from your beautiful piece of art.

Thank you again for always answering my questions on the mailing lists :D
Wei Shung

--0016e6dab507bf718b049f48585f--

From general-return-3198-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 25 10:37:27 2011
Return-Path: <general-return-3198-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29291 invoked from network); 25 Mar 2011 10:37:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Mar 2011 10:37:27 -0000
Received: (qmail 30365 invoked by uid 500); 25 Mar 2011 10:37:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29661 invoked by uid 500); 25 Mar 2011 10:37:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29495 invoked by uid 99); 25 Mar 2011 10:37:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 10:37:24 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [85.214.115.60] (HELO h1349259.stratoserver.net) (85.214.115.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 10:37:18 +0000
Received: from isabel-drost.de ([85.214.115.60] helo=motokokusanaghi.localnet)
	by h1349259.stratoserver.net with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
	(Exim 4.69)
	(envelope-from <isabel@apache.org>)
	id 1Q34O1-0005lb-Im; Fri, 25 Mar 2011 10:36:57 +0000
From: Isabel Drost <isabel@apache.org>
Reply-To: isabel@apache.org
To: general@hadoop.apache.org
Subject: BerlinBuzzwords 2011 Early Bird Ticket Period ends on April 7th.
Date: Fri, 25 Mar 2011 03:36:55 -0700
User-Agent: KMail/1.13.5 (Linux/2.6.32-5-amd64; KDE/4.4.5; x86_64; ; )
Cc: user@mahout.apache.org,
 user@hbase.apache.org
MIME-Version: 1.0
Content-Type: multipart/signed;
  boundary="nextPart3911021.xfUZaYUefh";
  protocol="application/pgp-signature";
  micalg=pgp-sha1
Content-Transfer-Encoding: 7bit
Message-Id: <201103250336.56455.isabel@apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

--nextPart3911021.xfUZaYUefh
Content-Type: Text/Plain;
  charset="us-ascii"
Content-Transfer-Encoding: 7bit


Hey folks,

just a short notice for those who haven't noticed we have only a
limited amount of Early-Bird tickets left and the Early-Bird period is
ends on April 7th. If you want to get one of the 30 remaining tickets
go and get one now here: http://berlinbuzzwords.de/content/tickets

While we are still working on the schedule and selecting speakers we
didn't send out any reject mail yet. So if you have submitted a talk
for BerlinBuzzwords 2011 you don't need to get a Early-Bird ticket
now. All potential speakers will be eligible for Early-Bird discount
even after April 7th.


regards,

Isabel

--nextPart3911021.xfUZaYUefh
Content-Type: application/pgp-signature; name=signature.asc 
Content-Description: This is a digitally signed message part.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEABECAAYFAk2McEgACgkQPJpCBAwIhbRpcgCfQt2qll68aQffEICMNR6asyKO
wlUAniVq0gcPuyC2Ec1avTWvjRpfPaM5
=2OgN
-----END PGP SIGNATURE-----

--nextPart3911021.xfUZaYUefh--

From general-return-3199-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 25 13:37:04 2011
Return-Path: <general-return-3199-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9405 invoked from network); 25 Mar 2011 13:37:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Mar 2011 13:37:03 -0000
Received: (qmail 75715 invoked by uid 500); 25 Mar 2011 13:37:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75657 invoked by uid 500); 25 Mar 2011 13:37:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75642 invoked by uid 99); 25 Mar 2011 13:37:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 13:37:02 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of binhara@gmail.com designates 74.125.82.42 as permitted sender)
Received: from [74.125.82.42] (HELO mail-ww0-f42.google.com) (74.125.82.42)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 13:36:56 +0000
Received: by wwk4 with SMTP id 4so9175181wwk.5
        for <multiple recipients>; Fri, 25 Mar 2011 06:36:33 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=OUMN1q6CNY2ObOvaPliG4BKqsqP5Mdk+DOnTytM+aJs=;
        b=xRbDqu8Y+rqjqs04RBZ0X/76VIfHfuiD1nkafR5RwprBXuu07ACYGWmmt5c9RynB3l
         qEcyR1mPbTRe/N5qZfLllbBdbmAiHY+OhraYMQ/plG3qLx2tqt2392TeJLn071zqF0JM
         mVLZge5p77yEjS+tg8daMF27R9vxxB7z2b/Qw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=nGUfoTH6sG0KR6+aQh1PQrsVdsbTgvHwb9+SplWrpE4345RT0JxsVAOtSO5q2dHWqx
         UG9lG2ljTeq9a2C7Tgo3EN+JLBe90C9r72EC2yG8BefJAs1uUclJLiD+WUsLKrqqObof
         372StbvePUeSsRdPtwLyz9mp+mEnGeGQZ+zII=
MIME-Version: 1.0
Received: by 10.216.64.139 with SMTP id c11mr1879923wed.46.1301060192900; Fri,
 25 Mar 2011 06:36:32 -0700 (PDT)
Received: by 10.216.21.82 with HTTP; Fri, 25 Mar 2011 06:36:32 -0700 (PDT)
Date: Fri, 25 Mar 2011 10:36:32 -0300
Message-ID: <AANLkTik0VqtU0bb_o+5BLWNvw1wTyudCqNRSww8oPRx0@mail.gmail.com>
Subject: problem with writeUTF
From: Alessandro Binhara <binhara@gmail.com>
To: cdh-user@cloudera.org, general@hadoop.apache.org, 
	hdfs-user@hadoop.apache.org
Content-Type: multipart/mixed; boundary=000e0cd56c5aa11936049f4eaeeb

--000e0cd56c5aa11936049f4eaeeb
Content-Type: multipart/alternative; boundary=000e0cd56c5aa1191e049f4eaee9

--000e0cd56c5aa1191e049f4eaee9
Content-Type: text/plain; charset=ISO-8859-1

Hello all ..

I write some programs to hadoop here .. and i had a problem with writeUTF
..
It put a null in every line begin .

See in attachement ..

Whats happen? How to remove this null ?

thanks

--000e0cd56c5aa1191e049f4eaee9
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hello all ..=A0<div><br></div><div>I write some programs to hadoop here .. =
and i had a problem with writeUTF ..=A0</div><div>It put a null in every li=
ne begin .=A0</div><div><br></div><div>See in attachement ..=A0</div><div><=
br></div>
<div>Whats happen? How to remove this null ?=A0</div><div><br></div><div>th=
anks=A0</div><div><br></div>

--000e0cd56c5aa1191e049f4eaee9--
--000e0cd56c5aa11936049f4eaeeb--

From general-return-3200-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 25 17:08:03 2011
Return-Path: <general-return-3200-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 26402 invoked from network); 25 Mar 2011 17:08:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Mar 2011 17:08:03 -0000
Received: (qmail 73489 invoked by uid 500); 25 Mar 2011 17:08:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73405 invoked by uid 500); 25 Mar 2011 17:08:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73389 invoked by uid 99); 25 Mar 2011 17:08:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 17:08:00 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of binhara@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 17:07:53 +0000
Received: by wyb40 with SMTP id 40so1734981wyb.35
        for <multiple recipients>; Fri, 25 Mar 2011 10:07:32 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:cc:content-type;
        bh=zD20KBQP8zKMNNv20kTNsbsabkx2o9eUjTaxmiv9k28=;
        b=yEafrKNkAN1ragXybtL5Ym57C4R1flrV8mTt8kfqVG7I/+Tqo7xq/HyIx/UiTHQEuB
         Dk9yZ7NMNwaHXfXGixxdGgql/ZzzgU0FN8Wpw+7hkKCyr8/VVtS7QppadQBnQjfvDAeM
         m3aM1j2H57lIlv5jwr8tUPfL2zLLx/GCbYOyA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=cVE+bE3Dl7Rq8zCq1ZDMH1WNK3pX+Hyx6qguiqwDj7hUl9vXmI3ne0+LWshbz9Pl+u
         VOmlyEB6C9q0mwaxJenZ1b9poQCbCMxPW9K8OuOhiHH/F+bAyiOyDIhgQAKi2E5j+eEr
         7KOD2HftDvPnRk1RrQxQHBhfb28lNi/znWFf8=
MIME-Version: 1.0
Received: by 10.216.64.139 with SMTP id c11mr2087797wed.46.1301072818931; Fri,
 25 Mar 2011 10:06:58 -0700 (PDT)
Received: by 10.216.21.82 with HTTP; Fri, 25 Mar 2011 10:06:58 -0700 (PDT)
In-Reply-To: <A2B4D100-89E6-4511-8843-ED32D43F68D3@gmail.com>
References: <AANLkTik0VqtU0bb_o+5BLWNvw1wTyudCqNRSww8oPRx0@mail.gmail.com>
	<A2B4D100-89E6-4511-8843-ED32D43F68D3@gmail.com>
Date: Fri, 25 Mar 2011 14:06:58 -0300
Message-ID: <AANLkTikUEiMdYvtS5ONBGRC_CMLS_OHUAy9PLPsoNsWM@mail.gmail.com>
Subject: Re: problem with writeUTF
From: Alessandro Binhara <binhara@gmail.com>
To: cdh-user@cloudera.org
Cc: Mapred Learn <mapred.learn@gmail.com>, 
	"hdfs-user@hadoop.apache.org" <hdfs-user@hadoop.apache.org>, 
	"general@hadoop.apache.org" <general@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=000e0cd56c5a330a91049f519f37
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd56c5a330a91049f519f37
Content-Type: text/plain; charset=ISO-8859-1

OHOHOHO Thanks ..
It s working now!!!!



On Fri, Mar 25, 2011 at 11:22 AM, Mapred Learn <mapred.learn@gmail.com>wrote:

> I also had this problem And ended up switching to :
> Write instead of writeUTF:
> Write(outputBytes, 0, outputBytes.length)
>
> Sent from my iPhone
>
> On Mar 25, 2011, at 6:36 AM, Alessandro Binhara <binhara@gmail.com> wrote:
>
> > Hello all ..
> >
> > I write some programs to hadoop here .. and i had a problem with writeUTF
> ..
> > It put a null in every line begin .
> >
> > See in attachement ..
> >
> > Whats happen? How to remove this null ?
> >
> > thanks
> >
> > <imagenull.png>
>

--000e0cd56c5a330a91049f519f37--

From general-return-3201-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 25 20:11:58 2011
Return-Path: <general-return-3201-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23850 invoked from network); 25 Mar 2011 20:11:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Mar 2011 20:11:57 -0000
Received: (qmail 49426 invoked by uid 500); 25 Mar 2011 20:11:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49375 invoked by uid 500); 25 Mar 2011 20:11:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49367 invoked by uid 99); 25 Mar 2011 20:11:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 20:11:56 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of binhara@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 20:11:48 +0000
Received: by wwi18 with SMTP id 18so1231928wwi.29
        for <general@hadoop.apache.org>; Fri, 25 Mar 2011 13:11:28 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=01XFGNYp+uH8Bd4scG9ZxuYEaBIESaOEOGFN2TGMfH8=;
        b=MDb+jd6B6J7lhemYTsgA9Jhtmzq1xg8C621+UEZQI5XtL3EKlZqmYfyxd/nWJtc/QT
         VP5L91dkWsoUf0Jzr+0vZa7CKWZQw0Zp5WO+uunHUpeJNk7sV4X4wUpCiq4zyFIpOeKz
         TmIqWV14QrySO8E4mY7mf2xLd9G5+kr6HZ4h0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=JWIjLuAGWk7GCYXMRHOsLIPhfEDkFH8vZBife2xWDCvZ3SH9Au3dTLoCjwT0w5eNfB
         5NaotIJZ/sYuVT8SGEfSgv4FbXVfx/yf6BTCA8wrSXszqPKsp4msrPKs1cwLrND/7M1W
         Uw2R0068SHO8YkJ/RlyO24aUACB6QgaEMf5Co=
MIME-Version: 1.0
Received: by 10.216.200.82 with SMTP id y60mr1121735wen.31.1301083888178; Fri,
 25 Mar 2011 13:11:28 -0700 (PDT)
Received: by 10.216.21.82 with HTTP; Fri, 25 Mar 2011 13:11:28 -0700 (PDT)
Date: Fri, 25 Mar 2011 17:11:28 -0300
Message-ID: <AANLkTi=6c1XHiTuTdCpQRkvoLNK5FpJe=7gqJ=npH=Ux@mail.gmail.com>
Subject: NameNode is not formatted.
From: Alessandro Binhara <binhara@gmail.com>
To: cdh-user@cloudera.org, general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6dab59efa5613049f54320e
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6dab59efa5613049f54320e
Content-Type: text/plain; charset=ISO-8859-1

Helo ... all...


I chance in hdfs-site.xml data dir , like this :

<property>
  <name>dfs.name.dir</name>
  <value>/var/hadoop/</value>
 </property>

<property>
  <name>dfs.data.dir</name>
  <value>/var/hadoop/</value>
 </property>

i try format a hdfs with
 su -s /bin/bash - hdfs -c 'hadoop namenode -format'

wel.. it ask if i would like RE Format a hdfs :
-->>> Re-format filesystem in /var/hadoop ? (Y or N)

I sad YES... but not happing .
I see a permission dir on /var/hadoop
drwxrwxrwx  2 root root  4096 2011-03-25 20:05 hadoop

Then a try run a name-node and i receive a message :

java.io.IOException: NameNode is not formatted.


Whats happen ???

thanks

--0016e6dab59efa5613049f54320e--

From general-return-3202-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 28 01:06:03 2011
Return-Path: <general-return-3202-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27943 invoked from network); 28 Mar 2011 01:06:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Mar 2011 01:06:02 -0000
Received: (qmail 61374 invoked by uid 500); 28 Mar 2011 01:06:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61300 invoked by uid 500); 28 Mar 2011 01:06:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61292 invoked by uid 99); 28 Mar 2011 01:06:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 01:06:00 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of kntreadway@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 01:05:54 +0000
Received: by bwz8 with SMTP id 8so3218952bwz.35
        for <general@hadoop.apache.org>; Sun, 27 Mar 2011 18:05:33 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:cc:content-type;
        bh=9gwNJ/izRd32xo/fhTk/h/rG4mCqNMd2DVRz9BamjEg=;
        b=msP9R0wP+ZpHBcIEghsKM+U0lFjBN+gHjJlk83DMejRzXpFll0tT4NG1QWC6sDOxD0
         9L6bmqJ7hz4NM+GH+E6aFI8GgmiAWG7T7aW0ilB1gSX/TQJB2nvpOaHxF60xlavpSx2g
         osgRYyISueWQ9Uem/UyaeZe7Z3Vc65ibEI050=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :cc:content-type;
        b=SlAxu1a0HPTcptoPrQSBVUojOBJ9JyiEcuDGc0N9lWLKS/+hl8a4hICnUSMtTt38au
         h/EPlZ8cTv637bl6vhamd/nBMxBAcbkFRkN9BvrUP4wu3CsrvWfhm01CliOns0LR9iMo
         mma20vC8nMJWckvQAa+RsxpBSFVtbzbI0b9+g=
Received: by 10.204.34.73 with SMTP id k9mr3231934bkd.189.1301274332995; Sun,
 27 Mar 2011 18:05:32 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.54.130 with HTTP; Sun, 27 Mar 2011 18:05:09 -0700 (PDT)
In-Reply-To: <AANLkTi=6c1XHiTuTdCpQRkvoLNK5FpJe=7gqJ=npH=Ux@mail.gmail.com>
References: <AANLkTi=6c1XHiTuTdCpQRkvoLNK5FpJe=7gqJ=npH=Ux@mail.gmail.com>
From: Nichole Treadway <kntreadway@gmail.com>
Date: Sun, 27 Mar 2011 21:05:09 -0400
Message-ID: <AANLkTim59Yh5iMgyFZiYeUDn=F5SjV_VzjM43-r-Hgdo@mail.gmail.com>
Subject: Re: NameNode is not formatted.
To: general@hadoop.apache.org
Cc: Alessandro Binhara <binhara@gmail.com>, cdh-user@cloudera.org
Content-Type: multipart/alternative; boundary=000325559e265fa032049f808a58

--000325559e265fa032049f808a58
Content-Type: text/plain; charset=ISO-8859-1

Try changing dfs.data.dir value to "/var/hadoop/dfs/data" and dfs.name.dir
to "/var/hadoop/dfs/name". Create this /var/hadoop/dfs directory with the
appropriate permissions. Re-format again and see if that helps.

On Fri, Mar 25, 2011 at 4:11 PM, Alessandro Binhara <binhara@gmail.com>wrote:

> Helo ... all...
>
>
> I chance in hdfs-site.xml data dir , like this :
>
> <property>
>  <name>dfs.name.dir</name>
>  <value>/var/hadoop/</value>
>  </property>
>
> <property>
>  <name>dfs.data.dir</name>
>  <value>/var/hadoop/</value>
>  </property>
>
> i try format a hdfs with
>  su -s /bin/bash - hdfs -c 'hadoop namenode -format'
>
> wel.. it ask if i would like RE Format a hdfs :
> -->>> Re-format filesystem in /var/hadoop ? (Y or N)
>
> I sad YES... but not happing .
> I see a permission dir on /var/hadoop
> drwxrwxrwx  2 root root  4096 2011-03-25 20:05 hadoop
>
> Then a try run a name-node and i receive a message :
>
> java.io.IOException: NameNode is not formatted.
>
>
> Whats happen ???
>
> thanks
>

--000325559e265fa032049f808a58--

From general-return-3203-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 28 18:28:52 2011
Return-Path: <general-return-3203-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97791 invoked from network); 28 Mar 2011 18:28:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Mar 2011 18:28:51 -0000
Received: (qmail 75864 invoked by uid 500); 28 Mar 2011 18:28:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75814 invoked by uid 500); 28 Mar 2011 18:28:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75801 invoked by uid 99); 28 Mar 2011 18:28:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 18:28:49 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of owen.omalley@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 18:28:40 +0000
Received: by vws7 with SMTP id 7so3856335vws.35
        for <general@hadoop.apache.org>; Mon, 28 Mar 2011 11:28:19 -0700 (PDT)
Received: by 10.52.73.196 with SMTP id n4mr5945030vdv.227.1301336899749;
        Mon, 28 Mar 2011 11:28:19 -0700 (PDT)
Received: from [10.72.184.42] (nat-dip4.cfw-a-gci.corp.yahoo.com [209.131.62.113])
        by mx.google.com with ESMTPS id y15sm776043vch.5.2011.03.28.11.28.17
        (version=SSLv3 cipher=OTHER);
        Mon, 28 Mar 2011 11:28:18 -0700 (PDT)
Cc: general@hadoop.apache.org
Message-Id: <4C3CD35C-B9DB-47ED-9B0E-BEDB869C7670@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: Owen O'Malley <omalley@apache.org>
In-Reply-To: <8A6FDBD1-0B3D-4EEA-9792-058FEDC37575@apache.org>
Content-Type: multipart/alternative; boundary=Apple-Mail-2-907149698
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Hadoop project wins MediaGuardian Innovation Award
Date: Mon, 28 Mar 2011 11:28:16 -0700
References: <8A6FDBD1-0B3D-4EEA-9792-058FEDC37575@apache.org>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-2-907149698
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed
Content-Transfer-Encoding: 7bit


On Mar 24, 2011, at 4:29 PM, Owen O'Malley wrote:

> The Hadoop project won the top MediaGuardian Innovation award.

I've put up some pictures of the event at:

http://developer.yahoo.com/blogs/hadoop/posts/2011/03/hadoop-project-wins-innovation-award/

-- Owen
--Apple-Mail-2-907149698--

From general-return-3204-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 29 06:28:34 2011
Return-Path: <general-return-3204-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6575 invoked from network); 29 Mar 2011 06:28:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Mar 2011 06:28:34 -0000
Received: (qmail 81271 invoked by uid 500); 29 Mar 2011 06:28:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81181 invoked by uid 500); 29 Mar 2011 06:28:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81173 invoked by uid 99); 29 Mar 2011 06:28:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 06:28:31 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of shv.hadoop@gmail.com designates 209.85.161.176 as permitted sender)
Received: from [209.85.161.176] (HELO mail-gx0-f176.google.com) (209.85.161.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 06:28:25 +0000
Received: by gxk7 with SMTP id 7so1882604gxk.35
        for <general@hadoop.apache.org>; Mon, 28 Mar 2011 23:28:04 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=aVyyAqxYMZcUIn4H7DUBuAovldRiyVaYBSkwoQD0JqU=;
        b=HX7CJZpNKaihbZ3ksr1UIaDEyxIu+K1JS2XdmXf1FQ5qWwcgFSJbFJSr3IkD8vitzU
         ss9zALEEX0d4lFlkxYHZuvI+PL0oNVmQ2dQKrmi61o+7DjskItzaNpPdgHlATLzjk1rI
         P4vajS7n/fMYLqkV6AsSEaTmjG/ZVovF+AIrI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=bcBHOkkx8n4ORIr7hknsaraMwBbMRl+FApGSjG9yxr3Q4YF7okPghMokIrza2EhZI2
         MZxOTd3BO/7w/igKnaou/aRXNJZobE94MSZvj25IMYnBml4RV/PJqRzOw/0goNNYoOyI
         D0YSHDTSMeFBy+UIq/D/6MjWkR9pAeRkE7i0c=
MIME-Version: 1.0
Received: by 10.236.182.138 with SMTP id o10mr6648365yhm.65.1301380083990;
 Mon, 28 Mar 2011 23:28:03 -0700 (PDT)
Received: by 10.146.168.19 with HTTP; Mon, 28 Mar 2011 23:28:03 -0700 (PDT)
In-Reply-To: <4C3CD35C-B9DB-47ED-9B0E-BEDB869C7670@apache.org>
References: <8A6FDBD1-0B3D-4EEA-9792-058FEDC37575@apache.org>
	<4C3CD35C-B9DB-47ED-9B0E-BEDB869C7670@apache.org>
Date: Mon, 28 Mar 2011 23:28:03 -0700
Message-ID: <AANLkTiktfx6y7pBpkFHG8CbJhFkiSy7XKt6i9WRLhCSg@mail.gmail.com>
Subject: Re: Hadoop project wins MediaGuardian Innovation Award
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf305b12909fc1e6049f99290b

--20cf305b12909fc1e6049f99290b
Content-Type: text/plain; charset=ISO-8859-1

I like that small "Fire exit" sign in the corner of the first picture. :-)

Congratulations to everybody!

--Konstantin


On Mon, Mar 28, 2011 at 11:28 AM, Owen O'Malley <omalley@apache.org> wrote:

>
> On Mar 24, 2011, at 4:29 PM, Owen O'Malley wrote:
>
>  The Hadoop project won the top MediaGuardian Innovation award.
>>
>
> I've put up some pictures of the event at:
>
>
> http://developer.yahoo.com/blogs/hadoop/posts/2011/03/hadoop-project-wins-innovation-award/
>
> -- Owen

--20cf305b12909fc1e6049f99290b--

From general-return-3205-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 29 09:32:43 2011
Return-Path: <general-return-3205-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22298 invoked from network); 29 Mar 2011 09:32:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Mar 2011 09:32:43 -0000
Received: (qmail 98200 invoked by uid 500); 29 Mar 2011 09:32:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98138 invoked by uid 500); 29 Mar 2011 09:32:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98130 invoked by uid 99); 29 Mar 2011 09:32:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 09:32:41 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 09:32:34 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 24860B7F34
	for <general@hadoop.apache.org>; Tue, 29 Mar 2011 10:32:12 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id wtroA2qHZyuJ for <general@hadoop.apache.org>;
	Tue, 29 Mar 2011 10:32:12 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 163D1B7F85
	for <general@hadoop.apache.org>; Tue, 29 Mar 2011 10:32:11 +0100 (BST)
MailScanner-NULL-Check: 1301995920.41091@qc1GDzkO9src4I0555gcDQ
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p2T9Vx27000400
	for <general@hadoop.apache.org>; Tue, 29 Mar 2011 10:31:59 +0100 (BST)
Message-ID: <4D91A70F.9020604@apache.org>
Date: Tue, 29 Mar 2011 10:31:59 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.14) Gecko/20110223 Thunderbird/3.1.8
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Hadoop project wins MediaGuardian Innovation Award
References: <8A6FDBD1-0B3D-4EEA-9792-058FEDC37575@apache.org> <4C3CD35C-B9DB-47ED-9B0E-BEDB869C7670@apache.org>
In-Reply-To: <4C3CD35C-B9DB-47ED-9B0E-BEDB869C7670@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p2T9Vx27000400
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 28/03/11 19:28, Owen O'Malley wrote:
>
> On Mar 24, 2011, at 4:29 PM, Owen O'Malley wrote:
>
>> The Hadoop project won the top MediaGuardian Innovation award.
>
> I've put up some pictures of the event at:
>
> http://developer.yahoo.com/blogs/hadoop/posts/2011/03/hadoop-project-wins-innovation-award/
>

Plus some more of Owen the next day holding the award and pointing at a 
branch diagram that scares everyone
http://www.flickr.com/photos/steve_l/sets/72157626356732562/

From general-return-3206-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 29 17:34:04 2011
Return-Path: <general-return-3206-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96305 invoked from network); 29 Mar 2011 17:34:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Mar 2011 17:34:03 -0000
Received: (qmail 87664 invoked by uid 500); 29 Mar 2011 17:34:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87580 invoked by uid 500); 29 Mar 2011 17:34:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87572 invoked by uid 99); 29 Mar 2011 17:34:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 17:34:01 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jdcryans@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 17:33:57 +0000
Received: by fxm7 with SMTP id 7so629506fxm.35
        for <general@hadoop.apache.org>; Tue, 29 Mar 2011 10:33:36 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=F3otnSY+AFzUGa2RJccVICOCHt0wI5qdrMsde/5mjRM=;
        b=C2zzcEw2PobwyURTjPrZNhb07c/O5SV3yoABUT+j1c00UM4TmWCqTTKIuEVUo/+h73
         xyG0/no6p2UqLd7kaPxYHp3tftVl47JlHfv+/Y0aDg/IgKKCAt+QkUx4pSUjkPa4xTU5
         PzBZPj9WbF5GI8IK+T1AOwiJ1Td5JYV6X6Ixc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=tarXbCzcHTSuLNL9ufyO3GnXxD9ncrBi0/v0YBH7X5c6ntk+O+VgQgRtsphVv3dwZ6
         nI1I7k194/jl14ssAFUF4q9KvRF/TbIznUYPiFwjXnBtAg9TC9Ni3ravIWmUObwE19d9
         fXPr/5SEd15WnaOcaFy3pDcKOq9pTFri0+JKg=
MIME-Version: 1.0
Received: by 10.223.66.200 with SMTP id o8mr83976fai.38.1301420016294; Tue, 29
 Mar 2011 10:33:36 -0700 (PDT)
Sender: jdcryans@gmail.com
Received: by 10.223.25.82 with HTTP; Tue, 29 Mar 2011 10:33:36 -0700 (PDT)
In-Reply-To: <46ED7CAE-5DE6-4453-8E04-802055EBE379@tynt.com>
References: <46ED7CAE-5DE6-4453-8E04-802055EBE379@tynt.com>
Date: Tue, 29 Mar 2011 10:33:36 -0700
X-Google-Sender-Auth: U205U_BbRd65e9w8okZk0B55JK4
Message-ID: <AANLkTikg03bDbxqXrbzbN3LTzBJycBRG1yQO8F1Wrx=T@mail.gmail.com>
Subject: Re: Hadoop in Canada
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

(moving to general@ since this is not a question regarding the usage
of the hadoop commons, which I BCC'd)

I moved from Montreal to SF a year and a half ago because I saw two
things 1) companies weren't interested (they are still trying to get
rid of COBOL or worse) or didn't have the data to use Hadoop (not
enough big companies) and 2) the universities were either uninterested
or just amused by this "new comer". I know of one company that really
does cool stuff with Hadoop in Montreal and it's Hopper
(www.hopper.travel, they are still in closed alpha AFAIK) who also
organized hackreduce.org last weekend. This is what their CEO has to
say to the question "Is there something you would do differently now
if you would start it over?":

Move to the Valley.

(see the rest here
http://nextmontreal.com/product-market-fit-hopper-travel-fred-lalonde/)

I'm sure there are a lot of other companies that are either
considering using or already using Hadoop to some extent in Canada
but, like anything else, only a portion of them are interested in
talking about it or even organizing an event.

I would actually love to see something getting organized and I'd be on
the first plane to Y**, but I'm afraid that to achieve any sort of
critical mass you'd have to fly in people from all the provinces. Air
Canada becomes a SPOF :P

Now that I think about it, there's probably enough Canucks around here
that use Hadoop that we could have our own little user group. If you
want to have a nice vacation and geek out with us, feel free to stop
by and say hi.

</rant>

J-D

On Tue, Mar 29, 2011 at 6:21 AM, James Seigel <james@tynt.com> wrote:
> Hello,
>
> You might remember me from a couple of weeks back asking if there were an=
y Calgary people interested in a =93meetup=94 about #bigdata or using hadoo=
p. =A0Well, I=92ve expanded my search a little to see if any of my Canadian=
 brothers and sisters are using the elephant for good or for evil. =A0It mi=
ght be harder to grab coffee, but it would be fun to see where everyone is.
>
> Shout out if you=92d like or ping me, I think it=92d be fun to chat!
>
> Cheers
> James Seigel
> Captain Hammer at Tynt.com

From general-return-3207-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 29 18:43:52 2011
Return-Path: <general-return-3207-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31045 invoked from network); 29 Mar 2011 18:43:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Mar 2011 18:43:52 -0000
Received: (qmail 76855 invoked by uid 500); 29 Mar 2011 18:43:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76797 invoked by uid 500); 29 Mar 2011 18:43:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76789 invoked by uid 99); 29 Mar 2011 18:43:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 18:43:50 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of msegel@navteq.com designates 204.120.70.37 as permitted sender)
Received: from [204.120.70.37] (HELO xmailfargo.navteq.com) (204.120.70.37)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 18:43:44 +0000
Received: from fargo-spamfw.navteq.com (navteq-spamfw.navteq.com [10.6.10.131])
	by xmailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id p2TIBGKP017858
	for <general@hadoop.apache.org>; Tue, 29 Mar 2011 13:11:16 -0500
Received: from imailfargo.navteq.com (localhost [127.0.0.1])
	by fargo-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id 7D2A44804F15
	for <general@hadoop.apache.org>; Tue, 29 Mar 2011 13:43:22 -0500 (CDT)
Received: from imailfargo.navteq.com (imailfargo.navteq.com [10.6.10.130]) by fargo-spamfw.navteq.com with ESMTP id CZyWSluNkQOxREiN for <general@hadoop.apache.org>; Tue, 29 Mar 2011 13:43:22 -0500 (CDT)
Received: from hq-ex-ht02.ad.navteq.com ([10.8.222.172])
	by imailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id p2TIhM1V025565
	for <general@hadoop.apache.org>; Tue, 29 Mar 2011 13:43:22 -0500
Received: from hq-ex-mb02.ad.navteq.com ([fe80::880d:318e:e860:a8db]) by
 hq-ex-ht02.ad.navteq.com ([fe80::24f6:ca50:734d:df6e%13]) with mapi; Tue, 29
 Mar 2011 13:43:21 -0500
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 29 Mar 2011 13:43:20 -0500
Subject: RE: Hadoop in Canada
Thread-Topic: Hadoop in Canada
Thread-Index: AcvuN4Cw1h7ekSp+RGKdSJJwBxydWgACYCWQ
Message-ID: <ABC24175AFD3BE4DA15F4CD375ED413D06BA841DF4@hq-ex-mb02.ad.navteq.com>
References: <46ED7CAE-5DE6-4453-8E04-802055EBE379@tynt.com>
 <AANLkTikg03bDbxqXrbzbN3LTzBJycBRG1yQO8F1Wrx=T@mail.gmail.com>
In-Reply-To: <AANLkTikg03bDbxqXrbzbN3LTzBJycBRG1yQO8F1Wrx=T@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1
X-Virus-Checked: Checked by ClamAV on apache.org

RnVubnkuLi4NCkJ1dCB0aGUgQ09CT0wgbW9kZWwga2luZCBvZiBsZW5kcyBpdHNlbGYgdG8gSGFk
b29wLiANCkl0IHdvdWxkIGJlIGludGVyZXN0aW5nIHRvIHdvcmsgb24gYSBtaWdyYXRpb24gcHJv
amVjdC4uLg0KDQoNCi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQpGcm9tOiBqZGNyeWFuc0Bn
bWFpbC5jb20gW21haWx0bzpqZGNyeWFuc0BnbWFpbC5jb21dIE9uIEJlaGFsZiBPZiBKZWFuLURh
bmllbCBDcnlhbnMNClNlbnQ6IFR1ZXNkYXksIE1hcmNoIDI5LCAyMDExIDEyOjM0IFBNDQpUbzog
Z2VuZXJhbEBoYWRvb3AuYXBhY2hlLm9yZw0KU3ViamVjdDogUmU6IEhhZG9vcCBpbiBDYW5hZGEN
Cg0KKG1vdmluZyB0byBnZW5lcmFsQCBzaW5jZSB0aGlzIGlzIG5vdCBhIHF1ZXN0aW9uIHJlZ2Fy
ZGluZyB0aGUgdXNhZ2UNCm9mIHRoZSBoYWRvb3AgY29tbW9ucywgd2hpY2ggSSBCQ0MnZCkNCg0K
SSBtb3ZlZCBmcm9tIE1vbnRyZWFsIHRvIFNGIGEgeWVhciBhbmQgYSBoYWxmIGFnbyBiZWNhdXNl
IEkgc2F3IHR3bw0KdGhpbmdzIDEpIGNvbXBhbmllcyB3ZXJlbid0IGludGVyZXN0ZWQgKHRoZXkg
YXJlIHN0aWxsIHRyeWluZyB0byBnZXQNCnJpZCBvZiBDT0JPTCBvciB3b3JzZSkgb3IgZGlkbid0
IGhhdmUgdGhlIGRhdGEgdG8gdXNlIEhhZG9vcCAobm90DQplbm91Z2ggYmlnIGNvbXBhbmllcykg
YW5kIDIpIHRoZSB1bml2ZXJzaXRpZXMgd2VyZSBlaXRoZXIgdW5pbnRlcmVzdGVkDQpvciBqdXN0
IGFtdXNlZCBieSB0aGlzICJuZXcgY29tZXIiLiBJIGtub3cgb2Ygb25lIGNvbXBhbnkgdGhhdCBy
ZWFsbHkNCmRvZXMgY29vbCBzdHVmZiB3aXRoIEhhZG9vcCBpbiBNb250cmVhbCBhbmQgaXQncyBI
b3BwZXINCih3d3cuaG9wcGVyLnRyYXZlbCwgdGhleSBhcmUgc3RpbGwgaW4gY2xvc2VkIGFscGhh
IEFGQUlLKSB3aG8gYWxzbw0Kb3JnYW5pemVkIGhhY2tyZWR1Y2Uub3JnIGxhc3Qgd2Vla2VuZC4g
VGhpcyBpcyB3aGF0IHRoZWlyIENFTyBoYXMgdG8NCnNheSB0byB0aGUgcXVlc3Rpb24gIklzIHRo
ZXJlIHNvbWV0aGluZyB5b3Ugd291bGQgZG8gZGlmZmVyZW50bHkgbm93DQppZiB5b3Ugd291bGQg
c3RhcnQgaXQgb3Zlcj8iOg0KDQpNb3ZlIHRvIHRoZSBWYWxsZXkuDQoNCihzZWUgdGhlIHJlc3Qg
aGVyZQ0KaHR0cDovL25leHRtb250cmVhbC5jb20vcHJvZHVjdC1tYXJrZXQtZml0LWhvcHBlci10
cmF2ZWwtZnJlZC1sYWxvbmRlLykNCg0KSSdtIHN1cmUgdGhlcmUgYXJlIGEgbG90IG9mIG90aGVy
IGNvbXBhbmllcyB0aGF0IGFyZSBlaXRoZXINCmNvbnNpZGVyaW5nIHVzaW5nIG9yIGFscmVhZHkg
dXNpbmcgSGFkb29wIHRvIHNvbWUgZXh0ZW50IGluIENhbmFkYQ0KYnV0LCBsaWtlIGFueXRoaW5n
IGVsc2UsIG9ubHkgYSBwb3J0aW9uIG9mIHRoZW0gYXJlIGludGVyZXN0ZWQgaW4NCnRhbGtpbmcg
YWJvdXQgaXQgb3IgZXZlbiBvcmdhbml6aW5nIGFuIGV2ZW50Lg0KDQpJIHdvdWxkIGFjdHVhbGx5
IGxvdmUgdG8gc2VlIHNvbWV0aGluZyBnZXR0aW5nIG9yZ2FuaXplZCBhbmQgSSdkIGJlIG9uDQp0
aGUgZmlyc3QgcGxhbmUgdG8gWSoqLCBidXQgSSdtIGFmcmFpZCB0aGF0IHRvIGFjaGlldmUgYW55
IHNvcnQgb2YNCmNyaXRpY2FsIG1hc3MgeW91J2QgaGF2ZSB0byBmbHkgaW4gcGVvcGxlIGZyb20g
YWxsIHRoZSBwcm92aW5jZXMuIEFpcg0KQ2FuYWRhIGJlY29tZXMgYSBTUE9GIDpQDQoNCk5vdyB0
aGF0IEkgdGhpbmsgYWJvdXQgaXQsIHRoZXJlJ3MgcHJvYmFibHkgZW5vdWdoIENhbnVja3MgYXJv
dW5kIGhlcmUNCnRoYXQgdXNlIEhhZG9vcCB0aGF0IHdlIGNvdWxkIGhhdmUgb3VyIG93biBsaXR0
bGUgdXNlciBncm91cC4gSWYgeW91DQp3YW50IHRvIGhhdmUgYSBuaWNlIHZhY2F0aW9uIGFuZCBn
ZWVrIG91dCB3aXRoIHVzLCBmZWVsIGZyZWUgdG8gc3RvcA0KYnkgYW5kIHNheSBoaS4NCg0KPC9y
YW50Pg0KDQpKLUQNCg0KT24gVHVlLCBNYXIgMjksIDIwMTEgYXQgNjoyMSBBTSwgSmFtZXMgU2Vp
Z2VsIDxqYW1lc0B0eW50LmNvbT4gd3JvdGU6DQo+IEhlbGxvLA0KPg0KPiBZb3UgbWlnaHQgcmVt
ZW1iZXIgbWUgZnJvbSBhIGNvdXBsZSBvZiB3ZWVrcyBiYWNrIGFza2luZyBpZiB0aGVyZSB3ZXJl
IGFueSBDYWxnYXJ5IHBlb3BsZSBpbnRlcmVzdGVkIGluIGEgIm1lZXR1cCIgYWJvdXQgI2JpZ2Rh
dGEgb3IgdXNpbmcgaGFkb29wLiCgV2VsbCwgSSd2ZSBleHBhbmRlZCBteSBzZWFyY2ggYSBsaXR0
bGUgdG8gc2VlIGlmIGFueSBvZiBteSBDYW5hZGlhbiBicm90aGVycyBhbmQgc2lzdGVycyBhcmUg
dXNpbmcgdGhlIGVsZXBoYW50IGZvciBnb29kIG9yIGZvciBldmlsLiCgSXQgbWlnaHQgYmUgaGFy
ZGVyIHRvIGdyYWIgY29mZmVlLCBidXQgaXQgd291bGQgYmUgZnVuIHRvIHNlZSB3aGVyZSBldmVy
eW9uZSBpcy4NCj4NCj4gU2hvdXQgb3V0IGlmIHlvdSdkIGxpa2Ugb3IgcGluZyBtZSwgSSB0aGlu
ayBpdCdkIGJlIGZ1biB0byBjaGF0IQ0KPg0KPiBDaGVlcnMNCj4gSmFtZXMgU2VpZ2VsDQo+IENh
cHRhaW4gSGFtbWVyIGF0IFR5bnQuY29tDQoNCg0KVGhlIGluZm9ybWF0aW9uIGNvbnRhaW5lZCBp
biB0aGlzIGNvbW11bmljYXRpb24gbWF5IGJlIENPTkZJREVOVElBTCBhbmQgaXMgaW50ZW5kZWQg
b25seSBmb3IgdGhlIHVzZSBvZiB0aGUgcmVjaXBpZW50KHMpIG5hbWVkIGFib3ZlLiAgSWYgeW91
IGFyZSBub3QgdGhlIGludGVuZGVkIHJlY2lwaWVudCwgeW91IGFyZSBoZXJlYnkgbm90aWZpZWQg
dGhhdCBhbnkgZGlzc2VtaW5hdGlvbiwgZGlzdHJpYnV0aW9uLCBvciBjb3B5aW5nIG9mIHRoaXMg
Y29tbXVuaWNhdGlvbiwgb3IgYW55IG9mIGl0cyBjb250ZW50cywgaXMgc3RyaWN0bHkgcHJvaGli
aXRlZC4gIElmIHlvdSBoYXZlIHJlY2VpdmVkIHRoaXMgY29tbXVuaWNhdGlvbiBpbiBlcnJvciwg
cGxlYXNlIG5vdGlmeSB0aGUgc2VuZGVyIGFuZCBkZWxldGUvZGVzdHJveSB0aGUgb3JpZ2luYWwg
bWVzc2FnZSBhbmQgYW55IGNvcHkgb2YgaXQgZnJvbSB5b3VyIGNvbXB1dGVyIG9yIHBhcGVyIGZp
bGVzLg0K

From general-return-3208-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 01:46:23 2011
Return-Path: <general-return-3208-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14179 invoked from network); 31 Mar 2011 01:46:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 01:46:23 -0000
Received: (qmail 28433 invoked by uid 500); 30 Mar 2011 21:42:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28366 invoked by uid 500); 30 Mar 2011 21:42:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28358 invoked by uid 99); 30 Mar 2011 21:42:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Mar 2011 21:42:22 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates 209.85.210.176 as permitted sender)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Mar 2011 21:42:16 +0000
Received: by iym1 with SMTP id 1so2478153iym.35
        for <general@hadoop.apache.org>; Wed, 30 Mar 2011 14:41:55 -0700 (PDT)
Received: by 10.42.150.65 with SMTP id z1mr1643762icv.503.1301521315081; Wed,
 30 Mar 2011 14:41:55 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.42.220.194 with HTTP; Wed, 30 Mar 2011 14:41:35 -0700 (PDT)
From: Todd Lipcon <todd@cloudera.com>
Date: Wed, 30 Mar 2011 14:41:35 -0700
Message-ID: <AANLkTi=owD6w2UA+-eCdDP67F=2NMChHLJxi1HYY+RG6@mail.gmail.com>
Subject: Maintenance of Hadoop 0.21 branch?
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba1efe86a72ff0049fba0bd3

--90e6ba1efe86a72ff0049fba0bd3
Content-Type: text/plain; charset=ISO-8859-1

Hi all,

Some recent discussion on HDFS-1786 has raised an interesting question: does
anyone plan on maintaining the 0.21 branch and eventually releasing an
0.21.1? Should we bother to commit bug fixes to this branch?

It seems to me that our time would be better spent getting 0.22 and trunk
back to a green state so we can talk about releasing them, rather than
applying patches to a branch with no releases planned.

Of course a decision now doesn't preclude anyone from stepping up later,
backporting patches to 0.21, and releasing an 0.21.1.

Thanks
-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

--90e6ba1efe86a72ff0049fba0bd3--

From general-return-3209-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 01:55:01 2011
Return-Path: <general-return-3209-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50175 invoked from network); 31 Mar 2011 01:55:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 01:55:01 -0000
Received: (qmail 42647 invoked by uid 500); 30 Mar 2011 21:51:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42579 invoked by uid 500); 30 Mar 2011 21:51:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42571 invoked by uid 99); 30 Mar 2011 21:51:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Mar 2011 21:51:00 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [216.252.110.212] (HELO web56203.mail.re3.yahoo.com) (216.252.110.212)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 30 Mar 2011 21:50:52 +0000
Received: (qmail 15108 invoked by uid 60001); 30 Mar 2011 21:50:29 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1301521829; bh=t7eNBSzIWrZUjKhdH2rsaEmDHIHIJOT2YoyIRP9zvHU=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=HU1ZYlgpt9zBbNO3HlbzsHfMgA/diXm4zxeJKWOeojTMDowZHrDAqkgJlBrdsdH+2C0GBmMoLde3XBvyTFT8jyHEU4wexCGPU5rXDxixE9bcjGuHczP1GIxQAhWakFQTDmARDl6gxXK4ZBYhStXqbKyNcJMg7vRSg5VLxK00anw=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=cZD/2hxMKG20KrG5ttQjDdszQI5BUhOAR8s80wsj2vab7h4rYvn1xpmpfhNsAQpi4I+LVWQ7+1r7lFbdYvYQ1hY0wpjNT7icneg55pW4ro7RxHLyTHqEbVrBpZThKDdRCa0R+Z6HJ/x1rmpdEiU710yaMz6FXhI7F0jhZeFqbWs=;
Message-ID: <332918.89435.qm@web56203.mail.re3.yahoo.com>
X-YMail-OSG: Qu5xDBMVM1kmKPdveI3KJt5mX9zhNZe2jSvqPcBPeq0sgF3
 nxJPhHIt.6IU6rqdDDutOtAJO.XAnqzmvx2dpe2N.CXPSxNqeFFc4FWv4LuM
 t5T.JsL2bVo0bzxdBkI1GyFS3CCplwSErJXEyB2xqcJo74T3EnK08IjK_gDg
 vFBWI8SYXL3lfy1WsC3EWm4A7M9FpOJDp_0n.QtGmeFWi9hCrZopXqv5jBTd
 _W9ST51j0lYfXQGoAIAOCX6SJxr8cDDh7eHe5Z5W8HM1Rr1zuq_xhoCKHpHq
 dL8tZfoej1oyjdXKL51KYwic9pQeEVSXaeouBVsqslezvROtoOARRwDm75Tn
 PhV5.wpdo4Zzc9yELRoEuGb90OhZdMM3266g4Dxioxk18Wr3UM.DAPcVad7C
 NInlXuVSRwzV_ZfoTUhECK.KIl5hZBk3j9w--
Received: from [209.131.62.113] by web56203.mail.re3.yahoo.com via HTTP; Wed, 30 Mar 2011 14:50:29 PDT
X-Mailer: YahooMailRC/559 YahooMailWebService/0.8.109.295617
References: <AANLkTi=owD6w2UA+-eCdDP67F=2NMChHLJxi1HYY+RG6@mail.gmail.com>
Date: Wed, 30 Mar 2011 14:50:29 -0700 (PDT)
From: "Tsz Wo \(Nicholas\), Sze" <s29752-hadoopgeneral@yahoo.com>
Subject: Re: Maintenance of Hadoop 0.21 branch?
To: general@hadoop.apache.org
In-Reply-To: <AANLkTi=owD6w2UA+-eCdDP67F=2NMChHLJxi1HYY+RG6@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-404604782-1301521829=:89435"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-404604782-1301521829=:89435
Content-Type: text/plain; charset=us-ascii

I recall that some users are using 0.21.   Should we discuss this on the users 
mailing lists?
Nicholas





________________________________
From: Todd Lipcon <todd@cloudera.com>
To: general@hadoop.apache.org
Sent: Wed, March 30, 2011 2:41:35 PM
Subject: Maintenance of Hadoop 0.21 branch?

Hi all,

Some recent discussion on HDFS-1786 has raised an interesting question: does
anyone plan on maintaining the 0.21 branch and eventually releasing an
0.21.1? Should we bother to commit bug fixes to this branch?

It seems to me that our time would be better spent getting 0.22 and trunk
back to a green state so we can talk about releasing them, rather than
applying patches to a branch with no releases planned.

Of course a decision now doesn't preclude anyone from stepping up later,
backporting patches to 0.21, and releasing an 0.21.1.

Thanks
-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

--0-404604782-1301521829=:89435--

From general-return-3212-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 07:50:42 2011
Return-Path: <general-return-3212-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42264 invoked from network); 31 Mar 2011 07:50:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 07:50:42 -0000
Received: (qmail 51332 invoked by uid 500); 31 Mar 2011 07:50:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51203 invoked by uid 500); 31 Mar 2011 07:50:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51195 invoked by uid 99); 31 Mar 2011 07:50:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 07:50:34 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [85.214.115.60] (HELO h1349259.stratoserver.net) (85.214.115.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 07:50:27 +0000
Received: from crt-01-tr.neofonie.de ([91.213.91.28] helo=essen.neofonie.priv)
	by h1349259.stratoserver.net with esmtpsa (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16)
	(Exim 4.69)
	(envelope-from <isabel@apache.org>)
	id 1Q5Cdo-000Fmm-HD
	for general@hadoop.apache.org; Thu, 31 Mar 2011 07:50:04 +0000
Date: Thu, 31 Mar 2011 09:42:36 +0200
From: Isabel Drost <isabel@apache.org>
To: general@hadoop.apache.org
Subject: Re: Hadoop in Canada
Message-ID: <20110331094236.5335a912@essen.neofonie.priv>
In-Reply-To: <AANLkTikg03bDbxqXrbzbN3LTzBJycBRG1yQO8F1Wrx=T@mail.gmail.com>
References: <46ED7CAE-5DE6-4453-8E04-802055EBE379@tynt.com>
	<AANLkTikg03bDbxqXrbzbN3LTzBJycBRG1yQO8F1Wrx=T@mail.gmail.com>
X-Mailer: Claws Mail 3.7.2 (GTK+ 2.14.4; x86_64-http://packman.links2linux.de-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

On Tue, 29 Mar 11 Jean-Daniel Cryans wrote:
> I would actually love to see something getting organized and I'd be on
> the first plane to Y**

Not exactly a meetup but Apache Con (<http://na11.apachecon.com/>) is
to take place in Vancouver, includes a "cloud computing" track focused
on Hadoop and friends, call for participation is open.

Isabel

From general-return-3300-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 14:56:06 2011
Return-Path: <general-return-3300-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BE47A36C8
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  2 May 2011 14:56:06 +0000 (UTC)
Received: (qmail 52225 invoked by uid 500); 2 May 2011 14:56:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52080 invoked by uid 500); 2 May 2011 14:56:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52064 invoked by uid 99); 2 May 2011 14:56:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 14:56:04 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of gmackey@cs.ucf.edu designates 132.170.216.154 as permitted sender)
Received: from [132.170.216.154] (HELO longwood.cs.ucf.edu) (132.170.216.154)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 14:55:58 +0000
Received: from longwood.cs.ucf.edu (localhost [127.0.0.1])
	by longwood.cs.ucf.edu (8.14.1/8.14.1) with ESMTP id p42EtYKi007311;
	Mon, 2 May 2011 10:55:34 -0400 (EDT)
Received: (from prayer@localhost)
	by longwood.cs.ucf.edu (8.14.1/8.14.4/Submit) id p42EtX5O007309;
	Mon, 2 May 2011 10:55:33 -0400 (EDT)
X-Authentication-Warning: longwood.cs.ucf.edu: prayer set sender to gmackey@mail.cs.ucf.edu using -f
Received: from [10.173.214.212] by mail.cs.ucf.edu
	with HTTP (Prayer-1.3.2); 02 May 2011 10:55:33 -0400
Date: 02 May 2011 10:55:33 -0400
From: gmackey@cs.ucf.edu
To: general@hadoop.apache.org
Cc: common-user@hadoop.apache.org
Subject: Re: questions about hadoop map reduce and compute intensive related applications
Message-ID: <Prayer.1.3.2.1105021055330.6905@longwood.cs.ucf.edu>
In-Reply-To: <BANLkTi=2EaAmxaibRPiX=AEkVeL8XKmdvA@mail.gmail.com>
References: <BANLkTinbKUikZGx_c9h6oN=_yveF8j2K6Q@mail.gmail.com>
 <BANLkTi=2EaAmxaibRPiX=AEkVeL8XKmdvA@mail.gmail.com>
X-Mailer: Prayer v1.3.2
Mime-Version: 1.0
Content-Type: text/plain; format=flowed; charset=ISO-8859-1
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on gondor.cs.ucf.edu
X-Virus-Scanned: clamav-milter 0.97 at longwood
X-Virus-Status: Clean
X-Virus-Checked: Checked by ClamAV on apache.org
X-Old-Spam-Status: No, score=-1.0 required=6.0 tests=ALL_TRUSTED
	autolearn=unavailable version=3.3.1

 
> Integrating MPI with map-reduce is currently difficult and/or very ugly, 
> however. Not impossible and there are hackish ways to do the job, but 
> they are hacks.

There is an project out of Sandia National Lab that puts MR and MPI 
together in a library if you're interested --> 
http://www.sandia.gov/~sjplimp/mapreduce.html

The project isn't mature yet, and I haven't actually used it myself.

-- 
--
Grant Mackey
PhD student Computer Engineering
University of Central Florida

From general-return-3301-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 19:03:31 2011
Return-Path: <general-return-3301-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 36A5F380D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  2 May 2011 19:03:31 +0000 (UTC)
Received: (qmail 16023 invoked by uid 500); 2 May 2011 19:03:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15921 invoked by uid 500); 2 May 2011 19:03:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15913 invoked by uid 99); 2 May 2011 19:03:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 19:03:29 +0000
X-ASF-Spam-Status: No, hits=3.3 required=5.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 19:03:23 +0000
Received: from sp1-ex07cas01.ds.corp.yahoo.com (sp1-ex07cas01.ds.corp.yahoo.com [216.252.116.137])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p42J2sVP084661
	for <general@hadoop.apache.org>; Mon, 2 May 2011 12:02:54 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1304362974;
	bh=2Tq7Ut2yQaddpQ7ojPId/LK/m3O+Fekw71ucVq0bcwU=;
	h=From:To:Date:Subject:Message-ID:Content-Type:MIME-Version;
	b=f8YYQM25x7zhvV03zU/Y7kCZadrK20xzvZfDug96hce/A0e+6yqp5b068cKiLKMGe
	 cvAjo+8/VHLaTlCVm4vmAIAy10SnFNine8X63mpq3MLQ2ULNoTGf+9MMgEf4mBhyR2
	 HjcFaD1SAA1zxgHsWlvhuN8R6s6WvcFwmfO4zVl0=
Received: from SP1-EX07VS02.ds.corp.yahoo.com ([216.252.116.135]) by
 sp1-ex07cas01.ds.corp.yahoo.com ([216.252.116.137]) with mapi; Mon, 2 May
 2011 12:02:53 -0700
From: Avik Dey <avik@yahoo-inc.com>
To: "'general@hadoop.apache.org'" <general@hadoop.apache.org>
Date: Mon, 2 May 2011 12:02:52 -0700
Subject: [hadoop] Hadoop Summit 2011 by Yahoo!: June 29th, Santa Clara
 Convention Center. Register and submit abstract for presentation:
 www.hadoopsummit.org
Thread-Topic: [hadoop] Hadoop Summit 2011 by Yahoo!: June 29th, Santa Clara
 Convention Center. Register and submit abstract for presentation:
 www.hadoopsummit.org
Thread-Index: Acv00YctyjaYKtraRSmrGT2TejdLtw==
Message-ID: <2931D8EDF4B12743B93584669F76353330A03F46EE@SP1-EX07VS02.ds.corp.yahoo.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_2931D8EDF4B12743B93584669F76353330A03F46EESP1EX07VS02ds_"
MIME-Version: 1.0

--_000_2931D8EDF4B12743B93584669F76353330A03F46EESP1EX07VS02ds_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi Folks,

The Hadoop Summit 2011 abstract submission closes on May 6th i.e. this Frid=
ay.

I know there are still some of you that are working on the presentation abs=
tract for the Summit, thought I would send a gentle reminder. :-)

See you at the Summit.

Avik


--_000_2931D8EDF4B12743B93584669F76353330A03F46EESP1EX07VS02ds_--

From general-return-3302-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 19:16:24 2011
Return-Path: <general-return-3302-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C7FBA35CF
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  2 May 2011 19:16:24 +0000 (UTC)
Received: (qmail 47724 invoked by uid 500); 2 May 2011 19:16:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47646 invoked by uid 500); 2 May 2011 19:16:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47638 invoked by uid 99); 2 May 2011 19:16:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 19:16:23 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 19:16:17 +0000
Received: by qwj9 with SMTP id 9so3970322qwj.35
        for <general@hadoop.apache.org>; Mon, 02 May 2011 12:15:55 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=joDKblDQ8s/8sS5Ne4SChIBt93VTUyHdh/DIZ/ZdAFI=;
        b=a1qnq7Bn3j3IfYInRlvR8ERXv/ynWw6YysS066aiMFI7sojpyA9jqcHOS6MrudX1U4
         97e4PavDXKxo+fEdSzuuR9iGK3TNMSnXfV+xapR8wMzpAsac0HwIzIsBqAmymQUXyoQJ
         ruQWu6p+tML9JjLnXpZwXOCEOBt1eACUVZ2zY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=tlqs7HJSU2NrHeF8sV7iNbM46Vas7SeFWWAoJ57OhwctHNUpyI3ETUNsxt20NdlNA4
         t1j+S8OoPXzzdtfrvQ+JdheDx2I0V88QQcGBxdcBrYFlQXSZcs1FU1tX8Lln88SVbnQU
         f+xz3xX3HDL1wnBQjhyludlEYvVSPib4uER3Y=
Received: by 10.224.28.193 with SMTP id n1mr6596938qac.225.1304363755705;
        Mon, 02 May 2011 12:15:55 -0700 (PDT)
Received: from [10.172.0.214] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id t17sm4177050qcs.11.2011.05.02.12.15.52
        (version=TLSv1/SSLv3 cipher=OTHER);
        Mon, 02 May 2011 12:15:54 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Discussions - Re: [VOTE] Release candidate 0.20.203.0-rc0
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <4DBEF0C1.2050306@apache.org>
Date: Tue, 3 May 2011 05:15:48 +1000
Content-Transfer-Encoding: quoted-printable
Message-Id: <05E53D57-D53A-4D05-AAA4-E99A12373ED2@Holsman.NET>
References: <12853C59-08F1-4F98-ACF3-800F06BC7A5F@apache.org> <4DBEF0C1.2050306@apache.org>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)

moving this thread to general@

On May 3, 2011, at 3:58 AM, Doug Cutting wrote:

>> Should we release
>> http://people.apache.org/~omalley/hadoop-0.20.203.0-rc0/?
>=20
> The patch selection process for this branch did not appear to be a
> community process.  A massive patch set was committed en-masse with no
> public discussion before or after about its specific composition.

guys...
1. do we agree this is an issue
2. if it is, how we do get the communication & discussion on list?

what do people think are the major issues that are stopping people =
talking about stuff on list are?=

From general-return-3303-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 20:06:11 2011
Return-Path: <general-return-3303-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5940E3DCE
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  2 May 2011 20:06:11 +0000 (UTC)
Received: (qmail 11871 invoked by uid 500); 2 May 2011 20:06:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11823 invoked by uid 500); 2 May 2011 20:06:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11809 invoked by uid 99); 2 May 2011 20:06:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 20:06:09 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 20:06:03 +0000
Received: from SP2-EX07CAS02.ds.corp.yahoo.com (sp2-ex07cas02.corp.sp2.yahoo.com [98.137.59.38])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p42K5P1c025481;
	Mon, 2 May 2011 13:05:25 -0700 (PDT)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by
 SP2-EX07CAS02.ds.corp.yahoo.com ([98.137.59.38]) with mapi; Mon, 2 May 2011
 13:05:25 -0700
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>,
        "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>
Date: Mon, 2 May 2011 13:05:14 -0700
Subject: Re: Discussions - Re: [VOTE] Release candidate 0.20.203.0-rc0
Thread-Topic: Discussions - Re: [VOTE] Release candidate 0.20.203.0-rc0
Thread-Index: AcwJBEYcLmBlmTsNT4u2XhdmLVQJTQ==
Message-ID: <38AE939A-0717-4220-A6F0-6093FF852060@yahoo-inc.com>
References: <12853C59-08F1-4F98-ACF3-800F06BC7A5F@apache.org>
 <4DBEF0C1.2050306@apache.org>
 <05E53D57-D53A-4D05-AAA4-E99A12373ED2@Holsman.NET>
In-Reply-To: <05E53D57-D53A-4D05-AAA4-E99A12373ED2@Holsman.NET>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


Hi folks,

This strikes me as a bit odd. I think we have already discussed this at len=
gth and agreed that a release could proceed.=20

Since then, Arun and Owen have worked actively to incorporated community fe=
edback into this release.=20

All parties making Hadoop releases other then Apache have already incorpora=
ted most of the patches in this release into their products, including doug=
's organization. I don't see how Hadoop's users benefit from Apache not inc=
orporating them into an Apache release.=20

As previously discussed, all parties are welcome to champion altenative rel=
eases from Apache if they want to invest in making Apache Hadoop better.

Thanks!!

E14

---
E14 - typing on glass

On May 2, 2011, at 12:16 PM, "Ian Holsman" <hadoop@holsman.net> wrote:

> moving this thread to general@
>=20
> On May 3, 2011, at 3:58 AM, Doug Cutting wrote:
>=20
>>> Should we release
>>> http://people.apache.org/~omalley/hadoop-0.20.203.0-rc0/?
>>=20
>> The patch selection process for this branch did not appear to be a
>> community process.  A massive patch set was committed en-masse with no
>> public discussion before or after about its specific composition.
>=20
> guys...
> 1. do we agree this is an issue
> 2. if it is, how we do get the communication & discussion on list?
>=20
> what do people think are the major issues that are stopping people talkin=
g about stuff on list are?

From general-return-3304-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 20:07:59 2011
Return-Path: <general-return-3304-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F383B3E9D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  2 May 2011 20:07:58 +0000 (UTC)
Received: (qmail 18043 invoked by uid 500); 2 May 2011 20:07:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17980 invoked by uid 500); 2 May 2011 20:07:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17964 invoked by uid 99); 2 May 2011 20:07:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 20:07:57 +0000
X-ASF-Spam-Status: No, hits=2.9 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 20:07:50 +0000
Received: from l2tp-8-66.corp.yahoo.com (l2tp-8-66.corp.yahoo.com [10.73.8.66])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p42K7GE3026130;
	Mon, 2 May 2011 13:07:17 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1304366837;
	bh=XKd0OT5gZEohdElsaM5iRFORVVSpRM2/KWpNUjoAHPk=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:Mime-Version:Subject:
	 Date:References;
	b=HTrU3fxc6SMVeFzcQUhUAtaP/xRtpRzu5qdSSz+SZ+bnQgdGVzztd/KKT45IQDpuc
	 6H34XvFdN/UgYdBShxKyasDsh93RZ3fhWpAG5TKx5u9webGF9W0w6FLKQmNMWSmJ4q
	 2YADKg8E2wmhxNHpDWTsoe4ceI3QyyhxhTKeGYgc=
Message-Id: <BB3A69B5-9A5A-4454-80C8-62EBD3CF47EC@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org, common-dev@hadoop.apache.org
In-Reply-To: <4DBEF0C1.2050306@apache.org>
Content-Type: multipart/alternative; boundary=Apple-Mail-9--357877675
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc0
Date: Mon, 2 May 2011 13:07:16 -0700
References: <12853C59-08F1-4F98-ACF3-800F06BC7A5F@apache.org> <4DBEF0C1.2050306@apache.org>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-9--357877675
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit

Doug,

On May 2, 2011, at 10:58 AM, Doug Cutting wrote:
> The patch selection process for this branch did not appear to be a
> community process.  A massive patch set was committed en-masse with no
> public discussion before or after about its specific composition.

Lets review:

# You proposed to release off the Yahoo security patchset first in  
April, 2010: http://s.apache.org/5Gv
# I started this discussion again in Jan, 2011: http://s.apache.org/uf
# We went through several iterations:
  - I first committed a jumbo patch upon which some reservations were  
expressed.
  - Owen went ahead and broke them up to commit individual patches to  
incorporate the provided feedback.
# Roy clearly clarified the way forward: http://s.apache.org/tD4  
(which Owen has since incorporatedk by breaking into individual  
patches).

Your current stance given the history, is surprising, to say the  
least... we have already discussed this. It is clear that the  
community (including downstream Apache projects like Pig, Hive and  
HCatalog) will substantially benefit from an Apache release of this  
improved codebase.

thanks,
Arun


--Apple-Mail-9--357877675--

From general-return-3305-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 20:40:49 2011
Return-Path: <general-return-3305-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 929C63DCA
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  2 May 2011 20:40:49 +0000 (UTC)
Received: (qmail 57130 invoked by uid 500); 2 May 2011 20:40:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57076 invoked by uid 500); 2 May 2011 20:40:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57068 invoked by uid 99); 2 May 2011 20:40:48 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 20:40:48 +0000
Received: from localhost (HELO [192.168.168.16]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 20:40:47 +0000
Message-ID: <4DBF16CF.3030905@apache.org>
Date: Mon, 02 May 2011 13:40:47 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.14) Gecko/20110223 Thunderbird/3.1.8
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc0
References: <12853C59-08F1-4F98-ACF3-800F06BC7A5F@apache.org> <4DBEF0C1.2050306@apache.org> <BB3A69B5-9A5A-4454-80C8-62EBD3CF47EC@yahoo-inc.com>
In-Reply-To: <BB3A69B5-9A5A-4454-80C8-62EBD3CF47EC@yahoo-inc.com>
X-Enigmail-Version: 1.1.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On 05/02/2011 01:07 PM, Arun C Murthy wrote:
> # Roy clearly clarified the way forward: http://s.apache.org/tD4 (which
> Owen has since incorporated by breaking into individual patches).

Roy suggested a three ways forward and possible outcomes:

Roy Fielding wrote:
>  a) break the changes down into a sequence of patches, create jira
>     issues for each one (or append to the existing issue), and then
>     provide the group with a list of the issue links so that people
>     can quickly +1 each one.  When it seems worthwhile to you, create
>     a branch off of some prior Apache release point in svn and commit
>     each patch to it until the branch is identical to (or, in your own
>     opinion, better than) the source code that you have tested locally.
>     Then RM a tarball and start a release vote.  Since all of this is
>     being done in jira and svn, others can help you do all but the
>     first part (breaking down the big patch).
> 
> or
> 
>  b) create a branch off of some prior Apache release point in svn
>     and replay the internal Y! commits on that branch until the branch
>     source code is identical to what you have tested locally.  Then
>     RM a tarball based on that branch and start a release vote.
>     Since the history is now in svn, others could do the RM bit if
>     you don't have time.
> 
> or
> 
>  c) create a branch off of some prior Apache release point in svn
>     and apply one big ugly patch to it.  Then RM a tarball based
>     on that branch and ask for a release vote.
> 
> You will note that none of the above requires a discussion on this
> list prior to the release vote, though (a) would likely result in
> more +1s than (b), and (b) would likely receive more +1s than (c).
> Regardless, the release vote is a lazy majority decision.
>
>  [ ... ]
>
> When the release vote happens, encourage folks to test and +1
> the release.  If it passes, woohoo!  If not, then listen to the
> reasons given by the other PMC members and see if you can make
> enough changes to the release to get those extra +1s.

I believe that Owen chose (b).  We're now at the release vote and I am a
PMC member giving reasons for my vote.

Also note that, on the common-dev thread, Eli & Tom have both noted a
number of inconsistencies between this set of patches and trunk, 0.22
and even prior 0.20 branches and releases.  In addition to the lack of
community involvement in patch selection, these issues concern me.

I cannot in good conscience vote for this release as a community product.

Doug

From general-return-3306-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 20:51:10 2011
Return-Path: <general-return-3306-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3F82C3397
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  2 May 2011 20:51:10 +0000 (UTC)
Received: (qmail 67126 invoked by uid 500); 2 May 2011 20:51:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67066 invoked by uid 500); 2 May 2011 20:51:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67057 invoked by uid 99); 2 May 2011 20:51:08 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 20:51:08 +0000
Received: from localhost (HELO [192.168.168.16]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 20:51:08 +0000
Message-ID: <4DBF193D.6060306@apache.org>
Date: Mon, 02 May 2011 13:51:09 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.14) Gecko/20110223 Thunderbird/3.1.8
MIME-Version: 1.0
To: general@hadoop.apache.org
CC: Eric Baldeschwieler <eric14@yahoo-inc.com>
Subject: Re: Discussions - Re: [VOTE] Release candidate 0.20.203.0-rc0
References: <12853C59-08F1-4F98-ACF3-800F06BC7A5F@apache.org> <4DBEF0C1.2050306@apache.org> <05E53D57-D53A-4D05-AAA4-E99A12373ED2@Holsman.NET> <38AE939A-0717-4220-A6F0-6093FF852060@yahoo-inc.com>
In-Reply-To: <38AE939A-0717-4220-A6F0-6093FF852060@yahoo-inc.com>
X-Enigmail-Version: 1.1.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On 05/02/2011 01:05 PM, Eric Baldeschwieler wrote:
> As previously discussed, all parties are welcome to champion
> altenative releases from Apache if they want to invest in making
> Apache Hadoop better.

I do not believe that different organizations should release their own
versions of Hadoop posing as Apache releases.  If folks wish to release
their own versions, then they should call them something else and
release them themselves.  The Apache Hadoop project should create
releases collaboratively, through an open process.  The standard means
is to start a branch from trunk or a prior release and propose patches
to that branch, one-by-one.  This candidate diverged sufficiently from
this pattern that, for me, it doesn't qualify as a community release.

Cheers,

Doug

From general-return-3307-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 21:00:23 2011
Return-Path: <general-return-3307-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E417438AB
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  2 May 2011 21:00:22 +0000 (UTC)
Received: (qmail 84763 invoked by uid 500); 2 May 2011 21:00:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84546 invoked by uid 500); 2 May 2011 21:00:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84538 invoked by uid 99); 2 May 2011 21:00:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 21:00:20 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 74.125.83.176 is neither permitted nor denied by domain of james@tynt.com)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 21:00:12 +0000
Received: by pve37 with SMTP id 37so4685089pve.35
        for <general@hadoop.apache.org>; Mon, 02 May 2011 13:59:50 -0700 (PDT)
Received: by 10.68.5.164 with SMTP id t4mr9190677pbt.167.1304369990793;
        Mon, 02 May 2011 13:59:50 -0700 (PDT)
Received: from [10.0.1.9] (S01060016cbc5f8b4.cg.shawcable.net [174.0.46.252])
        by mx.google.com with ESMTPS id m1sm1366120pbb.47.2011.05.02.13.59.49
        (version=TLSv1/SSLv3 cipher=OTHER);
        Mon, 02 May 2011 13:59:50 -0700 (PDT)
Content-Type: text/plain; charset=windows-1252
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: Discussions - Re: [VOTE] Release candidate 0.20.203.0-rc0
From: James Seigel <james@tynt.com>
In-Reply-To: <4DBF193D.6060306@apache.org>
Date: Mon, 2 May 2011 14:59:48 -0600
Content-Transfer-Encoding: quoted-printable
Message-Id: <CFD224E6-4EBE-497A-8186-307233F898DB@tynt.com>
References: <12853C59-08F1-4F98-ACF3-800F06BC7A5F@apache.org> <4DBEF0C1.2050306@apache.org> <05E53D57-D53A-4D05-AAA4-E99A12373ED2@Holsman.NET> <38AE939A-0717-4220-A6F0-6093FF852060@yahoo-inc.com> <4DBF193D.6060306@apache.org>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

Hello!

I guess I am concerned as a user of hadoop that the only way to get an =
=93endorsed=94 up-to-date version of hadoop one has to abandon the =
community and =93trust=94 a commercial release with its special sauce.

I am just hoping that the community can put together a nice stable =
up-to-date patched version.  That=92d be nice.  It probably won=92t =
change my commercial deploy, but it would give me something to compare =
with :)

Just my $0.02 (CND)
Cheers
James.

On 2011-05-02, at 2:51 PM, Doug Cutting wrote:

> On 05/02/2011 01:05 PM, Eric Baldeschwieler wrote:
>> As previously discussed, all parties are welcome to champion
>> altenative releases from Apache if they want to invest in making
>> Apache Hadoop better.
>=20
> I do not believe that different organizations should release their own
> versions of Hadoop posing as Apache releases.  If folks wish to =
release
> their own versions, then they should call them something else and
> release them themselves.  The Apache Hadoop project should create
> releases collaboratively, through an open process.  The standard means
> is to start a branch from trunk or a prior release and propose patches
> to that branch, one-by-one.  This candidate diverged sufficiently from
> this pattern that, for me, it doesn't qualify as a community release.
>=20
> Cheers,
>=20
> Doug


From general-return-3308-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 21:02:13 2011
Return-Path: <general-return-3308-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 998B131BE
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  2 May 2011 21:02:13 +0000 (UTC)
Received: (qmail 91366 invoked by uid 500); 2 May 2011 21:02:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91243 invoked by uid 500); 2 May 2011 21:02:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91227 invoked by uid 99); 2 May 2011 21:02:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 21:02:10 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 209.85.215.176 as permitted sender)
Received: from [209.85.215.176] (HELO mail-ey0-f176.google.com) (209.85.215.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 21:02:04 +0000
Received: by eya28 with SMTP id 28so2742565eya.35
        for <multiple recipients>; Mon, 02 May 2011 14:01:44 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.14.15.9 with SMTP id e9mr3337525eee.74.1304370104065; Mon, 02
 May 2011 14:01:44 -0700 (PDT)
Received: by 10.14.127.15 with HTTP; Mon, 2 May 2011 14:01:44 -0700 (PDT)
In-Reply-To: <38AE939A-0717-4220-A6F0-6093FF852060@yahoo-inc.com>
References: <12853C59-08F1-4F98-ACF3-800F06BC7A5F@apache.org>
	<4DBEF0C1.2050306@apache.org>
	<05E53D57-D53A-4D05-AAA4-E99A12373ED2@Holsman.NET>
	<38AE939A-0717-4220-A6F0-6093FF852060@yahoo-inc.com>
Date: Mon, 2 May 2011 14:01:44 -0700
Message-ID: <BANLkTik+T_2is3-ktvex3_JfLqyQjTO7CA@mail.gmail.com>
Subject: Re: Discussions - Re: [VOTE] Release candidate 0.20.203.0-rc0
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Cc: "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Hey Eric,

I don't have any objections to a release from
branch-0.20-security-203.  However when I examined the specific patch
set I noticed the are important implications with respect to
compatibility (of for 0.20.2 and 0.22), a question about project model
(eg not reviewing patches on jira before committing them, not having
patches go through trunk, etc), and some open questions for users (eg
is this the next dot release of the stable branch?).

I agree this is a valuable artifact, but that doesn't mean it's OK to
ignore compatibility concerns, etc.

I've listed specifics questions/comments here:
http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201105.mbox/%3CB=
ANLkTinZ=3Dxb6kJ5PTeLN5KKD9b-cwaM0OQ@mail.gmail.com%3E

Thanks,
Eli

On Mon, May 2, 2011 at 1:05 PM, Eric Baldeschwieler
<eric14@yahoo-inc.com> wrote:
>
> Hi folks,
>
> This strikes me as a bit odd. I think we have already discussed this at l=
ength and agreed that a release could proceed.
>
> Since then, Arun and Owen have worked actively to incorporated community =
feedback into this release.
>
> All parties making Hadoop releases other then Apache have already incorpo=
rated most of the patches in this release into their products, including do=
ug's organization. I don't see how Hadoop's users benefit from Apache not i=
ncorporating them into an Apache release.
>
> As previously discussed, all parties are welcome to champion altenative r=
eleases from Apache if they want to invest in making Apache Hadoop better.
>
> Thanks!!
>
> E14
>
> ---
> E14 - typing on glass
>
> On May 2, 2011, at 12:16 PM, "Ian Holsman" <hadoop@holsman.net> wrote:
>
>> moving this thread to general@
>>
>> On May 3, 2011, at 3:58 AM, Doug Cutting wrote:
>>
>>>> Should we release
>>>> http://people.apache.org/~omalley/hadoop-0.20.203.0-rc0/?
>>>
>>> The patch selection process for this branch did not appear to be a
>>> community process. =A0A massive patch set was committed en-masse with n=
o
>>> public discussion before or after about its specific composition.
>>
>> guys...
>> 1. do we agree this is an issue
>> 2. if it is, how we do get the communication & discussion on list?
>>
>> what do people think are the major issues that are stopping people talki=
ng about stuff on list are?
>

From general-return-3309-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 21:06:01 2011
Return-Path: <general-return-3309-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 14B283C92
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  2 May 2011 21:06:01 +0000 (UTC)
Received: (qmail 98276 invoked by uid 500); 2 May 2011 21:05:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98233 invoked by uid 500); 2 May 2011 21:05:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98225 invoked by uid 99); 2 May 2011 21:05:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 21:05:59 +0000
X-ASF-Spam-Status: No, hits=2.9 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 21:05:54 +0000
Received: from l2tp-8-66.corp.yahoo.com (l2tp-8-66.corp.yahoo.com [10.73.8.66])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p42L5GLF021901
	for <general@hadoop.apache.org>; Mon, 2 May 2011 14:05:16 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1304370316;
	bh=s2N5vivpa4mAwb5TRD0MWzPuXtANECqOs+0R1tNR8Aw=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:Mime-Version:Subject:
	 Date:References;
	b=hy3tWzS/rTfeILuqS1DtKN9WzkX5pi3DXwNn9/NW9G2qykHyWQU34W6MjTdYDN61F
	 cShlgexeWCTCULCLFrma6HSIz2BpDVyiy7LL8+Ej9bL856SYvkMCAzvv5K/YxHFM+v
	 aOT83PpQeWMIH8IzpXOziN34sr2jYqH66wBnTYjM=
Message-Id: <247DDA36-BCD7-4EE9-A8CD-AD8E62C94F16@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <4DBF16CF.3030905@apache.org>
Content-Type: multipart/alternative; boundary=Apple-Mail-11--354398301
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc0
Date: Mon, 2 May 2011 14:05:16 -0700
References: <12853C59-08F1-4F98-ACF3-800F06BC7A5F@apache.org> <4DBEF0C1.2050306@apache.org> <BB3A69B5-9A5A-4454-80C8-62EBD3CF47EC@yahoo-inc.com> <4DBF16CF.3030905@apache.org>
X-Mailer: Apple Mail (2.936)

--Apple-Mail-11--354398301
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit

Doug,

On May 2, 2011, at 1:40 PM, Doug Cutting wrote:
>
> Also note that, on the common-dev thread, Eli & Tom have both noted a
> number of inconsistencies between this set of patches and trunk, 0.22
> and even prior 0.20 branches and releases.  In addition to the lack of
> community involvement in patch selection, these issues concern me.
>
> I cannot in good conscience vote for this release as a community  
> product.

As I noted before you were the first one to propose this release off  
Yahoo security patch-set in April, 2010:
http://s.apache.org/5Gv

What has changed since? Clearly, the same situation exists today.

Also, please note that of the ~450 commits in the branch, only 30 odd  
jiras are yet to be committed to trunk:
http://s.apache.org/7Pe. So it's incorrect to state 'lack of community  
involvement'.

Assuming the technical inconsistencies are sorted out, are you willing  
to withdraw you objection?

thanks,
Arun


--Apple-Mail-11--354398301--

From general-return-3310-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 21:21:05 2011
Return-Path: <general-return-3310-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5047C3347
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  2 May 2011 21:21:05 +0000 (UTC)
Received: (qmail 16667 invoked by uid 500); 2 May 2011 21:21:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16619 invoked by uid 500); 2 May 2011 21:21:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16611 invoked by uid 99); 2 May 2011 21:21:03 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 21:21:03 +0000
Received: from localhost (HELO [192.168.168.16]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 21:21:03 +0000
Message-ID: <4DBF2040.90602@apache.org>
Date: Mon, 02 May 2011 14:21:04 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.14) Gecko/20110223 Thunderbird/3.1.8
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc0
References: <12853C59-08F1-4F98-ACF3-800F06BC7A5F@apache.org> <4DBEF0C1.2050306@apache.org> <BB3A69B5-9A5A-4454-80C8-62EBD3CF47EC@yahoo-inc.com> <4DBF16CF.3030905@apache.org> <247DDA36-BCD7-4EE9-A8CD-AD8E62C94F16@yahoo-inc.com>
In-Reply-To: <247DDA36-BCD7-4EE9-A8CD-AD8E62C94F16@yahoo-inc.com>
X-Enigmail-Version: 1.1.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On 05/02/2011 02:05 PM, Arun C Murthy wrote:
> As I noted before you were the first one to propose this release off
> Yahoo security patch-set in April, 2010:
> http://s.apache.org/5Gv
> 
> What has changed since? Clearly, the same situation exists today.

I have absolutely no objection in principle to an Apache 0.20 release
including security.  I object to the fact that this patchset started
from an arbitrary point and unilaterally applied a large set of patches
that are not well correlated with Jira, trunk or other 0.20 branches.

> Also, please note that of the ~450 commits in the branch, only 30 odd
> jiras are yet to be committed to trunk:
> http://s.apache.org/7Pe. So it's incorrect to state 'lack of community
> involvement'.

This should be easily discoverable from Jira: issues should use the
"fix-for" field to indicate which branches they've been merged to.  This
standard practice has not been observed for over 400 patches included in
this release candidate.

> Assuming the technical inconsistencies are sorted out, are you willing
> to withdraw you objection?

These are not just technical concerns.  How I vote on any future release
candidate will in part depend on how the community is involved in its
production.

Thanks,

Doug

From general-return-3311-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 21:34:07 2011
Return-Path: <general-return-3311-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E22F03A44
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  2 May 2011 21:34:06 +0000 (UTC)
Received: (qmail 35339 invoked by uid 500); 2 May 2011 21:34:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 35280 invoked by uid 500); 2 May 2011 21:34:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 35272 invoked by uid 99); 2 May 2011 21:34:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 21:34:05 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 21:34:00 +0000
Received: from l2tp-8-66.corp.yahoo.com (l2tp-8-66.corp.yahoo.com [10.73.8.66])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p42LXKdo078559
	for <general@hadoop.apache.org>; Mon, 2 May 2011 14:33:20 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1304372001;
	bh=g34q1YNv+Ilq0DjVz54fpehc6d/aYVjjosgSvNhwXhY=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=H7mpmPkwg61wOygpqYgHNwFG0Fz65S5qBV5wUsu70oBykDBx1C4reUWfQoxBtFWEi
	 Wj5pUNr7S4/aVb/RPBkTGM4VvAcCoIUBR8YmkXJ+lzXnX6WMM6Uk2YzQA3xiDqZHeW
	 lLn2wlazsPc0NsBN/+LmWQnSIy5bjEkkm1KTXA6A=
Message-Id: <5F90B548-37ED-4DB8-AAE3-D1C2BB5C700A@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <4DBF2040.90602@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc0
Date: Mon, 2 May 2011 14:33:20 -0700
References: <12853C59-08F1-4F98-ACF3-800F06BC7A5F@apache.org> <4DBEF0C1.2050306@apache.org> <BB3A69B5-9A5A-4454-80C8-62EBD3CF47EC@yahoo-inc.com> <4DBF16CF.3030905@apache.org> <247DDA36-BCD7-4EE9-A8CD-AD8E62C94F16@yahoo-inc.com> <4DBF2040.90602@apache.org>
X-Mailer: Apple Mail (2.936)


On May 2, 2011, at 2:21 PM, Doug Cutting wrote:

> On 05/02/2011 02:05 PM, Arun C Murthy wrote:
>> As I noted before you were the first one to propose this release off
>> Yahoo security patch-set in April, 2010:
>> http://s.apache.org/5Gv
>>
>> What has changed since? Clearly, the same situation exists today.
>
> I have absolutely no objection in principle to an Apache 0.20 release
> including security.  I object to the fact that this patchset started
> from an arbitrary point and unilaterally applied a large set of  
> patches
> that are not well correlated with Jira, trunk or other 0.20 branches.

Completely untrue.

This patchset started from 0.20.1 has is complete superset of 0.20.1.

We will work towards ensuring it is a complete superset of the last  
stable release: 0.20.2.

>
>> Also, please note that of the ~450 commits in the branch, only 30 odd
>> jiras are yet to be committed to trunk:
>> http://s.apache.org/7Pe. So it's incorrect to state 'lack of  
>> community
>> involvement'.
>
> This should be easily discoverable from Jira: issues should use the
> "fix-for" field to indicate which branches they've been merged to.   
> This
> standard practice has not been observed for over 400 patches  
> included in
> this release candidate.
>

This seems like parliamentary stalling procedures... sure they don't  
have 'fix-for' fields but they've been verified to be true from  
external committers:

http://s.apache.org/yX

Are you simply asking for someone to go through the 450 odd jiras and  
set 'fix-for' fields?

>> Assuming the technical inconsistencies are sorted out, are you  
>> willing
>> to withdraw you objection?
>
> These are not just technical concerns.  How I vote on any future  
> release
> candidate will in part depend on how the community is involved in its
> production.
>

I understand they aren't technical concerns.

I asked if you were willing to withdraw your objection if the  
technical concerns are satisfied. I think you answered my question -  
you will not withdraw your objection even if it's a technical issue.

thanks,
Arun


From general-return-3312-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 21:48:38 2011
Return-Path: <general-return-3312-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1CD9A3238
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  2 May 2011 21:48:38 +0000 (UTC)
Received: (qmail 49550 invoked by uid 500); 2 May 2011 21:48:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49466 invoked by uid 500); 2 May 2011 21:48:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49458 invoked by uid 99); 2 May 2011 21:48:36 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 21:48:36 +0000
Received: from localhost (HELO [192.168.168.16]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 21:48:36 +0000
Message-ID: <4DBF26B5.70409@apache.org>
Date: Mon, 02 May 2011 14:48:37 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.14) Gecko/20110223 Thunderbird/3.1.8
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc0
References: <12853C59-08F1-4F98-ACF3-800F06BC7A5F@apache.org> <4DBEF0C1.2050306@apache.org> <BB3A69B5-9A5A-4454-80C8-62EBD3CF47EC@yahoo-inc.com> <4DBF16CF.3030905@apache.org> <247DDA36-BCD7-4EE9-A8CD-AD8E62C94F16@yahoo-inc.com> <4DBF2040.90602@apache.org> <5F90B548-37ED-4DB8-AAE3-D1C2BB5C700A@yahoo-inc.com>
In-Reply-To: <5F90B548-37ED-4DB8-AAE3-D1C2BB5C700A@yahoo-inc.com>
X-Enigmail-Version: 1.1.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On 05/02/2011 02:33 PM, Arun C Murthy wrote:
> On May 2, 2011, at 2:21 PM, Doug Cutting wrote:
>> I have absolutely no objection in principle to an Apache 0.20 release
>> including security.  I object to the fact that this patchset started
>> from an arbitrary point and unilaterally applied a large set of patches
>> that are not well correlated with Jira, trunk or other 0.20 branches.
> 
> Completely untrue.

'Completely'?  Really?  Not a true bit in there?  Wow!

> This patchset started from 0.20.1 has is complete superset of 0.20.1.

0.20.1 isn't a branch, it's a tag.  The 0.20 branch includes many
post-0.20.1 patches that are not in this candidate.  Releases in a
series normally share a branch.

> I asked if you were willing to withdraw your objection if the technical
> concerns are satisfied. I think you answered my question - you will not
> withdraw your objection even if it's a technical issue.

That is not what I said.  If this release does not get enough votes then
perhaps another 0.20.203 release candidate will be proposed.  Its
process and contents will be different and I will judge it on the basis
of those when I vote.

Doug

From general-return-3313-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 21:50:11 2011
Return-Path: <general-return-3313-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DCD7D32EA
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  2 May 2011 21:50:10 +0000 (UTC)
Received: (qmail 52452 invoked by uid 500); 2 May 2011 21:50:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52403 invoked by uid 500); 2 May 2011 21:50:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52395 invoked by uid 99); 2 May 2011 21:50:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 21:50:09 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 21:50:02 +0000
Received: by qwj9 with SMTP id 9so4065680qwj.35
        for <general@hadoop.apache.org>; Mon, 02 May 2011 14:49:40 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=YJmsiepD8ZZXQTKYBChWOesLAtHL0eOs0GLByegTSvs=;
        b=Yd+V1DEwruYjBsWbBKm/YIY4gGzy0W36JFgPGE3MuuCIvynWijyjz9h0STIv5V2gaN
         w5j4Uwl/D6qajMc4Mu1Xo1w8t0YUaXr9tRaDZj2wXtD4f3xdJhA5B3xQ1OepAEypkwf+
         hbP0brDJPjVbxg9/nPcPMria+U8GDur3z/x0A=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=PnQElhCjFsMfEEMo+Cw6N3CgjIO5SyNJHFmyqXAzCUV5xYUVVB6Y8Y9fobB30gcy1m
         wird3lxLZW9fVDT2pO1JOwxcNxCwkga2Y/265YWpaCNKqzq9Y5Lz1JUBfrVLK17+YnmD
         34p4fLoeb8fXk4pq0gEuVEbbBZKugQEv6J0ms=
Received: by 10.229.17.7 with SMTP id q7mr6536903qca.145.1304372980331;
        Mon, 02 May 2011 14:49:40 -0700 (PDT)
Received: from [10.172.0.214] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id t17sm4262620qcs.23.2011.05.02.14.49.38
        (version=TLSv1/SSLv3 cipher=OTHER);
        Mon, 02 May 2011 14:49:39 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc0
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <5F90B548-37ED-4DB8-AAE3-D1C2BB5C700A@yahoo-inc.com>
Date: Tue, 3 May 2011 07:49:33 +1000
Content-Transfer-Encoding: quoted-printable
Message-Id: <DA227749-2244-4B61-B940-0C01F1E75FF3@Holsman.NET>
References: <12853C59-08F1-4F98-ACF3-800F06BC7A5F@apache.org> <4DBEF0C1.2050306@apache.org> <BB3A69B5-9A5A-4454-80C8-62EBD3CF47EC@yahoo-inc.com> <4DBF16CF.3030905@apache.org> <247DDA36-BCD7-4EE9-A8CD-AD8E62C94F16@yahoo-inc.com> <4DBF2040.90602@apache.org> <5F90B548-37ED-4DB8-AAE3-D1C2BB5C700A@yahoo-inc.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org


On May 3, 2011, at 7:33 AM, Arun C Murthy wrote:

>=20
> This patchset started from 0.20.1 has is complete superset of 0.20.1.
>=20
> We will work towards ensuring it is a complete superset of the last =
stable release: 0.20.2.

so are you intending to make it a superset for 203? or for a future =
release?=

From general-return-3314-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 21:55:27 2011
Return-Path: <general-return-3314-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 06CC534A3
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  2 May 2011 21:55:27 +0000 (UTC)
Received: (qmail 62760 invoked by uid 500); 2 May 2011 21:55:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62712 invoked by uid 500); 2 May 2011 21:55:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62704 invoked by uid 99); 2 May 2011 21:55:25 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 21:55:25 +0000
Received: from localhost (HELO [192.168.168.16]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 21:55:25 +0000
Message-ID: <4DBF284E.2020509@apache.org>
Date: Mon, 02 May 2011 14:55:26 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.14) Gecko/20110223 Thunderbird/3.1.8
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc0
References: <12853C59-08F1-4F98-ACF3-800F06BC7A5F@apache.org> <4DBEF0C1.2050306@apache.org> <BB3A69B5-9A5A-4454-80C8-62EBD3CF47EC@yahoo-inc.com> <4DBF16CF.3030905@apache.org> <247DDA36-BCD7-4EE9-A8CD-AD8E62C94F16@yahoo-inc.com> <4DBF2040.90602@apache.org> <5F90B548-37ED-4DB8-AAE3-D1C2BB5C700A@yahoo-inc.com>
In-Reply-To: <5F90B548-37ED-4DB8-AAE3-D1C2BB5C700A@yahoo-inc.com>
X-Enigmail-Version: 1.1.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On 05/02/2011 02:33 PM, Arun C Murthy wrote:
> We will work towards ensuring it is a complete superset of the last
> stable release: 0.20.2.

Great!  Who's 'we'?  Do you want any help with this?

Doug

From general-return-3315-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 22:06:13 2011
Return-Path: <general-return-3315-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 324F53A97
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  2 May 2011 22:06:13 +0000 (UTC)
Received: (qmail 73552 invoked by uid 500); 2 May 2011 22:06:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73507 invoked by uid 500); 2 May 2011 22:06:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73497 invoked by uid 99); 2 May 2011 22:06:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 22:06:11 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [98.139.52.228] (HELO nm7-vm0.bullet.mail.ac4.yahoo.com) (98.139.52.228)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 02 May 2011 22:06:04 +0000
Received: from [98.139.52.196] by nm7.bullet.mail.ac4.yahoo.com with NNFMP; 02 May 2011 22:05:43 -0000
Received: from [98.139.52.147] by tm9.bullet.mail.ac4.yahoo.com with NNFMP; 02 May 2011 22:05:43 -0000
Received: from [127.0.0.1] by omp1030.mail.ac4.yahoo.com with NNFMP; 02 May 2011 22:05:42 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 996854.96846.bm@omp1030.mail.ac4.yahoo.com
Received: (qmail 50690 invoked by uid 60001); 2 May 2011 22:05:42 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1304373942; bh=H2wFz++YCasumGdGuctn/+FFObYCBF9I6NlwNa+PiUA=; h=Message-ID:X-YMail-OSG:Received:X-RocketYMMF:X-Mailer:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=exGUzj2VjKn1PwLaSuWNI+KEcmhw21G9xkBIcbP+oKRbkd0ixuLeRlSpwLFy5u1v/gsbz3H4y5rqoSvQ29hJLHw/B8v8vg8p1V1M9nPz5/+jFS1NmLciUPncxAYyhcRcd7+Ja0eMw8UTE/VnNg26XBlZEJwN2dG0c55swbefosI=
Message-ID: <857369.41092.qm@web65511.mail.ac4.yahoo.com>
X-YMail-OSG: zoMFFgMVM1nU6PsguNPn53nXkXa8Ec.9Y4RDv3lPfPTqHgb
 bKaRgsP82B1Hy_b3Af153DDdhwgpirx1ToV02moS0Cd9hIuc0DjcJCJD9zeD
 DMZ2CT8ygI8LtJwIuJ6xJ.sM6Q5ukzD3hg4nhubzxQIRbA5dVsvijB1qpMfU
 ubzcFC0wedyHlYFQNgrywIh6_EByYmQAwjvhyEb5oW0h8fZafu20b8jzzYqG
 sDQ23l.w5BNiqMr9_mUndmceUX96ZN5FgSboSr9nn5ymUSKNy3_VWQ8RK56z
 rVcpYEfXbBMLmRWYK21b11Twix4kemOodBqF50CDbXiOo.unQ1yKk6HzvCov
 CLKGlcN7pGPf9cgwBb7V_CLuzalEEUREXXAq9jgY91n7DtXdZqAycz2rtXm5
 0D_b2EUdEbL_B_w--
Received: from [69.231.27.174] by web65511.mail.ac4.yahoo.com via HTTP; Mon, 02 May 2011 15:05:42 PDT
X-RocketYMMF: apurtell
X-Mailer: YahooMailClassic/12.0.2 YahooMailWebService/0.8.110.299900
Date: Mon, 2 May 2011 15:05:42 -0700 (PDT)
From: Andrew Purtell <apurtell@apache.org>
Reply-To: apurtell@apache.org
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc0
To: general@hadoop.apache.org
In-Reply-To: <4DBF2040.90602@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Most points in this thread are valid, having to do with the process of how =
the contribution was assembled; and specific technical aspects of it, e.g. =
JIRAs missing from branch 0.20.203 relative to branch 0.20. However,=0A=0A>=
 From: Doug Cutting <cutting@apache.org>=0A> > Assuming the technical incon=
sistencies are sorted out,=0A> > are you willing to withdraw you objection?=
=0A> =0A> These are not just technical concerns.=A0 How I vote on any futur=
e=0A> release candidate will in part depend on how the community is=0A> inv=
olved in its production.=0A=0AWhat strikes me, as an observer to this discu=
ssion, is that here "community" does not seem equated with Yahoo by implica=
tion. Perhaps I misread. Nevertheless, Yahoo retains a good percentage of a=
ctive Core developers with standing as both committers and high scale users=
, and these people produced the contribution that is branch 0.20.203, and t=
herefore by definition "the community" was entirely involved in its product=
ion.=0A=0AYahoo should be commended for advancing the state of branch 0.20 =
with an obvious commitment to donating the results to Apache. As a communit=
y we are lucky to have a strong contributor. Their security enhancements al=
low us and many others the option of strong authentication and user isolati=
on for multitenant deployments. =0A=0AA commercial vendor's product already=
 incorporates Yahoo's donated security enhancements. It would be regrettabl=
e if nontechnical factors ultimately prevents Apache from incorporating the=
 value of these contributions into an official release.=0A=0ASome technical=
 concerns seem reasonable. Regarding that:=0A=0A> From: Stack <stack@duboce=
.net>=0A> How hard would it be to get the patches Tom lists below into=0A> =
branch-0.20-security-203?=A0 I'd think it'd be an easier=0A> sell if it wer=
e a superset of all in 0.20, especially since it=0A> bears its name.=0A=0AT=
his suggestion makes a lot of sense. In addition, filing JIRAs for and post=
ing the diffs of the remaining differences could help the process as well, =
and would be good faith actions of an active contributor.=0A=0ABest regards=
,=0A=0A    - Andy=0A=0AProblems worthy of attack prove their worth by hitti=
ng back. - Piet Hein (via Tom White)=0A

From general-return-3316-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 22:15:04 2011
Return-Path: <general-return-3316-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3E2FE3C78
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  2 May 2011 22:15:04 +0000 (UTC)
Received: (qmail 84151 invoked by uid 500); 2 May 2011 22:15:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84091 invoked by uid 500); 2 May 2011 22:15:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84083 invoked by uid 99); 2 May 2011 22:15:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 22:15:02 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [98.139.52.215] (HELO nm18.bullet.mail.ac4.yahoo.com) (98.139.52.215)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 02 May 2011 22:14:55 +0000
Received: from [98.139.52.195] by nm18.bullet.mail.ac4.yahoo.com with NNFMP; 02 May 2011 22:14:34 -0000
Received: from [98.139.52.130] by tm8.bullet.mail.ac4.yahoo.com with NNFMP; 02 May 2011 22:14:34 -0000
Received: from [127.0.0.1] by omp1013.mail.ac4.yahoo.com with NNFMP; 02 May 2011 22:14:34 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 16509.80734.bm@omp1013.mail.ac4.yahoo.com
Received: (qmail 55300 invoked by uid 60001); 2 May 2011 22:14:33 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1304374473; bh=NaG3Lb8NwZp9xieW92C6RuQU/SMJurW++OgiY053KwY=; h=Message-ID:X-YMail-OSG:Received:X-RocketYMMF:X-Mailer:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=uBJsPR5+EueeYRki4FNOF7Vkw8Ajz4g4zfD7fAnKFgyMzEvl4WFwEKAdbwz9eMNHXiDxDcTlr37B06l5mLri1IGwpL5et3UhMAyzX+yOf2gtzdg12rI3u9qchI7GWSUpOsk0H2E3mjFh6pPHmReWKlWiRLQP0J9JZE0iOa94UKo=
Message-ID: <878851.54111.qm@web65511.mail.ac4.yahoo.com>
X-YMail-OSG: TXl6rjAVM1nxTn_cPing3fu5fis015qy1bk865sZQ.fwHxU
 4MBGBWuDlf.Ky.0IztTfd98J__gItnHkLPKbTRZlDXf9ZxLqIFOLEmmYzaLX
 _KIBF9vvTTe9EGjUKxlmXevYLtjTVxs2Y2k1nhXwJTMxQj6ElX7VUZqsXg8I
 v1ti1u5JWNYFBYBjomwIGFFKk_K06sn996ThHJ7.IrylcJCmj4ZrxVKDeU0i
 j.4ixso0GxolJ4ridfK9XsuE.EZo0qWgdekMmtoomIVyK4cHOi72JrgjF7g6
 SnNw.hEkw_SZsrTE3cStmoRSulEgJYkUs62dQCFl5LPQJOVUZD1R1YNI9FA1
 ySItxP0OFTFFJt2PMNCcoDMvECm4kUzszzzyPc.gP0vykPh5u1Rb5F8jo7Wn
 4UZRI9Tgf2N_pRQ--
Received: from [69.231.27.174] by web65511.mail.ac4.yahoo.com via HTTP; Mon, 02 May 2011 15:14:33 PDT
X-RocketYMMF: apurtell
X-Mailer: YahooMailClassic/12.0.2 YahooMailWebService/0.8.110.299900
Date: Mon, 2 May 2011 15:14:33 -0700 (PDT)
From: Andrew Purtell <apurtell@apache.org>
Reply-To: apurtell@apache.org
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc0
To: general@hadoop.apache.org
In-Reply-To: <857369.41092.qm@web65511.mail.ac4.yahoo.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

I would like to make one minor but important clarification:=0A=0AFrom:=0A=
=0A> It would be regrettable if=0A> nontechnical factors ultimately prevent=
s Apache from=0A> incorporating the value of these contributions into an=0A=
> official release.=0A=0ATo:=0A=0AIt would be regrettable if nontechnical f=
actors ultimately prevents Apache from  incorporating the value of these co=
ntributions into an official release OF 0.20. There are some not yet ready =
to take the leap to 0.22; who do not consider it proven.  =0A=0ASo in this =
regard I do not wish to minimize concerns about distracting from the succes=
s of 0.22 or later releases. =0A=0A> From: Andrew Purtell <apurtell@apache.=
org>=0A> Subject: Re: [VOTE] Release candidate 0.20.203.0-rc0=0A> To: gener=
al@hadoop.apache.org=0A> Date: Monday, May 2, 2011, 3:05 PM=0A=0A> Most poi=
nts in this thread are valid,=0A> having to do with the process of how the =
contribution was=0A> assembled; and specific technical aspects of it, e.g. =
JIRAs=0A> missing from branch 0.20.203 relative to branch 0.20.=0A> However=
,=0A> =0A> > > From: Doug Cutting <cutting@apache.org>=0A> > > Assuming the=
 technical inconsistencies are sorted=0A> > > out, are you willing to withd=
raw you objection?=0A> > =0A> > These are not just technical concerns.=A0 H=
ow I vote on=0A> > any future release candidate will in part depend on how=
=0A> > the community is involved in its production.=0A> =0A> What strikes m=
e, as an observer to this discussion, is that=0A> here "community" does not=
 seem equated with Yahoo by=0A> implication. Perhaps I misread. Nevertheles=
s, Yahoo retains=0A> a good percentage of active Core developers with stand=
ing as=0A> both committers and high scale users, and these people=0A> produ=
ced the contribution that is branch 0.20.203, and=0A> therefore by definiti=
on "the community" was entirely=0A> involved in its production.=0A> =0A> Ya=
hoo should be commended for advancing the state of branch=0A> 0.20 with an =
obvious commitment to donating the results to=0A> Apache. As a community we=
 are lucky to have a strong=0A> contributor. Their security enhancements al=
low us and many=0A> others the option of strong authentication and user=0A>=
 isolation for multitenant deployments. =0A> =0A> A commercial vendor's pro=
duct already incorporates Yahoo's=0A> donated security enhancements. It wou=
ld be regrettable if=0A> nontechnical factors ultimately prevents Apache fr=
om=0A> incorporating the value of these contributions into an=0A> official =
release.=0A> =0A> Some technical concerns seem reasonable. Regarding that:=
=0A> =0A> > From: Stack <stack@duboce.net>=0A> > How hard would it be to ge=
t the patches Tom lists=0A> below into=0A> > branch-0.20-security-203?=A0 I=
'd think it'd be an=0A> easier=0A> > sell if it were a superset of all in 0=
.20, especially=0A> since it=0A> > bears its name.=0A> =0A> This suggestion=
 makes a lot of sense. In addition, filing=0A> JIRAs for and posting the di=
ffs of the remaining differences=0A> could help the process as well, and wo=
uld be good faith=0A> actions of an active contributor.=0A> =0A> Best regar=
ds,=0A> =0A> =A0 =A0 - Andy=0A> =0A> Problems worthy of attack prove their =
worth by hitting=0A> back. - Piet Hein (via Tom White)=0A> =0A> 

From general-return-3317-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 22:16:07 2011
Return-Path: <general-return-3317-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E36D53F6C
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  2 May 2011 22:16:06 +0000 (UTC)
Received: (qmail 87102 invoked by uid 500); 2 May 2011 22:16:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87048 invoked by uid 500); 2 May 2011 22:16:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87036 invoked by uid 99); 2 May 2011 22:16:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 22:16:05 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 22:16:00 +0000
Received: from l2tp-8-66.corp.yahoo.com (l2tp-8-66.corp.yahoo.com [10.73.8.66])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p42MFDIt040212;
	Mon, 2 May 2011 15:15:13 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1304374513;
	bh=gJsgRgrT2xC9idfARiOmnVDA0l2YgfFPZj1x/TfnJNM=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=rTi6M449Dmp431lNxanza4LfptTXiXRRD0bc5OiloAh++KsPyePOLzmIir19noEAY
	 GUGaIwFSt4CPKDra/FLFC5eBgO4voLH4S+zcNhpP2KDQOMOxME+a0xMquKLXjzLXNN
	 wTSSa8pk0h+H9wf0vILA3NAsjWTxg+/UKjaxam+o=
Message-Id: <DD993727-BFB3-4421-A2D6-F348FC8B1BB7@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>,
        "apurtell@apache.org" <apurtell@apache.org>
In-Reply-To: <857369.41092.qm@web65511.mail.ac4.yahoo.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc0
Date: Mon, 2 May 2011 15:15:13 -0700
References: <857369.41092.qm@web65511.mail.ac4.yahoo.com>
X-Mailer: Apple Mail (2.936)


On May 2, 2011, at 3:05 PM, Andrew Purtell wrote:


> Some technical concerns seem reasonable. Regarding that:
>
>> From: Stack <stack@duboce.net>
>> How hard would it be to get the patches Tom lists below into
>> branch-0.20-security-203?  I'd think it'd be an easier
>> sell if it were a superset of all in 0.20, especially since it
>> bears its name.
>
> This suggestion makes a lot of sense. In addition, filing JIRAs for  
> and posting the diffs of the remaining differences could help the  
> process as well, and would be good faith actions of an active  
> contributor.
>

Agreed, I'm starting the effort to ensure the differences from 0.20.2  
are resolved.

 From my msg on this thread to common-dev@:

> # Remaining for 0.20.203
>  * HADOOP-5611
>  * HADOOP-5612
>  * HADOOP-5623
>  * HDFS-596
>  * HDFS-723
>  * HDFS-732
>  * HDFS-579
>  * MAPREDUCE-1070
>  * HADOOP-6315
>  * MAPREDUCE-1163

Suresh has kindly agreed to help me, appreciate help from others -  
particularly on the 0.20.3 changes.

thanks,
Arun



From general-return-3318-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 22:19:31 2011
Return-Path: <general-return-3318-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E8B883E87
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  2 May 2011 22:19:30 +0000 (UTC)
Received: (qmail 90036 invoked by uid 500); 2 May 2011 22:19:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89979 invoked by uid 500); 2 May 2011 22:19:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89971 invoked by uid 99); 2 May 2011 22:19:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 22:19:29 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 209.85.215.176 as permitted sender)
Received: from [209.85.215.176] (HELO mail-ey0-f176.google.com) (209.85.215.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 22:19:23 +0000
Received: by eya28 with SMTP id 28so2768829eya.35
        for <general@hadoop.apache.org>; Mon, 02 May 2011 15:19:03 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.14.51.20 with SMTP id a20mr3509458eec.213.1304374743400; Mon,
 02 May 2011 15:19:03 -0700 (PDT)
Received: by 10.14.127.15 with HTTP; Mon, 2 May 2011 15:19:03 -0700 (PDT)
In-Reply-To: <DD993727-BFB3-4421-A2D6-F348FC8B1BB7@yahoo-inc.com>
References: <857369.41092.qm@web65511.mail.ac4.yahoo.com>
	<DD993727-BFB3-4421-A2D6-F348FC8B1BB7@yahoo-inc.com>
Date: Mon, 2 May 2011 15:19:03 -0700
Message-ID: <BANLkTimoP2bN8rZDMrEhkCEah=0ApkKukA@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc0
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Cc: "apurtell@apache.org" <apurtell@apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Mon, May 2, 2011 at 3:15 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:
>
> On May 2, 2011, at 3:05 PM, Andrew Purtell wrote:
>
>
>> Some technical concerns seem reasonable. Regarding that:
>>
>>> From: Stack <stack@duboce.net>
>>> How hard would it be to get the patches Tom lists below into
>>> branch-0.20-security-203? =A0I'd think it'd be an easier
>>> sell if it were a superset of all in 0.20, especially since it
>>> bears its name.
>>
>> This suggestion makes a lot of sense. In addition, filing JIRAs for and
>> posting the diffs of the remaining differences could help the process as
>> well, and would be good faith actions of an active contributor.
>>
>
> Agreed, I'm starting the effort to ensure the differences from 0.20.2 are
> resolved.
>
> From my msg on this thread to common-dev@:
>
>> # Remaining for 0.20.203
>> =A0* HADOOP-5611
>> =A0* HADOOP-5612
>> =A0* HADOOP-5623
>> =A0* HDFS-596
>> =A0* HDFS-723
>> =A0* HDFS-732
>> =A0* HDFS-579
>> =A0* MAPREDUCE-1070
>> =A0* HADOOP-6315
>> =A0* MAPREDUCE-1163
>
> Suresh has kindly agreed to help me, appreciate help from others -
> particularly on the 0.20.3 changes.
>

I'd like to help.  Perhaps let's coordinate in a separate thread?

Thanks,
Eli

From general-return-3319-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 22:20:43 2011
Return-Path: <general-return-3319-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 340A231AF
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  2 May 2011 22:20:43 +0000 (UTC)
Received: (qmail 93091 invoked by uid 500); 2 May 2011 22:20:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93042 invoked by uid 500); 2 May 2011 22:20:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93034 invoked by uid 99); 2 May 2011 22:20:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 22:20:41 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 22:20:36 +0000
Received: from l2tp-8-66.corp.yahoo.com (l2tp-8-66.corp.yahoo.com [10.73.8.66])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p42MJbNk041390
	for <general@hadoop.apache.org>; Mon, 2 May 2011 15:19:44 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1304374784;
	bh=kDX4B2P5ycqNZPOPQ5QoZwZUMlFM2QVKo875XSXGIM8=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=swpg6cAzBG3eOq5L6aP1o0y6Twg3QYBQSjfY0wDhF2Y3pYgmcAW3fwcdb+7GSPZ5D
	 GxMIifAMkkZEBwbxE578ViYayNod82vniCsR76Bi/Ip6lNVUnJkV/Ft2B34bSeb+eO
	 ulggMvqoC6Jagl578dAv5fkIdPqWqE50dIEus2dY=
Message-Id: <410F98B8-6BFE-459A-BD75-46B5131F3F19@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <DA227749-2244-4B61-B940-0C01F1E75FF3@Holsman.NET>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc0
Date: Mon, 2 May 2011 15:19:36 -0700
References: <12853C59-08F1-4F98-ACF3-800F06BC7A5F@apache.org> <4DBEF0C1.2050306@apache.org> <BB3A69B5-9A5A-4454-80C8-62EBD3CF47EC@yahoo-inc.com> <4DBF16CF.3030905@apache.org> <247DDA36-BCD7-4EE9-A8CD-AD8E62C94F16@yahoo-inc.com> <4DBF2040.90602@apache.org> <5F90B548-37ED-4DB8-AAE3-D1C2BB5C700A@yahoo-inc.com> <DA227749-2244-4B61-B940-0C01F1E75FF3@Holsman.NET>
X-Mailer: Apple Mail (2.936)


On May 2, 2011, at 2:49 PM, Ian Holsman wrote:

>
> On May 3, 2011, at 7:33 AM, Arun C Murthy wrote:
>
>>
>> This patchset started from 0.20.1 has is complete superset of 0.20.1.
>>
>> We will work towards ensuring it is a complete superset of the last  
>> stable release: 0.20.2.
>
> so are you intending to make it a superset for 203? or for a future  
> release?

For 0.20.203, it behooves us to incorporate feedback for that.

Arun

From general-return-3320-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 22:25:50 2011
Return-Path: <general-return-3320-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 547803243
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  2 May 2011 22:25:50 +0000 (UTC)
Received: (qmail 1826 invoked by uid 500); 2 May 2011 22:25:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1781 invoked by uid 500); 2 May 2011 22:25:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1773 invoked by uid 99); 2 May 2011 22:25:48 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 22:25:48 +0000
Received: from localhost (HELO [192.168.168.16]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 22:25:48 +0000
Message-ID: <4DBF2F6D.8010709@apache.org>
Date: Mon, 02 May 2011 15:25:49 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.14) Gecko/20110223 Thunderbird/3.1.8
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc0
References: <857369.41092.qm@web65511.mail.ac4.yahoo.com>
In-Reply-To: <857369.41092.qm@web65511.mail.ac4.yahoo.com>
X-Enigmail-Version: 1.1.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On 05/02/2011 03:05 PM, Andrew Purtell wrote:
> What strikes me, as an observer to this discussion, is that here
> "community" does not seem equated with Yahoo by implication. Perhaps
> I misread. Nevertheless, Yahoo retains a good percentage of active
> Core developers with standing as both committers and high scale
> users, and these people produced the contribution that is branch
> 0.20.203, and therefore by definition "the community" was entirely
> involved in its production.

Whether or not a subset of contributors acts as the community depends on
whether others outside that subset have a reasonable opportunity to
become involved.  Until this release vote was called it wasn't entirely
clear to all what was happening with these branches.  Wider community
involvement is now starting as folks work to rationalize this 450-issue
patch with respect to past and future releases, Jira, etc.

> Yahoo should be commended for advancing the state of branch 0.20 with
> an obvious commitment to donating the results to Apache. As a
> community we are lucky to have a strong contributor. Their security
> enhancements allow us and many others the option of strong
> authentication and user isolation for multitenant deployments.

+1

> A commercial vendor's product already incorporates Yahoo's donated
> security enhancements. It would be regrettable if nontechnical
> factors ultimately prevents Apache from incorporating the value of
> these contributions into an official release.

+1

Cheers,

Doug

From general-return-3321-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 22:32:00 2011
Return-Path: <general-return-3321-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9AE443C26
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  2 May 2011 22:32:00 +0000 (UTC)
Received: (qmail 13497 invoked by uid 500); 2 May 2011 22:31:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13441 invoked by uid 500); 2 May 2011 22:31:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13433 invoked by uid 99); 2 May 2011 22:31:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 22:31:58 +0000
X-ASF-Spam-Status: No, hits=1.0 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_SOFTFAIL
X-Spam-Check-By: apache.org
Received-SPF: softfail (athena.apache.org: transitioning domain of jtcornelius@pentaho.com does not designate 98.129.35.4 as permitted sender)
Received: from [98.129.35.4] (HELO server505.appriver.com) (98.129.35.4)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 22:31:51 +0000
X-Note-AR-ScanTimeLocal: 5/2/2011 5:31:31 PM
X-Policy: GLOBAL - UNKNOWN
X-Primary: jtcornelius@pentaho.com
X-Note: This Email was scanned by AppRiver SecureTide
X-Virus-Scan: V-
X-Note-SnifferID: 0
X-Note: TCH-CT/SI:0-90/SG:2 5/2/2011 5:30:31 PM
X-GBUdb-Analysis: 0, 98.129.23.14, Ugly c=1 p=-0.554607 Source Normal
X-Signature-Violations:
	0-0-0-4117-c
X-Note: Spam Tests Failed: 
X-Country-Path: UNITED STATES->UNITED STATES->UNITED STATES
X-Note-Sending-IP: 98.129.23.14
X-Note-Reverse-DNS: smtp.exg5.exghost.com
X-Note-WHTLIST: jtcornelius@pentaho.com
X-Note: User Rule Hits: 
X-Note: Global Rule Hits: G195 G196 G197 G198 G202 G203 G302 
X-Note: Encrypt Rule Hits: 
X-Note: Mail Class: VALID
X-Note: Headers Injected
Received: from [98.129.23.14] (HELO smtp.exg5.exghost.com)
  by server505.appriver.com (CommuniGate Pro SMTP 5.3.13)
  with ESMTPS id 139716802 for general@hadoop.apache.org; Mon, 02 May 2011 17:31:31 -0500
Received: from MBX15.exg5.exghost.com ([169.254.1.20]) by
 ht01.exg5.exghost.com ([98.129.23.14]) with mapi; Mon, 2 May 2011 17:31:29
 -0500
From: Jake Cornelius <jtcornelius@pentaho.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 2 May 2011 17:31:29 -0500
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc0
Thread-Topic: [VOTE] Release candidate 0.20.203.0-rc0
Thread-Index: AcwJF+rv/4nhhKVGTcao2zDzzg+WuQAAMMkP
Message-ID: <nhpljt9wcefgmo79jixd99n6.1304375483558@email.android.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0



Doug Cutting <cutting@apache.org> wrote:


On 05/02/2011 03:05 PM, Andrew Purtell wrote:
> What strikes me, as an observer to this discussion, is that here
> "community" does not seem equated with Yahoo by implication. Perhaps
> I misread. Nevertheless, Yahoo retains a good percentage of active
> Core developers with standing as both committers and high scale
> users, and these people produced the contribution that is branch
> 0.20.203, and therefore by definition "the community" was entirely
> involved in its production.

Whether or not a subset of contributors acts as the community depends on
whether others outside that subset have a reasonable opportunity to
become involved.  Until this release vote was called it wasn't entirely
clear to all what was happening with these branches.  Wider community
involvement is now starting as folks work to rationalize this 450-issue
patch with respect to past and future releases, Jira, etc.

> Yahoo should be commended for advancing the state of branch 0.20 with
> an obvious commitment to donating the results to Apache. As a
> community we are lucky to have a strong contributor. Their security
> enhancements allow us and many others the option of strong
> authentication and user isolation for multitenant deployments.

+1

> A commercial vendor's product already incorporates Yahoo's donated
> security enhancements. It would be regrettable if nontechnical
> factors ultimately prevents Apache from incorporating the value of
> these contributions into an official release.

+1

Cheers,

Doug

From general-return-3322-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 00:41:56 2011
Return-Path: <general-return-3322-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5A474FC8
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue,  3 May 2011 00:41:56 +0000 (UTC)
Received: (qmail 69815 invoked by uid 500); 3 May 2011 00:41:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69749 invoked by uid 500); 3 May 2011 00:41:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69739 invoked by uid 99); 3 May 2011 00:41:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 00:41:54 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [208.97.132.74] (HELO homiemail-a73.g.dreamhost.com) (208.97.132.74)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 00:41:48 +0000
Received: from homiemail-a73.g.dreamhost.com (localhost [127.0.0.1])
	by homiemail-a73.g.dreamhost.com (Postfix) with ESMTP id E33671F007C
	for <general@hadoop.apache.org>; Mon,  2 May 2011 17:41:27 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; q=dns; s=
	gbiv.com; b=yyGfXDdf+NOt+aJ2QyZcJ6gcHijxeOX2nWc6K3fPiuBpGxUGRc43
	O4+3FNQycn83m5N6Urt1TkJ0MvTu+cMGxmp24eOmsM7RaAytaMnuGuDpQ2I3VQEH
	ZKWOFJr2JLakpJDEOZZUl7lmMCpKnQJmM9YITSfItQkQVWt81I34ifo=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; s=gbiv.com;
	 bh=7XLVJ9bIOqq5KKw3UI3KkSwG7P8=; b=6gfgZWA/OrYyh9AEbzBXbbl/EhDb
	96lHt3XEZkrey/b1WtGzV1r1tHrWEvUKJJl+bLS1lZzhv6KlhMaNYk8gCuOGJU3F
	IHl/m6221/wo3L25e+njEf0yadjQBSJ9fYEhhf2V1RO83e+wH0tM7x+YgglpAs7R
	d79/NBpqyecXu18=
Received: from [10.134.89.86] (unknown [75.103.10.98])
	(using TLSv1 with cipher AES128-SHA (128/128 bits))
	(No client certificate requested)
	(Authenticated sender: fielding@gbiv.com)
	by homiemail-a73.g.dreamhost.com (Postfix) with ESMTPSA id 92EAD1F0078
	for <general@hadoop.apache.org>; Mon,  2 May 2011 17:41:27 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: Discussions - Re: [VOTE] Release candidate 0.20.203.0-rc0
From: "Roy T. Fielding" <fielding@gbiv.com>
In-Reply-To: <05E53D57-D53A-4D05-AAA4-E99A12373ED2@Holsman.NET>
Date: Mon, 2 May 2011 17:41:26 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <81841C06-1C7B-4BAB-BF74-85B75D155C1D@gbiv.com>
References: <12853C59-08F1-4F98-ACF3-800F06BC7A5F@apache.org> <4DBEF0C1.2050306@apache.org> <05E53D57-D53A-4D05-AAA4-E99A12373ED2@Holsman.NET>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)

On May 2, 2011, at 12:15 PM, Ian Holsman wrote:

> moving this thread to general@
>=20
> On May 3, 2011, at 3:58 AM, Doug Cutting wrote:
>=20
>>> Should we release
>>> http://people.apache.org/~omalley/hadoop-0.20.203.0-rc0/?
>>=20
>> The patch selection process for this branch did not appear to be a
>> community process.  A massive patch set was committed en-masse with =
no
>> public discussion before or after about its specific composition.
>=20
> guys...
> 1. do we agree this is an issue

Of course it is an issue.  Anyone can make it an issue -- no
agreement is necessary.

> 2. if it is, how we do get the communication & discussion on list?

By communicating and discussing on list.  Like, for example,
by proposing a release vote and people objecting to it, followed
by a polite collaboration on ways to reduce objections if that
is needed to get a release out the door.

> what do people think are the major issues that are stopping people =
talking about stuff on list are?

The fact that people can vote on individual issues via jira,
which means that there is effectively no discussion of the
product as a whole on list.  I am constantly amazed at how
quiet it is in this project, at least until I remember that
most of the work is done exclusively via jira, unlike any of
my other followed projects that use jira.  I'd suggest that
the right place to hold any discussion is on the dev list,
but I am not on that list because it receives way too many
automated notifications.  Maybe it would help discussion on
dev if notices were sent elsewhere and only discussions were
held on dev.

By all means, produce a tarball and let the entire PMC vote
on it as the next release.  My personal preference is to not
allow anything that deviates from the major.minor.patch release
numbering that most software projects follow, but I don't have
a vote here.

It is perfectly reasonable for Doug (or anyone else) to vote
on a release based on a lack of version history, adequate
description of the sweet meats, or anything else that others
might consider non-technical.  This is a release vote!
It does not require consensus.  It requires minimal review
(usually meaning three +1s) and a majority opinion of those
on the PMC who choose to review the proposed release and vote.

....Roy=

From general-return-3323-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 04:12:13 2011
Return-Path: <general-return-3323-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 65B2B2348
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue,  3 May 2011 04:12:13 +0000 (UTC)
Received: (qmail 20861 invoked by uid 500); 3 May 2011 04:12:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20795 invoked by uid 500); 3 May 2011 04:12:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 20787 invoked by uid 99); 3 May 2011 04:12:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 04:12:08 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mbhandarkar@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 04:12:03 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:In-Reply-To:Accept-Language:
   Content-Language:X-MS-Has-Attach:X-MS-TNEF-Correlator:
   x-originating-ip:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version;
  b=muTyo1e1pOWtEvi469o/cGtWOL3LA83CWzhereOlyc92kC6KZV6UgIcQ
   X7jdsfAMqIUZ5IIZyQNjGvG+NToq0YWSK+rYOgckqKbtlhOZ5u3eA8f30
   WiOehcyIOnxsdac;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=mbhandarkar@linkedin.com; q=dns/txt;
  s=proddkim; t=1304395923; x=1335931923;
  h=from:to:subject:date:message-id:in-reply-to:content-id:
   content-transfer-encoding:mime-version;
  bh=efIJXWJC86muvedZp1tCJJmTsuP1I7ev8bYXOwyYSyk=;
  b=OgryO/UTZ3Fprz1PIfoOgopACXw9n03ODHikYaQ1GehxKudJpATISZBA
   KKT3cGZ5VHzC0RTHJn9CrSgqfB7u6pwqL4vjRn0byjKTehpzFXwCIEPMl
   5EYoBdrkp/u7U0f;
X-IronPort-AV: E=Sophos;i="4.64,307,1301900400"; 
   d="scan'208";a="22776366"
Received: from ESV4-EXC02.linkedin.biz ([fe80::4d74:48bd:e0bd:13ee]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi id 14.01.0251.000; Mon, 2
 May 2011 21:15:04 -0700
From: Milind Bhandarkar <mbhandarkar@linkedin.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: Discussions - Re: [VOTE] Release candidate 0.20.203.0-rc0
Thread-Topic: Discussions - Re: [VOTE] Release candidate 0.20.203.0-rc0
Thread-Index: AQHMBsMYhWGNisphykmPrKzgX+v3mZR6TOOAgAAVnwCAAFr7AP//xWQA
Date: Tue, 3 May 2011 04:15:03 +0000
Message-ID: <C9E4CB85.3625%mbhandarkar@linkedin.com>
In-Reply-To: <81841C06-1C7B-4BAB-BF74-85B75D155C1D@gbiv.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <EBF9D28F93688644995DB28167C37C2D@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


>It is perfectly reasonable for Doug (or anyone else) to vote
>on a release based on a lack of version history, adequate
>description of the sweet meats, or anything else that others
>might consider non-technical.  This is a release vote!
>It does not require consensus.  It requires minimal review
>(usually meaning three +1s) and a majority opinion of those
>on the PMC who choose to review the proposed release and vote.

Roy,

Thanks for reminding everyone that a release does not require consensus.

Regarding this release, I think anyone who runs a multi-tenant Hadoop
cluster will appreciate the user-limits feature that goes a long way to
ensure that an errant job does not take the entire cluster down. Your
operations and support people will thank you for deploying this release.

Recently I was discussing with operations folks at a company that operates
a Hadoop cluster based on a commercial distribution of Hadoop, and they
were excited to hear that they will have a way of making sure that their
cluster will not be taken down by an errant user/job, because that's one
big fear that keeps them awake.

FWIW, I am +1 for this release.

Arun, can you include a document that gives more details about what the
limits are, and how to modify your jobs to stay below these limits (I know
it is a cut-paste for you :-)?

- milind


From general-return-3324-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 05:18:23 2011
Return-Path: <general-return-3324-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C16512DCD
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue,  3 May 2011 05:18:23 +0000 (UTC)
Received: (qmail 82435 invoked by uid 500); 3 May 2011 05:18:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81881 invoked by uid 500); 3 May 2011 05:18:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81873 invoked by uid 99); 3 May 2011 05:18:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 05:18:20 +0000
X-ASF-Spam-Status: No, hits=3.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of eltonsky9404@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 05:18:15 +0000
Received: by wwi18 with SMTP id 18so7415001wwi.29
        for <general@hadoop.apache.org>; Mon, 02 May 2011 22:17:54 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=0xW0cDCRNQjyjH/8tQF3emvLZYsfnWw4ktko+UbMHAs=;
        b=kCGRq41vNFoVKWaYrPpXeedhxDZ8MtCL0kNNK7JngyKCNhltjapeiY0ePEqgpZR1eB
         wJ8j5sPnUlCvzTOWwl8AFzTZQhqpqh025OXiqZt2GjqvMEachbvXe+3Wxg82DXwp7L1b
         l60ZZMX+2vVNvFJXU+kWw6h61jY6BsVCjnegY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Qg54Zi1pJDahzPwukHkfvki4W8Cm+UgadsnvIL2mt5csCFZrABPKfzt6oZ7tgJcbwJ
         pJYxOsLepJOFmie8OSDfqb7Gpahvx6VsOXO3MTadeIe4EZ4dgknZpyptbQ4JzbDi7zH9
         OYkIDrb7A5OpM8XeatFLc+Kp6cqwcH+T4CCiM=
MIME-Version: 1.0
Received: by 10.227.5.205 with SMTP id 13mr3430827wbw.31.1304399874099; Mon,
 02 May 2011 22:17:54 -0700 (PDT)
Received: by 10.227.28.220 with HTTP; Mon, 2 May 2011 22:17:54 -0700 (PDT)
In-Reply-To: <Prayer.1.3.2.1105021055330.6905@longwood.cs.ucf.edu>
References: <BANLkTinbKUikZGx_c9h6oN=_yveF8j2K6Q@mail.gmail.com>
	<BANLkTi=2EaAmxaibRPiX=AEkVeL8XKmdvA@mail.gmail.com>
	<Prayer.1.3.2.1105021055330.6905@longwood.cs.ucf.edu>
Date: Tue, 3 May 2011 15:17:54 +1000
Message-ID: <BANLkTimpFC+Jtqqb1rZbQ9ax_nLPdQLF2g@mail.gmail.com>
Subject: Re: questions about hadoop map reduce and compute intensive related applications
From: elton sky <eltonsky9404@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175cf7da240ab904a2584385

--0015175cf7da240ab904a2584385
Content-Type: text/plain; charset=ISO-8859-1

thanks gmackey,

There is an project out of Sandia National Lab that puts MR and MPI together
> in a library if you're interested -->
> http://www.sandia.gov/~sjplimp/mapreduce.html


That is a implementation of MR using MPI. I saw that as well but haven't
tried it out.

I am actually looking at programming model level integration. You know, for
some applications, you can take advantage of high through put from MR and
message passing from MPI.

-Elton


On Tue, May 3, 2011 at 12:55 AM, <gmackey@cs.ucf.edu> wrote:

>
>  Integrating MPI with map-reduce is currently difficult and/or very ugly,
>> however. Not impossible and there are hackish ways to do the job, but they
>> are hacks.
>>
>
> There is an project out of Sandia National Lab that puts MR and MPI
> together in a library if you're interested -->
> http://www.sandia.gov/~sjplimp/mapreduce.html
>
> The project isn't mature yet, and I haven't actually used it myself.
>
> --
> --
> Grant Mackey
> PhD student Computer Engineering
> University of Central Florida
>

--0015175cf7da240ab904a2584385--

From general-return-3325-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 08:40:19 2011
Return-Path: <general-return-3325-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B97F72E9F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue,  3 May 2011 08:40:19 +0000 (UTC)
Received: (qmail 88330 invoked by uid 500); 3 May 2011 08:40:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87324 invoked by uid 500); 3 May 2011 08:40:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87309 invoked by uid 99); 3 May 2011 08:40:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 08:40:18 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of shv.hadoop@gmail.com designates 209.85.161.176 as permitted sender)
Received: from [209.85.161.176] (HELO mail-gx0-f176.google.com) (209.85.161.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 08:40:13 +0000
Received: by gxk7 with SMTP id 7so3058437gxk.35
        for <multiple recipients>; Tue, 03 May 2011 01:39:52 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=elJstl9zVKPMbEb1Gurl2xJyXqh/RHTmq21ZwDsfd7w=;
        b=ZKwJIsOHYvrjJQDEDSNgEiB//XRfnOmIxTm2smNmPgWMwXKhlEU8SmPu5o2dWWYH/a
         L4clzGmIXixAKdfEgjtCxtRsw1b9fciGCzNjFm2j9NutV4/0rLD90augh1C17dfJCKM/
         SQcUTWwbh/lQcWLOHEveQeBR82pwlN+TycIeo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=hnu381bxyyw5VKkLa+/jC6zsxS78YVxlgBHXBF9MTxU3Jf0ZDnW882bLHk9qzPIDwk
         fYWaKJUTTu9cV/OTDs6GWCgCtFu+kki/fl5J3PF0uCyAiVkFPPL+B0K9EdlKzzT2YeWg
         5LyEXSjbFOnXL+ECn9/U2uIcurO3GlknoyoB0=
MIME-Version: 1.0
Received: by 10.236.184.199 with SMTP id s47mr1711703yhm.356.1304411992495;
 Tue, 03 May 2011 01:39:52 -0700 (PDT)
Received: by 10.147.124.12 with HTTP; Tue, 3 May 2011 01:39:52 -0700 (PDT)
In-Reply-To: <8D047C89-6430-4FDE-A9C4-CFEE79644AC6@Holsman.NET>
References: <12853C59-08F1-4F98-ACF3-800F06BC7A5F@apache.org>
	<BANLkTinZ=xb6kJ5PTeLN5KKD9b-cwaM0OQ@mail.gmail.com>
	<BANLkTi=1sQoL68k_2UmA1_Oo2CyXsM3fAw@mail.gmail.com>
	<DBCE9300-0D9D-4A9A-99B4-878139A028EE@yahoo-inc.com>
	<61C1E656-0649-4097-8709-FDC596ED57E9@yahoo-inc.com>
	<F7921362-CC1B-435C-890A-E1BD181E03C0@yahoo-inc.com>
	<8D047C89-6430-4FDE-A9C4-CFEE79644AC6@Holsman.NET>
Date: Tue, 3 May 2011 01:39:52 -0700
Message-ID: <BANLkTi=JDBneLBHsCh1T-PF8C7wNbra0kw@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc0
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: common-dev@hadoop.apache.org, general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf303f651874174304a25b1534

--20cf303f651874174304a25b1534
Content-Type: text/plain; charset=ISO-8859-1

I think its a good idea to release hadoop-0.20.203. It moves Apache Hadoop a
step forward.

Looks like the technical difficulties are resolved now with latest Arun's
commits.
Being a superset of hadoop-0.20.2 it can be considered based on one of the
official Apache releases.
I don't think there was a lack of discussions on the lists about the issues
included in the release candidate. Todd did a thorough review of the entire
security branch. Many developers participated in discussions.
Agreeing with Stack I wish HBase was considered a primary target for Hadoop
support. But it is not realistic to have it in hadoop-0.20.203.
I have some experience running a version of this release candidate on a
large cluster. It works. I would add a couple of patches, which make it run
on Windows for me like HADOOP-7110, HADOOP-7126. But those are not blockers.

Thanks,
--Konstantin


On Mon, May 2, 2011 at 5:12 PM, Ian Holsman <hadoop@holsman.net> wrote:

>
> On May 3, 2011, at 9:58 AM, Arun C Murthy wrote:
>
> >>
> >> Owen, Suresh and I have committed everything on this list except
> >> HADOOP-6386 and HADOOP-6428. Not sure which of the two are relevant/
> >> necessary, I'll check with Cos.  Other than that hadoop-0.20.203 now a
> >> superset of hadoop-0.20.2.
> >>
> >
> > Missed adding HADOOP-5759 to that list, I'll check with Amareshwari
> before committing.
> >
> > Arun
>
> Thanks for doing this so fast Arun.
>
>

--20cf303f651874174304a25b1534--

From general-return-3326-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 14:17:54 2011
Return-Path: <general-return-3326-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8F2F22B17
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue,  3 May 2011 14:17:54 +0000 (UTC)
Received: (qmail 68012 invoked by uid 500); 3 May 2011 14:17:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67868 invoked by uid 500); 3 May 2011 14:17:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67860 invoked by uid 99); 3 May 2011 14:17:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 14:17:52 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 14:17:43 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id A3AB21BA925
	for <general@hadoop.apache.org>; Tue,  3 May 2011 15:17:20 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id 23o3T6Zuw18m for <general@hadoop.apache.org>;
	Tue,  3 May 2011 15:17:20 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by colossus.hpl.hp.com (Postfix) with ESMTP id F283C1BA91D
	for <general@hadoop.apache.org>; Tue,  3 May 2011 15:17:19 +0100 (BST)
MailScanner-NULL-Check: 1305037027.87133@8c1EBFxn6gL8cFRwSjsiVA
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p43EH7Ax019162
	for <general@hadoop.apache.org>; Tue, 3 May 2011 15:17:07 +0100 (BST)
Message-ID: <4DC00E63.1000507@apache.org>
Date: Tue, 03 May 2011 15:17:07 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.14) Gecko/20110223 Thunderbird/3.1.8
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Discussions - Re: [VOTE] Release candidate 0.20.203.0-rc0
References: <12853C59-08F1-4F98-ACF3-800F06BC7A5F@apache.org> <4DBEF0C1.2050306@apache.org> <05E53D57-D53A-4D05-AAA4-E99A12373ED2@Holsman.NET> <81841C06-1C7B-4BAB-BF74-85B75D155C1D@gbiv.com>
In-Reply-To: <81841C06-1C7B-4BAB-BF74-85B75D155C1D@gbiv.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p43EH7Ax019162
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 03/05/11 01:41, Roy T. Fielding wrote:
 >
  I am constantly amazed at how
> quiet it is in this project, at least until I remember that
> most of the work is done exclusively via jira, unlike any of
> my other followed projects that use jira.  I'd suggest that
> the right place to hold any discussion is on the dev list,
> but I am not on that list because it receives way too many
> automated notifications.  Maybe it would help discussion on
> dev if notices were sent elsewhere and only discussions were
> held on dev.

I've seen this before on the Maven lists, where there's mostly a stream 
of JIRA changes above anything else:
http://mail-archives.apache.org/mod_mbox/maven-dev/200510.mbox/browser

however, they've got no JIRA issues in their list now, which may imply 
all changes aren't going to the list, or they arent using it so much:
http://mail-archives.apache.org/mod_mbox/maven-dev/201104.mbox/browser

(pause: bisecting their list shows that in 1.mar.06 they forked JIRA to 
a separate list to hide the details of ongoing work)

In some ways it's a means of dealing with a large and fast moving 
codebase: you subscribe to the issues that matter to you, all the 
discussions on a specific feature are archived, etc.

However, it has some flaws
  -discouragement of community, you become a group of people working on 
JIRA issues, rather than on a large integrated project
  -with work spread across common, hdfs and mapreduce JIRAs and mailing 
lists, it's hard to keep all the things in your head -it is pretty much 
a full time job to do so. And I don't know about the others, but I don't 
have the time.
  -we need a way of gently moving people from those who use hadoop to 
those who develop it. To me, every end user is a warm engineering 
resource we just need to point at a problem that they care about. The 
scale of the project, its complexity, JIRA change rate and testing 
difficulties are all barriers to entry -you end up needing a team of people
  * someone to track all the issues and keep the design in their head
  * 1+ person to test
  * 1+ person to code
I don't know about others, but I can't do this on my own.

The attempt to split up into HDFS+MAPREDUCE was one tactic to deal with 
this, but it hasn't worked, we just have more mailing lists to track (or 
in my case, fall behind on).

votewise:

-I'm favour of shipping an apache release of 20.x that has the patches 
that Y! and others have added to deal with scale and availability -and 
which has been tested by them. This will provide an apache release for 
people to use in production systems -because the official apache 
releases have lagged the CDH and Y! releases.

-I'd like to see all the changes integrated into trunk too, as it 
doesn't make sense for a patch in this branch not to be in trunk.

Steve

From general-return-3327-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 17:02:36 2011
Return-Path: <general-return-3327-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 652862FCE
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue,  3 May 2011 17:02:36 +0000 (UTC)
Received: (qmail 65000 invoked by uid 500); 3 May 2011 17:02:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 64803 invoked by uid 500); 3 May 2011 17:02:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 64795 invoked by uid 99); 3 May 2011 17:02:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 17:02:34 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 17:02:28 +0000
Received: by ewy22 with SMTP id 22so131067ewy.35
        for <general@hadoop.apache.org>; Tue, 03 May 2011 10:02:08 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.14.122.81 with SMTP id s57mr17550eeh.195.1304442127760; Tue,
 03 May 2011 10:02:07 -0700 (PDT)
Received: by 10.14.37.13 with HTTP; Tue, 3 May 2011 10:02:06 -0700 (PDT)
In-Reply-To: <BANLkTi=JDBneLBHsCh1T-PF8C7wNbra0kw@mail.gmail.com>
References: <12853C59-08F1-4F98-ACF3-800F06BC7A5F@apache.org>
	<BANLkTinZ=xb6kJ5PTeLN5KKD9b-cwaM0OQ@mail.gmail.com>
	<BANLkTi=1sQoL68k_2UmA1_Oo2CyXsM3fAw@mail.gmail.com>
	<DBCE9300-0D9D-4A9A-99B4-878139A028EE@yahoo-inc.com>
	<61C1E656-0649-4097-8709-FDC596ED57E9@yahoo-inc.com>
	<F7921362-CC1B-435C-890A-E1BD181E03C0@yahoo-inc.com>
	<8D047C89-6430-4FDE-A9C4-CFEE79644AC6@Holsman.NET>
	<BANLkTi=JDBneLBHsCh1T-PF8C7wNbra0kw@mail.gmail.com>
Date: Tue, 3 May 2011 10:02:06 -0700
Message-ID: <BANLkTikevqLp7RyjSBNB1Vxzr-7nbCSXRg@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc0
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

I think we still need to incorporate the patches currently checked
into branch 0.20.  For example, Owen identified a major bug
(BooleanWritable's comparator is broken) and filed a jira
(HADOOP-6928) to put it in branch-0.20, where I reviewed it and
checked it in, so this bug would be fixed in the next stable release.
However this change is not in branch-0.20-security-203. Unless we put
the delta from branch-0.20 into this release, it is missing important
bug fixes that will cause it to regress against 20.3 (if it ever is
released).

I am also nervous about changes like the one identified by
HADOOP-7255. It looks like this change caused a significant regression
in TestDFSIO throughput. It changes the core Task class, the commit
log is a single line, and as far as I can tell it was not discussed or
reviewed by anyone in the community. Don't changes like this at least
deserve a jira before we release them?

Thanks,
Eli

On Tue, May 3, 2011 at 1:39 AM, Konstantin Shvachko
<shv.hadoop@gmail.com> wrote:
> I think its a good idea to release hadoop-0.20.203. It moves Apache Hadoo=
p a
> step forward.
>
> Looks like the technical difficulties are resolved now with latest Arun's
> commits.
> Being a superset of hadoop-0.20.2 it can be considered based on one of th=
e
> official Apache releases.
> I don't think there was a lack of discussions on the lists about the issu=
es
> included in the release candidate. Todd did a thorough review of the enti=
re
> security branch. Many developers participated in discussions.
> Agreeing with Stack I wish HBase was considered a primary target for Hado=
op
> support. But it is not realistic to have it in hadoop-0.20.203.
> I have some experience running a version of this release candidate on a
> large cluster. It works. I would add a couple of patches, which make it r=
un
> on Windows for me like HADOOP-7110, HADOOP-7126. But those are not blocke=
rs.
>
> Thanks,
> --Konstantin
>
>
> On Mon, May 2, 2011 at 5:12 PM, Ian Holsman <hadoop@holsman.net> wrote:
>
>>
>> On May 3, 2011, at 9:58 AM, Arun C Murthy wrote:
>>
>> >>
>> >> Owen, Suresh and I have committed everything on this list except
>> >> HADOOP-6386 and HADOOP-6428. Not sure which of the two are relevant/
>> >> necessary, I'll check with Cos. =A0Other than that hadoop-0.20.203 no=
w a
>> >> superset of hadoop-0.20.2.
>> >>
>> >
>> > Missed adding HADOOP-5759 to that list, I'll check with Amareshwari
>> before committing.
>> >
>> > Arun
>>
>> Thanks for doing this so fast Arun.
>>
>>
>

From general-return-3328-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 18:19:30 2011
Return-Path: <general-return-3328-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5899B2A1F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue,  3 May 2011 18:19:30 +0000 (UTC)
Received: (qmail 5470 invoked by uid 500); 3 May 2011 18:19:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5422 invoked by uid 500); 3 May 2011 18:19:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5414 invoked by uid 99); 3 May 2011 18:19:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 18:19:28 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of eli@cloudera.com designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 18:19:22 +0000
Received: by ewy22 with SMTP id 22so164937ewy.35
        for <general@hadoop.apache.org>; Tue, 03 May 2011 11:19:01 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.14.120.199 with SMTP id p47mr52631eeh.163.1304446741114; Tue,
 03 May 2011 11:19:01 -0700 (PDT)
Received: by 10.14.37.13 with HTTP; Tue, 3 May 2011 11:19:01 -0700 (PDT)
Date: Tue, 3 May 2011 11:19:01 -0700
Message-ID: <BANLkTinb+LA4xXW63c+n_qaPn9MJBsVeFw@mail.gmail.com>
Subject: Questions about the security branches
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hey guys,

Do all changes for 0.20.2xx release go through branch-0.20-security,
then get merged to a particular -2xx branch? Ie to include something
in a 2xx release do you merge from trunk -> -security -> security-2xx?

Why create a new branch for every new dot release? Ie if the intent
that the branch will be dead after release, why not release from a
single branch?  It's hard to see that each branch is a superset of the
previous.

Why is there a new 4th component to the version number?  Shouldn't we
stick with major.minor.revision?  Why not use 0.20.100, Arun's
original proposal?

I noticed a 0.20.205.0 fix version showed up recently.  Where's the
branch for that?

Thanks,
Eli

From general-return-3329-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 20:36:25 2011
Return-Path: <general-return-3329-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 521DB28CC
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue,  3 May 2011 20:36:25 +0000 (UTC)
Received: (qmail 50795 invoked by uid 500); 3 May 2011 20:36:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50751 invoked by uid 500); 3 May 2011 20:36:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50743 invoked by uid 99); 3 May 2011 20:36:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 20:36:23 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [208.97.132.66] (HELO homiemail-a31.g.dreamhost.com) (208.97.132.66)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 20:36:17 +0000
Received: from homiemail-a31.g.dreamhost.com (localhost [127.0.0.1])
	by homiemail-a31.g.dreamhost.com (Postfix) with ESMTP id E9BEF202045
	for <general@hadoop.apache.org>; Tue,  3 May 2011 13:35:56 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; q=dns; s=
	gbiv.com; b=nBD6RtZfqV1I57AFXnWb3TLX9k9OGx3S+nZjRib9gMyr1U5EH2AP
	iNgsTjf9iFI0dfklmp4aInRQye5wSqgpZodNtvLPl+ATYmQIrCAHiN4JYvDXPDN9
	wbRG/IdDAwJojS3FqUXZvuv2SxEG47bn5wFWvIk/F6mFHz1zZFJbsro=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; s=gbiv.com;
	 bh=9wOo7jfaM8DThwmmZtlucPoIk6s=; b=EA3E3GtNFEX3fJbg2TQ46Uh54lLf
	RjkTzBW/238dbL62CVOhrJCK+RZpxYyse3Pn9PJrJ53q6PC3QVJNsi2wHZPANyWk
	5ZvPPtSKp7p2/E6y8QJH3fbKIelPZaQCVr7jl9ZvZIdsZo8m8Yv+cPjBx6naZd44
	h4V0MndIl4zepP0=
Received: from [192.168.1.84] (99-21-208-82.lightspeed.irvnca.sbcglobal.net [99.21.208.82])
	(using TLSv1 with cipher AES128-SHA (128/128 bits))
	(No client certificate requested)
	(Authenticated sender: fielding@gbiv.com)
	by homiemail-a31.g.dreamhost.com (Postfix) with ESMTPSA id B0AA3202044
	for <general@hadoop.apache.org>; Tue,  3 May 2011 13:35:56 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: Discussions - Re: [VOTE] Release candidate 0.20.203.0-rc0
From: "Roy T. Fielding" <fielding@gbiv.com>
In-Reply-To: <81841C06-1C7B-4BAB-BF74-85B75D155C1D@gbiv.com>
Date: Tue, 3 May 2011 13:35:56 -0700
Content-Transfer-Encoding: 7bit
Message-Id: <878CEE04-B352-45B0-B399-FB53D6A7C2CB@gbiv.com>
References: <12853C59-08F1-4F98-ACF3-800F06BC7A5F@apache.org> <4DBEF0C1.2050306@apache.org> <05E53D57-D53A-4D05-AAA4-E99A12373ED2@Holsman.NET> <81841C06-1C7B-4BAB-BF74-85B75D155C1D@gbiv.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)

PLEASE NOTE

Voting +1 for a release means that you have downloaded the
source code package, verified its signatures, compiled it
on your platform of choice, and checked to your satisfaction
that it matches the source code we have in subversion and that
is is better (in your opinion) than the last Apache release
of the same name.

The ASF relies on that minimum amount of peer review to make
sure that we don't release trojan horses, license violations,
or other things that might get us sued as a foundation or as
individuals.  If you don't have time to do it yourself, then
vote +0 (with happy feelings) and hope that there are at least
three members of the PMC who do have that time.

DO NOT +1 a release just because it seems like progress.
Progress is in the doing, not the talking.

....Roy

From general-return-3330-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 21:04:47 2011
Return-Path: <general-return-3330-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DF1EB2FE5
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue,  3 May 2011 21:04:46 +0000 (UTC)
Received: (qmail 99831 invoked by uid 500); 3 May 2011 21:04:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99758 invoked by uid 500); 3 May 2011 21:04:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 99750 invoked by uid 99); 3 May 2011 21:04:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 21:04:45 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 209.85.215.176 as permitted sender)
Received: from [209.85.215.176] (HELO mail-ey0-f176.google.com) (209.85.215.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 21:04:38 +0000
Received: by eya28 with SMTP id 28so234624eya.35
        for <general@hadoop.apache.org>; Tue, 03 May 2011 14:04:18 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.14.122.81 with SMTP id s57mr171638eeh.195.1304456658139; Tue,
 03 May 2011 14:04:18 -0700 (PDT)
Received: by 10.14.37.13 with HTTP; Tue, 3 May 2011 14:04:18 -0700 (PDT)
In-Reply-To: <93B30E4D-73D3-495D-9BDD-D6E697BB9FA3@apache.org>
References: <BANLkTi=kPh42QkdnAJti+gZk3nNUPc8jMQ@mail.gmail.com>
	<93B30E4D-73D3-495D-9BDD-D6E697BB9FA3@apache.org>
Date: Tue, 3 May 2011 14:04:18 -0700
Message-ID: <BANLkTikUTuxBAZW-9NqmU9JVNgpQHfckBA@mail.gmail.com>
Subject: Re: Questions wrt security branches
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Tue, May 3, 2011 at 1:31 PM, Owen O'Malley <omalley@apache.org> wrote:
>
> On May 3, 2011, at 9:35 AM, Eli Collins wrote:
>
>> Do all changes for 0.20.2xx release go through branch-0.20-security,
>> then get merged to a particular -2xx branch?
>
> I've discussed this before on the lists, =A0but here goes:
>

Thanks.  I should have mentioned that I followed the thread below but
it was not clear that eg all changes have to go through -security to
ensure a -20(n+1) branch has the changes from -20n.

http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201103.mbox/%3CE=
59D9A63-CC4F-4E00-97C0-B11C84C23825@apache.org%3E

> branch-0.20-security is the major branch and all changes need to be commi=
tted to it.
>

Thanks for clarifying.

> The branches off of branch-0.20-security, namely branch-0.20-security-203=
 and branch-0.20-security-204 are the minor branches, which are branched of=
f of branch-0.20-security every month or two. Within a minor branch there a=
re only bug fixes.
>

Why the new branches instead of releasing from branch-0.20-security
the way previous release branches (eg branch-0.20) have worked?

> So this release, we are trying to get out the door is 0.20.203.0. A bug f=
ix to it would go into 0.20.203.1. New features like disk fail in place go =
into 0.20.204.0.
>

Shouldn't new features go to trunk and then the next minor release?

According to the project's current version scheme
(http://wiki.apache.org/hadoop/Roadmap eg major.minor.point), a point
release (eg the 203 in 0.20.203) "do not introduce new features or
make other improvements other than fixing bugs."

Is this a proposal to make branch-0.20-security a feature branch that
does releases, and the new version number is necessary to allow for
new features?  Doesn't this allow for releases from
branch-0.20-security and trunk to diverge in terms of feature set?

I think people would prefer we spend our energy putting new features
(not yet in trunk) in trunk so we don't create divergent releases.
Creating a new branch for new features seems like it will prolong our
current situation.

>> Why is there a new 4th component to the version number?
>
> The problem is that we need minor versions and there isn't space in the c=
urrent scheme. It would probably be clearer, if we called this release 1.0.=
 Then this looks like:
>
> branch-1
> branch-1.0
> branch-1.1
>
> with point releases off of it. When I floated the idea of using 1.0 last =
time, there was more consensus around using the 0.20.20X.Y naming.
>

I remember there being resistance to calling it 1.0 but I don't
remember consensus on a new feature branch or 4 part version scheme.

Not sure mirroring the YDH version numbers is what's most sense for an
Apache release:
http://www.flickr.com/photos/steve_l/5560919873/in/set-72157626356732562
Arun's proposal of using 0.20.100 seemed more logical.

>> I noticed a 0.20.205.0 fix version showed up recently. =A0Where's the
>> branch for that?
>
> It hasn't branched yet, but it will come off of branch-0.20-security.
>

So a fix for 0.20.205.0 (like any fix for a version not yet branched)
only gets checked into branch-0.20-security. Makes sense.

Thanks for the clarifications.

Thanks,
Eli

From general-return-3331-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 23:59:39 2011
Return-Path: <general-return-3331-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3E70B2BE7
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue,  3 May 2011 23:59:39 +0000 (UTC)
Received: (qmail 79712 invoked by uid 500); 3 May 2011 23:59:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79658 invoked by uid 500); 3 May 2011 23:59:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 79650 invoked by uid 99); 3 May 2011 23:59:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 23:59:37 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 23:59:30 +0000
Received: by bwz8 with SMTP id 8so989493bwz.35
        for <general@hadoop.apache.org>; Tue, 03 May 2011 16:59:10 -0700 (PDT)
Received: by 10.204.8.141 with SMTP id h13mr465188bkh.64.1304467150329; Tue,
 03 May 2011 16:59:10 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.205.81.143 with HTTP; Tue, 3 May 2011 16:58:50 -0700 (PDT)
In-Reply-To: <BANLkTikevqLp7RyjSBNB1Vxzr-7nbCSXRg@mail.gmail.com>
References: <12853C59-08F1-4F98-ACF3-800F06BC7A5F@apache.org>
 <BANLkTinZ=xb6kJ5PTeLN5KKD9b-cwaM0OQ@mail.gmail.com> <BANLkTi=1sQoL68k_2UmA1_Oo2CyXsM3fAw@mail.gmail.com>
 <DBCE9300-0D9D-4A9A-99B4-878139A028EE@yahoo-inc.com> <61C1E656-0649-4097-8709-FDC596ED57E9@yahoo-inc.com>
 <F7921362-CC1B-435C-890A-E1BD181E03C0@yahoo-inc.com> <8D047C89-6430-4FDE-A9C4-CFEE79644AC6@Holsman.NET>
 <BANLkTi=JDBneLBHsCh1T-PF8C7wNbra0kw@mail.gmail.com> <BANLkTikevqLp7RyjSBNB1Vxzr-7nbCSXRg@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Tue, 3 May 2011 16:58:50 -0700
Message-ID: <BANLkTimKKbkuPCz61TU=8-nO8Z6PYhf7Sg@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc0
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00151743fa561dd8b104a267edaf
X-Virus-Checked: Checked by ClamAV on apache.org

--00151743fa561dd8b104a267edaf
Content-Type: text/plain; charset=ISO-8859-1

Just to gauge what amount of stuff is in branch-0.20-security-203 I wrote a
quick script which does a comparison based on JIRAs mention in the commit
log. It output the following list of JIRAs that are in the branch but not
committed to trunk. I've marked many as N/A meaning that they don't apply to
trunk:

< HADOOP-1026
< HADOOP-6304 N/A
< HADOOP-6544
< HADOOP-6598 N/A
< HADOOP-6638
< HADOOP-6653 N/A
< HADOOP-6716 N/A
 < HADOOP-6718 N/A
< HADOOP-6728
< HADOOP-6745
< HADOOP-6757
< HADOOP-6776 N/A
< HADOOP-6808
< HADOOP-6810 N/A
< HADOOP-6832
< HADOOP-6855 N/A
< HADOOP-7143 N/A
< HADOOP-7190 N/A
< HADOOP-7232 N/A
< HADOOP-7243 N/A
< HADOOP-7246 (not sure about this)
< HADOOP-7247
< HADOOP-7253
< HDFS-1020 N/A
< HDFS-1022
< HDFS-1136 N/A
< HDFS-1153
< HDFS-1158
< HDFS-1313 N/A
< HDFS-495 N/A
< HDFS-6599 <-- must be a typo in commit log
< HDFS-740  N/A
< MAPREDUCE-1088
< MAPREDUCE-1100
< MAPREDUCE-1118
< MAPREDUCE-1207
< MAPREDUCE-1361 N/A
< MAPREDUCE-1376 N/A
< MAPREDUCE-1442 N/A
< MAPREDUCE-1521
< MAPREDUCE-1526 N/A
< MAPREDUCE-1550 N/A
< MAPREDUCE-1594 N/A
< MAPREDUCE-1641
< MAPREDUCE-1671
< MAPREDUCE-1672
< MAPREDUCE-1676
< MAPREDUCE-1677
< MAPREDUCE-1682
< MAPREDUCE-1687
< MAPREDUCE-1699 Unclear
< MAPREDUCE-1711 N/A
< MAPREDUCE-1713
< MAPREDUCE-1716
< MAPREDUCE-1730
< MAPREDUCE-1741
< MAPREDUCE-1744
< MAPREDUCE-1758
< MAPREDUCE-1759 N/A
< MAPREDUCE-1778 N/A
< MAPREDUCE-1790
< MAPREDUCE-1807 N/A
< MAPREDUCE-1854
< MAPREDUCE-1871
< MAPREDUCE-1872
< MAPREDUCE-1882
< MAPREDUCE-1889
< MAPREDUCE-1890
< MAPREDUCE-1914
< MAPREDUCE-1921
< MAPREDUCE-1933
< MAPREDUCE-1934
< MAPREDUCE-1938
< MAPREDUCE-1943
< MAPREDUCE-1954
< MAPREDUCE-1955
< MAPREDUCE-1960
< MAPREDUCE-1964
< MAPREDUCE-1966
< MAPREDUCE-1971
< MAPREDUCE-2005 N/A
< MAPREDUCE-2019
< MAPREDUCE-2055
< MAPREDUCE-2316
< MAPREDUCE-2355
< MAPREDUCE-291
< MAPREDUCE-323
< MAPREDUCE-339
< MAPREDUCE-517
< MAPREDUCE-6419 <-- doesnt exist, probably typo

Certainly some of these are new test cases, benchmark improvements, or
system tests. But many others are large new features (e.g new metrics
framework, separate JobHistory service). Others also introduce new
configurations (eg new JT based limits). In the list above there are 58 that
seem to be applicable, probably at least half of which are non-test code.

This list above doesn't include 192 patches that were committed to the
branch without reference to any JIRA in the commit message:
todd@todd-w510:~/git/hadoop-common$ for x in $(git rev-list
origin/branch-0.20..origin/branch-0.20-security-203 -- src) ; do git log -n1
$x | egrep -q '(MAPREDUCE|HDFS|HADOOP)[-:][0-9]+' || echo $x ; done | wc -l
192
Browsing through these, many have already been forward ported, or at least
had corresponding JIRAs opened. But it's very difficult to match them up and
evaluate which ones have been committed. Eli pointed out one earlier this
week that was done by a non-committer with no public review that introduced
an apparent performance regression; it's difficult to know whether there
might be others as well.

Rather than being a "maintenance release" (as is usually expected when
incrementing the third component of a version string) this is essentially a
separate trunk off of 0.20. I agree that the advancements in this branch are
many, and are a great set of contributions for the community. User limits
and security are two such that have been cited in this thread; unfortunately
the new improvements in limits haven't been committed to trunk, and the
security in trunk has a known root exploit. Do users really want to see us
putting these things in 20 without making sure they'll also show up in
future releases?

Looking at recent history of 204 it seems some more patches have gone in
there before going into trunk as well - for example MR-2429. Arun and Sid
are working on forward-porting it, and it's obviously not due to any kind of
bad intent that it was missed, but it underscores the dangers of having
essentially two trunks in ASF. I completely agree that there should be
long-term maintenance branches at the ASF, but we need to establish a clear
process to make sure that "maintenance" doesn't diverge into something else.

Here are two requests that others have made but I haven't seen an answer to
yet:

- Document the criteria by which developers can judge whether an improvement
should be included in branch-0.20-security. The inclusion criteria for the
branch as it stands is not clear -- given this branch's lineage, it clearly
used to be "things important for the Yahoo clusters", but that doesn't seem
like a reasonable community criterion. Up until now in Hadoop's history, the
criteria has always been "compatible bug fixes only", which doesn't describe
this branch either.

- Clearly establish the process that all patches must either be committed to
trunk first (and then backported), or include a comment on the JIRA
explaining why this is not necessary. Additionally we should decide whether
patches must be backported "through" 0.22 or if they may skip back from
trunk to 20-security. (I'm assuming 21 is dead here)

Perhaps this could go on a wiki page (or web site page) regarding the
currently active branches?

Thanks
-Todd

On Tue, May 3, 2011 at 10:02 AM, Eli Collins <eli@cloudera.com> wrote:

> I think we still need to incorporate the patches currently checked
> into branch 0.20.  For example, Owen identified a major bug
> (BooleanWritable's comparator is broken) and filed a jira
> (HADOOP-6928) to put it in branch-0.20, where I reviewed it and
> checked it in, so this bug would be fixed in the next stable release.
> However this change is not in branch-0.20-security-203. Unless we put
> the delta from branch-0.20 into this release, it is missing important
> bug fixes that will cause it to regress against 20.3 (if it ever is
> released).
>
> I am also nervous about changes like the one identified by
> HADOOP-7255. It looks like this change caused a significant regression
> in TestDFSIO throughput. It changes the core Task class, the commit
> log is a single line, and as far as I can tell it was not discussed or
> reviewed by anyone in the community. Don't changes like this at least
> deserve a jira before we release them?
>
> Thanks,
> Eli
>
> On Tue, May 3, 2011 at 1:39 AM, Konstantin Shvachko
> <shv.hadoop@gmail.com> wrote:
> > I think its a good idea to release hadoop-0.20.203. It moves Apache
> Hadoop a
> > step forward.
> >
> > Looks like the technical difficulties are resolved now with latest Arun's
> > commits.
> > Being a superset of hadoop-0.20.2 it can be considered based on one of
> the
> > official Apache releases.
> > I don't think there was a lack of discussions on the lists about the
> issues
> > included in the release candidate. Todd did a thorough review of the
> entire
> > security branch. Many developers participated in discussions.
> > Agreeing with Stack I wish HBase was considered a primary target for
> Hadoop
> > support. But it is not realistic to have it in hadoop-0.20.203.
> > I have some experience running a version of this release candidate on a
> > large cluster. It works. I would add a couple of patches, which make it
> run
> > on Windows for me like HADOOP-7110, HADOOP-7126. But those are not
> blockers.
> >
> > Thanks,
> > --Konstantin
> >
> >
> > On Mon, May 2, 2011 at 5:12 PM, Ian Holsman <hadoop@holsman.net> wrote:
> >
> >>
> >> On May 3, 2011, at 9:58 AM, Arun C Murthy wrote:
> >>
> >> >>
> >> >> Owen, Suresh and I have committed everything on this list except
> >> >> HADOOP-6386 and HADOOP-6428. Not sure which of the two are relevant/
> >> >> necessary, I'll check with Cos.  Other than that hadoop-0.20.203 now
> a
> >> >> superset of hadoop-0.20.2.
> >> >>
> >> >
> >> > Missed adding HADOOP-5759 to that list, I'll check with Amareshwari
> >> before committing.
> >> >
> >> > Arun
> >>
> >> Thanks for doing this so fast Arun.
> >>
> >>
> >
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--00151743fa561dd8b104a267edaf--

From general-return-3332-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 00:16:54 2011
Return-Path: <general-return-3332-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C27652A58
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  4 May 2011 00:16:54 +0000 (UTC)
Received: (qmail 97513 invoked by uid 500); 4 May 2011 00:16:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97459 invoked by uid 500); 4 May 2011 00:16:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97450 invoked by uid 99); 4 May 2011 00:16:53 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 00:16:53 +0000
Received: from localhost (HELO [192.168.168.16]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 00:16:53 +0000
Message-ID: <4DC09AF6.70403@apache.org>
Date: Tue, 03 May 2011 17:16:54 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.14) Gecko/20110223 Thunderbird/3.1.8
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc0
References: <12853C59-08F1-4F98-ACF3-800F06BC7A5F@apache.org> <4DBEF0C1.2050306@apache.org> <BB3A69B5-9A5A-4454-80C8-62EBD3CF47EC@yahoo-inc.com> <4DBF16CF.3030905@apache.org> <247DDA36-BCD7-4EE9-A8CD-AD8E62C94F16@yahoo-inc.com> <4DBF2040.90602@apache.org> <5F90B548-37ED-4DB8-AAE3-D1C2BB5C700A@yahoo-inc.com>
In-Reply-To: <5F90B548-37ED-4DB8-AAE3-D1C2BB5C700A@yahoo-inc.com>
X-Enigmail-Version: 1.1.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On 05/02/2011 02:33 PM, Arun C Murthy wrote:
> Are you simply asking for someone to go through the 450 odd jiras and
> set 'fix-for' fields?

Every other release we've made is well-correlated with Jira.  It should
not be difficult to achieve that for this one.  We could write a script
to take all 450 bug IDs from the change log and use Jira's command-line
tool to set the "fix-for" to be this 0.20+security release.  Would you
like help with that?

Doug

From general-return-3333-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 01:02:15 2011
Return-Path: <general-return-3333-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6B0582463
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  4 May 2011 01:02:15 +0000 (UTC)
Received: (qmail 29074 invoked by uid 500); 4 May 2011 01:02:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29019 invoked by uid 500); 4 May 2011 01:02:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29011 invoked by uid 99); 4 May 2011 01:02:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 01:02:13 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 01:02:03 +0000
Received: from SP2-EX07CAS05.ds.corp.yahoo.com (sp2-ex07cas05.corp.sp2.yahoo.com [98.137.59.39])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p4411HBb000726
	for <general@hadoop.apache.org>; Tue, 3 May 2011 18:01:17 -0700 (PDT)
Received: from SP2-EX07VS04.ds.corp.yahoo.com ([98.137.59.33]) by
 SP2-EX07CAS05.ds.corp.yahoo.com ([98.137.59.39]) with mapi; Tue, 3 May 2011
 18:01:17 -0700
From: Arun C Murthy <arunc@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 3 May 2011 18:01:12 -0700
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc0
Thread-Topic: [VOTE] Release candidate 0.20.203.0-rc0
Thread-Index: AcwJ9sVyeKRCnqhPTO655sAH70KJiQ==
Message-ID: <EDC1C3A4-003F-494E-B335-2D0CB1638738@yahoo-inc.com>
References: <12853C59-08F1-4F98-ACF3-800F06BC7A5F@apache.org>
 <4DBEF0C1.2050306@apache.org>
 <BB3A69B5-9A5A-4454-80C8-62EBD3CF47EC@yahoo-inc.com>
 <4DBF16CF.3030905@apache.org>
 <247DDA36-BCD7-4EE9-A8CD-AD8E62C94F16@yahoo-inc.com>
 <4DBF2040.90602@apache.org>
 <5F90B548-37ED-4DB8-AAE3-D1C2BB5C700A@yahoo-inc.com>
 <4DC09AF6.70403@apache.org>
In-Reply-To: <4DC09AF6.70403@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

On May 3, 2011, at 5:17 PM, "Doug Cutting" <cutting@apache.org> wrote:

> On 05/02/2011 02:33 PM, Arun C Murthy wrote:
>> Are you simply asking for someone to go through the 450 odd jiras and
>> set 'fix-for' fields?
>=20
> Every other release we've made is well-correlated with Jira.  It should
> not be difficult to achieve that for this one.  We could write a script
> to take all 450 bug IDs from the change log and use Jira's command-line
> tool to set the "fix-for" to be this 0.20+security release.  Would you
> like help with that?
>=20

Yes please, that would be great. Thanks!

Arun=20

From general-return-3334-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 17:32:46 2011
Return-Path: <general-return-3334-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C26E93830
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  4 May 2011 17:32:46 +0000 (UTC)
Received: (qmail 95260 invoked by uid 500); 4 May 2011 17:32:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94934 invoked by uid 500); 4 May 2011 17:32:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94925 invoked by uid 99); 4 May 2011 17:32:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 17:32:44 +0000
X-ASF-Spam-Status: No, hits=2.9 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 17:32:36 +0000
Received: from battlerock-lm.corp.yahoo.com (battlerock-lm.corp.yahoo.com [10.72.123.81])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p44HVt79014581
	for <general@hadoop.apache.org>; Wed, 4 May 2011 10:31:56 -0700 (PDT)
From: "Owen O'Malley" <omalley@apache.org>
Content-Type: multipart/alternative; boundary=Apple-Mail-1--194398535
Subject: [VOTE] Release candidate 0.20.203.0-rc1
Date: Wed, 4 May 2011 10:31:55 -0700
Message-Id: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org>
To: general@hadoop.apache.org
Mime-Version: 1.0 (Apple Message framework v1084)
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-1--194398535
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

Here's an updated release candidate for 0.20.203.0. I've incorporated =
the feedback and included all of the patches from 0.20.2, which is the =
last stable release. I also fixed the eclipse-plugin problem.=20

The candidate is at: =
http://people.apache.org/~omalley/hadoop-0.20.203.0-rc1/

Please download it, inspect it, compile it, and test it. Clearly, I'm =
+1.

-- Owen=

--Apple-Mail-1--194398535--

From general-return-3335-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 19:18:08 2011
Return-Path: <general-return-3335-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A70333CF6
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  4 May 2011 19:18:08 +0000 (UTC)
Received: (qmail 49644 invoked by uid 500); 4 May 2011 19:18:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49568 invoked by uid 500); 4 May 2011 19:18:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49560 invoked by uid 99); 4 May 2011 19:18:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 19:18:07 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 19:17:59 +0000
Received: by ewy22 with SMTP id 22so645158ewy.35
        for <general@hadoop.apache.org>; Wed, 04 May 2011 12:17:39 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.14.5.200 with SMTP id 48mr725501eel.131.1304536658555; Wed, 04
 May 2011 12:17:38 -0700 (PDT)
Received: by 10.14.37.13 with HTTP; Wed, 4 May 2011 12:17:38 -0700 (PDT)
In-Reply-To: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org>
References: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org>
Date: Wed, 4 May 2011 12:17:38 -0700
Message-ID: <BANLkTikc3OkoYjoOUyATXVfOt+=a2W8d_Q@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Wed, May 4, 2011 at 10:31 AM, Owen O'Malley <omalley@apache.org> wrote:
> Here's an updated release candidate for 0.20.203.0. I've incorporated the feedback and included all of the patches from 0.20.2, which is the last stable release. I also fixed the eclipse-plugin problem.
>
> The candidate is at: http://people.apache.org/~omalley/hadoop-0.20.203.0-rc1/
>
> Please download it, inspect it, compile it, and test it. Clearly, I'm +1.
>
> -- Owen

Hey Owen,

Thanks for incorporating all the feedback and additional changes. It's
great that this release won't be a regression against our previous
stable release.

I would like to call out that we are not just voting to adopt a
particular release, we are starting a new version scheme for the
project, doing new feature development on maintenance release branches
(before trunk), and we're saying it's OK to release software that
hasn't been reviewed by the community.

I'd like to hear from our development community not just that we want
to do a release from this branch but that we want to adopt these other
changes as well. Here's a summary of the major *remaining* issues and
a recommendation on how to proceed:

1. There are about ~50 changes that have jiras that are committed to
the branch that are not yet in trunk. The next release (0.22) will be
a regression against this release, with respect to these particular
changes. Recomendation: we should get these changes in trunk before
releasing so that new features do not show up in maintenace branches
first.

2. There are 192 patches that were committed to the branch without
reference to any Jira in the commit message. Some of these may have
already been forward ported, but it is very difficult to match them up
and evaluate which ones have been committed. Some are troublesome,
when spot checking the commits I found some that have been done by
non-committers with no public review that introduced an apparent
performance regressions (eg see HADOOP-7255). Recommendation: we
should update the commit log to make sure there is a jira for each
issue, and all changes have been reviewed/committed. This is the way
we've always done releases.

3. The new versioning scheme major.minor.point.X the new "X" component
allows for new feature development on point releases. Recomendation:
we should discuss in a separate thread whether we want to do new
feature development on maintenance branches and if so to adopt this
new version scheme.

Thanks,
Eli

From general-return-3336-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 20:14:06 2011
Return-Path: <general-return-3336-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1290831D2
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  4 May 2011 20:14:06 +0000 (UTC)
Received: (qmail 44155 invoked by uid 500); 4 May 2011 20:14:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44095 invoked by uid 500); 4 May 2011 20:14:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44087 invoked by uid 99); 4 May 2011 20:14:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 20:14:04 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 20:13:59 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=NAvODEm6X3mgKowqZtbwzHYZRbwhSZkPf6IMQgIzJcYjpoVWojH+b0ze
   Rk5H7ZrW8MSHvkxlEwdVPlpLD9TaPh3McrFrT+NIYVf63HYKcX9kw2WLR
   DmijQOGjQOrV4nP;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1304540039; x=1336076039;
  h=from:to:subject:date:message-id:references:in-reply-to:
   content-id:content-transfer-encoding:mime-version;
  bh=5xMJDnQiwZjegL3F9xjP/eBLAbTcqlXOugVVwoauiJk=;
  b=FYNM0sBG7tjSFThTkV72MKkkd0LhuYh+WtrZhtWY4YpC8KO2A6TecsfD
   c6phK6c3ZxKYDgfbFkN96V6r9uc89byjtiLeyBlWtaCds4pS6Ke+A2136
   ijZf0N1DG04tqM4;
X-IronPort-AV: E=Sophos;i="4.64,316,1301900400"; 
   d="scan'208";a="22852564"
Received: from ESV4-EXC03.linkedin.biz ([fe80::985a:a6b4:f1e1:23b0]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi id 14.01.0251.000; Wed, 4
 May 2011 13:17:03 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
Thread-Topic: [VOTE] Release candidate 0.20.203.0-rc1
Thread-Index: AQHMCoHKs40jdRozskC+8soYJjDqwpR9j9WA
Date: Wed, 4 May 2011 20:17:02 +0000
Message-ID: <450B3108-28FA-40B2-A7F3-28747B8D2863@linkedin.com>
References: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org>
In-Reply-To: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <A53B0B7A9CD0BE4AA6B7A9D8CD0DB497@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


On May 4, 2011, at 10:31 AM, Owen O'Malley wrote:

> Here's an updated release candidate for 0.20.203.0. I've incorporated the=
 feedback and included all of the patches from 0.20.2, which is the last st=
able release. I also fixed the eclipse-plugin problem.=20
>=20
> The candidate is at: http://people.apache.org/~omalley/hadoop-0.20.203.0-=
rc1/
>=20
> Please download it, inspect it, compile it, and test it. Clearly, I'm +1.


	Am I misreading this, or are the MR protocols out of sync between 0.20.203=
 and 0.21?  It would also appear that this is marked stable in 0.21. What i=
s the user impact?



From general-return-3337-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 20:27:19 2011
Return-Path: <general-return-3337-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E39A538D9
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  4 May 2011 20:27:18 +0000 (UTC)
Received: (qmail 77817 invoked by uid 500); 4 May 2011 20:27:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77747 invoked by uid 500); 4 May 2011 20:27:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77739 invoked by uid 99); 4 May 2011 20:27:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 20:27:17 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 20:27:12 +0000
Received: from [10.0.1.3] (snvvpn4-10-72-168-c8.hq.corp.yahoo.com [10.72.168.8])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p44KQZMC094736
	for <general@hadoop.apache.org>; Wed, 4 May 2011 13:26:37 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1304540797;
	bh=DvigmNJ2PyPP3nX1Q/7pnM1OgqAlHkFnvN8qLL+KVds=;
	h=Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:
	 Content-Transfer-Encoding:Message-Id:References:To;
	b=Bn36eQvUxBDlXTGppgUG+JlL0ykV9KyfoyTqJb+8SPcwh3KrnqOUIEvdTZn6IsCMc
	 8dYHPwv96/SZopOLfVvsQTrU1GkGaxEBawgpTzU3sSo7nlURt19jy31a0HQ/JR79am
	 0E7AjkwNFIPoHlKR6tvyNvcCcnILCuvoXPfamKdY=
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <BANLkTikc3OkoYjoOUyATXVfOt+=a2W8d_Q@mail.gmail.com>
Date: Wed, 4 May 2011 13:26:35 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <69F92CFE-509C-4163-ACC4-7A11640FB7C1@yahoo-inc.com>
References: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org> <BANLkTikc3OkoYjoOUyATXVfOt+=a2W8d_Q@mail.gmail.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1084)

Hi Folks,

This is a release vote, let's stay focused.  On this thread I think =
appropriate responses are either=20

+1 and some short commentary  (assuming you've tried it and it works)

or

-1 and some short commentary.  It would also be cool if you noted if =
you've tried it.

----

In the spirit of my feedback, I'll respond to this under another =
subject.

Thanks,

E14

On May 4, 2011, at 12:17 PM, Eli Collins wrote:

> On Wed, May 4, 2011 at 10:31 AM, Owen O'Malley <omalley@apache.org> =
wrote:
>> Here's an updated release candidate for 0.20.203.0. I've incorporated =
the feedback and included all of the patches from 0.20.2, which is the =
last stable release. I also fixed the eclipse-plugin problem.
>>=20
>> The candidate is at: =
http://people.apache.org/~omalley/hadoop-0.20.203.0-rc1/
>>=20
>> Please download it, inspect it, compile it, and test it. Clearly, I'm =
+1.
>>=20
>> -- Owen
>=20
> Hey Owen,
>=20
> Thanks for incorporating all the feedback and additional changes. It's
> great that this release won't be a regression against our previous
> stable release.
>=20
> I would like to call out that we are not just voting to adopt a
> particular release, we are starting a new version scheme for the
> project, doing new feature development on maintenance release branches
> (before trunk), and we're saying it's OK to release software that
> hasn't been reviewed by the community.
>=20
> I'd like to hear from our development community not just that we want
> to do a release from this branch but that we want to adopt these other
> changes as well. Here's a summary of the major *remaining* issues and
> a recommendation on how to proceed:
>=20
> 1. There are about ~50 changes that have jiras that are committed to
> the branch that are not yet in trunk. The next release (0.22) will be
> a regression against this release, with respect to these particular
> changes. Recomendation: we should get these changes in trunk before
> releasing so that new features do not show up in maintenace branches
> first.
>=20
> 2. There are 192 patches that were committed to the branch without
> reference to any Jira in the commit message. Some of these may have
> already been forward ported, but it is very difficult to match them up
> and evaluate which ones have been committed. Some are troublesome,
> when spot checking the commits I found some that have been done by
> non-committers with no public review that introduced an apparent
> performance regressions (eg see HADOOP-7255). Recommendation: we
> should update the commit log to make sure there is a jira for each
> issue, and all changes have been reviewed/committed. This is the way
> we've always done releases.
>=20
> 3. The new versioning scheme major.minor.point.X the new "X" component
> allows for new feature development on point releases. Recomendation:
> we should discuss in a separate thread whether we want to do new
> feature development on maintenance branches and if so to adopt this
> new version scheme.
>=20
> Thanks,
> Eli


From general-return-3338-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 22:03:59 2011
Return-Path: <general-return-3338-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E233C3C46
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  4 May 2011 22:03:58 +0000 (UTC)
Received: (qmail 92347 invoked by uid 500); 4 May 2011 22:03:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92307 invoked by uid 500); 4 May 2011 22:03:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 92299 invoked by uid 99); 4 May 2011 22:03:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:03:57 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:03:49 +0000
Received: by ewy22 with SMTP id 22so708339ewy.35
        for <general@hadoop.apache.org>; Wed, 04 May 2011 15:03:29 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.14.40.156 with SMTP id f28mr798303eeb.35.1304546609375; Wed,
 04 May 2011 15:03:29 -0700 (PDT)
Received: by 10.14.37.13 with HTTP; Wed, 4 May 2011 15:03:29 -0700 (PDT)
In-Reply-To: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org>
References: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org>
Date: Wed, 4 May 2011 15:03:29 -0700
Message-ID: <BANLkTik5gj0H4gmvy14Ux=g_r5Tt6D1=MA@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Wed, May 4, 2011 at 10:31 AM, Owen O'Malley <omalley@apache.org> wrote:
> Here's an updated release candidate for 0.20.203.0. I've incorporated the feedback and included all of the patches from 0.20.2, which is the last stable release. I also fixed the eclipse-plugin problem.
>
> The candidate is at: http://people.apache.org/~omalley/hadoop-0.20.203.0-rc1/
>
> Please download it, inspect it, compile it, and test it. Clearly, I'm +1.
>
> -- Owen

While rc2 is an improvement on rc1, I am -1 on this particular rc.  Rationale:

This rc contains many patches not yet committed to trunk. This would
cause the next major release (0.22) to be a feature regression against
our latest stable release (203), were 0.22 released soon.

This rc contains many patches not yet reviewed by the community via
the normal process (jira, patch against trunk, merge to a release
branch). I think we should respect the existing community process that
has been used for all previous releases.

This rc introduces a new development and braching model (new feature
development outside trunk) and Hadoop versioning scheme without
sufficient discussion or proposal of these changes with the community.

We should establish new process before the release, a release is not
the appropriate mechanism for changing our review and development
process or versioning .

I do support a release from branch-0.20-security that follows the
existing, established community process.

Thanks,
Eli

From general-return-3339-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 22:06:47 2011
Return-Path: <general-return-3339-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4C0DC3FA8
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  4 May 2011 22:06:47 +0000 (UTC)
Received: (qmail 97937 invoked by uid 500); 4 May 2011 22:06:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97887 invoked by uid 500); 4 May 2011 22:06:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97879 invoked by uid 99); 4 May 2011 22:06:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:06:45 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:06:39 +0000
Received: from l2tp-8-201.corp.yahoo.com (l2tp-8-201.corp.yahoo.com [10.73.8.201])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p44M5rNU072803
	for <general@hadoop.apache.org>; Wed, 4 May 2011 15:05:53 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1304546753;
	bh=u2CKIg3Z1PzjLQRll70R2mLSksVQuamfoIWp85SNFRM=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=EiQrKuwhlE/z9lq4BsYqGoOpuaZSzhaWP6DB5QfvIl/VAz6Rv//YSu8JfTnGsLjl6
	 VVh+66IdZJfM78RGlRJYuwhPtC/JBj0kvsa/E0Vtw8TT6IqfhM9/nen4S8DxJQiaYN
	 lu7NGB92VTXg3l4mEMXmyiFFXFSpMUPAU7WYd0m8=
Message-Id: <590F865D-2DF9-4A72-A7BC-2EE5EE9CB24A@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
Date: Wed, 4 May 2011 15:05:52 -0700
References: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

On May 4, 2011, at 10:31 AM, Owen O'Malley wrote:

> Here's an updated release candidate for 0.20.203.0. I've  
> incorporated the feedback and included all of the patches from  
> 0.20.2, which is the last stable release. I also fixed the eclipse- 
> plugin problem.
>
> The candidate is at: http://people.apache.org/~omalley/hadoop-0.20.203.0-rc1/
>
> Please download it, inspect it, compile it, and test it. Clearly,  
> I'm +1.
>

+1

Downloaded release, checked checksums, built, deployed single-node  
cluster.

Arun


From general-return-3340-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 22:07:10 2011
Return-Path: <general-return-3340-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1A54C3FDD
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  4 May 2011 22:07:10 +0000 (UTC)
Received: (qmail 806 invoked by uid 500); 4 May 2011 22:07:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 760 invoked by uid 500); 4 May 2011 22:07:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 752 invoked by uid 99); 4 May 2011 22:07:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:07:08 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:07:02 +0000
Received: from sp1-ex07cas01.ds.corp.yahoo.com (sp1-ex07cas01.ds.corp.yahoo.com [216.252.116.137])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p44M6Iov073016
	for <general@hadoop.apache.org>; Wed, 4 May 2011 15:06:18 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1304546778;
	bh=9L2fMYMjkerauRs/hx64E0y1oWXps2AIQHFl0XnDuCs=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:MIME-Version;
	b=FFNJKF6RB9Tuw+Li2a3xypBRsqSVyzAqvVBZjH+0afmVHEZvScNxwW+kqkA8RdUwQ
	 GqdPo9fdMKCzQmOk49fM0sBLx+0S/Wn4KTHyHm64vUAEnE2ChJProX15RprrD/PzaR
	 NtJMqUyE+GPequM1H31bNj84vvBUbQCKqtAu7gwI=
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas01.ds.corp.yahoo.com ([216.252.116.137]) with mapi; Wed, 4 May
 2011 15:06:18 -0700
From: Suresh Srinivas <sureshms@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 4 May 2011 15:06:16 -0700
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
Thread-Topic: [VOTE] Release candidate 0.20.203.0-rc1
Thread-Index: AcwKpz4FhTLAcTIqSb+M8SFDEET6ZgAAD7/6
Message-ID: <C9E71BE8.19FCF%sureshms@yahoo-inc.com>
In-Reply-To: <BANLkTik5gj0H4gmvy14Ux=g_r5Tt6D1=MA@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
user-agent: Microsoft-Entourage/13.8.0.101117
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Eli,

How many of these patches that you find troublesome are in CDH already?

Regards,
Suresh


On 5/4/11 3:03 PM, "Eli Collins" <eli@cloudera.com> wrote:

> On Wed, May 4, 2011 at 10:31 AM, Owen O'Malley <omalley@apache.org> wrote=
:
>> Here's an updated release candidate for 0.20.203.0. I've incorporated th=
e
>> feedback and included all of the patches from 0.20.2, which is the last
>> stable release. I also fixed the eclipse-plugin problem.
>>=20
>> The candidate is at: http://people.apache.org/~omalley/hadoop-0.20.203.0=
-rc1/
>>=20
>> Please download it, inspect it, compile it, and test it. Clearly, I'm +1=
.
>>=20
>> -- Owen
>=20
> While rc2 is an improvement on rc1, I am -1 on this particular rc.  Ratio=
nale:
>=20
> This rc contains many patches not yet committed to trunk. This would
> cause the next major release (0.22) to be a feature regression against
> our latest stable release (203), were 0.22 released soon.
>=20
> This rc contains many patches not yet reviewed by the community via
> the normal process (jira, patch against trunk, merge to a release
> branch). I think we should respect the existing community process that
> has been used for all previous releases.
>=20
> This rc introduces a new development and braching model (new feature
> development outside trunk) and Hadoop versioning scheme without
> sufficient discussion or proposal of these changes with the community.
>=20
> We should establish new process before the release, a release is not
> the appropriate mechanism for changing our review and development
> process or versioning .
>=20
> I do support a release from branch-0.20-security that follows the
> existing, established community process.
>=20
> Thanks,
> Eli


From general-return-3341-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 22:17:28 2011
Return-Path: <general-return-3341-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 12C2D3A6E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  4 May 2011 22:17:28 +0000 (UTC)
Received: (qmail 13661 invoked by uid 500); 4 May 2011 22:17:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13615 invoked by uid 500); 4 May 2011 22:17:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13607 invoked by uid 99); 4 May 2011 22:17:26 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:17:26 +0000
Received: from localhost (HELO [192.168.168.16]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:17:26 +0000
Message-ID: <4DC1D076.4040004@apache.org>
Date: Wed, 04 May 2011 15:17:26 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.14) Gecko/20110223 Thunderbird/3.1.8
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
References: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org>
In-Reply-To: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org>
X-Enigmail-Version: 1.1.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

-1

This candidate has lots of patches that are not in trunk, potentially
adding regressions to 0.22 and 0.23.  This should be addressed before we
release from 0.20-security.  We should also not move to four-component
version numbering.  A release from the 0.20-security branch should
perhaps be called 0.20.100.

Doug

On 05/04/2011 10:31 AM, Owen O'Malley wrote:
> Here's an updated release candidate for 0.20.203.0. I've incorporated the feedback and included all of the patches from 0.20.2, which is the last stable release. I also fixed the eclipse-plugin problem. 
> 
> The candidate is at: http://people.apache.org/~omalley/hadoop-0.20.203.0-rc1/
> 
> Please download it, inspect it, compile it, and test it. Clearly, I'm +1.
> 
> -- Owen

From general-return-3342-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 22:24:41 2011
Return-Path: <general-return-3342-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AA22837AF
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  4 May 2011 22:24:41 +0000 (UTC)
Received: (qmail 29469 invoked by uid 500); 4 May 2011 22:24:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29408 invoked by uid 500); 4 May 2011 22:24:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29400 invoked by uid 99); 4 May 2011 22:24:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:24:40 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:24:33 +0000
Received: by bwz8 with SMTP id 8so2366309bwz.35
        for <general@hadoop.apache.org>; Wed, 04 May 2011 15:24:12 -0700 (PDT)
Received: by 10.204.11.22 with SMTP id r22mr703834bkr.172.1304547852148; Wed,
 04 May 2011 15:24:12 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.205.81.143 with HTTP; Wed, 4 May 2011 15:23:52 -0700 (PDT)
In-Reply-To: <4DC1D076.4040004@apache.org>
References: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org> <4DC1D076.4040004@apache.org>
From: Todd Lipcon <todd@cloudera.com>
Date: Wed, 4 May 2011 15:23:52 -0700
Message-ID: <BANLkTimvDtHbTxYVvrcsSuGWBrNkjzqH=g@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0003255585b251e12304a27ab79f

--0003255585b251e12304a27ab79f
Content-Type: text/plain; charset=ISO-8859-1

-1 for the same reasons I outlined in my email yesterday. This is not a
community artifact following the community's processes, and thus should not
be an official release until those issues are addressed.

On Wed, May 4, 2011 at 3:17 PM, Doug Cutting <cutting@apache.org> wrote:

> -1
>
> This candidate has lots of patches that are not in trunk, potentially
> adding regressions to 0.22 and 0.23.  This should be addressed before we
> release from 0.20-security.  We should also not move to four-component
> version numbering.  A release from the 0.20-security branch should
> perhaps be called 0.20.100.
>
> Doug
>
> On 05/04/2011 10:31 AM, Owen O'Malley wrote:
> > Here's an updated release candidate for 0.20.203.0. I've incorporated the
> feedback and included all of the patches from 0.20.2, which is the last
> stable release. I also fixed the eclipse-plugin problem.
> >
> > The candidate is at:
> http://people.apache.org/~omalley/hadoop-0.20.203.0-rc1/
> >
> > Please download it, inspect it, compile it, and test it. Clearly, I'm +1.
> >
> > -- Owen
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--0003255585b251e12304a27ab79f--

From general-return-3343-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 22:24:43 2011
Return-Path: <general-return-3343-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ADF8337D9
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  4 May 2011 22:24:43 +0000 (UTC)
Received: (qmail 30885 invoked by uid 500); 4 May 2011 22:24:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30815 invoked by uid 500); 4 May 2011 22:24:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30805 invoked by uid 99); 4 May 2011 22:24:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:24:41 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:24:34 +0000
Received: by ewy22 with SMTP id 22so715273ewy.35
        for <general@hadoop.apache.org>; Wed, 04 May 2011 15:24:14 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.14.122.81 with SMTP id s57mr820258eeh.195.1304547854412; Wed,
 04 May 2011 15:24:14 -0700 (PDT)
Received: by 10.14.37.13 with HTTP; Wed, 4 May 2011 15:24:14 -0700 (PDT)
In-Reply-To: <C9E71BE8.19FCF%sureshms@yahoo-inc.com>
References: <BANLkTik5gj0H4gmvy14Ux=g_r5Tt6D1=MA@mail.gmail.com>
	<C9E71BE8.19FCF%sureshms@yahoo-inc.com>
Date: Wed, 4 May 2011 15:24:14 -0700
Message-ID: <BANLkTikrS9PTeErT15fs2yv+N4_wpPN7xQ@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

With my Cloudera hat on..

When we went through the 10x and 20x patches we only pulled a subset
of them, primarily for security and the general improvements that we
thought were good.  We found both incompatible changes and some
sketchy changes that we did not pull in from a quality perspective.
There is a big difference between a patch set that's acceptable for
Yahoo!'s user base and one that's a more general artifact.

When we evaluated the YDH patch sets we were using that frame of mind.
 I'm now looking it in terms of an Apache release. And the place to
review changes for an Apache release is on jira.

CDH3 is based on the latest stable Apache release (20.2) so it doesn't
regress against it.  I'm nervous about rebasing future releases on 203
because of the compatibility and quality implications.

Thanks,
Eli


On Wed, May 4, 2011 at 3:06 PM, Suresh Srinivas <sureshms@yahoo-inc.com> wr=
ote:
> Eli,
>
> How many of these patches that you find troublesome are in CDH already?
>
> Regards,
> Suresh
>
>
> On 5/4/11 3:03 PM, "Eli Collins" <eli@cloudera.com> wrote:
>
>> On Wed, May 4, 2011 at 10:31 AM, Owen O'Malley <omalley@apache.org> wrot=
e:
>>> Here's an updated release candidate for 0.20.203.0. I've incorporated t=
he
>>> feedback and included all of the patches from 0.20.2, which is the last
>>> stable release. I also fixed the eclipse-plugin problem.
>>>
>>> The candidate is at: http://people.apache.org/~omalley/hadoop-0.20.203.=
0-rc1/
>>>
>>> Please download it, inspect it, compile it, and test it. Clearly, I'm +=
1.
>>>
>>> -- Owen
>>
>> While rc2 is an improvement on rc1, I am -1 on this particular rc. =A0Ra=
tionale:
>>
>> This rc contains many patches not yet committed to trunk. This would
>> cause the next major release (0.22) to be a feature regression against
>> our latest stable release (203), were 0.22 released soon.
>>
>> This rc contains many patches not yet reviewed by the community via
>> the normal process (jira, patch against trunk, merge to a release
>> branch). I think we should respect the existing community process that
>> has been used for all previous releases.
>>
>> This rc introduces a new development and braching model (new feature
>> development outside trunk) and Hadoop versioning scheme without
>> sufficient discussion or proposal of these changes with the community.
>>
>> We should establish new process before the release, a release is not
>> the appropriate mechanism for changing our review and development
>> process or versioning .
>>
>> I do support a release from branch-0.20-security that follows the
>> existing, established community process.
>>
>> Thanks,
>> Eli
>
>

From general-return-3344-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 22:30:12 2011
Return-Path: <general-return-3344-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 01019396D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  4 May 2011 22:30:12 +0000 (UTC)
Received: (qmail 39016 invoked by uid 500); 4 May 2011 22:30:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38962 invoked by uid 500); 4 May 2011 22:30:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 38954 invoked by uid 99); 4 May 2011 22:30:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:30:10 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jghoman@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:30:04 +0000
Received: by wwi18 with SMTP id 18so1856700wwi.29
        for <general@hadoop.apache.org>; Wed, 04 May 2011 15:29:44 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type:content-transfer-encoding;
        bh=fmxIhh3AQswrvRAhbDi7PKtyHc7Tzk9ZR8b02+ax4QQ=;
        b=Ad1ub2unbCCV8MWtFN1gW1JSN8uHMKvDPNPMFFJGZx3Er3glamnRY/spm/tNH48zdg
         uZ+Db1diSxezHymY3rTFTqD0e+Pr1GnVrzBlGT5qkDwInsunHhRrv5ykvD+omyuT/W3X
         9EqHMJlpVxhrN3k+xE19wWu1AyAL/J1VWevbg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=ZQ+Xmp8uPUw6eZRw+HlXFWkjoGk+J08NzATalf1AzVpiaHtXn8oUPSioRF4ZQeWH3f
         L95SQ2HTheBgdCXs/jnpcSs72ol8OovqvHMYnTnwSSvHw07RAH53Jf2mY3Z/op4SFKR7
         7tpt/6azrQ+tdL6dosWkPFL0fIv4JPt7q31ug=
Received: by 10.216.203.195 with SMTP id f45mr5405504weo.89.1304548184097;
 Wed, 04 May 2011 15:29:44 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.136.80 with HTTP; Wed, 4 May 2011 15:29:14 -0700 (PDT)
In-Reply-To: <BANLkTikrS9PTeErT15fs2yv+N4_wpPN7xQ@mail.gmail.com>
References: <BANLkTik5gj0H4gmvy14Ux=g_r5Tt6D1=MA@mail.gmail.com>
 <C9E71BE8.19FCF%sureshms@yahoo-inc.com> <BANLkTikrS9PTeErT15fs2yv+N4_wpPN7xQ@mail.gmail.com>
From: Jakob Homan <jghoman@gmail.com>
Date: Wed, 4 May 2011 15:29:14 -0700
Message-ID: <BANLkTimjxRdf6PyTnbZUwrS_znZo00z-5w@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

@Eli >> This rc contains many patches not yet committed to trunk.
If you've compiled this list, can you post it?

On Wed, May 4, 2011 at 3:24 PM, Eli Collins <eli@cloudera.com> wrote:
> With my Cloudera hat on..
>
> When we went through the 10x and 20x patches we only pulled a subset
> of them, primarily for security and the general improvements that we
> thought were good. =A0We found both incompatible changes and some
> sketchy changes that we did not pull in from a quality perspective.
> There is a big difference between a patch set that's acceptable for
> Yahoo!'s user base and one that's a more general artifact.
>
> When we evaluated the YDH patch sets we were using that frame of mind.
> =A0I'm now looking it in terms of an Apache release. And the place to
> review changes for an Apache release is on jira.
>
> CDH3 is based on the latest stable Apache release (20.2) so it doesn't
> regress against it. =A0I'm nervous about rebasing future releases on 203
> because of the compatibility and quality implications.
>
> Thanks,
> Eli
>
>
> On Wed, May 4, 2011 at 3:06 PM, Suresh Srinivas <sureshms@yahoo-inc.com> =
wrote:
>> Eli,
>>
>> How many of these patches that you find troublesome are in CDH already?
>>
>> Regards,
>> Suresh
>>
>>
>> On 5/4/11 3:03 PM, "Eli Collins" <eli@cloudera.com> wrote:
>>
>>> On Wed, May 4, 2011 at 10:31 AM, Owen O'Malley <omalley@apache.org> wro=
te:
>>>> Here's an updated release candidate for 0.20.203.0. I've incorporated =
the
>>>> feedback and included all of the patches from 0.20.2, which is the las=
t
>>>> stable release. I also fixed the eclipse-plugin problem.
>>>>
>>>> The candidate is at: http://people.apache.org/~omalley/hadoop-0.20.203=
.0-rc1/
>>>>
>>>> Please download it, inspect it, compile it, and test it. Clearly, I'm =
+1.
>>>>
>>>> -- Owen
>>>
>>> While rc2 is an improvement on rc1, I am -1 on this particular rc. =A0R=
ationale:
>>>
>>> This rc contains many patches not yet committed to trunk. This would
>>> cause the next major release (0.22) to be a feature regression against
>>> our latest stable release (203), were 0.22 released soon.
>>>
>>> This rc contains many patches not yet reviewed by the community via
>>> the normal process (jira, patch against trunk, merge to a release
>>> branch). I think we should respect the existing community process that
>>> has been used for all previous releases.
>>>
>>> This rc introduces a new development and braching model (new feature
>>> development outside trunk) and Hadoop versioning scheme without
>>> sufficient discussion or proposal of these changes with the community.
>>>
>>> We should establish new process before the release, a release is not
>>> the appropriate mechanism for changing our review and development
>>> process or versioning .
>>>
>>> I do support a release from branch-0.20-security that follows the
>>> existing, established community process.
>>>
>>> Thanks,
>>> Eli
>>
>>
>

From general-return-3345-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 22:31:36 2011
Return-Path: <general-return-3345-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1692B3FFF
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  4 May 2011 22:31:36 +0000 (UTC)
Received: (qmail 41092 invoked by uid 500); 4 May 2011 22:31:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 41043 invoked by uid 500); 4 May 2011 22:31:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 41033 invoked by uid 99); 4 May 2011 22:31:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:31:34 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:31:29 +0000
Received: by bwz8 with SMTP id 8so2373036bwz.35
        for <general@hadoop.apache.org>; Wed, 04 May 2011 15:31:07 -0700 (PDT)
Received: by 10.204.82.143 with SMTP id b15mr1661095bkl.118.1304548267304;
 Wed, 04 May 2011 15:31:07 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.205.81.143 with HTTP; Wed, 4 May 2011 15:30:47 -0700 (PDT)
In-Reply-To: <BANLkTikrS9PTeErT15fs2yv+N4_wpPN7xQ@mail.gmail.com>
References: <BANLkTik5gj0H4gmvy14Ux=g_r5Tt6D1=MA@mail.gmail.com>
 <C9E71BE8.19FCF%sureshms@yahoo-inc.com> <BANLkTikrS9PTeErT15fs2yv+N4_wpPN7xQ@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Wed, 4 May 2011 15:30:47 -0700
Message-ID: <BANLkTikU1NmH0NM+Gu95PU9cWsHNQ_VbBg@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d7ef9010a71704a27ad0b3

--0016e6d7ef9010a71704a27ad0b3
Content-Type: text/plain; charset=ISO-8859-1

With Cloudera hat on, I agree with Eli's assessment.

With Apache hat on, I don't see how this is at all relevant to the task at
hand. I would make the same arguments against taking CDH3 and releasing it
as an ASF artifact -- we'd also have a certain amount of work to do to make
sure that all of the patches are in trunk, first. Additionally, I'd want to
outline what the inclusion criteria would be for that branch.

-Todd

On Wed, May 4, 2011 at 3:24 PM, Eli Collins <eli@cloudera.com> wrote:

> With my Cloudera hat on..
>
> When we went through the 10x and 20x patches we only pulled a subset
> of them, primarily for security and the general improvements that we
> thought were good.  We found both incompatible changes and some
> sketchy changes that we did not pull in from a quality perspective.
> There is a big difference between a patch set that's acceptable for
> Yahoo!'s user base and one that's a more general artifact.
>
> When we evaluated the YDH patch sets we were using that frame of mind.
>  I'm now looking it in terms of an Apache release. And the place to
> review changes for an Apache release is on jira.
>
> CDH3 is based on the latest stable Apache release (20.2) so it doesn't
> regress against it.  I'm nervous about rebasing future releases on 203
> because of the compatibility and quality implications.
>
> Thanks,
> Eli
>
>
> On Wed, May 4, 2011 at 3:06 PM, Suresh Srinivas <sureshms@yahoo-inc.com>
> wrote:
> > Eli,
> >
> > How many of these patches that you find troublesome are in CDH already?
> >
> > Regards,
> > Suresh
> >
> >
> > On 5/4/11 3:03 PM, "Eli Collins" <eli@cloudera.com> wrote:
> >
> >> On Wed, May 4, 2011 at 10:31 AM, Owen O'Malley <omalley@apache.org>
> wrote:
> >>> Here's an updated release candidate for 0.20.203.0. I've incorporated
> the
> >>> feedback and included all of the patches from 0.20.2, which is the last
> >>> stable release. I also fixed the eclipse-plugin problem.
> >>>
> >>> The candidate is at:
> http://people.apache.org/~omalley/hadoop-0.20.203.0-rc1/
> >>>
> >>> Please download it, inspect it, compile it, and test it. Clearly, I'm
> +1.
> >>>
> >>> -- Owen
> >>
> >> While rc2 is an improvement on rc1, I am -1 on this particular rc.
>  Rationale:
> >>
> >> This rc contains many patches not yet committed to trunk. This would
> >> cause the next major release (0.22) to be a feature regression against
> >> our latest stable release (203), were 0.22 released soon.
> >>
> >> This rc contains many patches not yet reviewed by the community via
> >> the normal process (jira, patch against trunk, merge to a release
> >> branch). I think we should respect the existing community process that
> >> has been used for all previous releases.
> >>
> >> This rc introduces a new development and braching model (new feature
> >> development outside trunk) and Hadoop versioning scheme without
> >> sufficient discussion or proposal of these changes with the community.
> >>
> >> We should establish new process before the release, a release is not
> >> the appropriate mechanism for changing our review and development
> >> process or versioning .
> >>
> >> I do support a release from branch-0.20-security that follows the
> >> existing, established community process.
> >>
> >> Thanks,
> >> Eli
> >
> >
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--0016e6d7ef9010a71704a27ad0b3--

From general-return-3346-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 22:36:46 2011
Return-Path: <general-return-3346-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 623C53E22
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  4 May 2011 22:36:46 +0000 (UTC)
Received: (qmail 47063 invoked by uid 500); 4 May 2011 22:36:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47013 invoked by uid 500); 4 May 2011 22:36:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47005 invoked by uid 99); 4 May 2011 22:36:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:36:44 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 209.85.215.176 as permitted sender)
Received: from [209.85.215.176] (HELO mail-ey0-f176.google.com) (209.85.215.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:36:37 +0000
Received: by eya28 with SMTP id 28so717192eya.35
        for <general@hadoop.apache.org>; Wed, 04 May 2011 15:36:17 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.14.50.196 with SMTP id z44mr783860eeb.227.1304548576913; Wed,
 04 May 2011 15:36:16 -0700 (PDT)
Received: by 10.14.37.13 with HTTP; Wed, 4 May 2011 15:36:16 -0700 (PDT)
In-Reply-To: <BANLkTimjxRdf6PyTnbZUwrS_znZo00z-5w@mail.gmail.com>
References: <BANLkTik5gj0H4gmvy14Ux=g_r5Tt6D1=MA@mail.gmail.com>
	<C9E71BE8.19FCF%sureshms@yahoo-inc.com>
	<BANLkTikrS9PTeErT15fs2yv+N4_wpPN7xQ@mail.gmail.com>
	<BANLkTimjxRdf6PyTnbZUwrS_znZo00z-5w@mail.gmail.com>
Date: Wed, 4 May 2011 15:36:16 -0700
Message-ID: <BANLkTi=pNF_aQrWPurfXRbjNdshfseNobg@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Wed, May 4, 2011 at 3:29 PM, Jakob Homan <jghoman@gmail.com> wrote:
> @Eli >> This rc contains many patches not yet committed to trunk.
> If you've compiled this list, can you post it?
>

Here's the list Todd posted yesterday:

http://mail-archives.apache.org/mod_mbox/hadoop-general/201105.mbox/%3CBANLkTimKKbkuPCz61TU=8-nO8Z6PYhf7Sg@mail.gmail.com%3E

Thanks,
Eli

From general-return-3347-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 22:41:48 2011
Return-Path: <general-return-3347-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 39CC23EC9
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  4 May 2011 22:41:48 +0000 (UTC)
Received: (qmail 50898 invoked by uid 500); 4 May 2011 22:41:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50841 invoked by uid 500); 4 May 2011 22:41:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50833 invoked by uid 99); 4 May 2011 22:41:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:41:46 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:41:40 +0000
Received: from sp1-ex07cas03.ds.corp.yahoo.com (sp1-ex07cas03.ds.corp.yahoo.com [216.252.116.151])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p44Mf2dK037183
	for <general@hadoop.apache.org>; Wed, 4 May 2011 15:41:02 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1304548863;
	bh=7uqtMbzD6mRwXizPzROVFvjVkTImwJBpsJnWY+nC3nc=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:MIME-Version;
	b=tcyGmrmtqigWic9iFXEsN72PUZ95G1pIQHHUlg3zr+uAP8ncZS/Sw15l6QvCd8uLq
	 yxbpyrZehK6cURbyw8FEyIqYtXKFGWpyDlfmRpXfXy26TaEMHZlddgPD2w4HZIjTNy
	 SUYpgkdL/gFfs9TBpVWoQWUHTUm98NVbfdQPe3a4=
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas03.ds.corp.yahoo.com ([216.252.116.151]) with mapi; Wed, 4 May
 2011 15:41:02 -0700
From: Suresh Srinivas <sureshms@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 4 May 2011 15:41:00 -0700
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
Thread-Topic: [VOTE] Release candidate 0.20.203.0-rc1
Thread-Index: AcwKqxAcJO5eMBmKSQao+BVskYWtvQAAUcNy
Message-ID: <C9E7240C.19FDC%sureshms@yahoo-inc.com>
In-Reply-To: <BANLkTikU1NmH0NM+Gu95PU9cWsHNQ_VbBg@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-Entourage/13.8.0.101117
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Here is a snippet from your blog -
http://www.cloudera.com/blog/2010/10/cdh3-beta-3-now-available/

------

Security Enhancements
As one of the primary contributors and largest production users of Hadoop,
Yahoo! publishes the source tree for the version of Hadoop that they run on
their production clusters. We are pleased to announce that we have merged
Yahoo=B9s source tree into CDH3b3. This merge brings many improvements
developed at Yahoo! into CDH, including improvements for MapReduce
scalability on 1000+-node clusters and several new tools for benchmarking
and testing Hadoop.
------

It would be great, if you can list how many of 192 changes were reviewed an=
d
became part of CDH.

Your -1 vote essentially blocks the changes that are already available in
CDH to be available from Apache open source!


On 5/4/11 3:30 PM, "Todd Lipcon" <todd@cloudera.com> wrote:

> With Cloudera hat on, I agree with Eli's assessment.
>=20
> With Apache hat on, I don't see how this is at all relevant to the task a=
t
> hand. I would make the same arguments against taking CDH3 and releasing i=
t
> as an ASF artifact -- we'd also have a certain amount of work to do to ma=
ke
> sure that all of the patches are in trunk, first. Additionally, I'd want =
to
> outline what the inclusion criteria would be for that branch.
>=20
> -Todd
>=20
> On Wed, May 4, 2011 at 3:24 PM, Eli Collins <eli@cloudera.com> wrote:
>=20
>> With my Cloudera hat on..
>>=20
>> When we went through the 10x and 20x patches we only pulled a subset
>> of them, primarily for security and the general improvements that we
>> thought were good.  We found both incompatible changes and some
>> sketchy changes that we did not pull in from a quality perspective.
>> There is a big difference between a patch set that's acceptable for
>> Yahoo!'s user base and one that's a more general artifact.
>>=20
>> When we evaluated the YDH patch sets we were using that frame of mind.
>>  I'm now looking it in terms of an Apache release. And the place to
>> review changes for an Apache release is on jira.
>>=20
>> CDH3 is based on the latest stable Apache release (20.2) so it doesn't
>> regress against it.  I'm nervous about rebasing future releases on 203
>> because of the compatibility and quality implications.
>>=20
>> Thanks,
>> Eli
>>=20
>>=20
>> On Wed, May 4, 2011 at 3:06 PM, Suresh Srinivas <sureshms@yahoo-inc.com>
>> wrote:
>>> Eli,
>>>=20
>>> How many of these patches that you find troublesome are in CDH already?
>>>=20
>>> Regards,
>>> Suresh
>>>=20
>>>=20
>>> On 5/4/11 3:03 PM, "Eli Collins" <eli@cloudera.com> wrote:
>>>=20
>>>> On Wed, May 4, 2011 at 10:31 AM, Owen O'Malley <omalley@apache.org>
>> wrote:
>>>>> Here's an updated release candidate for 0.20.203.0. I've incorporated
>> the
>>>>> feedback and included all of the patches from 0.20.2, which is the la=
st
>>>>> stable release. I also fixed the eclipse-plugin problem.
>>>>>=20
>>>>> The candidate is at:
>> http://people.apache.org/~omalley/hadoop-0.20.203.0-rc1/
>>>>>=20
>>>>> Please download it, inspect it, compile it, and test it. Clearly, I'm
>> +1.
>>>>>=20
>>>>> -- Owen
>>>>=20
>>>> While rc2 is an improvement on rc1, I am -1 on this particular rc.
>>  Rationale:
>>>>=20
>>>> This rc contains many patches not yet committed to trunk. This would
>>>> cause the next major release (0.22) to be a feature regression against
>>>> our latest stable release (203), were 0.22 released soon.
>>>>=20
>>>> This rc contains many patches not yet reviewed by the community via
>>>> the normal process (jira, patch against trunk, merge to a release
>>>> branch). I think we should respect the existing community process that
>>>> has been used for all previous releases.
>>>>=20
>>>> This rc introduces a new development and braching model (new feature
>>>> development outside trunk) and Hadoop versioning scheme without
>>>> sufficient discussion or proposal of these changes with the community.
>>>>=20
>>>> We should establish new process before the release, a release is not
>>>> the appropriate mechanism for changing our review and development
>>>> process or versioning .
>>>>=20
>>>> I do support a release from branch-0.20-security that follows the
>>>> existing, established community process.
>>>>=20
>>>> Thanks,
>>>> Eli
>>>=20
>>>=20
>>=20
>=20
>=20


From general-return-3348-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 22:51:56 2011
Return-Path: <general-return-3348-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3987433F6
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  4 May 2011 22:51:56 +0000 (UTC)
Received: (qmail 60973 invoked by uid 500); 4 May 2011 22:51:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60924 invoked by uid 500); 4 May 2011 22:51:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60916 invoked by uid 99); 4 May 2011 22:51:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:51:54 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:51:47 +0000
Received: by ewy22 with SMTP id 22so723392ewy.35
        for <general@hadoop.apache.org>; Wed, 04 May 2011 15:51:27 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.14.46.214 with SMTP id r62mr757945eeb.99.1304549486642; Wed,
 04 May 2011 15:51:26 -0700 (PDT)
Received: by 10.14.37.13 with HTTP; Wed, 4 May 2011 15:51:26 -0700 (PDT)
In-Reply-To: <C9E7240C.19FDC%sureshms@yahoo-inc.com>
References: <BANLkTikU1NmH0NM+Gu95PU9cWsHNQ_VbBg@mail.gmail.com>
	<C9E7240C.19FDC%sureshms@yahoo-inc.com>
Date: Wed, 4 May 2011 15:51:26 -0700
Message-ID: <BANLkTi=B+cc_677HC1wRnpcY73zuF8qekg@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

> Your -1 vote essentially blocks the changes that are already available in
> CDH to be available from Apache open source!

As Eric mentioned, this thread is about an Apache release, not CDH.

My -1 vote does not block these changes from being released via
Apache. You can not veto a release. Releases are lazy majority, the
release is only blocked if there are more -1 votes than +1 votes.

If these changes are contributed on jira, discussed and reviewed, and
committed to trunk I'm happy to support the release.  There's a big
difference between asking that a release respect the Apache community
process and blocking it. If you want to get the release out how about
contributing the work via the normal means so the community can review
it like we review all other code changes.

Thanks,
Eli

From general-return-3349-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 23:09:45 2011
Return-Path: <general-return-3349-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 176C83B8F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  4 May 2011 23:09:45 +0000 (UTC)
Received: (qmail 80499 invoked by uid 500); 4 May 2011 23:09:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80447 invoked by uid 500); 4 May 2011 23:09:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80439 invoked by uid 99); 4 May 2011 23:09:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 23:09:43 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [98.139.52.222] (HELO nm25.bullet.mail.ac4.yahoo.com) (98.139.52.222)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 04 May 2011 23:09:33 +0000
Received: from [98.139.52.195] by nm25.bullet.mail.ac4.yahoo.com with NNFMP; 04 May 2011 23:09:12 -0000
Received: from [98.139.52.171] by tm8.bullet.mail.ac4.yahoo.com with NNFMP; 04 May 2011 23:09:12 -0000
Received: from [127.0.0.1] by omp1054.mail.ac4.yahoo.com with NNFMP; 04 May 2011 23:09:12 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 549727.21227.bm@omp1054.mail.ac4.yahoo.com
Received: (qmail 54768 invoked by uid 60001); 4 May 2011 23:09:12 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1304550552; bh=YuKBVr2+P8/o31wTbbEnhFzhXKT53RV+PzPidPh6dBM=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=ieBnz6hhrX0jCKSbkYaAS/VxTJUNYvgEgM4hTX3ya1jLalNUBCCQjDkj0Jm55VDZ6dwGyJANGad7CxuyWkMZ8xsxXm5TLhjT8Cxwu3N7GnYS0WLW4CgSW/IGx75ZSgPHFuk2h//XGLiYAz/iZnc66yONDO11ME7vK4xepdLUht0=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=sNvxdnXTZsSnKDcDn6Pd8IgGuf9EeeCYGWpxEoEUGesquYJMzsk7S98UxGKy7hTW6qopCMwMUP7/+zKDujY/NTfUFMv4mom+2A3DYdxICU2OAaTRqyZ/Hy9/5dChVTaeCn7LX4PsNgvILBl1IqhTA7e45gdKiqsec14JNQ5jBYg=;
Message-ID: <321761.51699.qm@web65903.mail.ac4.yahoo.com>
X-YMail-OSG: cely7lcVM1ldAKpSSCG9EvLVP3QghvvKw8h3T8anOK4H2g2
 lAbDwIIe3sZMRo1OEivCAION1Cc_yzOr656m1Km74WdTBanLnnhCcCSBXrCo
 WeLEYzKkYmah7CLTdTto3cF9b3gYyIUrzobFTo9zD1qATyLy2c4zhzaHsJld
 m7bsqFlk3aPtw_MOua.5ZpKurMG1ak_psJblHxUiIrjYGp1kC_UOVFwWlcf5
 kV4ALUELsavX5HOCJsiF1dTc5RO1JNy2UEdLbaH.YiBNLVTKjC2w6npEfBf8
 Q5S76oIreqAzzjY3.eJwkS5i32x8cLRK07PiBMtkkZG33nsEHODTgsgcGXA1
 zUEFFiK63Srqqkb1_7L6CLv1p2bzN3m93y4LHWamb4WGUC_TORCiCEyTF5Lr
 baDlL.1eLujoKKlE1LpvihuEU80QfLwteMg--
Received: from [209.131.62.113] by web65903.mail.ac4.yahoo.com via HTTP; Wed, 04 May 2011 16:09:12 PDT
X-Mailer: YahooMailRC/559 YahooMailWebService/0.8.110.299900
References: <BANLkTik5gj0H4gmvy14Ux=g_r5Tt6D1=MA@mail.gmail.com> <C9E71BE8.19FCF%sureshms@yahoo-inc.com> <BANLkTikrS9PTeErT15fs2yv+N4_wpPN7xQ@mail.gmail.com> <BANLkTimjxRdf6PyTnbZUwrS_znZo00z-5w@mail.gmail.com> <BANLkTi=pNF_aQrWPurfXRbjNdshfseNobg@mail.gmail.com>
Date: Wed, 4 May 2011 16:09:12 -0700 (PDT)
From: "Tsz Wo \(Nicholas\), Sze" <s29752-hadoopgeneral@yahoo.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
To: general@hadoop.apache.org
In-Reply-To: <BANLkTi=pNF_aQrWPurfXRbjNdshfseNobg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-451605876-1304550552=:51699"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-451605876-1304550552=:51699
Content-Type: text/plain; charset=us-ascii

The list seems highly inaccurate.  Checked the first few N/A items.  All are 
false positives.

< HADOOP-6304 N/A -- fixed in trunk via HADOOP-7110 (Todd, it was fixed by you. 
Forgot?)
< HADOOP-6598 N/A -- moved to HADOOP-6763 and committed to trunk
< HADOOP-6653 N/A -- not applicable in trunk
< HADOOP-6716 N/A -- as part of HADOOP-6815 which was committed to trunk
< HADOOP-6718 N/A -- Incorporated in HADOOP-6706 for 0.22.
< HADOOP-6776 N/A -- Tom White said "This is fixed in trunk, so can be closed."

Regards,
Nicholas





________________________________
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Sent: Wed, May 4, 2011 3:36:16 PM
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1

On Wed, May 4, 2011 at 3:29 PM, Jakob Homan <jghoman@gmail.com> wrote:
> @Eli >> This rc contains many patches not yet committed to trunk.
> If you've compiled this list, can you post it?
>

Here's the list Todd posted yesterday:

http://mail-archives.apache.org/mod_mbox/hadoop-general/201105.mbox/%3CBANLkTimKKbkuPCz61TU=8-nO8Z6PYhf7Sg@mail.gmail.com%3E


Thanks,
Eli

--0-451605876-1304550552=:51699--

From general-return-3350-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 23:12:00 2011
Return-Path: <general-return-3350-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 400F33C3D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  4 May 2011 23:12:00 +0000 (UTC)
Received: (qmail 84753 invoked by uid 500); 4 May 2011 23:11:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84693 invoked by uid 500); 4 May 2011 23:11:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84685 invoked by uid 99); 4 May 2011 23:11:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 23:11:58 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of eli@cloudera.com designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 23:11:45 +0000
Received: by ewy22 with SMTP id 22so728900ewy.35
        for <general@hadoop.apache.org>; Wed, 04 May 2011 16:11:24 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.14.17.100 with SMTP id i76mr812605eei.67.1304550683871; Wed,
 04 May 2011 16:11:23 -0700 (PDT)
Received: by 10.14.37.13 with HTTP; Wed, 4 May 2011 16:11:23 -0700 (PDT)
In-Reply-To: <321761.51699.qm@web65903.mail.ac4.yahoo.com>
References: <BANLkTik5gj0H4gmvy14Ux=g_r5Tt6D1=MA@mail.gmail.com>
	<C9E71BE8.19FCF%sureshms@yahoo-inc.com>
	<BANLkTikrS9PTeErT15fs2yv+N4_wpPN7xQ@mail.gmail.com>
	<BANLkTimjxRdf6PyTnbZUwrS_znZo00z-5w@mail.gmail.com>
	<BANLkTi=pNF_aQrWPurfXRbjNdshfseNobg@mail.gmail.com>
	<321761.51699.qm@web65903.mail.ac4.yahoo.com>
Date: Wed, 4 May 2011 16:11:23 -0700
Message-ID: <BANLkTimniyMpWE0MtRSLpQV=E-w4PLkvXA@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Wed, May 4, 2011 at 4:09 PM, Tsz Wo (Nicholas), Sze
<s29752-hadoopgeneral@yahoo.com> wrote:
> The list seems highly inaccurate. =A0Checked the first few N/A items. =A0=
All are
> false positives.

Yes, that's why those are marked N/A ie "Not applicable". Check out
the non N/A ones.

Thanks,
Eli

From general-return-3351-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 23:13:20 2011
Return-Path: <general-return-3351-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0B2543C8D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  4 May 2011 23:13:20 +0000 (UTC)
Received: (qmail 87577 invoked by uid 500); 4 May 2011 23:13:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87531 invoked by uid 500); 4 May 2011 23:13:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87523 invoked by uid 99); 4 May 2011 23:13:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 23:13:18 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 23:13:12 +0000
Received: from l2tp-8-201.corp.yahoo.com (l2tp-8-201.corp.yahoo.com [10.73.8.201])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p44NBrOS046750
	for <general@hadoop.apache.org>; Wed, 4 May 2011 16:11:54 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1304550714;
	bh=4Y4EVUclYhx6KYuAqN1eIn/VPn2SlkPwFSOoeoGHHt4=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=va6J65AHObsB7ab7EMDr/dCFC7YNtYBqrmWa+k3QkBXriLIZJU1FSTrCB0zd8AAzk
	 /2ugxGnFQ7F7v+Gt61N7PjPNBeYPQtcRtxU7EGnFq664gQNA+4Yc4yICbotVjiHrA+
	 u/ioshWoNA9g5P4VzAadyEsa4Hb1DQ5ukFh7OOso=
Message-Id: <CB238EE8-1E06-4C6C-A7F5-CB00E2E073CB@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <321761.51699.qm@web65903.mail.ac4.yahoo.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
Date: Wed, 4 May 2011 16:11:53 -0700
References: <BANLkTik5gj0H4gmvy14Ux=g_r5Tt6D1=MA@mail.gmail.com> <C9E71BE8.19FCF%sureshms@yahoo-inc.com> <BANLkTikrS9PTeErT15fs2yv+N4_wpPN7xQ@mail.gmail.com> <BANLkTimjxRdf6PyTnbZUwrS_znZo00z-5w@mail.gmail.com> <BANLkTi=pNF_aQrWPurfXRbjNdshfseNobg@mail.gmail.com> <321761.51699.qm@web65903.mail.ac4.yahoo.com>
X-Mailer: Apple Mail (2.936)

On May 4, 2011, at 4:09 PM, Tsz Wo (Nicholas), Sze wrote:

> The list seems highly inaccurate.  Checked the first few N/A items.   
> All are
> false positives.
>

Also,  can you please provide a list on features which are not related  
to gridmix benchmarks or herriot tests?

Please remember, and I have said this on list and off-list, that many  
of the forward ports obviated the need for multiple patches which show  
up in the commit logs.

thanks,
Arun

> < HADOOP-6304 N/A -- fixed in trunk via HADOOP-7110 (Todd, it was  
> fixed by you.
> Forgot?)
> < HADOOP-6598 N/A -- moved to HADOOP-6763 and committed to trunk
> < HADOOP-6653 N/A -- not applicable in trunk
> < HADOOP-6716 N/A -- as part of HADOOP-6815 which was committed to  
> trunk
> < HADOOP-6718 N/A -- Incorporated in HADOOP-6706 for 0.22.
> < HADOOP-6776 N/A -- Tom White said "This is fixed in trunk, so can  
> be closed."
>
> Regards,
> Nicholas
>
>
>
>
>
> ________________________________
> From: Eli Collins <eli@cloudera.com>
> To: general@hadoop.apache.org
> Sent: Wed, May 4, 2011 3:36:16 PM
> Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
>
> On Wed, May 4, 2011 at 3:29 PM, Jakob Homan <jghoman@gmail.com> wrote:
>> @Eli >> This rc contains many patches not yet committed to trunk.
>> If you've compiled this list, can you post it?
>>
>
> Here's the list Todd posted yesterday:
>
> http://mail-archives.apache.org/mod_mbox/hadoop-general/201105.mbox/%3CBANLkTimKKbkuPCz61TU=8-nO8Z6PYhf7Sg@mail.gmail.com%3E
>
>
> Thanks,
> Eli


From general-return-3352-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 23:20:11 2011
Return-Path: <general-return-3352-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A82B6318C
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  4 May 2011 23:20:11 +0000 (UTC)
Received: (qmail 94612 invoked by uid 500); 4 May 2011 23:20:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94549 invoked by uid 500); 4 May 2011 23:20:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94541 invoked by uid 99); 4 May 2011 23:20:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 23:20:10 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 23:20:02 +0000
Received: from battlerock-lm.corp.yahoo.com (battlerock-lm.corp.yahoo.com [10.72.123.81])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p44NJBcH003126
	for <general@hadoop.apache.org>; Wed, 4 May 2011 16:19:11 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
From: "Owen O'Malley" <omalley@apache.org>
In-Reply-To: <450B3108-28FA-40B2-A7F3-28747B8D2863@linkedin.com>
Date: Wed, 4 May 2011 16:19:11 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <B8B4D228-084E-4879-89FB-650588DF24AD@apache.org>
References: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org> <450B3108-28FA-40B2-A7F3-28747B8D2863@linkedin.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org


On May 4, 2011, at 1:17 PM, Allen Wittenauer wrote:

> 	Am I misreading this, or are the MR protocols out of sync =
between 0.20.203 and 0.21?  It would also appear that this is marked =
stable in 0.21. What is the user impact?

The names of the protocols were changed, but the names of the protocols =
aren't user-facing. The protocols themselves also changed, as with all =
Hadoop major versions. (We need to switch to protobuf or something for =
RPC to provide wire compatibility.)=20

-- Owen=

From general-return-3353-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 23:29:28 2011
Return-Path: <general-return-3353-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6B03B3247
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  4 May 2011 23:29:28 +0000 (UTC)
Received: (qmail 6152 invoked by uid 500); 4 May 2011 23:29:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6103 invoked by uid 500); 4 May 2011 23:29:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6094 invoked by uid 99); 4 May 2011 23:29:27 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 23:29:27 +0000
Received: from localhost (HELO mail-fx0-f48.google.com) (127.0.0.1)
  (smtp-auth username mahadev, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 23:29:26 +0000
Received: by fxm7 with SMTP id 7so2146235fxm.35
        for <general@hadoop.apache.org>; Wed, 04 May 2011 16:29:25 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.223.106.70 with SMTP id w6mr1954171fao.68.1304551765108; Wed,
 04 May 2011 16:29:25 -0700 (PDT)
Received: by 10.223.144.197 with HTTP; Wed, 4 May 2011 16:29:25 -0700 (PDT)
In-Reply-To: <590F865D-2DF9-4A72-A7BC-2EE5EE9CB24A@yahoo-inc.com>
References: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org>
	<590F865D-2DF9-4A72-A7BC-2EE5EE9CB24A@yahoo-inc.com>
Date: Wed, 4 May 2011 16:29:25 -0700
Message-ID: <BANLkTimPQ8YPD6vJ4yog87HPVtgNvSkMJg@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
From: Mahadev Konar <mahadev@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

+1 for the release.

I downloaded the release, verified checksums, built and deployed. Ran
randomwriter jobs on it.

Everything passes.

-- 
thanks
mahadev
@mahadevkonar

On Wed, May 4, 2011 at 3:05 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:
> On May 4, 2011, at 10:31 AM, Owen O'Malley wrote:
>
>> Here's an updated release candidate for 0.20.203.0. I've incorporated the
>> feedback and included all of the patches from 0.20.2, which is the last
>> stable release. I also fixed the eclipse-plugin problem.
>>
>> The candidate is at:
>> http://people.apache.org/~omalley/hadoop-0.20.203.0-rc1/
>>
>> Please download it, inspect it, compile it, and test it. Clearly, I'm +1.
>>
>
> +1
>
> Downloaded release, checked checksums, built, deployed single-node cluster.
>
> Arun
>
>

From general-return-3354-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 23:33:28 2011
Return-Path: <general-return-3354-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 034B9334E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  4 May 2011 23:33:28 +0000 (UTC)
Received: (qmail 9200 invoked by uid 500); 4 May 2011 23:33:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 9134 invoked by uid 500); 4 May 2011 23:33:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 9126 invoked by uid 99); 4 May 2011 23:33:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 23:33:26 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 23:33:20 +0000
Received: by pwi16 with SMTP id 16so1083113pwi.35
        for <general@hadoop.apache.org>; Wed, 04 May 2011 16:32:59 -0700 (PDT)
Received: by 10.68.15.168 with SMTP id y8mr2172461pbc.337.1304551979070; Wed,
 04 May 2011 16:32:59 -0700 (PDT)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.68.41.230 with HTTP; Wed, 4 May 2011 16:32:39 -0700 (PDT)
In-Reply-To: <C9E71BE8.19FCF%sureshms@yahoo-inc.com>
References: <BANLkTik5gj0H4gmvy14Ux=g_r5Tt6D1=MA@mail.gmail.com> <C9E71BE8.19FCF%sureshms@yahoo-inc.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Wed, 4 May 2011 16:32:39 -0700
X-Google-Sender-Auth: BA_jgKIVeM3bnZEP3LmsPOcXo2o
Message-ID: <BANLkTi=_D1ihMU6yKKk06eycSaCunJWxUw@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Wed, May 4, 2011 at 15:06, Suresh Srinivas <sureshms@yahoo-inc.com> wrot=
e:
> Eli,
>
> How many of these patches that you find troublesome are in CDH already?

How is that relevant to the release vote and discrepancies listed in
Eli's email?

> Regards,
> Suresh
>
>
> On 5/4/11 3:03 PM, "Eli Collins" <eli@cloudera.com> wrote:
>
>> On Wed, May 4, 2011 at 10:31 AM, Owen O'Malley <omalley@apache.org> wrot=
e:
>>> Here's an updated release candidate for 0.20.203.0. I've incorporated t=
he
>>> feedback and included all of the patches from 0.20.2, which is the last
>>> stable release. I also fixed the eclipse-plugin problem.
>>>
>>> The candidate is at: http://people.apache.org/~omalley/hadoop-0.20.203.=
0-rc1/
>>>
>>> Please download it, inspect it, compile it, and test it. Clearly, I'm +=
1.
>>>
>>> -- Owen
>>
>> While rc2 is an improvement on rc1, I am -1 on this particular rc. =A0Ra=
tionale:
>>
>> This rc contains many patches not yet committed to trunk. This would
>> cause the next major release (0.22) to be a feature regression against
>> our latest stable release (203), were 0.22 released soon.
>>
>> This rc contains many patches not yet reviewed by the community via
>> the normal process (jira, patch against trunk, merge to a release
>> branch). I think we should respect the existing community process that
>> has been used for all previous releases.
>>
>> This rc introduces a new development and braching model (new feature
>> development outside trunk) and Hadoop versioning scheme without
>> sufficient discussion or proposal of these changes with the community.
>>
>> We should establish new process before the release, a release is not
>> the appropriate mechanism for changing our review and development
>> process or versioning .
>>
>> I do support a release from branch-0.20-security that follows the
>> existing, established community process.
>>
>> Thanks,
>> Eli
>
>

From general-return-3355-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 23:45:29 2011
Return-Path: <general-return-3355-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5F6FE3D91
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  4 May 2011 23:45:29 +0000 (UTC)
Received: (qmail 21229 invoked by uid 500); 4 May 2011 23:45:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21161 invoked by uid 500); 4 May 2011 23:45:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21153 invoked by uid 99); 4 May 2011 23:45:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 23:45:28 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 23:45:21 +0000
Received: by bwz8 with SMTP id 8so2437914bwz.35
        for <general@hadoop.apache.org>; Wed, 04 May 2011 16:45:01 -0700 (PDT)
Received: by 10.204.82.143 with SMTP id b15mr1711672bkl.118.1304552701145;
 Wed, 04 May 2011 16:45:01 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.205.81.143 with HTTP; Wed, 4 May 2011 16:44:41 -0700 (PDT)
In-Reply-To: <CB238EE8-1E06-4C6C-A7F5-CB00E2E073CB@yahoo-inc.com>
References: <BANLkTik5gj0H4gmvy14Ux=g_r5Tt6D1=MA@mail.gmail.com>
 <C9E71BE8.19FCF%sureshms@yahoo-inc.com> <BANLkTikrS9PTeErT15fs2yv+N4_wpPN7xQ@mail.gmail.com>
 <BANLkTimjxRdf6PyTnbZUwrS_znZo00z-5w@mail.gmail.com> <BANLkTi=pNF_aQrWPurfXRbjNdshfseNobg@mail.gmail.com>
 <321761.51699.qm@web65903.mail.ac4.yahoo.com> <CB238EE8-1E06-4C6C-A7F5-CB00E2E073CB@yahoo-inc.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Wed, 4 May 2011 16:44:41 -0700
Message-ID: <BANLkTi=aW6Toxtg7g=nFSV7h4gK68522wg@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d7ef9057b2ee04a27bd876
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d7ef9057b2ee04a27bd876
Content-Type: text/plain; charset=ISO-8859-1

On Wed, May 4, 2011 at 4:11 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:

> On May 4, 2011, at 4:09 PM, Tsz Wo (Nicholas), Sze wrote:
>
>  The list seems highly inaccurate.  Checked the first few N/A items.  All
>> are
>> false positives.
>>
>>
> Also,  can you please provide a list on features which are not related to
> gridmix benchmarks or herriot tests?
>

Here are a few I quickly pulled up:
MAPREDUCE-2316 (docs for improved capacity scheduler)
MAPREDUCE-2355 (adds new config for heartbeat dampening in MR)

"   BZ-4182948. Add statistics logging to Fred for better visibility into
startup time costs. (Matt Foley)"
- I believe I saw a note from Matt on the JIRA yesterday about this feature,
where he decided that the version done in 203 wasn't a good approach, and
it's done differently in trunk (not sure if done yet).

MAPREDUCE-2364 (important bug fix for localization)
- in fact most of localization is different in this branch compared to trunk
due to inclusion of MAPREDUCE-2378, the trunk version of which is still on
the "yahoo-merge" branch,.

"New cunters for FileInput/OutputFormat. New Counter
        MAP_OUTPUT_MATERIALZIED_BYTES. Related bugs: 4241034, 3418543,
4217546"
- not sure which JIRA this is, I think I've seen a JIRA for trunk, but not
committed.

- MAPREDUCE-1904, committed without JIRA as:
"        . Reducing new Path(), RawFileStatus() creation overhead in
LocalDirAllocator"
not in trunk

+    BZ4101537 .  When a queue is built without any access rights we explain
the
+    problem.  (dking, rvw ramach)  [attachment of 2010-11-24]
seems to be on trunk as MR-2411, but not committed, best I can tell, despite
the JIRA there being resolved (based on looking at QueueManager in trunk)

"        . Remove unnecessary reference to user configuration from
TaskDistributedCacheManager causing memory leaks"
Not in trunk, not sure which JIRA it might be.. probably part of 2178.

Major new feature: MAPREDUCE-323 - very large rework of how job history
files are managed
Major change: MAPREDUCE-1100/MAPREDUCE-1176: unresolved on trunk, though
probably will be attacked by different JIRAs
Major new ops-visible feature: "metrics2" system
Major new ops-visible feature: MAPREDUCE-291 job history can be viewed from
a separate server
Major new set of user-visible configurations: MAPREDUCE-1943 and friends
which implement new limits in MapReduce (eg MAPREDUCE-1872 as well)

I have code to work on, so I won't keep going, but this is from looking at
the last couple months of 203.

-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

--0016e6d7ef9057b2ee04a27bd876--

From general-return-3356-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 23:46:51 2011
Return-Path: <general-return-3356-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9F7CC372C
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  4 May 2011 23:46:51 +0000 (UTC)
Received: (qmail 22924 invoked by uid 500); 4 May 2011 23:46:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22868 invoked by uid 500); 4 May 2011 23:46:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22859 invoked by uid 99); 4 May 2011 23:46:50 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 23:46:50 +0000
Received: from localhost (HELO [192.168.168.16]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 23:46:49 +0000
Message-ID: <4DC1E56A.2010209@apache.org>
Date: Wed, 04 May 2011 16:46:50 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.14) Gecko/20110223 Thunderbird/3.1.8
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc0
References: <12853C59-08F1-4F98-ACF3-800F06BC7A5F@apache.org> <4DBEF0C1.2050306@apache.org> <BB3A69B5-9A5A-4454-80C8-62EBD3CF47EC@yahoo-inc.com> <4DBF16CF.3030905@apache.org> <247DDA36-BCD7-4EE9-A8CD-AD8E62C94F16@yahoo-inc.com> <4DBF2040.90602@apache.org> <5F90B548-37ED-4DB8-AAE3-D1C2BB5C700A@yahoo-inc.com> <4DC09AF6.70403@apache.org> <EDC1C3A4-003F-494E-B335-2D0CB1638738@yahoo-inc.com>
In-Reply-To: <EDC1C3A4-003F-494E-B335-2D0CB1638738@yahoo-inc.com>
X-Enigmail-Version: 1.1.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On 05/03/2011 06:01 PM, Arun C Murthy wrote:
> On May 3, 2011, at 5:17 PM, "Doug Cutting" <cutting@apache.org> wrote:
> 
>> On 05/02/2011 02:33 PM, Arun C Murthy wrote:
>>> Are you simply asking for someone to go through the 450 odd jiras and
>>> set 'fix-for' fields?
>>
>> Every other release we've made is well-correlated with Jira.  It should
>> not be difficult to achieve that for this one.  We could write a script
>> to take all 450 bug IDs from the change log and use Jira's command-line
>> tool to set the "fix-for" to be this 0.20+security release.  Would you
>> like help with that?
>>
> 
> Yes please, that would be great. Thanks!

Please find below a script that will add a fix-version to issues.

Doug

#!/bin/bash

# reads bug ids from standard input
# and adds the fixVersion named on command line

if [ $# -eq 0 ]
then
  echo "Usage: $0 bugid"
  exit 1
fi

fix=$1
echo Setting fix version to $fix.

server=https://issues.apache.org/jira
jira=./jira-cli-2.0.0/jira.sh

set -e

echo -n "Jira username: "
read user
echo -n "Jira password: "
stty -echo
read password
stty echo

while read issue
do
    # first read the old fix versions
    old=`$jira -a getFieldValue --server $server \
         --password $password --user $user \
         --issue $issue --field fixVersions | \
	 tail -n 1 | sed 's/([0-9]*)//g' | sed s/\'//g`

    # now update, adding new value
    # jira will ignore if this value is already present
    $jira -a updateIssue --server $server \
     --password $password --user $user \
     --issue $issue --fixVersions "${old},${fix}"
done

From general-return-3357-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 23:56:32 2011
Return-Path: <general-return-3357-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 248C3302E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed,  4 May 2011 23:56:32 +0000 (UTC)
Received: (qmail 29827 invoked by uid 500); 4 May 2011 23:56:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29775 invoked by uid 500); 4 May 2011 23:56:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29766 invoked by uid 99); 4 May 2011 23:56:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 23:56:30 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 23:56:24 +0000
Received: from l2tp-8-201.corp.yahoo.com (l2tp-8-201.corp.yahoo.com [10.73.8.201])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p44Ntgom059120
	for <general@hadoop.apache.org>; Wed, 4 May 2011 16:55:43 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1304553343;
	bh=S/fmmR6hku0Np/f2ltcuOpsy2EdIYfhET3ZXzHnIGQE=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=mHpCDCprYZFfkMs+d9avVxvNhB9F/VczHbqFi6PO5RdNFD0YjVh5ub5DEaCWMQ294
	 sZFktYMyUbeSdZp0W25HHpfnNTZO+roXvnkDITKJLuMXaP4jvn2OaJwK9m21X5Lozu
	 ZmZKSzHGQrwKvYchI0VpA+lnq5zVUZO3c8kQQrLk=
Message-Id: <96D8EC41-E319-4FCE-BF05-CDD9E06CC0AA@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <BANLkTi=aW6Toxtg7g=nFSV7h4gK68522wg@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
Date: Wed, 4 May 2011 16:55:42 -0700
References: <BANLkTik5gj0H4gmvy14Ux=g_r5Tt6D1=MA@mail.gmail.com> <C9E71BE8.19FCF%sureshms@yahoo-inc.com> <BANLkTikrS9PTeErT15fs2yv+N4_wpPN7xQ@mail.gmail.com> <BANLkTimjxRdf6PyTnbZUwrS_znZo00z-5w@mail.gmail.com> <BANLkTi=pNF_aQrWPurfXRbjNdshfseNobg@mail.gmail.com> <321761.51699.qm@web65903.mail.ac4.yahoo.com> <CB238EE8-1E06-4C6C-A7F5-CB00E2E073CB@yahoo-inc.com> <BANLkTi=aW6Toxtg7g=nFSV7h4gK68522wg@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

On May 4, 2011, at 4:44 PM, Todd Lipcon wrote:

> On Wed, May 4, 2011 at 4:11 PM, Arun C Murthy <acm@yahoo-inc.com>  
> wrote:
>
>> On May 4, 2011, at 4:09 PM, Tsz Wo (Nicholas), Sze wrote:
>>
>> The list seems highly inaccurate.  Checked the first few N/A  
>> items.  All
>>> are
>>> false positives.
>>>
>>>
>> Also,  can you please provide a list on features which are not  
>> related to
>> gridmix benchmarks or herriot tests?
>>
>
> Here are a few I quickly pulled up:

So, it's around 10? Approximately? Also, the ones you put up were  
reviewed via jira.

Please note that several of the ones you are pointing out are already  
in y-merge branch which is nearly trunk. including MR-2378 as you  
pointed out.

Thanks for the list, I'll ensure we work on forward porting them.

Arun


From general-return-3358-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 00:06:45 2011
Return-Path: <general-return-3358-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0E22D3693
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 00:06:45 +0000 (UTC)
Received: (qmail 40201 invoked by uid 500); 5 May 2011 00:06:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40140 invoked by uid 500); 5 May 2011 00:06:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40132 invoked by uid 99); 5 May 2011 00:06:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 00:06:43 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 00:06:36 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p4505xFN033775;
	Wed, 4 May 2011 17:05:59 -0700 (PDT)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by
 SP2-EX07CAS01.ds.corp.yahoo.com ([98.137.59.37]) with mapi; Wed, 4 May 2011
 17:05:59 -0700
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 4 May 2011 17:05:50 -0700
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
Thread-Topic: [VOTE] Release candidate 0.20.203.0-rc1
Thread-Index: AcwKuDasuitIdCivTxePKzk9K10thg==
Message-ID: <B746E44F-B6B3-40F2-BF75-C923E9B3962E@yahoo-inc.com>
References: <BANLkTik5gj0H4gmvy14Ux=g_r5Tt6D1=MA@mail.gmail.com>
 <C9E71BE8.19FCF%sureshms@yahoo-inc.com>
 <BANLkTikrS9PTeErT15fs2yv+N4_wpPN7xQ@mail.gmail.com>
 <BANLkTimjxRdf6PyTnbZUwrS_znZo00z-5w@mail.gmail.com>
 <BANLkTi=pNF_aQrWPurfXRbjNdshfseNobg@mail.gmail.com>
 <321761.51699.qm@web65903.mail.ac4.yahoo.com>
 <CB238EE8-1E06-4C6C-A7F5-CB00E2E073CB@yahoo-inc.com>
 <BANLkTi=aW6Toxtg7g=nFSV7h4gK68522wg@mail.gmail.com>
In-Reply-To: <BANLkTi=aW6Toxtg7g=nFSV7h4gK68522wg@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Hi folks,

Let's stay focused. Let's take the other threads onto other threads. This i=
s a vote.=20

To the extent naming is a problem, let's take that to a thread and find an =
acceptable proposal.=20

To the extent folks want to collaborate on certifying the release for total=
 lack of regression or collaborate on the cleanest possible merge, I think =
all interested parties should take these topics to another thread and divid=
e up the work.=20

If you've voted, you don't need to comment further on this thread, no matte=
r what company you work for!

Thanks,

---
E14 - typing on glass

On May 4, 2011, at 4:46 PM, "Todd Lipcon" <todd@cloudera.com> wrote:

> On Wed, May 4, 2011 at 4:11 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:
>=20
>> On May 4, 2011, at 4:09 PM, Tsz Wo (Nicholas), Sze wrote:
>>=20
>> The list seems highly inaccurate.  Checked the first few N/A items.  All
>>> are
>>> false positives.
>>>=20
>>>=20
>> Also,  can you please provide a list on features which are not related t=
o
>> gridmix benchmarks or herriot tests?
>>=20
>=20
> Here are a few I quickly pulled up:
> MAPREDUCE-2316 (docs for improved capacity scheduler)
> MAPREDUCE-2355 (adds new config for heartbeat dampening in MR)
>=20
> "   BZ-4182948. Add statistics logging to Fred for better visibility into
> startup time costs. (Matt Foley)"
> - I believe I saw a note from Matt on the JIRA yesterday about this featu=
re,
> where he decided that the version done in 203 wasn't a good approach, and
> it's done differently in trunk (not sure if done yet).
>=20
> MAPREDUCE-2364 (important bug fix for localization)
> - in fact most of localization is different in this branch compared to tr=
unk
> due to inclusion of MAPREDUCE-2378, the trunk version of which is still o=
n
> the "yahoo-merge" branch,.
>=20
> "New cunters for FileInput/OutputFormat. New Counter
>        MAP_OUTPUT_MATERIALZIED_BYTES. Related bugs: 4241034, 3418543,
> 4217546"
> - not sure which JIRA this is, I think I've seen a JIRA for trunk, but no=
t
> committed.
>=20
> - MAPREDUCE-1904, committed without JIRA as:
> "        . Reducing new Path(), RawFileStatus() creation overhead in
> LocalDirAllocator"
> not in trunk
>=20
> +    BZ4101537 .  When a queue is built without any access rights we expl=
ain
> the
> +    problem.  (dking, rvw ramach)  [attachment of 2010-11-24]
> seems to be on trunk as MR-2411, but not committed, best I can tell, desp=
ite
> the JIRA there being resolved (based on looking at QueueManager in trunk)
>=20
> "        . Remove unnecessary reference to user configuration from
> TaskDistributedCacheManager causing memory leaks"
> Not in trunk, not sure which JIRA it might be.. probably part of 2178.
>=20
> Major new feature: MAPREDUCE-323 - very large rework of how job history
> files are managed
> Major change: MAPREDUCE-1100/MAPREDUCE-1176: unresolved on trunk, though
> probably will be attacked by different JIRAs
> Major new ops-visible feature: "metrics2" system
> Major new ops-visible feature: MAPREDUCE-291 job history can be viewed fr=
om
> a separate server
> Major new set of user-visible configurations: MAPREDUCE-1943 and friends
> which implement new limits in MapReduce (eg MAPREDUCE-1872 as well)
>=20
> I have code to work on, so I won't keep going, but this is from looking a=
t
> the last couple months of 203.
>=20
> -Todd
> --=20
> Todd Lipcon
> Software Engineer, Cloudera

From general-return-3359-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 00:22:36 2011
Return-Path: <general-return-3359-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 578F93D11
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 00:22:36 +0000 (UTC)
Received: (qmail 54059 invoked by uid 500); 5 May 2011 00:22:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54011 invoked by uid 500); 5 May 2011 00:22:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54003 invoked by uid 99); 5 May 2011 00:22:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 00:22:34 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 00:22:28 +0000
Received: by ewy22 with SMTP id 22so745624ewy.35
        for <general@hadoop.apache.org>; Wed, 04 May 2011 17:22:08 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.14.17.100 with SMTP id i76mr829954eei.67.1304554928003; Wed,
 04 May 2011 17:22:08 -0700 (PDT)
Received: by 10.14.37.13 with HTTP; Wed, 4 May 2011 17:22:07 -0700 (PDT)
In-Reply-To: <B746E44F-B6B3-40F2-BF75-C923E9B3962E@yahoo-inc.com>
References: <BANLkTik5gj0H4gmvy14Ux=g_r5Tt6D1=MA@mail.gmail.com>
	<C9E71BE8.19FCF%sureshms@yahoo-inc.com>
	<BANLkTikrS9PTeErT15fs2yv+N4_wpPN7xQ@mail.gmail.com>
	<BANLkTimjxRdf6PyTnbZUwrS_znZo00z-5w@mail.gmail.com>
	<BANLkTi=pNF_aQrWPurfXRbjNdshfseNobg@mail.gmail.com>
	<321761.51699.qm@web65903.mail.ac4.yahoo.com>
	<CB238EE8-1E06-4C6C-A7F5-CB00E2E073CB@yahoo-inc.com>
	<BANLkTi=aW6Toxtg7g=nFSV7h4gK68522wg@mail.gmail.com>
	<B746E44F-B6B3-40F2-BF75-C923E9B3962E@yahoo-inc.com>
Date: Wed, 4 May 2011 17:22:07 -0700
Message-ID: <BANLkTinvH8uUAzOQUogqnBanioPZ3UmnHg@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Good suggestion, it would be helpful to hash out the issues around
compatibility, feature branches, version numbers, how to contribute at
Apache before putting up new votes that would be helpful, ie the vote
would go much smoother if all the issues with the previous vote were
addressed before starting a new one.

Thanks,
Eli

On Wed, May 4, 2011 at 5:05 PM, Eric Baldeschwieler
<eric14@yahoo-inc.com> wrote:
> Hi folks,
>
> Let's stay focused. Let's take the other threads onto other threads. This=
 is a vote.
>
> To the extent naming is a problem, let's take that to a thread and find a=
n acceptable proposal.
>
> To the extent folks want to collaborate on certifying the release for tot=
al lack of regression or collaborate on the cleanest possible merge, I thin=
k all interested parties should take these topics to another thread and div=
ide up the work.
>
> If you've voted, you don't need to comment further on this thread, no mat=
ter what company you work for!
>
> Thanks,
>
> ---
> E14 - typing on glass
>
> On May 4, 2011, at 4:46 PM, "Todd Lipcon" <todd@cloudera.com> wrote:
>
>> On Wed, May 4, 2011 at 4:11 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:
>>
>>> On May 4, 2011, at 4:09 PM, Tsz Wo (Nicholas), Sze wrote:
>>>
>>> The list seems highly inaccurate. =A0Checked the first few N/A items. =
=A0All
>>>> are
>>>> false positives.
>>>>
>>>>
>>> Also, =A0can you please provide a list on features which are not relate=
d to
>>> gridmix benchmarks or herriot tests?
>>>
>>
>> Here are a few I quickly pulled up:
>> MAPREDUCE-2316 (docs for improved capacity scheduler)
>> MAPREDUCE-2355 (adds new config for heartbeat dampening in MR)
>>
>> " =A0 BZ-4182948. Add statistics logging to Fred for better visibility i=
nto
>> startup time costs. (Matt Foley)"
>> - I believe I saw a note from Matt on the JIRA yesterday about this feat=
ure,
>> where he decided that the version done in 203 wasn't a good approach, an=
d
>> it's done differently in trunk (not sure if done yet).
>>
>> MAPREDUCE-2364 (important bug fix for localization)
>> - in fact most of localization is different in this branch compared to t=
runk
>> due to inclusion of MAPREDUCE-2378, the trunk version of which is still =
on
>> the "yahoo-merge" branch,.
>>
>> "New cunters for FileInput/OutputFormat. New Counter
>> =A0 =A0 =A0 =A0MAP_OUTPUT_MATERIALZIED_BYTES. Related bugs: 4241034, 341=
8543,
>> 4217546"
>> - not sure which JIRA this is, I think I've seen a JIRA for trunk, but n=
ot
>> committed.
>>
>> - MAPREDUCE-1904, committed without JIRA as:
>> " =A0 =A0 =A0 =A0. Reducing new Path(), RawFileStatus() creation overhea=
d in
>> LocalDirAllocator"
>> not in trunk
>>
>> + =A0 =A0BZ4101537 . =A0When a queue is built without any access rights =
we explain
>> the
>> + =A0 =A0problem. =A0(dking, rvw ramach) =A0[attachment of 2010-11-24]
>> seems to be on trunk as MR-2411, but not committed, best I can tell, des=
pite
>> the JIRA there being resolved (based on looking at QueueManager in trunk=
)
>>
>> " =A0 =A0 =A0 =A0. Remove unnecessary reference to user configuration fr=
om
>> TaskDistributedCacheManager causing memory leaks"
>> Not in trunk, not sure which JIRA it might be.. probably part of 2178.
>>
>> Major new feature: MAPREDUCE-323 - very large rework of how job history
>> files are managed
>> Major change: MAPREDUCE-1100/MAPREDUCE-1176: unresolved on trunk, though
>> probably will be attacked by different JIRAs
>> Major new ops-visible feature: "metrics2" system
>> Major new ops-visible feature: MAPREDUCE-291 job history can be viewed f=
rom
>> a separate server
>> Major new set of user-visible configurations: MAPREDUCE-1943 and friends
>> which implement new limits in MapReduce (eg MAPREDUCE-1872 as well)
>>
>> I have code to work on, so I won't keep going, but this is from looking =
at
>> the last couple months of 203.
>>
>> -Todd
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>

From general-return-3360-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 00:30:57 2011
Return-Path: <general-return-3360-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A7A23345C
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 00:30:57 +0000 (UTC)
Received: (qmail 59125 invoked by uid 500); 5 May 2011 00:30:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59065 invoked by uid 500); 5 May 2011 00:30:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59057 invoked by uid 99); 5 May 2011 00:30:56 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 00:30:56 +0000
Received: from localhost (HELO mail-fx0-f48.google.com) (127.0.0.1)
  (smtp-auth username mahadev, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 00:30:56 +0000
Received: by fxm7 with SMTP id 7so2189961fxm.35
        for <general@hadoop.apache.org>; Wed, 04 May 2011 17:30:54 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.223.106.70 with SMTP id w6mr1995575fao.68.1304555454277; Wed,
 04 May 2011 17:30:54 -0700 (PDT)
Received: by 10.223.144.197 with HTTP; Wed, 4 May 2011 17:30:54 -0700 (PDT)
In-Reply-To: <BANLkTinvH8uUAzOQUogqnBanioPZ3UmnHg@mail.gmail.com>
References: <BANLkTik5gj0H4gmvy14Ux=g_r5Tt6D1=MA@mail.gmail.com>
	<C9E71BE8.19FCF%sureshms@yahoo-inc.com>
	<BANLkTikrS9PTeErT15fs2yv+N4_wpPN7xQ@mail.gmail.com>
	<BANLkTimjxRdf6PyTnbZUwrS_znZo00z-5w@mail.gmail.com>
	<BANLkTi=pNF_aQrWPurfXRbjNdshfseNobg@mail.gmail.com>
	<321761.51699.qm@web65903.mail.ac4.yahoo.com>
	<CB238EE8-1E06-4C6C-A7F5-CB00E2E073CB@yahoo-inc.com>
	<BANLkTi=aW6Toxtg7g=nFSV7h4gK68522wg@mail.gmail.com>
	<B746E44F-B6B3-40F2-BF75-C923E9B3962E@yahoo-inc.com>
	<BANLkTinvH8uUAzOQUogqnBanioPZ3UmnHg@mail.gmail.com>
Date: Wed, 4 May 2011 17:30:54 -0700
Message-ID: <BANLkTineZVMp1Fgv+JFT7NNihvDeHxdrvA@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
From: Mahadev Konar <mahadev@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Eli,
  I think the intent from the email was to just vote on this thread,
which I agree with.
 Discussions should be done in a separate threads. Hopefully we can
all stick to just voting!

thanks
mahadev

On Wed, May 4, 2011 at 5:22 PM, Eli Collins <eli@cloudera.com> wrote:
> Good suggestion, it would be helpful to hash out the issues around
> compatibility, feature branches, version numbers, how to contribute at
> Apache before putting up new votes that would be helpful, ie the vote
> would go much smoother if all the issues with the previous vote were
> addressed before starting a new one.
>
> Thanks,
> Eli
>
> On Wed, May 4, 2011 at 5:05 PM, Eric Baldeschwieler
> <eric14@yahoo-inc.com> wrote:
>> Hi folks,
>>
>> Let's stay focused. Let's take the other threads onto other threads. Thi=
s is a vote.
>>
>> To the extent naming is a problem, let's take that to a thread and find =
an acceptable proposal.
>>
>> To the extent folks want to collaborate on certifying the release for to=
tal lack of regression or collaborate on the cleanest possible merge, I thi=
nk all interested parties should take these topics to another thread and di=
vide up the work.
>>
>> If you've voted, you don't need to comment further on this thread, no ma=
tter what company you work for!
>>
>> Thanks,
>>
>> ---
>> E14 - typing on glass
>>
>> On May 4, 2011, at 4:46 PM, "Todd Lipcon" <todd@cloudera.com> wrote:
>>
>>> On Wed, May 4, 2011 at 4:11 PM, Arun C Murthy <acm@yahoo-inc.com> wrote=
:
>>>
>>>> On May 4, 2011, at 4:09 PM, Tsz Wo (Nicholas), Sze wrote:
>>>>
>>>> The list seems highly inaccurate. =A0Checked the first few N/A items. =
=A0All
>>>>> are
>>>>> false positives.
>>>>>
>>>>>
>>>> Also, =A0can you please provide a list on features which are not relat=
ed to
>>>> gridmix benchmarks or herriot tests?
>>>>
>>>
>>> Here are a few I quickly pulled up:
>>> MAPREDUCE-2316 (docs for improved capacity scheduler)
>>> MAPREDUCE-2355 (adds new config for heartbeat dampening in MR)
>>>
>>> " =A0 BZ-4182948. Add statistics logging to Fred for better visibility =
into
>>> startup time costs. (Matt Foley)"
>>> - I believe I saw a note from Matt on the JIRA yesterday about this fea=
ture,
>>> where he decided that the version done in 203 wasn't a good approach, a=
nd
>>> it's done differently in trunk (not sure if done yet).
>>>
>>> MAPREDUCE-2364 (important bug fix for localization)
>>> - in fact most of localization is different in this branch compared to =
trunk
>>> due to inclusion of MAPREDUCE-2378, the trunk version of which is still=
 on
>>> the "yahoo-merge" branch,.
>>>
>>> "New cunters for FileInput/OutputFormat. New Counter
>>> =A0 =A0 =A0 =A0MAP_OUTPUT_MATERIALZIED_BYTES. Related bugs: 4241034, 34=
18543,
>>> 4217546"
>>> - not sure which JIRA this is, I think I've seen a JIRA for trunk, but =
not
>>> committed.
>>>
>>> - MAPREDUCE-1904, committed without JIRA as:
>>> " =A0 =A0 =A0 =A0. Reducing new Path(), RawFileStatus() creation overhe=
ad in
>>> LocalDirAllocator"
>>> not in trunk
>>>
>>> + =A0 =A0BZ4101537 . =A0When a queue is built without any access rights=
 we explain
>>> the
>>> + =A0 =A0problem. =A0(dking, rvw ramach) =A0[attachment of 2010-11-24]
>>> seems to be on trunk as MR-2411, but not committed, best I can tell, de=
spite
>>> the JIRA there being resolved (based on looking at QueueManager in trun=
k)
>>>
>>> " =A0 =A0 =A0 =A0. Remove unnecessary reference to user configuration f=
rom
>>> TaskDistributedCacheManager causing memory leaks"
>>> Not in trunk, not sure which JIRA it might be.. probably part of 2178.
>>>
>>> Major new feature: MAPREDUCE-323 - very large rework of how job history
>>> files are managed
>>> Major change: MAPREDUCE-1100/MAPREDUCE-1176: unresolved on trunk, thoug=
h
>>> probably will be attacked by different JIRAs
>>> Major new ops-visible feature: "metrics2" system
>>> Major new ops-visible feature: MAPREDUCE-291 job history can be viewed =
from
>>> a separate server
>>> Major new set of user-visible configurations: MAPREDUCE-1943 and friend=
s
>>> which implement new limits in MapReduce (eg MAPREDUCE-1872 as well)
>>>
>>> I have code to work on, so I won't keep going, but this is from looking=
 at
>>> the last couple months of 203.
>>>
>>> -Todd
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>
>



--=20
thanks
mahadev
@mahadevkonar

From general-return-3361-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 00:39:56 2011
Return-Path: <general-return-3361-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6EBB0357F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 00:39:56 +0000 (UTC)
Received: (qmail 72870 invoked by uid 500); 5 May 2011 00:39:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72825 invoked by uid 500); 5 May 2011 00:39:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72817 invoked by uid 99); 5 May 2011 00:39:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 00:39:55 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 209.85.215.176 as permitted sender)
Received: from [209.85.215.176] (HELO mail-ey0-f176.google.com) (209.85.215.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 00:39:48 +0000
Received: by eya28 with SMTP id 28so747979eya.35
        for <general@hadoop.apache.org>; Wed, 04 May 2011 17:39:28 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.14.5.200 with SMTP id 48mr834347eel.131.1304555968523; Wed, 04
 May 2011 17:39:28 -0700 (PDT)
Received: by 10.14.37.13 with HTTP; Wed, 4 May 2011 17:39:28 -0700 (PDT)
In-Reply-To: <BANLkTineZVMp1Fgv+JFT7NNihvDeHxdrvA@mail.gmail.com>
References: <BANLkTik5gj0H4gmvy14Ux=g_r5Tt6D1=MA@mail.gmail.com>
	<C9E71BE8.19FCF%sureshms@yahoo-inc.com>
	<BANLkTikrS9PTeErT15fs2yv+N4_wpPN7xQ@mail.gmail.com>
	<BANLkTimjxRdf6PyTnbZUwrS_znZo00z-5w@mail.gmail.com>
	<BANLkTi=pNF_aQrWPurfXRbjNdshfseNobg@mail.gmail.com>
	<321761.51699.qm@web65903.mail.ac4.yahoo.com>
	<CB238EE8-1E06-4C6C-A7F5-CB00E2E073CB@yahoo-inc.com>
	<BANLkTi=aW6Toxtg7g=nFSV7h4gK68522wg@mail.gmail.com>
	<B746E44F-B6B3-40F2-BF75-C923E9B3962E@yahoo-inc.com>
	<BANLkTinvH8uUAzOQUogqnBanioPZ3UmnHg@mail.gmail.com>
	<BANLkTineZVMp1Fgv+JFT7NNihvDeHxdrvA@mail.gmail.com>
Date: Wed, 4 May 2011 17:39:28 -0700
Message-ID: <BANLkTimpuUhDRL1S4iYQKp3A-eKWcHi8kg@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

The point is that these discussion should be sorted out, ie you don't
change your development and release model on a release VOTE thread,
you change it on a DISCUSSION thread.

Ie before we release this we should understand what that means. What
is being proposed is not just another release from branch-0.20 or
branch-0.22.

Thanks,
Eli

On Wed, May 4, 2011 at 5:30 PM, Mahadev Konar <mahadev@apache.org> wrote:
> Eli,
> =A0I think the intent from the email was to just vote on this thread,
> which I agree with.
> =A0Discussions should be done in a separate threads. Hopefully we can
> all stick to just voting!
>
> thanks
> mahadev
>
> On Wed, May 4, 2011 at 5:22 PM, Eli Collins <eli@cloudera.com> wrote:
>> Good suggestion, it would be helpful to hash out the issues around
>> compatibility, feature branches, version numbers, how to contribute at
>> Apache before putting up new votes that would be helpful, ie the vote
>> would go much smoother if all the issues with the previous vote were
>> addressed before starting a new one.
>>
>> Thanks,
>> Eli
>>
>> On Wed, May 4, 2011 at 5:05 PM, Eric Baldeschwieler
>> <eric14@yahoo-inc.com> wrote:
>>> Hi folks,
>>>
>>> Let's stay focused. Let's take the other threads onto other threads. Th=
is is a vote.
>>>
>>> To the extent naming is a problem, let's take that to a thread and find=
 an acceptable proposal.
>>>
>>> To the extent folks want to collaborate on certifying the release for t=
otal lack of regression or collaborate on the cleanest possible merge, I th=
ink all interested parties should take these topics to another thread and d=
ivide up the work.
>>>
>>> If you've voted, you don't need to comment further on this thread, no m=
atter what company you work for!
>>>
>>> Thanks,
>>>
>>> ---
>>> E14 - typing on glass
>>>
>>> On May 4, 2011, at 4:46 PM, "Todd Lipcon" <todd@cloudera.com> wrote:
>>>
>>>> On Wed, May 4, 2011 at 4:11 PM, Arun C Murthy <acm@yahoo-inc.com> wrot=
e:
>>>>
>>>>> On May 4, 2011, at 4:09 PM, Tsz Wo (Nicholas), Sze wrote:
>>>>>
>>>>> The list seems highly inaccurate. =A0Checked the first few N/A items.=
 =A0All
>>>>>> are
>>>>>> false positives.
>>>>>>
>>>>>>
>>>>> Also, =A0can you please provide a list on features which are not rela=
ted to
>>>>> gridmix benchmarks or herriot tests?
>>>>>
>>>>
>>>> Here are a few I quickly pulled up:
>>>> MAPREDUCE-2316 (docs for improved capacity scheduler)
>>>> MAPREDUCE-2355 (adds new config for heartbeat dampening in MR)
>>>>
>>>> " =A0 BZ-4182948. Add statistics logging to Fred for better visibility=
 into
>>>> startup time costs. (Matt Foley)"
>>>> - I believe I saw a note from Matt on the JIRA yesterday about this fe=
ature,
>>>> where he decided that the version done in 203 wasn't a good approach, =
and
>>>> it's done differently in trunk (not sure if done yet).
>>>>
>>>> MAPREDUCE-2364 (important bug fix for localization)
>>>> - in fact most of localization is different in this branch compared to=
 trunk
>>>> due to inclusion of MAPREDUCE-2378, the trunk version of which is stil=
l on
>>>> the "yahoo-merge" branch,.
>>>>
>>>> "New cunters for FileInput/OutputFormat. New Counter
>>>> =A0 =A0 =A0 =A0MAP_OUTPUT_MATERIALZIED_BYTES. Related bugs: 4241034, 3=
418543,
>>>> 4217546"
>>>> - not sure which JIRA this is, I think I've seen a JIRA for trunk, but=
 not
>>>> committed.
>>>>
>>>> - MAPREDUCE-1904, committed without JIRA as:
>>>> " =A0 =A0 =A0 =A0. Reducing new Path(), RawFileStatus() creation overh=
ead in
>>>> LocalDirAllocator"
>>>> not in trunk
>>>>
>>>> + =A0 =A0BZ4101537 . =A0When a queue is built without any access right=
s we explain
>>>> the
>>>> + =A0 =A0problem. =A0(dking, rvw ramach) =A0[attachment of 2010-11-24]
>>>> seems to be on trunk as MR-2411, but not committed, best I can tell, d=
espite
>>>> the JIRA there being resolved (based on looking at QueueManager in tru=
nk)
>>>>
>>>> " =A0 =A0 =A0 =A0. Remove unnecessary reference to user configuration =
from
>>>> TaskDistributedCacheManager causing memory leaks"
>>>> Not in trunk, not sure which JIRA it might be.. probably part of 2178.
>>>>
>>>> Major new feature: MAPREDUCE-323 - very large rework of how job histor=
y
>>>> files are managed
>>>> Major change: MAPREDUCE-1100/MAPREDUCE-1176: unresolved on trunk, thou=
gh
>>>> probably will be attacked by different JIRAs
>>>> Major new ops-visible feature: "metrics2" system
>>>> Major new ops-visible feature: MAPREDUCE-291 job history can be viewed=
 from
>>>> a separate server
>>>> Major new set of user-visible configurations: MAPREDUCE-1943 and frien=
ds
>>>> which implement new limits in MapReduce (eg MAPREDUCE-1872 as well)
>>>>
>>>> I have code to work on, so I won't keep going, but this is from lookin=
g at
>>>> the last couple months of 203.
>>>>
>>>> -Todd
>>>> --
>>>> Todd Lipcon
>>>> Software Engineer, Cloudera
>>>
>>
>
>
>
> --
> thanks
> mahadev
> @mahadevkonar
>

From general-return-3362-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 00:44:30 2011
Return-Path: <general-return-3362-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D8B3D372B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 00:44:30 +0000 (UTC)
Received: (qmail 76793 invoked by uid 500); 5 May 2011 00:44:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76738 invoked by uid 500); 5 May 2011 00:44:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76730 invoked by uid 99); 5 May 2011 00:44:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 00:44:29 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.171] (HELO mail-px0-f171.google.com) (209.85.212.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 00:44:25 +0000
Received: by pxi7 with SMTP id 7so1142434pxi.2
        for <general@hadoop.apache.org>; Wed, 04 May 2011 17:44:04 -0700 (PDT)
Received: by 10.68.15.168 with SMTP id y8mr2247377pbc.337.1304556244099; Wed,
 04 May 2011 17:44:04 -0700 (PDT)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.68.41.230 with HTTP; Wed, 4 May 2011 17:43:44 -0700 (PDT)
In-Reply-To: <BANLkTimpuUhDRL1S4iYQKp3A-eKWcHi8kg@mail.gmail.com>
References: <BANLkTik5gj0H4gmvy14Ux=g_r5Tt6D1=MA@mail.gmail.com>
 <C9E71BE8.19FCF%sureshms@yahoo-inc.com> <BANLkTikrS9PTeErT15fs2yv+N4_wpPN7xQ@mail.gmail.com>
 <BANLkTimjxRdf6PyTnbZUwrS_znZo00z-5w@mail.gmail.com> <BANLkTi=pNF_aQrWPurfXRbjNdshfseNobg@mail.gmail.com>
 <321761.51699.qm@web65903.mail.ac4.yahoo.com> <CB238EE8-1E06-4C6C-A7F5-CB00E2E073CB@yahoo-inc.com>
 <BANLkTi=aW6Toxtg7g=nFSV7h4gK68522wg@mail.gmail.com> <B746E44F-B6B3-40F2-BF75-C923E9B3962E@yahoo-inc.com>
 <BANLkTinvH8uUAzOQUogqnBanioPZ3UmnHg@mail.gmail.com> <BANLkTineZVMp1Fgv+JFT7NNihvDeHxdrvA@mail.gmail.com>
 <BANLkTimpuUhDRL1S4iYQKp3A-eKWcHi8kg@mail.gmail.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Wed, 4 May 2011 17:43:44 -0700
X-Google-Sender-Auth: CUiGBQrVBnER0AQhW8a145KO9hw
Message-ID: <BANLkTik5YqkryXkWJOrPxon0NaLZtQoK1w@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I tend to agree. Changing release model of Apache Hadoop train isn't
something that should be done in a hassle or as a part of release
voting.

If these questions aren't addressed - let's postpone the vote and
discuss all the complications or implications until they sorted out or
the consensus/compromise is reached.

Cos

On Wed, May 4, 2011 at 17:39, Eli Collins <eli@cloudera.com> wrote:
> The point is that these discussion should be sorted out, ie you don't
> change your development and release model on a release VOTE thread,
> you change it on a DISCUSSION thread.
>
> Ie before we release this we should understand what that means. What
> is being proposed is not just another release from branch-0.20 or
> branch-0.22.
>
> Thanks,
> Eli
>
> On Wed, May 4, 2011 at 5:30 PM, Mahadev Konar <mahadev@apache.org> wrote:
>> Eli,
>> =A0I think the intent from the email was to just vote on this thread,
>> which I agree with.
>> =A0Discussions should be done in a separate threads. Hopefully we can
>> all stick to just voting!
>>
>> thanks
>> mahadev
>>
>> On Wed, May 4, 2011 at 5:22 PM, Eli Collins <eli@cloudera.com> wrote:
>>> Good suggestion, it would be helpful to hash out the issues around
>>> compatibility, feature branches, version numbers, how to contribute at
>>> Apache before putting up new votes that would be helpful, ie the vote
>>> would go much smoother if all the issues with the previous vote were
>>> addressed before starting a new one.
>>>
>>> Thanks,
>>> Eli
>>>
>>> On Wed, May 4, 2011 at 5:05 PM, Eric Baldeschwieler
>>> <eric14@yahoo-inc.com> wrote:
>>>> Hi folks,
>>>>
>>>> Let's stay focused. Let's take the other threads onto other threads. T=
his is a vote.
>>>>
>>>> To the extent naming is a problem, let's take that to a thread and fin=
d an acceptable proposal.
>>>>
>>>> To the extent folks want to collaborate on certifying the release for =
total lack of regression or collaborate on the cleanest possible merge, I t=
hink all interested parties should take these topics to another thread and =
divide up the work.
>>>>
>>>> If you've voted, you don't need to comment further on this thread, no =
matter what company you work for!
>>>>
>>>> Thanks,
>>>>
>>>> ---
>>>> E14 - typing on glass
>>>>
>>>> On May 4, 2011, at 4:46 PM, "Todd Lipcon" <todd@cloudera.com> wrote:
>>>>
>>>>> On Wed, May 4, 2011 at 4:11 PM, Arun C Murthy <acm@yahoo-inc.com> wro=
te:
>>>>>
>>>>>> On May 4, 2011, at 4:09 PM, Tsz Wo (Nicholas), Sze wrote:
>>>>>>
>>>>>> The list seems highly inaccurate. =A0Checked the first few N/A items=
. =A0All
>>>>>>> are
>>>>>>> false positives.
>>>>>>>
>>>>>>>
>>>>>> Also, =A0can you please provide a list on features which are not rel=
ated to
>>>>>> gridmix benchmarks or herriot tests?
>>>>>>
>>>>>
>>>>> Here are a few I quickly pulled up:
>>>>> MAPREDUCE-2316 (docs for improved capacity scheduler)
>>>>> MAPREDUCE-2355 (adds new config for heartbeat dampening in MR)
>>>>>
>>>>> " =A0 BZ-4182948. Add statistics logging to Fred for better visibilit=
y into
>>>>> startup time costs. (Matt Foley)"
>>>>> - I believe I saw a note from Matt on the JIRA yesterday about this f=
eature,
>>>>> where he decided that the version done in 203 wasn't a good approach,=
 and
>>>>> it's done differently in trunk (not sure if done yet).
>>>>>
>>>>> MAPREDUCE-2364 (important bug fix for localization)
>>>>> - in fact most of localization is different in this branch compared t=
o trunk
>>>>> due to inclusion of MAPREDUCE-2378, the trunk version of which is sti=
ll on
>>>>> the "yahoo-merge" branch,.
>>>>>
>>>>> "New cunters for FileInput/OutputFormat. New Counter
>>>>> =A0 =A0 =A0 =A0MAP_OUTPUT_MATERIALZIED_BYTES. Related bugs: 4241034, =
3418543,
>>>>> 4217546"
>>>>> - not sure which JIRA this is, I think I've seen a JIRA for trunk, bu=
t not
>>>>> committed.
>>>>>
>>>>> - MAPREDUCE-1904, committed without JIRA as:
>>>>> " =A0 =A0 =A0 =A0. Reducing new Path(), RawFileStatus() creation over=
head in
>>>>> LocalDirAllocator"
>>>>> not in trunk
>>>>>
>>>>> + =A0 =A0BZ4101537 . =A0When a queue is built without any access righ=
ts we explain
>>>>> the
>>>>> + =A0 =A0problem. =A0(dking, rvw ramach) =A0[attachment of 2010-11-24=
]
>>>>> seems to be on trunk as MR-2411, but not committed, best I can tell, =
despite
>>>>> the JIRA there being resolved (based on looking at QueueManager in tr=
unk)
>>>>>
>>>>> " =A0 =A0 =A0 =A0. Remove unnecessary reference to user configuration=
 from
>>>>> TaskDistributedCacheManager causing memory leaks"
>>>>> Not in trunk, not sure which JIRA it might be.. probably part of 2178=
.
>>>>>
>>>>> Major new feature: MAPREDUCE-323 - very large rework of how job histo=
ry
>>>>> files are managed
>>>>> Major change: MAPREDUCE-1100/MAPREDUCE-1176: unresolved on trunk, tho=
ugh
>>>>> probably will be attacked by different JIRAs
>>>>> Major new ops-visible feature: "metrics2" system
>>>>> Major new ops-visible feature: MAPREDUCE-291 job history can be viewe=
d from
>>>>> a separate server
>>>>> Major new set of user-visible configurations: MAPREDUCE-1943 and frie=
nds
>>>>> which implement new limits in MapReduce (eg MAPREDUCE-1872 as well)
>>>>>
>>>>> I have code to work on, so I won't keep going, but this is from looki=
ng at
>>>>> the last couple months of 203.
>>>>>
>>>>> -Todd
>>>>> --
>>>>> Todd Lipcon
>>>>> Software Engineer, Cloudera
>>>>
>>>
>>
>>
>>
>> --
>> thanks
>> mahadev
>> @mahadevkonar
>>
>

From general-return-3363-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 01:00:04 2011
Return-Path: <general-return-3363-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0940F3D04
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 01:00:04 +0000 (UTC)
Received: (qmail 92420 invoked by uid 500); 5 May 2011 01:00:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92359 invoked by uid 500); 5 May 2011 01:00:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 92351 invoked by uid 99); 5 May 2011 01:00:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 01:00:02 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of tom@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 00:59:54 +0000
Received: by bwz8 with SMTP id 8so2494543bwz.35
        for <general@hadoop.apache.org>; Wed, 04 May 2011 17:59:34 -0700 (PDT)
Received: by 10.204.8.72 with SMTP id g8mr1777088bkg.103.1304557174344; Wed,
 04 May 2011 17:59:34 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.205.81.136 with HTTP; Wed, 4 May 2011 17:59:14 -0700 (PDT)
From: Tom White <tom@cloudera.com>
Date: Wed, 4 May 2011 17:59:14 -0700
Message-ID: <BANLkTimzcTyLoLppo9LxAJuXgzsqM7Ppew@mail.gmail.com>
Subject: [DISCUSSION] Release rules
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

One year ago (to the day!) Chris started a discussion about the
release manager role
(http://mail-archives.apache.org/mod_mbox/hadoop-general/201005.mbox/%3Ch2q1267dd3b1005041331r7d8f696di370a279ff605832f@mail.gmail.com%3E).
In light of today's disagreements, I think we should restart this
discussion and incorporate these rules into the bylaws, since it
formalizes our practices.

I'm happy to drive this. We could start by discussing Chris' proposal
(see clarifications in
http://mail-archives.apache.org/mod_mbox/hadoop-general/201005.mbox/%3Ct2y1267dd3b1005051201h7116e4caud75673ac9d5128d6@mail.gmail.com%3E),
then when we get consensus we can put the document on the website.
(BTW does anyone know if the bylaws were checked into SVN anywhere?
These belong together.)

Cheers,
Tom

From general-return-3364-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 01:11:21 2011
Return-Path: <general-return-3364-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A636695D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 01:11:21 +0000 (UTC)
Received: (qmail 1912 invoked by uid 500); 5 May 2011 01:11:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1746 invoked by uid 500); 5 May 2011 01:11:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1738 invoked by uid 99); 5 May 2011 01:11:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 01:11:20 +0000
X-ASF-Spam-Status: No, hits=3.3 required=5.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 01:11:12 +0000
Received: from sp1-ex07cas03.ds.corp.yahoo.com (sp1-ex07cas03.ds.corp.yahoo.com [216.252.116.151])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p451AMih051214
	for <general@hadoop.apache.org>; Wed, 4 May 2011 18:10:22 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1304557822;
	bh=6lKNMk6P34U+jTzjd3R2nC52t9tSOaYYNrdGFqVWKS4=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 MIME-Version;
	b=HSlpdOLM7dzKzWyTFdVrS+owQJGMw9bxQtw2yVR5qDkzTSJ0tkbIVyHYwzzky7ku6
	 No3pgLG8vfS7El2yUWDwHayQjpan1oT0zYEFIONlyDsuxGQPNFhEW3s22G13F3WOzO
	 2AhJNmUVDs0uIgUhfwnuogNCuD4Ow7fNxxTEjkNw=
Received: from SP1-EX07VS02.ds.corp.yahoo.com ([216.252.116.135]) by
 sp1-ex07cas03.ds.corp.yahoo.com ([216.252.116.151]) with mapi; Wed, 4 May
 2011 18:10:21 -0700
From: Devaraj Das <ddas@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 4 May 2011 18:10:19 -0700
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
Thread-Topic: [VOTE] Release candidate 0.20.203.0-rc1
Thread-Index: AcwKgWNdu2v/kISbSx+vh7BYIk9JfwAP8/Kk
Message-ID: <C9E7470B.34E6C%ddas@yahoo-inc.com>
In-Reply-To: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C9E7470B34E6Cddasyahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C9E7470B34E6Cddasyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

+1 based on some single node tests I did (with security ON).


On 5/4/11 10:31 AM, "Owen O'Malley" <omalley@apache.org> wrote:

Here's an updated release candidate for 0.20.203.0. I've incorporated the f=
eedback and included all of the patches from 0.20.2, which is the last stab=
le release. I also fixed the eclipse-plugin problem.

The candidate is at: http://people.apache.org/~omalley/hadoop-0.20.203.0-rc=
1/

Please download it, inspect it, compile it, and test it. Clearly, I'm +1.

-- Owen


--_000_C9E7470B34E6Cddasyahooinccom_--

From general-return-3365-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 01:12:32 2011
Return-Path: <general-return-3365-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B75BD986
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 01:12:32 +0000 (UTC)
Received: (qmail 3412 invoked by uid 500); 5 May 2011 01:12:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3365 invoked by uid 500); 5 May 2011 01:12:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3357 invoked by uid 99); 5 May 2011 01:12:31 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 01:12:31 +0000
Received: from localhost (HELO mail-iy0-f176.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 01:12:31 +0000
Received: by iym1 with SMTP id 1so2280453iym.35
        for <general@hadoop.apache.org>; Wed, 04 May 2011 18:12:30 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.139.202 with SMTP id f10mr788816ibu.36.1304557950478; Wed,
 04 May 2011 18:12:30 -0700 (PDT)
Received: by 10.231.143.3 with HTTP; Wed, 4 May 2011 18:12:30 -0700 (PDT)
In-Reply-To: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org>
References: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org>
Date: Wed, 4 May 2011 18:12:30 -0700
Message-ID: <BANLkTinkd3SwXrchfm5DUW=GEXG3Edx8SA@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

I'm +1 on releasing rc1. The signature and hashes match on the
artifact, ran some of the more aggressive MR tests. Reviewed changes
from rc0.

It looks like we need a FAQ for this release, if only to prevent the
same questions from being asked and answered across different threads
and lists. Reservations, regressions, and pending work can also be
documented there.

Right now, Apache Hadoop releases are not recommended by its
community. Instead, not only our end users, but other Apache projects
run Cloudera's distribution. From all those wearing their Apache hat,
I would like to see more effort directed toward a release that we can
recommend soon and less time spent compiling tasks to delay it.

Releasing this will complicate the documented process. However, that
process *has not produced a usable release* for the last two out of
six years. This is failure. Entertaining concerns like a one-to-one
correspondence between commits and JIRA issues is bizarre in this
context. Let's find a way to make progress instead of tossing
pharisaic accusations of illegitimacy. -C

On Wed, May 4, 2011 at 10:31 AM, Owen O'Malley <omalley@apache.org> wrote:
> Here's an updated release candidate for 0.20.203.0. I've incorporated the feedback and included all of the patches from 0.20.2, which is the last stable release. I also fixed the eclipse-plugin problem.
>
> The candidate is at: http://people.apache.org/~omalley/hadoop-0.20.203.0-rc1/
>
> Please download it, inspect it, compile it, and test it. Clearly, I'm +1.
>
> -- Owen

From general-return-3366-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 01:19:32 2011
Return-Path: <general-return-3366-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0C7441000
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 01:19:32 +0000 (UTC)
Received: (qmail 11381 invoked by uid 500); 5 May 2011 01:19:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11329 invoked by uid 500); 5 May 2011 01:19:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11321 invoked by uid 99); 5 May 2011 01:19:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 01:19:30 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 01:19:24 +0000
Received: from SP2-EX07CAS04.ds.corp.yahoo.com (sp2-ex07cas04.corp.sp2.yahoo.com [98.137.59.5])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p451IaQb027338;
	Wed, 4 May 2011 18:18:36 -0700 (PDT)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by
 SP2-EX07CAS04.ds.corp.yahoo.com ([98.137.59.5]) with mapi; Wed, 4 May 2011
 18:18:36 -0700
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 4 May 2011 18:18:24 -0700
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
Thread-Topic: [VOTE] Release candidate 0.20.203.0-rc1
Thread-Index: AcwKwluzZkZSDE6qR+WWzMYpk6J2Xw==
Message-ID: <C0BE4B0B-D99C-4E9B-9748-AD0FFD5B5766@yahoo-inc.com>
References: <BANLkTik5gj0H4gmvy14Ux=g_r5Tt6D1=MA@mail.gmail.com>
 <C9E71BE8.19FCF%sureshms@yahoo-inc.com>
 <BANLkTikrS9PTeErT15fs2yv+N4_wpPN7xQ@mail.gmail.com>
 <BANLkTimjxRdf6PyTnbZUwrS_znZo00z-5w@mail.gmail.com>
 <BANLkTi=pNF_aQrWPurfXRbjNdshfseNobg@mail.gmail.com>
 <321761.51699.qm@web65903.mail.ac4.yahoo.com>
 <CB238EE8-1E06-4C6C-A7F5-CB00E2E073CB@yahoo-inc.com>
 <BANLkTi=aW6Toxtg7g=nFSV7h4gK68522wg@mail.gmail.com>
 <B746E44F-B6B3-40F2-BF75-C923E9B3962E@yahoo-inc.com>
 <BANLkTinvH8uUAzOQUogqnBanioPZ3UmnHg@mail.gmail.com>
 <BANLkTineZVMp1Fgv+JFT7NNihvDeHxdrvA@mail.gmail.com>
 <BANLkTimpuUhDRL1S4iYQKp3A-eKWcHi8kg@mail.gmail.com>
 <BANLkTik5YqkryXkWJOrPxon0NaLZtQoK1w@mail.gmail.com>
In-Reply-To: <BANLkTik5YqkryXkWJOrPxon0NaLZtQoK1w@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Ok. I'll bite.

The point of a vote is to learn what everyone thinks. So far we have learne=
d:

1 - the team that is trying to contribute code and do a release thinks it i=
s ready.=20

2 - Cloudera does not think the release is a good idea.=20

No more talk between the Team contributing code and cloudera will educate u=
s further  Let's hear from the rest of the community.=20

In parallel on other threads, let's work out how to address concerns. That =
will be useful however the vote goes. I promise to continue to work with ev=
eryone to help drive releases.=20

We've called a vote, so let it proceed. That is how apache works.=20

Thanks!

---
E14 - typing on glass

PS this is my last comment on this thread. Start new ones if you are not ca=
sting a vote.=20

On May 4, 2011, at 5:45 PM, "Konstantin Boudnik" <cos@apache.org> wrote:

> I tend to agree. Changing release model of Apache Hadoop train isn't
> something that should be done in a hassle or as a part of release
> voting.
>=20
> If these questions aren't addressed - let's postpone the vote and
> discuss all the complications or implications until they sorted out or
> the consensus/compromise is reached.
>=20
> Cos
>=20
> On Wed, May 4, 2011 at 17:39, Eli Collins <eli@cloudera.com> wrote:
>> The point is that these discussion should be sorted out, ie you don't
>> change your development and release model on a release VOTE thread,
>> you change it on a DISCUSSION thread.
>>=20
>> Ie before we release this we should understand what that means. What
>> is being proposed is not just another release from branch-0.20 or
>> branch-0.22.
>>=20
>> Thanks,
>> Eli
>>=20
>> On Wed, May 4, 2011 at 5:30 PM, Mahadev Konar <mahadev@apache.org> wrote=
:
>>> Eli,
>>>  I think the intent from the email was to just vote on this thread,
>>> which I agree with.
>>>  Discussions should be done in a separate threads. Hopefully we can
>>> all stick to just voting!
>>>=20
>>> thanks
>>> mahadev
>>>=20
>>> On Wed, May 4, 2011 at 5:22 PM, Eli Collins <eli@cloudera.com> wrote:
>>>> Good suggestion, it would be helpful to hash out the issues around
>>>> compatibility, feature branches, version numbers, how to contribute at
>>>> Apache before putting up new votes that would be helpful, ie the vote
>>>> would go much smoother if all the issues with the previous vote were
>>>> addressed before starting a new one.
>>>>=20
>>>> Thanks,
>>>> Eli
>>>>=20
>>>> On Wed, May 4, 2011 at 5:05 PM, Eric Baldeschwieler
>>>> <eric14@yahoo-inc.com> wrote:
>>>>> Hi folks,
>>>>>=20
>>>>> Let's stay focused. Let's take the other threads onto other threads. =
This is a vote.
>>>>>=20
>>>>> To the extent naming is a problem, let's take that to a thread and fi=
nd an acceptable proposal.
>>>>>=20
>>>>> To the extent folks want to collaborate on certifying the release for=
 total lack of regression or collaborate on the cleanest possible merge, I =
think all interested parties should take these topics to another thread and=
 divide up the work.
>>>>>=20
>>>>> If you've voted, you don't need to comment further on this thread, no=
 matter what company you work for!
>>>>>=20
>>>>> Thanks,
>>>>>=20
>>>>> ---
>>>>> E14 - typing on glass
>>>>>=20
>>>>> On May 4, 2011, at 4:46 PM, "Todd Lipcon" <todd@cloudera.com> wrote:
>>>>>=20
>>>>>> On Wed, May 4, 2011 at 4:11 PM, Arun C Murthy <acm@yahoo-inc.com> wr=
ote:
>>>>>>=20
>>>>>>> On May 4, 2011, at 4:09 PM, Tsz Wo (Nicholas), Sze wrote:
>>>>>>>=20
>>>>>>> The list seems highly inaccurate.  Checked the first few N/A items.=
  All
>>>>>>>> are
>>>>>>>> false positives.
>>>>>>>>=20
>>>>>>>>=20
>>>>>>> Also,  can you please provide a list on features which are not rela=
ted to
>>>>>>> gridmix benchmarks or herriot tests?
>>>>>>>=20
>>>>>>=20
>>>>>> Here are a few I quickly pulled up:
>>>>>> MAPREDUCE-2316 (docs for improved capacity scheduler)
>>>>>> MAPREDUCE-2355 (adds new config for heartbeat dampening in MR)
>>>>>>=20
>>>>>> "   BZ-4182948. Add statistics logging to Fred for better visibility=
 into
>>>>>> startup time costs. (Matt Foley)"
>>>>>> - I believe I saw a note from Matt on the JIRA yesterday about this =
feature,
>>>>>> where he decided that the version done in 203 wasn't a good approach=
, and
>>>>>> it's done differently in trunk (not sure if done yet).
>>>>>>=20
>>>>>> MAPREDUCE-2364 (important bug fix for localization)
>>>>>> - in fact most of localization is different in this branch compared =
to trunk
>>>>>> due to inclusion of MAPREDUCE-2378, the trunk version of which is st=
ill on
>>>>>> the "yahoo-merge" branch,.
>>>>>>=20
>>>>>> "New cunters for FileInput/OutputFormat. New Counter
>>>>>>        MAP_OUTPUT_MATERIALZIED_BYTES. Related bugs: 4241034, 3418543=
,
>>>>>> 4217546"
>>>>>> - not sure which JIRA this is, I think I've seen a JIRA for trunk, b=
ut not
>>>>>> committed.
>>>>>>=20
>>>>>> - MAPREDUCE-1904, committed without JIRA as:
>>>>>> "        . Reducing new Path(), RawFileStatus() creation overhead in
>>>>>> LocalDirAllocator"
>>>>>> not in trunk
>>>>>>=20
>>>>>> +    BZ4101537 .  When a queue is built without any access rights we=
 explain
>>>>>> the
>>>>>> +    problem.  (dking, rvw ramach)  [attachment of 2010-11-24]
>>>>>> seems to be on trunk as MR-2411, but not committed, best I can tell,=
 despite
>>>>>> the JIRA there being resolved (based on looking at QueueManager in t=
runk)
>>>>>>=20
>>>>>> "        . Remove unnecessary reference to user configuration from
>>>>>> TaskDistributedCacheManager causing memory leaks"
>>>>>> Not in trunk, not sure which JIRA it might be.. probably part of 217=
8.
>>>>>>=20
>>>>>> Major new feature: MAPREDUCE-323 - very large rework of how job hist=
ory
>>>>>> files are managed
>>>>>> Major change: MAPREDUCE-1100/MAPREDUCE-1176: unresolved on trunk, th=
ough
>>>>>> probably will be attacked by different JIRAs
>>>>>> Major new ops-visible feature: "metrics2" system
>>>>>> Major new ops-visible feature: MAPREDUCE-291 job history can be view=
ed from
>>>>>> a separate server
>>>>>> Major new set of user-visible configurations: MAPREDUCE-1943 and fri=
ends
>>>>>> which implement new limits in MapReduce (eg MAPREDUCE-1872 as well)
>>>>>>=20
>>>>>> I have code to work on, so I won't keep going, but this is from look=
ing at
>>>>>> the last couple months of 203.
>>>>>>=20
>>>>>> -Todd
>>>>>> --
>>>>>> Todd Lipcon
>>>>>> Software Engineer, Cloudera
>>>>>=20
>>>>=20
>>>=20
>>>=20
>>>=20
>>> --
>>> thanks
>>> mahadev
>>> @mahadevkonar
>>>=20
>>=20

From general-return-3367-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 01:19:50 2011
Return-Path: <general-return-3367-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D26DE3E0
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 01:19:50 +0000 (UTC)
Received: (qmail 13070 invoked by uid 500); 5 May 2011 01:19:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13025 invoked by uid 500); 5 May 2011 01:19:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13016 invoked by uid 99); 5 May 2011 01:19:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 01:19:49 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 209.85.215.176 as permitted sender)
Received: from [209.85.215.176] (HELO mail-ey0-f176.google.com) (209.85.215.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 01:19:41 +0000
Received: by eya28 with SMTP id 28so757720eya.35
        for <general@hadoop.apache.org>; Wed, 04 May 2011 18:19:21 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.14.17.100 with SMTP id i76mr842992eei.67.1304558361545; Wed,
 04 May 2011 18:19:21 -0700 (PDT)
Received: by 10.14.37.13 with HTTP; Wed, 4 May 2011 18:19:21 -0700 (PDT)
In-Reply-To: <BANLkTinkd3SwXrchfm5DUW=GEXG3Edx8SA@mail.gmail.com>
References: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org>
	<BANLkTinkd3SwXrchfm5DUW=GEXG3Edx8SA@mail.gmail.com>
Date: Wed, 4 May 2011 18:19:21 -0700
Message-ID: <BANLkTimBqCw+9KtmaMxJMBP0qOBb4kyMQA@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

> Entertaining concerns like a one-to-one
> correspondence between commits and JIRA issues is bizarre in this
> context.

It's not about whether there's a jira, it's about whether the code was
reviewed.  We think code should be reviewed and vote on by the
community before releasing it. That's how we've always rolled.

Everyone agrees releases are too infrequent, that's not an excuse for
steam rolling the community.

Thanks,
Eli

From general-return-3368-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 01:25:10 2011
Return-Path: <general-return-3368-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 46EA1698
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 01:25:10 +0000 (UTC)
Received: (qmail 17794 invoked by uid 500); 5 May 2011 01:25:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17742 invoked by uid 500); 5 May 2011 01:25:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17734 invoked by uid 99); 5 May 2011 01:25:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 01:25:08 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 01:25:02 +0000
Received: by ewy22 with SMTP id 22so760160ewy.35
        for <general@hadoop.apache.org>; Wed, 04 May 2011 18:24:42 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.14.50.196 with SMTP id z44mr824894eeb.227.1304558682195; Wed,
 04 May 2011 18:24:42 -0700 (PDT)
Received: by 10.14.37.13 with HTTP; Wed, 4 May 2011 18:24:42 -0700 (PDT)
In-Reply-To: <C0BE4B0B-D99C-4E9B-9748-AD0FFD5B5766@yahoo-inc.com>
References: <BANLkTik5gj0H4gmvy14Ux=g_r5Tt6D1=MA@mail.gmail.com>
	<C9E71BE8.19FCF%sureshms@yahoo-inc.com>
	<BANLkTikrS9PTeErT15fs2yv+N4_wpPN7xQ@mail.gmail.com>
	<BANLkTimjxRdf6PyTnbZUwrS_znZo00z-5w@mail.gmail.com>
	<BANLkTi=pNF_aQrWPurfXRbjNdshfseNobg@mail.gmail.com>
	<321761.51699.qm@web65903.mail.ac4.yahoo.com>
	<CB238EE8-1E06-4C6C-A7F5-CB00E2E073CB@yahoo-inc.com>
	<BANLkTi=aW6Toxtg7g=nFSV7h4gK68522wg@mail.gmail.com>
	<B746E44F-B6B3-40F2-BF75-C923E9B3962E@yahoo-inc.com>
	<BANLkTinvH8uUAzOQUogqnBanioPZ3UmnHg@mail.gmail.com>
	<BANLkTineZVMp1Fgv+JFT7NNihvDeHxdrvA@mail.gmail.com>
	<BANLkTimpuUhDRL1S4iYQKp3A-eKWcHi8kg@mail.gmail.com>
	<BANLkTik5YqkryXkWJOrPxon0NaLZtQoK1w@mail.gmail.com>
	<C0BE4B0B-D99C-4E9B-9748-AD0FFD5B5766@yahoo-inc.com>
Date: Wed, 4 May 2011 18:24:42 -0700
Message-ID: <BANLkTimr4SkiQYnZLaLOMFPnxrTHoTigEw@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Wed, May 4, 2011 at 6:18 PM, Eric Baldeschwieler
<eric14@yahoo-inc.com> wrote:
> Ok. I'll bite.
>
> The point of a vote is to learn what everyone thinks. So far we have learned:
>
> 1 - the team that is trying to contribute code and do a release thinks it is ready.
>
> 2 - Cloudera does not think the release is a good idea.
>

I don't think that's true.  There's a difference between not
supporting a given rc and not supporting a release from this branch in
general.

With both of my hats on, I want code to be reviewed before being
release, I want releases to not regress against previous releases, I
don't want the next major release to regress against a stable release,
I want the community to discuss new version schemes and development
models vs adopting them by accident just because we voted on a
particular release.

Thanks,
Eli

From general-return-3369-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 01:25:13 2011
Return-Path: <general-return-3369-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F305B6E3
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 01:25:12 +0000 (UTC)
Received: (qmail 19430 invoked by uid 500); 5 May 2011 01:25:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19371 invoked by uid 500); 5 May 2011 01:25:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19363 invoked by uid 99); 5 May 2011 01:25:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 01:25:11 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jdcryans@gmail.com designates 209.85.213.48 as permitted sender)
Received: from [209.85.213.48] (HELO mail-yw0-f48.google.com) (209.85.213.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 01:25:05 +0000
Received: by ywk9 with SMTP id 9so874663ywk.35
        for <general@hadoop.apache.org>; Wed, 04 May 2011 18:24:44 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        bh=tKbzi7J8uP1lNFdoqA4VlXFQDvYBIfIyU9LbcSSjDW0=;
        b=rrV1uzoVZt7pmuA7LLCuXw6UZIQlVVjSsp1nbjdrjgotptyoYCH/mGQeazPH9ClgXm
         U3ZCDxq4BFo5lez3T/9iikmxbfMyRnVE7XOz8gUY4dO3UNI2/HA2VZrA726vpRQwiNMd
         w3FAqxum2ArRBJHGygRlwgDubXIyr1MXJ4KSM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=BKr3nOvneEFOi4Inbk5pigFHnxa+5/f0YZZqWx1OoATPgkKGxTQ64Q+Mp1Swhk3xYu
         GizQs+ULu+nyrVarNXvFr1m9mfh6wqDLf7VpI7pD7r7kBshUmfqz9jYOXDSQTWZclxzV
         fDxjbirRtEhAtxFA7ljBsbdQxSFSIGnGwazEM=
MIME-Version: 1.0
Received: by 10.101.134.20 with SMTP id l20mr1195114ann.5.1304558684066; Wed,
 04 May 2011 18:24:44 -0700 (PDT)
Sender: jdcryans@gmail.com
Received: by 10.100.4.10 with HTTP; Wed, 4 May 2011 18:24:43 -0700 (PDT)
In-Reply-To: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org>
References: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org>
Date: Wed, 4 May 2011 18:24:43 -0700
X-Google-Sender-Auth: WhYIzzCVXgpIaYRljiyjcpVma7Q
Message-ID: <BANLkTikH4PazAADyo8+=3d3v0bUHaS24HQ@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Non-biding -1.

I did download it and checked it out, but when I look at the
documentation I see it says "Hadoop 0.20 documentation" in the tab on
top. From what I can tell this isn't the branch 0.20 so I think it's
an error and from a user point of view this looks more like something
I would call 0.22 (although yes I understand this is 0.20 +security
+whatever).

Why would a single company push so hard to go against the "normal"
release process just for "the benefit of putting our work in the hands
of all hadoop users" is beyond me. It's not like people were begging
on the mailing lists to be able to get their hands on such a release
to the point where an emergency point release including tons of new
features is needed.

So to me the more logical reason would be monetary gains, that I would
understand better from a for-profit company. But then why go through
the hurdles of having such an ASF release when Y! isn't even selling
anything remotely related to Hadoop services? And why now?

But then there's this spinoff thing and it suddenly makes a lot more sense.

E14 said earlier that "That is how apache works."

I would say yes, maybe this is how it works, but I'm not sure I want
to see it working like _that_. The ASF shouldn't be the vehicle for a
single (future) company's wishes.

J-D

On Wed, May 4, 2011 at 10:31 AM, Owen O'Malley <omalley@apache.org> wrote:
> Here's an updated release candidate for 0.20.203.0. I've incorporated the feedback and included all of the patches from 0.20.2, which is the last stable release. I also fixed the eclipse-plugin problem.
>
> The candidate is at: http://people.apache.org/~omalley/hadoop-0.20.203.0-rc1/
>
> Please download it, inspect it, compile it, and test it. Clearly, I'm +1.
>
> -- Owen

From general-return-3370-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 01:28:01 2011
Return-Path: <general-return-3370-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9397E77B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 01:28:01 +0000 (UTC)
Received: (qmail 23963 invoked by uid 500); 5 May 2011 01:28:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23919 invoked by uid 500); 5 May 2011 01:28:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23911 invoked by uid 99); 5 May 2011 01:28:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 01:28:00 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mbhandarkar@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 01:27:54 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:In-Reply-To:Accept-Language:
   Content-Language:X-MS-Has-Attach:X-MS-TNEF-Correlator:
   x-originating-ip:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version;
  b=XMXcZv9supBRbam/NxkN4fn28GxlKflXq2ys3TLHcjc7xzFT1fEyKlej
   uYRiWCGufsfPLcSiEdVQd9WU8Gepu3hRqRVhz61IjVXEKSf04wjNJLuO/
   Y0I31XgRvnXJIRz;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=mbhandarkar@linkedin.com; q=dns/txt;
  s=proddkim; t=1304558874; x=1336094874;
  h=from:to:subject:date:message-id:in-reply-to:content-id:
   content-transfer-encoding:mime-version;
  bh=2BC4xa/42uaMMEldCICrjkm6+x46fa6tQaa/xsLnQNY=;
  b=An6J9xPdOHVP1Rjpb1xYe2RGPj4040AP3kDXz2SI+eCTaZHiPhdG6pTY
   kyFzb/z84XY9/ksvXnHqNrgQgYVrrQlqajo+ssAOrgL2JmdRu86faUteW
   7uMrntJypNMGG9b;
X-IronPort-AV: E=Sophos;i="4.64,317,1301900400"; 
   d="scan'208";a="22862744"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi id 14.01.0251.000; Wed, 4
 May 2011 18:30:57 -0700
From: Milind Bhandarkar <mbhandarkar@linkedin.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
Thread-Topic: [VOTE] Release candidate 0.20.203.0-rc1
Thread-Index: AcwKgWNdu2v/kISbSx+vh7BYIk9JfwAP8/KkAACZ7gA=
Date: Thu, 5 May 2011 01:30:57 +0000
Message-ID: <C9E749C9.393E%mbhandarkar@linkedin.com>
In-Reply-To: <C9E7470B.34E6C%ddas@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <D988B1A9FB4D5F48B7165FCD0A41B0A1@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

My (non-binding) vote for 0.20.203.0-rc1 is +1.

I downloaded, compiled, ran tests, ran my bigrams example, all ran
perfectly.
(I did a single node test without security on.)

The voting criteria I used are:

1. Is this a working release? : Yes
2. Does it take the codebase forward? : Yes
3. Does it have features that the user community might find valuable? : Yes

- milind

--=20
Milind Bhandarkar
mbhandarkar@linkedin.com
+1-650-776-3167






On 5/4/11 6:10 PM, "Devaraj Das" <ddas@yahoo-inc.com> wrote:

>+1 based on some single node tests I did (with security ON).
>
>
>On 5/4/11 10:31 AM, "Owen O'Malley" <omalley@apache.org> wrote:
>
>Here's an updated release candidate for 0.20.203.0. I've incorporated the
>feedback and included all of the patches from 0.20.2, which is the last
>stable release. I also fixed the eclipse-plugin problem.
>
>The candidate is at:
>http://people.apache.org/~omalley/hadoop-0.20.203.0-rc1/
>
>Please download it, inspect it, compile it, and test it. Clearly, I'm +1.
>
>-- Owen
>


From general-return-3371-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 01:57:16 2011
Return-Path: <general-return-3371-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E44CF3522
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 01:57:15 +0000 (UTC)
Received: (qmail 48793 invoked by uid 500); 5 May 2011 01:57:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48731 invoked by uid 500); 5 May 2011 01:57:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48723 invoked by uid 99); 5 May 2011 01:57:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 01:57:14 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [208.97.132.119] (HELO homiemail-a70.g.dreamhost.com) (208.97.132.119)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 01:57:06 +0000
Received: from homiemail-a70.g.dreamhost.com (localhost [127.0.0.1])
	by homiemail-a70.g.dreamhost.com (Postfix) with ESMTP id 5BEDF768061
	for <general@hadoop.apache.org>; Wed,  4 May 2011 18:56:45 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; q=dns; s=
	gbiv.com; b=o8K7ro00lw7lm14JUSdbP+RBQ6vxxpFgzaYdbIwZqVq6y1vS2KUA
	9327QkUQeYydfZLvRCZtr3mg3iSZNP117APxZcKh9uUoak9Vh2OgAD9vM8Ejw3Dd
	L8Rw06iUayYScVwB/FnsTITXvLZ0it5F8hgqDIqj/wuE2R9F9cDJKpI=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; s=gbiv.com;
	 bh=VoGFmmPH0I6KQnbDbYMrb5glrgw=; b=BjiNW3u3aWHiWJKLNEZImTlO+MGl
	8sfKwSpyynb2u05ko2PVKaFaFOHA3ZpdfO7xzQTYm/zhChyx3gFzbjKpOBXszaGa
	0HGSe2e3d6q0P8EX3oqtZm6GI8cwIKoEVywYJimjJ2LPmU5xa7kwhY7cWKfcR6ws
	dV5iDiAVEtTJCH4=
Received: from [10.134.89.86] (unknown [75.103.10.98])
	(using TLSv1 with cipher AES128-SHA (128/128 bits))
	(No client certificate requested)
	(Authenticated sender: fielding@gbiv.com)
	by homiemail-a70.g.dreamhost.com (Postfix) with ESMTPSA id 4252676805C
	for <general@hadoop.apache.org>; Wed,  4 May 2011 18:56:45 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
From: "Roy T. Fielding" <fielding@gbiv.com>
In-Reply-To: <BANLkTimpuUhDRL1S4iYQKp3A-eKWcHi8kg@mail.gmail.com>
Date: Wed, 4 May 2011 18:56:44 -0700
Content-Transfer-Encoding: 7bit
Message-Id: <79A1AED5-9005-4F58-BFDB-F7394CB2A69C@gbiv.com>
References: <BANLkTik5gj0H4gmvy14Ux=g_r5Tt6D1=MA@mail.gmail.com> <C9E71BE8.19FCF%sureshms@yahoo-inc.com> <BANLkTikrS9PTeErT15fs2yv+N4_wpPN7xQ@mail.gmail.com> <BANLkTimjxRdf6PyTnbZUwrS_znZo00z-5w@mail.gmail.com> <BANLkTi=pNF_aQrWPurfXRbjNdshfseNobg@mail.gmail.com> <321761.51699.qm@web65903.mail.ac4.yahoo.com> <CB238EE8-1E06-4C6C-A7F5-CB00E2E073CB@yahoo-inc.com> <BANLkTi=aW6Toxtg7g=nFSV7h4gK68522wg@mail.gmail.com> <B746E44F-B6B3-40F2-BF75-C923E9B3962E@yahoo-inc.com> <BANLkTinvH8uUAzOQUogqnBanioPZ3UmnHg@mail.gmail.com> <BANLkTineZVMp1Fgv+JFT7NNihvDeHxdrvA@mail.gmail.com> <BANLkTimpuUhDRL1S4iYQKp3A-eKWcHi8kg@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

On May 4, 2011, at 5:39 PM, Eli Collins wrote:

> The point is that these discussion should be sorted out, ie you don't
> change your development and release model on a release VOTE thread,
> you change it on a DISCUSSION thread.

That is no different than saying you have a right to veto a
release until the issue is addressed, which you don't have.

A release vote is a majority decision.  If the majority
decides to release, then whatever gets released will define
the new norm by which policies are assumed.  If not released,
then I suggest collaborating more on the policies before
trying to vote again.

Either way, we don't hold up a vote for the sake of a
policy discussion because voting is a more efficient
means of discovering if the policy really matters.

....Roy


From general-return-3372-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 02:13:10 2011
Return-Path: <general-return-3372-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 13F8B253F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 02:13:10 +0000 (UTC)
Received: (qmail 58888 invoked by uid 500); 5 May 2011 02:13:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58843 invoked by uid 500); 5 May 2011 02:13:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58835 invoked by uid 99); 5 May 2011 02:13:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 02:13:08 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [98.139.52.238] (HELO nm16-vm0.bullet.mail.ac4.yahoo.com) (98.139.52.238)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 05 May 2011 02:12:59 +0000
Received: from [98.139.52.193] by nm16.bullet.mail.ac4.yahoo.com with NNFMP; 05 May 2011 02:12:38 -0000
Received: from [98.139.52.177] by tm6.bullet.mail.ac4.yahoo.com with NNFMP; 05 May 2011 02:12:38 -0000
Received: from [127.0.0.1] by omp1060.mail.ac4.yahoo.com with NNFMP; 05 May 2011 02:12:38 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 692724.18193.bm@omp1060.mail.ac4.yahoo.com
Received: (qmail 93706 invoked by uid 60001); 5 May 2011 02:12:38 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1304561558; bh=B/xWcNe8oOLCoKwvxtlXIsnhWMN5+HbqOw1X7F6YaAM=; h=Message-ID:X-YMail-OSG:Received:X-RocketYMMF:X-Mailer:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=fXvso+6pFXb9YVb2EWFSzcI2JMgVeAxppOA+4+hhXHobjnyGn819kC/hSvPvxGEzzxV6fgMefb2C7uGs/XV7apyxr8L07+Cfa3vHJyQVK17pnNwdXGMocE282H9Pnjed+uIcK7xypLM7LCTXhBnhJny2Yu+nT1EpVnDpnFlkiEg=
Message-ID: <551643.49517.qm@web65511.mail.ac4.yahoo.com>
X-YMail-OSG: etasIvUVM1mqxe5pvAz97H8QCFJxZHwrTWmqese5Q0kP1yJ
 DeyxlqciFgRIljvrmi0KFd5ZMuO9NcKqXHIrwjK1PKKdWvsUK6qZHudKAYOM
 yvedJXxrYi5jTGtzj5UqRvnsYDSE8UHp8Q0pvIbwKSrhhmsdiGv_dC9f1QkY
 sjfV6g9pmXdcvvqyLh9GXZ6v_T7dt_IsP.lc8BnAY3tmeFn.pM8X6XpEbIBM
 FojMm2dC36mUnYzF.E9wFEfr5hpb8PoXdtJAyeEQEHfnuxgN3S0rD4Cs5jHN
 .g1ofhGAvSjKH5nwgTUqP2gYVf6fTkO_Ed0aOmWJ0bEFCvFgjCcNJGKplknJ
 mcGs0k7LHTBCDzj7WJYykW7bPhHyMP6EHxJvs498Uu11p0VTcvMPMNuBvcKL
 GlKygE.xalvFMBA--
Received: from [69.231.27.174] by web65511.mail.ac4.yahoo.com via HTTP; Wed, 04 May 2011 19:12:38 PDT
X-RocketYMMF: apurtell
X-Mailer: YahooMailClassic/12.0.2 YahooMailWebService/0.8.110.299900
Date: Wed, 4 May 2011 19:12:38 -0700 (PDT)
From: Andrew Purtell <apurtell@apache.org>
Reply-To: apurtell@apache.org
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Speculation either on the motives of those objecting to a release or of tho=
se making contributions or proposing a release does not advance progress. T=
he accusations and counter-accusations seen on this thread are regrettable =
and I feel less and less confident in the future of Apache Hadoop as time g=
oes on. As a strong believer in and advocate of open source as an answer to=
 technical and architectural challenges, I am pained to see the members of =
what should be a vibrant community litigating in an ultimately self-defeati=
ng way. If only this energy put into argument could be channeled into code =
or patches...=0A=0AIn open source, if opinions were code we would rule the =
world.=0A=0ASo what of this candidate?=0A=0AArtifact looks good, DFS tests =
are good, MR tests are good. Looked over some of the documentation and foun=
d no errors. To my knowledge this is now a superset of branch-0.20, address=
ing the reasonably determined deficit of rc0.=0A=0AThere seems no reason ot=
her issues cannot be addressed subsequently.=0A=0AThere has not been a rele=
ase of Apache Hadoop 0.20 since at least Feb 6 2010 yet since this time imp=
ortant security enhancements have been contributed, but in the form of an A=
pache product these are only available as patches on a non-release branch. =
Forward progress of the Apache product seems more important than achieving =
the perfect release in all eyes.=0A=0AFor example, append features remain o=
n a non-release branch. I would really have liked to see the append changes=
 included in this candidate, but this is not grounds for objection merely r=
egret, and I hope this can be covered by a subsequent release, perhaps soon=
.=0A=0AAfter security and append features are in 0.20, in my personal humbl=
e opinion the 0.20 release in total is sufficient and all attention should =
be paid to the next release (0.22 or whatever), except for critical bug fix=
es.=0A=0A+1=0A=0ABest regards,=0A=0A=A0 =A0 - Andy=0A=0AProblems worthy of =
attack prove their worth by hitting back. - Piet Hein (via Tom White)=0A

From general-return-3373-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 02:20:58 2011
Return-Path: <general-return-3373-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3B9452BA5
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 02:20:58 +0000 (UTC)
Received: (qmail 72522 invoked by uid 500); 5 May 2011 02:20:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72472 invoked by uid 500); 5 May 2011 02:20:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72464 invoked by uid 99); 5 May 2011 02:20:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 02:20:56 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 02:20:52 +0000
Received: by fxm7 with SMTP id 7so2260227fxm.35
        for <general@hadoop.apache.org>; Wed, 04 May 2011 19:20:30 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=JZSSNdZCGYVUv848RwUXDzue2A+y9jBRpZITZ+WaNXU=;
        b=R6lmth5+hJ/3oWIWtWcfaRqnYrc4PGFp4+Kmi1s8xn+03S1DciO0zJhTq19tAlDObN
         P1i1OFLWZO5amroqq2GfATNVyPiDNNP/DCGiBJsTI+J0DY4l/Ot8JRS0+Zpqu/RSIy5s
         1qXIXn4WLHcROPk5lkqGkXhElsoh0OCySnJ7k=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Z05ceWzwO5vx0VsfpUD368L7xZIAlQsiHwTeMOjJ1in0DFu4x5m/SlKwqClVwokzBO
         h1TN3hm18CocxW/3uH2bUH3Ehng58NHo4dZE8zkuYrS3XV/WNw/zLYuVItQ9C14uZid2
         O0LLePofkvq7GCGWGG8uXZMIkBPMaiEIH42lo=
MIME-Version: 1.0
Received: by 10.223.59.92 with SMTP id k28mr2108124fah.27.1304562030754; Wed,
 04 May 2011 19:20:30 -0700 (PDT)
Received: by 10.223.119.131 with HTTP; Wed, 4 May 2011 19:20:30 -0700 (PDT)
In-Reply-To: <79A1AED5-9005-4F58-BFDB-F7394CB2A69C@gbiv.com>
References: <BANLkTik5gj0H4gmvy14Ux=g_r5Tt6D1=MA@mail.gmail.com>
	<C9E71BE8.19FCF%sureshms@yahoo-inc.com>
	<BANLkTikrS9PTeErT15fs2yv+N4_wpPN7xQ@mail.gmail.com>
	<BANLkTimjxRdf6PyTnbZUwrS_znZo00z-5w@mail.gmail.com>
	<BANLkTi=pNF_aQrWPurfXRbjNdshfseNobg@mail.gmail.com>
	<321761.51699.qm@web65903.mail.ac4.yahoo.com>
	<CB238EE8-1E06-4C6C-A7F5-CB00E2E073CB@yahoo-inc.com>
	<BANLkTi=aW6Toxtg7g=nFSV7h4gK68522wg@mail.gmail.com>
	<B746E44F-B6B3-40F2-BF75-C923E9B3962E@yahoo-inc.com>
	<BANLkTinvH8uUAzOQUogqnBanioPZ3UmnHg@mail.gmail.com>
	<BANLkTineZVMp1Fgv+JFT7NNihvDeHxdrvA@mail.gmail.com>
	<BANLkTimpuUhDRL1S4iYQKp3A-eKWcHi8kg@mail.gmail.com>
	<79A1AED5-9005-4F58-BFDB-F7394CB2A69C@gbiv.com>
Date: Wed, 4 May 2011 19:20:30 -0700
Message-ID: <BANLkTimETA6KDp_eJGH1--C0E6SLbx2U4w@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00151744874e6e3dc704a27e0470

--00151744874e6e3dc704a27e0470
Content-Type: text/plain; charset=ISO-8859-1

+1.

I downloaded the bits, compiled and ran unit tests. Also, looked at the
source code to some extent. Looks good.

-dhruba

On Wed, May 4, 2011 at 6:56 PM, Roy T. Fielding <fielding@gbiv.com> wrote:

> On May 4, 2011, at 5:39 PM, Eli Collins wrote:
>
> > The point is that these discussion should be sorted out, ie you don't
> > change your development and release model on a release VOTE thread,
> > you change it on a DISCUSSION thread.
>
> That is no different than saying you have a right to veto a
> release until the issue is addressed, which you don't have.
>
> A release vote is a majority decision.  If the majority
> decides to release, then whatever gets released will define
> the new norm by which policies are assumed.  If not released,
> then I suggest collaborating more on the policies before
> trying to vote again.
>
> Either way, we don't hold up a vote for the sake of a
> policy discussion because voting is a more efficient
> means of discovering if the policy really matters.
>
> ....Roy
>
>


-- 
Connect to me at http://www.facebook.com/dhruba

--00151744874e6e3dc704a27e0470--

From general-return-3374-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 02:22:50 2011
Return-Path: <general-return-3374-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 087EC2BEB
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 02:22:50 +0000 (UTC)
Received: (qmail 76868 invoked by uid 500); 5 May 2011 02:22:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76811 invoked by uid 500); 5 May 2011 02:22:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76803 invoked by uid 99); 5 May 2011 02:22:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 02:22:48 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [208.97.132.66] (HELO homiemail-a64.g.dreamhost.com) (208.97.132.66)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 02:22:40 +0000
Received: from homiemail-a64.g.dreamhost.com (localhost [127.0.0.1])
	by homiemail-a64.g.dreamhost.com (Postfix) with ESMTP id 95BC343807F
	for <general@hadoop.apache.org>; Wed,  4 May 2011 19:22:19 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; q=dns; s=
	gbiv.com; b=xqnKz9MggVzrAT50/Lc4SB/5hWCqJJd7chzRU8inLlPgj3LB7ksV
	3sRubRUCfp0KM+p645Pk7aT0jfcEW6AUqoJBCl/LBjG4Ifd2JX94OdmM3lOcT3nA
	Qzcl+7a4IonS8yGyxc82chDA+f/uFr5fCUqAlLGR15AGLXL2rJ6uE2c=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; s=gbiv.com;
	 bh=eeSYhdeRccOmwvKCctpDMdlTHuY=; b=Ekb4AwMdmBwcctXIW0ve10OSx8Vo
	pmB7RyxegIX2UIUip+7B69diVDBwjrPtv0mrr9XYezGLE5x++SAHpg5NA0l4JBUN
	Orfu5AyKPD6e2jiCqPTzd2j+MlOx1vkTi+EyMVrDESom3pg+Ju41FH91xv6Mqrm+
	33tCXYNLMuTjHIU=
Received: from [10.134.89.86] (unknown [75.103.10.98])
	(using TLSv1 with cipher AES128-SHA (128/128 bits))
	(No client certificate requested)
	(Authenticated sender: fielding@gbiv.com)
	by homiemail-a64.g.dreamhost.com (Postfix) with ESMTPSA id 5B9E443807E
	for <general@hadoop.apache.org>; Wed,  4 May 2011 19:22:19 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
From: "Roy T. Fielding" <fielding@gbiv.com>
In-Reply-To: <BANLkTikH4PazAADyo8+=3d3v0bUHaS24HQ@mail.gmail.com>
Date: Wed, 4 May 2011 19:22:18 -0700
Content-Transfer-Encoding: 7bit
Message-Id: <3116A002-6EA6-4971-B7AD-261BCC53D76B@gbiv.com>
References: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org> <BANLkTikH4PazAADyo8+=3d3v0bUHaS24HQ@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

On May 4, 2011, at 6:24 PM, Jean-Daniel Cryans wrote:

> Non-biding -1.
> 
> I did download it and checked it out, but when I look at the
> documentation I see it says "Hadoop 0.20 documentation" in the tab on
> top. From what I can tell this isn't the branch 0.20 so I think it's
> an error and from a user point of view this looks more like something
> I would call 0.22 (although yes I understand this is 0.20 +security
> +whatever).
> 
> Why would a single company push so hard to go against the "normal"
> release process just for "the benefit of putting our work in the hands
> of all hadoop users" is beyond me. It's not like people were begging
> on the mailing lists to be able to get their hands on such a release
> to the point where an emergency point release including tons of new
> features is needed.
> 
> So to me the more logical reason would be monetary gains, that I would
> understand better from a for-profit company. But then why go through
> the hurdles of having such an ASF release when Y! isn't even selling
> anything remotely related to Hadoop services? And why now?
> 
> But then there's this spinoff thing and it suddenly makes a lot more sense.
> 
> E14 said earlier that "That is how apache works."
> 
> I would say yes, maybe this is how it works, but I'm not sure I want
> to see it working like _that_. The ASF shouldn't be the vehicle for a
> single (future) company's wishes.

The ASF is a vehicle for whomever wishes to collaborate on a
given project.  Collaboration means helping do the work.  Those
who do the work may do so for whatever reasons that they think
are good, whether it is because they feel like being charitable
today, they get paid a salary and the big boss said "work on
this part", or because they just have an itch worth scratching.

Apache does not care why people choose to collaborate or
how they choose to apply their own intellectual efforts.  We
welcome all forms of contribution under the terms of our license.

What we do require is a certain amount of civility regarding
our voting procedures and an emphasis on individual responsibility
for your votes.  Anyone caught *voting* a particular way just
because the boss says so will be dealt with severely.  Votes
are how we do quality control and make decisions, and no other
company can be allowed to make decisions for our non-profit.

....Roy

From general-return-3375-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 02:40:31 2011
Return-Path: <general-return-3375-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 148483E5
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 02:40:31 +0000 (UTC)
Received: (qmail 91479 invoked by uid 500); 5 May 2011 02:40:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91375 invoked by uid 500); 5 May 2011 02:40:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91367 invoked by uid 99); 5 May 2011 02:40:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 02:40:28 +0000
X-ASF-Spam-Status: No, hits=3.3 required=5.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 02:40:19 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p452dqes003793
	for <general@hadoop.apache.org>; Wed, 4 May 2011 19:39:52 -0700 (PDT)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by
 SP2-EX07CAS01.ds.corp.yahoo.com ([98.137.59.37]) with mapi; Wed, 4 May 2011
 19:39:52 -0700
From: Eric Yang <eyang@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 4 May 2011 19:39:51 -0700
Subject: [DISCUSSION] development process of Hadoop
Thread-Topic: [DISCUSSION] development process of Hadoop
Thread-Index: AcwKw2hV3hkBHc1BTRuS8ErMGgtbtwACkzGG
Message-ID: <C9E75C07.11858%eyang@yahoo-inc.com>
In-Reply-To: <BANLkTimr4SkiQYnZLaLOMFPnxrTHoTigEw@mail.gmail.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C9E75C0711858eyangyahooinccom_"
MIME-Version: 1.0

--_000_C9E75C0711858eyangyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

If we reflect back and see how the development community end up in its curr=
ent state for Hadoop.  There are development rapidly happening and tested i=
n all kind of organizations.  However, Hadoop committers are only committin=
g code that are interested by the sponsored companies.  People are coding d=
efensively to ensuring only self serving patches would be committed, and he=
lping others and merging problem are always prioritized secondary.  While t=
he world demand agility, the "review then commit" process is preventing pro=
gress from happening.  Committers are afraid to commit patches because revi=
ew hasn't took place.  By the time patch is reviewed, it does not apply pro=
perly.  People end up having to generate multiple version of patches to ens=
ure the code can be applied.  The large lag time between patch generation a=
nd reviewed is taking significant toll on the community and progress.

Yahoo have a great team of developers who improves Hadoop at faster pace wi=
th its own fork of the source code.  The reason that Yahoo was able to achi=
eve faster improvement with features was due to the ability to use source c=
ode repository tools properly.  Unfortunate for Yahoo, their source code re=
pository was not Apache svn trunk.  I applause Owen and Arun's effort for m=
en powering and backward/forward porting the changes between yahoo github a=
nd Apache svn.  There might be some jiras that needs to be merged into Hado=
op 0.20.203 branch to ensure the linage is correct.  The community should o=
ffer to help with detail listing of what is missing rather than vote -1 wit=
hout concise reasoning of what is missing.

JIRA is meant as a discussion and collaboration tool, but hadoop community =
intends to use it as the source code version control system with men powere=
d diff maker.  While spending time in the incubator with other project, the=
 mentors have explained that it is not ASF's philosophy to use "review then=
 commit".  Hadoop community should rethink if the community is using the ri=
ght tools for the right task.

Use JIRA, if there is large feature set that requires brain storming, and d=
evelopers should have the ability to make small incremental changes without=
 RTC.  This will ensure developers help each other rather than policing eac=
h other.

Any thoughts?

Regards,
Eric

--_000_C9E75C0711858eyangyahooinccom_--

From general-return-3376-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 03:34:52 2011
Return-Path: <general-return-3376-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9A3792357
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 03:34:52 +0000 (UTC)
Received: (qmail 14587 invoked by uid 500); 5 May 2011 03:34:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14495 invoked by uid 500); 5 May 2011 03:34:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14344 invoked by uid 99); 5 May 2011 03:34:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 03:34:47 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 03:34:40 +0000
Received: by qyk30 with SMTP id 30so1676867qyk.14
        for <general@hadoop.apache.org>; Wed, 04 May 2011 20:34:18 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=SoCYPDsOAk8s02GiBnHaqmv/J1g6LIOkXppI39Httuc=;
        b=hqRe7ZVMTtUVVQOXupev5JuvU6DQNXV+LU3et91A/xUgpYnpaZgxxecAl/852jAgHd
         Rjzhnn8M6c4sJMRXKS68jnEOQGBIid/jCOP869YwTWRTVPga8KVd8XnEUPHQHuSSSSVb
         U5KGsAVEwsnkgc8KQMXCSIl4f2fGX+LVdO+BY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=dSO41OEx7U46e2swZL1PlT8fwb4BFP6EdoZGF61btcqUJzQomDDVBl57qhn1XWF3sD
         VuiGDHwDXSJd7LhwbtyL/p60ZbOGxSXbq7lMNttNTbZrwXnwyYSEwvb0f9eK/WHxU7cD
         U2emLNWyPozonXlhyA9pKhWtiv6EGcdcltr9Q=
Received: by 10.229.64.233 with SMTP id f41mr1481007qci.130.1304566458600;
        Wed, 04 May 2011 20:34:18 -0700 (PDT)
Received: from [10.172.0.126] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id t17sm1448357qcs.35.2011.05.04.20.34.16
        (version=TLSv1/SSLv3 cipher=OTHER);
        Wed, 04 May 2011 20:34:17 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <3116A002-6EA6-4971-B7AD-261BCC53D76B@gbiv.com>
Date: Thu, 5 May 2011 13:34:11 +1000
Content-Transfer-Encoding: quoted-printable
Message-Id: <9234135B-FD91-4E92-B4F5-6E6F7EB38A94@Holsman.NET>
References: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org> <BANLkTikH4PazAADyo8+=3d3v0bUHaS24HQ@mail.gmail.com> <3116A002-6EA6-4971-B7AD-261BCC53D76B@gbiv.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

just as a Tally
we have
6+1's (andy.. is yours binding?? if so 7)
and 3 -1's.

so according to the votes so far we are releasing.. but according to our =
bylaws.. we need to wait 7 days for everyone to chime in.

--I
On May 5, 2011, at 12:22 PM, Roy T. Fielding wrote:

> On May 4, 2011, at 6:24 PM, Jean-Daniel Cryans wrote:
>=20
>> Non-biding -1.
>>=20
>> I did download it and checked it out, but when I look at the
>> documentation I see it says "Hadoop 0.20 documentation" in the tab on
>> top. =46rom what I can tell this isn't the branch 0.20 so I think =
it's
>> an error and from a user point of view this looks more like something
>> I would call 0.22 (although yes I understand this is 0.20 +security
>> +whatever).
>>=20
>> Why would a single company push so hard to go against the "normal"
>> release process just for "the benefit of putting our work in the =
hands
>> of all hadoop users" is beyond me. It's not like people were begging
>> on the mailing lists to be able to get their hands on such a release
>> to the point where an emergency point release including tons of new
>> features is needed.
>>=20
>> So to me the more logical reason would be monetary gains, that I =
would
>> understand better from a for-profit company. But then why go through
>> the hurdles of having such an ASF release when Y! isn't even selling
>> anything remotely related to Hadoop services? And why now?
>>=20
>> But then there's this spinoff thing and it suddenly makes a lot more =
sense.
>>=20
>> E14 said earlier that "That is how apache works."
>>=20
>> I would say yes, maybe this is how it works, but I'm not sure I want
>> to see it working like _that_. The ASF shouldn't be the vehicle for a
>> single (future) company's wishes.
>=20
> The ASF is a vehicle for whomever wishes to collaborate on a
> given project.  Collaboration means helping do the work.  Those
> who do the work may do so for whatever reasons that they think
> are good, whether it is because they feel like being charitable
> today, they get paid a salary and the big boss said "work on
> this part", or because they just have an itch worth scratching.
>=20
> Apache does not care why people choose to collaborate or
> how they choose to apply their own intellectual efforts.  We
> welcome all forms of contribution under the terms of our license.
>=20
> What we do require is a certain amount of civility regarding
> our voting procedures and an emphasis on individual responsibility
> for your votes.  Anyone caught *voting* a particular way just
> because the boss says so will be dealt with severely.  Votes
> are how we do quality control and make decisions, and no other
> company can be allowed to make decisions for our non-profit.
>=20
> ....Roy


From general-return-3377-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 04:17:37 2011
Return-Path: <general-return-3377-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B4CDE2CD1
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 04:17:37 +0000 (UTC)
Received: (qmail 38649 invoked by uid 500); 5 May 2011 04:17:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38612 invoked by uid 500); 5 May 2011 04:17:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 38604 invoked by uid 99); 5 May 2011 04:17:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 04:17:34 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.105 as permitted sender)
Received: from [17.148.16.105] (HELO asmtpout030.mac.com) (17.148.16.105)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 04:17:27 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.8] ([71.198.193.36])
 by asmtp030.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LKP002DPFWGSK70@asmtp030.mac.com>
 for general@hadoop.apache.org; Wed, 04 May 2011 21:17:06 -0700 (PDT)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-05-04_09:2011-05-04,2011-05-04,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1105040211
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <79A1AED5-9005-4F58-BFDB-F7394CB2A69C@gbiv.com>
Date: Wed, 04 May 2011 21:17:04 -0700
Message-id: <89EC003F-E125-44C5-932C-2157A307CFD6@mac.com>
References: <BANLkTik5gj0H4gmvy14Ux=g_r5Tt6D1=MA@mail.gmail.com>
 <C9E71BE8.19FCF%sureshms@yahoo-inc.com>
 <BANLkTikrS9PTeErT15fs2yv+N4_wpPN7xQ@mail.gmail.com>
 <BANLkTimjxRdf6PyTnbZUwrS_znZo00z-5w@mail.gmail.com>
 <BANLkTi=pNF_aQrWPurfXRbjNdshfseNobg@mail.gmail.com>
 <321761.51699.qm@web65903.mail.ac4.yahoo.com>
 <CB238EE8-1E06-4C6C-A7F5-CB00E2E073CB@yahoo-inc.com>
 <BANLkTi=aW6Toxtg7g=nFSV7h4gK68522wg@mail.gmail.com>
 <B746E44F-B6B3-40F2-BF75-C923E9B3962E@yahoo-inc.com>
 <BANLkTinvH8uUAzOQUogqnBanioPZ3UmnHg@mail.gmail.com>
 <BANLkTineZVMp1Fgv+JFT7NNihvDeHxdrvA@mail.gmail.com>
 <BANLkTimpuUhDRL1S4iYQKp3A-eKWcHi8kg@mail.gmail.com>
 <79A1AED5-9005-4F58-BFDB-F7394CB2A69C@gbiv.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)

I'm really not sure yet how to vote here.  I was going to vote +1 for what I was told by a number of Yahoo! committers would be a one time release as Yahoo! "comes back to Apache" after a hiatus last fall/winter and ended their own distribution.  Clearly this code was not all developed as a community process, but I was going to support a one time release of what they had developed in exclusion.

Then I read Roy's email, which confused me.  We would he or I or anyone else support this release setting precedent or policy since it would walk all over our bylaws, community process, and the consensus nature of our foundation?  This release vote is a lazy majority of the PMC, but other decisions rolled up in this are supposed to be lazy majority of active committers or, in the case of code changes, a lazy consensus.  Setting policy by this release means any sufficiently large group of committers could go off and develop on their own and then commit it to a branch and call a release.

Furthermore, it now sounds like this is possibly the first in a line of feature releases off this branch.  Bug fixes releases, sure.  But feature releases?  What's wrong with trunk?

Nige

On May 4, 2011, at 6:56 PM, Roy T. Fielding wrote:

> On May 4, 2011, at 5:39 PM, Eli Collins wrote:
> 
>> The point is that these discussion should be sorted out, ie you don't
>> change your development and release model on a release VOTE thread,
>> you change it on a DISCUSSION thread.
> 
> That is no different than saying you have a right to veto a
> release until the issue is addressed, which you don't have.
> 
> A release vote is a majority decision.  If the majority
> decides to release, then whatever gets released will define
> the new norm by which policies are assumed.  If not released,
> then I suggest collaborating more on the policies before
> trying to vote again.
> 
> Either way, we don't hold up a vote for the sake of a
> policy discussion because voting is a more efficient
> means of discovering if the policy really matters.
> 
> ....Roy
> 


From general-return-3378-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 04:55:18 2011
Return-Path: <general-return-3378-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4322524E9
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 04:55:18 +0000 (UTC)
Received: (qmail 53865 invoked by uid 500); 5 May 2011 04:55:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53712 invoked by uid 500); 5 May 2011 04:55:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53704 invoked by uid 99); 5 May 2011 04:55:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 04:55:16 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of hammer@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 04:55:09 +0000
Received: by bwz8 with SMTP id 8so2649312bwz.35
        for <general@hadoop.apache.org>; Wed, 04 May 2011 21:54:47 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.19.10 with SMTP id y10mr1827877bka.190.1304571287433; Wed,
 04 May 2011 21:54:47 -0700 (PDT)
Received: by 10.204.37.140 with HTTP; Wed, 4 May 2011 21:54:47 -0700 (PDT)
In-Reply-To: <89EC003F-E125-44C5-932C-2157A307CFD6@mac.com>
References: <BANLkTik5gj0H4gmvy14Ux=g_r5Tt6D1=MA@mail.gmail.com>
	<C9E71BE8.19FCF%sureshms@yahoo-inc.com>
	<BANLkTikrS9PTeErT15fs2yv+N4_wpPN7xQ@mail.gmail.com>
	<BANLkTimjxRdf6PyTnbZUwrS_znZo00z-5w@mail.gmail.com>
	<BANLkTi=pNF_aQrWPurfXRbjNdshfseNobg@mail.gmail.com>
	<321761.51699.qm@web65903.mail.ac4.yahoo.com>
	<CB238EE8-1E06-4C6C-A7F5-CB00E2E073CB@yahoo-inc.com>
	<BANLkTi=aW6Toxtg7g=nFSV7h4gK68522wg@mail.gmail.com>
	<B746E44F-B6B3-40F2-BF75-C923E9B3962E@yahoo-inc.com>
	<BANLkTinvH8uUAzOQUogqnBanioPZ3UmnHg@mail.gmail.com>
	<BANLkTineZVMp1Fgv+JFT7NNihvDeHxdrvA@mail.gmail.com>
	<BANLkTimpuUhDRL1S4iYQKp3A-eKWcHi8kg@mail.gmail.com>
	<79A1AED5-9005-4F58-BFDB-F7394CB2A69C@gbiv.com>
	<89EC003F-E125-44C5-932C-2157A307CFD6@mac.com>
Date: Wed, 4 May 2011 21:54:47 -0700
Message-ID: <BANLkTinsQWnrZBUA0=--5Z4WJ7Y1jTRFcQ@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0003255574062bf1c704a2802c0a

--0003255574062bf1c704a2802c0a
Content-Type: text/plain; charset=ISO-8859-1

-1.

As Roy says, "whatever gets released will define the new norm by which
policies are assumed", and I certainly don't want this project to change its
norms to accommodate bad practices. In particular, Eli presented three very
reasonable technical objections to this release. To summarize:

1) Let's get the JIRAs that are going into this release into trunk first.
2) Let's create a JIRA for each issue in the release.
3) Let's stick to the release numbering conventions established for this
project.

I know the folks at Yahoo! are all professional engineers and done
tremendous work to help get the project to this point. There's no doubt in
my mind they understand the validity of the above three technical
objections. In fact, many of them helped author our "How to Contribute"
page, which established these conventions:
wiki.apache.org/hadoop/HowToContribute. We develop new features against
trunk, we create JIRAs for each issue, we review code before it goes into
trunk, and we only update old releases with bug fixes.

I couldn't be more excited to have Yahoo! once again doing development in
Apache, and I hope that we can work together to get the work that you've
done in this branch into one of our upcoming feature releases.

I hope those who voted +1 before Roy clarified what a release vote will mean
for future project norms will reconsider their votes.

While there may be many competing agendas in this community, we all wish to
see Apache Hadoop releases of the highest quality. Changing our norms to
allow huge, unreviewed patch sets introducing new features into a past
release is a step in the wrong direction.

With a little bit of elbow grease, we can get the work done in this branch
into trunk, get 0.22 out the door, and be ready for a great 0.23 release.

Later,
Jeff

On Wed, May 4, 2011 at 9:17 PM, Nigel Daley <ndaley@mac.com> wrote:

> I'm really not sure yet how to vote here.  I was going to vote +1 for what
> I was told by a number of Yahoo! committers would be a one time release as
> Yahoo! "comes back to Apache" after a hiatus last fall/winter and ended
> their own distribution.  Clearly this code was not all developed as a
> community process, but I was going to support a one time release of what
> they had developed in exclusion.
>
> Then I read Roy's email, which confused me.  We would he or I or anyone
> else support this release setting precedent or policy since it would walk
> all over our bylaws, community process, and the consensus nature of our
> foundation?  This release vote is a lazy majority of the PMC, but other
> decisions rolled up in this are supposed to be lazy majority of active
> committers or, in the case of code changes, a lazy consensus.  Setting
> policy by this release means any sufficiently large group of committers
> could go off and develop on their own and then commit it to a branch and
> call a release.
>
> Furthermore, it now sounds like this is possibly the first in a line of
> feature releases off this branch.  Bug fixes releases, sure.  But feature
> releases?  What's wrong with trunk?
>
> Nige
>
> On May 4, 2011, at 6:56 PM, Roy T. Fielding wrote:
>
> > On May 4, 2011, at 5:39 PM, Eli Collins wrote:
> >
> >> The point is that these discussion should be sorted out, ie you don't
> >> change your development and release model on a release VOTE thread,
> >> you change it on a DISCUSSION thread.
> >
> > That is no different than saying you have a right to veto a
> > release until the issue is addressed, which you don't have.
> >
> > A release vote is a majority decision.  If the majority
> > decides to release, then whatever gets released will define
> > the new norm by which policies are assumed.  If not released,
> > then I suggest collaborating more on the policies before
> > trying to vote again.
> >
> > Either way, we don't hold up a vote for the sake of a
> > policy discussion because voting is a more efficient
> > means of discovering if the policy really matters.
> >
> > ....Roy
> >
>
>

--0003255574062bf1c704a2802c0a--

From general-return-3379-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 05:48:10 2011
Return-Path: <general-return-3379-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 087472D43
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 05:48:10 +0000 (UTC)
Received: (qmail 87958 invoked by uid 500); 5 May 2011 05:48:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87111 invoked by uid 500); 5 May 2011 05:48:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87103 invoked by uid 99); 5 May 2011 05:48:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 05:48:02 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of esammer@cloudera.com designates 209.85.210.176 as permitted sender)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 05:47:55 +0000
Received: by iym1 with SMTP id 1so2472446iym.35
        for <general@hadoop.apache.org>; Wed, 04 May 2011 22:47:34 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.43.58.15 with SMTP id wi15mr374478icb.411.1304574453110; Wed,
 04 May 2011 22:47:33 -0700 (PDT)
Received: by 10.42.213.199 with HTTP; Wed, 4 May 2011 22:47:33 -0700 (PDT)
In-Reply-To: <BANLkTinsQWnrZBUA0=--5Z4WJ7Y1jTRFcQ@mail.gmail.com>
References: <BANLkTik5gj0H4gmvy14Ux=g_r5Tt6D1=MA@mail.gmail.com>
	<C9E71BE8.19FCF%sureshms@yahoo-inc.com>
	<BANLkTikrS9PTeErT15fs2yv+N4_wpPN7xQ@mail.gmail.com>
	<BANLkTimjxRdf6PyTnbZUwrS_znZo00z-5w@mail.gmail.com>
	<BANLkTi=pNF_aQrWPurfXRbjNdshfseNobg@mail.gmail.com>
	<321761.51699.qm@web65903.mail.ac4.yahoo.com>
	<CB238EE8-1E06-4C6C-A7F5-CB00E2E073CB@yahoo-inc.com>
	<BANLkTi=aW6Toxtg7g=nFSV7h4gK68522wg@mail.gmail.com>
	<B746E44F-B6B3-40F2-BF75-C923E9B3962E@yahoo-inc.com>
	<BANLkTinvH8uUAzOQUogqnBanioPZ3UmnHg@mail.gmail.com>
	<BANLkTineZVMp1Fgv+JFT7NNihvDeHxdrvA@mail.gmail.com>
	<BANLkTimpuUhDRL1S4iYQKp3A-eKWcHi8kg@mail.gmail.com>
	<79A1AED5-9005-4F58-BFDB-F7394CB2A69C@gbiv.com>
	<89EC003F-E125-44C5-932C-2157A307CFD6@mac.com>
	<BANLkTinsQWnrZBUA0=--5Z4WJ7Y1jTRFcQ@mail.gmail.com>
Date: Wed, 4 May 2011 22:47:33 -0700
Message-ID: <BANLkTi=vjd_1yBdyudvt3Bk4rFKA2fe0Og@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec517a9a0dc583404a280e888

--bcaec517a9a0dc583404a280e888
Content-Type: text/plain; charset=ISO-8859-1

(non-binding) -1 for similar reasons to what Jeff and others have laid out,
and certainly if we're going to change the development process as a side
effect of a release vote.

On Wed, May 4, 2011 at 9:54 PM, Jeff Hammerbacher <hammer@cloudera.com>wrote:

> -1.
>
> As Roy says, "whatever gets released will define the new norm by which
> policies are assumed", and I certainly don't want this project to change
> its
> norms to accommodate bad practices. In particular, Eli presented three very
> reasonable technical objections to this release. To summarize:
>
> 1) Let's get the JIRAs that are going into this release into trunk first.
> 2) Let's create a JIRA for each issue in the release.
> 3) Let's stick to the release numbering conventions established for this
> project.
>
> I know the folks at Yahoo! are all professional engineers and done
> tremendous work to help get the project to this point. There's no doubt in
> my mind they understand the validity of the above three technical
> objections. In fact, many of them helped author our "How to Contribute"
> page, which established these conventions:
> wiki.apache.org/hadoop/HowToContribute. We develop new features against
> trunk, we create JIRAs for each issue, we review code before it goes into
> trunk, and we only update old releases with bug fixes.
>
> I couldn't be more excited to have Yahoo! once again doing development in
> Apache, and I hope that we can work together to get the work that you've
> done in this branch into one of our upcoming feature releases.
>
> I hope those who voted +1 before Roy clarified what a release vote will
> mean
> for future project norms will reconsider their votes.
>
> While there may be many competing agendas in this community, we all wish to
> see Apache Hadoop releases of the highest quality. Changing our norms to
> allow huge, unreviewed patch sets introducing new features into a past
> release is a step in the wrong direction.
>
> With a little bit of elbow grease, we can get the work done in this branch
> into trunk, get 0.22 out the door, and be ready for a great 0.23 release.
>
> Later,
> Jeff
>
> On Wed, May 4, 2011 at 9:17 PM, Nigel Daley <ndaley@mac.com> wrote:
>
> > I'm really not sure yet how to vote here.  I was going to vote +1 for
> what
> > I was told by a number of Yahoo! committers would be a one time release
> as
> > Yahoo! "comes back to Apache" after a hiatus last fall/winter and ended
> > their own distribution.  Clearly this code was not all developed as a
> > community process, but I was going to support a one time release of what
> > they had developed in exclusion.
> >
> > Then I read Roy's email, which confused me.  We would he or I or anyone
> > else support this release setting precedent or policy since it would walk
> > all over our bylaws, community process, and the consensus nature of our
> > foundation?  This release vote is a lazy majority of the PMC, but other
> > decisions rolled up in this are supposed to be lazy majority of active
> > committers or, in the case of code changes, a lazy consensus.  Setting
> > policy by this release means any sufficiently large group of committers
> > could go off and develop on their own and then commit it to a branch and
> > call a release.
> >
> > Furthermore, it now sounds like this is possibly the first in a line of
> > feature releases off this branch.  Bug fixes releases, sure.  But feature
> > releases?  What's wrong with trunk?
> >
> > Nige
> >
> > On May 4, 2011, at 6:56 PM, Roy T. Fielding wrote:
> >
> > > On May 4, 2011, at 5:39 PM, Eli Collins wrote:
> > >
> > >> The point is that these discussion should be sorted out, ie you don't
> > >> change your development and release model on a release VOTE thread,
> > >> you change it on a DISCUSSION thread.
> > >
> > > That is no different than saying you have a right to veto a
> > > release until the issue is addressed, which you don't have.
> > >
> > > A release vote is a majority decision.  If the majority
> > > decides to release, then whatever gets released will define
> > > the new norm by which policies are assumed.  If not released,
> > > then I suggest collaborating more on the policies before
> > > trying to vote again.
> > >
> > > Either way, we don't hold up a vote for the sake of a
> > > policy discussion because voting is a more efficient
> > > means of discovering if the policy really matters.
> > >
> > > ....Roy
> > >
> >
> >
>



-- 
Eric Sammer
twitter: esammer
data: www.cloudera.com

--bcaec517a9a0dc583404a280e888--

From general-return-3380-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 06:21:55 2011
Return-Path: <general-return-3380-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3CBF52146
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 06:21:55 +0000 (UTC)
Received: (qmail 24637 invoked by uid 500); 5 May 2011 06:21:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24423 invoked by uid 500); 5 May 2011 06:21:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24412 invoked by uid 99); 5 May 2011 06:21:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 06:21:44 +0000
X-ASF-Spam-Status: No, hits=-1.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_SOFTFAIL
X-Spam-Check-By: apache.org
Received-SPF: softfail (nike.apache.org: transitioning domain of matei@eecs.berkeley.edu does not designate 169.229.218.146 as permitted sender)
Received: from [169.229.218.146] (HELO cm05fe.IST.Berkeley.EDU) (169.229.218.146)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 06:21:36 +0000
Received: from c-67-164-92-113.hsd1.ca.comcast.net ([67.164.92.113] helo=[192.168.1.4])
	by cm05fe.ist.berkeley.edu with esmtpsa (TLSv1:AES128-SHA:128)
	(Exim 4.72)
	(auth plain:matei@eecs.berkeley.edu)
	(envelope-from <matei@eecs.berkeley.edu>)
	id 1QHrw1-0007oo-Hz
	for general@hadoop.apache.org; Wed, 04 May 2011 23:21:14 -0700
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
From: Matei Zaharia <matei@eecs.berkeley.edu>
In-Reply-To: <BANLkTi=vjd_1yBdyudvt3Bk4rFKA2fe0Og@mail.gmail.com>
Date: Wed, 4 May 2011 23:21:08 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <CB428114-99DC-4687-A1F0-952700BB21DA@eecs.berkeley.edu>
References: <BANLkTik5gj0H4gmvy14Ux=g_r5Tt6D1=MA@mail.gmail.com> <C9E71BE8.19FCF%sureshms@yahoo-inc.com> <BANLkTikrS9PTeErT15fs2yv+N4_wpPN7xQ@mail.gmail.com> <BANLkTimjxRdf6PyTnbZUwrS_znZo00z-5w@mail.gmail.com> <BANLkTi=pNF_aQrWPurfXRbjNdshfseNobg@mail.gmail.com> <321761.51699.qm@web65903.mail.ac4.yahoo.com> <CB238EE8-1E06-4C6C-A7F5-CB00E2E073CB@yahoo-inc.com> <BANLkTi=aW6Toxtg7g=nFSV7h4gK68522wg@mail.gmail.com> <B746E44F-B6B3-40F2-BF75-C923E9B3962E@yahoo-inc.com> <BANLkTinvH8uUAzOQUogqnBanioPZ3UmnHg@mail.gmail.com> <BANLkTineZVMp1Fgv+JFT7NNihvDeHxdrvA@mail.gmail.com> <BANLkTimpuUhDRL1S4iYQKp3A-eKWcHi8kg@mail.gmail.com> <79A1AED5-9005-4F58-BFDB-F7394CB2A69C@gbiv.com> <89EC003F-E125-44C5-932C-2157A307CFD6@mac.com> <BANLkTinsQWnrZBUA0=--5Z4WJ7Y1jTRFcQ@mail.gmail.com> <BANLkTi=vjd_1yBdyudvt3Bk4rFKA2fe0Og@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

I'm not going to cast a vote, but I'm concerned about this for the same =
reasons Eli brought up -- in particular, compatibility with 0.22. I'm an =
author of several patches that have gone into 0.21 and trunk, only to =
stay on hiatus for 2 years because the project hasn't made a stable =
release since 0.20. (Today, many of these patches are being used through =
CDH, which is great, but it would be nice to see them in an Apache =
release too.) This push of features into 0.20.203 makes a widely used =
0.22 seem even more distant. Can we at least get a confirmation that =
these changes will be included in 0.22, as well as a timeline?

To support a vibrant developer community, Apache Hadoop should not just =
be a mechanism for Yahoo and Cloudera to publish patches. It should =
include a well-defined process for smaller third-party contributors to =
push changes that will make it into a stable release within a reasonable =
time horizon. The lack of such a process has been a major cause for the =
slowdown in the project in my perspective.

Matei



On May 4, 2011, at 10:47 PM, Eric Sammer wrote:

> (non-binding) -1 for similar reasons to what Jeff and others have laid =
out,
> and certainly if we're going to change the development process as a =
side
> effect of a release vote.
>=20
> On Wed, May 4, 2011 at 9:54 PM, Jeff Hammerbacher =
<hammer@cloudera.com>wrote:
>=20
>> -1.
>>=20
>> As Roy says, "whatever gets released will define the new norm by =
which
>> policies are assumed", and I certainly don't want this project to =
change
>> its
>> norms to accommodate bad practices. In particular, Eli presented =
three very
>> reasonable technical objections to this release. To summarize:
>>=20
>> 1) Let's get the JIRAs that are going into this release into trunk =
first.
>> 2) Let's create a JIRA for each issue in the release.
>> 3) Let's stick to the release numbering conventions established for =
this
>> project.
>>=20
>> I know the folks at Yahoo! are all professional engineers and done
>> tremendous work to help get the project to this point. There's no =
doubt in
>> my mind they understand the validity of the above three technical
>> objections. In fact, many of them helped author our "How to =
Contribute"
>> page, which established these conventions:
>> wiki.apache.org/hadoop/HowToContribute. We develop new features =
against
>> trunk, we create JIRAs for each issue, we review code before it goes =
into
>> trunk, and we only update old releases with bug fixes.
>>=20
>> I couldn't be more excited to have Yahoo! once again doing =
development in
>> Apache, and I hope that we can work together to get the work that =
you've
>> done in this branch into one of our upcoming feature releases.
>>=20
>> I hope those who voted +1 before Roy clarified what a release vote =
will
>> mean
>> for future project norms will reconsider their votes.
>>=20
>> While there may be many competing agendas in this community, we all =
wish to
>> see Apache Hadoop releases of the highest quality. Changing our norms =
to
>> allow huge, unreviewed patch sets introducing new features into a =
past
>> release is a step in the wrong direction.
>>=20
>> With a little bit of elbow grease, we can get the work done in this =
branch
>> into trunk, get 0.22 out the door, and be ready for a great 0.23 =
release.
>>=20
>> Later,
>> Jeff
>>=20
>> On Wed, May 4, 2011 at 9:17 PM, Nigel Daley <ndaley@mac.com> wrote:
>>=20
>>> I'm really not sure yet how to vote here.  I was going to vote +1 =
for
>> what
>>> I was told by a number of Yahoo! committers would be a one time =
release
>> as
>>> Yahoo! "comes back to Apache" after a hiatus last fall/winter and =
ended
>>> their own distribution.  Clearly this code was not all developed as =
a
>>> community process, but I was going to support a one time release of =
what
>>> they had developed in exclusion.
>>>=20
>>> Then I read Roy's email, which confused me.  We would he or I or =
anyone
>>> else support this release setting precedent or policy since it would =
walk
>>> all over our bylaws, community process, and the consensus nature of =
our
>>> foundation?  This release vote is a lazy majority of the PMC, but =
other
>>> decisions rolled up in this are supposed to be lazy majority of =
active
>>> committers or, in the case of code changes, a lazy consensus.  =
Setting
>>> policy by this release means any sufficiently large group of =
committers
>>> could go off and develop on their own and then commit it to a branch =
and
>>> call a release.
>>>=20
>>> Furthermore, it now sounds like this is possibly the first in a line =
of
>>> feature releases off this branch.  Bug fixes releases, sure.  But =
feature
>>> releases?  What's wrong with trunk?
>>>=20
>>> Nige
>>>=20
>>> On May 4, 2011, at 6:56 PM, Roy T. Fielding wrote:
>>>=20
>>>> On May 4, 2011, at 5:39 PM, Eli Collins wrote:
>>>>=20
>>>>> The point is that these discussion should be sorted out, ie you =
don't
>>>>> change your development and release model on a release VOTE =
thread,
>>>>> you change it on a DISCUSSION thread.
>>>>=20
>>>> That is no different than saying you have a right to veto a
>>>> release until the issue is addressed, which you don't have.
>>>>=20
>>>> A release vote is a majority decision.  If the majority
>>>> decides to release, then whatever gets released will define
>>>> the new norm by which policies are assumed.  If not released,
>>>> then I suggest collaborating more on the policies before
>>>> trying to vote again.
>>>>=20
>>>> Either way, we don't hold up a vote for the sake of a
>>>> policy discussion because voting is a more efficient
>>>> means of discovering if the policy really matters.
>>>>=20
>>>> ....Roy
>>>>=20
>>>=20
>>>=20
>>=20
>=20
>=20
>=20
> --=20
> Eric Sammer
> twitter: esammer
> data: www.cloudera.com


From general-return-3381-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 06:31:58 2011
Return-Path: <general-return-3381-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3AB232D08
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 06:31:58 +0000 (UTC)
Received: (qmail 40422 invoked by uid 500); 5 May 2011 06:31:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40385 invoked by uid 500); 5 May 2011 06:31:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40377 invoked by uid 99); 5 May 2011 06:31:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 06:31:55 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of eli@cloudera.com designates 209.85.215.176 as permitted sender)
Received: from [209.85.215.176] (HELO mail-ey0-f176.google.com) (209.85.215.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 06:31:51 +0000
Received: by eya28 with SMTP id 28so819859eya.35
        for <general@hadoop.apache.org>; Wed, 04 May 2011 23:31:29 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.14.17.100 with SMTP id i76mr961383eei.67.1304577089659; Wed,
 04 May 2011 23:31:29 -0700 (PDT)
Received: by 10.14.37.13 with HTTP; Wed, 4 May 2011 23:31:29 -0700 (PDT)
In-Reply-To: <C9E75C07.11858%eyang@yahoo-inc.com>
References: <BANLkTimr4SkiQYnZLaLOMFPnxrTHoTigEw@mail.gmail.com>
	<C9E75C07.11858%eyang@yahoo-inc.com>
Date: Wed, 4 May 2011 23:31:29 -0700
Message-ID: <BANLkTinT5z_2bGVnWZmBQCwVxi-Y5Z=9fw@mail.gmail.com>
Subject: Re: [DISCUSSION] development process of Hadoop
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Wed, May 4, 2011 at 7:39 PM, Eric Yang <eyang@yahoo-inc.com> wrote:
> If we reflect back and see how the development community end up in its cu=
rrent state for Hadoop. =A0There are development rapidly happening and test=
ed in all kind of organizations. =A0However, Hadoop committers are only com=
mitting code that are interested by the sponsored companies. =A0People are =
coding defensively to ensuring only self serving patches would be committed=
, and helping others and merging problem are always prioritized secondary. =
=A0While the world demand agility, the "review then commit" process is prev=
enting progress from happening. =A0Committers are afraid to commit patches =
because review hasn't took place. =A0By the time patch is reviewed, it does=
 not apply properly. =A0People end up having to generate multiple version o=
f patches to ensure the code can be applied. =A0The large lag time between =
patch generation and reviewed is taking significant toll on the community a=
nd progress.
>
> Yahoo have a great team of developers who improves Hadoop at faster pace =
with its own fork of the source code. =A0The reason that Yahoo was able to =
achieve faster improvement with features was due to the ability to use sour=
ce code repository tools properly. =A0Unfortunate for Yahoo, their source c=
ode repository was not Apache svn trunk. =A0I applause Owen and Arun's effo=
rt for men powering and backward/forward porting the changes between yahoo =
github and Apache svn. =A0There might be some jiras that needs to be merged=
 into Hadoop 0.20.203 branch to ensure the linage is correct. =A0The commun=
ity should offer to help with detail listing of what is missing rather than=
 vote -1 without concise reasoning of what is missing.
>
> JIRA is meant as a discussion and collaboration tool, but hadoop communit=
y intends to use it as the source code version control system with men powe=
red diff maker. =A0While spending time in the incubator with other project,=
 the mentors have explained that it is not ASF's philosophy to use "review =
then commit".

ASF's policy is that projects make this decision for themselves:
http://www.apache.org/dev/project-creation.html

The Hadoop bylaws specify that code changes are lazy consensus, ie you
need a +1 from a committer. Technically the code doesn't have to be
reviewed before committing it, that's just been the norm.

I don't think jira is technically required either, it's just been the
norm. The vote for the patch has to happen on the lists, that happens
as a side effect of jira traffic going to the dev lists.

> Hadoop community should rethink if the community is using the right tools=
 for the right task.
>
> Use JIRA, if there is large feature set that requires brain storming, and=
 developers should have the ability to make small incremental changes witho=
ut RTC. =A0This will ensure developers help each other rather than policing=
 each other.
>
> Any thoughts?
>

I think you can move quickly with RTC or CTR, I've worked on RTC
projects that have moved quickly. It requires people dedicate
bandwidth to reviewing changes. If you do want all your code reviewed
(at some point) then you're ultimately limited by review bandwidth,
with either RTC or CTR.

The time it takes to file a jira is normally insignificant compared to
the time to create and test a change. The idea with using jira is that
you propose/discuss a change before creating code. You could do that
on the lists too. I agree using just a code review tool for small
stuff would be faster, eg things that don't require a bug #, release
note, etc.

Thanks,
Eli

From general-return-3382-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 06:32:58 2011
Return-Path: <general-return-3382-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 266A021F8
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 06:32:58 +0000 (UTC)
Received: (qmail 42667 invoked by uid 500); 5 May 2011 06:32:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42594 invoked by uid 500); 5 May 2011 06:32:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42586 invoked by uid 99); 5 May 2011 06:32:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 06:32:56 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 06:32:50 +0000
Received: from [192.168.1.71] (snvvpn4-10-72-168-c66.hq.corp.yahoo.com [10.72.168.66])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p456W7bJ015778
	for <general@hadoop.apache.org>; Wed, 4 May 2011 23:32:08 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1304577128;
	bh=cERPNIzrrDZfZ+c4ycQEyiimIfcrml3S8rg621pP64k=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=IZWJpShNohLfzi86zclxGlTRr+6tbK2zKbQIZiP7xqxfPrU+rV9NfhmP5HBAYQ+6y
	 lOHQysUho+KnyQDldu/QG27iLTfkYCdTjd4qko0U7Gh+IVq2xLUBb+JnZtFYao0U3U
	 2ZmNIbzRJm4zvj04es4t7LmyFdxOi7eFgukkzQAU=
Message-Id: <CAE2C4B4-5CB3-4533-90ED-587ECA2569F3@yahoo-inc.com>
From: Sanjay Radia <sradia@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
Date: Wed, 4 May 2011 23:32:06 -0700
References: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

+1
  downloaded, built, deployed on one node cluster.


sanjay

On May 4, 2011, at 10:31 AM, Owen O'Malley wrote:

> Here's an updated release candidate for 0.20.203.0. I've  
> incorporated the feedback and included all of the patches from  
> 0.20.2, which is the last stable release. I also fixed the eclipse- 
> plugin problem.
>
> The candidate is at: http://people.apache.org/~omalley/hadoop-0.20.203.0-rc1/
>
> Please download it, inspect it, compile it, and test it. Clearly,  
> I'm +1.
>
> -- Owen


From general-return-3383-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 06:41:18 2011
Return-Path: <general-return-3383-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9BA8E2BCD
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 06:41:18 +0000 (UTC)
Received: (qmail 57478 invoked by uid 500); 5 May 2011 06:41:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57428 invoked by uid 500); 5 May 2011 06:41:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57420 invoked by uid 99); 5 May 2011 06:41:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 06:41:15 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 209.85.215.176 as permitted sender)
Received: from [209.85.215.176] (HELO mail-ey0-f176.google.com) (209.85.215.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 06:41:09 +0000
Received: by eya28 with SMTP id 28so821781eya.35
        for <general@hadoop.apache.org>; Wed, 04 May 2011 23:40:48 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.14.46.214 with SMTP id r62mr905896eeb.99.1304577648574; Wed,
 04 May 2011 23:40:48 -0700 (PDT)
Received: by 10.14.37.13 with HTTP; Wed, 4 May 2011 23:40:48 -0700 (PDT)
In-Reply-To: <BANLkTimzcTyLoLppo9LxAJuXgzsqM7Ppew@mail.gmail.com>
References: <BANLkTimzcTyLoLppo9LxAJuXgzsqM7Ppew@mail.gmail.com>
Date: Wed, 4 May 2011 23:40:48 -0700
Message-ID: <BANLkTimVpqfrfhN2bq8QhfB_6FrnKyE+6w@mail.gmail.com>
Subject: Re: [DISCUSSION] Release rules
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Wed, May 4, 2011 at 5:59 PM, Tom White <tom@cloudera.com> wrote:
> One year ago (to the day!) Chris started a discussion about the
> release manager role
> (http://mail-archives.apache.org/mod_mbox/hadoop-general/201005.mbox/%3Ch2q1267dd3b1005041331r7d8f696di370a279ff605832f@mail.gmail.com%3E).
> In light of today's disagreements, I think we should restart this
> discussion and incorporate these rules into the bylaws, since it
> formalizes our practices.
>
> I'm happy to drive this. We could start by discussing Chris' proposal
> (see clarifications in
> http://mail-archives.apache.org/mod_mbox/hadoop-general/201005.mbox/%3Ct2y1267dd3b1005051201h7116e4caud75673ac9d5128d6@mail.gmail.com%3E),
> then when we get consensus we can put the document on the website.
> (BTW does anyone know if the bylaws were checked into SVN anywhere?
> These belong together.)

Sounds good to me. I like Chris' proposal, he was clear that "nothing
should be in (unreleased) 0.x that isn't also in trunk." so that may
needs to be revisited if we want to be consistent with today's vote.

I don't think the bylaws were checked in, we should do that first. How
about checking them into the site repo so they get generated as part
of the docs? Eg this is how Pig does it:
http://pig.apache.org/bylaws.html

Thanks,
Eli

From general-return-3384-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 06:50:26 2011
Return-Path: <general-return-3384-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DFA362201
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 06:50:26 +0000 (UTC)
Received: (qmail 69003 invoked by uid 500); 5 May 2011 06:50:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68950 invoked by uid 500); 5 May 2011 06:50:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68942 invoked by uid 99); 5 May 2011 06:50:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 06:50:23 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [216.243.76.15] (HELO Exchg-FE01.marklogic.com) (216.243.76.15)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 06:50:17 +0000
Received: from exchg-be.marklogic.com ([fe80::cdc9:46e4:439b:bc0e]) by
 Exchg-FE01.marklogic.com ([::1]) with mapi; Wed, 4 May 2011 23:49:56 -0700
From: Jane Chen <Jane.Chen@marklogic.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 4 May 2011 23:49:22 -0700
Subject: RE: [VOTE] Release candidate 0.20.203.0-rc1
Thread-Topic: [VOTE] Release candidate 0.20.203.0-rc1
Thread-Index: AcwK7KbUFUkVmeFCT9yPXrcqLvxjpAAAtGvQ
Message-ID: <E9E9B636B25C054B8E6B9131614B925A04B3BE86@EXCHG-BE.marklogic.com>
References: <BANLkTik5gj0H4gmvy14Ux=g_r5Tt6D1=MA@mail.gmail.com>
 <C9E71BE8.19FCF%sureshms@yahoo-inc.com>
 <BANLkTikrS9PTeErT15fs2yv+N4_wpPN7xQ@mail.gmail.com>
 <BANLkTimjxRdf6PyTnbZUwrS_znZo00z-5w@mail.gmail.com>
 <BANLkTi=pNF_aQrWPurfXRbjNdshfseNobg@mail.gmail.com>
 <321761.51699.qm@web65903.mail.ac4.yahoo.com>
 <CB238EE8-1E06-4C6C-A7F5-CB00E2E073CB@yahoo-inc.com>
 <BANLkTi=aW6Toxtg7g=nFSV7h4gK68522wg@mail.gmail.com>
 <B746E44F-B6B3-40F2-BF75-C923E9B3962E@yahoo-inc.com>
 <BANLkTinvH8uUAzOQUogqnBanioPZ3UmnHg@mail.gmail.com>
 <BANLkTineZVMp1Fgv+JFT7NNihvDeHxdrvA@mail.gmail.com>
 <BANLkTimpuUhDRL1S4iYQKp3A-eKWcHi8kg@mail.gmail.com>
 <79A1AED5-9005-4F58-BFDB-F7394CB2A69C@gbiv.com>
 <89EC003F-E125-44C5-932C-2157A307CFD6@mac.com>
 <BANLkTinsQWnrZBUA0=--5Z4WJ7Y1jTRFcQ@mail.gmail.com>
 <BANLkTi=vjd_1yBdyudvt3Bk4rFKA2fe0Og@mail.gmail.com>
 <CB428114-99DC-4687-A1F0-952700BB21DA@eecs.berkeley.edu>
In-Reply-To: <CB428114-99DC-4687-A1F0-952700BB21DA@eecs.berkeley.edu>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Agree.  As a new comer, I had trouble figuring out which version to adopt -=
- 0.20.2 vs. 0.21. This new release candidate seems to add more confusion t=
o general users.

Jane

-----Original Message-----
From: Matei Zaharia [mailto:matei@eecs.berkeley.edu]=20
Sent: Wednesday, May 04, 2011 11:21 PM
To: general@hadoop.apache.org
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1

I'm not going to cast a vote, but I'm concerned about this for the same rea=
sons Eli brought up -- in particular, compatibility with 0.22. I'm an autho=
r of several patches that have gone into 0.21 and trunk, only to stay on hi=
atus for 2 years because the project hasn't made a stable release since 0.2=
0. (Today, many of these patches are being used through CDH, which is great=
, but it would be nice to see them in an Apache release too.) This push of =
features into 0.20.203 makes a widely used 0.22 seem even more distant. Can=
 we at least get a confirmation that these changes will be included in 0.22=
, as well as a timeline?

To support a vibrant developer community, Apache Hadoop should not just be =
a mechanism for Yahoo and Cloudera to publish patches. It should include a =
well-defined process for smaller third-party contributors to push changes t=
hat will make it into a stable release within a reasonable time horizon. Th=
e lack of such a process has been a major cause for the slowdown in the pro=
ject in my perspective.

Matei



On May 4, 2011, at 10:47 PM, Eric Sammer wrote:

> (non-binding) -1 for similar reasons to what Jeff and others have laid ou=
t,
> and certainly if we're going to change the development process as a side
> effect of a release vote.
>=20
> On Wed, May 4, 2011 at 9:54 PM, Jeff Hammerbacher <hammer@cloudera.com>wr=
ote:
>=20
>> -1.
>>=20
>> As Roy says, "whatever gets released will define the new norm by which
>> policies are assumed", and I certainly don't want this project to change
>> its
>> norms to accommodate bad practices. In particular, Eli presented three v=
ery
>> reasonable technical objections to this release. To summarize:
>>=20
>> 1) Let's get the JIRAs that are going into this release into trunk first=
.
>> 2) Let's create a JIRA for each issue in the release.
>> 3) Let's stick to the release numbering conventions established for this
>> project.
>>=20
>> I know the folks at Yahoo! are all professional engineers and done
>> tremendous work to help get the project to this point. There's no doubt =
in
>> my mind they understand the validity of the above three technical
>> objections. In fact, many of them helped author our "How to Contribute"
>> page, which established these conventions:
>> wiki.apache.org/hadoop/HowToContribute. We develop new features against
>> trunk, we create JIRAs for each issue, we review code before it goes int=
o
>> trunk, and we only update old releases with bug fixes.
>>=20
>> I couldn't be more excited to have Yahoo! once again doing development i=
n
>> Apache, and I hope that we can work together to get the work that you've
>> done in this branch into one of our upcoming feature releases.
>>=20
>> I hope those who voted +1 before Roy clarified what a release vote will
>> mean
>> for future project norms will reconsider their votes.
>>=20
>> While there may be many competing agendas in this community, we all wish=
 to
>> see Apache Hadoop releases of the highest quality. Changing our norms to
>> allow huge, unreviewed patch sets introducing new features into a past
>> release is a step in the wrong direction.
>>=20
>> With a little bit of elbow grease, we can get the work done in this bran=
ch
>> into trunk, get 0.22 out the door, and be ready for a great 0.23 release=
.
>>=20
>> Later,
>> Jeff
>>=20
>> On Wed, May 4, 2011 at 9:17 PM, Nigel Daley <ndaley@mac.com> wrote:
>>=20
>>> I'm really not sure yet how to vote here.  I was going to vote +1 for
>> what
>>> I was told by a number of Yahoo! committers would be a one time release
>> as
>>> Yahoo! "comes back to Apache" after a hiatus last fall/winter and ended
>>> their own distribution.  Clearly this code was not all developed as a
>>> community process, but I was going to support a one time release of wha=
t
>>> they had developed in exclusion.
>>>=20
>>> Then I read Roy's email, which confused me.  We would he or I or anyone
>>> else support this release setting precedent or policy since it would wa=
lk
>>> all over our bylaws, community process, and the consensus nature of our
>>> foundation?  This release vote is a lazy majority of the PMC, but other
>>> decisions rolled up in this are supposed to be lazy majority of active
>>> committers or, in the case of code changes, a lazy consensus.  Setting
>>> policy by this release means any sufficiently large group of committers
>>> could go off and develop on their own and then commit it to a branch an=
d
>>> call a release.
>>>=20
>>> Furthermore, it now sounds like this is possibly the first in a line of
>>> feature releases off this branch.  Bug fixes releases, sure.  But featu=
re
>>> releases?  What's wrong with trunk?
>>>=20
>>> Nige
>>>=20
>>> On May 4, 2011, at 6:56 PM, Roy T. Fielding wrote:
>>>=20
>>>> On May 4, 2011, at 5:39 PM, Eli Collins wrote:
>>>>=20
>>>>> The point is that these discussion should be sorted out, ie you don't
>>>>> change your development and release model on a release VOTE thread,
>>>>> you change it on a DISCUSSION thread.
>>>>=20
>>>> That is no different than saying you have a right to veto a
>>>> release until the issue is addressed, which you don't have.
>>>>=20
>>>> A release vote is a majority decision.  If the majority
>>>> decides to release, then whatever gets released will define
>>>> the new norm by which policies are assumed.  If not released,
>>>> then I suggest collaborating more on the policies before
>>>> trying to vote again.
>>>>=20
>>>> Either way, we don't hold up a vote for the sake of a
>>>> policy discussion because voting is a more efficient
>>>> means of discovering if the policy really matters.
>>>>=20
>>>> ....Roy
>>>>=20
>>>=20
>>>=20
>>=20
>=20
>=20
>=20
> --=20
> Eric Sammer
> twitter: esammer
> data: www.cloudera.com


From general-return-3385-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 06:53:06 2011
Return-Path: <general-return-3385-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EDBA82252
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 06:53:05 +0000 (UTC)
Received: (qmail 73125 invoked by uid 500); 5 May 2011 06:53:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73082 invoked by uid 500); 5 May 2011 06:53:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73070 invoked by uid 99); 5 May 2011 06:53:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 06:53:03 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jdcryans@gmail.com designates 209.85.213.176 as permitted sender)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 06:52:56 +0000
Received: by yxd5 with SMTP id 5so941130yxd.35
        for <general@hadoop.apache.org>; Wed, 04 May 2011 23:52:35 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=pemoyu0QFBiafh+KxJrXqspUhGGXJXpEeESX9qHZo5k=;
        b=P0nW4/Mtg2ZkGylh698skYHwEa8vBqwfRF+bfcfyGDmWqCVkedjgMxJcW4LKFAWY84
         kqRxw37nxbsLnIRqiGkPeJcWXGtWb8lqhprmYA1+48RENKLlGb8YG+8Vu8nv/YiPToMf
         Jc0QIE4LILg6YqexcigOqGD265iseNTbCHLvs=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=UtQb6GHyrI6+o2RwQp7AmDYOhwQWBeptlEefj8d1Dw4VPkVOk1IslQPW/P8qgKIbI6
         OUajMF5OTOfZ20/q39dH1Tey/Yz5DNVIR3lhTGgQh78fmPmJuMTcujHsLyit4Bhx6iQB
         eSiBLJNH0Hbm9WyQstxF2bcnaZHmjrhaV+zCw=
MIME-Version: 1.0
Received: by 10.101.134.20 with SMTP id l20mr1329381ann.5.1304578355113; Wed,
 04 May 2011 23:52:35 -0700 (PDT)
Sender: jdcryans@gmail.com
Received: by 10.100.4.10 with HTTP; Wed, 4 May 2011 23:52:34 -0700 (PDT)
In-Reply-To: <3116A002-6EA6-4971-B7AD-261BCC53D76B@gbiv.com>
References: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org>
	<BANLkTikH4PazAADyo8+=3d3v0bUHaS24HQ@mail.gmail.com>
	<3116A002-6EA6-4971-B7AD-261BCC53D76B@gbiv.com>
Date: Wed, 4 May 2011 23:52:34 -0700
X-Google-Sender-Auth: Vcod_Y3urTf5YAYqYpbJHXT4H-E
Message-ID: <BANLkTi=nx_ymZHLapyBYdhPdQ53kG40zqg@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Roy,

On Wed, May 4, 2011 at 7:22 PM, Roy T. Fielding <fielding@gbiv.com> wrote:
> The ASF is a vehicle for whomever wishes to collaborate on a
> given project. =A0Collaboration means helping do the work. =A0Those
> who do the work may do so for whatever reasons that they think
> are good, whether it is because they feel like being charitable
> today, they get paid a salary and the big boss said "work on
> this part", or because they just have an itch worth scratching.
>
> Apache does not care why people choose to collaborate or
> how they choose to apply their own intellectual efforts. =A0We
> welcome all forms of contribution under the terms of our license.

I don't think I was arguing against the contribution of the code in
that branch, it's very welcome, but I'm questioning (and ranting
about) the motivation for releasing a version that even just by name
is a weird hulla-hoop around the usual development practices that
Hadoop has had in the past (not that it's set in stone).

So I wanted to contribute my negative non-binding vote to highlight
that this release is probably very confusing for the general user.
This is 0.20, but it's not. Also it has more numbers, and it starts at
203. Why doing this at all instead of just moving on with 0.22? Or is
0.22 bound to be like 0.21? It almost begs the question if this should
be called 0.22.0 then.

>
> What we do require is a certain amount of civility regarding
> our voting procedures and an emphasis on individual responsibility
> for your votes. =A0Anyone caught *voting* a particular way just
> because the boss says so will be dealt with severely. =A0Votes
> are how we do quality control and make decisions, and no other
> company can be allowed to make decisions for our non-profit.

Yeah I don't think that's a problem here, everyone seem to have their
very own strong opinions.

From general-return-3386-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 09:58:50 2011
Return-Path: <general-return-3386-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CF7BE2C7D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 09:58:50 +0000 (UTC)
Received: (qmail 72160 invoked by uid 500); 5 May 2011 09:58:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72090 invoked by uid 500); 5 May 2011 09:58:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 65696 invoked by uid 99); 5 May 2011 09:52:02 -0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of tvalderrama@tuenti.com designates 209.85.215.48 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=tuenti.com; s=corp;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=Pse/nqpWTyVi8eXWYRU98wr9i0G+boceclJR76kAg78=;
        b=jzU6LWQ27gYW+GWrFcnpRzBb/Os0iLHwVWwad4VWfpwrV1kI5W/fkrwHc4LbcVDEqF
         oOrVpoZsyfEdhs3eWGTY/HgcIstDYb33bg6HZyipoQ0TIfgCStIDMAjcCd2h+/aSA4j5
         0cGPyZhVmmffp/GrABBuEm3C6GS0qT2E0BT0Y=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=tuenti.com; s=corp;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=YVarVi4pvvzahqovcQGQQLutMVkt4ySlaOijhMHTtmqhiepZvpmp+PuA6Afny2FmTz
         P272LzfrHBrYciqx4wNIG5L3Vi1wdwgOFOa2DJ4aOt94qa1AIqPhNclPRSokbzWfznn9
         bST5X0AedypJoMIKWh4ZqN8Hjd59Qic3rGBJg=
MIME-Version: 1.0
Date: Thu, 5 May 2011 11:51:35 +0200
Message-ID: <BANLkTimgKU8p0QLKtVC+xrdSR9Vv5gjx5w@mail.gmail.com>
Subject: Re: [DISCUSSION] development process of Hadoop
From: Tony Valderrama <tvalderrama@tuenti.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hi, I just wanted to drop in a few thoughts from a new developer
working outside of the Hadoop developer community.

On Wed, May 4, 2011 at 7:39 PM, Eric Yang <eyang@yahoo-inc.com> wrote:
> While the world demand agility, the "review then commit" process is preventing progress
> from happening.  People end up having to generate multiple version of patches to ensure
> the code can be applied.  The large lag time between patch generation and reviewed
> is taking significant toll on the community and progress.

> Yahoo have a great team of developers who improves Hadoop at faster pace with its own
> fork of the source code.  The reason that Yahoo was able to achieve faster improvement with
> features was due to the ability to use source code repository tools properly.  Unfortunate
> for Yahoo, their source code repository was not Apache svn trunk.

I agree that the review process is broken.  However, the current
situation is exactly the result of a lack of adherence to this and
other processes.  Various subgroups within the community have
(intentionally or unintentionally) hijacked the project at different
times by avoiding community processes in the interest of agility or
commercial benefit, and the result is a highly fragmented project with
no clear direction.

>From the outside, Hadoop looks like a Yahoo/Cloudera project which
occasionally gets an Apache stamp.  Given the lack of adherence to
processes, as a non-Yahoo/Cloudera developer I have no way of breaking
into the development community.  Who's going to review or commit
patches I submit?  And which of the myriad versions should I even be
trying to patch against?  And given the speed with which undocumented
changes are being made, how am I supposed to figure out if my changes
are going to be relevant or viable next week?  We'd love to contribute
back, but it's just not clear that we or other small players have any
place within the Hadoop developer community.

Here at Tuenti, like various other small-to-midsize Hadoop users,
we've just forked 0.20 and devoted a couple of developers to
maintaining features that we need.  It would be nice to have shiny new
features in the Yahoo branch or the Facebook branch or the Cloudera
branch or the 0.22 branch (does Hadoop even have a trunk at the
moment?), but we'll favor our own stable and familiar branch over the
risky and hefty investment required to adopt a branch without clear
community support.


> Use JIRA, if there is large feature set that requires brain storming, and developers
> should have the ability to make small incremental changes without RTC.  This will ensure developers
> help each other rather than policing each other.

As an outsider, JIRA is the only way I've been able to follow the
changes to Hadoop's code and guess where the project is heading.
Permitting developers to commit without review or documentation will
just further exclude anyone who can't walk down the hall and knock on
an office door to ask about a commit.


Of course, take this with a grain of salt, since I don't claim to be a
part of the Hadoop developer community and I don't forsee Tuenti ever
playing a major role in the developer community.
~Tony

From general-return-3387-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 11:36:04 2011
Return-Path: <general-return-3387-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A4C952058
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 11:36:04 +0000 (UTC)
Received: (qmail 12437 invoked by uid 500); 5 May 2011 11:36:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12396 invoked by uid 500); 5 May 2011 11:36:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12388 invoked by uid 99); 5 May 2011 11:36:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 11:36:03 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 11:35:56 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id DACD51BA8FD
	for <general@hadoop.apache.org>; Thu,  5 May 2011 12:35:34 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id V3l1JJv1GK+F for <general@hadoop.apache.org>;
	Thu,  5 May 2011 12:35:34 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 434D71BAA23
	for <general@hadoop.apache.org>; Thu,  5 May 2011 12:35:33 +0100 (BST)
MailScanner-NULL-Check: 1305200122.23674@pIZMjsiymTh5AqvAudcm9g
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p45BZLCB013103
	for <general@hadoop.apache.org>; Thu, 5 May 2011 12:35:21 +0100 (BST)
Message-ID: <4DC28B79.7000100@apache.org>
Date: Thu, 05 May 2011 12:35:21 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.14) Gecko/20110223 Thunderbird/3.1.8
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSSION] development process of Hadoop
References: <BANLkTimgKU8p0QLKtVC+xrdSR9Vv5gjx5w@mail.gmail.com>
In-Reply-To: <BANLkTimgKU8p0QLKtVC+xrdSR9Vv5gjx5w@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p45BZLCB013103
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 05/05/11 10:51, Tony Valderrama wrote:
> Hi, I just wanted to drop in a few thoughts from a new developer
> working outside of the Hadoop developer community.
>
> On Wed, May 4, 2011 at 7:39 PM, Eric Yang<eyang@yahoo-inc.com>  wrote:
>> While the world demand agility, the "review then commit" process is preventing progress
>> from happening.  People end up having to generate multiple version of patches to ensure
>> the code can be applied.  The large lag time between patch generation and reviewed
>> is taking significant toll on the community and progress.
>
>> Yahoo have a great team of developers who improves Hadoop at faster pace with its own
>> fork of the source code.  The reason that Yahoo was able to achieve faster improvement with
>> features was due to the ability to use source code repository tools properly.  Unfortunate
>> for Yahoo, their source code repository was not Apache svn trunk.
>
> I agree that the review process is broken.  However, the current
> situation is exactly the result of a lack of adherence to this and
> other processes.  Various subgroups within the community have
> (intentionally or unintentionally) hijacked the project at different
> times by avoiding community processes in the interest of agility or
> commercial benefit, and the result is a highly fragmented project with
> no clear direction.
>
>  From the outside, Hadoop looks like a Yahoo/Cloudera project which
> occasionally gets an Apache stamp.  Given the lack of adherence to
> processes, as a non-Yahoo/Cloudera developer I have no way of breaking
> into the development community.  Who's going to review or commit
> patches I submit?  And which of the myriad versions should I even be
> trying to patch against?  And given the speed with which undocumented
> changes are being made, how am I supposed to figure out if my changes
> are going to be relevant or viable next week?  We'd love to contribute
> back, but it's just not clear that we or other small players have any
> place within the Hadoop developer community.

As someone who has commit rights but undercommits, here are my issues
  -I am not full time on hadoop, I have little time to keep my own code 
up to date, let alone review patches
  -I am not fully up to date with all the changes or subtleties in what 
is a big, complicated system
  -I don't want to break the big systems (Y!, Facebook) by introducing 
changes that work on my network and my (small, dynamic) clusters but 
which place limitations on scale. It's why I prefer review by those 
people who do work on large scale projects.

>
>> Use JIRA, if there is large feature set that requires brain storming, and developers
>> should have the ability to make small incremental changes without RTC.  This will ensure developers
>> help each other rather than policing each other.
>
> As an outsider, JIRA is the only way I've been able to follow the
> changes to Hadoop's code and guess where the project is heading.
> Permitting developers to commit without review or documentation will
> just further exclude anyone who can't walk down the hall and knock on
> an office door to ask about a commit.

I've worked in other ASF projects (Axis) where some large dev teams 
(IBM) used to make decisions in team meetings and propagate them. It's 
faster, but less community centric, and when a large dev team (IBM) get 
re-assigned internally everyone is left not just scrambling to catch up 
engineering-wise, but also to make sense of big chunks of 
under-documented code. At least the JIRA-based review process not only 
provides a discussion log, Hudson/Jenkins checks that there are tests, 
no extra warnings, etc.

What could be interesting would be
  -a move to Git to make it easier to pull in patches from other 
branches, and for people like Tony to have their own fork under SCM.
  -adoption of Gerrit for having each JIRA issue move from being a patch 
to a branch (local or remote), so that people can develop the code for 
an issue, others can pull it in and merge it, and so that the issue 
tracks live code, not dead patches
  -more testing of trunk in bigger real/virtual clusters

I don't know how we can do this, I'd love to hear about experiences 
others have with such a process.


From general-return-3388-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 16:19:26 2011
Return-Path: <general-return-3388-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1034B297D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 16:19:25 +0000 (UTC)
Received: (qmail 68150 invoked by uid 500); 5 May 2011 16:19:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68095 invoked by uid 500); 5 May 2011 16:19:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68085 invoked by uid 99); 5 May 2011 16:19:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 16:19:24 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [98.139.52.220] (HELO nm23.bullet.mail.ac4.yahoo.com) (98.139.52.220)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 05 May 2011 16:19:17 +0000
Received: from [98.139.52.188] by nm23.bullet.mail.ac4.yahoo.com with NNFMP; 05 May 2011 16:18:55 -0000
Received: from [98.139.52.135] by tm1.bullet.mail.ac4.yahoo.com with NNFMP; 05 May 2011 16:18:55 -0000
Received: from [127.0.0.1] by omp1018.mail.ac4.yahoo.com with NNFMP; 05 May 2011 16:18:55 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 777673.97233.bm@omp1018.mail.ac4.yahoo.com
Received: (qmail 28246 invoked by uid 60001); 5 May 2011 16:18:55 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1304612335; bh=CIrd4rM+rvXusqowN1gTztGpf6vqBtcYvabNQA3ai40=; h=Message-ID:X-YMail-OSG:Received:X-RocketYMMF:X-Mailer:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=zCNMzMYxjaiAcXP1kabAf84xVH6bTRuGwD4kGzGZoWMoEZfHKUa9S34a1BWzcKc/+6GdyFqwpUad/SGMUeIMWJRUqHGFH5tzd/dySJgOF/ss0VHkFrP2zj6nEOGrar2CjzBaQMR7Sm2EEkbQTRRVLH5YCNQO5P90UO+uQGQjpCg=
Message-ID: <142245.88843.qm@web65502.mail.ac4.yahoo.com>
X-YMail-OSG: z9Vf8jcVM1l4fP7MWsoztO2xcIWgOhFHyN_tmZrWx2Dxe0Z
 3V_gdQV_cXNbQ1xHyHyPVsxxpIhZaX0cYy0Vx_7DmgSuqHh4BRtLrOqzYjFO
 TPf6YqOzIC7csFy.dXT80fKd6nzPr6g_JCs6KhK12yQeRVtqUScEc5HyUysQ
 McBIKjxsZ0UX.LqNEyLuQQr1.V5mDWvJYkO4GOjqMZKyigcmh2rwlkxK9AIT
 YM5EBsTdnzyTbPOGhUrZFtrbCqfAzTigla6HZgtHj9KKOq4zKrUX8lcp6vfp
 .kZW_YyKv6YDcaOFZQ7pB4Oca048zoaRi8QGRwXf2F44H2S2NWQ64m09KmoL
 LEG7vBDTIBy_WE5woH0dewTELjIPkW1eDECAodwvBAOqnzPfGMUzeNBgWfG1
 t7PpP5v.Ynrah
Received: from [69.231.27.174] by web65502.mail.ac4.yahoo.com via HTTP; Thu, 05 May 2011 09:18:55 PDT
X-RocketYMMF: apurtell
X-Mailer: YahooMailClassic/12.0.2 YahooMailWebService/0.8.110.299900
Date: Thu, 5 May 2011 09:18:55 -0700 (PDT)
From: Andrew Purtell <apurtell@apache.org>
Reply-To: apurtell@apache.org
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
To: general@hadoop.apache.org
In-Reply-To: <9234135B-FD91-4E92-B4F5-6E6F7EB38A94@Holsman.NET>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii

Non binding.

  - Andy

--- On Wed, 5/4/11, Ian Holsman <hadoop@holsman.net> wrote:
> just as a Tally we have
> 6+1's (andy.. is yours binding?? if so 7)
> and 3 -1's.
> 
> so according to the votes so far we are releasing.. but
> according to our bylaws.. we need to wait 7 days for
> everyone to chime in.
> 
> --I


From general-return-3389-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 16:37:15 2011
Return-Path: <general-return-3389-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0B1AE22BC
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 16:37:15 +0000 (UTC)
Received: (qmail 1298 invoked by uid 500); 5 May 2011 16:37:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1250 invoked by uid 500); 5 May 2011 16:37:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1242 invoked by uid 99); 5 May 2011 16:37:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 16:37:13 +0000
X-ASF-Spam-Status: No, hits=2.0 required=5.0
	tests=RCVD_IN_BL_SPAMCOP_NET,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 16:37:07 +0000
Received: by pwi16 with SMTP id 16so1536733pwi.35
        for <general@hadoop.apache.org>; Thu, 05 May 2011 09:36:45 -0700 (PDT)
Received: by 10.68.47.231 with SMTP id g7mr3463611pbn.203.1304613405039; Thu,
 05 May 2011 09:36:45 -0700 (PDT)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.68.41.230 with HTTP; Thu, 5 May 2011 09:36:25 -0700 (PDT)
In-Reply-To: <BANLkTimVpqfrfhN2bq8QhfB_6FrnKyE+6w@mail.gmail.com>
References: <BANLkTimzcTyLoLppo9LxAJuXgzsqM7Ppew@mail.gmail.com> <BANLkTimVpqfrfhN2bq8QhfB_6FrnKyE+6w@mail.gmail.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Thu, 5 May 2011 09:36:25 -0700
X-Google-Sender-Auth: C9eoC3_QLeaLbY0rmxGzxZW2y44
Message-ID: <BANLkTinejGbkX1EUaT=mgu9r5weFL=ASZw@mail.gmail.com>
Subject: Re: [DISCUSSION] Release rules
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Wed, May 4, 2011 at 23:40, Eli Collins <eli@cloudera.com> wrote:
> On Wed, May 4, 2011 at 5:59 PM, Tom White <tom@cloudera.com> wrote:
>> One year ago (to the day!) Chris started a discussion about the
>> release manager role
>> (http://mail-archives.apache.org/mod_mbox/hadoop-general/201005.mbox/%3Ch2q1267dd3b1005041331r7d8f696di370a279ff605832f@mail.gmail.com%3E).
>> In light of today's disagreements, I think we should restart this
>> discussion and incorporate these rules into the bylaws, since it
>> formalizes our practices.
>>
>> I'm happy to drive this. We could start by discussing Chris' proposal
>> (see clarifications in
>> http://mail-archives.apache.org/mod_mbox/hadoop-general/201005.mbox/%3Ct2y1267dd3b1005051201h7116e4caud75673ac9d5128d6@mail.gmail.com%3E),
>> then when we get consensus we can put the document on the website.
>> (BTW does anyone know if the bylaws were checked into SVN anywhere?
>> These belong together.)
>
> Sounds good to me. I like Chris' proposal, he was clear that "nothing
> should be in (unreleased) 0.x that isn't also in trunk." so that may
> needs to be revisited if we want to be consistent with today's vote.
>
> I don't think the bylaws were checked in, we should do that first. How
> about checking them into the site repo so they get generated as part
> of the docs? Eg this is how Pig does it:
> http://pig.apache.org/bylaws.html

+1 makes sense.

>
> Thanks,
> Eli
>

From general-return-3390-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 17:03:20 2011
Return-Path: <general-return-3390-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CEAE3241E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 17:03:20 +0000 (UTC)
Received: (qmail 58261 invoked by uid 500); 5 May 2011 17:03:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58213 invoked by uid 500); 5 May 2011 17:03:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58205 invoked by uid 99); 5 May 2011 17:03:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 17:03:18 +0000
X-ASF-Spam-Status: No, hits=3.3 required=5.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 17:03:12 +0000
Received: from SP2-EX07CAS03.ds.corp.yahoo.com (sp2-ex07cas03.corp.sp2.yahoo.com [98.137.59.35])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p45H2Jga000424
	for <general@hadoop.apache.org>; Thu, 5 May 2011 10:02:19 -0700 (PDT)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by
 SP2-EX07CAS03.ds.corp.yahoo.com ([98.137.59.35]) with mapi; Thu, 5 May 2011
 10:02:19 -0700
From: Eric Yang <eyang@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 5 May 2011 10:02:16 -0700
Subject: Re: [DISCUSSION] development process of Hadoop
Thread-Topic: [DISCUSSION] development process of Hadoop
Thread-Index: AcwK7jErYGHl3b7MQomaUu3VtVwVUQAV/5e9
Message-ID: <C9E82628.118A1%eyang@yahoo-inc.com>
In-Reply-To: <BANLkTinT5z_2bGVnWZmBQCwVxi-Y5Z=9fw@mail.gmail.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C9E82628118A1eyangyahooinccom_"
MIME-Version: 1.0

--_000_C9E82628118A1eyangyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Instead of depending on review then commit practice being the norm,  Hadoop=
 committers can probably take advantage of the svn jira plugin.  People can=
 actively commit to svn as long as a jira number is reference in the commit=
.  The commit message will show up in JIRA and leave a trail of activities =
for reference.  Future committers can refer back to the code history to see=
 why the code is written the way it did.  It is less error prone to maintai=
n patch increments.  This seems like a solvable problem by tweaking the beh=
aviors of the hadoop committers.

Regards,
Eric

On 5/4/11 11:31 PM, "Eli Collins" <eli@cloudera.com> wrote:

On Wed, May 4, 2011 at 7:39 PM, Eric Yang <eyang@yahoo-inc.com> wrote:
> If we reflect back and see how the development community end up in its cu=
rrent state for Hadoop.  There are development rapidly happening and tested=
 in all kind of organizations.  However, Hadoop committers are only committ=
ing code that are interested by the sponsored companies.  People are coding=
 defensively to ensuring only self serving patches would be committed, and =
helping others and merging problem are always prioritized secondary.  While=
 the world demand agility, the "review then commit" process is preventing p=
rogress from happening.  Committers are afraid to commit patches because re=
view hasn't took place.  By the time patch is reviewed, it does not apply p=
roperly.  People end up having to generate multiple version of patches to e=
nsure the code can be applied.  The large lag time between patch generation=
 and reviewed is taking significant toll on the community and progress.
>
> Yahoo have a great team of developers who improves Hadoop at faster pace =
with its own fork of the source code.  The reason that Yahoo was able to ac=
hieve faster improvement with features was due to the ability to use source=
 code repository tools properly.  Unfortunate for Yahoo, their source code =
repository was not Apache svn trunk.  I applause Owen and Arun's effort for=
 men powering and backward/forward porting the changes between yahoo github=
 and Apache svn.  There might be some jiras that needs to be merged into Ha=
doop 0.20.203 branch to ensure the linage is correct.  The community should=
 offer to help with detail listing of what is missing rather than vote -1 w=
ithout concise reasoning of what is missing.
>
> JIRA is meant as a discussion and collaboration tool, but hadoop communit=
y intends to use it as the source code version control system with men powe=
red diff maker.  While spending time in the incubator with other project, t=
he mentors have explained that it is not ASF's philosophy to use "review th=
en commit".

ASF's policy is that projects make this decision for themselves:
http://www.apache.org/dev/project-creation.html

The Hadoop bylaws specify that code changes are lazy consensus, ie you
need a +1 from a committer. Technically the code doesn't have to be
reviewed before committing it, that's just been the norm.

I don't think jira is technically required either, it's just been the
norm. The vote for the patch has to happen on the lists, that happens
as a side effect of jira traffic going to the dev lists.

> Hadoop community should rethink if the community is using the right tools=
 for the right task.
>
> Use JIRA, if there is large feature set that requires brain storming, and=
 developers should have the ability to make small incremental changes witho=
ut RTC.  This will ensure developers help each other rather than policing e=
ach other.
>
> Any thoughts?
>

I think you can move quickly with RTC or CTR, I've worked on RTC
projects that have moved quickly. It requires people dedicate
bandwidth to reviewing changes. If you do want all your code reviewed
(at some point) then you're ultimately limited by review bandwidth,
with either RTC or CTR.

The time it takes to file a jira is normally insignificant compared to
the time to create and test a change. The idea with using jira is that
you propose/discuss a change before creating code. You could do that
on the lists too. I agree using just a code review tool for small
stuff would be faster, eg things that don't require a bug #, release
note, etc.

Thanks,
Eli


--_000_C9E82628118A1eyangyahooinccom_--

From general-return-3391-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 17:12:51 2011
Return-Path: <general-return-3391-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 34ABF2C99
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 17:12:51 +0000 (UTC)
Received: (qmail 74044 invoked by uid 500); 5 May 2011 17:12:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73991 invoked by uid 500); 5 May 2011 17:12:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73983 invoked by uid 99); 5 May 2011 17:12:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 17:12:49 +0000
X-ASF-Spam-Status: No, hits=3.3 required=5.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 17:12:41 +0000
Received: from SP2-EX07CAS03.ds.corp.yahoo.com (sp2-ex07cas03.corp.sp2.yahoo.com [98.137.59.35])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p45HC2gD045217
	for <general@hadoop.apache.org>; Thu, 5 May 2011 10:12:02 -0700 (PDT)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by
 SP2-EX07CAS03.ds.corp.yahoo.com ([98.137.59.35]) with mapi; Thu, 5 May 2011
 10:12:02 -0700
From: Eric Yang <eyang@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 5 May 2011 10:12:01 -0700
Subject: Re: [DISCUSSION] development process of Hadoop
Thread-Topic: [DISCUSSION] development process of Hadoop
Thread-Index: AcwLCySHr3U6SykjQSOV1rSGZbb/+wAPGezI
Message-ID: <C9E82871.118A5%eyang@yahoo-inc.com>
In-Reply-To: <BANLkTimgKU8p0QLKtVC+xrdSR9Vv5gjx5w@mail.gmail.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C9E82871118A5eyangyahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C9E82871118A5eyangyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

The by-law of +1 from committers is probably the less barrier that Apache c=
ould make, to prevent people putting in stuff that might not be compatible =
with Apache license.  I also agree that making a working trunk is probably =
the most important priority for Hadoop.  Once the trunk is working, it will=
 be much easier for Hadoop to grow new developers.

Regards,
Eric

On 5/5/11 2:51 AM, "Tony Valderrama" <tvalderrama@tuenti.com> wrote:

Hi, I just wanted to drop in a few thoughts from a new developer
working outside of the Hadoop developer community.

On Wed, May 4, 2011 at 7:39 PM, Eric Yang <eyang@yahoo-inc.com> wrote:
> While the world demand agility, the "review then commit" process is preve=
nting progress
> from happening.  People end up having to generate multiple version of pat=
ches to ensure
> the code can be applied.  The large lag time between patch generation and=
 reviewed
> is taking significant toll on the community and progress.

> Yahoo have a great team of developers who improves Hadoop at faster pace =
with its own
> fork of the source code.  The reason that Yahoo was able to achieve faste=
r improvement with
> features was due to the ability to use source code repository tools prope=
rly.  Unfortunate
> for Yahoo, their source code repository was not Apache svn trunk.

I agree that the review process is broken.  However, the current
situation is exactly the result of a lack of adherence to this and
other processes.  Various subgroups within the community have
(intentionally or unintentionally) hijacked the project at different
times by avoiding community processes in the interest of agility or
commercial benefit, and the result is a highly fragmented project with
no clear direction.

>From the outside, Hadoop looks like a Yahoo/Cloudera project which
occasionally gets an Apache stamp.  Given the lack of adherence to
processes, as a non-Yahoo/Cloudera developer I have no way of breaking
into the development community.  Who's going to review or commit
patches I submit?  And which of the myriad versions should I even be
trying to patch against?  And given the speed with which undocumented
changes are being made, how am I supposed to figure out if my changes
are going to be relevant or viable next week?  We'd love to contribute
back, but it's just not clear that we or other small players have any
place within the Hadoop developer community.

Here at Tuenti, like various other small-to-midsize Hadoop users,
we've just forked 0.20 and devoted a couple of developers to
maintaining features that we need.  It would be nice to have shiny new
features in the Yahoo branch or the Facebook branch or the Cloudera
branch or the 0.22 branch (does Hadoop even have a trunk at the
moment?), but we'll favor our own stable and familiar branch over the
risky and hefty investment required to adopt a branch without clear
community support.


> Use JIRA, if there is large feature set that requires brain storming, and=
 developers
> should have the ability to make small incremental changes without RTC.  T=
his will ensure developers
> help each other rather than policing each other.

As an outsider, JIRA is the only way I've been able to follow the
changes to Hadoop's code and guess where the project is heading.
Permitting developers to commit without review or documentation will
just further exclude anyone who can't walk down the hall and knock on
an office door to ask about a commit.


Of course, take this with a grain of salt, since I don't claim to be a
part of the Hadoop developer community and I don't forsee Tuenti ever
playing a major role in the developer community.
~Tony


--_000_C9E82871118A5eyangyahooinccom_--

From general-return-3392-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 17:24:41 2011
Return-Path: <general-return-3392-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 71A38261F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 17:24:41 +0000 (UTC)
Received: (qmail 98380 invoked by uid 500); 5 May 2011 17:24:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98317 invoked by uid 500); 5 May 2011 17:24:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98274 invoked by uid 99); 5 May 2011 17:24:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 17:24:38 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [85.214.115.60] (HELO h1349259.stratoserver.net) (85.214.115.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 17:24:32 +0000
Received: from isabel-drost.de ([85.214.115.60] helo=motokokusanaghi.localnet)
	by h1349259.stratoserver.net with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
	(Exim 4.69)
	(envelope-from <isabel@apache.org>)
	id 1QI2HY-0006lp-HR
	for general@hadoop.apache.org; Thu, 05 May 2011 17:24:08 +0000
From: Isabel Drost <isabel@apache.org>
Reply-To: isabel@apache.org
To: general@hadoop.apache.org
Subject: 32 days left to Berlin Buzzwords
Date: Thu, 5 May 2011 19:15:22 +0200
User-Agent: KMail/1.13.5 (Linux/2.6.32-5-amd64; KDE/4.4.5; x86_64; ; )
MIME-Version: 1.0
Content-Type: multipart/signed;
  boundary="nextPart2720268.oGGqCYLffK";
  protocol="application/pgp-signature";
  micalg=pgp-sha1
Content-Transfer-Encoding: 7bit
Message-Id: <201105051915.22361.isabel@apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

--nextPart2720268.oGGqCYLffK
Content-Type: text/plain;
  charset="us-ascii"
Content-Transfer-Encoding: 7bit

hey folks,

BerlinBuzzwords 2011 is close only 32 days left until the big Search,
Store and Scale opensource crowd is gathering
in Berlin on June 6th/7th.

The conference again focuses on the topics search,
data analysis and NoSQL. It is to take place on June 6/7th 2011 in Berlin.

We are looking forward to two awesome keynote speakers who shaped the world of
open source data analysis: Doug Cutting, founder of Apache Lucene and
Hadoop) as
well as Ted Dunning (Chief Application Architect at MapR Technologies
and active
developer at Apache Hadoop and Mahout).

We are amazed by the amount and quality of the talk submissions we
got. As a result
this year we have added one more track to the main conference. If you haven't
done so already, make sure to book your ticket now - early bird tickets are
already sold out since April 7th and there might not be many tickets left.

As we would like to give visitors of our main conference a reason to stay in
town for the whole week, we have been talking to local co-working spaces and
companies asking them for free space and WiFi to host Hackathons right after the
main conference - that is on June 8th through 10th.

If you would like to gather with fellow developers and users of your project,
fix bugs together, hack on new features or give users a hands-on introduction to
your tools, please submit your workshop proposal to our wiki:

http://berlinbuzzwords.de/node/428

Please note that slots are assigned on a first come first serve basis. We are
doing our best to get you connected, however space is limited.

The deal is simple: We get you in touch with a conference room provider. Your
event gets promoted in our schedule. Co-Ordination however is completely up to
you: Make sure to provide an interesting abstract, provide a Hackathon
registration area - see the Barcamp page for a good example:

http://berlinbuzzwords.de/wiki/barcamp

Attending Hackathons requires a Berlin Buzzwords ticket and (then free)
registration at the Hackathon in question.

Hope I see you all around in Berlin,


Isabel

--nextPart2720268.oGGqCYLffK
Content-Type: application/pgp-signature; name=signature.asc 
Content-Description: This is a digitally signed message part.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEABECAAYFAk3C2yoACgkQPJpCBAwIhbQvMgCdG1fvGGo8H2/PMe3bQTe3+Lls
wwsAnjB546uJIugKAO+VlKTnQg1qIAvP
=rQoy
-----END PGP SIGNATURE-----

--nextPart2720268.oGGqCYLffK--

From general-return-3393-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 17:33:09 2011
Return-Path: <general-return-3393-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DEBC82782
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 17:33:09 +0000 (UTC)
Received: (qmail 19764 invoked by uid 500); 5 May 2011 17:33:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19675 invoked by uid 500); 5 May 2011 17:33:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19659 invoked by uid 99); 5 May 2011 17:33:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 17:33:06 +0000
X-ASF-Spam-Status: No, hits=3.3 required=5.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 17:32:59 +0000
Received: from SP2-EX07CAS05.ds.corp.yahoo.com (sp2-ex07cas05.corp.sp2.yahoo.com [98.137.59.39])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p45HWB85011731
	for <general@hadoop.apache.org>; Thu, 5 May 2011 10:32:11 -0700 (PDT)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by
 SP2-EX07CAS05.ds.corp.yahoo.com ([98.137.59.39]) with mapi; Thu, 5 May 2011
 10:32:11 -0700
From: Eric Yang <eyang@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 5 May 2011 10:32:10 -0700
Subject: Re: [DISCUSSION] development process of Hadoop
Thread-Topic: [DISCUSSION] development process of Hadoop
Thread-Index: AcwLGKaaUWvmZ/WsQD6X0QVZUYVB5gAMbY8W
Message-ID: <C9E82D2A.118AD%eyang@yahoo-inc.com>
In-Reply-To: <4DC28B79.7000100@apache.org>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C9E82D2A118ADeyangyahooinccom_"
MIME-Version: 1.0

--_000_C9E82D2A118ADeyangyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Git is powerful in maintaining different branch of the source code.  Howeve=
r, it will only work if the entire community is willing to move to git.  Ma=
intaining svn and git hybrid, is a time consuming task that we are paying i=
n full price.  Hadoop community should work smarter for the source control.=
  What do people think about fully adopting git instead of svn?

Regards,
Eric

On 5/5/11 4:35 AM, "Steve Loughran" <stevel@apache.org> wrote:

On 05/05/11 10:51, Tony Valderrama wrote:
> Hi, I just wanted to drop in a few thoughts from a new developer
> working outside of the Hadoop developer community.
>
> On Wed, May 4, 2011 at 7:39 PM, Eric Yang<eyang@yahoo-inc.com>  wrote:
>> While the world demand agility, the "review then commit" process is prev=
enting progress
>> from happening.  People end up having to generate multiple version of pa=
tches to ensure
>> the code can be applied.  The large lag time between patch generation an=
d reviewed
>> is taking significant toll on the community and progress.
>
>> Yahoo have a great team of developers who improves Hadoop at faster pace=
 with its own
>> fork of the source code.  The reason that Yahoo was able to achieve fast=
er improvement with
>> features was due to the ability to use source code repository tools prop=
erly.  Unfortunate
>> for Yahoo, their source code repository was not Apache svn trunk.
>
> I agree that the review process is broken.  However, the current
> situation is exactly the result of a lack of adherence to this and
> other processes.  Various subgroups within the community have
> (intentionally or unintentionally) hijacked the project at different
> times by avoiding community processes in the interest of agility or
> commercial benefit, and the result is a highly fragmented project with
> no clear direction.
>
>  From the outside, Hadoop looks like a Yahoo/Cloudera project which
> occasionally gets an Apache stamp.  Given the lack of adherence to
> processes, as a non-Yahoo/Cloudera developer I have no way of breaking
> into the development community.  Who's going to review or commit
> patches I submit?  And which of the myriad versions should I even be
> trying to patch against?  And given the speed with which undocumented
> changes are being made, how am I supposed to figure out if my changes
> are going to be relevant or viable next week?  We'd love to contribute
> back, but it's just not clear that we or other small players have any
> place within the Hadoop developer community.

As someone who has commit rights but undercommits, here are my issues
  -I am not full time on hadoop, I have little time to keep my own code
up to date, let alone review patches
  -I am not fully up to date with all the changes or subtleties in what
is a big, complicated system
  -I don't want to break the big systems (Y!, Facebook) by introducing
changes that work on my network and my (small, dynamic) clusters but
which place limitations on scale. It's why I prefer review by those
people who do work on large scale projects.

>
>> Use JIRA, if there is large feature set that requires brain storming, an=
d developers
>> should have the ability to make small incremental changes without RTC.  =
This will ensure developers
>> help each other rather than policing each other.
>
> As an outsider, JIRA is the only way I've been able to follow the
> changes to Hadoop's code and guess where the project is heading.
> Permitting developers to commit without review or documentation will
> just further exclude anyone who can't walk down the hall and knock on
> an office door to ask about a commit.

I've worked in other ASF projects (Axis) where some large dev teams
(IBM) used to make decisions in team meetings and propagate them. It's
faster, but less community centric, and when a large dev team (IBM) get
re-assigned internally everyone is left not just scrambling to catch up
engineering-wise, but also to make sense of big chunks of
under-documented code. At least the JIRA-based review process not only
provides a discussion log, Hudson/Jenkins checks that there are tests,
no extra warnings, etc.

What could be interesting would be
  -a move to Git to make it easier to pull in patches from other
branches, and for people like Tony to have their own fork under SCM.
  -adoption of Gerrit for having each JIRA issue move from being a patch
to a branch (local or remote), so that people can develop the code for
an issue, others can pull it in and merge it, and so that the issue
tracks live code, not dead patches
  -more testing of trunk in bigger real/virtual clusters

I don't know how we can do this, I'd love to hear about experiences
others have with such a process.



--_000_C9E82D2A118ADeyangyahooinccom_--

From general-return-3394-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 17:37:14 2011
Return-Path: <general-return-3394-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 630302E4E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 17:37:14 +0000 (UTC)
Received: (qmail 26189 invoked by uid 500); 5 May 2011 17:37:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26133 invoked by uid 500); 5 May 2011 17:37:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26125 invoked by uid 99); 5 May 2011 17:37:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 17:37:12 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [98.139.52.229] (HELO nm7-vm1.bullet.mail.ac4.yahoo.com) (98.139.52.229)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 05 May 2011 17:37:03 +0000
Received: from [98.139.52.191] by nm7.bullet.mail.ac4.yahoo.com with NNFMP; 05 May 2011 17:36:42 -0000
Received: from [98.139.52.149] by tm4.bullet.mail.ac4.yahoo.com with NNFMP; 05 May 2011 17:36:41 -0000
Received: from [127.0.0.1] by omp1032.mail.ac4.yahoo.com with NNFMP; 05 May 2011 17:36:41 -0000
X-Yahoo-Newman-Property: ymail-5
X-Yahoo-Newman-Id: 889033.71772.bm@omp1032.mail.ac4.yahoo.com
Received: (qmail 99217 invoked by uid 60001); 5 May 2011 17:36:41 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1304617001; bh=NVyU6FOLid9dV9f9q2WjrxE+VTOwfP/24sAWV/h2N7Q=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=mSfmtkJanS2Vg+JhJub5xbh++n1DMlXFtyDVD3z5zS6SIeQ9ZX3f+sV1XvrPO6LidhZQwRteHCc3Su8yOMyUaVh6IbCFm9pZ0ZZ/9EUb+lvtoa7K61k3+7j0bVeDYpoi3uskMkhfCNcma6JmCK2qc/su+gbDdHcFbK8T+hbCT60=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=UmN2QD2lQJTc5tk50Ljt/El+Mh4OX+X5N1v8709QZSeZQLXupKwZsm4fPlhWnk19gPh4u0wXM+kw7PGrsqyvd7W3jijGsTRNesAB2q0EUrzIlJ+prhGbxBJuHYLBOYfoDGbg5QYRrYk4WKgTU1L4V6KQkc65yJ9h1Xfk3kYFQHI=;
Message-ID: <672310.96845.qm@web65907.mail.ac4.yahoo.com>
X-YMail-OSG: M2i.AJAVM1kX5pBm4algY..bvUj5HVys5xkgF8199Smwr.M
 QfjAn6QnmvdNSNYvWT_2hry.ZfIvz43FWeG6P3Wb_VB365z4QIeDrEOCth8s
 W0OrRnPRAuFDtSqEoQ0trSuHmCxW2ekVl5S.jrEnT2dGcR8g9jHRyV50Wu.X
 D76oCMLhNRy0EeLMKU1lpH_gn09aO5e4VQ8hWltjsL_YwSadRlw5GKKCDoLG
 ujvHP62uc9obuK6ryxBEhSLledNfo8V6kMchCYxtEC8SrbBNpgBpc_.LeXc0
 qsJEOTuE_V_r3n6aufN_.5B.fpSA3WWxe_rM.sU5Oq1V96SZ9orxPY54xUTB
 zZX0455LiEoO9w8dJsGoaLkftQtYPlS5OpGEvcKv9QhWEUS839DSpwr1L7bx
 gJiOprMYuyXatvaLbOaUTZxEvKufXzcyi_w--
Received: from [209.131.62.113] by web65907.mail.ac4.yahoo.com via HTTP; Thu, 05 May 2011 10:36:41 PDT
X-Mailer: YahooMailRC/559 YahooMailWebService/0.8.110.299900
References: <BANLkTimzcTyLoLppo9LxAJuXgzsqM7Ppew@mail.gmail.com> <BANLkTimVpqfrfhN2bq8QhfB_6FrnKyE+6w@mail.gmail.com> <BANLkTinejGbkX1EUaT=mgu9r5weFL=ASZw@mail.gmail.com>
Date: Thu, 5 May 2011 10:36:41 -0700 (PDT)
From: "Tsz Wo \(Nicholas\), Sze" <s29752-hadoopgeneral@yahoo.com>
Subject: Re: [DISCUSSION] Release rules
To: general@hadoop.apache.org
In-Reply-To: <BANLkTinejGbkX1EUaT=mgu9r5weFL=ASZw@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1926559672-1304617001=:96845"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-1926559672-1304617001=:96845
Content-Type: text/plain; charset=us-ascii

> I don't think the bylaws were checked in, we should do that first. How
> about checking them into the site repo so they get generated as part
> of the docs?

-1

Please don't check in anything before having a vote.  Thanks.

Nicholas





________________________________
From: Konstantin Boudnik <cos@apache.org>
To: general@hadoop.apache.org
Sent: Thu, May 5, 2011 9:36:25 AM
Subject: Re: [DISCUSSION] Release rules

On Wed, May 4, 2011 at 23:40, Eli Collins <eli@cloudera.com> wrote:
> On Wed, May 4, 2011 at 5:59 PM, Tom White <tom@cloudera.com> wrote:
>> One year ago (to the day!) Chris started a discussion about the
>> release manager role
>>(http://mail-archives.apache.org/mod_mbox/hadoop-general/201005.mbox/%3Ch2q1267dd3b1005041331r7d8f696di370a279ff605832f@mail.gmail.com%3E).
>>.
>> In light of today's disagreements, I think we should restart this
>> discussion and incorporate these rules into the bylaws, since it
>> formalizes our practices.
>>
>> I'm happy to drive this. We could start by discussing Chris' proposal
>> (see clarifications in
>>http://mail-archives.apache.org/mod_mbox/hadoop-general/201005.mbox/%3Ct2y1267dd3b1005051201h7116e4caud75673ac9d5128d6@mail.gmail.com%3E),
>>,
>> then when we get consensus we can put the document on the website.
>> (BTW does anyone know if the bylaws were checked into SVN anywhere?
>> These belong together.)
>
> Sounds good to me. I like Chris' proposal, he was clear that "nothing
> should be in (unreleased) 0.x that isn't also in trunk." so that may
> needs to be revisited if we want to be consistent with today's vote.
>
> I don't think the bylaws were checked in, we should do that first. How
> about checking them into the site repo so they get generated as part
> of the docs? Eg this is how Pig does it:
> http://pig.apache.org/bylaws.html

+1 makes sense.

>
> Thanks,
> Eli
>

--0-1926559672-1304617001=:96845--

From general-return-3395-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 17:53:12 2011
Return-Path: <general-return-3395-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 52CBB2722
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 17:53:12 +0000 (UTC)
Received: (qmail 49101 invoked by uid 500); 5 May 2011 17:53:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49053 invoked by uid 500); 5 May 2011 17:53:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49045 invoked by uid 99); 5 May 2011 17:53:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 17:53:10 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 17:53:05 +0000
Received: by bwz8 with SMTP id 8so3469602bwz.35
        for <general@hadoop.apache.org>; Thu, 05 May 2011 10:52:44 -0700 (PDT)
Received: by 10.204.82.143 with SMTP id b15mr214854bkl.118.1304617964234; Thu,
 05 May 2011 10:52:44 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.205.81.143 with HTTP; Thu, 5 May 2011 10:52:24 -0700 (PDT)
In-Reply-To: <C9E82D2A.118AD%eyang@yahoo-inc.com>
References: <4DC28B79.7000100@apache.org> <C9E82D2A.118AD%eyang@yahoo-inc.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Thu, 5 May 2011 10:52:24 -0700
Message-ID: <BANLkTin_1YK4a39jVtNOdG_bucK6MQhHEw@mail.gmail.com>
Subject: Re: [DISCUSSION] development process of Hadoop
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d7ef905367ec04a28b0ac2

--0016e6d7ef905367ec04a28b0ac2
Content-Type: text/plain; charset=ISO-8859-1

On Thu, May 5, 2011 at 10:32 AM, Eric Yang <eyang@yahoo-inc.com> wrote:

> Git is powerful in maintaining different branch of the source code.
>  However, it will only work if the entire community is willing to move to
> git.  Maintaining svn and git hybrid, is a time consuming task that we are
> paying in full price.  Hadoop community should work smarter for the source
> control.  What do people think about fully adopting git instead of svn?
>

+1 for Git as a tool. But using git makes it even _more_ important that we
have a clearly defined release process that outlines which branches are
meant to be released as official artifacts, and what the inclusion criteria
for those branches should be.

-Todd


> On 5/5/11 4:35 AM, "Steve Loughran" <stevel@apache.org> wrote:
>
> On 05/05/11 10:51, Tony Valderrama wrote:
> > Hi, I just wanted to drop in a few thoughts from a new developer
> > working outside of the Hadoop developer community.
> >
> > On Wed, May 4, 2011 at 7:39 PM, Eric Yang<eyang@yahoo-inc.com>  wrote:
> >> While the world demand agility, the "review then commit" process is
> preventing progress
> >> from happening.  People end up having to generate multiple version of
> patches to ensure
> >> the code can be applied.  The large lag time between patch generation
> and reviewed
> >> is taking significant toll on the community and progress.
> >
> >> Yahoo have a great team of developers who improves Hadoop at faster pace
> with its own
> >> fork of the source code.  The reason that Yahoo was able to achieve
> faster improvement with
> >> features was due to the ability to use source code repository tools
> properly.  Unfortunate
> >> for Yahoo, their source code repository was not Apache svn trunk.
> >
> > I agree that the review process is broken.  However, the current
> > situation is exactly the result of a lack of adherence to this and
> > other processes.  Various subgroups within the community have
> > (intentionally or unintentionally) hijacked the project at different
> > times by avoiding community processes in the interest of agility or
> > commercial benefit, and the result is a highly fragmented project with
> > no clear direction.
> >
> >  From the outside, Hadoop looks like a Yahoo/Cloudera project which
> > occasionally gets an Apache stamp.  Given the lack of adherence to
> > processes, as a non-Yahoo/Cloudera developer I have no way of breaking
> > into the development community.  Who's going to review or commit
> > patches I submit?  And which of the myriad versions should I even be
> > trying to patch against?  And given the speed with which undocumented
> > changes are being made, how am I supposed to figure out if my changes
> > are going to be relevant or viable next week?  We'd love to contribute
> > back, but it's just not clear that we or other small players have any
> > place within the Hadoop developer community.
>
> As someone who has commit rights but undercommits, here are my issues
>  -I am not full time on hadoop, I have little time to keep my own code
> up to date, let alone review patches
>  -I am not fully up to date with all the changes or subtleties in what
> is a big, complicated system
>  -I don't want to break the big systems (Y!, Facebook) by introducing
> changes that work on my network and my (small, dynamic) clusters but
> which place limitations on scale. It's why I prefer review by those
> people who do work on large scale projects.
>
> >
> >> Use JIRA, if there is large feature set that requires brain storming,
> and developers
> >> should have the ability to make small incremental changes without RTC.
>  This will ensure developers
> >> help each other rather than policing each other.
> >
> > As an outsider, JIRA is the only way I've been able to follow the
> > changes to Hadoop's code and guess where the project is heading.
> > Permitting developers to commit without review or documentation will
> > just further exclude anyone who can't walk down the hall and knock on
> > an office door to ask about a commit.
>
> I've worked in other ASF projects (Axis) where some large dev teams
> (IBM) used to make decisions in team meetings and propagate them. It's
> faster, but less community centric, and when a large dev team (IBM) get
> re-assigned internally everyone is left not just scrambling to catch up
> engineering-wise, but also to make sense of big chunks of
> under-documented code. At least the JIRA-based review process not only
> provides a discussion log, Hudson/Jenkins checks that there are tests,
> no extra warnings, etc.
>
> What could be interesting would be
>  -a move to Git to make it easier to pull in patches from other
> branches, and for people like Tony to have their own fork under SCM.
>  -adoption of Gerrit for having each JIRA issue move from being a patch
> to a branch (local or remote), so that people can develop the code for
> an issue, others can pull it in and merge it, and so that the issue
> tracks live code, not dead patches
>  -more testing of trunk in bigger real/virtual clusters
>
> I don't know how we can do this, I'd love to hear about experiences
> others have with such a process.
>
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

--0016e6d7ef905367ec04a28b0ac2--

From general-return-3396-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 18:10:43 2011
Return-Path: <general-return-3396-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 64561283C
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 18:10:43 +0000 (UTC)
Received: (qmail 91362 invoked by uid 500); 5 May 2011 18:10:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91304 invoked by uid 500); 5 May 2011 18:10:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91284 invoked by uid 99); 5 May 2011 18:10:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 18:10:41 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of eli@cloudera.com designates 209.85.215.176 as permitted sender)
Received: from [209.85.215.176] (HELO mail-ey0-f176.google.com) (209.85.215.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 18:10:35 +0000
Received: by eya28 with SMTP id 28so1071694eya.35
        for <general@hadoop.apache.org>; Thu, 05 May 2011 11:10:14 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.14.50.196 with SMTP id z44mr1345815eeb.227.1304619013874; Thu,
 05 May 2011 11:10:13 -0700 (PDT)
Received: by 10.14.37.13 with HTTP; Thu, 5 May 2011 11:10:13 -0700 (PDT)
In-Reply-To: <672310.96845.qm@web65907.mail.ac4.yahoo.com>
References: <BANLkTimzcTyLoLppo9LxAJuXgzsqM7Ppew@mail.gmail.com>
	<BANLkTimVpqfrfhN2bq8QhfB_6FrnKyE+6w@mail.gmail.com>
	<BANLkTinejGbkX1EUaT=mgu9r5weFL=ASZw@mail.gmail.com>
	<672310.96845.qm@web65907.mail.ac4.yahoo.com>
Date: Thu, 5 May 2011 11:10:13 -0700
Message-ID: <BANLkTikKe9e9uu6MH5KYqt_s9YwzifQPXA@mail.gmail.com>
Subject: Re: [DISCUSSION] Release rules
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Thu, May 5, 2011 at 10:36 AM, Tsz Wo (Nicholas), Sze
<s29752-hadoopgeneral@yahoo.com> wrote:
>> I don't think the bylaws were checked in, we should do that first. How
>> about checking them into the site repo so they get generated as part
>> of the docs?
>
> -1
>
> Please don't check in anything before having a vote. =A0Thanks.
>

The project already has bylaws.  Owen proposed them and they passed,
we should check them into svn.

http://mail-archives.apache.org/mod_mbox/hadoop-general/201011.mbox/%3CAANL=
kTikWUHBedykosKNqPweuOvu2qCKfcf_+HwwivsQa@mail.gmail.com%3E

Thanks,
Eli

From general-return-3397-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 18:33:54 2011
Return-Path: <general-return-3397-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0F92B27C4
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 18:33:54 +0000 (UTC)
Received: (qmail 62057 invoked by uid 500); 5 May 2011 18:33:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62005 invoked by uid 500); 5 May 2011 18:33:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61997 invoked by uid 99); 5 May 2011 18:33:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 18:33:52 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 18:33:44 +0000
Received: by qwj9 with SMTP id 9so2424170qwj.35
        for <general@hadoop.apache.org>; Thu, 05 May 2011 11:33:24 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        bh=5vb+DEvXxrYqean8J1lZnvTYocFdIXYmsg7tJzz+bzI=;
        b=aMsQ8newTGEYq78w3L8idJmBAWOt0IPmp9p8AjJ2wZvpw09PH39f1lr/nrf7f54dzK
         FNrHgDotVstycPAL1ieLBdpwsOIdKNxKirp+GjnUKjRwrByBQWx4hbWT3DelAFd8212u
         SVALPq1xyQ7dlFKYoksHXv56t1I61/pSUJC0I=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=Bq2UckUgAh7Qy4I0bLpmiRyk4KjvEk/LbEtCkFU7FhcgGd5oSFmV9OKXyUZ4Dbr/Bx
         JGUuIoaMC4wU/DSaaJz2miKDb1T9bc5GSCHg2WJejNmu6e0BPndP0hS3B8bTWd5fFRHb
         Yq36agrK12Co/xgHUebn7bIqjqM7wz4bQ9GjU=
MIME-Version: 1.0
Received: by 10.224.125.66 with SMTP id x2mr2635572qar.365.1304620403926; Thu,
 05 May 2011 11:33:23 -0700 (PDT)
Sender: saint.ack@gmail.com
Received: by 10.224.29.6 with HTTP; Thu, 5 May 2011 11:33:23 -0700 (PDT)
In-Reply-To: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org>
References: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org>
Date: Thu, 5 May 2011 11:33:23 -0700
X-Google-Sender-Auth: hkUyOF6KuZrt3ld1X_N_SaISasA
Message-ID: <BANLkTikEX=K9LjKdgJ3-yMRAR1XBoyae1A@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

I abstain: +/- 0.

I can't vote against the good work done by the crew at Y!

But I can't vote for a 0.20.clusterbomb release that railroads over
precedent compounding further the existing confusion that already
exists around the state of Hadoop.

Thanks,
St.Ack


On Wed, May 4, 2011 at 10:31 AM, Owen O'Malley <omalley@apache.org> wrote:
> Here's an updated release candidate for 0.20.203.0. I've incorporated the feedback and included all of the patches from 0.20.2, which is the last stable release. I also fixed the eclipse-plugin problem.
>
> The candidate is at: http://people.apache.org/~omalley/hadoop-0.20.203.0-rc1/
>
> Please download it, inspect it, compile it, and test it. Clearly, I'm +1.
>
> -- Owen

From general-return-3398-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 19:04:20 2011
Return-Path: <general-return-3398-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 267B22570
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 19:04:20 +0000 (UTC)
Received: (qmail 46781 invoked by uid 500); 5 May 2011 19:04:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46729 invoked by uid 500); 5 May 2011 19:04:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46721 invoked by uid 99); 5 May 2011 19:04:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 19:04:18 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 19:04:10 +0000
Received: from SP1-EX07CAS02.ds.corp.yahoo.com (sp1-ex07cas02.ds.corp.yahoo.com [216.252.116.138])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p45J2wrJ040047
	for <general@hadoop.apache.org>; Thu, 5 May 2011 12:02:58 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1304622178;
	bh=3TPUfkJhtcfhB+lLbOfzURHcBXBjZ1fpwZPj79b37O4=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:MIME-Version;
	b=Fs0l8uOIBc+FX3lHbuqJw89MKxAkaMvl08LkpjG7pFAgGdO00wP8WvAsDknaS3H2n
	 0zpo0KXF4aIr7YzDT4TBnatqzpwnBAT6LJM+B+2N5C2kHKc78Yw5gbvs5CiAWK+QbV
	 xR5F02J3BjyoUkBXn5/6U59Rd5878pKugIzf2gps=
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 SP1-EX07CAS02.ds.corp.yahoo.com ([216.252.116.167]) with mapi; Thu, 5 May
 2011 12:02:57 -0700
From: Suresh Srinivas <sureshms@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 5 May 2011 12:02:56 -0700
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
Thread-Topic: [VOTE] Release candidate 0.20.203.0-rc1
Thread-Index: AcwKremA54H4Jsq0RZSx6FH9bjiPHgAqSFpV
Message-ID: <C9E84270.1A05E%sureshms@yahoo-inc.com>
In-Reply-To: <BANLkTi=B+cc_677HC1wRnpcY73zuF8qekg@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
user-agent: Microsoft-Entourage/13.8.0.101117
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


+1

Downloaded the release, validated checksums, deployed a single-node
cluster, and ran some HDFS sanity tests, Web UI tests and mapreduce
examples.

Regards,
Suresh




From general-return-3399-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 19:45:49 2011
Return-Path: <general-return-3399-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C9F6021FE
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 19:45:49 +0000 (UTC)
Received: (qmail 37231 invoked by uid 500); 5 May 2011 19:45:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37155 invoked by uid 500); 5 May 2011 19:45:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37147 invoked by uid 99); 5 May 2011 19:45:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 19:45:48 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [98.139.52.237] (HELO nm15-vm1.bullet.mail.ac4.yahoo.com) (98.139.52.237)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 05 May 2011 19:45:40 +0000
Received: from [98.139.52.188] by nm15.bullet.mail.ac4.yahoo.com with NNFMP; 05 May 2011 19:45:18 -0000
Received: from [98.139.52.132] by tm1.bullet.mail.ac4.yahoo.com with NNFMP; 05 May 2011 19:45:18 -0000
Received: from [127.0.0.1] by omp1015.mail.ac4.yahoo.com with NNFMP; 05 May 2011 19:45:18 -0000
X-Yahoo-Newman-Property: ymail-5
X-Yahoo-Newman-Id: 97162.22607.bm@omp1015.mail.ac4.yahoo.com
Received: (qmail 46039 invoked by uid 60001); 5 May 2011 19:45:18 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1304624717; bh=DP+Yqtu6pqfphhpLS4Lp3SV3O2p05bkbNV1VIPIaPDQ=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=eTZLkjv2rNMX01G69U+MaSF8tM7X4Hk9mWRy1+3xr8UjClVb+U5N032UVmDm17g2NTr+XBor+VAmoSj1ARjVYX4RCFZ/87hkzFU5bwlg1bTJOoKawg9jHAfLuKRc0TL87e8Teq862nHJBzhDbYpEmuRMYGBzYLpBCB3SawK8N2E=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=UqD7XZX414KBejZnaePiraFIZL0DGnWQ3Wu3TllteYo+ZUyFPmQleY8zVGdK1rClrRjAt57dqPCYD338hDwtGLvcWeU+wgHbNQQ/uRLG4DvTTi+R89UwSyGen8JzHyn+17lRgIjrlQoVTDw4Y+Dj3D5sQnHvJLEDrT5xBJaulFE=;
Message-ID: <973709.45697.qm@web65903.mail.ac4.yahoo.com>
X-YMail-OSG: aBJPokYVM1lcdPvpX3g1PiTcsGSwgyOdlo92WTzL_skW.3J
 qp69euCTQcERdPU_RnD_TMNUS9SRKKgZGVEWZzZTlZj992o6noWJWsL_04DA
 Cnp.eU6Ve8n_I6msTCy3.oSb2X8xRMzYVfc98X49AET_SSNWK6i7l2Nz0syB
 sCXOAy2KO5DEldI9JyZAylCvKHhCR7efPFvPhBQ.TnX6880BGg7X_OflNjJ9
 6fpRvFI1tz7qYbbhzSClP7cCAu39xmhsN7ev23GNMziwI.mO85LUp5zjDXJl
 6SXdMoQCcoORaX65QfvlwyirClNNdsbqlXTJl_PLTym5wtWF9ZYpBMfM0Sct
 dbTGj_fHsubuA5_ryanbA48iAbBDYIha9YEyJ.Ozc3cadbLZ9D9pwufoSm3r
 LiBwR382m_hBY2g--
Received: from [209.131.62.113] by web65903.mail.ac4.yahoo.com via HTTP; Thu, 05 May 2011 12:45:17 PDT
X-Mailer: YahooMailRC/559 YahooMailWebService/0.8.110.299900
References: <BANLkTimzcTyLoLppo9LxAJuXgzsqM7Ppew@mail.gmail.com> <BANLkTimVpqfrfhN2bq8QhfB_6FrnKyE+6w@mail.gmail.com> <BANLkTinejGbkX1EUaT=mgu9r5weFL=ASZw@mail.gmail.com> <672310.96845.qm@web65907.mail.ac4.yahoo.com> <BANLkTikKe9e9uu6MH5KYqt_s9YwzifQPXA@mail.gmail.com>
Date: Thu, 5 May 2011 12:45:17 -0700 (PDT)
From: "Tsz Wo \(Nicholas\), Sze" <s29752-hadoopgeneral@yahoo.com>
Subject: Re: [DISCUSSION] Release rules
To: general@hadoop.apache.org
In-Reply-To: <BANLkTikKe9e9uu6MH5KYqt_s9YwzifQPXA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1490471232-1304624717=:45697"

--0-1490471232-1304624717=:45697
Content-Type: text/plain; charset=us-ascii

Withdraw my -1.

Eli, thanks for pointing it out about the vote.

Nicholas
  




________________________________
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Sent: Thu, May 5, 2011 11:10:13 AM
Subject: Re: [DISCUSSION] Release rules

On Thu, May 5, 2011 at 10:36 AM, Tsz Wo (Nicholas), Sze
<s29752-hadoopgeneral@yahoo.com> wrote:
>> I don't think the bylaws were checked in, we should do that first. How
>> about checking them into the site repo so they get generated as part
>> of the docs?
>
> -1
>
> Please don't check in anything before having a vote.  Thanks.
>

The project already has bylaws.  Owen proposed them and they passed,
we should check them into svn.

http://mail-archives.apache.org/mod_mbox/hadoop-general/201011.mbox/%3CAANLkTikWUHBedykosKNqPweuOvu2qCKfcf_+HwwivsQa@mail.gmail.com%3E


Thanks,
Eli

--0-1490471232-1304624717=:45697--

From general-return-3400-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 20:57:47 2011
Return-Path: <general-return-3400-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 02F7A277E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 20:57:47 +0000 (UTC)
Received: (qmail 30580 invoked by uid 500); 5 May 2011 20:57:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30530 invoked by uid 500); 5 May 2011 20:57:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30522 invoked by uid 99); 5 May 2011 20:57:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 20:57:45 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jghoman@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 20:57:39 +0000
Received: by wyb40 with SMTP id 40so2997110wyb.35
        for <general@hadoop.apache.org>; Thu, 05 May 2011 13:57:19 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type;
        bh=KrPtbqM9KAKXnwviBZ8T4ml53GogCQJkhuuA3IRx72U=;
        b=MYgFuWyTGwE0fyH04BI8BJ+mZq63mWglXm97LiNjL5Xz9QSrCS10xPKyLPhQj7fqp6
         OgiFuqYufNTTM+Yd+6QgAzqYustXjX5tzNUAspSoiwH9YK4JaJZKun1OfrPeJWwIHU6m
         gJ9agAU3H0J6CdXQM6O/pV01mFH6Fh4QLFM2Y=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=U+7RrR+cx/kDt2DYyZv0oJqvQtXO6RBi7G1Kg9nT3L+dtYrIot8y7T4EENr/kLeMZk
         IxUWs2EjBkBS0XkFC7ptq9cpI1JSqvokWVhMA/L1cYO8BOg65+KneKySoDdJR6PcG7bT
         6+uHDFCiWNfqq/WZE9A7AoqORntz6+YYeOyH0=
Received: by 10.216.203.195 with SMTP id f45mr6597620weo.89.1304629039162;
 Thu, 05 May 2011 13:57:19 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.136.80 with HTTP; Thu, 5 May 2011 13:56:49 -0700 (PDT)
In-Reply-To: <C9E84270.1A05E%sureshms@yahoo-inc.com>
References: <BANLkTi=B+cc_677HC1wRnpcY73zuF8qekg@mail.gmail.com> <C9E84270.1A05E%sureshms@yahoo-inc.com>
From: Jakob Homan <jghoman@gmail.com>
Date: Thu, 5 May 2011 13:56:49 -0700
Message-ID: <BANLkTi=S=9aUXXUmrzmNZTBeXt2eW1Catw@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Downloaded, verified, tested on single node cluster to my
satisfaction.  We've also brought this release up on a sizable cluster
and checked its basic sanity.

Regardless of the difficult path we've had over the past year, this is
a good chunk of code to get out to the community.  I'd much rather
explain a convoluted numbering system or what is or isn't in this
release than continue to apologize for having no release at all.

-Jakob


On Thu, May 5, 2011 at 12:02 PM, Suresh Srinivas <sureshms@yahoo-inc.com> wrote:
>
> +1
>
> Downloaded the release, validated checksums, deployed a single-node
> cluster, and ran some HDFS sanity tests, Web UI tests and mapreduce
> examples.
>
> Regards,
> Suresh
>
>
>
>

From general-return-3401-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 21:11:07 2011
Return-Path: <general-return-3401-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 864205B0
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu,  5 May 2011 21:11:07 +0000 (UTC)
Received: (qmail 49948 invoked by uid 500); 5 May 2011 21:11:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49898 invoked by uid 500); 5 May 2011 21:11:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49889 invoked by uid 99); 5 May 2011 21:11:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:11:05 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [208.97.132.119] (HELO homiemail-a64.g.dreamhost.com) (208.97.132.119)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:10:59 +0000
Received: from homiemail-a64.g.dreamhost.com (localhost [127.0.0.1])
	by homiemail-a64.g.dreamhost.com (Postfix) with ESMTP id 08EA0438080
	for <general@hadoop.apache.org>; Thu,  5 May 2011 14:10:39 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; q=dns; s=
	gbiv.com; b=ZDQZN+CgyE66KuyH7pMLDEfScTRNt/EAWvLL9oWUyyh13PjMe5DS
	qFdbjmZkE0KkEopCTE9oW1GqMPLAIiYs0xPS0ChB2RTilgL66VoM3jbV15xfdI9K
	hmGL/lDPmjKXsSulJAQFDWAceRustHt1r8CgMFY3AOth9OFuXEz7uO0=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; s=gbiv.com;
	 bh=kQ5rDZn05URrqFsRNEVJoLTO4pE=; b=wcgiulDLllF0k3SPZ0pkopfAqefG
	dEK4PCjzrj276CLs5wI+LiFxUWP+Sq+09rk5z9EZLR9KzlqaULnp2oXiOr/Z6dBO
	fYfs7SJ5S3xU80EX1ajqXr5Lt5dWTsGnq3Ho6AZFbosKs9snllBSJQnqN9OjvNAO
	UPnwkpfc6W/ZcQ4=
Received: from [10.134.89.86] (unknown [75.103.10.98])
	(using TLSv1 with cipher AES128-SHA (128/128 bits))
	(No client certificate requested)
	(Authenticated sender: fielding@gbiv.com)
	by homiemail-a64.g.dreamhost.com (Postfix) with ESMTPSA id C919E43807C
	for <general@hadoop.apache.org>; Thu,  5 May 2011 14:10:38 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
From: "Roy T. Fielding" <fielding@gbiv.com>
In-Reply-To: <BANLkTi=nx_ymZHLapyBYdhPdQ53kG40zqg@mail.gmail.com>
Date: Thu, 5 May 2011 14:10:38 -0700
Content-Transfer-Encoding: 7bit
Message-Id: <BD23CCCD-6B2F-4C58-AF9C-F34DD12D7CF2@gbiv.com>
References: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org> <BANLkTikH4PazAADyo8+=3d3v0bUHaS24HQ@mail.gmail.com> <3116A002-6EA6-4971-B7AD-261BCC53D76B@gbiv.com> <BANLkTi=nx_ymZHLapyBYdhPdQ53kG40zqg@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)

On May 4, 2011, at 11:52 PM, Jean-Daniel Cryans wrote:

> Roy,
> 
> On Wed, May 4, 2011 at 7:22 PM, Roy T. Fielding <fielding@gbiv.com> wrote:
>> The ASF is a vehicle for whomever wishes to collaborate on a
>> given project.  Collaboration means helping do the work.  Those
>> who do the work may do so for whatever reasons that they think
>> are good, whether it is because they feel like being charitable
>> today, they get paid a salary and the big boss said "work on
>> this part", or because they just have an itch worth scratching.
>> 
>> Apache does not care why people choose to collaborate or
>> how they choose to apply their own intellectual efforts.  We
>> welcome all forms of contribution under the terms of our license.
> 
> I don't think I was arguing against the contribution of the code in
> that branch, it's very welcome, but I'm questioning (and ranting
> about) the motivation for releasing a version that even just by name
> is a weird hulla-hoop around the usual development practices that
> Hadoop has had in the past (not that it's set in stone).

Yes, and I said that kind of questioning is not appropriate.
You are not responsible for other peoples' motivation.

> So I wanted to contribute my negative non-binding vote to highlight
> that this release is probably very confusing for the general user.
> This is 0.20, but it's not. Also it has more numbers, and it starts at
> 203. Why doing this at all instead of just moving on with 0.22? Or is
> 0.22 bound to be like 0.21? It almost begs the question if this should
> be called 0.22.0 then.

Yes, I already made that same point.  You don't need to talk about
motivation in order to do so.

If I had a vote, I would have voted -1 just because version numbers
do matter to users, the three number form is well-known, and minting
new versions is far cheaper than adding extra numbers.  I'd have cut
the release candidate as 0.30.  However, I did not do that work.
The person who did the work chose 0.20.203.0.  Anyone who doesn't
like that should vote accordingly and, preferably, make your
communication about such things more open in the future so
that you don't waste others' time on extra builds.  And if the
majority thinks releasing these bits are more important than my
concerns, then I have to accept that as the will of the project.

Please note also that policies are not technical discussions.
Likewise, version numbers are not technical. If those were
technical changes then anyone on the PMC could veto them,
which would effectively mean anyone could veto a release
and the project would quickly devolve into tyranny by minority.

Likewise, just because I said that a successful release defines
its own set of precedents (and therein policy), that doesn't
mean the project can't vote on a new policy the next day or
make another release that sets it again moving forward.
Progress is in the doing.

....Roy

From general-return-3402-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 00:42:47 2011
Return-Path: <general-return-3402-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0A28F2B78
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri,  6 May 2011 00:42:47 +0000 (UTC)
Received: (qmail 95670 invoked by uid 500); 6 May 2011 00:42:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95602 invoked by uid 500); 6 May 2011 00:42:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95594 invoked by uid 99); 6 May 2011 00:42:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 00:42:45 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [98.139.52.238] (HELO nm16-vm0.bullet.mail.ac4.yahoo.com) (98.139.52.238)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 06 May 2011 00:42:36 +0000
Received: from [98.139.52.190] by nm16.bullet.mail.ac4.yahoo.com with NNFMP; 06 May 2011 00:42:15 -0000
Received: from [98.139.52.144] by tm3.bullet.mail.ac4.yahoo.com with NNFMP; 06 May 2011 00:42:15 -0000
Received: from [127.0.0.1] by omp1027.mail.ac4.yahoo.com with NNFMP; 06 May 2011 00:42:15 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 211734.78341.bm@omp1027.mail.ac4.yahoo.com
Received: (qmail 91314 invoked by uid 60001); 6 May 2011 00:42:15 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1304642535; bh=X9DbxpfeR2fPYeKDE8fJLjUrRIwjBXfwGxKGCtZ20mk=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=cHIPgILjmkt+sTZQuprdu0GSFEWVUbhYlwHZmDfZDUYuNcHs+qXhGowHjB2uFO/yt4MUxN3ffY+rf8Z7QDfJaOE/ab3jQ5bjQwxDvnUMixs8zoDjd3ZdNH2fLSAQoxmuxvmWdBotXxr6sCF0AzoDRz6ZjV2zjvR7XXhCpdVAxj4=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=jrXEUioIubRFHhs1vLGCYpEDsw0KYtsBHKf19XVrgayrl5S9t7MxUsVAvU2hqh0kD3c1V1mEfByJu0nniii1xvcqWLNah5Uyg8WI2/qMG+EICbzT1NQviXv5eCKEZaHNkiDDqtl9gAPSw9EoT/2bgW9EdPoeu8gyr2B2fsXR+ks=;
Message-ID: <95165.87987.qm@web65908.mail.ac4.yahoo.com>
X-YMail-OSG: Yq1k6pkVM1ko1iltuLSogj7Ns5MnvKs28XK4vNRKG8mi6M6
 12iZYA6qfk4p6MJgj4I2zbDKoseRUgpa0XdqeJBhGxO4W6TmnurYn5UPz7KS
 uyFvgauzWRhm4W_5WAHeg7eb7ECM7YwZZc28rIvT.FjvH4RuFeSIblqHljUU
 k3Ya_gkvLHN7Q.KAvUIxl9r_G1OJ.KZGOrQL1GAKa8pwsGM1Fbx_R9nm8UR7
 A4dlfV.H1JoMolbWwRnQjh88uWENS4wPNc_I60_m0V6MSSyobGlBQRQrlaLJ
 TRA54QaIa5hRqY6GRpKQ4K9DxKY_J9220dKTnVL24sGJF0lD8A3ifOooGN74
 ASsqhJd464fKxClpbTfviWNJMqo6cNOL3e6R.UU8AuoTvXL172hjkYBWcYS5
 lleabX6N11as7VQ--
Received: from [209.131.62.113] by web65908.mail.ac4.yahoo.com via HTTP; Thu, 05 May 2011 17:42:14 PDT
X-Mailer: YahooMailRC/559 YahooMailWebService/0.8.110.299900
Date: Thu, 5 May 2011 17:42:14 -0700 (PDT)
From: "Tsz Wo \(Nicholas\), Sze" <s29752-hadoopgeneral@yahoo.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-2097846283-1304642534=:87987"

--0-2097846283-1304642534=:87987
Content-Type: text/plain; charset=us-ascii

+1 on 0.20.203.0-rc1.

I downloaded the release, verified signature, verified some message digests, 
started a one-node cluster and tested it with my MapReduce Math Library.  In 
particular, I have run

*DistMpMultTest*
It runs 5 jobs:
- 2 forward FFT jobs for calculating two 4086-dimensional DFTs.
- 1 backward FFT job for calculating componentwise multiplication using a c 
program with GMP lib, and a 4086-dimensional inverse DFTs.
- 1 componentwise summation job.
- 1 job for carrying.

*DistBbp*
It runs 14 jobs for computing the 1024 bits of pi right after the billionth bit 
positions.

Please see the details below.

Regards,
Nicholas


****** DistMpMultTest ******************************************************

distfft.VERSION = 20110124
distmpmult.VERSION = 20110307
distmpmult.LOCAL_THRESHOLD = 2^16 (=65536)

args = [#=2
  0: 18
  1: gen
]
dir              = DistMpMultTest-20110505-222621889
NEW: Zahlen(bitsPerDigit=32, digitsPerArray=4096, numArrays=128)

SchonhageStrassen.Factory.valueOf(numDigits=262144): 
  digitsPerOperand = 2^18 (=262144) (highest=262144)
  bitsPerElement   = 2^12 (=4096)
  D                = 2^12 (=4096)
  modulusExponent  = 10240 (ss_e=8224)
  efficiency       = 0.801171875 (N=2^24 (=16777216), n=10240)
NEW: SchonhageStrassen[modulas=2^10240 + 1, D=2^12 (=4096), bitsPerElement=2^12 
(=4096), Z=Zahlen(bitsPerDigit=2^5, digitLimit=2^32, digitsPerArray=2^12, 
numArrays=128)]
NEW: Zahlen(bitsPerDigit=32, digitsPerArray=4096, numArrays=1)
NEW: SchonhageStrassen[modulas=2^10240 + 1, D=2^12 (=4096), bitsPerElement=2^12 
(=4096), Z=Zahlen(bitsPerDigit=2^5, digitLimit=2^32, digitsPerArray=2^12, 
numArrays=1)]

numDigits        = 2^18 (=262144) (e=18)
digitsPerOperand = 262144
J                = 2^6 (=64)
K                = 2^6 (=64)

Verifier: STARTED
WorkGroup: Created DistMpMultTest with 2 threads.

multiply_mapreduce: c; x = [a, b]
  distfft: forward
    inverse           = false
    distmpbase: 
      Z                 = Zahlen
        bitsPerDigit    = 2^5 (=32)
        digitLimit      = 2^32 (=4294967296)
        digitsPerArray  = 2^12 (=4096)
        numArrays       = 1
        digitsSupported = 4096
      schonhagestrassen = SchonhageStrassen
        D              = 2^12 (=4096)
        D^(-1)         = -2^10228
        modulus        = 2^10240 + 1
        bitsPerElement = 2^12 (=4096)
        zeta           = [2^0, 2^5, 2^10, 2^15, 2^20, 2^25, 2^30, 2^35, 2^40, 
2^45, ...]
      J                 = 2^6 (=64)
      K                 = 2^6 (=64)
      dir               = DistMpMultTest-20110505-222621889
  digitsPerOperand = 262144
  descriptor = FunctionDescriptor(input=a, output=a')
  descriptor = FunctionDescriptor(input=b, output=b')
  forward a: DistFft(J=2^6 (=64), K=2^6 (=64), 
dir=DistMpMultTest-20110505-222621889, inverse=false)
  GmpMultiplier(Thread-260, count=0): cmd=[./gmp_mult, 16]
  GmpMultiplier(Thread-260, count=1): messager started.
  GmpMultiplier(Thread-260, count=1): 
  GmpMultiplier(Thread-260, count=1): START: Thu May  5 22:26:28 2011
  GmpMultiplier(Thread-260, count=1): VERSION=20110122b, GMP4.1.4
  GmpMultiplier(Thread-260, count=1): 
  GmpMultiplier(Thread-260, count=1): argc=2, argv=[./gmp_mult, 16]
  SUBMIT JOB: DistMpMultTest-20110505-222621889: a' = dft(a)
  forward b: DistFft(J=2^6 (=64), K=2^6 (=64), 
dir=DistMpMultTest-20110505-222621889, inverse=false)
  SUBMIT JOB: DistMpMultTest-20110505-222621889: b' = dft(b)
  Verifier: expected = + 0129ED2D B8CF7E1D A9BF33D8 9A294511 8508413C ...  
4EA85F30 EF79A552 1C080D0C 82F6882C 8CE9F355 (524,288 digits, 16,777,209 bits)
  descriptor = FunctionDescriptor(input=a' ** b', output=c')
  backward: DistFft(J=2^6 (=64), K=2^6 (=64), 
dir=DistMpMultTest-20110505-222621889, inverse=true)
  SUBMIT JOB: DistMpMultTest-20110505-222621889: c' = dft^-1(a' ** b')
          2272875ms (=37:52.875)     : backward, 
DistMpMultTest-20110505-222621889: c' = dft^-1(a' ** b')
  -----------------------------------------------------
  descriptor = FunctionDescriptor(input=c'_0 + c'_1 + c'_2, output=c'')
  summation: DistMpSum(J=2^6 (=64), K=2^6 (=64), 
dir=DistMpMultTest-20110505-222621889/c')
  SUBMIT JOB: DistMpMultTest-20110505-222621889/c': c'' = c'_0 + c'_1 + c'_2
          2805177ms (=46:45.177)     : summation, 
DistMpMultTest-20110505-222621889/c': c'' = c'_0 + c'_1 + c'_2
  -----------------------------------------------------
  descriptor = FunctionDescriptor(input=c'', output=c)
  carrying: DistCarrying(J=2^6 (=64), K=2^6 (=64), 
dir=DistMpMultTest-20110505-222621889/c')
  SUBMIT JOB: DistMpMultTest-20110505-222621889/c': c = carry(c'')
          3456765ms (=57:36.765)     : carrying, 
DistMpMultTest-20110505-222621889/c': c = carry(c'')
  -----------------------------------------------------
multiply_mapreduce: returns ZahlenDescriptor(numParts=2^6 (=64), 
elementsPerPart=64)
c = ZahlenDescriptor(numParts=2^6 (=64), elementsPerPart=64)
        3456769ms (=57:36.769)     : DistMpMult

           maxMemory = 963.00 MB
         totalMemory = 64.88 MB
          freeMemory = 26.32 MB

computed = + 0129ED2D B8CF7E1D A9BF33D8 9A294511 8508413C ...  4EA85F30 EF79A552 
1C080D0C 82F6882C 8CE9F355 (524,288 digits, 16,777,209 bits)
        3457374ms (=57:37.374)     : DONE (dir = 
DistMpMultTest-20110505-222621889)




****** DistBbp *************************************************************
Create file b1000000000-p1024-20110506-000707662.log

STARTUP Fri May 06 00:07:07 UTC 2011
  Started at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
  Printed at org.apache.hadoop.mp.pi.DistSum.<init>(DistSum.java:428)
WorkGroup: Created DistSum.OutputProcessor with 1 threads.
DistBbp.VERSION = 20100731

b = 1,000,000,000 (bits skipped)

precision = 1,024
nWorkers  = 14
nJobs     = 1
machine   = MapSide(p1,t1), m1t1
remoteDir = 100b-1024-test
tmpDir    = ../100b-1024-test


DistBbp.processExistingJobOutputs = true

Check existing job outputs from hdfs://NAMENODE/user/100b-1024-test ...

Read existing results from hdfs://NAMENODE/user/tsz/100b-1024-test ...
DistBbp.bellard = null
ADD : P8_1: -[n:value=1,delta=8,limit=400000393; 
e:value=999999999,delta=-20,limit=-996], parts.length=1
ADD : P8_3: -[n:value=3,delta=8,limit=400000395; 
e:value=999999994,delta=-20,limit=-996], parts.length=1
ADD : P8_5: +[n:value=5,delta=8,limit=400000397; 
e:value=999999989,delta=-20,limit=-996], parts.length=1
ADD : P8_7: +[n:value=7,delta=8,limit=400000399; 
e:value=999999984,delta=-20,limit=-996], parts.length=1
ADD : P20_21: +[n:value=21,delta=20,limit=1000000981; 
e:value=999999982,delta=-20,limit=-995], parts.length=1
ADD : P20_3: -[n:value=3,delta=20,limit=1000000983; 
e:value=1000000000,delta=-20,limit=-995], parts.length=1
ADD : P20_5: -[n:value=5,delta=20,limit=1000000985; 
e:value=999999996,delta=-20,limit=-995], parts.length=1
ADD : P20_7: -[n:value=7,delta=20,limit=1000000987; 
e:value=999999996,delta=-20,limit=-995], parts.length=1
ADD : P20_9: +[n:value=9,delta=20,limit=1000000989; 
e:value=999999994,delta=-20,limit=-995], parts.length=1
ADD : P20_11: -[n:value=11,delta=20,limit=1000000991; 
e:value=999999992,delta=-20,limit=-995], parts.length=1
ADD : P20_13: +[n:value=13,delta=20,limit=1000000993; 
e:value=999999990,delta=-20,limit=-995], parts.length=1
ADD : P20_15: +[n:value=15,delta=20,limit=1000000995; 
e:value=999999986,delta=-20,limit=-995], parts.length=1
ADD : P20_17: +[n:value=17,delta=20,limit=1000000997; 
e:value=999999986,delta=-20,limit=-995], parts.length=1
ADD : P20_19: -[n:value=19,delta=20,limit=1000000979; 
e:value=999999984,delta=-20,limit=-995], parts.length=1
           1102ms (=1.102s)        : EXECUTOR: 14 computation(s)
           1298ms (=1.298s)        : P20_5.job0006-20110506-000707683> starting, 
steps/cores = 50000049/1 = 50,000,049, ++nSubmittedJobs=1
P20_5.job0006-20110506-000707683> sleep(5s)
           7667ms (=7.667s)        : P20_15.job0011-20110506-000707683> 
starting, steps/cores = 50000049/1 = 50,000,049, ++nSubmittedJobs=2
P20_15.job0011-20110506-000707683> sleep(5s)
          13327ms (=13.327s)       : P8_7.job0003-20110506-000707683> starting, 
steps/cores = 50000049/1 = 50,000,049, ++nSubmittedJobs=3
P8_7.job0003-20110506-000707683> sleep(5s)
          18670ms (=18.670s)       : P20_3.job0005-20110506-000707683> starting, 
steps/cores = 50000049/1 = 50,000,049, ++nSubmittedJobs=4
P20_3.job0005-20110506-000707683> sleep(5s)
          24281ms (=24.281s)       : P8_5.job0002-20110506-000707683> starting, 
steps/cores = 50000049/1 = 50,000,049, ++nSubmittedJobs=5
P8_5.job0002-20110506-000707683> sleep(5s)
          29693ms (=29.693s)       : P8_1.job0000-20110506-000707683> starting, 
steps/cores = 50000049/1 = 50,000,049, ++nSubmittedJobs=6
P8_1.job0000-20110506-000707683> sleep(5s)
          34917ms (=34.917s)       : P20_19.job0013-20110506-000707683> 
starting, steps/cores = 50000048/1 = 50,000,048, ++nSubmittedJobs=7
P20_19.job0013-20110506-000707683> sleep(5s)
          40135ms (=40.135s)       : P20_21.job0004-20110506-000707683> 
starting, steps/cores = 50000048/1 = 50,000,048, ++nSubmittedJobs=8
P20_21.job0004-20110506-000707683> sleep(5s)
          45458ms (=45.458s)       : P20_13.job0010-20110506-000707683> 
starting, steps/cores = 50000049/1 = 50,000,049, ++nSubmittedJobs=9
P20_13.job0010-20110506-000707683> sleep(5s)
          50666ms (=50.666s)       : P20_11.job0009-20110506-000707683> 
starting, steps/cores = 50000049/1 = 50,000,049, ++nSubmittedJobs=10
P20_11.job0009-20110506-000707683> sleep(5s)
          55942ms (=55.942s)       : P20_7.job0007-20110506-000707683> starting, 
steps/cores = 50000049/1 = 50,000,049, ++nSubmittedJobs=11
P20_7.job0007-20110506-000707683> sleep(5s)
          61261ms (=1:01.261)      : P20_9.job0008-20110506-000707683> starting, 
steps/cores = 50000049/1 = 50,000,049, ++nSubmittedJobs=12
P20_9.job0008-20110506-000707683> sleep(5s)
          66577ms (=1:06.577)      : P20_17.job0012-20110506-000707683> 
starting, steps/cores = 50000049/1 = 50,000,049, ++nSubmittedJobs=13
P20_17.job0012-20110506-000707683> sleep(5s)
          71752ms (=1:11.752)      : P8_3.job0001-20110506-000707683> starting, 
steps/cores = 50000049/1 = 50,000,049, ++nSubmittedJobs=14
P8_3.job0001-20110506-000707683> sleep(5s)
         187749ms (=3:07.749)      : P20_5.job0006-20110506-000707683> 
timetaken=3:06.451, --nSubmittedJobs=13
         187753ms (=3:07.753)      : ++DistSum.OutputProcessor.count=1 for 
P20_5.job0006-20110506-000707683
> found 3 items in 
>hdfs://NAMENODE/user/100b-1024-test/P20_5.job0006-20110506-000707683/out
> write the result to 100b-1024-test/P20_5.job0006-20110506-000707683.writable
DistSum P20_5.job0006-20110506-000707683> duration=142210(2:22.210), 
sigma=-[n:value=5,delta=20,limit=1000000985; 
e:value=999999996,delta=-20,limit=-995], value=[32:7AB75B81 4B81A170 48F78314 
C1B677EF 6D8AA841 77D75680 AE63063C 378B6899 DBF6BB41 113B3C9E F6ADC215 2AA38330 
14796B10 16862533 C17A6A17 4893ED2C 23570119 471AADFC 493994FA CBA6DBF4 C60A3E46 
48E27ADC 6B2A442E 797C4011 40FCA6AF FE97AFAF 3FF6CB94 532F818F E2EA5E7F 7BC2FADF 
83B2F466 07F9A5C7 ]
         187804ms (=3:07.804)      : --DistSum.OutputProcessor.count=0 for 
P20_5.job0006-20110506-000707683
EXECUTOR: 1/14 done, estimated time: remaining=40.94 minutes, total=44.09 
minutes, sleep=2.300s
         193403ms (=3:13.403)      : P20_15.job0011-20110506-000707683> 
timetaken=3:05.735, --nSubmittedJobs=12
         193403ms (=3:13.403)      : ++DistSum.OutputProcessor.count=1 for 
P20_15.job0011-20110506-000707683
> found 3 items in 
>hdfs://NAMENODE/user/100b-1024-test/P20_15.job0011-20110506-000707683/out
> write the result to 100b-1024-test/P20_15.job0011-20110506-000707683.writable
DistSum P20_15.job0011-20110506-000707683> duration=141830(2:21.830), 
sigma=+[n:value=15,delta=20,limit=1000000995; 
e:value=999999986,delta=-20,limit=-995], value=[32:FDC65838 CB87FD4C 0C6883F1 
F6E7A8F4 903A3888 6C377451 138823A3 9B8AFB5D F1F60326 F5300E57 DFDDF086 83309616 
8FBB61F1 C3D2CC7A 5414D92E 3A27B56C 305F6A64 BD1BE14D CDDE3433 AD57BF9C C3CCCF98 
BB7AF093 A42E5EB8 7182B1A0 09058BF0 5C2EF773 A8B4D621 ADD9C960 32272AAD 0B060FFA 
D0932CC6 ACAF0A05 ]
         193419ms (=3:13.419)      : --DistSum.OutputProcessor.count=0 for 
P20_15.job0011-20110506-000707683
EXECUTOR: 2/14 done, estimated time: remaining=19.36 minutes, total=22.58 
minutes, sleep=2.200s
         338790ms (=5:38.790)      : P8_7.job0003-20110506-000707683> 
timetaken=5:25.463, --nSubmittedJobs=11


...


        1087176ms (=18:07.176)     : --DistSum.OutputProcessor.count=0 for 
P20_17.job0012-20110506-000707683
EXECUTOR: 13/14 done, estimated time: remaining=1.39 minutes, total=19.52 
minutes, sleep=1.100s
        1097443ms (=18:17.443)     : P8_3.job0001-20110506-000707683> 
timetaken=17:05.691, --nSubmittedJobs=0
        1097443ms (=18:17.443)     : ++DistSum.OutputProcessor.count=1 for 
P8_3.job0001-20110506-000707683
> found 3 items in 
>hdfs://NAMENODE/user/100b-1024-test/P8_3.job0001-20110506-000707683/out
> write the result to 100b-1024-test/P8_3.job0001-20110506-000707683.writable
DistSum P8_3.job0001-20110506-000707683> duration=142280(2:22.280), 
sigma=-[n:value=3,delta=8,limit=400000395; 
e:value=999999994,delta=-20,limit=-996], value=[32:9850C816 183C9765 11830320 
A12AB99B 81D93E98 061769A9 0FBBCAF7 3AEE5079 5673D8FA 0A352C71 C025AC12 5D482AC1 
276E834F C862BF85 D9B79DE5 1264B724 A5C5FD90 2677498B DDF05B2B D298E033 9AB3DC52 
2A3DB5AA 2835506D DF795BCB 743CC0FE 27AA4F1F 1F711BAB 52E4F4E6 5410CE20 CD27ACB6 
6C7F6810 95F97608 ]
        1097461ms (=18:17.461)     : --DistSum.OutputProcessor.count=0 for 
P8_3.job0001-20110506-000707683
EXECUTOR: 14/14 done, estimated time: remaining=0.00 ms, total=18.31 minutes, 
sleep=1s
Write to hdfs://NAMENODE/user/tsz/100b-1024-test/b1000000000-p1024 ...

b = 1,000,000,000 (bits skipped)

CPU time = 1991670ms = 33:11.670
END Fri May 06 00:25:26 UTC 2011
[32:
  3E08FF2B 03F1829E 05C038A3 2884A9C4 E7DEF417 875B1C22 D2DDDA11 D99573D8 
43F80107 AB3A56CC 

  C975C849 2DC5AD43 1D191704 9064DE41 67DE5C6E 0F264D33 D903CE49 F2324781 
D9D7ED45 FAA9272D 

  CAA42278 E8EF2FFF 450E2183 25FA9AB3 459D01AF AEF8AF79 DD6CC766 12152C31 
2F31ABCD DCF01DEC 

  67332105 643A438D 
]
****** END ******************************************************

--0-2097846283-1304642534=:87987--

From general-return-3403-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 10:08:54 2011
Return-Path: <general-return-3403-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E1B97381C
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri,  6 May 2011 10:08:53 +0000 (UTC)
Received: (qmail 27403 invoked by uid 500); 6 May 2011 10:08:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27317 invoked by uid 500); 6 May 2011 10:08:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27309 invoked by uid 99); 6 May 2011 10:08:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 10:08:51 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of lars.francke@gmail.com designates 209.85.220.176 as permitted sender)
Received: from [209.85.220.176] (HELO mail-vx0-f176.google.com) (209.85.220.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 10:08:45 +0000
Received: by vxa37 with SMTP id 37so5216490vxa.35
        for <general@hadoop.apache.org>; Fri, 06 May 2011 03:08:24 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type:content-transfer-encoding;
        bh=FUiiej8iqUeFP/KO4F9XhdvRg6zOlX5zZ/VorlW3OYg=;
        b=b4zxRwL3txeEuxfMcnEOJdsv9Fg21gDXwIfHOkjrQQvk56XvuovmD1Hrl4SlF51jmm
         zHbjppSxhxz+r8/uh2Cp7AmnyzGOQ876Lg4hh6WvjR7lcG0+vAEwUJCuXDk3r+xvZmX9
         4Jrqh1QTZiJHMLwcFYxq+/SF/qjwzjIfPvoZE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=ajdCpjBjT7TwBOFQZEIa6B0/LLgxIYtGVEblXsneOIkXa2bUa2NHn7DTbmGyI3rnD0
         /eIIk/267QsrSgqfDR8lxsnpx7d9PPAGcuZUcAb7zfNkhzmzXQnxhBdem7B0UjXmAIAD
         Eqmu0UK4bfnCxaVLrpbfC92nKotV+KXh1XBTo=
Received: by 10.52.98.227 with SMTP id el3mr1090276vdb.244.1304676504107; Fri,
 06 May 2011 03:08:24 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.52.115.133 with HTTP; Fri, 6 May 2011 03:07:44 -0700 (PDT)
In-Reply-To: <BANLkTi=aW6Toxtg7g=nFSV7h4gK68522wg@mail.gmail.com>
References: <BANLkTik5gj0H4gmvy14Ux=g_r5Tt6D1=MA@mail.gmail.com>
 <C9E71BE8.19FCF%sureshms@yahoo-inc.com> <BANLkTikrS9PTeErT15fs2yv+N4_wpPN7xQ@mail.gmail.com>
 <BANLkTimjxRdf6PyTnbZUwrS_znZo00z-5w@mail.gmail.com> <BANLkTi=pNF_aQrWPurfXRbjNdshfseNobg@mail.gmail.com>
 <321761.51699.qm@web65903.mail.ac4.yahoo.com> <CB238EE8-1E06-4C6C-A7F5-CB00E2E073CB@yahoo-inc.com>
 <BANLkTi=aW6Toxtg7g=nFSV7h4gK68522wg@mail.gmail.com>
From: Lars Francke <lars.francke@gmail.com>
Date: Fri, 6 May 2011 12:07:44 +0200
Message-ID: <BANLkTikT2zpd-k3DgmwXwm4vXy4KpnefxQ@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

> " =C2=A0 BZ-4182948. Add statistics logging to Fred for better visibility=
 into
> startup time costs. (Matt Foley)"
> - I believe I saw a note from Matt on the JIRA yesterday about this featu=
re,
> where he decided that the version done in 203 wasn't a good approach, and
> it's done differently in trunk (not sure if done yet).

Could anyone elaborate on what this "Fred" is that has been coming up
on these threads a few times now?

And is there something like a RELEASE NOTES draft that I could look
over? I try to follow these mailing lists as best as I can but I have
lost track of all the branches and features being worked on and I
can't imagine I'm the only one.

It would be nice to get an overview of what this release is all about
where work is being done etc.

Thanks!

Cheers,
Lars

From general-return-3404-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 11:53:15 2011
Return-Path: <general-return-3404-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 32B6A31A3
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri,  6 May 2011 11:53:15 +0000 (UTC)
Received: (qmail 55994 invoked by uid 500); 6 May 2011 11:53:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55940 invoked by uid 500); 6 May 2011 11:53:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55932 invoked by uid 99); 6 May 2011 11:53:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 11:53:13 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 11:53:04 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 4CB741BA94F
	for <general@hadoop.apache.org>; Fri,  6 May 2011 12:52:43 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id vbglCahS5bbT for <general@hadoop.apache.org>;
	Fri,  6 May 2011 12:52:43 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by colossus.hpl.hp.com (Postfix) with ESMTP id CDEB81BA401
	for <general@hadoop.apache.org>; Fri,  6 May 2011 12:52:42 +0100 (BST)
MailScanner-NULL-Check: 1305287551.10042@rkrJvcpTtrxiNl7C3UPJUQ
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p46BqU1B013372
	for <general@hadoop.apache.org>; Fri, 6 May 2011 12:52:30 +0100 (BST)
Message-ID: <4DC3E0FE.9060207@apache.org>
Date: Fri, 06 May 2011 12:52:30 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.14) Gecko/20110223 Thunderbird/3.1.8
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
References: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org> <BANLkTikH4PazAADyo8+=3d3v0bUHaS24HQ@mail.gmail.com> <3116A002-6EA6-4971-B7AD-261BCC53D76B@gbiv.com> <9234135B-FD91-4E92-B4F5-6E6F7EB38A94@Holsman.NET>
In-Reply-To: <9234135B-FD91-4E92-B4F5-6E6F7EB38A94@Holsman.NET>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p46BqU1B013372
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org


Vote: +1

I'm a committer not PMC, so it's non binding.


Why:
-looks and works OK on my desktop
-we've been using Y! releases, and this brings their branch back into 
the apache fold, meaning we can say "the official Apache release of 
Apache Hadoop is something you can use in production".
-I'm confident that the Y! team have tested this well.

I understand Eli's concerns that putting stuff in there that hasn't gone 
into trunk yet is danger. However, as the team makes no guarantees of 
100% compatibility between releases, I don't think it's critical. It's 
just something that needs to be addressed -which can be done after this 
release has shipped.


-Steve


From general-return-3405-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 12:45:39 2011
Return-Path: <general-return-3405-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C5EA239D5
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri,  6 May 2011 12:45:39 +0000 (UTC)
Received: (qmail 23752 invoked by uid 500); 6 May 2011 12:45:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23702 invoked by uid 500); 6 May 2011 12:45:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23694 invoked by uid 99); 6 May 2011 12:45:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 12:45:38 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 12:45:30 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 6A910B7F7D
	for <general@hadoop.apache.org>; Fri,  6 May 2011 13:45:09 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id dYpso4g29yNt for <general@hadoop.apache.org>;
	Fri,  6 May 2011 13:45:08 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 9DFC7B7F68
	for <general@hadoop.apache.org>; Fri,  6 May 2011 13:45:08 +0100 (BST)
MailScanner-NULL-Check: 1305290696.63219@pkpaUvQuS3CfHinNLcGT3g
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p46Ciu64014722
	for <general@hadoop.apache.org>; Fri, 6 May 2011 13:44:56 +0100 (BST)
Message-ID: <4DC3ED48.6000702@apache.org>
Date: Fri, 06 May 2011 13:44:56 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.14) Gecko/20110223 Thunderbird/3.1.8
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSSION] development process of Hadoop
References: <4DC28B79.7000100@apache.org> <C9E82D2A.118AD%eyang@yahoo-inc.com> <BANLkTin_1YK4a39jVtNOdG_bucK6MQhHEw@mail.gmail.com>
In-Reply-To: <BANLkTin_1YK4a39jVtNOdG_bucK6MQhHEw@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p46Ciu64014722
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 05/05/11 18:52, Todd Lipcon wrote:
> On Thu, May 5, 2011 at 10:32 AM, Eric Yang<eyang@yahoo-inc.com>  wrote:
>
>> Git is powerful in maintaining different branch of the source code.
>>   However, it will only work if the entire community is willing to move to
>> git.  Maintaining svn and git hybrid, is a time consuming task that we are
>> paying in full price.  Hadoop community should work smarter for the source
>> control.  What do people think about fully adopting git instead of svn?
>>
>
> +1 for Git as a tool. But using git makes it even _more_ important that we
> have a clearly defined release process that outlines which branches are
> meant to be released as official artifacts, and what the inclusion criteria
> for those branches should be.
>

I'm +0.9995 for git: some bits I like, some bits I don't (it's awful for 
binary data). And you need more than just a release process locked down, 
you need the developers understanding following a good process. If you 
have written down process docs there, I'd love to see them.

apache infrastructure are discussing git -what would be best would be to 
start with a non-critical project, such as one or more of the moved 
contrib projects (like MR-Unit), so we can see that it, gerrit, etc work 
well within the Hadoop developer world.




From general-return-3406-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 13:45:22 2011
Return-Path: <general-return-3406-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2F4263BBF
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri,  6 May 2011 13:45:22 +0000 (UTC)
Received: (qmail 29749 invoked by uid 500); 6 May 2011 13:45:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29700 invoked by uid 500); 6 May 2011 13:45:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29692 invoked by uid 99); 6 May 2011 13:45:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 13:45:20 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mlortiz@uci.cu designates 200.55.140.180 as permitted sender)
Received: from [200.55.140.180] (HELO mx3.uci.cu) (200.55.140.180)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 06 May 2011 13:45:16 +0000
Received: (qmail 27167 invoked by uid 507); 6 May 2011 13:44:48 -0000
Received: from 10.0.0.185 by ns3.uci.cu (envelope-from <mlortiz@uci.cu>, uid 501) with qmail-scanner-2.01st 
 (avp: 5.0.2.0. spamassassin: 3.0.6. perlscan: 2.01st.  
 Clear:RC:1(10.0.0.185):. 
 Processed in 0.639064 secs); 06 May 2011 13:44:48 -0000
Received: from unknown (HELO ucimail4.uci.cu) (10.0.0.185)
  by 0 with SMTP; 6 May 2011 13:44:48 -0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by ucimail4.uci.cu (Postfix) with ESMTP id C5DA313243A1;
	Fri,  6 May 2011 09:44:47 -0400 (CDT)
X-Virus-Scanned: amavisd-new at uci.cu
Received: from ucimail4.uci.cu ([127.0.0.1])
	by localhost (ucimail4.uci.cu [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id 5pRb0mtCiatk; Fri,  6 May 2011 09:44:47 -0400 (CDT)
Received: from [10.36.18.29] (marcosluis-aspire-5251.uci.cu [10.36.18.29])
	by ucimail4.uci.cu (Postfix) with ESMTP id 539231324371;
	Fri,  6 May 2011 09:44:47 -0400 (CDT)
Message-ID: <4DC402B7.8070900@uci.cu>
Date: Fri, 06 May 2011 09:46:23 -0430
From: Marcos Ortiz <mlortiz@uci.cu>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.14) Gecko/20110223 Lightning/1.0b2 Thunderbird/3.1.8
MIME-Version: 1.0
To: general@hadoop.apache.org
CC: Steve Loughran <stevel@apache.org>
Subject: Re: [DISCUSSION] development process of Hadoop
References: <4DC28B79.7000100@apache.org> <C9E82D2A.118AD%eyang@yahoo-inc.com> <BANLkTin_1YK4a39jVtNOdG_bucK6MQhHEw@mail.gmail.com> <4DC3ED48.6000702@apache.org>
In-Reply-To: <4DC3ED48.6000702@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit

On 05/06/2011 08:14 AM, Steve Loughran wrote:
> On 05/05/11 18:52, Todd Lipcon wrote:
>> On Thu, May 5, 2011 at 10:32 AM, Eric Yang<eyang@yahoo-inc.com>  wrote:
>>
>>> Git is powerful in maintaining different branch of the source code.
>>>   However, it will only work if the entire community is willing to 
>>> move to
>>> git.  Maintaining svn and git hybrid, is a time consuming task that 
>>> we are
>>> paying in full price.  Hadoop community should work smarter for the 
>>> source
>>> control.  What do people think about fully adopting git instead of svn?
>>>
>>
>> +1 for Git as a tool. But using git makes it even _more_ important 
>> that we
>> have a clearly defined release process that outlines which branches are
>> meant to be released as official artifacts, and what the inclusion 
>> criteria
>> for those branches should be.
>>
>
> I'm +0.9995 for git: some bits I like, some bits I don't (it's awful 
> for binary data). And you need more than just a release process locked 
> down, you need the developers understanding following a good process. 
> If you have written down process docs there, I'd love to see them.
>
> apache infrastructure are discussing git -what would be best would be 
> to start with a non-critical project, such as one or more of the moved 
> contrib projects (like MR-Unit), so we can see that it, gerrit, etc 
> work well within the Hadoop developer world.
>
>
>
+1 for Git

We migrated from SVN to Git for our completed infrastructure, for many 
reason:
- Git use much less space than SVN, all the changes are in a single .git
- Git is awesome for branching
- Another great advantage is that there are many developers that know 
Git, and how the development process can be greatly improved.

PostgreSQL, one of my favorites open source projects that I use on my 
daily work, migrated the development process to Git from CVS.

Regards.

-- 
Marcos Luís Ortíz Valmaseda
  Software Engineer (Large-Scaled Distributed Systems)
  University of Information Sciences,
  La Habana, Cuba
  Linux User # 418229
  http://about.me/marcosortiz


From general-return-3407-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 16:48:52 2011
Return-Path: <general-return-3407-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8C83C3F52
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri,  6 May 2011 16:48:52 +0000 (UTC)
Received: (qmail 60094 invoked by uid 500); 6 May 2011 16:48:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60030 invoked by uid 500); 6 May 2011 16:48:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60022 invoked by uid 99); 6 May 2011 16:48:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 16:48:50 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mbhandarkar@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 16:48:46 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:CC:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:In-Reply-To:Accept-Language:
   Content-Language:X-MS-Has-Attach:X-MS-TNEF-Correlator:
   x-originating-ip:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version;
  b=pamtRap3GtJk1lWhPrldiCY3kGnGgm2zyCyywCoUbi2ye8MYRJdljY/Q
   6pYIyDJ3aND/62ITN2n99lDIxV0JyoiR0U1ubDrT/dhHeV7yBbYA8bSgH
   i9vf472a9CDyzOe;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=mbhandarkar@linkedin.com; q=dns/txt;
  s=proddkim; t=1304700526; x=1336236526;
  h=from:to:cc:subject:date:message-id:in-reply-to:
   content-id:content-transfer-encoding:mime-version;
  bh=VF1bbQWV4VK30KhA1GifIljtApnL2HF/dk7C4ag3iPg=;
  b=pWYWtRQc5q0kSp8IPLVpRT5p0EuPZfbdgR0lLTZmyrYq3Sb1aNTmBqD+
   yHmBo2Y7UkPRX3gn8wlqaXp9MSmouOOmynyS0Thyk3SU7HmXr0WqfjxBX
   aHQOfBOKWq9aN0j;
X-IronPort-AV: E=Sophos;i="4.64,327,1301900400"; 
   d="scan'208";a="22924164"
Received: from ESV4-EXC02.linkedin.biz ([fe80::4d74:48bd:e0bd:13ee]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi id 14.01.0251.000; Fri, 6
 May 2011 09:51:29 -0700
From: Milind Bhandarkar <mbhandarkar@linkedin.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: Steve Loughran <stevel@apache.org>
Subject: Re: [DISCUSSION] development process of Hadoop
Thread-Topic: [DISCUSSION] development process of Hadoop
Thread-Index: AQHMCwoF5EVOIXU1nkuwTbIZGN61ApR+kEyAgABjsgCAAAWnAIABPG0AgAAZjYD//7UFgA==
Date: Fri, 6 May 2011 16:51:28 +0000
Message-ID: <C9E973E8.8664%mbhandarkar@linkedin.com>
In-Reply-To: <4DC402B7.8070900@uci.cu>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="iso-8859-1"
Content-ID: <5B9BCCFB9CF6524695285BCB4425F578@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

+1 for git.

When (not if) Apache Hadoop switches to git, I would recommend all to
consider the branching model beautifully described in
http://nvie.com/posts/a-successful-git-branching-model/.

- milind

--=20
Milind Bhandarkar
mbhandarkar@linkedin.com
+1-650-776-3167






On 5/6/11 7:16 AM, "Marcos Ortiz" <mlortiz@uci.cu> wrote:

>On 05/06/2011 08:14 AM, Steve Loughran wrote:
>> On 05/05/11 18:52, Todd Lipcon wrote:
>>> On Thu, May 5, 2011 at 10:32 AM, Eric Yang<eyang@yahoo-inc.com>  wrote:
>>>
>>>> Git is powerful in maintaining different branch of the source code.
>>>>   However, it will only work if the entire community is willing to
>>>> move to
>>>> git.  Maintaining svn and git hybrid, is a time consuming task that
>>>> we are
>>>> paying in full price.  Hadoop community should work smarter for the
>>>> source
>>>> control.  What do people think about fully adopting git instead of
>>>>svn?
>>>>
>>>
>>> +1 for Git as a tool. But using git makes it even _more_ important
>>> that we
>>> have a clearly defined release process that outlines which branches are
>>> meant to be released as official artifacts, and what the inclusion
>>> criteria
>>> for those branches should be.
>>>
>>
>> I'm +0.9995 for git: some bits I like, some bits I don't (it's awful
>> for binary data). And you need more than just a release process locked
>> down, you need the developers understanding following a good process.
>> If you have written down process docs there, I'd love to see them.
>>
>> apache infrastructure are discussing git -what would be best would be
>> to start with a non-critical project, such as one or more of the moved
>> contrib projects (like MR-Unit), so we can see that it, gerrit, etc
>> work well within the Hadoop developer world.
>>
>>
>>
>+1 for Git
>
>We migrated from SVN to Git for our completed infrastructure, for many
>reason:
>- Git use much less space than SVN, all the changes are in a single .git
>- Git is awesome for branching
>- Another great advantage is that there are many developers that know
>Git, and how the development process can be greatly improved.
>
>PostgreSQL, one of my favorites open source projects that I use on my
>daily work, migrated the development process to Git from CVS.
>
>Regards.
>
>--=20
>Marcos Lu=EDs Ort=EDz Valmaseda
>  Software Engineer (Large-Scaled Distributed Systems)
>  University of Information Sciences,
>  La Habana, Cuba
>  Linux User # 418229
>  http://about.me/marcosortiz
>


From general-return-3408-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 18:31:28 2011
Return-Path: <general-return-3408-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 49307365B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri,  6 May 2011 18:31:28 +0000 (UTC)
Received: (qmail 882 invoked by uid 500); 6 May 2011 18:31:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 804 invoked by uid 500); 6 May 2011 18:31:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 796 invoked by uid 99); 6 May 2011 18:31:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 18:31:26 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [98.139.53.197] (HELO nm11-vm1.bullet.mail.ac4.yahoo.com) (98.139.53.197)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 06 May 2011 18:31:17 +0000
Received: from [98.139.52.192] by nm11.bullet.mail.ac4.yahoo.com with NNFMP; 06 May 2011 18:30:56 -0000
Received: from [98.139.52.152] by tm5.bullet.mail.ac4.yahoo.com with NNFMP; 06 May 2011 18:30:56 -0000
Received: from [127.0.0.1] by omp1035.mail.ac4.yahoo.com with NNFMP; 06 May 2011 18:30:56 -0000
X-Yahoo-Newman-Property: ymail-5
X-Yahoo-Newman-Id: 571912.32407.bm@omp1035.mail.ac4.yahoo.com
Received: (qmail 81724 invoked by uid 60001); 6 May 2011 18:30:56 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1304706656; bh=nLYtR+F48n9fTQTUSdQTevldMzCT+LnBFUga6wqaBJg=; h=Message-ID:X-YMail-OSG:Received:X-RocketYMMF:X-Mailer:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=b81z5lLvpZDBTF066ANrEc/1AVFFzUH1Z5NXbajDpXAeoNiTym7gxzDDMd1zk2T6Th8GajMy3wiaEVXXLHP7aPxWWOVcl5FOCPWdwblMywd7bEz9hKfGQbejWHcM1dA2xEAJooqyMx/qSyB2QH0MUN0Mtd10x7b5maK3JeJtEd8=
Message-ID: <403388.69170.qm@web65509.mail.ac4.yahoo.com>
X-YMail-OSG: 2QW48NwVM1mTOrH1rzIlZMnTggOlvIqwlzAHiEA5DyMw0OL
 vgV0eoB0Ztb.F6FvqOdYmTi5Cz5O0RgkKh1Y67k6VMc2tClOC6c8EWkKjDIQ
 65gxWzIgtWiDV9iy0iFDaWPdiQi.T_DwHvDYLOkk0QxcJsxl1i9rf_FFDclB
 cOaVb.w3emyWpDGlMcwaSf.LzBJBIMiq3jBRuygQ5WcQAGTfxvjMdqG8DuRt
 qPhnxiy07biwC8npoII5fEvwIb6Y7ghXMiNVN4usk5rHTx800mKc3n4LOA3a
 eOa9taPZ_i5mkp8rH3TlA2oMt0ydlSRCDaGouAmHpjr7RNpY5bGvLlcrUoEC
 I5pmcIYWqGUmYJLnA8pmNDR3iWJFlKzBACq0ruIEEzoIPLUu1jmpdjFIxL7X
 rNmeqY2BJ2vF_80U21V1s7A3qlvc0t8LRxpWJVAofauq8O.yIDzGnchnvuyR
 MkL_94o0VZA--
Received: from [69.231.27.174] by web65509.mail.ac4.yahoo.com via HTTP; Fri, 06 May 2011 11:30:56 PDT
X-RocketYMMF: apurtell
X-Mailer: YahooMailClassic/12.0.2 YahooMailWebService/0.8.110.299900
Date: Fri, 6 May 2011 11:30:56 -0700 (PDT)
From: Andrew Purtell <apurtell@apache.org>
Reply-To: apurtell@apache.org
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
To: general@hadoop.apache.org
In-Reply-To: <4DC3E0FE.9060207@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org

> -we've been using Y! releases, and this brings their branch
> back into the apache fold, meaning we can say "the official
> Apache release of Apache Hadoop is something you can use in
> production".

And it is worth consideration that there is some measure of competition between Apache Hadoop and other projects, for example: 

  http://twitter.com/#!/CurtMonash/status/66343706743160832
  "@SethGrimes The @DataStax story is "Cassandra is mature,
  unlike Apache Hadoop, and hence will rescue Hadoop"" I've
  requested proof points..."

and they are happy to exploit any perception of division, lack of forward progress, and such. Perhaps this is additional, and I pray sufficient, motivation to cease the proxy battles, bury the hatchet, etc.

   - Andy


--- On Fri, 5/6/11, Steve Loughran <stevel@apache.org> wrote:

> From: Steve Loughran <stevel@apache.org>
> Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
> To: general@hadoop.apache.org
> Date: Friday, May 6, 2011, 4:52 AM
> 
> Vote: +1
> 
> I'm a committer not PMC, so it's non binding.
> 
> 
> Why:
> -looks and works OK on my desktop
> -we've been using Y! releases, and this brings their branch
> back into the apache fold, meaning we can say "the official
> Apache release of Apache Hadoop is something you can use in
> production".
> -I'm confident that the Y! team have tested this well.
> 
> I understand Eli's concerns that putting stuff in there
> that hasn't gone into trunk yet is danger. However, as the
> team makes no guarantees of 100% compatibility between
> releases, I don't think it's critical. It's just something
> that needs to be addressed -which can be done after this
> release has shipped.
> 
> 
> -Steve
> 
> 

From general-return-3409-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 19:03:33 2011
Return-Path: <general-return-3409-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B7A6F3EC4
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri,  6 May 2011 19:03:33 +0000 (UTC)
Received: (qmail 46075 invoked by uid 500); 6 May 2011 19:03:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45925 invoked by uid 500); 6 May 2011 19:03:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45917 invoked by uid 99); 6 May 2011 19:03:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 19:03:32 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of hammer@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 19:03:26 +0000
Received: by bwz8 with SMTP id 8so4682998bwz.35
        for <general@hadoop.apache.org>; Fri, 06 May 2011 12:03:05 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.19.82 with SMTP id z18mr1956808bka.175.1304708585537; Fri,
 06 May 2011 12:03:05 -0700 (PDT)
Received: by 10.204.37.204 with HTTP; Fri, 6 May 2011 12:03:05 -0700 (PDT)
Date: Fri, 6 May 2011 12:03:05 -0700
Message-ID: <BANLkTiksqFvXdN68Ki4ZK0y=GLmr7s=MJg@mail.gmail.com>
Subject: Apache Hadoop Hackathons: 5/11 and 5/18 in SF and Palo Alto
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00032555a726c6c04804a2a023c7
X-Virus-Checked: Checked by ClamAV on apache.org

--00032555a726c6c04804a2a023c7
Content-Type: text/plain; charset=ISO-8859-1

Hey,

The discussion this week about the 0.20.203.0 release has done a great job
of highlighting some issues in our development process; it's also done a
great job of lifting our mailing list activity metrics. After reading
through the various threads, it's clear that everyone agrees on two things:

1) We'd like to get the work done in the 0.20.x branches into trunk
2) We'd like to start releasing off of trunk again

To that end, a few folks from Yahoo!, Cloudera, StumbleUpon, and Facebook
would like to put together a series of Hackathons to 1) burn down the 13
(only 13!) remaining blockers for the 0.22 release and 2) forward-port the
work done in the 0.20.x branches into trunk.

Cloudera will be hosting Hackathons from 10 am to 6 pm in both our Palo Alto
and San Francisco offices next Wednesday, 5/11, and the following Wednesday,
5/18, to ensure both of these tasks get completed in short order. PMC
members Todd Lipcon and Tom White will lead the SF group and PMC members Eli
Collins and Patrick Hunt will lead the Palo Alto group.

Whether you're a long-time contributor or don't have a patch to your name,
now is a great time to get involved in Apache Hadoop development. Forward
porting patches is a lot easier than writing them from scratch, and you'll
have mentors present to help guide you through the patch testing and
submission process.

To sign up for the 5/11 Hackathon in either SF or PA, head over to
http://hadoophackathon.eventbrite.com.

As a reminder, the SF HUG will be held on 5/11 at 6 pm in the Cloudera SF
offices: http://www.meetup.com/hadoopsf/events/17354462/. If you can't make
it on 5/11, we'll send out a link next week for the 5/18 event.

Looking forward to getting the release train moving again!

Regards,
Jeff

p.s. if you'd like to participate remotely, email me directly and I'll see
about how we can teleconference you into the event.

--00032555a726c6c04804a2a023c7--

From general-return-3410-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 20:05:37 2011
Return-Path: <general-return-3410-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 110A53F9B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri,  6 May 2011 20:05:37 +0000 (UTC)
Received: (qmail 28119 invoked by uid 500); 6 May 2011 20:05:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28066 invoked by uid 500); 6 May 2011 20:05:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28058 invoked by uid 99); 6 May 2011 20:05:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 20:05:35 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 20:05:27 +0000
Received: by bwz8 with SMTP id 8so4756952bwz.35
        for <general@hadoop.apache.org>; Fri, 06 May 2011 13:05:07 -0700 (PDT)
Received: by 10.204.8.141 with SMTP id h13mr1336767bkh.64.1304712307192; Fri,
 06 May 2011 13:05:07 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.205.81.143 with HTTP; Fri, 6 May 2011 13:04:47 -0700 (PDT)
In-Reply-To: <4EB044AB-A4F8-4530-AB68-0A675D96BCF8@mac.com>
References: <C955EB7A.2E753%sureshms@yahoo-inc.com> <E2932D45-FF26-4579-A975-A98D2578622A@mac.com>
 <B98835A6-F4B7-426E-B22F-B2BEBE273C47@mac.com> <4112B27D-7DCF-4C8E-AD61-6884BA7E738C@mac.com>
 <844F39DC-0A5D-4AAE-AF8D-EFD4CC83C18A@mac.com> <BANLkTin+SyXE0tnfwhhH9P=wR0b7wCOHaw@mail.gmail.com>
 <BANLkTi=+cQBhsNjwdzB+fhcDkePJ=gSOqQ@mail.gmail.com> <9B529359-B589-4229-BA96-188C762A8E31@apache.org>
 <522E7C8B-C159-4520-A758-6E994195D0DC@mac.com> <83FF361E-EA96-4E82-9084-19D91BCEB608@mac.com>
 <BANLkTimLoMzdA2BCZtxSNbsvXqq39XiVtw@mail.gmail.com> <AF47F47F-4712-43BE-ACCA-B6B1F00FC697@yahoo-inc.com>
 <4EB044AB-A4F8-4530-AB68-0A675D96BCF8@mac.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Fri, 6 May 2011 13:04:47 -0700
Message-ID: <BANLkTikzjT_kv6E2jxJc7UOA8wJ9CK+k0g@mail.gmail.com>
Subject: Re: HADOOP-7106: Re-organize hadoop subversion layout
To: Nigel Daley <ndaley@mac.com>
Cc: general@hadoop.apache.org, Ian Holsman <ian@holsman.net>, 
	infrastructure-dev@apache.org
Content-Type: multipart/alternative; boundary=00151743fa569ab4e704a2a1010f
X-Virus-Checked: Checked by ClamAV on apache.org

--00151743fa569ab4e704a2a1010f
Content-Type: text/plain; charset=ISO-8859-1

Hey folks,

FYI I'm in the process of loading one of the SVN dumps onto a server here.

MAN is it slow. Going maybe 2 revisions/sec.... so I should have an SVN
replica in somewhere around a week to test with.

@Infra: I don't suppose it's possible to get a writable snapshot mounted
somehow? If I recall correctly, ZFS supports this and svn.apache.org runs
ZFS?


-Todd

On Fri, Apr 29, 2011 at 1:16 PM, Nigel Daley <ndaley@mac.com> wrote:

> I can't do this at 2pm now.  Todd, I suspect you want more time to try out
> the svn/git test anyways.
>
> Let's shoot for next Wednesday at 2pm.  Ian should be back by then too.
>  Any objections?
>
> Cheers,
> Nige
>
> On Apr 29, 2011, at 11:36 AM, Owen O'Malley wrote:
>
> >
> > On Apr 28, 2011, at 11:24 PM, Todd Lipcon wrote:
> >
> >> Wasn't sure how to go about doing that. I guess we need to talk to infra
> about it? Do you know how we might clone the SVN repos themselves to test
> with?
> >
> > It looks like there are svn dumps at http://svn-master.apache.org/dump/from 2 april 2011. You should be able to use those to setup a local
> subversion.
> >
> > -- Owen
> >
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

--00151743fa569ab4e704a2a1010f--

From general-return-3411-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 20:22:06 2011
Return-Path: <general-return-3411-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 44602364F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri,  6 May 2011 20:22:06 +0000 (UTC)
Received: (qmail 48019 invoked by uid 500); 6 May 2011 20:22:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47858 invoked by uid 500); 6 May 2011 20:22:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47660 invoked by uid 99); 6 May 2011 20:22:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 20:22:04 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of paul.joseph.davis@gmail.com designates 209.85.216.169 as permitted sender)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 20:21:57 +0000
Received: by qyk2 with SMTP id 2so5858606qyk.14
        for <general@hadoop.apache.org>; Fri, 06 May 2011 13:21:36 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:cc:content-type:content-transfer-encoding;
        bh=6rIfvMKaChVZzTEs81GA4q3MnIej/5avRKwJpHZHeTM=;
        b=rzekpelkifoLHx6KcsJcN2LZrnzL6EXSRGWMuDqGiSdfh/KckLg2clgjwDNvoOFNvw
         vkGXoi/g8AKABe5IZ3bPwIibNi6Ekya9am18p5h5dXICoGchZeLypjA1Q7BxR9xQ+o47
         XKZlv8MZLyisdcd+EQtB2IGrN72C1EmF7kOLM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :cc:content-type:content-transfer-encoding;
        b=J7Kls/w8COYpiEJ2+oPkjBwyS3778veFhm4MSD4GeW+9ltB9LJRiULOSmkRqb9l4up
         rWVhzgnrxklXvmd1jc8jbKm/PHnMiNb3PS+BdV5cJns/zmUCgnsxEZaTDLltXei9/Erq
         1IX7AdRndISVCp5ogvmvf14Va6vjkRfPmqkQQ=
Received: by 10.52.172.242 with SMTP id bf18mr458894vdc.161.1304713296058;
 Fri, 06 May 2011 13:21:36 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.52.101.193 with HTTP; Fri, 6 May 2011 13:20:56 -0700 (PDT)
In-Reply-To: <BANLkTikzjT_kv6E2jxJc7UOA8wJ9CK+k0g@mail.gmail.com>
References: <C955EB7A.2E753%sureshms@yahoo-inc.com> <E2932D45-FF26-4579-A975-A98D2578622A@mac.com>
 <B98835A6-F4B7-426E-B22F-B2BEBE273C47@mac.com> <4112B27D-7DCF-4C8E-AD61-6884BA7E738C@mac.com>
 <844F39DC-0A5D-4AAE-AF8D-EFD4CC83C18A@mac.com> <BANLkTin+SyXE0tnfwhhH9P=wR0b7wCOHaw@mail.gmail.com>
 <BANLkTi=+cQBhsNjwdzB+fhcDkePJ=gSOqQ@mail.gmail.com> <9B529359-B589-4229-BA96-188C762A8E31@apache.org>
 <522E7C8B-C159-4520-A758-6E994195D0DC@mac.com> <83FF361E-EA96-4E82-9084-19D91BCEB608@mac.com>
 <BANLkTimLoMzdA2BCZtxSNbsvXqq39XiVtw@mail.gmail.com> <AF47F47F-4712-43BE-ACCA-B6B1F00FC697@yahoo-inc.com>
 <4EB044AB-A4F8-4530-AB68-0A675D96BCF8@mac.com> <BANLkTikzjT_kv6E2jxJc7UOA8wJ9CK+k0g@mail.gmail.com>
From: Paul Davis <paul.joseph.davis@gmail.com>
Date: Fri, 6 May 2011 16:20:56 -0400
Message-ID: <BANLkTinA+FjqCcTjaUqbdG58-D0as6fNyw@mail.gmail.com>
Subject: Re: HADOOP-7106: Re-organize hadoop subversion layout
To: infrastructure-dev@apache.org
Cc: Nigel Daley <ndaley@mac.com>, general@hadoop.apache.org, 
	Ian Holsman <ian@holsman.net>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

You might want to look around at the options for imports more. I know
when I was doing git-svn clones it was doing something stupid like 1
rev/s until I found a --log-window setting. Not sure if that's from
the git layer or the svn layer but it made orders of magnitude
difference.

On Fri, May 6, 2011 at 4:04 PM, Todd Lipcon <todd@cloudera.com> wrote:
> Hey folks,
>
> FYI I'm in the process of loading one of the SVN dumps onto a server here=
.
>
> MAN is it slow. Going maybe 2 revisions/sec.... so I should have an SVN
> replica in somewhere around a week to test with.
>
> @Infra: I don't suppose it's possible to get a writable snapshot mounted
> somehow? If I recall correctly, ZFS supports this and svn.apache.org runs
> ZFS?
>
>
> -Todd
>
> On Fri, Apr 29, 2011 at 1:16 PM, Nigel Daley <ndaley@mac.com> wrote:
>
>> I can't do this at 2pm now. =A0Todd, I suspect you want more time to try=
 out
>> the svn/git test anyways.
>>
>> Let's shoot for next Wednesday at 2pm. =A0Ian should be back by then too=
.
>> =A0Any objections?
>>
>> Cheers,
>> Nige
>>
>> On Apr 29, 2011, at 11:36 AM, Owen O'Malley wrote:
>>
>> >
>> > On Apr 28, 2011, at 11:24 PM, Todd Lipcon wrote:
>> >
>> >> Wasn't sure how to go about doing that. I guess we need to talk to in=
fra
>> about it? Do you know how we might clone the SVN repos themselves to tes=
t
>> with?
>> >
>> > It looks like there are svn dumps at http://svn-master.apache.org/dump=
/from 2 april 2011. You should be able to use those to setup a local
>> subversion.
>> >
>> > -- Owen
>> >
>>
>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

From general-return-3412-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat May  7 00:46:07 2011
Return-Path: <general-return-3412-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EF4C93052
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat,  7 May 2011 00:46:06 +0000 (UTC)
Received: (qmail 79004 invoked by uid 500); 7 May 2011 00:46:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78948 invoked by uid 500); 7 May 2011 00:46:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78940 invoked by uid 99); 7 May 2011 00:46:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 00:46:05 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 00:45:59 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=Xk+oSpN+Fot1Lxsox3obaAUETASgSXa/7Z3ycBUC1E93I59g9qIx/0PQ
   QybrHsjfIhhT4k68GSx0uAoopke1pQFg6EObCTUagVXT2iJjEjsDyxDGL
   uc68OybACnIUzL0;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1304729159; x=1336265159;
  h=from:to:subject:date:message-id:references:in-reply-to:
   content-id:content-transfer-encoding:mime-version;
  bh=g0XhI5STImXmzsqBl4JpTf5Lb2eJZRb99odqSU1Cv9o=;
  b=GLB1kr0XdDSYM1dsjGH0Tk6XYMp8EQxsSWz61anJFCsmaIzlvhjjGMFz
   BhS1g19PC5g74UlGon0jVDJRHfZSAJ71vFVlSEv1JLxObkdRn2xZqd66U
   xyriPK2IZVecOoR;
X-IronPort-AV: E=Sophos;i="4.64,329,1301900400"; 
   d="scan'208";a="23531428"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi id 14.01.0251.000; Fri, 6
 May 2011 17:49:05 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
Thread-Topic: [VOTE] Release candidate 0.20.203.0-rc1
Thread-Index: AQHMCoHKs40jdRozskC+8soYJjDqwpR9roiAgAAAxwCAAAUFAIAAAdSAgAAC2wCAAALqAIABUn0AgAAf0oCAAdIfAA==
Date: Sat, 7 May 2011 00:49:04 +0000
Message-ID: <5B44D291-757D-4C0C-9AAB-2EAF55B4208D@linkedin.com>
References: <BANLkTi=B+cc_677HC1wRnpcY73zuF8qekg@mail.gmail.com>
 <C9E84270.1A05E%sureshms@yahoo-inc.com>
 <BANLkTi=S=9aUXXUmrzmNZTBeXt2eW1Catw@mail.gmail.com>
In-Reply-To: <BANLkTi=S=9aUXXUmrzmNZTBeXt2eW1Catw@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <7DE3616669EFF64AA867EF27908C188D@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


On May 5, 2011, at 1:56 PM, Jakob Homan wrote:

> +1
>=20
> Downloaded, verified, tested on single node cluster to my
> satisfaction.  We've also brought this release up on a sizable cluster
> and checked its basic sanity.

	All of you people doing single node tests are missing stuff.  For example,=
 the regression in how the secondary namenode addr stuff works vs. 0.20. =20

	By far, the biggest problem we've found is that the capacity scheduler doc=
umentation doesn't actually match what the code does.  I have a hunch that =
the unit tests were written/change to match the outcome, rather than test w=
hat is supposed to happen.  For us, this breakage makes it unusable out of =
the box and we'll likely either go back to our (relatively stable) backport=
 of 0.21's cap sched, try to fix the 0.20.203 code, or maybe even switch to=
 a completely different scheduler.=

From general-return-3413-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat May  7 00:49:43 2011
Return-Path: <general-return-3413-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3E96E3E4F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat,  7 May 2011 00:49:43 +0000 (UTC)
Received: (qmail 80703 invoked by uid 500); 7 May 2011 00:49:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80651 invoked by uid 500); 7 May 2011 00:49:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80643 invoked by uid 99); 7 May 2011 00:49:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 00:49:41 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 00:49:37 +0000
Received: from [10.0.1.3] (snvvpn1-10-72-244-c28.hq.corp.yahoo.com [10.72.244.28])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p470kgxb094469
	for <general@hadoop.apache.org>; Fri, 6 May 2011 17:46:43 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1304729203;
	bh=ydnPzOXRQVpBnU2VNdw7TlSHtRm8kee0wshzjroMwpU=;
	h=Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:
	 Content-Transfer-Encoding:Message-Id:References:To;
	b=josZoo9lI++sHmiREPVXRj2pZNHa1QSjaeR+PvDZpkgWTz5rFK1g8K6cZKE5ereta
	 UfuIfgRvl0fW42Yl2elp25+uM+/nKWxuUW9dAJIL6QvG1MC3XezR714ova6pDqbSaT
	 IvLHpifJ2dTJfs9kFi1/UoYeHHAFvwllYQZg7Kdo=
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: Apache Hadoop Hackathons: 5/11 and 5/18 in SF and Palo Alto
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <BANLkTiksqFvXdN68Ki4ZK0y=GLmr7s=MJg@mail.gmail.com>
Date: Fri, 6 May 2011 17:46:42 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <6D26A411-537F-4719-B95F-5C0A68077F34@yahoo-inc.com>
References: <BANLkTiksqFvXdN68Ki4ZK0y=GLmr7s=MJg@mail.gmail.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1084)

Hi Jeff,

Could you put me in contact with the folks from yahoo you are talking =
to?

Thanks,

E14

On May 6, 2011, at 12:03 PM, Jeff Hammerbacher wrote:

> Hey,
>=20
> The discussion this week about the 0.20.203.0 release has done a great =
job
> of highlighting some issues in our development process; it's also done =
a
> great job of lifting our mailing list activity metrics. After reading
> through the various threads, it's clear that everyone agrees on two =
things:
>=20
> 1) We'd like to get the work done in the 0.20.x branches into trunk
> 2) We'd like to start releasing off of trunk again
>=20
> To that end, a few folks from Yahoo!, Cloudera, StumbleUpon, and =
Facebook
> would like to put together a series of Hackathons to 1) burn down the =
13
> (only 13!) remaining blockers for the 0.22 release and 2) forward-port =
the
> work done in the 0.20.x branches into trunk.
>=20
> Cloudera will be hosting Hackathons from 10 am to 6 pm in both our =
Palo Alto
> and San Francisco offices next Wednesday, 5/11, and the following =
Wednesday,
> 5/18, to ensure both of these tasks get completed in short order. PMC
> members Todd Lipcon and Tom White will lead the SF group and PMC =
members Eli
> Collins and Patrick Hunt will lead the Palo Alto group.
>=20
> Whether you're a long-time contributor or don't have a patch to your =
name,
> now is a great time to get involved in Apache Hadoop development. =
Forward
> porting patches is a lot easier than writing them from scratch, and =
you'll
> have mentors present to help guide you through the patch testing and
> submission process.
>=20
> To sign up for the 5/11 Hackathon in either SF or PA, head over to
> http://hadoophackathon.eventbrite.com.
>=20
> As a reminder, the SF HUG will be held on 5/11 at 6 pm in the Cloudera =
SF
> offices: http://www.meetup.com/hadoopsf/events/17354462/. If you can't =
make
> it on 5/11, we'll send out a link next week for the 5/18 event.
>=20
> Looking forward to getting the release train moving again!
>=20
> Regards,
> Jeff
>=20
> p.s. if you'd like to participate remotely, email me directly and I'll =
see
> about how we can teleconference you into the event.


From general-return-3414-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat May  7 00:52:34 2011
Return-Path: <general-return-3414-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C6F723E76
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat,  7 May 2011 00:52:34 +0000 (UTC)
Received: (qmail 84289 invoked by uid 500); 7 May 2011 00:52:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84226 invoked by uid 500); 7 May 2011 00:52:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84218 invoked by uid 99); 7 May 2011 00:52:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 00:52:33 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mbhandarkar@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 00:52:27 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:In-Reply-To:Accept-Language:
   Content-Language:X-MS-Has-Attach:X-MS-TNEF-Correlator:
   x-originating-ip:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version;
  b=Hryp9fIqdJTLzOVMZ9ZYX1cniLeV0/CHca3MSm6GO1uAS+ipSFaxyGL2
   4C3UALtslvJvMqBawiPHc8g1Q+LiXTkKdETpfZLbGyB6gWJU5K409xR6c
   kldmEBS/z3Veerz;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=mbhandarkar@linkedin.com; q=dns/txt;
  s=proddkim; t=1304729547; x=1336265547;
  h=from:to:subject:date:message-id:in-reply-to:content-id:
   content-transfer-encoding:mime-version;
  bh=aSuAKKA2q/dhaTIuuW4Ii2ly1BedNFlxTRVnfJGkGeM=;
  b=R7PrnO45uey80/LI3ZYbiHTvxvuZQyQoUrPq2u9LOkHBLNQtferSVGyK
   6kl2e2foeL4N3aQUIUSFqjc45ASnQQtBr/8Oo/sdrrQCpMVKJ5XhTp2JM
   6PvB86D17NyLzed;
X-IronPort-AV: E=Sophos;i="4.64,329,1301900400"; 
   d="scan'208";a="23531513"
Received: from ESV4-EXC02.linkedin.biz ([fe80::4d74:48bd:e0bd:13ee]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi id 14.01.0251.000; Fri, 6
 May 2011 17:55:33 -0700
From: Milind Bhandarkar <mbhandarkar@linkedin.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: Apache Hadoop Hackathons: 5/11 and 5/18 in SF and Palo Alto
Thread-Topic: Apache Hadoop Hackathons: 5/11 and 5/18 in SF and Palo Alto
Thread-Index: AQHMDCDH76fHc6JVUECnxku7xExO05SA/Y0A//+MKAA=
Date: Sat, 7 May 2011 00:55:33 +0000
Message-ID: <C9E9E592.875C%mbhandarkar@linkedin.com>
In-Reply-To: <6D26A411-537F-4719-B95F-5C0A68077F34@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <4B3E205E6A8DDF44A77D97E9901B0CD3@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

(Channeling Allen Wittenauer...:-)

*Grabbing Popcorn...*

- milind

--=20
Milind Bhandarkar
mbhandarkar@linkedin.com







On 5/6/11 5:46 PM, "Eric Baldeschwieler" <eric14@yahoo-inc.com> wrote:

>Hi Jeff,
>
>Could you put me in contact with the folks from yahoo you are talking to?
>
>Thanks,
>
>E14
>
>On May 6, 2011, at 12:03 PM, Jeff Hammerbacher wrote:
>
>> Hey,
>>=20
>> The discussion this week about the 0.20.203.0 release has done a great
>>job
>> of highlighting some issues in our development process; it's also done a
>> great job of lifting our mailing list activity metrics. After reading
>> through the various threads, it's clear that everyone agrees on two
>>things:
>>=20
>> 1) We'd like to get the work done in the 0.20.x branches into trunk
>> 2) We'd like to start releasing off of trunk again
>>=20
>> To that end, a few folks from Yahoo!, Cloudera, StumbleUpon, and
>>Facebook
>> would like to put together a series of Hackathons to 1) burn down the 13
>> (only 13!) remaining blockers for the 0.22 release and 2) forward-port
>>the
>> work done in the 0.20.x branches into trunk.
>>=20
>> Cloudera will be hosting Hackathons from 10 am to 6 pm in both our Palo
>>Alto
>> and San Francisco offices next Wednesday, 5/11, and the following
>>Wednesday,
>> 5/18, to ensure both of these tasks get completed in short order. PMC
>> members Todd Lipcon and Tom White will lead the SF group and PMC
>>members Eli
>> Collins and Patrick Hunt will lead the Palo Alto group.
>>=20
>> Whether you're a long-time contributor or don't have a patch to your
>>name,
>> now is a great time to get involved in Apache Hadoop development.
>>Forward
>> porting patches is a lot easier than writing them from scratch, and
>>you'll
>> have mentors present to help guide you through the patch testing and
>> submission process.
>>=20
>> To sign up for the 5/11 Hackathon in either SF or PA, head over to
>> http://hadoophackathon.eventbrite.com.
>>=20
>> As a reminder, the SF HUG will be held on 5/11 at 6 pm in the Cloudera
>>SF
>> offices: http://www.meetup.com/hadoopsf/events/17354462/. If you can't
>>make
>> it on 5/11, we'll send out a link next week for the 5/18 event.
>>=20
>> Looking forward to getting the release train moving again!
>>=20
>> Regards,
>> Jeff
>>=20
>> p.s. if you'd like to participate remotely, email me directly and I'll
>>see
>> about how we can teleconference you into the event.
>


From general-return-3415-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat May  7 01:44:14 2011
Return-Path: <general-return-3415-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CFC073E2C
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat,  7 May 2011 01:44:14 +0000 (UTC)
Received: (qmail 8575 invoked by uid 500); 7 May 2011 01:44:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8518 invoked by uid 500); 7 May 2011 01:44:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8509 invoked by uid 99); 7 May 2011 01:44:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 01:44:08 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 01:44:01 +0000
Received: from SP2-EX07CAS05.ds.corp.yahoo.com (sp2-ex07cas05.corp.sp2.yahoo.com [98.137.59.39])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p471hZZF096755
	for <general@hadoop.apache.org>; Fri, 6 May 2011 18:43:35 -0700 (PDT)
Received: from SP2-EX07VS07.ds.corp.yahoo.com ([98.137.59.26]) by
 SP2-EX07CAS05.ds.corp.yahoo.com ([98.137.59.39]) with mapi; Fri, 6 May 2011
 18:43:35 -0700
From: Todd Papaioannou <toddp@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 6 May 2011 18:43:33 -0700
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
Thread-Topic: [VOTE] Release candidate 0.20.203.0-rc1
Thread-Index: AcwMWC1+DW2Ye9SjRjmnu3sJKhCKeA==
Message-ID: <C9E9F13F.91B8%toddp@yahoo-inc.com>
In-Reply-To: <5B44D291-757D-4C0C-9AAB-2EAF55B4208D@linkedin.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-MacOutlook/14.10.0.110310
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Allen,

Can you provide some more details into what issues you are seeing with the
capacity scheduler? Is it just the docs don't match the code, or are you
seeing real issues with job scheduling?

Thanks

ToddP

On 5/6/11 5:49 PM, "Allen Wittenauer" <awittenauer@linkedin.com> wrote:

>
>On May 5, 2011, at 1:56 PM, Jakob Homan wrote:
>
>> +1
>>=20
>> Downloaded, verified, tested on single node cluster to my
>> satisfaction.  We've also brought this release up on a sizable cluster
>> and checked its basic sanity.
>
>	All of you people doing single node tests are missing stuff.  For
>example, the regression in how the secondary namenode addr stuff works
>vs. 0.20. =20
>
>	By far, the biggest problem we've found is that the capacity scheduler
>documentation doesn't actually match what the code does.  I have a hunch
>that the unit tests were written/change to match the outcome, rather than
>test what is supposed to happen.  For us, this breakage makes it unusable
>out of the box and we'll likely either go back to our (relatively stable)
>backport of 0.21's cap sched, try to fix the 0.20.203 code, or maybe even
>switch to a completely different scheduler.


From general-return-3416-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat May  7 02:29:00 2011
Return-Path: <general-return-3416-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8A68427BD
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat,  7 May 2011 02:29:00 +0000 (UTC)
Received: (qmail 23404 invoked by uid 500); 7 May 2011 02:28:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23356 invoked by uid 500); 7 May 2011 02:28:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23348 invoked by uid 99); 7 May 2011 02:28:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 02:28:58 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 02:28:52 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=bgNaFuQw/75NImzhPuhug2Mo0t+bC5Idqbu58HcWy45wVoS6ti9Tx99d
   7WxJUX1QA895tf1/Oy/E0woSK32LjZzWPxjbYYbImAYJUMEdhrCbFHFWq
   ziz3BbgOD3WkCoE;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1304735332; x=1336271332;
  h=from:to:subject:date:message-id:references:in-reply-to:
   content-id:content-transfer-encoding:mime-version;
  bh=XlaI2sTGST6pZ+/fGtTxVY8KPxdTsVnbNgE5WqfwpAY=;
  b=arEJ6lUz5sm1E1KFgJPcMSNW/iEOaRmyW1oBB4VPUUqxvvEcttZUoc9L
   yCHZXdX509G5UCEr/QbNvqRVrIhfd+2f3s5bG3/IdZJVj2O4xYy6O+6Wi
   o1oRNR+rtQT5Ytu;
X-IronPort-AV: E=Sophos;i="4.64,329,1301900400"; 
   d="scan'208";a="22945328"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi id 14.01.0251.000; Fri, 6
 May 2011 19:31:59 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
Thread-Topic: [VOTE] Release candidate 0.20.203.0-rc1
Thread-Index: AQHMCoHKs40jdRozskC+8soYJjDqwpR9roiAgAAAxwCAAAUFAIAAAdSAgAAC2wCAAALqAIABUn0AgAAf0oCAAdIfAIAAEFOAgAAMkIA=
Date: Sat, 7 May 2011 02:31:58 +0000
Message-ID: <993A636D-D5F2-41C4-93DC-9B02E07F5B65@linkedin.com>
References: <C9E9F13F.91B8%toddp@yahoo-inc.com>
In-Reply-To: <C9E9F13F.91B8%toddp@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <04CAC0D8A8CA824BAA7F5FC7B01D9DE1@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


On May 6, 2011, at 6:43 PM, Todd Papaioannou wrote:

> Allen,
>=20
> Can you provide some more details into what issues you are seeing with th=
e
> capacity scheduler? Is it just the docs don't match the code, or are you
> seeing real issues with job scheduling?

	Jobs are definitely not getting the maximum number of task slots they shou=
ld be getting.  I'm suspecting a bug with how max-limit of -1 queues are ha=
ndled.  I'll actually be in the office next week to try and see if I can fi=
gure out where things are going haywire.

	[I filed a bug on this a few weeks ago before I left for vacation.  It was=
 basically ignored.]=

From general-return-3417-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat May  7 03:35:01 2011
Return-Path: <general-return-3417-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 129992768
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat,  7 May 2011 03:35:01 +0000 (UTC)
Received: (qmail 58902 invoked by uid 500); 7 May 2011 03:34:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58498 invoked by uid 500); 7 May 2011 03:34:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58487 invoked by uid 99); 7 May 2011 03:34:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 03:34:54 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 03:34:47 +0000
Received: by pve37 with SMTP id 37so2451303pve.35
        for <general@hadoop.apache.org>; Fri, 06 May 2011 20:34:25 -0700 (PDT)
Received: by 10.68.3.139 with SMTP id c11mr1608480pbc.277.1304739265080; Fri,
 06 May 2011 20:34:25 -0700 (PDT)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.68.41.38 with HTTP; Fri, 6 May 2011 20:34:05 -0700 (PDT)
In-Reply-To: <6D26A411-537F-4719-B95F-5C0A68077F34@yahoo-inc.com>
References: <BANLkTiksqFvXdN68Ki4ZK0y=GLmr7s=MJg@mail.gmail.com> <6D26A411-537F-4719-B95F-5C0A68077F34@yahoo-inc.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Fri, 6 May 2011 20:34:05 -0700
X-Google-Sender-Auth: UsUaDcjV8h52uEiabTKSsLTiX4g
Message-ID: <BANLkTinrJ6E4KpzHT2mR0kN4k=N6v2ePdw@mail.gmail.com>
Subject: Re: Apache Hadoop Hackathons: 5/11 and 5/18 in SF and Palo Alto
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Fri, May 6, 2011 at 17:46, Eric Baldeschwieler <eric14@yahoo-inc.com> wrote:
> Hi Jeff,
>
> Could you put me in contact with the folks from yahoo you are talking to?

Aren't you at Y! anymore, Eric? Wow!

> Thanks,
>
> E14
>
> On May 6, 2011, at 12:03 PM, Jeff Hammerbacher wrote:
>
>> Hey,
>>
>> The discussion this week about the 0.20.203.0 release has done a great job
>> of highlighting some issues in our development process; it's also done a
>> great job of lifting our mailing list activity metrics. After reading
>> through the various threads, it's clear that everyone agrees on two things:
>>
>> 1) We'd like to get the work done in the 0.20.x branches into trunk
>> 2) We'd like to start releasing off of trunk again
>>
>> To that end, a few folks from Yahoo!, Cloudera, StumbleUpon, and Facebook
>> would like to put together a series of Hackathons to 1) burn down the 13
>> (only 13!) remaining blockers for the 0.22 release and 2) forward-port the
>> work done in the 0.20.x branches into trunk.
>>
>> Cloudera will be hosting Hackathons from 10 am to 6 pm in both our Palo Alto
>> and San Francisco offices next Wednesday, 5/11, and the following Wednesday,
>> 5/18, to ensure both of these tasks get completed in short order. PMC
>> members Todd Lipcon and Tom White will lead the SF group and PMC members Eli
>> Collins and Patrick Hunt will lead the Palo Alto group.
>>
>> Whether you're a long-time contributor or don't have a patch to your name,
>> now is a great time to get involved in Apache Hadoop development. Forward
>> porting patches is a lot easier than writing them from scratch, and you'll
>> have mentors present to help guide you through the patch testing and
>> submission process.
>>
>> To sign up for the 5/11 Hackathon in either SF or PA, head over to
>> http://hadoophackathon.eventbrite.com.
>>
>> As a reminder, the SF HUG will be held on 5/11 at 6 pm in the Cloudera SF
>> offices: http://www.meetup.com/hadoopsf/events/17354462/. If you can't make
>> it on 5/11, we'll send out a link next week for the 5/18 event.
>>
>> Looking forward to getting the release train moving again!
>>
>> Regards,
>> Jeff
>>
>> p.s. if you'd like to participate remotely, email me directly and I'll see
>> about how we can teleconference you into the event.
>
>

From general-return-3418-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat May  7 04:52:12 2011
Return-Path: <general-return-3418-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7D9A82638
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat,  7 May 2011 04:52:12 +0000 (UTC)
Received: (qmail 93024 invoked by uid 500); 7 May 2011 04:52:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92203 invoked by uid 500); 7 May 2011 04:52:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 92187 invoked by uid 99); 7 May 2011 04:52:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 04:52:07 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 04:52:01 +0000
Received: from SP2-EX07CAS04.ds.corp.yahoo.com (sp2-ex07cas04.corp.sp2.yahoo.com [98.137.59.5])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p474pQV9089047
	for <general@hadoop.apache.org>; Fri, 6 May 2011 21:51:26 -0700 (PDT)
Received: from SP2-EX07VS04.ds.corp.yahoo.com ([98.137.59.33]) by
 SP2-EX07CAS04.ds.corp.yahoo.com ([98.137.59.5]) with mapi; Fri, 6 May 2011
 21:51:26 -0700
From: Arun C Murthy <arunc@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 6 May 2011 21:51:22 -0700
Subject: Re: Apache Hadoop Hackathons: 5/11 and 5/18 in SF and Palo Alto
Thread-Topic: Apache Hadoop Hackathons: 5/11 and 5/18 in SF and Palo Alto
Thread-Index: AcwMcmwKZlRvRfUPRb+9LEfdaL1h+A==
Message-ID: <9CFF0257-889B-44F4-9114-9C0534D1B7DE@yahoo-inc.com>
References: <BANLkTiksqFvXdN68Ki4ZK0y=GLmr7s=MJg@mail.gmail.com>
 <6D26A411-537F-4719-B95F-5C0A68077F34@yahoo-inc.com>
 <BANLkTinrJ6E4KpzHT2mR0kN4k=N6v2ePdw@mail.gmail.com>
In-Reply-To: <BANLkTinrJ6E4KpzHT2mR0kN4k=N6v2ePdw@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Cos,

These are difficult times for the community as a whole, the least we can do=
 it treat each other with respect and dignity in these public forums.=20

Your comment was in very poor taste, please refrain from such in the future=
.

Thank you,
Arun

Sent from my iPhone

On May 6, 2011, at 8:35 PM, "Konstantin Boudnik" <cos@apache.org> wrote:

> On Fri, May 6, 2011 at 17:46, Eric Baldeschwieler <eric14@yahoo-inc.com> =
wrote:
>> Hi Jeff,
>>=20
>> Could you put me in contact with the folks from yahoo you are talking to=
?
>=20
> Aren't you at Y! anymore, Eric? Wow!
>=20
>> Thanks,
>>=20
>> E14
>>=20
>> On May 6, 2011, at 12:03 PM, Jeff Hammerbacher wrote:
>>=20
>>> Hey,
>>>=20
>>> The discussion this week about the 0.20.203.0 release has done a great =
job
>>> of highlighting some issues in our development process; it's also done =
a
>>> great job of lifting our mailing list activity metrics. After reading
>>> through the various threads, it's clear that everyone agrees on two thi=
ngs:
>>>=20
>>> 1) We'd like to get the work done in the 0.20.x branches into trunk
>>> 2) We'd like to start releasing off of trunk again
>>>=20
>>> To that end, a few folks from Yahoo!, Cloudera, StumbleUpon, and Facebo=
ok
>>> would like to put together a series of Hackathons to 1) burn down the 1=
3
>>> (only 13!) remaining blockers for the 0.22 release and 2) forward-port =
the
>>> work done in the 0.20.x branches into trunk.
>>>=20
>>> Cloudera will be hosting Hackathons from 10 am to 6 pm in both our Palo=
 Alto
>>> and San Francisco offices next Wednesday, 5/11, and the following Wedne=
sday,
>>> 5/18, to ensure both of these tasks get completed in short order. PMC
>>> members Todd Lipcon and Tom White will lead the SF group and PMC member=
s Eli
>>> Collins and Patrick Hunt will lead the Palo Alto group.
>>>=20
>>> Whether you're a long-time contributor or don't have a patch to your na=
me,
>>> now is a great time to get involved in Apache Hadoop development. Forwa=
rd
>>> porting patches is a lot easier than writing them from scratch, and you=
'll
>>> have mentors present to help guide you through the patch testing and
>>> submission process.
>>>=20
>>> To sign up for the 5/11 Hackathon in either SF or PA, head over to
>>> http://hadoophackathon.eventbrite.com.
>>>=20
>>> As a reminder, the SF HUG will be held on 5/11 at 6 pm in the Cloudera =
SF
>>> offices: http://www.meetup.com/hadoopsf/events/17354462/. If you can't =
make
>>> it on 5/11, we'll send out a link next week for the 5/18 event.
>>>=20
>>> Looking forward to getting the release train moving again!
>>>=20
>>> Regards,
>>> Jeff
>>>=20
>>> p.s. if you'd like to participate remotely, email me directly and I'll =
see
>>> about how we can teleconference you into the event.
>>=20
>>=20

From general-return-3419-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat May  7 05:55:40 2011
Return-Path: <general-return-3419-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4E20F2DC8
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat,  7 May 2011 05:55:40 +0000 (UTC)
Received: (qmail 18223 invoked by uid 500); 7 May 2011 05:55:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17840 invoked by uid 500); 7 May 2011 05:55:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17832 invoked by uid 99); 7 May 2011 05:55:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 05:55:38 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [208.97.132.177] (HELO homiemail-a33.g.dreamhost.com) (208.97.132.177)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 05:55:30 +0000
Received: from homiemail-a33.g.dreamhost.com (localhost [127.0.0.1])
	by homiemail-a33.g.dreamhost.com (Postfix) with ESMTP id D8C7E59405E
	for <general@hadoop.apache.org>; Fri,  6 May 2011 22:55:08 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; q=dns; s=
	gbiv.com; b=llcOh8W/1u/0Ue4fq2Kj02CbDl7KeM36ZBD620qPjn0RoiYELOlb
	93b9Mi3aO+ElgxoNk/TPZxFFgn72Jabu5VCGkrhC2+ZKUDqecIuvDnK8RmtVTwvK
	We9Ep9E/XXJd+pEG9TUd1opHwPFAFgbnm3ELE9R6uyMhZs4xtlzP5q4=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; s=gbiv.com;
	 bh=Cxli3m8z7VtY/1w47Ld9xt5czZc=; b=XpbJp4gUdoHlgEakvI3iOOP7CWWz
	+Wx4xFEXIobdh41geWJZ5BB105yY7CnLHxJZwZWFOv0cCgMaH5OEixkuLk8kojmA
	GFawnpwJfy0oMNSqYBhhAohCfTrKB9cDfLHjUpxEYE1bTwf9iSCd5Yna3HxKAWMF
	6Yw/iiumHCWISH4=
Received: from [192.168.1.84] (99-21-208-82.lightspeed.irvnca.sbcglobal.net [99.21.208.82])
	(using TLSv1 with cipher AES128-SHA (128/128 bits))
	(No client certificate requested)
	(Authenticated sender: fielding@gbiv.com)
	by homiemail-a33.g.dreamhost.com (Postfix) with ESMTPSA id C5B5E594059
	for <general@hadoop.apache.org>; Fri,  6 May 2011 22:55:08 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: [DISCUSSION] development process of Hadoop
From: "Roy T. Fielding" <fielding@gbiv.com>
In-Reply-To: <C9E973E8.8664%mbhandarkar@linkedin.com>
Date: Fri, 6 May 2011 22:55:08 -0700
Content-Transfer-Encoding: 7bit
Message-Id: <C115C888-6035-40C8-8DA6-4E393D14C09B@gbiv.com>
References: <C9E973E8.8664%mbhandarkar@linkedin.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

Please do not turn this into yet another git discussion.  The issues
Hadoop is having, with extensive development on private branches rather
than collaboration on a common Apache trunk, is a direct result of how
many of the core developers are using git.  This project has quickly
become the poster boy for antisocial behavior via dvcs.

....Roy

From general-return-3420-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat May  7 06:14:36 2011
Return-Path: <general-return-3420-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CC128242A
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat,  7 May 2011 06:14:36 +0000 (UTC)
Received: (qmail 25022 invoked by uid 500); 7 May 2011 06:14:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24670 invoked by uid 500); 7 May 2011 06:14:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24661 invoked by uid 99); 7 May 2011 06:14:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 06:14:31 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of esammer@cloudera.com designates 209.85.210.176 as permitted sender)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 06:14:24 +0000
Received: by iym1 with SMTP id 1so5019213iym.35
        for <general@hadoop.apache.org>; Fri, 06 May 2011 23:14:04 -0700 (PDT)
Received: by 10.42.218.4 with SMTP id ho4mr3705317icb.344.1304748844049; Fri,
 06 May 2011 23:14:04 -0700 (PDT)
References: <FF816856-A1C6-4684-A176-A5A7D87F7ECF@apache.org>
 <BANLkTikH4PazAADyo8+=3d3v0bUHaS24HQ@mail.gmail.com> <3116A002-6EA6-4971-B7AD-261BCC53D76B@gbiv.com>
 <9234135B-FD91-4E92-B4F5-6E6F7EB38A94@Holsman.NET> <4DC3E0FE.9060207@apache.org>
From: Eric Sammer <esammer@cloudera.com>
In-Reply-To: <4DC3E0FE.9060207@apache.org>
Mime-Version: 1.0 (iPhone Mail 8H7)
Date: Fri, 6 May 2011 23:14:01 -0700
Message-ID: <-4353636876961290469@unknownmsgid>
Subject: Release compatibility was Re: [VOTE] Release candidate 0.20.203.0-rc1
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=20cf3054a0495ee70b04a2a983c3

--20cf3054a0495ee70b04a2a983c3
Content-Type: text/plain; charset=ISO-8859-1

On May 6, 2011, at 4:53 AM, Steve Loughran <stevel@apache.org> wrote:

I understand Eli's concerns that putting stuff in there that hasn't gone
into trunk yet is danger. However, as the team makes no guarantees of 100%
compatibility between releases, I don't think it's critical. It's just
something that needs to be addressed -which can be done after this release
has shipped.


I was under the impression that the community has been extremely strict
about compatibility between minor version bumps in the past. I though there
were specific guarantees and that was one of the reasons certain behaviors
have persisted so long.

Does this mean API changes can be made in minor releases and it can be made
backward compatible in future releases? That seems very, very counter to
various conversations that have happened in the past. I'm of the mind that
we should continue to promise what we've always promised and if that's
changing, let's make with the refactoring party!

Can some PMC'ers clarify this one for me?

TIA.
Sammer



-Steve

--20cf3054a0495ee70b04a2a983c3--

From general-return-3421-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat May  7 06:15:49 2011
Return-Path: <general-return-3421-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 302362681
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat,  7 May 2011 06:15:49 +0000 (UTC)
Received: (qmail 26662 invoked by uid 500); 7 May 2011 06:15:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26592 invoked by uid 500); 7 May 2011 06:15:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26555 invoked by uid 99); 7 May 2011 06:15:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 06:15:46 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mbhandarkar@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 06:15:42 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:In-Reply-To:Accept-Language:
   Content-Language:X-MS-Has-Attach:X-MS-TNEF-Correlator:
   x-originating-ip:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version;
  b=wBO+wz8rJqH7bxE2eCNJiVkMfx7ZAAt7e6r6lkyQgtpE7LCN05lW2R59
   mmZbTlzSqOZ6R90lhpdWt1+c344VOpyAAREamzLy8uah5lLxlG5RFsYE7
   1f+fCLG+3pKRB5O;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=mbhandarkar@linkedin.com; q=dns/txt;
  s=proddkim; t=1304748942; x=1336284942;
  h=from:to:subject:date:message-id:in-reply-to:content-id:
   content-transfer-encoding:mime-version;
  bh=z/hniA/H/pD6dEjnjBQJHrWx+7WamNMqZ3XFbGLD9lE=;
  b=o090sdpnyjn1Gf3wZKFOEnW8O0U1EPH2wYyD0ddqskxzfo+THCK8mGF1
   O44ps4Z3AUR4AqlWlhL3yHZDKfR+UiMma4lXPKW8S88wNiB/GUWRKq4ud
   SqHkjbSlApwgC4V;
X-IronPort-AV: E=Sophos;i="4.64,330,1301900400"; 
   d="scan'208";a="23536560"
Received: from ESV4-EXC02.linkedin.biz ([fe80::4d74:48bd:e0bd:13ee]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi id 14.01.0251.000; Fri, 6
 May 2011 23:18:49 -0700
From: Milind Bhandarkar <mbhandarkar@linkedin.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
Thread-Topic: [VOTE] Release candidate 0.20.203.0-rc1
Thread-Index: AcwKremAu2v/kISbSx+vh7BYIk9JfwAqSFpVABKlSoAAOmcUAAAB5x+AAAGw4QD//8kPAA==
Date: Sat, 7 May 2011 06:18:49 +0000
Message-ID: <C9EA2EFE.8790%mbhandarkar@linkedin.com>
In-Reply-To: <993A636D-D5F2-41C4-93DC-9B02E07F5B65@linkedin.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <5A729181F8FBFF49AED759CFE64A1B78@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Allen, there are per job limits, and per user limits in this branch. (So,
max capacity of -1 is for the queue, but within the queue, the per user
limits come into picture.) If I remember right, the defaults were based on
a certain assumption of how many users would be on a queue simultaneously.
Of course this would need to be set in the site-specific config.

- milind

--=20
Milind Bhandarkar
mbhandarkar@linkedin.com
+1-650-776-3167






On 5/6/11 7:31 PM, "Allen Wittenauer" <awittenauer@linkedin.com> wrote:

>
>On May 6, 2011, at 6:43 PM, Todd Papaioannou wrote:
>
>> Allen,
>>=20
>> Can you provide some more details into what issues you are seeing with
>>the
>> capacity scheduler? Is it just the docs don't match the code, or are you
>> seeing real issues with job scheduling?
>
>	Jobs are definitely not getting the maximum number of task slots they
>should be getting.  I'm suspecting a bug with how max-limit of -1 queues
>are handled.  I'll actually be in the office next week to try and see if
>I can figure out where things are going haywire.
>
>	[I filed a bug on this a few weeks ago before I left for vacation.  It
>was basically ignored.]


From general-return-3422-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat May  7 06:16:24 2011
Return-Path: <general-return-3422-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 67309291A
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat,  7 May 2011 06:16:24 +0000 (UTC)
Received: (qmail 28850 invoked by uid 500); 7 May 2011 06:16:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28782 invoked by uid 500); 7 May 2011 06:16:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28774 invoked by uid 99); 7 May 2011 06:16:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 06:16:22 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of esammer@cloudera.com designates 209.85.210.176 as permitted sender)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 06:16:17 +0000
Received: by iym1 with SMTP id 1so5019985iym.35
        for <general@hadoop.apache.org>; Fri, 06 May 2011 23:15:56 -0700 (PDT)
Received: by 10.42.218.4 with SMTP id ho4mr3707053icb.344.1304748956547; Fri,
 06 May 2011 23:15:56 -0700 (PDT)
References: <4DC28B79.7000100@apache.org> <C9E82D2A.118AD%eyang@yahoo-inc.com>
 <BANLkTin_1YK4a39jVtNOdG_bucK6MQhHEw@mail.gmail.com> <4DC3ED48.6000702@apache.org>
From: Eric Sammer <esammer@cloudera.com>
In-Reply-To: <4DC3ED48.6000702@apache.org>
Mime-Version: 1.0 (iPhone Mail 8H7)
Date: Fri, 6 May 2011 23:15:54 -0700
Message-ID: <2582414837582335589@unknownmsgid>
Subject: Re: [DISCUSSION] development process of Hadoop
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I think I speak for all the other mrunit committers when I say we're
happy to be the guinea pigs on this.

On May 6, 2011, at 5:45 AM, Steve Loughran <stevel@apache.org> wrote:

> On 05/05/11 18:52, Todd Lipcon wrote:
>> On Thu, May 5, 2011 at 10:32 AM, Eric Yang<eyang@yahoo-inc.com>  wrote:
>>
>>> Git is powerful in maintaining different branch of the source code.
>>>  However, it will only work if the entire community is willing to move =
to
>>> git.  Maintaining svn and git hybrid, is a time consuming task that we =
are
>>> paying in full price.  Hadoop community should work smarter for the sou=
rce
>>> control.  What do people think about fully adopting git instead of svn?
>>>
>>
>> +1 for Git as a tool. But using git makes it even _more_ important that =
we
>> have a clearly defined release process that outlines which branches are
>> meant to be released as official artifacts, and what the inclusion crite=
ria
>> for those branches should be.
>>
>
> I'm +0.9995 for git: some bits I like, some bits I don't (it's awful for =
binary data). And you need more than just a release process locked down, yo=
u need the developers understanding following a good process. If you have w=
ritten down process docs there, I'd love to see them.
>
> apache infrastructure are discussing git -what would be best would be to =
start with a non-critical project, such as one or more of the moved contrib=
 projects (like MR-Unit), so we can see that it, gerrit, etc work well with=
in the Hadoop developer world.
>
>
>

From general-return-3423-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat May  7 06:20:47 2011
Return-Path: <general-return-3423-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 38FB929E2
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat,  7 May 2011 06:20:47 +0000 (UTC)
Received: (qmail 30745 invoked by uid 500); 7 May 2011 06:20:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30627 invoked by uid 500); 7 May 2011 06:20:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30618 invoked by uid 99); 7 May 2011 06:20:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 06:20:45 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 06:20:39 +0000
Received: by pwi16 with SMTP id 16so2499512pwi.35
        for <general@hadoop.apache.org>; Fri, 06 May 2011 23:20:18 -0700 (PDT)
Received: by 10.68.42.8 with SMTP id j8mr5964680pbl.143.1304749218050; Fri, 06
 May 2011 23:20:18 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.68.41.38 with HTTP; Fri, 6 May 2011 23:19:58 -0700 (PDT)
In-Reply-To: <9CFF0257-889B-44F4-9114-9C0534D1B7DE@yahoo-inc.com>
References: <BANLkTiksqFvXdN68Ki4ZK0y=GLmr7s=MJg@mail.gmail.com>
 <6D26A411-537F-4719-B95F-5C0A68077F34@yahoo-inc.com> <BANLkTinrJ6E4KpzHT2mR0kN4k=N6v2ePdw@mail.gmail.com>
 <9CFF0257-889B-44F4-9114-9C0534D1B7DE@yahoo-inc.com>
From: Konstantin Boudnik <cos@boudnik.org>
Date: Fri, 6 May 2011 23:19:58 -0700
Message-ID: <BANLkTinFoiO-e=-GBOfHm-YFUttB9uHMkQ@mail.gmail.com>
Subject: Re: Apache Hadoop Hackathons: 5/11 and 5/18 in SF and Palo Alto
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Arun,

I believe you're reading to much into my comment - it was absolutely
well intent.

And I think you have to abstain from such reprimands in the future.
Should you have anything to say to me personally - pl. send me a
separate email next time.

With regards,
  Cos

On Fri, May 6, 2011 at 21:51, Arun C Murthy <arunc@yahoo-inc.com> wrote:
> Cos,
>
> These are difficult times for the community as a whole, the least we can do it treat each other with respect and dignity in these public forums.
>
> Your comment was in very poor taste, please refrain from such in the future.
>
> Thank you,
> Arun
>
> Sent from my iPhone
>
> On May 6, 2011, at 8:35 PM, "Konstantin Boudnik" <cos@apache.org> wrote:
>
>> On Fri, May 6, 2011 at 17:46, Eric Baldeschwieler <eric14@yahoo-inc.com> wrote:
>>> Hi Jeff,
>>>
>>> Could you put me in contact with the folks from yahoo you are talking to?
>>
>> Aren't you at Y! anymore, Eric? Wow!
>>
>>> Thanks,
>>>
>>> E14
>>>
>>> On May 6, 2011, at 12:03 PM, Jeff Hammerbacher wrote:
>>>
>>>> Hey,
>>>>
>>>> The discussion this week about the 0.20.203.0 release has done a great job
>>>> of highlighting some issues in our development process; it's also done a
>>>> great job of lifting our mailing list activity metrics. After reading
>>>> through the various threads, it's clear that everyone agrees on two things:
>>>>
>>>> 1) We'd like to get the work done in the 0.20.x branches into trunk
>>>> 2) We'd like to start releasing off of trunk again
>>>>
>>>> To that end, a few folks from Yahoo!, Cloudera, StumbleUpon, and Facebook
>>>> would like to put together a series of Hackathons to 1) burn down the 13
>>>> (only 13!) remaining blockers for the 0.22 release and 2) forward-port the
>>>> work done in the 0.20.x branches into trunk.
>>>>
>>>> Cloudera will be hosting Hackathons from 10 am to 6 pm in both our Palo Alto
>>>> and San Francisco offices next Wednesday, 5/11, and the following Wednesday,
>>>> 5/18, to ensure both of these tasks get completed in short order. PMC
>>>> members Todd Lipcon and Tom White will lead the SF group and PMC members Eli
>>>> Collins and Patrick Hunt will lead the Palo Alto group.
>>>>
>>>> Whether you're a long-time contributor or don't have a patch to your name,
>>>> now is a great time to get involved in Apache Hadoop development. Forward
>>>> porting patches is a lot easier than writing them from scratch, and you'll
>>>> have mentors present to help guide you through the patch testing and
>>>> submission process.
>>>>
>>>> To sign up for the 5/11 Hackathon in either SF or PA, head over to
>>>> http://hadoophackathon.eventbrite.com.
>>>>
>>>> As a reminder, the SF HUG will be held on 5/11 at 6 pm in the Cloudera SF
>>>> offices: http://www.meetup.com/hadoopsf/events/17354462/. If you can't make
>>>> it on 5/11, we'll send out a link next week for the 5/18 event.
>>>>
>>>> Looking forward to getting the release train moving again!
>>>>
>>>> Regards,
>>>> Jeff
>>>>
>>>> p.s. if you'd like to participate remotely, email me directly and I'll see
>>>> about how we can teleconference you into the event.
>>>
>>>
>

From general-return-3424-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat May  7 06:52:08 2011
Return-Path: <general-return-3424-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ABF8F2C5E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat,  7 May 2011 06:52:08 +0000 (UTC)
Received: (qmail 46944 invoked by uid 500); 7 May 2011 06:52:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46624 invoked by uid 500); 7 May 2011 06:52:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46615 invoked by uid 99); 7 May 2011 06:52:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 06:52:03 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mbhandarkar@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 06:51:58 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:In-Reply-To:Accept-Language:
   Content-Language:X-MS-Has-Attach:X-MS-TNEF-Correlator:
   x-originating-ip:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version;
  b=eqxPhR6LBYtyLyaKxrWsir7NONBK+QkhfFbePwfqSQChULR5FcH20tSz
   UYuF7ZXKzBbEXzVmXd7ji3aNosEoJvDWU5CVrpMTNtD45fEl1sQM6hbEV
   JnspxKoBiEwheEP;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=mbhandarkar@linkedin.com; q=dns/txt;
  s=proddkim; t=1304751118; x=1336287118;
  h=from:to:subject:date:message-id:in-reply-to:content-id:
   content-transfer-encoding:mime-version;
  bh=U0kCOOIrse4lPLhpCWb5uInfGuFlAgMnw3Q4l1aijfg=;
  b=nz5lg1Arp4fBuOWK3tb25O5kv2JGDMb9P7UkbyQDGFi9qB3Uw88LLeXy
   b4RUcQ1M2dogVtk11gmFm9bZBvgjzNXqFzhN4hFw3kWv6pY32ZhriLXrJ
   ZncuOOM99tcmHQ4;
X-IronPort-AV: E=Sophos;i="4.64,330,1301900400"; 
   d="scan'208";a="22947753"
Received: from ESV4-EXC02.linkedin.biz ([fe80::4d74:48bd:e0bd:13ee]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi id 14.01.0251.000; Fri, 6
 May 2011 23:55:05 -0700
From: Milind Bhandarkar <mbhandarkar@linkedin.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: Release compatibility was Re: [VOTE] Release candidate
 0.20.203.0-rc1
Thread-Topic: Release compatibility was Re: [VOTE] Release candidate
 0.20.203.0-rc1
Thread-Index: AQHMCsM2u2v/kISbSx+vh7BYIk9Jf5R99lUAgAAUFYCAAh2PAIABM8OA//+VJoA=
Date: Sat, 7 May 2011 06:55:05 +0000
Message-ID: <C9EA354D.87AE%mbhandarkar@linkedin.com>
In-Reply-To: <-4353636876961290469@unknownmsgid>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <92EEA0D7A6D88E40AE00889D1187D627@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

[I am not on PMC, but seeing that PMC may be busy with other issues, I
will try to answer your questions.]

Eric,

I think the thread=20
"http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C18C
5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com%3E" will answer your
questions. Here is the timeline as I see it:

1. Arun proposes to create a release from the security patchset. Says Doug
has proposed this earlier
(http://mail-archives.apache.org/mod_mbox/hadoop-general/201004.mbox/%3C4BD
1DFEA.5020908@apache.org%3E April 23, 2010) ("This has been proposed
earlier by Doug and did not get far due to concerns about the effect this
would have on development on trunk.") (August 24, 2010)

2. Lots of +1s, between August 24 to August 30 2010. One particular
comment is from Tom White: "I think it would be good to have a shared 0.20
Apache security branch.
Since security isn't in 0.21, and the 0.22 release is a some way off
as you mention, this would be useful for folks who want the security
features sooner (and want to use an Apache release)."

3. Arun volunteers to create a release (August 30, 2010)

4. Doug reminds Arun. (October 15, 2010)

5. Arun apologizes for not creating a branch because he was busy, because
he had a baby. (January 11, 2011)
=20
6. Lots of discussion about what to call it (the release, not the baby,
although I had a good laugh at Patrick Angeles's email: "You're gonna call
your kid 20.100?" ;-).

7. Arun proposes to call it 0.20.100: "I'm open to suggestions - how about
something like 20.100 to show that it's a big jump? Anything else?" Jan
12, 2011

8. Among others, Eli says: "+1 on 0.20.x   (where x is a J > 3)" on Jan
12, 2011.

So, as you can see, even if this release is called 0.20.x, the community
agreed that these are valuable patches to have, and despite backward
incompatibility, still have them in minor release.

- milind

--=20
Milind Bhandarkar
mbhandarkar@linkedin.com
+1-650-776-3167






On 5/6/11 11:14 PM, "Eric Sammer" <esammer@cloudera.com> wrote:

>On May 6, 2011, at 4:53 AM, Steve Loughran <stevel@apache.org> wrote:
>
>I understand Eli's concerns that putting stuff in there that hasn't gone
>into trunk yet is danger. However, as the team makes no guarantees of 100%
>compatibility between releases, I don't think it's critical. It's just
>something that needs to be addressed -which can be done after this release
>has shipped.
>
>
>I was under the impression that the community has been extremely strict
>about compatibility between minor version bumps in the past. I though
>there
>were specific guarantees and that was one of the reasons certain behaviors
>have persisted so long.
>
>Does this mean API changes can be made in minor releases and it can be
>made
>backward compatible in future releases? That seems very, very counter to
>various conversations that have happened in the past. I'm of the mind that
>we should continue to promise what we've always promised and if that's
>changing, let's make with the refactoring party!
>
>Can some PMC'ers clarify this one for me?
>
>TIA.
>Sammer
>
>
>
>-Steve


From general-return-3425-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat May  7 06:58:04 2011
Return-Path: <general-return-3425-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 615482CE4
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat,  7 May 2011 06:58:04 +0000 (UTC)
Received: (qmail 53997 invoked by uid 500); 7 May 2011 06:58:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53957 invoked by uid 500); 7 May 2011 06:58:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53949 invoked by uid 99); 7 May 2011 06:58:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 06:58:01 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 06:57:55 +0000
Received: by pzk10 with SMTP id 10so2507666pzk.35
        for <general@hadoop.apache.org>; Fri, 06 May 2011 23:57:33 -0700 (PDT)
Received: by 10.68.3.139 with SMTP id c11mr1827966pbc.277.1304751453174; Fri,
 06 May 2011 23:57:33 -0700 (PDT)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.68.41.38 with HTTP; Fri, 6 May 2011 23:57:13 -0700 (PDT)
In-Reply-To: <C9EA354D.87AE%mbhandarkar@linkedin.com>
References: <-4353636876961290469@unknownmsgid> <C9EA354D.87AE%mbhandarkar@linkedin.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Fri, 6 May 2011 23:57:13 -0700
X-Google-Sender-Auth: zvFhJ81I8h8KRWLnJEnCnuoUpqM
Message-ID: <BANLkTik_pcObRh-Z8NxJYGn+Br5iKCmgew@mail.gmail.com>
Subject: Re: Release compatibility was Re: [VOTE] Release candidate 0.20.203.0-rc1
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Wow! Great compilation, Milind! Very nice to have the sequence of events ha=
ndy.

Thanks,
  Cos

On Fri, May 6, 2011 at 23:55, Milind Bhandarkar
<mbhandarkar@linkedin.com> wrote:
> [I am not on PMC, but seeing that PMC may be busy with other issues, I
> will try to answer your questions.]
>
> Eric,
>
> I think the thread
> "http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C1=
8C
> 5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com%3E" will answer your
> questions. Here is the timeline as I see it:
>
> 1. Arun proposes to create a release from the security patchset. Says Dou=
g
> has proposed this earlier
> (http://mail-archives.apache.org/mod_mbox/hadoop-general/201004.mbox/%3C4=
BD
> 1DFEA.5020908@apache.org%3E April 23, 2010) ("This has been proposed
> earlier by Doug and did not get far due to concerns about the effect this
> would have on development on trunk.") (August 24, 2010)
>
> 2. Lots of +1s, between August 24 to August 30 2010. One particular
> comment is from Tom White: "I think it would be good to have a shared 0.2=
0
> Apache security branch.
> Since security isn't in 0.21, and the 0.22 release is a some way off
> as you mention, this would be useful for folks who want the security
> features sooner (and want to use an Apache release)."
>
> 3. Arun volunteers to create a release (August 30, 2010)
>
> 4. Doug reminds Arun. (October 15, 2010)
>
> 5. Arun apologizes for not creating a branch because he was busy, because
> he had a baby. (January 11, 2011)
>
> 6. Lots of discussion about what to call it (the release, not the baby,
> although I had a good laugh at Patrick Angeles's email: "You're gonna cal=
l
> your kid 20.100?" ;-).
>
> 7. Arun proposes to call it 0.20.100: "I'm open to suggestions - how abou=
t
> something like 20.100 to show that it's a big jump? Anything else?" Jan
> 12, 2011
>
> 8. Among others, Eli says: "+1 on 0.20.x =A0 (where x is a J > 3)" on Jan
> 12, 2011.
>
> So, as you can see, even if this release is called 0.20.x, the community
> agreed that these are valuable patches to have, and despite backward
> incompatibility, still have them in minor release.
>
> - milind
>
> --
> Milind Bhandarkar
> mbhandarkar@linkedin.com
> +1-650-776-3167
>
>
>
>
>
>
> On 5/6/11 11:14 PM, "Eric Sammer" <esammer@cloudera.com> wrote:
>
>>On May 6, 2011, at 4:53 AM, Steve Loughran <stevel@apache.org> wrote:
>>
>>I understand Eli's concerns that putting stuff in there that hasn't gone
>>into trunk yet is danger. However, as the team makes no guarantees of 100=
%
>>compatibility between releases, I don't think it's critical. It's just
>>something that needs to be addressed -which can be done after this releas=
e
>>has shipped.
>>
>>
>>I was under the impression that the community has been extremely strict
>>about compatibility between minor version bumps in the past. I though
>>there
>>were specific guarantees and that was one of the reasons certain behavior=
s
>>have persisted so long.
>>
>>Does this mean API changes can be made in minor releases and it can be
>>made
>>backward compatible in future releases? That seems very, very counter to
>>various conversations that have happened in the past. I'm of the mind tha=
t
>>we should continue to promise what we've always promised and if that's
>>changing, let's make with the refactoring party!
>>
>>Can some PMC'ers clarify this one for me?
>>
>>TIA.
>>Sammer
>>
>>
>>
>>-Steve
>
>

From general-return-3426-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat May  7 07:04:20 2011
Return-Path: <general-return-3426-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 31A0022AE
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat,  7 May 2011 07:04:20 +0000 (UTC)
Received: (qmail 56835 invoked by uid 500); 7 May 2011 07:04:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56793 invoked by uid 500); 7 May 2011 07:04:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56784 invoked by uid 99); 7 May 2011 07:04:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 07:04:18 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mbhandarkar@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 07:04:12 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:In-Reply-To:Accept-Language:
   Content-Language:X-MS-Has-Attach:X-MS-TNEF-Correlator:
   x-originating-ip:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version;
  b=Eeh2L4sfdlrCNo8z05yNkrqVYD6c0QRXowDY1m3IVex4G2JVOnoJ/ZBY
   la4zxxAAuSXY55KLTG+DszUSRw68saZ3i+pcBKGgG4uy8/Jxc0XpS9p+L
   kZ21AK3W16RBAfW;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=mbhandarkar@linkedin.com; q=dns/txt;
  s=proddkim; t=1304751852; x=1336287852;
  h=from:to:subject:date:message-id:in-reply-to:content-id:
   content-transfer-encoding:mime-version;
  bh=7P6oT0NOXWu+bqZw7koGlyq9RmP0xiPEDsco02QJHFU=;
  b=mVucZF7C403BZz8apoU3Fpo+sK1cd9sz76KA03d9lQC7Lcn1dwd7kumB
   cWqXYKHG/ip6MiHCtbE4otiVnMsxPCWBBa7ubnUBli5ml43dYpbGGPzFY
   zkd9HFxd9/eWz4+;
X-IronPort-AV: E=Sophos;i="4.64,330,1301900400"; 
   d="scan'208";a="23537006"
Received: from ESV4-EXC02.linkedin.biz ([fe80::4d74:48bd:e0bd:13ee]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi id 14.01.0251.000; Sat, 7
 May 2011 00:07:18 -0700
From: Milind Bhandarkar <mbhandarkar@linkedin.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: Release compatibility was Re: [VOTE] Release candidate
 0.20.203.0-rc1
Thread-Topic: Release compatibility was Re: [VOTE] Release candidate
 0.20.203.0-rc1
Thread-Index: AQHMCsM2u2v/kISbSx+vh7BYIk9Jf5R99lUAgAAUFYCAAh2PAIABM8OA//+VJoCAAHbsgP//jH4A
Date: Sat, 7 May 2011 07:07:17 +0000
Message-ID: <C9EA3C55.87DB%mbhandarkar@linkedin.com>
In-Reply-To: <BANLkTik_pcObRh-Z8NxJYGn+Br5iKCmgew@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <7B35465A06F1E9499AE8DDBCF1AA763F@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks Kos. Archived mailing lists come in handy. Many thanks to Apache to
have http://mail-archives.apache.org/mod_mbox/hadoop-general/.

- milind
--=20
Milind Bhandarkar
mbhandarkar@linkedin.com
+1-650-776-3167






On 5/6/11 11:57 PM, "Konstantin Boudnik" <cos@apache.org> wrote:

>Wow! Great compilation, Milind! Very nice to have the sequence of events
>handy.
>
>Thanks,
>  Cos
>
>On Fri, May 6, 2011 at 23:55, Milind Bhandarkar
><mbhandarkar@linkedin.com> wrote:
>> [I am not on PMC, but seeing that PMC may be busy with other issues, I
>> will try to answer your questions.]
>>
>> Eric,
>>
>> I think the thread
>>=20
>>"http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C1
>>8C
>> 5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com%3E" will answer your
>> questions. Here is the timeline as I see it:
>>
>> 1. Arun proposes to create a release from the security patchset. Says
>>Doug
>> has proposed this earlier
>>=20
>>(http://mail-archives.apache.org/mod_mbox/hadoop-general/201004.mbox/%3C4
>>BD
>> 1DFEA.5020908@apache.org%3E April 23, 2010) ("This has been proposed
>> earlier by Doug and did not get far due to concerns about the effect
>>this
>> would have on development on trunk.") (August 24, 2010)
>>
>> 2. Lots of +1s, between August 24 to August 30 2010. One particular
>> comment is from Tom White: "I think it would be good to have a shared
>>0.20
>> Apache security branch.
>> Since security isn't in 0.21, and the 0.22 release is a some way off
>> as you mention, this would be useful for folks who want the security
>> features sooner (and want to use an Apache release)."
>>
>> 3. Arun volunteers to create a release (August 30, 2010)
>>
>> 4. Doug reminds Arun. (October 15, 2010)
>>
>> 5. Arun apologizes for not creating a branch because he was busy,
>>because
>> he had a baby. (January 11, 2011)
>>
>> 6. Lots of discussion about what to call it (the release, not the baby,
>> although I had a good laugh at Patrick Angeles's email: "You're gonna
>>call
>> your kid 20.100?" ;-).
>>
>> 7. Arun proposes to call it 0.20.100: "I'm open to suggestions - how
>>about
>> something like 20.100 to show that it's a big jump? Anything else?" Jan
>> 12, 2011
>>
>> 8. Among others, Eli says: "+1 on 0.20.x   (where x is a J > 3)" on Jan
>> 12, 2011.
>>
>> So, as you can see, even if this release is called 0.20.x, the community
>> agreed that these are valuable patches to have, and despite backward
>> incompatibility, still have them in minor release.
>>
>> - milind
>>
>> --
>> Milind Bhandarkar
>> mbhandarkar@linkedin.com
>> +1-650-776-3167
>>
>>
>>
>>
>>
>>
>> On 5/6/11 11:14 PM, "Eric Sammer" <esammer@cloudera.com> wrote:
>>
>>>On May 6, 2011, at 4:53 AM, Steve Loughran <stevel@apache.org> wrote:
>>>
>>>I understand Eli's concerns that putting stuff in there that hasn't gone
>>>into trunk yet is danger. However, as the team makes no guarantees of
>>>100%
>>>compatibility between releases, I don't think it's critical. It's just
>>>something that needs to be addressed -which can be done after this
>>>release
>>>has shipped.
>>>
>>>
>>>I was under the impression that the community has been extremely strict
>>>about compatibility between minor version bumps in the past. I though
>>>there
>>>were specific guarantees and that was one of the reasons certain
>>>behaviors
>>>have persisted so long.
>>>
>>>Does this mean API changes can be made in minor releases and it can be
>>>made
>>>backward compatible in future releases? That seems very, very counter to
>>>various conversations that have happened in the past. I'm of the mind
>>>that
>>>we should continue to promise what we've always promised and if that's
>>>changing, let's make with the refactoring party!
>>>
>>>Can some PMC'ers clarify this one for me?
>>>
>>>TIA.
>>>Sammer
>>>
>>>
>>>
>>>-Steve
>>
>>


From general-return-3427-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat May  7 19:03:44 2011
Return-Path: <general-return-3427-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7C08D288D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat,  7 May 2011 19:03:44 +0000 (UTC)
Received: (qmail 56470 invoked by uid 500); 7 May 2011 19:03:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56382 invoked by uid 500); 7 May 2011 19:03:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56373 invoked by uid 99); 7 May 2011 19:03:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 19:03:42 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 19:03:37 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=vznJt0S6tmMfrNLSA6PrKcjMeXt0rZ3QW9IoF/4wxXvLHv6LmamYmTNJ
   FGxbC9H794IeycUc99jjnZPpAFOgz6h+dIJx7O5FWjxyeDaxSOYOXMqt4
   yScuLEY4ezG3s7F;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1304795017; x=1336331017;
  h=from:to:subject:date:message-id:references:in-reply-to:
   content-id:content-transfer-encoding:mime-version;
  bh=z8q14hCd77IEGiImE4hk4NAKMUdXp3ULAxYji9b11IU=;
  b=W3lUdp7x+uLSwLLwKjMBDwx+d/gRUjTcQpj36m7OC2r/yqHZNLH6PUf4
   ibEDMuTgWBh44+k3VUcpJA5a3WVDb8nKF2n4BrNA5zW1ifd3bXdoOlKI2
   a1TMAeQCcXh49kv;
X-IronPort-AV: E=Sophos;i="4.64,331,1301900400"; 
   d="scan'208";a="22953484"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi id 14.01.0251.000; Sat, 7
 May 2011 12:06:45 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc1
Thread-Topic: [VOTE] Release candidate 0.20.203.0-rc1
Thread-Index: AQHMCoHKs40jdRozskC+8soYJjDqwpR9roiAgAAAxwCAAAUFAIAAAdSAgAAC2wCAAALqAIABUn0AgAAf0oCAAdIfAIAAEFOAgAAMkICAAEBYgIAA1ZgA
Date: Sat, 7 May 2011 19:06:44 +0000
Message-ID: <AFF49313-5EC3-4A26-BFD6-C7209F62AEAC@linkedin.com>
References: <C9EA2EFE.8790%mbhandarkar@linkedin.com>
In-Reply-To: <C9EA2EFE.8790%mbhandarkar@linkedin.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <B4B7DC4FEEF75D4CACD3E22C147E3524@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


On May 6, 2011, at 11:18 PM, Milind Bhandarkar wrote:

> Allen, there are per job limits, and per user limits in this branch. (So,
> max capacity of -1 is for the queue, but within the queue, the per user
> limits come into picture.) If I remember right, the defaults were based o=
n
> a certain assumption of how many users would be on a queue simultaneously=
.
> Of course this would need to be set in the site-specific config.

	Yes, I'm aware of the changes.  What I'm basically saying is that even wit=
h those new limits taken into consideration, the math doesn't seem to hold =
up.



From general-return-3428-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat May  7 23:50:51 2011
Return-Path: <general-return-3428-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 56FB02711
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat,  7 May 2011 23:50:51 +0000 (UTC)
Received: (qmail 11318 invoked by uid 500); 7 May 2011 23:50:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11204 invoked by uid 500); 7 May 2011 23:50:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11196 invoked by uid 99); 7 May 2011 23:50:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 23:50:49 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of esammer@cloudera.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 23:50:43 +0000
Received: by iwr19 with SMTP id 19so5535249iwr.35
        for <general@hadoop.apache.org>; Sat, 07 May 2011 16:50:22 -0700 (PDT)
Received: by 10.42.229.135 with SMTP id ji7mr4206992icb.143.1304812220614;
 Sat, 07 May 2011 16:50:20 -0700 (PDT)
References: <C9EA354D.87AE%mbhandarkar@linkedin.com>
From: Eric Sammer <esammer@cloudera.com>
In-Reply-To: <C9EA354D.87AE%mbhandarkar@linkedin.com>
Mime-Version: 1.0 (iPhone Mail 8H7)
Date: Sat, 7 May 2011 16:50:17 -0700
Message-ID: <-1022679743169743093@unknownmsgid>
Subject: Re: Release compatibility was Re: [VOTE] Release candidate 0.20.203.0-rc1
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Milind:

Thanks for the pointer. I remember this thread. I guess my question
was unrelated to the specific release and more about the general mode
of development under normal release circumstances (ie. do we permit
backward incompatible changes between 0.22.0 and 0.22.1 or is this
something we've allowed just for the 203 release?).

I think it's important to be clear about what the MO is so end users
can plan upgrades appropriately.

Thanks!
Sammer

On May 6, 2011, at 11:52 PM, Milind Bhandarkar <mbhandarkar@linkedin.com> wrote:

> [I am not on PMC, but seeing that PMC may be busy with other issues, I
> will try to answer your questions.]
>
> Eric,
>
> I think the thread
> "http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C18C
> 5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com%3E" will answer your
> questions. Here is the timeline as I see it:
>
> 1. Arun proposes to create a release from the security patchset. Says Doug
> has proposed this earlier
> (http://mail-archives.apache.org/mod_mbox/hadoop-general/201004.mbox/%3C4BD
> 1DFEA.5020908@apache.org%3E April 23, 2010) ("This has been proposed
> earlier by Doug and did not get far due to concerns about the effect this
> would have on development on trunk.") (August 24, 2010)
>
> 2. Lots of +1s, between August 24 to August 30 2010. One particular
> comment is from Tom White: "I think it would be good to have a shared 0.20
> Apache security branch.
> Since security isn't in 0.21, and the 0.22 release is a some way off
> as you mention, this would be useful for folks who want the security
> features sooner (and want to use an Apache release)."
>
> 3. Arun volunteers to create a release (August 30, 2010)
>
> 4. Doug reminds Arun. (October 15, 2010)
>
> 5. Arun apologizes for not creating a branch because he was busy, because
> he had a baby. (January 11, 2011)
>
> 6. Lots of discussion about what to call it (the release, not the baby,
> although I had a good laugh at Patrick Angeles's email: "You're gonna call
> your kid 20.100?" ;-).
>
> 7. Arun proposes to call it 0.20.100: "I'm open to suggestions - how about
> something like 20.100 to show that it's a big jump? Anything else?" Jan
> 12, 2011
>
> 8. Among others, Eli says: "+1 on 0.20.x   (where x is a J > 3)" on Jan
> 12, 2011.
>
> So, as you can see, even if this release is called 0.20.x, the community
> agreed that these are valuable patches to have, and despite backward
> incompatibility, still have them in minor release.
>
> - milind
>
> --
> Milind Bhandarkar
> mbhandarkar@linkedin.com
> +1-650-776-3167
>
>
>
>
>
>
> On 5/6/11 11:14 PM, "Eric Sammer" <esammer@cloudera.com> wrote:
>
>> On May 6, 2011, at 4:53 AM, Steve Loughran <stevel@apache.org> wrote:
>>
>> I understand Eli's concerns that putting stuff in there that hasn't gone
>> into trunk yet is danger. However, as the team makes no guarantees of 100%
>> compatibility between releases, I don't think it's critical. It's just
>> something that needs to be addressed -which can be done after this release
>> has shipped.
>>
>>
>> I was under the impression that the community has been extremely strict
>> about compatibility between minor version bumps in the past. I though
>> there
>> were specific guarantees and that was one of the reasons certain behaviors
>> have persisted so long.
>>
>> Does this mean API changes can be made in minor releases and it can be
>> made
>> backward compatible in future releases? That seems very, very counter to
>> various conversations that have happened in the past. I'm of the mind that
>> we should continue to promise what we've always promised and if that's
>> changing, let's make with the refactoring party!
>>
>> Can some PMC'ers clarify this one for me?
>>
>> TIA.
>> Sammer
>>
>>
>>
>> -Steve
>

From general-return-3429-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sun May  8 01:36:44 2011
Return-Path: <general-return-3429-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4469B2F22
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sun,  8 May 2011 01:36:44 +0000 (UTC)
Received: (qmail 58464 invoked by uid 500); 8 May 2011 01:36:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58408 invoked by uid 500); 8 May 2011 01:36:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58400 invoked by uid 99); 8 May 2011 01:36:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 May 2011 01:36:42 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 May 2011 01:36:35 +0000
Received: by pve37 with SMTP id 37so2791828pve.35
        for <general@hadoop.apache.org>; Sat, 07 May 2011 18:36:13 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=mbn66dj47gc20U3UpvliEe9Lnu/63l8uT9W9sMLCH/c=;
        b=U++MXnMmpzQtFTi6+ZyqFIjUPvFVhDgUBSFBfceDKXdEi5OzlyFG9vT56tRa5I76dF
         tTrIYsL/ZqLNsOScGDkPTkUod37KtcaWyW8/ptZMFRFBROnfSa41Xt2aRRvjsxZzWe8d
         ZZMHId+bxRvNKfWVM2OffZet9hv/ipr11sfAc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=sfnzQ6SUgaNcYkzXJpNdH87sVbVQNgyDOCvC9xnlERQCqbWyBahaLOTveqsRmDPgLx
         NZOQAIDDKptMPlLJbNL3DN7krAjuI9BobzTAnZHX1UviMyrPUR3ukeT+vQAWiOT05G5m
         mOeoSMdI7ByARYeTs+jzV2PImr8FCyUEEcMVI=
Received: by 10.142.149.20 with SMTP id w20mr2992655wfd.137.1304818573457;
        Sat, 07 May 2011 18:36:13 -0700 (PDT)
Received: from [192.168.1.15] (124-148-142-55.dyn.iinet.net.au [124.148.142.55])
        by mx.google.com with ESMTPS id d3sm3136768pbh.73.2011.05.07.18.36.10
        (version=TLSv1/SSLv3 cipher=OTHER);
        Sat, 07 May 2011 18:36:12 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: Release compatibility was Re: [VOTE] Release candidate 0.20.203.0-rc1
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <-1022679743169743093@unknownmsgid>
Date: Sun, 8 May 2011 11:36:04 +1000
Content-Transfer-Encoding: quoted-printable
Message-Id: <5A50CB87-211B-4517-8E77-4AE0EEDC24F9@Holsman.NET>
References: <C9EA354D.87AE%mbhandarkar@linkedin.com> <-1022679743169743093@unknownmsgid>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org


On May 8, 2011, at 9:50 AM, Eric Sammer wrote:

> do we permit
> backward incompatible changes between 0.22.0 and 0.22.1 or is this
> something we've allowed just for the 203 release?

good question.
do we allow incompatible (smallish) features to be added to a 20.x =
release.
hoping that they will eventually be put into trunk at a later stage.
and if we need a process or something around it, or will just act on =
good faith that it will occur.=

From general-return-3430-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sun May  8 05:42:01 2011
Return-Path: <general-return-3430-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8386B3B7F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sun,  8 May 2011 05:42:01 +0000 (UTC)
Received: (qmail 45845 invoked by uid 500); 8 May 2011 05:41:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45506 invoked by uid 500); 8 May 2011 05:41:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45493 invoked by uid 99); 8 May 2011 05:41:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 May 2011 05:41:55 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of shv.hadoop@gmail.com designates 209.85.213.176 as permitted sender)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 May 2011 05:41:50 +0000
Received: by yxd5 with SMTP id 5so2202597yxd.35
        for <multiple recipients>; Sat, 07 May 2011 22:41:29 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=OKFDB1CnEXWwnKGi3kjZABUFerE0NradlI5M3RX7kuo=;
        b=kHJ69z7cxpXvlq4WdO6eIXZFeMfx3owTS5DLm2yZLz/6xdoPfqZwhcnqy4YKChPhSJ
         BrAh9JsG9z364IfgDxvO8a8JgVZTnlW+4odYLyc03hoTRDFK2uqsdz0MoXC1FKv+KuBY
         c7P61UAb97VTpltCkISWd3a9kSe7RE/81OnBE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=MyZ0FlhSYqfC0zwCOT5PmF/AC1o9HzmVZO+8gLwSkRYQpMtOCMde7FqkdHbdpKZSDW
         IOGiqIC3VEcW3xf+ab2EGnTTnima8ozukWt0fvwT2cUsln6b5kd0vvv7mYUS8DfQe43y
         fusuaQ/Z7andNSOVtW+mE2MJY2+zkTbYHP8Ig=
MIME-Version: 1.0
Received: by 10.236.109.19 with SMTP id r19mr918263yhg.289.1304833289001; Sat,
 07 May 2011 22:41:29 -0700 (PDT)
Received: by 10.147.124.12 with HTTP; Sat, 7 May 2011 22:41:28 -0700 (PDT)
In-Reply-To: <BANLkTi=JDBneLBHsCh1T-PF8C7wNbra0kw@mail.gmail.com>
References: <12853C59-08F1-4F98-ACF3-800F06BC7A5F@apache.org>
	<BANLkTinZ=xb6kJ5PTeLN5KKD9b-cwaM0OQ@mail.gmail.com>
	<BANLkTi=1sQoL68k_2UmA1_Oo2CyXsM3fAw@mail.gmail.com>
	<DBCE9300-0D9D-4A9A-99B4-878139A028EE@yahoo-inc.com>
	<61C1E656-0649-4097-8709-FDC596ED57E9@yahoo-inc.com>
	<F7921362-CC1B-435C-890A-E1BD181E03C0@yahoo-inc.com>
	<8D047C89-6430-4FDE-A9C4-CFEE79644AC6@Holsman.NET>
	<BANLkTi=JDBneLBHsCh1T-PF8C7wNbra0kw@mail.gmail.com>
Date: Sat, 7 May 2011 22:41:28 -0700
Message-ID: <BANLkTim68H=8ngBfzMsVrQOb9PmY7FviLQ@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc0
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: common-dev@hadoop.apache.org, general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0023547c8c5fae9c5604a2bd2c6f

--0023547c8c5fae9c5604a2bd2c6f
Content-Type: text/plain; charset=ISO-8859-1

-1 for rc1

I downloaded and ran the test target 3 times.

First run failed because my umask is defaulted to 0002, which is a known
problem HADOOP-5050 committed to 0.21 but not 0.20.
Set umask to 0022 and re-ran test twice. Both resulted in failure. Here is
the list of failed tests:
    [junit] Test org.apache.hadoop.mapred.TestJobTrackerRestart FAILED
(timeout)
    [junit] Test
org.apache.hadoop.mapred.TestJobTrackerRestartWithLostTracker FAILED
    [junit] Test org.apache.hadoop.mapred.TestJobTrackerSafeMode FAILED
    [junit] Test org.apache.hadoop.mapred.TestMiniMRMapRedDebugScript FAILED
    [junit] Test org.apache.hadoop.mapred.TestRecoveryManager FAILED
    [junit] Test org.apache.hadoop.mapred.TestTTMemoryReporting FAILED
    [junit] Test org.apache.hadoop.mapred.TestTaskTrackerLocalization FAILED
    [junit] Test org.apache.hadoop.hdfsproxy.TestHdfsProxy FAILED

I am in favor of releasing hadoop-0.20.203.
And we run a version of this release on a large cluster at eBay. I know it
works.
I understand the controversy behind it. I regret it hasn't been developed in
a true community way.
I think it nevertheless adds value to Apache Hadoop.
Lets just make sure it passes the tests.

Thanks,
--Konstantin



On Tue, May 3, 2011 at 1:39 AM, Konstantin Shvachko <shv.hadoop@gmail.com>wrote:

> I think its a good idea to release hadoop-0.20.203. It moves Apache Hadoop
> a step forward.
>
> Looks like the technical difficulties are resolved now with latest Arun's
> commits.
> Being a superset of hadoop-0.20.2 it can be considered based on one of the
> official Apache releases.
> I don't think there was a lack of discussions on the lists about the issues
> included in the release candidate. Todd did a thorough review of the entire
> security branch. Many developers participated in discussions.
> Agreeing with Stack I wish HBase was considered a primary target for Hadoop
> support. But it is not realistic to have it in hadoop-0.20.203.
> I have some experience running a version of this release candidate on a
> large cluster. It works. I would add a couple of patches, which make it run
> on Windows for me like HADOOP-7110, HADOOP-7126. But those are not blockers.
>
> Thanks,
> --Konstantin
>
>
> On Mon, May 2, 2011 at 5:12 PM, Ian Holsman <hadoop@holsman.net> wrote:
>
>>
>> On May 3, 2011, at 9:58 AM, Arun C Murthy wrote:
>>
>> >>
>> >> Owen, Suresh and I have committed everything on this list except
>> >> HADOOP-6386 and HADOOP-6428. Not sure which of the two are relevant/
>> >> necessary, I'll check with Cos.  Other than that hadoop-0.20.203 now a
>> >> superset of hadoop-0.20.2.
>> >>
>> >
>> > Missed adding HADOOP-5759 to that list, I'll check with Amareshwari
>> before committing.
>> >
>> > Arun
>>
>> Thanks for doing this so fast Arun.
>>
>>
>

--0023547c8c5fae9c5604a2bd2c6f--

From general-return-3431-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sun May  8 05:49:12 2011
Return-Path: <general-return-3431-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5AF0132A6
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sun,  8 May 2011 05:49:12 +0000 (UTC)
Received: (qmail 52798 invoked by uid 500); 8 May 2011 05:49:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52356 invoked by uid 500); 8 May 2011 05:49:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52348 invoked by uid 99); 8 May 2011 05:49:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 May 2011 05:49:10 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mbhandarkar@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 May 2011 05:49:05 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:In-Reply-To:Accept-Language:
   Content-Language:X-MS-Has-Attach:X-MS-TNEF-Correlator:
   x-originating-ip:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version;
  b=PKboxcdpv76KZLmDmMzMcn+Z5dZNyYVXoQWPtiwrMkeEiUGAwXIh1Hdy
   uPPhbt5CJhaoaw4YnCYmZjNELfvRvVo94T4q2TTFWgRedMM7Pk+qSU6y+
   5utBYIIOC5s+K5A;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=mbhandarkar@linkedin.com; q=dns/txt;
  s=proddkim; t=1304833745; x=1336369745;
  h=from:to:subject:date:message-id:in-reply-to:content-id:
   content-transfer-encoding:mime-version;
  bh=dO2QFKjqAOKU8F6Pmw0bQIXV6f6FNuEC5iYzVb21xII=;
  b=pJyZb107pLqaP9FDN3DaUUJjYvokbirTv1pAo0fmpC7p9wrOPgn5ERiq
   lNC4USO2GwfJV+VrnJ2k6MiGsh5Ha7Bn3XLcfi5m2GVPJaLyH5ou4sZ1K
   kP8xljmX1k6u+xd;
X-IronPort-AV: E=Sophos;i="4.64,333,1301900400"; 
   d="scan'208";a="22957725"
Received: from ESV4-EXC02.linkedin.biz ([fe80::4d74:48bd:e0bd:13ee]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi id 14.01.0251.000; Sat, 7
 May 2011 22:52:14 -0700
From: Milind Bhandarkar <mbhandarkar@linkedin.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: Release compatibility was Re: [VOTE] Release candidate
 0.20.203.0-rc1
Thread-Topic: Release compatibility was Re: [VOTE] Release candidate
 0.20.203.0-rc1
Thread-Index: AQHMCsM2u2v/kISbSx+vh7BYIk9Jf5R99lUAgAAUFYCAAh2PAIABM8OA//+VJoCAAZH4gP//7ssA
Date: Sun, 8 May 2011 05:52:13 +0000
Message-ID: <C9EB7862.883D%mbhandarkar@linkedin.com>
In-Reply-To: <-1022679743169743093@unknownmsgid>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <43F2F7A90BA20A4A8EE46A33E51A302A@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

[Mentioning again: I am not on the PMC, and this email contains
non-binding opinions based on my reading the general@hadoop.apache.org
emails.]

It is my understanding that, from the beginning, the 0.20+security was
always treated as an exception to the normal (I.e. Pre-0.20) release
process. (This has been confirmed by the mailing list threads, in which
many of those who are objecting to this release now - stating that it has
violated norms - have consented, actually argued for, breaking the norms.)

For whatever I have read on this mailing list before the vote for this
release, it looked like most of the community agreed that what Yahoo! Had
produced on their own branch, outside of Apache trunk, was important
contribution, and a release based on that would be a good idea, and that a
one-time release should proceed. (After all, whichever organization the
contributors belong to, many seem to indicate that they feel ashamed not
having an Apache release in more than a year.)

>From many emails on this thread, it has been clear to me, that it is a one
time concession given for parting ways from the normal process, and I hope
everyone understands that this is supposed to make Apache Hadoop releases
relevant once again.

So, to cut it short, the 0.20.203 backward incompatibilities etc have no
bearing on the "normal" process, in which no backward incompatibilities
should be allowed in minor releases. To answer your specific question, I
have no reason to believe that 0.22.1 could be backward incompatible with
0.22.0.=20
=20
- milind

--=20
Milind Bhandarkar
mbhandarkar@linkedin.com
+1-650-776-3167






On 5/7/11 4:50 PM, "Eric Sammer" <esammer@cloudera.com> wrote:

>Milind:
>
>Thanks for the pointer. I remember this thread. I guess my question
>was unrelated to the specific release and more about the general mode
>of development under normal release circumstances (ie. do we permit
>backward incompatible changes between 0.22.0 and 0.22.1 or is this
>something we've allowed just for the 203 release?).
>
>I think it's important to be clear about what the MO is so end users
>can plan upgrades appropriately.
>
>Thanks!
>Sammer
>
>On May 6, 2011, at 11:52 PM, Milind Bhandarkar <mbhandarkar@linkedin.com>
>wrote:
>
>> [I am not on PMC, but seeing that PMC may be busy with other issues, I
>> will try to answer your questions.]
>>
>> Eric,
>>
>> I think the thread
>>=20
>>"http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C1
>>8C
>> 5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com%3E" will answer your
>> questions. Here is the timeline as I see it:
>>
>> 1. Arun proposes to create a release from the security patchset. Says
>>Doug
>> has proposed this earlier
>>=20
>>(http://mail-archives.apache.org/mod_mbox/hadoop-general/201004.mbox/%3C4
>>BD
>> 1DFEA.5020908@apache.org%3E April 23, 2010) ("This has been proposed
>> earlier by Doug and did not get far due to concerns about the effect
>>this
>> would have on development on trunk.") (August 24, 2010)
>>
>> 2. Lots of +1s, between August 24 to August 30 2010. One particular
>> comment is from Tom White: "I think it would be good to have a shared
>>0.20
>> Apache security branch.
>> Since security isn't in 0.21, and the 0.22 release is a some way off
>> as you mention, this would be useful for folks who want the security
>> features sooner (and want to use an Apache release)."
>>
>> 3. Arun volunteers to create a release (August 30, 2010)
>>
>> 4. Doug reminds Arun. (October 15, 2010)
>>
>> 5. Arun apologizes for not creating a branch because he was busy,
>>because
>> he had a baby. (January 11, 2011)
>>
>> 6. Lots of discussion about what to call it (the release, not the baby,
>> although I had a good laugh at Patrick Angeles's email: "You're gonna
>>call
>> your kid 20.100?" ;-).
>>
>> 7. Arun proposes to call it 0.20.100: "I'm open to suggestions - how
>>about
>> something like 20.100 to show that it's a big jump? Anything else?" Jan
>> 12, 2011
>>
>> 8. Among others, Eli says: "+1 on 0.20.x   (where x is a J > 3)" on Jan
>> 12, 2011.
>>
>> So, as you can see, even if this release is called 0.20.x, the community
>> agreed that these are valuable patches to have, and despite backward
>> incompatibility, still have them in minor release.
>>
>> - milind
>>
>> --
>> Milind Bhandarkar
>> mbhandarkar@linkedin.com
>> +1-650-776-3167
>>
>>
>>
>>
>>
>>
>> On 5/6/11 11:14 PM, "Eric Sammer" <esammer@cloudera.com> wrote:
>>
>>> On May 6, 2011, at 4:53 AM, Steve Loughran <stevel@apache.org> wrote:
>>>
>>> I understand Eli's concerns that putting stuff in there that hasn't
>>>gone
>>> into trunk yet is danger. However, as the team makes no guarantees of
>>>100%
>>> compatibility between releases, I don't think it's critical. It's just
>>> something that needs to be addressed -which can be done after this
>>>release
>>> has shipped.
>>>
>>>
>>> I was under the impression that the community has been extremely strict
>>> about compatibility between minor version bumps in the past. I though
>>> there
>>> were specific guarantees and that was one of the reasons certain
>>>behaviors
>>> have persisted so long.
>>>
>>> Does this mean API changes can be made in minor releases and it can be
>>> made
>>> backward compatible in future releases? That seems very, very counter
>>>to
>>> various conversations that have happened in the past. I'm of the mind
>>>that
>>> we should continue to promise what we've always promised and if that's
>>> changing, let's make with the refactoring party!
>>>
>>> Can some PMC'ers clarify this one for me?
>>>
>>> TIA.
>>> Sammer
>>>
>>>
>>>
>>> -Steve
>>


From general-return-3432-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sun May  8 17:53:54 2011
Return-Path: <general-return-3432-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D4B663B04
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sun,  8 May 2011 17:53:53 +0000 (UTC)
Received: (qmail 79913 invoked by uid 500); 8 May 2011 17:53:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79853 invoked by uid 500); 8 May 2011 17:53:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 79845 invoked by uid 99); 8 May 2011 17:53:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 May 2011 17:53:51 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 May 2011 17:53:45 +0000
Received: from [10.0.1.3] (snvvpn3-10-72-76-c144.hq.corp.yahoo.com [10.72.76.144])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p48HrBnO069878
	for <general@hadoop.apache.org>; Sun, 8 May 2011 10:53:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1304877192;
	bh=mB0wTn1vc/mDLC9Xisa7psGAyst7x05I9sBCSi7gKKg=;
	h=Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:
	 Content-Transfer-Encoding:Message-Id:References:To;
	b=CsE6itbxjPhujEko8makgzlnKPLWXXTCxxsTG6toVUrjoLJJopmFTHVtVB5DzJNz6
	 QhpEE1EcqwCR+C1gQgsVtYVFi284tCbmgA91zWIAOkfxblrcnOlbp6yGktEMI6vTiK
	 vvwl1Z7mbGfNPQLtSZOg07/aQ5d38sw9MfC63msY=
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: Apache Hadoop Hackathons: 5/11 and 5/18 in SF and Palo Alto
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <BANLkTinrJ6E4KpzHT2mR0kN4k=N6v2ePdw@mail.gmail.com>
Date: Sun, 8 May 2011 10:53:19 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <5E9CCC0F-AD6F-4BE1-83CF-ACDD4D55BE75@yahoo-inc.com>
References: <BANLkTiksqFvXdN68Ki4ZK0y=GLmr7s=MJg@mail.gmail.com> <6D26A411-537F-4719-B95F-5C0A68077F34@yahoo-inc.com> <BANLkTinrJ6E4KpzHT2mR0kN4k=N6v2ePdw@mail.gmail.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Folks,

I guess this requires a clarification.

1) Of course I remain at Yahoo.

2) The yahoo hadoop team is in not way associated with this event.  This =
email chain is the first we've heard of it.  Sorry for our confusion.

3) Of course we're always happy to see folks investing in Hadoop.  It =
will be interesting to see if this proves an effective approach. =20

Thanks

E14

On May 6, 2011, at 8:34 PM, Konstantin Boudnik wrote:

> On Fri, May 6, 2011 at 17:46, Eric Baldeschwieler =
<eric14@yahoo-inc.com> wrote:
>> Hi Jeff,
>>=20
>> Could you put me in contact with the folks from yahoo you are talking =
to?
>=20
> Aren't you at Y! anymore, Eric? Wow!
>=20
>> Thanks,
>>=20
>> E14
>>=20
>> On May 6, 2011, at 12:03 PM, Jeff Hammerbacher wrote:
>>=20
>>> Hey,
>>>=20
>>> The discussion this week about the 0.20.203.0 release has done a =
great job
>>> of highlighting some issues in our development process; it's also =
done a
>>> great job of lifting our mailing list activity metrics. After =
reading
>>> through the various threads, it's clear that everyone agrees on two =
things:
>>>=20
>>> 1) We'd like to get the work done in the 0.20.x branches into trunk
>>> 2) We'd like to start releasing off of trunk again
>>>=20
>>> To that end, a few folks from Yahoo!, Cloudera, StumbleUpon, and =
Facebook
>>> would like to put together a series of Hackathons to 1) burn down =
the 13
>>> (only 13!) remaining blockers for the 0.22 release and 2) =
forward-port the
>>> work done in the 0.20.x branches into trunk.
>>>=20
>>> Cloudera will be hosting Hackathons from 10 am to 6 pm in both our =
Palo Alto
>>> and San Francisco offices next Wednesday, 5/11, and the following =
Wednesday,
>>> 5/18, to ensure both of these tasks get completed in short order. =
PMC
>>> members Todd Lipcon and Tom White will lead the SF group and PMC =
members Eli
>>> Collins and Patrick Hunt will lead the Palo Alto group.
>>>=20
>>> Whether you're a long-time contributor or don't have a patch to your =
name,
>>> now is a great time to get involved in Apache Hadoop development. =
Forward
>>> porting patches is a lot easier than writing them from scratch, and =
you'll
>>> have mentors present to help guide you through the patch testing and
>>> submission process.
>>>=20
>>> To sign up for the 5/11 Hackathon in either SF or PA, head over to
>>> http://hadoophackathon.eventbrite.com.
>>>=20
>>> As a reminder, the SF HUG will be held on 5/11 at 6 pm in the =
Cloudera SF
>>> offices: http://www.meetup.com/hadoopsf/events/17354462/. If you =
can't make
>>> it on 5/11, we'll send out a link next week for the 5/18 event.
>>>=20
>>> Looking forward to getting the release train moving again!
>>>=20
>>> Regards,
>>> Jeff
>>>=20
>>> p.s. if you'd like to participate remotely, email me directly and =
I'll see
>>> about how we can teleconference you into the event.
>>=20
>>=20


From general-return-3433-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sun May  8 18:13:08 2011
Return-Path: <general-return-3433-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8BD5830CA
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sun,  8 May 2011 18:13:08 +0000 (UTC)
Received: (qmail 87331 invoked by uid 500); 8 May 2011 18:13:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87276 invoked by uid 500); 8 May 2011 18:13:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87267 invoked by uid 99); 8 May 2011 18:13:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 May 2011 18:13:06 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 May 2011 18:13:01 +0000
Received: from [10.0.1.3] (snvvpn3-10-72-76-c144.hq.corp.yahoo.com [10.72.76.144])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p48IAMuT074091
	for <general@hadoop.apache.org>; Sun, 8 May 2011 11:10:23 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1304878223;
	bh=u92a/mqwSr8ZvbhL58FAeY2iwh8uzmxP+fNOZjrrP8E=;
	h=Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:
	 Content-Transfer-Encoding:Message-Id:References:To;
	b=s8b4d4CiFL408OYjMODKlGp9hVBV5eh3xyRAtWGh2EC4Mx6loS6GQLuG5ztrXSQZw
	 tz5fIVy6o1HpZdeyQs8xbrA1Hejw9aR9GFghDu1icLZN7v/B8RCmURZsBSrJP5M2v2
	 I5Bdob3izfpHfpQR/ggBBdydDgIRJDc8v0KcMVc4=
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: Release compatibility was Re: [VOTE] Release candidate 0.20.203.0-rc1
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <C9EB7862.883D%mbhandarkar@linkedin.com>
Date: Sun, 8 May 2011 11:10:22 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <215257F9-1F66-4A4D-B4CF-8999DC509866@yahoo-inc.com>
References: <C9EB7862.883D%mbhandarkar@linkedin.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1084)

I'd agree with this too. [same disclaimer as milind, not on PMC]

In general one would not expect to see an incompatible change added in a =
dot release (0.24.1 0.24.2).  I'd expect anything like that to require =
community discussion and support.

As milind summarized, we seem to have support for the addition of =
security to 20.  The existing mechanism of the required release vote =
will confirm or deny that.

I think it is important that compatible enhancements to hadoop are =
allowed into dot releases.  This is something that we've discussed but =
never finalized in the community.  It is the desire to put improvements =
into users hands more quickly that the next major release that drives =
orgs to produce private releases of hadoop.  In general, I think it is =
fair that such changes go into trunk first.  Exceptions to that also =
need discussion and support IMO.

I think the key to making progress is discussion and the idea that =
majority support, not consensus is what is needed to make exceptions to =
our process.  Process is useful, it reduces friction.  Process without =
exception is stifling. =20

On May 7, 2011, at 10:52 PM, Milind Bhandarkar wrote:

> [Mentioning again: I am not on the PMC, and this email contains
> non-binding opinions based on my reading the general@hadoop.apache.org
> emails.]
>=20
> It is my understanding that, from the beginning, the 0.20+security was
> always treated as an exception to the normal (I.e. Pre-0.20) release
> process. (This has been confirmed by the mailing list threads, in =
which
> many of those who are objecting to this release now - stating that it =
has
> violated norms - have consented, actually argued for, breaking the =
norms.)
>=20
> For whatever I have read on this mailing list before the vote for this
> release, it looked like most of the community agreed that what Yahoo! =
Had
> produced on their own branch, outside of Apache trunk, was important
> contribution, and a release based on that would be a good idea, and =
that a
> one-time release should proceed. (After all, whichever organization =
the
> contributors belong to, many seem to indicate that they feel ashamed =
not
> having an Apache release in more than a year.)
>=20
> =46rom many emails on this thread, it has been clear to me, that it is =
a one
> time concession given for parting ways from the normal process, and I =
hope
> everyone understands that this is supposed to make Apache Hadoop =
releases
> relevant once again.
>=20
> So, to cut it short, the 0.20.203 backward incompatibilities etc have =
no
> bearing on the "normal" process, in which no backward =
incompatibilities
> should be allowed in minor releases. To answer your specific question, =
I
> have no reason to believe that 0.22.1 could be backward incompatible =
with
> 0.22.0.=20
>=20
> - milind
>=20
> --=20
> Milind Bhandarkar
> mbhandarkar@linkedin.com
> +1-650-776-3167
>=20
>=20
>=20
>=20
>=20
>=20
> On 5/7/11 4:50 PM, "Eric Sammer" <esammer@cloudera.com> wrote:
>=20
>> Milind:
>>=20
>> Thanks for the pointer. I remember this thread. I guess my question
>> was unrelated to the specific release and more about the general mode
>> of development under normal release circumstances (ie. do we permit
>> backward incompatible changes between 0.22.0 and 0.22.1 or is this
>> something we've allowed just for the 203 release?).
>>=20
>> I think it's important to be clear about what the MO is so end users
>> can plan upgrades appropriately.
>>=20
>> Thanks!
>> Sammer
>>=20
>> On May 6, 2011, at 11:52 PM, Milind Bhandarkar =
<mbhandarkar@linkedin.com>
>> wrote:
>>=20
>>> [I am not on PMC, but seeing that PMC may be busy with other issues, =
I
>>> will try to answer your questions.]
>>>=20
>>> Eric,
>>>=20
>>> I think the thread
>>>=20
>>> =
"http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C1
>>> 8C
>>> 5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com%3E" will answer your
>>> questions. Here is the timeline as I see it:
>>>=20
>>> 1. Arun proposes to create a release from the security patchset. =
Says
>>> Doug
>>> has proposed this earlier
>>>=20
>>> =
(http://mail-archives.apache.org/mod_mbox/hadoop-general/201004.mbox/%3C4
>>> BD
>>> 1DFEA.5020908@apache.org%3E April 23, 2010) ("This has been proposed
>>> earlier by Doug and did not get far due to concerns about the effect
>>> this
>>> would have on development on trunk.") (August 24, 2010)
>>>=20
>>> 2. Lots of +1s, between August 24 to August 30 2010. One particular
>>> comment is from Tom White: "I think it would be good to have a =
shared
>>> 0.20
>>> Apache security branch.
>>> Since security isn't in 0.21, and the 0.22 release is a some way off
>>> as you mention, this would be useful for folks who want the security
>>> features sooner (and want to use an Apache release)."
>>>=20
>>> 3. Arun volunteers to create a release (August 30, 2010)
>>>=20
>>> 4. Doug reminds Arun. (October 15, 2010)
>>>=20
>>> 5. Arun apologizes for not creating a branch because he was busy,
>>> because
>>> he had a baby. (January 11, 2011)
>>>=20
>>> 6. Lots of discussion about what to call it (the release, not the =
baby,
>>> although I had a good laugh at Patrick Angeles's email: "You're =
gonna
>>> call
>>> your kid 20.100?" ;-).
>>>=20
>>> 7. Arun proposes to call it 0.20.100: "I'm open to suggestions - how
>>> about
>>> something like 20.100 to show that it's a big jump? Anything else?" =
Jan
>>> 12, 2011
>>>=20
>>> 8. Among others, Eli says: "+1 on 0.20.x   (where x is a J > 3)" on =
Jan
>>> 12, 2011.
>>>=20
>>> So, as you can see, even if this release is called 0.20.x, the =
community
>>> agreed that these are valuable patches to have, and despite backward
>>> incompatibility, still have them in minor release.
>>>=20
>>> - milind
>>>=20
>>> --
>>> Milind Bhandarkar
>>> mbhandarkar@linkedin.com
>>> +1-650-776-3167
>>>=20
>>>=20
>>>=20
>>>=20
>>>=20
>>>=20
>>> On 5/6/11 11:14 PM, "Eric Sammer" <esammer@cloudera.com> wrote:
>>>=20
>>>> On May 6, 2011, at 4:53 AM, Steve Loughran <stevel@apache.org> =
wrote:
>>>>=20
>>>> I understand Eli's concerns that putting stuff in there that hasn't
>>>> gone
>>>> into trunk yet is danger. However, as the team makes no guarantees =
of
>>>> 100%
>>>> compatibility between releases, I don't think it's critical. It's =
just
>>>> something that needs to be addressed -which can be done after this
>>>> release
>>>> has shipped.
>>>>=20
>>>>=20
>>>> I was under the impression that the community has been extremely =
strict
>>>> about compatibility between minor version bumps in the past. I =
though
>>>> there
>>>> were specific guarantees and that was one of the reasons certain
>>>> behaviors
>>>> have persisted so long.
>>>>=20
>>>> Does this mean API changes can be made in minor releases and it can =
be
>>>> made
>>>> backward compatible in future releases? That seems very, very =
counter
>>>> to
>>>> various conversations that have happened in the past. I'm of the =
mind
>>>> that
>>>> we should continue to promise what we've always promised and if =
that's
>>>> changing, let's make with the refactoring party!
>>>>=20
>>>> Can some PMC'ers clarify this one for me?
>>>>=20
>>>> TIA.
>>>> Sammer
>>>>=20
>>>>=20
>>>>=20
>>>> -Steve
>>>=20
>=20


From general-return-3434-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sun May  8 18:44:28 2011
Return-Path: <general-return-3434-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D92C131E5
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sun,  8 May 2011 18:44:28 +0000 (UTC)
Received: (qmail 16278 invoked by uid 500); 8 May 2011 18:44:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16192 invoked by uid 500); 8 May 2011 18:44:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16184 invoked by uid 99); 8 May 2011 18:44:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 May 2011 18:44:27 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.171] (HELO mail-px0-f171.google.com) (209.85.212.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 May 2011 18:44:22 +0000
Received: by pxi7 with SMTP id 7so3692618pxi.2
        for <general@hadoop.apache.org>; Sun, 08 May 2011 11:44:02 -0700 (PDT)
Received: by 10.68.37.36 with SMTP id v4mr1825715pbj.76.1304880242057; Sun, 08
 May 2011 11:44:02 -0700 (PDT)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.68.41.38 with HTTP; Sun, 8 May 2011 11:43:42 -0700 (PDT)
In-Reply-To: <5E9CCC0F-AD6F-4BE1-83CF-ACDD4D55BE75@yahoo-inc.com>
References: <BANLkTiksqFvXdN68Ki4ZK0y=GLmr7s=MJg@mail.gmail.com>
 <6D26A411-537F-4719-B95F-5C0A68077F34@yahoo-inc.com> <BANLkTinrJ6E4KpzHT2mR0kN4k=N6v2ePdw@mail.gmail.com>
 <5E9CCC0F-AD6F-4BE1-83CF-ACDD4D55BE75@yahoo-inc.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Sun, 8 May 2011 11:43:42 -0700
X-Google-Sender-Auth: TJWy8hekiuNj_FPlrjEoe_VOgzo
Message-ID: <BANLkTimWNYk3zZ3crdKU=uxb26YYAkha9Q@mail.gmail.com>
Subject: Re: Apache Hadoop Hackathons: 5/11 and 5/18 in SF and Palo Alto
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Thank you for clarification Eric!

I got a private email suggesting that my original comment might have
created certain friction and whatnot. For those who suspect me of evil
deeds I want to clarify my response too - I was genuinely surprised by
your reply to the OP and put this amusement in the email.

I am glad that we can put this confusion behind us now!

Thanks,
  Cos

On Sun, May 8, 2011 at 10:53, Eric Baldeschwieler <eric14@yahoo-inc.com> wr=
ote:
> Hi Folks,
>
> I guess this requires a clarification.
>
> 1) Of course I remain at Yahoo.
>
> 2) The yahoo hadoop team is in not way associated with this event. =A0Thi=
s email chain is the first we've heard of it. =A0Sorry for our confusion.
>
> 3) Of course we're always happy to see folks investing in Hadoop. =A0It w=
ill be interesting to see if this proves an effective approach.
>
> Thanks
>
> E14
>
> On May 6, 2011, at 8:34 PM, Konstantin Boudnik wrote:
>
>> On Fri, May 6, 2011 at 17:46, Eric Baldeschwieler <eric14@yahoo-inc.com>=
 wrote:
>>> Hi Jeff,
>>>
>>> Could you put me in contact with the folks from yahoo you are talking t=
o?
>>
>> Aren't you at Y! anymore, Eric? Wow!
>>
>>> Thanks,
>>>
>>> E14
>>>
>>> On May 6, 2011, at 12:03 PM, Jeff Hammerbacher wrote:
>>>
>>>> Hey,
>>>>
>>>> The discussion this week about the 0.20.203.0 release has done a great=
 job
>>>> of highlighting some issues in our development process; it's also done=
 a
>>>> great job of lifting our mailing list activity metrics. After reading
>>>> through the various threads, it's clear that everyone agrees on two th=
ings:
>>>>
>>>> 1) We'd like to get the work done in the 0.20.x branches into trunk
>>>> 2) We'd like to start releasing off of trunk again
>>>>
>>>> To that end, a few folks from Yahoo!, Cloudera, StumbleUpon, and Faceb=
ook
>>>> would like to put together a series of Hackathons to 1) burn down the =
13
>>>> (only 13!) remaining blockers for the 0.22 release and 2) forward-port=
 the
>>>> work done in the 0.20.x branches into trunk.
>>>>
>>>> Cloudera will be hosting Hackathons from 10 am to 6 pm in both our Pal=
o Alto
>>>> and San Francisco offices next Wednesday, 5/11, and the following Wedn=
esday,
>>>> 5/18, to ensure both of these tasks get completed in short order. PMC
>>>> members Todd Lipcon and Tom White will lead the SF group and PMC membe=
rs Eli
>>>> Collins and Patrick Hunt will lead the Palo Alto group.
>>>>
>>>> Whether you're a long-time contributor or don't have a patch to your n=
ame,
>>>> now is a great time to get involved in Apache Hadoop development. Forw=
ard
>>>> porting patches is a lot easier than writing them from scratch, and yo=
u'll
>>>> have mentors present to help guide you through the patch testing and
>>>> submission process.
>>>>
>>>> To sign up for the 5/11 Hackathon in either SF or PA, head over to
>>>> http://hadoophackathon.eventbrite.com.
>>>>
>>>> As a reminder, the SF HUG will be held on 5/11 at 6 pm in the Cloudera=
 SF
>>>> offices: http://www.meetup.com/hadoopsf/events/17354462/. If you can't=
 make
>>>> it on 5/11, we'll send out a link next week for the 5/18 event.
>>>>
>>>> Looking forward to getting the release train moving again!
>>>>
>>>> Regards,
>>>> Jeff
>>>>
>>>> p.s. if you'd like to participate remotely, email me directly and I'll=
 see
>>>> about how we can teleconference you into the event.
>>>
>>>
>
>

From general-return-3435-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sun May  8 19:56:29 2011
Return-Path: <general-return-3435-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ECFA63CA9
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sun,  8 May 2011 19:56:28 +0000 (UTC)
Received: (qmail 56576 invoked by uid 500); 8 May 2011 19:56:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56525 invoked by uid 500); 8 May 2011 19:56:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56517 invoked by uid 99); 8 May 2011 19:56:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 May 2011 19:56:27 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of eli@cloudera.com designates 209.85.215.176 as permitted sender)
Received: from [209.85.215.176] (HELO mail-ey0-f176.google.com) (209.85.215.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 May 2011 19:56:22 +0000
Received: by eya28 with SMTP id 28so1973164eya.35
        for <general@hadoop.apache.org>; Sun, 08 May 2011 12:56:01 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.14.50.196 with SMTP id z44mr2887523eeb.227.1304884560838; Sun,
 08 May 2011 12:56:00 -0700 (PDT)
Received: by 10.14.37.13 with HTTP; Sun, 8 May 2011 12:56:00 -0700 (PDT)
In-Reply-To: <5A50CB87-211B-4517-8E77-4AE0EEDC24F9@Holsman.NET>
References: <C9EA354D.87AE%mbhandarkar@linkedin.com>
	<-1022679743169743093@unknownmsgid>
	<5A50CB87-211B-4517-8E77-4AE0EEDC24F9@Holsman.NET>
Date: Sun, 8 May 2011 12:56:00 -0700
Message-ID: <BANLkTinbSXLOmsTnhvoF8T1HK2WFuzGx1A@mail.gmail.com>
Subject: Re: Release compatibility was Re: [VOTE] Release candidate 0.20.203.0-rc1
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Sat, May 7, 2011 at 6:36 PM, Ian Holsman <hadoop@holsman.net> wrote:
>
> On May 8, 2011, at 9:50 AM, Eric Sammer wrote:
>
>> do we permit
>> backward incompatible changes between 0.22.0 and 0.22.1 or is this
>> something we've allowed just for the 203 release?
>
> good question.
> do we allow incompatible (smallish) features to be added to a 20.x release.
> hoping that they will eventually be put into trunk at a later stage.
> and if we need a process or something around it, or will just act on good faith that it will occur.

We do allow it, as the 203 release shows. I don't think we need an
official process, our existing practice of filing a jira with the
appropriate fix version is sufficient. And this seems to be happening
already for all changes to future 20x releases.

Compatibility is just one reason it's important that features be
developed on trunk first. Security and other enhancements were
developed on 20 and forward ported to trunk. Almost two years later
the forward porting is still not complete - if we released from trunk
today, it would not support security. Ditto for append. Some people
who work on HBase consider the code in branch-20-append to be more
reliable than the code on trunk. This is the primary reason why people
are currently on 20.x releases.

People are of course free to contribute to Apache on whatever branch
they please, but I think as a development community we need to try to
make sure a future release off trunk is not a regression against
previous releases. This won't happen unless we invest in trunk. I
support a release from branch-20-security, I just wanted to see the
work go into trunk first. Ie I think the staging matters. (and I agree
with Ray wrt the version scheme but that's independent).

Fortunately, I don't think we're far off. The vast majority of
security work is in trunk (thanks to all those who did the porting),
there's probably only 50 to 100 bugs/enhancements in
branch-20-security not yet in trunk, and the append code (or rather
sync) to support HBase just needs some tests and debugging.

I agree with Eric that is is the desire to put improvements into users
hands more quickly that drives orgs to produce releases of Hadoop
outside Apache. I think the best way to address this is to get solid
releases coming off trunk on a regular basis again. Hence the push to
close out 22 and work on getting trunk in shape. Also, as a
development community, we're at our best when collaborating on trunk.

Thanks,
Eli

From general-return-3436-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May  9 21:37:25 2011
Return-Path: <general-return-3436-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ED6DA49DD
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon,  9 May 2011 21:37:24 +0000 (UTC)
Received: (qmail 46218 invoked by uid 500); 9 May 2011 21:37:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46128 invoked by uid 500); 9 May 2011 21:37:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46120 invoked by uid 99); 9 May 2011 21:37:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 21:37:22 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of hammer@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 21:37:18 +0000
Received: by bwz8 with SMTP id 8so7455233bwz.35
        for <general@hadoop.apache.org>; Mon, 09 May 2011 14:36:57 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.19.65 with SMTP id z1mr1004865bka.202.1304977016867; Mon,
 09 May 2011 14:36:56 -0700 (PDT)
Received: by 10.204.37.204 with HTTP; Mon, 9 May 2011 14:36:56 -0700 (PDT)
In-Reply-To: <BANLkTiksqFvXdN68Ki4ZK0y=GLmr7s=MJg@mail.gmail.com>
References: <BANLkTiksqFvXdN68Ki4ZK0y=GLmr7s=MJg@mail.gmail.com>
Date: Mon, 9 May 2011 14:36:56 -0700
Message-ID: <BANLkTi=5n7g+xB=kZ+RgGvwstkkb=opasA@mail.gmail.com>
Subject: Re: Apache Hadoop Hackathons: 5/11 and 5/18 in SF and Palo Alto
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000325558c3687caea04a2dea329

--000325558c3687caea04a2dea329
Content-Type: text/plain; charset=ISO-8859-1

Hey,

We've got 23 folks signed up for the Hackathon this Wednesday from
organizations like Cloudera, Facebook, Twitter, AOL, Trend Micro,
StumbleUpon, and Ngmoco. We'll also have the VP of the HBase PMC, Michael
Stack, and the VP of the Hive PMC, John Sichi. If you've been waiting to get
involved in Apache Hadoop, development, now is the time!

We've got room for 10 - 15 more. If you're in San Francisco or Palo Alto and
want to help out with Apache Hadoop development, sign up at
http://hadoophackathon.eventbrite.com.

Regards,
Jeff

On Fri, May 6, 2011 at 12:03 PM, Jeff Hammerbacher <hammer@cloudera.com>wrote:

> Hey,
>
> The discussion this week about the 0.20.203.0 release has done a great job
> of highlighting some issues in our development process; it's also done a
> great job of lifting our mailing list activity metrics. After reading
> through the various threads, it's clear that everyone agrees on two things:
>
> 1) We'd like to get the work done in the 0.20.x branches into trunk
> 2) We'd like to start releasing off of trunk again
>
> To that end, a few folks from Yahoo!, Cloudera, StumbleUpon, and Facebook
> would like to put together a series of Hackathons to 1) burn down the 13
> (only 13!) remaining blockers for the 0.22 release and 2) forward-port the
> work done in the 0.20.x branches into trunk.
>
> Cloudera will be hosting Hackathons from 10 am to 6 pm in both our Palo
> Alto and San Francisco offices next Wednesday, 5/11, and the following
> Wednesday, 5/18, to ensure both of these tasks get completed in short order.
> PMC members Todd Lipcon and Tom White will lead the SF group and PMC members
> Eli Collins and Patrick Hunt will lead the Palo Alto group.
>
> Whether you're a long-time contributor or don't have a patch to your name,
> now is a great time to get involved in Apache Hadoop development. Forward
> porting patches is a lot easier than writing them from scratch, and you'll
> have mentors present to help guide you through the patch testing and
> submission process.
>
> To sign up for the 5/11 Hackathon in either SF or PA, head over to
> http://hadoophackathon.eventbrite.com.
>
> As a reminder, the SF HUG will be held on 5/11 at 6 pm in the Cloudera SF
> offices: http://www.meetup.com/hadoopsf/events/17354462/. If you can't
> make it on 5/11, we'll send out a link next week for the 5/18 event.
>
> Looking forward to getting the release train moving again!
>
> Regards,
> Jeff
>
> p.s. if you'd like to participate remotely, email me directly and I'll see
> about how we can teleconference you into the event.
>
>
>

--000325558c3687caea04a2dea329--

From general-return-3437-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 10:30:23 2011
Return-Path: <general-return-3437-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D8EB65742
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 10 May 2011 10:30:23 +0000 (UTC)
Received: (qmail 84385 invoked by uid 500); 10 May 2011 10:30:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84332 invoked by uid 500); 10 May 2011 10:30:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84324 invoked by uid 99); 10 May 2011 10:30:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 10:30:21 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 10:30:13 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 5884C1BA75C
	for <general@hadoop.apache.org>; Tue, 10 May 2011 11:29:52 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id a7otMClnoYiv for <general@hadoop.apache.org>;
	Tue, 10 May 2011 11:29:51 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by colossus.hpl.hp.com (Postfix) with ESMTP id A0C9A1BA757
	for <general@hadoop.apache.org>; Tue, 10 May 2011 11:29:51 +0100 (BST)
MailScanner-NULL-Check: 1305628179.82761@0fnggZPH7zafn6yhccly9w
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p4AATcIP000110
	for <general@hadoop.apache.org>; Tue, 10 May 2011 11:29:38 +0100 (BST)
Message-ID: <4DC91392.2010308@apache.org>
Date: Tue, 10 May 2011 11:29:38 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.14) Gecko/20110223 Thunderbird/3.1.8
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Defining Hadoop Compatibility -revisiting-
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p4AATcIP000110
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org


Back in Jan 2011, I started a discussion about how to define Apache 
Hadoop Compatibility:
http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C4D46B6AD.2020802@apache.org%3E

I am now reading EMC HD "Enterprise Ready" Apache Hadoop datasheet

http://www.greenplum.com/sites/default/files/EMC_Greenplum_HD_DS_Final_1.pdf

It claims that their implementations are 100% compatible, even though 
the Enterprise edition uses a C filesystem. It also claims that both 
their software releases contain "Certified Stacks", without defining 
what Certified means, or who does the certification -only that it is an 
improvement.


I think we should revisit this issue before people with their own 
agendas define what compatibility with Apache Hadoop is for us


Licensing
-Use of the Hadoop codebase must follow the Apache License
http://www.apache.org/licenses/LICENSE-2.0
-plug in components that are dynamically linked to (Filesystems and 
schedulers) don't appear to be derivative works on my reading of this,

Naming
  -this is something for branding@apache, they will have their opinions. 
The key one is that the name "Apache Hadoop" must get used, and it's 
important to make clear it is a derivative work.
  -I don't think you can claim to have a Distribution/Fork/Version of 
Apache Hadoop if you swap out big chunks of it for alternate 
filesystems, MR engines, etc. Some description of this is needed
"Supports the Apache Hadoop MapReduce engine on top of Filesystem XYZ"

Compatibility
  -the definition of the Hadoop interfaces and classes is the Apache 
Source tree,
  -the definition of semantics of the Hadoop interfaces and classes is 
the Apache Source tree, including the test classes.
  -the verification that the actual semantics of an Apache Hadoop 
release is compatible with the expected semantics is that current and 
future tests pass
  -bug reports can highlight incompatibility with expectations of 
community users, and once incorporated into tests form part of the 
compatibility testing
  -vendors can claim and even certify their derivative works as 
compatible with other versions of their derivative works, but cannot 
claim compatibility with Apache Hadoop unless their code passes the 
tests and is consistent with the bug reports marked as ("by design"). 
Perhaps we should have tests that verify each of these "by design" 
bugreps to make them more formal.

Certification
  -I have no idea what this means in EMC's case, they just say "Certified"
  -As we don't do any certification ourselves, it would seem impossible 
for us to certify that any derivative work is compatible.
  -It may be best to state that nobody can certify their derivative as 
"compatible with Apache Hadoop" unless it passes all current test suites
  -And require that anyone who declares compatibility define what they 
mean by this

This is a good argument for getting more functional tests out there 
-whoever has more functional tests needs to get them into a test module 
that can be used to test real deployments.


From general-return-3438-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 18:06:34 2011
Return-Path: <general-return-3438-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 05B2F5902
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 10 May 2011 18:06:34 +0000 (UTC)
Received: (qmail 55892 invoked by uid 500); 10 May 2011 18:06:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55833 invoked by uid 500); 10 May 2011 18:06:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55825 invoked by uid 99); 10 May 2011 18:06:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 18:06:31 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of scott@richrelevance.com designates 64.78.17.18 as permitted sender)
Received: from [64.78.17.18] (HELO EXHUB018-3.exch018.msoutlookonline.net) (64.78.17.18)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 18:06:25 +0000
Received: from EXVMBX018-1.exch018.msoutlookonline.net ([64.78.17.47]) by
 EXHUB018-3.exch018.msoutlookonline.net ([64.78.17.18]) with mapi; Tue, 10 May
 2011 11:06:04 -0700
From: Scott Carey <scott@richrelevance.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: Steve Loughran <stevel@apache.org>
Date: Tue, 10 May 2011 11:06:22 -0700
Subject: Re: [DISCUSSION] development process of Hadoop
Thread-Topic: [DISCUSSION] development process of Hadoop
Thread-Index: AcwPPO0kpPpTBfNVTQusiHpJlrubyQ==
Message-ID: <C9EECA2C.3352F%scott@richrelevance.com>
In-Reply-To: <4DC402B7.8070900@uci.cu>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-MacOutlook/14.10.0.110310
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org



On 5/6/11 7:16 AM, "Marcos Ortiz" <mlortiz@uci.cu> wrote:

>>
>>
>+1 for Git
>
>We migrated from SVN to Git for our completed infrastructure, for many
>reason:
>- Git use much less space than SVN, all the changes are in a single .git

FWIW, svn 1.7 will have a single DB file too.  Though that project has
some chaos at the moment too and the release of 1.7 may be soon or a ways
away.  It is still slow over the network compared to git.  It is also
adding 'svn patch'.

>- Git is awesome for branching
>- Another great advantage is that there are many developers that know
>Git, and how the development process can be greatly improved.

There are also many developers who know svn, and many who don't know git.
That is not a clear win.

>
>PostgreSQL, one of my favorites open source projects that I use on my
>daily work, migrated the development process to Git from CVS.

Almost anything is better than CVS.


I don't feel that the primary cause of hadoop's situation is due to svn.
Git would help with merging patches that have become stale for sure, and
especially help on the client side for developers who need maintain many
concurrent contexts.  But there are many significant process issues at the
heart of the problem that are not due to the tools.

>
>Regards.
>
>--=20
>Marcos Lu=EDs Ort=EDz Valmaseda
>  Software Engineer (Large-Scaled Distributed Systems)
>  University of Information Sciences,
>  La Habana, Cuba
>  Linux User # 418229
>  http://about.me/marcosortiz
>


From general-return-3439-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 19:41:49 2011
Return-Path: <general-return-3439-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BECAE5C57
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 10 May 2011 19:41:49 +0000 (UTC)
Received: (qmail 21639 invoked by uid 500); 10 May 2011 19:41:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21583 invoked by uid 500); 10 May 2011 19:41:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21575 invoked by uid 99); 10 May 2011 19:41:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 19:41:48 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of scott@richrelevance.com designates 64.78.17.19 as permitted sender)
Received: from [64.78.17.19] (HELO EXHUB018-4.exch018.msoutlookonline.net) (64.78.17.19)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 19:41:41 +0000
Received: from EXVMBX018-1.exch018.msoutlookonline.net ([64.78.17.47]) by
 EXHUB018-4.exch018.msoutlookonline.net ([64.78.17.19]) with mapi; Tue, 10 May
 2011 12:41:20 -0700
From: Scott Carey <scott@richrelevance.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 10 May 2011 12:41:40 -0700
Subject: Re: Release compatibility was Re: [VOTE] Release candidate
 0.20.203.0-rc1
Thread-Topic: Release compatibility was Re: [VOTE] Release candidate
 0.20.203.0-rc1
Thread-Index: AcwPSjxJUiMi2kaTQZGGiW1fMVt2Ow==
Message-ID: <C9EEE05B.335A3%scott@richrelevance.com>
In-Reply-To: <215257F9-1F66-4A4D-B4CF-8999DC509866@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-MacOutlook/14.10.0.110310
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org



On 5/8/11 11:10 AM, "Eric Baldeschwieler" <eric14@yahoo-inc.com> wrote:

>I'd agree with this too. [same disclaimer as milind, not on PMC]
>
>In general one would not expect to see an incompatible change added in a
>dot release (0.24.1 0.24.2).  I'd expect anything like that to require
>community discussion and support.
>
>As milind summarized, we seem to have support for the addition of
>security to 20.  The existing mechanism of the required release vote will
>confirm or deny that.
>
>I think it is important that compatible enhancements to hadoop are
>allowed into dot releases.  This is something that we've discussed but
>never finalized in the community.  It is the desire to put improvements
>into users hands more quickly that the next major release that drives
>orgs to produce private releases of hadoop.  In general, I think it is
>fair that such changes go into trunk first.  Exceptions to that also need
>discussion and support IMO.

As an observer, this is a very important observation.  Sure, the default
is that dot releases are bugfix-onl.  But exceptions to these rules are
sometimes required and often beneficial to the health of the project.
Performance enhancements, minor features, and other items are sometimes
very low risk and the barrier to getting them to users earlier should be
lower. =20
These issues are the sort of things that get into non-Apache releases
quickly and drive the community away from the Apache release.  Its been
well proven through those vehicles that back-porting minor features and
improvements from trunk to an old release can be done safely.

>
>I think the key to making progress is discussion and the idea that
>majority support, not consensus is what is needed to make exceptions to
>our process.  Process is useful, it reduces friction.  Process without
>exception is stifling.

Absolutely -- for a subset of process exceptions, a lazy majority would be
much more useful than consensus.  Others are much more dangerous
(backwards compatibility breakage)

>
>On May 7, 2011, at 10:52 PM, Milind Bhandarkar wrote:
>
>> [Mentioning again: I am not on the PMC, and this email contains
>> non-binding opinions based on my reading the general@hadoop.apache.org
>> emails.]
>>=20
>> It is my understanding that, from the beginning, the 0.20+security was
>> always treated as an exception to the normal (I.e. Pre-0.20) release
>> process. (This has been confirmed by the mailing list threads, in which
>> many of those who are objecting to this release now - stating that it
>>has
>> violated norms - have consented, actually argued for, breaking the
>>norms.)
>>=20
>> For whatever I have read on this mailing list before the vote for this
>> release, it looked like most of the community agreed that what Yahoo!
>>Had
>> produced on their own branch, outside of Apache trunk, was important
>> contribution, and a release based on that would be a good idea, and
>>that a
>> one-time release should proceed. (After all, whichever organization the
>> contributors belong to, many seem to indicate that they feel ashamed not
>> having an Apache release in more than a year.)
>>=20
>> From many emails on this thread, it has been clear to me, that it is a
>>one
>> time concession given for parting ways from the normal process, and I
>>hope
>> everyone understands that this is supposed to make Apache Hadoop
>>releases
>> relevant once again.
>>=20
>> So, to cut it short, the 0.20.203 backward incompatibilities etc have no
>> bearing on the "normal" process, in which no backward incompatibilities
>> should be allowed in minor releases. To answer your specific question, I
>> have no reason to believe that 0.22.1 could be backward incompatible
>>with
>> 0.22.0.=20
>>=20
>> - milind
>>=20
>> --=20
>> Milind Bhandarkar
>> mbhandarkar@linkedin.com
>> +1-650-776-3167
>>=20
>>=20
>>=20
>>=20
>>=20
>>=20
>> On 5/7/11 4:50 PM, "Eric Sammer" <esammer@cloudera.com> wrote:
>>=20
>>> Milind:
>>>=20
>>> Thanks for the pointer. I remember this thread. I guess my question
>>> was unrelated to the specific release and more about the general mode
>>> of development under normal release circumstances (ie. do we permit
>>> backward incompatible changes between 0.22.0 and 0.22.1 or is this
>>> something we've allowed just for the 203 release?).
>>>=20
>>> I think it's important to be clear about what the MO is so end users
>>> can plan upgrades appropriately.
>>>=20
>>> Thanks!
>>> Sammer
>>>=20
>>> On May 6, 2011, at 11:52 PM, Milind Bhandarkar
>>><mbhandarkar@linkedin.com>
>>> wrote:
>>>=20
>>>> [I am not on PMC, but seeing that PMC may be busy with other issues, I
>>>> will try to answer your questions.]
>>>>=20
>>>> Eric,
>>>>=20
>>>> I think the thread
>>>>=20
>>>>=20
>>>>"http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3
>>>>C1
>>>> 8C
>>>> 5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com%3E" will answer your
>>>> questions. Here is the timeline as I see it:
>>>>=20
>>>> 1. Arun proposes to create a release from the security patchset. Says
>>>> Doug
>>>> has proposed this earlier
>>>>=20
>>>>=20
>>>>(http://mail-archives.apache.org/mod_mbox/hadoop-general/201004.mbox/%3
>>>>C4
>>>> BD
>>>> 1DFEA.5020908@apache.org%3E April 23, 2010) ("This has been proposed
>>>> earlier by Doug and did not get far due to concerns about the effect
>>>> this
>>>> would have on development on trunk.") (August 24, 2010)
>>>>=20
>>>> 2. Lots of +1s, between August 24 to August 30 2010. One particular
>>>> comment is from Tom White: "I think it would be good to have a shared
>>>> 0.20
>>>> Apache security branch.
>>>> Since security isn't in 0.21, and the 0.22 release is a some way off
>>>> as you mention, this would be useful for folks who want the security
>>>> features sooner (and want to use an Apache release)."
>>>>=20
>>>> 3. Arun volunteers to create a release (August 30, 2010)
>>>>=20
>>>> 4. Doug reminds Arun. (October 15, 2010)
>>>>=20
>>>> 5. Arun apologizes for not creating a branch because he was busy,
>>>> because
>>>> he had a baby. (January 11, 2011)
>>>>=20
>>>> 6. Lots of discussion about what to call it (the release, not the
>>>>baby,
>>>> although I had a good laugh at Patrick Angeles's email: "You're gonna
>>>> call
>>>> your kid 20.100?" ;-).
>>>>=20
>>>> 7. Arun proposes to call it 0.20.100: "I'm open to suggestions - how
>>>> about
>>>> something like 20.100 to show that it's a big jump? Anything else?"
>>>>Jan
>>>> 12, 2011
>>>>=20
>>>> 8. Among others, Eli says: "+1 on 0.20.x   (where x is a J > 3)" on
>>>>Jan
>>>> 12, 2011.
>>>>=20
>>>> So, as you can see, even if this release is called 0.20.x, the
>>>>community
>>>> agreed that these are valuable patches to have, and despite backward
>>>> incompatibility, still have them in minor release.
>>>>=20
>>>> - milind
>>>>=20
>>>> --
>>>> Milind Bhandarkar
>>>> mbhandarkar@linkedin.com
>>>> +1-650-776-3167
>>>>=20
>>>>=20
>>>>=20
>>>>=20
>>>>=20
>>>>=20
>>>> On 5/6/11 11:14 PM, "Eric Sammer" <esammer@cloudera.com> wrote:
>>>>=20
>>>>> On May 6, 2011, at 4:53 AM, Steve Loughran <stevel@apache.org> wrote:
>>>>>=20
>>>>> I understand Eli's concerns that putting stuff in there that hasn't
>>>>> gone
>>>>> into trunk yet is danger. However, as the team makes no guarantees of
>>>>> 100%
>>>>> compatibility between releases, I don't think it's critical. It's
>>>>>just
>>>>> something that needs to be addressed -which can be done after this
>>>>> release
>>>>> has shipped.
>>>>>=20
>>>>>=20
>>>>> I was under the impression that the community has been extremely
>>>>>strict
>>>>> about compatibility between minor version bumps in the past. I though
>>>>> there
>>>>> were specific guarantees and that was one of the reasons certain
>>>>> behaviors
>>>>> have persisted so long.
>>>>>=20
>>>>> Does this mean API changes can be made in minor releases and it can
>>>>>be
>>>>> made
>>>>> backward compatible in future releases? That seems very, very counter
>>>>> to
>>>>> various conversations that have happened in the past. I'm of the mind
>>>>> that
>>>>> we should continue to promise what we've always promised and if
>>>>>that's
>>>>> changing, let's make with the refactoring party!
>>>>>=20
>>>>> Can some PMC'ers clarify this one for me?
>>>>>=20
>>>>> TIA.
>>>>> Sammer
>>>>>=20
>>>>>=20
>>>>>=20
>>>>> -Steve
>>>>=20
>>=20
>


From general-return-3440-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 21:29:12 2011
Return-Path: <general-return-3440-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1D8E4551C
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 10 May 2011 21:29:12 +0000 (UTC)
Received: (qmail 753 invoked by uid 500); 10 May 2011 21:29:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 703 invoked by uid 500); 10 May 2011 21:29:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 695 invoked by uid 99); 10 May 2011 21:29:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 21:29:10 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 21:29:05 +0000
Received: by bwz8 with SMTP id 8so8915987bwz.35
        for <general@hadoop.apache.org>; Tue, 10 May 2011 14:28:44 -0700 (PDT)
Received: by 10.204.136.217 with SMTP id s25mr505937bkt.13.1305062924090; Tue,
 10 May 2011 14:28:44 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.205.82.13 with HTTP; Tue, 10 May 2011 14:28:24 -0700 (PDT)
In-Reply-To: <C9EEE05B.335A3%scott@richrelevance.com>
References: <215257F9-1F66-4A4D-B4CF-8999DC509866@yahoo-inc.com> <C9EEE05B.335A3%scott@richrelevance.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Tue, 10 May 2011 14:28:24 -0700
Message-ID: <BANLkTi=38uUOYGP4zvPfoqPHC3KEKrKnXA@mail.gmail.com>
Subject: Re: Release compatibility was Re: [VOTE] Release candidate 0.20.203.0-rc1
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175cd8d2ffff7e04a2f2a319

--0015175cd8d2ffff7e04a2f2a319
Content-Type: text/plain; charset=ISO-8859-1

On Tue, May 10, 2011 at 12:41 PM, Scott Carey <scott@richrelevance.com>wrote:

>
> As an observer, this is a very important observation.  Sure, the default
> is that dot releases are bugfix-onl.  But exceptions to these rules are
> sometimes required and often beneficial to the health of the project.
> Performance enhancements, minor features, and other items are sometimes
> very low risk and the barrier to getting them to users earlier should be
> lower.
>

I agree whole-heartedly.


> These issues are the sort of things that get into non-Apache releases
> quickly and drive the community away from the Apache release.  Its been
> well proven through those vehicles that back-porting minor features and
> improvements from trunk to an old release can be done safely.


However, one shouldn't understate the difficulty of agreeing on the
risk-reward tradeoff here. While risk is mostly technical, reward may vary
widely based on the userbase or organization.

For example, everyone would agree that security was a very risky feature to
add to 20, with known backward compatibilities and a lot of fallout. For
some people (both CDH and YDH), the security features were an absolute
necessity on a tight timeline, so the risk-reward decision was clear -- I've
heard from many users, though, that they saw none of the reward from
security and wished they hadn't had to endure the resulting changes and bugs
within the 0.20 series.

Another example is the 0.20-append patch series, which is indispensable for
the HBase community but seen as overly risky by those who do not use HBase.

So, while I'm in favor of "sustaining" release series like 0.20-security in
theory, I also think we need a clear inclusion criteria for such branches.
As I said in a previous email, the criteria used to be "low risk compatible
bug fixes only" with a vote process for any exceptions. 0.20-security is
obviously entirely different, but as yet remains undefined (it's way more
than just "security").

-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

--0015175cd8d2ffff7e04a2f2a319--

From general-return-3441-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 23:54:14 2011
Return-Path: <general-return-3441-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BA8515B74
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 10 May 2011 23:54:14 +0000 (UTC)
Received: (qmail 45217 invoked by uid 500); 10 May 2011 23:54:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45150 invoked by uid 500); 10 May 2011 23:54:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45142 invoked by uid 99); 10 May 2011 23:54:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 23:54:12 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 23:54:06 +0000
Received: by qwj9 with SMTP id 9so5582861qwj.35
        for <general@hadoop.apache.org>; Tue, 10 May 2011 16:53:44 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:from:content-type:content-transfer-encoding
         :subject:date:message-id:to:mime-version:x-mailer;
        bh=p/QXLVMeoy11TBW2OhQlGEWUAJQCOhtIKyj3bGqCpLo=;
        b=SUMQT6/zJB67wxbvBTTBAO+Oy4oqakOxK4rviR6V1nGmrao96+B8LD6tgBWwZsoxmx
         GAgKZ2tWDHmAQjgtitU0I7B/9kyL85Flj5xBzDB1CZMPjKfQfauSy6lFhKL0rP6uCF59
         whySn6LUmnmn0otYDCSbb4WLOUhdMJwQoYGsM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=from:content-type:content-transfer-encoding:subject:date:message-id
         :to:mime-version:x-mailer;
        b=kcKyC+Bh+VO/VIcG6GY+vacWnWh10MB72oUFqNtjEXh2ta5G8oIfXXWeTtH266Hbay
         hZ/bXW8gXwJOfVOkr2NDj5L9MA/yXD87oWW6V6AWVtDKq6rJDQg8eayJGaDju2jSuJv6
         QJKLHmKsP2Tuge49qMGBZWSjAVzTATIvBjCeQ=
Received: by 10.224.181.206 with SMTP id bz14mr7414277qab.353.1305071624531;
        Tue, 10 May 2011 16:53:44 -0700 (PDT)
Received: from [10.172.1.95] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id l10sm6268588qck.38.2011.05.10.16.53.42
        (version=SSLv3 cipher=OTHER);
        Tue, 10 May 2011 16:53:43 -0700 (PDT)
From: Ian Holsman <ian@holsman.net>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: OT: anyone else going to berlin buzzwords?
Date: Wed, 11 May 2011 09:53:40 +1000
Message-Id: <B4C351A8-CACF-42C5-AE02-8334B8A95712@holsman.net>
To: general@hadoop.apache.org
Mime-Version: 1.0 (Apple Message framework v1084)
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

If so.. I'll be there.. let's catch up.

http://www.berlinbuzzwords.de/

--
Ian Holsman
Ian@Holsman.net
PH: +1-703 879-3128 AOLIM: ianholsman Skype:iholsman

We are not afraid of the truth, in fact we plan on taking the truth out =
for a nice meal while we persuade it to adopt our views


From general-return-3442-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 00:58:27 2011
Return-Path: <general-return-3442-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BB71D57E7
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 00:58:27 +0000 (UTC)
Received: (qmail 91478 invoked by uid 500); 11 May 2011 00:58:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91420 invoked by uid 500); 11 May 2011 00:58:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91412 invoked by uid 99); 11 May 2011 00:58:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 00:58:25 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 00:58:20 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p4B0vSnY021729
	for <general@hadoop.apache.org>; Tue, 10 May 2011 17:57:28 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1305075449;
	bh=aq991iR76xtG5iSvBaKEobVon6oc58/GsGagFUq+PaM=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=gebk1mj2ausP6mnVSdT8xwWXaHdq+njbZZd+W2sCj/vIyalr0FDtjtEb103s75Obz
	 l9ZAvmm0BnNwRh52wIPWH3963Hqi771nB0ORvc38E905e2y3TVF6hY/ycMYhqbFrst
	 61Z714FGBPcVh3pd1Y4KHifI87Jo01c/iZBESFOs=
Message-Id: <34132EAF-B19F-4DE1-93F9-5E710888E155@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <BANLkTi=5n7g+xB=kZ+RgGvwstkkb=opasA@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Apache Hadoop Hackathons: 5/11 and 5/18 in SF and Palo Alto
Date: Tue, 10 May 2011 17:57:28 -0700
References: <BANLkTiksqFvXdN68Ki4ZK0y=GLmr7s=MJg@mail.gmail.com> <BANLkTi=5n7g+xB=kZ+RgGvwstkkb=opasA@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

Awesome.

I'm sorry to miss this (I'm neck deep in MR-279), but a heads up on  
forward porting from my end:
# Luke and Suresh are sick of me bugging them and just rolled up their  
sleeves to finish up all of the metrics2 work! Thanks guys! I've  
reviewed/committed Luke's https://issues.apache.org/jira/browse/HADOOP-6919 
  and Suresh finished up the rest.
# I'll work with Devaraj/Chris to finish up the TT security hole (https://issues.apache.org/jira/browse/MAPREDUCE-2178 
).

thanks,
Arun

On May 9, 2011, at 2:36 PM, Jeff Hammerbacher wrote:

> Hey,
>
> We've got 23 folks signed up for the Hackathon this Wednesday from
> organizations like Cloudera, Facebook, Twitter, AOL, Trend Micro,
> StumbleUpon, and Ngmoco. We'll also have the VP of the HBase PMC,  
> Michael
> Stack, and the VP of the Hive PMC, John Sichi. If you've been  
> waiting to get
> involved in Apache Hadoop, development, now is the time!
>
> We've got room for 10 - 15 more. If you're in San Francisco or Palo  
> Alto and
> want to help out with Apache Hadoop development, sign up at
> http://hadoophackathon.eventbrite.com.
>
> Regards,
> Jeff
>
> On Fri, May 6, 2011 at 12:03 PM, Jeff Hammerbacher <hammer@cloudera.com 
> >wrote:
>
>> Hey,
>>
>> The discussion this week about the 0.20.203.0 release has done a  
>> great job
>> of highlighting some issues in our development process; it's also  
>> done a
>> great job of lifting our mailing list activity metrics. After reading
>> through the various threads, it's clear that everyone agrees on two  
>> things:
>>
>> 1) We'd like to get the work done in the 0.20.x branches into trunk
>> 2) We'd like to start releasing off of trunk again
>>
>> To that end, a few folks from Yahoo!, Cloudera, StumbleUpon, and  
>> Facebook
>> would like to put together a series of Hackathons to 1) burn down  
>> the 13
>> (only 13!) remaining blockers for the 0.22 release and 2) forward- 
>> port the
>> work done in the 0.20.x branches into trunk.
>>
>> Cloudera will be hosting Hackathons from 10 am to 6 pm in both our  
>> Palo
>> Alto and San Francisco offices next Wednesday, 5/11, and the  
>> following
>> Wednesday, 5/18, to ensure both of these tasks get completed in  
>> short order.
>> PMC members Todd Lipcon and Tom White will lead the SF group and  
>> PMC members
>> Eli Collins and Patrick Hunt will lead the Palo Alto group.
>>
>> Whether you're a long-time contributor or don't have a patch to  
>> your name,
>> now is a great time to get involved in Apache Hadoop development.  
>> Forward
>> porting patches is a lot easier than writing them from scratch, and  
>> you'll
>> have mentors present to help guide you through the patch testing and
>> submission process.
>>
>> To sign up for the 5/11 Hackathon in either SF or PA, head over to
>> http://hadoophackathon.eventbrite.com.
>>
>> As a reminder, the SF HUG will be held on 5/11 at 6 pm in the  
>> Cloudera SF
>> offices: http://www.meetup.com/hadoopsf/events/17354462/. If you  
>> can't
>> make it on 5/11, we'll send out a link next week for the 5/18 event.
>>
>> Looking forward to getting the release train moving again!
>>
>> Regards,
>> Jeff
>>
>> p.s. if you'd like to participate remotely, email me directly and  
>> I'll see
>> about how we can teleconference you into the event.
>>
>>
>>


From general-return-3443-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 02:13:25 2011
Return-Path: <general-return-3443-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 58AAD4073
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 02:13:25 +0000 (UTC)
Received: (qmail 44244 invoked by uid 500); 11 May 2011 02:13:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44186 invoked by uid 500); 11 May 2011 02:13:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44178 invoked by uid 99); 11 May 2011 02:13:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 02:13:23 +0000
X-ASF-Spam-Status: No, hits=3.3 required=5.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 02:13:18 +0000
Received: from sp1-ex07cas03.ds.corp.yahoo.com (sp1-ex07cas03.ds.corp.yahoo.com [216.252.116.151])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p4B2Cm0r042588
	for <general@hadoop.apache.org>; Tue, 10 May 2011 19:12:48 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1305079968;
	bh=pmW1A4cyTS0fSBfuKadRnbcsaQUnf/otF++IhXpyktM=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 MIME-Version;
	b=MOxI2TWRHzED2KUKFn5xRBhAVZEaQFjpBmpaw4pZ2r/1yD62zpL4u+MjaOQR9040N
	 UaC9BuDzhYkNumG9Vn1yfhFG8gt2sJzfZBcScWyPPHki6I+j/kyxfXnE4ZgU3TWgcO
	 QBJigs7u8DaRPep4jpGUsb2pnsfXG6DbJgoyqeLM=
Received: from SP1-EX07VS02.ds.corp.yahoo.com ([216.252.116.135]) by
 sp1-ex07cas03.ds.corp.yahoo.com ([216.252.116.151]) with mapi; Tue, 10 May
 2011 19:12:48 -0700
From: Devaraj Das <ddas@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 10 May 2011 19:12:45 -0700
Subject: Re: OT: anyone else going to berlin buzzwords?
Thread-Topic: OT: anyone else going to berlin buzzwords?
Thread-Index: AcwPbZ4i/fjDzk0jT9OoS4Uo2ZRcrwAE0xLY
Message-ID: <C9EF3EAD.355B8%ddas@yahoo-inc.com>
In-Reply-To: <B4C351A8-CACF-42C5-AE02-8334B8A95712@holsman.net>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C9EF3EAD355B8ddasyahooinccom_"
MIME-Version: 1.0

--_000_C9EF3EAD355B8ddasyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

I'll be there .. Looking forward to meeting up with Ian and the other Hadoo=
p'ers...


On 5/10/11 4:53 PM, "Ian Holsman" <ian@holsman.net> wrote:

If so.. I'll be there.. let's catch up.

http://www.berlinbuzzwords.de/

--
Ian Holsman
Ian@Holsman.net
PH: +1-703 879-3128 AOLIM: ianholsman Skype:iholsman

We are not afraid of the truth, in fact we plan on taking the truth out for=
 a nice meal while we persuade it to adopt our views



--_000_C9EF3EAD355B8ddasyahooinccom_--

From general-return-3444-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 02:50:10 2011
Return-Path: <general-return-3444-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E587644EE
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 02:50:09 +0000 (UTC)
Received: (qmail 72061 invoked by uid 500); 11 May 2011 02:50:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72014 invoked by uid 500); 11 May 2011 02:50:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72006 invoked by uid 99); 11 May 2011 02:50:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 02:50:06 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 02:49:59 +0000
Received: by bwz8 with SMTP id 8so127700bwz.35
        for <general@hadoop.apache.org>; Tue, 10 May 2011 19:49:38 -0700 (PDT)
Received: by 10.204.10.81 with SMTP id o17mr5836313bko.94.1305082178142; Tue,
 10 May 2011 19:49:38 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.205.82.13 with HTTP; Tue, 10 May 2011 19:49:18 -0700 (PDT)
From: Todd Lipcon <todd@cloudera.com>
Date: Tue, 10 May 2011 19:49:18 -0700
Message-ID: <BANLkTinck2etdB0wvura=s5X5pP5XC05Yg@mail.gmail.com>
Subject: "newbie" label on JIRA
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175cd794a1832104a2f71fc4
X-Virus-Checked: Checked by ClamAV on apache.org

--0015175cd794a1832104a2f71fc4
Content-Type: text/plain; charset=ISO-8859-1

Hi all,

I spent this afternoon looking through JIRA to identify some issues that I
think would be good for new contributors to try their hand at. In my mind,
the qualities of such an issue are:

- fairly straightforward issue to solve (an experienced contributor would be
able to address it in 30-60 minutes)
- fairly tight scope (doesn't require understanding of a lot of different
moving pieces)
- easy to write a unit test for (so we get new contributors on the right
path of testing their changes)
- not likely to be controversial among contributors

I came up with about 25 of these from looking through the 0.22 and 0.23
"Affects Version" lists:
https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+in+(%22HADOOP%22,+%22MAPREDUCE%22,+%22HDFS%22)+and+labels+%3D+%22newbie%22

I'd like to encourage others to look through any JIRAs that they think fit
the bill, and add the same label. Then, we can point new contributors at
this list of JIRAs -- hopefully this will get them on the right path towards
understanding our project's workflow and give some nice positive
reinforcement since they should be easy to review and commit quickly.

Thanks!

-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

--0015175cd794a1832104a2f71fc4--

From general-return-3445-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 03:35:29 2011
Return-Path: <general-return-3445-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 19E774DF6
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 03:35:29 +0000 (UTC)
Received: (qmail 96577 invoked by uid 500); 11 May 2011 03:35:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95760 invoked by uid 500); 11 May 2011 03:35:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95746 invoked by uid 99); 11 May 2011 03:35:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 03:35:25 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of tom@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 03:35:19 +0000
Received: by bwz8 with SMTP id 8so157138bwz.35
        for <general@hadoop.apache.org>; Tue, 10 May 2011 20:34:59 -0700 (PDT)
Received: by 10.204.74.68 with SMTP id t4mr5322497bkj.22.1305084897190; Tue,
 10 May 2011 20:34:57 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.205.81.136 with HTTP; Tue, 10 May 2011 20:34:36 -0700 (PDT)
In-Reply-To: <34132EAF-B19F-4DE1-93F9-5E710888E155@yahoo-inc.com>
References: <BANLkTiksqFvXdN68Ki4ZK0y=GLmr7s=MJg@mail.gmail.com>
 <BANLkTi=5n7g+xB=kZ+RgGvwstkkb=opasA@mail.gmail.com> <34132EAF-B19F-4DE1-93F9-5E710888E155@yahoo-inc.com>
From: Tom White <tom@cloudera.com>
Date: Tue, 10 May 2011 20:34:36 -0700
Message-ID: <BANLkTi=-6s9CNRK=+KiE7qWD0Gjp_qZFKg@mail.gmail.com>
Subject: Re: Apache Hadoop Hackathons: 5/11 and 5/18 in SF and Palo Alto
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Tue, May 10, 2011 at 5:57 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:
> Awesome.
>
> I'm sorry to miss this (I'm neck deep in MR-279), but a heads up on forward
> porting from my end:
> # Luke and Suresh are sick of me bugging them and just rolled up their
> sleeves to finish up all of the metrics2 work! Thanks guys! I've
> reviewed/committed Luke's https://issues.apache.org/jira/browse/HADOOP-6919
> and Suresh finished up the rest.
> # I'll work with Devaraj/Chris to finish up the TT security hole
> (https://issues.apache.org/jira/browse/MAPREDUCE-2178).

Thanks Arun - that's great!

Tom

>
> thanks,
> Arun
>
> On May 9, 2011, at 2:36 PM, Jeff Hammerbacher wrote:
>
>> Hey,
>>
>> We've got 23 folks signed up for the Hackathon this Wednesday from
>> organizations like Cloudera, Facebook, Twitter, AOL, Trend Micro,
>> StumbleUpon, and Ngmoco. We'll also have the VP of the HBase PMC, Michael
>> Stack, and the VP of the Hive PMC, John Sichi. If you've been waiting to
>> get
>> involved in Apache Hadoop, development, now is the time!
>>
>> We've got room for 10 - 15 more. If you're in San Francisco or Palo Alto
>> and
>> want to help out with Apache Hadoop development, sign up at
>> http://hadoophackathon.eventbrite.com.
>>
>> Regards,
>> Jeff
>>
>> On Fri, May 6, 2011 at 12:03 PM, Jeff Hammerbacher
>> <hammer@cloudera.com>wrote:
>>
>>> Hey,
>>>
>>> The discussion this week about the 0.20.203.0 release has done a great
>>> job
>>> of highlighting some issues in our development process; it's also done a
>>> great job of lifting our mailing list activity metrics. After reading
>>> through the various threads, it's clear that everyone agrees on two
>>> things:
>>>
>>> 1) We'd like to get the work done in the 0.20.x branches into trunk
>>> 2) We'd like to start releasing off of trunk again
>>>
>>> To that end, a few folks from Yahoo!, Cloudera, StumbleUpon, and Facebook
>>> would like to put together a series of Hackathons to 1) burn down the 13
>>> (only 13!) remaining blockers for the 0.22 release and 2) forward-port
>>> the
>>> work done in the 0.20.x branches into trunk.
>>>
>>> Cloudera will be hosting Hackathons from 10 am to 6 pm in both our Palo
>>> Alto and San Francisco offices next Wednesday, 5/11, and the following
>>> Wednesday, 5/18, to ensure both of these tasks get completed in short
>>> order.
>>> PMC members Todd Lipcon and Tom White will lead the SF group and PMC
>>> members
>>> Eli Collins and Patrick Hunt will lead the Palo Alto group.
>>>
>>> Whether you're a long-time contributor or don't have a patch to your
>>> name,
>>> now is a great time to get involved in Apache Hadoop development. Forward
>>> porting patches is a lot easier than writing them from scratch, and
>>> you'll
>>> have mentors present to help guide you through the patch testing and
>>> submission process.
>>>
>>> To sign up for the 5/11 Hackathon in either SF or PA, head over to
>>> http://hadoophackathon.eventbrite.com.
>>>
>>> As a reminder, the SF HUG will be held on 5/11 at 6 pm in the Cloudera SF
>>> offices: http://www.meetup.com/hadoopsf/events/17354462/. If you can't
>>> make it on 5/11, we'll send out a link next week for the 5/18 event.
>>>
>>> Looking forward to getting the release train moving again!
>>>
>>> Regards,
>>> Jeff
>>>
>>> p.s. if you'd like to participate remotely, email me directly and I'll
>>> see
>>> about how we can teleconference you into the event.
>>>
>>>
>>>
>
>

From general-return-3446-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 04:51:58 2011
Return-Path: <general-return-3446-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 22B554F36
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 04:51:58 +0000 (UTC)
Received: (qmail 32953 invoked by uid 500); 11 May 2011 04:51:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32755 invoked by uid 500); 11 May 2011 04:51:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32747 invoked by uid 99); 11 May 2011 04:51:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 04:51:53 +0000
X-ASF-Spam-Status: No, hits=3.3 required=5.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 04:51:46 +0000
Received: from sp1-ex07cas03.ds.corp.yahoo.com (sp1-ex07cas03.ds.corp.yahoo.com [216.252.116.151])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p4B4p17F024356
	for <general@hadoop.apache.org>; Tue, 10 May 2011 21:51:01 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1305089462;
	bh=TdOH/d0WsrIzVstpywo7hizW0Zvc20n9lyW4x3fzwqE=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 MIME-Version;
	b=Vet2OBG2bYnsogGC28Bm22RyFZW8GrzVUjrpJOX2YWHFhkP0C3sOHDbU1pBGmhCnv
	 j5kLDgV0D6X/TYO9CfntnXzVQf+cYEUUPqMqcA3tZmMWkEEf+EkDB3DN1f7ntrW+LX
	 21eX2scy4cvxiZF3sAKFroNIYVoPO1sFNbnNFqvU=
Received: from SP1-EX07VS02.ds.corp.yahoo.com ([216.252.116.135]) by
 sp1-ex07cas03.ds.corp.yahoo.com ([216.252.116.151]) with mapi; Tue, 10 May
 2011 21:51:01 -0700
From: Devaraj Das <ddas@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 10 May 2011 21:50:58 -0700
Subject: Re: Apache Hadoop Hackathons: 5/11 and 5/18 in SF and Palo Alto
Thread-Topic: Apache Hadoop Hackathons: 5/11 and 5/18 in SF and Palo Alto
Thread-Index: AcwPjIxsJoki/2MDQ0iivHxen22nTAACnhCi
Message-ID: <C9EF63C2.355DB%ddas@yahoo-inc.com>
In-Reply-To: <BANLkTi=-6s9CNRK=+KiE7qWD0Gjp_qZFKg@mail.gmail.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C9EF63C2355DBddasyahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C9EF63C2355DBddasyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

I will get to MR-2178 soon. I & Chris had done a draft of the port to trunk=
 a few months ago. Hopefully, it is easy to continue from there and merge .=
..


On 5/10/11 8:34 PM, "Tom White" <tom@cloudera.com> wrote:

On Tue, May 10, 2011 at 5:57 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:
> Awesome.
>
> I'm sorry to miss this (I'm neck deep in MR-279), but a heads up on forwa=
rd
> porting from my end:
> # Luke and Suresh are sick of me bugging them and just rolled up their
> sleeves to finish up all of the metrics2 work! Thanks guys! I've
> reviewed/committed Luke's https://issues.apache.org/jira/browse/HADOOP-69=
19
> and Suresh finished up the rest.
> # I'll work with Devaraj/Chris to finish up the TT security hole
> (https://issues.apache.org/jira/browse/MAPREDUCE-2178).

Thanks Arun - that's great!

Tom

>
> thanks,
> Arun
>
> On May 9, 2011, at 2:36 PM, Jeff Hammerbacher wrote:
>
>> Hey,
>>
>> We've got 23 folks signed up for the Hackathon this Wednesday from
>> organizations like Cloudera, Facebook, Twitter, AOL, Trend Micro,
>> StumbleUpon, and Ngmoco. We'll also have the VP of the HBase PMC, Michae=
l
>> Stack, and the VP of the Hive PMC, John Sichi. If you've been waiting to
>> get
>> involved in Apache Hadoop, development, now is the time!
>>
>> We've got room for 10 - 15 more. If you're in San Francisco or Palo Alto
>> and
>> want to help out with Apache Hadoop development, sign up at
>> http://hadoophackathon.eventbrite.com.
>>
>> Regards,
>> Jeff
>>
>> On Fri, May 6, 2011 at 12:03 PM, Jeff Hammerbacher
>> <hammer@cloudera.com>wrote:
>>
>>> Hey,
>>>
>>> The discussion this week about the 0.20.203.0 release has done a great
>>> job
>>> of highlighting some issues in our development process; it's also done =
a
>>> great job of lifting our mailing list activity metrics. After reading
>>> through the various threads, it's clear that everyone agrees on two
>>> things:
>>>
>>> 1) We'd like to get the work done in the 0.20.x branches into trunk
>>> 2) We'd like to start releasing off of trunk again
>>>
>>> To that end, a few folks from Yahoo!, Cloudera, StumbleUpon, and Facebo=
ok
>>> would like to put together a series of Hackathons to 1) burn down the 1=
3
>>> (only 13!) remaining blockers for the 0.22 release and 2) forward-port
>>> the
>>> work done in the 0.20.x branches into trunk.
>>>
>>> Cloudera will be hosting Hackathons from 10 am to 6 pm in both our Palo
>>> Alto and San Francisco offices next Wednesday, 5/11, and the following
>>> Wednesday, 5/18, to ensure both of these tasks get completed in short
>>> order.
>>> PMC members Todd Lipcon and Tom White will lead the SF group and PMC
>>> members
>>> Eli Collins and Patrick Hunt will lead the Palo Alto group.
>>>
>>> Whether you're a long-time contributor or don't have a patch to your
>>> name,
>>> now is a great time to get involved in Apache Hadoop development. Forwa=
rd
>>> porting patches is a lot easier than writing them from scratch, and
>>> you'll
>>> have mentors present to help guide you through the patch testing and
>>> submission process.
>>>
>>> To sign up for the 5/11 Hackathon in either SF or PA, head over to
>>> http://hadoophackathon.eventbrite.com.
>>>
>>> As a reminder, the SF HUG will be held on 5/11 at 6 pm in the Cloudera =
SF
>>> offices: http://www.meetup.com/hadoopsf/events/17354462/. If you can't
>>> make it on 5/11, we'll send out a link next week for the 5/18 event.
>>>
>>> Looking forward to getting the release train moving again!
>>>
>>> Regards,
>>> Jeff
>>>
>>> p.s. if you'd like to participate remotely, email me directly and I'll
>>> see
>>> about how we can teleconference you into the event.
>>>
>>>
>>>
>
>


--_000_C9EF63C2355DBddasyahooinccom_--

From general-return-3447-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 05:01:30 2011
Return-Path: <general-return-3447-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B7620485E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 05:01:30 +0000 (UTC)
Received: (qmail 39455 invoked by uid 500); 11 May 2011 05:01:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39073 invoked by uid 500); 11 May 2011 05:01:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39065 invoked by uid 99); 11 May 2011 05:01:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 05:01:28 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.171] (HELO mail-px0-f171.google.com) (209.85.212.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 05:01:23 +0000
Received: by pxi7 with SMTP id 7so138229pxi.2
        for <general@hadoop.apache.org>; Tue, 10 May 2011 22:01:02 -0700 (PDT)
Received: by 10.68.57.235 with SMTP id l11mr3516514pbq.478.1305090062114; Tue,
 10 May 2011 22:01:02 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.68.62.230 with HTTP; Tue, 10 May 2011 22:00:42 -0700 (PDT)
In-Reply-To: <BANLkTinck2etdB0wvura=s5X5pP5XC05Yg@mail.gmail.com>
References: <BANLkTinck2etdB0wvura=s5X5pP5XC05Yg@mail.gmail.com>
From: Konstantin Boudnik <cos@boudnik.org>
Date: Tue, 10 May 2011 22:00:42 -0700
Message-ID: <BANLkTi=5BMHmVL1FC0dyahVcKsi0eyVSRg@mail.gmail.com>
Subject: Re: "newbie" label on JIRA
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Todd - this is a great idea and nice list of JIRAs to take care about!
I assume you are leaving 0.22 blockers to more experienced contributors, ri=
ght?
--
=A0 Take care,
Konstantin (Cos) Boudnik

On Tue, May 10, 2011 at 19:49, Todd Lipcon <todd@cloudera.com> wrote:
> Hi all,
>
> I spent this afternoon looking through JIRA to identify some issues that =
I
> think would be good for new contributors to try their hand at. In my mind=
,
> the qualities of such an issue are:
>
> - fairly straightforward issue to solve (an experienced contributor would=
 be
> able to address it in 30-60 minutes)
> - fairly tight scope (doesn't require understanding of a lot of different
> moving pieces)
> - easy to write a unit test for (so we get new contributors on the right
> path of testing their changes)
> - not likely to be controversial among contributors
>
> I came up with about 25 of these from looking through the 0.22 and 0.23
> "Affects Version" lists:
> https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=3Dtrue&jq=
lQuery=3Dproject+in+(%22HADOOP%22,+%22MAPREDUCE%22,+%22HDFS%22)+and+labels+=
%3D+%22newbie%22
>
> I'd like to encourage others to look through any JIRAs that they think fi=
t
> the bill, and add the same label. Then, we can point new contributors at
> this list of JIRAs -- hopefully this will get them on the right path towa=
rds
> understanding our project's workflow and give some nice positive
> reinforcement since they should be easy to review and commit quickly.
>
> Thanks!
>
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

From general-return-3448-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 05:25:19 2011
Return-Path: <general-return-3448-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2E7934EA0
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 05:25:19 +0000 (UTC)
Received: (qmail 57712 invoked by uid 500); 11 May 2011 05:25:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57662 invoked by uid 500); 11 May 2011 05:25:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57653 invoked by uid 99); 11 May 2011 05:25:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 05:25:14 +0000
X-ASF-Spam-Status: No, hits=3.3 required=5.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 05:25:09 +0000
Received: from sp1-ex07cas01.ds.corp.yahoo.com (sp1-ex07cas01.ds.corp.yahoo.com [216.252.116.137])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p4B5OZn9058549
	for <general@hadoop.apache.org>; Tue, 10 May 2011 22:24:35 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1305091475;
	bh=4w7/vTUhP/xpNWoRGCfdxzlNidPrXE6sdDBYWSzW900=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 MIME-Version;
	b=eA9hlEpprrIG+1QrHQ4HGG3pgpDxmmpcHT0zoYBIvoiHxqp0zmuYvxCKrue9xhJqk
	 9jujvH4enlqDQOOHaUrGS/uYOPLECgLlu0rI679iWF6iyvoYJYRAUGk+mvbMrnpUTA
	 cKeWkUg6qXuHw7qGoJwA8282GSNByxjZVH4hp+Qs=
Received: from SP1-EX07VS02.ds.corp.yahoo.com ([216.252.116.135]) by
 sp1-ex07cas01.ds.corp.yahoo.com ([216.252.116.137]) with mapi; Tue, 10 May
 2011 22:24:35 -0700
From: Devaraj Das <ddas@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 10 May 2011 22:24:34 -0700
Subject: Re: Release compatibility was Re: [VOTE] Release candidate
 0.20.203.0-rc1
Thread-Topic: Release compatibility was Re: [VOTE] Release candidate
 0.20.203.0-rc1
Thread-Index: AcwPWVVkXE4+QjyeTd+eW37NfwN21wAQmDv1
Message-ID: <C9EF6BA2.355E9%ddas@yahoo-inc.com>
In-Reply-To: <BANLkTi=38uUOYGP4zvPfoqPHC3KEKrKnXA@mail.gmail.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C9EF6BA2355E9ddasyahooinccom_"
MIME-Version: 1.0

--_000_C9EF6BA2355E9ddasyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Just so that everyone is on the same page w.r.t the compatibility between 2=
0.2 & 20.203 (don't think this is documented anywhere yet)..

The aim of the team working on Hadoop Security at Yahoo! was to make Hadoop=
 *secure*, and with *minimal* disruption to existing apps. I can't think of=
 any major user-facing incompatibilities other than the users having to run=
 a 'kinit' when they are working with a secure Hadoop cluster (of course th=
e admins need to do more work in order to set up a secure cluster). Also, s=
ecurity can switched off, and all the other enhancements (job limits, etc.)=
 are still available.. As per users/Operations/Solutions at Yahoo!, 20.secu=
rity was one of the smoothest upgrades ever.


On 5/10/11 2:28 PM, "Todd Lipcon" <todd@cloudera.com> wrote:

On Tue, May 10, 2011 at 12:41 PM, Scott Carey <scott@richrelevance.com>wrot=
e:

>
> As an observer, this is a very important observation.  Sure, the default
> is that dot releases are bugfix-onl.  But exceptions to these rules are
> sometimes required and often beneficial to the health of the project.
> Performance enhancements, minor features, and other items are sometimes
> very low risk and the barrier to getting them to users earlier should be
> lower.
>

I agree whole-heartedly.


> These issues are the sort of things that get into non-Apache releases
> quickly and drive the community away from the Apache release.  Its been
> well proven through those vehicles that back-porting minor features and
> improvements from trunk to an old release can be done safely.


However, one shouldn't understate the difficulty of agreeing on the
risk-reward tradeoff here. While risk is mostly technical, reward may vary
widely based on the userbase or organization.

For example, everyone would agree that security was a very risky feature to
add to 20, with known backward compatibilities and a lot of fallout. For
some people (both CDH and YDH), the security features were an absolute
necessity on a tight timeline, so the risk-reward decision was clear -- I'v=
e
heard from many users, though, that they saw none of the reward from
security and wished they hadn't had to endure the resulting changes and bug=
s
within the 0.20 series.

Another example is the 0.20-append patch series, which is indispensable for
the HBase community but seen as overly risky by those who do not use HBase.

So, while I'm in favor of "sustaining" release series like 0.20-security in
theory, I also think we need a clear inclusion criteria for such branches.
As I said in a previous email, the criteria used to be "low risk compatible
bug fixes only" with a vote process for any exceptions. 0.20-security is
obviously entirely different, but as yet remains undefined (it's way more
than just "security").

-Todd
--
Todd Lipcon
Software Engineer, Cloudera


--_000_C9EF6BA2355E9ddasyahooinccom_--

From general-return-3449-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 05:50:17 2011
Return-Path: <general-return-3449-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4368C4D5B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 05:50:17 +0000 (UTC)
Received: (qmail 77097 invoked by uid 500); 11 May 2011 05:50:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76951 invoked by uid 500); 11 May 2011 05:50:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76938 invoked by uid 99); 11 May 2011 05:50:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 05:50:12 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of atm@cloudera.com designates 209.85.216.169 as permitted sender)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 05:50:06 +0000
Received: by qyk2 with SMTP id 2so2638286qyk.14
        for <general@hadoop.apache.org>; Tue, 10 May 2011 22:49:45 -0700 (PDT)
Received: by 10.224.27.67 with SMTP id h3mr7723782qac.39.1305092985193; Tue,
 10 May 2011 22:49:45 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.85.71 with HTTP; Tue, 10 May 2011 22:49:15 -0700 (PDT)
In-Reply-To: <C9EF6BA2.355E9%ddas@yahoo-inc.com>
References: <BANLkTi=38uUOYGP4zvPfoqPHC3KEKrKnXA@mail.gmail.com> <C9EF6BA2.355E9%ddas@yahoo-inc.com>
From: "Aaron T. Myers" <atm@cloudera.com>
Date: Tue, 10 May 2011 22:49:15 -0700
Message-ID: <BANLkTinoW_MBSi9jSKOFK8i2=BMnNzU66w@mail.gmail.com>
Subject: Re: Release compatibility was Re: [VOTE] Release candidate 0.20.203.0-rc1
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec51ba3d1c8051604a2f9a39c
X-Virus-Checked: Checked by ClamAV on apache.org

--bcaec51ba3d1c8051604a2f9a39c
Content-Type: text/plain; charset=ISO-8859-1

On Tue, May 10, 2011 at 10:24 PM, Devaraj Das <ddas@yahoo-inc.com> wrote:

> I can't think of any major user-facing incompatibilities other than the
> users having to run a 'kinit' when they are working with a secure Hadoop
> cluster (of course the admins need to do more work in order to set up a
> secure cluster).


By far the most significant incompatibility that I've seen from a user
perspective is that setting hadoop.job.ugi no longer has any effect.
Granted, this interface wasn't used by a large percentage of users, but
those that were using it have no other alternative that is as flexible as
this was. There was discussion about this incompatibility last September on
the mailing lists[1]. The conclusion there was that supporting backward
compatibility for this interface was too difficult, semantically and
technically, to warrant support. This incompatibility is present whether or
not Kerberos support is enabled on the cluster.

I totally agree that the pain of upgrading to 0.20 security was felt
substantially more by admins/operators than by users.

--
Aaron T. Myers
Software Engineer, Cloudera

[1]
http://mail-archives.apache.org/mod_mbox/hadoop-general/201009.mbox/%3CAANLkTiknJ_SzRux7KhjhxVfUU9FBkNgvYnkpbz3G_+a4@mail.gmail.com%3E

--bcaec51ba3d1c8051604a2f9a39c--

From general-return-3450-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 05:55:18 2011
Return-Path: <general-return-3450-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 469324E6B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 05:55:18 +0000 (UTC)
Received: (qmail 81285 invoked by uid 500); 11 May 2011 05:55:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81231 invoked by uid 500); 11 May 2011 05:55:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81218 invoked by uid 99); 11 May 2011 05:55:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 05:55:14 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 05:55:09 +0000
Received: by bwz8 with SMTP id 8so238653bwz.35
        for <general@hadoop.apache.org>; Tue, 10 May 2011 22:54:48 -0700 (PDT)
Received: by 10.204.47.77 with SMTP id m13mr1458988bkf.67.1305093288195; Tue,
 10 May 2011 22:54:48 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.205.82.13 with HTTP; Tue, 10 May 2011 22:54:28 -0700 (PDT)
In-Reply-To: <BANLkTi=5BMHmVL1FC0dyahVcKsi0eyVSRg@mail.gmail.com>
References: <BANLkTinck2etdB0wvura=s5X5pP5XC05Yg@mail.gmail.com> <BANLkTi=5BMHmVL1FC0dyahVcKsi0eyVSRg@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Tue, 10 May 2011 22:54:28 -0700
Message-ID: <BANLkTimkhfjEgvCtzSo8jwZ1uhLk_H-zjQ@mail.gmail.com>
Subject: Re: "newbie" label on JIRA
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6dd9634d7758d04a2f9b538

--0016e6dd9634d7758d04a2f9b538
Content-Type: text/plain; charset=ISO-8859-1

On Tue, May 10, 2011 at 10:00 PM, Konstantin Boudnik <cos@boudnik.org>wrote:

> Todd - this is a great idea and nice list of JIRAs to take care about!
> I assume you are leaving 0.22 blockers to more experienced contributors,
> right?
>

Unfortunately most of the remaining 0.22 blockers are fairly complicated, so
better suited to more experienced folks. Of course if any new contributor
has a patch ready, that would be great!

Also, to be extra clear, the existence of a "newbie" tag on a JIRA shouldn't
preclude an experienced contributor from doing an issue themselves. It's
just to provide a nice list of starting points for those wanting to get
involved, since the total number of open JIRAs (and the complexity of a lot
of them) is pretty intimidating.

-Todd

On Tue, May 10, 2011 at 19:49, Todd Lipcon <todd@cloudera.com> wrote:
> > Hi all,
> >
> > I spent this afternoon looking through JIRA to identify some issues that
> I
> > think would be good for new contributors to try their hand at. In my
> mind,
> > the qualities of such an issue are:
> >
> > - fairly straightforward issue to solve (an experienced contributor would
> be
> > able to address it in 30-60 minutes)
> > - fairly tight scope (doesn't require understanding of a lot of different
> > moving pieces)
> > - easy to write a unit test for (so we get new contributors on the right
> > path of testing their changes)
> > - not likely to be controversial among contributors
> >
> > I came up with about 25 of these from looking through the 0.22 and 0.23
> > "Affects Version" lists:
> >
> https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+in+(%22HADOOP%22,+%22MAPREDUCE%22,+%22HDFS%22)+and+labels+%3D+%22newbie%22
> >
> > I'd like to encourage others to look through any JIRAs that they think
> fit
> > the bill, and add the same label. Then, we can point new contributors at
> > this list of JIRAs -- hopefully this will get them on the right path
> towards
> > understanding our project's workflow and give some nice positive
> > reinforcement since they should be easy to review and commit quickly.
> >
> > Thanks!
> >
> > -Todd
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--0016e6dd9634d7758d04a2f9b538--

From general-return-3451-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 07:00:56 2011
Return-Path: <general-return-3451-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3D8FC4562
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 07:00:56 +0000 (UTC)
Received: (qmail 49660 invoked by uid 500); 11 May 2011 07:00:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49612 invoked by uid 500); 11 May 2011 07:00:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49604 invoked by uid 99); 11 May 2011 07:00:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 07:00:53 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 07:00:46 +0000
Received: by bwz8 with SMTP id 8so279858bwz.35
        for <general@hadoop.apache.org>; Wed, 11 May 2011 00:00:26 -0700 (PDT)
Received: by 10.204.20.74 with SMTP id e10mr625579bkb.148.1305097226287; Wed,
 11 May 2011 00:00:26 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.205.82.13 with HTTP; Wed, 11 May 2011 00:00:06 -0700 (PDT)
In-Reply-To: <C9EF6BA2.355E9%ddas@yahoo-inc.com>
References: <BANLkTi=38uUOYGP4zvPfoqPHC3KEKrKnXA@mail.gmail.com> <C9EF6BA2.355E9%ddas@yahoo-inc.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Wed, 11 May 2011 00:00:06 -0700
Message-ID: <BANLkTiks+0+wqTtZ7UA_UV_vG8Zqf=uD2Q@mail.gmail.com>
Subject: Re: Release compatibility was Re: [VOTE] Release candidate 0.20.203.0-rc1
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00032555903e920d9604a2faa044
X-Virus-Checked: Checked by ClamAV on apache.org

--00032555903e920d9604a2faa044
Content-Type: text/plain; charset=ISO-8859-1

On Tue, May 10, 2011 at 10:24 PM, Devaraj Das <ddas@yahoo-inc.com> wrote:

> Just so that everyone is on the same page w.r.t the compatibility between
> 20.2 & 20.203 (don't think this is documented anywhere yet)..
>
> The aim of the team working on Hadoop Security at Yahoo! was to make Hadoop
> *secure*, and with *minimal* disruption to existing apps. I can't think of
> any major user-facing incompatibilities other than the users having to run a
> 'kinit' when they are working with a secure Hadoop cluster (of course the
> admins need to do more work in order to set up a secure cluster). Also,
> security can switched off, and all the other enhancements (job limits, etc.)
> are still available.. As per users/Operations/Solutions at Yahoo!,
> 20.security was one of the smoothest upgrades ever.
>

And I think you guys did a commendable job with this, given the scope of the
project! :)

But there were certainly plenty of bugs introduced along the way that
affected both secure and non-secure, and even now the security-able branches
don't function on any non-Sun JVM.

Again, I think for this particular case, most of the developers agreed on
the risk/reward trade-off, so I didn't want to start a discussion about
security being a good or bad decision to backport on to 0.20.

But, I'd love to know what our framework is for making such decisions in the
future, if we plan to maintain branches with feature backports as part of
Apache. (eg what scope of change requires what type of vote and when)

-Todd


>
> On 5/10/11 2:28 PM, "Todd Lipcon" <todd@cloudera.com> wrote:
>
> On Tue, May 10, 2011 at 12:41 PM, Scott Carey <scott@richrelevance.com
> >wrote:
>
> >
> > As an observer, this is a very important observation.  Sure, the default
> > is that dot releases are bugfix-onl.  But exceptions to these rules are
> > sometimes required and often beneficial to the health of the project.
> > Performance enhancements, minor features, and other items are sometimes
> > very low risk and the barrier to getting them to users earlier should be
> > lower.
> >
>
> I agree whole-heartedly.
>
>
> > These issues are the sort of things that get into non-Apache releases
> > quickly and drive the community away from the Apache release.  Its been
> > well proven through those vehicles that back-porting minor features and
> > improvements from trunk to an old release can be done safely.
>
>
> However, one shouldn't understate the difficulty of agreeing on the
> risk-reward tradeoff here. While risk is mostly technical, reward may vary
> widely based on the userbase or organization.
>
> For example, everyone would agree that security was a very risky feature to
> add to 20, with known backward compatibilities and a lot of fallout. For
> some people (both CDH and YDH), the security features were an absolute
> necessity on a tight timeline, so the risk-reward decision was clear --
> I've
> heard from many users, though, that they saw none of the reward from
> security and wished they hadn't had to endure the resulting changes and
> bugs
> within the 0.20 series.
>
> Another example is the 0.20-append patch series, which is indispensable for
> the HBase community but seen as overly risky by those who do not use HBase.
>
> So, while I'm in favor of "sustaining" release series like 0.20-security in
> theory, I also think we need a clear inclusion criteria for such branches.
> As I said in a previous email, the criteria used to be "low risk compatible
> bug fixes only" with a vote process for any exceptions. 0.20-security is
> obviously entirely different, but as yet remains undefined (it's way more
> than just "security").
>
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

--00032555903e920d9604a2faa044--

From general-return-3452-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 07:43:50 2011
Return-Path: <general-return-3452-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E75C24CB4
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 07:43:50 +0000 (UTC)
Received: (qmail 6574 invoked by uid 500); 11 May 2011 07:43:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5819 invoked by uid 500); 11 May 2011 07:43:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5811 invoked by uid 99); 11 May 2011 07:43:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 07:43:43 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [98.139.52.201] (HELO nm4.bullet.mail.ac4.yahoo.com) (98.139.52.201)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 11 May 2011 07:43:35 +0000
Received: from [98.139.52.194] by nm4.bullet.mail.ac4.yahoo.com with NNFMP; 11 May 2011 07:43:13 -0000
Received: from [98.139.52.158] by tm7.bullet.mail.ac4.yahoo.com with NNFMP; 11 May 2011 07:43:13 -0000
Received: from [127.0.0.1] by omp1041.mail.ac4.yahoo.com with NNFMP; 11 May 2011 07:43:13 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 821865.64793.bm@omp1041.mail.ac4.yahoo.com
Received: (qmail 66137 invoked by uid 60001); 11 May 2011 07:43:13 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1305099793; bh=PN2TrU/KCvEvJ7+++jDtoftd9Kq4u4VSzgHQJaMP8Q8=; h=Message-ID:X-YMail-OSG:Received:X-RocketYMMF:X-Mailer:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=t5m4yR9/AergglYr+TIS4o6onwWYxrcW5qv68dKovz2QWaFRCc3WExRjlxxqOBdgr5skb9HPfb+kS84wvq9vSOKv/qPPipemHzgRBpHLgTQke+u2tIyiCVsmjPEXZUgbBnsfVwTdygFIjuOWrTBnJjhb0Wf4S3LHw/91Qe7p66Y=
Message-ID: <655300.15287.qm@web65509.mail.ac4.yahoo.com>
X-YMail-OSG: BMFj6jcVM1n7HMe1Z70aQ4u._y4GR1Nm2goKPtowzbEgxmU
 N6ML6EMk4xks63W.YqdhdAnQZfNTuQivzGQRl8Og_VxqQZ9bNhV1uya4WFZR
 kvA.sEUGvYnRzsi3EVgwQkLcqSHZFXNnhpyW0Qos0wE5lYrjzpSykg6mN.Bq
 pUOd3cHPxKhhEmXpz4u1eRuZVWArtFd87J.Z_RPhrQqP4ngVmJHvJxo_tvfg
 G_FWVNDgCfA9GAsLkEaxe4pkbDLEZLbge1KpIsC1dZwzEl41f0W7x1Mp0qkl
 wujOaq9Qfq.O4UAvrZB58PKKF7.jitHK2SLDbOT9utt50pOAmRrVFifiF1sr
 Y5k5vkx8oOqONIvhKXuIBuPIK9hs3vWE34HZ05KzuQHERhy9qfIj4i15BxHJ
 8Iz0ZZm0LtTFraHizfzonTN.6WUgBnamfAb50dQjW5XJPEaC9LT79Jjh.5Dr
 WuQv1FYvO11mhuUaIMKKhOeykrUallloIhc2H
Received: from [60.251.45.163] by web65509.mail.ac4.yahoo.com via HTTP; Wed, 11 May 2011 00:43:13 PDT
X-RocketYMMF: apurtell
X-Mailer: YahooMailClassic/12.0.2 YahooMailWebService/0.8.110.299900
Date: Wed, 11 May 2011 00:43:13 -0700 (PDT)
From: Andrew Purtell <apurtell@apache.org>
Reply-To: apurtell@apache.org
Subject: Re: Defining Hadoop Compatibility -revisiting-
To: general@hadoop.apache.org
In-Reply-To: <4DC91392.2010308@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

> From: Steve Loughran <stevel@apache.org>=0A> Subject: Defining Hadoop Com=
patibility -revisiting-=0A> To: general@hadoop.apache.org=0A> Date: Tuesday=
, May 10, 2011, 3:29 AM=0A> =0A> Back in Jan 2011, I started a discussion a=
bout how to=0A> define Apache Hadoop Compatibility:=0A> http://mail-archive=
s.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C4D46B6AD.2020802@apache=
.org%3E=0A> =0A> I am now reading EMC HD "Enterprise Ready" Apache Hadoop=
=0A> datasheet=0A[...]=0A>  -I don't think you can claim to have a=0A> Dist=
ribution/Fork/Version of Apache Hadoop if you swap out=0A> big chunks of it=
 for alternate filesystems, MR engines, etc.=0A> Some description of this i=
s needed=0A> "Supports the Apache Hadoop MapReduce engine on top of=0A> Fil=
esystem XYZ"=0A=0AThis is also the case with Brisk, which replaces HDFS and=
 the standard JobTracker with Cassandra and a new JobTracker, and claims to=
 be a Hadoop distribution.=0A=0A  "Apache Hadoop TM Powered by Cassandra"=
=0A  http://www.datastax.com/products/brisk=0A=0A  "DataStax=E2=80=99 Brisk=
 is an enhanced open-source Apache Hadoop and=0A  Hive distribution that ut=
ilizes Apache Cassandra for many of=0A  its core services. [...]"=0A=0A  - =
 Andy=0A

From general-return-3453-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 09:09:19 2011
Return-Path: <general-return-3453-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2B2014C41
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 09:09:19 +0000 (UTC)
Received: (qmail 19955 invoked by uid 500); 11 May 2011 09:09:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19747 invoked by uid 500); 11 May 2011 09:09:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19739 invoked by uid 99); 11 May 2011 09:09:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 09:09:17 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 09:09:10 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 834E91BA934
	for <general@hadoop.apache.org>; Wed, 11 May 2011 10:08:48 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id tcwvCzru7ZY4 for <general@hadoop.apache.org>;
	Wed, 11 May 2011 10:08:48 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by colossus.hpl.hp.com (Postfix) with ESMTP id C98DF1BA8FC
	for <general@hadoop.apache.org>; Wed, 11 May 2011 10:08:47 +0100 (BST)
MailScanner-NULL-Check: 1305709715.53521@KaifiW7uEKCUFpC99a6iZg
Received: from [16.24.227.187] ([16.24.227.187])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p4B98YsV028091
	for <general@hadoop.apache.org>; Wed, 11 May 2011 10:08:34 +0100 (BST)
Message-ID: <4DCA520D.70405@apache.org>
Date: Wed, 11 May 2011 10:08:29 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-GB; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: OT: anyone else going to berlin buzzwords?
References: <B4C351A8-CACF-42C5-AE02-8334B8A95712@holsman.net>
In-Reply-To: <B4C351A8-CACF-42C5-AE02-8334B8A95712@holsman.net>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p4B98YsV028091
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 11/05/2011 00:53, Ian Holsman wrote:
> If so.. I'll be there.. let's catch up.
>
> http://www.berlinbuzzwords.de/
>

1. I'm going to be there, talking about Hadoop testing: Junit -> MrUnit, 
-> minimr -> single host, sampling, how to ramp to production scale

2. Isabel would like us to have a Hadoop Hackathon, which could be 
@Nokia (big Hadoop users) on the thursday. I could come to this, but 
will be flying home early afternoon, and am too busy with my talk and 
unrelated work deadlines to organise a hackathon.

I do think it would be good to have a hadoop hackathon of a sort there, 
and if someone wants to take up the challenge of organising a more 
detailed hadoop hackathon, maybe with some discussion of the MR 2.0 
engine, etc.

Who else from the -user and -dev lists will be there and is willing to 
help put this together?

-steve

From general-return-3454-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 09:26:05 2011
Return-Path: <general-return-3454-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BA0864641
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 09:26:05 +0000 (UTC)
Received: (qmail 52751 invoked by uid 500); 11 May 2011 09:26:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52694 invoked by uid 500); 11 May 2011 09:26:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52684 invoked by uid 99); 11 May 2011 09:26:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 09:26:03 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [145.100.8.49] (HELO smtp.sara.nl) (145.100.8.49)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 09:25:48 +0000
Received: from planck.ka.sara.nl (145.100.8.32) by sara-exch-fe1.ka.sara.nl
 (145.100.8.49) with Microsoft SMTP Server (TLS) id 14.0.702.0; Wed, 11 May
 2011 11:25:27 +0200
Received: from planck.ka.sara.nl ([145.100.8.32]) by planck.ka.sara.nl
 ([145.100.8.32]) with mapi; Wed, 11 May 2011 11:25:27 +0200
From: Evert Lammerts <Evert.Lammerts@sara.nl>
To: "'general@hadoop.apache.org'" <general@hadoop.apache.org>,
	"cdh-user@cloudera.org" <cdh-user@cloudera.org>,
	"'hdfs-user@hadoop.apache.org'" <hdfs-user@hadoop.apache.org>
Date: Wed, 11 May 2011 11:23:25 +0200
Subject: Stability issue - dead DN's
Thread-Topic: Stability issue - dead DN's
Thread-Index: AcwPvRRk2s4vP8/iRMSVf8vjhmk8oA==
Message-ID: <ADF94D8555C7A246B86A633685E0178AB90A474E7A@planck.ka.sara.nl>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/mixed;
	boundary="_002_ADF94D8555C7A246B86A633685E0178AB90A474E7Aplanckkasaran_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_002_ADF94D8555C7A246B86A633685E0178AB90A474E7Aplanckkasaran_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi list,

I notice that whenever our Hadoop installation is put under a heavy load we=
 lose one or two (on a total of five) datanodes. This results in IOExceptio=
ns, and affects the overall performance of the job being run. Can anybody g=
ive me advise or best practices on a different configuration to increase th=
e stability? Below I've included the specs of the cluster, the hadoop relat=
ed config and an example of when which things go wrong. Any help is very mu=
ch appreciated, and if I can provide any other info please let me know.

Cheers,
Evert

=3D=3D What goes wrong, and when =3D=3D

See attached a screenshot of Ganglia when the cluster is under load of a si=
ngle job. This job:
* reads ~1TB from HDFS
* writes ~200GB to HDFS
* runs 288 Mappers and 35 Reducers

When the job runs it takes all available Map and Reduce slots. The system s=
tarts swapping and there is a short time interval during which most cores a=
re in WAIT. After that the job really starts running. At around half way, o=
ne or two datanodes become unreachable and are marked as dead nodes. The am=
ount of under-replicated blocks becomes huge. Then some "java.io.IOExceptio=
n: Could not obtain block" are thrown in Mappers. The job does manage to fi=
nish successfully after around 3.5 hours, but my fear is that when we make =
the input much larger - which we want - the system becomes too unstable to =
finish the job.

Maybe worth mentioning - never know what might help diagnostics.  We notice=
 that memory usage becomes less when we switch our keys from Text to LongWr=
itable. Also, the Mappers are done in a fraction of the time. However, this=
 for some reason results in much more network traffic and makes Reducers ex=
tremely slow. We're working on figuring out what causes this.


=3D=3D The cluster =3D=3D

We have a cluster that consists of 6 Sun Thumpers running Hadoop 0.20.2 on =
CentOS 5.5. One of them acts as NN and JT, the other 5 run DN's and TT's. E=
ach node has:
* 16GB RAM
* 32GB swapspace
* 4 cores
* 11 LVM's of 4 x 500GB disks (2TB in total) for HDFS
* non-HDFS stuff on separate disks
* a 2x1GE bonded network interface for interconnects
* a 2x1GE bonded network interface for external access

I realize that this is not a well balanced system, but it's what we had ava=
ilable for a prototype environment. We're working on putting together a spe=
cification for a much larger production environment.


=3D=3D Hadoop config =3D=3D

Here some properties that I think might be relevant:

__CORE-SITE.XML__

fs.inmemory.size.mb: 200
mapreduce.task.io.sort.factor: 100
mapreduce.task.io.sort.mb: 200
# 1024*1024*4 MB, blocksize of the LVM's
io.file.buffer.size: 4194304

__HDFS-SITE.XML__

# 1024*1024*4*32 MB, 32 times the blocksize of the LVM's
dfs.block.size: 134217728
# Only 5 DN's, but this shouldn't hurt
dfs.namenode.handler.count: 40
# This got rid of the occasional "Could not obtain block"'s
dfs.datanode.max.xcievers: 4096

__MAPRED-SITE.XML__

mapred.tasktracker.map.tasks.maximum: 4
mapred.tasktracker.reduce.tasks.maximum: 4
mapred.child.java.opts: -Xmx2560m
mapreduce.reduce.shuffle.parallelcopies: 20
mapreduce.map.java.opts: -Xmx512m
mapreduce.reduce.java.opts: -Xmx512m
# Compression codecs are configured and seem to work fine
mapred.compress.map.output: true
mapred.map.output.compression.codec: com.hadoop.compression.lzo.LzoCodec


--_002_ADF94D8555C7A246B86A633685E0178AB90A474E7Aplanckkasaran_--

From general-return-3455-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 10:31:39 2011
Return-Path: <general-return-3455-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ABDE94174
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 10:31:39 +0000 (UTC)
Received: (qmail 33120 invoked by uid 500); 11 May 2011 10:31:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33066 invoked by uid 500); 11 May 2011 10:31:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33058 invoked by uid 99); 11 May 2011 10:31:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 10:31:38 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bernd.fondermann@googlemail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 10:31:33 +0000
Received: by bwz8 with SMTP id 8so434101bwz.35
        for <general@hadoop.apache.org>; Wed, 11 May 2011 03:31:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=SqOlqkbXSG5TOhvjXrv54ErLoFyD/MifSr77Pp6t8QA=;
        b=o8OmCFJa/8SOK0rLyoTNAbRR+NMNTVyFNrOSZSK0GRV2PkrRB+uUGJ0W4r6SUkLMv8
         aY7+cGakXYUQVNF3YXKTl0DWxu6UsGn+i/qmBm8TjPLB3hV+8vT67qthznpbdJsvgb6e
         XbyUGxKimCbZndwZiotUhamO9qC4a2J6iT2kc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=SSyhMcz1qFwTQHM0ueAix8aqjPURodjk76DYYf0wYVxwbh0yDzpiAdewKQ1lXXU1nO
         QeV+d0FGFMeC/zVTxMHVHcp1pT7yJYxZg78YTqflrXwfRJi+io8FI+vErQPRCRPLsfbM
         FKy8T00FDXfkNjbAdhCJ9vqhNR6dIomPoiAJ0=
MIME-Version: 1.0
Received: by 10.204.41.2 with SMTP id m2mr1893391bke.54.1305109872047; Wed, 11
 May 2011 03:31:12 -0700 (PDT)
Received: by 10.204.83.196 with HTTP; Wed, 11 May 2011 03:31:11 -0700 (PDT)
In-Reply-To: <4DCA520D.70405@apache.org>
References: <B4C351A8-CACF-42C5-AE02-8334B8A95712@holsman.net>
	<4DCA520D.70405@apache.org>
Date: Wed, 11 May 2011 12:31:11 +0200
Message-ID: <BANLkTimAH6U5yYwQhb=M7vxbN08+oztwLg@mail.gmail.com>
Subject: Re: OT: anyone else going to berlin buzzwords?
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Wed, May 11, 2011 at 11:08, Steve Loughran <stevel@apache.org> wrote:
> On 11/05/2011 00:53, Ian Holsman wrote:
>>
>> If so.. I'll be there.. let's catch up.
>>
>> http://www.berlinbuzzwords.de/
>>
>
> 1. I'm going to be there

1.1.
me too, but only for the conference on Mon + Tue.
Who's available for a Hadoop/Apache/etc food + beverages on Monday evening?

  Bernd

From general-return-3456-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 10:34:54 2011
Return-Path: <general-return-3456-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A0B845000
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 10:34:54 +0000 (UTC)
Received: (qmail 40118 invoked by uid 500); 11 May 2011 10:34:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40049 invoked by uid 500); 11 May 2011 10:34:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40041 invoked by uid 99); 11 May 2011 10:34:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 10:34:53 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 10:34:46 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 03F061BA943
	for <general@hadoop.apache.org>; Wed, 11 May 2011 11:34:25 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id GFAXuCV5C87F for <general@hadoop.apache.org>;
	Wed, 11 May 2011 11:34:24 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 6E7901BA93D
	for <general@hadoop.apache.org>; Wed, 11 May 2011 11:34:24 +0100 (BST)
MailScanner-NULL-Check: 1305714852.21315@i003BrJHnYdHngHBHDsvwQ
Received: from [16.24.227.187] ([16.24.227.187])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p4BAYBTp000281
	for <general@hadoop.apache.org>; Wed, 11 May 2011 11:34:11 +0100 (BST)
Message-ID: <4DCA661E.3020801@apache.org>
Date: Wed, 11 May 2011 11:34:06 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-GB; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Defining Hadoop Compatibility -revisiting-
References: <655300.15287.qm@web65509.mail.ac4.yahoo.com>
In-Reply-To: <655300.15287.qm@web65509.mail.ac4.yahoo.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p4BAYBTp000281
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 11/05/2011 08:43, Andrew Purtell wrote:
>> From: Steve Loughran<stevel@apache.org>

>>   -I don't think you can claim to have a
>> Distribution/Fork/Version of Apache Hadoop if you swap out
>> big chunks of it for alternate filesystems, MR engines, etc.
>> Some description of this is needed
>> "Supports the Apache Hadoop MapReduce engine on top of
>> Filesystem XYZ"
>
> This is also the case with Brisk, which replaces HDFS and the standard JobTracker with Cassandra and a new JobTracker, and claims to be a Hadoop distribution.
>
>    "Apache Hadoop TM Powered by Cassandra"
>    http://www.datastax.com/products/brisk
>
>    "DataStax’ Brisk is an enhanced open-source Apache Hadoop and
>    Hive distribution that utilizes Apache Cassandra for many of
>    its core services. [...]"
>

+1. It is something containing Hadoop interfaces and possibly 
source/artifacts, but I'm not sure how to describe it. It is just 
something that claims compatibility with Hadoop's filesystem and MR 
runtime. If Google chose to add the same interfaces to their platform 
within Google App Engine, it wouldn't be a Hadoop distro either.

I think it's important to set some definitions here *now* so that 
confusion doesn't set in.

From general-return-3457-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 10:49:46 2011
Return-Path: <general-return-3457-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4CB814806
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 10:49:46 +0000 (UTC)
Received: (qmail 54924 invoked by uid 500); 11 May 2011 10:49:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54708 invoked by uid 500); 11 May 2011 10:49:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54700 invoked by uid 99); 11 May 2011 10:49:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 10:49:44 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 10:49:37 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 90886B7E49
	for <general@hadoop.apache.org>; Wed, 11 May 2011 11:49:16 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id i+eqPTHPp-KE for <general@hadoop.apache.org>;
	Wed, 11 May 2011 11:49:15 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 396A2B7E42
	for <general@hadoop.apache.org>; Wed, 11 May 2011 11:49:14 +0100 (BST)
MailScanner-NULL-Check: 1305715743.12954@sOwZtvfyjTFAKc2HInRYCQ
Received: from [16.24.227.187] ([16.24.227.187])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p4BAn0Wn000684
	for <general@hadoop.apache.org>; Wed, 11 May 2011 11:49:00 +0100 (BST)
Message-ID: <4DCA6997.2040200@apache.org>
Date: Wed, 11 May 2011 11:48:55 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-GB; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: OT: anyone else going to berlin buzzwords?
References: <C9EF3EAD.355B8%ddas@yahoo-inc.com>
In-Reply-To: <C9EF3EAD.355B8%ddas@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p4BAn0Wn000684
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 11/05/2011 03:12, Devaraj Das wrote:
> I'll be there .. Looking forward to meeting up with Ian and the other Hadoop'ers...
>


OK, there's now a tentative plan for a Hadoop hackathon, i'd like a 
volunteer to do a tour of the MR-279 codebase, and we have a volunteer 
to talk about gridmix.

http://berlinbuzzwords.de/wiki/hadoop-hackathon

Being a hackathon and all, some coding theme would be good. Testing MR-279?

From general-return-3458-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 12:57:49 2011
Return-Path: <general-return-3458-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 23C7A45B1
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 12:57:49 +0000 (UTC)
Received: (qmail 35085 invoked by uid 500); 11 May 2011 12:57:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 35045 invoked by uid 500); 11 May 2011 12:57:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 35037 invoked by uid 99); 11 May 2011 12:57:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 12:57:47 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 12:57:42 +0000
Received: by qwj9 with SMTP id 9so290309qwj.35
        for <general@hadoop.apache.org>; Wed, 11 May 2011 05:57:21 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.224.27.20 with SMTP id g20mr7571578qac.15.1305118641140; Wed,
 11 May 2011 05:57:21 -0700 (PDT)
Received: by 10.224.67.136 with HTTP; Wed, 11 May 2011 05:57:21 -0700 (PDT)
In-Reply-To: <ADF94D8555C7A246B86A633685E0178AB90A474E7A@planck.ka.sara.nl>
References: <ADF94D8555C7A246B86A633685E0178AB90A474E7A@planck.ka.sara.nl>
Date: Wed, 11 May 2011 06:57:21 -0600
Message-ID: <BANLkTi=NaiXQPAkdS-LPW-Swev6XYD0Xjw@mail.gmail.com>
Subject: Re: Stability issue - dead DN's
From: Eric Fiala <eric@fiala.ca>
To: cdh-user@cloudera.org, general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec51a8524feb30204a2ff9ca2

--bcaec51a8524feb30204a2ff9ca2
Content-Type: text/plain; charset=ISO-8859-1

Evert,
>From my experience, hadoop loathes swap and you mention that all reduces and
mappers are running (8 total) and from the ganglia screenshot I see that you
have a thick crest of that purple swap.

If we do the math that means [ map.tasks.max * mapred.child.java.opts ]  +
 [ reduce.tasks.max * mapred.child.java.opts ] => or [ 4 * 2.5G ] + [ 4 *
2.5G ] is greater than the amount of physical RAM in the machine.
This doesn't account for the base tasktracker and datanode process + OS
overhead and whatever else may be hoarding resources on the systems.

I would play with this ratio, either less maps / reduces max - or lower your
child.java.opts so that when you are fully subscribed you are not using
more resource than the machine can offer.
Also, setting mapred.reduce.slowstart.completed.maps  to 1.00 or some other
value close to 1 would be one way to guarantee only 4 either maps or reduces
to be running at once and address (albeit in a duct tape like way) the
oversubscription problem you are seeing (this represents the fractions of
maps that should complete before initiating the reduce phase).

hth

EF


On Wed, May 11, 2011 at 3:23 AM, Evert Lammerts <Evert.Lammerts@sara.nl>wrote:

> Hi list,
>
> I notice that whenever our Hadoop installation is put under a heavy load we
> lose one or two (on a total of five) datanodes. This results in
> IOExceptions, and affects the overall performance of the job being run. Can
> anybody give me advise or best practices on a different configuration to
> increase the stability? Below I've included the specs of the cluster, the
> hadoop related config and an example of when which things go wrong. Any help
> is very much appreciated, and if I can provide any other info please let me
> know.
>
> Cheers,
> Evert
>
> == What goes wrong, and when ==
>
> See attached a screenshot of Ganglia when the cluster is under load of a
> single job. This job:
> * reads ~1TB from HDFS
> * writes ~200GB to HDFS
> * runs 288 Mappers and 35 Reducers
>
> When the job runs it takes all available Map and Reduce slots. The system
> starts swapping and there is a short time interval during which most cores
> are in WAIT. After that the job really starts running. At around half way,
> one or two datanodes become unreachable and are marked as dead nodes. The
> amount of under-replicated blocks becomes huge. Then some
> "java.io.IOException: Could not obtain block" are thrown in Mappers. The job
> does manage to finish successfully after around 3.5 hours, but my fear is
> that when we make the input much larger - which we want - the system becomes
> too unstable to finish the job.
>
> Maybe worth mentioning - never know what might help diagnostics.  We notice
> that memory usage becomes less when we switch our keys from Text to
> LongWritable. Also, the Mappers are done in a fraction of the time. However,
> this for some reason results in much more network traffic and makes Reducers
> extremely slow. We're working on figuring out what causes this.
>
>
> == The cluster ==
>
> We have a cluster that consists of 6 Sun Thumpers running Hadoop 0.20.2 on
> CentOS 5.5. One of them acts as NN and JT, the other 5 run DN's and TT's.
> Each node has:
> * 16GB RAM
> * 32GB swapspace
> * 4 cores
> * 11 LVM's of 4 x 500GB disks (2TB in total) for HDFS
> * non-HDFS stuff on separate disks
> * a 2x1GE bonded network interface for interconnects
> * a 2x1GE bonded network interface for external access
>
> I realize that this is not a well balanced system, but it's what we had
> available for a prototype environment. We're working on putting together a
> specification for a much larger production environment.
>
>
> == Hadoop config ==
>
> Here some properties that I think might be relevant:
>
> __CORE-SITE.XML__
>
> fs.inmemory.size.mb: 200
> mapreduce.task.io.sort.factor: 100
> mapreduce.task.io.sort.mb: 200
> # 1024*1024*4 MB, blocksize of the LVM's
> io.file.buffer.size: 4194304
>
> __HDFS-SITE.XML__
>
> # 1024*1024*4*32 MB, 32 times the blocksize of the LVM's
> dfs.block.size: 134217728
> # Only 5 DN's, but this shouldn't hurt
> dfs.namenode.handler.count: 40
> # This got rid of the occasional "Could not obtain block"'s
> dfs.datanode.max.xcievers: 4096
>
> __MAPRED-SITE.XML__
>
> mapred.tasktracker.map.tasks.maximum: 4
> mapred.tasktracker.reduce.tasks.maximum: 4
> mapred.child.java.opts: -Xmx2560m
> mapreduce.reduce.shuffle.parallelcopies: 20
> mapreduce.map.java.opts: -Xmx512m
> mapreduce.reduce.java.opts: -Xmx512m
> # Compression codecs are configured and seem to work fine
> mapred.compress.map.output: true
> mapred.map.output.compression.codec: com.hadoop.compression.lzo.LzoCodec
>
>

--bcaec51a8524feb30204a2ff9ca2--

From general-return-3459-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 13:21:52 2011
Return-Path: <general-return-3459-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 88C0C4629
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 13:21:52 +0000 (UTC)
Received: (qmail 76508 invoked by uid 500); 11 May 2011 13:21:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76425 invoked by uid 500); 11 May 2011 13:21:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76417 invoked by uid 99); 11 May 2011 13:21:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 13:21:51 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 209.85.160.48 is neither permitted nor denied by domain of james@tynt.com)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 13:21:43 +0000
Received: by pwi16 with SMTP id 16so326500pwi.35
        for <general@hadoop.apache.org>; Wed, 11 May 2011 06:21:22 -0700 (PDT)
Received: by 10.68.2.129 with SMTP id 1mr2076029pbu.145.1305118449411;
        Wed, 11 May 2011 05:54:09 -0700 (PDT)
Received: from [10.0.1.13] (S01060016cbc5f8b4.cg.shawcable.net [174.0.46.252])
        by mx.google.com with ESMTPS id p2sm5507255pbq.6.2011.05.11.05.54.08
        (version=TLSv1/SSLv3 cipher=OTHER);
        Wed, 11 May 2011 05:54:08 -0700 (PDT)
Content-Type: text/plain; charset=windows-1252
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: Stability issue - dead DN's
From: James Seigel <james@tynt.com>
In-Reply-To: <ADF94D8555C7A246B86A633685E0178AB90A474E7A@planck.ka.sara.nl>
Date: Wed, 11 May 2011 06:54:06 -0600
Content-Transfer-Encoding: quoted-printable
Message-Id: <763952E3-8AAD-4757-99DC-57ABC06BBC2C@tynt.com>
References: <ADF94D8555C7A246B86A633685E0178AB90A474E7A@planck.ka.sara.nl>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)

Evert,

What=92s the stack trace and what version of hadoop do you have =
installed Sir!

James.
On 2011-05-11, at 3:23 AM, Evert Lammerts wrote:

> Hi list,
>=20
> I notice that whenever our Hadoop installation is put under a heavy =
load we lose one or two (on a total of five) datanodes. This results in =
IOExceptions, and affects the overall performance of the job being run. =
Can anybody give me advise or best practices on a different =
configuration to increase the stability? Below I've included the specs =
of the cluster, the hadoop related config and an example of when which =
things go wrong. Any help is very much appreciated, and if I can provide =
any other info please let me know.
>=20
> Cheers,
> Evert
>=20
> =3D=3D What goes wrong, and when =3D=3D
>=20
> See attached a screenshot of Ganglia when the cluster is under load of =
a single job. This job:
> * reads ~1TB from HDFS
> * writes ~200GB to HDFS
> * runs 288 Mappers and 35 Reducers
>=20
> When the job runs it takes all available Map and Reduce slots. The =
system starts swapping and there is a short time interval during which =
most cores are in WAIT. After that the job really starts running. At =
around half way, one or two datanodes become unreachable and are marked =
as dead nodes. The amount of under-replicated blocks becomes huge. Then =
some "java.io.IOException: Could not obtain block" are thrown in =
Mappers. The job does manage to finish successfully after around 3.5 =
hours, but my fear is that when we make the input much larger - which we =
want - the system becomes too unstable to finish the job.
>=20
> Maybe worth mentioning - never know what might help diagnostics.  We =
notice that memory usage becomes less when we switch our keys from Text =
to LongWritable. Also, the Mappers are done in a fraction of the time. =
However, this for some reason results in much more network traffic and =
makes Reducers extremely slow. We're working on figuring out what causes =
this.
>=20
>=20
> =3D=3D The cluster =3D=3D
>=20
> We have a cluster that consists of 6 Sun Thumpers running Hadoop =
0.20.2 on CentOS 5.5. One of them acts as NN and JT, the other 5 run =
DN's and TT's. Each node has:
> * 16GB RAM
> * 32GB swapspace
> * 4 cores
> * 11 LVM's of 4 x 500GB disks (2TB in total) for HDFS
> * non-HDFS stuff on separate disks
> * a 2x1GE bonded network interface for interconnects
> * a 2x1GE bonded network interface for external access
>=20
> I realize that this is not a well balanced system, but it's what we =
had available for a prototype environment. We're working on putting =
together a specification for a much larger production environment.
>=20
>=20
> =3D=3D Hadoop config =3D=3D
>=20
> Here some properties that I think might be relevant:
>=20
> __CORE-SITE.XML__
>=20
> fs.inmemory.size.mb: 200
> mapreduce.task.io.sort.factor: 100
> mapreduce.task.io.sort.mb: 200
> # 1024*1024*4 MB, blocksize of the LVM's
> io.file.buffer.size: 4194304
>=20
> __HDFS-SITE.XML__
>=20
> # 1024*1024*4*32 MB, 32 times the blocksize of the LVM's
> dfs.block.size: 134217728
> # Only 5 DN's, but this shouldn't hurt
> dfs.namenode.handler.count: 40
> # This got rid of the occasional "Could not obtain block"'s
> dfs.datanode.max.xcievers: 4096
>=20
> __MAPRED-SITE.XML__
>=20
> mapred.tasktracker.map.tasks.maximum: 4
> mapred.tasktracker.reduce.tasks.maximum: 4
> mapred.child.java.opts: -Xmx2560m
> mapreduce.reduce.shuffle.parallelcopies: 20
> mapreduce.map.java.opts: -Xmx512m
> mapreduce.reduce.java.opts: -Xmx512m
> # Compression codecs are configured and seem to work fine
> mapred.compress.map.output: true
> mapred.map.output.compression.codec: =
com.hadoop.compression.lzo.LzoCodec
>=20


From general-return-3460-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 13:39:08 2011
Return-Path: <general-return-3460-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7F15F4EB3
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 13:39:08 +0000 (UTC)
Received: (qmail 19890 invoked by uid 500); 11 May 2011 13:39:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19826 invoked by uid 500); 11 May 2011 13:39:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19811 invoked by uid 99); 11 May 2011 13:39:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 13:39:04 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [145.100.8.49] (HELO smtp.sara.nl) (145.100.8.49)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 13:38:56 +0000
Received: from planck.ka.sara.nl (145.100.8.32) by sara-exch-fe1.ka.sara.nl
 (145.100.8.49) with Microsoft SMTP Server (TLS) id 14.0.702.0; Wed, 11 May
 2011 15:38:34 +0200
Received: from planck.ka.sara.nl ([145.100.8.32]) by planck.ka.sara.nl
 ([145.100.8.32]) with mapi; Wed, 11 May 2011 15:38:13 +0200
From: Evert Lammerts <Evert.Lammerts@sara.nl>
To: "'general@hadoop.apache.org'" <general@hadoop.apache.org>
Date: Wed, 11 May 2011 15:36:11 +0200
Subject: RE: Stability issue - dead DN's
Thread-Topic: Stability issue - dead DN's
Thread-Index: AcwP3m0sYkuQIkI6Sfiu1Yslgv3bpAAAXW6g
Message-ID: <ADF94D8555C7A246B86A633685E0178AB90A474E81@planck.ka.sara.nl>
References: <ADF94D8555C7A246B86A633685E0178AB90A474E7A@planck.ka.sara.nl>
 <763952E3-8AAD-4757-99DC-57ABC06BBC2C@tynt.com>
In-Reply-To: <763952E3-8AAD-4757-99DC-57ABC06BBC2C@tynt.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi James,

Hadoop version is 0.20.2 (find that and more on our setup also in my first =
mail, under heading "The cluster").

Below I) an example stacktrace of losing a datanode is and II) an example o=
f a "Could not obtain block" IOException.

Cheers,
Evert

11/05/11 15:06:43 INFO hdfs.DFSClient: Failed to connect to /192.168.28.214=
:50050, add to deadNodes and continue
java.net.SocketTimeoutException: 60000 millis timeout while waiting for cha=
nnel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=3D/192.168.28.209:50726 rem=
ote=3D/192.168.28.214:50050]
        at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeo=
ut.java:164)
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.j=
ava:155)
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.j=
ava:128)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
        at java.io.DataInputStream.readShort(DataInputStream.java:295)
        at org.apache.hadoop.hdfs.DFSClient$BlockReader.newBlockReader(DFSC=
lient.java:1478)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSC=
lient.java:1811)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.j=
ava:1948)
        at java.io.DataInputStream.readFully(DataInputStream.java:178)
        at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuf=
fer.java:63)
        at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.jav=
a:101)
        at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:=
1945)
        at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:=
1845)
        at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:=
1891)
        at org.apache.mahout.common.iterator.sequencefile.SequenceFileItera=
tor.computeNext(SequenceFileIterator.java:95)
        at org.apache.mahout.common.iterator.sequencefile.SequenceFileItera=
tor.computeNext(SequenceFileIterator.java:1)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(Abst=
ractIterator.java:135)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractItera=
tor.java:130)
        at nl.liacs.infrawatch.hadoop.kmeans.RandomSeedGenerator.buildRando=
m(RandomSeedGenerator.java:85)
        at nl.liacs.infrawatch.hadoop.kmeans.Job.run(Job.java:171)
        at nl.liacs.infrawatch.hadoop.kmeans.Job.main(Job.java:74)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessor=
Impl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod=
AccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:186)


11/05/10 09:43:47 INFO mapred.JobClient:  map 82% reduce 17% 11/05/10 09:44=
:39 INFO mapred.JobClient: Task Id :
attempt_201104121440_0122_m_000225_0, Status : FAILED
java.io.IOException: Could not obtain block:
blk_4397122445076815438_4097927
file=3D/user/joaquin/data/20081201/20081201.039
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(D=
FSClient.java:1993)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSC=
lient.java:1800)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.j=
ava:1948)
        at java.io.DataInputStream.read(DataInputStream.java:83)
        at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
        at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyVa=
lue(LineRecordReader.java:97)
        at nl.liacs.infrawatch.hadoop.kmeans.KeyValueLineRecordReader.nextK=
eyValue(KeyValueLineRecordReader.java:94)
        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKey=
Value(MapTask.java:455)
        at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.j=
ava:67)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:646)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupIn=
formation.java:1115)
        at org.apache.hadoop.mapred.Child.main(Child.java:262)

> -----Original Message-----
> From: James Seigel [mailto:james@tynt.com]
> Sent: woensdag 11 mei 2011 14:54
> To: general@hadoop.apache.org
> Subject: Re: Stability issue - dead DN's
>
> Evert,
>
> What's the stack trace and what version of hadoop do you have installed
> Sir!
>
> James.
> On 2011-05-11, at 3:23 AM, Evert Lammerts wrote:
>
> > Hi list,
> >
> > I notice that whenever our Hadoop installation is put under a heavy
> load we lose one or two (on a total of five) datanodes. This results in
> IOExceptions, and affects the overall performance of the job being run.
> Can anybody give me advise or best practices on a different
> configuration to increase the stability? Below I've included the specs
> of the cluster, the hadoop related config and an example of when which
> things go wrong. Any help is very much appreciated, and if I can
> provide any other info please let me know.
> >
> > Cheers,
> > Evert
> >
> > =3D=3D What goes wrong, and when =3D=3D
> >
> > See attached a screenshot of Ganglia when the cluster is under load
> of a single job. This job:
> > * reads ~1TB from HDFS
> > * writes ~200GB to HDFS
> > * runs 288 Mappers and 35 Reducers
> >
> > When the job runs it takes all available Map and Reduce slots. The
> system starts swapping and there is a short time interval during which
> most cores are in WAIT. After that the job really starts running. At
> around half way, one or two datanodes become unreachable and are marked
> as dead nodes. The amount of under-replicated blocks becomes huge. Then
> some "java.io.IOException: Could not obtain block" are thrown in
> Mappers. The job does manage to finish successfully after around 3.5
> hours, but my fear is that when we make the input much larger - which
> we want - the system becomes too unstable to finish the job.
> >
> > Maybe worth mentioning - never know what might help diagnostics.  We
> notice that memory usage becomes less when we switch our keys from Text
> to LongWritable. Also, the Mappers are done in a fraction of the time.
> However, this for some reason results in much more network traffic and
> makes Reducers extremely slow. We're working on figuring out what
> causes this.
> >
> >
> > =3D=3D The cluster =3D=3D
> >
> > We have a cluster that consists of 6 Sun Thumpers running Hadoop
> 0.20.2 on CentOS 5.5. One of them acts as NN and JT, the other 5 run
> DN's and TT's. Each node has:
> > * 16GB RAM
> > * 32GB swapspace
> > * 4 cores
> > * 11 LVM's of 4 x 500GB disks (2TB in total) for HDFS
> > * non-HDFS stuff on separate disks
> > * a 2x1GE bonded network interface for interconnects
> > * a 2x1GE bonded network interface for external access
> >
> > I realize that this is not a well balanced system, but it's what we
> had available for a prototype environment. We're working on putting
> together a specification for a much larger production environment.
> >
> >
> > =3D=3D Hadoop config =3D=3D
> >
> > Here some properties that I think might be relevant:
> >
> > __CORE-SITE.XML__
> >
> > fs.inmemory.size.mb: 200
> > mapreduce.task.io.sort.factor: 100
> > mapreduce.task.io.sort.mb: 200
> > # 1024*1024*4 MB, blocksize of the LVM's
> > io.file.buffer.size: 4194304
> >
> > __HDFS-SITE.XML__
> >
> > # 1024*1024*4*32 MB, 32 times the blocksize of the LVM's
> > dfs.block.size: 134217728
> > # Only 5 DN's, but this shouldn't hurt
> > dfs.namenode.handler.count: 40
> > # This got rid of the occasional "Could not obtain block"'s
> > dfs.datanode.max.xcievers: 4096
> >
> > __MAPRED-SITE.XML__
> >
> > mapred.tasktracker.map.tasks.maximum: 4
> > mapred.tasktracker.reduce.tasks.maximum: 4
> > mapred.child.java.opts: -Xmx2560m
> > mapreduce.reduce.shuffle.parallelcopies: 20
> > mapreduce.map.java.opts: -Xmx512m
> > mapreduce.reduce.java.opts: -Xmx512m
> > # Compression codecs are configured and seem to work fine
> > mapred.compress.map.output: true
> > mapred.map.output.compression.codec:
> com.hadoop.compression.lzo.LzoCodec
> >


From general-return-3461-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 13:41:36 2011
Return-Path: <general-return-3461-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3B93E4003
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 13:41:36 +0000 (UTC)
Received: (qmail 32406 invoked by uid 500); 11 May 2011 13:41:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32330 invoked by uid 500); 11 May 2011 13:41:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32322 invoked by uid 99); 11 May 2011 13:41:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 13:41:34 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 74.125.83.48 is neither permitted nor denied by domain of james@tynt.com)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 13:41:27 +0000
Received: by gwj22 with SMTP id 22so212219gwj.35
        for <general@hadoop.apache.org>; Wed, 11 May 2011 06:41:06 -0700 (PDT)
Received: by 10.101.12.5 with SMTP id p5mr5653847ani.39.1305121265828; Wed, 11
 May 2011 06:41:05 -0700 (PDT)
References: <ADF94D8555C7A246B86A633685E0178AB90A474E7A@planck.ka.sara.nl>
 <763952E3-8AAD-4757-99DC-57ABC06BBC2C@tynt.com> <ADF94D8555C7A246B86A633685E0178AB90A474E81@planck.ka.sara.nl>
From: James Seigel <james@tynt.com>
In-Reply-To: <ADF94D8555C7A246B86A633685E0178AB90A474E81@planck.ka.sara.nl>
Mime-Version: 1.0 (iPhone Mail 8J2)
Date: Wed, 11 May 2011 07:41:02 -0600
Message-ID: <8740675753567357118@unknownmsgid>
Subject: Re: Stability issue - dead DN's
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

I noticed you crossposted to cloudera, what version of theirs are you running?

Cheers
James

Sent from my mobile. Please excuse the typos.

On 2011-05-11, at 7:39 AM, Evert Lammerts <Evert.Lammerts@sara.nl> wrote:

> Hi James,
>
> Hadoop version is 0.20.2 (find that and more on our setup also in my first mail, under heading "The cluster").
>
> Below I) an example stacktrace of losing a datanode is and II) an example of a "Could not obtain block" IOException.
>
> Cheers,
> Evert
>
> 11/05/11 15:06:43 INFO hdfs.DFSClient: Failed to connect to /192.168.28.214:50050, add to deadNodes and continue
> java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/192.168.28.209:50726 remote=/192.168.28.214:50050]
>        at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
>        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>        at java.io.DataInputStream.readShort(DataInputStream.java:295)
>        at org.apache.hadoop.hdfs.DFSClient$BlockReader.newBlockReader(DFSClient.java:1478)
>        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1811)
>        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1948)
>        at java.io.DataInputStream.readFully(DataInputStream.java:178)
>        at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63)
>        at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
>        at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1945)
>        at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1845)
>        at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1891)
>        at org.apache.mahout.common.iterator.sequencefile.SequenceFileIterator.computeNext(SequenceFileIterator.java:95)
>        at org.apache.mahout.common.iterator.sequencefile.SequenceFileIterator.computeNext(SequenceFileIterator.java:1)
>        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135)
>        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130)
>        at nl.liacs.infrawatch.hadoop.kmeans.RandomSeedGenerator.buildRandom(RandomSeedGenerator.java:85)
>        at nl.liacs.infrawatch.hadoop.kmeans.Job.run(Job.java:171)
>        at nl.liacs.infrawatch.hadoop.kmeans.Job.main(Job.java:74)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
>
>
> 11/05/10 09:43:47 INFO mapred.JobClient:  map 82% reduce 17% 11/05/10 09:44:39 INFO mapred.JobClient: Task Id :
> attempt_201104121440_0122_m_000225_0, Status : FAILED
> java.io.IOException: Could not obtain block:
> blk_4397122445076815438_4097927
> file=/user/joaquin/data/20081201/20081201.039
>        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1993)
>        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1800)
>        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1948)
>        at java.io.DataInputStream.read(DataInputStream.java:83)
>        at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
>        at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:97)
>        at nl.liacs.infrawatch.hadoop.kmeans.KeyValueLineRecordReader.nextKeyValue(KeyValueLineRecordReader.java:94)
>        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:455)
>        at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
>        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
>        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:646)
>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
>        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at javax.security.auth.Subject.doAs(Subject.java:396)
>        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
>        at org.apache.hadoop.mapred.Child.main(Child.java:262)
>
>> -----Original Message-----
>> From: James Seigel [mailto:james@tynt.com]
>> Sent: woensdag 11 mei 2011 14:54
>> To: general@hadoop.apache.org
>> Subject: Re: Stability issue - dead DN's
>>
>> Evert,
>>
>> What's the stack trace and what version of hadoop do you have installed
>> Sir!
>>
>> James.
>> On 2011-05-11, at 3:23 AM, Evert Lammerts wrote:
>>
>>> Hi list,
>>>
>>> I notice that whenever our Hadoop installation is put under a heavy
>> load we lose one or two (on a total of five) datanodes. This results in
>> IOExceptions, and affects the overall performance of the job being run.
>> Can anybody give me advise or best practices on a different
>> configuration to increase the stability? Below I've included the specs
>> of the cluster, the hadoop related config and an example of when which
>> things go wrong. Any help is very much appreciated, and if I can
>> provide any other info please let me know.
>>>
>>> Cheers,
>>> Evert
>>>
>>> == What goes wrong, and when ==
>>>
>>> See attached a screenshot of Ganglia when the cluster is under load
>> of a single job. This job:
>>> * reads ~1TB from HDFS
>>> * writes ~200GB to HDFS
>>> * runs 288 Mappers and 35 Reducers
>>>
>>> When the job runs it takes all available Map and Reduce slots. The
>> system starts swapping and there is a short time interval during which
>> most cores are in WAIT. After that the job really starts running. At
>> around half way, one or two datanodes become unreachable and are marked
>> as dead nodes. The amount of under-replicated blocks becomes huge. Then
>> some "java.io.IOException: Could not obtain block" are thrown in
>> Mappers. The job does manage to finish successfully after around 3.5
>> hours, but my fear is that when we make the input much larger - which
>> we want - the system becomes too unstable to finish the job.
>>>
>>> Maybe worth mentioning - never know what might help diagnostics.  We
>> notice that memory usage becomes less when we switch our keys from Text
>> to LongWritable. Also, the Mappers are done in a fraction of the time.
>> However, this for some reason results in much more network traffic and
>> makes Reducers extremely slow. We're working on figuring out what
>> causes this.
>>>
>>>
>>> == The cluster ==
>>>
>>> We have a cluster that consists of 6 Sun Thumpers running Hadoop
>> 0.20.2 on CentOS 5.5. One of them acts as NN and JT, the other 5 run
>> DN's and TT's. Each node has:
>>> * 16GB RAM
>>> * 32GB swapspace
>>> * 4 cores
>>> * 11 LVM's of 4 x 500GB disks (2TB in total) for HDFS
>>> * non-HDFS stuff on separate disks
>>> * a 2x1GE bonded network interface for interconnects
>>> * a 2x1GE bonded network interface for external access
>>>
>>> I realize that this is not a well balanced system, but it's what we
>> had available for a prototype environment. We're working on putting
>> together a specification for a much larger production environment.
>>>
>>>
>>> == Hadoop config ==
>>>
>>> Here some properties that I think might be relevant:
>>>
>>> __CORE-SITE.XML__
>>>
>>> fs.inmemory.size.mb: 200
>>> mapreduce.task.io.sort.factor: 100
>>> mapreduce.task.io.sort.mb: 200
>>> # 1024*1024*4 MB, blocksize of the LVM's
>>> io.file.buffer.size: 4194304
>>>
>>> __HDFS-SITE.XML__
>>>
>>> # 1024*1024*4*32 MB, 32 times the blocksize of the LVM's
>>> dfs.block.size: 134217728
>>> # Only 5 DN's, but this shouldn't hurt
>>> dfs.namenode.handler.count: 40
>>> # This got rid of the occasional "Could not obtain block"'s
>>> dfs.datanode.max.xcievers: 4096
>>>
>>> __MAPRED-SITE.XML__
>>>
>>> mapred.tasktracker.map.tasks.maximum: 4
>>> mapred.tasktracker.reduce.tasks.maximum: 4
>>> mapred.child.java.opts: -Xmx2560m
>>> mapreduce.reduce.shuffle.parallelcopies: 20
>>> mapreduce.map.java.opts: -Xmx512m
>>> mapreduce.reduce.java.opts: -Xmx512m
>>> # Compression codecs are configured and seem to work fine
>>> mapred.compress.map.output: true
>>> mapred.map.output.compression.codec:
>> com.hadoop.compression.lzo.LzoCodec
>>>
>

From general-return-3462-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 13:42:45 2011
Return-Path: <general-return-3462-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E18E940B9
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 13:42:44 +0000 (UTC)
Received: (qmail 36060 invoked by uid 500); 11 May 2011 13:42:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36012 invoked by uid 500); 11 May 2011 13:42:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36004 invoked by uid 99); 11 May 2011 13:42:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 13:42:43 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 209.85.213.48 is neither permitted nor denied by domain of james@tynt.com)
Received: from [209.85.213.48] (HELO mail-yw0-f48.google.com) (209.85.213.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 13:42:38 +0000
Received: by ywk9 with SMTP id 9so213629ywk.35
        for <general@hadoop.apache.org>; Wed, 11 May 2011 06:42:17 -0700 (PDT)
Received: by 10.101.12.5 with SMTP id p5mr5654709ani.39.1305121337540; Wed, 11
 May 2011 06:42:17 -0700 (PDT)
References: <ADF94D8555C7A246B86A633685E0178AB90A474E7A@planck.ka.sara.nl>
 <763952E3-8AAD-4757-99DC-57ABC06BBC2C@tynt.com> <ADF94D8555C7A246B86A633685E0178AB90A474E81@planck.ka.sara.nl>
From: James Seigel <james@tynt.com>
In-Reply-To: <ADF94D8555C7A246B86A633685E0178AB90A474E81@planck.ka.sara.nl>
Mime-Version: 1.0 (iPhone Mail 8J2)
Date: Wed, 11 May 2011 07:42:14 -0600
Message-ID: <-142259448883561283@unknownmsgid>
Subject: Re: Stability issue - dead DN's
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Content-Type: text/plain; charset=ISO-8859-1

Have you dug into the dead DN logs as well?

James

Sent from my mobile. Please excuse the typos.

On 2011-05-11, at 7:39 AM, Evert Lammerts <Evert.Lammerts@sara.nl> wrote:

> Hi James,
>
> Hadoop version is 0.20.2 (find that and more on our setup also in my first mail, under heading "The cluster").
>
> Below I) an example stacktrace of losing a datanode is and II) an example of a "Could not obtain block" IOException.
>
> Cheers,
> Evert
>
> 11/05/11 15:06:43 INFO hdfs.DFSClient: Failed to connect to /192.168.28.214:50050, add to deadNodes and continue
> java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/192.168.28.209:50726 remote=/192.168.28.214:50050]
>        at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
>        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
>        at java.io.DataInputStream.readShort(DataInputStream.java:295)
>        at org.apache.hadoop.hdfs.DFSClient$BlockReader.newBlockReader(DFSClient.java:1478)
>        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1811)
>        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1948)
>        at java.io.DataInputStream.readFully(DataInputStream.java:178)
>        at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63)
>        at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
>        at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1945)
>        at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1845)
>        at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1891)
>        at org.apache.mahout.common.iterator.sequencefile.SequenceFileIterator.computeNext(SequenceFileIterator.java:95)
>        at org.apache.mahout.common.iterator.sequencefile.SequenceFileIterator.computeNext(SequenceFileIterator.java:1)
>        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135)
>        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130)
>        at nl.liacs.infrawatch.hadoop.kmeans.RandomSeedGenerator.buildRandom(RandomSeedGenerator.java:85)
>        at nl.liacs.infrawatch.hadoop.kmeans.Job.run(Job.java:171)
>        at nl.liacs.infrawatch.hadoop.kmeans.Job.main(Job.java:74)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
>
>
> 11/05/10 09:43:47 INFO mapred.JobClient:  map 82% reduce 17% 11/05/10 09:44:39 INFO mapred.JobClient: Task Id :
> attempt_201104121440_0122_m_000225_0, Status : FAILED
> java.io.IOException: Could not obtain block:
> blk_4397122445076815438_4097927
> file=/user/joaquin/data/20081201/20081201.039
>        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1993)
>        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1800)
>        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1948)
>        at java.io.DataInputStream.read(DataInputStream.java:83)
>        at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
>        at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:97)
>        at nl.liacs.infrawatch.hadoop.kmeans.KeyValueLineRecordReader.nextKeyValue(KeyValueLineRecordReader.java:94)
>        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:455)
>        at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
>        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
>        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:646)
>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
>        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at javax.security.auth.Subject.doAs(Subject.java:396)
>        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
>        at org.apache.hadoop.mapred.Child.main(Child.java:262)
>
>> -----Original Message-----
>> From: James Seigel [mailto:james@tynt.com]
>> Sent: woensdag 11 mei 2011 14:54
>> To: general@hadoop.apache.org
>> Subject: Re: Stability issue - dead DN's
>>
>> Evert,
>>
>> What's the stack trace and what version of hadoop do you have installed
>> Sir!
>>
>> James.
>> On 2011-05-11, at 3:23 AM, Evert Lammerts wrote:
>>
>>> Hi list,
>>>
>>> I notice that whenever our Hadoop installation is put under a heavy
>> load we lose one or two (on a total of five) datanodes. This results in
>> IOExceptions, and affects the overall performance of the job being run.
>> Can anybody give me advise or best practices on a different
>> configuration to increase the stability? Below I've included the specs
>> of the cluster, the hadoop related config and an example of when which
>> things go wrong. Any help is very much appreciated, and if I can
>> provide any other info please let me know.
>>>
>>> Cheers,
>>> Evert
>>>
>>> == What goes wrong, and when ==
>>>
>>> See attached a screenshot of Ganglia when the cluster is under load
>> of a single job. This job:
>>> * reads ~1TB from HDFS
>>> * writes ~200GB to HDFS
>>> * runs 288 Mappers and 35 Reducers
>>>
>>> When the job runs it takes all available Map and Reduce slots. The
>> system starts swapping and there is a short time interval during which
>> most cores are in WAIT. After that the job really starts running. At
>> around half way, one or two datanodes become unreachable and are marked
>> as dead nodes. The amount of under-replicated blocks becomes huge. Then
>> some "java.io.IOException: Could not obtain block" are thrown in
>> Mappers. The job does manage to finish successfully after around 3.5
>> hours, but my fear is that when we make the input much larger - which
>> we want - the system becomes too unstable to finish the job.
>>>
>>> Maybe worth mentioning - never know what might help diagnostics.  We
>> notice that memory usage becomes less when we switch our keys from Text
>> to LongWritable. Also, the Mappers are done in a fraction of the time.
>> However, this for some reason results in much more network traffic and
>> makes Reducers extremely slow. We're working on figuring out what
>> causes this.
>>>
>>>
>>> == The cluster ==
>>>
>>> We have a cluster that consists of 6 Sun Thumpers running Hadoop
>> 0.20.2 on CentOS 5.5. One of them acts as NN and JT, the other 5 run
>> DN's and TT's. Each node has:
>>> * 16GB RAM
>>> * 32GB swapspace
>>> * 4 cores
>>> * 11 LVM's of 4 x 500GB disks (2TB in total) for HDFS
>>> * non-HDFS stuff on separate disks
>>> * a 2x1GE bonded network interface for interconnects
>>> * a 2x1GE bonded network interface for external access
>>>
>>> I realize that this is not a well balanced system, but it's what we
>> had available for a prototype environment. We're working on putting
>> together a specification for a much larger production environment.
>>>
>>>
>>> == Hadoop config ==
>>>
>>> Here some properties that I think might be relevant:
>>>
>>> __CORE-SITE.XML__
>>>
>>> fs.inmemory.size.mb: 200
>>> mapreduce.task.io.sort.factor: 100
>>> mapreduce.task.io.sort.mb: 200
>>> # 1024*1024*4 MB, blocksize of the LVM's
>>> io.file.buffer.size: 4194304
>>>
>>> __HDFS-SITE.XML__
>>>
>>> # 1024*1024*4*32 MB, 32 times the blocksize of the LVM's
>>> dfs.block.size: 134217728
>>> # Only 5 DN's, but this shouldn't hurt
>>> dfs.namenode.handler.count: 40
>>> # This got rid of the occasional "Could not obtain block"'s
>>> dfs.datanode.max.xcievers: 4096
>>>
>>> __MAPRED-SITE.XML__
>>>
>>> mapred.tasktracker.map.tasks.maximum: 4
>>> mapred.tasktracker.reduce.tasks.maximum: 4
>>> mapred.child.java.opts: -Xmx2560m
>>> mapreduce.reduce.shuffle.parallelcopies: 20
>>> mapreduce.map.java.opts: -Xmx512m
>>> mapreduce.reduce.java.opts: -Xmx512m
>>> # Compression codecs are configured and seem to work fine
>>> mapred.compress.map.output: true
>>> mapred.map.output.compression.codec:
>> com.hadoop.compression.lzo.LzoCodec
>>>
>

From general-return-3463-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 15:04:01 2011
Return-Path: <general-return-3463-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 140804DE4
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 15:04:00 +0000 (UTC)
Received: (qmail 25944 invoked by uid 500); 11 May 2011 15:03:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25874 invoked by uid 500); 11 May 2011 15:03:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25866 invoked by uid 99); 11 May 2011 15:03:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:03:59 +0000
X-ASF-Spam-Status: No, hits=3.3 required=5.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:03:53 +0000
Received: from sp1-ex07cas03.ds.corp.yahoo.com (sp1-ex07cas03.ds.corp.yahoo.com [216.252.116.151])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p4BF2RmW017184
	for <general@hadoop.apache.org>; Wed, 11 May 2011 08:02:27 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1305126147;
	bh=mu7cc1tmjZtbVgaM8LH0TBHWVls6U0rU1bxHuEZ7SJI=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 MIME-Version;
	b=FCS/WCd2vXPSfw+lWPan7b8gP3tMgnoZsLrWNJMH/kiKFKd+4oWouu2JNftS/UdOp
	 Uh0KC4Jra7+guR737ij63wRJ0HrCIEzrfhNEsQsQ3suczFZXKxaWvW8hBBI93lYBbF
	 fLwYPDjZSjiGePGayQsQr58CslZdgu/Vic7hNpJs=
Received: from SP1-EX07VS02.ds.corp.yahoo.com ([216.252.116.135]) by
 sp1-ex07cas03.ds.corp.yahoo.com ([216.252.116.151]) with mapi; Wed, 11 May
 2011 08:02:27 -0700
From: Devaraj Das <ddas@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 11 May 2011 08:02:26 -0700
Subject: Re: OT: anyone else going to berlin buzzwords?
Thread-Topic: OT: anyone else going to berlin buzzwords?
Thread-Index: AcwPyTNg/oOFdfAqQ92h/sattuN1dwAIz0SI
Message-ID: <C9EFF312.35694%ddas@yahoo-inc.com>
In-Reply-To: <4DCA6997.2040200@apache.org>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C9EFF31235694ddasyahooinccom_"
MIME-Version: 1.0

--_000_C9EFF31235694ddasyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

I can do a tour of the MR-279 codebase but I am currently flying out on the=
 afternoon of Wednesday. Let me see if I can change the flight bookings.


On 5/11/11 3:48 AM, "Steve Loughran" <stevel@apache.org> wrote:

On 11/05/2011 03:12, Devaraj Das wrote:
> I'll be there .. Looking forward to meeting up with Ian and the other Had=
oop'ers...
>


OK, there's now a tentative plan for a Hadoop hackathon, i'd like a
volunteer to do a tour of the MR-279 codebase, and we have a volunteer
to talk about gridmix.

http://berlinbuzzwords.de/wiki/hadoop-hackathon

Being a hackathon and all, some coding theme would be good. Testing MR-279?


--_000_C9EFF31235694ddasyahooinccom_--

From general-return-3464-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 16:45:47 2011
Return-Path: <general-return-3464-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0E770408B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 16:45:47 +0000 (UTC)
Received: (qmail 46093 invoked by uid 500); 11 May 2011 16:45:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45970 invoked by uid 500); 11 May 2011 16:45:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45961 invoked by uid 99); 11 May 2011 16:45:45 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 16:45:45 +0000
Received: from localhost (HELO dhcp-02.private.iobm.com) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 16:45:45 +0000
Subject: Re: Stability issue - dead DN's
Mime-Version: 1.0 (Apple Message framework v1082)
Content-Type: text/plain; charset=us-ascii
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <BANLkTi=NaiXQPAkdS-LPW-Swev6XYD0Xjw@mail.gmail.com>
Date: Wed, 11 May 2011 09:45:44 -0700
Cc: <cdh-user@cloudera.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <DA1F03C5-7F7B-412A-84E7-774294085603@apache.org>
References: <ADF94D8555C7A246B86A633685E0178AB90A474E7A@planck.ka.sara.nl> <BANLkTi=NaiXQPAkdS-LPW-Swev6XYD0Xjw@mail.gmail.com>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On May 11, 2011, at 5:57 AM, Eric Fiala wrote:
>=20
> If we do the math that means [ map.tasks.max * mapred.child.java.opts =
]  +
> [ reduce.tasks.max * mapred.child.java.opts ] =3D> or [ 4 * 2.5G ] + [ =
4 *
> 2.5G ] is greater than the amount of physical RAM in the machine.
> This doesn't account for the base tasktracker and datanode process + =
OS
> overhead and whatever else may be hoarding resources on the systems.

	+1 to what Eric said.

	You've exhausted memory and now the whole system is falling =
apart. =20

> I would play with this ratio, either less maps / reduces max - or =
lower your
> child.java.opts so that when you are fully subscribed you are not =
using
> more resource than the machine can offer.

	Yup.

> Also, setting mapred.reduce.slowstart.completed.maps  to 1.00 or some =
other
> value close to 1 would be one way to guarantee only 4 either maps or =
reduces
> to be running at once and address (albeit in a duct tape like way) the
> oversubscription problem you are seeing (this represents the fractions =
of
> maps that should complete before initiating the reduce phase).

	slowstart isn't really going to help you much here.  All it =
takes is another job with the same settings running at the same time and =
processes will start dying again.  That said, the default for slowstart =
is incredibly stupid for the vast majority.  Something closer to .70 or =
.80 is more realistic.


>> * a 2x1GE bonded network interface for interconnects
>> * a 2x1GE bonded network interface for external access

	Multiple NICs on a box can sometimes cause big performance =
problems with Hadoop.  So watch your traffic carefully.



From general-return-3465-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 17:33:26 2011
Return-Path: <general-return-3465-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4EE984AEF
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 17:33:26 +0000 (UTC)
Received: (qmail 15778 invoked by uid 500); 11 May 2011 17:33:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15729 invoked by uid 500); 11 May 2011 17:33:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 10499 invoked by uid 99); 11 May 2011 17:28:44 -0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,MIME_QP_LONG_LINE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of im_gumby@hotmail.com designates 65.55.116.13 as permitted sender)
X-Originating-IP: [187.146.84.243]
X-Originating-Email: [im_gumby@hotmail.com]
Message-ID: <BLU0-SMTP107D4E21A69A6C860D8CBB2F0860@phx.gbl>
References: <ADF94D8555C7A246B86A633685E0178AB90A474E7A@planck.ka.sara.nl> <BANLkTi=NaiXQPAkdS-LPW-Swev6XYD0Xjw@mail.gmail.com> <DA1F03C5-7F7B-412A-84E7-774294085603@apache.org>
In-Reply-To: <DA1F03C5-7F7B-412A-84E7-774294085603@apache.org>
MIME-Version: 1.0 (iPad Mail 8H7)
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="us-ascii"
CC: "<general@hadoop.apache.org>" <general@hadoop.apache.org>,
 "<cdh-user@cloudera.org>" <cdh-user@cloudera.org>
X-Mailer: iPad Mail (8H7)
From: Michel Segel <michael_segel@hotmail.com>
Subject: Re: Stability issue - dead DN's
Date: Wed, 11 May 2011 12:22:28 -0500
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-OriginalArrivalTime: 11 May 2011 17:28:13.0395 (UTC) FILETIME=[CE4ABA30:01CC1000]
Sender: <im_gumby@hotmail.com>
X-Virus-Checked: Checked by ClamAV on apache.org

You really really don't want to do this.
Long story short... It won't work.=20

Just a suggestion.. You don't want anyone on your cluster itself. They shoul=
d interact wit edge nodes, which are 'Hadoop aware'. Then your cluster has a=
 single network to worry about.


Sent from a remote device. Please excuse any typos...

Mike Segel

On May 11, 2011, at 11:45 AM, Allen Wittenauer <aw@apache.org> wrote:

>=20
>=20
>=20
>=20
>>> * a 2x1GE bonded network interface for interconnects
>>> * a 2x1GE bonded network interface for external access
>=20
>    Multiple NICs on a box can sometimes cause big performance problems wit=
h Hadoop.  So watch your traffic carefully.
>=20
>=20
>=20

From general-return-3466-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 17:36:34 2011
Return-Path: <general-return-3466-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 37E1540B9
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 17:36:34 +0000 (UTC)
Received: (qmail 20241 invoked by uid 500); 11 May 2011 17:36:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20193 invoked by uid 500); 11 May 2011 17:36:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 20185 invoked by uid 99); 11 May 2011 17:36:32 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 17:36:32 +0000
Received: from localhost (HELO mail-qy0-f169.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 17:36:32 +0000
Received: by qyk2 with SMTP id 2so3068823qyk.14
        for <general@hadoop.apache.org>; Wed, 11 May 2011 10:36:31 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.6.196 with SMTP id a4mr7384972qca.143.1305135391453; Wed,
 11 May 2011 10:36:31 -0700 (PDT)
Received: by 10.229.235.131 with HTTP; Wed, 11 May 2011 10:36:31 -0700 (PDT)
In-Reply-To: <BANLkTim68H=8ngBfzMsVrQOb9PmY7FviLQ@mail.gmail.com>
References: <12853C59-08F1-4F98-ACF3-800F06BC7A5F@apache.org>
	<BANLkTinZ=xb6kJ5PTeLN5KKD9b-cwaM0OQ@mail.gmail.com>
	<BANLkTi=1sQoL68k_2UmA1_Oo2CyXsM3fAw@mail.gmail.com>
	<DBCE9300-0D9D-4A9A-99B4-878139A028EE@yahoo-inc.com>
	<61C1E656-0649-4097-8709-FDC596ED57E9@yahoo-inc.com>
	<F7921362-CC1B-435C-890A-E1BD181E03C0@yahoo-inc.com>
	<8D047C89-6430-4FDE-A9C4-CFEE79644AC6@Holsman.NET>
	<BANLkTi=JDBneLBHsCh1T-PF8C7wNbra0kw@mail.gmail.com>
	<BANLkTim68H=8ngBfzMsVrQOb9PmY7FviLQ@mail.gmail.com>
Date: Wed, 11 May 2011 10:36:31 -0700
Message-ID: <BANLkTi=KoCF9MB717JSFsZG3fgy2U+qF=w@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc0
From: "Owen O'Malley" <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175cb12064338c04a30383cd

--0015175cb12064338c04a30383cd
Content-Type: text/plain; charset=UTF-8

The vote on 0.20.203.0-rc1 passes. We have our first stable release in a
year. The votes were:

13 +1's (10 binding: Arun, Chris, Devaraj, Dhruba, Jakob, Mahadev, Nicholas,
Owen, Sanjay,
                             Suresh)
7 -1's (4 binding: Doug, Eli, Konstantin, Todd)

Konstantin, your issue with the test cases requiring a umask 02 is a good
point. I'll patch it and can roll a 0.20.203.1 release candidate.

-- Owen

--0015175cb12064338c04a30383cd--

From general-return-3467-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 17:38:40 2011
Return-Path: <general-return-3467-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ABF41411D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 17:38:40 +0000 (UTC)
Received: (qmail 22832 invoked by uid 500); 11 May 2011 17:38:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22776 invoked by uid 500); 11 May 2011 17:38:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22768 invoked by uid 99); 11 May 2011 17:38:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 17:38:39 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 17:38:31 +0000
Received: by qwj9 with SMTP id 9so501167qwj.35
        for <general@hadoop.apache.org>; Wed, 11 May 2011 10:38:10 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=h3T3cwsfEIvZCEWzw+phR8ZLWRpyGEmb6z9sldz/OSo=;
        b=BnTJYJ8v17eDCfwR52gVh+MjFbi8O62SgRdQCwtzbo05y1DzxfZzmLyLI2rFGJg0KR
         pze+Uwp9TllDJrz8ZCNrnmWgJEvP6jrmJDbTNLBAlGOe55/7UAba+EuK/oIjodxiwq4N
         PbbD5K0Dkz4bH9FMZ6AyLC3Q1HnXGBt/PpLAM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=IKV0s2tS4s5xGpKi0lWs03E9J1Xof9KJPPdU6e3ZVGqo+C/4VZSUf+JYR03v9cfNZy
         XKixOv+ztYsPezovG9SUk/gIKODOo1nYRGGvJYNGGCmGYoSMg3OCgMLhSMHy6kPHl1x3
         QvPxKnRjNhzE8WPvwHGRhyB+7SntUmzqkDCUs=
Received: by 10.229.77.205 with SMTP id h13mr7436461qck.287.1305135490658;
        Wed, 11 May 2011 10:38:10 -0700 (PDT)
Received: from [10.172.1.95] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id s16sm225734qco.13.2011.05.11.10.38.08
        (version=TLSv1/SSLv3 cipher=OTHER);
        Wed, 11 May 2011 10:38:09 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc0
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <BANLkTi=KoCF9MB717JSFsZG3fgy2U+qF=w@mail.gmail.com>
Date: Thu, 12 May 2011 03:38:05 +1000
Content-Transfer-Encoding: quoted-printable
Message-Id: <965E8AEC-2DF5-4A51-90E3-21D4C101E5CC@Holsman.NET>
References: <12853C59-08F1-4F98-ACF3-800F06BC7A5F@apache.org> <BANLkTinZ=xb6kJ5PTeLN5KKD9b-cwaM0OQ@mail.gmail.com> <BANLkTi=1sQoL68k_2UmA1_Oo2CyXsM3fAw@mail.gmail.com> <DBCE9300-0D9D-4A9A-99B4-878139A028EE@yahoo-inc.com> <61C1E656-0649-4097-8709-FDC596ED57E9@yahoo-inc.com> <F7921362-CC1B-435C-890A-E1BD181E03C0@yahoo-inc.com> <8D047C89-6430-4FDE-A9C4-CFEE79644AC6@Holsman.NET> <BANLkTi=JDBneLBHsCh1T-PF8C7wNbra0kw@mail.gmail.com> <BANLkTim68H=8ngBfzMsVrQOb9PmY7FviLQ@mail.gmail.com> <BANLkTi=KoCF9MB717JSFsZG3fgy2U+qF=w@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

congratulations guys!
On May 12, 2011, at 3:36 AM, Owen O'Malley wrote:

> The vote on 0.20.203.0-rc1 passes. We have our first stable release in =
a
> year. The votes were:
>=20
> 13 +1's (10 binding: Arun, Chris, Devaraj, Dhruba, Jakob, Mahadev, =
Nicholas,
> Owen, Sanjay,
>                             Suresh)
> 7 -1's (4 binding: Doug, Eli, Konstantin, Todd)
>=20
> Konstantin, your issue with the test cases requiring a umask 02 is a =
good
> point. I'll patch it and can roll a 0.20.203.1 release candidate.
>=20
> -- Owen


From general-return-3468-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 18:41:08 2011
Return-Path: <general-return-3468-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5B2D0487D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 18:41:08 +0000 (UTC)
Received: (qmail 28843 invoked by uid 500); 11 May 2011 18:41:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28790 invoked by uid 500); 11 May 2011 18:41:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28782 invoked by uid 99); 11 May 2011 18:41:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 18:41:06 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of shv.hadoop@gmail.com designates 74.125.83.48 as permitted sender)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 18:41:00 +0000
Received: by gwj22 with SMTP id 22so373689gwj.35
        for <general@hadoop.apache.org>; Wed, 11 May 2011 11:40:39 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=sPOvSUAM6q7od/QHPAM69MM0irOJGQV4K+KankL2LRg=;
        b=OOFL3tB4a13MgSg8pXnBYEfiUlWubsckCWRwOdJkY9lb9W8geMuzkViV5EC/822msO
         ONLXWubY9jmeMXRnah/uAKkbfFfLuIKtyhI3AxhCCmnxw4mq0+VmZ5cGIE5TjKo/d4H/
         dLP3Qt/2WJUieKEeQ5C3q3YkmDK4bKiBNiUjo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=c1qu4xp0STtQsN4pfvBkibPtrFrqXZfyOSz6nClX4um6u3Emg75AVk4XynVU2kNEaR
         hVGnEZdPcpNQHqpGyKLDP8oS7gUOJs5QHEA954et3E7OX/9E0kZP2rmSY9u1dPXVmkwa
         cyFyCXhuVhsXGhUd1ia/boqYnISBSFWyKQmYA=
MIME-Version: 1.0
Received: by 10.236.109.19 with SMTP id r19mr5485830yhg.289.1305139239747;
 Wed, 11 May 2011 11:40:39 -0700 (PDT)
Received: by 10.147.124.12 with HTTP; Wed, 11 May 2011 11:40:39 -0700 (PDT)
In-Reply-To: <BANLkTi=KoCF9MB717JSFsZG3fgy2U+qF=w@mail.gmail.com>
References: <12853C59-08F1-4F98-ACF3-800F06BC7A5F@apache.org>
	<BANLkTinZ=xb6kJ5PTeLN5KKD9b-cwaM0OQ@mail.gmail.com>
	<BANLkTi=1sQoL68k_2UmA1_Oo2CyXsM3fAw@mail.gmail.com>
	<DBCE9300-0D9D-4A9A-99B4-878139A028EE@yahoo-inc.com>
	<61C1E656-0649-4097-8709-FDC596ED57E9@yahoo-inc.com>
	<F7921362-CC1B-435C-890A-E1BD181E03C0@yahoo-inc.com>
	<8D047C89-6430-4FDE-A9C4-CFEE79644AC6@Holsman.NET>
	<BANLkTi=JDBneLBHsCh1T-PF8C7wNbra0kw@mail.gmail.com>
	<BANLkTim68H=8ngBfzMsVrQOb9PmY7FviLQ@mail.gmail.com>
	<BANLkTi=KoCF9MB717JSFsZG3fgy2U+qF=w@mail.gmail.com>
Date: Wed, 11 May 2011 11:40:39 -0700
Message-ID: <BANLkTikQAqsxc4wCU8LRY93yHGEjQLZLgw@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc0
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0023547c8c5fc4823604a30468e0

--0023547c8c5fc4823604a30468e0
Content-Type: text/plain; charset=ISO-8859-1

> Konstantin, your issue with the test cases requiring a umask 02 is a good
> point. I'll patch it and can roll a 0.20.203.1 release candidate.

umask is not a big concern. I reset it to standard 0022.
Still there were 8 other test failures: 7 in mapred, and 1 hdfsproxy.
Stable release should pass unit tests.
Lets make 0.20.203.1 stable.

Thanks,
--Konstantin

On Wed, May 11, 2011 at 10:36 AM, Owen O'Malley <omalley@apache.org> wrote:

> The vote on 0.20.203.0-rc1 passes. We have our first stable release in a
> year. The votes were:
>
> 13 +1's (10 binding: Arun, Chris, Devaraj, Dhruba, Jakob, Mahadev,
> Nicholas,
> Owen, Sanjay,
>                             Suresh)
> 7 -1's (4 binding: Doug, Eli, Konstantin, Todd)
>
> Konstantin, your issue with the test cases requiring a umask 02 is a good
> point. I'll patch it and can roll a 0.20.203.1 release candidate.
>
> -- Owen
>

--0023547c8c5fc4823604a30468e0--

From general-return-3469-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 19:09:40 2011
Return-Path: <general-return-3469-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 562B44B59
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 19:09:40 +0000 (UTC)
Received: (qmail 84436 invoked by uid 500); 11 May 2011 19:09:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84387 invoked by uid 500); 11 May 2011 19:09:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84379 invoked by uid 99); 11 May 2011 19:09:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 19:09:38 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of angadi@gmail.com designates 209.85.210.176 as permitted sender)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 19:09:32 +0000
Received: by iym1 with SMTP id 1so911165iym.35
        for <general@hadoop.apache.org>; Wed, 11 May 2011 12:09:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=YCstV0F+8qa1kAgOgo+y8sncPEXbclD3KPru74JwEKg=;
        b=t2f7Ues2KywoxUNWkzpTN6pEw4/O7RM24INiG+3cCpmV0ciafpNfIb5rfLnZQGUfvW
         dBnR7xeeo0Q8J81lV3zBW+ufiW8zvHaa/CUnceGX88oWq1smI8+iu2ls5stjLgh8ooi1
         CUqYx1LIGTNvhl1+jLCh/r/TumbDBOpx6/qzU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Hixcc9EfGs3KOJWlaMYy0zL/1PSD96xqdL5MiNA7J1EurTDRcKKa+0M8CIBs32zzeL
         Nb8soeOH6oi422qgc/+UkgtewHkebqDwMw8jZryZcxfxmTUCTV4mqFVNJ1tuHWm/BSSt
         CNe4NjzieNN60AT1n0ztMdjFOCRhN/S2jExgg=
MIME-Version: 1.0
Received: by 10.231.176.6 with SMTP id bc6mr7023950ibb.147.1305140952021; Wed,
 11 May 2011 12:09:12 -0700 (PDT)
Received: by 10.231.10.141 with HTTP; Wed, 11 May 2011 12:09:11 -0700 (PDT)
In-Reply-To: <BANLkTi=KoCF9MB717JSFsZG3fgy2U+qF=w@mail.gmail.com>
References: <12853C59-08F1-4F98-ACF3-800F06BC7A5F@apache.org>
	<BANLkTinZ=xb6kJ5PTeLN5KKD9b-cwaM0OQ@mail.gmail.com>
	<BANLkTi=1sQoL68k_2UmA1_Oo2CyXsM3fAw@mail.gmail.com>
	<DBCE9300-0D9D-4A9A-99B4-878139A028EE@yahoo-inc.com>
	<61C1E656-0649-4097-8709-FDC596ED57E9@yahoo-inc.com>
	<F7921362-CC1B-435C-890A-E1BD181E03C0@yahoo-inc.com>
	<8D047C89-6430-4FDE-A9C4-CFEE79644AC6@Holsman.NET>
	<BANLkTi=JDBneLBHsCh1T-PF8C7wNbra0kw@mail.gmail.com>
	<BANLkTim68H=8ngBfzMsVrQOb9PmY7FviLQ@mail.gmail.com>
	<BANLkTi=KoCF9MB717JSFsZG3fgy2U+qF=w@mail.gmail.com>
Date: Wed, 11 May 2011 12:09:11 -0700
Message-ID: <BANLkTikDF2gbkQ_Yp=bGTmXbainO6Zd91A@mail.gmail.com>
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc0
From: Raghu Angadi <angadi@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364c7ab3d3bc5c04a304ce1f

--0016364c7ab3d3bc5c04a304ce1f
Content-Type: text/plain; charset=ISO-8859-1

Congrats anad thanks to all the developers for such passion and hard work
they they put into a very important project.

+1 for getting new 0.20 release out with security. It is great news for
users for that new apache release is finally coming out. I hope most of the
technical as well as non-technical issues will resolved this or next
release.

There are many important issues raised in this thread and these are crucial
for future of Apache Hadoop. wanted to echo one of them in particular :
community is the most important aspect of the project and triumphs over the
rest.
I am sure the process would be much smoother going forward as we have more
frequent releases. This is probably the first real test for the
develop-a-large-feature-on-a-branch-and-merge process. Discussion here would
certainly lead to important improvements to the process.

Looking at from a different angle, Hadoop has a very enviable problem :
there is so much development it is very hard to co-ordinate and scale. It
has already scalled up a few times before, and with the leaders it has, it
is doing it again.

Raghu.

On Wed, May 11, 2011 at 10:36 AM, Owen O'Malley <omalley@apache.org> wrote:

> The vote on 0.20.203.0-rc1 passes. We have our first stable release in a
> year. The votes were:
>
> 13 +1's (10 binding: Arun, Chris, Devaraj, Dhruba, Jakob, Mahadev,
> Nicholas,
> Owen, Sanjay,
>                             Suresh)
> 7 -1's (4 binding: Doug, Eli, Konstantin, Todd)
>
> Konstantin, your issue with the test cases requiring a umask 02 is a good
> point. I'll patch it and can roll a 0.20.203.1 release candidate.
>
> -- Owen
>

--0016364c7ab3d3bc5c04a304ce1f--

From general-return-3470-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 20:21:31 2011
Return-Path: <general-return-3470-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EBA974900
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 20:21:30 +0000 (UTC)
Received: (qmail 34416 invoked by uid 500); 11 May 2011 20:21:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34330 invoked by uid 500); 11 May 2011 20:21:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34257 invoked by uid 99); 11 May 2011 20:21:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 20:21:30 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of owen.omalley@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 20:21:22 +0000
Received: by pwi16 with SMTP id 16so581618pwi.35
        for <general@hadoop.apache.org>; Wed, 11 May 2011 13:21:02 -0700 (PDT)
Received: by 10.68.17.232 with SMTP id r8mr14084652pbd.91.1305145262102;
        Wed, 11 May 2011 13:21:02 -0700 (PDT)
Received: from battlerock-lm.corp.yahoo.com (nat-dip4.cfw-a-gci.corp.yahoo.com [209.131.62.113])
        by mx.google.com with ESMTPS id j2sm181106pbg.60.2011.05.11.13.20.59
        (version=SSLv3 cipher=OTHER);
        Wed, 11 May 2011 13:21:00 -0700 (PDT)
From: Owen O'Malley <omalley@apache.org>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: Cleaning up documentation from website
Date: Wed, 11 May 2011 13:20:58 -0700
Message-Id: <FABE9911-352F-4AE2-96B1-27E66FB0369A@apache.org>
To: general@hadoop.apache.org
Mime-Version: 1.0 (Apple Message framework v1084)
X-Mailer: Apple Mail (2.1084)

We haven't cleaned up the version documentation in a long time:

lrwxrwxr-x  1 omalley   hadoop  11 May 11 20:14 current -> r0.20.203.0
drwxrwsr-x  7 cutting   hadoop  29 Nov 24  2008 r0.14.4
drwxrwsr-x  6 cutting   hadoop  22 Jan 23  2008 r0.15.2
drwxrwsr-x  6 nigel     hadoop  22 Jan 23  2008 r0.15.3
drwxrwsr-x  6 nigel     hadoop  28 Feb  8  2008 r0.16.0
drwxrwsr-x  6 nigel     hadoop  36 Mar 14  2008 r0.16.1
drwxrwsr-x  6 nigel     hadoop  36 Apr  3  2008 r0.16.2
drwxrwsr-x  6 nigel     hadoop  36 Apr 17  2008 r0.16.3
drwxrwsr-x  6 nigel     hadoop  36 May  9  2008 r0.16.4
drwxrwsr-x  6 nigel     hadoop  42 May 21  2008 r0.17.0
drwxrwsr-x  5 omalley   hadoop  42 Aug 20  2008 r0.17.1
drwxrwsr-x  7 omalley   hadoop  43 Dec 11  2008 r0.17.2
drwxrwsr-x  7 nigel     hadoop  51 Aug 22  2008 r0.18.0
drwxrwsr-x  7 nigel     hadoop  51 Sep 18  2008 r0.18.1
drwxrwsr-x  8 nigel     hadoop  52 Dec 11  2008 r0.18.2
drwxrwsr-x  8 nigel     hadoop  52 Jan 30  2009 r0.18.3
drwxrwsr-x  8 nigel     hadoop  58 Dec 11  2008 r0.19.0
drwxrwsr-x  8 nigel     hadoop  58 Feb 24  2009 r0.19.1
drwxrwsr-x  7 tomwhite  hadoop  57 Jun 30  2009 r0.19.2
drwxrwsr-x  7 nigel     hadoop  63 Apr  9  2009 r0.20.0
drwxrwsr-x  7 omalley   hadoop  64 Mar  1  2010 r0.20.1
drwxrwxr-x  7 cdouglas  hadoop  63 Aug 24  2010 r0.20.2
drwxrwxr-x  7 omalley   hadoop  63 May 11 20:11 r0.20.203.0
drwxrwxr-x  7 tomwhite  hadoop  31 Aug 17  2010 r0.21.0

Other than the history of who made each of the releases, there isn't a =
lot of value there. I propose that we trim this down to:

r0.20.2
r0.20.203.0
r0.21.0

Any concerns? If not, I'll clean up next week.

-- Owen=

From general-return-3471-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 20:55:49 2011
Return-Path: <general-return-3471-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 993414F79
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 20:55:49 +0000 (UTC)
Received: (qmail 95333 invoked by uid 500); 11 May 2011 20:55:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95288 invoked by uid 500); 11 May 2011 20:55:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95280 invoked by uid 99); 11 May 2011 20:55:48 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 20:55:48 +0000
Received: from localhost (HELO awittena-mn.linkedin.biz) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 20:55:47 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Cleaning up documentation from website
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <FABE9911-352F-4AE2-96B1-27E66FB0369A@apache.org>
Date: Wed, 11 May 2011 13:55:45 -0700
Content-Transfer-Encoding: 7bit
Message-Id: <4DED24B1-2180-4C78-BB8A-390BBAD0132E@apache.org>
References: <FABE9911-352F-4AE2-96B1-27E66FB0369A@apache.org>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On May 11, 2011, at 1:20 PM, Owen O'Malley wrote:

> We haven't cleaned up the version documentation in a long time:
> 
> lrwxrwxr-x  1 omalley   hadoop  11 May 11 20:14 current -> r0.20.203.0

	^--- this is a much more interesting discussion.

 	(What does "current" mean now?)
	

From general-return-3472-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 21:26:08 2011
Return-Path: <general-return-3472-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F2DA4456A
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 21:26:07 +0000 (UTC)
Received: (qmail 42366 invoked by uid 500); 11 May 2011 21:26:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42312 invoked by uid 500); 11 May 2011 21:26:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42304 invoked by uid 99); 11 May 2011 21:26:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:26:06 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:25:59 +0000
Received: from SP2-EX07CAS03.ds.corp.yahoo.com (sp2-ex07cas03.corp.sp2.yahoo.com [98.137.59.35])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p4BLOtB8076441;
	Wed, 11 May 2011 14:24:55 -0700 (PDT)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by
 SP2-EX07CAS03.ds.corp.yahoo.com ([98.137.59.35]) with mapi; Wed, 11 May 2011
 14:24:55 -0700
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 11 May 2011 14:24:47 -0700
Subject: Re: Defining Hadoop Compatibility -revisiting-
Thread-Topic: Defining Hadoop Compatibility -revisiting-
Thread-Index: AcwQId8TR15JwslTTzqga+rzSnJw+A==
Message-ID: <FDD1BE64-981D-4FA1-87BC-0FD4A880AB7A@yahoo-inc.com>
References: <4DC91392.2010308@apache.org>
In-Reply-To: <4DC91392.2010308@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

This is a really interesting topic!  I completely agree that we need to get=
 ahead of this.=20

I would be really interested in learning of any experience other apache pro=
jects, such as apache or tomcat have with these issues.=20

---
E14 - typing on glass

On May 10, 2011, at 6:31 AM, "Steve Loughran" <stevel@apache.org> wrote:

>=20
> Back in Jan 2011, I started a discussion about how to define Apache=20
> Hadoop Compatibility:
> http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C4D=
46B6AD.2020802@apache.org%3E
>=20
> I am now reading EMC HD "Enterprise Ready" Apache Hadoop datasheet
>=20
> http://www.greenplum.com/sites/default/files/EMC_Greenplum_HD_DS_Final_1.=
pdf
>=20
> It claims that their implementations are 100% compatible, even though=20
> the Enterprise edition uses a C filesystem. It also claims that both=20
> their software releases contain "Certified Stacks", without defining=20
> what Certified means, or who does the certification -only that it is an=20
> improvement.
>=20
>=20
> I think we should revisit this issue before people with their own=20
> agendas define what compatibility with Apache Hadoop is for us
>=20
>=20
> Licensing
> -Use of the Hadoop codebase must follow the Apache License
> http://www.apache.org/licenses/LICENSE-2.0
> -plug in components that are dynamically linked to (Filesystems and=20
> schedulers) don't appear to be derivative works on my reading of this,
>=20
> Naming
>  -this is something for branding@apache, they will have their opinions.=20
> The key one is that the name "Apache Hadoop" must get used, and it's=20
> important to make clear it is a derivative work.
>  -I don't think you can claim to have a Distribution/Fork/Version of=20
> Apache Hadoop if you swap out big chunks of it for alternate=20
> filesystems, MR engines, etc. Some description of this is needed
> "Supports the Apache Hadoop MapReduce engine on top of Filesystem XYZ"
>=20
> Compatibility
>  -the definition of the Hadoop interfaces and classes is the Apache=20
> Source tree,
>  -the definition of semantics of the Hadoop interfaces and classes is=20
> the Apache Source tree, including the test classes.
>  -the verification that the actual semantics of an Apache Hadoop=20
> release is compatible with the expected semantics is that current and=20
> future tests pass
>  -bug reports can highlight incompatibility with expectations of=20
> community users, and once incorporated into tests form part of the=20
> compatibility testing
>  -vendors can claim and even certify their derivative works as=20
> compatible with other versions of their derivative works, but cannot=20
> claim compatibility with Apache Hadoop unless their code passes the=20
> tests and is consistent with the bug reports marked as ("by design").=20
> Perhaps we should have tests that verify each of these "by design"=20
> bugreps to make them more formal.
>=20
> Certification
>  -I have no idea what this means in EMC's case, they just say "Certified"
>  -As we don't do any certification ourselves, it would seem impossible=20
> for us to certify that any derivative work is compatible.
>  -It may be best to state that nobody can certify their derivative as=20
> "compatible with Apache Hadoop" unless it passes all current test suites
>  -And require that anyone who declares compatibility define what they=20
> mean by this
>=20
> This is a good argument for getting more functional tests out there=20
> -whoever has more functional tests needs to get them into a test module=20
> that can be used to test real deployments.
>=20

From general-return-3473-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 21:30:38 2011
Return-Path: <general-return-3473-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3109A4B6D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 21:30:38 +0000 (UTC)
Received: (qmail 52758 invoked by uid 500); 11 May 2011 21:30:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52703 invoked by uid 500); 11 May 2011 21:30:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52695 invoked by uid 99); 11 May 2011 21:30:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:30:36 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of danharvey42@gmail.com designates 209.85.220.176 as permitted sender)
Received: from [209.85.220.176] (HELO mail-vx0-f176.google.com) (209.85.220.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:30:32 +0000
Received: by vxa37 with SMTP id 37so1089632vxa.35
        for <general@hadoop.apache.org>; Wed, 11 May 2011 14:30:11 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type;
        bh=J/MWmJB27Jt1F/HQP47VvlvcX7IPhChki7FkGMPx5kY=;
        b=blAGXQ5+2mr4C3YRT570TW3KXM5+purb4d35hXvtQELkcqAMthUfI0ciZkK9A6qwVk
         tPyCBAATs9IC63gF85EBKyKCgtGwWB5pmB1RpMYMeXmyj/EGDyVhecN4xvy19sSkyQ6w
         XC07aSJ9VC9hmko/7Qu+WswMdETEyf5TxbrFc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=OkS4Y2qynr/GSrd0FeApZqpUBj28oJwfeU2acq7lyUnhaJYkoqWnpmlEUoV7+pPgZ4
         LkOcJIcE8+KCDpxUb2/HchYwX9lCBIzK2SkficH7bNSS0aLbiNvdDgWGNKhwFXXQY9CS
         h9jsCuCtBOq4JDb0widbgLVKJRAnYIuM6dlPQ=
Received: by 10.52.74.99 with SMTP id s3mr2094913vdv.108.1305149411247; Wed,
 11 May 2011 14:30:11 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.52.89.226 with HTTP; Wed, 11 May 2011 14:29:51 -0700 (PDT)
In-Reply-To: <C9EFF312.35694%ddas@yahoo-inc.com>
References: <4DCA6997.2040200@apache.org> <C9EFF312.35694%ddas@yahoo-inc.com>
From: Dan <danharvey42@gmail.com>
Date: Wed, 11 May 2011 22:29:51 +0100
Message-ID: <BANLkTincfMk1K0rG7qv69PFpLwi646zKPw@mail.gmail.com>
Subject: Re: OT: anyone else going to berlin buzzwords?
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

I'll be heading there too, probably there till Thursday/Friday so
could attend a hackathon after the conference.

It would be great if it wasn't the same time as the HBase hackathon
which is planned for the 8th, or could be combined with it if it's
better on the Wednesday? I'm not sure who's organising that...

On Wed, May 11, 2011 at 4:02 PM, Devaraj Das <ddas@yahoo-inc.com> wrote:
> I can do a tour of the MR-279 codebase but I am currently flying out on the afternoon of Wednesday. Let me see if I can change the flight bookings.
>
>
> On 5/11/11 3:48 AM, "Steve Loughran" <stevel@apache.org> wrote:
>
> On 11/05/2011 03:12, Devaraj Das wrote:
>> I'll be there .. Looking forward to meeting up with Ian and the other Hadoop'ers...
>>
>
>
> OK, there's now a tentative plan for a Hadoop hackathon, i'd like a
> volunteer to do a tour of the MR-279 codebase, and we have a volunteer
> to talk about gridmix.
>
> http://berlinbuzzwords.de/wiki/hadoop-hackathon
>
> Being a hackathon and all, some coding theme would be good. Testing MR-279?
>
>

From general-return-3474-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 21:31:39 2011
Return-Path: <general-return-3474-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7A49642CB
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 21:31:39 +0000 (UTC)
Received: (qmail 54429 invoked by uid 500); 11 May 2011 21:31:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54331 invoked by uid 500); 11 May 2011 21:31:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54323 invoked by uid 99); 11 May 2011 21:31:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:31:37 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:31:32 +0000
Received: from SP1-EX07CAS02.ds.corp.yahoo.com (sp1-ex07cas02.ds.corp.yahoo.com [216.252.116.138])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p4BLTLMl054661
	for <general@hadoop.apache.org>; Wed, 11 May 2011 14:29:21 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1305149361;
	bh=kS15+hYwUZT4TpgKq8v6Oelw99wpRNHROFxFC619nPA=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:MIME-Version;
	b=C5L7/8WOCpROB8hGoJa9pJr9VhkQl1lfnIj3Bz0uIfqxCZQ9VI48CBuN3cTE7nrgo
	 5VcCDvUwyT2POOaMtWgDo3uJwNyDV9hRH/iexuLNlpiRBKp2LPRvv4F5tDAU2PS8Pk
	 mYGRHaxFgakablsbgE0Db+uYguGO+ybnScIhKlB0=
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 SP1-EX07CAS02.ds.corp.yahoo.com ([216.252.116.167]) with mapi; Wed, 11 May
 2011 14:29:21 -0700
From: Suresh Srinivas <sureshms@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 11 May 2011 14:29:19 -0700
Subject: Re: Apache Hadoop Hackathons: 5/11 and 5/18 in SF and Palo Alto
Thread-Topic: Apache Hadoop Hackathons: 5/11 and 5/18 in SF and Palo Alto
Thread-Index: AcwPjIxsJoki/2MDQ0iivHxen22nTAACnhCiACLd8jQ=
Message-ID: <C9F04DBF.1A2BA%sureshms@yahoo-inc.com>
In-Reply-To: <C9EF63C2.355DB%ddas@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-Entourage/13.8.0.101117
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

I am getting the final metrics patch HDFS-1117 in. Also here is the list
Jitendra had compiled about some of the HDFS patches flagged as not in
trunk:
HDFS-1020  - Yes, incorporated into 1201
HDFS-1022  - NA for trunk
HDFS-1136  - Yes, Fixed in hdfs-1178
HDFS-1153  - NO
HDFS-1158  - YES/NO, This issue is covered by HDFS-1161. Filed separate
issues for the suggested enhancements. by Eli.
HDFS-1313  - YES, It was backport of HDFS-481.
HDFS-495   - NA for trunk
HDFS-6599  - Is it HADOOP-6599? HADOOP-6599 is committed in common trunk.
HDFS-740   - This redirects to HADOOP-6344, which is committed to trunk.



On 5/10/11 9:50 PM, "Devaraj Das" <ddas@yahoo-inc.com> wrote:

> I will get to MR-2178 soon. I & Chris had done a draft of the port to tru=
nk a
> few months ago. Hopefully, it is easy to continue from there and merge ..=
.
>=20
>=20
> On 5/10/11 8:34 PM, "Tom White" <tom@cloudera.com> wrote:
>=20
> On Tue, May 10, 2011 at 5:57 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:
>> Awesome.
>>=20
>> I'm sorry to miss this (I'm neck deep in MR-279), but a heads up on forw=
ard
>> porting from my end:
>> # Luke and Suresh are sick of me bugging them and just rolled up their
>> sleeves to finish up all of the metrics2 work! Thanks guys! I've
>> reviewed/committed Luke's https://issues.apache.org/jira/browse/HADOOP-6=
919
>> and Suresh finished up the rest.
>> # I'll work with Devaraj/Chris to finish up the TT security hole
>> (https://issues.apache.org/jira/browse/MAPREDUCE-2178).
>=20
> Thanks Arun - that's great!
>=20
> Tom
>=20
>>=20
>> thanks,
>> Arun
>>=20
>> On May 9, 2011, at 2:36 PM, Jeff Hammerbacher wrote:
>>=20
>>> Hey,
>>>=20
>>> We've got 23 folks signed up for the Hackathon this Wednesday from
>>> organizations like Cloudera, Facebook, Twitter, AOL, Trend Micro,
>>> StumbleUpon, and Ngmoco. We'll also have the VP of the HBase PMC, Micha=
el
>>> Stack, and the VP of the Hive PMC, John Sichi. If you've been waiting t=
o
>>> get
>>> involved in Apache Hadoop, development, now is the time!
>>>=20
>>> We've got room for 10 - 15 more. If you're in San Francisco or Palo Alt=
o
>>> and
>>> want to help out with Apache Hadoop development, sign up at
>>> http://hadoophackathon.eventbrite.com.
>>>=20
>>> Regards,
>>> Jeff
>>>=20
>>> On Fri, May 6, 2011 at 12:03 PM, Jeff Hammerbacher
>>> <hammer@cloudera.com>wrote:
>>>=20
>>>> Hey,
>>>>=20
>>>> The discussion this week about the 0.20.203.0 release has done a great
>>>> job
>>>> of highlighting some issues in our development process; it's also done=
 a
>>>> great job of lifting our mailing list activity metrics. After reading
>>>> through the various threads, it's clear that everyone agrees on two
>>>> things:
>>>>=20
>>>> 1) We'd like to get the work done in the 0.20.x branches into trunk
>>>> 2) We'd like to start releasing off of trunk again
>>>>=20
>>>> To that end, a few folks from Yahoo!, Cloudera, StumbleUpon, and Faceb=
ook
>>>> would like to put together a series of Hackathons to 1) burn down the =
13
>>>> (only 13!) remaining blockers for the 0.22 release and 2) forward-port
>>>> the
>>>> work done in the 0.20.x branches into trunk.
>>>>=20
>>>> Cloudera will be hosting Hackathons from 10 am to 6 pm in both our Pal=
o
>>>> Alto and San Francisco offices next Wednesday, 5/11, and the following
>>>> Wednesday, 5/18, to ensure both of these tasks get completed in short
>>>> order.
>>>> PMC members Todd Lipcon and Tom White will lead the SF group and PMC
>>>> members
>>>> Eli Collins and Patrick Hunt will lead the Palo Alto group.
>>>>=20
>>>> Whether you're a long-time contributor or don't have a patch to your
>>>> name,
>>>> now is a great time to get involved in Apache Hadoop development. Forw=
ard
>>>> porting patches is a lot easier than writing them from scratch, and
>>>> you'll
>>>> have mentors present to help guide you through the patch testing and
>>>> submission process.
>>>>=20
>>>> To sign up for the 5/11 Hackathon in either SF or PA, head over to
>>>> http://hadoophackathon.eventbrite.com.
>>>>=20
>>>> As a reminder, the SF HUG will be held on 5/11 at 6 pm in the Cloudera=
 SF
>>>> offices: http://www.meetup.com/hadoopsf/events/17354462/. If you can't
>>>> make it on 5/11, we'll send out a link next week for the 5/18 event.
>>>>=20
>>>> Looking forward to getting the release train moving again!
>>>>=20
>>>> Regards,
>>>> Jeff
>>>>=20
>>>> p.s. if you'd like to participate remotely, email me directly and I'll
>>>> see
>>>> about how we can teleconference you into the event.
>>>>=20
>>>>=20
>>>>=20
>>=20
>>=20
>=20


From general-return-3475-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 21:43:12 2011
Return-Path: <general-return-3475-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CD1264E69
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 21:43:12 +0000 (UTC)
Received: (qmail 70409 invoked by uid 500); 11 May 2011 21:43:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70345 invoked by uid 500); 11 May 2011 21:43:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70331 invoked by uid 99); 11 May 2011 21:43:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:43:11 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mbhandarkar@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:43:05 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:In-Reply-To:Accept-Language:
   Content-Language:X-MS-Has-Attach:X-MS-TNEF-Correlator:
   x-originating-ip:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version;
  b=Hgg0LEoZU57zaz9vsEx8B0Xz5p3qXUXkwdsQgnR/z6K2Gpsk4CtEcCVg
   +8ABl87vcCXPi6dTUxJoLEm48ryPSrVR7vbqxehtyT88mJkqgWAUcyPWW
   XmFui7+MiibnS0n;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=mbhandarkar@linkedin.com; q=dns/txt;
  s=proddkim; t=1305150185; x=1336686185;
  h=from:to:subject:date:message-id:in-reply-to:content-id:
   content-transfer-encoding:mime-version;
  bh=mhZ3y4JljQ4/xqUuEWYeThJRpreCea7Qjtd/Timorww=;
  b=qNHWqHtb7OTxnut8YpkIkgsH5/p/aSBsM5f+L5DbVuK1F8fu3HdnuVs3
   jbv7TC5KQnCRaa+Y/+V3A2MCy9JJjEhtrFyZcOpUsQDVqzOdb4kFhiOuE
   3t/b6k6AI7tvH5A;
X-IronPort-AV: E=Sophos;i="4.64,354,1301900400"; 
   d="scan'208";a="23092385"
Received: from ESV4-EXC02.linkedin.biz ([fe80::4d74:48bd:e0bd:13ee]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi id 14.01.0251.000; Wed,
 11 May 2011 14:46:18 -0700
From: Milind Bhandarkar <mbhandarkar@linkedin.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: Defining Hadoop Compatibility -revisiting-
Thread-Topic: Defining Hadoop Compatibility -revisiting-
Thread-Index: AQHMDv3EIB+Y40jJf0ahsOjyQf2X+pSImxGA//+PqQA=
Date: Wed, 11 May 2011 21:46:18 +0000
Message-ID: <C9F04FDB.8C82%mbhandarkar@linkedin.com>
In-Reply-To: <FDD1BE64-981D-4FA1-87BC-0FD4A880AB7A@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <C346918A85E64641A06126E3306F8B88@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

I think it's time to separate out functional tests as a "Hadoop
Compatibility Kit (HCK)", similar to the Sun TCK for Java, but under ASL
2.0. Then "certification" would mean "Passes 100% of the HCK testsuite."

- milind
--=20
Milind Bhandarkar
mbhandarkar@linkedin.com






On 5/11/11 2:24 PM, "Eric Baldeschwieler" <eric14@yahoo-inc.com> wrote:

>This is a really interesting topic!  I completely agree that we need to
>get ahead of this.
>
>I would be really interested in learning of any experience other apache
>projects, such as apache or tomcat have with these issues.
>
>---
>E14 - typing on glass
>
>On May 10, 2011, at 6:31 AM, "Steve Loughran" <stevel@apache.org> wrote:
>
>>=20
>> Back in Jan 2011, I started a discussion about how to define Apache
>> Hadoop Compatibility:
>>=20
>>http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C4D
>>46B6AD.2020802@apache.org%3E
>>=20
>> I am now reading EMC HD "Enterprise Ready" Apache Hadoop datasheet
>>=20
>>=20
>>http://www.greenplum.com/sites/default/files/EMC_Greenplum_HD_DS_Final_1.
>>pdf
>>=20
>> It claims that their implementations are 100% compatible, even though
>> the Enterprise edition uses a C filesystem. It also claims that both
>> their software releases contain "Certified Stacks", without defining
>> what Certified means, or who does the certification -only that it is an
>> improvement.
>>=20
>>=20
>> I think we should revisit this issue before people with their own
>> agendas define what compatibility with Apache Hadoop is for us
>>=20
>>=20
>> Licensing
>> -Use of the Hadoop codebase must follow the Apache License
>> http://www.apache.org/licenses/LICENSE-2.0
>> -plug in components that are dynamically linked to (Filesystems and
>> schedulers) don't appear to be derivative works on my reading of this,
>>=20
>> Naming
>>  -this is something for branding@apache, they will have their opinions.
>> The key one is that the name "Apache Hadoop" must get used, and it's
>> important to make clear it is a derivative work.
>>  -I don't think you can claim to have a Distribution/Fork/Version of
>> Apache Hadoop if you swap out big chunks of it for alternate
>> filesystems, MR engines, etc. Some description of this is needed
>> "Supports the Apache Hadoop MapReduce engine on top of Filesystem XYZ"
>>=20
>> Compatibility
>>  -the definition of the Hadoop interfaces and classes is the Apache
>> Source tree,
>>  -the definition of semantics of the Hadoop interfaces and classes is
>> the Apache Source tree, including the test classes.
>>  -the verification that the actual semantics of an Apache Hadoop
>> release is compatible with the expected semantics is that current and
>> future tests pass
>>  -bug reports can highlight incompatibility with expectations of
>> community users, and once incorporated into tests form part of the
>> compatibility testing
>>  -vendors can claim and even certify their derivative works as
>> compatible with other versions of their derivative works, but cannot
>> claim compatibility with Apache Hadoop unless their code passes the
>> tests and is consistent with the bug reports marked as ("by design").
>> Perhaps we should have tests that verify each of these "by design"
>> bugreps to make them more formal.
>>=20
>> Certification
>>  -I have no idea what this means in EMC's case, they just say
>>"Certified"
>>  -As we don't do any certification ourselves, it would seem impossible
>> for us to certify that any derivative work is compatible.
>>  -It may be best to state that nobody can certify their derivative as
>> "compatible with Apache Hadoop" unless it passes all current test suites
>>  -And require that anyone who declares compatibility define what they
>> mean by this
>>=20
>> This is a good argument for getting more functional tests out there
>> -whoever has more functional tests needs to get them into a test module
>> that can be used to test real deployments.
>>=20


From general-return-3476-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 21:50:26 2011
Return-Path: <general-return-3476-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 75FC64577
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 21:50:26 +0000 (UTC)
Received: (qmail 83988 invoked by uid 500); 11 May 2011 21:50:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83915 invoked by uid 500); 11 May 2011 21:50:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83907 invoked by uid 99); 11 May 2011 21:50:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:50:24 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [216.86.168.182] (HELO mxout-07.mxes.net) (216.86.168.182)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:50:19 +0000
Received: from [192.168.254.196] (unknown [69.170.160.163])
	(using TLSv1 with cipher AES128-SHA (128/128 bits))
	(No client certificate requested)
	by smtp.mxes.net (Postfix) with ESMTPSA id C071B22E257
	for <general@hadoop.apache.org>; Wed, 11 May 2011 17:49:58 -0400 (EDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: OT: anyone else going to berlin buzzwords?
From: Chris K Wensel <chris@wensel.net>
In-Reply-To: <BANLkTincfMk1K0rG7qv69PFpLwi646zKPw@mail.gmail.com>
Date: Wed, 11 May 2011 14:49:57 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <B3D30587-0CFF-43B0-AD30-EAE7839F01EA@wensel.net>
References: <4DCA6997.2040200@apache.org> <C9EFF312.35694%ddas@yahoo-inc.com> <BANLkTincfMk1K0rG7qv69PFpLwi646zKPw@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)

i'll be there and speaking. but traveling on thursday, so will miss the =
hackathon unless its moved up.

ckw

On May 11, 2011, at 2:29 PM, Dan wrote:

> I'll be heading there too, probably there till Thursday/Friday so
> could attend a hackathon after the conference.
>=20
> It would be great if it wasn't the same time as the HBase hackathon
> which is planned for the 8th, or could be combined with it if it's
> better on the Wednesday? I'm not sure who's organising that...
>=20
> On Wed, May 11, 2011 at 4:02 PM, Devaraj Das <ddas@yahoo-inc.com> =
wrote:
>> I can do a tour of the MR-279 codebase but I am currently flying out =
on the afternoon of Wednesday. Let me see if I can change the flight =
bookings.
>>=20
>>=20
>> On 5/11/11 3:48 AM, "Steve Loughran" <stevel@apache.org> wrote:
>>=20
>> On 11/05/2011 03:12, Devaraj Das wrote:
>>> I'll be there .. Looking forward to meeting up with Ian and the =
other Hadoop'ers...
>>>=20
>>=20
>>=20
>> OK, there's now a tentative plan for a Hadoop hackathon, i'd like a
>> volunteer to do a tour of the MR-279 codebase, and we have a =
volunteer
>> to talk about gridmix.
>>=20
>> http://berlinbuzzwords.de/wiki/hadoop-hackathon
>>=20
>> Being a hackathon and all, some coding theme would be good. Testing =
MR-279?
>>=20
>>=20

--
Chris K Wensel
chris@concurrentinc.com
http://www.concurrentinc.com

-- Concurrent, Inc. offers mentoring, support for Cascading


From general-return-3477-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 22:11:02 2011
Return-Path: <general-return-3477-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C73FC4049
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 22:11:02 +0000 (UTC)
Received: (qmail 18179 invoked by uid 500); 11 May 2011 22:11:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18132 invoked by uid 500); 11 May 2011 22:11:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18124 invoked by uid 99); 11 May 2011 22:11:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 22:11:01 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of owen.omalley@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 22:10:52 +0000
Received: by pwi16 with SMTP id 16so641153pwi.35
        for <general@hadoop.apache.org>; Wed, 11 May 2011 15:10:31 -0700 (PDT)
Received: by 10.142.149.20 with SMTP id w20mr5608932wfd.137.1305151831202;
        Wed, 11 May 2011 15:10:31 -0700 (PDT)
Received: from battlerock-lm.corp.yahoo.com (nat-dip4.cfw-a-gci.corp.yahoo.com [209.131.62.113])
        by mx.google.com with ESMTPS id p40sm600972wfc.19.2011.05.11.15.10.28
        (version=SSLv3 cipher=OTHER);
        Wed, 11 May 2011 15:10:30 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: [VOTE] Release candidate 0.20.203.0-rc0
From: Owen O'Malley <omalley@apache.org>
In-Reply-To: <BANLkTikQAqsxc4wCU8LRY93yHGEjQLZLgw@mail.gmail.com>
Date: Wed, 11 May 2011 15:10:27 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <B5429A7F-FA77-46C1-8CBC-277F868354B3@apache.org>
References: <12853C59-08F1-4F98-ACF3-800F06BC7A5F@apache.org> <BANLkTinZ=xb6kJ5PTeLN5KKD9b-cwaM0OQ@mail.gmail.com> <BANLkTi=1sQoL68k_2UmA1_Oo2CyXsM3fAw@mail.gmail.com> <DBCE9300-0D9D-4A9A-99B4-878139A028EE@yahoo-inc.com> <61C1E656-0649-4097-8709-FDC596ED57E9@yahoo-inc.com> <F7921362-CC1B-435C-890A-E1BD181E03C0@yahoo-inc.com> <8D047C89-6430-4FDE-A9C4-CFEE79644AC6@Holsman.NET> <BANLkTi=JDBneLBHsCh1T-PF8C7wNbra0kw@mail.gmail.com> <BANLkTim68H=8ngBfzMsVrQOb9PmY7FviLQ@mail.gmail.com> <BANLkTi=KoCF9MB717JSFsZG3fgy2U+qF=w@mail.gmail.com> <BANLkTikQAqsxc4wCU8LRY93yHGEjQLZLgw@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org


On May 11, 2011, at 11:40 AM, Konstantin Shvachko wrote:

> umask is not a big concern. I reset it to standard 0022.

It still should be fixed.

> Still there were 8 other test failures: 7 in mapred, and 1 hdfsproxy.
> Stable release should pass unit tests.

All unit tests pass for me. Others have also run the unit tests and had =
them pass.
If you can figure out what is going wrong, that would be great.

> Lets make 0.20.203.1 stable.

+1

-- Owen=

From general-return-3478-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 22:12:57 2011
Return-Path: <general-return-3478-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 47E8540CC
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 22:12:57 +0000 (UTC)
Received: (qmail 21595 invoked by uid 500); 11 May 2011 22:12:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21548 invoked by uid 500); 11 May 2011 22:12:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21540 invoked by uid 99); 11 May 2011 22:12:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 22:12:55 +0000
X-ASF-Spam-Status: No, hits=2.9 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 22:12:51 +0000
Received: from oceanfarearth-lm.corp.yahoo.com (oceanfarearth-lm.corp.yahoo.com [10.72.113.156])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p4BMCDb8061306;
	Wed, 11 May 2011 15:12:13 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1305151933;
	bh=ECtTnxhv05l7zIEGO/7paVEX43AM4QLk0YJ8P2Zgiwc=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:Mime-Version:Subject:
	 Date:References;
	b=RHbHvjH5lqdftiQ1ZqlK9/gBStADNJfimECpF9nFxylFBsp09fPF05Vn6wpC6fMTW
	 A7PiGZTRfttX7Jvh2jR7YdRIR38haFWylxAXo63YG1XPjRR0sC9c54cAOcKlERgv8I
	 wZGwkuMZH85QEr6tuSmVJ9NXhzbsaMuEBtuZvW0s=
Message-Id: <1057A159-BE13-47D2-B746-1A66F0DBC982@yahoo-inc.com>
From: Sanjay Radia <sradia@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>,
        "Aaron T. Myers" <atm@cloudera.com>
In-Reply-To: <BANLkTinoW_MBSi9jSKOFK8i2=BMnNzU66w@mail.gmail.com>
Content-Type: multipart/alternative; boundary=Apple-Mail-2-427218634
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Release compatibility was Re: [VOTE] Release candidate 0.20.203.0-rc1
Date: Wed, 11 May 2011 15:12:12 -0700
References: <BANLkTi=38uUOYGP4zvPfoqPHC3KEKrKnXA@mail.gmail.com> <C9EF6BA2.355E9%ddas@yahoo-inc.com> <BANLkTinoW_MBSi9jSKOFK8i2=BMnNzU66w@mail.gmail.com>
X-Mailer: Apple Mail (2.936)

--Apple-Mail-2-427218634
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


On May 10, 2011, at 10:49 PM, Aaron T. Myers wrote:

> On Tue, May 10, 2011 at 10:24 PM, Devaraj Das <ddas@yahoo-inc.com>  
> wrote:
>
> .....
>
> By far the most significant incompatibility that I've seen from a user
> perspective is that setting hadoop.job.ugi no longer has any effect.
> Granted, this interface wasn't used by a large percentage of users,  
> but
> those that were using it have no other alternative that is as  
> flexible as
> this was. There was discussion about this incompatibility last  
> September on
> the mailing lists[1]. The conclusion there was that supporting  
> backward
> compatibility for this interface was too difficult, semantically and
> technically, to warrant support. This incompatibility is present  
> whether or
> not Kerberos support is enabled on the cluster.


That I why we added the interface audience and stability classification.
 From the very beginning (see my early proposals on HADOOP-5073) the  
security UGI
interfaces were targeted to be marked as limited private.
We completed the interface annotation in release 21, so some users did  
not realize that they should not
be using such interfaces.

sanjay


--Apple-Mail-2-427218634--

From general-return-3479-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 22:36:51 2011
Return-Path: <general-return-3479-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5D7D54351
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 22:36:51 +0000 (UTC)
Received: (qmail 60357 invoked by uid 500); 11 May 2011 22:36:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60302 invoked by uid 500); 11 May 2011 22:36:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60294 invoked by uid 99); 11 May 2011 22:36:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 22:36:49 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 22:36:43 +0000
Received: by qwj9 with SMTP id 9so687809qwj.35
        for <general@hadoop.apache.org>; Wed, 11 May 2011 15:36:22 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=kgexXDvs18zkzUcQ4b9wYY+QA+hMsO0f+6QfkqkG+gw=;
        b=ImGXB4ilg43DzDaCMMoNjSOQ4zwbb0fQj+YZ6NEzzsqKddZ7IHKLBw8FfFHi7jR22b
         0yk6lFhjzhSYTzpL3PJjUT0uMuX6Pit1Jnc9mhjPeUmFJyBh42eCD/a506bndWXGQafO
         12pIc2R8bZ+P6ckQbpyDDaihEfSnUjIS6L8iU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=j0+rv+GXAv37NPxLZnonvBhS+2iP40qaUDzLGbb/l9ZrQL/hU+rqRTe9JEBBDlEg1T
         L0fCjpswJC6ivWZ+gCgfTQHKF2Bk/CgJcyU3/+r0UOup7HTxtTkIW/J6OUa6Eh+BYpwp
         KacvAz1fbHLw4U23sbqPThdIn8s3zjWsvS+Bs=
Received: by 10.224.2.136 with SMTP id 8mr714573qaj.394.1305153382747;
        Wed, 11 May 2011 15:36:22 -0700 (PDT)
Received: from [10.172.1.95] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id m7sm366097qcg.5.2011.05.11.15.36.20
        (version=TLSv1/SSLv3 cipher=OTHER);
        Wed, 11 May 2011 15:36:21 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: Cleaning up documentation from website
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <FABE9911-352F-4AE2-96B1-27E66FB0369A@apache.org>
Date: Thu, 12 May 2011 08:36:16 +1000
Content-Transfer-Encoding: quoted-printable
Message-Id: <DA3EE4F3-E4E7-4857-8428-EBDC26ED25A5@Holsman.NET>
References: <FABE9911-352F-4AE2-96B1-27E66FB0369A@apache.org>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)

I think we need a 0.18 doc's as some people are still running this.
other than that (and what is 'current'.. it should point to the stable =
release imho) I'm good with it.
On May 12, 2011, at 6:20 AM, Owen O'Malley wrote:

> We haven't cleaned up the version documentation in a long time:
>=20
> lrwxrwxr-x  1 omalley   hadoop  11 May 11 20:14 current -> r0.20.203.0
> drwxrwsr-x  7 cutting   hadoop  29 Nov 24  2008 r0.14.4
> drwxrwsr-x  6 cutting   hadoop  22 Jan 23  2008 r0.15.2
> drwxrwsr-x  6 nigel     hadoop  22 Jan 23  2008 r0.15.3
> drwxrwsr-x  6 nigel     hadoop  28 Feb  8  2008 r0.16.0
> drwxrwsr-x  6 nigel     hadoop  36 Mar 14  2008 r0.16.1
> drwxrwsr-x  6 nigel     hadoop  36 Apr  3  2008 r0.16.2
> drwxrwsr-x  6 nigel     hadoop  36 Apr 17  2008 r0.16.3
> drwxrwsr-x  6 nigel     hadoop  36 May  9  2008 r0.16.4
> drwxrwsr-x  6 nigel     hadoop  42 May 21  2008 r0.17.0
> drwxrwsr-x  5 omalley   hadoop  42 Aug 20  2008 r0.17.1
> drwxrwsr-x  7 omalley   hadoop  43 Dec 11  2008 r0.17.2
> drwxrwsr-x  7 nigel     hadoop  51 Aug 22  2008 r0.18.0
> drwxrwsr-x  7 nigel     hadoop  51 Sep 18  2008 r0.18.1
> drwxrwsr-x  8 nigel     hadoop  52 Dec 11  2008 r0.18.2
> drwxrwsr-x  8 nigel     hadoop  52 Jan 30  2009 r0.18.3
> drwxrwsr-x  8 nigel     hadoop  58 Dec 11  2008 r0.19.0
> drwxrwsr-x  8 nigel     hadoop  58 Feb 24  2009 r0.19.1
> drwxrwsr-x  7 tomwhite  hadoop  57 Jun 30  2009 r0.19.2
> drwxrwsr-x  7 nigel     hadoop  63 Apr  9  2009 r0.20.0
> drwxrwsr-x  7 omalley   hadoop  64 Mar  1  2010 r0.20.1
> drwxrwxr-x  7 cdouglas  hadoop  63 Aug 24  2010 r0.20.2
> drwxrwxr-x  7 omalley   hadoop  63 May 11 20:11 r0.20.203.0
> drwxrwxr-x  7 tomwhite  hadoop  31 Aug 17  2010 r0.21.0
>=20
> Other than the history of who made each of the releases, there isn't a =
lot of value there. I propose that we trim this down to:
>=20
> r0.20.2
> r0.20.203.0
> r0.21.0
>=20
> Any concerns? If not, I'll clean up next week.
>=20
> -- Owen


From general-return-3480-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 22:42:35 2011
Return-Path: <general-return-3480-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8668744B0
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 22:42:35 +0000 (UTC)
Received: (qmail 66795 invoked by uid 500); 11 May 2011 22:42:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66737 invoked by uid 500); 11 May 2011 22:42:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66729 invoked by uid 99); 11 May 2011 22:42:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 22:42:33 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.220.176] (HELO mail-vx0-f176.google.com) (209.85.220.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 22:42:28 +0000
Received: by vxa37 with SMTP id 37so1153335vxa.35
        for <general@hadoop.apache.org>; Wed, 11 May 2011 15:42:07 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=1Ped8hsQjBjz2NQYzo8nUOdPtAOCdmlzvmNVEKBTcUA=;
        b=l18ZTDpYxjXAY2MBDve/XwcoXVApUlB9EPvy+5+7Klu7gcQw7x6H5vgoebFwhl5JSL
         wuLGujGeuhaBOGf4Da18NRkaVaGewyN/v292IS/5qU/p8KqC3R15TzHbMy/2paIQXd2W
         1iDgX61Q6ycBMWWwQpa/q02ieoNCvwNFT+HbQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=fApK+51WMPGXbW7Q5fl79VDDAElVN7cyLrpjjwCuYeSdkcMV4wgF7RjnKpCxr0FJrD
         7zkW2iXe4bJ+tHNIO/Tr0LXaFzBdzAfq7mb6oyb5pxiekp10/rInexixS6zgax1/Gggt
         ZOoXST9CiQuZvY3fe9let4uxRnSycFoc9U0SI=
Received: by 10.52.179.131 with SMTP id dg3mr985944vdc.9.1305153727788;
        Wed, 11 May 2011 15:42:07 -0700 (PDT)
Received: from [10.172.1.95] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id 15sm96891vdh.27.2011.05.11.15.42.05
        (version=TLSv1/SSLv3 cipher=OTHER);
        Wed, 11 May 2011 15:42:06 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: Defining Hadoop Compatibility -revisiting-
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <FDD1BE64-981D-4FA1-87BC-0FD4A880AB7A@yahoo-inc.com>
Date: Thu, 12 May 2011 08:42:01 +1000
Content-Transfer-Encoding: quoted-printable
Message-Id: <6407C0E3-934F-4C9D-9103-C738F6C1AED7@Holsman.NET>
References: <4DC91392.2010308@apache.org> <FDD1BE64-981D-4FA1-87BC-0FD4A880AB7A@yahoo-inc.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)

For apache (httpd I'm assuming you mean). we define compatibility as =
adherence to the set of RFC's that define the HTTP protocol.

I'm no expert in this (Roy is though), but we could attempt to do =
something similar when it comes to HDFS/Map-Reduce protocols. I'm not =
sure what benefit there would be to going to a RFC, as opposed to =
documenting the API on our site.


On May 12, 2011, at 7:24 AM, Eric Baldeschwieler wrote:

> This is a really interesting topic!  I completely agree that we need =
to get ahead of this.=20
>=20
> I would be really interested in learning of any experience other =
apache projects, such as apache or tomcat have with these issues.=20
>=20
> ---
> E14 - typing on glass
>=20
> On May 10, 2011, at 6:31 AM, "Steve Loughran" <stevel@apache.org> =
wrote:
>=20
>>=20
>> Back in Jan 2011, I started a discussion about how to define Apache=20=

>> Hadoop Compatibility:
>> =
http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C4D4=
6B6AD.2020802@apache.org%3E
>>=20
>> I am now reading EMC HD "Enterprise Ready" Apache Hadoop datasheet
>>=20
>> =
http://www.greenplum.com/sites/default/files/EMC_Greenplum_HD_DS_Final_1.p=
df
>>=20
>> It claims that their implementations are 100% compatible, even though=20=

>> the Enterprise edition uses a C filesystem. It also claims that both=20=

>> their software releases contain "Certified Stacks", without defining=20=

>> what Certified means, or who does the certification -only that it is =
an=20
>> improvement.
>>=20
>>=20
>> I think we should revisit this issue before people with their own=20
>> agendas define what compatibility with Apache Hadoop is for us
>>=20
>>=20
>> Licensing
>> -Use of the Hadoop codebase must follow the Apache License
>> http://www.apache.org/licenses/LICENSE-2.0
>> -plug in components that are dynamically linked to (Filesystems and=20=

>> schedulers) don't appear to be derivative works on my reading of =
this,
>>=20
>> Naming
>> -this is something for branding@apache, they will have their =
opinions.=20
>> The key one is that the name "Apache Hadoop" must get used, and it's=20=

>> important to make clear it is a derivative work.
>> -I don't think you can claim to have a Distribution/Fork/Version of=20=

>> Apache Hadoop if you swap out big chunks of it for alternate=20
>> filesystems, MR engines, etc. Some description of this is needed
>> "Supports the Apache Hadoop MapReduce engine on top of Filesystem =
XYZ"
>>=20
>> Compatibility
>> -the definition of the Hadoop interfaces and classes is the Apache=20
>> Source tree,
>> -the definition of semantics of the Hadoop interfaces and classes is=20=

>> the Apache Source tree, including the test classes.
>> -the verification that the actual semantics of an Apache Hadoop=20
>> release is compatible with the expected semantics is that current and=20=

>> future tests pass
>> -bug reports can highlight incompatibility with expectations of=20
>> community users, and once incorporated into tests form part of the=20
>> compatibility testing
>> -vendors can claim and even certify their derivative works as=20
>> compatible with other versions of their derivative works, but cannot=20=

>> claim compatibility with Apache Hadoop unless their code passes the=20=

>> tests and is consistent with the bug reports marked as ("by design").=20=

>> Perhaps we should have tests that verify each of these "by design"=20
>> bugreps to make them more formal.
>>=20
>> Certification
>> -I have no idea what this means in EMC's case, they just say =
"Certified"
>> -As we don't do any certification ourselves, it would seem impossible=20=

>> for us to certify that any derivative work is compatible.
>> -It may be best to state that nobody can certify their derivative as=20=

>> "compatible with Apache Hadoop" unless it passes all current test =
suites
>> -And require that anyone who declares compatibility define what they=20=

>> mean by this
>>=20
>> This is a good argument for getting more functional tests out there=20=

>> -whoever has more functional tests needs to get them into a test =
module=20
>> that can be used to test real deployments.
>>=20


From general-return-3481-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 22:57:24 2011
Return-Path: <general-return-3481-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7235B4C17
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 22:57:24 +0000 (UTC)
Received: (qmail 78212 invoked by uid 500); 11 May 2011 22:57:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78169 invoked by uid 500); 11 May 2011 22:57:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78161 invoked by uid 99); 11 May 2011 22:57:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 22:57:22 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of apache@jacobrideout.net designates 74.50.50.180 as permitted sender)
Received: from [74.50.50.180] (HELO jacobrideout.net) (74.50.50.180)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 22:57:16 +0000
Received: from mail-bw0-f48.google.com (mail-bw0-f48.google.com [209.85.214.48])
	(using TLSv1 with cipher RC4-SHA (128/128 bits))
	(No client certificate requested)
	by jacobrideout.net (Postfix) with ESMTPSA id BED35400B5
	for <general@hadoop.apache.org>; Wed, 11 May 2011 16:56:54 -0600 (MDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=jacobrideout.net;
	s=2009a; t=1305154615;
	bh=/S0qYIAxAxzGHoQhnCeHPODXPdTa0Blq+cn1KMZv7Ws=;
	h=MIME-Version:In-Reply-To:References:From:Date:Message-ID:Subject:
	 To:Content-Type:Content-Transfer-Encoding;
	b=Yz+iQllo78swHe8mLM0qRR4F7rFgZOs/8iAK/2ZuIAxE7z8y2g7RUIx6Z/Sdbu4YA
	 mN/+3r2KOqrMEFitaykM3BH4rEkzP2vIEZwrH2C1JCNoSiOTEuCxyBWZMdezytRwD1
	 RQhLN+jGQwTIvWJusYIpuGMR8QzilaB55APCy7O8=
Received: by bwz8 with SMTP id 8so1386608bwz.35
        for <general@hadoop.apache.org>; Wed, 11 May 2011 15:56:53 -0700 (PDT)
Received: by 10.204.38.88 with SMTP id a24mr525307bke.130.1305154613218; Wed,
 11 May 2011 15:56:53 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.84.28 with HTTP; Wed, 11 May 2011 15:56:33 -0700 (PDT)
In-Reply-To: <6407C0E3-934F-4C9D-9103-C738F6C1AED7@Holsman.NET>
References: <4DC91392.2010308@apache.org> <FDD1BE64-981D-4FA1-87BC-0FD4A880AB7A@yahoo-inc.com>
 <6407C0E3-934F-4C9D-9103-C738F6C1AED7@Holsman.NET>
From: Jacob R Rideout <apache@jacobrideout.net>
Date: Wed, 11 May 2011 16:56:33 -0600
Message-ID: <BANLkTik9NeKPEuoPO6ijyzbyidweHDFKBg@mail.gmail.com>
Subject: Re: Defining Hadoop Compatibility -revisiting-
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

What about defining compatibility as fully implementing all the
public-stable annotated interfaces for a particular release?

Jacob Rideout

On Wed, May 11, 2011 at 4:42 PM, Ian Holsman <hadoop@holsman.net> wrote:
> For apache (httpd I'm assuming you mean). we define compatibility as adhe=
rence to the set of RFC's that define the HTTP protocol.
>
> I'm no expert in this (Roy is though), but we could attempt to do somethi=
ng similar when it comes to HDFS/Map-Reduce protocols. I'm not sure what be=
nefit there would be to going to a RFC, as opposed to documenting the API o=
n our site.
>
>
> On May 12, 2011, at 7:24 AM, Eric Baldeschwieler wrote:
>
>> This is a really interesting topic! =C2=A0I completely agree that we nee=
d to get ahead of this.
>>
>> I would be really interested in learning of any experience other apache =
projects, such as apache or tomcat have with these issues.
>>
>> ---
>> E14 - typing on glass
>>
>> On May 10, 2011, at 6:31 AM, "Steve Loughran" <stevel@apache.org> wrote:
>>
>>>
>>> Back in Jan 2011, I started a discussion about how to define Apache
>>> Hadoop Compatibility:
>>> http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C=
4D46B6AD.2020802@apache.org%3E
>>>
>>> I am now reading EMC HD "Enterprise Ready" Apache Hadoop datasheet
>>>
>>> http://www.greenplum.com/sites/default/files/EMC_Greenplum_HD_DS_Final_=
1.pdf
>>>
>>> It claims that their implementations are 100% compatible, even though
>>> the Enterprise edition uses a C filesystem. It also claims that both
>>> their software releases contain "Certified Stacks", without defining
>>> what Certified means, or who does the certification -only that it is an
>>> improvement.
>>>
>>>
>>> I think we should revisit this issue before people with their own
>>> agendas define what compatibility with Apache Hadoop is for us
>>>
>>>
>>> Licensing
>>> -Use of the Hadoop codebase must follow the Apache License
>>> http://www.apache.org/licenses/LICENSE-2.0
>>> -plug in components that are dynamically linked to (Filesystems and
>>> schedulers) don't appear to be derivative works on my reading of this,
>>>
>>> Naming
>>> -this is something for branding@apache, they will have their opinions.
>>> The key one is that the name "Apache Hadoop" must get used, and it's
>>> important to make clear it is a derivative work.
>>> -I don't think you can claim to have a Distribution/Fork/Version of
>>> Apache Hadoop if you swap out big chunks of it for alternate
>>> filesystems, MR engines, etc. Some description of this is needed
>>> "Supports the Apache Hadoop MapReduce engine on top of Filesystem XYZ"
>>>
>>> Compatibility
>>> -the definition of the Hadoop interfaces and classes is the Apache
>>> Source tree,
>>> -the definition of semantics of the Hadoop interfaces and classes is
>>> the Apache Source tree, including the test classes.
>>> -the verification that the actual semantics of an Apache Hadoop
>>> release is compatible with the expected semantics is that current and
>>> future tests pass
>>> -bug reports can highlight incompatibility with expectations of
>>> community users, and once incorporated into tests form part of the
>>> compatibility testing
>>> -vendors can claim and even certify their derivative works as
>>> compatible with other versions of their derivative works, but cannot
>>> claim compatibility with Apache Hadoop unless their code passes the
>>> tests and is consistent with the bug reports marked as ("by design").
>>> Perhaps we should have tests that verify each of these "by design"
>>> bugreps to make them more formal.
>>>
>>> Certification
>>> -I have no idea what this means in EMC's case, they just say "Certified=
"
>>> -As we don't do any certification ourselves, it would seem impossible
>>> for us to certify that any derivative work is compatible.
>>> -It may be best to state that nobody can certify their derivative as
>>> "compatible with Apache Hadoop" unless it passes all current test suite=
s
>>> -And require that anyone who declares compatibility define what they
>>> mean by this
>>>
>>> This is a good argument for getting more functional tests out there
>>> -whoever has more functional tests needs to get them into a test module
>>> that can be used to test real deployments.
>>>
>
>

From general-return-3482-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 23:21:07 2011
Return-Path: <general-return-3482-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 750104D2F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 11 May 2011 23:21:07 +0000 (UTC)
Received: (qmail 5294 invoked by uid 500); 11 May 2011 23:21:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5239 invoked by uid 500); 11 May 2011 23:21:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5230 invoked by uid 99); 11 May 2011 23:21:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 23:21:06 +0000
X-ASF-Spam-Status: No, hits=3.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of akimball83@gmail.com designates 74.125.83.176 as permitted sender)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 23:21:01 +0000
Received: by pve37 with SMTP id 37so665277pve.35
        for <general@hadoop.apache.org>; Wed, 11 May 2011 16:20:41 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type;
        bh=aogb/1XFzzPVavCRTIzmVXuYqXAtccDycywXmlkho40=;
        b=h6y3cDhNvCMbE/wP5HUhocS7lt5Jmwd+Ok8OLOr9FWxGQWsjRjjpjxppEY8AneDRgZ
         aKrsnGNwjxplza8XOM2douAQqsQC7sUS65MxH3rFJFXDIlLLSVwkKKC1U+GsJLkyYIXm
         m2Z3dKq5X9XFtqtT7Pl9juoAmPTrnPREDqpgo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=X06vv6EMMjwyAcf2Zo+2NxU6cNMwFOEZ+WcpB7HsiB5bYexf3Haf2861JTep5pNa7u
         Ib+usjOcOdxYpvMCUrA9s7gH52PkJnmLrF8/WahmRmnFWBv8iXv+vJ8mU7v/ASPOMkcr
         PlcUCGTnW1LZatB0yQV8gS+PSKJPHtiHyxMCM=
Received: by 10.68.4.201 with SMTP id m9mr14674557pbm.177.1305156041052; Wed,
 11 May 2011 16:20:41 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.68.62.34 with HTTP; Wed, 11 May 2011 16:20:21 -0700 (PDT)
In-Reply-To: <BANLkTik9NeKPEuoPO6ijyzbyidweHDFKBg@mail.gmail.com>
References: <4DC91392.2010308@apache.org> <FDD1BE64-981D-4FA1-87BC-0FD4A880AB7A@yahoo-inc.com>
 <6407C0E3-934F-4C9D-9103-C738F6C1AED7@Holsman.NET> <BANLkTik9NeKPEuoPO6ijyzbyidweHDFKBg@mail.gmail.com>
From: Aaron Kimball <akimball83@gmail.com>
Date: Wed, 11 May 2011 16:20:21 -0700
Message-ID: <BANLkTikCDLMRhoLLKhTfJEgc7dOCYHonpQ@mail.gmail.com>
Subject: Re: Defining Hadoop Compatibility -revisiting-
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec520e52134115d04a308524e

--bcaec520e52134115d04a308524e
Content-Type: text/plain; charset=ISO-8859-1

What does it mean to "implement" those interfaces? I'm +1 for a TCK-based
definition. In addition to statically implementing a set of interfaces, each
interface also implicitly includes a set of acceptable inputs and predicted
outputs (or ranges of outputs) for those inputs.

- Aaron

On Wed, May 11, 2011 at 3:56 PM, Jacob R Rideout <apache@jacobrideout.net>wrote:

> What about defining compatibility as fully implementing all the
> public-stable annotated interfaces for a particular release?
>
> Jacob Rideout
>
> On Wed, May 11, 2011 at 4:42 PM, Ian Holsman <hadoop@holsman.net> wrote:
> > For apache (httpd I'm assuming you mean). we define compatibility as
> adherence to the set of RFC's that define the HTTP protocol.
> >
> > I'm no expert in this (Roy is though), but we could attempt to do
> something similar when it comes to HDFS/Map-Reduce protocols. I'm not sure
> what benefit there would be to going to a RFC, as opposed to documenting the
> API on our site.
> >
> >
> > On May 12, 2011, at 7:24 AM, Eric Baldeschwieler wrote:
> >
> >> This is a really interesting topic!  I completely agree that we need to
> get ahead of this.
> >>
> >> I would be really interested in learning of any experience other apache
> projects, such as apache or tomcat have with these issues.
> >>
> >> ---
> >> E14 - typing on glass
> >>
> >> On May 10, 2011, at 6:31 AM, "Steve Loughran" <stevel@apache.org>
> wrote:
> >>
> >>>
> >>> Back in Jan 2011, I started a discussion about how to define Apache
> >>> Hadoop Compatibility:
> >>>
> http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C4D46B6AD.2020802@apache.org%3E
> >>>
> >>> I am now reading EMC HD "Enterprise Ready" Apache Hadoop datasheet
> >>>
> >>>
> http://www.greenplum.com/sites/default/files/EMC_Greenplum_HD_DS_Final_1.pdf
> >>>
> >>> It claims that their implementations are 100% compatible, even though
> >>> the Enterprise edition uses a C filesystem. It also claims that both
> >>> their software releases contain "Certified Stacks", without defining
> >>> what Certified means, or who does the certification -only that it is an
> >>> improvement.
> >>>
> >>>
> >>> I think we should revisit this issue before people with their own
> >>> agendas define what compatibility with Apache Hadoop is for us
> >>>
> >>>
> >>> Licensing
> >>> -Use of the Hadoop codebase must follow the Apache License
> >>> http://www.apache.org/licenses/LICENSE-2.0
> >>> -plug in components that are dynamically linked to (Filesystems and
> >>> schedulers) don't appear to be derivative works on my reading of this,
> >>>
> >>> Naming
> >>> -this is something for branding@apache, they will have their opinions.
> >>> The key one is that the name "Apache Hadoop" must get used, and it's
> >>> important to make clear it is a derivative work.
> >>> -I don't think you can claim to have a Distribution/Fork/Version of
> >>> Apache Hadoop if you swap out big chunks of it for alternate
> >>> filesystems, MR engines, etc. Some description of this is needed
> >>> "Supports the Apache Hadoop MapReduce engine on top of Filesystem XYZ"
> >>>
> >>> Compatibility
> >>> -the definition of the Hadoop interfaces and classes is the Apache
> >>> Source tree,
> >>> -the definition of semantics of the Hadoop interfaces and classes is
> >>> the Apache Source tree, including the test classes.
> >>> -the verification that the actual semantics of an Apache Hadoop
> >>> release is compatible with the expected semantics is that current and
> >>> future tests pass
> >>> -bug reports can highlight incompatibility with expectations of
> >>> community users, and once incorporated into tests form part of the
> >>> compatibility testing
> >>> -vendors can claim and even certify their derivative works as
> >>> compatible with other versions of their derivative works, but cannot
> >>> claim compatibility with Apache Hadoop unless their code passes the
> >>> tests and is consistent with the bug reports marked as ("by design").
> >>> Perhaps we should have tests that verify each of these "by design"
> >>> bugreps to make them more formal.
> >>>
> >>> Certification
> >>> -I have no idea what this means in EMC's case, they just say
> "Certified"
> >>> -As we don't do any certification ourselves, it would seem impossible
> >>> for us to certify that any derivative work is compatible.
> >>> -It may be best to state that nobody can certify their derivative as
> >>> "compatible with Apache Hadoop" unless it passes all current test
> suites
> >>> -And require that anyone who declares compatibility define what they
> >>> mean by this
> >>>
> >>> This is a good argument for getting more functional tests out there
> >>> -whoever has more functional tests needs to get them into a test module
> >>> that can be used to test real deployments.
> >>>
> >
> >
>

--bcaec520e52134115d04a308524e--

From general-return-3483-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 01:04:42 2011
Return-Path: <general-return-3483-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 36C14466E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 12 May 2011 01:04:42 +0000 (UTC)
Received: (qmail 16347 invoked by uid 500); 12 May 2011 01:04:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16292 invoked by uid 500); 12 May 2011 01:04:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16284 invoked by uid 99); 12 May 2011 01:04:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 01:04:40 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 01:04:34 +0000
Received: from oceanfarearth-lm.corp.yahoo.com (oceanfarearth-lm.corp.yahoo.com [10.72.113.156])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p4C144Qn009000
	for <general@hadoop.apache.org>; Wed, 11 May 2011 18:04:04 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1305162244;
	bh=++IxRMYS8L7tVRhqs0F1avTHp8d1LGxMIAM7o6BbtRA=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=KZFSxKeYor716dgxtzhXIMdWvkO9NbilYvo8tyM/gDqIqfBBXbdW0FV4w3gTBTNO0
	 EznICApvF4nCkczcxk7XqxXkjUnSBmO4xlNd60+qjX/8W5IsUioUz4gGXfp6WojPWD
	 Av3LufBACQBPhjZyeeCV0v22LBr2A7lkDFxYYyyQ=
Message-Id: <DFDA2177-D0D3-45E5-8C96-AFE86E9A653A@yahoo-inc.com>
From: Sanjay Radia <sradia@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <BANLkTiksqFvXdN68Ki4ZK0y=GLmr7s=MJg@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Apache Hadoop Hackathons: 5/11 and 5/18 in SF and Palo Alto
Date: Wed, 11 May 2011 18:04:04 -0700
References: <BANLkTiksqFvXdN68Ki4ZK0y=GLmr7s=MJg@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

I was looking to forward porting MR-2358.

The original 20.203 does not apply because trunk has lots of  
improvements in MR since 20 due to context objects.
The attached trunk patch does not look correct. I will post a correct  
trunk patch if indeed the bug still exists in trunk.

Further the bug fixed by 2358 only surfaced during the testing of  
viewfs. Since viewfs is not part of 22 I would not worry
about this one for 22.

sanjay



On May 6, 2011, at 12:03 PM, Jeff Hammerbacher wrote:

> Hey,
>
> The discussion this week about the 0.20.203.0 release has done a  
> great job
> of highlighting some issues in our development process; it's also  
> done a
> great job of lifting our mailing list activity metrics. After reading
> through the various threads, it's clear that everyone agrees on two  
> things:
>
> 1) We'd like to get the work done in the 0.20.x branches into trunk
> 2) We'd like to start releasing off of trunk again
>
> To that end, a few folks from Yahoo!, Cloudera, StumbleUpon, and  
> Facebook
> would like to put together a series of Hackathons to 1) burn down  
> the 13
> (only 13!) remaining blockers for the 0.22 release and 2) forward- 
> port the
> work done in the 0.20.x branches into trunk.
>
> Cloudera will be hosting Hackathons from 10 am to 6 pm in both our  
> Palo Alto
> and San Francisco offices next Wednesday, 5/11, and the following  
> Wednesday,
> 5/18, to ensure both of these tasks get completed in short order. PMC
> members Todd Lipcon and Tom White will lead the SF group and PMC  
> members Eli
> Collins and Patrick Hunt will lead the Palo Alto group.
>
> Whether you're a long-time contributor or don't have a patch to your  
> name,
> now is a great time to get involved in Apache Hadoop development.  
> Forward
> porting patches is a lot easier than writing them from scratch, and  
> you'll
> have mentors present to help guide you through the patch testing and
> submission process.
>
> To sign up for the 5/11 Hackathon in either SF or PA, head over to
> http://hadoophackathon.eventbrite.com.
>
> As a reminder, the SF HUG will be held on 5/11 at 6 pm in the  
> Cloudera SF
> offices: http://www.meetup.com/hadoopsf/events/17354462/. If you  
> can't make
> it on 5/11, we'll send out a link next week for the 5/18 event.
>
> Looking forward to getting the release train moving again!
>
> Regards,
> Jeff
>
> p.s. if you'd like to participate remotely, email me directly and  
> I'll see
> about how we can teleconference you into the event.


From general-return-3484-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 02:26:29 2011
Return-Path: <general-return-3484-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 743B950EE
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 12 May 2011 02:26:29 +0000 (UTC)
Received: (qmail 63861 invoked by uid 500); 12 May 2011 02:26:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63804 invoked by uid 500); 12 May 2011 02:26:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63796 invoked by uid 99); 12 May 2011 02:26:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 02:26:28 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mcsrivas@gmail.com designates 209.85.210.48 as permitted sender)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 02:26:23 +0000
Received: by pzk10 with SMTP id 10so740755pzk.35
        for <general@hadoop.apache.org>; Wed, 11 May 2011 19:26:03 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=6BevyoxSZxR9KfMF/vREoE2rGC94qA40BMpa5pGKnik=;
        b=U5ALXtxSbOQzvSA3OcVBGBSlO8AQzIs7Nz3yqs/gBzyW7QBO2gQRK1Puh0O+dtTqx2
         4BzmmJQrxoJS4Mk+1a1uyEPdQ3HyhUFA1DRsSbA2J6VuxrGz5Gev6Keo6jcBuhpnq9Wk
         3vwWayNUeNlfzeHBPIdPVr6rzoXiVXca2/CRI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=QnWQKI0WkabpySHf0Q6IzDBAkYkZTvSuhUgO5r42G8Ep5iP1y1B3OyZ6AaCv7jE6Zt
         xcVtCcyf1m/ioiA0wqB9HuiRFR0CQMJx236RWg3tzWGWJb4aORWlfSbd+rJV4HZT3EFR
         oy+An8LJQD/PsKxTDpPprkkElQozBDVnYyPaE=
MIME-Version: 1.0
Received: by 10.68.59.169 with SMTP id a9mr3043831pbr.60.1305167163385; Wed,
 11 May 2011 19:26:03 -0700 (PDT)
Received: by 10.68.56.193 with HTTP; Wed, 11 May 2011 19:26:03 -0700 (PDT)
In-Reply-To: <C9F04FDB.8C82%mbhandarkar@linkedin.com>
References: <FDD1BE64-981D-4FA1-87BC-0FD4A880AB7A@yahoo-inc.com>
	<C9F04FDB.8C82%mbhandarkar@linkedin.com>
Date: Wed, 11 May 2011 19:26:03 -0700
Message-ID: <BANLkTimnQxosSzWVyE_sZMn-eJv_pT0jvA@mail.gmail.com>
Subject: Re: Defining Hadoop Compatibility -revisiting-
From: "M. C. Srivas" <mcsrivas@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec53af1682566c504a30ae993

--bcaec53af1682566c504a30ae993
Content-Type: text/plain; charset=ISO-8859-1

While the HCK is a great idea to check quickly if an implementation is
"compliant",  we still need a written specification to define what is meant
by compliance, something akin to a set of RFC's, or a set of docs like the
 IEEE POSIX specifications.

For example, the POSIX.1c pthreads API has a written document that specifies
all the function calls, input params, return values, and error codes. It
clearly indicates what any POSIX-complaint threads package needs to support,
and what are vendor-specific non-portable extensions that one can use at
one's own risk.

Currently we have 2 sets of API  in the DFS and Map/Reduce layers, and the
specification is extracted only by looking at the code, or (where the code
is non-trivial) by writing really bizarre test programs to examine corner
cases. Further, the interaction between a mix of the old and new APIs is not
specified anywhere. Such specifications are vitally important when
implementing libraries like Cascading, Mahout, etc. For example, an
application might open a file using the new API, and pass that stream into a
library that manipulates the stream using some of the old API ... what is
then the expectation of the state of the stream when the library call
returns?

Sanjay Radia @ Y! already started specifying some the DFS APIs to nail such
things down. There's similar good effort in the Map/Reduce and  Avro spaces,
but it seems to have stalled somewhat. We should continue it.

Doing such specs would be a great service to the community and the users of
Hadoop. It provides them
   (a) clear-cut docs on how to use the Hadoop APIs
   (b) wider choice of Hadoop implementations by freeing them from vendor
lock-in.

Once we have such specification, the HCK becomes meaningful (since the HCK
itself will be buggy initially).


On Wed, May 11, 2011 at 2:46 PM, Milind Bhandarkar <mbhandarkar@linkedin.com
> wrote:

> I think it's time to separate out functional tests as a "Hadoop
> Compatibility Kit (HCK)", similar to the Sun TCK for Java, but under ASL
> 2.0. Then "certification" would mean "Passes 100% of the HCK testsuite."
>
> - milind
> --
> Milind Bhandarkar
> mbhandarkar@linkedin.com
>
>
>
>
>
>
> On 5/11/11 2:24 PM, "Eric Baldeschwieler" <eric14@yahoo-inc.com> wrote:
>
> >This is a really interesting topic!  I completely agree that we need to
> >get ahead of this.
> >
> >I would be really interested in learning of any experience other apache
> >projects, such as apache or tomcat have with these issues.
> >
> >---
> >E14 - typing on glass
> >
> >On May 10, 2011, at 6:31 AM, "Steve Loughran" <stevel@apache.org> wrote:
> >
> >>
> >> Back in Jan 2011, I started a discussion about how to define Apache
> >> Hadoop Compatibility:
> >>
> >>
> http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C4D
> >>46B6AD.2020802@apache.org%3E
> >>
> >> I am now reading EMC HD "Enterprise Ready" Apache Hadoop datasheet
> >>
> >>
> >>http://www.greenplum.com/sites/default/files/EMC_Greenplum_HD_DS_Final_1
> .
> >>pdf
> >>
> >> It claims that their implementations are 100% compatible, even though
> >> the Enterprise edition uses a C filesystem. It also claims that both
> >> their software releases contain "Certified Stacks", without defining
> >> what Certified means, or who does the certification -only that it is an
> >> improvement.
> >>
> >>
> >> I think we should revisit this issue before people with their own
> >> agendas define what compatibility with Apache Hadoop is for us
> >>
> >>
> >> Licensing
> >> -Use of the Hadoop codebase must follow the Apache License
> >> http://www.apache.org/licenses/LICENSE-2.0
> >> -plug in components that are dynamically linked to (Filesystems and
> >> schedulers) don't appear to be derivative works on my reading of this,
> >>
> >> Naming
> >>  -this is something for branding@apache, they will have their opinions.
> >> The key one is that the name "Apache Hadoop" must get used, and it's
> >> important to make clear it is a derivative work.
> >>  -I don't think you can claim to have a Distribution/Fork/Version of
> >> Apache Hadoop if you swap out big chunks of it for alternate
> >> filesystems, MR engines, etc. Some description of this is needed
> >> "Supports the Apache Hadoop MapReduce engine on top of Filesystem XYZ"
> >>
> >> Compatibility
> >>  -the definition of the Hadoop interfaces and classes is the Apache
> >> Source tree,
> >>  -the definition of semantics of the Hadoop interfaces and classes is
> >> the Apache Source tree, including the test classes.
> >>  -the verification that the actual semantics of an Apache Hadoop
> >> release is compatible with the expected semantics is that current and
> >> future tests pass
> >>  -bug reports can highlight incompatibility with expectations of
> >> community users, and once incorporated into tests form part of the
> >> compatibility testing
> >>  -vendors can claim and even certify their derivative works as
> >> compatible with other versions of their derivative works, but cannot
> >> claim compatibility with Apache Hadoop unless their code passes the
> >> tests and is consistent with the bug reports marked as ("by design").
> >> Perhaps we should have tests that verify each of these "by design"
> >> bugreps to make them more formal.
> >>
> >> Certification
> >>  -I have no idea what this means in EMC's case, they just say
> >>"Certified"
> >>  -As we don't do any certification ourselves, it would seem impossible
> >> for us to certify that any derivative work is compatible.
> >>  -It may be best to state that nobody can certify their derivative as
> >> "compatible with Apache Hadoop" unless it passes all current test suites
> >>  -And require that anyone who declares compatibility define what they
> >> mean by this
> >>
> >> This is a good argument for getting more functional tests out there
> >> -whoever has more functional tests needs to get them into a test module
> >> that can be used to test real deployments.
> >>
>
>

--bcaec53af1682566c504a30ae993--

From general-return-3485-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 04:17:47 2011
Return-Path: <general-return-3485-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9B9265F66
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 12 May 2011 04:17:47 +0000 (UTC)
Received: (qmail 28620 invoked by uid 500); 12 May 2011 04:17:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28567 invoked by uid 500); 12 May 2011 04:17:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28559 invoked by uid 99); 12 May 2011 04:17:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 04:17:35 +0000
X-ASF-Spam-Status: No, hits=3.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jameswarren3@gmail.com designates 209.85.218.48 as permitted sender)
Received: from [209.85.218.48] (HELO mail-yi0-f48.google.com) (209.85.218.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 04:17:29 +0000
Received: by yia28 with SMTP id 28so568673yia.35
        for <general@hadoop.apache.org>; Wed, 11 May 2011 21:17:08 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:reply-to:sender:in-reply-to
         :references:from:date:x-google-sender-auth:message-id:subject:to
         :content-type;
        bh=3doTDzlyH6jsFiIIKOwsd9BFjbTZCs9ehdy27y0JSCM=;
        b=YTFoAKNocyF1cHo1h5EzJXB8UcYkbc/KBHNGi1nkpvpCSF3lFknl4DSRTPEEF8hkUU
         Rmm6cGS3VXHdzzOk6bmMqrT8QM6v/vx4Q7Xhd0RRlaN5uy3bLjwXifazBRf03ig64Vi1
         QWpcbz1FSDu1enGeO2kztvTb2ZnYjfCDLLT0k=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:reply-to:sender:in-reply-to:references:from:date
         :x-google-sender-auth:message-id:subject:to:content-type;
        b=Xu7yC2zKuXyZ2Q2EPfUmfgYtcNYI24Hqb4wYv4PL0ZxK86hzQBTUIav+QrPP8p0oi1
         VcNuFi/HyZAyKBNjg97xdYlMH55qZpCyCa+kZUP8OncpnFotKdC0P1tqyIjy/bAHf5Dk
         Hn9CtOFg/k5CiHYHOTYQMH4IV0zqk/jK45xOQ=
Received: by 10.236.95.140 with SMTP id p12mr11040490yhf.191.1305173828235;
 Wed, 11 May 2011 21:17:08 -0700 (PDT)
MIME-Version: 1.0
Reply-To: james.warren@stanfordalumni.org
Sender: jameswarren3@gmail.com
Received: by 10.236.110.16 with HTTP; Wed, 11 May 2011 21:16:28 -0700 (PDT)
In-Reply-To: <BANLkTi=5n7g+xB=kZ+RgGvwstkkb=opasA@mail.gmail.com>
References: <BANLkTiksqFvXdN68Ki4ZK0y=GLmr7s=MJg@mail.gmail.com> <BANLkTi=5n7g+xB=kZ+RgGvwstkkb=opasA@mail.gmail.com>
From: James Warren <james.warren@stanfordalumni.org>
Date: Wed, 11 May 2011 21:16:28 -0700
X-Google-Sender-Auth: q-lSF3kO6MFETztpIaep0ZKVu-I
Message-ID: <BANLkTik9ANgnPZOyBv6wVwVhugR2Lw9BNg@mail.gmail.com>
Subject: Re: Apache Hadoop Hackathons: 5/11 and 5/18 in SF and Palo Alto
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0023547c8c3366f0ec04a30c76b1
X-Virus-Checked: Checked by ClamAV on apache.org

--0023547c8c3366f0ec04a30c76b1
Content-Type: text/plain; charset=ISO-8859-1

Many thanks to Cloudera for hosting today's hackathons!  Under the helpful
guidance of Eli Collins, I submitted my first patch.  It's great to get off
the sidelines and into the game.

+1 for more of these community building events.

Thanks again,
-James Warren

On Mon, May 9, 2011 at 2:36 PM, Jeff Hammerbacher <hammer@cloudera.com>wrote:

> Hey,
>
> We've got 23 folks signed up for the Hackathon this Wednesday from
> organizations like Cloudera, Facebook, Twitter, AOL, Trend Micro,
> StumbleUpon, and Ngmoco. We'll also have the VP of the HBase PMC, Michael
> Stack, and the VP of the Hive PMC, John Sichi. If you've been waiting to
> get
> involved in Apache Hadoop, development, now is the time!
>
> We've got room for 10 - 15 more. If you're in San Francisco or Palo Alto
> and
> want to help out with Apache Hadoop development, sign up at
> http://hadoophackathon.eventbrite.com.
>
> Regards,
> Jeff
>
> On Fri, May 6, 2011 at 12:03 PM, Jeff Hammerbacher <hammer@cloudera.com
> >wrote:
>
> > Hey,
> >
> > The discussion this week about the 0.20.203.0 release has done a great
> job
> > of highlighting some issues in our development process; it's also done a
> > great job of lifting our mailing list activity metrics. After reading
> > through the various threads, it's clear that everyone agrees on two
> things:
> >
> > 1) We'd like to get the work done in the 0.20.x branches into trunk
> > 2) We'd like to start releasing off of trunk again
> >
> > To that end, a few folks from Yahoo!, Cloudera, StumbleUpon, and Facebook
> > would like to put together a series of Hackathons to 1) burn down the 13
> > (only 13!) remaining blockers for the 0.22 release and 2) forward-port
> the
> > work done in the 0.20.x branches into trunk.
> >
> > Cloudera will be hosting Hackathons from 10 am to 6 pm in both our Palo
> > Alto and San Francisco offices next Wednesday, 5/11, and the following
> > Wednesday, 5/18, to ensure both of these tasks get completed in short
> order.
> > PMC members Todd Lipcon and Tom White will lead the SF group and PMC
> members
> > Eli Collins and Patrick Hunt will lead the Palo Alto group.
> >
> > Whether you're a long-time contributor or don't have a patch to your
> name,
> > now is a great time to get involved in Apache Hadoop development. Forward
> > porting patches is a lot easier than writing them from scratch, and
> you'll
> > have mentors present to help guide you through the patch testing and
> > submission process.
> >
> > To sign up for the 5/11 Hackathon in either SF or PA, head over to
> > http://hadoophackathon.eventbrite.com.
> >
> > As a reminder, the SF HUG will be held on 5/11 at 6 pm in the Cloudera SF
> > offices: http://www.meetup.com/hadoopsf/events/17354462/. If you can't
> > make it on 5/11, we'll send out a link next week for the 5/18 event.
> >
> > Looking forward to getting the release train moving again!
> >
> > Regards,
> > Jeff
> >
> > p.s. if you'd like to participate remotely, email me directly and I'll
> see
> > about how we can teleconference you into the event.
> >
> >
> >
>

--0023547c8c3366f0ec04a30c76b1--

From general-return-3486-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 04:38:52 2011
Return-Path: <general-return-3486-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 445CE502E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 12 May 2011 04:38:52 +0000 (UTC)
Received: (qmail 42851 invoked by uid 500); 12 May 2011 04:38:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42774 invoked by uid 500); 12 May 2011 04:38:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42766 invoked by uid 99); 12 May 2011 04:38:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 04:38:46 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.220.176] (HELO mail-vx0-f176.google.com) (209.85.220.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 04:38:37 +0000
Received: by vxa37 with SMTP id 37so1372562vxa.35
        for <general@hadoop.apache.org>; Wed, 11 May 2011 21:38:16 -0700 (PDT)
Received: by 10.52.110.135 with SMTP id ia7mr4229701vdb.294.1305175096100;
 Wed, 11 May 2011 21:38:16 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.52.112.7 with HTTP; Wed, 11 May 2011 21:37:56 -0700 (PDT)
X-Originating-IP: [67.160.196.149]
In-Reply-To: <BANLkTimnQxosSzWVyE_sZMn-eJv_pT0jvA@mail.gmail.com>
References: <FDD1BE64-981D-4FA1-87BC-0FD4A880AB7A@yahoo-inc.com>
 <C9F04FDB.8C82%mbhandarkar@linkedin.com> <BANLkTimnQxosSzWVyE_sZMn-eJv_pT0jvA@mail.gmail.com>
From: Ted Dunning <tdunning@maprtech.com>
Date: Wed, 11 May 2011 21:37:56 -0700
Message-ID: <BANLkTimd-w=j-=cCnzvc1kVzcienZOU8Cw@mail.gmail.com>
Subject: Re: Defining Hadoop Compatibility -revisiting-
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec5486072f9077704a30cc1fd
X-Virus-Checked: Checked by ClamAV on apache.org

--bcaec5486072f9077704a30cc1fd
Content-Type: text/plain; charset=ISO-8859-1

As a specific example of how these are important, over in Mahout-land we
have been wrestling with determining just what it means to have dependencies
in the lib directory inside a jar.  This isn't documented, behaves
differently in different versions of Hadoop and means that some Mahout
programs work sometimes, but fail in high-profile locations like Twitter.

On Wed, May 11, 2011 at 7:26 PM, M. C. Srivas <mcsrivas@gmail.com> wrote:

> Such specifications are vitally important when
> implementing libraries like Cascading, Mahout, etc. For example, an
> application might open a file using the new API, and pass that stream into
> a
> library that manipulates the stream using some of the old API ... what is
> then the expectation of the state of the stream when the library call
> returns?
>

--bcaec5486072f9077704a30cc1fd--

From general-return-3487-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 07:12:53 2011
Return-Path: <general-return-3487-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6712657E1
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 12 May 2011 07:12:53 +0000 (UTC)
Received: (qmail 66473 invoked by uid 500); 12 May 2011 07:12:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66080 invoked by uid 500); 12 May 2011 07:12:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66072 invoked by uid 99); 12 May 2011 07:12:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 07:12:45 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 07:12:39 +0000
Received: by ewy22 with SMTP id 22so533303ewy.35
        for <general@hadoop.apache.org>; Thu, 12 May 2011 00:12:19 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.14.46.214 with SMTP id r62mr4309368eeb.99.1305184338765; Thu,
 12 May 2011 00:12:18 -0700 (PDT)
Received: by 10.14.37.13 with HTTP; Thu, 12 May 2011 00:12:18 -0700 (PDT)
In-Reply-To: <DA3EE4F3-E4E7-4857-8428-EBDC26ED25A5@Holsman.NET>
References: <FABE9911-352F-4AE2-96B1-27E66FB0369A@apache.org>
	<DA3EE4F3-E4E7-4857-8428-EBDC26ED25A5@Holsman.NET>
Date: Thu, 12 May 2011 00:12:18 -0700
Message-ID: <BANLkTikMBwRscfR80OmejQLa4a_2HqwcLw@mail.gmail.com>
Subject: Re: Cleaning up documentation from website
From: Eli Collins <eli@cloudera.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Wednesday, May 11, 2011, Ian Holsman <hadoop@holsman.net> wrote:
> I think we need a 0.18 doc's as some people are still running this.
> other than that (and what is 'current'.. it should point to the stable release imho) I'm good with it.

There are some non trivial 0.19 clusters running  too.

What is our definition of stable release? Stable normally means no new
features, just enhancements and bug fixes. And API stability.

Thanks,
Eli

From general-return-3488-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 09:13:30 2011
Return-Path: <general-return-3488-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C14415C97
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 12 May 2011 09:13:30 +0000 (UTC)
Received: (qmail 97281 invoked by uid 500); 12 May 2011 09:13:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97183 invoked by uid 500); 12 May 2011 09:13:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97175 invoked by uid 99); 12 May 2011 09:13:28 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 09:13:28 +0000
Received: from localhost (HELO [192.168.42.176]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 09:13:28 +0000
Message-ID: <4DCBA4B3.2070704@apache.org>
Date: Thu, 12 May 2011 11:13:23 +0200
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: OT: anyone else going to berlin buzzwords?
References: <B4C351A8-CACF-42C5-AE02-8334B8A95712@holsman.net>	<4DCA520D.70405@apache.org> <BANLkTimAH6U5yYwQhb=M7vxbN08+oztwLg@mail.gmail.com>
In-Reply-To: <BANLkTimAH6U5yYwQhb=M7vxbN08+oztwLg@mail.gmail.com>
X-Enigmail-Version: 1.1.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

I'll be there, but only for the conference on Monday and Tuesday.

Doug

On 05/11/2011 12:31 PM, Bernd Fondermann wrote:
> On Wed, May 11, 2011 at 11:08, Steve Loughran <stevel@apache.org> wrote:
>> On 11/05/2011 00:53, Ian Holsman wrote:
>>>
>>> If so.. I'll be there.. let's catch up.
>>>
>>> http://www.berlinbuzzwords.de/
>>>
>>
>> 1. I'm going to be there
> 
> 1.1.
> me too, but only for the conference on Mon + Tue.
> Who's available for a Hadoop/Apache/etc food + beverages on Monday evening?
> 
>   Bernd

From general-return-3489-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 09:23:52 2011
Return-Path: <general-return-3489-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2A88A5234
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 12 May 2011 09:23:52 +0000 (UTC)
Received: (qmail 4998 invoked by uid 500); 12 May 2011 09:23:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4948 invoked by uid 500); 12 May 2011 09:23:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4940 invoked by uid 99); 12 May 2011 09:23:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 09:23:50 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 09:23:42 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 5CA81B7E3C
	for <general@hadoop.apache.org>; Thu, 12 May 2011 10:23:20 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id 8iFiwQ7sQNW5 for <general@hadoop.apache.org>;
	Thu, 12 May 2011 10:23:19 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 6023FB7E3F
	for <general@hadoop.apache.org>; Thu, 12 May 2011 10:23:19 +0100 (BST)
MailScanner-NULL-Check: 1305796987.23926@AVvdiBsLQ7w1oaPZwxfTWw
Received: from [16.24.209.249] (jleen.emea.hpqcorp.net [16.24.209.249])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p4C9N6dS001605
	for <general@hadoop.apache.org>; Thu, 12 May 2011 10:23:06 +0100 (BST)
Message-ID: <4DCBA6F5.90006@apache.org>
Date: Thu, 12 May 2011 10:23:01 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-GB; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Defining Hadoop Compatibility -revisiting-
References: <4DC91392.2010308@apache.org> <FDD1BE64-981D-4FA1-87BC-0FD4A880AB7A@yahoo-inc.com>
In-Reply-To: <FDD1BE64-981D-4FA1-87BC-0FD4A880AB7A@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p4C9N6dS001605
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 11/05/2011 22:24, Eric Baldeschwieler wrote:
> This is a really interesting topic!  I completely agree that we need to get ahead of this.
>
> I would be really interested in learning of any experience other apache projects, such as apache or tomcat have with these issues.

I don't know about apache httpd

Tomcat is the JCP reference implementation of JSP, the JSP Jar is 
broadly reused, and the JCP program defines a test kit (with licensing 
T&Cs) to define compatibility. That is because the JCP program was 
designed to split specification from implementation.

Hadoop doesn't have that, which is a strength and a weakness. Strong: 
agility. Weakness: compatibility between versions as well as with others.

I think Sun NFS might be a good example of similar defacto standard, or 
MS SMB -it is up to others to show they are compatible with what is 
effective the reference implementation. Being closed source, there is no 
option for anyone to include SunOS NFS or MS SMB in their products -the 
issue of "how much of SunOS NFS to include before you have to stop 
calling it that" never arose.




From general-return-3490-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 09:33:41 2011
Return-Path: <general-return-3490-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 72CB2558B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 12 May 2011 09:33:41 +0000 (UTC)
Received: (qmail 14463 invoked by uid 500); 12 May 2011 09:33:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14407 invoked by uid 500); 12 May 2011 09:33:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14399 invoked by uid 99); 12 May 2011 09:33:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 09:33:40 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 09:33:30 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 569EDB7EBE
	for <general@hadoop.apache.org>; Thu, 12 May 2011 10:33:10 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id 355DBJ40t5Pf for <general@hadoop.apache.org>;
	Thu, 12 May 2011 10:33:04 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 35EEBB7EB3
	for <general@hadoop.apache.org>; Thu, 12 May 2011 10:33:03 +0100 (BST)
MailScanner-NULL-Check: 1305797571.9747@b0jqKJWy0m2uIebuzcUTAQ
Received: from [16.24.209.249] (jleen.emea.hpqcorp.net [16.24.209.249])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p4C9WpUb002165
	for <general@hadoop.apache.org>; Thu, 12 May 2011 10:32:51 +0100 (BST)
Message-ID: <4DCBA93E.2050005@apache.org>
Date: Thu, 12 May 2011 10:32:46 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-GB; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Defining Hadoop Compatibility -revisiting-
References: <FDD1BE64-981D-4FA1-87BC-0FD4A880AB7A@yahoo-inc.com>	<C9F04FDB.8C82%mbhandarkar@linkedin.com> <BANLkTimnQxosSzWVyE_sZMn-eJv_pT0jvA@mail.gmail.com>
In-Reply-To: <BANLkTimnQxosSzWVyE_sZMn-eJv_pT0jvA@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p4C9WpUb002165
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 12/05/2011 03:26, M. C. Srivas wrote:
> While the HCK is a great idea to check quickly if an implementation is
> "compliant",  we still need a written specification to define what is meant
> by compliance, something akin to a set of RFC's, or a set of docs like the
>   IEEE POSIX specifications.
>
> For example, the POSIX.1c pthreads API has a written document that specifies
> all the function calls, input params, return values, and error codes. It
> clearly indicates what any POSIX-complaint threads package needs to support,
> and what are vendor-specific non-portable extensions that one can use at
> one's own risk.

I have been known to be critical of standards bodies in the past
http://www.waterfall2006.com/loughran.html

And I've been in them. It is absolutely essential that the Hadoop stack 
doesn't become controlled by a standards body, as then you become 
controlled by whoever can afford to send the most people to the 
standards events -and make behind the scenes deals with others to get 
votes through.

> Currently we have 2 sets of API  in the DFS and Map/Reduce layers, and the
> specification is extracted only by looking at the code, or (where the code
> is non-trivial) by writing really bizarre test programs to examine corner
> cases. Further, the interaction between a mix of the old and new APIs is not
> specified anywhere. Such specifications are vitally important when
> implementing libraries like Cascading, Mahout, etc. For example, an
> application might open a file using the new API, and pass that stream into a
> library that manipulates the stream using some of the old API ... what is
> then the expectation of the state of the stream when the library call
> returns?
>
> Sanjay Radia @ Y! already started specifying some the DFS APIs to nail such
> things down. There's similar good effort in the Map/Reduce and  Avro spaces,
> but it seems to have stalled somewhat. We should continue it.
>
> Doing such specs would be a great service to the community and the users of
> Hadoop. It provides them
>     (a) clear-cut docs on how to use the Hadoop APIs#

+1

>     (b) wider choice of Hadoop implementations by freeing them from vendor
> lock-in.

=0

They won't be hadoop implementations, they will be "something that is 
compatible with the Apache Hadoop API as defined in v 0.x of the Hadoop 
compatibility kit". Furthermore, there's the issue of any google patents 
-while google have given Hadoop permission to them, that may not apply 
to other things that implement compatible APIs.

I also think that the Hadoop team need to be the one's who own the 
interfaces and tests, define the tests as a functional test suite for 
testing Hadoop distributions, and reserve the right to make changes to 
the interfaces, semantics and tests as suits the teams needs. The input 
from others -especially related community projects- are important, but, 
to be ruthless, the compatibility issues with things that aren't really 
Apache Hadoop are less important. you choose to reimplement Hadoop, you 
take on the costs of staying current.

>
> Once we have such specification, the HCK becomes meaningful (since the HCK
> itself will be buggy initially).




From general-return-3491-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 09:34:43 2011
Return-Path: <general-return-3491-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 99C605891
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 12 May 2011 09:34:43 +0000 (UTC)
Received: (qmail 17541 invoked by uid 500); 12 May 2011 09:34:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17494 invoked by uid 500); 12 May 2011 09:34:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17486 invoked by uid 99); 12 May 2011 09:34:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 09:34:42 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 09:34:33 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 618BA1BA98E
	for <general@hadoop.apache.org>; Thu, 12 May 2011 10:34:12 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id MWn6mKr+39oo for <general@hadoop.apache.org>;
	Thu, 12 May 2011 10:34:12 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by colossus.hpl.hp.com (Postfix) with ESMTP id E9C7B1BA96E
	for <general@hadoop.apache.org>; Thu, 12 May 2011 10:34:11 +0100 (BST)
MailScanner-NULL-Check: 1305797639.63819@igY1648vUPtCM24qMdnKKQ
Received: from [16.24.209.249] (jleen.emea.hpqcorp.net [16.24.209.249])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p4C9Xu5c002209
	for <general@hadoop.apache.org>; Thu, 12 May 2011 10:33:56 +0100 (BST)
Message-ID: <4DCBA97F.5010200@apache.org>
Date: Thu, 12 May 2011 10:33:51 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-GB; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Defining Hadoop Compatibility -revisiting-
References: <4DC91392.2010308@apache.org> <FDD1BE64-981D-4FA1-87BC-0FD4A880AB7A@yahoo-inc.com> <6407C0E3-934F-4C9D-9103-C738F6C1AED7@Holsman.NET> <BANLkTik9NeKPEuoPO6ijyzbyidweHDFKBg@mail.gmail.com> <BANLkTikCDLMRhoLLKhTfJEgc7dOCYHonpQ@mail.gmail.com>
In-Reply-To: <BANLkTikCDLMRhoLLKhTfJEgc7dOCYHonpQ@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p4C9Xu5c002209
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 12/05/2011 00:20, Aaron Kimball wrote:
> What does it mean to "implement" those interfaces? I'm +1 for a TCK-based
> definition. In addition to statically implementing a set of interfaces, each
> interface also implicitly includes a set of acceptable inputs and predicted
> outputs (or ranges of outputs) for those inputs.
>

+1: Parnas's definition of "interface" is signature+semantics. Java 
interface files just define the signature, not behaviour. A test kit is 
needed.



From general-return-3492-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 09:49:28 2011
Return-Path: <general-return-3492-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E7D845EAF
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 12 May 2011 09:49:28 +0000 (UTC)
Received: (qmail 31461 invoked by uid 500); 12 May 2011 09:49:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31402 invoked by uid 500); 12 May 2011 09:49:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31394 invoked by uid 99); 12 May 2011 09:49:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 09:49:27 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of msegel@navteq.com designates 65.167.11.240 as permitted sender)
Received: from [65.167.11.240] (HELO xmailchi.navteq.com) (65.167.11.240)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 09:49:21 +0000
Received: from chicago-spamfw.navteq.com (imailchi [10.8.5.161])
	by xmailchi.navteq.com (8.13.8/8.13.6) with ESMTP id p4C9mvUu026282
	for <general@hadoop.apache.org>; Thu, 12 May 2011 04:48:57 -0500
Received: from imailchi.hq.navteq.com (localhost [127.0.0.1])
	by chicago-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id 33E9541C21A6
	for <general@hadoop.apache.org>; Thu, 12 May 2011 04:48:57 -0500 (CDT)
Received: from imailchi.hq.navteq.com ([10.8.120.104]) by chicago-spamfw.navteq.com with ESMTP id PQZD6z7hqaGHD0GR for <general@hadoop.apache.org>; Thu, 12 May 2011 04:48:57 -0500 (CDT)
Received: from hq-ex-ht02.ad.navteq.com (hq-ex-ht02.ad.navteq.com [10.8.222.172])
	by imailchi.hq.navteq.com (8.12.9/8.12.9) with ESMTP id p4C9mvNe023551;
	Thu, 12 May 2011 04:48:57 -0500
Received: from hq-ex-mb03.ad.navteq.com ([fe80::c4dd:7b21:5c22:cfe4]) by
 hq-ex-ht02.ad.navteq.com ([fe80::24f6:ca50:734d:df6e%13]) with mapi; Thu, 12
 May 2011 04:48:56 -0500
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 12 May 2011 04:49:08 -0500
Subject: Re: Defining Hadoop Compatibility -revisiting-
Thread-Topic: Defining Hadoop Compatibility -revisiting-
Thread-Index: AcwQic86KG3YNpOOReaZJbYFy54s+Q==
Message-ID: <003DD45B-DBE7-47F1-BB51-7566C47E239A@navteq.com>
References: <FDD1BE64-981D-4FA1-87BC-0FD4A880AB7A@yahoo-inc.com>
 <C9F04FDB.8C82%mbhandarkar@linkedin.com>
 <BANLkTimnQxosSzWVyE_sZMn-eJv_pT0jvA@mail.gmail.com>
 <4DCBA93E.2050005@apache.org>
In-Reply-To: <4DCBA93E.2050005@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_003DD45BDBE747F1BB517566C47E239Anavteqcom_"
MIME-Version: 1.0

--_000_003DD45BDBE747F1BB517566C47E239Anavteqcom_
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1

While IANAL...

As long as any implementation follows Apache's license regarding derivative works, it's fair game.  (this is my understanding YMMV)

The APL is very liberal in what one can do with a derivative work...

Surely Apache has some lawyers who can summarize what is allowable when talking about a derivative work and what is not?

Note these are my opinions only and do not reflect the opinions of anyone else. Any resemblance to a coherent thought is pure coincidence.....


Sent from a remote device. Please excuse any typos...

Mike Segel

On May 12, 2011, at 4:33 AM, "Steve Loughran" <stevel@apache.org<mailto:stevel@apache.org>> wrote:


   (b) wider choice of Hadoop implementations by freeing them from vendor
lock-in.

=0

They won't be hadoop implementations, they will be "something that is
compatible with the Apache Hadoop API as defined in v 0.x of the Hadoop
compatibility kit". Furthermore, there's the issue of any google patents
-while google have given Hadoop permission to them, that may not apply
to other things that implement compatible APIs.




The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above.  If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited.  If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.

--_000_003DD45BDBE747F1BB517566C47E239Anavteqcom_--

From general-return-3493-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 15:45:15 2011
Return-Path: <general-return-3493-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D3BD45E39
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 12 May 2011 15:45:15 +0000 (UTC)
Received: (qmail 11010 invoked by uid 500); 12 May 2011 15:45:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 10963 invoked by uid 500); 12 May 2011 15:45:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 10955 invoked by uid 99); 12 May 2011 15:45:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 15:45:14 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 15:45:07 +0000
Received: by vws7 with SMTP id 7so1914172vws.35
        for <general@hadoop.apache.org>; Thu, 12 May 2011 08:44:46 -0700 (PDT)
Received: by 10.52.181.99 with SMTP id dv3mr544411vdc.14.1305215086238; Thu,
 12 May 2011 08:44:46 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.52.112.7 with HTTP; Thu, 12 May 2011 08:44:26 -0700 (PDT)
X-Originating-IP: [67.160.196.149]
In-Reply-To: <4DCBA4B3.2070704@apache.org>
References: <B4C351A8-CACF-42C5-AE02-8334B8A95712@holsman.net>
 <4DCA520D.70405@apache.org> <BANLkTimAH6U5yYwQhb=M7vxbN08+oztwLg@mail.gmail.com>
 <4DCBA4B3.2070704@apache.org>
From: Ted Dunning <tdunning@maprtech.com>
Date: Thu, 12 May 2011 08:44:26 -0700
Message-ID: <BANLkTimievMK7XMGYb5VZGcgHqj9aorMxQ@mail.gmail.com>
Subject: Re: OT: anyone else going to berlin buzzwords?
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec548a5bd921c7204a31611aa

--bcaec548a5bd921c7204a31611aa
Content-Type: text/plain; charset=ISO-8859-1

I will be there Monday-Thursday.

Monday sounds good for food and beverages.

On Thu, May 12, 2011 at 2:13 AM, Doug Cutting <cutting@apache.org> wrote:

> I'll be there, but only for the conference on Monday and Tuesday.
>
> Doug
>
> On 05/11/2011 12:31 PM, Bernd Fondermann wrote:
> > On Wed, May 11, 2011 at 11:08, Steve Loughran <stevel@apache.org> wrote:
> >> On 11/05/2011 00:53, Ian Holsman wrote:
> >>>
> >>> If so.. I'll be there.. let's catch up.
> >>>
> >>> http://www.berlinbuzzwords.de/
> >>>
> >>
> >> 1. I'm going to be there
> >
> > 1.1.
> > me too, but only for the conference on Mon + Tue.
> > Who's available for a Hadoop/Apache/etc food + beverages on Monday
> evening?
> >
> >   Bernd
>

--bcaec548a5bd921c7204a31611aa--

From general-return-3494-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 16:42:32 2011
Return-Path: <general-return-3494-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6B7215B34
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 12 May 2011 16:42:32 +0000 (UTC)
Received: (qmail 11270 invoked by uid 500); 12 May 2011 16:42:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11218 invoked by uid 500); 12 May 2011 16:42:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11209 invoked by uid 99); 12 May 2011 16:42:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 16:42:30 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mbhandarkar@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 16:42:26 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:In-Reply-To:Accept-Language:
   Content-Language:X-MS-Has-Attach:X-MS-TNEF-Correlator:
   x-originating-ip:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version;
  b=lf62ucnZK24HT9MRxrJ2hmAQ7wh0IDWUFl1GUzSE2PqI3h6FW2kLDR2c
   6nj2oL0YNodD/e3RnR/RjE03O3nfZBa221cKhCVd8y3CkZotKD/oOki+O
   HgFbvdtJboS8WfD;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=mbhandarkar@linkedin.com; q=dns/txt;
  s=proddkim; t=1305218546; x=1336754546;
  h=from:to:subject:date:message-id:in-reply-to:content-id:
   content-transfer-encoding:mime-version;
  bh=pRbLfifJeupCSoErF03Csy2MMz4Rv7Vkoq25ucmP3oE=;
  b=rXw29tY0FwKJem5LKs453KRozGpUtLe7+NnVlzYPt2XB8F+Betq5l8/e
   iq2w76HeIBlMvKqok0YIekeRCElyxzcUMlFl8xKNd1kAQKAPqMbdoAocD
   qmlzynq0f71+y4E;
X-IronPort-AV: E=Sophos;i="4.64,359,1301900400"; 
   d="scan'208";a="23121408"
Received: from ESV4-EXC02.linkedin.biz ([fe80::4d74:48bd:e0bd:13ee]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi id 14.01.0251.000; Thu,
 12 May 2011 09:45:41 -0700
From: Milind Bhandarkar <mbhandarkar@linkedin.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: Defining Hadoop Compatibility -revisiting-
Thread-Topic: Defining Hadoop Compatibility -revisiting-
Thread-Index: AQHMDv3EIB+Y40jJf0ahsOjyQf2X+pSImxGA//+PqQCAAMSEgIAAedIA
Date: Thu, 12 May 2011 16:45:41 +0000
Message-ID: <C9F15AFB.8D9C%mbhandarkar@linkedin.com>
In-Reply-To: <BANLkTimnQxosSzWVyE_sZMn-eJv_pT0jvA@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <143F15C7163C4A46BF2FAFA4AC9713C6@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

HCK and written specifications are not mutually exclusive. However, given
the evolving nature of Hadoop APIs, functional tests need to evolve as
well, and having them tied to a "current stable" version is easier to do
than it is to tie the written specifications.

- milind

--=20
Milind Bhandarkar
mbhandarkar@linkedin.com
+1-650-776-3167






On 5/11/11 7:26 PM, "M. C. Srivas" <mcsrivas@gmail.com> wrote:

>While the HCK is a great idea to check quickly if an implementation is
>"compliant",  we still need a written specification to define what is
>meant
>by compliance, something akin to a set of RFC's, or a set of docs like the
> IEEE POSIX specifications.
>
>For example, the POSIX.1c pthreads API has a written document that
>specifies
>all the function calls, input params, return values, and error codes. It
>clearly indicates what any POSIX-complaint threads package needs to
>support,
>and what are vendor-specific non-portable extensions that one can use at
>one's own risk.
>
>Currently we have 2 sets of API  in the DFS and Map/Reduce layers, and the
>specification is extracted only by looking at the code, or (where the code
>is non-trivial) by writing really bizarre test programs to examine corner
>cases. Further, the interaction between a mix of the old and new APIs is
>not
>specified anywhere. Such specifications are vitally important when
>implementing libraries like Cascading, Mahout, etc. For example, an
>application might open a file using the new API, and pass that stream
>into a
>library that manipulates the stream using some of the old API ... what is
>then the expectation of the state of the stream when the library call
>returns?
>
>Sanjay Radia @ Y! already started specifying some the DFS APIs to nail
>such
>things down. There's similar good effort in the Map/Reduce and  Avro
>spaces,
>but it seems to have stalled somewhat. We should continue it.
>
>Doing such specs would be a great service to the community and the users
>of
>Hadoop. It provides them
>   (a) clear-cut docs on how to use the Hadoop APIs
>   (b) wider choice of Hadoop implementations by freeing them from vendor
>lock-in.
>
>Once we have such specification, the HCK becomes meaningful (since the HCK
>itself will be buggy initially).
>
>
>On Wed, May 11, 2011 at 2:46 PM, Milind Bhandarkar
><mbhandarkar@linkedin.com
>> wrote:
>
>> I think it's time to separate out functional tests as a "Hadoop
>> Compatibility Kit (HCK)", similar to the Sun TCK for Java, but under ASL
>> 2.0. Then "certification" would mean "Passes 100% of the HCK testsuite."
>>
>> - milind
>> --
>> Milind Bhandarkar
>> mbhandarkar@linkedin.com
>>
>>
>>
>>
>>
>>
>> On 5/11/11 2:24 PM, "Eric Baldeschwieler" <eric14@yahoo-inc.com> wrote:
>>
>> >This is a really interesting topic!  I completely agree that we need to
>> >get ahead of this.
>> >
>> >I would be really interested in learning of any experience other apache
>> >projects, such as apache or tomcat have with these issues.
>> >
>> >---
>> >E14 - typing on glass
>> >
>> >On May 10, 2011, at 6:31 AM, "Steve Loughran" <stevel@apache.org>
>>wrote:
>> >
>> >>
>> >> Back in Jan 2011, I started a discussion about how to define Apache
>> >> Hadoop Compatibility:
>> >>
>> >>
>>=20
>>http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C4D
>> >>46B6AD.2020802@apache.org%3E
>> >>
>> >> I am now reading EMC HD "Enterprise Ready" Apache Hadoop datasheet
>> >>
>> >>
>>=20
>>>>http://www.greenplum.com/sites/default/files/EMC_Greenplum_HD_DS_Final_
>>>>1
>> .
>> >>pdf
>> >>
>> >> It claims that their implementations are 100% compatible, even though
>> >> the Enterprise edition uses a C filesystem. It also claims that both
>> >> their software releases contain "Certified Stacks", without defining
>> >> what Certified means, or who does the certification -only that it is
>>an
>> >> improvement.
>> >>
>> >>
>> >> I think we should revisit this issue before people with their own
>> >> agendas define what compatibility with Apache Hadoop is for us
>> >>
>> >>
>> >> Licensing
>> >> -Use of the Hadoop codebase must follow the Apache License
>> >> http://www.apache.org/licenses/LICENSE-2.0
>> >> -plug in components that are dynamically linked to (Filesystems and
>> >> schedulers) don't appear to be derivative works on my reading of
>>this,
>> >>
>> >> Naming
>> >>  -this is something for branding@apache, they will have their
>>opinions.
>> >> The key one is that the name "Apache Hadoop" must get used, and it's
>> >> important to make clear it is a derivative work.
>> >>  -I don't think you can claim to have a Distribution/Fork/Version of
>> >> Apache Hadoop if you swap out big chunks of it for alternate
>> >> filesystems, MR engines, etc. Some description of this is needed
>> >> "Supports the Apache Hadoop MapReduce engine on top of Filesystem
>>XYZ"
>> >>
>> >> Compatibility
>> >>  -the definition of the Hadoop interfaces and classes is the Apache
>> >> Source tree,
>> >>  -the definition of semantics of the Hadoop interfaces and classes is
>> >> the Apache Source tree, including the test classes.
>> >>  -the verification that the actual semantics of an Apache Hadoop
>> >> release is compatible with the expected semantics is that current and
>> >> future tests pass
>> >>  -bug reports can highlight incompatibility with expectations of
>> >> community users, and once incorporated into tests form part of the
>> >> compatibility testing
>> >>  -vendors can claim and even certify their derivative works as
>> >> compatible with other versions of their derivative works, but cannot
>> >> claim compatibility with Apache Hadoop unless their code passes the
>> >> tests and is consistent with the bug reports marked as ("by design").
>> >> Perhaps we should have tests that verify each of these "by design"
>> >> bugreps to make them more formal.
>> >>
>> >> Certification
>> >>  -I have no idea what this means in EMC's case, they just say
>> >>"Certified"
>> >>  -As we don't do any certification ourselves, it would seem
>>impossible
>> >> for us to certify that any derivative work is compatible.
>> >>  -It may be best to state that nobody can certify their derivative as
>> >> "compatible with Apache Hadoop" unless it passes all current test
>>suites
>> >>  -And require that anyone who declares compatibility define what they
>> >> mean by this
>> >>
>> >> This is a good argument for getting more functional tests out there
>> >> -whoever has more functional tests needs to get them into a test
>>module
>> >> that can be used to test real deployments.
>> >>
>>
>>


From general-return-3495-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 16:45:51 2011
Return-Path: <general-return-3495-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5C19B52BB
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 12 May 2011 16:45:51 +0000 (UTC)
Received: (qmail 17353 invoked by uid 500); 12 May 2011 16:45:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17297 invoked by uid 500); 12 May 2011 16:45:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17289 invoked by uid 99); 12 May 2011 16:45:49 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 16:45:49 +0000
Received: from localhost (HELO [172.16.37.77]) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 16:45:49 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Defining Hadoop Compatibility -revisiting-
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <4DCBA6F5.90006@apache.org>
Date: Thu, 12 May 2011 09:45:48 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <D49BFE4D-64FB-4CC0-8224-A3B962F2145B@apache.org>
References: <4DC91392.2010308@apache.org> <FDD1BE64-981D-4FA1-87BC-0FD4A880AB7A@yahoo-inc.com> <4DCBA6F5.90006@apache.org>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On May 12, 2011, at 2:23 AM, Steve Loughran wrote:
> I think Sun NFS might be a good example of similar defacto standard, =
or MS SMB -it is up to others to show they are compatible with what is =
effective the reference implementation. Being closed source, there is no =
option for anyone to include SunOS NFS or MS SMB in their products -the =
issue of "how much of SunOS NFS to include before you have to stop =
calling it that" never arose.

	SMB and FAT are better examples given how much of NFS and =
associated protocols are actually defined in RFC's (1094 being the first =
one). :)

	FAT compatibility between devices is notoriously bad if you go =
beyond simple files.=

From general-return-3496-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 18:02:32 2011
Return-Path: <general-return-3496-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B5BDE5731
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 12 May 2011 18:02:32 +0000 (UTC)
Received: (qmail 61966 invoked by uid 500); 12 May 2011 18:02:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61921 invoked by uid 500); 12 May 2011 18:02:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61912 invoked by uid 99); 12 May 2011 18:02:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 18:02:31 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bernd.fondermann@googlemail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 18:02:26 +0000
Received: by bwz8 with SMTP id 8so2438587bwz.35
        for <general@hadoop.apache.org>; Thu, 12 May 2011 11:02:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type:content-transfer-encoding;
        bh=mygstrN+jW6nhWpepYhx+TJjq5scZ24+arJNtAwrCNI=;
        b=gJFdXlLSskAaunZtQLxdnNp9shmkleyvIkVnYroNXPymUn7hHZ3hA4WmI5S8Brybed
         inAlSOzkMzTwTnwu/tEA/GqUik06dlXPV7P3KfIW3nHJqfsJulb0r4YBlD6FEcwClZcg
         7ltckYjjvFP2BxRXwSpRczzG2ZLxAqTwA7qcw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=e0CHS50wzsqfyY6GLHJapoYJs11vy40CyTkrGCj+pfain0dnyC0JbDh2lqUZcdImhv
         oPcQCnbAI/akbPV+ce4AcILyjhw0iPGJZ4bHCPWSs1R5rfEsl1JHsFewcPemACYOeayK
         iT7Cm+HcyaPmWrPoLg8InKgIdpXdzOMYei3G0=
MIME-Version: 1.0
Received: by 10.204.41.2 with SMTP id m2mr525127bke.54.1305223325398; Thu, 12
 May 2011 11:02:05 -0700 (PDT)
Received: by 10.204.83.196 with HTTP; Thu, 12 May 2011 11:02:05 -0700 (PDT)
In-Reply-To: <BANLkTimievMK7XMGYb5VZGcgHqj9aorMxQ@mail.gmail.com>
References: <B4C351A8-CACF-42C5-AE02-8334B8A95712@holsman.net>
	<4DCA520D.70405@apache.org>
	<BANLkTimAH6U5yYwQhb=M7vxbN08+oztwLg@mail.gmail.com>
	<4DCBA4B3.2070704@apache.org>
	<BANLkTimievMK7XMGYb5VZGcgHqj9aorMxQ@mail.gmail.com>
Date: Thu, 12 May 2011 20:02:05 +0200
Message-ID: <BANLkTik8SKbwsvN8Amp+xPvKEDA7nJk2vg@mail.gmail.com>
Subject: Re: OT: anyone else going to berlin buzzwords?
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I will try to determine a location nearby the buzz, make a reservation
and post to the appropriate lists.

  Bernd

On Thu, May 12, 2011 at 17:44, Ted Dunning <tdunning@maprtech.com> wrote:
> I will be there Monday-Thursday.
>
> Monday sounds good for food and beverages.
>
> On Thu, May 12, 2011 at 2:13 AM, Doug Cutting <cutting@apache.org> wrote:
>
>> I'll be there, but only for the conference on Monday and Tuesday.
>>
>> Doug
>>
>> On 05/11/2011 12:31 PM, Bernd Fondermann wrote:
>> > On Wed, May 11, 2011 at 11:08, Steve Loughran <stevel@apache.org> wrot=
e:
>> >> On 11/05/2011 00:53, Ian Holsman wrote:
>> >>>
>> >>> If so.. I'll be there.. let's catch up.
>> >>>
>> >>> http://www.berlinbuzzwords.de/
>> >>>
>> >>
>> >> 1. I'm going to be there
>> >
>> > 1.1.
>> > me too, but only for the conference on Mon + Tue.
>> > Who's available for a Hadoop/Apache/etc food + beverages on Monday
>> evening?
>> >
>> > =A0 Bernd
>>
>

From general-return-3497-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 18:13:31 2011
Return-Path: <general-return-3497-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 46C7C551F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 12 May 2011 18:13:31 +0000 (UTC)
Received: (qmail 85171 invoked by uid 500); 12 May 2011 18:13:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85108 invoked by uid 500); 12 May 2011 18:13:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85100 invoked by uid 99); 12 May 2011 18:13:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 18:13:29 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 18:13:24 +0000
Received: by vws7 with SMTP id 7so2102412vws.35
        for <general@hadoop.apache.org>; Thu, 12 May 2011 11:13:03 -0700 (PDT)
Received: by 10.52.112.69 with SMTP id io5mr724847vdb.94.1305223983045; Thu,
 12 May 2011 11:13:03 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.52.112.7 with HTTP; Thu, 12 May 2011 11:12:43 -0700 (PDT)
X-Originating-IP: [64.105.168.204]
In-Reply-To: <BANLkTik8SKbwsvN8Amp+xPvKEDA7nJk2vg@mail.gmail.com>
References: <B4C351A8-CACF-42C5-AE02-8334B8A95712@holsman.net>
 <4DCA520D.70405@apache.org> <BANLkTimAH6U5yYwQhb=M7vxbN08+oztwLg@mail.gmail.com>
 <4DCBA4B3.2070704@apache.org> <BANLkTimievMK7XMGYb5VZGcgHqj9aorMxQ@mail.gmail.com>
 <BANLkTik8SKbwsvN8Amp+xPvKEDA7nJk2vg@mail.gmail.com>
From: Ted Dunning <tdunning@maprtech.com>
Date: Thu, 12 May 2011 11:12:43 -0700
Message-ID: <BANLkTinLnjBcqQRxnBL8nf_GQXurAMn==g@mail.gmail.com>
Subject: Re: OT: anyone else going to berlin buzzwords?
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec548a9f1dc9c1104a3182383

--bcaec548a9f1dc9c1104a3182383
Content-Type: text/plain; charset=ISO-8859-1

Actually, I just heard from Isabel that there will be a Buzzwords barbecue
that evening.

Should we all just meet there and then adjourn for beers after dinner?

On Thu, May 12, 2011 at 11:02 AM, Bernd Fondermann <
bernd.fondermann@googlemail.com> wrote:

> I will try to determine a location nearby the buzz, make a reservation
> and post to the appropriate lists.
>
>  Bernd
>
> On Thu, May 12, 2011 at 17:44, Ted Dunning <tdunning@maprtech.com> wrote:
> > I will be there Monday-Thursday.
> >
> > Monday sounds good for food and beverages.
> >
> > On Thu, May 12, 2011 at 2:13 AM, Doug Cutting <cutting@apache.org>
> wrote:
> >
> >> I'll be there, but only for the conference on Monday and Tuesday.
> >>
> >> Doug
> >>
> >> On 05/11/2011 12:31 PM, Bernd Fondermann wrote:
> >> > On Wed, May 11, 2011 at 11:08, Steve Loughran <stevel@apache.org>
> wrote:
> >> >> On 11/05/2011 00:53, Ian Holsman wrote:
> >> >>>
> >> >>> If so.. I'll be there.. let's catch up.
> >> >>>
> >> >>> http://www.berlinbuzzwords.de/
> >> >>>
> >> >>
> >> >> 1. I'm going to be there
> >> >
> >> > 1.1.
> >> > me too, but only for the conference on Mon + Tue.
> >> > Who's available for a Hadoop/Apache/etc food + beverages on Monday
> >> evening?
> >> >
> >> >   Bernd
> >>
> >
>

--bcaec548a9f1dc9c1104a3182383--

From general-return-3498-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 19:48:48 2011
Return-Path: <general-return-3498-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 825E45513
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 12 May 2011 19:48:48 +0000 (UTC)
Received: (qmail 24832 invoked by uid 500); 12 May 2011 19:48:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24785 invoked by uid 500); 12 May 2011 19:48:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24777 invoked by uid 99); 12 May 2011 19:48:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 19:48:46 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of bernd.fondermann@googlemail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 19:48:41 +0000
Received: by bwz8 with SMTP id 8so2579255bwz.35
        for <general@hadoop.apache.org>; Thu, 12 May 2011 12:48:20 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=hrzdmYkGm0CF8TKAeg6lvX/jeb1GG0cFa0TbmvLCNUY=;
        b=e/It02Yd/qa1o99xooiLC18k9F6fS3WA91TWFI035/3K1Dop0ycelYNm5GLcXJuUmx
         /REl1SIuqkXrd8L1BxM1YvB7jna4Fo18svxR+bUhsftZas6J5xPOOcXHBFEMYUn4nzD9
         B3tmpQXH0vC/6rpG9odfKsDxQtAm995Tj9pl0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=xEtrT+8i4wmGp3GBtstQAEQBc0UaZx2Y7rtbNLjg++VaoWDNevsclkoC0zkpO2nB4/
         Ky6W8aZoZ4LClVCfRg0myhjBZaszbTcGPQf6GXYC9KnCDzufRJl5IOe3BMhfqPMVkEUc
         YYs05TtB2/3tNeoi1oIA+tryPktZiYVbzAukU=
MIME-Version: 1.0
Received: by 10.204.49.138 with SMTP id v10mr641947bkf.108.1305229700829; Thu,
 12 May 2011 12:48:20 -0700 (PDT)
Received: by 10.204.83.196 with HTTP; Thu, 12 May 2011 12:48:20 -0700 (PDT)
In-Reply-To: <BANLkTinLnjBcqQRxnBL8nf_GQXurAMn==g@mail.gmail.com>
References: <B4C351A8-CACF-42C5-AE02-8334B8A95712@holsman.net>
	<4DCA520D.70405@apache.org>
	<BANLkTimAH6U5yYwQhb=M7vxbN08+oztwLg@mail.gmail.com>
	<4DCBA4B3.2070704@apache.org>
	<BANLkTimievMK7XMGYb5VZGcgHqj9aorMxQ@mail.gmail.com>
	<BANLkTik8SKbwsvN8Amp+xPvKEDA7nJk2vg@mail.gmail.com>
	<BANLkTinLnjBcqQRxnBL8nf_GQXurAMn==g@mail.gmail.com>
Date: Thu, 12 May 2011 21:48:20 +0200
Message-ID: <BANLkTinVjfAfNmQLjMf7Qa74HOM5yGgqjA@mail.gmail.com>
Subject: Re: OT: anyone else going to berlin buzzwords?
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Thu, May 12, 2011 at 20:12, Ted Dunning <tdunning@maprtech.com> wrote:
> Actually, I just heard from Isabel that there will be a Buzzwords barbecue
> that evening.

Sorry about that, I checked the programme and pinged Isabel to make
sure we don't get in conflict with the conf.
And I definitively don't want to miss that so...

>
> Should we all just meet there and then adjourn for beers after dinner?

... +1

  Bernd

From general-return-3499-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 21:40:58 2011
Return-Path: <general-return-3499-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0797C5BDA
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 12 May 2011 21:40:58 +0000 (UTC)
Received: (qmail 75260 invoked by uid 500); 12 May 2011 21:40:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75207 invoked by uid 500); 12 May 2011 21:40:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75199 invoked by uid 99); 12 May 2011 21:40:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 21:40:56 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of hammer@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 21:40:49 +0000
Received: by bwz8 with SMTP id 8so2721226bwz.35
        for <general@hadoop.apache.org>; Thu, 12 May 2011 14:40:29 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.136.217 with SMTP id s25mr698544bkt.13.1305236428872; Thu,
 12 May 2011 14:40:28 -0700 (PDT)
Received: by 10.204.37.204 with HTTP; Thu, 12 May 2011 14:40:28 -0700 (PDT)
Date: Thu, 12 May 2011 14:40:28 -0700
Message-ID: <BANLkTinDNbnhWgFwX-v=mwjHz5j+7oXUNg@mail.gmail.com>
Subject: Apache Hadoop Hackathon: 5/18 in Palo Alto and San Francisco
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175cd8d2b0dd8a04a31b09ec
X-Virus-Checked: Checked by ClamAV on apache.org

--0015175cd8d2b0dd8a04a31b09ec
Content-Type: text/plain; charset=ISO-8859-1

Hey,

Thanks to everyone who came out for the Apache Hadoop Hackathon yesterday in
Palo Alto and San Francisco. We had 35 people sign up from a great cross
section of companies: Yahoo!, Cloudera, Facebook, Apple, Twitter,
Foursquare, AOL, Ngmoco, StumbleUpon, Trend Micro, Conviva, and more. We had
committers from the HBase, Hive, Pig, and Oozie projects ensuring their
projects work with the upcoming 0.22 release, and a number of folks got
their first patches up. The 0.22 branch is now being built by Jenkins and
we're in good shape to get a release candidate up soon.

We had so much fun that we're going to do it again!

Join us next Wednesday, May 18th, in either San Francisco or Palo Alto, and
help us continue the march to get feature development back on trunk. We'll
have a special guest in Palo Alto: Nigel Daley, Apache Hadoop PMC member and
the release manager for the 0.22 release. If you're a sysadmin or devops
person and want to help us build and test Apache Hadoop, this Hackathon will
be a great chance to learn about how it's done. Of course we'd also love to
have developers looking to get their first patches into Hadoop as well.

Sign up at http://hadoophackathon.eventbrite.com and we'll see you
Wednesday.

Regards,
Jeff

p.s. if you're looking to prepare for the Hackathon, check out the issues
tagged "newbie" at
https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&jqlQuery=labels+%3D+newbie
.

--0015175cd8d2b0dd8a04a31b09ec--

From general-return-3500-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 22:27:26 2011
Return-Path: <general-return-3500-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 177F45EED
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 12 May 2011 22:27:26 +0000 (UTC)
Received: (qmail 17143 invoked by uid 500); 12 May 2011 22:27:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17084 invoked by uid 500); 12 May 2011 22:27:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17076 invoked by uid 99); 12 May 2011 22:27:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 22:27:24 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 22:27:18 +0000
Received: by pve37 with SMTP id 37so1298716pve.35
        for <general@hadoop.apache.org>; Thu, 12 May 2011 15:26:56 -0700 (PDT)
Received: by 10.68.57.235 with SMTP id l11mr939135pbq.478.1305239216067; Thu,
 12 May 2011 15:26:56 -0700 (PDT)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.68.62.230 with HTTP; Thu, 12 May 2011 15:26:36 -0700 (PDT)
In-Reply-To: <BANLkTikCDLMRhoLLKhTfJEgc7dOCYHonpQ@mail.gmail.com>
References: <4DC91392.2010308@apache.org> <FDD1BE64-981D-4FA1-87BC-0FD4A880AB7A@yahoo-inc.com>
 <6407C0E3-934F-4C9D-9103-C738F6C1AED7@Holsman.NET> <BANLkTik9NeKPEuoPO6ijyzbyidweHDFKBg@mail.gmail.com>
 <BANLkTikCDLMRhoLLKhTfJEgc7dOCYHonpQ@mail.gmail.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Thu, 12 May 2011 15:26:36 -0700
X-Google-Sender-Auth: N7kw19DYEd9bq0SdDVD8DOVuLRQ
Message-ID: <BANLkTi=m3akJmgYxDNqNED4hiPgWEpS8nQ@mail.gmail.com>
Subject: Re: Defining Hadoop Compatibility -revisiting-
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

TCK (or JCK initially) was done as a tool to basically compare Java
Lang specs with a particular implementation including but not limited
to an extensive suite of say compiler tests.

So I assume before we can embark on any sort of HCK suite some formal
specs would have to be defined. It's rather hard to say that
implementation X is(not) compatible with Apache Hadoop for the lack of
API and spec level definition of what really comprise such an animal.

As was mentioned someplace else in the thread there's certain effort
happening to document DFS, MR, and Avro APIs. Seems like a very good
start for Hadoop specs at large.
--
=A0 Take care,
Konstantin (Cos) Boudnik

On Wed, May 11, 2011 at 16:20, Aaron Kimball <akimball83@gmail.com> wrote:
> What does it mean to "implement" those interfaces? I'm +1 for a TCK-based
> definition. In addition to statically implementing a set of interfaces, e=
ach
> interface also implicitly includes a set of acceptable inputs and predict=
ed
> outputs (or ranges of outputs) for those inputs.
>
> - Aaron
>
> On Wed, May 11, 2011 at 3:56 PM, Jacob R Rideout <apache@jacobrideout.net=
>wrote:
>
>> What about defining compatibility as fully implementing all the
>> public-stable annotated interfaces for a particular release?
>>
>> Jacob Rideout
>>
>> On Wed, May 11, 2011 at 4:42 PM, Ian Holsman <hadoop@holsman.net> wrote:
>> > For apache (httpd I'm assuming you mean). we define compatibility as
>> adherence to the set of RFC's that define the HTTP protocol.
>> >
>> > I'm no expert in this (Roy is though), but we could attempt to do
>> something similar when it comes to HDFS/Map-Reduce protocols. I'm not su=
re
>> what benefit there would be to going to a RFC, as opposed to documenting=
 the
>> API on our site.
>> >
>> >
>> > On May 12, 2011, at 7:24 AM, Eric Baldeschwieler wrote:
>> >
>> >> This is a really interesting topic! =A0I completely agree that we nee=
d to
>> get ahead of this.
>> >>
>> >> I would be really interested in learning of any experience other apac=
he
>> projects, such as apache or tomcat have with these issues.
>> >>
>> >> ---
>> >> E14 - typing on glass
>> >>
>> >> On May 10, 2011, at 6:31 AM, "Steve Loughran" <stevel@apache.org>
>> wrote:
>> >>
>> >>>
>> >>> Back in Jan 2011, I started a discussion about how to define Apache
>> >>> Hadoop Compatibility:
>> >>>
>> http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C4=
D46B6AD.2020802@apache.org%3E
>> >>>
>> >>> I am now reading EMC HD "Enterprise Ready" Apache Hadoop datasheet
>> >>>
>> >>>
>> http://www.greenplum.com/sites/default/files/EMC_Greenplum_HD_DS_Final_1=
.pdf
>> >>>
>> >>> It claims that their implementations are 100% compatible, even thoug=
h
>> >>> the Enterprise edition uses a C filesystem. It also claims that both
>> >>> their software releases contain "Certified Stacks", without defining
>> >>> what Certified means, or who does the certification -only that it is=
 an
>> >>> improvement.
>> >>>
>> >>>
>> >>> I think we should revisit this issue before people with their own
>> >>> agendas define what compatibility with Apache Hadoop is for us
>> >>>
>> >>>
>> >>> Licensing
>> >>> -Use of the Hadoop codebase must follow the Apache License
>> >>> http://www.apache.org/licenses/LICENSE-2.0
>> >>> -plug in components that are dynamically linked to (Filesystems and
>> >>> schedulers) don't appear to be derivative works on my reading of thi=
s,
>> >>>
>> >>> Naming
>> >>> -this is something for branding@apache, they will have their opinion=
s.
>> >>> The key one is that the name "Apache Hadoop" must get used, and it's
>> >>> important to make clear it is a derivative work.
>> >>> -I don't think you can claim to have a Distribution/Fork/Version of
>> >>> Apache Hadoop if you swap out big chunks of it for alternate
>> >>> filesystems, MR engines, etc. Some description of this is needed
>> >>> "Supports the Apache Hadoop MapReduce engine on top of Filesystem XY=
Z"
>> >>>
>> >>> Compatibility
>> >>> -the definition of the Hadoop interfaces and classes is the Apache
>> >>> Source tree,
>> >>> -the definition of semantics of the Hadoop interfaces and classes is
>> >>> the Apache Source tree, including the test classes.
>> >>> -the verification that the actual semantics of an Apache Hadoop
>> >>> release is compatible with the expected semantics is that current an=
d
>> >>> future tests pass
>> >>> -bug reports can highlight incompatibility with expectations of
>> >>> community users, and once incorporated into tests form part of the
>> >>> compatibility testing
>> >>> -vendors can claim and even certify their derivative works as
>> >>> compatible with other versions of their derivative works, but cannot
>> >>> claim compatibility with Apache Hadoop unless their code passes the
>> >>> tests and is consistent with the bug reports marked as ("by design")=
.
>> >>> Perhaps we should have tests that verify each of these "by design"
>> >>> bugreps to make them more formal.
>> >>>
>> >>> Certification
>> >>> -I have no idea what this means in EMC's case, they just say
>> "Certified"
>> >>> -As we don't do any certification ourselves, it would seem impossibl=
e
>> >>> for us to certify that any derivative work is compatible.
>> >>> -It may be best to state that nobody can certify their derivative as
>> >>> "compatible with Apache Hadoop" unless it passes all current test
>> suites
>> >>> -And require that anyone who declares compatibility define what they
>> >>> mean by this
>> >>>
>> >>> This is a good argument for getting more functional tests out there
>> >>> -whoever has more functional tests needs to get them into a test mod=
ule
>> >>> that can be used to test real deployments.
>> >>>
>> >
>> >
>>
>

From general-return-3501-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 22:30:46 2011
Return-Path: <general-return-3501-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 722D75296
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 12 May 2011 22:30:46 +0000 (UTC)
Received: (qmail 23230 invoked by uid 500); 12 May 2011 22:30:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23171 invoked by uid 500); 12 May 2011 22:30:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23163 invoked by uid 99); 12 May 2011 22:30:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 22:30:45 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.171] (HELO mail-px0-f171.google.com) (209.85.212.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 22:30:41 +0000
Received: by pxi7 with SMTP id 7so1686818pxi.2
        for <general@hadoop.apache.org>; Thu, 12 May 2011 15:30:20 -0700 (PDT)
Received: by 10.68.37.36 with SMTP id v4mr1098569pbj.76.1305239420037; Thu, 12
 May 2011 15:30:20 -0700 (PDT)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.68.62.230 with HTTP; Thu, 12 May 2011 15:30:00 -0700 (PDT)
In-Reply-To: <C9F15AFB.8D9C%mbhandarkar@linkedin.com>
References: <BANLkTimnQxosSzWVyE_sZMn-eJv_pT0jvA@mail.gmail.com> <C9F15AFB.8D9C%mbhandarkar@linkedin.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Thu, 12 May 2011 15:30:00 -0700
X-Google-Sender-Auth: 9kIhJTaHuYcKfOZPDk3CbMWkt-U
Message-ID: <BANLkTimOJAwc2Y9ixj2etCUnDe+9hfbi+g@mail.gmail.com>
Subject: Re: Defining Hadoop Compatibility -revisiting-
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Thu, May 12, 2011 at 09:45, Milind Bhandarkar
<mbhandarkar@linkedin.com> wrote:
> HCK and written specifications are not mutually exclusive. However, given
> the evolving nature of Hadoop APIs, functional tests need to evolve as

I would actually expand it to 'functional and system tests' because
latter are capable of validating inter-component iterations not
coverable by functional tests.

Cos

> well, and having them tied to a "current stable" version is easier to do
> than it is to tie the written specifications.
>
> - milind
>
> --
> Milind Bhandarkar
> mbhandarkar@linkedin.com
> +1-650-776-3167
>
>
>
>
>
>
> On 5/11/11 7:26 PM, "M. C. Srivas" <mcsrivas@gmail.com> wrote:
>
>>While the HCK is a great idea to check quickly if an implementation is
>>"compliant", =A0we still need a written specification to define what is
>>meant
>>by compliance, something akin to a set of RFC's, or a set of docs like th=
e
>> IEEE POSIX specifications.
>>
>>For example, the POSIX.1c pthreads API has a written document that
>>specifies
>>all the function calls, input params, return values, and error codes. It
>>clearly indicates what any POSIX-complaint threads package needs to
>>support,
>>and what are vendor-specific non-portable extensions that one can use at
>>one's own risk.
>>
>>Currently we have 2 sets of API =A0in the DFS and Map/Reduce layers, and =
the
>>specification is extracted only by looking at the code, or (where the cod=
e
>>is non-trivial) by writing really bizarre test programs to examine corner
>>cases. Further, the interaction between a mix of the old and new APIs is
>>not
>>specified anywhere. Such specifications are vitally important when
>>implementing libraries like Cascading, Mahout, etc. For example, an
>>application might open a file using the new API, and pass that stream
>>into a
>>library that manipulates the stream using some of the old API ... what is
>>then the expectation of the state of the stream when the library call
>>returns?
>>
>>Sanjay Radia @ Y! already started specifying some the DFS APIs to nail
>>such
>>things down. There's similar good effort in the Map/Reduce and =A0Avro
>>spaces,
>>but it seems to have stalled somewhat. We should continue it.
>>
>>Doing such specs would be a great service to the community and the users
>>of
>>Hadoop. It provides them
>> =A0 (a) clear-cut docs on how to use the Hadoop APIs
>> =A0 (b) wider choice of Hadoop implementations by freeing them from vend=
or
>>lock-in.
>>
>>Once we have such specification, the HCK becomes meaningful (since the HC=
K
>>itself will be buggy initially).
>>
>>
>>On Wed, May 11, 2011 at 2:46 PM, Milind Bhandarkar
>><mbhandarkar@linkedin.com
>>> wrote:
>>
>>> I think it's time to separate out functional tests as a "Hadoop
>>> Compatibility Kit (HCK)", similar to the Sun TCK for Java, but under AS=
L
>>> 2.0. Then "certification" would mean "Passes 100% of the HCK testsuite.=
"
>>>
>>> - milind
>>> --
>>> Milind Bhandarkar
>>> mbhandarkar@linkedin.com
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 5/11/11 2:24 PM, "Eric Baldeschwieler" <eric14@yahoo-inc.com> wrote:
>>>
>>> >This is a really interesting topic! =A0I completely agree that we need=
 to
>>> >get ahead of this.
>>> >
>>> >I would be really interested in learning of any experience other apach=
e
>>> >projects, such as apache or tomcat have with these issues.
>>> >
>>> >---
>>> >E14 - typing on glass
>>> >
>>> >On May 10, 2011, at 6:31 AM, "Steve Loughran" <stevel@apache.org>
>>>wrote:
>>> >
>>> >>
>>> >> Back in Jan 2011, I started a discussion about how to define Apache
>>> >> Hadoop Compatibility:
>>> >>
>>> >>
>>>
>>>http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C4=
D
>>> >>46B6AD.2020802@apache.org%3E
>>> >>
>>> >> I am now reading EMC HD "Enterprise Ready" Apache Hadoop datasheet
>>> >>
>>> >>
>>>
>>>>>http://www.greenplum.com/sites/default/files/EMC_Greenplum_HD_DS_Final=
_
>>>>>1
>>> .
>>> >>pdf
>>> >>
>>> >> It claims that their implementations are 100% compatible, even thoug=
h
>>> >> the Enterprise edition uses a C filesystem. It also claims that both
>>> >> their software releases contain "Certified Stacks", without defining
>>> >> what Certified means, or who does the certification -only that it is
>>>an
>>> >> improvement.
>>> >>
>>> >>
>>> >> I think we should revisit this issue before people with their own
>>> >> agendas define what compatibility with Apache Hadoop is for us
>>> >>
>>> >>
>>> >> Licensing
>>> >> -Use of the Hadoop codebase must follow the Apache License
>>> >> http://www.apache.org/licenses/LICENSE-2.0
>>> >> -plug in components that are dynamically linked to (Filesystems and
>>> >> schedulers) don't appear to be derivative works on my reading of
>>>this,
>>> >>
>>> >> Naming
>>> >> =A0-this is something for branding@apache, they will have their
>>>opinions.
>>> >> The key one is that the name "Apache Hadoop" must get used, and it's
>>> >> important to make clear it is a derivative work.
>>> >> =A0-I don't think you can claim to have a Distribution/Fork/Version =
of
>>> >> Apache Hadoop if you swap out big chunks of it for alternate
>>> >> filesystems, MR engines, etc. Some description of this is needed
>>> >> "Supports the Apache Hadoop MapReduce engine on top of Filesystem
>>>XYZ"
>>> >>
>>> >> Compatibility
>>> >> =A0-the definition of the Hadoop interfaces and classes is the Apach=
e
>>> >> Source tree,
>>> >> =A0-the definition of semantics of the Hadoop interfaces and classes=
 is
>>> >> the Apache Source tree, including the test classes.
>>> >> =A0-the verification that the actual semantics of an Apache Hadoop
>>> >> release is compatible with the expected semantics is that current an=
d
>>> >> future tests pass
>>> >> =A0-bug reports can highlight incompatibility with expectations of
>>> >> community users, and once incorporated into tests form part of the
>>> >> compatibility testing
>>> >> =A0-vendors can claim and even certify their derivative works as
>>> >> compatible with other versions of their derivative works, but cannot
>>> >> claim compatibility with Apache Hadoop unless their code passes the
>>> >> tests and is consistent with the bug reports marked as ("by design")=
.
>>> >> Perhaps we should have tests that verify each of these "by design"
>>> >> bugreps to make them more formal.
>>> >>
>>> >> Certification
>>> >> =A0-I have no idea what this means in EMC's case, they just say
>>> >>"Certified"
>>> >> =A0-As we don't do any certification ourselves, it would seem
>>>impossible
>>> >> for us to certify that any derivative work is compatible.
>>> >> =A0-It may be best to state that nobody can certify their derivative=
 as
>>> >> "compatible with Apache Hadoop" unless it passes all current test
>>>suites
>>> >> =A0-And require that anyone who declares compatibility define what t=
hey
>>> >> mean by this
>>> >>
>>> >> This is a good argument for getting more functional tests out there
>>> >> -whoever has more functional tests needs to get them into a test
>>>module
>>> >> that can be used to test real deployments.
>>> >>
>>>
>>>
>
>

From general-return-3502-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 03:34:45 2011
Return-Path: <general-return-3502-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AE181499B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 03:34:45 +0000 (UTC)
Received: (qmail 59274 invoked by uid 500); 13 May 2011 03:34:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59127 invoked by uid 500); 13 May 2011 03:34:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59118 invoked by uid 99); 13 May 2011 03:34:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 03:34:43 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mbhandarkar@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 03:34:36 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:In-Reply-To:Accept-Language:
   Content-Language:X-MS-Has-Attach:X-MS-TNEF-Correlator:
   x-originating-ip:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version;
  b=ihr9hJWhJZ74DBwgFQctINTcjFKwnloOIdtiM8DuuL9rH3onMPZ7GjCt
   WXEd6L3kjYPSkD6IYaEk1ltRlHjycwnInPPt6wlBgZ3UPpANhVyQUwRlj
   i6SIk6y1XW3bEKq;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=mbhandarkar@linkedin.com; q=dns/txt;
  s=proddkim; t=1305257675; x=1336793675;
  h=from:to:subject:date:message-id:in-reply-to:content-id:
   content-transfer-encoding:mime-version;
  bh=q/ZZnOecBVpEy5URypeSzRn2UzbDBEEYvMzdwhfkZbg=;
  b=Ii1GDeldo29DgClPJ0L1otukVIaB//7f8smPRsiu7QDja+QKLYcyC6oj
   xOzrsWX91+n+ajgPgFE3J5ZnPjk8YSOXd1GrepSD1WAF7tp8mefN6WtVR
   +Mad92C9oARjUS6;
X-IronPort-AV: E=Sophos;i="4.64,362,1301900400"; 
   d="scan'208";a="23147751"
Received: from ESV4-EXC02.linkedin.biz ([fe80::4d74:48bd:e0bd:13ee]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi id 14.01.0251.000; Thu,
 12 May 2011 20:37:50 -0700
From: Milind Bhandarkar <mbhandarkar@linkedin.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: Defining Hadoop Compatibility -revisiting-
Thread-Topic: Defining Hadoop Compatibility -revisiting-
Thread-Index: AQHMEPOnIB+Y40jJf0ahsOjyQf2X+pSKG1eA
Date: Fri, 13 May 2011 03:37:49 +0000
Message-ID: <C9F1F17B.8EF3%mbhandarkar@linkedin.com>
In-Reply-To: <BANLkTi=m3akJmgYxDNqNED4hiPgWEpS8nQ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <6C27115C4549904298D782B8530DB158@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

The problem with (only) specs is that they are written in natural
language, and subject to human interpretation, and since humans are bad at
natural language interpretation, this gives rise to something called
standards bodies and lawyers, and that has never been good for anyone in
the past ;-)

Now consider this scenario:

$ bin/hadoop jar hck-0.20.2.jar --config <myconfig/dir>
... Bunch of output ...
Result: Tests run: 1000, Successful: 999 Failed: 1

This is much easier to interpret, even for humans.

The intention of formally defining compatibility is so that the programs
written for Apache Hadoop run unmodified for other open-source /
closed-source systems that claim to be "Apache Hadoop Compatible". Unless
it can be verified easily, the compatibility definition has no meaning.
So, standards that are only documented are useless.

By the way, one should also define "Apache Hadoop Source Compatible", and
"Apache Hadoop Binary Compatible", depending on whether one recompiles
src/hck/**.java and rebuilds hck.jar or not.

- milind

--=20
Milind Bhandarkar
mbhandarkar@linkedin.com
+1-650-776-3167






On 5/12/11 3:26 PM, "Konstantin Boudnik" <cos@apache.org> wrote:

>TCK (or JCK initially) was done as a tool to basically compare Java
>Lang specs with a particular implementation including but not limited
>to an extensive suite of say compiler tests.
>
>So I assume before we can embark on any sort of HCK suite some formal
>specs would have to be defined. It's rather hard to say that
>implementation X is(not) compatible with Apache Hadoop for the lack of
>API and spec level definition of what really comprise such an animal.
>
>As was mentioned someplace else in the thread there's certain effort
>happening to document DFS, MR, and Avro APIs. Seems like a very good
>start for Hadoop specs at large.
>--
>  Take care,
>Konstantin (Cos) Boudnik
>
>On Wed, May 11, 2011 at 16:20, Aaron Kimball <akimball83@gmail.com> wrote:
>> What does it mean to "implement" those interfaces? I'm +1 for a
>>TCK-based
>> definition. In addition to statically implementing a set of interfaces,
>>each
>> interface also implicitly includes a set of acceptable inputs and
>>predicted
>> outputs (or ranges of outputs) for those inputs.
>>
>> - Aaron
>>
>> On Wed, May 11, 2011 at 3:56 PM, Jacob R Rideout
>><apache@jacobrideout.net>wrote:
>>
>>> What about defining compatibility as fully implementing all the
>>> public-stable annotated interfaces for a particular release?
>>>
>>> Jacob Rideout
>>>
>>> On Wed, May 11, 2011 at 4:42 PM, Ian Holsman <hadoop@holsman.net>
>>>wrote:
>>> > For apache (httpd I'm assuming you mean). we define compatibility as
>>> adherence to the set of RFC's that define the HTTP protocol.
>>> >
>>> > I'm no expert in this (Roy is though), but we could attempt to do
>>> something similar when it comes to HDFS/Map-Reduce protocols. I'm not
>>>sure
>>> what benefit there would be to going to a RFC, as opposed to
>>>documenting the
>>> API on our site.
>>> >
>>> >
>>> > On May 12, 2011, at 7:24 AM, Eric Baldeschwieler wrote:
>>> >
>>> >> This is a really interesting topic!  I completely agree that we
>>>need to
>>> get ahead of this.
>>> >>
>>> >> I would be really interested in learning of any experience other
>>>apache
>>> projects, such as apache or tomcat have with these issues.
>>> >>
>>> >> ---
>>> >> E14 - typing on glass
>>> >>
>>> >> On May 10, 2011, at 6:31 AM, "Steve Loughran" <stevel@apache.org>
>>> wrote:
>>> >>
>>> >>>
>>> >>> Back in Jan 2011, I started a discussion about how to define Apache
>>> >>> Hadoop Compatibility:
>>> >>>
>>>=20
>>>http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C4
>>>D46B6AD.2020802@apache.org%3E
>>> >>>
>>> >>> I am now reading EMC HD "Enterprise Ready" Apache Hadoop datasheet
>>> >>>
>>> >>>
>>>=20
>>>http://www.greenplum.com/sites/default/files/EMC_Greenplum_HD_DS_Final_1
>>>.pdf
>>> >>>
>>> >>> It claims that their implementations are 100% compatible, even
>>>though
>>> >>> the Enterprise edition uses a C filesystem. It also claims that
>>>both
>>> >>> their software releases contain "Certified Stacks", without
>>>defining
>>> >>> what Certified means, or who does the certification -only that it
>>>is an
>>> >>> improvement.
>>> >>>
>>> >>>
>>> >>> I think we should revisit this issue before people with their own
>>> >>> agendas define what compatibility with Apache Hadoop is for us
>>> >>>
>>> >>>
>>> >>> Licensing
>>> >>> -Use of the Hadoop codebase must follow the Apache License
>>> >>> http://www.apache.org/licenses/LICENSE-2.0
>>> >>> -plug in components that are dynamically linked to (Filesystems and
>>> >>> schedulers) don't appear to be derivative works on my reading of
>>>this,
>>> >>>
>>> >>> Naming
>>> >>> -this is something for branding@apache, they will have their
>>>opinions.
>>> >>> The key one is that the name "Apache Hadoop" must get used, and
>>>it's
>>> >>> important to make clear it is a derivative work.
>>> >>> -I don't think you can claim to have a Distribution/Fork/Version of
>>> >>> Apache Hadoop if you swap out big chunks of it for alternate
>>> >>> filesystems, MR engines, etc. Some description of this is needed
>>> >>> "Supports the Apache Hadoop MapReduce engine on top of Filesystem
>>>XYZ"
>>> >>>
>>> >>> Compatibility
>>> >>> -the definition of the Hadoop interfaces and classes is the Apache
>>> >>> Source tree,
>>> >>> -the definition of semantics of the Hadoop interfaces and classes
>>>is
>>> >>> the Apache Source tree, including the test classes.
>>> >>> -the verification that the actual semantics of an Apache Hadoop
>>> >>> release is compatible with the expected semantics is that current
>>>and
>>> >>> future tests pass
>>> >>> -bug reports can highlight incompatibility with expectations of
>>> >>> community users, and once incorporated into tests form part of the
>>> >>> compatibility testing
>>> >>> -vendors can claim and even certify their derivative works as
>>> >>> compatible with other versions of their derivative works, but
>>>cannot
>>> >>> claim compatibility with Apache Hadoop unless their code passes the
>>> >>> tests and is consistent with the bug reports marked as ("by
>>>design").
>>> >>> Perhaps we should have tests that verify each of these "by design"
>>> >>> bugreps to make them more formal.
>>> >>>
>>> >>> Certification
>>> >>> -I have no idea what this means in EMC's case, they just say
>>> "Certified"
>>> >>> -As we don't do any certification ourselves, it would seem
>>>impossible
>>> >>> for us to certify that any derivative work is compatible.
>>> >>> -It may be best to state that nobody can certify their derivative
>>>as
>>> >>> "compatible with Apache Hadoop" unless it passes all current test
>>> suites
>>> >>> -And require that anyone who declares compatibility define what
>>>they
>>> >>> mean by this
>>> >>>
>>> >>> This is a good argument for getting more functional tests out there
>>> >>> -whoever has more functional tests needs to get them into a test
>>>module
>>> >>> that can be used to test real deployments.
>>> >>>
>>> >
>>> >
>>>
>>


From general-return-3503-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 03:37:37 2011
Return-Path: <general-return-3503-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A904549FA
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 03:37:37 +0000 (UTC)
Received: (qmail 61886 invoked by uid 500); 13 May 2011 03:37:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61841 invoked by uid 500); 13 May 2011 03:37:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61833 invoked by uid 99); 13 May 2011 03:37:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 03:37:35 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mbhandarkar@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 03:37:30 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:In-Reply-To:Accept-Language:
   Content-Language:X-MS-Has-Attach:X-MS-TNEF-Correlator:
   x-originating-ip:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version;
  b=tjs713dLyka8YNkeSTjtKvr5cHLH0ySCina9kODTx82WeVdQ9diH1x2G
   76d6f5BJtzbZ3wjV1vrS84hhtqP+Qd6uEigSaQkYt2/bfBGbW36KLkCLk
   zP1zS1vbi7cscWX;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=mbhandarkar@linkedin.com; q=dns/txt;
  s=proddkim; t=1305257850; x=1336793850;
  h=from:to:subject:date:message-id:in-reply-to:content-id:
   content-transfer-encoding:mime-version;
  bh=ag5AkGY4reBwm6Yr333odYmT7RQU5cWGPJlnyHIwQXs=;
  b=bYXLChJCapZ4ObhqOQA8qxkg0FeToOFheNzBItSLVmwn0hmawGcPVG5F
   0ikuuL88dkYMnhu4YllbRhN7uyHAvGjAnonN8B/QoZdTSXFuvoRpeSQVN
   4Crm9YVOcvDEQ0g;
X-IronPort-AV: E=Sophos;i="4.64,362,1301900400"; 
   d="scan'208";a="23147791"
Received: from ESV4-EXC02.linkedin.biz ([fe80::4d74:48bd:e0bd:13ee]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi id 14.01.0251.000; Thu,
 12 May 2011 20:40:46 -0700
From: Milind Bhandarkar <mbhandarkar@linkedin.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: Defining Hadoop Compatibility -revisiting-
Thread-Topic: Defining Hadoop Compatibility -revisiting-
Thread-Index: AQHMDv3EIB+Y40jJf0ahsOjyQf2X+pSImxGA//+PqQCAAMSEgIAAedIAgADWjwD//+B3AA==
Date: Fri, 13 May 2011 03:40:46 +0000
Message-ID: <C9F1F4D2.8F11%mbhandarkar@linkedin.com>
In-Reply-To: <BANLkTimOJAwc2Y9ixj2etCUnDe+9hfbi+g@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <9A8C926288334646A358E8450C86792B@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Cos,

Can you give me an example of a "system test" that is not a functional
test ? My assumption was that the functionality being tested is specific
to a component, and that inter-component interactions (that's what you
meant, right?) would be taken care by the public interface and semantics
of a component API.

- milind

--=20
Milind Bhandarkar
mbhandarkar@linkedin.com
+1-650-776-3167






On 5/12/11 3:30 PM, "Konstantin Boudnik" <cos@apache.org> wrote:

>On Thu, May 12, 2011 at 09:45, Milind Bhandarkar
><mbhandarkar@linkedin.com> wrote:
>> HCK and written specifications are not mutually exclusive. However,
>>given
>> the evolving nature of Hadoop APIs, functional tests need to evolve as
>
>I would actually expand it to 'functional and system tests' because
>latter are capable of validating inter-component iterations not
>coverable by functional tests.
>
>Cos
>
>> well, and having them tied to a "current stable" version is easier to do
>> than it is to tie the written specifications.
>>
>> - milind
>>
>> --
>> Milind Bhandarkar
>> mbhandarkar@linkedin.com
>> +1-650-776-3167
>>
>>
>>
>>
>>
>>
>> On 5/11/11 7:26 PM, "M. C. Srivas" <mcsrivas@gmail.com> wrote:
>>
>>>While the HCK is a great idea to check quickly if an implementation is
>>>"compliant",  we still need a written specification to define what is
>>>meant
>>>by compliance, something akin to a set of RFC's, or a set of docs like
>>>the
>>> IEEE POSIX specifications.
>>>
>>>For example, the POSIX.1c pthreads API has a written document that
>>>specifies
>>>all the function calls, input params, return values, and error codes. It
>>>clearly indicates what any POSIX-complaint threads package needs to
>>>support,
>>>and what are vendor-specific non-portable extensions that one can use at
>>>one's own risk.
>>>
>>>Currently we have 2 sets of API  in the DFS and Map/Reduce layers, and
>>>the
>>>specification is extracted only by looking at the code, or (where the
>>>code
>>>is non-trivial) by writing really bizarre test programs to examine
>>>corner
>>>cases. Further, the interaction between a mix of the old and new APIs is
>>>not
>>>specified anywhere. Such specifications are vitally important when
>>>implementing libraries like Cascading, Mahout, etc. For example, an
>>>application might open a file using the new API, and pass that stream
>>>into a
>>>library that manipulates the stream using some of the old API ... what
>>>is
>>>then the expectation of the state of the stream when the library call
>>>returns?
>>>
>>>Sanjay Radia @ Y! already started specifying some the DFS APIs to nail
>>>such
>>>things down. There's similar good effort in the Map/Reduce and  Avro
>>>spaces,
>>>but it seems to have stalled somewhat. We should continue it.
>>>
>>>Doing such specs would be a great service to the community and the users
>>>of
>>>Hadoop. It provides them
>>>   (a) clear-cut docs on how to use the Hadoop APIs
>>>   (b) wider choice of Hadoop implementations by freeing them from
>>>vendor
>>>lock-in.
>>>
>>>Once we have such specification, the HCK becomes meaningful (since the
>>>HCK
>>>itself will be buggy initially).
>>>
>>>
>>>On Wed, May 11, 2011 at 2:46 PM, Milind Bhandarkar
>>><mbhandarkar@linkedin.com
>>>> wrote:
>>>
>>>> I think it's time to separate out functional tests as a "Hadoop
>>>> Compatibility Kit (HCK)", similar to the Sun TCK for Java, but under
>>>>ASL
>>>> 2.0. Then "certification" would mean "Passes 100% of the HCK
>>>>testsuite."
>>>>
>>>> - milind
>>>> --
>>>> Milind Bhandarkar
>>>> mbhandarkar@linkedin.com
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 5/11/11 2:24 PM, "Eric Baldeschwieler" <eric14@yahoo-inc.com>
>>>>wrote:
>>>>
>>>> >This is a really interesting topic!  I completely agree that we need
>>>>to
>>>> >get ahead of this.
>>>> >
>>>> >I would be really interested in learning of any experience other
>>>>apache
>>>> >projects, such as apache or tomcat have with these issues.
>>>> >
>>>> >---
>>>> >E14 - typing on glass
>>>> >
>>>> >On May 10, 2011, at 6:31 AM, "Steve Loughran" <stevel@apache.org>
>>>>wrote:
>>>> >
>>>> >>
>>>> >> Back in Jan 2011, I started a discussion about how to define Apache
>>>> >> Hadoop Compatibility:
>>>> >>
>>>> >>
>>>>
>>>>http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C
>>>>4D
>>>> >>46B6AD.2020802@apache.org%3E
>>>> >>
>>>> >> I am now reading EMC HD "Enterprise Ready" Apache Hadoop datasheet
>>>> >>
>>>> >>
>>>>
>>>>>>http://www.greenplum.com/sites/default/files/EMC_Greenplum_HD_DS_Fina
>>>>>>l_
>>>>>>1
>>>> .
>>>> >>pdf
>>>> >>
>>>> >> It claims that their implementations are 100% compatible, even
>>>>though
>>>> >> the Enterprise edition uses a C filesystem. It also claims that
>>>>both
>>>> >> their software releases contain "Certified Stacks", without
>>>>defining
>>>> >> what Certified means, or who does the certification -only that it
>>>>is
>>>>an
>>>> >> improvement.
>>>> >>
>>>> >>
>>>> >> I think we should revisit this issue before people with their own
>>>> >> agendas define what compatibility with Apache Hadoop is for us
>>>> >>
>>>> >>
>>>> >> Licensing
>>>> >> -Use of the Hadoop codebase must follow the Apache License
>>>> >> http://www.apache.org/licenses/LICENSE-2.0
>>>> >> -plug in components that are dynamically linked to (Filesystems and
>>>> >> schedulers) don't appear to be derivative works on my reading of
>>>>this,
>>>> >>
>>>> >> Naming
>>>> >>  -this is something for branding@apache, they will have their
>>>>opinions.
>>>> >> The key one is that the name "Apache Hadoop" must get used, and
>>>>it's
>>>> >> important to make clear it is a derivative work.
>>>> >>  -I don't think you can claim to have a Distribution/Fork/Version
>>>>of
>>>> >> Apache Hadoop if you swap out big chunks of it for alternate
>>>> >> filesystems, MR engines, etc. Some description of this is needed
>>>> >> "Supports the Apache Hadoop MapReduce engine on top of Filesystem
>>>>XYZ"
>>>> >>
>>>> >> Compatibility
>>>> >>  -the definition of the Hadoop interfaces and classes is the Apache
>>>> >> Source tree,
>>>> >>  -the definition of semantics of the Hadoop interfaces and classes
>>>>is
>>>> >> the Apache Source tree, including the test classes.
>>>> >>  -the verification that the actual semantics of an Apache Hadoop
>>>> >> release is compatible with the expected semantics is that current
>>>>and
>>>> >> future tests pass
>>>> >>  -bug reports can highlight incompatibility with expectations of
>>>> >> community users, and once incorporated into tests form part of the
>>>> >> compatibility testing
>>>> >>  -vendors can claim and even certify their derivative works as
>>>> >> compatible with other versions of their derivative works, but
>>>>cannot
>>>> >> claim compatibility with Apache Hadoop unless their code passes the
>>>> >> tests and is consistent with the bug reports marked as ("by
>>>>design").
>>>> >> Perhaps we should have tests that verify each of these "by design"
>>>> >> bugreps to make them more formal.
>>>> >>
>>>> >> Certification
>>>> >>  -I have no idea what this means in EMC's case, they just say
>>>> >>"Certified"
>>>> >>  -As we don't do any certification ourselves, it would seem
>>>>impossible
>>>> >> for us to certify that any derivative work is compatible.
>>>> >>  -It may be best to state that nobody can certify their derivative
>>>>as
>>>> >> "compatible with Apache Hadoop" unless it passes all current test
>>>>suites
>>>> >>  -And require that anyone who declares compatibility define what
>>>>they
>>>> >> mean by this
>>>> >>
>>>> >> This is a good argument for getting more functional tests out there
>>>> >> -whoever has more functional tests needs to get them into a test
>>>>module
>>>> >> that can be used to test real deployments.
>>>> >>
>>>>
>>>>
>>
>>


From general-return-3504-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 04:06:29 2011
Return-Path: <general-return-3504-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8FEC34788
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 04:06:29 +0000 (UTC)
Received: (qmail 81174 invoked by uid 500); 13 May 2011 04:06:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81090 invoked by uid 500); 13 May 2011 04:06:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81078 invoked by uid 99); 13 May 2011 04:06:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 04:06:26 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.220.176] (HELO mail-vx0-f176.google.com) (209.85.220.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 04:06:21 +0000
Received: by vxa37 with SMTP id 37so2557502vxa.35
        for <general@hadoop.apache.org>; Thu, 12 May 2011 21:06:00 -0700 (PDT)
Received: by 10.52.112.69 with SMTP id io5mr1371632vdb.94.1305259560049; Thu,
 12 May 2011 21:06:00 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.52.112.7 with HTTP; Thu, 12 May 2011 21:05:40 -0700 (PDT)
X-Originating-IP: [67.160.196.149]
In-Reply-To: <C9F1F17B.8EF3%mbhandarkar@linkedin.com>
References: <BANLkTi=m3akJmgYxDNqNED4hiPgWEpS8nQ@mail.gmail.com> <C9F1F17B.8EF3%mbhandarkar@linkedin.com>
From: Ted Dunning <tdunning@maprtech.com>
Date: Thu, 12 May 2011 21:05:40 -0700
Message-ID: <BANLkTi=zFtSowx2vtDxMFWCSQxp0-GD8fA@mail.gmail.com>
Subject: Re: Defining Hadoop Compatibility -revisiting-
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec548a9f16a9cdf04a3206c74

--bcaec548a9f16a9cdf04a3206c74
Content-Type: text/plain; charset=ISO-8859-1

Did anybody propose natural language only specifications?

On Thu, May 12, 2011 at 8:37 PM, Milind Bhandarkar <mbhandarkar@linkedin.com
> wrote:

> The problem with (only) specs is that they are written in natural
> language, and subject to human interpretation,
>

--bcaec548a9f16a9cdf04a3206c74--

From general-return-3505-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 04:49:17 2011
Return-Path: <general-return-3505-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 49C754B66
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 04:49:17 +0000 (UTC)
Received: (qmail 12157 invoked by uid 500); 13 May 2011 04:49:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12119 invoked by uid 500); 13 May 2011 04:49:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12111 invoked by uid 99); 13 May 2011 04:49:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 04:49:14 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mbhandarkar@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 04:49:08 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:In-Reply-To:Accept-Language:
   Content-Language:X-MS-Has-Attach:X-MS-TNEF-Correlator:
   x-originating-ip:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version;
  b=rnqw7eRXfyLdHXj2sS72eelL381jlBsnXXC1WtCbGxHAJjzx8bYQhAU9
   c7+R7a7J58O2Ksw5A344f1JeWg3+Y+iWONQ0b3M7GKxW1vx2lI3ndte+B
   3/kCzDFCrgcwkfh;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=mbhandarkar@linkedin.com; q=dns/txt;
  s=proddkim; t=1305262148; x=1336798148;
  h=from:to:subject:date:message-id:in-reply-to:content-id:
   content-transfer-encoding:mime-version;
  bh=eVXy9Lbp5iPjpwGm3KJsReLQUTzq4lGb2zGVGBN6gyM=;
  b=hfym8Eb/8AYWSoW4MXQbdkFM48ZFaaJML9KmXc8k/UgelUeylcghUvkh
   xhzmMOzTgDPA5ilRrXr49jlY9U4GYAHRwY7H1RjnswMWD/gyOXOneKWqb
   ILoGas8uqekwTR2;
X-IronPort-AV: E=Sophos;i="4.64,362,1301900400"; 
   d="scan'208";a="23148751"
Received: from ESV4-EXC02.linkedin.biz ([fe80::4d74:48bd:e0bd:13ee]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi id 14.01.0251.000; Thu,
 12 May 2011 21:52:23 -0700
From: Milind Bhandarkar <mbhandarkar@linkedin.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: Defining Hadoop Compatibility -revisiting-
Thread-Topic: Defining Hadoop Compatibility -revisiting-
Thread-Index: AQHMEPOnIB+Y40jJf0ahsOjyQf2X+pSKG1eAgAB+JQD//5axgA==
Date: Fri, 13 May 2011 04:52:23 +0000
Message-ID: <C9F20588.8F34%mbhandarkar@linkedin.com>
In-Reply-To: <BANLkTi=zFtSowx2vtDxMFWCSQxp0-GD8fA@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <9C7DFA44E173F1489918182DF0B03FD3@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Ok, my mistake. They have only asked for documented specifications. I may
have been influenced by all the specifications I have read. All of them
were in English, which is characterized as a natural language.

But then, if you are proposing a specification in a non-natural-language,
isn't that called a test suite ? Or is there a middle ground ?

- milind

--=20
Milind Bhandarkar
mbhandarkar@linkedin.com
+1-650-776-3167






On 5/12/11 9:05 PM, "Ted Dunning" <tdunning@maprtech.com> wrote:

>Did anybody propose natural language only specifications?
>
>On Thu, May 12, 2011 at 8:37 PM, Milind Bhandarkar
><mbhandarkar@linkedin.com
>> wrote:
>
>> The problem with (only) specs is that they are written in natural
>> language, and subject to human interpretation,
>>


From general-return-3506-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 05:06:42 2011
Return-Path: <general-return-3506-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 91F2F42AB
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 05:06:42 +0000 (UTC)
Received: (qmail 20483 invoked by uid 500); 13 May 2011 05:06:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20323 invoked by uid 500); 13 May 2011 05:06:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 20315 invoked by uid 99); 13 May 2011 05:06:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 05:06:36 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 05:06:32 +0000
Received: from [10.0.1.3] (snvvpn4-10-72-168-c47.hq.corp.yahoo.com [10.72.168.47])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p4D55v0c094242
	for <general@hadoop.apache.org>; Thu, 12 May 2011 22:05:58 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1305263158;
	bh=1z/vHT4/lHSjffGhTvnc6UbANmEBMm3c75nmeg20jOs=;
	h=Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:
	 Content-Transfer-Encoding:Message-Id:References:To;
	b=xduOpalmKQ0LWG1ctNUev8SGRd1x2U7yWr+/jw0R3FshuVMgFNEM6gdtD/AJ3rTVn
	 sl3byinB9O/9gYpMursDi43MKmiGAA/k1ejFitbOvOMfrQQZis1YKcxtX1LJhROtzC
	 qpiXEE4KaqLSaaYVdi16nqa8XsZ1yhGq2D4C71YI=
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: Defining Hadoop Compatibility -revisiting-
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <4DCBA93E.2050005@apache.org>
Date: Thu, 12 May 2011 22:05:56 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <205D1C0D-975B-4497-AAF2-F64684665241@yahoo-inc.com>
References: <FDD1BE64-981D-4FA1-87BC-0FD4A880AB7A@yahoo-inc.com> <C9F04FDB.8C82%mbhandarkar@linkedin.com> <BANLkTimnQxosSzWVyE_sZMn-eJv_pT0jvA@mail.gmail.com> <4DCBA93E.2050005@apache.org>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1084)

label:
print "+1";
goto label;

I could not agree more with everything you said steve!  The Apache =
Hadoop project should own the definition of Apache Hadoop.  Hadoop is =
far from done.  The interfaces need to keep evolving to get to a place =
where we can be proud of them.

I support "vendors" building replacement components for Apache Hadoop =
components.  That will benefit the community, give folks choices and =
challenge us to make Apache Hadoop even better.  I think it is critical =
that Apache Hadoop remain a living / evolving work that is driven by =
those who are willing to contribute their work to it and that the result =
of that evolution is the reference implementation that vendors must =
match & exceed to play.

I'd love to see more effort to add specifications and compatibility =
tests to Apache Hadoop.  We'll continue to invest in specs and see what =
we can do about tests.  I  encourage folks who wish to demonstrate =
compatibility and use the Apache Hadoop trademark with their products to =
help contribute such work to Apache Hadoop.  We should include these =
things with the code under SVN with our normal patch peer review.

On May 12, 2011, at 2:32 AM, Steve Loughran wrote:

> On 12/05/2011 03:26, M. C. Srivas wrote:
>> While the HCK is a great idea to check quickly if an implementation =
is
>> "compliant",  we still need a written specification to define what is =
meant
>> by compliance, something akin to a set of RFC's, or a set of docs =
like the
>>  IEEE POSIX specifications.
>>=20
>> For example, the POSIX.1c pthreads API has a written document that =
specifies
>> all the function calls, input params, return values, and error codes. =
It
>> clearly indicates what any POSIX-complaint threads package needs to =
support,
>> and what are vendor-specific non-portable extensions that one can use =
at
>> one's own risk.
>=20
> I have been known to be critical of standards bodies in the past
> http://www.waterfall2006.com/loughran.html
>=20
> And I've been in them. It is absolutely essential that the Hadoop =
stack=20
> doesn't become controlled by a standards body, as then you become=20
> controlled by whoever can afford to send the most people to the=20
> standards events -and make behind the scenes deals with others to get=20=

> votes through.
>=20
>> Currently we have 2 sets of API  in the DFS and Map/Reduce layers, =
and the
>> specification is extracted only by looking at the code, or (where the =
code
>> is non-trivial) by writing really bizarre test programs to examine =
corner
>> cases. Further, the interaction between a mix of the old and new APIs =
is not
>> specified anywhere. Such specifications are vitally important when
>> implementing libraries like Cascading, Mahout, etc. For example, an
>> application might open a file using the new API, and pass that stream =
into a
>> library that manipulates the stream using some of the old API ... =
what is
>> then the expectation of the state of the stream when the library call
>> returns?
>>=20
>> Sanjay Radia @ Y! already started specifying some the DFS APIs to =
nail such
>> things down. There's similar good effort in the Map/Reduce and  Avro =
spaces,
>> but it seems to have stalled somewhat. We should continue it.
>>=20
>> Doing such specs would be a great service to the community and the =
users of
>> Hadoop. It provides them
>>    (a) clear-cut docs on how to use the Hadoop APIs#
>=20
> +1
>=20
>>    (b) wider choice of Hadoop implementations by freeing them from =
vendor
>> lock-in.
>=20
> =3D0
>=20
> They won't be hadoop implementations, they will be "something that is=20=

> compatible with the Apache Hadoop API as defined in v 0.x of the =
Hadoop=20
> compatibility kit". Furthermore, there's the issue of any google =
patents=20
> -while google have given Hadoop permission to them, that may not apply=20=

> to other things that implement compatible APIs.
>=20
> I also think that the Hadoop team need to be the one's who own the=20
> interfaces and tests, define the tests as a functional test suite for=20=

> testing Hadoop distributions, and reserve the right to make changes to=20=

> the interfaces, semantics and tests as suits the teams needs. The =
input=20
> from others -especially related community projects- are important, =
but,=20
> to be ruthless, the compatibility issues with things that aren't =
really=20
> Apache Hadoop are less important. you choose to reimplement Hadoop, =
you=20
> take on the costs of staying current.
>=20
>>=20
>> Once we have such specification, the HCK becomes meaningful (since =
the HCK
>> itself will be buggy initially).
>=20
>=20
>=20


From general-return-3507-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 05:39:05 2011
Return-Path: <general-return-3507-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3BFA7404F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 05:39:05 +0000 (UTC)
Received: (qmail 49755 invoked by uid 500); 13 May 2011 05:39:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49711 invoked by uid 500); 13 May 2011 05:39:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49702 invoked by uid 99); 13 May 2011 05:39:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 05:39:01 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 05:38:55 +0000
Received: by vws7 with SMTP id 7so2617452vws.35
        for <general@hadoop.apache.org>; Thu, 12 May 2011 22:38:34 -0700 (PDT)
Received: by 10.52.181.99 with SMTP id dv3mr1539359vdc.14.1305265114235; Thu,
 12 May 2011 22:38:34 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.52.112.7 with HTTP; Thu, 12 May 2011 22:38:14 -0700 (PDT)
X-Originating-IP: [67.160.196.149]
In-Reply-To: <C9F20588.8F34%mbhandarkar@linkedin.com>
References: <BANLkTi=zFtSowx2vtDxMFWCSQxp0-GD8fA@mail.gmail.com> <C9F20588.8F34%mbhandarkar@linkedin.com>
From: Ted Dunning <tdunning@maprtech.com>
Date: Thu, 12 May 2011 22:38:14 -0700
Message-ID: <BANLkTikPuePYzkuBT70a10NxGboxVimAHg@mail.gmail.com>
Subject: Re: Defining Hadoop Compatibility -revisiting-
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec548a5bd78c4ae04a321b705

--bcaec548a5bd78c4ae04a321b705
Content-Type: text/plain; charset=ISO-8859-1

I would say that an English spec with associated test suite is a middle
ground.

On Thu, May 12, 2011 at 9:52 PM, Milind Bhandarkar <mbhandarkar@linkedin.com
> wrote:

> Ok, my mistake. They have only asked for documented specifications. I may
> have been influenced by all the specifications I have read. All of them
> were in English, which is characterized as a natural language.
>
> But then, if you are proposing a specification in a non-natural-language,
> isn't that called a test suite ? Or is there a middle ground ?
>
> - milind
>
> --
> Milind Bhandarkar
> mbhandarkar@linkedin.com
> +1-650-776-3167
>
>
>
>
>
>
> On 5/12/11 9:05 PM, "Ted Dunning" <tdunning@maprtech.com> wrote:
>
> >Did anybody propose natural language only specifications?
>

--bcaec548a5bd78c4ae04a321b705--

From general-return-3508-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 06:12:59 2011
Return-Path: <general-return-3508-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5605A4E97
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 06:12:59 +0000 (UTC)
Received: (qmail 77038 invoked by uid 500); 13 May 2011 06:12:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76858 invoked by uid 500); 13 May 2011 06:12:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76843 invoked by uid 99); 13 May 2011 06:12:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 06:12:57 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 06:12:53 +0000
Received: by pzk10 with SMTP id 10so1507788pzk.35
        for <general@hadoop.apache.org>; Thu, 12 May 2011 23:12:32 -0700 (PDT)
Received: by 10.68.34.70 with SMTP id x6mr1722393pbi.344.1305267152136; Thu,
 12 May 2011 23:12:32 -0700 (PDT)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.68.62.230 with HTTP; Thu, 12 May 2011 23:12:12 -0700 (PDT)
In-Reply-To: <BANLkTikPuePYzkuBT70a10NxGboxVimAHg@mail.gmail.com>
References: <BANLkTi=zFtSowx2vtDxMFWCSQxp0-GD8fA@mail.gmail.com>
 <C9F20588.8F34%mbhandarkar@linkedin.com> <BANLkTikPuePYzkuBT70a10NxGboxVimAHg@mail.gmail.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Thu, 12 May 2011 23:12:12 -0700
X-Google-Sender-Auth: lWP3cJEj_Z_zgTrY65Astl_icKU
Message-ID: <BANLkTimMqUzw2r+-5mPzkByZYjViO3vz_A@mail.gmail.com>
Subject: Re: Defining Hadoop Compatibility -revisiting-
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

The way it has been done in JCK was a specs written in somewhat
formalized language and a tool (called testgen, written in Perl if I
remember correctly) which was dynamically generating a lot of lang
tests. I think this is a middle ground Milind has mentioned.

BTW, it was a _huge_ effort: Sun had two team working on TCK - ~40+
people - working for a few years on that thing.
--
=A0 Take care,
Konstantin (Cos) Boudnik

On Thu, May 12, 2011 at 22:38, Ted Dunning <tdunning@maprtech.com> wrote:
> I would say that an English spec with associated test suite is a middle
> ground.
>
> On Thu, May 12, 2011 at 9:52 PM, Milind Bhandarkar <mbhandarkar@linkedin.=
com
>> wrote:
>
>> Ok, my mistake. They have only asked for documented specifications. I ma=
y
>> have been influenced by all the specifications I have read. All of them
>> were in English, which is characterized as a natural language.
>>
>> But then, if you are proposing a specification in a non-natural-language=
,
>> isn't that called a test suite ? Or is there a middle ground ?
>>
>> - milind
>>
>> --
>> Milind Bhandarkar
>> mbhandarkar@linkedin.com
>> +1-650-776-3167
>>
>>
>>
>>
>>
>>
>> On 5/12/11 9:05 PM, "Ted Dunning" <tdunning@maprtech.com> wrote:
>>
>> >Did anybody propose natural language only specifications?
>>
>

From general-return-3509-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 06:16:30 2011
Return-Path: <general-return-3509-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 54E7146F1
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 06:16:30 +0000 (UTC)
Received: (qmail 95723 invoked by uid 500); 13 May 2011 06:16:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95668 invoked by uid 500); 13 May 2011 06:16:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95660 invoked by uid 99); 13 May 2011 06:16:28 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 06:16:28 +0000
Received: from localhost (HELO [90.168.237.215]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 06:16:26 +0000
Message-ID: <4DCCCCB2.6020607@apache.org>
Date: Fri, 13 May 2011 08:16:18 +0200
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Defining Hadoop Compatibility -revisiting-
References: <4DC91392.2010308@apache.org>
In-Reply-To: <4DC91392.2010308@apache.org>
X-Enigmail-Version: 1.1.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Certification semms like mission creep.  Our mission is to produce
open-source software.  If we wish to produce testing software, that
seems fine.  But running a certification program for non-open-source
software seems like a different task.

The Hadoop mark should only be used to refer to open-source software
produced by the ASF.  If other folks wish to make factual statements
concerning our software, e.g., that their proprietary software passes
tests that we've created, that may be fine, but I don't think we should
validate those claims by granting certifications to institutions.  That
ventures outside the mission of the ASF.  We are not an accrediting
organization.

Doug

On 05/10/2011 12:29 PM, Steve Loughran wrote:
> 
> Back in Jan 2011, I started a discussion about how to define Apache
> Hadoop Compatibility:
> http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C4D46B6AD.2020802@apache.org%3E
> 
> 
> I am now reading EMC HD "Enterprise Ready" Apache Hadoop datasheet
> 
> http://www.greenplum.com/sites/default/files/EMC_Greenplum_HD_DS_Final_1.pdf
> 
> 
> It claims that their implementations are 100% compatible, even though
> the Enterprise edition uses a C filesystem. It also claims that both
> their software releases contain "Certified Stacks", without defining
> what Certified means, or who does the certification -only that it is an
> improvement.
> 
> 
> I think we should revisit this issue before people with their own
> agendas define what compatibility with Apache Hadoop is for us
> 
> 
> Licensing
> -Use of the Hadoop codebase must follow the Apache License
> http://www.apache.org/licenses/LICENSE-2.0
> -plug in components that are dynamically linked to (Filesystems and
> schedulers) don't appear to be derivative works on my reading of this,
> 
> Naming
>  -this is something for branding@apache, they will have their opinions.
> The key one is that the name "Apache Hadoop" must get used, and it's
> important to make clear it is a derivative work.
>  -I don't think you can claim to have a Distribution/Fork/Version of
> Apache Hadoop if you swap out big chunks of it for alternate
> filesystems, MR engines, etc. Some description of this is needed
> "Supports the Apache Hadoop MapReduce engine on top of Filesystem XYZ"
> 
> Compatibility
>  -the definition of the Hadoop interfaces and classes is the Apache
> Source tree,
>  -the definition of semantics of the Hadoop interfaces and classes is
> the Apache Source tree, including the test classes.
>  -the verification that the actual semantics of an Apache Hadoop release
> is compatible with the expected semantics is that current and future
> tests pass
>  -bug reports can highlight incompatibility with expectations of
> community users, and once incorporated into tests form part of the
> compatibility testing
>  -vendors can claim and even certify their derivative works as
> compatible with other versions of their derivative works, but cannot
> claim compatibility with Apache Hadoop unless their code passes the
> tests and is consistent with the bug reports marked as ("by design").
> Perhaps we should have tests that verify each of these "by design"
> bugreps to make them more formal.
> 
> Certification
>  -I have no idea what this means in EMC's case, they just say "Certified"
>  -As we don't do any certification ourselves, it would seem impossible
> for us to certify that any derivative work is compatible.
>  -It may be best to state that nobody can certify their derivative as
> "compatible with Apache Hadoop" unless it passes all current test suites
>  -And require that anyone who declares compatibility define what they
> mean by this
> 
> This is a good argument for getting more functional tests out there
> -whoever has more functional tests needs to get them into a test module
> that can be used to test real deployments.
> 

From general-return-3510-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 06:25:42 2011
Return-Path: <general-return-3510-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9F8454A7C
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 06:25:42 +0000 (UTC)
Received: (qmail 3474 invoked by uid 500); 13 May 2011 06:25:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3017 invoked by uid 500); 13 May 2011 06:25:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3009 invoked by uid 99); 13 May 2011 06:25:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 06:25:39 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 06:25:33 +0000
Received: by pve37 with SMTP id 37so1487012pve.35
        for <general@hadoop.apache.org>; Thu, 12 May 2011 23:25:11 -0700 (PDT)
Received: by 10.68.57.235 with SMTP id l11mr1559473pbq.478.1305267911048; Thu,
 12 May 2011 23:25:11 -0700 (PDT)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.68.62.230 with HTTP; Thu, 12 May 2011 23:24:51 -0700 (PDT)
In-Reply-To: <C9F1F4D2.8F11%mbhandarkar@linkedin.com>
References: <BANLkTimOJAwc2Y9ixj2etCUnDe+9hfbi+g@mail.gmail.com> <C9F1F4D2.8F11%mbhandarkar@linkedin.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Thu, 12 May 2011 23:24:51 -0700
X-Google-Sender-Auth: Y5YHFWFgZb22fXvUi-G9rRnipfI
Message-ID: <BANLkTi=4LOHwbJP6qqwKSN0MJ+OSTYtnaQ@mail.gmail.com>
Subject: Re: Defining Hadoop Compatibility -revisiting-
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Thu, May 12, 2011 at 20:40, Milind Bhandarkar
<mbhandarkar@linkedin.com> wrote:
> Cos,
>
> Can you give me an example of a "system test" that is not a functional
> test ? My assumption was that the functionality being tested is specific
> to a component, and that inter-component interactions (that's what you
> meant, right?) would be taken care by the public interface and semantics
> of a component API.

Milind, kinda... However, to exercise inter-component interactions via
component APIs one needs to have tests which are beyond functional or
component realm (e.g. system). At some point  I was part of a team
working on integration validation framework for Hadoop (FIT) which was
addressing inter-component interaction validations essentially
guaranteeing their compatibility. Components being Hadoop, Pig, Oozie,
etc. - thus massaging the whole stack of application and covering a
lot of use cases.

Having a framework like this and a set of test cases available for
Hadoop community is a great benefit because one can quickly make sure
that a Hadoop stack built from a set of components is working
property. Another use case is to run the same set of tests - versioned
separately from the product itself - against previous and a next
release validating their compatibility at the functional level (sorta
what you have mentioned).

This doesn't by the way deploy if we'd choose to work on HCK or not,
however HCK might be eventually based on top of such a framework.

Cos

> - milind
>
> --
> Milind Bhandarkar
> mbhandarkar@linkedin.com
> +1-650-776-3167
>
>
>
>
>
>
> On 5/12/11 3:30 PM, "Konstantin Boudnik" <cos@apache.org> wrote:
>
>>On Thu, May 12, 2011 at 09:45, Milind Bhandarkar
>><mbhandarkar@linkedin.com> wrote:
>>> HCK and written specifications are not mutually exclusive. However,
>>>given
>>> the evolving nature of Hadoop APIs, functional tests need to evolve as
>>
>>I would actually expand it to 'functional and system tests' because
>>latter are capable of validating inter-component iterations not
>>coverable by functional tests.
>>
>>Cos
>>
>>> well, and having them tied to a "current stable" version is easier to d=
o
>>> than it is to tie the written specifications.
>>>
>>> - milind
>>>
>>> --
>>> Milind Bhandarkar
>>> mbhandarkar@linkedin.com
>>> +1-650-776-3167
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 5/11/11 7:26 PM, "M. C. Srivas" <mcsrivas@gmail.com> wrote:
>>>
>>>>While the HCK is a great idea to check quickly if an implementation is
>>>>"compliant", =A0we still need a written specification to define what is
>>>>meant
>>>>by compliance, something akin to a set of RFC's, or a set of docs like
>>>>the
>>>> IEEE POSIX specifications.
>>>>
>>>>For example, the POSIX.1c pthreads API has a written document that
>>>>specifies
>>>>all the function calls, input params, return values, and error codes. I=
t
>>>>clearly indicates what any POSIX-complaint threads package needs to
>>>>support,
>>>>and what are vendor-specific non-portable extensions that one can use a=
t
>>>>one's own risk.
>>>>
>>>>Currently we have 2 sets of API =A0in the DFS and Map/Reduce layers, an=
d
>>>>the
>>>>specification is extracted only by looking at the code, or (where the
>>>>code
>>>>is non-trivial) by writing really bizarre test programs to examine
>>>>corner
>>>>cases. Further, the interaction between a mix of the old and new APIs i=
s
>>>>not
>>>>specified anywhere. Such specifications are vitally important when
>>>>implementing libraries like Cascading, Mahout, etc. For example, an
>>>>application might open a file using the new API, and pass that stream
>>>>into a
>>>>library that manipulates the stream using some of the old API ... what
>>>>is
>>>>then the expectation of the state of the stream when the library call
>>>>returns?
>>>>
>>>>Sanjay Radia @ Y! already started specifying some the DFS APIs to nail
>>>>such
>>>>things down. There's similar good effort in the Map/Reduce and =A0Avro
>>>>spaces,
>>>>but it seems to have stalled somewhat. We should continue it.
>>>>
>>>>Doing such specs would be a great service to the community and the user=
s
>>>>of
>>>>Hadoop. It provides them
>>>> =A0 (a) clear-cut docs on how to use the Hadoop APIs
>>>> =A0 (b) wider choice of Hadoop implementations by freeing them from
>>>>vendor
>>>>lock-in.
>>>>
>>>>Once we have such specification, the HCK becomes meaningful (since the
>>>>HCK
>>>>itself will be buggy initially).
>>>>
>>>>
>>>>On Wed, May 11, 2011 at 2:46 PM, Milind Bhandarkar
>>>><mbhandarkar@linkedin.com
>>>>> wrote:
>>>>
>>>>> I think it's time to separate out functional tests as a "Hadoop
>>>>> Compatibility Kit (HCK)", similar to the Sun TCK for Java, but under
>>>>>ASL
>>>>> 2.0. Then "certification" would mean "Passes 100% of the HCK
>>>>>testsuite."
>>>>>
>>>>> - milind
>>>>> --
>>>>> Milind Bhandarkar
>>>>> mbhandarkar@linkedin.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 5/11/11 2:24 PM, "Eric Baldeschwieler" <eric14@yahoo-inc.com>
>>>>>wrote:
>>>>>
>>>>> >This is a really interesting topic! =A0I completely agree that we ne=
ed
>>>>>to
>>>>> >get ahead of this.
>>>>> >
>>>>> >I would be really interested in learning of any experience other
>>>>>apache
>>>>> >projects, such as apache or tomcat have with these issues.
>>>>> >
>>>>> >---
>>>>> >E14 - typing on glass
>>>>> >
>>>>> >On May 10, 2011, at 6:31 AM, "Steve Loughran" <stevel@apache.org>
>>>>>wrote:
>>>>> >
>>>>> >>
>>>>> >> Back in Jan 2011, I started a discussion about how to define Apach=
e
>>>>> >> Hadoop Compatibility:
>>>>> >>
>>>>> >>
>>>>>
>>>>>http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3=
C
>>>>>4D
>>>>> >>46B6AD.2020802@apache.org%3E
>>>>> >>
>>>>> >> I am now reading EMC HD "Enterprise Ready" Apache Hadoop datasheet
>>>>> >>
>>>>> >>
>>>>>
>>>>>>>http://www.greenplum.com/sites/default/files/EMC_Greenplum_HD_DS_Fin=
a
>>>>>>>l_
>>>>>>>1
>>>>> .
>>>>> >>pdf
>>>>> >>
>>>>> >> It claims that their implementations are 100% compatible, even
>>>>>though
>>>>> >> the Enterprise edition uses a C filesystem. It also claims that
>>>>>both
>>>>> >> their software releases contain "Certified Stacks", without
>>>>>defining
>>>>> >> what Certified means, or who does the certification -only that it
>>>>>is
>>>>>an
>>>>> >> improvement.
>>>>> >>
>>>>> >>
>>>>> >> I think we should revisit this issue before people with their own
>>>>> >> agendas define what compatibility with Apache Hadoop is for us
>>>>> >>
>>>>> >>
>>>>> >> Licensing
>>>>> >> -Use of the Hadoop codebase must follow the Apache License
>>>>> >> http://www.apache.org/licenses/LICENSE-2.0
>>>>> >> -plug in components that are dynamically linked to (Filesystems an=
d
>>>>> >> schedulers) don't appear to be derivative works on my reading of
>>>>>this,
>>>>> >>
>>>>> >> Naming
>>>>> >> =A0-this is something for branding@apache, they will have their
>>>>>opinions.
>>>>> >> The key one is that the name "Apache Hadoop" must get used, and
>>>>>it's
>>>>> >> important to make clear it is a derivative work.
>>>>> >> =A0-I don't think you can claim to have a Distribution/Fork/Versio=
n
>>>>>of
>>>>> >> Apache Hadoop if you swap out big chunks of it for alternate
>>>>> >> filesystems, MR engines, etc. Some description of this is needed
>>>>> >> "Supports the Apache Hadoop MapReduce engine on top of Filesystem
>>>>>XYZ"
>>>>> >>
>>>>> >> Compatibility
>>>>> >> =A0-the definition of the Hadoop interfaces and classes is the Apa=
che
>>>>> >> Source tree,
>>>>> >> =A0-the definition of semantics of the Hadoop interfaces and class=
es
>>>>>is
>>>>> >> the Apache Source tree, including the test classes.
>>>>> >> =A0-the verification that the actual semantics of an Apache Hadoop
>>>>> >> release is compatible with the expected semantics is that current
>>>>>and
>>>>> >> future tests pass
>>>>> >> =A0-bug reports can highlight incompatibility with expectations of
>>>>> >> community users, and once incorporated into tests form part of the
>>>>> >> compatibility testing
>>>>> >> =A0-vendors can claim and even certify their derivative works as
>>>>> >> compatible with other versions of their derivative works, but
>>>>>cannot
>>>>> >> claim compatibility with Apache Hadoop unless their code passes th=
e
>>>>> >> tests and is consistent with the bug reports marked as ("by
>>>>>design").
>>>>> >> Perhaps we should have tests that verify each of these "by design"
>>>>> >> bugreps to make them more formal.
>>>>> >>
>>>>> >> Certification
>>>>> >> =A0-I have no idea what this means in EMC's case, they just say
>>>>> >>"Certified"
>>>>> >> =A0-As we don't do any certification ourselves, it would seem
>>>>>impossible
>>>>> >> for us to certify that any derivative work is compatible.
>>>>> >> =A0-It may be best to state that nobody can certify their derivati=
ve
>>>>>as
>>>>> >> "compatible with Apache Hadoop" unless it passes all current test
>>>>>suites
>>>>> >> =A0-And require that anyone who declares compatibility define what
>>>>>they
>>>>> >> mean by this
>>>>> >>
>>>>> >> This is a good argument for getting more functional tests out ther=
e
>>>>> >> -whoever has more functional tests needs to get them into a test
>>>>>module
>>>>> >> that can be used to test real deployments.
>>>>> >>
>>>>>
>>>>>
>>>
>>>
>
>

From general-return-3511-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 06:54:11 2011
Return-Path: <general-return-3511-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 919144AB7
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 06:54:11 +0000 (UTC)
Received: (qmail 49558 invoked by uid 500); 13 May 2011 06:54:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49386 invoked by uid 500); 13 May 2011 06:54:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49377 invoked by uid 99); 13 May 2011 06:54:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 06:54:08 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mbhandarkar@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 06:54:02 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:In-Reply-To:Accept-Language:
   Content-Language:X-MS-Has-Attach:X-MS-TNEF-Correlator:
   x-originating-ip:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version;
  b=eystJ4GMvwaebeJcUJI5+ksaXbWjXR7eytonowpQPGgcofK3TYCUc453
   iLBK2TW/EN8RP1SlKLWNc/TVP3jMQrP69xHsBXxhX9000tm58fDTrVHbh
   bwgc4eisBYtFDFY;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=mbhandarkar@linkedin.com; q=dns/txt;
  s=proddkim; t=1305269642; x=1336805642;
  h=from:to:subject:date:message-id:in-reply-to:content-id:
   content-transfer-encoding:mime-version;
  bh=4lfbstlA1TlzQaACldpUopg/YTRA/dj7NgZD2X0huSg=;
  b=IDVtdRzIEdt3q8k7aOBYr+7TUx8cxPe9Xug915Mag4kXjIUDO0mC/sSY
   oZW4bisrcdxMO55fjy/k8Et3JXVUKg1FNnMtkXAGTEN2mA1wz+VYgS1+3
   KVtH6ym7DTjGXxQ;
X-IronPort-AV: E=Sophos;i="4.64,362,1301900400"; 
   d="scan'208";a="23150739"
Received: from ESV4-EXC02.linkedin.biz ([fe80::4d74:48bd:e0bd:13ee]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi id 14.01.0251.000; Thu,
 12 May 2011 23:57:17 -0700
From: Milind Bhandarkar <mbhandarkar@linkedin.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: Defining Hadoop Compatibility -revisiting-
Thread-Topic: Defining Hadoop Compatibility -revisiting-
Thread-Index: AQHMEPOnIB+Y40jJf0ahsOjyQf2X+pSKG1eAgAB+JQD//5axgIAAgywA//+fuYA=
Date: Fri, 13 May 2011 06:57:17 +0000
Message-ID: <C9F21EB2.8F52%mbhandarkar@linkedin.com>
In-Reply-To: <BANLkTikPuePYzkuBT70a10NxGboxVimAHg@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <92CD4637F8D7924C9C1E8F7AE5B0841E@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Sure. As I said before, they are not mutually exclusive. Just stating my
experience that specs without a test suite are of no use. If I were to
prioritize, I would give priority to a TCK over natural-language specs.
That's all.

So far, I have seen many replacements for HDFS as InputFormat and
OutputFormat that reads from or writes to different data sources and syncs
only. It is easily imaginable to have a pluggable app managers and
resource manager after MR-279 (other than local, which is part of Apache
Hadoop, but not "compatible", think distributed cache).

So, we would need a spec and a test suite per component (I.e. App manager,
resource manager, current scheduler, replication target chooser,
authentication, authorization) now. If the binary protocols were to be
crystallized, I can imagine others implementing only the datanode, or a
task tracker. So we would need protocol-level compatibility suite for
individual daemons as well.

I agree with one of the statements that Steve L made, that "Hadoop has an
enviable problem of too much activity." If one follows the activities in
commercial world, open source, academic and industry-sponsored R&D, one
quickly realizes that writing RFCs for all the above components and fixing
them without versioning is cumbersome and difficult optimistically, and
near impossible realistically. Also, my experience is that keeping
standards documentation for an evolving technology up-to-date with the
proper implementation is a pipe-dream at best. A test suite that gets
compiled and run every time a new version comes out is within the realm of
possibility.

Therefore, all I am saying is that, while a POSIX-like spec is a "nice to
have", a test-suite that defines compatibility is a must.

- milind

--=20
Milind Bhandarkar
mbhandarkar@linkedin.com
+1-650-776-3167






On 5/12/11 10:38 PM, "Ted Dunning" <tdunning@maprtech.com> wrote:

>I would say that an English spec with associated test suite is a middle
>ground.
>
>On Thu, May 12, 2011 at 9:52 PM, Milind Bhandarkar
><mbhandarkar@linkedin.com
>> wrote:
>
>> Ok, my mistake. They have only asked for documented specifications. I
>>may
>> have been influenced by all the specifications I have read. All of them
>> were in English, which is characterized as a natural language.
>>
>> But then, if you are proposing a specification in a
>>non-natural-language,
>> isn't that called a test suite ? Or is there a middle ground ?
>>
>> - milind
>>
>> --
>> Milind Bhandarkar
>> mbhandarkar@linkedin.com
>> +1-650-776-3167
>>
>>
>>
>>
>>
>>
>> On 5/12/11 9:05 PM, "Ted Dunning" <tdunning@maprtech.com> wrote:
>>
>> >Did anybody propose natural language only specifications?
>>


From general-return-3512-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 07:08:06 2011
Return-Path: <general-return-3512-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DCFF94393
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 07:08:06 +0000 (UTC)
Received: (qmail 77648 invoked by uid 500); 13 May 2011 07:08:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77600 invoked by uid 500); 13 May 2011 07:08:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77592 invoked by uid 99); 13 May 2011 07:08:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 07:08:04 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mbhandarkar@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 07:07:58 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:In-Reply-To:Accept-Language:
   Content-Language:X-MS-Has-Attach:X-MS-TNEF-Correlator:
   x-originating-ip:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version;
  b=u72I0W6rAm/JPdDyUv7/Qs7g7YjZGQCXAXLaMyq3t3oMbejFh5A8coLK
   Xgy8sAjqA4YMNKgZLgh5JzNfwudze9zWhZiJoVwhYqNOvmc7V5gxq0CWu
   XYHqq4iz+tkWPIJ;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=mbhandarkar@linkedin.com; q=dns/txt;
  s=proddkim; t=1305270477; x=1336806477;
  h=from:to:subject:date:message-id:in-reply-to:content-id:
   content-transfer-encoding:mime-version;
  bh=i6XB3LDwOrB9qh7Tf7k2zJAkgBwtDcJj5CyoQAmjKGY=;
  b=USfbEVfVIj+ir68JlGglF/ayl/t7qT5aXDUgFDh2tw453DFAYgC+WIF1
   TW2kKkUQs8k14vpEUe3VftREUykKRO//+t+kt69eTzUrFxReuSKcFfnOu
   ZgUtpXo4zDMqvub;
X-IronPort-AV: E=Sophos;i="4.64,362,1301900400"; 
   d="scan'208";a="23150984"
Received: from ESV4-EXC02.linkedin.biz ([fe80::4d74:48bd:e0bd:13ee]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi id 14.01.0251.000; Fri,
 13 May 2011 00:11:13 -0700
From: Milind Bhandarkar <mbhandarkar@linkedin.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: Defining Hadoop Compatibility -revisiting-
Thread-Topic: Defining Hadoop Compatibility -revisiting-
Thread-Index: AQHMDv3EIB+Y40jJf0ahsOjyQf2X+pSImxGA//+PqQCAAMSEgIAAedIAgADWjwD//+B3AIAApDWA//+WlgA=
Date: Fri, 13 May 2011 07:11:12 +0000
Message-ID: <C9F22405.8F84%mbhandarkar@linkedin.com>
In-Reply-To: <BANLkTi=4LOHwbJP6qqwKSN0MJ+OSTYtnaQ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <A30241E3460938449A7483371B6E6D75@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Cos,

I remember the issues about the "inter-component interactions" at that
point when you were part of the Yahoo Hadoop FIT team (I was on the other
side of the same floor, remember ? ;-)

Things like, "Can Pig take full URIs as input, and so works with viewfs",
"Can Local jobtracker still use HDFS as input and output", "Can Oozie use
local file system to keep workflows, while the jars were located on hdfs"
etc came up often.

Each of these issues were component-interaction issues, and were results
of making DistributedFileSystem a public class, or some subtle dependency
on the semantics of a particular method in an interface, which were not
explicit in the syntax.

That's an issue with interface-compatibility, and so merely compiling
against a particular interface is not a solution. One needs a test-suite.

(With annotations in Java, one can impose more semantic restrictions on
the interface, that can be automatically checked against at runtime. But
is limited to individual methods, or the full class. Code generation using
perl or whatever is similar in capability.)

- milind
--=20
Milind Bhandarkar
mbhandarkar@linkedin.com
+1-650-776-3167






On 5/12/11 11:24 PM, "Konstantin Boudnik" <cos@apache.org> wrote:

>On Thu, May 12, 2011 at 20:40, Milind Bhandarkar
><mbhandarkar@linkedin.com> wrote:
>> Cos,
>>
>> Can you give me an example of a "system test" that is not a functional
>> test ? My assumption was that the functionality being tested is specific
>> to a component, and that inter-component interactions (that's what you
>> meant, right?) would be taken care by the public interface and semantics
>> of a component API.
>
>Milind, kinda... However, to exercise inter-component interactions via
>component APIs one needs to have tests which are beyond functional or
>component realm (e.g. system). At some point  I was part of a team
>working on integration validation framework for Hadoop (FIT) which was
>addressing inter-component interaction validations essentially
>guaranteeing their compatibility. Components being Hadoop, Pig, Oozie,
>etc. - thus massaging the whole stack of application and covering a
>lot of use cases.
>
>Having a framework like this and a set of test cases available for
>Hadoop community is a great benefit because one can quickly make sure
>that a Hadoop stack built from a set of components is working
>property. Another use case is to run the same set of tests - versioned
>separately from the product itself - against previous and a next
>release validating their compatibility at the functional level (sorta
>what you have mentioned).
>
>This doesn't by the way deploy if we'd choose to work on HCK or not,
>however HCK might be eventually based on top of such a framework.
>
>Cos
>
>> - milind
>>
>> --
>> Milind Bhandarkar
>> mbhandarkar@linkedin.com
>> +1-650-776-3167
>>
>>
>>
>>
>>
>>
>> On 5/12/11 3:30 PM, "Konstantin Boudnik" <cos@apache.org> wrote:
>>
>>>On Thu, May 12, 2011 at 09:45, Milind Bhandarkar
>>><mbhandarkar@linkedin.com> wrote:
>>>> HCK and written specifications are not mutually exclusive. However,
>>>>given
>>>> the evolving nature of Hadoop APIs, functional tests need to evolve as
>>>
>>>I would actually expand it to 'functional and system tests' because
>>>latter are capable of validating inter-component iterations not
>>>coverable by functional tests.
>>>
>>>Cos
>>>
>>>> well, and having them tied to a "current stable" version is easier to
>>>>do
>>>> than it is to tie the written specifications.
>>>>
>>>> - milind
>>>>
>>>> --
>>>> Milind Bhandarkar
>>>> mbhandarkar@linkedin.com
>>>> +1-650-776-3167
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 5/11/11 7:26 PM, "M. C. Srivas" <mcsrivas@gmail.com> wrote:
>>>>
>>>>>While the HCK is a great idea to check quickly if an implementation is
>>>>>"compliant",  we still need a written specification to define what is
>>>>>meant
>>>>>by compliance, something akin to a set of RFC's, or a set of docs like
>>>>>the
>>>>> IEEE POSIX specifications.
>>>>>
>>>>>For example, the POSIX.1c pthreads API has a written document that
>>>>>specifies
>>>>>all the function calls, input params, return values, and error codes.
>>>>>It
>>>>>clearly indicates what any POSIX-complaint threads package needs to
>>>>>support,
>>>>>and what are vendor-specific non-portable extensions that one can use
>>>>>at
>>>>>one's own risk.
>>>>>
>>>>>Currently we have 2 sets of API  in the DFS and Map/Reduce layers, and
>>>>>the
>>>>>specification is extracted only by looking at the code, or (where the
>>>>>code
>>>>>is non-trivial) by writing really bizarre test programs to examine
>>>>>corner
>>>>>cases. Further, the interaction between a mix of the old and new APIs
>>>>>is
>>>>>not
>>>>>specified anywhere. Such specifications are vitally important when
>>>>>implementing libraries like Cascading, Mahout, etc. For example, an
>>>>>application might open a file using the new API, and pass that stream
>>>>>into a
>>>>>library that manipulates the stream using some of the old API ... what
>>>>>is
>>>>>then the expectation of the state of the stream when the library call
>>>>>returns?
>>>>>
>>>>>Sanjay Radia @ Y! already started specifying some the DFS APIs to nail
>>>>>such
>>>>>things down. There's similar good effort in the Map/Reduce and  Avro
>>>>>spaces,
>>>>>but it seems to have stalled somewhat. We should continue it.
>>>>>
>>>>>Doing such specs would be a great service to the community and the
>>>>>users
>>>>>of
>>>>>Hadoop. It provides them
>>>>>   (a) clear-cut docs on how to use the Hadoop APIs
>>>>>   (b) wider choice of Hadoop implementations by freeing them from
>>>>>vendor
>>>>>lock-in.
>>>>>
>>>>>Once we have such specification, the HCK becomes meaningful (since the
>>>>>HCK
>>>>>itself will be buggy initially).
>>>>>
>>>>>
>>>>>On Wed, May 11, 2011 at 2:46 PM, Milind Bhandarkar
>>>>><mbhandarkar@linkedin.com
>>>>>> wrote:
>>>>>
>>>>>> I think it's time to separate out functional tests as a "Hadoop
>>>>>> Compatibility Kit (HCK)", similar to the Sun TCK for Java, but under
>>>>>>ASL
>>>>>> 2.0. Then "certification" would mean "Passes 100% of the HCK
>>>>>>testsuite."
>>>>>>
>>>>>> - milind
>>>>>> --
>>>>>> Milind Bhandarkar
>>>>>> mbhandarkar@linkedin.com
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 5/11/11 2:24 PM, "Eric Baldeschwieler" <eric14@yahoo-inc.com>
>>>>>>wrote:
>>>>>>
>>>>>> >This is a really interesting topic!  I completely agree that we
>>>>>>need
>>>>>>to
>>>>>> >get ahead of this.
>>>>>> >
>>>>>> >I would be really interested in learning of any experience other
>>>>>>apache
>>>>>> >projects, such as apache or tomcat have with these issues.
>>>>>> >
>>>>>> >---
>>>>>> >E14 - typing on glass
>>>>>> >
>>>>>> >On May 10, 2011, at 6:31 AM, "Steve Loughran" <stevel@apache.org>
>>>>>>wrote:
>>>>>> >
>>>>>> >>
>>>>>> >> Back in Jan 2011, I started a discussion about how to define
>>>>>>Apache
>>>>>> >> Hadoop Compatibility:
>>>>>> >>
>>>>>> >>
>>>>>>
>>>>>>http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%
>>>>>>3C
>>>>>>4D
>>>>>> >>46B6AD.2020802@apache.org%3E
>>>>>> >>
>>>>>> >> I am now reading EMC HD "Enterprise Ready" Apache Hadoop
>>>>>>datasheet
>>>>>> >>
>>>>>> >>
>>>>>>
>>>>>>>>http://www.greenplum.com/sites/default/files/EMC_Greenplum_HD_DS_Fi
>>>>>>>>na
>>>>>>>>l_
>>>>>>>>1
>>>>>> .
>>>>>> >>pdf
>>>>>> >>
>>>>>> >> It claims that their implementations are 100% compatible, even
>>>>>>though
>>>>>> >> the Enterprise edition uses a C filesystem. It also claims that
>>>>>>both
>>>>>> >> their software releases contain "Certified Stacks", without
>>>>>>defining
>>>>>> >> what Certified means, or who does the certification -only that it
>>>>>>is
>>>>>>an
>>>>>> >> improvement.
>>>>>> >>
>>>>>> >>
>>>>>> >> I think we should revisit this issue before people with their own
>>>>>> >> agendas define what compatibility with Apache Hadoop is for us
>>>>>> >>
>>>>>> >>
>>>>>> >> Licensing
>>>>>> >> -Use of the Hadoop codebase must follow the Apache License
>>>>>> >> http://www.apache.org/licenses/LICENSE-2.0
>>>>>> >> -plug in components that are dynamically linked to (Filesystems
>>>>>>and
>>>>>> >> schedulers) don't appear to be derivative works on my reading of
>>>>>>this,
>>>>>> >>
>>>>>> >> Naming
>>>>>> >>  -this is something for branding@apache, they will have their
>>>>>>opinions.
>>>>>> >> The key one is that the name "Apache Hadoop" must get used, and
>>>>>>it's
>>>>>> >> important to make clear it is a derivative work.
>>>>>> >>  -I don't think you can claim to have a Distribution/Fork/Version
>>>>>>of
>>>>>> >> Apache Hadoop if you swap out big chunks of it for alternate
>>>>>> >> filesystems, MR engines, etc. Some description of this is needed
>>>>>> >> "Supports the Apache Hadoop MapReduce engine on top of Filesystem
>>>>>>XYZ"
>>>>>> >>
>>>>>> >> Compatibility
>>>>>> >>  -the definition of the Hadoop interfaces and classes is the
>>>>>>Apache
>>>>>> >> Source tree,
>>>>>> >>  -the definition of semantics of the Hadoop interfaces and
>>>>>>classes
>>>>>>is
>>>>>> >> the Apache Source tree, including the test classes.
>>>>>> >>  -the verification that the actual semantics of an Apache Hadoop
>>>>>> >> release is compatible with the expected semantics is that current
>>>>>>and
>>>>>> >> future tests pass
>>>>>> >>  -bug reports can highlight incompatibility with expectations of
>>>>>> >> community users, and once incorporated into tests form part of
>>>>>>the
>>>>>> >> compatibility testing
>>>>>> >>  -vendors can claim and even certify their derivative works as
>>>>>> >> compatible with other versions of their derivative works, but
>>>>>>cannot
>>>>>> >> claim compatibility with Apache Hadoop unless their code passes
>>>>>>the
>>>>>> >> tests and is consistent with the bug reports marked as ("by
>>>>>>design").
>>>>>> >> Perhaps we should have tests that verify each of these "by
>>>>>>design"
>>>>>> >> bugreps to make them more formal.
>>>>>> >>
>>>>>> >> Certification
>>>>>> >>  -I have no idea what this means in EMC's case, they just say
>>>>>> >>"Certified"
>>>>>> >>  -As we don't do any certification ourselves, it would seem
>>>>>>impossible
>>>>>> >> for us to certify that any derivative work is compatible.
>>>>>> >>  -It may be best to state that nobody can certify their
>>>>>>derivative
>>>>>>as
>>>>>> >> "compatible with Apache Hadoop" unless it passes all current test
>>>>>>suites
>>>>>> >>  -And require that anyone who declares compatibility define what
>>>>>>they
>>>>>> >> mean by this
>>>>>> >>
>>>>>> >> This is a good argument for getting more functional tests out
>>>>>>there
>>>>>> >> -whoever has more functional tests needs to get them into a test
>>>>>>module
>>>>>> >> that can be used to test real deployments.
>>>>>> >>
>>>>>>
>>>>>>
>>>>
>>>>
>>
>>


From general-return-3513-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 07:21:40 2011
Return-Path: <general-return-3513-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C2E2C4B15
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 07:21:40 +0000 (UTC)
Received: (qmail 91733 invoked by uid 500); 13 May 2011 07:21:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91011 invoked by uid 500); 13 May 2011 07:21:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90982 invoked by uid 99); 13 May 2011 07:21:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 07:21:33 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mbhandarkar@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 07:21:28 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:In-Reply-To:Accept-Language:
   Content-Language:X-MS-Has-Attach:X-MS-TNEF-Correlator:
   x-originating-ip:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version;
  b=dfRgmifZrD59WZLkEO9jqT1TZsZbo7o7Mi883xbPFCIyd87+dgmqp+i0
   z4QXoQIPOZACVyNs8yeemBQEiyBP6XLUzegd/3RNwcKI9/n9TtiV7yRWQ
   lQudEYkeLXO+YpR;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=mbhandarkar@linkedin.com; q=dns/txt;
  s=proddkim; t=1305271288; x=1336807288;
  h=from:to:subject:date:message-id:in-reply-to:content-id:
   content-transfer-encoding:mime-version;
  bh=dxuhBEwwJyw76wPIo+Lg6Gp3qLMvmwBT+S2JdgsHTrs=;
  b=zKAWIk84A7nFMXzXf9vfzHJ/rAEUUeSyGdW70NNf0EX7O+57nh83ARn2
   BX0zVYFXehjXh8r6gvITnMSaWy0HuG3IiJK5qBU3uDxYCb/KUvnhC3p15
   /cQLXr7BPNWNQtK;
X-IronPort-AV: E=Sophos;i="4.64,362,1301900400"; 
   d="scan'208";a="23151524"
Received: from ESV4-EXC02.linkedin.biz ([fe80::4d74:48bd:e0bd:13ee]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi id 14.01.0251.000; Fri,
 13 May 2011 00:24:44 -0700
From: Milind Bhandarkar <mbhandarkar@linkedin.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: Defining Hadoop Compatibility -revisiting-
Thread-Topic: Defining Hadoop Compatibility -revisiting-
Thread-Index: AQHMETVFIB+Y40jJf0ahsOjyQf2X+pSKWjoA
Date: Fri, 13 May 2011 07:24:43 +0000
Message-ID: <C9F227D5.8FA4%mbhandarkar@linkedin.com>
In-Reply-To: <4DCCCCB2.6020607@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <4CD6C50103C78243AA918CBD64B6F73E@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

+1.

Apache foundation or contributors to Apache should not waste their energy
providing such certification.

Compatibility claims should be easily verifiable by users of these
proprietary systems or independent observers, if a test-suite were readily
available to run.

>The Hadoop mark should only be used to refer to open-source software
>produced by the ASF.


IANAL, but Steve is questioning usage of "Apache Hadoop Compatible" in PR
material of commercial software. Is this considered as usage of "The
Hadoop mark" ?

- milind

--=20
Milind Bhandarkar
mbhandarkar@linkedin.com
+1-650-776-3167






On 5/12/11 11:16 PM, "Doug Cutting" <cutting@apache.org> wrote:

>Certification semms like mission creep.  Our mission is to produce
>open-source software.  If we wish to produce testing software, that
>seems fine.  But running a certification program for non-open-source
>software seems like a different task.
>
>The Hadoop mark should only be used to refer to open-source software
>produced by the ASF.  If other folks wish to make factual statements
>concerning our software, e.g., that their proprietary software passes
>tests that we've created, that may be fine, but I don't think we should
>validate those claims by granting certifications to institutions.  That
>ventures outside the mission of the ASF.  We are not an accrediting
>organization.
>
>Doug
>
>On 05/10/2011 12:29 PM, Steve Loughran wrote:
>>=20
>> Back in Jan 2011, I started a discussion about how to define Apache
>> Hadoop Compatibility:
>>=20
>>http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C4D
>>46B6AD.2020802@apache.org%3E
>>=20
>>=20
>> I am now reading EMC HD "Enterprise Ready" Apache Hadoop datasheet
>>=20
>>=20
>>http://www.greenplum.com/sites/default/files/EMC_Greenplum_HD_DS_Final_1.
>>pdf
>>=20
>>=20
>> It claims that their implementations are 100% compatible, even though
>> the Enterprise edition uses a C filesystem. It also claims that both
>> their software releases contain "Certified Stacks", without defining
>> what Certified means, or who does the certification -only that it is an
>> improvement.
>>=20
>>=20
>> I think we should revisit this issue before people with their own
>> agendas define what compatibility with Apache Hadoop is for us
>>=20
>>=20
>> Licensing
>> -Use of the Hadoop codebase must follow the Apache License
>> http://www.apache.org/licenses/LICENSE-2.0
>> -plug in components that are dynamically linked to (Filesystems and
>> schedulers) don't appear to be derivative works on my reading of this,
>>=20
>> Naming
>>  -this is something for branding@apache, they will have their opinions.
>> The key one is that the name "Apache Hadoop" must get used, and it's
>> important to make clear it is a derivative work.
>>  -I don't think you can claim to have a Distribution/Fork/Version of
>> Apache Hadoop if you swap out big chunks of it for alternate
>> filesystems, MR engines, etc. Some description of this is needed
>> "Supports the Apache Hadoop MapReduce engine on top of Filesystem XYZ"
>>=20
>> Compatibility
>>  -the definition of the Hadoop interfaces and classes is the Apache
>> Source tree,
>>  -the definition of semantics of the Hadoop interfaces and classes is
>> the Apache Source tree, including the test classes.
>>  -the verification that the actual semantics of an Apache Hadoop release
>> is compatible with the expected semantics is that current and future
>> tests pass
>>  -bug reports can highlight incompatibility with expectations of
>> community users, and once incorporated into tests form part of the
>> compatibility testing
>>  -vendors can claim and even certify their derivative works as
>> compatible with other versions of their derivative works, but cannot
>> claim compatibility with Apache Hadoop unless their code passes the
>> tests and is consistent with the bug reports marked as ("by design").
>> Perhaps we should have tests that verify each of these "by design"
>> bugreps to make them more formal.
>>=20
>> Certification
>>  -I have no idea what this means in EMC's case, they just say
>>"Certified"
>>  -As we don't do any certification ourselves, it would seem impossible
>> for us to certify that any derivative work is compatible.
>>  -It may be best to state that nobody can certify their derivative as
>> "compatible with Apache Hadoop" unless it passes all current test suites
>>  -And require that anyone who declares compatibility define what they
>> mean by this
>>=20
>> This is a good argument for getting more functional tests out there
>> -whoever has more functional tests needs to get them into a test module
>> that can be used to test real deployments.
>>=20


From general-return-3514-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 08:53:16 2011
Return-Path: <general-return-3514-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 780F842EC
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 08:53:16 +0000 (UTC)
Received: (qmail 56208 invoked by uid 500); 13 May 2011 08:53:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56163 invoked by uid 500); 13 May 2011 08:53:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56150 invoked by uid 99); 13 May 2011 08:53:14 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 08:53:14 +0000
Received: from localhost (HELO [90.169.151.44]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 08:53:14 +0000
Message-ID: <4DCCF172.5010504@apache.org>
Date: Fri, 13 May 2011 10:53:06 +0200
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Defining Hadoop Compatibility -revisiting-
References: <C9F227D5.8FA4%mbhandarkar@linkedin.com>
In-Reply-To: <C9F227D5.8FA4%mbhandarkar@linkedin.com>
X-Enigmail-Version: 1.1.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On 05/13/2011 09:24 AM, Milind Bhandarkar wrote:
> IANAL, but Steve is questioning usage of "Apache Hadoop Compatible" in PR
> material of commercial software. Is this considered as usage of "The
> Hadoop mark" ?

It's a usage.  Is it permitted?  Let's consider.

"EMC Greenplum HD Enterprise Edition - The Enterprise Edition is a 100
percent interface-compatible implementation of the Apache Hadoop stack."

The trademark question is whether this creates confusion about what
Hadoop means (not acceptable) or whether it's just a statement about
Hadoop (acceptable).  On one hand, it might be read to say, "EE is
Hadoop", which creates confusion, on the other it might be read to say,
"EE's API is a superset of Hadoop's API", a statement of fact that may
or may not be true.  I think the intended reading is the latter, but it
should probably be stated more clearly: Hadoop has an API but Hadoop is
not its API.  Perhaps we should to ask them to clarify this?

"EMC Greenplum HD Community Edition - The Community Edition is a 100
percent open source certified and supported version of the Apache Hadoop
stack comprising HDFS, MapReduce, Zookeeper, Hive and HBase."

Here "certified" is probably just intended to mean that the software
uses a "certified" open source license, e.g., listed at
http://www.opensource.org/licenses/.  However they should say that this
"includes" or "contains" the various Apache products, not that it "is" them.

Doug

From general-return-3515-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 09:52:04 2011
Return-Path: <general-return-3515-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 87B6D4224
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 09:52:04 +0000 (UTC)
Received: (qmail 16019 invoked by uid 500); 13 May 2011 09:52:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15968 invoked by uid 500); 13 May 2011 09:52:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15960 invoked by uid 99); 13 May 2011 09:52:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 09:52:02 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [145.100.8.49] (HELO smtp.sara.nl) (145.100.8.49)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 09:51:56 +0000
Received: from planck.ka.sara.nl (145.100.8.32) by sara-exch-fe1.ka.sara.nl
 (145.100.8.49) with Microsoft SMTP Server (TLS) id 14.0.702.0; Fri, 13 May
 2011 11:51:27 +0200
Received: from planck.ka.sara.nl ([145.100.8.32]) by planck.ka.sara.nl
 ([145.100.8.32]) with mapi; Fri, 13 May 2011 11:51:07 +0200
From: Evert Lammerts <Evert.Lammerts@sara.nl>
To: 'Eric Fiala' <eric@fiala.ca>, "cdh-user@cloudera.org"
	<cdh-user@cloudera.org>, "general@hadoop.apache.org"
	<general@hadoop.apache.org>
Date: Fri, 13 May 2011 11:49:06 +0200
Subject: RE: Stability issue - dead DN's
Thread-Topic: Stability issue - dead DN's
Thread-Index: AcwP2vzr3MhYPAnJTSu7BVhGQzJBtgBdyqHw
Message-ID: <ADF94D8555C7A246B86A633685E0178AB90A474E93@planck.ka.sara.nl>
References: <ADF94D8555C7A246B86A633685E0178AB90A474E7A@planck.ka.sara.nl>
 <BANLkTi=NaiXQPAkdS-LPW-Swev6XYD0Xjw@mail.gmail.com>
In-Reply-To: <BANLkTi=NaiXQPAkdS-LPW-Swev6XYD0Xjw@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

> From my experience, hadoop loathes swap and you mention that all reduces =
and mappers are running (8 total) and from the ganglia screenshot I see tha=
t you have a thick crest of that purple swap.

I know, it's ugly isn't it :) My understanding is that this is partly due t=
o forked processes though.

> If we do the math that means [ map.tasks.max * mapred.child.java.opts ]  =
+  [ reduce.tasks.max * mapred.child.java.opts ] =3D> or [ 4 * 2.5G ] + [ 4=
 * 2.5G ] is greater than the amount of physical RAM in the machine.
> This doesn't account for the base tasktracker and datanode process + OS o=
verhead and whatever else may be hoarding resources on the systems.

This makes me feel stupid :) Your right, I've just screwed it down, we'll s=
ee how it performs now.

> I would play with this ratio, either less maps / reduces max - or lower y=
our child.java.opts so that when you are fully subscribed you are not using=
 more resource than the machine can offer.
> Also, setting mapred.reduce.slowstart.completed.maps   to 1.00 or some ot=
her value close to 1 would be one way to guarantee only 4 either maps or re=
duces to be running at once and address (albeit in a duct tape like way) th=
e oversubscription problem you are seeing (this represents the fractions of=
 maps that should complete before initiating the reduce phase).

This is a new one for me. I get Allen's point that on a multi tenant cluste=
r this won't fix the problem, but the default is definitely not a good one.=
 Starting reduce tasks as soon as map tasks start running is hardly ever us=
eful, and just takes up slots that could be used by others.

Thanks a bunch for the suggestions!

Cheers,
Evert



On Wed, May 11, 2011 at 3:23 AM, Evert Lammerts <Evert.Lammerts@sara.nl> wr=
ote:
Hi list,

I notice that whenever our Hadoop installation is put under a heavy load we=
 lose one or two (on a total of five) datanodes. This results in IOExceptio=
ns, and affects the overall performance of the job being run. Can anybody g=
ive me advise or best practices on a different configuration to increase th=
e stability? Below I've included the specs of the cluster, the hadoop relat=
ed config and an example of when which things go wrong. Any help is very mu=
ch appreciated, and if I can provide any other info please let me know.

Cheers,
Evert

=3D=3D What goes wrong, and when =3D=3D

See attached a screenshot of Ganglia when the cluster is under load of a si=
ngle job. This job:
* reads ~1TB from HDFS
* writes ~200GB to HDFS
* runs 288 Mappers and 35 Reducers

When the job runs it takes all available Map and Reduce slots. The system s=
tarts swapping and there is a short time interval during which most cores a=
re in WAIT. After that the job really starts running. At around half way, o=
ne or two datanodes become unreachable and are marked as dead nodes. The am=
ount of under-replicated blocks becomes huge. Then some "java.io.IOExceptio=
n: Could not obtain block" are thrown in Mappers. The job does manage to fi=
nish successfully after around 3.5 hours, but my fear is that when we make =
the input much larger - which we want - the system becomes too unstable to =
finish the job.

Maybe worth mentioning - never know what might help diagnostics.  We notice=
 that memory usage becomes less when we switch our keys from Text to LongWr=
itable. Also, the Mappers are done in a fraction of the time. However, this=
 for some reason results in much more network traffic and makes Reducers ex=
tremely slow. We're working on figuring out what causes this.


=3D=3D The cluster =3D=3D

We have a cluster that consists of 6 Sun Thumpers running Hadoop 0.20.2 on =
CentOS 5.5. One of them acts as NN and JT, the other 5 run DN's and TT's. E=
ach node has:
* 16GB RAM
* 32GB swapspace
* 4 cores
* 11 LVM's of 4 x 500GB disks (2TB in total) for HDFS
* non-HDFS stuff on separate disks
* a 2x1GE bonded network interface for interconnects
* a 2x1GE bonded network interface for external access

I realize that this is not a well balanced system, but it's what we had ava=
ilable for a prototype environment. We're working on putting together a spe=
cification for a much larger production environment.


=3D=3D Hadoop config =3D=3D

Here some properties that I think might be relevant:

__CORE-SITE.XML__

fs.inmemory.size.mb: 200
mapreduce.task.io.sort.factor: 100
mapreduce.task.io.sort.mb: 200
# 1024*1024*4 MB, blocksize of the LVM's
io.file.buffer.size: 4194304

__HDFS-SITE.XML__

# 1024*1024*4*32 MB, 32 times the blocksize of the LVM's
dfs.block.size: 134217728
# Only 5 DN's, but this shouldn't hurt
dfs.namenode.handler.count: 40
# This got rid of the occasional "Could not obtain block"'s
dfs.datanode.max.xcievers: 4096

__MAPRED-SITE.XML__

mapred.tasktracker.map.tasks.maximum: 4
mapred.tasktracker.reduce.tasks.maximum: 4
mapred.child.java.opts: -Xmx2560m
mapreduce.reduce.shuffle.parallelcopies: 20
mapreduce.map.java.opts: -Xmx512m
mapreduce.reduce.java.opts: -Xmx512m
# Compression codecs are configured and seem to work fine
mapred.compress.map.output: true
mapred.map.output.compression.codec: com.hadoop.compression.lzo.LzoCodec


From general-return-3516-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 09:57:13 2011
Return-Path: <general-return-3516-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6F6CC43C2
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 09:57:13 +0000 (UTC)
Received: (qmail 25877 invoked by uid 500); 13 May 2011 09:57:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25818 invoked by uid 500); 13 May 2011 09:57:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25810 invoked by uid 99); 13 May 2011 09:57:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 09:57:11 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [145.100.8.49] (HELO smtp.sara.nl) (145.100.8.49)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 09:57:04 +0000
Received: from planck.ka.sara.nl (145.100.8.32) by sara-exch-fe1.ka.sara.nl
 (145.100.8.49) with Microsoft SMTP Server (TLS) id 14.0.702.0; Fri, 13 May
 2011 11:56:42 +0200
Received: from planck.ka.sara.nl ([145.100.8.32]) by planck.ka.sara.nl
 ([145.100.8.32]) with mapi; Fri, 13 May 2011 11:56:42 +0200
From: Evert Lammerts <Evert.Lammerts@sara.nl>
To: "<cdh-user@cloudera.org>" <cdh-user@cloudera.org>,
	"<general@hadoop.apache.org>" <general@hadoop.apache.org>
Date: Fri, 13 May 2011 11:54:40 +0200
Subject: RE: Stability issue - dead DN's
Thread-Topic: Stability issue - dead DN's
Thread-Index: AcwQANbHLbwnC3EHT6WAQGeZF8GBhABUUebQ
Message-ID: <ADF94D8555C7A246B86A633685E0178AB90A474E94@planck.ka.sara.nl>
References: <ADF94D8555C7A246B86A633685E0178AB90A474E7A@planck.ka.sara.nl>
 <BANLkTi=NaiXQPAkdS-LPW-Swev6XYD0Xjw@mail.gmail.com>
 <DA1F03C5-7F7B-412A-84E7-774294085603@apache.org>
 <BLU0-SMTP107D4E21A69A6C860D8CBB2F0860@phx.gbl>
In-Reply-To: <BLU0-SMTP107D4E21A69A6C860D8CBB2F0860@phx.gbl>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi Mike,

> You really really don't want to do this.
> Long story short... It won't work.

Can you elaborate? Are you talking about the bonded interfaces or about hav=
ing a separated network for interconnects and external network? What can go=
 wrong there?

>
> Just a suggestion.. You don't want anyone on your cluster itself. They
> should interact wit edge nodes, which are 'Hadoop aware'. Then your
> cluster has a single network to worry about.

That's our current setup. We have a single headnode that is used as a SPOE.=
 However, I'd like to change that on our future production system. We want =
to implement Kerberos for authentication, and let users interact with the c=
luster from their own machine. This would enable them to submit their jobs =
from the local IDE. The only way to do this is by opening up Hadoop ports f=
or the world, is my understanding: if people interact with HDFS they need t=
o be able to interact with all nodes, right? What would be the argument aga=
inst this?

Cheers,
Evert

>
>
> Sent from a remote device. Please excuse any typos...
>
> Mike Segel
>
> On May 11, 2011, at 11:45 AM, Allen Wittenauer <aw@apache.org> wrote:
>
> >
> >
> >
> >
> >>> * a 2x1GE bonded network interface for interconnects
> >>> * a 2x1GE bonded network interface for external access
> >
> >    Multiple NICs on a box can sometimes cause big performance
> problems with Hadoop.  So watch your traffic carefully.
> >
> >
> >

From general-return-3517-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 12:37:12 2011
Return-Path: <general-return-3517-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6291E43E9
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 12:37:12 +0000 (UTC)
Received: (qmail 55555 invoked by uid 500); 13 May 2011 12:37:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55503 invoked by uid 500); 13 May 2011 12:37:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55495 invoked by uid 99); 13 May 2011 12:37:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 12:37:10 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of msegel@navteq.com designates 204.120.70.37 as permitted sender)
Received: from [204.120.70.37] (HELO xmailfargo.navteq.com) (204.120.70.37)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 12:37:03 +0000
Received: from fargo-spamfw.navteq.com (navteq-spamfw.navteq.com [10.6.10.131])
	by xmailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id p4DC2dnb014017
	for <general@hadoop.apache.org>; Fri, 13 May 2011 07:02:39 -0500
Received: from imailfargo.navteq.com (localhost [127.0.0.1])
	by fargo-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP
	id 432A44970B51; Fri, 13 May 2011 07:36:42 -0500 (CDT)
Received: from imailfargo.navteq.com (imailfargo.navteq.com [10.6.10.130]) by fargo-spamfw.navteq.com with ESMTP id 7LaLSDGUpeU90Ul9; Fri, 13 May 2011 07:36:42 -0500 (CDT)
Received: from hq-ex-ht01.ad.navteq.com ([10.8.222.51])
	by imailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id p4DCafxf024570;
	Fri, 13 May 2011 07:36:41 -0500
Received: from hq-ex-mb03.ad.navteq.com ([fe80::c4dd:7b21:5c22:cfe4]) by
 hq-ex-ht01.ad.navteq.com ([fe80::55a6:5b94:5c00:fe39%12]) with mapi; Fri, 13
 May 2011 07:36:41 -0500
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "<cdh-user@cloudera.org>" <cdh-user@cloudera.org>,
        "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Date: Fri, 13 May 2011 07:36:53 -0500
Subject: Re: Stability issue - dead DN's
Thread-Topic: Stability issue - dead DN's
Thread-Index: AcwRamh2hWksDMtuRoWpOL9mqoklxQ==
Message-ID: <FAAD6507-80D9-453A-BCED-916532688B49@navteq.com>
References: <ADF94D8555C7A246B86A633685E0178AB90A474E7A@planck.ka.sara.nl>
 <BANLkTi=NaiXQPAkdS-LPW-Swev6XYD0Xjw@mail.gmail.com>
 <DA1F03C5-7F7B-412A-84E7-774294085603@apache.org>
 <BLU0-SMTP107D4E21A69A6C860D8CBB2F0860@phx.gbl>
 <ADF94D8555C7A246B86A633685E0178AB90A474E94@planck.ka.sara.nl>
In-Reply-To: <ADF94D8555C7A246B86A633685E0178AB90A474E94@planck.ka.sara.nl>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1
X-Virus-Checked: Checked by ClamAV on apache.org

Bonded will work but you may not see the performance you would expect.  I=
f you need >1 GBe, go 10GBe less headache and has even more headroom.

Multiple interfaces won't work. Or I should say didn't work in past relea=
ses.=20
If you think about it, clients have to connect to each node. So having tw=
o interfaces and trying to manage them makes no sense.=20

Add to this trying to manage this in DNS ... Why make more work for yours=
elf?
Going from memory... It looked like you rDNS had to match you hostnames s=
o your internal interfaces had to match hostnames so you had an inverted =
network.

If you draw out your network topology you end up with a ladder.=20
You would be better off (IMHO) to create a subnet where only your edge se=
rvers are dual nic'd.
But then if your cluster is for development... Now your PCs can't be used=
 as clients...

Does this make sense?


Sent from a remote device. Please excuse any typos...

Mike Segel

On May 13, 2011, at 4:57 AM, "Evert Lammerts" <Evert.Lammerts@sara.nl> wr=
ote:

> Hi Mike,
>=20
>> You really really don't want to do this.
>> Long story short... It won't work.
>=20
> Can you elaborate? Are you talking about the bonded interfaces or about=
 having a separated network for interconnects and external network? What =
can go wrong there?
>=20
>>=20
>> Just a suggestion.. You don't want anyone on your cluster itself. They
>> should interact wit edge nodes, which are 'Hadoop aware'. Then your
>> cluster has a single network to worry about.
>=20
> That's our current setup. We have a single headnode that is used as a S=
POE. However, I'd like to change that on our future production system. We=
 want to implement Kerberos for authentication, and let users interact wi=
th the cluster from their own machine. This would enable them to submit t=
heir jobs from the local IDE. The only way to do this is by opening up Ha=
doop ports for the world, is my understanding: if people interact with HD=
FS they need to be able to interact with all nodes, right? What would be =
the argument against this?
>=20
> Cheers,
> Evert
>=20
>>=20
>>=20
>> Sent from a remote device. Please excuse any typos...
>>=20
>> Mike Segel
>>=20
>> On May 11, 2011, at 11:45 AM, Allen Wittenauer <aw@apache.org> wrote:
>>=20
>>>=20
>>>=20
>>>=20
>>>=20
>>>>> * a 2x1GE bonded network interface for interconnects
>>>>> * a 2x1GE bonded network interface for external access
>>>=20
>>>   Multiple NICs on a box can sometimes cause big performance
>> problems with Hadoop.  So watch your traffic carefully.
>>>=20
>>>=20
>>>=20


The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.

From general-return-3518-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 13:44:27 2011
Return-Path: <general-return-3518-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 31BC04C05
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 13:44:27 +0000 (UTC)
Received: (qmail 79650 invoked by uid 500); 13 May 2011 13:44:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79605 invoked by uid 500); 13 May 2011 13:44:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 79597 invoked by uid 99); 13 May 2011 13:44:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 13:44:25 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.220.176] (HELO mail-vx0-f176.google.com) (209.85.220.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 13:44:19 +0000
Received: by vxa37 with SMTP id 37so2903241vxa.35
        for <general@hadoop.apache.org>; Fri, 13 May 2011 06:43:58 -0700 (PDT)
Received: by 10.52.110.135 with SMTP id ia7mr2042845vdb.294.1305294238058;
 Fri, 13 May 2011 06:43:58 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.52.112.7 with HTTP; Fri, 13 May 2011 06:43:37 -0700 (PDT)
X-Originating-IP: [67.160.196.149]
In-Reply-To: <4DCCF172.5010504@apache.org>
References: <C9F227D5.8FA4%mbhandarkar@linkedin.com> <4DCCF172.5010504@apache.org>
From: Ted Dunning <tdunning@maprtech.com>
Date: Fri, 13 May 2011 06:43:37 -0700
Message-ID: <BANLkTi=5Nn91bUmpxq=t7VSmAg0pSbO0tQ@mail.gmail.com>
Subject: Re: Defining Hadoop Compatibility -revisiting-
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec54860726309c704a3287fc4

--bcaec54860726309c704a3287fc4
Content-Type: text/plain; charset=ISO-8859-1

I thought the word "comprising" meant includes, not is.

On Fri, May 13, 2011 at 1:53 AM, Doug Cutting <cutting@apache.org> wrote:

> "EMC Greenplum HD Community Edition - The Community Edition is a 100
> percent open source certified and supported version of the Apache Hadoop
> stack comprising HDFS, MapReduce, Zookeeper, Hive and HBase."
>
> Here "certified" is probably just intended to mean that the software
> uses a "certified" open source license, e.g., listed at
> http://www.opensource.org/licenses/.  However they should say that this
> "includes" or "contains" the various Apache products, not that it "is"
> them.
>

--bcaec54860726309c704a3287fc4--

From general-return-3519-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 14:42:02 2011
Return-Path: <general-return-3519-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1A1224200
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 14:42:02 +0000 (UTC)
Received: (qmail 3160 invoked by uid 500); 13 May 2011 14:42:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3106 invoked by uid 500); 13 May 2011 14:42:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3097 invoked by uid 99); 13 May 2011 14:42:00 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 14:42:00 +0000
Received: from localhost (HELO mail-qy0-f176.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 14:41:59 +0000
Received: by qyk30 with SMTP id 30so1888006qyk.14
        for <general@hadoop.apache.org>; Fri, 13 May 2011 07:41:58 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.78.22 with SMTP id i22mr1274301qck.33.1305297718371; Fri,
 13 May 2011 07:41:58 -0700 (PDT)
Received: by 10.229.189.9 with HTTP; Fri, 13 May 2011 07:41:58 -0700 (PDT)
Date: Fri, 13 May 2011 07:41:58 -0700
Message-ID: <BANLkTi=1+++j55GER6Tvuw=pRbvApKmy-g@mail.gmail.com>
Subject: What is "Hadoop?" Was: Defining Hadoop Compatibility -revisiting-
From: "Owen O'Malley" <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00235429d8f4d467c904a3294eda

--00235429d8f4d467c904a3294eda
Content-Type: text/plain; charset=UTF-8

On Tue, May 10, 2011 at 3:29 AM, Steve Loughran <stevel@apache.org> wrote:

> I think we should revisit this issue before people with their own agendas
> define what compatibility with Apache Hadoop is for us
>

I agree completely. As you point out, this week we've had a flood of
products calling themselves "Hadoop" or "Distribution of Hadoop" that
include only a part of Hadoop. This is will dilute Apache's Hadoop trademark
and create consumer confusion.

Licensing
> -Use of the Hadoop codebase must follow the Apache License
> http://www.apache.org/licenses/LICENSE-2.0
> -plug in components that are dynamically linked to (Filesystems and
> schedulers) don't appear to be derivative works on my reading of this,
>

+1
Plugins are usually considered independent works. Note that the Apache
license does permit commercial closed-source derivative works. A company
could take Hadoop's code, modify it, and sell a binary release as long as
they meet the conditions of the Apache license.


> Naming
>  -this is something for branding@apache, they will have their opinions.
> The key one is that the name "Apache Hadoop" must get used, and it's
> important to make clear it is a derivative work.
>  -I don't think you can claim to have a Distribution/Fork/Version of Apache
> Hadoop if you swap out big chunks of it for alternate filesystems, MR
> engines, etc. Some description of this is needed
> "Supports the Apache Hadoop MapReduce engine on top of Filesystem XYZ"
>

The Hadoop name is the primary tool that the project has for minimizing
customer confusion. I think we need to create a very clear definition of
what can be called Hadoop and what can not. Apache gives the PMCs a fair
amount of latitude in picking the policy for their project name and I think
we need to do so.

Given the large number of so-called Hadoop products that are being released,
I believe that we should require "Hadoop" to mean specifically the Apache
Hadoop releases (possibly with a few critical security patches).

Projects that are derivative works can either be "powered by Apache Hadoop,"
or "based on Apache Hadoop."

What do others think?

-- Owen

--00235429d8f4d467c904a3294eda--

From general-return-3520-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 14:50:43 2011
Return-Path: <general-return-3520-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E782C49FB
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 14:50:42 +0000 (UTC)
Received: (qmail 29370 invoked by uid 500); 13 May 2011 14:50:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29177 invoked by uid 500); 13 May 2011 14:50:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29168 invoked by uid 99); 13 May 2011 14:50:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 14:50:41 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cutting@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 14:50:35 +0000
Received: by vws7 with SMTP id 7so2997632vws.35
        for <general@hadoop.apache.org>; Fri, 13 May 2011 07:50:14 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=ICq4A5MgxF1FimDwlwQFNDlFj8MEwzheCzJdHwtmNYI=;
        b=lCnKOQwdpFq54L1j+2DnZhcgLhI79ofVegJS+gbRgfQw2Tr2S17SKdS2x3DUSPDJl+
         mQmJMU3MZKNb9qU8E8S8/FsrRq+NrN1Kj/RpC430ZcaNS8On0MxoE44oBKZON9D3I5XE
         8ditRtwulW0/Ne3LMYXt1zam9O4R/e4gDzrjM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=ba9eq+pzOEFexyLQiK1TlbGTijZ4SBAwKUeTd6BrOIDqxnOl4q3+I0nqgKJfHukXE0
         HoJNaeXBeSu416p33xw6CVLaYrK3q1pKGXJP0fXQhu8AKQoaLpMnfdiVsnXS4NrhNsKq
         IIm4Pvo+oEuqS2WZ+LznbrzNXuupXpP777M5I=
MIME-Version: 1.0
Received: by 10.52.76.10 with SMTP id g10mr2141351vdw.252.1305298214241; Fri,
 13 May 2011 07:50:14 -0700 (PDT)
Received: by 10.52.156.170 with HTTP; Fri, 13 May 2011 07:50:14 -0700 (PDT)
Received: by 10.52.156.170 with HTTP; Fri, 13 May 2011 07:50:14 -0700 (PDT)
In-Reply-To: <BANLkTi=5Nn91bUmpxq=t7VSmAg0pSbO0tQ@mail.gmail.com>
References: <C9F227D5.8FA4%mbhandarkar@linkedin.com>
	<4DCCF172.5010504@apache.org>
	<BANLkTi=5Nn91bUmpxq=t7VSmAg0pSbO0tQ@mail.gmail.com>
Date: Fri, 13 May 2011 16:50:14 +0200
Message-ID: <BANLkTinEB=Vi_gZnu1FihnFA_eGXy8U2oQ@mail.gmail.com>
Subject: Re: Defining Hadoop Compatibility -revisiting-
From: Doug Cutting <cutting@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf3071d06662c48f04a3296c4b
X-Virus-Checked: Checked by ClamAV on apache.org

--20cf3071d06662c48f04a3296c4b
Content-Type: text/plain; charset=UTF-8

Yes, but there's an "is" earlier in the sentence.

Doug
On May 13, 2011 3:44 PM, "Ted Dunning" <tdunning@maprtech.com> wrote:
> I thought the word "comprising" meant includes, not is.
>
> On Fri, May 13, 2011 at 1:53 AM, Doug Cutting <cutting@apache.org> wrote:
>
>> "EMC Greenplum HD Community Edition - The Community Edition is a 100
>> percent open source certified and supported version of the Apache Hadoop
>> stack comprising HDFS, MapReduce, Zookeeper, Hive and HBase."
>>
>> Here "certified" is probably just intended to mean that the software
>> uses a "certified" open source license, e.g., listed at
>> http://www.opensource.org/licenses/. However they should say that this
>> "includes" or "contains" the various Apache products, not that it "is"
>> them.
>>

--20cf3071d06662c48f04a3296c4b--

From general-return-3521-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 15:20:17 2011
Return-Path: <general-return-3521-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 772864D36
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 15:20:17 +0000 (UTC)
Received: (qmail 253 invoked by uid 500); 13 May 2011 15:20:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 108 invoked by uid 500); 13 May 2011 15:20:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 100 invoked by uid 99); 13 May 2011 15:20:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 15:20:15 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 15:20:08 +0000
Received: from sp1-ex07cas03.ds.corp.yahoo.com (sp1-ex07cas03.ds.corp.yahoo.com [216.252.116.151])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p4DFJd7Z057982
	for <general@hadoop.apache.org>; Fri, 13 May 2011 08:19:39 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1305299979;
	bh=wFYvHHcwAnUStGDWyErC8FtPgbVzWFs2aZzQ473CXIU=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:MIME-Version;
	b=HWWJXGE6Azxv0SM0WZg/KBxRh2cqE9E7JMAI4nTmW5ztw/2XzrUK3N6LDIstVTvc5
	 +8iCH7plt0D8QTGn3zJ/lCf7lss/kWxW80b17p7KQKmWf59dQwcQ2T5jg4suvNhjvk
	 TbTF7CE6xxylX7wvh9KQ+91JW6ffGa38xIHT9o1Q=
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas03.ds.corp.yahoo.com ([216.252.116.151]) with mapi; Fri, 13 May
 2011 08:19:38 -0700
From: Nathan Roberts <nroberts@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 13 May 2011 08:19:30 -0700
Subject: Re: Defining Hadoop Compatibility -revisiting-
Thread-Topic: Defining Hadoop Compatibility -revisiting-
Thread-Index: AcwRfTDyAeDQKalfQJipUYQWWspNuQAA/aoJ
Message-ID: <C9F2B632.111D6%nroberts@yahoo-inc.com>
In-Reply-To: <BANLkTinEB=Vi_gZnu1FihnFA_eGXy8U2oQ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Key seems to be how one would interpret "version". Replace it with a synony=
m
like "variant" and this may be the intent.



On 5/13/11 9:50 AM, "Doug Cutting" <cutting@gmail.com> wrote:

> Yes, but there's an "is" earlier in the sentence.
>=20
> Doug
> On May 13, 2011 3:44 PM, "Ted Dunning" <tdunning@maprtech.com> wrote:
>> I thought the word "comprising" meant includes, not is.
>>=20
>> On Fri, May 13, 2011 at 1:53 AM, Doug Cutting <cutting@apache.org> wrote=
:
>>=20
>>> "EMC Greenplum HD Community Edition - The Community Edition is a 100
>>> percent open source certified and supported version of the Apache Hadoo=
p
>>> stack comprising HDFS, MapReduce, Zookeeper, Hive and HBase."
>>>=20
>>> Here "certified" is probably just intended to mean that the software
>>> uses a "certified" open source license, e.g., listed at
>>> http://www.opensource.org/licenses/. However they should say that this
>>> "includes" or "contains" the various Apache products, not that it "is"
>>> them.
>>>=20
>=20


From general-return-3522-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 17:28:46 2011
Return-Path: <general-return-3522-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 99B7A479D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 17:28:46 +0000 (UTC)
Received: (qmail 99826 invoked by uid 500); 13 May 2011 17:28:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99778 invoked by uid 500); 13 May 2011 17:28:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 99770 invoked by uid 99); 13 May 2011 17:28:45 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 17:28:45 +0000
Received: from localhost (HELO [172.16.37.108]) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 17:28:44 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Defining Hadoop Compatibility -revisiting-
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <4DCCF172.5010504@apache.org>
Date: Fri, 13 May 2011 10:28:41 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <99AB77CF-025E-4E4A-BD7F-BDA362E633BF@apache.org>
References: <C9F227D5.8FA4%mbhandarkar@linkedin.com> <4DCCF172.5010504@apache.org>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On May 13, 2011, at 1:53 AM, Doug Cutting wrote:
> Here "certified" is probably just intended to mean that the software
> uses a "certified" open source license, e.g., listed at
> http://www.opensource.org/licenses/.  However they should say that =
this
> "includes" or "contains" the various Apache products, not that it "is" =
them.

	If it has a modified version of Hadoop (i.e., not an actual =
Apache release or patches which have never been committed to trunk), are =
they allowed to say "includes Apache Hadoop"?  At what point is it not =
Apache Hadoop?


From general-return-3523-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 17:33:06 2011
Return-Path: <general-return-3523-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D1F3B4725
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 17:33:06 +0000 (UTC)
Received: (qmail 9768 invoked by uid 500); 13 May 2011 17:33:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 9716 invoked by uid 500); 13 May 2011 17:33:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 9708 invoked by uid 99); 13 May 2011 17:33:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 17:33:05 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of msegel@navteq.com designates 65.167.11.240 as permitted sender)
Received: from [65.167.11.240] (HELO xmailchi.navteq.com) (65.167.11.240)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 17:33:00 +0000
Received: from chicago-spamfw.navteq.com (imailchi [10.8.5.161])
	by xmailchi.navteq.com (8.13.8/8.13.6) with ESMTP id p4DHWdjL025672
	for <general@hadoop.apache.org>; Fri, 13 May 2011 12:32:39 -0500
Received: from imailchi.hq.navteq.com (localhost [127.0.0.1])
	by chicago-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id C014D442246F
	for <general@hadoop.apache.org>; Fri, 13 May 2011 12:32:38 -0500 (CDT)
Received: from imailchi.hq.navteq.com (imailchi.hq.navteq.com [10.8.120.104]) by chicago-spamfw.navteq.com with ESMTP id NJhjIqIDdHc7SKDL for <general@hadoop.apache.org>; Fri, 13 May 2011 12:32:38 -0500 (CDT)
Received: from hq-ex-ht02.ad.navteq.com (hq-ex-ht02.ad.navteq.com [10.8.222.172])
	by imailchi.hq.navteq.com (8.12.9/8.12.9) with ESMTP id p4DHWcNe021920;
	Fri, 13 May 2011 12:32:38 -0500
Received: from hq-ex-mb03.ad.navteq.com ([fe80::c4dd:7b21:5c22:cfe4]) by
 hq-ex-ht02.ad.navteq.com ([fe80::24f6:ca50:734d:df6e%13]) with mapi; Fri, 13
 May 2011 12:32:38 -0500
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 13 May 2011 12:32:52 -0500
Subject: Re: Defining Hadoop Compatibility -revisiting-
Thread-Topic: Defining Hadoop Compatibility -revisiting-
Thread-Index: AcwRk8D/d408HxkoREGgU8PWykB5Jw==
Message-ID: <D510AC3F-1AD8-4595-9C54-E0DF3EB9FE25@navteq.com>
References: <C9F227D5.8FA4%mbhandarkar@linkedin.com>
 <4DCCF172.5010504@apache.org>
 <99AB77CF-025E-4E4A-BD7F-BDA362E633BF@apache.org>
In-Reply-To: <99AB77CF-025E-4E4A-BD7F-BDA362E633BF@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1

My first read was that they used the term Apache Hadoop in reference of A=
pache's release. They referenced their release as Hadoop.


Sent from a remote device. Please excuse any typos...

Mike Segel

On May 13, 2011, at 12:28 PM, "Allen Wittenauer" <aw@apache.org> wrote:

>=20
> On May 13, 2011, at 1:53 AM, Doug Cutting wrote:
>> Here "certified" is probably just intended to mean that the software
>> uses a "certified" open source license, e.g., listed at
>> http://www.opensource.org/licenses/.  However they should say that thi=
s
>> "includes" or "contains" the various Apache products, not that it "is"=
 them.
>=20
>    If it has a modified version of Hadoop (i.e., not an actual Apache r=
elease or patches which have never been committed to trunk), are they all=
owed to say "includes Apache Hadoop"?  At what point is it not Apache Had=
oop?
>=20


The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.

From general-return-3524-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 17:48:25 2011
Return-Path: <general-return-3524-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E1B55456C
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 17:48:25 +0000 (UTC)
Received: (qmail 35485 invoked by uid 500); 13 May 2011 17:48:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 35437 invoked by uid 500); 13 May 2011 17:48:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 35429 invoked by uid 99); 13 May 2011 17:48:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 17:48:24 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 17:48:18 +0000
Received: by pwi16 with SMTP id 16so1853784pwi.35
        for <general@hadoop.apache.org>; Fri, 13 May 2011 10:47:56 -0700 (PDT)
Received: by 10.68.3.139 with SMTP id c11mr2505282pbc.277.1305308876070; Fri,
 13 May 2011 10:47:56 -0700 (PDT)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.68.62.230 with HTTP; Fri, 13 May 2011 10:47:36 -0700 (PDT)
In-Reply-To: <C9F22405.8F84%mbhandarkar@linkedin.com>
References: <BANLkTi=4LOHwbJP6qqwKSN0MJ+OSTYtnaQ@mail.gmail.com> <C9F22405.8F84%mbhandarkar@linkedin.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Fri, 13 May 2011 10:47:36 -0700
X-Google-Sender-Auth: 7js-RwcawQVZPTFGNoEJHLVYySI
Message-ID: <BANLkTinrhUpwA=3jonnrn8Bj5mB2mQpsFA@mail.gmail.com>
Subject: Re: Defining Hadoop Compatibility -revisiting-
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Fri, May 13, 2011 at 00:11, Milind Bhandarkar
<mbhandarkar@linkedin.com> wrote:
> Cos,
>
> I remember the issues about the "inter-component interactions" at that
> point when you were part of the Yahoo Hadoop FIT team (I was on the other
> side of the same floor, remember ? ;-)

Vaguely ;) Of course I remember. But I prefer not to mentioned any
internal technologies developed for private companies after getting
lashes for that.

> Things like, "Can Pig take full URIs as input, and so works with viewfs",
> "Can Local jobtracker still use HDFS as input and output", "Can Oozie use
> local file system to keep workflows, while the jars were located on hdfs"
> etc came up often.
>
> Each of these issues were component-interaction issues, and were results
> of making DistributedFileSystem a public class, or some subtle dependency
> on the semantics of a particular method in an interface, which were not
> explicit in the syntax.
>
> That's an issue with interface-compatibility, and so merely compiling
> against a particular interface is not a solution. One needs a test-suite.

One needs more than a mere test-suite if experience teaches us
anything. FIT and its continuation turns to be a complex program (not
only in a sense of computer code) with many moving parts, bells and
whistles. One of those was a set of specs actually written in English
language. The downside is that someone needs to keep them up to day,
translate them into test cases or teach others how to do it, etc. That
exactly why TCK was using a test generator and used somewhat
formalized spec language.

Cos

> (With annotations in Java, one can impose more semantic restrictions on
> the interface, that can be automatically checked against at runtime. But
> is limited to individual methods, or the full class. Code generation usin=
g
> perl or whatever is similar in capability.)
>
> - milind
> --
> Milind Bhandarkar
> mbhandarkar@linkedin.com
> +1-650-776-3167
>
>
>
>
>
>
> On 5/12/11 11:24 PM, "Konstantin Boudnik" <cos@apache.org> wrote:
>
>>On Thu, May 12, 2011 at 20:40, Milind Bhandarkar
>><mbhandarkar@linkedin.com> wrote:
>>> Cos,
>>>
>>> Can you give me an example of a "system test" that is not a functional
>>> test ? My assumption was that the functionality being tested is specifi=
c
>>> to a component, and that inter-component interactions (that's what you
>>> meant, right?) would be taken care by the public interface and semantic=
s
>>> of a component API.
>>
>>Milind, kinda... However, to exercise inter-component interactions via
>>component APIs one needs to have tests which are beyond functional or
>>component realm (e.g. system). At some point =A0I was part of a team
>>working on integration validation framework for Hadoop (FIT) which was
>>addressing inter-component interaction validations essentially
>>guaranteeing their compatibility. Components being Hadoop, Pig, Oozie,
>>etc. - thus massaging the whole stack of application and covering a
>>lot of use cases.
>>
>>Having a framework like this and a set of test cases available for
>>Hadoop community is a great benefit because one can quickly make sure
>>that a Hadoop stack built from a set of components is working
>>property. Another use case is to run the same set of tests - versioned
>>separately from the product itself - against previous and a next
>>release validating their compatibility at the functional level (sorta
>>what you have mentioned).
>>
>>This doesn't by the way deploy if we'd choose to work on HCK or not,
>>however HCK might be eventually based on top of such a framework.
>>
>>Cos
>>
>>> - milind
>>>
>>> --
>>> Milind Bhandarkar
>>> mbhandarkar@linkedin.com
>>> +1-650-776-3167
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 5/12/11 3:30 PM, "Konstantin Boudnik" <cos@apache.org> wrote:
>>>
>>>>On Thu, May 12, 2011 at 09:45, Milind Bhandarkar
>>>><mbhandarkar@linkedin.com> wrote:
>>>>> HCK and written specifications are not mutually exclusive. However,
>>>>>given
>>>>> the evolving nature of Hadoop APIs, functional tests need to evolve a=
s
>>>>
>>>>I would actually expand it to 'functional and system tests' because
>>>>latter are capable of validating inter-component iterations not
>>>>coverable by functional tests.
>>>>
>>>>Cos
>>>>
>>>>> well, and having them tied to a "current stable" version is easier to
>>>>>do
>>>>> than it is to tie the written specifications.
>>>>>
>>>>> - milind
>>>>>
>>>>> --
>>>>> Milind Bhandarkar
>>>>> mbhandarkar@linkedin.com
>>>>> +1-650-776-3167
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 5/11/11 7:26 PM, "M. C. Srivas" <mcsrivas@gmail.com> wrote:
>>>>>
>>>>>>While the HCK is a great idea to check quickly if an implementation i=
s
>>>>>>"compliant", =A0we still need a written specification to define what =
is
>>>>>>meant
>>>>>>by compliance, something akin to a set of RFC's, or a set of docs lik=
e
>>>>>>the
>>>>>> IEEE POSIX specifications.
>>>>>>
>>>>>>For example, the POSIX.1c pthreads API has a written document that
>>>>>>specifies
>>>>>>all the function calls, input params, return values, and error codes.
>>>>>>It
>>>>>>clearly indicates what any POSIX-complaint threads package needs to
>>>>>>support,
>>>>>>and what are vendor-specific non-portable extensions that one can use
>>>>>>at
>>>>>>one's own risk.
>>>>>>
>>>>>>Currently we have 2 sets of API =A0in the DFS and Map/Reduce layers, =
and
>>>>>>the
>>>>>>specification is extracted only by looking at the code, or (where the
>>>>>>code
>>>>>>is non-trivial) by writing really bizarre test programs to examine
>>>>>>corner
>>>>>>cases. Further, the interaction between a mix of the old and new APIs
>>>>>>is
>>>>>>not
>>>>>>specified anywhere. Such specifications are vitally important when
>>>>>>implementing libraries like Cascading, Mahout, etc. For example, an
>>>>>>application might open a file using the new API, and pass that stream
>>>>>>into a
>>>>>>library that manipulates the stream using some of the old API ... wha=
t
>>>>>>is
>>>>>>then the expectation of the state of the stream when the library call
>>>>>>returns?
>>>>>>
>>>>>>Sanjay Radia @ Y! already started specifying some the DFS APIs to nai=
l
>>>>>>such
>>>>>>things down. There's similar good effort in the Map/Reduce and =A0Avr=
o
>>>>>>spaces,
>>>>>>but it seems to have stalled somewhat. We should continue it.
>>>>>>
>>>>>>Doing such specs would be a great service to the community and the
>>>>>>users
>>>>>>of
>>>>>>Hadoop. It provides them
>>>>>> =A0 (a) clear-cut docs on how to use the Hadoop APIs
>>>>>> =A0 (b) wider choice of Hadoop implementations by freeing them from
>>>>>>vendor
>>>>>>lock-in.
>>>>>>
>>>>>>Once we have such specification, the HCK becomes meaningful (since th=
e
>>>>>>HCK
>>>>>>itself will be buggy initially).
>>>>>>
>>>>>>
>>>>>>On Wed, May 11, 2011 at 2:46 PM, Milind Bhandarkar
>>>>>><mbhandarkar@linkedin.com
>>>>>>> wrote:
>>>>>>
>>>>>>> I think it's time to separate out functional tests as a "Hadoop
>>>>>>> Compatibility Kit (HCK)", similar to the Sun TCK for Java, but unde=
r
>>>>>>>ASL
>>>>>>> 2.0. Then "certification" would mean "Passes 100% of the HCK
>>>>>>>testsuite."
>>>>>>>
>>>>>>> - milind
>>>>>>> --
>>>>>>> Milind Bhandarkar
>>>>>>> mbhandarkar@linkedin.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 5/11/11 2:24 PM, "Eric Baldeschwieler" <eric14@yahoo-inc.com>
>>>>>>>wrote:
>>>>>>>
>>>>>>> >This is a really interesting topic! =A0I completely agree that we
>>>>>>>need
>>>>>>>to
>>>>>>> >get ahead of this.
>>>>>>> >
>>>>>>> >I would be really interested in learning of any experience other
>>>>>>>apache
>>>>>>> >projects, such as apache or tomcat have with these issues.
>>>>>>> >
>>>>>>> >---
>>>>>>> >E14 - typing on glass
>>>>>>> >
>>>>>>> >On May 10, 2011, at 6:31 AM, "Steve Loughran" <stevel@apache.org>
>>>>>>>wrote:
>>>>>>> >
>>>>>>> >>
>>>>>>> >> Back in Jan 2011, I started a discussion about how to define
>>>>>>>Apache
>>>>>>> >> Hadoop Compatibility:
>>>>>>> >>
>>>>>>> >>
>>>>>>>
>>>>>>>http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/=
%
>>>>>>>3C
>>>>>>>4D
>>>>>>> >>46B6AD.2020802@apache.org%3E
>>>>>>> >>
>>>>>>> >> I am now reading EMC HD "Enterprise Ready" Apache Hadoop
>>>>>>>datasheet
>>>>>>> >>
>>>>>>> >>
>>>>>>>
>>>>>>>>>http://www.greenplum.com/sites/default/files/EMC_Greenplum_HD_DS_F=
i
>>>>>>>>>na
>>>>>>>>>l_
>>>>>>>>>1
>>>>>>> .
>>>>>>> >>pdf
>>>>>>> >>
>>>>>>> >> It claims that their implementations are 100% compatible, even
>>>>>>>though
>>>>>>> >> the Enterprise edition uses a C filesystem. It also claims that
>>>>>>>both
>>>>>>> >> their software releases contain "Certified Stacks", without
>>>>>>>defining
>>>>>>> >> what Certified means, or who does the certification -only that i=
t
>>>>>>>is
>>>>>>>an
>>>>>>> >> improvement.
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> I think we should revisit this issue before people with their ow=
n
>>>>>>> >> agendas define what compatibility with Apache Hadoop is for us
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> Licensing
>>>>>>> >> -Use of the Hadoop codebase must follow the Apache License
>>>>>>> >> http://www.apache.org/licenses/LICENSE-2.0
>>>>>>> >> -plug in components that are dynamically linked to (Filesystems
>>>>>>>and
>>>>>>> >> schedulers) don't appear to be derivative works on my reading of
>>>>>>>this,
>>>>>>> >>
>>>>>>> >> Naming
>>>>>>> >> =A0-this is something for branding@apache, they will have their
>>>>>>>opinions.
>>>>>>> >> The key one is that the name "Apache Hadoop" must get used, and
>>>>>>>it's
>>>>>>> >> important to make clear it is a derivative work.
>>>>>>> >> =A0-I don't think you can claim to have a Distribution/Fork/Vers=
ion
>>>>>>>of
>>>>>>> >> Apache Hadoop if you swap out big chunks of it for alternate
>>>>>>> >> filesystems, MR engines, etc. Some description of this is needed
>>>>>>> >> "Supports the Apache Hadoop MapReduce engine on top of Filesyste=
m
>>>>>>>XYZ"
>>>>>>> >>
>>>>>>> >> Compatibility
>>>>>>> >> =A0-the definition of the Hadoop interfaces and classes is the
>>>>>>>Apache
>>>>>>> >> Source tree,
>>>>>>> >> =A0-the definition of semantics of the Hadoop interfaces and
>>>>>>>classes
>>>>>>>is
>>>>>>> >> the Apache Source tree, including the test classes.
>>>>>>> >> =A0-the verification that the actual semantics of an Apache Hado=
op
>>>>>>> >> release is compatible with the expected semantics is that curren=
t
>>>>>>>and
>>>>>>> >> future tests pass
>>>>>>> >> =A0-bug reports can highlight incompatibility with expectations =
of
>>>>>>> >> community users, and once incorporated into tests form part of
>>>>>>>the
>>>>>>> >> compatibility testing
>>>>>>> >> =A0-vendors can claim and even certify their derivative works as
>>>>>>> >> compatible with other versions of their derivative works, but
>>>>>>>cannot
>>>>>>> >> claim compatibility with Apache Hadoop unless their code passes
>>>>>>>the
>>>>>>> >> tests and is consistent with the bug reports marked as ("by
>>>>>>>design").
>>>>>>> >> Perhaps we should have tests that verify each of these "by
>>>>>>>design"
>>>>>>> >> bugreps to make them more formal.
>>>>>>> >>
>>>>>>> >> Certification
>>>>>>> >> =A0-I have no idea what this means in EMC's case, they just say
>>>>>>> >>"Certified"
>>>>>>> >> =A0-As we don't do any certification ourselves, it would seem
>>>>>>>impossible
>>>>>>> >> for us to certify that any derivative work is compatible.
>>>>>>> >> =A0-It may be best to state that nobody can certify their
>>>>>>>derivative
>>>>>>>as
>>>>>>> >> "compatible with Apache Hadoop" unless it passes all current tes=
t
>>>>>>>suites
>>>>>>> >> =A0-And require that anyone who declares compatibility define wh=
at
>>>>>>>they
>>>>>>> >> mean by this
>>>>>>> >>
>>>>>>> >> This is a good argument for getting more functional tests out
>>>>>>>there
>>>>>>> >> -whoever has more functional tests needs to get them into a test
>>>>>>>module
>>>>>>> >> that can be used to test real deployments.
>>>>>>> >>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>>
>
>

From general-return-3525-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 21:15:40 2011
Return-Path: <general-return-3525-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4D03741F8
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 21:15:40 +0000 (UTC)
Received: (qmail 94560 invoked by uid 500); 13 May 2011 21:15:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94513 invoked by uid 500); 13 May 2011 21:15:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94505 invoked by uid 99); 13 May 2011 21:15:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 21:15:38 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [145.100.8.49] (HELO smtp.sara.nl) (145.100.8.49)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 21:15:33 +0000
Received: from planck.ka.sara.nl (145.100.8.32) by sara-exch-fe1.ka.sara.nl
 (145.100.8.49) with Microsoft SMTP Server (TLS) id 14.0.702.0; Fri, 13 May
 2011 23:15:09 +0200
Received: from planck.ka.sara.nl ([145.100.8.32]) by planck.ka.sara.nl
 ([145.100.8.32]) with mapi; Fri, 13 May 2011 23:14:48 +0200
From: Evert Lammerts <Evert.Lammerts@sara.nl>
To: "cdh-user@cloudera.org" <cdh-user@cloudera.org>,
	"general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 13 May 2011 23:14:47 +0200
Subject: RE: Stability issue - dead DN's
Thread-Topic: Stability issue - dead DN's
Thread-Index: AcwRamh2hWksDMtuRoWpOL9mqoklxQARO6Jr
Message-ID: <ADF94D8555C7A246B86A633685E0178AB90A35CA07@planck.ka.sara.nl>
References: <ADF94D8555C7A246B86A633685E0178AB90A474E7A@planck.ka.sara.nl>
 <BANLkTi=NaiXQPAkdS-LPW-Swev6XYD0Xjw@mail.gmail.com>
 <DA1F03C5-7F7B-412A-84E7-774294085603@apache.org>
 <BLU0-SMTP107D4E21A69A6C860D8CBB2F0860@phx.gbl>
 <ADF94D8555C7A246B86A633685E0178AB90A474E94@planck.ka.sara.nl>,<FAAD6507-80D9-453A-BCED-916532688B49@navteq.com>
In-Reply-To: <FAAD6507-80D9-453A-BCED-916532688B49@navteq.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi Mike,

Thanks for trying to help out.

I had a talk with our networking guys this afternoon. According to them (an=
d this is way out of my area of expertise, so excuse any mistakes) multiple=
 interfaces shouldn't be a problem. We could set up a nameserver to resolve=
 hostnames to addresses in our private space when the request comes from on=
e of the nodes, and route this traffic over a single interface. Any other r=
equest can be resolved to an address in the public space, which is bound to=
 an other interface. In our current setup we're not even resolving hostname=
s in our private address space through a nameserver - we do it with an ugly=
 hack in /etc/hosts. And it seems to work alright.

Having said that, our problems are still not completely gone even after adj=
usting the maximum allowed RAM for tasks - although things are lots better.=
 While writing this mail three out of five DN's were marked as dead. There =
still is some swapping going on, but the cores are not spending any time in=
 WAIT, so this shouldn't be the cause of anything. See below a trace from a=
 dead DN - any thoughts are appreciated!

Cheers,
Evert

2011-05-13 23:13:27,716 INFO org.apache.hadoop.hdfs.server.datanode.DataNod=
e: Received block blk_-9131821326787012529_2915672 src: /192.168.28.211:601=
36 dest: /192.168.28.214:50050 of size 382425
2011-05-13 23:13:27,915 INFO org.apache.hadoop.hdfs.server.datanode.DataNod=
e: Exception in receiveBlock for block blk_-9132067116195286882_130888 java=
.io.EOFException: while trying to read 3744913 bytes
2011-05-13 23:13:27,925 INFO org.apache.hadoop.hdfs.server.datanode.DataNod=
e.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.214:35139, byt=
es: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001437_0=
, offset: 196608, srvID: DS-443352839-145.100.2.183-50050-1291128673616, bl=
ockid: blk_-9163184839986480695_4112368, duration: 6254000
2011-05-13 23:13:28,032 INFO org.apache.hadoop.hdfs.server.datanode.DataNod=
e: Received block blk_-9149862728087355005_3793421 src: /192.168.28.210:411=
97 dest: /192.168.28.214:50050 of size 245767
2011-05-13 23:13:28,033 WARN org.apache.hadoop.hdfs.server.datanode.DataNod=
e: Block blk_-9132067116195286882_130888 unfinalized and removed.=20
2011-05-13 23:13:28,033 INFO org.apache.hadoop.hdfs.server.datanode.DataNod=
e: writeBlock blk_-9132067116195286882_130888 received exception java.io.EO=
FException: while trying to read 3744913 bytes
2011-05-13 23:13:28,033 ERROR org.apache.hadoop.hdfs.server.datanode.DataNo=
de: DatanodeRegistration(192.168.28.214:50050, storageID=3DDS-443352839-145=
.100.2.183-50050-1291128673616, infoPort=3D50075, ipcPort=3D50020):DataXcei=
ver
java.io.EOFException: while trying to read 3744913 bytes
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(B=
lockReceiver.java:270)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPac=
ket(BlockReceiver.java:357)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePack=
et(BlockReceiver.java:378)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBloc=
k(BlockReceiver.java:534)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(Da=
taXceiver.java:417)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiv=
er.java:122)
2011-05-13 23:13:28,038 INFO org.apache.hadoop.hdfs.server.datanode.DataNod=
e.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.214:32910, byt=
es: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001443_0=
, offset: 197632, srvID: DS-443352839-145.100.2.183-50050-1291128673616, bl=
ockid: blk_-9163184839986480695_4112368, duration: 4323000
2011-05-13 23:13:28,038 INFO org.apache.hadoop.hdfs.server.datanode.DataNod=
e.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.214:35138, byt=
es: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001440_0=
, offset: 197120, srvID: DS-443352839-145.100.2.183-50050-1291128673616, bl=
ockid: blk_-9163184839986480695_4112368, duration: 5573000
2011-05-13 23:13:28,159 INFO org.apache.hadoop.hdfs.server.datanode.DataNod=
e.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.212:38574, byt=
es: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001444_0=
, offset: 197632, srvID: DS-443352839-145.100.2.183-50050-1291128673616, bl=
ockid: blk_-9163184839986480695_4112368, duration: 16939000
2011-05-13 23:13:28,209 INFO org.apache.hadoop.hdfs.server.datanode.DataNod=
e: Received block blk_-9123390874940601805_2898225 src: /192.168.28.210:442=
27 dest: /192.168.28.214:50050 of size 300441
2011-05-13 23:13:28,217 INFO org.apache.hadoop.hdfs.server.datanode.DataNod=
e.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.213:42364, byt=
es: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001451_0=
, offset: 198656, srvID: DS-443352839-145.100.2.183-50050-1291128673616, bl=
ockid: blk_-9163184839986480695_4112368, duration: 5291000
2011-05-13 23:13:28,252 INFO org.apache.hadoop.hdfs.server.datanode.DataNod=
e.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.214:32930, byt=
es: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001436_0=
, offset: 0, srvID: DS-443352839-145.100.2.183-50050-1291128673616, blockid=
: blk_-1800696633107072247_4099834, duration: 5099000
2011-05-13 23:13:28,256 INFO org.apache.hadoop.hdfs.server.datanode.DataNod=
e.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.213:42363, byt=
es: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001458_0=
, offset: 199680, srvID: DS-443352839-145.100.2.183-50050-1291128673616, bl=
ockid: blk_-9163184839986480695_4112368, duration: 4945000
2011-05-13 23:13:28,257 INFO org.apache.hadoop.hdfs.server.datanode.DataNod=
e.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.214:35137, byt=
es: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001436_0=
, offset: 196608, srvID: DS-443352839-145.100.2.183-50050-1291128673616, bl=
ockid: blk_-9163184839986480695_4112368, duration: 4159000
2011-05-13 23:13:28,258 INFO org.apache.hadoop.hdfs.server.datanode.DataNod=
e: Exception in receiveBlock for block blk_-9140444589483291821_3585975 jav=
a.io.EOFException: while trying to read 100 bytes
2011-05-13 23:13:28,258 WARN org.apache.hadoop.hdfs.server.datanode.DataNod=
e: Block blk_-9140444589483291821_3585975 unfinalized and removed.=20
2011-05-13 23:13:28,258 INFO org.apache.hadoop.hdfs.server.datanode.DataNod=
e: writeBlock blk_-9140444589483291821_3585975 received exception java.io.E=
OFException: while trying to read 100 bytes
2011-05-13 23:13:28,259 ERROR org.apache.hadoop.hdfs.server.datanode.DataNo=
de: DatanodeRegistration(192.168.28.214:50050, storageID=3DDS-443352839-145=
.100.2.183-50050-1291128673616, infoPort=3D50075, ipcPort=3D50020):DataXcei=
ver
java.io.EOFException: while trying to read 100 bytes
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(B=
lockReceiver.java:270)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPac=
ket(BlockReceiver.java:357)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePack=
et(BlockReceiver.java:378)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBloc=
k(BlockReceiver.java:534)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(Da=
taXceiver.java:417)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiv=
er.java:122)
2011-05-13 23:13:28,264 INFO org.apache.hadoop.hdfs.server.datanode.DataNod=
e.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.212:38553, byt=
es: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001441_0=
, offset: 0, srvID: DS-443352839-145.100.2.183-50050-1291128673616, blockid=
: blk_-5819719631677148140_4098274, duration: 5625000
2011-05-13 23:13:28,264 INFO org.apache.hadoop.hdfs.server.datanode.DataNod=
e.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.212:38535, byt=
es: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001438_0=
, offset: 196608, srvID: DS-443352839-145.100.2.183-50050-1291128673616, bl=
ockid: blk_-9163184839986480695_4112368, duration: 4473000
2011-05-13 23:13:28,265 INFO org.apache.hadoop.hdfs.server.datanode.DataNod=
e: DatanodeRegistration(192.168.28.214:50050, storageID=3DDS-443352839-145.=
100.2.183-50050-1291128673616, infoPort=3D50075, ipcPort=3D50020): Exceptio=
n writing block blk_-9150014886921014525_2267869 to mirror 192.168.28.213:5=
0050
java.io.IOException: The stream is closed
        at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStrea=
m.java:108)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.ja=
va:65)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123=
)
        at java.io.DataOutputStream.flush(DataOutputStream.java:106)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBloc=
k(BlockReceiver.java:540)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(Da=
taXceiver.java:417)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiv=
er.java:122)

2011-05-13 23:13:28,265 INFO org.apache.hadoop.hdfs.server.datanode.DataNod=
e.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.213:45484, byt=
es: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001432_0=
, offset: 0, srvID: DS-443352839-145.100.2.183-50050-1291128673616, blockid=
: blk_405051931214094755_4098504, duration: 5597000
2011-05-13 23:13:28,273 INFO org.apache.hadoop.hdfs.server.datanode.DataNod=
e: Received block blk_-9150014886921014525_2267869 src: /192.168.28.211:492=
08 dest: /192.168.28.214:50050 of size 3033173
2011-05-13 23:13:28,313 INFO org.apache.hadoop.hdfs.server.datanode.DataNod=
e: Received block blk_-9144765354308563975_3310572 src: /192.168.28.211:515=
92 dest: /192.168.28.214:50050 of size 242383

________________________________________
From: Segel, Mike [msegel@navteq.com]
Sent: Friday, May 13, 2011 2:36 PM
To: general@hadoop.apache.org
Cc: <cdh-user@cloudera.org>; <general@hadoop.apache.org>
Subject: Re: Stability issue - dead DN's

Bonded will work but you may not see the performance you would expect.  If =
you need >1 GBe, go 10GBe less headache and has even more headroom.

Multiple interfaces won't work. Or I should say didn't work in past release=
s.
If you think about it, clients have to connect to each node. So having two =
interfaces and trying to manage them makes no sense.

Add to this trying to manage this in DNS ... Why make more work for yoursel=
f?
Going from memory... It looked like you rDNS had to match you hostnames so =
your internal interfaces had to match hostnames so you had an inverted netw=
ork.

If you draw out your network topology you end up with a ladder.
You would be better off (IMHO) to create a subnet where only your edge serv=
ers are dual nic'd.
But then if your cluster is for development... Now your PCs can't be used a=
s clients...

Does this make sense?


Sent from a remote device. Please excuse any typos...

Mike Segel

On May 13, 2011, at 4:57 AM, "Evert Lammerts" <Evert.Lammerts@sara.nl> wrot=
e:

> Hi Mike,
>
>> You really really don't want to do this.
>> Long story short... It won't work.
>
> Can you elaborate? Are you talking about the bonded interfaces or about h=
aving a separated network for interconnects and external network? What can =
go wrong there?
>
>>
>> Just a suggestion.. You don't want anyone on your cluster itself. They
>> should interact wit edge nodes, which are 'Hadoop aware'. Then your
>> cluster has a single network to worry about.
>
> That's our current setup. We have a single headnode that is used as a SPO=
E. However, I'd like to change that on our future production system. We wan=
t to implement Kerberos for authentication, and let users interact with the=
 cluster from their own machine. This would enable them to submit their job=
s from the local IDE. The only way to do this is by opening up Hadoop ports=
 for the world, is my understanding: if people interact with HDFS they need=
 to be able to interact with all nodes, right? What would be the argument a=
gainst this?
>
> Cheers,
> Evert
>
>>
>>
>> Sent from a remote device. Please excuse any typos...
>>
>> Mike Segel
>>
>> On May 11, 2011, at 11:45 AM, Allen Wittenauer <aw@apache.org> wrote:
>>
>>>
>>>
>>>
>>>
>>>>> * a 2x1GE bonded network interface for interconnects
>>>>> * a 2x1GE bonded network interface for external access
>>>
>>>   Multiple NICs on a box can sometimes cause big performance
>> problems with Hadoop.  So watch your traffic carefully.
>>>
>>>
>>>


The information contained in this communication may be CONFIDENTIAL and is =
intended only for the use of the recipient(s) named above.  If you are not =
the intended recipient, you are hereby notified that any dissemination, dis=
tribution, or copying of this communication, or any of its contents, is str=
ictly prohibited.  If you have received this communication in error, please=
 notify the sender and delete/destroy the original message and any copy of =
it from your computer or paper files.

From general-return-3526-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 21:56:00 2011
Return-Path: <general-return-3526-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 15E1447F4
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 21:56:00 +0000 (UTC)
Received: (qmail 52568 invoked by uid 500); 13 May 2011 21:55:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52509 invoked by uid 500); 13 May 2011 21:55:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52501 invoked by uid 99); 13 May 2011 21:55:58 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 21:55:58 +0000
Received: from localhost (HELO [90.169.178.234]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 21:55:58 +0000
Message-ID: <4DCDA8E7.5090809@apache.org>
Date: Fri, 13 May 2011 23:55:51 +0200
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Defining Hadoop Compatibility -revisiting-
References: <C9F227D5.8FA4%mbhandarkar@linkedin.com> <4DCCF172.5010504@apache.org> <99AB77CF-025E-4E4A-BD7F-BDA362E633BF@apache.org>
In-Reply-To: <99AB77CF-025E-4E4A-BD7F-BDA362E633BF@apache.org>
X-Enigmail-Version: 1.1.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On 05/13/2011 07:28 PM, Allen Wittenauer wrote:
> If it has a modified version of Hadoop (i.e., not an actual Apache
> release or patches which have never been committed to trunk), are
> they allowed to say "includes Apache Hadoop"?

No.  Those are the two cases we permit.  We used to say that it was
enough for a patch to be in Jira, but Roy clarified last year that
committed to trunk is a better line, since that means the code has been
reviewed and accepted by the community.

Doug

From general-return-3527-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 22:13:55 2011
Return-Path: <general-return-3527-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1691D4FBE
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 22:13:55 +0000 (UTC)
Received: (qmail 65586 invoked by uid 500); 13 May 2011 22:13:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65527 invoked by uid 500); 13 May 2011 22:13:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65519 invoked by uid 99); 13 May 2011 22:13:53 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:13:53 +0000
Received: from localhost (HELO awittena-mn.linkedin.biz) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:13:53 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Defining Hadoop Compatibility -revisiting-
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <4DCDA8E7.5090809@apache.org>
Date: Fri, 13 May 2011 15:13:44 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <2ED3A7F5-E549-4545-9395-094A10EA4EED@apache.org>
References: <C9F227D5.8FA4%mbhandarkar@linkedin.com> <4DCCF172.5010504@apache.org> <99AB77CF-025E-4E4A-BD7F-BDA362E633BF@apache.org> <4DCDA8E7.5090809@apache.org>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On May 13, 2011, at 2:55 PM, Doug Cutting wrote:

> On 05/13/2011 07:28 PM, Allen Wittenauer wrote:
>> If it has a modified version of Hadoop (i.e., not an actual Apache
>> release or patches which have never been committed to trunk), are
>> they allowed to say "includes Apache Hadoop"?
>=20
> No.  Those are the two cases we permit.  We used to say that it was
> enough for a patch to be in Jira, but Roy clarified last year that
> committed to trunk is a better line, since that means the code has =
been
> reviewed and accepted by the community.


	So what do we do about companies that release a product that =
says "includes Apache Hadoop" but includes patches that aren't committed =
to trunk?



From general-return-3528-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 22:16:09 2011
Return-Path: <general-return-3528-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 41C8F4363
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 22:16:09 +0000 (UTC)
Received: (qmail 68469 invoked by uid 500); 13 May 2011 22:16:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68417 invoked by uid 500); 13 May 2011 22:16:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68409 invoked by uid 99); 13 May 2011 22:16:07 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:16:07 +0000
Received: from localhost (HELO [90.169.178.234]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:16:07 +0000
Message-ID: <4DCDADA0.5090704@apache.org>
Date: Sat, 14 May 2011 00:16:00 +0200
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Defining Hadoop Compatibility -revisiting-
References: <C9F227D5.8FA4%mbhandarkar@linkedin.com> <4DCCF172.5010504@apache.org> <99AB77CF-025E-4E4A-BD7F-BDA362E633BF@apache.org> <4DCDA8E7.5090809@apache.org> <2ED3A7F5-E549-4545-9395-094A10EA4EED@apache.org>
In-Reply-To: <2ED3A7F5-E549-4545-9395-094A10EA4EED@apache.org>
X-Enigmail-Version: 1.1.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On 05/14/2011 12:13 AM, Allen Wittenauer wrote:
> So what do we do about companies that release a product that says "includes Apache Hadoop" but includes patches that aren't committed to trunk?

We yell at them to get those patches into trunk already.  This policy
was clarified after that product was shipping.

Doug

From general-return-3529-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 22:17:53 2011
Return-Path: <general-return-3529-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 80F304B42
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 22:17:53 +0000 (UTC)
Received: (qmail 71209 invoked by uid 500); 13 May 2011 22:17:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71151 invoked by uid 500); 13 May 2011 22:17:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71143 invoked by uid 99); 13 May 2011 22:17:52 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:17:52 +0000
Received: from localhost (HELO awittena-mn.linkedin.biz) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:17:51 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Defining Hadoop Compatibility -revisiting-
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <4DCDADA0.5090704@apache.org>
Date: Fri, 13 May 2011 15:17:46 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <8A7603B9-A6C0-4FC7-8B45-CEF3035244F0@apache.org>
References: <C9F227D5.8FA4%mbhandarkar@linkedin.com> <4DCCF172.5010504@apache.org> <99AB77CF-025E-4E4A-BD7F-BDA362E633BF@apache.org> <4DCDA8E7.5090809@apache.org> <2ED3A7F5-E549-4545-9395-094A10EA4EED@apache.org> <4DCDADA0.5090704@apache.org>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On May 13, 2011, at 3:16 PM, Doug Cutting wrote:

> On 05/14/2011 12:13 AM, Allen Wittenauer wrote:
>> So what do we do about companies that release a product that says =
"includes Apache Hadoop" but includes patches that aren't committed to =
trunk?
>=20
> We yell at them to get those patches into trunk already.  This policy
> was clarified after that product was shipping.


	... and if those patches are rejected by the community?=

From general-return-3530-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 22:18:47 2011
Return-Path: <general-return-3530-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5F3924FD6
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 22:18:47 +0000 (UTC)
Received: (qmail 73489 invoked by uid 500); 13 May 2011 22:18:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73428 invoked by uid 500); 13 May 2011 22:18:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73420 invoked by uid 99); 13 May 2011 22:18:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:18:45 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of eli@cloudera.com designates 209.85.215.176 as permitted sender)
Received: from [209.85.215.176] (HELO mail-ey0-f176.google.com) (209.85.215.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:18:39 +0000
Received: by eya28 with SMTP id 28so1281839eya.35
        for <general@hadoop.apache.org>; Fri, 13 May 2011 15:18:18 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.14.122.81 with SMTP id s57mr881080eeh.195.1305325097908; Fri,
 13 May 2011 15:18:17 -0700 (PDT)
Received: by 10.14.37.13 with HTTP; Fri, 13 May 2011 15:18:17 -0700 (PDT)
In-Reply-To: <2ED3A7F5-E549-4545-9395-094A10EA4EED@apache.org>
References: <C9F227D5.8FA4%mbhandarkar@linkedin.com>
	<4DCCF172.5010504@apache.org>
	<99AB77CF-025E-4E4A-BD7F-BDA362E633BF@apache.org>
	<4DCDA8E7.5090809@apache.org>
	<2ED3A7F5-E549-4545-9395-094A10EA4EED@apache.org>
Date: Fri, 13 May 2011 15:18:17 -0700
Message-ID: <BANLkTimC1GekQQA2ZEB5ZHOXYoeMesvvJw@mail.gmail.com>
Subject: Re: Defining Hadoop Compatibility -revisiting-
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Fri, May 13, 2011 at 3:13 PM, Allen Wittenauer <aw@apache.org> wrote:
>
> On May 13, 2011, at 2:55 PM, Doug Cutting wrote:
>
>> On 05/13/2011 07:28 PM, Allen Wittenauer wrote:
>>> If it has a modified version of Hadoop (i.e., not an actual Apache
>>> release or patches which have never been committed to trunk), are
>>> they allowed to say "includes Apache Hadoop"?
>>
>> No. =A0Those are the two cases we permit. =A0We used to say that it was
>> enough for a patch to be in Jira, but Roy clarified last year that
>> committed to trunk is a better line, since that means the code has been
>> reviewed and accepted by the community.
>
>
> =A0 =A0 =A0 =A0So what do we do about companies that release a product th=
at says "includes Apache Hadoop" but includes patches that aren't committed=
 to trunk?
>

It's not just companies by the way, we the Hadoop project just made an
official Apache release that contains patches not yet in trunk...

From general-return-3531-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 22:22:59 2011
Return-Path: <general-return-3531-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CB0DE46FE
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 22:22:59 +0000 (UTC)
Received: (qmail 77923 invoked by uid 500); 13 May 2011 22:22:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77877 invoked by uid 500); 13 May 2011 22:22:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77869 invoked by uid 99); 13 May 2011 22:22:58 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:22:58 +0000
Received: from localhost (HELO [90.169.178.234]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:22:58 +0000
Message-ID: <4DCDAF3D.7060409@apache.org>
Date: Sat, 14 May 2011 00:22:53 +0200
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Defining Hadoop Compatibility -revisiting-
References: <C9F227D5.8FA4%mbhandarkar@linkedin.com> <4DCCF172.5010504@apache.org> <99AB77CF-025E-4E4A-BD7F-BDA362E633BF@apache.org> <4DCDA8E7.5090809@apache.org> <2ED3A7F5-E549-4545-9395-094A10EA4EED@apache.org> <4DCDADA0.5090704@apache.org> <8A7603B9-A6C0-4FC7-8B45-CEF3035244F0@apache.org>
In-Reply-To: <8A7603B9-A6C0-4FC7-8B45-CEF3035244F0@apache.org>
X-Enigmail-Version: 1.1.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On 05/14/2011 12:17 AM, Allen Wittenauer wrote:
> 	... and if those patches are rejected by the community?

It would be very strange, since they've mostly been released in 203,
although not yet having been committed to trunk.

Doug

From general-return-3532-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 22:28:43 2011
Return-Path: <general-return-3532-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 455284966
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 22:28:43 +0000 (UTC)
Received: (qmail 86475 invoked by uid 500); 13 May 2011 22:28:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86422 invoked by uid 500); 13 May 2011 22:28:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86414 invoked by uid 99); 13 May 2011 22:28:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:28:41 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [208.97.132.177] (HELO homiemail-a71.g.dreamhost.com) (208.97.132.177)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:28:34 +0000
Received: from homiemail-a71.g.dreamhost.com (localhost [127.0.0.1])
	by homiemail-a71.g.dreamhost.com (Postfix) with ESMTP id C55D2428076
	for <general@hadoop.apache.org>; Fri, 13 May 2011 15:28:12 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; q=dns; s=
	gbiv.com; b=EcmvcGNMCozTez3IYXsJN+cDj6iIN4tutUb2/mpTT/yemy3bDW+y
	qm2o48jQgGOQ5I9TSZMEnyJRCRFUIfDMdhpZncSgTM70myKo1cX4amhW49OGKtay
	+a39X75vuP5av+UZM1/3GsJGSmIFSzYcP8jTHHaH5IYs1K+wYuJv7WM=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; s=gbiv.com;
	 bh=tsfxa0WJnZeX7tJJHpvpoUV48w8=; b=5QjY+M8+9mLZZEaS4G1etmpb2UG6
	OnsCQjuFeMxqDvkSAV/lIXwUYfhuaVC8uQKSfUBUslXqy4hI7l4aqo+fHAzpM6EV
	2DoWPFyBHrFvkc3/mD3IWHWbVoT2tvwNapMr0FtcblSGzrgR10G4fEwrUX3Rz6nm
	YCDQxUmA1wv1wk0=
Received: from [10.134.89.86] (unknown [75.103.10.98])
	(using TLSv1 with cipher AES128-SHA (128/128 bits))
	(No client certificate requested)
	(Authenticated sender: fielding@gbiv.com)
	by homiemail-a71.g.dreamhost.com (Postfix) with ESMTPSA id AF44442806E
	for <general@hadoop.apache.org>; Fri, 13 May 2011 15:28:12 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: Defining Hadoop Compatibility -revisiting-
From: "Roy T. Fielding" <fielding@gbiv.com>
In-Reply-To: <4DCDA8E7.5090809@apache.org>
Date: Fri, 13 May 2011 15:26:45 -0700
Content-Transfer-Encoding: 7bit
Message-Id: <318DA92B-E9BB-4E8F-9FFB-4AA70E6FF1EB@gbiv.com>
References: <C9F227D5.8FA4%mbhandarkar@linkedin.com> <4DCCF172.5010504@apache.org> <99AB77CF-025E-4E4A-BD7F-BDA362E633BF@apache.org> <4DCDA8E7.5090809@apache.org>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

On May 13, 2011, at 2:55 PM, Doug Cutting wrote:

> On 05/13/2011 07:28 PM, Allen Wittenauer wrote:
>> If it has a modified version of Hadoop (i.e., not an actual Apache
>> release or patches which have never been committed to trunk), are
>> they allowed to say "includes Apache Hadoop"?
> 
> No.  Those are the two cases we permit.  We used to say that it was
> enough for a patch to be in Jira, but Roy clarified last year that
> committed to trunk is a better line, since that means the code has been
> reviewed and accepted by the community.

Committed to a releasable branch, actually.  In other words, a
branch (including trunk) that the PMC is collaborating on towards
release at some point in the future.

....Roy


From general-return-3533-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 22:33:58 2011
Return-Path: <general-return-3533-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DA0DF4CFC
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 22:33:58 +0000 (UTC)
Received: (qmail 95556 invoked by uid 500); 13 May 2011 22:33:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95491 invoked by uid 500); 13 May 2011 22:33:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95483 invoked by uid 99); 13 May 2011 22:33:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:33:57 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of msegel@navteq.com designates 65.167.11.240 as permitted sender)
Received: from [65.167.11.240] (HELO xmailchi.navteq.com) (65.167.11.240)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:33:50 +0000
Received: from chicago-spamfw.navteq.com (imailchi [10.8.5.161])
	by xmailchi.navteq.com (8.13.8/8.13.6) with ESMTP id p4DMXT90030667
	for <general@hadoop.apache.org>; Fri, 13 May 2011 17:33:29 -0500
Received: from imailchi.hq.navteq.com (localhost [127.0.0.1])
	by chicago-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id AC0FA4422F17
	for <general@hadoop.apache.org>; Fri, 13 May 2011 17:33:28 -0500 (CDT)
Received: from imailchi.hq.navteq.com ([10.8.120.104]) by chicago-spamfw.navteq.com with ESMTP id ldKdSsY4BjgkxuDE for <general@hadoop.apache.org>; Fri, 13 May 2011 17:33:28 -0500 (CDT)
Received: from hq-ex-ht01.ad.navteq.com (hq-ex-ht01.ad.navteq.com [10.8.222.51])
	by imailchi.hq.navteq.com (8.12.9/8.12.9) with ESMTP id p4DMXSNe023532
	for <general@hadoop.apache.org>; Fri, 13 May 2011 17:33:28 -0500
Received: from hq-ex-mb03.ad.navteq.com ([fe80::c4dd:7b21:5c22:cfe4]) by
 hq-ex-ht01.ad.navteq.com ([fe80::55a6:5b94:5c00:fe39%12]) with mapi; Fri, 13
 May 2011 17:33:28 -0500
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 13 May 2011 17:33:40 -0500
Subject: Re: Stability issue - dead DN's
Thread-Topic: Stability issue - dead DN's
Thread-Index: AcwRvcdfcMDGgpIHSLChYJEq3fXs1w==
Message-ID: <DABA1F29-2DC3-480E-ABD6-F690F95CD018@navteq.com>
References: <ADF94D8555C7A246B86A633685E0178AB90A474E7A@planck.ka.sara.nl>
 <BANLkTi=NaiXQPAkdS-LPW-Swev6XYD0Xjw@mail.gmail.com>
 <DA1F03C5-7F7B-412A-84E7-774294085603@apache.org>
 <BLU0-SMTP107D4E21A69A6C860D8CBB2F0860@phx.gbl>
 <ADF94D8555C7A246B86A633685E0178AB90A474E94@planck.ka.sara.nl>
 <FAAD6507-80D9-453A-BCED-916532688B49@navteq.com>
 <ADF94D8555C7A246B86A633685E0178AB90A35CA07@planck.ka.sara.nl>
In-Reply-To: <ADF94D8555C7A246B86A633685E0178AB90A35CA07@planck.ka.sara.nl>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1
X-Virus-Checked: Checked by ClamAV on apache.org

Ok...

Hum, look, I've been force fed a couple of margaritas so, my memory is a =
bit foggy...
You say your clients connect on nic A. Your cluster connects on nic B.

What happens when you want to upload a file from your client to HDFS? Or =
even access it?

=2E.. ;-)



Sent from a remote device. Please excuse any typos...

Mike Segel

On May 13, 2011, at 4:15 PM, "Evert Lammerts" <Evert.Lammerts@sara.nl> wr=
ote:

> Hi Mike,
>=20
> Thanks for trying to help out.
>=20
> I had a talk with our networking guys this afternoon. According to them=
 (and this is way out of my area of expertise, so excuse any mistakes) mu=
ltiple interfaces shouldn't be a problem. We could set up a nameserver to=
 resolve hostnames to addresses in our private space when the request com=
es from one of the nodes, and route this traffic over a single interface.=
 Any other request can be resolved to an address in the public space, whi=
ch is bound to an other interface. In our current setup we're not even re=
solving hostnames in our private address space through a nameserver - we =
do it with an ugly hack in /etc/hosts. And it seems to work alright.
>=20
> Having said that, our problems are still not completely gone even after=
 adjusting the maximum allowed RAM for tasks - although things are lots b=
etter. While writing this mail three out of five DN's were marked as dead=
=2E There still is some swapping going on, but the cores are not spending=
 any time in WAIT, so this shouldn't be the cause of anything. See below =
a trace from a dead DN - any thoughts are appreciated!
>=20
> Cheers,
> Evert
>=20
> 2011-05-13 23:13:27,716 INFO org.apache.hadoop.hdfs.server.datanode.Dat=
aNode: Received block blk_-9131821326787012529_2915672 src: /192.168.28.2=
11:60136 dest: /192.168.28.214:50050 of size 382425
> 2011-05-13 23:13:27,915 INFO org.apache.hadoop.hdfs.server.datanode.Dat=
aNode: Exception in receiveBlock for block blk_-9132067116195286882_13088=
8 java.io.EOFException: while trying to read 3744913 bytes
> 2011-05-13 23:13:27,925 INFO org.apache.hadoop.hdfs.server.datanode.Dat=
aNode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.214:3513=
9, bytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_=
001437_0, offset: 196608, srvID: DS-443352839-145.100.2.183-50050-1291128=
673616, blockid: blk_-9163184839986480695_4112368, duration: 6254000
> 2011-05-13 23:13:28,032 INFO org.apache.hadoop.hdfs.server.datanode.Dat=
aNode: Received block blk_-9149862728087355005_3793421 src: /192.168.28.2=
10:41197 dest: /192.168.28.214:50050 of size 245767
> 2011-05-13 23:13:28,033 WARN org.apache.hadoop.hdfs.server.datanode.Dat=
aNode: Block blk_-9132067116195286882_130888 unfinalized and removed.
> 2011-05-13 23:13:28,033 INFO org.apache.hadoop.hdfs.server.datanode.Dat=
aNode: writeBlock blk_-9132067116195286882_130888 received exception java=
=2Eio.EOFException: while trying to read 3744913 bytes
> 2011-05-13 23:13:28,033 ERROR org.apache.hadoop.hdfs.server.datanode.Da=
taNode: DatanodeRegistration(192.168.28.214:50050, storageID=3DDS-4433528=
39-145.100.2.183-50050-1291128673616, infoPort=3D50075, ipcPort=3D50020):=
DataXceiver
> java.io.EOFException: while trying to read 3744913 bytes
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBu=
f(BlockReceiver.java:270)
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNext=
Packet(BlockReceiver.java:357)
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveP=
acket(BlockReceiver.java:378)
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveB=
lock(BlockReceiver.java:534)
>        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock=
(DataXceiver.java:417)
>        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXc=
eiver.java:122)
> 2011-05-13 23:13:28,038 INFO org.apache.hadoop.hdfs.server.datanode.Dat=
aNode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.214:3291=
0, bytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_=
001443_0, offset: 197632, srvID: DS-443352839-145.100.2.183-50050-1291128=
673616, blockid: blk_-9163184839986480695_4112368, duration: 4323000
> 2011-05-13 23:13:28,038 INFO org.apache.hadoop.hdfs.server.datanode.Dat=
aNode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.214:3513=
8, bytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_=
001440_0, offset: 197120, srvID: DS-443352839-145.100.2.183-50050-1291128=
673616, blockid: blk_-9163184839986480695_4112368, duration: 5573000
> 2011-05-13 23:13:28,159 INFO org.apache.hadoop.hdfs.server.datanode.Dat=
aNode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.212:3857=
4, bytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_=
001444_0, offset: 197632, srvID: DS-443352839-145.100.2.183-50050-1291128=
673616, blockid: blk_-9163184839986480695_4112368, duration: 16939000
> 2011-05-13 23:13:28,209 INFO org.apache.hadoop.hdfs.server.datanode.Dat=
aNode: Received block blk_-9123390874940601805_2898225 src: /192.168.28.2=
10:44227 dest: /192.168.28.214:50050 of size 300441
> 2011-05-13 23:13:28,217 INFO org.apache.hadoop.hdfs.server.datanode.Dat=
aNode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.213:4236=
4, bytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_=
001451_0, offset: 198656, srvID: DS-443352839-145.100.2.183-50050-1291128=
673616, blockid: blk_-9163184839986480695_4112368, duration: 5291000
> 2011-05-13 23:13:28,252 INFO org.apache.hadoop.hdfs.server.datanode.Dat=
aNode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.214:3293=
0, bytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_=
001436_0, offset: 0, srvID: DS-443352839-145.100.2.183-50050-129112867361=
6, blockid: blk_-1800696633107072247_4099834, duration: 5099000
> 2011-05-13 23:13:28,256 INFO org.apache.hadoop.hdfs.server.datanode.Dat=
aNode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.213:4236=
3, bytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_=
001458_0, offset: 199680, srvID: DS-443352839-145.100.2.183-50050-1291128=
673616, blockid: blk_-9163184839986480695_4112368, duration: 4945000
> 2011-05-13 23:13:28,257 INFO org.apache.hadoop.hdfs.server.datanode.Dat=
aNode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.214:3513=
7, bytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_=
001436_0, offset: 196608, srvID: DS-443352839-145.100.2.183-50050-1291128=
673616, blockid: blk_-9163184839986480695_4112368, duration: 4159000
> 2011-05-13 23:13:28,258 INFO org.apache.hadoop.hdfs.server.datanode.Dat=
aNode: Exception in receiveBlock for block blk_-9140444589483291821_35859=
75 java.io.EOFException: while trying to read 100 bytes
> 2011-05-13 23:13:28,258 WARN org.apache.hadoop.hdfs.server.datanode.Dat=
aNode: Block blk_-9140444589483291821_3585975 unfinalized and removed.
> 2011-05-13 23:13:28,258 INFO org.apache.hadoop.hdfs.server.datanode.Dat=
aNode: writeBlock blk_-9140444589483291821_3585975 received exception jav=
a.io.EOFException: while trying to read 100 bytes
> 2011-05-13 23:13:28,259 ERROR org.apache.hadoop.hdfs.server.datanode.Da=
taNode: DatanodeRegistration(192.168.28.214:50050, storageID=3DDS-4433528=
39-145.100.2.183-50050-1291128673616, infoPort=3D50075, ipcPort=3D50020):=
DataXceiver
> java.io.EOFException: while trying to read 100 bytes
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBu=
f(BlockReceiver.java:270)
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNext=
Packet(BlockReceiver.java:357)
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveP=
acket(BlockReceiver.java:378)
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveB=
lock(BlockReceiver.java:534)
>        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock=
(DataXceiver.java:417)
>        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXc=
eiver.java:122)
> 2011-05-13 23:13:28,264 INFO org.apache.hadoop.hdfs.server.datanode.Dat=
aNode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.212:3855=
3, bytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_=
001441_0, offset: 0, srvID: DS-443352839-145.100.2.183-50050-129112867361=
6, blockid: blk_-5819719631677148140_4098274, duration: 5625000
> 2011-05-13 23:13:28,264 INFO org.apache.hadoop.hdfs.server.datanode.Dat=
aNode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.212:3853=
5, bytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_=
001438_0, offset: 196608, srvID: DS-443352839-145.100.2.183-50050-1291128=
673616, blockid: blk_-9163184839986480695_4112368, duration: 4473000
> 2011-05-13 23:13:28,265 INFO org.apache.hadoop.hdfs.server.datanode.Dat=
aNode: DatanodeRegistration(192.168.28.214:50050, storageID=3DDS-44335283=
9-145.100.2.183-50050-1291128673616, infoPort=3D50075, ipcPort=3D50020): =
Exception writing block blk_-9150014886921014525_2267869 to mirror 192.16=
8.28.213:50050
> java.io.IOException: The stream is closed
>        at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputSt=
ream.java:108)
>        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream=
=2Ejava:65)
>        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:=
123)
>        at java.io.DataOutputStream.flush(DataOutputStream.java:106)
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveB=
lock(BlockReceiver.java:540)
>        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock=
(DataXceiver.java:417)
>        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXc=
eiver.java:122)
>=20
> 2011-05-13 23:13:28,265 INFO org.apache.hadoop.hdfs.server.datanode.Dat=
aNode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.213:4548=
4, bytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_=
001432_0, offset: 0, srvID: DS-443352839-145.100.2.183-50050-129112867361=
6, blockid: blk_405051931214094755_4098504, duration: 5597000
> 2011-05-13 23:13:28,273 INFO org.apache.hadoop.hdfs.server.datanode.Dat=
aNode: Received block blk_-9150014886921014525_2267869 src: /192.168.28.2=
11:49208 dest: /192.168.28.214:50050 of size 3033173
> 2011-05-13 23:13:28,313 INFO org.apache.hadoop.hdfs.server.datanode.Dat=
aNode: Received block blk_-9144765354308563975_3310572 src: /192.168.28.2=
11:51592 dest: /192.168.28.214:50050 of size 242383
>=20
> ________________________________________
> From: Segel, Mike [msegel@navteq.com]
> Sent: Friday, May 13, 2011 2:36 PM
> To: general@hadoop.apache.org
> Cc: <cdh-user@cloudera.org>; <general@hadoop.apache.org>
> Subject: Re: Stability issue - dead DN's
>=20
> Bonded will work but you may not see the performance you would expect. =
 If you need >1 GBe, go 10GBe less headache and has even more headroom.
>=20
> Multiple interfaces won't work. Or I should say didn't work in past rel=
eases.
> If you think about it, clients have to connect to each node. So having =
two interfaces and trying to manage them makes no sense.
>=20
> Add to this trying to manage this in DNS ... Why make more work for you=
rself?
> Going from memory... It looked like you rDNS had to match you hostnames=
 so your internal interfaces had to match hostnames so you had an inverte=
d network.
>=20
> If you draw out your network topology you end up with a ladder.
> You would be better off (IMHO) to create a subnet where only your edge =
servers are dual nic'd.
> But then if your cluster is for development... Now your PCs can't be us=
ed as clients...
>=20
> Does this make sense?
>=20
>=20
> Sent from a remote device. Please excuse any typos...
>=20
> Mike Segel
>=20
> On May 13, 2011, at 4:57 AM, "Evert Lammerts" <Evert.Lammerts@sara.nl> =
wrote:
>=20
>> Hi Mike,
>>=20
>>> You really really don't want to do this.
>>> Long story short... It won't work.
>>=20
>> Can you elaborate? Are you talking about the bonded interfaces or abou=
t having a separated network for interconnects and external network? What=
 can go wrong there?
>>=20
>>>=20
>>> Just a suggestion.. You don't want anyone on your cluster itself. The=
y
>>> should interact wit edge nodes, which are 'Hadoop aware'. Then your
>>> cluster has a single network to worry about.
>>=20
>> That's our current setup. We have a single headnode that is used as a =
SPOE. However, I'd like to change that on our future production system. W=
e want to implement Kerberos for authentication, and let users interact w=
ith the cluster from their own machine. This would enable them to submit =
their jobs from the local IDE. The only way to do this is by opening up H=
adoop ports for the world, is my understanding: if people interact with H=
DFS they need to be able to interact with all nodes, right? What would be=
 the argument against this?
>>=20
>> Cheers,
>> Evert
>>=20
>>>=20
>>>=20
>>> Sent from a remote device. Please excuse any typos...
>>>=20
>>> Mike Segel
>>>=20
>>> On May 11, 2011, at 11:45 AM, Allen Wittenauer <aw@apache.org> wrote:
>>>=20
>>>>=20
>>>>=20
>>>>=20
>>>>=20
>>>>>> * a 2x1GE bonded network interface for interconnects
>>>>>> * a 2x1GE bonded network interface for external access
>>>>=20
>>>>  Multiple NICs on a box can sometimes cause big performance
>>> problems with Hadoop.  So watch your traffic carefully.
>>>>=20
>>>>=20
>>>>=20
>=20
>=20
> The information contained in this communication may be CONFIDENTIAL and=
 is intended only for the use of the recipient(s) named above.  If you ar=
e not the intended recipient, you are hereby notified that any disseminat=
ion, distribution, or copying of this communication, or any of its conten=
ts, is strictly prohibited.  If you have received this communication in e=
rror, please notify the sender and delete/destroy the original message an=
d any copy of it from your computer or paper files.


The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.

From general-return-3534-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 22:54:38 2011
Return-Path: <general-return-3534-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4F38248DB
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 22:54:38 +0000 (UTC)
Received: (qmail 23968 invoked by uid 500); 13 May 2011 22:54:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23918 invoked by uid 500); 13 May 2011 22:54:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23910 invoked by uid 99); 13 May 2011 22:54:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:54:36 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.220.176] (HELO mail-vx0-f176.google.com) (209.85.220.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:54:29 +0000
Received: by vxa37 with SMTP id 37so3486833vxa.35
        for <general@hadoop.apache.org>; Fri, 13 May 2011 15:54:08 -0700 (PDT)
Received: by 10.52.110.135 with SMTP id ia7mr2832589vdb.294.1305327248065;
 Fri, 13 May 2011 15:54:08 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.52.112.7 with HTTP; Fri, 13 May 2011 15:53:48 -0700 (PDT)
X-Originating-IP: [64.105.168.204]
In-Reply-To: <2ED3A7F5-E549-4545-9395-094A10EA4EED@apache.org>
References: <C9F227D5.8FA4%mbhandarkar@linkedin.com> <4DCCF172.5010504@apache.org>
 <99AB77CF-025E-4E4A-BD7F-BDA362E633BF@apache.org> <4DCDA8E7.5090809@apache.org>
 <2ED3A7F5-E549-4545-9395-094A10EA4EED@apache.org>
From: Ted Dunning <tdunning@maprtech.com>
Date: Fri, 13 May 2011 15:53:48 -0700
Message-ID: <BANLkTik=EkwPxvcw2-vw=hsofsJYRgyaGg@mail.gmail.com>
Subject: Re: Defining Hadoop Compatibility -revisiting-
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec5486072efc2f104a3302ec0

--bcaec5486072efc2f104a3302ec0
Content-Type: text/plain; charset=ISO-8859-1

But "distribution Z includes X" kind of implies the existence of some such
that X != Y, Y != empty-set and X+Y = Z, at least in common usage.

Isn't that the same as a non-trunk change?

So doesn't this mean that your question reduces to the question of what
happens when non-Apache changes are made to an Apache release?  And isn't
that the definition of a derived work?

 On Fri, May 13, 2011 at 3:13 PM, Allen Wittenauer <aw@apache.org> wrote:

>
> On May 13, 2011, at 2:55 PM, Doug Cutting wrote:
>
> > On 05/13/2011 07:28 PM, Allen Wittenauer wrote:
> >> If it has a modified version of Hadoop (i.e., not an actual Apache
> >> release or patches which have never been committed to trunk), are
> >> they allowed to say "includes Apache Hadoop"?
> >
> > No.  Those are the two cases we permit.  We used to say that it was
> > enough for a patch to be in Jira, but Roy clarified last year that
> > committed to trunk is a better line, since that means the code has been
> > reviewed and accepted by the community.
>
>
>        So what do we do about companies that release a product that says
> "includes Apache Hadoop" but includes patches that aren't committed to
> trunk?
>
>
>

--bcaec5486072efc2f104a3302ec0--

From general-return-3535-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 22:57:56 2011
Return-Path: <general-return-3535-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 136894991
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 22:57:56 +0000 (UTC)
Received: (qmail 29032 invoked by uid 500); 13 May 2011 22:57:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28981 invoked by uid 500); 13 May 2011 22:57:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28973 invoked by uid 99); 13 May 2011 22:57:54 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:57:54 +0000
Received: from localhost (HELO awittena-mn.linkedin.biz) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:57:54 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Defining Hadoop Compatibility -revisiting-
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <BANLkTik=EkwPxvcw2-vw=hsofsJYRgyaGg@mail.gmail.com>
Date: Fri, 13 May 2011 15:57:48 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <BEC4433A-2270-47CF-960D-CB416B20FD4A@apache.org>
References: <C9F227D5.8FA4%mbhandarkar@linkedin.com> <4DCCF172.5010504@apache.org> <99AB77CF-025E-4E4A-BD7F-BDA362E633BF@apache.org> <4DCDA8E7.5090809@apache.org> <2ED3A7F5-E549-4545-9395-094A10EA4EED@apache.org> <BANLkTik=EkwPxvcw2-vw=hsofsJYRgyaGg@mail.gmail.com>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On May 13, 2011, at 3:53 PM, Ted Dunning wrote:

> But "distribution Z includes X" kind of implies the existence of some =
such
> that X !=3D Y, Y !=3D empty-set and X+Y =3D Z, at least in common =
usage.
>=20
> Isn't that the same as a non-trunk change?
>=20
> So doesn't this mean that your question reduces to the question of =
what
> happens when non-Apache changes are made to an Apache release?  And =
isn't
> that the definition of a derived work?


	Yup. Which is why I doubt *any* commercial entity can claim =
"includes Apache Hadoop" (including Cloudera).



From general-return-3536-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 23:17:45 2011
Return-Path: <general-return-3536-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8823F4C5E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 13 May 2011 23:17:45 +0000 (UTC)
Received: (qmail 44192 invoked by uid 500); 13 May 2011 23:17:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44140 invoked by uid 500); 13 May 2011 23:17:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44132 invoked by uid 99); 13 May 2011 23:17:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 23:17:43 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 23:17:37 +0000
Received: by vws7 with SMTP id 7so3524866vws.35
        for <general@hadoop.apache.org>; Fri, 13 May 2011 16:17:16 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=qGnKWqcLH00M3I/+k4egXwuNyuaIJkXNFsLdAy8nzkk=;
        b=MSS97qs7Xirn9OUwla2u4yBa5pvgyLFIHlg4gG4Uui8DYQGQzTC391GLpW2CuccDwX
         ofs0dKBbOp4OzKdwGoy0iDRCR5vt2x3m4SAYH5UPZ0BiLZunEgRWpCiZYTq/ZrE/bkWt
         W9MrQonZAydGrxYf0Sn/8gN+dQueCuCupn7X4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=GUdqfHLCt8CzvgR3HuLFnV9ecW4S3NtBQdn5Gd3UBvlHlTZJhnvvuUlGrK9TUDlQ2W
         jKMSP0sYYj4UKPSvLdOlHCKxgyuN+a/8ANVUNrdQgxeZZns0d5CAszkpmx9bdFyRpKC2
         V8m9GQ5873IXv3/mfKO2Ey8VqmeIKBygRj4+I=
Received: by 10.52.112.227 with SMTP id it3mr2941803vdb.84.1305328636806;
        Fri, 13 May 2011 16:17:16 -0700 (PDT)
Received: from [10.172.0.47] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id q2sm912627vbv.22.2011.05.13.16.17.14
        (version=TLSv1/SSLv3 cipher=OTHER);
        Fri, 13 May 2011 16:17:15 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: What is "Hadoop?" Was: Defining Hadoop Compatibility -revisiting-
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <BANLkTi=1+++j55GER6Tvuw=pRbvApKmy-g@mail.gmail.com>
Date: Sat, 14 May 2011 09:17:11 +1000
Content-Transfer-Encoding: quoted-printable
Message-Id: <3B4F5C8A-3404-4435-9EC3-88438BCC5730@Holsman.NET>
References: <BANLkTi=1+++j55GER6Tvuw=pRbvApKmy-g@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)


On May 14, 2011, at 12:41 AM, Owen O'Malley wrote:

> On Tue, May 10, 2011 at 3:29 AM, Steve Loughran <stevel@apache.org> =
wrote:
>=20
>> I think we should revisit this issue before people with their own =
agendas
>> define what compatibility with Apache Hadoop is for us
>>=20
>=20
> I agree completely. As you point out, this week we've had a flood of
> products calling themselves "Hadoop" or "Distribution of Hadoop" that
> include only a part of Hadoop. This is will dilute Apache's Hadoop =
trademark
> and create consumer confusion.
>=20
> Licensing
>> -Use of the Hadoop codebase must follow the Apache License
>> http://www.apache.org/licenses/LICENSE-2.0
>> -plug in components that are dynamically linked to (Filesystems and
>> schedulers) don't appear to be derivative works on my reading of =
this,
>>=20
>=20
> +1
> Plugins are usually considered independent works. Note that the Apache
> license does permit commercial closed-source derivative works. A =
company
> could take Hadoop's code, modify it, and sell a binary release as long =
as
> they meet the conditions of the Apache license.
>=20
>=20
>> Naming
>> -this is something for branding@apache, they will have their =
opinions.
>> The key one is that the name "Apache Hadoop" must get used, and it's
>> important to make clear it is a derivative work.
>> -I don't think you can claim to have a Distribution/Fork/Version of =
Apache
>> Hadoop if you swap out big chunks of it for alternate filesystems, MR
>> engines, etc. Some description of this is needed
>> "Supports the Apache Hadoop MapReduce engine on top of Filesystem =
XYZ"
>>=20
>=20
> The Hadoop name is the primary tool that the project has for =
minimizing
> customer confusion. I think we need to create a very clear definition =
of
> what can be called Hadoop and what can not. Apache gives the PMCs a =
fair
> amount of latitude in picking the policy for their project name and I =
think
> we need to do so.
>=20
> Given the large number of so-called Hadoop products that are being =
released,
> I believe that we should require "Hadoop" to mean specifically the =
Apache
> Hadoop releases (possibly with a few critical security patches).
>=20
> Projects that are derivative works can either be "powered by Apache =
Hadoop,"
> or "based on Apache Hadoop."
>=20
> What do others think?
>=20
I think thats a great idea.=20
Maybe we should also create names/marks around the interfaces as well.

> -- Owen


From general-return-3537-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 08:53:54 2011
Return-Path: <general-return-3537-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5709B6544
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat, 14 May 2011 08:53:54 +0000 (UTC)
Received: (qmail 82729 invoked by uid 500); 14 May 2011 08:53:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82642 invoked by uid 500); 14 May 2011 08:53:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82634 invoked by uid 99); 14 May 2011 08:53:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 08:53:51 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [145.100.8.49] (HELO smtp.sara.nl) (145.100.8.49)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 08:53:44 +0000
Received: from planck.ka.sara.nl (145.100.8.32) by sara-exch-fe1.ka.sara.nl
 (145.100.8.49) with Microsoft SMTP Server (TLS) id 14.0.702.0; Sat, 14 May
 2011 10:53:23 +0200
Received: from planck.ka.sara.nl ([145.100.8.32]) by planck.ka.sara.nl
 ([145.100.8.32]) with mapi; Sat, 14 May 2011 10:53:02 +0200
From: Evert Lammerts <Evert.Lammerts@sara.nl>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Sat, 14 May 2011 10:53:01 +0200
Subject: RE: Stability issue - dead DN's
Thread-Topic: Stability issue - dead DN's
Thread-Index: AcwRvcdfcMDGgpIHSLChYJEq3fXs1wAVPNYp
Message-ID: <ADF94D8555C7A246B86A633685E0178AB90A35CA0A@planck.ka.sara.nl>
References: <ADF94D8555C7A246B86A633685E0178AB90A474E7A@planck.ka.sara.nl>
 <BANLkTi=NaiXQPAkdS-LPW-Swev6XYD0Xjw@mail.gmail.com>
 <DA1F03C5-7F7B-412A-84E7-774294085603@apache.org>
 <BLU0-SMTP107D4E21A69A6C860D8CBB2F0860@phx.gbl>
 <ADF94D8555C7A246B86A633685E0178AB90A474E94@planck.ka.sara.nl>
 <FAAD6507-80D9-453A-BCED-916532688B49@navteq.com>
 <ADF94D8555C7A246B86A633685E0178AB90A35CA07@planck.ka.sara.nl>,<DABA1F29-2DC3-480E-ABD6-F690F95CD018@navteq.com>
In-Reply-To: <DABA1F29-2DC3-480E-ABD6-F690F95CD018@navteq.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Ok, I'll give this scenario a try (in spite of the intoxication ;-)).

=3D putting or getting a file =3D
A client will access the NameNode first and get a list of hostnames. These =
will resolve to addresses either in public or in private space, depending o=
n whether the request to the nameserver was made by a machine in public or =
in private space. Each node has one NIC listening on its address in private=
 space and one on its address in public space. The Hadoop daemons are bound=
 to 0.0.0.0:*. Reverse DNS will return an address in private space when the=
 client connects from one of the nodes, and (obviously) an address in publi=
c space when the request came through WAN.

I'm not sure what could go wrong here... On Monday I'll recheck this scenar=
io with our HPN guys as well.

Cheers,
Evert


________________________________________
From: Segel, Mike [msegel@navteq.com]
Sent: Saturday, May 14, 2011 12:33 AM
To: general@hadoop.apache.org
Subject: Re: Stability issue - dead DN's

Ok...

Hum, look, I've been force fed a couple of margaritas so, my memory is a bi=
t foggy...
You say your clients connect on nic A. Your cluster connects on nic B.

What happens when you want to upload a file from your client to HDFS? Or ev=
en access it?

... ;-)



Sent from a remote device. Please excuse any typos...

Mike Segel

On May 13, 2011, at 4:15 PM, "Evert Lammerts" <Evert.Lammerts@sara.nl> wrot=
e:

> Hi Mike,
>
> Thanks for trying to help out.
>
> I had a talk with our networking guys this afternoon. According to them (=
and this is way out of my area of expertise, so excuse any mistakes) multip=
le interfaces shouldn't be a problem. We could set up a nameserver to resol=
ve hostnames to addresses in our private space when the request comes from =
one of the nodes, and route this traffic over a single interface. Any other=
 request can be resolved to an address in the public space, which is bound =
to an other interface. In our current setup we're not even resolving hostna=
mes in our private address space through a nameserver - we do it with an ug=
ly hack in /etc/hosts. And it seems to work alright.
>
> Having said that, our problems are still not completely gone even after a=
djusting the maximum allowed RAM for tasks - although things are lots bette=
r. While writing this mail three out of five DN's were marked as dead. Ther=
e still is some swapping going on, but the cores are not spending any time =
in WAIT, so this shouldn't be the cause of anything. See below a trace from=
 a dead DN - any thoughts are appreciated!
>
> Cheers,
> Evert
>
> 2011-05-13 23:13:27,716 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode: Received block blk_-9131821326787012529_2915672 src: /192.168.28.211:6=
0136 dest: /192.168.28.214:50050 of size 382425
> 2011-05-13 23:13:27,915 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode: Exception in receiveBlock for block blk_-9132067116195286882_130888 ja=
va.io.EOFException: while trying to read 3744913 bytes
> 2011-05-13 23:13:27,925 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.214:35139, b=
ytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001437=
_0, offset: 196608, srvID: DS-443352839-145.100.2.183-50050-1291128673616, =
blockid: blk_-9163184839986480695_4112368, duration: 6254000
> 2011-05-13 23:13:28,032 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode: Received block blk_-9149862728087355005_3793421 src: /192.168.28.210:4=
1197 dest: /192.168.28.214:50050 of size 245767
> 2011-05-13 23:13:28,033 WARN org.apache.hadoop.hdfs.server.datanode.DataN=
ode: Block blk_-9132067116195286882_130888 unfinalized and removed.
> 2011-05-13 23:13:28,033 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode: writeBlock blk_-9132067116195286882_130888 received exception java.io.=
EOFException: while trying to read 3744913 bytes
> 2011-05-13 23:13:28,033 ERROR org.apache.hadoop.hdfs.server.datanode.Data=
Node: DatanodeRegistration(192.168.28.214:50050, storageID=3DDS-443352839-1=
45.100.2.183-50050-1291128673616, infoPort=3D50075, ipcPort=3D50020):DataXc=
eiver
> java.io.EOFException: while trying to read 3744913 bytes
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(=
BlockReceiver.java:270)
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPa=
cket(BlockReceiver.java:357)
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePac=
ket(BlockReceiver.java:378)
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlo=
ck(BlockReceiver.java:534)
>        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(D=
ataXceiver.java:417)
>        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXcei=
ver.java:122)
> 2011-05-13 23:13:28,038 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.214:32910, b=
ytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001443=
_0, offset: 197632, srvID: DS-443352839-145.100.2.183-50050-1291128673616, =
blockid: blk_-9163184839986480695_4112368, duration: 4323000
> 2011-05-13 23:13:28,038 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.214:35138, b=
ytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001440=
_0, offset: 197120, srvID: DS-443352839-145.100.2.183-50050-1291128673616, =
blockid: blk_-9163184839986480695_4112368, duration: 5573000
> 2011-05-13 23:13:28,159 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.212:38574, b=
ytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001444=
_0, offset: 197632, srvID: DS-443352839-145.100.2.183-50050-1291128673616, =
blockid: blk_-9163184839986480695_4112368, duration: 16939000
> 2011-05-13 23:13:28,209 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode: Received block blk_-9123390874940601805_2898225 src: /192.168.28.210:4=
4227 dest: /192.168.28.214:50050 of size 300441
> 2011-05-13 23:13:28,217 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.213:42364, b=
ytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001451=
_0, offset: 198656, srvID: DS-443352839-145.100.2.183-50050-1291128673616, =
blockid: blk_-9163184839986480695_4112368, duration: 5291000
> 2011-05-13 23:13:28,252 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.214:32930, b=
ytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001436=
_0, offset: 0, srvID: DS-443352839-145.100.2.183-50050-1291128673616, block=
id: blk_-1800696633107072247_4099834, duration: 5099000
> 2011-05-13 23:13:28,256 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.213:42363, b=
ytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001458=
_0, offset: 199680, srvID: DS-443352839-145.100.2.183-50050-1291128673616, =
blockid: blk_-9163184839986480695_4112368, duration: 4945000
> 2011-05-13 23:13:28,257 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.214:35137, b=
ytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001436=
_0, offset: 196608, srvID: DS-443352839-145.100.2.183-50050-1291128673616, =
blockid: blk_-9163184839986480695_4112368, duration: 4159000
> 2011-05-13 23:13:28,258 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode: Exception in receiveBlock for block blk_-9140444589483291821_3585975 j=
ava.io.EOFException: while trying to read 100 bytes
> 2011-05-13 23:13:28,258 WARN org.apache.hadoop.hdfs.server.datanode.DataN=
ode: Block blk_-9140444589483291821_3585975 unfinalized and removed.
> 2011-05-13 23:13:28,258 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode: writeBlock blk_-9140444589483291821_3585975 received exception java.io=
.EOFException: while trying to read 100 bytes
> 2011-05-13 23:13:28,259 ERROR org.apache.hadoop.hdfs.server.datanode.Data=
Node: DatanodeRegistration(192.168.28.214:50050, storageID=3DDS-443352839-1=
45.100.2.183-50050-1291128673616, infoPort=3D50075, ipcPort=3D50020):DataXc=
eiver
> java.io.EOFException: while trying to read 100 bytes
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(=
BlockReceiver.java:270)
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPa=
cket(BlockReceiver.java:357)
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePac=
ket(BlockReceiver.java:378)
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlo=
ck(BlockReceiver.java:534)
>        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(D=
ataXceiver.java:417)
>        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXcei=
ver.java:122)
> 2011-05-13 23:13:28,264 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.212:38553, b=
ytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001441=
_0, offset: 0, srvID: DS-443352839-145.100.2.183-50050-1291128673616, block=
id: blk_-5819719631677148140_4098274, duration: 5625000
> 2011-05-13 23:13:28,264 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.212:38535, b=
ytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001438=
_0, offset: 196608, srvID: DS-443352839-145.100.2.183-50050-1291128673616, =
blockid: blk_-9163184839986480695_4112368, duration: 4473000
> 2011-05-13 23:13:28,265 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode: DatanodeRegistration(192.168.28.214:50050, storageID=3DDS-443352839-14=
5.100.2.183-50050-1291128673616, infoPort=3D50075, ipcPort=3D50020): Except=
ion writing block blk_-9150014886921014525_2267869 to mirror 192.168.28.213=
:50050
> java.io.IOException: The stream is closed
>        at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStre=
am.java:108)
>        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.j=
ava:65)
>        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:12=
3)
>        at java.io.DataOutputStream.flush(DataOutputStream.java:106)
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlo=
ck(BlockReceiver.java:540)
>        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(D=
ataXceiver.java:417)
>        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXcei=
ver.java:122)
>
> 2011-05-13 23:13:28,265 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.213:45484, b=
ytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001432=
_0, offset: 0, srvID: DS-443352839-145.100.2.183-50050-1291128673616, block=
id: blk_405051931214094755_4098504, duration: 5597000
> 2011-05-13 23:13:28,273 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode: Received block blk_-9150014886921014525_2267869 src: /192.168.28.211:4=
9208 dest: /192.168.28.214:50050 of size 3033173
> 2011-05-13 23:13:28,313 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode: Received block blk_-9144765354308563975_3310572 src: /192.168.28.211:5=
1592 dest: /192.168.28.214:50050 of size 242383
>
> ________________________________________
> From: Segel, Mike [msegel@navteq.com]
> Sent: Friday, May 13, 2011 2:36 PM
> To: general@hadoop.apache.org
> Cc: <cdh-user@cloudera.org>; <general@hadoop.apache.org>
> Subject: Re: Stability issue - dead DN's
>
> Bonded will work but you may not see the performance you would expect.  I=
f you need >1 GBe, go 10GBe less headache and has even more headroom.
>
> Multiple interfaces won't work. Or I should say didn't work in past relea=
ses.
> If you think about it, clients have to connect to each node. So having tw=
o interfaces and trying to manage them makes no sense.
>
> Add to this trying to manage this in DNS ... Why make more work for yours=
elf?
> Going from memory... It looked like you rDNS had to match you hostnames s=
o your internal interfaces had to match hostnames so you had an inverted ne=
twork.
>
> If you draw out your network topology you end up with a ladder.
> You would be better off (IMHO) to create a subnet where only your edge se=
rvers are dual nic'd.
> But then if your cluster is for development... Now your PCs can't be used=
 as clients...
>
> Does this make sense?
>
>
> Sent from a remote device. Please excuse any typos...
>
> Mike Segel
>
> On May 13, 2011, at 4:57 AM, "Evert Lammerts" <Evert.Lammerts@sara.nl> wr=
ote:
>
>> Hi Mike,
>>
>>> You really really don't want to do this.
>>> Long story short... It won't work.
>>
>> Can you elaborate? Are you talking about the bonded interfaces or about =
having a separated network for interconnects and external network? What can=
 go wrong there?
>>
>>>
>>> Just a suggestion.. You don't want anyone on your cluster itself. They
>>> should interact wit edge nodes, which are 'Hadoop aware'. Then your
>>> cluster has a single network to worry about.
>>
>> That's our current setup. We have a single headnode that is used as a SP=
OE. However, I'd like to change that on our future production system. We wa=
nt to implement Kerberos for authentication, and let users interact with th=
e cluster from their own machine. This would enable them to submit their jo=
bs from the local IDE. The only way to do this is by opening up Hadoop port=
s for the world, is my understanding: if people interact with HDFS they nee=
d to be able to interact with all nodes, right? What would be the argument =
against this?
>>
>> Cheers,
>> Evert
>>
>>>
>>>
>>> Sent from a remote device. Please excuse any typos...
>>>
>>> Mike Segel
>>>
>>> On May 11, 2011, at 11:45 AM, Allen Wittenauer <aw@apache.org> wrote:
>>>
>>>>
>>>>
>>>>
>>>>
>>>>>> * a 2x1GE bonded network interface for interconnects
>>>>>> * a 2x1GE bonded network interface for external access
>>>>
>>>>  Multiple NICs on a box can sometimes cause big performance
>>> problems with Hadoop.  So watch your traffic carefully.
>>>>
>>>>
>>>>
>
>
> The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are no=
t the intended recipient, you are hereby notified that any dissemination, d=
istribution, or copying of this communication, or any of its contents, is s=
trictly prohibited.  If you have received this communication in error, plea=
se notify the sender and delete/destroy the original message and any copy o=
f it from your computer or paper files.


The information contained in this communication may be CONFIDENTIAL and is =
intended only for the use of the recipient(s) named above.  If you are not =
the intended recipient, you are hereby notified that any dissemination, dis=
tribution, or copying of this communication, or any of its contents, is str=
ictly prohibited.  If you have received this communication in error, please=
 notify the sender and delete/destroy the original message and any copy of =
it from your computer or paper files.

From general-return-3538-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 20:09:38 2011
Return-Path: <general-return-3538-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D69A16D81
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat, 14 May 2011 20:09:38 +0000 (UTC)
Received: (qmail 26524 invoked by uid 500); 14 May 2011 20:09:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26471 invoked by uid 500); 14 May 2011 20:09:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26463 invoked by uid 99); 14 May 2011 20:09:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 20:09:37 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [145.100.8.49] (HELO smtp.sara.nl) (145.100.8.49)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 20:09:29 +0000
Received: from planck.ka.sara.nl (145.100.8.32) by sara-exch-fe1.ka.sara.nl
 (145.100.8.49) with Microsoft SMTP Server (TLS) id 14.0.702.0; Sat, 14 May
 2011 22:09:07 +0200
Received: from planck.ka.sara.nl ([145.100.8.32]) by planck.ka.sara.nl
 ([145.100.8.32]) with mapi; Sat, 14 May 2011 22:09:07 +0200
From: Evert Lammerts <Evert.Lammerts@sara.nl>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Sat, 14 May 2011 22:08:01 +0200
Subject: RE: Stability issue - dead DN's
Thread-Topic: Stability issue - dead DN's
Thread-Index: AcwRvcdfcMDGgpIHSLChYJEq3fXs1wAVPNYpABf5Yeo=
Message-ID: <ADF94D8555C7A246B86A633685E0178AB90A35CA0F@planck.ka.sara.nl>
References: <ADF94D8555C7A246B86A633685E0178AB90A474E7A@planck.ka.sara.nl>
 <BANLkTi=NaiXQPAkdS-LPW-Swev6XYD0Xjw@mail.gmail.com>
 <DA1F03C5-7F7B-412A-84E7-774294085603@apache.org>
 <BLU0-SMTP107D4E21A69A6C860D8CBB2F0860@phx.gbl>
 <ADF94D8555C7A246B86A633685E0178AB90A474E94@planck.ka.sara.nl>
 <FAAD6507-80D9-453A-BCED-916532688B49@navteq.com>
 <ADF94D8555C7A246B86A633685E0178AB90A35CA07@planck.ka.sara.nl>,<DABA1F29-2DC3-480E-ABD6-F690F95CD018@navteq.com>,<ADF94D8555C7A246B86A633685E0178AB90A35CA0A@planck.ka.sara.nl>
In-Reply-To: <ADF94D8555C7A246B86A633685E0178AB90A35CA0A@planck.ka.sara.nl>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Just to check: the NN gives back hostnames of DN's to the client when getti=
ng or putting data, and not IP addresses right?

Cheers,
Evert

________________________________________
From: Evert Lammerts [Evert.Lammerts@sara.nl]
Sent: Saturday, May 14, 2011 10:53 AM
To: general@hadoop.apache.org
Subject: RE: Stability issue - dead DN's

Ok, I'll give this scenario a try (in spite of the intoxication ;-)).

=3D putting or getting a file =3D
A client will access the NameNode first and get a list of hostnames. These =
will resolve to addresses either in public or in private space, depending o=
n whether the request to the nameserver was made by a machine in public or =
in private space. Each node has one NIC listening on its address in private=
 space and one on its address in public space. The Hadoop daemons are bound=
 to 0.0.0.0:*. Reverse DNS will return an address in private space when the=
 client connects from one of the nodes, and (obviously) an address in publi=
c space when the request came through WAN.

I'm not sure what could go wrong here... On Monday I'll recheck this scenar=
io with our HPN guys as well.

Cheers,
Evert


________________________________________
From: Segel, Mike [msegel@navteq.com]
Sent: Saturday, May 14, 2011 12:33 AM
To: general@hadoop.apache.org
Subject: Re: Stability issue - dead DN's

Ok...

Hum, look, I've been force fed a couple of margaritas so, my memory is a bi=
t foggy...
You say your clients connect on nic A. Your cluster connects on nic B.

What happens when you want to upload a file from your client to HDFS? Or ev=
en access it?

... ;-)



Sent from a remote device. Please excuse any typos...

Mike Segel

On May 13, 2011, at 4:15 PM, "Evert Lammerts" <Evert.Lammerts@sara.nl> wrot=
e:

> Hi Mike,
>
> Thanks for trying to help out.
>
> I had a talk with our networking guys this afternoon. According to them (=
and this is way out of my area of expertise, so excuse any mistakes) multip=
le interfaces shouldn't be a problem. We could set up a nameserver to resol=
ve hostnames to addresses in our private space when the request comes from =
one of the nodes, and route this traffic over a single interface. Any other=
 request can be resolved to an address in the public space, which is bound =
to an other interface. In our current setup we're not even resolving hostna=
mes in our private address space through a nameserver - we do it with an ug=
ly hack in /etc/hosts. And it seems to work alright.
>
> Having said that, our problems are still not completely gone even after a=
djusting the maximum allowed RAM for tasks - although things are lots bette=
r. While writing this mail three out of five DN's were marked as dead. Ther=
e still is some swapping going on, but the cores are not spending any time =
in WAIT, so this shouldn't be the cause of anything. See below a trace from=
 a dead DN - any thoughts are appreciated!
>
> Cheers,
> Evert
>
> 2011-05-13 23:13:27,716 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode: Received block blk_-9131821326787012529_2915672 src: /192.168.28.211:6=
0136 dest: /192.168.28.214:50050 of size 382425
> 2011-05-13 23:13:27,915 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode: Exception in receiveBlock for block blk_-9132067116195286882_130888 ja=
va.io.EOFException: while trying to read 3744913 bytes
> 2011-05-13 23:13:27,925 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.214:35139, b=
ytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001437=
_0, offset: 196608, srvID: DS-443352839-145.100.2.183-50050-1291128673616, =
blockid: blk_-9163184839986480695_4112368, duration: 6254000
> 2011-05-13 23:13:28,032 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode: Received block blk_-9149862728087355005_3793421 src: /192.168.28.210:4=
1197 dest: /192.168.28.214:50050 of size 245767
> 2011-05-13 23:13:28,033 WARN org.apache.hadoop.hdfs.server.datanode.DataN=
ode: Block blk_-9132067116195286882_130888 unfinalized and removed.
> 2011-05-13 23:13:28,033 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode: writeBlock blk_-9132067116195286882_130888 received exception java.io.=
EOFException: while trying to read 3744913 bytes
> 2011-05-13 23:13:28,033 ERROR org.apache.hadoop.hdfs.server.datanode.Data=
Node: DatanodeRegistration(192.168.28.214:50050, storageID=3DDS-443352839-1=
45.100.2.183-50050-1291128673616, infoPort=3D50075, ipcPort=3D50020):DataXc=
eiver
> java.io.EOFException: while trying to read 3744913 bytes
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(=
BlockReceiver.java:270)
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPa=
cket(BlockReceiver.java:357)
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePac=
ket(BlockReceiver.java:378)
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlo=
ck(BlockReceiver.java:534)
>        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(D=
ataXceiver.java:417)
>        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXcei=
ver.java:122)
> 2011-05-13 23:13:28,038 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.214:32910, b=
ytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001443=
_0, offset: 197632, srvID: DS-443352839-145.100.2.183-50050-1291128673616, =
blockid: blk_-9163184839986480695_4112368, duration: 4323000
> 2011-05-13 23:13:28,038 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.214:35138, b=
ytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001440=
_0, offset: 197120, srvID: DS-443352839-145.100.2.183-50050-1291128673616, =
blockid: blk_-9163184839986480695_4112368, duration: 5573000
> 2011-05-13 23:13:28,159 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.212:38574, b=
ytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001444=
_0, offset: 197632, srvID: DS-443352839-145.100.2.183-50050-1291128673616, =
blockid: blk_-9163184839986480695_4112368, duration: 16939000
> 2011-05-13 23:13:28,209 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode: Received block blk_-9123390874940601805_2898225 src: /192.168.28.210:4=
4227 dest: /192.168.28.214:50050 of size 300441
> 2011-05-13 23:13:28,217 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.213:42364, b=
ytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001451=
_0, offset: 198656, srvID: DS-443352839-145.100.2.183-50050-1291128673616, =
blockid: blk_-9163184839986480695_4112368, duration: 5291000
> 2011-05-13 23:13:28,252 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.214:32930, b=
ytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001436=
_0, offset: 0, srvID: DS-443352839-145.100.2.183-50050-1291128673616, block=
id: blk_-1800696633107072247_4099834, duration: 5099000
> 2011-05-13 23:13:28,256 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.213:42363, b=
ytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001458=
_0, offset: 199680, srvID: DS-443352839-145.100.2.183-50050-1291128673616, =
blockid: blk_-9163184839986480695_4112368, duration: 4945000
> 2011-05-13 23:13:28,257 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.214:35137, b=
ytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001436=
_0, offset: 196608, srvID: DS-443352839-145.100.2.183-50050-1291128673616, =
blockid: blk_-9163184839986480695_4112368, duration: 4159000
> 2011-05-13 23:13:28,258 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode: Exception in receiveBlock for block blk_-9140444589483291821_3585975 j=
ava.io.EOFException: while trying to read 100 bytes
> 2011-05-13 23:13:28,258 WARN org.apache.hadoop.hdfs.server.datanode.DataN=
ode: Block blk_-9140444589483291821_3585975 unfinalized and removed.
> 2011-05-13 23:13:28,258 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode: writeBlock blk_-9140444589483291821_3585975 received exception java.io=
.EOFException: while trying to read 100 bytes
> 2011-05-13 23:13:28,259 ERROR org.apache.hadoop.hdfs.server.datanode.Data=
Node: DatanodeRegistration(192.168.28.214:50050, storageID=3DDS-443352839-1=
45.100.2.183-50050-1291128673616, infoPort=3D50075, ipcPort=3D50020):DataXc=
eiver
> java.io.EOFException: while trying to read 100 bytes
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(=
BlockReceiver.java:270)
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPa=
cket(BlockReceiver.java:357)
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePac=
ket(BlockReceiver.java:378)
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlo=
ck(BlockReceiver.java:534)
>        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(D=
ataXceiver.java:417)
>        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXcei=
ver.java:122)
> 2011-05-13 23:13:28,264 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.212:38553, b=
ytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001441=
_0, offset: 0, srvID: DS-443352839-145.100.2.183-50050-1291128673616, block=
id: blk_-5819719631677148140_4098274, duration: 5625000
> 2011-05-13 23:13:28,264 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.212:38535, b=
ytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001438=
_0, offset: 196608, srvID: DS-443352839-145.100.2.183-50050-1291128673616, =
blockid: blk_-9163184839986480695_4112368, duration: 4473000
> 2011-05-13 23:13:28,265 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode: DatanodeRegistration(192.168.28.214:50050, storageID=3DDS-443352839-14=
5.100.2.183-50050-1291128673616, infoPort=3D50075, ipcPort=3D50020): Except=
ion writing block blk_-9150014886921014525_2267869 to mirror 192.168.28.213=
:50050
> java.io.IOException: The stream is closed
>        at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStre=
am.java:108)
>        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.j=
ava:65)
>        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:12=
3)
>        at java.io.DataOutputStream.flush(DataOutputStream.java:106)
>        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlo=
ck(BlockReceiver.java:540)
>        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(D=
ataXceiver.java:417)
>        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXcei=
ver.java:122)
>
> 2011-05-13 23:13:28,265 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode.clienttrace: src: /192.168.28.214:50050, dest: /192.168.28.213:45484, b=
ytes: 0, op: HDFS_READ, cliID: DFSClient_attempt_201105131125_0025_m_001432=
_0, offset: 0, srvID: DS-443352839-145.100.2.183-50050-1291128673616, block=
id: blk_405051931214094755_4098504, duration: 5597000
> 2011-05-13 23:13:28,273 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode: Received block blk_-9150014886921014525_2267869 src: /192.168.28.211:4=
9208 dest: /192.168.28.214:50050 of size 3033173
> 2011-05-13 23:13:28,313 INFO org.apache.hadoop.hdfs.server.datanode.DataN=
ode: Received block blk_-9144765354308563975_3310572 src: /192.168.28.211:5=
1592 dest: /192.168.28.214:50050 of size 242383
>
> ________________________________________
> From: Segel, Mike [msegel@navteq.com]
> Sent: Friday, May 13, 2011 2:36 PM
> To: general@hadoop.apache.org
> Cc: <cdh-user@cloudera.org>; <general@hadoop.apache.org>
> Subject: Re: Stability issue - dead DN's
>
> Bonded will work but you may not see the performance you would expect.  I=
f you need >1 GBe, go 10GBe less headache and has even more headroom.
>
> Multiple interfaces won't work. Or I should say didn't work in past relea=
ses.
> If you think about it, clients have to connect to each node. So having tw=
o interfaces and trying to manage them makes no sense.
>
> Add to this trying to manage this in DNS ... Why make more work for yours=
elf?
> Going from memory... It looked like you rDNS had to match you hostnames s=
o your internal interfaces had to match hostnames so you had an inverted ne=
twork.
>
> If you draw out your network topology you end up with a ladder.
> You would be better off (IMHO) to create a subnet where only your edge se=
rvers are dual nic'd.
> But then if your cluster is for development... Now your PCs can't be used=
 as clients...
>
> Does this make sense?
>
>
> Sent from a remote device. Please excuse any typos...
>
> Mike Segel
>
> On May 13, 2011, at 4:57 AM, "Evert Lammerts" <Evert.Lammerts@sara.nl> wr=
ote:
>
>> Hi Mike,
>>
>>> You really really don't want to do this.
>>> Long story short... It won't work.
>>
>> Can you elaborate? Are you talking about the bonded interfaces or about =
having a separated network for interconnects and external network? What can=
 go wrong there?
>>
>>>
>>> Just a suggestion.. You don't want anyone on your cluster itself. They
>>> should interact wit edge nodes, which are 'Hadoop aware'. Then your
>>> cluster has a single network to worry about.
>>
>> That's our current setup. We have a single headnode that is used as a SP=
OE. However, I'd like to change that on our future production system. We wa=
nt to implement Kerberos for authentication, and let users interact with th=
e cluster from their own machine. This would enable them to submit their jo=
bs from the local IDE. The only way to do this is by opening up Hadoop port=
s for the world, is my understanding: if people interact with HDFS they nee=
d to be able to interact with all nodes, right? What would be the argument =
against this?
>>
>> Cheers,
>> Evert
>>
>>>
>>>
>>> Sent from a remote device. Please excuse any typos...
>>>
>>> Mike Segel
>>>
>>> On May 11, 2011, at 11:45 AM, Allen Wittenauer <aw@apache.org> wrote:
>>>
>>>>
>>>>
>>>>
>>>>
>>>>>> * a 2x1GE bonded network interface for interconnects
>>>>>> * a 2x1GE bonded network interface for external access
>>>>
>>>>  Multiple NICs on a box can sometimes cause big performance
>>> problems with Hadoop.  So watch your traffic carefully.
>>>>
>>>>
>>>>
>
>
> The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are no=
t the intended recipient, you are hereby notified that any dissemination, d=
istribution, or copying of this communication, or any of its contents, is s=
trictly prohibited.  If you have received this communication in error, plea=
se notify the sender and delete/destroy the original message and any copy o=
f it from your computer or paper files.


The information contained in this communication may be CONFIDENTIAL and is =
intended only for the use of the recipient(s) named above.  If you are not =
the intended recipient, you are hereby notified that any dissemination, dis=
tribution, or copying of this communication, or any of its contents, is str=
ictly prohibited.  If you have received this communication in error, please=
 notify the sender and delete/destroy the original message and any copy of =
it from your computer or paper files.

From general-return-3539-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 05:19:53 2011
Return-Path: <general-return-3539-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 998376F14
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 16 May 2011 05:19:53 +0000 (UTC)
Received: (qmail 38250 invoked by uid 500); 16 May 2011 05:19:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37869 invoked by uid 500); 16 May 2011 05:19:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37860 invoked by uid 99); 16 May 2011 05:19:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 05:19:45 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 05:19:40 +0000
Received: from [10.0.1.3] (snvvpn4-10-72-168-c136.hq.corp.yahoo.com [10.72.168.136])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p4G5Ispg062453
	for <general@hadoop.apache.org>; Sun, 15 May 2011 22:18:56 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1305523136;
	bh=6l/dREewmWaD2UWiCVswgloVIJtGyxlfCKtzoi6yDqg=;
	h=Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:
	 Content-Transfer-Encoding:Message-Id:References:To;
	b=F3cpbPZOc1TavZpWFBlxTC0FWAYMh6P2Oyu8ICXe6h/09MoHYhAkWfYtkr1ogHL3j
	 tbla4/XBp8H+iT8KR1A6Pb5QrdsOlb1lqql5lGfrfR6x26/RJbJtIZ/4XqUU49pias
	 2ikCY/N/tMeChnOUqnjp0ZOrU7RUNPd/PKkHBG00=
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: What is "Hadoop?" Was: Defining Hadoop Compatibility -revisiting-
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <3B4F5C8A-3404-4435-9EC3-88438BCC5730@Holsman.NET>
Date: Sun, 15 May 2011 22:18:53 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <0F7B3B62-03AB-40AE-AF03-68669A963CD8@yahoo-inc.com>
References: <BANLkTi=1+++j55GER6Tvuw=pRbvApKmy-g@mail.gmail.com> <3B4F5C8A-3404-4435-9EC3-88438BCC5730@Holsman.NET>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1084)

Interesting point!  I can see a future where there are many folks mixing =
and matching hadoop and non-hadoop components.  Swapping out HDFS seems =
particularly popular.=20

On May 13, 2011, at 4:17 PM, Ian Holsman wrote:
...
> I think thats a great idea.=20
> Maybe we should also create names/marks around the interfaces as well.
>=20
>> -- Owen
>=20


From general-return-3540-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 05:35:18 2011
Return-Path: <general-return-3540-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E025B6566
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 16 May 2011 05:35:17 +0000 (UTC)
Received: (qmail 49869 invoked by uid 500); 16 May 2011 05:35:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49806 invoked by uid 500); 16 May 2011 05:35:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49798 invoked by uid 99); 16 May 2011 05:35:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 05:35:15 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 05:35:10 +0000
Received: from [10.0.1.3] (snvvpn4-10-72-168-c136.hq.corp.yahoo.com [10.72.168.136])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p4G5YX6M066673
	for <general@hadoop.apache.org>; Sun, 15 May 2011 22:34:35 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1305524075;
	bh=zIzRqARSiAT/cJ9mk7FgW5KIpc9vWbUr2bkKhea+pWc=;
	h=Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:
	 Content-Transfer-Encoding:Message-Id:References:To;
	b=HG4EQIGq7BFYZdK5MoRe6Kkf0q/BlNpLXI9rIUvj55SPLk7jsjs+wMjI3tG37q8ne
	 BITcV32s6z0AiTrLjpgNF+jN3T/GKtjoAny6pRHhcrHS2ttAYZkh8Gvr9sJ8yNkV7Q
	 rHxG+uIdCIEWuA3zjzLt90cHmeVI3RGOEfAUNG4c=
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: Defining Hadoop Compatibility -revisiting-
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <4DCCCCB2.6020607@apache.org>
Date: Sun, 15 May 2011 22:34:33 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <76368DCE-92A1-4D90-AA04-3074E6BE68AB@yahoo-inc.com>
References: <4DC91392.2010308@apache.org> <4DCCCCB2.6020607@apache.org>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1084)

Good point.

On May 12, 2011, at 11:16 PM, Doug Cutting wrote:

> Certification semms like mission creep.  Our mission is to produce
> open-source software.  If we wish to produce testing software, that
> seems fine.  But running a certification program for non-open-source
> software seems like a different task.
>=20
> The Hadoop mark should only be used to refer to open-source software
> produced by the ASF.  If other folks wish to make factual statements
> concerning our software, e.g., that their proprietary software passes
> tests that we've created, that may be fine, but I don't think we =
should
> validate those claims by granting certifications to institutions.  =
That
> ventures outside the mission of the ASF.  We are not an accrediting
> organization.
>=20
> Doug
>=20
> On 05/10/2011 12:29 PM, Steve Loughran wrote:
>>=20
>> Back in Jan 2011, I started a discussion about how to define Apache
>> Hadoop Compatibility:
>> =
http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C4D4=
6B6AD.2020802@apache.org%3E
>>=20
>>=20
>> I am now reading EMC HD "Enterprise Ready" Apache Hadoop datasheet
>>=20
>> =
http://www.greenplum.com/sites/default/files/EMC_Greenplum_HD_DS_Final_1.p=
df
>>=20
>>=20
>> It claims that their implementations are 100% compatible, even though
>> the Enterprise edition uses a C filesystem. It also claims that both
>> their software releases contain "Certified Stacks", without defining
>> what Certified means, or who does the certification -only that it is =
an
>> improvement.
>>=20
>>=20
>> I think we should revisit this issue before people with their own
>> agendas define what compatibility with Apache Hadoop is for us
>>=20
>>=20
>> Licensing
>> -Use of the Hadoop codebase must follow the Apache License
>> http://www.apache.org/licenses/LICENSE-2.0
>> -plug in components that are dynamically linked to (Filesystems and
>> schedulers) don't appear to be derivative works on my reading of =
this,
>>=20
>> Naming
>> -this is something for branding@apache, they will have their =
opinions.
>> The key one is that the name "Apache Hadoop" must get used, and it's
>> important to make clear it is a derivative work.
>> -I don't think you can claim to have a Distribution/Fork/Version of
>> Apache Hadoop if you swap out big chunks of it for alternate
>> filesystems, MR engines, etc. Some description of this is needed
>> "Supports the Apache Hadoop MapReduce engine on top of Filesystem =
XYZ"
>>=20
>> Compatibility
>> -the definition of the Hadoop interfaces and classes is the Apache
>> Source tree,
>> -the definition of semantics of the Hadoop interfaces and classes is
>> the Apache Source tree, including the test classes.
>> -the verification that the actual semantics of an Apache Hadoop =
release
>> is compatible with the expected semantics is that current and future
>> tests pass
>> -bug reports can highlight incompatibility with expectations of
>> community users, and once incorporated into tests form part of the
>> compatibility testing
>> -vendors can claim and even certify their derivative works as
>> compatible with other versions of their derivative works, but cannot
>> claim compatibility with Apache Hadoop unless their code passes the
>> tests and is consistent with the bug reports marked as ("by design").
>> Perhaps we should have tests that verify each of these "by design"
>> bugreps to make them more formal.
>>=20
>> Certification
>> -I have no idea what this means in EMC's case, they just say =
"Certified"
>> -As we don't do any certification ourselves, it would seem impossible
>> for us to certify that any derivative work is compatible.
>> -It may be best to state that nobody can certify their derivative as
>> "compatible with Apache Hadoop" unless it passes all current test =
suites
>> -And require that anyone who declares compatibility define what they
>> mean by this
>>=20
>> This is a good argument for getting more functional tests out there
>> -whoever has more functional tests needs to get them into a test =
module
>> that can be used to test real deployments.
>>=20


From general-return-3541-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 05:35:42 2011
Return-Path: <general-return-3541-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6958A6576
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 16 May 2011 05:35:42 +0000 (UTC)
Received: (qmail 51416 invoked by uid 500); 16 May 2011 05:35:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51262 invoked by uid 500); 16 May 2011 05:35:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51254 invoked by uid 99); 16 May 2011 05:35:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 05:35:40 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 05:35:34 +0000
Received: from [10.0.1.3] (snvvpn4-10-72-168-c136.hq.corp.yahoo.com [10.72.168.136])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p4G5YX6N066673
	for <general@hadoop.apache.org>; Sun, 15 May 2011 22:34:56 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1305524096;
	bh=ROJsPuYSDGQhH5HMkSvfv4N/yrunrErKFi625OnQqHY=;
	h=Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:
	 Content-Transfer-Encoding:Message-Id:References:To;
	b=u5YVZlvKAFbexyjsK8Z8eldPImxxYDoJiP4wPhS8ZcJWbG8wCykP1+r5AW5QIns00
	 SKf8hU0aeuaWZ1miVJ0S0LWxLa+Lt+/agaBnOPJ7mHjQwkSjCK4Dp2HD3oBM3IsjIc
	 eZJlFTcHhyhwpjyjK+U+chUm59jdz475A7FXjD9o=
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: Defining Hadoop Compatibility -revisiting-
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <C9F21EB2.8F52%mbhandarkar@linkedin.com>
Date: Sun, 15 May 2011 22:34:56 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <B2218E92-BBE9-4A60-B8C2-654FD1C13461@yahoo-inc.com>
References: <C9F21EB2.8F52%mbhandarkar@linkedin.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

Good point. =20

Tests are a must for the Hadoop community to meet its own goals (quality =
and backwards compatibility).  Writing detailed specs for something that =
is evolving this quickly is challenging.  Also in a lot of cases, =
documenting the current APIs to POSIX like detail will mainly convince =
us that we need to design new APIs that are more like POSIX.

The good news is that this is open source and we can crowd source both =
activities, although it will still require a lot of work from our =
committers to validate and integrate this sort of contribution.

E14

On May 12, 2011, at 11:57 PM, Milind Bhandarkar wrote:

...

> Therefore, all I am saying is that, while a POSIX-like spec is a "nice =
to
> have", a test-suite that defines compatibility is a must.
>=20
> - milind


From general-return-3542-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 10:50:52 2011
Return-Path: <general-return-3542-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 992676AB6
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 16 May 2011 10:50:52 +0000 (UTC)
Received: (qmail 66019 invoked by uid 500); 16 May 2011 10:50:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65941 invoked by uid 500); 16 May 2011 10:50:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65933 invoked by uid 99); 16 May 2011 10:50:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 10:50:50 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 10:50:44 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id BCBC9B7E0E
	for <general@hadoop.apache.org>; Mon, 16 May 2011 11:50:21 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id yPtNHTId4Q0A for <general@hadoop.apache.org>;
	Mon, 16 May 2011 11:50:17 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 6AA19B7D65
	for <general@hadoop.apache.org>; Mon, 16 May 2011 11:50:17 +0100 (BST)
MailScanner-NULL-Check: 1306147805.48942@HN6lz3kfG8z8cu6Rbe40Fw
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p4GAo324005307
	for <general@hadoop.apache.org>; Mon, 16 May 2011 11:50:04 +0100 (BST)
Message-ID: <4DD1015B.4090905@apache.org>
Date: Mon, 16 May 2011 11:50:03 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Defining Hadoop Compatibility -revisiting-
References: <C9F20588.8F34%mbhandarkar@linkedin.com>
In-Reply-To: <C9F20588.8F34%mbhandarkar@linkedin.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p4GAo324005307
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 13/05/11 05:52, Milind Bhandarkar wrote:
> Ok, my mistake. They have only asked for documented specifications. I may
> have been influenced by all the specifications I have read. All of them
> were in English, which is characterized as a natural language.
>
> But then, if you are proposing a specification in a non-natural-language,
> isn't that called a test suite ? Or is there a middle ground ?

There's formal specifications in languages like Z, We don't really want 
to go there if we can help it, as all it lets you do is prove 
correctness if you're a mathematician, and I haven't found the 
mathematician plugin for Jenkins yet.

There's also languages like Extended ML, from Sanella et al, who may be 
familiar to Doug from his time in the frozen lands of the north 
(edinburgh):
	http://homepages.inf.ed.ac.uk/dts/eml/
Some of the bits of spec in this language can be executed, as long as 
you don't start declaring things about state over time. Again, though, 
it's hard work, unless your target language is, say ML or Haskell, as 
there you can jump from Specification to Implementation fairly rapidly

Where the formal stuff is good for is things like consistency protocols, 
so I'd hope someone did get out the proofs for Zookeeper, so the rest of 
us can rely on it working.

-Steve



From general-return-3543-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 11:02:37 2011
Return-Path: <general-return-3543-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CF55F6913
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 16 May 2011 11:02:37 +0000 (UTC)
Received: (qmail 80651 invoked by uid 500); 16 May 2011 11:02:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80565 invoked by uid 500); 16 May 2011 11:02:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80557 invoked by uid 99); 16 May 2011 11:02:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 11:02:35 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 11:02:27 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 1F71C1BA5A5
	for <general@hadoop.apache.org>; Mon, 16 May 2011 12:02:05 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id uNk7LbEzEwsC for <general@hadoop.apache.org>;
	Mon, 16 May 2011 12:02:04 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 9E05B1BA534
	for <general@hadoop.apache.org>; Mon, 16 May 2011 12:02:04 +0100 (BST)
MailScanner-NULL-Check: 1306148512.654@xHKeAASKaWnqshtZ1RElVg
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p4GB1qpg005671
	for <general@hadoop.apache.org>; Mon, 16 May 2011 12:01:52 +0100 (BST)
Message-ID: <4DD10420.7060004@apache.org>
Date: Mon, 16 May 2011 12:01:52 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Defining Hadoop Compatibility -revisiting-
References: <C9F227D5.8FA4%mbhandarkar@linkedin.com> <4DCCF172.5010504@apache.org> <99AB77CF-025E-4E4A-BD7F-BDA362E633BF@apache.org> <4DCDA8E7.5090809@apache.org> <2ED3A7F5-E549-4545-9395-094A10EA4EED@apache.org> <BANLkTik=EkwPxvcw2-vw=hsofsJYRgyaGg@mail.gmail.com> <BEC4433A-2270-47CF-960D-CB416B20FD4A@apache.org>
In-Reply-To: <BEC4433A-2270-47CF-960D-CB416B20FD4A@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p4GB1qpg005671
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 13/05/11 23:57, Allen Wittenauer wrote:
>
> On May 13, 2011, at 3:53 PM, Ted Dunning wrote:
>
>> But "distribution Z includes X" kind of implies the existence of some such
>> that X != Y, Y != empty-set and X+Y = Z, at least in common usage.
>>
>> Isn't that the same as a non-trunk change?
>>
>> So doesn't this mean that your question reduces to the question of what
>> happens when non-Apache changes are made to an Apache release?  And isn't
>> that the definition of a derived work?
>
>
> 	Yup. Which is why I doubt *any* commercial entity can claim "includes Apache Hadoop" (including Cloudera).
>
>

but they can claim it is a derivative work, which CDH clearly is, 
(Though if we were to come up with a formal declaration of what a 
derivative work is, we'd have to handle the fact that it is a superset. 
Even worse, you may realise a release is the ordered application of a 
sequence of patches, and if the patches are applied in a different order 
you may end up with a different body of source code...)

Something that implements the APIs may not be a derivative work, 
depending on how much of the original code is in there. You could look 
at the base classes and interfaces and produce a clean room 
implementation (relying on the notion that interfaces are a list of 
facts and not copyrightable in the US), but whoever does that may 
encounter the issue that Google's donation of the right to use their MR 
patent may not apply to such implementations.

From general-return-3544-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 11:16:12 2011
Return-Path: <general-return-3544-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7AA4A669F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 16 May 2011 11:16:12 +0000 (UTC)
Received: (qmail 92626 invoked by uid 500); 16 May 2011 11:16:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92575 invoked by uid 500); 16 May 2011 11:16:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 92566 invoked by uid 99); 16 May 2011 11:16:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 11:16:11 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 11:16:02 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 7CC9B1BA757
	for <general@hadoop.apache.org>; Mon, 16 May 2011 12:15:41 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id as5UGAoM6Urn for <general@hadoop.apache.org>;
	Mon, 16 May 2011 12:15:41 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 0A9FA1BA5A5
	for <general@hadoop.apache.org>; Mon, 16 May 2011 12:15:40 +0100 (BST)
MailScanner-NULL-Check: 1306149329.09556@sPfeztXRZBWZVEaHUTxVKg
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p4GBFRrR006035
	for <general@hadoop.apache.org>; Mon, 16 May 2011 12:15:27 +0100 (BST)
Message-ID: <4DD1074F.1040009@apache.org>
Date: Mon, 16 May 2011 12:15:27 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Defining Hadoop Compatibility -revisiting-
References: <C9F227D5.8FA4%mbhandarkar@linkedin.com> <4DCCF172.5010504@apache.org> <99AB77CF-025E-4E4A-BD7F-BDA362E633BF@apache.org> <4DCDA8E7.5090809@apache.org> <2ED3A7F5-E549-4545-9395-094A10EA4EED@apache.org> <4DCDADA0.5090704@apache.org>
In-Reply-To: <4DCDADA0.5090704@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p4GBFRrR006035
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 13/05/11 23:16, Doug Cutting wrote:
> On 05/14/2011 12:13 AM, Allen Wittenauer wrote:
>> So what do we do about companies that release a product that says "includes Apache Hadoop" but includes patches that aren't committed to trunk?
>
> We yell at them to get those patches into trunk already.  This policy
> was clarified after that product was shipping.
>
> Doug

I distributed some RPMs with my lifecycle branch in, I can't remember 
what I called them, but I'd better revist all my .spec files to make 
sure the text is valid. Even with 0.21 JARs, what should I call it?


sf-apache-hadoop-operations.rpm

"This RPM contains the JAR artifacts of Apache Hadoop 0.21 and SmartFrog 
components to manage hadoop clusters,  manipulate the distributed 
filesystems, and submit MapReduce jobs"

Would that work?

From general-return-3545-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 11:21:08 2011
Return-Path: <general-return-3545-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 748836888
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 16 May 2011 11:21:08 +0000 (UTC)
Received: (qmail 98786 invoked by uid 500); 16 May 2011 11:21:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98721 invoked by uid 500); 16 May 2011 11:21:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98713 invoked by uid 99); 16 May 2011 11:21:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 11:21:06 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 11:20:58 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 00DE41BA5A5
	for <general@hadoop.apache.org>; Mon, 16 May 2011 12:20:37 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id 6uJWg395+iYx for <general@hadoop.apache.org>;
	Mon, 16 May 2011 12:20:36 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 77A091BA534
	for <general@hadoop.apache.org>; Mon, 16 May 2011 12:20:36 +0100 (BST)
MailScanner-NULL-Check: 1306149624.59408@MK2D6cYjf8Y2qQoUy5zxYQ
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p4GBKOtQ006148
	for <general@hadoop.apache.org>; Mon, 16 May 2011 12:20:24 +0100 (BST)
Message-ID: <4DD10878.4040109@apache.org>
Date: Mon, 16 May 2011 12:20:24 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Defining Hadoop Compatibility -revisiting-
References: <4DC91392.2010308@apache.org> <4DCCCCB2.6020607@apache.org>
In-Reply-To: <4DCCCCB2.6020607@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p4GBKOtQ006148
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 13/05/11 07:16, Doug Cutting wrote:
> Certification semms like mission creep.  Our mission is to produce
> open-source software.  If we wish to produce testing software, that
> seems fine.  But running a certification program for non-open-source
> software seems like a different task.
>

+1

That said, some stricter definition of public interfaces may be useful 
for the related projects, as a consistent open source stack is strongly 
beneficial.

> The Hadoop mark should only be used to refer to open-source software
> produced by the ASF.  If other folks wish to make factual statements
> concerning our software, e.g., that their proprietary software passes
> tests that we've created, that may be fine, but I don't think we should
> validate those claims by granting certifications to institutions.  That
> ventures outside the mission of the ASF.  We are not an accrediting
> organization.

+1. Apache is not a standards body, except in the form of "de-facto 
standards defined by working code and their test suite"

What it does have a strict rules about naming. We should formalise them 
and publish them on the wiki, then whenever some product gets 
press-released (it's like a beta-release, only earlier in the 
lifecycle), the vendor can be directed to the page and reminded of the 
T&Cs of the license and any trade marks.

What does this mean for T-Shirts and Stickers, incidentally?


From general-return-3546-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 12:01:00 2011
Return-Path: <general-return-3546-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F01E66C8E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 16 May 2011 12:01:00 +0000 (UTC)
Received: (qmail 53729 invoked by uid 500); 16 May 2011 12:00:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53651 invoked by uid 500); 16 May 2011 12:00:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53643 invoked by uid 99); 16 May 2011 12:00:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 12:00:59 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of msegel@navteq.com designates 65.167.11.240 as permitted sender)
Received: from [65.167.11.240] (HELO xmailchi.navteq.com) (65.167.11.240)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 12:00:51 +0000
Received: from chicago-spamfw.navteq.com (imailchi [10.8.5.161])
	by xmailchi.navteq.com (8.13.8/8.13.6) with ESMTP id p4GC0T8l011832
	for <general@hadoop.apache.org>; Mon, 16 May 2011 07:00:29 -0500
Received: from imailchi.hq.navteq.com (localhost [127.0.0.1])
	by chicago-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id 5291841CA57D
	for <general@hadoop.apache.org>; Mon, 16 May 2011 07:00:29 -0500 (CDT)
Received: from imailchi.hq.navteq.com ([10.8.120.104]) by chicago-spamfw.navteq.com with ESMTP id aqEVORZ8HS07o1gw for <general@hadoop.apache.org>; Mon, 16 May 2011 07:00:29 -0500 (CDT)
Received: from hq-ex-ht02.ad.navteq.com (hq-ex-ht02.ad.navteq.com [10.8.222.172])
	by imailchi.hq.navteq.com (8.12.9/8.12.9) with ESMTP id p4GC0TNe029270
	for <general@hadoop.apache.org>; Mon, 16 May 2011 07:00:29 -0500
Received: from hq-ex-mb03.ad.navteq.com ([fe80::c4dd:7b21:5c22:cfe4]) by
 hq-ex-ht02.ad.navteq.com ([fe80::24f6:ca50:734d:df6e%13]) with mapi; Mon, 16
 May 2011 07:00:28 -0500
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 16 May 2011 07:00:44 -0500
Subject: Re: Defining Hadoop Compatibility -revisiting-
Thread-Topic: Defining Hadoop Compatibility -revisiting-
Thread-Index: AcwTwNj/zlsDHXdeTBGDSG818Gfdng==
Message-ID: <A4FB782E-2433-47F1-AE91-CEFC42654C3A@navteq.com>
References: <C9F227D5.8FA4%mbhandarkar@linkedin.com>
 <4DCCF172.5010504@apache.org>
 <99AB77CF-025E-4E4A-BD7F-BDA362E633BF@apache.org>
 <4DCDA8E7.5090809@apache.org>
 <2ED3A7F5-E549-4545-9395-094A10EA4EED@apache.org>
 <BANLkTik=EkwPxvcw2-vw=hsofsJYRgyaGg@mail.gmail.com>
 <BEC4433A-2270-47CF-960D-CB416B20FD4A@apache.org>
 <4DD10420.7060004@apache.org>
In-Reply-To: <4DD10420.7060004@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1
X-Virus-Checked: Checked by ClamAV on apache.org

But Cloudera's release is a bit murky.

The math example is a bit flawed...

X represents the set of stable releases.
Y represents the set of available patches.
C represents the set of Cloudera releases.

So if C contains a release X(n) plus a set of patches that is contained i=
n Y,
Then does it not have the right to be considered Apache Hadoop?
It's my understanding is that any enhancement to Hadoop is made available=
 to Apache and will eventually make it into a later release...

So while it may not be 'official' release X(z), all of it's components ar=
e in Apache.
(note: I'm talking about the core components and not Cloudera's additiona=
l toolsets that encompass Hadoop.)

Cloudera is clearly a derivative work.
And IMHO is the only one which can say ... 'Includes Apache Hadoop'.

That doesn't mean that others can't, depending on how they implemented th=
eir changes.
Based on EMC marketing material, they've done a rip and replace of HDFS.
So it wouldn't be a superset since it doesn't contain a complete subset, =
but contains code that implements the API... So they can't say 'Includes =
Apache Hadoop',but they can say it's a derivative work based on Apache Ha=
doop and then go on to show how and why, in their opinion their product i=
s better.(that's marketing for you...)

Clearly there are others out there...=20
Hadoop on Cassandra as an example...

Fragmentation of Hadoop will occur. It's inevitable. Too much money is on=
 the table...

But because Apache's licensing is so open, Apache will have a hard time c=
ontrolling derivative works... =20
I believe that Steve is incorrect in his assertion concerning potential l=
oss of any patent protection. Again Apache's licensing is very open and a=
s long as they follow Apache's Ts and Cs, they are covered.

Note: because I am sending this from my email address at my client, I am =
obliged to say that this email is my opinion and does not reflect on the =
opinion of my client...
(you know the rest....)

Sent from a remote device. Please excuse any typos...

Mike Segel

On May 16, 2011, at 6:02 AM, "Steve Loughran" <stevel@apache.org> wrote:

> On 13/05/11 23:57, Allen Wittenauer wrote:
>>=20
>> On May 13, 2011, at 3:53 PM, Ted Dunning wrote:
>>=20
>>> But "distribution Z includes X" kind of implies the existence of some=
 such
>>> that X !=3D Y, Y !=3D empty-set and X+Y =3D Z, at least in common usa=
ge.
>>>=20
>>> Isn't that the same as a non-trunk change?
>>>=20
>>> So doesn't this mean that your question reduces to the question of wh=
at
>>> happens when non-Apache changes are made to an Apache release?  And i=
sn't
>>> that the definition of a derived work?
>>=20
>>=20
>>    Yup. Which is why I doubt *any* commercial entity can claim "includ=
es Apache Hadoop" (including Cloudera).
>>=20
>>=20
>=20
> but they can claim it is a derivative work, which CDH clearly is,=20
> (Though if we were to come up with a formal declaration of what a=20
> derivative work is, we'd have to handle the fact that it is a superset.=
=20
> Even worse, you may realise a release is the ordered application of a=20
> sequence of patches, and if the patches are applied in a different orde=
r=20
> you may end up with a different body of source code...)
>=20
> Something that implements the APIs may not be a derivative work,=20
> depending on how much of the original code is in there. You could look=20
> at the base classes and interfaces and produce a clean room=20
> implementation (relying on the notion that interfaces are a list of=20
> facts and not copyrightable in the US), but whoever does that may=20
> encounter the issue that Google's donation of the right to use their MR=
=20
> patent may not apply to such implementations.


The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.

From general-return-3547-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 14:12:03 2011
Return-Path: <general-return-3547-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D9FDF68E6
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 16 May 2011 14:12:03 +0000 (UTC)
Received: (qmail 74200 invoked by uid 500); 16 May 2011 14:12:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74150 invoked by uid 500); 16 May 2011 14:12:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74142 invoked by uid 99); 16 May 2011 14:12:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 14:12:02 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 14:11:53 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 412411BA2FB
	for <general@hadoop.apache.org>; Mon, 16 May 2011 15:11:32 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id RrZjPKvrNq2a for <general@hadoop.apache.org>;
	Mon, 16 May 2011 15:11:31 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 99B6B1BA264
	for <general@hadoop.apache.org>; Mon, 16 May 2011 15:11:31 +0100 (BST)
MailScanner-NULL-Check: 1306159879.68@tu5wJpRoBroUMiEn+Qsy7w
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p4GEBI0V011077
	for <general@hadoop.apache.org>; Mon, 16 May 2011 15:11:19 +0100 (BST)
Message-ID: <4DD13086.1040608@apache.org>
Date: Mon, 16 May 2011 15:11:18 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Defining Hadoop Compatibility -revisiting-
References: <C9F227D5.8FA4%mbhandarkar@linkedin.com> <4DCCF172.5010504@apache.org> <99AB77CF-025E-4E4A-BD7F-BDA362E633BF@apache.org> <4DCDA8E7.5090809@apache.org> <2ED3A7F5-E549-4545-9395-094A10EA4EED@apache.org> <BANLkTik=EkwPxvcw2-vw=hsofsJYRgyaGg@mail.gmail.com> <BEC4433A-2270-47CF-960D-CB416B20FD4A@apache.org> <4DD10420.7060004@apache.org> <A4FB782E-2433-47F1-AE91-CEFC42654C3A@navteq.com>
In-Reply-To: <A4FB782E-2433-47F1-AE91-CEFC42654C3A@navteq.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p4GEBI0V011077
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 16/05/11 13:00, Segel, Mike wrote:
> But Cloudera's release is a bit murky.
>
> The math example is a bit flawed...
>
> X represents the set of stable releases.
> Y represents the set of available patches.
> C represents the set of Cloudera releases.
>
> So if C contains a release X(n) plus a set of patches that is contained in Y,
> Then does it not have the right to be considered Apache Hadoop?
> It's my understanding is that any enhancement to Hadoop is made available to Apache and will eventually make it into a later release...

It certainly contains it.

Now, if you want to make life more complex:
-view the contributions to the code base as a series of patches P1...Pn, 
each of which changes the code.
-These patches are essentially functions that transform the source S to 
a new state S'.
-the initial state of the source codebase is S0.

Hypothesis: the order in which the patch functions are applied 
determines the final state of the source tree.

If patches P1 and P2 were applied in order, you would get a state

S' = P2(P1(S0))

Applying the patches in a different order, you get a new final state.
S'' = P1(P2(S0))


Question for the maths people then is: can you be sure that S' and S'' 
are the same. As it would seem to me that it depends on the nature of 
the function. It could be that the set of functions that SVN supports 
guarantees sameness, but given conflict resolution problems I've 
encountered in the past, I doubt this.

Assuming that my belief holds: that the order in which a series of SVN 
patches are executed determines the final state of the source tree, then 
saying the patch sets -the set of functions applied to the source- of 
two codebases are equivalent does not mean the final state of the code 
is the same unless the sequence of application is also the same.

That would then define an apache release as a strictly ordered sequence 
of patches, or at least an sequence of operations that leads to the same 
final code state, such as S0.20.3

(oh look, I've just written a formal definition of what a release is, 
though I've avoided defining what a function is. View them as planar 
projections in cartesian space or something)


>
> So while it may not be 'official' release X(z), all of it's components are in Apache.
> (note: I'm talking about the core components and not Cloudera's additional toolsets that encompass Hadoop.)
>
> Cloudera is clearly a derivative work.
> And IMHO is the only one which can say ... 'Includes Apache Hadoop'.

Once you start thinking about the ordering of the patch functions it 
gets complicated.

> That doesn't mean that others can't, depending on how they implemented their changes.

yes, though again it depends on the sequence of functions applied to the 
released sourcecode, such as S0.20.3, to the version they ship.

> So it wouldn't be a superset since it doesn't contain a complete subset, but contains code that implements the API... So they can't say 'Includes Apache Hadoop',but they can say it's a derivative work based on Apache Hadoop and then go on to show how and why, in their opinion their product is better.(that's marketing for you...)

I agree

> Fragmentation of Hadoop will occur. It's inevitable. Too much money is on the table...

Clearly, but there are still some questions we can resolve here
  -what do they call their products?
  -how can they support assertions that their code is compatible if the 
series of patches they have applied to the codebase are not externally 
visible?
  -what are the concerns of the community about naming and branching?


>
> But because Apache's licensing is so open, Apache will have a hard time controlling derivative works...

The Apache license permits anyone to fork and take that fork in house or 
closed source. Most people are considered daft to do this except for 
quick fixes, because any closed source takes on the task of writing the 
functions needed to transform it from the released state to one that 
matches customer needs. (i.e. the working state)


> I believe that Steve is incorrect in his assertion concerning potential loss of any patent protection. Again Apache's licensing is very open and as long as they follow Apache's Ts and Cs, they are covered.

Possibly. I avoid such legal issues.

-steve

From general-return-3548-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 16:38:54 2011
Return-Path: <general-return-3548-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 24403696A
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 16 May 2011 16:38:54 +0000 (UTC)
Received: (qmail 69999 invoked by uid 500); 16 May 2011 16:38:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69951 invoked by uid 500); 16 May 2011 16:38:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69943 invoked by uid 99); 16 May 2011 16:38:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 16:38:52 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [145.100.8.49] (HELO smtp.sara.nl) (145.100.8.49)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 16:38:44 +0000
Received: from planck.ka.sara.nl (145.100.8.32) by sara-exch-fe1.ka.sara.nl
 (145.100.8.49) with Microsoft SMTP Server (TLS) id 14.0.702.0; Mon, 16 May
 2011 18:38:21 +0200
Received: from planck.ka.sara.nl ([145.100.8.32]) by planck.ka.sara.nl
 ([145.100.8.32]) with mapi; Mon, 16 May 2011 18:38:21 +0200
From: Evert Lammerts <Evert.Lammerts@sara.nl>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 16 May 2011 18:37:36 +0200
Subject: MAPREDUCE-5
Thread-Topic: MAPREDUCE-5
Thread-Index: AQHME+eqslwvbrLwnkCsqdYAsEnFDA==
Message-ID: <ADF94D8555C7A246B86A633685E0178AB90A35CA1E@planck.ka.sara.nl>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/mixed;
	boundary="_002_ADF94D8555C7A246B86A633685E0178AB90A35CA1Eplanckkasaran_"
MIME-Version: 1.0

--_002_ADF94D8555C7A246B86A633685E0178AB90A35CA1Eplanckkasaran_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

When Reducers start running during a certain job (mapred.reduce.slowstart.c=
ompleted.maps =3D 0.8) it takes about 20 minutes before the DN stopd reacti=
ng. This seems to be due to a number of Exceptions in the TT - at least, it=
's the only place I'm seeing errors. The three recurring ones are getMapOut=
put, EOFException and IllegalStateException. It seems related to https://is=
sues.apache.org/jira/browse/MAPREDUCE-5. See an excerpt from the logs attac=
hed.

We're running Hadoop 0.20.2 on a 6 node (test) cluster with:

# java -version
java version "1.6.0_24"
Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)

Can anybody shed some light on this?

Thanks a bunch,
Evert

--_002_ADF94D8555C7A246B86A633685E0178AB90A35CA1Eplanckkasaran_
Content-Type: text/x-log; name="tt.log"
Content-Description: tt.log
Content-Disposition: attachment; filename="tt.log"; size=18782;
	creation-date="Mon, 16 May 2011 18:59:24 GMT";
	modification-date="Mon, 16 May 2011 18:59:24 GMT"
Content-Transfer-Encoding: base64

CWF0IG9yZy5tb3J0YmF5LmlvLm5pby5TZWxlY3RDaGFubmVsRW5kUG9pbnQuZmx1c2goU2VsZWN0
Q2hhbm5lbEVuZFBvaW50LmphdmE6MjIxKQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuSHR0cEdlbmVy
YXRvci5mbHVzaChIdHRwR2VuZXJhdG9yLmphdmE6NzI1KQoJLi4uIDI3IG1vcmUKCjIwMTEtMDUt
MTUgMTU6NTA6NDMsNjcwIElORk8gb3JnLmFwYWNoZS5oYWRvb3AubWFwcmVkLlRhc2tUcmFja2Vy
LmNsaWVudHRyYWNlOiBzcmM6IDE5Mi4xNjguMjguMjE0OjUwMDYwLCBkZXN0OiAxOTIuMTY4LjI4
LjIxNDo1MjMxMywgYnl0ZXM6IDY1NTM2LCBvcDogTUFQUkVEX1NIVUZGTEUsIGNsaUlEOiBhdHRl
bXB0XzIwMTEwNTE0MjEzN18wMDAxX21fMDAwNjk0XzAsIGR1cmF0aW9uOiAxOTAyNDEwMDAKMjAx
MS0wNS0xNSAxNTo1MDo0Myw2NzAgRVJST1Igb3JnLm1vcnRiYXkubG9nOiAvbWFwT3V0cHV0Cmph
dmEubGFuZy5JbGxlZ2FsU3RhdGVFeGNlcHRpb246IENvbW1pdHRlZAoJYXQgb3JnLm1vcnRiYXku
amV0dHkuUmVzcG9uc2UucmVzZXRCdWZmZXIoUmVzcG9uc2UuamF2YToxMDIzKQoJYXQgb3JnLm1v
cnRiYXkuamV0dHkuUmVzcG9uc2Uuc2VuZEVycm9yKFJlc3BvbnNlLmphdmE6MjQwKQoJYXQgb3Jn
LmFwYWNoZS5oYWRvb3AubWFwcmVkLlRhc2tUcmFja2VyJE1hcE91dHB1dFNlcnZsZXQuZG9HZXQo
VGFza1RyYWNrZXIuamF2YTozNzE4KQoJYXQgamF2YXguc2VydmxldC5odHRwLkh0dHBTZXJ2bGV0
LnNlcnZpY2UoSHR0cFNlcnZsZXQuamF2YTo3MDcpCglhdCBqYXZheC5zZXJ2bGV0Lmh0dHAuSHR0
cFNlcnZsZXQuc2VydmljZShIdHRwU2VydmxldC5qYXZhOjgyMCkKCWF0IG9yZy5tb3J0YmF5Lmpl
dHR5LnNlcnZsZXQuU2VydmxldEhvbGRlci5oYW5kbGUoU2VydmxldEhvbGRlci5qYXZhOjUxMSkK
CWF0IG9yZy5tb3J0YmF5LmpldHR5LnNlcnZsZXQuU2VydmxldEhhbmRsZXIkQ2FjaGVkQ2hhaW4u
ZG9GaWx0ZXIoU2VydmxldEhhbmRsZXIuamF2YToxMjIxKQoJYXQgb3JnLmFwYWNoZS5oYWRvb3Au
aHR0cC5IdHRwU2VydmVyJFF1b3RpbmdJbnB1dEZpbHRlci5kb0ZpbHRlcihIdHRwU2VydmVyLmph
dmE6ODI0KQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuc2VydmxldC5TZXJ2bGV0SGFuZGxlciRDYWNo
ZWRDaGFpbi5kb0ZpbHRlcihTZXJ2bGV0SGFuZGxlci5qYXZhOjEyMTIpCglhdCBvcmcubW9ydGJh
eS5qZXR0eS5zZXJ2bGV0LlNlcnZsZXRIYW5kbGVyLmhhbmRsZShTZXJ2bGV0SGFuZGxlci5qYXZh
OjM5OSkKCWF0IG9yZy5tb3J0YmF5LmpldHR5LnNlY3VyaXR5LlNlY3VyaXR5SGFuZGxlci5oYW5k
bGUoU2VjdXJpdHlIYW5kbGVyLmphdmE6MjE2KQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuc2Vydmxl
dC5TZXNzaW9uSGFuZGxlci5oYW5kbGUoU2Vzc2lvbkhhbmRsZXIuamF2YToxODIpCglhdCBvcmcu
bW9ydGJheS5qZXR0eS5oYW5kbGVyLkNvbnRleHRIYW5kbGVyLmhhbmRsZShDb250ZXh0SGFuZGxl
ci5qYXZhOjc2NikKCWF0IG9yZy5tb3J0YmF5LmpldHR5LndlYmFwcC5XZWJBcHBDb250ZXh0Lmhh
bmRsZShXZWJBcHBDb250ZXh0LmphdmE6NDUwKQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuaGFuZGxl
ci5Db250ZXh0SGFuZGxlckNvbGxlY3Rpb24uaGFuZGxlKENvbnRleHRIYW5kbGVyQ29sbGVjdGlv
bi5qYXZhOjIzMCkKCWF0IG9yZy5tb3J0YmF5LmpldHR5LmhhbmRsZXIuSGFuZGxlcldyYXBwZXIu
aGFuZGxlKEhhbmRsZXJXcmFwcGVyLmphdmE6MTUyKQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuU2Vy
dmVyLmhhbmRsZShTZXJ2ZXIuamF2YTozMjYpCglhdCBvcmcubW9ydGJheS5qZXR0eS5IdHRwQ29u
bmVjdGlvbi5oYW5kbGVSZXF1ZXN0KEh0dHBDb25uZWN0aW9uLmphdmE6NTQyKQoJYXQgb3JnLm1v
cnRiYXkuamV0dHkuSHR0cENvbm5lY3Rpb24kUmVxdWVzdEhhbmRsZXIuaGVhZGVyQ29tcGxldGUo
SHR0cENvbm5lY3Rpb24uamF2YTo5MjgpCglhdCBvcmcubW9ydGJheS5qZXR0eS5IdHRwUGFyc2Vy
LnBhcnNlTmV4dChIdHRwUGFyc2VyLmphdmE6NTQ5KQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuSHR0
cFBhcnNlci5wYXJzZUF2YWlsYWJsZShIdHRwUGFyc2VyLmphdmE6MjEyKQoJYXQgb3JnLm1vcnRi
YXkuamV0dHkuSHR0cENvbm5lY3Rpb24uaGFuZGxlKEh0dHBDb25uZWN0aW9uLmphdmE6NDA0KQoJ
YXQgb3JnLm1vcnRiYXkuaW8ubmlvLlNlbGVjdENoYW5uZWxFbmRQb2ludC5ydW4oU2VsZWN0Q2hh
bm5lbEVuZFBvaW50LmphdmE6NDEwKQoJYXQgb3JnLm1vcnRiYXkudGhyZWFkLlF1ZXVlZFRocmVh
ZFBvb2wkUG9vbFRocmVhZC5ydW4oUXVldWVkVGhyZWFkUG9vbC5qYXZhOjU4MikKMjAxMS0wNS0x
NSAxNTo1MDo0Myw3MzEgSU5GTyBvcmcuYXBhY2hlLmhhZG9vcC5tYXByZWQuVGFza1RyYWNrZXI6
IGF0dGVtcHRfMjAxMTA1MTQyMTM3XzAwMDFfcl8wMDAwMTBfMCAwLjA4MjE5MzQzJSByZWR1Y2Ug
PiBjb3B5ICg2ODUgb2YgMjc3OCBhdCAzLjI0IE1CL3MpID4gCjIwMTEtMDUtMTUgMTU6NTA6NDQs
MTE1IElORk8gb3JnLmFwYWNoZS5oYWRvb3AubWFwcmVkLlRhc2tUcmFja2VyOiBhdHRlbXB0XzIw
MTEwNTE0MjEzN18wMDAxX21fMDAyMjY4XzAgMC45Nzg0NTQlIAoyMDExLTA1LTE1IDE1OjUwOjQ0
LDEzNCBJTkZPIG9yZy5hcGFjaGUuaGFkb29wLm1hcHJlZC5UYXNrVHJhY2tlci5jbGllbnR0cmFj
ZTogc3JjOiAxOTIuMTY4LjI4LjIxNDo1MDA2MCwgZGVzdDogMTkyLjE2OC4yOC4yMTM6MzM0Nzcs
IGJ5dGVzOiA2ODY5NjI4LCBvcDogTUFQUkVEX1NIVUZGTEUsIGNsaUlEOiBhdHRlbXB0XzIwMTEw
NTE0MjEzN18wMDAxX21fMDAwNjQ3XzAsIGR1cmF0aW9uOiA0MzI4MDQ4MDAwCjIwMTEtMDUtMTUg
MTU6NTA6NDQsMjM4IElORk8gb3JnLmFwYWNoZS5oYWRvb3AubWFwcmVkLlRhc2tUcmFja2VyLmNs
aWVudHRyYWNlOiBzcmM6IDE5Mi4xNjguMjguMjE0OjUwMDYwLCBkZXN0OiAxOTIuMTY4LjI4LjIx
MzozMzQ3NywgYnl0ZXM6IDE4LCBvcDogTUFQUkVEX1NIVUZGTEUsIGNsaUlEOiBhdHRlbXB0XzIw
MTEwNTE0MjEzN18wMDAxX21fMDAwNjcwXzAsIGR1cmF0aW9uOiAzMTQwMDAKMjAxMS0wNS0xNSAx
NTo1MDo0NCwyOTkgSU5GTyBvcmcuYXBhY2hlLmhhZG9vcC5tYXByZWQuVGFza1RyYWNrZXIuY2xp
ZW50dHJhY2U6IHNyYzogMTkyLjE2OC4yOC4yMTQ6NTAwNjAsIGRlc3Q6IDE5Mi4xNjguMjguMjEz
OjMzNDcyLCBieXRlczogNzA0MjkxNiwgb3A6IE1BUFJFRF9TSFVGRkxFLCBjbGlJRDogYXR0ZW1w
dF8yMDExMDUxNDIxMzdfMDAwMV9tXzAwMDY3MV8wLCBkdXJhdGlvbjogMTA4ODM1NjUwMDAKMjAx
MS0wNS0xNSAxNTo1MDo0NCwzODIgSU5GTyBvcmcuYXBhY2hlLmhhZG9vcC5tYXByZWQuVGFza1Ry
YWNrZXI6IGF0dGVtcHRfMjAxMTA1MTQyMTM3XzAwMDFfbV8wMDIyNjdfMCAwLjg5OTY4NDYlIAoy
MDExLTA1LTE1IDE1OjUwOjQ0LDU3NCBXQVJOIG9yZy5hcGFjaGUuaGFkb29wLm1hcHJlZC5UYXNr
VHJhY2tlcjogZ2V0TWFwT3V0cHV0KGF0dGVtcHRfMjAxMTA1MTQyMTM3XzAwMDFfbV8wMDA2NDlf
MCwxKSBmYWlsZWQgOgpvcmcubW9ydGJheS5qZXR0eS5Fb2ZFeGNlcHRpb24KCWF0IG9yZy5tb3J0
YmF5LmpldHR5Lkh0dHBHZW5lcmF0b3IuZmx1c2goSHR0cEdlbmVyYXRvci5qYXZhOjc5MSkKCWF0
IG9yZy5tb3J0YmF5LmpldHR5LkFic3RyYWN0R2VuZXJhdG9yJE91dHB1dC5mbHVzaChBYnN0cmFj
dEdlbmVyYXRvci5qYXZhOjU2OSkKCWF0IG9yZy5tb3J0YmF5LmpldHR5Lkh0dHBDb25uZWN0aW9u
JE91dHB1dC5mbHVzaChIdHRwQ29ubmVjdGlvbi5qYXZhOjEwMTIpCglhdCBvcmcubW9ydGJheS5q
ZXR0eS5BYnN0cmFjdEdlbmVyYXRvciRPdXRwdXQud3JpdGUoQWJzdHJhY3RHZW5lcmF0b3IuamF2
YTo2NTEpCglhdCBvcmcubW9ydGJheS5qZXR0eS5BYnN0cmFjdEdlbmVyYXRvciRPdXRwdXQud3Jp
dGUoQWJzdHJhY3RHZW5lcmF0b3IuamF2YTo1ODApCglhdCBvcmcuYXBhY2hlLmhhZG9vcC5tYXBy
ZWQuVGFza1RyYWNrZXIkTWFwT3V0cHV0U2VydmxldC5kb0dldChUYXNrVHJhY2tlci5qYXZhOjM2
OTMpCglhdCBqYXZheC5zZXJ2bGV0Lmh0dHAuSHR0cFNlcnZsZXQuc2VydmljZShIdHRwU2Vydmxl
dC5qYXZhOjcwNykKCWF0IGphdmF4LnNlcnZsZXQuaHR0cC5IdHRwU2VydmxldC5zZXJ2aWNlKEh0
dHBTZXJ2bGV0LmphdmE6ODIwKQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuc2VydmxldC5TZXJ2bGV0
SG9sZGVyLmhhbmRsZShTZXJ2bGV0SG9sZGVyLmphdmE6NTExKQoJYXQgb3JnLm1vcnRiYXkuamV0
dHkuc2VydmxldC5TZXJ2bGV0SGFuZGxlciRDYWNoZWRDaGFpbi5kb0ZpbHRlcihTZXJ2bGV0SGFu
ZGxlci5qYXZhOjEyMjEpCglhdCBvcmcuYXBhY2hlLmhhZG9vcC5odHRwLkh0dHBTZXJ2ZXIkUXVv
dGluZ0lucHV0RmlsdGVyLmRvRmlsdGVyKEh0dHBTZXJ2ZXIuamF2YTo4MjQpCglhdCBvcmcubW9y
dGJheS5qZXR0eS5zZXJ2bGV0LlNlcnZsZXRIYW5kbGVyJENhY2hlZENoYWluLmRvRmlsdGVyKFNl
cnZsZXRIYW5kbGVyLmphdmE6MTIxMikKCWF0IG9yZy5tb3J0YmF5LmpldHR5LnNlcnZsZXQuU2Vy
dmxldEhhbmRsZXIuaGFuZGxlKFNlcnZsZXRIYW5kbGVyLmphdmE6Mzk5KQoJYXQgb3JnLm1vcnRi
YXkuamV0dHkuc2VjdXJpdHkuU2VjdXJpdHlIYW5kbGVyLmhhbmRsZShTZWN1cml0eUhhbmRsZXIu
amF2YToyMTYpCglhdCBvcmcubW9ydGJheS5qZXR0eS5zZXJ2bGV0LlNlc3Npb25IYW5kbGVyLmhh
bmRsZShTZXNzaW9uSGFuZGxlci5qYXZhOjE4MikKCWF0IG9yZy5tb3J0YmF5LmpldHR5LmhhbmRs
ZXIuQ29udGV4dEhhbmRsZXIuaGFuZGxlKENvbnRleHRIYW5kbGVyLmphdmE6NzY2KQoJYXQgb3Jn
Lm1vcnRiYXkuamV0dHkud2ViYXBwLldlYkFwcENvbnRleHQuaGFuZGxlKFdlYkFwcENvbnRleHQu
amF2YTo0NTApCglhdCBvcmcubW9ydGJheS5qZXR0eS5oYW5kbGVyLkNvbnRleHRIYW5kbGVyQ29s
bGVjdGlvbi5oYW5kbGUoQ29udGV4dEhhbmRsZXJDb2xsZWN0aW9uLmphdmE6MjMwKQoJYXQgb3Jn
Lm1vcnRiYXkuamV0dHkuaGFuZGxlci5IYW5kbGVyV3JhcHBlci5oYW5kbGUoSGFuZGxlcldyYXBw
ZXIuamF2YToxNTIpCglhdCBvcmcubW9ydGJheS5qZXR0eS5TZXJ2ZXIuaGFuZGxlKFNlcnZlci5q
YXZhOjMyNikKCWF0IG9yZy5tb3J0YmF5LmpldHR5Lkh0dHBDb25uZWN0aW9uLmhhbmRsZVJlcXVl
c3QoSHR0cENvbm5lY3Rpb24uamF2YTo1NDIpCglhdCBvcmcubW9ydGJheS5qZXR0eS5IdHRwQ29u
bmVjdGlvbiRSZXF1ZXN0SGFuZGxlci5oZWFkZXJDb21wbGV0ZShIdHRwQ29ubmVjdGlvbi5qYXZh
OjkyOCkKCWF0IG9yZy5tb3J0YmF5LmpldHR5Lkh0dHBQYXJzZXIucGFyc2VOZXh0KEh0dHBQYXJz
ZXIuamF2YTo1NDkpCglhdCBvcmcubW9ydGJheS5qZXR0eS5IdHRwUGFyc2VyLnBhcnNlQXZhaWxh
YmxlKEh0dHBQYXJzZXIuamF2YToyMTIpCglhdCBvcmcubW9ydGJheS5qZXR0eS5IdHRwQ29ubmVj
dGlvbi5oYW5kbGUoSHR0cENvbm5lY3Rpb24uamF2YTo0MDQpCglhdCBvcmcubW9ydGJheS5pby5u
aW8uU2VsZWN0Q2hhbm5lbEVuZFBvaW50LnJ1bihTZWxlY3RDaGFubmVsRW5kUG9pbnQuamF2YTo0
MTApCglhdCBvcmcubW9ydGJheS50aHJlYWQuUXVldWVkVGhyZWFkUG9vbCRQb29sVGhyZWFkLnJ1
bihRdWV1ZWRUaHJlYWRQb29sLmphdmE6NTgyKQpDYXVzZWQgYnk6IGphdmEuaW8uSU9FeGNlcHRp
b246IEJyb2tlbiBwaXBlCglhdCBzdW4ubmlvLmNoLkZpbGVEaXNwYXRjaGVyLndyaXRlMChOYXRp
dmUgTWV0aG9kKQoJYXQgc3VuLm5pby5jaC5Tb2NrZXREaXNwYXRjaGVyLndyaXRlKFNvY2tldERp
c3BhdGNoZXIuamF2YToyOSkKCWF0IHN1bi5uaW8uY2guSU9VdGlsLndyaXRlRnJvbU5hdGl2ZUJ1
ZmZlcihJT1V0aWwuamF2YTo3MikKCWF0IHN1bi5uaW8uY2guSU9VdGlsLndyaXRlKElPVXRpbC5q
YXZhOjQzKQoJYXQgc3VuLm5pby5jaC5Tb2NrZXRDaGFubmVsSW1wbC53cml0ZShTb2NrZXRDaGFu
bmVsSW1wbC5qYXZhOjMzNCkKCWF0IG9yZy5tb3J0YmF5LmlvLm5pby5DaGFubmVsRW5kUG9pbnQu
Zmx1c2goQ2hhbm5lbEVuZFBvaW50LmphdmE6MTcwKQoJYXQgb3JnLm1vcnRiYXkuaW8ubmlvLlNl
bGVjdENoYW5uZWxFbmRQb2ludC5mbHVzaChTZWxlY3RDaGFubmVsRW5kUG9pbnQuamF2YToyMjEp
CglhdCBvcmcubW9ydGJheS5qZXR0eS5IdHRwR2VuZXJhdG9yLmZsdXNoKEh0dHBHZW5lcmF0b3Iu
amF2YTo3MjUpCgkuLi4gMjYgbW9yZQoKMjAxMS0wNS0xNSAxNTo1MDo0NCw1NzQgV0FSTiBvcmcu
bW9ydGJheS5sb2c6IENvbW1pdHRlZCBiZWZvcmUgNDEwIGdldE1hcE91dHB1dChhdHRlbXB0XzIw
MTEwNTE0MjEzN18wMDAxX21fMDAwNjQ5XzAsMSkgZmFpbGVkIDoKb3JnLm1vcnRiYXkuamV0dHku
RW9mRXhjZXB0aW9uCglhdCBvcmcubW9ydGJheS5qZXR0eS5IdHRwR2VuZXJhdG9yLmZsdXNoKEh0
dHBHZW5lcmF0b3IuamF2YTo3OTEpCglhdCBvcmcubW9ydGJheS5qZXR0eS5BYnN0cmFjdEdlbmVy
YXRvciRPdXRwdXQuZmx1c2goQWJzdHJhY3RHZW5lcmF0b3IuamF2YTo1NjkpCglhdCBvcmcubW9y
dGJheS5qZXR0eS5IdHRwQ29ubmVjdGlvbiRPdXRwdXQuZmx1c2goSHR0cENvbm5lY3Rpb24uamF2
YToxMDEyKQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuQWJzdHJhY3RHZW5lcmF0b3IkT3V0cHV0Lndy
aXRlKEFic3RyYWN0R2VuZXJhdG9yLmphdmE6NjUxKQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuQWJz
dHJhY3RHZW5lcmF0b3IkT3V0cHV0LndyaXRlKEFic3RyYWN0R2VuZXJhdG9yLmphdmE6NTgwKQoJ
YXQgb3JnLmFwYWNoZS5oYWRvb3AubWFwcmVkLlRhc2tUcmFja2VyJE1hcE91dHB1dFNlcnZsZXQu
ZG9HZXQoVGFza1RyYWNrZXIuamF2YTozNjkzKQoJYXQgamF2YXguc2VydmxldC5odHRwLkh0dHBT
ZXJ2bGV0LnNlcnZpY2UoSHR0cFNlcnZsZXQuamF2YTo3MDcpCglhdCBqYXZheC5zZXJ2bGV0Lmh0
dHAuSHR0cFNlcnZsZXQuc2VydmljZShIdHRwU2VydmxldC5qYXZhOjgyMCkKCWF0IG9yZy5tb3J0
YmF5LmpldHR5LnNlcnZsZXQuU2VydmxldEhvbGRlci5oYW5kbGUoU2VydmxldEhvbGRlci5qYXZh
OjUxMSkKCWF0IG9yZy5tb3J0YmF5LmpldHR5LnNlcnZsZXQuU2VydmxldEhhbmRsZXIkQ2FjaGVk
Q2hhaW4uZG9GaWx0ZXIoU2VydmxldEhhbmRsZXIuamF2YToxMjIxKQoJYXQgb3JnLmFwYWNoZS5o
YWRvb3AuaHR0cC5IdHRwU2VydmVyJFF1b3RpbmdJbnB1dEZpbHRlci5kb0ZpbHRlcihIdHRwU2Vy
dmVyLmphdmE6ODI0KQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuc2VydmxldC5TZXJ2bGV0SGFuZGxl
ciRDYWNoZWRDaGFpbi5kb0ZpbHRlcihTZXJ2bGV0SGFuZGxlci5qYXZhOjEyMTIpCglhdCBvcmcu
bW9ydGJheS5qZXR0eS5zZXJ2bGV0LlNlcnZsZXRIYW5kbGVyLmhhbmRsZShTZXJ2bGV0SGFuZGxl
ci5qYXZhOjM5OSkKCWF0IG9yZy5tb3J0YmF5LmpldHR5LnNlY3VyaXR5LlNlY3VyaXR5SGFuZGxl
ci5oYW5kbGUoU2VjdXJpdHlIYW5kbGVyLmphdmE6MjE2KQoJYXQgb3JnLm1vcnRiYXkuamV0dHku
c2VydmxldC5TZXNzaW9uSGFuZGxlci5oYW5kbGUoU2Vzc2lvbkhhbmRsZXIuamF2YToxODIpCglh
dCBvcmcubW9ydGJheS5qZXR0eS5oYW5kbGVyLkNvbnRleHRIYW5kbGVyLmhhbmRsZShDb250ZXh0
SGFuZGxlci5qYXZhOjc2NikKCWF0IG9yZy5tb3J0YmF5LmpldHR5LndlYmFwcC5XZWJBcHBDb250
ZXh0LmhhbmRsZShXZWJBcHBDb250ZXh0LmphdmE6NDUwKQoJYXQgb3JnLm1vcnRiYXkuamV0dHku
aGFuZGxlci5Db250ZXh0SGFuZGxlckNvbGxlY3Rpb24uaGFuZGxlKENvbnRleHRIYW5kbGVyQ29s
bGVjdGlvbi5qYXZhOjIzMCkKCWF0IG9yZy5tb3J0YmF5LmpldHR5LmhhbmRsZXIuSGFuZGxlcldy
YXBwZXIuaGFuZGxlKEhhbmRsZXJXcmFwcGVyLmphdmE6MTUyKQoJYXQgb3JnLm1vcnRiYXkuamV0
dHkuU2VydmVyLmhhbmRsZShTZXJ2ZXIuamF2YTozMjYpCglhdCBvcmcubW9ydGJheS5qZXR0eS5I
dHRwQ29ubmVjdGlvbi5oYW5kbGVSZXF1ZXN0KEh0dHBDb25uZWN0aW9uLmphdmE6NTQyKQoJYXQg
b3JnLm1vcnRiYXkuamV0dHkuSHR0cENvbm5lY3Rpb24kUmVxdWVzdEhhbmRsZXIuaGVhZGVyQ29t
cGxldGUoSHR0cENvbm5lY3Rpb24uamF2YTo5MjgpCglhdCBvcmcubW9ydGJheS5qZXR0eS5IdHRw
UGFyc2VyLnBhcnNlTmV4dChIdHRwUGFyc2VyLmphdmE6NTQ5KQoJYXQgb3JnLm1vcnRiYXkuamV0
dHkuSHR0cFBhcnNlci5wYXJzZUF2YWlsYWJsZShIdHRwUGFyc2VyLmphdmE6MjEyKQoJYXQgb3Jn
Lm1vcnRiYXkuamV0dHkuSHR0cENvbm5lY3Rpb24uaGFuZGxlKEh0dHBDb25uZWN0aW9uLmphdmE6
NDA0KQoJYXQgb3JnLm1vcnRiYXkuaW8ubmlvLlNlbGVjdENoYW5uZWxFbmRQb2ludC5ydW4oU2Vs
ZWN0Q2hhbm5lbEVuZFBvaW50LmphdmE6NDEwKQoJYXQgb3JnLm1vcnRiYXkudGhyZWFkLlF1ZXVl
ZFRocmVhZFBvb2wkUG9vbFRocmVhZC5ydW4oUXVldWVkVGhyZWFkUG9vbC5qYXZhOjU4MikKQ2F1
c2VkIGJ5OiBqYXZhLmlvLklPRXhjZXB0aW9uOiBCcm9rZW4gcGlwZQoJYXQgc3VuLm5pby5jaC5G
aWxlRGlzcGF0Y2hlci53cml0ZTAoTmF0aXZlIE1ldGhvZCkKCWF0IHN1bi5uaW8uY2guU29ja2V0
RGlzcGF0Y2hlci53cml0ZShTb2NrZXREaXNwYXRjaGVyLmphdmE6MjkpCglhdCBzdW4ubmlvLmNo
LklPVXRpbC53cml0ZUZyb21OYXRpdmVCdWZmZXIoSU9VdGlsLmphdmE6NzIpCglhdCBzdW4ubmlv
LmNoLklPVXRpbC53cml0ZShJT1V0aWwuamF2YTo0MykKCWF0IHN1bi5uaW8uY2guU29ja2V0Q2hh
bm5lbEltcGwud3JpdGUoU29ja2V0Q2hhbm5lbEltcGwuamF2YTozMzQpCglhdCBvcmcubW9ydGJh
eS5pby5uaW8uQ2hhbm5lbEVuZFBvaW50LmZsdXNoKENoYW5uZWxFbmRQb2ludC5qYXZhOjE3MCkK
CWF0IG9yZy5tb3J0YmF5LmlvLm5pby5TZWxlY3RDaGFubmVsRW5kUG9pbnQuZmx1c2goU2VsZWN0
Q2hhbm5lbEVuZFBvaW50LmphdmE6MjIxKQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuSHR0cEdlbmVy
YXRvci5mbHVzaChIdHRwR2VuZXJhdG9yLmphdmE6NzI1KQoJLi4uIDI2IG1vcmUKCjIwMTEtMDUt
MTUgMTU6NTA6NDQsNTc0IElORk8gb3JnLmFwYWNoZS5oYWRvb3AubWFwcmVkLlRhc2tUcmFja2Vy
LmNsaWVudHRyYWNlOiBzcmM6IDE5Mi4xNjguMjguMjE0OjUwMDYwLCBkZXN0OiAxOTIuMTY4LjI4
LjIxMzozMzQ3NywgYnl0ZXM6IDY1NTM2LCBvcDogTUFQUkVEX1NIVUZGTEUsIGNsaUlEOiBhdHRl
bXB0XzIwMTEwNTE0MjEzN18wMDAxX21fMDAwNjQ5XzAsIGR1cmF0aW9uOiAyNDk5NTQwMDAKMjAx
MS0wNS0xNSAxNTo1MDo0NCw1NzQgRVJST1Igb3JnLm1vcnRiYXkubG9nOiAvbWFwT3V0cHV0Cmph
dmEubGFuZy5JbGxlZ2FsU3RhdGVFeGNlcHRpb246IENvbW1pdHRlZAoJYXQgb3JnLm1vcnRiYXku
amV0dHkuUmVzcG9uc2UucmVzZXRCdWZmZXIoUmVzcG9uc2UuamF2YToxMDIzKQoJYXQgb3JnLm1v
cnRiYXkuamV0dHkuUmVzcG9uc2Uuc2VuZEVycm9yKFJlc3BvbnNlLmphdmE6MjQwKQoJYXQgb3Jn
LmFwYWNoZS5oYWRvb3AubWFwcmVkLlRhc2tUcmFja2VyJE1hcE91dHB1dFNlcnZsZXQuZG9HZXQo
VGFza1RyYWNrZXIuamF2YTozNzE4KQoJYXQgamF2YXguc2VydmxldC5odHRwLkh0dHBTZXJ2bGV0
LnNlcnZpY2UoSHR0cFNlcnZsZXQuamF2YTo3MDcpCglhdCBqYXZheC5zZXJ2bGV0Lmh0dHAuSHR0
cFNlcnZsZXQuc2VydmljZShIdHRwU2VydmxldC5qYXZhOjgyMCkKCWF0IG9yZy5tb3J0YmF5Lmpl
dHR5LnNlcnZsZXQuU2VydmxldEhvbGRlci5oYW5kbGUoU2VydmxldEhvbGRlci5qYXZhOjUxMSkK
CWF0IG9yZy5tb3J0YmF5LmpldHR5LnNlcnZsZXQuU2VydmxldEhhbmRsZXIkQ2FjaGVkQ2hhaW4u
ZG9GaWx0ZXIoU2VydmxldEhhbmRsZXIuamF2YToxMjIxKQoJYXQgb3JnLmFwYWNoZS5oYWRvb3Au
aHR0cC5IdHRwU2VydmVyJFF1b3RpbmdJbnB1dEZpbHRlci5kb0ZpbHRlcihIdHRwU2VydmVyLmph
dmE6ODI0KQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuc2VydmxldC5TZXJ2bGV0SGFuZGxlciRDYWNo
ZWRDaGFpbi5kb0ZpbHRlcihTZXJ2bGV0SGFuZGxlci5qYXZhOjEyMTIpCglhdCBvcmcubW9ydGJh
eS5qZXR0eS5zZXJ2bGV0LlNlcnZsZXRIYW5kbGVyLmhhbmRsZShTZXJ2bGV0SGFuZGxlci5qYXZh
OjM5OSkKCWF0IG9yZy5tb3J0YmF5LmpldHR5LnNlY3VyaXR5LlNlY3VyaXR5SGFuZGxlci5oYW5k
bGUoU2VjdXJpdHlIYW5kbGVyLmphdmE6MjE2KQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuc2Vydmxl
dC5TZXNzaW9uSGFuZGxlci5oYW5kbGUoU2Vzc2lvbkhhbmRsZXIuamF2YToxODIpCglhdCBvcmcu
bW9ydGJheS5qZXR0eS5oYW5kbGVyLkNvbnRleHRIYW5kbGVyLmhhbmRsZShDb250ZXh0SGFuZGxl
ci5qYXZhOjc2NikKCWF0IG9yZy5tb3J0YmF5LmpldHR5LndlYmFwcC5XZWJBcHBDb250ZXh0Lmhh
bmRsZShXZWJBcHBDb250ZXh0LmphdmE6NDUwKQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuaGFuZGxl
ci5Db250ZXh0SGFuZGxlckNvbGxlY3Rpb24uaGFuZGxlKENvbnRleHRIYW5kbGVyQ29sbGVjdGlv
bi5qYXZhOjIzMCkKCWF0IG9yZy5tb3J0YmF5LmpldHR5LmhhbmRsZXIuSGFuZGxlcldyYXBwZXIu
aGFuZGxlKEhhbmRsZXJXcmFwcGVyLmphdmE6MTUyKQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuU2Vy
dmVyLmhhbmRsZShTZXJ2ZXIuamF2YTozMjYpCglhdCBvcmcubW9ydGJheS5qZXR0eS5IdHRwQ29u
bmVjdGlvbi5oYW5kbGVSZXF1ZXN0KEh0dHBDb25uZWN0aW9uLmphdmE6NTQyKQoJYXQgb3JnLm1v
cnRiYXkuamV0dHkuSHR0cENvbm5lY3Rpb24kUmVxdWVzdEhhbmRsZXIuaGVhZGVyQ29tcGxldGUo
SHR0cENvbm5lY3Rpb24uamF2YTo5MjgpCglhdCBvcmcubW9ydGJheS5qZXR0eS5IdHRwUGFyc2Vy
LnBhcnNlTmV4dChIdHRwUGFyc2VyLmphdmE6NTQ5KQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuSHR0
cFBhcnNlci5wYXJzZUF2YWlsYWJsZShIdHRwUGFyc2VyLmphdmE6MjEyKQoJYXQgb3JnLm1vcnRi
YXkuamV0dHkuSHR0cENvbm5lY3Rpb24uaGFuZGxlKEh0dHBDb25uZWN0aW9uLmphdmE6NDA0KQoJ
YXQgb3JnLm1vcnRiYXkuaW8ubmlvLlNlbGVjdENoYW5uZWxFbmRQb2ludC5ydW4oU2VsZWN0Q2hh
bm5lbEVuZFBvaW50LmphdmE6NDEwKQoJYXQgb3JnLm1vcnRiYXkudGhyZWFkLlF1ZXVlZFRocmVh
ZFBvb2wkUG9vbFRocmVhZC5ydW4oUXVldWVkVGhyZWFkUG9vbC5qYXZhOjU4MikKMjAxMS0wNS0x
NSAxNTo1MDo0NCw2NDggSU5GTyBvcmcuYXBhY2hlLmhhZG9vcC5tYXByZWQuVGFza1RyYWNrZXI6
IGF0dGVtcHRfMjAxMTA1MTQyMTM3XzAwMDFfbV8wMDIyNTVfMCAxLjAlIAoyMDExLTA1LTE1IDE1
OjUwOjQ0LDk1MyBJTkZPIG9yZy5hcGFjaGUuaGFkb29wLm1hcHJlZC5UYXNrVHJhY2tlci5jbGll
bnR0cmFjZTogc3JjOiAxOTIuMTY4LjI4LjIxNDo1MDA2MCwgZGVzdDogMTkyLjE2OC4yOC4yMTM6
MzM0OTAsIGJ5dGVzOiA2NjQ2MDg3LCBvcDogTUFQUkVEX1NIVUZGTEUsIGNsaUlEOiBhdHRlbXB0
XzIwMTEwNTE0MjEzN18wMDAxX21fMDAwMzgyXzAsIGR1cmF0aW9uOiAzNzQ1MDEwMDAKMjAxMS0w
NS0xNSAxNTo1MDo0NCw5OTUgV0FSTiBvcmcuYXBhY2hlLmhhZG9vcC5tYXByZWQuVGFza1RyYWNr
ZXI6IGdldE1hcE91dHB1dChhdHRlbXB0XzIwMTEwNTE0MjEzN18wMDAxX21fMDAwMzk0XzAsMTQp
IGZhaWxlZCA6Cm9yZy5tb3J0YmF5LmpldHR5LkVvZkV4Y2VwdGlvbgoJYXQgb3JnLm1vcnRiYXku
amV0dHkuSHR0cEdlbmVyYXRvci5mbHVzaChIdHRwR2VuZXJhdG9yLmphdmE6NzkxKQoJYXQgb3Jn
Lm1vcnRiYXkuamV0dHkuQWJzdHJhY3RHZW5lcmF0b3IkT3V0cHV0LmZsdXNoKEFic3RyYWN0R2Vu
ZXJhdG9yLmphdmE6NTY5KQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuSHR0cENvbm5lY3Rpb24kT3V0
cHV0LmZsdXNoKEh0dHBDb25uZWN0aW9uLmphdmE6MTAxMikKCWF0IG9yZy5tb3J0YmF5LmpldHR5
LkFic3RyYWN0R2VuZXJhdG9yJE91dHB1dC53cml0ZShBYnN0cmFjdEdlbmVyYXRvci5qYXZhOjY1
MSkKCWF0IG9yZy5tb3J0YmF5LmpldHR5LkFic3RyYWN0R2VuZXJhdG9yJE91dHB1dC53cml0ZShB
YnN0cmFjdEdlbmVyYXRvci5qYXZhOjU4MCkKCWF0IG9yZy5hcGFjaGUuaGFkb29wLm1hcHJlZC5U
YXNrVHJhY2tlciRNYXBPdXRwdXRTZXJ2bGV0LmRvR2V0KFRhc2tUcmFja2VyLmphdmE6MzY5MykK
CWF0IGphdmF4LnNlcnZsZXQuaHR0cC5IdHRwU2VydmxldC5zZXJ2aWNlKEh0dHBTZXJ2bGV0Lmph
dmE6NzA3KQoJYXQgamF2YXguc2VydmxldC5odHRwLkh0dHBTZXJ2bGV0LnNlcnZpY2UoSHR0cFNl
cnZsZXQuamF2YTo4MjApCglhdCBvcmcubW9ydGJheS5qZXR0eS5zZXJ2bGV0LlNlcnZsZXRIb2xk
ZXIuaGFuZGxlKFNlcnZsZXRIb2xkZXIuamF2YTo1MTEpCglhdCBvcmcubW9ydGJheS5qZXR0eS5z
ZXJ2bGV0LlNlcnZsZXRIYW5kbGVyJENhY2hlZENoYWluLmRvRmlsdGVyKFNlcnZsZXRIYW5kbGVy
LmphdmE6MTIyMSkKCWF0IG9yZy5hcGFjaGUuaGFkb29wLmh0dHAuSHR0cFNlcnZlciRRdW90aW5n
SW5wdXRGaWx0ZXIuZG9GaWx0ZXIoSHR0cFNlcnZlci5qYXZhOjgyNCkKCWF0IG9yZy5tb3J0YmF5
LmpldHR5LnNlcnZsZXQuU2VydmxldEhhbmRsZXIkQ2FjaGVkQ2hhaW4uZG9GaWx0ZXIoU2Vydmxl
dEhhbmRsZXIuamF2YToxMjEyKQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuc2VydmxldC5TZXJ2bGV0
SGFuZGxlci5oYW5kbGUoU2VydmxldEhhbmRsZXIuamF2YTozOTkpCglhdCBvcmcubW9ydGJheS5q
ZXR0eS5zZWN1cml0eS5TZWN1cml0eUhhbmRsZXIuaGFuZGxlKFNlY3VyaXR5SGFuZGxlci5qYXZh
OjIxNikKCWF0IG9yZy5tb3J0YmF5LmpldHR5LnNlcnZsZXQuU2Vzc2lvbkhhbmRsZXIuaGFuZGxl
KFNlc3Npb25IYW5kbGVyLmphdmE6MTgyKQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuaGFuZGxlci5D
b250ZXh0SGFuZGxlci5oYW5kbGUoQ29udGV4dEhhbmRsZXIuamF2YTo3NjYpCglhdCBvcmcubW9y
dGJheS5qZXR0eS53ZWJhcHAuV2ViQXBwQ29udGV4dC5oYW5kbGUoV2ViQXBwQ29udGV4dC5qYXZh
OjQ1MCkKCWF0IG9yZy5tb3J0YmF5LmpldHR5LmhhbmRsZXIuQ29udGV4dEhhbmRsZXJDb2xsZWN0
aW9uLmhhbmRsZShDb250ZXh0SGFuZGxlckNvbGxlY3Rpb24uamF2YToyMzApCglhdCBvcmcubW9y
dGJheS5qZXR0eS5oYW5kbGVyLkhhbmRsZXJXcmFwcGVyLmhhbmRsZShIYW5kbGVyV3JhcHBlci5q
YXZhOjE1MikKCWF0IG9yZy5tb3J0YmF5LmpldHR5LlNlcnZlci5oYW5kbGUoU2VydmVyLmphdmE6
MzI2KQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuSHR0cENvbm5lY3Rpb24uaGFuZGxlUmVxdWVzdChI
dHRwQ29ubmVjdGlvbi5qYXZhOjU0MikKCWF0IG9yZy5tb3J0YmF5LmpldHR5Lkh0dHBDb25uZWN0
aW9uJFJlcXVlc3RIYW5kbGVyLmhlYWRlckNvbXBsZXRlKEh0dHBDb25uZWN0aW9uLmphdmE6OTI4
KQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuSHR0cFBhcnNlci5wYXJzZU5leHQoSHR0cFBhcnNlci5q
YXZhOjU0OSkKCWF0IG9yZy5tb3J0YmF5LmpldHR5Lkh0dHBQYXJzZXIucGFyc2VBdmFpbGFibGUo
SHR0cFBhcnNlci5qYXZhOjIxMikKCWF0IG9yZy5tb3J0YmF5LmpldHR5Lkh0dHBDb25uZWN0aW9u
LmhhbmRsZShIdHRwQ29ubmVjdGlvbi5qYXZhOjQwNCkKCWF0IG9yZy5tb3J0YmF5LmlvLm5pby5T
ZWxlY3RDaGFubmVsRW5kUG9pbnQucnVuKFNlbGVjdENoYW5uZWxFbmRQb2ludC5qYXZhOjQxMCkK
CWF0IG9yZy5tb3J0YmF5LnRocmVhZC5RdWV1ZWRUaHJlYWRQb29sJFBvb2xUaHJlYWQucnVuKFF1
ZXVlZFRocmVhZFBvb2wuamF2YTo1ODIpCkNhdXNlZCBieTogamF2YS5pby5JT0V4Y2VwdGlvbjog
QnJva2VuIHBpcGUKCWF0IHN1bi5uaW8uY2guRmlsZURpc3BhdGNoZXIud3JpdGUwKE5hdGl2ZSBN
ZXRob2QpCglhdCBzdW4ubmlvLmNoLlNvY2tldERpc3BhdGNoZXIud3JpdGUoU29ja2V0RGlzcGF0
Y2hlci5qYXZhOjI5KQoJYXQgc3VuLm5pby5jaC5JT1V0aWwud3JpdGVGcm9tTmF0aXZlQnVmZmVy
KElPVXRpbC5qYXZhOjcyKQoJYXQgc3VuLm5pby5jaC5JT1V0aWwud3JpdGUoSU9VdGlsLmphdmE6
NDMpCglhdCBzdW4ubmlvLmNoLlNvY2tldENoYW5uZWxJbXBsLndyaXRlKFNvY2tldENoYW5uZWxJ
bXBsLmphdmE6MzM0KQoJYXQgb3JnLm1vcnRiYXkuaW8ubmlvLkNoYW5uZWxFbmRQb2ludC5mbHVz
aChDaGFubmVsRW5kUG9pbnQuamF2YToxNzApCglhdCBvcmcubW9ydGJheS5pby5uaW8uU2VsZWN0
Q2hhbm5lbEVuZFBvaW50LmZsdXNoKFNlbGVjdENoYW5uZWxFbmRQb2ludC5qYXZhOjIyMSkKCWF0
IG9yZy5tb3J0YmF5LmpldHR5Lkh0dHBHZW5lcmF0b3IuZmx1c2goSHR0cEdlbmVyYXRvci5qYXZh
OjcyNSkKCS4uLiAyNiBtb3JlCgoyMDExLTA1LTE1IDE1OjUwOjQ0LDk5NSBXQVJOIG9yZy5tb3J0
YmF5LmxvZzogQ29tbWl0dGVkIGJlZm9yZSA0MTAgZ2V0TWFwT3V0cHV0KGF0dGVtcHRfMjAxMTA1
MTQyMTM3XzAwMDFfbV8wMDAzOTRfMCwxNCkgZmFpbGVkIDoKb3JnLm1vcnRiYXkuamV0dHkuRW9m
RXhjZXB0aW9uCglhdCBvcmcubW9ydGJheS5qZXR0eS5IdHRwR2VuZXJhdG9yLmZsdXNoKEh0dHBH
ZW5lcmF0b3IuamF2YTo3OTEpCglhdCBvcmcubW9ydGJheS5qZXR0eS5BYnN0cmFjdEdlbmVyYXRv
ciRPdXRwdXQuZmx1c2goQWJzdHJhY3RHZW5lcmF0b3IuamF2YTo1NjkpCglhdCBvcmcubW9ydGJh
eS5qZXR0eS5IdHRwQ29ubmVjdGlvbiRPdXRwdXQuZmx1c2goSHR0cENvbm5lY3Rpb24uamF2YTox
MDEyKQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuQWJzdHJhY3RHZW5lcmF0b3IkT3V0cHV0LndyaXRl
KEFic3RyYWN0R2VuZXJhdG9yLmphdmE6NjUxKQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuQWJzdHJh
Y3RHZW5lcmF0b3IkT3V0cHV0LndyaXRlKEFic3RyYWN0R2VuZXJhdG9yLmphdmE6NTgwKQoJYXQg
b3JnLmFwYWNoZS5oYWRvb3AubWFwcmVkLlRhc2tUcmFja2VyJE1hcE91dHB1dFNlcnZsZXQuZG9H
ZXQoVGFza1RyYWNrZXIuamF2YTozNjkzKQoJYXQgamF2YXguc2VydmxldC5odHRwLkh0dHBTZXJ2
bGV0LnNlcnZpY2UoSHR0cFNlcnZsZXQuamF2YTo3MDcpCglhdCBqYXZheC5zZXJ2bGV0Lmh0dHAu
SHR0cFNlcnZsZXQuc2VydmljZShIdHRwU2VydmxldC5qYXZhOjgyMCkKCWF0IG9yZy5tb3J0YmF5
LmpldHR5LnNlcnZsZXQuU2VydmxldEhvbGRlci5oYW5kbGUoU2VydmxldEhvbGRlci5qYXZhOjUx
MSkKCWF0IG9yZy5tb3J0YmF5LmpldHR5LnNlcnZsZXQuU2VydmxldEhhbmRsZXIkQ2FjaGVkQ2hh
aW4uZG9GaWx0ZXIoU2VydmxldEhhbmRsZXIuamF2YToxMjIxKQoJYXQgb3JnLmFwYWNoZS5oYWRv
b3AuaHR0cC5IdHRwU2VydmVyJFF1b3RpbmdJbnB1dEZpbHRlci5kb0ZpbHRlcihIdHRwU2VydmVy
LmphdmE6ODI0KQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuc2VydmxldC5TZXJ2bGV0SGFuZGxlciRD
YWNoZWRDaGFpbi5kb0ZpbHRlcihTZXJ2bGV0SGFuZGxlci5qYXZhOjEyMTIpCglhdCBvcmcubW9y
dGJheS5qZXR0eS5zZXJ2bGV0LlNlcnZsZXRIYW5kbGVyLmhhbmRsZShTZXJ2bGV0SGFuZGxlci5q
YXZhOjM5OSkKCWF0IG9yZy5tb3J0YmF5LmpldHR5LnNlY3VyaXR5LlNlY3VyaXR5SGFuZGxlci5o
YW5kbGUoU2VjdXJpdHlIYW5kbGVyLmphdmE6MjE2KQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuc2Vy
dmxldC5TZXNzaW9uSGFuZGxlci5oYW5kbGUoU2Vzc2lvbkhhbmRsZXIuamF2YToxODIpCglhdCBv
cmcubW9ydGJheS5qZXR0eS5oYW5kbGVyLkNvbnRleHRIYW5kbGVyLmhhbmRsZShDb250ZXh0SGFu
ZGxlci5qYXZhOjc2NikKCWF0IG9yZy5tb3J0YmF5LmpldHR5LndlYmFwcC5XZWJBcHBDb250ZXh0
LmhhbmRsZShXZWJBcHBDb250ZXh0LmphdmE6NDUwKQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuaGFu
ZGxlci5Db250ZXh0SGFuZGxlckNvbGxlY3Rpb24uaGFuZGxlKENvbnRleHRIYW5kbGVyQ29sbGVj
dGlvbi5qYXZhOjIzMCkKCWF0IG9yZy5tb3J0YmF5LmpldHR5LmhhbmRsZXIuSGFuZGxlcldyYXBw
ZXIuaGFuZGxlKEhhbmRsZXJXcmFwcGVyLmphdmE6MTUyKQoJYXQgb3JnLm1vcnRiYXkuamV0dHku
U2VydmVyLmhhbmRsZShTZXJ2ZXIuamF2YTozMjYpCglhdCBvcmcubW9ydGJheS5qZXR0eS5IdHRw
Q29ubmVjdGlvbi5oYW5kbGVSZXF1ZXN0KEh0dHBDb25uZWN0aW9uLmphdmE6NTQyKQoJYXQgb3Jn
Lm1vcnRiYXkuamV0dHkuSHR0cENvbm5lY3Rpb24kUmVxdWVzdEhhbmRsZXIuaGVhZGVyQ29tcGxl
dGUoSHR0cENvbm5lY3Rpb24uamF2YTo5MjgpCglhdCBvcmcubW9ydGJheS5qZXR0eS5IdHRwUGFy
c2VyLnBhcnNlTmV4dChIdHRwUGFyc2VyLmphdmE6NTQ5KQoJYXQgb3JnLm1vcnRiYXkuamV0dHku
SHR0cFBhcnNlci5wYXJzZUF2YWlsYWJsZShIdHRwUGFyc2VyLmphdmE6MjEyKQoJYXQgb3JnLm1v
cnRiYXkuamV0dHkuSHR0cENvbm5lY3Rpb24uaGFuZGxlKEh0dHBDb25uZWN0aW9uLmphdmE6NDA0
KQoJYXQgb3JnLm1vcnRiYXkuaW8ubmlvLlNlbGVjdENoYW5uZWxFbmRQb2ludC5ydW4oU2VsZWN0
Q2hhbm5lbEVuZFBvaW50LmphdmE6NDEwKQoJYXQgb3JnLm1vcnRiYXkudGhyZWFkLlF1ZXVlZFRo
cmVhZFBvb2wkUG9vbFRocmVhZC5ydW4oUXVldWVkVGhyZWFkUG9vbC5qYXZhOjU4MikKQ2F1c2Vk
IGJ5OiBqYXZhLmlvLklPRXhjZXB0aW9uOiBCcm9rZW4gcGlwZQoJYXQgc3VuLm5pby5jaC5GaWxl
RGlzcGF0Y2hlci53cml0ZTAoTmF0aXZlIE1ldGhvZCkKCWF0IHN1bi5uaW8uY2guU29ja2V0RGlz
cGF0Y2hlci53cml0ZShTb2NrZXREaXNwYXRjaGVyLmphdmE6MjkpCglhdCBzdW4ubmlvLmNoLklP
VXRpbC53cml0ZUZyb21OYXRpdmVCdWZmZXIoSU9VdGlsLmphdmE6NzIpCglhdCBzdW4ubmlvLmNo
LklPVXRpbC53cml0ZShJT1V0aWwuamF2YTo0MykKCWF0IHN1bi5uaW8uY2guU29ja2V0Q2hhbm5l
bEltcGwud3JpdGUoU29ja2V0Q2hhbm5lbEltcGwuamF2YTozMzQpCglhdCBvcmcubW9ydGJheS5p
by5uaW8uQ2hhbm5lbEVuZFBvaW50LmZsdXNoKENoYW5uZWxFbmRQb2ludC5qYXZhOjE3MCkKCWF0
IG9yZy5tb3J0YmF5LmlvLm5pby5TZWxlY3RDaGFubmVsRW5kUG9pbnQuZmx1c2goU2VsZWN0Q2hh
bm5lbEVuZFBvaW50LmphdmE6MjIxKQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuSHR0cEdlbmVyYXRv
ci5mbHVzaChIdHRwR2VuZXJhdG9yLmphdmE6NzI1KQoJLi4uIDI2IG1vcmUKCjIwMTEtMDUtMTUg
MTU6NTA6NDQsOTk1IElORk8gb3JnLmFwYWNoZS5oYWRvb3AubWFwcmVkLlRhc2tUcmFja2VyLmNs
aWVudHRyYWNlOiBzcmM6IDE5Mi4xNjguMjguMjE0OjUwMDYwLCBkZXN0OiAxOTIuMTY4LjI4LjIx
MzozMzQ5MCwgYnl0ZXM6IDEzMTA3Miwgb3A6IE1BUFJFRF9TSFVGRkxFLCBjbGlJRDogYXR0ZW1w
dF8yMDExMDUxNDIxMzdfMDAwMV9tXzAwMDM5NF8wLCBkdXJhdGlvbjogMzMyNjAwMDAKMjAxMS0w
NS0xNSAxNTo1MDo0NCw5OTUgRVJST1Igb3JnLm1vcnRiYXkubG9nOiAvbWFwT3V0cHV0CmphdmEu
bGFuZy5JbGxlZ2FsU3RhdGVFeGNlcHRpb246IENvbW1pdHRlZAoJYXQgb3JnLm1vcnRiYXkuamV0
dHkuUmVzcG9uc2UucmVzZXRCdWZmZXIoUmVzcG9uc2UuamF2YToxMDIzKQoJYXQgb3JnLm1vcnRi
YXkuamV0dHkuUmVzcG9uc2Uuc2VuZEVycm9yKFJlc3BvbnNlLmphdmE6MjQwKQoJYXQgb3JnLmFw
YWNoZS5oYWRvb3AubWFwcmVkLlRhc2tUcmFja2VyJE1hcE91dHB1dFNlcnZsZXQuZG9HZXQoVGFz
a1RyYWNrZXIuamF2YTozNzE4KQoJYXQgamF2YXguc2VydmxldC5odHRwLkh0dHBTZXJ2bGV0LnNl
cnZpY2UoSHR0cFNlcnZsZXQuamF2YTo3MDcpCglhdCBqYXZheC5zZXJ2bGV0Lmh0dHAuSHR0cFNl
cnZsZXQuc2VydmljZShIdHRwU2VydmxldC5qYXZhOjgyMCkKCWF0IG9yZy5tb3J0YmF5LmpldHR5
LnNlcnZsZXQuU2VydmxldEhvbGRlci5oYW5kbGUoU2VydmxldEhvbGRlci5qYXZhOjUxMSkKCWF0
IG9yZy5tb3J0YmF5LmpldHR5LnNlcnZsZXQuU2VydmxldEhhbmRsZXIkQ2FjaGVkQ2hhaW4uZG9G
aWx0ZXIoU2VydmxldEhhbmRsZXIuamF2YToxMjIxKQoJYXQgb3JnLmFwYWNoZS5oYWRvb3AuaHR0
cC5IdHRwU2VydmVyJFF1b3RpbmdJbnB1dEZpbHRlci5kb0ZpbHRlcihIdHRwU2VydmVyLmphdmE6
ODI0KQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuc2VydmxldC5TZXJ2bGV0SGFuZGxlciRDYWNoZWRD
aGFpbi5kb0ZpbHRlcihTZXJ2bGV0SGFuZGxlci5qYXZhOjEyMTIpCglhdCBvcmcubW9ydGJheS5q
ZXR0eS5zZXJ2bGV0LlNlcnZsZXRIYW5kbGVyLmhhbmRsZShTZXJ2bGV0SGFuZGxlci5qYXZhOjM5
OSkKCWF0IG9yZy5tb3J0YmF5LmpldHR5LnNlY3VyaXR5LlNlY3VyaXR5SGFuZGxlci5oYW5kbGUo
U2VjdXJpdHlIYW5kbGVyLmphdmE6MjE2KQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuc2VydmxldC5T
ZXNzaW9uSGFuZGxlci5oYW5kbGUoU2Vzc2lvbkhhbmRsZXIuamF2YToxODIpCglhdCBvcmcubW9y
dGJheS5qZXR0eS5oYW5kbGVyLkNvbnRleHRIYW5kbGVyLmhhbmRsZShDb250ZXh0SGFuZGxlci5q
YXZhOjc2NikKCWF0IG9yZy5tb3J0YmF5LmpldHR5LndlYmFwcC5XZWJBcHBDb250ZXh0LmhhbmRs
ZShXZWJBcHBDb250ZXh0LmphdmE6NDUwKQoJYXQgb3JnLm1vcnRiYXkuamV0dHkuaGFuZGxlci5D
b250ZXh0SGFuZGxlckNvbGxlY3Rpb24uaGFuZGxlKENvbnRleHRIYW5kbGVyQ29sbGVjdGlvbi5q
YXZhOjIzMCkKCWF0IG9yZy5tb3J0YmF5LmpldHR5LmhhbmRsZXIuSGFuZGxlcldyYXBwZXIuaGFu
ZGxlKEhhbmRsZXJXcmFwcGVyLmphdmE6MTUyKQo=

--_002_ADF94D8555C7A246B86A633685E0178AB90A35CA1Eplanckkasaran_--

From general-return-3549-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 17:19:32 2011
Return-Path: <general-return-3549-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 65D6C6087
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 16 May 2011 17:19:32 +0000 (UTC)
Received: (qmail 56097 invoked by uid 500); 16 May 2011 17:19:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55883 invoked by uid 500); 16 May 2011 17:19:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55874 invoked by uid 99); 16 May 2011 17:19:31 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 17:19:31 +0000
Received: from localhost (HELO awittena-mn.linkedin.biz) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 17:19:30 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Defining Hadoop Compatibility -revisiting-
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <A4FB782E-2433-47F1-AE91-CEFC42654C3A@navteq.com>
Date: Mon, 16 May 2011 10:19:29 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <434E417B-C21F-4649-9D72-4E6DCA558C18@apache.org>
References: <C9F227D5.8FA4%mbhandarkar@linkedin.com> <4DCCF172.5010504@apache.org> <99AB77CF-025E-4E4A-BD7F-BDA362E633BF@apache.org> <4DCDA8E7.5090809@apache.org> <2ED3A7F5-E549-4545-9395-094A10EA4EED@apache.org> <BANLkTik=EkwPxvcw2-vw=hsofsJYRgyaGg@mail.gmail.com> <BEC4433A-2270-47CF-960D-CB416B20FD4A@apache.org> <4DD10420.7060004@apache.org> <A4FB782E-2433-47F1-AE91-CEFC42654C3A@navteq.com>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On May 16, 2011, at 5:00 AM, Segel, Mike wrote:
> X represents the set of stable releases.
> Y represents the set of available patches.
> C represents the set of Cloudera releases.
>=20
> So if C contains a release X(n) plus a set of patches that is =
contained in Y,
> Then does it not have the right to be considered Apache Hadoop?
> It's my understanding is that any enhancement to Hadoop is made =
available to Apache and will eventually make it into a later release...

	This assumption is probably wrong.  It likely wouldn't be hard =
to find patches made in Cloudera Hadoop that have been rejected from =
Apache Hadoop.  I know some of the code in Cloudera Hadoop 2 was =
definitely rejected.  If Cloudera Hadoop 3's lineage is based upon 2...

=09


From general-return-3550-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 18:04:05 2011
Return-Path: <general-return-3550-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 46C316B7D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 16 May 2011 18:04:05 +0000 (UTC)
Received: (qmail 29016 invoked by uid 500); 16 May 2011 18:04:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28960 invoked by uid 500); 16 May 2011 18:04:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28952 invoked by uid 99); 16 May 2011 18:04:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 18:04:03 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [145.100.8.49] (HELO smtp.sara.nl) (145.100.8.49)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 18:03:54 +0000
Received: from planck.ka.sara.nl (145.100.8.32) by sara-exch-fe1.ka.sara.nl
 (145.100.8.49) with Microsoft SMTP Server (TLS) id 14.0.702.0; Mon, 16 May
 2011 20:03:34 +0200
Received: from planck.ka.sara.nl ([145.100.8.32]) by planck.ka.sara.nl
 ([145.100.8.32]) with mapi; Mon, 16 May 2011 20:03:34 +0200
From: Evert Lammerts <Evert.Lammerts@sara.nl>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 16 May 2011 20:03:34 +0200
Subject: Acceptance tests
Thread-Topic: Acceptance tests
Thread-Index: AcwT85JwHeXJRxetTCmdP3G6Kzlskw==
Message-ID: <ADF94D8555C7A246B86A633685E0178AB90A35CA1B@planck.ka.sara.nl>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Hi all,

What acceptance tests are people using when buying clusters for Hadoop? Any=
 pointers to relevant methods?

Thanks,
Evert Lammerts=

From general-return-3551-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 19:46:19 2011
Return-Path: <general-return-3551-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 235456AF1
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 16 May 2011 19:46:19 +0000 (UTC)
Received: (qmail 5946 invoked by uid 500); 16 May 2011 19:46:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5888 invoked by uid 500); 16 May 2011 19:46:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5879 invoked by uid 99); 16 May 2011 19:46:17 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 19:46:17 +0000
Received: from localhost (HELO awittena-mn.linkedin.biz) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 19:46:17 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Acceptance tests
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <ADF94D8555C7A246B86A633685E0178AB90A35CA1B@planck.ka.sara.nl>
Date: Mon, 16 May 2011 12:46:16 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <3A2CF811-0C7E-4B05-8DB1-95A4D1FF8ACB@apache.org>
References: <ADF94D8555C7A246B86A633685E0178AB90A35CA1B@planck.ka.sara.nl>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On May 16, 2011, at 11:03 AM, Evert Lammerts wrote:

> Hi all,
>=20
> What acceptance tests are people using when buying clusters for =
Hadoop? Any pointers to relevant methods?


	We get some test nodes from various manufacturers.  We do some =
raw IO benchmarking vs. our other nodes.  We add them to our various =
grids to see how they perform real world, paying attn to avg task time =
turn around for certain jobs.   Since we know where our current machines =
are at, we can look at price per perf improvements.

	Other random things that I think are important:

		a) Unless someone shares their entire *-site.xml data, =
most published benchmarks on the net are mostly useless.  Simple things =
like block size have a big impact.

		b) Test your actual workload.  Synthetic benchmarks are =
just that--synthetic.  They may not reflect that particular nuances of =
your job.

		c) Establish a baseline. If you have no hardware today, =
then at least establish something on EC2 to compare.

		d) Make sure you talk to multiple vendors.

		e) Any advice anyone gives you on config is likely going =
to be wrong.=

From general-return-3552-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 21:09:54 2011
Return-Path: <general-return-3552-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 05EFF6597
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 16 May 2011 21:09:54 +0000 (UTC)
Received: (qmail 69534 invoked by uid 500); 16 May 2011 21:09:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69471 invoked by uid 500); 16 May 2011 21:09:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69463 invoked by uid 99); 16 May 2011 21:09:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 21:09:52 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of eli@cloudera.com designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 21:09:47 +0000
Received: by ewy22 with SMTP id 22so2096721ewy.35
        for <general@hadoop.apache.org>; Mon, 16 May 2011 14:09:25 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.14.122.81 with SMTP id s57mr1883894eeh.195.1305580165512; Mon,
 16 May 2011 14:09:25 -0700 (PDT)
Received: by 10.14.37.13 with HTTP; Mon, 16 May 2011 14:09:25 -0700 (PDT)
In-Reply-To: <434E417B-C21F-4649-9D72-4E6DCA558C18@apache.org>
References: <C9F227D5.8FA4%mbhandarkar@linkedin.com>
	<4DCCF172.5010504@apache.org>
	<99AB77CF-025E-4E4A-BD7F-BDA362E633BF@apache.org>
	<4DCDA8E7.5090809@apache.org>
	<2ED3A7F5-E549-4545-9395-094A10EA4EED@apache.org>
	<BANLkTik=EkwPxvcw2-vw=hsofsJYRgyaGg@mail.gmail.com>
	<BEC4433A-2270-47CF-960D-CB416B20FD4A@apache.org>
	<4DD10420.7060004@apache.org>
	<A4FB782E-2433-47F1-AE91-CEFC42654C3A@navteq.com>
	<434E417B-C21F-4649-9D72-4E6DCA558C18@apache.org>
Date: Mon, 16 May 2011 14:09:25 -0700
Message-ID: <BANLkTinXg7twM7KUrKpHpTycyaxnPmnEfg@mail.gmail.com>
Subject: Re: Defining Hadoop Compatibility -revisiting-
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Mon, May 16, 2011 at 10:19 AM, Allen Wittenauer <aw@apache.org> wrote:
>
> On May 16, 2011, at 5:00 AM, Segel, Mike wrote:
>> X represents the set of stable releases.
>> Y represents the set of available patches.
>> C represents the set of Cloudera releases.
>>
>> So if C contains a release X(n) plus a set of patches that is contained =
in Y,
>> Then does it not have the right to be considered Apache Hadoop?
>> It's my understanding is that any enhancement to Hadoop is made availabl=
e to Apache and will eventually make it into a later release...
>
> =A0 =A0 =A0 =A0This assumption is probably wrong. =A0It likely wouldn't b=
e hard to find patches made in Cloudera Hadoop that have been rejected from=
 Apache Hadoop. =A0I know some of the code in Cloudera Hadoop 2 was definit=
ely rejected. =A0If Cloudera Hadoop 3's lineage is based upon 2...

Allen,

There are few things in Hadoop in CDH that are not in trunk,
branch-20-security, or branch-20-append.  The stuff in this category
is not major (eg HADOOP-6605, better JAVA_HOME detection).

One of the things we and others are busy doing is getting the work
from CDH3 and 20x (formerly YDH) checked into trunk so a future
release won't regress against these 20-based releases.

Most projects in CDH are not heavily patched btw, they're close to an
upstream Apache release.  Hadoop is the exception.
https://ccp.cloudera.com/display/DOC/Downloading+CDH+Releases

Thanks,
Eli

From general-return-3553-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 21:18:55 2011
Return-Path: <general-return-3553-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C2A1D6B9C
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 16 May 2011 21:18:55 +0000 (UTC)
Received: (qmail 80884 invoked by uid 500); 16 May 2011 21:18:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80818 invoked by uid 500); 16 May 2011 21:18:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80810 invoked by uid 99); 16 May 2011 21:18:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 21:18:54 +0000
X-ASF-Spam-Status: No, hits=3.3 required=5.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 21:18:46 +0000
Received: from SP2-EX07CAS03.ds.corp.yahoo.com (sp2-ex07cas03.corp.sp2.yahoo.com [98.137.59.35])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p4GLHwx3041328
	for <general@hadoop.apache.org>; Mon, 16 May 2011 14:17:58 -0700 (PDT)
Received: from SP2-EX07VS03.ds.corp.yahoo.com ([98.137.59.32]) by
 SP2-EX07CAS03.ds.corp.yahoo.com ([98.137.59.35]) with mapi; Mon, 16 May 2011
 14:17:58 -0700
From: Matthew Foley <mattf@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: Matthew Foley <mattf@yahoo-inc.com>
Date: Mon, 16 May 2011 14:17:57 -0700
Subject: Re: Defining Hadoop Compatibility -revisiting-
Thread-Topic: Defining Hadoop Compatibility -revisiting-
Thread-Index: AcwUDrqxSBO3RyeeSueeq1hAW5ouXg==
Message-ID: <54800038-3C7F-4D2C-A54B-D7F8A8F7F94F@yahoo-inc.com>
References: <C9F227D5.8FA4%mbhandarkar@linkedin.com>
 <4DCCF172.5010504@apache.org>
 <99AB77CF-025E-4E4A-BD7F-BDA362E633BF@apache.org>
 <4DCDA8E7.5090809@apache.org>
 <2ED3A7F5-E549-4545-9395-094A10EA4EED@apache.org>
 <BANLkTik=EkwPxvcw2-vw=hsofsJYRgyaGg@mail.gmail.com>
 <BEC4433A-2270-47CF-960D-CB416B20FD4A@apache.org>
 <4DD10420.7060004@apache.org>
 <A4FB782E-2433-47F1-AE91-CEFC42654C3A@navteq.com>
In-Reply-To: <A4FB782E-2433-47F1-AE91-CEFC42654C3A@navteq.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_548000383C7F4D2CA54BD7F8A8F7F94Fyahooinccom_"
MIME-Version: 1.0

--_000_548000383C7F4D2CA54BD7F8A8F7F94Fyahooinccom_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

It's important to distinguish between the name "Hadoop", which is protected=
 by trademark law,
and the Hadoop implementation, which is licensed as opensource under copyri=
ght law.

The term "derivative work" is, I believe, only relevant under copyright law=
, not trademark law.
(N.B., I'm not a lawyer -- and this email is my opinion, not my employer's.=
)  Since the Apache License
explicitly allows derivative works, I don't think it's a useful term for th=
is discussion.

However, the ASF, and by delegation the Hadoop PMC, has a lot of control ov=
er the name,
and how we allow it to be used, under trademark law.  But to keep our right=
s under that
law, we have to enforce the trademark consistently.  So it's good that we'r=
e having this discussion,
and it's important to reach a conclusion, document it, and enforce it consi=
stently.

There are a lot of subtleties; for instance, if I recall correctly from my =
days with Adobe and
PostScript(R), someone who has not licensed a trademark "X" can still claim=
 "compatible with X"
as long as they ALSO make clear that the product is NOT, itself, an "X".  B=
ut you really need
a lawyer to get into that stuff.

--Matt


On May 16, 2011, at 5:00 AM, Segel, Mike wrote:

But Cloudera's release is a bit murky.

The math example is a bit flawed...

X represents the set of stable releases.
Y represents the set of available patches.
C represents the set of Cloudera releases.

So if C contains a release X(n) plus a set of patches that is contained in =
Y,
Then does it not have the right to be considered Apache Hadoop?
It's my understanding is that any enhancement to Hadoop is made available t=
o Apache and will eventually make it into a later release...

So while it may not be 'official' release X(z), all of it's components are =
in Apache.
(note: I'm talking about the core components and not Cloudera's additional =
toolsets that encompass Hadoop.)

Cloudera is clearly a derivative work.
And IMHO is the only one which can say ... 'Includes Apache Hadoop'.

That doesn't mean that others can't, depending on how they implemented thei=
r changes.
Based on EMC marketing material, they've done a rip and replace of HDFS.
So it wouldn't be a superset since it doesn't contain a complete subset, bu=
t contains code that implements the API... So they can't say 'Includes Apac=
he Hadoop',but they can say it's a derivative work based on Apache Hadoop a=
nd then go on to show how and why, in their opinion their product is better=
.(that's marketing for you...)

Clearly there are others out there...
Hadoop on Cassandra as an example...

Fragmentation of Hadoop will occur. It's inevitable. Too much money is on t=
he table...

But because Apache's licensing is so open, Apache will have a hard time con=
trolling derivative works...
I believe that Steve is incorrect in his assertion concerning potential los=
s of any patent protection. Again Apache's licensing is very open and as lo=
ng as they follow Apache's Ts and Cs, they are covered.

Note: because I am sending this from my email address at my client, I am ob=
liged to say that this email is my opinion and does not reflect on the opin=
ion of my client...
(you know the rest....)

Sent from a remote device. Please excuse any typos...

Mike Segel

On May 16, 2011, at 6:02 AM, "Steve Loughran" <stevel@apache.org<mailto:ste=
vel@apache.org>> wrote:

On 13/05/11 23:57, Allen Wittenauer wrote:

On May 13, 2011, at 3:53 PM, Ted Dunning wrote:

But "distribution Z includes X" kind of implies the existence of some such
that X !=3D Y, Y !=3D empty-set and X+Y =3D Z, at least in common usage.

Isn't that the same as a non-trunk change?

So doesn't this mean that your question reduces to the question of what
happens when non-Apache changes are made to an Apache release?  And isn't
that the definition of a derived work?


  Yup. Which is why I doubt *any* commercial entity can claim "includes Apa=
che Hadoop" (including Cloudera).



but they can claim it is a derivative work, which CDH clearly is,
(Though if we were to come up with a formal declaration of what a
derivative work is, we'd have to handle the fact that it is a superset.
Even worse, you may realise a release is the ordered application of a
sequence of patches, and if the patches are applied in a different order
you may end up with a different body of source code...)

Something that implements the APIs may not be a derivative work,
depending on how much of the original code is in there. You could look
at the base classes and interfaces and produce a clean room
implementation (relying on the notion that interfaces are a list of
facts and not copyrightable in the US), but whoever does that may
encounter the issue that Google's donation of the right to use their MR
patent may not apply to such implementations.


The information contained in this communication may be CONFIDENTIAL and is =
intended only for the use of the recipient(s) named above.  If you are not =
the intended recipient, you are hereby notified that any dissemination, dis=
tribution, or copying of this communication, or any of its contents, is str=
ictly prohibited.  If you have received this communication in error, please=
 notify the sender and delete/destroy the original message and any copy of =
it from your computer or paper files.


--_000_548000383C7F4D2CA54BD7F8A8F7F94Fyahooinccom_--

From general-return-3554-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 21:25:47 2011
Return-Path: <general-return-3554-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DA50864D2
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 16 May 2011 21:25:47 +0000 (UTC)
Received: (qmail 97582 invoked by uid 500); 16 May 2011 21:25:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97527 invoked by uid 500); 16 May 2011 21:25:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97519 invoked by uid 99); 16 May 2011 21:25:46 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 21:25:46 +0000
Received: from localhost (HELO awittena-mn.linkedin.biz) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 21:25:46 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Defining Hadoop Compatibility -revisiting-
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <BANLkTinXg7twM7KUrKpHpTycyaxnPmnEfg@mail.gmail.com>
Date: Mon, 16 May 2011 14:25:44 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <D972DAFB-26F5-4285-8D10-48C2BB2A5A4E@apache.org>
References: <C9F227D5.8FA4%mbhandarkar@linkedin.com> <4DCCF172.5010504@apache.org> <99AB77CF-025E-4E4A-BD7F-BDA362E633BF@apache.org> <4DCDA8E7.5090809@apache.org> <2ED3A7F5-E549-4545-9395-094A10EA4EED@apache.org> <BANLkTik=EkwPxvcw2-vw=hsofsJYRgyaGg@mail.gmail.com> <BEC4433A-2270-47CF-960D-CB416B20FD4A@apache.org> <4DD10420.7060004@apache.org> <A4FB782E-2433-47F1-AE91-CEFC42654C3A@navteq.com> <434E417B-C21F-4649-9D72-4E6DCA558C18@apache.org> <BANLkTinXg7twM7KUrKpHpTycyaxnPmnEfg@mail.gmail.com>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)


On May 16, 2011, at 2:09 PM, Eli Collins wrote:
>=20
> Allen,
>=20
> There are few things in Hadoop in CDH that are not in trunk,
> branch-20-security, or branch-20-append.  The stuff in this category
> is not major (eg HADOOP-6605, better JAVA_HOME detection).

	But that's my point:  when is it no longer Apache Hadoop?  How =
major does a change need to be under the line?    In the case of CDH2 =
and 3, in order to test it out, I actually had to back out some of =
Cloudera's "improvements" in order to even test whereas I didn't under =
Apache.  Is this another place where we only seem to care about APIs and =
say to hell with the rest of the stack?



From general-return-3555-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 21:29:29 2011
Return-Path: <general-return-3555-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7FF9A66BC
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 16 May 2011 21:29:29 +0000 (UTC)
Received: (qmail 3909 invoked by uid 500); 16 May 2011 21:29:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3859 invoked by uid 500); 16 May 2011 21:29:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3851 invoked by uid 99); 16 May 2011 21:29:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 21:29:27 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of eli@cloudera.com designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 21:29:21 +0000
Received: by ewy22 with SMTP id 22so2103399ewy.35
        for <general@hadoop.apache.org>; Mon, 16 May 2011 14:29:00 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.14.122.81 with SMTP id s57mr1888906eeh.195.1305581340274; Mon,
 16 May 2011 14:29:00 -0700 (PDT)
Received: by 10.14.37.13 with HTTP; Mon, 16 May 2011 14:29:00 -0700 (PDT)
In-Reply-To: <D972DAFB-26F5-4285-8D10-48C2BB2A5A4E@apache.org>
References: <C9F227D5.8FA4%mbhandarkar@linkedin.com>
	<4DCCF172.5010504@apache.org>
	<99AB77CF-025E-4E4A-BD7F-BDA362E633BF@apache.org>
	<4DCDA8E7.5090809@apache.org>
	<2ED3A7F5-E549-4545-9395-094A10EA4EED@apache.org>
	<BANLkTik=EkwPxvcw2-vw=hsofsJYRgyaGg@mail.gmail.com>
	<BEC4433A-2270-47CF-960D-CB416B20FD4A@apache.org>
	<4DD10420.7060004@apache.org>
	<A4FB782E-2433-47F1-AE91-CEFC42654C3A@navteq.com>
	<434E417B-C21F-4649-9D72-4E6DCA558C18@apache.org>
	<BANLkTinXg7twM7KUrKpHpTycyaxnPmnEfg@mail.gmail.com>
	<D972DAFB-26F5-4285-8D10-48C2BB2A5A4E@apache.org>
Date: Mon, 16 May 2011 14:29:00 -0700
Message-ID: <BANLkTi=E9u67P201cAC_F5XAkHi4hxmefQ@mail.gmail.com>
Subject: Re: Defining Hadoop Compatibility -revisiting-
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Mon, May 16, 2011 at 2:25 PM, Allen Wittenauer <aw@apache.org> wrote:
>
> On May 16, 2011, at 2:09 PM, Eli Collins wrote:
>>
>> Allen,
>>
>> There are few things in Hadoop in CDH that are not in trunk,
>> branch-20-security, or branch-20-append. =A0The stuff in this category
>> is not major (eg HADOOP-6605, better JAVA_HOME detection).
>
> =A0 =A0 =A0 =A0But that's my point: =A0when is it no longer Apache Hadoop=
? =A0How major does a change need to be under the line? =A0 =A0In the case =
of CDH2 and 3, in order to test it out, I actually had to back out some of =
Cloudera's "improvements" in order to even test whereas I didn't under Apac=
he. =A0Is this another place where we only seem to care about APIs and say =
to hell with the rest of the stack?
>

I don't think anyone is saying to hell with the rest of the stack, and
 everyone I've spoken to is on-board with a future release that
doesn't require lots of backporting from feature branches.

Thanks,
Eli

From general-return-3556-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 21:43:22 2011
Return-Path: <general-return-3556-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C06D76170
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 16 May 2011 21:43:22 +0000 (UTC)
Received: (qmail 41342 invoked by uid 500); 16 May 2011 21:43:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 41284 invoked by uid 500); 16 May 2011 21:43:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 41276 invoked by uid 99); 16 May 2011 21:43:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 21:43:21 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 21:43:16 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=S4N4v3hJ99UIu9m1vR1jWAGJ3sUF5gVnTVk0ACqqR7cfSKiJcZgqaPPE
   zKA6AzGj3kBZmGfmjjQRFEpknFTXhReMpZWzSR59OUYU+i6qIkv3G/xAh
   o/8rQp/Rwg0Tc92;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1305582196; x=1337118196;
  h=from:to:subject:date:message-id:references:in-reply-to:
   content-id:content-transfer-encoding:mime-version;
  bh=y/z7MUEfr/CuhwsCW7yGGIyqQ1oUN4MkRMaTu7ghAbQ=;
  b=tXBgo/TgDbd9DEPs1Lg0pnmfn+U95g/Zbr34/rUVGg7ggVvaGuk89oLi
   /NPjecBqKVjNJ7IMYVaeKlStzaKvzOGYgG4dO4nxpx6f17943kZ4F817U
   ARS13O4CgRMj1z/;
X-IronPort-AV: E=Sophos;i="4.65,221,1304319600"; 
   d="scan'208";a="23241731"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi id 14.01.0251.000; Mon,
 16 May 2011 14:42:55 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: Defining Hadoop Compatibility -revisiting-
Thread-Topic: Defining Hadoop Compatibility -revisiting-
Thread-Index: AQHMDv3EAOfNkuENQUmqX7bDeLVeOZSKwecAgAATHoCAABixAIAAkA6AgABKpYCAAAT/AIAACzIAgAABHgCAA+73AIAAEHMAgABZDoCAAEA+gIAABI8AgAAA6gCAAAPiAA==
Date: Mon, 16 May 2011 21:42:54 +0000
Message-ID: <6E1C9C7A-646A-44A2-9EB0-A11B7291B7B2@linkedin.com>
References: <C9F227D5.8FA4%mbhandarkar@linkedin.com>
 <4DCCF172.5010504@apache.org>
 <99AB77CF-025E-4E4A-BD7F-BDA362E633BF@apache.org>
 <4DCDA8E7.5090809@apache.org>
 <2ED3A7F5-E549-4545-9395-094A10EA4EED@apache.org>
 <BANLkTik=EkwPxvcw2-vw=hsofsJYRgyaGg@mail.gmail.com>
 <BEC4433A-2270-47CF-960D-CB416B20FD4A@apache.org>
 <4DD10420.7060004@apache.org>
 <A4FB782E-2433-47F1-AE91-CEFC42654C3A@navteq.com>
 <434E417B-C21F-4649-9D72-4E6DCA558C18@apache.org>
 <BANLkTinXg7twM7KUrKpHpTycyaxnPmnEfg@mail.gmail.com>
 <D972DAFB-26F5-4285-8D10-48C2BB2A5A4E@apache.org>
 <BANLkTi=E9u67P201cAC_F5XAkHi4hxmefQ@mail.gmail.com>
In-Reply-To: <BANLkTi=E9u67P201cAC_F5XAkHi4hxmefQ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <AEB44086DAAAED48A74D50980DE6C2BF@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


On May 16, 2011, at 2:29 PM, Eli Collins wrote:

> On Mon, May 16, 2011 at 2:25 PM, Allen Wittenauer <aw@apache.org> wrote:
>>=20
>> On May 16, 2011, at 2:09 PM, Eli Collins wrote:
>>>=20
>>> Allen,
>>>=20
>>> There are few things in Hadoop in CDH that are not in trunk,
>>> branch-20-security, or branch-20-append.  The stuff in this category
>>> is not major (eg HADOOP-6605, better JAVA_HOME detection).
>>=20
>>        But that's my point:  when is it no longer Apache Hadoop?  How ma=
jor does a change need to be under the line?    In the case of CDH2 and 3, =
in order to test it out, I actually had to back out some of Cloudera's "imp=
rovements" in order to even test whereas I didn't under Apache.  Is this an=
other place where we only seem to care about APIs and say to hell with the =
rest of the stack?
>>=20
>=20
> I don't think anyone is saying to hell with the rest of the stack, and
> everyone I've spoken to is on-board with a future release that
> doesn't require lots of backporting from feature branches.

	You've missed my point.

	Does "Hadoop compatibility" and the ability to say "includes Apache Hadoop=
" only apply when we're talking about MR and HDFS APIs?  =

From general-return-3557-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 21:59:44 2011
Return-Path: <general-return-3557-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1370872C6
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 16 May 2011 21:59:44 +0000 (UTC)
Received: (qmail 88656 invoked by uid 500); 16 May 2011 21:59:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88599 invoked by uid 500); 16 May 2011 21:59:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88591 invoked by uid 99); 16 May 2011 21:59:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 21:59:42 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 21:59:36 +0000
Received: by qyk30 with SMTP id 30so3545770qyk.14
        for <general@hadoop.apache.org>; Mon, 16 May 2011 14:59:15 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=laNOSyYp9caSB7NHWWILaHrZAp+/pyZDF2Wf2htXlT0=;
        b=CLyqT+sYncgPpGPLC0JfQVPyCeWGNl/ctkXlwfJ2wEfOQRAIex2uJnecZJ+/8QWHA0
         7SKX8GepmXAYqHvejc3aSLI/jdqLCOridkVyZL7wkV1e3F0BFZvtBaLpJ3eNSJYsOphS
         rT2CG9MPpELeV5y+osHkrBZdwf7kd9Xd1qYy0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=HyMv/ahjYl0FdROfA0P5bOAxGmw8fy30tp1CshkuNuIRFBH4wYSXi9pdUsbFNGRIfW
         L0zNwzhUl2viNPywQruc+gOA86Xf/ct+BokkyyXfd9GWzwOGDm5q1OI3kmBC7/YIeypS
         C2gJfzS663jXXiqEGo/CAbKXLjz1yEYXG60gg=
Received: by 10.224.195.67 with SMTP id eb3mr3762001qab.324.1305583155610;
        Mon, 16 May 2011 14:59:15 -0700 (PDT)
Received: from [10.172.1.246] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id t17sm3506368qcs.11.2011.05.16.14.59.13
        (version=TLSv1/SSLv3 cipher=OTHER);
        Mon, 16 May 2011 14:59:14 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: Defining Hadoop Compatibility -revisiting-
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <6E1C9C7A-646A-44A2-9EB0-A11B7291B7B2@linkedin.com>
Date: Tue, 17 May 2011 07:59:10 +1000
Content-Transfer-Encoding: quoted-printable
Message-Id: <B1A21BA0-69A9-45B6-AC18-E374DCA7B424@Holsman.NET>
References: <C9F227D5.8FA4%mbhandarkar@linkedin.com> <4DCCF172.5010504@apache.org> <99AB77CF-025E-4E4A-BD7F-BDA362E633BF@apache.org> <4DCDA8E7.5090809@apache.org> <2ED3A7F5-E549-4545-9395-094A10EA4EED@apache.org> <BANLkTik=EkwPxvcw2-vw=hsofsJYRgyaGg@mail.gmail.com> <BEC4433A-2270-47CF-960D-CB416B20FD4A@apache.org> <4DD10420.7060004@apache.org> <A4FB782E-2433-47F1-AE91-CEFC42654C3A@navteq.com> <434E417B-C21F-4649-9D72-4E6DCA558C18@apache.org> <BANLkTinXg7twM7KUrKpHpTycyaxnPmnEfg@mail.gmail.com> <D972DAFB-26F5-4285-8D10-48C2BB2A5A4E@apache.org> <BANLkTi=E9u67P201cAC_F5XAkHi4hxmefQ@mail.gmail.com> <6E1C9C7A-646A-44A2-9EB0-A11B7291B7B2@linkedin.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)


>=20
> 	Does "Hadoop compatibility" and the ability to say "includes =
Apache Hadoop" only apply when we're talking about MR and HDFS APIs? =20


It is confusing isn't it.

We could go down the route java did and say that the API's are 'hadoop' =
and ours is just a reference implementation of it. (but others pointed =
out, we don't want to become a certification group)

Out of curiosity, how good is our test suite in exercising our APIs?=20
Is it sophisticated enough to capture someone adding a =
functionality-changing patch (eg the append one). and have it flag it as =
a test-failure?=20


From general-return-3558-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 00:10:25 2011
Return-Path: <general-return-3558-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4F5346C90
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 17 May 2011 00:10:25 +0000 (UTC)
Received: (qmail 44108 invoked by uid 500); 17 May 2011 00:10:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44042 invoked by uid 500); 17 May 2011 00:10:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44034 invoked by uid 99); 17 May 2011 00:10:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 00:10:23 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of hammer@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 00:10:17 +0000
Received: by bwz8 with SMTP id 8so114663bwz.35
        for <general@hadoop.apache.org>; Mon, 16 May 2011 17:09:57 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.233.14 with SMTP id jw14mr21453bkb.40.1305590997246; Mon,
 16 May 2011 17:09:57 -0700 (PDT)
Received: by 10.204.37.204 with HTTP; Mon, 16 May 2011 17:09:57 -0700 (PDT)
In-Reply-To: <BANLkTinDNbnhWgFwX-v=mwjHz5j+7oXUNg@mail.gmail.com>
References: <BANLkTinDNbnhWgFwX-v=mwjHz5j+7oXUNg@mail.gmail.com>
Date: Mon, 16 May 2011 17:09:57 -0700
Message-ID: <BANLkTinxV5nexyuEiSt9fJ41907+QMrJyA@mail.gmail.com>
Subject: Re: Apache Hadoop Hackathon: 5/18 in Palo Alto and San Francisco
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=485b3979da849ce37704a36d972a
X-Virus-Checked: Checked by ClamAV on apache.org

--485b3979da849ce37704a36d972a
Content-Type: text/plain; charset=ISO-8859-1

Hey,

We've got a great group coming together again on Wednesday for an Apache
Hadoop Hackathon in Palo Alto and San Francisco. Sign up at
http://hadoophackathon.eventbrite.com.

As a reminder, we'll have Nigel Daley, the release manager for 0.22, present
in Palo Alto. If you have build and release or testing skills and would like
to contribute to Apache Hadoop, your skills will be highly valued. Please
consider coming out to meet Nigel and help out on that front!

Also, there's a Hadoop User Group at Yahoo! in Sunnyvale on Wednesday
evening: http://www.meetup.com/hadoop/events/16805258. There will be a large
caravan leaving from the Palo Alto Hackathon to attend the HUG, so if you
want to Hadoop all day, stop by Palo Alto and join us for the trip south.

Regards,
Jeff

On Thu, May 12, 2011 at 2:40 PM, Jeff Hammerbacher <hammer@cloudera.com>wrote:

> Hey,
>
> Thanks to everyone who came out for the Apache Hadoop Hackathon yesterday
> in Palo Alto and San Francisco. We had 35 people sign up from a great cross
> section of companies: Yahoo!, Cloudera, Facebook, Apple, Twitter,
> Foursquare, AOL, Ngmoco, StumbleUpon, Trend Micro, Conviva, and more. We had
> committers from the HBase, Hive, Pig, and Oozie projects ensuring their
> projects work with the upcoming 0.22 release, and a number of folks got
> their first patches up. The 0.22 branch is now being built by Jenkins and
> we're in good shape to get a release candidate up soon.
>
> We had so much fun that we're going to do it again!
>
> Join us next Wednesday, May 18th, in either San Francisco or Palo Alto, and
> help us continue the march to get feature development back on trunk. We'll
> have a special guest in Palo Alto: Nigel Daley, Apache Hadoop PMC member and
> the release manager for the 0.22 release. If you're a sysadmin or devops
> person and want to help us build and test Apache Hadoop, this Hackathon will
> be a great chance to learn about how it's done. Of course we'd also love to
> have developers looking to get their first patches into Hadoop as well.
>
> Sign up at http://hadoophackathon.eventbrite.com and we'll see you
> Wednesday.
>
> Regards,
> Jeff
>
> p.s. if you're looking to prepare for the Hackathon, check out the issues
> tagged "newbie" at
> https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&jqlQuery=labels+%3D+newbie
> .
>

--485b3979da849ce37704a36d972a--

From general-return-3559-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 00:40:17 2011
Return-Path: <general-return-3559-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CBEDB6B0C
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 17 May 2011 00:40:17 +0000 (UTC)
Received: (qmail 77711 invoked by uid 500); 17 May 2011 00:40:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77668 invoked by uid 500); 17 May 2011 00:40:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77660 invoked by uid 99); 17 May 2011 00:40:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 00:40:16 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of msegel@navteq.com designates 204.120.70.37 as permitted sender)
Received: from [204.120.70.37] (HELO xmailfargo.navteq.com) (204.120.70.37)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 00:40:09 +0000
Received: from fargo-spamfw.navteq.com (navteq-spamfw.navteq.com [10.6.10.131])
	by xmailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id p4H05avC031713
	for <general@hadoop.apache.org>; Mon, 16 May 2011 19:05:36 -0500
Received: from imailfargo.navteq.com (localhost [127.0.0.1])
	by fargo-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP
	id 1298448A52DF; Mon, 16 May 2011 19:39:48 -0500 (CDT)
Received: from imailfargo.navteq.com (imailfargo.navteq.com [10.6.10.130]) by fargo-spamfw.navteq.com with ESMTP id 48vjfnU69p3cH53j; Mon, 16 May 2011 19:39:48 -0500 (CDT)
Received: from hq-ex-ht02.ad.navteq.com ([10.8.222.172])
	by imailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id p4H0dlEu020205;
	Mon, 16 May 2011 19:39:47 -0500
Received: from hq-ex-mb03.ad.navteq.com ([fe80::c4dd:7b21:5c22:cfe4]) by
 hq-ex-ht02.ad.navteq.com ([fe80::24f6:ca50:734d:df6e%13]) with mapi; Mon, 16
 May 2011 19:39:47 -0500
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>,
        Matthew Foley
	<mattf@yahoo-inc.com>
Date: Mon, 16 May 2011 19:40:03 -0500
Subject: Re: Defining Hadoop Compatibility -revisiting-
Thread-Topic: Defining Hadoop Compatibility -revisiting-
Thread-Index: AcwUKuv060GtYE4ZSOG5VRabJbnqzw==
Message-ID: <1EB061D5-C193-4FC2-85AA-A4C2C8302B8D@navteq.com>
References: <C9F227D5.8FA4%mbhandarkar@linkedin.com>
 <4DCCF172.5010504@apache.org>
 <99AB77CF-025E-4E4A-BD7F-BDA362E633BF@apache.org>
 <4DCDA8E7.5090809@apache.org>
 <2ED3A7F5-E549-4545-9395-094A10EA4EED@apache.org>
 <BANLkTik=EkwPxvcw2-vw=hsofsJYRgyaGg@mail.gmail.com>
 <BEC4433A-2270-47CF-960D-CB416B20FD4A@apache.org>
 <4DD10420.7060004@apache.org>
 <A4FB782E-2433-47F1-AE91-CEFC42654C3A@navteq.com>
 <54800038-3C7F-4D2C-A54B-D7F8A8F7F94F@yahoo-inc.com>
In-Reply-To: <54800038-3C7F-4D2C-A54B-D7F8A8F7F94F@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1
X-Virus-Checked: Checked by ClamAV on apache.org

I just checked... TESS said no trademarks for Hadoop.
So... what TM protection? :-)

You are correct about derivative works. It's a moot point as long as the =
derivative work follows the T&Cs...



Sent from a remote device. Please excuse any typos...

Mike Segel

On May 16, 2011, at 4:18 PM, "Matthew Foley" <mattf@yahoo-inc.com> wrote:

> It's important to distinguish between the name "Hadoop", which is prote=
cted by trademark law,
> and the Hadoop implementation, which is licensed as opensource under co=
pyright law.
>=20
> The term "derivative work" is, I believe, only relevant under copyright=
 law, not trademark law.
> (N.B., I'm not a lawyer -- and this email is my opinion, not my employe=
r's.)  Since the Apache License
> explicitly allows derivative works, I don't think it's a useful term fo=
r this discussion.
>=20
> However, the ASF, and by delegation the Hadoop PMC, has a lot of contro=
l over the name,
> and how we allow it to be used, under trademark law.  But to keeps our =
rights under that
> law, we have to enforce the trademark consistently.  So it's good that =
we're having this discussion,
> and it's important to reach a conclusion, document it, and enforce it c=
onsistently.
>=20
> There are a lot of subtleties; for instance, if I recall correctly from=
 my days with Adobe and
> PostScript(R), someone who has not licensed a trademark "X" can still c=
laim "compatible with X"
> as long as they ALSO make clear that the product is NOT, itself, an "X"=
=2E  But you really need
> a lawyer to get into that stuff.
>=20
> --Matt
>=20
>=20
> On May 16, 2011, at 5:00 AM, Segel, Mike wrote:
>=20
> But Cloudera's release is a bit murky.
>=20
> The math example is a bit flawed...
>=20
> X represents the set of stable releases.
> Y represents the set of available patches.
> C represents the set of Cloudera releases.
>=20
> So if C contains a release X(n) plus a set of patches that is contained=
 in Y,
> Then does it not have the right to be considered Apache Hadoop?
> It's my understanding is that any enhancement to Hadoop is made availab=
le to Apache and will eventually make it into a later release...
>=20
> So while it may not be 'official' release X(z), all of it's components =
are in Apache.
> (note: I'm talking about the core components and not Cloudera's additio=
nal toolsets that encompass Hadoop.)
>=20
> Cloudera is clearly a derivative work.
> And IMHO is the only one which can say ... 'Includes Apache Hadoop'.
>=20
> That doesn't mean that others can't, depending on how they implemented =
their changes.
> Based on EMC marketing material, they've done a rip and replace of HDFS=
=2E
> So it wouldn't be a superset since it doesn't contain a complete subset=
, but contains code that implements the API... So they can't say 'Include=
s Apache Hadoop',but they can say it's a derivative work based on Apache =
Hadoop and then go on to show how and why, in their opinion their product=
 is better.(that's marketing for you...)
>=20
> Clearly there are others out there...
> Hadoop on Cassandra as an example...
>=20
> Fragmentation of Hadoop will occur. It's inevitable. Too much money is =
on the table...
>=20
> But because Apache's licensing is so open, Apache will have a hard time=
 controlling derivative works...
> I believe that Steve is incorrect in his assertion concerning potential=
 loss of any patent protection. Again Apache's licensing is very open and=
 as long as they follow Apache's Ts and Cs, they are covered.
>=20
> Note: because I am sending this from my email address at my client, I a=
m obliged to say that this email is my opinion and does not reflect on th=
e opinion of my client...
> (you know the rest....)
>=20
> Sent from a remote device. Please excuse any typos...
>=20
> Mike Segel
>=20
> On May 16, 2011, at 6:02 AM, "Steve Loughran" <stevel@apache.org<mailto=
:stevel@apache.org>> wrote:
>=20
> On 13/05/11 23:57, Allen Wittenauer wrote:
>=20
> On May 13, 2011, at 3:53 PM, Ted Dunning wrote:
>=20
> But "distribution Z includes X" kind of implies the existence of some s=
uch
> that X !=3D Y, Y !=3D empty-set and X+Y =3D Z, at least in common usage=
=2E
>=20
> Isn't that the same as a non-trunk change?
>=20
> So doesn't this mean that your question reduces to the question of what
> happens when non-Apache changes are made to an Apache release?  And isn=
't
> that the definition of a derived work?
>=20
>=20
>  Yup. Which is why I doubt *any* commercial entity can claim "includes =
Apache Hadoop" (including Cloudera).
>=20
>=20
>=20
> but they can claim it is a derivative work, which CDH clearly is,
> (Though if we were to come up with a formal declaration of what a
> derivative work is, we'd have to handle the fact that it is a superset.
> Even worse, you may realise a release is the ordered application of a
> sequence of patches, and if the patches are applied in a different orde=
r
> you may end up with a different body of source code...)
>=20
> Something that implements the APIs may not be a derivative work,
> depending on how much of the original code is in there. You could look
> at the base classes and interfaces and produce a clean room
> implementation (relying on the notion that interfaces are a list of
> facts and not copyrightable in the US), but whoever does that may
> encounter the issue that Google's donation of the right to use their MR
> patent may not apply to such implementations.
>=20
>=20
> The information contained in this communication may be CONFIDENTIAL and=
 is intended only for the use of the recipient(s) named above.  If you ar=
e not the intended recipient, you are hereby notified that any disseminat=
ion, distribution, or copying of this communication, or any of its conten=
ts, is strictly prohibited.  If you have received this communication in e=
rror, please notify the sender and delete/destroy the original message an=
d any copy of it from your computer or paper files.
>=20


The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.

From general-return-3560-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 01:07:09 2011
Return-Path: <general-return-3560-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 05DB66741
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 17 May 2011 01:07:09 +0000 (UTC)
Received: (qmail 5804 invoked by uid 500); 17 May 2011 01:07:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5733 invoked by uid 500); 17 May 2011 01:07:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5724 invoked by uid 99); 17 May 2011 01:07:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 01:07:07 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of charmalloc@allthingshadoop.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 01:07:02 +0000
Received: by iwr19 with SMTP id 19so29508iwr.35
        for <general@hadoop.apache.org>; Mon, 16 May 2011 18:06:41 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.141.20 with SMTP id k20mr21852ibu.132.1305594401650; Mon,
 16 May 2011 18:06:41 -0700 (PDT)
Received: by 10.231.14.141 with HTTP; Mon, 16 May 2011 18:06:41 -0700 (PDT)
X-Originating-IP: [72.88.211.41]
In-Reply-To: <BANLkTinxV5nexyuEiSt9fJ41907+QMrJyA@mail.gmail.com>
References: <BANLkTinDNbnhWgFwX-v=mwjHz5j+7oXUNg@mail.gmail.com>
	<BANLkTinxV5nexyuEiSt9fJ41907+QMrJyA@mail.gmail.com>
Date: Mon, 16 May 2011 21:06:41 -0400
Message-ID: <BANLkTimJFR1xM7o7G_FmQk=gLR3xJ4Gexw@mail.gmail.com>
Subject: Re: Apache Hadoop Hackathon: 5/18 in Palo Alto and San Francisco
From: Joe Stein <charmalloc@allthingshadoop.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6464fc687f71d04a36e62d8

--0016e6464fc687f71d04a36e62d8
Content-Type: text/plain; charset=ISO-8859-1

Any chance for something in the east (NYC) or do I need to start nagging the
wife and kids that west coast weather is the way to go?

I will post on the NYC HUG maybe we can get some Hack together contrib to
Hadoop.... but maybe some evening/day that a few commiters on this list that
are in NYC (not sure if there are any based in NYC?) they can run a mini
hackathon to help get folks in contrib mode? that would rock!

Maybe even some standard type of "Hack @ HUG" that every HUG can do with
their users (build, patch, build, deploy, test) if they want...

Thanks to all contributors and commiters for your hard work and continued
efforts/dedication.

On Mon, May 16, 2011 at 8:09 PM, Jeff Hammerbacher <hammer@cloudera.com>wrote:

> Hey,
>
> We've got a great group coming together again on Wednesday for an Apache
> Hadoop Hackathon in Palo Alto and San Francisco. Sign up at
> http://hadoophackathon.eventbrite.com.
>
> As a reminder, we'll have Nigel Daley, the release manager for 0.22,
> present
> in Palo Alto. If you have build and release or testing skills and would
> like
> to contribute to Apache Hadoop, your skills will be highly valued. Please
> consider coming out to meet Nigel and help out on that front!
>
> Also, there's a Hadoop User Group at Yahoo! in Sunnyvale on Wednesday
> evening: http://www.meetup.com/hadoop/events/16805258. There will be a
> large
> caravan leaving from the Palo Alto Hackathon to attend the HUG, so if you
> want to Hadoop all day, stop by Palo Alto and join us for the trip south.
>
> Regards,
> Jeff
>
> On Thu, May 12, 2011 at 2:40 PM, Jeff Hammerbacher <hammer@cloudera.com
> >wrote:
>
> > Hey,
> >
> > Thanks to everyone who came out for the Apache Hadoop Hackathon yesterday
> > in Palo Alto and San Francisco. We had 35 people sign up from a great
> cross
> > section of companies: Yahoo!, Cloudera, Facebook, Apple, Twitter,
> > Foursquare, AOL, Ngmoco, StumbleUpon, Trend Micro, Conviva, and more. We
> had
> > committers from the HBase, Hive, Pig, and Oozie projects ensuring their
> > projects work with the upcoming 0.22 release, and a number of folks got
> > their first patches up. The 0.22 branch is now being built by Jenkins and
> > we're in good shape to get a release candidate up soon.
> >
> > We had so much fun that we're going to do it again!
> >
> > Join us next Wednesday, May 18th, in either San Francisco or Palo Alto,
> and
> > help us continue the march to get feature development back on trunk.
> We'll
> > have a special guest in Palo Alto: Nigel Daley, Apache Hadoop PMC member
> and
> > the release manager for the 0.22 release. If you're a sysadmin or devops
> > person and want to help us build and test Apache Hadoop, this Hackathon
> will
> > be a great chance to learn about how it's done. Of course we'd also love
> to
> > have developers looking to get their first patches into Hadoop as well.
> >
> > Sign up at http://hadoophackathon.eventbrite.com and we'll see you
> > Wednesday.
> >
> > Regards,
> > Jeff
> >
> > p.s. if you're looking to prepare for the Hackathon, check out the issues
> > tagged "newbie" at
> >
> https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&jqlQuery=labels+%3D+newbie
> > .
> >
>



-- 

/*
Joe Stein
http://www.linkedin.com/in/charmalloc
Twitter: @allthingshadoop
*/

--0016e6464fc687f71d04a36e62d8--

From general-return-3561-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 01:12:28 2011
Return-Path: <general-return-3561-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D10F9697E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 17 May 2011 01:12:28 +0000 (UTC)
Received: (qmail 12183 invoked by uid 500); 17 May 2011 01:12:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12134 invoked by uid 500); 17 May 2011 01:12:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12126 invoked by uid 99); 17 May 2011 01:12:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 01:12:27 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of scott@richrelevance.com designates 64.78.17.19 as permitted sender)
Received: from [64.78.17.19] (HELO EXHUB018-4.exch018.msoutlookonline.net) (64.78.17.19)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 01:12:20 +0000
Received: from EXVMBX018-1.exch018.msoutlookonline.net ([64.78.17.47]) by
 EXHUB018-4.exch018.msoutlookonline.net ([64.78.17.19]) with mapi; Mon, 16 May
 2011 18:11:59 -0700
From: Scott Carey <scott@richrelevance.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: Matthew Foley <mattf@yahoo-inc.com>
Date: Mon, 16 May 2011 18:12:22 -0700
Subject: Re: Defining Hadoop Compatibility -revisiting-
Thread-Topic: Defining Hadoop Compatibility -revisiting-
Thread-Index: AcwUL2s7rmbwhQEvRESvHXsJsF+fKw==
Message-ID: <C9F71692.35BAB%scott@richrelevance.com>
In-Reply-To: <1EB061D5-C193-4FC2-85AA-A4C2C8302B8D@navteq.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-MacOutlook/14.10.0.110310
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

On trademarks, what about the phrase:  "New distribution for Apache
Hadoop"?  I've seen that used, and its something that replaces most of the
stack.  I believe "Apache Hadoop" is trademarked in this context, even if
Hadoop alone isn't.
"Compatible with Apache Hadoop" is a smaller issue, defining some rough
guidelines for various forms of compatibility is useful for the community
(and reputable vendors), abuse of that will at least become obvious.  But
"distribution for Apache Hadoop" (not too sure what 'for' means here)?  Is
there any TM protection?  A proprietary derivative work with most of the
guts replaced is not an Apache Hadoop distribution, nor a distribution for
Apache Hadoop.

On 5/16/11 5:40 PM, "Segel, Mike" <msegel@navteq.com> wrote:

>I just checked... TESS said no trademarks for Hadoop.
>So... what TM protection? :-)
>
>You are correct about derivative works. It's a moot point as long as the
>derivative work follows the T&Cs...
>
>
>
>Sent from a remote device. Please excuse any typos...
>
>Mike Segel
>
>On May 16, 2011, at 4:18 PM, "Matthew Foley" <mattf@yahoo-inc.com> wrote:
>
>> It's important to distinguish between the name "Hadoop", which is
>>protected by trademark law,
>> and the Hadoop implementation, which is licensed as opensource under
>>copyright law.
>>=20
>> The term "derivative work" is, I believe, only relevant under copyright
>>law, not trademark law.
>> (N.B., I'm not a lawyer -- and this email is my opinion, not my
>>employer's.)  Since the Apache License
>> explicitly allows derivative works, I don't think it's a useful term
>>for this discussion.
>>=20
>> However, the ASF, and by delegation the Hadoop PMC, has a lot of
>>control over the name,
>> and how we allow it to be used, under trademark law.  But to keeps our
>>rights under that
>> law, we have to enforce the trademark consistently.  So it's good that
>>we're having this discussion,
>> and it's important to reach a conclusion, document it, and enforce it
>>consistently.
>>=20
>> There are a lot of subtleties; for instance, if I recall correctly from
>>my days with Adobe and
>> PostScript(R), someone who has not licensed a trademark "X" can still
>>claim "compatible with X"
>> as long as they ALSO make clear that the product is NOT, itself, an
>>"X".  But you really need
>> a lawyer to get into that stuff.
>>=20
>> --Matt
>>=20
>>=20
>> On May 16, 2011, at 5:00 AM, Segel, Mike wrote:
>>=20
>> But Cloudera's release is a bit murky.
>>=20
>> The math example is a bit flawed...
>>=20
>> X represents the set of stable releases.
>> Y represents the set of available patches.
>> C represents the set of Cloudera releases.
>>=20
>> So if C contains a release X(n) plus a set of patches that is contained
>>in Y,
>> Then does it not have the right to be considered Apache Hadoop?
>> It's my understanding is that any enhancement to Hadoop is made
>>available to Apache and will eventually make it into a later release...
>>=20
>> So while it may not be 'official' release X(z), all of it's components
>>are in Apache.
>> (note: I'm talking about the core components and not Cloudera's
>>additional toolsets that encompass Hadoop.)
>>=20
>> Cloudera is clearly a derivative work.
>> And IMHO is the only one which can say ... 'Includes Apache Hadoop'.
>>=20
>> That doesn't mean that others can't, depending on how they implemented
>>their changes.
>> Based on EMC marketing material, they've done a rip and replace of HDFS.
>> So it wouldn't be a superset since it doesn't contain a complete
>>subset, but contains code that implements the API... So they can't say
>>'Includes Apache Hadoop',but they can say it's a derivative work based
>>on Apache Hadoop and then go on to show how and why, in their opinion
>>their product is better.(that's marketing for you...)
>>=20
>> Clearly there are others out there...
>> Hadoop on Cassandra as an example...
>>=20
>> Fragmentation of Hadoop will occur. It's inevitable. Too much money is
>>on the table...
>>=20
>> But because Apache's licensing is so open, Apache will have a hard time
>>controlling derivative works...
>> I believe that Steve is incorrect in his assertion concerning potential
>>loss of any patent protection. Again Apache's licensing is very open and
>>as long as they follow Apache's Ts and Cs, they are covered.
>>=20
>> Note: because I am sending this from my email address at my client, I
>>am obliged to say that this email is my opinion and does not reflect on
>>the opinion of my client...
>> (you know the rest....)
>>=20
>> Sent from a remote device. Please excuse any typos...
>>=20
>> Mike Segel
>>=20
>> On May 16, 2011, at 6:02 AM, "Steve Loughran"
>><stevel@apache.org<mailto:stevel@apache.org>> wrote:
>>=20
>> On 13/05/11 23:57, Allen Wittenauer wrote:
>>=20
>> On May 13, 2011, at 3:53 PM, Ted Dunning wrote:
>>=20
>> But "distribution Z includes X" kind of implies the existence of some
>>such
>> that X !=3D Y, Y !=3D empty-set and X+Y =3D Z, at least in common usage.
>>=20
>> Isn't that the same as a non-trunk change?
>>=20
>> So doesn't this mean that your question reduces to the question of what
>> happens when non-Apache changes are made to an Apache release?  And
>>isn't
>> that the definition of a derived work?
>>=20
>>=20
>>  Yup. Which is why I doubt *any* commercial entity can claim "includes
>>Apache Hadoop" (including Cloudera).
>>=20
>>=20
>>=20
>> but they can claim it is a derivative work, which CDH clearly is,
>> (Though if we were to come up with a formal declaration of what a
>> derivative work is, we'd have to handle the fact that it is a superset.
>> Even worse, you may realise a release is the ordered application of a
>> sequence of patches, and if the patches are applied in a different order
>> you may end up with a different body of source code...)
>>=20
>> Something that implements the APIs may not be a derivative work,
>> depending on how much of the original code is in there. You could look
>> at the base classes and interfaces and produce a clean room
>> implementation (relying on the notion that interfaces are a list of
>> facts and not copyrightable in the US), but whoever does that may
>> encounter the issue that Google's donation of the right to use their MR
>> patent may not apply to such implementations.
>>=20
>>=20
>> The information contained in this communication may be CONFIDENTIAL and
>>is intended only for the use of the recipient(s) named above.  If you
>>are not the intended recipient, you are hereby notified that any
>>dissemination, distribution, or copying of this communication, or any of
>>its contents, is strictly prohibited.  If you have received this
>>communication in error, please notify the sender and delete/destroy the
>>original message and any copy of it from your computer or paper files.
>>=20
>
>
>The information contained in this communication may be CONFIDENTIAL and
>is intended only for the use of the recipient(s) named above.  If you are
>not the intended recipient, you are hereby notified that any
>dissemination, distribution, or copying of this communication, or any of
>its contents, is strictly prohibited.  If you have received this
>communication in error, please notify the sender and delete/destroy the
>original message and any copy of it from your computer or paper files.


From general-return-3562-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 01:50:59 2011
Return-Path: <general-return-3562-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3E5836A00
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 17 May 2011 01:50:59 +0000 (UTC)
Received: (qmail 36944 invoked by uid 500); 17 May 2011 01:50:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36881 invoked by uid 500); 17 May 2011 01:50:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36873 invoked by uid 99); 17 May 2011 01:50:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 01:50:57 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of msegel@navteq.com designates 65.167.11.240 as permitted sender)
Received: from [65.167.11.240] (HELO xmailchi.navteq.com) (65.167.11.240)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 01:50:53 +0000
Received: from chicago-spamfw.navteq.com (imailchi [10.8.5.161])
	by xmailchi.navteq.com (8.13.8/8.13.6) with ESMTP id p4H1oWH4027188
	for <general@hadoop.apache.org>; Mon, 16 May 2011 20:50:32 -0500
Received: from imailchi.hq.navteq.com (localhost [127.0.0.1])
	by chicago-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP
	id 382B13DE90C1; Mon, 16 May 2011 20:50:32 -0500 (CDT)
Received: from imailchi.hq.navteq.com ([10.8.120.104]) by chicago-spamfw.navteq.com with ESMTP id TubBt7ggt2Ro6nuG; Mon, 16 May 2011 20:50:32 -0500 (CDT)
Received: from hq-ex-ht02.ad.navteq.com (hq-ex-ht02.ad.navteq.com [10.8.222.172])
	by imailchi.hq.navteq.com (8.12.9/8.12.9) with ESMTP id p4H1oWNe024968;
	Mon, 16 May 2011 20:50:32 -0500
Received: from hq-ex-mb03.ad.navteq.com ([fe80::c4dd:7b21:5c22:cfe4]) by
 hq-ex-ht02.ad.navteq.com ([fe80::24f6:ca50:734d:df6e%13]) with mapi; Mon, 16
 May 2011 20:50:31 -0500
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>,
        Matthew Foley
	<mattf@yahoo-inc.com>
Date: Mon, 16 May 2011 20:50:47 -0500
Subject: Re: Defining Hadoop Compatibility -revisiting-
Thread-Topic: Defining Hadoop Compatibility -revisiting-
Thread-Index: AcwUNM2FOGATVu6GQTaVLC2hgNCHzw==
Message-ID: <938B2F6A-CC9F-41C3-8EC7-E5FD048AEA26@navteq.com>
References: <C9F71692.35BAB%scott@richrelevance.com>
In-Reply-To: <C9F71692.35BAB%scott@richrelevance.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1

Let me clarify...
I searched on Hadoop as a term in any TM.=20
Nothing came back...

This means that Apache Hadoop didn't show up.

Note the following: I did the basic search. I wouldn't be surprised that =
someone from Apache comes back and says see TM xxxxxxxx ...

-Mike

Sent from a remote device. Please excuse any typos...

Mike Segel

On May 16, 2011, at 8:12 PM, Scott Carey <scott@richrelevance.com> wrote:

> On trademarks, what about the phrase:  "New distribution for Apache
> Hadoop"?  I've seen that used, and its something that replaces most of =
the
> stack.  I believe "Apache Hadoop" is trademarked in this context, even =
if
> Hadoop alone isn't.
> "Compatible with Apache Hadoop" is a smaller issue, defining some rough
> guidelines for various forms of compatibility is useful for the communi=
ty
> (and reputable vendors), abuse of that will at least become obvious.  B=
ut
> "distribution for Apache Hadoop" (not too sure what 'for' means here)? =
 Is
> there any TM protection?  A proprietary derivative work with most of th=
e
> guts replaced is not an Apache Hadoop distribution, nor a distribution =
for
> Apache Hadoop.
>=20
> On 5/16/11 5:40 PM, "Segel, Mike" <msegel@navteq.com> wrote:
>=20
>> I just checked... TESS said no trademarks for Hadoop.
>> So... what TM protection? :-)
>>=20
>> You are correct about derivative works. It's a moot point as long as t=
he
>> derivative work follows the T&Cs...
>>=20
>>=20
>>=20
>> Sent from a remote device. Please excuse any typos...
>>=20
>> Mike Segel
>>=20
>> On May 16, 2011, at 4:18 PM, "Matthew Foley" <mattf@yahoo-inc.com> wro=
te:
>>=20
>>> It's important to distinguish between the name "Hadoop", which is
>>> protected by trademark law,
>>> and the Hadoop implementation, which is licensed as opensource under
>>> copyright law.
>>>=20
>>> The term "derivative work" is, I believe, only relevant under copyrig=
ht
>>> law, not trademark law.
>>> (N.B., I'm not a lawyer -- and this email is my opinion, not my
>>> employer's.)  Since the Apache License
>>> explicitly allows derivative works, I don't think it's a useful term
>>> for this discussion.
>>>=20
>>> However, the ASF, and by delegation the Hadoop PMC, has a lot of
>>> control over the name,
>>> and how we allow it to be used, under trademark law.  But to keeps ou=
r
>>> rights under that
>>> law, we have to enforce the trademark consistently.  So it's good tha=
t
>>> we're having this discussion,
>>> and it's important to reach a conclusion, document it, and enforce it
>>> consistently.
>>>=20
>>> There are a lot of subtleties; for instance, if I recall correctly fr=
om
>>> my days with Adobe and
>>> PostScript(R), someone who has not licensed a trademark "X" can still
>>> claim "compatible with X"
>>> as long as they ALSO make clear that the product is NOT, itself, an
>>> "X".  But you really need
>>> a lawyer to get into that stuff.
>>>=20
>>> --Matt
>>>=20
>>>=20
>>> On May 16, 2011, at 5:00 AM, Segel, Mike wrote:
>>>=20
>>> But Cloudera's release is a bit murky.
>>>=20
>>> The math example is a bit flawed...
>>>=20
>>> X represents the set of stable releases.
>>> Y represents the set of available patches.
>>> C represents the set of Cloudera releases.
>>>=20
>>> So if C contains a release X(n) plus a set of patches that is contain=
ed
>>> in Y,
>>> Then does it not have the right to be considered Apache Hadoop?
>>> It's my understanding is that any enhancement to Hadoop is made
>>> available to Apache and will eventually make it into a later release.=
=2E.
>>>=20
>>> So while it may not be 'official' release X(z), all of it's component=
s
>>> are in Apache.
>>> (note: I'm talking about the core components and not Cloudera's
>>> additional toolsets that encompass Hadoop.)
>>>=20
>>> Cloudera is clearly a derivative work.
>>> And IMHO is the only one which can say ... 'Includes Apache Hadoop'.
>>>=20
>>> That doesn't mean that others can't, depending on how they implemente=
d
>>> their changes.
>>> Based on EMC marketing material, they've done a rip and replace of HD=
FS.
>>> So it wouldn't be a superset since it doesn't contain a complete
>>> subset, but contains code that implements the API... So they can't sa=
y
>>> 'Includes Apache Hadoop',but they can say it's a derivative work base=
d
>>> on Apache Hadoop and then go on to show how and why, in their opinion
>>> their product is better.(that's marketing for you...)
>>>=20
>>> Clearly there are others out there...
>>> Hadoop on Cassandra as an example...
>>>=20
>>> Fragmentation of Hadoop will occur. It's inevitable. Too much money i=
s
>>> on the table...
>>>=20
>>> But because Apache's licensing is so open, Apache will have a hard ti=
me
>>> controlling derivative works...
>>> I believe that Steve is incorrect in his assertion concerning potenti=
al
>>> loss of any patent protection. Again Apache's licensing is very open =
and
>>> as long as they follow Apache's Ts and Cs, they are covered.
>>>=20
>>> Note: because I am sending this from my email address at my client, I
>>> am obliged to say that this email is my opinion and does not reflect =
on
>>> the opinion of my client...
>>> (you know the rest....)
>>>=20
>>> Sent from a remote device. Please excuse any typos...
>>>=20
>>> Mike Segel
>>>=20
>>> On May 16, 2011, at 6:02 AM, "Steve Loughran"
>>> <stevel@apache.org<mailto:stevel@apache.org>> wrote:
>>>=20
>>> On 13/05/11 23:57, Allen Wittenauer wrote:
>>>=20
>>> On May 13, 2011, at 3:53 PM, Ted Dunning wrote:
>>>=20
>>> But "distribution Z includes X" kind of implies the existence of some
>>> such
>>> that X !=3D Y, Y !=3D empty-set and X+Y =3D Z, at least in common usa=
ge.
>>>=20
>>> Isn't that the same as a non-trunk change?
>>>=20
>>> So doesn't this mean that your question reduces to the question of wh=
at
>>> happens when non-Apache changes are made to an Apache release?  And
>>> isn't
>>> that the definition of a derived work?
>>>=20
>>>=20
>>> Yup. Which is why I doubt *any* commercial entity can claim "includes
>>> Apache Hadoop" (including Cloudera).
>>>=20
>>>=20
>>>=20
>>> but they can claim it is a derivative work, which CDH clearly is,
>>> (Though if we were to come up with a formal declaration of what a
>>> derivative work is, we'd have to handle the fact that it is a superse=
t.
>>> Even worse, you may realise a release is the ordered application of a
>>> sequence of patches, and if the patches are applied in a different or=
der
>>> you may end up with a different body of source code...)
>>>=20
>>> Something that implements the APIs may not be a derivative work,
>>> depending on how much of the original code is in there. You could loo=
k
>>> at the base classes and interfaces and produce a clean room
>>> implementation (relying on the notion that interfaces are a list of
>>> facts and not copyrightable in the US), but whoever does that may
>>> encounter the issue that Google's donation of the right to use their =
MR
>>> patent may not apply to such implementations.
>>>=20
>>>=20
>>> The information contained in this communication may be CONFIDENTIAL a=
nd
>>> is intended only for the use of the recipient(s) named above.  If you
>>> are not the intended recipient, you are hereby notified that any
>>> dissemination, distribution, or copying of this communication, or any=
 of
>>> its contents, is strictly prohibited.  If you have received this
>>> communication in error, please notify the sender and delete/destroy t=
he
>>> original message and any copy of it from your computer or paper files=
=2E
>>>=20
>>=20
>>=20
>> The information contained in this communication may be CONFIDENTIAL an=
d
>> is intended only for the use of the recipient(s) named above.  If you =
are
>> not the intended recipient, you are hereby notified that any
>> dissemination, distribution, or copying of this communication, or any =
of
>> its contents, is strictly prohibited.  If you have received this
>> communication in error, please notify the sender and delete/destroy th=
e
>> original message and any copy of it from your computer or paper files.
>=20


The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.

From general-return-3563-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 01:53:06 2011
Return-Path: <general-return-3563-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 94BCA6A4F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 17 May 2011 01:53:06 +0000 (UTC)
Received: (qmail 40063 invoked by uid 500); 17 May 2011 01:53:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39998 invoked by uid 500); 17 May 2011 01:53:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39989 invoked by uid 99); 17 May 2011 01:53:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 01:53:05 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 01:52:59 +0000
Received: by pve37 with SMTP id 37so34058pve.35
        for <general@hadoop.apache.org>; Mon, 16 May 2011 18:52:37 -0700 (PDT)
Received: by 10.68.3.139 with SMTP id c11mr151693pbc.277.1305597157063; Mon,
 16 May 2011 18:52:37 -0700 (PDT)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.68.62.230 with HTTP; Mon, 16 May 2011 18:52:17 -0700 (PDT)
In-Reply-To: <B1A21BA0-69A9-45B6-AC18-E374DCA7B424@Holsman.NET>
References: <C9F227D5.8FA4%mbhandarkar@linkedin.com> <4DCCF172.5010504@apache.org>
 <99AB77CF-025E-4E4A-BD7F-BDA362E633BF@apache.org> <4DCDA8E7.5090809@apache.org>
 <2ED3A7F5-E549-4545-9395-094A10EA4EED@apache.org> <BANLkTik=EkwPxvcw2-vw=hsofsJYRgyaGg@mail.gmail.com>
 <BEC4433A-2270-47CF-960D-CB416B20FD4A@apache.org> <4DD10420.7060004@apache.org>
 <A4FB782E-2433-47F1-AE91-CEFC42654C3A@navteq.com> <434E417B-C21F-4649-9D72-4E6DCA558C18@apache.org>
 <BANLkTinXg7twM7KUrKpHpTycyaxnPmnEfg@mail.gmail.com> <D972DAFB-26F5-4285-8D10-48C2BB2A5A4E@apache.org>
 <BANLkTi=E9u67P201cAC_F5XAkHi4hxmefQ@mail.gmail.com> <6E1C9C7A-646A-44A2-9EB0-A11B7291B7B2@linkedin.com>
 <B1A21BA0-69A9-45B6-AC18-E374DCA7B424@Holsman.NET>
From: Konstantin Boudnik <cos@apache.org>
Date: Mon, 16 May 2011 18:52:17 -0700
X-Google-Sender-Auth: uFckeOqvaYXIhblhlWaG3O7N2ZQ
Message-ID: <BANLkTik12zJD4L8XBfmvf28jXLBYC6=8yA@mail.gmail.com>
Subject: Re: Defining Hadoop Compatibility -revisiting-
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

We have the following method coverage:
Common ~60%
HDFS      ~80%
MR          ~70%
(better analysis will be available after our projects are connected to
Sonar, I think).

While method coverage isn't completely adequate answer to your
question, I'd say there is a possibility to sneak in some semantical
and even API changes which might go entirely unvalidated by the test
suites. It isn't very high, but it does exist.

A better approach to validate semantics is to run cluster tests (e.g.
system tests) which have a better potentials to exercise public APIs
than functional tests. There's HADOOP-7278 to address this for 0.22
(and potentially others)
--
=A0 Take care,
Konstantin (Cos) Boudnik

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.

On Mon, May 16, 2011 at 14:59, Ian Holsman <hadoop@holsman.net> wrote:
>
>>
>> =A0 =A0 =A0 Does "Hadoop compatibility" and the ability to say "includes=
 Apache Hadoop" only apply when we're talking about MR and HDFS APIs?
>
>
> It is confusing isn't it.
>
> We could go down the route java did and say that the API's are 'hadoop' a=
nd ours is just a reference implementation of it. (but others pointed out, =
we don't want to become a certification group)
>
> Out of curiosity, how good is our test suite in exercising our APIs?
> Is it sophisticated enough to capture someone adding a functionality-chan=
ging patch (eg the append one). and have it flag it as a test-failure?
>
>

From general-return-3564-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 02:33:29 2011
Return-Path: <general-return-3564-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 248804322
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 17 May 2011 02:33:29 +0000 (UTC)
Received: (qmail 66565 invoked by uid 500); 17 May 2011 02:33:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66506 invoked by uid 500); 17 May 2011 02:33:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66498 invoked by uid 99); 17 May 2011 02:33:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 02:33:27 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 02:33:21 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p4H2Wmkf067966;
	Mon, 16 May 2011 19:32:48 -0700 (PDT)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by
 SP2-EX07CAS01.ds.corp.yahoo.com ([98.137.59.37]) with mapi; Mon, 16 May 2011
 19:32:48 -0700
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>,
        Matthew Foley
	<mattf@yahoo-inc.com>
Date: Mon, 16 May 2011 19:32:36 -0700
Subject: Re: Defining Hadoop Compatibility -revisiting-
Thread-Topic: Defining Hadoop Compatibility -revisiting-
Thread-Index: AcwUOrWd4eLrbM55Qmmy70FDIxobXQ==
Message-ID: <3CF0B8B5-2146-4E26-9F44-C012A5DFB7A5@yahoo-inc.com>
References: <C9F71692.35BAB%scott@richrelevance.com>
 <938B2F6A-CC9F-41C3-8EC7-E5FD048AEA26@navteq.com>
In-Reply-To: <938B2F6A-CC9F-41C3-8EC7-E5FD048AEA26@navteq.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

My understanding is that a history if defending your trade mark is more imp=
ortant than registration. Apache does defend Hadoop.=20

---
E14 - typing on glass

On May 16, 2011, at 6:52 PM, "Segel, Mike" <msegel@navteq.com> wrote:

> Let me clarify...
> I searched on Hadoop as a term in any TM.=20
> Nothing came back...
>=20
> This means that Apache Hadoop didn't show up.
>=20
> Note the following: I did the basic search. I wouldn't be surprised that =
someone from Apache comes back and says see TM xxxxxxxx ...
>=20
> -Mike
>=20
> Sent from a remote device. Please excuse any typos...
>=20
> Mike Segel
>=20
> On May 16, 2011, at 8:12 PM, Scott Carey <scott@richrelevance.com> wrote:
>=20
>> On trademarks, what about the phrase:  "New distribution for Apache
>> Hadoop"?  I've seen that used, and its something that replaces most of t=
he
>> stack.  I believe "Apache Hadoop" is trademarked in this context, even i=
f
>> Hadoop alone isn't.
>> "Compatible with Apache Hadoop" is a smaller issue, defining some rough
>> guidelines for various forms of compatibility is useful for the communit=
y
>> (and reputable vendors), abuse of that will at least become obvious.  Bu=
t
>> "distribution for Apache Hadoop" (not too sure what 'for' means here)?  =
Is
>> there any TM protection?  A proprietary derivative work with most of the
>> guts replaced is not an Apache Hadoop distribution, nor a distribution f=
or
>> Apache Hadoop.
>>=20
>> On 5/16/11 5:40 PM, "Segel, Mike" <msegel@navteq.com> wrote:
>>=20
>>> I just checked... TESS said no trademarks for Hadoop.
>>> So... what TM protection? :-)
>>>=20
>>> You are correct about derivative works. It's a moot point as long as th=
e
>>> derivative work follows the T&Cs...
>>>=20
>>>=20
>>>=20
>>> Sent from a remote device. Please excuse any typos...
>>>=20
>>> Mike Segel
>>>=20
>>> On May 16, 2011, at 4:18 PM, "Matthew Foley" <mattf@yahoo-inc.com> wrot=
e:
>>>=20
>>>> It's important to distinguish between the name "Hadoop", which is
>>>> protected by trademark law,
>>>> and the Hadoop implementation, which is licensed as opensource under
>>>> copyright law.
>>>>=20
>>>> The term "derivative work" is, I believe, only relevant under copyrigh=
t
>>>> law, not trademark law.
>>>> (N.B., I'm not a lawyer -- and this email is my opinion, not my
>>>> employer's.)  Since the Apache License
>>>> explicitly allows derivative works, I don't think it's a useful term
>>>> for this discussion.
>>>>=20
>>>> However, the ASF, and by delegation the Hadoop PMC, has a lot of
>>>> control over the name,
>>>> and how we allow it to be used, under trademark law.  But to keeps our
>>>> rights under that
>>>> law, we have to enforce the trademark consistently.  So it's good that
>>>> we're having this discussion,
>>>> and it's important to reach a conclusion, document it, and enforce it
>>>> consistently.
>>>>=20
>>>> There are a lot of subtleties; for instance, if I recall correctly fro=
m
>>>> my days with Adobe and
>>>> PostScript(R), someone who has not licensed a trademark "X" can still
>>>> claim "compatible with X"
>>>> as long as they ALSO make clear that the product is NOT, itself, an
>>>> "X".  But you really need
>>>> a lawyer to get into that stuff.
>>>>=20
>>>> --Matt
>>>>=20
>>>>=20
>>>> On May 16, 2011, at 5:00 AM, Segel, Mike wrote:
>>>>=20
>>>> But Cloudera's release is a bit murky.
>>>>=20
>>>> The math example is a bit flawed...
>>>>=20
>>>> X represents the set of stable releases.
>>>> Y represents the set of available patches.
>>>> C represents the set of Cloudera releases.
>>>>=20
>>>> So if C contains a release X(n) plus a set of patches that is containe=
d
>>>> in Y,
>>>> Then does it not have the right to be considered Apache Hadoop?
>>>> It's my understanding is that any enhancement to Hadoop is made
>>>> available to Apache and will eventually make it into a later release..=
.
>>>>=20
>>>> So while it may not be 'official' release X(z), all of it's components
>>>> are in Apache.
>>>> (note: I'm talking about the core components and not Cloudera's
>>>> additional toolsets that encompass Hadoop.)
>>>>=20
>>>> Cloudera is clearly a derivative work.
>>>> And IMHO is the only one which can say ... 'Includes Apache Hadoop'.
>>>>=20
>>>> That doesn't mean that others can't, depending on how they implemented
>>>> their changes.
>>>> Based on EMC marketing material, they've done a rip and replace of HDF=
S.
>>>> So it wouldn't be a superset since it doesn't contain a complete
>>>> subset, but contains code that implements the API... So they can't say
>>>> 'Includes Apache Hadoop',but they can say it's a derivative work based
>>>> on Apache Hadoop and then go on to show how and why, in their opinion
>>>> their product is better.(that's marketing for you...)
>>>>=20
>>>> Clearly there are others out there...
>>>> Hadoop on Cassandra as an example...
>>>>=20
>>>> Fragmentation of Hadoop will occur. It's inevitable. Too much money is
>>>> on the table...
>>>>=20
>>>> But because Apache's licensing is so open, Apache will have a hard tim=
e
>>>> controlling derivative works...
>>>> I believe that Steve is incorrect in his assertion concerning potentia=
l
>>>> loss of any patent protection. Again Apache's licensing is very open a=
nd
>>>> as long as they follow Apache's Ts and Cs, they are covered.
>>>>=20
>>>> Note: because I am sending this from my email address at my client, I
>>>> am obliged to say that this email is my opinion and does not reflect o=
n
>>>> the opinion of my client...
>>>> (you know the rest....)
>>>>=20
>>>> Sent from a remote device. Please excuse any typos...
>>>>=20
>>>> Mike Segel
>>>>=20
>>>> On May 16, 2011, at 6:02 AM, "Steve Loughran"
>>>> <stevel@apache.org<mailto:stevel@apache.org>> wrote:
>>>>=20
>>>> On 13/05/11 23:57, Allen Wittenauer wrote:
>>>>=20
>>>> On May 13, 2011, at 3:53 PM, Ted Dunning wrote:
>>>>=20
>>>> But "distribution Z includes X" kind of implies the existence of some
>>>> such
>>>> that X !=3D Y, Y !=3D empty-set and X+Y =3D Z, at least in common usag=
e.
>>>>=20
>>>> Isn't that the same as a non-trunk change?
>>>>=20
>>>> So doesn't this mean that your question reduces to the question of wha=
t
>>>> happens when non-Apache changes are made to an Apache release?  And
>>>> isn't
>>>> that the definition of a derived work?
>>>>=20
>>>>=20
>>>> Yup. Which is why I doubt *any* commercial entity can claim "includes
>>>> Apache Hadoop" (including Cloudera).
>>>>=20
>>>>=20
>>>>=20
>>>> but they can claim it is a derivative work, which CDH clearly is,
>>>> (Though if we were to come up with a formal declaration of what a
>>>> derivative work is, we'd have to handle the fact that it is a superset=
.
>>>> Even worse, you may realise a release is the ordered application of a
>>>> sequence of patches, and if the patches are applied in a different ord=
er
>>>> you may end up with a different body of source code...)
>>>>=20
>>>> Something that implements the APIs may not be a derivative work,
>>>> depending on how much of the original code is in there. You could look
>>>> at the base classes and interfaces and produce a clean room
>>>> implementation (relying on the notion that interfaces are a list of
>>>> facts and not copyrightable in the US), but whoever does that may
>>>> encounter the issue that Google's donation of the right to use their M=
R
>>>> patent may not apply to such implementations.
>>>>=20
>>>>=20
>>>> The information contained in this communication may be CONFIDENTIAL an=
d
>>>> is intended only for the use of the recipient(s) named above.  If you
>>>> are not the intended recipient, you are hereby notified that any
>>>> dissemination, distribution, or copying of this communication, or any =
of
>>>> its contents, is strictly prohibited.  If you have received this
>>>> communication in error, please notify the sender and delete/destroy th=
e
>>>> original message and any copy of it from your computer or paper files.
>>>>=20
>>>=20
>>>=20
>>> The information contained in this communication may be CONFIDENTIAL and
>>> is intended only for the use of the recipient(s) named above.  If you a=
re
>>> not the intended recipient, you are hereby notified that any
>>> dissemination, distribution, or copying of this communication, or any o=
f
>>> its contents, is strictly prohibited.  If you have received this
>>> communication in error, please notify the sender and delete/destroy the
>>> original message and any copy of it from your computer or paper files.
>>=20
>=20
>=20
> The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are no=
t the intended recipient, you are hereby notified that any dissemination, d=
istribution, or copying of this communication, or any of its contents, is s=
trictly prohibited.  If you have received this communication in error, plea=
se notify the sender and delete/destroy the original message and any copy o=
f it from your computer or paper files.

From general-return-3565-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 02:53:04 2011
Return-Path: <general-return-3565-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E43084EAE
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 17 May 2011 02:53:04 +0000 (UTC)
Received: (qmail 74896 invoked by uid 500); 17 May 2011 02:53:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74842 invoked by uid 500); 17 May 2011 02:53:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74834 invoked by uid 99); 17 May 2011 02:53:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 02:53:02 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [98.139.53.216] (HELO nm21-vm0.bullet.mail.ac4.yahoo.com) (98.139.53.216)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 17 May 2011 02:52:54 +0000
Received: from [98.139.52.190] by nm21.bullet.mail.ac4.yahoo.com with NNFMP; 17 May 2011 02:52:33 -0000
Received: from [98.139.52.159] by tm3.bullet.mail.ac4.yahoo.com with NNFMP; 17 May 2011 02:52:33 -0000
Received: from [127.0.0.1] by omp1042.mail.ac4.yahoo.com with NNFMP; 17 May 2011 02:52:33 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 660526.74064.bm@omp1042.mail.ac4.yahoo.com
Received: (qmail 8002 invoked by uid 60001); 17 May 2011 02:52:33 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1305600753; bh=aD0X+/2RFVpJLGMKwWeftiP4tjcj7KxhGUQilTpVxl4=; h=Message-ID:X-YMail-OSG:Received:X-RocketYMMF:X-Mailer:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=5bsN2VDFoe1Okx56ITtx/2ZWV0EcrynHcLLxFwFhQGM5FsJfwbMT9JY5x1u14qicLnkkvYl4x+2J1axSh9LrQywmn3ELux7Hu8sOSMTnaxcZiIaMb93Nmlw1UjkEeqff2pRLAg4hga23db+eqh7vwlJJoh6O2cpFxjbxN3mYtcs=
Message-ID: <477039.7995.qm@web65506.mail.ac4.yahoo.com>
X-YMail-OSG: 4sYWDZsVM1nvSRHQUrWvA_vzA2mEJpHbuhi9pMrEpZpY1xS
 SOcwHSgqObYpmlPMcSV5vhkn.8D4JusGgSvb0HSSPI1SFv19PGMx47NFwMFA
 .2so1I.JnRDbrYOqdkw5L3nzYziiERi8IhU6XoX8iATCcukE2GQkinVQc26F
 fiTtRcd.DyixX7vz3xkjoTtn7GsjzXi6VWvFXku13twVgqHQ3WMiVVwCxY.N
 DMvB.Vp68Ztn0nKM3p.zGafcxfacP5L8rTvtRrpI39W09MXSUXLSOssyvflJ
 QT9G59N5OkhHGhTaWj4C0G8ULRekZhFX0e_gxJ5twegORgrOBVgZRqTk5Ah4
 aEyP1CoGsqfXnXnZShPSY08d0BVLqrSCpBJXiAF8FuokH4UA07tmoWrVEROV
 vyW1ZBudWsc9ZhQ--
Received: from [220.135.41.196] by web65506.mail.ac4.yahoo.com via HTTP; Mon, 16 May 2011 19:52:33 PDT
X-RocketYMMF: apurtell
X-Mailer: YahooMailClassic/12.0.2 YahooMailWebService/0.8.111.303096
Date: Mon, 16 May 2011 19:52:33 -0700 (PDT)
From: Andrew Purtell <apurtell@apache.org>
Reply-To: apurtell@apache.org
Subject: Re: Defining Hadoop Compatibility -revisiting-
To: general@hadoop.apache.org
Cc: Matthew Foley <mattf@yahoo-inc.com>
In-Reply-To: <C9F71692.35BAB%scott@richrelevance.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

> On trademarks, what about the phrase:  "New distribution for Apache=0A> H=
adoop"?  I've seen that used, and its something that=0A> replaces most of t=
he stack. [...] A proprietary derivative work with=0A> most of the guts rep=
laced is not an Apache Hadoop distribution, nor=0A> a distribution for Apac=
he Hadoop.=0A=0AIMHO, this is the key issue. Allowing proprietary derivativ=
e works that provide Hadoop compatible APIs to claim they are Hadoop will p=
rovoke endless confusion, argument, claim, and counter-claim, and poison th=
e well for all involved with Apache Hadoop.=0A=0ABest regards,=0A=0A    - A=
ndy=0A=0AProblems worthy of attack prove their worth by hitting back. - Pie=
t Hein (via Tom White)=0A=0A=0A--- On Mon, 5/16/11, Scott Carey <scott@rich=
relevance.com> wrote:=0A=0A> From: Scott Carey <scott@richrelevance.com>=0A=
> Subject: Re: Defining Hadoop Compatibility -revisiting-=0A> To: "general@=
hadoop.apache.org" <general@hadoop.apache.org>=0A> Cc: "Matthew Foley" <mat=
tf@yahoo-inc.com>=0A> Date: Monday, May 16, 2011, 6:12 PM=0A> On trademarks=
, what about the phrase:=A0 "New distribution for Apache=0A> Hadoop"?=A0 I'=
ve seen that used, and its something that replaces most=0A> of the stack.=
=A0 I believe "Apache Hadoop" is trademarked in this=0A> context, even if H=
adoop alone isn't. "Compatible with Apache Hadoop"=0A> is a smaller issue, =
defining some rough guidelines for various forms=0A> of compatibility is us=
eful for the community (and reputable vendors),=0A> abuse of that will at l=
east become obvious.=A0 But "distribution for=0A> Apache Hadoop" (not too s=
ure what 'for' means here)?=A0 Is there any=0A> TM protection?=A0 A proprie=
tary derivative work with most of the=0A> guts replaced is not an Apache Ha=
doop distribution, nor a=0A> distribution for Apache Hadoop.=0A> =0A> On 5/=
16/11 5:40 PM, "Segel, Mike" <msegel@navteq.com>=0A> wrote:=0A> =0A> >I jus=
t checked... TESS said no trademarks for Hadoop.=0A> >So... what TM protect=
ion? :-)=0A> >=0A> >You are correct about derivative works. It's a moot=0A>=
 point as long as the=0A> >derivative work follows the T&Cs...=0A> >=0A> >=
=0A> >=0A> >Sent from a remote device. Please excuse any typos...=0A> >=0A>=
 >Mike Segel=0A> >=0A> >On May 16, 2011, at 4:18 PM, "Matthew Foley" <mattf=
@yahoo-inc.com>=0A> wrote:=0A> >=0A> >> It's important to distinguish betwe=
en the name=0A> "Hadoop", which is=0A> >>protected by trademark law,=0A> >>=
 and the Hadoop implementation, which is licensed=0A> as opensource under=
=0A> >>copyright law.=0A> >> =0A> >> The term "derivative work" is, I belie=
ve, only=0A> relevant under copyright=0A> >>law, not trademark law.=0A> >> =
(N.B., I'm not a lawyer -- and this email is my=0A> opinion, not my=0A> >>e=
mployer's.)=A0 Since the Apache License=0A> >> explicitly allows derivative=
 works, I don't think=0A> it's a useful term=0A> >>for this discussion.=0A>=
 >> =0A> >> However, the ASF, and by delegation the Hadoop=0A> PMC, has a l=
ot of=0A> >>control over the name,=0A> >> and how we allow it to be used, u=
nder trademark=0A> law.=A0 But to keeps our=0A> >>rights under that=0A> >> =
law, we have to enforce the trademark=0A> consistently.=A0 So it's good tha=
t=0A> >>we're having this discussion,=0A> >> and it's important to reach a =
conclusion, document=0A> it, and enforce it=0A> >>consistently.=0A> >> =0A>=
 >> There are a lot of subtleties; for instance, if I=0A> recall correctly =
from=0A> >>my days with Adobe and=0A> >> PostScript(R), someone who has not=
 licensed a=0A> trademark "X" can still=0A> >>claim "compatible with X"=0A>=
 >> as long as they ALSO make clear that the product=0A> is NOT, itself, an=
=0A> >>"X".=A0 But you really need=0A> >> a lawyer to get into that stuff.=
=0A> >> =0A> >> --Matt=0A> >> =0A> >> =0A> >> On May 16, 2011, at 5:00 AM, =
Segel, Mike wrote:=0A> >> =0A> >> But Cloudera's release is a bit murky.=0A=
> >> =0A> >> The math example is a bit flawed...=0A> >> =0A> >> X represent=
s the set of stable releases.=0A> >> Y represents the set of available patc=
hes.=0A> >> C represents the set of Cloudera releases.=0A> >> =0A> >> So if=
 C contains a release X(n) plus a set of=0A> patches that is contained=0A> =
>>in Y,=0A> >> Then does it not have the right to be considered=0A> Apache =
Hadoop?=0A> >> It's my understanding is that any enhancement to=0A> Hadoop =
is made=0A> >>available to Apache and will eventually make it=0A> into a la=
ter release...=0A> >> =0A> >> So while it may not be 'official' release X(z=
),=0A> all of it's components=0A> >>are in Apache.=0A> >> (note: I'm talkin=
g about the core components and=0A> not Cloudera's=0A> >>additional toolset=
s that encompass Hadoop.)=0A> >> =0A> >> Cloudera is clearly a derivative w=
ork.=0A> >> And IMHO is the only one which can say ...=0A> 'Includes Apache=
 Hadoop'.=0A> >> =0A> >> That doesn't mean that others can't, depending on=
=0A> how they implemented=0A> >>their changes.=0A> >> Based on EMC marketin=
g material, they've done a=0A> rip and replace of HDFS.=0A> >> So it wouldn=
't be a superset since it doesn't=0A> contain a complete=0A> >>subset, but =
contains code that implements the=0A> API... So they can't say=0A> >>'Inclu=
des Apache Hadoop',but they can say it's a=0A> derivative work based=0A> >>=
on Apache Hadoop and then go on to show how and=0A> why, in their opinion=
=0A> >>their product is better.(that's marketing for=0A> you...)=0A> >> =0A=
> >> Clearly there are others out there...=0A> >> Hadoop on Cassandra as an=
 example...=0A> >> =0A> >> Fragmentation of Hadoop will occur. It's=0A> ine=
vitable. Too much money is=0A> >>on the table...=0A> >> =0A> >> But because=
 Apache's licensing is so open, Apache=0A> will have a hard time=0A> >>cont=
rolling derivative works...=0A> >> I believe that Steve is incorrect in his=
 assertion=0A> concerning potential=0A> >>loss of any patent protection. Ag=
ain Apache's=0A> licensing is very open and=0A> >>as long as they follow Ap=
ache's Ts and Cs, they are=0A> covered.=0A> >> =0A> >> Note: because I am s=
ending this from my email=0A> address at my client, I=0A> >>am obliged to s=
ay that this email is my opinion and=0A> does not reflect on=0A> >>the opin=
ion of my client...=0A> >> (you know the rest....)=0A> >> =0A> >> Sent from=
 a remote device. Please excuse any=0A> typos...=0A> >> =0A> >> Mike Segel=
=0A> >> =0A> >> On May 16, 2011, at 6:02 AM, "Steve Loughran"=0A> >><stevel=
@apache.org<mailto:stevel@apache.org>>=0A> wrote:=0A> >> =0A> >> On 13/05/1=
1 23:57, Allen Wittenauer wrote:=0A> >> =0A> >> On May 13, 2011, at 3:53 PM=
, Ted Dunning wrote:=0A> >> =0A> >> But "distribution Z includes X" kind of=
 implies=0A> the existence of some=0A> >>such=0A> >> that X !=3D Y, Y !=3D =
empty-set and X+Y =3D Z, at least=0A> in common usage.=0A> >> =0A> >> Isn't=
 that the same as a non-trunk change?=0A> >> =0A> >> So doesn't this mean t=
hat your question reduces to=0A> the question of what=0A> >> happens when n=
on-Apache changes are made to an=0A> Apache release?=A0 And=0A> >>isn't=0A>=
 >> that the definition of a derived work?=0A> >> =0A> >> =0A> >>=A0 Yup. W=
hich is why I doubt *any* commercial=0A> entity can claim "includes=0A> >>A=
pache Hadoop" (including Cloudera).=0A> >> =0A> >> =0A> >> =0A> >> but they=
 can claim it is a derivative work, which=0A> CDH clearly is,=0A> >> (Thoug=
h if we were to come up with a formal=0A> declaration of what a=0A> >> deri=
vative work is, we'd have to handle the fact=0A> that it is a superset.=0A>=
 >> Even worse, you may realise a release is the=0A> ordered application of=
 a=0A> >> sequence of patches, and if the patches are=0A> applied in a diff=
erent order=0A> >> you may end up with a different body of source=0A> code.=
..)=0A> >> =0A> >> Something that implements the APIs may not be a=0A> deri=
vative work,=0A> >> depending on how much of the original code is in=0A> th=
ere. You could look=0A> >> at the base classes and interfaces and produce a=
=0A> clean room=0A> >> implementation (relying on the notion that=0A> inter=
faces are a list of=0A> >> facts and not copyrightable in the US), but=0A> =
whoever does that may=0A> >> encounter the issue that Google's donation of =
the=0A> right to use their MR=0A> >> patent may not apply to such implement=
ations.=0A> >> =0A> >> =0A> >> The information contained in this communicat=
ion=0A> may be CONFIDENTIAL and=0A> >>is intended only for the use of the r=
ecipient(s)=0A> named above.=A0 If you=0A> >>are not the intended recipient=
, you are hereby=0A> notified that any=0A> >>dissemination, distribution, o=
r copying of this=0A> communication, or any of=0A> >>its contents, is stric=
tly prohibited.=A0 If you=0A> have received this=0A> >>communication in err=
or, please notify the sender=0A> and delete/destroy the=0A> >>original mess=
age and any copy of it from your=0A> computer or paper files.=0A> >> =0A> >=
=0A> >=0A> >The information contained in this communication may be=0A> CONF=
IDENTIAL and=0A> >is intended only for the use of the recipient(s) named=0A=
> above.=A0 If you are=0A> >not the intended recipient, you are hereby noti=
fied=0A> that any=0A> >dissemination, distribution, or copying of this=0A> =
communication, or any of=0A> >its contents, is strictly prohibited.=A0 If y=
ou have=0A> received this=0A> >communication in error, please notify the se=
nder and=0A> delete/destroy the=0A> >original message and any copy of it fr=
om your computer=0A> or paper files.=0A> =0A> 

From general-return-3566-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 09:20:41 2011
Return-Path: <general-return-3566-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4B74F4886
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 17 May 2011 09:20:41 +0000 (UTC)
Received: (qmail 15253 invoked by uid 500); 17 May 2011 09:20:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15182 invoked by uid 500); 17 May 2011 09:20:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15174 invoked by uid 99); 17 May 2011 09:20:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 09:20:39 +0000
X-ASF-Spam-Status: No, hits=3.3 required=5.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 09:20:31 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p4H9JHaZ009919
	for <general@hadoop.apache.org>; Tue, 17 May 2011 02:19:17 -0700 (PDT)
Received: from SP2-EX07VS03.ds.corp.yahoo.com ([98.137.59.32]) by
 SP2-EX07CAS01.ds.corp.yahoo.com ([98.137.59.37]) with mapi; Tue, 17 May 2011
 02:19:17 -0700
From: Matthew Foley <mattf@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: Matthew Foley <mattf@yahoo-inc.com>
Date: Tue, 17 May 2011 02:19:08 -0700
Subject: Re: Defining Hadoop Compatibility -revisiting-
Thread-Topic: Defining Hadoop Compatibility -revisiting-
Thread-Index: AcwUc36MXcEeGdVESJyEZQQeYplkIg==
Message-ID: <CA18BF37-B892-4E05-9BCA-44AE1B290604@yahoo-inc.com>
References: <C9F71692.35BAB%scott@richrelevance.com>
In-Reply-To: <C9F71692.35BAB%scott@richrelevance.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_CA18BF37B8924E059BCA44AE1B290604yahooinccom_"
MIME-Version: 1.0

--_000_CA18BF37B8924E059BCA44AE1B290604yahooinccom_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

TESS only has "registered" trademarks -- that's the kind of trademark you p=
ut an "(R)" next to.
But you can have an ordinary unregistered trademark -- the kind you put a "=
tm" next to --
just by claiming it, and then promoting and defending it.

In the second paragraph of our bylaws<http://hadoop.apache.org/bylaws.html>=
 we claim:
      The foundation holds the trademark on the name "Hadoop" and copyright=
 on
      Apache code including the code in the Hadoop codebase.
In the LICENSE.txt file in our distribution, clause 6 of the Apache License=
 states,
      6. Trademarks. This License does not grant permission to use the trad=
e
      names, trademarks, service marks, or product names of the Licensor,
      except as required for reasonable and customary use in describing the
      origin of the Work and reproducing the content of the NOTICE file.
Very important!  This exclusion of trademarks from the opensource license i=
s normal
and appropriate, precisely because it is the primary tool for preventing co=
nfusion and
fragmentation in an opensource marketplace.

However, we also have to promote and defend the trademark.  I'll leave it u=
p to the
lawyers to define exactly what that means, but it does take some effort.  W=
e should
probably use the "tm" annotation in our logo GIFs and our primary market-fa=
cing
documents (not so much in the code).  Perhaps the PMC or Apache Board can a=
sk some
of our sponsor organizations to help out with some techdocs and trademark l=
egal review
assistance.  It's not really a topic for us lay people to argue over, it ju=
st wastes time.

WRT Scott's comments below, understand that trademark lawyers love to talk =
about using
trademarks only as "modifiers", i.e., essentially as adjectives.  We should=
n't say "Chevrolet"
as a thing, we should say "Chevrolet (tm) cars".  There are many kinds of c=
ars, and "Chevrolet"
is a mark distinguishing THIS kind of car from the others.  You can't trade=
mark THINGS, you
can only trademark DISTINGUISHING MARKS that differentiate things in the ma=
rketplace.
And it has nothing to do with the underlying technology.

So here are three statements I believe to be true:

1. We should defend against the usage "Apache Hadoop", "Cloudera Hadoop",
"Yahoo Hadoop", and "EMC Hadoop".  This would imply that Hadoop was a THING=
,
and those other words -- all trademarks! -- are the modifiers.  Not accepta=
ble.

2. If the PMC chooses to, it is okay to allow usages like "Cloudera distrib=
ution of
Hadoop", "Yahoo distribution of Hadoop", and "EMC distribution of Hadoop", =
or even
"powered by Hadoop", "built on Hadoop", "Hadoop inside", or whatever, where
"Hadoop" is understood to mean "Hadoop distributed computing platform"
(as opposed to other kinds of distributed computing platform product).
ALL of those usages are claiming to "be Hadoop" in some sense, so they are
protected by the trademark on "Hadoop".  Therefore, they can only be used i=
f each
of those companies obtain a license from Apache to use the Hadoop trademark=
,
which should include an agreement about correct use of the mark.  And curre=
ntly
they DON'T have such a license, because the Apache License specifically exc=
ludes
trademarks!

3. If other companies choose to sell distributed computing platform product=
s that
are NOT named "Hadoop", but their marketing literature says these products =
are
"compatible with Hadoop" while also making clear that they are not Hadoop a=
nd
don't claim to be Hadoop -- we probably can't do anything about it.  Claims=
 of
compatibility are generally protected for the sake of competition in the ma=
rketplace.

If We-The-Community can come to an agreement about what compatibility means=
,
we can build it into the license to use the "Hadoop" trademark (as in #2 ab=
ove), then
enforce it with peer pressure and market rejection of non-conforming produc=
ts (#3).
But the only real help from the law will be in distinguishing case #2 from =
case #3.

--Matt
"I am not a lawyer, the above is just my opinion, and does not represent th=
e opinion of my employer."


On May 16, 2011, at 6:12 PM, Scott Carey wrote:

On trademarks, what about the phrase:  "New distribution for Apache
Hadoop"?  I've seen that used, and its something that replaces most of the
stack.  I believe "Apache Hadoop" is trademarked in this context, even if
Hadoop alone isn't.
"Compatible with Apache Hadoop" is a smaller issue, defining some rough
guidelines for various forms of compatibility is useful for the community
(and reputable vendors), abuse of that will at least become obvious.  But
"distribution for Apache Hadoop" (not too sure what 'for' means here)?  Is
there any TM protection?  A proprietary derivative work with most of the
guts replaced is not an Apache Hadoop distribution, nor a distribution for
Apache Hadoop.

On 5/16/11 5:40 PM, "Segel, Mike" <msegel@navteq.com<mailto:msegel@navteq.c=
om>> wrote:

I just checked... TESS said no trademarks for Hadoop.
So... what TM protection? :-)

You are correct about derivative works. It's a moot point as long as the
derivative work follows the T&Cs...



Sent from a remote device. Please excuse any typos...

Mike Segel

On May 16, 2011, at 4:18 PM, "Matthew Foley" <mattf@yahoo-inc.com<mailto:ma=
ttf@yahoo-inc.com>> wrote:

It's important to distinguish between the name "Hadoop", which is
protected by trademark law,
and the Hadoop implementation, which is licensed as opensource under
copyright law.

The term "derivative work" is, I believe, only relevant under copyright
law, not trademark law.
(N.B., I'm not a lawyer -- and this email is my opinion, not my
employer's.)  Since the Apache License
explicitly allows derivative works, I don't think it's a useful term
for this discussion.

However, the ASF, and by delegation the Hadoop PMC, has a lot of
control over the name,
and how we allow it to be used, under trademark law.  But to keeps our
rights under that
law, we have to enforce the trademark consistently.  So it's good that
we're having this discussion,
and it's important to reach a conclusion, document it, and enforce it
consistently.

There are a lot of subtleties; for instance, if I recall correctly from
my days with Adobe and
PostScript(R), someone who has not licensed a trademark "X" can still
claim "compatible with X"
as long as they ALSO make clear that the product is NOT, itself, an
"X".  But you really need
a lawyer to get into that stuff.

--Matt


On May 16, 2011, at 5:00 AM, Segel, Mike wrote:

But Cloudera's release is a bit murky.

The math example is a bit flawed...

X represents the set of stable releases.
Y represents the set of available patches.
C represents the set of Cloudera releases.

So if C contains a release X(n) plus a set of patches that is contained
in Y,
Then does it not have the right to be considered Apache Hadoop?
It's my understanding is that any enhancement to Hadoop is made
available to Apache and will eventually make it into a later release...

So while it may not be 'official' release X(z), all of it's components
are in Apache.
(note: I'm talking about the core components and not Cloudera's
additional toolsets that encompass Hadoop.)

Cloudera is clearly a derivative work.
And IMHO is the only one which can say ... 'Includes Apache Hadoop'.

That doesn't mean that others can't, depending on how they implemented
their changes.
Based on EMC marketing material, they've done a rip and replace of HDFS.
So it wouldn't be a superset since it doesn't contain a complete
subset, but contains code that implements the API... So they can't say
'Includes Apache Hadoop',but they can say it's a derivative work based
on Apache Hadoop and then go on to show how and why, in their opinion
their product is better.(that's marketing for you...)

Clearly there are others out there...
Hadoop on Cassandra as an example...

Fragmentation of Hadoop will occur. It's inevitable. Too much money is
on the table...

But because Apache's licensing is so open, Apache will have a hard time
controlling derivative works...
I believe that Steve is incorrect in his assertion concerning potential
loss of any patent protection. Again Apache's licensing is very open and
as long as they follow Apache's Ts and Cs, they are covered.

Note: because I am sending this from my email address at my client, I
am obliged to say that this email is my opinion and does not reflect on
the opinion of my client...
(you know the rest....)

Sent from a remote device. Please excuse any typos...

Mike Segel

On May 16, 2011, at 6:02 AM, "Steve Loughran"
<stevel@apache.org<mailto:stevel@apache.org>> wrote:

On 13/05/11 23:57, Allen Wittenauer wrote:

On May 13, 2011, at 3:53 PM, Ted Dunning wrote:

But "distribution Z includes X" kind of implies the existence of some
such
that X !=3D Y, Y !=3D empty-set and X+Y =3D Z, at least in common usage.

Isn't that the same as a non-trunk change?

So doesn't this mean that your question reduces to the question of what
happens when non-Apache changes are made to an Apache release?  And
isn't
that the definition of a derived work?


Yup. Which is why I doubt *any* commercial entity can claim "includes
Apache Hadoop" (including Cloudera).



but they can claim it is a derivative work, which CDH clearly is,
(Though if we were to come up with a formal declaration of what a
derivative work is, we'd have to handle the fact that it is a superset.
Even worse, you may realise a release is the ordered application of a
sequence of patches, and if the patches are applied in a different order
you may end up with a different body of source code...)

Something that implements the APIs may not be a derivative work,
depending on how much of the original code is in there. You could look
at the base classes and interfaces and produce a clean room
implementation (relying on the notion that interfaces are a list of
facts and not copyrightable in the US), but whoever does that may
encounter the issue that Google's donation of the right to use their MR
patent may not apply to such implementations.


The information contained in this communication may be CONFIDENTIAL and
is intended only for the use of the recipient(s) named above.  If you
are not the intended recipient, you are hereby notified that any
dissemination, distribution, or copying of this communication, or any of
its contents, is strictly prohibited.  If you have received this
communication in error, please notify the sender and delete/destroy the
original message and any copy of it from your computer or paper files.



The information contained in this communication may be CONFIDENTIAL and
is intended only for the use of the recipient(s) named above.  If you are
not the intended recipient, you are hereby notified that any
dissemination, distribution, or copying of this communication, or any of
its contents, is strictly prohibited.  If you have received this
communication in error, please notify the sender and delete/destroy the
original message and any copy of it from your computer or paper files.



--_000_CA18BF37B8924E059BCA44AE1B290604yahooinccom_--

From general-return-3567-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 12:53:10 2011
Return-Path: <general-return-3567-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0B28E4915
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 17 May 2011 12:53:10 +0000 (UTC)
Received: (qmail 78097 invoked by uid 500); 17 May 2011 12:53:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78042 invoked by uid 500); 17 May 2011 12:53:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78034 invoked by uid 99); 17 May 2011 12:53:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 12:53:08 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of msegel@navteq.com designates 65.167.11.240 as permitted sender)
Received: from [65.167.11.240] (HELO xmailchi.navteq.com) (65.167.11.240)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 12:53:01 +0000
Received: from chicago-spamfw.navteq.com (imailchi [10.8.5.161])
	by xmailchi.navteq.com (8.13.8/8.13.6) with ESMTP id p4HCqdSx032437
	for <general@hadoop.apache.org>; Tue, 17 May 2011 07:52:40 -0500
Received: from imailchi.hq.navteq.com (localhost [127.0.0.1])
	by chicago-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id 8F39F4425A0B
	for <general@hadoop.apache.org>; Tue, 17 May 2011 07:52:39 -0500 (CDT)
Received: from imailchi.hq.navteq.com (imailchi.hq.navteq.com [10.8.120.104]) by chicago-spamfw.navteq.com with ESMTP id 5MsrP6QHvAfbEJcO for <general@hadoop.apache.org>; Tue, 17 May 2011 07:52:39 -0500 (CDT)
Received: from hq-ex-ht01.ad.navteq.com (hq-ex-ht01.ad.navteq.com [10.8.222.51])
	by imailchi.hq.navteq.com (8.12.9/8.12.9) with ESMTP id p4HCqdNe002614
	for <general@hadoop.apache.org>; Tue, 17 May 2011 07:52:39 -0500
Received: from hq-ex-mb03.ad.navteq.com ([fe80::c4dd:7b21:5c22:cfe4]) by
 hq-ex-ht01.ad.navteq.com ([fe80::55a6:5b94:5c00:fe39%12]) with mapi; Tue, 17
 May 2011 07:52:39 -0500
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 17 May 2011 07:52:37 -0500
Subject: RE: Defining Hadoop Compatibility -revisiting-
Thread-Topic: Defining Hadoop Compatibility -revisiting-
Thread-Index: AcwUc36MXcEeGdVESJyEZQQeYplkIgAHIjdw
Message-ID: <3798688F8784154192FDA3388F4C2081D59C0C@hq-ex-mb03.ad.navteq.com>
References: <C9F71692.35BAB%scott@richrelevance.com>
 <CA18BF37-B892-4E05-9BCA-44AE1B290604@yahoo-inc.com>
In-Reply-To: <CA18BF37-B892-4E05-9BCA-44AE1B290604@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1
X-Virus-Checked: Checked by ClamAV on apache.org

Well that would explain it, although Apache itself is a Registered Trade =
Mark.

I agree you need to step up the enforcement because you don't want Hadoop=
 to become the next Kleenex... :-)


-----Original Message-----
From: Matthew Foley [mailto:mattf@yahoo-inc.com]=20
Sent: Tuesday, May 17, 2011 4:19 AM
To: general@hadoop.apache.org
Cc: Matthew Foley
Subject: Re: Defining Hadoop Compatibility -revisiting-

TESS only has "registered" trademarks -- that's the kind of trademark you=
 put an "(R)" next to.
But you can have an ordinary unregistered trademark -- the kind you put a=
 "tm" next to --
just by claiming it, and then promoting and defending it.

In the second paragraph of our bylaws<http://hadoop.apache.org/bylaws.htm=
l> we claim:
      The foundation holds the trademark on the name "Hadoop" and copyrig=
ht on
      Apache code including the code in the Hadoop codebase.
In the LICENSE.txt file in our distribution, clause 6 of the Apache Licen=
se states,
      6. Trademarks. This License does not grant permission to use the tr=
ade
      names, trademarks, service marks, or product names of the Licensor,
      except as required for reasonable and customary use in describing t=
he
      origin of the Work and reproducing the content of the NOTICE file.
Very important!  This exclusion of trademarks from the opensource license=
 is normal
and appropriate, precisely because it is the primary tool for preventing =
confusion and
fragmentation in an opensource marketplace.

However, we also have to promote and defend the trademark.  I'll leave it=
 up to the
lawyers to define exactly what that means, but it does take some effort. =
 We should
probably use the "tm" annotation in our logo GIFs and our primary market-=
facing
documents (not so much in the code).  Perhaps the PMC or Apache Board can=
 ask some
of our sponsor organizations to help out with some techdocs and trademark=
 legal review
assistance.  It's not really a topic for us lay people to argue over, it =
just wastes time.

WRT Scott's comments below, understand that trademark lawyers love to tal=
k about using
trademarks only as "modifiers", i.e., essentially as adjectives.  We shou=
ldn't say "Chevrolet"
as a thing, we should say "Chevrolet (tm) cars".  There are many kinds of=
 cars, and "Chevrolet"
is a mark distinguishing THIS kind of car from the others.  You can't tra=
demark THINGS, you
can only trademark DISTINGUISHING MARKS that differentiate things in the =
marketplace.
And it has nothing to do with the underlying technology.

So here are three statements I believe to be true:

1. We should defend against the usage "Apache Hadoop", "Cloudera Hadoop",
"Yahoo Hadoop", and "EMC Hadoop".  This would imply that Hadoop was a THI=
NG,
and those other words -- all trademarks! -- are the modifiers.  Not accep=
table.

2. If the PMC chooses to, it is okay to allow usages like "Cloudera distr=
ibution of
Hadoop", "Yahoo distribution of Hadoop", and "EMC distribution of Hadoop"=
, or even
"powered by Hadoop", "built on Hadoop", "Hadoop inside", or whatever, whe=
re
"Hadoop" is understood to mean "Hadoop distributed computing platform"
(as opposed to other kinds of distributed computing platform product).
ALL of those usages are claiming to "be Hadoop" in some sense, so they ar=
e
protected by the trademark on "Hadoop".  Therefore, they can only be used=
 if each
of those companies obtain a license from Apache to use the Hadoop tradema=
rk,
which should include an agreement about correct use of the mark.  And cur=
rently
they DON'T have such a license, because the Apache License specifically e=
xcludes
trademarks!

3. If other companies choose to sell distributed computing platform produ=
cts that
are NOT named "Hadoop", but their marketing literature says these product=
s are
"compatible with Hadoop" while also making clear that they are not Hadoop=
 and
don't claim to be Hadoop -- we probably can't do anything about it.  Clai=
ms of
compatibility are generally protected for the sake of competition in the =
marketplace.

If We-The-Community can come to an agreement about what compatibility mea=
ns,
we can build it into the license to use the "Hadoop" trademark (as in #2 =
above), then
enforce it with peer pressure and market rejection of non-conforming prod=
ucts (#3).
But the only real help from the law will be in distinguishing case #2 fro=
m case #3.

--Matt
"I am not a lawyer, the above is just my opinion, and does not represent =
the opinion of my employer."


On May 16, 2011, at 6:12 PM, Scott Carey wrote:

On trademarks, what about the phrase:  "New distribution for Apache
Hadoop"?  I've seen that used, and its something that replaces most of th=
e
stack.  I believe "Apache Hadoop" is trademarked in this context, even if
Hadoop alone isn't.
"Compatible with Apache Hadoop" is a smaller issue, defining some rough
guidelines for various forms of compatibility is useful for the community
(and reputable vendors), abuse of that will at least become obvious.  But
"distribution for Apache Hadoop" (not too sure what 'for' means here)?  I=
s
there any TM protection?  A proprietary derivative work with most of the
guts replaced is not an Apache Hadoop distribution, nor a distribution fo=
r
Apache Hadoop.

On 5/16/11 5:40 PM, "Segel, Mike" <msegel@navteq.com<mailto:msegel@navteq=
=2Ecom>> wrote:

I just checked... TESS said no trademarks for Hadoop.
So... what TM protection? :-)

You are correct about derivative works. It's a moot point as long as the
derivative work follows the T&Cs...



Sent from a remote device. Please excuse any typos...

Mike Segel

On May 16, 2011, at 4:18 PM, "Matthew Foley" <mattf@yahoo-inc.com<mailto:=
mattf@yahoo-inc.com>> wrote:

It's important to distinguish between the name "Hadoop", which is
protected by trademark law,
and the Hadoop implementation, which is licensed as opensource under
copyright law.

The term "derivative work" is, I believe, only relevant under copyright
law, not trademark law.
(N.B., I'm not a lawyer -- and this email is my opinion, not my
employer's.)  Since the Apache License
explicitly allows derivative works, I don't think it's a useful term
for this discussion.

However, the ASF, and by delegation the Hadoop PMC, has a lot of
control over the name,
and how we allow it to be used, under trademark law.  But to keeps our
rights under that
law, we have to enforce the trademark consistently.  So it's good that
we're having this discussion,
and it's important to reach a conclusion, document it, and enforce it
consistently.

There are a lot of subtleties; for instance, if I recall correctly from
my days with Adobe and
PostScript(R), someone who has not licensed a trademark "X" can still
claim "compatible with X"
as long as they ALSO make clear that the product is NOT, itself, an
"X".  But you really need
a lawyer to get into that stuff.

--Matt


On May 16, 2011, at 5:00 AM, Segel, Mike wrote:

But Cloudera's release is a bit murky.

The math example is a bit flawed...

X represents the set of stable releases.
Y represents the set of available patches.
C represents the set of Cloudera releases.

So if C contains a release X(n) plus a set of patches that is contained
in Y,
Then does it not have the right to be considered Apache Hadoop?
It's my understanding is that any enhancement to Hadoop is made
available to Apache and will eventually make it into a later release...

So while it may not be 'official' release X(z), all of it's components
are in Apache.
(note: I'm talking about the core components and not Cloudera's
additional toolsets that encompass Hadoop.)

Cloudera is clearly a derivative work.
And IMHO is the only one which can say ... 'Includes Apache Hadoop'.

That doesn't mean that others can't, depending on how they implemented
their changes.
Based on EMC marketing material, they've done a rip and replace of HDFS.
So it wouldn't be a superset since it doesn't contain a complete
subset, but contains code that implements the API... So they can't say
'Includes Apache Hadoop',but they can say it's a derivative work based
on Apache Hadoop and then go on to show how and why, in their opinion
their product is better.(that's marketing for you...)

Clearly there are others out there...
Hadoop on Cassandra as an example...

Fragmentation of Hadoop will occur. It's inevitable. Too much money is
on the table...

But because Apache's licensing is so open, Apache will have a hard time
controlling derivative works...
I believe that Steve is incorrect in his assertion concerning potential
loss of any patent protection. Again Apache's licensing is very open and
as long as they follow Apache's Ts and Cs, they are covered.

Note: because I am sending this from my email address at my client, I
am obliged to say that this email is my opinion and does not reflect on
the opinion of my client...
(you know the rest....)

Sent from a remote device. Please excuse any typos...

Mike Segel

On May 16, 2011, at 6:02 AM, "Steve Loughran"
<stevel@apache.org<mailto:stevel@apache.org>> wrote:

On 13/05/11 23:57, Allen Wittenauer wrote:

On May 13, 2011, at 3:53 PM, Ted Dunning wrote:

But "distribution Z includes X" kind of implies the existence of some
such
that X !=3D Y, Y !=3D empty-set and X+Y =3D Z, at least in common usage.

Isn't that the same as a non-trunk change?

So doesn't this mean that your question reduces to the question of what
happens when non-Apache changes are made to an Apache release?  And
isn't
that the definition of a derived work?


Yup. Which is why I doubt *any* commercial entity can claim "includes
Apache Hadoop" (including Cloudera).



but they can claim it is a derivative work, which CDH clearly is,
(Though if we were to come up with a formal declaration of what a
derivative work is, we'd have to handle the fact that it is a superset.
Even worse, you may realise a release is the ordered application of a
sequence of patches, and if the patches are applied in a different order
you may end up with a different body of source code...)

Something that implements the APIs may not be a derivative work,
depending on how much of the original code is in there. You could look
at the base classes and interfaces and produce a clean room
implementation (relying on the notion that interfaces are a list of
facts and not copyrightable in the US), but whoever does that may
encounter the issue that Google's donation of the right to use their MR
patent may not apply to such implementations.


The information contained in this communication may be CONFIDENTIAL and
is intended only for the use of the recipient(s) named above.  If you
are not the intended recipient, you are hereby notified that any
dissemination, distribution, or copying of this communication, or any of
its contents, is strictly prohibited.  If you have received this
communication in error, please notify the sender and delete/destroy the
original message and any copy of it from your computer or paper files.



The information contained in this communication may be CONFIDENTIAL and
is intended only for the use of the recipient(s) named above.  If you are
not the intended recipient, you are hereby notified that any
dissemination, distribution, or copying of this communication, or any of
its contents, is strictly prohibited.  If you have received this
communication in error, please notify the sender and delete/destroy the
original message and any copy of it from your computer or paper files.




The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.

From general-return-3568-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 13:24:50 2011
Return-Path: <general-return-3568-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B0E764F5A
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 17 May 2011 13:24:50 +0000 (UTC)
Received: (qmail 35709 invoked by uid 500); 17 May 2011 13:24:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 35651 invoked by uid 500); 17 May 2011 13:24:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 35642 invoked by uid 99); 17 May 2011 13:24:49 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 13:24:49 +0000
Received: from localhost (HELO [90.168.248.176]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 13:24:48 +0000
Message-ID: <4DD2770E.2080204@apache.org>
Date: Tue, 17 May 2011 15:24:30 +0200
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Defining Hadoop Compatibility -revisiting-
References: <C9F71692.35BAB%scott@richrelevance.com> <CA18BF37-B892-4E05-9BCA-44AE1B290604@yahoo-inc.com>
In-Reply-To: <CA18BF37-B892-4E05-9BCA-44AE1B290604@yahoo-inc.com>
X-Enigmail-Version: 1.1.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Matt,

Have you read Apache's trademark policy page?

http://www.apache.org/foundation/marks/

Apache does not generally license its trademarks.  Constructions like,
"Acme Foo powered by Apache Bar" are generally permitted as they are not
deemed to create confusion about the origin of Bar.

Cheers,

Doug

On 05/17/2011 11:19 AM, Matthew Foley wrote:
> TESS only has "registered" trademarks -- that's the kind of trademark you put an "(R)" next to.
> But you can have an ordinary unregistered trademark -- the kind you put a "tm" next to --
> just by claiming it, and then promoting and defending it.
> 
> In the second paragraph of our bylaws<http://hadoop.apache.org/bylaws.html> we claim:
>       The foundation holds the trademark on the name "Hadoop" and copyright on
>       Apache code including the code in the Hadoop codebase.
> In the LICENSE.txt file in our distribution, clause 6 of the Apache License states,
>       6. Trademarks. This License does not grant permission to use the trade
>       names, trademarks, service marks, or product names of the Licensor,
>       except as required for reasonable and customary use in describing the
>       origin of the Work and reproducing the content of the NOTICE file.
> Very important!  This exclusion of trademarks from the opensource license is normal
> and appropriate, precisely because it is the primary tool for preventing confusion and
> fragmentation in an opensource marketplace.
> 
> However, we also have to promote and defend the trademark.  I'll leave it up to the
> lawyers to define exactly what that means, but it does take some effort.  We should
> probably use the "tm" annotation in our logo GIFs and our primary market-facing
> documents (not so much in the code).  Perhaps the PMC or Apache Board can ask some
> of our sponsor organizations to help out with some techdocs and trademark legal review
> assistance.  It's not really a topic for us lay people to argue over, it just wastes time.
> 
> WRT Scott's comments below, understand that trademark lawyers love to talk about using
> trademarks only as "modifiers", i.e., essentially as adjectives.  We shouldn't say "Chevrolet"
> as a thing, we should say "Chevrolet (tm) cars".  There are many kinds of cars, and "Chevrolet"
> is a mark distinguishing THIS kind of car from the others.  You can't trademark THINGS, you
> can only trademark DISTINGUISHING MARKS that differentiate things in the marketplace.
> And it has nothing to do with the underlying technology.
> 
> So here are three statements I believe to be true:
> 
> 1. We should defend against the usage "Apache Hadoop", "Cloudera Hadoop",
> "Yahoo Hadoop", and "EMC Hadoop".  This would imply that Hadoop was a THING,
> and those other words -- all trademarks! -- are the modifiers.  Not acceptable.
> 
> 2. If the PMC chooses to, it is okay to allow usages like "Cloudera distribution of
> Hadoop", "Yahoo distribution of Hadoop", and "EMC distribution of Hadoop", or even
> "powered by Hadoop", "built on Hadoop", "Hadoop inside", or whatever, where
> "Hadoop" is understood to mean "Hadoop distributed computing platform"
> (as opposed to other kinds of distributed computing platform product).
> ALL of those usages are claiming to "be Hadoop" in some sense, so they are
> protected by the trademark on "Hadoop".  Therefore, they can only be used if each
> of those companies obtain a license from Apache to use the Hadoop trademark,
> which should include an agreement about correct use of the mark.  And currently
> they DON'T have such a license, because the Apache License specifically excludes
> trademarks!
> 
> 3. If other companies choose to sell distributed computing platform products that
> are NOT named "Hadoop", but their marketing literature says these products are
> "compatible with Hadoop" while also making clear that they are not Hadoop and
> don't claim to be Hadoop -- we probably can't do anything about it.  Claims of
> compatibility are generally protected for the sake of competition in the marketplace.
> 
> If We-The-Community can come to an agreement about what compatibility means,
> we can build it into the license to use the "Hadoop" trademark (as in #2 above), then
> enforce it with peer pressure and market rejection of non-conforming products (#3).
> But the only real help from the law will be in distinguishing case #2 from case #3.
> 
> --Matt
> "I am not a lawyer, the above is just my opinion, and does not represent the opinion of my employer."
> 
> 
> On May 16, 2011, at 6:12 PM, Scott Carey wrote:
> 
> On trademarks, what about the phrase:  "New distribution for Apache
> Hadoop"?  I've seen that used, and its something that replaces most of the
> stack.  I believe "Apache Hadoop" is trademarked in this context, even if
> Hadoop alone isn't.
> "Compatible with Apache Hadoop" is a smaller issue, defining some rough
> guidelines for various forms of compatibility is useful for the community
> (and reputable vendors), abuse of that will at least become obvious.  But
> "distribution for Apache Hadoop" (not too sure what 'for' means here)?  Is
> there any TM protection?  A proprietary derivative work with most of the
> guts replaced is not an Apache Hadoop distribution, nor a distribution for
> Apache Hadoop.
> 
> On 5/16/11 5:40 PM, "Segel, Mike" <msegel@navteq.com<mailto:msegel@navteq.com>> wrote:
> 
> I just checked... TESS said no trademarks for Hadoop.
> So... what TM protection? :-)
> 
> You are correct about derivative works. It's a moot point as long as the
> derivative work follows the T&Cs...
> 
> 
> 
> Sent from a remote device. Please excuse any typos...
> 
> Mike Segel
> 
> On May 16, 2011, at 4:18 PM, "Matthew Foley" <mattf@yahoo-inc.com<mailto:mattf@yahoo-inc.com>> wrote:
> 
> It's important to distinguish between the name "Hadoop", which is
> protected by trademark law,
> and the Hadoop implementation, which is licensed as opensource under
> copyright law.
> 
> The term "derivative work" is, I believe, only relevant under copyright
> law, not trademark law.
> (N.B., I'm not a lawyer -- and this email is my opinion, not my
> employer's.)  Since the Apache License
> explicitly allows derivative works, I don't think it's a useful term
> for this discussion.
> 
> However, the ASF, and by delegation the Hadoop PMC, has a lot of
> control over the name,
> and how we allow it to be used, under trademark law.  But to keeps our
> rights under that
> law, we have to enforce the trademark consistently.  So it's good that
> we're having this discussion,
> and it's important to reach a conclusion, document it, and enforce it
> consistently.
> 
> There are a lot of subtleties; for instance, if I recall correctly from
> my days with Adobe and
> PostScript(R), someone who has not licensed a trademark "X" can still
> claim "compatible with X"
> as long as they ALSO make clear that the product is NOT, itself, an
> "X".  But you really need
> a lawyer to get into that stuff.
> 
> --Matt
> 
> 
> On May 16, 2011, at 5:00 AM, Segel, Mike wrote:
> 
> But Cloudera's release is a bit murky.
> 
> The math example is a bit flawed...
> 
> X represents the set of stable releases.
> Y represents the set of available patches.
> C represents the set of Cloudera releases.
> 
> So if C contains a release X(n) plus a set of patches that is contained
> in Y,
> Then does it not have the right to be considered Apache Hadoop?
> It's my understanding is that any enhancement to Hadoop is made
> available to Apache and will eventually make it into a later release...
> 
> So while it may not be 'official' release X(z), all of it's components
> are in Apache.
> (note: I'm talking about the core components and not Cloudera's
> additional toolsets that encompass Hadoop.)
> 
> Cloudera is clearly a derivative work.
> And IMHO is the only one which can say ... 'Includes Apache Hadoop'.
> 
> That doesn't mean that others can't, depending on how they implemented
> their changes.
> Based on EMC marketing material, they've done a rip and replace of HDFS.
> So it wouldn't be a superset since it doesn't contain a complete
> subset, but contains code that implements the API... So they can't say
> 'Includes Apache Hadoop',but they can say it's a derivative work based
> on Apache Hadoop and then go on to show how and why, in their opinion
> their product is better.(that's marketing for you...)
> 
> Clearly there are others out there...
> Hadoop on Cassandra as an example...
> 
> Fragmentation of Hadoop will occur. It's inevitable. Too much money is
> on the table...
> 
> But because Apache's licensing is so open, Apache will have a hard time
> controlling derivative works...
> I believe that Steve is incorrect in his assertion concerning potential
> loss of any patent protection. Again Apache's licensing is very open and
> as long as they follow Apache's Ts and Cs, they are covered.
> 
> Note: because I am sending this from my email address at my client, I
> am obliged to say that this email is my opinion and does not reflect on
> the opinion of my client...
> (you know the rest....)
> 
> Sent from a remote device. Please excuse any typos...
> 
> Mike Segel
> 
> On May 16, 2011, at 6:02 AM, "Steve Loughran"
> <stevel@apache.org<mailto:stevel@apache.org>> wrote:
> 
> On 13/05/11 23:57, Allen Wittenauer wrote:
> 
> On May 13, 2011, at 3:53 PM, Ted Dunning wrote:
> 
> But "distribution Z includes X" kind of implies the existence of some
> such
> that X != Y, Y != empty-set and X+Y = Z, at least in common usage.
> 
> Isn't that the same as a non-trunk change?
> 
> So doesn't this mean that your question reduces to the question of what
> happens when non-Apache changes are made to an Apache release?  And
> isn't
> that the definition of a derived work?
> 
> 
> Yup. Which is why I doubt *any* commercial entity can claim "includes
> Apache Hadoop" (including Cloudera).
> 
> 
> 
> but they can claim it is a derivative work, which CDH clearly is,
> (Though if we were to come up with a formal declaration of what a
> derivative work is, we'd have to handle the fact that it is a superset.
> Even worse, you may realise a release is the ordered application of a
> sequence of patches, and if the patches are applied in a different order
> you may end up with a different body of source code...)
> 
> Something that implements the APIs may not be a derivative work,
> depending on how much of the original code is in there. You could look
> at the base classes and interfaces and produce a clean room
> implementation (relying on the notion that interfaces are a list of
> facts and not copyrightable in the US), but whoever does that may
> encounter the issue that Google's donation of the right to use their MR
> patent may not apply to such implementations.
> 
> 
> The information contained in this communication may be CONFIDENTIAL and
> is intended only for the use of the recipient(s) named above.  If you
> are not the intended recipient, you are hereby notified that any
> dissemination, distribution, or copying of this communication, or any of
> its contents, is strictly prohibited.  If you have received this
> communication in error, please notify the sender and delete/destroy the
> original message and any copy of it from your computer or paper files.
> 
> 
> 
> The information contained in this communication may be CONFIDENTIAL and
> is intended only for the use of the recipient(s) named above.  If you are
> not the intended recipient, you are hereby notified that any
> dissemination, distribution, or copying of this communication, or any of
> its contents, is strictly prohibited.  If you have received this
> communication in error, please notify the sender and delete/destroy the
> original message and any copy of it from your computer or paper files.
> 
> 
> 

From general-return-3569-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 17:54:01 2011
Return-Path: <general-return-3569-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 115934241
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 17 May 2011 17:54:01 +0000 (UTC)
Received: (qmail 39535 invoked by uid 500); 17 May 2011 17:53:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39478 invoked by uid 500); 17 May 2011 17:53:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39470 invoked by uid 99); 17 May 2011 17:53:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 17:53:59 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 17:53:51 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p4HHr9JY098419
	for <general@hadoop.apache.org>; Tue, 17 May 2011 10:53:09 -0700 (PDT)
Received: from SP2-EX07VS03.ds.corp.yahoo.com ([98.137.59.32]) by
 SP2-EX07CAS01.ds.corp.yahoo.com ([98.137.59.37]) with mapi; Tue, 17 May 2011
 10:53:09 -0700
From: Matthew Foley <mattf@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: Matthew Foley <mattf@yahoo-inc.com>
Date: Tue, 17 May 2011 10:53:07 -0700
Subject: Re: Defining Hadoop Compatibility -revisiting-
Thread-Topic: Defining Hadoop Compatibility -revisiting-
Thread-Index: AcwUu0ef5rLyQdHoTXmZ6Li0qXoZsA==
Message-ID: <38C5317C-45F8-4913-8BEE-3C1442961A55@yahoo-inc.com>
References: <C9F71692.35BAB%scott@richrelevance.com>
 <CA18BF37-B892-4E05-9BCA-44AE1B290604@yahoo-inc.com>
 <4DD2770E.2080204@apache.org>
In-Reply-To: <4DD2770E.2080204@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

> Constructions like, "Acme Foo powered by Apache Bar" are generally permit=
ted...

Hi Doug,
Great document, very typical of company Trademark Policy documents.
It needs to be read in conjunction with the FAQ at
        http://www.apache.org/foundation/marks/faq/
which is where the "powered by" mark usage is authorized. =20
The "Apache Project Branding Requirements" for PMCs,
        http://www.apache.org/foundation/marks/pmcs.html
goes into more depth, apparently covering all of what I said in my prior em=
ail,
and more.

The FAQ about the "powered by" usage is not a general permission to use=20
similar constructions; rather it states 8 prescriptive guidelines for exact=
ly=20
how it is okay to use this specific construction.  The first one of those=20
requirements is that it can only be used for:
        "products or services that are supersets of the functionality of an=
 Apache=20
        product, or services [that] are run atop Apache products"

And this statement of permission in the publicly available FAQ constitutes =
a license,=20
so it is imprecise to say that ASF doesn't license its trademarks. :-)

The Project Branding page authorizes PMCs to develop their own "Powered by =
<project>"=20
or "<project> Inside" programs.  Has the Hadoop PMC done so?  Is there a we=
b page=20
for that?  Our "PoweredBy" page is only a list of companies and products.

I do need to clarify one thing I said below w.r.t. the "Apache Hadoop" cons=
truction.
It's perfectly fine to say "Apache Hadoop" as long as there is NOT ALSO a "=
Yahoo Hadoop",
a "Cloudera Hadoop", and an "EMC Hadoop".  And indeed such competing usages=
 are=20
forbidden by this very fine Trademark Policy document, while mandating the =
"Apache
Hadoop" usage.

Cheers,
--Matt


On May 17, 2011, at 6:24 AM, Doug Cutting wrote:

Matt,

Have you read Apache's trademark policy page?

http://www.apache.org/foundation/marks/

Apache does not generally license its trademarks.  Constructions like,
"Acme Foo powered by Apache Bar" are generally permitted as they are not
deemed to create confusion about the origin of Bar.

Cheers,

Doug

On 05/17/2011 11:19 AM, Matthew Foley wrote:
> TESS only has "registered" trademarks -- that's the kind of trademark you=
 put an "(R)" next to.
> But you can have an ordinary unregistered trademark -- the kind you put a=
 "tm" next to --
> just by claiming it, and then promoting and defending it.
>=20
> In the second paragraph of our bylaws<http://hadoop.apache.org/bylaws.htm=
l> we claim:
>      The foundation holds the trademark on the name "Hadoop" and copyrigh=
t on
>      Apache code including the code in the Hadoop codebase.
> In the LICENSE.txt file in our distribution, clause 6 of the Apache Licen=
se states,
>      6. Trademarks. This License does not grant permission to use the tra=
de
>      names, trademarks, service marks, or product names of the Licensor,
>      except as required for reasonable and customary use in describing th=
e
>      origin of the Work and reproducing the content of the NOTICE file.
> Very important!  This exclusion of trademarks from the opensource license=
 is normal
> and appropriate, precisely because it is the primary tool for preventing =
confusion and
> fragmentation in an opensource marketplace.
>=20
> However, we also have to promote and defend the trademark.  I'll leave it=
 up to the
> lawyers to define exactly what that means, but it does take some effort. =
 We should
> probably use the "tm" annotation in our logo GIFs and our primary market-=
facing
> documents (not so much in the code).  Perhaps the PMC or Apache Board can=
 ask some
> of our sponsor organizations to help out with some techdocs and trademark=
 legal review
> assistance.  It's not really a topic for us lay people to argue over, it =
just wastes time.
>=20
> WRT Scott's comments below, understand that trademark lawyers love to tal=
k about using
> trademarks only as "modifiers", i.e., essentially as adjectives.  We shou=
ldn't say "Chevrolet"
> as a thing, we should say "Chevrolet (tm) cars".  There are many kinds of=
 cars, and "Chevrolet"
> is a mark distinguishing THIS kind of car from the others.  You can't tra=
demark THINGS, you
> can only trademark DISTINGUISHING MARKS that differentiate things in the =
marketplace.
> And it has nothing to do with the underlying technology.
>=20
> So here are three statements I believe to be true:
>=20
> 1. We should defend against the usage "Apache Hadoop", "Cloudera Hadoop",
> "Yahoo Hadoop", and "EMC Hadoop".  This would imply that Hadoop was a THI=
NG,
> and those other words -- all trademarks! -- are the modifiers.  Not accep=
table.
>=20
> 2. If the PMC chooses to, it is okay to allow usages like "Cloudera distr=
ibution of
> Hadoop", "Yahoo distribution of Hadoop", and "EMC distribution of Hadoop"=
, or even
> "powered by Hadoop", "built on Hadoop", "Hadoop inside", or whatever, whe=
re
> "Hadoop" is understood to mean "Hadoop distributed computing platform"
> (as opposed to other kinds of distributed computing platform product).
> ALL of those usages are claiming to "be Hadoop" in some sense, so they ar=
e
> protected by the trademark on "Hadoop".  Therefore, they can only be used=
 if each
> of those companies obtain a license from Apache to use the Hadoop tradema=
rk,
> which should include an agreement about correct use of the mark.  And cur=
rently
> they DON'T have such a license, because the Apache License specifically e=
xcludes
> trademarks!
>=20
> 3. If other companies choose to sell distributed computing platform produ=
cts that
> are NOT named "Hadoop", but their marketing literature says these product=
s are
> "compatible with Hadoop" while also making clear that they are not Hadoop=
 and
> don't claim to be Hadoop -- we probably can't do anything about it.  Clai=
ms of
> compatibility are generally protected for the sake of competition in the =
marketplace.
>=20
> If We-The-Community can come to an agreement about what compatibility mea=
ns,
> we can build it into the license to use the "Hadoop" trademark (as in #2 =
above), then
> enforce it with peer pressure and market rejection of non-conforming prod=
ucts (#3).
> But the only real help from the law will be in distinguishing case #2 fro=
m case #3.
>=20
> --Matt
> "I am not a lawyer, the above is just my opinion, and does not represent =
the opinion of my employer."
>=20
>=20
> On May 16, 2011, at 6:12 PM, Scott Carey wrote:
>=20
> On trademarks, what about the phrase:  "New distribution for Apache
> Hadoop"?  I've seen that used, and its something that replaces most of th=
e
> stack.  I believe "Apache Hadoop" is trademarked in this context, even if
> Hadoop alone isn't.
> "Compatible with Apache Hadoop" is a smaller issue, defining some rough
> guidelines for various forms of compatibility is useful for the community
> (and reputable vendors), abuse of that will at least become obvious.  But
> "distribution for Apache Hadoop" (not too sure what 'for' means here)?  I=
s
> there any TM protection?  A proprietary derivative work with most of the
> guts replaced is not an Apache Hadoop distribution, nor a distribution fo=
r
> Apache Hadoop.
>=20
> On 5/16/11 5:40 PM, "Segel, Mike" <msegel@navteq.com<mailto:msegel@navteq=
.com>> wrote:
>=20
> I just checked... TESS said no trademarks for Hadoop.
> So... what TM protection? :-)
>=20
> You are correct about derivative works. It's a moot point as long as the
> derivative work follows the T&Cs...
>=20
>=20
>=20
> Sent from a remote device. Please excuse any typos...
>=20
> Mike Segel
>=20
> On May 16, 2011, at 4:18 PM, "Matthew Foley" <mattf@yahoo-inc.com<mailto:=
mattf@yahoo-inc.com>> wrote:
>=20
> It's important to distinguish between the name "Hadoop", which is
> protected by trademark law,
> and the Hadoop implementation, which is licensed as opensource under
> copyright law.
>=20
> The term "derivative work" is, I believe, only relevant under copyright
> law, not trademark law.
> (N.B., I'm not a lawyer -- and this email is my opinion, not my
> employer's.)  Since the Apache License
> explicitly allows derivative works, I don't think it's a useful term
> for this discussion.
>=20
> However, the ASF, and by delegation the Hadoop PMC, has a lot of
> control over the name,
> and how we allow it to be used, under trademark law.  But to keeps our
> rights under that
> law, we have to enforce the trademark consistently.  So it's good that
> we're having this discussion,
> and it's important to reach a conclusion, document it, and enforce it
> consistently.
>=20
> There are a lot of subtleties; for instance, if I recall correctly from
> my days with Adobe and
> PostScript(R), someone who has not licensed a trademark "X" can still
> claim "compatible with X"
> as long as they ALSO make clear that the product is NOT, itself, an
> "X".  But you really need
> a lawyer to get into that stuff.
>=20
> --Matt
>=20
>=20
> On May 16, 2011, at 5:00 AM, Segel, Mike wrote:
>=20
> But Cloudera's release is a bit murky.
>=20
> The math example is a bit flawed...
>=20
> X represents the set of stable releases.
> Y represents the set of available patches.
> C represents the set of Cloudera releases.
>=20
> So if C contains a release X(n) plus a set of patches that is contained
> in Y,
> Then does it not have the right to be considered Apache Hadoop?
> It's my understanding is that any enhancement to Hadoop is made
> available to Apache and will eventually make it into a later release...
>=20
> So while it may not be 'official' release X(z), all of it's components
> are in Apache.
> (note: I'm talking about the core components and not Cloudera's
> additional toolsets that encompass Hadoop.)
>=20
> Cloudera is clearly a derivative work.
> And IMHO is the only one which can say ... 'Includes Apache Hadoop'.
>=20
> That doesn't mean that others can't, depending on how they implemented
> their changes.
> Based on EMC marketing material, they've done a rip and replace of HDFS.
> So it wouldn't be a superset since it doesn't contain a complete
> subset, but contains code that implements the API... So they can't say
> 'Includes Apache Hadoop',but they can say it's a derivative work based
> on Apache Hadoop and then go on to show how and why, in their opinion
> their product is better.(that's marketing for you...)
>=20
> Clearly there are others out there...
> Hadoop on Cassandra as an example...
>=20
> Fragmentation of Hadoop will occur. It's inevitable. Too much money is
> on the table...
>=20
> But because Apache's licensing is so open, Apache will have a hard time
> controlling derivative works...
> I believe that Steve is incorrect in his assertion concerning potential
> loss of any patent protection. Again Apache's licensing is very open and
> as long as they follow Apache's Ts and Cs, they are covered.
>=20
> Note: because I am sending this from my email address at my client, I
> am obliged to say that this email is my opinion and does not reflect on
> the opinion of my client...
> (you know the rest....)
>=20
> Sent from a remote device. Please excuse any typos...
>=20
> Mike Segel
>=20
> On May 16, 2011, at 6:02 AM, "Steve Loughran"
> <stevel@apache.org<mailto:stevel@apache.org>> wrote:
>=20
> On 13/05/11 23:57, Allen Wittenauer wrote:
>=20
> On May 13, 2011, at 3:53 PM, Ted Dunning wrote:
>=20
> But "distribution Z includes X" kind of implies the existence of some
> such
> that X !=3D Y, Y !=3D empty-set and X+Y =3D Z, at least in common usage.
>=20
> Isn't that the same as a non-trunk change?
>=20
> So doesn't this mean that your question reduces to the question of what
> happens when non-Apache changes are made to an Apache release?  And
> isn't
> that the definition of a derived work?
>=20
>=20
> Yup. Which is why I doubt *any* commercial entity can claim "includes
> Apache Hadoop" (including Cloudera).
>=20
>=20
>=20
> but they can claim it is a derivative work, which CDH clearly is,
> (Though if we were to come up with a formal declaration of what a
> derivative work is, we'd have to handle the fact that it is a superset.
> Even worse, you may realise a release is the ordered application of a
> sequence of patches, and if the patches are applied in a different order
> you may end up with a different body of source code...)
>=20
> Something that implements the APIs may not be a derivative work,
> depending on how much of the original code is in there. You could look
> at the base classes and interfaces and produce a clean room
> implementation (relying on the notion that interfaces are a list of
> facts and not copyrightable in the US), but whoever does that may
> encounter the issue that Google's donation of the right to use their MR
> patent may not apply to such implementations.
>=20
>=20
> The information contained in this communication may be CONFIDENTIAL and
> is intended only for the use of the recipient(s) named above.  If you
> are not the intended recipient, you are hereby notified that any
> dissemination, distribution, or copying of this communication, or any of
> its contents, is strictly prohibited.  If you have received this
> communication in error, please notify the sender and delete/destroy the
> original message and any copy of it from your computer or paper files.
>=20
>=20
>=20
> The information contained in this communication may be CONFIDENTIAL and
> is intended only for the use of the recipient(s) named above.  If you are
> not the intended recipient, you are hereby notified that any
> dissemination, distribution, or copying of this communication, or any of
> its contents, is strictly prohibited.  If you have received this
> communication in error, please notify the sender and delete/destroy the
> original message and any copy of it from your computer or paper files.
>=20
>=20
>=20


From general-return-3570-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 13:20:56 2011
Return-Path: <general-return-3570-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 84DAB6B08
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 18 May 2011 13:20:56 +0000 (UTC)
Received: (qmail 91758 invoked by uid 500); 18 May 2011 13:20:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91695 invoked by uid 500); 18 May 2011 13:20:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91683 invoked by uid 99); 18 May 2011 13:20:54 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 13:20:54 +0000
Received: from localhost (HELO [90.169.153.176]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 13:20:53 +0000
Message-ID: <4DD3C7B1.5010408@apache.org>
Date: Wed, 18 May 2011 15:20:49 +0200
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.17) Gecko/20110424 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Defining Hadoop Compatibility -revisiting-
References: <C9F71692.35BAB%scott@richrelevance.com> <CA18BF37-B892-4E05-9BCA-44AE1B290604@yahoo-inc.com> <4DD2770E.2080204@apache.org> <38C5317C-45F8-4913-8BEE-3C1442961A55@yahoo-inc.com>
In-Reply-To: <38C5317C-45F8-4913-8BEE-3C1442961A55@yahoo-inc.com>
X-Enigmail-Version: 1.1.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On 05/17/2011 07:53 PM, Matthew Foley wrote:
> And this statement of permission in the publicly available FAQ constitutes a license, 
> so it is imprecise to say that ASF doesn't license its trademarks. :-)

That's not the way I interpret it.  I believe that a license would be
required to permit a use that might create confusion while the FAQ
provides sample stock phrases that are not thought to create confusion,
i.e., nominal uses.

> The Project Branding page authorizes PMCs to develop their own "Powered by <project>" 
> or "<project> Inside" programs.  Has the Hadoop PMC done so?  Is there a web page 
> for that?  Our "PoweredBy" page is only a list of companies and products.

There is an open issue to create a distinct "Powered by Hadoop" logo:

https://issues.apache.org/jira/browse/HADOOP-7020

Doug

From general-return-3571-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 14:56:46 2011
Return-Path: <general-return-3571-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9C37B63B7
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 18 May 2011 14:56:46 +0000 (UTC)
Received: (qmail 46044 invoked by uid 500); 18 May 2011 14:56:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45995 invoked by uid 500); 18 May 2011 14:56:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45987 invoked by uid 99); 18 May 2011 14:56:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 14:56:45 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jcoveney@gmail.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 14:56:41 +0000
Received: by gya6 with SMTP id 6so806807gya.35
        for <general@hadoop.apache.org>; Wed, 18 May 2011 07:56:20 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=bDiHEwLp0x8PokHbSmx4usn3FE2uqy0tLtee4ZGYgME=;
        b=rEumMADRYiPs23GAv/muj1eAMmz408xlZi4EL5istDRF/M4bUtbOsjvQ2L8ZdiwmCc
         gQ5zmmlQja9QNM+p+gBDNtjaYsgeRkPoav6u0m+bBWs6wAhafrwk0HQ/0Uvrx/XcnuZO
         YRl4wFKFyYwNL43aTeU4fiOvXwD0RPzlXs41g=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=FAK0O/9cmrKuug++fWEyC6sVZ4JtDWl8RSmUblNU31vtLHREThfhjSDxr9UVQubV3R
         xhDWabJVTAZV+H4squsHnc38/27KpRq7Sg+EGKtw5who3EwG0U7kGH9+1tvWb22jvkqs
         ogu8qE0ExAlNECaovw4Pm84BPfTV03NF1q51E=
MIME-Version: 1.0
Received: by 10.236.109.36 with SMTP id r24mr2152308yhg.327.1305730580493;
 Wed, 18 May 2011 07:56:20 -0700 (PDT)
Received: by 10.236.208.5 with HTTP; Wed, 18 May 2011 07:56:20 -0700 (PDT)
In-Reply-To: <BANLkTimJFR1xM7o7G_FmQk=gLR3xJ4Gexw@mail.gmail.com>
References: <BANLkTinDNbnhWgFwX-v=mwjHz5j+7oXUNg@mail.gmail.com>
	<BANLkTinxV5nexyuEiSt9fJ41907+QMrJyA@mail.gmail.com>
	<BANLkTimJFR1xM7o7G_FmQk=gLR3xJ4Gexw@mail.gmail.com>
Date: Wed, 18 May 2011 10:56:20 -0400
Message-ID: <BANLkTinPpLr+0YLj9m4N4V8p7dDP8JCTow@mail.gmail.com>
Subject: Re: Apache Hadoop Hackathon: 5/18 in Palo Alto and San Francisco
From: Jonathan Coveney <jcoveney@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=002354867bb66c379704a38e1719

--002354867bb66c379704a38e1719
Content-Type: text/plain; charset=ISO-8859-1

I think east coast Hackathons could be great (in DC here).

2011/5/16 Joe Stein <charmalloc@allthingshadoop.com>

> Any chance for something in the east (NYC) or do I need to start nagging
> the
> wife and kids that west coast weather is the way to go?
>
> I will post on the NYC HUG maybe we can get some Hack together contrib to
> Hadoop.... but maybe some evening/day that a few commiters on this list
> that
> are in NYC (not sure if there are any based in NYC?) they can run a mini
> hackathon to help get folks in contrib mode? that would rock!
>
> Maybe even some standard type of "Hack @ HUG" that every HUG can do with
> their users (build, patch, build, deploy, test) if they want...
>
> Thanks to all contributors and commiters for your hard work and continued
> efforts/dedication.
>
> On Mon, May 16, 2011 at 8:09 PM, Jeff Hammerbacher <hammer@cloudera.com
> >wrote:
>
> > Hey,
> >
> > We've got a great group coming together again on Wednesday for an Apache
> > Hadoop Hackathon in Palo Alto and San Francisco. Sign up at
> > http://hadoophackathon.eventbrite.com.
> >
> > As a reminder, we'll have Nigel Daley, the release manager for 0.22,
> > present
> > in Palo Alto. If you have build and release or testing skills and would
> > like
> > to contribute to Apache Hadoop, your skills will be highly valued. Please
> > consider coming out to meet Nigel and help out on that front!
> >
> > Also, there's a Hadoop User Group at Yahoo! in Sunnyvale on Wednesday
> > evening: http://www.meetup.com/hadoop/events/16805258. There will be a
> > large
> > caravan leaving from the Palo Alto Hackathon to attend the HUG, so if you
> > want to Hadoop all day, stop by Palo Alto and join us for the trip south.
> >
> > Regards,
> > Jeff
> >
> > On Thu, May 12, 2011 at 2:40 PM, Jeff Hammerbacher <hammer@cloudera.com
> > >wrote:
> >
> > > Hey,
> > >
> > > Thanks to everyone who came out for the Apache Hadoop Hackathon
> yesterday
> > > in Palo Alto and San Francisco. We had 35 people sign up from a great
> > cross
> > > section of companies: Yahoo!, Cloudera, Facebook, Apple, Twitter,
> > > Foursquare, AOL, Ngmoco, StumbleUpon, Trend Micro, Conviva, and more.
> We
> > had
> > > committers from the HBase, Hive, Pig, and Oozie projects ensuring their
> > > projects work with the upcoming 0.22 release, and a number of folks got
> > > their first patches up. The 0.22 branch is now being built by Jenkins
> and
> > > we're in good shape to get a release candidate up soon.
> > >
> > > We had so much fun that we're going to do it again!
> > >
> > > Join us next Wednesday, May 18th, in either San Francisco or Palo Alto,
> > and
> > > help us continue the march to get feature development back on trunk.
> > We'll
> > > have a special guest in Palo Alto: Nigel Daley, Apache Hadoop PMC
> member
> > and
> > > the release manager for the 0.22 release. If you're a sysadmin or
> devops
> > > person and want to help us build and test Apache Hadoop, this Hackathon
> > will
> > > be a great chance to learn about how it's done. Of course we'd also
> love
> > to
> > > have developers looking to get their first patches into Hadoop as well.
> > >
> > > Sign up at http://hadoophackathon.eventbrite.com and we'll see you
> > > Wednesday.
> > >
> > > Regards,
> > > Jeff
> > >
> > > p.s. if you're looking to prepare for the Hackathon, check out the
> issues
> > > tagged "newbie" at
> > >
> >
> https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&jqlQuery=labels+%3D+newbie
> > > .
> > >
> >
>
>
>
> --
>
> /*
> Joe Stein
> http://www.linkedin.com/in/charmalloc
> Twitter: @allthingshadoop
> */
>

--002354867bb66c379704a38e1719--

From general-return-3572-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 19:08:10 2011
Return-Path: <general-return-3572-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 547A84DF9
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 19 May 2011 19:08:10 +0000 (UTC)
Received: (qmail 45845 invoked by uid 500); 19 May 2011 19:08:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45151 invoked by uid 500); 19 May 2011 19:08:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44806 invoked by uid 99); 19 May 2011 19:08:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 19:08:04 +0000
X-ASF-Spam-Status: No, hits=3.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of akimball83@gmail.com designates 209.85.218.48 as permitted sender)
Received: from [209.85.218.48] (HELO mail-yi0-f48.google.com) (209.85.218.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 19:07:59 +0000
Received: by yia28 with SMTP id 28so1516494yia.35
        for <multiple recipients>; Thu, 19 May 2011 12:07:38 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:from:date:message-id:subject:to
         :content-type;
        bh=QYGAyAp17qezq0CAMMIagNfNjGjyfgM3swzaQrrIugI=;
        b=vsBBVJ5KYLpVPbjOeC2cZhHryF6RvHKypYwHxrhp2beRexCITVr+EayzKz2Naje9zM
         ttQNRU8rtUJe4KQT77a9Qqq72B/hBJZrRwMJTu3Qd2sOsd+4iFrFMS2qsItH9MR7/jc3
         lll8OC9DLZ5QB0I3n/MgYxvFvzI8WzBMoKP88=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type;
        b=jNOCNzPpOqoq1ffjEITIa1RfYLF8KkEufFpdgAO6Pqc/eWn7F88XBUuh0/RYxinZfs
         vCrVuURYD7Fp/wssKcotjitx/q+TO49jWvVl0D1QHv8yWNm3EzMdUn/RSOFfGUR5L/rz
         DRiFlQxkCUzgA4DrrGHO0pg/ERQGItu9CUqis=
Received: by 10.151.88.42 with SMTP id q42mr2808603ybl.402.1305832058118; Thu,
 19 May 2011 12:07:38 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.151.110.18 with HTTP; Thu, 19 May 2011 12:07:18 -0700 (PDT)
From: Aaron Kimball <akimball83@gmail.com>
Date: Thu, 19 May 2011 12:07:18 -0700
Message-ID: <BANLkTinRg=t-h54tStsjneutnrVh_1BjwA@mail.gmail.com>
Subject: Next SF HUG: June 8, at RichRelevance
To: general@hadoop.apache.org, core-user@hadoop.apache.org, 
	user@pig.apache.org, user@hive.apache.org
Content-Type: multipart/alternative; boundary=000e0cd7070cf5e44004a3a5b74a

--000e0cd7070cf5e44004a3a5b74a
Content-Type: text/plain; charset=ISO-8859-1

We had a great time after the Cloudera hackathon last week with our monthly
SF Hadoop User Group meetup. I'd like to thank Cloudera again for hosting
such a successful event.

Our next meetup will be held Wednesday, June 8, from 6pm to 8pm.

This meetup will be hosted by our friends at RichRelevance. Their office is
at 275 Battery St. #1150, though we'll be using their 9th floor conference
space.

As usual, we will use the discussion-based "unconference" format. At the
beginning of the meetup we will collaboratively construct an agenda
consisting of several discussion breakout groups. All participants may
propose a topic and volunteer to facilitate a discussion. All Hadoop-related
topics are encouraged, and all members of the Hadoop community are welcome.

Event schedule:

   - 6pm - Welcome
   - 6:30pm - Introductions; start creating agenda
   - Breakout sessions begin as soon as we're ready
   - 8pm - Conclusion


Food and refreshments will be provided, courtesy of RichRelevance.

If you're going to attend, please RSVP at http://bit.ly/kxaJqa.

Hope to see you all there!
- Aaron Kimball

--000e0cd7070cf5e44004a3a5b74a--

From general-return-3573-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 21:17:45 2011
Return-Path: <general-return-3573-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BD2334EDF
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 19 May 2011 21:17:45 +0000 (UTC)
Received: (qmail 71773 invoked by uid 500); 19 May 2011 21:17:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71683 invoked by uid 500); 19 May 2011 21:17:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71667 invoked by uid 99); 19 May 2011 21:17:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:17:43 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jghoman@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:17:37 +0000
Received: by wyb40 with SMTP id 40so3522770wyb.35
        for <multiple recipients>; Thu, 19 May 2011 14:17:16 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:from:date:message-id:subject:to
         :content-type;
        bh=AOh2qB2OPmVca7Ydx7cVL2R3ULjlss4RoGiFAz1zmvI=;
        b=nARZIkHrxUPPnw7HMSGZY+B8MIiioExKHmIIJpr2Imi5rgzrmPw3b+xMnlHx+BozWz
         HBzrrt+SpXezv7VpwsQFO/AGVMnS4ws3I0jM7QGtkJpW4jtLCZYPGL5CI8f12udPg2M0
         0UWiW/iYEcGeSbNbyL9rb4J1yb1EEAIihiaS4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type;
        b=xXH+ig/IvGfO6Uh5lapWji8Q3OFTy5qRMag1PU1EWOXLywTUKmGO20m0Nv2Po1NVQZ
         BZn1OVy8eaxVsMEeGlf8548wdj2UT/UncdzP+O3B2OKaW8n3Whc5Wkp0/SCYkcNPgYmW
         5CLnvFSrGYhNQ6IC4IspQJxCeIoe9NZzx/lUU=
Received: by 10.216.235.158 with SMTP id u30mr3328012weq.104.1305839836068;
 Thu, 19 May 2011 14:17:16 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.136.80 with HTTP; Thu, 19 May 2011 14:16:46 -0700 (PDT)
From: Jakob Homan <jghoman@gmail.com>
Date: Thu, 19 May 2011 14:16:46 -0700
Message-ID: <BANLkTi=GdqMHFg0_1qUS23NmRpgedA0ADA@mail.gmail.com>
Subject: Please join me in welcoming Matt Foley as the newest Apache Hadoop
 HDFS committer
To: general@hadoop.apache.org, hdfs-dev@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On behalf of the PMC, I am pleased to announce that Matt Foley has
been elected a committer in the Apache Hadoop HDFS project.  Please
join me in congratulating Matt and thanking him for his sustained,
substantial contributions thus far.  Matt is a great addition to our
community!

Jakob
Apache Hadoop PMC

From general-return-3574-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 21:19:46 2011
Return-Path: <general-return-3574-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 076064960
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 19 May 2011 21:19:46 +0000 (UTC)
Received: (qmail 77052 invoked by uid 500); 19 May 2011 21:19:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76956 invoked by uid 500); 19 May 2011 21:19:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76940 invoked by uid 99); 19 May 2011 21:19:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:19:43 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of atm@cloudera.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:19:37 +0000
Received: by qwj9 with SMTP id 9so2247318qwj.35
        for <multiple recipients>; Thu, 19 May 2011 14:19:16 -0700 (PDT)
Received: by 10.229.68.106 with SMTP id u42mr2633272qci.284.1305839956261;
 Thu, 19 May 2011 14:19:16 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.217.79 with HTTP; Thu, 19 May 2011 14:18:46 -0700 (PDT)
In-Reply-To: <BANLkTi=GdqMHFg0_1qUS23NmRpgedA0ADA@mail.gmail.com>
References: <BANLkTi=GdqMHFg0_1qUS23NmRpgedA0ADA@mail.gmail.com>
From: "Aaron T. Myers" <atm@cloudera.com>
Date: Thu, 19 May 2011 14:18:46 -0700
Message-ID: <BANLkTi=RkcyU7LLxVkqpJz+h+iKKNEH6zQ@mail.gmail.com>
Subject: Re: Please join me in welcoming Matt Foley as the newest Apache
 Hadoop HDFS committer
To: hdfs-dev@hadoop.apache.org
Cc: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f91426b9fbfb04a3a78e3d

--001485f91426b9fbfb04a3a78e3d
Content-Type: text/plain; charset=ISO-8859-1

Congratulations, Matt. It's a pleasure working with you. Well-deserved.

--
Aaron T. Myers
Software Engineer, Cloudera



On Thu, May 19, 2011 at 2:16 PM, Jakob Homan <jghoman@gmail.com> wrote:

> On behalf of the PMC, I am pleased to announce that Matt Foley has
> been elected a committer in the Apache Hadoop HDFS project.  Please
> join me in congratulating Matt and thanking him for his sustained,
> substantial contributions thus far.  Matt is a great addition to our
> community!
>
> Jakob
> Apache Hadoop PMC
>

--001485f91426b9fbfb04a3a78e3d--

From general-return-3575-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 21:25:56 2011
Return-Path: <general-return-3575-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 811114E03
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 19 May 2011 21:25:56 +0000 (UTC)
Received: (qmail 90706 invoked by uid 500); 19 May 2011 21:25:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90604 invoked by uid 500); 19 May 2011 21:25:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90588 invoked by uid 99); 19 May 2011 21:25:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:25:54 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:25:47 +0000
Received: by ewy22 with SMTP id 22so1561171ewy.35
        for <multiple recipients>; Thu, 19 May 2011 14:25:26 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.14.43.19 with SMTP id k19mr1253657eeb.187.1305840326495; Thu,
 19 May 2011 14:25:26 -0700 (PDT)
Received: by 10.14.22.6 with HTTP; Thu, 19 May 2011 14:25:26 -0700 (PDT)
In-Reply-To: <BANLkTi=GdqMHFg0_1qUS23NmRpgedA0ADA@mail.gmail.com>
References: <BANLkTi=GdqMHFg0_1qUS23NmRpgedA0ADA@mail.gmail.com>
Date: Thu, 19 May 2011 14:25:26 -0700
Message-ID: <BANLkTimTXbHmm38ngF+kNLN_MgM0y4+n9g@mail.gmail.com>
Subject: Re: Please join me in welcoming Matt Foley as the newest Apache
 Hadoop HDFS committer
From: Eli Collins <eli@cloudera.com>
To: hdfs-dev@hadoop.apache.org
Cc: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Welcome Matt!

On Thu, May 19, 2011 at 2:16 PM, Jakob Homan <jghoman@gmail.com> wrote:
> On behalf of the PMC, I am pleased to announce that Matt Foley has
> been elected a committer in the Apache Hadoop HDFS project. =A0Please
> join me in congratulating Matt and thanking him for his sustained,
> substantial contributions thus far. =A0Matt is a great addition to our
> community!
>
> Jakob
> Apache Hadoop PMC
>

From general-return-3576-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 21:30:02 2011
Return-Path: <general-return-3576-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 258ED4F4B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 19 May 2011 21:30:02 +0000 (UTC)
Received: (qmail 96861 invoked by uid 500); 19 May 2011 21:30:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96681 invoked by uid 500); 19 May 2011 21:30:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96666 invoked by uid 99); 19 May 2011 21:30:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:30:00 +0000
X-ASF-Spam-Status: No, hits=3.3 required=5.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:29:51 +0000
Received: from SP2-EX07CAS02.ds.corp.yahoo.com (sp2-ex07cas02.corp.sp2.yahoo.com [98.137.59.38])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p4JLSrlX054114;
	Thu, 19 May 2011 14:28:53 -0700 (PDT)
Received: from SP2-EX07VS04.ds.corp.yahoo.com ([98.137.59.33]) by
 SP2-EX07CAS02.ds.corp.yahoo.com ([98.137.59.38]) with mapi; Thu, 19 May 2011
 14:28:52 -0700
From: Kan Zhang <kan@yahoo-inc.com>
To: "hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org>,
        "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 19 May 2011 14:28:49 -0700
Subject: Re: Please join me in welcoming Matt Foley as the newest Apache
 Hadoop HDFS committer
Thread-Topic: Please join me in welcoming Matt Foley as the newest Apache
 Hadoop HDFS committer
Thread-Index: AcwWajnac070BELwSjCymRdtOggFPgAAYQJN
Message-ID: <C9FAD9A1.34C6C%kan@yahoo-inc.com>
In-Reply-To: <BANLkTi=GdqMHFg0_1qUS23NmRpgedA0ADA@mail.gmail.com>
Accept-Language: en, en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en, en-US
Content-Type: multipart/alternative;
	boundary="_000_C9FAD9A134C6Ckanyahooinccom_"
MIME-Version: 1.0

--_000_C9FAD9A134C6Ckanyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Congrats, Matt!


On 5/19/11 2:16 PM, "Jakob Homan" <jghoman@gmail.com> wrote:

On behalf of the PMC, I am pleased to announce that Matt Foley has
been elected a committer in the Apache Hadoop HDFS project.  Please
join me in congratulating Matt and thanking him for his sustained,
substantial contributions thus far.  Matt is a great addition to our
community!

Jakob
Apache Hadoop PMC


--_000_C9FAD9A134C6Ckanyahooinccom_--

From general-return-3577-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 21:33:26 2011
Return-Path: <general-return-3577-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 09AA4412A
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 19 May 2011 21:33:26 +0000 (UTC)
Received: (qmail 880 invoked by uid 500); 19 May 2011 21:33:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 797 invoked by uid 500); 19 May 2011 21:33:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 783 invoked by uid 99); 19 May 2011 21:33:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:33:24 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.215.176] (HELO mail-ey0-f176.google.com) (209.85.215.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:33:16 +0000
Received: by eya28 with SMTP id 28so1557557eya.35
        for <multiple recipients>; Thu, 19 May 2011 14:32:54 -0700 (PDT)
Received: by 10.14.0.133 with SMTP id 5mr1323274eeb.144.1305840773865;
        Thu, 19 May 2011 14:32:53 -0700 (PDT)
Received: from 172.18.202.200 (bda-178-239-85-56.bis7.eu.blackberry.com [178.239.85.56])
        by mx.google.com with ESMTPS id y3sm2146438eeh.23.2011.05.19.14.32.51
        (version=SSLv3 cipher=OTHER);
        Thu, 19 May 2011 14:32:52 -0700 (PDT)
X-rim-org-msg-ref-id:1773068962
Message-ID:<1773068962-1305840770-cardhu_decombobulator_blackberry.rim.net-1605458944-@b2.c9.bise7.blackberry>
Reply-To: michelan@hermanus.cc
X-Priority: Normal
Sensitivity: Normal
Importance: Normal
Subject: Re: Please join me in welcoming Matt Foley as the newest ApacheHadoop HDFS committer
To: general@hadoop.apache.org,"hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org>
From: "Michelan Arendse" <michelan@hermanus.cc>
Date: Thu, 19 May 2011 21:32:49 +0000
Content-Type: text/plain
MIME-Version: 1.0

Welcome Matt.
------Original Message------
From: Kan Zhang
To: hdfs-dev@hadoop.apache.org
To: general@hadoop.apache.org
ReplyTo: general@hadoop.apache.org
Subject: Re: Please join me in welcoming Matt Foley as the newest ApacheHadoop HDFS committer
Sent: May 19, 2011 11:28 PM

Congrats, Matt!


On 5/19/11 2:16 PM, "Jakob Homan" <jghoman@gmail.com> wrote:

On behalf of the PMC, I am pleased to announce that Matt Foley has
been elected a committer in the Apache Hadoop HDFS project.  Please
join me in congratulating Matt and thanking him for his sustained,
substantial contributions thus far.  Matt is a great addition to our
community!

Jakob
Apache Hadoop PMC





From general-return-3578-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 21:35:45 2011
Return-Path: <general-return-3578-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A41E746D1
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 19 May 2011 21:35:45 +0000 (UTC)
Received: (qmail 5894 invoked by uid 500); 19 May 2011 21:35:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5717 invoked by uid 500); 19 May 2011 21:35:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5709 invoked by uid 99); 19 May 2011 21:35:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:35:44 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mlortiz@uci.cu designates 200.55.140.180 as permitted sender)
Received: from [200.55.140.180] (HELO mx3.uci.cu) (200.55.140.180)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 19 May 2011 21:35:39 +0000
Received: (qmail 27189 invoked by uid 507); 19 May 2011 21:35:09 -0000
Received: from 10.0.0.185 by ns3.uci.cu (envelope-from <mlortiz@uci.cu>, uid 501) with qmail-scanner-2.01st 
 (avp: 5.0.2.0. spamassassin: 3.0.6. perlscan: 2.01st.  
 Clear:RC:1(10.0.0.185):. 
 Processed in 0.626323 secs); 19 May 2011 21:35:09 -0000
Received: from unknown (HELO ucimail4.uci.cu) (10.0.0.185)
  by 0 with SMTP; 19 May 2011 21:35:09 -0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by ucimail4.uci.cu (Postfix) with ESMTP id 2D8FD132497F;
	Thu, 19 May 2011 17:35:09 -0400 (CDT)
X-Virus-Scanned: amavisd-new at uci.cu
Received: from ucimail4.uci.cu ([127.0.0.1])
	by localhost (ucimail4.uci.cu [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id KEkH5XaozydI; Thu, 19 May 2011 17:35:08 -0400 (CDT)
Received: from [10.36.18.29] (marcosluis-aspire-5251.uci.cu [10.36.18.29])
	by ucimail4.uci.cu (Postfix) with ESMTP id D2D571324912;
	Thu, 19 May 2011 17:35:08 -0400 (CDT)
Message-ID: <4DD5947D.3000404@uci.cu>
Date: Thu, 19 May 2011 17:36:53 -0430
From: Marcos Ortiz <mlortiz@uci.cu>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110424 Lightning/1.0b2 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
CC: Jakob Homan <jghoman@gmail.com>, hdfs-dev@hadoop.apache.org
Subject: Re: Please join me in welcoming Matt Foley as the newest Apache Hadoop
 HDFS committer
References: <BANLkTi=GdqMHFg0_1qUS23NmRpgedA0ADA@mail.gmail.com>
In-Reply-To: <BANLkTi=GdqMHFg0_1qUS23NmRpgedA0ADA@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit

On 05/19/2011 04:46 PM, Jakob Homan wrote:
> On behalf of the PMC, I am pleased to announce that Matt Foley has
> been elected a committer in the Apache Hadoop HDFS project.  Please
> join me in congratulating Matt and thanking him for his sustained,
> substantial contributions thus far.  Matt is a great addition to our
> community!
>
> Jakob
> Apache Hadoop PMC
Congratulations, Matt

-- 
Marcos Luís Ortíz Valmaseda
  Software Engineer (Large-Scaled Distributed Systems)
  University of Information Sciences,
  La Habana, Cuba
  Linux User # 418229
  http://about.me/marcosortiz


From general-return-3579-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 21:36:25 2011
Return-Path: <general-return-3579-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E74C546FA
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 19 May 2011 21:36:24 +0000 (UTC)
Received: (qmail 8567 invoked by uid 500); 19 May 2011 21:36:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8509 invoked by uid 500); 19 May 2011 21:36:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8501 invoked by uid 99); 19 May 2011 21:36:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:36:23 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:36:16 +0000
Received: by qwj9 with SMTP id 9so2257010qwj.35
        for <general@hadoop.apache.org>; Thu, 19 May 2011 14:35:56 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:date:x-google-sender-auth
         :message-id:subject:from:to:content-type;
        bh=jR/bXI7zeYbnLyIx7juYGDU0YWK2vgKSMxOoupCDf+M=;
        b=DSjbtVlhzkeBpHObWbJ0uJ8WQykwgjdVMhASOt9Bc/sYgdgGkvVVJq7rXT0Ex8m2AJ
         p3T2GnLmGnx4xYrW/05xj0mXd/swQJY9ipcwF4z3kI6ektuLXfrgk1huXcLPxHmP9C66
         9ngcePj6vRdS6mGfYx+Hr94ybK9dRDxEJMyAY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        b=gyN9Kk70d7RXB3NcohoLjnSpehRyyCWlgOoz6+F5s7ShbEMRt4ixdMRxB1+y+DIVtb
         9dpjJsgUVBtirJWvm7cKlR4scUVqNkOQ0HMOKupFxP90z3y3WzMcXd1+a9foiN4cp/T1
         34THmECjAqUmG7ebSxzhBzazTl0mMg6avVG9U=
MIME-Version: 1.0
Received: by 10.224.196.193 with SMTP id eh1mr2755204qab.236.1305840955892;
 Thu, 19 May 2011 14:35:55 -0700 (PDT)
Sender: saint.ack@gmail.com
Received: by 10.224.19.208 with HTTP; Thu, 19 May 2011 14:35:55 -0700 (PDT)
Date: Thu, 19 May 2011 14:35:55 -0700
X-Google-Sender-Auth: RngRJ8p1HZcF4DENigF9bhnozA4
Message-ID: <BANLkTinnwuMy_STziezGfLrW6fxgJ-8dsA@mail.gmail.com>
Subject: ANN: HBase 0.90.3 available for download
From: Stack <stack@duboce.net>
To: Hbase-User <user@hbase.apache.org>, general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

The Apache HBase team is happy to announce that HBase 0.90.3 is
available from the Apache mirror of choice:

  http://www.apache.org/dyn/closer.cgi/hbase/

HBase 0.90.3 is a maintenance release that fixes several important bugs
since version 0.90.3, while retaining API and data compatibility. The
release notes may be found on the Apache JIRA:

  https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12316313

Users upgrading from previous versions of 0.90 may upgrade clients and servers
separately, though it is recommended that both be upgraded. If upgrading
from a version of HBase prior to 0.90.0, please read the notes accompanying
that release:

  http://osdir.com/ml/general-hadoop-apache/2011-01/msg00208.html

As always, many thanks to those who contributed to this release!

-The HBase Team

From general-return-3580-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 12:43:17 2011
Return-Path: <general-return-3580-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E6485699F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 20 May 2011 12:43:16 +0000 (UTC)
Received: (qmail 26437 invoked by uid 500); 20 May 2011 12:43:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26361 invoked by uid 500); 20 May 2011 12:43:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26353 invoked by uid 99); 20 May 2011 12:43:14 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 12:43:14 +0000
Received: from localhost (HELO [10.0.0.77]) (127.0.0.1)
  (smtp-auth username gsingers, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 12:43:14 +0000
From: Grant Ingersoll <gsingers@apache.org>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: Spring Scale-A-Thon RTP 2011 -- June 18th
Date: Fri, 20 May 2011 08:43:12 -0400
Message-Id: <ABAB186A-98D5-48FB-A4B7-D873BDA4C822@apache.org>
To: general@hadoop.apache.org
Mime-Version: 1.0 (Apple Message framework v1084)
X-Mailer: Apple Mail (2.1084)

Those in the Raleigh/Durham/Chapel Hill North Carolina area or those =
willing to travel might be interested in the upcoming Scale-A-Thon that =
a few of us from the Triangle Hadoop User's Group (www.trihug.org) are =
putting on.  Scale-A-Thon is a one day hacker's event aimed at devs =
wanting to hack on things like Hadoop, Mahout, HBase, Cassandra, etc.  =
This first time event is scheduled for June 18th.  You can learn more at =
http://www.scaleathon.com/2011/05/spring-scale-a-thon-rtp-2011/.  It's a =
free event, but is limited to the first 35 who register and tickets are =
going fast.

Hope to see you there,
Grant=

From general-return-3581-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 16:28:22 2011
Return-Path: <general-return-3581-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 649E94353
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Mon, 23 May 2011 16:28:22 +0000 (UTC)
Received: (qmail 24681 invoked by uid 500); 23 May 2011 16:28:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24619 invoked by uid 500); 23 May 2011 16:28:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24610 invoked by uid 99); 23 May 2011 16:28:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 16:28:20 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 16:28:15 +0000
Received: from oceanfarearth-lm.corp.yahoo.com (oceanfarearth-lm.corp.yahoo.com [10.72.113.156])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p4NGRVjD008596
	for <general@hadoop.apache.org>; Mon, 23 May 2011 09:27:31 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1306168051;
	bh=xBNXMzKUJda/HXyFqkspYX3q+smQCYk9WSV0yoL9X3Y=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=KnjmKxhaQ6zbpaWRnhGoR3eJHDYQ3JTg8M2nezBZoEgLnoRdHvgFuIOVV4c7c27IM
	 0KY4WWDaU0MWtoKoIgB5TMeeqkHKj8HgHO9ry8bfMtsJcO1fZt1iMxoEDOPNWh/+TI
	 dzqY4lstgXiybkrFG1cOdkfBAeGOIwL6vDyWuVAA=
Message-Id: <B655D278-10D9-42D9-8A4B-9E00510E751A@yahoo-inc.com>
From: Sanjay Radia <sradia@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <4DCCCCB2.6020607@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Defining Hadoop Compatibility -revisiting-
Date: Mon, 23 May 2011 09:27:31 -0700
References: <4DC91392.2010308@apache.org> <4DCCCCB2.6020607@apache.org>
X-Mailer: Apple Mail (2.936)

Agree.

On May 12, 2011, at 11:16 PM, Doug Cutting wrote:

> Certification semms like mission creep.  Our mission is to produce
> open-source software.  If we wish to produce testing software, that
> seems fine.  But running a certification program for non-open-source
> software seems like a different task.
>
> The Hadoop mark should only be used to refer to open-source software
> produced by the ASF.  If other folks wish to make factual statements
> concerning our software, e.g., that their proprietary software passes
> tests that we've created, that may be fine, but I don't think we  
> should
> validate those claims by granting certifications to institutions.   
> That
> ventures outside the mission of the ASF.  We are not an accrediting
> organization.
>
> Doug
>
> On 05/10/2011 12:29 PM, Steve Loughran wrote:
>>
>> Back in Jan 2011, I started a discussion about how to define Apache
>> Hadoop Compatibility:
>> http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C4D46B6AD.2020802@apache.org%3E
>>
>>
>> I am now reading EMC HD "Enterprise Ready" Apache Hadoop datasheet
>>
>> http://www.greenplum.com/sites/default/files/EMC_Greenplum_HD_DS_Final_1.pdf
>>
>>
>> It claims that their implementations are 100% compatible, even though
>> the Enterprise edition uses a C filesystem. It also claims that both
>> their software releases contain "Certified Stacks", without defining
>> what Certified means, or who does the certification -only that it  
>> is an
>> improvement.
>>
>>
>> I think we should revisit this issue before people with their own
>> agendas define what compatibility with Apache Hadoop is for us
>>
>>
>> Licensing
>> -Use of the Hadoop codebase must follow the Apache License
>> http://www.apache.org/licenses/LICENSE-2.0
>> -plug in components that are dynamically linked to (Filesystems and
>> schedulers) don't appear to be derivative works on my reading of  
>> this,
>>
>> Naming
>> -this is something for branding@apache, they will have their  
>> opinions.
>> The key one is that the name "Apache Hadoop" must get used, and it's
>> important to make clear it is a derivative work.
>> -I don't think you can claim to have a Distribution/Fork/Version of
>> Apache Hadoop if you swap out big chunks of it for alternate
>> filesystems, MR engines, etc. Some description of this is needed
>> "Supports the Apache Hadoop MapReduce engine on top of Filesystem  
>> XYZ"
>>
>> Compatibility
>> -the definition of the Hadoop interfaces and classes is the Apache
>> Source tree,
>> -the definition of semantics of the Hadoop interfaces and classes is
>> the Apache Source tree, including the test classes.
>> -the verification that the actual semantics of an Apache Hadoop  
>> release
>> is compatible with the expected semantics is that current and future
>> tests pass
>> -bug reports can highlight incompatibility with expectations of
>> community users, and once incorporated into tests form part of the
>> compatibility testing
>> -vendors can claim and even certify their derivative works as
>> compatible with other versions of their derivative works, but cannot
>> claim compatibility with Apache Hadoop unless their code passes the
>> tests and is consistent with the bug reports marked as ("by design").
>> Perhaps we should have tests that verify each of these "by design"
>> bugreps to make them more formal.
>>
>> Certification
>> -I have no idea what this means in EMC's case, they just say  
>> "Certified"
>> -As we don't do any certification ourselves, it would seem impossible
>> for us to certify that any derivative work is compatible.
>> -It may be best to state that nobody can certify their derivative as
>> "compatible with Apache Hadoop" unless it passes all current test  
>> suites
>> -And require that anyone who declares compatibility define what they
>> mean by this
>>
>> This is a good argument for getting more functional tests out there
>> -whoever has more functional tests needs to get them into a test  
>> module
>> that can be used to test real deployments.
>>


From general-return-3582-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 16:24:38 2011
Return-Path: <general-return-3582-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 476AD6744
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 24 May 2011 16:24:38 +0000 (UTC)
Received: (qmail 33256 invoked by uid 500); 24 May 2011 16:24:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33190 invoked by uid 500); 24 May 2011 16:24:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33182 invoked by uid 99); 24 May 2011 16:24:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 16:24:36 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 16:24:26 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id C39FE1BA95D
	for <general@hadoop.apache.org>; Tue, 24 May 2011 17:24:05 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id 3-7-zmt7WnEJ for <general@hadoop.apache.org>;
	Tue, 24 May 2011 17:24:05 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 3F6E41BA310
	for <general@hadoop.apache.org>; Tue, 24 May 2011 17:24:05 +0100 (BST)
MailScanner-NULL-Check: 1306859033.1297@CR2B36gh1q+FE4lOBYZFbQ
Received: from [16.25.172.146] (dhcp-172-146.hpl.hp.com [16.25.172.146])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p4OGNqq4012877
	for <general@hadoop.apache.org>; Tue, 24 May 2011 17:23:52 +0100 (BST)
Message-ID: <4DDBDB98.6040302@apache.org>
Date: Tue, 24 May 2011 17:23:52 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110419 Red Hat/3.1.10-1.el6_0 Thunderbird/3.1.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Defining Hadoop Compatibility -revisiting-
References: <4DC91392.2010308@apache.org> <4DCCCCB2.6020607@apache.org> <B655D278-10D9-42D9-8A4B-9E00510E751A@yahoo-inc.com>
In-Reply-To: <B655D278-10D9-42D9-8A4B-9E00510E751A@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p4OGNqq4012877
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org


I've drafted a policy on the wiki based on this discussion.

http://wiki.apache.org/hadoop/Defining%20Hadoop

Others need to look at, edit, etc, then we can vote on whether to take 
it into the managed documentation.

From general-return-3583-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 06:47:17 2011
Return-Path: <general-return-3583-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0FEFD6E06
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 26 May 2011 06:47:17 +0000 (UTC)
Received: (qmail 50581 invoked by uid 500); 26 May 2011 06:47:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50404 invoked by uid 500); 26 May 2011 06:47:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50389 invoked by uid 99); 26 May 2011 06:47:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 06:47:06 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.102 as permitted sender)
Received: from [17.148.16.102] (HELO asmtpout027.mac.com) (17.148.16.102)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 06:46:58 +0000
MIME-version: 1.0
Content-type: multipart/alternative;
 boundary="Boundary_(ID_5KSBVT9feIVA4w/dMwA6zw)"
Received: from [10.0.1.4] ([76.103.137.188])
 by asmtp027.mac.com (Oracle Communications Messaging Exchange Server 7u4-18.01
 64bit (built Jul 15 2010)) with ESMTPSA id <0LLS008AUITPO420@asmtp027.mac.com>
 for general@hadoop.apache.org; Wed, 25 May 2011 23:46:38 -0700 (PDT)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.4.6813,1.0.148,0.0.0000
 definitions=2011-05-26_02:2011-05-26,2011-05-26,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1105250235
From: Nigel Daley <ndaley@mac.com>
Subject: Update on 0.22
Date: Wed, 25 May 2011 23:46:36 -0700
Message-id: <1F63BF19-F939-432C-9B77-8ADC4731A05E@mac.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)

--Boundary_(ID_5KSBVT9feIVA4w/dMwA6zw)
Content-type: text/plain; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT

Looks like we're down to 12 blockers on 0.22. 

* Thanks to Cloudera for hosting a couple hack-a-thons over the past couple of weeks which helped get this number down.
* Thanks to Devaraj Das for volunteering to get https://issues.apache.org/jira/browse/MAPREDUCE-2178 committed.
* Thanks to Tom White for getting a CI build running that creates the actual release artifact.

I'm planning to commit https://issues.apache.org/jira/browse/HADOOP-7106 (SVN reorg) this Friday at 2pm PDT.  Todd, were you able to test git history based on your svn dump and import?

Cheers,
Nige

--Boundary_(ID_5KSBVT9feIVA4w/dMwA6zw)--

From general-return-3584-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 21:46:17 2011
Return-Path: <general-return-3584-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 413A7493D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 27 May 2011 21:46:17 +0000 (UTC)
Received: (qmail 93177 invoked by uid 500); 27 May 2011 21:46:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92500 invoked by uid 500); 27 May 2011 21:46:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 92173 invoked by uid 99); 27 May 2011 21:46:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 21:46:13 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.100 as permitted sender)
Received: from [17.148.16.100] (HELO asmtpout025.mac.com) (17.148.16.100)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 21:46:05 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.45.111.134] ([67.136.90.250])
 by asmtp025.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010))
 with ESMTPSA id <0LLV0010PJ2UHA70@asmtp025.mac.com>; Fri,
 27 May 2011 14:44:55 -0700 (PDT)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.4.6813,1.0.148,0.0.0000
 definitions=2011-05-27_06:2011-05-27,2011-05-27,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1105270168
Subject: Re: Update on 0.22
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <1F63BF19-F939-432C-9B77-8ADC4731A05E@mac.com>
Date: Fri, 27 May 2011 14:45:08 -0700
Message-id: <44FF77C5-A535-4D8B-B461-978C803F0D33@mac.com>
References: <1F63BF19-F939-432C-9B77-8ADC4731A05E@mac.com>
To: general@hadoop.apache.org, common-dev@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

I'm starting this now.

Nige

On May 25, 2011, at 11:46 PM, Nigel Daley wrote:
> I'm planning to commit https://issues.apache.org/jira/browse/HADOOP-7106 (SVN reorg) this Friday at 2pm PDT.  Todd, were you able to test git history based on your svn dump and import?
> 
> Cheers,
> Nige


From general-return-3585-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat May 28 01:22:42 2011
Return-Path: <general-return-3585-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8E5B6478F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat, 28 May 2011 01:22:42 +0000 (UTC)
Received: (qmail 48856 invoked by uid 500); 28 May 2011 01:22:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48780 invoked by uid 500); 28 May 2011 01:22:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48765 invoked by uid 99); 28 May 2011 01:22:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 May 2011 01:22:39 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.95 as permitted sender)
Received: from [17.148.16.95] (HELO asmtpout020.mac.com) (17.148.16.95)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 May 2011 01:22:30 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.45.111.134] ([67.136.90.250])
 by asmtp020.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010))
 with ESMTPSA id <0LLV00ASRT4WS270@asmtp020.mac.com>; Fri,
 27 May 2011 18:22:09 -0700 (PDT)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.4.6813,1.0.148,0.0.0000
 definitions=2011-05-27_08:2011-05-28,2011-05-27,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1105270230
Subject: Re: Update on 0.22
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <44FF77C5-A535-4D8B-B461-978C803F0D33@mac.com>
Date: Fri, 27 May 2011 18:22:08 -0700
Cc: common-dev@hadoop.apache.org
Message-id: <5E56623C-A3BB-4AFF-87DA-59370D031785@mac.com>
References: <1F63BF19-F939-432C-9B77-8ADC4731A05E@mac.com>
 <44FF77C5-A535-4D8B-B461-978C803F0D33@mac.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

I had to call this off as the auth and email patch was out of date.  I'll reschedule for next Friday at 2pm.

Nige

On May 27, 2011, at 2:45 PM, Nigel Daley wrote:

> I'm starting this now.
> 
> Nige
> 
> On May 25, 2011, at 11:46 PM, Nigel Daley wrote:
>> I'm planning to commit https://issues.apache.org/jira/browse/HADOOP-7106 (SVN reorg) this Friday at 2pm PDT.  Todd, were you able to test git history based on your svn dump and import?
>> 
>> Cheers,
>> Nige
> 


From general-return-3586-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sun May 29 13:22:21 2011
Return-Path: <general-return-3586-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 662104B02
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sun, 29 May 2011 13:22:21 +0000 (UTC)
Received: (qmail 59022 invoked by uid 500); 29 May 2011 13:22:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58189 invoked by uid 500); 29 May 2011 13:22:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57624 invoked by uid 99); 29 May 2011 13:22:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 May 2011 13:22:16 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [88.198.2.104] (HELO koch.ro) (88.198.2.104)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 May 2011 13:22:09 +0000
Received: from 22-94.77-83.cust.bluewin.ch ([83.77.94.22] helo=t61.localnet)
	by koch.ro with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
	(Exim 4.72)
	(envelope-from <thomas@koch.ro>)
	id 1QQfwC-0001Z1-88; Sun, 29 May 2011 15:21:48 +0200
From: Thomas Koch <thomas@koch.ro>
Reply-To: thomas@koch.ro
To: general@hadoop.apache.org,
 dev@zookeeper.apache.org,
 dev@lucene.apache.org,
 dev@nutch.apache.org,
 lily-developers@googlegroups.com
Subject: Keysigning @ Berlin Buzzwords
Date: Sun, 29 May 2011 13:46:43 +0200
User-Agent: KMail/1.13.5 (Linux/2.6.38-2-amd64; KDE/4.4.5; x86_64; ; )
MIME-Version: 1.0
Content-Type: text/plain;
  charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Message-Id: <201105291346.44102.thomas@koch.ro>
X-Virus-Checked: Checked by ClamAV on apache.org

=46rom my blog:
<http://www.koch.ro/blog/index.php?/archives/150-Keysigning-Berlin-
Buzzwords.html>

Whenever you download some software from the Apache Software Foundation, th=
ere=20
is a small .asc file besides every software release. This tiny file is a=20
cryptographic signature to protect you from accidentally downloading and=20
running malware.
However this signature is only as good as the web-of-trust between you and =
the=20
Release Manager of the Software project.
At BerlinBuzzWords many developers and users will gather at one place. This=
 is=20
a unique occasion to strengthen the web-of-trust. Therefor we want to=20
encourage and remind you to use the occassion for keysigning! For lack of t=
ime=20
there won't be an official keysigning party, but you have occassion during=
=20
lunch, coffee break or the barbecue to do quick one-to-one keysignings.
You should bring print-outs of your key fingerprints and identification=20
documents. If you put the fingerprints in your batch (see picture) everyone=
=20
can easily see
them and ask you for a quick trust exchange. We may also mark a "keysigning=
=20
corner" somewhere where people can go and meet others for key signing.

Hints:

    * under Debian (and Ubuntu) you can get a PDF to print out your=20
fingerprints many times via:
      sudo aptitude install signing-party ghostscript=20
      gpg-key2ps KEYID | ps2pdf - fingerprints.pdf
    * the signing-party package also contains the tool "caff" to batch proc=
ess=20
the signing of keys after the event
    * please only use keys with at least 2024 bits!
    * please make sure that you've uploaded your key to a public keyserver
    * I'm happy to answer all remaining questions at the BarCamp on Sunday =
or=20
during the event

Best regards,

Thomas Koch, http://www.koch.ro

From general-return-3587-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 22:09:00 2011
Return-Path: <general-return-3587-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AAF0B475B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 31 May 2011 22:09:00 +0000 (UTC)
Received: (qmail 227 invoked by uid 500); 31 May 2011 22:08:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 160 invoked by uid 500); 31 May 2011 22:08:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 152 invoked by uid 99); 31 May 2011 22:08:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 22:08:58 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of owen.omalley@gmail.com designates 209.85.210.48 as permitted sender)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 22:08:49 +0000
Received: by pzk10 with SMTP id 10so3087666pzk.35
        for <general@hadoop.apache.org>; Tue, 31 May 2011 15:08:28 -0700 (PDT)
Received: by 10.68.36.194 with SMTP id s2mr2515566pbj.119.1306879708132;
        Tue, 31 May 2011 15:08:28 -0700 (PDT)
Received: from battlerock-lm.corp.yahoo.com (nat-dip4.cfw-a-gci.corp.yahoo.com [209.131.62.113])
        by mx.google.com with ESMTPS id p5sm388810pbk.52.2011.05.31.15.08.23
        (version=SSLv3 cipher=OTHER);
        Tue, 31 May 2011 15:08:27 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: Defining Hadoop Compatibility -revisiting-
From: Owen O'Malley <omalley@apache.org>
In-Reply-To: <4DDBDB98.6040302@apache.org>
Date: Tue, 31 May 2011 15:08:22 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <A93FE831-B0B9-40F1-94E9-2E4D7B8BD6BC@apache.org>
References: <4DC91392.2010308@apache.org> <4DCCCCB2.6020607@apache.org> <B655D278-10D9-42D9-8A4B-9E00510E751A@yahoo-inc.com> <4DDBDB98.6040302@apache.org>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org


On May 24, 2011, at 9:23 AM, Steve Loughran wrote:

>=20
> I've drafted a policy on the wiki based on this discussion.
>=20
> http://wiki.apache.org/hadoop/Defining%20Hadoop
>=20
> Others need to look at, edit, etc, then we can vote on whether to take =
it into the managed documentation.

I think it looks great. Thanks for your work at drafting it, Steve. It =
addresses the rapidly growing problem of=20

Do we need the escape clause that states "or other products which have =
written approval from the VP, Apache Brand Management?" I think that =
will just cause lots of petitions from the numerous companies with =
rationalizations about why their exception should be allowed. If we =
limit Hadoop to mean precisely the Apache releases, there is no =
ambiguity or room to argue.

-- Owen=

From general-return-3588-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 23:10:25 2011
Return-Path: <general-return-3588-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 844F0416A
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 31 May 2011 23:10:25 +0000 (UTC)
Received: (qmail 74437 invoked by uid 500); 31 May 2011 23:10:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74388 invoked by uid 500); 31 May 2011 23:10:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74373 invoked by uid 99); 31 May 2011 23:10:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 23:10:22 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jdcryans@gmail.com designates 74.125.83.48 as permitted sender)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 23:10:16 +0000
Received: by gwj22 with SMTP id 22so3051164gwj.35
        for <general@hadoop.apache.org>; Tue, 31 May 2011 16:09:55 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:cc:content-type
         :content-transfer-encoding;
        bh=ozoJRLKVVaSeJBOvjVDjQNnk2tQ2mLYqLA0ywvezsFg=;
        b=E0KP74m2OJlLRwvuf6IVhTGSi/GzQimgQF7K4/+OW1Z9LfFD2P2uHKoRi3mS9fqGRi
         c3h/tRKV6/Tq/Zrq3hxu5Ybo7ZhKBIDNAJ/PdIJeL7CBwKnQKHou5ZJSD4UPbHy9DMN+
         RS3bCvqMgkW+PLmZZSdbB85HItN5QaLN1ImaM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:cc:content-type
         :content-transfer-encoding;
        b=Ze5ZDt979BOmgovSNy2Ts+NwKeRPENUHKOyErmZ5V0C1KhJY4QPbY7sD6kkKHOOP2T
         12duSK3IZQ+genXSGF3Sy0gAhF8WA2mziZTmABC45tvkze616WtbW6a8ylb61k7FuuCx
         22dngQcmw5UpWV4q7RqLT7VwtfMOVs7RQjYOE=
MIME-Version: 1.0
Received: by 10.101.189.34 with SMTP id r34mr4131407anp.77.1306883395465; Tue,
 31 May 2011 16:09:55 -0700 (PDT)
Sender: jdcryans@gmail.com
Received: by 10.100.239.20 with HTTP; Tue, 31 May 2011 16:09:55 -0700 (PDT)
In-Reply-To: <BANLkTi=gvZcNmge+b3Ez9YNmjuDNGZ_QFA@mail.gmail.com>
References: <BANLkTinnwuMy_STziezGfLrW6fxgJ-8dsA@mail.gmail.com>
	<BANLkTi=gvZcNmge+b3Ez9YNmjuDNGZ_QFA@mail.gmail.com>
Date: Tue, 31 May 2011 16:09:55 -0700
X-Google-Sender-Auth: MT_LUu9Gx3S-izf05WWfa_4NoHE
Message-ID: <BANLkTi=mWOHbK8GH8JBkHoGNiFRnnETNUg@mail.gmail.com>
Subject: Re: ANN: HBase 0.90.3 available for download
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: user@hbase.apache.org
Cc: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

It's all under the tags in the github repo: https://github.com/apache/hbase

J-D

On Tue, May 31, 2011 at 4:06 PM, Jack Levin <magnito@gmail.com> wrote:
> Hello, is there a git repo URL I could use to check out that code version=
?
>
> -Jack
>
> On Thu, May 19, 2011 at 2:35 PM, Stack <stack@duboce.net> wrote:
>> The Apache HBase team is happy to announce that HBase 0.90.3 is
>> available from the Apache mirror of choice:
>>
>> =A0http://www.apache.org/dyn/closer.cgi/hbase/
>>
>> HBase 0.90.3 is a maintenance release that fixes several important bugs
>> since version 0.90.3, while retaining API and data compatibility. The
>> release notes may be found on the Apache JIRA:
>>
>> =A0https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=3D12=
310753&version=3D12316313
>>
>> Users upgrading from previous versions of 0.90 may upgrade clients and s=
ervers
>> separately, though it is recommended that both be upgraded. If upgrading
>> from a version of HBase prior to 0.90.0, please read the notes accompany=
ing
>> that release:
>>
>> =A0http://osdir.com/ml/general-hadoop-apache/2011-01/msg00208.html
>>
>> As always, many thanks to those who contributed to this release!
>>
>> -The HBase Team
>>
>

From general-return-3589-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 23:12:09 2011
Return-Path: <general-return-3589-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 796BC4217
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 31 May 2011 23:12:09 +0000 (UTC)
Received: (qmail 82089 invoked by uid 500); 31 May 2011 23:12:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81931 invoked by uid 500); 31 May 2011 23:12:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 68303 invoked by uid 99); 31 May 2011 23:07:01 -0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of magnito@gmail.com designates 74.125.83.48 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:cc:content-type
         :content-transfer-encoding;
        bh=r9pg4QDd8rb5JwRtC6f8dZU+7tkJeoiu7rCfAneU/WY=;
        b=TTjwYcMmcNYc/kN5JTt2jeiPwKu+oMyHnOKM/Q9f9/0nCUKN+czevfhe0Oh0YTL8wy
         1F3YWghOPk8/pwC1pKCEaNQskDJLpIATRGmkddL8+nuQjOWDNOQgrchAViGUVapNoN/K
         q9apmEHRgdE9+3j1DZaw3cMGRIDTZq1uX02Ss=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type:content-transfer-encoding;
        b=ln4zOwMtzXsCCksakupzp31NxKx+wdYdx8HVtLu6M/M9h+P15e6kEBAZMjZignzlgf
         lHg8HJW5hksGFU1GGyIEOTB1kyp5e4dUOoNyfqKGyPc3omBbJkskL9EypaKte71DT6Dl
         xhIDV7QVkFOaw2NF6KZ4+JKpehKDDMZmRkk10=
MIME-Version: 1.0
In-Reply-To: <BANLkTinnwuMy_STziezGfLrW6fxgJ-8dsA@mail.gmail.com>
References: <BANLkTinnwuMy_STziezGfLrW6fxgJ-8dsA@mail.gmail.com>
Date: Tue, 31 May 2011 16:06:35 -0700
Message-ID: <BANLkTi=gvZcNmge+b3Ez9YNmjuDNGZ_QFA@mail.gmail.com>
Subject: Re: ANN: HBase 0.90.3 available for download
From: Jack Levin <magnito@gmail.com>
To: user@hbase.apache.org
Cc: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hello, is there a git repo URL I could use to check out that code version?

-Jack

On Thu, May 19, 2011 at 2:35 PM, Stack <stack@duboce.net> wrote:
> The Apache HBase team is happy to announce that HBase 0.90.3 is
> available from the Apache mirror of choice:
>
> =A0http://www.apache.org/dyn/closer.cgi/hbase/
>
> HBase 0.90.3 is a maintenance release that fixes several important bugs
> since version 0.90.3, while retaining API and data compatibility. The
> release notes may be found on the Apache JIRA:
>
> =A0https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=3D123=
10753&version=3D12316313
>
> Users upgrading from previous versions of 0.90 may upgrade clients and se=
rvers
> separately, though it is recommended that both be upgraded. If upgrading
> from a version of HBase prior to 0.90.0, please read the notes accompanyi=
ng
> that release:
>
> =A0http://osdir.com/ml/general-hadoop-apache/2011-01/msg00208.html
>
> As always, many thanks to those who contributed to this release!
>
> -The HBase Team
>

From general-return-3590-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 23:12:28 2011
Return-Path: <general-return-3590-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BE43A4250
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 31 May 2011 23:12:28 +0000 (UTC)
Received: (qmail 84057 invoked by uid 500); 31 May 2011 23:12:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83971 invoked by uid 500); 31 May 2011 23:12:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83924 invoked by uid 99); 31 May 2011 23:12:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 23:12:25 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [98.139.53.194] (HELO nm10-vm0.bullet.mail.ac4.yahoo.com) (98.139.53.194)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 31 May 2011 23:12:17 +0000
Received: from [98.139.52.197] by nm10.bullet.mail.ac4.yahoo.com with NNFMP; 31 May 2011 23:11:56 -0000
Received: from [98.139.52.133] by tm10.bullet.mail.ac4.yahoo.com with NNFMP; 31 May 2011 23:11:56 -0000
Received: from [127.0.0.1] by omp1016.mail.ac4.yahoo.com with NNFMP; 31 May 2011 23:11:56 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 324022.91035.bm@omp1016.mail.ac4.yahoo.com
Received: (qmail 9256 invoked by uid 60001); 31 May 2011 23:11:56 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1306883516; bh=21IIFxxy2bYm9XnKUPyknJ0dWmhqxh4g1spYis3zHbc=; h=Message-ID:X-YMail-OSG:Received:X-RocketYMMF:X-Mailer:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=bzzjVpyEQtwAzGBhCNy6ULRR7n0kLpkmcKE2FPj3Zbz2s/HCn47ALldyOthXQhLklgnhfCHKjLT/SEXQAOer/N+e2NmTswZZ3Ag2ECaVAJIrvGMggdNuFok1+FkEBbUOWDJ0zB1VBAKmP2pO1PGExYC7cPX1+wBlKeigOzKUkRQ=
Message-ID: <937378.9182.qm@web65503.mail.ac4.yahoo.com>
X-YMail-OSG: CxB9jqEVM1nQBlhFz7caBjJZrsysb7vGT.3Wb8EI3Bsvmvy
 28Rwn_ghYneRrv3EhPB.hwfhhjvRFDtqJwaPLos3B7zZwx60.jStzfKRsrRk
 1jph15pq93bnNOC8ye2NncEnn1Ivkc_QB_JIjKl36Tr6kEpSNfQRfB_0r53g
 4uYnaXxn..jQoR4avuhY5SBkiym9zKRNpzuKTYnF.8udJ_HcRs.GSdJ3WBQR
 CMWwwFM4Ycu85u45uHNwX48aGj53p_5tGvtd.cpbHvQGRW8et6Jtt6dnQsDC
 vIoTJjB2KUYnh33QmgPnB1DJr6lVQOTd9DTDjpAXa04ywVB0.9DeTXrpwfVo
 nlj3YCmy1tHDSdWY3CTchJJywzdwVxQUxlxQxDlyC2uyj1gfcFDA_ICNt0Bu
 pM2KDueU2eXyi6dpZdzAH2e7gRHJad_cORyN_ucHFRss4bDRqtJmzIUjNgag
 YXGNOSkxE.koiTOiNdYkoVb_gcDgZv2VqkFPVY2wHcycWmlaILUwO5t_P.nf
 PJVwR5r6lwc1tMp7KvbfxG14F
Received: from [64.168.229.50] by web65503.mail.ac4.yahoo.com via HTTP; Tue, 31 May 2011 16:11:55 PDT
X-RocketYMMF: apurtell
X-Mailer: YahooMailClassic/14.0.1 YahooMailWebService/0.8.111.303096
Date: Tue, 31 May 2011 16:11:55 -0700 (PDT)
From: Andrew Purtell <apurtell@apache.org>
Reply-To: apurtell@apache.org
Subject: Re: ANN: HBase 0.90.3 available for download
To: user@hbase.apache.org
Cc: general@hadoop.apache.org
In-Reply-To: <BANLkTi=gvZcNmge+b3Ez9YNmjuDNGZ_QFA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org

> From: Jack Levin <magnito@gmail.com>
> Hello, is there a git repo URL I could use to check out that
> code version?

git://git.apache.org/hbase.git

or 

git://github.com/apache/hbase.git

or

https://github.com/apache/hbase.git

Then checkout tag '0.90.3'

From general-return-1276-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 03:13:33 2010
Return-Path: <general-return-1276-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 15990 invoked from network); 1 Apr 2010 03:13:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 03:13:32 -0000
Received: (qmail 8349 invoked by uid 500); 1 Apr 2010 03:13:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8145 invoked by uid 500); 1 Apr 2010 03:13:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8117 invoked by uid 99); 1 Apr 2010 03:13:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 03:13:29 +0000
X-ASF-Spam-Status: No, hits=-0.1 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [65.115.85.73] (HELO smtp-outbound-2.vmware.com) (65.115.85.73)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 03:13:21 +0000
Received: from jupiter.vmware.com (mailhost5.vmware.com [10.16.68.131])
	by smtp-outbound-2.vmware.com (Postfix) with ESMTP id 5493918065;
	Wed, 31 Mar 2010 20:12:59 -0700 (PDT)
Received: from pa-excaht04.vmware.com (pa-excaht04.vmware.com [10.113.81.155])
	by jupiter.vmware.com (Postfix) with ESMTP id 4AAADDC168;
	Wed, 31 Mar 2010 20:12:59 -0700 (PDT)
Received: from EXCH-MBX-22.vmware.com ([10.113.81.177]) by
 pa-excaht04.vmware.com ([10.113.81.155]) with mapi; Wed, 31 Mar 2010 20:12:59
 -0700
From: Zhanlei Ma <zma@vmware.com>
To: "hive-user@hadoop.apache.org" <hive-user@hadoop.apache.org>,
	"common-user@hadoop.apache.org" <common-user@hadoop.apache.org>,
	"general@hadoop.apache.org" <general@hadoop.apache.org>,
	"mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>
Date: Wed, 31 Mar 2010 20:12:56 -0700
Subject: How to Recommission?
Thread-Topic: How to Recommission?
Thread-Index: AcrRSTnY7FRKEnlYTMunq01Wom7ORA==
Message-ID: <F21534570FE52940A8A9366D82C11E8B044E89A1D8@EXCH-MBX-22.vmware.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_F21534570FE52940A8A9366D82C11E8B044E89A1D8EXCHMBX22vmwa_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_F21534570FE52940A8A9366D82C11E8B044E89A1D8EXCHMBX22vmwa_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

How to Recommission or decommission DataNode(s) in hadoop???
Decommission(Del some Datanodes):
On a large cluster removing one or two data-nodes will not lead to any data=
 loss, because name-node will replicate their blocks as long as it

will detect that the nodes are dead. With a large number of nodes getting r=
emoved or dying the probability of losing data is higher.

Hadoop offers the decommission feature to retire a set of existing data-nod=
es. The nodes to be retired should be included into the exclude file,

and the exclude file name should be specified as a configuration parameter =
dfs.hosts.exclude. This file should have been specified during

namenode startup. It could be a zero length file. You must use the full hos=
tname, ip or ip:port format in this file. Then the shell command

bin/hadoop dfsadmin -refreshNodes

should be called, which forces the name-node to re-read the exclude file an=
d start the decommission process.

Decommission does not happen momentarily since it requires replication of p=
otentially a large number of blocks and we do not want the cluster to

be overwhelmed with just this one job. The decommission progress can be mon=
itored on the name-node Web UI. Until all blocks are replicated the

node will be in "Decommission In Progress" state. When decommission is done=
 the state will change to "Decommissioned". The nodes can be removed

whenever decommission is finished.



But how to Recommission? Wish your help.
Thanks.

--_000_F21534570FE52940A8A9366D82C11E8B044E89A1D8EXCHMBX22vmwa_--

From general-return-1277-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 03:18:01 2010
Return-Path: <general-return-1277-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16772 invoked from network); 1 Apr 2010 03:18:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 03:18:00 -0000
Received: (qmail 11358 invoked by uid 500); 1 Apr 2010 03:17:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11261 invoked by uid 500); 1 Apr 2010 03:17:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11251 invoked by uid 99); 1 Apr 2010 03:17:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 03:17:57 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_HELO_PASS,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [64.81.36.244] (HELO foobar.kobold.org) (64.81.36.244)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 03:17:49 +0000
Received: from owl.kobold.org (owl.kobold.org [192.168.1.7])
	(authenticated bits=0)
	by foobar.kobold.org (8.14.1/8.14.1) with ESMTP id o3120Xsq032186
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Wed, 31 Mar 2010 19:00:37 -0700
Message-ID: <4BB41041.9040409@hep.caltech.edu>
Date: Wed, 31 Mar 2010 20:17:21 -0700
From: Michael Thomas <thomas@hep.caltech.edu>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.8) Gecko/20100301 Fedora/3.0.3-1.fc12 Thunderbird/3.0.3
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: How to Recommission?
References: <F21534570FE52940A8A9366D82C11E8B044E89A1D8@EXCH-MBX-22.vmware.com>
In-Reply-To: <F21534570FE52940A8A9366D82C11E8B044E89A1D8@EXCH-MBX-22.vmware.com>
Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms020907020401030906050007"
X-Virus-Checked: Checked by ClamAV on apache.org

--------------ms020907020401030906050007
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: quoted-printable

On 03/31/2010 08:12 PM, Zhanlei Ma wrote:
> How to Recommission or decommission DataNode(s) in hadoop???
> Decommission(Del some Datanodes):
> On a large cluster removing one or two data-nodes will not lead to any =
data loss, because name-node will replicate their blocks as long as it
>
[...]
>
>
>
> But how to Recommission? Wish your help.
> Thanks.
>

Remove the hostname from your dfs.hosts.exclude file and run 'hadoop=20
dfsadmin -refreshNodes'.  Then start the datanode process in the=20
'recommissioned' datanode again.

--Mike


--------------ms020907020401030906050007
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIMVjCC
A/gwggLgoAMCAQICASkwDQYJKoZIhvcNAQEFBQAwdTETMBEGCgmSJomT8ixkARkWA25ldDES
MBAGCgmSJomT8ixkARkWAkVTMQ4wDAYDVQQKEwVFU25ldDEgMB4GA1UECxMXQ2VydGlmaWNh
dGUgQXV0aG9yaXRpZXMxGDAWBgNVBAMTD0VTbmV0IFJvb3QgQ0EgMTAeFw0wMjEyMDUwODAw
MDBaFw0xMzAxMjUwODAwMDBaMGkxEzARBgoJkiaJk/IsZAEZFgNvcmcxGDAWBgoJkiaJk/Is
ZAEZFghET0VHcmlkczEgMB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9yaXRpZXMxFjAUBgNV
BAMTDURPRUdyaWRzIENBIDEwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC09dYj
YaPbCD5mtbiQb7Ka3y1qAm0ZcqKCFciWcfe8Kwcuy9tjHuIsLf9ZItdkDW4xy8sua9nJlx3K
lwjtumTMtOtg35KZCknUd8KM4VGTSFdLVG9AbNayef76caVCGM1+jyF0Lq03kauGOPTcNfZe
1TZa3e1c9rc8ljV5OSWa/mfsCACyS5zFIWu0yIDNyJdf+n0hwaPN53wllpJ30taD+JBjQ7h2
k4xRWzeaznLOb9OztZVRA/1sVze+iczFh2xwa4VdGy0eIIPw1pfvYwxO36rm0S109qvbsNla
roPRbxerPKakQLpKe034Xcx7gBPqUk/FxoRRWin5EWN3rz9LAgMBAAGjgZ4wgZswDgYDVR0P
AQH/BAQDAgGGMBEGCWCGSAGG+EIBAQQEAwIAhzAdBgNVHQ4EFgQUyhkdEo5upDhdQtQxDgjb
2Y0XDV0wHwYDVR0jBBgwFoAUvF1NSC/4NZRZq1yJSz7RsjoUAeowDwYDVR0TAQH/BAUwAwEB
/zAlBgNVHREEHjAcgRpET0VHcmlkcy1DQS0xQGRvZWdyaWRzLm9yZzANBgkqhkiG9w0BAQUF
AAOCAQEAZNVrIDLqe39CEOiJt7Q7EpBPhAihMvDTSf/42u0SMbUmChww4mLmph5DBghZUVF8
Yn59kRZMn1QLOtO1HzLqvAvPITacZVPlJgG2IXzlR636YghZFAycbIUEOJDBHR4vtQO1KDxg
ZwvAbtmKIoxvhUCq2xsfFt9kCBBn+JYtQ6O5LsBJq3PmuubeMcc7mbQAfJZ7h/3QghgkFIhm
E1+LBXPJbkuP8vgfg6h2BKoAf5TFfZECgGZKimfN110tBvfedGZwYYd3/GsJc83B0JN1gny0
gqNVPm392UchXGeBRrHnm2gkhIkr48Oq6EmNGV9/a6XfbplQW/JWbtPVPWkaizCCBCkwggMR
oAMCAQICAwChZzANBgkqhkiG9w0BAQUFADBpMRMwEQYKCZImiZPyLGQBGRYDb3JnMRgwFgYK
CZImiZPyLGQBGRYIRE9FR3JpZHMxIDAeBgNVBAsTF0NlcnRpZmljYXRlIEF1dGhvcml0aWVz
MRYwFAYDVQQDEw1ET0VHcmlkcyBDQSAxMB4XDTEwMDEyNTE4MzEyMFoXDTExMDEyNTE4MzEy
MFowYDETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCGRvZWdyaWRzMQ8w
DQYDVQQLEwZQZW9wbGUxHjAcBgNVBAMTFU1pY2hhZWwgVGhvbWFzIDcyNTM3NDCCASIwDQYJ
KoZIhvcNAQEBBQADggEPADCCAQoCggEBALchCLXStiJKw5u2+nFW0020jqo1X8/kRB9kus8a
6Op5ds3j5O3IZkyZ41tQwE9Rc0yRDiDGpT1jL/n2LK1HPXvZbasmPCxxTMY3kNTm4fWAnRS7
UzLo0Adaz81bcT8Buo7Il1FCS684LhsK19JChqU2/iY8G7H0uBImj6QDLR0+w5QsLfD7aOXx
M8Njhse1ZqMyjwjhw5FqWyWNhlKi1oW42eXtYhbsqyAQZSXLoz2SwqBnLTRqcRMn4GgvB83Z
1FObyb79HFiuqlbaMlXWy0s8ywajJnMG8LsgwkNJZQLJsPalz5rwe/pOG1abasLMBVpJGGSB
AQktY8s5M+U9RlMCAwEAAaOB4jCB3zARBglghkgBhvhCAQEEBAMCBeAwDgYDVR0PAQH/BAQD
AgTwMDYGA1UdIAQvMC0wDQYLKoZIhvdMAwcBAwEwDAYKKoZIhvdMBQICATAOBgwqhkiG90wF
AgMCAQEwPgYDVR0fBDcwNTAzoDGgL4YtaHR0cDovL2NybC5kb2Vncmlkcy5vcmcvMWMzZjJj
YTgvMWMzZjJjYTguY3JsMCEGA1UdEQQaMBiBFnRob21hc0BoZXAuY2FsdGVjaC5lZHUwHwYD
VR0jBBgwFoAUyhkdEo5upDhdQtQxDgjb2Y0XDV0wDQYJKoZIhvcNAQEFBQADggEBAGcleFpm
d9Ho37gHZrrF0zulSQ3Aemrbx2O4EVJotQbtAs2v4nnH0eAa0F37BXRA6DAvgaC4YODA6z5M
HHE2mnsIebv0nS8DLj1yN7C1nqDSUXlaIzExEGQ1vXG2XlVo6pMZgJd2ML9+dyUUOcnNj0kQ
GeMysWBHbDChdfVG8y2PuGzoFY/ikNtWWhTNcPVZHIL5I2JCYqV5lyLQhlkC0Q5eqbqYKAq1
GuQmLO8hPfzkb6qbAkzxB8/FFqjlPgBgFcd9Nf2fAI14kI77iRbEuu8jSTPx13p6uYvcX4hy
1cW4Y7lnE8bBkAFWrUuNGc+aw7Ang6rJANWXtF/tN+UqUOcwggQpMIIDEaADAgECAgMAoWcw
DQYJKoZIhvcNAQEFBQAwaTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkW
CERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UEAxMN
RE9FR3JpZHMgQ0EgMTAeFw0xMDAxMjUxODMxMjBaFw0xMTAxMjUxODMxMjBaMGAxEzARBgoJ
kiaJk/IsZAEZFgNvcmcxGDAWBgoJkiaJk/IsZAEZFghkb2VncmlkczEPMA0GA1UECxMGUGVv
cGxlMR4wHAYDVQQDExVNaWNoYWVsIFRob21hcyA3MjUzNzQwggEiMA0GCSqGSIb3DQEBAQUA
A4IBDwAwggEKAoIBAQC3IQi10rYiSsObtvpxVtNNtI6qNV/P5EQfZLrPGujqeXbN4+TtyGZM
meNbUMBPUXNMkQ4gxqU9Yy/59iytRz172W2rJjwscUzGN5DU5uH1gJ0Uu1My6NAHWs/NW3E/
AbqOyJdRQkuvOC4bCtfSQoalNv4mPBux9LgSJo+kAy0dPsOULC3w+2jl8TPDY4bHtWajMo8I
4cORalsljYZSotaFuNnl7WIW7KsgEGUly6M9ksKgZy00anETJ+BoLwfN2dRTm8m+/RxYrqpW
2jJV1stLPMsGoyZzBvC7IMJDSWUCybD2pc+a8Hv6ThtWm2rCzAVaSRhkgQEJLWPLOTPlPUZT
AgMBAAGjgeIwgd8wEQYJYIZIAYb4QgEBBAQDAgXgMA4GA1UdDwEB/wQEAwIE8DA2BgNVHSAE
LzAtMA0GCyqGSIb3TAMHAQMBMAwGCiqGSIb3TAUCAgEwDgYMKoZIhvdMBQIDAgEBMD4GA1Ud
HwQ3MDUwM6AxoC+GLWh0dHA6Ly9jcmwuZG9lZ3JpZHMub3JnLzFjM2YyY2E4LzFjM2YyY2E4
LmNybDAhBgNVHREEGjAYgRZ0aG9tYXNAaGVwLmNhbHRlY2guZWR1MB8GA1UdIwQYMBaAFMoZ
HRKObqQ4XULUMQ4I29mNFw1dMA0GCSqGSIb3DQEBBQUAA4IBAQBnJXhaZnfR6N+4B2a6xdM7
pUkNwHpq28djuBFSaLUG7QLNr+J5x9HgGtBd+wV0QOgwL4GguGDgwOs+TBxxNpp7CHm79J0v
Ay49cjewtZ6g0lF5WiMxMRBkNb1xtl5VaOqTGYCXdjC/fnclFDnJzY9JEBnjMrFgR2wwoXX1
RvMtj7hs6BWP4pDbVloUzXD1WRyC+SNiQmKleZci0IZZAtEOXqm6mCgKtRrkJizvIT385G+q
mwJM8QfPxRao5T4AYBXHfTX9nwCNeJCO+4kWxLrvI0kz8dd6ermL3F+IctXFuGO5ZxPGwZAB
Vq1LjRnPmsOwJ4OqyQDVl7Rf7TflKlDnMYIDXjCCA1oCAQEwcDBpMRMwEQYKCZImiZPyLGQB
GRYDb3JnMRgwFgYKCZImiZPyLGQBGRYIRE9FR3JpZHMxIDAeBgNVBAsTF0NlcnRpZmljYXRl
IEF1dGhvcml0aWVzMRYwFAYDVQQDEw1ET0VHcmlkcyBDQSAxAgMAoWcwCQYFKw4DAhoFAKCC
AcMwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTAwNDAxMDMx
NzIxWjAjBgkqhkiG9w0BCQQxFgQUip2eR4Jlx75fxDsJTGHCO0hvMT4wXwYJKoZIhvcNAQkP
MVIwUDALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3
DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMH8GCSsGAQQBgjcQBDFyMHAwaTETMBEG
CgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERPRUdyaWRzMSAwHgYDVQQLExdD
ZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3JpZHMgQ0EgMQIDAKFnMIGB
BgsqhkiG9w0BCRACCzFyoHAwaTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixk
ARkWCERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UE
AxMNRE9FR3JpZHMgQ0EgMQIDAKFnMA0GCSqGSIb3DQEBAQUABIIBADJzxwW5decknac3qoOs
WbQBigXcAKWp7FZyiSoKyEsfeiRthkm7vYWnVBfJMmXm0NBfJnU4r3Oe59S8sgtWHHLbtMJF
AhYpuY9DleTFZwh+4IFeSjtB/RKDrFmrNGFeg+1yzxX3qS0lTFc5HLw7zFcKZ0jiBao02sVM
IWzARTB4itUERfCbBxUCdfRDJ6DSm6Zh/LZz4y9bAd8VuK9KvWoqM6TR2ezMOklyGALw3xBT
YbB5X28gBuwojEqn/dLW58oDQlolpmqTE2ldkkAud1sbUDQ3Zw8Kx3wHof69D2FDQZAPZ4Rv
mXYx+hpbsE0qzNfWKojjaQIKiTqQW4eP9yAAAAAAAAA=
--------------ms020907020401030906050007--

From general-return-1278-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 03:27:08 2010
Return-Path: <general-return-1278-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17890 invoked from network); 1 Apr 2010 03:27:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 03:27:08 -0000
Received: (qmail 20187 invoked by uid 500); 1 Apr 2010 03:27:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18709 invoked by uid 500); 1 Apr 2010 03:27:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18679 invoked by uid 99); 1 Apr 2010 03:27:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 03:27:03 +0000
X-ASF-Spam-Status: No, hits=-3.1 required=10.0
	tests=AWL,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 03:26:58 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:Received:Received:From:To:Subject:
   Thread-Topic:Thread-Index:Date:Message-ID:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:user-agent:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version:Return-Path:
   X-OriginalArrivalTime;
  b=Sa1Md/x6lFIj0GTYl4JUuVISsHfHAz8bF1RW9TObHO0pTKJGjP3poG8z
   DjaFVGg2FNm9/gVbX8Y53OtAYu78jlDPtDd53OWCJ6zGx3HLMoneTEu2c
   TEzWwb14odrRCN3;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1270092418; x=1301628418;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20How=20to=20Recommission?|Date:=20Thu,
   =201=20Apr=202010=2003:20:38=20+0000|Message-ID:=20<C7D95
   F16.3F8B%awittenauer@linkedin.com>|To:=20"general@hadoop.
   apache.org"=20<general@hadoop.apache.org>,=0D=0A=09"hive-
   user@hadoop.apache.org"=20<hive-user@hadoop.apache.org>,
   =0D=0A=09"common-user@hadoop.apache.org"=20<common-user@h
   adoop.apache.org>,=0D=0A=09"mapreduce-user@hadoop.apache.
   org"=20<mapreduce-user@hadoop.apache.org>|MIME-Version:
   =201.0|Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<486a03af-953f-4ae2-8681-00a70a3fffde>
   |In-Reply-To:=20<F21534570FE52940A8A9366D82C11E8B044E89A1
   D8@EXCH-MBX-22.vmware.com>;
  bh=Ega8eF5n3LkNYSe82fbahFnCfB1ZiL/gb5SM2Mlh2jo=;
  b=XsmWKSJ7nCb7aoeb/P1BZUXqpPtdCoyw+/rAFCnaQ/Zl1hY6oWXl90Bt
   ADOFFZppCqeG5A2GEDwo2uyMKQ9mBmPmS/lk80uxo0j3NXQzY5IOw9+jl
   zxJR7QN/SPn1sLo;
X-IronPort-AV: E=Sophos;i="4.51,345,1267430400"; 
   d="scan'208";a="11709058"
Received: from esv4-exctest.linkedin.biz ([172.18.46.60]) by CORP-MAIL.linkedin.biz with Microsoft SMTPSVC(6.0.3790.3959);
	 Wed, 31 Mar 2010 20:20:40 -0700
Received: from ESV4-CAS01.linkedin.biz (172.18.46.140) by
 esv4-exctest.linkedin.biz (172.18.46.60) with Microsoft SMTP Server (TLS) id
 14.0.682.1; Wed, 31 Mar 2010 20:20:39 -0700
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi; Wed, 31 Mar 2010
 20:20:39 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>,
	"hive-user@hadoop.apache.org" <hive-user@hadoop.apache.org>,
	"common-user@hadoop.apache.org" <common-user@hadoop.apache.org>,
	"mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>
Subject: Re: How to Recommission?
Thread-Topic: How to Recommission?
Thread-Index: AcrRSTnY7FRKEnlYTMunq01Wom7ORAAARL7o
Date: Thu, 1 Apr 2010 03:20:38 +0000
Message-ID: <C7D95F16.3F8B%awittenauer@linkedin.com>
In-Reply-To: <F21534570FE52940A8A9366D82C11E8B044E89A1D8@EXCH-MBX-22.vmware.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-Entourage/13.4.0.100208
Content-Type: text/plain; charset="us-ascii"
Content-ID: <486a03af-953f-4ae2-8681-00a70a3fffde>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginalArrivalTime: 01 Apr 2010 03:20:40.0229 (UTC) FILETIME=[4E24A550:01CAD14A]




On 3/31/10 8:12 PM, "Zhanlei Ma" <zma@vmware.com> wrote:

> But how to Recommission? Wish your help.

Take them out of dfs.exclude and refreshnodes again.



From general-return-1279-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 03:29:07 2010
Return-Path: <general-return-1279-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 18170 invoked from network); 1 Apr 2010 03:29:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 03:29:06 -0000
Received: (qmail 23155 invoked by uid 500); 1 Apr 2010 03:29:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22975 invoked by uid 500); 1 Apr 2010 03:29:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22967 invoked by uid 99); 1 Apr 2010 03:29:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 03:29:05 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [65.115.85.69] (HELO smtp-outbound-1.vmware.com) (65.115.85.69)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 03:28:58 +0000
Received: from mailhost3.vmware.com (mailhost3.vmware.com [10.16.27.45])
	by smtp-outbound-1.vmware.com (Postfix) with ESMTP id C55B817002
	for <general@hadoop.apache.org>; Wed, 31 Mar 2010 20:28:36 -0700 (PDT)
Received: from pa-excaht03.vmware.com (pa-excaht03.vmware.com [10.113.81.154])
	by mailhost3.vmware.com (Postfix) with ESMTP id B93C4CD902
	for <general@hadoop.apache.org>; Wed, 31 Mar 2010 20:28:36 -0700 (PDT)
Received: from EXCH-MBX-22.vmware.com ([10.113.81.177]) by
 pa-excaht03.vmware.com ([10.113.81.154]) with mapi; Wed, 31 Mar 2010 20:28:36
 -0700
From: Zhanlei Ma <zma@vmware.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 31 Mar 2010 20:28:34 -0700
Subject: RE: How to Recommission?
Thread-Topic: How to Recommission?
Thread-Index: AcrRSe9fULyTds46RvWIhcIWc1eEggAAXEsg
Message-ID: <F21534570FE52940A8A9366D82C11E8B044E89A206@EXCH-MBX-22.vmware.com>
References: <F21534570FE52940A8A9366D82C11E8B044E89A1D8@EXCH-MBX-22.vmware.com>
 <4BB41041.9040409@hep.caltech.edu>
In-Reply-To: <4BB41041.9040409@hep.caltech.edu>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="gb2312"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

SSBnb3QgaXQsVGhhbmtzDQoNCi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQpGcm9tOiBNaWNo
YWVsIFRob21hcyBbbWFpbHRvOnRob21hc0BoZXAuY2FsdGVjaC5lZHVdIA0KU2VudDogMjAxMMTq
NNTCMcjVIDExOjE3DQpUbzogZ2VuZXJhbEBoYWRvb3AuYXBhY2hlLm9yZw0KU3ViamVjdDogUmU6
IEhvdyB0byBSZWNvbW1pc3Npb24/DQoNCk9uIDAzLzMxLzIwMTAgMDg6MTIgUE0sIFpoYW5sZWkg
TWEgd3JvdGU6DQo+IEhvdyB0byBSZWNvbW1pc3Npb24gb3IgZGVjb21taXNzaW9uIERhdGFOb2Rl
KHMpIGluIGhhZG9vcD8/Pw0KPiBEZWNvbW1pc3Npb24oRGVsIHNvbWUgRGF0YW5vZGVzKToNCj4g
T24gYSBsYXJnZSBjbHVzdGVyIHJlbW92aW5nIG9uZSBvciB0d28gZGF0YS1ub2RlcyB3aWxsIG5v
dCBsZWFkIHRvIGFueSBkYXRhIGxvc3MsIGJlY2F1c2UgbmFtZS1ub2RlIHdpbGwgcmVwbGljYXRl
IHRoZWlyIGJsb2NrcyBhcyBsb25nIGFzIGl0DQo+DQpbLi4uXQ0KPg0KPg0KPg0KPiBCdXQgaG93
IHRvIFJlY29tbWlzc2lvbj8gV2lzaCB5b3VyIGhlbHAuDQo+IFRoYW5rcy4NCj4NCg0KUmVtb3Zl
IHRoZSBob3N0bmFtZSBmcm9tIHlvdXIgZGZzLmhvc3RzLmV4Y2x1ZGUgZmlsZSBhbmQgcnVuICdo
YWRvb3AgDQpkZnNhZG1pbiAtcmVmcmVzaE5vZGVzJy4gIFRoZW4gc3RhcnQgdGhlIGRhdGFub2Rl
IHByb2Nlc3MgaW4gdGhlIA0KJ3JlY29tbWlzc2lvbmVkJyBkYXRhbm9kZSBhZ2Fpbi4NCg0KLS1N
aWtlDQoNCg==

From general-return-1280-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 11:20:20 2010
Return-Path: <general-return-1280-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 7683 invoked from network); 1 Apr 2010 11:20:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 11:20:20 -0000
Received: (qmail 48865 invoked by uid 500); 1 Apr 2010 11:20:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48697 invoked by uid 500); 1 Apr 2010 11:20:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48688 invoked by uid 99); 1 Apr 2010 11:20:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 11:20:17 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=AWL,HTML_MESSAGE,HTML_OBFUSCATE_05_10,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 11:20:10 +0000
Received: from EGL-EX07CAS02.ds.corp.yahoo.com (egl-ex07cas02.eglbp.corp.yahoo.com [203.83.248.209])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o31BIrTG011233
	for <general@hadoop.apache.org>; Thu, 1 Apr 2010 04:18:53 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:accept-language:content-language:x-ms-has-attach:
	x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=Gs3GWFWEAAKYk3YppWn1KR7+XmRHRq61y9G+PIwrwZktgBrNSQreJU6D9gOUw3eR
Received: from EGL-EX07VS01.ds.corp.yahoo.com ([203.83.248.205]) by
 EGL-EX07CAS02.ds.corp.yahoo.com ([203.83.248.216]) with mapi; Thu, 1 Apr 2010
 16:48:52 +0530
From: "Giridharan  Kesavan" <gkesavan@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 1 Apr 2010 16:48:51 +0530
Subject: HADOOP-6671 - To use maven for hadoop common builds
Thread-Topic: HADOOP-6671 - To use maven for hadoop common builds
Thread-Index: AcrRjRssjAQqHsuAH0CAy/QdN8LW6A==
Message-ID: <C7DA7EF3.119B1%gkesavan@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C7DA7EF3119B1gkesavanyahooinccom_"
MIME-Version: 1.0

--_000_C7DA7EF3119B1gkesavanyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Hello,

I 've filed this jira "https://issues.apache.org/jira/browse/HADOOP-6671" f=
or porting hadoop-common builds to maven.
Please let me know of your thoughts and suggestion, if any.

Thanks,
giri

--_000_C7DA7EF3119B1gkesavanyahooinccom_--

From general-return-1281-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 15:06:11 2010
Return-Path: <general-return-1281-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 86645 invoked from network); 1 Apr 2010 15:06:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 15:06:11 -0000
Received: (qmail 97457 invoked by uid 500); 1 Apr 2010 15:06:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97388 invoked by uid 500); 1 Apr 2010 15:06:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97380 invoked by uid 99); 1 Apr 2010 15:06:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 15:06:09 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=FREEMAIL_FROM,SPF_NEUTRAL,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [98.136.183.74] (HELO web114514.mail.gq1.yahoo.com) (98.136.183.74)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 01 Apr 2010 15:06:02 +0000
Received: (qmail 74635 invoked by uid 60001); 1 Apr 2010 15:05:39 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ymail.com; s=s1024; t=1270134339; bh=hkrmBdXuS2DsC58ZcyHrghxVNwM+IfKbAKcJhVCQ5ZA=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=FU+7mLAbcd4HQXUR3cNwfIlGGdrwk6Zg0hRfBAJMBMJtFsYuJelJa68g3HDeDHB10QA8z0NKwfEGpKaXAN1PRrsolV9DZw7wUa+WgHj0p6LaXIu3opYJjX2Ltt4erM89GtA6WZ2PXLuh7mcNd30kJTMxzlaSjjzcTNcZ9QY4byc=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=ymail.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=poG8GSGEfpzAbu3xMlCHNG6WMy69K1GYW+wvembJX0X17cQZ3S7oQWYZGKkGcpfvj9wPgwzXOWOlqOR5krMyoxhp0t6QZEiT3rSsS9pcv1NKynwM7sdsr3eveFezWb1Q/lxUBt57rS7WZoZuq8mQBco4QJjrAwLb/N8ZawdRoMg=;
Message-ID: <629658.74161.qm@web114514.mail.gq1.yahoo.com>
X-YMail-OSG: m9BqknUVM1l3h6dGWvNV6qG5y5lXOWLCweqiay_p.TWath1
 UoSxi19Y7kAQddbd0klO8G6SxdSYRhoq578FXNwrJydBzORc0MnZWkbIuuhx
 z8TtKEWnGCLrq2xqcp44SjK7t.a.Yv5LpkKTuwcQDA9Z6y3QNnuOm9EzLKrb
 J49YROc4j2AJ3oM_naSyHh0nkZCB2UcIlRwzQlPncFaorPOBubn5nOf_VW1u
 LJ9SAWr03fbsexQ2Se4h8w00XSVNwxAxTJSGcjv.cIrZ9dd0qG4vjBCaWv8h
 7sSvk2jfFHDrbnXUc4w.O.sKUOzBGidhWgBBvjANfoIgUHHR91GOlfPRDXWA
 bK9mW4kQdQ01RwNGUE8Tph5yxD4cUjOzM_C.8mR2ozmTwxnXI_AAYLHDbiR3
 oTT2Lpfglat14sz1SCt3dE5IKiSvUbZYH3aXMbQ2I_0_km45KKkcSMm.MYma
 wisy_WLemT9dHw_DHi1H9H6kPPD6FknbfDmah0yqViunpfkXGIjKjfXcSaNQ
 KHSarGzMECUNMGTkgFF9dyUXHCNd4sGOSLw--
Received: from [72.177.134.58] by web114514.mail.gq1.yahoo.com via HTTP; Thu, 01 Apr 2010 08:05:39 PDT
X-Mailer: YahooMailRC/324.3 YahooMailWebService/0.8.100.260964
Date: Thu, 1 Apr 2010 08:05:39 -0700 (PDT)
From: Bill Abernathy <billabernathy@ymail.com>
Subject: [VOTE] Replace HBase with Cassandra
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org


I propose a vote to replace HBase as a Hadoop sub-project with Cassandra.

Why members should vote for this proposal:

* Twitter is using Cassandra
* Cassandra sounds like someone you'd trust your kids with, "HBase" sounds like a pathogen.
* You'll never have to kill -9 a namenode
* The Cassandra logo looks better on tshirts
* Hadoop has two sub-projects beginning with "H" and none with "C"
* Twitter is using Cassandra

The vote will remain open for 72 hours.

Thanks,

--
Bill Abernathy
billabernathy@ymail.com



      

From general-return-1282-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 15:12:53 2010
Return-Path: <general-return-1282-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88287 invoked from network); 1 Apr 2010 15:12:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 15:12:52 -0000
Received: (qmail 3055 invoked by uid 500); 1 Apr 2010 15:12:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3013 invoked by uid 500); 1 Apr 2010 15:12:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3005 invoked by uid 99); 1 Apr 2010 15:12:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 15:12:51 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 15:12:46 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:Received:Received:From:To:Subject:
   Thread-Topic:Thread-Index:Date:Message-ID:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:user-agent:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version:Return-Path:
   X-OriginalArrivalTime;
  b=AC97mtE9fCpYc6zNOVD9W8lSj/e5UWh/+9+tBdrDrvkBYV4b+/ZbNLoD
   iFkVz+WeYCpCwm3IdZldMjN8jBeDRjUdavvr2wYqA2kK520fTWk3dblSl
   uKKqwTyq0VxGh0J;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1270134765; x=1301670765;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20[VOTE]=20Replace=20HBase=20with=20Cassa
   ndra|Date:=20Thu,=201=20Apr=202010=2015:12:13=20+0000
   |Message-ID:=20<C7DA05DD.3FAA%awittenauer@linkedin.com>
   |To:=20"general@hadoop.apache.org"=20<general@hadoop.apac
   he.org>|MIME-Version:=201.0|Content-Transfer-Encoding:=20
   quoted-printable|Content-ID:=20<906e227a-7982-4af3-b115-6
   7ae87a7c22d>|In-Reply-To:=20<629658.74161.qm@web114514.ma
   il.gq1.yahoo.com>;
  bh=c+ZNEmLtst2+AYdJGuMI0JDUiA5BnV1D+5KiWY0MZHI=;
  b=dLadOi1lWSvSBoXjqmSbAyjClD2LdXgGHbgwXDlvNNUelZ0iesq6/XO4
   X/UiN1VI9/jCxkLEgyYlSNdqlMW6Ck6zc8a8n38bggfCvE/oYqqcXzQTA
   1gi2X5OJmGw1/eg;
X-IronPort-AV: E=Sophos;i="4.51,348,1267430400"; 
   d="scan'208";a="11716669"
Received: from esv4-exctest.linkedin.biz ([172.18.46.60]) by CORP-MAIL.linkedin.biz with Microsoft SMTPSVC(6.0.3790.3959);
	 Thu, 1 Apr 2010 08:12:15 -0700
Received: from ESV4-CAS01.linkedin.biz (172.18.46.140) by
 esv4-exctest.linkedin.biz (172.18.46.60) with Microsoft SMTP Server (TLS) id
 14.0.682.1; Thu, 1 Apr 2010 08:12:14 -0700
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi; Thu, 1 Apr 2010 08:12:14
 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: [VOTE] Replace HBase with Cassandra
Thread-Topic: [VOTE] Replace HBase with Cassandra
Thread-Index: AQHK0azd5qLg+2GMuk6gobeVJdxv+pINuBCA
Date: Thu, 1 Apr 2010 15:12:13 +0000
Message-ID: <C7DA05DD.3FAA%awittenauer@linkedin.com>
In-Reply-To: <629658.74161.qm@web114514.mail.gq1.yahoo.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-Entourage/13.4.0.100208
Content-Type: text/plain; charset="us-ascii"
Content-ID: <906e227a-7982-4af3-b115-67ae87a7c22d>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginalArrivalTime: 01 Apr 2010 15:12:15.0369 (UTC) FILETIME=[B66DFF90:01CAD1AD]
X-Virus-Checked: Checked by ClamAV on apache.org


I'm turning off my email today.


On 4/1/10 8:05 AM, "Bill Abernathy" <billabernathy@ymail.com> wrote:

>=20
> I propose a vote to replace HBase as a Hadoop sub-project with Cassandra.
>=20
> Why members should vote for this proposal:
>=20
> * Twitter is using Cassandra
> * Cassandra sounds like someone you'd trust your kids with, "HBase" sound=
s
> like a pathogen.
> * You'll never have to kill -9 a namenode
> * The Cassandra logo looks better on tshirts
> * Hadoop has two sub-projects beginning with "H" and none with "C"
> * Twitter is using Cassandra
>=20
> The vote will remain open for 72 hours.
>=20
> Thanks,
>=20
> --
> Bill Abernathy
> billabernathy@ymail.com
>=20
>=20
>=20
>      =20


From general-return-1283-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 15:20:00 2010
Return-Path: <general-return-1283-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97309 invoked from network); 1 Apr 2010 15:19:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 15:19:59 -0000
Received: (qmail 18019 invoked by uid 500); 1 Apr 2010 15:19:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17979 invoked by uid 500); 1 Apr 2010 15:19:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17970 invoked by uid 99); 1 Apr 2010 15:19:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 15:19:58 +0000
X-ASF-Spam-Status: No, hits=2.5 required=10.0
	tests=FREEMAIL_FROM,FREEMAIL_REPLY,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of owen.omalley@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 15:19:52 +0000
Received: by pwi7 with SMTP id 7so1109897pwi.35
        for <general@hadoop.apache.org>; Thu, 01 Apr 2010 08:19:31 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:from:to
         :in-reply-to:content-type:content-transfer-encoding:x-mailer
         :mime-version:subject:date:references;
        bh=lVhI50R2MRBSTBvCWcV8jZuQgaglm078ZPfyl4j+KYs=;
        b=FvsTlEbGKh9j4hUHXsyuNMu6+PzfZo2eIRid/8LzcSLznAox+UGUGBXb16M0UUsr+4
         V/UB5aviDVpy6VDyielJ9fw/XwrAqW/5XALxX/d1edCvQ31kYNFelidOyaVGaIh7MXa1
         ISYIcG5X/1EY7HJAaLn3beULtH24jndeevRCw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:from:to:in-reply-to:content-type
         :content-transfer-encoding:x-mailer:mime-version:subject:date
         :references;
        b=SBcBFNUKPAaQFNYaxWKuoQuVQUnD6DOH40Ocx/6reR2sSNgomvAhVtUsvIE+zJRIts
         4f2yHFhoXRLx/pVjhGXNjX2qLn2LxcjZQ1+TZt/gPwlqYr8BhMVV8MSsu1RCPijiqqIO
         TM9fBwUM09gfHVp7LB0CXRhbrdOiXr9+D1H5o=
Received: by 10.141.108.19 with SMTP id k19mr565499rvm.110.1270135170723;
        Thu, 01 Apr 2010 08:19:30 -0700 (PDT)
Received: from [10.14.254.105] ([166.205.138.131])
        by mx.google.com with ESMTPS id 20sm6949288pzk.3.2010.04.01.08.19.28
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Thu, 01 Apr 2010 08:19:29 -0700 (PDT)
Message-Id: <90F2C2FE-BDDC-4BEC-901D-9BE917107BDF@gmail.com>
From: Owen O'Malley <owen.omalley@gmail.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <C7DA05DD.3FAA%awittenauer@linkedin.com>
Content-Type: text/plain;
	charset=us-ascii;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit
X-Mailer: iPhone Mail (7E18)
Mime-Version: 1.0 (iPhone Mail 7E18)
Subject: Re: [VOTE] Replace HBase with Cassandra
Date: Thu, 1 Apr 2010 08:19:21 -0700
References: <C7DA05DD.3FAA%awittenauer@linkedin.com>
X-Virus-Checked: Checked by ClamAV on apache.org

I would have gone with the starting line, "Since HBase will soon  
become an Apache top level project..."

-- Owen

On Apr 1, 2010, at 8:12 AM, Allen Wittenauer  
<awittenauer@linkedin.com> wrote:

>
> I'm turning off my email today.
>
>
> On 4/1/10 8:05 AM, "Bill Abernathy" <billabernathy@ymail.com> wrote:
>
>>
>> I propose a vote to replace HBase as a Hadoop sub-project with  
>> Cassandra.
>>
>> Why members should vote for this proposal:
>>
>> * Twitter is using Cassandra
>> * Cassandra sounds like someone you'd trust your kids with, "HBase"  
>> sounds
>> like a pathogen.
>> * You'll never have to kill -9 a namenode
>> * The Cassandra logo looks better on tshirts
>> * Hadoop has two sub-projects beginning with "H" and none with "C"
>> * Twitter is using Cassandra
>>
>> The vote will remain open for 72 hours.
>>
>> Thanks,
>>
>> --
>> Bill Abernathy
>> billabernathy@ymail.com
>>
>>
>>
>>
>

From general-return-1284-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 02 18:59:47 2010
Return-Path: <general-return-1284-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38660 invoked from network); 2 Apr 2010 18:59:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Apr 2010 18:59:47 -0000
Received: (qmail 78516 invoked by uid 500); 2 Apr 2010 18:59:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77876 invoked by uid 500); 2 Apr 2010 18:59:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77654 invoked by uid 99); 2 Apr 2010 18:59:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 18:59:42 +0000
X-ASF-Spam-Status: No, hits=1.0 required=10.0
	tests=AWL,HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 18:59:36 +0000
Received: from SP1-EX07CAS02.ds.corp.yahoo.com (sp1-ex07cas02.ds.corp.yahoo.com [216.252.116.138])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o32IwEVR050459;
	Fri, 2 Apr 2010 11:58:14 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:accept-language:content-language:x-ms-has-attach:
	x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=Rt0JJfESr8EYhvBHk0+P/GrjA8E25RX6NfZVw6pDReqBlqB98uLlE4REevqUxlbC
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 SP1-EX07CAS02.ds.corp.yahoo.com ([216.252.116.167]) with mapi; Fri, 2 Apr
 2010 11:58:14 -0700
From: Dekel Tankel <dekel@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>,
        "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Date: Fri, 2 Apr 2010 11:58:13 -0700
Subject: Hadoop Bay Area User Group - April 21st -  Agenda and registration
 available!
Thread-Topic: Hadoop Bay Area User Group - April 21st -  Agenda and
 registration available!
Thread-Index: AcrSlnIpAEqUnd84RWOGjIrQGwpYFA==
Message-ID: <46E2672895E8744A97D7577A6110858B4FE055C949@SP1-EX07VS01.ds.corp.yahoo.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_46E2672895E8744A97D7577A6110858B4FE055C949SP1EX07VS01ds_"
MIME-Version: 1.0

--_000_46E2672895E8744A97D7577A6110858B4FE055C949SP1EX07VS01ds_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi all,

RSVP is open for the next monthly Bay Area Hadoop user group at the Yahoo! =
Sunnyvale Campus, Wednesday, 21st, 6PM

Registration and Agenda are available here http://www.meetup.com/hadoop/cal=
endar/13002132/

Looking forward to seeing you there!

Dekel


--_000_46E2672895E8744A97D7577A6110858B4FE055C949SP1EX07VS01ds_--

From general-return-1285-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 02 22:26:25 2010
Return-Path: <general-return-1285-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77508 invoked from network); 2 Apr 2010 22:26:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Apr 2010 22:26:25 -0000
Received: (qmail 74967 invoked by uid 500); 2 Apr 2010 22:26:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74850 invoked by uid 500); 2 Apr 2010 22:26:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 60331 invoked by uid 99); 2 Apr 2010 09:33:14 -0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
X-pt: isis.lip6.fr
X-Authentication-Warning: ari-31-201-04.infop6.jussieu.fr: 2961083 set sender to jonathan.lejeune@etu.upmc.fr using -f
Subject: Project for studies
From: LEJEUNE JONATHAN <jonathan.lejeune@etu.upmc.fr>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Date: Fri, 02 Apr 2010 12:32:17 +0200
Message-Id: <1270204337.19663.16.camel@ari-31-201-04.infop6.jussieu.fr>
Mime-Version: 1.0
X-Mailer: Evolution 2.12.3 (2.12.3-8.el5_2.3) 
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.3 (isis.lip6.fr [132.227.60.2]); Fri, 02 Apr 2010 11:32:43 +0200 (CEST)
X-Spam-Score: 8 (********) M_LOTO
X-Spam-Report: Content analysis details:   (8.0 points, 3.0 required)
                pts rule name              description
               --- ---------              -----------
                8.0 M_LOTO                 From: une lotterie ou un jeu
X-Virus-Checked: Checked by ClamAV on apache.org
X-Old-Spam-Status: Yes, hits=8 required=3
               
X-Scanned-By: MIMEDefang 2.63 on 132.227.60.2

Dear project members,

We are two french students from Universit=C3=A9 Pierre et Marie Curie (Pari=
s
6), and we are working on the Hadoop project (framework mapreduce). We
are trying to modify the faults tolerance by replicating tasks (map
tasks and reduce tasks).=20
We are going to start tests.So we need to have an access to repository's
Hadoop.
Is it possible ? What is the procedure ?

Thanks.=20

Regards.=20

Jonathan LEJEUNE and Madeleine PIFFARETTI

From general-return-1286-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 02 22:28:44 2010
Return-Path: <general-return-1286-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77767 invoked from network); 2 Apr 2010 22:28:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Apr 2010 22:28:44 -0000
Received: (qmail 80123 invoked by uid 500); 2 Apr 2010 22:28:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80040 invoked by uid 500); 2 Apr 2010 22:28:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80032 invoked by uid 99); 2 Apr 2010 22:28:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 22:28:43 +0000
X-ASF-Spam-Status: No, hits=-0.6 required=10.0
	tests=AWL,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ryanobjc@gmail.com designates 209.85.223.203 as permitted sender)
Received: from [209.85.223.203] (HELO mail-iw0-f203.google.com) (209.85.223.203)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 22:28:37 +0000
Received: by iwn41 with SMTP id 41so1656621iwn.20
        for <general@hadoop.apache.org>; Fri, 02 Apr 2010 15:28:17 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=5xcycrxHfXSTamd2XNODKTHsE+w3/cZCpJPYDh5Q68I=;
        b=nH4WIZH5FhLbKYpmxh05bqBoCKAlS6cWnu3cezUqb7VV1VluPBqCHpcjvDmtHuk176
         TcShx4p+P6o4M6pkNrQY3ocv9nsKhwxkxD5UqfHYR7L0LqWrtAReGZrLdYAAVt8SUfR5
         5fZSe10pX2jLM6HyBRP6+kWjgoNrDV691MyOQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=gE6Rk3DjEstnZQ0wp4hKNzU3jyluxNoV71sfKvMiu64iEtaZ/kiyPSN/ZSdYOea5WF
         sB9OSF/36Yt7ROAdr7KT0sMEIG23B+pJjuWvAHNNOUrl6bDtQmFwaygm7CDmPIi5ELVj
         rFCcFZvbIOMURoIcF7JOTYQwK8P/gf/CqkFGM=
MIME-Version: 1.0
Received: by 10.231.48.199 with HTTP; Fri, 2 Apr 2010 15:28:17 -0700 (PDT)
In-Reply-To: <1270204337.19663.16.camel@ari-31-201-04.infop6.jussieu.fr>
References: <1270204337.19663.16.camel@ari-31-201-04.infop6.jussieu.fr>
Date: Fri, 2 Apr 2010 15:28:17 -0700
Received: by 10.231.150.195 with SMTP id z3mr1111656ibv.3.1270247297368; Fri, 
	02 Apr 2010 15:28:17 -0700 (PDT)
Message-ID: <p2h78568af11004021528h34fc9e18qe55d6115804f27f7@mail.gmail.com>
Subject: Re: Project for studies
From: Ryan Rawson <ryanobjc@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

You probably want the git mirrors which are listed here:

http://git.apache.org/

Generally speaking, you must create and submit patches in the Hadoop
JIRA system to get code into Hadoop.

Good luck,
-ryan

On Fri, Apr 2, 2010 at 3:32 AM, LEJEUNE JONATHAN
<jonathan.lejeune@etu.upmc.fr> wrote:
> Dear project members,
>
> We are two french students from Universit=E9 Pierre et Marie Curie (Paris
> 6), and we are working on the Hadoop project (framework mapreduce). We
> are trying to modify the faults tolerance by replicating tasks (map
> tasks and reduce tasks).
> We are going to start tests.So we need to have an access to repository's
> Hadoop.
> Is it possible ? What is the procedure ?
>
> Thanks.
>
> Regards.
>
> Jonathan LEJEUNE and Madeleine PIFFARETTI
>

From general-return-1287-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 05 06:57:04 2010
Return-Path: <general-return-1287-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35526 invoked from network); 5 Apr 2010 06:57:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Apr 2010 06:57:03 -0000
Received: (qmail 46838 invoked by uid 500); 5 Apr 2010 06:57:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46591 invoked by uid 500); 5 Apr 2010 06:57:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46583 invoked by uid 99); 5 Apr 2010 06:57:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 06:57:00 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=HTML_MESSAGE,MIME_QP_LONG_LINE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [203.91.198.74] (HELO wipro-blr-out02.wipro.com) (203.91.198.74)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 06:56:55 +0000
X-AuditID: cb5bdd58-b7bf7ae000006db3-7c-4bb989a08d03
Received: from blr-ec-aa03.wipro.com ( [10.201.18.42])
	by wipro-blr-out02.wipro.com (Symantec Mail Security) with SMTP id 01.F3.28083.0A989BB4; Mon,  5 Apr 2010 12:26:32 +0530 (IST)
Received: from blr-ec-bh02.wipro.com ([10.201.50.92]) by blr-ec-aa03.wipro.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Mon, 5 Apr 2010 12:26:32 +0530
Received: from BLR-EC-MBX08.wipro.com ([10.201.51.212]) by blr-ec-bh02.wipro.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Mon, 5 Apr 2010 12:26:33 +0530
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01CAD48D.1FD3781B"
Subject: Need information on interfacing Project Voldemort with BDB Sore
Date: Mon, 5 Apr 2010 12:26:24 +0530
Message-ID: <C037A228E2155B408E90822BF6EA059D09FED5FC@BLR-EC-MBX08.wipro.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Need information on interfacing Project Voldemort with BDB Sore
Thread-Index: AcrUjRrnbcwk0ZKVRWeebEGrHHQa7Q==
From: <psuresh.prasad@wipro.com>
To: <general@hadoop.apache.org>
X-OriginalArrivalTime: 05 Apr 2010 06:56:33.0541 (UTC) FILETIME=[20923350:01CAD48D]
X-Brightmail-Tracker: AAAAAQAAAZE=

------_=_NextPart_001_01CAD48D.1FD3781B
Content-Type: text/plain;
	charset="us-ascii"
content-transfer-encoding: quoted-printable

Hi ,

 

I was successful in creating offline data for Project Voldemort   with
Read-only store  and swapping the stores .

But I don't find any examples where the Hadoop is interfaced with
Project Voldemort with BDB  stores.

Please can someone share the examples.

 

Regards,

Suresh Prasad

 


Please do not print this email unless it is absolutely necessary. =0A=
=0A=
The information contained in this electronic message and any attachments to=
 this message are intended for the exclusive use of the addressee(s) and may=
 contain proprietary, confidential or privileged information. If you are not=
 the intended recipient, you should not disseminate, distribute or copy this=
 e-mail. Please notify the sender immediately and destroy all copies of this=
 message and any attachments. =0A=
=0A=
WARNING: Computer viruses can be transmitted via email. The recipient should=
 check this email and any attachments for the presence of viruses. The compa=
ny accepts no liability for any damage caused by any virus transmitted by th=
is email. =0A=
=0A=
www.wipro.com

------_=_NextPart_001_01CAD48D.1FD3781B--

From general-return-1288-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 05 20:36:09 2010
Return-Path: <general-return-1288-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 3857 invoked from network); 5 Apr 2010 20:36:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Apr 2010 20:36:09 -0000
Received: (qmail 18886 invoked by uid 500); 5 Apr 2010 20:36:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18816 invoked by uid 500); 5 Apr 2010 20:36:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18808 invoked by uid 99); 5 Apr 2010 20:36:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 20:36:07 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mona@apture.com designates 67.221.235.209 as permitted sender)
Received: from [67.221.235.209] (HELO web06.apture.com) (67.221.235.209)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 20:35:59 +0000
Received: (qmail 1984 invoked by uid 89); 5 Apr 2010 20:35:38 -0000
Received: by simscan 1.1.0 ppid: 1953, pid: 1955, t: 15.4003s
         scanners: regex: 1.1.0 spam: 3.2.5
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
	morpheus.contegix.com
X-Spam-Level: *
Received: from unknown (HELO ?192.168.42.165?) (mona@apture.com@71.133.225.222)
  by 67-221-235-209.contegix.com with ESMTPA; 5 Apr 2010 20:35:22 -0000
From: Mona Gandhi <mona@apture.com>
Content-Type: multipart/alternative; boundary=Apple-Mail-2-134740413
Subject: Additional jar's to support input formats
Date: Mon, 5 Apr 2010 13:35:16 -0700
Message-Id: <C4792DFC-8642-4483-A5F5-1185C7ABF948@apture.com>
To: general@hadoop.apache.org
Mime-Version: 1.0 (Apple Message framework v1077)
X-Mailer: Apple Mail (2.1077)
X-Virus-Checked: Checked by ClamAV on apache.org
X-Old-Spam-Status: No, score=1.9 required=10.0 tests=ALL_TRUSTED,FH_DATE_PAST_20XX,
	HTML_MESSAGE autolearn=no version=3.2.5

--Apple-Mail-2-134740413
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

Where should i add additional jar's to support different input formats =
like AvroInputFormat?

Here's a link to a successful build - =
http://hudson.brokenco.de/job/Avro_Trunk/lastBuild/

Thanks in advance,
Mona


--Apple-Mail-2-134740413--

From general-return-1289-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 02:37:46 2010
Return-Path: <general-return-1289-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78539 invoked from network); 6 Apr 2010 02:37:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 02:37:46 -0000
Received: (qmail 11486 invoked by uid 500); 6 Apr 2010 02:37:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11359 invoked by uid 500); 6 Apr 2010 02:37:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11349 invoked by uid 99); 6 Apr 2010 02:37:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 02:37:43 +0000
X-ASF-Spam-Status: No, hits=-0.1 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [65.115.85.73] (HELO smtp-outbound-2.vmware.com) (65.115.85.73)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 02:37:36 +0000
Received: from mailhost3.vmware.com (mailhost3.vmware.com [10.16.27.45])
	by smtp-outbound-2.vmware.com (Postfix) with ESMTP id 792E05303C
	for <general@hadoop.apache.org>; Mon,  5 Apr 2010 19:37:16 -0700 (PDT)
Received: from pa-exht02.vmware.com (pa-exht02.vmware.com [10.113.81.168])
	by mailhost3.vmware.com (Postfix) with ESMTP id 71DC7CD91E
	for <general@hadoop.apache.org>; Mon,  5 Apr 2010 19:37:16 -0700 (PDT)
Received: from EXCH-MBX-22.vmware.com ([10.113.81.177]) by
 pa-exht02.vmware.com ([10.113.81.168]) with mapi; Mon, 5 Apr 2010 19:37:16
 -0700
From: Zhanlei Ma <zma@vmware.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 5 Apr 2010 19:37:11 -0700
Subject: Any good examples?
Thread-Topic: Any good examples?
Thread-Index: AcrVMg95aP8lPVSgQ0aoHGnEnhi91A==
Message-ID: <F21534570FE52940A8A9366D82C11E8B044E99C6A2@EXCH-MBX-22.vmware.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-cr-hashedpuzzle: A264 BLK+ BVNi BXd4 Br++ B+o6 CcKo CflN DdyW DqWK E4Lp
 FFe+ FeaB GvZp HYps
 Jyau;1;ZwBlAG4AZQByAGEAbABAAGgAYQBkAG8AbwBwAC4AYQBwAGEAYwBoAGUALgBvAHIAZwA=;Sosha1_v1;7;{2EBE7596-3AB5-414E-B1F1-E67C43C75F31};egBtAGEAQAB2AG0AdwBhAHIAZQAuAGMAbwBtAA==;Tue,
 06 Apr 2010 02:37:11 GMT;QQBuAHkAIABnAG8AbwBkACAAZQB4AGEAbQBwAGwAZQBzAD8A
x-cr-puzzleid: {2EBE7596-3AB5-414E-B1F1-E67C43C75F31}
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_F21534570FE52940A8A9366D82C11E8B044E99C6A2EXCHMBX22vmwa_"
MIME-Version: 1.0

--_000_F21534570FE52940A8A9366D82C11E8B044E99C6A2EXCHMBX22vmwa_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

HI all:
    In the test of hadoop, it is a problem to find the good example and the=
 test big data.Terasort is good, but the data generated by teragen is not s=
table. Does Anyone has other good examples and the data source to recommend=
 me. Thanks for you.

--_000_F21534570FE52940A8A9366D82C11E8B044E99C6A2EXCHMBX22vmwa_--

From general-return-1290-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 02:40:53 2010
Return-Path: <general-return-1290-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78892 invoked from network); 6 Apr 2010 02:40:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 02:40:53 -0000
Received: (qmail 15931 invoked by uid 500); 6 Apr 2010 02:40:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15880 invoked by uid 500); 6 Apr 2010 02:40:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15872 invoked by uid 99); 6 Apr 2010 02:40:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 02:40:52 +0000
X-ASF-Spam-Status: No, hits=1.8 required=10.0
	tests=AWL,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.219.215] (HELO mail-ew0-f215.google.com) (209.85.219.215)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 02:40:48 +0000
Received: by ewy7 with SMTP id 7so67859ewy.31
        for <general@hadoop.apache.org>; Mon, 05 Apr 2010 19:40:26 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.154.205 with HTTP; Mon, 5 Apr 2010 19:40:26 -0700 (PDT)
In-Reply-To: <F21534570FE52940A8A9366D82C11E8B044E99C6A2@EXCH-MBX-22.vmware.com>
References: <F21534570FE52940A8A9366D82C11E8B044E99C6A2@EXCH-MBX-22.vmware.com>
Date: Mon, 5 Apr 2010 19:40:26 -0700
Received: by 10.213.42.73 with SMTP id r9mr153062ebe.40.1270521626541; Mon, 05 
	Apr 2010 19:40:26 -0700 (PDT)
Message-ID: <r2u23d54121004051940z47f60728w7a3b1e46b522458a@mail.gmail.com>
Subject: Re: Any good examples?
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Check out http://infochimps.org/datasets for a number of free,
moderately sized, datasets. They won't be terabytes in size, but
they're a good place to start if you're looking for real data to play
with.

On Mon, Apr 5, 2010 at 7:37 PM, Zhanlei Ma <zma@vmware.com> wrote:
> HI all:
> =A0 =A0In the test of hadoop, it is a problem to find the good example an=
d the test big data.Terasort is good, but the data generated by teragen is =
not stable. Does Anyone has other good examples and the data source to reco=
mmend me. Thanks for you.
>



--=20
Eric Sammer
phone: +1-917-287-2675
twitter: esammer
data: www.cloudera.com

From general-return-1291-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 02:50:32 2010
Return-Path: <general-return-1291-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 80041 invoked from network); 6 Apr 2010 02:50:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 02:50:32 -0000
Received: (qmail 26817 invoked by uid 500); 6 Apr 2010 02:50:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26678 invoked by uid 500); 6 Apr 2010 02:50:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26669 invoked by uid 99); 6 Apr 2010 02:50:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 02:50:30 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [65.115.85.73] (HELO smtp-outbound-2.vmware.com) (65.115.85.73)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 02:50:25 +0000
Received: from mailhost2.vmware.com (mailhost2.vmware.com [10.16.67.167])
	by smtp-outbound-2.vmware.com (Postfix) with ESMTP id C6FF5B063
	for <general@hadoop.apache.org>; Mon,  5 Apr 2010 19:50:03 -0700 (PDT)
Received: from pa-exht03.vmware.com (pa-exht03.vmware.com [10.16.45.223])
	by mailhost2.vmware.com (Postfix) with ESMTP id B926C8E858
	for <general@hadoop.apache.org>; Mon,  5 Apr 2010 19:50:03 -0700 (PDT)
Received: from EXCH-MBX-22.vmware.com ([10.113.81.177]) by
 pa-exht03.vmware.com ([10.16.45.223]) with mapi; Mon, 5 Apr 2010 19:50:03
 -0700
From: Zhanlei Ma <zma@vmware.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 5 Apr 2010 19:50:01 -0700
Subject: RE: Any good examples?
Thread-Topic: Any good examples?
Thread-Index: AcrVMpUexkwLRP8jSP6jcS+tvVAkYgAATYnQ
Message-ID: <F21534570FE52940A8A9366D82C11E8B044E99C6BD@EXCH-MBX-22.vmware.com>
References: <F21534570FE52940A8A9366D82C11E8B044E99C6A2@EXCH-MBX-22.vmware.com>
 <r2u23d54121004051940z47f60728w7a3b1e46b522458a@mail.gmail.com>
In-Reply-To: <r2u23d54121004051940z47f60728w7a3b1e46b522458a@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

VGhhbmsgRXJpYyBTYW1tZXIhDQoNCi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQpGcm9tOiBF
cmljIFNhbW1lciBbbWFpbHRvOmVzYW1tZXJAY2xvdWRlcmEuY29tXSANClNlbnQ6IDIwMTDlubQ0
5pyINuaXpSAxMDo0MA0KVG86IGdlbmVyYWxAaGFkb29wLmFwYWNoZS5vcmcNClN1YmplY3Q6IFJl
OiBBbnkgZ29vZCBleGFtcGxlcz8NCg0KQ2hlY2sgb3V0IGh0dHA6Ly9pbmZvY2hpbXBzLm9yZy9k
YXRhc2V0cyBmb3IgYSBudW1iZXIgb2YgZnJlZSwNCm1vZGVyYXRlbHkgc2l6ZWQsIGRhdGFzZXRz
LiBUaGV5IHdvbid0IGJlIHRlcmFieXRlcyBpbiBzaXplLCBidXQNCnRoZXkncmUgYSBnb29kIHBs
YWNlIHRvIHN0YXJ0IGlmIHlvdSdyZSBsb29raW5nIGZvciByZWFsIGRhdGEgdG8gcGxheQ0Kd2l0
aC4NCg0KT24gTW9uLCBBcHIgNSwgMjAxMCBhdCA3OjM3IFBNLCBaaGFubGVpIE1hIDx6bWFAdm13
YXJlLmNvbT4gd3JvdGU6DQo+IEhJIGFsbDoNCj4gwqAgwqBJbiB0aGUgdGVzdCBvZiBoYWRvb3As
IGl0IGlzIGEgcHJvYmxlbSB0byBmaW5kIHRoZSBnb29kIGV4YW1wbGUgYW5kIHRoZSB0ZXN0IGJp
ZyBkYXRhLlRlcmFzb3J0IGlzIGdvb2QsIGJ1dCB0aGUgZGF0YSBnZW5lcmF0ZWQgYnkgdGVyYWdl
biBpcyBub3Qgc3RhYmxlLiBEb2VzIEFueW9uZSBoYXMgb3RoZXIgZ29vZCBleGFtcGxlcyBhbmQg
dGhlIGRhdGEgc291cmNlIHRvIHJlY29tbWVuZCBtZS4gVGhhbmtzIGZvciB5b3UuDQo+DQoNCg0K
DQotLSANCkVyaWMgU2FtbWVyDQpwaG9uZTogKzEtOTE3LTI4Ny0yNjc1DQp0d2l0dGVyOiBlc2Ft
bWVyDQpkYXRhOiB3d3cuY2xvdWRlcmEuY29tDQo=

From general-return-1292-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 09:22:32 2010
Return-Path: <general-return-1292-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46072 invoked from network); 6 Apr 2010 09:22:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 09:22:32 -0000
Received: (qmail 2919 invoked by uid 500); 6 Apr 2010 09:22:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2801 invoked by uid 500); 6 Apr 2010 09:22:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2790 invoked by uid 99); 6 Apr 2010 09:22:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 09:22:28 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=AWL,HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 09:22:22 +0000
Received: from SP1-EX07CAS02.ds.corp.yahoo.com (sp1-ex07cas02.ds.corp.yahoo.com [216.252.116.138])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o369LVWM017011
	for <general@hadoop.apache.org>; Tue, 6 Apr 2010 02:21:32 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:in-reply-to:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=qzQthXmLJ+P2cF0k0l7nUqqMU9l6887+qvWMZdRgRpTuZ2Xr7/9RwjF4fGyjFQ5i
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 SP1-EX07CAS02.ds.corp.yahoo.com ([216.252.116.167]) with mapi; Tue, 6 Apr
 2010 02:21:31 -0700
From: Rekha Joshi <rekhajos@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 6 Apr 2010 02:21:30 -0700
Subject: Re: Additional jar's to support input formats
Thread-Topic: Additional jar's to support input formats
Thread-Index: AcrU/7PhUSiSrVX/RMiJJcghLL4/owAatab1
Message-ID: <C7E0FAF2.DB61%rekhajos@yahoo-inc.com>
In-Reply-To: <C4792DFC-8642-4483-A5F5-1185C7ABF948@apture.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C7E0FAF2DB61rekhajosyahooinccom_"
MIME-Version: 1.0

--_000_C7E0FAF2DB61rekhajosyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

I don't think its a matter of adding to -libjars as to use avro serializati=
on hadoop needs some updates.Some discussion with Doug, Aaron @ MAPREDUCE-8=
15

On 4/6/10 2:05 AM, "Mona Gandhi" <mona@apture.com> wrote:

Where should i add additional jar's to support different input formats like=
 AvroInputFormat?

Here's a link to a successful build - http://hudson.brokenco.de/job/Avro_Tr=
unk/lastBuild/

Thanks in advance,
Mona



--_000_C7E0FAF2DB61rekhajosyahooinccom_--

From general-return-1293-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 14:44:21 2010
Return-Path: <general-return-1293-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31380 invoked from network); 6 Apr 2010 14:44:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 14:44:20 -0000
Received: (qmail 55995 invoked by uid 500); 6 Apr 2010 14:44:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55938 invoked by uid 500); 6 Apr 2010 14:44:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55930 invoked by uid 99); 6 Apr 2010 14:44:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 14:44:19 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.219.215] (HELO mail-ew0-f215.google.com) (209.85.219.215)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 14:44:12 +0000
Received: by ewy7 with SMTP id 7so189487ewy.31
        for <general@hadoop.apache.org>; Tue, 06 Apr 2010 07:43:52 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.51.201 with HTTP; Tue, 6 Apr 2010 07:43:51 -0700 (PDT)
Date: Tue, 6 Apr 2010 10:43:51 -0400
Received: by 10.213.52.208 with SMTP id j16mr4169868ebg.39.1270565031489; Tue, 
	06 Apr 2010 07:43:51 -0700 (PDT)
Message-ID: <m2r643a61251004060743ge24767e9n7038fe038c4f04de@mail.gmail.com>
Subject: hadoop on demand setup: Failed to retrieve 'hdfs' service address
From: Kevin Van Workum <vanw@sabalcore.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hello,

I'm trying to setup hadoop on demand (HOD) on my cluster. I'm
currently unable to "allocate cluster". I'm starting hod with the
following command:

/usr/local/hadoop-0.20.2/hod/bin/hod -c
/usr/local/hadoop-0.20.2/hod/conf/hodrc -t
/b/01/vanw/hod/hadoop-0.20.2.tar.gz -o "allocate ~/hod 3"
--ringmaster.log-dir=/tmp -b 4

The job starts on the nodes and I see the ringmaster running on the
MotherSuperior. The ringmaster-main.log file is created, but is empty.
I don't see any associated processes running on the other 2 nodes in
the job.

The critical errors are as follows:

[2010-04-06 10:34:13,630] CRITICAL/50 hadoop:298 - Failed to retrieve
'hdfs' service address.
[2010-04-06 10:34:13,631] DEBUG/10 hadoop:631 - Cleaning up cluster id
238366.jman, as cluster could not be allocated.
[2010-04-06 10:34:13,632] DEBUG/10 hadoop:635 - Calling rm.stop()
[2010-04-06 10:34:13,639] DEBUG/10 hadoop:637 - Returning from rm.stop()
[2010-04-06 10:34:13,639] CRITICAL/50 hod:401 - Cannot allocate
cluster /b/01/vanw/hod
[2010-04-06 10:34:14,149] DEBUG/10 hod:597 - return code: 7

The contents of the hodrc file is:

[hod]
stream                          = True
java-home                       = /usr/local/jdk1.6.0_02
cluster                         = orange
cluster-factor                  = 1.8
xrs-port-range                  = 32768-65536
debug                           = 4
allocate-wait-time              = 3600
temp-dir                        = /tmp/hod

[ringmaster]
register                        = True
stream                          = False
temp-dir                        = /tmp/hod
http-port-range                 = 8000-9000
work-dirs                       = /tmp/hod/1,/tmp/hod/2
xrs-port-range                  = 32768-65536
debug                           = 4

[hodring]
stream                          = False
temp-dir                        = /tmp/hod
register                        = True
java-home                       = /usr/local/jdk1.6.0_02
http-port-range                 = 8000-9000
xrs-port-range                  = 32768-65536
debug                           = 4

[resource_manager]
queue                           = dque
batch-home                      = /usr/local/torque-2.3.7
id                              = torque
env-vars                       =
HOD_PYTHON_HOME=/usr/local/python-2.5.5/bin/python

[gridservice-mapred]
external                        = False
pkgs                            = /usr/local/hadoop-0.20.2
tracker_port                    = 8030
info_port                       = 50080

[gridservice-hdfs]
external                        = False
pkgs                            = /usr/local/hadoop-0.20.2
fs_port                         = 8020
info_port                       = 50070


Some other useful information:
Linux 2.6.18-128.7.1.el5
Python 2.5.5
Twisted 10.0.0
zope 3.3.0
java version "1.6.0_02"

-- 
Kevin Van Workum, PhD
Sabalcore Computing Inc.
Run your code on 500 processors.
Sign up for a free trial account.
www.sabalcore.com
877-492-8027 ext. 11

From general-return-1294-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 14:51:20 2010
Return-Path: <general-return-1294-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33307 invoked from network); 6 Apr 2010 14:51:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 14:51:20 -0000
Received: (qmail 71805 invoked by uid 500); 6 Apr 2010 14:51:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71697 invoked by uid 500); 6 Apr 2010 14:51:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71689 invoked by uid 99); 6 Apr 2010 14:51:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 14:51:19 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.219.215] (HELO mail-ew0-f215.google.com) (209.85.219.215)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 14:51:14 +0000
Received: by ewy7 with SMTP id 7so194524ewy.31
        for <general@hadoop.apache.org>; Tue, 06 Apr 2010 07:50:52 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.51.201 with HTTP; Tue, 6 Apr 2010 07:50:52 -0700 (PDT)
In-Reply-To: <m2r643a61251004060743ge24767e9n7038fe038c4f04de@mail.gmail.com>
References: <m2r643a61251004060743ge24767e9n7038fe038c4f04de@mail.gmail.com>
Date: Tue, 6 Apr 2010 10:50:52 -0400
Received: by 10.213.42.78 with SMTP id r14mr4206418ebe.11.1270565452133; Tue, 
	06 Apr 2010 07:50:52 -0700 (PDT)
Message-ID: <l2t643a61251004060750r56b8a60et1cd0e30a51f096c4@mail.gmail.com>
Subject: Re: hadoop on demand setup: Failed to retrieve 'hdfs' service address
From: Kevin Van Workum <vanw@sabalcore.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Tue, Apr 6, 2010 at 10:43 AM, Kevin Van Workum <vanw@sabalcore.com> wrot=
e:
> Hello,
>
> I'm trying to setup hadoop on demand (HOD) on my cluster. I'm
> currently unable to "allocate cluster". I'm starting hod with the
> following command:
>
> /usr/local/hadoop-0.20.2/hod/bin/hod -c
> /usr/local/hadoop-0.20.2/hod/conf/hodrc -t
> /b/01/vanw/hod/hadoop-0.20.2.tar.gz -o "allocate ~/hod 3"
> --ringmaster.log-dir=3D/tmp -b 4
>
> The job starts on the nodes and I see the ringmaster running on the
> MotherSuperior. The ringmaster-main.log file is created, but is empty.
> I don't see any associated processes running on the other 2 nodes in
> the job.

Update: After making sure debug =3D 4 was set on the nodes, I'm seeing
these errors in ringmaster-main.log:
[2010-04-06 10:47:43,183] DEBUG/10 ringMaster:479 - getServiceAddr name: hd=
fs
[2010-04-06 10:47:43,184] DEBUG/10 ringMaster:487 - getServiceAddr
service: <hodlib.GridServices.hdfs.Hdfs instance at 0x122d24d0>
[2010-04-06 10:47:43,186] DEBUG/10 ringMaster:504 - getServiceAddr
addr hdfs: not found

>
> The critical errors are as follows:
>
> [2010-04-06 10:34:13,630] CRITICAL/50 hadoop:298 - Failed to retrieve
> 'hdfs' service address.
> [2010-04-06 10:34:13,631] DEBUG/10 hadoop:631 - Cleaning up cluster id
> 238366.jman, as cluster could not be allocated.
> [2010-04-06 10:34:13,632] DEBUG/10 hadoop:635 - Calling rm.stop()
> [2010-04-06 10:34:13,639] DEBUG/10 hadoop:637 - Returning from rm.stop()
> [2010-04-06 10:34:13,639] CRITICAL/50 hod:401 - Cannot allocate
> cluster /b/01/vanw/hod
> [2010-04-06 10:34:14,149] DEBUG/10 hod:597 - return code: 7
>
> The contents of the hodrc file is:
>
> [hod]
> stream =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D True
> java-home =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D /usr/local/jdk1=
.6.0_02
> cluster =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D orange
> cluster-factor =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D 1.8
> xrs-port-range =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D 32768-65536
> debug =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D 4
> allocate-wait-time =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D 3600
> temp-dir =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D /tmp/hod
>
> [ringmaster]
> register =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D True
> stream =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D False
> temp-dir =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D /tmp/hod
> http-port-range =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D 8000-9000
> work-dirs =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D /tmp/hod/1,/tmp=
/hod/2
> xrs-port-range =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D 32768-65536
> debug =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D 4
>
> [hodring]
> stream =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D False
> temp-dir =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D /tmp/hod
> register =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D True
> java-home =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D /usr/local/jdk1=
.6.0_02
> http-port-range =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D 8000-9000
> xrs-port-range =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D 32768-65536
> debug =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D 4
>
> [resource_manager]
> queue =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D dque
> batch-home =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D /usr/local/torq=
ue-2.3.7
> id =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D torque
> env-vars =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D
> HOD_PYTHON_HOME=3D/usr/local/python-2.5.5/bin/python
>
> [gridservice-mapred]
> external =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D False
> pkgs =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D /usr/loca=
l/hadoop-0.20.2
> tracker_port =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D 8030
> info_port =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D 50080
>
> [gridservice-hdfs]
> external =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D False
> pkgs =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D /usr/loca=
l/hadoop-0.20.2
> fs_port =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D 8020
> info_port =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D 50070
>
>
> Some other useful information:
> Linux 2.6.18-128.7.1.el5
> Python 2.5.5
> Twisted 10.0.0
> zope 3.3.0
> java version "1.6.0_02"
>
> --
> Kevin Van Workum, PhD
> Sabalcore Computing Inc.
> Run your code on 500 processors.
> Sign up for a free trial account.
> www.sabalcore.com
> 877-492-8027 ext. 11
>



--=20
Kevin Van Workum, PhD
Sabalcore Computing Inc.
Run your code on 500 processors.
Sign up for a free trial account.
www.sabalcore.com
877-492-8027 ext. 11

From general-return-1295-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 18:32:43 2010
Return-Path: <general-return-1295-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95449 invoked from network); 6 Apr 2010 18:32:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 18:32:43 -0000
Received: (qmail 5210 invoked by uid 500); 6 Apr 2010 18:32:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5176 invoked by uid 500); 6 Apr 2010 18:32:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5168 invoked by uid 99); 6 Apr 2010 18:32:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 18:32:41 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 06 Apr 2010 18:32:39 +0000
Received: (qmail 95396 invoked by uid 99); 6 Apr 2010 18:32:17 -0000
Received: from localhost.apache.org (HELO [192.168.42.156]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 18:32:17 +0000
Message-ID: <4BBB7E30.8040602@apache.org>
Date: Tue, 06 Apr 2010 11:32:16 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100317)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Q about use of avro logs with hadoop streaming
References: <859D76D2-50AE-4E66-974A-3B47621323F6@apture.com>
In-Reply-To: <859D76D2-50AE-4E66-974A-3B47621323F6@apture.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Mona Gandhi wrote:
> i currently use avro version 1.3.0 to log data. I am having difficulty processing these avro logs via a map reduce job written in Python using hadoop streaming(v 0.21.0). 

Mona,

There is currently no support for Avro data in streaming.  One could use 
a shell command to convert Avro data to lines of text (e.g., Avro's 
'tojson' tool) but that would be rather inefficient.

A good approach would be something akin to Hadoop Pipes: we implement a 
Java mapper and reducer that use an Avro protocol to communicate with a 
subprocess over standard input and output, transmitting input and output 
records as raw binary.  The subprocess would deserialize inputs, call 
the user-provided mapper or reducer function, then serialize outputs 
back.  This would require no changes to Hadoop and could be included in 
Avro.  We'd provide implementations of this protocol for the various 
languages, Python, Ruby, C, C++, etc., enabling high-performance 
mapreduce programs over Avro data for all of these.

The existing Hadoop Pipes implementation would be a good starting point 
for this work, as it already uses the same technique, although with a 
Hadoop Writable-based protocol and with only a C++ implementation.

I've filed an issue in Jira to track this:

https://issues.apache.org/jira/browse/AVRO-512

I might have a chance to work on this later this month.

Doug


From general-return-1296-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 20:15:24 2010
Return-Path: <general-return-1296-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17427 invoked from network); 6 Apr 2010 20:15:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 20:15:24 -0000
Received: (qmail 25037 invoked by uid 500); 6 Apr 2010 20:15:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25000 invoked by uid 500); 6 Apr 2010 20:15:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24992 invoked by uid 99); 6 Apr 2010 20:15:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 20:15:23 +0000
X-ASF-Spam-Status: No, hits=-62.9 required=10.0
	tests=AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 06 Apr 2010 20:15:22 +0000
Received: (qmail 17404 invoked by uid 99); 6 Apr 2010 20:15:02 -0000
Received: from localhost.apache.org (HELO [192.168.42.156]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 20:15:02 +0000
Message-ID: <4BBB9644.7020201@apache.org>
Date: Tue, 06 Apr 2010 13:15:00 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100317)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSS] Avro as TLP?
References: <4BB28E7C.90300@apache.org>
In-Reply-To: <4BB28E7C.90300@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Doug Cutting wrote:
> Does the Hadoop community have any questions or concerns about this 
> proposal?

Since no issues were raised, I'll now call the vote.

Doug

From general-return-1297-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 20:16:13 2010
Return-Path: <general-return-1297-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17652 invoked from network); 6 Apr 2010 20:16:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 20:16:13 -0000
Received: (qmail 27611 invoked by uid 500); 6 Apr 2010 20:16:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27573 invoked by uid 500); 6 Apr 2010 20:16:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27565 invoked by uid 99); 6 Apr 2010 20:16:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 20:16:12 +0000
X-ASF-Spam-Status: No, hits=-62.6 required=10.0
	tests=AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 06 Apr 2010 20:16:10 +0000
Received: (qmail 17531 invoked by uid 99); 6 Apr 2010 20:15:50 -0000
Received: from localhost.apache.org (HELO [192.168.42.156]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 20:15:50 +0000
Message-ID: <4BBB9675.1090909@apache.org>
Date: Tue, 06 Apr 2010 13:15:49 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100317)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: [VOTE] Avro as TLP?
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Please vote as to whether you think Avro should become a top-level
Apache project, as mentioned previously on this list.

I've included below a draft board resolution.  It lists all current
committers as the initial members of the project management committee
(PMC) and Matt Massie as the initial chair.

Does the Hadoop PMC support sending this proposal on to the board?

Thanks,

Doug

------------

     X. Establish the Apache Avro Project

        WHEREAS, the Board of Directors deems it to be in the best
        interests of the Foundation and consistent with the
        Foundation's purpose to establish a Project Management
        Committee charged with the creation and maintenance of
        open-source software related to data serialization
        for distribution at no charge to the public.

        NOW, THEREFORE, BE IT RESOLVED, that a Project Management
        Committee (PMC), to be known as the "Apache Avro Project",
        be and hereby is established pursuant to Bylaws of the
        Foundation; and be it further

        RESOLVED, that the Apache Avro Project be and hereby is
        responsible for the creation and maintenance of software
        related to data serialization; and be it further

        RESOLVED, that the office of "Vice President, Apache Avro" be
        and hereby is created, the person holding such office to
        serve at the direction of the Board of Directors as the chair
        of the Apache Avro Project, and to have primary responsibility
        for management of the projects within the scope of
        responsibility of the Apache Avro Project; and be it further

        RESOLVED, that the persons listed immediately below be and
        hereby are appointed to serve as the initial members of the
        Apache Avro Project:

          * Doug Cutting         <cutting@apache.org>
          * Jeff Hammerbacher    <hammer@apache.org>
          * Jeff Hodges          <jmhodges@apache.org>
          * Matt Massie          <massie@apache.org>
          * Philip Zeyliger      <philz@apache.org>
          * Scott Banachowski    <sbanacho@apache.org>
          * Scott Carey          <scottcarey@apache.org>
          * Sharad Agarwal       <sharadag@apache.org>
          * Thiruvalluvan M. G.  <thiru@apache.org>

        NOW, THEREFORE, BE IT FURTHER RESOLVED, that Matt Massie
        be appointed to the office of Vice President, Apache Avro, to
        serve in accordance with and subject to the direction of the
        Board of Directors and the Bylaws of the Foundation until
        death, resignation, retirement, removal or disqualification,
        or until a successor is appointed; and be it further

        RESOLVED, that the initial Apache Avro PMC be and hereby is
        tasked with the creation of a set of bylaws intended to
        encourage open development and increased participation in the
        Apache Avro Project; and be it further

        RESOLVED, that the Apache Avro Project be and hereby
        is tasked with the migration and rationalization of the Apache
        Hadoop Avro sub-project; and be it further

        RESOLVED, that all responsibilities pertaining to the Apache
        Hadoop Avro sub-project encumbered upon the
        Apache Hadoop Project are hereafter discharged.


From general-return-1298-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 20:25:27 2010
Return-Path: <general-return-1298-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19552 invoked from network); 6 Apr 2010 20:25:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 20:25:27 -0000
Received: (qmail 39357 invoked by uid 500); 6 Apr 2010 20:25:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39313 invoked by uid 500); 6 Apr 2010 20:25:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39305 invoked by uid 99); 6 Apr 2010 20:25:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 20:25:26 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of owen.omalley@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 20:25:18 +0000
Received: by pwi7 with SMTP id 7so312642pwi.35
        for <general@hadoop.apache.org>; Tue, 06 Apr 2010 13:24:57 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type;
        bh=NcJw4HQsakholpQ5UL6K4tYLwFwUAfGmzPOiO+et4y0=;
        b=n8mAaqtT7kKtr+cKsja0XTJAI6jAG4kwh03elBGotmo62adZE0obXpV06UqsYKMvz8
         GbbBfSd1X5D2IZ4BKzgzq3560/X1c/jGxJxWinFsMiknlo+XKOcYraR5J5nFqi4zYi9Z
         cou+opBtcPihiNYsIIyo3JiaZ5AzUr4vwViL8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=hMhrOkozjtxyz4PnkPZPjZnvfIP9ojlHlKxa1zSpnnMogG9Ryny+eKSouHSbl/2j9k
         GykipyJ8aFK1giDx+i7zi/zn4lyZj3D4NPqrrFu6TRwrIMZEWzpMM1bdSHe58UYU8CPn
         ryoRQNfRgSJU/sQ1hNphMyr8ADmKXy/YcSg5M=
MIME-Version: 1.0
Received: by 10.231.168.20 with HTTP; Tue, 6 Apr 2010 13:24:56 -0700 (PDT)
In-Reply-To: <4BBB9675.1090909@apache.org>
References: <4BBB9675.1090909@apache.org>
Date: Tue, 6 Apr 2010 13:24:56 -0700
Received: by 10.142.55.18 with SMTP id d18mr2812183wfa.170.1270585497158; Tue, 
	06 Apr 2010 13:24:57 -0700 (PDT)
Message-ID: <n2q5f7f740b1004061324uec231ca3m361722bba7a09fa6@mail.gmail.com>
Subject: Re: [VOTE] Avro as TLP?
From: "Owen O'Malley" <owen.omalley@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636b2bb873703470483973dbd
X-Virus-Checked: Checked by ClamAV on apache.org

--001636b2bb873703470483973dbd
Content-Type: text/plain; charset=UTF-8

On Tue, Apr 6, 2010 at 1:15 PM, Doug Cutting <cutting@apache.org> wrote:

> Does the Hadoop PMC support sending this proposal on to the board?
>

+1

--001636b2bb873703470483973dbd--

From general-return-1299-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 21:51:49 2010
Return-Path: <general-return-1299-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37718 invoked from network); 6 Apr 2010 21:51:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 21:51:48 -0000
Received: (qmail 37959 invoked by uid 500); 6 Apr 2010 21:51:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37913 invoked by uid 500); 6 Apr 2010 21:51:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37905 invoked by uid 99); 6 Apr 2010 21:51:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 21:51:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 06 Apr 2010 21:51:45 +0000
Received: (qmail 37696 invoked by uid 99); 6 Apr 2010 21:51:23 -0000
Received: from localhost.apache.org (HELO mail-yx0-f172.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 21:51:23 +0000
Received: by yxe2 with SMTP id 2so195990yxe.2
        for <general@hadoop.apache.org>; Tue, 06 Apr 2010 14:51:22 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.147.133 with HTTP; Tue, 6 Apr 2010 14:51:22 -0700 (PDT)
In-Reply-To: <4BBB9675.1090909@apache.org>
References: <4BBB9675.1090909@apache.org>
Date: Tue, 6 Apr 2010 14:51:22 -0700
Received: by 10.231.145.145 with SMTP id d17mr13503ibv.30.1270590682402; Tue, 
	06 Apr 2010 14:51:22 -0700 (PDT)
Message-ID: <o2h1267dd3b1004061451k90885a60k63feb12bf468910d@mail.gmail.com>
Subject: Re: [VOTE] Avro as TLP?
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

> Does the Hadoop PMC support sending this proposal on to the board?

+1 -C

From general-return-1300-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 22:02:27 2010
Return-Path: <general-return-1300-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39823 invoked from network); 6 Apr 2010 22:02:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 22:02:27 -0000
Received: (qmail 47147 invoked by uid 500); 6 Apr 2010 22:02:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47100 invoked by uid 500); 6 Apr 2010 22:02:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47092 invoked by uid 99); 6 Apr 2010 22:02:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 22:02:26 +0000
X-ASF-Spam-Status: No, hits=-0.3 required=10.0
	tests=AWL,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.218.217] (HELO mail-bw0-f217.google.com) (209.85.218.217)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 22:02:20 +0000
Received: by bwz9 with SMTP id 9so345109bwz.29
        for <general@hadoop.apache.org>; Tue, 06 Apr 2010 15:01:58 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.98.16 with HTTP; Tue, 6 Apr 2010 15:01:57 -0700 (PDT)
In-Reply-To: <4BBB9675.1090909@apache.org>
References: <4BBB9675.1090909@apache.org>
Date: Tue, 6 Apr 2010 15:01:57 -0700
Received: by 10.204.7.212 with SMTP id e20mr8521472bke.117.1270591317787; Tue, 
	06 Apr 2010 15:01:57 -0700 (PDT)
Message-ID: <h2uac79ea401004061501jc0f45c5bka76f3a0b893c60d2@mail.gmail.com>
Subject: Re: [VOTE] Avro as TLP?
From: Tom White <tom@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Tue, Apr 6, 2010 at 1:15 PM, Doug Cutting <cutting@apache.org> wrote:
> Please vote as to whether you think Avro should become a top-level
> Apache project, as mentioned previously on this list.
>
> I've included below a draft board resolution. =A0It lists all current
> committers as the initial members of the project management committee
> (PMC) and Matt Massie as the initial chair.
>
> Does the Hadoop PMC support sending this proposal on to the board?

+1

Tom

>
> Thanks,
>
> Doug
>
> ------------
>
> =A0 =A0X. Establish the Apache Avro Project
>
> =A0 =A0 =A0 WHEREAS, the Board of Directors deems it to be in the best
> =A0 =A0 =A0 interests of the Foundation and consistent with the
> =A0 =A0 =A0 Foundation's purpose to establish a Project Management
> =A0 =A0 =A0 Committee charged with the creation and maintenance of
> =A0 =A0 =A0 open-source software related to data serialization
> =A0 =A0 =A0 for distribution at no charge to the public.
>
> =A0 =A0 =A0 NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> =A0 =A0 =A0 Committee (PMC), to be known as the "Apache Avro Project",
> =A0 =A0 =A0 be and hereby is established pursuant to Bylaws of the
> =A0 =A0 =A0 Foundation; and be it further
>
> =A0 =A0 =A0 RESOLVED, that the Apache Avro Project be and hereby is
> =A0 =A0 =A0 responsible for the creation and maintenance of software
> =A0 =A0 =A0 related to data serialization; and be it further
>
> =A0 =A0 =A0 RESOLVED, that the office of "Vice President, Apache Avro" be
> =A0 =A0 =A0 and hereby is created, the person holding such office to
> =A0 =A0 =A0 serve at the direction of the Board of Directors as the chair
> =A0 =A0 =A0 of the Apache Avro Project, and to have primary responsibilit=
y
> =A0 =A0 =A0 for management of the projects within the scope of
> =A0 =A0 =A0 responsibility of the Apache Avro Project; and be it further
>
> =A0 =A0 =A0 RESOLVED, that the persons listed immediately below be and
> =A0 =A0 =A0 hereby are appointed to serve as the initial members of the
> =A0 =A0 =A0 Apache Avro Project:
>
> =A0 =A0 =A0 =A0 * Doug Cutting =A0 =A0 =A0 =A0 <cutting@apache.org>
> =A0 =A0 =A0 =A0 * Jeff Hammerbacher =A0 =A0<hammer@apache.org>
> =A0 =A0 =A0 =A0 * Jeff Hodges =A0 =A0 =A0 =A0 =A0<jmhodges@apache.org>
> =A0 =A0 =A0 =A0 * Matt Massie =A0 =A0 =A0 =A0 =A0<massie@apache.org>
> =A0 =A0 =A0 =A0 * Philip Zeyliger =A0 =A0 =A0<philz@apache.org>
> =A0 =A0 =A0 =A0 * Scott Banachowski =A0 =A0<sbanacho@apache.org>
> =A0 =A0 =A0 =A0 * Scott Carey =A0 =A0 =A0 =A0 =A0<scottcarey@apache.org>
> =A0 =A0 =A0 =A0 * Sharad Agarwal =A0 =A0 =A0 <sharadag@apache.org>
> =A0 =A0 =A0 =A0 * Thiruvalluvan M. G. =A0<thiru@apache.org>
>
> =A0 =A0 =A0 NOW, THEREFORE, BE IT FURTHER RESOLVED, that Matt Massie
> =A0 =A0 =A0 be appointed to the office of Vice President, Apache Avro, to
> =A0 =A0 =A0 serve in accordance with and subject to the direction of the
> =A0 =A0 =A0 Board of Directors and the Bylaws of the Foundation until
> =A0 =A0 =A0 death, resignation, retirement, removal or disqualification,
> =A0 =A0 =A0 or until a successor is appointed; and be it further
>
> =A0 =A0 =A0 RESOLVED, that the initial Apache Avro PMC be and hereby is
> =A0 =A0 =A0 tasked with the creation of a set of bylaws intended to
> =A0 =A0 =A0 encourage open development and increased participation in the
> =A0 =A0 =A0 Apache Avro Project; and be it further
>
> =A0 =A0 =A0 RESOLVED, that the Apache Avro Project be and hereby
> =A0 =A0 =A0 is tasked with the migration and rationalization of the Apach=
e
> =A0 =A0 =A0 Hadoop Avro sub-project; and be it further
>
> =A0 =A0 =A0 RESOLVED, that all responsibilities pertaining to the Apache
> =A0 =A0 =A0 Hadoop Avro sub-project encumbered upon the
> =A0 =A0 =A0 Apache Hadoop Project are hereafter discharged.
>
>

From general-return-1301-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 22:35:33 2010
Return-Path: <general-return-1301-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45845 invoked from network); 6 Apr 2010 22:35:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 22:35:33 -0000
Received: (qmail 85683 invoked by uid 500); 6 Apr 2010 22:35:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85611 invoked by uid 500); 6 Apr 2010 22:35:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85602 invoked by uid 99); 6 Apr 2010 22:35:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 22:35:32 +0000
X-ASF-Spam-Status: No, hits=0.4 required=10.0
	tests=AWL,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.30.64] (HELO qmta07.emeryville.ca.mail.comcast.net) (76.96.30.64)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 22:35:23 +0000
Received: from omta20.emeryville.ca.mail.comcast.net ([76.96.30.87])
	by qmta07.emeryville.ca.mail.comcast.net with comcast
	id 2NAb1e00H1smiN4A7Nb4bk; Tue, 06 Apr 2010 22:35:04 +0000
Received: from wlan-snve-152-247.corp.yahoo.com ([209.131.62.115])
	by omta20.emeryville.ca.mail.comcast.net with comcast
	id 2Nai1e0032VBGtd8gNar4i; Tue, 06 Apr 2010 22:34:58 +0000
Message-Id: <15631D3C-EEA4-41CF-B0E9-58E54CDA1A0F@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: HBase as TLP?
Date: Tue, 6 Apr 2010 15:34:39 -0700
X-Mailer: Apple Mail (2.936)

Stack,
     Was there any resolution from the HBase community about moving to  
a TLP?

-- Owen

From general-return-1302-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 22:45:39 2010
Return-Path: <general-return-1302-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47761 invoked from network); 6 Apr 2010 22:45:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 22:45:39 -0000
Received: (qmail 96591 invoked by uid 500); 6 Apr 2010 22:45:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96539 invoked by uid 500); 6 Apr 2010 22:45:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96531 invoked by uid 99); 6 Apr 2010 22:45:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 22:45:38 +0000
X-ASF-Spam-Status: No, hits=-0.8 required=10.0
	tests=AWL,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 22:45:33 +0000
Received: from [127.0.0.1] (gentlepaint-lx.corp.yahoo.com [10.72.185.127])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o36MiRKg019631
	for <general@hadoop.apache.org>; Tue, 6 Apr 2010 15:44:29 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=vEHZ2hw3fek0KdJse2ptkjBerHXgd53SxSvKelc/bb6T3t7dw53zMjP29L6xw4UT
Message-ID: <4BBBB94B.3080502@yahoo-inc.com>
Date: Tue, 06 Apr 2010 15:44:27 -0700
From: Konstantin Shvachko <shv@yahoo-inc.com>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Avro as TLP?
References: <4BBB9675.1090909@apache.org>
In-Reply-To: <4BBB9675.1090909@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

+1

--Konstantin

On 4/6/2010 1:15 PM, Doug Cutting wrote:
> Please vote as to whether you think Avro should become a top-level
> Apache project, as mentioned previously on this list.
>
> I've included below a draft board resolution. It lists all current
> committers as the initial members of the project management committee
> (PMC) and Matt Massie as the initial chair.
>
> Does the Hadoop PMC support sending this proposal on to the board?
>
> Thanks,
>
> Doug
>
> ------------
>
> X. Establish the Apache Avro Project
>
> WHEREAS, the Board of Directors deems it to be in the best
> interests of the Foundation and consistent with the
> Foundation's purpose to establish a Project Management
> Committee charged with the creation and maintenance of
> open-source software related to data serialization
> for distribution at no charge to the public.
>
> NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> Committee (PMC), to be known as the "Apache Avro Project",
> be and hereby is established pursuant to Bylaws of the
> Foundation; and be it further
>
> RESOLVED, that the Apache Avro Project be and hereby is
> responsible for the creation and maintenance of software
> related to data serialization; and be it further
>
> RESOLVED, that the office of "Vice President, Apache Avro" be
> and hereby is created, the person holding such office to
> serve at the direction of the Board of Directors as the chair
> of the Apache Avro Project, and to have primary responsibility
> for management of the projects within the scope of
> responsibility of the Apache Avro Project; and be it further
>
> RESOLVED, that the persons listed immediately below be and
> hereby are appointed to serve as the initial members of the
> Apache Avro Project:
>
> * Doug Cutting <cutting@apache.org>
> * Jeff Hammerbacher <hammer@apache.org>
> * Jeff Hodges <jmhodges@apache.org>
> * Matt Massie <massie@apache.org>
> * Philip Zeyliger <philz@apache.org>
> * Scott Banachowski <sbanacho@apache.org>
> * Scott Carey <scottcarey@apache.org>
> * Sharad Agarwal <sharadag@apache.org>
> * Thiruvalluvan M. G. <thiru@apache.org>
>
> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Matt Massie
> be appointed to the office of Vice President, Apache Avro, to
> serve in accordance with and subject to the direction of the
> Board of Directors and the Bylaws of the Foundation until
> death, resignation, retirement, removal or disqualification,
> or until a successor is appointed; and be it further
>
> RESOLVED, that the initial Apache Avro PMC be and hereby is
> tasked with the creation of a set of bylaws intended to
> encourage open development and increased participation in the
> Apache Avro Project; and be it further
>
> RESOLVED, that the Apache Avro Project be and hereby
> is tasked with the migration and rationalization of the Apache
> Hadoop Avro sub-project; and be it further
>
> RESOLVED, that all responsibilities pertaining to the Apache
> Hadoop Avro sub-project encumbered upon the
> Apache Hadoop Project are hereafter discharged.
>
>


From general-return-1303-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 06:34:12 2010
Return-Path: <general-return-1303-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33499 invoked from network); 7 Apr 2010 06:34:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 06:34:12 -0000
Received: (qmail 51366 invoked by uid 500); 7 Apr 2010 06:34:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50169 invoked by uid 500); 7 Apr 2010 06:34:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50151 invoked by uid 99); 7 Apr 2010 06:34:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 06:34:09 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of rosegun38@gmail.com designates 209.85.221.193 as permitted sender)
Received: from [209.85.221.193] (HELO mail-qy0-f193.google.com) (209.85.221.193)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 06:34:02 +0000
Received: by qyk31 with SMTP id 31so1282975qyk.20
        for <general@hadoop.apache.org>; Tue, 06 Apr 2010 23:33:42 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type;
        bh=6t8EFcz6xtr35c1Z2YeI7pRgrmB0z+3WQD5QrDP67iQ=;
        b=UAjwx0Rk6FqHHwlVoDr+ddE0cwPN3h+Kkpqev3sF9Ghd5Ztil4RXrQSZ7vXT/o0Cf0
         zVngxqwwgJfSSB9UYVxStE60JF4QKm0MCVYxqHuNwJc9Yh5C8KF1D7pUPvGhLHTXfBPR
         rQyBl74SzW+NtpsHPM/totNQmEVtY3bJ2Yg7M=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=vo8lOp1b8nijOFuY8vzIBmcbtSLo97SBCYheni0l5kI1zxVtpOG+ZHqU/xgeRRw33S
         cls1crkD5foEbi/qY7b6GZ4TZ8uhOMSyjz4q9nPcrgQWFOrVan5vEgSR2QiFnCFxv+9r
         RWg09l9tDdcDIPUWX0hH8zE6GpNjy1k7Y38Ps=
MIME-Version: 1.0
Received: by 10.224.74.70 with HTTP; Tue, 6 Apr 2010 23:33:41 -0700 (PDT)
In-Reply-To: <4BBB9644.7020201@apache.org>
References: <4BB28E7C.90300@apache.org> <4BBB9644.7020201@apache.org>
Date: Wed, 7 Apr 2010 14:33:41 +0800
Received: by 10.224.73.89 with SMTP id p25mr1938953qaj.82.1270622021554; Tue, 
	06 Apr 2010 23:33:41 -0700 (PDT)
Message-ID: <r2sbd8f42271004062333vf876fb15m711ee82355e0e5bc@mail.gmail.com>
Subject: Re: [DISCUSS] Avro as TLP?
From: Jason Mark <rosegun38@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00c09f93d9723d466404839fbe7f

--00c09f93d9723d466404839fbe7f
Content-Type: text/plain; charset=ISO-8859-1

+1

2010/4/7 Doug Cutting <cutting@apache.org>

> Doug Cutting wrote:
>
>> Does the Hadoop community have any questions or concerns about this
>> proposal?
>>
>
> Since no issues were raised, I'll now call the vote.
>
> Doug
>

--00c09f93d9723d466404839fbe7f--

From general-return-1304-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 06:46:44 2010
Return-Path: <general-return-1304-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35017 invoked from network); 7 Apr 2010 06:46:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 06:46:44 -0000
Received: (qmail 59793 invoked by uid 500); 7 Apr 2010 06:46:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59643 invoked by uid 500); 7 Apr 2010 06:46:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59634 invoked by uid 99); 7 Apr 2010 06:46:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 06:46:40 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 06:46:33 +0000
Received: from [192.168.1.71] ([10.72.244.164])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o376jMjr049877
	for <general@hadoop.apache.org>; Tue, 6 Apr 2010 23:45:22 -0700 (PDT)
Message-Id: <40BF62EE-DCAB-48CE-B063-F06243662F75@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <4BBB9675.1090909@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Avro as TLP?
Date: Tue, 6 Apr 2010 23:45:22 -0700
References: <4BBB9675.1090909@apache.org>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

On Apr 6, 2010, at 1:15 PM, Doug Cutting wrote:
> Does the Hadoop PMC support sending this proposal on to the board?
>
> Thanks,

+1

Arun


From general-return-1305-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 16:54:01 2010
Return-Path: <general-return-1305-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72320 invoked from network); 7 Apr 2010 16:54:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 16:54:01 -0000
Received: (qmail 32932 invoked by uid 500); 7 Apr 2010 16:54:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32865 invoked by uid 500); 7 Apr 2010 16:53:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32849 invoked by uid 99); 7 Apr 2010 16:53:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 16:53:59 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 16:53:51 +0000
Received: from [10.73.135.252] ([10.73.135.252])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o37Gqmt6079801;
	Wed, 7 Apr 2010 09:52:48 -0700 (PDT)
Message-ID: <4BBCB864.6010608@apache.org>
Date: Wed, 07 Apr 2010 09:52:52 -0700
From: Patrick Hunt <phunt@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4
MIME-Version: 1.0
To: general@hadoop.apache.org
CC: Doug Cutting <cutting@apache.org>
Subject: Re: [VOTE] Avro as TLP?
References: <4BBB9675.1090909@apache.org>
In-Reply-To: <4BBB9675.1090909@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Patrick

On 04/06/2010 01:15 PM, Doug Cutting wrote:
> Please vote as to whether you think Avro should become a top-level
> Apache project, as mentioned previously on this list.
>
> I've included below a draft board resolution. It lists all current
> committers as the initial members of the project management committee
> (PMC) and Matt Massie as the initial chair.
>
> Does the Hadoop PMC support sending this proposal on to the board?
>
> Thanks,
>
> Doug
>
> ------------
>
> X. Establish the Apache Avro Project
>
> WHEREAS, the Board of Directors deems it to be in the best
> interests of the Foundation and consistent with the
> Foundation's purpose to establish a Project Management
> Committee charged with the creation and maintenance of
> open-source software related to data serialization
> for distribution at no charge to the public.
>
> NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> Committee (PMC), to be known as the "Apache Avro Project",
> be and hereby is established pursuant to Bylaws of the
> Foundation; and be it further
>
> RESOLVED, that the Apache Avro Project be and hereby is
> responsible for the creation and maintenance of software
> related to data serialization; and be it further
>
> RESOLVED, that the office of "Vice President, Apache Avro" be
> and hereby is created, the person holding such office to
> serve at the direction of the Board of Directors as the chair
> of the Apache Avro Project, and to have primary responsibility
> for management of the projects within the scope of
> responsibility of the Apache Avro Project; and be it further
>
> RESOLVED, that the persons listed immediately below be and
> hereby are appointed to serve as the initial members of the
> Apache Avro Project:
>
> * Doug Cutting <cutting@apache.org>
> * Jeff Hammerbacher <hammer@apache.org>
> * Jeff Hodges <jmhodges@apache.org>
> * Matt Massie <massie@apache.org>
> * Philip Zeyliger <philz@apache.org>
> * Scott Banachowski <sbanacho@apache.org>
> * Scott Carey <scottcarey@apache.org>
> * Sharad Agarwal <sharadag@apache.org>
> * Thiruvalluvan M. G. <thiru@apache.org>
>
> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Matt Massie
> be appointed to the office of Vice President, Apache Avro, to
> serve in accordance with and subject to the direction of the
> Board of Directors and the Bylaws of the Foundation until
> death, resignation, retirement, removal or disqualification,
> or until a successor is appointed; and be it further
>
> RESOLVED, that the initial Apache Avro PMC be and hereby is
> tasked with the creation of a set of bylaws intended to
> encourage open development and increased participation in the
> Apache Avro Project; and be it further
>
> RESOLVED, that the Apache Avro Project be and hereby
> is tasked with the migration and rationalization of the Apache
> Hadoop Avro sub-project; and be it further
>
> RESOLVED, that all responsibilities pertaining to the Apache
> Hadoop Avro sub-project encumbered upon the
> Apache Hadoop Project are hereafter discharged.
>

From general-return-1306-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 17:02:43 2010
Return-Path: <general-return-1306-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 75474 invoked from network); 7 Apr 2010 17:02:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 17:02:43 -0000
Received: (qmail 50931 invoked by uid 500); 7 Apr 2010 17:02:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50895 invoked by uid 500); 7 Apr 2010 17:02:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50885 invoked by uid 99); 7 Apr 2010 17:02:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 17:02:42 +0000
X-ASF-Spam-Status: No, hits=-0.6 required=10.0
	tests=AWL,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 209.85.221.193 as permitted sender)
Received: from [209.85.221.193] (HELO mail-qy0-f193.google.com) (209.85.221.193)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 17:02:36 +0000
Received: by qyk31 with SMTP id 31so2047984qyk.20
        for <general@hadoop.apache.org>; Wed, 07 Apr 2010 10:02:15 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:received:message-id:subject
         :from:to:content-type:content-transfer-encoding;
        bh=0gqkHgtzcD90CjhcRz2cla1Nt2pLPYQnp7nUEuYueh8=;
        b=PAISfAGbiw8E/ZPSuXQYCA0nPCl+eYMGxyYzVqwQuP3By8tw6TI/nR6XfyZly1PwYu
         a1SYhTNfBm7NamzQ86ru2uZh0pCya3BK6uw0QLlPb6QzBtcGFX5N0Osu52z9aiec3BOc
         4LNmDbkksjJwSh1etGS/7pmpK1xGIBIVCtp08=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=Rpmb0Usi8Kk77kFs51dkqd8Y5tcsb6kHZnjQOlgChDFkZRnaFQILOwiMH2MBx9cttL
         71yGFV3UYESvQ3BpOdCkzycPp2MzBtyzkid40JNW7OkvGRka3APt+mdvftz9hhavpTtT
         id3LJvlfLw2sndWTrnSVoaBmDVxruopqNz0DU=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
Received: by 10.229.10.133 with HTTP; Wed, 7 Apr 2010 10:02:14 -0700 (PDT)
In-Reply-To: <15631D3C-EEA4-41CF-B0E9-58E54CDA1A0F@apache.org>
References: <15631D3C-EEA4-41CF-B0E9-58E54CDA1A0F@apache.org>
Date: Wed, 7 Apr 2010 10:02:14 -0700
X-Google-Sender-Auth: 1401346c14fe9599
Received: by 10.229.218.21 with SMTP id ho21mr2691684qcb.79.1270659734996; 
	Wed, 07 Apr 2010 10:02:14 -0700 (PDT)
Message-ID: <u2u7c962aed1004071002t8a12885aj29479e93300171ba@mail.gmail.com>
Subject: Re: HBase as TLP?
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Consensus would seem to favor going to TLP.  Let me run a quick vote.
Will be back.
Thanks,
St.Ack

On Tue, Apr 6, 2010 at 3:34 PM, Owen O'Malley <omalley@apache.org> wrote:
> Stack,
> =A0 =A0Was there any resolution from the HBase community about moving to =
a TLP?
>
> -- Owen
>

From general-return-1307-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 17:30:03 2010
Return-Path: <general-return-1307-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82656 invoked from network); 7 Apr 2010 17:30:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 17:30:02 -0000
Received: (qmail 93145 invoked by uid 500); 7 Apr 2010 17:30:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93096 invoked by uid 500); 7 Apr 2010 17:30:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93079 invoked by uid 99); 7 Apr 2010 17:30:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 17:30:01 +0000
X-ASF-Spam-Status: No, hits=-0.5 required=10.0
	tests=AWL,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 17:29:57 +0000
Received: from [10.73.135.252] (wifi-e-135-252.corp.yahoo.com [10.73.135.252])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o37HSu0N062492;
	Wed, 7 Apr 2010 10:28:56 -0700 (PDT)
Message-ID: <4BBCC0D8.1030407@apache.org>
Date: Wed, 07 Apr 2010 10:28:56 -0700
From: Patrick Hunt <phunt@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4
MIME-Version: 1.0
To: "Owen O'Malley" <omalley@apache.org>, general@hadoop.apache.org
Subject: ZooKeeper as TLP discussion results.
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Please find the results of the ZooKeeper as TLP discussion here.

http://bit.ly/c4fuZT

There was consensus amongst the development team that we will stay as a 
subproject of Hadoop for the time being. Full details of the discussion 
can be found in the thread provided.

Patrick

From general-return-1308-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 17:31:14 2010
Return-Path: <general-return-1308-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82937 invoked from network); 7 Apr 2010 17:31:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 17:31:14 -0000
Received: (qmail 95654 invoked by uid 500); 7 Apr 2010 17:31:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95624 invoked by uid 500); 7 Apr 2010 17:31:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95616 invoked by uid 99); 7 Apr 2010 17:31:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 17:31:13 +0000
X-ASF-Spam-Status: No, hits=0.4 required=10.0
	tests=AWL,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 17:31:08 +0000
Received: from pugholehand-lm.corp.yahoo.com (pugholehand-lm.corp.yahoo.com [10.72.115.44])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o37HTs01095168
	for <general@hadoop.apache.org>; Wed, 7 Apr 2010 10:29:54 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=pi2SCOF+txvgnt6KLOEmgER/KqueVABNZJS2OSl4u0kyt0kGOn2FLtuwRruv1HwT
Message-Id: <11F4AE05-627B-4A41-B5D3-A497D56BC93A@yahoo-inc.com>
From: Alan Gates <gates@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <4BBB9675.1090909@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Avro as TLP?
Date: Wed, 7 Apr 2010 10:29:53 -0700
References: <4BBB9675.1090909@apache.org>
X-Mailer: Apple Mail (2.936)

+1

Alan.

On Apr 6, 2010, at 1:15 PM, Doug Cutting wrote:

> Does the Hadoop PMC support sending this proposal on to the board?


From general-return-1309-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 19:01:18 2010
Return-Path: <general-return-1309-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 11427 invoked from network); 7 Apr 2010 19:01:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 19:01:17 -0000
Received: (qmail 3331 invoked by uid 500); 7 Apr 2010 19:01:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3298 invoked by uid 500); 7 Apr 2010 19:01:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3290 invoked by uid 99); 7 Apr 2010 19:01:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 19:01:16 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 07 Apr 2010 19:01:14 +0000
Received: (qmail 10995 invoked by uid 99); 7 Apr 2010 19:00:51 -0000
Received: from localhost.apache.org (HELO [192.168.42.156]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 19:00:51 +0000
Message-ID: <4BBCD662.80809@apache.org>
Date: Wed, 07 Apr 2010 12:00:50 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100317)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: ZooKeeper as TLP discussion results.
References: <4BBCC0D8.1030407@apache.org>
In-Reply-To: <4BBCC0D8.1030407@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Patrick Hunt wrote:
> Please find the results of the ZooKeeper as TLP discussion here.
> 
> http://bit.ly/c4fuZT
> 
> There was consensus amongst the development team that we will stay as a 
> subproject of Hadoop for the time being. Full details of the discussion 
> can be found in the thread provided.

In that discussion you state:
 > a) I do not think ZooKeeper currently has a sufficiently large and
 > diverse enough community such that it can fend for itself as a
 > TLP

How does the Hadoop PMC assist Zookeeper?  I see very little evidence 
that the Hadoop PMC is involved at all day-to-day operations of 
Zookeeper.  Currently the Hadoop PMC must vote on new Zookeeper 
committers and releases.  These votes usually pass with the minimum 
required number of PMC votes, often only after an appeal is made for 
more votes.  This does not seem like significant oversight.  Beyond 
that, I see little evidence of involvement by the Hadoop PMC in 
Zookeeper.  So I don't see this as a strong argument not to become a TLP.

If a group of committers is operating independently, then they ought to 
be an independent TLP.  The fact that you're a subproject operating 
independently only hides the lack of diversity, it doesn't help it.

 > b) Loss of branding and discover-ability

This also seems a poor reason to remain a sub-project.  We can retain 
prominent links to Zookeeper from hadoop.apache.org regardless of how 
the projects are structured.

 > c) "if ain't broke don't fix it". I have frequent interactions with
 > Hadoop PMC/Chair and an Apache board member.

That should not change as a TLP.  Apache encourages cross-project 
communication and collaboration.

The things that would change if Zookeeper were a TLP are:
  1. its official website would be at zookeeper.apache.org.
  2. it could vote for committers and releases directly rather than 
through the Hadoop PMC
  3. it would submit its quarterly report to the board directly, instead 
of via the Hadoop PMC.

That's pretty much it.

The board has a well established report and review system.  Each 
project's quarterly report is closely read and must be individually 
approved by a quorum of the board's members.  PMCs tend not to have such 
a review mechanism for subprojects.  Historically subprojects that 
develop problems have been slow to identify, and the problems have 
worsened in the meantime.  The focus of each PMC should be on direct 
decisions about code, committers and releases.  The board's job is to 
make sure that each PMC operates effectively.  These are different 
responsibilities and require different processes.

No one is going to force Zookeeper to become a TLP against its will, and 
no change must be made immediately.  But I think such a move would be 
easy to make, have significant upside for the project in simplifying its 
formal votes, and have significant upside to the foundation in 
facilitating the project's direct interaction with the board.

Doug

From general-return-1310-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 21:44:15 2010
Return-Path: <general-return-1310-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59888 invoked from network); 7 Apr 2010 21:44:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 21:44:15 -0000
Received: (qmail 26138 invoked by uid 500); 7 Apr 2010 21:44:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26082 invoked by uid 500); 7 Apr 2010 21:44:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26074 invoked by uid 99); 7 Apr 2010 21:44:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 21:44:14 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of dhruba@gmail.com designates 209.85.211.174 as permitted sender)
Received: from [209.85.211.174] (HELO mail-yw0-f174.google.com) (209.85.211.174)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 21:44:08 +0000
Received: by ywh4 with SMTP id 4so731831ywh.5
        for <general@hadoop.apache.org>; Wed, 07 Apr 2010 14:43:47 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type;
        bh=JnR0249Iin7+VnwT6Uv1xayzqka99BGOTefdp/QQ6hA=;
        b=jvRlAX7lPrbJ8X697JgkD+ACXlZVFDfhatAhU0WInWPqyZwSt9fAanPI7BJN/DHK/E
         cXa2/ARqgdJbdsGpLEDKsCsPuJj7oj2rl6liqHD59N0XOK0TupZx6ROXBeRuKtYDGl1x
         tLSU3GAUF4UT1QTtVTSBiUlwLmUJ2UAEc/zZA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=UvTCCFfxecrR6qDn5XMgO7HI+LPhtcDBcZ3qkjxKYh6x6ZiFHLXmVhISFnvEhbV/KX
         6n7F0Jnulu6/Zd2oCD0xDFmO7JrqPGsLsbmiqCAO0mKHqcfE/6wX2UA5NQDTGdchz2C7
         iNkaWff3VBNIi/k4motQ+C+BJogp4FGeHzHAk=
MIME-Version: 1.0
Received: by 10.151.50.17 with HTTP; Wed, 7 Apr 2010 14:43:47 -0700 (PDT)
In-Reply-To: <11F4AE05-627B-4A41-B5D3-A497D56BC93A@yahoo-inc.com>
References: <4BBB9675.1090909@apache.org>
	 <11F4AE05-627B-4A41-B5D3-A497D56BC93A@yahoo-inc.com>
Date: Wed, 7 Apr 2010 14:43:47 -0700
Received: by 10.150.254.17 with SMTP id b17mr1777774ybi.36.1270676627612; Wed, 
	07 Apr 2010 14:43:47 -0700 (PDT)
Message-ID: <o2x4aa34eb71004071443u2016e8dfxa67f1dedd897047e@mail.gmail.com>
Subject: Re: [VOTE] Avro as TLP?
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd349c00365be0483ac75c7
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd349c00365be0483ac75c7
Content-Type: text/plain; charset=ISO-8859-1

+1

-dhruba

On Wed, Apr 7, 2010 at 10:29 AM, Alan Gates <gates@yahoo-inc.com> wrote:

> +1
>
> Alan.
>
>
> On Apr 6, 2010, at 1:15 PM, Doug Cutting wrote:
>
>  Does the Hadoop PMC support sending this proposal on to the board?
>>
>
>


-- 
Connect to me at http://www.facebook.com/dhruba

--000e0cd349c00365be0483ac75c7--

From general-return-1311-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 00:04:39 2010
Return-Path: <general-return-1311-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 89249 invoked from network); 8 Apr 2010 00:04:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 00:04:38 -0000
Received: (qmail 56591 invoked by uid 500); 8 Apr 2010 00:04:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56483 invoked by uid 500); 8 Apr 2010 00:04:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 85335 invoked by uid 99); 7 Apr 2010 18:43:37 -0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.103 as permitted sender)
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; charset=us-ascii; format=flowed
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0
 reason=mlx engine=5.0.0-0908210000 definitions=main-1004070129
Message-id: <1F12CBA8-F3F3-4A5A-8F01-BD7B299A6DEC@mac.com>
From: Nigel Daley <ndaley@mac.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-reply-to: <4BBCB864.6010608@apache.org>
X-Mailer: iPhone Mail (7E18)
Subject: Re: [VOTE] Avro as TLP?
Date: Wed, 07 Apr 2010 11:42:31 -0700
References: <4BBB9675.1090909@apache.org> <4BBCB864.6010608@apache.org>

+1

Cheers,
Nige

> On 04/06/2010 01:15 PM, Doug Cutting wrote:
>> Please vote as to whether you think Avro should become a top-level
>> Apache project, as mentioned previously on this list.
>>
>> I've included below a draft board resolution. It lists all current
>> committers as the initial members of the project management committee
>> (PMC) and Matt Massie as the initial chair.
>>
>> Does the Hadoop PMC support sending this proposal on to the board?
>>
>> Thanks,
>>
>> Doug
>>
>> ------------
>>
>> X. Establish the Apache Avro Project
>>
>> WHEREAS, the Board of Directors deems it to be in the best
>> interests of the Foundation and consistent with the
>> Foundation's purpose to establish a Project Management
>> Committee charged with the creation and maintenance of
>> open-source software related to data serialization
>> for distribution at no charge to the public.
>>
>> NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>> Committee (PMC), to be known as the "Apache Avro Project",
>> be and hereby is established pursuant to Bylaws of the
>> Foundation; and be it further
>>
>> RESOLVED, that the Apache Avro Project be and hereby is
>> responsible for the creation and maintenance of software
>> related to data serialization; and be it further
>>
>> RESOLVED, that the office of "Vice President, Apache Avro" be
>> and hereby is created, the person holding such office to
>> serve at the direction of the Board of Directors as the chair
>> of the Apache Avro Project, and to have primary responsibility
>> for management of the projects within the scope of
>> responsibility of the Apache Avro Project; and be it further
>>
>> RESOLVED, that the persons listed immediately below be and
>> hereby are appointed to serve as the initial members of the
>> Apache Avro Project:
>>
>> * Doug Cutting <cutting@apache.org>
>> * Jeff Hammerbacher <hammer@apache.org>
>> * Jeff Hodges <jmhodges@apache.org>
>> * Matt Massie <massie@apache.org>
>> * Philip Zeyliger <philz@apache.org>
>> * Scott Banachowski <sbanacho@apache.org>
>> * Scott Carey <scottcarey@apache.org>
>> * Sharad Agarwal <sharadag@apache.org>
>> * Thiruvalluvan M. G. <thiru@apache.org>
>>
>> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Matt Massie
>> be appointed to the office of Vice President, Apache Avro, to
>> serve in accordance with and subject to the direction of the
>> Board of Directors and the Bylaws of the Foundation until
>> death, resignation, retirement, removal or disqualification,
>> or until a successor is appointed; and be it further
>>
>> RESOLVED, that the initial Apache Avro PMC be and hereby is
>> tasked with the creation of a set of bylaws intended to
>> encourage open development and increased participation in the
>> Apache Avro Project; and be it further
>>
>> RESOLVED, that the Apache Avro Project be and hereby
>> is tasked with the migration and rationalization of the Apache
>> Hadoop Avro sub-project; and be it further
>>
>> RESOLVED, that all responsibilities pertaining to the Apache
>> Hadoop Avro sub-project encumbered upon the
>> Apache Hadoop Project are hereafter discharged.
>>

From general-return-1312-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 01:15:06 2010
Return-Path: <general-return-1312-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4639 invoked from network); 8 Apr 2010 01:15:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 01:15:05 -0000
Received: (qmail 98888 invoked by uid 500); 8 Apr 2010 01:15:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98825 invoked by uid 500); 8 Apr 2010 01:15:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98817 invoked by uid 99); 8 Apr 2010 01:15:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 01:15:04 +0000
X-ASF-Spam-Status: No, hits=-0.3 required=10.0
	tests=AWL,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 01:14:58 +0000
Received: from SNV-EXPF01.ds.corp.yahoo.com (snv-expf01.ds.corp.yahoo.com [207.126.227.250])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o381E9nS003787
	for <general@hadoop.apache.org>; Wed, 7 Apr 2010 18:14:09 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:x-originalarrivaltime;
	b=UPrxJ3z9cHU69xpN9yw3dYTVP5/hlXMSX0lcvINII6TcyyWfAY50SyB122jkYWsV
Received: from SNV-EXVS09.ds.corp.yahoo.com ([207.126.227.86]) by SNV-EXPF01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Wed, 7 Apr 2010 18:14:09 -0700
Received: from 10.73.146.106 ([10.73.146.106]) by SNV-EXVS09.ds.corp.yahoo.com ([207.126.227.84]) via Exchange Front-End Server snv-webmail.corp.yahoo.com ([207.126.227.59]) with Microsoft Exchange Server HTTP-DAV ;
 Thu,  8 Apr 2010 01:14:09 +0000
User-Agent: Microsoft-Entourage/12.23.0.091001
Date: Wed, 07 Apr 2010 18:14:08 -0700
Subject: Re: [VOTE] Avro as TLP?
From: Mahadev Konar <mahadev@yahoo-inc.com>
To: <general@hadoop.apache.org>
Message-ID: <C7E27BF0.3316A%mahadev@yahoo-inc.com>
Thread-Topic: [VOTE] Avro as TLP?
Thread-Index: AcrWuMm2x+uX32onaEWwwvvP3zi75g==
In-Reply-To: <1F12CBA8-F3F3-4A5A-8F01-BD7B299A6DEC@mac.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-OriginalArrivalTime: 08 Apr 2010 01:14:09.0830 (UTC) FILETIME=[CACDEC60:01CAD6B8]

+1... 


Thanks
mahadev


On 4/7/10 11:42 AM, "Nigel Daley" <ndaley@mac.com> wrote:

> +1
> 
> Cheers,
> Nige
> 
>> On 04/06/2010 01:15 PM, Doug Cutting wrote:
>>> Please vote as to whether you think Avro should become a top-level
>>> Apache project, as mentioned previously on this list.
>>> 
>>> I've included below a draft board resolution. It lists all current
>>> committers as the initial members of the project management committee
>>> (PMC) and Matt Massie as the initial chair.
>>> 
>>> Does the Hadoop PMC support sending this proposal on to the board?
>>> 
>>> Thanks,
>>> 
>>> Doug
>>> 
>>> ------------
>>> 
>>> X. Establish the Apache Avro Project
>>> 
>>> WHEREAS, the Board of Directors deems it to be in the best
>>> interests of the Foundation and consistent with the
>>> Foundation's purpose to establish a Project Management
>>> Committee charged with the creation and maintenance of
>>> open-source software related to data serialization
>>> for distribution at no charge to the public.
>>> 
>>> NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>>> Committee (PMC), to be known as the "Apache Avro Project",
>>> be and hereby is established pursuant to Bylaws of the
>>> Foundation; and be it further
>>> 
>>> RESOLVED, that the Apache Avro Project be and hereby is
>>> responsible for the creation and maintenance of software
>>> related to data serialization; and be it further
>>> 
>>> RESOLVED, that the office of "Vice President, Apache Avro" be
>>> and hereby is created, the person holding such office to
>>> serve at the direction of the Board of Directors as the chair
>>> of the Apache Avro Project, and to have primary responsibility
>>> for management of the projects within the scope of
>>> responsibility of the Apache Avro Project; and be it further
>>> 
>>> RESOLVED, that the persons listed immediately below be and
>>> hereby are appointed to serve as the initial members of the
>>> Apache Avro Project:
>>> 
>>> * Doug Cutting <cutting@apache.org>
>>> * Jeff Hammerbacher <hammer@apache.org>
>>> * Jeff Hodges <jmhodges@apache.org>
>>> * Matt Massie <massie@apache.org>
>>> * Philip Zeyliger <philz@apache.org>
>>> * Scott Banachowski <sbanacho@apache.org>
>>> * Scott Carey <scottcarey@apache.org>
>>> * Sharad Agarwal <sharadag@apache.org>
>>> * Thiruvalluvan M. G. <thiru@apache.org>
>>> 
>>> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Matt Massie
>>> be appointed to the office of Vice President, Apache Avro, to
>>> serve in accordance with and subject to the direction of the
>>> Board of Directors and the Bylaws of the Foundation until
>>> death, resignation, retirement, removal or disqualification,
>>> or until a successor is appointed; and be it further
>>> 
>>> RESOLVED, that the initial Apache Avro PMC be and hereby is
>>> tasked with the creation of a set of bylaws intended to
>>> encourage open development and increased participation in the
>>> Apache Avro Project; and be it further
>>> 
>>> RESOLVED, that the Apache Avro Project be and hereby
>>> is tasked with the migration and rationalization of the Apache
>>> Hadoop Avro sub-project; and be it further
>>> 
>>> RESOLVED, that all responsibilities pertaining to the Apache
>>> Hadoop Avro sub-project encumbered upon the
>>> Apache Hadoop Project are hereafter discharged.
>>> 


From general-return-1313-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 02:30:03 2010
Return-Path: <general-return-1313-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14004 invoked from network); 8 Apr 2010 02:30:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 02:30:03 -0000
Received: (qmail 59425 invoked by uid 500); 8 Apr 2010 02:30:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59369 invoked by uid 500); 8 Apr 2010 02:30:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59361 invoked by uid 99); 8 Apr 2010 02:30:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 02:30:02 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [216.252.110.215] (HELO web56206.mail.re3.yahoo.com) (216.252.110.215)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 08 Apr 2010 02:29:53 +0000
Received: (qmail 23865 invoked by uid 60001); 8 Apr 2010 02:29:31 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1270693771; bh=BPsm2o1/VcRqikYn2dtQo+tiouax/u4eNE02efuGFWk=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=jh10qbwu+cfCvEr8ZObvrauYI4QzBP81xS2mYSUEq9M3n3oevxY5cNIyvC0EgEKTNL/rD/vGjf+njINPYssBTz8i8z4aDXj+90/qR9RMN6lil8EW3vohYijXxcpRgRclo8r9uHR4RxtD5YEkjx8ApfXg0Cu0CyQq8FyctM89/CY=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=nEIwyAaBJ2dfSLcT2XaINDwVk8swE09sZ8bZYUw5q9xjylf3wvccExrNg4zEqYZ2R0dVeqphsV0chaazY+2lHlU8m2lgFIA5zrIYdY02d6TQzd2TB2rfaNpo8PYPZhovypR2jwd/vYvMZXXsefBOCi2JNiTOQF8jx0m3EfCzvAw=;
Message-ID: <434909.23548.qm@web56206.mail.re3.yahoo.com>
X-YMail-OSG: XAdBf0gVM1k2NpvXMVxaWPPex1WkyoKb4.VYlC4ZFjWH4BX
 8igjJrVWN0OkYXqbFmoWLucQOASYEU0gRCBnKX7mUuaW4aeHdo4tJY4xzJuJ
 aK8dbw0JFiOiQiZO12Wq8qPquuHnf_5oagluqz6NO6WLf7IFJ9M3SKS47hki
 9YCJyZAADaZ2cX7f.2dR_sZQsdWZZzZwdQZjt4U_WXQzbKcCMHps8VaFJJ_O
 U1WBt14hSWgIYTwI29JMpCxV9K5UGjdyh7pUIMIAtW6fNZasGmlTqHekgruC
 egTlWx94OCFPFB_cD_BSVnTRU5Gcl73bsjMEXq6ziS0UVrtC4nFw6jExbNBt
 BNg--
Received: from [98.207.82.20] by web56206.mail.re3.yahoo.com via HTTP; Wed, 07 Apr 2010 19:29:31 PDT
X-Mailer: YahooMailRC/324.3 YahooMailWebService/0.8.100.260964
References: <4BBB9675.1090909@apache.org>
Date: Wed, 7 Apr 2010 19:29:31 -0700 (PDT)
From: "Tsz Wo \(Nicholas\), Sze" <s29752-hadoopgeneral@yahoo.com>
Subject: Re: [VOTE] Avro as TLP?
To: general@hadoop.apache.org
In-Reply-To: <4BBB9675.1090909@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org

+1
Nicholas Sze




----- Original Message ----
> From: Doug Cutting <cutting@apache.org>
> To: general@hadoop.apache.org
> Sent: Tue, April 6, 2010 1:15:49 PM
> Subject: [VOTE] Avro as TLP?
> 
> Please vote as to whether you think Avro should become a top-level
Apache 
> project, as mentioned previously on this list.

I've included below a 
> draft board resolution.  It lists all current
committers as the initial 
> members of the project management committee
(PMC) and Matt Massie as the 
> initial chair.

Does the Hadoop PMC support sending this proposal on to 
> the board?

Thanks,

Doug

------------

    
> X. Establish the Apache Avro Project

       WHEREAS, the 
> Board of Directors deems it to be in the best
       interests 
> of the Foundation and consistent with the
       Foundation's 
> purpose to establish a Project Management
       Committee 
> charged with the creation and maintenance of
      
> open-source software related to data serialization
       for 
> distribution at no charge to the public.

       NOW, 
> THEREFORE, BE IT RESOLVED, that a Project Management
      
> Committee (PMC), to be known as the "Apache Avro Project",
    
>    be and hereby is established pursuant to Bylaws of the
    
>    Foundation; and be it further

       RESOLVED, 
> that the Apache Avro Project be and hereby is
      
> responsible for the creation and maintenance of software
      
> related to data serialization; and be it further

      
> RESOLVED, that the office of "Vice President, Apache Avro" be
    
>    and hereby is created, the person holding such office to
  
>      serve at the direction of the Board of Directors as the 
> chair
       of the Apache Avro Project, and to have primary 
> responsibility
       for management of the projects within 
> the scope of
       responsibility of the Apache Avro Project; 
> and be it further

       RESOLVED, that the persons listed 
> immediately below be and
       hereby are appointed to serve 
> as the initial members of the
       Apache Avro 
> Project:

         * Doug Cutting      
>    <
> href="mailto:cutting@apache.org">cutting@apache.org>
    
>      * Jeff Hammerbacher    <
> ymailto="mailto:hammer@apache.org" 
> href="mailto:hammer@apache.org">hammer@apache.org>
    
>      * Jeff Hodges          <
> ymailto="mailto:jmhodges@apache.org" 
> href="mailto:jmhodges@apache.org">jmhodges@apache.org>
    
>      * Matt Massie          <
> ymailto="mailto:massie@apache.org" 
> href="mailto:massie@apache.org">massie@apache.org>
    
>      * Philip Zeyliger      <
> ymailto="mailto:philz@apache.org" 
> href="mailto:philz@apache.org">philz@apache.org>
      
>    * Scott Banachowski    <
> ymailto="mailto:sbanacho@apache.org" 
> href="mailto:sbanacho@apache.org">sbanacho@apache.org>
    
>      * Scott Carey          <
> ymailto="mailto:scottcarey@apache.org" 
> href="mailto:scottcarey@apache.org">scottcarey@apache.org>
  
>        * Sharad Agarwal       <
> ymailto="mailto:sharadag@apache.org" 
> href="mailto:sharadag@apache.org">sharadag@apache.org>
    
>      * Thiruvalluvan M. G.  <
> ymailto="mailto:thiru@apache.org" 
> href="mailto:thiru@apache.org">thiru@apache.org>

    
>    NOW, THEREFORE, BE IT FURTHER RESOLVED, that Matt Massie
  
>      be appointed to the office of Vice President, Apache Avro, 
> to
       serve in accordance with and subject to the 
> direction of the
       Board of Directors and the Bylaws of 
> the Foundation until
       death, resignation, retirement, 
> removal or disqualification,
       or until a successor is 
> appointed; and be it further

       RESOLVED, that the 
> initial Apache Avro PMC be and hereby is
       tasked with 
> the creation of a set of bylaws intended to
       encourage 
> open development and increased participation in the
      
> Apache Avro Project; and be it further

       RESOLVED, 
> that the Apache Avro Project be and hereby
       is tasked 
> with the migration and rationalization of the Apache
      
> Hadoop Avro sub-project; and be it further

      
> RESOLVED, that all responsibilities pertaining to the Apache
    
>    Hadoop Avro sub-project encumbered upon the
      
> Apache Hadoop Project are hereafter discharged.

From general-return-1314-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 04:03:22 2010
Return-Path: <general-return-1314-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 32946 invoked from network); 8 Apr 2010 04:03:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 04:03:21 -0000
Received: (qmail 36199 invoked by uid 500); 8 Apr 2010 04:03:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36100 invoked by uid 500); 8 Apr 2010 04:03:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36092 invoked by uid 99); 8 Apr 2010 04:03:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 04:03:19 +0000
X-ASF-Spam-Status: No, hits=-0.6 required=10.0
	tests=AWL,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 209.85.221.193 as permitted sender)
Received: from [209.85.221.193] (HELO mail-qy0-f193.google.com) (209.85.221.193)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 04:03:14 +0000
Received: by qyk31 with SMTP id 31so2844930qyk.20
        for <general@hadoop.apache.org>; Wed, 07 Apr 2010 21:02:54 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:date
         :x-google-sender-auth:received:message-id:subject:from:to
         :content-type;
        bh=V8FjidTOtuvZGjz+pl41aE2iEaiwvSSE9TQuhBd3ftk=;
        b=qB7b2SVsAZ2+UFt0LECZkGgLRXSyGZpF//1fesjWGLPhzDT1XOTazFVfWGfJL7rrCp
         kP7dG0uhZFKHqP9TD+JIe0cKvKQYVZuTjaE2s0z2ohLPUF2VynO77MforKeqv7GTvsoD
         43qB8f8dK7x4mjQfGjQYdfhDNJG41RKregrYg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        b=dLesiwaamS+18XNj9nFf1jZaGU///hUHW415QomrNo2BYwMyfTBin+GwTxLywcpMeL
         3gNC+ykrMHywA9JYuV4UD7PwHfQOzMnYDNDCuryu3/dNa0tPDoTpHZBedTIf34RZg1B6
         7N/Gd8+ZweOlOffhXJfo50dZiHghOtt9t/oyY=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
Received: by 10.229.10.133 with HTTP; Wed, 7 Apr 2010 21:02:54 -0700 (PDT)
Date: Wed, 7 Apr 2010 21:02:54 -0700
X-Google-Sender-Auth: 84aad20640e708a3
Received: by 10.229.211.140 with SMTP id go12mr1117964qcb.32.1270699374139; 
	Wed, 07 Apr 2010 21:02:54 -0700 (PDT)
Message-ID: <x2l7c962aed1004072102j3ca29379he9f4095be7736f53@mail.gmail.com>
Subject: [DISCUSS] HBase as TLP
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

The HBase subproject has voted to become a TLP: http://su.pr/1g0HAN

Does the Hadoop community have any questions or concerns about this proposal?

Please don't vote yet in response to this.  I'll call a formal vote
after questions, if any, have been resolved.

Thanks,
St.Ack
(Thanks for the boiler plate Doug Cutting)

From general-return-1315-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 08:42:23 2010
Return-Path: <general-return-1315-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78422 invoked from network); 8 Apr 2010 08:42:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 08:42:23 -0000
Received: (qmail 57248 invoked by uid 500); 8 Apr 2010 08:42:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57167 invoked by uid 500); 8 Apr 2010 08:42:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57159 invoked by uid 99); 8 Apr 2010 08:42:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 08:42:21 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of bernd.fondermann@googlemail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 08:42:14 +0000
Received: by wwb39 with SMTP id 39so81110wwb.35
        for <general@hadoop.apache.org>; Thu, 08 Apr 2010 01:41:54 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=kUe1xjSRJFSXi5eJQ7BQFRXHMQFd3UDwImepJlsvZR0=;
        b=vyL+/NcaSnC/Lagax5wcXOEYydH0UxU3x/tYFvANqHlyNpgyxij4IAG1lfaCRm/eup
         GihiGPr3yJJTujG9PzFqaeIeIT6juAcF+ttYJwVlCi7bAhOLyyRUf3ki1/IbWsWiQm7l
         UHw2ajgzWelxPpEdzYvK2JQ6ASl3n1cpWHX1s=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=J+PL/PTXyHapiGJ11LbtJBhgTOPXThgqYCoPl6qRcOozr6pgbaVIJgYn9+jaXBZvyi
         cxir8a1n5Bf2rz4oJWHskuC3tCZgvdmfqQ6NmvGDr49Ra5IURpu00i2u0xBNQWazrQO1
         aLDIYu5UojLqIFTvr8arN0slH4tDYzGqRvuCs=
MIME-Version: 1.0
Received: by 10.216.16.35 with HTTP; Thu, 8 Apr 2010 01:41:54 -0700 (PDT)
In-Reply-To: <4BBCD662.80809@apache.org>
References: <4BBCC0D8.1030407@apache.org> <4BBCD662.80809@apache.org>
Date: Thu, 8 Apr 2010 10:41:54 +0200
Received: by 10.216.165.5 with SMTP id d5mr86247wel.143.1270716114409; Thu, 08 
	Apr 2010 01:41:54 -0700 (PDT)
Message-ID: <t2o88f6e29a1004080141u26531c1eo7576b9d060447f5e@mail.gmail.com>
Subject: Re: ZooKeeper as TLP discussion results.
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Wed, Apr 7, 2010 at 21:00, Doug Cutting <cutting@apache.org> wrote:
> Patrick Hunt wrote:
>>
>> Please find the results of the ZooKeeper as TLP discussion here.
>>
>> http://bit.ly/c4fuZT
>>
>> There was consensus amongst the development team that we will stay as a
>> subproject of Hadoop for the time being. Full details of the discussion =
can
>> be found in the thread provided.
>
> In that discussion you state:
>> a) I do not think ZooKeeper currently has a sufficiently large and
>> diverse enough community such that it can fend for itself as a
>> TLP
>
> How does the Hadoop PMC assist Zookeeper? =A0I see very little evidence t=
hat
> the Hadoop PMC is involved at all day-to-day operations of Zookeeper.
> =A0Currently the Hadoop PMC must vote on new Zookeeper committers and
> releases. =A0These votes usually pass with the minimum required number of=
 PMC
> votes, often only after an appeal is made for more votes. =A0This does no=
t
> seem like significant oversight. =A0Beyond that, I see little evidence of
> involvement by the Hadoop PMC in Zookeeper. =A0So I don't see this as a s=
trong
> argument not to become a TLP.
>
> If a group of committers is operating independently, then they ought to b=
e
> an independent TLP. =A0The fact that you're a subproject operating
> independently only hides the lack of diversity, it doesn't help it.
>
>> b) Loss of branding and discover-ability
>
> This also seems a poor reason to remain a sub-project. =A0We can retain
> prominent links to Zookeeper from hadoop.apache.org regardless of how the
> projects are structured.
>
>> c) "if ain't broke don't fix it". I have frequent interactions with
>> Hadoop PMC/Chair and an Apache board member.
>
> That should not change as a TLP. =A0Apache encourages cross-project
> communication and collaboration.
>
> The things that would change if Zookeeper were a TLP are:
> =A01. its official website would be at zookeeper.apache.org.
> =A02. it could vote for committers and releases directly rather than thro=
ugh
> the Hadoop PMC
> =A03. it would submit its quarterly report to the board directly, instead=
 of
> via the Hadoop PMC.
>
> That's pretty much it.
>
> The board has a well established report and review system. =A0Each projec=
t's
> quarterly report is closely read and must be individually approved by a
> quorum of the board's members. =A0PMCs tend not to have such a review
> mechanism for subprojects. =A0Historically subprojects that develop probl=
ems
> have been slow to identify, and the problems have worsened in the meantim=
e.
> =A0The focus of each PMC should be on direct decisions about code, commit=
ters
> and releases. =A0The board's job is to make sure that each PMC operates
> effectively. =A0These are different responsibilities and require differen=
t
> processes.
>
> No one is going to force Zookeeper to become a TLP against its will, and =
no
> change must be made immediately. =A0But I think such a move would be easy=
 to
> make, have significant upside for the project in simplifying its formal
> votes, and have significant upside to the foundation in facilitating the
> project's direct interaction with the board.
>
> Doug

+1 to all of this.

Additional comments.
If there would be 3 Zookeeper committers on the Hadoop PMC, then the
lack-of-oversight problem would go away and releases could be voted on
without delay.
I don't want to discuss why this currently is like it is - this should
really be discussed on private@hadoop.a.o.

However, going TLP requires experienced people. That means Zookeeper
will probably need Hadoop PMC members or others (ASF members, etc) to
join the initial PMC.
Otherwise, in my opinion the proper path could either be Zookeeper
growing up here for a while until a proper PMC can be forked from
Hadoop, or going through - hold your breath - incubation.

You know, this whole umbrella discussion should not result in a mass
exodus into TLPs which might collapse sooner or later.

   Bernd

From general-return-1316-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 09:25:50 2010
Return-Path: <general-return-1316-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85412 invoked from network); 8 Apr 2010 09:25:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 09:25:50 -0000
Received: (qmail 2359 invoked by uid 500); 8 Apr 2010 09:25:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2285 invoked by uid 500); 8 Apr 2010 09:25:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2276 invoked by uid 99); 8 Apr 2010 09:25:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 09:25:48 +0000
X-ASF-Spam-Status: No, hits=-0.1 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [65.115.85.73] (HELO smtp-outbound-2.vmware.com) (65.115.85.73)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 09:25:42 +0000
Received: from jupiter.vmware.com (mailhost5.vmware.com [10.16.68.131])
	by smtp-outbound-2.vmware.com (Postfix) with ESMTP id 9F55B460B5
	for <general@hadoop.apache.org>; Thu,  8 Apr 2010 02:25:21 -0700 (PDT)
Received: from pa-exht02.vmware.com (pa-exht02.vmware.com [10.113.81.168])
	by jupiter.vmware.com (Postfix) with ESMTP id 913CBDC297
	for <general@hadoop.apache.org>; Thu,  8 Apr 2010 02:25:21 -0700 (PDT)
Received: from EXCH-MBX-22.vmware.com ([10.113.81.177]) by
 pa-exht02.vmware.com ([10.113.81.168]) with mapi; Thu, 8 Apr 2010 02:25:21
 -0700
From: Zhanlei Ma <zma@vmware.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 8 Apr 2010 02:25:17 -0700
Subject: reserch the code of hadoop
Thread-Topic: reserch the code of hadoop
Thread-Index: AcrW/WcPsoK+yk6cR9WGwhHU1vPAIA==
Message-ID: <F21534570FE52940A8A9366D82C11E8B044EA8910A@EXCH-MBX-22.vmware.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_F21534570FE52940A8A9366D82C11E8B044EA8910AEXCHMBX22vmwa_"
MIME-Version: 1.0

--_000_F21534570FE52940A8A9366D82C11E8B044EA8910AEXCHMBX22vmwa_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

HI all:
   We know the fault tolence of hadoop when a datanode is failed, as runnin=
g hadoop. Howerver which source part resolving the situation? And how to so=
lve? Looking forward your reply.
   Best regards and thanks!
                                        zma

--_000_F21534570FE52940A8A9366D82C11E8B044EA8910AEXCHMBX22vmwa_--

From general-return-1317-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 10:09:09 2010
Return-Path: <general-return-1317-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 92186 invoked from network); 8 Apr 2010 10:09:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 10:09:09 -0000
Received: (qmail 53424 invoked by uid 500); 8 Apr 2010 10:09:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53232 invoked by uid 500); 8 Apr 2010 10:09:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53220 invoked by uid 99); 8 Apr 2010 10:09:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 10:09:07 +0000
X-ASF-Spam-Status: No, hits=-2.4 required=10.0
	tests=AWL,RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 10:08:59 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 303731BA4FA
	for <general@hadoop.apache.org>; Thu,  8 Apr 2010 11:08:37 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id yulPgOlqC8mw for <general@hadoop.apache.org>;
	Thu,  8 Apr 2010 11:08:36 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by colossus.hpl.hp.com (Postfix) with ESMTPS id B2C061BA4EF
	for <general@hadoop.apache.org>; Thu,  8 Apr 2010 11:08:36 +0100 (BST)
MailScanner-NULL-Check: 1271326104.79577@lGz9BcdwogcuL6P/BlGsaA
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o38A8KnF007748
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Thu, 8 Apr 2010 11:08:21 +0100 (BST)
Message-ID: <4BBDAB14.1050806@apache.org>
Date: Thu, 08 Apr 2010 11:08:20 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100228)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: reserch the code of hadoop
References: <F21534570FE52940A8A9366D82C11E8B044EA8910A@EXCH-MBX-22.vmware.com>
In-Reply-To: <F21534570FE52940A8A9366D82C11E8B044EA8910A@EXCH-MBX-22.vmware.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o38A8KnF007748
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

Zhanlei Ma wrote:
> HI all:
>    We know the fault tolence of hadoop when a datanode is failed, as running hadoop. 


>Howerver which source part resolving the situation? And how to solve? 

stuff in hadoop-hdfs; get on the hdfs-dev@hadoop.apache.org mailing list 
to discuss these problems.

Namenode: needs some way of feeding changes to peers in a (changing, 
volatile) set of namenodes such that any can failover to become the 
primary, or you do a complete -any NN accepts operations- world and you 
are in the domain of HA distributed databases, along with the 
performance problems

datanode: rebinding on failure

dfsclient: move away from a single DNS URL and do some rebinding lookup

This is all serious code and hard to test; everyone running production 
clusters would like HA, but not at the expense of performance *or any of 
their existing data*. You need to start talking to Yahoo! and Facebook 
before you begin touching the code to understand their needs, as the big 
datacentre teams will veto any change that doesn't work for them. The 
problems of small clusters, where small is a few hundred TB on tens of 
servers, are considered dealt with and it is the big facilities where 
the problems turn up. This is why the historical focus has been on rapid 
recovery and  (ideally) zero data loss over continuous availability

-steve

From general-return-1318-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 16:11:29 2010
Return-Path: <general-return-1318-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83945 invoked from network); 8 Apr 2010 16:11:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 16:11:28 -0000
Received: (qmail 22142 invoked by uid 500); 8 Apr 2010 16:11:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21963 invoked by uid 500); 8 Apr 2010 16:11:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21955 invoked by uid 99); 8 Apr 2010 16:11:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 16:11:27 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 08 Apr 2010 16:11:25 +0000
Received: (qmail 83900 invoked by uid 99); 8 Apr 2010 16:11:03 -0000
Received: from localhost.apache.org (HELO [192.168.168.103]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 16:11:03 +0000
Message-ID: <4BBE0016.4020502@apache.org>
Date: Thu, 08 Apr 2010 09:11:02 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100317)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: ZooKeeper as TLP discussion results.
References: <4BBCC0D8.1030407@apache.org> <4BBCD662.80809@apache.org> <t2o88f6e29a1004080141u26531c1eo7576b9d060447f5e@mail.gmail.com>
In-Reply-To: <t2o88f6e29a1004080141u26531c1eo7576b9d060447f5e@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Bernd Fondermann wrote:
> However, going TLP requires experienced people. That means Zookeeper
> will probably need Hadoop PMC members or others (ASF members, etc) to
> join the initial PMC.

The Hadoop PMC should use criteria similar to the Incubator for 
graduation to TLP.

http://incubator.apache.org/incubation/Incubation_Policy.html#Minimum+Graduation+Requirements

The only requirement in doubt is diversity.  Those diversity 
requirements are soft, not hard and fast.  When I raised this, several 
board members indicated that they would rather have non-diverse TLPs 
than non-diverse subprojects, since the board could then directly 
observe whether diversity is improving within the project, and, if it is 
not, provide assistance.

> Otherwise, in my opinion the proper path could either be Zookeeper
> growing up here for a while until a proper PMC can be forked from
> Hadoop, or going through - hold your breath - incubation.

These projects are already effectively in incubation, under the Hadoop 
PMC rather than under the Incubator PMC.  The question today is whether 
they're ready to graduate.  The board has asked for a report on this.

We need not rush into anything.  If we have doubts about some cases, 
then our report to the board might include questions about the criteria 
to promote subprojects to TLP, to help clarify things.

If we, as a PMC, wish to wait for diversity in a subproject to improve, 
then we must commit to regular monitoring that subproject's diversity, 
and, if it's not improving, try to figure out why, and act to help it 
improve.  Just as podlings should not languish in the incubator for 
years, a sub-project's goal should be promotion to TLP within a year or 
so of founding.  If we don't want to commit to this oversight, then 
either the board or the incubator PMC could.

Doug

From general-return-1319-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 16:53:05 2010
Return-Path: <general-return-1319-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 92841 invoked from network); 8 Apr 2010 16:53:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 16:53:05 -0000
Received: (qmail 85237 invoked by uid 500); 8 Apr 2010 16:53:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85162 invoked by uid 500); 8 Apr 2010 16:53:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85146 invoked by uid 99); 8 Apr 2010 16:53:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 16:53:04 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 16:52:55 +0000
Received: from [10.73.135.250] (wifi-e-135-250.corp.yahoo.com [10.73.135.250])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o38GqHKv013419;
	Thu, 8 Apr 2010 09:52:18 -0700 (PDT)
Message-ID: <4BBE09C1.6010706@apache.org>
Date: Thu, 08 Apr 2010 09:52:17 -0700
From: Patrick Hunt <phunt@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4
MIME-Version: 1.0
To: general@hadoop.apache.org
CC: Doug Cutting <cutting@apache.org>
Subject: Re: ZooKeeper as TLP discussion results.
References: <4BBCC0D8.1030407@apache.org> <4BBCD662.80809@apache.org> <t2o88f6e29a1004080141u26531c1eo7576b9d060447f5e@mail.gmail.com> <4BBE0016.4020502@apache.org>
In-Reply-To: <4BBE0016.4020502@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org


On 04/08/2010 09:11 AM, Doug Cutting wrote:
> http://incubator.apache.org/incubation/Incubation_Policy.html#Minimum+Graduation+Requirements

I can speak only for ZK but we are hitting on all cylinders from this 
list. All the process/infra/legal items have long ago been committed. We 
have strong partnerships with multiple Apache projects (HBase and Solr 
in particular) and a dedication to "the apache way".

 > The only requirement in doubt is diversity.

If you read through the full thread, and not just the snippets Doug 
pulled, you would see a strong community and commitment to both 
increasing diversity and gaining TLP status. The hurdle we face today 
wrt diversity is that while usage is picking up, it's been historically 
hard for us to gain contributors. This is primarily due to the fact that 
we have a) highly targeted & complex project, and b) relatively mature 
project and user base. We are not a new project, where the majority of 
code is yet to be written. We are working to increase diversity, (again, 
see the thread for some discussion of this) however it's slow going. 
Recently we have seen a big upswing in users (twitter, digg, Solr, 
Cassandra, Neo4j, etc...) due to our efforts at evangelism. The hope is 
that these new users, along with some new areas for development, would 
stimulate contributor growth. If you have any ideas or could provide 
help in this area please feel free to discuss on our dev list.

Patrick

From general-return-1320-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 17:14:01 2010
Return-Path: <general-return-1320-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98689 invoked from network); 8 Apr 2010 17:14:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 17:14:01 -0000
Received: (qmail 20035 invoked by uid 500); 8 Apr 2010 17:14:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19989 invoked by uid 500); 8 Apr 2010 17:14:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19981 invoked by uid 99); 8 Apr 2010 17:14:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 17:14:00 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=AWL,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 17:13:56 +0000
Received: from pugholehand-lm.corp.yahoo.com (pugholehand-lm.corp.yahoo.com [10.72.115.44])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o38HCqFR021775
	for <general@hadoop.apache.org>; Thu, 8 Apr 2010 10:12:52 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=wZGE3hrHNA33/WomZ10R5Fw+vdsZTgQKrR1C/XymJyEkQIhE4c/9khJIELxXojYv
Message-Id: <3F1902B5-BAD3-440D-8D35-6EBC3297B939@yahoo-inc.com>
From: Alan Gates <gates@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <x2l7c962aed1004072102j3ca29379he9f4095be7736f53@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSS] HBase as TLP
Date: Thu, 8 Apr 2010 10:12:52 -0700
References: <x2l7c962aed1004072102j3ca29379he9f4095be7736f53@mail.gmail.com>
X-Mailer: Apple Mail (2.936)

I have a concern.  HDFS, mapreduce, Hbase, Hive, and Pig taken  
together form a coherent software stack.  I suspect that many users  
see this as a whole, in the same way they see a Linux distribution as  
a whole, without remembering that Linux is really the kernel while  
other GNU components are added into the distribution.  HDFS and  
mapreduce form a base on which the other projects depend.  Hbase,  
Hive, and Pig function to extend the base to many more users, both in  
terms of making it easier to use and bringing new functionality.

Thus splitting them up does not make sense to me.  They form a whole,  
why not keep them as a whole?

I know that the response will be that becoming a top level project  
doesn't mean they cannot continue to function as a whole; that this is  
merely a governance issue and not an issue of how projects work  
together (see for example http://bit.ly/9ylAYS).  But I remain  
skeptical.  The structure of governing hierarchies always influence  
cohesion of a group.  It would help me if advocates of this position  
could point to successful, separate Apache TLPs that are either  
completely dependent on another project (as HBase would be on Hadoop)  
or significantly dependent to extend functionality (as Hadoop would be  
on Hbase).

This is not to say that I do not understand the value of having PMCs  
from these growing subprojects report directly to Apache.  But I am  
concerned that we are letting this valid concern overrule other  
equally valid concerns without considering the tradeoffs.

Alan.

On Apr 7, 2010, at 9:02 PM, Stack wrote:

> The HBase subproject has voted to become a TLP: http://su.pr/1g0HAN
>
> Does the Hadoop community have any questions or concerns about this  
> proposal?
>
> Please don't vote yet in response to this.  I'll call a formal vote
> after questions, if any, have been resolved.
>
> Thanks,
> St.Ack
> (Thanks for the boiler plate Doug Cutting)


From general-return-1321-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 17:28:56 2010
Return-Path: <general-return-1321-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4164 invoked from network); 8 Apr 2010 17:28:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 17:28:55 -0000
Received: (qmail 44286 invoked by uid 500); 8 Apr 2010 17:28:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44240 invoked by uid 500); 8 Apr 2010 17:28:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44232 invoked by uid 99); 8 Apr 2010 17:28:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 17:28:54 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 17:28:49 +0000
Received: from pugholehand-lm.corp.yahoo.com (pugholehand-lm.corp.yahoo.com [10.72.115.44])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o38HS0S4028162
	for <general@hadoop.apache.org>; Thu, 8 Apr 2010 10:28:00 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:content-type:
	content-transfer-encoding:mime-version:subject:date:x-mailer;
	b=wvfOdQMvscjKfJNmLaEwSlaPXW92mYwY9JpPAvHeunX3yXzbujMQmPUgjnxEMA/i
Message-Id: <41E8B39D-3CF2-4415-B4D2-9CD37C4FF26F@yahoo-inc.com>
From: Alan Gates <gates@yahoo-inc.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Pig as TLP discussion
Date: Thu, 8 Apr 2010 10:28:00 -0700
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

You can find the discussion of Pig becoming a TLP here:

http://www.mail-archive.com/pig-dev@hadoop.apache.org/msg08589.html

As the discussion was varied I will not attempt to summarize it other  
than to note that the majority of voters felt that Pig should not  
become a TLP at this time.

Alan.

From general-return-1322-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 17:51:23 2010
Return-Path: <general-return-1322-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14668 invoked from network); 8 Apr 2010 17:51:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 17:51:23 -0000
Received: (qmail 96489 invoked by uid 500); 8 Apr 2010 17:51:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96453 invoked by uid 500); 8 Apr 2010 17:51:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96445 invoked by uid 99); 8 Apr 2010 17:51:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 17:51:22 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 08 Apr 2010 17:51:19 +0000
Received: (qmail 14590 invoked by uid 99); 8 Apr 2010 17:50:57 -0000
Received: from localhost.apache.org (HELO [192.168.168.103]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 17:50:57 +0000
Message-ID: <4BBE1780.1080102@apache.org>
Date: Thu, 08 Apr 2010 10:50:56 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100317)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSS] HBase as TLP
References: <x2l7c962aed1004072102j3ca29379he9f4095be7736f53@mail.gmail.com> <3F1902B5-BAD3-440D-8D35-6EBC3297B939@yahoo-inc.com>
In-Reply-To: <3F1902B5-BAD3-440D-8D35-6EBC3297B939@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Alan Gates wrote:
> It would help me if advocates of this position could point to 
> successful, separate Apache TLPs that are either completely dependent on 
> another project (as HBase would be on Hadoop) or significantly dependent 
> to extend functionality (as Hadoop would be on Hbase).

HTTPD requires APR.

Jackrabbit, Roller and probably others require Lucene.

Nutch requires Hadoop.

Continuum and others require Derby.

Doug

From general-return-1323-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 22:06:30 2010
Return-Path: <general-return-1323-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82401 invoked from network); 8 Apr 2010 22:06:29 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 22:06:29 -0000
Received: (qmail 92500 invoked by uid 500); 8 Apr 2010 22:06:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92457 invoked by uid 500); 8 Apr 2010 22:06:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 92449 invoked by uid 99); 8 Apr 2010 22:06:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 22:06:28 +0000
X-ASF-Spam-Status: No, hits=-61.7 required=10.0
	tests=AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 08 Apr 2010 22:06:27 +0000
Received: (qmail 82239 invoked by uid 99); 8 Apr 2010 22:06:07 -0000
Received: from localhost.apache.org (HELO [192.168.168.103]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 22:06:06 +0000
Message-ID: <4BBE534E.4030805@apache.org>
Date: Thu, 08 Apr 2010 15:06:06 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100317)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Pig as TLP discussion
References: <41E8B39D-3CF2-4415-B4D2-9CD37C4FF26F@yahoo-inc.com>
In-Reply-To: <41E8B39D-3CF2-4415-B4D2-9CD37C4FF26F@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Alan Gates wrote:
> As the discussion was varied I will not attempt to summarize it other 
> than to note that the majority of voters felt that Pig should not become 
> a TLP at this time.

Fundamentally the question should not be, "do you want to become a TLP" 
but rather "why can't you become a TLP"?  Compelling answers might be:

- "Because we're actually not a distinct community.  Most of our 
committers are also active in other parts of Hadoop.  We're a separate 
code tree so we can release on separate cycles."

or perhaps

- "Because most of our code is still coming in batched commits from a 
single company.  All the real development is happening in their private 
repository, not in public with community involvement."

The former would indicate that you should perhaps remain in Hadoop 
indefinitely.  The latter would indicate that the Hadoop PMC should help 
your developers understand the importance of community development at 
Apache, or, failing that, encourage your code to leave Apache for 
Github, Google Code or Sourceforge, which has no such requirements. 
Apache's mission is to support, open collaborative development. 
Organizationally, each independent developer community, once it's 
operating, should ideally report directly to the board.

I saw no such compelling arguments in the discussion you cited.  Did I 
miss them?

Alternately, the Hadoop PMC could decide it wishes to continue to flout 
the advice of the board against umbrella projects.  If we did this, we 
might consider developing an internal, regularly scheduled, subproject 
review process.  We could present this process to the board as part of 
our rationale for keeping several subprojects.

Doug


From general-return-1324-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 00:15:38 2010
Return-Path: <general-return-1324-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 2016 invoked from network); 9 Apr 2010 00:15:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 00:15:38 -0000
Received: (qmail 93341 invoked by uid 500); 9 Apr 2010 00:15:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93290 invoked by uid 500); 9 Apr 2010 00:15:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93282 invoked by uid 99); 9 Apr 2010 00:15:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 00:15:36 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 209.85.221.193 as permitted sender)
Received: from [209.85.221.193] (HELO mail-qy0-f193.google.com) (209.85.221.193)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 00:15:30 +0000
Received: by qyk31 with SMTP id 31so4364251qyk.20
        for <general@hadoop.apache.org>; Thu, 08 Apr 2010 17:15:09 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:received:message-id:subject
         :from:to:content-type:content-transfer-encoding;
        bh=L9vmAMBhUCUT/mRBpMozdR3PSLCQqdiDUAk1+fvTbgg=;
        b=BGvyVSbJ6IOyhSIfalPTPviY4T3k4CtLLLxWRYwsga8uZpFfFz0+M/ovfuLkiqtl3e
         s7Kw6b634oCohFRVQfvvthPhtpgUxNnkf4gNyYEMbFgjz4gtgGSxTGsDEZiOWsqSUmF+
         Mpr0xZl5oOIB69+z3wCkSgKb/3je91o/8YsEk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=wwMn/7ct49H6KhfKwkr3qmW53ArI2y86AVJmK4h3CuwAaMm2agi6AqVA5Aw1SeWZ4L
         QvlN+u4UYzKtByir/erchdgeHcWQJ3Tnx4aE7yN2hLekWYCau3A02WPBcjJ6BS3CBvX8
         9rktaM3PeYC8OIDX/rItf0L55+4Xr0X6y5Taw=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
Received: by 10.229.10.133 with HTTP; Thu, 8 Apr 2010 17:15:09 -0700 (PDT)
In-Reply-To: <3F1902B5-BAD3-440D-8D35-6EBC3297B939@yahoo-inc.com>
References: <x2l7c962aed1004072102j3ca29379he9f4095be7736f53@mail.gmail.com>
	 <3F1902B5-BAD3-440D-8D35-6EBC3297B939@yahoo-inc.com>
Date: Thu, 8 Apr 2010 17:15:09 -0700
X-Google-Sender-Auth: 596469d0a421a024
Received: by 10.229.99.143 with SMTP id u15mr1230822qcn.105.1270772109290; 
	Thu, 08 Apr 2010 17:15:09 -0700 (PDT)
Message-ID: <n2j7c962aed1004081715q7ae4a675k7c855618cc8b0996@mail.gmail.com>
Subject: Re: [DISCUSS] HBase as TLP
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Thu, Apr 8, 2010 at 10:12 AM, Alan Gates <gates@yahoo-inc.com> wrote:
> I have a concern. =A0HDFS, mapreduce, Hbase, Hive, and Pig taken together=
 form
> a coherent software stack.

... if it had been designed by Dr. Frankenstein (half-joke!).

I think it a stretch describing the Hadoop suite 'coherent'.  Pieces
in the above stack can be swapped out and other components that do
similar function put in their place.  Rare is the install that uses
all of the elements above (and the above is not a complete list of all
Hadoop subprojects).  Some of the components have inter-dependencies
but others have no dependency on other subprojects at all and can be
run w/o reference to other pieces of the Hadoop set.  For many, Hadoop
looks more like a 'bucket' of software than it is a 'coherent'
software stack.

I'm trying to undo the premise upon which the rest of your mail depends.
St.Ack

From general-return-1325-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 01:42:08 2010
Return-Path: <general-return-1325-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21441 invoked from network); 9 Apr 2010 01:42:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 01:42:07 -0000
Received: (qmail 65922 invoked by uid 500); 9 Apr 2010 01:42:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65872 invoked by uid 500); 9 Apr 2010 01:42:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65864 invoked by uid 99); 9 Apr 2010 01:42:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 01:42:06 +0000
X-ASF-Spam-Status: No, hits=1.3 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jaybooth@gmail.com designates 209.85.211.174 as permitted sender)
Received: from [209.85.211.174] (HELO mail-yw0-f174.google.com) (209.85.211.174)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 01:42:02 +0000
Received: by ywh4 with SMTP id 4so1425136ywh.5
        for <general@hadoop.apache.org>; Thu, 08 Apr 2010 18:41:41 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type;
        bh=hmidAE7DwD06PGoMOrnXACMpvwLspGScCTmL5YS0y+M=;
        b=R58gzJG2mO0g30xCp8ej06bKsmDI1Ud8BLQhHtznL5qdYfLtxFP+TiMkBxOyU0jIsD
         pb+Mk9w7EaHWHMCEGgZOuyCgV7D6mdbg183rk7rmiqXeewAqLWB04FWl6FXjElBIGAsc
         hnvBydQ0bXI/wpFGAJSASq9iRqd9jhelw2CLE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=JfkKRLkYr7ANVNvXcn+HEhLgQ422Plp+p9mnpmZcqaPq+xlG97OE/6UNjN0KVx6eSM
         UQNVdybN1er6luXM5KzZAEJIEFHR1r1DRom/W0Nz+9ZfXjD7Sz2t1vSBvbjZT41WLI0M
         gzeMQaImnyn33mOkkHiPwrSdL1nTQndjYBsWw=
MIME-Version: 1.0
Received: by 10.100.201.9 with HTTP; Thu, 8 Apr 2010 18:41:41 -0700 (PDT)
In-Reply-To: <n2j7c962aed1004081715q7ae4a675k7c855618cc8b0996@mail.gmail.com>
References: <x2l7c962aed1004072102j3ca29379he9f4095be7736f53@mail.gmail.com>
	 <3F1902B5-BAD3-440D-8D35-6EBC3297B939@yahoo-inc.com>
	 <n2j7c962aed1004081715q7ae4a675k7c855618cc8b0996@mail.gmail.com>
Date: Thu, 8 Apr 2010 21:41:41 -0400
Received: by 10.100.236.21 with SMTP id j21mr1590676anh.67.1270777301055; Thu, 
	08 Apr 2010 18:41:41 -0700 (PDT)
Message-ID: <p2m54dc3c51004081841y4d86190ey126dbff961df024@mail.gmail.com>
Subject: Re: [DISCUSS] HBase as TLP
From: Jay Booth <jaybooth@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636b2b5ea9e3ad50483c3e5ba

--001636b2b5ea9e3ad50483c3e5ba
Content-Type: text/plain; charset=ISO-8859-1

What if the projects were:

A)  split out to TLPs because they do seem to have reached that level of
individual community

but,

B)  The projects could somehow jointly put out an integrated build
containing the above projects and let users run whatever they want out of
it?

That would require a lot of coordination but would make a heck of a 1.0
release, if there's a reference build then it'll make it easier to
identify/fix cross-component bugs and the release process might surface more
of them early.  It seems like that could help with a number of St.Ack's
concerns about reanimated monsters.  Maybe it could be owned by the Common
project, with participation from the project leads?  Just an idea.

-Jay

On Thu, Apr 8, 2010 at 8:15 PM, Stack <stack@duboce.net> wrote:

> On Thu, Apr 8, 2010 at 10:12 AM, Alan Gates <gates@yahoo-inc.com> wrote:
> > I have a concern.  HDFS, mapreduce, Hbase, Hive, and Pig taken together
> form
> > a coherent software stack.
>
> ... if it had been designed by Dr. Frankenstein (half-joke!).
>
> I think it a stretch describing the Hadoop suite 'coherent'.  Pieces
> in the above stack can be swapped out and other components that do
> similar function put in their place.  Rare is the install that uses
> all of the elements above (and the above is not a complete list of all
> Hadoop subprojects).  Some of the components have inter-dependencies
> but others have no dependency on other subprojects at all and can be
> run w/o reference to other pieces of the Hadoop set.  For many, Hadoop
> looks more like a 'bucket' of software than it is a 'coherent'
> software stack.
>
> I'm trying to undo the premise upon which the rest of your mail depends.
> St.Ack
>

--001636b2b5ea9e3ad50483c3e5ba--

From general-return-1326-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 02:03:49 2010
Return-Path: <general-return-1326-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25967 invoked from network); 9 Apr 2010 02:03:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 02:03:49 -0000
Received: (qmail 80425 invoked by uid 500); 9 Apr 2010 02:03:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80360 invoked by uid 500); 9 Apr 2010 02:03:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80351 invoked by uid 99); 9 Apr 2010 02:03:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 02:03:48 +0000
X-ASF-Spam-Status: No, hits=-0.5 required=10.0
	tests=AWL,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 02:03:41 +0000
Received: from wlan-guest-164.corp.yahoo.com ([10.72.76.118])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o39221bQ097071
	for <general@hadoop.apache.org>; Thu, 8 Apr 2010 19:02:01 -0700 (PDT)
Message-Id: <C9980F2D-E394-426B-8282-3E1095784561@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <p2m54dc3c51004081841y4d86190ey126dbff961df024@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSS] HBase as TLP
Date: Thu, 8 Apr 2010 19:02:01 -0700
References: <x2l7c962aed1004072102j3ca29379he9f4095be7736f53@mail.gmail.com> <3F1902B5-BAD3-440D-8D35-6EBC3297B939@yahoo-inc.com> <n2j7c962aed1004081715q7ae4a675k7c855618cc8b0996@mail.gmail.com> <p2m54dc3c51004081841y4d86190ey126dbff961df024@mail.gmail.com>
X-Mailer: Apple Mail (2.936)


On Apr 8, 2010, at 6:41 PM, Jay Booth wrote:

> What if the projects were:
>
> A)  split out to TLPs because they do seem to have reached that  
> level of
> individual community
>
> but,
>
> B)  The projects could somehow jointly put out an integrated build
> containing the above projects and let users run whatever they want  
> out of
> it?
>
> That would require a lot of coordination but would make a heck of a  
> 1.0
> release,


1.0 release of what?

Arun

From general-return-1327-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 03:40:50 2010
Return-Path: <general-return-1327-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42187 invoked from network); 9 Apr 2010 03:40:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 03:40:50 -0000
Received: (qmail 39831 invoked by uid 500); 9 Apr 2010 03:40:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39794 invoked by uid 500); 9 Apr 2010 03:40:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39786 invoked by uid 99); 9 Apr 2010 03:40:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 03:40:48 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jaybooth@gmail.com designates 74.125.83.48 as permitted sender)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 03:40:42 +0000
Received: by gwaa12 with SMTP id a12so1510913gwa.35
        for <general@hadoop.apache.org>; Thu, 08 Apr 2010 20:40:21 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type;
        bh=TCvgiOjL25YQgpJAGCxEFU3ZACzk4QJPOadXazZotIo=;
        b=mxqWJLcI6yH4Y1+0LuNHYApm/1ysE9FO6J9No+jxbjcT2AUb+VHl2bVi4pCqfggYCw
         kzk1ihSAjf9SJ07guHAdFAB5HkzUcCEcRWGftSHJc22dQ+fucNU6G6JbBPGJZfWcifSJ
         AKWbpJ/QQlwgogyaKLN++spJLv23Di4Eq48WY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=ZWDf3euX42oDb2arE90HkQIHwBWwN30UWNvaEql5JV5+cBrHZzGU2+BXCxkUiaKcsX
         9m/h39KdRNLOX2lZS3FGO2dAYid/cLq0EbNvRrwijStDoHXR5rt5HP94kraL939ZPaJs
         ztOuEfUGvbmWGNlCZ/dkEE6UfNfHcm33JWcL0=
MIME-Version: 1.0
Received: by 10.100.201.9 with HTTP; Thu, 8 Apr 2010 20:40:21 -0700 (PDT)
In-Reply-To: <C9980F2D-E394-426B-8282-3E1095784561@yahoo-inc.com>
References: <x2l7c962aed1004072102j3ca29379he9f4095be7736f53@mail.gmail.com>
	 <3F1902B5-BAD3-440D-8D35-6EBC3297B939@yahoo-inc.com>
	 <n2j7c962aed1004081715q7ae4a675k7c855618cc8b0996@mail.gmail.com>
	 <p2m54dc3c51004081841y4d86190ey126dbff961df024@mail.gmail.com>
	 <C9980F2D-E394-426B-8282-3E1095784561@yahoo-inc.com>
Date: Thu, 8 Apr 2010 23:40:21 -0400
Received: by 10.101.137.13 with SMTP id p13mr1542325ann.199.1270784421238; 
	Thu, 08 Apr 2010 20:40:21 -0700 (PDT)
Message-ID: <r2g54dc3c51004082040p9807d80fn5eba502289afc67b@mail.gmail.com>
Subject: Re: [DISCUSS] HBase as TLP
From: Jay Booth <jaybooth@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e68ee28c0399d10483c58eea
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e68ee28c0399d10483c58eea
Content-Type: text/plain; charset=ISO-8859-1

Not sure exactly what I meant by "1.0 of what", "Hadoop" I guess, I was
trying to address the concerns raised, which I share -- Alan's concern is
that if the projects are completely separate from each other, that might
decrease visibility as to the demands they're placing on each other when
integrated, and St.Ack mentioned the frankenstein factor which I think we've
all felt some pain from, and which may get worse after the project split.
What's the standard way to deploy the three, even?  Is there one?

If the PMCs jointly maintained some sort of 'stable integrated build' which
took in new releases from the TLPs as they were released after a soak
period, it could provide a common touchstone that bugs could be tested
against and cross-component patches delivered against, potentially
increasing visibility of cross-component issues while providing a less
cobbled-together system to administrate.  On the other side, though, if
executed wrong, you'd be creating a committee of committees and possibly
undoing some of the benefits of going TLP in the first place, especially if
politics heat up over what goes into the 'standard' build.  I think it could
be viable though.



On Thu, Apr 8, 2010 at 10:02 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:

>
> On Apr 8, 2010, at 6:41 PM, Jay Booth wrote:
>
>  What if the projects were:
>>
>> A)  split out to TLPs because they do seem to have reached that level of
>> individual community
>>
>> but,
>>
>> B)  The projects could somehow jointly put out an integrated build
>> containing the above projects and let users run whatever they want out of
>> it?
>>
>> That would require a lot of coordination but would make a heck of a 1.0
>> release,
>>
>
>
> 1.0 release of what?
>
> Arun
>

--0016e68ee28c0399d10483c58eea--

From general-return-1328-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 04:01:26 2010
Return-Path: <general-return-1328-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44770 invoked from network); 9 Apr 2010 04:01:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 04:01:26 -0000
Received: (qmail 52742 invoked by uid 500); 9 Apr 2010 04:01:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52594 invoked by uid 500); 9 Apr 2010 04:01:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52586 invoked by uid 99); 9 Apr 2010 04:01:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 04:01:24 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.218.217] (HELO mail-bw0-f217.google.com) (209.85.218.217)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 04:01:18 +0000
Received: by bwz9 with SMTP id 9so2336112bwz.29
        for <general@hadoop.apache.org>; Thu, 08 Apr 2010 21:00:58 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.56.133 with HTTP; Thu, 8 Apr 2010 21:00:57 -0700 (PDT)
In-Reply-To: <r2g54dc3c51004082040p9807d80fn5eba502289afc67b@mail.gmail.com>
References: <x2l7c962aed1004072102j3ca29379he9f4095be7736f53@mail.gmail.com>
	 <3F1902B5-BAD3-440D-8D35-6EBC3297B939@yahoo-inc.com>
	 <n2j7c962aed1004081715q7ae4a675k7c855618cc8b0996@mail.gmail.com>
	 <p2m54dc3c51004081841y4d86190ey126dbff961df024@mail.gmail.com>
	 <C9980F2D-E394-426B-8282-3E1095784561@yahoo-inc.com>
	 <r2g54dc3c51004082040p9807d80fn5eba502289afc67b@mail.gmail.com>
Date: Thu, 8 Apr 2010 21:00:57 -0700
Received: by 10.204.19.137 with SMTP id a9mr1217569bkb.24.1270785657949; Thu, 
	08 Apr 2010 21:00:57 -0700 (PDT)
Message-ID: <i2zac79ea401004082100hf5da28f3vab8881fc102dc0bc@mail.gmail.com>
Subject: Re: [DISCUSS] HBase as TLP
From: Tom White <tom@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Eclipse does big bang releases of multiple components, but I believe
it requires a huge amount of coordination and planning. Instead, I
think the direction Hadoop should move in is to stabilize and clearly
demarcate its core filesystem and MapReduce interfaces, so that
projects like HBase, Pig, and Hive can run against multiple versions
of core. Their release cycles are already largely decoupled from core,
so the question about whether they become TLPs is more to do with
project governance than with release coordination.

Cheers,
Tom

On Thu, Apr 8, 2010 at 8:40 PM, Jay Booth <jaybooth@gmail.com> wrote:
> Not sure exactly what I meant by "1.0 of what", "Hadoop" I guess, I was
> trying to address the concerns raised, which I share -- Alan's concern is
> that if the projects are completely separate from each other, that might
> decrease visibility as to the demands they're placing on each other when
> integrated, and St.Ack mentioned the frankenstein factor which I think we=
've
> all felt some pain from, and which may get worse after the project split.
> What's the standard way to deploy the three, even? =A0Is there one?
>
> If the PMCs jointly maintained some sort of 'stable integrated build' whi=
ch
> took in new releases from the TLPs as they were released after a soak
> period, it could provide a common touchstone that bugs could be tested
> against and cross-component patches delivered against, potentially
> increasing visibility of cross-component issues while providing a less
> cobbled-together system to administrate. =A0On the other side, though, if
> executed wrong, you'd be creating a committee of committees and possibly
> undoing some of the benefits of going TLP in the first place, especially =
if
> politics heat up over what goes into the 'standard' build. =A0I think it =
could
> be viable though.
>
>
>
> On Thu, Apr 8, 2010 at 10:02 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:
>
>>
>> On Apr 8, 2010, at 6:41 PM, Jay Booth wrote:
>>
>> =A0What if the projects were:
>>>
>>> A) =A0split out to TLPs because they do seem to have reached that level=
 of
>>> individual community
>>>
>>> but,
>>>
>>> B) =A0The projects could somehow jointly put out an integrated build
>>> containing the above projects and let users run whatever they want out =
of
>>> it?
>>>
>>> That would require a lot of coordination but would make a heck of a 1.0
>>> release,
>>>
>>
>>
>> 1.0 release of what?
>>
>> Arun
>>
>

From general-return-1329-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 04:07:40 2010
Return-Path: <general-return-1329-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45403 invoked from network); 9 Apr 2010 04:07:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 04:07:40 -0000
Received: (qmail 55081 invoked by uid 500); 9 Apr 2010 04:07:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55046 invoked by uid 500); 9 Apr 2010 04:07:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55038 invoked by uid 99); 9 Apr 2010 04:07:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 04:07:38 +0000
X-ASF-Spam-Status: No, hits=-1.4 required=10.0
	tests=AWL,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of imyousuf@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 04:07:34 +0000
Received: by vws9 with SMTP id 9so1461597vws.35
        for <general@hadoop.apache.org>; Thu, 08 Apr 2010 21:07:13 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=JRM4Cu3nxqlKDdI4xkbJJMbI4JOKEyxxPHPmqcDdV+o=;
        b=vcInwbAtKp6V0Pq4ulojyZFqgovHTgEPOb1e/V78mUqqVmYevjn6s/9AUEST14FMAa
         Pr4Ag2Hy7Xxky7WvF84N0T8do2MLx4kpBA83CI8DqBjHiEehutYW7FUteFXLz3gg+8LS
         3hTErP6d8uHezUaTE4J83TkJ3Eqh7iysRTdrg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=S7hlJIvukWkp/UVoqdSHnKnroOF9U/2k+voVDL9oQjhKt6Y/biFR6+Nu/Q9gGAWAt3
         BqoaOumDkS7+mlWs2cJKhPYalNJUWdcub+yX1Ih0OTT4aBkfMFWFdCeHUQoTbZ1ZPJt/
         MyX+dwOE8yeteQwM+F0RzjNhQhy3jTKrwvp7c=
MIME-Version: 1.0
Received: by 10.220.100.210 with HTTP; Thu, 8 Apr 2010 21:07:13 -0700 (PDT)
In-Reply-To: <i2zac79ea401004082100hf5da28f3vab8881fc102dc0bc@mail.gmail.com>
References: <x2l7c962aed1004072102j3ca29379he9f4095be7736f53@mail.gmail.com>
	 <3F1902B5-BAD3-440D-8D35-6EBC3297B939@yahoo-inc.com>
	 <n2j7c962aed1004081715q7ae4a675k7c855618cc8b0996@mail.gmail.com>
	 <p2m54dc3c51004081841y4d86190ey126dbff961df024@mail.gmail.com>
	 <C9980F2D-E394-426B-8282-3E1095784561@yahoo-inc.com>
	 <r2g54dc3c51004082040p9807d80fn5eba502289afc67b@mail.gmail.com>
	 <i2zac79ea401004082100hf5da28f3vab8881fc102dc0bc@mail.gmail.com>
Date: Fri, 9 Apr 2010 09:52:13 +0545
Received: by 10.220.62.204 with SMTP id y12mr741241vch.186.1270786033117; Thu, 
	08 Apr 2010 21:07:13 -0700 (PDT)
Message-ID: <p2n7bfdc29a1004082107j24f21354rb9b351b7ee6c587e@mail.gmail.com>
Subject: Re: [DISCUSS] HBase as TLP
From: Imran M Yousuf <imyousuf@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

+1

I feel the same. From following HBase seeing its releases depending
directly on Hadoop release gets me thinking...

Best regards,

Imran

On Fri, Apr 9, 2010 at 9:45 AM, Tom White <tom@cloudera.com> wrote:
> Eclipse does big bang releases of multiple components, but I believe
> it requires a huge amount of coordination and planning. Instead, I
> think the direction Hadoop should move in is to stabilize and clearly
> demarcate its core filesystem and MapReduce interfaces, so that
> projects like HBase, Pig, and Hive can run against multiple versions
> of core. Their release cycles are already largely decoupled from core,
> so the question about whether they become TLPs is more to do with
> project governance than with release coordination.
>
> Cheers,
> Tom
>
> On Thu, Apr 8, 2010 at 8:40 PM, Jay Booth <jaybooth@gmail.com> wrote:
>> Not sure exactly what I meant by "1.0 of what", "Hadoop" I guess, I was
>> trying to address the concerns raised, which I share -- Alan's concern i=
s
>> that if the projects are completely separate from each other, that might
>> decrease visibility as to the demands they're placing on each other when
>> integrated, and St.Ack mentioned the frankenstein factor which I think w=
e've
>> all felt some pain from, and which may get worse after the project split=
.
>> What's the standard way to deploy the three, even? =A0Is there one?
>>
>> If the PMCs jointly maintained some sort of 'stable integrated build' wh=
ich
>> took in new releases from the TLPs as they were released after a soak
>> period, it could provide a common touchstone that bugs could be tested
>> against and cross-component patches delivered against, potentially
>> increasing visibility of cross-component issues while providing a less
>> cobbled-together system to administrate. =A0On the other side, though, i=
f
>> executed wrong, you'd be creating a committee of committees and possibly
>> undoing some of the benefits of going TLP in the first place, especially=
 if
>> politics heat up over what goes into the 'standard' build. =A0I think it=
 could
>> be viable though.
>>
>>
>>
>> On Thu, Apr 8, 2010 at 10:02 PM, Arun C Murthy <acm@yahoo-inc.com> wrote=
:
>>
>>>
>>> On Apr 8, 2010, at 6:41 PM, Jay Booth wrote:
>>>
>>> =A0What if the projects were:
>>>>
>>>> A) =A0split out to TLPs because they do seem to have reached that leve=
l of
>>>> individual community
>>>>
>>>> but,
>>>>
>>>> B) =A0The projects could somehow jointly put out an integrated build
>>>> containing the above projects and let users run whatever they want out=
 of
>>>> it?
>>>>
>>>> That would require a lot of coordination but would make a heck of a 1.=
0
>>>> release,
>>>>
>>>
>>>
>>> 1.0 release of what?
>>>
>>> Arun
>>>
>>
>



--=20
Imran M Yousuf
Entrepreneur & Software Engineer
Smart IT Engineering
Dhaka, Bangladesh
Email: imran@smartitengineering.com
Blog: http://imyousuf-tech.blogs.smartitengineering.com/
Mobile: +880-1711402557

From general-return-1330-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 06:10:22 2010
Return-Path: <general-return-1330-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59910 invoked from network); 9 Apr 2010 06:10:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 06:10:22 -0000
Received: (qmail 28137 invoked by uid 500); 9 Apr 2010 06:10:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28032 invoked by uid 500); 9 Apr 2010 06:10:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28024 invoked by uid 99); 9 Apr 2010 06:10:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 06:10:21 +0000
X-ASF-Spam-Status: No, hits=1.4 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jaybooth@gmail.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 06:10:16 +0000
Received: by gyf1 with SMTP id 1so1549675gyf.35
        for <general@hadoop.apache.org>; Thu, 08 Apr 2010 23:09:55 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type;
        bh=DicNeCDD552o3UUGdrTWUeQXdnBJSck/KI0f/J2v7Vs=;
        b=vTDpa1G3jUqM3nZ5BjTTzQOR0uYPsvijPkltG/j6shyczS1j3KcinvmR4/9W6X7iSy
         Gf4WNHXbtk1h5/3l7d/Ec+O2R1gDPJfB7oTCgHAoIXR5+tFE9e8oBmfUvH/KUqleRZbw
         VGs4ljKNDNgBA9E+nJ+ZC/MO0sVDoqM/CAano=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=J93Ze785EUMOO0jl2iz7ytNDM5yhJgyCFsTXPbANmks3XlxkTcTwmQq1a4q7D+kFxW
         S1Y3Jc5uVWiZ4nSIYNEJIFH8B1v9EFYSyEUfrKe0p2ARcxAAYjCw6tHVf6qTnrsUsCgN
         izyPTgZiqVRW61x7bQiGsqNV4x2OwywbpF+QE=
MIME-Version: 1.0
Received: by 10.100.201.9 with HTTP; Thu, 8 Apr 2010 23:09:54 -0700 (PDT)
In-Reply-To: <p2n7bfdc29a1004082107j24f21354rb9b351b7ee6c587e@mail.gmail.com>
References: <x2l7c962aed1004072102j3ca29379he9f4095be7736f53@mail.gmail.com>
	 <3F1902B5-BAD3-440D-8D35-6EBC3297B939@yahoo-inc.com>
	 <n2j7c962aed1004081715q7ae4a675k7c855618cc8b0996@mail.gmail.com>
	 <p2m54dc3c51004081841y4d86190ey126dbff961df024@mail.gmail.com>
	 <C9980F2D-E394-426B-8282-3E1095784561@yahoo-inc.com>
	 <r2g54dc3c51004082040p9807d80fn5eba502289afc67b@mail.gmail.com>
	 <i2zac79ea401004082100hf5da28f3vab8881fc102dc0bc@mail.gmail.com>
	 <p2n7bfdc29a1004082107j24f21354rb9b351b7ee6c587e@mail.gmail.com>
Date: Fri, 9 Apr 2010 02:09:54 -0400
Received: by 10.101.136.33 with SMTP id o33mr1894029ann.63.1270793394269; Thu, 
	08 Apr 2010 23:09:54 -0700 (PDT)
Message-ID: <x2s54dc3c51004082309nfd39c80bk2da33ceca0d7cf93@mail.gmail.com>
Subject: Re: [DISCUSS] HBase as TLP
From: Jay Booth <jaybooth@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636d3496ad942aa0483c7a442

--001636d3496ad942aa0483c7a442
Content-Type: text/plain; charset=ISO-8859-1

Alright, I totally agree.  Thanks for putting it that way.

-Jay

On Fri, Apr 9, 2010 at 12:07 AM, Imran M Yousuf <imyousuf@gmail.com> wrote:

> +1
>
> I feel the same. From following HBase seeing its releases depending
> directly on Hadoop release gets me thinking...
>
> Best regards,
>
> Imran
>
> On Fri, Apr 9, 2010 at 9:45 AM, Tom White <tom@cloudera.com> wrote:
> > Eclipse does big bang releases of multiple components, but I believe
> > it requires a huge amount of coordination and planning. Instead, I
> > think the direction Hadoop should move in is to stabilize and clearly
> > demarcate its core filesystem and MapReduce interfaces, so that
> > projects like HBase, Pig, and Hive can run against multiple versions
> > of core. Their release cycles are already largely decoupled from core,
> > so the question about whether they become TLPs is more to do with
> > project governance than with release coordination.
> >
> > Cheers,
> > Tom
> >
> > On Thu, Apr 8, 2010 at 8:40 PM, Jay Booth <jaybooth@gmail.com> wrote:
> >> Not sure exactly what I meant by "1.0 of what", "Hadoop" I guess, I was
> >> trying to address the concerns raised, which I share -- Alan's concern
> is
> >> that if the projects are completely separate from each other, that might
> >> decrease visibility as to the demands they're placing on each other when
> >> integrated, and St.Ack mentioned the frankenstein factor which I think
> we've
> >> all felt some pain from, and which may get worse after the project
> split.
> >> What's the standard way to deploy the three, even?  Is there one?
> >>
> >> If the PMCs jointly maintained some sort of 'stable integrated build'
> which
> >> took in new releases from the TLPs as they were released after a soak
> >> period, it could provide a common touchstone that bugs could be tested
> >> against and cross-component patches delivered against, potentially
> >> increasing visibility of cross-component issues while providing a less
> >> cobbled-together system to administrate.  On the other side, though, if
> >> executed wrong, you'd be creating a committee of committees and possibly
> >> undoing some of the benefits of going TLP in the first place, especially
> if
> >> politics heat up over what goes into the 'standard' build.  I think it
> could
> >> be viable though.
> >>
> >>
> >>
> >> On Thu, Apr 8, 2010 at 10:02 PM, Arun C Murthy <acm@yahoo-inc.com>
> wrote:
> >>
> >>>
> >>> On Apr 8, 2010, at 6:41 PM, Jay Booth wrote:
> >>>
> >>>  What if the projects were:
> >>>>
> >>>> A)  split out to TLPs because they do seem to have reached that level
> of
> >>>> individual community
> >>>>
> >>>> but,
> >>>>
> >>>> B)  The projects could somehow jointly put out an integrated build
> >>>> containing the above projects and let users run whatever they want out
> of
> >>>> it?
> >>>>
> >>>> That would require a lot of coordination but would make a heck of a
> 1.0
> >>>> release,
> >>>>
> >>>
> >>>
> >>> 1.0 release of what?
> >>>
> >>> Arun
> >>>
> >>
> >
>
>
>
> --
> Imran M Yousuf
> Entrepreneur & Software Engineer
> Smart IT Engineering
> Dhaka, Bangladesh
> Email: imran@smartitengineering.com
> Blog: http://imyousuf-tech.blogs.smartitengineering.com/
> Mobile: +880-1711402557
>

--001636d3496ad942aa0483c7a442--

From general-return-1331-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 13:34:56 2010
Return-Path: <general-return-1331-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49444 invoked from network); 9 Apr 2010 13:34:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 13:34:56 -0000
Received: (qmail 93995 invoked by uid 500); 9 Apr 2010 13:34:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93910 invoked by uid 500); 9 Apr 2010 13:34:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93902 invoked by uid 99); 9 Apr 2010 13:34:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 13:34:54 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,MIME_QP_LONG_LINE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [96.6.114.79] (HELO prod-mail-xrelay04.akamai.com) (96.6.114.79)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 13:34:48 +0000
Received: from prod-mail-xrelay04.akamai.com (localhost [127.0.0.1])
	by postfix.imss70 (Postfix) with ESMTP id C086B95353
	for <general@hadoop.apache.org>; Fri,  9 Apr 2010 13:34:26 +0000 (GMT)
Received: from prod-mail-relay03.akamai.com (unknown [172.27.8.26])
	by prod-mail-xrelay04.akamai.com (Postfix) with ESMTP id AD8CD95352
	for <general@hadoop.apache.org>; Fri,  9 Apr 2010 13:34:26 +0000 (GMT)
Received: from USMA1VM-FEGATE3.kendall.corp.akamai.com (exchange.kendall.corp.akamai.com [172.17.120.149])
	by prod-mail-relay03.akamai.com (Postfix) with ESMTP id 91D712FD62
	for <general@hadoop.apache.org>; Fri,  9 Apr 2010 13:34:26 +0000 (GMT)
Received: from MAVS2.kendall.corp.akamai.com ([172.17.33.20]) by USMA1VM-FEGATE3.kendall.corp.akamai.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Fri, 9 Apr 2010 09:34:26 -0400
Received: from 172.19.113.103 ([172.19.113.103]) by MAVS2.kendall.corp.akamai.com ([172.17.33.102]) via Exchange Front-End Server exchange.akamai.com ([172.17.120.149]) with Microsoft Exchange Server HTTP-DAV ;
 Fri,  9 Apr 2010 13:34:25 +0000
User-Agent: Microsoft-Entourage/12.24.0.100205
Date: Fri, 09 Apr 2010 09:34:24 -0400
Subject: Different exception handling on corrupt GZip file reading
From: Richard Weber <riweber@akamai.com>
To: Hadoop <general@hadoop.apache.org>
Message-ID: <C7E4A520.D3AA%riweber@akamai.com>
Thread-Topic: Different exception handling on corrupt GZip file reading
Thread-Index: AcrX6V4gX9KGeNhIC0uktMcdChQ3Xw==
Mime-version: 1.0
Content-type: multipart/alternative;
	boundary="B_3353650464_38430317"
X-OriginalArrivalTime: 09 Apr 2010 13:34:26.0053 (UTC) FILETIME=[5F597350:01CAD7E9]

--B_3353650464_38430317
Content-type: text/plain;
	charset="ISO-8859-1"
Content-transfer-encoding: quoted-printable

Maybe this is a =B3dumb=B2 question.  In our situation, we process a ton of log
files all gzipped.  Some of those files may be truncated for a a variety of
reasons resulting in a corrupted gzip file.

Now using the default TextInputFormat and LineRecordReader, Hadoop will
happily churn along until it hits a corrupted file.  Once it hits the file,
it throws exceptions, tries to restart on that file and ultimately fails.  =
I
originally tried using the Skipped Records feature, but these exceptions ar=
e
happening at the IO level, not record level.

My solution has been to just make a new SafeTextInputFormat and
SafeLineRecordReader class.  The only difference between these classes and
the non-safe classes is that it has a try {} block in the nextKeyValue() fn=
=B9
when it does the readLine.  If an exception occurs, then the file is closed
out.

My question really boils down to: Is there a reason this isn=B9t in the Hadoo=
p
libary to start with?  Even if there was a flag to raise the exception, or
just let it keep flowing with bad input data.

It=B9s really more of a gripe that I need to reimplement the above 2 classes
just to have a try catch block, and then to make sure I use these classes
for my input format.

Thanks

--Rick

--B_3353650464_38430317--


From general-return-1332-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 10 14:34:01 2010
Return-Path: <general-return-1332-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27411 invoked from network); 10 Apr 2010 14:34:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Apr 2010 14:34:01 -0000
Received: (qmail 96960 invoked by uid 500); 10 Apr 2010 14:34:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96898 invoked by uid 500); 10 Apr 2010 14:33:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96890 invoked by uid 99); 10 Apr 2010 14:33:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Apr 2010 14:33:59 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of magawake@gmail.com designates 209.85.210.184 as permitted sender)
Received: from [209.85.210.184] (HELO mail-yx0-f184.google.com) (209.85.210.184)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Apr 2010 14:33:51 +0000
Received: by yxe14 with SMTP id 14so2001997yxe.5
        for <general@hadoop.apache.org>; Sat, 10 Apr 2010 07:33:30 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:received:message-id
         :subject:from:to:content-type;
        bh=6pmkQ3LAklZbDZ3C0h8leSY54LhiqSRId0+8skCe0uI=;
        b=l3bbWS2hpkZeOwOA5h8HdaNyYlY/Ni9wy/OahKH8wYV4U61YdlGG0GdiHftvPwF10G
         Lf+pBede+SMoQ6H3VFklZJrvbXJxKyTWqcZTviUH+RqNNq9s7vIIHDtJH6QAPVJbrv3R
         hmi37o9SLuYtapROsl/xYgqeqslzd/0XfzhJ4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=XYGIs5gH4jkRMaxUMkrC9EKCp/CdRUjNrcObcvKFgAVSiNjQm46r+I0O6gTQ0/pW+o
         PL+eiTpx8MLeUnMwO3O3NXeRxqPo/4Bdl67OAevAaoeeEX2B5WdJbak9H6w+3GzcFhZa
         +bwlJEl6h6gr9Nt4dN5WcPLH0wXoiw9QVOt/8=
MIME-Version: 1.0
Received: by 10.231.191.130 with HTTP; Sat, 10 Apr 2010 07:33:30 -0700 (PDT)
Date: Sat, 10 Apr 2010 10:33:30 -0400
Received: by 10.101.21.13 with SMTP id y13mr2631173ani.210.1270910010816; Sat, 
	10 Apr 2010 07:33:30 -0700 (PDT)
Message-ID: <h2y1cbd6f831004100733xf3ab6298u278df45f8d80d3ad@mail.gmail.com>
Subject: rack awareness question
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
X-Virus-Checked: Checked by ClamAV on apache.org

Lets say I had a mapping like this,

192.168.0.2 -> /rackdefault


I would now like to map

192.168.0.2 -> /rack02

What is the procedure to do this? is this possible to do while the
cluster is running?

TIA

From general-return-1333-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 00:31:02 2010
Return-Path: <general-return-1333-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 74568 invoked from network); 12 Apr 2010 00:31:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 00:31:02 -0000
Received: (qmail 45148 invoked by uid 500); 12 Apr 2010 00:31:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45091 invoked by uid 500); 12 Apr 2010 00:31:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45083 invoked by uid 99); 12 Apr 2010 00:31:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 00:31:00 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 00:30:54 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:Received:Received:From:To:Subject:
   Thread-Topic:Thread-Index:Date:Message-ID:References:
   In-Reply-To:Accept-Language:Content-Language:
   X-MS-Has-Attach:X-MS-TNEF-Correlator:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version:
   Return-Path:X-OriginalArrivalTime;
  b=HX/wDFkN5+nYiBevr1kVj4caJC7dcDVuSiBl9xmmZtkhdhLvKFJNczZe
   KE418Vjr0s1UOwPmG3VEW1rNVnjjQeWtQ8SO2FKnEgqBMAJnhb1L/ZUbC
   esewzyz9oslQCra;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1271032254; x=1302568254;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20rack=20awareness=20question|Date:=20Mon
   ,=2012=20Apr=202010=2000:30:36=20+0000|Message-ID:=20<EBF
   EB403-3709-44E2-A930-11A07A753AE5@linkedin.com>|To:=20"<g
   eneral@hadoop.apache.org>"=20<general@hadoop.apache.org>
   |MIME-Version:=201.0|Content-Transfer-Encoding:=20quoted-
   printable|Content-ID:=20<9de96d17-5a70-49e2-9122-1a9b4cc8
   ad30>|In-Reply-To:=20<h2y1cbd6f831004100733xf3ab6298u278d
   f45f8d80d3ad@mail.gmail.com>|References:=20<h2y1cbd6f8310
   04100733xf3ab6298u278df45f8d80d3ad@mail.gmail.com>;
  bh=XrJnLCxaUKhNekSyver+SrHLe3v2nBiXpsCeXUvNGX0=;
  b=OCYgOFmZdo2AdeOBjdwM7+r161edOx5lElsW+JwYR+pmraNVaP3Jo5wV
   Sy8SaG9VIFghnrxjwhmbgCa1LrH056ZSdZE+9OabR5VMOdZxB1WORqo35
   41rE/GdKoVbo+Vw;
X-IronPort-AV: E=Sophos;i="4.52,186,1270450800"; 
   d="scan'208";a="12255059"
Received: from esv4-exctest.linkedin.biz ([172.18.46.60]) by CORP-MAIL.linkedin.biz with Microsoft SMTPSVC(6.0.3790.3959);
	 Sun, 11 Apr 2010 17:30:33 -0700
Received: from ESV4-CAS02.linkedin.biz (172.18.46.142) by
 esv4-exctest.linkedin.biz (172.18.46.60) with Microsoft SMTP Server (TLS) id
 14.0.682.1; Sun, 11 Apr 2010 17:30:32 -0700
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi; Sun, 11 Apr 2010
 17:30:32 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: rack awareness question
Thread-Topic: rack awareness question
Thread-Index: AQHK2Lrd83fFTEyGQkeCRFATVCpFBpIecp8A
Date: Mon, 12 Apr 2010 00:30:36 +0000
Message-ID: <EBFEB403-3709-44E2-A930-11A07A753AE5@linkedin.com>
References: <h2y1cbd6f831004100733xf3ab6298u278df45f8d80d3ad@mail.gmail.com>
In-Reply-To: <h2y1cbd6f831004100733xf3ab6298u278df45f8d80d3ad@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Content-Type: text/plain; charset="us-ascii"
Content-ID: <9de96d17-5a70-49e2-9122-1a9b4cc8ad30>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginalArrivalTime: 12 Apr 2010 00:30:33.0347 (UTC) FILETIME=[5CE94D30:01CAD9D7]
X-Virus-Checked: Checked by ClamAV on apache.org


On Apr 10, 2010, at 7:33 AM, Mag Gam wrote:
> What is the procedure to do this?

Change the topology script, bounce NN and JT.


> is this possible to do while the
> cluster is running?

No.  There is a jira i filed on turning the cache into an actual cache w/TT=
Ls, etc.=

From general-return-1334-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 01:34:51 2010
Return-Path: <general-return-1334-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82777 invoked from network); 12 Apr 2010 01:34:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 01:34:50 -0000
Received: (qmail 79287 invoked by uid 500); 12 Apr 2010 01:34:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79208 invoked by uid 500); 12 Apr 2010 01:34:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 79197 invoked by uid 99); 12 Apr 2010 01:34:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 01:34:49 +0000
X-ASF-Spam-Status: No, hits=0.6 required=10.0
	tests=AWL,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.92.24] (HELO qw-out-2122.google.com) (74.125.92.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 01:34:43 +0000
Received: by qw-out-2122.google.com with SMTP id 8so2229473qwh.35
        for <general@hadoop.apache.org>; Sun, 11 Apr 2010 18:34:22 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.250.138 with HTTP; Sun, 11 Apr 2010 18:28:34 -0700 (PDT)
In-Reply-To: <EBFEB403-3709-44E2-A930-11A07A753AE5@linkedin.com>
References: <h2y1cbd6f831004100733xf3ab6298u278df45f8d80d3ad@mail.gmail.com>
	 <EBFEB403-3709-44E2-A930-11A07A753AE5@linkedin.com>
Date: Sun, 11 Apr 2010 18:28:34 -0700
Received: by 10.229.241.82 with SMTP id ld18mr41092qcb.60.1271035714274; Sun, 
	11 Apr 2010 18:28:34 -0700 (PDT)
Message-ID: <q2pc9118d4c1004111828va42fc159s1db13fbbb10aca9f@mail.gmail.com>
Subject: Re: rack awareness question
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163630eda93f087304840010ec

--00163630eda93f087304840010ec
Content-Type: text/plain; charset=ISO-8859-1

> No.  There is a jira i filed on turning the cache into an actual cache
> w/TTLs, etc.


https://issues.apache.org/jira/browse/HDFS-870, for those who like
hyperlinks.

--00163630eda93f087304840010ec--

From general-return-1335-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 06:05:56 2010
Return-Path: <general-return-1335-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30699 invoked from network); 12 Apr 2010 06:05:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 06:05:56 -0000
Received: (qmail 2613 invoked by uid 500); 12 Apr 2010 06:05:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2342 invoked by uid 500); 12 Apr 2010 06:05:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2326 invoked by uid 99); 12 Apr 2010 06:05:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 06:05:52 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [62.99.145.2] (HELO mx.inode.at) (62.99.145.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 06:05:45 +0000
Received: from [213.47.112.79] (port=3766 helo=localhost)
	by smartmx-02.inode.at with esmtpa (Exim 4.69)
	(envelope-from <rho@devc.at>)
	id 1O1Clv-0000cZ-Ag; Mon, 12 Apr 2010 08:05:23 +0200
Date: Mon, 12 Apr 2010 08:05:22 +0200
From: Robert Barta <rho@devc.at>
To: general@hadoop.apache.org
Subject: Re: rack awareness question
Message-ID: <20100412060522.GA32132@odman.int.devc.at>
Reply-To: rho@devc.at
References: <h2y1cbd6f831004100733xf3ab6298u278df45f8d80d3ad@mail.gmail.com> <EBFEB403-3709-44E2-A930-11A07A753AE5@linkedin.com> <q2pc9118d4c1004111828va42fc159s1db13fbbb10aca9f@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <q2pc9118d4c1004111828va42fc159s1db13fbbb10aca9f@mail.gmail.com>
Sender: Robert Barta <rho@devc.at>
User-Agent: Mutt/1.5.18 (2008-05-17)

On Sun, Apr 11, 2010 at 06:28:34PM -0700, Jeff Hammerbacher wrote:
> > No.  There is a jira i filed on turning the cache into an actual cache
> > w/TTLs, etc.
> 
> 
> https://issues.apache.org/jira/browse/HDFS-870, for those who like
> hyperlinks.

Links are always appreciated as I feed here my semantic network
about MapReduce/Hadoop/... with them:

   http://tmip.devc.at/test/mapreduce/.surface/x3/seadragon.html

So if you have the feeling that your favourite tool is
underrepresented there, feel free to provide me with more material.

\rho

From general-return-1336-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 11:38:02 2010
Return-Path: <general-return-1336-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96810 invoked from network); 12 Apr 2010 11:38:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 11:38:02 -0000
Received: (qmail 90036 invoked by uid 500); 12 Apr 2010 11:38:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89510 invoked by uid 500); 12 Apr 2010 11:37:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89502 invoked by uid 99); 12 Apr 2010 11:37:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 11:37:59 +0000
X-ASF-Spam-Status: No, hits=-2.2 required=10.0
	tests=AWL,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of magawake@gmail.com designates 209.85.223.198 as permitted sender)
Received: from [209.85.223.198] (HELO mail-iw0-f198.google.com) (209.85.223.198)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 11:37:54 +0000
Received: by iwn36 with SMTP id 36so4494855iwn.29
        for <general@hadoop.apache.org>; Mon, 12 Apr 2010 04:37:33 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=WV7HpgJbsXbnrwCEUry16j7TVxgsnhY7em8JackvvjI=;
        b=pgBEfc46FCVRFnZsR1mLzvUJ8LIKZpAgGLPJjR/zigW+XbIVp42iHkJwKk/O8r4siP
         y6ka5c99dW/14gWgKJwP6q0VLQo9uIxQDQbTDrk3WsE5sHuYoj5Yv1mLTWtzocJjbOt4
         HBNNAyJGVupg/sudgBzrMrI4/ny8JJZxBJzMM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=UiGKKs1eeSxUMFMT0h31ApiBa/zWlv5rmLUjLoN1arH+Y8tqFRsxlT2dwrrBkMq8xD
         fg4ajU81Ku4waqk+7QLyt6QKiOqoLj1aWsomb8oheA0Yp6ZuA8XXir3CWdyL6VDPrHMc
         QS3TaQX2V3cj7uw09uyXM6vg6e2X5MIuYhKCA=
MIME-Version: 1.0
Received: by 10.231.191.130 with HTTP; Mon, 12 Apr 2010 04:37:33 -0700 (PDT)
In-Reply-To: <EBFEB403-3709-44E2-A930-11A07A753AE5@linkedin.com>
References: <h2y1cbd6f831004100733xf3ab6298u278df45f8d80d3ad@mail.gmail.com>
	 <EBFEB403-3709-44E2-A930-11A07A753AE5@linkedin.com>
Date: Mon, 12 Apr 2010 07:37:33 -0400
Received: by 10.231.170.14 with SMTP id b14mr1848781ibz.54.1271072253602; Mon, 
	12 Apr 2010 04:37:33 -0700 (PDT)
Message-ID: <t2h1cbd6f831004120437nf86c7809te27fd31fb6c706c3@mail.gmail.com>
Subject: Re: rack awareness question
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Thanks.

What is 'JT'? JobTracker?


On Sun, Apr 11, 2010 at 8:30 PM, Allen Wittenauer
<awittenauer@linkedin.com> wrote:
>
> On Apr 10, 2010, at 7:33 AM, Mag Gam wrote:
>> What is the procedure to do this?
>
> Change the topology script, bounce NN and JT.
>
>
>> is this possible to do while the
>> cluster is running?
>
> No. =C2=A0There is a jira i filed on turning the cache into an actual cac=
he w/TTLs, etc.

From general-return-1337-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 15:42:38 2010
Return-Path: <general-return-1337-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77539 invoked from network); 12 Apr 2010 15:42:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 15:42:37 -0000
Received: (qmail 27476 invoked by uid 500); 12 Apr 2010 15:42:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27369 invoked by uid 500); 12 Apr 2010 15:42:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27361 invoked by uid 99); 12 Apr 2010 15:42:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 15:42:36 +0000
X-ASF-Spam-Status: No, hits=-0.8 required=10.0
	tests=AWL,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 74.125.92.27 as permitted sender)
Received: from [74.125.92.27] (HELO qw-out-2122.google.com) (74.125.92.27)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 15:42:30 +0000
Received: by qw-out-2122.google.com with SMTP id 8so2464092qwh.35
        for <general@hadoop.apache.org>; Mon, 12 Apr 2010 08:42:10 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:received:message-id:subject
         :from:to:content-type:content-transfer-encoding;
        bh=VFHtcjBWczpUNS5Kw1RJdPpNeQih0+lxGnn6QpQhsaM=;
        b=dlV+fRa5gvR3B9krZt3IBZr2lN/uzvu36ovSiJFS5v6i+si1mvKBDi25/VgnSLt7rF
         YmB7maF0eF6FKs9dr0KCFTR4Xk1+o0OooVqkMobf9NDaAeVPWV8zHpTE7dI2w2L05cRI
         QPHjZcMd2UFo9h9ZIE+/ihBNoznglDECDIvsI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=fVfHfJbauhjfrBszT8OuD8W1/06kEpL7Ni1jFRywcD4ypNP1Eo1WD9IWESweM8YM3o
         wrqbJcjs87gcAackYmPOspOZhXdnUJTwE2vMNl5gfyeVH8s+Qarb9hRJQSH8Vdw1sDH2
         1VZzfNGXR/CWfOrjx0FdPQc2uIqVtIp7UvYZQ=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
Received: by 10.229.10.133 with HTTP; Mon, 12 Apr 2010 08:42:09 -0700 (PDT)
In-Reply-To: <x2s54dc3c51004082309nfd39c80bk2da33ceca0d7cf93@mail.gmail.com>
References: <x2l7c962aed1004072102j3ca29379he9f4095be7736f53@mail.gmail.com>
	 <3F1902B5-BAD3-440D-8D35-6EBC3297B939@yahoo-inc.com>
	 <n2j7c962aed1004081715q7ae4a675k7c855618cc8b0996@mail.gmail.com>
	 <p2m54dc3c51004081841y4d86190ey126dbff961df024@mail.gmail.com>
	 <C9980F2D-E394-426B-8282-3E1095784561@yahoo-inc.com>
	 <r2g54dc3c51004082040p9807d80fn5eba502289afc67b@mail.gmail.com>
	 <i2zac79ea401004082100hf5da28f3vab8881fc102dc0bc@mail.gmail.com>
	 <p2n7bfdc29a1004082107j24f21354rb9b351b7ee6c587e@mail.gmail.com>
	 <x2s54dc3c51004082309nfd39c80bk2da33ceca0d7cf93@mail.gmail.com>
Date: Mon, 12 Apr 2010 08:42:09 -0700
X-Google-Sender-Auth: 55620509235ee58b
Received: by 10.229.193.18 with SMTP id ds18mr6082257qcb.14.1271086929976; 
	Mon, 12 Apr 2010 08:42:09 -0700 (PDT)
Message-ID: <r2k7c962aed1004120842mcb58e680vb2062c7de84ebe2f@mail.gmail.com>
Subject: Re: [DISCUSS] HBase as TLP
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Its been a while since there's been a peep out of this thread so I'll
now move this topic to a vote.

Thanks to all who contributed to the discussion.
St.Ack

On Thu, Apr 8, 2010 at 11:09 PM, Jay Booth <jaybooth@gmail.com> wrote:
> Alright, I totally agree. =A0Thanks for putting it that way.
>
> -Jay
>
> On Fri, Apr 9, 2010 at 12:07 AM, Imran M Yousuf <imyousuf@gmail.com> wrot=
e:
>
>> +1
>>
>> I feel the same. From following HBase seeing its releases depending
>> directly on Hadoop release gets me thinking...
>>
>> Best regards,
>>
>> Imran
>>
>> On Fri, Apr 9, 2010 at 9:45 AM, Tom White <tom@cloudera.com> wrote:
>> > Eclipse does big bang releases of multiple components, but I believe
>> > it requires a huge amount of coordination and planning. Instead, I
>> > think the direction Hadoop should move in is to stabilize and clearly
>> > demarcate its core filesystem and MapReduce interfaces, so that
>> > projects like HBase, Pig, and Hive can run against multiple versions
>> > of core. Their release cycles are already largely decoupled from core,
>> > so the question about whether they become TLPs is more to do with
>> > project governance than with release coordination.
>> >
>> > Cheers,
>> > Tom
>> >
>> > On Thu, Apr 8, 2010 at 8:40 PM, Jay Booth <jaybooth@gmail.com> wrote:
>> >> Not sure exactly what I meant by "1.0 of what", "Hadoop" I guess, I w=
as
>> >> trying to address the concerns raised, which I share -- Alan's concer=
n
>> is
>> >> that if the projects are completely separate from each other, that mi=
ght
>> >> decrease visibility as to the demands they're placing on each other w=
hen
>> >> integrated, and St.Ack mentioned the frankenstein factor which I thin=
k
>> we've
>> >> all felt some pain from, and which may get worse after the project
>> split.
>> >> What's the standard way to deploy the three, even? =A0Is there one?
>> >>
>> >> If the PMCs jointly maintained some sort of 'stable integrated build'
>> which
>> >> took in new releases from the TLPs as they were released after a soak
>> >> period, it could provide a common touchstone that bugs could be teste=
d
>> >> against and cross-component patches delivered against, potentially
>> >> increasing visibility of cross-component issues while providing a les=
s
>> >> cobbled-together system to administrate. =A0On the other side, though=
, if
>> >> executed wrong, you'd be creating a committee of committees and possi=
bly
>> >> undoing some of the benefits of going TLP in the first place, especia=
lly
>> if
>> >> politics heat up over what goes into the 'standard' build. =A0I think=
 it
>> could
>> >> be viable though.
>> >>
>> >>
>> >>
>> >> On Thu, Apr 8, 2010 at 10:02 PM, Arun C Murthy <acm@yahoo-inc.com>
>> wrote:
>> >>
>> >>>
>> >>> On Apr 8, 2010, at 6:41 PM, Jay Booth wrote:
>> >>>
>> >>> =A0What if the projects were:
>> >>>>
>> >>>> A) =A0split out to TLPs because they do seem to have reached that l=
evel
>> of
>> >>>> individual community
>> >>>>
>> >>>> but,
>> >>>>
>> >>>> B) =A0The projects could somehow jointly put out an integrated buil=
d
>> >>>> containing the above projects and let users run whatever they want =
out
>> of
>> >>>> it?
>> >>>>
>> >>>> That would require a lot of coordination but would make a heck of a
>> 1.0
>> >>>> release,
>> >>>>
>> >>>
>> >>>
>> >>> 1.0 release of what?
>> >>>
>> >>> Arun
>> >>>
>> >>
>> >
>>
>>
>>
>> --
>> Imran M Yousuf
>> Entrepreneur & Software Engineer
>> Smart IT Engineering
>> Dhaka, Bangladesh
>> Email: imran@smartitengineering.com
>> Blog: http://imyousuf-tech.blogs.smartitengineering.com/
>> Mobile: +880-1711402557
>>
>

From general-return-1338-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 15:57:22 2010
Return-Path: <general-return-1338-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87151 invoked from network); 12 Apr 2010 15:57:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 15:57:21 -0000
Received: (qmail 59971 invoked by uid 500); 12 Apr 2010 15:57:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59855 invoked by uid 500); 12 Apr 2010 15:57:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59842 invoked by uid 99); 12 Apr 2010 15:57:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 15:57:20 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 74.125.92.24 as permitted sender)
Received: from [74.125.92.24] (HELO qw-out-2122.google.com) (74.125.92.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 15:57:12 +0000
Received: by qw-out-2122.google.com with SMTP id 8so2470569qwh.35
        for <multiple recipients>; Mon, 12 Apr 2010 08:56:51 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:date
         :x-google-sender-auth:received:message-id:subject:from:to
         :content-type;
        bh=++z8ceir4yeWezuJoC4DQP/oAQJ9UmvCMpZGThdlWkY=;
        b=GXgeeXTsuaAywCm+x0gSZiPunjYTeSv3oyBGTCOHXiS47hLPJtrnVMPTx5a3kS1+rS
         /agX07V8wQiJljilCwK/0aCEMS26YJUDGxsoacVrOleMbAWLp29PYiUEEqkjhJQv9edX
         4ZNHST322pz8rqwLqcbmwhlE3oWi/wnmeRo5s=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        b=o6EqmId63/pQmVpkX0m5xOdfvea38jDEecmgBcZY49DYRYK65yM1sSl0kf6s+p2Hml
         Kqk0N4bnLwSF1XtKf9ujUB3wazBolmg+gRGkytoekCFQOoy8XYVpkP2l/Irc5eUfM6zO
         g4vO/ORdY7QE67n9Lgpj0vKxQmw3LYcezeOOk=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
Received: by 10.229.10.133 with HTTP; Mon, 12 Apr 2010 08:56:51 -0700 (PDT)
Date: Mon, 12 Apr 2010 08:56:51 -0700
X-Google-Sender-Auth: ca2f50e17d7c1ac2
Received: by 10.229.218.204 with SMTP id hr12mr188968qcb.101.1271087811372; 
	Mon, 12 Apr 2010 08:56:51 -0700 (PDT)
Message-ID: <m2l7c962aed1004120856hddd9e41dt77f0ba3e259c9171@mail.gmail.com>
Subject: [VOTE] HBase as TLP?
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org, hbase-dev@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Please vote as to whether you think HBase should become a top-level
Apache project.

I've included below a draft board resolution.  It lists all current
active HBase committers as initial members of the project management
committee (PMC) and myself, Michael Stack, as the initial chair.

Do the good people of Hadoop support sending this request on to the Hadoop PMC?

Yours,
St.Ack

------------

   X. Establish the Apache HBase Project

      WHEREAS, the Board of Directors deems it to be in the best
      interests of the Foundation and consistent with the
      Foundation's purpose to establish a Project Management
      Committee charged with the creation and maintenance of
      open-source software related to data serialization
      for distribution at no charge to the public.

      NOW, THEREFORE, BE IT RESOLVED, that a Project Management
      Committee (PMC), to be known as the "Apache HBase Project",
      be and hereby is established pursuant to Bylaws of the
      Foundation; and be it further

      RESOLVED, that the Apache HBase Project be and hereby is
      responsible for the creation and maintenance of software
      related to a distributed database; and be it further

      RESOLVED, that the office of "Vice President, Apache HBase" be
      and hereby is created, the person holding such office to
      serve at the direction of the Board of Directors as the chair
      of the Apache HBase Project, and to have primary responsibility
      for management of the projects within the scope of
      responsibility of the Apache HBase Project; and be it further

      RESOLVED, that the persons listed immediately below be and
      hereby are appointed to serve as the initial members of the
      Apache HBase Project:

        * Michael Stack <stack@apache.org>
        * Jonathan Gray <jgray@apache.org>
        * Andrew Purtell <apurtell@apache.org>
        * John-Daniel Cryans <jdcryans@apache.org>
        * Ryan Rawson <rawson@apache.org>
        * Lars George <larsgeorge@apache.org>

      NOW, THEREFORE, BE IT FURTHER RESOLVED, that Michael Stack
      be appointed to the office of Vice President, Apache HBase, to
      serve in accordance with and subject to the direction of the
      Board of Directors and the Bylaws of the Foundation until
      death, resignation, retirement, removal or disqualification,
      or until a successor is appointed; and be it further

      RESOLVED, that the initial Apache HBase PMC be and hereby is
      tasked with the creation of a set of bylaws intended to
      encourage open development and increased participation in the
      Apache HBase Project; and be it further

      RESOLVED, that the Apache HBase Project be and hereby
      is tasked with the migration and rationalization of the Apache
      Hadoop HBase sub-project; and be it further

      RESOLVED, that all responsibilities pertaining to the Apache
      Hadoop HBase sub-project encumbered upon the
      Apache Hadoop Project are hereafter discharged.

From general-return-1339-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 16:19:11 2010
Return-Path: <general-return-1339-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 99664 invoked from network); 12 Apr 2010 16:19:10 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 16:19:10 -0000
Received: (qmail 83265 invoked by uid 500); 12 Apr 2010 16:19:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83203 invoked by uid 500); 12 Apr 2010 16:19:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83195 invoked by uid 99); 12 Apr 2010 16:19:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 16:19:09 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 12 Apr 2010 16:19:07 +0000
Received: (qmail 99270 invoked by uid 99); 12 Apr 2010 16:18:45 -0000
Received: from localhost.apache.org (HELO [192.168.168.102]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 16:18:45 +0000
Message-ID: <4BC347E4.9050706@apache.org>
Date: Mon, 12 Apr 2010 09:18:44 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100317)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: [RESULT] [VOTE] Avro as TLP?
References: <4BBB9675.1090909@apache.org>
In-Reply-To: <4BBB9675.1090909@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

This passes, with 11 +1 votes and no -1 votes.

Owen, can you please add this resolution to the agenda for next week's 
board meeting?

Thanks,

Doug

Doug Cutting wrote:
> Please vote as to whether you think Avro should become a top-level
> Apache project, as mentioned previously on this list.
> 
> I've included below a draft board resolution.  It lists all current
> committers as the initial members of the project management committee
> (PMC) and Matt Massie as the initial chair.
> 
> Does the Hadoop PMC support sending this proposal on to the board?
> 
> Thanks,
> 
> Doug
> 
> ------------
> 
>     X. Establish the Apache Avro Project
> 
>        WHEREAS, the Board of Directors deems it to be in the best
>        interests of the Foundation and consistent with the
>        Foundation's purpose to establish a Project Management
>        Committee charged with the creation and maintenance of
>        open-source software related to data serialization
>        for distribution at no charge to the public.
> 
>        NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>        Committee (PMC), to be known as the "Apache Avro Project",
>        be and hereby is established pursuant to Bylaws of the
>        Foundation; and be it further
> 
>        RESOLVED, that the Apache Avro Project be and hereby is
>        responsible for the creation and maintenance of software
>        related to data serialization; and be it further
> 
>        RESOLVED, that the office of "Vice President, Apache Avro" be
>        and hereby is created, the person holding such office to
>        serve at the direction of the Board of Directors as the chair
>        of the Apache Avro Project, and to have primary responsibility
>        for management of the projects within the scope of
>        responsibility of the Apache Avro Project; and be it further
> 
>        RESOLVED, that the persons listed immediately below be and
>        hereby are appointed to serve as the initial members of the
>        Apache Avro Project:
> 
>          * Doug Cutting         <cutting@apache.org>
>          * Jeff Hammerbacher    <hammer@apache.org>
>          * Jeff Hodges          <jmhodges@apache.org>
>          * Matt Massie          <massie@apache.org>
>          * Philip Zeyliger      <philz@apache.org>
>          * Scott Banachowski    <sbanacho@apache.org>
>          * Scott Carey          <scottcarey@apache.org>
>          * Sharad Agarwal       <sharadag@apache.org>
>          * Thiruvalluvan M. G.  <thiru@apache.org>
> 
>        NOW, THEREFORE, BE IT FURTHER RESOLVED, that Matt Massie
>        be appointed to the office of Vice President, Apache Avro, to
>        serve in accordance with and subject to the direction of the
>        Board of Directors and the Bylaws of the Foundation until
>        death, resignation, retirement, removal or disqualification,
>        or until a successor is appointed; and be it further
> 
>        RESOLVED, that the initial Apache Avro PMC be and hereby is
>        tasked with the creation of a set of bylaws intended to
>        encourage open development and increased participation in the
>        Apache Avro Project; and be it further
> 
>        RESOLVED, that the Apache Avro Project be and hereby
>        is tasked with the migration and rationalization of the Apache
>        Hadoop Avro sub-project; and be it further
> 
>        RESOLVED, that all responsibilities pertaining to the Apache
>        Hadoop Avro sub-project encumbered upon the
>        Apache Hadoop Project are hereafter discharged.
> 

From general-return-1340-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 16:52:38 2010
Return-Path: <general-return-1340-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 18847 invoked from network); 12 Apr 2010 16:52:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 16:52:38 -0000
Received: (qmail 35341 invoked by uid 500); 12 Apr 2010 16:52:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 35153 invoked by uid 500); 12 Apr 2010 16:52:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 35140 invoked by uid 99); 12 Apr 2010 16:52:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 16:52:37 +0000
X-ASF-Spam-Status: No, hits=0.4 required=10.0
	tests=AWL,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 16:52:32 +0000
Received: by wwb29 with SMTP id 29so1718256wwb.35
        for <multiple recipients>; Mon, 12 Apr 2010 09:52:10 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.158.5 with HTTP; Mon, 12 Apr 2010 09:52:10 -0700 (PDT)
In-Reply-To: <m2l7c962aed1004120856hddd9e41dt77f0ba3e259c9171@mail.gmail.com>
References: <m2l7c962aed1004120856hddd9e41dt77f0ba3e259c9171@mail.gmail.com>
Date: Mon, 12 Apr 2010 09:52:10 -0700
Received: by 10.216.88.79 with SMTP id z57mr2544321wee.22.1271091130142; Mon, 
	12 Apr 2010 09:52:10 -0700 (PDT)
Message-ID: <u2w23d54121004120952j6a503986h61c6c87339ec67b@mail.gmail.com>
Subject: Re: [VOTE] HBase as TLP?
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Cc: hbase-dev@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

+1

On Mon, Apr 12, 2010 at 8:56 AM, Stack <stack@duboce.net> wrote:
> Please vote as to whether you think HBase should become a top-level
> Apache project.
>
> I've included below a draft board resolution. =A0It lists all current
> active HBase committers as initial members of the project management
> committee (PMC) and myself, Michael Stack, as the initial chair.
>
> Do the good people of Hadoop support sending this request on to the Hadoo=
p PMC?
>
> Yours,
> St.Ack
>
> ------------
>
> =A0 X. Establish the Apache HBase Project
>
> =A0 =A0 =A0WHEREAS, the Board of Directors deems it to be in the best
> =A0 =A0 =A0interests of the Foundation and consistent with the
> =A0 =A0 =A0Foundation's purpose to establish a Project Management
> =A0 =A0 =A0Committee charged with the creation and maintenance of
> =A0 =A0 =A0open-source software related to data serialization
> =A0 =A0 =A0for distribution at no charge to the public.
>
> =A0 =A0 =A0NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> =A0 =A0 =A0Committee (PMC), to be known as the "Apache HBase Project",
> =A0 =A0 =A0be and hereby is established pursuant to Bylaws of the
> =A0 =A0 =A0Foundation; and be it further
>
> =A0 =A0 =A0RESOLVED, that the Apache HBase Project be and hereby is
> =A0 =A0 =A0responsible for the creation and maintenance of software
> =A0 =A0 =A0related to a distributed database; and be it further
>
> =A0 =A0 =A0RESOLVED, that the office of "Vice President, Apache HBase" be
> =A0 =A0 =A0and hereby is created, the person holding such office to
> =A0 =A0 =A0serve at the direction of the Board of Directors as the chair
> =A0 =A0 =A0of the Apache HBase Project, and to have primary responsibilit=
y
> =A0 =A0 =A0for management of the projects within the scope of
> =A0 =A0 =A0responsibility of the Apache HBase Project; and be it further
>
> =A0 =A0 =A0RESOLVED, that the persons listed immediately below be and
> =A0 =A0 =A0hereby are appointed to serve as the initial members of the
> =A0 =A0 =A0Apache HBase Project:
>
> =A0 =A0 =A0 =A0* Michael Stack <stack@apache.org>
> =A0 =A0 =A0 =A0* Jonathan Gray <jgray@apache.org>
> =A0 =A0 =A0 =A0* Andrew Purtell <apurtell@apache.org>
> =A0 =A0 =A0 =A0* John-Daniel Cryans <jdcryans@apache.org>
> =A0 =A0 =A0 =A0* Ryan Rawson <rawson@apache.org>
> =A0 =A0 =A0 =A0* Lars George <larsgeorge@apache.org>
>
> =A0 =A0 =A0NOW, THEREFORE, BE IT FURTHER RESOLVED, that Michael Stack
> =A0 =A0 =A0be appointed to the office of Vice President, Apache HBase, to
> =A0 =A0 =A0serve in accordance with and subject to the direction of the
> =A0 =A0 =A0Board of Directors and the Bylaws of the Foundation until
> =A0 =A0 =A0death, resignation, retirement, removal or disqualification,
> =A0 =A0 =A0or until a successor is appointed; and be it further
>
> =A0 =A0 =A0RESOLVED, that the initial Apache HBase PMC be and hereby is
> =A0 =A0 =A0tasked with the creation of a set of bylaws intended to
> =A0 =A0 =A0encourage open development and increased participation in the
> =A0 =A0 =A0Apache HBase Project; and be it further
>
> =A0 =A0 =A0RESOLVED, that the Apache HBase Project be and hereby
> =A0 =A0 =A0is tasked with the migration and rationalization of the Apache
> =A0 =A0 =A0Hadoop HBase sub-project; and be it further
>
> =A0 =A0 =A0RESOLVED, that all responsibilities pertaining to the Apache
> =A0 =A0 =A0Hadoop HBase sub-project encumbered upon the
> =A0 =A0 =A0Apache Hadoop Project are hereafter discharged.
>



--=20
Eric Sammer
phone: +1-917-287-2675
twitter: esammer
data: www.cloudera.com

From general-return-1341-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 18:21:45 2010
Return-Path: <general-return-1341-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55653 invoked from network); 12 Apr 2010 18:21:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 18:21:45 -0000
Received: (qmail 90789 invoked by uid 500); 12 Apr 2010 18:21:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90623 invoked by uid 500); 12 Apr 2010 18:21:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90615 invoked by uid 99); 12 Apr 2010 18:21:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 18:21:44 +0000
X-ASF-Spam-Status: No, hits=-2.9 required=10.0
	tests=AWL,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 18:21:39 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:Received:Received:From:To:Subject:
   Thread-Topic:Thread-Index:Date:Message-ID:References:
   In-Reply-To:Accept-Language:Content-Language:
   X-MS-Has-Attach:X-MS-TNEF-Correlator:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version:
   Return-Path:X-OriginalArrivalTime;
  b=RMqQRswGBxUXeUl0ySKI8xGAJveLw93zVleCFpY1HaSpSgzPlipRb47j
   sgpSeYAtTXg35sJJaJVElJPcTCv6kkF/61JHP/isG+z6lUUzUpO/nOa2/
   F/H0J09iOOElyhE;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1271096499; x=1302632499;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20rack=20awareness=20question|Date:=20Mon
   ,=2012=20Apr=202010=2018:21:16=20+0000|Message-ID:=20<732
   D302F-02DD-4D45-8023-6484445CB5E9@linkedin.com>|To:=20"<g
   eneral@hadoop.apache.org>"=20<general@hadoop.apache.org>
   |MIME-Version:=201.0|Content-Transfer-Encoding:=20quoted-
   printable|Content-ID:=20<e5caa996-50d8-459c-9b69-cb006265
   cfea>|In-Reply-To:=20<q2pc9118d4c1004111828va42fc159s1db1
   3fbbb10aca9f@mail.gmail.com>|References:=20<h2y1cbd6f8310
   04100733xf3ab6298u278df45f8d80d3ad@mail.gmail.com>=0D=0A
   =20<EBFEB403-3709-44E2-A930-11A07A753AE5@linkedin.com>=0D
   =0A=20<q2pc9118d4c1004111828va42fc159s1db13fbbb10aca9f@ma
   il.gmail.com>;
  bh=QqT6sNPV6diVYX1MXQLgup5DghAVXvi/vc28/TcXTLU=;
  b=v1NAVx96IW21ze96H5dG/l/3uFfO9eeGiq2PNFL2vB0sBoRoo6BnwPjE
   WuKOv7+p+qeFLpyN+whVZTY1NrvHxDXcqQ7Vszm22EDTWnulGkQTJ69hL
   RYKnnbTKFlbG09m;
X-IronPort-AV: E=Sophos;i="4.52,192,1270450800"; 
   d="scan'208";a="11913400"
Received: from esv4-exctest.linkedin.biz ([172.18.46.60]) by CORP-MAIL.linkedin.biz with Microsoft SMTPSVC(6.0.3790.3959);
	 Mon, 12 Apr 2010 11:21:19 -0700
Received: from ESV4-CAS02.linkedin.biz (172.18.46.142) by
 esv4-exctest.linkedin.biz (172.18.46.60) with Microsoft SMTP Server (TLS) id
 14.0.682.1; Mon, 12 Apr 2010 11:21:18 -0700
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi; Mon, 12 Apr 2010
 11:21:18 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: rack awareness question
Thread-Topic: rack awareness question
Thread-Index: AQHK2Lrd83fFTEyGQkeCRFATVCpFBpIecp8AgAAQMwCAARryAA==
Date: Mon, 12 Apr 2010 18:21:16 +0000
Message-ID: <732D302F-02DD-4D45-8023-6484445CB5E9@linkedin.com>
References: <h2y1cbd6f831004100733xf3ab6298u278df45f8d80d3ad@mail.gmail.com>
 <EBFEB403-3709-44E2-A930-11A07A753AE5@linkedin.com>
 <q2pc9118d4c1004111828va42fc159s1db13fbbb10aca9f@mail.gmail.com>
In-Reply-To: <q2pc9118d4c1004111828va42fc159s1db13fbbb10aca9f@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Content-Type: text/plain; charset="us-ascii"
Content-ID: <e5caa996-50d8-459c-9b69-cb006265cfea>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginalArrivalTime: 12 Apr 2010 18:21:19.0211 (UTC) FILETIME=[F26E2BB0:01CADA6C]


On Apr 11, 2010, at 6:28 PM, Jeff Hammerbacher wrote:

>> No.  There is a jira i filed on turning the cache into an actual cache
>> w/TTLs, etc.
>=20
>=20
> https://issues.apache.org/jira/browse/HDFS-870, for those who like
> hyperlinks.

Thank you for supporting my laziness. ;)


From general-return-1342-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 19:06:09 2010
Return-Path: <general-return-1342-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73851 invoked from network); 12 Apr 2010 19:06:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 19:06:09 -0000
Received: (qmail 66871 invoked by uid 500); 12 Apr 2010 19:06:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66776 invoked by uid 500); 12 Apr 2010 19:06:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66768 invoked by uid 99); 12 Apr 2010 19:06:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 19:06:08 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.6.228.93] (HELO n13.bullet.mail.ac4.yahoo.com) (74.6.228.93)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 12 Apr 2010 19:05:58 +0000
Received: from [76.13.12.67] by n13.bullet.mail.ac4.yahoo.com with NNFMP; 12 Apr 2010 19:05:36 -0000
Received: from [76.13.10.181] by t8.bullet.mail.ac4.yahoo.com with NNFMP; 12 Apr 2010 19:05:36 -0000
Received: from [127.0.0.1] by omp122.mail.ac4.yahoo.com with NNFMP; 12 Apr 2010 19:05:36 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 683659.87141.bm@omp122.mail.ac4.yahoo.com
Received: (qmail 92543 invoked by uid 60001); 12 Apr 2010 19:05:36 -0000
Message-ID: <432237.92455.qm@web65513.mail.ac4.yahoo.com>
X-YMail-OSG: 9dlNexcVM1kJyjB2selUsiVKYdjmHtlj.JZMmPC36ZLdHNj
 TsOUGQiNDlcDAOs4O6xsGWt6MelZyF0ISHoaikRByXytn_e9FvikcxDW75c.
 JYAmg.Sm2kFEQ4R1GQO7GJpD9GbCryB7OhKE7pEF46yh7IQFclNQIAVehsn3
 nUa.Iw_d5tUguJRJACNw7w2kh_CXF6J581D9wG2HVyTQeICIsH0n3q8cYbb1
 YBJlqgti0KP2UP48IljVIJw2GyGBsiEDYNVHibZ4ISgyc.o9Krcq_8G71xzo
 eHCP4hSIWoQvP7zKXkk0Vlr2tYV3NvlPir753orJMNe5jZD0Bkk6z_IntPA-
 -
Received: from [69.108.153.136] by web65513.mail.ac4.yahoo.com via HTTP; Mon, 12 Apr 2010 12:05:36 PDT
X-RocketYMMF: apurtell
X-Mailer: YahooMailClassic/10.0.8 YahooMailWebService/0.8.100.260964
Date: Mon, 12 Apr 2010 12:05:36 -0700 (PDT)
From: Andrew Purtell <apurtell@apache.org>
Reply-To: apurtell@apache.org
Subject: Re: [VOTE] HBase as TLP?
To: general@hadoop.apache.org, hbase-dev@hadoop.apache.org
In-Reply-To: <m2l7c962aed1004120856hddd9e41dt77f0ba3e259c9171@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org

+1

  - Andy

> From: Stack
> Subject: [VOTE] HBase as TLP?
>
> Please vote as to whether you think HBase should become a top-level
> Apache project.



      


From general-return-1343-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 19:12:20 2010
Return-Path: <general-return-1343-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 75524 invoked from network); 12 Apr 2010 19:12:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 19:12:20 -0000
Received: (qmail 76794 invoked by uid 500); 12 Apr 2010 19:12:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76735 invoked by uid 500); 12 Apr 2010 19:12:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76727 invoked by uid 99); 12 Apr 2010 19:12:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 19:12:19 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of timrobertson100@gmail.com designates 209.85.219.215 as permitted sender)
Received: from [209.85.219.215] (HELO mail-ew0-f215.google.com) (209.85.219.215)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 19:12:11 +0000
Received: by ewy7 with SMTP id 7so1808282ewy.31
        for <general@hadoop.apache.org>; Mon, 12 Apr 2010 12:11:51 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=a/VEFip1wNAr1A+TS9Q6k5u6BbB1D4rQUpGuAfEET2E=;
        b=h6AEaS22TBlRNl6PUJtrxeRzOGcf58Hc0i9pDkita8hoqjph9KfojdyB5W1Upm/yzp
         3bSXxcNn9DOoUrSz0iijVQGx4wv5dpaxUqBrV0xmPu0INiff0eY6H3M717ZLi7DdouNQ
         Z8seS23C0gIfeBHDFgvFFOAk2TjLJuX3+wtII=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=sfkNpYXod8koOaz4BYsnDnHtvghfTf2AHL8R6VUYb6s5kWOc/UlXkDIAh+Q44BGxI1
         Knct6u2G4AJ6uIJ9wKrv6W5xyBFtqUaNCsCtfKDVyZgfdhoRA2SSq56tRk0uFpvsgkzx
         /by5lZjKsaWMtlVvLdNpcX/HELCGpbMFyyj+M=
MIME-Version: 1.0
Received: by 10.103.169.2 with HTTP; Mon, 12 Apr 2010 12:11:51 -0700 (PDT)
In-Reply-To: <432237.92455.qm@web65513.mail.ac4.yahoo.com>
References: <m2l7c962aed1004120856hddd9e41dt77f0ba3e259c9171@mail.gmail.com>
	 <432237.92455.qm@web65513.mail.ac4.yahoo.com>
Date: Mon, 12 Apr 2010 21:11:51 +0200
Received: by 10.102.12.13 with SMTP id 13mr2332090mul.133.1271099511096; Mon, 
	12 Apr 2010 12:11:51 -0700 (PDT)
Message-ID: <z2u32120a6a1004121211nf43d724eudf61194d1b092dba@mail.gmail.com>
Subject: Re: [VOTE] HBase as TLP?
From: Tim Robertson <timrobertson100@gmail.com>
To: general@hadoop.apache.org, apurtell@apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

+1

- Tim

On Mon, Apr 12, 2010 at 9:05 PM, Andrew Purtell <apurtell@apache.org> wrote=
:
> +1
>
> =A0- Andy
>
>> From: Stack
>> Subject: [VOTE] HBase as TLP?
>>
>> Please vote as to whether you think HBase should become a top-level
>> Apache project.
>
>
>
>
>
>

From general-return-1344-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 22:10:26 2010
Return-Path: <general-return-1344-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63197 invoked from network); 12 Apr 2010 22:10:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 22:10:26 -0000
Received: (qmail 8680 invoked by uid 500); 12 Apr 2010 22:10:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8634 invoked by uid 500); 12 Apr 2010 22:10:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8619 invoked by uid 99); 12 Apr 2010 22:10:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 22:10:25 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of lars.george@gmail.com designates 74.125.83.48 as permitted sender)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 22:10:17 +0000
Received: by gwaa12 with SMTP id a12so2985283gwa.35
        for <multiple recipients>; Mon, 12 Apr 2010 15:09:56 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:cc:content-type
         :content-transfer-encoding;
        bh=sneGWTyh9SwzaKVdKhvw8kZKXlysWGxNwa8VsdDfQHU=;
        b=WH5YmjGMSGQUP3OIAJSl1rfj1sApscyjTkKy+ZVM0aqiFcj0xLYx4QOuBFH5csaCOC
         8h2eQ/kFmdAKpzkMHx90+efYtp7qd4+gXh+vz0er5NQEJ5SDHVWwSdVrVdVO+4TSFrt1
         zIqGJIUvX7MPrgRrVR4/QaexjxYllTYxQlD+o=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type:content-transfer-encoding;
        b=TkJfL5R7ARvKXyhTNoEabjiRVaYbrf6dh8mgHA+E5PIva+YSVImySPGNMHFK07cV2l
         DIvi2xxnxFOs1s5Fi/VjHXudAt/R5dKEzd1Cc/Pv1xAWJ4a9s565qcHQU8s6jrZwEkkv
         ILEgCA2yUD4P/4NShmDOc3AVSbJEfjs9KYluQ=
MIME-Version: 1.0
Received: by 10.151.100.5 with HTTP; Mon, 12 Apr 2010 15:09:56 -0700 (PDT)
In-Reply-To: <432237.92455.qm@web65513.mail.ac4.yahoo.com>
References: <m2l7c962aed1004120856hddd9e41dt77f0ba3e259c9171@mail.gmail.com>
	 <432237.92455.qm@web65513.mail.ac4.yahoo.com>
Date: Tue, 13 Apr 2010 00:09:56 +0200
Received: by 10.151.60.7 with SMTP id n7mr4101546ybk.183.1271110196755; Mon, 
	12 Apr 2010 15:09:56 -0700 (PDT)
Message-ID: <y2z61770b881004121509td61462d4g65759fffc0b910@mail.gmail.com>
Subject: Re: [VOTE] HBase as TLP?
From: Lars George <lars.george@gmail.com>
To: hbase-dev@hadoop.apache.org
Cc: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

+1

---
Lars

On Mon, Apr 12, 2010 at 9:05 PM, Andrew Purtell <apurtell@apache.org> wrote=
:
> +1
>
> =A0- Andy
>
>> From: Stack
>> Subject: [VOTE] HBase as TLP?
>>
>> Please vote as to whether you think HBase should become a top-level
>> Apache project.
>
>
>
>
>
>

From general-return-1345-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 22:21:16 2010
Return-Path: <general-return-1345-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70769 invoked from network); 12 Apr 2010 22:21:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 22:21:15 -0000
Received: (qmail 20150 invoked by uid 500); 12 Apr 2010 22:21:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20044 invoked by uid 500); 12 Apr 2010 22:21:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 20021 invoked by uid 99); 12 Apr 2010 22:21:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 22:21:13 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jdcryans@gmail.com designates 74.125.83.48 as permitted sender)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 22:21:06 +0000
Received: by gwaa12 with SMTP id a12so2991549gwa.35
        for <multiple recipients>; Mon, 12 Apr 2010 15:20:45 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:received:message-id:subject
         :from:to:cc:content-type:content-transfer-encoding;
        bh=jvjcwPhSi11Z8BMKiuRmz+LXEcQeikwE+zvPEBdtEDA=;
        b=RfZwGgedlsciOPEftV7GnLGBeTm0VLgdMsD4WSe19OZ4MPbktv6yLyqMT+r9bO59bG
         8q3M0Tt70oRCrVHX37W+jdpo7qlnncRInPgrHFXzXz5n6S4Qnkp2cl8BghsVIkGKRL7O
         sAPRX+e2Ac4az+nAigIO3vqpJvclddO/mkLls=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:cc:content-type
         :content-transfer-encoding;
        b=rKANZL/OA08gJAMTEz8WMx/S6IuWQCaoNBwMm6J122qheIFb9vQX9YHLimhXRFpIVu
         dJhPE18abrikSMn8dH+z135nX/CbL+LtcQY5ZWw0ci2aXvBk3phBsKXmuNwd5Q8nb9mj
         XCRIHEfU1aMsJSpMuSeqR2vT+BvnxZVWVLIRw=
MIME-Version: 1.0
Sender: jdcryans@gmail.com
Received: by 10.90.89.20 with HTTP; Mon, 12 Apr 2010 15:20:45 -0700 (PDT)
In-Reply-To: <m2l7c962aed1004120856hddd9e41dt77f0ba3e259c9171@mail.gmail.com>
References: <m2l7c962aed1004120856hddd9e41dt77f0ba3e259c9171@mail.gmail.com>
Date: Tue, 13 Apr 2010 00:20:45 +0200
X-Google-Sender-Auth: 71122ad0dbbf5a5e
Received: by 10.91.192.14 with SMTP id u14mr269695agp.98.1271110845611; Mon, 
	12 Apr 2010 15:20:45 -0700 (PDT)
Message-ID: <g2l31a243e71004121520g736c86aayd375811b3f00dbee@mail.gmail.com>
Subject: Re: [VOTE] HBase as TLP?
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: general@hadoop.apache.org
Cc: hbase-dev@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

+1

On Mon, Apr 12, 2010 at 5:56 PM, Stack <stack@duboce.net> wrote:
> Please vote as to whether you think HBase should become a top-level
> Apache project.
>
> I've included below a draft board resolution. =A0It lists all current
> active HBase committers as initial members of the project management
> committee (PMC) and myself, Michael Stack, as the initial chair.
>
> Do the good people of Hadoop support sending this request on to the Hadoo=
p PMC?
>
> Yours,
> St.Ack
>
> ------------
>
> =A0 X. Establish the Apache HBase Project
>
> =A0 =A0 =A0WHEREAS, the Board of Directors deems it to be in the best
> =A0 =A0 =A0interests of the Foundation and consistent with the
> =A0 =A0 =A0Foundation's purpose to establish a Project Management
> =A0 =A0 =A0Committee charged with the creation and maintenance of
> =A0 =A0 =A0open-source software related to data serialization
> =A0 =A0 =A0for distribution at no charge to the public.
>
> =A0 =A0 =A0NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> =A0 =A0 =A0Committee (PMC), to be known as the "Apache HBase Project",
> =A0 =A0 =A0be and hereby is established pursuant to Bylaws of the
> =A0 =A0 =A0Foundation; and be it further
>
> =A0 =A0 =A0RESOLVED, that the Apache HBase Project be and hereby is
> =A0 =A0 =A0responsible for the creation and maintenance of software
> =A0 =A0 =A0related to a distributed database; and be it further
>
> =A0 =A0 =A0RESOLVED, that the office of "Vice President, Apache HBase" be
> =A0 =A0 =A0and hereby is created, the person holding such office to
> =A0 =A0 =A0serve at the direction of the Board of Directors as the chair
> =A0 =A0 =A0of the Apache HBase Project, and to have primary responsibilit=
y
> =A0 =A0 =A0for management of the projects within the scope of
> =A0 =A0 =A0responsibility of the Apache HBase Project; and be it further
>
> =A0 =A0 =A0RESOLVED, that the persons listed immediately below be and
> =A0 =A0 =A0hereby are appointed to serve as the initial members of the
> =A0 =A0 =A0Apache HBase Project:
>
> =A0 =A0 =A0 =A0* Michael Stack <stack@apache.org>
> =A0 =A0 =A0 =A0* Jonathan Gray <jgray@apache.org>
> =A0 =A0 =A0 =A0* Andrew Purtell <apurtell@apache.org>
> =A0 =A0 =A0 =A0* John-Daniel Cryans <jdcryans@apache.org>
> =A0 =A0 =A0 =A0* Ryan Rawson <rawson@apache.org>
> =A0 =A0 =A0 =A0* Lars George <larsgeorge@apache.org>
>
> =A0 =A0 =A0NOW, THEREFORE, BE IT FURTHER RESOLVED, that Michael Stack
> =A0 =A0 =A0be appointed to the office of Vice President, Apache HBase, to
> =A0 =A0 =A0serve in accordance with and subject to the direction of the
> =A0 =A0 =A0Board of Directors and the Bylaws of the Foundation until
> =A0 =A0 =A0death, resignation, retirement, removal or disqualification,
> =A0 =A0 =A0or until a successor is appointed; and be it further
>
> =A0 =A0 =A0RESOLVED, that the initial Apache HBase PMC be and hereby is
> =A0 =A0 =A0tasked with the creation of a set of bylaws intended to
> =A0 =A0 =A0encourage open development and increased participation in the
> =A0 =A0 =A0Apache HBase Project; and be it further
>
> =A0 =A0 =A0RESOLVED, that the Apache HBase Project be and hereby
> =A0 =A0 =A0is tasked with the migration and rationalization of the Apache
> =A0 =A0 =A0Hadoop HBase sub-project; and be it further
>
> =A0 =A0 =A0RESOLVED, that all responsibilities pertaining to the Apache
> =A0 =A0 =A0Hadoop HBase sub-project encumbered upon the
> =A0 =A0 =A0Apache Hadoop Project are hereafter discharged.
>

From general-return-1346-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 22:39:00 2010
Return-Path: <general-return-1346-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83038 invoked from network); 12 Apr 2010 22:38:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 22:38:59 -0000
Received: (qmail 45300 invoked by uid 500); 12 Apr 2010 22:38:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45162 invoked by uid 500); 12 Apr 2010 22:38:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45146 invoked by uid 99); 12 Apr 2010 22:38:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 22:38:58 +0000
X-ASF-Spam-Status: No, hits=-0.6 required=10.0
	tests=AWL,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 22:38:52 +0000
Received: from SNV-EXBH01.ds.corp.yahoo.com (snv-exbh01.ds.corp.yahoo.com [207.126.227.249])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o3CMc6nN091124;
	Mon, 12 Apr 2010 15:38:07 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:x-originalarrivaltime;
	b=2qlDVOvg3ZaY5dP41BYfUhwB7niL6KiLmq+XvMV4dnUZs7kgVc1BBtt38hOKvq0l
Received: from SNV-EXVS10.ds.corp.yahoo.com ([207.126.227.89]) by SNV-EXBH01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Mon, 12 Apr 2010 15:38:07 -0700
Received: from 172.21.148.59 ([172.21.148.59]) by SNV-EXVS10.ds.corp.yahoo.com ([207.126.227.103]) via Exchange Front-End Server snv-webmail.corp.yahoo.com ([207.126.227.60]) with Microsoft Exchange Server HTTP-DAV ;
 Mon, 12 Apr 2010 22:37:48 +0000
User-Agent: Microsoft-Entourage/12.24.0.100205
Date: Mon, 12 Apr 2010 15:37:45 -0700
Subject: Re: [VOTE] HBase as TLP?
From: Amir Youssefi <amiry@yahoo-inc.com>
To: <general@hadoop.apache.org>, <hbase-dev@hadoop.apache.org>
Message-ID: <C7E8EEC9.1E96A%amiry@yahoo-inc.com>
Thread-Topic: [VOTE] HBase as TLP?
Thread-Index: AcrakMUTC/bAdlt7okeMZyhtFSzGVw==
In-Reply-To: <m2l7c962aed1004120856hddd9e41dt77f0ba3e259c9171@mail.gmail.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-OriginalArrivalTime: 12 Apr 2010 22:38:07.0007 (UTC) FILETIME=[D23142F0:01CADA90]

+1

-ay

On 4/12/10 8:56 AM, "Stack" <stack@duboce.net> wrote:

> Please vote as to whether you think HBase should become a top-level
> Apache project.
> 
> I've included below a draft board resolution.  It lists all current
> active HBase committers as initial members of the project management
> committee (PMC) and myself, Michael Stack, as the initial chair.
> 
> Do the good people of Hadoop support sending this request on to the Hadoop
> PMC?
> 
> Yours,
> St.Ack
> 
> ------------
> 
>    X. Establish the Apache HBase Project
> 
>       WHEREAS, the Board of Directors deems it to be in the best
>       interests of the Foundation and consistent with the
>       Foundation's purpose to establish a Project Management
>       Committee charged with the creation and maintenance of
>       open-source software related to data serialization
>       for distribution at no charge to the public.
> 
>       NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>       Committee (PMC), to be known as the "Apache HBase Project",
>       be and hereby is established pursuant to Bylaws of the
>       Foundation; and be it further
> 
>       RESOLVED, that the Apache HBase Project be and hereby is
>       responsible for the creation and maintenance of software
>       related to a distributed database; and be it further
> 
>       RESOLVED, that the office of "Vice President, Apache HBase" be
>       and hereby is created, the person holding such office to
>       serve at the direction of the Board of Directors as the chair
>       of the Apache HBase Project, and to have primary responsibility
>       for management of the projects within the scope of
>       responsibility of the Apache HBase Project; and be it further
> 
>       RESOLVED, that the persons listed immediately below be and
>       hereby are appointed to serve as the initial members of the
>       Apache HBase Project:
> 
>         * Michael Stack <stack@apache.org>
>         * Jonathan Gray <jgray@apache.org>
>         * Andrew Purtell <apurtell@apache.org>
>         * John-Daniel Cryans <jdcryans@apache.org>
>         * Ryan Rawson <rawson@apache.org>
>         * Lars George <larsgeorge@apache.org>
> 
>       NOW, THEREFORE, BE IT FURTHER RESOLVED, that Michael Stack
>       be appointed to the office of Vice President, Apache HBase, to
>       serve in accordance with and subject to the direction of the
>       Board of Directors and the Bylaws of the Foundation until
>       death, resignation, retirement, removal or disqualification,
>       or until a successor is appointed; and be it further
> 
>       RESOLVED, that the initial Apache HBase PMC be and hereby is
>       tasked with the creation of a set of bylaws intended to
>       encourage open development and increased participation in the
>       Apache HBase Project; and be it further
> 
>       RESOLVED, that the Apache HBase Project be and hereby
>       is tasked with the migration and rationalization of the Apache
>       Hadoop HBase sub-project; and be it further
> 
>       RESOLVED, that all responsibilities pertaining to the Apache
>       Hadoop HBase sub-project encumbered upon the
>       Apache Hadoop Project are hereafter discharged.


From general-return-1347-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 22:42:28 2010
Return-Path: <general-return-1347-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84720 invoked from network); 12 Apr 2010 22:42:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 22:42:28 -0000
Received: (qmail 50583 invoked by uid 500); 12 Apr 2010 22:42:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50520 invoked by uid 500); 12 Apr 2010 22:42:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50506 invoked by uid 99); 12 Apr 2010 22:42:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 22:42:26 +0000
X-ASF-Spam-Status: No, hits=-0.3 required=10.0
	tests=AWL,MIME_QP_LONG_LINE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 22:42:20 +0000
Received: from SNV-EXPF01.ds.corp.yahoo.com (snv-expf01.ds.corp.yahoo.com [207.126.227.250])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o3CMfcZ2084463;
	Mon, 12 Apr 2010 15:41:39 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:cc:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:x-originalarrivaltime;
	b=k902QHVpZBVRwpKa6dWFZ3bFeD+uzRhQai9k6vUxiR9kwfapx9JfkjZwkvrqNTLy
Received: from SNV-EXVS09.ds.corp.yahoo.com ([207.126.227.86]) by SNV-EXPF01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Mon, 12 Apr 2010 15:41:38 -0700
Received: from 10.73.146.106 ([10.73.146.106]) by SNV-EXVS09.ds.corp.yahoo.com ([207.126.227.84]) via Exchange Front-End Server snv-webmail.corp.yahoo.com ([207.126.227.59]) with Microsoft Exchange Server HTTP-DAV ;
 Mon, 12 Apr 2010 22:41:06 +0000
User-Agent: Microsoft-Entourage/12.24.0.100205
Date: Mon, 12 Apr 2010 15:41:06 -0700
Subject: Re: [VOTE] HBase as TLP?
From: Mahadev Konar <mahadev@yahoo-inc.com>
To: <general@hadoop.apache.org>
CC: <hbase-dev@hadoop.apache.org>
Message-ID: <C7E8EF92.3363F%mahadev@yahoo-inc.com>
Thread-Topic: [VOTE] HBase as TLP?
Thread-Index: AcrakTzhSkukb7lFYUeglHISHYP5ow==
In-Reply-To: <g2l31a243e71004121520g736c86aayd375811b3f00dbee@mail.gmail.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="ISO-8859-1"
Content-transfer-encoding: quoted-printable
X-OriginalArrivalTime: 12 Apr 2010 22:41:38.0822 (UTC) FILETIME=[5071AA60:01CADA91]

+1=20

Thanks
mahadev


On 4/12/10 3:20 PM, "Jean-Daniel Cryans" <jdcryans@apache.org> wrote:

> +1
>=20
> On Mon, Apr 12, 2010 at 5:56 PM, Stack <stack@duboce.net> wrote:
>> Please vote as to whether you think HBase should become a top-level
>> Apache project.
>>=20
>> I've included below a draft board resolution. =A0It lists all current
>> active HBase committers as initial members of the project management
>> committee (PMC) and myself, Michael Stack, as the initial chair.
>>=20
>> Do the good people of Hadoop support sending this request on to the Hado=
op
>> PMC?
>>=20
>> Yours,
>> St.Ack
>>=20
>> ------------
>>=20
>> =A0 X. Establish the Apache HBase Project
>>=20
>> =A0 =A0 =A0WHEREAS, the Board of Directors deems it to be in the best
>> =A0 =A0 =A0interests of the Foundation and consistent with the
>> =A0 =A0 =A0Foundation's purpose to establish a Project Management
>> =A0 =A0 =A0Committee charged with the creation and maintenance of
>> =A0 =A0 =A0open-source software related to data serialization
>> =A0 =A0 =A0for distribution at no charge to the public.
>>=20
>> =A0 =A0 =A0NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>> =A0 =A0 =A0Committee (PMC), to be known as the "Apache HBase Project",
>> =A0 =A0 =A0be and hereby is established pursuant to Bylaws of the
>> =A0 =A0 =A0Foundation; and be it further
>>=20
>> =A0 =A0 =A0RESOLVED, that the Apache HBase Project be and hereby is
>> =A0 =A0 =A0responsible for the creation and maintenance of software
>> =A0 =A0 =A0related to a distributed database; and be it further
>>=20
>> =A0 =A0 =A0RESOLVED, that the office of "Vice President, Apache HBase" be
>> =A0 =A0 =A0and hereby is created, the person holding such office to
>> =A0 =A0 =A0serve at the direction of the Board of Directors as the chair
>> =A0 =A0 =A0of the Apache HBase Project, and to have primary responsibility
>> =A0 =A0 =A0for management of the projects within the scope of
>> =A0 =A0 =A0responsibility of the Apache HBase Project; and be it further
>>=20
>> =A0 =A0 =A0RESOLVED, that the persons listed immediately below be and
>> =A0 =A0 =A0hereby are appointed to serve as the initial members of the
>> =A0 =A0 =A0Apache HBase Project:
>>=20
>> =A0 =A0 =A0 =A0* Michael Stack <stack@apache.org>
>> =A0 =A0 =A0 =A0* Jonathan Gray <jgray@apache.org>
>> =A0 =A0 =A0 =A0* Andrew Purtell <apurtell@apache.org>
>> =A0 =A0 =A0 =A0* John-Daniel Cryans <jdcryans@apache.org>
>> =A0 =A0 =A0 =A0* Ryan Rawson <rawson@apache.org>
>> =A0 =A0 =A0 =A0* Lars George <larsgeorge@apache.org>
>>=20
>> =A0 =A0 =A0NOW, THEREFORE, BE IT FURTHER RESOLVED, that Michael Stack
>> =A0 =A0 =A0be appointed to the office of Vice President, Apache HBase, to
>> =A0 =A0 =A0serve in accordance with and subject to the direction of the
>> =A0 =A0 =A0Board of Directors and the Bylaws of the Foundation until
>> =A0 =A0 =A0death, resignation, retirement, removal or disqualification,
>> =A0 =A0 =A0or until a successor is appointed; and be it further
>>=20
>> =A0 =A0 =A0RESOLVED, that the initial Apache HBase PMC be and hereby is
>> =A0 =A0 =A0tasked with the creation of a set of bylaws intended to
>> =A0 =A0 =A0encourage open development and increased participation in the
>> =A0 =A0 =A0Apache HBase Project; and be it further
>>=20
>> =A0 =A0 =A0RESOLVED, that the Apache HBase Project be and hereby
>> =A0 =A0 =A0is tasked with the migration and rationalization of the Apache
>> =A0 =A0 =A0Hadoop HBase sub-project; and be it further
>>=20
>> =A0 =A0 =A0RESOLVED, that all responsibilities pertaining to the Apache
>> =A0 =A0 =A0Hadoop HBase sub-project encumbered upon the
>> =A0 =A0 =A0Apache Hadoop Project are hereafter discharged.
>>=20


From general-return-1348-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 22:43:31 2010
Return-Path: <general-return-1348-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85511 invoked from network); 12 Apr 2010 22:43:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 22:43:30 -0000
Received: (qmail 52858 invoked by uid 500); 12 Apr 2010 22:43:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52784 invoked by uid 500); 12 Apr 2010 22:43:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52768 invoked by uid 99); 12 Apr 2010 22:43:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 22:43:29 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.218.217] (HELO mail-bw0-f217.google.com) (209.85.218.217)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 22:43:22 +0000
Received: by bwz9 with SMTP id 9so5167283bwz.29
        for <multiple recipients>; Mon, 12 Apr 2010 15:43:02 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.70.77 with HTTP; Mon, 12 Apr 2010 15:43:02 -0700 (PDT)
In-Reply-To: <m2l7c962aed1004120856hddd9e41dt77f0ba3e259c9171@mail.gmail.com>
References: <m2l7c962aed1004120856hddd9e41dt77f0ba3e259c9171@mail.gmail.com>
Date: Mon, 12 Apr 2010 15:43:02 -0700
Received: by 10.204.34.201 with SMTP id m9mr5515907bkd.127.1271112182129; Mon, 
	12 Apr 2010 15:43:02 -0700 (PDT)
Message-ID: <y2pac79ea401004121543x82efe618l927aaea67388798c@mail.gmail.com>
Subject: Re: [VOTE] HBase as TLP?
From: Tom White <tom@cloudera.com>
To: hbase-dev@hadoop.apache.org
Cc: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Tom

On Mon, Apr 12, 2010 at 8:56 AM, Stack <stack@duboce.net> wrote:
> Please vote as to whether you think HBase should become a top-level
> Apache project.
>
> I've included below a draft board resolution. =A0It lists all current
> active HBase committers as initial members of the project management
> committee (PMC) and myself, Michael Stack, as the initial chair.
>
> Do the good people of Hadoop support sending this request on to the Hadoo=
p PMC?
>
> Yours,
> St.Ack
>
> ------------
>
> =A0 X. Establish the Apache HBase Project
>
> =A0 =A0 =A0WHEREAS, the Board of Directors deems it to be in the best
> =A0 =A0 =A0interests of the Foundation and consistent with the
> =A0 =A0 =A0Foundation's purpose to establish a Project Management
> =A0 =A0 =A0Committee charged with the creation and maintenance of
> =A0 =A0 =A0open-source software related to data serialization
> =A0 =A0 =A0for distribution at no charge to the public.
>
> =A0 =A0 =A0NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> =A0 =A0 =A0Committee (PMC), to be known as the "Apache HBase Project",
> =A0 =A0 =A0be and hereby is established pursuant to Bylaws of the
> =A0 =A0 =A0Foundation; and be it further
>
> =A0 =A0 =A0RESOLVED, that the Apache HBase Project be and hereby is
> =A0 =A0 =A0responsible for the creation and maintenance of software
> =A0 =A0 =A0related to a distributed database; and be it further
>
> =A0 =A0 =A0RESOLVED, that the office of "Vice President, Apache HBase" be
> =A0 =A0 =A0and hereby is created, the person holding such office to
> =A0 =A0 =A0serve at the direction of the Board of Directors as the chair
> =A0 =A0 =A0of the Apache HBase Project, and to have primary responsibilit=
y
> =A0 =A0 =A0for management of the projects within the scope of
> =A0 =A0 =A0responsibility of the Apache HBase Project; and be it further
>
> =A0 =A0 =A0RESOLVED, that the persons listed immediately below be and
> =A0 =A0 =A0hereby are appointed to serve as the initial members of the
> =A0 =A0 =A0Apache HBase Project:
>
> =A0 =A0 =A0 =A0* Michael Stack <stack@apache.org>
> =A0 =A0 =A0 =A0* Jonathan Gray <jgray@apache.org>
> =A0 =A0 =A0 =A0* Andrew Purtell <apurtell@apache.org>
> =A0 =A0 =A0 =A0* John-Daniel Cryans <jdcryans@apache.org>
> =A0 =A0 =A0 =A0* Ryan Rawson <rawson@apache.org>
> =A0 =A0 =A0 =A0* Lars George <larsgeorge@apache.org>
>
> =A0 =A0 =A0NOW, THEREFORE, BE IT FURTHER RESOLVED, that Michael Stack
> =A0 =A0 =A0be appointed to the office of Vice President, Apache HBase, to
> =A0 =A0 =A0serve in accordance with and subject to the direction of the
> =A0 =A0 =A0Board of Directors and the Bylaws of the Foundation until
> =A0 =A0 =A0death, resignation, retirement, removal or disqualification,
> =A0 =A0 =A0or until a successor is appointed; and be it further
>
> =A0 =A0 =A0RESOLVED, that the initial Apache HBase PMC be and hereby is
> =A0 =A0 =A0tasked with the creation of a set of bylaws intended to
> =A0 =A0 =A0encourage open development and increased participation in the
> =A0 =A0 =A0Apache HBase Project; and be it further
>
> =A0 =A0 =A0RESOLVED, that the Apache HBase Project be and hereby
> =A0 =A0 =A0is tasked with the migration and rationalization of the Apache
> =A0 =A0 =A0Hadoop HBase sub-project; and be it further
>
> =A0 =A0 =A0RESOLVED, that all responsibilities pertaining to the Apache
> =A0 =A0 =A0Hadoop HBase sub-project encumbered upon the
> =A0 =A0 =A0Apache Hadoop Project are hereafter discharged.
>

From general-return-1349-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 22:49:27 2010
Return-Path: <general-return-1349-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88161 invoked from network); 12 Apr 2010 22:49:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 22:49:27 -0000
Received: (qmail 58507 invoked by uid 500); 12 Apr 2010 22:49:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58476 invoked by uid 500); 12 Apr 2010 22:49:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58464 invoked by uid 99); 12 Apr 2010 22:49:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 22:49:26 +0000
X-ASF-Spam-Status: No, hits=-0.2 required=10.0
	tests=AWL,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 22:49:21 +0000
Received: from [10.72.76.34] ([10.72.76.34])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o3CMlUrZ037018;
	Mon, 12 Apr 2010 15:47:31 -0700 (PDT)
Message-ID: <4BC3A304.9050701@apache.org>
Date: Mon, 12 Apr 2010 15:47:32 -0700
From: Patrick Hunt <phunt@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] HBase as TLP?
References: <m2l7c962aed1004120856hddd9e41dt77f0ba3e259c9171@mail.gmail.com> <y2pac79ea401004121543x82efe618l927aaea67388798c@mail.gmail.com>
In-Reply-To: <y2pac79ea401004121543x82efe618l927aaea67388798c@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

+1

Patrick

On 04/12/2010 03:43 PM, Tom White wrote:
> +1
>
> Tom
>
> On Mon, Apr 12, 2010 at 8:56 AM, Stack<stack@duboce.net>  wrote:
>> Please vote as to whether you think HBase should become a top-level
>> Apache project.
>>
>> I've included below a draft board resolution.  It lists all current
>> active HBase committers as initial members of the project management
>> committee (PMC) and myself, Michael Stack, as the initial chair.
>>
>> Do the good people of Hadoop support sending this request on to the Hadoop PMC?
>>
>> Yours,
>> St.Ack
>>
>> ------------
>>
>>    X. Establish the Apache HBase Project
>>
>>       WHEREAS, the Board of Directors deems it to be in the best
>>       interests of the Foundation and consistent with the
>>       Foundation's purpose to establish a Project Management
>>       Committee charged with the creation and maintenance of
>>       open-source software related to data serialization
>>       for distribution at no charge to the public.
>>
>>       NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>>       Committee (PMC), to be known as the "Apache HBase Project",
>>       be and hereby is established pursuant to Bylaws of the
>>       Foundation; and be it further
>>
>>       RESOLVED, that the Apache HBase Project be and hereby is
>>       responsible for the creation and maintenance of software
>>       related to a distributed database; and be it further
>>
>>       RESOLVED, that the office of "Vice President, Apache HBase" be
>>       and hereby is created, the person holding such office to
>>       serve at the direction of the Board of Directors as the chair
>>       of the Apache HBase Project, and to have primary responsibility
>>       for management of the projects within the scope of
>>       responsibility of the Apache HBase Project; and be it further
>>
>>       RESOLVED, that the persons listed immediately below be and
>>       hereby are appointed to serve as the initial members of the
>>       Apache HBase Project:
>>
>>         * Michael Stack<stack@apache.org>
>>         * Jonathan Gray<jgray@apache.org>
>>         * Andrew Purtell<apurtell@apache.org>
>>         * John-Daniel Cryans<jdcryans@apache.org>
>>         * Ryan Rawson<rawson@apache.org>
>>         * Lars George<larsgeorge@apache.org>
>>
>>       NOW, THEREFORE, BE IT FURTHER RESOLVED, that Michael Stack
>>       be appointed to the office of Vice President, Apache HBase, to
>>       serve in accordance with and subject to the direction of the
>>       Board of Directors and the Bylaws of the Foundation until
>>       death, resignation, retirement, removal or disqualification,
>>       or until a successor is appointed; and be it further
>>
>>       RESOLVED, that the initial Apache HBase PMC be and hereby is
>>       tasked with the creation of a set of bylaws intended to
>>       encourage open development and increased participation in the
>>       Apache HBase Project; and be it further
>>
>>       RESOLVED, that the Apache HBase Project be and hereby
>>       is tasked with the migration and rationalization of the Apache
>>       Hadoop HBase sub-project; and be it further
>>
>>       RESOLVED, that all responsibilities pertaining to the Apache
>>       Hadoop HBase sub-project encumbered upon the
>>       Apache Hadoop Project are hereafter discharged.
>>

From general-return-1350-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 23:20:34 2010
Return-Path: <general-return-1350-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4866 invoked from network); 12 Apr 2010 23:20:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 23:20:34 -0000
Received: (qmail 88414 invoked by uid 500); 12 Apr 2010 23:20:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88375 invoked by uid 500); 12 Apr 2010 23:20:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88367 invoked by uid 99); 12 Apr 2010 23:20:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 23:20:33 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 23:20:25 +0000
Received: from wlanvpn-mc2e-246-192.corp.yahoo.com ([172.21.148.192])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o3CNIU5d002415
	for <general@hadoop.apache.org>; Mon, 12 Apr 2010 16:19:06 -0700 (PDT)
Message-Id: <CF6FB75B-924A-4CA8-90EA-3B599B082BBA@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <m2l7c962aed1004120856hddd9e41dt77f0ba3e259c9171@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] HBase as TLP?
Date: Mon, 12 Apr 2010 16:19:06 -0700
References: <m2l7c962aed1004120856hddd9e41dt77f0ba3e259c9171@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Arun

On Apr 12, 2010, at 8:56 AM, Stack wrote:

> Please vote as to whether you think HBase should become a top-level
> Apache project.
>
> I've included below a draft board resolution.  It lists all current
> active HBase committers as initial members of the project management
> committee (PMC) and myself, Michael Stack, as the initial chair.
>
> Do the good people of Hadoop support sending this request on to the  
> Hadoop PMC?
>
> Yours,
> St.Ack
>
> ------------
>
>   X. Establish the Apache HBase Project
>
>      WHEREAS, the Board of Directors deems it to be in the best
>      interests of the Foundation and consistent with the
>      Foundation's purpose to establish a Project Management
>      Committee charged with the creation and maintenance of
>      open-source software related to data serialization
>      for distribution at no charge to the public.
>
>      NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>      Committee (PMC), to be known as the "Apache HBase Project",
>      be and hereby is established pursuant to Bylaws of the
>      Foundation; and be it further
>
>      RESOLVED, that the Apache HBase Project be and hereby is
>      responsible for the creation and maintenance of software
>      related to a distributed database; and be it further
>
>      RESOLVED, that the office of "Vice President, Apache HBase" be
>      and hereby is created, the person holding such office to
>      serve at the direction of the Board of Directors as the chair
>      of the Apache HBase Project, and to have primary responsibility
>      for management of the projects within the scope of
>      responsibility of the Apache HBase Project; and be it further
>
>      RESOLVED, that the persons listed immediately below be and
>      hereby are appointed to serve as the initial members of the
>      Apache HBase Project:
>
>        * Michael Stack <stack@apache.org>
>        * Jonathan Gray <jgray@apache.org>
>        * Andrew Purtell <apurtell@apache.org>
>        * John-Daniel Cryans <jdcryans@apache.org>
>        * Ryan Rawson <rawson@apache.org>
>        * Lars George <larsgeorge@apache.org>
>
>      NOW, THEREFORE, BE IT FURTHER RESOLVED, that Michael Stack
>      be appointed to the office of Vice President, Apache HBase, to
>      serve in accordance with and subject to the direction of the
>      Board of Directors and the Bylaws of the Foundation until
>      death, resignation, retirement, removal or disqualification,
>      or until a successor is appointed; and be it further
>
>      RESOLVED, that the initial Apache HBase PMC be and hereby is
>      tasked with the creation of a set of bylaws intended to
>      encourage open development and increased participation in the
>      Apache HBase Project; and be it further
>
>      RESOLVED, that the Apache HBase Project be and hereby
>      is tasked with the migration and rationalization of the Apache
>      Hadoop HBase sub-project; and be it further
>
>      RESOLVED, that all responsibilities pertaining to the Apache
>      Hadoop HBase sub-project encumbered upon the
>      Apache Hadoop Project are hereafter discharged.


From general-return-1351-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 23:57:49 2010
Return-Path: <general-return-1351-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 32693 invoked from network); 12 Apr 2010 23:57:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 23:57:49 -0000
Received: (qmail 15739 invoked by uid 500); 12 Apr 2010 23:57:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15705 invoked by uid 500); 12 Apr 2010 23:57:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15697 invoked by uid 99); 12 Apr 2010 23:57:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 23:57:48 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jessesanford@gmail.com designates 209.85.222.189 as permitted sender)
Received: from [209.85.222.189] (HELO mail-pz0-f189.google.com) (209.85.222.189)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 23:57:42 +0000
Received: by pzk27 with SMTP id 27so5144433pzk.2
        for <general@hadoop.apache.org>; Mon, 12 Apr 2010 16:57:21 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:received:message-id:subject:from:to:content-type;
        bh=MSWJXMfafFJXdNkujq23g/qeAUNz7gdbKwLYb+WkyXk=;
        b=aXIn1scqjNN8cBwgxAfNIgmRA6tPgzQszHmqh3/t5QpTQDNkHVIzeBw+ZDn+X+SRjR
         tAR6Ty3ZN6Lri9V99vbfpi2nzmqkeJDHJ2istjR9P+tH//CD1H9/PsEeXUDIaz0v3v+4
         hb3AsmfFcxV2GdnqmV4zdBEXXa5fZCinWt9AY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=jGMYEzXvPkSlSJwOaJtaVdelVzSSFEdrT/q4QpI5KG0qpxpcETvHBFSey2kW3hp5/T
         XPde1FdgBq8bbhi5/A9qDmVZc3XNB27w7+Jvs1LxPmG8ywdeMpBs0J7qq57fP//3jHed
         aGjCHAtFXluwfgGLdzp9eQKgMvD8ECVkFQ69k=
MIME-Version: 1.0
Received: by 10.143.19.6 with HTTP; Mon, 12 Apr 2010 16:57:20 -0700 (PDT)
Received: by 10.143.19.6 with HTTP; Mon, 12 Apr 2010 16:57:20 -0700 (PDT)
In-Reply-To: <CF6FB75B-924A-4CA8-90EA-3B599B082BBA@yahoo-inc.com>
References: <m2l7c962aed1004120856hddd9e41dt77f0ba3e259c9171@mail.gmail.com>
	 <CF6FB75B-924A-4CA8-90EA-3B599B082BBA@yahoo-inc.com>
Date: Mon, 12 Apr 2010 19:57:20 -0400
Received: by 10.142.119.3 with SMTP id r3mr2058999wfc.145.1271116641009; Mon, 
	12 Apr 2010 16:57:21 -0700 (PDT)
Message-ID: <t2hb4354f221004121657nae7c16f6gfca8aea739bee617@mail.gmail.com>
Subject: Re: [VOTE] HBase as TLP?
From: Jesse Sanford <jessesanford@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636e0b6b9db01ad048412e7e4
X-Virus-Checked: Checked by ClamAV on apache.org

--001636e0b6b9db01ad048412e7e4
Content-Type: text/plain; charset=ISO-8859-1

+1

On Apr 12, 2010 7:20 PM, "Arun C Murthy" <acm@yahoo-inc.com> wrote:

+1

Arun



On Apr 12, 2010, at 8:56 AM, Stack wrote:

> Please vote as to whether you think HBase should beco...

--001636e0b6b9db01ad048412e7e4--

From general-return-1352-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 00:52:41 2010
Return-Path: <general-return-1352-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70281 invoked from network); 13 Apr 2010 00:52:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 00:52:41 -0000
Received: (qmail 48571 invoked by uid 500); 13 Apr 2010 00:52:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48485 invoked by uid 500); 13 Apr 2010 00:52:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48469 invoked by uid 99); 13 Apr 2010 00:52:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 00:52:40 +0000
X-ASF-Spam-Status: No, hits=1.0 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_SOFTFAIL
X-Spam-Check-By: apache.org
Received-SPF: softfail (athena.apache.org: transitioning domain of john@beforedawnsolutions.com does not designate 209.85.221.187 as permitted sender)
Received: from [209.85.221.187] (HELO mail-qy0-f187.google.com) (209.85.221.187)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 00:52:32 +0000
Received: by qyk17 with SMTP id 17so2270644qyk.9
        for <multiple recipients>; Mon, 12 Apr 2010 17:52:11 -0700 (PDT)
Received: by 10.229.217.196 with SMTP id hn4mr953899qcb.94.1271119930849;
        Mon, 12 Apr 2010 17:52:10 -0700 (PDT)
Received: from [192.168.1.115] (c-71-196-131-13.hsd1.co.comcast.net [71.196.131.13])
        by mx.google.com with ESMTPS id w30sm6801872qce.22.2010.04.12.17.52.09
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Mon, 12 Apr 2010 17:52:10 -0700 (PDT)
Subject: Re: [VOTE] HBase as TLP?
Mime-Version: 1.0 (Apple Message framework v1078)
Content-Type: text/plain; charset=us-ascii
From: John Martyniak <john@beforedawnsolutions.com>
In-Reply-To: <m2l7c962aed1004120856hddd9e41dt77f0ba3e259c9171@mail.gmail.com>
Date: Mon, 12 Apr 2010 20:52:06 -0400
Cc: hbase-dev@hadoop.apache.org
Content-Transfer-Encoding: quoted-printable
Message-Id: <AB29A3AD-E842-400B-A80F-1DA1F57374E0@beforedawnsolutions.com>
References: <m2l7c962aed1004120856hddd9e41dt77f0ba3e259c9171@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1078)

Can somebody tell me why this is a good idea?

Isn't HBase tightly coupled with Hadoop/HDFS.  Isn't that the benefit of =
it.

-John

On Apr 12, 2010, at 11:56 AM, Stack wrote:

> Please vote as to whether you think HBase should become a top-level
> Apache project.
>=20
> I've included below a draft board resolution.  It lists all current
> active HBase committers as initial members of the project management
> committee (PMC) and myself, Michael Stack, as the initial chair.
>=20
> Do the good people of Hadoop support sending this request on to the =
Hadoop PMC?
>=20
> Yours,
> St.Ack
>=20
> ------------
>=20
>   X. Establish the Apache HBase Project
>=20
>      WHEREAS, the Board of Directors deems it to be in the best
>      interests of the Foundation and consistent with the
>      Foundation's purpose to establish a Project Management
>      Committee charged with the creation and maintenance of
>      open-source software related to data serialization
>      for distribution at no charge to the public.
>=20
>      NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>      Committee (PMC), to be known as the "Apache HBase Project",
>      be and hereby is established pursuant to Bylaws of the
>      Foundation; and be it further
>=20
>      RESOLVED, that the Apache HBase Project be and hereby is
>      responsible for the creation and maintenance of software
>      related to a distributed database; and be it further
>=20
>      RESOLVED, that the office of "Vice President, Apache HBase" be
>      and hereby is created, the person holding such office to
>      serve at the direction of the Board of Directors as the chair
>      of the Apache HBase Project, and to have primary responsibility
>      for management of the projects within the scope of
>      responsibility of the Apache HBase Project; and be it further
>=20
>      RESOLVED, that the persons listed immediately below be and
>      hereby are appointed to serve as the initial members of the
>      Apache HBase Project:
>=20
>        * Michael Stack <stack@apache.org>
>        * Jonathan Gray <jgray@apache.org>
>        * Andrew Purtell <apurtell@apache.org>
>        * John-Daniel Cryans <jdcryans@apache.org>
>        * Ryan Rawson <rawson@apache.org>
>        * Lars George <larsgeorge@apache.org>
>=20
>      NOW, THEREFORE, BE IT FURTHER RESOLVED, that Michael Stack
>      be appointed to the office of Vice President, Apache HBase, to
>      serve in accordance with and subject to the direction of the
>      Board of Directors and the Bylaws of the Foundation until
>      death, resignation, retirement, removal or disqualification,
>      or until a successor is appointed; and be it further
>=20
>      RESOLVED, that the initial Apache HBase PMC be and hereby is
>      tasked with the creation of a set of bylaws intended to
>      encourage open development and increased participation in the
>      Apache HBase Project; and be it further
>=20
>      RESOLVED, that the Apache HBase Project be and hereby
>      is tasked with the migration and rationalization of the Apache
>      Hadoop HBase sub-project; and be it further
>=20
>      RESOLVED, that all responsibilities pertaining to the Apache
>      Hadoop HBase sub-project encumbered upon the
>      Apache Hadoop Project are hereafter discharged.


From general-return-1353-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 00:53:03 2010
Return-Path: <general-return-1353-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70536 invoked from network); 13 Apr 2010 00:53:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 00:53:03 -0000
Received: (qmail 50495 invoked by uid 500); 13 Apr 2010 00:53:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50449 invoked by uid 500); 13 Apr 2010 00:53:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50441 invoked by uid 99); 13 Apr 2010 00:53:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 00:53:02 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of boyuzhang35@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 00:52:55 +0000
Received: by wwb29 with SMTP id 29so1973202wwb.35
        for <general@hadoop.apache.org>; Mon, 12 Apr 2010 17:52:34 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type;
        bh=T0Dae1CGJwiUwj6diKdjwBRU4jUONagDo7hjGCwKF7k=;
        b=YBDCi+tqeoBwEs9cqYoqi4+Sw4JqZuwFX+ph3XuS6yzj4eM8DXASXmdy9G3nMTjZQv
         spdZX1VCyBLTQlhef/sVDRTEIvJwyLWGIuY1/+6+HcBstHE3JeXWagCPJIWC4fsazzL6
         iG7Ql5M0WmGibN4XS4lindq8GmEZN5nRO+Tcg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=XY0zD79DSy+9Cz5fq5cJ8eoVA6kZfStnqE/xAnG1oCMC+DCRjxU6iG2P7PtZWvInHR
         aePVSe9dCpGSXMJttbJbfokfZp3z2NzG3pBWBgVSuLkiH60FfZfhXTJrgpyX87eJaLKv
         WJiJPMwwjvgFLztjiTnR+FbkRLlZnRUbNZwMg=
MIME-Version: 1.0
Received: by 10.216.29.195 with HTTP; Mon, 12 Apr 2010 17:52:34 -0700 (PDT)
In-Reply-To: <m2r643a61251004060743ge24767e9n7038fe038c4f04de@mail.gmail.com>
References: <m2r643a61251004060743ge24767e9n7038fe038c4f04de@mail.gmail.com>
Date: Mon, 12 Apr 2010 20:52:34 -0400
Received: by 10.216.169.84 with SMTP id m62mr2578361wel.130.1271119954660; 
	Mon, 12 Apr 2010 17:52:34 -0700 (PDT)
Message-ID: <s2v873f8eb61004121752vc05a79f3qa2a25727adafe40a@mail.gmail.com>
Subject: Re: hadoop on demand setup: Failed to retrieve 'hdfs' service address
From: Boyu Zhang <boyuzhang35@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e65b60a05d4f59048413add2
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e65b60a05d4f59048413add2
Content-Type: text/plain; charset=ISO-8859-1

Hi Kevin,

Sorry to bother again, I am wondering in order to get HOD to work, do we
need to install all the prerequisite software like passwordless ssh? Thanks
a lot!

Boyu

On Tue, Apr 6, 2010 at 10:43 AM, Kevin Van Workum <vanw@sabalcore.com>wrote:

> Hello,
>
> I'm trying to setup hadoop on demand (HOD) on my cluster. I'm
> currently unable to "allocate cluster". I'm starting hod with the
> following command:
>
> /usr/local/hadoop-0.20.2/hod/bin/hod -c
> /usr/local/hadoop-0.20.2/hod/conf/hodrc -t
> /b/01/vanw/hod/hadoop-0.20.2.tar.gz -o "allocate ~/hod 3"
> --ringmaster.log-dir=/tmp -b 4
>
> The job starts on the nodes and I see the ringmaster running on the
> MotherSuperior. The ringmaster-main.log file is created, but is empty.
> I don't see any associated processes running on the other 2 nodes in
> the job.
>
> The critical errors are as follows:
>
> [2010-04-06 10:34:13,630] CRITICAL/50 hadoop:298 - Failed to retrieve
> 'hdfs' service address.
> [2010-04-06 10:34:13,631] DEBUG/10 hadoop:631 - Cleaning up cluster id
> 238366.jman, as cluster could not be allocated.
> [2010-04-06 10:34:13,632] DEBUG/10 hadoop:635 - Calling rm.stop()
> [2010-04-06 10:34:13,639] DEBUG/10 hadoop:637 - Returning from rm.stop()
> [2010-04-06 10:34:13,639] CRITICAL/50 hod:401 - Cannot allocate
> cluster /b/01/vanw/hod
> [2010-04-06 10:34:14,149] DEBUG/10 hod:597 - return code: 7
>
> The contents of the hodrc file is:
>
> [hod]
> stream                          = True
> java-home                       = /usr/local/jdk1.6.0_02
> cluster                         = orange
> cluster-factor                  = 1.8
> xrs-port-range                  = 32768-65536
> debug                           = 4
> allocate-wait-time              = 3600
> temp-dir                        = /tmp/hod
>
> [ringmaster]
> register                        = True
> stream                          = False
> temp-dir                        = /tmp/hod
> http-port-range                 = 8000-9000
> work-dirs                       = /tmp/hod/1,/tmp/hod/2
> xrs-port-range                  = 32768-65536
> debug                           = 4
>
> [hodring]
> stream                          = False
> temp-dir                        = /tmp/hod
> register                        = True
> java-home                       = /usr/local/jdk1.6.0_02
> http-port-range                 = 8000-9000
> xrs-port-range                  = 32768-65536
> debug                           = 4
>
> [resource_manager]
> queue                           = dque
> batch-home                      = /usr/local/torque-2.3.7
> id                              = torque
> env-vars                       =
> HOD_PYTHON_HOME=/usr/local/python-2.5.5/bin/python
>
> [gridservice-mapred]
> external                        = False
> pkgs                            = /usr/local/hadoop-0.20.2
> tracker_port                    = 8030
> info_port                       = 50080
>
> [gridservice-hdfs]
> external                        = False
> pkgs                            = /usr/local/hadoop-0.20.2
> fs_port                         = 8020
> info_port                       = 50070
>
>
> Some other useful information:
> Linux 2.6.18-128.7.1.el5
> Python 2.5.5
> Twisted 10.0.0
> zope 3.3.0
> java version "1.6.0_02"
>
> --
> Kevin Van Workum, PhD
> Sabalcore Computing Inc.
> Run your code on 500 processors.
> Sign up for a free trial account.
> www.sabalcore.com
> 877-492-8027 ext. 11
>

--0016e65b60a05d4f59048413add2--

From general-return-1354-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 01:03:14 2010
Return-Path: <general-return-1354-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76629 invoked from network); 13 Apr 2010 01:03:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 01:03:13 -0000
Received: (qmail 61216 invoked by uid 500); 13 Apr 2010 01:03:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61170 invoked by uid 500); 13 Apr 2010 01:03:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61156 invoked by uid 99); 13 Apr 2010 01:03:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 01:03:12 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 01:03:06 +0000
Received: from pugholehand-lm.corp.yahoo.com (pugholehand-lm.corp.yahoo.com [10.72.115.44])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o3D11Ate055011;
	Mon, 12 Apr 2010 18:01:10 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=cc:message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=UeLnJMC6/KvNGALvv8/yfJU+Wgdp1wPFDcUn2LZ5V2VWIENkSbD5gUeyrKAn/MvL
Cc: <hbase-dev@hadoop.apache.org>
Message-Id: <E96C84F7-53C0-424C-96D7-C832704C5452@yahoo-inc.com>
From: Alan Gates <gates@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <AB29A3AD-E842-400B-A80F-1DA1F57374E0@beforedawnsolutions.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] HBase as TLP?
Date: Mon, 12 Apr 2010 18:01:10 -0700
References: <m2l7c962aed1004120856hddd9e41dt77f0ba3e259c9171@mail.gmail.com> <AB29A3AD-E842-400B-A80F-1DA1F57374E0@beforedawnsolutions.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

For a discussion of this exact questions see:

http://mail-archives.apache.org/mod_mbox/hadoop-general/201004.mbox/%3cx2l7c962aed1004072102j3ca29379he9f4095be7736f53@mail.gmail.com%3e

Alan.

On Apr 12, 2010, at 5:52 PM, John Martyniak wrote:

> Can somebody tell me why this is a good idea?
>
> Isn't HBase tightly coupled with Hadoop/HDFS.  Isn't that the  
> benefit of it.
>
> -John
>
> On Apr 12, 2010, at 11:56 AM, Stack wrote:
>
>> Please vote as to whether you think HBase should become a top-level
>> Apache project.
>>
>> I've included below a draft board resolution.  It lists all current
>> active HBase committers as initial members of the project management
>> committee (PMC) and myself, Michael Stack, as the initial chair.
>>
>> Do the good people of Hadoop support sending this request on to the  
>> Hadoop PMC?
>>
>> Yours,
>> St.Ack
>>
>> ------------
>>
>>  X. Establish the Apache HBase Project
>>
>>     WHEREAS, the Board of Directors deems it to be in the best
>>     interests of the Foundation and consistent with the
>>     Foundation's purpose to establish a Project Management
>>     Committee charged with the creation and maintenance of
>>     open-source software related to data serialization
>>     for distribution at no charge to the public.
>>
>>     NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>>     Committee (PMC), to be known as the "Apache HBase Project",
>>     be and hereby is established pursuant to Bylaws of the
>>     Foundation; and be it further
>>
>>     RESOLVED, that the Apache HBase Project be and hereby is
>>     responsible for the creation and maintenance of software
>>     related to a distributed database; and be it further
>>
>>     RESOLVED, that the office of "Vice President, Apache HBase" be
>>     and hereby is created, the person holding such office to
>>     serve at the direction of the Board of Directors as the chair
>>     of the Apache HBase Project, and to have primary responsibility
>>     for management of the projects within the scope of
>>     responsibility of the Apache HBase Project; and be it further
>>
>>     RESOLVED, that the persons listed immediately below be and
>>     hereby are appointed to serve as the initial members of the
>>     Apache HBase Project:
>>
>>       * Michael Stack <stack@apache.org>
>>       * Jonathan Gray <jgray@apache.org>
>>       * Andrew Purtell <apurtell@apache.org>
>>       * John-Daniel Cryans <jdcryans@apache.org>
>>       * Ryan Rawson <rawson@apache.org>
>>       * Lars George <larsgeorge@apache.org>
>>
>>     NOW, THEREFORE, BE IT FURTHER RESOLVED, that Michael Stack
>>     be appointed to the office of Vice President, Apache HBase, to
>>     serve in accordance with and subject to the direction of the
>>     Board of Directors and the Bylaws of the Foundation until
>>     death, resignation, retirement, removal or disqualification,
>>     or until a successor is appointed; and be it further
>>
>>     RESOLVED, that the initial Apache HBase PMC be and hereby is
>>     tasked with the creation of a set of bylaws intended to
>>     encourage open development and increased participation in the
>>     Apache HBase Project; and be it further
>>
>>     RESOLVED, that the Apache HBase Project be and hereby
>>     is tasked with the migration and rationalization of the Apache
>>     Hadoop HBase sub-project; and be it further
>>
>>     RESOLVED, that all responsibilities pertaining to the Apache
>>     Hadoop HBase sub-project encumbered upon the
>>     Apache Hadoop Project are hereafter discharged.
>


From general-return-1355-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 01:18:38 2010
Return-Path: <general-return-1355-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85694 invoked from network); 13 Apr 2010 01:18:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 01:18:38 -0000
Received: (qmail 81128 invoked by uid 500); 13 Apr 2010 01:18:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81075 invoked by uid 500); 13 Apr 2010 01:18:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81059 invoked by uid 99); 13 Apr 2010 01:18:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 01:18:36 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 01:18:29 +0000
Received: by vws10 with SMTP id 10so757522vws.35
        for <multiple recipients>; Mon, 12 Apr 2010 18:18:07 -0700 (PDT)
Received: by 10.220.62.141 with SMTP id x13mr2612968vch.104.1271121487672;
        Mon, 12 Apr 2010 18:18:07 -0700 (PDT)
Received: from [10.0.0.106] ([38.104.141.102])
        by mx.google.com with ESMTPS id 31sm111843420vws.7.2010.04.12.18.18.06
        (version=SSLv3 cipher=RC4-MD5);
        Mon, 12 Apr 2010 18:18:06 -0700 (PDT)
Message-ID: <4BC3C64E.8090805@cloudera.com>
Date: Mon, 12 Apr 2010 18:18:06 -0700
From: Amr Awadallah <aaa@cloudera.com>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4 ThunderBrowse/3.2.8.1
MIME-Version: 1.0
To: general@hadoop.apache.org
CC: Tom White <tom@cloudera.com>, hbase-dev@hadoop.apache.org
Subject: Re: [VOTE] HBase as TLP?
References: <m2l7c962aed1004120856hddd9e41dt77f0ba3e259c9171@mail.gmail.com> <y2pac79ea401004121543x82efe618l927aaea67388798c@mail.gmail.com>
In-Reply-To: <y2pac79ea401004121543x82efe618l927aaea67388798c@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

+1

From general-return-1356-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 02:53:03 2010
Return-Path: <general-return-1356-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 11599 invoked from network); 13 Apr 2010 02:53:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 02:53:03 -0000
Received: (qmail 26943 invoked by uid 500); 13 Apr 2010 02:53:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26862 invoked by uid 500); 13 Apr 2010 02:53:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26854 invoked by uid 99); 13 Apr 2010 02:53:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 02:53:01 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.217.225] (HELO mail-gx0-f225.google.com) (209.85.217.225)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 02:52:55 +0000
Received: by gxk25 with SMTP id 25so1572489gxk.11
        for <general@hadoop.apache.org>; Mon, 12 Apr 2010 19:52:21 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.100.196.13 with HTTP; Mon, 12 Apr 2010 19:52:21 -0700 (PDT)
In-Reply-To: <s2v873f8eb61004121752vc05a79f3qa2a25727adafe40a@mail.gmail.com>
References: <m2r643a61251004060743ge24767e9n7038fe038c4f04de@mail.gmail.com>
	 <s2v873f8eb61004121752vc05a79f3qa2a25727adafe40a@mail.gmail.com>
Date: Mon, 12 Apr 2010 22:52:21 -0400
Received: by 10.101.150.1 with SMTP id c1mr8871371ano.112.1271127141125; Mon, 
	12 Apr 2010 19:52:21 -0700 (PDT)
Message-ID: <y2w643a61251004121952nb64aad34wf982d058d7c8ab4a@mail.gmail.com>
Subject: Re: hadoop on demand setup: Failed to retrieve 'hdfs' service address
From: Kevin Van Workum <vanw@sabalcore.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Mon, Apr 12, 2010 at 8:52 PM, Boyu Zhang <boyuzhang35@gmail.com> wrote:
> Hi Kevin,
>
> Sorry to bother again, I am wondering in order to get HOD to work, do we
> need to install all the prerequisite software like passwordless ssh? Than=
ks
> a lot!

SSH is not needed for HOD, it uses pbsdsh to launch processes on the nodes.

HOD seems to be very sensitive about the python version: 2.4 and 2.6
don't work, you need 2.5.

HOD is a little more flexible with Java, 1.5 and 1.6 seem to both work
for me. Also, the most recent versions of Twisted and zope seem to be
fine.

>
> Boyu
>
> On Tue, Apr 6, 2010 at 10:43 AM, Kevin Van Workum <vanw@sabalcore.com>wro=
te:
>
>> Hello,
>>
>> I'm trying to setup hadoop on demand (HOD) on my cluster. I'm
>> currently unable to "allocate cluster". I'm starting hod with the
>> following command:
>>
>> /usr/local/hadoop-0.20.2/hod/bin/hod -c
>> /usr/local/hadoop-0.20.2/hod/conf/hodrc -t
>> /b/01/vanw/hod/hadoop-0.20.2.tar.gz -o "allocate ~/hod 3"
>> --ringmaster.log-dir=3D/tmp -b 4
>>
>> The job starts on the nodes and I see the ringmaster running on the
>> MotherSuperior. The ringmaster-main.log file is created, but is empty.
>> I don't see any associated processes running on the other 2 nodes in
>> the job.
>>
>> The critical errors are as follows:
>>
>> [2010-04-06 10:34:13,630] CRITICAL/50 hadoop:298 - Failed to retrieve
>> 'hdfs' service address.
>> [2010-04-06 10:34:13,631] DEBUG/10 hadoop:631 - Cleaning up cluster id
>> 238366.jman, as cluster could not be allocated.
>> [2010-04-06 10:34:13,632] DEBUG/10 hadoop:635 - Calling rm.stop()
>> [2010-04-06 10:34:13,639] DEBUG/10 hadoop:637 - Returning from rm.stop()
>> [2010-04-06 10:34:13,639] CRITICAL/50 hod:401 - Cannot allocate
>> cluster /b/01/vanw/hod
>> [2010-04-06 10:34:14,149] DEBUG/10 hod:597 - return code: 7
>>
>> The contents of the hodrc file is:
>>
>> [hod]
>> stream =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D True
>> java-home =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D /usr/local/jdk=
1.6.0_02
>> cluster =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D orange
>> cluster-factor =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D 1.8
>> xrs-port-range =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D 32768-65536
>> debug =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D 4
>> allocate-wait-time =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D 3600
>> temp-dir =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D /tmp/hod
>>
>> [ringmaster]
>> register =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D True
>> stream =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D False
>> temp-dir =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D /tmp/hod
>> http-port-range =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D 8000-9000
>> work-dirs =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D /tmp/hod/1,/tm=
p/hod/2
>> xrs-port-range =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D 32768-65536
>> debug =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D 4
>>
>> [hodring]
>> stream =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D False
>> temp-dir =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D /tmp/hod
>> register =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D True
>> java-home =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D /usr/local/jdk=
1.6.0_02
>> http-port-range =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D 8000-9000
>> xrs-port-range =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D 32768-65536
>> debug =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D 4
>>
>> [resource_manager]
>> queue =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D dque
>> batch-home =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D /usr/local/tor=
que-2.3.7
>> id =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D torque
>> env-vars =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D
>> HOD_PYTHON_HOME=3D/usr/local/python-2.5.5/bin/python
>>
>> [gridservice-mapred]
>> external =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D False
>> pkgs =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D /usr/loc=
al/hadoop-0.20.2
>> tracker_port =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D 8030
>> info_port =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D 50080
>>
>> [gridservice-hdfs]
>> external =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D False
>> pkgs =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=3D /usr/loc=
al/hadoop-0.20.2
>> fs_port =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D 8020
>> info_port =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D 50070
>>
>>
>> Some other useful information:
>> Linux 2.6.18-128.7.1.el5
>> Python 2.5.5
>> Twisted 10.0.0
>> zope 3.3.0
>> java version "1.6.0_02"
>>
>> --
>> Kevin Van Workum, PhD
>> Sabalcore Computing Inc.
>> Run your code on 500 processors.
>> Sign up for a free trial account.
>> www.sabalcore.com
>> 877-492-8027 ext. 11
>>
>



--=20
Kevin Van Workum, PhD
Sabalcore Computing Inc.
Run your code on 500 processors.
Sign up for a free trial account.
www.sabalcore.com
877-492-8027 ext. 11

From general-return-1357-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 04:56:16 2010
Return-Path: <general-return-1357-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 26637 invoked from network); 13 Apr 2010 04:56:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 04:56:15 -0000
Received: (qmail 87400 invoked by uid 500); 13 Apr 2010 04:56:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87355 invoked by uid 500); 13 Apr 2010 04:56:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87347 invoked by uid 99); 13 Apr 2010 04:56:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 04:56:14 +0000
X-ASF-Spam-Status: No, hits=-2.1 required=10.0
	tests=AWL,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of magawake@gmail.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 04:56:07 +0000
Received: by gyf1 with SMTP id 1so3177377gyf.35
        for <general@hadoop.apache.org>; Mon, 12 Apr 2010 21:55:47 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:received:message-id
         :subject:from:to:content-type;
        bh=+qfKPN0z0TTzz0LK0qNAjP8Hcv//brotrDRFMW060DQ=;
        b=I/PyRatR7NQlUNtTXRi8h9kKv2Qn8oJBR/qNtorywzKBFDSDh0ACAo2Dx4egJCZp9i
         FKZuxrUuD+iKFVHbTkhKRerLfW59twTVBuDzHTZ1CaN7BEorylPr7+cqq/LSGpC0+TXH
         c4GesrrmAyFFId501WDBzeRV+ymuUMeUePm38=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=uDVACgOjBMK4qXPPiGy7mqreN893ZNYxkvx1/Yb6yGdgWSdKURhz+MQcSknykEW0Mc
         e8p6thTxMA1BLeXUkopcNkTLlFs13GVzMS/PacElxa4qF1ls3RePGbVsfdlYO55GmwaT
         HokO20vl1BmGclI7KC4NoeY+KpI0NJBmxIaco=
MIME-Version: 1.0
Received: by 10.231.191.130 with HTTP; Mon, 12 Apr 2010 21:55:46 -0700 (PDT)
Date: Tue, 13 Apr 2010 00:55:46 -0400
Received: by 10.101.211.36 with SMTP id n36mr8987706anq.120.1271134546998; 
	Mon, 12 Apr 2010 21:55:46 -0700 (PDT)
Message-ID: <g2o1cbd6f831004122155qd6ac83cex23572f90e83161d7@mail.gmail.com>
Subject: backing up namenode
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8

I have 1 name node and I would like to back it up. What is the correct
procedure to do this? Does the filesystem have to stay idle when the
backup occurs? I have seen some searches about rsync, but I haven't
seen any examples. Can someone please provide more information about
this?

TIA

From general-return-1358-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 04:56:40 2010
Return-Path: <general-return-1358-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 26726 invoked from network); 13 Apr 2010 04:56:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 04:56:39 -0000
Received: (qmail 88394 invoked by uid 500); 13 Apr 2010 04:56:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88356 invoked by uid 500); 13 Apr 2010 04:56:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88346 invoked by uid 99); 13 Apr 2010 04:56:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 04:56:38 +0000
X-ASF-Spam-Status: No, hits=-2.0 required=10.0
	tests=AWL,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of magawake@gmail.com designates 209.85.223.197 as permitted sender)
Received: from [209.85.223.197] (HELO mail-iw0-f197.google.com) (209.85.223.197)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 04:56:33 +0000
Received: by iwn35 with SMTP id 35so1669029iwn.21
        for <general@hadoop.apache.org>; Mon, 12 Apr 2010 21:56:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=wRyt+Lm54ZNzSWDX9ijRUB6OZNYdfO/sswVUUFzBYWc=;
        b=MHv/tfU6Wqim6jQLaz4IWtUcfOGQ2cF+Z411DqaMWGKJ37gt/IgokQklwxcRr799LC
         68PJ0ASx51vBAGGlj8MAVsZ9AY0qohBh0ddar6V1tLKnLik7uGvxg+3sBsmaPVGG90r+
         9mpGUDZ5l0VczHBa/yuxskbw/7+L5GSp1WfGo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=MY8HTA5omOeo1oeBa1YfQAHdOTW4v0UyIBNtO0B/sjawh/kFTWhdv32wIXEP7vWNQK
         lwCohMCPujlaWjGX57jMoq9okYqiCFg2m0LsPOInL7kkFVLWInk8Mq1u/8ZycME/YQFD
         UWGu5fNv5+sQ2K6X+sqOBkbOFHNvt2MT/qqa4=
MIME-Version: 1.0
Received: by 10.231.191.130 with HTTP; Mon, 12 Apr 2010 21:56:12 -0700 (PDT)
In-Reply-To: <732D302F-02DD-4D45-8023-6484445CB5E9@linkedin.com>
References: <h2y1cbd6f831004100733xf3ab6298u278df45f8d80d3ad@mail.gmail.com>
	 <EBFEB403-3709-44E2-A930-11A07A753AE5@linkedin.com>
	 <q2pc9118d4c1004111828va42fc159s1db13fbbb10aca9f@mail.gmail.com>
	 <732D302F-02DD-4D45-8023-6484445CB5E9@linkedin.com>
Date: Tue, 13 Apr 2010 00:56:12 -0400
Received: by 10.231.152.75 with SMTP id f11mr2377723ibw.50.1271134572771; Mon, 
	12 Apr 2010 21:56:12 -0700 (PDT)
Message-ID: <j2n1cbd6f831004122156r282cd3ecte36734e3b1a1aab3@mail.gmail.com>
Subject: Re: rack awareness question
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Thanks again. You have been very helpful...

On Mon, Apr 12, 2010 at 2:21 PM, Allen Wittenauer
<awittenauer@linkedin.com> wrote:
>
> On Apr 11, 2010, at 6:28 PM, Jeff Hammerbacher wrote:
>
>>> No. =C2=A0There is a jira i filed on turning the cache into an actual c=
ache
>>> w/TTLs, etc.
>>
>>
>> https://issues.apache.org/jira/browse/HDFS-870, for those who like
>> hyperlinks.
>
> Thank you for supporting my laziness. ;)
>
>

From general-return-1359-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 11:36:23 2010
Return-Path: <general-return-1359-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 844 invoked from network); 13 Apr 2010 11:36:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 11:36:23 -0000
Received: (qmail 53940 invoked by uid 500); 13 Apr 2010 11:36:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53753 invoked by uid 500); 13 Apr 2010 11:36:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53724 invoked by uid 99); 13 Apr 2010 11:36:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 11:36:20 +0000
X-ASF-Spam-Status: No, hits=-0.1 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [65.115.85.73] (HELO smtp-outbound-2.vmware.com) (65.115.85.73)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 11:36:12 +0000
Received: from mailhost3.vmware.com (mailhost3.vmware.com [10.16.27.45])
	by smtp-outbound-2.vmware.com (Postfix) with ESMTP id 623C44609E
	for <general@hadoop.apache.org>; Tue, 13 Apr 2010 04:35:52 -0700 (PDT)
Received: from pa-exht01.vmware.com (pa-exht01.vmware.com [10.113.81.167])
	by mailhost3.vmware.com (Postfix) with ESMTP id 5C69ACD91E
	for <general@hadoop.apache.org>; Tue, 13 Apr 2010 04:35:52 -0700 (PDT)
Received: from EXCH-MBX-22.vmware.com ([10.113.81.177]) by
 pa-exht01.vmware.com ([10.113.81.167]) with mapi; Tue, 13 Apr 2010 04:35:52
 -0700
From: Zhanlei Ma <zma@vmware.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 13 Apr 2010 04:35:45 -0700
Subject: how to do if jobtracker failed
Thread-Topic: how to do if jobtracker failed
Thread-Index: Acra/XR8Gy1qOG/iSbSK6+CjrdRtuQ==
Message-ID: <F21534570FE52940A8A9366D82C11E8B044EBFE351@EXCH-MBX-22.vmware.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-cr-hashedpuzzle: AFBx BHxj BLD2 BTE7 DyMC EP0n ERnT EcJH HGfr H8rW IH9u
 KA+M LQPS L7V1 L9hX
 MDgw;1;ZwBlAG4AZQByAGEAbABAAGgAYQBkAG8AbwBwAC4AYQBwAGEAYwBoAGUALgBvAHIAZwA=;Sosha1_v1;7;{64934E66-41EA-4552-982E-8B437267A911};egBtAGEAQAB2AG0AdwBhAHIAZQAuAGMAbwBtAA==;Tue,
 13 Apr 2010 11:35:45
 GMT;aABvAHcAIAB0AG8AIABkAG8AIABpAGYAIABqAG8AYgB0AHIAYQBjAGsAZQByACAAZgBhAGkAbABlAGQA
x-cr-puzzleid: {64934E66-41EA-4552-982E-8B437267A911}
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_F21534570FE52940A8A9366D82C11E8B044EBFE351EXCHMBX22vmwa_"
MIME-Version: 1.0

--_000_F21534570FE52940A8A9366D82C11E8B044EBFE351EXCHMBX22vmwa_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi all:
When running job, how will hadoopp do if the jobtracker failed? Looking for=
ward your answer!
Thanks
                                                                           =
                                              Zhanlei Ma

--_000_F21534570FE52940A8A9366D82C11E8B044EBFE351EXCHMBX22vmwa_--

From general-return-1360-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 11:41:53 2010
Return-Path: <general-return-1360-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1509 invoked from network); 13 Apr 2010 11:41:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 11:41:53 -0000
Received: (qmail 60404 invoked by uid 500); 13 Apr 2010 11:41:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60313 invoked by uid 500); 13 Apr 2010 11:41:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60305 invoked by uid 99); 13 Apr 2010 11:41:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 11:41:51 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [216.252.110.213] (HELO web56204.mail.re3.yahoo.com) (216.252.110.213)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 13 Apr 2010 11:41:43 +0000
Received: (qmail 19975 invoked by uid 60001); 13 Apr 2010 11:41:21 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1271158881; bh=V4J+Vyw7l5lquCFXZRal0jLgkDERnpMR59B1khRXfRI=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=tPL6JWx4ln+sd8KkXRDlXtH6iK/E4MEERKaMKHRaAkp/qBG2MBQo/wDYgHsZhyGL4RZdO2aMzdYI0sGDh35XDvxR6keqrjTeI/5btF4f+SoRwMuIio0TXBovTWT1F962gYkqNdgHqqCENRW92gbR8ehBD9mBDWBOr93/h4liyIE=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=uEXdKH4rhCAIUqXFi7r282uSsjDphbDc01RUy88tAhePiHCTAMrIDCSN5C95mf1uTFHt8tbbyhkujEcXKMe1a2Yi4OM5h4oqrVApd5TWg8fhUXg04yQj4Yiu1NJwjjxBZIEEtTSWlgX8D27eWVKdlDkpq7UchmQANflkWWhYFNE=;
Message-ID: <601540.54306.qm@web56204.mail.re3.yahoo.com>
X-YMail-OSG: WUcVm7kVM1l9Hcef1_NAK11x1XR386ZC3xWBFSKneR3kMF0
 zdzaYHCaUa0pgP164q8oePC6mVYytHMzz4pVOZefinfLxWAc04voeoOw9tvt
 ag5YaQXuVDPwBeTWd.T3_LTvw6Z9w_k35dylFRs5dk7cWlbijkwE6C91cWos
 8WXPzPC91Jxtj2X1AiDzm6x.7m2b_KTdKOdtmIQSwjQ.xEzKplTrRaM.IuD.
 f35HJA1blglqLW93yznQQVWDy3yvJyNR.h1Rm_aZaq_zxI0XCAxGDMix9nLc
 Vs9Bu1SMd52mRDVahESOhEkA1sk1TE4ZzE0NCMH5NZPMgRzuPBrYMh_F0U4G
 5dR.eXGGzCkvUR8_dxth6xyoij8JoFw--
Received: from [98.207.82.20] by web56204.mail.re3.yahoo.com via HTTP; Tue, 13 Apr 2010 04:41:16 PDT
X-Mailer: YahooMailRC/348.3 YahooMailWebService/0.8.100.260964
References: <m2l7c962aed1004120856hddd9e41dt77f0ba3e259c9171@mail.gmail.com>
Date: Tue, 13 Apr 2010 04:41:16 -0700 (PDT)
From: "Tsz Wo \(Nicholas\), Sze" <s29752-hadoopgeneral@yahoo.com>
Subject: Re: [VOTE] HBase as TLP?
To: general@hadoop.apache.org, hbase-dev@hadoop.apache.org
In-Reply-To: <m2l7c962aed1004120856hddd9e41dt77f0ba3e259c9171@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org

+1
Nicholas Sze




----- Original Message ----
> From: Stack <stack@duboce.net>
> To: general@hadoop.apache.org; hbase-dev@hadoop.apache.org
> Sent: Mon, April 12, 2010 8:56:51 AM
> Subject: [VOTE] HBase as TLP?
> 
> Please vote as to whether you think HBase should become a top-level
Apache 
> project.

I've included below a draft board resolution.  It lists all 
> current
active HBase committers as initial members of the project 
> management
committee (PMC) and myself, Michael Stack, as the initial 
> chair.

Do the good people of Hadoop support sending this request on to 
> the Hadoop PMC?

Yours,
St.Ack

------------

   X. 
> Establish the Apache HBase Project

      WHEREAS, the 
> Board of Directors deems it to be in the best
      interests 
> of the Foundation and consistent with the
      Foundation's 
> purpose to establish a Project Management
      Committee 
> charged with the creation and maintenance of
      open-source 
> software related to data serialization
      for distribution 
> at no charge to the public.

      NOW, THEREFORE, BE IT 
> RESOLVED, that a Project Management
      Committee (PMC), to 
> be known as the "Apache HBase Project",
      be and hereby is 
> established pursuant to Bylaws of the
      Foundation; and be 
> it further

      RESOLVED, that the Apache HBase Project 
> be and hereby is
      responsible for the creation and 
> maintenance of software
      related to a distributed 
> database; and be it further

      RESOLVED, that the 
> office of "Vice President, Apache HBase" be
      and hereby 
> is created, the person holding such office to
      serve at 
> the direction of the Board of Directors as the chair
      of 
> the Apache HBase Project, and to have primary responsibility
    
>   for management of the projects within the scope of
    
>   responsibility of the Apache HBase Project; and be it 
> further

      RESOLVED, that the persons listed 
> immediately below be and
      hereby are appointed to serve 
> as the initial members of the
      Apache HBase 
> Project:

        * Michael Stack <
> ymailto="mailto:stack@apache.org" 
> href="mailto:stack@apache.org">stack@apache.org>
      
>   * Jonathan Gray <
> href="mailto:jgray@apache.org">jgray@apache.org>
      
>   * Andrew Purtell <
> href="mailto:apurtell@apache.org">apurtell@apache.org>
    
>     * John-Daniel Cryans <
> href="mailto:jdcryans@apache.org">jdcryans@apache.org>
    
>     * Ryan Rawson <
> href="mailto:rawson@apache.org">rawson@apache.org>
    
>     * Lars George <
> href="mailto:larsgeorge@apache.org">larsgeorge@apache.org>

  
>     NOW, THEREFORE, BE IT FURTHER RESOLVED, that Michael 
> Stack
      be appointed to the office of Vice President, 
> Apache HBase, to
      serve in accordance with and subject to 
> the direction of the
      Board of Directors and the Bylaws 
> of the Foundation until
      death, resignation, retirement, 
> removal or disqualification,
      or until a successor is 
> appointed; and be it further

      RESOLVED, that the 
> initial Apache HBase PMC be and hereby is
      tasked with 
> the creation of a set of bylaws intended to
      encourage 
> open development and increased participation in the
      
> Apache HBase Project; and be it further

      RESOLVED, 
> that the Apache HBase Project be and hereby
      is tasked 
> with the migration and rationalization of the Apache
      
> Hadoop HBase sub-project; and be it further

      
> RESOLVED, that all responsibilities pertaining to the Apache
    
>   Hadoop HBase sub-project encumbered upon the
      
> Apache Hadoop Project are hereafter discharged.

From general-return-1361-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 12:30:31 2010
Return-Path: <general-return-1361-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22157 invoked from network); 13 Apr 2010 12:30:31 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 12:30:31 -0000
Received: (qmail 43540 invoked by uid 500); 13 Apr 2010 12:30:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 43468 invoked by uid 500); 13 Apr 2010 12:30:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 43453 invoked by uid 99); 13 Apr 2010 12:30:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 12:30:29 +0000
X-ASF-Spam-Status: No, hits=-1.1 required=10.0
	tests=AWL,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of imyousuf@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 12:30:24 +0000
Received: by vws10 with SMTP id 10so954761vws.35
        for <multiple recipients>; Tue, 13 Apr 2010 05:30:03 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:cc:content-type
         :content-transfer-encoding;
        bh=mBbmazcd+OlETo3cnmV4U9snBIV/NDY5nwFZm6yFgWE=;
        b=ep/hV3YlAQv1YTS/Q+X7ElknsKELt0CEomNh3SY9lH41jNFpf1PP85SKegpO5U6woQ
         RmzXeR+XoRkq8RKqIn5IpLgQIxo7REdD9vNHpXSM5WCzAOrMwHCRt9zjHiw9IvpoLptF
         97tVbY69mjJWLJ9D8Hapvw3dHn2LKp2WMPfMQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type:content-transfer-encoding;
        b=RautyUDrEHUm++razqkeoKirxDL+LRbxv0Yb75raPBd9dHJdvXDvNpwjwrPrrsg+xm
         z3JRXL9QLGgTaozoGST5WqLOnPdvNXLaEHV1+xpcGmn9U2C+jw7JqBbHI+Vc6yMkoAFK
         QQH0IFV1B/uEPt5LpS22BH1hjS/A45xjO4otA=
MIME-Version: 1.0
Received: by 10.220.100.210 with HTTP; Tue, 13 Apr 2010 05:30:03 -0700 (PDT)
In-Reply-To: <601540.54306.qm@web56204.mail.re3.yahoo.com>
References: <m2l7c962aed1004120856hddd9e41dt77f0ba3e259c9171@mail.gmail.com>
	 <601540.54306.qm@web56204.mail.re3.yahoo.com>
Date: Tue, 13 Apr 2010 18:30:03 +0600
Received: by 10.220.107.13 with SMTP id z13mr2943628vco.190.1271161803247; 
	Tue, 13 Apr 2010 05:30:03 -0700 (PDT)
Message-ID: <t2y7bfdc29a1004130530s178a6a78x92eba9ad639ce2fe@mail.gmail.com>
Subject: Re: [VOTE] HBase as TLP?
From: Imran M Yousuf <imyousuf@gmail.com>
To: general@hadoop.apache.org
Cc: hbase-dev@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

+1

/Imran

On Tue, Apr 13, 2010 at 5:41 PM, Tsz Wo (Nicholas), Sze
<s29752-hadoopgeneral@yahoo.com> wrote:
> +1
> Nicholas Sze
>
>
>
>
> ----- Original Message ----
>> From: Stack <stack@duboce.net>
>> To: general@hadoop.apache.org; hbase-dev@hadoop.apache.org
>> Sent: Mon, April 12, 2010 8:56:51 AM
>> Subject: [VOTE] HBase as TLP?
>>
>> Please vote as to whether you think HBase should become a top-level
> Apache
>> project.
>
> I've included below a draft board resolution. =A0It lists all
>> current
> active HBase committers as initial members of the project
>> management
> committee (PMC) and myself, Michael Stack, as the initial
>> chair.
>
> Do the good people of Hadoop support sending this request on to
>> the Hadoop PMC?
>
> Yours,
> St.Ack
>
> ------------
>
> =A0 X.
>> Establish the Apache HBase Project
>
> =A0 =A0 =A0WHEREAS, the
>> Board of Directors deems it to be in the best
> =A0 =A0 =A0interests
>> of the Foundation and consistent with the
> =A0 =A0 =A0Foundation's
>> purpose to establish a Project Management
> =A0 =A0 =A0Committee
>> charged with the creation and maintenance of
> =A0 =A0 =A0open-source
>> software related to data serialization
> =A0 =A0 =A0for distribution
>> at no charge to the public.
>
> =A0 =A0 =A0NOW, THEREFORE, BE IT
>> RESOLVED, that a Project Management
> =A0 =A0 =A0Committee (PMC), to
>> be known as the "Apache HBase Project",
> =A0 =A0 =A0be and hereby is
>> established pursuant to Bylaws of the
> =A0 =A0 =A0Foundation; and be
>> it further
>
> =A0 =A0 =A0RESOLVED, that the Apache HBase Project
>> be and hereby is
> =A0 =A0 =A0responsible for the creation and
>> maintenance of software
> =A0 =A0 =A0related to a distributed
>> database; and be it further
>
> =A0 =A0 =A0RESOLVED, that the
>> office of "Vice President, Apache HBase" be
> =A0 =A0 =A0and hereby
>> is created, the person holding such office to
> =A0 =A0 =A0serve at
>> the direction of the Board of Directors as the chair
> =A0 =A0 =A0of
>> the Apache HBase Project, and to have primary responsibility
>
>> =A0 for management of the projects within the scope of
>
>> =A0 responsibility of the Apache HBase Project; and be it
>> further
>
> =A0 =A0 =A0RESOLVED, that the persons listed
>> immediately below be and
> =A0 =A0 =A0hereby are appointed to serve
>> as the initial members of the
> =A0 =A0 =A0Apache HBase
>> Project:
>
> =A0 =A0 =A0 =A0* Michael Stack <
>> ymailto=3D"mailto:stack@apache.org"
>> href=3D"mailto:stack@apache.org">stack@apache.org>
>
>> =A0 * Jonathan Gray <
>> href=3D"mailto:jgray@apache.org">jgray@apache.org>
>
>> =A0 * Andrew Purtell <
>> href=3D"mailto:apurtell@apache.org">apurtell@apache.org>
>
>> =A0 =A0 * John-Daniel Cryans <
>> href=3D"mailto:jdcryans@apache.org">jdcryans@apache.org>
>
>> =A0 =A0 * Ryan Rawson <
>> href=3D"mailto:rawson@apache.org">rawson@apache.org>
>
>> =A0 =A0 * Lars George <
>> href=3D"mailto:larsgeorge@apache.org">larsgeorge@apache.org>
>
>
>> =A0 =A0 NOW, THEREFORE, BE IT FURTHER RESOLVED, that Michael
>> Stack
> =A0 =A0 =A0be appointed to the office of Vice President,
>> Apache HBase, to
> =A0 =A0 =A0serve in accordance with and subject to
>> the direction of the
> =A0 =A0 =A0Board of Directors and the Bylaws
>> of the Foundation until
> =A0 =A0 =A0death, resignation, retirement,
>> removal or disqualification,
> =A0 =A0 =A0or until a successor is
>> appointed; and be it further
>
> =A0 =A0 =A0RESOLVED, that the
>> initial Apache HBase PMC be and hereby is
> =A0 =A0 =A0tasked with
>> the creation of a set of bylaws intended to
> =A0 =A0 =A0encourage
>> open development and increased participation in the
>
>> Apache HBase Project; and be it further
>
> =A0 =A0 =A0RESOLVED,
>> that the Apache HBase Project be and hereby
> =A0 =A0 =A0is tasked
>> with the migration and rationalization of the Apache
>
>> Hadoop HBase sub-project; and be it further
>
>
>> RESOLVED, that all responsibilities pertaining to the Apache
>
>> =A0 Hadoop HBase sub-project encumbered upon the
>
>> Apache Hadoop Project are hereafter discharged.
>



--=20
Imran M Yousuf
Entrepreneur & Software Engineer
Smart IT Engineering
Dhaka, Bangladesh
Email: imran@smartitengineering.com
Blog: http://imyousuf-tech.blogs.smartitengineering.com/
Mobile: +880-1711402557

From general-return-1362-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 14:43:46 2010
Return-Path: <general-return-1362-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56514 invoked from network); 13 Apr 2010 14:43:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 14:43:45 -0000
Received: (qmail 60929 invoked by uid 500); 13 Apr 2010 14:43:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60850 invoked by uid 500); 13 Apr 2010 14:43:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60842 invoked by uid 99); 13 Apr 2010 14:43:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 14:43:44 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of boyuzhang35@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 14:43:37 +0000
Received: by wyb42 with SMTP id 42so1699840wyb.35
        for <general@hadoop.apache.org>; Tue, 13 Apr 2010 07:43:16 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type;
        bh=ATlIWX5YNu0M+dzDBo/iI80+pKrmVxernuGENco93Dw=;
        b=CGTx/ptupftHPOk3n+G0AA6Sq0jclNM2CHVsdmHBMYlu5B70RpmLmjy1ByX9/4bdxG
         P89flVGiyo1giubvfP22nwWUaxaO3IVw26XUTiqr+F4oplB9l5DvSk1jGaH2b201tMvp
         AJbL7gxdW+8v53bnsT9KCx578cmgrrykVAIj4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=LKZ2NxOSoZQeCIFCV4Dn42GdIbnTe6Qkv7Fp57Vdb8YKEunVf8fdT5MIJH13qwH7Fx
         kFqDyfoAGYK3cJAyI5YbR9o3thRpqvgjbmBkhsDCO/oOhqJhIUHhK2YD3PHkoNtP50ff
         hQ4gIFe11wUlJga9/bktiP0uRMY7Vu8Fn0Ngs=
MIME-Version: 1.0
Received: by 10.216.29.195 with HTTP; Tue, 13 Apr 2010 07:43:16 -0700 (PDT)
In-Reply-To: <y2w643a61251004121952nb64aad34wf982d058d7c8ab4a@mail.gmail.com>
References: <m2r643a61251004060743ge24767e9n7038fe038c4f04de@mail.gmail.com>
	 <s2v873f8eb61004121752vc05a79f3qa2a25727adafe40a@mail.gmail.com>
	 <y2w643a61251004121952nb64aad34wf982d058d7c8ab4a@mail.gmail.com>
Date: Tue, 13 Apr 2010 10:43:16 -0400
Received: by 10.216.156.133 with SMTP id m5mr3501546wek.115.1271169796515; 
	Tue, 13 Apr 2010 07:43:16 -0700 (PDT)
Message-ID: <m2j873f8eb61004130743n25835d2bg6b820814be5faeca@mail.gmail.com>
Subject: Re: hadoop on demand setup: Failed to retrieve 'hdfs' service address
From: Boyu Zhang <boyuzhang35@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64f7fae2baf0804841f487f
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e64f7fae2baf0804841f487f
Content-Type: text/plain; charset=ISO-8859-1

Thanks a lot for the reply. I will go over the scripts to see the details.

And I am a little confused about which process starts the hdfs and mapreduce
daemons, the ringmaster or the hodring?

And I am wondering how do the hadoop daemons work once they are up. Do they
communicate in the same way without HOD (daemons talk to each other ) or
everything has to go through ringmaster and hodring? Thanks a lot for the
time and help!

Boyu

On Mon, Apr 12, 2010 at 10:52 PM, Kevin Van Workum <vanw@sabalcore.com>wrote:

> On Mon, Apr 12, 2010 at 8:52 PM, Boyu Zhang <boyuzhang35@gmail.com> wrote:
> > Hi Kevin,
> >
> > Sorry to bother again, I am wondering in order to get HOD to work, do we
> > need to install all the prerequisite software like passwordless ssh?
> Thanks
> > a lot!
>
> SSH is not needed for HOD, it uses pbsdsh to launch processes on the nodes.
>
> HOD seems to be very sensitive about the python version: 2.4 and 2.6
> don't work, you need 2.5.
>
> HOD is a little more flexible with Java, 1.5 and 1.6 seem to both work
> for me. Also, the most recent versions of Twisted and zope seem to be
> fine.
>
> >
> > Boyu
> >
> > On Tue, Apr 6, 2010 at 10:43 AM, Kevin Van Workum <vanw@sabalcore.com
> >wrote:
> >
> >> Hello,
> >>
> >> I'm trying to setup hadoop on demand (HOD) on my cluster. I'm
> >> currently unable to "allocate cluster". I'm starting hod with the
> >> following command:
> >>
> >> /usr/local/hadoop-0.20.2/hod/bin/hod -c
> >> /usr/local/hadoop-0.20.2/hod/conf/hodrc -t
> >> /b/01/vanw/hod/hadoop-0.20.2.tar.gz -o "allocate ~/hod 3"
> >> --ringmaster.log-dir=/tmp -b 4
> >>
> >> The job starts on the nodes and I see the ringmaster running on the
> >> MotherSuperior. The ringmaster-main.log file is created, but is empty.
> >> I don't see any associated processes running on the other 2 nodes in
> >> the job.
> >>
> >> The critical errors are as follows:
> >>
> >> [2010-04-06 10:34:13,630] CRITICAL/50 hadoop:298 - Failed to retrieve
> >> 'hdfs' service address.
> >> [2010-04-06 10:34:13,631] DEBUG/10 hadoop:631 - Cleaning up cluster id
> >> 238366.jman, as cluster could not be allocated.
> >> [2010-04-06 10:34:13,632] DEBUG/10 hadoop:635 - Calling rm.stop()
> >> [2010-04-06 10:34:13,639] DEBUG/10 hadoop:637 - Returning from rm.stop()
> >> [2010-04-06 10:34:13,639] CRITICAL/50 hod:401 - Cannot allocate
> >> cluster /b/01/vanw/hod
> >> [2010-04-06 10:34:14,149] DEBUG/10 hod:597 - return code: 7
> >>
> >> The contents of the hodrc file is:
> >>
> >> [hod]
> >> stream                          = True
> >> java-home                       = /usr/local/jdk1.6.0_02
> >> cluster                         = orange
> >> cluster-factor                  = 1.8
> >> xrs-port-range                  = 32768-65536
> >> debug                           = 4
> >> allocate-wait-time              = 3600
> >> temp-dir                        = /tmp/hod
> >>
> >> [ringmaster]
> >> register                        = True
> >> stream                          = False
> >> temp-dir                        = /tmp/hod
> >> http-port-range                 = 8000-9000
> >> work-dirs                       = /tmp/hod/1,/tmp/hod/2
> >> xrs-port-range                  = 32768-65536
> >> debug                           = 4
> >>
> >> [hodring]
> >> stream                          = False
> >> temp-dir                        = /tmp/hod
> >> register                        = True
> >> java-home                       = /usr/local/jdk1.6.0_02
> >> http-port-range                 = 8000-9000
> >> xrs-port-range                  = 32768-65536
> >> debug                           = 4
> >>
> >> [resource_manager]
> >> queue                           = dque
> >> batch-home                      = /usr/local/torque-2.3.7
> >> id                              = torque
> >> env-vars                       =
> >> HOD_PYTHON_HOME=/usr/local/python-2.5.5/bin/python
> >>
> >> [gridservice-mapred]
> >> external                        = False
> >> pkgs                            = /usr/local/hadoop-0.20.2
> >> tracker_port                    = 8030
> >> info_port                       = 50080
> >>
> >> [gridservice-hdfs]
> >> external                        = False
> >> pkgs                            = /usr/local/hadoop-0.20.2
> >> fs_port                         = 8020
> >> info_port                       = 50070
> >>
> >>
> >> Some other useful information:
> >> Linux 2.6.18-128.7.1.el5
> >> Python 2.5.5
> >> Twisted 10.0.0
> >> zope 3.3.0
> >> java version "1.6.0_02"
> >>
> >> --
> >> Kevin Van Workum, PhD
> >> Sabalcore Computing Inc.
> >> Run your code on 500 processors.
> >> Sign up for a free trial account.
> >> www.sabalcore.com
> >> 877-492-8027 ext. 11
> >>
> >
>
>
>
> --
> Kevin Van Workum, PhD
> Sabalcore Computing Inc.
> Run your code on 500 processors.
> Sign up for a free trial account.
> www.sabalcore.com
> 877-492-8027 ext. 11
>

--0016e64f7fae2baf0804841f487f--

From general-return-1363-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 14:47:23 2010
Return-Path: <general-return-1363-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57147 invoked from network); 13 Apr 2010 14:47:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 14:47:22 -0000
Received: (qmail 71306 invoked by uid 500); 13 Apr 2010 14:47:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71152 invoked by uid 500); 13 Apr 2010 14:47:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71144 invoked by uid 99); 13 Apr 2010 14:47:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 14:47:21 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dan.harvey@mendeley.com designates 72.14.220.159 as permitted sender)
Received: from [72.14.220.159] (HELO fg-out-1718.google.com) (72.14.220.159)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 14:47:16 +0000
Received: by fg-out-1718.google.com with SMTP id l26so892444fgb.11
        for <general@hadoop.apache.org>; Tue, 13 Apr 2010 07:46:54 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.223.121.20 with HTTP; Tue, 13 Apr 2010 07:46:52 -0700 (PDT)
Date: Tue, 13 Apr 2010 15:46:52 +0100
Received: by 10.223.8.67 with SMTP id g3mr3401542fag.107.1271170012942; Tue, 
	13 Apr 2010 07:46:52 -0700 (PDT)
Message-ID: <q2ha7988b8e1004130746t8ab3e3dap5607f7f8e53053b8@mail.gmail.com>
Subject: DBInputFormat number of mappers
From: Dan Harvey <dan.harvey@mendeley.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001517477f2a12112504841f55dc

--001517477f2a12112504841f55dc
Content-Type: text/plain; charset=ISO-8859-1

Hi,

I'm importing data from a mysql database using the DBInputFormat to go over
the rows in a table and put them into HBase with the mapper but I can't find
a way to increase the number of maps it splits the input into. I am running
this on a cluster where we have 5 nodes and each node has a maximum of 2 map
tasks. So for example if I set the number of rows to import to be 10,000,000
then there will only 2 maps tasks and use only two of the nodes..

I've tried increasing the limit manually in the code with :

job.getConfiguration().setInt("mapred.map.tasks", 4);

increasing the number on the command line to set the same property, and also
increasing the number of map tasks per node.
But in all cases mapred.map.tasks is set to 2 in the job xml config file.

I've had a look at the code and DBInputFormat splits the total number of
rows over mapred.map.tasks, so I'm guessing it's just getting that to
change.

It would be great if anyone has any ideas what's going on?

Thanks,

-- 
Dan Harvey | Datamining Engineer
www.mendeley.com/profiles/dan-harvey

Mendeley Limited | London, UK | www.mendeley.com
Registered in England and Wales | Company Number 6419015

--001517477f2a12112504841f55dc--

From general-return-1364-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 15:09:33 2010
Return-Path: <general-return-1364-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63474 invoked from network); 13 Apr 2010 15:09:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 15:09:33 -0000
Received: (qmail 17724 invoked by uid 500); 13 Apr 2010 15:09:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17682 invoked by uid 500); 13 Apr 2010 15:09:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17674 invoked by uid 99); 13 Apr 2010 15:09:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 15:09:32 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of dan.harvey@mendeley.com designates 72.14.220.152 as permitted sender)
Received: from [72.14.220.152] (HELO fg-out-1718.google.com) (72.14.220.152)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 15:09:25 +0000
Received: by fg-out-1718.google.com with SMTP id 22so1882651fge.11
        for <general@hadoop.apache.org>; Tue, 13 Apr 2010 08:09:04 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.223.121.20 with HTTP; Tue, 13 Apr 2010 08:09:04 -0700 (PDT)
In-Reply-To: <q2ha7988b8e1004130746t8ab3e3dap5607f7f8e53053b8@mail.gmail.com>
References: <q2ha7988b8e1004130746t8ab3e3dap5607f7f8e53053b8@mail.gmail.com>
Date: Tue, 13 Apr 2010 16:09:04 +0100
Received: by 10.223.76.132 with SMTP id c4mr3279325fak.106.1271171344392; Tue, 
	13 Apr 2010 08:09:04 -0700 (PDT)
Message-ID: <m2va7988b8e1004130809l40cde215icd5e1ab561b92f49@mail.gmail.com>
Subject: Re: DBInputFormat number of mappers
From: Dan Harvey <dan.harvey@mendeley.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015174c102e6e624104841fa4f0
X-Virus-Checked: Checked by ClamAV on apache.org

--0015174c102e6e624104841fa4f0
Content-Type: text/plain; charset=ISO-8859-1

Right, after sending this e-mail out that started working straight away with
no changes... So setting the number of mappers in the code using :-

job.getConfiguration().setInt("mapred.map.tasks", 4);

allowed me to specify the number of splits/map tasks.

Which lead me to the second problem I've been getting for awhile. When I
start a hadoop job using DBInputFormat as the input if I use 5 splits say
one will start straight away and the others will stay in the initializing
until it is done then carry on one at a time. This doesn't happen all the
time though and using the same code and database some will sometimes start
in parallel!

I've read this has happened to others before but no clear solution was found
then.

Has anyone else had this before or found a way to solve it?

Thanks,

On 13 April 2010 15:46, Dan Harvey <dan.harvey@mendeley.com> wrote:

> Hi,
>
> I'm importing data from a mysql database using the DBInputFormat to go over
> the rows in a table and put them into HBase with the mapper but I can't find
> a way to increase the number of maps it splits the input into. I am running
> this on a cluster where we have 5 nodes and each node has a maximum of 2 map
> tasks. So for example if I set the number of rows to import to be 10,000,000
> then there will only 2 maps tasks and use only two of the nodes..
>
> I've tried increasing the limit manually in the code with :
>
> job.getConfiguration().setInt("mapred.map.tasks", 4);
>
> increasing the number on the command line to set the same property, and
> also increasing the number of map tasks per node.
> But in all cases mapred.map.tasks is set to 2 in the job xml config file.
>
> I've had a look at the code and DBInputFormat splits the total number of
> rows over mapred.map.tasks, so I'm guessing it's just getting that to
> change.
>
> It would be great if anyone has any ideas what's going on?
>
> Thanks,
>
> --
> Dan Harvey | Datamining Engineer
> www.mendeley.com/profiles/dan-harvey
>
> Mendeley Limited | London, UK | www.mendeley.com
> Registered in England and Wales | Company Number 6419015
>



-- 
Dan Harvey | Datamining Engineer
www.mendeley.com/profiles/dan-harvey

Mendeley Limited | London, UK | www.mendeley.com
Registered in England and Wales | Company Number 6419015

--0015174c102e6e624104841fa4f0--

From general-return-1365-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 15:30:07 2010
Return-Path: <general-return-1365-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68792 invoked from network); 13 Apr 2010 15:30:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 15:30:06 -0000
Received: (qmail 43600 invoked by uid 500); 13 Apr 2010 15:30:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 43558 invoked by uid 500); 13 Apr 2010 15:30:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 43544 invoked by uid 99); 13 Apr 2010 15:30:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 15:30:05 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of fzappert@gmail.com designates 209.85.221.187 as permitted sender)
Received: from [209.85.221.187] (HELO mail-qy0-f187.google.com) (209.85.221.187)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 15:29:58 +0000
Received: by qyk17 with SMTP id 17so2985995qyk.9
        for <multiple recipients>; Tue, 13 Apr 2010 08:29:37 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:cc:content-type;
        bh=Mxzf4ioWN+BrTjKY1T7xiuXn5SuVqAXLpomUhqQ7Za8=;
        b=Y7p6MbWFNgReb/yvghmWxf/OAjwveTH7MpN+YR9CZklsOpzG21DAmM4U92Q5m/7MKf
         ndjnuEaxyNRiT5VhyLILx2iIvwcEOLu9dfeTs7uqe32wm9J0l1Uic+J2LGjzV8goT7jF
         Xd0bVAwKTep7iKV4p2QE2zvRfom6pThiBoXt0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=k+slzFwYrTLM2DWrvswu/KgyfZbU/EXvVERkckGuQFE1E04m/06+vrTyiHDHTGL5Hq
         d3qAxEIURXG8KHJMvVrWtipdaDavH8VO+koQ2VUW0EK2MB7xv8VELxH96sXorHSR7o1b
         5GW4HO72t0ALylib9If0q8S+GGYD4qMKnQEmg=
MIME-Version: 1.0
Received: by 10.229.96.17 with HTTP; Tue, 13 Apr 2010 08:29:37 -0700 (PDT)
In-Reply-To: <t2y7bfdc29a1004130530s178a6a78x92eba9ad639ce2fe@mail.gmail.com>
References: <m2l7c962aed1004120856hddd9e41dt77f0ba3e259c9171@mail.gmail.com>
	 <601540.54306.qm@web56204.mail.re3.yahoo.com>
	 <t2y7bfdc29a1004130530s178a6a78x92eba9ad639ce2fe@mail.gmail.com>
Date: Tue, 13 Apr 2010 08:29:37 -0700
Received: by 10.229.224.79 with SMTP id in15mr2179104qcb.76.1271172577435; 
	Tue, 13 Apr 2010 08:29:37 -0700 (PDT)
Message-ID: <v2z14048aeb1004130829tbeed3d80r6859e4022bd71439@mail.gmail.com>
Subject: Re: [VOTE] HBase as TLP?
From: Fred Zappert <fzappert@gmail.com>
To: general@hadoop.apache.org
Cc: hbase-dev@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016363b8edced1f8004841fedd1
X-Virus-Checked: Checked by ClamAV on apache.org

--0016363b8edced1f8004841fedd1
Content-Type: text/plain; charset=ISO-8859-1

+1


Fred Zappert
W: (206) 266-8375
M: (415) 640-2980
http://www.linkedin.com/in/fredzappert

On Tue, Apr 13, 2010 at 5:30 AM, Imran M Yousuf <imyousuf@gmail.com> wrote:

> +1
>
> /Imran
>
> On Tue, Apr 13, 2010 at 5:41 PM, Tsz Wo (Nicholas), Sze
> <s29752-hadoopgeneral@yahoo.com> wrote:
> > +1
> > Nicholas Sze
> >
> >
> >
> >
> > ----- Original Message ----
> >> From: Stack <stack@duboce.net>
> >> To: general@hadoop.apache.org; hbase-dev@hadoop.apache.org
> >> Sent: Mon, April 12, 2010 8:56:51 AM
> >> Subject: [VOTE] HBase as TLP?
> >>
> >> Please vote as to whether you think HBase should become a top-level
> > Apache
> >> project.
> >
> > I've included below a draft board resolution.  It lists all
> >> current
> > active HBase committers as initial members of the project
> >> management
> > committee (PMC) and myself, Michael Stack, as the initial
> >> chair.
> >
> > Do the good people of Hadoop support sending this request on to
> >> the Hadoop PMC?
> >
> > Yours,
> > St.Ack
> >
> > ------------
> >
> >   X.
> >> Establish the Apache HBase Project
> >
> >      WHEREAS, the
> >> Board of Directors deems it to be in the best
> >      interests
> >> of the Foundation and consistent with the
> >      Foundation's
> >> purpose to establish a Project Management
> >      Committee
> >> charged with the creation and maintenance of
> >      open-source
> >> software related to data serialization
> >      for distribution
> >> at no charge to the public.
> >
> >      NOW, THEREFORE, BE IT
> >> RESOLVED, that a Project Management
> >      Committee (PMC), to
> >> be known as the "Apache HBase Project",
> >      be and hereby is
> >> established pursuant to Bylaws of the
> >      Foundation; and be
> >> it further
> >
> >      RESOLVED, that the Apache HBase Project
> >> be and hereby is
> >      responsible for the creation and
> >> maintenance of software
> >      related to a distributed
> >> database; and be it further
> >
> >      RESOLVED, that the
> >> office of "Vice President, Apache HBase" be
> >      and hereby
> >> is created, the person holding such office to
> >      serve at
> >> the direction of the Board of Directors as the chair
> >      of
> >> the Apache HBase Project, and to have primary responsibility
> >
> >>   for management of the projects within the scope of
> >
> >>   responsibility of the Apache HBase Project; and be it
> >> further
> >
> >      RESOLVED, that the persons listed
> >> immediately below be and
> >      hereby are appointed to serve
> >> as the initial members of the
> >      Apache HBase
> >> Project:
> >
> >        * Michael Stack <
> >> ymailto="mailto:stack@apache.org"
> >> href="mailto:stack@apache.org">stack@apache.org>
> >
> >>   * Jonathan Gray <
> >> href="mailto:jgray@apache.org">jgray@apache.org>
> >
> >>   * Andrew Purtell <
> >> href="mailto:apurtell@apache.org">apurtell@apache.org>
> >
> >>     * John-Daniel Cryans <
> >> href="mailto:jdcryans@apache.org">jdcryans@apache.org>
> >
> >>     * Ryan Rawson <
> >> href="mailto:rawson@apache.org">rawson@apache.org>
> >
> >>     * Lars George <
> >> href="mailto:larsgeorge@apache.org">larsgeorge@apache.org>
> >
> >
> >>     NOW, THEREFORE, BE IT FURTHER RESOLVED, that Michael
> >> Stack
> >      be appointed to the office of Vice President,
> >> Apache HBase, to
> >      serve in accordance with and subject to
> >> the direction of the
> >      Board of Directors and the Bylaws
> >> of the Foundation until
> >      death, resignation, retirement,
> >> removal or disqualification,
> >      or until a successor is
> >> appointed; and be it further
> >
> >      RESOLVED, that the
> >> initial Apache HBase PMC be and hereby is
> >      tasked with
> >> the creation of a set of bylaws intended to
> >      encourage
> >> open development and increased participation in the
> >
> >> Apache HBase Project; and be it further
> >
> >      RESOLVED,
> >> that the Apache HBase Project be and hereby
> >      is tasked
> >> with the migration and rationalization of the Apache
> >
> >> Hadoop HBase sub-project; and be it further
> >
> >
> >> RESOLVED, that all responsibilities pertaining to the Apache
> >
> >>   Hadoop HBase sub-project encumbered upon the
> >
> >> Apache Hadoop Project are hereafter discharged.
> >
>
>
>
> --
> Imran M Yousuf
> Entrepreneur & Software Engineer
> Smart IT Engineering
> Dhaka, Bangladesh
> Email: imran@smartitengineering.com
> Blog: http://imyousuf-tech.blogs.smartitengineering.com/
> Mobile: +880-1711402557
>

--0016363b8edced1f8004841fedd1--

From general-return-1366-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 16:45:39 2010
Return-Path: <general-return-1366-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66478 invoked from network); 13 Apr 2010 16:26:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 16:26:20 -0000
Received: (qmail 77171 invoked by uid 500); 13 Apr 2010 16:26:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76998 invoked by uid 500); 13 Apr 2010 16:26:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76852 invoked by uid 99); 13 Apr 2010 16:26:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 16:26:17 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 16:26:09 +0000
Received: from [192.168.1.71] ([10.72.76.39])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o3DGOc6Q047370
	for <general@hadoop.apache.org>; Tue, 13 Apr 2010 09:24:39 -0700 (PDT)
Message-Id: <1CCD1C49-71A3-46B2-878E-74781E0815EE@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: JIRA snafu?
Date: Tue, 13 Apr 2010 09:24:38 -0700
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

I can no longer grant permission to Apache for my patches on JIRA.  
Anyone else?

thanks,
Arun

From general-return-1368-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 17:41:38 2010
Return-Path: <general-return-1368-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 48804 invoked from network); 13 Apr 2010 16:47:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 16:47:19 -0000
Received: (qmail 49225 invoked by uid 500); 13 Apr 2010 16:47:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49163 invoked by uid 500); 13 Apr 2010 16:47:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49147 invoked by uid 99); 13 Apr 2010 16:47:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 16:47:17 +0000
X-ASF-Spam-Status: No, hits=0.8 required=10.0
	tests=AWL,HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 16:47:11 +0000
Received: from sp1-ex07cas01.ds.corp.yahoo.com (sp1-ex07cas01.ds.corp.yahoo.com [216.252.116.137])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o3DGjO9g059169
	for <general@hadoop.apache.org>; Tue, 13 Apr 2010 09:45:24 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:in-reply-to:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=TOA4iqzOsu7Rqe6W70qR6uRvovClPblyY3Imqme7vsKXmhVNITpWW79FjLk5hMIQ
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas01.ds.corp.yahoo.com ([216.252.116.137]) with mapi; Tue, 13 Apr
 2010 09:45:23 -0700
From: Suresh Srinivas <sureshms@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 13 Apr 2010 09:45:22 -0700
Subject: Re: JIRA snafu?
Thread-Topic: JIRA snafu?
Thread-Index: AcrbJkQcbpRQsmXsTsOd4RXS9cdrrwAAm/6S
Message-ID: <C7E9EDB0.21894%sureshms@yahoo-inc.com>
In-Reply-To: <1CCD1C49-71A3-46B2-878E-74781E0815EE@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C7E9EDB021894sureshmsyahooinccom_"
MIME-Version: 1.0

--_000_C7E9EDB021894sureshmsyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

I had the same problem yesterday.


On 4/13/10 8:24 AM, "Arun C Murthy" <acm@yahoo-inc.com> wrote:

I can no longer grant permission to Apache for my patches on JIRA.
Anyone else?

thanks,
Arun


--_000_C7E9EDB021894sureshmsyahooinccom_--

From general-return-1367-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 17:43:48 2010
Return-Path: <general-return-1367-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47005 invoked from network); 13 Apr 2010 16:46:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 16:46:34 -0000
Received: (qmail 40558 invoked by uid 500); 13 Apr 2010 16:46:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40513 invoked by uid 500); 13 Apr 2010 16:46:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40501 invoked by uid 99); 13 Apr 2010 16:46:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 16:46:32 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 16:46:26 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:Received:Received:From:To:Subject:
   Thread-Topic:Thread-Index:Date:Message-ID:References:
   In-Reply-To:Accept-Language:Content-Language:
   X-MS-Has-Attach:X-MS-TNEF-Correlator:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version:
   Return-Path:X-OriginalArrivalTime;
  b=ZW9feciwglknhNWv+jztcWtubch8LdI3Ljpy4fPrHIWQpijNkW8W0O+g
   6QIf1IL4mcZZIy0usnHXh/9lrCEhZMAV7CFkaJsAu/DLscsSDs4tJFIJ6
   lhPMUWBwJpzB/r2;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1271177186; x=1302713186;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20JIRA=20snafu?|Date:=20Tue,=2013=20Apr
   =202010=2016:46:04=20+0000|Message-ID:=20<0FF6B01F-2F20-4
   9B2-B689-94B544637CBD@linkedin.com>|To:=20"<general@hadoo
   p.apache.org>"=20<general@hadoop.apache.org>
   |MIME-Version:=201.0|Content-Transfer-Encoding:=20quoted-
   printable|Content-ID:=20<016fd015-e61b-46c7-8829-824e3f0b
   c526>|In-Reply-To:=20<1CCD1C49-71A3-46B2-878E-74781E0815E
   E@yahoo-inc.com>|References:=20<1CCD1C49-71A3-46B2-878E-7
   4781E0815EE@yahoo-inc.com>;
  bh=Qp5qkxCufDNrqmyOepO1QqAXf1s8jJzicyCwqsFDuZ8=;
  b=uI9v0FZxH+yjezlWS0BuBCIUsaOhfrtiCsmHWYlAGWc8opGcJJUB4Wc0
   N3erhoLbryYnX6zJ1UjckHy3ymKHx3eCy5UOpfZQurQtmoFg9CvcKJSiW
   M+jbweZXONdQAbf;
X-IronPort-AV: E=Sophos;i="4.52,198,1270450800"; 
   d="scan'208";a="12290539"
Received: from esv4-exctest.linkedin.biz ([172.18.46.60]) by CORP-MAIL.linkedin.biz with Microsoft SMTPSVC(6.0.3790.3959);
	 Tue, 13 Apr 2010 09:46:05 -0700
Received: from ESV4-CAS02.linkedin.biz (172.18.46.142) by
 esv4-exctest.linkedin.biz (172.18.46.60) with Microsoft SMTP Server (TLS) id
 14.0.682.1; Tue, 13 Apr 2010 09:46:04 -0700
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi; Tue, 13 Apr 2010
 09:46:04 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: JIRA snafu?
Thread-Topic: JIRA snafu?
Thread-Index: AQHK2yYM3Wjwn2J7K0+LJWJ86ZUH7JIhEKkA
Date: Tue, 13 Apr 2010 16:46:04 +0000
Message-ID: <0FF6B01F-2F20-49B2-B689-94B544637CBD@linkedin.com>
References: <1CCD1C49-71A3-46B2-878E-74781E0815EE@yahoo-inc.com>
In-Reply-To: <1CCD1C49-71A3-46B2-878E-74781E0815EE@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Content-Type: text/plain; charset="us-ascii"
Content-ID: <016fd015-e61b-46c7-8829-824e3f0bc526>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginalArrivalTime: 13 Apr 2010 16:46:05.0452 (UTC) FILETIME=[CF2D7CC0:01CADB28]
X-Virus-Checked: Checked by ClamAV on apache.org


On Apr 13, 2010, at 9:24 AM, Arun C Murthy wrote:

> I can no longer grant permission to Apache for my patches on JIRA. Anyone=
 else?

I haven't tried, but likely related to issues.apache.org getting compromise=
d.



From general-return-1369-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 18:53:32 2010
Return-Path: <general-return-1369-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72896 invoked from network); 13 Apr 2010 17:13:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 17:13:08 -0000
Received: (qmail 18174 invoked by uid 500); 13 Apr 2010 17:13:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17998 invoked by uid 500); 13 Apr 2010 17:13:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17990 invoked by uid 99); 13 Apr 2010 17:13:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 17:13:06 +0000
X-ASF-Spam-Status: No, hits=0.2 required=10.0
	tests=AWL,FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bernd.fondermann@googlemail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 17:13:02 +0000
Received: by wyb42 with SMTP id 42so1775859wyb.35
        for <general@hadoop.apache.org>; Tue, 13 Apr 2010 10:12:40 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type;
        bh=IznoQf8HMT+CZ7NRKQOIR1e7mFfZ7LLioKYyJvzsKjY=;
        b=jwYGpJ8I36PxO0TSX3Qb+klHIwO+JYJVPIH87XTv86TC4A/UHjDgMLe12hXKAVCu+z
         8sX9SXGYXmQsVlu4PxCSqBHypyhhGiWiHTKPOPA3PjwymXZHM3scWVkVAgPYYa2nwdCL
         ocQfVPlywTe1ks9daOoMr1IY3fu/wOy0PUUdU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=i+qkSuOuATqPoyi/6VoU/XDRUvH3KTCblM3CbGpApvtMyENWOMwDG1JE53uwtSxrHX
         EQabUHrQ4R8P72g2JyD7dwIR6NSF0/4MjsK2QwWDrQpdtLCobY4vMP1a15g0wkuku93n
         cpZ5zdvBKNndmWc/zcegrvLXuqnjahV0aC4A0=
MIME-Version: 1.0
Received: by 10.216.230.75 with HTTP; Tue, 13 Apr 2010 10:12:40 -0700 (PDT)
In-Reply-To: <1CCD1C49-71A3-46B2-878E-74781E0815EE@yahoo-inc.com>
References: <1CCD1C49-71A3-46B2-878E-74781E0815EE@yahoo-inc.com>
Date: Tue, 13 Apr 2010 19:12:40 +0200
Received: by 10.216.88.212 with SMTP id a62mr3610842wef.72.1271178760481; Tue, 
	13 Apr 2010 10:12:40 -0700 (PDT)
Message-ID: <p2s88f6e29a1004131012ie0e92c8rb7382c2fa09ddde6@mail.gmail.com>
Subject: Re: JIRA snafu?
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

AFAIK, JIRA has been re-installed.
(BTW, please change your passwords.)

You can report this on IRC #asfinfra, or infrastructure@a.o.

  Bernd


On Tue, Apr 13, 2010 at 18:24, Arun C Murthy <acm@yahoo-inc.com> wrote:
> I can no longer grant permission to Apache for my patches on JIRA. Anyone
> else?
>
> thanks,
> Arun
>

From general-return-1370-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 18:54:02 2010
Return-Path: <general-return-1370-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 80359 invoked from network); 13 Apr 2010 17:14:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 17:14:18 -0000
Received: (qmail 23877 invoked by uid 500); 13 Apr 2010 17:14:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23840 invoked by uid 500); 13 Apr 2010 17:14:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23832 invoked by uid 99); 13 Apr 2010 17:14:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 17:14:17 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=AWL,HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 17:14:10 +0000
Received: from sp1-ex07cas03.ds.corp.yahoo.com (sp1-ex07cas03.ds.corp.yahoo.com [216.252.116.151])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o3DHDRRC072588
	for <general@hadoop.apache.org>; Tue, 13 Apr 2010 10:13:27 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:in-reply-to:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=JU18knTudHEEGvysag2LTv2rqqt26p34QqrLR7dqYrv7QVFmrV+7ChE8m+UW5sIA
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas03.ds.corp.yahoo.com ([216.252.116.151]) with mapi; Tue, 13 Apr
 2010 10:13:26 -0700
From: Suresh Srinivas <sureshms@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 13 Apr 2010 10:13:26 -0700
Subject: Re: JIRA snafu?
Thread-Topic: JIRA snafu?
Thread-Index: AcrbJkQcbpRQsmXsTsOd4RXS9cdrrwAAm/6SAAD6yQE=
Message-ID: <C7E9F443.218A2%sureshms@yahoo-inc.com>
In-Reply-To: <C7E9EDB0.21894%sureshms@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C7E9F443218A2sureshmsyahooinccom_"
MIME-Version: 1.0

--_000_C7E9F443218A2sureshmsyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

I also see that File Attachments no longer show time when the attachment wa=
s uploaded.

On 4/13/10 9:45 AM, "Suresh Srinivas" <sureshms@yahoo-inc.com> wrote:

I had the same problem yesterday.


On 4/13/10 8:24 AM, "Arun C Murthy" <acm@yahoo-inc.com> wrote:

I can no longer grant permission to Apache for my patches on JIRA.
Anyone else?

thanks,
Arun


--_000_C7E9F443218A2sureshmsyahooinccom_--

From general-return-1371-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 06:47:02 2010
Return-Path: <general-return-1371-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88933 invoked from network); 14 Apr 2010 06:47:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 06:47:02 -0000
Received: (qmail 74770 invoked by uid 500); 14 Apr 2010 06:47:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74550 invoked by uid 500); 14 Apr 2010 06:47:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74541 invoked by uid 99); 14 Apr 2010 06:47:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 06:47:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 14 Apr 2010 06:46:57 +0000
Received: (qmail 88828 invoked by uid 99); 14 Apr 2010 06:46:35 -0000
Received: from localhost.apache.org (HELO mail-iw0-f198.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 06:46:35 +0000
Received: by iwn36 with SMTP id 36so6166878iwn.29
        for <multiple recipients>; Tue, 13 Apr 2010 23:46:34 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.168.78 with HTTP; Tue, 13 Apr 2010 23:46:34 -0700 (PDT)
Date: Wed, 14 Apr 2010 07:46:34 +0100
Received: by 10.231.60.19 with SMTP id n19mr3123479ibh.79.1271227594823; Tue, 
	13 Apr 2010 23:46:34 -0700 (PDT)
Message-ID: <u2n1267dd3b1004132346gae872ec3g54d3e8b3d221c68e@mail.gmail.com>
Subject: Subprojects and TLP status
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org, private@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Most of Hadoop's subprojects have discussed becoming top-level Apache
projects (TLPs) in the last few weeks. Most have expressed a desire to
remain in Hadoop. The salient parts of the discussions I've read tend
to focus on three aspects: a technical dependence on Hadoop,
additional overhead as a TLP, and visibility both within the Hadoop
ecosystem and in the open source community generally.

Life as a TLP: this is not much harder than being a Hadoop subproject,
and the Apache preferences being tossed around- particularly
"insufficiently diverse"- are not blockers. Every subproject needs to
write a section of the report Hadoop sends to the board; almost the
same report, sent to a new address. The initial cost is similarly
light: copy bylaws, send a few notes to INFRA, and follow some
directions. I think the estimated costs are far higher than they will
be in practice. Inertia is a powerful force, but it should be
overcome. The directions are here, and should not intimidating:

http://apache.org/dev/project-creation.html

Visibility: the Hadoop site does not need to change. For each
subproject, we can literally change the hyperlinks to point to the new
page and be done. Long-term, linking to all ASF projects that run on
Hadoop from a prominent page is something we all want. So particularly
in the medium-term that most are considering: visibility through the
website will not change. Each subproject will still be linked from the
front page.

Hadoop would not be nearly as popular as it is without Zookeeper,
HBase, Hive, and Pig. All statistics on work in shared MapReduce
clusters show that users vastly prefer running Pig and Hive queries to
writing MapReduce jobs. HBase continues to push features in HDFS that
increase its adoption and relevance outside MapReduce, while sharing
some of its NoSQL limelight. Zookeeper is not only a linchpin in real
workloads, but many proposals for future features require it. The
bottom line is that MapReduce and HDFS need these projects for
visibility and adoption in precisely the same way. I don't think
separate TLPs will uncouple the broader community from one another.

Technical dependence: this has two dimensions. First, influencing
MapReduce and HDFS. This is nonsense. Earning influence by
contributing to a subproject is the only way to push code changes;
nobody from any of these projects has violated that by unilaterally
committing to HDFS or MapReduce, anyway. And anyone cynical enough to
believe that MapReduce and HDFS would deliberately screw over or
ignore dependent projects because they don't have PMC members is
plainly unsuited to community-driven development. I understand that
these projects need to protect their users, but lobbying rights are
not an actual benefit.

Second, being a coherent part of the Hadoop ecosystem. It is (mostly)
true that Hadoop current offers a set of mutually compatible
frameworks. It is not true that moving them to separate Apache
projects would make solutions less coherent or affect existing or
future users at all. The cohesion between projects' governance is
sufficiently weak to justify independent units, but the real
dependencies between the projects are strong enough to keep us engaged
with one another. And it's not as if other projects- Cascading, for
example- aren't also organisms adapted and specialized for life in
Hadoop.

Arguments on technical dependence are ignoring the nature of the
existing interactions. Besides, weak technical dependencies are not a
necessary prerequisite for a subproject's independence.

As for what was *not* said in these discussions, there is no argument
that every one of these subprojects has a distinct, autonomous
community. There was also no argument that the Hadoop PMC offers any
valuable oversight, given that the representatives of its fiefdoms are
too consumed by provincial matters to participate in neighboring
governance. Most releases I've voted on: I run the unit tests, check
the signature, verify the checksum, and know literally nothing else
about its content. I have often never heard the names of many proposed
committers and even some proposed PMC members. Right now, subprojects
with enough PMC members essentially vote out their own releases and
vote in their own committers: TLPs in all but name.

The Hadoop club- in conferences, meetups, technical debates, etc.- is
broad, diverse, and intertwined, but communities of developers have
already clustered around subprojects. Allowing that each cluster
should govern itself is a dry, practical matter, not an existential
crisis. -C

From general-return-1372-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 07:59:53 2010
Return-Path: <general-return-1372-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 26671 invoked from network); 14 Apr 2010 07:59:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 07:59:53 -0000
Received: (qmail 32073 invoked by uid 500); 14 Apr 2010 07:59:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31819 invoked by uid 500); 14 Apr 2010 07:59:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31807 invoked by uid 99); 14 Apr 2010 07:59:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 07:59:51 +0000
X-ASF-Spam-Status: No, hits=0.2 required=10.0
	tests=AWL,FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bernd.fondermann@googlemail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 07:59:46 +0000
Received: by wyb42 with SMTP id 42so2059592wyb.35
        for <general@hadoop.apache.org>; Wed, 14 Apr 2010 00:59:25 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=Jwprcb6KNuuva2ym2Ovn8qJ6ZBe6nGUJe92eaoe8+yc=;
        b=XXRoTOJERZc90jB9u3c+4xIf3RZdiGNA24CB+tfB4gg22rvcMEB7JYIRZqpV7Wlft5
         JbUBZKbTEKJPQ6rCk6aPVGnkvAmSP8x8U+NdH17KirSoBX9fxfUE1H6N4HibIculLEDa
         ptVLzuBFtcpLVVXqBx6v+4fNufLbJFwb1zs8A=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=aNQ03/MwiGo9QIdzEMnYtBFZh/G3rY3STksjBpVYwr4Q+9R3UNiEtlMl2i2Y/U4O0v
         0up4oBK0WuXFdZWy17ouaA7FcU5u27IADNhnQ/k9pzMr9wrJvqO9p39iQmGZGUZwcvFi
         u2GjYDxwzz26OeltF13YxQjEQJJzSmANAViWY=
MIME-Version: 1.0
Received: by 10.216.230.75 with HTTP; Wed, 14 Apr 2010 00:59:25 -0700 (PDT)
In-Reply-To: <p2s88f6e29a1004131012ie0e92c8rb7382c2fa09ddde6@mail.gmail.com>
References: <1CCD1C49-71A3-46B2-878E-74781E0815EE@yahoo-inc.com>
	 <p2s88f6e29a1004131012ie0e92c8rb7382c2fa09ddde6@mail.gmail.com>
Date: Wed, 14 Apr 2010 09:59:25 +0200
Received: by 10.216.157.141 with SMTP id o13mr4604242wek.163.1271231965354; 
	Wed, 14 Apr 2010 00:59:25 -0700 (PDT)
Message-ID: <h2k88f6e29a1004140059v428d799fmbd18fdafdef09d07@mail.gmail.com>
Subject: Re: JIRA snafu?
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Opened
https://issues.apache.org/jira/browse/INFRA-2615

  Bernd

On Tue, Apr 13, 2010 at 19:12, Bernd Fondermann
<bernd.fondermann@googlemail.com> wrote:
> AFAIK, JIRA has been re-installed.
> (BTW, please change your passwords.)
>
> You can report this on IRC #asfinfra, or infrastructure@a.o.
>
> =A0Bernd
>
>
> On Tue, Apr 13, 2010 at 18:24, Arun C Murthy <acm@yahoo-inc.com> wrote:
>> I can no longer grant permission to Apache for my patches on JIRA. Anyon=
e
>> else?
>>
>> thanks,
>> Arun
>>
>

From general-return-1373-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 10:36:04 2010
Return-Path: <general-return-1373-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29884 invoked from network); 14 Apr 2010 10:36:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 10:36:04 -0000
Received: (qmail 14844 invoked by uid 500); 14 Apr 2010 10:36:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14698 invoked by uid 500); 14 Apr 2010 10:36:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14690 invoked by uid 99); 14 Apr 2010 10:36:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 10:36:02 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 10:35:52 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 796BCB7E68
	for <general@hadoop.apache.org>; Wed, 14 Apr 2010 11:35:31 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id pKVDObYYTa9h for <general@hadoop.apache.org>;
	Wed, 14 Apr 2010 11:35:30 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id B2C1DB7E67
	for <general@hadoop.apache.org>; Wed, 14 Apr 2010 11:35:30 +0100 (BST)
MailScanner-NULL-Check: 1271846118.65709@smvIvwtpwU4Z6vsUIXRz8w
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o3EAZHej006411
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Wed, 14 Apr 2010 11:35:18 +0100 (BST)
Message-ID: <4BC59A65.20701@apache.org>
Date: Wed, 14 Apr 2010 11:35:17 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100228)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: JIRA snafu?
References: <1CCD1C49-71A3-46B2-878E-74781E0815EE@yahoo-inc.com>
In-Reply-To: <1CCD1C49-71A3-46B2-878E-74781E0815EE@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o3EAZHej006411
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

Arun C Murthy wrote:
> I can no longer grant permission to Apache for my patches on JIRA. 
> Anyone else?

the emergency release of jira doesn't have the extra legal stuff, until 
they get that in you are being viewed as implicitly granting permissions.

-steve

From general-return-1374-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 11:02:21 2010
Return-Path: <general-return-1374-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44386 invoked from network); 14 Apr 2010 11:02:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 11:02:21 -0000
Received: (qmail 33830 invoked by uid 500); 14 Apr 2010 11:02:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33647 invoked by uid 500); 14 Apr 2010 11:02:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33639 invoked by uid 99); 14 Apr 2010 11:02:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 11:02:17 +0000
X-ASF-Spam-Status: No, hits=0.2 required=10.0
	tests=AWL,FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bernd.fondermann@googlemail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 11:02:12 +0000
Received: by wyb42 with SMTP id 42so2123112wyb.35
        for <general@hadoop.apache.org>; Wed, 14 Apr 2010 04:01:51 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type;
        bh=E2kCG+6ctZSMeryH1JoW2ahEDc51Sw2IxIT5++qCAgQ=;
        b=EOixUV13aO3/l94JAIm7vjx2liHlCPL7qQ3u4Y3HpmgEEFkl1YDZPFo4KN5iTfxbRS
         X3Sr328LCfWLO2wrNQ8X1rxbjvFw68k/QRIv1VVturvoLPZUPM7EfKiwse/DojZL30ts
         WD6WhISeeAMuqLacYxcigRLB/N+RYr3Z12SNM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=qm1LZp3Pj0QFkUM4hhMnHde/I5AWanbg1g/MjZJrmsGju+UJmP0p/VzXMn5qxgxkCf
         yxUyJc4BwDqj4/EmulAFVTZsdx1Pq5+kQVtXdvm+LEufWU5K7t/J/4zEJcuePlQgiAcQ
         kIHPasWtPbusdONZXW7qYw/IohA4U89vvdq50=
MIME-Version: 1.0
Received: by 10.216.230.75 with HTTP; Wed, 14 Apr 2010 04:01:51 -0700 (PDT)
In-Reply-To: <4BC59A65.20701@apache.org>
References: <1CCD1C49-71A3-46B2-878E-74781E0815EE@yahoo-inc.com>
	 <4BC59A65.20701@apache.org>
Date: Wed, 14 Apr 2010 13:01:51 +0200
Received: by 10.216.157.141 with SMTP id o13mr4842480wek.163.1271242911564; 
	Wed, 14 Apr 2010 04:01:51 -0700 (PDT)
Message-ID: <p2s88f6e29a1004140401l5797704cl3287b884d39280da@mail.gmail.com>
Subject: Re: JIRA snafu?
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Apr 14, 2010 at 12:35, Steve Loughran <stevel@apache.org> wrote:
> Arun C Murthy wrote:
>>
>> I can no longer grant permission to Apache for my patches on JIRA. Anyone
>> else?
>
> the emergency release of jira doesn't have the extra legal stuff, until they
> get that in you are being viewed as implicitly granting permissions.

But it would help if contributors would explicitly comment that they
grant permission.
In a few months at least I won't be able to remember when and if JIRA
had been down.

  Bernd

From general-return-1375-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 11:49:05 2010
Return-Path: <general-return-1375-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78270 invoked from network); 14 Apr 2010 11:49:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 11:49:05 -0000
Received: (qmail 94847 invoked by uid 500); 14 Apr 2010 11:49:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94652 invoked by uid 500); 14 Apr 2010 11:49:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94644 invoked by uid 99); 14 Apr 2010 11:49:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 11:49:02 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=AWL,RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 11:48:53 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 69F83B7E77
	for <general@hadoop.apache.org>; Wed, 14 Apr 2010 12:48:32 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id Dy9pdRmrzZXV for <general@hadoop.apache.org>;
	Wed, 14 Apr 2010 12:48:31 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 7E8D2B7E65
	for <general@hadoop.apache.org>; Wed, 14 Apr 2010 12:48:30 +0100 (BST)
MailScanner-NULL-Check: 1271850499.2477@GylAWR+PGAAbfBceDCZUng
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o3EBmHHg008093
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Wed, 14 Apr 2010 12:48:18 +0100 (BST)
Message-ID: <4BC5AB81.3040103@apache.org>
Date: Wed, 14 Apr 2010 12:48:17 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100228)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: JIRA snafu?
References: <1CCD1C49-71A3-46B2-878E-74781E0815EE@yahoo-inc.com>	 <4BC59A65.20701@apache.org> <p2s88f6e29a1004140401l5797704cl3287b884d39280da@mail.gmail.com>
In-Reply-To: <p2s88f6e29a1004140401l5797704cl3287b884d39280da@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o3EBmHHg008093
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

Bernd Fondermann wrote:
> On Wed, Apr 14, 2010 at 12:35, Steve Loughran <stevel@apache.org> wrote:
>> Arun C Murthy wrote:
>>> I can no longer grant permission to Apache for my patches on JIRA. Anyone
>>> else?
>> the emergency release of jira doesn't have the extra legal stuff, until they
>> get that in you are being viewed as implicitly granting permissions.
> 
> But it would help if contributors would explicitly comment that they
> grant permission.
> In a few months at least I won't be able to remember when and if JIRA
> had been down.

+1


From general-return-1376-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 13:40:20 2010
Return-Path: <general-return-1376-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50506 invoked from network); 14 Apr 2010 13:40:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 13:40:20 -0000
Received: (qmail 84909 invoked by uid 500); 14 Apr 2010 13:40:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84794 invoked by uid 500); 14 Apr 2010 13:40:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84786 invoked by uid 99); 14 Apr 2010 13:40:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 13:40:19 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=AWL,HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of msegel@navteq.com designates 204.120.70.37 as permitted sender)
Received: from [204.120.70.37] (HELO xmailfargo.navteq.com) (204.120.70.37)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 13:40:14 +0000
Received: from fargo-spamfw.navteq.com (navteq-spamfw.navteq.com [10.6.10.131])
	by xmailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id o3EDN29s000499
	for <general@hadoop.apache.org>; Wed, 14 Apr 2010 08:23:02 -0500
Received: from imailfargo.navteq.com (localhost [127.0.0.1])
	by fargo-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id 185C848F1A0F
	for <general@hadoop.apache.org>; Wed, 14 Apr 2010 08:39:53 -0500 (CDT)
Received: from imailfargo.navteq.com (imailfargo.navteq.com [10.6.10.130]) by fargo-spamfw.navteq.com with ESMTP id E2nTUvylN58lE8bs for <general@hadoop.apache.org>; Wed, 14 Apr 2010 08:39:53 -0500 (CDT)
Received: from hq-ex-ht01.ad.navteq.com ([10.8.222.51])
	by imailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id o3EDdq2X014035
	for <general@hadoop.apache.org>; Wed, 14 Apr 2010 08:39:52 -0500
Received: from hq-ex-mb02.ad.navteq.com ([fe80::880d:318e:e860:a8db]) by
 hq-ex-ht01.ad.navteq.com ([fe80::55a6:5b94:5c00:fe39%13]) with mapi; Wed, 14
 Apr 2010 08:39:52 -0500
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 14 Apr 2010 08:39:50 -0500
Subject: Custom Class Loader for Hadoop M/R jobs?
Thread-Topic: Custom Class Loader for Hadoop M/R jobs?
Thread-Index: Acrb1/TEma6fZVD1SWq9JJCGy5HxHw==
Message-ID: <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E2CB@hq-ex-mb02.ad.navteq.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_ABC24175AFD3BE4DA15F4CD375ED413D0607C7E2CBhqexmb02adnav_"
MIME-Version: 1.0

--_000_ABC24175AFD3BE4DA15F4CD375ED413D0607C7E2CBhqexmb02adnav_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1

Hi,

Ok, here's a bit of a bizarre  issue...

How do you handle class collisions between Hadoop and your m/r job which =
calls other 3rd party classes.

An example: Hadoop has an older version of an open source jar in its /lib=
 directory. You're interfacing with a 3rd party OS tool that uses a later=
 release of the same jar.

You can modify the classpath, and that might work. But the better way is =
to create a Custom Class Loader. (Non-trivial)

Looking at the Configuration class, it looks like there are a couple of m=
ethods that deal with loading a class in to the configuration so that the=
 m/r jobs can have access to them on each node.

Is this the correct intended use, or am I missing something?
Has anyone done something like this?

Thx

-Mike

Michael Segel
Architect,  R&D
NAVTEQ
425 West Randolph Street
Chicago, IL 60606
(T)  +1 312-780-3432
(C)  +1 312-952-8175
www.navteq.com<http://www.navteq.com/>



The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.

--_000_ABC24175AFD3BE4DA15F4CD375ED413D0607C7E2CBhqexmb02adnav_--

From general-return-1377-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 14:49:04 2010
Return-Path: <general-return-1377-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98762 invoked from network); 14 Apr 2010 14:49:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 14:49:04 -0000
Received: (qmail 8945 invoked by uid 500); 14 Apr 2010 14:49:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8799 invoked by uid 500); 14 Apr 2010 14:49:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8787 invoked by uid 99); 14 Apr 2010 14:49:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 14:49:02 +0000
X-ASF-Spam-Status: No, hits=-0.2 required=10.0
	tests=AWL,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.218.211] (HELO mail-bw0-f211.google.com) (209.85.218.211)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 14:48:56 +0000
Received: by bwz3 with SMTP id 3so217621bwz.11
        for <multiple recipients>; Wed, 14 Apr 2010 07:48:34 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.78.211 with HTTP; Wed, 14 Apr 2010 07:48:33 -0700 (PDT)
In-Reply-To: <u2n1267dd3b1004132346gae872ec3g54d3e8b3d221c68e@mail.gmail.com>
References: <u2n1267dd3b1004132346gae872ec3g54d3e8b3d221c68e@mail.gmail.com>
Date: Wed, 14 Apr 2010 07:48:33 -0700
Received: by 10.204.140.70 with SMTP id h6mr8689551bku.1.1271256513906; Wed, 
	14 Apr 2010 07:48:33 -0700 (PDT)
Message-ID: <n2sac79ea401004140748ybca41f35gcd01db13ce4faff9@mail.gmail.com>
Subject: Re: Subprojects and TLP status
From: Tom White <tom@cloudera.com>
To: general@hadoop.apache.org
Cc: private@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

+1

Thank you for writing this, Chris.

Tom

On Tue, Apr 13, 2010 at 11:46 PM, Chris Douglas <cdouglas@apache.org> wrote:
> Most of Hadoop's subprojects have discussed becoming top-level Apache
> projects (TLPs) in the last few weeks. Most have expressed a desire to
> remain in Hadoop. The salient parts of the discussions I've read tend
> to focus on three aspects: a technical dependence on Hadoop,
> additional overhead as a TLP, and visibility both within the Hadoop
> ecosystem and in the open source community generally.
>
> Life as a TLP: this is not much harder than being a Hadoop subproject,
> and the Apache preferences being tossed around- particularly
> "insufficiently diverse"- are not blockers. Every subproject needs to
> write a section of the report Hadoop sends to the board; almost the
> same report, sent to a new address. The initial cost is similarly
> light: copy bylaws, send a few notes to INFRA, and follow some
> directions. I think the estimated costs are far higher than they will
> be in practice. Inertia is a powerful force, but it should be
> overcome. The directions are here, and should not intimidating:
>
> http://apache.org/dev/project-creation.html
>
> Visibility: the Hadoop site does not need to change. For each
> subproject, we can literally change the hyperlinks to point to the new
> page and be done. Long-term, linking to all ASF projects that run on
> Hadoop from a prominent page is something we all want. So particularly
> in the medium-term that most are considering: visibility through the
> website will not change. Each subproject will still be linked from the
> front page.
>
> Hadoop would not be nearly as popular as it is without Zookeeper,
> HBase, Hive, and Pig. All statistics on work in shared MapReduce
> clusters show that users vastly prefer running Pig and Hive queries to
> writing MapReduce jobs. HBase continues to push features in HDFS that
> increase its adoption and relevance outside MapReduce, while sharing
> some of its NoSQL limelight. Zookeeper is not only a linchpin in real
> workloads, but many proposals for future features require it. The
> bottom line is that MapReduce and HDFS need these projects for
> visibility and adoption in precisely the same way. I don't think
> separate TLPs will uncouple the broader community from one another.
>
> Technical dependence: this has two dimensions. First, influencing
> MapReduce and HDFS. This is nonsense. Earning influence by
> contributing to a subproject is the only way to push code changes;
> nobody from any of these projects has violated that by unilaterally
> committing to HDFS or MapReduce, anyway. And anyone cynical enough to
> believe that MapReduce and HDFS would deliberately screw over or
> ignore dependent projects because they don't have PMC members is
> plainly unsuited to community-driven development. I understand that
> these projects need to protect their users, but lobbying rights are
> not an actual benefit.
>
> Second, being a coherent part of the Hadoop ecosystem. It is (mostly)
> true that Hadoop current offers a set of mutually compatible
> frameworks. It is not true that moving them to separate Apache
> projects would make solutions less coherent or affect existing or
> future users at all. The cohesion between projects' governance is
> sufficiently weak to justify independent units, but the real
> dependencies between the projects are strong enough to keep us engaged
> with one another. And it's not as if other projects- Cascading, for
> example- aren't also organisms adapted and specialized for life in
> Hadoop.
>
> Arguments on technical dependence are ignoring the nature of the
> existing interactions. Besides, weak technical dependencies are not a
> necessary prerequisite for a subproject's independence.
>
> As for what was *not* said in these discussions, there is no argument
> that every one of these subprojects has a distinct, autonomous
> community. There was also no argument that the Hadoop PMC offers any
> valuable oversight, given that the representatives of its fiefdoms are
> too consumed by provincial matters to participate in neighboring
> governance. Most releases I've voted on: I run the unit tests, check
> the signature, verify the checksum, and know literally nothing else
> about its content. I have often never heard the names of many proposed
> committers and even some proposed PMC members. Right now, subprojects
> with enough PMC members essentially vote out their own releases and
> vote in their own committers: TLPs in all but name.
>
> The Hadoop club- in conferences, meetups, technical debates, etc.- is
> broad, diverse, and intertwined, but communities of developers have
> already clustered around subprojects. Allowing that each cluster
> should govern itself is a dry, practical matter, not an existential
> crisis. -C
>

From general-return-1378-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 14:56:51 2010
Return-Path: <general-return-1378-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 3680 invoked from network); 14 Apr 2010 14:56:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 14:56:51 -0000
Received: (qmail 41012 invoked by uid 500); 14 Apr 2010 14:56:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40913 invoked by uid 500); 14 Apr 2010 14:56:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40905 invoked by uid 99); 14 Apr 2010 14:56:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 14:56:50 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.218.211] (HELO mail-bw0-f211.google.com) (209.85.218.211)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 14:56:43 +0000
Received: by bwz3 with SMTP id 3so226292bwz.11
        for <general@hadoop.apache.org>; Wed, 14 Apr 2010 07:56:22 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.78.211 with HTTP; Wed, 14 Apr 2010 07:56:17 -0700 (PDT)
In-Reply-To: <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E2CB@hq-ex-mb02.ad.navteq.com>
References: <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E2CB@hq-ex-mb02.ad.navteq.com>
Date: Wed, 14 Apr 2010 07:56:17 -0700
Received: by 10.204.35.75 with SMTP id o11mr8450184bkd.214.1271256977447; Wed, 
	14 Apr 2010 07:56:17 -0700 (PDT)
Message-ID: <i2yac79ea401004140756qd6dd3e3cz3ed8e8713c63da17@mail.gmail.com>
Subject: Re: Custom Class Loader for Hadoop M/R jobs?
From: Tom White <tom@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

I agree that it writing a custom classloader is non-trivial (and I
don't know if anyone has done this in the MapReduce context), so this
would best be handled by the framework. I've opened
https://issues.apache.org/jira/browse/MAPREDUCE-1700 for this.

Tom

On Wed, Apr 14, 2010 at 6:39 AM, Segel, Mike <msegel@navteq.com> wrote:
> Hi,
>
> Ok, here's a bit of a bizarre =A0issue...
>
> How do you handle class collisions between Hadoop and your m/r job which =
calls other 3rd party classes.
>
> An example: Hadoop has an older version of an open source jar in its /lib=
 directory. You're interfacing with a 3rd party OS tool that uses a later r=
elease of the same jar.
>
> You can modify the classpath, and that might work. But the better way is =
to create a Custom Class Loader. (Non-trivial)
>
> Looking at the Configuration class, it looks like there are a couple of m=
ethods that deal with loading a class in to the configuration so that the m=
/r jobs can have access to them on each node.
>
> Is this the correct intended use, or am I missing something?
> Has anyone done something like this?
>
> Thx
>
> -Mike
>
> Michael Segel
> Architect, =A0R&D
> NAVTEQ
> 425 West Randolph Street
> Chicago, IL 60606
> (T) =A0+1 312-780-3432
> (C) =A0+1 312-952-8175
> www.navteq.com<http://www.navteq.com/>
>
>
>
> The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above. =A0If you are =
not the intended recipient, you are hereby notified that any dissemination,=
 distribution, or copying of this communication, or any of its contents, is=
 strictly prohibited. =A0If you have received this communication in error, =
please notify the sender and delete/destroy the original message and any co=
py of it from your computer or paper files.
>

From general-return-1379-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 16:21:12 2010
Return-Path: <general-return-1379-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 65353 invoked from network); 14 Apr 2010 16:21:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 16:21:11 -0000
Received: (qmail 22921 invoked by uid 500); 14 Apr 2010 16:21:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22853 invoked by uid 500); 14 Apr 2010 16:21:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22833 invoked by uid 99); 14 Apr 2010 16:21:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 16:21:10 +0000
X-ASF-Spam-Status: No, hits=0.2 required=10.0
	tests=AWL,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 16:21:04 +0000
Received: from localhost ([10.72.168.172])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o3EGJqdB096170;
	Wed, 14 Apr 2010 09:19:54 -0700 (PDT)
Date: Wed, 14 Apr 2010 09:19:51 -0700
From: Konstantin Boudnik <cos@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Cc: "private@hadoop.apache.org" <private@hadoop.apache.org>
Subject: Re: Subprojects and TLP status
Message-ID: <20100414161949.GA28485@goodenter-lm.local>
References: <u2n1267dd3b1004132346gae872ec3g54d3e8b3d221c68e@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=koi8-r
Content-Disposition: inline
In-Reply-To: <u2n1267dd3b1004132346gae872ec3g54d3e8b3d221c68e@mail.gmail.com>
X-Organization: Grid computing (Hadoop)
User-Agent: Mutt/1.5.20 (2009-06-14)

Great summing up of the Hadoop business as it is today.

+1 

On Tue, Apr 13, 2010 at 11:46PM, Chris Douglas wrote:
> Most of Hadoop's subprojects have discussed becoming top-level Apache
> projects (TLPs) in the last few weeks. Most have expressed a desire to
> remain in Hadoop. The salient parts of the discussions I've read tend
> to focus on three aspects: a technical dependence on Hadoop,
> additional overhead as a TLP, and visibility both within the Hadoop
> ecosystem and in the open source community generally.
> 
> Life as a TLP: this is not much harder than being a Hadoop subproject,
> and the Apache preferences being tossed around- particularly
> "insufficiently diverse"- are not blockers. Every subproject needs to
> write a section of the report Hadoop sends to the board; almost the
> same report, sent to a new address. The initial cost is similarly
> light: copy bylaws, send a few notes to INFRA, and follow some
> directions. I think the estimated costs are far higher than they will
> be in practice. Inertia is a powerful force, but it should be
> overcome. The directions are here, and should not intimidating:
> 
> http://apache.org/dev/project-creation.html
> 
> Visibility: the Hadoop site does not need to change. For each
> subproject, we can literally change the hyperlinks to point to the new
> page and be done. Long-term, linking to all ASF projects that run on
> Hadoop from a prominent page is something we all want. So particularly
> in the medium-term that most are considering: visibility through the
> website will not change. Each subproject will still be linked from the
> front page.
> 
> Hadoop would not be nearly as popular as it is without Zookeeper,
> HBase, Hive, and Pig. All statistics on work in shared MapReduce
> clusters show that users vastly prefer running Pig and Hive queries to
> writing MapReduce jobs. HBase continues to push features in HDFS that
> increase its adoption and relevance outside MapReduce, while sharing
> some of its NoSQL limelight. Zookeeper is not only a linchpin in real
> workloads, but many proposals for future features require it. The
> bottom line is that MapReduce and HDFS need these projects for
> visibility and adoption in precisely the same way. I don't think
> separate TLPs will uncouple the broader community from one another.
> 
> Technical dependence: this has two dimensions. First, influencing
> MapReduce and HDFS. This is nonsense. Earning influence by
> contributing to a subproject is the only way to push code changes;
> nobody from any of these projects has violated that by unilaterally
> committing to HDFS or MapReduce, anyway. And anyone cynical enough to
> believe that MapReduce and HDFS would deliberately screw over or
> ignore dependent projects because they don't have PMC members is
> plainly unsuited to community-driven development. I understand that
> these projects need to protect their users, but lobbying rights are
> not an actual benefit.
> 
> Second, being a coherent part of the Hadoop ecosystem. It is (mostly)
> true that Hadoop current offers a set of mutually compatible
> frameworks. It is not true that moving them to separate Apache
> projects would make solutions less coherent or affect existing or
> future users at all. The cohesion between projects' governance is
> sufficiently weak to justify independent units, but the real
> dependencies between the projects are strong enough to keep us engaged
> with one another. And it's not as if other projects- Cascading, for
> example- aren't also organisms adapted and specialized for life in
> Hadoop.
> 
> Arguments on technical dependence are ignoring the nature of the
> existing interactions. Besides, weak technical dependencies are not a
> necessary prerequisite for a subproject's independence.
> 
> As for what was *not* said in these discussions, there is no argument
> that every one of these subprojects has a distinct, autonomous
> community. There was also no argument that the Hadoop PMC offers any
> valuable oversight, given that the representatives of its fiefdoms are
> too consumed by provincial matters to participate in neighboring
> governance. Most releases I've voted on: I run the unit tests, check
> the signature, verify the checksum, and know literally nothing else
> about its content. I have often never heard the names of many proposed
> committers and even some proposed PMC members. Right now, subprojects
> with enough PMC members essentially vote out their own releases and
> vote in their own committers: TLPs in all but name.
> 
> The Hadoop club- in conferences, meetups, technical debates, etc.- is
> broad, diverse, and intertwined, but communities of developers have
> already clustered around subprojects. Allowing that each cluster
> should govern itself is a dry, practical matter, not an existential
> crisis. -C

From general-return-1380-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 16:29:03 2010
Return-Path: <general-return-1380-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69318 invoked from network); 14 Apr 2010 16:29:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 16:29:02 -0000
Received: (qmail 33249 invoked by uid 500); 14 Apr 2010 16:29:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33217 invoked by uid 500); 14 Apr 2010 16:29:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33209 invoked by uid 99); 14 Apr 2010 16:29:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 16:29:01 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 16:28:56 +0000
Received: by wwb29 with SMTP id 29so204838wwb.35
        for <general@hadoop.apache.org>; Wed, 14 Apr 2010 09:28:35 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.158.5 with HTTP; Wed, 14 Apr 2010 09:28:35 -0700 (PDT)
In-Reply-To: <m2va7988b8e1004130809l40cde215icd5e1ab561b92f49@mail.gmail.com>
References: <q2ha7988b8e1004130746t8ab3e3dap5607f7f8e53053b8@mail.gmail.com>
	 <m2va7988b8e1004130809l40cde215icd5e1ab561b92f49@mail.gmail.com>
Date: Wed, 14 Apr 2010 09:28:35 -0700
Received: by 10.216.90.9 with SMTP id d9mr4463118wef.95.1271262515364; Wed, 14 
	Apr 2010 09:28:35 -0700 (PDT)
Message-ID: <u2l23d54121004140928v58286578u2d66ed08196722e0@mail.gmail.com>
Subject: Re: DBInputFormat number of mappers
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

If you're performing a simple import of an entire table, sqoop may
make your life easier. It gives you a reasonable command line client
for importing single tables or an entire database (provided there is a
JDBC driver available for it). Sqoop comes with Cloudera's
distribution for Hadoop or you can snag the source from
http://github.com/cloudera/sqoop

If sqoop isn't appealing for some reason, you can at least take a look
at the code and see what Aaron did under the hood.

On Tue, Apr 13, 2010 at 8:09 AM, Dan Harvey <dan.harvey@mendeley.com> wrote:
> Right, after sending this e-mail out that started working straight away with
> no changes... So setting the number of mappers in the code using :-
>
> job.getConfiguration().setInt("mapred.map.tasks", 4);
>
> allowed me to specify the number of splits/map tasks.
>
> Which lead me to the second problem I've been getting for awhile. When I
> start a hadoop job using DBInputFormat as the input if I use 5 splits say
> one will start straight away and the others will stay in the initializing
> until it is done then carry on one at a time. This doesn't happen all the
> time though and using the same code and database some will sometimes start
> in parallel!
>
> I've read this has happened to others before but no clear solution was found
> then.
>
> Has anyone else had this before or found a way to solve it?
>
> Thanks,
>
> On 13 April 2010 15:46, Dan Harvey <dan.harvey@mendeley.com> wrote:
>
>> Hi,
>>
>> I'm importing data from a mysql database using the DBInputFormat to go over
>> the rows in a table and put them into HBase with the mapper but I can't find
>> a way to increase the number of maps it splits the input into. I am running
>> this on a cluster where we have 5 nodes and each node has a maximum of 2 map
>> tasks. So for example if I set the number of rows to import to be 10,000,000
>> then there will only 2 maps tasks and use only two of the nodes..
>>
>> I've tried increasing the limit manually in the code with :
>>
>> job.getConfiguration().setInt("mapred.map.tasks", 4);
>>
>> increasing the number on the command line to set the same property, and
>> also increasing the number of map tasks per node.
>> But in all cases mapred.map.tasks is set to 2 in the job xml config file.
>>
>> I've had a look at the code and DBInputFormat splits the total number of
>> rows over mapred.map.tasks, so I'm guessing it's just getting that to
>> change.
>>
>> It would be great if anyone has any ideas what's going on?
>>
>> Thanks,
>>
>> --
>> Dan Harvey | Datamining Engineer
>> www.mendeley.com/profiles/dan-harvey
>>
>> Mendeley Limited | London, UK | www.mendeley.com
>> Registered in England and Wales | Company Number 6419015
>>
>
>
>
> --
> Dan Harvey | Datamining Engineer
> www.mendeley.com/profiles/dan-harvey
>
> Mendeley Limited | London, UK | www.mendeley.com
> Registered in England and Wales | Company Number 6419015
>



-- 
Eric Sammer
phone: +1-917-287-2675
twitter: esammer
data: www.cloudera.com

From general-return-1381-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 17:03:27 2010
Return-Path: <general-return-1381-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 89003 invoked from network); 14 Apr 2010 17:03:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 17:03:26 -0000
Received: (qmail 83730 invoked by uid 500); 14 Apr 2010 17:03:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83616 invoked by uid 500); 14 Apr 2010 17:03:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83608 invoked by uid 99); 14 Apr 2010 17:03:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 17:03:25 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=AWL,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [64.78.17.19] (HELO EXHUB018-4.exch018.msoutlookonline.net) (64.78.17.19)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 17:03:19 +0000
Received: from EXVMBX018-1.exch018.msoutlookonline.net ([64.78.17.47]) by
 EXHUB018-4.exch018.msoutlookonline.net ([64.78.17.19]) with mapi; Wed, 14 Apr
 2010 10:02:58 -0700
From: Scott Carey <scott@richrelevance.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 14 Apr 2010 10:01:38 -0700
Subject: Re: Custom Class Loader for Hadoop M/R jobs?
Thread-Topic: Custom Class Loader for Hadoop M/R jobs?
Thread-Index: Acrb9FSGrtsm/Ye1TKucQGBMPunIbQ==
Message-ID: <C97199E2-890B-4F00-A9D0-376327DB9D00@richrelevance.com>
References: <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E2CB@hq-ex-mb02.ad.navteq.com>
In-Reply-To: <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E2CB@hq-ex-mb02.ad.navteq.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Depending on what the dependency is, you might be able to just remove it fr=
om hadoop's lib directory on your cluster.

For me, Hadoop's later versions has jackson-1.0.1 in its lib directory and =
that breaks usage of Avro in a M/R job among other things.  However, the fe=
ature that uses this library is unimportant to me (configuration dump in JS=
ON format) so I just removed the jar.

-Scott
=20
On Apr 14, 2010, at 6:39 AM, Segel, Mike wrote:

> Hi,
>=20
> Ok, here's a bit of a bizarre  issue...
>=20
> How do you handle class collisions between Hadoop and your m/r job which =
calls other 3rd party classes.
>=20
> An example: Hadoop has an older version of an open source jar in its /lib=
 directory. You're interfacing with a 3rd party OS tool that uses a later r=
elease of the same jar.
>=20
> You can modify the classpath, and that might work. But the better way is =
to create a Custom Class Loader. (Non-trivial)
>=20
> Looking at the Configuration class, it looks like there are a couple of m=
ethods that deal with loading a class in to the configuration so that the m=
/r jobs can have access to them on each node.
>=20
> Is this the correct intended use, or am I missing something?
> Has anyone done something like this?
>=20
> Thx
>=20
> -Mike
>=20
> Michael Segel
> Architect,  R&D
> NAVTEQ
> 425 West Randolph Street
> Chicago, IL 60606
> (T)  +1 312-780-3432
> (C)  +1 312-952-8175
> www.navteq.com<http://www.navteq.com/>
>=20
>=20
>=20
> The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are no=
t the intended recipient, you are hereby notified that any dissemination, d=
istribution, or copying of this communication, or any of its contents, is s=
trictly prohibited.  If you have received this communication in error, plea=
se notify the sender and delete/destroy the original message and any copy o=
f it from your computer or paper files.


From general-return-1382-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 17:17:20 2010
Return-Path: <general-return-1382-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93699 invoked from network); 14 Apr 2010 17:17:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 17:17:20 -0000
Received: (qmail 3454 invoked by uid 500); 14 Apr 2010 17:17:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3415 invoked by uid 500); 14 Apr 2010 17:17:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3407 invoked by uid 99); 14 Apr 2010 17:17:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 17:17:19 +0000
X-ASF-Spam-Status: No, hits=1.3 required=10.0
	tests=AWL,HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 17:17:14 +0000
Received: from wlanvpn-mc2e-247-251.corp.yahoo.com (wlanvpn-mc2e-247-251.corp.yahoo.com [172.21.149.251])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o3EHFcbj051189
	for <general@hadoop.apache.org>; Wed, 14 Apr 2010 10:15:38 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:mime-version:
	subject:date:references:x-mailer;
	b=RpR/PlHYZSCxCHs4a6PN5ErArhJK/j0E82x/oWGpDe3y9Jy2BSKnPzbwZN+kshCy
Message-Id: <DB3048EA-F0C1-4C65-9780-786819121D40@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <p2s88f6e29a1004140401l5797704cl3287b884d39280da@mail.gmail.com>
Content-Type: multipart/alternative; boundary=Apple-Mail-1-900362466
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: JIRA snafu?
Date: Wed, 14 Apr 2010 10:15:38 -0700
References: <1CCD1C49-71A3-46B2-878E-74781E0815EE@yahoo-inc.com> <4BC59A65.20701@apache.org> <p2s88f6e29a1004140401l5797704cl3287b884d39280da@mail.gmail.com>
X-Mailer: Apple Mail (2.936)

--Apple-Mail-1-900362466
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


On Apr 14, 2010, at 4:01 AM, Bernd Fondermann wrote:

> On Wed, Apr 14, 2010 at 12:35, Steve Loughran <stevel@apache.org>  
> wrote:
>> Arun C Murthy wrote:
>>>
>>> I can no longer grant permission to Apache for my patches on JIRA.  
>>> Anyone
>>> else?
>>
>> the emergency release of jira doesn't have the extra legal stuff,  
>> until they
>> get that in you are being viewed as implicitly granting permissions.
>
> But it would help if contributors would explicitly comment that they
> grant permission.
> In a few months at least I won't be able to remember when and if JIRA
> had been down.
>

+1

Case in point: http://tinyurl.com/yyfnfz6

Arun
--Apple-Mail-1-900362466--

From general-return-1383-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 17:34:05 2010
Return-Path: <general-return-1383-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 3712 invoked from network); 14 Apr 2010 17:34:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 17:34:05 -0000
Received: (qmail 35772 invoked by uid 500); 14 Apr 2010 17:34:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 35583 invoked by uid 500); 14 Apr 2010 17:34:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 35575 invoked by uid 99); 14 Apr 2010 17:34:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 17:34:04 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of msegel@navteq.com designates 65.167.11.240 as permitted sender)
Received: from [65.167.11.240] (HELO xmailchi.navteq.com) (65.167.11.240)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 17:33:57 +0000
Received: from chicago-spamfw.navteq.com (imailchi [10.8.5.161])
	by xmailchi.navteq.com (8.13.8/8.13.6) with ESMTP id o3EHFUev004772
	for <general@hadoop.apache.org>; Wed, 14 Apr 2010 12:15:30 -0500
Received: from imailchi.hq.navteq.com (localhost [127.0.0.1])
	by chicago-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id 368FA3C44FE6
	for <general@hadoop.apache.org>; Wed, 14 Apr 2010 12:33:36 -0500 (CDT)
Received: from imailchi.hq.navteq.com (imailchi.hq.navteq.com [10.8.120.104]) by chicago-spamfw.navteq.com with ESMTP id xNpVKVEadViZYcbO for <general@hadoop.apache.org>; Wed, 14 Apr 2010 12:33:36 -0500 (CDT)
Received: from hq-ex-ht01.ad.navteq.com (hq-ex-ht01.ad.navteq.com [10.8.222.51])
	by imailchi.hq.navteq.com (8.12.9/8.12.9) with ESMTP id o3EHXaGq014896
	for <general@hadoop.apache.org>; Wed, 14 Apr 2010 12:33:36 -0500
Received: from hq-ex-mb02.ad.navteq.com ([fe80::880d:318e:e860:a8db]) by
 hq-ex-ht01.ad.navteq.com ([fe80::55a6:5b94:5c00:fe39%13]) with mapi; Wed, 14
 Apr 2010 12:33:30 -0500
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 14 Apr 2010 12:33:29 -0500
Subject: RE: Custom Class Loader for Hadoop M/R jobs?
Thread-Topic: Custom Class Loader for Hadoop M/R jobs?
Thread-Index: Acrb9FSGrtsm/Ye1TKucQGBMPunIbQAA9n4g
Message-ID: <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E54A@hq-ex-mb02.ad.navteq.com>
References: <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E2CB@hq-ex-mb02.ad.navteq.com>
 <C97199E2-890B-4F00-A9D0-376327DB9D00@richrelevance.com>
In-Reply-To: <C97199E2-890B-4F00-A9D0-376327DB9D00@richrelevance.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1
X-Virus-Checked: Checked by ClamAV on apache.org

Scott,

While that may work for a quick fix. Its not a good long term solution an=
d you then run in to a problem where you upgrade your hadoop release and =
the removed jar is replaced or if you replace the jar, it possible to get=
 overwritten.

In this specific instance, the Jackson libraries are not that important a=
nd they can be replaced.
But that doesn't mean that this issue won't come up again and its somethi=
ng you can't easily pop out and replace.

This is why I'm looking at custom class loading and trying to understand =
what can be accomplished with the methods in the Configuration class.=20

Thx

-Mike


-----Original Message-----
From: Scott Carey [mailto:scott@richrelevance.com]=20
Sent: Wednesday, April 14, 2010 12:02 PM
To: general@hadoop.apache.org
Subject: Re: Custom Class Loader for Hadoop M/R jobs?

Depending on what the dependency is, you might be able to just remove it =
from hadoop's lib directory on your cluster.

For me, Hadoop's later versions has jackson-1.0.1 in its lib directory an=
d that breaks usage of Avro in a M/R job among other things.  However, th=
e feature that uses this library is unimportant to me (configuration dump=
 in JSON format) so I just removed the jar.

-Scott
=20
On Apr 14, 2010, at 6:39 AM, Segel, Mike wrote:

> Hi,
>=20
> Ok, here's a bit of a bizarre  issue...
>=20
> How do you handle class collisions between Hadoop and your m/r job whic=
h calls other 3rd party classes.
>=20
> An example: Hadoop has an older version of an open source jar in its /l=
ib directory. You're interfacing with a 3rd party OS tool that uses a lat=
er release of the same jar.
>=20
> You can modify the classpath, and that might work. But the better way i=
s to create a Custom Class Loader. (Non-trivial)
>=20
> Looking at the Configuration class, it looks like there are a couple of=
 methods that deal with loading a class in to the configuration so that t=
he m/r jobs can have access to them on each node.
>=20
> Is this the correct intended use, or am I missing something?
> Has anyone done something like this?
>=20
> Thx
>=20
> -Mike
>=20
> Michael Segel
> Architect,  R&D
> NAVTEQ
> 425 West Randolph Street
> Chicago, IL 60606
> (T)  +1 312-780-3432
> (C)  +1 312-952-8175
> www.navteq.com<http://www.navteq.com/>
>=20
>=20
>=20
> The information contained in this communication may be CONFIDENTIAL and=
 is intended only for the use of the recipient(s) named above.  If you ar=
e not the intended recipient, you are hereby notified that any disseminat=
ion, distribution, or copying of this communication, or any of its conten=
ts, is strictly prohibited.  If you have received this communication in e=
rror, please notify the sender and delete/destroy the original message an=
d any copy of it from your computer or paper files.



The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.

From general-return-1384-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 17:55:53 2010
Return-Path: <general-return-1384-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19670 invoked from network); 14 Apr 2010 17:55:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 17:55:53 -0000
Received: (qmail 79982 invoked by uid 500); 14 Apr 2010 17:55:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79876 invoked by uid 500); 14 Apr 2010 17:55:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 79867 invoked by uid 99); 14 Apr 2010 17:55:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 17:55:52 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of invisiblesun@mac.com designates 17.148.16.105 as permitted sender)
Received: from [17.148.16.105] (HELO asmtpout030.mac.com) (17.148.16.105)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 17:55:46 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; charset=us-ascii
Received: from [172.27.35.143]
 (c-67-169-39-169.hsd1.ca.comcast.net [67.169.39.169])
 by asmtp030.mac.com (Sun Java(tm) System Messaging Server 6.3-8.01 (built Dec
 16 2008; 32bit)) with ESMTPSA id <0L0V0055OOG0GK60@asmtp030.mac.com> for
 general@hadoop.apache.org; Wed, 14 Apr 2010 10:55:14 -0700 (PDT)
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0
 reason=mlx engine=5.0.0-0908210000 definitions=main-1004140192
Subject: Re: Custom Class Loader for Hadoop M/R jobs?
From: Julia Smith <invisiblesun@mac.com>
In-reply-to:
 <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E54A@hq-ex-mb02.ad.navteq.com>
Date: Wed, 14 Apr 2010 10:55:11 -0700
Message-id: <19CC33EF-FA84-46CB-B61D-385CB92688A5@mac.com>
References:
 <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E2CB@hq-ex-mb02.ad.navteq.com>
 <C97199E2-890B-4F00-A9D0-376327DB9D00@richrelevance.com>
 <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E54A@hq-ex-mb02.ad.navteq.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1078)

This might be of use to you, as I had a related problem years ago:

http://www.developertutorials.com/tutorials/java/simplify-application-delivery-one-jar-050422/page5.html


The trick is a one-jar solution allows you to embed jar files and load classes from them. Rolling up all your jar variants in a single jar and then selectively loading the desired version might provide you a useful solution. The source is there, so you could start with a working classloader and perhaps apply a few tweaks.

On Apr 14, 2010, at 10:33 AM, Segel, Mike wrote:

> Scott,
> 
> While that may work for a quick fix. Its not a good long term solution and you then run in to a problem where you upgrade your hadoop release and the removed jar is replaced or if you replace the jar, it possible to get overwritten.
> 
> In this specific instance, the Jackson libraries are not that important and they can be replaced.
> But that doesn't mean that this issue won't come up again and its something you can't easily pop out and replace.
> 
> This is why I'm looking at custom class loading and trying to understand what can be accomplished with the methods in the Configuration class. 
> 
> Thx
> 
> -Mike
> 
> 
> -----Original Message-----
> From: Scott Carey [mailto:scott@richrelevance.com] 
> Sent: Wednesday, April 14, 2010 12:02 PM
> To: general@hadoop.apache.org
> Subject: Re: Custom Class Loader for Hadoop M/R jobs?
> 
> Depending on what the dependency is, you might be able to just remove it from hadoop's lib directory on your cluster.
> 
> For me, Hadoop's later versions has jackson-1.0.1 in its lib directory and that breaks usage of Avro in a M/R job among other things.  However, the feature that uses this library is unimportant to me (configuration dump in JSON format) so I just removed the jar.
> 
> -Scott
> 
> On Apr 14, 2010, at 6:39 AM, Segel, Mike wrote:
> 
>> Hi,
>> 
>> Ok, here's a bit of a bizarre  issue...
>> 
>> How do you handle class collisions between Hadoop and your m/r job which calls other 3rd party classes.
>> 
>> An example: Hadoop has an older version of an open source jar in its /lib directory. You're interfacing with a 3rd party OS tool that uses a later release of the same jar.
>> 
>> You can modify the classpath, and that might work. But the better way is to create a Custom Class Loader. (Non-trivial)
>> 
>> Looking at the Configuration class, it looks like there are a couple of methods that deal with loading a class in to the configuration so that the m/r jobs can have access to them on each node.
>> 
>> Is this the correct intended use, or am I missing something?
>> Has anyone done something like this?
>> 
>> Thx
>> 
>> -Mike
>> 
>> Michael Segel
>> Architect,  R&D
>> NAVTEQ
>> 425 West Randolph Street
>> Chicago, IL 60606
>> (T)  +1 312-780-3432
>> (C)  +1 312-952-8175
>> www.navteq.com<http://www.navteq.com/>
>> 
>> 
>> 
>> The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above.  If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited.  If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
> 
> 
> 
> The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above.  If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited.  If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.


From general-return-1385-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 18:09:53 2010
Return-Path: <general-return-1385-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31147 invoked from network); 14 Apr 2010 18:09:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 18:09:53 -0000
Received: (qmail 3893 invoked by uid 500); 14 Apr 2010 18:09:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3797 invoked by uid 500); 14 Apr 2010 18:09:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3789 invoked by uid 99); 14 Apr 2010 18:09:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 18:09:52 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=AWL,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [64.78.17.17] (HELO EXHUB018-2.exch018.msoutlookonline.net) (64.78.17.17)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 18:09:45 +0000
Received: from EXVMBX018-1.exch018.msoutlookonline.net ([64.78.17.47]) by
 EXHUB018-2.exch018.msoutlookonline.net ([64.78.17.17]) with mapi; Wed, 14 Apr
 2010 11:09:25 -0700
From: Scott Carey <scott@richrelevance.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 14 Apr 2010 11:08:06 -0700
Subject: Re: Custom Class Loader for Hadoop M/R jobs?
Thread-Topic: Custom Class Loader for Hadoop M/R jobs?
Thread-Index: Acrb/Z03qb+/Dhq2Q4uKqzSDobm4rA==
Message-ID: <42721ED4-DC28-47AF-A456-D3FF321013EF@richrelevance.com>
References: <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E2CB@hq-ex-mb02.ad.navteq.com>
 <C97199E2-890B-4F00-A9D0-376327DB9D00@richrelevance.com>
 <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E54A@hq-ex-mb02.ad.navteq.com>
In-Reply-To: <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E54A@hq-ex-mb02.ad.navteq.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

My long term suggestions are in https://issues.apache.org/jira/browse/MAPRE=
DUCE-1700.  The framework definitely needs to handle this and not place the=
 burden on users, IMO. But that won't help you in the short term.

Whether removing or replacing a Hadoop jar is an acceptable option to you (=
or others) in the short term is up to you.  Obviously, its not a great long=
 term solution but if you (or someone else) has to make it work ASAP, it mi=
ght be the only option.  In our case, we package our own rpm and have a few=
 custom patches to Hadoop so removing one jar is a trivial thing to do in t=
he short / medium term.

-Scott

On Apr 14, 2010, at 10:33 AM, Segel, Mike wrote:

> Scott,
>=20
> While that may work for a quick fix. Its not a good long term solution an=
d you then run in to a problem where you upgrade your hadoop release and th=
e removed jar is replaced or if you replace the jar, it possible to get ove=
rwritten.
>=20
> In this specific instance, the Jackson libraries are not that important a=
nd they can be replaced.
> But that doesn't mean that this issue won't come up again and its somethi=
ng you can't easily pop out and replace.
>=20
> This is why I'm looking at custom class loading and trying to understand =
what can be accomplished with the methods in the Configuration class.=20
>=20
> Thx
>=20
> -Mike
>=20
>=20
> -----Original Message-----
> From: Scott Carey [mailto:scott@richrelevance.com]=20
> Sent: Wednesday, April 14, 2010 12:02 PM
> To: general@hadoop.apache.org
> Subject: Re: Custom Class Loader for Hadoop M/R jobs?
>=20
> Depending on what the dependency is, you might be able to just remove it =
from hadoop's lib directory on your cluster.
>=20
> For me, Hadoop's later versions has jackson-1.0.1 in its lib directory an=
d that breaks usage of Avro in a M/R job among other things.  However, the =
feature that uses this library is unimportant to me (configuration dump in =
JSON format) so I just removed the jar.
>=20
> -Scott
>=20
> On Apr 14, 2010, at 6:39 AM, Segel, Mike wrote:
>=20
>> Hi,
>>=20
>> Ok, here's a bit of a bizarre  issue...
>>=20
>> How do you handle class collisions between Hadoop and your m/r job which=
 calls other 3rd party classes.
>>=20
>> An example: Hadoop has an older version of an open source jar in its /li=
b directory. You're interfacing with a 3rd party OS tool that uses a later =
release of the same jar.
>>=20
>> You can modify the classpath, and that might work. But the better way is=
 to create a Custom Class Loader. (Non-trivial)
>>=20
>> Looking at the Configuration class, it looks like there are a couple of =
methods that deal with loading a class in to the configuration so that the =
m/r jobs can have access to them on each node.
>>=20
>> Is this the correct intended use, or am I missing something?
>> Has anyone done something like this?
>>=20
>> Thx
>>=20
>> -Mike
>>=20
>> Michael Segel
>> Architect,  R&D
>> NAVTEQ
>> 425 West Randolph Street
>> Chicago, IL 60606
>> (T)  +1 312-780-3432
>> (C)  +1 312-952-8175
>> www.navteq.com<http://www.navteq.com/>
>>=20
>>=20
>>=20
>> The information contained in this communication may be CONFIDENTIAL and =
is intended only for the use of the recipient(s) named above.  If you are n=
ot the intended recipient, you are hereby notified that any dissemination, =
distribution, or copying of this communication, or any of its contents, is =
strictly prohibited.  If you have received this communication in error, ple=
ase notify the sender and delete/destroy the original message and any copy =
of it from your computer or paper files.
>=20
>=20
>=20
> The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are no=
t the intended recipient, you are hereby notified that any dissemination, d=
istribution, or copying of this communication, or any of its contents, is s=
trictly prohibited.  If you have received this communication in error, plea=
se notify the sender and delete/destroy the original message and any copy o=
f it from your computer or paper files.


From general-return-1386-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 18:20:20 2010
Return-Path: <general-return-1386-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37446 invoked from network); 14 Apr 2010 18:20:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 18:20:20 -0000
Received: (qmail 17984 invoked by uid 500); 14 Apr 2010 18:20:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17947 invoked by uid 500); 14 Apr 2010 18:20:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17939 invoked by uid 99); 14 Apr 2010 18:20:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 18:20:19 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of chris.cooper@navteq.com designates 65.167.11.240 as permitted sender)
Received: from [65.167.11.240] (HELO xmailchi.navteq.com) (65.167.11.240)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 18:20:12 +0000
Received: from chicago-spamfw.navteq.com (imailchi [10.8.5.161])
	by xmailchi.navteq.com (8.13.8/8.13.6) with ESMTP id o3EI1i9w006908
	for <general@hadoop.apache.org>; Wed, 14 Apr 2010 13:01:44 -0500
Received: from imailchi.hq.navteq.com (localhost [127.0.0.1])
	by chicago-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id 098894318401
	for <general@hadoop.apache.org>; Wed, 14 Apr 2010 13:19:49 -0500 (CDT)
Received: from imailchi.hq.navteq.com (imailchi.hq.navteq.com [10.8.120.104]) by chicago-spamfw.navteq.com with ESMTP id YsB8whFHrccGj39t for <general@hadoop.apache.org>; Wed, 14 Apr 2010 13:19:49 -0500 (CDT)
Received: from hq-ex-ht01.ad.navteq.com (hq-ex-ht01.ad.navteq.com [10.8.222.51])
	by imailchi.hq.navteq.com (8.12.9/8.12.9) with ESMTP id o3EIJnGq020175
	for <general@hadoop.apache.org>; Wed, 14 Apr 2010 13:19:50 -0500
Received: from hq-ex-mb01.ad.navteq.com ([fe80::1dd0:ea36:7d35:db66]) by
 hq-ex-ht01.ad.navteq.com ([fe80::55a6:5b94:5c00:fe39%13]) with mapi; Wed, 14
 Apr 2010 13:19:49 -0500
From: "Cooper, Chris" <chris.cooper@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 14 Apr 2010 13:19:49 -0500
Subject: RE: Custom Class Loader for Hadoop M/R jobs?
Thread-Topic: Custom Class Loader for Hadoop M/R jobs?
Thread-Index: Acrb/Z03qb+/Dhq2Q4uKqzSDobm4rAAAJeNg
Message-ID: <76A09B97FD0B434ABCD9A75F688CF7E50373D81E@hq-ex-mb01.ad.navteq.com>
References: <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E2CB@hq-ex-mb02.ad.navteq.com>
 <C97199E2-890B-4F00-A9D0-376327DB9D00@richrelevance.com>
 <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E54A@hq-ex-mb02.ad.navteq.com>
 <42721ED4-DC28-47AF-A456-D3FF321013EF@richrelevance.com>
In-Reply-To: <42721ED4-DC28-47AF-A456-D3FF321013EF@richrelevance.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1
X-Virus-Checked: Checked by ClamAV on apache.org

Scott,

I think the direction your comments in https://issues.apache.org/jira/bro=
wse/MAPREDUCE-1700 is spot on.  You should be looking at J2EE container c=
lass loader hierarchies.  I've attached a couple of good links that cover=
 this approach.


http://www.ibm.com/developerworks/websphere/library/techarticles/0112_deb=
oer/deboer.html
http://www.objectsource.com/j2eechapters/Ch21-ClassLoaders_and_J2EE.htm

I'm sure Mike and I would both be willing to work with you to contribute =
a solution if you're interested.

Best regards,

CC

-----Original Message-----
From: Scott Carey [mailto:scott@richrelevance.com]=20
Sent: Wednesday, April 14, 2010 1:08 PM
To: general@hadoop.apache.org
Subject: Re: Custom Class Loader for Hadoop M/R jobs?

My long term suggestions are in https://issues.apache.org/jira/browse/MAP=
REDUCE-1700.  The framework definitely needs to handle this and not place=
 the burden on users, IMO. But that won't help you in the short term.

Whether removing or replacing a Hadoop jar is an acceptable option to you=
 (or others) in the short term is up to you.  Obviously, its not a great =
long term solution but if you (or someone else) has to make it work ASAP,=
 it might be the only option.  In our case, we package our own rpm and ha=
ve a few custom patches to Hadoop so removing one jar is a trivial thing =
to do in the short / medium term.

-Scott

On Apr 14, 2010, at 10:33 AM, Segel, Mike wrote:

> Scott,
>=20
> While that may work for a quick fix. Its not a good long term solution =
and you then run in to a problem where you upgrade your hadoop release an=
d the removed jar is replaced or if you replace the jar, it possible to g=
et overwritten.
>=20
> In this specific instance, the Jackson libraries are not that important=
 and they can be replaced.
> But that doesn't mean that this issue won't come up again and its somet=
hing you can't easily pop out and replace.
>=20
> This is why I'm looking at custom class loading and trying to understan=
d what can be accomplished with the methods in the Configuration class.=20
>=20
> Thx
>=20
> -Mike
>=20
>=20
> -----Original Message-----
> From: Scott Carey [mailto:scott@richrelevance.com]=20
> Sent: Wednesday, April 14, 2010 12:02 PM
> To: general@hadoop.apache.org
> Subject: Re: Custom Class Loader for Hadoop M/R jobs?
>=20
> Depending on what the dependency is, you might be able to just remove i=
t from hadoop's lib directory on your cluster.
>=20
> For me, Hadoop's later versions has jackson-1.0.1 in its lib directory =
and that breaks usage of Avro in a M/R job among other things.  However, =
the feature that uses this library is unimportant to me (configuration du=
mp in JSON format) so I just removed the jar.
>=20
> -Scott
>=20
> On Apr 14, 2010, at 6:39 AM, Segel, Mike wrote:
>=20
>> Hi,
>>=20
>> Ok, here's a bit of a bizarre  issue...
>>=20
>> How do you handle class collisions between Hadoop and your m/r job whi=
ch calls other 3rd party classes.
>>=20
>> An example: Hadoop has an older version of an open source jar in its /=
lib directory. You're interfacing with a 3rd party OS tool that uses a la=
ter release of the same jar.
>>=20
>> You can modify the classpath, and that might work. But the better way =
is to create a Custom Class Loader. (Non-trivial)
>>=20
>> Looking at the Configuration class, it looks like there are a couple o=
f methods that deal with loading a class in to the configuration so that =
the m/r jobs can have access to them on each node.
>>=20
>> Is this the correct intended use, or am I missing something?
>> Has anyone done something like this?
>>=20
>> Thx
>>=20
>> -Mike
>>=20
>> Michael Segel
>> Architect,  R&D
>> NAVTEQ
>> 425 West Randolph Street
>> Chicago, IL 60606
>> (T)  +1 312-780-3432
>> (C)  +1 312-952-8175
>> www.navteq.com<http://www.navteq.com/>
>>=20
>>=20
>>=20
>> The information contained in this communication may be CONFIDENTIAL an=
d is intended only for the use of the recipient(s) named above.  If you a=
re not the intended recipient, you are hereby notified that any dissemina=
tion, distribution, or copying of this communication, or any of its conte=
nts, is strictly prohibited.  If you have received this communication in =
error, please notify the sender and delete/destroy the original message a=
nd any copy of it from your computer or paper files.
>=20
>=20
>=20
> The information contained in this communication may be CONFIDENTIAL and=
 is intended only for the use of the recipient(s) named above.  If you ar=
e not the intended recipient, you are hereby notified that any disseminat=
ion, distribution, or copying of this communication, or any of its conten=
ts, is strictly prohibited.  If you have received this communication in e=
rror, please notify the sender and delete/destroy the original message an=
d any copy of it from your computer or paper files.



The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.

From general-return-1387-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 20:30:01 2010
Return-Path: <general-return-1387-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81376 invoked from network); 14 Apr 2010 20:30:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 20:30:01 -0000
Received: (qmail 15605 invoked by uid 500); 14 Apr 2010 20:29:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15542 invoked by uid 500); 14 Apr 2010 20:29:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15300 invoked by uid 99); 14 Apr 2010 20:29:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 20:29:59 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=FREEMAIL_FROM,FRT_ADOBE2,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 74.125.92.27 as permitted sender)
Received: from [74.125.92.27] (HELO qw-out-2122.google.com) (74.125.92.27)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 20:29:52 +0000
Received: by qw-out-2122.google.com with SMTP id 8so223157qwh.35
        for <general@hadoop.apache.org>; Wed, 14 Apr 2010 13:29:32 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:received:message-id:subject
         :from:to:content-type:content-transfer-encoding;
        bh=jJpi4DAfLgk3fueXPg2WT9yFzK1WWuXVC096WQZ2PQ4=;
        b=J3pjjhsnB72H4xI8WcHvykWevVagEfNnlWDHViBmyu4+ABsCf3USDA1CYukqC7loLH
         pGeaNlffbRvIsVQGF/SBrtEFgj00yMOt2qsu5L/CtbaTBmCEO9fSAlHAdTmkIZnFyXtY
         AVHrF9eHOzMbXZmOYg6xoLpPtaKekX6GpxUf0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=q9VBrlF9tZUZTasrVvk9yhY+WNpIVvhMmzIitVoLi2ZHMZg2Pfv8yg4BQ4fC5ITWM6
         +RptogVSByKKj2a86E5+XCCVEQgqhhujB/RLpY/IJPH6RmaKK8mmko9FNTmIMQi9vw+A
         +18gXLRy/OBWIz8ufIHsxLSwaIchvsQYcnqzQ=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
Received: by 10.229.18.11 with HTTP; Wed, 14 Apr 2010 13:28:46 -0700 (PDT)
In-Reply-To: <z2m7c962aed1004141257o7ab2135bnf3d00da31715b0db@mail.gmail.com>
References: <m2l7c962aed1004120856hddd9e41dt77f0ba3e259c9171@mail.gmail.com>
	 <BA087D3D-D948-48A9-9989-D54F5AF7C4EF@adobe.com>
	 <z2m7c962aed1004141257o7ab2135bnf3d00da31715b0db@mail.gmail.com>
Date: Wed, 14 Apr 2010 13:28:46 -0700
X-Google-Sender-Auth: 4451f0d2a7cd72bb
Received: by 10.229.227.5 with SMTP id iy5mr2272948qcb.29.1271276927003; Wed, 
	14 Apr 2010 13:28:47 -0700 (PDT)
Message-ID: <j2x7c962aed1004141328m883a6e41pd793f2cb4d5c96d5@mail.gmail.com>
Subject: Re: [VOTE] HBase as TLP?
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

This passes, with 14 +1 votes and no -1 votes.

Owen, can you please add this resolution to the agenda for next week's
board meeting?

Thanks,
St.Ack

On Wed, Apr 14, 2010 at 12:57 PM, Stack <stack@duboce.net> wrote:
> This passes, with 14 +1 votes and no -1 votes.
>
> Owen, can you please add this resolution to the agenda for next week's
> board meeting?
>
> Thanks,
> St.Ack
>
> On Tue, Apr 13, 2010 at 12:25 PM, Cosmin Lehene <clehene@adobe.com> wrote=
:
>> +1
>>
>>
>> On Apr 12, 2010, at 6:56 PM, Stack wrote:
>>
>>> Please vote as to whether you think HBase should become a top-level
>>> Apache project.
>>>
>>> I've included below a draft board resolution. =A0It lists all current
>>> active HBase committers as initial members of the project management
>>> committee (PMC) and myself, Michael Stack, as the initial chair.
>>>
>>> Do the good people of Hadoop support sending this request on to the Had=
oop PMC?
>>>
>>> Yours,
>>> St.Ack
>>>
>>> ------------
>>>
>>> =A0 X. Establish the Apache HBase Project
>>>
>>> =A0 =A0 =A0WHEREAS, the Board of Directors deems it to be in the best
>>> =A0 =A0 =A0interests of the Foundation and consistent with the
>>> =A0 =A0 =A0Foundation's purpose to establish a Project Management
>>> =A0 =A0 =A0Committee charged with the creation and maintenance of
>>> =A0 =A0 =A0open-source software related to data serialization
>>> =A0 =A0 =A0for distribution at no charge to the public.
>>>
>>> =A0 =A0 =A0NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>>> =A0 =A0 =A0Committee (PMC), to be known as the "Apache HBase Project",
>>> =A0 =A0 =A0be and hereby is established pursuant to Bylaws of the
>>> =A0 =A0 =A0Foundation; and be it further
>>>
>>> =A0 =A0 =A0RESOLVED, that the Apache HBase Project be and hereby is
>>> =A0 =A0 =A0responsible for the creation and maintenance of software
>>> =A0 =A0 =A0related to a distributed database; and be it further
>>>
>>> =A0 =A0 =A0RESOLVED, that the office of "Vice President, Apache HBase" =
be
>>> =A0 =A0 =A0and hereby is created, the person holding such office to
>>> =A0 =A0 =A0serve at the direction of the Board of Directors as the chai=
r
>>> =A0 =A0 =A0of the Apache HBase Project, and to have primary responsibil=
ity
>>> =A0 =A0 =A0for management of the projects within the scope of
>>> =A0 =A0 =A0responsibility of the Apache HBase Project; and be it furthe=
r
>>>
>>> =A0 =A0 =A0RESOLVED, that the persons listed immediately below be and
>>> =A0 =A0 =A0hereby are appointed to serve as the initial members of the
>>> =A0 =A0 =A0Apache HBase Project:
>>>
>>> =A0 =A0 =A0 =A0* Michael Stack <stack@apache.org>
>>> =A0 =A0 =A0 =A0* Jonathan Gray <jgray@apache.org>
>>> =A0 =A0 =A0 =A0* Andrew Purtell <apurtell@apache.org>
>>> =A0 =A0 =A0 =A0* John-Daniel Cryans <jdcryans@apache.org>
>>> =A0 =A0 =A0 =A0* Ryan Rawson <rawson@apache.org>
>>> =A0 =A0 =A0 =A0* Lars George <larsgeorge@apache.org>
>>>
>>> =A0 =A0 =A0NOW, THEREFORE, BE IT FURTHER RESOLVED, that Michael Stack
>>> =A0 =A0 =A0be appointed to the office of Vice President, Apache HBase, =
to
>>> =A0 =A0 =A0serve in accordance with and subject to the direction of the
>>> =A0 =A0 =A0Board of Directors and the Bylaws of the Foundation until
>>> =A0 =A0 =A0death, resignation, retirement, removal or disqualification,
>>> =A0 =A0 =A0or until a successor is appointed; and be it further
>>>
>>> =A0 =A0 =A0RESOLVED, that the initial Apache HBase PMC be and hereby is
>>> =A0 =A0 =A0tasked with the creation of a set of bylaws intended to
>>> =A0 =A0 =A0encourage open development and increased participation in th=
e
>>> =A0 =A0 =A0Apache HBase Project; and be it further
>>>
>>> =A0 =A0 =A0RESOLVED, that the Apache HBase Project be and hereby
>>> =A0 =A0 =A0is tasked with the migration and rationalization of the Apac=
he
>>> =A0 =A0 =A0Hadoop HBase sub-project; and be it further
>>>
>>> =A0 =A0 =A0RESOLVED, that all responsibilities pertaining to the Apache
>>> =A0 =A0 =A0Hadoop HBase sub-project encumbered upon the
>>> =A0 =A0 =A0Apache Hadoop Project are hereafter discharged.
>>
>>
>

From general-return-1388-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 11:09:43 2010
Return-Path: <general-return-1388-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 79145 invoked from network); 15 Apr 2010 11:09:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 11:09:43 -0000
Received: (qmail 3427 invoked by uid 500); 15 Apr 2010 11:09:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3284 invoked by uid 500); 15 Apr 2010 11:09:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3271 invoked by uid 99); 15 Apr 2010 11:09:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 11:09:38 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [203.101.103.66] (HELO krishna.sisl.co.in) (203.101.103.66)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 11:09:27 +0000
Received: from kaveri.sisl.co.in (kaveri.sisl.co.in [132.186.192.4])
	by krishna.sisl.co.in (Postfix) with ESMTP id 8B7AB13CC29
	for <general@hadoop.apache.org>; Thu, 15 Apr 2010 16:45:41 +0530 (IST)
Received: from INBLRK77N2MSX.in002.siemens.net (inblrk77n2msx.in002.siemens.net [132.186.221.163])
	by kaveri.sisl.co.in (Postfix) with ESMTP id 273CD1F800E
	for <general@hadoop.apache.org>; Thu, 15 Apr 2010 16:38:34 +0530 (IST)
Received: from INBLRK77M1MSX.in002.siemens.net ([132.186.221.151]) by
 INBLRK77N2MSX.in002.siemens.net ([132.186.221.163]) with mapi; Thu, 15 Apr
 2010 16:36:14 +0530
From: "Goel, Mohit IN BLR SISL" <mohit.goel@siemens.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 15 Apr 2010 16:36:15 +0530
Subject: Query regarding Hadoop and cloud infrastructure
Thread-Topic: Query regarding Hadoop and cloud infrastructure
Thread-Index: Acrci6p/C1PlZUJkSNmnZxaz1u52aQ==
Message-ID: <2658E54B540D284981EA57E6A549EA70A1777C16F4@INBLRK77M1MSX.in002.siemens.net>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
x-tm-as-product-ver: SMEX-8.0.0.4125-3.800.1026-17320.004
x-tm-as-result: No--48.794300-8.000000-2
x-tm-as-user-approved-sender: No
x-tm-as-user-blocked-sender: No
Content-Type: multipart/alternative;
	boundary="_000_2658E54B540D284981EA57E6A549EA70A1777C16F4INBLRK77M1MSX_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_2658E54B540D284981EA57E6A549EA70A1777C16F4INBLRK77M1MSX_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hello,

I have a general query regarding usage of Hadoop with my cloud infrastructu=
re. I am trying to achieve scaling up and scaling down in cloud using Hadoo=
p.

I have set up a cloud infrastructure which creates images consists of OS an=
d applications. To access user applications, instance of image has to launc=
h. Now I want to make this running or launched instance scalable based on s=
ome condition like -

a)       If no. of users who are accessing the application which is hosting=
 in cloud (i.e. in instance) are more then it should run one more instance =
of image and if no. of users are less then instances should be terminated.
b)       If CPU usage is more then one more instance of image should run or=
 if CPU usage is less then it should terminate the instance.

Can I achieve these goals using Hadoop?

Please guide me and let me know the possible solution.

Thanks in advance.

--------------------------------------------------------
With best regards,
Mohit Goel

________________________________
Important notice: This e-mail and any attachment there to contains corporat=
e proprietary information. If you have received it by mistake, please notif=
y us immediately by reply e-mail and delete this e-mail and its attachments=
 from your system.
Thank You.

--_000_2658E54B540D284981EA57E6A549EA70A1777C16F4INBLRK77M1MSX_--

From general-return-1389-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 12:16:03 2010
Return-Path: <general-return-1389-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96386 invoked from network); 15 Apr 2010 12:16:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 12:16:03 -0000
Received: (qmail 70241 invoked by uid 500); 15 Apr 2010 12:16:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70133 invoked by uid 500); 15 Apr 2010 12:16:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70125 invoked by uid 99); 15 Apr 2010 12:16:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 12:16:01 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=AWL,RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 12:15:54 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 65B86B7E0C
	for <general@hadoop.apache.org>; Thu, 15 Apr 2010 13:15:32 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id KucwxeAlx4Tu for <general@hadoop.apache.org>;
	Thu, 15 Apr 2010 13:15:31 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 75BEBB7E0B
	for <general@hadoop.apache.org>; Thu, 15 Apr 2010 13:15:31 +0100 (BST)
MailScanner-NULL-Check: 1271938519.53005@UxXeEOUoo5w/zQtDHi95zg
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o3FCFIEY010695
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Thu, 15 Apr 2010 13:15:19 +0100 (BST)
Message-ID: <4BC70356.5050700@apache.org>
Date: Thu, 15 Apr 2010 13:15:18 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100228)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Query regarding Hadoop and cloud infrastructure
References: <2658E54B540D284981EA57E6A549EA70A1777C16F4@INBLRK77M1MSX.in002.siemens.net>
In-Reply-To: <2658E54B540D284981EA57E6A549EA70A1777C16F4@INBLRK77M1MSX.in002.siemens.net>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o3FCFIEY010695
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

Goel, Mohit IN BLR SISL wrote:
> Hello,
> 
> I have a general query regarding usage of Hadoop with my cloud infrastructure. I am trying to achieve scaling up and scaling down in cloud using Hadoop.
> 
> I have set up a cloud infrastructure which creates images consists of OS and applications. To access user applications, instance of image has to launch. Now I want to make this running or launched instance scalable based on some condition like -
> 
> a)       If no. of users who are accessing the application which is hosting in cloud (i.e. in instance) are more then it should run one more instance of image and if no. of users are less then instances should be terminated.
> b)       If CPU usage is more then one more instance of image should run or if CPU usage is less then it should terminate the instance.
> 
> Can I achieve these goals using Hadoop?
> 

1. Hadoop on Demand, HOD, does some of this
2. Hadoop on EC2 does some of this
3. I've been doing some of this, too; I have some slides up where I 
discuss issues
http://www.slideshare.net/steve_l/new-roles-for-the-cloud

One funny for Hadoop is that it likes locality, and it likes machines 
with TB of physical storage, which doesn't fit in quite as well with the 
VM-on-demand story. If you look at my slides, you can see that 
everything expects stable hostnames, reacts to failure by blacklisting, 
not by killing the VM and creating a new one with the same HDFS volumes 
mounted. There is room for improvement!



From general-return-1390-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 13:09:45 2010
Return-Path: <general-return-1390-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17800 invoked from network); 15 Apr 2010 13:09:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 13:09:45 -0000
Received: (qmail 32544 invoked by uid 500); 15 Apr 2010 13:09:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32480 invoked by uid 500); 15 Apr 2010 13:09:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32472 invoked by uid 99); 15 Apr 2010 13:09:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 13:09:43 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dan.harvey@mendeley.com designates 209.85.218.211 as permitted sender)
Received: from [209.85.218.211] (HELO mail-bw0-f211.google.com) (209.85.218.211)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 13:09:39 +0000
Received: by bwz3 with SMTP id 3so1263242bwz.11
        for <general@hadoop.apache.org>; Thu, 15 Apr 2010 06:09:17 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.239.157.131 with HTTP; Thu, 15 Apr 2010 06:09:16 -0700 (PDT)
In-Reply-To: <u2l23d54121004140928v58286578u2d66ed08196722e0@mail.gmail.com>
References: <q2ha7988b8e1004130746t8ab3e3dap5607f7f8e53053b8@mail.gmail.com>
	 <m2va7988b8e1004130809l40cde215icd5e1ab561b92f49@mail.gmail.com>
	 <u2l23d54121004140928v58286578u2d66ed08196722e0@mail.gmail.com>
Date: Thu, 15 Apr 2010 14:09:16 +0100
Received: by 10.239.177.212 with SMTP id w20mr13283hbf.126.1271336956752; Thu, 
	15 Apr 2010 06:09:16 -0700 (PDT)
Message-ID: <s2ha7988b8e1004150609kb3e67dbct8fe49a2aa5b8bebd@mail.gmail.com>
Subject: Re: DBInputFormat number of mappers
From: Dan Harvey <dan.harvey@mendeley.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636499eb3b274920484463356

--001636499eb3b274920484463356
Content-Type: text/plain; charset=ISO-8859-1

Unfortunately the tables I'm importing need application logic to extract the
data full which means I can use sqoop, but I will be using that for all the
simpler tables.

I think I found the problem was with locking the tables in mysql which
caused the mappers to run one at a time, so I've re-wrote the code so I'm
not locking tables now and that's fixed it from happening completely. It's
odd that it didn't always go one at a time though but I guess that might
just been to do with how the mysql server was dealing with the queries at
different times.

Thanks,

On 14 April 2010 17:28, Eric Sammer <esammer@cloudera.com> wrote:

> If you're performing a simple import of an entire table, sqoop may
> make your life easier. It gives you a reasonable command line client
> for importing single tables or an entire database (provided there is a
> JDBC driver available for it). Sqoop comes with Cloudera's
> distribution for Hadoop or you can snag the source from
> http://github.com/cloudera/sqoop
>
> If sqoop isn't appealing for some reason, you can at least take a look
> at the code and see what Aaron did under the hood.
>
> On Tue, Apr 13, 2010 at 8:09 AM, Dan Harvey <dan.harvey@mendeley.com>
> wrote:
> > Right, after sending this e-mail out that started working straight away
> with
> > no changes... So setting the number of mappers in the code using :-
> >
> > job.getConfiguration().setInt("mapred.map.tasks", 4);
> >
> > allowed me to specify the number of splits/map tasks.
> >
> > Which lead me to the second problem I've been getting for awhile. When I
> > start a hadoop job using DBInputFormat as the input if I use 5 splits say
> > one will start straight away and the others will stay in the initializing
> > until it is done then carry on one at a time. This doesn't happen all the
> > time though and using the same code and database some will sometimes
> start
> > in parallel!
> >
> > I've read this has happened to others before but no clear solution was
> found
> > then.
> >
> > Has anyone else had this before or found a way to solve it?
> >
> > Thanks,
> >
> > On 13 April 2010 15:46, Dan Harvey <dan.harvey@mendeley.com> wrote:
> >
> >> Hi,
> >>
> >> I'm importing data from a mysql database using the DBInputFormat to go
> over
> >> the rows in a table and put them into HBase with the mapper but I can't
> find
> >> a way to increase the number of maps it splits the input into. I am
> running
> >> this on a cluster where we have 5 nodes and each node has a maximum of 2
> map
> >> tasks. So for example if I set the number of rows to import to be
> 10,000,000
> >> then there will only 2 maps tasks and use only two of the nodes..
> >>
> >> I've tried increasing the limit manually in the code with :
> >>
> >> job.getConfiguration().setInt("mapred.map.tasks", 4);
> >>
> >> increasing the number on the command line to set the same property, and
> >> also increasing the number of map tasks per node.
> >> But in all cases mapred.map.tasks is set to 2 in the job xml config
> file.
> >>
> >> I've had a look at the code and DBInputFormat splits the total number of
> >> rows over mapred.map.tasks, so I'm guessing it's just getting that to
> >> change.
> >>
> >> It would be great if anyone has any ideas what's going on?
> >>
> >> Thanks,
> >>
> >> --
> >> Dan Harvey | Datamining Engineer
> >> www.mendeley.com/profiles/dan-harvey
> >>
> >> Mendeley Limited | London, UK | www.mendeley.com
> >> Registered in England and Wales | Company Number 6419015
> >>
> >
> >
> >
> > --
> > Dan Harvey | Datamining Engineer
> > www.mendeley.com/profiles/dan-harvey
> >
> > Mendeley Limited | London, UK | www.mendeley.com
> > Registered in England and Wales | Company Number 6419015
> >
>
>
>
> --
> Eric Sammer
> phone: +1-917-287-2675
> twitter: esammer
> data: www.cloudera.com
>



-- 
Dan Harvey | Datamining Engineer
www.mendeley.com/profiles/dan-harvey

Mendeley Limited | London, UK | www.mendeley.com
Registered in England and Wales | Company Number 6419015

--001636499eb3b274920484463356--

From general-return-1391-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 13:23:27 2010
Return-Path: <general-return-1391-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21256 invoked from network); 15 Apr 2010 13:23:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 13:23:27 -0000
Received: (qmail 57885 invoked by uid 500); 15 Apr 2010 13:23:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57737 invoked by uid 500); 15 Apr 2010 13:23:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57729 invoked by uid 99); 15 Apr 2010 13:23:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 13:23:26 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 13:23:16 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id C17B2B7E09
	for <general@hadoop.apache.org>; Thu, 15 Apr 2010 14:22:55 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id BWShxJr+axBR for <general@hadoop.apache.org>;
	Thu, 15 Apr 2010 14:22:55 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 092C6B7E03
	for <general@hadoop.apache.org>; Thu, 15 Apr 2010 14:22:54 +0100 (BST)
MailScanner-NULL-Check: 1271942563.1266@aBbCMr4Q2ITmCmkX7ixagA
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o3FDMd07012709
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Thu, 15 Apr 2010 14:22:39 +0100 (BST)
Message-ID: <4BC7131F.10500@apache.org>
Date: Thu, 15 Apr 2010 14:22:39 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100228)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Custom Class Loader for Hadoop M/R jobs?
References: <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E2CB@hq-ex-mb02.ad.navteq.com> <C97199E2-890B-4F00-A9D0-376327DB9D00@richrelevance.com> <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E54A@hq-ex-mb02.ad.navteq.com> <42721ED4-DC28-47AF-A456-D3FF321013EF@richrelevance.com>
In-Reply-To: <42721ED4-DC28-47AF-A456-D3FF321013EF@richrelevance.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o3FDMd07012709
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

Scott Carey wrote:
> In our case, we package our own rpm and have a few custom patches to Hadoop so removing one jar is a trivial thing to do in the short / medium term.

Added my thoughts on CLs to the MAPREDUCE-1700 bugrep.

I do my own RPMs too, Ivy driven, tested by installing onto local VMs.

There is scope for having a hadoop-redist project that contains 
everything needed to create RPMs or debs of Hadoop releases, with your 
own JARs in. volunteers to do this are welcome...




From general-return-1392-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 14:30:51 2010
Return-Path: <general-return-1392-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 41312 invoked from network); 15 Apr 2010 14:30:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 14:30:50 -0000
Received: (qmail 90861 invoked by uid 500); 15 Apr 2010 14:30:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90816 invoked by uid 500); 15 Apr 2010 14:30:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90808 invoked by uid 99); 15 Apr 2010 14:30:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 14:30:49 +0000
X-ASF-Spam-Status: No, hits=-0.1 required=10.0
	tests=AWL,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of msegel@navteq.com designates 65.167.11.240 as permitted sender)
Received: from [65.167.11.240] (HELO xmailchi.navteq.com) (65.167.11.240)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 14:30:44 +0000
Received: from chicago-spamfw.navteq.com (imailchi [10.8.5.161])
	by xmailchi.navteq.com (8.13.8/8.13.6) with ESMTP id o3FECFJc006138
	for <general@hadoop.apache.org>; Thu, 15 Apr 2010 09:12:15 -0500
Received: from imailchi.hq.navteq.com (localhost [127.0.0.1])
	by chicago-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id CCD93431A0D5
	for <general@hadoop.apache.org>; Thu, 15 Apr 2010 09:30:22 -0500 (CDT)
Received: from imailchi.hq.navteq.com (imailchi.hq.navteq.com [10.8.120.104]) by chicago-spamfw.navteq.com with ESMTP id ioYjae9Ii2EWEXdS for <general@hadoop.apache.org>; Thu, 15 Apr 2010 09:30:22 -0500 (CDT)
Received: from hq-ex-ht01.ad.navteq.com (hq-ex-ht01.ad.navteq.com [10.8.222.51])
	by imailchi.hq.navteq.com (8.12.9/8.12.9) with ESMTP id o3FEUMGq023027
	for <general@hadoop.apache.org>; Thu, 15 Apr 2010 09:30:22 -0500
Received: from hq-ex-mb02.ad.navteq.com ([fe80::880d:318e:e860:a8db]) by
 hq-ex-ht01.ad.navteq.com ([fe80::55a6:5b94:5c00:fe39%13]) with mapi; Thu, 15
 Apr 2010 09:30:22 -0500
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 15 Apr 2010 09:30:21 -0500
Subject: RE: Query regarding Hadoop and cloud infrastructure
Thread-Topic: Query regarding Hadoop and cloud infrastructure
Thread-Index: AcrclWvyFX02vJhrS5CafCyi5wEZqwAD2gsg
Message-ID: <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E95A@hq-ex-mb02.ad.navteq.com>
References: <2658E54B540D284981EA57E6A549EA70A1777C16F4@INBLRK77M1MSX.in002.siemens.net>
 <4BC70356.5050700@apache.org>
In-Reply-To: <4BC70356.5050700@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1

Steve,=20
Outside of a EC2 or a commercial site which sells time on a 'cloud', I wo=
uld argue against trying to do HOD or build a dynamic cloud for a corpora=
te environment.

Corporate clouds tend to be static in terms of usage. Meaning that they a=
re being built for a task and any changes are not dynamic enough to justi=
fy HOD.

I sat through a presentation from Sun. A nice guy, but in the end, I and =
others thought it was a way to make Sun's hardware (Sorry err I mean Orac=
le) relevant in the Hadoop world. Its counter to the concept of developin=
g 'white box' commodity hardware.

I'm not sold on virtualization, but its just my opinion and not necessari=
ly shared by anyone, which means I need to make the following statement:

The opinions expressed in this post are mine and mine alone. They do not =
reflect the opinions or position of my client, or my employer. Any resemb=
lance to a rational coherent thought is pure coincidence.=20

-Mike

-----Original Message-----
From: Steve Loughran [mailto:stevel@apache.org]=20
Sent: Thursday, April 15, 2010 7:15 AM
To: general@hadoop.apache.org
Subject: Re: Query regarding Hadoop and cloud infrastructure

Goel, Mohit IN BLR SISL wrote:
> Hello,
>=20
> I have a general query regarding usage of Hadoop with my cloud infrastr=
ucture. I am trying to achieve scaling up and scaling down in cloud using=
 Hadoop.
>=20
> I have set up a cloud infrastructure which creates images consists of O=
S and applications. To access user applications, instance of image has to=
 launch. Now I want to make this running or launched instance scalable ba=
sed on some condition like -
>=20
> a)       If no. of users who are accessing the application which is hos=
ting in cloud (i.e. in instance) are more then it should run one more ins=
tance of image and if no. of users are less then instances should be term=
inated.
> b)       If CPU usage is more then one more instance of image should ru=
n or if CPU usage is less then it should terminate the instance.
>=20
> Can I achieve these goals using Hadoop?
>=20

1. Hadoop on Demand, HOD, does some of this
2. Hadoop on EC2 does some of this
3. I've been doing some of this, too; I have some slides up where I=20
discuss issues
http://www.slideshare.net/steve_l/new-roles-for-the-cloud

One funny for Hadoop is that it likes locality, and it likes machines=20
with TB of physical storage, which doesn't fit in quite as well with the=20
VM-on-demand story. If you look at my slides, you can see that=20
everything expects stable hostnames, reacts to failure by blacklisting,=20
not by killing the VM and creating a new one with the same HDFS volumes=20
mounted. There is room for improvement!




The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.

From general-return-1393-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 15:17:40 2010
Return-Path: <general-return-1393-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58121 invoked from network); 15 Apr 2010 15:17:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 15:17:40 -0000
Received: (qmail 70626 invoked by uid 500); 15 Apr 2010 15:17:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70593 invoked by uid 500); 15 Apr 2010 15:17:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70584 invoked by uid 99); 15 Apr 2010 15:17:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 15:17:39 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of earcam@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 15:17:33 +0000
Received: by wwb29 with SMTP id 29so913696wwb.35
        for <general@hadoop.apache.org>; Thu, 15 Apr 2010 08:17:11 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :from:date:received:message-id:subject:to:content-type;
        bh=AXqyPS1HzI8Bo8AgPVaRH0AxwmTMs9URgnCLO/ZM0l0=;
        b=Ntse061lsxUDDCwrRuNsilIHL+JEvs5PFra6P96Jdlj6bL+aCvlPebE3uwbDDR8Zgi
         FGiUVVx6xuNg/972xdM0WbZVvGUIjNcfSakP0b/Kz0KhFw+bejG1kGhIMb3N6kIf1rVR
         8g7+d6wEzEJtEKab1Te+KN2tjdMl2a2QZzf90=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=fEHn5p5908vO9YRFrMu3sTv6LjiwREh4RHhdosxTwmA1ajQ6rQKSbdCH1BEUXIow6G
         InaxHpmL4B0eUL/iPyfHTFlR1zfjW6oi0j9DwsYMSAnISYIvqmocIptrDeD0L3BE4afE
         LRYRSejLZT3CYmEIOVhrXZ5vEPgCj4wFZLpXY=
MIME-Version: 1.0
Received: by 10.216.49.84 with HTTP; Thu, 15 Apr 2010 08:16:49 -0700 (PDT)
In-Reply-To: <76A09B97FD0B434ABCD9A75F688CF7E50373D81E@hq-ex-mb01.ad.navteq.com>
References: <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E2CB@hq-ex-mb02.ad.navteq.com> 
	<C97199E2-890B-4F00-A9D0-376327DB9D00@richrelevance.com> <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E54A@hq-ex-mb02.ad.navteq.com> 
	<42721ED4-DC28-47AF-A456-D3FF321013EF@richrelevance.com> <76A09B97FD0B434ABCD9A75F688CF7E50373D81E@hq-ex-mb01.ad.navteq.com>
From: Caspar MacRae <earcam@gmail.com>
Date: Thu, 15 Apr 2010 16:16:49 +0100
Received: by 10.216.89.81 with SMTP id b59mr227765wef.97.1271344630739; Thu, 
	15 Apr 2010 08:17:10 -0700 (PDT)
Message-ID: <w2xe5058b6a1004150816md06db24dxac42fa65a3a49739@mail.gmail.com>
Subject: Re: Custom Class Loader for Hadoop M/R jobs?
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d77c7c1a33eb048447fd6e
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d77c7c1a33eb048447fd6e
Content-Type: text/plain; charset=UTF-8

Hi all,

I think the problem(s) lies deeper and should be solved at more fundamental
level: OSGi (http://www.osgi.org)

This has the *classloader*, *modularity* and *distribution* management
maturity that IMHO Hadoop clearly needs (from what I know, albeit circa
1.9).

It's 10 years old, not headline app-server-tastic, nor flavour of the month
http://java.dzone.com/articles/osgi-feast-or-famine - but that's the point,
this proven tech.  And it'll eventually be present at the lowest levels of
Java, eg. project jigsaw: http://openjdk.java.net/projects/jigsaw/ which is
a precursor to completing JSR-291

<http://markmail.org/message/5fjx7pzq6kmwagch> A number of people have tried
to introduce OSGi to Hadoop but it seems their efforts *may* have been
ignored by those in the meritocratic circle of power - this is a real shame,
perhaps Jira voting is the way to draw attention to this?  You could vote on
this ticket that's over 2 years old...
https://issues.apache.org/jira/browse/MAPREDUCE-243

OSGi can easily solve the classloading and lifecycle management issues in
Hadoop, and brings a lot more besides.  Can someone please explain to me the
rationale for continuing to ignore such an obvious and elegant solution?


Best regards,
Caspar
<http://techdistrict.kirkk.com/2010/02/26/osgi-devcon-slides/>


On 14 April 2010 19:19, Cooper, Chris <chris.cooper@navteq.com> wrote:

> Scott,
>
> I think the direction your comments in
> https://issues.apache.org/jira/browse/MAPREDUCE-1700 is spot on.  You
> should be looking at J2EE container class loader hierarchies.  I've attached
> a couple of good links that cover this approach.
>
>
>
> http://www.ibm.com/developerworks/websphere/library/techarticles/0112_deboer/deboer.html
> http://www.objectsource.com/j2eechapters/Ch21-ClassLoaders_and_J2EE.htm
>
> I'm sure Mike and I would both be willing to work with you to contribute a
> solution if you're interested.
>
> Best regards,
>
> CC
>
> -----Original Message-----
> From: Scott Carey [mailto:scott@richrelevance.com]
> Sent: Wednesday, April 14, 2010 1:08 PM
> To: general@hadoop.apache.org
> Subject: Re: Custom Class Loader for Hadoop M/R jobs?
>
> My long term suggestions are in
> https://issues.apache.org/jira/browse/MAPREDUCE-1700.  The framework
> definitely needs to handle this and not place the burden on users, IMO. But
> that won't help you in the short term.
>
> Whether removing or replacing a Hadoop jar is an acceptable option to you
> (or others) in the short term is up to you.  Obviously, its not a great long
> term solution but if you (or someone else) has to make it work ASAP, it
> might be the only option.  In our case, we package our own rpm and have a
> few custom patches to Hadoop so removing one jar is a trivial thing to do in
> the short / medium term.
>
> -Scott
>
> On Apr 14, 2010, at 10:33 AM, Segel, Mike wrote:
>
> > Scott,
> >
> > While that may work for a quick fix. Its not a good long term solution
> and you then run in to a problem where you upgrade your hadoop release and
> the removed jar is replaced or if you replace the jar, it possible to get
> overwritten.
> >
> > In this specific instance, the Jackson libraries are not that important
> and they can be replaced.
> > But that doesn't mean that this issue won't come up again and its
> something you can't easily pop out and replace.
> >
> > This is why I'm looking at custom class loading and trying to understand
> what can be accomplished with the methods in the Configuration class.
> >
> > Thx
> >
> > -Mike
> >
> >
> > -----Original Message-----
> > From: Scott Carey [mailto:scott@richrelevance.com]
> > Sent: Wednesday, April 14, 2010 12:02 PM
> > To: general@hadoop.apache.org
> > Subject: Re: Custom Class Loader for Hadoop M/R jobs?
> >
> > Depending on what the dependency is, you might be able to just remove it
> from hadoop's lib directory on your cluster.
> >
> > For me, Hadoop's later versions has jackson-1.0.1 in its lib directory
> and that breaks usage of Avro in a M/R job among other things.  However, the
> feature that uses this library is unimportant to me (configuration dump in
> JSON format) so I just removed the jar.
> >
> > -Scott
> >
> > On Apr 14, 2010, at 6:39 AM, Segel, Mike wrote:
> >
> >> Hi,
> >>
> >> Ok, here's a bit of a bizarre  issue...
> >>
> >> How do you handle class collisions between Hadoop and your m/r job which
> calls other 3rd party classes.
> >>
> >> An example: Hadoop has an older version of an open source jar in its
> /lib directory. You're interfacing with a 3rd party OS tool that uses a
> later release of the same jar.
> >>
> >> You can modify the classpath, and that might work. But the better way is
> to create a Custom Class Loader. (Non-trivial)
> >>
> >> Looking at the Configuration class, it looks like there are a couple of
> methods that deal with loading a class in to the configuration so that the
> m/r jobs can have access to them on each node.
> >>
> >> Is this the correct intended use, or am I missing something?
> >> Has anyone done something like this?
> >>
> >> Thx
> >>
> >> -Mike
> >>
> >> Michael Segel
> >> Architect,  R&D
> >> NAVTEQ
> >> 425 West Randolph Street
> >> Chicago, IL 60606
> >> (T)  +1 312-780-3432
> >> (C)  +1 312-952-8175
> >> www.navteq.com<http://www.navteq.com/>
> >>
> >>
> >>
> >> The information contained in this communication may be CONFIDENTIAL and
> is intended only for the use of the recipient(s) named above.  If you are
> not the intended recipient, you are hereby notified that any dissemination,
> distribution, or copying of this communication, or any of its contents, is
> strictly prohibited.  If you have received this communication in error,
> please notify the sender and delete/destroy the original message and any
> copy of it from your computer or paper files.
> >
> >
> >
> > The information contained in this communication may be CONFIDENTIAL and
> is intended only for the use of the recipient(s) named above.  If you are
> not the intended recipient, you are hereby notified that any dissemination,
> distribution, or copying of this communication, or any of its contents, is
> strictly prohibited.  If you have received this communication in error,
> please notify the sender and delete/destroy the original message and any
> copy of it from your computer or paper files.
>
>
>
> The information contained in this communication may be CONFIDENTIAL and is
> intended only for the use of the recipient(s) named above.  If you are not
> the intended recipient, you are hereby notified that any dissemination,
> distribution, or copying of this communication, or any of its contents, is
> strictly prohibited.  If you have received this communication in error,
> please notify the sender and delete/destroy the original message and any
> copy of it from your computer or paper files.
>

--0016e6d77c7c1a33eb048447fd6e--

From general-return-1394-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 16:21:39 2010
Return-Path: <general-return-1394-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 75817 invoked from network); 15 Apr 2010 16:21:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 16:21:38 -0000
Received: (qmail 66309 invoked by uid 500); 15 Apr 2010 16:21:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66263 invoked by uid 500); 15 Apr 2010 16:21:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66255 invoked by uid 99); 15 Apr 2010 16:21:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 16:21:37 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.92.24] (HELO qw-out-2122.google.com) (74.125.92.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 16:21:31 +0000
Received: by qw-out-2122.google.com with SMTP id 8so596133qwh.35
        for <general@hadoop.apache.org>; Thu, 15 Apr 2010 09:21:10 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.92.20 with HTTP; Thu, 15 Apr 2010 09:20:50 -0700 (PDT)
In-Reply-To: <s2ha7988b8e1004150609kb3e67dbct8fe49a2aa5b8bebd@mail.gmail.com>
References: <q2ha7988b8e1004130746t8ab3e3dap5607f7f8e53053b8@mail.gmail.com> 
	<m2va7988b8e1004130809l40cde215icd5e1ab561b92f49@mail.gmail.com> 
	<u2l23d54121004140928v58286578u2d66ed08196722e0@mail.gmail.com> 
	<s2ha7988b8e1004150609kb3e67dbct8fe49a2aa5b8bebd@mail.gmail.com>
From: Aaron Kimball <aaron@cloudera.com>
Date: Thu, 15 Apr 2010 09:20:50 -0700
Received: by 10.229.98.129 with SMTP id q1mr156094qcn.100.1271348470245; Thu, 
	15 Apr 2010 09:21:10 -0700 (PDT)
Message-ID: <g2yd6d7c4411004150920lb255c5bs819321d7fa697da0@mail.gmail.com>
Subject: Re: DBInputFormat number of mappers
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364274e7f46853048448e1dd
X-Virus-Checked: Checked by ClamAV on apache.org

--0016364274e7f46853048448e1dd
Content-Type: text/plain; charset=ISO-8859-1

Hi Dan,

It's also worth pointing out that DBInputFormat's queries are written in
such a way as to make parallelism more likely to hurt than to help. Each
mapper submits a query to the database that does a full table scan followed
by an ORDER BY clause which has to run database-side. Only after this part
of the plan is complete does it select the appropriate output data slice
with LIMIT and OFFSET.

The moral of this story: If you're using CDH, consider using
DataDrivenDBInputFormat instead, which generates queries that actually run
in linear time :) (DataDrivenDBIF is also in Apache trunk, so those of you
running the straight Apache source will get it in the next release.)

Also, if you're using Sqoop, consider just importing the data with
--as-sequencefile so that the records are dumped in a way that fully
preserves everything, including binary data. Then run a second MapReduce job
that runs your extraction code over that input.

- Aaron


On Thu, Apr 15, 2010 at 6:09 AM, Dan Harvey <dan.harvey@mendeley.com> wrote:

> Unfortunately the tables I'm importing need application logic to extract
> the
> data full which means I can use sqoop, but I will be using that for all the
> simpler tables.
>
> I think I found the problem was with locking the tables in mysql which
> caused the mappers to run one at a time, so I've re-wrote the code so I'm
> not locking tables now and that's fixed it from happening completely. It's
> odd that it didn't always go one at a time though but I guess that might
> just been to do with how the mysql server was dealing with the queries at
> different times.
>
> Thanks,
>
> On 14 April 2010 17:28, Eric Sammer <esammer@cloudera.com> wrote:
>
> > If you're performing a simple import of an entire table, sqoop may
> > make your life easier. It gives you a reasonable command line client
> > for importing single tables or an entire database (provided there is a
> > JDBC driver available for it). Sqoop comes with Cloudera's
> > distribution for Hadoop or you can snag the source from
> > http://github.com/cloudera/sqoop
> >
> > If sqoop isn't appealing for some reason, you can at least take a look
> > at the code and see what Aaron did under the hood.
> >
> > On Tue, Apr 13, 2010 at 8:09 AM, Dan Harvey <dan.harvey@mendeley.com>
> > wrote:
> > > Right, after sending this e-mail out that started working straight away
> > with
> > > no changes... So setting the number of mappers in the code using :-
> > >
> > > job.getConfiguration().setInt("mapred.map.tasks", 4);
> > >
> > > allowed me to specify the number of splits/map tasks.
> > >
> > > Which lead me to the second problem I've been getting for awhile. When
> I
> > > start a hadoop job using DBInputFormat as the input if I use 5 splits
> say
> > > one will start straight away and the others will stay in the
> initializing
> > > until it is done then carry on one at a time. This doesn't happen all
> the
> > > time though and using the same code and database some will sometimes
> > start
> > > in parallel!
> > >
> > > I've read this has happened to others before but no clear solution was
> > found
> > > then.
> > >
> > > Has anyone else had this before or found a way to solve it?
> > >
> > > Thanks,
> > >
> > > On 13 April 2010 15:46, Dan Harvey <dan.harvey@mendeley.com> wrote:
> > >
> > >> Hi,
> > >>
> > >> I'm importing data from a mysql database using the DBInputFormat to go
> > over
> > >> the rows in a table and put them into HBase with the mapper but I
> can't
> > find
> > >> a way to increase the number of maps it splits the input into. I am
> > running
> > >> this on a cluster where we have 5 nodes and each node has a maximum of
> 2
> > map
> > >> tasks. So for example if I set the number of rows to import to be
> > 10,000,000
> > >> then there will only 2 maps tasks and use only two of the nodes..
> > >>
> > >> I've tried increasing the limit manually in the code with :
> > >>
> > >> job.getConfiguration().setInt("mapred.map.tasks", 4);
> > >>
> > >> increasing the number on the command line to set the same property,
> and
> > >> also increasing the number of map tasks per node.
> > >> But in all cases mapred.map.tasks is set to 2 in the job xml config
> > file.
> > >>
> > >> I've had a look at the code and DBInputFormat splits the total number
> of
> > >> rows over mapred.map.tasks, so I'm guessing it's just getting that to
> > >> change.
> > >>
> > >> It would be great if anyone has any ideas what's going on?
> > >>
> > >> Thanks,
> > >>
> > >> --
> > >> Dan Harvey | Datamining Engineer
> > >> www.mendeley.com/profiles/dan-harvey
> > >>
> > >> Mendeley Limited | London, UK | www.mendeley.com
> > >> Registered in England and Wales | Company Number 6419015
> > >>
> > >
> > >
> > >
> > > --
> > > Dan Harvey | Datamining Engineer
> > > www.mendeley.com/profiles/dan-harvey
> > >
> > > Mendeley Limited | London, UK | www.mendeley.com
> > > Registered in England and Wales | Company Number 6419015
> > >
> >
> >
> >
> > --
> > Eric Sammer
> > phone: +1-917-287-2675
> > twitter: esammer
> > data: www.cloudera.com
> >
>
>
>
> --
> Dan Harvey | Datamining Engineer
> www.mendeley.com/profiles/dan-harvey
>
> Mendeley Limited | London, UK | www.mendeley.com
> Registered in England and Wales | Company Number 6419015
>

--0016364274e7f46853048448e1dd--

From general-return-1395-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 16:28:08 2010
Return-Path: <general-return-1395-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77550 invoked from network); 15 Apr 2010 16:28:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 16:28:08 -0000
Received: (qmail 79519 invoked by uid 500); 15 Apr 2010 16:28:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79484 invoked by uid 500); 15 Apr 2010 16:28:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 79476 invoked by uid 99); 15 Apr 2010 16:28:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 16:28:07 +0000
X-ASF-Spam-Status: No, hits=-0.1 required=10.0
	tests=AWL,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of msegel@navteq.com designates 65.167.11.240 as permitted sender)
Received: from [65.167.11.240] (HELO xmailchi.navteq.com) (65.167.11.240)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 16:28:03 +0000
Received: from chicago-spamfw.navteq.com (imailchi [10.8.5.161])
	by xmailchi.navteq.com (8.13.8/8.13.6) with ESMTP id o3FG9Ysq010501
	for <general@hadoop.apache.org>; Thu, 15 Apr 2010 11:09:34 -0500
Received: from imailchi.hq.navteq.com (localhost [127.0.0.1])
	by chicago-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id 1AC4D3FD9752
	for <general@hadoop.apache.org>; Thu, 15 Apr 2010 11:27:42 -0500 (CDT)
Received: from imailchi.hq.navteq.com (imailchi.hq.navteq.com [10.8.120.104]) by chicago-spamfw.navteq.com with ESMTP id 0QIUNuVzjXhZNJuI for <general@hadoop.apache.org>; Thu, 15 Apr 2010 11:27:42 -0500 (CDT)
Received: from hq-ex-ht01.ad.navteq.com (hq-ex-ht01.ad.navteq.com [10.8.222.51])
	by imailchi.hq.navteq.com (8.12.9/8.12.9) with ESMTP id o3FGRgGq006133
	for <general@hadoop.apache.org>; Thu, 15 Apr 2010 11:27:42 -0500
Received: from hq-ex-mb02.ad.navteq.com ([fe80::880d:318e:e860:a8db]) by
 hq-ex-ht01.ad.navteq.com ([fe80::55a6:5b94:5c00:fe39%13]) with mapi; Thu, 15
 Apr 2010 11:27:41 -0500
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 15 Apr 2010 11:27:39 -0500
Subject: RE: Custom Class Loader for Hadoop M/R jobs?
Thread-Topic: Custom Class Loader for Hadoop M/R jobs?
Thread-Index: Acrcrsv1HP7aIy/rQ2KtQAKO5HdmwAACCp0A
Message-ID: <ABC24175AFD3BE4DA15F4CD375ED413D0607CD62ED@hq-ex-mb02.ad.navteq.com>
References: <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E2CB@hq-ex-mb02.ad.navteq.com>
 	<C97199E2-890B-4F00-A9D0-376327DB9D00@richrelevance.com>
 <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E54A@hq-ex-mb02.ad.navteq.com>
 	<42721ED4-DC28-47AF-A456-D3FF321013EF@richrelevance.com>
 <76A09B97FD0B434ABCD9A75F688CF7E50373D81E@hq-ex-mb01.ad.navteq.com>
 <w2xe5058b6a1004150816md06db24dxac42fa65a3a49739@mail.gmail.com>
In-Reply-To: <w2xe5058b6a1004150816md06db24dxac42fa65a3a49739@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-cr-puzzleid: {A9C4C54B-E4F3-4DDC-9DC7-AA1AB08D51BC}
x-cr-hashedpuzzle: BQ8o CvJN DHmm Dd3U Df3d FqyH GE4I GXqT Gd25 G6Z6 HQEb
 IEkw Kxwd K6Fd K8DI
 LIzr;1;ZwBlAG4AZQByAGEAbABAAGgAYQBkAG8AbwBwAC4AYQBwAGEAYwBoAGUALgBvAHIAZwA=;Sosha1_v1;7;{A9C4C54B-E4F3-4DDC-9DC7-AA1AB08D51BC};bQBzAGUAZwBlAGwAQABjAGgAaQAuAG4AYQB2AHQAZQBxAC4AYwBvAG0A;Thu,
 15 Apr 2010 16:27:39
 GMT;UgBFADoAIABDAHUAcwB0AG8AbQAgAEMAbABhAHMAcwAgAEwAbwBhAGQAZQByACAAZgBvAHIAIABIAGEAZABvAG8AcAAgAE0ALwBSACAAagBvAGIAcwA/AA==
acceptlanguage: en-US
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1

Caspar,

While IANAL, I did a first blush read of OSGi's Specification License, Version 1.0. on the website link: http://www.osgi.org/Specifications/Licensing as well as the license on the download page: http://www.osgi.org/Download/Release4V42?info=nothanks.

In a nutshell, OSGi's Specification License, Version 1.0. is incompatible with Apache's. 

That alone can explain why Apache projects won't touch OSGi.

>From a more practical issue, this problem is solved in other Apache projects, so why not borrow from another Apache project like Tomcat?

Does that make sense? It appears OSGi is a non-starter.

HTH

-Mike


-----Original Message-----
From: Caspar MacRae [mailto:earcam@gmail.com] 
Sent: Thursday, April 15, 2010 10:17 AM
To: general@hadoop.apache.org
Subject: Re: Custom Class Loader for Hadoop M/R jobs?

Hi all,

I think the problem(s) lies deeper and should be solved at more fundamental
level: OSGi (http://www.osgi.org)

This has the *classloader*, *modularity* and *distribution* management
maturity that IMHO Hadoop clearly needs (from what I know, albeit circa
1.9).

It's 10 years old, not headline app-server-tastic, nor flavour of the month
http://java.dzone.com/articles/osgi-feast-or-famine - but that's the point,
this proven tech.  And it'll eventually be present at the lowest levels of
Java, eg. project jigsaw: http://openjdk.java.net/projects/jigsaw/ which is
a precursor to completing JSR-291

<http://markmail.org/message/5fjx7pzq6kmwagch> A number of people have tried
to introduce OSGi to Hadoop but it seems their efforts *may* have been
ignored by those in the meritocratic circle of power - this is a real shame,
perhaps Jira voting is the way to draw attention to this?  You could vote on
this ticket that's over 2 years old...
https://issues.apache.org/jira/browse/MAPREDUCE-243

OSGi can easily solve the classloading and lifecycle management issues in
Hadoop, and brings a lot more besides.  Can someone please explain to me the
rationale for continuing to ignore such an obvious and elegant solution?


Best regards,
Caspar
<http://techdistrict.kirkk.com/2010/02/26/osgi-devcon-slides/>


On 14 April 2010 19:19, Cooper, Chris <chris.cooper@navteq.com> wrote:

> Scott,
>
> I think the direction your comments in
> https://issues.apache.org/jira/browse/MAPREDUCE-1700 is spot on.  You
> should be looking at J2EE container class loader hierarchies.  I've attached
> a couple of good links that cover this approach.
>
>
>
> http://www.ibm.com/developerworks/websphere/library/techarticles/0112_deboer/deboer.html
> http://www.objectsource.com/j2eechapters/Ch21-ClassLoaders_and_J2EE.htm
>
> I'm sure Mike and I would both be willing to work with you to contribute a
> solution if you're interested.
>
> Best regards,
>
> CC
>
> -----Original Message-----
> From: Scott Carey [mailto:scott@richrelevance.com]
> Sent: Wednesday, April 14, 2010 1:08 PM
> To: general@hadoop.apache.org
> Subject: Re: Custom Class Loader for Hadoop M/R jobs?
>
> My long term suggestions are in
> https://issues.apache.org/jira/browse/MAPREDUCE-1700.  The framework
> definitely needs to handle this and not place the burden on users, IMO. But
> that won't help you in the short term.
>
> Whether removing or replacing a Hadoop jar is an acceptable option to you
> (or others) in the short term is up to you.  Obviously, its not a great long
> term solution but if you (or someone else) has to make it work ASAP, it
> might be the only option.  In our case, we package our own rpm and have a
> few custom patches to Hadoop so removing one jar is a trivial thing to do in
> the short / medium term.
>
> -Scott
>
> On Apr 14, 2010, at 10:33 AM, Segel, Mike wrote:
>
> > Scott,
> >
> > While that may work for a quick fix. Its not a good long term solution
> and you then run in to a problem where you upgrade your hadoop release and
> the removed jar is replaced or if you replace the jar, it possible to get
> overwritten.
> >
> > In this specific instance, the Jackson libraries are not that important
> and they can be replaced.
> > But that doesn't mean that this issue won't come up again and its
> something you can't easily pop out and replace.
> >
> > This is why I'm looking at custom class loading and trying to understand
> what can be accomplished with the methods in the Configuration class.
> >
> > Thx
> >
> > -Mike
> >
> >
> > -----Original Message-----
> > From: Scott Carey [mailto:scott@richrelevance.com]
> > Sent: Wednesday, April 14, 2010 12:02 PM
> > To: general@hadoop.apache.org
> > Subject: Re: Custom Class Loader for Hadoop M/R jobs?
> >
> > Depending on what the dependency is, you might be able to just remove it
> from hadoop's lib directory on your cluster.
> >
> > For me, Hadoop's later versions has jackson-1.0.1 in its lib directory
> and that breaks usage of Avro in a M/R job among other things.  However, the
> feature that uses this library is unimportant to me (configuration dump in
> JSON format) so I just removed the jar.
> >
> > -Scott
> >
> > On Apr 14, 2010, at 6:39 AM, Segel, Mike wrote:
> >
> >> Hi,
> >>
> >> Ok, here's a bit of a bizarre  issue...
> >>
> >> How do you handle class collisions between Hadoop and your m/r job which
> calls other 3rd party classes.
> >>
> >> An example: Hadoop has an older version of an open source jar in its
> /lib directory. You're interfacing with a 3rd party OS tool that uses a
> later release of the same jar.
> >>
> >> You can modify the classpath, and that might work. But the better way is
> to create a Custom Class Loader. (Non-trivial)
> >>
> >> Looking at the Configuration class, it looks like there are a couple of
> methods that deal with loading a class in to the configuration so that the
> m/r jobs can have access to them on each node.
> >>
> >> Is this the correct intended use, or am I missing something?
> >> Has anyone done something like this?
> >>
> >> Thx
> >>
> >> -Mike
> >>
> >> Michael Segel
> >> Architect,  R&D
> >> NAVTEQ
> >> 425 West Randolph Street
> >> Chicago, IL 60606
> >> (T)  +1 312-780-3432
> >> (C)  +1 312-952-8175
> >> www.navteq.com<http://www.navteq.com/>
> >>
> >>
> >>
> >> The information contained in this communication may be CONFIDENTIAL and
> is intended only for the use of the recipient(s) named above.  If you are
> not the intended recipient, you are hereby notified that any dissemination,
> distribution, or copying of this communication, or any of its contents, is
> strictly prohibited.  If you have received this communication in error,
> please notify the sender and delete/destroy the original message and any
> copy of it from your computer or paper files.
> >
> >
> >
> > The information contained in this communication may be CONFIDENTIAL and
> is intended only for the use of the recipient(s) named above.  If you are
> not the intended recipient, you are hereby notified that any dissemination,
> distribution, or copying of this communication, or any of its contents, is
> strictly prohibited.  If you have received this communication in error,
> please notify the sender and delete/destroy the original message and any
> copy of it from your computer or paper files.
>
>
>
> The information contained in this communication may be CONFIDENTIAL and is
> intended only for the use of the recipient(s) named above.  If you are not
> the intended recipient, you are hereby notified that any dissemination,
> distribution, or copying of this communication, or any of its contents, is
> strictly prohibited.  If you have received this communication in error,
> please notify the sender and delete/destroy the original message and any
> copy of it from your computer or paper files.
>


The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above.  If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited.  If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.

From general-return-1396-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 16:29:47 2010
Return-Path: <general-return-1396-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78165 invoked from network); 15 Apr 2010 16:29:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 16:29:46 -0000
Received: (qmail 82806 invoked by uid 500); 15 Apr 2010 16:29:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82774 invoked by uid 500); 15 Apr 2010 16:29:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82766 invoked by uid 99); 15 Apr 2010 16:29:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 16:29:45 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.92.24] (HELO qw-out-2122.google.com) (74.125.92.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 16:29:39 +0000
Received: by qw-out-2122.google.com with SMTP id 8so599596qwh.35
        for <general@hadoop.apache.org>; Thu, 15 Apr 2010 09:29:18 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.92.20 with HTTP; Thu, 15 Apr 2010 09:28:58 -0700 (PDT)
In-Reply-To: <C7E4A520.D3AA%riweber@akamai.com>
References: <AcrX6V4gX9KGeNhIC0uktMcdChQ3Xw==> <C7E4A520.D3AA%riweber@akamai.com>
From: Aaron Kimball <aaron@cloudera.com>
Date: Thu, 15 Apr 2010 09:28:58 -0700
Received: by 10.229.251.72 with SMTP id mr8mr281407qcb.30.1271348958777; Thu, 
	15 Apr 2010 09:29:18 -0700 (PDT)
Message-ID: <z2gd6d7c4411004150928q283450d5paa6daa57c2a4f6e1@mail.gmail.com>
Subject: Re: Different exception handling on corrupt GZip file reading
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016363b808c12d2b5048448ff43
X-Virus-Checked: Checked by ClamAV on apache.org

--0016363b808c12d2b5048448ff43
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

If you ever wonder "why doesn't Hadoop do _REASONABLE_THING_X_", the answer
is usually one of:

* Somebody made a mistake the first time it got written
* Nobody needed quite that corner case before
* Maybe people thought that was useful, but didn't know how to fix it, or
were too lazy to contribute the code :)

In any case, it's just code -- there's likely not an ideological reason tha=
t
some feature is missing. I'd strongly encourage you to file a ticket on JIR=
A
and post your code as a patch. Then we can help you clean it up and get it
in there for everyone.

- Aaron


On Fri, Apr 9, 2010 at 6:34 AM, Richard Weber <riweber@akamai.com> wrote:

> Maybe this is a =B3dumb=B2 question.  In our situation, we process a ton =
of log
> files all gzipped.  Some of those files may be truncated for a a variety =
of
> reasons resulting in a corrupted gzip file.
>
> Now using the default TextInputFormat and LineRecordReader, Hadoop will
> happily churn along until it hits a corrupted file.  Once it hits the fil=
e,
> it throws exceptions, tries to restart on that file and ultimately fails.
>  I
> originally tried using the Skipped Records feature, but these exceptions
> are
> happening at the IO level, not record level.
>
> My solution has been to just make a new SafeTextInputFormat and
> SafeLineRecordReader class.  The only difference between these classes an=
d
> the non-safe classes is that it has a try {} block in the nextKeyValue()
> fn=B9
> when it does the readLine.  If an exception occurs, then the file is clos=
ed
> out.
>
> My question really boils down to: Is there a reason this isn=B9t in the
> Hadoop
> libary to start with?  Even if there was a flag to raise the exception, o=
r
> just let it keep flowing with bad input data.
>
> It=B9s really more of a gripe that I need to reimplement the above 2 clas=
ses
> just to have a try catch block, and then to make sure I use these classes
> for my input format.
>
> Thanks
>
> --Rick
>

--0016363b808c12d2b5048448ff43--

From general-return-1397-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 16:35:54 2010
Return-Path: <general-return-1397-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 79317 invoked from network); 15 Apr 2010 16:35:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 16:35:53 -0000
Received: (qmail 92882 invoked by uid 500); 15 Apr 2010 16:35:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92825 invoked by uid 500); 15 Apr 2010 16:35:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 92817 invoked by uid 99); 15 Apr 2010 16:35:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 16:35:52 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [194.246.122.12] (HELO gw2.consol.de) (194.246.122.12)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 16:35:46 +0000
Received: from sol1.bb.consol.de (sol1.bb.consol.de [10.250.0.71])
	by gw2.consol.de (8.14.4/8.14.3) with ESMTP id o3FGZPdQ034856
	for <general@hadoop.apache.org>; Thu, 15 Apr 2010 18:35:25 +0200 (CEST)
	(envelope-from marcel.may@consol.de)
Received: from [IPv6:::1] (gw2.consol.de [10.250.0.12])
	by sol1.bb.consol.de (8.13.8+Sun/8.13.7) with ESMTP id o3FGZPDS000766
	for <general@hadoop.apache.org>; Thu, 15 Apr 2010 18:35:25 +0200 (CEST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1077)
Subject: Re: Custom Class Loader for Hadoop M/R jobs?
From: Marcel May <marcel.may@consol.de>
In-Reply-To: <ABC24175AFD3BE4DA15F4CD375ED413D0607CD62ED@hq-ex-mb02.ad.navteq.com>
Date: Thu, 15 Apr 2010 18:35:25 +0200
Content-Transfer-Encoding: quoted-printable
Message-Id: <9825E237-841F-4B16-A8A8-13B30C2E4051@consol.de>
References: <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E2CB@hq-ex-mb02.ad.navteq.com> <C97199E2-890B-4F00-A9D0-376327DB9D00@richrelevance.com> <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E54A@hq-ex-mb02.ad.navteq.com> <42721ED4-DC28-47AF-A456-D3FF321013EF@richrelevance.com> <76A09B97FD0B434ABCD9A75F688CF7E50373D81E@hq-ex-mb01.ad.navteq.com> <w2xe5058b6a1004150816md06db24dxac42fa65a3a49739@mail.gmail.com> <ABC24175AFD3BE4DA15F4CD375ED413D0607CD62ED@hq-ex-mb02.ad.navteq.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1077)
X-purgate-type: clean
X-purgate-Ad: Categorized by eleven eXpurgate (R) http://www.eleven.de
X-purgate: clean
X-purgate: This mail is considered clean (visit http://www.eleven.de for further information)
X-purgate-size: 811
X-purgate-ID: 153102::1271349325-00011504-3277CF8F/0-0/0-0
X-Virus-Scanned: clamav-milter 0.96 at gw2.consol.de
X-Virus-Status: Clean
X-Virus-Checked: Checked by ClamAV on apache.org


On Apr 15, 2010, at 6:27 PM, Segel, Mike wrote:

> Caspar,
>=20
> While IANAL, I did a first blush read of OSGi's Specification License, =
Version 1.0. on the website link: =
http://www.osgi.org/Specifications/Licensing as well as the license on =
the download page: =
http://www.osgi.org/Download/Release4V42?info=3Dnothanks.
>=20
> In a nutshell, OSGi's Specification License, Version 1.0. is =
incompatible with Apache's.=20
>=20
> That alone can explain why Apache projects won't touch OSGi.
>=20

http://felix.apache.org/

> =46rom a more practical issue, this problem is solved in other Apache =
projects, so why not borrow from another Apache project like Tomcat?
>=20
> Does that make sense? It appears OSGi is a non-starter.
>=20
> HTH
>=20
> -Mike
...

Cheers,
Marcel=

From general-return-1398-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 17:24:24 2010
Return-Path: <general-return-1398-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95849 invoked from network); 15 Apr 2010 17:24:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 17:24:24 -0000
Received: (qmail 79594 invoked by uid 500); 15 Apr 2010 17:24:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79493 invoked by uid 500); 15 Apr 2010 17:24:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 79485 invoked by uid 99); 15 Apr 2010 17:24:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 17:24:23 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of msegel@navteq.com designates 204.120.70.37 as permitted sender)
Received: from [204.120.70.37] (HELO xmailfargo.navteq.com) (204.120.70.37)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 17:24:16 +0000
Received: from fargo-spamfw.navteq.com (navteq-spamfw.navteq.com [10.6.10.131])
	by xmailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id o3FH7118024319
	for <general@hadoop.apache.org>; Thu, 15 Apr 2010 12:07:01 -0500
Received: from imailfargo.navteq.com (localhost [127.0.0.1])
	by fargo-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id 5D9B74209597
	for <general@hadoop.apache.org>; Thu, 15 Apr 2010 12:23:55 -0500 (CDT)
Received: from imailfargo.navteq.com (imailfargo.navteq.com [10.6.10.130]) by fargo-spamfw.navteq.com with ESMTP id AybYd6rklvAhW0dO for <general@hadoop.apache.org>; Thu, 15 Apr 2010 12:23:55 -0500 (CDT)
Received: from hq-ex-ht01.ad.navteq.com ([10.8.222.51])
	by imailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id o3FHNtjD012657
	for <general@hadoop.apache.org>; Thu, 15 Apr 2010 12:23:55 -0500
Received: from hq-ex-mb02.ad.navteq.com ([fe80::880d:318e:e860:a8db]) by
 hq-ex-ht01.ad.navteq.com ([fe80::55a6:5b94:5c00:fe39%13]) with mapi; Thu, 15
 Apr 2010 12:23:54 -0500
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 15 Apr 2010 12:23:54 -0500
Subject: RE: Custom Class Loader for Hadoop M/R jobs?
Thread-Topic: Custom Class Loader for Hadoop M/R jobs?
Thread-Index: AcrcubiKcHs0dwFDQDOmKSRJXh/j4gABmVLg
Message-ID: <ABC24175AFD3BE4DA15F4CD375ED413D0607CD6342@hq-ex-mb02.ad.navteq.com>
References: <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E2CB@hq-ex-mb02.ad.navteq.com>
 <C97199E2-890B-4F00-A9D0-376327DB9D00@richrelevance.com>
 <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E54A@hq-ex-mb02.ad.navteq.com>
 <42721ED4-DC28-47AF-A456-D3FF321013EF@richrelevance.com>
 <76A09B97FD0B434ABCD9A75F688CF7E50373D81E@hq-ex-mb01.ad.navteq.com>
 <w2xe5058b6a1004150816md06db24dxac42fa65a3a49739@mail.gmail.com>
 <ABC24175AFD3BE4DA15F4CD375ED413D0607CD62ED@hq-ex-mb02.ad.navteq.com>
 <9825E237-841F-4B16-A8A8-13B30C2E4051@consol.de>
In-Reply-To: <9825E237-841F-4B16-A8A8-13B30C2E4051@consol.de>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1
X-Virus-Checked: Checked by ClamAV on apache.org

Ok, so are you using OSGi's code, or just writing your own code to implem=
ent a published API/specification?

I realize this is getting off topic, so you may just want to e-mail me di=
rectly.

Thx

-Mike

-----Original Message-----
From: Marcel May [mailto:marcel.may@consol.de]=20
Sent: Thursday, April 15, 2010 11:35 AM
To: general@hadoop.apache.org
Subject: Re: Custom Class Loader for Hadoop M/R jobs?


On Apr 15, 2010, at 6:27 PM, Segel, Mike wrote:

> Caspar,
>=20
> While IANAL, I did a first blush read of OSGi's Specification License, =
Version 1.0. on the website link: http://www.osgi.org/Specifications/Lice=
nsing as well as the license on the download page: http://www.osgi.org/Do=
wnload/Release4V42?info=3Dnothanks.
>=20
> In a nutshell, OSGi's Specification License, Version 1.0. is incompatib=
le with Apache's.=20
>=20
> That alone can explain why Apache projects won't touch OSGi.
>=20

http://felix.apache.org/

> From a more practical issue, this problem is solved in other Apache pro=
jects, so why not borrow from another Apache project like Tomcat?
>=20
> Does that make sense? It appears OSGi is a non-starter.
>=20
> HTH
>=20
> -Mike
=2E..

Cheers,
Marcel


The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.

From general-return-1399-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 17:47:46 2010
Return-Path: <general-return-1399-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 2274 invoked from network); 15 Apr 2010 17:47:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 17:47:46 -0000
Received: (qmail 26616 invoked by uid 500); 15 Apr 2010 17:47:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26574 invoked by uid 500); 15 Apr 2010 17:47:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26566 invoked by uid 99); 15 Apr 2010 17:47:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 17:47:45 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of earcam@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 17:47:40 +0000
Received: by wwb29 with SMTP id 29so1008494wwb.35
        for <general@hadoop.apache.org>; Thu, 15 Apr 2010 10:47:18 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :from:date:received:message-id:subject:to:content-type;
        bh=d+nrQmyPICQFMJHG2LseNIVqWtU8VcTjq4PfSY9W4Cg=;
        b=okL6J9F6P1D4eKRnXBhiTdnymutg2iMk/af3OFNAB+UEbPrVxFoS0TqWGXY8uYr2lZ
         On7FeH2L/DvT/EWQOL0nrrpSNjI388EB3SMKYl3ZdQypssYI1Z1knNnVO2dQ6hfOyaGo
         +gHa5oycwmtnSrAwjS+AELmxXrzbk5CsRdKxo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=Js68JjYC32wlr+2QTxJW6NDj6+mPZLynf8CgNnjkcwXcqwJpg6AEfTwUx9vTNMNdPy
         iySHsGuA8BUGpmTIIFz2O6PF9DE9bU0l9yvnj7D4j1PDHvb46Ly5d3z9MuMovIzd4h72
         zpKl9rr/Me7ySDPjT8+daElQDsmlSxBJ5YrRo=
MIME-Version: 1.0
Received: by 10.216.49.84 with HTTP; Thu, 15 Apr 2010 10:40:13 -0700 (PDT)
In-Reply-To: <ABC24175AFD3BE4DA15F4CD375ED413D0607CD6342@hq-ex-mb02.ad.navteq.com>
References: <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E2CB@hq-ex-mb02.ad.navteq.com> 
	<C97199E2-890B-4F00-A9D0-376327DB9D00@richrelevance.com> <ABC24175AFD3BE4DA15F4CD375ED413D0607C7E54A@hq-ex-mb02.ad.navteq.com> 
	<42721ED4-DC28-47AF-A456-D3FF321013EF@richrelevance.com> <76A09B97FD0B434ABCD9A75F688CF7E50373D81E@hq-ex-mb01.ad.navteq.com> 
	<w2xe5058b6a1004150816md06db24dxac42fa65a3a49739@mail.gmail.com> 
	<ABC24175AFD3BE4DA15F4CD375ED413D0607CD62ED@hq-ex-mb02.ad.navteq.com> 
	<9825E237-841F-4B16-A8A8-13B30C2E4051@consol.de> <ABC24175AFD3BE4DA15F4CD375ED413D0607CD6342@hq-ex-mb02.ad.navteq.com>
From: Caspar MacRae <earcam@gmail.com>
Date: Thu, 15 Apr 2010 18:40:13 +0100
Received: by 10.216.178.85 with SMTP id e63mr395702wem.156.1271353638519; Thu, 
	15 Apr 2010 10:47:18 -0700 (PDT)
Message-ID: <v2ne5058b6a1004151040v26dcdf73g4ff8635c806bcb90@mail.gmail.com>
Subject: Re: Custom Class Loader for Hadoop M/R jobs?
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e65b61d00202f404844a1659

--0016e65b61d00202f404844a1659
Content-Type: text/plain; charset=UTF-8

ServiceMix and ActiveMQ are also using OSGi in their core AFAIK.

The way this (these) problems have been solved is more of an afterthought,
and regardless is quite simplistic (thinking of classloader isolation in web
containers, app servers, and the monolithic WARs produced as a result),
however OSGi's approach to modularity provides a truly elegant solution.

wrt to the ticket https://issues.apache.org/jira/browse/MAPREDUCE-243, Dave
Savage is a nice chap who works for Paremus (an enterprise OSGi provider) I
spoke with him at the recent OSGi devcon in London and he did quite a bit of
work to get this working for private clients - I'm sure with his expertise
it would not be a huge effort to get this rolling again.




On 15 April 2010 18:23, Segel, Mike <msegel@navteq.com> wrote:

> Ok, so are you using OSGi's code, or just writing your own code to
> implement a published API/specification?
>
> I realize this is getting off topic, so you may just want to e-mail me
> directly.
>
> Thx
>
> -Mike
>
> -----Original Message-----
> From: Marcel May [mailto:marcel.may@consol.de]
> Sent: Thursday, April 15, 2010 11:35 AM
> To: general@hadoop.apache.org
> Subject: Re: Custom Class Loader for Hadoop M/R jobs?
>
>
> On Apr 15, 2010, at 6:27 PM, Segel, Mike wrote:
>
> > Caspar,
> >
> > While IANAL, I did a first blush read of OSGi's Specification License,
> Version 1.0. on the website link:
> http://www.osgi.org/Specifications/Licensing as well as the license on the
> download page: http://www.osgi.org/Download/Release4V42?info=nothanks.
> >
> > In a nutshell, OSGi's Specification License, Version 1.0. is incompatible
> with Apache's.
> >
> > That alone can explain why Apache projects won't touch OSGi.
> >
>
> http://felix.apache.org/
>
> > From a more practical issue, this problem is solved in other Apache
> projects, so why not borrow from another Apache project like Tomcat?
> >
> > Does that make sense? It appears OSGi is a non-starter.
> >
> > HTH
> >
> > -Mike
> ...
>
> Cheers,
> Marcel
>
>
> The information contained in this communication may be CONFIDENTIAL and is
> intended only for the use of the recipient(s) named above.  If you are not
> the intended recipient, you are hereby notified that any dissemination,
> distribution, or copying of this communication, or any of its contents, is
> strictly prohibited.  If you have received this communication in error,
> please notify the sender and delete/destroy the original message and any
> copy of it from your computer or paper files.
>

--0016e65b61d00202f404844a1659--

From general-return-1400-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 07:48:39 2010
Return-Path: <general-return-1400-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39093 invoked from network); 16 Apr 2010 07:48:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 07:48:39 -0000
Received: (qmail 69525 invoked by uid 500); 16 Apr 2010 07:48:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69358 invoked by uid 500); 16 Apr 2010 07:48:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69350 invoked by uid 99); 16 Apr 2010 07:48:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 07:48:37 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of shendong.liu@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 07:48:31 +0000
Received: by vws10 with SMTP id 10so1040930vws.35
        for <general@hadoop.apache.org>; Fri, 16 Apr 2010 00:48:09 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:received:message-id
         :subject:from:to:content-type;
        bh=ZeprjAdv7XkgLAzdXrZpnyLM6gX0FkHBD7kaia1Y/Cc=;
        b=prgOCgyy1dqkXqsBKvuIo12RHdcGQz7msHUQ231BNoheektxpQ6cNPgEK6Hu0pksPK
         N3vKZrVw7Eb8pCkyEiY0hEz4kuU5HFY7dyxotzzt9jt6DsCdF4dsm3LxQjLxFzaX+1El
         amd6I+TgqUyP4hRLTNtHN6WfnGu0P1FbhAxuU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=O5D/j0jOZkAiiikXmrHOZz6DmRs9+WOZQZlQZxQ80FE6ZPaYXuVij7ANGsuWPNHDbw
         pvZ8u3xRwC9bgoEwzHJOP/dQ4V69wv1Htw42Hi1XdC9XsPdsTqzl7nnPxOL0cfOTnhO1
         xrCsKsQZxsrLF9X+wRXfrEns26muWdzouCTFY=
MIME-Version: 1.0
Received: by 10.220.95.65 with HTTP; Fri, 16 Apr 2010 00:48:09 -0700 (PDT)
Date: Fri, 16 Apr 2010 15:48:09 +0800
Received: by 10.220.121.216 with SMTP id i24mr727189vcr.115.1271404089869; 
	Fri, 16 Apr 2010 00:48:09 -0700 (PDT)
Message-ID: <u2j4798ffdd1004160048g1ab091dfy30c70cc2f339fb3d@mail.gmail.com>
Subject: hello all get a hadoop example
From: =?UTF-8?B?5YiY55CG5b+X?= <shendong.liu@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504501651a248748048455d503

--00504501651a248748048455d503
Content-Type: text/plain; charset=UTF-8

hello all

please give me a hadoop java example . thanks !

Lizhi

--00504501651a248748048455d503--

From general-return-1401-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 08:02:15 2010
Return-Path: <general-return-1401-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46413 invoked from network); 16 Apr 2010 08:02:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 08:02:14 -0000
Received: (qmail 78249 invoked by uid 500); 16 Apr 2010 08:02:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78142 invoked by uid 500); 16 Apr 2010 08:02:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78134 invoked by uid 99); 16 Apr 2010 08:02:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 08:02:13 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of slimtbourbi@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 08:02:05 +0000
Received: by wyb42 with SMTP id 42so1058906wyb.35
        for <general@hadoop.apache.org>; Fri, 16 Apr 2010 01:01:45 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type;
        bh=1gDNWzv09O3+Dh0NhJNipZKTQn7JQG6ZNDBKCYmgq/k=;
        b=UUHFYx6zvhMW97D4qQirGuckjB9YRi6mwbrbr6j2rs6E/JsNDZg8H8XIUPuCuZYp6W
         1GPoX3GC9Xu0Kka1/hv0zo0kdclfEze1e6/ctkm+55NnO5Ta/JPAzAygglo4lCP9fbc/
         Qu7PgLNPbnx8coXdIBTDUGCuZZFXzi1pDQzlE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=mVaVylvKkIxHnaMr7npRzz04efQrNxw1Je1s7K1Xm8zVcXATGOs2VRumGa8he/ZT1N
         c19add3jz2FtR4yjMc08NE0e7AHZaPAiCejFM8cyEhgfMrNkUxFz8Utfl+4yecrR5WUd
         4XPxo4DC4Jqrf2yapH2zE5aROsdkefTpO7K1s=
MIME-Version: 1.0
Received: by 10.216.15.200 with HTTP; Fri, 16 Apr 2010 01:01:45 -0700 (PDT)
In-Reply-To: <u2j4798ffdd1004160048g1ab091dfy30c70cc2f339fb3d@mail.gmail.com>
References: <u2j4798ffdd1004160048g1ab091dfy30c70cc2f339fb3d@mail.gmail.com>
Date: Fri, 16 Apr 2010 10:01:45 +0200
Received: by 10.216.86.148 with SMTP id w20mr1365882wee.112.1271404905337; 
	Fri, 16 Apr 2010 01:01:45 -0700 (PDT)
Message-ID: <x2r78aa0af91004160101w170c271ape09b8b22be26ee25@mail.gmail.com>
Subject: Re: hello all get a hadoop example
From: slim tebourbi <slimtbourbi@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d7ef38bf95a004845605ed
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d7ef38bf95a004845605ed
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

see this :
http://hadoop.apache.org/common/docs/current/mapred_tutorial.html

<http://hadoop.apache.org/common/docs/current/mapred_tutorial.html>Slim
Tebourbi

2010/4/16 =E5=88=98=E7=90=86=E5=BF=97 <shendong.liu@gmail.com>

> hello all
>
> please give me a hadoop java example . thanks !
>
> Lizhi
>

--0016e6d7ef38bf95a004845605ed--

From general-return-1402-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 09:05:37 2010
Return-Path: <general-return-1402-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69347 invoked from network); 16 Apr 2010 09:05:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 09:05:36 -0000
Received: (qmail 50348 invoked by uid 500); 16 Apr 2010 09:05:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49754 invoked by uid 500); 16 Apr 2010 09:05:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49744 invoked by uid 99); 16 Apr 2010 09:05:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 09:05:32 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [72.14.220.159] (HELO fg-out-1718.google.com) (72.14.220.159)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 09:05:26 +0000
Received: by fg-out-1718.google.com with SMTP id e12so168318fga.11
        for <general@hadoop.apache.org>; Fri, 16 Apr 2010 02:05:05 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.239.134.145 with HTTP; Fri, 16 Apr 2010 02:05:05 -0700 (PDT)
Date: Fri, 16 Apr 2010 10:05:05 +0100
Received: by 10.103.84.21 with SMTP id m21mr869088mul.106.1271408705470; Fri, 
	16 Apr 2010 02:05:05 -0700 (PDT)
Message-ID: <j2jfd848f8b1004160205t797b0553t70b9dfb86d4f835@mail.gmail.com>
Subject: Re: Custom Class Loader for Hadoop M/R jobs?
From: David Savage <david.savage@paremus.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hi there,

Many thanks to Caspar for the kind words. Yep we have had Hadoop
running in an OSGi environment in the past - though it was for a proof
of concept so I've not tracked the current Hadoop trunk to see if that
patch still applies. I'm more than happy to try to offer advice if
this is something you wish to pursue, though I'd also highly recommend
the Felix mailing lists (user@felix.apache.org or
dev@felix.apache.org) as excellent sources of OSGi wisdom.

Regarding licensing again IANAL but here's the list of projects at
apache are using OSGi (that I am aware of) so I'm pretty certain it's
compatible...

http://felix.apache.org
http://aries.apache.org
http://sling.apache.org
http://tuscany.apache.org
http://servicemix.apache.org
http://activemq.apache.org
http://cxf.apache.org

Also I know there's been work to make other library projects such as
Apache Commons OSGi compatible.

Regarding custom classloaders I'd tend to advise checking if OSGi
solves your problems first before trying to role your own - the
benefits of using a common classloading model is that your components
can then interact with other frameworks - also if you run into complex
problems their are a wealth of experts out there who can help you -
where as with a custom approach you are on your own.

Regards,

Dave

> On Thu, 15 Apr 2010 10:47, Caspar MacRae wrote:
>
> ServiceMix and ActiveMQ are also using OSGi in their core AFAIK.
>
> The way this (these) problems have been solved is more of an afterthought,
> and regardless is quite simplistic (thinking of classloader isolation in web
> containers, app servers, and the monolithic WARs produced as a result),
> however OSGi's approach to modularity provides a truly elegant solution.
>
> wrt to the ticket https://issues.apache.org/jira/browse/MAPREDUCE-243, Dave
> Savage is a nice chap who works for Paremus (an enterprise OSGi provider) I
> spoke with him at the recent OSGi devcon in London and he did quite a bit of
> work to get this working for private clients - I'm sure with his expertise
> it would not be a huge effort to get this rolling again.
>
> > On 15 April 2010 18:23, Segel, Mike <mse...@navteq.com> wrote:
> >
> > Ok, so are you using OSGi's code, or just writing your own code to
> > implement a published API/specification?
> >
> > I realize this is getting off topic, so you may just want to e-mail me
> > directly.
> >
> > Thx
> >
> > -Mike
> >
> > -----Original Message-----
> > From: Marcel May [mailto:marcel....@consol.de]
> > Sent: Thursday, April 15, 2010 11:35 AM
> > To: general@hadoop.apache.org
> > Subject: Re: Custom Class Loader for Hadoop M/R jobs?
> >
> >
> > On Apr 15, 2010, at 6:27 PM, Segel, Mike wrote:
> >
> > > Caspar,
> > >
> > > While IANAL, I did a first blush read of OSGi's Specification License,
> > Version 1.0. on the website link:
> > http://www.osgi.org/Specifications/Licensing as well as the license on the
> > download page: http://www.osgi.org/Download/Release4V42?info=nothanks.
> > >
> > > In a nutshell, OSGi's Specification License, Version 1.0. is incompatible
> > with Apache's.
> > >
> > > That alone can explain why Apache projects won't touch OSGi.
> > >
> >
> > http://felix.apache.org/
> >
> > > From a more practical issue, this problem is solved in other Apache
> > projects, so why not borrow from another Apache project like Tomcat?
> > >
> > > Does that make sense? It appears OSGi is a non-starter.
> > >
> > > HTH
> > >
> > > -Mike
> > ...
> >
> > Cheers,
> > Marcel
> >
> >
> > The information contained in this communication may be CONFIDENTIAL and is
> > intended only for the use of the recipient(s) named above.  If you are not
> > the intended recipient, you are hereby notified that any dissemination,
> > distribution, or copying of this communication, or any of its contents, is
> > strictly prohibited.  If you have received this communication in error,
> > please notify the sender and delete/destroy the original message and any
> > copy of it from your computer or paper files.
> >

From general-return-1403-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 09:09:17 2010
Return-Path: <general-return-1403-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70121 invoked from network); 16 Apr 2010 09:09:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 09:09:16 -0000
Received: (qmail 53247 invoked by uid 500); 16 Apr 2010 09:09:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53215 invoked by uid 500); 16 Apr 2010 09:09:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53207 invoked by uid 99); 16 Apr 2010 09:09:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 09:09:15 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=AWL,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.218.211] (HELO mail-bw0-f211.google.com) (209.85.218.211)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 09:09:10 +0000
Received: by bwz3 with SMTP id 3so2202458bwz.11
        for <general@hadoop.apache.org>; Fri, 16 Apr 2010 02:08:49 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.239.134.145 with HTTP; Fri, 16 Apr 2010 02:08:48 -0700 (PDT)
In-Reply-To: <j2jfd848f8b1004160205t797b0553t70b9dfb86d4f835@mail.gmail.com>
References: <j2jfd848f8b1004160205t797b0553t70b9dfb86d4f835@mail.gmail.com>
Date: Fri, 16 Apr 2010 10:08:48 +0100
Received: by 10.102.14.25 with SMTP id 25mr890781mun.30.1271408928341; Fri, 16 
	Apr 2010 02:08:48 -0700 (PDT)
Message-ID: <r2mfd848f8b1004160208sbac6d0c8hb268435519093ab4@mail.gmail.com>
Subject: Re: Custom Class Loader for Hadoop M/R jobs?
From: David Savage <david.savage@paremus.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Fri, Apr 16, 2010 at 10:05 AM, David Savage <david.savage@paremus.com> w=
rote:
> Hi there,
>
> Many thanks to Caspar for the kind words. Yep we have had Hadoop
> running in an OSGi environment in the past - though it was for a proof
> of concept so I've not tracked the current Hadoop trunk to see if that
> patch still applies. I'm more than happy to try to offer advice if
> this is something you wish to pursue, though I'd also highly recommend
> the Felix mailing lists (user@felix.apache.org or
> dev@felix.apache.org) as excellent sources of OSGi wisdom.
>
> Regarding licensing again IANAL but here's the list of projects at
> apache are using OSGi (that I am aware of) so I'm pretty certain it's
> compatible...
>
> http://felix.apache.org
> http://aries.apache.org

Woops:

http://incubator.apache.org/aries/

> http://sling.apache.org
> http://tuscany.apache.org
> http://servicemix.apache.org
> http://activemq.apache.org
> http://cxf.apache.org
>
> Also I know there's been work to make other library projects such as
> Apache Commons OSGi compatible.
>
> Regarding custom classloaders I'd tend to advise checking if OSGi
> solves your problems first before trying to role your own - the
> benefits of using a common classloading model is that your components
> can then interact with other frameworks - also if you run into complex
> problems their are a wealth of experts out there who can help you -
> where as with a custom approach you are on your own.
>
> Regards,
>
> Dave
>
>> On Thu, 15 Apr 2010 10:47, Caspar MacRae wrote:
>>
>> ServiceMix and ActiveMQ are also using OSGi in their core AFAIK.
>>
>> The way this (these) problems have been solved is more of an afterthough=
t,
>> and regardless is quite simplistic (thinking of classloader isolation in=
 web
>> containers, app servers, and the monolithic WARs produced as a result),
>> however OSGi's approach to modularity provides a truly elegant solution.
>>
>> wrt to the ticket https://issues.apache.org/jira/browse/MAPREDUCE-243, D=
ave
>> Savage is a nice chap who works for Paremus (an enterprise OSGi provider=
) I
>> spoke with him at the recent OSGi devcon in London and he did quite a bi=
t of
>> work to get this working for private clients - I'm sure with his experti=
se
>> it would not be a huge effort to get this rolling again.
>>
>> > On 15 April 2010 18:23, Segel, Mike <mse...@navteq.com> wrote:
>> >
>> > Ok, so are you using OSGi's code, or just writing your own code to
>> > implement a published API/specification?
>> >
>> > I realize this is getting off topic, so you may just want to e-mail me
>> > directly.
>> >
>> > Thx
>> >
>> > -Mike
>> >
>> > -----Original Message-----
>> > From: Marcel May [mailto:marcel....@consol.de]
>> > Sent: Thursday, April 15, 2010 11:35 AM
>> > To: general@hadoop.apache.org
>> > Subject: Re: Custom Class Loader for Hadoop M/R jobs?
>> >
>> >
>> > On Apr 15, 2010, at 6:27 PM, Segel, Mike wrote:
>> >
>> > > Caspar,
>> > >
>> > > While IANAL, I did a first blush read of OSGi's Specification Licens=
e,
>> > Version 1.0. on the website link:
>> > http://www.osgi.org/Specifications/Licensing as well as the license on=
 the
>> > download page: http://www.osgi.org/Download/Release4V42?info=3Dnothank=
s.
>> > >
>> > > In a nutshell, OSGi's Specification License, Version 1.0. is incompa=
tible
>> > with Apache's.
>> > >
>> > > That alone can explain why Apache projects won't touch OSGi.
>> > >
>> >
>> > http://felix.apache.org/
>> >
>> > > From a more practical issue, this problem is solved in other Apache
>> > projects, so why not borrow from another Apache project like Tomcat?
>> > >
>> > > Does that make sense? It appears OSGi is a non-starter.
>> > >
>> > > HTH
>> > >
>> > > -Mike
>> > ...
>> >
>> > Cheers,
>> > Marcel
>> >
>> >
>> > The information contained in this communication may be CONFIDENTIAL an=
d is
>> > intended only for the use of the recipient(s) named above. =A0If you a=
re not
>> > the intended recipient, you are hereby notified that any dissemination=
,
>> > distribution, or copying of this communication, or any of its contents=
, is
>> > strictly prohibited. =A0If you have received this communication in err=
or,
>> > please notify the sender and delete/destroy the original message and a=
ny
>> > copy of it from your computer or paper files.
>> >
>

From general-return-1404-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 10:32:46 2010
Return-Path: <general-return-1404-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85999 invoked from network); 16 Apr 2010 10:32:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 10:32:46 -0000
Received: (qmail 40486 invoked by uid 500); 16 Apr 2010 10:32:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40452 invoked by uid 500); 16 Apr 2010 10:32:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40444 invoked by uid 99); 16 Apr 2010 10:32:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 10:32:44 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 10:32:33 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 4D558B7EB7
	for <general@hadoop.apache.org>; Fri, 16 Apr 2010 11:32:13 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id naIcnSDYHBp5 for <general@hadoop.apache.org>;
	Fri, 16 Apr 2010 11:32:01 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id A0B7FB7EA9
	for <general@hadoop.apache.org>; Fri, 16 Apr 2010 11:32:01 +0100 (BST)
MailScanner-NULL-Check: 1272018709.48166@qmmc7VlU9sjszqMlQDCxog
Received: from [16.24.242.20] ([16.24.242.20])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o3GAVlR3009981
	for <general@hadoop.apache.org>; Fri, 16 Apr 2010 11:31:48 +0100 (BST)
Message-ID: <4BC83C8E.4030704@apache.org>
Date: Fri, 16 Apr 2010 11:31:42 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-GB; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: JIRA snafu?
References: <1CCD1C49-71A3-46B2-878E-74781E0815EE@yahoo-inc.com>
In-Reply-To: <1CCD1C49-71A3-46B2-878E-74781E0815EE@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o3GAVlR3009981
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 13/04/2010 17:24, Arun C Murthy wrote:
> I can no longer grant permission to Apache for my patches on JIRA.
> Anyone else?
>
> thanks,
> Arun
the other troublespot right now is that some people (myself included) 
cant take ownership of issues, mark things as closed, etc. Reported to 
the infra team, but as you can imagine, they are fairly busy right now


From general-return-1405-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 11:31:46 2010
Return-Path: <general-return-1405-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97209 invoked from network); 16 Apr 2010 11:31:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 11:31:46 -0000
Received: (qmail 12212 invoked by uid 500); 16 Apr 2010 11:31:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11976 invoked by uid 500); 16 Apr 2010 11:31:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11968 invoked by uid 99); 16 Apr 2010 11:31:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 11:31:42 +0000
X-ASF-Spam-Status: No, hits=2.6 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of michael_segel@hotmail.com designates 65.55.34.16 as permitted sender)
Received: from [65.55.34.16] (HELO col0-omc1-s6.col0.hotmail.com) (65.55.34.16)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 11:31:36 +0000
Received: from COL117-W57 ([65.55.34.7]) by col0-omc1-s6.col0.hotmail.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Fri, 16 Apr 2010 04:31:15 -0700
Message-ID: <COL117-W57280AFE0D67E190A44EF58F0E0@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_c216e6b3-391a-4e30-bff0-6d4b78237654_"
X-Originating-IP: [173.15.87.34]
From: Michael Segel <michael_segel@hotmail.com>
To: <general@hadoop.apache.org>
Subject: RE: Custom Class Loader for Hadoop M/R jobs?
Date: Fri, 16 Apr 2010 06:31:15 -0500
Importance: Normal
In-Reply-To: <j2jfd848f8b1004160205t797b0553t70b9dfb86d4f835@mail.gmail.com>
References: <j2jfd848f8b1004160205t797b0553t70b9dfb86d4f835@mail.gmail.com>
MIME-Version: 1.0
X-OriginalArrivalTime: 16 Apr 2010 11:31:15.0943 (UTC) FILETIME=[53646770:01CADD58]

--_c216e6b3-391a-4e30-bff0-6d4b78237654_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


David=2C

The only reason to roll your own is if you do this in a specific map reduce=
 job and the functionality doesn't already exist.
And no=2C I'm not suggesting that.

As to OSGi for a solution=2C sure=2C that would work. I think Chris was loo=
king at something like Tomcat because they have also had to deal with this =
issue=2C so a solution exists. I wasn't aware of the OSGi implementations s=
o that too is an option.

The reason I started to look at this in terms of the Configuration class is=
 that it looks like some of the pieces are already in place and I was wonde=
ring if the class already has enough of a solution in place to allow for cu=
stom class loads?

Thanks to everyone!

-Mike

> Date: Fri=2C 16 Apr 2010 10:05:05 +0100
> Subject: Re: Custom Class Loader for Hadoop M/R jobs?
> From: david.savage@paremus.com
> To: general@hadoop.apache.org
>=20
> Hi there=2C
>=20
> Many thanks to Caspar for the kind words. Yep we have had Hadoop
> running in an OSGi environment in the past - though it was for a proof
> of concept so I've not tracked the current Hadoop trunk to see if that
> patch still applies. I'm more than happy to try to offer advice if
> this is something you wish to pursue=2C though I'd also highly recommend
> the Felix mailing lists (user@felix.apache.org or
> dev@felix.apache.org) as excellent sources of OSGi wisdom.
>=20
> Regarding licensing again IANAL but here's the list of projects at
> apache are using OSGi (that I am aware of) so I'm pretty certain it's
> compatible...
>=20
> http://felix.apache.org
> http://aries.apache.org
> http://sling.apache.org
> http://tuscany.apache.org
> http://servicemix.apache.org
> http://activemq.apache.org
> http://cxf.apache.org
>=20
> Also I know there's been work to make other library projects such as
> Apache Commons OSGi compatible.
>=20
> Regarding custom classloaders I'd tend to advise checking if OSGi
> solves your problems first before trying to role your own - the
> benefits of using a common classloading model is that your components
> can then interact with other frameworks - also if you run into complex
> problems their are a wealth of experts out there who can help you -
> where as with a custom approach you are on your own.
>=20
> Regards=2C
>=20
> Dave
>=20
> > On Thu=2C 15 Apr 2010 10:47=2C Caspar MacRae wrote:
> >
> > ServiceMix and ActiveMQ are also using OSGi in their core AFAIK.
> >
> > The way this (these) problems have been solved is more of an afterthoug=
ht=2C
> > and regardless is quite simplistic (thinking of classloader isolation i=
n web
> > containers=2C app servers=2C and the monolithic WARs produced as a resu=
lt)=2C
> > however OSGi's approach to modularity provides a truly elegant solution=
.
> >
> > wrt to the ticket https://issues.apache.org/jira/browse/MAPREDUCE-243=
=2C Dave
> > Savage is a nice chap who works for Paremus (an enterprise OSGi provide=
r) I
> > spoke with him at the recent OSGi devcon in London and he did quite a b=
it of
> > work to get this working for private clients - I'm sure with his expert=
ise
> > it would not be a huge effort to get this rolling again.
> >
> > > On 15 April 2010 18:23=2C Segel=2C Mike <mse...@navteq.com> wrote:
> > >
> > > Ok=2C so are you using OSGi's code=2C or just writing your own code t=
o
> > > implement a published API/specification?
> > >
> > > I realize this is getting off topic=2C so you may just want to e-mail=
 me
> > > directly.
> > >
> > > Thx
> > >
> > > -Mike
> > >
> > > -----Original Message-----
> > > From: Marcel May [mailto:marcel....@consol.de]
> > > Sent: Thursday=2C April 15=2C 2010 11:35 AM
> > > To: general@hadoop.apache.org
> > > Subject: Re: Custom Class Loader for Hadoop M/R jobs?
> > >
> > >
> > > On Apr 15=2C 2010=2C at 6:27 PM=2C Segel=2C Mike wrote:
> > >
> > > > Caspar=2C
> > > >
> > > > While IANAL=2C I did a first blush read of OSGi's Specification Lic=
ense=2C
> > > Version 1.0. on the website link:
> > > http://www.osgi.org/Specifications/Licensing as well as the license o=
n the
> > > download page: http://www.osgi.org/Download/Release4V42?info=3Dnothan=
ks.
> > > >
> > > > In a nutshell=2C OSGi's Specification License=2C Version 1.0. is in=
compatible
> > > with Apache's.
> > > >
> > > > That alone can explain why Apache projects won't touch OSGi.
> > > >
> > >
> > > http://felix.apache.org/
> > >
> > > > From a more practical issue=2C this problem is solved in other Apac=
he
> > > projects=2C so why not borrow from another Apache project like Tomcat=
?
> > > >
> > > > Does that make sense? It appears OSGi is a non-starter.
> > > >
> > > > HTH
> > > >
> > > > -Mike
> > > ...
> > >
> > > Cheers=2C
> > > Marcel
> > >
> > >
> > > The information contained in this communication may be CONFIDENTIAL a=
nd is
> > > intended only for the use of the recipient(s) named above.  If you ar=
e not
> > > the intended recipient=2C you are hereby notified that any disseminat=
ion=2C
> > > distribution=2C or copying of this communication=2C or any of its con=
tents=2C is
> > > strictly prohibited.  If you have received this communication in erro=
r=2C
> > > please notify the sender and delete/destroy the original message and =
any
> > > copy of it from your computer or paper files.
> > >
 		 	   		 =20
_________________________________________________________________
The New Busy is not the old busy. Search=2C chat and e-mail from your inbox=
.
http://www.windowslive.com/campaign/thenewbusy?ocid=3DPID28326::T:WLMTAGL:O=
N:WL:en-US:WM_HMP:042010_3=

--_c216e6b3-391a-4e30-bff0-6d4b78237654_--

From general-return-1406-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 15:46:19 2010
Return-Path: <general-return-1406-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4928 invoked from network); 16 Apr 2010 15:46:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 15:46:19 -0000
Received: (qmail 33616 invoked by uid 500); 16 Apr 2010 15:46:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33555 invoked by uid 500); 16 Apr 2010 15:46:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33547 invoked by uid 99); 16 Apr 2010 15:46:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 15:46:17 +0000
X-ASF-Spam-Status: No, hits=1.7 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of shendong.liu@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 15:46:11 +0000
Received: by vws10 with SMTP id 10so1234335vws.35
        for <general@hadoop.apache.org>; Fri, 16 Apr 2010 08:45:50 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type;
        bh=HNkFduYh691Ypcmn4/qikWHcbN5ndft3pcAEuSWwdgE=;
        b=KNfoFwiORS2FRRvwBiLi93b2lOEIyrG3EdrqcId52zTcyqbnEEJ3LmyUYTHqzHlY4K
         Wb3fGc6XzaWBuAzc86yXugXL58j5GqbIWz3TO59REtiru1WoL2rEpIbkK18ApMJUyQPP
         2X558A9JPvUjAioEu6bU5S7R+jImGgIW+sWN8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Tlj3P4wrkgZrm2pj4rLOKui46pndrCscBAKu3Mb5J+Xenz5HNLMtCe3+eNUR4DQ6Wd
         jWptkZ4BZ6+xxEyODuyuDKv2JkNfPbYiW+t4hKEBQas5MdrzoM9W98jP1eMMxjJnf6DU
         yii3y8qdK3E3tW6PC3UjPgLmg0lUNEhwRr1TY=
MIME-Version: 1.0
Received: by 10.220.95.65 with HTTP; Fri, 16 Apr 2010 08:45:50 -0700 (PDT)
In-Reply-To: <x2r78aa0af91004160101w170c271ape09b8b22be26ee25@mail.gmail.com>
References: <u2j4798ffdd1004160048g1ab091dfy30c70cc2f339fb3d@mail.gmail.com>
	 <x2r78aa0af91004160101w170c271ape09b8b22be26ee25@mail.gmail.com>
Date: Fri, 16 Apr 2010 23:45:50 +0800
Received: by 10.220.107.71 with SMTP id a7mr1057107vcp.231.1271432750090; Fri, 
	16 Apr 2010 08:45:50 -0700 (PDT)
Message-ID: <h2p4798ffdd1004160845k49bfac34oa884af3bf864fc0f@mail.gmail.com>
Subject: Re: hello all get a hadoop example
From: steven liu <shendong.liu@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00c09f9057616ccb6d04845c81b6

--00c09f9057616ccb6d04845c81b6
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

thanks Slim Tebourbi

2010/4/16 slim tebourbi <slimtbourbi@gmail.com>

> see this :
> http://hadoop.apache.org/common/docs/current/mapred_tutorial.html
>
> <http://hadoop.apache.org/common/docs/current/mapred_tutorial.html>Slim
> Tebourbi
>
> 2010/4/16 =E5=88=98=E7=90=86=E5=BF=97 <shendong.liu@gmail.com>
>
> > hello all
> >
> > please give me a hadoop java example . thanks !
> >
> > Lizhi
> >
>

--00c09f9057616ccb6d04845c81b6--

From general-return-1407-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 18:40:43 2010
Return-Path: <general-return-1407-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57422 invoked from network); 16 Apr 2010 18:40:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 18:40:43 -0000
Received: (qmail 40287 invoked by uid 500); 16 Apr 2010 18:40:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40232 invoked by uid 500); 16 Apr 2010 18:40:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40224 invoked by uid 99); 16 Apr 2010 18:40:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 18:40:42 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ctubbsii@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 18:40:34 +0000
Received: by wwb29 with SMTP id 29so1675683wwb.35
        for <general@hadoop.apache.org>; Fri, 16 Apr 2010 11:40:14 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=19l6a5dyLPcj9vYnA8RN3VcrB+J2g6jfLgBjtWV/u/0=;
        b=OBU2UanaNG6qVYMN9jwr7FddNcGZbqWR3dEFPsHlrH1rur+V48LP0vtLxA1R8vhvq+
         DqlxrpP2ptcgOrUioXKHQvAP/SO2/hG+R3sEnbt39axpA7LgqzLDTpcHTDEwuwlIn0ky
         EuC6j3LnkLsb1QFALGG466w3FcNvNITHUmEZQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=Y5PMDWHOY3gO6DTwGF0dvwNak3e3xvTpZZ1Jnbkv1d6zc7sOnmM8q5/A/5FUkZg2q6
         MLS8PPe6LwnRacwsidjK070gV4u6nxurVdcFG6NWvRFmPPQN0Tcbdm2CqPgMdAcqjYtz
         VXWlKRr4osn9HFoQuJL4z1Pln4Oej3GvGtJrA=
MIME-Version: 1.0
Received: by 10.216.36.76 with HTTP; Fri, 16 Apr 2010 11:40:14 -0700 (PDT)
In-Reply-To: <x2r78aa0af91004160101w170c271ape09b8b22be26ee25@mail.gmail.com>
References: <u2j4798ffdd1004160048g1ab091dfy30c70cc2f339fb3d@mail.gmail.com>
	 <x2r78aa0af91004160101w170c271ape09b8b22be26ee25@mail.gmail.com>
Date: Fri, 16 Apr 2010 14:40:14 -0400
Received: by 10.216.182.142 with SMTP id o14mr2437310wem.137.1271443214337; 
	Fri, 16 Apr 2010 11:40:14 -0700 (PDT)
Message-ID: <x2r9db00c201004161140s140bbd9cn2cf5cdd9e24bc02e@mail.gmail.com>
Subject: Re: hello all get a hadoop example
From: Christopher Tubbs <ctubbsii@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

That documentation is SO out of date.

On Fri, Apr 16, 2010 at 4:01 AM, slim tebourbi <slimtbourbi@gmail.com> wrot=
e:
> see this :
> http://hadoop.apache.org/common/docs/current/mapred_tutorial.html
>
> <http://hadoop.apache.org/common/docs/current/mapred_tutorial.html>Slim
> Tebourbi
>
> 2010/4/16 =E5=88=98=E7=90=86=E5=BF=97 <shendong.liu@gmail.com>
>
>> hello all
>>
>> please give me a hadoop java example . thanks !
>>
>> Lizhi
>>
>

From general-return-1408-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 17 22:05:44 2010
Return-Path: <general-return-1408-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35461 invoked from network); 17 Apr 2010 22:05:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Apr 2010 22:05:44 -0000
Received: (qmail 99788 invoked by uid 500); 17 Apr 2010 22:05:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99571 invoked by uid 500); 17 Apr 2010 22:05:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 38557 invoked by uid 99); 13 Apr 2010 03:19:02 -0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cryptcom@gmail.com designates 209.85.212.48 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:references:message-id:from:to
         :in-reply-to:content-type:content-transfer-encoding:x-mailer
         :mime-version:subject:date:cc;
        bh=iSHXF2FT8RZ+eEZw3JUJtBeJxkuRcr9J3hyutAiHmjY=;
        b=CyH4nZJQcqct0Vb8Ipy35fh9pjsaxqFpAMySX2yAsraT5IRfunyh0hLpg1pPN1Sp8z
         /17/alUi4MXf8PrIpyvxQ+Etn8S6/doYdRPv0XvVOoOggG7NAZw81O93ZLpGvAL9lfhQ
         cZ7i+/aYo7b30jKSMTXOlbqUbG9sbnJk8YUmQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=references:message-id:from:to:in-reply-to:content-type
         :content-transfer-encoding:x-mailer:mime-version:subject:date:cc;
        b=U+V9amvwJWE+HcM8TPtuXJrAZbOqICFXVYzfkyKN5djcri7LJhaUr6eIhZqLSzcMTV
         ayaZ3UKA6lLwBSL5Af7SkBEaDHPB/GqWHEr5rTorWi3BwBJehaJeJVsFiFupm9Ng6uUG
         VymsoyTEXosiXG2Zq45Lyysl20Sa03umFzX8E=
References: <m2l7c962aed1004120856hddd9e41dt77f0ba3e259c9171@mail.gmail.com> <AB29A3AD-E842-400B-A80F-1DA1F57374E0@beforedawnsolutions.com>
Message-Id: <747595C5-5D91-445B-B2FD-32B5D944BCD6@gmail.com>
From: Joe Stein <cryptcom@gmail.com>
To: "hbase-dev@hadoop.apache.org" <hbase-dev@hadoop.apache.org>
In-Reply-To: <AB29A3AD-E842-400B-A80F-1DA1F57374E0@beforedawnsolutions.com>
Content-Type: text/plain;
	charset=us-ascii;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit
X-Mailer: iPhone Mail (7E18)
Mime-Version: 1.0 (iPhone Mail 7E18)
Subject: Re: [VOTE] HBase as TLP?
Date: Mon, 12 Apr 2010 23:18:25 -0400
Cc: "general@hadoop.apache.org" <general@hadoop.apache.org>,
 "hbase-dev@hadoop.apache.org" <hbase-dev@hadoop.apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

It is a good idea because the segment is already fragmented and  
honestly hbase is often a second thought, and should not be, because  
it is seen as tightly coupled.  I find hbase and Cassandra the leaders  
of the pack and welcome hbase TLP to knock out dumb others and  
hopefully tighten up the options for newbies (how is that for a run on  
sentance).

Just my $.029843366347 cents on it since you asked.

/*
Joe Stein
http://www.linkedin.com/in/charmalloc
*/

On Apr 12, 2010, at 8:52 PM, John Martyniak <john@beforedawnsolutions.com 
 > wrote:

> Can somebody tell me why this is a good idea?
>
> Isn't HBase tightly coupled with Hadoop/HDFS.  Isn't that the  
> benefit of it.
>
> -John
>
> On Apr 12, 2010, at 11:56 AM, Stack wrote:
>
>> Please vote as to whether you think HBase should become a top-level
>> Apache project.
>>
>> I've included below a draft board resolution.  It lists all current
>> active HBase committers as initial members of the project management
>> committee (PMC) and myself, Michael Stack, as the initial chair.
>>
>> Do the good people of Hadoop support sending this request on to the  
>> Hadoop PMC?
>>
>> Yours,
>> St.Ack
>>
>> ------------
>>
>>  X. Establish the Apache HBase Project
>>
>>     WHEREAS, the Board of Directors deems it to be in the best
>>     interests of the Foundation and consistent with the
>>     Foundation's purpose to establish a Project Management
>>     Committee charged with the creation and maintenance of
>>     open-source software related to data serialization
>>     for distribution at no charge to the public.
>>
>>     NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>>     Committee (PMC), to be known as the "Apache HBase Project",
>>     be and hereby is established pursuant to Bylaws of the
>>     Foundation; and be it further
>>
>>     RESOLVED, that the Apache HBase Project be and hereby is
>>     responsible for the creation and maintenance of software
>>     related to a distributed database; and be it further
>>
>>     RESOLVED, that the office of "Vice President, Apache HBase" be
>>     and hereby is created, the person holding such office to
>>     serve at the direction of the Board of Directors as the chair
>>     of the Apache HBase Project, and to have primary responsibility
>>     for management of the projects within the scope of
>>     responsibility of the Apache HBase Project; and be it further
>>
>>     RESOLVED, that the persons listed immediately below be and
>>     hereby are appointed to serve as the initial members of the
>>     Apache HBase Project:
>>
>>       * Michael Stack <stack@apache.org>
>>       * Jonathan Gray <jgray@apache.org>
>>       * Andrew Purtell <apurtell@apache.org>
>>       * John-Daniel Cryans <jdcryans@apache.org>
>>       * Ryan Rawson <rawson@apache.org>
>>       * Lars George <larsgeorge@apache.org>
>>
>>     NOW, THEREFORE, BE IT FURTHER RESOLVED, that Michael Stack
>>     be appointed to the office of Vice President, Apache HBase, to
>>     serve in accordance with and subject to the direction of the
>>     Board of Directors and the Bylaws of the Foundation until
>>     death, resignation, retirement, removal or disqualification,
>>     or until a successor is appointed; and be it further
>>
>>     RESOLVED, that the initial Apache HBase PMC be and hereby is
>>     tasked with the creation of a set of bylaws intended to
>>     encourage open development and increased participation in the
>>     Apache HBase Project; and be it further
>>
>>     RESOLVED, that the Apache HBase Project be and hereby
>>     is tasked with the migration and rationalization of the Apache
>>     Hadoop HBase sub-project; and be it further
>>
>>     RESOLVED, that all responsibilities pertaining to the Apache
>>     Hadoop HBase sub-project encumbered upon the
>>     Apache Hadoop Project are hereafter discharged.
>

From general-return-1409-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 17 22:05:53 2010
Return-Path: <general-return-1409-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35543 invoked from network); 17 Apr 2010 22:05:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Apr 2010 22:05:53 -0000
Received: (qmail 1242 invoked by uid 500); 17 Apr 2010 22:05:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1198 invoked by uid 500); 17 Apr 2010 22:05:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 54007 invoked by uid 99); 13 Apr 2010 19:26:11 -0000
X-ASF-Spam-Status: No, hits=-2.5 required=10.0
	tests=AWL,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of clehene@adobe.com designates 64.18.1.23 as permitted sender)
From: Cosmin Lehene <clehene@adobe.com>
To: "hbase-dev@hadoop.apache.org" <hbase-dev@hadoop.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 13 Apr 2010 20:25:35 +0100
Subject: Re: [VOTE] HBase as TLP?
Thread-Topic: [VOTE] HBase as TLP?
Thread-Index: AcrbPxnu9nsZy1KfTFmtKyxZnshv0g==
Message-ID: <BA087D3D-D948-48A9-9989-D54F5AF7C4EF@adobe.com>
References: <m2l7c962aed1004120856hddd9e41dt77f0ba3e259c9171@mail.gmail.com>
In-Reply-To: <m2l7c962aed1004120856hddd9e41dt77f0ba3e259c9171@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

+1


On Apr 12, 2010, at 6:56 PM, Stack wrote:

> Please vote as to whether you think HBase should become a top-level
> Apache project.
>=20
> I've included below a draft board resolution.  It lists all current
> active HBase committers as initial members of the project management
> committee (PMC) and myself, Michael Stack, as the initial chair.
>=20
> Do the good people of Hadoop support sending this request on to the Hadoo=
p PMC?
>=20
> Yours,
> St.Ack
>=20
> ------------
>=20
>   X. Establish the Apache HBase Project
>=20
>      WHEREAS, the Board of Directors deems it to be in the best
>      interests of the Foundation and consistent with the
>      Foundation's purpose to establish a Project Management
>      Committee charged with the creation and maintenance of
>      open-source software related to data serialization
>      for distribution at no charge to the public.
>=20
>      NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>      Committee (PMC), to be known as the "Apache HBase Project",
>      be and hereby is established pursuant to Bylaws of the
>      Foundation; and be it further
>=20
>      RESOLVED, that the Apache HBase Project be and hereby is
>      responsible for the creation and maintenance of software
>      related to a distributed database; and be it further
>=20
>      RESOLVED, that the office of "Vice President, Apache HBase" be
>      and hereby is created, the person holding such office to
>      serve at the direction of the Board of Directors as the chair
>      of the Apache HBase Project, and to have primary responsibility
>      for management of the projects within the scope of
>      responsibility of the Apache HBase Project; and be it further
>=20
>      RESOLVED, that the persons listed immediately below be and
>      hereby are appointed to serve as the initial members of the
>      Apache HBase Project:
>=20
>        * Michael Stack <stack@apache.org>
>        * Jonathan Gray <jgray@apache.org>
>        * Andrew Purtell <apurtell@apache.org>
>        * John-Daniel Cryans <jdcryans@apache.org>
>        * Ryan Rawson <rawson@apache.org>
>        * Lars George <larsgeorge@apache.org>
>=20
>      NOW, THEREFORE, BE IT FURTHER RESOLVED, that Michael Stack
>      be appointed to the office of Vice President, Apache HBase, to
>      serve in accordance with and subject to the direction of the
>      Board of Directors and the Bylaws of the Foundation until
>      death, resignation, retirement, removal or disqualification,
>      or until a successor is appointed; and be it further
>=20
>      RESOLVED, that the initial Apache HBase PMC be and hereby is
>      tasked with the creation of a set of bylaws intended to
>      encourage open development and increased participation in the
>      Apache HBase Project; and be it further
>=20
>      RESOLVED, that the Apache HBase Project be and hereby
>      is tasked with the migration and rationalization of the Apache
>      Hadoop HBase sub-project; and be it further
>=20
>      RESOLVED, that all responsibilities pertaining to the Apache
>      Hadoop HBase sub-project encumbered upon the
>      Apache Hadoop Project are hereafter discharged.


From general-return-1410-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 17 22:06:27 2010
Return-Path: <general-return-1410-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35642 invoked from network); 17 Apr 2010 22:06:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Apr 2010 22:06:27 -0000
Received: (qmail 2404 invoked by uid 500); 17 Apr 2010 22:06:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2206 invoked by uid 500); 17 Apr 2010 22:06:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 72881 invoked by uid 99); 14 Apr 2010 04:35:44 -0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=HTML_MESSAGE,MIME_QP_LONG_LINE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
X-AuditID: cb5bdd58-b7bf7ae000006db3-c0-4bc546036190
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01CADB8B.E08FE5E0"
Subject: how to split data
Date: Wed, 14 Apr 2010 10:05:14 +0530
Message-ID: <1E766DC73403B144A7FF3C3A3888D0C7143229@BLR-EC-MBX02.wipro.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: how to split data
Thread-Index: Acrbi+B85RcIu5uYSc2MHIqkbqfjQg==
From: <lee.sungeun@wipro.com>
To: <general@hadoop.apache.org>
X-OriginalArrivalTime: 14 Apr 2010 04:35:14.0992 (UTC) FILETIME=[E0AE0B00:01CADB8B]
X-Brightmail-Tracker: AAAAAQAAAZE=
X-Virus-Checked: Checked by ClamAV on apache.org

------_=_NextPart_001_01CADB8B.E08FE5E0
Content-Type: text/plain;
	charset="iso-8859-1"
content-transfer-encoding: quoted-printable

Hi
i am studying about hadoop and sending a mail to know how to split data. if=
 the block size is 64MB and i am supposed to hadle 129MB. How will it split?=
 (64,64,1) or (64,64,64) . i think first one is right. but i want to get inf=
ormaiton more detail.
so if possible, just let me know it.
 
Regards
sungeun lee

Please do not print this email unless it is absolutely necessary. =0A=
=0A=
The information contained in this electronic message and any attachments to=
 this message are intended for the exclusive use of the addressee(s) and may=
 contain proprietary, confidential or privileged information. If you are not=
 the intended recipient, you should not disseminate, distribute or copy this=
 e-mail. Please notify the sender immediately and destroy all copies of this=
 message and any attachments. =0A=
=0A=
WARNING: Computer viruses can be transmitted via email. The recipient should=
 check this email and any attachments for the presence of viruses. The compa=
ny accepts no liability for any damage caused by any virus transmitted by th=
is email. =0A=
=0A=
www.wipro.com

------_=_NextPart_001_01CADB8B.E08FE5E0--

From general-return-1411-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 17 22:06:45 2010
Return-Path: <general-return-1411-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35722 invoked from network); 17 Apr 2010 22:06:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Apr 2010 22:06:45 -0000
Received: (qmail 3447 invoked by uid 500); 17 Apr 2010 22:06:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3235 invoked by uid 500); 17 Apr 2010 22:06:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 76787 invoked by uid 99); 15 Apr 2010 07:49:00 -0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of rectech@126.com designates 123.125.50.112 as permitted sender)
Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes
To: general@hadoop.apache.org
Date: Thu, 15 Apr 2010 15:48:29 +0800
Subject: strange error with node name
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
From: rec <rectech@126.com>
Organization: recfloyd
Message-ID: <op.va61a3vxwm0a25@localhost>
User-Agent: Opera Mail/10.10 (Linux)
X-CM-TRANSID:j9KowLAb0QPNxMZLQUYJAA--.1082S2
X-Coremail-Antispam: 1Uf129KBjDUn29KB7ZKAUJUUUUU529EdanIXcx71UUUUU7v73
	VFW2AGmfu7bjvjm3AaLaJ3UbIYCTnIWIevJa73UjIFyTuYvjxUYD73DUUUU
X-CM-SenderInfo: xuhf3vlfk6ij2wof0z/1tbiDwyzsUnsSpgPgQAAs4


Hi guys, I've got a strange error when doing my first try of hadoop 0.20.  
To try the clustering, I've got 3 machines with the same OS, Ubuntu 9.10.  
One of them for master and the other 2 for slave.
After starting, I turn to the page  
http://MASTER_DOMAIN:50070/dfsnodelist.jsp?whatNodes=LIVE. I got 2 live  
nodes, whose names are both "localhost", but not the domain names of the 2  
slaves.
I did added the 2 slaves' domain name to /etc/hosts file.
There's no error in log files, but where did I go wrong ?

Thanks advanced.
Rec


From general-return-1412-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 17 22:44:08 2010
Return-Path: <general-return-1412-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 51686 invoked from network); 17 Apr 2010 22:44:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Apr 2010 22:44:08 -0000
Received: (qmail 21074 invoked by uid 500); 17 Apr 2010 22:44:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21021 invoked by uid 500); 17 Apr 2010 22:44:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21013 invoked by uid 99); 17 Apr 2010 22:44:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Apr 2010 22:44:07 +0000
X-ASF-Spam-Status: No, hits=-0.1 required=10.0
	tests=AWL,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of msegel@navteq.com designates 65.167.11.240 as permitted sender)
Received: from [65.167.11.240] (HELO xmailchi.navteq.com) (65.167.11.240)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Apr 2010 22:44:01 +0000
Received: from chicago-spamfw.navteq.com (imailchi [10.8.5.161])
	by xmailchi.navteq.com (8.13.8/8.13.6) with ESMTP id o3HMPQxW029509
	for <general@hadoop.apache.org>; Sat, 17 Apr 2010 17:25:26 -0500
Received: from imailchi.hq.navteq.com (localhost [127.0.0.1])
	by chicago-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id 7028B2EC3E3A
	for <general@hadoop.apache.org>; Sat, 17 Apr 2010 17:43:39 -0500 (CDT)
Received: from imailchi.hq.navteq.com (imailchi.hq.navteq.com [10.8.120.104]) by chicago-spamfw.navteq.com with ESMTP id QmAtQxRO0HeqXBqA for <general@hadoop.apache.org>; Sat, 17 Apr 2010 17:43:39 -0500 (CDT)
Received: from hq-ex-ht02.ad.navteq.com (hq-ex-ht02.ad.navteq.com [10.8.222.172])
	by imailchi.hq.navteq.com (8.12.9/8.12.9) with ESMTP id o3HMhdGq001624
	for <general@hadoop.apache.org>; Sat, 17 Apr 2010 17:43:39 -0500
Received: from hq-ex-mb02.ad.navteq.com ([fe80::880d:318e:e860:a8db]) by
 hq-ex-ht02.ad.navteq.com ([fe80::24f6:ca50:734d:df6e%14]) with mapi; Sat, 17
 Apr 2010 17:43:37 -0500
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Sat, 17 Apr 2010 17:43:37 -0500
Subject: RE: how to split data
Thread-Topic: how to split data
Thread-Index: Acrbi+B85RcIu5uYSc2MHIqkbqfjQgC8feiQ
Message-ID: <ABC24175AFD3BE4DA15F4CD375ED413D0607CD6B13@hq-ex-mb02.ad.navteq.com>
References: <1E766DC73403B144A7FF3C3A3888D0C7143229@BLR-EC-MBX02.wipro.com>
In-Reply-To: <1E766DC73403B144A7FF3C3A3888D0C7143229@BLR-EC-MBX02.wipro.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1

Sungeun,

Its really the second one. Files are split in to blocks depending on a co=
nfiguration parameter. By default its 64MB. Using your example, the split=
s are 64,64,1 however the 1 MB is within an allocated block of size 64MB.=
=20

HTH
-Mike

-----Original Message-----
From: lee.sungeun@wipro.com [mailto:lee.sungeun@wipro.com]=20
Sent: Tuesday, April 13, 2010 11:35 PM
To: general@hadoop.apache.org
Subject: how to split data

Hi
i am studying about hadoop and sending a mail to know how to split data. =
if the block size is 64MB and i am supposed to hadle 129MB. How will it s=
plit? (64,64,1) or (64,64,64) . i think first one is right. but i want to=
 get informaiton more detail.
so if possible, just let me know it.
=20
Regards
sungeun lee

Please do not print this email unless it is absolutely necessary.=20

The information contained in this electronic message and any attachments =
to this message are intended for the exclusive use of the addressee(s) an=
d may contain proprietary, confidential or privileged information. If you=
 are not the intended recipient, you should not disseminate, distribute o=
r copy this e-mail. Please notify the sender immediately and destroy all =
copies of this message and any attachments.=20

WARNING: Computer viruses can be transmitted via email. The recipient sho=
uld check this email and any attachments for the presence of viruses. The=
 company accepts no liability for any damage caused by any virus transmit=
ted by this email.=20

www.wipro.com


The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.

From general-return-1413-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 19 02:38:24 2010
Return-Path: <general-return-1413-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27333 invoked from network); 19 Apr 2010 02:38:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Apr 2010 02:38:24 -0000
Received: (qmail 88753 invoked by uid 500); 19 Apr 2010 02:38:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88584 invoked by uid 500); 19 Apr 2010 02:38:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88576 invoked by uid 99); 19 Apr 2010 02:38:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 02:38:21 +0000
X-ASF-Spam-Status: No, hits=4.8 required=10.0
	tests=AWL,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of sonalgoyal4@gmail.com designates 74.125.83.176 as permitted sender)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 02:38:17 +0000
Received: by pvg7 with SMTP id 7so2635615pvg.35
        for <general@hadoop.apache.org>; Sun, 18 Apr 2010 19:37:57 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type;
        bh=vvNFx++LjOUzu95PkK0H815fq9J8NA1cETumnnP0x1w=;
        b=PVHPcYOyip8VJuFp71VZ+6zyS2UG48wFYXYItQUYjBBm9LfqPlQy0pnTKVegZCOppI
         km5HWJMMeZUQIRUpn5ynqaEAisvGnGjQb/t6cgddPNYYwAOQ5vzu3KPpvNHTBYrUFOPo
         cXxUCN97LTxZIrh12QFh7mxmssD5qIjbVfsnY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=i6tcUV2t/7BnwDN/aaxBF9/qU1DHVg1Z2iwEGQmWtS+pwvwqVdqPl7EFeKIKuFOzY8
         5prA7Y7YFleGLaNWAL6deU3NjbzQZthWUYrAgNc62tAP0mUtJdHo8fMU6sTAnPSeupNJ
         8+tcA9RJzNM9XKGeZDBCFZUbiTeB+WP0pimP4=
MIME-Version: 1.0
Received: by 10.142.133.15 with HTTP; Sun, 18 Apr 2010 19:37:56 -0700 (PDT)
In-Reply-To: <s2ha7988b8e1004150609kb3e67dbct8fe49a2aa5b8bebd@mail.gmail.com>
References: <q2ha7988b8e1004130746t8ab3e3dap5607f7f8e53053b8@mail.gmail.com>
	 <m2va7988b8e1004130809l40cde215icd5e1ab561b92f49@mail.gmail.com>
	 <u2l23d54121004140928v58286578u2d66ed08196722e0@mail.gmail.com>
	 <s2ha7988b8e1004150609kb3e67dbct8fe49a2aa5b8bebd@mail.gmail.com>
Date: Mon, 19 Apr 2010 08:07:56 +0530
Received: by 10.143.21.2 with SMTP id y2mr1784134wfi.239.1271644676838; Sun, 
	18 Apr 2010 19:37:56 -0700 (PDT)
Message-ID: <y2h9f26e661004181937xc08de939hb70d7868d3b5d3d0@mail.gmail.com>
Subject: Re: DBInputFormat number of mappers
From: Sonal Goyal <sonalgoyal4@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502cbe63e576304848dd9e5

--00504502cbe63e576304848dd9e5
Content-Type: text/plain; charset=ISO-8859-1

Hi Dan,

You can also check http://code.google.com/p/hiho/ for importing query based
data using bound range queries.

Thanks and Regards,
Sonal
www.meghsoft.com


On Thu, Apr 15, 2010 at 6:39 PM, Dan Harvey <dan.harvey@mendeley.com> wrote:

> Unfortunately the tables I'm importing need application logic to extract
> the
> data full which means I can use sqoop, but I will be using that for all the
> simpler tables.
>
> I think I found the problem was with locking the tables in mysql which
> caused the mappers to run one at a time, so I've re-wrote the code so I'm
> not locking tables now and that's fixed it from happening completely. It's
> odd that it didn't always go one at a time though but I guess that might
> just been to do with how the mysql server was dealing with the queries at
> different times.
>
> Thanks,
>
> On 14 April 2010 17:28, Eric Sammer <esammer@cloudera.com> wrote:
>
> > If you're performing a simple import of an entire table, sqoop may
> > make your life easier. It gives you a reasonable command line client
> > for importing single tables or an entire database (provided there is a
> > JDBC driver available for it). Sqoop comes with Cloudera's
> > distribution for Hadoop or you can snag the source from
> > http://github.com/cloudera/sqoop
> >
> > If sqoop isn't appealing for some reason, you can at least take a look
> > at the code and see what Aaron did under the hood.
> >
> > On Tue, Apr 13, 2010 at 8:09 AM, Dan Harvey <dan.harvey@mendeley.com>
> > wrote:
> > > Right, after sending this e-mail out that started working straight away
> > with
> > > no changes... So setting the number of mappers in the code using :-
> > >
> > > job.getConfiguration().setInt("mapred.map.tasks", 4);
> > >
> > > allowed me to specify the number of splits/map tasks.
> > >
> > > Which lead me to the second problem I've been getting for awhile. When
> I
> > > start a hadoop job using DBInputFormat as the input if I use 5 splits
> say
> > > one will start straight away and the others will stay in the
> initializing
> > > until it is done then carry on one at a time. This doesn't happen all
> the
> > > time though and using the same code and database some will sometimes
> > start
> > > in parallel!
> > >
> > > I've read this has happened to others before but no clear solution was
> > found
> > > then.
> > >
> > > Has anyone else had this before or found a way to solve it?
> > >
> > > Thanks,
> > >
> > > On 13 April 2010 15:46, Dan Harvey <dan.harvey@mendeley.com> wrote:
> > >
> > >> Hi,
> > >>
> > >> I'm importing data from a mysql database using the DBInputFormat to go
> > over
> > >> the rows in a table and put them into HBase with the mapper but I
> can't
> > find
> > >> a way to increase the number of maps it splits the input into. I am
> > running
> > >> this on a cluster where we have 5 nodes and each node has a maximum of
> 2
> > map
> > >> tasks. So for example if I set the number of rows to import to be
> > 10,000,000
> > >> then there will only 2 maps tasks and use only two of the nodes..
> > >>
> > >> I've tried increasing the limit manually in the code with :
> > >>
> > >> job.getConfiguration().setInt("mapred.map.tasks", 4);
> > >>
> > >> increasing the number on the command line to set the same property,
> and
> > >> also increasing the number of map tasks per node.
> > >> But in all cases mapred.map.tasks is set to 2 in the job xml config
> > file.
> > >>
> > >> I've had a look at the code and DBInputFormat splits the total number
> of
> > >> rows over mapred.map.tasks, so I'm guessing it's just getting that to
> > >> change.
> > >>
> > >> It would be great if anyone has any ideas what's going on?
> > >>
> > >> Thanks,
> > >>
> > >> --
> > >> Dan Harvey | Datamining Engineer
> > >> www.mendeley.com/profiles/dan-harvey
> > >>
> > >> Mendeley Limited | London, UK | www.mendeley.com
> > >> Registered in England and Wales | Company Number 6419015
> > >>
> > >
> > >
> > >
> > > --
> > > Dan Harvey | Datamining Engineer
> > > www.mendeley.com/profiles/dan-harvey
> > >
> > > Mendeley Limited | London, UK | www.mendeley.com
> > > Registered in England and Wales | Company Number 6419015
> > >
> >
> >
> >
> > --
> > Eric Sammer
> > phone: +1-917-287-2675
> > twitter: esammer
> > data: www.cloudera.com
> >
>
>
>
> --
> Dan Harvey | Datamining Engineer
> www.mendeley.com/profiles/dan-harvey
>
> Mendeley Limited | London, UK | www.mendeley.com
> Registered in England and Wales | Company Number 6419015
>

--00504502cbe63e576304848dd9e5--

From general-return-1414-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 19 08:23:16 2010
Return-Path: <general-return-1414-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14359 invoked from network); 19 Apr 2010 08:23:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Apr 2010 08:23:15 -0000
Received: (qmail 99000 invoked by uid 500); 19 Apr 2010 08:23:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98649 invoked by uid 500); 19 Apr 2010 08:23:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98641 invoked by uid 99); 19 Apr 2010 08:23:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 08:23:11 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [203.101.103.66] (HELO krishna.sisl.co.in) (203.101.103.66)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 08:23:04 +0000
Received: from kaveri.sisl.co.in (kaveri.sisl.co.in [132.186.192.4])
	by krishna.sisl.co.in (Postfix) with ESMTP id BA75713CC1F
	for <general@hadoop.apache.org>; Mon, 19 Apr 2010 13:59:56 +0530 (IST)
Received: from INBLRK77N3MSX.in002.siemens.net (inblrk77n3msx.in002.siemens.net [132.186.221.164])
	by kaveri.sisl.co.in (Postfix) with ESMTP id 19E821F800A
	for <general@hadoop.apache.org>; Mon, 19 Apr 2010 13:52:09 +0530 (IST)
Received: from INBLRK77M1MSX.in002.siemens.net ([132.186.221.151]) by
 INBLRK77N3MSX.in002.siemens.net ([132.186.221.164]) with mapi; Mon, 19 Apr
 2010 13:50:48 +0530
From: "Goel, Mohit IN BLR SISL" <mohit.goel@siemens.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 19 Apr 2010 13:50:47 +0530
Subject: RE: Query regarding Hadoop and cloud infrastructure
Thread-Topic: Query regarding Hadoop and cloud infrastructure
Thread-Index: AcrclX8Y1mUOnKA6QuGrrzYk20mRSgDAt6ag
Message-ID: <2658E54B540D284981EA57E6A549EA70A177803B07@INBLRK77M1MSX.in002.siemens.net>
References: <2658E54B540D284981EA57E6A549EA70A1777C16F4@INBLRK77M1MSX.in002.siemens.net>
 <4BC70356.5050700@apache.org>
In-Reply-To: <4BC70356.5050700@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
x-tm-as-product-ver: SMEX-8.0.0.4125-3.800.1026-17328.005
x-tm-as-result: No--39.204700-8.000000-31
x-tm-as-user-approved-sender: No
x-tm-as-user-blocked-sender: No
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


Hello,

Thanks for the reply.

It means that we can achieve scaling up and provisioning using Hadoop. Now =
the question is How to do this?

Can I get source code to achieve the same? Or is there any service availabl=
e using which we can implement provisioning and scaling up feature in our c=
loud infrastructure?

--------------------------------------------------------
With best regards,
Mohit Goel

-----Original Message-----
From: Steve Loughran [mailto:stevel@apache.org]
Sent: Thursday, April 15, 2010 5:45 PM
To: general@hadoop.apache.org
Subject: Re: Query regarding Hadoop and cloud infrastructure

Goel, Mohit IN BLR SISL wrote:
> Hello,
>
> I have a general query regarding usage of Hadoop with my cloud infrastruc=
ture. I am trying to achieve scaling up and scaling down in cloud using Had=
oop.
>
> I have set up a cloud infrastructure which creates images consists of OS =
and applications. To access user applications, instance of image has to lau=
nch. Now I want to make this running or launched instance scalable based on=
 some condition like -
>
> a)       If no. of users who are accessing the application which is hosti=
ng in cloud (i.e. in instance) are more then it should run one more instanc=
e of image and if no. of users are less then instances should be terminated=
.
> b)       If CPU usage is more then one more instance of image should run =
or if CPU usage is less then it should terminate the instance.
>
> Can I achieve these goals using Hadoop?
>

1. Hadoop on Demand, HOD, does some of this
2. Hadoop on EC2 does some of this
3. I've been doing some of this, too; I have some slides up where I
discuss issues
http://www.slideshare.net/steve_l/new-roles-for-the-cloud

One funny for Hadoop is that it likes locality, and it likes machines
with TB of physical storage, which doesn't fit in quite as well with the
VM-on-demand story. If you look at my slides, you can see that
everything expects stable hostnames, reacts to failure by blacklisting,
not by killing the VM and creating a new one with the same HDFS volumes
mounted. There is room for improvement!



Important notice: This e-mail and any attachment there to contains corporat=
e proprietary information. If you have received it by mistake, please notif=
y us immediately by reply e-mail and delete this e-mail and its attachments=
 from your system.
Thank You.

From general-return-1415-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 19 10:52:24 2010
Return-Path: <general-return-1415-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 89294 invoked from network); 19 Apr 2010 10:52:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Apr 2010 10:52:24 -0000
Received: (qmail 78915 invoked by uid 500); 19 Apr 2010 10:52:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78555 invoked by uid 500); 19 Apr 2010 10:52:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78547 invoked by uid 99); 19 Apr 2010 10:52:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 10:52:19 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 10:52:09 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id EA1CEB7D8E
	for <general@hadoop.apache.org>; Mon, 19 Apr 2010 11:51:47 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id VcJbGpiJZfSA for <general@hadoop.apache.org>;
	Mon, 19 Apr 2010 11:51:37 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 1F3B6B7D85
	for <general@hadoop.apache.org>; Mon, 19 Apr 2010 11:51:36 +0100 (BST)
MailScanner-NULL-Check: 1272279085.23819@JyvlwSEKFaB7Tbx7ktz+iQ
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o3JApNSx011312
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Mon, 19 Apr 2010 11:51:24 +0100 (BST)
Message-ID: <4BCC35AC.3030507@apache.org>
Date: Mon, 19 Apr 2010 11:51:24 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100228)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Query regarding Hadoop and cloud infrastructure
References: <2658E54B540D284981EA57E6A549EA70A1777C16F4@INBLRK77M1MSX.in002.siemens.net> <4BC70356.5050700@apache.org> <2658E54B540D284981EA57E6A549EA70A177803B07@INBLRK77M1MSX.in002.siemens.net>
In-Reply-To: <2658E54B540D284981EA57E6A549EA70A177803B07@INBLRK77M1MSX.in002.siemens.net>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o3JApNSx011312
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

Goel, Mohit IN BLR SISL wrote:
> Hello,
> 
> Thanks for the reply.
> 
> It means that we can achieve scaling up and provisioning using Hadoop. Now the question is How to do this?
> 
> Can I get source code to achieve the same? Or is there any service available using which we can implement provisioning and scaling up feature in our cloud infrastructure?

That depends on your infrastructure. HOD is probably your best starting 
point, if you have something running eucalyptus there are more options

From general-return-1416-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 19 13:33:41 2010
Return-Path: <general-return-1416-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81670 invoked from network); 19 Apr 2010 13:33:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Apr 2010 13:33:41 -0000
Received: (qmail 7775 invoked by uid 500); 19 Apr 2010 13:33:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7691 invoked by uid 500); 19 Apr 2010 13:33:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7683 invoked by uid 99); 19 Apr 2010 13:33:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 13:33:39 +0000
X-ASF-Spam-Status: No, hits=4.1 required=10.0
	tests=AWL,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mikelin36@gmail.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 13:33:32 +0000
Received: by gyf1 with SMTP id 1so2578763gyf.35
        for <general@hadoop.apache.org>; Mon, 19 Apr 2010 06:33:11 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:received:message-id
         :subject:from:to:content-type;
        bh=bnnCBhRR7aH+ZldW82BhOWEm57pr2nFMSDggvyiN1Ls=;
        b=PTqMplJNSVhevKePMFABjv9fcoWrLe9RBFt5AIbsLJg5XnYUm+MQyg/w2mTVeBJ91r
         0PyCHuP6MhBcQ8nBMpSctn+lpc65xDPPNeqUikn0/BLdo2FsWiUvgvTR2HnBRFpBt77r
         dU8rMOxkzEYhxXlxaYMXqfWC0F+UpLNewOLek=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=h1614uCsH8yFua9FXzU3qRhsWuGRXyMDg7Axw9QSxLyDa24+eccRsXA1Uhmn1D7hiG
         4zHQjjOxvH2uZMno1r0JjF2gmuuKUKmB8yGP9WTndx2KguJPNNa/NpMrmr9bmR8oo/gk
         m0ehmHKjzr4YrBeXHkilDFeAQdV9s5Xw0pAwY=
MIME-Version: 1.0
Received: by 10.100.242.11 with HTTP; Mon, 19 Apr 2010 06:33:09 -0700 (PDT)
Date: Mon, 19 Apr 2010 21:33:09 +0800
Received: by 10.100.56.30 with SMTP id e30mr13458594ana.38.1271683990213; Mon, 
	19 Apr 2010 06:33:10 -0700 (PDT)
Message-ID: <h2o999b56ef1004190633jea5b15a2hf1924116bae09840@mail.gmail.com>
Subject: Error while random accessing Lzo file in Hadoop
From: Yuting Lin <mikelin36@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f9a6428185fd04849700c7

--001485f9a6428185fd04849700c7
Content-Type: text/plain; charset=ISO-8859-1

Hi all

I try to random access the sequence file which is blocked compressed by
"com.hadoop.compression.lzo.LzoCodec.class".  However, when the program
contains more than one seek(offset) followed by next(), it leads unexpected
error in Java (The offset passed to seek() is the beginning position of the
block).

If the sequence file is compressed by
"org.apache.hadoop.io.compress.GzipCodec.class", there is no such error.

My Linux is 32-bit 2.6.27, Lzo version is 0.1.0. OS, Java version is 1.6.0.
The log of error is shown below.

Is there any approach to random access the lzo-file in Hadoop? Thanks.
-
Regards
Yuting

---------------  S Y S T E M  ---------------

OS:lenny/sid

uname:Linux 2.6.27-14-generic #1 SMP Mon Aug 31 13:01:41 UTC 2009 i686
libc:glibc 2.8.90 NPTL 2.8.90
rlimit: STACK 8192k, CORE 0k, NPROC 26607, NOFILE 1024, AS infinity
load average:0.03 0.11 0.14

CPU:total 2 (2 cores per cpu, 1 threads per core) family 6 model 15 stepping
11, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3

Memory: 4k page, physical 3369768k(1217868k free), swap 1140572k(1140572k
free)

vm_info: Java HotSpot(TM) Server VM (11.0-b15) for linux-x86 JRE
(1.6.0_10-b33), built on Sep 26 2008 01:05:20 by "java_re" with gcc 3.2.1-7a
(J2SE release)

---------------  T H R E A D  ---------------

Current thread (0x09542000):  JavaThread "main" [_thread_in_native,
id=22346, stack(0xb7e6e000,0xb7ebf000)]

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR),
si_addr=0x09a82000

Registers:
EAX=0x00000000, EBX=0x7181cff4, ECX=0x01048f83, EDX=0x00074de0
ESP=0xb7ebd6c4, EBP=0xb7ebd6f8, ESI=0x09a0c7b5, EDI=0x09a0d21b
EIP=0x716d644b, CR2=0x09a82000, EFLAGS=0x00210212

Top of Stack: (sp=0xb7ebd6c4)
0xb7ebd6c4:   00fd419e 00000000 01048f82 09a0d21f
0xb7ebd6d4:   09a0c7b9 099fc003 09a0d21b 099fc21e
0xb7ebd6e4:   095c644c 00000022 7181cff4 09542114
0xb7ebd6f4:   09542000 b7ebd7f8 7181a994 099fc000
0xb7ebd704:   00000003 09a0d000 b7ebd764 00000000
0xb7ebd714:   00000000 b4dec439 716d6200 099fc002
0xb7ebd724:   00000000 ae190570 b7ebd72c 72a18ae0
0xb7ebd734:   b7ebd848 00000003 00010000 72a18af0

Instructions: (pc=0x716d644b)
0x716d643b:   26 00 00 00 00 8b 44 16 04 8b 7d e4 83 6d cc 04
0x716d644b:   89 44 17 04 83 c2 04 83 7d cc 03 77 e8 8d 41 fb
...

--001485f9a6428185fd04849700c7--

From general-return-1417-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 19 17:56:51 2010
Return-Path: <general-return-1417-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 28936 invoked from network); 19 Apr 2010 17:56:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Apr 2010 17:56:51 -0000
Received: (qmail 12607 invoked by uid 500); 19 Apr 2010 17:56:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12522 invoked by uid 500); 19 Apr 2010 17:56:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12514 invoked by uid 99); 19 Apr 2010 17:56:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 17:56:49 +0000
X-ASF-Spam-Status: No, hits=2.5 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jonathan.seidman@gmail.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 17:56:43 +0000
Received: by gyf1 with SMTP id 1so2689210gyf.35
        for <general@hadoop.apache.org>; Mon, 19 Apr 2010 10:56:23 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:received:message-id
         :subject:from:to:content-type;
        bh=ad9HKP44qKGwbmh9rOBgOxE7MqdxKpq3YsIIDHJAj1s=;
        b=X2G5klZL3tMvwI0kYOqtWs2Vd8OEmobetTw3usm+/FCjGrAJD6o3A8vB21qGvu29hC
         tQiSK4o8J/dJxs2pWKYL0N0VNX8nkC7fa3Ro7iPtBN3HQJLg/qgR1jfumOzJSXsTA2Af
         Rerendh1qX0JBRMA4aCbPJ79NhLSXFUyot3s8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=BZoKwYS/KLkJmjR66uVkPKslWEb1KWdFIH936G6uoO25lrjjFnWEfqsxiDlUnbfib9
         c7rc1w4yOxBbiquR8b+UCLiSUtHqJzA2CnQLNEQzAr++fjHJoli9cnxfi2BkHnSs9vj1
         nb44CghNPzs2CBcM7Y+pd1Ivj6is+Lw/DVn38=
MIME-Version: 1.0
Received: by 10.100.165.5 with HTTP; Mon, 19 Apr 2010 10:56:19 -0700 (PDT)
Date: Mon, 19 Apr 2010 12:56:19 -0500
Received: by 10.101.131.4 with SMTP id i4mr7799917ann.174.1271699779231; Mon, 
	19 Apr 2010 10:56:19 -0700 (PDT)
Message-ID: <l2y7ef0f1a91004191056t621af3bqf6b02c1509cc863b@mail.gmail.com>
Subject: [PROPOSAL] Chicago Hadoop User Group
From: Jonathan Seidman <jonathan.seidman@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c92a3f9a5c7504849aadee

--001636c92a3f9a5c7504849aadee
Content-Type: text/plain; charset=ISO-8859-1

Despite a thriving software industry, the Chicago area seems to lack a user
group for Hadoop and related technologies - Hive, HBase, nosql, cloud
computing, etc. If you agree that us Chicagoans deserve a HUG, shoot me an
email. My employer, located in the west loop, will provide space so
facilities shouldn't be an issue.

Thanks.

Jonathan Seidman

--001636c92a3f9a5c7504849aadee--

From general-return-1418-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 19 17:57:59 2010
Return-Path: <general-return-1418-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29293 invoked from network); 19 Apr 2010 17:57:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Apr 2010 17:57:59 -0000
Received: (qmail 15677 invoked by uid 500); 19 Apr 2010 17:57:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15621 invoked by uid 500); 19 Apr 2010 17:57:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15613 invoked by uid 99); 19 Apr 2010 17:57:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 17:57:58 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of msegel@navteq.com designates 65.167.11.240 as permitted sender)
Received: from [65.167.11.240] (HELO xmailchi.navteq.com) (65.167.11.240)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 17:57:50 +0000
Received: from chicago-spamfw.navteq.com (imailchi [10.8.5.161])
	by xmailchi.navteq.com (8.13.8/8.13.6) with ESMTP id o3JHdACC027558
	for <general@hadoop.apache.org>; Mon, 19 Apr 2010 12:39:10 -0500
Received: from imailchi.hq.navteq.com (localhost [127.0.0.1])
	by chicago-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id 1D8E23FE4355
	for <general@hadoop.apache.org>; Mon, 19 Apr 2010 12:57:29 -0500 (CDT)
Received: from imailchi.hq.navteq.com (imailchi.hq.navteq.com [10.8.120.104]) by chicago-spamfw.navteq.com with ESMTP id nZCshzxoLQvc8jCA for <general@hadoop.apache.org>; Mon, 19 Apr 2010 12:57:29 -0500 (CDT)
Received: from hq-ex-ht01.ad.navteq.com (hq-ex-ht01.ad.navteq.com [10.8.222.51])
	by imailchi.hq.navteq.com (8.12.9/8.12.9) with ESMTP id o3JHvSGq009700
	for <general@hadoop.apache.org>; Mon, 19 Apr 2010 12:57:29 -0500
Received: from hq-ex-mb02.ad.navteq.com ([fe80::880d:318e:e860:a8db]) by
 hq-ex-ht01.ad.navteq.com ([fe80::55a6:5b94:5c00:fe39%13]) with mapi; Mon, 19
 Apr 2010 12:57:28 -0500
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 19 Apr 2010 12:57:27 -0500
Subject: RE: [PROPOSAL] Chicago Hadoop User Group
Thread-Topic: [PROPOSAL] Chicago Hadoop User Group
Thread-Index: Acrf6bGH1M1Ag/exTSSC5iEHKfzluQAAAm5g
Message-ID: <ABC24175AFD3BE4DA15F4CD375ED413D0607CD6FFF@hq-ex-mb02.ad.navteq.com>
References: <l2y7ef0f1a91004191056t621af3bqf6b02c1509cc863b@mail.gmail.com>
In-Reply-To: <l2y7ef0f1a91004191056t621af3bqf6b02c1509cc863b@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1
X-Virus-Checked: Checked by ClamAV on apache.org

Sure,
Please contact me.

-Mike

-----Original Message-----
From: Jonathan Seidman [mailto:jonathan.seidman@gmail.com]=20
Sent: Monday, April 19, 2010 12:56 PM
To: general@hadoop.apache.org
Subject: [PROPOSAL] Chicago Hadoop User Group

Despite a thriving software industry, the Chicago area seems to lack a us=
er
group for Hadoop and related technologies - Hive, HBase, nosql, cloud
computing, etc. If you agree that us Chicagoans deserve a HUG, shoot me a=
n
email. My employer, located in the west loop, will provide space so
facilities shouldn't be an issue.

Thanks.

Jonathan Seidman


The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.

From general-return-1419-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 19 18:22:26 2010
Return-Path: <general-return-1419-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 36530 invoked from network); 19 Apr 2010 18:22:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Apr 2010 18:22:26 -0000
Received: (qmail 45727 invoked by uid 500); 19 Apr 2010 18:22:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45665 invoked by uid 500); 19 Apr 2010 18:22:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45657 invoked by uid 99); 19 Apr 2010 18:22:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 18:22:24 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of kapilmistry@gmail.com designates 209.85.219.215 as permitted sender)
Received: from [209.85.219.215] (HELO mail-ew0-f215.google.com) (209.85.219.215)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 18:22:17 +0000
Received: by ewy7 with SMTP id 7so992265ewy.31
        for <general@hadoop.apache.org>; Mon, 19 Apr 2010 11:21:56 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :from:date:received:message-id:subject:to:content-type;
        bh=/A069xzmZ4y0TRMg0su0DA//QJa3JxtwyE8w5zV0t8c=;
        b=LjYBMRwYaEwJ3hufknzM9/2nIDRrz19aCDq5cjzryOHhYrOgQ3ti91YPQy517a34Kq
         S8djk4/um2rzHq1hOD8Ge7eYvEGBUIZUv47CC6ExSl6OxYM1dvaI9yY2QvIHQRxHChEY
         Ge7CYV7GJnYuf2d9U1nideOPDVI+jo6Y3zUoo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=BBhDyM0kftN4qcFuAE+UD7fQXNIQaQFfT5B1cZn2DoAt3X2Ip4dhdYNmL1202kJ78I
         4/xGeZzKdmsNU71TtlugPwYCv0UNUsGrN5UXjP0v0hYAP97ie4TL5gWOCxzdrH5TEqiQ
         +V6vpBb7IuEQyn8zRk0Dx//49zzXwHiAs6ajk=
MIME-Version: 1.0
Received: by 10.213.10.1 with HTTP; Mon, 19 Apr 2010 11:21:34 -0700 (PDT)
In-Reply-To: <l2y7ef0f1a91004191056t621af3bqf6b02c1509cc863b@mail.gmail.com>
References: <l2y7ef0f1a91004191056t621af3bqf6b02c1509cc863b@mail.gmail.com>
From: Kapil Mistry <kapilmistry@gmail.com>
Date: Mon, 19 Apr 2010 13:21:34 -0500
Received: by 10.213.2.79 with SMTP id 15mr2795392ebi.41.1271701316098; Mon, 19 
	Apr 2010 11:21:56 -0700 (PDT)
Message-ID: <q2va0d063721004191121k424bd64ice1f3d2fdf9dea53@mail.gmail.com>
Subject: Re: [PROPOSAL] Chicago Hadoop User Group
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00151747681834ee6f04849b09ce
X-Virus-Checked: Checked by ClamAV on apache.org

--00151747681834ee6f04849b09ce
Content-Type: text/plain; charset=ISO-8859-1

I would be interested as well.

Kapil

On Mon, Apr 19, 2010 at 12:56 PM, Jonathan Seidman <
jonathan.seidman@gmail.com> wrote:

> Despite a thriving software industry, the Chicago area seems to lack a user
> group for Hadoop and related technologies - Hive, HBase, nosql, cloud
> computing, etc. If you agree that us Chicagoans deserve a HUG, shoot me an
> email. My employer, located in the west loop, will provide space so
> facilities shouldn't be an issue.
>
> Thanks.
>
> Jonathan Seidman
>

--00151747681834ee6f04849b09ce--

From general-return-1420-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 19 19:06:54 2010
Return-Path: <general-return-1420-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56065 invoked from network); 19 Apr 2010 19:06:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Apr 2010 19:06:53 -0000
Received: (qmail 6975 invoked by uid 500); 19 Apr 2010 19:06:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6907 invoked by uid 500); 19 Apr 2010 19:06:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6899 invoked by uid 99); 19 Apr 2010 19:06:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 19:06:52 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 19:06:45 +0000
Received: from wlan-snve-153-90.corp.yahoo.com (snvvpn4-10-72-168-c140.hq.corp.yahoo.com [10.72.168.140])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o3JJ5ope062301
	for <general@hadoop.apache.org>; Mon, 19 Apr 2010 12:05:50 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=HXuTkKt2LJ7eFOKt3syCHoXmYoROX4QXgQ/nEGFaIkYpQoGcMJpdJOwMT2rA/JGC
Message-Id: <8C2892F2-D5B2-4F86-A0E2-F0493CC1BFD6@yahoo-inc.com>
From: Hong Tang <htang@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <h2o999b56ef1004190633jea5b15a2hf1924116bae09840@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Error while random accessing Lzo file in Hadoop
Date: Mon, 19 Apr 2010 12:05:50 -0700
References: <h2o999b56ef1004190633jea5b15a2hf1924116bae09840@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

Yuting,

Could you please report the bug at http://code.google.com/p/hadoop-gpl-compression/issues/list? 
  Also it would be helpful if you could share your test program and  
sample data.

-Hong

On Apr 19, 2010, at 6:33 AM, Yuting Lin wrote:

> Hi all
>
> I try to random access the sequence file which is blocked compressed  
> by
> "com.hadoop.compression.lzo.LzoCodec.class".  However, when the  
> program
> contains more than one seek(offset) followed by next(), it leads  
> unexpected
> error in Java (The offset passed to seek() is the beginning position  
> of the
> block).
>
> If the sequence file is compressed by
> "org.apache.hadoop.io.compress.GzipCodec.class", there is no such  
> error.
>
> My Linux is 32-bit 2.6.27, Lzo version is 0.1.0. OS, Java version is  
> 1.6.0.
> The log of error is shown below.
>
> Is there any approach to random access the lzo-file in Hadoop? Thanks.
> -
> Regards
> Yuting
>
> ---------------  S Y S T E M  ---------------
>
> OS:lenny/sid
>
> uname:Linux 2.6.27-14-generic #1 SMP Mon Aug 31 13:01:41 UTC 2009 i686
> libc:glibc 2.8.90 NPTL 2.8.90
> rlimit: STACK 8192k, CORE 0k, NPROC 26607, NOFILE 1024, AS infinity
> load average:0.03 0.11 0.14
>
> CPU:total 2 (2 cores per cpu, 1 threads per core) family 6 model 15  
> stepping
> 11, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3
>
> Memory: 4k page, physical 3369768k(1217868k free), swap  
> 1140572k(1140572k
> free)
>
> vm_info: Java HotSpot(TM) Server VM (11.0-b15) for linux-x86 JRE
> (1.6.0_10-b33), built on Sep 26 2008 01:05:20 by "java_re" with gcc  
> 3.2.1-7a
> (J2SE release)
>
> ---------------  T H R E A D  ---------------
>
> Current thread (0x09542000):  JavaThread "main" [_thread_in_native,
> id=22346, stack(0xb7e6e000,0xb7ebf000)]
>
> siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR),
> si_addr=0x09a82000
>
> Registers:
> EAX=0x00000000, EBX=0x7181cff4, ECX=0x01048f83, EDX=0x00074de0
> ESP=0xb7ebd6c4, EBP=0xb7ebd6f8, ESI=0x09a0c7b5, EDI=0x09a0d21b
> EIP=0x716d644b, CR2=0x09a82000, EFLAGS=0x00210212
>
> Top of Stack: (sp=0xb7ebd6c4)
> 0xb7ebd6c4:   00fd419e 00000000 01048f82 09a0d21f
> 0xb7ebd6d4:   09a0c7b9 099fc003 09a0d21b 099fc21e
> 0xb7ebd6e4:   095c644c 00000022 7181cff4 09542114
> 0xb7ebd6f4:   09542000 b7ebd7f8 7181a994 099fc000
> 0xb7ebd704:   00000003 09a0d000 b7ebd764 00000000
> 0xb7ebd714:   00000000 b4dec439 716d6200 099fc002
> 0xb7ebd724:   00000000 ae190570 b7ebd72c 72a18ae0
> 0xb7ebd734:   b7ebd848 00000003 00010000 72a18af0
>
> Instructions: (pc=0x716d644b)
> 0x716d643b:   26 00 00 00 00 8b 44 16 04 8b 7d e4 83 6d cc 04
> 0x716d644b:   89 44 17 04 83 c2 04 83 7d cc 03 77 e8 8d 41 fb
> ...


From general-return-1421-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 19 19:10:43 2010
Return-Path: <general-return-1421-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57327 invoked from network); 19 Apr 2010 19:10:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Apr 2010 19:10:43 -0000
Received: (qmail 11816 invoked by uid 500); 19 Apr 2010 19:10:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11782 invoked by uid 500); 19 Apr 2010 19:10:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11774 invoked by uid 99); 19 Apr 2010 19:10:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 19:10:42 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ryanobjc@gmail.com designates 209.85.212.176 as permitted sender)
Received: from [209.85.212.176] (HELO mail-px0-f176.google.com) (209.85.212.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 19:10:34 +0000
Received: by pxi10 with SMTP id 10so112502pxi.35
        for <general@hadoop.apache.org>; Mon, 19 Apr 2010 12:10:13 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=fLIRkYR8O5dmaWT8zwwE11BdmrePIFPBEAhwYXMH/Ls=;
        b=tgxR4b+8+BQ9FoUU7SYaY2R9ReKcsLwSEli+XUs1LcpTL2T/6IQYx0kZh8eS6vIR80
         LlCVgdKLzVCP74WSjKzOCVqAKfiuL0J3qAi0+9kx5AoFDUzi9coCeb6EtzzNGsqeRsM6
         uQPZPdo6S+R/WqArxWAlkzFg4dkqg/6LNzGhg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=eAOJsp9j1HTPKOZwHgWi+2VLujICFqQZ9ZNT0MZIBFhy6RxPzbg+MndqW67MgF/AOb
         jSSt+mxhs7NF+N28duX3N6NFWJa8amiSiAEAKBVpg/wUJs+jcBW+H6Qe/pW1U4fXcwUs
         ZVRPh9lGCLfTXfkAc6QVffmEaKhiYFHR13+uE=
MIME-Version: 1.0
Received: by 10.231.20.212 with HTTP; Mon, 19 Apr 2010 12:10:12 -0700 (PDT)
In-Reply-To: <h2o999b56ef1004190633jea5b15a2hf1924116bae09840@mail.gmail.com>
References: <h2o999b56ef1004190633jea5b15a2hf1924116bae09840@mail.gmail.com>
Date: Mon, 19 Apr 2010 12:10:12 -0700
Received: by 10.140.247.16 with SMTP id u16mr4650717rvh.215.1271704213479; 
	Mon, 19 Apr 2010 12:10:13 -0700 (PDT)
Message-ID: <h2h78568af11004191210j8b28ecb6o3af1eb4ac0c3f81@mail.gmail.com>
Subject: Re: Error while random accessing Lzo file in Hadoop
From: Ryan Rawson <ryanobjc@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Hey,

You are running an extremely old JVM.  Could you try with JDK
1.6.0_19?  (or at least 14)

Also your ulimit -n is fairly low 1024 file handles is not enough.  Try 32k

-ryan

On Mon, Apr 19, 2010 at 6:33 AM, Yuting Lin <mikelin36@gmail.com> wrote:
> Hi all
>
> I try to random access the sequence file which is blocked compressed by
> "com.hadoop.compression.lzo.LzoCodec.class". =A0However, when the program
> contains more than one seek(offset) followed by next(), it leads unexpect=
ed
> error in Java (The offset passed to seek() is the beginning position of t=
he
> block).
>
> If the sequence file is compressed by
> "org.apache.hadoop.io.compress.GzipCodec.class", there is no such error.
>
> My Linux is 32-bit 2.6.27, Lzo version is 0.1.0. OS, Java version is 1.6.=
0.
> The log of error is shown below.
>
> Is there any approach to random access the lzo-file in Hadoop? Thanks.
> -
> Regards
> Yuting
>
> --------------- =A0S Y S T E M =A0---------------
>
> OS:lenny/sid
>
> uname:Linux 2.6.27-14-generic #1 SMP Mon Aug 31 13:01:41 UTC 2009 i686
> libc:glibc 2.8.90 NPTL 2.8.90
> rlimit: STACK 8192k, CORE 0k, NPROC 26607, NOFILE 1024, AS infinity
> load average:0.03 0.11 0.14
>
> CPU:total 2 (2 cores per cpu, 1 threads per core) family 6 model 15 stepp=
ing
> 11, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3
>
> Memory: 4k page, physical 3369768k(1217868k free), swap 1140572k(1140572k
> free)
>
> vm_info: Java HotSpot(TM) Server VM (11.0-b15) for linux-x86 JRE
> (1.6.0_10-b33), built on Sep 26 2008 01:05:20 by "java_re" with gcc 3.2.1=
-7a
> (J2SE release)
>
> --------------- =A0T H R E A D =A0---------------
>
> Current thread (0x09542000): =A0JavaThread "main" [_thread_in_native,
> id=3D22346, stack(0xb7e6e000,0xb7ebf000)]
>
> siginfo:si_signo=3DSIGSEGV: si_errno=3D0, si_code=3D1 (SEGV_MAPERR),
> si_addr=3D0x09a82000
>
> Registers:
> EAX=3D0x00000000, EBX=3D0x7181cff4, ECX=3D0x01048f83, EDX=3D0x00074de0
> ESP=3D0xb7ebd6c4, EBP=3D0xb7ebd6f8, ESI=3D0x09a0c7b5, EDI=3D0x09a0d21b
> EIP=3D0x716d644b, CR2=3D0x09a82000, EFLAGS=3D0x00210212
>
> Top of Stack: (sp=3D0xb7ebd6c4)
> 0xb7ebd6c4: =A0 00fd419e 00000000 01048f82 09a0d21f
> 0xb7ebd6d4: =A0 09a0c7b9 099fc003 09a0d21b 099fc21e
> 0xb7ebd6e4: =A0 095c644c 00000022 7181cff4 09542114
> 0xb7ebd6f4: =A0 09542000 b7ebd7f8 7181a994 099fc000
> 0xb7ebd704: =A0 00000003 09a0d000 b7ebd764 00000000
> 0xb7ebd714: =A0 00000000 b4dec439 716d6200 099fc002
> 0xb7ebd724: =A0 00000000 ae190570 b7ebd72c 72a18ae0
> 0xb7ebd734: =A0 b7ebd848 00000003 00010000 72a18af0
>
> Instructions: (pc=3D0x716d644b)
> 0x716d643b: =A0 26 00 00 00 00 8b 44 16 04 8b 7d e4 83 6d cc 04
> 0x716d644b: =A0 89 44 17 04 83 c2 04 83 7d cc 03 77 e8 8d 41 fb
> ...
>

From general-return-1422-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 02:31:31 2010
Return-Path: <general-return-1422-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17652 invoked from network); 20 Apr 2010 02:31:31 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 02:31:31 -0000
Received: (qmail 64824 invoked by uid 500); 20 Apr 2010 02:31:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 64564 invoked by uid 500); 20 Apr 2010 02:31:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 64556 invoked by uid 99); 20 Apr 2010 02:31:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 02:31:28 +0000
X-ASF-Spam-Status: No, hits=4.1 required=10.0
	tests=AWL,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mikelin36@gmail.com designates 209.85.217.225 as permitted sender)
Received: from [209.85.217.225] (HELO mail-gx0-f225.google.com) (209.85.217.225)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 02:31:22 +0000
Received: by gxk25 with SMTP id 25so3121049gxk.11
        for <general@hadoop.apache.org>; Mon, 19 Apr 2010 19:31:01 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type;
        bh=dpX5Zmw4X5qpNtpxgXTzhB65xFQ16GqWa+lzvWt1f1A=;
        b=sjbQNeNqlw086jypO2TIlR7Gu3pvwNibVNuF6HSohK+RUFTDjHAe9VnPQlOn62eQkn
         v+U/C0/ffEpvvpOjBojQRtB0/evQXdn0jllbpAatlRC5Uuk1OM2a4QBqev75MoCHWmxD
         oUL1TKmQJSWAm+lLPZJ9/N1qQyzaDl8M+6NhY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=plKXphZG1vN5NFIJmqqCNLPGDgIFyBVq8Mn/jR2maF9h+gi1qkt/qDWU/REbym8lGV
         Nu9SrE/E8UOBdWhkwCL04wBk1cHyp+yCasWKoA9cO326vQgLsiUGZVw10tFpgRfXt9lH
         yWS4T3lLW28Zl31SyYhdpMHrvGFxLJtfYWkk4=
MIME-Version: 1.0
Received: by 10.100.242.11 with HTTP; Mon, 19 Apr 2010 19:30:55 -0700 (PDT)
In-Reply-To: <h2h78568af11004191210j8b28ecb6o3af1eb4ac0c3f81@mail.gmail.com>
References: <h2o999b56ef1004190633jea5b15a2hf1924116bae09840@mail.gmail.com>
	 <h2h78568af11004191210j8b28ecb6o3af1eb4ac0c3f81@mail.gmail.com>
Date: Tue, 20 Apr 2010 10:30:55 +0800
Received: by 10.101.192.33 with SMTP id u33mr14907975anp.237.1271730655407; 
	Mon, 19 Apr 2010 19:30:55 -0700 (PDT)
Message-ID: <q2g999b56ef1004191930ka3f112ck2580835657571c7f@mail.gmail.com>
Subject: Re: Error while random accessing Lzo file in Hadoop
From: Yuting Lin <mikelin36@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c92c56f79d580484a1ddf0

--001636c92c56f79d580484a1ddf0
Content-Type: text/plain; charset=ISO-8859-1

Hi Hong and Ryan

Thanks for your suggestion.

I update the jdk to 1.6.0_20 and try but it doesn't solves the problems. The
nofile doesn't affect the codes here since there is only one thread read the
file, and it's the second pair of seek() and next() that caused the error.

I have reported the bug at
http://code.google.com/p/hadoop-gpl-compression/issues/list and shared the
test program.

Thanks.
-
Regards
Yuting

On Tue, Apr 20, 2010 at 3:10 AM, Ryan Rawson <ryanobjc@gmail.com> wrote:

> Hey,
>
> You are running an extremely old JVM.  Could you try with JDK
> 1.6.0_19?  (or at least 14)
>
> Also your ulimit -n is fairly low 1024 file handles is not enough.  Try 32k
>
> -ryan
>
> On Mon, Apr 19, 2010 at 6:33 AM, Yuting Lin <mikelin36@gmail.com> wrote:
> > Hi all
> >
> > I try to random access the sequence file which is blocked compressed by
> > "com.hadoop.compression.lzo.LzoCodec.class".  However, when the program
> > contains more than one seek(offset) followed by next(), it leads
> unexpected
> > error in Java (The offset passed to seek() is the beginning position of
> the
> > block).
> >
> > If the sequence file is compressed by
> > "org.apache.hadoop.io.compress.GzipCodec.class", there is no such error.
> >
> > My Linux is 32-bit 2.6.27, Lzo version is 0.1.0. OS, Java version is
> 1.6.0.
> > The log of error is shown below.
> >
> > Is there any approach to random access the lzo-file in Hadoop? Thanks.
> > -
> > Regards
> > Yuting
> >
> > ---------------  S Y S T E M  ---------------
> >
> > OS:lenny/sid
> >
> > uname:Linux 2.6.27-14-generic #1 SMP Mon Aug 31 13:01:41 UTC 2009 i686
> > libc:glibc 2.8.90 NPTL 2.8.90
> > rlimit: STACK 8192k, CORE 0k, NPROC 26607, NOFILE 1024, AS infinity
> > load average:0.03 0.11 0.14
> >
> > CPU:total 2 (2 cores per cpu, 1 threads per core) family 6 model 15
> stepping
> > 11, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3
> >
> > Memory: 4k page, physical 3369768k(1217868k free), swap 1140572k(1140572k
> > free)
> >
> > vm_info: Java HotSpot(TM) Server VM (11.0-b15) for linux-x86 JRE
> > (1.6.0_10-b33), built on Sep 26 2008 01:05:20 by "java_re" with gcc
> 3.2.1-7a
> > (J2SE release)
> >
> > ---------------  T H R E A D  ---------------
> >
> > Current thread (0x09542000):  JavaThread "main" [_thread_in_native,
> > id=22346, stack(0xb7e6e000,0xb7ebf000)]
> >
> > siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR),
> > si_addr=0x09a82000
> >
> > Registers:
> > EAX=0x00000000, EBX=0x7181cff4, ECX=0x01048f83, EDX=0x00074de0
> > ESP=0xb7ebd6c4, EBP=0xb7ebd6f8, ESI=0x09a0c7b5, EDI=0x09a0d21b
> > EIP=0x716d644b, CR2=0x09a82000, EFLAGS=0x00210212
> >
> > Top of Stack: (sp=0xb7ebd6c4)
> > 0xb7ebd6c4:   00fd419e 00000000 01048f82 09a0d21f
> > 0xb7ebd6d4:   09a0c7b9 099fc003 09a0d21b 099fc21e
> > 0xb7ebd6e4:   095c644c 00000022 7181cff4 09542114
> > 0xb7ebd6f4:   09542000 b7ebd7f8 7181a994 099fc000
> > 0xb7ebd704:   00000003 09a0d000 b7ebd764 00000000
> > 0xb7ebd714:   00000000 b4dec439 716d6200 099fc002
> > 0xb7ebd724:   00000000 ae190570 b7ebd72c 72a18ae0
> > 0xb7ebd734:   b7ebd848 00000003 00010000 72a18af0
> >
> > Instructions: (pc=0x716d644b)
> > 0x716d643b:   26 00 00 00 00 8b 44 16 04 8b 7d e4 83 6d cc 04
> > 0x716d644b:   89 44 17 04 83 c2 04 83 7d cc 03 77 e8 8d 41 fb
> > ...
> >
>

--001636c92c56f79d580484a1ddf0--

From general-return-1423-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 03:29:10 2010
Return-Path: <general-return-1423-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35031 invoked from network); 20 Apr 2010 03:29:10 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 03:29:10 -0000
Received: (qmail 2589 invoked by uid 500); 20 Apr 2010 03:29:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2193 invoked by uid 500); 20 Apr 2010 03:29:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2185 invoked by uid 99); 20 Apr 2010 03:29:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 03:29:05 +0000
X-ASF-Spam-Status: No, hits=1.4 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of haoguohua@gmail.com designates 209.85.211.174 as permitted sender)
Received: from [209.85.211.174] (HELO mail-yw0-f174.google.com) (209.85.211.174)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 03:28:59 +0000
Received: by ywh4 with SMTP id 4so2745634ywh.5
        for <general@hadoop.apache.org>; Mon, 19 Apr 2010 20:28:38 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type;
        bh=qGBzBnofMCXmEWuXsqGpRS8vOyuHJNYlDJOmC+p8yoY=;
        b=fSZQBoueaTtAcaZbFIbIF5B5G3VIcp/0FUlNeSuVCNalOMT7zMD0kgW84Jnh1YscFz
         KBkPIu0Nog7awOpOIaHnb60UWA7FadP3w0fuRfmprLe02FsIvvrRaDYKjTn7688tb3cm
         Srn2Ca+7/vHiKbfygUs7/2HwhuwaGbHJW46Kw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=TNLPRGnwtq7Hd7cRpypIhiLd2VNFFnxJMhmfAAyWWqLVbeooiVaIF+85JOVEh92nvQ
         YgNpE5ZBr7Eq8EIV/i6SlbIpnCy0DuiRUMwjLwknjnwp55XEssl/05CPuGy3FeP0pJBV
         hffclwn35Qi37Ub7o5lnRia2aM1relbWiTTvI=
MIME-Version: 1.0
Received: by 10.151.111.18 with HTTP; Mon, 19 Apr 2010 20:28:38 -0700 (PDT)
In-Reply-To: <q2va0d063721004191121k424bd64ice1f3d2fdf9dea53@mail.gmail.com>
References: <l2y7ef0f1a91004191056t621af3bqf6b02c1509cc863b@mail.gmail.com>
	 <q2va0d063721004191121k424bd64ice1f3d2fdf9dea53@mail.gmail.com>
Date: Mon, 19 Apr 2010 22:28:38 -0500
Received: by 10.150.235.9 with SMTP id i9mr6764029ybh.239.1271734118057; Mon, 
	19 Apr 2010 20:28:38 -0700 (PDT)
Message-ID: <p2u519188da1004192028i866febacj9a19fd643f7e7f35@mail.gmail.com>
Subject: Re: [PROPOSAL] Chicago Hadoop User Group
From: Guohua Hao <haoguohua@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd3417a5b096c0484a2acdf

--000e0cd3417a5b096c0484a2acdf
Content-Type: text/plain; charset=ISO-8859-1

I would like to join as well.

-- Guohua

On Mon, Apr 19, 2010 at 1:21 PM, Kapil Mistry <kapilmistry@gmail.com> wrote:

> I would be interested as well.
>
> Kapil
>
> On Mon, Apr 19, 2010 at 12:56 PM, Jonathan Seidman <
> jonathan.seidman@gmail.com> wrote:
>
> > Despite a thriving software industry, the Chicago area seems to lack a
> user
> > group for Hadoop and related technologies - Hive, HBase, nosql, cloud
> > computing, etc. If you agree that us Chicagoans deserve a HUG, shoot me
> an
> > email. My employer, located in the west loop, will provide space so
> > facilities shouldn't be an issue.
> >
> > Thanks.
> >
> > Jonathan Seidman
> >
>

--000e0cd3417a5b096c0484a2acdf--

From general-return-1424-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 03:33:21 2010
Return-Path: <general-return-1424-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 36356 invoked from network); 20 Apr 2010 03:33:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 03:33:21 -0000
Received: (qmail 5667 invoked by uid 500); 20 Apr 2010 03:33:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5527 invoked by uid 500); 20 Apr 2010 03:33:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5519 invoked by uid 99); 20 Apr 2010 03:33:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 03:33:17 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=AWL,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of msegel@navteq.com designates 65.167.11.240 as permitted sender)
Received: from [65.167.11.240] (HELO xmailchi.navteq.com) (65.167.11.240)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 03:33:11 +0000
Received: from chicago-spamfw.navteq.com (imailchi [10.8.5.161])
	by xmailchi.navteq.com (8.13.8/8.13.6) with ESMTP id o3K3EPou009302
	for <general@hadoop.apache.org>; Mon, 19 Apr 2010 22:14:25 -0500
Received: from imailchi.hq.navteq.com (localhost [127.0.0.1])
	by chicago-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id 7F4DC43220F9
	for <general@hadoop.apache.org>; Mon, 19 Apr 2010 22:32:45 -0500 (CDT)
Received: from imailchi.hq.navteq.com (imailchi.hq.navteq.com [10.8.120.104]) by chicago-spamfw.navteq.com with ESMTP id WlTn4vIlkqgc5XvQ for <general@hadoop.apache.org>; Mon, 19 Apr 2010 22:32:45 -0500 (CDT)
Received: from hq-ex-ht01.ad.navteq.com (hq-ex-ht01.ad.navteq.com [10.8.222.51])
	by imailchi.hq.navteq.com (8.12.9/8.12.9) with ESMTP id o3K3WjGq027745
	for <general@hadoop.apache.org>; Mon, 19 Apr 2010 22:32:45 -0500
Received: from hq-ex-mb02.ad.navteq.com ([fe80::880d:318e:e860:a8db]) by
 hq-ex-ht01.ad.navteq.com ([fe80::55a6:5b94:5c00:fe39%13]) with mapi; Mon, 19
 Apr 2010 22:32:44 -0500
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 19 Apr 2010 22:32:43 -0500
Subject: RE: [PROPOSAL] Chicago Hadoop User Group
Thread-Topic: [PROPOSAL] Chicago Hadoop User Group
Thread-Index: AcrgOaTxC3apEgyKRS6PL7IMEFfMWgAAA+JQ
Message-ID: <ABC24175AFD3BE4DA15F4CD375ED413D0607D373B1@hq-ex-mb02.ad.navteq.com>
References: <l2y7ef0f1a91004191056t621af3bqf6b02c1509cc863b@mail.gmail.com>
	 <q2va0d063721004191121k424bd64ice1f3d2fdf9dea53@mail.gmail.com>
 <p2u519188da1004192028i866febacj9a19fd643f7e7f35@mail.gmail.com>
In-Reply-To: <p2u519188da1004192028i866febacj9a19fd643f7e7f35@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1

Hey!
Not to steal Jonathan's thunder, but if you want to e-mail me your contac=
t info, I'll create a spread sheet so we can figure out a date/time/place=
, and create a mailing list to take this offline from the general mailing=
 list.

I'm sure that guys who are more than an hour (non-rush hour traffic) outs=
ide of Chicago will want to venture in to town. (Ok maybe someone from Mi=
lwaukee might hop a train?)


Thx

-Mike


-----Original Message-----
From: Guohua Hao [mailto:haoguohua@gmail.com]=20
Sent: Monday, April 19, 2010 10:29 PM
To: general@hadoop.apache.org
Subject: Re: [PROPOSAL] Chicago Hadoop User Group

I would like to join as well.

-- Guohua

On Mon, Apr 19, 2010 at 1:21 PM, Kapil Mistry <kapilmistry@gmail.com> wro=
te:

> I would be interested as well.
>
> Kapil
>
> On Mon, Apr 19, 2010 at 12:56 PM, Jonathan Seidman <
> jonathan.seidman@gmail.com> wrote:
>
> > Despite a thriving software industry, the Chicago area seems to lack =
a
> user
> > group for Hadoop and related technologies - Hive, HBase, nosql, cloud
> > computing, etc. If you agree that us Chicagoans deserve a HUG, shoot =
me
> an
> > email. My employer, located in the west loop, will provide space so
> > facilities shouldn't be an issue.
> >
> > Thanks.
> >
> > Jonathan Seidman
> >
>


The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.

From general-return-1425-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 05:57:28 2010
Return-Path: <general-return-1425-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59602 invoked from network); 20 Apr 2010 05:57:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 05:57:27 -0000
Received: (qmail 64365 invoked by uid 500); 20 Apr 2010 05:57:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63924 invoked by uid 500); 20 Apr 2010 05:57:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63913 invoked by uid 99); 20 Apr 2010 05:57:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 05:57:23 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 05:57:15 +0000
Received: from [192.168.2.121] ([10.73.152.37])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o3K5uQlk068562
	for <general@hadoop.apache.org>; Mon, 19 Apr 2010 22:56:26 -0700 (PDT)
Message-Id: <87E99CEB-0A57-4434-8C4A-AE6C3384E56B@yahoo-inc.com>
From: Hong Tang <htang@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <q2g999b56ef1004191930ka3f112ck2580835657571c7f@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Error while random accessing Lzo file in Hadoop
Date: Mon, 19 Apr 2010 22:56:26 -0700
References: <h2o999b56ef1004190633jea5b15a2hf1924116bae09840@mail.gmail.com> <h2h78568af11004191210j8b28ecb6o3af1eb4ac0c3f81@mail.gmail.com> <q2g999b56ef1004191930ka3f112ck2580835657571c7f@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

Yuting,

Thanks for reporting the bug, I will take a look.

-Hong

On Apr 19, 2010, at 7:30 PM, Yuting Lin wrote:

> Hi Hong and Ryan
>
> Thanks for your suggestion.
>
> I update the jdk to 1.6.0_20 and try but it doesn't solves the  
> problems. The
> nofile doesn't affect the codes here since there is only one thread  
> read the
> file, and it's the second pair of seek() and next() that caused the  
> error.
>
> I have reported the bug at
> http://code.google.com/p/hadoop-gpl-compression/issues/list and  
> shared the
> test program.
>
> Thanks.
> -
> Regards
> Yuting
>
> On Tue, Apr 20, 2010 at 3:10 AM, Ryan Rawson <ryanobjc@gmail.com>  
> wrote:
>
>> Hey,
>>
>> You are running an extremely old JVM.  Could you try with JDK
>> 1.6.0_19?  (or at least 14)
>>
>> Also your ulimit -n is fairly low 1024 file handles is not enough.   
>> Try 32k
>>
>> -ryan
>>
>> On Mon, Apr 19, 2010 at 6:33 AM, Yuting Lin <mikelin36@gmail.com>  
>> wrote:
>>> Hi all
>>>
>>> I try to random access the sequence file which is blocked  
>>> compressed by
>>> "com.hadoop.compression.lzo.LzoCodec.class".  However, when the  
>>> program
>>> contains more than one seek(offset) followed by next(), it leads
>> unexpected
>>> error in Java (The offset passed to seek() is the beginning  
>>> position of
>> the
>>> block).
>>>
>>> If the sequence file is compressed by
>>> "org.apache.hadoop.io.compress.GzipCodec.class", there is no such  
>>> error.
>>>
>>> My Linux is 32-bit 2.6.27, Lzo version is 0.1.0. OS, Java version is
>> 1.6.0.
>>> The log of error is shown below.
>>>
>>> Is there any approach to random access the lzo-file in Hadoop?  
>>> Thanks.
>>> -
>>> Regards
>>> Yuting
>>>
>>> ---------------  S Y S T E M  ---------------
>>>
>>> OS:lenny/sid
>>>
>>> uname:Linux 2.6.27-14-generic #1 SMP Mon Aug 31 13:01:41 UTC 2009  
>>> i686
>>> libc:glibc 2.8.90 NPTL 2.8.90
>>> rlimit: STACK 8192k, CORE 0k, NPROC 26607, NOFILE 1024, AS infinity
>>> load average:0.03 0.11 0.14
>>>
>>> CPU:total 2 (2 cores per cpu, 1 threads per core) family 6 model 15
>> stepping
>>> 11, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3
>>>
>>> Memory: 4k page, physical 3369768k(1217868k free), swap  
>>> 1140572k(1140572k
>>> free)
>>>
>>> vm_info: Java HotSpot(TM) Server VM (11.0-b15) for linux-x86 JRE
>>> (1.6.0_10-b33), built on Sep 26 2008 01:05:20 by "java_re" with gcc
>> 3.2.1-7a
>>> (J2SE release)
>>>
>>> ---------------  T H R E A D  ---------------
>>>
>>> Current thread (0x09542000):  JavaThread "main" [_thread_in_native,
>>> id=22346, stack(0xb7e6e000,0xb7ebf000)]
>>>
>>> siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR),
>>> si_addr=0x09a82000
>>>
>>> Registers:
>>> EAX=0x00000000, EBX=0x7181cff4, ECX=0x01048f83, EDX=0x00074de0
>>> ESP=0xb7ebd6c4, EBP=0xb7ebd6f8, ESI=0x09a0c7b5, EDI=0x09a0d21b
>>> EIP=0x716d644b, CR2=0x09a82000, EFLAGS=0x00210212
>>>
>>> Top of Stack: (sp=0xb7ebd6c4)
>>> 0xb7ebd6c4:   00fd419e 00000000 01048f82 09a0d21f
>>> 0xb7ebd6d4:   09a0c7b9 099fc003 09a0d21b 099fc21e
>>> 0xb7ebd6e4:   095c644c 00000022 7181cff4 09542114
>>> 0xb7ebd6f4:   09542000 b7ebd7f8 7181a994 099fc000
>>> 0xb7ebd704:   00000003 09a0d000 b7ebd764 00000000
>>> 0xb7ebd714:   00000000 b4dec439 716d6200 099fc002
>>> 0xb7ebd724:   00000000 ae190570 b7ebd72c 72a18ae0
>>> 0xb7ebd734:   b7ebd848 00000003 00010000 72a18af0
>>>
>>> Instructions: (pc=0x716d644b)
>>> 0x716d643b:   26 00 00 00 00 8b 44 16 04 8b 7d e4 83 6d cc 04
>>> 0x716d644b:   89 44 17 04 83 c2 04 83 7d cc 03 77 e8 8d 41 fb
>>> ...
>>>
>>


From general-return-1426-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 14:07:42 2010
Return-Path: <general-return-1426-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16692 invoked from network); 20 Apr 2010 14:07:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 14:07:42 -0000
Received: (qmail 52105 invoked by uid 500); 20 Apr 2010 14:07:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52031 invoked by uid 500); 20 Apr 2010 14:07:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52023 invoked by uid 99); 20 Apr 2010 14:07:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 14:07:40 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jonathan.seidman@gmail.com designates 74.125.83.48 as permitted sender)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 14:07:33 +0000
Received: by gwb1 with SMTP id 1so68415gwb.35
        for <general@hadoop.apache.org>; Tue, 20 Apr 2010 07:07:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type;
        bh=YTnX6gsOwKtMGzOeV1r01nTXDPnSgzDNpvHdCdtEj9w=;
        b=EU8ctf8P+laR37EH/YOvhwwrIcnLfqv4AdNRnWVE5lWQSDbRR7OYSNypvHT4KbashM
         JjWca97VJtx1Xy7MRP0hOweAgpHWf6P9SKh6D6GOUmnoz9FFJXAvuRTnu+DIxHZgj9H8
         8GCTpeEcq9462l+y1Wch7wyhNe6KNngwjsHLg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=fJXYtG08rrfoH43R2WkRzTTlqRvKwaG6QtrE4mZ2wirZBod6u+/SoystidA6ZnAC7a
         b5sj8APRd26e07tP3+CyThm8Z7X0xLHrnwXfi4geN9cjDqmJKVn3cqXF8GpFL3eAblVX
         ufsN7LEJ4qkFwN4NINFnvyfB1FNso6hq5xVlc=
MIME-Version: 1.0
Received: by 10.100.165.5 with HTTP; Tue, 20 Apr 2010 07:07:10 -0700 (PDT)
In-Reply-To: <ABC24175AFD3BE4DA15F4CD375ED413D0607D373B1@hq-ex-mb02.ad.navteq.com>
References: <l2y7ef0f1a91004191056t621af3bqf6b02c1509cc863b@mail.gmail.com>
	 <q2va0d063721004191121k424bd64ice1f3d2fdf9dea53@mail.gmail.com>
	 <p2u519188da1004192028i866febacj9a19fd643f7e7f35@mail.gmail.com>
	 <ABC24175AFD3BE4DA15F4CD375ED413D0607D373B1@hq-ex-mb02.ad.navteq.com>
Date: Tue, 20 Apr 2010 09:07:10 -0500
Received: by 10.100.56.30 with SMTP id e30mr17141273ana.38.1271772430865; Tue, 
	20 Apr 2010 07:07:10 -0700 (PDT)
Message-ID: <s2n7ef0f1a91004200707re6dbb383o7054b2d4d095a349@mail.gmail.com>
Subject: Re: [PROPOSAL] Chicago Hadoop User Group
From: Jonathan Seidman <jonathan.seidman@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f9a642fb3f940484ab9793
X-Virus-Checked: Checked by ClamAV on apache.org

--001485f9a642fb3f940484ab9793
Content-Type: text/plain; charset=ISO-8859-1

MIke - Thanks! That'd be awesome. I'll send you an email to sync up.

Jonathan

On Mon, Apr 19, 2010 at 10:32 PM, Segel, Mike <msegel@navteq.com> wrote:

> Hey!
> Not to steal Jonathan's thunder, but if you want to e-mail me your contact
> info, I'll create a spread sheet so we can figure out a date/time/place, and
> create a mailing list to take this offline from the general mailing list.
>
> I'm sure that guys who are more than an hour (non-rush hour traffic)
> outside of Chicago will want to venture in to town. (Ok maybe someone from
> Milwaukee might hop a train?)
>
>
> Thx
>
> -Mike
>
>
> -----Original Message-----
> From: Guohua Hao [mailto:haoguohua@gmail.com]
> Sent: Monday, April 19, 2010 10:29 PM
> To: general@hadoop.apache.org
> Subject: Re: [PROPOSAL] Chicago Hadoop User Group
>
> I would like to join as well.
>
> -- Guohua
>
> On Mon, Apr 19, 2010 at 1:21 PM, Kapil Mistry <kapilmistry@gmail.com>
> wrote:
>
> > I would be interested as well.
> >
> > Kapil
> >
> > On Mon, Apr 19, 2010 at 12:56 PM, Jonathan Seidman <
> > jonathan.seidman@gmail.com> wrote:
> >
> > > Despite a thriving software industry, the Chicago area seems to lack a
> > user
> > > group for Hadoop and related technologies - Hive, HBase, nosql, cloud
> > > computing, etc. If you agree that us Chicagoans deserve a HUG, shoot me
> > an
> > > email. My employer, located in the west loop, will provide space so
> > > facilities shouldn't be an issue.
> > >
> > > Thanks.
> > >
> > > Jonathan Seidman
> > >
> >
>
>
> The information contained in this communication may be CONFIDENTIAL and is
> intended only for the use of the recipient(s) named above.  If you are not
> the intended recipient, you are hereby notified that any dissemination,
> distribution, or copying of this communication, or any of its contents, is
> strictly prohibited.  If you have received this communication in error,
> please notify the sender and delete/destroy the original message and any
> copy of it from your computer or paper files.
>

--001485f9a642fb3f940484ab9793--

From general-return-1427-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 07:11:23 2010
Return-Path: <general-return-1427-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24127 invoked from network); 21 Apr 2010 07:11:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 07:11:23 -0000
Received: (qmail 9953 invoked by uid 500); 21 Apr 2010 07:11:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 9667 invoked by uid 500); 21 Apr 2010 07:11:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 9652 invoked by uid 99); 21 Apr 2010 07:11:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 07:11:19 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [88.198.2.104] (HELO koch.ro) (88.198.2.104)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 07:11:11 +0000
Received: from 84-72-85-88.dclient.hispeed.ch ([84.72.85.88] helo=jona.localnet)
	by koch.ro with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
	(Exim 4.69)
	(envelope-from <thomas@koch.ro>)
	id 1O4Tzy-0004Gc-Q1
	for general@hadoop.apache.org; Wed, 21 Apr 2010 09:05:26 +0200
From: Thomas Koch <thomas@koch.ro>
Reply-To: thomas@koch.ro
To: general@hadoop.apache.org
Subject: German speaking company to run a hadoop cluster with crawlers and search
Date: Wed, 21 Apr 2010 09:10:48 +0200
User-Agent: KMail/1.12.4 (Linux/2.6.32-4-amd64; KDE/4.3.4; x86_64; ; )
MIME-Version: 1.0
Content-Type: text/plain;
  charset="us-ascii"
Content-Transfer-Encoding: 7bit
Message-Id: <201004210910.49092.thomas@koch.ro>
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

I'm searching for a german speaking company in central Europe, which already 
runs hadoop clusters, is experienced in information retrieval and crawling.
I'd like to know, if there would be interest to take over the task of running 
the backend for an already running and selling media monitoring company in 
Switzerland.
This is ATM a private request from me, the sole and only developer who is in 
charge to keep this thing running 24/7 and to implement new features in 
parallel...

Best regards,

Thomas Koch, http://www.koch.ro

From general-return-1428-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 11:09:36 2010
Return-Path: <general-return-1428-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83786 invoked from network); 21 Apr 2010 11:09:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 11:09:36 -0000
Received: (qmail 65467 invoked by uid 500); 21 Apr 2010 11:09:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65096 invoked by uid 500); 21 Apr 2010 11:09:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65088 invoked by uid 99); 21 Apr 2010 11:09:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 11:09:30 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of magawake@gmail.com designates 209.85.211.174 as permitted sender)
Received: from [209.85.211.174] (HELO mail-yw0-f174.google.com) (209.85.211.174)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 11:09:23 +0000
Received: by ywh4 with SMTP id 4so3429470ywh.5
        for <general@hadoop.apache.org>; Wed, 21 Apr 2010 04:09:02 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:received:message-id
         :subject:from:to:content-type;
        bh=hqQjz0lJB8j1qAqjPEyKtb5zPByzf8OhkmYgejRY5vM=;
        b=vTALgj1Nsg1RRbszSfURyIWzHGpbfFD0A/kZuLSqajvPWNqxxmkTljeh1aHflJDeKv
         KgXw95Ywj+pVdw87V6XA+f+DtI8ho3PzZm9lWkrBxBO7R43obZ1IcYgQwUWRfTEaaELP
         BZrBA9PznMgN3BHUEfry/JsomsK7ikZ/p4LXU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=dRiWXIMlgEQIkFPjC68lTFi8tnNa+QVecbHX7qM8exo7qxxRLxeOelM5NCNhnRIEIX
         xuWYm/wl8onxS+JJdOZiZs/YWuYFVs8ZtfoYE+wQJyNvx6NLfHh7h5ZD78QEIUg5srsD
         ae8RrvvHWT8ZnJyxMIA69+tZ+YnLxEpPfabpU=
MIME-Version: 1.0
Received: by 10.231.4.21 with HTTP; Wed, 21 Apr 2010 04:09:01 -0700 (PDT)
Date: Wed, 21 Apr 2010 07:09:01 -0400
Received: by 10.101.175.37 with SMTP id c37mr19395865anp.56.1271848141754; 
	Wed, 21 Apr 2010 04:09:01 -0700 (PDT)
Message-ID: <l2k1cbd6f831004210409red34ff40pbe3d42771aa645a8@mail.gmail.com>
Subject: dfs.data.dir
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
X-Virus-Checked: Checked by ClamAV on apache.org

I would like to add/remove more data directories to my hdfs installation.

Currently, what I do is decommision the entire node and then remove
all content from dfs.data.dir and renable the node. But is there an
easier way? Each of my node consists of 2TB of data and I don't want
to waste the time...

From general-return-1429-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 17:42:33 2010
Return-Path: <general-return-1429-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44652 invoked from network); 21 Apr 2010 17:42:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 17:42:33 -0000
Received: (qmail 55122 invoked by uid 500); 21 Apr 2010 17:42:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55004 invoked by uid 500); 21 Apr 2010 17:42:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54996 invoked by uid 99); 21 Apr 2010 17:42:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 17:42:31 +0000
X-ASF-Spam-Status: No, hits=-76.8 required=10.0
	tests=AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 21 Apr 2010 17:42:30 +0000
Received: (qmail 44382 invoked by uid 99); 21 Apr 2010 17:42:10 -0000
Received: from localhost.apache.org (HELO [192.168.168.107]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 17:42:10 +0000
Message-ID: <4BCF38F1.4090103@apache.org>
Date: Wed, 21 Apr 2010 10:42:09 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100411)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: how to split data
References: <1E766DC73403B144A7FF3C3A3888D0C7143229@BLR-EC-MBX02.wipro.com>
In-Reply-To: <1E766DC73403B144A7FF3C3A3888D0C7143229@BLR-EC-MBX02.wipro.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

lee.sungeun@wipro.com wrote:
> if the block size is 64MB and i am supposed to hadle 129MB. How will it split? (64,64,1) or (64,64,64)

(64,65)

FileInputFormat permits the final split to exceed the desired split size 
by up to 10%.

Doug

From general-return-1430-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 19:34:12 2010
Return-Path: <general-return-1430-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29676 invoked from network); 21 Apr 2010 19:34:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 19:34:11 -0000
Received: (qmail 85655 invoked by uid 500); 21 Apr 2010 19:34:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85595 invoked by uid 500); 21 Apr 2010 19:34:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85587 invoked by uid 99); 21 Apr 2010 19:34:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 19:34:09 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,MIME_BASE64_TEXT,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of msegel@navteq.com designates 204.120.70.37 as permitted sender)
Received: from [204.120.70.37] (HELO xmailfargo.navteq.com) (204.120.70.37)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 19:34:02 +0000
Received: from fargo-spamfw.navteq.com (navteq-spamfw.navteq.com [10.6.10.131])
	by xmailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id o3LJGUkq009085
	for <general@hadoop.apache.org>; Wed, 21 Apr 2010 14:16:30 -0500
Received: from imailfargo.navteq.com (localhost [127.0.0.1])
	by fargo-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id 6DE8047F289F
	for <general@hadoop.apache.org>; Wed, 21 Apr 2010 14:33:40 -0500 (CDT)
Received: from imailfargo.navteq.com (imailfargo.navteq.com [10.6.10.130]) by fargo-spamfw.navteq.com with ESMTP id EAIbFRbIghRXaT1o for <general@hadoop.apache.org>; Wed, 21 Apr 2010 14:33:40 -0500 (CDT)
Received: from hq-ex-ht01.ad.navteq.com (hq-ex-ht01.ad.navteq.com [10.8.222.51])
	by imailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id o3LJXY8Q017980
	for <general@hadoop.apache.org>; Wed, 21 Apr 2010 14:33:35 -0500
Received: from hq-ex-mb02.ad.navteq.com ([fe80::880d:318e:e860:a8db]) by
 hq-ex-ht01.ad.navteq.com ([fe80::55a6:5b94:5c00:fe39%13]) with mapi; Wed, 21
 Apr 2010 14:33:34 -0500
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 21 Apr 2010 14:33:33 -0500
Subject: [Announce] Local Chicago area Hadoop User Group (CHUG)
Thread-Topic: [Announce] Local Chicago area Hadoop User Group (CHUG)
Thread-Index: AcrhiYfGWxS0Lma4TESRHE88DO+iPg==
Message-ID: <ABC24175AFD3BE4DA15F4CD375ED413D0607D37F67@hq-ex-mb02.ad.navteq.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_ABC24175AFD3BE4DA15F4CD375ED413D0607D37F67hqexmb02adnav_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_ABC24175AFD3BE4DA15F4CD375ED413D0607D37F67hqexmb02adnav_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1

All,


I've  set up a Meetup site for the Chicago area Hadoop User Group.

http://www.meetup.com/Chicago-area-Hadoop-User-Group-CHUG/

With the majority of those interested working downtown, I think our first=
 meeting will be downtown.
Haven't yet set up a date, or location.... But this is just the first ste=
p.

Please feel free to spread the word.

Thx

-Mike


Michael Segel
Architect,  R&D
NAVTEQ
425 West Randolph Street
Chicago, IL 60606
(T)  +1 312-780-3432
(C)  +1 312-952-8175
www.navteq.com<http://www.navteq.com/>



The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.

--_000_ABC24175AFD3BE4DA15F4CD375ED413D0607D37F67hqexmb02adnav_--

From general-return-1431-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 21:34:28 2010
Return-Path: <general-return-1431-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1798 invoked from network); 21 Apr 2010 21:34:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 21:34:28 -0000
Received: (qmail 58759 invoked by uid 500); 21 Apr 2010 21:34:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58532 invoked by uid 500); 21 Apr 2010 21:34:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58524 invoked by uid 99); 21 Apr 2010 21:34:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 21:34:26 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 21:34:18 +0000
Received: by pwi7 with SMTP id 7so5461510pwi.35
        for <general@hadoop.apache.org>; Wed, 21 Apr 2010 14:33:57 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.142.100.13 with HTTP; Wed, 21 Apr 2010 14:33:56 -0700 (PDT)
In-Reply-To: <l2k1cbd6f831004210409red34ff40pbe3d42771aa645a8@mail.gmail.com>
References: <l2k1cbd6f831004210409red34ff40pbe3d42771aa645a8@mail.gmail.com>
Date: Wed, 21 Apr 2010 14:33:56 -0700
Received: by 10.142.207.18 with SMTP id e18mr4094591wfg.158.1271885637033; 
	Wed, 21 Apr 2010 14:33:57 -0700 (PDT)
Message-ID: <m2pdfe484f01004211433l61ce4be6rdfe94ebc3bc24064@mail.gmail.com>
Subject: Re: dfs.data.dir
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hey Mag,

You can bring down the datanode daemon, add the extra dfs.data.dir and
then restart. Since blocks are round robin'd the new directory will
have lower utilization (one other directories are full it will start
catching up). If that's not OK you can re-balance the directories by
hand with cp when the datanode is down (before you restart it).  If
this takes you longer than 10 minutes the blocks on that datanode will
start getting re-replicated but when you bring the datanode back up
the namenode will notice the over-replicated blocks and remove them.

Thanks,
Eli

On Wed, Apr 21, 2010 at 4:09 AM, Mag Gam <magawake@gmail.com> wrote:
> I would like to add/remove more data directories to my hdfs installation.
>
> Currently, what I do is decommision the entire node and then remove
> all content from dfs.data.dir and renable the node. But is there an
> easier way? Each of my node consists of 2TB of data and I don't want
> to waste the time...
>

From general-return-1432-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 12:42:34 2010
Return-Path: <general-return-1432-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67617 invoked from network); 22 Apr 2010 12:42:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 12:42:34 -0000
Received: (qmail 91010 invoked by uid 500); 22 Apr 2010 12:42:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90897 invoked by uid 500); 22 Apr 2010 12:42:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90889 invoked by uid 99); 22 Apr 2010 12:42:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 12:42:32 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 12:42:22 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id EE835B7E29
	for <general@hadoop.apache.org>; Thu, 22 Apr 2010 13:42:01 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id eIZowgjnjdKk for <general@hadoop.apache.org>;
	Thu, 22 Apr 2010 13:41:50 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 8985EB7E25
	for <general@hadoop.apache.org>; Thu, 22 Apr 2010 13:41:49 +0100 (BST)
MailScanner-NULL-Check: 1272544898.40915@AoG9MTUraCorLtUoLXV+9A
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o3MCfZwa025684
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Thu, 22 Apr 2010 13:41:36 +0100 (BST)
Message-ID: <4BD04400.90800@apache.org>
Date: Thu, 22 Apr 2010 13:41:36 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100228)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: dfs.data.dir
References: <l2k1cbd6f831004210409red34ff40pbe3d42771aa645a8@mail.gmail.com> <m2pdfe484f01004211433l61ce4be6rdfe94ebc3bc24064@mail.gmail.com>
In-Reply-To: <m2pdfe484f01004211433l61ce4be6rdfe94ebc3bc24064@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o3MCfZwa025684
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

Eli Collins wrote:
> Hey Mag,
> 
> You can bring down the datanode daemon, add the extra dfs.data.dir and
> then restart. Since blocks are round robin'd the new directory will
> have lower utilization (one other directories are full it will start
> catching up). If that's not OK you can re-balance the directories by
> hand with cp when the datanode is down (before you restart it).  If
> this takes you longer than 10 minutes the blocks on that datanode will
> start getting re-replicated but when you bring the datanode back up
> the namenode will notice the over-replicated blocks and remove them.

that brings up a couple of issues I've been thinking about now that 
workers can go to 6+ HDDs/node

* a way to measure the distribution across disks, rather than just 
nodes. DfsClient doesn't provide enough info here yet.
* a way to triger some rebalancing on a single node, to say "position 
stuff more fairly". You don't need to worry about network traffic, just 
local disk load and CPU time, so it should be simpler.

From general-return-1433-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 17:40:18 2010
Return-Path: <general-return-1433-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77755 invoked from network); 22 Apr 2010 17:40:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 17:40:18 -0000
Received: (qmail 9679 invoked by uid 500); 22 Apr 2010 17:40:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 9584 invoked by uid 500); 22 Apr 2010 17:40:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 9575 invoked by uid 99); 22 Apr 2010 17:40:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 17:40:16 +0000
X-ASF-Spam-Status: No, hits=-0.5 required=10.0
	tests=AWL,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [68.142.207.182] (HELO web81903.mail.mud.yahoo.com) (68.142.207.182)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 22 Apr 2010 17:40:08 +0000
Received: (qmail 34529 invoked by uid 60001); 22 Apr 2010 17:39:47 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1271957987; bh=9/fuQLALgTK2J0uLp4uNkkl9LHld2j8bK+yjwms3vVg=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=il4F4NK87S2WPi4345jw2AnphPuo4gYe2IVEquo/mkVMFKRUhS3NX8ZeenO1niAePY6f6svtX1FH7g0hx5Yr2RPxwJumtZ1jAfzgbamEXwOE8OReGptt5PPqQi32Aabl2+K9E1vb448Gg/OEMdpQfCtagy+Qo2pqIj/SQGD14As=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=UtwhtDoRbEc1EFSVboZBXJH/lX/4ThOgBEegH15lLebw60xSqTpZBOTNriK2ui7sEd85g+hWf/4fOhtmVNpb3GkvWd9U5s9YGZ/FjCCd/B1fPwjABcmN0hhFO6ghHiNZTo2g0sI97RP+aq8EacuVC5Cd5gMMszOSKYyCWqJzpDE=;
Message-ID: <225.32177.qm@web81903.mail.mud.yahoo.com>
X-YMail-OSG: 0BKIJKYVM1kr1qxNdncqj8DJWyrMP3i_0IsGpvPZViw149T
 MKBCSjenqjkS09a8j.i4mtgAb_EmQTt8aCulr7qkBpU2zrO5rS.a5IO3.bQ_
 P.KS0Pn2tPtGLpodDjf6XsfYT6Ap6tOpXBwMVUzJlbllTr_P8Na81l2j0J11
 PwGcFTQr8BRPMBSvbgYAXFOPyEF5GD9P.d.vvJNt2IA7unZjsSpdjLtwkqQJ
 jsVJIY8jwSvB7QSxp.U.0SkMMwtrvUW2Urp4G9uzhNOS32Qjkl3OrvZ7dTUr
 wBEBmWW6JEZA9.OERRhk5CDe0T_kdHBQoviTm
Received: from [64.140.94.70] by web81903.mail.mud.yahoo.com via HTTP; Thu, 22 Apr 2010 10:39:46 PDT
X-Mailer: YahooMailClassic/10.1.9 YahooMailWebService/0.8.102.267879
Date: Thu, 22 Apr 2010 10:39:46 -0700 (PDT)
From: Rana Aich <ranaaich@yahoo.com>
Subject: Where is the 'user' directory
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii

Hi,

I'm a newbie in this sphere...I've downloaded Hadoop yesterday and am going thru the commands listed in the Quick Start page.

I went to http://localhost:50070 the DFS home page.
>From there I clicked 'Browse the File System' link.

There I could see two directories:
/tmp
/user

and all there subdirectories and files.

But when I go to my File sytem I'm unable to find 'user' directory. Though the DFS home page lists it under "/" (root directory) - I couldn't find any directory of that name. Interestingly /tmp is there.

I ran 'find / -name' as Superuser for "user" directory but there is none.

Am I missing something here?

Thanking you all in advance.
-raich

From general-return-1434-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 17:46:17 2010
Return-Path: <general-return-1434-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81295 invoked from network); 22 Apr 2010 17:46:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 17:46:17 -0000
Received: (qmail 21324 invoked by uid 500); 22 Apr 2010 17:46:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21278 invoked by uid 500); 22 Apr 2010 17:46:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21270 invoked by uid 99); 22 Apr 2010 17:46:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 17:46:16 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 17:46:09 +0000
Received: by wyb40 with SMTP id 40so190633wyb.35
        for <general@hadoop.apache.org>; Thu, 22 Apr 2010 10:45:49 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.164.203 with HTTP; Thu, 22 Apr 2010 10:45:49 -0700 (PDT)
In-Reply-To: <225.32177.qm@web81903.mail.mud.yahoo.com>
References: <225.32177.qm@web81903.mail.mud.yahoo.com>
Date: Thu, 22 Apr 2010 13:45:49 -0400
Received: by 10.216.89.81 with SMTP id b59mr558346wef.97.1271958349508; Thu, 
	22 Apr 2010 10:45:49 -0700 (PDT)
Message-ID: <i2q23d54121004221045h5c28793ds5f896947f520590a@mail.gmail.com>
Subject: Re: Where is the 'user' directory
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

HDFS is a user space file system meaning it is a virtual file system,
different from the OS file systems (like ext3, xfs). To access files
on HDFS you use the 'hadoop fs' command. Ex:

hadoop fs -ls /

This is different than your OS / partition and will contain the same
view as the web UI.

On Thu, Apr 22, 2010 at 1:39 PM, Rana Aich <ranaaich@yahoo.com> wrote:
> Hi,
>
> I'm a newbie in this sphere...I've downloaded Hadoop yesterday and am going thru the commands listed in the Quick Start page.
>
> I went to http://localhost:50070 the DFS home page.
> From there I clicked 'Browse the File System' link.
>
> There I could see two directories:
> /tmp
> /user
>
> and all there subdirectories and files.
>
> But when I go to my File sytem I'm unable to find 'user' directory. Though the DFS home page lists it under "/" (root directory) - I couldn't find any directory of that name. Interestingly /tmp is there.
>
> I ran 'find / -name' as Superuser for "user" directory but there is none.
>
> Am I missing something here?
>
> Thanking you all in advance.
> -raich
>



-- 
Eric Sammer
phone: +1-917-287-2675
twitter: esammer
data: www.cloudera.com

From general-return-1435-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 19:59:48 2010
Return-Path: <general-return-1435-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 61956 invoked from network); 22 Apr 2010 19:59:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 19:59:48 -0000
Received: (qmail 58455 invoked by uid 500); 22 Apr 2010 19:59:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58420 invoked by uid 500); 22 Apr 2010 19:59:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58411 invoked by uid 99); 22 Apr 2010 19:59:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 19:59:45 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=AWL,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 19:59:40 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:Received:Received:From:To:Subject:
   Thread-Topic:Thread-Index:Date:Message-ID:References:
   In-Reply-To:Accept-Language:Content-Language:
   X-MS-Has-Attach:X-MS-TNEF-Correlator:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version:
   Return-Path:X-OriginalArrivalTime;
  b=L0NGGDS0BmVQhMichuZuzpz0T5MQCmKSavEhp9D/vx7JPQGpwGNtbGvh
   DDXN+eFIgWiNVOW/QnLLuJVnjZjK+BDW9JTPCDFqbCEi7AXmc8FH2l/t+
   eIWzm7lp98upEkk;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1271966380; x=1303502380;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20dfs.data.dir|Date:=20Thu,=2022=20Apr=20
   2010=2019:59:18=20+0000|Message-ID:=20<51FEB071-B163-4B01
   -A08A-EB4DD9645923@linkedin.com>|To:=20"<general@hadoop.a
   pache.org>"=20<general@hadoop.apache.org>|MIME-Version:
   =201.0|Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<25c6fce2-8d3b-4ac2-8c2d-a17dca0af983>
   |In-Reply-To:=20<4BD04400.90800@apache.org>|References:
   =20<l2k1cbd6f831004210409red34ff40pbe3d42771aa645a8@mail.
   gmail.com>=0D=0A=20<m2pdfe484f01004211433l61ce4be6rdfe94e
   bc3bc24064@mail.gmail.com>=0D=0A=20<4BD04400.90800@apache
   .org>;
  bh=QKuYumno2DbWzE7An23Rd54R9ez8NI4Ylnnw27EwYW4=;
  b=WzpAmdES+gmoUY34Hi/V58F8OsrurDIinwgWgFJ6W4klN2K0obrnzPJW
   TnzY+33pNa22NC58KmEEMfJrO6740xP7ZdpPSFrJoI+WxdlHLjbFnH+mr
   8V4gb28DsRUKPql;
X-IronPort-AV: E=Sophos;i="4.52,258,1270450800"; 
   d="scan'208";a="12124020"
Received: from esv4-exctest.linkedin.biz ([172.18.46.60]) by CORP-MAIL.linkedin.biz with Microsoft SMTPSVC(6.0.3790.3959);
	 Thu, 22 Apr 2010 12:59:20 -0700
Received: from ESV4-CAS01.linkedin.biz (172.18.46.140) by
 esv4-exctest.linkedin.biz (172.18.46.60) with Microsoft SMTP Server (TLS) id
 14.0.682.1; Thu, 22 Apr 2010 12:59:20 -0700
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi; Thu, 22 Apr 2010
 12:59:19 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: dfs.data.dir
Thread-Topic: dfs.data.dir
Thread-Index: AQHK4UMhXQAV3PvzLEKrqdMha3hZoZIt54MAgAD9mQCAAHpLAA==
Date: Thu, 22 Apr 2010 19:59:18 +0000
Message-ID: <51FEB071-B163-4B01-A08A-EB4DD9645923@linkedin.com>
References: <l2k1cbd6f831004210409red34ff40pbe3d42771aa645a8@mail.gmail.com>
 <m2pdfe484f01004211433l61ce4be6rdfe94ebc3bc24064@mail.gmail.com>
 <4BD04400.90800@apache.org>
In-Reply-To: <4BD04400.90800@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Content-Type: text/plain; charset="us-ascii"
Content-ID: <25c6fce2-8d3b-4ac2-8c2d-a17dca0af983>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginalArrivalTime: 22 Apr 2010 19:59:20.0905 (UTC) FILETIME=[4C52F390:01CAE256]


On Apr 22, 2010, at 5:41 AM, Steve Loughran wrote:
> that brings up a couple of issues I've been thinking about now that worke=
rs can go to 6+ HDDs/node
>=20
> * a way to measure the distribution across disks, rather than just nodes.=
 DfsClient doesn't provide enough info here yet.

What should probably happen is that instead of throwing you to the file bro=
wser, clicking on a host from the live nodes page should probably put you o=
n a "stats about this node" page.

> * a way to triger some rebalancing on a single node, to say "position stu=
ff more fairly". You don't need to worry about network traffic, just local =
disk load and CPU time, so it should be simpler.


Yup.  Working with 8 drives per node, it is interesting to see how unbalanc=
ed the data gets after a while.  [Luckily, we have MR tmp space segregated =
off so I'm sure it would be a lot worse if we didn't!]

Someone should file a jira. :)


From general-return-1436-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 21:37:16 2010
Return-Path: <general-return-1436-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16398 invoked from network); 22 Apr 2010 21:37:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 21:37:15 -0000
Received: (qmail 12416 invoked by uid 500); 22 Apr 2010 21:37:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12368 invoked by uid 500); 22 Apr 2010 21:37:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12360 invoked by uid 99); 22 Apr 2010 21:37:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 21:37:14 +0000
X-ASF-Spam-Status: No, hits=-0.1 required=10.0
	tests=AWL,FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of owen.omalley@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 21:37:07 +0000
Received: by wwb29 with SMTP id 29so5415841wwb.35
        for <general@hadoop.apache.org>; Thu, 22 Apr 2010 14:36:45 -0700 (PDT)
Received: by 10.216.154.7 with SMTP id g7mr674497wek.7.1271972205801;
        Thu, 22 Apr 2010 14:36:45 -0700 (PDT)
Received: from wlan-snve-152-231.corp.yahoo.com (nat-dip6.cfw-a-gci.corp.yahoo.com [209.131.62.115])
        by mx.google.com with ESMTPS id e82sm233314wej.16.2010.04.22.14.36.43
        (version=SSLv3 cipher=RC4-MD5);
        Thu, 22 Apr 2010 14:36:44 -0700 (PDT)
Message-Id: <C3B2314E-7FC9-4301-AFE2-AFECEA6333B6@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: [VOTE] Should we approve the Hadoop security logo?
Date: Thu, 22 Apr 2010 14:36:41 -0700
X-Mailer: Apple Mail (2.936)

All,
   Should we allow use of the Hadoop security logo (hadoop-security- 
logo.jpg) from HADOOP-6721. Clearly, I'm +1.

-- Owen

From general-return-1437-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 21:44:54 2010
Return-Path: <general-return-1437-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 20125 invoked from network); 22 Apr 2010 21:44:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 21:44:54 -0000
Received: (qmail 24547 invoked by uid 500); 22 Apr 2010 21:44:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24489 invoked by uid 500); 22 Apr 2010 21:44:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24480 invoked by uid 99); 22 Apr 2010 21:44:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 21:44:53 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 22 Apr 2010 21:44:50 +0000
Received: (qmail 19963 invoked by uid 99); 22 Apr 2010 21:44:28 -0000
Received: from localhost.apache.org (HELO [192.168.168.106]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 21:44:28 +0000
Message-ID: <4BD0C33C.6070400@apache.org>
Date: Thu, 22 Apr 2010 14:44:28 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100411)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Should we approve the Hadoop security logo?
References: <C3B2314E-7FC9-4301-AFE2-AFECEA6333B6@apache.org>
In-Reply-To: <C3B2314E-7FC9-4301-AFE2-AFECEA6333B6@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Owen O'Malley wrote:
>   Should we allow use of the Hadoop security logo 
> (hadoop-security-logo.jpg) from HADOOP-6721. Clearly, I'm +1.

Allow use where?  What's the intended purpose for this logo?

As Hadoop releases add security features, won't the existing logo suffice?

Doug

From general-return-1438-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 22:18:10 2010
Return-Path: <general-return-1438-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37137 invoked from network); 22 Apr 2010 22:18:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 22:18:09 -0000
Received: (qmail 56232 invoked by uid 500); 22 Apr 2010 22:18:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56082 invoked by uid 500); 22 Apr 2010 22:18:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56074 invoked by uid 99); 22 Apr 2010 22:18:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 22:18:08 +0000
X-ASF-Spam-Status: No, hits=0.2 required=10.0
	tests=AWL,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.62.56] (HELO qmta06.westchester.pa.mail.comcast.net) (76.96.62.56)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 22:18:00 +0000
Received: from omta10.westchester.pa.mail.comcast.net ([76.96.62.28])
	by qmta06.westchester.pa.mail.comcast.net with comcast
	id 8a8B1e0090cZkys56mHgxx; Thu, 22 Apr 2010 22:17:40 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta10.westchester.pa.mail.comcast.net with comcast
	id 8mHX1e0032SbwD53WmHaDM; Thu, 22 Apr 2010 22:17:38 +0000
Message-Id: <D964E5AC-96CA-4D7C-9991-1587C57E8FE7@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <4BD0C33C.6070400@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Should we approve the Hadoop security logo?
Date: Thu, 22 Apr 2010 15:17:28 -0700
References: <C3B2314E-7FC9-4301-AFE2-AFECEA6333B6@apache.org> <4BD0C33C.6070400@apache.org>
X-Mailer: Apple Mail (2.936)


On Apr 22, 2010, at 2:44 PM, Doug Cutting wrote:

> Owen O'Malley wrote:
>>  Should we allow use of the Hadoop security logo (hadoop-security- 
>> logo.jpg) from HADOOP-6721. Clearly, I'm +1.
>
> Allow use where?  What's the intended purpose for this logo?

The short term desire is to make t-shirts with a Hadoop security logo.  
It will also be used in documentation and literature relating to the  
Hadoop security features.

-- Owen

From general-return-1439-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 02:36:07 2010
Return-Path: <general-return-1439-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 15807 invoked from network); 23 Apr 2010 02:36:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 02:36:07 -0000
Received: (qmail 61326 invoked by uid 500); 23 Apr 2010 02:36:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61135 invoked by uid 500); 23 Apr 2010 02:36:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61127 invoked by uid 99); 23 Apr 2010 02:36:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 02:36:05 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.212.176] (HELO mail-px0-f176.google.com) (209.85.212.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 02:35:59 +0000
Received: by pxi10 with SMTP id 10so900886pxi.35
        for <general@hadoop.apache.org>; Thu, 22 Apr 2010 19:35:38 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.142.100.13 with HTTP; Thu, 22 Apr 2010 19:35:38 -0700 (PDT)
In-Reply-To: <51FEB071-B163-4B01-A08A-EB4DD9645923@linkedin.com>
References: <l2k1cbd6f831004210409red34ff40pbe3d42771aa645a8@mail.gmail.com>
	 <m2pdfe484f01004211433l61ce4be6rdfe94ebc3bc24064@mail.gmail.com>
	 <4BD04400.90800@apache.org>
	 <51FEB071-B163-4B01-A08A-EB4DD9645923@linkedin.com>
Date: Thu, 22 Apr 2010 19:35:38 -0700
Received: by 10.143.85.4 with SMTP id n4mr2452238wfl.198.1271990138091; Thu, 
	22 Apr 2010 19:35:38 -0700 (PDT)
Message-ID: <m2xdfe484f01004221935w5fcebdb2r25deee2dcff3f4c0@mail.gmail.com>
Subject: Re: dfs.data.dir
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Thu, Apr 22, 2010 at 12:59 PM, Allen Wittenauer
<awittenauer@linkedin.com> wrote:
>
>> * a way to triger some rebalancing on a single node, to say "position st=
uff more fairly". You don't need to worry about network traffic, just local=
 disk load and CPU time, so it should be simpler.
>
>
> Yup. =A0Working with 8 drives per node, it is interesting to see how unba=
lanced the data gets after a while. =A0[Luckily, we have MR tmp space segre=
gated off so I'm sure it would be a lot worse if we didn't!]

How unbalanced do you see your mounts?  I've mostly heard of this
being an issue when new disks are added.

Thanks,
Eli

From general-return-1440-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 04:18:45 2010
Return-Path: <general-return-1440-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34794 invoked from network); 23 Apr 2010 04:18:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 04:18:45 -0000
Received: (qmail 29034 invoked by uid 500); 23 Apr 2010 04:18:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28866 invoked by uid 500); 23 Apr 2010 04:18:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28858 invoked by uid 99); 23 Apr 2010 04:18:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 04:18:43 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of gautam.singaraju@gmail.com designates 209.85.217.225 as permitted sender)
Received: from [209.85.217.225] (HELO mail-gx0-f225.google.com) (209.85.217.225)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 04:18:38 +0000
Received: by gxk25 with SMTP id 25so5258739gxk.11
        for <general@hadoop.apache.org>; Thu, 22 Apr 2010 21:18:17 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:received:message-id
         :subject:from:to:content-type;
        bh=mmszG50e7AjhhxCU4QLMM2CIDWCfG9m1Teu0CskvK40=;
        b=MwbUOBPviARVyYbU1J6JsFGG6tFC8qjo76xyztO6kWq+6BPgGeDDwAXmkzUKqbIodc
         yBd0cYx5mghBiXp5kZTxCI2aA+tsb7cBpowYGfZEPWG2ujkN90WxiU5IMLyh/xXE1gV3
         q7Heoom5aebA41MSr51jU0JeX01xO0NrbGMy8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=TL8ITrDbotY86b5TRDB9Qm6IQdVt2AsqxgwObS2foD5SUusRbylyZEHwtrOhfiIej3
         lwrQ1z/gFzNJ7814f0mvSnIjJ0dIN3hlwUCVQISFE1Yd5YqTbHxFMUS456wjc+Ihcw1q
         ZOVPq0LH5YdQrb4HQiPkDCLkuwsvh6Sn6j24o=
MIME-Version: 1.0
Received: by 10.150.230.19 with HTTP; Thu, 22 Apr 2010 21:18:16 -0700 (PDT)
Date: Fri, 23 Apr 2010 00:18:16 -0400
Received: by 10.150.120.22 with SMTP id s22mr11067040ybc.199.1271996296771; 
	Thu, 22 Apr 2010 21:18:16 -0700 (PDT)
Message-ID: <g2jf8b2ccd41004222118kb9c46c54oa7bc835bc0b5fcfd@mail.gmail.com>
Subject: DBOutputFormat over SSH?
From: Gautam Singaraju <gautam.singaraju@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

All,

I have a use-case where I need to crunch a large amount of data and
push to the results (comparatively a smaller set) to a mysql db at a
remote location. As per security concerns, only SSH ports are open. I
tried using Java Secure Channel [1] in combination with some custom
JDBC code from the reducers.

Can anyone comment on the performance of DBOutputFormat? Have there
been any efforts to tunnel this through SSH? This is going to be an
expensive operation; any suggestions would be welcome.

[1] http://www.jcraft.com/jsch/
---
Gautam Singaraju

From general-return-1441-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 04:28:14 2010
Return-Path: <general-return-1441-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37007 invoked from network); 23 Apr 2010 04:28:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 04:28:14 -0000
Received: (qmail 33040 invoked by uid 500); 23 Apr 2010 04:28:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32880 invoked by uid 500); 23 Apr 2010 04:28:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32872 invoked by uid 99); 23 Apr 2010 04:28:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 04:28:12 +0000
X-ASF-Spam-Status: No, hits=2.8 required=10.0
	tests=AWL,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of sonalgoyal4@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 04:28:07 +0000
Received: by pwi7 with SMTP id 7so6638286pwi.35
        for <general@hadoop.apache.org>; Thu, 22 Apr 2010 21:27:47 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type;
        bh=JnTY6gJa2ztDNePvOahNjLH/RvG9raTK2HQj/cCKk+4=;
        b=VKjOs074/RfQhGSlLX+vfwXiZd5lLK1TMKp46ALMvUgUjjSteakJ0+9yIDMIS8N7X7
         f1izZFoCVCfl6ZZRHQItNSim8xD1ouEXSO3jJbyz/f8tB9WObhXJT8lblrBS9ahBQzgE
         FMgEoDlkvK9mgcMgkNfjmX65jcXKSpwtbCabg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=ZrHa3d54Xx77NFMxTVbyKWnj97iMXlfmxab9NLbafQ3XShL11n6w13o7tc3ojiGbyx
         L5e+37vg+CMh+F9h84u7mppwJgrh0LbqbRLNhfqm4ELCeoouYdV9uZ13F6uIcmEUR6Yz
         5DNgRaZyWTXEvlj9x8iqsKU4dsIQf593HzquU=
MIME-Version: 1.0
Received: by 10.142.211.14 with HTTP; Thu, 22 Apr 2010 21:27:44 -0700 (PDT)
In-Reply-To: <g2jf8b2ccd41004222118kb9c46c54oa7bc835bc0b5fcfd@mail.gmail.com>
References: <g2jf8b2ccd41004222118kb9c46c54oa7bc835bc0b5fcfd@mail.gmail.com>
Date: Fri, 23 Apr 2010 09:57:44 +0530
Received: by 10.143.136.7 with SMTP id o7mr5069033wfn.101.1271996864385; Thu, 
	22 Apr 2010 21:27:44 -0700 (PDT)
Message-ID: <u2i9f26e661004222127iea301d45z3494e059497dbdb3@mail.gmail.com>
Subject: Re: DBOutputFormat over SSH?
From: Sonal Goyal <sonalgoyal4@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd3fcec41d41a0484dfd9c3

--000e0cd3fcec41d41a0484dfd9c3
Content-Type: text/plain; charset=ISO-8859-1

Hi Gautam,

DBOutputFormat inserts records one by one, which is inherently slow. You can
use open source Apache licensed hiho framework which provides MySQL's "load
data infile " functionality.  It may be more suited to your needs.

HIHO is available at http://code.google.com/p/hiho/

I havent tested it over ssh, please let me know if you need any help setting
it up.

Thanks and Regards,
Sonal
www.meghsoft.com


On Fri, Apr 23, 2010 at 9:48 AM, Gautam Singaraju <
gautam.singaraju@gmail.com> wrote:

> All,
>
> I have a use-case where I need to crunch a large amount of data and
> push to the results (comparatively a smaller set) to a mysql db at a
> remote location. As per security concerns, only SSH ports are open. I
> tried using Java Secure Channel [1] in combination with some custom
> JDBC code from the reducers.
>
> Can anyone comment on the performance of DBOutputFormat? Have there
> been any efforts to tunnel this through SSH? This is going to be an
> expensive operation; any suggestions would be welcome.
>
> [1] http://www.jcraft.com/jsch/
> ---
> Gautam Singaraju
>

--000e0cd3fcec41d41a0484dfd9c3--

From general-return-1442-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 04:29:31 2010
Return-Path: <general-return-1442-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37159 invoked from network); 23 Apr 2010 04:29:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 04:29:30 -0000
Received: (qmail 34155 invoked by uid 500); 23 Apr 2010 04:29:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34052 invoked by uid 500); 23 Apr 2010 04:29:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34044 invoked by uid 99); 23 Apr 2010 04:29:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 04:29:29 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 04:29:23 +0000
Received: by wyb40 with SMTP id 40so477084wyb.35
        for <general@hadoop.apache.org>; Thu, 22 Apr 2010 21:29:02 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.164.203 with HTTP; Thu, 22 Apr 2010 21:29:00 -0700 (PDT)
In-Reply-To: <g2jf8b2ccd41004222118kb9c46c54oa7bc835bc0b5fcfd@mail.gmail.com>
References: <g2jf8b2ccd41004222118kb9c46c54oa7bc835bc0b5fcfd@mail.gmail.com>
Date: Fri, 23 Apr 2010 00:29:00 -0400
Received: by 10.216.88.4 with SMTP id z4mr1474277wee.121.1271996940383; Thu, 
	22 Apr 2010 21:29:00 -0700 (PDT)
Message-ID: <r2z23d54121004222129gcc126a5egc00ff7fbb3d0c7bf@mail.gmail.com>
Subject: Re: DBOutputFormat over SSH?
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

In general, you'll want to avoid tunneling permanent production code
over ssh tunnels. They're flaky and do not recover from network
interruption in any reasonable way. If you need to do this, a vpn is
the correct approach. Linux easily will do ipsec p2p tunnels that are
reasonably secure. If you really only have port 22 then I suppose
that's your only option but I really would reevaluate the security
policy.

Either way, it's going to be slow due to the encryption overhead but
if it's a small amount of data, that may be fine.

On Fri, Apr 23, 2010 at 12:18 AM, Gautam Singaraju
<gautam.singaraju@gmail.com> wrote:
> All,
>
> I have a use-case where I need to crunch a large amount of data and
> push to the results (comparatively a smaller set) to a mysql db at a
> remote location. As per security concerns, only SSH ports are open. I
> tried using Java Secure Channel [1] in combination with some custom
> JDBC code from the reducers.
>
> Can anyone comment on the performance of DBOutputFormat? Have there
> been any efforts to tunnel this through SSH? This is going to be an
> expensive operation; any suggestions would be welcome.
>
> [1] http://www.jcraft.com/jsch/
> ---
> Gautam Singaraju
>



-- 
Eric Sammer
phone: +1-917-287-2675
twitter: esammer
data: www.cloudera.com

From general-return-1443-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 04:32:29 2010
Return-Path: <general-return-1443-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37349 invoked from network); 23 Apr 2010 04:32:29 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 04:32:29 -0000
Received: (qmail 36723 invoked by uid 500); 23 Apr 2010 04:32:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36551 invoked by uid 500); 23 Apr 2010 04:32:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36543 invoked by uid 99); 23 Apr 2010 04:32:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 04:32:27 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 04:32:19 +0000
Received: from sp1-ex07cas01.ds.corp.yahoo.com (sp1-ex07cas01.ds.corp.yahoo.com [216.252.116.137])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o3N4Vl9A012932
	for <general@hadoop.apache.org>; Thu, 22 Apr 2010 21:31:47 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:in-reply-to:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=WCjz+23aOsE80cXAIO60CNgVpkt2jJVtFnAj7HwZtMqh5I95UZu0aeEACDmRalOY
Received: from SP1-EX07VS02.ds.corp.yahoo.com ([216.252.116.135]) by
 sp1-ex07cas01.ds.corp.yahoo.com ([216.252.116.137]) with mapi; Thu, 22 Apr
 2010 21:31:47 -0700
From: Devaraj Das <ddas@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 22 Apr 2010 21:31:45 -0700
Subject: Re: [VOTE] Should we approve the Hadoop security logo?
Thread-Topic: [VOTE] Should we approve the Hadoop security logo?
Thread-Index: AcriacNZWOmzZxyfQISpU83yVZoDsAANB3hx
Message-ID: <C7F670C1.1C2D7%ddas@yahoo-inc.com>
In-Reply-To: <D964E5AC-96CA-4D7C-9991-1587C57E8FE7@apache.org>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C7F670C11C2D7ddasyahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C7F670C11C2D7ddasyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

+1 for using the logo at least in t-shirts & security related documentation=
.


On 4/22/10 3:17 PM, "Owen O'Malley" <omalley@apache.org> wrote:



On Apr 22, 2010, at 2:44 PM, Doug Cutting wrote:

> Owen O'Malley wrote:
>>  Should we allow use of the Hadoop security logo (hadoop-security-
>> logo.jpg) from HADOOP-6721. Clearly, I'm +1.
>
> Allow use where?  What's the intended purpose for this logo?

The short term desire is to make t-shirts with a Hadoop security logo.
It will also be used in documentation and literature relating to the
Hadoop security features.

-- Owen


--_000_C7F670C11C2D7ddasyahooinccom_--

From general-return-1444-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 04:36:04 2010
Return-Path: <general-return-1444-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38784 invoked from network); 23 Apr 2010 04:36:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 04:36:04 -0000
Received: (qmail 39318 invoked by uid 500); 23 Apr 2010 04:36:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39195 invoked by uid 500); 23 Apr 2010 04:36:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39187 invoked by uid 99); 23 Apr 2010 04:36:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 04:36:02 +0000
X-ASF-Spam-Status: No, hits=0.6 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 04:35:58 +0000
Received: by gyf1 with SMTP id 1so5109429gyf.35
        for <general@hadoop.apache.org>; Thu, 22 Apr 2010 21:35:36 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type;
        bh=/EUbMpIuvKravLvi71+I1JfVt19va2iIe5ZYlaqFZBc=;
        b=eMhp6lriY3A+Ro/z93mvIDAryrnW/HS+Ck7IEJqhSy6mvhGS+ps9VG2UFn+Er3X6ms
         tBtIsfxlHoXLohSTvE0GtEVtT5Q8GofKtL63Zho2c6ScomwQB6GSuLYo8/W6XdGpqlNL
         5Ob+6pCMr1rMH1VSISi0258Pw53Wz22eiARWM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=UO2DcGbAEIPqjffs5cWSWhFcHw86sN6rs7g1wP1vg1Y2vvSRebn2jAXZEz74Un/a5C
         iq8qMiMen5ggvCX0EP1UE7gvKT9+1FF09dv9YXegJ5I1oVE2/k7ziY/RB28FnvKtlXMy
         V/LHgBn75PGTjKwfUqDNNQVyPFichXA8zMR4Y=
MIME-Version: 1.0
Received: by 10.151.38.8 with HTTP; Thu, 22 Apr 2010 21:35:36 -0700 (PDT)
In-Reply-To: <C7F670C1.1C2D7%ddas@yahoo-inc.com>
References: <D964E5AC-96CA-4D7C-9991-1587C57E8FE7@apache.org>
	 <C7F670C1.1C2D7%ddas@yahoo-inc.com>
Date: Thu, 22 Apr 2010 21:35:36 -0700
Received: by 10.151.59.15 with SMTP id m15mr10793514ybk.246.1271997336199; 
	Thu, 22 Apr 2010 21:35:36 -0700 (PDT)
Message-ID: <g2o4aa34eb71004222135t35c11e58re98613ae3087c77b@mail.gmail.com>
Subject: Re: [VOTE] Should we approve the Hadoop security logo?
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00151750e14c612d660484dff53f

--00151750e14c612d660484dff53f
Content-Type: text/plain; charset=ISO-8859-1

Are we saying that we create a separate logo for every major feature we put
into Hadoop?

thanks,
dhruba

On Thu, Apr 22, 2010 at 9:31 PM, Devaraj Das <ddas@yahoo-inc.com> wrote:

> +1 for using the logo at least in t-shirts & security related
> documentation.
>
>
> On 4/22/10 3:17 PM, "Owen O'Malley" <omalley@apache.org> wrote:
>
>
>
> On Apr 22, 2010, at 2:44 PM, Doug Cutting wrote:
>
> > Owen O'Malley wrote:
> >>  Should we allow use of the Hadoop security logo (hadoop-security-
> >> logo.jpg) from HADOOP-6721. Clearly, I'm +1.
> >
> > Allow use where?  What's the intended purpose for this logo?
>
> The short term desire is to make t-shirts with a Hadoop security logo.
> It will also be used in documentation and literature relating to the
> Hadoop security features.
>
> -- Owen
>
>


-- 
Connect to me at http://www.facebook.com/dhruba

--00151750e14c612d660484dff53f--

From general-return-1445-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 04:59:09 2010
Return-Path: <general-return-1445-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45097 invoked from network); 23 Apr 2010 04:59:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 04:59:09 -0000
Received: (qmail 51949 invoked by uid 500); 23 Apr 2010 04:59:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51688 invoked by uid 500); 23 Apr 2010 04:59:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51675 invoked by uid 99); 23 Apr 2010 04:59:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 04:59:07 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of gautam.singaraju@gmail.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 04:59:02 +0000
Received: by gyf1 with SMTP id 1so5119310gyf.35
        for <general@hadoop.apache.org>; Thu, 22 Apr 2010 21:58:41 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=oWsmk7/ZLQqgfFkFK/ccDKv44/A2WdDiYvw/4GeWrko=;
        b=dYYYuKUR/BF2q/w4yf+qXzAzPH8U11D22VuK/d/ByUZrVUxqrWGf8Imw2n9bW5Yd/K
         1Txqb+VgDXGtrHgdrgy6SZR2WxY8enUIzAHfiz3/gDU2bZWPsaPf3c9GAlWLxZ0L2Ll2
         sO9RKJDB1YgLOcjw1ix+RXrRN6aPALoOujf0g=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=pm7K86ivwVy8/AofXMFD+NNBKaoTA0xpiRQx9euUy0N3Uak9JBZpAspbMpzGCHSJYt
         j7zJVLg/7Zyj1CUOsO24Zk+xuLhkpm+UfulPfHrMwyzsB0ShFJqY0tOIrH6/DtMU2Owx
         dllrN8ALNdaouWOxukMjOvUQglfaHqWgKJmTI=
MIME-Version: 1.0
Received: by 10.150.230.19 with HTTP; Thu, 22 Apr 2010 21:58:41 -0700 (PDT)
In-Reply-To: <u2i9f26e661004222127iea301d45z3494e059497dbdb3@mail.gmail.com>
References: <g2jf8b2ccd41004222118kb9c46c54oa7bc835bc0b5fcfd@mail.gmail.com>
	 <u2i9f26e661004222127iea301d45z3494e059497dbdb3@mail.gmail.com>
Date: Fri, 23 Apr 2010 00:58:41 -0400
Received: by 10.150.13.7 with SMTP id 7mr2648867ybm.238.1271998721584; Thu, 22 
	Apr 2010 21:58:41 -0700 (PDT)
Message-ID: <z2nf8b2ccd41004222158z8835b302v9e56c6b08c605a99@mail.gmail.com>
Subject: Re: DBOutputFormat over SSH?
From: Gautam Singaraju <gautam.singaraju@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Thanks Sonal, I will check it out.
---
Gautam



On Fri, Apr 23, 2010 at 12:27 AM, Sonal Goyal <sonalgoyal4@gmail.com> wrote=
:
> Hi Gautam,
>
> DBOutputFormat inserts records one by one, which is inherently slow. You =
can
> use open source Apache licensed hiho framework which provides MySQL's "lo=
ad
> data infile " functionality. =A0It may be more suited to your needs.
>
> HIHO is available at http://code.google.com/p/hiho/
>
> I havent tested it over ssh, please let me know if you need any help sett=
ing
> it up.
>
> Thanks and Regards,
> Sonal
> www.meghsoft.com
>
>
> On Fri, Apr 23, 2010 at 9:48 AM, Gautam Singaraju <
> gautam.singaraju@gmail.com> wrote:
>
>> All,
>>
>> I have a use-case where I need to crunch a large amount of data and
>> push to the results (comparatively a smaller set) to a mysql db at a
>> remote location. As per security concerns, only SSH ports are open. I
>> tried using Java Secure Channel [1] in combination with some custom
>> JDBC code from the reducers.
>>
>> Can anyone comment on the performance of DBOutputFormat? Have there
>> been any efforts to tunnel this through SSH? This is going to be an
>> expensive operation; any suggestions would be welcome.
>>
>> [1] http://www.jcraft.com/jsch/
>> ---
>> Gautam Singaraju
>>
>

From general-return-1446-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 05:02:32 2010
Return-Path: <general-return-1446-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46252 invoked from network); 23 Apr 2010 05:02:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 05:02:32 -0000
Received: (qmail 53901 invoked by uid 500); 23 Apr 2010 05:02:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53810 invoked by uid 500); 23 Apr 2010 05:02:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53796 invoked by uid 99); 23 Apr 2010 05:02:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 05:02:31 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of gautam.singaraju@gmail.com designates 74.125.83.48 as permitted sender)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 05:02:26 +0000
Received: by gwb1 with SMTP id 1so1974617gwb.35
        for <general@hadoop.apache.org>; Thu, 22 Apr 2010 22:02:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type;
        bh=CVEIBErb9s3hsffX815jko2BdR7lcPmGV4Zq62RVXpk=;
        b=QTOV++XvW8+jpukjPHXgiW6JEQHVdq9zbuKczk2R8jVNLeXEI+l/mRLwmLbCnbqF+q
         X99dOED3zZJpc/hnbx5800piq5po9YPgefcbyy+wft0cpRSnqJc5wWM8eXmDd68eB8Zj
         ubtsMQV9Prjl2L+bocNogyyqIgw15uSbRh7Zw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=bJgIe6DjHAY6/62FFGniZK7p4MSsfYxYLM53HSNInN55tKMk+NUmz+75bDky+FrlTH
         ebd7u6w059yqRGN5m2il4mqkwPy3JdyLpzN4wdH6h88KLnDu3fJCeHplwtRgJr/lN96d
         hKrJKF12CDdHFYWg+t50qzgND6RauCk+P6tyg=
MIME-Version: 1.0
Received: by 10.150.230.19 with HTTP; Thu, 22 Apr 2010 22:02:05 -0700 (PDT)
In-Reply-To: <r2z23d54121004222129gcc126a5egc00ff7fbb3d0c7bf@mail.gmail.com>
References: <g2jf8b2ccd41004222118kb9c46c54oa7bc835bc0b5fcfd@mail.gmail.com>
	 <r2z23d54121004222129gcc126a5egc00ff7fbb3d0c7bf@mail.gmail.com>
Date: Fri, 23 Apr 2010 01:02:05 -0400
Received: by 10.150.187.19 with SMTP id k19mr10978265ybf.96.1271998925532; 
	Thu, 22 Apr 2010 22:02:05 -0700 (PDT)
Message-ID: <y2rf8b2ccd41004222202m5dde78edx1176e2e3a5e7dab8@mail.gmail.com>
Subject: Re: DBOutputFormat over SSH?
From: Gautam Singaraju <gautam.singaraju@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Then there is a need to design a local differential DB and synchronize
the database with the remote location. That synchronization could be
over SSH and should be atomic.
---
Gautam



On Fri, Apr 23, 2010 at 12:29 AM, Eric Sammer <esammer@cloudera.com> wrote:
> In general, you'll want to avoid tunneling permanent production code
> over ssh tunnels. They're flaky and do not recover from network
> interruption in any reasonable way. If you need to do this, a vpn is
> the correct approach. Linux easily will do ipsec p2p tunnels that are
> reasonably secure. If you really only have port 22 then I suppose
> that's your only option but I really would reevaluate the security
> policy.
>
> Either way, it's going to be slow due to the encryption overhead but
> if it's a small amount of data, that may be fine.
>
> On Fri, Apr 23, 2010 at 12:18 AM, Gautam Singaraju
> <gautam.singaraju@gmail.com> wrote:
>> All,
>>
>> I have a use-case where I need to crunch a large amount of data and
>> push to the results (comparatively a smaller set) to a mysql db at a
>> remote location. As per security concerns, only SSH ports are open. I
>> tried using Java Secure Channel [1] in combination with some custom
>> JDBC code from the reducers.
>>
>> Can anyone comment on the performance of DBOutputFormat? Have there
>> been any efforts to tunnel this through SSH? This is going to be an
>> expensive operation; any suggestions would be welcome.
>>
>> [1] http://www.jcraft.com/jsch/
>> ---
>> Gautam Singaraju
>>
>
>
>
> --
> Eric Sammer
> phone: +1-917-287-2675
> twitter: esammer
> data: www.cloudera.com
>

From general-return-1447-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 05:27:55 2010
Return-Path: <general-return-1447-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 52600 invoked from network); 23 Apr 2010 05:27:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 05:27:55 -0000
Received: (qmail 72905 invoked by uid 500); 23 Apr 2010 05:27:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72661 invoked by uid 500); 23 Apr 2010 05:27:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72653 invoked by uid 99); 23 Apr 2010 05:27:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 05:27:53 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.62.24] (HELO qmta02.westchester.pa.mail.comcast.net) (76.96.62.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 05:27:45 +0000
Received: from omta15.westchester.pa.mail.comcast.net ([76.96.62.87])
	by qmta02.westchester.pa.mail.comcast.net with comcast
	id 8tM41e0011swQuc52tTRdu; Fri, 23 Apr 2010 05:27:25 +0000
Received: from [10.0.0.13] ([209.131.62.115])
	by omta15.westchester.pa.mail.comcast.net with comcast
	id 8tTF1e0062VBGtd3btTJSl; Fri, 23 Apr 2010 05:27:23 +0000
Message-Id: <121803A3-CFB9-489B-96EF-027234E55D25@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <00e801cae28f$98d27310$ca775930$@com>
Content-Type: multipart/alternative; boundary=Apple-Mail-13--512025332
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: License for Google's patent
Date: Thu, 22 Apr 2010 22:27:14 -0700
References: <00e801cae28f$98d27310$ca775930$@com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-13--512025332
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit

All,
    We got the following email from Larry Rosen, Apache's legal counsel.

-- Owen

On Apr 22, 2010, at 7:49 PM, Lawrence Rosen wrote:

> To: ASF Board
>
> Several weeks ago I sought clarification from Google about its  
> recent patent 7,650,331 ["System and method for efficient large- 
> scale data processing"] that may be infringed by implementation of  
> the Apache Hadoop and Apache MapReduce projects.  I just received  
> word from Google's general counsel that "we have granted a license  
> for Hadoop, terms of which are specified in the CLA."
>
> I am very pleased to reassure the Apache community about Google's  
> continued generosity and commitment to ASF and open source. Will  
> someone here please inform the Apache Hadoop and Apache MapReduce  
> projects that they need not worry about this patent.
>
> Best regards,
>
> /Larry
>
>
> Lawrence Rosen
> Rosenlaw & Einschlag, a technology law firm (www.rosenlaw.com)
> 3001 King Ranch Road, Ukiah, CA 95482
> Office: 707-485-1242    Cell: 707-478-8932
> Apache Software Foundation, member and counsel (www.apache.org)
> Open Web Foundation, board member (www.openwebfoundation.org)
> Stanford University, Instructor in Law
> Author, Open Source Licensing: Software Freedom and Intellectual  
> Property Law (Prentice Hall 2004)
>
>


--Apple-Mail-13--512025332--

From general-return-1448-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 05:31:28 2010
Return-Path: <general-return-1448-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53991 invoked from network); 23 Apr 2010 05:31:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 05:31:28 -0000
Received: (qmail 75023 invoked by uid 500); 23 Apr 2010 05:31:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74897 invoked by uid 500); 23 Apr 2010 05:31:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74888 invoked by uid 99); 23 Apr 2010 05:31:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 05:31:26 +0000
X-ASF-Spam-Status: No, hits=0.2 required=10.0
	tests=AWL,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.27.212] (HELO qmta14.emeryville.ca.mail.comcast.net) (76.96.27.212)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 05:31:18 +0000
Received: from omta03.emeryville.ca.mail.comcast.net ([76.96.30.27])
	by qmta14.emeryville.ca.mail.comcast.net with comcast
	id 8haU1e0070b6N64AEtWzLh; Fri, 23 Apr 2010 05:30:59 +0000
Received: from [10.0.0.13] ([209.131.62.115])
	by omta03.emeryville.ca.mail.comcast.net with comcast
	id 8tWp1e00L2VBGtd8PtWsJJ; Fri, 23 Apr 2010 05:30:56 +0000
Message-Id: <8B5126F1-EAA7-44BE-B692-D0F7EC6638A6@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <g2o4aa34eb71004222135t35c11e58re98613ae3087c77b@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Should we approve the Hadoop security logo?
Date: Thu, 22 Apr 2010 22:30:48 -0700
References: <D964E5AC-96CA-4D7C-9991-1587C57E8FE7@apache.org> <C7F670C1.1C2D7%ddas@yahoo-inc.com> <g2o4aa34eb71004222135t35c11e58re98613ae3087c77b@mail.gmail.com>
X-Mailer: Apple Mail (2.936)


On Apr 22, 2010, at 9:35 PM, Dhruba Borthakur wrote:

> Are we saying that we create a separate logo for every major feature  
> we put
> into Hadoop?

The issue is that to stay within Apache rules, all variants of the  
Hadoop elephant must be official Hadoop logos. Clearly it is not  
necessary to have a new logo for a feature, but it can be fun. This  
logo certainly doesn't harm the Hadoop image and conveys the intended  
meaning.

*Laugh* It is amusing to imagine what append's logo would look like.  
If you wanted to make an append (or some other feature) logo, the logo  
looked nice, and didn't harm the Hadoop image, that would be  
reasonable in my opinion.

-- Owen

From general-return-1449-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 06:21:50 2010
Return-Path: <general-return-1449-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 65053 invoked from network); 23 Apr 2010 06:21:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 06:21:49 -0000
Received: (qmail 13490 invoked by uid 500); 23 Apr 2010 06:21:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13156 invoked by uid 500); 23 Apr 2010 06:21:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13143 invoked by uid 99); 23 Apr 2010 06:21:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 06:21:48 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 06:21:39 +0000
Received: from SNV-EXBH01.ds.corp.yahoo.com (snv-exbh01.ds.corp.yahoo.com [207.126.227.249])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o3N6Jl7F050362
	for <general@hadoop.apache.org>; Thu, 22 Apr 2010 23:19:54 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:return-path:x-originalarrivaltime;
	b=W3fSQNPOI0I1PAws+e6KlDPulzY5VDmn9MsltCVOTzpsT2NbEDzJCs7C+O3op6Y9
Received: from SNV-EXVS09.ds.corp.yahoo.com ([207.126.227.86]) by SNV-EXBH01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Thu, 22 Apr 2010 23:18:50 -0700
Received: from 10.72.168.28 ([10.72.168.28]) by SNV-EXVS09.ds.corp.yahoo.com ([207.126.227.84]) via Exchange Front-End Server snv-webmail.corp.yahoo.com ([207.126.227.59]) with Microsoft Exchange Server HTTP-DAV ;
 Fri, 23 Apr 2010 06:18:33 +0000
User-Agent: Microsoft-Entourage/12.24.0.100205
Date: Thu, 22 Apr 2010 23:18:33 -0700
Subject: Re: [VOTE] Should we approve the Hadoop security logo?
From: Hairong Kuang <hairong@yahoo-inc.com>
To: <general@hadoop.apache.org>
Message-ID: <C7F689C9.1D5D0%hairong@yahoo-inc.com>
Thread-Topic: [VOTE] Should we approve the Hadoop security logo?
Thread-Index: AcrirMyyecbsV9K+hk2pi97oOeCaYA==
In-Reply-To: <8B5126F1-EAA7-44BE-B692-D0F7EC6638A6@apache.org>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-OriginalArrivalTime: 23 Apr 2010 06:18:50.0391 (UTC) FILETIME=[D7106A70:01CAE2AC]
X-Virus-Checked: Checked by ClamAV on apache.org

+1

I personally love to see that my daughter wears a T-shirt with a secure
Hadoop logo to show off she is the youngest Hadoop contributor. :-)

Hairong


On 4/22/10 10:30 PM, "Owen O'Malley" <omalley@apache.org> wrote:

> 
> On Apr 22, 2010, at 9:35 PM, Dhruba Borthakur wrote:
> 
>> Are we saying that we create a separate logo for every major feature
>> we put
>> into Hadoop?
> 
> The issue is that to stay within Apache rules, all variants of the
> Hadoop elephant must be official Hadoop logos. Clearly it is not
> necessary to have a new logo for a feature, but it can be fun. This
> logo certainly doesn't harm the Hadoop image and conveys the intended
> meaning.
> 
> *Laugh* It is amusing to imagine what append's logo would look like.
> If you wanted to make an append (or some other feature) logo, the logo
> looked nice, and didn't harm the Hadoop image, that would be
> reasonable in my opinion.
> 
> -- Owen


From general-return-1450-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 09:19:55 2010
Return-Path: <general-return-1450-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9694 invoked from network); 23 Apr 2010 09:19:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 09:19:54 -0000
Received: (qmail 63641 invoked by uid 500); 23 Apr 2010 09:19:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63515 invoked by uid 500); 23 Apr 2010 09:19:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63505 invoked by uid 99); 23 Apr 2010 09:19:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 09:19:53 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 09:19:45 +0000
Received: from EGL-EX07CAS02.ds.corp.yahoo.com (egl-ex07cas02.eglbp.corp.yahoo.com [203.83.248.209])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o3N9I7ux045469
	for <general@hadoop.apache.org>; Fri, 23 Apr 2010 02:18:21 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:in-reply-to:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=cxPNYbwX5UbmmKFFNlaBzfwWugcj0C8yTohZx6RrHQ2wYHH+3Ob1maWItlspavZM
Received: from EGL-EX07VS01.ds.corp.yahoo.com ([203.83.248.205]) by
 EGL-EX07CAS02.ds.corp.yahoo.com ([203.83.248.216]) with mapi; Fri, 23 Apr
 2010 14:45:09 +0530
From: "Ankur C. Goel" <gankur@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 23 Apr 2010 14:45:07 +0530
Subject: Re: License for Google's patent
Thread-Topic: License for Google's patent
Thread-Index: AcripcYujVOZKWdlSs2lGlD5j8XcnAAH7ELv
Message-ID: <C7F762F3.1164A%gankur@yahoo-inc.com>
In-Reply-To: <121803A3-CFB9-489B-96EF-027234E55D25@apache.org>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C7F762F31164Agankuryahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C7F762F31164Agankuryahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

This is such a good news for the community. Is the CLA available for public=
 viewing ?

-@nkur


On 4/23/10 10:57 AM, "Owen O'Malley" <omalley@apache.org> wrote:

All,
    We got the following email from Larry Rosen, Apache's legal counsel.

-- Owen

On Apr 22, 2010, at 7:49 PM, Lawrence Rosen wrote:

> To: ASF Board
>
> Several weeks ago I sought clarification from Google about its
> recent patent 7,650,331 ["System and method for efficient large-
> scale data processing"] that may be infringed by implementation of
> the Apache Hadoop and Apache MapReduce projects.  I just received
> word from Google's general counsel that "we have granted a license
> for Hadoop, terms of which are specified in the CLA."
>
> I am very pleased to reassure the Apache community about Google's
> continued generosity and commitment to ASF and open source. Will
> someone here please inform the Apache Hadoop and Apache MapReduce
> projects that they need not worry about this patent.
>
> Best regards,
>
> /Larry
>
>
> Lawrence Rosen
> Rosenlaw & Einschlag, a technology law firm (www.rosenlaw.com)
> 3001 King Ranch Road, Ukiah, CA 95482
> Office: 707-485-1242    Cell: 707-478-8932
> Apache Software Foundation, member and counsel (www.apache.org)
> Open Web Foundation, board member (www.openwebfoundation.org)
> Stanford University, Instructor in Law
> Author, Open Source Licensing: Software Freedom and Intellectual
> Property Law (Prentice Hall 2004)
>
>



--_000_C7F762F31164Agankuryahooinccom_--

From general-return-1451-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 11:10:46 2010
Return-Path: <general-return-1451-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34303 invoked from network); 23 Apr 2010 11:10:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 11:10:45 -0000
Received: (qmail 90205 invoked by uid 500); 23 Apr 2010 11:10:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89823 invoked by uid 500); 23 Apr 2010 11:10:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89815 invoked by uid 99); 23 Apr 2010 11:10:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 11:10:43 +0000
X-ASF-Spam-Status: No, hits=0.2 required=10.0
	tests=AWL,FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bernd.fondermann@googlemail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 11:10:38 +0000
Received: by wyb40 with SMTP id 40so637281wyb.35
        for <general@hadoop.apache.org>; Fri, 23 Apr 2010 04:10:17 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=vJfV2cDuknscq5B1uUIcjqmqT44h1TWWI1pGakKidLM=;
        b=JMEaHs1j2KZTh5y95H6c/mAJy8seplbCzXeSwGyM8TTQpIH64UD/8rHUQr3JLXkiud
         EOM7NODQaYaScLZPAqo5BhSwlcpSrniORJhopWIEnVvhIZTyVfFRf27+ooM7eMPovAWJ
         pOv/wchYEzMtkGWVUUHb4dtv07toE63oe6MQw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=FC4x7eJuILOtZXGhOdYGpdpLCRYImTqhN3+zWjwr6pLZBKT/TdJvS9awSoD9lYGidy
         Hu+sQgi+XGi9sw8DoD593udQtiIyeKtBkq1f+CPWcMzEnTXBm0S+/XgbL44izhD6kjzL
         hCE0yDdW+JMfuPVJAsPEE/JgDvncXwsF0M3Cg=
MIME-Version: 1.0
Received: by 10.216.186.15 with HTTP; Fri, 23 Apr 2010 04:10:17 -0700 (PDT)
In-Reply-To: <C7F762F3.1164A%gankur@yahoo-inc.com>
References: <121803A3-CFB9-489B-96EF-027234E55D25@apache.org>
	 <C7F762F3.1164A%gankur@yahoo-inc.com>
Date: Fri, 23 Apr 2010 13:10:17 +0200
Received: by 10.216.88.12 with SMTP id z12mr2962260wee.165.1272021017401; Fri, 
	23 Apr 2010 04:10:17 -0700 (PDT)
Message-ID: <h2j88f6e29a1004230410i59593c5bt581e3b1a494429ba@mail.gmail.com>
Subject: Re: License for Google's patent
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Fri, Apr 23, 2010 at 11:15, Ankur C. Goel <gankur@yahoo-inc.com> wrote:
> Is the CLA available for public viewing ?

CLAs contain personal data, so generally: No, don't think so.

  Bernd

>
> -@nkur
>
>
> On 4/23/10 10:57 AM, "Owen O'Malley" <omalley@apache.org> wrote:
>
> All,
> =A0 =A0We got the following email from Larry Rosen, Apache's legal counse=
l.
>
> -- Owen
>
> On Apr 22, 2010, at 7:49 PM, Lawrence Rosen wrote:
>
>> To: ASF Board
>>
>> Several weeks ago I sought clarification from Google about its
>> recent patent 7,650,331 ["System and method for efficient large-
>> scale data processing"] that may be infringed by implementation of
>> the Apache Hadoop and Apache MapReduce projects. =A0I just received
>> word from Google's general counsel that "we have granted a license
>> for Hadoop, terms of which are specified in the CLA."
>>
>> I am very pleased to reassure the Apache community about Google's
>> continued generosity and commitment to ASF and open source. Will
>> someone here please inform the Apache Hadoop and Apache MapReduce
>> projects that they need not worry about this patent.
>>
>> Best regards,
>>
>> /Larry
>>
>>
>> Lawrence Rosen
>> Rosenlaw & Einschlag, a technology law firm (www.rosenlaw.com)
>> 3001 King Ranch Road, Ukiah, CA 95482
>> Office: 707-485-1242 =A0 =A0Cell: 707-478-8932
>> Apache Software Foundation, member and counsel (www.apache.org)
>> Open Web Foundation, board member (www.openwebfoundation.org)
>> Stanford University, Instructor in Law
>> Author, Open Source Licensing: Software Freedom and Intellectual
>> Property Law (Prentice Hall 2004)
>>
>>
>
>
>

From general-return-1452-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 17:59:33 2010
Return-Path: <general-return-1452-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37598 invoked from network); 23 Apr 2010 17:59:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 17:59:33 -0000
Received: (qmail 46160 invoked by uid 500); 23 Apr 2010 17:59:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46093 invoked by uid 500); 23 Apr 2010 17:59:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46085 invoked by uid 99); 23 Apr 2010 17:59:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 17:59:31 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 23 Apr 2010 17:59:29 +0000
Received: (qmail 37568 invoked by uid 99); 23 Apr 2010 17:59:07 -0000
Received: from localhost.apache.org (HELO [192.168.168.106]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 17:59:07 +0000
Message-ID: <4BD1DFEA.5020908@apache.org>
Date: Fri, 23 Apr 2010 10:59:06 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100411)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: [DISCUSS] secure 0.20-based branch
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Y!'s developed extensive security features based on the 0.20 branch. The 
0.20 versions of the individual patches appear in Jira, but these have 
not been committed to any branch in Apache's SVN.  Y! has periodically 
pushed out versions of these as Yahoo!'s Distribution of Hadoop at 
github, and Cloudera is likely to make a 0.20-based distribution 
including these as well.

Shouldn't we commit these all to some 0.20-based branch at Apache?  I'd 
earlier (on common-dev) suggested we might start a 1.0 branch based on 
0.20, then add a 1.1 branch with the security patches.  If that were 
done, the 0.21 release could perhaps instead be called 1.2.  But, 
regardless of the naming, it would be good to have the 0.20 versions of 
all of the security patches committed to a branch at Apache so that we 
can make a release that includes them, patches can be targeted against 
this branch, etc.

What do others think?

Doug


From general-return-1453-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 18:35:17 2010
Return-Path: <general-return-1453-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53140 invoked from network); 23 Apr 2010 18:35:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 18:35:17 -0000
Received: (qmail 94579 invoked by uid 500); 23 Apr 2010 18:35:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94485 invoked by uid 500); 23 Apr 2010 18:35:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94476 invoked by uid 99); 23 Apr 2010 18:35:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 18:35:16 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=AWL,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 18:35:11 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:Received:Received:From:To:Subject:
   Thread-Topic:Thread-Index:Date:Message-ID:References:
   In-Reply-To:Accept-Language:Content-Language:
   X-MS-Has-Attach:X-MS-TNEF-Correlator:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version:
   Return-Path:X-OriginalArrivalTime;
  b=DlYxzHNoInKtcH27z8vRGFX+cUwW+qyhIu+QHoIAPaSmawbBrlSq+xyi
   oX9Zs5elJOlkXePNgEOo/EwlYNphNirMf6prelr4b+caHeTpSnxZu6+yg
   FxN2pC9dPDBcnbt;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1272047711; x=1303583711;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20[DISCUSS]=20secure=200.20-based=20branc
   h|Date:=20Fri,=2023=20Apr=202010=2018:34:49=20+0000
   |Message-ID:=20<792D1884-3AF7-4087-AC78-B1872D895138@link
   edin.com>|To:=20"<general@hadoop.apache.org>"=20<general@
   hadoop.apache.org>|MIME-Version:=201.0
   |Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<f6a5dce0-dc3d-42e6-b1d0-dd9f7003bc8b>
   |In-Reply-To:=20<4BD1DFEA.5020908@apache.org>|References:
   =20<4BD1DFEA.5020908@apache.org>;
  bh=PqN7SQiW9ekR5m3qPRqQBOn1aPU2k/mw8ilWSoQ8N+w=;
  b=CRRQM7LPBBLCvt5JhNIlgVkwytdbSUaeCHizFH9ZsDLcGYFsMMuKf4f3
   tROuy3DIqNf+DY4rBfqfEoukWguylVDYXPn6pGhoBtCeJI4RxLODjrNdR
   sjF4hXZBoSRhzbU;
X-IronPort-AV: E=Sophos;i="4.52,263,1270450800"; 
   d="scan'208";a="12500953"
Received: from esv4-exctest.linkedin.biz ([172.18.46.60]) by CORP-MAIL.linkedin.biz with Microsoft SMTPSVC(6.0.3790.3959);
	 Fri, 23 Apr 2010 11:34:51 -0700
Received: from ESV4-CAS01.linkedin.biz (172.18.46.140) by
 esv4-exctest.linkedin.biz (172.18.46.60) with Microsoft SMTP Server (TLS) id
 14.0.682.1; Fri, 23 Apr 2010 11:34:50 -0700
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi; Fri, 23 Apr 2010
 11:34:50 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: [DISCUSS] secure 0.20-based branch
Thread-Topic: [DISCUSS] secure 0.20-based branch
Thread-Index: AQHK4w67e+CPFqDOjkKEj2kXIrJNoJIw1oqA
Date: Fri, 23 Apr 2010 18:34:49 +0000
Message-ID: <792D1884-3AF7-4087-AC78-B1872D895138@linkedin.com>
References: <4BD1DFEA.5020908@apache.org>
In-Reply-To: <4BD1DFEA.5020908@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Content-Type: text/plain; charset="us-ascii"
Content-ID: <f6a5dce0-dc3d-42e6-b1d0-dd9f7003bc8b>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginalArrivalTime: 23 Apr 2010 18:34:51.0179 (UTC) FILETIME=[A8F1E7B0:01CAE313]


On Apr 23, 2010, at 10:59 AM, Doug Cutting wrote:
> What do others think?

That 0.20 is not 1.0 quality, no matter how hard people want to believe it =
is true.  The API may be stable, but the ops support sucks.




From general-return-1454-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 18:58:55 2010
Return-Path: <general-return-1454-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57872 invoked from network); 23 Apr 2010 18:58:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 18:58:55 -0000
Received: (qmail 13605 invoked by uid 500); 23 Apr 2010 18:58:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13565 invoked by uid 500); 23 Apr 2010 18:58:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13557 invoked by uid 99); 23 Apr 2010 18:58:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 18:58:54 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 23 Apr 2010 18:58:52 +0000
Received: (qmail 57827 invoked by uid 99); 23 Apr 2010 18:58:30 -0000
Received: from localhost.apache.org (HELO [192.168.168.106]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 18:58:30 +0000
Message-ID: <4BD1EDD6.8060004@apache.org>
Date: Fri, 23 Apr 2010 11:58:30 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100411)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSS] secure 0.20-based branch
References: <4BD1DFEA.5020908@apache.org> <792D1884-3AF7-4087-AC78-B1872D895138@linkedin.com>
In-Reply-To: <792D1884-3AF7-4087-AC78-B1872D895138@linkedin.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Allen Wittenauer wrote:
> That 0.20 is not 1.0 quality, no matter how hard people want to believe it is true.

Allen, my question was, "regardless of the naming" should we try to 
merge all of the 0.20-based security patches to a branch in Apache's 
subversion?

As for the naming, the major release number does not make a claim about 
quality or features, but rather about compatibility.  1.0 would 
presumably be the lowest quality and least featured release in the 1.x 
series, but everything in that series should be API compatible with 1.0. 
  Every release in the 2.x series might not be compatible with 1.0. 
Point releases add features, dot releases add quality.  So 1.0.1 would 
only improve quality, while 1.1.0 would add features while maintaining 
compatibility.

Doug

From general-return-1455-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 20:36:19 2010
Return-Path: <general-return-1455-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88510 invoked from network); 23 Apr 2010 20:36:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 20:36:19 -0000
Received: (qmail 27171 invoked by uid 500); 23 Apr 2010 20:36:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27132 invoked by uid 500); 23 Apr 2010 20:36:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27124 invoked by uid 99); 23 Apr 2010 20:36:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 20:36:18 +0000
X-ASF-Spam-Status: No, hits=-1310.3 required=10.0
	tests=ALL_TRUSTED,AWL,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 23 Apr 2010 20:36:16 +0000
Received: (qmail 88410 invoked by uid 99); 23 Apr 2010 20:35:56 -0000
Received: from localhost.apache.org (HELO mail-iw0-f203.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 20:35:56 +0000
Received: by iwn41 with SMTP id 41so7153261iwn.20
        for <general@hadoop.apache.org>; Fri, 23 Apr 2010 13:35:55 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.190.197 with SMTP id dj5mr148451ibb.58.1272054955529; Fri, 
	23 Apr 2010 13:35:55 -0700 (PDT)
Received: by 10.231.152.206 with HTTP; Fri, 23 Apr 2010 13:35:55 -0700 (PDT)
In-Reply-To: <w2i1267dd3b1004231235k6a58b298wca4d3f014822f788@mail.gmail.com>
References: <4BD1DFEA.5020908@apache.org>
	 <792D1884-3AF7-4087-AC78-B1872D895138@linkedin.com>
	 <w2i1267dd3b1004231235k6a58b298wca4d3f014822f788@mail.gmail.com>
Date: Fri, 23 Apr 2010 21:35:55 +0100
Message-ID: <o2g1267dd3b1004231335y40ea2b48i23647e6272a18679@mail.gmail.com>
Subject: [DISCUSS] secure 0.20-based branch
From: Chris Douglas <cdouglas@apache.org>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Please, let's not repeat the discussion on 0.20 as 1.0 in this thread.

I oppose the immortality of the 0.20 branch for the same reasons I
opposed it on common-dev. From a technical perspective, nothing has
been more destructive to the momentum and focus of this project than
the perpetual backporting and development on this branch. Yahoo,
Cloudera, and Facebook have their reasons for building fortresses on
the sands of 0.20, but Apache has a year of development beyond that.
It's a dark, unmapped jungle at the moment, but what you propose will
only exacerbate that problem by establishing a fourth settlement on
that sad oasis.

I vote no. Apache doesn't need to participate in the ridiculous
exercise of porting 0.20 to 0.22. Why not support (and aid) Tom's
effort to stabilize trunk? -C

On Friday, April 23, 2010, Doug Cutting <cutting@apache.org> wrote:
> Allen Wittenauer wrote:
>
> That 0.20 is not 1.0 quality, no matter how hard people want to believe i=
t is true.
>
>
> Allen, my question was, "regardless of the naming" should we try to merge=
 all of the 0.20-based security patches to a branch in Apache's subversion?
>
> As for the naming, the major release number does not make a claim about q=
uality or features, but rather about compatibility. =A01.0 would presumably=
 be the lowest quality and least featured release in the 1.x series, but ev=
erything in that series should be API compatible with 1.0. =A0Every release=
 in the 2.x series might not be compatible with 1.0. Point releases add fea=
tures, dot releases add quality. =A0So 1.0.1 would only improve quality, wh=
ile 1.1.0 would add features while maintaining compatibility.
>
> Doug
>

From general-return-1456-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 20:42:15 2010
Return-Path: <general-return-1456-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 90568 invoked from network); 23 Apr 2010 20:42:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 20:42:15 -0000
Received: (qmail 39323 invoked by uid 500); 23 Apr 2010 20:42:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39279 invoked by uid 500); 23 Apr 2010 20:42:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39271 invoked by uid 99); 23 Apr 2010 20:42:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 20:42:13 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 23 Apr 2010 20:42:11 +0000
Received: (qmail 90389 invoked by uid 99); 23 Apr 2010 20:41:49 -0000
Received: from localhost.apache.org (HELO [192.168.168.106]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 20:41:49 +0000
Message-ID: <4BD2060D.7060604@apache.org>
Date: Fri, 23 Apr 2010 13:41:49 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100411)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSS] secure 0.20-based branch
References: <4BD1DFEA.5020908@apache.org>	 <792D1884-3AF7-4087-AC78-B1872D895138@linkedin.com>	 <w2i1267dd3b1004231235k6a58b298wca4d3f014822f788@mail.gmail.com> <o2g1267dd3b1004231335y40ea2b48i23647e6272a18679@mail.gmail.com>
In-Reply-To: <o2g1267dd3b1004231335y40ea2b48i23647e6272a18679@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Chris Douglas wrote:
> Why not support (and aid) Tom's effort to stabilize trunk?

I do support Tom's efforts and do not see these as mutually exclusive.

Doug

From general-return-1457-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 20:52:17 2010
Return-Path: <general-return-1457-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95751 invoked from network); 23 Apr 2010 20:52:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 20:52:16 -0000
Received: (qmail 56333 invoked by uid 500); 23 Apr 2010 20:52:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56271 invoked by uid 500); 23 Apr 2010 20:52:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56263 invoked by uid 99); 23 Apr 2010 20:52:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 20:52:15 +0000
X-ASF-Spam-Status: No, hits=-74.0 required=10.0
	tests=AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 23 Apr 2010 20:52:14 +0000
Received: (qmail 95626 invoked by uid 99); 23 Apr 2010 20:51:54 -0000
Received: from localhost.apache.org (HELO [192.168.168.106]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 20:51:54 +0000
Message-ID: <4BD20869.3060304@apache.org>
Date: Fri, 23 Apr 2010 13:51:53 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100411)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSS] secure 0.20-based branch
References: <4BD1DFEA.5020908@apache.org>	 <792D1884-3AF7-4087-AC78-B1872D895138@linkedin.com>	 <w2i1267dd3b1004231235k6a58b298wca4d3f014822f788@mail.gmail.com> <o2g1267dd3b1004231335y40ea2b48i23647e6272a18679@mail.gmail.com>
In-Reply-To: <o2g1267dd3b1004231335y40ea2b48i23647e6272a18679@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Chris Douglas wrote:
> From a technical perspective, nothing has
> been more destructive to the momentum and focus of this project than
> the perpetual backporting and development on this branch.

I'm not proposing any more back- or forward-porting than will be done 
anyway.  I'm proposing we commit all the 0.20 security patches to a repo 
at Apache.  This could be as simple as copying Y!'s github repo 
wholesale to a branch at Apache.  Or, with a bit more effort, we can 
probably apply the patches from Jira in the right order.  Otherwise, 
Cloudera, Facebook and others will have to duplicate this effort.  Then 
bugfix patches and releases can be made against this repo rather than 
everyone having to assemble and maintain their own 0.20 security 
patchset from scratch.  Everyone will be using this patchset for quite 
some time.  Why shouldn't we share a repo that contains it?

Doug

From general-return-1458-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 21:31:10 2010
Return-Path: <general-return-1458-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 8270 invoked from network); 23 Apr 2010 21:31:10 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 21:31:10 -0000
Received: (qmail 12682 invoked by uid 500); 23 Apr 2010 21:31:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12636 invoked by uid 500); 23 Apr 2010 21:31:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12628 invoked by uid 99); 23 Apr 2010 21:31:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 21:31:09 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 209.85.211.174 as permitted sender)
Received: from [209.85.211.174] (HELO mail-yw0-f174.google.com) (209.85.211.174)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 21:31:04 +0000
Received: by ywh4 with SMTP id 4so1276943ywh.5
        for <general@hadoop.apache.org>; Fri, 23 Apr 2010 14:30:43 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=ursIC59jTCO8ghro7VIx5EWO5PZXOvvqqwBx3WCwStc=;
        b=DpZm5bt/Cf/31b86SHVdi8CWX0eKkQ6w6CfNtZofRyhzWfpViNux+tEa8r3h5xA2FA
         DqJ9OltKxmUHxzaFHbnM4PDOp+zsO/ZjIURkVLJi4v2JM+AnqkQbawNhp6z+++Kh9orh
         lBtC7BRgjGQX3pWI91huJMSTpw0qvHY4UxV+k=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=BA+TNI8yFQouSFc+WNCWoLvYlkwiGtwxbiyscDezXS48hVLL+9Vy2xvBO6AjpdLBZx
         k8aYblx2bRJqsj5dcq0Zoq2zv/OphB8H27w1e8EejN8n3+Ik+Ot5NNzzZkax2L7uPUb/
         MauR65b8i8ran3t9bxTZI7qjLFozq6Ku3Uvbw=
MIME-Version: 1.0
Received: by 10.150.14.5 with SMTP id 5mr687549ybn.17.1272058243147; Fri, 23 
	Apr 2010 14:30:43 -0700 (PDT)
Received: by 10.151.38.8 with HTTP; Fri, 23 Apr 2010 14:30:43 -0700 (PDT)
In-Reply-To: <4BD20869.3060304@apache.org>
References: <4BD1DFEA.5020908@apache.org>
	 <792D1884-3AF7-4087-AC78-B1872D895138@linkedin.com>
	 <w2i1267dd3b1004231235k6a58b298wca4d3f014822f788@mail.gmail.com>
	 <o2g1267dd3b1004231335y40ea2b48i23647e6272a18679@mail.gmail.com>
	 <4BD20869.3060304@apache.org>
Date: Fri, 23 Apr 2010 14:30:43 -0700
Message-ID: <m2q4aa34eb71004231430od3439bcehd74d72f4247543ad@mail.gmail.com>
Subject: Re: [DISCUSS] secure 0.20-based branch
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd76242b7683c0484ee23e2

--000e0cd76242b7683c0484ee23e2
Content-Type: text/plain; charset=ISO-8859-1

I support Doug's idea whole-heartedly. The question that remains is "who
gets to test and stabilize this new branch"? I am proposing that we
designate a owner for this branch and it is the onus of the owner of this
branch to test/stabilize that branch.

thanks,
dhruba

On Fri, Apr 23, 2010 at 1:51 PM, Doug Cutting <cutting@apache.org> wrote:

> Chris Douglas wrote:
>
>> From a technical perspective, nothing has
>> been more destructive to the momentum and focus of this project than
>> the perpetual backporting and development on this branch.
>>
>
> I'm not proposing any more back- or forward-porting than will be done
> anyway.  I'm proposing we commit all the 0.20 security patches to a repo at
> Apache.  This could be as simple as copying Y!'s github repo wholesale to a
> branch at Apache.  Or, with a bit more effort, we can probably apply the
> patches from Jira in the right order.  Otherwise, Cloudera, Facebook and
> others will have to duplicate this effort.  Then bugfix patches and releases
> can be made against this repo rather than everyone having to assemble and
> maintain their own 0.20 security patchset from scratch.  Everyone will be
> using this patchset for quite some time.  Why shouldn't we share a repo that
> contains it?
>
> Doug
>



-- 
Connect to me at http://www.facebook.com/dhruba

--000e0cd76242b7683c0484ee23e2--

From general-return-1459-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 23:03:24 2010
Return-Path: <general-return-1459-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25195 invoked from network); 23 Apr 2010 23:03:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 23:03:24 -0000
Received: (qmail 80447 invoked by uid 500); 23 Apr 2010 23:03:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80400 invoked by uid 500); 23 Apr 2010 23:03:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80392 invoked by uid 99); 23 Apr 2010 23:03:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 23:03:23 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 23 Apr 2010 23:03:20 +0000
Received: (qmail 25032 invoked by uid 99); 23 Apr 2010 23:02:59 -0000
Received: from localhost.apache.org (HELO [192.168.168.106]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 23:02:59 +0000
Message-ID: <4BD22722.5060009@apache.org>
Date: Fri, 23 Apr 2010 16:02:58 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100411)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSS] secure 0.20-based branch
References: <4BD1DFEA.5020908@apache.org>	 <792D1884-3AF7-4087-AC78-B1872D895138@linkedin.com>	 <w2i1267dd3b1004231235k6a58b298wca4d3f014822f788@mail.gmail.com>	 <o2g1267dd3b1004231335y40ea2b48i23647e6272a18679@mail.gmail.com>	 <4BD20869.3060304@apache.org> <m2q4aa34eb71004231430od3439bcehd74d72f4247543ad@mail.gmail.com>
In-Reply-To: <m2q4aa34eb71004231430od3439bcehd74d72f4247543ad@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Dhruba Borthakur wrote:
> I support Doug's idea whole-heartedly. The question that remains is "who
> gets to test and stabilize this new branch"?

As a starting point, Y!'s distribution includes a patch list at:

http://github.com/yahoo/hadoop-common/commits/yahoo-hadoop-0.20

Cloudera lists its patches in the tarball inside the source rpm:

http://archive.cloudera.com/redhat/cdh/3/SRPMS/hadoop-0.20-0.20.2+228-1.src.rpm

We could perhaps start with the intersection of these and vote on that. 
  If there are patches missing from this list that you believe are well 
tested and critical to Facebook, then these could be nominated as well. 
  We'd require a consensus vote on each patch.

> I am proposing that we
> designate a owner for this branch and it is the onus of the owner of this
> branch to test/stabilize that branch.

I'm happy to call votes on patch lists, merge patches to the branch, run 
unit tests and roll release candidates, although I'd love help.  If we 
stick to patches and combinations of patches that folks are already 
testing elsewhere, then this should be a stable branch.  The first 
release on this branch should be declared alpha until its tested in a 
variety of environments.  Folks should of course not immediately put 
into production any release from this branch (or any other branch) 
without some testing of their own.  If folks prefer to continue to use 
releases blessed by Y! or Cloudera, then we'd at least make the patch 
lists of those releases considerably shorter.  This branch would 
simplify sharing of bugfixes even if we don't make releases from it, 
since it would already contain the patches common to most production 
environments.

Doug

From general-return-1460-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 24 05:38:32 2010
Return-Path: <general-return-1460-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 8500 invoked from network); 24 Apr 2010 05:38:31 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Apr 2010 05:38:31 -0000
Received: (qmail 31602 invoked by uid 500); 24 Apr 2010 05:38:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31325 invoked by uid 500); 24 Apr 2010 05:38:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31317 invoked by uid 99); 24 Apr 2010 05:38:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Apr 2010 05:38:29 +0000
X-ASF-Spam-Status: No, hits=-1317.3 required=10.0
	tests=ALL_TRUSTED,AWL,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Sat, 24 Apr 2010 05:38:28 +0000
Received: (qmail 8475 invoked by uid 99); 24 Apr 2010 05:38:08 -0000
Received: from localhost.apache.org (HELO mail-iw0-f198.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Apr 2010 05:38:08 +0000
Received: by iwn36 with SMTP id 36so4009772iwn.29
        for <general@hadoop.apache.org>; Fri, 23 Apr 2010 22:38:07 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.162.85 with SMTP id u21mr309421ibx.95.1272087487637; Fri, 
	23 Apr 2010 22:38:07 -0700 (PDT)
Received: by 10.231.152.206 with HTTP; Fri, 23 Apr 2010 22:38:07 -0700 (PDT)
In-Reply-To: <4BD22722.5060009@apache.org>
References: <4BD1DFEA.5020908@apache.org>
	 <792D1884-3AF7-4087-AC78-B1872D895138@linkedin.com>
	 <w2i1267dd3b1004231235k6a58b298wca4d3f014822f788@mail.gmail.com>
	 <o2g1267dd3b1004231335y40ea2b48i23647e6272a18679@mail.gmail.com>
	 <4BD20869.3060304@apache.org>
	 <m2q4aa34eb71004231430od3439bcehd74d72f4247543ad@mail.gmail.com>
	 <4BD22722.5060009@apache.org>
Date: Sat, 24 Apr 2010 06:38:07 +0100
Message-ID: <w2n1267dd3b1004232238i3a848f33xeabade312fcc22dd@mail.gmail.com>
Subject: [DISCUSS] secure 0.20-based branch
From: Chris Douglas <cdouglas@apache.org>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

> I'm not proposing any more back- or forward-porting than will be done any=
way.

That's probably true for this release, but what about HDFS-200? With
security and sync in 0.20, there is less motivation to move back to
trunk, which has diverged significantly. Moving off of 0.20 will be a
struggle without these supplements. With a weak trunk, the
justifications that led to the current state will remain in force.

> =A0Or, with a bit more effort, we can probably apply the patches from Jir=
a in the right order. =A0Otherwise, Cloudera, Facebook and others will have=
 to duplicate this effort.

Cloudera, Facebook, and others could also help to finish this work in
trunk. Or clone from github if the need is dire.

> Then bugfix patches and releases can be made against this repo rather tha=
n everyone having to assemble and maintain their own 0.20 security patchset=
 from scratch. =A0Everyone will be using this patchset for quite some time.=
 =A0Why shouldn't we share a repo that contains it?

Because trunk is the shared repository that contains the security
work. And a working append. And dozens of smaller, but important
features including the 1.0 APIs. Symlinks. Optimizations to the
shuffle. Splittable bzip compression. Stability and scalability fixes
to the NameNode and JobTracker. Unicorns and happiness.

Stabilizing, packaging, and testing trunk is drudgery, but it can be shared=
.

I can see the value in restarting collaboration between major
contributors by reestablishing a common branch, and 0.20 will probably
be more successful in that respect, at least earlier. However, I
continue to oppose sinking combined energy into 0.20 at the expense of
trunk, for reasons already discussed at length. -C

> Doug
>

From general-return-1461-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 25 04:47:17 2010
Return-Path: <general-return-1461-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69380 invoked from network); 25 Apr 2010 04:47:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Apr 2010 04:47:17 -0000
Received: (qmail 49370 invoked by uid 500); 25 Apr 2010 04:47:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49044 invoked by uid 500); 25 Apr 2010 04:47:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49035 invoked by uid 99); 25 Apr 2010 04:47:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 04:47:13 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of gautam.singaraju@gmail.com designates 74.125.83.48 as permitted sender)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 04:47:06 +0000
Received: by gwj16 with SMTP id 16so210864gwj.35
        for <general@hadoop.apache.org>; Sat, 24 Apr 2010 21:46:45 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=9ZZCO+m2aO2AvGr/1AeiAVUrhqGyfyQ7iP12Su/7WiM=;
        b=I4N1ZQFT7GtG9CnB27CziPPM9/Tyc00+OGjT5H6FfgqRakhZoda9bnVSWN9ujo/4Sz
         9WJW5rFlX1qlPJHnD0ZOLatnVP2+wJ6NU62xItZOjFodBfmv1aFnp/4YX6twCnz5/BsG
         A5bvzut7kSZ92YuCN2LZ12F67xWI0K7qYKtBY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=Kxgqtg204qwR51ghQUK/MhRp0Wjfl7Fk0kgV+p6BaaLaN4chj09KmFZUcLeD0vOFN8
         W47ha+cu3ziKyc5fcMqnTGsGSGmD+Uw3x8adiq0gIYRCETdl6RQcTJrqYgwckJz/Yuwa
         maFrHq90VBtfPt5NueQhhtp64Sc9ubzB6WiQc=
MIME-Version: 1.0
Received: by 10.150.248.3 with SMTP id v3mr2399073ybh.82.1272170805622; Sat, 
	24 Apr 2010 21:46:45 -0700 (PDT)
Received: by 10.150.230.19 with HTTP; Sat, 24 Apr 2010 21:46:45 -0700 (PDT)
Date: Sun, 25 Apr 2010 00:46:45 -0400
Message-ID: <l2pf8b2ccd41004242146kea04a168l8721f6e14656c1a6@mail.gmail.com>
Subject: jobclient.getFS throwing an NullPointerException
From: Gautam Singaraju <gautam.singaraju@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

All,

After a reduce task, I would like to have a handle to
DistributedFileSystem. We have the deprecated hadoop-site.xml instead
of core-site.xml, mapred-site.xml and hdfs-site.xml. When I perform a
jobclient.getFS() it throws a NullPointerException.

I am not sure if the configuration is an issue. Any tips would be appreciated?
---
Gautam

From general-return-1462-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 25 19:11:47 2010
Return-Path: <general-return-1462-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35583 invoked from network); 25 Apr 2010 19:11:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Apr 2010 19:11:47 -0000
Received: (qmail 8208 invoked by uid 500); 25 Apr 2010 19:11:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8148 invoked by uid 500); 25 Apr 2010 19:11:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8140 invoked by uid 99); 25 Apr 2010 19:11:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 19:11:45 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of magawake@gmail.com designates 209.85.217.225 as permitted sender)
Received: from [209.85.217.225] (HELO mail-gx0-f225.google.com) (209.85.217.225)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 19:11:37 +0000
Received: by gxk25 with SMTP id 25so6466959gxk.11
        for <general@hadoop.apache.org>; Sun, 25 Apr 2010 12:11:16 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=4TNZWtxSzBqUU75vtw3MZWgUQei5L0GpeFdTR+PNWPs=;
        b=ghzwjUkzO73k66LiZtEodAxaUnr3kiD6iBwaYI6oOyJ5ikaV9fxAq2Mgdjj3c8QJX0
         85kvG9Z82Fn4ABK7l0wGWhoXAZ2+GHV4qVZwPzir2hwGdeMNaAQvVcnxU4VJm4BdP5WB
         MWaRn54tvftR30pwmXQxuclJEEZE1nfbngSMM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=u/0l1923OkhaSgJeStR26KVUKfzr2JuQvKstwOA8qTHPw86O2BHbfDjDWcuR+sxnL9
         Zf2P14ZV5XhMsvF2xZEgpm+MULxWXCrVPLKc+VrWU6/1KQFv17v20kxYyGyUc7i/hpt7
         v5OmkevIWGZQIwTy3zeerP0IBMFoCjDVU/mNI=
MIME-Version: 1.0
Received: by 10.101.9.21 with SMTP id m21mr3483293ani.119.1272222676673; Sun, 
	25 Apr 2010 12:11:16 -0700 (PDT)
Received: by 10.231.4.21 with HTTP; Sun, 25 Apr 2010 12:11:16 -0700 (PDT)
Date: Sun, 25 Apr 2010 15:11:16 -0400
Message-ID: <n2o1cbd6f831004251211j54fd4350gc2409ebd80927914@mail.gmail.com>
Subject: replace namenode
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
X-Virus-Checked: Checked by ClamAV on apache.org

Hello All,

I have been testing hdfs and have been very happy. Currently, I run it
on a 200 node cluster equating to 250TB :-). So far everything has
been good. My question is I would like to replace my namenode to
something more reliable. Is there an easy way to do this?

I am thinking of rsycing the namenode metadata (not sure where it is,
but I am hoping someone guide me to that) to the new node and have all
datanode's conf/core-site.xml point to new node.

That should be it, correct?

TIA

From general-return-1463-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 15:26:26 2010
Return-Path: <general-return-1463-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34761 invoked from network); 26 Apr 2010 15:26:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 15:26:26 -0000
Received: (qmail 84950 invoked by uid 500); 26 Apr 2010 15:26:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84869 invoked by uid 500); 26 Apr 2010 15:26:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84861 invoked by uid 99); 26 Apr 2010 15:26:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 15:26:24 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of gautam.singaraju@gmail.com designates 209.85.222.189 as permitted sender)
Received: from [209.85.222.189] (HELO mail-pz0-f189.google.com) (209.85.222.189)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 15:26:19 +0000
Received: by pzk27 with SMTP id 27so9562151pzk.2
        for <general@hadoop.apache.org>; Mon, 26 Apr 2010 08:25:59 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=m/5prcw+jqaZ8hHrr1k5U5DIvm8zCQ6ClwZQ86IY6Go=;
        b=vMGTzJE4SCefZHJBnI3i0hJ9/gR9uJcHmrcVKW7A81OI3NxlvQAk3dxgZeScZHAbNg
         /WgqDjXLcUESg7wiGoG3Euj4QmOio5S/5e1smE7tWeeR5h0oIXmSiuhuK/UrazLcBwOS
         EpSF6qGA5T4QE/afH9Y0C+US8qhxYich93Sd4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=x5IZ7F8Zkl1JePf5SqHncsJX6TB7R4caw0UJWM26Fjys+SuqM9G6UyTkjifzxq8x++
         F9Vp22II7PJf9VQqJ2v8QNRpZkk1l5m5RyTDHxggfxRuBM1ZjcYaU2URaovF/l93atEx
         iCfqIzGhFqtGYK+KLmwoYL7QC7xKSE76r/wAk=
MIME-Version: 1.0
Received: by 10.114.237.2 with SMTP id k2mr4422282wah.214.1272295558221; Mon, 
	26 Apr 2010 08:25:58 -0700 (PDT)
Received: by 10.150.230.19 with HTTP; Mon, 26 Apr 2010 08:25:58 -0700 (PDT)
In-Reply-To: <r2z23d54121004222129gcc126a5egc00ff7fbb3d0c7bf@mail.gmail.com>
References: <g2jf8b2ccd41004222118kb9c46c54oa7bc835bc0b5fcfd@mail.gmail.com>
	 <r2z23d54121004222129gcc126a5egc00ff7fbb3d0c7bf@mail.gmail.com>
Date: Mon, 26 Apr 2010 11:25:58 -0400
Message-ID: <h2nf8b2ccd41004260825h339e8768n968547ced0628918@mail.gmail.com>
Subject: Re: DBOutputFormat over SSH?
From: Gautam Singaraju <gautam.singaraju@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

I should take a look at Vaidya for performance analysis, but yes,
inserting using LOAD DATA LOCAL INFILE is got to be the fastest. My
numbers show its atleast 9-10 times faster than the next mechanism. I
am loading to an empty table, without much indexes; otherwise, I might
need to disable indexing and re-enable indexing after the load data.

I am yet to check out Hivo; and got a couple of questions. For using
inload functionality of MySQL, I had to copy the results to local and
sequentially loaded it to MYSQL. Does Hivo perform a parallel Load
Data Local? Does the reducers perform this task upon close? That would
mean multiple connections to the DB and could be faster.

Thanks!
---
Gautam



On Fri, Apr 23, 2010 at 12:29 AM, Eric Sammer <esammer@cloudera.com> wrote:
> In general, you'll want to avoid tunneling permanent production code
> over ssh tunnels. They're flaky and do not recover from network
> interruption in any reasonable way. If you need to do this, a vpn is
> the correct approach. Linux easily will do ipsec p2p tunnels that are
> reasonably secure. If you really only have port 22 then I suppose
> that's your only option but I really would reevaluate the security
> policy.
>
> Either way, it's going to be slow due to the encryption overhead but
> if it's a small amount of data, that may be fine.
>
> On Fri, Apr 23, 2010 at 12:18 AM, Gautam Singaraju
> <gautam.singaraju@gmail.com> wrote:
>> All,
>>
>> I have a use-case where I need to crunch a large amount of data and
>> push to the results (comparatively a smaller set) to a mysql db at a
>> remote location. As per security concerns, only SSH ports are open. I
>> tried using Java Secure Channel [1] in combination with some custom
>> JDBC code from the reducers.
>>
>> Can anyone comment on the performance of DBOutputFormat? Have there
>> been any efforts to tunnel this through SSH? This is going to be an
>> expensive operation; any suggestions would be welcome.
>>
>> [1] http://www.jcraft.com/jsch/
>> ---
>> Gautam Singaraju
>>
>
>
>
> --
> Eric Sammer
> phone: +1-917-287-2675
> twitter: esammer
> data: www.cloudera.com
>

From general-return-1464-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 15:27:15 2010
Return-Path: <general-return-1464-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34901 invoked from network); 26 Apr 2010 15:27:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 15:27:15 -0000
Received: (qmail 86156 invoked by uid 500); 26 Apr 2010 15:27:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86083 invoked by uid 500); 26 Apr 2010 15:27:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86075 invoked by uid 99); 26 Apr 2010 15:27:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 15:27:14 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of gautam.singaraju@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 15:27:06 +0000
Received: by pwi7 with SMTP id 7so8713210pwi.35
        for <general@hadoop.apache.org>; Mon, 26 Apr 2010 08:26:45 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=gMpNK7LDpR3s1lKbp9cd0oS7r21vt64M20cX93pPr3w=;
        b=lPUL3rxu1QsfHXJOXS9wsRiPSBVYzWgmyUIlM2t3RdCfXoPsRcs7ljMqCXJbDZco9p
         Gn6g71ffuA4gBEKPjJcZ21rEuuXWvmtXu7vsOJiF1Z+qvs3+bK9YCmXXnFhXBiWdi0Fo
         70BiNJ564+O1M+W0pTSJONk0V4sZWXq9glzl0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Oi06YW6KsmMj4PdAW31mlsQHygo3hKuKOvEgz2JCHMdykKKobqTFhV6ZT+rcXS+dYS
         XFRkePQs/qTjIDYx46ycjgauY7R/oxe/jNkEIby/ayzbX5B80BvKgv4LDzEg/U60QV2X
         jdMYZZQWgQwwiNojUxxwL2lW6wW7Jn/JZlyaA=
MIME-Version: 1.0
Received: by 10.142.67.30 with SMTP id p30mr1934250wfa.154.1272295605368; Mon, 
	26 Apr 2010 08:26:45 -0700 (PDT)
Received: by 10.150.230.19 with HTTP; Mon, 26 Apr 2010 08:26:45 -0700 (PDT)
In-Reply-To: <l2pf8b2ccd41004242146kea04a168l8721f6e14656c1a6@mail.gmail.com>
References: <l2pf8b2ccd41004242146kea04a168l8721f6e14656c1a6@mail.gmail.com>
Date: Mon, 26 Apr 2010 11:26:45 -0400
Message-ID: <z2lf8b2ccd41004260826g7607c066k639667ee9bc7ff50@mail.gmail.com>
Subject: Re: jobclient.getFS throwing an NullPointerException
From: Gautam Singaraju <gautam.singaraju@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Figured it out, thanks!
---
Gautam



On Sun, Apr 25, 2010 at 12:46 AM, Gautam Singaraju
<gautam.singaraju@gmail.com> wrote:
> All,
>
> After a reduce task, I would like to have a handle to
> DistributedFileSystem. We have the deprecated hadoop-site.xml instead
> of core-site.xml, mapred-site.xml and hdfs-site.xml. When I perform a
> jobclient.getFS() it throws a NullPointerException.
>
> I am not sure if the configuration is an issue. Any tips would be appreciated?
> ---
> Gautam
>

From general-return-1465-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 15:43:39 2010
Return-Path: <general-return-1465-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43960 invoked from network); 26 Apr 2010 15:43:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 15:43:39 -0000
Received: (qmail 23619 invoked by uid 500); 26 Apr 2010 15:43:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23566 invoked by uid 500); 26 Apr 2010 15:43:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23558 invoked by uid 99); 26 Apr 2010 15:43:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 15:43:38 +0000
X-ASF-Spam-Status: No, hits=-2.2 required=10.0
	tests=AWL,RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 15:43:29 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 38DD71BA4EE
	for <general@hadoop.apache.org>; Mon, 26 Apr 2010 16:43:06 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id 9kNgLLJvcYUV for <general@hadoop.apache.org>;
	Mon, 26 Apr 2010 16:42:56 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by colossus.hpl.hp.com (Postfix) with ESMTPS id 11D4E1BA331
	for <general@hadoop.apache.org>; Mon, 26 Apr 2010 16:42:55 +0100 (BST)
MailScanner-NULL-Check: 1272901364.63208@3UFQFH87AGtPpNGI8IZmNA
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o3QFghcK017659
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Mon, 26 Apr 2010 16:42:44 +0100 (BST)
Message-ID: <4BD5B473.90807@apache.org>
Date: Mon, 26 Apr 2010 16:42:43 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100228)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSS] secure 0.20-based branch
References: <4BD1DFEA.5020908@apache.org>
In-Reply-To: <4BD1DFEA.5020908@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o3QFghcK017659
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

Doug Cutting wrote:
> Y!'s developed extensive security features based on the 0.20 branch. The 
> 0.20 versions of the individual patches appear in Jira, but these have 
> not been committed to any branch in Apache's SVN.  Y! has periodically 
> pushed out versions of these as Yahoo!'s Distribution of Hadoop at 
> github, and Cloudera is likely to make a 0.20-based distribution 
> including these as well.
> 
> Shouldn't we commit these all to some 0.20-based branch at Apache?  I'd 
> earlier (on common-dev) suggested we might start a 1.0 branch based on 
> 0.20, then add a 1.1 branch with the security patches.  If that were 
> done, the 0.21 release could perhaps instead be called 1.2.  But, 
> regardless of the naming, it would be good to have the 0.20 versions of 
> all of the security patches committed to a branch at Apache so that we 
> can make a release that includes them, patches can be targeted against 
> this branch, etc.
> 
> What do others think?

I'm biased as I don't have that much data in HDFS right now, but I'm 
mostly in favour of SVN_TRUNK as where things go, not adding new 
features to the existing stuff

From general-return-1466-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 16:46:39 2010
Return-Path: <general-return-1466-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68313 invoked from network); 26 Apr 2010 16:46:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 16:46:38 -0000
Received: (qmail 29215 invoked by uid 500); 26 Apr 2010 16:46:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29132 invoked by uid 500); 26 Apr 2010 16:46:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29121 invoked by uid 99); 26 Apr 2010 16:46:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 16:46:37 +0000
X-ASF-Spam-Status: No, hits=-2.2 required=10.0
	tests=AWL,RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 16:46:29 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 611481BA682
	for <general@hadoop.apache.org>; Mon, 26 Apr 2010 17:46:07 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id 7Q61Mko0gSNa for <general@hadoop.apache.org>;
	Mon, 26 Apr 2010 17:45:57 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by colossus.hpl.hp.com (Postfix) with ESMTPS id 600A71BA2B8
	for <general@hadoop.apache.org>; Mon, 26 Apr 2010 17:45:57 +0100 (BST)
MailScanner-NULL-Check: 1272905145.79199@ADbWXsv9qU6NMasUkN8BTw
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o3QGjiX8018963
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Mon, 26 Apr 2010 17:45:45 +0100 (BST)
Message-ID: <4BD5C338.3030602@apache.org>
Date: Mon, 26 Apr 2010 17:45:44 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100228)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: dfs.data.dir
References: <l2k1cbd6f831004210409red34ff40pbe3d42771aa645a8@mail.gmail.com> <m2pdfe484f01004211433l61ce4be6rdfe94ebc3bc24064@mail.gmail.com> <4BD04400.90800@apache.org> <51FEB071-B163-4B01-A08A-EB4DD9645923@linkedin.com>
In-Reply-To: <51FEB071-B163-4B01-A08A-EB4DD9645923@linkedin.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o3QGjiX8018963
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

Allen Wittenauer wrote:
> On Apr 22, 2010, at 5:41 AM, Steve Loughran wrote:
>> that brings up a couple of issues I've been thinking about now that workers can go to 6+ HDDs/node
>>
>> * a way to measure the distribution across disks, rather than just nodes. DfsClient doesn't provide enough info here yet.
> 
> What should probably happen is that instead of throwing you to the file browser, clicking on a host from the live nodes page should probably put you on a "stats about this node" page.

I don't want to do any of this by hand. I want machine readable content 
something can aggregate over time.

> 
>> * a way to triger some rebalancing on a single node, to say "position stuff more fairly". You don't need to worry about network traffic, just local disk load and CPU time, so it should be simpler.
> 
> 
> Yup.  Working with 8 drives per node, it is interesting to see how unbalanced the data gets after a while.  [Luckily, we have MR tmp space segregated off so I'm sure it would be a lot worse if we didn't!]
> 
> Someone should file a jira. :)

Especially if someone else offers to fix it.


From general-return-1467-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 16:54:42 2010
Return-Path: <general-return-1467-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73761 invoked from network); 26 Apr 2010 16:54:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 16:54:41 -0000
Received: (qmail 38942 invoked by uid 500); 26 Apr 2010 16:54:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38835 invoked by uid 500); 26 Apr 2010 16:54:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 38827 invoked by uid 99); 26 Apr 2010 16:54:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 16:54:40 +0000
X-ASF-Spam-Status: No, hits=-0.2 required=10.0
	tests=AWL,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 16:54:35 +0000
Received: from [10.73.145.24] (travelsoon-lm.corp.yahoo.com [10.73.145.24])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o3QGrkfM082614
	for <general@hadoop.apache.org>; Mon, 26 Apr 2010 09:53:46 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=gIZmSE8UCr1hGb8Seae5tNZ9ThELfQ5HFQ0Ex7IIh+vxMJPtiNGypiI4W/GBl6ST
Message-ID: <4BD5C516.3090408@yahoo-inc.com>
Date: Mon, 26 Apr 2010 09:53:42 -0700
From: Jakob Homan <jhoman@yahoo-inc.com>
User-Agent: Thunderbird 2.0.0.12 (Macintosh/20080213)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSS] secure 0.20-based branch
References: <4BD1DFEA.5020908@apache.org> <4BD5B473.90807@apache.org>
In-Reply-To: <4BD5B473.90807@apache.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

I can't see any way in which staying mired in developing on 20 rather 
than trunk is beneficial to the long-term health of the project. 20.0 
was released in late April, 2009 and we've not had a major release since 
then, which is hurting the project significantly. Packaging and testing 
a security-enabled release is very difficult, let me assure you, and 
will require a significant, concerted effort that would take away a huge 
number of cycles from releasing 21 - whatever form 21 takes - and fixing 
bugs on trunk.  We're in the process of moving all the work we did on 
Y!'s 20 to trunk so that we can have a great 21 (&& 22).

-jakob


From general-return-1468-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 20:44:58 2010
Return-Path: <general-return-1468-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 18419 invoked from network); 26 Apr 2010 20:44:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 20:44:58 -0000
Received: (qmail 2972 invoked by uid 500); 26 Apr 2010 20:44:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2910 invoked by uid 500); 26 Apr 2010 20:44:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2902 invoked by uid 99); 26 Apr 2010 20:44:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 20:44:56 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=AWL,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of asrabkin@gmail.com designates 209.85.221.193 as permitted sender)
Received: from [209.85.221.193] (HELO mail-qy0-f193.google.com) (209.85.221.193)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 20:44:50 +0000
Received: by qyk31 with SMTP id 31so17305353qyk.20
        for <general@hadoop.apache.org>; Mon, 26 Apr 2010 13:44:29 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=qbV/RENCRYouTWocnhgbF7dKxE9fW3Pyls2vgB99WeM=;
        b=wJIkj5oRa9OX9dmBiFMwsnXmZmNXJg9BTMke3FcL/iTY6FZ3c0woDXj8+pd/4YDMCw
         11GZdVnggr3t/KFyVuSYzbRwtIPYtO+H4ahEZukFpJ7qikeNcb6vsoIL7H5x37NyxGhh
         C1D7rcMG3q9V5Mm4q15o3SAn1bi5rnJvwsc/o=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=PU7IAgjg3JKimo17Q2ndtfymLkXMpXPiAst9LvGZ+afhjMO23qle1LAECe4V3LM5JH
         2V4iAyJSklN3zpyhcE/gu+Bbk+TLlTu+3oXZ3Fu8wIyVPugDJy8q/vphWaVma1eMDWNS
         o41PyHwA1a9H3h79zHPA2ev5jeWsGO8gYoulE=
MIME-Version: 1.0
Received: by 10.229.214.7 with SMTP id gy7mr5595963qcb.12.1272314669335; Mon, 
	26 Apr 2010 13:44:29 -0700 (PDT)
Received: by 10.229.82.17 with HTTP; Mon, 26 Apr 2010 13:44:29 -0700 (PDT)
Date: Mon, 26 Apr 2010 13:44:29 -0700
Message-ID: <m2r39b0afc01004261344mb12be54yba790f8f849eaa82@mail.gmail.com>
Subject: ANNOUNCE: Chukwa 0.4 released
From: Ariel Rabkin <asrabkin@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

The Chukwa team is pleased to announce the release of Chukwa 0.4.0,
our second public release.

Chukwa is a log collection framework designed to facilitate
large-scale log storage and processing with Hadoop. Chukwa  has been
tested at scale and used in several production settings, and is
reasonably robust and well behaved.

- This release fixes many bugs, improves documentation, and adds
several more collection tools, such as the ability to collect UDP
packets.
- There is one important backwards-incompatible change. Instead of
separate scripts for starting Chukwa agents, collectors, etc, there is
now a single bin/chukwa taking a mode parameter.

See http://hadoop.apache.org/chukwa/docs/r0.4.0/ for an overview of
Chukwa and http://www.apache.org/dyn/closer.cgi/hadoop/chukwa/ for
download links. And feel free to email chukwa-user@hadoop.apache.org with
any questions or concerns.


--Ari

-- 
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department

From general-return-1469-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 05:31:43 2010
Return-Path: <general-return-1469-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78125 invoked from network); 27 Apr 2010 05:31:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 05:31:43 -0000
Received: (qmail 44750 invoked by uid 500); 27 Apr 2010 05:31:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44491 invoked by uid 500); 27 Apr 2010 05:31:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44483 invoked by uid 99); 27 Apr 2010 05:31:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 05:31:41 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.221.193] (HELO mail-qy0-f193.google.com) (209.85.221.193)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 05:31:35 +0000
Received: by qyk31 with SMTP id 31so18063854qyk.20
        for <general@hadoop.apache.org>; Mon, 26 Apr 2010 22:31:14 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.191.1 with SMTP id dk1mr6626819qcb.18.1272346274547; Mon, 
	26 Apr 2010 22:31:14 -0700 (PDT)
Received: by 10.229.22.140 with HTTP; Mon, 26 Apr 2010 22:31:14 -0700 (PDT)
Date: Mon, 26 Apr 2010 22:31:14 -0700
Message-ID: <q2u17d463271004262231i6913dd22x6c52d66719b5880a@mail.gmail.com>
Subject: MR & Hadoop industry survey, make your voice heard
From: John Kreisa <john@cloudera.com>
To: general <general@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=0016363b85f6b9c9cf048531336e
X-Virus-Checked: Checked by ClamAV on apache.org

--0016363b85f6b9c9cf048531336e
Content-Type: text/plain; charset=ISO-8859-1

Greetings Hadoop community,

We know there are many exciting uses of Hadoop out there in a wide range of
industries and now there is an opportunity to make your voice heard. Leading
industry analyst the 451 Group is running a brief survey where the purpose
is to collect information on how enterprise IT organizations use MapReduce
and IT management log search technologies. There are only 8 questions on
this survey and one of the choices of MapReduce implementations is Hadoop.

I wanted to make you aware of the survey and ask you to help make sure that
Hadoop is represented by taking a few minutes to take part in it.

 http://www.surveymonkey.com/s/D6QCV3W

Cheers,

John

-- 
John Kreisa
VP Marketing
Cloudera
phone: +1 (408) 425 7600
blog: cloudera.com/blog
twitter: twitter.com/cloudera
web: www.cloudera.com

--0016363b85f6b9c9cf048531336e--

From general-return-1470-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 21:17:46 2010
Return-Path: <general-return-1470-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93130 invoked from network); 27 Apr 2010 21:17:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 21:17:46 -0000
Received: (qmail 92400 invoked by uid 500); 27 Apr 2010 21:17:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92346 invoked by uid 500); 27 Apr 2010 21:17:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 92338 invoked by uid 99); 27 Apr 2010 21:17:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 21:17:44 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [64.78.17.18] (HELO EXHUB018-3.exch018.msoutlookonline.net) (64.78.17.18)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 21:17:35 +0000
Received: from EXVMBX018-1.exch018.msoutlookonline.net ([64.78.17.47]) by
 EXHUB018-3.exch018.msoutlookonline.net ([64.78.17.18]) with mapi; Tue, 27 Apr
 2010 14:17:14 -0700
From: Scott Carey <scott@richrelevance.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 27 Apr 2010 14:15:52 -0700
Subject: Re: [DISCUSS] secure 0.20-based branch
Thread-Topic: [DISCUSS] secure 0.20-based branch
Thread-Index: AcrmTwGIRLd5JJRARPWlONBwHtpDTg==
Message-ID: <F7699C4E-25FF-48EF-930C-E82329B10459@richrelevance.com>
References: <4BD1DFEA.5020908@apache.org>
 <792D1884-3AF7-4087-AC78-B1872D895138@linkedin.com>
 <w2i1267dd3b1004231235k6a58b298wca4d3f014822f788@mail.gmail.com>
 <o2g1267dd3b1004231335y40ea2b48i23647e6272a18679@mail.gmail.com>
 <4BD20869.3060304@apache.org>
 <m2q4aa34eb71004231430od3439bcehd74d72f4247543ad@mail.gmail.com>
 <4BD22722.5060009@apache.org>
 <w2n1267dd3b1004232238i3a848f33xeabade312fcc22dd@mail.gmail.com>
In-Reply-To: <w2n1267dd3b1004232238i3a848f33xeabade312fcc22dd@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


On Apr 23, 2010, at 10:38 PM, Chris Douglas wrote:

>> I'm not proposing any more back- or forward-porting than will be done an=
yway.
>>=20
>=20
> Because trunk is the shared repository that contains the security
> work. And a working append. And dozens of smaller, but important
> features including the 1.0 APIs. Symlinks. Optimizations to the
> shuffle. Splittable bzip compression. Stability and scalability fixes
> to the NameNode and JobTracker. Unicorns and happiness.
>=20

I'm for anything that gets all the goodies above out in a release.  I don't=
 care if they all get in one release or if its spread out over 2 or 3.
Right now, about 1/4 of the above (e.g. happiness, but no unicorns) is in C=
DH2/3.  Trunk has stalled, getting new --  CORE -- features requires using =
other branches.=20

Although I would like to see the changes that these other branches have in =
apache's SVN, they belong in trunk.  0.20 is old already.  Its the old, sta=
ble branch now and new stuff should go into newer releases.  I've been wait=
ing for things like the Shuffle refactor (30% performance improvement for s=
ome of my job flows) for a long time.

Just because Y! is not going to upgrade their deployment past their branch =
for a long time does not mean the rest of the community has to wait.  I liv=
ed on 0.19.2 in production until very recently -- it became a solid branch =
without Y! or Facebook.  Without the same testing muscle, it might take 1 o=
r two more minor releases to stabilize, but the community's release schedul=
e IMO desperately needs to become more independent of the biggest players. =
=20

Trunk should be moved forward and incorporate Cloudera and Yahoo's improvem=
ents aggressively.  Its OK to have a 0.x.0 release that isn't completely st=
able yet, or backed by the biggest users.   It is important to incorporate =
improvements made by productive contributors into actual releases in a time=
ly fashion, or else those contributors will roll their own versions and eve=
ntually diverge significantly from the community rather than wait to get va=
lue from their work.


> Stabilizing, packaging, and testing trunk is drudgery, but it can be shar=
ed.
>=20
> I can see the value in restarting collaboration between major
> contributors by reestablishing a common branch, and 0.20 will probably
> be more successful in that respect, at least earlier. However, I
> continue to oppose sinking combined energy into 0.20 at the expense of
> trunk, for reasons already discussed at length. -C
>=20

I would love to see an apache release with new, useful features and enhance=
ments.  That could be a 0.20 with all or most of the Y! and Cloudera stuff =
in there.  However, if any such effort slows down progress on trunk -- forg=
et it.  Get a 0.21 or 0.22 out with whatever features are ready, and move t=
he ball forward on trunk.   We should not encourage 0.20 to live forever. =
=20

0.21 and 0.22 should be releases that are compelling enough for Y!, Clouder=
a, and anyone else with their own customizations to want to move to for the=
ir own sake.


>> Doug
>>=20


From general-return-1471-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 15:36:01 2010
Return-Path: <general-return-1471-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60929 invoked from network); 28 Apr 2010 15:36:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 15:36:01 -0000
Received: (qmail 21240 invoked by uid 500); 28 Apr 2010 15:36:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21185 invoked by uid 500); 28 Apr 2010 15:35:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21177 invoked by uid 99); 28 Apr 2010 15:35:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 15:35:59 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=AWL,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of sonalgoyal4@gmail.com designates 209.85.223.174 as permitted sender)
Received: from [209.85.223.174] (HELO mail-iw0-f174.google.com) (209.85.223.174)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 15:35:54 +0000
Received: by iwn4 with SMTP id 4so6494886iwn.31
        for <general@hadoop.apache.org>; Wed, 28 Apr 2010 08:35:34 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=6kPvPsLlATLHCgkmm9rZNQ0x9ZWAUeRmLAytVm/bAHI=;
        b=UA4rSP3eedXH8GkElSqx5+W2pLw3zHClNH+/z8Nw7PViBwyYcu+3ftexBfSgM6sft5
         kxzQH9u3Wupv5OSS0IfOmFyrLaxgfXJC2u97xDlz1mbspwRN+spogPJ7GuA657g0WYom
         QNTB2LzOwryuQr0kmBc8zCMNCJo4ESYE+Vr6w=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=hK8IkPMnBEYs1yHz3OaBWXCX8rew62gqRSBlIVCl6RJ9yZGFFM62+P62v7S8vXo2V1
         OjLdzBEON27Mtn9voj70Rpms88jNJaKSfsoMiuIliHHG6BSaUA3kqTQFiKxheMIexo8S
         QgURN4k3uLUcrjbgcTna6PO57zMo5mP+5mel4=
MIME-Version: 1.0
Received: by 10.142.247.16 with SMTP id u16mr4543894wfh.217.1272468932797; 
	Wed, 28 Apr 2010 08:35:32 -0700 (PDT)
Received: by 10.142.211.14 with HTTP; Wed, 28 Apr 2010 08:35:32 -0700 (PDT)
In-Reply-To: <h2nf8b2ccd41004260825h339e8768n968547ced0628918@mail.gmail.com>
References: <g2jf8b2ccd41004222118kb9c46c54oa7bc835bc0b5fcfd@mail.gmail.com>
	 <r2z23d54121004222129gcc126a5egc00ff7fbb3d0c7bf@mail.gmail.com>
	 <h2nf8b2ccd41004260825h339e8768n968547ced0628918@mail.gmail.com>
Date: Wed, 28 Apr 2010 21:05:32 +0530
Message-ID: <l2m9f26e661004280835j13568ce8p9d4817d21d43f49c@mail.gmail.com>
Subject: Re: DBOutputFormat over SSH?
From: Sonal Goyal <sonalgoyal4@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502cc65ba24dd04854dc215

--00504502cc65ba24dd04854dc215
Content-Type: text/plain; charset=ISO-8859-1

Gautam,

Answers inline.

Thanks and Regards,
Sonal
www.meghsoft.com


On Mon, Apr 26, 2010 at 8:55 PM, Gautam Singaraju <
gautam.singaraju@gmail.com> wrote:

> I should take a look at Vaidya for performance analysis, but yes,
> inserting using LOAD DATA LOCAL INFILE is got to be the fastest. My
> numbers show its atleast 9-10 times faster than the next mechanism. I
> am loading to an empty table, without much indexes; otherwise, I might
> need to disable indexing and re-enable indexing after the load data.
>
> I am yet to check out Hivo; and got a couple of questions. For using
> inload functionality of MySQL, I had to copy the results to local and
> sequentially loaded it to MYSQL. Does Hivo perform a parallel Load
> Data Local? Does the reducers perform this task upon close? That would
> mean multiple connections to the DB and could be faster.
>
> yes, parallel load data local. HDFS files are loaded in parallel to the
MySQL database. For efficiency, loading is in map only job. Number of
mappers are determined by number of files.


> Thanks!
> ---
> Gautam
>
>
>
> On Fri, Apr 23, 2010 at 12:29 AM, Eric Sammer <esammer@cloudera.com>
> wrote:
> > In general, you'll want to avoid tunneling permanent production code
> > over ssh tunnels. They're flaky and do not recover from network
> > interruption in any reasonable way. If you need to do this, a vpn is
> > the correct approach. Linux easily will do ipsec p2p tunnels that are
> > reasonably secure. If you really only have port 22 then I suppose
> > that's your only option but I really would reevaluate the security
> > policy.
> >
> > Either way, it's going to be slow due to the encryption overhead but
> > if it's a small amount of data, that may be fine.
> >
> > On Fri, Apr 23, 2010 at 12:18 AM, Gautam Singaraju
> > <gautam.singaraju@gmail.com> wrote:
> >> All,
> >>
> >> I have a use-case where I need to crunch a large amount of data and
> >> push to the results (comparatively a smaller set) to a mysql db at a
> >> remote location. As per security concerns, only SSH ports are open. I
> >> tried using Java Secure Channel [1] in combination with some custom
> >> JDBC code from the reducers.
> >>
> >> Can anyone comment on the performance of DBOutputFormat? Have there
> >> been any efforts to tunnel this through SSH? This is going to be an
> >> expensive operation; any suggestions would be welcome.
> >>
> >> [1] http://www.jcraft.com/jsch/
> >> ---
> >> Gautam Singaraju
> >>
> >
> >
> >
> > --
> > Eric Sammer
> > phone: +1-917-287-2675
> > twitter: esammer
> > data: www.cloudera.com
> >
>

--00504502cc65ba24dd04854dc215--

From general-return-1472-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 15:41:52 2010
Return-Path: <general-return-1472-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63762 invoked from network); 28 Apr 2010 15:41:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 15:41:52 -0000
Received: (qmail 32636 invoked by uid 500); 28 Apr 2010 15:41:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32570 invoked by uid 500); 28 Apr 2010 15:41:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32562 invoked by uid 99); 28 Apr 2010 15:41:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 15:41:51 +0000
X-ASF-Spam-Status: No, hits=0.2 required=10.0
	tests=AWL,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.27.211] (HELO qmta11.emeryville.ca.mail.comcast.net) (76.96.27.211)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 15:41:42 +0000
Received: from omta04.emeryville.ca.mail.comcast.net ([76.96.30.35])
	by qmta11.emeryville.ca.mail.comcast.net with comcast
	id B1pK1e0050lTkoCAB3hPcP; Wed, 28 Apr 2010 15:41:23 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta04.emeryville.ca.mail.comcast.net with comcast
	id B3hC1e0012SbwD58Q3hEDG; Wed, 28 Apr 2010 15:41:20 +0000
Cc: general@hadoop.apache.org
Message-Id: <6C75A0AB-36E2-4E78-850E-F025140CE4F1@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: Owen O'Malley <omalley@apache.org>
In-Reply-To: <C3B2314E-7FC9-4301-AFE2-AFECEA6333B6@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Should we approve the Hadoop security logo?
Date: Wed, 28 Apr 2010 08:41:10 -0700
References: <C3B2314E-7FC9-4301-AFE2-AFECEA6333B6@apache.org>
X-Mailer: Apple Mail (2.936)

On Apr 22, 2010, at 2:36 PM, Owen O'Malley wrote:

> All,
>  Should we allow use of the Hadoop security logo (hadoop-security- 
> logo.jpg) from HADOOP-6721. Clearly, I'm +1.

With 4 +1's (including Eli's from the jira) the vote passes.

Thanks,
    Owen

From general-return-1473-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 16:26:53 2010
Return-Path: <general-return-1473-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85018 invoked from network); 28 Apr 2010 16:26:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 16:26:53 -0000
Received: (qmail 97961 invoked by uid 500); 28 Apr 2010 16:26:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97903 invoked by uid 500); 28 Apr 2010 16:26:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97895 invoked by uid 99); 28 Apr 2010 16:26:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 16:26:52 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [67.195.23.152] (HELO n1-vm0.bullet.mail.gq1.yahoo.com) (67.195.23.152)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 28 Apr 2010 16:26:43 +0000
Received: from [67.195.9.81] by n1.bullet.mail.gq1.yahoo.com with NNFMP; 28 Apr 2010 16:26:19 -0000
Received: from [67.195.9.97] by t1.bullet.mail.gq1.yahoo.com with NNFMP; 28 Apr 2010 16:26:19 -0000
Received: from [127.0.0.1] by omp101.mail.gq1.yahoo.com with NNFMP; 28 Apr 2010 16:26:19 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 569059.36010.bm@omp101.mail.gq1.yahoo.com
Received: (qmail 31385 invoked by uid 60001); 28 Apr 2010 16:26:19 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1272471979; bh=+cSMtXGzTrJ3/cn6982yHCW4ixwQhqg7ZGOPcscl2kc=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=ZmY0HMFtwmdcIJcvJ4MHxxVz5mWwW83jI0lQDONULm/+q5vywX/SFKu9AELI9XAQNHX/3SB7qLGYDrbccDzulZkAJ034eco3Yya/4DcgP51PyBN55FyZavxGtBQBrN3B3gjJ8EePiEMGMTExzdOkBgTtT1GrjymTCgZIOUt4sO8=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=FOYjRTirIQFHsSxe8lnaDXqCcWGX79gpwO08TeDdnbP1CVtAl4IlqWFyTB2y1RrjbMJJe1TRSoEua1NzKNu6jcHXsfrBserR7y2e/6XV9sQbEwC0MjWVwFcWzJHQyhuiRrv/+r3NMMqlVJga6GxNmZgS21PdW2LVrDFXgd5xQTc=;
Message-ID: <170889.24561.qm@web111809.mail.gq1.yahoo.com>
X-YMail-OSG: FujhOfYVM1mU4yOeTZGrbA5R7wirovKCgKA0zun0k9IpAYr
 8NKXI0EA6iH8GRHxkMIlpSbt41yLbAIWHDrMTpp9CHF0xA3cFLvIwaiF9M5d
 b1NLTVeCwSXi2idxiwwO8d0IYDqXES4DoglxTG2nmrMhV4dEnEZBIlg7CHqn
 kNKD8YQwpst5psNfHCdC638rXVTEKkFguRM7.trOH7TAnOxomUvzgHZD1fjj
 inIcXrjkB9FOMkpUTH2hVGDOD86QdCV74xUUm_nVh4beEyUvo1EnROOWtpv4
 Ps30n4kQpxJp1IC5sv42fLGF.432X6uTgkiWVCEzEMxOp
Received: from [198.144.195.34] by web111809.mail.gq1.yahoo.com via HTTP; Wed, 28 Apr 2010 09:26:19 PDT
X-Mailer: YahooMailClassic/10.1.11 YahooMailWebService/0.8.103.269680
Date: Wed, 28 Apr 2010 09:26:19 -0700 (PDT)
From: A Anand <ajay2000@yahoo.com>
Subject: Spreadsheet analytics product on Hadoop
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1096905557-1272471979=:24561"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-1096905557-1272471979=:24561
Content-Type: text/plain; charset=us-ascii

Datameer recently announced the beta availability of their Data Analytics Solution (DAS) product which provides spreadsheet based analytics on Hadoop. Current participants in the beta program include leaders in security, financial services, telecommunications and internet research. To learn more, check out www.datameer.com.
Ajay


      
--0-1096905557-1272471979=:24561--


From general-return-1474-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 17:33:17 2010
Return-Path: <general-return-1474-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 99022 invoked from network); 28 Apr 2010 17:33:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 17:33:15 -0000
Received: (qmail 88573 invoked by uid 500); 28 Apr 2010 17:33:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88536 invoked by uid 500); 28 Apr 2010 17:33:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88528 invoked by uid 99); 28 Apr 2010 17:33:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 17:33:14 +0000
X-ASF-Spam-Status: No, hits=-0.5 required=10.0
	tests=AWL,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [64.78.17.18] (HELO EXHUB018-3.exch018.msoutlookonline.net) (64.78.17.18)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 17:33:06 +0000
Received: from EXVMBX018-1.exch018.msoutlookonline.net ([64.78.17.47]) by
 EXHUB018-3.exch018.msoutlookonline.net ([64.78.17.18]) with mapi; Wed, 28 Apr
 2010 10:32:46 -0700
From: Scott Carey <scott@richrelevance.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 28 Apr 2010 10:31:23 -0700
Subject: Re: dfs.data.dir
Thread-Topic: dfs.data.dir
Thread-Index: Acrm+NCTosPNR3rdSMOQgjoEX0Gfug==
Message-ID: <C10C5AEA-CF09-4830-88DA-E7CBF5FF5926@richrelevance.com>
References: <l2k1cbd6f831004210409red34ff40pbe3d42771aa645a8@mail.gmail.com>
 <m2pdfe484f01004211433l61ce4be6rdfe94ebc3bc24064@mail.gmail.com>
 <4BD04400.90800@apache.org>
 <51FEB071-B163-4B01-A08A-EB4DD9645923@linkedin.com>
 <4BD5C338.3030602@apache.org>
In-Reply-To: <4BD5C338.3030602@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


On Apr 26, 2010, at 9:45 AM, Steve Loughran wrote:

> Allen Wittenauer wrote:
>> On Apr 22, 2010, at 5:41 AM, Steve Loughran wrote:
>>> that brings up a couple of issues I've been thinking about now that wor=
kers can go to 6+ HDDs/node
>>>=20
>>> * a way to measure the distribution across disks, rather than just node=
s. DfsClient doesn't provide enough info here yet.
>>=20
>> What should probably happen is that instead of throwing you to the file =
browser, clicking on a host from the live nodes page should probably put yo=
u on a "stats about this node" page.
>=20
> I don't want to do any of this by hand. I want machine readable content=20
> something can aggregate over time.
>=20
>>=20
>>> * a way to triger some rebalancing on a single node, to say "position s=
tuff more fairly". You don't need to worry about network traffic, just loca=
l disk load and CPU time, so it should be simpler.
>>=20
>>=20
>> Yup.  Working with 8 drives per node, it is interesting to see how unbal=
anced the data gets after a while.  [Luckily, we have MR tmp space segregat=
ed off so I'm sure it would be a lot worse if we didn't!]
>>=20
>> Someone should file a jira. :)
>=20
> Especially if someone else offers to fix it.
>=20

Should be trivial to at least make the new block allocation choose which de=
vice to allocate the block on with a weighted roulette algorithm instead of=
 round-robin.


From general-return-1475-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 09:29:21 2010
Return-Path: <general-return-1475-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27553 invoked from network); 29 Apr 2010 09:29:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 09:29:20 -0000
Received: (qmail 82699 invoked by uid 500); 29 Apr 2010 09:29:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82540 invoked by uid 500); 29 Apr 2010 09:29:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82526 invoked by uid 99); 29 Apr 2010 09:29:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 09:29:16 +0000
X-ASF-Spam-Status: No, hits=-2.1 required=10.0
	tests=AWL,RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 09:29:07 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 2642BB7E3A
	for <general@hadoop.apache.org>; Thu, 29 Apr 2010 10:28:44 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id FOnCwS+NZ6Uc for <general@hadoop.apache.org>;
	Thu, 29 Apr 2010 10:28:34 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 0FE46B7E39
	for <general@hadoop.apache.org>; Thu, 29 Apr 2010 10:28:33 +0100 (BST)
MailScanner-NULL-Check: 1273138102.36808@+hWZzTFY0htn5DTX/6FkEw
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o3T9SL6k022759
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Thu, 29 Apr 2010 10:28:21 +0100 (BST)
Message-ID: <4BD95145.8040204@apache.org>
Date: Thu, 29 Apr 2010 10:28:37 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100228)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: dfs.data.dir
References: <l2k1cbd6f831004210409red34ff40pbe3d42771aa645a8@mail.gmail.com> <m2pdfe484f01004211433l61ce4be6rdfe94ebc3bc24064@mail.gmail.com> <4BD04400.90800@apache.org> <51FEB071-B163-4B01-A08A-EB4DD9645923@linkedin.com> <4BD5C338.3030602@apache.org> <C10C5AEA-CF09-4830-88DA-E7CBF5FF5926@richrelevance.com>
In-Reply-To: <C10C5AEA-CF09-4830-88DA-E7CBF5FF5926@richrelevance.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o3T9SL6k022759
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

Scott Carey wrote:
> On Apr 26, 2010, at 9:45 AM, Steve Loughran wrote:
> 
>> Allen Wittenauer wrote:
>>> On Apr 22, 2010, at 5:41 AM, Steve Loughran wrote:
>>>> that brings up a couple of issues I've been thinking about now that workers can go to 6+ HDDs/node
>>>>
>>>> * a way to measure the distribution across disks, rather than just nodes. DfsClient doesn't provide enough info here yet.
>>> What should probably happen is that instead of throwing you to the file browser, clicking on a host from the live nodes page should probably put you on a "stats about this node" page.
>> I don't want to do any of this by hand. I want machine readable content 
>> something can aggregate over time.
>>
>>>> * a way to triger some rebalancing on a single node, to say "position stuff more fairly". You don't need to worry about network traffic, just local disk load and CPU time, so it should be simpler.
>>>
>>> Yup.  Working with 8 drives per node, it is interesting to see how unbalanced the data gets after a while.  [Luckily, we have MR tmp space segregated off so I'm sure it would be a lot worse if we didn't!]
>>>
>>> Someone should file a jira. :)
>> Especially if someone else offers to fix it.
>>
> 
> Should be trivial to at least make the new block allocation choose which device to allocate the block on with a weighted roulette algorithm instead of round-robin.
> 

I'd go for another plugin point with the default impl being round-robin, 
let people come up with other strategies, and add a way from DfsClient 
to measure distribution.

Other strategies could use specific knowledge about the disks -are they 
raided, can you hot swap, is the disk playing up, etc, etc, feature 
creep that I encourage people to explore.

From general-return-1476-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 09:58:36 2010
Return-Path: <general-return-1476-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35503 invoked from network); 29 Apr 2010 09:58:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 09:58:36 -0000
Received: (qmail 34471 invoked by uid 500); 29 Apr 2010 09:58:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34122 invoked by uid 500); 29 Apr 2010 09:58:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34114 invoked by uid 99); 29 Apr 2010 09:58:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 09:58:34 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.212.176] (HELO mail-px0-f176.google.com) (209.85.212.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 09:58:27 +0000
Received: by pxi10 with SMTP id 10so3683377pxi.35
        for <general@hadoop.apache.org>; Thu, 29 Apr 2010 02:58:05 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.143.87.2 with SMTP id p2mr5163300wfl.323.1272535085121; Thu, 
	29 Apr 2010 02:58:05 -0700 (PDT)
Received: by 10.142.188.16 with HTTP; Thu, 29 Apr 2010 02:58:05 -0700 (PDT)
In-Reply-To: <4BD95145.8040204@apache.org>
References: <l2k1cbd6f831004210409red34ff40pbe3d42771aa645a8@mail.gmail.com>
	 <m2pdfe484f01004211433l61ce4be6rdfe94ebc3bc24064@mail.gmail.com>
	 <4BD04400.90800@apache.org>
	 <51FEB071-B163-4B01-A08A-EB4DD9645923@linkedin.com>
	 <4BD5C338.3030602@apache.org>
	 <C10C5AEA-CF09-4830-88DA-E7CBF5FF5926@richrelevance.com>
	 <4BD95145.8040204@apache.org>
Date: Thu, 29 Apr 2010 02:58:05 -0700
Message-ID: <v2mc9118d4c1004290258mdcd620eelfd776bb979fa9452@mail.gmail.com>
Subject: Re: dfs.data.dir
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd5c884b683c504855d2953
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd5c884b683c504855d2953
Content-Type: text/plain; charset=ISO-8859-1

> I'd go for another plugin point with the default impl being round-robin,
> let people come up with other strategies, and add a way from DfsClient to
> measure distribution.
>

https://issues.apache.org/jira/browse/HDFS-1120 and
https://issues.apache.org/jira/browse/HDFS-1121

--000e0cd5c884b683c504855d2953--

From general-return-1477-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 01:14:22 2010
Return-Path: <general-return-1477-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81023 invoked from network); 30 Apr 2010 01:14:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 01:14:20 -0000
Received: (qmail 70054 invoked by uid 500); 30 Apr 2010 01:14:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69446 invoked by uid 500); 30 Apr 2010 01:14:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69020 invoked by uid 99); 30 Apr 2010 01:14:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 01:14:15 +0000
X-ASF-Spam-Status: No, hits=1.3 required=10.0
	tests=AWL,HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 01:14:08 +0000
Received: from SP1-EX07CAS02.ds.corp.yahoo.com (sp1-ex07cas02.ds.corp.yahoo.com [216.252.116.138])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o3U1DcF2011620;
	Thu, 29 Apr 2010 18:13:38 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:accept-language:content-language:x-ms-has-attach:
	x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=NAkHDklL2gCG+aQjym5QMLJG4SkRDK2DCcBa14qYy006h7l+tPqmBAzREaw+dHCI
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 SP1-EX07CAS02.ds.corp.yahoo.com ([216.252.116.167]) with mapi; Thu, 29 Apr
 2010 18:13:37 -0700
From: Dekel Tankel <dekel@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>,
        "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Date: Thu, 29 Apr 2010 18:13:36 -0700
Subject: Hadoop Summit - Call for Presentations Extended!
Thread-Topic: Hadoop Summit - Call for Presentations Extended!
Thread-Index: AcroAlvHuwAPMgtuR6quSXWd0pEzwA==
Message-ID: <46E2672895E8744A97D7577A6110858B4FE9F24798@SP1-EX07VS01.ds.corp.yahoo.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_46E2672895E8744A97D7577A6110858B4FE9F24798SP1EX07VS01ds_"
MIME-Version: 1.0

--_000_46E2672895E8744A97D7577A6110858B4FE9F24798SP1EX07VS01ds_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi

The call for presentations for Hadoop summit has been extended until *May 1=
0th*

Leverage this extended opportunity to share your experience on Hadoop and s=
ubmit abstracts here:

*http://developer.yahoo.com/events/hadoopsummit2010/presentationguidelines.=
html*

                Dekel

--_000_46E2672895E8744A97D7577A6110858B4FE9F24798SP1EX07VS01ds_--

From general-return-1478-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 19:05:13 2010
Return-Path: <general-return-1478-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 13462 invoked from network); 30 Apr 2010 19:05:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 19:05:13 -0000
Received: (qmail 5447 invoked by uid 500); 30 Apr 2010 19:05:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5235 invoked by uid 500); 30 Apr 2010 19:05:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 38641 invoked by uid 99); 30 Apr 2010 17:59:47 -0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Message-Id: <B2858AB5-EE1D-44B2-9764-F5302D24CA86@apache.org>
From: Arun C Murthy <acmurthy@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: [ANNOUNCE] Hemanth Yamijala as new member of Apache Hadoop PMC
Date: Fri, 30 Apr 2010 10:59:23 -0700
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

The Apache Hadoop PMC as voted to make Hemanth Yamijala a member of  
the PMC.

Congratulations Hemanth!

Thanks again for all your valuable contributions, we look forward to  
more from you!

Arun


From general-return-3-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 03 14:24:17 2008
Return-Path: <general-return-3-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 36762 invoked from network); 3 Jul 2008 14:24:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 3 Jul 2008 14:24:17 -0000
Received: (qmail 73048 invoked by uid 500); 3 Jul 2008 14:24:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73033 invoked by uid 500); 3 Jul 2008 14:24:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73020 invoked by uid 99); 3 Jul 2008 14:24:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jul 2008 07:24:02 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [121.0.17.65] (HELO smtp.alibaba-inc.com) (121.0.17.65)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jul 2008 14:23:10 +0000
X-IronPort-AV: E=Sophos;i="4.27,742,1204473600"; 
   d="scan'208";a="20326642"
Subject: question about bin/hadoop namenode -format
From: Yi Zhao <yi.zhao@alibaba-inc.com>
To: general@hadoop.apache.org
Content-Type: text/plain
Date: Thu, 03 Jul 2008 22:25:27 +0800
Message-Id: <1215095127.3384.6.camel@localhost.localdomain>
Mime-Version: 1.0
X-Mailer: Evolution 2.22.2 
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

when I execute the command "bin/hadoop namenode -format" get the message
below:

08/07/03 22:22:03 INFO dfs.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = zhaoyi-laptop/127.0.0.1
STARTUP_MSG:   args = [-format]
************************************************************/
Re-format filesystem in /home/zhaoyi/devel/hadoop/fs/name ? (Y or N) y
Format aborted in /home/zhaoyi/devel/hadoop/fs/name
08/07/03 22:22:08 INFO dfs.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at zhaoyi-laptop/127.0.0.1
************************************************************/

My slave is localhost
My master is localhost
I can ssh localhost without password.

what wrong with it???

any help is appreciate;

regards,
Yi


From general-return-4-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 03 15:01:05 2008
Return-Path: <general-return-4-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 49213 invoked from network); 3 Jul 2008 15:01:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 3 Jul 2008 15:01:05 -0000
Received: (qmail 40394 invoked by uid 500); 3 Jul 2008 15:01:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40375 invoked by uid 500); 3 Jul 2008 15:01:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40364 invoked by uid 99); 3 Jul 2008 15:01:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jul 2008 08:01:06 -0700
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 209.17.190.200 is neither permitted nor denied by domain of enis.soz.nutch@gmail.com)
Received: from [209.17.190.200] (HELO mail.ortakdns.com) (209.17.190.200)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jul 2008 15:00:15 +0000
Received: from [192.168.2.15] (unknown [85.105.135.220])
	by mail.ortakdns.com (Postfix) with ESMTPA id 0854C52C001
	for <general@hadoop.apache.org>; Thu,  3 Jul 2008 18:05:22 +0300 (EEST)
Message-ID: <486CE990.8000805@gmail.com>
Date: Thu, 03 Jul 2008 18:00:32 +0300
From: Enis Soztutar <enis.soz.nutch@gmail.com>
User-Agent: Thunderbird 2.0.0.14 (X11/20080502)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: question about bin/hadoop namenode -format
References: <1215095127.3384.6.camel@localhost.localdomain>
In-Reply-To: <1215095127.3384.6.camel@localhost.localdomain>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

you should you uppercase Y : )

Yi Zhao wrote:
> when I execute the command "bin/hadoop namenode -format" get the message
> below:
>
> 08/07/03 22:22:03 INFO dfs.NameNode: STARTUP_MSG: 
> /************************************************************
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG:   host = zhaoyi-laptop/127.0.0.1
> STARTUP_MSG:   args = [-format]
> ************************************************************/
> Re-format filesystem in /home/zhaoyi/devel/hadoop/fs/name ? (Y or N) y
> Format aborted in /home/zhaoyi/devel/hadoop/fs/name
> 08/07/03 22:22:08 INFO dfs.NameNode: SHUTDOWN_MSG: 
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at zhaoyi-laptop/127.0.0.1
> ************************************************************/
>
> My slave is localhost
> My master is localhost
> I can ssh localhost without password.
>
> what wrong with it???
>
> any help is appreciate;
>
> regards,
> Yi
>
>
>   

From general-return-5-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 04 06:16:01 2008
Return-Path: <general-return-5-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 17550 invoked from network); 4 Jul 2008 06:16:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 4 Jul 2008 06:16:01 -0000
Received: (qmail 75411 invoked by uid 500); 4 Jul 2008 06:16:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75394 invoked by uid 500); 4 Jul 2008 06:16:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75383 invoked by uid 99); 4 Jul 2008 06:16:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jul 2008 23:16:02 -0700
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=SPF_PASS,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [121.0.17.65] (HELO smtp.alibaba-inc.com) (121.0.17.65)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jul 2008 06:15:08 +0000
X-IronPort-AV: E=Sophos;i="4.30,300,1212336000"; 
   d="scan'208";a="20440106"
Subject: can't find any datanode after start-all
From: Yi Zhao <yi.zhao@alibaba-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <486CE990.8000805@gmail.com>
References: <1215095127.3384.6.camel@localhost.localdomain>
	 <486CE990.8000805@gmail.com>
Content-Type: text/plain; charset=UTF-8
Date: Fri, 04 Jul 2008 14:17:27 +0800
Message-Id: <1215152247.3257.12.camel@localhost.localdomain>
Mime-Version: 1.0
X-Mailer: Evolution 2.22.2 
Content-Transfer-Encoding: 8bit
X-Virus-Checked: Checked by ClamAV on apache.org

hi, all. when I try to setup a hadoop enviroment which have one master
and one slave
master: localhost
slave: 10.62.164.60

after start-all.sh, I got the information
from http://localhost:50070/dfshealth.jsp as "There are no datanodes in
the cluster" 

any help ia appreciated, thanks:D

my hadoop-site.xml is is below:
------------------------------------
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
	<property>
		<name>fs.default.name</name>
		<value>localhost:7770</value>
		<description></description>
	</property>
	<property>
		<name>mapred.job.tracker</name>
		<value>localhost:7771</value>
		<description></description>
	</property>
	<property>
		<name>hadoop.tmp.dir</name>
		<value>/home/zhaoyi/devel/hadoop/tmp</value>
		<description></description>
	</property>
	<property>
		<name>dfs.name.dir</name>
		<value>/home/zhaoyi/devel/hadoop/fs/name</value>
		<description></description>
	</property>
	<property>
		<name>dfs.data.dir</name>
		<value>/home/zhaoyi/devel/hadoop/fs/data</value>
		<description></description>
	</property>
	<property>
		<name>dfs.replication</name>
		<value>1</value>
		<description></description>
	</property>
</configuration>


From general-return-6-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 04 13:45:44 2008
Return-Path: <general-return-6-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 67955 invoked from network); 4 Jul 2008 13:45:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 4 Jul 2008 13:45:44 -0000
Received: (qmail 25199 invoked by uid 500); 4 Jul 2008 13:45:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25179 invoked by uid 500); 4 Jul 2008 13:45:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25168 invoked by uid 99); 4 Jul 2008 13:45:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jul 2008 06:45:45 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [121.0.17.65] (HELO smtp.alibaba-inc.com) (121.0.17.65)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jul 2008 13:44:53 +0000
X-IronPort-AV: E=Sophos;i="4.30,302,1212336000"; 
   d="scan'208";a="71266548"
Subject: how to disperse the data uniformly?
From: Yi Zhao <yi.zhao@alibaba-inc.com>
To: general <general@hadoop.apache.org>
Content-Type: text/plain; charset=UTF-8
Date: Fri, 04 Jul 2008 21:47:14 +0800
Message-Id: <1215179234.26802.5.camel@localhost.localdomain>
Mime-Version: 1.0
X-Mailer: Evolution 2.22.2 
Content-Transfer-Encoding: 8bit
X-Virus-Checked: Checked by ClamAV on apache.org

I have two datanode as below:
10.62.136.10
10.62.136.11
one namenode below:
10.62.136.10
master:
10.62.136.10
salves:
10.62.136.10
10.62.136.11

when I put a file 600M and a file 20M into dfs, I found that all data is
centralized in one datanode!!

how to disperse all the data to all datanode uniformly?

thanks.

my hadoop-site.xml is:
------------------------------
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
	<property>
		<name>fs.default.name</name>
		<value>10.62.142.62:7770</value>
		<description>the name of the default file system, either the literal
string "local" or a host:port for DFS.</description>
	</property>
	<property>
		<name>mapred.job.tracker</name>
		<value>10.62.142.62:7771</value>
		<description>the host and port that MapReduce job tracker runs at. if
"local", then jobs are run in-process as a single map and reduce
task.</description>
	</property>
	<property>
		<name>hadoop.tmp.dir</name>
		<value>/home/kongming/devel/hadoop/tmp</value>
		<description>a base for other temporary directories.</description>
	</property>
	<property>
		<name>dfs.name.dir</name>
		<value>/home/kongming/devel/hadoop/fs/name</value>
		<description>determines where on the local filesystem the DFS name
node should store the name table. if this is a comma-delimited list of
directories then the name table is replicated in all of the directories,
for redundancy.</description>
	</property>
	<property>
		<name>dfs.data.dir</name>
		<value>/home/kongming/devel/hadoop/fs/data</value>
		<description>determines where on the local filesystem on DFS data node
should store its blocks. if this is a comma-delimited list of
directories, then data will be stored in all named directories,
typically on different devices. derectories that do not exist are
ignored.</description>
	</property>
	<property>
		<name>dfs.replication</name>
		<value>1</value>
		<description>default block replication. the actual number of
replications can be specified when the file is created. the default is
used if replication is not specified in create time</description>
	</property>
</configuration>


From general-return-7-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 05 19:14:35 2008
Return-Path: <general-return-7-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 16277 invoked from network); 5 Jul 2008 19:14:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 5 Jul 2008 19:14:35 -0000
Received: (qmail 55781 invoked by uid 500); 5 Jul 2008 19:14:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55763 invoked by uid 500); 5 Jul 2008 19:14:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55752 invoked by uid 99); 5 Jul 2008 19:14:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jul 2008 12:14:36 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of gmackey@cs.ucf.edu designates 132.170.108.1 as permitted sender)
Received: from [132.170.108.1] (HELO longwood.cs.ucf.edu) (132.170.108.1)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jul 2008 19:13:41 +0000
Received: from longwood.cs.ucf.edu (localhost [127.0.0.1])
	by longwood.cs.ucf.edu (8.14.1/8.14.1) with ESMTP id m65JDmFc004106
	for <general@hadoop.apache.org>; Sat, 5 Jul 2008 15:13:51 -0400 (EDT)
Received: (from prayer@localhost)
	by longwood.cs.ucf.edu (8.14.1/8.14.1/Submit) id m65JDb3K004090
	for general@hadoop.apache.org; Sat, 5 Jul 2008 15:13:37 -0400 (EDT)
X-Authentication-Warning: longwood.cs.ucf.edu: prayer set sender to gmackey@cs.ucf.edu using -f
From: gmackey@cs.ucf.edu
To: general@hadoop.apache.org
Subject: Re: how to disperse the data uniformly?
Date: 05 Jul 2008 15:13:37 -0400
X-Mailer: Prayer v1.0.16
X-Originating-IP: [70.56.213.162]
In-Reply-To: <1215179234.26802.5.camel@localhost.localdomain>
Message-ID: <Prayer.1.0.16.0807051513370.3646@longwood.cs.ucf.edu>
References: <1215179234.26802.5.camel@localhost.localdomain>
Mime-Version: 1.0
Content-Type: text/plain; format=flowed; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Spam-Level: **
X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on gondor.cs.ucf.edu
X-Virus-Checked: Checked by ClamAV on apache.org
X-Old-Spam-Status: No, score=2.2 required=5.0 tests=FH_RELAY_NODNS,J_CHICKENPOX_44,
	RDNS_NONE autolearn=no version=3.2.4

for starters, you should probably change your replication value to=20
something other than 1, it kinda defeats the purpose of a parallel=20
distributed file system otherwise. What version of hadoop are you running?=
=20
you should just be able to run the command

$bin/hadoop balancer=20

to get the nodes balanced. However I don't remember what the tolerance=20
value on the balancer is for the generic hadoop-default.xml file is. Since=
=20
you only have 620MB on the cluster, then it might be within hadoop's range=
=20
of okay data distribution. Run the balancer first and see what happens\

 - Grant

On Jul 4 2008, Yi Zhao wrote:

>I have two datanode as below:
>10.62.136.10
>=EF=BB=BF10.62.136.11
>one namenode below:
>10.62.136.10
>master:
>=EF=BB=BF10.62.136.10
>salves:
>=EF=BB=BF10.62.136.10
>=EF=BB=BF10.62.136.11
>
>when I put a file 600M and a file 20M into dfs, I found that all data is
>centralized in one datanode!!
>
>how to disperse all the data to all datanode uniformly?
>
>thanks.
>
>my hadoop-site.xml is:
>------------------------------
><?xml version=3D"1.0"?>
><?xml-stylesheet type=3D"text/xsl" href=3D"configuration.xsl"?>
>
><!-- Put site-specific property overrides in this file. -->
>
><configuration>
>=09<property>
>=09=09<name>fs.default.name</name>
>=09=09<value>10.62.142.62:7770</value>
>=09=09<description>the name of the default file system, either the literal
>string "local" or a host:port for DFS.</description>
>=09</property>
>=09<property>
>=09=09<name>mapred.job.tracker</name>
>=09=09<value>10.62.142.62:7771</value>
>=09=09<description>the host and port that MapReduce job tracker runs at. i=
f
>"local", then jobs are run in-process as a single map and reduce
>task.</description>
>=09</property>
>=09<property>
>=09=09<name>hadoop.tmp.dir</name>
>=09=09<value>/home/kongming/devel/hadoop/tmp</value>
>=09=09<description>a base for other temporary directories.</description>
>=09</property>
>=09<property>
>=09=09<name>dfs.name.dir</name>
>=09=09<value>/home/kongming/devel/hadoop/fs/name</value>
>=09=09<description>determines where on the local filesystem the DFS name
>node should store the name table. if this is a comma-delimited list of
>directories then the name table is replicated in all of the directories,
>for redundancy.</description>
>=09</property>
>=09<property>
>=09=09<name>dfs.data.dir</name>
>=09=09<value>/home/kongming/devel/hadoop/fs/data</value>
>=09=09<description>determines where on the local filesystem on DFS data no=
de
>should store its blocks. if this is a comma-delimited list of
>directories, then data will be stored in all named directories,
>typically on different devices. derectories that do not exist are
>ignored.</description>
>=09</property>
>=09<property>
>=09=09<name>dfs.replication</name>
>=09=09<value>1</value>
>=09=09<description>default block replication. the actual number of
>replications can be specified when the file is created. the default is
>used if replication is not specified in create time</description>
>=09</property>
></configuration>
>

--=20
Grant Mackey
UCF Researcher
Eng. III Rm238


From general-return-8-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 14 10:11:33 2008
Return-Path: <general-return-8-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 36788 invoked from network); 14 Jul 2008 10:11:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 14 Jul 2008 10:11:33 -0000
Received: (qmail 84212 invoked by uid 500); 14 Jul 2008 10:11:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84172 invoked by uid 500); 14 Jul 2008 10:11:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84151 invoked by uid 99); 14 Jul 2008 10:11:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Jul 2008 03:11:33 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [121.0.17.65] (HELO smtp.alibaba-inc.com) (121.0.17.65)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Jul 2008 10:10:40 +0000
X-IronPort-AV: E=Sophos;i="4.30,358,1212336000"; 
   d="scan'208";a="21830280"
Subject: how to distribute the data to all the datanodes?
From: Yi Zhao <yi.zhao@alibaba-inc.com>
To: general <general@hadoop.apache.org>
Content-Type: text/plain; charset=UTF-8
Date: Mon, 14 Jul 2008 18:12:12 +0800
Message-Id: <1216030332.3992.19.camel@localhost.localdomain>
Mime-Version: 1.0
X-Mailer: Evolution 2.22.2 
Content-Transfer-Encoding: 8bit
X-Virus-Checked: Checked by ClamAV on apache.org

hi, all
I have a hadoop cluster which have one master and three datanodes.

I want to put a local file about 128M intpu hdfs, I have set the
block-size to 10M

when I set the replication to 0,
I found that all the data distributed to the node which I execute the
command 'bin/hadoop dfs -put file.gz input', so this node's disk space
is used about 128M, but other nodes has no disk space used.

when I set the replication to 3,
I found that every nodes have the same data, so every nodes is about
128M disk space used.

what should I do? I'm using hadoop-0.15.2.

any one can help me?

thanks.


From general-return-9-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 15 17:30:11 2008
Return-Path: <general-return-9-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 77023 invoked from network); 15 Jul 2008 17:30:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 15 Jul 2008 17:30:09 -0000
Received: (qmail 11571 invoked by uid 500); 15 Jul 2008 17:30:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11530 invoked by uid 500); 15 Jul 2008 17:30:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11519 invoked by uid 99); 15 Jul 2008 17:30:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jul 2008 10:30:09 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of gmackey@cs.ucf.edu designates 132.170.108.1 as permitted sender)
Received: from [132.170.108.1] (HELO longwood.cs.ucf.edu) (132.170.108.1)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jul 2008 17:29:15 +0000
Received: from longwood.cs.ucf.edu (localhost [127.0.0.1])
	by longwood.cs.ucf.edu (8.14.1/8.14.1) with ESMTP id m6FHTRQ8026540
	for <general@hadoop.apache.org>; Tue, 15 Jul 2008 13:29:30 -0400 (EDT)
Received: (from prayer@localhost)
	by longwood.cs.ucf.edu (8.14.1/8.14.1/Submit) id m6FHTRlZ026538
	for general@hadoop.apache.org; Tue, 15 Jul 2008 13:29:27 -0400 (EDT)
X-Authentication-Warning: longwood.cs.ucf.edu: prayer set sender to gmackey@cs.ucf.edu using -f
From: gmackey@cs.ucf.edu
To: general@hadoop.apache.org
Subject: Re: how to distribute the data to all the datanodes?
Date: 15 Jul 2008 13:29:26 -0400
X-Mailer: Prayer v1.0.16
X-Originating-IP: [128.165.72.17]
In-Reply-To: <1216030332.3992.19.camel@localhost.localdomain>
Message-ID: <Prayer.1.0.16.0807151329260.25785@longwood.cs.ucf.edu>
References: <1216030332.3992.19.camel@localhost.localdomain>
Mime-Version: 1.0
Content-Type: text/plain; format=flowed; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Spam-Level: *
X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on gondor.cs.ucf.edu
X-Virus-Checked: Checked by ClamAV on apache.org
X-Old-Spam-Status: No, score=1.6 required=5.0 tests=FH_RELAY_NODNS,RDNS_NONE
	autolearn=no version=3.2.4


I'm failing to see your question. If you only want one copy of the data=20
stored, but you want the 128MB to be replicated over your data nodes, then=
=20
you need to set the replication factor to 1. I'm surprised that it let you=
=20
set the factor to 0.

IF you were to set the replication value any higher than 1, then multiple=
=20
copies would exist, for redundancy, and would be distributed across the=20
three nodes.

hope this helps

 - Grant

On Jul 14 2008, Yi Zhao wrote:

>hi, all
>I have a hadoop cluster which have one master and three datanodes.
>
>I want to put a local file about 128M intpu hdfs, I have set the
>block-size to 10M
>
>when I set the replication to 0,
>I found that all the data distributed to the node which I execute the
>command 'bin/hadoop dfs -put file.gz input', so this node's disk space
>is used about 128M, but other nodes has no disk space used.
>
>when I set the replication to 3,
>I found that every nodes have the same data, so every nodes is about
>128M disk space used.
>
>what should I do? =EF=BB=BFI'm using hadoop-0.15.2.
>
>any one can help me?
>
>thanks.
>

--=20
Grant Mackey
UCF Researcher
Eng. III Rm238


From general-return-10-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 24 07:59:02 2008
Return-Path: <general-return-10-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 26607 invoked from network); 24 Jul 2008 07:59:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 24 Jul 2008 07:59:02 -0000
Received: (qmail 57923 invoked by uid 500); 24 Jul 2008 07:59:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57902 invoked by uid 500); 24 Jul 2008 07:59:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57886 invoked by uid 99); 24 Jul 2008 07:59:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Jul 2008 00:59:01 -0700
X-ASF-Spam-Status: No, hits=2.4 required=10.0
	tests=SPF_PASS,TVD_FW_GRAPHIC_NAME_LONG
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [210.150.84.113] (HELO connecty.co.jp) (210.150.84.113)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 24 Jul 2008 07:58:04 +0000
Received: (qmail 24818 invoked by SAV 20080723.038 by uid 0); 24 Jul 2008 16:58:25 +0900
X-Authentication: hideki.kajiwara was authenticated by 210.150.84.113
 at  24 Jul 2008 16:58:25 +0900
Received: from unknown (HELO connecty.co.jp) (124.25.242.126)
  by ps27.suite2.arena.ne.jp (210.150.84.113) with SMTP; 24 Jul 2008 16:58:25 +0900
From: Hideki KAJIWARA <hideki.kajiwara@connecty.co.jp>
To: general@hadoop.apache.org
Cc: tomoharu.fujita@connecty.co.jp
Subject: redundant configuration of the namenode
Date: Thu, 24 Jul 2008 16:58:17 +0900
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="Boundary-bqsQeP5FSnJRtWaOIURmd"
X-Mailer: HidemaruMail 4.83 (WinNT,501)
Message-Id: <C9C8ED63085DCAhideki.kajiwara@connecty.co.jp>
X-Virus-Checked: Checked by ClamAV on apache.org

--Boundary-bqsQeP5FSnJRtWaOIURmd
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Description: Mail message body

It thinks about the redundant configuration of the name node that used 
keepalived (active/standby). (Refer to attached file redundant.gif and 
configuration file hadoop-site.xml. )

The NFS mount does the directory "/dfs/name" on the standby machine to "
/dfs/namerep" set to dfs.name.dir in the configuration file. 
As a result, it thinks about making of file system metadata tedious. 

It became a result like attached file result.gif, and the file was not 
complete though the 
failover had been executed while writing the data of 100MB by using the 
sample program. 


Hereafter, reproduction procedure. 

1.The sample program is executed. 
java sample.HadoopWriteToDFS 104857600 /dfs/users/kajiwara/test.data

2.It is kill as for namenode process of the active machine. 
(Then, start-dfs.sh starts on the standby machine. )

After a while, the file is not complete as the above-mentioned though 
the command ends normally. 

I want you to teach if there is a method of solving above-mentioned 
issue. 

H.KAJIWARA
hideki.kajiwara@connecty.co.jp

--Boundary-bqsQeP5FSnJRtWaOIURmd
Content-Type: text/xml; charset=us-ascii; name="hadoop-site.xml"
Content-Disposition: attachment; filename="hadoop-site.xml"
Content-Transfer-Encoding: base64

PD94bWwgdmVyc2lvbj0iMS4wIj8+DQo8Y29uZmlndXJhdGlvbj4NCiAgICA8cHJvcGVydHk+
DQogICAgICAgIDxuYW1lPmZzLmxpc3Rlbi5hZGRyZXNzPC9uYW1lPg0KICAgICAgICA8dmFs
dWU+MC4wLjAuMDo5MDAwPC92YWx1ZT4NCiAgICA8L3Byb3BlcnR5Pg0KICAgIDxwcm9wZXJ0
eT4NCiAgICAgICAgPG5hbWU+ZGZzLm5hbWUuZGlyPC9uYW1lPg0KPCEtLWZvciBhY3RpdmUg
TmFtZW5vZGUtLT4NCiAgICAgICAgPHZhbHVlPi9kZnMvbmFtZSwvZGZzL25hbWVyZXA8L3Zh
bHVlPg0KPCEtLWZvciBzdGFuZGJ5IE5hbWVub2RlDQogICAgICAgIDx2YWx1ZT4vZGZzL25h
bWU8L3ZhbHVlPg0KLS0+DQogICAgICAgIDxkZXNjcmlwdGlvbj5EZXRlcm1pbmVzIHdoZXJl
IG9uIHRoZSBsb2NhbCBmaWxlc3lzdGVtIHRoZSBERlMgbmFtZSBub2RlDQogICAgICAgIHNo
b3VsZCBzdG9yZSB0aGUgbmFtZSB0YWJsZS4gIElmIHRoaXMgaXMgYSBjb21tYS1kZWxpbWl0
ZWQgbGlzdA0KICAgICAgICBvZiBkaXJlY3RvcmllcyB0aGVuIHRoZSBuYW1lIHRhYmxlIGlz
IHJlcGxpY2F0ZWQgaW4gYWxsIG9mIHRoZQ0KICAgICAgICBkaXJlY3RvcmllcywgZm9yIHJl
ZHVuZGFuY3kuIDwvZGVzY3JpcHRpb24+DQogICAgPC9wcm9wZXJ0eT4NCiAgICA8cHJvcGVy
dHk+DQogICAgICAgIDxuYW1lPmZzLmRlZmF1bHQubmFtZTwvbmFtZT4NCiAgICAgICAgPHZh
bHVlPmhkZnM6Ly9kZnMtbWFzdGVyOjkwMDA8L3ZhbHVlPg0KICAgICAgICA8ZGVzY3JpcHRp
b24+VGhlIG5hbWUgb2YgdGhlIGRlZmF1bHQgZmlsZSBzeXN0ZW0uICBBIFVSSSB3aG9zZQ0K
ICAgICAgICBzY2hlbWUgYW5kIGF1dGhvcml0eSBkZXRlcm1pbmUgdGhlIEZpbGVTeXN0ZW0g
aW1wbGVtZW50YXRpb24uICBUaGUNCiAgICAgICAgdXJpJ3Mgc2NoZW1lIGRldGVybWluZXMg
dGhlIGNvbmZpZyBwcm9wZXJ0eSAoZnMuU0NIRU1FLmltcGwpIG5hbWluZw0KICAgICAgICB0
aGUgRmlsZVN5c3RlbSBpbXBsZW1lbnRhdGlvbiBjbGFzcy4gIFRoZSB1cmkncyBhdXRob3Jp
dHkgaXMgdXNlZCB0bw0KICAgICAgICBkZXRlcm1pbmUgdGhlIGhvc3QsIHBvcnQsIGV0Yy4g
Zm9yIGEgZmlsZXN5c3RlbS48L2Rlc2NyaXB0aW9uPg0KICAgIDwvcHJvcGVydHk+DQogICAg
PHByb3BlcnR5Pg0KICAgICAgICA8bmFtZT5kZnMuZGF0YS5kaXI8L25hbWU+DQogICAgICAg
IDx2YWx1ZT4vZGZzL2V4cG9ydDwvdmFsdWU+DQogICAgICAgIDxkZXNjcmlwdGlvbj5EZXRl
cm1pbmVzIHdoZXJlIG9uIHRoZSBsb2NhbCBmaWxlc3lzdGVtIGFuIERGUyBkYXRhIG5vZGUN
CiAgICAgICAgc2hvdWxkIHN0b3JlIGl0cyBibG9ja3MuICBJZiB0aGlzIGlzIGEgY29tbWEt
ZGVsaW1pdGVkDQogICAgICAgIGxpc3Qgb2YgZGlyZWN0b3JpZXMsIHRoZW4gZGF0YSB3aWxs
IGJlIHN0b3JlZCBpbiBhbGwgbmFtZWQNCiAgICAgICAgZGlyZWN0b3JpZXMsIHR5cGljYWxs
eSBvbiBkaWZmZXJlbnQgZGV2aWNlcy4NCiAgICAgICAgRGlyZWN0b3JpZXMgdGhhdCBkbyBu
b3QgZXhpc3QgYXJlIGlnbm9yZWQuDQogICAgICAgIDwvZGVzY3JpcHRpb24+DQogICAgPC9w
cm9wZXJ0eT4NCiAgICA8cHJvcGVydHk+DQogICAgICAgIDxuYW1lPmRmcy5yZXBsaWNhdGlv
bjwvbmFtZT4NCiAgICAgICAgPHZhbHVlPjI8L3ZhbHVlPg0KICAgICAgICA8ZGVzY3JpcHRp
b24+RGVmYXVsdCBibG9jayByZXBsaWNhdGlvbi4NCiAgICAgICAgVGhlIGFjdHVhbCBudW1i
ZXIgb2YgcmVwbGljYXRpb25zIGNhbiBiZSBzcGVjaWZpZWQgd2hlbiB0aGUgZmlsZSBpcyBj
cmVhdGVkLg0KICAgICAgICBUaGUgZGVmYXVsdCBpcyB1c2VkIGlmIHJlcGxpY2F0aW9uIGlz
IG5vdCBzcGVjaWZpZWQgaW4gY3JlYXRlIHRpbWUuDQogICAgICAgIDwvZGVzY3JpcHRpb24+
DQogICAgPC9wcm9wZXJ0eT4NCiAgICA8cHJvcGVydHk+DQogICAgICAgIDxuYW1lPmRmcy5w
ZXJtaXNzaW9ucy5zdXBlcmdyb3VwPC9uYW1lPg0KICAgICAgICA8dmFsdWU+ZGZzPC92YWx1
ZT4NCiAgICAgICAgPGRlc2NyaXB0aW9uPlRoZSBuYW1lIG9mIHRoZSBncm91cCBvZiBzdXBl
ci11c2Vycy48L2Rlc2NyaXB0aW9uPg0KICAgIDwvcHJvcGVydHk+DQoNCjwhLS1mb3IgTWFw
UmVkdWNlLS0+DQogICAgPHByb3BlcnR5Pg0KICAgICAgICA8bmFtZT5tYXByZWQuam9iLnRy
YWNrZXI8L25hbWU+DQogICAgICAgIDx2YWx1ZT5kZnMtbWFzdGVyOjEwMDAwPC92YWx1ZT4N
CiAgICAgICAgPGRlc2NyaXB0aW9uPlRoZSBob3N0IGFuZCBwb3J0IHRoYXQgdGhlIE1hcFJl
ZHVjZSBqb2IgdHJhY2tlciBydW5zDQogICAgICAgIGF0LiAgSWYgImxvY2FsIiwgdGhlbiBq
b2JzIGFyZSBydW4gaW4tcHJvY2VzcyBhcyBhIHNpbmdsZSBtYXANCiAgICAgICAgYW5kIHJl
ZHVjZSB0YXNrLg0KICAgICAgICA8L2Rlc2NyaXB0aW9uPg0KICAgIDwvcHJvcGVydHk+DQog
ICAgPHByb3BlcnR5Pg0KICAgICAgICA8bmFtZT5tYXByZWQuc3lzdGVtLmRpcjwvbmFtZT4N
CiAgICAgICAgPHZhbHVlPi9tYXByZWQvc3lzdGVtPC92YWx1ZT4NCiAgICAgICAgPGRlc2Ny
aXB0aW9uPlRoZSBzaGFyZWQgZGlyZWN0b3J5IHdoZXJlIE1hcFJlZHVjZSBzdG9yZXMgY29u
dHJvbCBmaWxlcy4NCiAgICAgICAgPC9kZXNjcmlwdGlvbj4NCiAgICA8L3Byb3BlcnR5Pg0K
PC9jb25maWd1cmF0aW9uPg0K

--Boundary-bqsQeP5FSnJRtWaOIURmd--

From general-return-11-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 24 09:17:02 2008
Return-Path: <general-return-11-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 98098 invoked from network); 24 Jul 2008 09:17:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 24 Jul 2008 09:17:02 -0000
Received: (qmail 80259 invoked by uid 500); 24 Jul 2008 09:17:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80209 invoked by uid 500); 24 Jul 2008 09:17:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80198 invoked by uid 99); 24 Jul 2008 09:17:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Jul 2008 02:17:02 -0700
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dynaturtle.cai@gmail.com designates 209.85.142.185 as permitted sender)
Received: from [209.85.142.185] (HELO ti-out-0910.google.com) (209.85.142.185)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Jul 2008 09:16:08 +0000
Received: by ti-out-0910.google.com with SMTP id d27so1476474tid.9
        for <general@hadoop.apache.org>; Thu, 24 Jul 2008 02:16:15 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:to
         :subject:mime-version:content-type;
        bh=jhax1/nq4pneuJeVzwEtvQNgpBHl0FMnNiWipWUI32Q=;
        b=jrQ+lRQG5tI6Ib+y8JXWuOA72UMuqGlVjrvBq54Coa9EXlLxIlnQor4COTVKEQE4S9
         wITXuHyK8hdWfqS1q6sOv+EFHcImMkuJCpgolEpmIP/Lr31CNa/ur3IXYB7uRVVnezIw
         Q5uLiJfaMYCaWbgK/MK+qp45R16ggUFIf+vLw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:mime-version:content-type;
        b=OgQUW73LYgx6hhiBLtXOZsw8HFRRvrQ5lfg/o+9SIcdw3dd19ZqNBLg3r1yAj6FNUo
         HxTHW2nuuK799/Tcm0t76T6uES1aTydoZavY3ZSnWK3UfNTUY0FcyL6qR68K/NY9Qb1n
         LTEtbJjD9uFmMYVVre+tpIOws03lK7KbjAlM8=
Received: by 10.110.47.17 with SMTP id u17mr8495tiu.49.1216890975119;
        Thu, 24 Jul 2008 02:16:15 -0700 (PDT)
Received: by 10.110.49.13 with HTTP; Thu, 24 Jul 2008 02:16:15 -0700 (PDT)
Message-ID: <448f7bd20807240216t1cf71609ve766e95112357553@mail.gmail.com>
Date: Thu, 24 Jul 2008 17:16:15 +0800
From: "Cai Xiao(dynaturtle)" <dynaturtle.cai@gmail.com>
To: general@hadoop.apache.org
Subject: Can anyone explain how to use hadoop plugin for eclipse?
MIME-Version: 1.0
Content-Type: multipart/alternative; 
	boundary="----=_Part_32719_11809156.1216890975161"
X-Virus-Checked: Checked by ClamAV on apache.org

------=_Part_32719_11809156.1216890975161
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

I am new to hadoop. And could anyone tell me how I can use hadoop eclipse
plugin?
I have troubled with how to use it.
-- 
Department of Spatial Information Science&Technology,
School of Earth&Space Science, Peking University

ADD: Room 3028, Building 46, Peking University, Beijing, P.R. China
Zip Code: 100871

------=_Part_32719_11809156.1216890975161--

From general-return-12-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 24 09:35:33 2008
Return-Path: <general-return-12-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 19543 invoked from network); 24 Jul 2008 09:35:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 24 Jul 2008 09:35:33 -0000
Received: (qmail 2559 invoked by uid 500); 24 Jul 2008 09:35:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2510 invoked by uid 500); 24 Jul 2008 09:35:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2499 invoked by uid 99); 24 Jul 2008 09:35:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Jul 2008 02:35:33 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of thopham.asnet@gmail.com designates 72.14.246.241 as permitted sender)
Received: from [72.14.246.241] (HELO ag-out-0708.google.com) (72.14.246.241)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Jul 2008 09:34:38 +0000
Received: by ag-out-0708.google.com with SMTP id 9so15852135agd.9
        for <general@hadoop.apache.org>; Thu, 24 Jul 2008 02:35:03 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=ZB2CPDosK2gQ/kntvnMacicLcws3XQ1P10OHm2T1XzE=;
        b=vjHbgyJbsIYmVEUjzZGBtmx7xjTvl2POZNECBWIu0EU5hmRGZ8wKcxa3KIHKUkVhdU
         6bAPtpiNhziwJditHdw+kmZ2ohX2l+YmmntPwu+EjT9JzwJCpkk1VKDCAW5aKguIiMJh
         HIAaT0e4D0uxZXojyiDK+/pF+C70AfQryBryo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:content-type:content-transfer-encoding;
        b=MIisSlLZiPwR+X+RKYHhZlE7D3w6xBepFaynzVAHIMrofBm6ucQyvmWvmaFQRpToSW
         0akSknOyQpbFtcedHsY6+BjNdJfN9hjhPg67Sui6oQuodS490Z0vivgckiPeBfGH6Mhp
         cCF1qSFDniMIoVC6FKBQNXswngE2NeIxCqImc=
Received: by 10.110.41.17 with SMTP id o17mr53177tio.18.1216892101590;
        Thu, 24 Jul 2008 02:35:01 -0700 (PDT)
Received: from ?192.168.1.11? ( [117.2.89.183])
        by mx.google.com with ESMTPS id w12sm17831409tib.1.2008.07.24.02.34.57
        (version=SSLv3 cipher=RC4-MD5);
        Thu, 24 Jul 2008 02:34:59 -0700 (PDT)
Message-ID: <48884C8E.5020800@gmail.com>
Date: Thu, 24 Jul 2008 16:34:06 +0700
From: "thopham.asnet" <thopham.asnet@gmail.com>
User-Agent: Thunderbird 2.0.0.14 (Windows/20080421)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Can anyone explain how to use hadoop plugin for eclipse?
References: <448f7bd20807240216t1cf71609ve766e95112357553@mail.gmail.com>
In-Reply-To: <448f7bd20807240216t1cf71609ve766e95112357553@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hey Cai Xiao,
You should do the steps below:

1. Open the path $HADOOP_HOME/contrib/eclipse-plugin, you will find a 
hadoop plugin for eclipse. Copy this .jar file to plugin folder of Eclipse.
2. Start Eclipse, select Windows\Preferences, then select Hadoop 
Map/Reduce node, you have to point to the root directory of Hadoop 
installation.
3. Restart Eclipse
4. Open Map/Reduce perspective in Eclipse
5. Create a new Map/Reduce server location by the instruction of the dialog.
6. Start your HDFS server and Map/Reduce master
7. Now, you can browse HDFS directory and submit your job to Map/Reduce 
server

Good luck and have fun,

Ps: Don't forget setting necessary environment variables.

Best regards,
Tho Pham

Cai Xiao(dynaturtle) wrote:
> I am new to hadoop. And could anyone tell me how I can use hadoop eclipse
> plugin?
> I have troubled with how to use it.
>   


From general-return-13-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 24 10:15:36 2008
Return-Path: <general-return-13-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 40574 invoked from network); 24 Jul 2008 10:15:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 24 Jul 2008 10:15:35 -0000
Received: (qmail 54507 invoked by uid 500); 24 Jul 2008 10:15:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54481 invoked by uid 500); 24 Jul 2008 10:15:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54470 invoked by uid 99); 24 Jul 2008 10:15:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Jul 2008 03:15:35 -0700
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dynaturtle.cai@gmail.com designates 209.85.142.184 as permitted sender)
Received: from [209.85.142.184] (HELO ti-out-0910.google.com) (209.85.142.184)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Jul 2008 10:14:39 +0000
Received: by ti-out-0910.google.com with SMTP id d27so1491975tid.9
        for <general@hadoop.apache.org>; Thu, 24 Jul 2008 03:15:03 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:to
         :subject:in-reply-to:mime-version:content-type:references;
        bh=Tv4UOh9Rvtdn5dgsqRHmxzp7VX/r7UONsw4zHz+h9Hw=;
        b=itsF4pSJiFGsI69UiYjREs385bEd4MsbFu+qlKA1byT7hWVhf3Vo7ZxU3WpYym+7pt
         7URv50oy5nxHgsmE2ikp5UQnf11xxD4UwmMUWle9SrTjDI9Ugod31Ek85W4eo7VxPhWE
         edej5JxVw4dofRpm6evsk517TXKcaGGsLcZsU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version
         :content-type:references;
        b=gz3Ox3LyMuj+B9WDIaSOxwX6kWrubMLIEEpzXiL6HcR/hblrNARHTe4BofTKZV/WfR
         yofyXXHI/epgCDrFI40VusE5JuP8TMZpe12xtLAvqlYNAfQIdUinXyS2t4nCvnXu5RYu
         XgVHZ0SB6YvWWY2GaKpZZQzeUu1oJggh56iaM=
Received: by 10.110.46.3 with SMTP id t3mr77855tit.33.1216894502913;
        Thu, 24 Jul 2008 03:15:02 -0700 (PDT)
Received: by 10.110.49.13 with HTTP; Thu, 24 Jul 2008 03:15:02 -0700 (PDT)
Message-ID: <448f7bd20807240315y55a5dd22v196586d2812308ff@mail.gmail.com>
Date: Thu, 24 Jul 2008 18:15:02 +0800
From: "Cai Xiao(dynaturtle)" <dynaturtle.cai@gmail.com>
To: general@hadoop.apache.org
Subject: Re: Can anyone explain how to use hadoop plugin for eclipse?
In-Reply-To: <48884C8E.5020800@gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; 
	boundary="----=_Part_36684_11034478.1216894502961"
References: <448f7bd20807240216t1cf71609ve766e95112357553@mail.gmail.com>
	 <48884C8E.5020800@gmail.com>
X-Virus-Checked: Checked by ClamAV on apache.org

------=_Part_36684_11034478.1216894502961
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Hello, Tho Pham,
          Thanks for your nice instruction.
However, I still have some problems when it comes step5
my server info is as following:
hostname: cadev18.asc.cnz.alimama.com
installation directory: hadoop
username: cloud15

And I have set my map/reduce master as:
cadev18.asc.cnz.alimama.com
my user name as:
cloud15
And left other parameters as default.
However,  the server is not correctly connected. Eclipse cannot read the
directory correctly.
So can you give me any suggestin? Is there any other parameter I have to
set.
Thanks

Regards
      Xiao

On Thu, Jul 24, 2008 at 5:34 PM, thopham.asnet <thopham.asnet@gmail.com>
wrote:

> Hey Cai Xiao,
> You should do the steps below:
>
> 1. Open the path $HADOOP_HOME/contrib/eclipse-plugin, you will find a
> hadoop plugin for eclipse. Copy this .jar file to plugin folder of Eclipse.
> 2. Start Eclipse, select Windows\Preferences, then select Hadoop Map/Reduce
> node, you have to point to the root directory of Hadoop installation.
> 3. Restart Eclipse
> 4. Open Map/Reduce perspective in Eclipse
> 5. Create a new Map/Reduce server location by the instruction of the
> dialog.
> 6. Start your HDFS server and Map/Reduce master
> 7. Now, you can browse HDFS directory and submit your job to Map/Reduce
> server
>
> Good luck and have fun,
>
> Ps: Don't forget setting necessary environment variables.
>
> Best regards,
> Tho Pham
>
>
> Cai Xiao(dynaturtle) wrote:
>
>> I am new to hadoop. And could anyone tell me how I can use hadoop eclipse
>> plugin?
>> I have troubled with how to use it.
>>
>>
>
>


-- 
Department of Spatial Information Science&Technology,
School of Earth&Space Science, Peking University

ADD: Room 3028, Building 46, Peking University, Beijing, P.R. China
Zip Code: 100871

------=_Part_36684_11034478.1216894502961--

From general-return-423-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 01 21:59:13 2009
Return-Path: <general-return-423-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 41717 invoked from network); 1 Aug 2009 21:59:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Aug 2009 21:59:13 -0000
Received: (qmail 39495 invoked by uid 500); 1 Aug 2009 21:59:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39423 invoked by uid 500); 1 Aug 2009 21:59:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39413 invoked by uid 99); 1 Aug 2009 21:59:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 01 Aug 2009 21:59:16 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ctubbsii@gmail.com designates 209.85.221.188 as permitted sender)
Received: from [209.85.221.188] (HELO mail-qy0-f188.google.com) (209.85.221.188)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 01 Aug 2009 21:59:06 +0000
Received: by qyk26 with SMTP id 26so3384011qyk.5
        for <general@hadoop.apache.org>; Sat, 01 Aug 2009 14:58:45 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :x-enigmail-version:content-type:content-transfer-encoding;
        bh=602EASQjyJZKhhmYQEEwUMrpiHcJhPkKKRZgcSbGw5s=;
        b=HrZwjelyd91ZmQJBu40QUQxXDJlk97HphdbzR4Fq6eEwSywnaHV65sU+u3w3svMvgW
         Vam1XrD0i15NGTwUc/A0rhDOPMq4VIsPj2qCvY+gdkFx7DX9xPD659No/pmVtHzEJ5v8
         a8tldeSt5F1OQFjfjMiEtdWltXFIcjQb+3s0c=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:x-enigmail-version:content-type
         :content-transfer-encoding;
        b=WZ0Vp0H3/1uTQoTcoe6cBaO6e3sazrsw7DHxZaE9O4IwHT9rkGNri2IV8k18h8w9Nh
         KbB7u+2FfjMp/q2mr3OmI8y40P+xOVHnZU1DXUdyiznXdL3TK2RyaE2p/DxXZSVxRwhT
         CsRwHgKnvivhwemKX/Sq/kyHzWIE+wVpHBXMY=
Received: by 10.224.11.142 with SMTP id t14mr3334910qat.356.1249163925730;
        Sat, 01 Aug 2009 14:58:45 -0700 (PDT)
Received: from ?192.168.90.4? (pool-71-246-212-121.washdc.fios.verizon.net [71.246.212.121])
        by mx.google.com with ESMTPS id 8sm9328433qwj.56.2009.08.01.14.58.44
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Sat, 01 Aug 2009 14:58:45 -0700 (PDT)
Message-ID: <4A74BA93.6000802@gmail.com>
Date: Sat, 01 Aug 2009 17:58:43 -0400
From: Christopher L Tubbs II <ctubbsii@gmail.com>
User-Agent: Thunderbird 2.0.0.21 (X11/20090320)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: mapreduce classpath issue
References: <2C52DBBEC4855C438BB330CB0D3B465903BFADA5@SNV-EXVS01.ds.corp.yahoo.com> <4A6A7D7C.8030202@gmail.com>
In-Reply-To: <4A6A7D7C.8030202@gmail.com>
X-Enigmail-Version: 0.96.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

I previously sent the following message requesting assistance with
classpath issues. I've now appeared to resolve the issue, in that the
libjars that I specify are added to the classpath for the mapper tasks,
but NOT for the run() method of the Tool.

I can get around this by specifying HADOOP_CLASSPATH environment
variable with the identical libjar listing, but this seems to be a bit
clunky for a quick mapreduce job.

Shouldn't libjars also place the jars on the hadoop classpath for the
thread that executes run()?

Right now, to avoid setting the libjars twice, I use (syntax may not be
exactly right, I'm going by memory):
  job.setInputFormat(conf.getClassByName("my.package.Class")))

However, in my InputFormat class, I also have static methods that add
information to the conf for the mapper task, in the same way that
FileInputFormat and FileOutputFormat do, so as to abstract away the
implementation details of those configuration settings.

So, while I can set the input format / output format by class name, I
cannot access these static methods in the run() thread unless I use
reflection.

It seems very odd that libjars only works for the mapper/reducer tasks
and not for the Tool.run() method. The two workarounds I've found are,
as I've stated above:
 1. duplicating the libjars listing on the HADOOP_CLASSPATH environment
variable for the Tool thread, and
 2. grabbing the class from the conf.getClassByName() and using
reflection to call the static methods to configure the class' options.

On a side note, it also seems very clumsy that the Tool requires to
extend Configured so that you can create the JobConf from the one
retrieved from getConf() so that the libjars works at all.

Any thoughts or suggestions for an easier way?


Christopher L Tubbs II wrote:
> So, I've been having an issue running on Mac OS, with classpath for
> MapReduce jobs on version 0.20.0
> 
> The basic idea is that I want to add a jar as a dependency to a
> MapReduce job. I know I can add the jar to HADOOP_CLASSPATH, but I do
> not have access to hadoop-env.xml (administrator restricted).
> 
> I have been using the "hadoop jar" command to execute a class with a
> main() contained within my jar. My class extends Configured and
> implements Tool. The main() calls ToolRunner.run() to launch the job,
> and I specify -libjars on the command line, which I see from the source,
> uses the GenericOptionsParser to strip the "-libjars x.jar,y.jar,z.jar"
> from the arguments passed to my class. I then get errors that I can't
> find classes contained within my library jars.
> 
> From within run(), my debugging code demonstrates that the jar files are
> readable, and I can getResource() to verify that the classes are
> available in those jars. However, I still get errors, because it doesn't
> seem that at any point, the JVM is using the contextClassLoader that the
> resources were added with.
> 
> Perhaps this is a bug with Mac OS's JVM 1.6, or am I doing something
> wrong? Everything works fine in linux with the identical setup.
> 
> 
> -
> Chris

From general-return-424-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 15:24:57 2009
Return-Path: <general-return-424-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27920 invoked from network); 11 Aug 2009 15:24:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 15:24:57 -0000
Received: (qmail 21368 invoked by uid 500); 11 Aug 2009 15:25:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21314 invoked by uid 500); 11 Aug 2009 15:25:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21304 invoked by uid 99); 11 Aug 2009 15:25:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 15:25:03 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of fredwang222@gmail.com designates 209.85.200.172 as permitted sender)
Received: from [209.85.200.172] (HELO wf-out-1314.google.com) (209.85.200.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 15:24:55 +0000
Received: by wf-out-1314.google.com with SMTP id 23so1349777wfg.2
        for <general@hadoop.apache.org>; Tue, 11 Aug 2009 08:24:34 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=PN7LJeNPtR2Hgc//MOV9N7hLzCQcEtRvE+5EceHh7UI=;
        b=DgaKkJb3Qld/ZBVb/B3Q40vHgMKjW/QGb5xhCC0RY/2NYXeHZJLUmdRHk2CXyfWcx3
         GrstER+Dbek2J1edjesSCzf41N1g9h9UJvljCroibAGarunwfjC4ujP/CIyWfwQCe8Gn
         yqeyBu+iRlzEmA+F7wVGPODOJ7ZAR5gdCK/sI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=WDCUQpYmQHJV+AXZQ1HcZLzXtF3MA190MmceffwqVdqWFHbulSOVDrpLnhXwtiAryq
         qEp0rdkOvVXi5DWvc7JmYvwoegRfv1jZ6EzHBhn+Vgvb3C00HfrOMc1Y5h3103qKAdOg
         WXm390iaeRwnPJNDHM5W9xNnIvCVaSktql4UY=
MIME-Version: 1.0
Received: by 10.142.233.21 with SMTP id f21mr1225767wfh.229.1250004274748; 
	Tue, 11 Aug 2009 08:24:34 -0700 (PDT)
Date: Tue, 11 Aug 2009 23:24:34 +0800
Message-ID: <702261350908110824j54ddcf6bre1146d09628061ad@mail.gmail.com>
Subject: run WordCount example have many fetch-failure issues randomly but 
	finally succeed,
From: fred wang <fredwang222@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd2e04cc39d490470df4c94
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd2e04cc39d490470df4c94
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

 It runs very well at the first several times, but when run it again, the
hadoop system become pretty much slow and show the following errors during
the runnig process(map 100% reduce: 22%) though it finally succeed.
    Info mapred.JobClient: Task Id : attempt_xxx_0030_m_002_0, Status:
FAILED: many fetch_failures
 When I run it again, it become OK until same failures after another several
time run.  Any potential reasons?  How can I probe this issue? Thx.

--000e0cd2e04cc39d490470df4c94--

From general-return-425-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 04:32:08 2009
Return-Path: <general-return-425-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50697 invoked from network); 14 Aug 2009 04:32:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 04:32:08 -0000
Received: (qmail 44260 invoked by uid 500); 14 Aug 2009 04:32:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44159 invoked by uid 500); 14 Aug 2009 04:32:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44148 invoked by uid 99); 14 Aug 2009 04:32:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 04:32:13 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of eelco.hillenius@gmail.com designates 209.85.210.185 as permitted sender)
Received: from [209.85.210.185] (HELO mail-yx0-f185.google.com) (209.85.210.185)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 04:32:05 +0000
Received: by yxe15 with SMTP id 15so1735874yxe.5
        for <general@hadoop.apache.org>; Thu, 13 Aug 2009 21:31:44 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:from:date:message-id
         :subject:to:content-type:content-transfer-encoding;
        bh=JcEgH3Mr4qsR1c3GNrxetDaFcovmrTzLatz/skLn2CU=;
        b=cWoBzAwmeqWLos943LOgL31RB5M8u4o54urCUXqTxfEXWo/oRdCKyaDD07NY2r63yM
         XpDd6cIe1dzzVKgWYH+g88+Qs21/DVxTwg4ZWxEPZYUXO9VveQNysSGcUhJDKX3DIDd4
         tgIZ1kTLJZhaLXA1zjqB9282z4gjExTpvz+ro=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        b=wm1kwfm8Ad3LW85pc7ArY5NpuZrLi9uRn4W2PtmtVmaS7CRZYQMliEkDYenRCuUOM9
         Y/lcpdGsZTYBLY7RkTmlV1o+6UZmt5u8jHlh7+h0r0erp4OoSKSEnUIsNALH4Vohkb63
         Fp0oYV2dgkB0TRGb9lsfNXMlCx+j77bOz2Bcc=
MIME-Version: 1.0
Received: by 10.150.58.8 with SMTP id g8mr2080561yba.347.1250224304832; Thu, 
	13 Aug 2009 21:31:44 -0700 (PDT)
From: Eelco Hillenius <eelco.hillenius@gmail.com>
Date: Thu, 13 Aug 2009 21:31:24 -0700
Message-ID: <ceb609d80908132131h71c9346kce97f2416549f450@mail.gmail.com>
Subject: HUG Sunnyvale on the 20th of August?
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hi all,

I was wondering whether there will be a Hadoop User Group event on
20th of August. I happen to be in town (Sunnyvale), and I would love
to meet a few Hadoopers.

Cheers,

Eelco

From general-return-426-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 17:51:17 2009
Return-Path: <general-return-426-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69180 invoked from network); 14 Aug 2009 17:51:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 17:51:17 -0000
Received: (qmail 9977 invoked by uid 500); 14 Aug 2009 17:51:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 9910 invoked by uid 500); 14 Aug 2009 17:51:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 9900 invoked by uid 99); 14 Aug 2009 17:51:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 17:51:22 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 17:51:10 +0000
Received: from [172.21.149.58] (wlanvpn-mc2e-247-58.corp.yahoo.com [172.21.149.58])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n7EHoVnq012194
	for <general@hadoop.apache.org>; Fri, 14 Aug 2009 10:50:31 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=N8k2fwZtqqRWNQFrPmV0BrgZ3IIGDxhiH/0LfsRjdJ+TP3Rw1VkrJRTqolRc/kZY
Message-ID: <4A85A3E6.4040703@yahoo-inc.com>
Date: Fri, 14 Aug 2009 10:50:30 -0700
From: Jakob Homan <jhoman@yahoo-inc.com>
User-Agent: Thunderbird 2.0.0.12 (Macintosh/20080213)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: HUG Sunnyvale on the 20th of August?
References: <ceb609d80908132131h71c9346kce97f2416549f450@mail.gmail.com>
In-Reply-To: <ceb609d80908132131h71c9346kce97f2416549f450@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Eelco-
    The August meetup will be the 19th at Yahoo!.  For this month you 
can sign up to attend at meetup 
(http://www.meetup.com/hadoop/calendar/10877366/?action=detail&eventId=10877366) 
rather than the regular Yahoo! Upcoming site.  Hope to see you there.

Cheers-

Jakob Homan
Hadoop at Yahoo!

Eelco Hillenius wrote:
> Hi all,
> 
> I was wondering whether there will be a Hadoop User Group event on
> 20th of August. I happen to be in town (Sunnyvale), and I would love
> to meet a few Hadoopers.
> 
> Cheers,
> 
> Eelco


From general-return-427-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 17:55:02 2009
Return-Path: <general-return-427-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70308 invoked from network); 14 Aug 2009 17:55:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 17:55:02 -0000
Received: (qmail 13930 invoked by uid 500); 14 Aug 2009 17:55:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13832 invoked by uid 500); 14 Aug 2009 17:55:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13822 invoked by uid 99); 14 Aug 2009 17:55:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 17:55:08 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of eelco.hillenius@gmail.com designates 209.85.211.198 as permitted sender)
Received: from [209.85.211.198] (HELO mail-yw0-f198.google.com) (209.85.211.198)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 17:54:57 +0000
Received: by ywh36 with SMTP id 36so2634505ywh.31
        for <general@hadoop.apache.org>; Fri, 14 Aug 2009 10:54:37 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=ooTHeGN4ONpDTjLpzcP7LUwGT0/fX4B8YQ5fPNVJHsY=;
        b=NQWRVP+wJ9xyWUYnh4A4OGA/VsjwW9XSZW0M5lYFVBrCFEebBQzEYZ/LpO/luf4GD+
         2ferzTe2vSLT9Q+zttSWLLg6hfMt0LVF3BGu9nBpcRIbJbi67uZdhzvn/nbKFMkcN6LM
         m2gkU9esskznM8uDtYRrRUAjH4blhv9Qe++4E=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=BW8UAGeMhrkYgijG36XDKfBD2uTUS8nA0Ivh8DCWLI7cSG3Fsv9YNAwzK9ileu0vRU
         6JZUPUHEHyEIrZplSt2TimkAp1AcNfa+HSZVTGkM3BCrqgRw/H7zdoRn4FYXMLLlXtvn
         gyhLMYRK+5eTKJqLVqJvj/Q2t6jgoCOjW1DIQ=
MIME-Version: 1.0
Received: by 10.150.141.13 with SMTP id o13mr2790359ybd.99.1250272477093; Fri, 
	14 Aug 2009 10:54:37 -0700 (PDT)
In-Reply-To: <4A85A3E6.4040703@yahoo-inc.com>
References: <ceb609d80908132131h71c9346kce97f2416549f450@mail.gmail.com> 
	<4A85A3E6.4040703@yahoo-inc.com>
From: Eelco Hillenius <eelco.hillenius@gmail.com>
Date: Fri, 14 Aug 2009 10:54:17 -0700
Message-ID: <ceb609d80908141054y7245f510h6b5e60d0994000fc@mail.gmail.com>
Subject: Re: HUG Sunnyvale on the 20th of August?
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Ah, thanks. Hope to make it!

Eelco

On Fri, Aug 14, 2009 at 10:50 AM, Jakob Homan<jhoman@yahoo-inc.com> wrote:
> Eelco-
> =A0 The August meetup will be the 19th at Yahoo!. =A0For this month you c=
an sign
> up to attend at meetup
> (http://www.meetup.com/hadoop/calendar/10877366/?action=3Ddetail&eventId=
=3D10877366)
> rather than the regular Yahoo! Upcoming site. =A0Hope to see you there.
>
> Cheers-
>
> Jakob Homan
> Hadoop at Yahoo!

From general-return-428-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 18:42:49 2009
Return-Path: <general-return-428-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4264 invoked from network); 14 Aug 2009 18:42:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 18:42:48 -0000
Received: (qmail 81586 invoked by uid 500); 14 Aug 2009 18:42:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81529 invoked by uid 500); 14 Aug 2009 18:42:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81519 invoked by uid 99); 14 Aug 2009 18:42:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 18:42:54 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 18:42:43 +0000
Received: from [192.168.0.197] (UNKNOWN-10-72-168-7.yahoo.com [10.72.168.7] (may be forged))
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n7EIetcI022273
	for <general@hadoop.apache.org>; Fri, 14 Aug 2009 11:40:55 -0700 (PDT)
Message-Id: <06027605-FB87-4F35-A40E-3E01C9B0B65B@yahoo-inc.com>
From: Alan Gates <gates@yahoo-inc.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Fwd: Amazon Elastic MapReduce Now Supports Apache Pig 
Date: Fri, 14 Aug 2009 11:40:55 -0700
References: <C97921224B64994093A1067F2D08834B176BA9BBFA@EX-SEA5-A.ant.amazon.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org



Begin forwarded message:

> From: "Sirota, Peter" <sirota@amazon.com>
> Date: August 10, 2009 6:17:26 PM PDT
> To: "pig-user@hadoop.apache.org" <pig-user@hadoop.apache.org>
> Subject: Amazon Elastic MapReduce Now Supports Apache Pig
> Reply-To: pig-user@hadoop.apache.org
>
> Dear Pig Users,
>
> We are excited to announce that Amazon Elastic MapReduce now  
> supports Apache Pig - making the service even more compelling for  
> large data set processing and analytics. Apache Pig is a platform  
> for analyzing large data sets that consists of a high-level language  
> for expressing data analysis programs, coupled with infrastructure  
> for evaluating these programs.
>
> Learn more in the Pig and Amazon Elastic MapReduce tutorial (http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2729&categoryID=269 
> )
>
> Watch a video (http://s3.amazonaws.com/awsVideos/AmazonElasticMapReduce/ElasticMapReduce-PigTutorial.html 
> )
>
> Use a sample Apache log processing application (http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2728&categoryID=263 
> )
>
>
> Sincerely,
> The Amazon Elastic MapReduce Team


From general-return-429-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 18:56:23 2009
Return-Path: <general-return-429-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6919 invoked from network); 14 Aug 2009 18:56:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 18:56:23 -0000
Received: (qmail 98112 invoked by uid 500); 14 Aug 2009 18:56:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98028 invoked by uid 500); 14 Aug 2009 18:56:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98018 invoked by uid 99); 14 Aug 2009 18:56:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 18:56:29 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 18:56:18 +0000
Received: from [10.72.185.127] (gentlepaint-lx.corp.yahoo.com [10.72.185.127])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n7EIsbLU068400
	for <general@hadoop.apache.org>; Fri, 14 Aug 2009 11:54:38 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=XqmbadhbrAT6+v/wNOJfaBqeKie5Em0ADwdxy2e2cvu3MCgz51QYacDUtYbarQPt
Message-ID: <4A85B2ED.2020900@yahoo-inc.com>
Date: Fri, 14 Aug 2009 11:54:37 -0700
From: Konstantin Shvachko <shv@yahoo-inc.com>
User-Agent: Thunderbird 2.0.0.22 (Windows/20090605)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: HUG Sunnyvale on the 20th of August?
References: <ceb609d80908132131h71c9346kce97f2416549f450@mail.gmail.com> 	<4A85A3E6.4040703@yahoo-inc.com> <ceb609d80908141054y7245f510h6b5e60d0994000fc@mail.gmail.com>
In-Reply-To: <ceb609d80908141054y7245f510h6b5e60d0994000fc@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

It is the 19th, right?
--Konstantin

Eelco Hillenius wrote:
> Ah, thanks. Hope to make it!
> 
> Eelco
> 
> On Fri, Aug 14, 2009 at 10:50 AM, Jakob Homan<jhoman@yahoo-inc.com> wrote:
>> Eelco-
>>   The August meetup will be the 19th at Yahoo!.  For this month you can sign
>> up to attend at meetup
>> (http://www.meetup.com/hadoop/calendar/10877366/?action=detail&eventId=10877366)
>> rather than the regular Yahoo! Upcoming site.  Hope to see you there.
>>
>> Cheers-
>>
>> Jakob Homan
>> Hadoop at Yahoo!
> 

From general-return-430-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 19:30:53 2009
Return-Path: <general-return-430-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17320 invoked from network); 14 Aug 2009 19:30:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 19:30:53 -0000
Received: (qmail 45338 invoked by uid 500); 14 Aug 2009 19:30:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45266 invoked by uid 500); 14 Aug 2009 19:30:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45256 invoked by uid 99); 14 Aug 2009 19:30:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 19:30:59 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eelco.hillenius@gmail.com designates 209.85.211.198 as permitted sender)
Received: from [209.85.211.198] (HELO mail-yw0-f198.google.com) (209.85.211.198)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 19:30:50 +0000
Received: by ywh36 with SMTP id 36so2725533ywh.31
        for <general@hadoop.apache.org>; Fri, 14 Aug 2009 12:30:29 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:from:date:message-id
         :subject:to:content-type:content-transfer-encoding;
        bh=zNPl8VN7F7iUMqiYQQMi9MMhmFYjXFAlMzPi88UVTbk=;
        b=GqgrBsBzviaN8K3U0cLNrvx8a792ALeJTu594MGxFMpKTU0Go98NF1jXK0oNiATDJq
         gMCVgQb5J7tCZ4VZ3aZhLG9POsZDnpX2uPMLZ7lQlqWlUbH0ZBxzXU3o4C1Dhs1AojV/
         LbFkcTyO/v7gBOzcg8DwUVbHmcKJ0X5jLqmIw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        b=n6OOK0ivGQbFvmQY9imjmh6M/noOO8hx1YCkoW9JS3LEkKrVmzELyBsHD/V6hfT6cK
         HD3a6JVuGYrJ14FT37oj7hagADKyirHwSYeUTgHww/gVvTIgfxyDSqY8ee3xbdt24B9J
         b84bANSkYrzwF8J2rDtjvmIYNSPTusBbq+yss=
MIME-Version: 1.0
Received: by 10.150.237.9 with SMTP id k9mr2872700ybh.108.1250278229188; Fri, 
	14 Aug 2009 12:30:29 -0700 (PDT)
From: Eelco Hillenius <eelco.hillenius@gmail.com>
Date: Fri, 14 Aug 2009 12:30:09 -0700
Message-ID: <ceb609d80908141230w5945ddcdn6f10f92f0e97c04a@mail.gmail.com>
Subject: comparing sub/ 3rd party projects that abstract map/reduce
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

Would people mind sharing their opinions about the relative strengths
and weaknesses of Hive, Chukwa and Pig (and possibly other
alternatives)?

I'm only just getting into Hadoop, but it seems to me these libs have
a considerable overlap depending on what you use them for? I plan to
check those sub projects out anyway, and I'm definitively not trying
to start a flame war here, but it would be great to hear some opinions
about what areas these projects are particularly useful for and what
might need some work etc.

As a bit of context, we (Teachscape) are considering Hadoop for
storing audit log files and extracting information from them. The
audit logging we do is application specific, e.g. user Foo deleted
Survey X, user Bar moved organization node AAB to AB, etc, and besides
the need to run a couple of fixed reports weekly (mainly that give our
customers some insight in how they are using our application), I
expect us to want to create queries on the fly to e.g. track down
problems. I don't expect our developers to have a problem writing Map/
Reduce programs, but I do like the idea of a higher level way of
extracting information.

Any thoughts would be greatly appreciated,

Eelco

From general-return-431-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 21:31:10 2009
Return-Path: <general-return-431-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58878 invoked from network); 14 Aug 2009 21:31:10 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 21:31:10 -0000
Received: (qmail 88517 invoked by uid 500); 14 Aug 2009 21:31:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88434 invoked by uid 500); 14 Aug 2009 21:31:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88424 invoked by uid 99); 14 Aug 2009 21:31:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 21:31:16 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of eelco.hillenius@gmail.com designates 209.85.217.227 as permitted sender)
Received: from [209.85.217.227] (HELO mail-gx0-f227.google.com) (209.85.217.227)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 21:31:06 +0000
Received: by gxk27 with SMTP id 27so2265574gxk.12
        for <general@hadoop.apache.org>; Fri, 14 Aug 2009 14:30:46 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=aC+k40AGTt32i8uJ2bthAEL9VM1ELEm6n99YSA4LLBQ=;
        b=Gim3u6UnwC+MlmuX372SMWorKxhweTiS26BbdZhpQUx63bYv7bM7sVPoAr0ydk1tJq
         h04HwcoLhUPJz+1tzBFO9oJ4O715JLsrtDYcsAAbYslRohNNBS6Pa25IVxRLJyyDkWKr
         Wx94Stls94Tlfho0YVoVdIkrWVQ9KlId1WY9M=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=QPWhnDJcYT/BpZH5rJmpZU7qXV/FI1Aey/SHKpD8DOLXP4nwcme3Cal8kiKHIXiIz7
         nWsfgDSZKaVTcy8eC+9pDGd9SVK3tlWvLNJCv03wT3U5BspXrn/J2aI/sKY6dphzUFMF
         WvSw6ICn7GXiI1wID0GRtY+gKnw7NYh2EFJ6g=
MIME-Version: 1.0
Received: by 10.150.208.21 with SMTP id f21mr2983819ybg.80.1250285446084; Fri, 
	14 Aug 2009 14:30:46 -0700 (PDT)
In-Reply-To: <ceb609d80908141230w5945ddcdn6f10f92f0e97c04a@mail.gmail.com>
References: <ceb609d80908141230w5945ddcdn6f10f92f0e97c04a@mail.gmail.com>
From: Eelco Hillenius <eelco.hillenius@gmail.com>
Date: Fri, 14 Aug 2009 14:30:26 -0700
Message-ID: <ceb609d80908141430p297c5403l3cc2b3b9123fe7e1@mail.gmail.com>
Subject: Re: comparing sub/ 3rd party projects that abstract map/reduce
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

I found some discussions on this (like
http://www.nabble.com/HBase,-Hive,-Pig-and-other-Hadoop-based-technologies-td19287896.html),
pls disregard my email.

Thanks,

Eelco

On Fri, Aug 14, 2009 at 12:30 PM, Eelco
Hillenius<eelco.hillenius@gmail.com> wrote:
> Hi,
>
> Would people mind sharing their opinions about the relative strengths
> and weaknesses of Hive, Chukwa and Pig (and possibly other
> alternatives)?
>
> I'm only just getting into Hadoop, but it seems to me these libs have
> a considerable overlap depending on what you use them for? I plan to
> check those sub projects out anyway, and I'm definitively not trying
> to start a flame war here, but it would be great to hear some opinions
> about what areas these projects are particularly useful for and what
> might need some work etc.
>
> As a bit of context, we (Teachscape) are considering Hadoop for
> storing audit log files and extracting information from them. The
> audit logging we do is application specific, e.g. user Foo deleted
> Survey X, user Bar moved organization node AAB to AB, etc, and besides
> the need to run a couple of fixed reports weekly (mainly that give our
> customers some insight in how they are using our application), I
> expect us to want to create queries on the fly to e.g. track down
> problems. I don't expect our developers to have a problem writing Map/
> Reduce programs, but I do like the idea of a higher level way of
> extracting information.
>
> Any thoughts would be greatly appreciated,
>
> Eelco
>

From general-return-432-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 15 05:18:54 2009
Return-Path: <general-return-432-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29192 invoked from network); 15 Aug 2009 05:18:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Aug 2009 05:18:53 -0000
Received: (qmail 21288 invoked by uid 500); 15 Aug 2009 05:19:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21187 invoked by uid 500); 15 Aug 2009 05:19:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21176 invoked by uid 99); 15 Aug 2009 05:18:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Aug 2009 05:18:59 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of lamfeeling@126.com designates 220.181.15.112 as permitted sender)
Received: from [220.181.15.112] (HELO m15-112.126.com) (220.181.15.112)
    by apache.org (qpsmtpd/0.29) with SMTP; Sat, 15 Aug 2009 05:18:47 +0000
Received: from SongPC (unknown [114.221.89.98])
	by smtp2 (Coremail) with SMTP id DMmowLD7K2sgRYZKH9xuCQ--.63719S2;
	Sat, 15 Aug 2009 13:18:24 +0800 (CST)
From: =?gb2312?B?wfjLyQ==?= <lamfeeling@126.com>
To: <general@hadoop.apache.org>
References: <ceb609d80908132131h71c9346kce97f2416549f450@mail.gmail.com>
In-Reply-To: <ceb609d80908132131h71c9346kce97f2416549f450@mail.gmail.com>
Subject: [AD]A www crawler for Hadoop Application (Pagerank Computing Included)
Date: Sat, 15 Aug 2009 13:18:23 +0800
Message-ID: <001201ca1d67$cff5f440$6fe1dcc0$@com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="gb2312"
Content-Transfer-Encoding: 7bit
X-Mailer: Microsoft Office Outlook 12.0
Thread-Index: AcocmDTNzHQBzUJfQzGD6VdxJJH3XwAzHNdg
Content-Language: zh-cn
x-cr-hashedpuzzle: Aa2R AyQB BVd4 CGYI CjDk C7aj EOvH Egm8 Fes3 Gex8 HXnI HZdb IVEO JNlE JX0r KGWI;1;ZwBlAG4AZQByAGEAbABAAGgAYQBkAG8AbwBwAC4AYQBwAGEAYwBoAGUALgBvAHIAZwA=;Sosha1_v1;7;{4C244B7A-06C4-4FC8-B1B4-B04A33FD95D2};bABhAG0AZgBlAGUAbABpAG4AZwBAADEAMgA2AC4AYwBvAG0A;Sat, 15 Aug 2009 05:18:20 GMT;WwBBAEQAXQBBACAAdwB3AHcAIABjAHIAYQB3AGwAZQByACAAZgBvAHIAIABIAGEAZABvAG8AcAAgAEEAcABwAGwAaQBjAGEAdABpAG8AbgAgACgAUABhAGcAZQByAGEAbgBrACAAQwBvAG0AcAB1AHQAaQBuAGcAIABJAG4AYwBsAHUAZABlAGQAKQA=
x-cr-puzzleid: {4C244B7A-06C4-4FC8-B1B4-B04A33FD95D2}
X-CM-TRANSID: DMmowLD7K2sgRYZKH9xuCQ--.63719S2
X-Coremail-Antispam: 1Uf129KBjvdXoWrKF4Utry3GF4DZr4DZFWxXrb_yoWkWwb_u3
	sxWrnFgw4kArZ7Xw42vrs0yFZ5W3yFyFWrZwn5Ww17Gr95Z3ykWFZ7KFZxWFW3XrZ0kan5
	Zr1Yqw1Yyry09jkaLaAFLSUrUUUUUbIjqfuFe4nvWSU5nxnvy29KBjDU0xBIdaVrnRJUUU
	qKb7IF0VCYb41lb7IF0VCF04k20xv_Gry3M7k042IE42xK82IY64kIx2x0424lb7IF0VCF
	I7km07C26c804VAKzcIF0wAYjsxI4VWUJwAYFVCjjxCrM7AC8VAFwI0_Jr0_Gr1l1I0E4x
	80FVCIwcAKzIAtM7C26IkvcIIF6IxKo4kEV4yl1IIY67AEw4v_Jr0_Jr4l84ACjcxK6xII
	jxv20xvE14v26w1j6s0DM28EF7xvwVC0I7IYx2IY6xkF7I0E14v26rxl6s0DM28EF7xvwV
	C2z280aVAFwI0_GcCE3s1l84ACjcxK6I8E87Iv6xkF7I0E14v26rxl6s0DM2kK67kvxFCE
	548m6r1fGryUXwAqjxCE34x0Y48IcwAqx4xG64xvF2IEw4CE5I8CrVC2j2WlYx0E2Ix0cI
	8IcVAFwI0_Jr0_Jr4lYx0Ex4A2jsIE14v26r1j6r4UM4x0Y48IcxkI7VAKI48JM4IEnf9E
	lVAFpTB2q-sK649IAas0WaI_GwCjxxvEw4Wlc7Ca8VAvwVA2a4k0FcxrMxkIecxEwVAFwV
	W8WwCY0x0Ix7I2Y4AK6F4j6FyUMxAIw28IcxkI7VAKI48JMxCjnVAqn7xvrwC2zVAF1VAY
	17CE14v26r1j6r15MIIYrxkI7VAKI48JMIIF0xvEx4A2jsIE14v26r1j6r4UMIIF0xvEx4
	A2jsIEc7CjxVAFwI0_Jr0_GrUvcSsGvfC2KfnxnUUI43ZEXa7IUjjYLPUUUUU==
X-CM-SenderInfo: xodpwvxhol0wa6rslhhfrp/
X-Virus-Checked: Checked by ClamAV on apache.org

Dear all:
	These days, I have wrote an hadoop www crawler for my project, and I
hope this little open source program may help you in work.
	
	I originally decided to use the Nutch, however, soon I found it was
so complicated to modify and develop my own work on it.
	So I studied the basic mechanism of Nutch Crawler, and rewrite some
core components with my own code, which is called project Joycrawler.

	I aim to build a functional and convenient tool for some web data
mining jobs and links analysis hadoop programs, so the whole project
Joycrawler is a standard program and can be simply reused in other hadoop
applications.

	It is compatible with hadoop 0.20.0( both Apache and Yahoo's
distribution), and include a Pagerank computing program, which is also a
standard hadoop program.

	I hope following information is helpful for you.

	Project home: http://code.google.com/p/joycrawler/
	Or try it now:
http://joycrawler.googlecode.com/files/joycrawler-0.11.0.zip
	For manual: http://joycrawler.googlecode.com/files/Readme-0.11.1.pdf

	Contact:	lamfeeling@126.com

Best Regards
Song Liu in Soochow University



From general-return-433-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 16 05:33:48 2009
Return-Path: <general-return-433-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57504 invoked from network); 16 Aug 2009 05:33:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Aug 2009 05:33:48 -0000
Received: (qmail 39550 invoked by uid 500); 16 Aug 2009 05:33:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39440 invoked by uid 500); 16 Aug 2009 05:33:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 41960 invoked by uid 99); 11 Aug 2009 01:24:23 -0000
X-ASF-Spam-Status: No, hits=-19.0 required=10.0
	tests=ENV_AND_HDR_SPF_MATCH,RCVD_IN_DNSWL_MED,SPF_PASS,USER_IN_DEF_SPF_WL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of sirota@amazon.com designates 207.171.184.25 as permitted sender)
X-IronPort-AV: E=Sophos;i="4.43,356,1246838400"; 
   d="scan'208";a="253987113"
From: "Sirota, Peter" <sirota@amazon.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 10 Aug 2009 18:23:45 -0700
Subject: Amazon Elastic MapReduce Now Supports Apache Pig 
Thread-Topic: Amazon Elastic MapReduce Now Supports Apache Pig 
Thread-Index: AcoaGcn3+u0GiiWtQOifJVsAt0z+wQAB119QAAAbBkAAACljIA==
Message-ID: <C97921224B64994093A1067F2D08834B176BA9BC12@EX-SEA5-A.ant.amazon.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-cr-puzzleid: {4BBBFED8-7C51-4C23-8AA8-D6B25D35A715}
x-cr-hashedpuzzle: hUM= AWmD AcAA BHNY CPJS DkM5 D4ph EwPR FFiw FLp9 FPUn
 FePK HjOs J3YH KQat
 Kfgv;1;ZwBlAG4AZQByAGEAbABAAGgAYQBkAG8AbwBwAC4AYQBwAGEAYwBoAGUALgBvAHIAZwA=;Sosha1_v1;7;{4BBBFED8-7C51-4C23-8AA8-D6B25D35A715};cwBpAHIAbwB0AGEAQABhAG0AYQB6AG8AbgAuAGMAbwBtAA==;Tue,
 11 Aug 2009 01:23:45
 GMT;QQBtAGEAegBvAG4AIABFAGwAYQBzAHQAaQBjACAATQBhAHAAUgBlAGQAdQBjAGUAIABOAG8AdwAgAFMAdQBwAHAAbwByAHQAcwAgAEEAcABhAGMAaABlACAAUABpAGcA
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Dear Hadoop Users,

We are excited to announce that Amazon Elastic MapReduce now supports Apach=
e Pig - making the service even more compelling for large data set processi=
ng and analytics. Apache Pig is a platform for analyzing large data sets th=
at consists of a high-level language for expressing data analysis programs,=
 coupled with infrastructure for evaluating these programs.

Learn more in the Pig and Amazon Elastic MapReduce tutorial (http://develop=
er.amazonwebservices.com/connect/entry.jspa?externalID=3D2729&categoryID=3D=
269)=20

Watch a video (http://s3.amazonaws.com/awsVideos/AmazonElasticMapReduce/Ela=
sticMapReduce-PigTutorial.html)

Use a sample Apache log processing application (http://developer.amazonwebs=
ervices.com/connect/entry.jspa?externalID=3D2728&categoryID=3D263)=20


Sincerely,
The Amazon Elastic MapReduce Team

From general-return-434-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 17 22:23:35 2009
Return-Path: <general-return-434-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38552 invoked from network); 17 Aug 2009 22:23:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Aug 2009 22:23:34 -0000
Received: (qmail 48529 invoked by uid 500); 17 Aug 2009 21:21:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48109 invoked by uid 500); 17 Aug 2009 21:21:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47924 invoked by uid 99); 17 Aug 2009 21:21:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 21:21:52 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 21:21:42 +0000
Received: from thickbeside-lm.corp.yahoo.com (thickbeside-lm.corp.yahoo.com [10.72.109.129])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n7HLL3wl054767;
	Mon, 17 Aug 2009 14:21:03 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:content-type:
	content-transfer-encoding:subject:mime-version:date:x-mailer;
	b=M8e57rL2JVwVEwPTXNC1W4Rhpg0MQHo2OScyKQlubWco0+Ab6pb+gBblY26nrsXJ
Message-Id: <F98D6F45-DEA7-4B70-8F1B-BFFA423C52C4@yahoo-inc.com>
From: Nigel Daley <ndaley@yahoo-inc.com>
To: general@hadoop.apache.org, common-user@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Subject: Impact of MS search deal on Y! Hadoop development
Mime-Version: 1.0 (Apple Message framework v935.3)
Date: Mon, 17 Aug 2009 14:21:03 -0700
X-Mailer: Apple Mail (2.935.3)
X-Virus-Checked: Checked by ClamAV on apache.org

Ok, this is old news now...but we continue to get questions.  For  
those that didn't see it, Eric Baldeschwieler wrote a blog a couple  
weeks ago on Yahoo!'s continued strong investment in Hadoop.

http://developer.yahoo.net/blogs/hadoop/2009/07/news_flash_hadoop_development.html

Nige

From general-return-435-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 20 13:54:27 2009
Return-Path: <general-return-435-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77109 invoked from network); 20 Aug 2009 13:54:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Aug 2009 13:54:25 -0000
Received: (qmail 5692 invoked by uid 500); 20 Aug 2009 13:54:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5624 invoked by uid 500); 20 Aug 2009 13:54:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5613 invoked by uid 99); 20 Aug 2009 13:54:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 13:54:43 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of hkumar.arora@gmail.com designates 209.85.223.176 as permitted sender)
Received: from [209.85.223.176] (HELO mail-iw0-f176.google.com) (209.85.223.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 13:54:33 +0000
Received: by iwn6 with SMTP id 6so498342iwn.5
        for <general@hadoop.apache.org>; Thu, 20 Aug 2009 06:54:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=73K2tEqxmRPFysHN4Z1kPWDT4Fw/HL7eTkCLQq+FAx8=;
        b=Rqkf1WFpd0haXECJuOT7SYHoWxYl5Al19kzUJsNzEdLqa837ixMlGtbb+2QwzEFZye
         38bqPbbuZpKBv0OpUgcj4jxktayAB+pcop9SKhHpe0r20od3cFeoi1SRp27Tewhffvk5
         U/218LX2MmU3IICf64V4OuACxGsTSkpHgPaBw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=UdR5CS+4VV3ah0eWdNWUXa6bjNHCK3msnf1YuHow40Dgiu1T73O093UmxZhMuGTuXl
         QqP1DQ9oUesE7O2jSgSnDhVD+B+LjaP8SoKgO/KSGWYgU1POKN2OKDAQBW8jvtKM+yRH
         s3mAzVkK9PDf1JB51Wquq51AxdaNz9lmW7MEs=
MIME-Version: 1.0
Received: by 10.231.37.141 with SMTP id x13mr4469538ibd.39.1250776451808; Thu, 
	20 Aug 2009 06:54:11 -0700 (PDT)
Date: Thu, 20 Aug 2009 16:54:11 +0300
Message-ID: <e497591c0908200654kb6ba97gb2b8a63fc2d263d6@mail.gmail.com>
Subject: Invalid argument for option USER_DATA_FILE
From: Harshit Kumar <hkumar.arora@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0003255769461a8000047193162b
X-Virus-Checked: Checked by ClamAV on apache.org

--0003255769461a8000047193162b
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

Hi
When I try to execute *hadoop-ec2 launch-cluster test-cluster 2*, it
executes, but keep waiting at "Waiting for instance to start", find below
the exact display as it shows on my screen


$ bin/hadoop-ec2 launch-cluster test-cluster 2
Testing for existing master in group: test-cluster
Creating group test-cluster-master
GROUP   test-cluster-master     Group for Hadoop Master.
GROUP           test-cluster-master
PERMISSION              test-cluster-master     ALLOWS  all
FROM    USER    5282-1142-6451  GRPNAME test-cluster-master
GROUP           test-cluster-master
PERMISSION              test-cluster-master     ALLOWS  tcp     22      22
FROM    CIDR    0.0.0.0/0
GROUP           test-cluster-master
PERMISSION              test-cluster-master     ALLOWS  tcp     50030
50030
FROM    CIDR    0.0.0.0/0
GROUP           test-cluster-master
PERMISSION              test-cluster-master     ALLOWS  tcp     50060
50060
FROM    CIDR    0.0.0.0/0
Creating group test-cluster
GROUP   test-cluster    Group for Hadoop Slaves.
GROUP           test-cluster
PERMISSION              test-cluster    ALLOWS  all                     FROM
USER    5282-1142-6451  GRPNAME test-cluster
GROUP           test-cluster
PERMISSION              test-cluster    ALLOWS  tcp     22      22      FROM
CIDR    0.0.0.0/0
GROUP           test-cluster
PERMISSION              test-cluster    ALLOWS  tcp     50030   50030   FROM
CIDR    0.0.0.0/0
GROUP           test-cluster
PERMISSION              test-cluster    ALLOWS  tcp     50060   50060   FROM
CIDR    0.0.0.0/0
GROUP           test-cluster-master
PERMISSION              test-cluster-master     ALLOWS  all
FROM    USER    5282-1142-6451  GRPNAME test-cluster
GROUP           test-cluster
PERMISSION              test-cluster    ALLOWS  all                     FROM
USER    5282-1142-6451  GRPNAME test-cluster-master
Starting master with AMI ami-fa6a8e93
Invalid argument for option '-f, --user-data-file DATA-FILE':
'/home/bike/hadoo
-0.19.2/src/contrib/ec2/bin/hadoop-ec2-init-remote.sh' (-h for usage)
Waiting for instance  to start
...............................................................................
...............................................................................

It just keeps on producing new dots, thats it and i guess this process will
never finish,

I tried hours to search for the solution on the net, but unsuccessful.

I will appreciate if anyone can  help me with this problem?

H. Kumar
Phone(Mobile): +82-10-2892-9663
Phone(Office): +82-31-
skype: harshit900
Blog: http://harshitkumar.wordpress.com
Website: http:/kumarharmuscat.tripod.com

--0003255769461a8000047193162b--

From general-return-436-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 20 14:21:28 2009
Return-Path: <general-return-436-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94324 invoked from network); 20 Aug 2009 14:21:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Aug 2009 14:21:24 -0000
Received: (qmail 74477 invoked by uid 500); 20 Aug 2009 14:21:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74420 invoked by uid 500); 20 Aug 2009 14:21:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74410 invoked by uid 99); 20 Aug 2009 14:21:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 14:21:42 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [209.85.219.209] (HELO mail-ew0-f209.google.com) (209.85.219.209)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 14:21:31 +0000
Received: by ewy5 with SMTP id 5so196845ewy.36
        for <general@hadoop.apache.org>; Thu, 20 Aug 2009 07:21:10 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.21.205 with SMTP id r55mr1901850wer.175.1250778068719; 
	Thu, 20 Aug 2009 07:21:08 -0700 (PDT)
In-Reply-To: <e497591c0908200654kb6ba97gb2b8a63fc2d263d6@mail.gmail.com>
References: <e497591c0908200654kb6ba97gb2b8a63fc2d263d6@mail.gmail.com>
Date: Thu, 20 Aug 2009 15:21:08 +0100
Message-ID: <ac79ea400908200721s44499077na7a7f7f5742a85dd@mail.gmail.com>
Subject: Re: Invalid argument for option USER_DATA_FILE
From: Tom White <tom@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Harshit,

Please see comment below.

Cheers,
Tom

P.S. Questions like this are best sent to common-user.

On Thu, Aug 20, 2009 at 2:54 PM, Harshit Kumar<hkumar.arora@gmail.com> wrot=
e:
> Hi
> When I try to execute *hadoop-ec2 launch-cluster test-cluster 2*, it
> executes, but keep waiting at "Waiting for instance to start", find below
> the exact display as it shows on my screen
>
>
> $ bin/hadoop-ec2 launch-cluster test-cluster 2
> Testing for existing master in group: test-cluster
> Creating group test-cluster-master
> GROUP =A0 test-cluster-master =A0 =A0 Group for Hadoop Master.
> GROUP =A0 =A0 =A0 =A0 =A0 test-cluster-master
> PERMISSION =A0 =A0 =A0 =A0 =A0 =A0 =A0test-cluster-master =A0 =A0 ALLOWS =
=A0all
> FROM =A0 =A0USER =A0 =A05282-1142-6451 =A0GRPNAME test-cluster-master
> GROUP =A0 =A0 =A0 =A0 =A0 test-cluster-master
> PERMISSION =A0 =A0 =A0 =A0 =A0 =A0 =A0test-cluster-master =A0 =A0 ALLOWS =
=A0tcp =A0 =A0 22 =A0 =A0 =A022
> FROM =A0 =A0CIDR =A0 =A00.0.0.0/0
> GROUP =A0 =A0 =A0 =A0 =A0 test-cluster-master
> PERMISSION =A0 =A0 =A0 =A0 =A0 =A0 =A0test-cluster-master =A0 =A0 ALLOWS =
=A0tcp =A0 =A0 50030
> 50030
> FROM =A0 =A0CIDR =A0 =A00.0.0.0/0
> GROUP =A0 =A0 =A0 =A0 =A0 test-cluster-master
> PERMISSION =A0 =A0 =A0 =A0 =A0 =A0 =A0test-cluster-master =A0 =A0 ALLOWS =
=A0tcp =A0 =A0 50060
> 50060
> FROM =A0 =A0CIDR =A0 =A00.0.0.0/0
> Creating group test-cluster
> GROUP =A0 test-cluster =A0 =A0Group for Hadoop Slaves.
> GROUP =A0 =A0 =A0 =A0 =A0 test-cluster
> PERMISSION =A0 =A0 =A0 =A0 =A0 =A0 =A0test-cluster =A0 =A0ALLOWS =A0all =
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 FROM
> USER =A0 =A05282-1142-6451 =A0GRPNAME test-cluster
> GROUP =A0 =A0 =A0 =A0 =A0 test-cluster
> PERMISSION =A0 =A0 =A0 =A0 =A0 =A0 =A0test-cluster =A0 =A0ALLOWS =A0tcp =
=A0 =A0 22 =A0 =A0 =A022 =A0 =A0 =A0FROM
> CIDR =A0 =A00.0.0.0/0
> GROUP =A0 =A0 =A0 =A0 =A0 test-cluster
> PERMISSION =A0 =A0 =A0 =A0 =A0 =A0 =A0test-cluster =A0 =A0ALLOWS =A0tcp =
=A0 =A0 50030 =A0 50030 =A0 FROM
> CIDR =A0 =A00.0.0.0/0
> GROUP =A0 =A0 =A0 =A0 =A0 test-cluster
> PERMISSION =A0 =A0 =A0 =A0 =A0 =A0 =A0test-cluster =A0 =A0ALLOWS =A0tcp =
=A0 =A0 50060 =A0 50060 =A0 FROM
> CIDR =A0 =A00.0.0.0/0
> GROUP =A0 =A0 =A0 =A0 =A0 test-cluster-master
> PERMISSION =A0 =A0 =A0 =A0 =A0 =A0 =A0test-cluster-master =A0 =A0 ALLOWS =
=A0all
> FROM =A0 =A0USER =A0 =A05282-1142-6451 =A0GRPNAME test-cluster
> GROUP =A0 =A0 =A0 =A0 =A0 test-cluster
> PERMISSION =A0 =A0 =A0 =A0 =A0 =A0 =A0test-cluster =A0 =A0ALLOWS =A0all =
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 FROM
> USER =A0 =A05282-1142-6451 =A0GRPNAME test-cluster-master
> Starting master with AMI ami-fa6a8e93
> Invalid argument for option '-f, --user-data-file DATA-FILE':
> '/home/bike/hadoo
> -0.19.2/src/contrib/ec2/bin/hadoop-ec2-init-remote.sh' (-h for usage)

The path looks odd here: a missing "p" in "hadoop", and it's also
split over two lines. What is the actual location of the
hadoop-ec2-init-remote.sh file on your machine?

> Waiting for instance =A0to start
> .........................................................................=
......
> .........................................................................=
......
>
> It just keeps on producing new dots, thats it and i guess this process wi=
ll
> never finish,
>
> I tried hours to search for the solution on the net, but unsuccessful.
>
> I will appreciate if anyone can =A0help me with this problem?
>
> H. Kumar
> Phone(Mobile): +82-10-2892-9663
> Phone(Office): +82-31-
> skype: harshit900
> Blog: http://harshitkumar.wordpress.com
> Website: http:/kumarharmuscat.tripod.com
>

From general-return-437-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 20 15:57:36 2009
Return-Path: <general-return-437-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38831 invoked from network); 20 Aug 2009 15:57:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Aug 2009 15:57:35 -0000
Received: (qmail 33182 invoked by uid 500); 20 Aug 2009 15:57:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33119 invoked by uid 500); 20 Aug 2009 15:57:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33109 invoked by uid 99); 20 Aug 2009 15:57:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 15:57:53 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of hkumar.arora@gmail.com designates 209.85.223.176 as permitted sender)
Received: from [209.85.223.176] (HELO mail-iw0-f176.google.com) (209.85.223.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 15:57:43 +0000
Received: by iwn6 with SMTP id 6so545861iwn.5
        for <general@hadoop.apache.org>; Thu, 20 Aug 2009 08:57:22 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=Qnl/01o7KGwnzUlRlhKSNtxpIfHm2UN/33C1yn22jrQ=;
        b=O+ev4ca/VfPCYgixSlTaymOocW/ece7wyrldwfGrkbYh0oYJbn9iUfQEG5sg9cIAGc
         xM9EmiC/hwnBVIAN/YNrLe9Wc7ZMuZB5nIEn43dnvfCH1ICbDJBgBQ8EM8szMMvmXwDz
         05mprI7eveLZsDmsoc8mT6lO4rHRMul9GnrxA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=QO99XJbSpVTToFt6stQTYisVHXJdmf9iChJLY1KFShdaGEH27DDIp3Qv5RSZZLes5z
         yHckiSp9DoTlZqhnJ4aBf9XNgJDzfjCF7fpvDZevygW+cucy/eeJxFiSBw7pl1q+5tcl
         cnKWT2KpJ6QHThJKRtZq4b8k8RoNTE27K3f70=
MIME-Version: 1.0
Received: by 10.231.34.3 with SMTP id j3mr4542833ibd.43.1250783842559; Thu, 20 
	Aug 2009 08:57:22 -0700 (PDT)
In-Reply-To: <ac79ea400908200721s44499077na7a7f7f5742a85dd@mail.gmail.com>
References: <e497591c0908200654kb6ba97gb2b8a63fc2d263d6@mail.gmail.com>
	 <ac79ea400908200721s44499077na7a7f7f5742a85dd@mail.gmail.com>
Date: Thu, 20 Aug 2009 18:57:22 +0300
Message-ID: <e497591c0908200857u28b2c85cx12d9e6bd8d246f06@mail.gmail.com>
Subject: Re: Invalid argument for option USER_DATA_FILE
From: Harshit Kumar <hkumar.arora@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00221532cda4a06b3d047194ce26
X-Virus-Checked: Checked by ClamAV on apache.org

--00221532cda4a06b3d047194ce26
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

Thanks Tom for your reply.
Actually that missing 'p' is bcos i copied it wrongly.

Here i paste the error message again

Starting master with AMI ami-fa6a8e93
Invalid argument for option '-f, --user-data-file DATA-FILE':
'/home/bike/hadoop-0.19.2/src/contrib/ec2/bin/hadoop-ec2-init-remote.sh' (-h
for usage)

The actual location of adoop-ec2-init-remote.sh  is this
C:\cygwin\home\bike\hadoop-0.19.2\src\contrib\ec2\bin

Thanks once again for responding.

I am also posting the original msg at common-user@hadoop.apache.org, thanks
for letting me know.

Regards
H. Kumar
Phone(Mobile): +82-10-2892-9663
Phone(Office): +82-31-
skype: harshit900
Blog: http://harshitkumar.wordpress.com
Website: http:/kumarharmuscat.tripod.com


2009/8/20 Tom White <tom@cloudera.com>

> Hi Harshit,
>
> Please see comment below.
>
> Cheers,
> Tom
>
> P.S. Questions like this are best sent to common-user.
>
> On Thu, Aug 20, 2009 at 2:54 PM, Harshit Kumar<hkumar.arora@gmail.com>
> wrote:
> > Hi
> > When I try to execute *hadoop-ec2 launch-cluster test-cluster 2*, it
> > executes, but keep waiting at "Waiting for instance to start", find below
> > the exact display as it shows on my screen
> >
> >
> > $ bin/hadoop-ec2 launch-cluster test-cluster 2
> > Testing for existing master in group: test-cluster
> > Creating group test-cluster-master
> > GROUP   test-cluster-master     Group for Hadoop Master.
> > GROUP           test-cluster-master
> > PERMISSION              test-cluster-master     ALLOWS  all
> > FROM    USER    5282-1142-6451  GRPNAME test-cluster-master
> > GROUP           test-cluster-master
> > PERMISSION              test-cluster-master     ALLOWS  tcp     22
>  22
> > FROM    CIDR    0.0.0.0/0
> > GROUP           test-cluster-master
> > PERMISSION              test-cluster-master     ALLOWS  tcp     50030
> > 50030
> > FROM    CIDR    0.0.0.0/0
> > GROUP           test-cluster-master
> > PERMISSION              test-cluster-master     ALLOWS  tcp     50060
> > 50060
> > FROM    CIDR    0.0.0.0/0
> > Creating group test-cluster
> > GROUP   test-cluster    Group for Hadoop Slaves.
> > GROUP           test-cluster
> > PERMISSION              test-cluster    ALLOWS  all
> FROM
> > USER    5282-1142-6451  GRPNAME test-cluster
> > GROUP           test-cluster
> > PERMISSION              test-cluster    ALLOWS  tcp     22      22
>  FROM
> > CIDR    0.0.0.0/0
> > GROUP           test-cluster
> > PERMISSION              test-cluster    ALLOWS  tcp     50030   50030
> FROM
> > CIDR    0.0.0.0/0
> > GROUP           test-cluster
> > PERMISSION              test-cluster    ALLOWS  tcp     50060   50060
> FROM
> > CIDR    0.0.0.0/0
> > GROUP           test-cluster-master
> > PERMISSION              test-cluster-master     ALLOWS  all
> > FROM    USER    5282-1142-6451  GRPNAME test-cluster
> > GROUP           test-cluster
> > PERMISSION              test-cluster    ALLOWS  all
> FROM
> > USER    5282-1142-6451  GRPNAME test-cluster-master
> > Starting master with AMI ami-fa6a8e93
> > Invalid argument for option '-f, --user-data-file DATA-FILE':
> > '/home/bike/hadoo
> > -0.19.2/src/contrib/ec2/bin/hadoop-ec2-init-remote.sh' (-h for usage)
>
> The path looks odd here: a missing "p" in "hadoop", and it's also
> split over two lines. What is the actual location of the
> hadoop-ec2-init-remote.sh file on your machine?
>
> > Waiting for instance  to start
> >
> ...............................................................................
> >
> ...............................................................................
> >
> > It just keeps on producing new dots, thats it and i guess this process
> will
> > never finish,
> >
> > I tried hours to search for the solution on the net, but unsuccessful.
> >
> > I will appreciate if anyone can  help me with this problem?
> >
> > H. Kumar
> > Phone(Mobile): +82-10-2892-9663
> > Phone(Office): +82-31-
> > skype: harshit900
> > Blog: http://harshitkumar.wordpress.com
> > Website: http:/kumarharmuscat.tripod.com
> >
>

--00221532cda4a06b3d047194ce26--

From general-return-440-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 22 06:33:09 2009
Return-Path: <general-return-440-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46298 invoked from network); 22 Aug 2009 06:33:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Aug 2009 06:33:09 -0000
Received: (qmail 48352 invoked by uid 500); 22 Aug 2009 06:33:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48289 invoked by uid 500); 22 Aug 2009 06:33:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48279 invoked by uid 99); 22 Aug 2009 06:33:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 Aug 2009 06:33:30 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.30.56] (HELO QMTA06.emeryville.ca.mail.comcast.net) (76.96.30.56)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 Aug 2009 06:33:19 +0000
Received: from OMTA02.emeryville.ca.mail.comcast.net ([76.96.30.19])
	by QMTA06.emeryville.ca.mail.comcast.net with comcast
	id XJN01c0030QkzPwA6JYyMp; Sat, 22 Aug 2009 06:32:58 +0000
Received: from [10.0.0.59] ([98.234.216.58])
	by OMTA02.emeryville.ca.mail.comcast.net with comcast
	id XJYt1c0041GAoR68NJYxGa; Sat, 22 Aug 2009 06:32:58 +0000
Message-Id: <2B8EEA26-46D4-4D77-997A-A3394790613F@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: [VOTE] Create hbase-issues email list
Date: Fri, 21 Aug 2009 23:32:52 -0700
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

I think we should create an hbase-issues lists to track that detailed  
jira traffic and reduce the traffic on hbase-dev. It will replicate  
the structure that we have in common, hdfs, and mapreduce. Clearly,  
I'm +1.

-- Owen

From general-return-441-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 22 06:58:44 2009
Return-Path: <general-return-441-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 48148 invoked from network); 22 Aug 2009 06:58:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Aug 2009 06:58:43 -0000
Received: (qmail 53998 invoked by uid 500); 22 Aug 2009 06:59:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53913 invoked by uid 500); 22 Aug 2009 06:59:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53903 invoked by uid 99); 22 Aug 2009 06:59:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 Aug 2009 06:59:05 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.198.238] (HELO rv-out-0506.google.com) (209.85.198.238)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 Aug 2009 06:58:56 +0000
Received: by rv-out-0506.google.com with SMTP id k40so398141rvb.29
        for <general@hadoop.apache.org>; Fri, 21 Aug 2009 23:58:33 -0700 (PDT)
Received: by 10.140.188.15 with SMTP id l15mr1223809rvf.119.1250924313810;
        Fri, 21 Aug 2009 23:58:33 -0700 (PDT)
Received: from ?192.168.42.100? (75-101-123-136.dsl.static.sonic.net [75.101.123.136])
        by mx.google.com with ESMTPS id k2sm4396478rvb.3.2009.08.21.23.58.31
        (version=SSLv3 cipher=RC4-MD5);
        Fri, 21 Aug 2009 23:58:32 -0700 (PDT)
Message-ID: <4A8F970D.7030804@cloudera.com>
Date: Fri, 21 Aug 2009 23:58:21 -0700
From: Amr Awadallah <aaa@cloudera.com>
Organization: Cloudera, Inc.
User-Agent: Thunderbird 2.0.0.23 (Windows/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Create hbase-issues email list
References: <2B8EEA26-46D4-4D77-997A-A3394790613F@apache.org>
In-Reply-To: <2B8EEA26-46D4-4D77-997A-A3394790613F@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Owen O'Malley wrote:
> I think we should create an hbase-issues lists to track that detailed 
> jira traffic and reduce the traffic on hbase-dev. It will replicate 
> the structure that we have in common, hdfs, and mapreduce. Clearly, 
> I'm +1.
>
> -- Owen

From general-return-442-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 23 02:11:43 2009
Return-Path: <general-return-442-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12838 invoked from network); 23 Aug 2009 02:11:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Aug 2009 02:11:43 -0000
Received: (qmail 2329 invoked by uid 500); 22 Aug 2009 22:12:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2259 invoked by uid 500); 22 Aug 2009 22:12:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2249 invoked by uid 99); 22 Aug 2009 22:12:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 Aug 2009 22:12:05 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 Aug 2009 22:11:55 +0000
Received: from [192.168.1.64] (snvvpn3-10-72-76-c95.hq.corp.yahoo.com [10.72.76.95])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n7MM9vUa036858
	for <general@hadoop.apache.org>; Sat, 22 Aug 2009 15:09:57 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=Lv6vMjCCqwYmSLfaYyLivUPiuv1Z0iW6L8JXeo4MWWIi+/Osys+qYFnuUVYUrZuZ
Message-Id: <D5A89B61-4CD1-4917-9DD7-F1D43A0C35CB@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <2B8EEA26-46D4-4D77-997A-A3394790613F@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Create hbase-issues email list
Date: Sat, 22 Aug 2009 15:09:58 -0700
References: <2B8EEA26-46D4-4D77-997A-A3394790613F@apache.org>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Arun

On Aug 21, 2009, at 11:32 PM, Owen O'Malley wrote:

> I think we should create an hbase-issues lists to track that  
> detailed jira traffic and reduce the traffic on hbase-dev. It will  
> replicate the structure that we have in common, hdfs, and mapreduce.  
> Clearly, I'm +1.
>
> -- Owen


From general-return-443-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 23 02:34:59 2009
Return-Path: <general-return-443-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23261 invoked from network); 23 Aug 2009 02:34:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Aug 2009 02:34:59 -0000
Received: (qmail 26877 invoked by uid 500); 23 Aug 2009 02:35:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26803 invoked by uid 500); 23 Aug 2009 02:35:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26792 invoked by uid 99); 23 Aug 2009 02:35:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 23 Aug 2009 02:35:21 +0000
X-ASF-Spam-Status: No, hits=-8.0 required=10.0
	tests=RCVD_IN_DNSWL_HI,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of Jim.Kellerman@microsoft.com designates 131.107.115.214 as permitted sender)
Received: from [131.107.115.214] (HELO smtp.microsoft.com) (131.107.115.214)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 23 Aug 2009 02:35:11 +0000
Received: from TK5EX14MLTC102.redmond.corp.microsoft.com (157.54.79.180) by
 TK5-EXGWY-E803.partners.extranet.microsoft.com (10.251.56.169) with Microsoft
 SMTP Server (TLS) id 8.2.176.0; Sat, 22 Aug 2009 19:34:48 -0700
Received: from TK5EX14MBXC118.redmond.corp.microsoft.com ([157.54.97.26]) by
 TK5EX14MLTC102.redmond.corp.microsoft.com ([157.54.79.180]) with mapi; Sat,
 22 Aug 2009 19:34:42 -0700
From: "Jim Kellerman (POWERSET)" <Jim.Kellerman@microsoft.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: RE: [VOTE] Create hbase-issues email list
Thread-Topic: [VOTE] Create hbase-issues email list
Thread-Index: AQHKIvKS7HhDXaoVk0W8RJwKtjpmUJCy7pYA
Date: Sun, 23 Aug 2009 02:34:39 +0000
Message-ID: <E9C6F5B134FDA3468879CB4E533CF1C80EC71A@TK5EX14MBXC118.redmond.corp.microsoft.com>
References: <2B8EEA26-46D4-4D77-997A-A3394790613F@apache.org>
In-Reply-To: <2B8EEA26-46D4-4D77-997A-A3394790613F@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

+1

-----Original Message-----
From: Owen O'Malley [mailto:omalley@apache.org]
Sent: Friday, August 21, 2009 11:33 PM
To: general@hadoop.apache.org
Subject: [VOTE] Create hbase-issues email list

I think we should create an hbase-issues lists to track that detailed jira =
traffic and reduce the traffic on hbase-dev. It will replicate the structur=
e that we have in common, hdfs, and mapreduce. Clearly, I'm +1.

-- Owen


From general-return-444-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 23 03:07:36 2009
Return-Path: <general-return-444-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35453 invoked from network); 23 Aug 2009 03:07:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Aug 2009 03:07:36 -0000
Received: (qmail 44900 invoked by uid 500); 23 Aug 2009 03:07:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44711 invoked by uid 500); 23 Aug 2009 03:07:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44701 invoked by uid 99); 23 Aug 2009 03:07:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 23 Aug 2009 03:07:57 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of hdspork@gmail.com designates 209.85.223.176 as permitted sender)
Received: from [209.85.223.176] (HELO mail-iw0-f176.google.com) (209.85.223.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 23 Aug 2009 03:07:48 +0000
Received: by iwn6 with SMTP id 6so700824iwn.5
        for <general@hadoop.apache.org>; Sat, 22 Aug 2009 20:07:27 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=w9bskveKmJMoDnzrhySe1KmUrxhPRrf1z2ECOcthjc0=;
        b=Pnnz7kiaZVSt1FiS4jLGZek5LDLvXOCY9QuQ00M42Bqf5kfWdVYLTxrU1/nuGxFQGa
         6OUD1kOUlyiwwv7i/D7QP72U6shETKn5xGgwuFsK7TNVwqMYSGbHqzfvxfCVOsK0cOTJ
         evVE2AaP6e8FsRJx101tUtTG/3twQKVSHSMFU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=vl5HTFziUXS21+js2c9fPL2cgowRxd4PgNJrn3/9hXcclwXTMoW2XIkIIzJ6SJSO7b
         2u7gp8ezmFeFPCQq3DAvzLN36A7rXOnPV2l+2YS3JylgXmcy14s6vNVU8lrWe1Bt/wug
         RnPfxOeAqZ7bV1If+jxnAYHzyf7Fk6zGU7NV8=
MIME-Version: 1.0
Received: by 10.231.18.69 with SMTP id v5mr1492738iba.26.1250996847244; Sat, 
	22 Aug 2009 20:07:27 -0700 (PDT)
In-Reply-To: <E9C6F5B134FDA3468879CB4E533CF1C80EC71A@TK5EX14MBXC118.redmond.corp.microsoft.com>
References: <2B8EEA26-46D4-4D77-997A-A3394790613F@apache.org>
	 <E9C6F5B134FDA3468879CB4E533CF1C80EC71A@TK5EX14MBXC118.redmond.corp.microsoft.com>
Date: Sun, 23 Aug 2009 11:07:27 +0800
Message-ID: <dd4ea7200908222007k54345569v32a99edd4d0330f6@mail.gmail.com>
Subject: Re: [VOTE] Create hbase-issues email list
From: He ZY <hdspork@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00221504666fb1f6bc0471c6660c
X-Virus-Checked: Checked by ClamAV on apache.org

--00221504666fb1f6bc0471c6660c
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

+1

On Sun, Aug 23, 2009 at 10:34 AM, Jim Kellerman (POWERSET) <
Jim.Kellerman@microsoft.com> wrote:

> +1
>
> -----Original Message-----
> From: Owen O'Malley [mailto:omalley@apache.org]
> Sent: Friday, August 21, 2009 11:33 PM
> To: general@hadoop.apache.org
> Subject: [VOTE] Create hbase-issues email list
>
> I think we should create an hbase-issues lists to track that detailed jira
> traffic and reduce the traffic on hbase-dev. It will replicate the structure
> that we have in common, hdfs, and mapreduce. Clearly, I'm +1.
>
> -- Owen
>
>

--00221504666fb1f6bc0471c6660c--

From general-return-445-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 23 06:18:57 2009
Return-Path: <general-return-445-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 52749 invoked from network); 23 Aug 2009 06:18:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Aug 2009 06:18:57 -0000
Received: (qmail 10899 invoked by uid 500); 23 Aug 2009 06:19:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 10819 invoked by uid 500); 23 Aug 2009 06:19:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 10808 invoked by uid 99); 23 Aug 2009 06:19:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 23 Aug 2009 06:19:18 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of digvijaysinghshaktawat@gmail.com designates 209.85.216.190 as permitted sender)
Received: from [209.85.216.190] (HELO mail-px0-f190.google.com) (209.85.216.190)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 23 Aug 2009 06:19:09 +0000
Received: by pxi28 with SMTP id 28so4643913pxi.2
        for <general@hadoop.apache.org>; Sat, 22 Aug 2009 23:18:48 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=VZlmbFlYdTd5ZpvkEHaxBF2u7ALUjQVRXNgMMp01uE4=;
        b=FeLZLfhSp1qe/8LQK/fR9p4r2WWGMb5QVPzfnwCRQrOKSn7AtQpyCmEJLsbacoWA6k
         gPTgCnUGGMAuM8j+MwxWkDFgKmM25d5HPZTpaAJBMuYvpxdbxkCARjufh/rAowhdvSpO
         Lh2i6pO/6z3qUfINXbplZdpWtQH5WMmw3+D6s=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=q6VflqMvZUzVKjIKtU6qlnY6ei7M8lJHoSmor3iz4Cn5hWi1+0GfFBMoyCk7LBjheH
         XG8jYEo/nnX1F4RR8Lg8G4oC5YSJYQnQZxOMxUKrbwPrORjkbUyFgbcHDJRmR62FBY5K
         psIKKkX7DSCeUvmpnRrtX4gmZp7CCCViHwE30=
MIME-Version: 1.0
Received: by 10.143.129.3 with SMTP id g3mr304156wfn.240.1251008328242; Sat, 
	22 Aug 2009 23:18:48 -0700 (PDT)
Date: Sun, 23 Aug 2009 11:48:48 +0530
Message-ID: <bf8befe30908222318y12193b09h7b8e5dfeb22f877d@mail.gmail.com>
Subject: Need Guidance
From: digvijay singh shaktawat <digvijaysinghshaktawat@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd5c38004179f0471c91300
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd5c38004179f0471c91300
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi,
     We are two undergraduate student and have are determined to do a
project in cloud computing as are final Btech project.  We are absolutely
new to hadoop and have arround six months to work on it. We wanted
suggestions regarding some interesting work possible with hadoop.

-Digvijay Singh
-Pushpak Aggarwal

--000e0cd5c38004179f0471c91300--

From general-return-446-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 23 08:58:12 2009
Return-Path: <general-return-446-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 61854 invoked from network); 23 Aug 2009 08:58:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Aug 2009 08:58:12 -0000
Received: (qmail 53410 invoked by uid 500); 23 Aug 2009 08:58:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53317 invoked by uid 500); 23 Aug 2009 08:58:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53307 invoked by uid 99); 23 Aug 2009 08:58:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 23 Aug 2009 08:58:33 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of tzehon.tan@gmail.com designates 209.85.216.190 as permitted sender)
Received: from [209.85.216.190] (HELO mail-px0-f190.google.com) (209.85.216.190)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 23 Aug 2009 08:58:25 +0000
Received: by pxi28 with SMTP id 28so4673265pxi.2
        for <general@hadoop.apache.org>; Sun, 23 Aug 2009 01:58:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=VmroeTLz2X00uKQPzFyv0f3S8wTh8TP/qT29s8GKOS0=;
        b=g3+td+J14uFxg8LBT++2/v2K4/+MehC7A/NnZnHAVHXnUROqtArl32zQvq9YBuTugY
         pkB8U1JzYz9/VaUj+f9WpH0PFEBjc01FcR3EYCAlrgwEFSMIQjn20CcamUULBYma1hiP
         KqVplTkSSP+890nZoZLmeEEovW8YtHJtXbfZ4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=XvBoT4YaFyWjUO2H/grkm83m/h/Sx0GU/L1eCX8UDxyk2HKcnMqELvqLDkdFcVqcaZ
         wTsOb/Avn78ig7BmzLCujPBoKs61sYkp9eyFavrISS3I0NV5liWEKQ1CMPyP6XeRfWIO
         sfRzxW4Jp8tH15NatwGcgZFoA1kylMqCh2//A=
MIME-Version: 1.0
Received: by 10.142.249.41 with SMTP id w41mr259969wfh.347.1251017885320; Sun, 
	23 Aug 2009 01:58:05 -0700 (PDT)
In-Reply-To: <2B8EEA26-46D4-4D77-997A-A3394790613F@apache.org>
References: <2B8EEA26-46D4-4D77-997A-A3394790613F@apache.org>
Date: Sun, 23 Aug 2009 16:58:05 +0800
Message-ID: <fca34e2c0908230158n3afa4975x24b03750f2978513@mail.gmail.com>
Subject: Re: [VOTE] Create hbase-issues email list
From: Tze Hon Tan <tzehon.tan@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636ed6870a9851e0471cb4cac
X-Virus-Checked: Checked by ClamAV on apache.org

--001636ed6870a9851e0471cb4cac
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

+1

On Sat, Aug 22, 2009 at 2:32 PM, Owen O'Malley <omalley@apache.org> wrote:

> I think we should create an hbase-issues lists to track that detailed jira
> traffic and reduce the traffic on hbase-dev. It will replicate the structure
> that we have in common, hdfs, and mapreduce. Clearly, I'm +1.
>
> -- Owen
>

--001636ed6870a9851e0471cb4cac--

From general-return-447-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 23 09:36:24 2009
Return-Path: <general-return-447-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 65196 invoked from network); 23 Aug 2009 09:36:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Aug 2009 09:36:24 -0000
Received: (qmail 60594 invoked by uid 500); 23 Aug 2009 09:36:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60508 invoked by uid 500); 23 Aug 2009 09:36:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60497 invoked by uid 99); 23 Aug 2009 09:36:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 23 Aug 2009 09:36:45 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yeahyf@gmail.com designates 209.85.218.214 as permitted sender)
Received: from [209.85.218.214] (HELO mail-bw0-f214.google.com) (209.85.218.214)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 23 Aug 2009 09:36:37 +0000
Received: by bwz10 with SMTP id 10so1079330bwz.29
        for <general@hadoop.apache.org>; Sun, 23 Aug 2009 02:36:15 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=epKCdkgFyGGRfEEyGcrZtDyazPU3ZRzmHB/JTbSHMlU=;
        b=mfnDHfMwoCnFsysVxmVqggkTPvPDmDs0pObePvaDihO1jCtg9z08bWvIvCMi0Z7bmJ
         sMrBMXlfq5How+5sBN2uodm9vS5fbPG51jhamlIakiIYMph4i+KLqohVZ8589jgCPh79
         8rid+M5odxOa456CBjL4y4NEoIUb/g2ELWnF8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=WZiO6KYnObEPgg8OrNVa0QgT/ECs6jwLng9keGEeoPVz4xZi4U1TpUKz5/wBb03Oy3
         uZF6lFIjACIiUhGUYu8oAovMyylG6xwAwoX0so3XSYcdLDnvQgHtgoTlx4CRMOW7QuFa
         96MXF0hEl4LAQoJf5ylUOfXEkQy3Bm3/BxUbc=
MIME-Version: 1.0
Received: by 10.239.141.142 with SMTP id c14mr348829hba.1.1251020175683; Sun, 
	23 Aug 2009 02:36:15 -0700 (PDT)
In-Reply-To: <fca34e2c0908230158n3afa4975x24b03750f2978513@mail.gmail.com>
References: <2B8EEA26-46D4-4D77-997A-A3394790613F@apache.org>
	 <fca34e2c0908230158n3afa4975x24b03750f2978513@mail.gmail.com>
Date: Sun, 23 Aug 2009 17:36:15 +0800
Message-ID: <3b13e4710908230236h16e6b5ccrd3171a7d240de9fa@mail.gmail.com>
Subject: Re: [VOTE] Create hbase-issues email list
From: yangfeng <yeahyf@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f6c5882db2040471cbd5ca
X-Virus-Checked: Checked by ClamAV on apache.org

--001485f6c5882db2040471cbd5ca
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

+1

2009/8/23 Tze Hon Tan <tzehon.tan@gmail.com>

> +1
>
> On Sat, Aug 22, 2009 at 2:32 PM, Owen O'Malley <omalley@apache.org> wrote:
>
> > I think we should create an hbase-issues lists to track that detailed
> jira
> > traffic and reduce the traffic on hbase-dev. It will replicate the
> structure
> > that we have in common, hdfs, and mapreduce. Clearly, I'm +1.
> >
> > -- Owen
> >
>

--001485f6c5882db2040471cbd5ca--

From general-return-448-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 23 15:53:57 2009
Return-Path: <general-return-448-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 15283 invoked from network); 23 Aug 2009 15:53:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Aug 2009 15:53:57 -0000
Received: (qmail 75448 invoked by uid 500); 23 Aug 2009 15:54:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75375 invoked by uid 500); 23 Aug 2009 15:54:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75365 invoked by uid 99); 23 Aug 2009 15:54:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 23 Aug 2009 15:54:17 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of luduling@gmail.com designates 209.85.222.200 as permitted sender)
Received: from [209.85.222.200] (HELO mail-pz0-f200.google.com) (209.85.222.200)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 23 Aug 2009 15:54:09 +0000
Received: by pzk38 with SMTP id 38so569468pzk.5
        for <general@hadoop.apache.org>; Sun, 23 Aug 2009 08:53:48 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=IxH0LgPj6LG6aGgQbqfcIOcTF7rJAMQ6daRIV2RBXJE=;
        b=p8NxWY+cywd17MBnExnmsVW0EdaOfQ+L+U8V+i04SHuDfiOQcOidlC5TrG2CYvS+Rp
         fPmJlkU4VkBXVZadqGxeuzI9azTu6udnenpn2vMr6UNtisK38kPrLqXFFSsmSvuezQE7
         ixYItIt4vqQWohjLO5sVxBEhBUuGsh0J7Rklg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=h1YdHrhS0qG+Wc6CPM2sDSAo9xv3kFflBINUwaIJbV10wNldISzRw949pb/du5qxd4
         kfwJFNZD1dEZ88nlu853XVIfWo5QPdxh+clS6lRyDxCpoao+BpqPJIZTmFc/cq0Rk30O
         OzAEG7lHZE/Xa+/HtpDaBLBlEUJggPEX/MYTU=
MIME-Version: 1.0
Received: by 10.142.9.12 with SMTP id 12mr316608wfi.100.1251042828727; Sun, 23 
	Aug 2009 08:53:48 -0700 (PDT)
In-Reply-To: <2B8EEA26-46D4-4D77-997A-A3394790613F@apache.org>
References: <2B8EEA26-46D4-4D77-997A-A3394790613F@apache.org>
Date: Sun, 23 Aug 2009 23:53:48 +0800
Message-ID: <64e78c10908230853y79e81b56j18f9bd4ecbfbad55@mail.gmail.com>
Subject: Re: [VOTE] Create hbase-issues email list
From: Duling Lu <luduling@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502aff467b6ab0471d11bde
X-Virus-Checked: Checked by ClamAV on apache.org

--00504502aff467b6ab0471d11bde
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

+1

2009/8/22 Owen O'Malley <omalley@apache.org>

> I think we should create an hbase-issues lists to track that detailed jira
> traffic and reduce the traffic on hbase-dev. It will replicate the structure
> that we have in common, hdfs, and mapreduce. Clearly, I'm +1.
>
> -- Owen
>

--00504502aff467b6ab0471d11bde--

From general-return-449-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 23 16:41:20 2009
Return-Path: <general-return-449-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 18928 invoked from network); 23 Aug 2009 16:41:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Aug 2009 16:41:20 -0000
Received: (qmail 366 invoked by uid 500); 23 Aug 2009 16:41:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 286 invoked by uid 500); 23 Aug 2009 16:41:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 276 invoked by uid 99); 23 Aug 2009 16:41:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 23 Aug 2009 16:41:40 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 23 Aug 2009 16:41:28 +0000
Received: from [192.168.1.64] (snvvpn2-10-73-152-c159.hq.corp.yahoo.com [10.73.152.159])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n7NGdV5s083852
	for <general@hadoop.apache.org>; Sun, 23 Aug 2009 09:39:32 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=UP7sVVlhdXPh0mxmi4QKv7XleWCc4FkyAMhyaZjamrvQMWWlbnlxiSjpGxay7K9Y
Message-Id: <00D49934-7287-473F-AEF5-86F2BC883383@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <bf8befe30908222318y12193b09h7b8e5dfeb22f877d@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Need Guidance
Date: Sun, 23 Aug 2009 09:39:31 -0700
References: <bf8befe30908222318y12193b09h7b8e5dfeb22f877d@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

Digvijay/Pushpak,

  Welcome to Hadoop!

  We maintain a list of issues we need help with over here: http://wiki.apache.org/hadoop/ProjectSuggestions

  In particular, we could use a *lot* of help with our tests and their  
automation etc. - good luck!

Arun

On Aug 22, 2009, at 11:18 PM, digvijay singh shaktawat wrote:

> Hi,
>     We are two undergraduate student and have are determined to do a
> project in cloud computing as are final Btech project.  We are  
> absolutely
> new to hadoop and have arround six months to work on it. We wanted
> suggestions regarding some interesting work possible with hadoop.
>
> -Digvijay Singh
> -Pushpak Aggarwal


From general-return-450-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 25 10:40:35 2009
Return-Path: <general-return-450-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53543 invoked from network); 25 Aug 2009 10:40:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Aug 2009 10:40:35 -0000
Received: (qmail 66037 invoked by uid 500); 25 Aug 2009 10:41:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65966 invoked by uid 500); 25 Aug 2009 10:40:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65956 invoked by uid 99); 25 Aug 2009 10:40:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 10:40:59 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 10:40:48 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 62562B7CCE
	for <general@hadoop.apache.org>; Tue, 25 Aug 2009 11:40:27 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id Ki58OCwSa1Lo for <general@hadoop.apache.org>;
	Tue, 25 Aug 2009 11:40:20 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id B4581B7D06
	for <general@hadoop.apache.org>; Tue, 25 Aug 2009 11:40:20 +0100 (BST)
MailScanner-NULL-Check: 1251801608.44532@GNNFO3GJC3icHRLrU7v7zw
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id n7PAe7XF007435
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Tue, 25 Aug 2009 11:40:07 +0100 (BST)
Message-ID: <4A93BF87.3040100@apache.org>
Date: Tue, 25 Aug 2009 11:40:07 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090812)
MIME-Version: 1.0
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Hadoop security list?
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: n7PAe7XF007435
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org


Assuming that eventually Hadoop will add security, are there plans for a 
hadoop security list?

In the mean time, if anyone were to have concerns about a bit of the 
system, should they just go as JIRA issues on the relevant project?

From general-return-451-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 25 12:24:29 2009
Return-Path: <general-return-451-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 86787 invoked from network); 25 Aug 2009 12:24:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Aug 2009 12:24:29 -0000
Received: (qmail 69292 invoked by uid 500); 25 Aug 2009 12:24:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69234 invoked by uid 500); 25 Aug 2009 12:24:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69224 invoked by uid 99); 25 Aug 2009 12:24:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 12:24:54 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of nicc777@gmail.com designates 209.85.218.214 as permitted sender)
Received: from [209.85.218.214] (HELO mail-bw0-f214.google.com) (209.85.218.214)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 12:24:44 +0000
Received: by bwz10 with SMTP id 10so2090530bwz.29
        for <general@hadoop.apache.org>; Tue, 25 Aug 2009 05:24:24 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=lgRIYiAgyYO8/oaZGEp0T2u/sJdIAOg0udRDMV2sOBA=;
        b=xdXNx+IwhYeU1IwOynjHow1nF7IXXmOe8JaVBLIEj5AYQ/clA63m53kKGIgBl+7w1z
         6Uo4ZryT2s5U1oVuT0M51LJQaVr3F8LMv1iSZdtBoknqP+dEQO3ABTanxScqfqAzJ1ee
         v+XvprfrcwnQthPXL8VB4IqvVXq4zMTUEQKzQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Xxe2SXZxJevqwydyWWxOmI+8VeayPUVYyC/drW2lhgVBqBUsQqVGoZbBIpNySfDs0V
         URr19dEIQQ2XfQ3NQmTd+fvGZxZWLrjOsIfJz/w1PkxuPZu2mavLeP1Q2tKix7dkoYCU
         W+mrMjczTMbch+PLL+jEIURACsedulX0Xb2jY=
MIME-Version: 1.0
Received: by 10.204.8.138 with SMTP id h10mr2403915bkh.187.1251203064347; Tue, 
	25 Aug 2009 05:24:24 -0700 (PDT)
In-Reply-To: <4A93BF87.3040100@apache.org>
References: <4A93BF87.3040100@apache.org>
Date: Tue, 25 Aug 2009 14:24:24 +0200
Message-ID: <e21a770d0908250524v6aee68b6u92c00e952023d44c@mail.gmail.com>
Subject: Re: Hadoop security list?
From: Nico Coetzee <nicc777@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00151758852e313d3a0471f66a1e
X-Virus-Checked: Checked by ClamAV on apache.org

--00151758852e313d3a0471f66a1e
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Perhaps a stupid question, but what do you mean by "security"? If you run
Hadoop on Linux there is a lot of things you can to "secure" your cluster.


On Tue, Aug 25, 2009 at 12:40 PM, Steve Loughran <stevel@apache.org> wrote:

>
> Assuming that eventually Hadoop will add security, are there plans for a
> hadoop security list?
>
> In the mean time, if anyone were to have concerns about a bit of the
> system, should they just go as JIRA issues on the relevant project?
>

--00151758852e313d3a0471f66a1e--

From general-return-452-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 25 12:58:49 2009
Return-Path: <general-return-452-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1228 invoked from network); 25 Aug 2009 12:58:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Aug 2009 12:58:49 -0000
Received: (qmail 20941 invoked by uid 500); 25 Aug 2009 12:59:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20816 invoked by uid 500); 25 Aug 2009 12:59:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 20793 invoked by uid 99); 25 Aug 2009 12:59:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 12:59:11 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 12:58:59 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 051861BA3D6
	for <general@hadoop.apache.org>; Tue, 25 Aug 2009 13:58:39 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id nr3BlAZNbKUj for <general@hadoop.apache.org>;
	Tue, 25 Aug 2009 13:58:38 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by colossus.hpl.hp.com (Postfix) with ESMTPS id 84BC71BA357
	for <general@hadoop.apache.org>; Tue, 25 Aug 2009 13:58:38 +0100 (BST)
MailScanner-NULL-Check: 1251809906.2768@T+BTsEsa8oi8TKPSH2HSaw
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id n7PCwP9M015461
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Tue, 25 Aug 2009 13:58:25 +0100 (BST)
Message-ID: <4A93DFF1.4030703@apache.org>
Date: Tue, 25 Aug 2009 13:58:25 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Hadoop security list?
References: <4A93BF87.3040100@apache.org> <e21a770d0908250524v6aee68b6u92c00e952023d44c@mail.gmail.com>
In-Reply-To: <e21a770d0908250524v6aee68b6u92c00e952023d44c@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: n7PCwP9M015461
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

Nico Coetzee wrote:
> Perhaps a stupid question, but what do you mean by "security"? If you run
> Hadoop on Linux there is a lot of things you can to "secure" your cluster.
> 

where to report security issues with Hadoop?

Some asf projects have them - httpd, tomcat, ws, others dont (ant, 
log4j). I'm wondering where hadoop fits in on this scale of things

From general-return-453-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 25 13:16:48 2009
Return-Path: <general-return-453-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 7660 invoked from network); 25 Aug 2009 13:16:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Aug 2009 13:16:48 -0000
Received: (qmail 45886 invoked by uid 500); 25 Aug 2009 13:17:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45824 invoked by uid 500); 25 Aug 2009 13:17:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45814 invoked by uid 99); 25 Aug 2009 13:17:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 13:17:12 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [66.51.199.98] (HELO mail10.dslextreme.com) (66.51.199.98)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 25 Aug 2009 13:17:03 +0000
Received: (qmail 24706 invoked from network); 25 Aug 2009 13:16:40 -0000
Received: from unknown (HELO [192.168.0.100]) (208.127.242.153)
	by mail10.dslextreme.com with (DHE-RSA-AES256-SHA encrypted) SMTP; Tue, 25 Aug 2009 06:16:40 -0700
Message-ID: <4A93E43B.7050500@cloudera.com>
Date: Tue, 25 Aug 2009 06:16:43 -0700
From: Amr Awadallah <aaa@cloudera.com>
User-Agent: Thunderbird 2.0.0.22 (Windows/20090605)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Hadoop security list?
References: <4A93BF87.3040100@apache.org> <e21a770d0908250524v6aee68b6u92c00e952023d44c@mail.gmail.com> <4A93DFF1.4030703@apache.org>
In-Reply-To: <4A93DFF1.4030703@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-MagicMail-UUID: 86fc4bf0-9179-11de-bcce-00188bf6df8c
X-Virus-Checked: Checked by ClamAV on apache.org

I propose we just use general@ for now for security announcements until 
it deserves getting its own list.

-- amr
> Nico Coetzee wrote:
>> Perhaps a stupid question, but what do you mean by "security"? If you 
>> run
>> Hadoop on Linux there is a lot of things you can to "secure" your 
>> cluster.
>>
>
> where to report security issues with Hadoop?
>
> Some asf projects have them - httpd, tomcat, ws, others dont (ant, 
> log4j). I'm wondering where hadoop fits in on this scale of things


From general-return-454-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 25 15:42:27 2009
Return-Path: <general-return-454-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 36996 invoked from network); 25 Aug 2009 15:42:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Aug 2009 15:42:27 -0000
Received: (qmail 14913 invoked by uid 500); 25 Aug 2009 15:42:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14819 invoked by uid 500); 25 Aug 2009 15:42:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14809 invoked by uid 99); 25 Aug 2009 15:42:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 15:42:51 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.27.228] (HELO QMTA15.emeryville.ca.mail.comcast.net) (76.96.27.228)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 15:42:41 +0000
Received: from OMTA09.emeryville.ca.mail.comcast.net ([76.96.30.20])
	by QMTA15.emeryville.ca.mail.comcast.net with comcast
	id YeMg1c0080S2fkCAFfiNuj; Tue, 25 Aug 2009 15:42:22 +0000
Received: from [10.0.0.59] ([209.131.62.115])
	by OMTA09.emeryville.ca.mail.comcast.net with comcast
	id Yfi91c0082VBGtd8VfiESt; Tue, 25 Aug 2009 15:42:20 +0000
Message-Id: <CC7CE677-2EEE-47FC-A3B4-859CE2788563@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: [VOTE] create security@hadoop.apache.org
Date: Tue, 25 Aug 2009 08:42:08 -0700
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

With the upcoming freeze of Map/Reduce 0.21, for the first time, we  
have code that is critical from a system security perspective. As  
outlined on http://www.apache.org/security/committers.html this will  
be a private list with only committers that are working on the  
security aspects of Hadoop. The use of the list will be strictly  
limited to collecting security issues from users so that they can  
fixed and released before they are announced on public forums, such as  
jira. Clearly, I'm +1.

On a related note, my plan is to work on getting full Kerberos  
integration for 0.22. We've come a long way without any authentication  
in Hadoop, but it is getting important to start closing the holes.

-- Owen

From general-return-455-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 25 16:53:52 2009
Return-Path: <general-return-455-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94151 invoked from network); 25 Aug 2009 16:53:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Aug 2009 16:53:52 -0000
Received: (qmail 34808 invoked by uid 500); 25 Aug 2009 16:54:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34722 invoked by uid 500); 25 Aug 2009 16:54:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34702 invoked by uid 99); 25 Aug 2009 16:54:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 16:54:15 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 16:54:04 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 68290B7D52
	for <general@hadoop.apache.org>; Tue, 25 Aug 2009 17:53:43 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id h18SCuu4UJZ9 for <general@hadoop.apache.org>;
	Tue, 25 Aug 2009 17:53:36 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id A4E02B7D51
	for <general@hadoop.apache.org>; Tue, 25 Aug 2009 17:53:36 +0100 (BST)
MailScanner-NULL-Check: 1251824004.75459@dzjmPc/aBqeGxwDv5kWO7w
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id n7PGrOcc029598
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Tue, 25 Aug 2009 17:53:24 +0100 (BST)
Message-ID: <4A941704.8010300@apache.org>
Date: Tue, 25 Aug 2009 17:53:24 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] create security@hadoop.apache.org
References: <CC7CE677-2EEE-47FC-A3B4-859CE2788563@apache.org>
In-Reply-To: <CC7CE677-2EEE-47FC-A3B4-859CE2788563@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: n7PGrOcc029598
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

Owen O'Malley wrote:
> With the upcoming freeze of Map/Reduce 0.21, for the first time, we have 
> code that is critical from a system security perspective. As outlined on 
> http://www.apache.org/security/committers.html this will be a private 
> list with only committers that are working on the security aspects of 
> Hadoop. The use of the list will be strictly limited to collecting 
> security issues from users so that they can fixed and released before 
> they are announced on public forums, such as jira. Clearly, I'm +1.
> 
> On a related note, my plan is to work on getting full Kerberos 
> integration for 0.22. We've come a long way without any authentication 
> in Hadoop, but it is getting important to start closing the holes.
> 
> -- Owen

+1

From general-return-456-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 25 16:57:45 2009
Return-Path: <general-return-456-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95660 invoked from network); 25 Aug 2009 16:57:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Aug 2009 16:57:45 -0000
Received: (qmail 39590 invoked by uid 500); 25 Aug 2009 16:58:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39518 invoked by uid 500); 25 Aug 2009 16:58:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39508 invoked by uid 99); 25 Aug 2009 16:58:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 16:58:10 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.219.209] (HELO mail-ew0-f209.google.com) (209.85.219.209)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 16:58:02 +0000
Received: by ewy5 with SMTP id 5so2001141ewy.36
        for <general@hadoop.apache.org>; Tue, 25 Aug 2009 09:57:40 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.72.85 with SMTP id s63mr1384107wed.0.1251219458194; Tue, 
	25 Aug 2009 09:57:38 -0700 (PDT)
In-Reply-To: <CC7CE677-2EEE-47FC-A3B4-859CE2788563@apache.org>
References: <CC7CE677-2EEE-47FC-A3B4-859CE2788563@apache.org>
Date: Tue, 25 Aug 2009 17:57:38 +0100
Message-ID: <ac79ea400908250957w4726819bvf960e46290b9d3e0@mail.gmail.com>
Subject: Re: [VOTE] create security@hadoop.apache.org
From: Tom White <tom@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Tom

On Tue, Aug 25, 2009 at 4:42 PM, Owen O'Malley<omalley@apache.org> wrote:
> With the upcoming freeze of Map/Reduce 0.21, for the first time, we have
> code that is critical from a system security perspective. As outlined on
> http://www.apache.org/security/committers.html this will be a private list
> with only committers that are working on the security aspects of Hadoop. The
> use of the list will be strictly limited to collecting security issues from
> users so that they can fixed and released before they are announced on
> public forums, such as jira. Clearly, I'm +1.
>
> On a related note, my plan is to work on getting full Kerberos integration
> for 0.22. We've come a long way without any authentication in Hadoop, but it
> is getting important to start closing the holes.
>
> -- Owen
>

From general-return-457-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 25 17:00:01 2009
Return-Path: <general-return-457-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97012 invoked from network); 25 Aug 2009 17:00:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Aug 2009 17:00:00 -0000
Received: (qmail 42178 invoked by uid 500); 25 Aug 2009 17:00:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42096 invoked by uid 500); 25 Aug 2009 17:00:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 41935 invoked by uid 99); 25 Aug 2009 17:00:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 17:00:24 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 17:00:14 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n7PGxmRY041829
	for <general@hadoop.apache.org>; Tue, 25 Aug 2009 09:59:48 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=jtvy7kwRSPbdVcYLLeVkugm3frzL2IAKwhm/sE26BCBQ4Dr2xfsWADkZn6rMlZVG
Message-Id: <1ABE40C8-A86D-4940-B91B-05896805A182@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <CC7CE677-2EEE-47FC-A3B4-859CE2788563@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] create security@hadoop.apache.org
Date: Tue, 25 Aug 2009 09:59:48 -0700
References: <CC7CE677-2EEE-47FC-A3B4-859CE2788563@apache.org>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Arun

On Aug 25, 2009, at 8:42 AM, Owen O'Malley wrote:

> With the upcoming freeze of Map/Reduce 0.21, for the first time, we  
> have code that is critical from a system security perspective. As  
> outlined on http://www.apache.org/security/committers.html this will  
> be a private list with only committers that are working on the  
> security aspects of Hadoop. The use of the list will be strictly  
> limited to collecting security issues from users so that they can  
> fixed and released before they are announced on public forums, such  
> as jira. Clearly, I'm +1.
>
> On a related note, my plan is to work on getting full Kerberos  
> integration for 0.22. We've come a long way without any  
> authentication in Hadoop, but it is getting important to start  
> closing the holes.
>
> -- Owen


From general-return-458-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 25 17:15:48 2009
Return-Path: <general-return-458-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 3187 invoked from network); 25 Aug 2009 17:15:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Aug 2009 17:15:48 -0000
Received: (qmail 65746 invoked by uid 500); 25 Aug 2009 17:16:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65685 invoked by uid 500); 25 Aug 2009 17:16:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65675 invoked by uid 99); 25 Aug 2009 17:16:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 17:16:12 +0000
X-ASF-Spam-Status: No, hits=4.2 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 17:16:00 +0000
Received: from sp1-ex07cas01.ds.corp.yahoo.com (sp1-ex07cas01.ds.corp.yahoo.com [216.252.116.137])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n7PHCmVx093165
	for <general@hadoop.apache.org>; Tue, 25 Aug 2009 10:12:48 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:in-reply-to:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=VPqMUgfV/F83eEJJkYE58V6Ufk6oGEcxBDXbARNuRqhzT4C52zTFSN+qNzB4wkyc
Received: from SP1-EX07VS02.ds.corp.yahoo.com ([216.252.116.135]) by
 sp1-ex07cas01.ds.corp.yahoo.com ([216.252.116.137]) with mapi; Tue, 25 Aug
 2009 10:12:48 -0700
From: Devaraj Das <ddas@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 25 Aug 2009 10:12:45 -0700
Subject: Re: [VOTE] create security@hadoop.apache.org
Thread-Topic: [VOTE] create security@hadoop.apache.org
Thread-Index: AcolmsgALArR0jXFSgaPOGUpEXaDFQADHsoN
Message-ID: <C6BA1965.63676%ddas@yahoo-inc.com>
In-Reply-To: <CC7CE677-2EEE-47FC-A3B4-859CE2788563@apache.org>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C6BA196563676ddasyahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C6BA196563676ddasyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

+1


On 8/25/09 9:12 PM, "Owen O'Malley" <omalley@apache.org> wrote:

With the upcoming freeze of Map/Reduce 0.21, for the first time, we
have code that is critical from a system security perspective. As
outlined on http://www.apache.org/security/committers.html this will
be a private list with only committers that are working on the
security aspects of Hadoop. The use of the list will be strictly
limited to collecting security issues from users so that they can
fixed and released before they are announced on public forums, such as
jira. Clearly, I'm +1.

On a related note, my plan is to work on getting full Kerberos
integration for 0.22. We've come a long way without any authentication
in Hadoop, but it is getting important to start closing the holes.

-- Owen


--_000_C6BA196563676ddasyahooinccom_--

From general-return-459-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 25 17:23:41 2009
Return-Path: <general-return-459-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 7686 invoked from network); 25 Aug 2009 17:23:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Aug 2009 17:23:40 -0000
Received: (qmail 81022 invoked by uid 500); 25 Aug 2009 17:24:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80976 invoked by uid 500); 25 Aug 2009 17:24:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80878 invoked by uid 99); 25 Aug 2009 17:24:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 17:24:04 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 17:23:53 +0000
Received: from SNV-EXBH01.ds.corp.yahoo.com (snv-exbh01.ds.corp.yahoo.com [207.126.227.249])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n7PHNBxD027253
	for <general@hadoop.apache.org>; Tue, 25 Aug 2009 10:23:16 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:return-path:x-originalarrivaltime;
	b=b2ALZR9YeCQK4mwRKbHMgarSMMmrydt0VIQr+onQOsEKQ1g+rWFM+4cWPfOuUy4E
Received: from SNV-EXVS09.ds.corp.yahoo.com ([207.126.227.87]) by SNV-EXBH01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Tue, 25 Aug 2009 10:23:11 -0700
Received: from 10.73.146.106 ([10.73.146.106]) by SNV-EXVS09.ds.corp.yahoo.com ([207.126.227.84]) via Exchange Front-End Server snv-webmail.corp.yahoo.com ([207.126.227.60]) with Microsoft Exchange Server HTTP-DAV ;
 Tue, 25 Aug 2009 17:22:27 +0000
User-Agent: Microsoft-Entourage/12.20.0.090605
Date: Tue, 25 Aug 2009 10:22:27 -0700
Subject: Re: [VOTE] create security@hadoop.apache.org
From: Mahadev Konar <mahadev@yahoo-inc.com>
To: <general@hadoop.apache.org>
Message-ID: <C6B96BE3.22EB0%mahadev@yahoo-inc.com>
Thread-Topic: [VOTE] create security@hadoop.apache.org
Thread-Index: AcolmsgALArR0jXFSgaPOGUpEXaDFQADHsoNAABWuZo=
In-Reply-To: <C6BA1965.63676%ddas@yahoo-inc.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-OriginalArrivalTime: 25 Aug 2009 17:23:11.0287 (UTC) FILETIME=[B874B470:01CA25A8]
X-Virus-Checked: Checked by ClamAV on apache.org

+1

mahadev


On 8/25/09 10:12 AM, "Devaraj Das" <ddas@yahoo-inc.com> wrote:

> +1
> 
> 
> On 8/25/09 9:12 PM, "Owen O'Malley" <omalley@apache.org> wrote:
> 
> With the upcoming freeze of Map/Reduce 0.21, for the first time, we
> have code that is critical from a system security perspective. As
> outlined on http://www.apache.org/security/committers.html this will
> be a private list with only committers that are working on the
> security aspects of Hadoop. The use of the list will be strictly
> limited to collecting security issues from users so that they can
> fixed and released before they are announced on public forums, such as
> jira. Clearly, I'm +1.
> 
> On a related note, my plan is to work on getting full Kerberos
> integration for 0.22. We've come a long way without any authentication
> in Hadoop, but it is getting important to start closing the holes.
> 
> -- Owen
> 


From general-return-460-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 25 17:26:12 2009
Return-Path: <general-return-460-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9047 invoked from network); 25 Aug 2009 17:26:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Aug 2009 17:26:12 -0000
Received: (qmail 83249 invoked by uid 500); 25 Aug 2009 17:26:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83185 invoked by uid 500); 25 Aug 2009 17:26:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83175 invoked by uid 99); 25 Aug 2009 17:26:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 17:26:36 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 25 Aug 2009 17:26:35 +0000
Received: (qmail 8798 invoked by uid 99); 25 Aug 2009 17:25:49 -0000
Received: from localhost.apache.org (HELO [192.168.168.101]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 17:25:49 +0000
Message-ID: <4A941EB5.4000303@apache.org>
Date: Tue, 25 Aug 2009 10:26:13 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] create security@hadoop.apache.org
References: <CC7CE677-2EEE-47FC-A3B4-859CE2788563@apache.org>
In-Reply-To: <CC7CE677-2EEE-47FC-A3B4-859CE2788563@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

+1 Like other private ASF lists, we must remember to use this list only 
when privacy is required and not for general discussion of security.

Doug

Owen O'Malley wrote:
> With the upcoming freeze of Map/Reduce 0.21, for the first time, we have 
> code that is critical from a system security perspective. As outlined on 
> http://www.apache.org/security/committers.html this will be a private 
> list with only committers that are working on the security aspects of 
> Hadoop. The use of the list will be strictly limited to collecting 
> security issues from users so that they can fixed and released before 
> they are announced on public forums, such as jira. Clearly, I'm +1.
> 
> On a related note, my plan is to work on getting full Kerberos 
> integration for 0.22. We've come a long way without any authentication 
> in Hadoop, but it is getting important to start closing the holes.
> 
> -- Owen

From general-return-461-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 25 17:44:00 2009
Return-Path: <general-return-461-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19914 invoked from network); 25 Aug 2009 17:44:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Aug 2009 17:44:00 -0000
Received: (qmail 7251 invoked by uid 500); 25 Aug 2009 17:44:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7202 invoked by uid 500); 25 Aug 2009 17:44:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7192 invoked by uid 99); 25 Aug 2009 17:44:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 17:44:24 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [216.252.110.212] (HELO web56203.mail.re3.yahoo.com) (216.252.110.212)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 25 Aug 2009 17:44:13 +0000
Received: (qmail 78242 invoked by uid 60001); 25 Aug 2009 17:43:52 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1251222232; bh=5TgXzL3Hxm906sHZTXiTgElPV1CTPf+3nGoumYHW+w8=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=emcK9baM3B9PaG5dXgGwcXVyKDwmeltcyaNz1KTaI31Do+6KVtMmoR5RTOqDgU7jhF63Df6JpTNQoQ2WFNuTxvProQ7m3HXi/CZEgAnO/3GxwcKoqag2xotSXChFLebNJP/9l03hymh9I/mNiU9ZeA82/m+oAScXeOyiXQohVLg=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=Ytgfw4zkqzIjBlUmj5gkEWkSnKhYjZj7WmBvH5xkULfQynEA61P3KQj4jzXL+TpbVWsXrsLmqJ7VyGkEJfRMmv9ZMErLuFSV+e+7LmXRU5SjJ+CbcStsvmue0+vvdXulv+dilbCb81Q2MnFgnItbiZm5+4KY0GaQ5xBxOAEFkr8=;
Message-ID: <328288.78170.qm@web56203.mail.re3.yahoo.com>
X-YMail-OSG: wdnWZLIVM1nmH3y8.6G8ImcleC_wKJt50b5YjMg0CrOkYDst16wurkWkwFIJaLOtpMXj_WvFwY0XkamLcnBUR8v3xg2Lo5bHGasQG7sgdB_RgawVNUGLEIDzMT6K6FRSkqSpXxFXUR20EQKJPau5whfGFKcxqFhJAduSyJ81VAAwucPVoDGGoZwqsHmXh9xt0b9FUPRdgp3nFbk7nSvB0pbU3Anz9uO0PszKtgeyCNc.o0Zgfkn8aDa7_d1atVjGDcRUe5UbHLfVff9LyMD7kj9T1yKSKccAf0a0mp5V3JoiNOS8gWXBqpvXKyf9NejXHs_uwa_vmis-
Received: from [216.145.54.7] by web56203.mail.re3.yahoo.com via HTTP; Tue, 25 Aug 2009 10:43:52 PDT
X-Mailer: YahooMailRC/1358.27 YahooMailWebService/0.7.338.2
References: <CC7CE677-2EEE-47FC-A3B4-859CE2788563@apache.org>
Date: Tue, 25 Aug 2009 10:43:52 -0700 (PDT)
From: "Tsz Wo \(Nicholas\), Sze" <s29752-hadoopgeneral@yahoo.com>
Subject: Re: [VOTE] create security@hadoop.apache.org
To: general@hadoop.apache.org
In-Reply-To: <CC7CE677-2EEE-47FC-A3B4-859CE2788563@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org

+1



----- Original Message ----
> From: Owen O'Malley <omalley@apache.org>
> To: general@hadoop.apache.org
> Sent: Tuesday, August 25, 2009 8:42:08 AM
> Subject: [VOTE] create security@hadoop.apache.org
> 
> With the upcoming freeze of Map/Reduce 0.21, for the first time, we have code 
> that is critical from a system security perspective. As outlined on 
> http://www.apache.org/security/committers.html this will be a private list with 
> only committers that are working on the security aspects of Hadoop. The use of 
> the list will be strictly limited to collecting security issues from users so 
> that they can fixed and released before they are announced on public forums, 
> such as jira. Clearly, I'm +1.
> 
> On a related note, my plan is to work on getting full Kerberos integration for 
> 0.22. We've come a long way without any authentication in Hadoop, but it is 
> getting important to start closing the holes.
> 
> -- Owen


From general-return-462-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 01:10:49 2009
Return-Path: <general-return-462-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24642 invoked from network); 26 Aug 2009 01:10:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 01:10:49 -0000
Received: (qmail 73055 invoked by uid 500); 26 Aug 2009 01:11:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72978 invoked by uid 500); 26 Aug 2009 01:11:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72967 invoked by uid 99); 26 Aug 2009 01:11:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 01:11:13 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of tzehon.tan@gmail.com designates 209.85.222.200 as permitted sender)
Received: from [209.85.222.200] (HELO mail-pz0-f200.google.com) (209.85.222.200)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 01:11:04 +0000
Received: by pzk38 with SMTP id 38so1889393pzk.5
        for <general@hadoop.apache.org>; Tue, 25 Aug 2009 18:10:43 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=pABhwp619Gd4EuXIrc9x4feg5MVp8szvzAluMoCvbOU=;
        b=A3g8ptsWSPCtaY+ThR/p1kGUtorYW+JdUcPbAK63sHZQquOLQmSmbCkyDX7YfU9hwF
         Ub00sF1A2nWf++nyrDch2TzGmErykJ6lLZ+e4n5guiNJV65Cs+sqe4WafeVoacSGSCNY
         kNiJb6/8EhDEVAhmfmsXbm+gZctlHCtPcaFfA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=W/bUWQ2t7BboawSZYLhaq4uQbGNgjo3qhA7eweRlFPiuWtt5H9J5xSI6+IU4BPfLjz
         YRgt0K6KP7nexOolSj469Csdyu9g25GPvdyyMUCoiNc9QoOboo+9J+zPnPqpWeuV4wIO
         uY+Rn5ftiJa+ug0p28NzeN94Fw7F7k4/dASjw=
MIME-Version: 1.0
Received: by 10.142.201.18 with SMTP id y18mr457614wff.257.1251249042944; Tue, 
	25 Aug 2009 18:10:42 -0700 (PDT)
In-Reply-To: <CC7CE677-2EEE-47FC-A3B4-859CE2788563@apache.org>
References: <CC7CE677-2EEE-47FC-A3B4-859CE2788563@apache.org>
Date: Wed, 26 Aug 2009 09:10:42 +0800
Message-ID: <fca34e2c0908251810t3b8b14f9wccce59cdad046bec@mail.gmail.com>
Subject: Re: [VOTE] create security@hadoop.apache.org
From: Tze Hon Tan <tzehon.tan@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd22a0cbaf4030472011ebc
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd22a0cbaf4030472011ebc
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

+1

On Tue, Aug 25, 2009 at 11:42 PM, Owen O'Malley <omalley@apache.org> wrote:

> With the upcoming freeze of Map/Reduce 0.21, for the first time, we have
> code that is critical from a system security perspective. As outlined on
> http://www.apache.org/security/committers.html this will be a private list
> with only committers that are working on the security aspects of Hadoop. The
> use of the list will be strictly limited to collecting security issues from
> users so that they can fixed and released before they are announced on
> public forums, such as jira. Clearly, I'm +1.
>
> On a related note, my plan is to work on getting full Kerberos integration
> for 0.22. We've come a long way without any authentication in Hadoop, but it
> is getting important to start closing the holes.
>
> -- Owen
>

--000e0cd22a0cbaf4030472011ebc--

From general-return-463-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 01:16:11 2009
Return-Path: <general-return-463-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25484 invoked from network); 26 Aug 2009 01:16:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 01:16:11 -0000
Received: (qmail 79245 invoked by uid 500); 26 Aug 2009 01:16:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79155 invoked by uid 500); 26 Aug 2009 01:16:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 79145 invoked by uid 99); 26 Aug 2009 01:16:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 01:16:35 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of fzappert@gmail.com designates 74.125.92.26 as permitted sender)
Received: from [74.125.92.26] (HELO qw-out-2122.google.com) (74.125.92.26)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 01:16:27 +0000
Received: by qw-out-2122.google.com with SMTP id 8so1783025qwh.35
        for <general@hadoop.apache.org>; Tue, 25 Aug 2009 18:16:06 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=i4pJQ+sNtU+vjdZb7p0DVm5jRjyTq7m1ZikCNMgs1Cc=;
        b=TLzStNdy7AefGMidlS5ffbQquVtpJJsB2nOFpGHUdufsXUASRlFinUSpSWQDenOXOR
         9hJeAWLt2Hje3DuDG0NjosEie25CDVhvB3opB4yg9tzr4Dx6n1sw4nT1CwMlEjZCq4rZ
         Tl7QjwQ7Odii+8bA7yJJ0yjhZ3gdMIfjCXFbE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=XAV3za119pdnuvIrKbOCtK3wpRGH0qsEFC6rGzb0YymIkNhdzrqSCSTCuUzekzav//
         TM4mplx+A1+8vticeALhql2sJ5svSlA81yYkdazQ2QnlTJHaO4nlif51HUpDKFMnb3Mk
         7o95B3yGUWoI69Oo2izmx0wC7xinb/wBHSUTM=
MIME-Version: 1.0
Received: by 10.229.16.73 with SMTP id n9mr1861517qca.70.1251249366537; Tue, 
	25 Aug 2009 18:16:06 -0700 (PDT)
In-Reply-To: <fca34e2c0908251810t3b8b14f9wccce59cdad046bec@mail.gmail.com>
References: <CC7CE677-2EEE-47FC-A3B4-859CE2788563@apache.org>
	 <fca34e2c0908251810t3b8b14f9wccce59cdad046bec@mail.gmail.com>
Date: Tue, 25 Aug 2009 18:16:06 -0700
Message-ID: <14048aeb0908251816h8d7c55o27632ec2b18ebcc6@mail.gmail.com>
Subject: Re: [VOTE] create security@hadoop.apache.org
From: Fred Zappert <fzappert@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175cb4ca04954604720132e9
X-Virus-Checked: Checked by ClamAV on apache.org

--0015175cb4ca04954604720132e9
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

+1  in general.
+1 for Kerberos to handle authentication and authorization

On Tue, Aug 25, 2009 at 6:10 PM, Tze Hon Tan <tzehon.tan@gmail.com> wrote:

> +1
>
> On Tue, Aug 25, 2009 at 11:42 PM, Owen O'Malley <omalley@apache.org>
> wrote:
>
> > With the upcoming freeze of Map/Reduce 0.21, for the first time, we have
> > code that is critical from a system security perspective. As outlined on
> > http://www.apache.org/security/committers.html this will be a private
> list
> > with only committers that are working on the security aspects of Hadoop.
> The
> > use of the list will be strictly limited to collecting security issues
> from
> > users so that they can fixed and released before they are announced on
> > public forums, such as jira. Clearly, I'm +1.
> >
> > On a related note, my plan is to work on getting full Kerberos
> integration
> > for 0.22. We've come a long way without any authentication in Hadoop, but
> it
> > is getting important to start closing the holes.
> >
> > -- Owen
> >
>

--0015175cb4ca04954604720132e9--

From general-return-464-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 04:50:37 2009
Return-Path: <general-return-464-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97736 invoked from network); 26 Aug 2009 04:50:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 04:50:37 -0000
Received: (qmail 65984 invoked by uid 500); 26 Aug 2009 04:50:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65666 invoked by uid 500); 26 Aug 2009 04:50:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65640 invoked by uid 99); 26 Aug 2009 04:50:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 04:50:57 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 04:50:45 +0000
Received: from traveling-laptop-66-251.eglbp.corp.yahoo.com (socks3.corp.yahoo.com [216.145.54.15])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n7Q4o333013692
	for <general@hadoop.apache.org>; Tue, 25 Aug 2009 21:50:04 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=XEALn49AuWrjTFjHEPMX0TFQHsYb/9OKbhgMIoYIK4kw5aAPPd+NR794XAA4S+Ry
Message-ID: <4A94BEFA.6090303@yahoo-inc.com>
Date: Wed, 26 Aug 2009 10:20:02 +0530
From: Hemanth Yamijala <yhemanth@yahoo-inc.com>
User-Agent: Thunderbird 2.0.0.23 (X11/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] create security@hadoop.apache.org
References: <CC7CE677-2EEE-47FC-A3B4-859CE2788563@apache.org> <fca34e2c0908251810t3b8b14f9wccce59cdad046bec@mail.gmail.com>
In-Reply-To: <fca34e2c0908251810t3b8b14f9wccce59cdad046bec@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Tze Hon Tan wrote:
> +1
>
> On Tue, Aug 25, 2009 at 11:42 PM, Owen O'Malley <omalley@apache.org> wrote:
>
>   
>> With the upcoming freeze of Map/Reduce 0.21, for the first time, we have
>> code that is critical from a system security perspective. As outlined on
>> http://www.apache.org/security/committers.html this will be a private list
>> with only committers that are working on the security aspects of Hadoop. The
>> use of the list will be strictly limited to collecting security issues from
>> users so that they can fixed and released before they are announced on
>> public forums, such as jira. Clearly, I'm +1.
>>
>> On a related note, my plan is to work on getting full Kerberos integration
>> for 0.22. We've come a long way without any authentication in Hadoop, but it
>> is getting important to start closing the holes.
>>
>> -- Owen
>>
>>     
>
>   


From general-return-465-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 17:58:16 2009
Return-Path: <general-return-465-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46371 invoked from network); 26 Aug 2009 17:58:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 17:58:16 -0000
Received: (qmail 30169 invoked by uid 500); 26 Aug 2009 17:58:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30098 invoked by uid 500); 26 Aug 2009 17:58:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30088 invoked by uid 99); 26 Aug 2009 17:58:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 17:58:15 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 209.85.216.191 as permitted sender)
Received: from [209.85.216.191] (HELO mail-px0-f191.google.com) (209.85.216.191)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 17:58:07 +0000
Received: by pxi29 with SMTP id 29so350638pxi.30
        for <general@hadoop.apache.org>; Wed, 26 Aug 2009 10:57:47 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=JTx96o/K+P+uRrDEeoHrrPUB44vqXqYqxY50F7lyvbs=;
        b=hM1DOosxPzdNZroXCKkwMWQzQTSR8LKuvlMkq7asBaqf53dPoBY0324ZRxIQ0urSCs
         qn4haYxxTsrMDQHV73n7zD2my0VNHsyOnLl00XRC4Ze5PNamrEgGiEUETUGRK5uVyytc
         ibdiFAVq6jLAAYbgBjJvo+ikZt7f7Gk03Ajns=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=m2ELgg0mfEaqSa1u4+JFFoLq+IBTU2qMGHdtpGsSRyUA71Jp/g4lNYMwddwn1WRPXb
         a/P1B494yK4HjSSxMJmQHnRWQ0dHxOk3R03ptyRhAb8iPkXVW5by5hhLnDQf+b+qUSVQ
         MjFYst/9vHBk8Ik22vEhQ6TLVndpiPgocs9EA=
MIME-Version: 1.0
Received: by 10.140.192.11 with SMTP id p11mr3925788rvf.92.1251309467600; Wed, 
	26 Aug 2009 10:57:47 -0700 (PDT)
In-Reply-To: <4A94BEFA.6090303@yahoo-inc.com>
References: <CC7CE677-2EEE-47FC-A3B4-859CE2788563@apache.org>
	 <fca34e2c0908251810t3b8b14f9wccce59cdad046bec@mail.gmail.com>
	 <4A94BEFA.6090303@yahoo-inc.com>
Date: Wed, 26 Aug 2009 10:57:47 -0700
Message-ID: <4aa34eb70908261057j75f304den91da21148fdb7e84@mail.gmail.com>
Subject: Re: [VOTE] create security@hadoop.apache.org
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd20fa652069804720f302e
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd20fa652069804720f302e
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

+1

On Tue, Aug 25, 2009 at 9:50 PM, Hemanth Yamijala <yhemanth@yahoo-inc.com>wrote:

> +1
>
>
> Tze Hon Tan wrote:
>
>> +1
>>
>> On Tue, Aug 25, 2009 at 11:42 PM, Owen O'Malley <omalley@apache.org>
>> wrote:
>>
>>
>>
>>> With the upcoming freeze of Map/Reduce 0.21, for the first time, we have
>>> code that is critical from a system security perspective. As outlined on
>>> http://www.apache.org/security/committers.html this will be a private
>>> list
>>> with only committers that are working on the security aspects of Hadoop.
>>> The
>>> use of the list will be strictly limited to collecting security issues
>>> from
>>> users so that they can fixed and released before they are announced on
>>> public forums, such as jira. Clearly, I'm +1.
>>>
>>> On a related note, my plan is to work on getting full Kerberos
>>> integration
>>> for 0.22. We've come a long way without any authentication in Hadoop, but
>>> it
>>> is getting important to start closing the holes.
>>>
>>> -- Owen
>>>
>>>
>>>
>>
>>
>>
>
>

--000e0cd20fa652069804720f302e--

From general-return-466-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 05:04:33 2009
Return-Path: <general-return-466-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54617 invoked from network); 27 Aug 2009 05:04:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 05:04:33 -0000
Received: (qmail 80491 invoked by uid 500); 27 Aug 2009 05:04:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80341 invoked by uid 500); 27 Aug 2009 05:04:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80331 invoked by uid 99); 27 Aug 2009 05:04:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 05:04:32 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 05:04:19 +0000
Received: from SNV-EXPF01.ds.corp.yahoo.com (snv-expf01.ds.corp.yahoo.com [207.126.227.250])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n7R53Z0D086232
	for <general@hadoop.apache.org>; Wed, 26 Aug 2009 22:03:37 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:x-mimeole:content-class:mime-version:
	content-type:subject:date:message-id:x-ms-has-attach:
	x-ms-tnef-correlator:thread-topic:thread-index:from:to:x-originalarrivaltime;
	b=RFX3IR5YWGxgBGrEalaOyBfrccKFlemhoPotfvrJw2Gvo4HVFHJChCU0ERecyNVi
Received: from SNV-EXVS01.ds.corp.yahoo.com ([216.145.51.185]) by SNV-EXPF01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Wed, 26 Aug 2009 22:03:36 -0700
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01CA26D3.B34FE854"
Subject: Re: [VOTE] create security@hadoop.apache.org
Date: Wed, 26 Aug 2009 22:03:22 -0700
Message-ID: <2C52DBBEC4855C438BB330CB0D3B465903BFADEA@SNV-EXVS01.ds.corp.yahoo.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: [VOTE] create security@hadoop.apache.org
thread-index: Acom07NGPNB/YjapSW20aEMfWfKMgA==
From: "Sharad Agarwal" <sharadag@yahoo-inc.com>
To: <general@hadoop.apache.org>
X-OriginalArrivalTime: 27 Aug 2009 05:03:36.0330 (UTC) FILETIME=[BBBF1EA0:01CA26D3]
X-Virus-Checked: Checked by ClamAV on apache.org

------_=_NextPart_001_01CA26D3.B34FE854
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

+1

Dhruba Borthakur wrote:
> +1
>
> On Tue, Aug 25, 2009 at 9:50 PM, Hemanth Yamijala =
<yhemanth@yahoo-inc.com>wrote:
>
>> +1
>>
>>
>> Tze Hon Tan wrote:
>>
>>> +1
>>>
>>> On Tue, Aug 25, 2009 at 11:42 PM, Owen O'Malley <omalley@apache.org>
>>> wrote:
>>>
>>>
>>>
>>>> With the upcoming freeze of Map/Reduce 0.21, for the first time, we =
have
>>>> code that is critical from a system security perspective. As =
outlined on
>>>> http://www.apache.org/security/committers.html this will be a =
private
>>>> list
>>>> with only committers that are working on the security aspects of =
Hadoop.
>>>> The
>>>> use of the list will be strictly limited to collecting security =
issues
>>>> from
>>>> users so that they can fixed and released before they are announced =
on
>>>> public forums, such as jira. Clearly, I'm +1.
>>>>
>>>> On a related note, my plan is to work on getting full Kerberos
>>>> integration
>>>> for 0.22. We've come a long way without any authentication in =
Hadoop, but
>>>> it
>>>> is getting important to start closing the holes.
>>>>
>>>> -- Owen
>>>>
>>>>
>>>>
>>>
>>>
>>
>


------_=_NextPart_001_01CA26D3.B34FE854--

From general-return-467-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 09:24:20 2009
Return-Path: <general-return-467-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33604 invoked from network); 27 Aug 2009 09:24:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 09:24:20 -0000
Received: (qmail 32853 invoked by uid 500); 27 Aug 2009 09:24:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32785 invoked by uid 500); 27 Aug 2009 09:24:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32775 invoked by uid 99); 27 Aug 2009 09:24:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 09:24:19 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of harshad@students.iiit.ac.in designates 121.242.23.201 as permitted sender)
Received: from [121.242.23.201] (HELO students.iiit.ac.in) (121.242.23.201)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 09:24:10 +0000
MailScanner-NULL-Check: 1251969824.99942@01LAqnQRTzYcD9vuk7wl4w
Received: from students.iiit.ac.in (localhost.localdomain [127.0.0.1])
	by students.iiit.ac.in (8.13.8/8.13.8) with ESMTP id n7R9Nf3i000696
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Thu, 27 Aug 2009 14:53:41 +0530
Received: (from apache@localhost)
	by students.iiit.ac.in (8.13.8/8.14.1/Submit) id n7R9NfKS000650;
	Thu, 27 Aug 2009 14:53:41 +0530
From: harshad@students.iiit.ac.in
X-Authentication-Warning: students.iiit.ac.in: apache set sender to harshad@localhost using -f
Received: from 172.17.11.61
        (SquirrelMail authenticated user harshad)
        by students.iiit.ac.in with HTTP;
        Thu, 27 Aug 2009 14:53:41 +0530 (IST)
Message-ID: <47805.172.17.11.61.1251365021.squirrel@students.iiit.ac.in>
Date: Thu, 27 Aug 2009 14:53:41 +0530 (IST)
Subject: Error in WordCount example in Hadoop
To: general@hadoop.apache.org
Cc: amit_s@research.iiit.ac.in, srinivasg@students.iiit.ac.in,
        padmini_m@research.iiit.ac.in
User-Agent: SquirrelMail/1.4.8-4.el5
MIME-Version: 1.0
Content-Type: text/plain;charset=iso-8859-1
Content-Transfer-Encoding: 8bit
X-Priority: 3 (Normal)
Importance: Normal
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on students.iiit.ac.in
X-yoursite-MailScanner-Information: Please contact the IIIT Server Room  for more information
X-MailScanner-ID: n7R9Nf3i000696
X-yoursite-MailScanner: Found to be clean
X-yoursite-MailScanner-From: harshad@students.iiit.ac.in
X-Virus-Checked: Checked by ClamAV on apache.org
X-Old-Spam-Status: No, score=-1.4 required=5.0 tests=ALL_TRUSTED autolearn=failed
	version=3.2.5, No

Hi all,

When I execute this command on my Hadoop to execute the WordCount example

" bin/hadoop jar wordcount.jar org.myorg.WordCount input output"

it gives me the following exception:

java.lang.ClassNotFoundException: org.myorg.WordCount
	at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
	at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:336)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:158)
	at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
	at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)

Please help me out.

Bye
Harshad Shrikhande



-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.


From general-return-468-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 28 17:19:03 2009
Return-Path: <general-return-468-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31959 invoked from network); 28 Aug 2009 17:19:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Aug 2009 17:19:03 -0000
Received: (qmail 22703 invoked by uid 500); 28 Aug 2009 17:19:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22639 invoked by uid 500); 28 Aug 2009 17:19:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22628 invoked by uid 99); 28 Aug 2009 17:19:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 17:19:02 +0000
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.62.16] (HELO QMTA01.westchester.pa.mail.comcast.net) (76.96.62.16)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 17:18:51 +0000
Received: from OMTA06.westchester.pa.mail.comcast.net ([76.96.62.51])
	by QMTA01.westchester.pa.mail.comcast.net with comcast
	id Zn3S1c00516LCl051tJW26; Fri, 28 Aug 2009 17:18:30 +0000
Received: from tryarticle-lm.corp.yahoo.com ([209.131.62.113])
	by OMTA06.westchester.pa.mail.comcast.net with comcast
	id ZtJD1c00c2SbwD53StJG47; Fri, 28 Aug 2009 17:18:28 +0000
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <47805.172.17.11.61.1251365021.squirrel@students.iiit.ac.in>
Subject: Re: Error in WordCount example in Hadoop
X-Priority: 3 (Normal)
References: <47805.172.17.11.61.1251365021.squirrel@students.iiit.ac.in>
Message-Id: <CED84EFD-3DFB-47F3-8615-56FAB3834911@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Date: Fri, 28 Aug 2009 10:18:12 -0700
Cc: amit_s@research.iiit.ac.in,
 srinivasg@students.iiit.ac.in,
 padmini_m@research.iiit.ac.in
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


On Aug 27, 2009, at 2:23 AM, harshad@students.iiit.ac.in wrote:

> Hi all,
>
> When I execute this command on my Hadoop to execute the WordCount  
> example

Please move map/reduce user questions to mapreduce-user@hadoop.apache.org

-- Owen

From general-return-469-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 28 17:29:46 2009
Return-Path: <general-return-469-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45019 invoked from network); 28 Aug 2009 17:29:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Aug 2009 17:29:46 -0000
Received: (qmail 39303 invoked by uid 500); 28 Aug 2009 17:29:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39232 invoked by uid 500); 28 Aug 2009 17:29:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39222 invoked by uid 99); 28 Aug 2009 17:29:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 17:29:45 +0000
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.30.40] (HELO QMTA04.emeryville.ca.mail.comcast.net) (76.96.30.40)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 17:29:34 +0000
Received: from OMTA07.emeryville.ca.mail.comcast.net ([76.96.30.59])
	by QMTA04.emeryville.ca.mail.comcast.net with comcast
	id Zooo1c0031GXsucA4tVDfA; Fri, 28 Aug 2009 17:29:13 +0000
Received: from tryarticle-lm.corp.yahoo.com ([209.131.62.113])
	by OMTA07.emeryville.ca.mail.comcast.net with comcast
	id ZtV51c0082SbwD58TtV7rg; Fri, 28 Aug 2009 17:29:11 +0000
Message-Id: <14FF0B69-F0F4-4847-8B4B-1D2064558467@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <CC7CE677-2EEE-47FC-A3B4-859CE2788563@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] create security@hadoop.apache.org
Date: Fri, 28 Aug 2009 10:29:04 -0700
References: <CC7CE677-2EEE-47FC-A3B4-859CE2788563@apache.org>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


On Aug 25, 2009, at 8:42 AM, Owen O'Malley wrote:

> With the upcoming freeze of Map/Reduce 0.21, for the first time, we  
> have code that is critical from a system security perspective. As  
> outlined on http://www.apache.org/security/committers.html this will  
> be a private list with only committers that are working on the  
> security aspects of Hadoop. The use of the list will be strictly  
> limited to collecting security issues from users so that they can  
> fixed and released before they are announced on public forums, such  
> as jira. Clearly, I'm +1.

With 13 +1's (7 from the PMC) and no -1's, the vote passes. I'll file  
the jira to create the lists.

https://issues.apache.org/jira/browse/INFRA-2203

-- Owen

From general-return-470-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 30 00:55:07 2009
Return-Path: <general-return-470-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 62342 invoked from network); 30 Aug 2009 00:55:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Aug 2009 00:55:07 -0000
Received: (qmail 72600 invoked by uid 500); 30 Aug 2009 00:55:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72517 invoked by uid 500); 30 Aug 2009 00:55:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 94280 invoked by uid 99); 29 Aug 2009 21:05:04 -0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of deepak.halale@gmail.com designates 209.85.219.208 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=ie0rv9DN6PKBHhRjUbJETtBm+B4GQrBGp5pXYC+YAYM=;
        b=mPu3UgpMb8Z0p5Z5hyL5X85eYWHD3oEGKqr9478fQ4gpIbmu3vbEI0NmUrmUvUTcMg
         K12uGdIQqkmy4oDxrG1YruoZf7DiVfxz0jzV972/3O1vvpunksenKy7i4/cx+zdHGUWF
         br3M6GhNR677Lu1klHIeqRVVht4PUN3qtJjaA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=TecKKW3mnGAENqsHFt9uRjwhYUUIIUj9LUDS5CrbFEqOfJMO/7ssj9nJErTawAXAM6
         Ib6uZTZ1Buy++uZz+WpoJNr9T3aSsu3EvcPDPdL8E67JpQeXb3W7Un8VOauET9xDGd/u
         fkjvpquRbnEQH2BJL+XUN+rtigwC4JOPfhGa8=
MIME-Version: 1.0
Date: Sat, 29 Aug 2009 17:04:35 -0400
Message-ID: <15678300908291404p5554c8ewda75d45d87bc5a75@mail.gmail.com>
Subject: Hadoop mapReduce clarification
From: Deepak Halale <deepak.halale@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d783fcdf426604724e25a5
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d783fcdf426604724e25a5
Content-Type: text/plain; charset=ISO-8859-1

Hi
I am new to Hadoop and trying to write mapReduce job which gets data from
database
i.e x number of fields and then processes it .

can you point me in direction where to get this information.

I was able to run the example mapReduce wordcount sample on single machine ,
but not getting the gist how the data with mulitpe columns in the flat file
or otherwise is processed


Thank you
Deepak H

--0016e6d783fcdf426604724e25a5--

From general-return-471-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 30 03:41:51 2009
Return-Path: <general-return-471-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87957 invoked from network); 30 Aug 2009 03:41:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Aug 2009 03:41:51 -0000
Received: (qmail 15785 invoked by uid 500); 30 Aug 2009 03:41:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15646 invoked by uid 500); 30 Aug 2009 03:41:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15625 invoked by uid 99); 30 Aug 2009 03:41:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 30 Aug 2009 03:41:50 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yeahyf@gmail.com designates 209.85.220.225 as permitted sender)
Received: from [209.85.220.225] (HELO mail-fx0-f225.google.com) (209.85.220.225)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 30 Aug 2009 03:41:40 +0000
Received: by fxm25 with SMTP id 25so2251071fxm.29
        for <general@hadoop.apache.org>; Sat, 29 Aug 2009 20:41:20 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=m4bBv+eFpDzf2Sp+8I1WB+ZmotsTrpGIuKvpUF3FRp4=;
        b=ICAD75RSfILB+hkZkJXREDdBTVNGQAOUOU0JN99Hile8mtBZYufrRiXDQVDvzgOEEB
         C3FzH5jypSm8VuY1ya8DK2C8+TE1Yxk4PcgojZ/MKpZoAGy7HQrvqNCucGkHH9B3xUH2
         7MDh3ni53DJ7MygtKsf9LvNyVPX+yxdEPk16I=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=q9yEd0EjrT9cGwFmfQ/SGe7yIX8o9o9mK1JJdcujeAZGsKuTiEVREshrVNYER3oso3
         4BM6oE2+Yr3QEzicU0aJVWOV+7+fNYc236kIgoLVBSBHVndTpevGlZifN4vUXf0B0rMb
         kMSkCP9MDHdQS+ikgtNRkiO2ordwFncJZEpsg=
MIME-Version: 1.0
Received: by 10.239.179.80 with SMTP id c16mr258364hbg.155.1251603680558; Sat, 
	29 Aug 2009 20:41:20 -0700 (PDT)
In-Reply-To: <CED84EFD-3DFB-47F3-8615-56FAB3834911@apache.org>
References: <47805.172.17.11.61.1251365021.squirrel@students.iiit.ac.in>
	 <CED84EFD-3DFB-47F3-8615-56FAB3834911@apache.org>
Date: Sun, 30 Aug 2009 11:41:20 +0800
Message-ID: <3b13e4710908292041t45d16850q713768dcef59afae@mail.gmail.com>
Subject: Re: Error in WordCount example in Hadoop
From: yangfeng <yeahyf@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f7cc06c780a7047253b043
X-Virus-Checked: Checked by ClamAV on apache.org

--001485f7cc06c780a7047253b043
Content-Type: text/plain; charset=ISO-8859-1

Before, please exec "export HADOOP_CLASSPATH=."

2009/8/29 Owen O'Malley <omalley@apache.org>

>
> On Aug 27, 2009, at 2:23 AM, harshad@students.iiit.ac.in wrote:
>
>  Hi all,
>>
>> When I execute this command on my Hadoop to execute the WordCount example
>>
>
> Please move map/reduce user questions to mapreduce-user@hadoop.apache.org
>
> -- Owen
>

--001485f7cc06c780a7047253b043--

From general-return-98-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 04 21:05:22 2008
Return-Path: <general-return-98-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 84030 invoked from network); 4 Dec 2008 21:05:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 4 Dec 2008 21:05:21 -0000
Received: (qmail 5862 invoked by uid 500); 4 Dec 2008 21:05:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5775 invoked by uid 500); 4 Dec 2008 21:05:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5694 invoked by uid 99); 4 Dec 2008 21:05:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Dec 2008 13:05:25 -0800
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [207.126.228.149] (HELO rsmtp1.corp.yahoo.com) (207.126.228.149)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Dec 2008 21:03:56 +0000
Received: from [10.72.73.120] (snvvpn1-10-72-73-c120.hq.corp.yahoo.com [10.72.73.120])
	(authenticated bits=0)
	by rsmtp1.corp.yahoo.com (8.13.8/8.13.8/y.rout) with ESMTP id mB4L3PbE057513
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Thu, 4 Dec 2008 13:03:26 -0800 (PST)
Message-ID: <4938459D.9020401@apache.org>
Date: Thu, 04 Dec 2008 13:03:25 -0800
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.18 (X11/20081125)
MIME-Version: 1.0
To: announce@apache.org, zookeeper-dev@hadoop.apache.org,
        zookeeper-user@hadoop.apache.org, core-user@hadoop.apache.org,
        general@hadoop.apache.org
Subject: [ANNOUNCE] Apache ZooKeeper 3.0.1
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

The Apache ZooKeeper team is proud to announce Apache ZooKeeper version 
3.0.1.

ZooKeeper is a high-performance coordination service for distributed 
applications. It exposes common services - such as naming, configuration 
management, synchronization, and group services - in a simple interface 
so you don't have to write them from scratch. You can use it 
off-the-shelf to implement consensus, group management, leader election, 
and presence protocols. And you can build on it for your own, specific 
needs.

If you are upgrading from version 2.2.1 on SourceForge be sure to review 
the 3.0.1 release notes for migration instructions.

For ZooKeeper release details and downloads, visit:
http://hadoop.apache.org/zookeeper/releases.html

ZooKeeper 3.0.1 Release Notes are at:
http://hadoop.apache.org/zookeeper/docs/r3.0.1/releasenotes.html

Regards,

The ZooKeeper Team

From general-return-99-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Dec 08 19:07:01 2008
Return-Path: <general-return-99-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 62405 invoked from network); 8 Dec 2008 19:07:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 8 Dec 2008 19:07:01 -0000
Received: (qmail 95201 invoked by uid 500); 8 Dec 2008 19:07:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94809 invoked by uid 500); 8 Dec 2008 19:07:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94694 invoked by uid 99); 8 Dec 2008 19:07:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Dec 2008 11:07:03 -0800
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Dec 2008 19:06:48 +0000
Received: from SNV-EXBH01.ds.corp.yahoo.com (snv-exbh01.ds.corp.yahoo.com [207.126.227.249])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id mB8J680c024701;
	Mon, 8 Dec 2008 11:06:08 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:x-mimeole:content-class:mime-version:
	content-type:subject:date:message-id:x-ms-has-attach:
	x-ms-tnef-correlator:thread-topic:thread-index:from:to:return-path:x-originalarrivaltime;
	b=A2k5NMkYJGRv5NNLsanFFTOQ9LvUQxc7+NG7rVp8T2q0irxGZeEWT6bDJQN9eHaF
Received: from SNV-EXVS09.ds.corp.yahoo.com ([207.126.227.86]) by SNV-EXBH01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Mon, 8 Dec 2008 11:06:08 -0800
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C95968.07187B08"
Subject: Pig 0.1.1 Released!
Date: Mon, 8 Dec 2008 11:06:07 -0800
Message-ID: <236D5828CF7F5749AF1BCB7F87596DE102F366E6@SNV-EXVS09.ds.corp.yahoo.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Pig 0.1.1 Released!
Thread-Index: AclZaAZ1zoQjfPbJQFu1+QFgGZ9uHg==
From: "Olga Natkovich" <olgan@yahoo-inc.com>
To: <announce@apache.org>, <pig-dev@hadoop.apache.org>,
        <pig-user@hadoop.apache.org>, <core-user@hadoop.apache.org>,
        <general@hadoop.apache.org>
X-OriginalArrivalTime: 08 Dec 2008 19:06:08.0734 (UTC) FILETIME=[07194FE0:01C95968]
X-Virus-Checked: Checked by ClamAV on apache.org

------_=_NextPart_001_01C95968.07187B08
Content-Type: text/plain;
	charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable

Hi,

Pig  team is happy to announce Pig 0.1.1 release.=20

Pig is Hadoop subproject which provides high-level data-flow language
and execution framework for parallel computation on Hadoop clusters.
More details about Pig can be found at http://hadoop.apache.org/pig/.

The highlight of this release is integration with Hadoop 0.18. The
details of the release can be found at
http://hadoop.apache.org/pig/releases.html.

Olga


------_=_NextPart_001_01C95968.07187B08--

From general-return-100-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Dec 21 18:38:59 2008
Return-Path: <general-return-100-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 86527 invoked from network); 21 Dec 2008 18:38:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 21 Dec 2008 18:38:56 -0000
Received: (qmail 56490 invoked by uid 500); 21 Dec 2008 18:38:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56468 invoked by uid 500); 21 Dec 2008 18:38:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 16326 invoked by uid 99); 20 Dec 2008 09:37:03 -0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of samqjf@163.com designates 220.181.12.14 as permitted sender)
Date: Sat, 20 Dec 2008 17:36:46 +0800
From: "samqjf" <samqjf@163.com>
To: general <general@hadoop.apache.org>
Subject: how to Learning the hadoop
Message-ID: <200812201736252969570@163.com>
X-mailer: Foxmail 6, 9, 201, 16 [cn]
Mime-Version: 1.0
Content-Type: multipart/alternative;
	boundary="=====003_Dragon046188523748_====="
X-Coremail-Antispam: 1Uf129KBjDUn29KB7ZKAUJUUUUUYxn0WfASr-VFAUDa7-sFnT
	9fnUUIcSsGvfJTRUUUjSkYjxAI6xkYrwAYjxAI6xAIw28IcVAK0I8IjxAxM7k0a2IF6F1U
	M7kC6x804xWl14x267AKxVWUJVW8JwAFxVCF77xC6IxKo4kEV4yl1I0EscIYIxCEI4klw4
	CSwwAFIxvE14AKwVWUJVWUGwAq048E620vw7xCY7CEYx808205XwAq048E620vw7xCY7CE
	4x8GYI0EYx1l5I8CrVCF0I0E4I0vr24lYx0E2Ix0cI8IcVAFwI0_JrI_JrylYx0Ex4A2js
	IE14v26r1j6r4UM4x0Y48IcxkI7VAKI48JM4IEnf9ElVAFpTB2q-sK649IAas0WaI_GwCj
	xxvEa2IrMxkFs20EY4vE8sxKj4xv1wCY1Ik26cxK6xAEc7vF620IrwCY1Ik26cxK6s8Yrw
	CY02Avz4vE14v_GFWlc2IjII80xcxEwVAKI48JMxCjnVAqn7xvrwC2zVAF1VAY17CE14v2
	6r1j6r15Ms8CjcxG0xvEn4CqF7xvrbIYCTnIWIevJa73UjIFyTuYvjxUg4SrUUUUU
X-Virus-Checked: Checked by ClamAV on apache.org

--=====003_Dragon046188523748_=====
Content-Type: text/plain;
	charset="gb2312"
Content-Transfer-Encoding: 7bit



Ladies and gentlemen, I would like to know what to do hadoop. What software can run on it?


Where there are Chinese information about the hadoop?


2008-12-20 



samqjf 

--=====003_Dragon046188523748_=====--



From general-return-101-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Dec 21 18:56:24 2008
Return-Path: <general-return-101-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 89678 invoked from network); 21 Dec 2008 18:56:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 21 Dec 2008 18:56:24 -0000
Received: (qmail 67033 invoked by uid 500); 21 Dec 2008 18:56:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67008 invoked by uid 500); 21 Dec 2008 18:56:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66996 invoked by uid 99); 21 Dec 2008 18:56:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 21 Dec 2008 10:56:23 -0800
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of kerdezixe@gmail.com designates 209.85.219.21 as permitted sender)
Received: from [209.85.219.21] (HELO mail-ew0-f21.google.com) (209.85.219.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 21 Dec 2008 18:56:16 +0000
Received: by ewy14 with SMTP id 14so1846701ewy.5
        for <general@hadoop.apache.org>; Sun, 21 Dec 2008 10:55:55 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:to
         :subject:in-reply-to:mime-version:content-type
         :content-transfer-encoding:content-disposition:references;
        bh=g4gr7Q+GI/JygIR8KZ6RDeE377HDeMseiknpges1BEQ=;
        b=bnkNyCVbaBiCFfZcyTiXVK9w57uBcavck8hwtE4ZLSSihhKmlzJLQHN2cMUC3pOi2g
         JGotKec0ncsxz2i3fI6jZe22TGW0jqOyTrALAmJagXeyCIeYDawD1LrYEg/eJDexEBu7
         0QBnFXDYuju3YJCzfN+CUaeKsAYTdHETx9lEc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version
         :content-type:content-transfer-encoding:content-disposition
         :references;
        b=dFhJd7znsYtxTboIA80EdsvePwTCDyv9V4i/+vIhr2ALDGaN7P6kFvFMAAsIJsOVo3
         U5mX1KJVBpqcW4+JJmx6+GvmyR3gsUc+7iYiWTcq8K8Z8dXzSoeU/Yl/576mKyvFQmvk
         +tfZ3Dae+2mo2p/kkPNkliQRdu7sMMorGcaCc=
Received: by 10.210.67.4 with SMTP id p4mr6420772eba.157.1229885755580;
        Sun, 21 Dec 2008 10:55:55 -0800 (PST)
Received: by 10.210.54.16 with HTTP; Sun, 21 Dec 2008 10:55:55 -0800 (PST)
Message-ID: <8a1bfe660812211055g651dc688p2f0d8c38bc87cccc@mail.gmail.com>
Date: Sun, 21 Dec 2008 19:55:55 +0100
From: "Laurent Laborde" <kerdezixe@gmail.com>
To: general@hadoop.apache.org
Subject: Re: how to Learning the hadoop
In-Reply-To: <200812201736252969570@163.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <200812201736252969570@163.com>
X-Virus-Checked: Checked by ClamAV on apache.org

On Sat, Dec 20, 2008 at 10:36 AM, samqjf <samqjf@163.com> wrote:
>
> Ladies and gentlemen, I would like to know what to do hadoop. What software can run on it?

Hi !
I'm still new to hadoop, but you can do a lot of things with it.

The 2 keys of hadoop are :
- HDFS : A distributed filesystem, something similar to the Google FileSystem.
- MapReduce : Well... there is a lot of paper about mapreduce. (mostly
from Google, they "invented" it).

As long as you have a small (a few MB) or a very large dataset (Many
TB, PB ?) and a problem that can be parallised with MapReduce, Hadoop
may be for you :)

Hadoop have some very interesting Subprojects if you're not planning
to deal directly with MapReduce task.
- HBase : an implementation of Google BigTable.
- Pig : i don't know how to explain it, but i'm crazy about it. it's
really cool. Very usefull for, eg : parsing huge file (log file, text,
csv, ...), filter, group, order, ...
- Hutsh : A search engine ... that use Lucene and hadoop. Something
similar to Google Search ;)
- and more ...

I suggest to read the Google papers about mapreduce, bigtable, GFS.
Papers about hadoop
Paper about the many subprojects of hadoop.

Please correct me if i'm wrong, i'm still newbie here :)
(so far... the most i did was a search engine with around 2 millions
URL indexed).

I'm expecting to use it at work for log parsing. (webiste with millons
of hits/day).
But not yet, i'm just playing with hadoop for now :)

-- 
F4FQM
Kerunix Flan
Laurent Laborde

From general-return-102-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Dec 26 19:42:17 2008
Return-Path: <general-return-102-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 57127 invoked from network); 26 Dec 2008 19:42:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 26 Dec 2008 19:42:17 -0000
Received: (qmail 89582 invoked by uid 500); 26 Dec 2008 19:42:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89556 invoked by uid 500); 26 Dec 2008 19:42:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89545 invoked by uid 99); 26 Dec 2008 19:42:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Dec 2008 11:42:16 -0800
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [96.9.130.69] (HELO sophia.codeminders.com) (96.9.130.69)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Dec 2008 19:42:08 +0000
Received: from [10.86.81.100] (dsl092-248-163.sfo4.dsl.speakeasy.net [66.92.248.163])
	(using TLSv1 with cipher AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sophia.codeminders.com (Postfix) with ESMTPSA id 4BA4A8186D
	for <general@hadoop.apache.org>; Fri, 26 Dec 2008 11:41:53 -0800 (PST)
Message-Id: <DBDE9CCF-061D-4AFF-8B16-5360495E8018@codeminders.com>
From: Vadim Zaliva <lord@codeminders.com>
To: general@hadoop.apache.org
Content-Type: multipart/signed; boundary=Apple-Mail-70-757718558; micalg=sha1; protocol="application/pkcs7-signature"
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: cluster machines dying
Date: Fri, 26 Dec 2008 11:41:45 -0800
X-Mailer: Apple Mail (2.930.3)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-70-757718558
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit

Hi!

I am experiencing a strange problem with hadoop-0.19. I have set up  
small
cluster with 4 machines. These are older machines, which have been under
heavy use for years. Once I have started to run hadoop on them they
unexplainably die after approximately 24 hours of use.

I have not even run any tasks on they yet. They are now accumulating  
some data
files, being copied in DFS. There is nothing in syslog prior to the  
crash.

Memory usage is moderate (no swap is used). CPU load is under 1. Does  
anybody seen
something like this? What steps I could take to diagnose this problem  
further?

Thanks!

Sincerely,
Vadim

P.S.

Java version: java version "1.6.0_11"

Kernel: Linux datamining1 2.6.23.1-42.fc8 #1 SMP Tue Oct 30 13:55:12  
EDT 2007 i686 i686 i386 GNU/Linux

--
"La perfection est atteinte non quand il ne reste rien a ajouter, mais
quand il ne reste rien a enlever."  (Antoine de Saint-Exupery)




--Apple-Mail-70-757718558
Content-Disposition: attachment;
	filename=smime.p7s
Content-Type: application/pkcs7-signature;
	name=smime.p7s
Content-Transfer-Encoding: base64

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIGQDCCAvkw
ggJioAMCAQICEERMoY4Lfo7a7sLhgKLeLcswDQYJKoZIhvcNAQEFBQAwYjELMAkGA1UEBhMCWkEx
JTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQ
ZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA4MDgwNDE4MDY1MloXDTA5MDgwNDE4MDY1
MlowXTEPMA0GA1UEBBMGWmFsaXZhMQ4wDAYDVQQqEwVWYWRpbTEVMBMGA1UEAxMMVmFkaW0gWmFs
aXZhMSMwIQYJKoZIhvcNAQkBFhRsb3JkQGNvZGVtaW5kZXJzLmNvbTCCASIwDQYJKoZIhvcNAQEB
BQADggEPADCCAQoCggEBAJp/hSCd9qKL3z67jLQ/rBeXMRgkL43ZL7wTXjUnS8OfVGYeRT2jLN06
JEouCV0ndiQwDT1Am8xq2IdZC1h/CEZJGrvj663uvrxFCadh2pqT2pFnq7VCiBsekFyc/uhE8NOF
It8doKZmzKrQQJ2iX94EdhEIP/mACO3+n9j0TBqSa7z4hfn8DYNW6MkZX4VGlcAm/d3ueTlcoiQ5
lrcTw3i5aK4SBTiReLaHQKoyU2FYvk0A3728MQjcU4sIJDpUTKCnzLMLmJHaSgdq9Ey7nDSUKb0F
CHVjsgp+dr/6ozRgmvIcs+WfEo1ELBuFfi+U6ESXlOy26qH3coOS7LXT9gsCAwEAAaMxMC8wHwYD
VR0RBBgwFoEUbG9yZEBjb2RlbWluZGVycy5jb20wDAYDVR0TAQH/BAIwADANBgkqhkiG9w0BAQUF
AAOBgQCbuIPK+DCnk/9WGG2Wv1kdmqj41IjBR7oXQ5GsWfDi6uKGWdzz/Z628YcirynXqr3bgJze
IvJ5XbkFuGe0SHl5HsXtKBeTNjqBS+lZQ8+eXkxuJTrFGzvn72EhY1E1NmYntVaCblizQmZIu/aq
zScU5ibqWqntjn8P7Iu3GSE9QTCCAz8wggKooAMCAQICAQ0wDQYJKoZIhvcNAQEFBQAwgdExCzAJ
BgNVBAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJuIENhcGUxEjAQBgNVBAcTCUNhcGUgVG93bjEaMBgG
A1UEChMRVGhhd3RlIENvbnN1bHRpbmcxKDAmBgNVBAsTH0NlcnRpZmljYXRpb24gU2VydmljZXMg
RGl2aXNpb24xJDAiBgNVBAMTG1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBDQTErMCkGCSqGSIb3
DQEJARYccGVyc29uYWwtZnJlZW1haWxAdGhhd3RlLmNvbTAeFw0wMzA3MTcwMDAwMDBaFw0xMzA3
MTYyMzU5NTlaMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5
KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBDQTCBnzAN
BgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAxKY8VXNV+065yplaHmjAdQRwnd/p/6Me7L3N9VvyGna9
fww6YfK/Uc4B1OVQCjDXAmNaLIkVcI7dyfArhVqqP3FWy688Cwfn8R+RNiQqE88r1fOCdz0Dviv+
uxg+B79AgAJk16emu59l0cUqVIUPSAR/p7bRPGEEQB5kGXJgt/sCAwEAAaOBlDCBkTASBgNVHRMB
Af8ECDAGAQH/AgEAMEMGA1UdHwQ8MDowOKA2oDSGMmh0dHA6Ly9jcmwudGhhd3RlLmNvbS9UaGF3
dGVQZXJzb25hbEZyZWVtYWlsQ0EuY3JsMAsGA1UdDwQEAwIBBjApBgNVHREEIjAgpB4wHDEaMBgG
A1UEAxMRUHJpdmF0ZUxhYmVsMi0xMzgwDQYJKoZIhvcNAQEFBQADgYEASIzRUIPqCy7MDaNmrGcP
f6+svsIXoUOWlJ1/TCG4+DYfqi2fNi/A9BxQIJNwPP2t4WFiw9k6GX6EsZkbAMUaC4J0niVQlGLH
2ydxVyWN3amcOY6MIE9lX5Xa9/eH1sYITq726jTlEBpbNU1341YheILcIRk13iSx0x1G/11fZU8x
ggMQMIIDDAIBATB2MGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAo
UHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBDQQIQ
REyhjgt+jtruwuGAot4tyzAJBgUrDgMCGgUAoIIBbzAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcB
MBwGCSqGSIb3DQEJBTEPFw0wODEyMjYxOTQxNDVaMCMGCSqGSIb3DQEJBDEWBBQ9CZgpmeZf619C
T+CDb9ihdEQY3DCBhQYJKwYBBAGCNxAEMXgwdjBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhh
d3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVt
YWlsIElzc3VpbmcgQ0ECEERMoY4Lfo7a7sLhgKLeLcswgYcGCyqGSIb3DQEJEAILMXigdjBiMQsw
CQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UE
AxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0ECEERMoY4Lfo7a7sLhgKLeLcsw
DQYJKoZIhvcNAQEBBQAEggEAQB/kQA+lGNjfj/nRsZYpZlMenH/nXstmmDokTBFMb1iu3u+XUc67
tf1hJB3vpFXuYZ7zq6qxXiM/5nwiwWF5QCD7oflHMlqFrs22QtfXt3jjHNXJYjinPnLbwx2EWwd4
MQt9CgFYQXDPY4YBWGKZOPxXpiw9bsGwrN8ZPjRxyP/zeH2KeqVuhJBt9WRYQwNZghsErgZ/hfc2
M8UvsSmEZlOP3O0gyq8V6Qb/QOfgPRFZtlFO5kww4FLqu/OF46SMi+IYqqhvys7szm5YQSo1u1hp
4v0IJ0kwkbSvRqqek+LE9AYNA0aER7gecKagucNCwMeBEiHizmSexL9FX5EXDgAAAAAAAA==

--Apple-Mail-70-757718558--

From general-return-1722-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 01 05:24:15 2010
Return-Path: <general-return-1722-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 20274 invoked from network); 1 Jul 2010 05:24:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jul 2010 05:24:14 -0000
Received: (qmail 56628 invoked by uid 500); 1 Jul 2010 05:24:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56318 invoked by uid 500); 1 Jul 2010 05:24:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56310 invoked by uid 99); 1 Jul 2010 05:24:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 05:24:10 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of renatoj.marroquin@gmail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 05:24:03 +0000
Received: by iwn39 with SMTP id 39so2089415iwn.35
        for <general@hadoop.apache.org>; Wed, 30 Jun 2010 22:23:42 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=CzXMv/X/txZZItX+Rm+O7/CBqHm9J/OrzGR8QK9nx60=;
        b=A7W+FrJLB7GbPx6p9EV/FjF5M9vAaeaG/zCsp1US+NuV5zRS/BK+Po58jPifMgXOp7
         KLYrsi3GrkAvBckZzRkyKS7+VTxKCqbrXPlwh9qvtuR757y0AKwOXRU5jXISnE5/tSm4
         q1xdp/SVsMiQqpdc00Liu1XCn6A1mXhAKO4WY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=FPjbTFlxA9HKbNBD+3X2fAgxK8u3zWlKl4bhB0odNk8h4nbkMJmobeQghuYLW6Gs/l
         f6IdpIGKfDAhfAOcunWUSpmXuPI3be0GGlAx7PD5EsZh9AtgHFePvqNHD/iIkd4rhJj6
         BBYXnS3yxAlj7MY3My8RGbFiPVk/HCVIzab08=
MIME-Version: 1.0
Received: by 10.231.169.74 with SMTP id x10mr10291542iby.121.1277961822806; 
	Wed, 30 Jun 2010 22:23:42 -0700 (PDT)
Received: by 10.231.33.200 with HTTP; Wed, 30 Jun 2010 22:23:42 -0700 (PDT)
In-Reply-To: <AANLkTilAe0jkMxwf1wg1u6TJr8sWUObgayKU5RJQtMZL@mail.gmail.com>
References: <AANLkTine5opXB6KXLejtKYC7zyiRam6a2Qw7RwkPR8fo@mail.gmail.com>
	<30B5A69F3E33FE4C83F4C832F5D6E2CF0254F734@CINMLVEM12.e2k.ad.ge.com>
	<AANLkTilAe0jkMxwf1wg1u6TJr8sWUObgayKU5RJQtMZL@mail.gmail.com>
Date: Thu, 1 Jul 2010 02:23:42 -0300
Message-ID: <AANLkTinitR3cqiLnkxo2H-AQjRMELzvyLAQ12KbkhFng@mail.gmail.com>
Subject: Re: Hadoop Platform
From: =?ISO-8859-1?Q?Renato_Marroqu=EDn_Mogrovejo?= <renatoj.marroquin@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016369c8ab27c2564048a4cac2b
X-Virus-Checked: Checked by ClamAV on apache.org

--0016369c8ab27c2564048a4cac2b
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Thanks everybody,

I will definitely try to get in contact with OpenCirrus, and if not I can a=
t
least test with some amazon hw so I can prove the application concept. And
the Whirr project is really interesting, I will give it a look specially
because we really don't want to depend on a single service provider.
Thanks again for the suggestions.


Renato M.


2010/6/30 Jeff Hammerbacher <hammer@cloudera.com>

> If you want to use an open API and be cloud-provider agnostic, check out
> Whirr: http://incubator.apache.org/projects/whirr.html.
>
> On Wed, Jun 30, 2010 at 10:52 AM, Yan, Weizhong (GE, Research)
> <yan@ge.com>wrote:
>
> > Why don't you just simply try Amazon elastic MapReduce?
> > http://aws.amazon.com/elasticmapreduce/
> >
> > -----Original Message-----
> > From: Renato Marroqu=EDn Mogrovejo [mailto:renatoj.marroquin@gmail.com]
> > Sent: Wednesday, June 30, 2010 11:47 AM
> > To: general@hadoop.apache.org
> > Subject: Hadoop Platform
> >
> > Hi everyone,
> >
> > I read a while ago that maybe IBM or Yahoo! were going to provide a
> "cloud"
> > for academic purposes. Does anybody know if those intentions
> materialised?
> > or if there any other ( free or not so expensive ) platform to test
> Hadoop
> > jobs??? I mean, obviously there would be several limitations on it, but=
 I
> > don't have access to a cluster and I have some Hadoop jobs I need to te=
st
> > them, so I can prove to my people the power of it (:
> > Thanks in advance.
> >
> >
> > Renato M.
> >
>

--0016369c8ab27c2564048a4cac2b--

From general-return-1723-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 01 06:42:16 2010
Return-Path: <general-return-1723-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45506 invoked from network); 1 Jul 2010 06:42:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jul 2010 06:42:15 -0000
Received: (qmail 22260 invoked by uid 500); 1 Jul 2010 06:42:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21794 invoked by uid 500); 1 Jul 2010 06:42:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21785 invoked by uid 99); 1 Jul 2010 06:42:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 06:42:10 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [68.180.199.172] (HELO web46307.mail.sp1.yahoo.com) (68.180.199.172)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 01 Jul 2010 06:42:01 +0000
Received: (qmail 53053 invoked by uid 60001); 1 Jul 2010 06:40:35 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1277966435; bh=DTN5iQ/XmCC4leYsRyWsfZNsUc5w5XmtE5PQS4tuumw=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=va76FPDEEpVczUu4ZSGPUUYiQivnNZdOTqx9F2r3w67K5gxNfPR/J0iqBN7a1MI0QYtxzRBgqW4fncU3T1GKeettNgkv++KjNqC0/h4bmgmI56SvWfxQdq5AkRbxta2Fv0DTjZ/poXl9ZZ6Epvlf+gMN72tBGCz5OAfVsiZiLig=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=5tAsdZ/xQXZBGPtQdo40H3zL8867ATUBlbtlWItui1EdDx4MSFzohXPu9acqSOzMN//xdLSj9pUwbFrhohUOfkol+swxiEXNMtNZI4RHNOWBnN+IaEKJ4VZWSaAyAbnkY/2GWprRlp9WUO8j27dfNRza3cK4hNPM3q8ZKXGCckY=;
Message-ID: <699135.52032.qm@web46307.mail.sp1.yahoo.com>
X-YMail-OSG: C6Nly2QVM1lztWj.IlWHpvTq4DfHVEuSIwNuqDBVMo87H52
 fSlsc6vtgxhZbQgqXnfvZLdSe64GWv6tTw2JTKTIT9Uyf0VR50opNIogLbxW
 PXz8woJ0MJIWrMYG1KaMUUMq1GmTycEKIT5210eGhy5gfAR2w6tFsidXSZOC
 2uPozq7zDR7hb4aoN_zbZDmCSYAA_nIimsIisvwQJfmL99Rej1YULkTXZrAS
 UK9JQIu8ZwAKTtRZ2LEdplV7snqo8_tIr.L_8Jrn9tfiqKqc496IcT2kyRyz
 R7n5heZQLqfVftdVrt6N5WVd8JPw-
Received: from [188.74.76.16] by web46307.mail.sp1.yahoo.com via HTTP; Wed, 30 Jun 2010 23:40:35 PDT
X-Mailer: YahooMailRC/420.4 YahooMailWebService/0.8.104.274457
Date: Wed, 30 Jun 2010 23:40:35 -0700 (PDT)
From: abc xyz <fabc_xyz111@yahoo.com>
Subject: Chaining Map-Reduce
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1002225173-1277966435=:52032"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-1002225173-1277966435=:52032
Content-Type: text/plain; charset=us-ascii

Hi everyone,

I know that it is possible in Hadoop 0.19 and onwards to have any number of 
mappers, then a reducer, and then any number of mappers chained in a single job. 
Is it possible to have more than one reducers in this chain? If yes, how??

Thanks



      
--0-1002225173-1277966435=:52032--

From general-return-1724-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 01 12:48:44 2010
Return-Path: <general-return-1724-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 357 invoked from network); 1 Jul 2010 12:48:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jul 2010 12:48:44 -0000
Received: (qmail 67811 invoked by uid 500); 1 Jul 2010 12:48:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67655 invoked by uid 500); 1 Jul 2010 12:48:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67646 invoked by uid 99); 1 Jul 2010 12:48:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 12:48:39 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 12:48:30 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id A112EB7CF8
	for <general@hadoop.apache.org>; Thu,  1 Jul 2010 13:47:39 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id FMozr8ZZlj8i for <general@hadoop.apache.org>;
	Thu,  1 Jul 2010 13:47:39 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id E3565B7CEA
	for <general@hadoop.apache.org>; Thu,  1 Jul 2010 13:47:38 +0100 (BST)
MailScanner-NULL-Check: 1278593246.49046@mT4r/Skfz20V8zgUMMSjOw
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o61ClOW4005760
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Thu, 1 Jul 2010 13:47:25 +0100 (BST)
Message-ID: <4C2C8E5C.8070305@apache.org>
Date: Thu, 01 Jul 2010 13:47:24 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100228)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Hadoop Platform
References: <AANLkTine5opXB6KXLejtKYC7zyiRam6a2Qw7RwkPR8fo@mail.gmail.com> <AANLkTil0DwbxxIFB2lIWRwo_p4n71Yf7cDGRPBisvY0u@mail.gmail.com>
In-Reply-To: <AANLkTil0DwbxxIFB2lIWRwo_p4n71Yf7cDGRPBisvY0u@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o61ClOW4005760
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

Hemanth Yamijala wrote:
> Renato,
> 
>> I read a while ago that maybe IBM or Yahoo! were going to provide a "cloud"
>> for academic purposes. Does anybody know if those intentions materialised?
>> or if there any other ( free or not so expensive ) platform to test Hadoop
>> jobs??? I mean, obviously there would be several limitations on it, but I
>> don't have access to a cluster and I have some Hadoop jobs I need to test
>> them, so I can prove to my people the power of it (:
>> Thanks in advance.
> 
> Can you check if OpenCirrus (https://opencirrus.org/) can fit your
> bill ? It seems like some one like a faculty member can request access
> by submitting a proposal.
> 

The problem that OpenCirrus and presumably whatever IBM  and Google are 
  doing is that it has to be research; the funding/tax situation doesn't 
let you do "product" or "play"; this is why we can't use those boxes for 
beta-testing Hadoop 0.21 unless we have a dataset and app that really is 
research. The other one is datasets, nobody wants to dedicate PB of 
storage to individual project's datasets, far easier to have shared 
datasets of one form or other.

what OpenCirrus do offer is the ability not just to run MR jobs over a 
cluster but the ability to rededicate a chunk of physical datacentre 
(not virtualised, real machines) to different uses. Which means you need 
a process for allocating machines, getting your research done on them 
etc. It's very biased towards "research of what the future of cloud 
computing" will be, rather than just "some boxes to run MR jobs on". If 
you do have some interesting ideas in that area, and you are at a 
research establishment which is part of the consortium, then yes, try 
and get some of the machines.


From general-return-1725-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 01 12:54:47 2010
Return-Path: <general-return-1725-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1802 invoked from network); 1 Jul 2010 12:54:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jul 2010 12:54:46 -0000
Received: (qmail 76987 invoked by uid 500); 1 Jul 2010 12:54:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76775 invoked by uid 500); 1 Jul 2010 12:54:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76767 invoked by uid 99); 1 Jul 2010 12:54:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 12:54:43 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 12:54:35 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 3AE25B7E61
	for <general@hadoop.apache.org>; Thu,  1 Jul 2010 13:54:15 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id KKVvML1LB8bR for <general@hadoop.apache.org>;
	Thu,  1 Jul 2010 13:54:14 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 5C5F3B7E28
	for <general@hadoop.apache.org>; Thu,  1 Jul 2010 13:54:14 +0100 (BST)
MailScanner-NULL-Check: 1278593641.86013@tyA6HuJyGJCRH4OiESj/vQ
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o61Cs0wC005866
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Thu, 1 Jul 2010 13:54:01 +0100 (BST)
Message-ID: <4C2C8FE9.1040706@apache.org>
Date: Thu, 01 Jul 2010 13:54:01 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100228)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Hadoop Platform
References: <AANLkTine5opXB6KXLejtKYC7zyiRam6a2Qw7RwkPR8fo@mail.gmail.com>	<30B5A69F3E33FE4C83F4C832F5D6E2CF0254F734@CINMLVEM12.e2k.ad.ge.com>	<AANLkTilAe0jkMxwf1wg1u6TJr8sWUObgayKU5RJQtMZL@mail.gmail.com> <AANLkTinitR3cqiLnkxo2H-AQjRMELzvyLAQ12KbkhFng@mail.gmail.com>
In-Reply-To: <AANLkTinitR3cqiLnkxo2H-AQjRMELzvyLAQ12KbkhFng@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o61Cs0wC005866
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

Renato Marroquín Mogrovejo wrote:
> Thanks everybody,
> 
> I will definitely try to get in contact with OpenCirrus, and if not I can at
> least test with some amazon hw so I can prove the application concept. 

There's a lot to be said for bringing up 3 or 4 virtualbox or vmware VMs 
and playing with that locally, as it does make sure that you have 
distribution right. CPU performance on VMs is pretty good, it's only 
disk IO that (currently) suffers.

 >And
> the Whirr project is really interesting, I will give it a look specially
> because we really don't want to depend on a single service provider.
> Thanks again for the suggestions.

I've been doing some work on GUIs/tooling too; this is something that 
could really front end what whirr does, though it also needs a 
cross-infrastructure HA story for managing the persistent data the 
tooling needs too -user accounts, current set of machines, etc etc.

http://www.slideshare.net/steve_l/farming-hadoop-inthecloud
> 
> 
> Renato M.
> 
> 
> 2010/6/30 Jeff Hammerbacher <hammer@cloudera.com>
> 
>> If you want to use an open API and be cloud-provider agnostic, check out
>> Whirr: http://incubator.apache.org/projects/whirr.html.
>>
>> On Wed, Jun 30, 2010 at 10:52 AM, Yan, Weizhong (GE, Research)
>> <yan@ge.com>wrote:
>>
>>> Why don't you just simply try Amazon elastic MapReduce?
>>> http://aws.amazon.com/elasticmapreduce/
>>>
>>> -----Original Message-----
>>> From: Renato Marroquín Mogrovejo [mailto:renatoj.marroquin@gmail.com]
>>> Sent: Wednesday, June 30, 2010 11:47 AM
>>> To: general@hadoop.apache.org
>>> Subject: Hadoop Platform
>>>
>>> Hi everyone,
>>>
>>> I read a while ago that maybe IBM or Yahoo! were going to provide a
>> "cloud"
>>> for academic purposes. Does anybody know if those intentions
>> materialised?
>>> or if there any other ( free or not so expensive ) platform to test
>> Hadoop
>>> jobs??? I mean, obviously there would be several limitations on it, but I
>>> don't have access to a cluster and I have some Hadoop jobs I need to test
>>> them, so I can prove to my people the power of it (:
>>> Thanks in advance.
>>>
>>>
>>> Renato M.
>>>
> 


From general-return-1726-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 01 15:16:30 2010
Return-Path: <general-return-1726-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50193 invoked from network); 1 Jul 2010 15:16:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jul 2010 15:16:30 -0000
Received: (qmail 95489 invoked by uid 500); 1 Jul 2010 15:16:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95274 invoked by uid 500); 1 Jul 2010 15:16:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95266 invoked by uid 99); 1 Jul 2010 15:16:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 15:16:28 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of gmackey@cs.ucf.edu designates 132.170.216.154 as permitted sender)
Received: from [132.170.216.154] (HELO longwood.cs.ucf.edu) (132.170.216.154)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 15:16:20 +0000
Received: from longwood.cs.ucf.edu (localhost [127.0.0.1])
	by longwood.cs.ucf.edu (8.14.1/8.14.1) with ESMTP id o61FFveQ028767
	for <general@hadoop.apache.org>; Thu, 1 Jul 2010 11:15:58 -0400 (EDT)
Received: (from apache@localhost)
	by longwood.cs.ucf.edu (8.14.1/8.14.4/Submit) id o61FFv54028765
	for general@hadoop.apache.org; Thu, 1 Jul 2010 11:15:57 -0400 (EDT)
X-Authentication-Warning: longwood.cs.ucf.edu: apache set sender to gmackey@cs.ucf.edu using -f
Received: from proxyout.lanl.gov (proxyout.lanl.gov [192.12.184.2]) by
 mail.cs.ucf.edu (Horde Framework) with HTTP; Thu, 01 Jul 2010 11:15:57
 -0400
Message-ID: <20100701111557.87054g9doprm1kow@mail.cs.ucf.edu>
Date: Thu, 01 Jul 2010 11:15:57 -0400
From: Grant Mackey <gmackey@cs.ucf.edu>
To: general@hadoop.apache.org
Subject: Re: Chaining Map-Reduce
References: <699135.52032.qm@web46307.mail.sp1.yahoo.com>
In-Reply-To: <699135.52032.qm@web46307.mail.sp1.yahoo.com>
MIME-Version: 1.0
Content-Type: text/plain;
 charset=ISO-8859-1;
 DelSp="Yes";
 format="flowed"
Content-Disposition: inline
Content-Transfer-Encoding: 7bit
User-Agent: Internet Messaging Program (IMP) 4.3.3
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on gondor.cs.ucf.edu
X-Virus-Scanned: clamav-milter 0.96.1 at longwood
X-Virus-Status: Clean
X-Virus-Checked: Checked by ClamAV on apache.org
X-Old-Spam-Status: No, score=-1.4 required=6.0 tests=ALL_TRUSTED autolearn=failed
	version=3.2.5

Look at the cascade project

-->http://www.cascading.org/

  - Grant

Quoting abc xyz <fabc_xyz111@yahoo.com>:

> Hi everyone,
>
> I know that it is possible in Hadoop 0.19 and onwards to have any number of
> mappers, then a reducer, and then any number of mappers chained in a  
> single job.
> Is it possible to have more than one reducers in this chain? If yes, how??
>
> Thanks
>
>
>
>



Grant Mackey
UCF Research Assistant
Engineering III
Rm 238 Cubicle 1

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.


From general-return-1727-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 03:03:03 2010
Return-Path: <general-return-1727-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78453 invoked from network); 2 Jul 2010 03:03:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 03:03:02 -0000
Received: (qmail 9649 invoked by uid 500); 2 Jul 2010 03:03:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 9406 invoked by uid 500); 2 Jul 2010 03:02:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 9398 invoked by uid 99); 2 Jul 2010 03:02:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 03:02:58 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,HTML_OBFUSCATE_10_20,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates 209.85.216.176 as permitted sender)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 03:02:51 +0000
Received: by qyk12 with SMTP id 12so220450qyk.14
        for <general@hadoop.apache.org>; Thu, 01 Jul 2010 20:02:30 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=/+9UvD0g9tBHVZHqPdt8HwkS6Tn500dDVrptxJe0oX8=;
        b=F33MgoA4BICiTbFt77WOfCqHz967qjJLfbby93fmBJVPUYtqG2dnDvoi7k4cXJtlQd
         JYXXi+W+/PS6UKdch/y6sKu0kO37C6qVt+9GpDBkmwMb2osTEpivga/N1fdvwtXayp80
         f64TXY97YJJWNM2BOWFGOUZ1sXqi9B9kNsqq0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=HcNPL/uDjhYEP3WtxH6aGm7Micr1IDIUSHQGtObmmspfvVW0b+g80o/J+EtVF3zlLB
         sZGHlZnSjOYwufyxIy5q1mfswgbXa/5ukqjg2vPwWcjQrAgfo/8xda9ofV6w6ghssKIN
         TUTsw4Y2n235KyJm4wZIJME76ZqouIksnWYzQ=
MIME-Version: 1.0
Received: by 10.224.28.78 with SMTP id l14mr19895qac.31.1278039750591; Thu, 01 
	Jul 2010 20:02:30 -0700 (PDT)
Received: by 10.229.189.134 with HTTP; Thu, 1 Jul 2010 20:02:30 -0700 (PDT)
Date: Thu, 1 Jul 2010 20:02:30 -0700
Message-ID: <AANLkTin_yUUdo3B8OMK9Kglfv3-Yx91rwDP4ZqzGgY1n@mail.gmail.com>
Subject: problem starting cdh3b2 jobtracker
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175ce08c57c4af048a5ed11e
X-Virus-Checked: Checked by ClamAV on apache.org

--0015175ce08c57c4af048a5ed11e
Content-Type: text/plain; charset=ISO-8859-1

We installed cdh3b2 0.20.2+320 and saw some strange error in jobtracker log:

2010-07-02 01:49:31,977 INFO org.apache.hadoop.mapred.JobTracker: JobTracker
up at: 9001
2010-07-02 01:49:31,977 INFO org.apache.hadoop.mapred.JobTracker: JobTracker
webserver: 50030
2010-07-02 01:49:31,988 WARN org.apache.hadoop.mapred.JobTracker: Error
starting tracker: java.io.IOException: Cannot create toBeDeleted in
/data1/mapred/local
    at
org.apache.hadoop.util.MRAsyncDiskService.<init>(MRAsyncDiskService.java:85)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1688)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:199)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:191)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3765)

2010-07-02 01:49:32,990 INFO org.apache.hadoop.mapred.JobTracker: Scheduler
configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
2010-07-02 01:49:32,991 FATAL org.apache.hadoop.mapred.JobTracker:
java.net.BindException: Problem binding to
sjc1-hadoop0.sjc1.ciq.com/10.201.8.204:9001<http://sjc1-hadoop0.sjc1.carrieriq.com/10.201.8.204:9001>:
Address already in use
    at org.apache.hadoop.ipc.Server.bind(Server.java:198)
    at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:261)
    at org.apache.hadoop.ipc.Server.<init>(Server.java:1043)
    at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:492)
    at org.apache.hadoop.ipc.RPC.getServer(RPC.java:454)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1628)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:199)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:191)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3765)
Caused by: java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind(Native Method)
    at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
    at org.apache.hadoop.ipc.Server.bind(Server.java:196)
    ... 8 more

2010-07-02 01:49:32,992 INFO org.apache.hadoop.mapred.JobTracker:
SHUTDOWN_MSG:

But 9001 wasn't used:
[sjc1-hadoop0.sjc1:hadoop 25618]netstat -nta | grep 9001
[sjc1-hadoop0.sjc1:hadoop 25619]netstat -nta | grep 9000
tcp        0      0 10.201.8.204:9000           0.0.0.0:*
LISTEN
tcp        0      0 10.201.8.204:9000           10.201.8.214:4223
ESTABLISHED
tcp        0      0 10.201.8.204:9000           10.201.8.212:49074
ESTABLISHED
tcp        0      0 10.201.8.204:9000           10.201.8.206:11910
ESTABLISHED
tcp        0      0 10.201.8.204:9000           10.201.8.210:62611
ESTABLISHED
tcp        0      0 10.201.8.204:9000           10.201.8.213:1299
ESTABLISHED
tcp        0      0 10.201.8.204:9000           10.201.8.205:9756
ESTABLISHED
tcp        0      0 10.201.8.204:9000           10.201.8.207:59207
ESTABLISHED

Here is output from ifconfig:
bond0     Link encap:Ethernet  HWaddr 00:30:48:60:53:94
          inet addr:10.201.8.204  Bcast:10.201.8.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:351496605 errors:0 dropped:1015 overruns:0 frame:0
          TX packets:178144953 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:119420730164 (111.2 GiB)  TX bytes:120002123131 (111.7
GiB)

eth0      Link encap:Ethernet  HWaddr 00:30:48:60:53:94
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:351496605 errors:0 dropped:1015 overruns:0 frame:0
          TX packets:178144953 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:119420730164 (111.2 GiB)  TX bytes:120002123131 (111.7
GiB)
          Interrupt:161

eth1      Link encap:Ethernet  HWaddr 00:30:48:60:53:94
          UP BROADCAST SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
          Interrupt:169

Has anyone encountered similar issue ?

--0015175ce08c57c4af048a5ed11e--

From general-return-1728-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 06:15:09 2010
Return-Path: <general-return-1728-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23693 invoked from network); 2 Jul 2010 06:15:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 06:15:08 -0000
Received: (qmail 5769 invoked by uid 500); 2 Jul 2010 06:15:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5497 invoked by uid 500); 2 Jul 2010 06:15:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5484 invoked by uid 99); 2 Jul 2010 06:15:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 06:15:04 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eltonsky9404@gmail.com designates 209.85.216.169 as permitted sender)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 06:14:56 +0000
Received: by qyk32 with SMTP id 32so289238qyk.14
        for <general@hadoop.apache.org>; Thu, 01 Jul 2010 23:13:35 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=Z6EwRxpL+L8fXYpexHkXFSrePams1QsdTpKMMFtOtvg=;
        b=PhU6paSlxidd0cE889l06YdKJu1IzEF0hme+6RNAeKJgBEURz1nW3T+32iklS+afV/
         oKAGV2OM4XFBzD3We5Z/9OacD9O0valyKDiTAUSs3kzurjaB9i/t6bh2ivfcVzdlkMyA
         Yx0SGKXmN7P71wEGsm+mxczpuX9d+h9PRqxDg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=kZtKkDslC+Cp9+QM1J/GKBeVtdjm2cxy/3JeHnoOJqSyhY+uD9Njqh0rQ6zmov/uPK
         JE6P5LrCgXPjzqAFRga4VIAo/C3WN/rbeaAI36dNQAJI4hYoTyU/a/be2pqfdP523BAR
         9suV7ViodaKn0ufEJu6hPHcQwAj6Jd32NqVb4=
MIME-Version: 1.0
Received: by 10.224.87.6 with SMTP id u6mr96737qal.214.1278051215262; Thu, 01 
	Jul 2010 23:13:35 -0700 (PDT)
Received: by 10.224.89.18 with HTTP; Thu, 1 Jul 2010 23:13:35 -0700 (PDT)
Date: Fri, 2 Jul 2010 16:13:35 +1000
Message-ID: <AANLkTiljgFWtvH2dQHhB44-8mXQKlAhtOPp5N1K3tJ3S@mail.gmail.com>
Subject: Why single thread for HDFS?
From: elton sky <eltonsky9404@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00c09f99e549b0c5ec048a617cb5
X-Virus-Checked: Checked by ClamAV on apache.org

--00c09f99e549b0c5ec048a617cb5
Content-Type: text/plain; charset=ISO-8859-1

I guess this question was igored, so I just post it again.

>From my understanding, HDFS uses a single thread to do read and write.
Since a file is composed of many blocks and each block is stored as a file
in the underlying FS, we can do some parallelism on block base.
When read across multi-blocks, threads can be used to read all blocks. When
write, we can calculate the offset of each block and write to all of them
simultaneously.

Is this right?

--00c09f99e549b0c5ec048a617cb5--

From general-return-1729-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 07:18:46 2010
Return-Path: <general-return-1729-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49191 invoked from network); 2 Jul 2010 07:18:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 07:18:46 -0000
Received: (qmail 69819 invoked by uid 500); 2 Jul 2010 07:18:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69510 invoked by uid 500); 2 Jul 2010 07:18:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69497 invoked by uid 99); 2 Jul 2010 07:18:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 07:18:41 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [68.180.199.173] (HELO web46308.mail.sp1.yahoo.com) (68.180.199.173)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 02 Jul 2010 07:18:34 +0000
Received: (qmail 62799 invoked by uid 60001); 2 Jul 2010 07:17:08 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1278055028; bh=2pjE1hcZ5fN30DboLRlCkLuP4upt7/TyCFhn1b1OjLM=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=4PTdjqyGAIkMMGc64yMBcsgOA2E9/JCg/JXxFNLeN0s2bvxA//GW4Ico34pOpKmzNImqmJU+sdmAiGhBuolkAC4swAsrctycaID+nJxgCGIiIoTkrQMOOfYoY3JV69LrZE8S/k5yuZHl2lCMpJlt4slT3xDfUCFlnIbm9a9CoD4=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=B1DB2D4XXU7gncemxXnNy/Tghwuk8CGUVR22F92C25k+6oKB2UenMSC2nDAGZU25l4sAncrxj1Pau1IzPso7nkbMRA667gZEkgCrqiMMLcPfB9lzC/oT396urzLqXWelBmAvcwnWbhMtNUHrc8XPuLlg6/N4H0AOcgbIgPpwE2M=;
Message-ID: <351370.62305.qm@web46308.mail.sp1.yahoo.com>
X-YMail-OSG: KuE8xV4VM1lx9U2El9C.uow_CVZaa_4zmBVdh3XsJwKpGeb
 nEBLgcvREemc7IkmVQvfHrKabMGaGbvVyjQxcmgSEG.UQh64fVdb.JdbdykG
 aBOwVYnBvFrtL5jBfWHaxNWQaEbapkCeHTu3lPwaQtb2d1.nCMtnPm8.CuGO
 drHmEKwO02MNwUFmRtniRlV1UGTVNraayMogXygUHIT7S5JOEhq1a266WX_a
 yB12lpn.VhxDyKAH7AKcmvDbpCSHcDYiDDx2py4vdEqab6.knyku6CKHgrHx
 coZpPlTXWzPw7YSWcafEqCZ4xYhf0Y4Ck2SveK4aXTFwnHyEKPLBpdvJj1eQ
 h1GSQcRyuuBoLHA--
Received: from [188.74.76.201] by web46308.mail.sp1.yahoo.com via HTTP; Fri, 02 Jul 2010 00:17:08 PDT
X-Mailer: YahooMailRC/420.4 YahooMailWebService/0.8.104.274457
References: <699135.52032.qm@web46307.mail.sp1.yahoo.com> <20100701111557.87054g9doprm1kow@mail.cs.ucf.edu>
Date: Fri, 2 Jul 2010 00:17:08 -0700 (PDT)
From: abc xyz <fabc_xyz111@yahoo.com>
Subject: Re: Chaining Map-Reduce
To: general@hadoop.apache.org
In-Reply-To: <20100701111557.87054g9doprm1kow@mail.cs.ucf.edu>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1835254229-1278055028=:62305"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-1835254229-1278055028=:62305
Content-Type: text/plain; charset=us-ascii



Thanks Grant, but i want to do it without using any external library. Is there 
some way of it to have more than one reducers in a single job? 




________________________________
From: Grant Mackey <gmackey@cs.ucf.edu>
To: general@hadoop.apache.org
Sent: Thu, July 1, 2010 4:15:57 PM
Subject: Re: Chaining Map-Reduce

Look at the cascade project

-->http://www.cascading.org/

  - Grant

Quoting abc xyz <fabc_xyz111@yahoo.com>:

> Hi everyone,
>
> I know that it is possible in Hadoop 0.19 and onwards to have any number of
> mappers, then a reducer, and then any number of mappers chained in a  
> single job.
> Is it possible to have more than one reducers in this chain? If yes, how??
>
> Thanks
>
>
>
>



Grant Mackey
UCF Research Assistant
Engineering III
Rm 238 Cubicle 1

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.


      
--0-1835254229-1278055028=:62305--

From general-return-1730-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 07:22:12 2010
Return-Path: <general-return-1730-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49648 invoked from network); 2 Jul 2010 07:22:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 07:22:12 -0000
Received: (qmail 74872 invoked by uid 500); 2 Jul 2010 07:22:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74691 invoked by uid 500); 2 Jul 2010 07:22:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74683 invoked by uid 99); 2 Jul 2010 07:22:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 07:22:08 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [68.180.199.176] (HELO web46311.mail.sp1.yahoo.com) (68.180.199.176)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 02 Jul 2010 07:21:59 +0000
Received: (qmail 62940 invoked by uid 60001); 2 Jul 2010 07:20:34 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1278055234; bh=zXA7QJdIOFuXk1Ow0e3ZFz4PjUGKcYsvhGCd4i77ZdA=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=oRv/ujMYpTvpdDb1g9ZkY0ib30/j4FsvM1Z1ZLz50qc6Y1hUMtUaMci646QTpPisHpWG6VvQ/KXlRzGMZaSrxOXw0kWtfGejAmSwJbzaVdixEPamdLZGUcOkVwTR01rBJv2/qZE+bXT61/bf8JWCzDrH32Am3EigCsvJUbDo1ew=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=1pjfuuhFToTZwyJjnL+7apTDonDcoc7ZMfqDgjUOTzM/xZJmWMb8An34trBd/blm8hM7Hc+py9044TeBOP/6P4u6Wv+t2k495vopLp1d+M965z8A979ElIpvFeWkk8PWXH62+cFrmVGXxv3/saNcp5JjvObhZLkKJhOzkyLrZUw=;
Message-ID: <447928.62922.qm@web46311.mail.sp1.yahoo.com>
X-YMail-OSG: hGsG6pMVM1kjr..fAQ5TiO4kFd1VM1WkU0270zedQvW6QLE
 ua9yazIqMxIo0xtHUcW7WY8IRlb60s5TwTbOVmUnCC3kd0I3PBC5pBio9vBI
 eyUQkKVJexVx5CDSOuXtM73ybgO9gKCPBu5L_k.hAVyM.ZVwfpv8Rgrs4PND
 CYs8.yJLgCbBvHtpxIizTjnRrfF4nNxySputPMCHpzY3Zc8u7TTRLo7jWeyC
 v4S3fFx8vPaIfDZAYooi58PXX7PGP55JzCfraJETe8g6A2Q3hMyoDOfAabqb
 _7ggrlw82875cCrpqT80QaFTvEZT5D7JRAQ8-
Received: from [188.74.76.201] by web46311.mail.sp1.yahoo.com via HTTP; Fri, 02 Jul 2010 00:20:33 PDT
X-Mailer: YahooMailRC/420.4 YahooMailWebService/0.8.104.274457
Date: Fri, 2 Jul 2010 00:20:33 -0700 (PDT)
From: abc xyz <fabc_xyz111@yahoo.com>
Subject: Displaying Map output in MapReduce
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1345269571-1278055233=:62922"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-1345269571-1278055233=:62922
Content-Type: text/plain; charset=us-ascii

Hi folks,

I want to just display the output of my mapper on console or want to log it in 
some file. System.out.println and Log4j's logger are not displaying the output. 
How can I see the intermediate results?

Cheers



      
--0-1345269571-1278055233=:62922--

From general-return-1731-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 07:33:11 2010
Return-Path: <general-return-1731-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53047 invoked from network); 2 Jul 2010 07:33:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 07:33:11 -0000
Received: (qmail 90865 invoked by uid 500); 2 Jul 2010 07:33:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89298 invoked by uid 500); 2 Jul 2010 07:33:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88585 invoked by uid 99); 2 Jul 2010 07:33:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 07:33:06 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cabiwan@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 07:32:58 +0000
Received: by fxm16 with SMTP id 16so2765232fxm.35
        for <general@hadoop.apache.org>; Fri, 02 Jul 2010 00:32:38 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=5qoEL1h+v7isT8MJUL7lEqUFwj/s3Fc/F+cfhkPSpyI=;
        b=uQHDLYdnlmbh39Qbo6ZQG8B+W9hLyeSx79wKeuqJ0BRkA7JG6OyjNhCcsmOJGIthuq
         2hE3Nf38sh+cDWhzxkJvXeRxJJA3L/QPH2tIYWU+1Rvhq80ITyNIqNhblpz7OpwRWjy7
         P16pJon3MQNF+EFW5GUFSP+4WF8vnEu0EmEaU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=hcBzKnosaT/i6K0SSEwY5/Nc7rC2JDyGa7ecvSemyEOU7PCaOncA1tuBC5ueRUDAFg
         iH0rnhaLDtZo2dj7BAUPSIKmIBeyL+zizwA2wqlKBEMW1jSiFXx47qSdIEqjHorYHOiH
         SH+R2pnOK6H/qf4pLsuUegERkQ81dWdaNfPZE=
MIME-Version: 1.0
Received: by 10.103.173.6 with SMTP id a6mr84487mup.116.1278055957366; Fri, 02 
	Jul 2010 00:32:37 -0700 (PDT)
Received: by 10.103.5.16 with HTTP; Fri, 2 Jul 2010 00:32:37 -0700 (PDT)
In-Reply-To: <447928.62922.qm@web46311.mail.sp1.yahoo.com>
References: <447928.62922.qm@web46311.mail.sp1.yahoo.com>
Date: Fri, 2 Jul 2010 09:32:37 +0200
Message-ID: <AANLkTilyIwJaPJDsqGdr860BF7iWQHUUuXz1a3RE7v7z@mail.gmail.com>
Subject: Re: Displaying Map output in MapReduce
From: Alberto Luengo Cabanillas <cabiwan@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163642741b5788b6048a62975b
X-Virus-Checked: Checked by ClamAV on apache.org

--00163642741b5788b6048a62975b
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi! Supposing that you=B4re using a virtualized environment, you should be
able to get to JobTracker Web Admin Interface from http://<your_virtualized=
_
machine_ip>:50070/logs/userlogs. In case you=B4re running Hadoop from insid=
e a
cluster frontend, use this machine=B4s IP. Inside this directory, it will
exist another one called "userlogs" where you will see all map and reduce
tasks launch by the JobTracker and the TaskTrackers. These tasks will look
like 'job_[m,r]_0_xxxxxx' (or similar). Inside of every one of them there
will be three subdirectories (stdout, stdlog and stderr) which will contain
all your debugging messages.
Inside my MapReduce development programs I use the Java *LOG* class in a
sentence like this:

private static final Log LOG =3D
LogFactory.getLog(<your_class_name>.class.getName());
LOG.info("***********INSIDE MY REDUCER SETUP**********");

Besides these messages, a normal "System.out.println(...)" will also appear
in the above directories.

Hope it helps!

2010/7/2 abc xyz <fabc_xyz111@yahoo.com>

> Hi folks,
>
> I want to just display the output of my mapper on console or want to log =
it
> in
> some file. System.out.println and Log4j's logger are not displaying the
> output.
> How can I see the intermediate results?
>
> Cheers
>
>
>






--=20
Alberto

--00163642741b5788b6048a62975b--

From general-return-1732-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 07:39:54 2010
Return-Path: <general-return-1732-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55419 invoked from network); 2 Jul 2010 07:39:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 07:39:53 -0000
Received: (qmail 97591 invoked by uid 500); 2 Jul 2010 07:39:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97365 invoked by uid 500); 2 Jul 2010 07:39:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97357 invoked by uid 99); 2 Jul 2010 07:39:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 07:39:49 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 07:39:41 +0000
Received: from EGL-EX07CAS02.ds.corp.yahoo.com (egl-ex07cas02.eglbp.corp.yahoo.com [203.83.248.209])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o627bYUH091596
	for <general@hadoop.apache.org>; Fri, 2 Jul 2010 00:37:34 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:in-reply-to:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=w4hkYqRrz2WWaStttGndaSHpxZlPu64uFXhlKAVUAs83IqTMQbybLTNwMOffPgPd
Received: from EGL-EX07VS02.ds.corp.yahoo.com ([203.83.248.206]) by
 EGL-EX07CAS02.ds.corp.yahoo.com ([203.83.248.216]) with mapi; Fri, 2 Jul 2010
 13:07:33 +0530
From: Venkatesh S <svenkat@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 2 Jul 2010 13:07:01 +0530
Subject: Re: Chaining Map-Reduce
Thread-Topic: Chaining Map-Reduce
Thread-Index: AcsZtuLbL6HZO4+TS9Kdd6/YMnG1OgAAnjwl
Message-ID: <C85394F5.3AB34%svenkat@yahoo-inc.com>
In-Reply-To: <351370.62305.qm@web46308.mail.sp1.yahoo.com>
Accept-Language: en, en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en, en-US
Content-Type: multipart/alternative;
	boundary="_000_C85394F53AB34svenkatyahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C85394F53AB34svenkatyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

You could use ChainMapper and ChainReducer API in hadoop.

--_000_C85394F53AB34svenkatyahooinccom_--

From general-return-1733-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 07:45:42 2010
Return-Path: <general-return-1733-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56770 invoked from network); 2 Jul 2010 07:45:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 07:45:42 -0000
Received: (qmail 11093 invoked by uid 500); 2 Jul 2010 07:45:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 10817 invoked by uid 500); 2 Jul 2010 07:45:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 10809 invoked by uid 99); 2 Jul 2010 07:45:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 07:45:38 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of fariha.atta.pk@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 07:45:32 +0000
Received: by fxm16 with SMTP id 16so2776615fxm.35
        for <general@hadoop.apache.org>; Fri, 02 Jul 2010 00:45:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=bH0kHhmqG0mHonO1aRfyxZDVS9jvC1R5wtD1zOpJiVk=;
        b=QrbiYOjFPPr3YvnCiq8W3ETBJ6XZ5eKeV2a/PJXLSPCwACNYeRQYWI3da3aAIIdAbT
         zWxHiOxuXpigK+JKu8tNLoTyG4SMJ3EGQVemAQU+4zrN/D7u8vvdz0W51BtABs1FO00i
         qSlCMQ5LDHYdxrWiGzFV6lmKzY2EjgZgXP8/0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=RC+uS47QB4BSEpskYdO9uxdQakLfeachFxfANrL9Jhqrkwwp45Fo6Xs1hjg4zMOKJ6
         QIqjbY8gSfljQXxH4YwvGDHe9aRe2hOwHIwRvVATJrYTPQzAf7Bf8yW3kYkcqrEswFuj
         d2gy5wPa7gL9Nzd3W8fBsYfCFTQJDOi46QBT0=
MIME-Version: 1.0
Received: by 10.223.103.80 with SMTP id j16mr311768fao.34.1278056712098; Fri, 
	02 Jul 2010 00:45:12 -0700 (PDT)
Received: by 10.223.19.5 with HTTP; Fri, 2 Jul 2010 00:45:12 -0700 (PDT)
In-Reply-To: <C85394F5.3AB34%svenkat@yahoo-inc.com>
References: <351370.62305.qm@web46308.mail.sp1.yahoo.com>
	<C85394F5.3AB34%svenkat@yahoo-inc.com>
Date: Fri, 2 Jul 2010 08:45:12 +0100
Message-ID: <AANLkTinjmTNRKYbSw3ULo5p64Ag4aip7yfz631OApPqr@mail.gmail.com>
Subject: Re: Chaining Map-Reduce
From: Fariha Atta <fariha.atta.pk@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163684820253d5b0048a62c48a
X-Virus-Checked: Checked by ClamAV on apache.org

--00163684820253d5b0048a62c48a
Content-Type: text/plain; charset=ISO-8859-1

But it provides chaining like Map+ | Reduce | Map* i.e. we can use only one
reducer.


On Fri, Jul 2, 2010 at 8:37 AM, Venkatesh S <svenkat@yahoo-inc.com> wrote:

> You could use ChainMapper and ChainReducer API in hadoop.
>

--00163684820253d5b0048a62c48a--

From general-return-1734-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 07:57:25 2010
Return-Path: <general-return-1734-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59005 invoked from network); 2 Jul 2010 07:57:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 07:57:25 -0000
Received: (qmail 24598 invoked by uid 500); 2 Jul 2010 07:57:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24336 invoked by uid 500); 2 Jul 2010 07:57:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24327 invoked by uid 99); 2 Jul 2010 07:57:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 07:57:21 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yhemanth@gmail.com designates 209.85.212.176 as permitted sender)
Received: from [209.85.212.176] (HELO mail-px0-f176.google.com) (209.85.212.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 07:57:14 +0000
Received: by pxi11 with SMTP id 11so2444594pxi.35
        for <general@hadoop.apache.org>; Fri, 02 Jul 2010 00:55:52 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=oj418F9gDqL3oCY4Doyo5hbAxaSvGP9ZLbyoxEC47bM=;
        b=KaoUOJ9pjYjoAi4jmVuxe0OJti3QTSXckp1+8xiKxm73cP4+L8BPQLm95xlrz4jC4p
         +o/9+kU0z21ocWiIoj5KaFyyWYqQtn+6Zx2ai/URrGJKECjcm6V8F2sCxB8KQWo8tSEN
         4C/JmSAHQpw3o0RPrXeFAPmlDE+REvLEZDZ2I=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=didIBwgrpLJGK8CLeBxZYTtEYTxZObi0MuDl8AtHnQ4K2ofqhUZimIpvEUSTihxCqr
         tF9EC+Z+8bRgTgyJwtBmgtWytfXIG9cdlEWAP6eosNxYJ1Jg5fc5riXDKF8I2XYEkIFK
         KtC21cc5U+IpX40COFgFwTi9kdzrr4C38Inj8=
MIME-Version: 1.0
Received: by 10.142.163.15 with SMTP id l15mr519976wfe.97.1278057352391; Fri, 
	02 Jul 2010 00:55:52 -0700 (PDT)
Received: by 10.142.188.19 with HTTP; Fri, 2 Jul 2010 00:55:52 -0700 (PDT)
In-Reply-To: <AANLkTiljgFWtvH2dQHhB44-8mXQKlAhtOPp5N1K3tJ3S@mail.gmail.com>
References: <AANLkTiljgFWtvH2dQHhB44-8mXQKlAhtOPp5N1K3tJ3S@mail.gmail.com>
Date: Fri, 2 Jul 2010 13:25:52 +0530
Message-ID: <AANLkTikCO_TnPI-xBt_QQljWkTBWJ4lq3nGhHPSEjYNr@mail.gmail.com>
Subject: Re: Why single thread for HDFS?
From: Hemanth Yamijala <yhemanth@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

Can you please post this on hdfs-dev@hadoop.apache.org ? I suspect the
most qualified people to answer this question would all be on that
list.

Hemanth

On Fri, Jul 2, 2010 at 11:43 AM, elton sky <eltonsky9404@gmail.com> wrote:
> I guess this question was igored, so I just post it again.
>
> From my understanding, HDFS uses a single thread to do read and write.
> Since a file is composed of many blocks and each block is stored as a file
> in the underlying FS, we can do some parallelism on block base.
> When read across multi-blocks, threads can be used to read all blocks. When
> write, we can calculate the offset of each block and write to all of them
> simultaneously.
>
> Is this right?
>

From general-return-1735-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 08:15:13 2010
Return-Path: <general-return-1735-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66413 invoked from network); 2 Jul 2010 08:15:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 08:15:13 -0000
Received: (qmail 43239 invoked by uid 500); 2 Jul 2010 08:15:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42922 invoked by uid 500); 2 Jul 2010 08:15:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42913 invoked by uid 99); 2 Jul 2010 08:15:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 08:15:08 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [68.180.199.177] (HELO web46312.mail.sp1.yahoo.com) (68.180.199.177)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 02 Jul 2010 08:15:00 +0000
Received: (qmail 7642 invoked by uid 60001); 2 Jul 2010 08:14:38 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1278058478; bh=NB3PSDOzCdgNQwZbEDfPW4uUg6PmfFkF5GgVhR2pPt4=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=nDpb3KYWj+pmEOvaaadj4MnA7ZFO8h5kRrJuNGfs9TQwZmldjFRPWRb2uNS+QSWv59nqjqFudBJin8zK1jMl2UmlpsMs7WsWhgWueJ0BtqtkAC30hmC/hMaVcWiwFqRsfVDmV731cRM63zdg8NjGu1Ev+vPQpRWdDOkRyh5tGmM=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=SglJeYrGjfhlF6P2Nt+9byLjdK2aUyMGbIoiCvMq94N1d+NEOJCM61jw4XROq5BT+WjL4xD07FWziA3POZ/of1mbmNBOYXxOGzSz5OqfElZYaOfXqiLU9tVL8KzePUyYVuaT3VqNgQZb9sC1uAnkeEUgHlboVNvJmbVfagk5zO8=;
Message-ID: <790746.6243.qm@web46312.mail.sp1.yahoo.com>
X-YMail-OSG: uZBA7EwVM1lsWjXav57mxyCp2iX5uiEe7cEwcjUFw9Qt20V
 7WJWuojYoDZ1Uf9P9yAi2sq5566YC5wFMgo8zxqTHVe6CCwHT.vX8rwPok9t
 QHSBR_GYkiu3Sy7tRqUMrKq8B8rFuA5EHOvl1ZHfYwKsr4Q6o1KFaXdkVNPH
 HVZNLZ863pMMxhW4KmSWMg.C0MiwwclTKA0H5ampSb14dsCbevHKL_.AvVWM
 hVOzgoLNXYIu3c6RTMteMRWYjHoL8AQjAj_O1si4sRkMhTRvPG8WXJ.u9sy2
 QbS9Y68izsDOcPpkL9OLeL1hsb.sY1JjZgpk-
Received: from [129.215.149.98] by web46312.mail.sp1.yahoo.com via HTTP; Fri, 02 Jul 2010 01:14:38 PDT
X-Mailer: YahooMailRC/420.4 YahooMailWebService/0.8.104.274457
References: <351370.62305.qm@web46308.mail.sp1.yahoo.com> <C85394F5.3AB34%svenkat@yahoo-inc.com> <AANLkTinjmTNRKYbSw3ULo5p64Ag4aip7yfz631OApPqr@mail.gmail.com>
Date: Fri, 2 Jul 2010 01:14:38 -0700 (PDT)
From: abc xyz <fabc_xyz111@yahoo.com>
Subject: Re: Chaining Map-Reduce
To: general@hadoop.apache.org
In-Reply-To: <AANLkTinjmTNRKYbSw3ULo5p64Ag4aip7yfz631OApPqr@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-445714589-1278058478=:6243"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-445714589-1278058478=:6243
Content-Type: text/plain; charset=us-ascii

one option can be to use multiple chained jobs using JobControl object. 
But can anyone tell whether in this case the output of first job be written to 
the disk? Actually I want to minimize access to the disk.

Cheers



________________________________
From: Fariha Atta <fariha.atta.pk@gmail.com>
To: general@hadoop.apache.org
Sent: Fri, July 2, 2010 8:45:12 AM
Subject: Re: Chaining Map-Reduce

But it provides chaining like Map+ | Reduce | Map* i.e. we can use only one
reducer.


On Fri, Jul 2, 2010 at 8:37 AM, Venkatesh S <svenkat@yahoo-inc.com> wrote:

> You could use ChainMapper and ChainReducer API in hadoop.
>



      
--0-445714589-1278058478=:6243--

From general-return-1736-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 11:47:52 2010
Return-Path: <general-return-1736-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 20422 invoked from network); 2 Jul 2010 11:47:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 11:47:52 -0000
Received: (qmail 71495 invoked by uid 500); 2 Jul 2010 11:47:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71068 invoked by uid 500); 2 Jul 2010 11:47:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71060 invoked by uid 99); 2 Jul 2010 11:47:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 11:47:47 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=RCVD_ILLEGAL_IP,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [192.5.164.99] (HELO camv02-relay2.casc.gd-ais.com) (192.5.164.99)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 11:47:39 +0000
Received: from ([10.73.100.22])
	by camv02-relay2.casc.gd-ais.com with SMTP  id 5203374.38952284;
	Fri, 02 Jul 2010 04:47:14 -0700
Received: from eadc01-cahprd01.ad.gd-ais.com ([10.120.80.11]) by camv02-fes01.ad.gd-ais.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Fri, 2 Jul 2010 04:47:14 -0700
Received: from EADC01-MABPRD01.ad.gd-ais.com ([169.254.1.214]) by
 eadc01-cahprd01.ad.gd-ais.com ([10.120.80.11]) with mapi; Fri, 2 Jul 2010
 06:47:12 -0500
From: "Kilbride, James P." <James.Kilbride@gd-ais.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 2 Jul 2010 06:47:08 -0500
Subject: HBASE/HADOOP Examples
Thread-Topic: HBASE/HADOOP Examples
Thread-Index: AcsZ3E1BbdniIUrQRCK6gI8frlWefQ==
Message-ID: <035C7FA04C0BF64B8143C76523E7ED742FCFEEA0A5@EADC01-MABPRD01.ad.gd-ais.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-cr-puzzleid: {C6F21294-460D-4787-91F0-33C25FE572A6}
x-cr-hashedpuzzle: fSI= B9Gj CoLf DEh6 Er2i F/+x GVTk Hqa+ H81C I7DN JEtY
 JIxi KuF3 Lequ LwZo
 MDYX;1;ZwBlAG4AZQByAGEAbABAAGgAYQBkAG8AbwBwAC4AYQBwAGEAYwBoAGUALgBvAHIAZwA=;Sosha1_v1;7;{C6F21294-460D-4787-91F0-33C25FE572A6};agBhAG0AZQBzAC4AawBpAGwAYgByAGkAZABlAEAAZwBkAC0AYQBpAHMALgBjAG8AbQA=;Fri,
 02 Jul 2010 11:47:08
 GMT;SABCAEEAUwBFAC8ASABBAEQATwBPAFAAIABFAHgAYQBtAHAAbABlAHMA
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginalArrivalTime: 02 Jul 2010 11:47:14.0080 (UTC) FILETIME=[504B4200:01CB19DC]
X-Virus-Checked: Checked by ClamAV on apache.org

I've found examples using the older mapred interface but not the newer mapr=
educe interface. I want to write a mapper that is configured to only pull o=
ut specific rows(which are the mapper's keys) and a specific column's value=
(which is the mapper's value).


Is there any examples of something like this available?

James Kilbride


From general-return-1737-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 13:27:19 2010
Return-Path: <general-return-1737-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50588 invoked from network); 2 Jul 2010 13:27:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 13:27:19 -0000
Received: (qmail 94835 invoked by uid 500); 2 Jul 2010 13:27:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94495 invoked by uid 500); 2 Jul 2010 13:27:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94487 invoked by uid 99); 2 Jul 2010 13:27:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 13:27:14 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of msegel@navteq.com designates 204.120.70.37 as permitted sender)
Received: from [204.120.70.37] (HELO xmailfargo.navteq.com) (204.120.70.37)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 13:27:06 +0000
Received: from fargo-spamfw.navteq.com (navteq-spamfw.navteq.com [10.6.10.131])
	by xmailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id o62D6Rb4004381
	for <general@hadoop.apache.org>; Fri, 2 Jul 2010 08:06:27 -0500
Received: from imailfargo.navteq.com (localhost [127.0.0.1])
	by fargo-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id 471E04048C1B
	for <general@hadoop.apache.org>; Fri,  2 Jul 2010 08:26:23 -0500 (CDT)
Received: from imailfargo.navteq.com (imailfargo.navteq.com [10.6.10.130]) by fargo-spamfw.navteq.com with ESMTP id Zfb41WqyQKH8ScLX for <general@hadoop.apache.org>; Fri, 02 Jul 2010 08:26:23 -0500 (CDT)
Received: from hq-ex-ht01.ad.navteq.com ([10.8.222.51])
	by imailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id o62DQMr4025256
	for <general@hadoop.apache.org>; Fri, 2 Jul 2010 08:26:23 -0500
Received: from hq-ex-mb02.ad.navteq.com ([fe80::880d:318e:e860:a8db]) by
 hq-ex-ht01.ad.navteq.com ([fe80::55a6:5b94:5c00:fe39%13]) with mapi; Fri, 2
 Jul 2010 08:26:22 -0500
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 2 Jul 2010 08:26:21 -0500
Subject: RE: Why single thread for HDFS?
Thread-Topic: Why single thread for HDFS?
Thread-Index: AcsZvDbvoaWZhLY8R8mjH+zHQq9A5AAKf4wA
Message-ID: <ABC24175AFD3BE4DA15F4CD375ED413D06084AF156@hq-ex-mb02.ad.navteq.com>
References: <AANLkTiljgFWtvH2dQHhB44-8mXQKlAhtOPp5N1K3tJ3S@mail.gmail.com>
 <AANLkTikCO_TnPI-xBt_QQljWkTBWJ4lq3nGhHPSEjYNr@mail.gmail.com>
In-Reply-To: <AANLkTikCO_TnPI-xBt_QQljWkTBWJ4lq3nGhHPSEjYNr@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1
X-Virus-Checked: Checked by ClamAV on apache.org

Actually they also listen here and this is a basic question...

I'm not an expert, but how does having multiple threads really help this =
problem?

I'm assuming you're talking about a map/reduce job and not some specific =
client code which is being run on a client outside of the cloud/cluster..=
=2E.

I wasn't aware that you could easily synchronize threads running on diffe=
rent JVMs. ;-)

Your parallelism comes from multiple tasks running on different nodes wit=
hin the cloud. By default you get one map/reduce job per block. You can w=
rite your own splitter to increase this and then get more parallelism.=20

HTH

-Mike


-----Original Message-----
From: Hemanth Yamijala [mailto:yhemanth@gmail.com]=20
Sent: Friday, July 02, 2010 2:56 AM
To: general@hadoop.apache.org
Subject: Re: Why single thread for HDFS?

Hi,

Can you please post this on hdfs-dev@hadoop.apache.org ? I suspect the
most qualified people to answer this question would all be on that
list.

Hemanth

On Fri, Jul 2, 2010 at 11:43 AM, elton sky <eltonsky9404@gmail.com> wrote=
:
> I guess this question was igored, so I just post it again.
>
> From my understanding, HDFS uses a single thread to do read and write.
> Since a file is composed of many blocks and each block is stored as a f=
ile
> in the underlying FS, we can do some parallelism on block base.
> When read across multi-blocks, threads can be used to read all blocks. =
When
> write, we can calculate the offset of each block and write to all of th=
em
> simultaneously.
>
> Is this right?
>


The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.

From general-return-1738-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 15:25:52 2010
Return-Path: <general-return-1738-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78623 invoked from network); 2 Jul 2010 15:25:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 15:25:52 -0000
Received: (qmail 21917 invoked by uid 500); 2 Jul 2010 15:25:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21782 invoked by uid 500); 2 Jul 2010 15:25:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21769 invoked by uid 99); 2 Jul 2010 15:25:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 15:25:50 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jaybooth@gmail.com designates 209.85.210.48 as permitted sender)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 15:25:43 +0000
Received: by pzk35 with SMTP id 35so1130850pzk.35
        for <general@hadoop.apache.org>; Fri, 02 Jul 2010 08:24:32 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=2o/LEMnqbAzZW5HwfmRtwgVZ6sSvaErPSk/OYEj60k8=;
        b=rTj50XAvbnUQOCd3PJB4qSP/6UpwJMwebbmx8Ar5Qz4gefCp/ZBw/Qw4TJYRbaUHE9
         oqx7nIaFqkyyNl8W/hayz/WghVUHRQlWitl5AM4qrQ6EHqBg575a56Lu7+TlB88XHGEh
         JiyALxng3NizC53CMjDSlm5PKFl3/Vz6KPdjY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=sI/P+6Sd1d9epQ1+0QbpRVbSRwuWtJ3ZwE0p4w2IQVCxfNwgxGP/WvolOxakn+RQ3c
         CteSKI2+vKqMbZNDsYBDmv3+tgkpf5ybbBbAnEhK3DFdB5YQnGz9ImLRmbleM3OpgDdO
         +AFdUxBcyIl+2sh9DiWtsMityPaZeTH/yvmsw=
MIME-Version: 1.0
Received: by 10.142.177.21 with SMTP id z21mr1298027wfe.65.1278084272255; Fri, 
	02 Jul 2010 08:24:32 -0700 (PDT)
Received: by 10.142.188.1 with HTTP; Fri, 2 Jul 2010 08:24:32 -0700 (PDT)
In-Reply-To: <ABC24175AFD3BE4DA15F4CD375ED413D06084AF156@hq-ex-mb02.ad.navteq.com>
References: <AANLkTiljgFWtvH2dQHhB44-8mXQKlAhtOPp5N1K3tJ3S@mail.gmail.com>
	<AANLkTikCO_TnPI-xBt_QQljWkTBWJ4lq3nGhHPSEjYNr@mail.gmail.com>
	<ABC24175AFD3BE4DA15F4CD375ED413D06084AF156@hq-ex-mb02.ad.navteq.com>
Date: Fri, 2 Jul 2010 11:24:32 -0400
Message-ID: <AANLkTilk2wfZ10leX0iEeNwJozMDdfHNiQ09rEAnbPDd@mail.gmail.com>
Subject: Re: Why single thread for HDFS?
From: Jay Booth <jaybooth@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Yeah, a good way to think of it is that parallelism is achieved at the
application level.

On the input side, you can process multiple files in parallel or one
file in parallel by logically splitting and opening multiple readers
of the same file at multiple points.  Each of these readers is single
threaded, because, well, you're returning a stream of bytes in order.
It's inherently serial.

On the reduce side, multiple reduces run, writing to multiple files in
the same directory.  Again, you can't really write to a single file in
parallel effectively -- you can't write byte 26 before byte 25,
because the file's not that long yet.

Theoretically, maybe you could have all reduces write to the same file
by allocating some amount of space ahead of time and writing to the
blocks in parallel - in practice, you very rarely know how big your
output is going to be before it's produced, so this doesn't really
work.  Multiple files in the same directory achieves the same goal
much more elegantly, without exposing a bunch of internal details of
the filesystem to user space.

Does that make sense?



On Fri, Jul 2, 2010 at 9:26 AM, Segel, Mike <msegel@navteq.com> wrote:
> Actually they also listen here and this is a basic question...
>
> I'm not an expert, but how does having multiple threads really help this =
problem?
>
> I'm assuming you're talking about a map/reduce job and not some specific =
client code which is being run on a client outside of the cloud/cluster....
>
> I wasn't aware that you could easily synchronize threads running on diffe=
rent JVMs. ;-)
>
> Your parallelism comes from multiple tasks running on different nodes wit=
hin the cloud. By default you get one map/reduce job per block. You can wri=
te your own splitter to increase this and then get more parallelism.
>
> HTH
>
> -Mike
>
>
> -----Original Message-----
> From: Hemanth Yamijala [mailto:yhemanth@gmail.com]
> Sent: Friday, July 02, 2010 2:56 AM
> To: general@hadoop.apache.org
> Subject: Re: Why single thread for HDFS?
>
> Hi,
>
> Can you please post this on hdfs-dev@hadoop.apache.org ? I suspect the
> most qualified people to answer this question would all be on that
> list.
>
> Hemanth
>
> On Fri, Jul 2, 2010 at 11:43 AM, elton sky <eltonsky9404@gmail.com> wrote=
:
>> I guess this question was igored, so I just post it again.
>>
>> From my understanding, HDFS uses a single thread to do read and write.
>> Since a file is composed of many blocks and each block is stored as a fi=
le
>> in the underlying FS, we can do some parallelism on block base.
>> When read across multi-blocks, threads can be used to read all blocks. W=
hen
>> write, we can calculate the offset of each block and write to all of the=
m
>> simultaneously.
>>
>> Is this right?
>>
>
>
> The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above. =A0If you are =
not the intended recipient, you are hereby notified that any dissemination,=
 distribution, or copying of this communication, or any of its contents, is=
 strictly prohibited. =A0If you have received this communication in error, =
please notify the sender and delete/destroy the original message and any co=
py of it from your computer or paper files.
>

From general-return-1739-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 17:11:07 2010
Return-Path: <general-return-1739-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 7926 invoked from network); 2 Jul 2010 17:11:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 17:11:07 -0000
Received: (qmail 28464 invoked by uid 500); 2 Jul 2010 17:11:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28366 invoked by uid 500); 2 Jul 2010 17:11:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28358 invoked by uid 99); 2 Jul 2010 17:11:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 17:11:05 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates 209.85.216.169 as permitted sender)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 17:10:57 +0000
Received: by qyk32 with SMTP id 32so535709qyk.14
        for <general@hadoop.apache.org>; Fri, 02 Jul 2010 10:10:37 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=rZg116dBf/qhu2o7mBgUxYZLd24GMBiP+M1dN4xFIbc=;
        b=qpmfYGpmKVQf+C/ltLetFmJCt1dcp6N6AycS3k0R3d15n9WHlN2m/WfwE6Jb/OtN16
         ou89aPprB3hU9noOv5od4tKjS0mszxpQo23nkEeUyaNw/WMMid78q9CrnPJhJQ+kzCQ3
         zzTD01+IveE6isgsH5BuHGFnrvvPIJFDKPt6I=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=aqVU7FhAezYxnh3KxN9trLWHBaYn3mRcLLXOJwKq76w+M0UM2JOoJ7TsYK7fEwYf1U
         NnrfS5eMDDZkHwjvVXR/tLs8IHfPWv14y8o/MIE4CqYIJhJgdvDNXS55sXoW8juH9EE/
         1RG8EFBZQzbBueOUCgwGYLNgJANTXU/MkBQ0U=
MIME-Version: 1.0
Received: by 10.224.28.78 with SMTP id l14mr611889qac.31.1278090636624; Fri, 
	02 Jul 2010 10:10:36 -0700 (PDT)
Received: by 10.229.189.134 with HTTP; Fri, 2 Jul 2010 10:10:36 -0700 (PDT)
In-Reply-To: <AANLkTilyIwJaPJDsqGdr860BF7iWQHUUuXz1a3RE7v7z@mail.gmail.com>
References: <447928.62922.qm@web46311.mail.sp1.yahoo.com>
	<AANLkTilyIwJaPJDsqGdr860BF7iWQHUUuXz1a3RE7v7z@mail.gmail.com>
Date: Fri, 2 Jul 2010 10:10:36 -0700
Message-ID: <AANLkTikHMfe8gOyTsH6_CNYTQX0ZskjMtod1p1f_Mbxe@mail.gmail.com>
Subject: Re: Displaying Map output in MapReduce
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175ce08c630578048a6aaa02
X-Virus-Checked: Checked by ClamAV on apache.org

--0015175ce08c630578048a6aaa02
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Minor correction - port is 50075:
http://10.32.56.159:50075/logs/userlogs/

On Fri, Jul 2, 2010 at 12:32 AM, Alberto Luengo Cabanillas <
cabiwan@gmail.com> wrote:

> Hi! Supposing that you=B4re using a virtualized environment, you should b=
e
> able to get to JobTracker Web Admin Interface from http://
> <your_virtualized_
> machine_ip>:50070/logs/userlogs. In case you=B4re running Hadoop from ins=
ide
> a
> cluster frontend, use this machine=B4s IP. Inside this directory, it will
> exist another one called "userlogs" where you will see all map and reduce
> tasks launch by the JobTracker and the TaskTrackers. These tasks will loo=
k
> like 'job_[m,r]_0_xxxxxx' (or similar). Inside of every one of them there
> will be three subdirectories (stdout, stdlog and stderr) which will conta=
in
> all your debugging messages.
> Inside my MapReduce development programs I use the Java *LOG* class in a
> sentence like this:
>
> private static final Log LOG =3D
> LogFactory.getLog(<your_class_name>.class.getName());
> LOG.info("***********INSIDE MY REDUCER SETUP**********");
>
> Besides these messages, a normal "System.out.println(...)" will also appe=
ar
> in the above directories.
>
> Hope it helps!
>
> 2010/7/2 abc xyz <fabc_xyz111@yahoo.com>
>
> > Hi folks,
> >
> > I want to just display the output of my mapper on console or want to lo=
g
> it
> > in
> > some file. System.out.println and Log4j's logger are not displaying the
> > output.
> > How can I see the intermediate results?
> >
> > Cheers
> >
> >
> >
>
>
>
>
>
>
> --
> Alberto
>

--0015175ce08c630578048a6aaa02--

From general-return-1740-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 18:37:51 2010
Return-Path: <general-return-1740-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38310 invoked from network); 2 Jul 2010 18:37:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 18:37:50 -0000
Received: (qmail 34238 invoked by uid 500); 2 Jul 2010 18:37:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34180 invoked by uid 500); 2 Jul 2010 18:37:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34172 invoked by uid 99); 2 Jul 2010 18:37:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 18:37:49 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jdixon@omniti.com designates 8.8.38.6 as permitted sender)
Received: from [8.8.38.6] (HELO edge.omniti.com) (8.8.38.6)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 18:37:41 +0000
DKIM-Signature: v=1; a=rsa-sha256; d=omniti.com; s=s1024; c=relaxed/relaxed;
	q=dns/txt; i=@omniti.com; t=1278095840;
	h=From:Subject:Date:To;
	bh=5mW/o0Yolk6i0ziVhBG6Cou4Hi1vnQwwKX1QuEi5AVE=;
	b=aaiGkLgsqruxK6dClKmdSHzdKylWol4tYTNzcXENUpe8uS6z/W4PqM6x4HsEz3Cl
	DCLhaEqlEaSHuy9C1I/7f6yQcrM4ivgZspUDhNlekakw21E1GMbHGaHzOtpjbl7B
	VEiqxt/fdd7M50D5FSKlRuW/fmjqjQyMu335qGxBjN4=;
Authentication-Results:  edge smtp.user=jdixon@omniti.com; auth=pass (LOGIN)
Received: from [68.55.0.29] ([68.55.0.29:59137] helo=omniti.com)
	by edge (envelope-from <jdixon@omniti.com>)
	(ecelerity 2.2.2.35 r(26636M)) with ESMTPSA (cipher=AES256-SHA) 
	id B5/78-17327-0E13E2C4; Fri, 02 Jul 2010 14:37:20 -0400
Date: Fri, 2 Jul 2010 14:37:17 -0400
From: Jason Dixon <jdixon@omniti.com>
To: general@hadoop.apache.org
Subject: CFP for Surge Scalability Conference 2010
Message-ID: <20100702183716.GE29381@omniti.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.18 (2008-05-17)
X-Virus-Checked: Checked by ClamAV on apache.org

A quick reminder that there's one week left to submit your abstract for
this year's Surge Scalability Conference.  The event is taking place on
Sept 30 and Oct 1, 2010 in Baltimore, MD.  Surge focuses on case studies
that address production failures and the re-engineering efforts that led
to victory in Web Applications or Internet Architectures.

Our Keynote speakers include John Allspaw and Theo Schlossnagle.  We are
currently accepting submissions for the Call For Papers through July
9th.  You can find more information, including suggested topics and our
current list of speakers, online:

http://omniti.com/surge/2010

I'd also like to urge folks who are planning to attend, to get your
session passes sooner rather than later.  We have limited seating and we
are on track to sell out early.  For more information, including the
CFP, sponsorship of the event, or participating as an exhibitor, please
visit the Surge website or contact us at surge@omniti.com.

Thanks,

-- 
Jason Dixon
OmniTI Computer Consulting, Inc.
jdixon@omniti.com
443.325.1357 x.241

From general-return-1741-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 03 16:34:14 2010
Return-Path: <general-return-1741-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98297 invoked from network); 3 Jul 2010 16:34:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jul 2010 16:34:14 -0000
Received: (qmail 62934 invoked by uid 500); 3 Jul 2010 16:34:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62463 invoked by uid 500); 3 Jul 2010 16:34:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62448 invoked by uid 99); 3 Jul 2010 16:34:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Jul 2010 16:34:11 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [68.180.199.181] (HELO web46316.mail.sp1.yahoo.com) (68.180.199.181)
    by apache.org (qpsmtpd/0.29) with SMTP; Sat, 03 Jul 2010 16:34:01 +0000
Received: (qmail 3153 invoked by uid 60001); 3 Jul 2010 16:33:39 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1278174819; bh=TnnNYTHRRh+GC542QcHr2EnscnDQQ2bo5gvz7MAwZRM=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=Vt/aDLMhvz7493/Gw9b8pagpSnC8AVrb9aC1kbJZ8YxZpXYLR2+tmvYkGXzCWbPLDHnCp/y+f+ocAv6eTFp6GRerKEbFjrDUTNHSJzizEc/vfU+JNPBP1irXIrzEA2J6Ais+vUyAOwUp6FsdSgRRuNULfuZ+py1Ii4z9qkDIIXU=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=ke+U/vhxw7rbSM1qEFVSu8Xrnjt9WkGmFl8XCPfmt8ETM1HZdMs9StBIGfM112dlyeenaT1YEeBKFcSPRYS/7NTKybkPrC0GELvTjLX7T1Dju6eRo/PvT/gkGaFhzAhh7zo6qpSI8eA71kEAy4pkogxr6bDBtLJdgOvpby4nCwQ=;
Message-ID: <482872.98634.qm@web46316.mail.sp1.yahoo.com>
X-YMail-OSG: xgcX5LgVM1ntRgk9jSvfOWiUAeYf4yl7sNonJMfAPIdOjAj
 8D4xYKm_1jAu1jWkwUDpDc7L_TF8vCIU6GYFR7K0zEe0J2X0oVoOQ7JuAjpo
 vPVk0dtVz248pEVDNr4h7EXqa0evoy9jad1vSlHj_Uk_8qtr420EUIfjEJpU
 rfHJjvX5HH9P7p8nJ24aC.pEEpnQFJmj9m0rOlIO40Y4x.5MiSZxjeRsNesK
 IPzuCKbcGrr4KiZzA6jGi_YQggf33NNwZM9mJiuHKnjXhWT.XM1MPR0NqOJs
 vFbPTziLWBKUP.e4M6hVl5gq90Zc.lo7XpxJi
Received: from [188.74.76.201] by web46316.mail.sp1.yahoo.com via HTTP; Sat, 03 Jul 2010 09:33:39 PDT
X-Mailer: YahooMailRC/420.4 YahooMailWebService/0.8.104.274457
Date: Sat, 3 Jul 2010 09:33:39 -0700 (PDT)
From: abc xyz <fabc_xyz111@yahoo.com>
Subject: Partitioned Datasets Map/Reduce
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1646732546-1278174819=:98634"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-1646732546-1278174819=:98634
Content-Type: text/plain; charset=us-ascii

Hello everyone,

I have written my custom partitioner for partitioning datasets. I want  to 
partition two datasets using the same partitioner and then in the  next 
mapreduce job, I want each mapper to handle the same partition from  the two 
sources and perform some function such as joining etc. How I  can I ensure that 
one mapper gets the split that corresponds to same  partition from both the 
sources? 


Any help would be highly appreciated.


      
--0-1646732546-1278174819=:98634--

From general-return-1742-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 05 05:48:35 2010
Return-Path: <general-return-1742-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 15724 invoked from network); 5 Jul 2010 05:48:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jul 2010 05:48:34 -0000
Received: (qmail 99278 invoked by uid 500); 5 Jul 2010 05:48:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98977 invoked by uid 500); 5 Jul 2010 05:48:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98969 invoked by uid 99); 5 Jul 2010 05:48:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jul 2010 05:48:28 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of brandonusa@gmail.com designates 209.85.210.48 as permitted sender)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jul 2010 05:48:20 +0000
Received: by pzk36 with SMTP id 36so108861pzk.35
        for <general@hadoop.apache.org>; Sun, 04 Jul 2010 22:47:09 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:references:message-id:from:to
         :in-reply-to:content-type:content-transfer-encoding:x-mailer
         :mime-version:subject:date:cc;
        bh=uM0E/o0BpoqO6jafelTTLK2AAVDPfOjjgIqGQZJ9Vrk=;
        b=IlW6ucftcEmWvtvDyuBgakYUSaMCObp8CF2ILG8UUCyBWVO4DH9JyBog2RGrs5f1lp
         2WAc9wOKQqyQj/VKBPAH+kg/cWaqdDcFfVHLzPIxvXt/nbUCtNYGlhQvgKwKf9g4VqYR
         xkxHorrzAkSTpdbnxuxu44Z/YCBORDFVXUdLY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=references:message-id:from:to:in-reply-to:content-type
         :content-transfer-encoding:x-mailer:mime-version:subject:date:cc;
        b=NqWVIhG0jJs6+I8D6/piT4XvunB5Fm0Yx5khk9ARRejuV9nvug7D32CRnpp+bVmHNv
         y8C2/4wwjtAAkZsSJtHsv0bQ4SqrcOAFdCpWsCkKnoxF7HXUqTZKpHo91OBbVvzXc0Nm
         fKxhNv/wQc4sSP9pGVFSV9exX3x+TEB3vL0OE=
Received: by 10.142.188.13 with SMTP id l13mr2465980wff.54.1278308829784;
        Sun, 04 Jul 2010 22:47:09 -0700 (PDT)
Received: from [192.168.1.3] (cpe-76-94-135-238.socal.res.rr.com [76.94.135.238])
        by mx.google.com with ESMTPS id z1sm4153715wfd.15.2010.07.04.22.47.07
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Sun, 04 Jul 2010 22:47:08 -0700 (PDT)
References: <AANLkTiljgFWtvH2dQHhB44-8mXQKlAhtOPp5N1K3tJ3S@mail.gmail.com> <AANLkTikCO_TnPI-xBt_QQljWkTBWJ4lq3nGhHPSEjYNr@mail.gmail.com> <ABC24175AFD3BE4DA15F4CD375ED413D06084AF156@hq-ex-mb02.ad.navteq.com> <AANLkTilk2wfZ10leX0iEeNwJozMDdfHNiQ09rEAnbPDd@mail.gmail.com>
Message-Id: <4A6F17FD-7E27-4535-8F8E-B106064F2EA0@gmail.com>
From: Bardia Afshin <brandonusa@gmail.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <AANLkTilk2wfZ10leX0iEeNwJozMDdfHNiQ09rEAnbPDd@mail.gmail.com>
Content-Type: text/plain;
	charset=us-ascii;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit
X-Mailer: iPhone Mail (7E18)
Mime-Version: 1.0 (iPhone Mail 7E18)
Subject: Re: Why single thread for HDFS?
Date: Sun, 4 Jul 2010 22:47:02 -0700
Cc: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

What's the unsubcribe link?

Sent from my iPhone

On Jul 2, 2010, at 8:24 AM, Jay Booth <jaybooth@gmail.com> wrote:

> Yeah, a good way to think of it is that parallelism is achieved at the
> application level.
>
> On the input side, you can process multiple files in parallel or one
> file in parallel by logically splitting and opening multiple readers
> of the same file at multiple points.  Each of these readers is single
> threaded, because, well, you're returning a stream of bytes in order.
> It's inherently serial.
>
> On the reduce side, multiple reduces run, writing to multiple files in
> the same directory.  Again, you can't really write to a single file in
> parallel effectively -- you can't write byte 26 before byte 25,
> because the file's not that long yet.
>
> Theoretically, maybe you could have all reduces write to the same file
> by allocating some amount of space ahead of time and writing to the
> blocks in parallel - in practice, you very rarely know how big your
> output is going to be before it's produced, so this doesn't really
> work.  Multiple files in the same directory achieves the same goal
> much more elegantly, without exposing a bunch of internal details of
> the filesystem to user space.
>
> Does that make sense?
>
>
>
> On Fri, Jul 2, 2010 at 9:26 AM, Segel, Mike <msegel@navteq.com> wrote:
>> Actually they also listen here and this is a basic question...
>>
>> I'm not an expert, but how does having multiple threads really help  
>> this problem?
>>
>> I'm assuming you're talking about a map/reduce job and not some  
>> specific client code which is being run on a client outside of the  
>> cloud/cluster....
>>
>> I wasn't aware that you could easily synchronize threads running on  
>> different JVMs. ;-)
>>
>> Your parallelism comes from multiple tasks running on different  
>> nodes within the cloud. By default you get one map/reduce job per  
>> block. You can write your own splitter to increase this and then  
>> get more parallelism.
>>
>> HTH
>>
>> -Mike
>>
>>
>> -----Original Message-----
>> From: Hemanth Yamijala [mailto:yhemanth@gmail.com]
>> Sent: Friday, July 02, 2010 2:56 AM
>> To: general@hadoop.apache.org
>> Subject: Re: Why single thread for HDFS?
>>
>> Hi,
>>
>> Can you please post this on hdfs-dev@hadoop.apache.org ? I suspect  
>> the
>> most qualified people to answer this question would all be on that
>> list.
>>
>> Hemanth
>>
>> On Fri, Jul 2, 2010 at 11:43 AM, elton sky <eltonsky9404@gmail.com>  
>> wrote:
>>> I guess this question was igored, so I just post it again.
>>>
>>> From my understanding, HDFS uses a single thread to do read and  
>>> write.
>>> Since a file is composed of many blocks and each block is stored  
>>> as a file
>>> in the underlying FS, we can do some parallelism on block base.
>>> When read across multi-blocks, threads can be used to read all  
>>> blocks. When
>>> write, we can calculate the offset of each block and write to all  
>>> of them
>>> simultaneously.
>>>
>>> Is this right?
>>>
>>
>>
>> The information contained in this communication may be CONFIDENTIAL  
>> and is intended only for the use of the recipient(s) named above.   
>> If you are not the intended recipient, you are hereby notified that  
>> any dissemination, distribution, or copying of this communication,  
>> or any of its contents, is strictly prohibited.  If you have  
>> received this communication in error, please notify the sender and  
>> delete/destroy the original message and any copy of it from your  
>> computer or paper files.
>>

From general-return-1743-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 05 06:08:41 2010
Return-Path: <general-return-1743-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21465 invoked from network); 5 Jul 2010 06:08:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jul 2010 06:08:41 -0000
Received: (qmail 12854 invoked by uid 500); 5 Jul 2010 06:08:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12490 invoked by uid 500); 5 Jul 2010 06:08:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12482 invoked by uid 99); 5 Jul 2010 06:08:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jul 2010 06:08:37 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of bernd.fondermann@googlemail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jul 2010 06:08:30 +0000
Received: by wyb29 with SMTP id 29so3984236wyb.35
        for <general@hadoop.apache.org>; Sun, 04 Jul 2010 23:08:10 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=DSioEhsK8Ps9BhWr1w3YK6tKGGVIpjQCgnbSwrrTVNg=;
        b=raKyatGuwFTjXV7Bqllw7AmH1+eMs3t+ZGLPyhPw3Fsy6I6OGZSymxEIi1aXeHRiBx
         pRjCd312XfYDoQIaMXSiJpa1NPbaEVeoW7G11rjBACMvKEWn5K6xOhee+EgNEtOHPiBO
         7Y3wI43HADPd6mjprPg9zaKiBGHXHvLqP+uHs=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=MZCwcfQZNsbhm7reOS/3uAoLG3ZR0p+JoXoybh3n53mpSievexdRJiCz6rqXxPBafg
         lalST6cvVXOJQm5KUvtyXKZKDL7bLNz1PLFc3QwdZ8SBr8PjIGcXVkW+3s6UlMUWQ/pT
         ikpNpEdqqLc1WwIt3Vzbs5yr2UFOeIVzokdRM=
MIME-Version: 1.0
Received: by 10.227.129.139 with SMTP id o11mr2871508wbs.144.1278310090684; 
	Sun, 04 Jul 2010 23:08:10 -0700 (PDT)
Received: by 10.216.173.209 with HTTP; Sun, 4 Jul 2010 23:08:10 -0700 (PDT)
In-Reply-To: <4A6F17FD-7E27-4535-8F8E-B106064F2EA0@gmail.com>
References: <AANLkTiljgFWtvH2dQHhB44-8mXQKlAhtOPp5N1K3tJ3S@mail.gmail.com>
	<AANLkTikCO_TnPI-xBt_QQljWkTBWJ4lq3nGhHPSEjYNr@mail.gmail.com>
	<ABC24175AFD3BE4DA15F4CD375ED413D06084AF156@hq-ex-mb02.ad.navteq.com>
	<AANLkTilk2wfZ10leX0iEeNwJozMDdfHNiQ09rEAnbPDd@mail.gmail.com>
	<4A6F17FD-7E27-4535-8F8E-B106064F2EA0@gmail.com>
Date: Mon, 5 Jul 2010 08:08:10 +0200
Message-ID: <AANLkTinK5tYVHAoFDKwdys2BqMHNz5AmsKiW5Phn8YOk@mail.gmail.com>
Subject: Re: Why single thread for HDFS?
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Mon, Jul 5, 2010 at 07:47, Bardia Afshin <brandonusa@gmail.com> wrote:
> What's the unsubcribe link?

To unsubscribe, send mail to
general-unsubscribe@hadoop.apache.org

Many Apache MLs have an unsubscribe footer.
Anyone volunteering to make this happen for this list, too?

  Bernd

From general-return-1744-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 05 07:34:57 2010
Return-Path: <general-return-1744-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43397 invoked from network); 5 Jul 2010 07:34:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jul 2010 07:34:56 -0000
Received: (qmail 59415 invoked by uid 500); 5 Jul 2010 07:34:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59106 invoked by uid 500); 5 Jul 2010 07:34:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59098 invoked by uid 99); 5 Jul 2010 07:34:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jul 2010 07:34:51 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jul 2010 07:34:44 +0000
Received: by vws19 with SMTP id 19so8353576vws.35
        for <general@hadoop.apache.org>; Mon, 05 Jul 2010 00:34:22 -0700 (PDT)
Received: by 10.220.58.74 with SMTP id f10mr1308515vch.86.1278315262222; Mon, 
	05 Jul 2010 00:34:22 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.220.17.134 with HTTP; Mon, 5 Jul 2010 00:34:02 -0700 (PDT)
In-Reply-To: <AANLkTikHMfe8gOyTsH6_CNYTQX0ZskjMtod1p1f_Mbxe@mail.gmail.com>
References: <447928.62922.qm@web46311.mail.sp1.yahoo.com> <AANLkTilyIwJaPJDsqGdr860BF7iWQHUUuXz1a3RE7v7z@mail.gmail.com> 
	<AANLkTikHMfe8gOyTsH6_CNYTQX0ZskjMtod1p1f_Mbxe@mail.gmail.com>
From: Aaron Kimball <aaron@cloudera.com>
Date: Mon, 5 Jul 2010 00:34:02 -0700
Message-ID: <AANLkTilmJ2_uTB9j9V5GT2xsN84Z0lnYgBTPf2P0XrHN@mail.gmail.com>
Subject: Re: Displaying Map output in MapReduce
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00235445b87e1da17d048a9ef72c
X-Virus-Checked: Checked by ClamAV on apache.org

--00235445b87e1da17d048a9ef72c
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

If you set the number of reduce tasks to zero, the outputs of the mappers
will be sent directly to the OutputFormat. You can debug your map phase of =
a
job by disabling reduce and inspecting the mapper outputs, and then
re-enable the reducer after you've got the mapping part of the job running
correctly.

-- Aaron

On Fri, Jul 2, 2010 at 10:10 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> Minor correction - port is 50075:
> http://10.32.56.159:50075/logs/userlogs/
>
> On Fri, Jul 2, 2010 at 12:32 AM, Alberto Luengo Cabanillas <
> cabiwan@gmail.com> wrote:
>
> > Hi! Supposing that you=B4re using a virtualized environment, you should=
 be
> > able to get to JobTracker Web Admin Interface from http://
> > <your_virtualized_
> > machine_ip>:50070/logs/userlogs. In case you=B4re running Hadoop from
> inside
> > a
> > cluster frontend, use this machine=B4s IP. Inside this directory, it wi=
ll
> > exist another one called "userlogs" where you will see all map and redu=
ce
> > tasks launch by the JobTracker and the TaskTrackers. These tasks will
> look
> > like 'job_[m,r]_0_xxxxxx' (or similar). Inside of every one of them the=
re
> > will be three subdirectories (stdout, stdlog and stderr) which will
> contain
> > all your debugging messages.
> > Inside my MapReduce development programs I use the Java *LOG* class in =
a
> > sentence like this:
> >
> > private static final Log LOG =3D
> > LogFactory.getLog(<your_class_name>.class.getName());
> > LOG.info("***********INSIDE MY REDUCER SETUP**********");
> >
> > Besides these messages, a normal "System.out.println(...)" will also
> appear
> > in the above directories.
> >
> > Hope it helps!
> >
> > 2010/7/2 abc xyz <fabc_xyz111@yahoo.com>
> >
> > > Hi folks,
> > >
> > > I want to just display the output of my mapper on console or want to
> log
> > it
> > > in
> > > some file. System.out.println and Log4j's logger are not displaying t=
he
> > > output.
> > > How can I see the intermediate results?
> > >
> > > Cheers
> > >
> > >
> > >
> >
> >
> >
> >
> >
> >
> > --
> > Alberto
> >
>

--00235445b87e1da17d048a9ef72c--

From general-return-1745-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 05 07:39:58 2010
Return-Path: <general-return-1745-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44652 invoked from network); 5 Jul 2010 07:39:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jul 2010 07:39:58 -0000
Received: (qmail 71842 invoked by uid 500); 5 Jul 2010 07:39:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71636 invoked by uid 500); 5 Jul 2010 07:39:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71626 invoked by uid 99); 5 Jul 2010 07:39:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jul 2010 07:39:53 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jul 2010 07:39:45 +0000
Received: by vws19 with SMTP id 19so8357588vws.35
        for <general@hadoop.apache.org>; Mon, 05 Jul 2010 00:38:24 -0700 (PDT)
Received: by 10.220.58.74 with SMTP id f10mr1311811vch.86.1278315501099; Mon, 
	05 Jul 2010 00:38:21 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.220.17.134 with HTTP; Mon, 5 Jul 2010 00:38:01 -0700 (PDT)
In-Reply-To: <4C29C36D.2030806@apache.org>
References: <AANLkTil1b6fgdChEJO5t17-OcTVl2htVJ__Vpdf2D_L0@mail.gmail.com> 
	<AANLkTimkK7rLbj42lxluTOeqyHAug5_bssuGN5C1jtda@mail.gmail.com> 
	<AANLkTin0LUFfRsx-m0MbtNRIcbsv8gz0luVd5UAGdYig@mail.gmail.com> 
	<4C29C36D.2030806@apache.org>
From: Aaron Kimball <aaron@cloudera.com>
Date: Mon, 5 Jul 2010 00:38:01 -0700
Message-ID: <AANLkTilm_PRVXr0T6G2OkLwpYHct8gTEEfPVNkrsUcRi@mail.gmail.com>
Subject: Re: Can we modify files in HDFS?
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00235445b87e5a9b89048a9f0519
X-Virus-Checked: Checked by ClamAV on apache.org

--00235445b87e5a9b89048a9f0519
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Jun 29, 2010 at 2:57 AM, Steve Loughran <stevel@apache.org> wrote:

> elton sky wrote:
>
>> thanx Jeff,
>>
>> So...it is a significant drawback.
>> As a matter of fact, there are many cases we need to modify.
>>
>
>
> When people say "Hadoop filesystems are not posix", this is what they mean.
> No locks, no read/write. seeking discouraged. Even append is something that
> is just stabilising. to be fair though, even NFS is quirky and that's been
> around since Ether-net was considered so cutting edge it had a hyphen in the
> title.
>
> HDFS delivers availability through redundant copies across multiple
> machines: you can read your data on or near any machine with a copy of the
> data. Think what you'd need for full seek and read/write actions
>
>
> * seek would kill bulk IO perf on classic rotating-disk HDDs, and nobody
> can afford to build a petabyte filestore out of SSDs yet. You should be
> streaming, not seeking.
>
> * to do writes, you'd need to lock out access to the files, which implies a
> distributed lock infrastructure (zookeeper?) or deal with conflicting
> writes.
>
> * if you want immediate update writes you'd need to push out the changes to
> the (existing) nodes, and deal with queueing up pending changes to machines
> that are currently offline in a way that I don't want to think about.
>
> * if you want slower-update writes (eventual consistency), then things may
> be slightly simpler -you'd need a lock on writing and each write would
> eventually be pushed out to the readers with a bit better bandwidth and CPU
> scheduling flexibility , but there's still that offline node problem. If a
> node that was down comes up, how does it know it's data is out of date and
> where does it get the data from? What will it do if all other nodes that
> have updated data are offline.
>
>
>
> > I dont understand why Yahoo didn't provoid that functionality. And as I
> know
> > no one else is working on this. Why is that?
>
> It's because it scares us and we are happier writing code to live in a
> world where you don't seek and patch files, but instead add new data and
> delete old stuff. I don't know what the Cassandra and HBase teams do here.
>

It's my understanding that HBase stores datasets in reasonably small files
(a few hundred MB each?) where deltas to a section are in more files that
are actually stacked on top of older ones. Every so often a garbage
collection routine compacts a stack of files for a slice of the table down
into a single file for that slice.


And yes, Yahoo (nor anybody else, for that matter) has not provided this
functionality because it's a lot trickier to add true overwrite capability
in a distributed environment than one might think at first glance. The
engineering cost of developing modification support has, for all
parties interested thus far, been higher than the cost of working around
this limitation. :)



> -steve
>
>
>
>
>

--00235445b87e5a9b89048a9f0519--

From general-return-1746-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 05 08:13:44 2010
Return-Path: <general-return-1746-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56485 invoked from network); 5 Jul 2010 08:13:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jul 2010 08:13:43 -0000
Received: (qmail 20149 invoked by uid 500); 5 Jul 2010 08:13:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18659 invoked by uid 500); 5 Jul 2010 08:13:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18651 invoked by uid 99); 5 Jul 2010 08:13:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jul 2010 08:13:35 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=CTYPE_001C_B,HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [145.100.8.50] (HELO sara-exch-fe2.ka.sara.nl) (145.100.8.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jul 2010 08:13:26 +0000
Received: from planck.ka.sara.nl (145.100.8.32) by sara-exch-fe2.ka.sara.nl
 (145.100.8.50) with Microsoft SMTP Server (TLS) id 14.0.694.0; Mon, 5 Jul
 2010 10:12:44 +0200
Received: from planck.ka.sara.nl ([145.100.8.32]) by planck.ka.sara.nl
 ([145.100.8.32]) with mapi; Mon, 5 Jul 2010 10:12:44 +0200
From: Evert Lammerts <Evert.Lammerts@sara.nl>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 5 Jul 2010 10:12:43 +0200
Subject: Hadoop versions & distributions
Thread-Topic: Hadoop versions & distributions
Thread-Index: AcscGdXRU3173thcQcCGXH49/En1Yg==
Message-ID: <ADF94D8555C7A246B86A633685E0178AB8F2803C36@planck.ka.sara.nl>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/signed; protocol="application/x-pkcs7-signature";
	micalg=SHA1; boundary="----=_NextPart_000_0000_01CB1C2A.9A605600"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

------=_NextPart_000_0000_01CB1C2A.9A605600
Content-Type: multipart/alternative;
	boundary="----=_NextPart_001_0001_01CB1C2A.9A605600"


------=_NextPart_001_0001_01CB1C2A.9A605600
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit

There are a number of different versions and distributions of Hadoop which,
as far as I understand, all differ from each other. I know that in the
0.20-append branch, files in HDFS can be appended, and that the Y!
distribution (0.20.S) implements security features through Kerberos. And
then there are the 0.20.3 and 0.22.0 branches. And trunk of course, which I
guess is 0.20.2 nowadays? In addition to that there are distributions by
Cloudera(CDH2 / 3beta) and IBM (IDAH).

 

>From my perspective, setting up a pilot cluster for a small number of users
from different institutes, security (0.20.S) is very attractive - scientists
like the idea of shielding their data and logic from other users. But what
will I miss if I choose Y!'s distribution over all of these other options?

 

Regards,

 

Evert Lammerts


------=_NextPart_001_0001_01CB1C2A.9A605600
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" =
xmlns=3D"http://www.w3.org/TR/REC-html40">

<head>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<meta name=3DGenerator content=3D"Microsoft Word 12 (filtered medium)">
<style>
<!--
 /* Font Definitions */
 @font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
	{font-family:"Malgun Gothic";
	panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
	{font-family:"\@Malgun Gothic";
	panose-1:0 0 0 0 0 0 0 0 0 0;}
 /* Style Definitions */
 p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0cm;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:purple;
	text-decoration:underline;}
span.EmailStyle17
	{mso-style-type:personal-compose;
	font-family:"Calibri","sans-serif";
	color:windowtext;}
.MsoChpDefault
	{mso-style-type:export-only;}
@page Section1
	{size:612.0pt 792.0pt;
	margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.Section1
	{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
 <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
 <o:shapelayout v:ext=3D"edit">
  <o:idmap v:ext=3D"edit" data=3D"1" />
 </o:shapelayout></xml><![endif]-->
</head>

<body lang=3DEN-US link=3Dblue vlink=3Dpurple>

<div class=3DSection1>

<p class=3DMsoNormal>There are a number of different versions and =
distributions of
Hadoop which, as far as I understand, all differ from each other. I know =
that
in the 0.20-append branch, files in HDFS can be appended, and that the =
Y! distribution
(0.20.S) implements security features through Kerberos. And then there =
are the 0.20.3
and 0.22.0 branches. And trunk of course, which I guess is 0.20.2 =
nowadays? In
addition to that there are distributions by Cloudera(CDH2 / 3beta) and =
IBM (IDAH).<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>From my perspective, setting up a pilot cluster for =
a small
number of users from different institutes, security (0.20.S) is very =
attractive
&#8211; scientists like the idea of shielding their data and logic from =
other
users. But what will I miss if I choose Y!&#8217;s distribution over all =
of
these other options?<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>Regards,<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal><span =
style=3D'font-size:10.0pt;font-family:"Courier New"'>Evert
Lammerts<o:p></o:p></span></p>

</div>

</body>

</html>

------=_NextPart_001_0001_01CB1C2A.9A605600--

------=_NextPart_000_0000_01CB1C2A.9A605600
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIPUDCCBDIw
ggMaoAMCAQICAQEwDQYJKoZIhvcNAQEFBQAwezELMAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0
ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0
ZWQxITAfBgNVBAMMGEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczAeFw0wNDAxMDEwMDAwMDBaFw0y
ODEyMzEyMzU5NTlaMHsxCzAJBgNVBAYTAkdCMRswGQYDVQQIDBJHcmVhdGVyIE1hbmNoZXN0ZXIx
EDAOBgNVBAcMB1NhbGZvcmQxGjAYBgNVBAoMEUNvbW9kbyBDQSBMaW1pdGVkMSEwHwYDVQQDDBhB
QUEgQ2VydGlmaWNhdGUgU2VydmljZXMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC+
QJ30buHqdoccTUVEjr5GyIMGncEq/hgfjuQC+vOrXVCKFjELmgbQxXAizUktVGPMtm5oRgtT6stM
JMC8ck7q8RWu9FSaEgrDerIzYOLaiVXzIljz3tzP74OGooyUT59o8piQRoQnx3a/48w1LIteB2Rl
gsBIsKiR+WGfdiBQqJHHZrXreGIDVvCKGhPqMaMeoJn9OPb2JzJYbwf1a7j7FCuvt6rM1mNfc4za
BZmoOKjLF3g2UazpnvR4Oo3PD9lC4pgMqy+fDgHe75+ZSfEt36x0TRuYtUfF5SnR+ZAYx2KcvoPH
Jns+iiXHwN2d5jVoECCdj9je0sOEnA1e6C/JAgMBAAGjgcAwgb0wHQYDVR0OBBYEFKARCiM+lvEH
7OKvKe+CpX/QMKS0MA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MHsGA1UdHwR0MHIw
OKA2oDSGMmh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0FBQUNlcnRpZmljYXRlU2VydmljZXMuY3Js
MDagNKAyhjBodHRwOi8vY3JsLmNvbW9kby5uZXQvQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmww
DQYJKoZIhvcNAQEFBQADggEBAAhW/ALwm+j/pPrWe8ZEgM5PxMX2AFjMpra8FEloBHbo5u5d7AIP
YNaNUBhPJk4B4+awpe6/vHRUQb/9/BK4x09a9IlgBX9gtwVK8/bxwr/EuXSGti19a8zS80bdL8bg
asPDNAMsfZbdWsIOpwqZwQWLqwwv81w6z2w3VQmH3lNAbFjv/LarZW4E9hvcPOBaFcae2fFZSDAh
ZQNs7Okhc+ybA6HgN62gFRiP+roCzqcsqRATLNTlCCarIpdg+JBedNSimlO98qlo4KJuwtdssaMP
nr/raOdW8q7y4ys4OgmBtWuF174t7T8at7Jj4vViLILUagBBUPE5g5+V6TaWmG4wggTdMIIDxaAD
AgECAhBxkvvmGV+sTRKFdHE0ohinMA0GCSqGSIb3DQEBBQUAMHsxCzAJBgNVBAYTAkdCMRswGQYD
VQQIDBJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcMB1NhbGZvcmQxGjAYBgNVBAoMEUNvbW9k
byBDQSBMaW1pdGVkMSEwHwYDVQQDDBhBQUEgQ2VydGlmaWNhdGUgU2VydmljZXMwHhcNMDQwMTAx
MDAwMDAwWhcNMjgxMjMxMjM1OTU5WjCBrjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYD
VQQHEw5TYWx0IExha2UgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYD
VQQLExhodHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1VU0VSRmlyc3QtQ2xp
ZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBFbWFpbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBALI5haTyfatBO2JGN67NwWB1vDll+UoaR6K5zEjMapjVTTUZuaRC5c5J4oovHnzSMQfHTrSD
ZJ0uKdWiZMSFvYVRNXmkTmiQexx6pJKoF/KYFfKTzMmkMpW7DE8wvZigC4vlbhuiRvp4vKJvq1le
pS/Pytptqi/rrKGzaqq3Lmc1i3nhHmmI4uZGzaCl6r4LznY6eg6b6vzaJ1s9cx8i5khhxkzzabGo
Lhu21DEgLLyCio6kDqXXiUP8FlqvHXHXEVnauocNr/rz4cLwpMVnjNbWVDreCqS6A3ezZcj9HtN0
YqoYymiTHqGFfvVHZcv4TVcodNI0/zC27vZiMBSMLOsCAwEAAaOCAScwggEjMB8GA1UdIwQYMBaA
FKARCiM+lvEH7OKvKe+CpX/QMKS0MB0GA1UdDgQWBBSJgmd9xJ0mcABLtFBIfN49rgRufTAOBgNV
HQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUH
AwQwEQYDVR0gBAowCDAGBgRVHSAAMHsGA1UdHwR0MHIwOKA2oDSGMmh0dHA6Ly9jcmwuY29tb2Rv
Y2EuY29tL0FBQUNlcnRpZmljYXRlU2VydmljZXMuY3JsMDagNKAyhjBodHRwOi8vY3JsLmNvbW9k
by5uZXQvQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwEQYJYIZIAYb4QgEBBAQDAgEGMA0GCSqG
SIb3DQEBBQUAA4IBAQCdlcs8uH6lCcQevwvCx3aOOTyUxhCqTwzJ4KuEXYlU4GU7820cfDcsJVRf
liH8N4SRnRXcFE+Bz1Qda2xFYMct+ZdRTPlmyjyggoymyPDi6dRK+ew/VsnddozDggFPbADzHhph
dARHA6nGQFeRvGUixSdnT1fbZFrZjR+6hi/0Bq6cae3p9M8pF9jgSp8aIC+XTFG7RgfEijdOIOMJ
MWjHnsSLneh+EbwyaBCWEZhE2CpRYE2I63Q630MGMsg5Vow6EVLTQaRDA/Tt7zMn2zngFE4mydj1
OeKJuJNdtykmQeqzm66D/Hd1yujKtf7iZUpjPkTE0MNeh3OpmByvfxV/MIIGNTCCBR2gAwIBAgIR
APUHLR99vIaqc6YYFoh++L8wDQYJKoZIhvcNAQEFBQAwga4xCzAJBgNVBAYTAlVTMQswCQYDVQQI
EwJVVDEXMBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNVBAoTFVRoZSBVU0VSVFJVU1QgTmV0
d29yazEhMB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3QuY29tMTYwNAYDVQQDEy1VVE4tVVNF
UkZpcnN0LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1haWwwHhcNMTAwMjA4MDAwMDAwWhcN
MTEwMjA4MjM1OTU5WjCB4DE1MDMGA1UECxMsQ29tb2RvIFRydXN0IE5ldHdvcmsgLSBQRVJTT05B
IE5PVCBWQUxJREFURUQxRjBEBgNVBAsTPVRlcm1zIGFuZCBDb25kaXRpb25zIG9mIHVzZTogaHR0
cDovL3d3dy5jb21vZG8ubmV0L3JlcG9zaXRvcnkxHzAdBgNVBAsTFihjKTIwMDMgQ29tb2RvIExp
bWl0ZWQxFzAVBgNVBAMTDkV2ZXJ0IExhbW1lcnRzMSUwIwYJKoZIhvcNAQkBFhZldmVydC5sYW1t
ZXJ0c0BzYXJhLm5sMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtYAj4sXuBqEwPxYY
BMkYLPGrnqT35USIERoWs26xtSY+dLVc7HqaHDZ1lU5m9wFx/Jqph1urlRuAcCj03kGSS23Ta6kt
HTRMYRCHI3nOLfEEbH4POT+wDytORUHNeEKp/PrBn1kmKk+dk+bJI7MNdy2XkiEeO+OqKmLiBGkT
glIxngOeqgqR2IvmVv2hLAyzq8Ax20inwveZsDMa7QdQLSVUaX1kSRSihwz4Jw9X/K+6oA3ZLGp9
pZYnFHBNTHM2DR/C7dNwQgbZS7TY/Jb1kObYNhwD2JzzJlkoe0blR17MeJkF7/+Y414j0AdPFB7I
ggZ4Tm7Maheer9cIQSsvIQIDAQABo4ICGDCCAhQwHwYDVR0jBBgwFoAUiYJnfcSdJnAAS7RQSHze
Pa4Ebn0wHQYDVR0OBBYEFNTyUAqBNg9JXuiPHa9jfLvRM3IyMA4GA1UdDwEB/wQEAwIFoDAMBgNV
HRMBAf8EAjAAMCAGA1UdJQQZMBcGCCsGAQUFBwMEBgsrBgEEAbIxAQMFAjARBglghkgBhvhCAQEE
BAMCBSAwRgYDVR0gBD8wPTA7BgwrBgEEAbIxAQIBAQEwKzApBggrBgEFBQcCARYdaHR0cHM6Ly9z
ZWN1cmUuY29tb2RvLm5ldC9DUFMwgaUGA1UdHwSBnTCBmjBMoEqgSIZGaHR0cDovL2NybC5jb21v
ZG9jYS5jb20vVVROLVVTRVJGaXJzdC1DbGllbnRBdXRoZW50aWNhdGlvbmFuZEVtYWlsLmNybDBK
oEigRoZEaHR0cDovL2NybC5jb21vZG8ubmV0L1VUTi1VU0VSRmlyc3QtQ2xpZW50QXV0aGVudGlj
YXRpb25hbmRFbWFpbC5jcmwwbAYIKwYBBQUHAQEEYDBeMDYGCCsGAQUFBzAChipodHRwOi8vY3J0
LmNvbW9kb2NhLmNvbS9VVE5BQUFDbGllbnRDQS5jcnQwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3Nw
LmNvbW9kb2NhLmNvbTAhBgNVHREEGjAYgRZldmVydC5sYW1tZXJ0c0BzYXJhLm5sMA0GCSqGSIb3
DQEBBQUAA4IBAQBaJvuiBPgqdq9MEYH9dj2vW8hprAIBEnOOvfXdoP604kBhSbj3s/l0ZQyQY35W
YVAltuJQ365uJKigPQHxS+slpUOAJPEVNmwPUIW0GAjxCPxgLdJgAc4jOV8f7oF3pnfI4ukCmjPN
LpaylZzMhFt+t6zDWiMtUqyxK/4on62LoLp2D88lxwxI3q3i00bPlV0wZ+mjzS7vrQGcUgDPWEBL
FD0I3pR+Z1EH1qjelBmBJV9paVzfzmoU93LIDJA7/MEUXd3JiMZGEoVd2qE5qIZO4fNJ+Cyo6tu1
ECSQxoL6qN+sBwekiFA9Q/zlUp0HfU6Jt8WmtZ44SOh67LcmCL4ZMYIEaDCCBGQCAQEwgcQwga4x
CzAJBgNVBAYTAlVTMQswCQYDVQQIEwJVVDEXMBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNV
BAoTFVRoZSBVU0VSVFJVU1QgTmV0d29yazEhMB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3Qu
Y29tMTYwNAYDVQQDEy1VVE4tVVNFUkZpcnN0LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1h
aWwCEQD1By0ffbyGqnOmGBaIfvi/MAkGBSsOAwIaBQCgggJ4MBgGCSqGSIb3DQEJAzELBgkqhkiG
9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTEwMDcwNTA4MTI0MVowIwYJKoZIhvcNAQkEMRYEFP54v6RG
fT6L5qDh49dCUmQJS01UMGcGCSqGSIb3DQEJDzFaMFgwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwIC
AgCAMA0GCCqGSIb3DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMAcGBSsOAwIaMAoGCCqG
SIb3DQIFMIHVBgkrBgEEAYI3EAQxgccwgcQwga4xCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJVVDEX
MBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNVBAoTFVRoZSBVU0VSVFJVU1QgTmV0d29yazEh
MB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3QuY29tMTYwNAYDVQQDEy1VVE4tVVNFUkZpcnN0
LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1haWwCEQD1By0ffbyGqnOmGBaIfvi/MIHXBgsq
hkiG9w0BCRACCzGBx6CBxDCBrjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5T
YWx0IExha2UgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYDVQQLExho
dHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1VU0VSRmlyc3QtQ2xpZW50IEF1
dGhlbnRpY2F0aW9uIGFuZCBFbWFpbAIRAPUHLR99vIaqc6YYFoh++L8wDQYJKoZIhvcNAQEBBQAE
ggEAa4jDHbSsjjJ/eKiYH6OVmsDDgcEmZy/3wOqotzW74nEsBCM+lcw+01KIyNg5opVZ28ppx/7T
+jJZ3gfg8U1sDPa8lc4MCku0NxGWdCYb1qoVy2j2p69fbgdKZYpJID6GWlkNqLKcsO1MAcIWcZUh
pAptmpUZyk6Jr5qYbYakrL7edj72m9GZOWZW5g5PpUX3f1al+D7c3oRQxM7Bfv0TKGnZPQrb8+ZA
+LRNk+p4te2v7wmqCDgemzEJrsMUTi1jLWgXsOY1SEoYfeyP947VBIj6gRgMxcQrSoEniDWandLD
KgBg16aze9jVsePQ+fjZiAj2Wa6YRABG6Zp7822BoAAAAAAAAA==

------=_NextPart_000_0000_01CB1C2A.9A605600--

From general-return-1747-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 05 08:31:13 2010
Return-Path: <general-return-1747-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59489 invoked from network); 5 Jul 2010 08:31:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jul 2010 08:31:12 -0000
Received: (qmail 35409 invoked by uid 500); 5 Jul 2010 08:31:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 35069 invoked by uid 500); 5 Jul 2010 08:31:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 35061 invoked by uid 99); 5 Jul 2010 08:31:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jul 2010 08:31:08 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of zhangguoping96@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jul 2010 08:31:00 +0000
Received: by vws19 with SMTP id 19so8415223vws.35
        for <general@hadoop.apache.org>; Mon, 05 Jul 2010 01:29:39 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=jhzhrBGrvWeqbW4C2rpb8jWyVQgnvGuhZIiW+rOgwPw=;
        b=ewghYEhV+8zKtuz/gqVA1VZB6XTkek1UMa2Hi8okIWGzafO//xE6PgeJsVThMU7Bih
         VyzoZP+IqJ7BWc9RI2YdYDbCwjHZnCeNburFR87WVXdzjoskxJVh8016CtgTEXeQMBtf
         G0E0lkX5Aj4oM1pcWQK4qHrjKJX7+Dl2BQtKw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=R9iPk0EbFzhvHOZyBdUx0MdiviRxNftFX5ikYj10H5h5/c0NhlTD824zuVAjg9u9MM
         gYO5jD4feEbxZfInF4sTKLJ2mk+mM53ZMBDeB8MMps9TjditjubvcqmS6AdYOqiaFgrS
         zzoWfs04hgUmdWIT/MJHy5qrmmdrOZuXPcEeo=
MIME-Version: 1.0
Received: by 10.229.183.84 with SMTP id cf20mr1294924qcb.93.1278318579392; 
	Mon, 05 Jul 2010 01:29:39 -0700 (PDT)
Received: by 10.42.4.213 with HTTP; Mon, 5 Jul 2010 01:29:39 -0700 (PDT)
Date: Mon, 5 Jul 2010 16:29:39 +0800
Message-ID: <AANLkTinCufLiyHIcg6o4KaFbAGj3MF3WLe-5t1_faBnb@mail.gmail.com>
Subject: Is org.apache.hadoop.mapred.lib.MultipleOutputFormat deprecated?
From: zhangguoping zhangguoping <zhangguoping96@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163630ff65d5a337048a9fbc3f
X-Virus-Checked: Checked by ClamAV on apache.org

--00163630ff65d5a337048a9fbc3f
Content-Type: text/plain; charset=ISO-8859-1

Hi,

Is org.apache.hadoop.mapred.lib.MultipleOutputFormat deprecated? I did not
find @deprecated comments in source file in 0.20.2

But I cannot use following:

job.setOutputFormatClass(  org.apache.hadoop.mapred.lib.MultipleOutputFormat
)


The type does not match.

--00163630ff65d5a337048a9fbc3f--

From general-return-1748-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 05 12:10:10 2010
Return-Path: <general-return-1748-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24213 invoked from network); 5 Jul 2010 12:10:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jul 2010 12:10:09 -0000
Received: (qmail 98776 invoked by uid 500); 5 Jul 2010 12:10:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98240 invoked by uid 500); 5 Jul 2010 12:10:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98231 invoked by uid 99); 5 Jul 2010 12:10:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jul 2010 12:10:04 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eltonsky9404@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jul 2010 12:09:57 +0000
Received: by qwd7 with SMTP id 7so2661102qwd.35
        for <general@hadoop.apache.org>; Mon, 05 Jul 2010 05:08:47 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=YhHZwWn4dpL7kXvdV4QyS87LIuTAQCcgpBQdCBpr7P4=;
        b=DCedBqSstu7ri3PM1m3ZSA+mugOIv6J+tocf4nITn1njqUY/ZmUQ8KSwDsNgvs4Avy
         fLoFl4aYm9LM/eP/GXCOXOgsQjdMKYKC3QUygasnBNJmFXq11VfUvVByxeYudOQXEJwO
         wKM8mWwruFR5fDsmSVB9b78SWtx94ngZEQpnc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=MwK8SoTTomJz8ZlLjSqzohpYOoxKLE64jhzFFIMWcBy3zU16PdH2fBrV2JpFtSZtqr
         hDkm21WBVzZkCxRT8OgJWV3fbtkN6kqL57nl9aBjm062alPepYJqPFnjXgLdJjdritXr
         nCst/jL/urALOdkPqOxHIaW1ig/5sKiBeXYjI=
MIME-Version: 1.0
Received: by 10.224.59.195 with SMTP id m3mr1006558qah.34.1278331727110; Mon, 
	05 Jul 2010 05:08:47 -0700 (PDT)
Received: by 10.224.89.18 with HTTP; Mon, 5 Jul 2010 05:08:47 -0700 (PDT)
In-Reply-To: <AANLkTilk2wfZ10leX0iEeNwJozMDdfHNiQ09rEAnbPDd@mail.gmail.com>
References: <AANLkTiljgFWtvH2dQHhB44-8mXQKlAhtOPp5N1K3tJ3S@mail.gmail.com>
	<AANLkTikCO_TnPI-xBt_QQljWkTBWJ4lq3nGhHPSEjYNr@mail.gmail.com>
	<ABC24175AFD3BE4DA15F4CD375ED413D06084AF156@hq-ex-mb02.ad.navteq.com>
	<AANLkTilk2wfZ10leX0iEeNwJozMDdfHNiQ09rEAnbPDd@mail.gmail.com>
Date: Mon, 5 Jul 2010 22:08:47 +1000
Message-ID: <AANLkTimmN23EV17S1g-2LITHQrasMmh6qbSj13CJyfb5@mail.gmail.com>
Subject: Re: Why single thread for HDFS?
From: elton sky <eltonsky9404@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00c09fa21ac37fe6c7048aa2cc54
X-Virus-Checked: Checked by ClamAV on apache.org

--00c09fa21ac37fe6c7048aa2cc54
Content-Type: text/plain; charset=ISO-8859-1

Segel, Jay
Thanks for reply!

>Your parallelism comes from multiple tasks running on different nodes
within the cloud. By >default you get one map/reduce job per block. You can
write your own splitter to increase >this and then get more parallelism.
sounds like an elegant solution. We can modify the 'distcp', using a simple
MR job, make it based on block rather than file.

>in practice, you very rarely know how big your output is going to be before
it's produced, so >this doesn't really work
I think you got the point why Yahoo make this design descision.
Multithreading only applicable when you know the size of the file, like copy
existing files, so you can split them and feed to different threads.

On Sat, Jul 3, 2010 at 1:24 AM, Jay Booth <jaybooth@gmail.com> wrote:

> Yeah, a good way to think of it is that parallelism is achieved at the
> application level.
>
> On the input side, you can process multiple files in parallel or one
> file in parallel by logically splitting and opening multiple readers
> of the same file at multiple points.  Each of these readers is single
> threaded, because, well, you're returning a stream of bytes in order.
> It's inherently serial.
>
> On the reduce side, multiple reduces run, writing to multiple files in
> the same directory.  Again, you can't really write to a single file in
> parallel effectively -- you can't write byte 26 before byte 25,
> because the file's not that long yet.
>
> Theoretically, maybe you could have all reduces write to the same file
> by allocating some amount of space ahead of time and writing to the
> blocks in parallel - in practice, you very rarely know how big your
> output is going to be before it's produced, so this doesn't really
> work.  Multiple files in the same directory achieves the same goal
> much more elegantly, without exposing a bunch of internal details of
> the filesystem to user space.
>
> Does that make sense?
>
>
>
> On Fri, Jul 2, 2010 at 9:26 AM, Segel, Mike <msegel@navteq.com> wrote:
> > Actually they also listen here and this is a basic question...
> >
> > I'm not an expert, but how does having multiple threads really help this
> problem?
> >
> > I'm assuming you're talking about a map/reduce job and not some specific
> client code which is being run on a client outside of the cloud/cluster....
> >
> > I wasn't aware that you could easily synchronize threads running on
> different JVMs. ;-)
> >
> > Your parallelism comes from multiple tasks running on different nodes
> within the cloud. By default you get one map/reduce job per block. You can
> write your own splitter to increase this and then get more parallelism.
> >
> > HTH
> >
> > -Mike
> >
> >
> > -----Original Message-----
> > From: Hemanth Yamijala [mailto:yhemanth@gmail.com]
> > Sent: Friday, July 02, 2010 2:56 AM
> > To: general@hadoop.apache.org
> > Subject: Re: Why single thread for HDFS?
> >
> > Hi,
> >
> > Can you please post this on hdfs-dev@hadoop.apache.org ? I suspect the
> > most qualified people to answer this question would all be on that
> > list.
> >
> > Hemanth
> >
> > On Fri, Jul 2, 2010 at 11:43 AM, elton sky <eltonsky9404@gmail.com>
> wrote:
> >> I guess this question was igored, so I just post it again.
> >>
> >> From my understanding, HDFS uses a single thread to do read and write.
> >> Since a file is composed of many blocks and each block is stored as a
> file
> >> in the underlying FS, we can do some parallelism on block base.
> >> When read across multi-blocks, threads can be used to read all blocks.
> When
> >> write, we can calculate the offset of each block and write to all of
> them
> >> simultaneously.
> >>
> >> Is this right?
> >>
> >
> >
> > The information contained in this communication may be CONFIDENTIAL and
> is intended only for the use of the recipient(s) named above.  If you are
> not the intended recipient, you are hereby notified that any dissemination,
> distribution, or copying of this communication, or any of its contents, is
> strictly prohibited.  If you have received this communication in error,
> please notify the sender and delete/destroy the original message and any
> copy of it from your computer or paper files.
> >
>

--00c09fa21ac37fe6c7048aa2cc54--

From general-return-1749-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 05 17:02:47 2010
Return-Path: <general-return-1749-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95242 invoked from network); 5 Jul 2010 17:02:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jul 2010 17:02:47 -0000
Received: (qmail 56301 invoked by uid 500); 5 Jul 2010 17:02:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56243 invoked by uid 500); 5 Jul 2010 17:02:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56235 invoked by uid 99); 5 Jul 2010 17:02:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jul 2010 17:02:45 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jul 2010 17:02:38 +0000
Received: by iwn37 with SMTP id 37so3877663iwn.35
        for <general@hadoop.apache.org>; Mon, 05 Jul 2010 10:02:16 -0700 (PDT)
Received: by 10.231.148.131 with SMTP id p3mr3806624ibv.18.1278349336286; Mon, 
	05 Jul 2010 10:02:16 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.161.72 with HTTP; Mon, 5 Jul 2010 10:01:56 -0700 (PDT)
In-Reply-To: <AANLkTimmN23EV17S1g-2LITHQrasMmh6qbSj13CJyfb5@mail.gmail.com>
References: <AANLkTiljgFWtvH2dQHhB44-8mXQKlAhtOPp5N1K3tJ3S@mail.gmail.com> 
	<AANLkTikCO_TnPI-xBt_QQljWkTBWJ4lq3nGhHPSEjYNr@mail.gmail.com> 
	<ABC24175AFD3BE4DA15F4CD375ED413D06084AF156@hq-ex-mb02.ad.navteq.com> 
	<AANLkTilk2wfZ10leX0iEeNwJozMDdfHNiQ09rEAnbPDd@mail.gmail.com> 
	<AANLkTimmN23EV17S1g-2LITHQrasMmh6qbSj13CJyfb5@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 5 Jul 2010 10:01:56 -0700
Message-ID: <AANLkTikNr4C2mjzLSwxUFglxtbfEalCwQ8clyzCNl_8a@mail.gmail.com>
Subject: Re: Why single thread for HDFS?
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485e7e92a169a0c048aa6e67d
X-Virus-Checked: Checked by ClamAV on apache.org

--001485e7e92a169a0c048aa6e67d
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Jul 5, 2010 at 5:08 AM, elton sky <eltonsky9404@gmail.com> wrote:

> Segel, Jay
> Thanks for reply!
>
> >Your parallelism comes from multiple tasks running on different nodes
> within the cloud. By >default you get one map/reduce job per block. You can
> write your own splitter to increase >this and then get more parallelism.
> sounds like an elegant solution. We can modify the 'distcp', using a simple
> MR job, make it based on block rather than file.
>

There's actually an open ticket somewhere to make distcp do this using the
new concat() API in the NameNode. concat() allows several files to be
combined into one file at the metadata level, so long as a number of
restrictions are met. The work hasn't been done yet, but the concat() call
is there and waiting for a user.

-Todd


>
> >in practice, you very rarely know how big your output is going to be
> before
> it's produced, so >this doesn't really work
> I think you got the point why Yahoo make this design descision.
> Multithreading only applicable when you know the size of the file, like
> copy
> existing files, so you can split them and feed to different threads.
>
> On Sat, Jul 3, 2010 at 1:24 AM, Jay Booth <jaybooth@gmail.com> wrote:
>
> > Yeah, a good way to think of it is that parallelism is achieved at the
> > application level.
> >
> > On the input side, you can process multiple files in parallel or one
> > file in parallel by logically splitting and opening multiple readers
> > of the same file at multiple points.  Each of these readers is single
> > threaded, because, well, you're returning a stream of bytes in order.
> > It's inherently serial.
> >
> > On the reduce side, multiple reduces run, writing to multiple files in
> > the same directory.  Again, you can't really write to a single file in
> > parallel effectively -- you can't write byte 26 before byte 25,
> > because the file's not that long yet.
> >
> > Theoretically, maybe you could have all reduces write to the same file
> > by allocating some amount of space ahead of time and writing to the
> > blocks in parallel - in practice, you very rarely know how big your
> > output is going to be before it's produced, so this doesn't really
> > work.  Multiple files in the same directory achieves the same goal
> > much more elegantly, without exposing a bunch of internal details of
> > the filesystem to user space.
> >
> > Does that make sense?
> >
> >
> >
> > On Fri, Jul 2, 2010 at 9:26 AM, Segel, Mike <msegel@navteq.com> wrote:
> > > Actually they also listen here and this is a basic question...
> > >
> > > I'm not an expert, but how does having multiple threads really help
> this
> > problem?
> > >
> > > I'm assuming you're talking about a map/reduce job and not some
> specific
> > client code which is being run on a client outside of the
> cloud/cluster....
> > >
> > > I wasn't aware that you could easily synchronize threads running on
> > different JVMs. ;-)
> > >
> > > Your parallelism comes from multiple tasks running on different nodes
> > within the cloud. By default you get one map/reduce job per block. You
> can
> > write your own splitter to increase this and then get more parallelism.
> > >
> > > HTH
> > >
> > > -Mike
> > >
> > >
> > > -----Original Message-----
> > > From: Hemanth Yamijala [mailto:yhemanth@gmail.com]
> > > Sent: Friday, July 02, 2010 2:56 AM
> > > To: general@hadoop.apache.org
> > > Subject: Re: Why single thread for HDFS?
> > >
> > > Hi,
> > >
> > > Can you please post this on hdfs-dev@hadoop.apache.org ? I suspect the
> > > most qualified people to answer this question would all be on that
> > > list.
> > >
> > > Hemanth
> > >
> > > On Fri, Jul 2, 2010 at 11:43 AM, elton sky <eltonsky9404@gmail.com>
> > wrote:
> > >> I guess this question was igored, so I just post it again.
> > >>
> > >> From my understanding, HDFS uses a single thread to do read and write.
> > >> Since a file is composed of many blocks and each block is stored as a
> > file
> > >> in the underlying FS, we can do some parallelism on block base.
> > >> When read across multi-blocks, threads can be used to read all blocks.
> > When
> > >> write, we can calculate the offset of each block and write to all of
> > them
> > >> simultaneously.
> > >>
> > >> Is this right?
> > >>
> > >
> > >
> > > The information contained in this communication may be CONFIDENTIAL and
> > is intended only for the use of the recipient(s) named above.  If you are
> > not the intended recipient, you are hereby notified that any
> dissemination,
> > distribution, or copying of this communication, or any of its contents,
> is
> > strictly prohibited.  If you have received this communication in error,
> > please notify the sender and delete/destroy the original message and any
> > copy of it from your computer or paper files.
> > >
> >
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--001485e7e92a169a0c048aa6e67d--

From general-return-1750-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 05 21:22:22 2010
Return-Path: <general-return-1750-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 65360 invoked from network); 5 Jul 2010 21:22:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jul 2010 21:22:22 -0000
Received: (qmail 24264 invoked by uid 500); 5 Jul 2010 21:22:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24173 invoked by uid 500); 5 Jul 2010 21:22:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24165 invoked by uid 99); 5 Jul 2010 21:22:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jul 2010 21:22:20 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jul 2010 21:22:14 +0000
Received: by iwn37 with SMTP id 37so4167033iwn.35
        for <general@hadoop.apache.org>; Mon, 05 Jul 2010 14:20:52 -0700 (PDT)
Received: by 10.231.79.4 with SMTP id n4mr3966296ibk.16.1278364852316; Mon, 05 
	Jul 2010 14:20:52 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.161.72 with HTTP; Mon, 5 Jul 2010 14:20:32 -0700 (PDT)
In-Reply-To: <ADF94D8555C7A246B86A633685E0178AB8F2803C36@planck.ka.sara.nl>
References: <ADF94D8555C7A246B86A633685E0178AB8F2803C36@planck.ka.sara.nl>
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 5 Jul 2010 14:20:32 -0700
Message-ID: <AANLkTik_BPtK9Lf4q55a-Okcn6cdhC7ndsCI5DxT8H7I@mail.gmail.com>
Subject: Re: Hadoop versions & distributions
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64ba554ea6a13048aaa822d
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e64ba554ea6a13048aaa822d
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

On Mon, Jul 5, 2010 at 1:12 AM, Evert Lammerts <Evert.Lammerts@sara.nl>wrot=
e:

>  There are a number of different versions and distributions of Hadoop
> which, as far as I understand, all differ from each other. I know that in
> the 0.20-append branch, files in HDFS can be appended, and that the Y!
> distribution (0.20.S) implements security features through Kerberos. And
> then there are the 0.20.3 and 0.22.0 branches. And trunk of course, which=
 I
> guess is 0.20.2 nowadays? In addition to that there are distributions by
> Cloudera(CDH2 / 3beta) and IBM (IDAH).
>
>
>
> From my perspective, setting up a pilot cluster for a small number of use=
rs
> from different institutes, security (0.20.S) is very attractive =96 scien=
tists
> like the idea of shielding their data and logic from other users. But wha=
t
> will I miss if I choose Y!=92s distribution over all of these other optio=
ns?
>
>
Hi Evert,

Y!'s distribution does contain a good set of patches, and we at Cloudera ar=
e
always keeping track of the ydist git repository to incorporate those
changes into CDH. Currently, ydist contains the security patch series, but
doesn't include the recent append work. CDH3b2 includes the append work, bu=
t
not security as of yet -- we are currently integrating security and it
should be available in the next beta.

Aside from the specific patches included, it's worth noting that the Y! dis=
t
is a git repository, rather than a full binary-and-source distribution of
Hadoop and related tools. CDH includes not just the core hadoop components
but also integrates many other important ecosystem components including Pig=
,
Hive, Oozie, HBase, Zookeeper, Flume, etc.

Thanks
-Todd

--=20
Todd Lipcon
Software Engineer, Cloudera

--0016e64ba554ea6a13048aaa822d--

From general-return-1751-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 00:02:03 2010
Return-Path: <general-return-1751-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91096 invoked from network); 6 Jul 2010 00:02:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 00:02:02 -0000
Received: (qmail 96971 invoked by uid 500); 6 Jul 2010 00:02:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96897 invoked by uid 500); 6 Jul 2010 00:02:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96889 invoked by uid 99); 6 Jul 2010 00:02:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 00:02:01 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eltonsky9404@gmail.com designates 209.85.216.176 as permitted sender)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 00:01:54 +0000
Received: by qyk12 with SMTP id 12so2225544qyk.14
        for <general@hadoop.apache.org>; Mon, 05 Jul 2010 17:01:33 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=doxNBTENzw9XIiLIExL4kSNl6NOwqZKzN7JxZgyhFdw=;
        b=XltMEfhcqFwnQHPv6lWFb3OVREazlFRoOwumc+PedtTwO6JAlZKgjjas9RJlVk50VS
         5k1V3d6W3ebckA4v0yJjBg+3FD6DohM52lO7t715jHIaYixAxywAx6dgBcWk1xRyKym0
         7rv91qRTkvdTbaSR59othnhUN7s2+DGD7vrmE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=LA9FrY/x0dD4FCc1GwB2jXYdC3mJFkgPpbrEdVOokHirB//Tpt+3dRPS1k4dBqq2CZ
         9SZ0r1XJjy4zRblnDPocX7OLFsNwV82paWdupRJ7dNRySvJcOuQEGlM4a7BV4cuJ3IUv
         4iz6Ds9b2+/8Md4LbhoINVj66f+hQOg3JUtl8=
MIME-Version: 1.0
Received: by 10.224.75.149 with SMTP id y21mr1519575qaj.36.1278374491633; Mon, 
	05 Jul 2010 17:01:31 -0700 (PDT)
Received: by 10.224.89.18 with HTTP; Mon, 5 Jul 2010 17:01:31 -0700 (PDT)
In-Reply-To: <AANLkTikNr4C2mjzLSwxUFglxtbfEalCwQ8clyzCNl_8a@mail.gmail.com>
References: <AANLkTiljgFWtvH2dQHhB44-8mXQKlAhtOPp5N1K3tJ3S@mail.gmail.com>
	<AANLkTikCO_TnPI-xBt_QQljWkTBWJ4lq3nGhHPSEjYNr@mail.gmail.com>
	<ABC24175AFD3BE4DA15F4CD375ED413D06084AF156@hq-ex-mb02.ad.navteq.com>
	<AANLkTilk2wfZ10leX0iEeNwJozMDdfHNiQ09rEAnbPDd@mail.gmail.com>
	<AANLkTimmN23EV17S1g-2LITHQrasMmh6qbSj13CJyfb5@mail.gmail.com>
	<AANLkTikNr4C2mjzLSwxUFglxtbfEalCwQ8clyzCNl_8a@mail.gmail.com>
Date: Tue, 6 Jul 2010 10:01:31 +1000
Message-ID: <AANLkTinC804Q953e4_4x80U_3l36EUERH4UNOz1I30Ww@mail.gmail.com>
Subject: Re: Why single thread for HDFS?
From: elton sky <eltonsky9404@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485e988e876b81e048aacc19b
X-Virus-Checked: Checked by ClamAV on apache.org

--001485e988e876b81e048aacc19b
Content-Type: text/plain; charset=ISO-8859-1

>There's actually an open ticket somewhere to make distcp do this using the
>new concat() API in the NameNode.
Where can I find that "open ticket"?

>concat() allows several files to be combined into one file at the metadata
level, so long as a number of
>restrictions are met. The work hasn't been done yet, but the concat() call
is there and waiting for a user.
Well, this sounds good when you have many small files, you concat() them
into a big one. I am talking about split a big file into blocks and copy all
a few blocks in parallel.


On Tue, Jul 6, 2010 at 3:01 AM, Todd Lipcon <todd@cloudera.com> wrote:

> On Mon, Jul 5, 2010 at 5:08 AM, elton sky <eltonsky9404@gmail.com> wrote:
>
> > Segel, Jay
> > Thanks for reply!
> >
> > >Your parallelism comes from multiple tasks running on different nodes
> > within the cloud. By >default you get one map/reduce job per block. You
> can
> > write your own splitter to increase >this and then get more parallelism.
> > sounds like an elegant solution. We can modify the 'distcp', using a
> simple
> > MR job, make it based on block rather than file.
> >
>
> There's actually an open ticket somewhere to make distcp do this using the
> new concat() API in the NameNode. concat() allows several files to be
> combined into one file at the metadata level, so long as a number of
> restrictions are met. The work hasn't been done yet, but the concat() call
> is there and waiting for a user.
>
> -Todd
>
>
> >
> > >in practice, you very rarely know how big your output is going to be
> > before
> > it's produced, so >this doesn't really work
> > I think you got the point why Yahoo make this design descision.
> > Multithreading only applicable when you know the size of the file, like
> > copy
> > existing files, so you can split them and feed to different threads.
> >
> > On Sat, Jul 3, 2010 at 1:24 AM, Jay Booth <jaybooth@gmail.com> wrote:
> >
> > > Yeah, a good way to think of it is that parallelism is achieved at the
> > > application level.
> > >
> > > On the input side, you can process multiple files in parallel or one
> > > file in parallel by logically splitting and opening multiple readers
> > > of the same file at multiple points.  Each of these readers is single
> > > threaded, because, well, you're returning a stream of bytes in order.
> > > It's inherently serial.
> > >
> > > On the reduce side, multiple reduces run, writing to multiple files in
> > > the same directory.  Again, you can't really write to a single file in
> > > parallel effectively -- you can't write byte 26 before byte 25,
> > > because the file's not that long yet.
> > >
> > > Theoretically, maybe you could have all reduces write to the same file
> > > by allocating some amount of space ahead of time and writing to the
> > > blocks in parallel - in practice, you very rarely know how big your
> > > output is going to be before it's produced, so this doesn't really
> > > work.  Multiple files in the same directory achieves the same goal
> > > much more elegantly, without exposing a bunch of internal details of
> > > the filesystem to user space.
> > >
> > > Does that make sense?
> > >
> > >
> > >
> > > On Fri, Jul 2, 2010 at 9:26 AM, Segel, Mike <msegel@navteq.com> wrote:
> > > > Actually they also listen here and this is a basic question...
> > > >
> > > > I'm not an expert, but how does having multiple threads really help
> > this
> > > problem?
> > > >
> > > > I'm assuming you're talking about a map/reduce job and not some
> > specific
> > > client code which is being run on a client outside of the
> > cloud/cluster....
> > > >
> > > > I wasn't aware that you could easily synchronize threads running on
> > > different JVMs. ;-)
> > > >
> > > > Your parallelism comes from multiple tasks running on different nodes
> > > within the cloud. By default you get one map/reduce job per block. You
> > can
> > > write your own splitter to increase this and then get more parallelism.
> > > >
> > > > HTH
> > > >
> > > > -Mike
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Hemanth Yamijala [mailto:yhemanth@gmail.com]
> > > > Sent: Friday, July 02, 2010 2:56 AM
> > > > To: general@hadoop.apache.org
> > > > Subject: Re: Why single thread for HDFS?
> > > >
> > > > Hi,
> > > >
> > > > Can you please post this on hdfs-dev@hadoop.apache.org ? I suspect
> the
> > > > most qualified people to answer this question would all be on that
> > > > list.
> > > >
> > > > Hemanth
> > > >
> > > > On Fri, Jul 2, 2010 at 11:43 AM, elton sky <eltonsky9404@gmail.com>
> > > wrote:
> > > >> I guess this question was igored, so I just post it again.
> > > >>
> > > >> From my understanding, HDFS uses a single thread to do read and
> write.
> > > >> Since a file is composed of many blocks and each block is stored as
> a
> > > file
> > > >> in the underlying FS, we can do some parallelism on block base.
> > > >> When read across multi-blocks, threads can be used to read all
> blocks.
> > > When
> > > >> write, we can calculate the offset of each block and write to all of
> > > them
> > > >> simultaneously.
> > > >>
> > > >> Is this right?
> > > >>
> > > >
> > > >
> > > > The information contained in this communication may be CONFIDENTIAL
> and
> > > is intended only for the use of the recipient(s) named above.  If you
> are
> > > not the intended recipient, you are hereby notified that any
> > dissemination,
> > > distribution, or copying of this communication, or any of its contents,
> > is
> > > strictly prohibited.  If you have received this communication in error,
> > > please notify the sender and delete/destroy the original message and
> any
> > > copy of it from your computer or paper files.
> > > >
> > >
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

--001485e988e876b81e048aacc19b--

From general-return-1752-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 00:49:17 2010
Return-Path: <general-return-1752-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5382 invoked from network); 6 Jul 2010 00:49:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 00:49:17 -0000
Received: (qmail 12492 invoked by uid 500); 6 Jul 2010 00:49:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12416 invoked by uid 500); 6 Jul 2010 00:49:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12408 invoked by uid 99); 6 Jul 2010 00:49:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 00:49:15 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 00:49:09 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version;
  b=AuQgPFPH9Gg1vT1I7qvAhy9LNetKSSKWtP/ERIpQ6nlL2GW2YQGYHHXz
   LlN/jL+KGYFoyoLwCXH/3jAoBjFTUTi1CWKUf7ZW9LiMiUy8W1c38pMLi
   fi1+2j4P1WoAaGv;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1278377348; x=1309913348;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20Why=20single=20thread=20for=20HDFS?
   |Date:=20Tue,=206=20Jul=202010=2000:47:46=20+0000
   |Message-ID:=20<5C9D1E1E-8299-4E97-B552-2DF69F5231BE@link
   edin.com>|To:=20"<general@hadoop.apache.org>"=20<general@
   hadoop.apache.org>|MIME-Version:=201.0
   |Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<d15ed14e-2a2e-47c8-8ba5-ba90b779b198>
   |In-Reply-To:=20<AANLkTinC804Q953e4_4x80U_3l36EUERH4UNOz1
   I30Ww@mail.gmail.com>|References:=20<AANLkTiljgFWtvH2dQHh
   B44-8mXQKlAhtOPp5N1K3tJ3S@mail.gmail.com>=0D=0A=20<AANLkT
   ikCO_TnPI-xBt_QQljWkTBWJ4lq3nGhHPSEjYNr@mail.gmail.com>
   =0D=0A=20<ABC24175AFD3BE4DA15F4CD375ED413D06084AF156@hq-e
   x-mb02.ad.navteq.com>=0D=0A=20<AANLkTilk2wfZ10leX0iEeNwJo
   zMDdfHNiQ09rEAnbPDd@mail.gmail.com>=0D=0A=20<AANLkTimmN23
   EV17S1g-2LITHQrasMmh6qbSj13CJyfb5@mail.gmail.com>=0D=0A
   =20<AANLkTikNr4C2mjzLSwxUFglxtbfEalCwQ8clyzCNl_8a@mail.gm
   ail.com>=0D=0A=20<AANLkTinC804Q953e4_4x80U_3l36EUERH4UNOz
   1I30Ww@mail.gmail.com>;
  bh=knG8dbJh3kHNif0GAgf8Ny1Aq0Ofr2MURF0r4QuIXOQ=;
  b=UINv3QCQaXvDHSgk6mdDT6FDgII9P5B+DbWk7sXQQqid4XIo8mE04m+T
   Cub2DUnfUxuoXJvwcmwQeyYDUTVTe3f9Keydjc5biaoY5YM+2aO/CsePM
   cnUD9i5oDtBUVlj;
X-IronPort-AV: E=Sophos;i="4.53,543,1272870000"; 
   d="scan'208";a="13434079"
Received: from ESV4-EXC02.linkedin.biz ([fe80::4d74:48bd:e0bd:13ee]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi; Mon, 5 Jul 2010 17:47:47
 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: Why single thread for HDFS?
Thread-Topic: Why single thread for HDFS?
Thread-Index: AQHLGa3sMMIbVYChWkWbc/hs3B8qDZKdufgAgABcVoCAACEFAIAEgE6AgABR5wCAAHU7gIAADOwA
Date: Tue, 6 Jul 2010 00:47:46 +0000
Message-ID: <5C9D1E1E-8299-4E97-B552-2DF69F5231BE@linkedin.com>
References: <AANLkTiljgFWtvH2dQHhB44-8mXQKlAhtOPp5N1K3tJ3S@mail.gmail.com>
 <AANLkTikCO_TnPI-xBt_QQljWkTBWJ4lq3nGhHPSEjYNr@mail.gmail.com>
 <ABC24175AFD3BE4DA15F4CD375ED413D06084AF156@hq-ex-mb02.ad.navteq.com>
 <AANLkTilk2wfZ10leX0iEeNwJozMDdfHNiQ09rEAnbPDd@mail.gmail.com>
 <AANLkTimmN23EV17S1g-2LITHQrasMmh6qbSj13CJyfb5@mail.gmail.com>
 <AANLkTikNr4C2mjzLSwxUFglxtbfEalCwQ8clyzCNl_8a@mail.gmail.com>
 <AANLkTinC804Q953e4_4x80U_3l36EUERH4UNOz1I30Ww@mail.gmail.com>
In-Reply-To: <AANLkTinC804Q953e4_4x80U_3l36EUERH4UNOz1I30Ww@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Content-Type: text/plain; charset="us-ascii"
Content-ID: <d15ed14e-2a2e-47c8-8ba5-ba90b779b198>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


On Jul 5, 2010, at 5:01 PM, elton sky wrote:
> Well, this sounds good when you have many small files, you concat() them
> into a big one. I am talking about split a big file into blocks and copy =
all
> a few blocks in parallel.

Basically, your point is that hadoop dfs -cp is relatively slow and could b=
e made faster.  If HDFS had a more multi-threaded design, it would make cp =
operations faster. =20

This sounds like a particularly high cost for an operation that is rarely u=
tilized.  [This is much more interesting in a distcp context, but even then=
 not that great.  distcp in my experience is usually used to push a bunch o=
f files, so you get your parallelism at the file level.  Typically these ar=
e part files are usually the same approx. size.]



From general-return-1753-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 03:47:06 2010
Return-Path: <general-return-1753-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44168 invoked from network); 6 Jul 2010 03:47:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 03:47:05 -0000
Received: (qmail 83004 invoked by uid 500); 6 Jul 2010 03:47:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82680 invoked by uid 500); 6 Jul 2010 03:47:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82672 invoked by uid 99); 6 Jul 2010 03:47:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 03:47:01 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eltonsky9404@gmail.com designates 209.85.216.176 as permitted sender)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 03:46:55 +0000
Received: by qyk12 with SMTP id 12so2306081qyk.14
        for <general@hadoop.apache.org>; Mon, 05 Jul 2010 20:46:34 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=sH+JTXZZDaO7KuFJN4Fh72xYavW6uaud0luUXs4kOqc=;
        b=vZrLrfJ+5I/EpUNE5ndmMJKuqUgmqFaWiYxD0cEtSiTbEKXn2QEfV7rGI+8+VAaNaw
         uylcqhBv1wQpkhb9vR+8UAMjWgwX8BGykl56jv+WZzB7wUeyTO+zHw1L2VpcMqF1y98r
         I1ZWHaW7OWXozlwDnlzc98XITrZTWvqAtJiZ0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=F5Ztd+90PnS7TxbZyMGrQ6DQn+oDLmSKFZqfsOSe3boLcnOZjGwJ01gZwrEpo4qgxF
         r46xj47p/lVR6UoYzxQfsKGKxHnQbaBm8LgyzgG0HvEOD22/VD3JvenAGn3zjUOFV0ru
         Mt3gYpF+rIY1abvRKCbVb8Xg86kmni12+6q/M=
MIME-Version: 1.0
Received: by 10.224.2.211 with SMTP id 19mr2008832qak.103.1278387994376; Mon, 
	05 Jul 2010 20:46:34 -0700 (PDT)
Received: by 10.224.89.18 with HTTP; Mon, 5 Jul 2010 20:46:34 -0700 (PDT)
In-Reply-To: <5C9D1E1E-8299-4E97-B552-2DF69F5231BE@linkedin.com>
References: <AANLkTiljgFWtvH2dQHhB44-8mXQKlAhtOPp5N1K3tJ3S@mail.gmail.com>
	<AANLkTikCO_TnPI-xBt_QQljWkTBWJ4lq3nGhHPSEjYNr@mail.gmail.com>
	<ABC24175AFD3BE4DA15F4CD375ED413D06084AF156@hq-ex-mb02.ad.navteq.com>
	<AANLkTilk2wfZ10leX0iEeNwJozMDdfHNiQ09rEAnbPDd@mail.gmail.com>
	<AANLkTimmN23EV17S1g-2LITHQrasMmh6qbSj13CJyfb5@mail.gmail.com>
	<AANLkTikNr4C2mjzLSwxUFglxtbfEalCwQ8clyzCNl_8a@mail.gmail.com>
	<AANLkTinC804Q953e4_4x80U_3l36EUERH4UNOz1I30Ww@mail.gmail.com>
	<5C9D1E1E-8299-4E97-B552-2DF69F5231BE@linkedin.com>
Date: Tue, 6 Jul 2010 13:46:34 +1000
Message-ID: <AANLkTimp7y0EYCThQ4WTKx6lW6BGYgkHHEocI3EV94q_@mail.gmail.com>
Subject: Re: Why single thread for HDFS?
From: elton sky <eltonsky9404@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175cb83c4a3b6b048aafe623
X-Virus-Checked: Checked by ClamAV on apache.org

--0015175cb83c4a3b6b048aafe623
Content-Type: text/plain; charset=ISO-8859-1

>Basically, your point is that hadoop dfs -cp is relatively slow and could
be made faster.  If HDFS had a more multi-threaded >design, itwould make cp
operations faster.
What I mean is, if we have the size of a file we can parallel by calculating
blocks. Otherwise we couldn't.


On Tue, Jul 6, 2010 at 10:47 AM, Allen Wittenauer
<awittenauer@linkedin.com>wrote:

>
> On Jul 5, 2010, at 5:01 PM, elton sky wrote:
> > Well, this sounds good when you have many small files, you concat() them
> > into a big one. I am talking about split a big file into blocks and copy
> all
> > a few blocks in parallel.
>
> Basically, your point is that hadoop dfs -cp is relatively slow and could
> be made faster.  If HDFS had a more multi-threaded design, it would make cp
> operations faster.
>
> This sounds like a particularly high cost for an operation that is rarely
> utilized.  [This is much more interesting in a distcp context, but even then
> not that great.  distcp in my experience is usually used to push a bunch of
> files, so you get your parallelism at the file level.  Typically these are
> part files are usually the same approx. size.]
>
>
>

--0015175cb83c4a3b6b048aafe623--

From general-return-1754-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 04:41:26 2010
Return-Path: <general-return-1754-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53808 invoked from network); 6 Jul 2010 04:41:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 04:41:25 -0000
Received: (qmail 4158 invoked by uid 500); 6 Jul 2010 04:41:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3668 invoked by uid 500); 6 Jul 2010 04:41:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3660 invoked by uid 99); 6 Jul 2010 04:41:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 04:41:20 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yhemanth@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 04:41:13 +0000
Received: by pwj2 with SMTP id 2so623100pwj.35
        for <general@hadoop.apache.org>; Mon, 05 Jul 2010 21:40:49 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=7TxFa00BIZxyfpAeC1K2sP4jSWJ3GeTkfufiiSOkBkg=;
        b=rxlwJ2MqUKmAifc2iQgezz+uywqwkh3dCfjM403DtCLBOPu9H/HiYSPDtR90G2dEVX
         iO1xq920VOLgOT+zkO7Fk7lLgr8iqbkRCWsZyOHCW/TWNpnUChpNRcx+MjlJ2RBMl0wS
         SqtmbCrYmr8a1B2LcQaKlbAMYd4pBEv5/r7bY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=CWigFMd7DteoD706FGcn1LkLYtcYn2rPkFFu3pt2NE2c93ss/fskWylcL8wc6esvbm
         jAhZI0ovnR9GDgtZsU90+YYI6D+Lo9okEQ261jqCIUCIF6uIIhLmacE5qZbZEVWgdSpe
         44boSO0+TPjaB9yPLgiCG2KkOZdZfuc7ZEtaM=
MIME-Version: 1.0
Received: by 10.142.215.19 with SMTP id n19mr4576992wfg.258.1278391249325; 
	Mon, 05 Jul 2010 21:40:49 -0700 (PDT)
Received: by 10.142.188.19 with HTTP; Mon, 5 Jul 2010 21:40:49 -0700 (PDT)
In-Reply-To: <482872.98634.qm@web46316.mail.sp1.yahoo.com>
References: <482872.98634.qm@web46316.mail.sp1.yahoo.com>
Date: Tue, 6 Jul 2010 10:10:49 +0530
Message-ID: <AANLkTil1vxAd4VWjbDjIo-m5WflbhKVSsNFNmLcmExm6@mail.gmail.com>
Subject: Re: Partitioned Datasets Map/Reduce
From: Hemanth Yamijala <yhemanth@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

> I have written my custom partitioner for partitioning datasets. I want =
=A0to
> partition two datasets using the same partitioner and then in the =A0next
> mapreduce job, I want each mapper to handle the same partition from =A0th=
e two
> sources and perform some function such as joining etc. How I =A0can I ens=
ure that
> one mapper gets the split that corresponds to same =A0partition from both=
 the
> sources?
>

Not really an answer to your specific question, but have you taken a
look at Pig (http://hadoop.apache.org/pig) which is suitable for
operations like Joining data sets ?

From general-return-1755-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 07:31:38 2010
Return-Path: <general-return-1755-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98384 invoked from network); 6 Jul 2010 07:31:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 07:31:38 -0000
Received: (qmail 41569 invoked by uid 500); 6 Jul 2010 07:31:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 41181 invoked by uid 500); 6 Jul 2010 07:31:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 41173 invoked by uid 99); 6 Jul 2010 07:31:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 07:31:34 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of gautam.singaraju@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 07:31:27 +0000
Received: by vws19 with SMTP id 19so10155212vws.35
        for <general@hadoop.apache.org>; Tue, 06 Jul 2010 00:30:06 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=HTKyjgjVVnwBhmlnOPncktaELR+3zhA2/9l5JOL6caE=;
        b=F8G0N30dSvp6zH1Y3FdclMzxaFEba3mzrigsDU+6c6BmOFxrzj4aomhwnK9t+s+00z
         2fDlOAJsK1YvezrZefwcogslVYXzAAYsnaT/xNGSQ4xeDS8AHSwhl8stco1A2chjgQa0
         O466ghCP21aLWWeuu7+At+XjSCKdbigINjyQM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=A3paq3SOj2Em2EYgwe6k7DtuHZwjyVzV6bgYxLM2+Z++UfAztqekxMPx54FXQn6Gv8
         PTscKFGSEis/+3H00HIincXzQ0hqljp+4ytKoXEba4bQyITCfCsKaNeRHHwXhn9LCvP5
         x0BxXljIbJg/bj/oEbxqG/lRbIBpzw+1A2z2o=
MIME-Version: 1.0
Received: by 10.229.223.140 with SMTP id ik12mr2297681qcb.50.1278401406196; 
	Tue, 06 Jul 2010 00:30:06 -0700 (PDT)
Received: by 10.229.249.136 with HTTP; Tue, 6 Jul 2010 00:30:06 -0700 (PDT)
In-Reply-To: <AANLkTimp7y0EYCThQ4WTKx6lW6BGYgkHHEocI3EV94q_@mail.gmail.com>
References: <AANLkTiljgFWtvH2dQHhB44-8mXQKlAhtOPp5N1K3tJ3S@mail.gmail.com>
	<AANLkTikCO_TnPI-xBt_QQljWkTBWJ4lq3nGhHPSEjYNr@mail.gmail.com>
	<ABC24175AFD3BE4DA15F4CD375ED413D06084AF156@hq-ex-mb02.ad.navteq.com>
	<AANLkTilk2wfZ10leX0iEeNwJozMDdfHNiQ09rEAnbPDd@mail.gmail.com>
	<AANLkTimmN23EV17S1g-2LITHQrasMmh6qbSj13CJyfb5@mail.gmail.com>
	<AANLkTikNr4C2mjzLSwxUFglxtbfEalCwQ8clyzCNl_8a@mail.gmail.com>
	<AANLkTinC804Q953e4_4x80U_3l36EUERH4UNOz1I30Ww@mail.gmail.com>
	<5C9D1E1E-8299-4E97-B552-2DF69F5231BE@linkedin.com>
	<AANLkTimp7y0EYCThQ4WTKx6lW6BGYgkHHEocI3EV94q_@mail.gmail.com>
Date: Tue, 6 Jul 2010 00:30:06 -0700
Message-ID: <AANLkTinHouGalTJqqemJEb55fXme9qXaG0cg4rI0X7I-@mail.gmail.com>
Subject: Re: Why single thread for HDFS?
From: Gautam Singaraju <gautam.singaraju@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

To add to Jay Booth's points, adding multi-threaded capability to HDFS
will bring down the performance. Consider a production server where
4-5 jobs are running on a low-end commodity server. Currently, that is
4 threads reading and writing from the hard disk. Making it a
multi-threaded read and write will create many threads (Number of Jobs
* Default HDFS Block size * 1024 KB/ file system block sizes). For a
low-end hard disk with limited RPM cycles, a higher number of threads
will decrease the performance. As the number of disk access increase
from 1, the throughput will increase. But after 3-4 parallel disk
accesses, the performance will start to decrease. You can use
performance analytics tools (like IOMeter) to identify the *ideal*
number of parallel disk accesses for a specified hard-disk.

---
Gautam



On Mon, Jul 5, 2010 at 8:46 PM, elton sky <eltonsky9404@gmail.com> wrote:
>>Basically, your point is that hadoop dfs -cp is relatively slow and could
> be made faster. =A0If HDFS had a more multi-threaded >design, itwould mak=
e cp
> operations faster.
> What I mean is, if we have the size of a file we can parallel by calculat=
ing
> blocks. Otherwise we couldn't.
>
>
> On Tue, Jul 6, 2010 at 10:47 AM, Allen Wittenauer
> <awittenauer@linkedin.com>wrote:
>
>>
>> On Jul 5, 2010, at 5:01 PM, elton sky wrote:
>> > Well, this sounds good when you have many small files, you concat() th=
em
>> > into a big one. I am talking about split a big file into blocks and co=
py
>> all
>> > a few blocks in parallel.
>>
>> Basically, your point is that hadoop dfs -cp is relatively slow and coul=
d
>> be made faster. =A0If HDFS had a more multi-threaded design, it would ma=
ke cp
>> operations faster.
>>
>> This sounds like a particularly high cost for an operation that is rarel=
y
>> utilized. =A0[This is much more interesting in a distcp context, but eve=
n then
>> not that great. =A0distcp in my experience is usually used to push a bun=
ch of
>> files, so you get your parallelism at the file level. =A0Typically these=
 are
>> part files are usually the same approx. size.]
>>
>>
>>
>

From general-return-1756-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 08:50:51 2010
Return-Path: <general-return-1756-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16122 invoked from network); 6 Jul 2010 08:50:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 08:50:51 -0000
Received: (qmail 2339 invoked by uid 500); 6 Jul 2010 08:50:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1996 invoked by uid 500); 6 Jul 2010 08:50:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1983 invoked by uid 99); 6 Jul 2010 08:50:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 08:50:46 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [68.180.199.181] (HELO web46316.mail.sp1.yahoo.com) (68.180.199.181)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 06 Jul 2010 08:50:36 +0000
Received: (qmail 63183 invoked by uid 60001); 6 Jul 2010 08:50:14 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1278406214; bh=Vvz+ai4YJU0ZSe+dTL5QTx4XwGCU5YWCXcU1TGJbHPI=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=hzl1WcHEYUFBG7ph2MIcZwnjRUWB0goVY9iw+VUbGm+fjPtx416pW4MOqSlTkQZHekZv15lrDRc79my7wHDdCgA454jyWoof5yP8oBWDVFhQrLqQRdJyddJFDfOVohmpst9xvpHDZraD24VA7+xQLR/zvHQGRxDBfENIMCzqK38=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=kvBO+I0CuyxzpWgxiqAOfTf2nM1HdDJh/rMwKzojvdxd8UtN+Ztfafa1kfeHvu62S+1eSGWZF5JdlTAXE6LJt4FG6IYdQh16hzMARZ6cot67l8NG3N5E2DAfYQB30i/AH8Z9iYrA2VQMVwI85GHs6XVc5SpVNv7EDJiGRcj0z80=;
Message-ID: <614591.61443.qm@web46316.mail.sp1.yahoo.com>
X-YMail-OSG: a1bvpV4VM1lgtJkM0KlUBSddAhQOoEM5e.5gP0X_vQdpSyt
 bDIuxfLefqZFJIEcqhrYeq8_nCqHHeHaDVwICTi8gVo6C1reAuEA2WupNJlB
 hG6YVrEz9nebMTzs7E5ELm9AoPRnEARfrmGb_piGExJUB7Sm5.sBdWetn0u9
 c6HDqq5.efHfFJBCwGDdJ9IQTAi1_BfQdlD9y1Ub.d711UXKCUREbezjj4Q0
 3bLzxOLbUkauMMM8.gwXCvad38yyXt6SpiLD0nmXzZYLoEW5pyOtrCzBfzRj
 tMrM0AHfuWJD37pfWmLM219qdNegrM_Z3aQGhPf_izjaCDjZ8U7KxEFY1E96
 yLDG7G6.DEAc-
Received: from [188.74.76.201] by web46316.mail.sp1.yahoo.com via HTTP; Tue, 06 Jul 2010 01:50:14 PDT
X-Mailer: YahooMailRC/420.4 YahooMailWebService/0.8.104.274457
References: <482872.98634.qm@web46316.mail.sp1.yahoo.com> <AANLkTil1vxAd4VWjbDjIo-m5WflbhKVSsNFNmLcmExm6@mail.gmail.com>
Date: Tue, 6 Jul 2010 01:50:14 -0700 (PDT)
From: abc xyz <fabc_xyz111@yahoo.com>
Subject: Re: Partitioned Datasets Map/Reduce
To: general@hadoop.apache.org
In-Reply-To: <AANLkTil1vxAd4VWjbDjIo-m5WflbhKVSsNFNmLcmExm6@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-648482194-1278406214=:61443"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-648482194-1278406214=:61443
Content-Type: text/plain; charset=us-ascii



well, I want to do some experimentation with hadoop. I need to partition two 
datasets using same partitioning function and then in the next job, take the 
same partition from both datasets and apply some operation in the mapper. But 
how to ensure to get the same partition from both sources in one mapper??



________________________________
From: Hemanth Yamijala <yhemanth@gmail.com>
To: general@hadoop.apache.org
Sent: Tue, July 6, 2010 5:40:49 AM
Subject: Re: Partitioned Datasets Map/Reduce

Hi,

> I have written my custom partitioner for partitioning datasets. I want  to
> partition two datasets using the same partitioner and then in the  next
> mapreduce job, I want each mapper to handle the same partition from  the two
> sources and perform some function such as joining etc. How I  can I ensure 
that
> one mapper gets the split that corresponds to same  partition from both the
> sources?
>

Not really an answer to your specific question, but have you taken a
look at Pig (http://hadoop.apache.org/pig) which is suitable for
operations like Joining data sets ?



      
--0-648482194-1278406214=:61443--

From general-return-1757-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 08:54:19 2010
Return-Path: <general-return-1757-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16600 invoked from network); 6 Jul 2010 08:54:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 08:54:18 -0000
Received: (qmail 6319 invoked by uid 500); 6 Jul 2010 08:54:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6057 invoked by uid 500); 6 Jul 2010 08:54:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6048 invoked by uid 99); 6 Jul 2010 08:54:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 08:54:15 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of zhangguoping96@gmail.com designates 209.85.210.48 as permitted sender)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 08:54:07 +0000
Received: by pzk36 with SMTP id 36so545359pzk.35
        for <general@hadoop.apache.org>; Tue, 06 Jul 2010 01:53:46 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=bd5gmHrJQrrxmLTLmPkyj16YgMW4xl9/v5Gq5PFilkA=;
        b=yBQkYJh05we6Tbk4+lmKrmi0eKq/gmVSlwBGj7nkjQyyVR0KSUGGs6sgLlGFhKOiTy
         ofIYUWtnPrKjSzC9Oq6CcQKhFVdcbuH/frYcJm0p6xxmywquXTChbDNKAWembaj763xB
         t0CZrUJ1tFjk1ogABobykhsVdDuuk0K4MvG/s=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=j9hA3sJm6IT/bQsW8yY9lZfstOtaWVIcJPJn1PpZ9SFCmyAAopXZSFbhuZ5JO8qkJK
         VIkOI2IHyMdRWdguLiVcu1U2/kA0cLzgR8aLfx1W0GVqrJXMgyVXRb7UUawRkLM3rVVl
         dUosPF+I1wIvomYkUinP+Qw5uEEg8lQJYLi/Y=
MIME-Version: 1.0
Received: by 10.114.131.13 with SMTP id e13mr4631831wad.201.1278406425826; 
	Tue, 06 Jul 2010 01:53:45 -0700 (PDT)
Received: by 10.42.4.213 with HTTP; Tue, 6 Jul 2010 01:53:45 -0700 (PDT)
Date: Tue, 6 Jul 2010 16:53:45 +0800
Message-ID: <AANLkTiltXwXZrgCNFugScg8Jgze-nfc-dwwF0xvH3eYt@mail.gmail.com>
Subject: Distributed Cache
From: zhangguoping zhangguoping <zhangguoping96@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636417fb5e3d66d048ab43039
X-Virus-Checked: Checked by ClamAV on apache.org

--001636417fb5e3d66d048ab43039
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

>From the book: "Hadoop The definitive guide" -- P242
>>
When you launch a job, Hadoop copies the files specified by the -files and
-archives options to the jobtracker=92s filesystem (normally HDFS). Then,
before a task
is run, the tasktracker copies the files from the jobtracker=92s filesystem=
 to
a local disk=97
the cache=97so the task can access the files.
>>

I wonder why hadoop wants to copy the files to jobtracker's filesystem.
Since it is already in HDFS, it should be available to tasks.
Any considerations?

--001636417fb5e3d66d048ab43039--

From general-return-1758-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 09:13:08 2010
Return-Path: <general-return-1758-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23479 invoked from network); 6 Jul 2010 09:13:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 09:13:08 -0000
Received: (qmail 30471 invoked by uid 500); 6 Jul 2010 09:13:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30227 invoked by uid 500); 6 Jul 2010 09:13:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30219 invoked by uid 99); 6 Jul 2010 09:13:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 09:13:03 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yhemanth@gmail.com designates 74.125.83.176 as permitted sender)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 09:12:55 +0000
Received: by pvc21 with SMTP id 21so643264pvc.35
        for <general@hadoop.apache.org>; Tue, 06 Jul 2010 02:12:34 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=VpwNb6eyEP2O+GXnbaQjVPTS+zxn38gU1vYWDODBb9Y=;
        b=Pw3oGWrprgg/Xv5DJ9XUVQEO87E6mI5ZbkyoQs4Mj46O1eoKv2DKhxXY/FGG9yCTFM
         Y5aaLF114zi6Kyk+kDCb0FASV6lGSxv04AWBqhzlPZVRV2/8HvxSfDbJZTq2Xwj6r7oq
         vt26ePFkCki6NCsciabnWLL0wqL99jUubLGGM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=w/sjpHVmuSZCFxoFHlnh3i3ri1pxxbgDYUdERMIRpo/0HZvhKQcI+X+6ZL4mwp3Mnb
         1ZekBvOQlGes2GrKI3UFGed6mV+fC7MTWHJa3v23FtpBEF+RHgzB9zdpA5WkMrvCou7J
         90WO6rZ+Y/0wOEWaJ59UOiXb57aPa6B3w16No=
MIME-Version: 1.0
Received: by 10.142.232.13 with SMTP id e13mr4905500wfh.196.1278407553735; 
	Tue, 06 Jul 2010 02:12:33 -0700 (PDT)
Received: by 10.142.188.19 with HTTP; Tue, 6 Jul 2010 02:12:33 -0700 (PDT)
In-Reply-To: <AANLkTiltXwXZrgCNFugScg8Jgze-nfc-dwwF0xvH3eYt@mail.gmail.com>
References: <AANLkTiltXwXZrgCNFugScg8Jgze-nfc-dwwF0xvH3eYt@mail.gmail.com>
Date: Tue, 6 Jul 2010 14:42:33 +0530
Message-ID: <AANLkTimw6tQwcepPsyR6cgcXzjxc5HuXiT7AgVk8BX0B@mail.gmail.com>
Subject: Re: Distributed Cache
From: Hemanth Yamijala <yhemanth@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

> From the book: "Hadoop The definitive guide" -- P242
>>>
> When you launch a job, Hadoop copies the files specified by the -files an=
d
> -archives options to the jobtracker=92s filesystem (normally HDFS). Then,
> before a task
> is run, the tasktracker copies the files from the jobtracker=92s filesyst=
em to
> a local disk=97
> the cache=97so the task can access the files.
>>>
>
> I wonder why hadoop wants to copy the files to jobtracker's filesystem.
> Since it is already in HDFS, it should be available to tasks.
> Any considerations?

Unlike input data files for M/R tasks, -files and -archives are
options to copy additional files (like any configuration files etc)
that all the M/R tasks might need when running. Such files typically
need to be transferred from the local machine where the job is
launched to the cluster nodes where the tasks run. Think of them as
convenient shortcuts to distribute files to all the tasks.

Makes sense ?

Thanks
Hemanth

From general-return-1759-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 09:19:14 2010
Return-Path: <general-return-1759-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24659 invoked from network); 6 Jul 2010 09:19:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 09:19:13 -0000
Received: (qmail 37739 invoked by uid 500); 6 Jul 2010 09:19:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37519 invoked by uid 500); 6 Jul 2010 09:19:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37511 invoked by uid 99); 6 Jul 2010 09:19:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 09:19:10 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 09:19:02 +0000
Received: from EGL-EX07CAS01.ds.corp.yahoo.com (egl-ex07cas01.eglbp.corp.yahoo.com [203.83.248.208])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o669H49A049106
	for <general@hadoop.apache.org>; Tue, 6 Jul 2010 02:17:05 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:in-reply-to:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=O7XUMuVsm9WIHRHZoARJNSokCNIsm7lwHaObboqNMP+x9nB7s67XtcLivtXoD7hm
Received: from EGL-EX07VS01.ds.corp.yahoo.com ([203.83.248.205]) by
 EGL-EX07CAS01.ds.corp.yahoo.com ([203.83.248.215]) with mapi; Tue, 6 Jul 2010
 14:47:03 +0530
From: Amareshwari Sri Ramadasu <amarsri@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 6 Jul 2010 14:47:02 +0530
Subject: Re: Distributed Cache
Thread-Topic: Distributed Cache
Thread-Index: Acsc6OEaboUnXHGRTYir40pJ0i630QAAx04v
Message-ID: <C858F266.19820%amarsri@yahoo-inc.com>
In-Reply-To: <AANLkTiltXwXZrgCNFugScg8Jgze-nfc-dwwF0xvH3eYt@mail.gmail.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C858F26619820amarsriyahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C858F26619820amarsriyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


If they are already in JT's FS, it does not copy them. It copies them only =
if they are on local FS or some other FS.

On 7/6/10 2:23 PM, "zhangguoping zhangguoping" <zhangguoping96@gmail.com> w=
rote:


I wonder why hadoop wants to copy the files to jobtracker's filesystem.
Since it is already in HDFS, it should be available to tasks.
Any considerations?


--_000_C858F26619820amarsriyahooinccom_--

From general-return-1760-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 09:45:15 2010
Return-Path: <general-return-1760-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31516 invoked from network); 6 Jul 2010 09:45:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 09:45:15 -0000
Received: (qmail 64107 invoked by uid 500); 6 Jul 2010 09:45:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63402 invoked by uid 500); 6 Jul 2010 09:45:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63385 invoked by uid 99); 6 Jul 2010 09:45:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 09:45:09 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [87.248.114.165] (HELO web24101.mail.ird.yahoo.com) (87.248.114.165)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 06 Jul 2010 09:45:00 +0000
Received: (qmail 77114 invoked by uid 60001); 6 Jul 2010 09:43:40 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1278409420; bh=AtDlA2KF2ZbcE9isGlTF2hi7NlAy5Lxy0bKqnQTk2aI=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=P2sX5TGwKrSlZ2PN+ucfatogvOcT9fymQAMDGYadadmfI33G7T3rjim3OdH1rMt18ZuG8LcgnRZx81R8y3G/VEmJ1lF1MgLBvpCPq8T3rVoryJIvQEiFwS8TyTPIPqvGuXRpQzxAZ03+0/EckLXmnO3yCCh0cC6R0qPoSayeDs4=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=XG9tNkA2OnmJrNQ6MZ7S5IcMbaJ4daNkJ6fdiHqWNMwtxoXjrt2qvZDy1ICCqJLD5O805HahwFZxAyhvFfr2K27ma5B8OctDmoUDk9IvLCj+W6LUuPpen08DFcQd2Z+X+aQ8qNHcz2IOIEztmuV4aGWAHsIbGpsVi11k/0zH+vo=;
Message-ID: <24760.76231.qm@web24101.mail.ird.yahoo.com>
X-YMail-OSG: PB1DS2QVM1mWZRRgo9RaAIt37qcfEgLN_j4Q3oMaxe8jreF
 .X7alC61G5rDDcJMdD9Gdi_v1kYBs0CFNI4EIYTEaKgIW.v_15T03RaRoOWx
 Qkn8T452OC3wKItg9dXIS5Jm3nS5tSd4ecYBfWld.wZtzmSH1V3qUpKVNAub
 JIZXmu.yJY2r.vPS_FwDO0RYLfq3dSMiTwnfg
Received: from [188.74.76.201] by web24101.mail.ird.yahoo.com via HTTP; Tue, 06 Jul 2010 09:43:39 GMT
X-Mailer: YahooMailRC/397.8 YahooMailWebService/0.8.104.274457
Date: Tue, 6 Jul 2010 09:43:39 +0000 (GMT)
From: Denim Live <denim.live@yahoo.com>
Subject: how to find the id of each map task?
To: general@hadoop.apache.org, mapreduce-user@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-300312025-1278409419=:76231"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-300312025-1278409419=:76231
Content-Type: text/plain; charset=us-ascii

Hello
I want to get the id of each mapper and reducer task because I want to tag the output of these mappers and reducers according to the mapper and reducer id. How can I retrieve the ids of each?
Thanks


      
--0-300312025-1278409419=:76231--

From general-return-1761-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 12:40:03 2010
Return-Path: <general-return-1761-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81765 invoked from network); 6 Jul 2010 12:40:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 12:40:02 -0000
Received: (qmail 1927 invoked by uid 500); 6 Jul 2010 12:40:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1607 invoked by uid 500); 6 Jul 2010 12:39:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1599 invoked by uid 99); 6 Jul 2010 12:39:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 12:39:57 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=RCVD_ILLEGAL_IP,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [192.5.164.99] (HELO camv02-relay2.casc.gd-ais.com) (192.5.164.99)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 12:39:49 +0000
Received: from ([10.73.100.22])
	by camv02-relay2.casc.gd-ais.com with SMTP  id 5203374.39508376;
	Tue, 06 Jul 2010 05:38:09 -0700
Received: from eadc01-cahprd01.ad.gd-ais.com ([10.120.80.11]) by camv02-fes01.ad.gd-ais.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Tue, 6 Jul 2010 05:38:09 -0700
Received: from EADC01-MABPRD01.ad.gd-ais.com ([169.254.1.214]) by
 eadc01-cahprd01.ad.gd-ais.com ([10.120.80.11]) with mapi; Tue, 6 Jul 2010
 07:38:08 -0500
From: "Kilbride, James P." <James.Kilbride@gd-ais.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 6 Jul 2010 07:38:07 -0500
Subject: MapReduce HBASE examples
Thread-Topic: MapReduce HBASE examples
Thread-Index: AcsdCBYXCcXHK05oQgyZi/cl6sZ8dA==
Message-ID: <035C7FA04C0BF64B8143C76523E7ED742FCFEEA6CD@EADC01-MABPRD01.ad.gd-ais.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginalArrivalTime: 06 Jul 2010 12:38:09.0131 (UTC) FILETIME=[16E60BB0:01CB1D08]
X-Virus-Checked: Checked by ClamAV on apache.org

All,

The examples in the hbase examples, and on the hadoop wiki all reference th=
e deprecated interfaces of the mapred package. Are there any examples of ho=
w to use hbase as the input for a mapreduce job, that uses the mapreduce pa=
ckage instead? I'm looking to set up a job which will read from a hbase tab=
le based on a row value passed into the job, and which starts the map the r=
ow values(as the value key) and the column names(or value) as the map value=
s.

James Kilbride

From general-return-1762-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 14:07:01 2010
Return-Path: <general-return-1762-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 7544 invoked from network); 6 Jul 2010 14:07:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 14:07:00 -0000
Received: (qmail 4418 invoked by uid 500); 6 Jul 2010 14:07:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4150 invoked by uid 500); 6 Jul 2010 14:06:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4142 invoked by uid 99); 6 Jul 2010 14:06:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 14:06:57 +0000
X-ASF-Spam-Status: No, hits=4.7 required=10.0
	tests=FREEMAIL_FROM,FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of michael_segel@hotmail.com designates 65.55.34.25 as permitted sender)
Received: from [65.55.34.25] (HELO col0-omc1-s15.col0.hotmail.com) (65.55.34.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 14:06:50 +0000
Received: from COL117-W14 ([65.55.34.9]) by col0-omc1-s15.col0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Tue, 6 Jul 2010 07:06:07 -0700
Message-ID: <COL117-W1485F3B9503653BB189B938FB20@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_ecc786b2-3d21-445b-8cea-4116f6ae1387_"
X-Originating-IP: [173.15.87.33]
From: Michael Segel <michael_segel@hotmail.com>
To: <general@hadoop.apache.org>
Subject: RE: Why single thread for HDFS?
Date: Tue, 6 Jul 2010 09:06:07 -0500
Importance: Normal
In-Reply-To: <AANLkTinHouGalTJqqemJEb55fXme9qXaG0cg4rI0X7I-@mail.gmail.com>
References:
 <AANLkTiljgFWtvH2dQHhB44-8mXQKlAhtOPp5N1K3tJ3S@mail.gmail.com>,<AANLkTikCO_TnPI-xBt_QQljWkTBWJ4lq3nGhHPSEjYNr@mail.gmail.com>,<ABC24175AFD3BE4DA15F4CD375ED413D06084AF156@hq-ex-mb02.ad.navteq.com>,<AANLkTilk2wfZ10leX0iEeNwJozMDdfHNiQ09rEAnbPDd@mail.gmail.com>,<AANLkTimmN23EV17S1g-2LITHQrasMmh6qbSj13CJyfb5@mail.gmail.com>,<AANLkTikNr4C2mjzLSwxUFglxtbfEalCwQ8clyzCNl_8a@mail.gmail.com>,<AANLkTinC804Q953e4_4x80U_3l36EUERH4UNOz1I30Ww@mail.gmail.com>,<5C9D1E1E-8299-4E97-B552-2DF69F5231BE@linkedin.com>,<AANLkTimp7y0EYCThQ4WTKx6lW6BGYgkHHEocI3EV94q_@mail.gmail.com>,<AANLkTinHouGalTJqqemJEb55fXme9qXaG0cg4rI0X7I-@mail.gmail.com>
MIME-Version: 1.0
X-OriginalArrivalTime: 06 Jul 2010 14:06:07.0210 (UTC) FILETIME=[60E0FCA0:01CB1D14]
X-Virus-Checked: Checked by ClamAV on apache.org

--_ecc786b2-3d21-445b-8cea-4116f6ae1387_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


Uhm...

That's not really true. It gets a bit more complicated than that.

If you're talking about M/R jobs=2C you don't want to do threads in your ma=
p() routine=2C while this is possible=2C its going to be really hard to jus=
tify the extra parallelism along with the need to wait for all of the threa=
ds to complete before you can end the map() method.=20

If you're talking about a way to copy files from one cluster to another... =
in hadoop... you can find out the block lists that make up the file. As lon=
g as the file is static=2C meaning no one is writing/spliting/compacting th=
e file=2C you could copy it. Here being multi threaded could work.=20
You'd have one thread per block that will read from one machine=2C and then=
 write directly to the other. Of course you'll need to figure out where to =
write the block=2C or rather tie in to HDFS.

This is more complex than a M/R job but will work.

If you're reading from the cloud and then writing to the UNIX file system=
=2C you want to write the blocks in serial order. (KISS).

JMHO

-Mike


> Date: Tue=2C 6 Jul 2010 00:30:06 -0700
> Subject: Re: Why single thread for HDFS?
> From: gautam.singaraju@gmail.com
> To: general@hadoop.apache.org
>=20
> To add to Jay Booth's points=2C adding multi-threaded capability to HDFS
> will bring down the performance. Consider a production server where
> 4-5 jobs are running on a low-end commodity server. Currently=2C that is
> 4 threads reading and writing from the hard disk. Making it a
> multi-threaded read and write will create many threads (Number of Jobs
> * Default HDFS Block size * 1024 KB/ file system block sizes). For a
> low-end hard disk with limited RPM cycles=2C a higher number of threads
> will decrease the performance. As the number of disk access increase
> from 1=2C the throughput will increase. But after 3-4 parallel disk
> accesses=2C the performance will start to decrease. You can use
> performance analytics tools (like IOMeter) to identify the *ideal*
> number of parallel disk accesses for a specified hard-disk.
>=20
> ---
> Gautam
>=20
>=20
>=20
> On Mon=2C Jul 5=2C 2010 at 8:46 PM=2C elton sky <eltonsky9404@gmail.com> =
wrote:
> >>Basically=2C your point is that hadoop dfs -cp is relatively slow and c=
ould
> > be made faster.  If HDFS had a more multi-threaded >design=2C itwould m=
ake cp
> > operations faster.
> > What I mean is=2C if we have the size of a file we can parallel by calc=
ulating
> > blocks. Otherwise we couldn't.
> >
> >
> > On Tue=2C Jul 6=2C 2010 at 10:47 AM=2C Allen Wittenauer
> > <awittenauer@linkedin.com>wrote:
> >
> >>
> >> On Jul 5=2C 2010=2C at 5:01 PM=2C elton sky wrote:
> >> > Well=2C this sounds good when you have many small files=2C you conca=
t() them
> >> > into a big one. I am talking about split a big file into blocks and =
copy
> >> all
> >> > a few blocks in parallel.
> >>
> >> Basically=2C your point is that hadoop dfs -cp is relatively slow and =
could
> >> be made faster.  If HDFS had a more multi-threaded design=2C it would =
make cp
> >> operations faster.
> >>
> >> This sounds like a particularly high cost for an operation that is rar=
ely
> >> utilized.  [This is much more interesting in a distcp context=2C but e=
ven then
> >> not that great.  distcp in my experience is usually used to push a bun=
ch of
> >> files=2C so you get your parallelism at the file level.  Typically the=
se are
> >> part files are usually the same approx. size.]
> >>
> >>
> >>
> >
 		 	   		 =20
_________________________________________________________________
Hotmail has tools for the New Busy. Search=2C chat and e-mail from your inb=
ox.
http://www.windowslive.com/campaign/thenewbusy?ocid=3DPID28326::T:WLMTAGL:O=
N:WL:en-US:WM_HMP:042010_1=

--_ecc786b2-3d21-445b-8cea-4116f6ae1387_--

From general-return-1763-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 14:11:40 2010
Return-Path: <general-return-1763-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 8464 invoked from network); 6 Jul 2010 14:11:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 14:11:40 -0000
Received: (qmail 15278 invoked by uid 500); 6 Jul 2010 14:11:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15184 invoked by uid 500); 6 Jul 2010 14:11:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15176 invoked by uid 99); 6 Jul 2010 14:11:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 14:11:37 +0000
X-ASF-Spam-Status: No, hits=4.7 required=10.0
	tests=FREEMAIL_FROM,FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of michael_segel@hotmail.com designates 65.55.34.17 as permitted sender)
Received: from [65.55.34.17] (HELO col0-omc1-s7.col0.hotmail.com) (65.55.34.17)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 14:11:29 +0000
Received: from COL117-W33 ([65.55.34.8]) by col0-omc1-s7.col0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Tue, 6 Jul 2010 07:10:46 -0700
Message-ID: <COL117-W33C4D11B7A0F2EBA9E9A1E8FB20@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_808c5ad0-cf97-4c6c-ad9f-170321d51d62_"
X-Originating-IP: [173.15.87.33]
From: Michael Segel <michael_segel@hotmail.com>
To: <general@hadoop.apache.org>
Subject: RE: Why single thread for HDFS?
Date: Tue, 6 Jul 2010 09:10:46 -0500
Importance: Normal
In-Reply-To: <AANLkTimp7y0EYCThQ4WTKx6lW6BGYgkHHEocI3EV94q_@mail.gmail.com>
References:
 <AANLkTiljgFWtvH2dQHhB44-8mXQKlAhtOPp5N1K3tJ3S@mail.gmail.com>,<AANLkTikCO_TnPI-xBt_QQljWkTBWJ4lq3nGhHPSEjYNr@mail.gmail.com>,<ABC24175AFD3BE4DA15F4CD375ED413D06084AF156@hq-ex-mb02.ad.navteq.com>,<AANLkTilk2wfZ10leX0iEeNwJozMDdfHNiQ09rEAnbPDd@mail.gmail.com>,<AANLkTimmN23EV17S1g-2LITHQrasMmh6qbSj13CJyfb5@mail.gmail.com>,<AANLkTikNr4C2mjzLSwxUFglxtbfEalCwQ8clyzCNl_8a@mail.gmail.com>,<AANLkTinC804Q953e4_4x80U_3l36EUERH4UNOz1I30Ww@mail.gmail.com>,<5C9D1E1E-8299-4E97-B552-2DF69F5231BE@linkedin.com>,<AANLkTimp7y0EYCThQ4WTKx6lW6BGYgkHHEocI3EV94q_@mail.gmail.com>
MIME-Version: 1.0
X-OriginalArrivalTime: 06 Jul 2010 14:10:46.0909 (UTC) FILETIME=[0797AAD0:01CB1D15]
X-Virus-Checked: Checked by ClamAV on apache.org

--_808c5ad0-cf97-4c6c-ad9f-170321d51d62_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable



If all you want to do is to have a faster -cp option=2C then if you know yo=
ur intial block list and location=2C you need to generate the target bloc l=
ist and then create a single thread per block and process each block in a s=
eparate thread.

You don't need to use the local disk and just read/write each block in 'pag=
ed' increments. (pages as in 4/16/32/64K page sizes.)=20
(This removes the i/o argument raised by another poster.)

This may be faster than the current process.

HTH

-Mike

> Date: Tue=2C 6 Jul 2010 13:46:34 +1000
> Subject: Re: Why single thread for HDFS?
> From: eltonsky9404@gmail.com
> To: general@hadoop.apache.org
>=20
> >Basically=2C your point is that hadoop dfs -cp is relatively slow and co=
uld
> be made faster.  If HDFS had a more multi-threaded >design=2C itwould mak=
e cp
> operations faster.
> What I mean is=2C if we have the size of a file we can parallel by calcul=
ating
> blocks. Otherwise we couldn't.
>=20
>=20
> On Tue=2C Jul 6=2C 2010 at 10:47 AM=2C Allen Wittenauer
> <awittenauer@linkedin.com>wrote:
>=20
> >
> > On Jul 5=2C 2010=2C at 5:01 PM=2C elton sky wrote:
> > > Well=2C this sounds good when you have many small files=2C you concat=
() them
> > > into a big one. I am talking about split a big file into blocks and c=
opy
> > all
> > > a few blocks in parallel.
> >
> > Basically=2C your point is that hadoop dfs -cp is relatively slow and c=
ould
> > be made faster.  If HDFS had a more multi-threaded design=2C it would m=
ake cp
> > operations faster.
> >
> > This sounds like a particularly high cost for an operation that is rare=
ly
> > utilized.  [This is much more interesting in a distcp context=2C but ev=
en then
> > not that great.  distcp in my experience is usually used to push a bunc=
h of
> > files=2C so you get your parallelism at the file level.  Typically thes=
e are
> > part files are usually the same approx. size.]
> >
> >
> >
 		 	   		 =20
_________________________________________________________________
Hotmail is redefining busy with tools for the New Busy. Get more from your =
inbox.
http://www.windowslive.com/campaign/thenewbusy?ocid=3DPID28326::T:WLMTAGL:O=
N:WL:en-US:WM_HMP:042010_2=

--_808c5ad0-cf97-4c6c-ad9f-170321d51d62_--

From general-return-1764-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 15:31:26 2010
Return-Path: <general-return-1764-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31685 invoked from network); 6 Jul 2010 15:31:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 15:31:25 -0000
Received: (qmail 28078 invoked by uid 500); 6 Jul 2010 15:31:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27919 invoked by uid 500); 6 Jul 2010 15:31:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27906 invoked by uid 99); 6 Jul 2010 15:31:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 15:31:23 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 15:31:13 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id B8F981BA743
	for <general@hadoop.apache.org>; Tue,  6 Jul 2010 16:30:52 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id q3AaZm7jrrmg for <general@hadoop.apache.org>;
	Tue,  6 Jul 2010 16:30:52 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by colossus.hpl.hp.com (Postfix) with ESMTPS id 3E9371BA732
	for <general@hadoop.apache.org>; Tue,  6 Jul 2010 16:30:51 +0100 (BST)
MailScanner-NULL-Check: 1279035039.80413@Q0yHY460Y2nBaKfZqrqJ0Q
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o66FUcgj025309
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Tue, 6 Jul 2010 16:30:39 +0100 (BST)
Message-ID: <4C334C20.9010105@apache.org>
Date: Tue, 06 Jul 2010 16:30:40 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100228)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Why single thread for HDFS?
References: <AANLkTiljgFWtvH2dQHhB44-8mXQKlAhtOPp5N1K3tJ3S@mail.gmail.com>,<AANLkTikCO_TnPI-xBt_QQljWkTBWJ4lq3nGhHPSEjYNr@mail.gmail.com>,<ABC24175AFD3BE4DA15F4CD375ED413D06084AF156@hq-ex-mb02.ad.navteq.com>,<AANLkTilk2wfZ10leX0iEeNwJozMDdfHNiQ09rEAnbPDd@mail.gmail.com>,<AANLkTimmN23EV17S1g-2LITHQrasMmh6qbSj13CJyfb5@mail.gmail.com>,<AANLkTikNr4C2mjzLSwxUFglxtbfEalCwQ8clyzCNl_8a@mail.gmail.com>,<AANLkTinC804Q953e4_4x80U_3l36EUERH4UNOz1I30Ww@mail.gmail.com>,<5C9D1E1E-8299-4E97-B552-2DF69F5231BE@linkedin.com>,<AANLkTimp7y0EYCThQ4WTKx6lW6BGYgkHHEocI3EV94q_@mail.gmail.com>,<AANLkTinHouGalTJqqemJEb55fXme9qXaG0cg4rI0X7I-@mail.gmail.com> <COL117-W1485F3B9503653BB189B938FB20@phx.gbl>
In-Reply-To: <COL117-W1485F3B9503653BB189B938FB20@phx.gbl>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o66FUcgj025309
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

Michael Segel wrote:
> Uhm...
> 
> That's not really true. It gets a bit more complicated than that.
> 
> If you're talking about M/R jobs, you don't want to do threads in your map() routine, while this is possible, its going to be really hard to justify the extra parallelism along with the need to wait for all of the threads to complete before you can end the map() method. 
> 
> If you're talking about a way to copy files from one cluster to another... in hadoop... you can find out the block lists that make up the file. As long as the file is static, meaning no one is writing/spliting/compacting the file, you could copy it. Here being multi threaded could work. 
> You'd have one thread per block that will read from one machine, and then write directly to the other. Of course you'll need to figure out where to write the block, or rather tie in to HDFS.

There's a paper by Russ Perry using HDFS as a filestore for raster 
processing, where he modified DfsClient to get all the locations of a 
file, and let the caller decide where to read blocks from.

http://www.hpl.hp.com/techreports/2009/HPL-2009-345.html

the advantage of this is that the caller can do the striping across 
machines, keep every server busy by asking for files from each of them. 
Of course, this ignores the trend to many-HDD servers; DfsClient can't 
currently see which physical disk a file is on, which you'd need if the 
client wanted to keep every disk on every server busy during a big read

From general-return-1765-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 16:11:11 2010
Return-Path: <general-return-1765-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45403 invoked from network); 6 Jul 2010 16:11:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 16:11:11 -0000
Received: (qmail 78363 invoked by uid 500); 6 Jul 2010 16:11:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78304 invoked by uid 500); 6 Jul 2010 16:11:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78296 invoked by uid 99); 6 Jul 2010 16:11:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 16:11:09 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.176 as permitted sender)
Received: from [209.85.161.176] (HELO mail-gx0-f176.google.com) (209.85.161.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 16:11:03 +0000
Received: by gxk7 with SMTP id 7so1679177gxk.35
        for <general@hadoop.apache.org>; Tue, 06 Jul 2010 09:10:42 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=LhNdTvoYnIj6fYa4MCyLJBYdLN5M2KtrLSkgnBrzjAc=;
        b=MgLTz9Si+uRU5G7BNVbezV6z4rtGQv/u3qG7JK7D3wes83r4MT7sLx8LzY6OLPd+eV
         4iwIcUPkyTwitGe5u0hVEPG2Qz/EnlPFT20ncxvYcpnJjY0crf1MzvAG6ocCHcnMOcn/
         tdoHGVWPhx/A2uorafhbBth1Y4waVW14nTG5E=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=C95UOn8kHXYAZvIqeQR8riTINr2r656dvfcRcxnK8J9w3sp6+XjdrdsukR/UeSKER9
         jotXVpFlRdC+c7IZo9MxQQJZqRZjEdhBbQ1kGCFW7Vy1k6DYVZ7gbHeIUg7afGd82wQE
         8RbAbaSzeGdSYwJJwgQRgNKefdXON9YqPjdeo=
Received: by 10.229.235.197 with SMTP id kh5mr2843307qcb.237.1278432641518; 
	Tue, 06 Jul 2010 09:10:41 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.231.142 with HTTP; Tue, 6 Jul 2010 09:10:21 -0700 (PDT)
In-Reply-To: <035C7FA04C0BF64B8143C76523E7ED742FCFEEA6CD@EADC01-MABPRD01.ad.gd-ais.com>
References: <035C7FA04C0BF64B8143C76523E7ED742FCFEEA6CD@EADC01-MABPRD01.ad.gd-ais.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Tue, 6 Jul 2010 21:40:21 +0530
Message-ID: <AANLkTilST4ufbVEiWZBBuz5h4sbPHvgP6eXKOS05Z_W3@mail.gmail.com>
Subject: Re: MapReduce HBASE examples
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

I believe this article will help you understand the new (not anymore)
API+HBase MR: http://kdpeterson.net/blog/2009/09/minimal-hbase-mapreduce-ex=
ample.html
[Look at the second example, which uses the Put object]

On Tue, Jul 6, 2010 at 6:08 PM, Kilbride, James P.
<James.Kilbride@gd-ais.com> wrote:
> All,
>
> The examples in the hbase examples, and on the hadoop wiki all reference =
the deprecated interfaces of the mapred package. Are there any examples of =
how to use hbase as the input for a mapreduce job, that uses the mapreduce =
package instead? I'm looking to set up a job which will read from a hbase t=
able based on a row value passed into the job, and which starts the map the=
 row values(as the value key) and the column names(or value) as the map val=
ues.
>
> James Kilbride
>



--=20
Harsh J
www.harshj.com

From general-return-1766-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 16:41:30 2010
Return-Path: <general-return-1766-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 52034 invoked from network); 6 Jul 2010 16:41:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 16:41:30 -0000
Received: (qmail 18992 invoked by uid 500); 6 Jul 2010 16:41:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18950 invoked by uid 500); 6 Jul 2010 16:41:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18942 invoked by uid 99); 6 Jul 2010 16:41:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 16:41:28 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=RCVD_ILLEGAL_IP,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [192.5.164.99] (HELO camv02-relay2.casc.gd-ais.com) (192.5.164.99)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 16:41:22 +0000
Received: from ([10.73.100.22])
	by camv02-relay2.casc.gd-ais.com with SMTP  id 5203374.39553526;
	Tue, 06 Jul 2010 09:39:43 -0700
Received: from eadc01-cahprd02.ad.gd-ais.com ([10.120.80.12]) by camv02-fes01.ad.gd-ais.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Tue, 6 Jul 2010 09:39:42 -0700
Received: from EADC01-MABPRD01.ad.gd-ais.com ([169.254.1.214]) by
 eadc01-cahprd02.ad.gd-ais.com ([10.120.80.12]) with mapi; Tue, 6 Jul 2010
 11:38:53 -0500
From: "Kilbride, James P." <James.Kilbride@gd-ais.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 6 Jul 2010 11:38:51 -0500
Subject: RE: MapReduce HBASE examples
Thread-Topic: MapReduce HBASE examples
Thread-Index: AcsdJe0ESW8pcpuYRAaV7qn4wbxLbAAAq/WQ
Message-ID: <035C7FA04C0BF64B8143C76523E7ED742FCFF49D50@EADC01-MABPRD01.ad.gd-ais.com>
References: <035C7FA04C0BF64B8143C76523E7ED742FCFEEA6CD@EADC01-MABPRD01.ad.gd-ais.com>
 <AANLkTilST4ufbVEiWZBBuz5h4sbPHvgP6eXKOS05Z_W3@mail.gmail.com>
In-Reply-To: <AANLkTilST4ufbVEiWZBBuz5h4sbPHvgP6eXKOS05Z_W3@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginalArrivalTime: 06 Jul 2010 16:39:42.0902 (UTC) FILETIME=[D5DBFD60:01CB1D29]
X-Virus-Checked: Checked by ClamAV on apache.org

This is an interesting start but I'm really interested in the opposite dire=
ction, where hbase is the input to my map reduce job, and then I'm going to=
 push some data into reducers which ultimately I'm okay with them just writ=
ing it to a file.

I get the impression that I need to set up a TableInputFormat type of objec=
t. But I since job only allows you to do setInputFormatClass, I'm not sure =
how to dynamically configure the inputFormatClass to accept some parameters=
 to limit the input format scan on the table to only specific rows. Here's =
the general thrust of what I'm trying to do with MapReduce and HBase.

I have a table called People which has rows of people(names, ids, whatever =
is used for identifying a person in the system). That table also has a colu=
mn family called relatives where the column ids are the names of relatives =
for the person. I want to pass into the inputFormat object the names of the=
 people I want it to look up. And the mapper should get the persons name as=
 the key and the columnFamily relatives as the value(that's the result of t=
he scan limitations I'm putting into place).=20

I then will retrieve the relatives(in the map function), look at relationsh=
ips between them and push onto the context the relatives name(keyOut) and a=
 floating point value(valueOut). The reducer will combine all these floatin=
g point values for each relative and output(in a file is fine) the relative=
s name and cumulative score.

But I can't seem to figure out how to set up a job that uses the TableInput=
Format I want, and which also allows me to set the parameter for it so that=
 it will only give me the people I ask for when I run the program not the e=
ntire table.=20

Does this make any sense?

James Kilbride

-----Original Message-----
From: Harsh J [mailto:qwertymaniac@gmail.com]=20
Sent: Tuesday, July 06, 2010 12:10 PM
To: general@hadoop.apache.org
Subject: Re: MapReduce HBASE examples

I believe this article will help you understand the new (not anymore)
API+HBase MR: http://kdpeterson.net/blog/2009/09/minimal-hbase-mapreduce-ex=
ample.html
[Look at the second example, which uses the Put object]

On Tue, Jul 6, 2010 at 6:08 PM, Kilbride, James P.
<James.Kilbride@gd-ais.com> wrote:
> All,
>
> The examples in the hbase examples, and on the hadoop wiki all reference =
the deprecated interfaces of the mapred package. Are there any examples of =
how to use hbase as the input for a mapreduce job, that uses the mapreduce =
package instead? I'm looking to set up a job which will read from a hbase t=
able based on a row value passed into the job, and which starts the map the=
 row values(as the value key) and the column names(or value) as the map val=
ues.
>
> James Kilbride
>



--=20
Harsh J
www.harshj.com

From general-return-1767-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 16:54:00 2010
Return-Path: <general-return-1767-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54961 invoked from network); 6 Jul 2010 16:54:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 16:54:00 -0000
Received: (qmail 35904 invoked by uid 500); 6 Jul 2010 16:53:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 35449 invoked by uid 500); 6 Jul 2010 16:53:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 35354 invoked by uid 99); 6 Jul 2010 16:53:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 16:53:55 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jdcryans@gmail.com designates 209.85.161.176 as permitted sender)
Received: from [209.85.161.176] (HELO mail-gx0-f176.google.com) (209.85.161.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 16:53:49 +0000
Received: by gxk7 with SMTP id 7so1718232gxk.35
        for <general@hadoop.apache.org>; Tue, 06 Jul 2010 09:53:28 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        bh=9z43/BaInM50DAjplqBZb3htX2KumwOLRsg3P0Z9o7I=;
        b=J1j5VwwerVnv+KTYHQNoAwDFu8kkp+Sp16LGKzv+736GYNAvcnKi0iY/YfutYBBaOr
         Ws2QFWEWyJU7itLmVPzAqaaDeflhfm/FJmj8/FwNYpICB5y1KWazxd0gXzU7h6IYeBM+
         JUKu1rJvBMOSPFC5bh7QBjSZTpAwH/nURlZW0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=ZQom9v0mktIQirw6zfr2wu1CUWT+lJI2TYwbo3UR/mxwC2GxW2RaQ4ah2MhaclU7qa
         B7RjPYBshoe1sfKT8BeqwXHQHmuzVQOrlOCeefvjPAhc1FD4ReItUO2IKFKFjpHcN97k
         /DFblARhPU/Qx2XdMeVk2ViJElDo+60GpcMtQ=
MIME-Version: 1.0
Received: by 10.224.79.144 with SMTP id p16mr353339qak.247.1278435208316; Tue, 
	06 Jul 2010 09:53:28 -0700 (PDT)
Sender: jdcryans@gmail.com
Received: by 10.229.44.139 with HTTP; Tue, 6 Jul 2010 09:53:28 -0700 (PDT)
In-Reply-To: <035C7FA04C0BF64B8143C76523E7ED742FCFF49D50@EADC01-MABPRD01.ad.gd-ais.com>
References: <035C7FA04C0BF64B8143C76523E7ED742FCFEEA6CD@EADC01-MABPRD01.ad.gd-ais.com>
	<AANLkTilST4ufbVEiWZBBuz5h4sbPHvgP6eXKOS05Z_W3@mail.gmail.com>
	<035C7FA04C0BF64B8143C76523E7ED742FCFF49D50@EADC01-MABPRD01.ad.gd-ais.com>
Date: Tue, 6 Jul 2010 09:53:28 -0700
X-Google-Sender-Auth: G31SyVsbWfc4bSsEzUoXO4WCKek
Message-ID: <AANLkTilhTI59EGECUn2Eo2zXad_CiPmZGIItxr6ILbaX@mail.gmail.com>
Subject: Re: MapReduce HBASE examples
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00c09f9b0b7f75cdd6048abae41d
X-Virus-Checked: Checked by ClamAV on apache.org

--00c09f9b0b7f75cdd6048abae41d
Content-Type: text/plain; charset=ISO-8859-1

>
>
> Does this make any sense?
>
>
Not in a MapReduce context, what you want to do is a LIKE with a bunch of
values right? Since a mapper will always read all the input that it's given
(minus some filters like you can do with HBase), whatever you do will always
end up being a full table scan. You "could" solve your problem by
configuring your Scan object with a RowFilter that knows about the names you
are looking for, but that still ends up being a full scan on the region
server side so it will be slow and will generate a lot of IO.

WRT examples, HBase ships with a couple of utility classes that can also be
used as examples. The Export class has the Scan configuration stuff:
http://github.com/apache/hbase/blob/0.20/src/java/org/apache/hadoop/hbase/mapreduce/Export.java

J-D

--00c09f9b0b7f75cdd6048abae41d--

From general-return-1768-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 17:02:39 2010
Return-Path: <general-return-1768-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56702 invoked from network); 6 Jul 2010 17:02:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 17:02:39 -0000
Received: (qmail 45074 invoked by uid 500); 6 Jul 2010 17:02:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45013 invoked by uid 500); 6 Jul 2010 17:02:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45005 invoked by uid 99); 6 Jul 2010 17:02:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 17:02:37 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=RCVD_ILLEGAL_IP,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [192.5.164.99] (HELO camv02-relay2.casc.gd-ais.com) (192.5.164.99)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 17:02:31 +0000
Received: from ([10.73.100.22])
	by camv02-relay2.casc.gd-ais.com with SMTP  id 5203374.39557778;
	Tue, 06 Jul 2010 10:02:00 -0700
Received: from eadc01-cahprd01.ad.gd-ais.com ([10.120.80.11]) by camv02-fes01.ad.gd-ais.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Tue, 6 Jul 2010 10:02:06 -0700
Received: from EADC01-MABPRD01.ad.gd-ais.com ([169.254.1.214]) by
 eadc01-cahprd01.ad.gd-ais.com ([10.120.80.11]) with mapi; Tue, 6 Jul 2010
 12:02:05 -0500
From: "Kilbride, James P." <James.Kilbride@gd-ais.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 6 Jul 2010 12:02:02 -0500
Subject: RE: MapReduce HBASE examples
Thread-Topic: MapReduce HBASE examples
Thread-Index: AcsdK9eOc56LPevEQsq5cO35WNA8ZgAAEZjQ
Message-ID: <035C7FA04C0BF64B8143C76523E7ED742FCFF49DA6@EADC01-MABPRD01.ad.gd-ais.com>
References: <035C7FA04C0BF64B8143C76523E7ED742FCFEEA6CD@EADC01-MABPRD01.ad.gd-ais.com>
	<AANLkTilST4ufbVEiWZBBuz5h4sbPHvgP6eXKOS05Z_W3@mail.gmail.com>
	<035C7FA04C0BF64B8143C76523E7ED742FCFF49D50@EADC01-MABPRD01.ad.gd-ais.com>
 <AANLkTilhTI59EGECUn2Eo2zXad_CiPmZGIItxr6ILbaX@mail.gmail.com>
In-Reply-To: <AANLkTilhTI59EGECUn2Eo2zXad_CiPmZGIItxr6ILbaX@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginalArrivalTime: 06 Jul 2010 17:02:06.0291 (UTC) FILETIME=[F694E230:01CB1D2C]
X-Virus-Checked: Checked by ClamAV on apache.org

So, if that's the case, and you argument makes sense understanding how scan=
 versus get works, I'd have to write a custom InputFormat class that looks =
like the TableInputFormat class, but uses a get(or series of gets) rather t=
han a scan object as the current table mapper does?=20

James Kilbride

-----Original Message-----
From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean-Dani=
el Cryans
Sent: Tuesday, July 06, 2010 12:53 PM
To: general@hadoop.apache.org
Subject: Re: MapReduce HBASE examples

>
>
> Does this make any sense?
>
>
Not in a MapReduce context, what you want to do is a LIKE with a bunch of
values right? Since a mapper will always read all the input that it's given
(minus some filters like you can do with HBase), whatever you do will alway=
s
end up being a full table scan. You "could" solve your problem by
configuring your Scan object with a RowFilter that knows about the names yo=
u
are looking for, but that still ends up being a full scan on the region
server side so it will be slow and will generate a lot of IO.

WRT examples, HBase ships with a couple of utility classes that can also be
used as examples. The Export class has the Scan configuration stuff:
http://github.com/apache/hbase/blob/0.20/src/java/org/apache/hadoop/hbase/m=
apreduce/Export.java

J-D

From general-return-1769-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 17:12:01 2010
Return-Path: <general-return-1769-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 62646 invoked from network); 6 Jul 2010 17:12:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 17:12:00 -0000
Received: (qmail 58844 invoked by uid 500); 6 Jul 2010 17:11:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58809 invoked by uid 500); 6 Jul 2010 17:11:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58801 invoked by uid 99); 6 Jul 2010 17:11:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 17:11:59 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jdcryans@gmail.com designates 209.85.161.176 as permitted sender)
Received: from [209.85.161.176] (HELO mail-gx0-f176.google.com) (209.85.161.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 17:11:52 +0000
Received: by gxk7 with SMTP id 7so1734307gxk.35
        for <general@hadoop.apache.org>; Tue, 06 Jul 2010 10:11:31 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        bh=bPxyY67fF5ENGzxSf1bAuV5oxwKUwoZ7XwNHjvmQCfc=;
        b=u5LTgNJR0AHFaq67kfUc7OWACjRm9o+rVGipTeK9+We381CwdjDc9ghQwUaJFidd9J
         9cCRtZNPkiS4IFF6LyoL9e7Whvjhuss8GUGajQDfdpLqiBYLO4DC3aKSPmuF0RR6guXy
         +xinOW6pH5Zkud17ea4+5mN43FvOpmNPydiMo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=thCFK5HHkhYQbSQZZ2txIQz4RzvQkcNfUGeHpbei7NE+aeKaW9DeEWVtB3SDezhrGs
         zjRE6bSI45/X6y3jMrLSexiGL4o6lj3Z091Qs9jOrnMy/Zerhi5iE2YXJSJyg5jKVQLv
         7VJzFwrOMv4whnR4AdLaBqXMnSrjCC7qkiiQ8=
MIME-Version: 1.0
Received: by 10.224.115.27 with SMTP id g27mr2667300qaq.369.1278436291396; 
	Tue, 06 Jul 2010 10:11:31 -0700 (PDT)
Sender: jdcryans@gmail.com
Received: by 10.229.44.139 with HTTP; Tue, 6 Jul 2010 10:11:31 -0700 (PDT)
In-Reply-To: <035C7FA04C0BF64B8143C76523E7ED742FCFF49DA6@EADC01-MABPRD01.ad.gd-ais.com>
References: <035C7FA04C0BF64B8143C76523E7ED742FCFEEA6CD@EADC01-MABPRD01.ad.gd-ais.com>
	<AANLkTilST4ufbVEiWZBBuz5h4sbPHvgP6eXKOS05Z_W3@mail.gmail.com>
	<035C7FA04C0BF64B8143C76523E7ED742FCFF49D50@EADC01-MABPRD01.ad.gd-ais.com>
	<AANLkTilhTI59EGECUn2Eo2zXad_CiPmZGIItxr6ILbaX@mail.gmail.com>
	<035C7FA04C0BF64B8143C76523E7ED742FCFF49DA6@EADC01-MABPRD01.ad.gd-ais.com>
Date: Tue, 6 Jul 2010 10:11:31 -0700
X-Google-Sender-Auth: GsiyXEepbgb15VVEPPwgT6GAiWo
Message-ID: <AANLkTilZRFetj36jx_yo1uF9tDbZki6mYZRPYg2dWvBo@mail.gmail.com>
Subject: Re: MapReduce HBASE examples
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00c09f9b09f50446e7048abb2534
X-Virus-Checked: Checked by ClamAV on apache.org

--00c09f9b09f50446e7048abb2534
Content-Type: text/plain; charset=ISO-8859-1

That won't be very efficient either... are you trying to do this for a real
time user request. If so, it really isn't the way you want to go.

If you are in a batch processing situation, I'd say it depends on how many
rows you have VS how many you need to retrieve eg scanning 2B rows only to
find 10 rows really doesn't make sense. How do you determine which users you
need to process? How big is your dataset? I understand that you wish to use
the MR-provided functionalities of grouping and such, but simply issuing a
bunch of Gets in parallel may just be easier to write and maintain.

J-D

On Tue, Jul 6, 2010 at 10:02 AM, Kilbride, James P. <
James.Kilbride@gd-ais.com> wrote:

> So, if that's the case, and you argument makes sense understanding how scan
> versus get works, I'd have to write a custom InputFormat class that looks
> like the TableInputFormat class, but uses a get(or series of gets) rather
> than a scan object as the current table mapper does?
>
> James Kilbride
>
> -----Original Message-----
> From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of
> Jean-Daniel Cryans
> Sent: Tuesday, July 06, 2010 12:53 PM
> To: general@hadoop.apache.org
> Subject: Re: MapReduce HBASE examples
>
> >
> >
> > Does this make any sense?
> >
> >
> Not in a MapReduce context, what you want to do is a LIKE with a bunch of
> values right? Since a mapper will always read all the input that it's given
> (minus some filters like you can do with HBase), whatever you do will
> always
> end up being a full table scan. You "could" solve your problem by
> configuring your Scan object with a RowFilter that knows about the names
> you
> are looking for, but that still ends up being a full scan on the region
> server side so it will be slow and will generate a lot of IO.
>
> WRT examples, HBase ships with a couple of utility classes that can also be
> used as examples. The Export class has the Scan configuration stuff:
>
> http://github.com/apache/hbase/blob/0.20/src/java/org/apache/hadoop/hbase/mapreduce/Export.java
>
> J-D
>

--00c09f9b09f50446e7048abb2534--

From general-return-1770-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 17:53:10 2010
Return-Path: <general-return-1770-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70878 invoked from network); 6 Jul 2010 17:53:10 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 17:53:10 -0000
Received: (qmail 5278 invoked by uid 500); 6 Jul 2010 17:53:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5154 invoked by uid 500); 6 Jul 2010 17:53:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5146 invoked by uid 99); 6 Jul 2010 17:53:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 17:53:08 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=RCVD_ILLEGAL_IP,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [137.100.120.43] (HELO mnbm01-relay1.mnb.gd-ais.com) (137.100.120.43)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 17:53:01 +0000
Received: from ([160.207.224.15])
	by mnbm01-relay1.mnb.gd-ais.com with SMTP  id 5202712.275883862;
	Tue, 06 Jul 2010 12:50:58 -0500
Received: from eadc01-cahprd01.ad.gd-ais.com ([10.120.80.11]) by mnbm01-fes01.ad.gd-ais.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Tue, 6 Jul 2010 12:50:57 -0500
Received: from EADC01-MABPRD01.ad.gd-ais.com ([169.254.1.214]) by
 eadc01-cahprd01.ad.gd-ais.com ([10.120.80.11]) with mapi; Tue, 6 Jul 2010
 12:50:58 -0500
From: "Kilbride, James P." <James.Kilbride@gd-ais.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 6 Jul 2010 12:50:55 -0500
Subject: RE: MapReduce HBASE examples
Thread-Topic: MapReduce HBASE examples
Thread-Index: AcsdLtmMqeDVDfbaQD6ko2+gKHt7DQAAumcA
Message-ID: <035C7FA04C0BF64B8143C76523E7ED742FCFF49E2B@EADC01-MABPRD01.ad.gd-ais.com>
References: <035C7FA04C0BF64B8143C76523E7ED742FCFEEA6CD@EADC01-MABPRD01.ad.gd-ais.com>
	<AANLkTilST4ufbVEiWZBBuz5h4sbPHvgP6eXKOS05Z_W3@mail.gmail.com>
	<035C7FA04C0BF64B8143C76523E7ED742FCFF49D50@EADC01-MABPRD01.ad.gd-ais.com>
	<AANLkTilhTI59EGECUn2Eo2zXad_CiPmZGIItxr6ILbaX@mail.gmail.com>
	<035C7FA04C0BF64B8143C76523E7ED742FCFF49DA6@EADC01-MABPRD01.ad.gd-ais.com>
 <AANLkTilZRFetj36jx_yo1uF9tDbZki6mYZRPYg2dWvBo@mail.gmail.com>
In-Reply-To: <AANLkTilZRFetj36jx_yo1uF9tDbZki6mYZRPYg2dWvBo@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginalArrivalTime: 06 Jul 2010 17:50:57.0991 (UTC) FILETIME=[CA02CD70:01CB1D33]
X-Virus-Checked: Checked by ClamAV on apache.org

I'm assuming the rows being pulled back are smaller than the full row set o=
f the entire database. So say the 10 out of 2B case. But, each row has a co=
lumn family who's 'columns' are actually rowIds in the database. (basically=
 my one to many relationship mapping). I'm not trying to use MR for the ini=
tial get of 10 columns, but rather the fact that each of those 10 initial r=
ows generates potentially hundreds or thousands of other calls.=20

I am trying to do this for a real time user request, but I expect the total=
 processing to take some time so it's more of a user initiated call. There =
also may be dozens of users making the request at any given time so I want =
to farm this out into the MR world so that multiple instances of the job ca=
n be running(with completely different starting rows) at any given time.=20

I could do this using a serialized local process but I explicitly want some=
 of my processing, which could take some time, happening out in the map red=
uce world to take advantage of spare cycles elsewhere, as well as potential=
 data locality and the fact that it is a parallelizable problem seems to im=
ply that M/R would be a logical way to do it.

James Kilbride

-----Original Message-----
From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean-Dani=
el Cryans
Sent: Tuesday, July 06, 2010 1:12 PM
To: general@hadoop.apache.org
Subject: Re: MapReduce HBASE examples

That won't be very efficient either... are you trying to do this for a real
time user request. If so, it really isn't the way you want to go.

If you are in a batch processing situation, I'd say it depends on how many
rows you have VS how many you need to retrieve eg scanning 2B rows only to
find 10 rows really doesn't make sense. How do you determine which users yo=
u
need to process? How big is your dataset? I understand that you wish to use
the MR-provided functionalities of grouping and such, but simply issuing a
bunch of Gets in parallel may just be easier to write and maintain.

J-D

On Tue, Jul 6, 2010 at 10:02 AM, Kilbride, James P. <
James.Kilbride@gd-ais.com> wrote:

> So, if that's the case, and you argument makes sense understanding how sc=
an
> versus get works, I'd have to write a custom InputFormat class that looks
> like the TableInputFormat class, but uses a get(or series of gets) rather
> than a scan object as the current table mapper does?
>
> James Kilbride
>
> -----Original Message-----
> From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of
> Jean-Daniel Cryans
> Sent: Tuesday, July 06, 2010 12:53 PM
> To: general@hadoop.apache.org
> Subject: Re: MapReduce HBASE examples
>
> >
> >
> > Does this make any sense?
> >
> >
> Not in a MapReduce context, what you want to do is a LIKE with a bunch of
> values right? Since a mapper will always read all the input that it's giv=
en
> (minus some filters like you can do with HBase), whatever you do will
> always
> end up being a full table scan. You "could" solve your problem by
> configuring your Scan object with a RowFilter that knows about the names
> you
> are looking for, but that still ends up being a full scan on the region
> server side so it will be slow and will generate a lot of IO.
>
> WRT examples, HBase ships with a couple of utility classes that can also =
be
> used as examples. The Export class has the Scan configuration stuff:
>
> http://github.com/apache/hbase/blob/0.20/src/java/org/apache/hadoop/hbase=
/mapreduce/Export.java
>
> J-D
>

From general-return-1771-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 19:16:18 2010
Return-Path: <general-return-1771-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5050 invoked from network); 6 Jul 2010 19:16:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 19:16:17 -0000
Received: (qmail 96679 invoked by uid 500); 6 Jul 2010 19:16:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96621 invoked by uid 500); 6 Jul 2010 19:16:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96613 invoked by uid 99); 6 Jul 2010 19:16:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 19:16:15 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of gmackey@cs.ucf.edu designates 132.170.216.154 as permitted sender)
Received: from [132.170.216.154] (HELO longwood.cs.ucf.edu) (132.170.216.154)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 19:16:08 +0000
Received: from longwood.cs.ucf.edu (localhost [127.0.0.1])
	by longwood.cs.ucf.edu (8.14.1/8.14.1) with ESMTP id o66JFivV002916;
	Tue, 6 Jul 2010 15:15:45 -0400 (EDT)
Received: (from apache@localhost)
	by longwood.cs.ucf.edu (8.14.1/8.14.4/Submit) id o66JFisO002913;
	Tue, 6 Jul 2010 15:15:44 -0400 (EDT)
X-Authentication-Warning: longwood.cs.ucf.edu: apache set sender to gmackey@cs.ucf.edu using -f
Received: from proxyout.lanl.gov (proxyout.lanl.gov [192.12.184.2]) by
 mail.cs.ucf.edu (Horde Framework) with HTTP; Tue, 06 Jul 2010 15:15:43
 -0400
Message-ID: <20100706151543.15177jed70raesws@mail.cs.ucf.edu>
Date: Tue, 06 Jul 2010 15:15:43 -0400
From: Grant Mackey <gmackey@cs.ucf.edu>
To: general@hadoop.apache.org, abc xyz <fabc_xyz111@yahoo.com>
Subject: Re: Chaining Map-Reduce
References: <351370.62305.qm@web46308.mail.sp1.yahoo.com>
 <C85394F5.3AB34%svenkat@yahoo-inc.com>
 <AANLkTinjmTNRKYbSw3ULo5p64Ag4aip7yfz631OApPqr@mail.gmail.com>
 <790746.6243.qm@web46312.mail.sp1.yahoo.com>
In-Reply-To: <790746.6243.qm@web46312.mail.sp1.yahoo.com>
MIME-Version: 1.0
Content-Type: text/plain;
 charset=ISO-8859-1;
 DelSp="Yes";
 format="flowed"
Content-Disposition: inline
Content-Transfer-Encoding: 7bit
User-Agent: Internet Messaging Program (IMP) 4.3.3
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on gondor.cs.ucf.edu
X-Virus-Scanned: clamav-milter 0.96.1 at longwood
X-Virus-Status: Clean
X-Virus-Checked: Checked by ClamAV on apache.org
X-Old-Spam-Status: No, score=-1.4 required=6.0 tests=ALL_TRUSTED autolearn=failed
	version=3.2.5

Oh, of course it will. From what I know about using this it just  
allows you to launch 1 job instead of several. Every time a Map/Reduce  
pair finishes it will dump to the HDFS (or whatever you're using)

  - Grant

Quoting abc xyz <fabc_xyz111@yahoo.com>:

> one option can be to use multiple chained jobs using JobControl object.
> But can anyone tell whether in this case the output of first job be  
> written to
> the disk? Actually I want to minimize access to the disk.
>
> Cheers
>
>
>
> ________________________________
> From: Fariha Atta <fariha.atta.pk@gmail.com>
> To: general@hadoop.apache.org
> Sent: Fri, July 2, 2010 8:45:12 AM
> Subject: Re: Chaining Map-Reduce
>
> But it provides chaining like Map+ | Reduce | Map* i.e. we can use only one
> reducer.
>
>
> On Fri, Jul 2, 2010 at 8:37 AM, Venkatesh S <svenkat@yahoo-inc.com> wrote:
>
>> You could use ChainMapper and ChainReducer API in hadoop.
>>
>
>
>
>



Grant Mackey
UCF Research Assistant
Engineering III
Rm 238 Cubicle 1

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.


From general-return-1772-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 21:39:44 2010
Return-Path: <general-return-1772-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45654 invoked from network); 6 Jul 2010 21:39:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 21:39:43 -0000
Received: (qmail 56375 invoked by uid 500); 6 Jul 2010 21:39:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56282 invoked by uid 500); 6 Jul 2010 21:39:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56274 invoked by uid 99); 6 Jul 2010 21:39:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 21:39:41 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jdcryans@gmail.com designates 209.85.213.176 as permitted sender)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 21:39:35 +0000
Received: by yxj4 with SMTP id 4so932435yxj.35
        for <general@hadoop.apache.org>; Tue, 06 Jul 2010 14:39:14 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        bh=ZEdaa6EM+GfwVOYDjs+RJXMc6OLBBjW9eWyJ4JKYHxM=;
        b=JzEZMSOdDx5kfQjkrLk3oDFDoSjvUlNYml7UR8B8hKsZ9RiMXQumpiZZw5gip0IFOa
         xm7bGrbKEfVdfAZfe4C/1yNUGszS0CmU2ZLz9QUgvywdldVFIxzUHTdwclsCqvu47r2Y
         s6iBejTHg979J7eMSOkKhGywi04bPSlLA+X2M=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=NCKB0hl3gZbSHiUNpZlY+uyQgZLxVcSZtcGgSFMKvMmYm79ujfl5Q4eN2QwoFlUesj
         CSIN8o4j5il+tit0MFAKLZL3K4GlUIhKlJoxeakbHAEVJedJ7eho1lYzXzgHxc+hvP22
         NbQdIUDVSB6b+AV6Mtwh2wrks+QSmqCCEBwPQ=
MIME-Version: 1.0
Received: by 10.229.248.195 with SMTP id mh3mr3157249qcb.206.1278452354383; 
	Tue, 06 Jul 2010 14:39:14 -0700 (PDT)
Sender: jdcryans@gmail.com
Received: by 10.229.44.139 with HTTP; Tue, 6 Jul 2010 14:39:14 -0700 (PDT)
In-Reply-To: <035C7FA04C0BF64B8143C76523E7ED742FCFF49E2B@EADC01-MABPRD01.ad.gd-ais.com>
References: <035C7FA04C0BF64B8143C76523E7ED742FCFEEA6CD@EADC01-MABPRD01.ad.gd-ais.com>
	<AANLkTilST4ufbVEiWZBBuz5h4sbPHvgP6eXKOS05Z_W3@mail.gmail.com>
	<035C7FA04C0BF64B8143C76523E7ED742FCFF49D50@EADC01-MABPRD01.ad.gd-ais.com>
	<AANLkTilhTI59EGECUn2Eo2zXad_CiPmZGIItxr6ILbaX@mail.gmail.com>
	<035C7FA04C0BF64B8143C76523E7ED742FCFF49DA6@EADC01-MABPRD01.ad.gd-ais.com>
	<AANLkTilZRFetj36jx_yo1uF9tDbZki6mYZRPYg2dWvBo@mail.gmail.com>
	<035C7FA04C0BF64B8143C76523E7ED742FCFF49E2B@EADC01-MABPRD01.ad.gd-ais.com>
Date: Tue, 6 Jul 2010 14:39:14 -0700
X-Google-Sender-Auth: gmmMhjQpj7R87mIdsgizdHtZK0E
Message-ID: <AANLkTilPO-E6yVXxA0hYQUEqwrUMWkFkB2lxuAJyWSej@mail.gmail.com>
Subject: Re: MapReduce HBASE examples
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: general@hadoop.apache.org, user@hbase.apache.org
Content-Type: multipart/alternative; boundary=0016e64caa9e720207048abee21d
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e64caa9e720207048abee21d
Content-Type: text/plain; charset=ISO-8859-1

(moving the thread to the HBase user mailing list, on reply please remove
the general@ since this is not a general question)

It is indeed a parallelizable problem that could use a job management
system, but in your case I don't think MR is the right solution. You will
have to do all sorts weird tweaks and in the end you won't get much out of
it since you basically want to process a tiny portion of the whole dataset.
You also talk about possible localisation, but I don't see that being
a particularly strong argument in what you describe. Yes, you could start
one mapper per region that contains some of the rows you are looking for,
but the cost of starting and managing those JVMs is high compared to just
starting one that does the work (since it can be done easily in a single
process that can be multi-threaded).

To sum up, using MR on a small dataset is basically having all the
disadvantages for almost none of the advantages.

Instead you could look into running Gearman (or similar) on those machines
and that would give you exactly what you need IMHO.

J-D

On Tue, Jul 6, 2010 at 10:50 AM, Kilbride, James P. <
James.Kilbride@gd-ais.com> wrote:

> I'm assuming the rows being pulled back are smaller than the full row set
> of the entire database. So say the 10 out of 2B case. But, each row has a
> column family who's 'columns' are actually rowIds in the database.
> (basically my one to many relationship mapping). I'm not trying to use MR
> for the initial get of 10 columns, but rather the fact that each of those 10
> initial rows generates potentially hundreds or thousands of other calls.
>
> I am trying to do this for a real time user request, but I expect the total
> processing to take some time so it's more of a user initiated call. There
> also may be dozens of users making the request at any given time so I want
> to farm this out into the MR world so that multiple instances of the job can
> be running(with completely different starting rows) at any given time.
>
> I could do this using a serialized local process but I explicitly want some
> of my processing, which could take some time, happening out in the map
> reduce world to take advantage of spare cycles elsewhere, as well as
> potential data locality and the fact that it is a parallelizable problem
> seems to imply that M/R would be a logical way to do it.
>
> James Kilbride
>
> -----Original Message-----
> From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of
> Jean-Daniel Cryans
> Sent: Tuesday, July 06, 2010 1:12 PM
> To: general@hadoop.apache.org
> Subject: Re: MapReduce HBASE examples
>
> That won't be very efficient either... are you trying to do this for a real
> time user request. If so, it really isn't the way you want to go.
>
> If you are in a batch processing situation, I'd say it depends on how many
> rows you have VS how many you need to retrieve eg scanning 2B rows only to
> find 10 rows really doesn't make sense. How do you determine which users
> you
> need to process? How big is your dataset? I understand that you wish to use
> the MR-provided functionalities of grouping and such, but simply issuing a
> bunch of Gets in parallel may just be easier to write and maintain.
>
> J-D
>
> On Tue, Jul 6, 2010 at 10:02 AM, Kilbride, James P. <
> James.Kilbride@gd-ais.com> wrote:
>
> > So, if that's the case, and you argument makes sense understanding how
> scan
> > versus get works, I'd have to write a custom InputFormat class that looks
> > like the TableInputFormat class, but uses a get(or series of gets) rather
> > than a scan object as the current table mapper does?
> >
> > James Kilbride
> >
> > -----Original Message-----
> > From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of
> > Jean-Daniel Cryans
> > Sent: Tuesday, July 06, 2010 12:53 PM
> > To: general@hadoop.apache.org
> > Subject: Re: MapReduce HBASE examples
> >
> > >
> > >
> > > Does this make any sense?
> > >
> > >
> > Not in a MapReduce context, what you want to do is a LIKE with a bunch of
> > values right? Since a mapper will always read all the input that it's
> given
> > (minus some filters like you can do with HBase), whatever you do will
> > always
> > end up being a full table scan. You "could" solve your problem by
> > configuring your Scan object with a RowFilter that knows about the names
> > you
> > are looking for, but that still ends up being a full scan on the region
> > server side so it will be slow and will generate a lot of IO.
> >
> > WRT examples, HBase ships with a couple of utility classes that can also
> be
> > used as examples. The Export class has the Scan configuration stuff:
> >
> >
> http://github.com/apache/hbase/blob/0.20/src/java/org/apache/hadoop/hbase/mapreduce/Export.java
> >
> > J-D
> >
>

--0016e64caa9e720207048abee21d--

From general-return-1773-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 07 01:13:16 2010
Return-Path: <general-return-1773-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 3583 invoked from network); 7 Jul 2010 01:13:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jul 2010 01:13:16 -0000
Received: (qmail 33528 invoked by uid 500); 7 Jul 2010 01:13:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33446 invoked by uid 500); 7 Jul 2010 01:13:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33438 invoked by uid 99); 7 Jul 2010 01:13:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 01:13:14 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates 209.85.216.169 as permitted sender)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 01:13:07 +0000
Received: by qyk32 with SMTP id 32so2807904qyk.14
        for <general@hadoop.apache.org>; Tue, 06 Jul 2010 18:11:46 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=TIbkue3CeBg5Bg2hLA5D1c/D85w5WED7YDZVjgJVCAk=;
        b=Dp9XCObdeA1i1dQbpdTnvM+50PWJM9OjtMq2WQg/0qcr+12RP6NtGD87a8+iCmu0f+
         1VUoOkm8PwtAT8EXCdBTV5OIg5peHL7J9C+twIrUZmPC8gEMnvrRS3dJCXsKQ7Z+hZNt
         aDhApFU3OBEBcSYr0fgSjJw3oIqO6PQx99ZT4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=pIiC1wS+0yeMidT7CCRKaozLhf268h7E7o0e4e6RkNPvOkZDOfUnXPBelLCWrcFbqo
         G40du2Z9AljPgvPOkgRp8FuzXL8Kft2NeyaPycxowRltCc0aYxjWioJzEtj+6IRDHR/A
         mx4SftX44D82/AdinqgVhvVgAkqMXUnOVY21w=
MIME-Version: 1.0
Received: by 10.224.106.4 with SMTP id v4mr3059802qao.305.1278465106149; Tue, 
	06 Jul 2010 18:11:46 -0700 (PDT)
Received: by 10.229.189.134 with HTTP; Tue, 6 Jul 2010 18:11:46 -0700 (PDT)
In-Reply-To: <AANLkTinCufLiyHIcg6o4KaFbAGj3MF3WLe-5t1_faBnb@mail.gmail.com>
References: <AANLkTinCufLiyHIcg6o4KaFbAGj3MF3WLe-5t1_faBnb@mail.gmail.com>
Date: Tue, 6 Jul 2010 18:11:46 -0700
Message-ID: <AANLkTimIoRAUNRhpjDtmJ5jwKe3pb7ENMM7fVGckwIju@mail.gmail.com>
Subject: Re: Is org.apache.hadoop.mapred.lib.MultipleOutputFormat deprecated?
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00c09f83a5b8828406048ac1da23
X-Virus-Checked: Checked by ClamAV on apache.org

--00c09f83a5b8828406048ac1da23
Content-Type: text/plain; charset=ISO-8859-1

Usually MultipleSequenceFileOutputFormat or MultipleTextOutputFormat is
used.

You need to use:
jobConf.setOutputFormat()

On Mon, Jul 5, 2010 at 1:29 AM, zhangguoping zhangguoping <
zhangguoping96@gmail.com> wrote:

> Hi,
>
> Is org.apache.hadoop.mapred.lib.MultipleOutputFormat deprecated? I did not
> find @deprecated comments in source file in 0.20.2
>
> But I cannot use following:
>
> job.setOutputFormatClass(
>  org.apache.hadoop.mapred.lib.MultipleOutputFormat
> )
>
>
> The type does not match.
>

--00c09f83a5b8828406048ac1da23--

From general-return-1774-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 07 02:11:42 2010
Return-Path: <general-return-1774-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17250 invoked from network); 7 Jul 2010 02:11:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jul 2010 02:11:41 -0000
Received: (qmail 65131 invoked by uid 500); 7 Jul 2010 02:11:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65073 invoked by uid 500); 7 Jul 2010 02:11:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65065 invoked by uid 99); 7 Jul 2010 02:11:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 02:11:40 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eltonsky9404@gmail.com designates 209.85.216.176 as permitted sender)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 02:11:32 +0000
Received: by qyk12 with SMTP id 12so2816810qyk.14
        for <general@hadoop.apache.org>; Tue, 06 Jul 2010 19:10:11 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=ZazK6nk6KOba8rmpkh7dVRJ6kplqof8mIw1ceI5Z7YY=;
        b=WzeCiTrDvTzO9zNIgrf67syFoYOvqM7muJTIhDHpkH+ZtB94X9Lka995OKlpjwFRcJ
         bHaoSl5SXu/Vf0EEmBdemmy0WEiS5vp0mRHVhlQal1vlfJ2FAJvWCwHBZoKHOE7CPxOM
         YdstIEW4mrt2es/22buZcBHmSV5vmhD12e1AU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=WMKNki9qZ/jqN6Y6jz4G3+nACLlNx6XNK+xisM4vOQYEWydXqGlGxr9VlmCiFpwGE0
         PsXF07mmQWUg/ujy9cXWKcUsVlrmwSMTpIHO0keiOTKkkbWcx1yj5buNY+OxfbHftQ9M
         xko6IEPVHfEO1oIbJq5zYu86XIAw16Mf7QxUg=
MIME-Version: 1.0
Received: by 10.224.52.32 with SMTP id f32mr3056206qag.352.1278468610981; Tue, 
	06 Jul 2010 19:10:10 -0700 (PDT)
Received: by 10.224.89.18 with HTTP; Tue, 6 Jul 2010 19:10:10 -0700 (PDT)
In-Reply-To: <4C334C20.9010105@apache.org>
References: <AANLkTiljgFWtvH2dQHhB44-8mXQKlAhtOPp5N1K3tJ3S@mail.gmail.com>
	<AANLkTikCO_TnPI-xBt_QQljWkTBWJ4lq3nGhHPSEjYNr@mail.gmail.com>
	<ABC24175AFD3BE4DA15F4CD375ED413D06084AF156@hq-ex-mb02.ad.navteq.com>
	<AANLkTilk2wfZ10leX0iEeNwJozMDdfHNiQ09rEAnbPDd@mail.gmail.com>
	<AANLkTimmN23EV17S1g-2LITHQrasMmh6qbSj13CJyfb5@mail.gmail.com>
	<AANLkTikNr4C2mjzLSwxUFglxtbfEalCwQ8clyzCNl_8a@mail.gmail.com>
	<AANLkTinC804Q953e4_4x80U_3l36EUERH4UNOz1I30Ww@mail.gmail.com>
	<5C9D1E1E-8299-4E97-B552-2DF69F5231BE@linkedin.com>
	<AANLkTimp7y0EYCThQ4WTKx6lW6BGYgkHHEocI3EV94q_@mail.gmail.com>
	<AANLkTinHouGalTJqqemJEb55fXme9qXaG0cg4rI0X7I-@mail.gmail.com>
	<COL117-W1485F3B9503653BB189B938FB20@phx.gbl>
	<4C334C20.9010105@apache.org>
Date: Wed, 7 Jul 2010 12:10:10 +1000
Message-ID: <AANLkTik7k_H6_1rtDiY5e_fxwpX7l9HrB1wvb6CD6iD8@mail.gmail.com>
Subject: Re: Why single thread for HDFS?
From: elton sky <eltonsky9404@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00c09f88d2796a0155048ac2abad
X-Virus-Checked: Checked by ClamAV on apache.org

--00c09f88d2796a0155048ac2abad
Content-Type: text/plain; charset=ISO-8859-1

Steve,

Seems HP has done block based parallel reading from different datanodes.
Though not from disk level, they achieve 4Gb/s rate with 9 readers (500Mb/s
each).
I didn't see anywhere I can download their code to play around, pity~

BTW, can we specify which disk to read from with Java?

On Wed, Jul 7, 2010 at 1:30 AM, Steve Loughran <stevel@apache.org> wrote:

> Michael Segel wrote:
>
>> Uhm...
>>
>> That's not really true. It gets a bit more complicated than that.
>>
>> If you're talking about M/R jobs, you don't want to do threads in your
>> map() routine, while this is possible, its going to be really hard to
>> justify the extra parallelism along with the need to wait for all of the
>> threads to complete before you can end the map() method.
>> If you're talking about a way to copy files from one cluster to another...
>> in hadoop... you can find out the block lists that make up the file. As long
>> as the file is static, meaning no one is writing/spliting/compacting the
>> file, you could copy it. Here being multi threaded could work. You'd have
>> one thread per block that will read from one machine, and then write
>> directly to the other. Of course you'll need to figure out where to write
>> the block, or rather tie in to HDFS.
>>
>
> There's a paper by Russ Perry using HDFS as a filestore for raster
> processing, where he modified DfsClient to get all the locations of a file,
> and let the caller decide where to read blocks from.
>
> http://www.hpl.hp.com/techreports/2009/HPL-2009-345.html
>
> the advantage of this is that the caller can do the striping across
> machines, keep every server busy by asking for files from each of them. Of
> course, this ignores the trend to many-HDD servers; DfsClient can't
> currently see which physical disk a file is on, which you'd need if the
> client wanted to keep every disk on every server busy during a big read
>

--00c09f88d2796a0155048ac2abad--

From general-return-1775-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 07 09:46:22 2010
Return-Path: <general-return-1775-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25243 invoked from network); 7 Jul 2010 09:46:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jul 2010 09:46:22 -0000
Received: (qmail 22302 invoked by uid 500); 7 Jul 2010 09:46:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21971 invoked by uid 500); 7 Jul 2010 09:46:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21963 invoked by uid 99); 7 Jul 2010 09:46:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 09:46:16 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 09:46:06 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 7739D1BA57D
	for <general@hadoop.apache.org>; Wed,  7 Jul 2010 10:45:45 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id Aq2pHuOEInMQ for <general@hadoop.apache.org>;
	Wed,  7 Jul 2010 10:45:45 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by colossus.hpl.hp.com (Postfix) with ESMTPS id D7FD01BA440
	for <general@hadoop.apache.org>; Wed,  7 Jul 2010 10:45:44 +0100 (BST)
MailScanner-NULL-Check: 1279100732.30003@ZOlSYnGZ0XYm8yyStS/PgQ
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o679jUhV015208
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Wed, 7 Jul 2010 10:45:31 +0100 (BST)
Message-ID: <4C344CBC.4090307@apache.org>
Date: Wed, 07 Jul 2010 10:45:32 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100228)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Why single thread for HDFS?
References: <AANLkTiljgFWtvH2dQHhB44-8mXQKlAhtOPp5N1K3tJ3S@mail.gmail.com>	<AANLkTikCO_TnPI-xBt_QQljWkTBWJ4lq3nGhHPSEjYNr@mail.gmail.com>	<ABC24175AFD3BE4DA15F4CD375ED413D06084AF156@hq-ex-mb02.ad.navteq.com>	<AANLkTilk2wfZ10leX0iEeNwJozMDdfHNiQ09rEAnbPDd@mail.gmail.com>	<AANLkTimmN23EV17S1g-2LITHQrasMmh6qbSj13CJyfb5@mail.gmail.com>	<AANLkTikNr4C2mjzLSwxUFglxtbfEalCwQ8clyzCNl_8a@mail.gmail.com>	<AANLkTinC804Q953e4_4x80U_3l36EUERH4UNOz1I30Ww@mail.gmail.com>	<5C9D1E1E-8299-4E97-B552-2DF69F5231BE@linkedin.com>	<AANLkTimp7y0EYCThQ4WTKx6lW6BGYgkHHEocI3EV94q_@mail.gmail.com>	<AANLkTinHouGalTJqqemJEb55fXme9qXaG0cg4rI0X7I-@mail.gmail.com>	<COL117-W1485F3B9503653BB189B938FB20@phx.gbl>	<4C334C20.9010105@apache.org> <AANLkTik7k_H6_1rtDiY5e_fxwpX7l9HrB1wvb6CD6iD8@mail.gmail.com>
In-Reply-To: <AANLkTik7k_H6_1rtDiY5e_fxwpX7l9HrB1wvb6CD6iD8@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o679jUhV015208
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

elton sky wrote:
> Steve,
> 
> Seems HP has done block based parallel reading from different datanodes.

yes; very much like IBM's GPFS, only with JBOD storage and the option of 
running code near the data when appropriate.

> Though not from disk level, they achieve 4Gb/s rate with 9 readers (500Mb/s
> each).
> I didn't see anywhere I can download their code to play around, pity~


I do have access to that code if I can get at the right bit of the 
repository, if you really want me to look at it in detail ask, with the 
caveats that I'm away for the rest of the month and somewhat busy. Apart 
from that there's no reason why I shouldn't be able to make the changes 
to DfsClient public. Keep reminding me :)


> BTW, can we specify which disk to read from with Java?
> 

I think right now you get a list of blocks via 
DfsClient.getBlockLocations(); this is a list of hosts where blocks 
live. There is no data about which disk on the specific host.

I belive that what Russ did was move the decisions from DfsInputStream 
-which picks a block location for you, with a bias to the local host- 
and instead lets the calling program make the decision as to where to 
fetch each block. This meant he could set the renderer up to request 
blocks from different hosts.

He had tried to use the JT to schedule the rendering code, but that 
didn't work as MapReduce has the notion of "reduction": less data out 
than in, so it moves work to where the data is. In rendering it's more 
MapExpand; the operation is the transformation of PDF pages into 600dpi 
32bpp bitmaps, which then need to be streamed to the (very large) 
printer at its print rate, in the correct order. It was easiest to have 
a specific machine on the cluster -with no datanodes or TTs- set up to 
do the rendering, and just ask the filesystem for where things are.

Like I said, I don't think there was anything tricky done in DfsClient, 
more a matter of making some data known internally to the  DfsClient 
code public, so that the client app can decide where to fetch data. If 
the DfsClient knew which HDD the data was on in a datanode, the client 
app could use that in its decision making too, so that if the 9 machines 
each had 6 HDDs, you could keep them all busy.

From general-return-1776-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 07 17:01:38 2010
Return-Path: <general-return-1776-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58860 invoked from network); 7 Jul 2010 17:01:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jul 2010 17:01:37 -0000
Received: (qmail 95014 invoked by uid 500); 7 Jul 2010 17:01:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94605 invoked by uid 500); 7 Jul 2010 17:01:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94582 invoked by uid 99); 7 Jul 2010 17:01:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 17:01:30 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of enis.soz.nutch@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 17:01:24 +0000
Received: by vws10 with SMTP id 10so1459460vws.35
        for <multiple recipients>; Wed, 07 Jul 2010 10:00:03 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=03mY+A/fTqD/lW60i9GAlq/OFZBZG/wyUQoaxQlOdV8=;
        b=LICI/oBvOFXaTeQ2izbf5pQwSY/16G2OPW8CErbq7sCvMf/UcgAkc5jr0VaqEvQEs5
         p9qmVzZ9LwwaGDn8vQMGdDfKmTVBY6kQND3njNl1D0cMTZ64geDReNKh2eCb5SR0OTYa
         UKjojj8BShYKrgT2+UZr9X0wy21jl6pb7URhw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=Mo9/GL7mvxTYt/Bjup870oryHjFg4yQIXcMxzYEpKFvxU6C2tuUnNCH2P7ayZU9MoI
         DoVBmMdCzluR+xwEL+Aek3Y0gPKdrC1hkXW/kAUNjYRNv5R4xPHtMeyLN3Xyfzy3O08l
         B80KWUgcjdKY1i1Ul+QG7YXvoDk+IHm2Vxpzs=
MIME-Version: 1.0
Received: by 10.220.76.74 with SMTP id b10mr3542937vck.78.1278522001850; Wed, 
	07 Jul 2010 10:00:01 -0700 (PDT)
Received: by 10.220.106.145 with HTTP; Wed, 7 Jul 2010 10:00:01 -0700 (PDT)
Date: Wed, 7 Jul 2010 20:00:01 +0300
Message-ID: <AANLkTil4EW97QBS8fdxKZdxwM2hEaG3Id8Qqa9Tu2Z2V@mail.gmail.com>
Subject: Introducing Gora, an ORM for NoSQL data stores
From: Enis Soztutar <enis.soz.nutch@gmail.com>
To: general@hadoop.apache.org, common-user@hadoop.apache.org, 
	user@avro.apache.org, user@hbase.apache.org, gora-dev@googlegroups.com
Content-Type: multipart/alternative; boundary=0016e64769e4c203c8048acf1962
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e64769e4c203c8048acf1962
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi all,

First off, sorry for cross-posting, but I think the announcement is relevan=
t
to hadoop, hbase and avro communities, all.

We would like to introduce a new project, called Gora. Gora is a Java ORM
layer for column stores, SQL databases, key-value stores and document
databases. The design goal is to have a common API to access and manage
multiple data stores. Gora differs from Java ORM frameworks, in that the
special focus is given to column oriented data bases, like Apache HBase and
Apache Cassandra.

Gora, in no way, is a replacement for Hibernate, DataNucleus or [insert the
name of your favorite ORM project]. But we think we differ from traditional
data stores from the following perspectives.
 - Gora is specifically designed with NoSQL data stores in mind. For
example, the API is based on <key, value> pairs, rather than just beans.
Also, we believe that the ORM layer should be tuned for batch operations
(like first class object re-use support),
 - Gora uses Avro, to generate data beans from avro schemas. Moreover, most
of the serializations are delegated to avro. For example, a map is
serialized to a field (if not configured otherwise) using Avro
serialization.
 - Gora provides first-class support for Hadoop MapReduce. DataStore
implementations are responsible for partitioning the data (which is then
converted to Hadoop Splits), and all the locality information is again
obtained from the data store. Developing MapReduce jobs with Gora is really
easy.
 - The long term goal for Gora is to be an intermediate data format for
popular big-data and search related projects. In the middle term, we plan t=
o
support Cassandra, Cascading, Pig and Solr. Think of the possibilities when
you can use the same data structures to persist objects to Hbase, SQL and
Solr. And use Pig or Cascading in jobs to mine the data stored at
HBase/Cassandra/SQL/etc.

Gora works as follows. You define the data structures for your domain using
regular Avro Json schemas. Then instead of compiling the avro files with
Avro's compiler, you compile the files with GoraCompiler. Generated keep
track of the persistency information along with the data. Then for each dat=
a
back-end, you define a mapping file which contains class fields to data
store specific schema configuration. For example, HBase mapping files,
define the column families, and mappings from fields to columns or column
families, whereas SQL mapping files define mappings for table fields.

Gora has started in NutchBase(http://github.com/dogacan/nutchbase), a branc=
h
of Apache Nutch(http://nutch.apache.org/) which is being used as a basis fo=
r
what will become Nutch 2.0. For the second version of the popular open
source web search project, an abstraction layer was needed so that the core
data structures for Nutch would no longer be kept as flat files on Hadoop.
We wanted to be able to use popular NoSQL databases (HBase, Cassandra,
Hypertable, etc), optionally flat files, and SQL databases (especially
embedded zero-conf SQL databases). So Gora as a project was born.

Gora is now in pre-alpha stage, with a public release planned before the en=
d
of the year. Documentation is also very sparse at this point. However, the
code is already used at NutchBase and will be used in Nutch 2.0. We
currently support HBase, plain Avro data files and SQL. Cassandra support i=
s
coming soon. Of course, the current set of developers is very small, and we
need your help in achieving these goals. So feel free to contribute in any
way you see fit. We believe in the Apache way of development and in fact,
one of the possible paths for Gora is to be accepted as a sub project of
Incubator or Hadoop (we welcome any feedback on this).

Lastly, you can find the project at http://github.com/enis/gora/. Some
example code is at
http://github.com/enis/gora/tree/master/gora-core/src/examples/ and
http://github.com/dogacan/nutchbase.

Feel free to use this list, or gora-dev@googlegroups.com for further
discussion.

Thanks,
Enis S=C3=B6ztutar

tl;dr Gora is an ORM layer with a specific focus on NoSQL data stores. It
has HBase, SQL, Avro and Mapreduce support.

--0016e64769e4c203c8048acf1962--

From general-return-1777-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 07 18:32:29 2010
Return-Path: <general-return-1777-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 99989 invoked from network); 7 Jul 2010 18:32:29 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jul 2010 18:32:29 -0000
Received: (qmail 50346 invoked by uid 500); 7 Jul 2010 18:32:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50231 invoked by uid 500); 7 Jul 2010 18:32:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50223 invoked by uid 99); 7 Jul 2010 18:32:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 18:32:27 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 18:32:21 +0000
Received: by pwj8 with SMTP id 8so3007673pwj.23
        for <hadoop-general@apache.org>; Wed, 07 Jul 2010 11:31:58 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.142.215.7 with SMTP id n7mr8612979wfg.63.1278527517831; Wed, 
	07 Jul 2010 11:31:57 -0700 (PDT)
Received: by 10.142.48.11 with HTTP; Wed, 7 Jul 2010 11:31:57 -0700 (PDT)
Date: Wed, 7 Jul 2010 11:31:57 -0700
Message-ID: <AANLkTildXbBQzm4HhOeuLm9N9oRafz83FczT1cozysA7@mail.gmail.com>
Subject: Announcement: XLDB 4
From: Jeff Hammerbacher <hammer@cloudera.com>
To: hadoop-general@apache.org
Content-Type: multipart/alternative; boundary=000e0cd3039e8932ac048ad062b4
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd3039e8932ac048ad062b4
Content-Type: text/plain; charset=ISO-8859-1

Hey,

I'm helping put together a conference on scientific data management, and I
think the Hadoop community has a lot to say on this front. If you have an
interest in this area and will be in the Bay Area the first week of October,
please read on! Otherwise, my apologies for the noise.

4th Extremely Large Databases (XLDB4) Conference
October 6-7, 2010
SLAC National Accelerator Laboratory
Menlo Park, California

XLDB, now including a conference open to all interested parties, is the
gathering place for the community managing and analyzing data at extreme
scale.

Leading practitioners from science, industry, and academia will discuss
lessons learned and real-world solutions for handling terabytes and
petabytes of data. Experts will present emerging trends that will
significantly impact how extremely large databases are built and used.

More information, including the program, and online registration are
available at <http://www.xldb.org/4>. Space is limited, so if you would like
to attend, make sure to register soon. Early registration ($100) ends on
July 31; late registration is $150.

If you are uncertain whether to attend, I would recommend reading the
article recently written by Curt Monash: <
http://www.dbms2.com/2010/07/01/why-you-should-go-to-xldb4/>

For questions, please contact us by email using <
xldb-admin@slac.stanford.edu> or contact me directly at <hammer@cloudera.com
>.

Regards,
Jeff

--000e0cd3039e8932ac048ad062b4--

From general-return-1778-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 01:19:51 2010
Return-Path: <general-return-1778-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27721 invoked from network); 8 Jul 2010 01:19:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 01:19:51 -0000
Received: (qmail 50727 invoked by uid 500); 8 Jul 2010 01:19:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50471 invoked by uid 500); 8 Jul 2010 01:19:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50456 invoked by uid 99); 8 Jul 2010 01:19:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 01:19:46 +0000
X-ASF-Spam-Status: No, hits=4.2 required=10.0
	tests=FS_REPLICA,HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [208.65.144.65] (HELO p01c11o142.mxlogic.net) (208.65.144.65)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 01:19:36 +0000
Received: from unknown [216.166.12.98]
	by p01c11o142.mxlogic.net(mxl_mta-6.7.0-0)
	with SMTP id 297253c4.0.37537.00-359.95847.p01c11o142.mxlogic.net (envelope-from <aramakrishnan@languageweaver.com>);
	Wed, 07 Jul 2010 19:19:16 -0600 (MDT)
X-MXL-Hash: 4c352794280f37ae-671998c7a9202917c62b6def895b8f0cb7835f9a
Received: from AUSP01VMBX08.collaborationhost.net ([10.2.8.97]) by
 AUSP01MHUB07.collaborationhost.net ([10.2.8.242]) with mapi; Wed, 7 Jul 2010
 20:18:57 -0500
From: Arun Ramakrishnan <aramakrishnan@languageweaver.com>
To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>,
	"general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 7 Jul 2010 20:18:56 -0500
Subject: rebalancing replication help
Thread-Topic: rebalancing replication help
Thread-Index: AcseOv4vjF6u6yf8SfScJKEMXDcPRAAAEigg
Message-ID: <C3AD6464AC81DC4AB14FFEA31391866A79321409AA@AUSP01VMBX08.collaborationhost.net>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C3AD6464AC81DC4AB14FFEA31391866A79321409AAAUSP01VMBX08c_"
MIME-Version: 1.0
X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010070601)]
X-MAIL-FROM: <aramakrishnan@languageweaver.com>
X-SOURCE-IP: [216.166.12.98]
X-AnalysisOut: [v=1.0 c=1 a=wbKXeunVgZ0A:10 a=VphdPIyG4kEA:10 a=IRTT+DMug4]
X-AnalysisOut: [7mVLgYdXzmDg==:17 a=GBQTkU0xDUXbRKSuH2oA:9 a=GDFg6kl203xBh]
X-AnalysisOut: [7Lwvs9DWpIHVtAA:4 a=CjuIK1q_8ugA:10 a=SSmOFEACAAAA:8 a=Y2V]
X-AnalysisOut: [NeNrzAAAA:8 a=yMhMjlubAAAA:8 a=TW66zc2HAAAA:8 a=HQ31llbKAA]
X-AnalysisOut: [AA:8 a=KOJE2Elat-tJl6-joL0A:9 a=n_mscvfbrtTaj5My_nwA:7 a=e]
X-AnalysisOut: [bbMM1CHFjmRuFgwP5J0u5ZpRi8A:4]
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C3AD6464AC81DC4AB14FFEA31391866A79321409AAAUSP01VMBX08c_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Looks like there is not much activity in the hdfs-user list. So, am reposti=
ng it in the general list.

Hi guys.
  I have a few related questions. I am going to layout the steps I have tak=
en. Please comment on what I can do better.

  I was trying to to add 5 nodes to my existing 10 node cluster and also in=
crease the replication factor from 2 to 3.
I thought I don't have to run the balancer cause it would most likely put t=
he new replicas into the new nodes.

There are about 500k blocks.
I wanted to get it all stabilized(replication and balancing) within 24 hour=
s. Its more than 24 hours now and fsck reports 30% under replication. Is th=
ere a way to force hdfs to use balance/replicate more aggressively.

It would be great if someone explained what/when things happen to blocks in=
 the context of

1)      Rebalancing

2)      -setrep

3)      Restarting cluster with a higher/lower replication factor.

A few questions and a few issues here.

1)      When you restart the cluster with a higher than previous replicatio=
n value. Does it also apply to existing blocks or only to new blocks being =
created ?

2)      Does the balancer take into account under replication of blocks or =
does it blindly start moving existing blocks to reach threshold ?


A very specific problem .  I am having this strange problem where the -setr=
ep hangs on one particular block for hours. Is this because its corrupt ?. =
But, fsck said its healthy.


Thanks
Arun


--_000_C3AD6464AC81DC4AB14FFEA31391866A79321409AAAUSP01VMBX08c_--

From general-return-1779-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 01:38:34 2010
Return-Path: <general-return-1779-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29447 invoked from network); 8 Jul 2010 01:38:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 01:38:34 -0000
Received: (qmail 67235 invoked by uid 500); 8 Jul 2010 01:38:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67022 invoked by uid 500); 8 Jul 2010 01:38:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67014 invoked by uid 99); 8 Jul 2010 01:38:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 01:38:32 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eltonsky9404@gmail.com designates 209.85.216.176 as permitted sender)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 01:38:24 +0000
Received: by qyk12 with SMTP id 12so3359716qyk.14
        for <general@hadoop.apache.org>; Wed, 07 Jul 2010 18:37:03 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=MUNMD0XZzjd87JD2QO5q6SmoWY31lWcNHe+8PWHh/fA=;
        b=rgASaW8ktU1vxn92QROuc9/zIf6bfr8BPIKylmlb+PnEO5qvxoU4JFAR/buDVoEnm9
         9YypJZWWvPRJSK6PSNlo3UpuYdtMGLsHNVSdh+v6dxDs+EzeG0oPbJGFnKtKKTo3gqxM
         BBzy6Zm8/drlLAl7pdFMfpF6HAwZQ3wSRmz20=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=RL4utIXZMruIHdh13AS2aDQH83qEA2X1zCzvOq/ktyl6NLno/SkQS4DwphMEAjd/Rx
         wBzTyUXcDomMivgYwmsNrCZP6h5qdjB2Q7s/oU9WukBoG96rbCKM2paYmsb29GP7pe9F
         C50X00ENwr9vvAl/T+b4loRnuk9p69FIcoXK8=
MIME-Version: 1.0
Received: by 10.224.43.232 with SMTP id x40mr2192724qae.73.1278553023698; Wed, 
	07 Jul 2010 18:37:03 -0700 (PDT)
Received: by 10.224.89.18 with HTTP; Wed, 7 Jul 2010 18:37:03 -0700 (PDT)
In-Reply-To: <4C344CBC.4090307@apache.org>
References: <AANLkTiljgFWtvH2dQHhB44-8mXQKlAhtOPp5N1K3tJ3S@mail.gmail.com>
	<AANLkTikCO_TnPI-xBt_QQljWkTBWJ4lq3nGhHPSEjYNr@mail.gmail.com>
	<ABC24175AFD3BE4DA15F4CD375ED413D06084AF156@hq-ex-mb02.ad.navteq.com>
	<AANLkTilk2wfZ10leX0iEeNwJozMDdfHNiQ09rEAnbPDd@mail.gmail.com>
	<AANLkTimmN23EV17S1g-2LITHQrasMmh6qbSj13CJyfb5@mail.gmail.com>
	<AANLkTikNr4C2mjzLSwxUFglxtbfEalCwQ8clyzCNl_8a@mail.gmail.com>
	<AANLkTinC804Q953e4_4x80U_3l36EUERH4UNOz1I30Ww@mail.gmail.com>
	<5C9D1E1E-8299-4E97-B552-2DF69F5231BE@linkedin.com>
	<AANLkTimp7y0EYCThQ4WTKx6lW6BGYgkHHEocI3EV94q_@mail.gmail.com>
	<AANLkTinHouGalTJqqemJEb55fXme9qXaG0cg4rI0X7I-@mail.gmail.com>
	<COL117-W1485F3B9503653BB189B938FB20@phx.gbl>
	<4C334C20.9010105@apache.org>
	<AANLkTik7k_H6_1rtDiY5e_fxwpX7l9HrB1wvb6CD6iD8@mail.gmail.com>
	<4C344CBC.4090307@apache.org>
Date: Thu, 8 Jul 2010 11:37:03 +1000
Message-ID: <AANLkTil4PPlYvc2AhwunJiXX3_O655oMQyKLJhdo0_fO@mail.gmail.com>
Subject: Re: Why single thread for HDFS?
From: elton sky <eltonsky9404@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00c09f89958dcdda02048ad65256
X-Virus-Checked: Checked by ClamAV on apache.org

--00c09f89958dcdda02048ad65256
Content-Type: text/plain; charset=ISO-8859-1

Steve,

I do have access to that code if I can get at the right bit of the
> repository, if you really want me to look at it in detail ask, with the
> caveats that I'm away for the rest of the month and somewhat busy. Apart
> from that there's no reason why I shouldn't be able to make the changes to
> DfsClient public. Keep reminding me :)
>

Sounds great. can you please make me know (email or other) how can I access
to that code.


>
>

>
I think right now you get a list of blocks via
> DfsClient.getBlockLocations(); this is a list of hosts where blocks live.
> There is no data about which disk on the specific host.
>
> I belive that what Russ did was move the decisions from DfsInputStream
> -which picks a block location for you, with a bias to the local host- and
> instead lets the calling program make the decision as to where to fetch each
> block. This meant he could set the renderer up to request blocks from
> different hosts.
>
> He had tried to use the JT to schedule the rendering code, but that didn't
> work as MapReduce has the notion of "reduction": less data out than in, so
> it moves work to where the data is. In rendering it's more MapExpand; the
> operation is the transformation of PDF pages into 600dpi 32bpp bitmaps,
> which then need to be streamed to the (very large) printer at its print
> rate, in the correct order. It was easiest to have a specific machine on the
> cluster -with no datanodes or TTs- set up to do the rendering, and just ask
> the filesystem for where things are.
>
> Like I said, I don't think there was anything tricky done in DfsClient,
> more a matter of making some data known internally to the  DfsClient code
> public, so that the client app can decide where to fetch data. If the
> DfsClient knew which HDD the data was on in a datanode, the client app could
> use that in its decision making too, so that if the 9 machines each had 6
> HDDs, you could keep them all busy.
>

Firstly, because nodes in a cluster are usually VMs. But if each vm attached
to multiple physical disk, we can parallel;


>If you're talking about M/R jobs, you don't want to do threads in your
map() routine, while this is possible, its going to be really hard >to
justify the extra parallelism along with the need to wait for all of the
threads to complete before you can end the map() method.

Secondly, I agree with Gautam, Micheal. In a MR job, maybe it's not good
idea to read input parallely in map() method. Because map reads a line from
HDFS each time to process. This solution looks simple & elegent, though it
keeps the connection to source for a long time, until the end of map.
I can think about some trick, like letting map() pull and process for the
first block worth of input, and simultaneously pull the left input data to
local disk of map() in other threads. But this sounds messy...

Aside above 2, I think we can do disk level parallel.

--00c09f89958dcdda02048ad65256--

From general-return-1780-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 07:51:45 2010
Return-Path: <general-return-1780-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 15898 invoked from network); 8 Jul 2010 07:51:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 07:51:45 -0000
Received: (qmail 40484 invoked by uid 500); 8 Jul 2010 07:51:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39719 invoked by uid 500); 8 Jul 2010 07:51:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39699 invoked by uid 99); 8 Jul 2010 07:51:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 07:51:39 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [87.248.114.165] (HELO web24101.mail.ird.yahoo.com) (87.248.114.165)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 08 Jul 2010 07:51:30 +0000
Received: (qmail 37774 invoked by uid 60001); 8 Jul 2010 07:51:09 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1278575469; bh=b0HBg+IpV38CeoWlkZfAH5wJVxzB671lxOkwMB1t8Ko=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:MIME-Version:Content-Type; b=YGGMdRzByw+MkhKJymSl3awvKS7EdpDVc/yO5uQSqaZDh9zjXfu1a2rcYHcJEoV9qDHQX968bq6qJfVeQXz1K/8oFDE+XxSEtDNBOv8jQUZU5zyhUDymMEXXdMlC3vr3FHsqQByJMSF8I7omyCE8acI9u00Qrxo3lwd46RfB+rA=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:MIME-Version:Content-Type;
  b=LVC7jUFgm1pN0/aX8dSjwx39EmBpkHtz55Z6XfZMFmaRohJCyU5EaD9Gj2Rw1NA/1i0+I/yV0u31pjbeZFpsPs94rBZgNDBQCOTP9cevOccDPeKi3kpR/gpkC5WoJrCLFAOCqAY7H5mggI5acFlnBZdmg017PFg8VxUY9YgXIYo=;
Message-ID: <328470.31709.qm@web24101.mail.ird.yahoo.com>
X-YMail-OSG: Yd6XupsVM1lkgG59mHbrfgxq3VtXk5eDz0ps1IGDI2LBp0U
 JO_DUDkKPWidy.p3x6TIz8NMBovrXjzkgsYcLINjG3ijY_Bcp3yXtE_YZd.c
 .wk2G4b.AhLYunfZVdd1v9nvPlQBeKzkqyFbNgmlVygAIjHgnP0ab.jpahn5
 A7U2q_Cl9DqldBkcz777rZEGNYrMa9TFL5h4f
Received: from [188.74.76.201] by web24101.mail.ird.yahoo.com via HTTP; Thu, 08 Jul 2010 07:51:09 GMT
X-Mailer: YahooMailRC/420.4 YahooMailWebService/0.8.104.274457
Date: Thu, 8 Jul 2010 07:51:09 +0000 (GMT)
From: Denim Live <denim.live@yahoo.com>
Subject: Finding the time it took for a mapreduce job to get executed
To: general@hadoop.apache.org
Cc: mapreduce-user@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1904840433-1278575469=:31709"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-1904840433-1278575469=:31709
Content-Type: text/plain; charset=us-ascii

Hi folks,

I want to determine the exact time it took for my mapreduce job to get executed 
for some anaylsis purpose. How can I calculate it?

Thanks


      
--0-1904840433-1278575469=:31709--

From general-return-1781-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 08:09:06 2010
Return-Path: <general-return-1781-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22695 invoked from network); 8 Jul 2010 08:09:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 08:09:05 -0000
Received: (qmail 50706 invoked by uid 500); 8 Jul 2010 08:09:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50675 invoked by uid 500); 8 Jul 2010 08:09:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50667 invoked by uid 99); 8 Jul 2010 08:09:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 08:09:00 +0000
X-ASF-Spam-Status: No, hits=3.8 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_BRBL_LASTEXT,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of v.amit@verchaska.com designates 203.123.141.155 as permitted sender)
Received: from [203.123.141.155] (HELO mail.verchaska.com) (203.123.141.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 08:08:52 +0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by mail.verchaska.com (Postfix) with ESMTP id F095311DBAF4
	for <general@hadoop.apache.org>; Thu,  8 Jul 2010 13:37:06 +0530 (IST)
X-Virus-Scanned: amavisd-new at mail.verchaska.com
Received: from mail.verchaska.com ([127.0.0.1])
	by localhost (mail.verchaska.com [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id utawjIFMvHOn for <general@hadoop.apache.org>;
	Thu,  8 Jul 2010 13:37:02 +0530 (IST)
Received: from [192.168.0.186] (unknown [192.168.0.186])
	by mail.verchaska.com (Postfix) with ESMTP id 4598B11DBAF0
	for <general@hadoop.apache.org>; Thu,  8 Jul 2010 13:37:02 +0530 (IST)
Message-ID: <4C35877A.3090705@verchaska.com>
Date: Thu, 08 Jul 2010 13:38:26 +0530
From: amit kumar verma <v.amit@verchaska.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.2.4) Gecko/20100608 Thunderbird/3.1
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: jni files
Content-Type: multipart/alternative;
 boundary="------------050806000704060908020706"
X-Virus-Checked: Checked by ClamAV on apache.org

--------------050806000704060908020706
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

  Hi,

I developed a project which is using some native jni files 
(liblemur_jni.so), earlier i use to run application jar by using 
-Djava.library.path=/PATH_TO_JNI_FILES, but am not able to the same with 
./hadoop jar command.

I followed 
http://hadoop.apache.org/common/docs/r0.18.3/native_libraries.html

   1. First copy the library to the HDFS.
      bin/hadoop fs -copyFromLocal mylib.so.1 /libraries/mylib.so.1
   2. The job launching program should contain the following:
      DistributedCache.createSymlink(conf);
      DistributedCache.addCacheFile("hdfs://*
      /192.168.0.153:50075*/libraries/mylib.so.1#mylib.so", conf);
   3. The map/reduce task can contain:
      System.loadLibrary("mylib.so");

  but getting error :

Exception in thread "main" java.io.IOException: Call to* 
/192.168.0.153:50075* failed on local exception: java.io.EOFException
         at org.apache.hadoop.ipc.Client.wrapException(Client.java:775)
         at org.apache.hadoop.ipc.Client.call(Client.java:743)
         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
         at $Proxy1.getProtocolVersion(Unknown Source)
         at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
         at 
org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
         at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
         at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
         at 
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
         at 
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
         at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
         at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
         at 
org.apache.hadoop.filecache.DistributedCache.getTimestamp(DistributedCache.java:506)
         at 
org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:640)
         at 
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
         at org.apache.hadoop.mapreduce.Job.submit(Job.java:432)
         at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:447)
         at com.i4dweb.trobo.grid.WordCountNew.main(WordCountNew.java:49)
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
         at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
         at java.lang.reflect.Method.invoke(Method.java:597)
         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.io.EOFException
         at java.io.DataInputStream.readInt(DataInputStream.java:375)
         at 
org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:508)
         at org.apache.hadoop.ipc.Client$Connection.run(Client.java:446)


Please advice.

-- 
Thanks,
Amit Kumar Verma
Verchaska Infotech Pvt. Ltd.



--------------050806000704060908020706--

From general-return-1782-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 08:56:57 2010
Return-Path: <general-return-1782-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31104 invoked from network); 8 Jul 2010 08:56:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 08:56:57 -0000
Received: (qmail 99176 invoked by uid 500); 8 Jul 2010 08:56:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98817 invoked by uid 500); 8 Jul 2010 08:56:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98799 invoked by uid 99); 8 Jul 2010 08:56:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 08:56:51 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [68.180.199.169] (HELO web46304.mail.sp1.yahoo.com) (68.180.199.169)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 08 Jul 2010 08:56:42 +0000
Received: (qmail 57037 invoked by uid 60001); 8 Jul 2010 08:56:20 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1278579380; bh=/JFpmzy5kxbeqd88J2jwrEYcw7sVWwTc+DiunYF9A7g=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:MIME-Version:Content-Type; b=WRb0A/+Emrh4P8XDzJOeLvF1/VPKUB2Bdz5wnwZN/RO5QhgmvVSRTyp6U0oCsuLKGbSNxzjJvssJOBFQQmhTYPnx8NsDjaJ8WrSfik/180EvjsJd/r9RmB068cTh/Nft05sTL3jzecVk/Eq3CUIlp00fQGgAmmJdr9B8ezZqvp8=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:MIME-Version:Content-Type;
  b=YEMt1oF0CPOPZSIAK6I6yfw3m5Jlc5bbKYllLADzeoQJvaBXhuyMdZrY+sGJI3TANFxJ4WnHSJQ3MRTvAG8cmdKT00GWnQ+4w6FFWURpiOPeamV3eo+Y778FEAc5qL9sJ55Mbs2h4+k/Yz52GtvxJWTDfcWEhOPDJ9k8MazbZxM=;
Message-ID: <287553.55933.qm@web46304.mail.sp1.yahoo.com>
X-YMail-OSG: zDKRh5MVM1mn0A1zEDyQECjtwwzuZkVgRwxbTzdmA.CS.0g
 3jBPW9NP7FNc9Y9bNtp08i.vtYTfud_I9uQpzH_ehDajz9uSiZKa0aN7JZHF
 DTra7T8QmEPeLodYIMyQ4MoFvSZ2nDWj1.5.6JF0eNUb7TuWX2cxJ8pF9wkJ
 U2Uk0CzZ_EZkrgqL01gohDnX5ylhlqALHh3ixobfsF6zO2Jj33jhpo4PwMuw
 .O7SHxgbAApZTsRvthhDCXCuep_heE.qcXakVQmJQqDLPtkhH2RqXGWJugRE
 LDeDX6jM5vWDEZtMHbEBdmB._905XtfGg1n8-
Received: from [188.74.76.201] by web46304.mail.sp1.yahoo.com via HTTP; Thu, 08 Jul 2010 01:56:20 PDT
X-Mailer: YahooMailRC/420.4 YahooMailWebService/0.8.104.276605
Date: Thu, 8 Jul 2010 01:56:20 -0700 (PDT)
From: abc xyz <fabc_xyz111@yahoo.com>
Subject: getting the partition number in a reducer to which a key-value pair belongs
To: common-user@hadoop.apache.org
Cc: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1148301366-1278579380=:55933"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-1148301366-1278579380=:55933
Content-Type: text/plain; charset=us-ascii

Hello everyone,
when i am processing a given key-{set of values} pair in reducer function, how 
can I get the partition number to which this key-{set of values} belong to? How 
is it possible to get this partition number without adding extra information 
about the partition number with each key-value pair during partitioning?
Cheers



      
--0-1148301366-1278579380=:55933--

From general-return-1783-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 09:13:49 2010
Return-Path: <general-return-1783-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38432 invoked from network); 8 Jul 2010 09:13:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 09:13:49 -0000
Received: (qmail 27090 invoked by uid 500); 8 Jul 2010 09:13:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26878 invoked by uid 500); 8 Jul 2010 09:13:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26866 invoked by uid 99); 8 Jul 2010 09:13:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 09:13:45 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [68.180.199.166] (HELO web46301.mail.sp1.yahoo.com) (68.180.199.166)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 08 Jul 2010 09:13:36 +0000
Received: (qmail 16562 invoked by uid 60001); 8 Jul 2010 09:13:14 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1278580394; bh=Ioyj8+wQUEhZvtLEzDwX8MxEh3hgWox8+yvS5/oufAg=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:MIME-Version:Content-Type; b=p8dLWh9sqIJt0N9DjGrFNIYc2/ByG6arHsKUm+9j3F9ynGWAHbDbtwglw1tfrZMe/6TcKL4gT6V1v5EiM5E/rOWqzxM7KxweoDUCWT6AvG2GoRJ3JeH5LQspsXCDAaqyTgxci3IERkatQcy8SHaSwy6lg/Z3eHb/c41s/F+wCF4=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:MIME-Version:Content-Type;
  b=6KpwsUVJh3hQXuTal6JhdWkwKNnmuUExsIqqf2c6OLo6cRAUFPr8WN+V4YDGfrYIFG/FWYShb71yVjRKNvBCB8f9bHdWGQNUC4oEiX9RrOYe69Lh3fBafYwUUbWP+YqZi/ly8gQpQAMVEZ0Oquud17oHOVV/A5xF1aWFvsBFPK4=;
Message-ID: <300130.9635.qm@web46301.mail.sp1.yahoo.com>
X-YMail-OSG: pczUS6wVM1lz8xs8r7BIjg6MvekqsI0_jQz5sOC8JB34xQu
 uOngel4rQbom7Fz2FYf.qIBc7nGDXYWvAircHK9Fcx894RnftsSABITSBTjH
 qmXwq.lqWPbKvfb9FMTRVPOcuKORrv_1lZmEsGYNNcprrFnvLohVAGclKzQN
 6cf0zPEoTjVyKllFzBS4SuWZWPNsZAndRQaL_NCrsReIzmSHWH13chR0Oo74
 5BJt9oOf6dY4HrI5tnwAKrjfHUuwO.at8eU.h2afemJTObxwz2c_l62i_0.w
 8_4yxkO6VAV5bF4_HGsfgh5s6JS9TQIP.IRmi
Received: from [188.74.76.201] by web46301.mail.sp1.yahoo.com via HTTP; Thu, 08 Jul 2010 02:13:14 PDT
X-Mailer: YahooMailRC/420.4 YahooMailWebService/0.8.104.276605
Date: Thu, 8 Jul 2010 02:13:14 -0700 (PDT)
From: abc xyz <fabc_xyz111@yahoo.com>
Subject: naming the output fle of reduce to the partition number
To: common-user@hadoop.apache.org
Cc: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-780567361-1278580394=:9635"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-780567361-1278580394=:9635
Content-Type: text/plain; charset=us-ascii

Hi Everyone,
I am having some problem with naming the output file of each reduce task  with 
the partition number. First of all, how can  I get the partition number  within 
each reduce? Second, How am I going to name the output file with that  partition 
number? 

I have looked to the MultipleTextOutputFormat. It can generate a new file  with 
the name of my choice for each key. But I want to name the output file for  each 
partition with the name of my choice i.e. with the partition number. 

Please help me in this regard.
 
Thanks


      
--0-780567361-1278580394=:9635--

From general-return-1784-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 09:35:24 2010
Return-Path: <general-return-1784-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43205 invoked from network); 8 Jul 2010 09:35:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 09:35:24 -0000
Received: (qmail 62098 invoked by uid 500); 8 Jul 2010 09:35:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61700 invoked by uid 500); 8 Jul 2010 09:35:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61687 invoked by uid 99); 8 Jul 2010 09:35:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 09:35:19 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [68.180.199.168] (HELO web46303.mail.sp1.yahoo.com) (68.180.199.168)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 08 Jul 2010 09:35:08 +0000
Received: (qmail 73291 invoked by uid 60001); 8 Jul 2010 09:34:46 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1278581686; bh=usJmeLvnQSGFptpaO/pfyI7spQQCzuHCcETUndVKZGc=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=tOXaRsagHEPlgM7Fmu3G632pniJ/+0wv10RUOGnTfYE4tJVlFyQwnrmU1UCwhWiCPAmgi1r38nFgt7UTiAuuzOac64KX6H4hspLQLOSlbAJ0CxhtxmyMqRM01Y7JWFh6i+EFVJk7JhAdHhbPR6479SXb7yPZcTayO4lc0noPus0=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=BYqNGneOOZo6ljmdjjtYNHLH0V4fU/ix6rJzFl11xg7u00ewLltD8T32SPa1/GGxmMK1iL5mVc3lrfr8G6rTgPsVA8gScMQ5FVHIXHeXVS6dGJ/t1vMlvHUcxsQLQ4Ex6T+NTtcYBUxynNd82TGDvy/wiZfENCU3un3rjShBJcY=;
Message-ID: <383036.66480.qm@web46303.mail.sp1.yahoo.com>
X-YMail-OSG: imK5t70VM1kFwlvdv.hrEgG1AYCpk3ZzHUWvtiLoHHZJeZm
 PwPfY.gRoQwNhZ_4_hVmVulDmOFYO3nZ6W2Sa5NOl2mdcL2sj7JPFqL1hDkz
 vz1AmVbU3vhfaISwi06c.YmB2NG9refRVBB3mIvmKwj1jW8unkSP1ihZkCA6
 x0r391io614G_j5_KKU6V_dw3nlWv4muhuNgEHCpKK0Ffauihzq2dBeEJG.t
 WsWMgXg_WJ4Qrr7TcSiVTGmKhJ9N.jZt6JwAXuXqs1LmrcZAIUiYC_YwS7M1
 0I8rGYzBMXybcZmGm4s5eFFIybpA2D3.HUCzWQSY28LMxBWdhA2Zhr3wFhIi
 ZSZaQvSbS
Received: from [188.74.76.201] by web46303.mail.sp1.yahoo.com via HTTP; Thu, 08 Jul 2010 02:34:46 PDT
X-Mailer: YahooMailRC/420.4 YahooMailWebService/0.8.104.276605
References: <287553.55933.qm@web46304.mail.sp1.yahoo.com>
Date: Thu, 8 Jul 2010 02:34:46 -0700 (PDT)
From: abc xyz <fabc_xyz111@yahoo.com>
Subject: Re: getting the partition number in a reducer to which a key-value pair belongs
To: general@hadoop.apache.org
In-Reply-To: <287553.55933.qm@web46304.mail.sp1.yahoo.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1465763519-1278581686=:66480"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-1465763519-1278581686=:66480
Content-Type: text/plain; charset=us-ascii

This has worked for me:
jobconf.getInt( "mapred.task.partition",  0);
in the reducer.





________________________________
From: abc xyz <fabc_xyz111@yahoo.com>
To: common-user@hadoop.apache.org
Cc: general@hadoop.apache.org
Sent: Thu, July 8, 2010 9:56:20 AM
Subject: getting the partition number in a reducer to which a key-value pair 
belongs

Hello everyone,
when i am processing a given key-{set of values} pair in reducer function, how 
can I get the partition number to which this key-{set of values} belong to? How 
is it possible to get this partition number without adding extra information 
about the partition number with each key-value pair during partitioning?
Cheers


      
--0-1465763519-1278581686=:66480--

From general-return-1785-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 17:27:31 2010
Return-Path: <general-return-1785-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91771 invoked from network); 8 Jul 2010 17:27:31 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 17:27:31 -0000
Received: (qmail 61625 invoked by uid 500); 8 Jul 2010 17:27:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61438 invoked by uid 500); 8 Jul 2010 17:27:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61430 invoked by uid 99); 8 Jul 2010 17:27:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 17:27:29 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 17:27:22 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version;
  b=d/qM3Yd/htXdff5crizTCEadVlAKh6B0xsiDVgU6ydW5OxVso0U6eM+o
   fa4B+9FJylWKjsvCOakFMUR26F7hMEAGH7UoDn1xgOO78GuxRVXmCBO4/
   GUSsFwfycWc31Us;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1278610042; x=1310146042;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20jni=20files|Date:=20Thu,=208=20Jul=2020
   10=2017:27:00=20+0000|Message-ID:=20<354823AA-8AF3-4FF4-9
   606-EAAD236E1835@linkedin.com>|To:=20"<general@hadoop.apa
   che.org>"=20<general@hadoop.apache.org>|MIME-Version:=201
   .0|Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<b3b09244-72b5-41f7-9a15-4b027e61d7a5>
   |In-Reply-To:=20<4C35877A.3090705@verchaska.com>
   |References:=20<4C35877A.3090705@verchaska.com>;
  bh=2sWaZZpqHubMqloRFnrXr2LdQ5eMLaewfYHraE6PEcs=;
  b=glUz1suV+KFYD8pzYCvuk1j9V+Q68WPfW0hOI/X6ggsFv2GkgiVqlS9D
   taOBiewwv94Ca/exMrunacX0q0gBvHAAGQmHPz+UPhMlpRXeImp1TI0rS
   CutYafWtL548+OF;
X-IronPort-AV: E=Sophos;i="4.53,559,1272870000"; 
   d="scan'208";a="13912495"
Received: from ESV4-EXC02.linkedin.biz ([fe80::4d74:48bd:e0bd:13ee]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi; Thu, 8 Jul 2010 10:27:01
 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: jni files
Thread-Topic: jni files
Thread-Index: AQHLHnTVV1c7eHpkqEaCeEw37pEITJKnvfoA
Date: Thu, 8 Jul 2010 17:27:00 +0000
Message-ID: <354823AA-8AF3-4FF4-9606-EAAD236E1835@linkedin.com>
References: <4C35877A.3090705@verchaska.com>
In-Reply-To: <4C35877A.3090705@verchaska.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Content-Type: text/plain; charset="us-ascii"
Content-ID: <b3b09244-72b5-41f7-9a15-4b027e61d7a5>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


On Jul 8, 2010, at 1:08 AM, amit kumar verma wrote:

>     DistributedCache.addCacheFile("hdfs://*
>     /192.168.0.153:50075*/libraries/mylib.so.1#mylib.so", conf);

Do you actually have asterisks in this?  If so, that's the problem.


From general-return-1786-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 17:32:51 2010
Return-Path: <general-return-1786-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 92645 invoked from network); 8 Jul 2010 17:32:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 17:32:51 -0000
Received: (qmail 66793 invoked by uid 500); 8 Jul 2010 17:32:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66732 invoked by uid 500); 8 Jul 2010 17:32:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66724 invoked by uid 99); 8 Jul 2010 17:32:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 17:32:49 +0000
X-ASF-Spam-Status: No, hits=3.8 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_BRBL_LASTEXT,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of v.amit@verchaska.com designates 203.123.141.155 as permitted sender)
Received: from [203.123.141.155] (HELO mail.verchaska.com) (203.123.141.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 17:32:41 +0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by mail.verchaska.com (Postfix) with ESMTP id 1B0AB11DBAF0
	for <general@hadoop.apache.org>; Thu,  8 Jul 2010 23:00:11 +0530 (IST)
X-Virus-Scanned: amavisd-new at mail.verchaska.com
Received: from mail.verchaska.com ([127.0.0.1])
	by localhost (mail.verchaska.com [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id FDPsIPPj2cYa for <general@hadoop.apache.org>;
	Thu,  8 Jul 2010 23:00:08 +0530 (IST)
Received: from [192.168.0.186] (unknown [192.168.0.186])
	by mail.verchaska.com (Postfix) with ESMTP id 4B5DB11DBAEF
	for <general@hadoop.apache.org>; Thu,  8 Jul 2010 23:00:08 +0530 (IST)
Message-ID: <4C360B76.8080405@verchaska.com>
Date: Thu, 08 Jul 2010 23:01:34 +0530
From: amit kumar verma <v.amit@verchaska.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.2.4) Gecko/20100608 Thunderbird/3.1
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: jni files
References: <4C35877A.3090705@verchaska.com> <354823AA-8AF3-4FF4-9606-EAAD236E1835@linkedin.com>
In-Reply-To: <354823AA-8AF3-4FF4-9606-EAAD236E1835@linkedin.com>
Content-Type: multipart/alternative;
 boundary="------------070600040301020308000600"
X-Virus-Checked: Checked by ClamAV on apache.org

--------------070600040301020308000600
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

  Hi,

No there is no asterisks sign there, it came as i tried to make this 
*bold*.

Thanks,
Amit Kumar Verma
Verchaska Infotech Pvt. Ltd.



On 07/08/2010 10:57 PM, Allen Wittenauer wrote:
> On Jul 8, 2010, at 1:08 AM, amit kumar verma wrote:
>
>>      DistributedCache.addCacheFile("hdfs://*
>>      /192.168.0.153:50075*/libraries/mylib.so.1#mylib.so", conf);
> Do you actually have asterisks in this?  If so, that's the problem.
>
>

--------------070600040301020308000600--

From general-return-1787-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 19:03:37 2010
Return-Path: <general-return-1787-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 28338 invoked from network); 8 Jul 2010 19:03:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 19:03:37 -0000
Received: (qmail 85719 invoked by uid 500); 8 Jul 2010 19:03:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85657 invoked by uid 500); 8 Jul 2010 19:03:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85498 invoked by uid 99); 8 Jul 2010 19:03:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 19:03:34 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [87.248.114.170] (HELO web24106.mail.ird.yahoo.com) (87.248.114.170)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 08 Jul 2010 19:03:24 +0000
Received: (qmail 44981 invoked by uid 60001); 8 Jul 2010 19:02:03 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1278615723; bh=z83Oxm9MprZyaOvFUhpRV7AP9Tk01q95wQchRfY7x3Q=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:MIME-Version:Content-Type; b=kmOg3duvsNGhmd47jjrVCVunmsywueCg2GC7f0Fe/bhknSI5uIOqWoQ1LVy/YJGAx76CdetEdawElpADFBnHVIBBxvE+QLCf+4paKNpdOEeh8q5A+cqxqQHtbvJlcu3319eb28sNTzZOqwAQZ+PoSn2z0Ahw6cYpaz7rQ0jZs9Y=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:MIME-Version:Content-Type;
  b=3jHN0wjPWoUAKOlaaq4DCMVfGfj5fswk9fpvXuarZXGv2f8Y0yBJDQS4iy08vkAT0QzaMTrL5v4x2A4aluazPq3pKtA6U0Y/12OsgsBNjiH42PJu4bngpTAekoMY50IejVsLKx1bvL0PStN9in5v9ueRsSIleKHnp5Cw5zIhDAc=;
Message-ID: <863416.44787.qm@web24106.mail.ird.yahoo.com>
X-YMail-OSG: W6hwyTwVM1nmPRJIkFjrO0UkI39zkZekdYi1TBL.GBoXC4O
 Af90sOji3W4VXaDQkDk7mcXqNfSHvARq.op0yMAJy6v_RD.1ytWJemkw6FR3
 _d1cDwFgdKf3HdpaoM5H1tJLaicpy.pRn.aIKtYhPk3kKV86dP0yWBWA.3Yr
 LJ_tWQu3keM3IGiA.X1G2mQ1glJOpw8gk5qy_c_L9L5yH7QB8.eIzqpHBBxb
 Oy71PzLy0yQ--
Received: from [188.74.76.201] by web24106.mail.ird.yahoo.com via HTTP; Thu, 08 Jul 2010 19:02:03 GMT
X-Mailer: YahooMailRC/420.4 YahooMailWebService/0.8.104.274457
Date: Thu, 8 Jul 2010 19:02:03 +0000 (GMT)
From: Denim Live <denim.live@yahoo.com>
Subject: Distributed Cache file issue
To: mapreduce-user@hadoop.apache.org
Cc: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-252320042-1278615723=:44787"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-252320042-1278615723=:44787
Content-Type: text/plain; charset=us-ascii

Hello all,

As a new user of hadoop, I am having some problems with understanding some 
things. I am writing a program to load a file to the distributed cache and read 
this file in each mapper. In my driver program, I have added the file to my 
distributed cache using:
        
       Path p=new 
Path("hdfs://localhost:9100/user/denimLive/denim/DCache/Orders.txt");
        DistributedCache.addCacheFile(p.toUri(), conf);

In the configure method of the mapper, I am reading the file from cache using:
            Path[] cacheFiles=DistributedCache.getFileClassPaths(conf);
            BufferedReader joinReader=new BufferedReader(new 
FileReader(cacheFiles[0].toString()));

however, the cacheFiles variable has null value in it. 

There is something mentioned on the Yahoo tutorial for hadoop about distributed 
cache which I do not understand:

As a cautionary note: If you use the local JobRunner in   Hadoop (i.e., what 
happens if you call JobClient.runJob()in a program with no or an empty 
hadoop-conf.xmlaccessible),   then no local data directory is created; the 
getLocalCacheFiles()call will return an empty set of   results. Unit test code 
should take this into account."

what does this mean? I am executing my program in pseudo-distributed mode on 
windows using Eclipse.

Any suggestion in this regard is highly valued. 

Thanks  in advance.



      
--0-252320042-1278615723=:44787--

From general-return-1788-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 08:35:32 2010
Return-Path: <general-return-1788-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68672 invoked from network); 9 Jul 2010 08:35:31 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 08:35:31 -0000
Received: (qmail 77781 invoked by uid 500); 9 Jul 2010 08:35:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77364 invoked by uid 500); 9 Jul 2010 08:35:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77355 invoked by uid 99); 9 Jul 2010 08:35:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 08:35:27 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yhemanth@gmail.com designates 209.85.212.176 as permitted sender)
Received: from [209.85.212.176] (HELO mail-px0-f176.google.com) (209.85.212.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 08:35:21 +0000
Received: by pxi11 with SMTP id 11so1417626pxi.35
        for <general@hadoop.apache.org>; Fri, 09 Jul 2010 01:35:00 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=B8euoDIRmfHxMrhxC5JhCwKlWXGcCYRDqcdwmsottHw=;
        b=NF2kdGOynpG8GuBb/SRRh+Na+2UdQMpKS3KNOwq/NJFSG3/XaZ8bAH8YWqHakCyUwa
         LDMX0+kWU14QvjtirNb49gE1h6nHl18qPHrmGARZbO40yeksJ8SkNOUy7aK90XOXEDSI
         p0Tx9guKzj8QhXA4L/eg/oFstkGr8Memacio4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=KZGt2zxDdI4GNzubmUOO3zVZpqRKqYu6S14NldNOKdIfWzCy2YStB/JiVBsRGaLfdO
         9R4vif4LbnOc6nyzlcQVwjvmz/XPicQf9qpPb4Vs0hcYNGQa7f4YJVucBH7/CbmqMs5K
         bus77D5ejmk4qDPr9R06IO/AiH1a14emNUgh4=
MIME-Version: 1.0
Received: by 10.142.140.20 with SMTP id n20mr8604574wfd.41.1278664499918; Fri, 
	09 Jul 2010 01:34:59 -0700 (PDT)
Received: by 10.142.188.19 with HTTP; Fri, 9 Jul 2010 01:34:59 -0700 (PDT)
In-Reply-To: <354823AA-8AF3-4FF4-9606-EAAD236E1835@linkedin.com>
References: <4C35877A.3090705@verchaska.com>
	<354823AA-8AF3-4FF4-9606-EAAD236E1835@linkedin.com>
Date: Fri, 9 Jul 2010 14:04:59 +0530
Message-ID: <AANLkTikwQaAEbX9q8hoNITrU5d_E-8wWBhfYskhX9d8U@mail.gmail.com>
Subject: Re: jni files
From: Hemanth Yamijala <yhemanth@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

Possibly another silly question, but can you cross check if the
versions of Hadoop on the client and the server are the same ?

Thanks
hemanth

On Thu, Jul 8, 2010 at 10:57 PM, Allen Wittenauer
<awittenauer@linkedin.com> wrote:
>
> On Jul 8, 2010, at 1:08 AM, amit kumar verma wrote:
>
>> =A0 =A0 DistributedCache.addCacheFile("hdfs://*
>> =A0 =A0 /192.168.0.153:50075*/libraries/mylib.so.1#mylib.so", conf);
>
> Do you actually have asterisks in this? =A0If so, that's the problem.
>
>

From general-return-1789-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 09:10:14 2010
Return-Path: <general-return-1789-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81054 invoked from network); 9 Jul 2010 09:10:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 09:10:13 -0000
Received: (qmail 41378 invoked by uid 500); 9 Jul 2010 09:10:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 41009 invoked by uid 500); 9 Jul 2010 09:10:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 41001 invoked by uid 99); 9 Jul 2010 09:10:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 09:10:09 +0000
X-ASF-Spam-Status: No, hits=1.6 required=10.0
	tests=RCVD_IN_BRBL_LASTEXT,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of v.amit@verchaska.com designates 203.123.141.155 as permitted sender)
Received: from [203.123.141.155] (HELO mail.verchaska.com) (203.123.141.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 09:10:03 +0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by mail.verchaska.com (Postfix) with ESMTP id C7F5511DBAF8
	for <general@hadoop.apache.org>; Fri,  9 Jul 2010 14:37:39 +0530 (IST)
X-Virus-Scanned: amavisd-new at mail.verchaska.com
Received: from mail.verchaska.com ([127.0.0.1])
	by localhost (mail.verchaska.com [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id d-2aGEBpyd7R for <general@hadoop.apache.org>;
	Fri,  9 Jul 2010 14:37:35 +0530 (IST)
Received: from [192.168.0.186] (unknown [192.168.0.186])
	by mail.verchaska.com (Postfix) with ESMTP id 60E2211DBAF3
	for <general@hadoop.apache.org>; Fri,  9 Jul 2010 14:37:35 +0530 (IST)
Message-ID: <4C36E730.1060004@verchaska.com>
Date: Fri, 09 Jul 2010 14:39:04 +0530
From: amit kumar verma <v.amit@verchaska.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.2.4) Gecko/20100608 Thunderbird/3.1
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: jni files
References: <4C35877A.3090705@verchaska.com>	<354823AA-8AF3-4FF4-9606-EAAD236E1835@linkedin.com> <AANLkTikwQaAEbX9q8hoNITrU5d_E-8wWBhfYskhX9d8U@mail.gmail.com>
In-Reply-To: <AANLkTikwQaAEbX9q8hoNITrU5d_E-8wWBhfYskhX9d8U@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

  Hi Hemant,

The version are same as copied it to all client machine.

I think I got a solution. As I read more about hadoop and JNI, I learned 
that I need to copy jni files to 
HADOOP_INSTALLATION_DIR//lib/native/Linux-xxx-xxx. I though my linux 
machine is Linux-i386-32. then I found in 
"org.apache.hadoop.util.PlatformName" class gives you your machine type 
and its Linux-amd64-64 and asa I copied jni files to this directory 
error are not coming.

Though full code is still not running as I developed the application 
using java.file class and i am still thinking how to make changes so 
that it can access hdfs !!!  Do i need to change my all API with respect 
to HDFS and rewrite using hadoop fs or ??!!!

It will be great if someone advice on this.



Thanks,
Amit Kumar Verma
Verchaska Infotech Pvt. Ltd.



On 07/09/2010 02:04 PM, Hemanth Yamijala wrote:
> Hi,
>
> Possibly another silly question, but can you cross check if the
> versions of Hadoop on the client and the server are the same ?
>
> Thanks
> hemanth
>
> On Thu, Jul 8, 2010 at 10:57 PM, Allen Wittenauer
> <awittenauer@linkedin.com>  wrote:
>> On Jul 8, 2010, at 1:08 AM, amit kumar verma wrote:
>>
>>>      DistributedCache.addCacheFile("hdfs://*
>>>      /192.168.0.153:50075*/libraries/mylib.so.1#mylib.so", conf);
>> Do you actually have asterisks in this?  If so, that's the problem.
>>
>>

From general-return-1790-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 09:18:49 2010
Return-Path: <general-return-1790-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82627 invoked from network); 9 Jul 2010 09:18:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 09:18:49 -0000
Received: (qmail 50255 invoked by uid 500); 9 Jul 2010 09:18:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49961 invoked by uid 500); 9 Jul 2010 09:18:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49953 invoked by uid 99); 9 Jul 2010 09:18:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 09:18:44 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yhemanth@gmail.com designates 74.125.83.176 as permitted sender)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 09:18:38 +0000
Received: by pvc21 with SMTP id 21so1455434pvc.35
        for <general@hadoop.apache.org>; Fri, 09 Jul 2010 02:17:17 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=4QbP/Uuko7zzTV4hPAJXYR4Jfn/gSx1Aiu4QEob5dSw=;
        b=XFWxqPi+qvBVWL9qre2pffb7ukGxmbH5GJ3WAHe3KvvqzmDshxhyf1s5eSMH2MJ27B
         9cWMxcHMS9N9bXxc0jaVoQaSNW4WH86alkmKSgmVJGFhaFfMvm9a7Pf8HOjwyB+l8qvS
         kRZ0n8Opye5Dl1N81s5Mtmpd+FpzwtB9z5IZQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=mTKK5SEIPQnriTpx8oXrJpvfDnCYfUqGoAp3IMXnx2Uie1GdE7BnCHVKNLBV3iotUS
         ZnxKwWYZ6b3UgaY3V/lNPGJLpN7AMYxkeM4sARjL14soEAe/8vKFEKtPyVzbY8BqDGrT
         Vk0BuxyUHveSr5i/S+u991O36jgr1/fu1huUo=
MIME-Version: 1.0
Received: by 10.142.174.4 with SMTP id w4mr4257090wfe.160.1278667037172; Fri, 
	09 Jul 2010 02:17:17 -0700 (PDT)
Received: by 10.142.188.19 with HTTP; Fri, 9 Jul 2010 02:17:17 -0700 (PDT)
In-Reply-To: <4C36E730.1060004@verchaska.com>
References: <4C35877A.3090705@verchaska.com>
	<354823AA-8AF3-4FF4-9606-EAAD236E1835@linkedin.com>
	<AANLkTikwQaAEbX9q8hoNITrU5d_E-8wWBhfYskhX9d8U@mail.gmail.com>
	<4C36E730.1060004@verchaska.com>
Date: Fri, 9 Jul 2010 14:47:17 +0530
Message-ID: <AANLkTikI6BTduu0skWGNQbQFyR0lDcvDTVfzElKWqltM@mail.gmail.com>
Subject: Re: jni files
From: Hemanth Yamijala <yhemanth@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Amit,

On Fri, Jul 9, 2010 at 2:39 PM, amit kumar verma <v.amit@verchaska.com> wro=
te:
> =A0Hi Hemant,
>
> The version are same as copied it to all client machine.
>
> I think I got a solution. As I read more about hadoop and JNI, I learned
> that I need to copy jni files to
> HADOOP_INSTALLATION_DIR//lib/native/Linux-xxx-xxx. I though my linux mach=
ine
> is Linux-i386-32. then I found in "org.apache.hadoop.util.PlatformName"
> class gives you your machine type and its Linux-amd64-64 and asa I copied
> jni files to this directory error are not coming.
>
> Though full code is still not running as I developed the application usin=
g
> java.file class and i am still thinking how to make changes so that it ca=
n
> access hdfs !!! =A0Do i need to change my all API with respect to HDFS an=
d
> rewrite using hadoop fs or ??!!!
>

To access files from HDFS, you should use the Hadoop FileSystem API.
Please take a look at the Javadoc and also a tutorial such as this:
http://developer.yahoo.com/hadoop/tutorial/module2.html#programmatically
for more information.

> It will be great if someone advice on this.
>
>
>
> Thanks,
> Amit Kumar Verma
> Verchaska Infotech Pvt. Ltd.
>
>
>
> On 07/09/2010 02:04 PM, Hemanth Yamijala wrote:
>>
>> Hi,
>>
>> Possibly another silly question, but can you cross check if the
>> versions of Hadoop on the client and the server are the same ?
>>
>> Thanks
>> hemanth
>>
>> On Thu, Jul 8, 2010 at 10:57 PM, Allen Wittenauer
>> <awittenauer@linkedin.com> =A0wrote:
>>>
>>> On Jul 8, 2010, at 1:08 AM, amit kumar verma wrote:
>>>
>>>> =A0 =A0 DistributedCache.addCacheFile("hdfs://*
>>>> =A0 =A0 /192.168.0.153:50075*/libraries/mylib.so.1#mylib.so", conf);
>>>
>>> Do you actually have asterisks in this? =A0If so, that's the problem.
>>>
>>>
>

From general-return-1791-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 09:31:37 2010
Return-Path: <general-return-1791-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84845 invoked from network); 9 Jul 2010 09:31:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 09:31:37 -0000
Received: (qmail 60858 invoked by uid 500); 9 Jul 2010 09:31:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60566 invoked by uid 500); 9 Jul 2010 09:31:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60558 invoked by uid 99); 9 Jul 2010 09:31:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 09:31:33 +0000
X-ASF-Spam-Status: No, hits=1.6 required=10.0
	tests=RCVD_IN_BRBL_LASTEXT,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of v.amit@verchaska.com designates 203.123.141.155 as permitted sender)
Received: from [203.123.141.155] (HELO mail.verchaska.com) (203.123.141.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 09:31:26 +0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by mail.verchaska.com (Postfix) with ESMTP id 550C511DBAFB
	for <general@hadoop.apache.org>; Fri,  9 Jul 2010 14:59:06 +0530 (IST)
X-Virus-Scanned: amavisd-new at mail.verchaska.com
Received: from mail.verchaska.com ([127.0.0.1])
	by localhost (mail.verchaska.com [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id FTJnMCrNZ61C for <general@hadoop.apache.org>;
	Fri,  9 Jul 2010 14:59:03 +0530 (IST)
Received: from [192.168.0.186] (unknown [192.168.0.186])
	by mail.verchaska.com (Postfix) with ESMTP id 35AD611DBAEF
	for <general@hadoop.apache.org>; Fri,  9 Jul 2010 14:59:03 +0530 (IST)
Message-ID: <4C36EC38.1080703@verchaska.com>
Date: Fri, 09 Jul 2010 15:00:32 +0530
From: amit kumar verma <v.amit@verchaska.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.2.4) Gecko/20100608 Thunderbird/3.1
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: jni files
References: <4C35877A.3090705@verchaska.com>	<354823AA-8AF3-4FF4-9606-EAAD236E1835@linkedin.com>	<AANLkTikwQaAEbX9q8hoNITrU5d_E-8wWBhfYskhX9d8U@mail.gmail.com>	<4C36E730.1060004@verchaska.com> <AANLkTikI6BTduu0skWGNQbQFyR0lDcvDTVfzElKWqltM@mail.gmail.com>
In-Reply-To: <AANLkTikI6BTduu0skWGNQbQFyR0lDcvDTVfzElKWqltM@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

  Hi  Hemanth,

Yeah I have gone through the api documentation and there is no issue in 
accessing files from HDFS, but my concern is what about the API which 
already got developed without hadoop. OK, what I mean, I developed an 
application when I didn't know about the hadoop, but as now I need to 
implement grid environment so I am looking for Hadoop.

So no the question is, how can use the same code to work for HDFS, do I 
need to change my code and use hadoop API to used the HDFS. If that is 
the case then the change will be major, or there is any way where the 
default java.file can be integrated with hdfs.

Did you get the issue ??

Thanks,
Amit Kumar Verma
Verchaska Infotech Pvt. Ltd.



On 07/09/2010 02:47 PM, Hemanth Yamijala wrote:
> Amit,
>
> On Fri, Jul 9, 2010 at 2:39 PM, amit kumar verma<v.amit@verchaska.com>  wrote:
>>   Hi Hemant,
>>
>> The version are same as copied it to all client machine.
>>
>> I think I got a solution. As I read more about hadoop and JNI, I learned
>> that I need to copy jni files to
>> HADOOP_INSTALLATION_DIR//lib/native/Linux-xxx-xxx. I though my linux machine
>> is Linux-i386-32. then I found in "org.apache.hadoop.util.PlatformName"
>> class gives you your machine type and its Linux-amd64-64 and asa I copied
>> jni files to this directory error are not coming.
>>
>> Though full code is still not running as I developed the application using
>> java.file class and i am still thinking how to make changes so that it can
>> access hdfs !!!  Do i need to change my all API with respect to HDFS and
>> rewrite using hadoop fs or ??!!!
>>
> To access files from HDFS, you should use the Hadoop FileSystem API.
> Please take a look at the Javadoc and also a tutorial such as this:
> http://developer.yahoo.com/hadoop/tutorial/module2.html#programmatically
> for more information.
>
>> It will be great if someone advice on this.
>>
>>
>>
>> Thanks,
>> Amit Kumar Verma
>> Verchaska Infotech Pvt. Ltd.
>>
>>
>>
>> On 07/09/2010 02:04 PM, Hemanth Yamijala wrote:
>>> Hi,
>>>
>>> Possibly another silly question, but can you cross check if the
>>> versions of Hadoop on the client and the server are the same ?
>>>
>>> Thanks
>>> hemanth
>>>
>>> On Thu, Jul 8, 2010 at 10:57 PM, Allen Wittenauer
>>> <awittenauer@linkedin.com>    wrote:
>>>> On Jul 8, 2010, at 1:08 AM, amit kumar verma wrote:
>>>>
>>>>>      DistributedCache.addCacheFile("hdfs://*
>>>>>      /192.168.0.153:50075*/libraries/mylib.so.1#mylib.so", conf);
>>>> Do you actually have asterisks in this?  If so, that's the problem.
>>>>
>>>>

From general-return-1792-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 09:49:40 2010
Return-Path: <general-return-1792-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87896 invoked from network); 9 Jul 2010 09:49:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 09:49:39 -0000
Received: (qmail 86718 invoked by uid 500); 9 Jul 2010 09:49:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86503 invoked by uid 500); 9 Jul 2010 09:49:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86495 invoked by uid 99); 9 Jul 2010 09:49:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 09:49:35 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yhemanth@gmail.com designates 74.125.83.176 as permitted sender)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 09:49:28 +0000
Received: by pvc21 with SMTP id 21so1477724pvc.35
        for <general@hadoop.apache.org>; Fri, 09 Jul 2010 02:48:07 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=DqNzTZmLukh6MmPodiJLmbputc7gftQZO3lSTvvYpOw=;
        b=L9+Tl3VNFiZjOH3Qdd8Mb06m3eztFifFHBrmDYVtxhZo0F34Qj+Ka/wCMpIdqnuhI4
         jlGJRczU5aP3AaBFOgFymW8nBqpyfUW23ZRqnFrs6HXvvy0076L4aLueSPHEhfVsSYo4
         h7ZfXaigVd+EcHC6/MADKJX5CEkftTpK+8/iw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=AMPaRGaMaAiH5uKhvjxhKrJ/ZAjPqHFnxJGcHoMddbw00iveeRyXaO18ty8kG7kEBQ
         v4Vk1/zYnIfAuP3kunioKqizW0VjTHfI9/PEys3gZbrt2lBOnaY5FACnjOrxEalElvcW
         v8dxKX+9ecRTLglCpdIMUF2FKQ876TxcnAhsw=
MIME-Version: 1.0
Received: by 10.142.136.1 with SMTP id j1mr11245492wfd.181.1278668887270; Fri, 
	09 Jul 2010 02:48:07 -0700 (PDT)
Received: by 10.142.188.19 with HTTP; Fri, 9 Jul 2010 02:48:07 -0700 (PDT)
In-Reply-To: <4C36EC38.1080703@verchaska.com>
References: <4C35877A.3090705@verchaska.com>
	<354823AA-8AF3-4FF4-9606-EAAD236E1835@linkedin.com>
	<AANLkTikwQaAEbX9q8hoNITrU5d_E-8wWBhfYskhX9d8U@mail.gmail.com>
	<4C36E730.1060004@verchaska.com>
	<AANLkTikI6BTduu0skWGNQbQFyR0lDcvDTVfzElKWqltM@mail.gmail.com>
	<4C36EC38.1080703@verchaska.com>
Date: Fri, 9 Jul 2010 15:18:07 +0530
Message-ID: <AANLkTimS_C6me30OhE_EKJK5YDm0Qj1zEZDyLROAHYxD@mail.gmail.com>
Subject: Re: jni files
From: Hemanth Yamijala <yhemanth@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Amit,

On Fri, Jul 9, 2010 at 3:00 PM, amit kumar verma <v.amit@verchaska.com> wro=
te:
> =A0Hi =A0Hemanth,
>
> Yeah I have gone through the api documentation and there is no issue in
> accessing files from HDFS, but my concern is what about the API which
> already got developed without hadoop. OK, what I mean, I developed an
> application when I didn't know about the hadoop, but as now I need to
> implement grid environment so I am looking for Hadoop.
>
> So no the question is, how can use the same code to work for HDFS, do I n=
eed
> to change my code and use hadoop API to used the HDFS. If that is the cas=
e
> then the change will be major, or there is any way where the default
> java.file can be integrated with hdfs.
>
> Did you get the issue ??
>

Yes, I think I do. Unfortunately, AFAIK, there's no easy way out. If
your application had previously used Java I/O File APIs, they need to
be migrated to the Hadoop FS API. If you are moving from a
non-Distributed application to Hadoop for a reason (such as handling
scale for e.g.) the investment will be well worth the effort, IMHO.

> Thanks,
> Amit Kumar Verma
> Verchaska Infotech Pvt. Ltd.
>
>
>
> On 07/09/2010 02:47 PM, Hemanth Yamijala wrote:
>>
>> Amit,
>>
>> On Fri, Jul 9, 2010 at 2:39 PM, amit kumar verma<v.amit@verchaska.com>
>> =A0wrote:
>>>
>>> =A0Hi Hemant,
>>>
>>> The version are same as copied it to all client machine.
>>>
>>> I think I got a solution. As I read more about hadoop and JNI, I learne=
d
>>> that I need to copy jni files to
>>> HADOOP_INSTALLATION_DIR//lib/native/Linux-xxx-xxx. I though my linux
>>> machine
>>> is Linux-i386-32. then I found in "org.apache.hadoop.util.PlatformName"
>>> class gives you your machine type and its Linux-amd64-64 and asa I copi=
ed
>>> jni files to this directory error are not coming.
>>>
>>> Though full code is still not running as I developed the application
>>> using
>>> java.file class and i am still thinking how to make changes so that it
>>> can
>>> access hdfs !!! =A0Do i need to change my all API with respect to HDFS =
and
>>> rewrite using hadoop fs or ??!!!
>>>
>> To access files from HDFS, you should use the Hadoop FileSystem API.
>> Please take a look at the Javadoc and also a tutorial such as this:
>> http://developer.yahoo.com/hadoop/tutorial/module2.html#programmatically
>> for more information.
>>
>>> It will be great if someone advice on this.
>>>
>>>
>>>
>>> Thanks,
>>> Amit Kumar Verma
>>> Verchaska Infotech Pvt. Ltd.
>>>
>>>
>>>
>>> On 07/09/2010 02:04 PM, Hemanth Yamijala wrote:
>>>>
>>>> Hi,
>>>>
>>>> Possibly another silly question, but can you cross check if the
>>>> versions of Hadoop on the client and the server are the same ?
>>>>
>>>> Thanks
>>>> hemanth
>>>>
>>>> On Thu, Jul 8, 2010 at 10:57 PM, Allen Wittenauer
>>>> <awittenauer@linkedin.com> =A0 =A0wrote:
>>>>>
>>>>> On Jul 8, 2010, at 1:08 AM, amit kumar verma wrote:
>>>>>
>>>>>> =A0 =A0 DistributedCache.addCacheFile("hdfs://*
>>>>>> =A0 =A0 /192.168.0.153:50075*/libraries/mylib.so.1#mylib.so", conf);
>>>>>
>>>>> Do you actually have asterisks in this? =A0If so, that's the problem.
>>>>>
>>>>>
>

From general-return-1793-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 10:23:31 2010
Return-Path: <general-return-1793-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12277 invoked from network); 9 Jul 2010 10:23:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 10:23:30 -0000
Received: (qmail 24611 invoked by uid 500); 9 Jul 2010 10:23:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24182 invoked by uid 500); 9 Jul 2010 10:23:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24174 invoked by uid 99); 9 Jul 2010 10:23:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 10:23:25 +0000
X-ASF-Spam-Status: No, hits=1.6 required=10.0
	tests=RCVD_IN_BRBL_LASTEXT,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of v.amit@verchaska.com designates 203.123.141.155 as permitted sender)
Received: from [203.123.141.155] (HELO mail.verchaska.com) (203.123.141.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 10:23:19 +0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by mail.verchaska.com (Postfix) with ESMTP id 6636411DBAF3
	for <general@hadoop.apache.org>; Fri,  9 Jul 2010 15:51:24 +0530 (IST)
X-Virus-Scanned: amavisd-new at mail.verchaska.com
Received: from mail.verchaska.com ([127.0.0.1])
	by localhost (mail.verchaska.com [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id JHQQO6Kkkucm for <general@hadoop.apache.org>;
	Fri,  9 Jul 2010 15:51:18 +0530 (IST)
Received: from [192.168.0.186] (unknown [192.168.0.186])
	by mail.verchaska.com (Postfix) with ESMTP id 6AC9211DB886
	for <general@hadoop.apache.org>; Fri,  9 Jul 2010 15:51:18 +0530 (IST)
Message-ID: <4C36F877.50302@verchaska.com>
Date: Fri, 09 Jul 2010 15:52:47 +0530
From: amit kumar verma <v.amit@verchaska.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.2.4) Gecko/20100608 Thunderbird/3.1
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: jni files
References: <4C35877A.3090705@verchaska.com>	<354823AA-8AF3-4FF4-9606-EAAD236E1835@linkedin.com>	<AANLkTikwQaAEbX9q8hoNITrU5d_E-8wWBhfYskhX9d8U@mail.gmail.com>	<4C36E730.1060004@verchaska.com>	<AANLkTikI6BTduu0skWGNQbQFyR0lDcvDTVfzElKWqltM@mail.gmail.com>	<4C36EC38.1080703@verchaska.com> <AANLkTimS_C6me30OhE_EKJK5YDm0Qj1zEZDyLROAHYxD@mail.gmail.com>
In-Reply-To: <AANLkTimS_C6me30OhE_EKJK5YDm0Qj1zEZDyLROAHYxD@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

  Hi Hemanth,

Thanks for your kind response and support. But what if I am using third 
party API and it also uses the java IO File ?? I think there must be 
some way to use hdfs by default with changing the code !!

Thanks,
Amit Kumar Verma
Verchaska Infotech Pvt. Ltd.



On 07/09/2010 03:18 PM, Hemanth Yamijala wrote:
> Amit,
>
> On Fri, Jul 9, 2010 at 3:00 PM, amit kumar verma<v.amit@verchaska.com>  wrote:
>>   Hi  Hemanth,
>>
>> Yeah I have gone through the api documentation and there is no issue in
>> accessing files from HDFS, but my concern is what about the API which
>> already got developed without hadoop. OK, what I mean, I developed an
>> application when I didn't know about the hadoop, but as now I need to
>> implement grid environment so I am looking for Hadoop.
>>
>> So no the question is, how can use the same code to work for HDFS, do I need
>> to change my code and use hadoop API to used the HDFS. If that is the case
>> then the change will be major, or there is any way where the default
>> java.file can be integrated with hdfs.
>>
>> Did you get the issue ??
>>
> Yes, I think I do. Unfortunately, AFAIK, there's no easy way out. If
> your application had previously used Java I/O File APIs, they need to
> be migrated to the Hadoop FS API. If you are moving from a
> non-Distributed application to Hadoop for a reason (such as handling
> scale for e.g.) the investment will be well worth the effort, IMHO.
>
>> Thanks,
>> Amit Kumar Verma
>> Verchaska Infotech Pvt. Ltd.
>>
>>
>>
>> On 07/09/2010 02:47 PM, Hemanth Yamijala wrote:
>>> Amit,
>>>
>>> On Fri, Jul 9, 2010 at 2:39 PM, amit kumar verma<v.amit@verchaska.com>
>>>   wrote:
>>>>   Hi Hemant,
>>>>
>>>> The version are same as copied it to all client machine.
>>>>
>>>> I think I got a solution. As I read more about hadoop and JNI, I learned
>>>> that I need to copy jni files to
>>>> HADOOP_INSTALLATION_DIR//lib/native/Linux-xxx-xxx. I though my linux
>>>> machine
>>>> is Linux-i386-32. then I found in "org.apache.hadoop.util.PlatformName"
>>>> class gives you your machine type and its Linux-amd64-64 and asa I copied
>>>> jni files to this directory error are not coming.
>>>>
>>>> Though full code is still not running as I developed the application
>>>> using
>>>> java.file class and i am still thinking how to make changes so that it
>>>> can
>>>> access hdfs !!!  Do i need to change my all API with respect to HDFS and
>>>> rewrite using hadoop fs or ??!!!
>>>>
>>> To access files from HDFS, you should use the Hadoop FileSystem API.
>>> Please take a look at the Javadoc and also a tutorial such as this:
>>> http://developer.yahoo.com/hadoop/tutorial/module2.html#programmatically
>>> for more information.
>>>
>>>> It will be great if someone advice on this.
>>>>
>>>>
>>>>
>>>> Thanks,
>>>> Amit Kumar Verma
>>>> Verchaska Infotech Pvt. Ltd.
>>>>
>>>>
>>>>
>>>> On 07/09/2010 02:04 PM, Hemanth Yamijala wrote:
>>>>> Hi,
>>>>>
>>>>> Possibly another silly question, but can you cross check if the
>>>>> versions of Hadoop on the client and the server are the same ?
>>>>>
>>>>> Thanks
>>>>> hemanth
>>>>>
>>>>> On Thu, Jul 8, 2010 at 10:57 PM, Allen Wittenauer
>>>>> <awittenauer@linkedin.com>      wrote:
>>>>>> On Jul 8, 2010, at 1:08 AM, amit kumar verma wrote:
>>>>>>
>>>>>>>      DistributedCache.addCacheFile("hdfs://*
>>>>>>>      /192.168.0.153:50075*/libraries/mylib.so.1#mylib.so", conf);
>>>>>> Do you actually have asterisks in this?  If so, that's the problem.
>>>>>>
>>>>>>

From general-return-1794-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 15:06:01 2010
Return-Path: <general-return-1794-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12729 invoked from network); 9 Jul 2010 15:06:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 15:06:01 -0000
Received: (qmail 85614 invoked by uid 500); 9 Jul 2010 15:06:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85494 invoked by uid 500); 9 Jul 2010 15:05:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85485 invoked by uid 99); 9 Jul 2010 15:05:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 15:05:59 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jdixon@omniti.com designates 8.8.38.6 as permitted sender)
Received: from [8.8.38.6] (HELO edge.omniti.com) (8.8.38.6)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 15:05:51 +0000
DKIM-Signature: v=1; a=rsa-sha256; d=omniti.com; s=s1024; c=relaxed/relaxed;
	q=dns/txt; i=@omniti.com; t=1278687913;
	h=From:Subject:Date:To;
	bh=4JESy4pcnlNxRWe0D3mXe8goYSw6mgxQuRTfP0+6sWM=;
	b=QiBhR3Y9tFugxAZonjtcYYJ6E0X0hKBmdErYd9uCvUcXZlNQ/gTJupnUkSnUahJq
	6Xlvwc8HpVw9PiY7hTxyE8KEK7nW1W+wIVsq8D405BWQtMu4/IilusPmicSHHjbH
	K77Dzxov7R8Bj6AqlkatwePa1QUywrXE7ABQH+xIORA=;
Authentication-Results:  edge smtp.user=jdixon@omniti.com; auth=pass (LOGIN)
Received: from [68.55.0.29] ([68.55.0.29:63216] helo=omniti.com)
	by edge (envelope-from <jdixon@omniti.com>)
	(ecelerity 2.2.2.35 r(26636M)) with ESMTPSA (cipher=AES256-SHA) 
	id 4D/1C-17327-8AA373C4; Fri, 09 Jul 2010 11:05:13 -0400
Date: Fri, 9 Jul 2010 11:05:09 -0400
From: Jason Dixon <jdixon@omniti.com>
To: general@hadoop.apache.org
Subject: Last day to submit your Surge 2010 CFP!
Message-ID: <20100709150509.GM4133@omniti.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.18 (2008-05-17)
X-Virus-Checked: Checked by ClamAV on apache.org

Today is your last chance to submit a CFP abstract for the 2010 Surge
Scalability Conference.  The event is taking place on Sept 30 and Oct 1,
2010 in Baltimore, MD.  Surge focuses on case studies that address
production failures and the re-engineering efforts that led to victory
in Web Applications or Internet Architectures.

You can find more information, including suggested topics and our
current list of speakers, online:

http://omniti.com/surge/2010

The final lineup should be available on the conference website next
week.  If you have questions about the CFP, attending Surge, or having
your business sponsor/exhibit at Surge 2010, please contact us at
surge@omniti.com.

Thanks!

-- 
Jason Dixon
OmniTI Computer Consulting, Inc.
jdixon@omniti.com
443.325.1357 x.241

From general-return-1795-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 17:16:29 2010
Return-Path: <general-return-1795-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49185 invoked from network); 9 Jul 2010 17:16:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 17:16:28 -0000
Received: (qmail 63433 invoked by uid 500); 9 Jul 2010 17:16:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63072 invoked by uid 500); 9 Jul 2010 17:16:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63042 invoked by uid 99); 9 Jul 2010 17:16:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 17:16:24 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of shujamughal@gmail.com designates 74.125.83.176 as permitted sender)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 17:16:17 +0000
Received: by pvc21 with SMTP id 21so1821727pvc.35
        for <multiple recipients>; Fri, 09 Jul 2010 10:14:55 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=22+XsjgGQjF7uP0/NmrpSXLnSwQj1Sw7zuaDai3KEJ4=;
        b=PlTNVNpArLTbydCiUXW4OUaiI4yUnTKiYMTq6jPHtG6XdEllq3KB2z6s0CM+0iyxYN
         0MxCvcKswG7TuLrFEwHCRbSt2rMxTg8XS5n6+vrJbeDe27OqRcjUauMIn4luJvxoZOKh
         XYpqNYijNz95J4bM47xNwgu9s+lLU4wCS6IF0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=Dc2bUOWZI0shj+vsX2BpJm17qx3spFwguH0oqhjBS20408b882xQBe7cVLqhx9TYVF
         jz6X/0GhILveBSefYQWdzOXXm+y7NO3Jt+YFg6CLiyiootFRwnsEsiM6oDcTCOB93qUS
         nzxHOqNb5WcBcAxZi2iCp52Tvp5RFsEyQGoQA=
MIME-Version: 1.0
Received: by 10.142.192.4 with SMTP id p4mr1483185wff.285.1278695695285; Fri, 
	09 Jul 2010 10:14:55 -0700 (PDT)
Received: by 10.143.19.15 with HTTP; Fri, 9 Jul 2010 10:14:55 -0700 (PDT)
Date: Fri, 9 Jul 2010 22:14:55 +0500
Message-ID: <AANLkTiljuf4VYz5y627W589rKxHz-jllhsFy58stkF7K@mail.gmail.com>
Subject: java.lang.OutOfMemoryError: Java heap space
From: Shuja Rehman <shujamughal@gmail.com>
To: hive-user@hadoop.apache.org, general@hadoop.apache.org, 
	mapreduce-user@hadoop.apache.org, mapreduce-issues@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd28c90b1a0d2048af78a6a
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd28c90b1a0d2048af78a6a
Content-Type: text/plain; charset=ISO-8859-1

Hi All

I am facing a hard problem. I am running a map reduce job using streaming
but it fails and it gives the following error.

Caught: java.lang.OutOfMemoryError: Java heap space
	at Nodemapper5.parseXML(Nodemapper5.groovy:25)
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess
failed with code 1
	at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:362)
	at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:572)
	at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:136)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
	at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
	at org.apache.hadoop.mapred.Child.main(Child.java:170)



I have increased the heap size in hadoop-env.sh and make it 2000M. Also I
tell the job manually by following line.

-D mapred.child.java.opts=-Xmx2000M \

but it still gives the error. The same job runs fine if i run on shell using
1024M heap size like

cat file.xml | /root/Nodemapper5.groovy


Any clue?????????

Thanks in advance.


-- 
Regards
Shuja-ur-Rehman Baig
_________________________________
MS CS - School of Science and Engineering
Lahore University of Management Sciences (LUMS)
Sector U, DHA, Lahore, 54792, Pakistan
Cell: +92 3214207445

--000e0cd28c90b1a0d2048af78a6a--

From general-return-1796-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 17:39:46 2010
Return-Path: <general-return-1796-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53291 invoked from network); 9 Jul 2010 17:39:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 17:39:46 -0000
Received: (qmail 92691 invoked by uid 500); 9 Jul 2010 17:39:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92533 invoked by uid 500); 9 Jul 2010 17:39:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 41807 invoked by uid 99); 8 Jul 2010 17:08:25 -0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
From: Grant Ingersoll <gsingers@apache.org>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
Subject: Triangle (Raleigh, Durham, Chapel Hill) Area Hadoop Users Group First Meeting: July 20th
Date: Thu, 8 Jul 2010 13:07:58 -0400
Message-Id: <C1AFDEFF-2EA3-48F0-9EC3-E90BE6BD33C3@apache.org>
To: general@hadoop.apache.org,
 user@mahout.apache.org
Mime-Version: 1.0 (Apple Message framework v1081)
X-Mailer: Apple Mail (2.1081)
X-Virus-Checked: Checked by ClamAV on apache.org

Interested in learning more about Hadoop or finding out what=92s going =
on with Hadoop in the Triangle?

Come out for the first meeting of the Triangle Hadoop Users Group =
scheduled for 7pm on Tuesday July 20th at Bronto Software.

Find out more at http://www.trihug.org

Three presenters are lined up:

	=95 Intro To Hadoop - Jeff Turner
	=95 Setting up Your First Cluster - Chad Vawter
	=95 Processing Megadata With Hadoop and Python - Ryan Cox
Join the mailing list and follow us @trihug.


Cheers,
Grant=

From general-return-1797-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 18:24:34 2010
Return-Path: <general-return-1797-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73104 invoked from network); 9 Jul 2010 18:24:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 18:24:34 -0000
Received: (qmail 38558 invoked by uid 500); 9 Jul 2010 18:24:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38467 invoked by uid 500); 9 Jul 2010 18:24:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 38453 invoked by uid 99); 9 Jul 2010 18:24:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 18:24:32 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 18:24:26 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version;
  b=Iv3qWzrJjuSUS3D4kvmumDcqU5jGTHBqA9mSfTO0de+TSOz5J6FIrP1C
   sCacNYVbNiRWiP4etn258Hthzn9iK21oj0jG6I+YjN/CbNTzLs6qUTc+v
   HuZVzUKDNuKbJh3;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1278699866; x=1310235866;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20jni=20files|Date:=20Fri,=209=20Jul=2020
   10=2018:24:04=20+0000|Message-ID:=20<B3D02B51-3ED5-4125-A
   900-2BE21E1C1B93@linkedin.com>|To:=20"<general@hadoop.apa
   che.org>"=20<general@hadoop.apache.org>|MIME-Version:=201
   .0|Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<4490e8c1-f16b-4913-99a4-781637033c25>
   |In-Reply-To:=20<4C36E730.1060004@verchaska.com>
   |References:=20<4C35877A.3090705@verchaska.com>=0D=0A=09<
   354823AA-8AF3-4FF4-9606-EAAD236E1835@linkedin.com>=0D=0A
   =20<AANLkTikwQaAEbX9q8hoNITrU5d_E-8wWBhfYskhX9d8U@mail.gm
   ail.com>=0D=0A=20<4C36E730.1060004@verchaska.com>;
  bh=rwYSR7OBzn2iWNc87REMWv2TAkybnSeUyz9WyaRoIt4=;
  b=FtX2JZgnG/zY6R0kXEToVu4by69YMYcW0jCGQwGT5JfDJ7AubUfQV5nX
   G/pnqxjlFlpxCmGWaoO2Su8/11+aZYdnOBOXdkcJCJQrm9QXl56sMHLl6
   PDzIHF19ZEVqwBr;
X-IronPort-AV: E=Sophos;i="4.55,174,1278313200"; 
   d="scan'208";a="13524905"
Received: from ESV4-EXC02.linkedin.biz ([fe80::4d74:48bd:e0bd:13ee]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi; Fri, 9 Jul 2010 11:24:05
 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: jni files
Thread-Topic: jni files
Thread-Index: AQHLHnTVV1c7eHpkqEaCeEw37pEITJKnvfoAgAD9sICAAAmGAIAAmxEA
Date: Fri, 9 Jul 2010 18:24:04 +0000
Message-ID: <B3D02B51-3ED5-4125-A900-2BE21E1C1B93@linkedin.com>
References: <4C35877A.3090705@verchaska.com>
	<354823AA-8AF3-4FF4-9606-EAAD236E1835@linkedin.com>
 <AANLkTikwQaAEbX9q8hoNITrU5d_E-8wWBhfYskhX9d8U@mail.gmail.com>
 <4C36E730.1060004@verchaska.com>
In-Reply-To: <4C36E730.1060004@verchaska.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Content-Type: text/plain; charset="us-ascii"
Content-ID: <4490e8c1-f16b-4913-99a4-781637033c25>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


On Jul 9, 2010, at 2:09 AM, amit kumar verma wrote:
> I think I got a solution. As I read more about hadoop and JNI, I learned =
that I need to copy jni files to HADOOP_INSTALLATION_DIR//lib/native/Linux-=
xxx-xxx.

lib/native/xxx are for the native compression libraries.  They are not for =
user-level map reduce code.



From general-return-1798-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 19:20:11 2010
Return-Path: <general-return-1798-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87904 invoked from network); 9 Jul 2010 19:20:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 19:20:11 -0000
Received: (qmail 1407 invoked by uid 500); 9 Jul 2010 19:20:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1188 invoked by uid 500); 9 Jul 2010 19:20:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1179 invoked by uid 99); 9 Jul 2010 19:20:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 19:20:09 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of abhishek.vit@gmail.com designates 209.85.161.176 as permitted sender)
Received: from [209.85.161.176] (HELO mail-gx0-f176.google.com) (209.85.161.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 19:20:02 +0000
Received: by gxk7 with SMTP id 7so2297754gxk.35
        for <general@hadoop.apache.org>; Fri, 09 Jul 2010 12:19:41 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:cc:content-type;
        bh=AOxILQjtp34pVxMoFjhuYtUdRkzfBFzDorql0lqODPg=;
        b=lhADrtcB4Lgk0ad43X/NO/UIJy0avLPIAoBdsoxyk7/T/y9oTRYhyUNlcu8dGNFqlu
         Ptta4wiglykc1ICWhSfPUk4t4nD4bG38EBn/uQFbwYtO+XhEmXPlvHXggFtOLIL4pJfF
         V9d0h+LM4YOGCjLm1wgAVFC3ozHypKK7wqdVY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=Q08Dz8Fe5mH7NPhriLt1CakYtBC8beFppNETMcvzZ9D4UwV15GxfZER1khD052yAp8
         fleWNbU79sGQL8pXcpDYfuWttHJcfO7l86dbVpWQGGn6SSHyXc6ynylYkD7fAGlXJtyz
         OScz4knsw0suDi4BRJZixjtZagSY/XUwUOwrc=
MIME-Version: 1.0
Received: by 10.224.65.147 with SMTP id j19mr5852758qai.189.1278703180915; 
	Fri, 09 Jul 2010 12:19:40 -0700 (PDT)
Received: by 10.229.246.131 with HTTP; Fri, 9 Jul 2010 12:19:40 -0700 (PDT)
In-Reply-To: <C1AFDEFF-2EA3-48F0-9EC3-E90BE6BD33C3@apache.org>
References: <C1AFDEFF-2EA3-48F0-9EC3-E90BE6BD33C3@apache.org>
Date: Fri, 9 Jul 2010 15:19:40 -0400
Message-ID: <AANLkTik639LXV9N2c6zo910v5tKU9YNfb7BLID6IWnsz@mail.gmail.com>
Subject: Re: Triangle (Raleigh, Durham, Chapel Hill) Area Hadoop Users Group 
	First Meeting: July 20th
From: Abhishek Pratap <abhishek.vit@gmail.com>
To: general@hadoop.apache.org
Cc: user@mahout.apache.org
Content-Type: multipart/alternative; boundary=00c09f88cf8cdf2d4c048af9488e
X-Virus-Checked: Checked by ClamAV on apache.org

--00c09f88cf8cdf2d4c048af9488e
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Hi Grant

The agenda for this meeting looks really interesting for newbies like me.
Any chance you guys can record videos. I am assuming getting talks live on
air will be tougher.

Thanks!
-Abhi

On Thu, Jul 8, 2010 at 1:07 PM, Grant Ingersoll <gsingers@apache.org> wrote=
:

> Interested in learning more about Hadoop or finding out what=92s going on
> with Hadoop in the Triangle?
>
> Come out for the first meeting of the Triangle Hadoop Users Group schedul=
ed
> for 7pm on Tuesday July 20th at Bronto Software.
>
> Find out more at http://www.trihug.org
>
> Three presenters are lined up:
>
>        =95 Intro To Hadoop - Jeff Turner
>        =95 Setting up Your First Cluster - Chad Vawter
>        =95 Processing Megadata With Hadoop and Python - Ryan Cox
> Join the mailing list and follow us @trihug.
>
>
> Cheers,
> Grant

--00c09f88cf8cdf2d4c048af9488e--

From general-return-1799-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 12 12:15:55 2010
Return-Path: <general-return-1799-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 7805 invoked from network); 12 Jul 2010 12:15:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jul 2010 12:15:54 -0000
Received: (qmail 61705 invoked by uid 500); 12 Jul 2010 12:15:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61162 invoked by uid 500); 12 Jul 2010 12:15:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61154 invoked by uid 99); 12 Jul 2010 12:15:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 12:15:47 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jamesisaac.dell@gmail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 12:15:40 +0000
Received: by iwn37 with SMTP id 37so6798036iwn.35
        for <general@hadoop.apache.org>; Mon, 12 Jul 2010 05:15:19 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=RAWiycnJHcMRg3mCETd9DyaCA/Bs/0CXvhl308MAZFg=;
        b=TilLH2grhUyoj48jRs/DhHjxeOwFq8Kx1aJfFbN1+j4GdJ2R+qD9tr3699vg4G3OMA
         0T4X1A2/i/5lRoj/hNC+sYwAzITGrCQM0scczTw/bA3n8neOIeE19Ct52+T9UQDI275q
         mfDSJhdRE0ZH/56Tcstvpu5NzjTeHjXow+8Xg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=uhqq9VdHZjDlQWZb5T6nZoQp6QWNjycxK3y7PZCPFVsmEA3JG54n+IaTdbYg7AocSi
         WynrqKekLO4AyIrIafWYEIUG6iYdZpR/3FagRRroacIM+cQoCb6Ow4H4wyvNBU0ZVhNZ
         E3Q+N1esTEO3qf4OprI+uHXIr4Gm0yYfUzaOE=
MIME-Version: 1.0
Received: by 10.231.161.80 with SMTP id q16mr13160885ibx.142.1278936918897; 
	Mon, 12 Jul 2010 05:15:18 -0700 (PDT)
Received: by 10.231.143.72 with HTTP; Mon, 12 Jul 2010 05:15:18 -0700 (PDT)
Date: Mon, 12 Jul 2010 17:45:18 +0530
Message-ID: <AANLkTinnda4iVR94vHyOiVOCX0nfCWltt2ukaEtvmTNz@mail.gmail.com>
Subject: Exception in thread "main" java.lang.ClassNotFoundException in Hadoop
From: james isaac <jamesisaac.dell@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=005045013e15bdb33a048b2fb426
X-Virus-Checked: Checked by ClamAV on apache.org

--005045013e15bdb33a048b2fb426
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Hi,

I have just started my career with Hadoop. I tried to execute the example
given in http://hadoop.apache.org/common/docs/r0.20.1/mapred_tutorial.htmla=
s
explained in the tutorial. I was not able to execute the program as
described in the tutorial. I am getting the error as shown below. I tried
executing with some other examples. They are also showing the same
=93ClassNotFoundException=94. I tried to find the solution in various websi=
tes
but i could not find it. Please help me to find the problem.

Exception in thread "main"
java.lang.ClassNotFoundException: org.myorg.WordCount
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.util.RunJar.main(RunJar.java:149)

--005045013e15bdb33a048b2fb426--

From general-return-1800-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 12 15:46:24 2010
Return-Path: <general-return-1800-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 92824 invoked from network); 12 Jul 2010 15:46:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jul 2010 15:46:24 -0000
Received: (qmail 74472 invoked by uid 500); 12 Jul 2010 15:46:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74410 invoked by uid 500); 12 Jul 2010 15:46:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74397 invoked by uid 99); 12 Jul 2010 15:46:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 15:46:22 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates 209.85.216.176 as permitted sender)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 15:46:14 +0000
Received: by qyk12 with SMTP id 12so5794471qyk.14
        for <general@hadoop.apache.org>; Mon, 12 Jul 2010 08:44:53 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=m3VpwpW8VhKjd+VSsUOM6GFRcz1402v1aA12L70Mttg=;
        b=nBoo9S2/x72BtuUMHiqlqAlPuI781+DIy5B1vNvJYeFYZOCX8QpV/+tqlD2fY7/BcG
         sVLEwTzjKqhRy/q75iUEq63EsdKnhw+H6kRT/ZAUpB8krHnXtxJb1oiiH2GyIgylANnk
         1kmhLddc8r9eTBtLnQ7dLGR8iUa0tUTIh+4qQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=vcfD3zRf2kt/DmIdHbKgclEhDfZhsqTumPgCyxmZzrqGR9C8OilbTqw84GjdZSqfsi
         2nxtR920HZpX2vWCLMWLUXdmzKUvxO17gIDD8hokAWNdxUD/sVEBbE2OqT/rcTh0NXPD
         289dS2TnZo9YCB0+odftXyZeBSP0bbDLOqQjc=
MIME-Version: 1.0
Received: by 10.224.64.209 with SMTP id f17mr7901804qai.138.1278949493646; 
	Mon, 12 Jul 2010 08:44:53 -0700 (PDT)
Received: by 10.229.75.85 with HTTP; Mon, 12 Jul 2010 08:44:47 -0700 (PDT)
In-Reply-To: <AANLkTinnda4iVR94vHyOiVOCX0nfCWltt2ukaEtvmTNz@mail.gmail.com>
References: <AANLkTinnda4iVR94vHyOiVOCX0nfCWltt2ukaEtvmTNz@mail.gmail.com>
Date: Mon, 12 Jul 2010 08:44:47 -0700
Message-ID: <AANLkTik4AFGGi6A93gmjMjBhHhIVNhtZrudE_6ORsqJu@mail.gmail.com>
Subject: Re: Exception in thread "main" java.lang.ClassNotFoundException in 
	Hadoop
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f8cdd2412827048b32a2f3
X-Virus-Checked: Checked by ClamAV on apache.org

--001485f8cdd2412827048b32a2f3
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Please give the commandline you used.
You should have specified the jar containing WordCount class.

On Mon, Jul 12, 2010 at 5:15 AM, james isaac <jamesisaac.dell@gmail.com>wro=
te:

> Hi,
>
> I have just started my career with Hadoop. I tried to execute the example
> given in
> http://hadoop.apache.org/common/docs/r0.20.1/mapred_tutorial.htmlas
> explained in the tutorial. I was not able to execute the program as
> described in the tutorial. I am getting the error as shown below. I tried
> executing with some other examples. They are also showing the same
> =93ClassNotFoundException=94. I tried to find the solution in various web=
sites
> but i could not find it. Please help me to find the problem.
>
> Exception in thread "main"
> java.lang.ClassNotFoundException: org.myorg.WordCount
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
>

--001485f8cdd2412827048b32a2f3--

From general-return-1801-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 12 19:40:16 2010
Return-Path: <general-return-1801-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6181 invoked from network); 12 Jul 2010 19:40:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jul 2010 19:40:16 -0000
Received: (qmail 3154 invoked by uid 500); 12 Jul 2010 19:40:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2966 invoked by uid 500); 12 Jul 2010 19:40:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2954 invoked by uid 99); 12 Jul 2010 19:40:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 19:40:14 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 19:40:08 +0000
Received: by pvc21 with SMTP id 21so4138229pvc.35
        for <general@hadoop.apache.org>; Mon, 12 Jul 2010 12:39:46 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.142.163.15 with SMTP id l15mr17598716wfe.97.1278963585988; 
	Mon, 12 Jul 2010 12:39:45 -0700 (PDT)
Received: by 10.143.18.6 with HTTP; Mon, 12 Jul 2010 12:39:45 -0700 (PDT)
Date: Mon, 12 Jul 2010 12:39:45 -0700
Message-ID: <AANLkTilRca-SDdWa0rZtNjdbnnzk3m0AG9JGwUBSGIaK@mail.gmail.com>
Subject: HEP proposal
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

A while back we started discussing on list (http://bit.ly/aFj9Ya) and
at the contributors meeting (http://bit.ly/aj4Y7I) a more coordinated
way to describe, socialize and shepherd enhancements to Hadoop.
Thanks for all the feedback.  Most of it was encouraging so I wrote up
a draft proposal with specifics to discuss here.  After incorporating
feedback I'll send out another revision for vote.

Thanks,
Eli


HEP: 1
Title: HEP Purpose and Guidelines
Author: Eli Collins
Status: Draft


What is a HEP?
==============

HEP stands for Hadoop Enhancement Proposal, and is based on Python's
PEP (Python Enhancement Proposal) [1].  A HEP is a document that
describes a new feature, it's rationale, and issues the feature needs
to address in order to be successuflly incorporated.

The intent is for HEPs to be the primary mechanism for proposing
significant new features to core Hadoop (common, HDFS and MapReduce),
incorporating community feedback, and recording the proposal.  Going
through the HEP process should improve the chances that a proposal is
successful.

While HEPs do not need to come with code, they are a mechanism to
propose features to the community, with the intent of contributing the
feature, rather than request the community implement a feature.

HEPs must be consistent with Apache bylaws [2], for example, the HEP
workflow takes place on the public Apache Hadoop lists.


When is a HEP Required?
=======================

HEPs should not impede casual contribution to Hadoop.  Small
improvements and bugs do not require HEPs.  Not all features need
HEPs.  While the decision is subjective, here are some guidelines to
indicate a HEP should be considered:

- The feature impacts backwards compatibility (eg modifies released
public APIs in an incompatible way).

- The feature requires that an existing component be substantially
re-designed (eg NameNode modified to use Bookkeeper).

- The implementation impact multiple parts of the system (eg symbolic
links versus adding a pluggable component like a codec).

- The feature impacts the entire development community (eg converts
the build system to use maven).


HEP Workflow
============

The author of a HEP should first try to determine if their idea is
HEP-able by sending mail to the general, or the project-specific lists
if the scope of the idea is limited to the project.  This gives the
author a chance to flesh out the proposal, address intial concerns,
and figure out whether it has a chance of being accepted.  The
author's role is to build consensus, and gather dissenting opinions.

Following this discussion the author should draft a HEP proposal
following the HEP template. The proposal should accurately reflect and
address feedback and dissenting opinions.  For example, flesh out
sections on backwards compatibility or testing. The author should send
the draft of the proposal to hep@hadoop.apache.org for review.  This
is a new, public list for editors and those interested in following
the review process.

A set of editors reviews incoming HEPs. Each HEP is assigned a single
primary editor. An editor may volunteer if they feel particular
functional expertise is required or assign HEPs to editors round
robin.

The editor reviews the proposal and may request it be updated if it
does not sufficiently address feedback raised during discussion, eg
why the proposal is not redundant with existing functionality, or is
technically sound, sufficiently motivated, covers backwards
compatibility, etc. As updates are necessary, the HEP author can check
in new versions if they have commit permissions, or can email new HEP
versions to the editor for committing. In order to ensure HEP
proposals make progress the editor should respond to proposal drafts
within two weeks of receiving them (or the proposer can request
another editor), and the proposer should generate updates to the draft
within two weeks of receiving feedback from the editor.

The editor's role is to determine if the proposal is complete, so that
the proposal can be voted on, not whether they agree with the proposal
itself.  The editor's involvement should increase the chance that a
HEP proposal makes it to a vote.

Once the editor deems the proposal is complete they add it to a
versioned HEP repository and the author posts the proposal to
general@hadoop.apache.org for vote.  HEP votes, like Apache procedural
votes, use majority rule [3]. Successful HEPs are assigned a number,
unsuccessful HEPs remain drafts.

The editors are apointed and removed by the PMC informally, similar to
how the Apache Board appoints shepherds to projects.


HEP Contents
============

Each HEP should contain the following:

1. Preamble -- Including the HEP number, a short descriptive title,
and the names of the authors.

2. Abstract -- A short (~200 word) description of the technical issue
being addressed.

3. Copyright/public domain -- Each HEP must either be explicitly
labelled as placed in the public domain (see this HEP as an example).

4. Design -- A high-level explanation of the design. It should cover
intended use cases, failure scenarios, and impact on the existing
system.

5. Motivation -- The motivation spells out the use case for the
feature and the benefits it provides.

6. Rationale -- The rationale describes what motivated the design and
why particular design decisions were made.  It should describe
alternate designs that were considered and related work, e.g. how the
feature is designed in other systems. It should also consider whether
the feature could be achieved by layering atop the existing system
rather than modifying it.

The rationale should provide evidence of consensus within the
community and discuss important objections or concerns raised during
discussion.

7. Backwards Compatibility -- All HEPs that introduce backwards
incompatibilities must include a section describing these
incompatibilities and their severity.  The HEP must explain how the
author proposes to deal with these incompatibilities.  HEP submissions
without a sufficient backwards compatibility treatise may be rejected
outright.


HEP Template
============

HEPs should be plain text with minimal structural markup that adheres
to a rigid style.  You can use this HEP as an example. Each HEP starts
with a header that contains the HEP number (or empty if the number has
not yet been assigned), title, list of authors and status (Draft,
Accepted, Rejected, or Withdrawn).


Auxiliary Files
===============

HEPs may include auxiliary files such as diagrams.  Such files must be
named ``hep-XXXX-Y.ext``, where "XXXX" is the HEP number, "Y" is a
serial number (starting at 1), and "ext" is replaced by the actual
file extension (e.g. "png").


References
==========

1. http://www.python.org/dev/peps/pep-0001

2. http://www.apache.org/foundation/bylaws.html

3. http://www.apache.org/foundation/voting.html


Copyright
=========

This document has been placed in the public domain.

From general-return-1802-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 13 05:46:39 2010
Return-Path: <general-return-1802-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87638 invoked from network); 13 Jul 2010 05:46:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jul 2010 05:46:38 -0000
Received: (qmail 71354 invoked by uid 500); 13 Jul 2010 05:46:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71050 invoked by uid 500); 13 Jul 2010 05:46:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 80293 invoked by uid 99); 13 Jul 2010 03:08:37 -0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of greensight@gmail.com designates 209.85.212.48 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=c0UOYEr5EOX7KXPL1bCJ3QoVLRNLiFvRdmVLwEnCe7A=;
        b=S0eCRdNU8e4oaNZsbaJSizf8DWjCRYhFa6zhGPscmdOlbfYD3x8J+qMVZOJsuv4yCp
         nAbNZbtRe6e88AhPavRILasBsBEg2DyIbja7Z+vnLjy80tq7Z+diHs5nU+r+W5WoThF7
         wV2X/qvsBirn3xmhOEnuGo12kLCrOQcP9gsKU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=GCpzYBXciAn8aHPlAHNJ9jlwlwnquq2LvwP2syadtiYvSn1uxzrpkcs8zxRL8ixbzJ
         S+kFmS60chImiy0gzoIKg8+GT1r6oXTbwcOLJvgsZ5+aNxKcOMdQTRhUn+8QWAe8SGWW
         65Szmt0xEpmZrZhbB3mjaXR179fG2//WT9Qqo=
MIME-Version: 1.0
Date: Tue, 13 Jul 2010 11:08:09 +0800
Message-ID: <AANLkTikubILyGLWqYIjOeQD3jsQylM-7qRT9BZc5bQTD@mail.gmail.com>
Subject: Invite a keynote talk on Hadoop in the coming Apache Aisa Roadshow in 
	Shanghai (August 14-15)
From: Jack Cai <greensight@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6441ec8d5b9c6048b3c2de2
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6441ec8d5b9c6048b3c2de2
Content-Type: text/plain; charset=ISO-8859-1

There will be an Apache Asia Roadshow event on August 14 - 15 in Shanghai,
China. As Hadoop is a very hot topic in China, we would like to invite a
keynote talk on Hadoop in the event. The event could cover $2-3K travel cost
for the speaker. Visit here [1] for more information on the event.

If someone is interested in delivering the keynote, please drop me a note.
Thanks!

-Jack

[1] http://www.apachecon.com/2010/Asia/cfp

--0016e6441ec8d5b9c6048b3c2de2--

From general-return-1803-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 13 06:00:11 2010
Return-Path: <general-return-1803-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 89385 invoked from network); 13 Jul 2010 06:00:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jul 2010 06:00:11 -0000
Received: (qmail 78852 invoked by uid 500); 13 Jul 2010 06:00:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78572 invoked by uid 500); 13 Jul 2010 06:00:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78563 invoked by uid 99); 13 Jul 2010 06:00:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 06:00:06 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jamesisaac.dell@gmail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 05:59:58 +0000
Received: by iwn37 with SMTP id 37so8177315iwn.35
        for <general@hadoop.apache.org>; Mon, 12 Jul 2010 22:59:37 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=aSbNwCWY7MyKUgj9+fy5f+0FnWCuePGc45mCOBK7i7M=;
        b=WB55hd9EA/Ms4+dRugWiAEE0oGWApjf9QBhRe5F9FUwtIRAmccXDzMUYWhJRYFTg70
         RKUzgSLU9REnNMGQeo9ecBPeGk5nzwpa3gj9nrzO8OZsVqBnpyw6p+G1urtZnRH66mZ5
         VC+2v1//zk/3ICwVleUL1Mut9ca93P8MfRKzc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=ZCzynUJgswP9RPzh3WXL7zFvDkgN4B3yDcn2gyJ1Km65PwJZYos+D2f5NPofZu2/Ds
         HlmKAbFfCSejoeV2DVBt4oV6X5pfnL3xjQD+agSjATmByO1takEahuqJ6WHfySyd2G48
         6obfO8afpcUZQkFD3dbXacxB9aLBxXhGJlOFE=
MIME-Version: 1.0
Received: by 10.231.184.168 with SMTP id ck40mr14738882ibb.174.1279000774469; 
	Mon, 12 Jul 2010 22:59:34 -0700 (PDT)
Received: by 10.231.143.72 with HTTP; Mon, 12 Jul 2010 22:59:34 -0700 (PDT)
In-Reply-To: <AANLkTik4AFGGi6A93gmjMjBhHhIVNhtZrudE_6ORsqJu@mail.gmail.com>
References: <AANLkTinnda4iVR94vHyOiVOCX0nfCWltt2ukaEtvmTNz@mail.gmail.com>
	<AANLkTik4AFGGi6A93gmjMjBhHhIVNhtZrudE_6ORsqJu@mail.gmail.com>
Date: Tue, 13 Jul 2010 11:29:34 +0530
Message-ID: <AANLkTilZN81QEtePblrymRsHRZE6WpnRW2hpjt4fFia3@mail.gmail.com>
Subject: Re: Exception in thread "main" java.lang.ClassNotFoundException in 
	Hadoop
From: james isaac <jamesisaac.dell@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=005045016033d472d4048b3e9245
X-Virus-Checked: Checked by ClamAV on apache.org

--005045016033d472d4048b3e9245
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

This is the command i used and prompt is pointing to the current directory
where wordcount.jar resides. I have also set the path for the HADOOP_HOME
and added the bin directory of hadoop to the classpath.

current directory =3D /home/user/demo/wordcount
HADOOP_HOME=3D/home/user/hadoop_sws/hadoop-0.20.2
$ hadoop jar wordcount.jar org.myorg.WordCount /hdfs/data/input
/hdfs/data/output



On Mon, Jul 12, 2010 at 9:14 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> Please give the commandline you used.
> You should have specified the jar containing WordCount class.
>
> On Mon, Jul 12, 2010 at 5:15 AM, james isaac <jamesisaac.dell@gmail.com
> >wrote:
>
> > Hi,
> >
> > I have just started my career with Hadoop. I tried to execute the examp=
le
> > given in
> > http://hadoop.apache.org/common/docs/r0.20.1/mapred_tutorial.htmlas
> > explained in the tutorial. I was not able to execute the program as
> > described in the tutorial. I am getting the error as shown below. I tri=
ed
> > executing with some other examples. They are also showing the same
> > =93ClassNotFoundException=94. I tried to find the solution in various
> websites
> > but i could not find it. Please help me to find the problem.
> >
> > Exception in thread "main"
> > java.lang.ClassNotFoundException: org.myorg.WordCount
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> > at java.lang.Class.forName0(Native Method)
> > at java.lang.Class.forName(Class.java:247)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
> >
>

--005045016033d472d4048b3e9245--

From general-return-1804-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 13 20:55:32 2010
Return-Path: <general-return-1804-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87550 invoked from network); 13 Jul 2010 20:55:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jul 2010 20:55:32 -0000
Received: (qmail 92124 invoked by uid 500); 13 Jul 2010 20:55:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91870 invoked by uid 500); 13 Jul 2010 20:55:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91862 invoked by uid 99); 13 Jul 2010 20:55:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 20:55:29 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 20:55:23 +0000
Received: from [127.0.0.1] (gentlepaint-lx.corp.yahoo.com [10.72.185.127])
	by mrout2.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o6DKsN4u094247
	for <general@hadoop.apache.org>; Tue, 13 Jul 2010 13:54:25 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=zReCAXrNRJY5ps0WMeqR1/8Be5aDQmKmFKRkxYF2jxqjOT+rV5Rc9RcJYVVOBjHZ
Message-ID: <4C3CD27F.60003@yahoo-inc.com>
Date: Tue, 13 Jul 2010 13:54:23 -0700
From: Konstantin Shvachko <shv@yahoo-inc.com>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.4) Gecko/20100608 Thunderbird/3.1
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: HEP proposal
References: <AANLkTilRca-SDdWa0rZtNjdbnnzk3m0AG9JGwUBSGIaK@mail.gmail.com>
In-Reply-To: <AANLkTilRca-SDdWa0rZtNjdbnnzk3m0AG9JGwUBSGIaK@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Eli,

Thanks for a really good proposal.
Some questions / comments:

On voting
1. Which voting rule?
http://www.apache.org/foundation/glossary.html#ConsensusApproval
http://www.apache.org/foundation/glossary.html#MajorityApproval
I think you mean the MajorityApproval as it does not have veto rule.
So may be it's just clarifying the reference.

2. Who can vote?
Usually PMCs have Binding Votes.
Would be good to have a sentence clarifying this.

3. How long does the vote go?
Usual 3 days may not be enough. One week is reasonable?

4. Discussion on public lists.
A HEP can evolve from a jira, then it should be counted as a public
discussion. I think it makes sense even to continue the discussion
there if so. I propose to add the following:
=========================================
idea is HEP-able by sending mail to the general, or the project-specific lists
+	(including Jiras)
if the scope of the idea is limited to the project.
=========================================

5. How the set of editors is selected?
    "The editors are apointed and removed by the PMC informally, similar to
    how the Apache Board appoints shepherds to projects."
This needs a reference. How does Apache Board appoints shepherds?

6. The level of design details.
I think HEP should have a pretty detailed design. When people vote they
will want to be sure the design can lead to a reasonable implementation.
Should we say "implementation-ready design", rather than
"A high-level explanation of the design."
Or just
"A _detailed_ explanation of the design."

7. Typos:
successuflly, apointed, intial

Thanks,
--Konstantin

On 7/12/2010 12:39 PM, Eli Collins wrote:
> A while back we started discussing on list (http://bit.ly/aFj9Ya) and
> at the contributors meeting (http://bit.ly/aj4Y7I) a more coordinated
> way to describe, socialize and shepherd enhancements to Hadoop.
> Thanks for all the feedback.  Most of it was encouraging so I wrote up
> a draft proposal with specifics to discuss here.  After incorporating
> feedback I'll send out another revision for vote.
>
> Thanks,
> Eli
>
>
> HEP: 1
> Title: HEP Purpose and Guidelines
> Author: Eli Collins
> Status: Draft
>
>
> What is a HEP?
> ==============
>
> HEP stands for Hadoop Enhancement Proposal, and is based on Python's
> PEP (Python Enhancement Proposal) [1].  A HEP is a document that
> describes a new feature, it's rationale, and issues the feature needs
> to address in order to be successuflly incorporated.
>
> The intent is for HEPs to be the primary mechanism for proposing
> significant new features to core Hadoop (common, HDFS and MapReduce),
> incorporating community feedback, and recording the proposal.  Going
> through the HEP process should improve the chances that a proposal is
> successful.
>
> While HEPs do not need to come with code, they are a mechanism to
> propose features to the community, with the intent of contributing the
> feature, rather than request the community implement a feature.
>
> HEPs must be consistent with Apache bylaws [2], for example, the HEP
> workflow takes place on the public Apache Hadoop lists.
>
>
> When is a HEP Required?
> =======================
>
> HEPs should not impede casual contribution to Hadoop.  Small
> improvements and bugs do not require HEPs.  Not all features need
> HEPs.  While the decision is subjective, here are some guidelines to
> indicate a HEP should be considered:
>
> - The feature impacts backwards compatibility (eg modifies released
> public APIs in an incompatible way).
>
> - The feature requires that an existing component be substantially
> re-designed (eg NameNode modified to use Bookkeeper).
>
> - The implementation impact multiple parts of the system (eg symbolic
> links versus adding a pluggable component like a codec).
>
> - The feature impacts the entire development community (eg converts
> the build system to use maven).
>
>
> HEP Workflow
> ============
>
> The author of a HEP should first try to determine if their idea is
> HEP-able by sending mail to the general, or the project-specific lists
> if the scope of the idea is limited to the project.  This gives the
> author a chance to flesh out the proposal, address intial concerns,
> and figure out whether it has a chance of being accepted.  The
> author's role is to build consensus, and gather dissenting opinions.
>
> Following this discussion the author should draft a HEP proposal
> following the HEP template. The proposal should accurately reflect and
> address feedback and dissenting opinions.  For example, flesh out
> sections on backwards compatibility or testing. The author should send
> the draft of the proposal to hep@hadoop.apache.org for review.  This
> is a new, public list for editors and those interested in following
> the review process.
>
> A set of editors reviews incoming HEPs. Each HEP is assigned a single
> primary editor. An editor may volunteer if they feel particular
> functional expertise is required or assign HEPs to editors round
> robin.
>
> The editor reviews the proposal and may request it be updated if it
> does not sufficiently address feedback raised during discussion, eg
> why the proposal is not redundant with existing functionality, or is
> technically sound, sufficiently motivated, covers backwards
> compatibility, etc. As updates are necessary, the HEP author can check
> in new versions if they have commit permissions, or can email new HEP
> versions to the editor for committing. In order to ensure HEP
> proposals make progress the editor should respond to proposal drafts
> within two weeks of receiving them (or the proposer can request
> another editor), and the proposer should generate updates to the draft
> within two weeks of receiving feedback from the editor.
>
> The editor's role is to determine if the proposal is complete, so that
> the proposal can be voted on, not whether they agree with the proposal
> itself.  The editor's involvement should increase the chance that a
> HEP proposal makes it to a vote.
>
> Once the editor deems the proposal is complete they add it to a
> versioned HEP repository and the author posts the proposal to
> general@hadoop.apache.org for vote.  HEP votes, like Apache procedural
> votes, use majority rule [3]. Successful HEPs are assigned a number,
> unsuccessful HEPs remain drafts.
>
> The editors are apointed and removed by the PMC informally, similar to
> how the Apache Board appoints shepherds to projects.
>
>
> HEP Contents
> ============
>
> Each HEP should contain the following:
>
> 1. Preamble -- Including the HEP number, a short descriptive title,
> and the names of the authors.
>
> 2. Abstract -- A short (~200 word) description of the technical issue
> being addressed.
>
> 3. Copyright/public domain -- Each HEP must either be explicitly
> labelled as placed in the public domain (see this HEP as an example).
>
> 4. Design -- A high-level explanation of the design. It should cover
> intended use cases, failure scenarios, and impact on the existing
> system.
>
> 5. Motivation -- The motivation spells out the use case for the
> feature and the benefits it provides.
>
> 6. Rationale -- The rationale describes what motivated the design and
> why particular design decisions were made.  It should describe
> alternate designs that were considered and related work, e.g. how the
> feature is designed in other systems. It should also consider whether
> the feature could be achieved by layering atop the existing system
> rather than modifying it.
>
> The rationale should provide evidence of consensus within the
> community and discuss important objections or concerns raised during
> discussion.
>
> 7. Backwards Compatibility -- All HEPs that introduce backwards
> incompatibilities must include a section describing these
> incompatibilities and their severity.  The HEP must explain how the
> author proposes to deal with these incompatibilities.  HEP submissions
> without a sufficient backwards compatibility treatise may be rejected
> outright.
>
>
> HEP Template
> ============
>
> HEPs should be plain text with minimal structural markup that adheres
> to a rigid style.  You can use this HEP as an example. Each HEP starts
> with a header that contains the HEP number (or empty if the number has
> not yet been assigned), title, list of authors and status (Draft,
> Accepted, Rejected, or Withdrawn).
>
>
> Auxiliary Files
> ===============
>
> HEPs may include auxiliary files such as diagrams.  Such files must be
> named ``hep-XXXX-Y.ext``, where "XXXX" is the HEP number, "Y" is a
> serial number (starting at 1), and "ext" is replaced by the actual
> file extension (e.g. "png").
>
>
> References
> ==========
>
> 1. http://www.python.org/dev/peps/pep-0001
>
> 2. http://www.apache.org/foundation/bylaws.html
>
> 3. http://www.apache.org/foundation/voting.html
>
>
> Copyright
> =========
>
> This document has been placed in the public domain.
>


From general-return-1805-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 17:47:50 2010
Return-Path: <general-return-1805-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69308 invoked from network); 14 Jul 2010 17:47:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 17:47:49 -0000
Received: (qmail 66657 invoked by uid 500); 14 Jul 2010 17:47:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66574 invoked by uid 500); 14 Jul 2010 17:47:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66566 invoked by uid 99); 14 Jul 2010 17:47:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 17:47:47 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 17:47:41 +0000
Received: by pvc21 with SMTP id 21so5959254pvc.35
        for <general@hadoop.apache.org>; Wed, 14 Jul 2010 10:46:19 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.142.125.20 with SMTP id x20mr1388165wfc.89.1279129579081; Wed, 
	14 Jul 2010 10:46:19 -0700 (PDT)
Received: by 10.142.105.5 with HTTP; Wed, 14 Jul 2010 10:46:18 -0700 (PDT)
In-Reply-To: <4C3CD27F.60003@yahoo-inc.com>
References: <AANLkTilRca-SDdWa0rZtNjdbnnzk3m0AG9JGwUBSGIaK@mail.gmail.com>
	<4C3CD27F.60003@yahoo-inc.com>
Date: Wed, 14 Jul 2010 10:46:18 -0700
Message-ID: <AANLkTina2dxXd61b5zj4ZpS6-MK3jpzx47ecc3xZkP-r@mail.gmail.com>
Subject: Re: HEP proposal
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Hey Konstantin,

Thanks for taking a look, comments in-line.

On Tue, Jul 13, 2010 at 1:54 PM, Konstantin Shvachko <shv@yahoo-inc.com> wr=
ote:
> Eli,
>
> Thanks for a really good proposal.
> Some questions / comments:
>
> On voting
> 1. Which voting rule?
> http://www.apache.org/foundation/glossary.html#ConsensusApproval
> http://www.apache.org/foundation/glossary.html#MajorityApproval
> I think you mean the MajorityApproval as it does not have veto rule.
> So may be it's just clarifying the reference.

Good point, clarified so it's majority approval.

> 2. Who can vote?
> Usually PMCs have Binding Votes.
> Would be good to have a sentence clarifying this.

Yup, added.

> 3. How long does the vote go?
> Usual 3 days may not be enough. One week is reasonable?

Specified one week.

> 4. Discussion on public lists.
> A HEP can evolve from a jira, then it should be counted as a public
> discussion. I think it makes sense even to continue the discussion
> there if so.

Agreed, changed the wording to "If the scope of the idea is limited to
a specific project the discussion may happen on the project-specific
list or jira."

> 5. How the set of editors is selected?
> =A0 "The editors are apointed and removed by the PMC informally, similar =
to
> =A0 how the Apache Board appoints shepherds to projects."
> This needs a reference. How does Apache Board appoints shepherds?

Good question, anyone know? Since it's informal I imagine shepherds
volunteer. The editors could be a subset of the PMC that either
volunteers or is rotated periodically.

> 6. The level of design details.
> I think HEP should have a pretty detailed design. When people vote they
> will want to be sure the design can lead to a reasonable implementation.
> Should we say "implementation-ready design", rather than
> "A high-level explanation of the design."
> Or just
> "A _detailed_ explanation of the design."

Rewrote this section, tried to make it more explicit about giving both
a high-level view and complete enough description so the design can
lead to a reasonable implementation. Also added that this section
should cover how to test the design.

> 7. Typos:
> successuflly, apointed, intial

Fixed.

Updated draft follows.

Thanks,
Eli



HEP: 1
Title: HEP Purpose and Guidelines
Author: Eli Collins
Status: Draft


What is a HEP?
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

HEP stands for Hadoop Enhancement Proposal, and is based on Python's
PEP (Python Enhancement Proposal) [1].  A HEP is a document that
describes a new feature, it's rationale, and issues the feature needs
to address in order to be successfully incorporated.

The intent is for HEPs to be the primary mechanism for proposing
significant new features to core Hadoop (common, HDFS and MapReduce),
incorporating community feedback, and recording the proposal.  Going
through the HEP process should improve the chances that a proposal is
successful.

While HEPs do not need to come with code, they are a mechanism to
propose features to the community, with the intent of contributing the
feature, rather than request the community implement a feature.

HEPs must be consistent with Apache bylaws [2], for example, the HEP
workflow takes place on the public Apache Hadoop lists.


When is a HEP Required?
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

HEPs should not impede casual contribution to Hadoop.  Small
improvements and bugs do not require HEPs.  Not all features need
HEPs.  While the decision is subjective, here are some guidelines to
indicate a HEP should be considered:

- The feature impacts backwards compatibility (eg modifies released
public APIs in an incompatible way).

- The feature requires that an existing component be substantially
re-designed (eg NameNode modified to use Bookkeeper).

- The implementation impact multiple parts of the system (eg symbolic
links versus adding a pluggable component like a codec).

- The feature impacts the entire development community (eg converts
the build system to use maven).


HEP Workflow
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

The author of a HEP should first try to determine if their idea is
HEP-able by sending mail to the general list.  If the scope of the
idea is limited to a specific project the discussion may happen on the
project-specific list or jira.  This gives the author a chance to
flesh out the proposal, address initial concerns, and figure out
whether it has a chance of being accepted.  The author's role is to
build consensus, and gather dissenting opinions.

Following this discussion the author should draft a HEP proposal
following the HEP template. The proposal should accurately reflect and
address feedback and dissenting opinions.  For example, flesh out
sections on backwards compatibility or testing. The author should send
the draft of the proposal to hep@hadoop.apache.org for review.  This
is a new, public list for editors and those interested in following
the review process.

A set of editors reviews incoming HEPs. Each HEP is assigned a single
primary editor. An editor may volunteer if they feel particular
functional expertise is required or assign HEPs to editors round
robin.

The editor reviews the proposal and may request it be updated if it
does not sufficiently address feedback raised during discussion, eg
why the proposal is not redundant with existing functionality, or is
technically sound, sufficiently motivated, covers backwards
compatibility, etc. As updates are necessary, the HEP author can check
in new versions if they have commit permissions, or can email new HEP
versions to the editor for committing. In order to ensure HEP
proposals make progress the editor should respond to proposal drafts
within two weeks of receiving them (or the proposer can request
another editor), and the proposer should generate updates to the draft
within two weeks of receiving feedback from the editor.

The editor's role is to determine if the proposal is complete, so that
the proposal can be voted on, not whether they agree with the proposal
itself.  The editor's involvement should increase the chance that a
HEP proposal makes it to a vote.

Once the editor deems the proposal is complete they add it to a
versioned HEP repository and the author posts the proposal to
general@hadoop.apache.org for vote.  HEP votes, like Apache procedural
votes, use majority approval [3]. Only PMC members have binding votes.
Votes are open for a period of 1 week to allow all active voters time
to consider the proposal. Successful HEPs are assigned a number,
unsuccessful HEPs remain drafts.

The editors are appointed and removed by the PMC informally, similar
to how the Apache Board appoints shepherds to projects.


HEP Contents
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

Each HEP should contain the following:

1. Preamble -- Including the HEP number, a short descriptive title,
and the names of the authors.

2. Abstract -- A short (~200 word) description of the technical issue
being addressed.

3. Copyright/public domain -- Each HEP must either be explicitly
labelled as placed in the public domain (see this HEP as an example).

4. Design -- This section should give both a high-level view and a
complete description of the feature.  While the design does not need
to cover implementation detail it should be clear to the reader that
the design can lead to a reasonable implementation.  This section
should cover intended use cases, failure scenarios, strategies for
testing, and impact on the existing system.

5. Motivation -- The motivation spells out the use case for the
feature and the benefits it provides.

6. Rationale -- The rationale describes what motivated the design and
why particular design decisions were made.  It should describe
alternate designs that were considered and related work, e.g. how the
feature is designed in other systems. It should also consider whether
the feature could be achieved by layering atop the existing system
rather than modifying it.

The rationale should provide evidence of consensus within the
community and discuss important objections or concerns raised during
discussion.

7. Backwards Compatibility -- All HEPs that introduce backwards
incompatibilities must include a section describing these
incompatibilities and their severity.  The HEP must explain how the
author proposes to deal with these incompatibilities.  HEP submissions
without a sufficient backwards compatibility treatise may be rejected
outright.


HEP Template
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

HEPs should be plain text with minimal structural markup that adheres
to a rigid style.  You can use this HEP as an example. Each HEP starts
with a header that contains the HEP number (or empty if the number has
not yet been assigned), title, list of authors and status (Draft,
Accepted, Rejected, or Withdrawn).


Auxiliary Files
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

HEPs may include auxiliary files such as diagrams.  Such files must be
named ``hep-XXXX-Y.ext``, where "XXXX" is the HEP number, "Y" is a
serial number (starting at 1), and "ext" is replaced by the actual
file extension (e.g. "png").


References
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

1. http://www.python.org/dev/peps/pep-0001

2. http://www.apache.org/foundation/bylaws.html

3. http://www.apache.org/foundation/glossary.html#MajorityApproval


Copyright
=3D=3D=3D=3D=3D=3D=3D=3D=3D

This document has been placed in the public domain.

From general-return-1806-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 17:55:45 2010
Return-Path: <general-return-1806-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 71402 invoked from network); 14 Jul 2010 17:55:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 17:55:44 -0000
Received: (qmail 82903 invoked by uid 500); 14 Jul 2010 17:55:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82844 invoked by uid 500); 14 Jul 2010 17:55:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82836 invoked by uid 99); 14 Jul 2010 17:55:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 17:55:42 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.212.176] (HELO mail-px0-f176.google.com) (209.85.212.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 17:55:35 +0000
Received: by pxi11 with SMTP id 11so149615pxi.35
        for <general@hadoop.apache.org>; Wed, 14 Jul 2010 10:55:09 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.142.213.13 with SMTP id l13mr204800wfg.155.1279130108988; Wed, 
	14 Jul 2010 10:55:08 -0700 (PDT)
Received: by 10.142.105.5 with HTTP; Wed, 14 Jul 2010 10:55:08 -0700 (PDT)
In-Reply-To: <284E0999-F935-47A2-B051-7241BF8A4B39@apache.org>
References: <b23e01b50909291345g47ffb267l76fe193ea72652a3@mail.gmail.com>
	<FE14B90F-1F11-4F68-8A6B-3F00DB99D050@apache.org>
	<4ACCBEE7.3070105@apache.org>
	<DDB254D5-1F8E-4B75-B8C1-3B3D095BBBC3@yahoo-inc.com>
	<4ACE1292.6000702@apache.org>
	<b23e01b50910140711y545fa031u7fb199594db33bd6@mail.gmail.com>
	<4ADE4055.3020002@yahoo-inc.com>
	<5764186A-DA78-412B-ADD4-5A9F50866732@yahoo-inc.com>
	<b23e01b50910291356w17c98e18md0239f30c2aa0a5@mail.gmail.com>
	<4AEB7728.3000008@yahoo-inc.com>
	<284E0999-F935-47A2-B051-7241BF8A4B39@apache.org>
Date: Wed, 14 Jul 2010 10:55:08 -0700
Message-ID: <AANLkTinw4_fk0d8u86wThuCQmXEOxyddqMj4T4jfXtZq@mail.gmail.com>
Subject: Re: [VOTE] Proposed bylaws for the Hadoop project
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hey Owen,

Will there an updated draft of the bylaws to vote on? I was thinking
of referencing hadoop bylaws in a draft of the HEP and remembered this
vote didn't terminate.

Thanks,
Eli


On Mon, Nov 2, 2009 at 12:52 PM, Owen O'Malley <omalley@apache.org> wrote:
> We should have created bylaws when we were established as a PMC back in Jan
> 2008, but it has become clear that we need to define them. I looked around
> and the Ant project had bylaws that were both clear and fairly complete. I
> made some minor edits to match what we do. Please look them over and vote.
> In a recursive usage, let's use 2/3 majority to approve these bylaws.
>
> -- Owen
>
>
>
>
>

From general-return-1807-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 18:58:23 2010
Return-Path: <general-return-1807-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12538 invoked from network); 14 Jul 2010 18:58:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 18:58:22 -0000
Received: (qmail 90513 invoked by uid 500); 14 Jul 2010 18:58:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90463 invoked by uid 500); 14 Jul 2010 18:58:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90455 invoked by uid 99); 14 Jul 2010 18:58:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 18:58:21 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 14 Jul 2010 18:58:18 +0000
Received: (qmail 12502 invoked by uid 99); 14 Jul 2010 18:57:56 -0000
Received: from localhost.apache.org (HELO [192.168.42.217]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 18:57:56 +0000
Message-ID: <4C3E08B4.60205@apache.org>
Date: Wed, 14 Jul 2010 11:57:56 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.10) Gecko/20100527 Thunderbird/3.0.5
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: HEP proposal
References: <AANLkTilRca-SDdWa0rZtNjdbnnzk3m0AG9JGwUBSGIaK@mail.gmail.com>	<4C3CD27F.60003@yahoo-inc.com> <AANLkTina2dxXd61b5zj4ZpS6-MK3jpzx47ecc3xZkP-r@mail.gmail.com>
In-Reply-To: <AANLkTina2dxXd61b5zj4ZpS6-MK3jpzx47ecc3xZkP-r@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 07/14/2010 10:46 AM, Eli Collins wrote:
>> 5. How the set of editors is selected?
>>    "The editors are apointed and removed by the PMC informally, similar to
>>    how the Apache Board appoints shepherds to projects."
>> This needs a reference. How does Apache Board appoints shepherds?
>
> Good question, anyone know? Since it's informal I imagine shepherds
> volunteer. The editors could be a subset of the PMC that either
> volunteers or is rotated periodically.

Shepherds are assigned randomly by the chair 48 hours before each 
meeting.  Perhaps a better analogy for an editor is an Incubator Champion?

http://incubator.apache.org/incubation/Roles_and_Responsibilities.html#Champion

Champions are volunteers.  So, rather than assigning an editor, a HEP 
might ask for one to volunteer.  Any PMC member might volunteer.

Doug

From general-return-1808-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 19:13:34 2010
Return-Path: <general-return-1808-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23770 invoked from network); 14 Jul 2010 19:13:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 19:13:33 -0000
Received: (qmail 9976 invoked by uid 500); 14 Jul 2010 19:13:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 9914 invoked by uid 500); 14 Jul 2010 19:13:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 9901 invoked by uid 99); 14 Jul 2010 19:13:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 19:13:32 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 19:13:25 +0000
Received: from localhost (snvvpn3-10-72-76-c139.hq.corp.yahoo.com [10.72.76.139])
	by mrout2.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o6EJCk0N019296
	for <general@hadoop.apache.org>; Wed, 14 Jul 2010 12:12:46 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=date:from:to:subject:message-id:references:mime-version:
	content-type:content-disposition:in-reply-to:x-organization:user-agent;
	b=l2ZVOhIe3k7veQ5EYz2wFl8msgiZpKJ3KLVJ88dZlpcoV97So2M3EfaU3+VuoRtJ
Date: Wed, 14 Jul 2010 12:12:46 -0700
From: Konstantin Boudnik <cos@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: HEP proposal
Message-ID: <20100714191245.GF3418@goodenter-lm.boudnik.org>
References: <AANLkTilRca-SDdWa0rZtNjdbnnzk3m0AG9JGwUBSGIaK@mail.gmail.com>
 <4C3CD27F.60003@yahoo-inc.com>
 <AANLkTina2dxXd61b5zj4ZpS6-MK3jpzx47ecc3xZkP-r@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature"; boundary="TU+u6i6jrDPzmlWF"
Content-Disposition: inline
In-Reply-To: <AANLkTina2dxXd61b5zj4ZpS6-MK3jpzx47ecc3xZkP-r@mail.gmail.com>
X-Organization: Yahoo! Hadoop (Grid computing)
User-Agent: Mutt/1.5.20 (2009-06-14)
X-Virus-Checked: Checked by ClamAV on apache.org

--TU+u6i6jrDPzmlWF
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

I have been following this discussion for some time now and the only questi=
on
came to my mind: why mimicking PEP? Is it so astonishingly successful or is
it much better than Apache voting or RFC process (from where it has been
apparently derived).

So far I see HEP as an over-complicated process for a process sake. I'd
appreciate if some one can chip-in and tell me if and where I'm wrong.

Thanks,
  Cos

On Wed, Jul 14, 2010 at 10:46AM, Eli Collins wrote:
>    Hey Konstantin,
>=20
>    Thanks for taking a look, comments in-line.
>=20
>    On Tue, Jul 13, 2010 at 1:54 PM, Konstantin Shvachko <shv@yahoo-inc.co=
m>
>    wrote:
>    > Eli,
>    >
>    > Thanks for a really good proposal.
>    > Some questions / comments:
>    >
>    > On voting
>    > 1. Which voting rule?
>    > http://www.apache.org/foundation/glossary.html#ConsensusApproval
>    > http://www.apache.org/foundation/glossary.html#MajorityApproval
>    > I think you mean the MajorityApproval as it does not have veto rule.
>    > So may be it's just clarifying the reference.
>=20
>    Good point, clarified so it's majority approval.
>=20
>    > 2. Who can vote?
>    > Usually PMCs have Binding Votes.
>    > Would be good to have a sentence clarifying this.
>=20
>    Yup, added.
>=20
>    > 3. How long does the vote go?
>    > Usual 3 days may not be enough. One week is reasonable?
>=20
>    Specified one week.
>=20
>    > 4. Discussion on public lists.
>    > A HEP can evolve from a jira, then it should be counted as a public
>    > discussion. I think it makes sense even to continue the discussion
>    > there if so.
>=20
>    Agreed, changed the wording to "If the scope of the idea is limited to
>    a specific project the discussion may happen on the project-specific
>    list or jira."
>=20
>    > 5. How the set of editors is selected?
>    >   "The editors are apointed and removed by the PMC informally, simil=
ar
>    to
>    >   how the Apache Board appoints shepherds to projects."
>    > This needs a reference. How does Apache Board appoints shepherds?
>=20
>    Good question, anyone know? Since it's informal I imagine shepherds
>    volunteer. The editors could be a subset of the PMC that either
>    volunteers or is rotated periodically.
>=20
>    > 6. The level of design details.
>    > I think HEP should have a pretty detailed design. When people vote t=
hey
>    > will want to be sure the design can lead to a reasonable implementat=
ion.
>    > Should we say "implementation-ready design", rather than
>    > "A high-level explanation of the design."
>    > Or just
>    > "A _detailed_ explanation of the design."
>=20
>    Rewrote this section, tried to make it more explicit about giving both
>    a high-level view and complete enough description so the design can
>    lead to a reasonable implementation. Also added that this section
>    should cover how to test the design.
>=20
>    > 7. Typos:
>    > successuflly, apointed, intial
>=20
>    Fixed.
>=20
>    Updated draft follows.
>=20
>    Thanks,
>    Eli
>=20
>    HEP: 1
>    Title: HEP Purpose and Guidelines
>    Author: Eli Collins
>    Status: Draft
>=20
>    What is a HEP?
>    =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>=20
>    HEP stands for Hadoop Enhancement Proposal, and is based on Python's
>    PEP (Python Enhancement Proposal) [1].  A HEP is a document that
>    describes a new feature, it's rationale, and issues the feature needs
>    to address in order to be successfully incorporated.
>=20
>    The intent is for HEPs to be the primary mechanism for proposing
>    significant new features to core Hadoop (common, HDFS and MapReduce),
>    incorporating community feedback, and recording the proposal.  Going
>    through the HEP process should improve the chances that a proposal is
>    successful.
>=20
>    While HEPs do not need to come with code, they are a mechanism to
>    propose features to the community, with the intent of contributing the
>    feature, rather than request the community implement a feature.
>=20
>    HEPs must be consistent with Apache bylaws [2], for example, the HEP
>    workflow takes place on the public Apache Hadoop lists.
>=20
>    When is a HEP Required?
>    =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>=20
>    HEPs should not impede casual contribution to Hadoop.  Small
>    improvements and bugs do not require HEPs.  Not all features need
>    HEPs.  While the decision is subjective, here are some guidelines to
>    indicate a HEP should be considered:
>=20
>    - The feature impacts backwards compatibility (eg modifies released
>    public APIs in an incompatible way).
>=20
>    - The feature requires that an existing component be substantially
>    re-designed (eg NameNode modified to use Bookkeeper).
>=20
>    - The implementation impact multiple parts of the system (eg symbolic
>    links versus adding a pluggable component like a codec).
>=20
>    - The feature impacts the entire development community (eg converts
>    the build system to use maven).
>=20
>    HEP Workflow
>    =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>=20
>    The author of a HEP should first try to determine if their idea is
>    HEP-able by sending mail to the general list.  If the scope of the
>    idea is limited to a specific project the discussion may happen on the
>    project-specific list or jira.  This gives the author a chance to
>    flesh out the proposal, address initial concerns, and figure out
>    whether it has a chance of being accepted.  The author's role is to
>    build consensus, and gather dissenting opinions.
>=20
>    Following this discussion the author should draft a HEP proposal
>    following the HEP template. The proposal should accurately reflect and
>    address feedback and dissenting opinions.  For example, flesh out
>    sections on backwards compatibility or testing. The author should send
>    the draft of the proposal to hep@hadoop.apache.org for review.  This
>    is a new, public list for editors and those interested in following
>    the review process.
>=20
>    A set of editors reviews incoming HEPs. Each HEP is assigned a single
>    primary editor. An editor may volunteer if they feel particular
>    functional expertise is required or assign HEPs to editors round
>    robin.
>=20
>    The editor reviews the proposal and may request it be updated if it
>    does not sufficiently address feedback raised during discussion, eg
>    why the proposal is not redundant with existing functionality, or is
>    technically sound, sufficiently motivated, covers backwards
>    compatibility, etc. As updates are necessary, the HEP author can check
>    in new versions if they have commit permissions, or can email new HEP
>    versions to the editor for committing. In order to ensure HEP
>    proposals make progress the editor should respond to proposal drafts
>    within two weeks of receiving them (or the proposer can request
>    another editor), and the proposer should generate updates to the draft
>    within two weeks of receiving feedback from the editor.
>=20
>    The editor's role is to determine if the proposal is complete, so that
>    the proposal can be voted on, not whether they agree with the proposal
>    itself.  The editor's involvement should increase the chance that a
>    HEP proposal makes it to a vote.
>=20
>    Once the editor deems the proposal is complete they add it to a
>    versioned HEP repository and the author posts the proposal to
>    general@hadoop.apache.org for vote.  HEP votes, like Apache procedural
>    votes, use majority approval [3]. Only PMC members have binding votes.
>    Votes are open for a period of 1 week to allow all active voters time
>    to consider the proposal. Successful HEPs are assigned a number,
>    unsuccessful HEPs remain drafts.
>=20
>    The editors are appointed and removed by the PMC informally, similar
>    to how the Apache Board appoints shepherds to projects.
>=20
>    HEP Contents
>    =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>=20
>    Each HEP should contain the following:
>=20
>    1. Preamble -- Including the HEP number, a short descriptive title,
>    and the names of the authors.
>=20
>    2. Abstract -- A short (~200 word) description of the technical issue
>    being addressed.
>=20
>    3. Copyright/public domain -- Each HEP must either be explicitly
>    labelled as placed in the public domain (see this HEP as an example).
>=20
>    4. Design -- This section should give both a high-level view and a
>    complete description of the feature.  While the design does not need
>    to cover implementation detail it should be clear to the reader that
>    the design can lead to a reasonable implementation.  This section
>    should cover intended use cases, failure scenarios, strategies for
>    testing, and impact on the existing system.
>=20
>    5. Motivation -- The motivation spells out the use case for the
>    feature and the benefits it provides.
>=20
>    6. Rationale -- The rationale describes what motivated the design and
>    why particular design decisions were made.  It should describe
>    alternate designs that were considered and related work, e.g. how the
>    feature is designed in other systems. It should also consider whether
>    the feature could be achieved by layering atop the existing system
>    rather than modifying it.
>=20
>    The rationale should provide evidence of consensus within the
>    community and discuss important objections or concerns raised during
>    discussion.
>=20
>    7. Backwards Compatibility -- All HEPs that introduce backwards
>    incompatibilities must include a section describing these
>    incompatibilities and their severity.  The HEP must explain how the
>    author proposes to deal with these incompatibilities.  HEP submissions
>    without a sufficient backwards compatibility treatise may be rejected
>    outright.
>=20
>    HEP Template
>    =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>=20
>    HEPs should be plain text with minimal structural markup that adheres
>    to a rigid style.  You can use this HEP as an example. Each HEP starts
>    with a header that contains the HEP number (or empty if the number has
>    not yet been assigned), title, list of authors and status (Draft,
>    Accepted, Rejected, or Withdrawn).
>=20
>    Auxiliary Files
>    =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>=20
>    HEPs may include auxiliary files such as diagrams.  Such files must be
>    named ``hep-XXXX-Y.ext``, where "XXXX" is the HEP number, "Y" is a
>    serial number (starting at 1), and "ext" is replaced by the actual
>    file extension (e.g. "png").
>=20
>    References
>    =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>=20
>    1. http://www.python.org/dev/peps/pep-0001
>=20
>    2. http://www.apache.org/foundation/bylaws.html
>=20
>    3. http://www.apache.org/foundation/glossary.html#MajorityApproval
>=20
>    Copyright
>    =3D=3D=3D=3D=3D=3D=3D=3D=3D
>=20
>    This document has been placed in the public domain.

--TU+u6i6jrDPzmlWF
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)

iF4EAREIAAYFAkw+DC0ACgkQMqXifkwDoaGyCAD/YQO2XmFYRJaA1j1W54vBajXg
8C252gCSr056wtKDcGYBAJqAet0YEsOu+g+clF77Vu+3r3I0TL5RHia4Im1n+6Tq
=lscd
-----END PGP SIGNATURE-----

--TU+u6i6jrDPzmlWF--

From general-return-1809-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 19:56:37 2010
Return-Path: <general-return-1809-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38740 invoked from network); 14 Jul 2010 19:56:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 19:56:37 -0000
Received: (qmail 49328 invoked by uid 500); 14 Jul 2010 19:56:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49241 invoked by uid 500); 14 Jul 2010 19:56:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49233 invoked by uid 99); 14 Jul 2010 19:56:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 19:56:35 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 19:56:29 +0000
Received: by pzk36 with SMTP id 36so72900pzk.35
        for <general@hadoop.apache.org>; Wed, 14 Jul 2010 12:56:07 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.142.213.13 with SMTP id l13mr452654wfg.155.1279137367318; Wed, 
	14 Jul 2010 12:56:07 -0700 (PDT)
Received: by 10.142.105.5 with HTTP; Wed, 14 Jul 2010 12:56:07 -0700 (PDT)
In-Reply-To: <20100714191245.GF3418@goodenter-lm.boudnik.org>
References: <AANLkTilRca-SDdWa0rZtNjdbnnzk3m0AG9JGwUBSGIaK@mail.gmail.com>
	<4C3CD27F.60003@yahoo-inc.com>
	<AANLkTina2dxXd61b5zj4ZpS6-MK3jpzx47ecc3xZkP-r@mail.gmail.com>
	<20100714191245.GF3418@goodenter-lm.boudnik.org>
Date: Wed, 14 Jul 2010 12:56:07 -0700
Message-ID: <AANLkTilE1lkbzff_QtDbUJLxNyzC8n00Curvzt3-_nm9@mail.gmail.com>
Subject: Re: HEP proposal
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Wed, Jul 14, 2010 at 12:12 PM, Konstantin Boudnik <cos@yahoo-inc.com> wr=
ote:
> I have been following this discussion for some time now and the only ques=
tion
> came to my mind: why mimicking PEP? Is it so astonishingly successful or =
is
> it much better than Apache voting or RFC process (from where it has been
> apparently derived).

I thought it would be more fruitful to adapt an existing, working
model rather than invent a new one. I based it on PEP after looking at
what other projects used; PEP seems to strike a balance between no
structure and more heavy weight processes (JSR, RFC).  I'm open to
other models if there's something you think is more suitable.

> So far I see HEP as an over-complicated process for a process sake. I'd a=
ppreciate if some one can chip-in and tell me if and where I'm wrong.

I think that's a reasonable opinion. There are communities that
function without process.

In the limited time I've been working on Hadoop there's been tension
between substantially modifying the system and preserving it's
stability. It seemed to me that some additional structure around
discussing change up front would help relieve some of that tension.
There is value in thinking and discussing what you're doing up front
in a way that's visible to the community. HEP essentially enforces
that. If that happens naturally I agree we don't need the process. If
people think what we've currently got is sufficient I'm happy to chalk
it up as a learning experience, don't want people to adopt it just
because I've taken the time to draft something.

Thanks,
Eli

> Thanks,
> =A0Cos
>
> On Wed, Jul 14, 2010 at 10:46AM, Eli Collins wrote:
>> =A0 =A0Hey Konstantin,
>>
>> =A0 =A0Thanks for taking a look, comments in-line.
>>
>> =A0 =A0On Tue, Jul 13, 2010 at 1:54 PM, Konstantin Shvachko <shv@yahoo-i=
nc.com>
>> =A0 =A0wrote:
>> =A0 =A0> Eli,
>> =A0 =A0>
>> =A0 =A0> Thanks for a really good proposal.
>> =A0 =A0> Some questions / comments:
>> =A0 =A0>
>> =A0 =A0> On voting
>> =A0 =A0> 1. Which voting rule?
>> =A0 =A0> http://www.apache.org/foundation/glossary.html#ConsensusApprova=
l
>> =A0 =A0> http://www.apache.org/foundation/glossary.html#MajorityApproval
>> =A0 =A0> I think you mean the MajorityApproval as it does not have veto =
rule.
>> =A0 =A0> So may be it's just clarifying the reference.
>>
>> =A0 =A0Good point, clarified so it's majority approval.
>>
>> =A0 =A0> 2. Who can vote?
>> =A0 =A0> Usually PMCs have Binding Votes.
>> =A0 =A0> Would be good to have a sentence clarifying this.
>>
>> =A0 =A0Yup, added.
>>
>> =A0 =A0> 3. How long does the vote go?
>> =A0 =A0> Usual 3 days may not be enough. One week is reasonable?
>>
>> =A0 =A0Specified one week.
>>
>> =A0 =A0> 4. Discussion on public lists.
>> =A0 =A0> A HEP can evolve from a jira, then it should be counted as a pu=
blic
>> =A0 =A0> discussion. I think it makes sense even to continue the discuss=
ion
>> =A0 =A0> there if so.
>>
>> =A0 =A0Agreed, changed the wording to "If the scope of the idea is limit=
ed to
>> =A0 =A0a specific project the discussion may happen on the project-speci=
fic
>> =A0 =A0list or jira."
>>
>> =A0 =A0> 5. How the set of editors is selected?
>> =A0 =A0> =A0 "The editors are apointed and removed by the PMC informally=
, similar
>> =A0 =A0to
>> =A0 =A0> =A0 how the Apache Board appoints shepherds to projects."
>> =A0 =A0> This needs a reference. How does Apache Board appoints shepherd=
s?
>>
>> =A0 =A0Good question, anyone know? Since it's informal I imagine shepher=
ds
>> =A0 =A0volunteer. The editors could be a subset of the PMC that either
>> =A0 =A0volunteers or is rotated periodically.
>>
>> =A0 =A0> 6. The level of design details.
>> =A0 =A0> I think HEP should have a pretty detailed design. When people v=
ote they
>> =A0 =A0> will want to be sure the design can lead to a reasonable implem=
entation.
>> =A0 =A0> Should we say "implementation-ready design", rather than
>> =A0 =A0> "A high-level explanation of the design."
>> =A0 =A0> Or just
>> =A0 =A0> "A _detailed_ explanation of the design."
>>
>> =A0 =A0Rewrote this section, tried to make it more explicit about giving=
 both
>> =A0 =A0a high-level view and complete enough description so the design c=
an
>> =A0 =A0lead to a reasonable implementation. Also added that this section
>> =A0 =A0should cover how to test the design.
>>
>> =A0 =A0> 7. Typos:
>> =A0 =A0> successuflly, apointed, intial
>>
>> =A0 =A0Fixed.
>>
>> =A0 =A0Updated draft follows.
>>
>> =A0 =A0Thanks,
>> =A0 =A0Eli
>>
>> =A0 =A0HEP: 1
>> =A0 =A0Title: HEP Purpose and Guidelines
>> =A0 =A0Author: Eli Collins
>> =A0 =A0Status: Draft
>>
>> =A0 =A0What is a HEP?
>> =A0 =A0=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>
>> =A0 =A0HEP stands for Hadoop Enhancement Proposal, and is based on Pytho=
n's
>> =A0 =A0PEP (Python Enhancement Proposal) [1]. =A0A HEP is a document tha=
t
>> =A0 =A0describes a new feature, it's rationale, and issues the feature n=
eeds
>> =A0 =A0to address in order to be successfully incorporated.
>>
>> =A0 =A0The intent is for HEPs to be the primary mechanism for proposing
>> =A0 =A0significant new features to core Hadoop (common, HDFS and MapRedu=
ce),
>> =A0 =A0incorporating community feedback, and recording the proposal. =A0=
Going
>> =A0 =A0through the HEP process should improve the chances that a proposa=
l is
>> =A0 =A0successful.
>>
>> =A0 =A0While HEPs do not need to come with code, they are a mechanism to
>> =A0 =A0propose features to the community, with the intent of contributin=
g the
>> =A0 =A0feature, rather than request the community implement a feature.
>>
>> =A0 =A0HEPs must be consistent with Apache bylaws [2], for example, the =
HEP
>> =A0 =A0workflow takes place on the public Apache Hadoop lists.
>>
>> =A0 =A0When is a HEP Required?
>> =A0 =A0=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
>>
>> =A0 =A0HEPs should not impede casual contribution to Hadoop. =A0Small
>> =A0 =A0improvements and bugs do not require HEPs. =A0Not all features ne=
ed
>> =A0 =A0HEPs. =A0While the decision is subjective, here are some guidelin=
es to
>> =A0 =A0indicate a HEP should be considered:
>>
>> =A0 =A0- The feature impacts backwards compatibility (eg modifies releas=
ed
>> =A0 =A0public APIs in an incompatible way).
>>
>> =A0 =A0- The feature requires that an existing component be substantiall=
y
>> =A0 =A0re-designed (eg NameNode modified to use Bookkeeper).
>>
>> =A0 =A0- The implementation impact multiple parts of the system (eg symb=
olic
>> =A0 =A0links versus adding a pluggable component like a codec).
>>
>> =A0 =A0- The feature impacts the entire development community (eg conver=
ts
>> =A0 =A0the build system to use maven).
>>
>> =A0 =A0HEP Workflow
>> =A0 =A0=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>
>> =A0 =A0The author of a HEP should first try to determine if their idea i=
s
>> =A0 =A0HEP-able by sending mail to the general list. =A0If the scope of =
the
>> =A0 =A0idea is limited to a specific project the discussion may happen o=
n the
>> =A0 =A0project-specific list or jira. =A0This gives the author a chance =
to
>> =A0 =A0flesh out the proposal, address initial concerns, and figure out
>> =A0 =A0whether it has a chance of being accepted. =A0The author's role i=
s to
>> =A0 =A0build consensus, and gather dissenting opinions.
>>
>> =A0 =A0Following this discussion the author should draft a HEP proposal
>> =A0 =A0following the HEP template. The proposal should accurately reflec=
t and
>> =A0 =A0address feedback and dissenting opinions. =A0For example, flesh o=
ut
>> =A0 =A0sections on backwards compatibility or testing. The author should=
 send
>> =A0 =A0the draft of the proposal to hep@hadoop.apache.org for review. =
=A0This
>> =A0 =A0is a new, public list for editors and those interested in followi=
ng
>> =A0 =A0the review process.
>>
>> =A0 =A0A set of editors reviews incoming HEPs. Each HEP is assigned a si=
ngle
>> =A0 =A0primary editor. An editor may volunteer if they feel particular
>> =A0 =A0functional expertise is required or assign HEPs to editors round
>> =A0 =A0robin.
>>
>> =A0 =A0The editor reviews the proposal and may request it be updated if =
it
>> =A0 =A0does not sufficiently address feedback raised during discussion, =
eg
>> =A0 =A0why the proposal is not redundant with existing functionality, or=
 is
>> =A0 =A0technically sound, sufficiently motivated, covers backwards
>> =A0 =A0compatibility, etc. As updates are necessary, the HEP author can =
check
>> =A0 =A0in new versions if they have commit permissions, or can email new=
 HEP
>> =A0 =A0versions to the editor for committing. In order to ensure HEP
>> =A0 =A0proposals make progress the editor should respond to proposal dra=
fts
>> =A0 =A0within two weeks of receiving them (or the proposer can request
>> =A0 =A0another editor), and the proposer should generate updates to the =
draft
>> =A0 =A0within two weeks of receiving feedback from the editor.
>>
>> =A0 =A0The editor's role is to determine if the proposal is complete, so=
 that
>> =A0 =A0the proposal can be voted on, not whether they agree with the pro=
posal
>> =A0 =A0itself. =A0The editor's involvement should increase the chance th=
at a
>> =A0 =A0HEP proposal makes it to a vote.
>>
>> =A0 =A0Once the editor deems the proposal is complete they add it to a
>> =A0 =A0versioned HEP repository and the author posts the proposal to
>> =A0 =A0general@hadoop.apache.org for vote. =A0HEP votes, like Apache pro=
cedural
>> =A0 =A0votes, use majority approval [3]. Only PMC members have binding v=
otes.
>> =A0 =A0Votes are open for a period of 1 week to allow all active voters =
time
>> =A0 =A0to consider the proposal. Successful HEPs are assigned a number,
>> =A0 =A0unsuccessful HEPs remain drafts.
>>
>> =A0 =A0The editors are appointed and removed by the PMC informally, simi=
lar
>> =A0 =A0to how the Apache Board appoints shepherds to projects.
>>
>> =A0 =A0HEP Contents
>> =A0 =A0=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>
>> =A0 =A0Each HEP should contain the following:
>>
>> =A0 =A01. Preamble -- Including the HEP number, a short descriptive titl=
e,
>> =A0 =A0and the names of the authors.
>>
>> =A0 =A02. Abstract -- A short (~200 word) description of the technical i=
ssue
>> =A0 =A0being addressed.
>>
>> =A0 =A03. Copyright/public domain -- Each HEP must either be explicitly
>> =A0 =A0labelled as placed in the public domain (see this HEP as an examp=
le).
>>
>> =A0 =A04. Design -- This section should give both a high-level view and =
a
>> =A0 =A0complete description of the feature. =A0While the design does not=
 need
>> =A0 =A0to cover implementation detail it should be clear to the reader t=
hat
>> =A0 =A0the design can lead to a reasonable implementation. =A0This secti=
on
>> =A0 =A0should cover intended use cases, failure scenarios, strategies fo=
r
>> =A0 =A0testing, and impact on the existing system.
>>
>> =A0 =A05. Motivation -- The motivation spells out the use case for the
>> =A0 =A0feature and the benefits it provides.
>>
>> =A0 =A06. Rationale -- The rationale describes what motivated the design=
 and
>> =A0 =A0why particular design decisions were made. =A0It should describe
>> =A0 =A0alternate designs that were considered and related work, e.g. how=
 the
>> =A0 =A0feature is designed in other systems. It should also consider whe=
ther
>> =A0 =A0the feature could be achieved by layering atop the existing syste=
m
>> =A0 =A0rather than modifying it.
>>
>> =A0 =A0The rationale should provide evidence of consensus within the
>> =A0 =A0community and discuss important objections or concerns raised dur=
ing
>> =A0 =A0discussion.
>>
>> =A0 =A07. Backwards Compatibility -- All HEPs that introduce backwards
>> =A0 =A0incompatibilities must include a section describing these
>> =A0 =A0incompatibilities and their severity. =A0The HEP must explain how=
 the
>> =A0 =A0author proposes to deal with these incompatibilities. =A0HEP subm=
issions
>> =A0 =A0without a sufficient backwards compatibility treatise may be reje=
cted
>> =A0 =A0outright.
>>
>> =A0 =A0HEP Template
>> =A0 =A0=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>
>> =A0 =A0HEPs should be plain text with minimal structural markup that adh=
eres
>> =A0 =A0to a rigid style. =A0You can use this HEP as an example. Each HEP=
 starts
>> =A0 =A0with a header that contains the HEP number (or empty if the numbe=
r has
>> =A0 =A0not yet been assigned), title, list of authors and status (Draft,
>> =A0 =A0Accepted, Rejected, or Withdrawn).
>>
>> =A0 =A0Auxiliary Files
>> =A0 =A0=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>
>> =A0 =A0HEPs may include auxiliary files such as diagrams. =A0Such files =
must be
>> =A0 =A0named ``hep-XXXX-Y.ext``, where "XXXX" is the HEP number, "Y" is =
a
>> =A0 =A0serial number (starting at 1), and "ext" is replaced by the actua=
l
>> =A0 =A0file extension (e.g. "png").
>>
>> =A0 =A0References
>> =A0 =A0=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>
>> =A0 =A01. http://www.python.org/dev/peps/pep-0001
>>
>> =A0 =A02. http://www.apache.org/foundation/bylaws.html
>>
>> =A0 =A03. http://www.apache.org/foundation/glossary.html#MajorityApprova=
l
>>
>> =A0 =A0Copyright
>> =A0 =A0=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>
>> =A0 =A0This document has been placed in the public domain.
>

From general-return-1810-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 20:09:25 2010
Return-Path: <general-return-1810-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46729 invoked from network); 14 Jul 2010 20:09:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 20:09:25 -0000
Received: (qmail 61799 invoked by uid 500); 14 Jul 2010 20:09:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61665 invoked by uid 500); 14 Jul 2010 20:09:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61656 invoked by uid 99); 14 Jul 2010 20:09:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 20:09:23 +0000
X-ASF-Spam-Status: No, hits=4.7 required=10.0
	tests=FREEMAIL_FROM,FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 20:09:15 +0000
Received: by gyg10 with SMTP id 10so205208gyg.35
        for <general@hadoop.apache.org>; Wed, 14 Jul 2010 13:08:54 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=HOFSBg9pDcFHxqCBCSRblSipZBj8Sshs43ik/0FW+Y0=;
        b=KQE6xHgXWYIm5AShKyfpFmC74xCIYRserK8aEpPYGa0v3kLBngGQAKZb0kCfKQQnFq
         uf9iDiT4mvzu5b3WvMsnEjZqBQPf1DUuc85AkF/S4o9hvl9JjUub1xWUDHG6A7uZfRFP
         1NQvSoxYDTh3hNojRmRWlDk/O56Tpu+kbtuHs=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=YrfoPucyROd3ToGUMTXa2gCeTvku+Ez3bE72A2aTmL5II8jYJI4DuLkfa1mQf7DGJl
         hVq0fMPUPsEDLfc6/+93VMGxDLP6epg3Ul4Icc/1Ds87jketD2MsD47sic8W4VKocrsQ
         eBcwaE4M+8B0m6+lMHKKfDMJmt4a6w+OJ5w1g=
MIME-Version: 1.0
Received: by 10.101.68.6 with SMTP id v6mr19119258ank.117.1279138134647; Wed, 
	14 Jul 2010 13:08:54 -0700 (PDT)
Received: by 10.101.4.24 with HTTP; Wed, 14 Jul 2010 13:08:54 -0700 (PDT)
In-Reply-To: <AANLkTilZN81QEtePblrymRsHRZE6WpnRW2hpjt4fFia3@mail.gmail.com>
References: <AANLkTinnda4iVR94vHyOiVOCX0nfCWltt2ukaEtvmTNz@mail.gmail.com>
	<AANLkTik4AFGGi6A93gmjMjBhHhIVNhtZrudE_6ORsqJu@mail.gmail.com>
	<AANLkTilZN81QEtePblrymRsHRZE6WpnRW2hpjt4fFia3@mail.gmail.com>
Date: Wed, 14 Jul 2010 13:08:54 -0700
Message-ID: <AANLkTin2VtbPRizRxKIegjH0Vz6gxb2kDsLF6I68oEgL@mail.gmail.com>
Subject: Re: Exception in thread "main" java.lang.ClassNotFoundException in 
	Hadoop
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016368e1ee8226407048b5e8e27
X-Virus-Checked: Checked by ClamAV on apache.org

--0016368e1ee8226407048b5e8e27
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

I assume you have used jar command to confirm that org.myorg.WordCount is i=
n
wordcount.jar

On Mon, Jul 12, 2010 at 10:59 PM, james isaac <jamesisaac.dell@gmail.com>wr=
ote:

> This is the command i used and prompt is pointing to the current director=
y
> where wordcount.jar resides. I have also set the path for the HADOOP_HOME
> and added the bin directory of hadoop to the classpath.
>
> current directory =3D /home/user/demo/wordcount
> HADOOP_HOME=3D/home/user/hadoop_sws/hadoop-0.20.2
> $ hadoop jar wordcount.jar org.myorg.WordCount /hdfs/data/input
> /hdfs/data/output
>
>
>
> On Mon, Jul 12, 2010 at 9:14 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > Please give the commandline you used.
> > You should have specified the jar containing WordCount class.
> >
> > On Mon, Jul 12, 2010 at 5:15 AM, james isaac <jamesisaac.dell@gmail.com
> > >wrote:
> >
> > > Hi,
> > >
> > > I have just started my career with Hadoop. I tried to execute the
> example
> > > given in
> > > http://hadoop.apache.org/common/docs/r0.20.1/mapred_tutorial.htmlas
> > > explained in the tutorial. I was not able to execute the program as
> > > described in the tutorial. I am getting the error as shown below. I
> tried
> > > executing with some other examples. They are also showing the same
> > > =93ClassNotFoundException=94. I tried to find the solution in various
> > websites
> > > but i could not find it. Please help me to find the problem.
> > >
> > > Exception in thread "main"
> > > java.lang.ClassNotFoundException: org.myorg.WordCount
> > > at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> > > at java.security.AccessController.doPrivileged(Native Method)
> > > at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> > > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> > > at java.lang.Class.forName0(Native Method)
> > > at java.lang.Class.forName(Class.java:247)
> > > at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
> > >
> >
>

--0016368e1ee8226407048b5e8e27--

From general-return-1811-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 22:01:53 2010
Return-Path: <general-return-1811-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98668 invoked from network); 14 Jul 2010 22:01:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 22:01:52 -0000
Received: (qmail 23298 invoked by uid 500); 14 Jul 2010 22:01:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23244 invoked by uid 500); 14 Jul 2010 22:01:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23236 invoked by uid 99); 14 Jul 2010 22:01:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 22:01:51 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.212.176] (HELO mail-px0-f176.google.com) (209.85.212.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 22:01:43 +0000
Received: by pxi11 with SMTP id 11so323448pxi.35
        for <general@hadoop.apache.org>; Wed, 14 Jul 2010 15:00:21 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.142.142.15 with SMTP id p15mr18481449wfd.208.1279144820950; 
	Wed, 14 Jul 2010 15:00:20 -0700 (PDT)
Received: by 10.142.105.5 with HTTP; Wed, 14 Jul 2010 15:00:20 -0700 (PDT)
In-Reply-To: <4C3E08B4.60205@apache.org>
References: <AANLkTilRca-SDdWa0rZtNjdbnnzk3m0AG9JGwUBSGIaK@mail.gmail.com>
	<4C3CD27F.60003@yahoo-inc.com>
	<AANLkTina2dxXd61b5zj4ZpS6-MK3jpzx47ecc3xZkP-r@mail.gmail.com>
	<4C3E08B4.60205@apache.org>
Date: Wed, 14 Jul 2010 15:00:20 -0700
Message-ID: <AANLkTiklz1m_splseCb03x9IC1dQD0wpMLORcu0Oe_bd@mail.gmail.com>
Subject: Re: HEP proposal
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd2e6b8abc64c048b601cc8
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd2e6b8abc64c048b601cc8
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Jul 14, 2010 at 11:57 AM, Doug Cutting <cutting@apache.org> wrote:

> On 07/14/2010 10:46 AM, Eli Collins wrote:
>
>> 5. How the set of editors is selected?
>>>   "The editors are apointed and removed by the PMC informally, similar to
>>>   how the Apache Board appoints shepherds to projects."
>>> This needs a reference. How does Apache Board appoints shepherds?
>>>
>>
>> Good question, anyone know? Since it's informal I imagine shepherds
>> volunteer. The editors could be a subset of the PMC that either
>> volunteers or is rotated periodically.
>>
>
> Shepherds are assigned randomly by the chair 48 hours before each meeting.
>  Perhaps a better analogy for an editor is an Incubator Champion?
>
>
> http://incubator.apache.org/incubation/Roles_and_Responsibilities.html#Champion
>
> Champions are volunteers.  So, rather than assigning an editor, a HEP might
> ask for one to volunteer.  Any PMC member might volunteer.
>

That sounds reasonable. Having a PMC member volunteer will likely result in
someone who has more expertise in the area and will perhaps be more
responsive.

Thanks,
Eli

--000e0cd2e6b8abc64c048b601cc8--

From general-return-1812-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 01:25:42 2010
Return-Path: <general-return-1812-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67299 invoked from network); 15 Jul 2010 01:25:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 01:25:41 -0000
Received: (qmail 21612 invoked by uid 500); 15 Jul 2010 01:25:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21562 invoked by uid 500); 15 Jul 2010 01:25:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21554 invoked by uid 99); 15 Jul 2010 01:25:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 01:25:39 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 01:25:33 +0000
Received: by vws11 with SMTP id 11so472521vws.35
        for <general@hadoop.apache.org>; Wed, 14 Jul 2010 18:24:12 -0700 (PDT)
Received: by 10.220.157.140 with SMTP id b12mr9219815vcx.16.1279157052157; 
	Wed, 14 Jul 2010 18:24:12 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.220.174.100 with HTTP; Wed, 14 Jul 2010 18:23:52 -0700 (PDT)
In-Reply-To: <AANLkTiklz1m_splseCb03x9IC1dQD0wpMLORcu0Oe_bd@mail.gmail.com>
References: <AANLkTilRca-SDdWa0rZtNjdbnnzk3m0AG9JGwUBSGIaK@mail.gmail.com> 
	<4C3CD27F.60003@yahoo-inc.com> <AANLkTina2dxXd61b5zj4ZpS6-MK3jpzx47ecc3xZkP-r@mail.gmail.com> 
	<4C3E08B4.60205@apache.org> <AANLkTiklz1m_splseCb03x9IC1dQD0wpMLORcu0Oe_bd@mail.gmail.com>
From: Aaron Kimball <aaron@cloudera.com>
Date: Wed, 14 Jul 2010 18:23:52 -0700
Message-ID: <AANLkTil_iFA_WiJVtg4EvbaPvM7a_WRiPuXy9z8OOHbT@mail.gmail.com>
Subject: Re: HEP proposal
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=e0cb4e8877d7b4adfb048b62f525
X-Virus-Checked: Checked by ClamAV on apache.org

--e0cb4e8877d7b4adfb048b62f525
Content-Type: text/plain; charset=ISO-8859-1

Eli,

Great work. I like where this is going.

Here's something that I think might be problematic:

3. Copyright/public domain -- Each HEP must either be explicitly
labelled as placed in the public domain (see this HEP as an example).


"must either be placed in the public domain, or..?" I assume the alternative
is an open source copyright license? You should state what this is. PEP's
alternative is "licensed under the Open Publication License."  (
http://www.opencontent.org/openpub/) Is that your intent?

Upon acceptance of a PEP, the PEP workflow states that it should be checked
into svn. Are we planning to host accepted HEPs in the Hadoop svn? If so, we
should specify that non-public-domain HEPs should be placed under a license
listed on http://www.apache.org/legal/resolved.html. The Open Publication
License isn't listed there. Nor are the GFDL and other documentation
licenses (presumably not because they're not specifically allowed, but
because Apache deals with code more than documentation). Note that
CC/Attribution/Share-Alike is allowed. You may want to recommend that one?

- Aaron


On Wed, Jul 14, 2010 at 3:00 PM, Eli Collins <eli@cloudera.com> wrote:

> On Wed, Jul 14, 2010 at 11:57 AM, Doug Cutting <cutting@apache.org> wrote:
>
> > On 07/14/2010 10:46 AM, Eli Collins wrote:
> >
> >> 5. How the set of editors is selected?
> >>>   "The editors are apointed and removed by the PMC informally, similar
> to
> >>>   how the Apache Board appoints shepherds to projects."
> >>> This needs a reference. How does Apache Board appoints shepherds?
> >>>
> >>
> >> Good question, anyone know? Since it's informal I imagine shepherds
> >> volunteer. The editors could be a subset of the PMC that either
> >> volunteers or is rotated periodically.
> >>
> >
> > Shepherds are assigned randomly by the chair 48 hours before each
> meeting.
> >  Perhaps a better analogy for an editor is an Incubator Champion?
> >
> >
> >
> http://incubator.apache.org/incubation/Roles_and_Responsibilities.html#Champion
> >
> > Champions are volunteers.  So, rather than assigning an editor, a HEP
> might
> > ask for one to volunteer.  Any PMC member might volunteer.
> >
>
> That sounds reasonable. Having a PMC member volunteer will likely result in
> someone who has more expertise in the area and will perhaps be more
> responsive.
>
> Thanks,
> Eli
>

--e0cb4e8877d7b4adfb048b62f525--

From general-return-1813-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 05:37:57 2010
Return-Path: <general-return-1813-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33475 invoked from network); 15 Jul 2010 05:37:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 05:37:57 -0000
Received: (qmail 60390 invoked by uid 500); 15 Jul 2010 05:37:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60022 invoked by uid 500); 15 Jul 2010 05:37:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60006 invoked by uid 99); 15 Jul 2010 05:37:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 05:37:53 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of dhruba@gmail.com designates 74.125.83.48 as permitted sender)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 05:37:47 +0000
Received: by gwj20 with SMTP id 20so379287gwj.35
        for <general@hadoop.apache.org>; Wed, 14 Jul 2010 22:36:26 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=MxHRGGPVqQFmCLXb0MdmYiyCU2LUmVgm687xzlIJBcI=;
        b=PbDe9uKhM+mXGKRyT8E7lpDa+dCwnNTxePCHh0sB8ZDhmWHJWfjflBJc/CyLloRAlH
         8oc6rIFJLN65ep2vn/Q1plG9p3PhzDogu8br9CIIyXM8VTXpVdgdXsIJumkXRZfCx80H
         uY8fLAK7t2645XjmyuyaKu584Ae6WYbKoLRGU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=SxXnYE37VljchRRTrWijtAcmINuuZ4NtX9tQZrMgt92oHAoBCaouzUOmvuhPy4Y51E
         2XyLIrpHBueBmBNLjHnR1FvkuJTzb+Sg2aKhRW2KJY/mV3BdW7tMUZmAQX0bogoV04ER
         XfsxUMh0zbHcBU7TNstgNnBw5vt/B/Z3HeKI8=
MIME-Version: 1.0
Received: by 10.151.18.21 with SMTP id v21mr8977454ybi.237.1279172185839; Wed, 
	14 Jul 2010 22:36:25 -0700 (PDT)
Received: by 10.151.107.2 with HTTP; Wed, 14 Jul 2010 22:36:25 -0700 (PDT)
In-Reply-To: <AANLkTil_iFA_WiJVtg4EvbaPvM7a_WRiPuXy9z8OOHbT@mail.gmail.com>
References: <AANLkTilRca-SDdWa0rZtNjdbnnzk3m0AG9JGwUBSGIaK@mail.gmail.com>
	<4C3CD27F.60003@yahoo-inc.com>
	<AANLkTina2dxXd61b5zj4ZpS6-MK3jpzx47ecc3xZkP-r@mail.gmail.com>
	<4C3E08B4.60205@apache.org>
	<AANLkTiklz1m_splseCb03x9IC1dQD0wpMLORcu0Oe_bd@mail.gmail.com>
	<AANLkTil_iFA_WiJVtg4EvbaPvM7a_WRiPuXy9z8OOHbT@mail.gmail.com>
Date: Wed, 14 Jul 2010 22:36:25 -0700
Message-ID: <AANLkTimAfxAXe0y0i6Bk0QfUkwv0RJCSqJ8zTlbIubFo@mail.gmail.com>
Subject: Re: HEP proposal
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cdf1704be602d048b667bec
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cdf1704be602d048b667bec
Content-Type: text/plain; charset=ISO-8859-1

on a related note, since a HEP-initiator might not be a apache committer,
does it mean that it is the responsibility of the editor/champion to check
versions of the HEP proposal into svn?

thanks,
dhruba

On Wed, Jul 14, 2010 at 6:23 PM, Aaron Kimball <aaron@cloudera.com> wrote:

> Eli,
>
> Great work. I like where this is going.
>
> Here's something that I think might be problematic:
>
> 3. Copyright/public domain -- Each HEP must either be explicitly
> labelled as placed in the public domain (see this HEP as an example).
>
>
> "must either be placed in the public domain, or..?" I assume the
> alternative
> is an open source copyright license? You should state what this is. PEP's
> alternative is "licensed under the Open Publication License."  (
> http://www.opencontent.org/openpub/) Is that your intent?
>
> Upon acceptance of a PEP, the PEP workflow states that it should be checked
> into svn. Are we planning to host accepted HEPs in the Hadoop svn? If so,
> we
> should specify that non-public-domain HEPs should be placed under a license
> listed on http://www.apache.org/legal/resolved.html. The Open Publication
> License isn't listed there. Nor are the GFDL and other documentation
> licenses (presumably not because they're not specifically allowed, but
> because Apache deals with code more than documentation). Note that
> CC/Attribution/Share-Alike is allowed. You may want to recommend that one?
>
> - Aaron
>
>
> On Wed, Jul 14, 2010 at 3:00 PM, Eli Collins <eli@cloudera.com> wrote:
>
> > On Wed, Jul 14, 2010 at 11:57 AM, Doug Cutting <cutting@apache.org>
> wrote:
> >
> > > On 07/14/2010 10:46 AM, Eli Collins wrote:
> > >
> > >> 5. How the set of editors is selected?
> > >>>   "The editors are apointed and removed by the PMC informally,
> similar
> > to
> > >>>   how the Apache Board appoints shepherds to projects."
> > >>> This needs a reference. How does Apache Board appoints shepherds?
> > >>>
> > >>
> > >> Good question, anyone know? Since it's informal I imagine shepherds
> > >> volunteer. The editors could be a subset of the PMC that either
> > >> volunteers or is rotated periodically.
> > >>
> > >
> > > Shepherds are assigned randomly by the chair 48 hours before each
> > meeting.
> > >  Perhaps a better analogy for an editor is an Incubator Champion?
> > >
> > >
> > >
> >
> http://incubator.apache.org/incubation/Roles_and_Responsibilities.html#Champion
> > >
> > > Champions are volunteers.  So, rather than assigning an editor, a HEP
> > might
> > > ask for one to volunteer.  Any PMC member might volunteer.
> > >
> >
> > That sounds reasonable. Having a PMC member volunteer will likely result
> in
> > someone who has more expertise in the area and will perhaps be more
> > responsive.
> >
> > Thanks,
> > Eli
> >
>



-- 
Connect to me at http://www.facebook.com/dhruba

--000e0cdf1704be602d048b667bec--

From general-return-1814-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 06:41:33 2010
Return-Path: <general-return-1814-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57644 invoked from network); 15 Jul 2010 06:41:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 06:41:33 -0000
Received: (qmail 95887 invoked by uid 500); 15 Jul 2010 06:41:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95495 invoked by uid 500); 15 Jul 2010 06:41:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95487 invoked by uid 99); 15 Jul 2010 06:41:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 06:41:29 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 06:41:22 +0000
Received: by pzk36 with SMTP id 36so276368pzk.35
        for <general@hadoop.apache.org>; Wed, 14 Jul 2010 23:41:01 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.142.211.6 with SMTP id j6mr23044062wfg.164.1279176060676; Wed, 
	14 Jul 2010 23:41:00 -0700 (PDT)
Received: by 10.142.105.5 with HTTP; Wed, 14 Jul 2010 23:41:00 -0700 (PDT)
In-Reply-To: <AANLkTil_iFA_WiJVtg4EvbaPvM7a_WRiPuXy9z8OOHbT@mail.gmail.com>
References: <AANLkTilRca-SDdWa0rZtNjdbnnzk3m0AG9JGwUBSGIaK@mail.gmail.com>
	<4C3CD27F.60003@yahoo-inc.com>
	<AANLkTina2dxXd61b5zj4ZpS6-MK3jpzx47ecc3xZkP-r@mail.gmail.com>
	<4C3E08B4.60205@apache.org>
	<AANLkTiklz1m_splseCb03x9IC1dQD0wpMLORcu0Oe_bd@mail.gmail.com>
	<AANLkTil_iFA_WiJVtg4EvbaPvM7a_WRiPuXy9z8OOHbT@mail.gmail.com>
Date: Wed, 14 Jul 2010 23:41:00 -0700
Message-ID: <AANLkTikSBkEsRg2d45NGmF_eaht_dBAjhydIYTYGsTTU@mail.gmail.com>
Subject: Re: HEP proposal
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Wed, Jul 14, 2010 at 6:23 PM, Aaron Kimball <aaron@cloudera.com> wrote:
> Eli,
>
> Great work. I like where this is going.
>
> Here's something that I think might be problematic:
>
> 3. Copyright/public domain -- Each HEP must either be explicitly
> labelled as placed in the public domain (see this HEP as an example).
>
>
> "must either be placed in the public domain, or..?" I assume the alternat=
ive
> is an open source copyright license?

I was assuming HEPs would just be placed in the public domain, period.
Like design docs posted on jira they're just text posted to a mailing
list, not sure checking them into svn needs to require they have an
alternate license. Something I'm missing?

Thanks,
Eli



You should state what this is. PEP's
> alternative is "licensed under the Open Publication License." =A0(
> http://www.opencontent.org/openpub/) Is that your intent?
>
> Upon acceptance of a PEP, the PEP workflow states that it should be check=
ed
> into svn. Are we planning to host accepted HEPs in the Hadoop svn? If so,=
 we
> should specify that non-public-domain HEPs should be placed under a licen=
se
> listed on http://www.apache.org/legal/resolved.html. The Open Publication
> License isn't listed there. Nor are the GFDL and other documentation
> licenses (presumably not because they're not specifically allowed, but
> because Apache deals with code more than documentation). Note that
> CC/Attribution/Share-Alike is allowed. You may want to recommend that one=
?
>
> - Aaron
>
>
> On Wed, Jul 14, 2010 at 3:00 PM, Eli Collins <eli@cloudera.com> wrote:
>
>> On Wed, Jul 14, 2010 at 11:57 AM, Doug Cutting <cutting@apache.org> wrot=
e:
>>
>> > On 07/14/2010 10:46 AM, Eli Collins wrote:
>> >
>> >> 5. How the set of editors is selected?
>> >>> =A0 "The editors are apointed and removed by the PMC informally, sim=
ilar
>> to
>> >>> =A0 how the Apache Board appoints shepherds to projects."
>> >>> This needs a reference. How does Apache Board appoints shepherds?
>> >>>
>> >>
>> >> Good question, anyone know? Since it's informal I imagine shepherds
>> >> volunteer. The editors could be a subset of the PMC that either
>> >> volunteers or is rotated periodically.
>> >>
>> >
>> > Shepherds are assigned randomly by the chair 48 hours before each
>> meeting.
>> > =A0Perhaps a better analogy for an editor is an Incubator Champion?
>> >
>> >
>> >
>> http://incubator.apache.org/incubation/Roles_and_Responsibilities.html#C=
hampion
>> >
>> > Champions are volunteers. =A0So, rather than assigning an editor, a HE=
P
>> might
>> > ask for one to volunteer. =A0Any PMC member might volunteer.
>> >
>>
>> That sounds reasonable. Having a PMC member volunteer will likely result=
 in
>> someone who has more expertise in the area and will perhaps be more
>> responsive.
>>
>> Thanks,
>> Eli
>>
>

From general-return-1815-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 06:42:40 2010
Return-Path: <general-return-1815-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57876 invoked from network); 15 Jul 2010 06:42:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 06:42:40 -0000
Received: (qmail 96764 invoked by uid 500); 15 Jul 2010 06:42:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96693 invoked by uid 500); 15 Jul 2010 06:42:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96684 invoked by uid 99); 15 Jul 2010 06:42:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 06:42:36 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.212.176] (HELO mail-px0-f176.google.com) (209.85.212.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 06:42:31 +0000
Received: by pxi11 with SMTP id 11so545534pxi.35
        for <general@hadoop.apache.org>; Wed, 14 Jul 2010 23:42:09 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.142.199.20 with SMTP id w20mr22126907wff.27.1279176129508; 
	Wed, 14 Jul 2010 23:42:09 -0700 (PDT)
Received: by 10.142.105.5 with HTTP; Wed, 14 Jul 2010 23:42:09 -0700 (PDT)
In-Reply-To: <AANLkTimAfxAXe0y0i6Bk0QfUkwv0RJCSqJ8zTlbIubFo@mail.gmail.com>
References: <AANLkTilRca-SDdWa0rZtNjdbnnzk3m0AG9JGwUBSGIaK@mail.gmail.com>
	<4C3CD27F.60003@yahoo-inc.com>
	<AANLkTina2dxXd61b5zj4ZpS6-MK3jpzx47ecc3xZkP-r@mail.gmail.com>
	<4C3E08B4.60205@apache.org>
	<AANLkTiklz1m_splseCb03x9IC1dQD0wpMLORcu0Oe_bd@mail.gmail.com>
	<AANLkTil_iFA_WiJVtg4EvbaPvM7a_WRiPuXy9z8OOHbT@mail.gmail.com>
	<AANLkTimAfxAXe0y0i6Bk0QfUkwv0RJCSqJ8zTlbIubFo@mail.gmail.com>
Date: Wed, 14 Jul 2010 23:42:09 -0700
Message-ID: <AANLkTil2vn1ZP_lBpgJwz1SL76_6NbO4EQgVwZiVHOhh@mail.gmail.com>
Subject: Re: HEP proposal
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Yup.

"As updates are necessary, the HEP author can check
in new versions if they have commit permissions, or can email new HEP
versions to the editor for committing."

Thanks,
Eli

On Wed, Jul 14, 2010 at 10:36 PM, Dhruba Borthakur <dhruba@gmail.com> wrote=
:
> on a related note, since a HEP-initiator might not be a apache committer,
> does it mean that it is the responsibility of the editor/champion to chec=
k
> versions of the HEP proposal into svn?
>
> thanks,
> dhruba
>
> On Wed, Jul 14, 2010 at 6:23 PM, Aaron Kimball <aaron@cloudera.com> wrote=
:
>
>> Eli,
>>
>> Great work. I like where this is going.
>>
>> Here's something that I think might be problematic:
>>
>> 3. Copyright/public domain -- Each HEP must either be explicitly
>> labelled as placed in the public domain (see this HEP as an example).
>>
>>
>> "must either be placed in the public domain, or..?" I assume the
>> alternative
>> is an open source copyright license? You should state what this is. PEP'=
s
>> alternative is "licensed under the Open Publication License." =A0(
>> http://www.opencontent.org/openpub/) Is that your intent?
>>
>> Upon acceptance of a PEP, the PEP workflow states that it should be chec=
ked
>> into svn. Are we planning to host accepted HEPs in the Hadoop svn? If so=
,
>> we
>> should specify that non-public-domain HEPs should be placed under a lice=
nse
>> listed on http://www.apache.org/legal/resolved.html. The Open Publicatio=
n
>> License isn't listed there. Nor are the GFDL and other documentation
>> licenses (presumably not because they're not specifically allowed, but
>> because Apache deals with code more than documentation). Note that
>> CC/Attribution/Share-Alike is allowed. You may want to recommend that on=
e?
>>
>> - Aaron
>>
>>
>> On Wed, Jul 14, 2010 at 3:00 PM, Eli Collins <eli@cloudera.com> wrote:
>>
>> > On Wed, Jul 14, 2010 at 11:57 AM, Doug Cutting <cutting@apache.org>
>> wrote:
>> >
>> > > On 07/14/2010 10:46 AM, Eli Collins wrote:
>> > >
>> > >> 5. How the set of editors is selected?
>> > >>> =A0 "The editors are apointed and removed by the PMC informally,
>> similar
>> > to
>> > >>> =A0 how the Apache Board appoints shepherds to projects."
>> > >>> This needs a reference. How does Apache Board appoints shepherds?
>> > >>>
>> > >>
>> > >> Good question, anyone know? Since it's informal I imagine shepherds
>> > >> volunteer. The editors could be a subset of the PMC that either
>> > >> volunteers or is rotated periodically.
>> > >>
>> > >
>> > > Shepherds are assigned randomly by the chair 48 hours before each
>> > meeting.
>> > > =A0Perhaps a better analogy for an editor is an Incubator Champion?
>> > >
>> > >
>> > >
>> >
>> http://incubator.apache.org/incubation/Roles_and_Responsibilities.html#C=
hampion
>> > >
>> > > Champions are volunteers. =A0So, rather than assigning an editor, a =
HEP
>> > might
>> > > ask for one to volunteer. =A0Any PMC member might volunteer.
>> > >
>> >
>> > That sounds reasonable. Having a PMC member volunteer will likely resu=
lt
>> in
>> > someone who has more expertise in the area and will perhaps be more
>> > responsive.
>> >
>> > Thanks,
>> > Eli
>> >
>>
>
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
>

From general-return-1816-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 08:28:32 2010
Return-Path: <general-return-1816-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17964 invoked from network); 15 Jul 2010 08:28:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 08:28:32 -0000
Received: (qmail 15738 invoked by uid 500); 15 Jul 2010 08:28:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15152 invoked by uid 500); 15 Jul 2010 08:28:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15143 invoked by uid 99); 15 Jul 2010 08:28:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 08:28:27 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_HELO_PASS,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [212.227.17.9] (HELO moutng.kundenserver.de) (212.227.17.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 08:28:20 +0000
Received: from [192.168.2.101] (p4FE235EF.dip.t-dialin.net [79.226.53.239])
	by mrelayeu.kundenserver.de (node=mrbap1) with ESMTP (Nemesis)
	id 0LbQ1i-1Ows1r2Bty-00kRQh; Thu, 15 Jul 2010 10:27:29 +0200
Subject: [ANNOUNCE] London Open Source Search meetup - Wed 28 July
From: =?ISO-8859-1?Q?Ren=E9?= Kriegler <rk@rene-kriegler.de>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Date: Thu, 15 Jul 2010 10:27:29 +0200
Message-Id: <1279182449.7495.11.camel@rene-laptop>
Mime-Version: 1.0
X-Mailer: Evolution 2.22.3.1 
Content-Transfer-Encoding: quoted-printable
X-Provags-ID: V02:K0:LkTEB2E+hk17N803dtqNLp4UDXhJ1wfSQv2VskjrcUT
 ZGOIm9obTorSsRmDJpnQND1MZLAz94YkNrmNOnBx1+6ydFoXY5
 wUdEMwzQyk7F7bOENh3aJatni6UNmTGqIXzR+PMeKz3gWAVdPz
 EnZUK2atDz/V7b7SqWZzNIeyRvCm4diIuw794YnxkWYggfgWfk
 wse5pYqJmpJ9LfW3SWQn2Ns9EV/uX8IegXD7yyGr8k=
X-Virus-Checked: Checked by ClamAV on apache.org

Hi all,

We are organising another open source search social evening in London on
Wednesday the 28 July.

As usual the plan is to get together and chat about search technology,
from Lucene to Solr, Hadoop, Mahout, Xapian and the like - bringing
together people from across the field to discuss ideas and ask questions
over a quiet drink.

For directions to this meetup and for the Meetup.com group see:

http://www.meetup.com/london-search-social/=20

Please come along if you can!


Ren=C3=A9 & Rich (Marr)



From general-return-1817-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 17:01:48 2010
Return-Path: <general-return-1817-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 40554 invoked from network); 15 Jul 2010 17:01:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 17:01:48 -0000
Received: (qmail 26670 invoked by uid 500); 15 Jul 2010 17:01:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26552 invoked by uid 500); 15 Jul 2010 17:01:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26544 invoked by uid 99); 15 Jul 2010 17:01:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 17:01:46 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 15 Jul 2010 17:01:43 +0000
Received: (qmail 40471 invoked by uid 99); 15 Jul 2010 17:01:22 -0000
Received: from localhost.apache.org (HELO [192.168.42.217]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 17:01:22 +0000
Message-ID: <4C3F3EE1.10208@apache.org>
Date: Thu, 15 Jul 2010 10:01:21 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.10) Gecko/20100527 Thunderbird/3.0.5
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: HEP proposal
References: <AANLkTilRca-SDdWa0rZtNjdbnnzk3m0AG9JGwUBSGIaK@mail.gmail.com>	<4C3CD27F.60003@yahoo-inc.com>	<AANLkTina2dxXd61b5zj4ZpS6-MK3jpzx47ecc3xZkP-r@mail.gmail.com>	<4C3E08B4.60205@apache.org>	<AANLkTiklz1m_splseCb03x9IC1dQD0wpMLORcu0Oe_bd@mail.gmail.com>	<AANLkTil_iFA_WiJVtg4EvbaPvM7a_WRiPuXy9z8OOHbT@mail.gmail.com> <AANLkTikSBkEsRg2d45NGmF_eaht_dBAjhydIYTYGsTTU@mail.gmail.com>
In-Reply-To: <AANLkTikSBkEsRg2d45NGmF_eaht_dBAjhydIYTYGsTTU@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 07/14/2010 11:41 PM, Eli Collins wrote:
> I was assuming HEPs would just be placed in the public domain, period.
> Like design docs posted on jira they're just text posted to a mailing
> list, not sure checking them into svn needs to require they have an
> alternate license. Something I'm missing?

If they were distributed with releases then they should be under the 
Apache license, but I don't see that HEPs will be distributed with 
releases, so public domain seems sufficient.

Doug

From general-return-1818-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 17:45:13 2010
Return-Path: <general-return-1818-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 64298 invoked from network); 15 Jul 2010 17:45:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 17:45:13 -0000
Received: (qmail 88419 invoked by uid 500); 15 Jul 2010 17:45:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88336 invoked by uid 500); 15 Jul 2010 17:45:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88328 invoked by uid 99); 15 Jul 2010 17:45:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 17:45:11 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jonathan.seidman@gmail.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 17:45:04 +0000
Received: by gyg10 with SMTP id 10so1177220gyg.35
        for <general@hadoop.apache.org>; Thu, 15 Jul 2010 10:43:43 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=BUPaO4uyrpDLbGHDkGPJZXUqiCgIAs1M17DtOKa2tbo=;
        b=b6LhI9SdOgTgkcPR1Mz5Yue5v+SsGdSp0oNZwtIHxEJjR0wTfWn33R0kHvHC/VRGtt
         95/2yDaVzTszkgoGgchSdFFrzel/u4coRf0yDMX10+vcSk+1Ec4HivyMB2wSYyo2iA5V
         Vlbg6kh9PDtRAUat2ptPllqHBcN9E2uuFa8rc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=oauj07SnUvjczj7gycJNUcsxw1fNhzAqCxD7JOhZJbnw1PJoVe3RIIKG/H3dXdZ5AH
         JGsBmbcmCC2vQyXppg1t/7Zg0AqBS8FiU1wtmgRlqLc/Br+Km8g+AF7sYFbI4ysOp9kO
         fkuhws+l8J+rsf7Z5J21k+8QwWlIUm/BOw35A=
MIME-Version: 1.0
Received: by 10.101.171.40 with SMTP id y40mr5366ano.7.1279215822714; Thu, 15 
	Jul 2010 10:43:42 -0700 (PDT)
Received: by 10.100.46.4 with HTTP; Thu, 15 Jul 2010 10:43:42 -0700 (PDT)
Date: Thu, 15 Jul 2010 12:43:42 -0500
Message-ID: <AANLkTimcA_D60ITpXlqs9Dxb_MkRa74IOiIPspzLrazi@mail.gmail.com>
Subject: [ANNOUNCE] Chicago area Hadoop User Group Meeting - July 21st
From: Jonathan Seidman <jonathan.seidman@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c923ccb43662048b70a49b
X-Virus-Checked: Checked by ClamAV on apache.org

--001636c923ccb43662048b70a49b
Content-Type: text/plain; charset=ISO-8859-1

The next meeting of the Chicago area Hadoop User Group will be help next
Wednesday, July 21st from 6:00-8:00PM. The meeting will be held at  Orbitz
Worldwide headquarters located in the Citigroup building at 500 West Madison
in downtown Chicago. We'll hear from Collin Bennett of Open Data Group who
will talk about work he's been doing to apply Hadoop to solve data analysis
problems.

Much thanks to our host Orbitz and our sponsor Cloudera.

More details and RSVP here: http://www.meetup.com/Chicago-area-Hadoop-User-
Group-CHUG/

Hope to see you there!

Jonathan

--001636c923ccb43662048b70a49b--

From general-return-1819-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 18:41:21 2010
Return-Path: <general-return-1819-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91189 invoked from network); 15 Jul 2010 18:41:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 18:41:21 -0000
Received: (qmail 61887 invoked by uid 500); 15 Jul 2010 18:41:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61825 invoked by uid 500); 15 Jul 2010 18:41:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61816 invoked by uid 99); 15 Jul 2010 18:41:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 18:41:19 +0000
X-ASF-Spam-Status: No, hits=4.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_BL_SPAMCOP_NET,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mdwasti@hotmail.com designates 65.54.190.22 as permitted sender)
Received: from [65.54.190.22] (HELO bay0-omc1-s11.bay0.hotmail.com) (65.54.190.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 18:41:09 +0000
Received: from BAY145-W60 ([65.54.190.61]) by bay0-omc1-s11.bay0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Thu, 15 Jul 2010 11:40:27 -0700
Message-ID: <BAY145-w60C3857131FF5F21F17FDDBEBB0@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_06b5619b-6c8f-4678-bee4-618ea52b7b1f_"
X-Originating-IP: [208.84.47.48]
From: Syed Wasti <mdwasti@hotmail.com>
To: <general@hadoop.apache.org>
Subject: Data Block Size ?
Date: Thu, 15 Jul 2010 11:40:27 -0700
Importance: Normal
MIME-Version: 1.0
X-OriginalArrivalTime: 15 Jul 2010 18:40:27.0164 (UTC) FILETIME=[317E65C0:01CB244D]
X-Virus-Checked: Checked by ClamAV on apache.org

--_06b5619b-6c8f-4678-bee4-618ea52b7b1f_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


Hi=2C

I am new to hadoop and looking for some answers to clear my basic concepts =
on Hadoop.



Will it matter what the data block size is ?=20

It is recommended to have a block size of 64 MB=2C but if we want to have t=
he data block size to 128 MB=2C should this effect the performance ?

Does the size of the map jobs created on each datanodes in anyway depend th=
e block size ?



Thanks for the support.



Regards

Syed


 		 	   		  =

--_06b5619b-6c8f-4678-bee4-618ea52b7b1f_--

From general-return-1820-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 18:49:35 2010
Return-Path: <general-return-1820-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93342 invoked from network); 15 Jul 2010 18:49:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 18:49:35 -0000
Received: (qmail 71648 invoked by uid 500); 15 Jul 2010 18:49:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71511 invoked by uid 500); 15 Jul 2010 18:49:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71503 invoked by uid 99); 15 Jul 2010 18:49:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 18:49:33 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 18:49:27 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version;
  b=W3haW7mDEoZnqSijZmFeWXpEZBLVgT1CYn2WSVjAaRXEiX6ugg+s+N1u
   v/+U7dV6FH4eWzgg13eUGqTHwbNz7XsSPZd48BS2lgxT4MnxQslqHpjlZ
   4UXk+v3tDv4xkZC;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1279219767; x=1310755767;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20Data=20Block=20Size=20?|Date:=20Thu,=20
   15=20Jul=202010=2018:49:04=20+0000|Message-ID:=20<6FB97C9
   2-99C8-42BC-A7C8-6724F0C3ADCD@linkedin.com>|To:=20"<gener
   al@hadoop.apache.org>"=20<general@hadoop.apache.org>
   |MIME-Version:=201.0|Content-Transfer-Encoding:=20quoted-
   printable|Content-ID:=20<bf3e868a-5bb7-49fe-bd4b-9e2b6ea4
   e262>|In-Reply-To:=20<BAY145-w60C3857131FF5F21F17FDDBEBB0
   @phx.gbl>|References:=20<BAY145-w60C3857131FF5F21F17FDDBE
   BB0@phx.gbl>;
  bh=Mu0/j/YPUbaUokxRJsXKe1EueMFE7Gb8yggDrFRnbLQ=;
  b=wBqRQMU1P6mrrgI4Yks5idd20SsJzB9bxzqYNiqv/5NmyQiMrdEETQ+T
   fBLvJLBi8onUU/WJZ3uIj8I/QbKs/pmxbJW+1/OcH4xlGdtNf76R9Vy3K
   nxWEUwBRf0m7pNQ;
X-IronPort-AV: E=Sophos;i="4.55,210,1278313200"; 
   d="scan'208";a="13645916"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi; Thu, 15 Jul 2010
 11:49:05 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: Data Block Size ?
Thread-Topic: Data Block Size ?
Thread-Index: AQHLJE1Ri3UPDvrVXk6fTiZOZu6/ZJKyyYkA
Date: Thu, 15 Jul 2010 18:49:04 +0000
Message-ID: <6FB97C92-99C8-42BC-A7C8-6724F0C3ADCD@linkedin.com>
References: <BAY145-w60C3857131FF5F21F17FDDBEBB0@phx.gbl>
In-Reply-To: <BAY145-w60C3857131FF5F21F17FDDBEBB0@phx.gbl>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Content-Type: text/plain; charset="us-ascii"
Content-ID: <bf3e868a-5bb7-49fe-bd4b-9e2b6ea4e262>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


On Jul 15, 2010, at 11:40 AM, Syed Wasti wrote:

> Will it matter what the data block size is ?=20

Yes.

> It is recommended to have a block size of 64 MB, but if we want to have t=
he data block size to 128 MB, should this effect the performance ?

Yes.

FWIW, we run with 128MB.

> Does the size of the map jobs created on each datanodes in anyway depend =
the block size ?

Yes.

Unless told otherwise, Hadoop will generally use the # of maps =3D=3D # of =
blocks.  So if you have fewer blocks to process, you'll have fewer maps to =
do more work.  This is not necessarily a bad thing; it all depends upon you=
r workload, size of grid, etc.


From general-return-1821-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 19:14:05 2010
Return-Path: <general-return-1821-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4620 invoked from network); 15 Jul 2010 19:14:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 19:14:05 -0000
Received: (qmail 514 invoked by uid 500); 15 Jul 2010 19:14:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 312 invoked by uid 500); 15 Jul 2010 19:14:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 304 invoked by uid 99); 15 Jul 2010 19:14:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 19:14:03 +0000
X-ASF-Spam-Status: No, hits=4.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_BL_SPAMCOP_NET,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mdwasti@hotmail.com designates 65.54.190.29 as permitted sender)
Received: from [65.54.190.29] (HELO bay0-omc1-s18.bay0.hotmail.com) (65.54.190.29)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 19:13:55 +0000
Received: from BAY145-W26 ([65.54.190.61]) by bay0-omc1-s18.bay0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Thu, 15 Jul 2010 12:13:34 -0700
Message-ID: <BAY145-w26CB9B9B0997B92BA89659BEBB0@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_205c9cf3-1b6f-44e2-ae6d-45613cebdf73_"
X-Originating-IP: [208.84.47.48]
From: Syed Wasti <mdwasti@hotmail.com>
To: Hadoop <general@hadoop.apache.org>
Subject: RE: Data Block Size ?
Date: Thu, 15 Jul 2010 12:13:34 -0700
Importance: Normal
In-Reply-To: <6FB97C92-99C8-42BC-A7C8-6724F0C3ADCD@linkedin.com>
References:
 <BAY145-w60C3857131FF5F21F17FDDBEBB0@phx.gbl>,<6FB97C92-99C8-42BC-A7C8-6724F0C3ADCD@linkedin.com>
MIME-Version: 1.0
X-OriginalArrivalTime: 15 Jul 2010 19:13:34.0209 (UTC) FILETIME=[D1DD6710:01CB2451]
X-Virus-Checked: Checked by ClamAV on apache.org

--_205c9cf3-1b6f-44e2-ae6d-45613cebdf73_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


Thank you Allen.
So=2C is it fair to assume that if I have smaller block size (64 MB)=2C the=
n my blocks are distributed across more datanodes and because my blocks are=
 around more datanodes=2C then my map jobs should also run on different dat=
anodes and becuase the maps size will be smaller=2C it should execute faste=
r using less resources.=20
Should this work this way ? or is there any algorithm on how the blocks sho=
uld be distributed across the datanodes and where should the replication co=
pies should go ?

Lets say=2C I have a file of 640 MB and a cluster with 5 datanodes and conf=
igured the block size to be 64 MB. How will this be distributed ?=20

Regards
Syed Wasti

> From: awittenauer@linkedin.com
> To: general@hadoop.apache.org
> Subject: Re: Data Block Size ?
> Date: Thu=2C 15 Jul 2010 18:49:04 +0000
>=20
>=20
> On Jul 15=2C 2010=2C at 11:40 AM=2C Syed Wasti wrote:
>=20
> > Will it matter what the data block size is ?=20
>=20
> Yes.
>=20
> > It is recommended to have a block size of 64 MB=2C but if we want to ha=
ve the data block size to 128 MB=2C should this effect the performance ?
>=20
> Yes.
>=20
> FWIW=2C we run with 128MB.
>=20
> > Does the size of the map jobs created on each datanodes in anyway depen=
d the block size ?
>=20
> Yes.
>=20
> Unless told otherwise=2C Hadoop will generally use the # of maps =3D=3D #=
 of blocks.  So if you have fewer blocks to process=2C you'll have fewer ma=
ps to do more work.  This is not necessarily a bad thing=3B it all depends =
upon your workload=2C size of grid=2C etc.
>=20
 		 	   		  =

--_205c9cf3-1b6f-44e2-ae6d-45613cebdf73_--

From general-return-1822-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 20:44:51 2010
Return-Path: <general-return-1822-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39311 invoked from network); 15 Jul 2010 20:44:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 20:44:51 -0000
Received: (qmail 2480 invoked by uid 500); 15 Jul 2010 20:44:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2361 invoked by uid 500); 15 Jul 2010 20:44:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2346 invoked by uid 99); 15 Jul 2010 20:44:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 20:44:47 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.213.48] (HELO mail-yw0-f48.google.com) (209.85.213.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 20:44:41 +0000
Received: by ywe9 with SMTP id 9so211703ywe.35
        for <general@hadoop.apache.org>; Thu, 15 Jul 2010 13:43:31 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.224.17.205 with SMTP id t13mr14367qaa.216.1279226611210; Thu, 
	15 Jul 2010 13:43:31 -0700 (PDT)
Received: by 10.229.40.19 with HTTP; Thu, 15 Jul 2010 13:43:31 -0700 (PDT)
Date: Thu, 15 Jul 2010 13:43:31 -0700
Message-ID: <AANLkTimnp8vQCDBg2aYDMGweUmAEoCAjylr4sd9PFTBq@mail.gmail.com>
Subject: Hadoop World 2010 - Call for Speakers
From: John Kreisa <john@cloudera.com>
To: general <general@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=00c09f93e0febf9e4f048b732723
X-Virus-Checked: Checked by ClamAV on apache.org

--00c09f93e0febf9e4f048b732723
Content-Type: text/plain; charset=ISO-8859-1

Hello Hadoop Fans,

We are currently seeking submissions for Hadoop World which returns to NYC
on October 12th. We want to build on last years success and have added more
tracks and more sessions so there are more speaking opportunities.

We have had a few requests to extend the submission deadline so we wanted to
let everyone know we are accepting proposals until August 2nd.

You'll find more details here:
http://www.cloudera.com/hadoop-world-nyc

We look forward to hearing from you,

John

-- 
John Kreisa
VP Marketing
Cloudera
phone: +1 (408) 425 7600
blog: cloudera.com/blog
twitter: twitter.com/cloudera
web: www.cloudera.com

--00c09f93e0febf9e4f048b732723--

From general-return-1823-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 18:07:11 2010
Return-Path: <general-return-1823-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 61189 invoked from network); 16 Jul 2010 18:07:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 18:07:11 -0000
Received: (qmail 74738 invoked by uid 500); 16 Jul 2010 18:07:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73917 invoked by uid 500); 16 Jul 2010 18:07:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73904 invoked by uid 99); 16 Jul 2010 18:07:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 18:07:07 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 18:07:01 +0000
Received: from [10.72.120.91] (tryarticle-lm.corp.yahoo.com [10.72.120.91])
	by mrout1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o6GI4uIY095556
	for <general@hadoop.apache.org>; Fri, 16 Jul 2010 11:04:56 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:content-type:
	content-transfer-encoding:mime-version:subject:date:x-mailer;
	b=a+MkKLeXWxBcmg1SAY2W9ITI/WMR1KLQUSI96mpW0WrVhvvsQwypbpoiYE6m95d1
Message-Id: <4F3AA93C-56A0-4465-B945-00B501321AC0@yahoo-inc.com>
From: "Owen O'Malley" <oom@yahoo-inc.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Chukwa moving out of Hadoop
Date: Fri, 16 Jul 2010 11:04:56 -0700
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

All,
    As part of Chukwa's migration from Hadoop to incubator, I've moved  
the subversion over to incubator. Over the next couple weeks, we'll  
move the web pages, etc. I just didn't want anyone who isn't tracking  
the chukwa-dev list to get surprised.

-- Owen

From general-return-1824-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 21:00:40 2010
Return-Path: <general-return-1824-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39256 invoked from network); 16 Jul 2010 21:00:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 21:00:40 -0000
Received: (qmail 5695 invoked by uid 500); 16 Jul 2010 21:00:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5475 invoked by uid 500); 16 Jul 2010 21:00:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5467 invoked by uid 99); 16 Jul 2010 21:00:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 21:00:38 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of Peter.Minearo@reardencommerce.com designates 64.41.141.18 as permitted sender)
Received: from [64.41.141.18] (HELO barracuda.mygazoo.com) (64.41.141.18)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 21:00:30 +0000
X-ASG-Debug-ID: 1279314001-659400290004-IjwG86
Received: from sccorpmail01.mygazoo.com (sccorpmail01.mygazoo.com [10.38.20.8]) by barracuda.mygazoo.com with ESMTP id FNOwtNi2uw0Qelw5 for <general@hadoop.apache.org>; Fri, 16 Jul 2010 14:00:02 -0700 (PDT)
X-Barracuda-Envelope-From: Peter.Minearo@Reardencommerce.com
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01CB2529.DB239CD8"
X-ASG-Orig-Subj: Hadoop and XML
Subject: Hadoop and XML
Date: Fri, 16 Jul 2010 14:00:00 -0700
Message-ID: <5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F46C@sccorpmail01.mygazoo.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Hadoop and XML
Thread-Index: AcslKdrY7z6P57aVRyC7BpzRMKis/g==
From: "Peter Minearo" <Peter.Minearo@Reardencommerce.com>
To: <general@hadoop.apache.org>
X-Barracuda-Connect: sccorpmail01.mygazoo.com[10.38.20.8]
X-Barracuda-Start-Time: 1279314002
X-Barracuda-URL: http://10.38.16.14:8000/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at mygazoo.com
X-Virus-Checked: Checked by ClamAV on apache.org

------_=_NextPart_001_01CB2529.DB239CD8
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

I have an XML file that has sparse data in it.  I am running a MapReduce
Job that reads in an XML file, pulls out a Key from within the XML
snippet and then hands back the Key and the XML snippet (as the Value)
to the OutputCollector.  The reason is to sort the file back into order.
Below is the snippet of code.=20
=20
public class XmlMapper extends MapReduceBase implements Mapper {
=20
 private Text keyText =3D new Text();
 private Text valueText =3D new Text();
=20
 @SuppressWarnings("unchecked")
 public void map(Object key, Object value, OutputCollector output,
Reporter reporter) throws IOException {
  Text valueText =3D (Text)value;
  String valueString =3D new String(valueText.getBytes(), "UTF-8");
  String keyString =3D getXmlKey(valueString);
  getKeyText().set(keyString);
  getValueText().set(valueString);
  output.collect(getKeyText(), getValueText());
 }
=20
=20
 public Text getKeyText() {
  return keyText;
 }
=20

 public void setKeyText(Text keyText) {
  this.keyText =3D keyText;
 }
=20

 public Text getValueText() {
  return valueText;
 }
=20

 public void setValueText(Text valueText) {
  this.valueText =3D valueText;
 }
=20

 private String getXmlKey(String value) {
        // Get the Key from the XML in the value.
 }
=20
}
=20
The XML snippet from the Value is fine when it is passed into the map()
method.  I am not changing any data either, just pulling out information
for the key.  The problem I am seeing is between the Map phase and the
Reduce phase, the XML is getting munged.  For Example:
=20
 </PrivateRate>
  </PrivateRateSet>te>
=20
It is my understanding that Hadoop uses the same instance of the Key and
Value object when calling the Map method.  What changes is the data
within those instances.  So, I ran an experiment where I do not have
different Key or Value Text Objects.  I reuse the ones passed into the
method, like below:
=20
public class XmlMapper extends MapReduceBase implements Mapper {
=20
 @SuppressWarnings("unchecked")
 public void map(Object key, Object value, OutputCollector output,
Reporter reporter) throws IOException {
  Text keyText =3D (Text)key;
  Text valueText =3D (Text)value;
  String valueString =3D new String(valueText.getBytes(), "UTF-8");
  String keyString =3D getXmlKey(valueString);
  keyText.set(keyString);
  valueText.set(valueString);
  output.collect(keyText, valueText);
 }
=20
=20
 private String getXmlKey(String value) {
        // Get the Key from the XML in the value.
 }
=20
}
=20
What was interesting about this is the fact that the XML was getting
munged within the Map Phase.  When I changed over to the code at the
top, the Map phase was fine.  However, the Reduce phase picks up the
munged XML.  Trying to debug the problem, I came across this method in
the Text Object:
=20
public void set(byte[] utf8, int start, int len) {
    setCapacity(len, false);
    System.arraycopy(utf8, start, bytes, 0, len);
    this.length =3D len;
}
=20
If the "bytes" array had a length of 1000 and the "utf8" array has a
length of 500; doing a System.arraycopy() would only copy the first 500
from "utf8" to "bytes" but leave the last 500 in "bytes" alone.  Could
this be the cause of the XML munging?
=20
All of this leads me to a few questions:
=20
1) Has anyone successfully used XML snippets as the data format within a
MapReduce job; not just reading from the file but used during the
shuffle?
2) Is anyone seeing this problem with XML or any other format?
3) Does anyone know what is going on?
4) Is this a bug?
=20

Thanks,
=20
Peter=20
=20
=20

------_=_NextPart_001_01CB2529.DB239CD8--

From general-return-1825-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 21:31:14 2010
Return-Path: <general-return-1825-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50382 invoked from network); 16 Jul 2010 21:31:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 21:31:13 -0000
Received: (qmail 39820 invoked by uid 500); 16 Jul 2010 21:31:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39753 invoked by uid 500); 16 Jul 2010 21:31:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39745 invoked by uid 99); 16 Jul 2010 21:31:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 21:31:12 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of soumya.sbanerjee@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 21:31:05 +0000
Received: by pwj2 with SMTP id 2so1824260pwj.35
        for <general@hadoop.apache.org>; Fri, 16 Jul 2010 14:29:43 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=kc5fvUIHLHqCE5/MScLL6vEN1/uBxojICmk1w9LUfoM=;
        b=xz78ZbJGHVLyS5HHphP/tuk2Ce1rB42ziMbFMpghizcLbBxtLEEE1tr0uvmpX+3Sr/
         lXCOqBoJ9OUMNr5g8OngNrQXuVwveY3ZNph/0bbpDgBLVizMKyelezWyYxDIPimES81g
         dZ7hvO5EIAmaqnUNdgQSlswHbrs0IFpRru1sQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=p+MehNS/MpS6pDgUnRqsTnkqsAHbwL9eQuMkFw9yZz6cbqKjoWhyFkr42mARw2Ds0c
         l6MpQ3YfuyrIr5Qf3yfGcRr4BJA2Lhy0tQzEwAPZek+PmQba3pSctoAg3aqgU3dDit7r
         pYcWP/DSDf4gv8V7wkPK85LZ/TdaH8MVryVmQ=
MIME-Version: 1.0
Received: by 10.142.147.7 with SMTP id u7mr2178296wfd.217.1279315783189; Fri, 
	16 Jul 2010 14:29:43 -0700 (PDT)
Received: by 10.142.187.10 with HTTP; Fri, 16 Jul 2010 14:29:43 -0700 (PDT)
In-Reply-To: <5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F46C@sccorpmail01.mygazoo.com>
References: <5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F46C@sccorpmail01.mygazoo.com>
Date: Sat, 17 Jul 2010 02:59:43 +0530
Message-ID: <AANLkTilvuqho0l3U9ETg5HHT4Yz5uUkyjX_xqBvmh505@mail.gmail.com>
Subject: Re: Hadoop and XML
From: Soumya Banerjee <soumya.sbanerjee@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd2281ed00b72048b87eac2
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd2281ed00b72048b87eac2
Content-Type: text/plain; charset=ISO-8859-1

Hi,

Can you please share the code of the job submission client ?

Also can you try creating the output key and values in the map method(method
lacal) ?
Make sure you are not using multi threaded map task configuration.

map()
{
private Text keyText = new Text();
 private Text valueText = new Text();

//rest of the code
}

Soumya.

On Sat, Jul 17, 2010 at 2:30 AM, Peter Minearo <
Peter.Minearo@reardencommerce.com> wrote:

> I have an XML file that has sparse data in it.  I am running a MapReduce
> Job that reads in an XML file, pulls out a Key from within the XML
> snippet and then hands back the Key and the XML snippet (as the Value)
> to the OutputCollector.  The reason is to sort the file back into order.
> Below is the snippet of code.
>
> public class XmlMapper extends MapReduceBase implements Mapper {
>
>  private Text keyText = new Text();
>  private Text valueText = new Text();
>
>  @SuppressWarnings("unchecked")
>  public void map(Object key, Object value, OutputCollector output,
> Reporter reporter) throws IOException {
>  Text valueText = (Text)value;
>  String valueString = new String(valueText.getBytes(), "UTF-8");
>  String keyString = getXmlKey(valueString);
>  getKeyText().set(keyString);
>  getValueText().set(valueString);
>  output.collect(getKeyText(), getValueText());
>  }
>
>
>  public Text getKeyText() {
>  return keyText;
>  }
>
>
>  public void setKeyText(Text keyText) {
>  this.keyText = keyText;
>  }
>
>
>  public Text getValueText() {
>  return valueText;
>  }
>
>
>  public void setValueText(Text valueText) {
>  this.valueText = valueText;
>  }
>
>
>  private String getXmlKey(String value) {
>        // Get the Key from the XML in the value.
>  }
>
> }
>
> The XML snippet from the Value is fine when it is passed into the map()
> method.  I am not changing any data either, just pulling out information
> for the key.  The problem I am seeing is between the Map phase and the
> Reduce phase, the XML is getting munged.  For Example:
>
>  </PrivateRate>
>  </PrivateRateSet>te>
>
> It is my understanding that Hadoop uses the same instance of the Key and
> Value object when calling the Map method.  What changes is the data
> within those instances.  So, I ran an experiment where I do not have
> different Key or Value Text Objects.  I reuse the ones passed into the
> method, like below:
>
> public class XmlMapper extends MapReduceBase implements Mapper {
>
>  @SuppressWarnings("unchecked")
>  public void map(Object key, Object value, OutputCollector output,
> Reporter reporter) throws IOException {
>  Text keyText = (Text)key;
>  Text valueText = (Text)value;
>  String valueString = new String(valueText.getBytes(), "UTF-8");
>  String keyString = getXmlKey(valueString);
>  keyText.set(keyString);
>  valueText.set(valueString);
>  output.collect(keyText, valueText);
>  }
>
>
>  private String getXmlKey(String value) {
>        // Get the Key from the XML in the value.
>  }
>
> }
>
> What was interesting about this is the fact that the XML was getting
> munged within the Map Phase.  When I changed over to the code at the
> top, the Map phase was fine.  However, the Reduce phase picks up the
> munged XML.  Trying to debug the problem, I came across this method in
> the Text Object:
>
> public void set(byte[] utf8, int start, int len) {
>    setCapacity(len, false);
>    System.arraycopy(utf8, start, bytes, 0, len);
>    this.length = len;
> }
>
> If the "bytes" array had a length of 1000 and the "utf8" array has a
> length of 500; doing a System.arraycopy() would only copy the first 500
> from "utf8" to "bytes" but leave the last 500 in "bytes" alone.  Could
> this be the cause of the XML munging?
>
> All of this leads me to a few questions:
>
> 1) Has anyone successfully used XML snippets as the data format within a
> MapReduce job; not just reading from the file but used during the
> shuffle?
> 2) Is anyone seeing this problem with XML or any other format?
> 3) Does anyone know what is going on?
> 4) Is this a bug?
>
>
> Thanks,
>
> Peter
>
>
>

--000e0cd2281ed00b72048b87eac2--

From general-return-1826-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 21:49:00 2010
Return-Path: <general-return-1826-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55675 invoked from network); 16 Jul 2010 21:48:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 21:48:59 -0000
Received: (qmail 58729 invoked by uid 500); 16 Jul 2010 21:48:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58644 invoked by uid 500); 16 Jul 2010 21:48:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58635 invoked by uid 99); 16 Jul 2010 21:48:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 21:48:58 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of Peter.Minearo@reardencommerce.com designates 64.41.141.19 as permitted sender)
Received: from [64.41.141.19] (HELO barracuda.mygazoo.com) (64.41.141.19)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 21:48:51 +0000
X-ASG-Debug-ID: 1279316905-77a055df0002-IjwG86
Received: from sccorpmail01.mygazoo.com (corpmail.mygazoo.com [10.38.20.8]) by barracuda.mygazoo.com with ESMTP id YJblYLP8wlki6DJB for <general@hadoop.apache.org>; Fri, 16 Jul 2010 14:48:27 -0700 (PDT)
X-Barracuda-Envelope-From: Peter.Minearo@Reardencommerce.com
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/mixed;
	boundary="----_=_NextPart_001_01CB2530.96664D62"
X-ASG-Orig-Subj: RE: Hadoop and XML
Subject: RE: Hadoop and XML
Date: Fri, 16 Jul 2010 14:44:18 -0700
Message-ID: <5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D051@sccorpmail01.mygazoo.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: <5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D051@sccorpmail01.mygazoo.com>
Thread-Topic: Hadoop and XML
Thread-Index: AcslLje2Q6DQ1fmUQ6eYG3y1yEV9+QAAdMi6
References: <5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F46C@sccorpmail01.mygazoo.com> <AANLkTilvuqho0l3U9ETg5HHT4Yz5uUkyjX_xqBvmh505@mail.gmail.com>
From: "Peter Minearo" <Peter.Minearo@Reardencommerce.com>
To: <general@hadoop.apache.org>
X-Barracuda-Connect: corpmail.mygazoo.com[10.38.20.8]
X-Barracuda-Start-Time: 1279316907
X-Barracuda-URL: http://10.38.16.13:8000/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at mygazoo.com
X-Virus-Checked: Checked by ClamAV on apache.org

------_=_NextPart_001_01CB2530.96664D62
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


I am not using multi-threaded Map tasks.  Also, if I understand your =
second question correctly:
"Also can you try creating the output key and values in the map =
method(method lacal) ?"
In the first code snippet I am doing exactly that.

Below is the class that runs the Job.

public class HadoopJobClient {

	private static final Log LOGGER =3D =
LogFactory.getLog(Prds.class.getName());
=09
	public static void main(String[] args) {
		JobConf conf =3D new JobConf(Prds.class);
	=09
		conf.set("xmlinput.start", "<PrivateRateSet>");
		conf.set("xmlinput.end", "</PrivateRateSet>");
	=09
		conf.setJobName("PRDS Parse");

		conf.setOutputKeyClass(Text.class);
		conf.setOutputValueClass(Text.class);

		conf.setMapperClass(PrdsMapper.class);
		conf.setReducerClass(PrdsReducer.class);

		conf.setInputFormat(XmlInputFormat.class);
		conf.setOutputFormat(TextOutputFormat.class);

		FileInputFormat.setInputPaths(conf, new Path(args[0]));
		FileOutputFormat.setOutputPath(conf, new Path(args[1]));

		// Run the job
		try {
			JobClient.runJob(conf);
		} catch (IOException e) {
			LOGGER.error(e.getMessage(), e);
		}

	}
=09
=09
}




-----Original Message-----
From: Soumya Banerjee [mailto:soumya.sbanerjee@gmail.com]
Sent: Fri 7/16/2010 2:29 PM
To: general@hadoop.apache.org
Subject: Re: Hadoop and XML
=20
Hi,

Can you please share the code of the job submission client ?

Also can you try creating the output key and values in the map =
method(method
lacal) ?
Make sure you are not using multi threaded map task configuration.

map()
{
private Text keyText =3D new Text();
 private Text valueText =3D new Text();

//rest of the code
}

Soumya.

On Sat, Jul 17, 2010 at 2:30 AM, Peter Minearo <
Peter.Minearo@reardencommerce.com> wrote:

> I have an XML file that has sparse data in it.  I am running a =
MapReduce
> Job that reads in an XML file, pulls out a Key from within the XML
> snippet and then hands back the Key and the XML snippet (as the Value)
> to the OutputCollector.  The reason is to sort the file back into =
order.
> Below is the snippet of code.
>
> public class XmlMapper extends MapReduceBase implements Mapper {
>
>  private Text keyText =3D new Text();
>  private Text valueText =3D new Text();
>
>  @SuppressWarnings("unchecked")
>  public void map(Object key, Object value, OutputCollector output,
> Reporter reporter) throws IOException {
>  Text valueText =3D (Text)value;
>  String valueString =3D new String(valueText.getBytes(), "UTF-8");
>  String keyString =3D getXmlKey(valueString);
>  getKeyText().set(keyString);
>  getValueText().set(valueString);
>  output.collect(getKeyText(), getValueText());
>  }
>
>
>  public Text getKeyText() {
>  return keyText;
>  }
>
>
>  public void setKeyText(Text keyText) {
>  this.keyText =3D keyText;
>  }
>
>
>  public Text getValueText() {
>  return valueText;
>  }
>
>
>  public void setValueText(Text valueText) {
>  this.valueText =3D valueText;
>  }
>
>
>  private String getXmlKey(String value) {
>        // Get the Key from the XML in the value.
>  }
>
> }
>
> The XML snippet from the Value is fine when it is passed into the =
map()
> method.  I am not changing any data either, just pulling out =
information
> for the key.  The problem I am seeing is between the Map phase and the
> Reduce phase, the XML is getting munged.  For Example:
>
>  </PrivateRate>
>  </PrivateRateSet>te>
>
> It is my understanding that Hadoop uses the same instance of the Key =
and
> Value object when calling the Map method.  What changes is the data
> within those instances.  So, I ran an experiment where I do not have
> different Key or Value Text Objects.  I reuse the ones passed into the
> method, like below:
>
> public class XmlMapper extends MapReduceBase implements Mapper {
>
>  @SuppressWarnings("unchecked")
>  public void map(Object key, Object value, OutputCollector output,
> Reporter reporter) throws IOException {
>  Text keyText =3D (Text)key;
>  Text valueText =3D (Text)value;
>  String valueString =3D new String(valueText.getBytes(), "UTF-8");
>  String keyString =3D getXmlKey(valueString);
>  keyText.set(keyString);
>  valueText.set(valueString);
>  output.collect(keyText, valueText);
>  }
>
>
>  private String getXmlKey(String value) {
>        // Get the Key from the XML in the value.
>  }
>
> }
>
> What was interesting about this is the fact that the XML was getting
> munged within the Map Phase.  When I changed over to the code at the
> top, the Map phase was fine.  However, the Reduce phase picks up the
> munged XML.  Trying to debug the problem, I came across this method in
> the Text Object:
>
> public void set(byte[] utf8, int start, int len) {
>    setCapacity(len, false);
>    System.arraycopy(utf8, start, bytes, 0, len);
>    this.length =3D len;
> }
>
> If the "bytes" array had a length of 1000 and the "utf8" array has a
> length of 500; doing a System.arraycopy() would only copy the first =
500
> from "utf8" to "bytes" but leave the last 500 in "bytes" alone.  Could
> this be the cause of the XML munging?
>
> All of this leads me to a few questions:
>
> 1) Has anyone successfully used XML snippets as the data format within =
a
> MapReduce job; not just reading from the file but used during the
> shuffle?
> 2) Is anyone seeing this problem with XML or any other format?
> 3) Does anyone know what is going on?
> 4) Is this a bug?
>
>
> Thanks,
>
> Peter
>
>
>


------_=_NextPart_001_01CB2530.96664D62--

From general-return-1827-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 21:52:59 2010
Return-Path: <general-return-1827-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56480 invoked from network); 16 Jul 2010 21:52:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 21:52:58 -0000
Received: (qmail 64059 invoked by uid 500); 16 Jul 2010 21:52:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63998 invoked by uid 500); 16 Jul 2010 21:52:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63990 invoked by uid 99); 16 Jul 2010 21:52:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 21:52:57 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of Peter.Minearo@reardencommerce.com designates 64.41.141.19 as permitted sender)
Received: from [64.41.141.19] (HELO barracuda.mygazoo.com) (64.41.141.19)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 21:52:50 +0000
X-ASG-Debug-ID: 1279317149-77a056810001-IjwG86
Received: from sccorpmail01.mygazoo.com (corpmail.mygazoo.com [10.38.20.8]) by barracuda.mygazoo.com with ESMTP id a0egHRybD8HoBPgt for <general@hadoop.apache.org>; Fri, 16 Jul 2010 14:52:29 -0700 (PDT)
X-Barracuda-Envelope-From: Peter.Minearo@Reardencommerce.com
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
X-ASG-Orig-Subj: RE: Hadoop and XML
Subject: RE: Hadoop and XML
Date: Fri, 16 Jul 2010 14:51:39 -0700
Message-ID: <5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F4BA@sccorpmail01.mygazoo.com>
In-Reply-To: <5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D051@sccorpmail01.mygazoo.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Hadoop and XML
Thread-Index: AcslLje2Q6DQ1fmUQ6eYG3y1yEV9+QAAdMi6AAAyKqA=
References: <5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F46C@sccorpmail01.mygazoo.com> <AANLkTilvuqho0l3U9ETg5HHT4Yz5uUkyjX_xqBvmh505@mail.gmail.com> <5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D051@sccorpmail01.mygazoo.com>
From: "Peter Minearo" <Peter.Minearo@Reardencommerce.com>
To: <general@hadoop.apache.org>
X-Barracuda-Connect: corpmail.mygazoo.com[10.38.20.8]
X-Barracuda-Start-Time: 1279317149
X-Barracuda-URL: http://10.38.16.13:8000/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at mygazoo.com
X-Virus-Checked: Checked by ClamAV on apache.org

Whoops....right after I sent it and someone else made a suggestion; I
realized what question 2 was about.  I can try that, but wouldn't that
cause Object bloat?  During the Hadoop training I went through; it was
mentioned to reuse the returning Key and Value objects to keep the
number of Objects created down to a minimum.  Is this not really a valid
point?

=20

-----Original Message-----
From: Peter Minearo [mailto:Peter.Minearo@Reardencommerce.com]=20
Sent: Friday, July 16, 2010 2:44 PM
To: general@hadoop.apache.org
Subject: RE: Hadoop and XML


I am not using multi-threaded Map tasks.  Also, if I understand your
second question correctly:
"Also can you try creating the output key and values in the map
method(method lacal) ?"
In the first code snippet I am doing exactly that.

Below is the class that runs the Job.

public class HadoopJobClient {

	private static final Log LOGGER =3D
LogFactory.getLog(Prds.class.getName());
=09
	public static void main(String[] args) {
		JobConf conf =3D new JobConf(Prds.class);
	=09
		conf.set("xmlinput.start", "<PrivateRateSet>");
		conf.set("xmlinput.end", "</PrivateRateSet>");
	=09
		conf.setJobName("PRDS Parse");

		conf.setOutputKeyClass(Text.class);
		conf.setOutputValueClass(Text.class);

		conf.setMapperClass(PrdsMapper.class);
		conf.setReducerClass(PrdsReducer.class);

		conf.setInputFormat(XmlInputFormat.class);
		conf.setOutputFormat(TextOutputFormat.class);

		FileInputFormat.setInputPaths(conf, new Path(args[0]));
		FileOutputFormat.setOutputPath(conf, new Path(args[1]));

		// Run the job
		try {
			JobClient.runJob(conf);
		} catch (IOException e) {
			LOGGER.error(e.getMessage(), e);
		}

	}
=09
=09
}




-----Original Message-----
From: Soumya Banerjee [mailto:soumya.sbanerjee@gmail.com]
Sent: Fri 7/16/2010 2:29 PM
To: general@hadoop.apache.org
Subject: Re: Hadoop and XML
=20
Hi,

Can you please share the code of the job submission client ?

Also can you try creating the output key and values in the map
method(method
lacal) ?
Make sure you are not using multi threaded map task configuration.

map()
{
private Text keyText =3D new Text();
 private Text valueText =3D new Text();

//rest of the code
}

Soumya.

On Sat, Jul 17, 2010 at 2:30 AM, Peter Minearo <
Peter.Minearo@reardencommerce.com> wrote:

> I have an XML file that has sparse data in it.  I am running a=20
> MapReduce Job that reads in an XML file, pulls out a Key from within=20
> the XML snippet and then hands back the Key and the XML snippet (as=20
> the Value) to the OutputCollector.  The reason is to sort the file
back into order.
> Below is the snippet of code.
>
> public class XmlMapper extends MapReduceBase implements Mapper {
>
>  private Text keyText =3D new Text();
>  private Text valueText =3D new Text();
>
>  @SuppressWarnings("unchecked")
>  public void map(Object key, Object value, OutputCollector output,=20
> Reporter reporter) throws IOException {  Text valueText =3D =
(Text)value;

> String valueString =3D new String(valueText.getBytes(), "UTF-8"); =20
> String keyString =3D getXmlKey(valueString); =20
> getKeyText().set(keyString);  getValueText().set(valueString); =20
> output.collect(getKeyText(), getValueText());  }
>
>
>  public Text getKeyText() {
>  return keyText;
>  }
>
>
>  public void setKeyText(Text keyText) {  this.keyText =3D keyText;  }
>
>
>  public Text getValueText() {
>  return valueText;
>  }
>
>
>  public void setValueText(Text valueText) {  this.valueText =3D=20
> valueText;  }
>
>
>  private String getXmlKey(String value) {
>        // Get the Key from the XML in the value.
>  }
>
> }
>
> The XML snippet from the Value is fine when it is passed into the=20
> map() method.  I am not changing any data either, just pulling out=20
> information for the key.  The problem I am seeing is between the Map=20
> phase and the Reduce phase, the XML is getting munged.  For Example:
>
>  </PrivateRate>
>  </PrivateRateSet>te>
>
> It is my understanding that Hadoop uses the same instance of the Key=20
> and Value object when calling the Map method.  What changes is the=20
> data within those instances.  So, I ran an experiment where I do not=20
> have different Key or Value Text Objects.  I reuse the ones passed=20
> into the method, like below:
>
> public class XmlMapper extends MapReduceBase implements Mapper {
>
>  @SuppressWarnings("unchecked")
>  public void map(Object key, Object value, OutputCollector output,=20
> Reporter reporter) throws IOException {  Text keyText =3D (Text)key; =20
> Text valueText =3D (Text)value;  String valueString =3D new=20
> String(valueText.getBytes(), "UTF-8");  String keyString =3D=20
> getXmlKey(valueString);  keyText.set(keyString); =20
> valueText.set(valueString);  output.collect(keyText, valueText);  }
>
>
>  private String getXmlKey(String value) {
>        // Get the Key from the XML in the value.
>  }
>
> }
>
> What was interesting about this is the fact that the XML was getting=20
> munged within the Map Phase.  When I changed over to the code at the=20
> top, the Map phase was fine.  However, the Reduce phase picks up the=20
> munged XML.  Trying to debug the problem, I came across this method in

> the Text Object:
>
> public void set(byte[] utf8, int start, int len) {
>    setCapacity(len, false);
>    System.arraycopy(utf8, start, bytes, 0, len);
>    this.length =3D len;
> }
>
> If the "bytes" array had a length of 1000 and the "utf8" array has a=20
> length of 500; doing a System.arraycopy() would only copy the first=20
> 500 from "utf8" to "bytes" but leave the last 500 in "bytes" alone. =20
> Could this be the cause of the XML munging?
>
> All of this leads me to a few questions:
>
> 1) Has anyone successfully used XML snippets as the data format within

> a MapReduce job; not just reading from the file but used during the=20
> shuffle?
> 2) Is anyone seeing this problem with XML or any other format?
> 3) Does anyone know what is going on?
> 4) Is this a bug?
>
>
> Thanks,
>
> Peter
>
>
>


From general-return-1828-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 22:10:25 2010
Return-Path: <general-return-1828-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 64279 invoked from network); 16 Jul 2010 22:10:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 22:10:24 -0000
Received: (qmail 76122 invoked by uid 500); 16 Jul 2010 22:10:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76070 invoked by uid 500); 16 Jul 2010 22:10:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76062 invoked by uid 99); 16 Jul 2010 22:10:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 22:10:23 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of Peter.Minearo@reardencommerce.com designates 64.41.141.18 as permitted sender)
Received: from [64.41.141.18] (HELO barracuda.mygazoo.com) (64.41.141.18)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 22:10:16 +0000
X-ASG-Debug-ID: 1279318178-4d7700290001-IjwG86
Received: from sccorpmail01.mygazoo.com ([10.38.20.8]) by barracuda.mygazoo.com with ESMTP id oYkoEyk8nHoW6jS1 for <general@hadoop.apache.org>; Fri, 16 Jul 2010 15:09:38 -0700 (PDT)
X-Barracuda-Envelope-From: Peter.Minearo@Reardencommerce.com
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/mixed;
	boundary="----_=_NextPart_001_01CB2533.8B8049BA"
X-ASG-Orig-Subj: RE: Hadoop and XML
Subject: RE: Hadoop and XML
Date: Fri, 16 Jul 2010 15:07:56 -0700
Message-ID: <5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D053@sccorpmail01.mygazoo.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: <5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D053@sccorpmail01.mygazoo.com>
Thread-Topic: Hadoop and XML
Thread-Index: AcslLje2Q6DQ1fmUQ6eYG3y1yEV9+QAAdMi6AAAyKqAAAKE0kA==
References: <5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F46C@sccorpmail01.mygazoo.com> <AANLkTilvuqho0l3U9ETg5HHT4Yz5uUkyjX_xqBvmh505@mail.gmail.com> <5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D051@sccorpmail01.mygazoo.com> <5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F4BA@sccorpmail01.mygazoo.com>
From: "Peter Minearo" <Peter.Minearo@Reardencommerce.com>
To: <general@hadoop.apache.org>
X-Barracuda-Connect: UNKNOWN[10.38.20.8]
X-Barracuda-Start-Time: 1279318178
X-Barracuda-URL: http://10.38.16.14:8000/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at mygazoo.com
X-Virus-Checked: Checked by ClamAV on apache.org

------_=_NextPart_001_01CB2533.8B8049BA
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Moving the variable to a local variable did not seem to work:


</PrivateRateSet>vateRateSet>



public void map(Object key, Object value, OutputCollector output, =
Reporter reporter) throws IOException {
		Text valueText =3D (Text)value;
		String valueString =3D new String(valueText.getBytes(), "UTF-8");
		String keyString =3D getXmlKey(valueString);
		Text returnKeyText =3D new Text();
		Text returnValueText =3D new Text();
		returnKeyText.set(keyString);
		returnValueText.set(valueString);
		output.collect(returnKeyText, returnValueText);
}

-----Original Message-----
From: Peter Minearo [mailto:Peter.Minearo@Reardencommerce.com]
Sent: Fri 7/16/2010 2:51 PM
To: general@hadoop.apache.org
Subject: RE: Hadoop and XML
=20
Whoops....right after I sent it and someone else made a suggestion; I
realized what question 2 was about.  I can try that, but wouldn't that
cause Object bloat?  During the Hadoop training I went through; it was
mentioned to reuse the returning Key and Value objects to keep the
number of Objects created down to a minimum.  Is this not really a valid
point?

=20

-----Original Message-----
From: Peter Minearo [mailto:Peter.Minearo@Reardencommerce.com]=20
Sent: Friday, July 16, 2010 2:44 PM
To: general@hadoop.apache.org
Subject: RE: Hadoop and XML


I am not using multi-threaded Map tasks.  Also, if I understand your
second question correctly:
"Also can you try creating the output key and values in the map
method(method lacal) ?"
In the first code snippet I am doing exactly that.

Below is the class that runs the Job.

public class HadoopJobClient {

	private static final Log LOGGER =3D
LogFactory.getLog(Prds.class.getName());
=09
	public static void main(String[] args) {
		JobConf conf =3D new JobConf(Prds.class);
	=09
		conf.set("xmlinput.start", "<PrivateRateSet>");
		conf.set("xmlinput.end", "</PrivateRateSet>");
	=09
		conf.setJobName("PRDS Parse");

		conf.setOutputKeyClass(Text.class);
		conf.setOutputValueClass(Text.class);

		conf.setMapperClass(PrdsMapper.class);
		conf.setReducerClass(PrdsReducer.class);

		conf.setInputFormat(XmlInputFormat.class);
		conf.setOutputFormat(TextOutputFormat.class);

		FileInputFormat.setInputPaths(conf, new Path(args[0]));
		FileOutputFormat.setOutputPath(conf, new Path(args[1]));

		// Run the job
		try {
			JobClient.runJob(conf);
		} catch (IOException e) {
			LOGGER.error(e.getMessage(), e);
		}

	}
=09
=09
}




-----Original Message-----
From: Soumya Banerjee [mailto:soumya.sbanerjee@gmail.com]
Sent: Fri 7/16/2010 2:29 PM
To: general@hadoop.apache.org
Subject: Re: Hadoop and XML
=20
Hi,

Can you please share the code of the job submission client ?

Also can you try creating the output key and values in the map
method(method
lacal) ?
Make sure you are not using multi threaded map task configuration.

map()
{
private Text keyText =3D new Text();
 private Text valueText =3D new Text();

//rest of the code
}

Soumya.

On Sat, Jul 17, 2010 at 2:30 AM, Peter Minearo <
Peter.Minearo@reardencommerce.com> wrote:

> I have an XML file that has sparse data in it.  I am running a=20
> MapReduce Job that reads in an XML file, pulls out a Key from within=20
> the XML snippet and then hands back the Key and the XML snippet (as=20
> the Value) to the OutputCollector.  The reason is to sort the file
back into order.
> Below is the snippet of code.
>
> public class XmlMapper extends MapReduceBase implements Mapper {
>
>  private Text keyText =3D new Text();
>  private Text valueText =3D new Text();
>
>  @SuppressWarnings("unchecked")
>  public void map(Object key, Object value, OutputCollector output,=20
> Reporter reporter) throws IOException {  Text valueText =3D =
(Text)value;

> String valueString =3D new String(valueText.getBytes(), "UTF-8"); =20
> String keyString =3D getXmlKey(valueString); =20
> getKeyText().set(keyString);  getValueText().set(valueString); =20
> output.collect(getKeyText(), getValueText());  }
>
>
>  public Text getKeyText() {
>  return keyText;
>  }
>
>
>  public void setKeyText(Text keyText) {  this.keyText =3D keyText;  }
>
>
>  public Text getValueText() {
>  return valueText;
>  }
>
>
>  public void setValueText(Text valueText) {  this.valueText =3D=20
> valueText;  }
>
>
>  private String getXmlKey(String value) {
>        // Get the Key from the XML in the value.
>  }
>
> }
>
> The XML snippet from the Value is fine when it is passed into the=20
> map() method.  I am not changing any data either, just pulling out=20
> information for the key.  The problem I am seeing is between the Map=20
> phase and the Reduce phase, the XML is getting munged.  For Example:
>
>  </PrivateRate>
>  </PrivateRateSet>te>
>
> It is my understanding that Hadoop uses the same instance of the Key=20
> and Value object when calling the Map method.  What changes is the=20
> data within those instances.  So, I ran an experiment where I do not=20
> have different Key or Value Text Objects.  I reuse the ones passed=20
> into the method, like below:
>
> public class XmlMapper extends MapReduceBase implements Mapper {
>
>  @SuppressWarnings("unchecked")
>  public void map(Object key, Object value, OutputCollector output,=20
> Reporter reporter) throws IOException {  Text keyText =3D (Text)key; =20
> Text valueText =3D (Text)value;  String valueString =3D new=20
> String(valueText.getBytes(), "UTF-8");  String keyString =3D=20
> getXmlKey(valueString);  keyText.set(keyString); =20
> valueText.set(valueString);  output.collect(keyText, valueText);  }
>
>
>  private String getXmlKey(String value) {
>        // Get the Key from the XML in the value.
>  }
>
> }
>
> What was interesting about this is the fact that the XML was getting=20
> munged within the Map Phase.  When I changed over to the code at the=20
> top, the Map phase was fine.  However, the Reduce phase picks up the=20
> munged XML.  Trying to debug the problem, I came across this method in

> the Text Object:
>
> public void set(byte[] utf8, int start, int len) {
>    setCapacity(len, false);
>    System.arraycopy(utf8, start, bytes, 0, len);
>    this.length =3D len;
> }
>
> If the "bytes" array had a length of 1000 and the "utf8" array has a=20
> length of 500; doing a System.arraycopy() would only copy the first=20
> 500 from "utf8" to "bytes" but leave the last 500 in "bytes" alone. =20
> Could this be the cause of the XML munging?
>
> All of this leads me to a few questions:
>
> 1) Has anyone successfully used XML snippets as the data format within

> a MapReduce job; not just reading from the file but used during the=20
> shuffle?
> 2) Is anyone seeing this problem with XML or any other format?
> 3) Does anyone know what is going on?
> 4) Is this a bug?
>
>
> Thanks,
>
> Peter
>
>
>



------_=_NextPart_001_01CB2533.8B8049BA--

From general-return-1829-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 17 04:44:29 2010
Return-Path: <general-return-1829-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53296 invoked from network); 17 Jul 2010 04:44:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jul 2010 04:44:28 -0000
Received: (qmail 45120 invoked by uid 500); 17 Jul 2010 04:44:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44705 invoked by uid 500); 17 Jul 2010 04:44:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44697 invoked by uid 99); 17 Jul 2010 04:44:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Jul 2010 04:44:24 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Jul 2010 04:44:17 +0000
Received: by qwd7 with SMTP id 7so1341398qwd.35
        for <general@hadoop.apache.org>; Fri, 16 Jul 2010 21:43:56 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=9uylSC+BenIg0k0i3HYEbDmdOLamq3y0kGKLHTRnraE=;
        b=p6dKPorP+LsnF/ApbYAjaFlS0wTva1BmoW+gWxd2wydWOgShebLJwUGsNEbfPFpIQQ
         dixQLeZOZ1Y6rvk/p6Bl2e8obn6/ygPbRuMYPMSVYd6/TZ8HcMVPP27JLNkkVwqnJxHb
         BO1vVGqkDkV1dtPE73/tUDdDZUV06BkPXITLc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=ecpfYzQeePkNyHHy0TKtHg5YEXSBB/hUN02mslA+LuP7AS/nY/DNM4jYv8pw0JD9Ia
         8PkJOJoUGkSmxN18NCHNHh1T34ZWULMeBBYudzkFp13QtXI0hctfP+cpqJ43tTQLya5M
         WGtGn48JDBnglxHhMliOvXCtiBc33SqKWr6eg=
MIME-Version: 1.0
Received: by 10.224.65.80 with SMTP id h16mr1680068qai.101.1279341834707; Fri, 
	16 Jul 2010 21:43:54 -0700 (PDT)
Received: by 10.229.75.85 with HTTP; Fri, 16 Jul 2010 21:43:54 -0700 (PDT)
In-Reply-To: <5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D053@sccorpmail01.mygazoo.com>
References: <5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F46C@sccorpmail01.mygazoo.com>
	<AANLkTilvuqho0l3U9ETg5HHT4Yz5uUkyjX_xqBvmh505@mail.gmail.com>
	<5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D051@sccorpmail01.mygazoo.com>
	<5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F4BA@sccorpmail01.mygazoo.com>
	<5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D053@sccorpmail01.mygazoo.com>
Date: Fri, 16 Jul 2010 21:43:54 -0700
Message-ID: <AANLkTilAzw5J1oL8raqTmPtxi9BGpuEwHXimEaUdfOzL@mail.gmail.com>
Subject: Re: Hadoop and XML
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00c09f8e5b8f9aaa3b048b8dfb67
X-Virus-Checked: Checked by ClamAV on apache.org

--00c09f8e5b8f9aaa3b048b8dfb67
Content-Type: text/plain; charset=ISO-8859-1

>From an earlier post:
http://oobaloo.co.uk/articles/2010/1/20/processing-xml-in-hadoop.html

On Fri, Jul 16, 2010 at 3:07 PM, Peter Minearo <
Peter.Minearo@reardencommerce.com> wrote:

> Moving the variable to a local variable did not seem to work:
>
>
> </PrivateRateSet>vateRateSet>
>
>
>
> public void map(Object key, Object value, OutputCollector output, Reporter
> reporter) throws IOException {
>                Text valueText = (Text)value;
>                String valueString = new String(valueText.getBytes(),
> "UTF-8");
>                String keyString = getXmlKey(valueString);
>                 Text returnKeyText = new Text();
>                Text returnValueText = new Text();
>                returnKeyText.set(keyString);
>                returnValueText.set(valueString);
>                output.collect(returnKeyText, returnValueText);
> }
>
> -----Original Message-----
> From: Peter Minearo [mailto:Peter.Minearo@Reardencommerce.com]
> Sent: Fri 7/16/2010 2:51 PM
> To: general@hadoop.apache.org
> Subject: RE: Hadoop and XML
>
> Whoops....right after I sent it and someone else made a suggestion; I
> realized what question 2 was about.  I can try that, but wouldn't that
> cause Object bloat?  During the Hadoop training I went through; it was
> mentioned to reuse the returning Key and Value objects to keep the
> number of Objects created down to a minimum.  Is this not really a valid
> point?
>
>
>
> -----Original Message-----
> From: Peter Minearo [mailto:Peter.Minearo@Reardencommerce.com]
> Sent: Friday, July 16, 2010 2:44 PM
> To: general@hadoop.apache.org
> Subject: RE: Hadoop and XML
>
>
> I am not using multi-threaded Map tasks.  Also, if I understand your
> second question correctly:
> "Also can you try creating the output key and values in the map
> method(method lacal) ?"
> In the first code snippet I am doing exactly that.
>
> Below is the class that runs the Job.
>
> public class HadoopJobClient {
>
>        private static final Log LOGGER =
> LogFactory.getLog(Prds.class.getName());
>
>        public static void main(String[] args) {
>                JobConf conf = new JobConf(Prds.class);
>
>                conf.set("xmlinput.start", "<PrivateRateSet>");
>                conf.set("xmlinput.end", "</PrivateRateSet>");
>
>                conf.setJobName("PRDS Parse");
>
>                conf.setOutputKeyClass(Text.class);
>                conf.setOutputValueClass(Text.class);
>
>                conf.setMapperClass(PrdsMapper.class);
>                conf.setReducerClass(PrdsReducer.class);
>
>                conf.setInputFormat(XmlInputFormat.class);
>                conf.setOutputFormat(TextOutputFormat.class);
>
>                FileInputFormat.setInputPaths(conf, new Path(args[0]));
>                FileOutputFormat.setOutputPath(conf, new Path(args[1]));
>
>                // Run the job
>                try {
>                        JobClient.runJob(conf);
>                } catch (IOException e) {
>                        LOGGER.error(e.getMessage(), e);
>                }
>
>        }
>
>
> }
>
>
>
>
> -----Original Message-----
> From: Soumya Banerjee [mailto:soumya.sbanerjee@gmail.com]
> Sent: Fri 7/16/2010 2:29 PM
> To: general@hadoop.apache.org
> Subject: Re: Hadoop and XML
>
> Hi,
>
> Can you please share the code of the job submission client ?
>
> Also can you try creating the output key and values in the map
> method(method
> lacal) ?
> Make sure you are not using multi threaded map task configuration.
>
> map()
> {
> private Text keyText = new Text();
>  private Text valueText = new Text();
>
> //rest of the code
> }
>
> Soumya.
>
> On Sat, Jul 17, 2010 at 2:30 AM, Peter Minearo <
> Peter.Minearo@reardencommerce.com> wrote:
>
> > I have an XML file that has sparse data in it.  I am running a
> > MapReduce Job that reads in an XML file, pulls out a Key from within
> > the XML snippet and then hands back the Key and the XML snippet (as
> > the Value) to the OutputCollector.  The reason is to sort the file
> back into order.
> > Below is the snippet of code.
> >
> > public class XmlMapper extends MapReduceBase implements Mapper {
> >
> >  private Text keyText = new Text();
> >  private Text valueText = new Text();
> >
> >  @SuppressWarnings("unchecked")
> >  public void map(Object key, Object value, OutputCollector output,
> > Reporter reporter) throws IOException {  Text valueText = (Text)value;
>
> > String valueString = new String(valueText.getBytes(), "UTF-8");
> > String keyString = getXmlKey(valueString);
> > getKeyText().set(keyString);  getValueText().set(valueString);
> > output.collect(getKeyText(), getValueText());  }
> >
> >
> >  public Text getKeyText() {
> >  return keyText;
> >  }
> >
> >
> >  public void setKeyText(Text keyText) {  this.keyText = keyText;  }
> >
> >
> >  public Text getValueText() {
> >  return valueText;
> >  }
> >
> >
> >  public void setValueText(Text valueText) {  this.valueText =
> > valueText;  }
> >
> >
> >  private String getXmlKey(String value) {
> >        // Get the Key from the XML in the value.
> >  }
> >
> > }
> >
> > The XML snippet from the Value is fine when it is passed into the
> > map() method.  I am not changing any data either, just pulling out
> > information for the key.  The problem I am seeing is between the Map
> > phase and the Reduce phase, the XML is getting munged.  For Example:
> >
> >  </PrivateRate>
> >  </PrivateRateSet>te>
> >
> > It is my understanding that Hadoop uses the same instance of the Key
> > and Value object when calling the Map method.  What changes is the
> > data within those instances.  So, I ran an experiment where I do not
> > have different Key or Value Text Objects.  I reuse the ones passed
> > into the method, like below:
> >
> > public class XmlMapper extends MapReduceBase implements Mapper {
> >
> >  @SuppressWarnings("unchecked")
> >  public void map(Object key, Object value, OutputCollector output,
> > Reporter reporter) throws IOException {  Text keyText = (Text)key;
> > Text valueText = (Text)value;  String valueString = new
> > String(valueText.getBytes(), "UTF-8");  String keyString =
> > getXmlKey(valueString);  keyText.set(keyString);
> > valueText.set(valueString);  output.collect(keyText, valueText);  }
> >
> >
> >  private String getXmlKey(String value) {
> >        // Get the Key from the XML in the value.
> >  }
> >
> > }
> >
> > What was interesting about this is the fact that the XML was getting
> > munged within the Map Phase.  When I changed over to the code at the
> > top, the Map phase was fine.  However, the Reduce phase picks up the
> > munged XML.  Trying to debug the problem, I came across this method in
>
> > the Text Object:
> >
> > public void set(byte[] utf8, int start, int len) {
> >    setCapacity(len, false);
> >    System.arraycopy(utf8, start, bytes, 0, len);
> >    this.length = len;
> > }
> >
> > If the "bytes" array had a length of 1000 and the "utf8" array has a
> > length of 500; doing a System.arraycopy() would only copy the first
> > 500 from "utf8" to "bytes" but leave the last 500 in "bytes" alone.
> > Could this be the cause of the XML munging?
> >
> > All of this leads me to a few questions:
> >
> > 1) Has anyone successfully used XML snippets as the data format within
>
> > a MapReduce job; not just reading from the file but used during the
> > shuffle?
> > 2) Is anyone seeing this problem with XML or any other format?
> > 3) Does anyone know what is going on?
> > 4) Is this a bug?
> >
> >
> > Thanks,
> >
> > Peter
> >
> >
> >
>
>
>

--00c09f8e5b8f9aaa3b048b8dfb67--

From general-return-1830-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 19 15:02:07 2010
Return-Path: <general-return-1830-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83281 invoked from network); 19 Jul 2010 15:02:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Jul 2010 15:02:07 -0000
Received: (qmail 49091 invoked by uid 500); 19 Jul 2010 15:02:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48983 invoked by uid 500); 19 Jul 2010 15:02:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48975 invoked by uid 99); 19 Jul 2010 15:02:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 15:02:05 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of Peter.Minearo@reardencommerce.com designates 64.41.141.18 as permitted sender)
Received: from [64.41.141.18] (HELO barracuda.mygazoo.com) (64.41.141.18)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 15:01:58 +0000
X-ASG-Debug-ID: 1279551697-505a000e0001-IjwG86
Received: from sccorpmail01.mygazoo.com (sccorpmail01.mygazoo.com [10.38.20.8]) by barracuda.mygazoo.com with ESMTP id Gg3gPmKJZxFUPjFw for <general@hadoop.apache.org>; Mon, 19 Jul 2010 08:01:37 -0700 (PDT)
X-Barracuda-Envelope-From: Peter.Minearo@Reardencommerce.com
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
X-ASG-Orig-Subj: RE: Hadoop and XML
Subject: RE: Hadoop and XML
Date: Mon, 19 Jul 2010 08:01:35 -0700
Message-ID: <5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F5EB@sccorpmail01.mygazoo.com>
In-Reply-To: <AANLkTilAzw5J1oL8raqTmPtxi9BGpuEwHXimEaUdfOzL@mail.gmail.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Hadoop and XML
Thread-Index: Acslar6i5AAclaPBQQ6y7iOQiqwm+wB5Qqgg
References: <5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F46C@sccorpmail01.mygazoo.com><AANLkTilvuqho0l3U9ETg5HHT4Yz5uUkyjX_xqBvmh505@mail.gmail.com><5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D051@sccorpmail01.mygazoo.com><5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F4BA@sccorpmail01.mygazoo.com><5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D053@sccorpmail01.mygazoo.com> <AANLkTilAzw5J1oL8raqTmPtxi9BGpuEwHXimEaUdfOzL@mail.gmail.com>
From: "Peter Minearo" <Peter.Minearo@Reardencommerce.com>
To: <general@hadoop.apache.org>
X-Barracuda-Connect: sccorpmail01.mygazoo.com[10.38.20.8]
X-Barracuda-Start-Time: 1279551697
X-Barracuda-URL: http://10.38.16.14:8000/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at mygazoo.com
X-Virus-Checked: Checked by ClamAV on apache.org

I am already using XmlInputFormat.  The input into the Map phase is not
the problem.  The problem lays in between the Map and Reduce phase.=20

BTW - The article is correct.  DO NOT USE StreamXmlRecordReader.
XmlInputFormat is a lot faster.  From my testing, StreamXmlRecordReader
took 8 minutes to read a 1 GB XML document; where as, XmlInputFormat was
under 2 minutes. (Using 2 Core, 8GB machines)
=20

-----Original Message-----
From: Ted Yu [mailto:yuzhihong@gmail.com]=20
Sent: Friday, July 16, 2010 9:44 PM
To: general@hadoop.apache.org
Subject: Re: Hadoop and XML

>From an earlier post:
http://oobaloo.co.uk/articles/2010/1/20/processing-xml-in-hadoop.html

On Fri, Jul 16, 2010 at 3:07 PM, Peter Minearo <
Peter.Minearo@reardencommerce.com> wrote:

> Moving the variable to a local variable did not seem to work:
>
>
> </PrivateRateSet>vateRateSet>
>
>
>
> public void map(Object key, Object value, OutputCollector output,=20
> Reporter
> reporter) throws IOException {
>                Text valueText =3D (Text)value;
>                String valueString =3D new String(valueText.getBytes(), =

> "UTF-8");
>                String keyString =3D getXmlKey(valueString);
>                 Text returnKeyText =3D new Text();
>                Text returnValueText =3D new Text();
>                returnKeyText.set(keyString);
>                returnValueText.set(valueString);
>                output.collect(returnKeyText, returnValueText); }
>
> -----Original Message-----
> From: Peter Minearo [mailto:Peter.Minearo@Reardencommerce.com]
> Sent: Fri 7/16/2010 2:51 PM
> To: general@hadoop.apache.org
> Subject: RE: Hadoop and XML
>
> Whoops....right after I sent it and someone else made a suggestion; I=20
> realized what question 2 was about.  I can try that, but wouldn't that

> cause Object bloat?  During the Hadoop training I went through; it was

> mentioned to reuse the returning Key and Value objects to keep the=20
> number of Objects created down to a minimum.  Is this not really a=20
> valid point?
>
>
>
> -----Original Message-----
> From: Peter Minearo [mailto:Peter.Minearo@Reardencommerce.com]
> Sent: Friday, July 16, 2010 2:44 PM
> To: general@hadoop.apache.org
> Subject: RE: Hadoop and XML
>
>
> I am not using multi-threaded Map tasks.  Also, if I understand your=20
> second question correctly:
> "Also can you try creating the output key and values in the map=20
> method(method lacal) ?"
> In the first code snippet I am doing exactly that.
>
> Below is the class that runs the Job.
>
> public class HadoopJobClient {
>
>        private static final Log LOGGER =3D=20
> LogFactory.getLog(Prds.class.getName());
>
>        public static void main(String[] args) {
>                JobConf conf =3D new JobConf(Prds.class);
>
>                conf.set("xmlinput.start", "<PrivateRateSet>");
>                conf.set("xmlinput.end", "</PrivateRateSet>");
>
>                conf.setJobName("PRDS Parse");
>
>                conf.setOutputKeyClass(Text.class);
>                conf.setOutputValueClass(Text.class);
>
>                conf.setMapperClass(PrdsMapper.class);
>                conf.setReducerClass(PrdsReducer.class);
>
>                conf.setInputFormat(XmlInputFormat.class);
>                conf.setOutputFormat(TextOutputFormat.class);
>
>                FileInputFormat.setInputPaths(conf, new Path(args[0]));
>                FileOutputFormat.setOutputPath(conf, new=20
> Path(args[1]));
>
>                // Run the job
>                try {
>                        JobClient.runJob(conf);
>                } catch (IOException e) {
>                        LOGGER.error(e.getMessage(), e);
>                }
>
>        }
>
>
> }
>
>
>
>
> -----Original Message-----
> From: Soumya Banerjee [mailto:soumya.sbanerjee@gmail.com]
> Sent: Fri 7/16/2010 2:29 PM
> To: general@hadoop.apache.org
> Subject: Re: Hadoop and XML
>
> Hi,
>
> Can you please share the code of the job submission client ?
>
> Also can you try creating the output key and values in the map=20
> method(method
> lacal) ?
> Make sure you are not using multi threaded map task configuration.
>
> map()
> {
> private Text keyText =3D new Text();
>  private Text valueText =3D new Text();
>
> //rest of the code
> }
>
> Soumya.
>
> On Sat, Jul 17, 2010 at 2:30 AM, Peter Minearo <=20
> Peter.Minearo@reardencommerce.com> wrote:
>
> > I have an XML file that has sparse data in it.  I am running a=20
> > MapReduce Job that reads in an XML file, pulls out a Key from within

> > the XML snippet and then hands back the Key and the XML snippet (as=20
> > the Value) to the OutputCollector.  The reason is to sort the file
> back into order.
> > Below is the snippet of code.
> >
> > public class XmlMapper extends MapReduceBase implements Mapper {
> >
> >  private Text keyText =3D new Text();
> >  private Text valueText =3D new Text();
> >
> >  @SuppressWarnings("unchecked")
> >  public void map(Object key, Object value, OutputCollector output,=20
> > Reporter reporter) throws IOException {  Text valueText =3D=20
> > (Text)value;
>
> > String valueString =3D new String(valueText.getBytes(), "UTF-8");=20
> > String keyString =3D getXmlKey(valueString);=20
> > getKeyText().set(keyString);  getValueText().set(valueString);=20
> > output.collect(getKeyText(), getValueText());  }
> >
> >
> >  public Text getKeyText() {
> >  return keyText;
> >  }
> >
> >
> >  public void setKeyText(Text keyText) {  this.keyText =3D keyText;  =
}
> >
> >
> >  public Text getValueText() {
> >  return valueText;
> >  }
> >
> >
> >  public void setValueText(Text valueText) {  this.valueText =3D=20
> > valueText;  }
> >
> >
> >  private String getXmlKey(String value) {
> >        // Get the Key from the XML in the value.
> >  }
> >
> > }
> >
> > The XML snippet from the Value is fine when it is passed into the
> > map() method.  I am not changing any data either, just pulling out=20
> > information for the key.  The problem I am seeing is between the Map

> > phase and the Reduce phase, the XML is getting munged.  For Example:
> >
> >  </PrivateRate>
> >  </PrivateRateSet>te>
> >
> > It is my understanding that Hadoop uses the same instance of the Key

> > and Value object when calling the Map method.  What changes is the=20
> > data within those instances.  So, I ran an experiment where I do not

> > have different Key or Value Text Objects.  I reuse the ones passed=20
> > into the method, like below:
> >
> > public class XmlMapper extends MapReduceBase implements Mapper {
> >
> >  @SuppressWarnings("unchecked")
> >  public void map(Object key, Object value, OutputCollector output,=20
> > Reporter reporter) throws IOException {  Text keyText =3D (Text)key; =

> > Text valueText =3D (Text)value;  String valueString =3D new=20
> > String(valueText.getBytes(), "UTF-8");  String keyString =3D=20
> > getXmlKey(valueString);  keyText.set(keyString);=20
> > valueText.set(valueString);  output.collect(keyText, valueText);  }
> >
> >
> >  private String getXmlKey(String value) {
> >        // Get the Key from the XML in the value.
> >  }
> >
> > }
> >
> > What was interesting about this is the fact that the XML was getting

> > munged within the Map Phase.  When I changed over to the code at the

> > top, the Map phase was fine.  However, the Reduce phase picks up the

> > munged XML.  Trying to debug the problem, I came across this method=20
> > in
>
> > the Text Object:
> >
> > public void set(byte[] utf8, int start, int len) {
> >    setCapacity(len, false);
> >    System.arraycopy(utf8, start, bytes, 0, len);
> >    this.length =3D len;
> > }
> >
> > If the "bytes" array had a length of 1000 and the "utf8" array has a

> > length of 500; doing a System.arraycopy() would only copy the first=20
> > 500 from "utf8" to "bytes" but leave the last 500 in "bytes" alone.
> > Could this be the cause of the XML munging?
> >
> > All of this leads me to a few questions:
> >
> > 1) Has anyone successfully used XML snippets as the data format=20
> > within
>
> > a MapReduce job; not just reading from the file but used during the=20
> > shuffle?
> > 2) Is anyone seeing this problem with XML or any other format?
> > 3) Does anyone know what is going on?
> > 4) Is this a bug?
> >
> >
> > Thanks,
> >
> > Peter
> >
> >
> >
>
>
>

From general-return-1831-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 19 16:08:51 2010
Return-Path: <general-return-1831-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 11651 invoked from network); 19 Jul 2010 16:08:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Jul 2010 16:08:50 -0000
Received: (qmail 39305 invoked by uid 500); 19 Jul 2010 16:08:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39237 invoked by uid 500); 19 Jul 2010 16:08:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39229 invoked by uid 99); 19 Jul 2010 16:08:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 16:08:48 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates 209.85.213.48 as permitted sender)
Received: from [209.85.213.48] (HELO mail-yw0-f48.google.com) (209.85.213.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 16:08:42 +0000
Received: by ywe9 with SMTP id 9so640586ywe.35
        for <general@hadoop.apache.org>; Mon, 19 Jul 2010 09:08:21 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=30f1K7u7SWWYUWonVNtUmFpCDcDFh3HKeFqNFpd+cHw=;
        b=YZsLz3o4QIfRjSEF6A2hOeCdcjr1hwzQOhmbUo7klusioZx9yN/GwOi6eXKyIzcazF
         aylj/cFiAlf7r8Dh8sVP7Ck+B7rXyPiIxNmLHhIHfEcvfbFiZQ+PM6J443dicMR1BlE/
         KbOy3VTFW78CZyDLXB1GFubwd5px+mK5Smaic=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=YdlfZXI98VRgzzb5khXOPrWVL/494qPLbFB95lK5wrx/rxh4fIx0Iiky/CyGfknNlm
         EyB5SmF3eyiHl8Cwq84qdGgOhKZSRmr/QGQxczwU+OkL2u7yJUc1r8eDM2gyqOzO8YFb
         dj2sG+7pvRLkjfw25d/qj4L8sXLK8ciBJ74Jk=
MIME-Version: 1.0
Received: by 10.100.38.4 with SMTP id l4mr993490anl.134.1279555699661; Mon, 19 
	Jul 2010 09:08:19 -0700 (PDT)
Received: by 10.101.4.24 with HTTP; Mon, 19 Jul 2010 09:08:19 -0700 (PDT)
In-Reply-To: <5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F5EB@sccorpmail01.mygazoo.com>
References: <5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F46C@sccorpmail01.mygazoo.com>
	<AANLkTilvuqho0l3U9ETg5HHT4Yz5uUkyjX_xqBvmh505@mail.gmail.com>
	<5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D051@sccorpmail01.mygazoo.com>
	<5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F4BA@sccorpmail01.mygazoo.com>
	<5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D053@sccorpmail01.mygazoo.com>
	<AANLkTilAzw5J1oL8raqTmPtxi9BGpuEwHXimEaUdfOzL@mail.gmail.com>
	<5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F5EB@sccorpmail01.mygazoo.com>
Date: Mon, 19 Jul 2010 09:08:19 -0700
Message-ID: <AANLkTinDkca4ABkityQiyu9GGX-NddzE0pMkAiIRN9J3@mail.gmail.com>
Subject: Re: Hadoop and XML
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e640d3d2f2de30048bbfc6e0
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e640d3d2f2de30048bbfc6e0
Content-Type: text/plain; charset=ISO-8859-1

For your initial question on Text.set().
Text.setCapacity() allocates new byte array. Since keepData is false, old
data wouldn't be copied over.

On Mon, Jul 19, 2010 at 8:01 AM, Peter Minearo <
Peter.Minearo@reardencommerce.com> wrote:

> I am already using XmlInputFormat.  The input into the Map phase is not
> the problem.  The problem lays in between the Map and Reduce phase.
>
> BTW - The article is correct.  DO NOT USE StreamXmlRecordReader.
> XmlInputFormat is a lot faster.  From my testing, StreamXmlRecordReader
> took 8 minutes to read a 1 GB XML document; where as, XmlInputFormat was
> under 2 minutes. (Using 2 Core, 8GB machines)
>
>
> -----Original Message-----
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: Friday, July 16, 2010 9:44 PM
> To: general@hadoop.apache.org
> Subject: Re: Hadoop and XML
>
> From an earlier post:
> http://oobaloo.co.uk/articles/2010/1/20/processing-xml-in-hadoop.html
>
> On Fri, Jul 16, 2010 at 3:07 PM, Peter Minearo <
> Peter.Minearo@reardencommerce.com> wrote:
>
> > Moving the variable to a local variable did not seem to work:
> >
> >
> > </PrivateRateSet>vateRateSet>
> >
> >
> >
> > public void map(Object key, Object value, OutputCollector output,
> > Reporter
> > reporter) throws IOException {
> >                Text valueText = (Text)value;
> >                String valueString = new String(valueText.getBytes(),
> > "UTF-8");
> >                String keyString = getXmlKey(valueString);
> >                 Text returnKeyText = new Text();
> >                Text returnValueText = new Text();
> >                returnKeyText.set(keyString);
> >                returnValueText.set(valueString);
> >                output.collect(returnKeyText, returnValueText); }
> >
> > -----Original Message-----
> > From: Peter Minearo [mailto:Peter.Minearo@Reardencommerce.com]
> > Sent: Fri 7/16/2010 2:51 PM
> > To: general@hadoop.apache.org
> > Subject: RE: Hadoop and XML
> >
> > Whoops....right after I sent it and someone else made a suggestion; I
> > realized what question 2 was about.  I can try that, but wouldn't that
>
> > cause Object bloat?  During the Hadoop training I went through; it was
>
> > mentioned to reuse the returning Key and Value objects to keep the
> > number of Objects created down to a minimum.  Is this not really a
> > valid point?
> >
> >
> >
> > -----Original Message-----
> > From: Peter Minearo [mailto:Peter.Minearo@Reardencommerce.com]
> > Sent: Friday, July 16, 2010 2:44 PM
> > To: general@hadoop.apache.org
> > Subject: RE: Hadoop and XML
> >
> >
> > I am not using multi-threaded Map tasks.  Also, if I understand your
> > second question correctly:
> > "Also can you try creating the output key and values in the map
> > method(method lacal) ?"
> > In the first code snippet I am doing exactly that.
> >
> > Below is the class that runs the Job.
> >
> > public class HadoopJobClient {
> >
> >        private static final Log LOGGER =
> > LogFactory.getLog(Prds.class.getName());
> >
> >        public static void main(String[] args) {
> >                JobConf conf = new JobConf(Prds.class);
> >
> >                conf.set("xmlinput.start", "<PrivateRateSet>");
> >                conf.set("xmlinput.end", "</PrivateRateSet>");
> >
> >                conf.setJobName("PRDS Parse");
> >
> >                conf.setOutputKeyClass(Text.class);
> >                conf.setOutputValueClass(Text.class);
> >
> >                conf.setMapperClass(PrdsMapper.class);
> >                conf.setReducerClass(PrdsReducer.class);
> >
> >                conf.setInputFormat(XmlInputFormat.class);
> >                conf.setOutputFormat(TextOutputFormat.class);
> >
> >                FileInputFormat.setInputPaths(conf, new Path(args[0]));
> >                FileOutputFormat.setOutputPath(conf, new
> > Path(args[1]));
> >
> >                // Run the job
> >                try {
> >                        JobClient.runJob(conf);
> >                } catch (IOException e) {
> >                        LOGGER.error(e.getMessage(), e);
> >                }
> >
> >        }
> >
> >
> > }
> >
> >
> >
> >
> > -----Original Message-----
> > From: Soumya Banerjee [mailto:soumya.sbanerjee@gmail.com]
> > Sent: Fri 7/16/2010 2:29 PM
> > To: general@hadoop.apache.org
> > Subject: Re: Hadoop and XML
> >
> > Hi,
> >
> > Can you please share the code of the job submission client ?
> >
> > Also can you try creating the output key and values in the map
> > method(method
> > lacal) ?
> > Make sure you are not using multi threaded map task configuration.
> >
> > map()
> > {
> > private Text keyText = new Text();
> >  private Text valueText = new Text();
> >
> > //rest of the code
> > }
> >
> > Soumya.
> >
> > On Sat, Jul 17, 2010 at 2:30 AM, Peter Minearo <
> > Peter.Minearo@reardencommerce.com> wrote:
> >
> > > I have an XML file that has sparse data in it.  I am running a
> > > MapReduce Job that reads in an XML file, pulls out a Key from within
>
> > > the XML snippet and then hands back the Key and the XML snippet (as
> > > the Value) to the OutputCollector.  The reason is to sort the file
> > back into order.
> > > Below is the snippet of code.
> > >
> > > public class XmlMapper extends MapReduceBase implements Mapper {
> > >
> > >  private Text keyText = new Text();
> > >  private Text valueText = new Text();
> > >
> > >  @SuppressWarnings("unchecked")
> > >  public void map(Object key, Object value, OutputCollector output,
> > > Reporter reporter) throws IOException {  Text valueText =
> > > (Text)value;
> >
> > > String valueString = new String(valueText.getBytes(), "UTF-8");
> > > String keyString = getXmlKey(valueString);
> > > getKeyText().set(keyString);  getValueText().set(valueString);
> > > output.collect(getKeyText(), getValueText());  }
> > >
> > >
> > >  public Text getKeyText() {
> > >  return keyText;
> > >  }
> > >
> > >
> > >  public void setKeyText(Text keyText) {  this.keyText = keyText;  }
> > >
> > >
> > >  public Text getValueText() {
> > >  return valueText;
> > >  }
> > >
> > >
> > >  public void setValueText(Text valueText) {  this.valueText =
> > > valueText;  }
> > >
> > >
> > >  private String getXmlKey(String value) {
> > >        // Get the Key from the XML in the value.
> > >  }
> > >
> > > }
> > >
> > > The XML snippet from the Value is fine when it is passed into the
> > > map() method.  I am not changing any data either, just pulling out
> > > information for the key.  The problem I am seeing is between the Map
>
> > > phase and the Reduce phase, the XML is getting munged.  For Example:
> > >
> > >  </PrivateRate>
> > >  </PrivateRateSet>te>
> > >
> > > It is my understanding that Hadoop uses the same instance of the Key
>
> > > and Value object when calling the Map method.  What changes is the
> > > data within those instances.  So, I ran an experiment where I do not
>
> > > have different Key or Value Text Objects.  I reuse the ones passed
> > > into the method, like below:
> > >
> > > public class XmlMapper extends MapReduceBase implements Mapper {
> > >
> > >  @SuppressWarnings("unchecked")
> > >  public void map(Object key, Object value, OutputCollector output,
> > > Reporter reporter) throws IOException {  Text keyText = (Text)key;
> > > Text valueText = (Text)value;  String valueString = new
> > > String(valueText.getBytes(), "UTF-8");  String keyString =
> > > getXmlKey(valueString);  keyText.set(keyString);
> > > valueText.set(valueString);  output.collect(keyText, valueText);  }
> > >
> > >
> > >  private String getXmlKey(String value) {
> > >        // Get the Key from the XML in the value.
> > >  }
> > >
> > > }
> > >
> > > What was interesting about this is the fact that the XML was getting
>
> > > munged within the Map Phase.  When I changed over to the code at the
>
> > > top, the Map phase was fine.  However, the Reduce phase picks up the
>
> > > munged XML.  Trying to debug the problem, I came across this method
> > > in
> >
> > > the Text Object:
> > >
> > > public void set(byte[] utf8, int start, int len) {
> > >    setCapacity(len, false);
> > >    System.arraycopy(utf8, start, bytes, 0, len);
> > >    this.length = len;
> > > }
> > >
> > > If the "bytes" array had a length of 1000 and the "utf8" array has a
>
> > > length of 500; doing a System.arraycopy() would only copy the first
> > > 500 from "utf8" to "bytes" but leave the last 500 in "bytes" alone.
> > > Could this be the cause of the XML munging?
> > >
> > > All of this leads me to a few questions:
> > >
> > > 1) Has anyone successfully used XML snippets as the data format
> > > within
> >
> > > a MapReduce job; not just reading from the file but used during the
> > > shuffle?
> > > 2) Is anyone seeing this problem with XML or any other format?
> > > 3) Does anyone know what is going on?
> > > 4) Is this a bug?
> > >
> > >
> > > Thanks,
> > >
> > > Peter
> > >
> > >
> > >
> >
> >
> >
>

--0016e640d3d2f2de30048bbfc6e0--

From general-return-1832-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 19 18:22:34 2010
Return-Path: <general-return-1832-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44661 invoked from network); 19 Jul 2010 18:22:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Jul 2010 18:22:33 -0000
Received: (qmail 27874 invoked by uid 500); 19 Jul 2010 18:22:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27811 invoked by uid 500); 19 Jul 2010 18:22:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27803 invoked by uid 99); 19 Jul 2010 18:22:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 18:22:32 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of michael_segel@hotmail.com designates 65.55.34.21 as permitted sender)
Received: from [65.55.34.21] (HELO col0-omc1-s11.col0.hotmail.com) (65.55.34.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 18:22:23 +0000
Received: from COL117-W14 ([65.55.34.7]) by col0-omc1-s11.col0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Mon, 19 Jul 2010 11:22:01 -0700
Message-ID: <COL117-W1436E4A01193F93A457C698FBF0@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_79691937-4b13-4ad4-87fc-0d0ab86a3fb1_"
X-Originating-IP: [65.167.11.254]
From: Michael Segel <michael_segel@hotmail.com>
To: <general@hadoop.apache.org>
Subject: Dynamic counters in Hadoop M/R jobs...
Date: Mon, 19 Jul 2010 13:22:02 -0500
Importance: Normal
MIME-Version: 1.0
X-OriginalArrivalTime: 19 Jul 2010 18:22:01.0740 (UTC) FILETIME=[484314C0:01CB276F]
X-Virus-Checked: Checked by ClamAV on apache.org

--_79691937-4b13-4ad4-87fc-0d0ab86a3fb1_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


Hi=2C

In looking at "The Definitive Guide"  pgs 211-218 looking at the documentat=
ion for counters have a question about how to use the

dynamic (String=2C String) methods to create/access counters.

Looking at the IdentityTableMap class=2C the following map() method exists(=
):
public void map(ImmutableBytesWritable key=2C
                RowResult value=2C
                org.apache.hadoop.mapred.OutputCollector<ImmutableBytesWrit=
able=2CRowResult> output=2C
                org.apache.hadoop.mapred.Reporter reporter)
         throws IOExceptionThis method is deprecated in 20.5

So how can you get the Reporter so you can use the 'reporter.incrCounter(St=
ring group=2CString counter=2C long amount)'
method. (See pg 214 code fragment)=20

I must be missing something.

Thx

-Mike


 		 	   		 =20
_________________________________________________________________
The New Busy is not the too busy. Combine all your e-mail accounts with Hot=
mail.
http://www.windowslive.com/campaign/thenewbusy?tile=3Dmultiaccount&ocid=3DP=
ID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_4=

--_79691937-4b13-4ad4-87fc-0d0ab86a3fb1_--

From general-return-1833-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 19 18:22:38 2010
Return-Path: <general-return-1833-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44731 invoked from network); 19 Jul 2010 18:22:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Jul 2010 18:22:38 -0000
Received: (qmail 29004 invoked by uid 500); 19 Jul 2010 18:22:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28946 invoked by uid 500); 19 Jul 2010 18:22:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28938 invoked by uid 99); 19 Jul 2010 18:22:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 18:22:36 +0000
X-ASF-Spam-Status: No, hits=-0.1 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of msilva@specificmedia.com designates 208.65.144.82 as permitted sender)
Received: from [208.65.144.82] (HELO p02c11o149.mxlogic.net) (208.65.144.82)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 18:22:27 +0000
Received: from unknown [206.82.192.117] (EHLO p02c11o149.mxlogic.net)
	by p02c11o149.mxlogic.net(mxl_mta-6.7.0-0)
	with ESMTP id ec7944c4.4d74f940.17399.00-596.41956.p02c11o149.mxlogic.net (envelope-from <msilva@specificmedia.com>);
	Mon, 19 Jul 2010 12:22:06 -0600 (MDT)
X-MXL-Hash: 4c4497ce1102db40-e8ac709177645ec1f2576d9acae5d6ac2d5d9714
Received: from unknown [206.82.192.117]
	by p02c11o149.mxlogic.net(mxl_mta-6.7.0-0)
	with SMTP id e87944c4.0.16074.00-390.41449.p02c11o149.mxlogic.net (envelope-from <msilva@specificmedia.com>);
	Mon, 19 Jul 2010 12:22:02 -0600 (MDT)
X-MXL-Hash: 4c4497ca375c720a-a71f418326d3d46bbd85a9aea96283c7d8cfb899
Received: from west-corp-ex02.specificmedia.local ([192.168.251.16]) by
 WEST-CORP-EX01.specificmedia.local ([192.168.251.15]) with mapi; Mon, 19 Jul
 2010 11:19:26 -0700
From: Matias Silva <MSilva@specificmedia.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 19 Jul 2010 11:19:26 -0700
Subject: Hive Returns Empty Results
Thread-Topic: Hive Returns Empty Results
Thread-Index: AcsnbuvKJUMrbgyvSE6JxrJ1xnL/IA==
Message-ID: <1D39027E38D41B4C91702F159EF19160253914A9C2@west-corp-ex02.specificmedia.local>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_1D39027E38D41B4C91702F159EF19160253914A9C2westcorpex02s_"
MIME-Version: 1.0
X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010070601)]
X-MAIL-FROM: <msilva@specificmedia.com>
X-SOURCE-IP: [206.82.192.117]
X-AnalysisOut: [v=1.0 c=1 a=VphdPIyG4kEA:10 a=YpQMAEZSN0BV7U5AHc4XUw==:17 ]
X-AnalysisOut: [a=DRd3HSgQ8n8vC0nKMRcA:9 a=MucCq7ZUDrmT8o1dJzFEpNStuRQA:4 ]
X-AnalysisOut: [a=CjuIK1q_8ugA:10 a=SSmOFEACAAAA:8 a=Y2VNeNrzAAAA:8 a=yMhM]
X-AnalysisOut: [jlubAAAA:8 a=TW66zc2HAAAA:8 a=HQ31llbKAAAA:8 a=V2BWb4iUXoc]
X-AnalysisOut: [h7CQ8JaQA:9 a=Scua6usYcg9TCaPIwGkA:7 a=tUJzjhdtsylBKWY9jOm]
X-AnalysisOut: [eqyna1QIA:4]
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_1D39027E38D41B4C91702F159EF19160253914A9C2westcorpex02s_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi Everyone, my hive interactive shell query returns with empty results whe=
n there should be some results.  I noticed this from time
to time.  Has this happened to anybody? and if so what can I do to resolve =
it assuming that you have resolved the
issue.

I'm using hadoop-hive-0.4.1+14.5-1

Thanks for your time and knowledge.

Best,
Matt


--_000_1D39027E38D41B4C91702F159EF19160253914A9C2westcorpex02s_--

From general-return-1834-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 19 20:32:49 2010
Return-Path: <general-return-1834-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88572 invoked from network); 19 Jul 2010 20:32:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Jul 2010 20:32:48 -0000
Received: (qmail 27027 invoked by uid 500); 19 Jul 2010 20:32:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26971 invoked by uid 500); 19 Jul 2010 20:32:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26963 invoked by uid 99); 19 Jul 2010 20:32:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 20:32:47 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.27.212] (HELO qmta14.emeryville.ca.mail.comcast.net) (76.96.27.212)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 20:32:37 +0000
Received: from omta24.emeryville.ca.mail.comcast.net ([76.96.30.92])
	by qmta14.emeryville.ca.mail.comcast.net with comcast
	id jqwK1e0061zF43QAEwYFNU; Mon, 19 Jul 2010 20:32:15 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta24.emeryville.ca.mail.comcast.net with comcast
	id jwY61e00L2SbwD58kwY9G8; Mon, 19 Jul 2010 20:32:13 +0000
Message-Id: <C68A4752-05BA-4A8E-97EE-A18B6344C69B@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <1D39027E38D41B4C91702F159EF19160253914A9C2@west-corp-ex02.specificmedia.local>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Hive Returns Empty Results
Date: Mon, 19 Jul 2010 13:32:06 -0700
References: <1D39027E38D41B4C91702F159EF19160253914A9C2@west-corp-ex02.specificmedia.local>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


On Jul 19, 2010, at 11:19 AM, Matias Silva wrote:

> Hi Everyone, my hive interactive shell query returns with empty  
> results when there should be some results.  I noticed this from time
> to time.  Has this happened to anybody? and if so what can I do to  
> resolve it assuming that you have resolved the
> issue.

General is for announcements that cross all of the sub-projects.  
Please move this over to hive-user@hadoop.apache.org

Thanks,
    Owen

From general-return-1835-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 19 20:54:30 2010
Return-Path: <general-return-1835-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95422 invoked from network); 19 Jul 2010 20:54:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Jul 2010 20:54:30 -0000
Received: (qmail 54732 invoked by uid 500); 19 Jul 2010 20:54:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54651 invoked by uid 500); 19 Jul 2010 20:54:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54643 invoked by uid 99); 19 Jul 2010 20:54:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 20:54:28 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of zookog@gmail.com designates 209.85.216.176 as permitted sender)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 20:54:21 +0000
Received: by qyk12 with SMTP id 12so2912879qyk.14
        for <general@hadoop.apache.org>; Mon, 19 Jul 2010 13:53:00 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=KqLVIIMXi0bhnZTXH6zfl337QZLcwXDsHLCJ8lgJrUY=;
        b=sD6uCNQzyia57sgbz4c7htOVlnOhYzeTVZ8Ueb7bP0H1nlHxkyb+FTz2Bmuq5aLzJE
         +JqOVzF1iAm+iNakAKB5HF2GPtVsjrlHDCAWRoH5CNBvwPlkW+U1L/mr/vXPrRHNGsLa
         nAu3MqQuTGyDkOcoiDnc0bN8YmBkbxcrLd/1E=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:date:x-google-sender-auth:message-id:subject
         :from:to:content-type:content-transfer-encoding;
        b=R9iUtKXiCRh0aIvpAepzoyh5dyBpkvD5WCeT5lBd5DeEEMZ+UdtPWogd9xdZphRIPa
         OkL7uaNEYRRfpZm4C3uOYR/y85z88WYltelSgXPhLfTFJjr3S+TDOVIl6szXBClVwBUZ
         KM8lQaZdVBWv+GI4mV8m+DeSERNxmTB+W5Eiw=
MIME-Version: 1.0
Received: by 10.224.40.207 with SMTP id l15mr5321934qae.224.1279572780378; 
	Mon, 19 Jul 2010 13:53:00 -0700 (PDT)
Sender: zookog@gmail.com
Received: by 10.229.181.80 with HTTP; Mon, 19 Jul 2010 13:53:00 -0700 (PDT)
Date: Mon, 19 Jul 2010 14:53:00 -0600
X-Google-Sender-Auth: FZAYzsE2jKzLjCHki4mmOfK752A
Message-ID: <AANLkTin05yQ9inPe6Ir3QKUfomXit-92I7otSkR9vs36@mail.gmail.com>
Subject: ANNOUNCING Tahoe, the Least-Authority File System, v1.7.1
From: "Zooko O'Whielacronx" <zooko@zooko.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Folks:

Tahoe-LAFS is a secure, distributed filesystem that can be used as an
alternate filesystem to HDFS, as described in these slides [1] and
implemented in this project named hadoop-lafs.

We just released v1.7.1 of Tahoe-LAFS. Here is the release
announcement. If you try it out then by all means let us know what you
think!

Regards,

Zooko Wilcox-O'Hearn

[1] http://docs.google.com/fileview?id=3D0B9iCsXQ_HuEpN2NlNDgwMTMtNWRjOC00Y=
zMwLWExYjMtYzVmM2FiZGRhNjA4&hl=3Den&pli=3D1
[2] http://code.google.com/p/hadoop-lafs


ANNOUNCING Tahoe, the Least-Authority File System, v1.7.1

The Tahoe-LAFS team is pleased to announce the immediate
availability of version 1.7.1 of Tahoe-LAFS, an extremely
reliable distributed storage system.

Tahoe-LAFS is the first distributed storage system which
offers "provider-independent security"=E2=80=94meaning that not even
the operators of your storage servers can read or alter your
data without your consent. Here is the one-page explanation of
its unique security and fault-tolerance properties:

http://tahoe-lafs.org/source/tahoe/trunk/docs/about.html

Tahoe-LAFS v1.7.1 is the successor to v1.7.0, which was
released June 18, 2010 [1].

v1.7.1 is a stable minor release which adds several bugfixes
and improvements in security, forward-compatibility,
packaging, usability, and portability. See the NEWS file [2]
for details.


WHAT IS IT GOOD FOR?

With Tahoe-LAFS, you distribute your filesystem across
multiple servers, and even if some of the servers are
compromised by by an attacker, the entire filesystem continues
to work correctly, and continues to preserve your privacy and
security. You can easily share specific files and directories
with other people.

In addition to the core storage system itself, volunteers have
built other projects on top of Tahoe-LAFS and have integrated
Tahoe-LAFS with existing systems.

These include frontends for Windows, Macintosh, JavaScript,
iPhone, and Android, and plugins for Hadoop, bzr, mercurial,
duplicity, TiddlyWiki, and more. See the Related Projects page
on the wiki [3].

We believe that strong cryptography, Free and Open Source
Software, erasure coding, and principled engineering practices
make Tahoe-LAFS safer than RAID, removable drive, tape,
on-line backup or "cloud storage" systems.

This software is developed under test-driven development, and
there are no known bugs or security flaws which would
compromise confidentiality or data integrity under recommended
use. (For all currently known issues please see the
known_issues.txt file [4].)


COMPATIBILITY

This release is fully compatible with the version 1 series of
Tahoe-LAFS. Clients from this release can write files and
directories in the format used by clients of all versions back
to v1.0 (which was released March 25, 2008). Clients from this
release can read files and directories produced by clients of
all versions since v1.0. Servers from this release can serve
clients of all versions back to v1.0 and clients from this
release can use servers of all versions back to v1.0.

This is the tenth release in the version 1 series. This series
of Tahoe-LAFS will be actively supported and maintained for
the forseeable future, and future versions of Tahoe-LAFS will
retain the ability to read and write files compatible with
this series.


LICENCE

You may use this package under the GNU General Public License,
version 2 or, at your option, any later version. See the file
"COPYING.GPL" [5] for the terms of the GNU General Public
License, version 2.

You may use this package under the Transitive Grace Period
Public Licence, version 1 or, at your option, any later
version. (The Transitive Grace Period Public Licence has
requirements similar to the GPL except that it allows you to
delay for up to twelve months after you redistribute a derived
work before releasing the source code of your derived work.)
See the file "COPYING.TGPPL.html" [6] for the terms of the
Transitive Grace Period Public Licence, version 1.

(You may choose to use this package under the terms of either
licence, at your option.)


INSTALLATION

Tahoe-LAFS works on Linux, Mac OS X, Windows, Cygwin, Solaris,
*BSD, and probably most other systems. Start with
"docs/quickstart.html" [7].


HACKING AND COMMUNITY

Please join us on the mailing list [8]. Patches are gratefully
accepted -- the RoadMap page [9] shows the next improvements
that we plan to make and CREDITS [10] lists the names of people
who've contributed to the project. The Dev page [11] contains
resources for hackers.


SPONSORSHIP

Tahoe-LAFS was originally developed by Allmydata, Inc., a
provider of commercial backup services. After discontinuing
funding of Tahoe-LAFS R&D in early 2009, they have continued
to provide servers, bandwidth, small personal gifts as tokens
of appreciation, and bug reports. Thank you to Allmydata,
Inc. for their generous and public-spirited support.

Google, Inc. is sponsoring Tahoe-LAFS development as part of
the Google Summer of Code 2010. Google suggested that we
should apply for the Summer of Code program, and when we did
they generously awarded four sponsorships to students from
around the world to hack on Tahoe-LAFS this summer. Thank you
to Google, Inc. for their generous and public-spirited
support.


HACK TAHOE-LAFS!

If you can find a security flaw in Tahoe-LAFS which is serious
enough that feel compelled to warn our users and issue a fix,
then we will award you with a customized t-shirts with your
exploit printed on it and add you to the "Hack Tahoe-LAFS Hall
Of Fame" [12].


ACKNOWLEDGEMENTS

This is the fifth release of Tahoe-LAFS to be created solely
as a labor of love by volunteers. Thank you very much to the
team of "hackers in the public interest" who make Tahoe-LAFS
possible.

David-Sarah Hopwood and Zooko Wilcox-O'Hearn
on behalf of the Tahoe-LAFS team

July 18, 2010
Rainhill, Merseyside, UK and Boulder, Colorado, USA


[1] http://tahoe-lafs.org/trac/tahoe/browser/relnotes.txt?rev=3D4514
[2] http://tahoe-lafs.org/trac/tahoe/browser/NEWS?rev=3D4577
[3] http://tahoe-lafs.org/trac/tahoe/wiki/RelatedProjects
[4] http://tahoe-lafs.org/trac/tahoe/browser/docs/known_issues.txt
[5] http://tahoe-lafs.org/trac/tahoe/browser/COPYING.GPL
[6] http://tahoe-lafs.org/source/tahoe/trunk/COPYING.TGPPL.html
[7] http://tahoe-lafs.org/source/tahoe/trunk/docs/quickstart.html
[8] http://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev
[9] http://tahoe-lafs.org/trac/tahoe/roadmap
[10] http://tahoe-lafs.org/trac/tahoe/browser/CREDITS?rev=3D4567
[11] http://tahoe-lafs.org/trac/tahoe/wiki/Dev
[12] http://tahoe-lafs.org/hacktahoelafs/

From general-return-1836-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 00:12:09 2010
Return-Path: <general-return-1836-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76391 invoked from network); 20 Jul 2010 00:12:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 00:12:09 -0000
Received: (qmail 81144 invoked by uid 500); 20 Jul 2010 00:12:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81077 invoked by uid 500); 20 Jul 2010 00:12:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81069 invoked by uid 99); 20 Jul 2010 00:12:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 00:12:07 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of anthony.urso@gmail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 00:12:01 +0000
Received: by iwn37 with SMTP id 37so7550160iwn.35
        for <general@hadoop.apache.org>; Mon, 19 Jul 2010 17:10:39 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:content-type:content-transfer-encoding;
        bh=1NnwWjc1q7tZncQx71TgwgWvfO6P7+oIhH/Z9tZbXbU=;
        b=piYwXLQTBQTVfZkDfNIzvEvWK+Sn2lzyiZqygOtxvI3svm80hlMg5eRMQ62a8HsIvE
         uxlz4P0soknhXy+GSzbHLUlWUKAIbbGjTu3RPN4dFexaK1zek20y951jeFnnqw8vg1wE
         74Fe6qrIL2uKnHSGfLWPquMsj9AAS531h5umU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=ac78QtxyK1oX+QkIMlpdAYZYzRWiiVEF/WGYBBMpAXsJMLS+0FPRnNuNOXPKF2y5Am
         548e9FMTij72SBlSZ5CSia2UrMolvcWj+KTIjsBG5sDM2titRCatlYDiKfl1JptSIHOG
         Cer0gOs4NXeDmhIefMku/SjT83xk4ibQIpNHo=
MIME-Version: 1.0
Received: by 10.231.34.135 with SMTP id l7mr6618360ibd.148.1279584639737; Mon, 
	19 Jul 2010 17:10:39 -0700 (PDT)
Sender: anthony.urso@gmail.com
Received: by 10.231.17.202 with HTTP; Mon, 19 Jul 2010 17:10:39 -0700 (PDT)
In-Reply-To: <COL117-W1436E4A01193F93A457C698FBF0@phx.gbl>
References: <COL117-W1436E4A01193F93A457C698FBF0@phx.gbl>
Date: Mon, 19 Jul 2010 17:10:39 -0700
X-Google-Sender-Auth: 9JyOE-duyhLr-h_k1lxjK90DZms
Message-ID: <AANLkTimIk1TwQYY-D4ZnqBry43yH_zTANXjEB_jx6QzH@mail.gmail.com>
Subject: Re: Dynamic counters in Hadoop M/R jobs...
From: Anthony Urso <anthonyu@cs.ucla.edu>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

The new Mapper class passes a Map.Context object to the map() method.
>From this you can get a StatusReporter object  which  can produce a
named Counter object and increment it.

http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduc=
e/Mapper.Context.html
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduc=
e/StatusReporter.html
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduc=
e/Counter.html

Alternatively, the old APIs are supposed to be temporarily
de-deprecated in the next stable release, so you can use them as
described by that book.

On Mon, Jul 19, 2010 at 11:22 AM, Michael Segel
<michael_segel@hotmail.com> wrote:
>
> Hi,
>
> In looking at "The Definitive Guide" =A0pgs 211-218 looking at the docume=
ntation for counters have a question about how to use the
>
> dynamic (String, String) methods to create/access counters.
>
> Looking at the IdentityTableMap class, the following map() method exists(=
):
> public void map(ImmutableBytesWritable key,
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0RowResult value,
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0org.apache.hadoop.mapred.OutputCollector<I=
mmutableBytesWritable,RowResult> output,
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0org.apache.hadoop.mapred.Reporter reporter=
)
> =A0 =A0 =A0 =A0 throws IOExceptionThis method is deprecated in 20.5
>
> So how can you get the Reporter so you can use the 'reporter.incrCounter(=
String group,String counter, long amount)'
> method. (See pg 214 code fragment)
>
> I must be missing something.
>
> Thx
>
> -Mike
>
>
>
> _________________________________________________________________
> The New Busy is not the too busy. Combine all your e-mail accounts with H=
otmail.
> http://www.windowslive.com/campaign/thenewbusy?tile=3Dmultiaccount&ocid=
=3DPID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_4

From general-return-1837-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 01:57:09 2010
Return-Path: <general-return-1837-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22082 invoked from network); 20 Jul 2010 01:57:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 01:57:09 -0000
Received: (qmail 62023 invoked by uid 500); 20 Jul 2010 01:57:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61981 invoked by uid 500); 20 Jul 2010 01:57:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61973 invoked by uid 99); 20 Jul 2010 01:57:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 01:57:07 +0000
X-ASF-Spam-Status: No, hits=-0.1 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of swatt@us.ibm.com designates 32.97.110.159 as permitted sender)
Received: from [32.97.110.159] (HELO e38.co.us.ibm.com) (32.97.110.159)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 01:56:57 +0000
Received: from d03relay03.boulder.ibm.com (d03relay03.boulder.ibm.com [9.17.195.228])
	by e38.co.us.ibm.com (8.14.4/8.13.1) with ESMTP id o6K1nY6S015477
	for <general@hadoop.apache.org>; Mon, 19 Jul 2010 19:49:34 -0600
Received: from d03av01.boulder.ibm.com (d03av01.boulder.ibm.com [9.17.195.167])
	by d03relay03.boulder.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id o6K1uaiK104252
	for <general@hadoop.apache.org>; Mon, 19 Jul 2010 19:56:36 -0600
Received: from d03av01.boulder.ibm.com (loopback [127.0.0.1])
	by d03av01.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id o6K1uaLF030574
	for <general@hadoop.apache.org>; Mon, 19 Jul 2010 19:56:36 -0600
Received: from d03nm123.boulder.ibm.com (d03nm123.boulder.ibm.com [9.17.195.149])
	by d03av01.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id o6K1uaxu030571
	for <general@hadoop.apache.org>; Mon, 19 Jul 2010 19:56:36 -0600
In-Reply-To: <AANLkTimcA_D60ITpXlqs9Dxb_MkRa74IOiIPspzLrazi@mail.gmail.com>
References: <AANLkTimcA_D60ITpXlqs9Dxb_MkRa74IOiIPspzLrazi@mail.gmail.com>
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: [ANNOUNCE] Austin Hadoop User Group July Meeting - 7/22
X-KeepSent: D0AEE89B:632ECFDE-86257766:0008E85D;
 type=4; name=$KeepSent
X-Mailer: Lotus Notes Release 8.5 December 05, 2008
Message-ID: <OFD0AEE89B.632ECFDE-ON86257766.0008E85D-86257766.000AABBB@us.ibm.com>
From: Stephen Watt <swatt@us.ibm.com>
Date: Mon, 19 Jul 2010 20:56:34 -0500
X-MIMETrack: Serialize by Router on D03NM123/03/M/IBM(Release 8.5.1HF41 | October 22, 2009) at
 07/19/2010 19:56:35,
	Serialize complete at 07/19/2010 19:56:35
Content-Type: multipart/alternative; boundary="=_alternative 000AABBA86257766_="
X-Virus-Checked: Checked by ClamAV on apache.org

--=_alternative 000AABBA86257766_=
Content-Type: text/plain; charset="US-ASCII"

Hi Folks

For those of you who will be in the Austin, Texas this coming Thursday, we 
are having a joint meetup with the Semantic Web Austin group. Our two 
speakers will present on topics that address both Big Data and Semantic 
Analysis. Our meetings are:

1) Awesome
2) Catered 
3) Provide a great to chance to hear about what is going on with Big Data 
in Texas. 

I hope to see you out there - 
http://austinhug.blogspot.com/2010/06/july-meeting.html

Regards
Steve Watt
--=_alternative 000AABBA86257766_=--

From general-return-1838-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 02:02:28 2010
Return-Path: <general-return-1838-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23219 invoked from network); 20 Jul 2010 02:02:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 02:02:28 -0000
Received: (qmail 70069 invoked by uid 500); 20 Jul 2010 02:02:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70007 invoked by uid 500); 20 Jul 2010 02:02:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69999 invoked by uid 99); 20 Jul 2010 02:02:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 02:02:26 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of msegel@navteq.com designates 204.120.70.37 as permitted sender)
Received: from [204.120.70.37] (HELO xmailfargo.navteq.com) (204.120.70.37)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 02:02:20 +0000
Received: from fargo-spamfw.navteq.com (navteq-spamfw.navteq.com [10.6.10.131])
	by xmailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id o6K1bk3v029735
	for <general@hadoop.apache.org>; Mon, 19 Jul 2010 20:37:46 -0500
Received: from imailfargo.navteq.com (localhost [127.0.0.1])
	by fargo-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id 6305647F6885
	for <general@hadoop.apache.org>; Mon, 19 Jul 2010 20:58:49 -0500 (CDT)
Received: from imailfargo.navteq.com (imailfargo.navteq.com [10.6.10.130]) by fargo-spamfw.navteq.com with ESMTP id FHAxFdiwDztaRfAy for <general@hadoop.apache.org>; Mon, 19 Jul 2010 20:58:49 -0500 (CDT)
Received: from hq-ex-ht02.ad.navteq.com ([10.8.222.172])
	by imailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id o6K1wn5W000582
	for <general@hadoop.apache.org>; Mon, 19 Jul 2010 20:58:49 -0500
Received: from hq-ex-mb02.ad.navteq.com ([fe80::880d:318e:e860:a8db]) by
 hq-ex-ht02.ad.navteq.com ([fe80::24f6:ca50:734d:df6e%14]) with mapi; Mon, 19
 Jul 2010 20:58:49 -0500
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 19 Jul 2010 20:58:47 -0500
Subject: RE: Dynamic counters in Hadoop M/R jobs...
Thread-Topic: Dynamic counters in Hadoop M/R jobs...
Thread-Index: AcsnoDI6be7U+cKKSqeUfaaJFVRQ6QACJOZQ
Message-ID: <ABC24175AFD3BE4DA15F4CD375ED413D061487031D@hq-ex-mb02.ad.navteq.com>
References: <COL117-W1436E4A01193F93A457C698FBF0@phx.gbl>
 <AANLkTimIk1TwQYY-D4ZnqBry43yH_zTANXjEB_jx6QzH@mail.gmail.com>
In-Reply-To: <AANLkTimIk1TwQYY-D4ZnqBry43yH_zTANXjEB_jx6QzH@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1
X-Virus-Checked: Checked by ClamAV on apache.org

VGhhbmtzIGZvciB0aGUgcmVwbHkuDQoNCkdvdCBhIGNvdXBsZSBvZiBxdWVzdGlvbnMuLi4NCg0K
SSBzYXcgdGhhdCBJIGNvdWxkIGdldCB0aGUgU3RhdHVzUmVwb3J0ZXIgZnJvbSB0aGUgTWFwcGVy
LkNvbnRleHQuDQpCdXQgdGhlIEFQSSBkb2Vzbid0IGhhdmUgYSB3YXkgdG8gZ2V0IHRoZSBTdGF0
dXNSZXBvcnQgZnJvbSB0aGUgTWFwcGVyLkNvbnRleHQuDQoNCklmIHlvdSdyZSBnb2luZyB0byBn
byB0byB0aGUgTWFwcGVyLkNvbnRleHQgeW91IGNhbiBnZXQgdGhlIGNvdW50ZXJzLg0KQW5kIHlv
dSBoYXZlIGEgZ2V0Q291bnRlcihTdHJpbmcsIFN0cmluZykuIFRoZW4geW91IGNhbiBhbHdheXMg
aW5jcmVtZW50IGl0Lg0KDQpCdXQgdGhpcyBraW5kIG9mIGJlZ3MgYSBxdWVzdGlvbiB3aGljaCBp
c24ndCBkb2N1bWVudGVkLi4uDQoNCklmIEkgaGF2ZSBhIG1hcHBlci5Db250ZXh0IGFuZCBJIHNh
eSBnZXRDb3VudGVyKCJGb28iLCJGb29iYXIgQ291bnQiKTsNCldoYXQgaGFwcGVucyBpZiB0aGVy
ZSBpc24ndCBhIGNvdW50ZXIgKCJGb28iLCJGb29iYXIgQ291bnQiKSA/DQoNCldoYXQgSSBtZWFu
IGlzIGhvdyBkbyB5b3UgYWRkIGEgZHluYW1pYyBjb3VudGVyIG9yIGlmIEkgYXNrIGZvciBhIGNv
dW50ZXIgdGhhdCBkb2Vzbid0IGFscmVhZHkgZXhpc3QsIHdpbGwgaXQgY3JlYXRlIGl0Pw0KKFRo
aXMgaXNuJ3QgZG9jdW1lbnRlZC4uLiBhbmQgaXRzIG5vdCBzYWZlIHRvIGFzc3VtZSBhbnl0aGlu
Zy4pDQoNClNvcnJ5LCBidXQgZHluYW1pYyBjb3VudGVycyBpc24ndCByZWFsbHkgd2VsbCBkb2N1
bWVudGVkLg0KDQotTWlrZQ0KDQotLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KRnJvbTogYW50
aG9ueS51cnNvQGdtYWlsLmNvbSBbbWFpbHRvOmFudGhvbnkudXJzb0BnbWFpbC5jb21dIE9uIEJl
aGFsZiBPZiBBbnRob255IFVyc28NClNlbnQ6IE1vbmRheSwgSnVseSAxOSwgMjAxMCA3OjExIFBN
DQpUbzogZ2VuZXJhbEBoYWRvb3AuYXBhY2hlLm9yZw0KU3ViamVjdDogUmU6IER5bmFtaWMgY291
bnRlcnMgaW4gSGFkb29wIE0vUiBqb2JzLi4uDQoNClRoZSBuZXcgTWFwcGVyIGNsYXNzIHBhc3Nl
cyBhIE1hcC5Db250ZXh0IG9iamVjdCB0byB0aGUgbWFwKCkgbWV0aG9kLg0KRnJvbSB0aGlzIHlv
dSBjYW4gZ2V0IGEgU3RhdHVzUmVwb3J0ZXIgb2JqZWN0ICB3aGljaCAgY2FuIHByb2R1Y2UgYQ0K
bmFtZWQgQ291bnRlciBvYmplY3QgYW5kIGluY3JlbWVudCBpdC4NCg0KaHR0cDovL2hhZG9vcC5h
cGFjaGUub3JnL2NvbW1vbi9kb2NzL2N1cnJlbnQvYXBpL29yZy9hcGFjaGUvaGFkb29wL21hcHJl
ZHVjZS9NYXBwZXIuQ29udGV4dC5odG1sDQpodHRwOi8vaGFkb29wLmFwYWNoZS5vcmcvY29tbW9u
L2RvY3MvY3VycmVudC9hcGkvb3JnL2FwYWNoZS9oYWRvb3AvbWFwcmVkdWNlL1N0YXR1c1JlcG9y
dGVyLmh0bWwNCmh0dHA6Ly9oYWRvb3AuYXBhY2hlLm9yZy9jb21tb24vZG9jcy9jdXJyZW50L2Fw
aS9vcmcvYXBhY2hlL2hhZG9vcC9tYXByZWR1Y2UvQ291bnRlci5odG1sDQoNCkFsdGVybmF0aXZl
bHksIHRoZSBvbGQgQVBJcyBhcmUgc3VwcG9zZWQgdG8gYmUgdGVtcG9yYXJpbHkNCmRlLWRlcHJl
Y2F0ZWQgaW4gdGhlIG5leHQgc3RhYmxlIHJlbGVhc2UsIHNvIHlvdSBjYW4gdXNlIHRoZW0gYXMN
CmRlc2NyaWJlZCBieSB0aGF0IGJvb2suDQoNCk9uIE1vbiwgSnVsIDE5LCAyMDEwIGF0IDExOjIy
IEFNLCBNaWNoYWVsIFNlZ2VsDQo8bWljaGFlbF9zZWdlbEBob3RtYWlsLmNvbT4gd3JvdGU6DQo+
DQo+IEhpLA0KPg0KPiBJbiBsb29raW5nIGF0ICJUaGUgRGVmaW5pdGl2ZSBHdWlkZSIgoHBncyAy
MTEtMjE4IGxvb2tpbmcgYXQgdGhlIGRvY3VtZW50YXRpb24gZm9yIGNvdW50ZXJzIGhhdmUgYSBx
dWVzdGlvbiBhYm91dCBob3cgdG8gdXNlIHRoZQ0KPg0KPiBkeW5hbWljIChTdHJpbmcsIFN0cmlu
ZykgbWV0aG9kcyB0byBjcmVhdGUvYWNjZXNzIGNvdW50ZXJzLg0KPg0KPiBMb29raW5nIGF0IHRo
ZSBJZGVudGl0eVRhYmxlTWFwIGNsYXNzLCB0aGUgZm9sbG93aW5nIG1hcCgpIG1ldGhvZCBleGlz
dHMoKToNCj4gcHVibGljIHZvaWQgbWFwKEltbXV0YWJsZUJ5dGVzV3JpdGFibGUga2V5LA0KPiCg
IKAgoCCgIKAgoCCgIKBSb3dSZXN1bHQgdmFsdWUsDQo+IKAgoCCgIKAgoCCgIKAgoG9yZy5hcGFj
aGUuaGFkb29wLm1hcHJlZC5PdXRwdXRDb2xsZWN0b3I8SW1tdXRhYmxlQnl0ZXNXcml0YWJsZSxS
b3dSZXN1bHQ+IG91dHB1dCwNCj4goCCgIKAgoCCgIKAgoCCgb3JnLmFwYWNoZS5oYWRvb3AubWFw
cmVkLlJlcG9ydGVyIHJlcG9ydGVyKQ0KPiCgIKAgoCCgIHRocm93cyBJT0V4Y2VwdGlvblRoaXMg
bWV0aG9kIGlzIGRlcHJlY2F0ZWQgaW4gMjAuNQ0KPg0KPiBTbyBob3cgY2FuIHlvdSBnZXQgdGhl
IFJlcG9ydGVyIHNvIHlvdSBjYW4gdXNlIHRoZSAncmVwb3J0ZXIuaW5jckNvdW50ZXIoU3RyaW5n
IGdyb3VwLFN0cmluZyBjb3VudGVyLCBsb25nIGFtb3VudCknDQo+IG1ldGhvZC4gKFNlZSBwZyAy
MTQgY29kZSBmcmFnbWVudCkNCj4NCj4gSSBtdXN0IGJlIG1pc3Npbmcgc29tZXRoaW5nLg0KPg0K
PiBUaHgNCj4NCj4gLU1pa2UNCj4NCj4NCj4NCj4gX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18NCj4gVGhlIE5ldyBCdXN5IGlz
IG5vdCB0aGUgdG9vIGJ1c3kuIENvbWJpbmUgYWxsIHlvdXIgZS1tYWlsIGFjY291bnRzIHdpdGgg
SG90bWFpbC4NCj4gaHR0cDovL3d3dy53aW5kb3dzbGl2ZS5jb20vY2FtcGFpZ24vdGhlbmV3YnVz
eT90aWxlPW11bHRpYWNjb3VudCZvY2lkPVBJRDI4MzI2OjpUOldMTVRBR0w6T046V0w6ZW4tVVM6
V01fSE1QOjA0MjAxMF80DQoNCg0KVGhlIGluZm9ybWF0aW9uIGNvbnRhaW5lZCBpbiB0aGlzIGNv
bW11bmljYXRpb24gbWF5IGJlIENPTkZJREVOVElBTCBhbmQgaXMgaW50ZW5kZWQgb25seSBmb3Ig
dGhlIHVzZSBvZiB0aGUgcmVjaXBpZW50KHMpIG5hbWVkIGFib3ZlLiAgSWYgeW91IGFyZSBub3Qg
dGhlIGludGVuZGVkIHJlY2lwaWVudCwgeW91IGFyZSBoZXJlYnkgbm90aWZpZWQgdGhhdCBhbnkg
ZGlzc2VtaW5hdGlvbiwgZGlzdHJpYnV0aW9uLCBvciBjb3B5aW5nIG9mIHRoaXMgY29tbXVuaWNh
dGlvbiwgb3IgYW55IG9mIGl0cyBjb250ZW50cywgaXMgc3RyaWN0bHkgcHJvaGliaXRlZC4gIElm
IHlvdSBoYXZlIHJlY2VpdmVkIHRoaXMgY29tbXVuaWNhdGlvbiBpbiBlcnJvciwgcGxlYXNlIG5v
dGlmeSB0aGUgc2VuZGVyIGFuZCBkZWxldGUvZGVzdHJveSB0aGUgb3JpZ2luYWwgbWVzc2FnZSBh
bmQgYW55IGNvcHkgb2YgaXQgZnJvbSB5b3VyIGNvbXB1dGVyIG9yIHBhcGVyIGZpbGVzLg0K

From general-return-1839-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 13:02:01 2010
Return-Path: <general-return-1839-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14654 invoked from network); 20 Jul 2010 13:02:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 13:02:00 -0000
Received: (qmail 1894 invoked by uid 500); 20 Jul 2010 13:01:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1449 invoked by uid 500); 20 Jul 2010 13:01:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1441 invoked by uid 99); 20 Jul 2010 13:01:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 13:01:55 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 13:01:48 +0000
Received: by iwn37 with SMTP id 37so8352581iwn.35
        for <general@hadoop.apache.org>; Tue, 20 Jul 2010 06:01:26 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.166.139 with SMTP id m11mr7248209iby.136.1279630885237; 
	Tue, 20 Jul 2010 06:01:25 -0700 (PDT)
Received: by 10.231.184.9 with HTTP; Tue, 20 Jul 2010 06:01:25 -0700 (PDT)
In-Reply-To: <AANLkTinDkca4ABkityQiyu9GGX-NddzE0pMkAiIRN9J3@mail.gmail.com>
References: <5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F46C@sccorpmail01.mygazoo.com>
	<AANLkTilvuqho0l3U9ETg5HHT4Yz5uUkyjX_xqBvmh505@mail.gmail.com>
	<5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D051@sccorpmail01.mygazoo.com>
	<5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F4BA@sccorpmail01.mygazoo.com>
	<5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D053@sccorpmail01.mygazoo.com>
	<AANLkTilAzw5J1oL8raqTmPtxi9BGpuEwHXimEaUdfOzL@mail.gmail.com>
	<5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F5EB@sccorpmail01.mygazoo.com>
	<AANLkTinDkca4ABkityQiyu9GGX-NddzE0pMkAiIRN9J3@mail.gmail.com>
Date: Tue, 20 Jul 2010 06:01:25 -0700
Message-ID: <AANLkTimScTowrYvQNcC5-ccxwVlwRyWnOdQ4asBWjNUW@mail.gmail.com>
Subject: Re: Hadoop and XML
From: Jeff Bean <jwfbean@cloudera.com>
To: general@hadoop.apache.org, 
	Peter Minearo <Peter.Minearo@reardencommerce.com>
Content-Type: multipart/alternative; boundary=005045013c645bb619048bd148fc
X-Virus-Checked: Checked by ClamAV on apache.org

--005045013c645bb619048bd148fc
Content-Type: text/plain; charset=ISO-8859-1

I think the problem is here:

String valueString = new String(valueText.getBytes(), "UTF-8");

Javadoc for Text says:

*getBytes<http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/Text.html#getBytes%28%29>
*()
          Returns the raw bytes; however, only data up to
getLength()<http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/Text.html#getLength%28%29>is
valid.

So try getting the length, truncating the byte array at the value returned
by getLength() and THEN converting it to a String.

Jeff

On Mon, Jul 19, 2010 at 9:08 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> For your initial question on Text.set().
> Text.setCapacity() allocates new byte array. Since keepData is false, old
> data wouldn't be copied over.
>
> On Mon, Jul 19, 2010 at 8:01 AM, Peter Minearo <
> Peter.Minearo@reardencommerce.com> wrote:
>
> > I am already using XmlInputFormat.  The input into the Map phase is not
> > the problem.  The problem lays in between the Map and Reduce phase.
> >
> > BTW - The article is correct.  DO NOT USE StreamXmlRecordReader.
> > XmlInputFormat is a lot faster.  From my testing, StreamXmlRecordReader
> > took 8 minutes to read a 1 GB XML document; where as, XmlInputFormat was
> > under 2 minutes. (Using 2 Core, 8GB machines)
> >
> >
> > -----Original Message-----
> > From: Ted Yu [mailto:yuzhihong@gmail.com]
> > Sent: Friday, July 16, 2010 9:44 PM
> > To: general@hadoop.apache.org
> > Subject: Re: Hadoop and XML
> >
> > From an earlier post:
> > http://oobaloo.co.uk/articles/2010/1/20/processing-xml-in-hadoop.html
> >
> > On Fri, Jul 16, 2010 at 3:07 PM, Peter Minearo <
> > Peter.Minearo@reardencommerce.com> wrote:
> >
> > > Moving the variable to a local variable did not seem to work:
> > >
> > >
> > > </PrivateRateSet>vateRateSet>
> > >
> > >
> > >
> > > public void map(Object key, Object value, OutputCollector output,
> > > Reporter
> > > reporter) throws IOException {
> > >                Text valueText = (Text)value;
> > >                String valueString = new String(valueText.getBytes(),
> > > "UTF-8");
> > >                String keyString = getXmlKey(valueString);
> > >                 Text returnKeyText = new Text();
> > >                Text returnValueText = new Text();
> > >                returnKeyText.set(keyString);
> > >                returnValueText.set(valueString);
> > >                output.collect(returnKeyText, returnValueText); }
> > >
> > > -----Original Message-----
> > > From: Peter Minearo [mailto:Peter.Minearo@Reardencommerce.com]
> > > Sent: Fri 7/16/2010 2:51 PM
> > > To: general@hadoop.apache.org
> > > Subject: RE: Hadoop and XML
> > >
> > > Whoops....right after I sent it and someone else made a suggestion; I
> > > realized what question 2 was about.  I can try that, but wouldn't that
> >
> > > cause Object bloat?  During the Hadoop training I went through; it was
> >
> > > mentioned to reuse the returning Key and Value objects to keep the
> > > number of Objects created down to a minimum.  Is this not really a
> > > valid point?
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Peter Minearo [mailto:Peter.Minearo@Reardencommerce.com]
> > > Sent: Friday, July 16, 2010 2:44 PM
> > > To: general@hadoop.apache.org
> > > Subject: RE: Hadoop and XML
> > >
> > >
> > > I am not using multi-threaded Map tasks.  Also, if I understand your
> > > second question correctly:
> > > "Also can you try creating the output key and values in the map
> > > method(method lacal) ?"
> > > In the first code snippet I am doing exactly that.
> > >
> > > Below is the class that runs the Job.
> > >
> > > public class HadoopJobClient {
> > >
> > >        private static final Log LOGGER =
> > > LogFactory.getLog(Prds.class.getName());
> > >
> > >        public static void main(String[] args) {
> > >                JobConf conf = new JobConf(Prds.class);
> > >
> > >                conf.set("xmlinput.start", "<PrivateRateSet>");
> > >                conf.set("xmlinput.end", "</PrivateRateSet>");
> > >
> > >                conf.setJobName("PRDS Parse");
> > >
> > >                conf.setOutputKeyClass(Text.class);
> > >                conf.setOutputValueClass(Text.class);
> > >
> > >                conf.setMapperClass(PrdsMapper.class);
> > >                conf.setReducerClass(PrdsReducer.class);
> > >
> > >                conf.setInputFormat(XmlInputFormat.class);
> > >                conf.setOutputFormat(TextOutputFormat.class);
> > >
> > >                FileInputFormat.setInputPaths(conf, new Path(args[0]));
> > >                FileOutputFormat.setOutputPath(conf, new
> > > Path(args[1]));
> > >
> > >                // Run the job
> > >                try {
> > >                        JobClient.runJob(conf);
> > >                } catch (IOException e) {
> > >                        LOGGER.error(e.getMessage(), e);
> > >                }
> > >
> > >        }
> > >
> > >
> > > }
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Soumya Banerjee [mailto:soumya.sbanerjee@gmail.com]
> > > Sent: Fri 7/16/2010 2:29 PM
> > > To: general@hadoop.apache.org
> > > Subject: Re: Hadoop and XML
> > >
> > > Hi,
> > >
> > > Can you please share the code of the job submission client ?
> > >
> > > Also can you try creating the output key and values in the map
> > > method(method
> > > lacal) ?
> > > Make sure you are not using multi threaded map task configuration.
> > >
> > > map()
> > > {
> > > private Text keyText = new Text();
> > >  private Text valueText = new Text();
> > >
> > > //rest of the code
> > > }
> > >
> > > Soumya.
> > >
> > > On Sat, Jul 17, 2010 at 2:30 AM, Peter Minearo <
> > > Peter.Minearo@reardencommerce.com> wrote:
> > >
> > > > I have an XML file that has sparse data in it.  I am running a
> > > > MapReduce Job that reads in an XML file, pulls out a Key from within
> >
> > > > the XML snippet and then hands back the Key and the XML snippet (as
> > > > the Value) to the OutputCollector.  The reason is to sort the file
> > > back into order.
> > > > Below is the snippet of code.
> > > >
> > > > public class XmlMapper extends MapReduceBase implements Mapper {
> > > >
> > > >  private Text keyText = new Text();
> > > >  private Text valueText = new Text();
> > > >
> > > >  @SuppressWarnings("unchecked")
> > > >  public void map(Object key, Object value, OutputCollector output,
> > > > Reporter reporter) throws IOException {  Text valueText =
> > > > (Text)value;
> > >
> > > > String valueString = new String(valueText.getBytes(), "UTF-8");
> > > > String keyString = getXmlKey(valueString);
> > > > getKeyText().set(keyString);  getValueText().set(valueString);
> > > > output.collect(getKeyText(), getValueText());  }
> > > >
> > > >
> > > >  public Text getKeyText() {
> > > >  return keyText;
> > > >  }
> > > >
> > > >
> > > >  public void setKeyText(Text keyText) {  this.keyText = keyText;  }
> > > >
> > > >
> > > >  public Text getValueText() {
> > > >  return valueText;
> > > >  }
> > > >
> > > >
> > > >  public void setValueText(Text valueText) {  this.valueText =
> > > > valueText;  }
> > > >
> > > >
> > > >  private String getXmlKey(String value) {
> > > >        // Get the Key from the XML in the value.
> > > >  }
> > > >
> > > > }
> > > >
> > > > The XML snippet from the Value is fine when it is passed into the
> > > > map() method.  I am not changing any data either, just pulling out
> > > > information for the key.  The problem I am seeing is between the Map
> >
> > > > phase and the Reduce phase, the XML is getting munged.  For Example:
> > > >
> > > >  </PrivateRate>
> > > >  </PrivateRateSet>te>
> > > >
> > > > It is my understanding that Hadoop uses the same instance of the Key
> >
> > > > and Value object when calling the Map method.  What changes is the
> > > > data within those instances.  So, I ran an experiment where I do not
> >
> > > > have different Key or Value Text Objects.  I reuse the ones passed
> > > > into the method, like below:
> > > >
> > > > public class XmlMapper extends MapReduceBase implements Mapper {
> > > >
> > > >  @SuppressWarnings("unchecked")
> > > >  public void map(Object key, Object value, OutputCollector output,
> > > > Reporter reporter) throws IOException {  Text keyText = (Text)key;
> > > > Text valueText = (Text)value;  String valueString = new
> > > > String(valueText.getBytes(), "UTF-8");  String keyString =
> > > > getXmlKey(valueString);  keyText.set(keyString);
> > > > valueText.set(valueString);  output.collect(keyText, valueText);  }
> > > >
> > > >
> > > >  private String getXmlKey(String value) {
> > > >        // Get the Key from the XML in the value.
> > > >  }
> > > >
> > > > }
> > > >
> > > > What was interesting about this is the fact that the XML was getting
> >
> > > > munged within the Map Phase.  When I changed over to the code at the
> >
> > > > top, the Map phase was fine.  However, the Reduce phase picks up the
> >
> > > > munged XML.  Trying to debug the problem, I came across this method
> > > > in
> > >
> > > > the Text Object:
> > > >
> > > > public void set(byte[] utf8, int start, int len) {
> > > >    setCapacity(len, false);
> > > >    System.arraycopy(utf8, start, bytes, 0, len);
> > > >    this.length = len;
> > > > }
> > > >
> > > > If the "bytes" array had a length of 1000 and the "utf8" array has a
> >
> > > > length of 500; doing a System.arraycopy() would only copy the first
> > > > 500 from "utf8" to "bytes" but leave the last 500 in "bytes" alone.
> > > > Could this be the cause of the XML munging?
> > > >
> > > > All of this leads me to a few questions:
> > > >
> > > > 1) Has anyone successfully used XML snippets as the data format
> > > > within
> > >
> > > > a MapReduce job; not just reading from the file but used during the
> > > > shuffle?
> > > > 2) Is anyone seeing this problem with XML or any other format?
> > > > 3) Does anyone know what is going on?
> > > > 4) Is this a bug?
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > Peter
> > > >
> > > >
> > > >
> > >
> > >
> > >
> >
>

--005045013c645bb619048bd148fc--

From general-return-1840-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 14:53:31 2010
Return-Path: <general-return-1840-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50056 invoked from network); 20 Jul 2010 14:53:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 14:53:30 -0000
Received: (qmail 48589 invoked by uid 500); 20 Jul 2010 14:53:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48153 invoked by uid 500); 20 Jul 2010 14:53:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48135 invoked by uid 99); 20 Jul 2010 14:53:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 14:53:26 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of timrobertson100@gmail.com designates 209.85.213.176 as permitted sender)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 14:53:18 +0000
Received: by yxm34 with SMTP id 34so2118515yxm.35
        for <multiple recipients>; Tue, 20 Jul 2010 07:52:57 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=7uUw4miireLQdFD2Ykadz8Juiof+36npLrjTVTi27Cc=;
        b=cjOYxVXyOpnIuN+5Ed3aWQJxGA1jLbpT5GPzB2rM5zWcJoHI7OEaAZFXFjQLFjZTFg
         SgPj1Eh4gn+P/hkxVDCVd58N/zay1ZZ5b9bhKOjjeHiGuMK8dMRdXcfExDLSEjjDfRV4
         zpU7fDYahwMGt3Aw2LXUMtYzz/enHjWpbZu74=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=vXyrX24yG98Pzf/kQqdFRevU1BhF4XT4Kohv2LiBrQwLP0w1vSFFfOuRrCeMA1ILcT
         u2Y5jHLnn5iLSqujgZgGA7z+tnv/g6KZNh3qFrLK8ZQWuO+kVWY/olB+P31E+1Adu9B4
         IU8OQ8ewGIOclN8vpWRT8/IsgqSgLTm1Vg0iM=
MIME-Version: 1.0
Received: by 10.151.25.10 with SMTP id c10mr483719ybj.19.1279637577465; Tue, 
	20 Jul 2010 07:52:57 -0700 (PDT)
Received: by 10.150.197.21 with HTTP; Tue, 20 Jul 2010 07:52:57 -0700 (PDT)
Date: Tue, 20 Jul 2010 16:52:57 +0200
Message-ID: <AANLkTimiol0QLyISg4RP54t3GN5AsExoxbli0T7GXMe9@mail.gmail.com>
Subject: Developers sought for web crawling project
From: Tim Robertson <timrobertson100@gmail.com>
To: general@hadoop.apache.org, user@hbase.apache.org, 
	hive-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hi all,

Disclosure: I have been an active member of the Hadoop / HBase / Hive
mailing lists for some time.  I am not a recruiter, but looking to
increase a development team that I lead.  I sincerely apologize if
this message is against mailing list etiquette; I have not seen any
guidelines forbidding this.

We are looking to expand the development team based in Copenhagen,
Denmark.  We run and operate an indexing system that provides access
to biodiversity information, and specifically 100s millions of point
based observations of species occurrence documented in the past
century.  Our live system is outgrowing a MySQL database, and we are
already tentatively using Hive, Hadoop and have run some tests using
HBase.  Over the next 16 months we will rework the whole system,
including custom reporting, custom maps and other visualizations, real
time search indexes, reducing latency, quality control integration,
custom vocabulary work etc.  Our final architecture will likely
include Hive, Hadoop, HBase, Lucene (SOLR / ElasticSearch?), Sqoop and
PostGIS.  If you are familiar with these technologies, and the thought
of some time working in Europe appeals to you, then we'd love to hear
from you.  Guidelines for applications are on the advert:
http://tinyurl.com/gbif-dev-job

Thanks,
Tim

From general-return-1841-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 15:08:35 2010
Return-Path: <general-return-1841-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60215 invoked from network); 20 Jul 2010 15:08:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 15:08:34 -0000
Received: (qmail 75231 invoked by uid 500); 20 Jul 2010 15:08:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75093 invoked by uid 500); 20 Jul 2010 15:08:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75079 invoked by uid 99); 20 Jul 2010 15:08:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 15:08:32 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of paliwalashish@gmail.com designates 209.85.216.176 as permitted sender)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 15:08:27 +0000
Received: by qyk12 with SMTP id 12so3437110qyk.14
        for <multiple recipients>; Tue, 20 Jul 2010 08:07:06 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:cc:content-type
         :content-transfer-encoding;
        bh=LU1Ot8e3HTcOLO6R3xl+iAoOPTNRjkjl9DNE/Ke/vUg=;
        b=iGIDYPtcLq6jvS/Op0TEhwHhFIJkE1egiP7M3T+xBrQNEMJe6g4A77zKG107eqlo32
         2/4fawf5zui2EfDeTOc7xbnrfRfqFsVnLuIhAU+w0jJHOxj9JBj1IvdWEkme96KMH9D/
         IBWTjRfzadssx1Nvvq4/dnky9wt8AgSdHADVU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type:content-transfer-encoding;
        b=T70M9baLMUkpGO/T/sxhytw2YkA8JNT+8mYJpHdSa3tv0qOaWeW0LihamSR4+zgeo+
         clS3TqpM7oHC8UWw7lXfSB1t5Cr7YyZE5JCG1gDkg27zXz22DxxzP5E9PRltbzw/ZAFq
         +wXljihHO1EjOjBmnHLo3NJswhNcbmVhU4mnI=
MIME-Version: 1.0
Received: by 10.224.37.11 with SMTP id v11mr1027383qad.100.1279638425940; Tue, 
	20 Jul 2010 08:07:05 -0700 (PDT)
Received: by 10.229.80.74 with HTTP; Tue, 20 Jul 2010 08:07:05 -0700 (PDT)
In-Reply-To: <AANLkTimiol0QLyISg4RP54t3GN5AsExoxbli0T7GXMe9@mail.gmail.com>
References: <AANLkTimiol0QLyISg4RP54t3GN5AsExoxbli0T7GXMe9@mail.gmail.com>
Date: Tue, 20 Jul 2010 20:37:05 +0530
Message-ID: <AANLkTikd_54w62lU-B1plr1j1ygekpZgoJy7JpMgItDB@mail.gmail.com>
Subject: Re: Developers sought for web crawling project
From: Ashish <paliwalashish@gmail.com>
To: user@hbase.apache.org
Cc: general@hadoop.apache.org, hive-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Tim,

You can try jobs@apache.org, its meant for this purpose.

Sorry for the noise. I belong to Apache MINA team and just getting
started here :)

thanks
ashish

On Tue, Jul 20, 2010 at 8:22 PM, Tim Robertson
<timrobertson100@gmail.com> wrote:
> Hi all,
>
> Disclosure: I have been an active member of the Hadoop / HBase / Hive
> mailing lists for some time. =A0I am not a recruiter, but looking to
> increase a development team that I lead. =A0I sincerely apologize if
> this message is against mailing list etiquette; I have not seen any
> guidelines forbidding this.
>
> We are looking to expand the development team based in Copenhagen,
> Denmark. =A0We run and operate an indexing system that provides access
> to biodiversity information, and specifically 100s millions of point
> based observations of species occurrence documented in the past
> century. =A0Our live system is outgrowing a MySQL database, and we are
> already tentatively using Hive, Hadoop and have run some tests using
> HBase. =A0Over the next 16 months we will rework the whole system,
> including custom reporting, custom maps and other visualizations, real
> time search indexes, reducing latency, quality control integration,
> custom vocabulary work etc. =A0Our final architecture will likely
> include Hive, Hadoop, HBase, Lucene (SOLR / ElasticSearch?), Sqoop and
> PostGIS. =A0If you are familiar with these technologies, and the thought
> of some time working in Europe appeals to you, then we'd love to hear
> from you. =A0Guidelines for applications are on the advert:
> http://tinyurl.com/gbif-dev-job
>
> Thanks,
> Tim
>



--=20
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal

From general-return-1842-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 15:58:20 2010
Return-Path: <general-return-1842-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 75803 invoked from network); 20 Jul 2010 15:58:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 15:58:20 -0000
Received: (qmail 46114 invoked by uid 500); 20 Jul 2010 15:58:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46049 invoked by uid 500); 20 Jul 2010 15:58:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46041 invoked by uid 99); 20 Jul 2010 15:58:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 15:58:18 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 15:58:12 +0000
Received: by vws7 with SMTP id 7so3611254vws.35
        for <general@hadoop.apache.org>; Tue, 20 Jul 2010 08:56:50 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=2WIAlKILISOjTbqdljgotbH+fxY/hmCqcgriq/w88Fg=;
        b=F2q8AF2RifjU8anuW0u+By47Yp1aiFyIHxDhnvWkwIKYnnFZaw9ekOTAjvhzQ4DzBW
         bixFDZKn+Wpq3sOI95ahvX7VRw8tc9lZi9lS1UM2dzxd50vJ0iuyWVMkGlDpazgiCakm
         46eaarwMLPr6K8DrsfxMNaRJsEr70qMcXsVqo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=gfgCJn5Blk13XECfh1aZQQ1FzapueqSRNYrn3893ydRnHE7x0AGd0pNU2gKERfJygS
         7bpQe+X2xZrHkvG3pQRDam41M/Z79xbXgHRUrhpd1XRdZuSibl6nGqoNBYuUSuScip9r
         p4XiL+fJTWuSztzRygobrGnf3M1a5YVhQmpWg=
MIME-Version: 1.0
Received: by 10.224.29.16 with SMTP id o16mr2470901qac.343.1279641410532; Tue, 
	20 Jul 2010 08:56:50 -0700 (PDT)
Received: by 10.229.75.85 with HTTP; Tue, 20 Jul 2010 08:56:50 -0700 (PDT)
In-Reply-To: <AANLkTimScTowrYvQNcC5-ccxwVlwRyWnOdQ4asBWjNUW@mail.gmail.com>
References: <5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F46C@sccorpmail01.mygazoo.com>
	<AANLkTilvuqho0l3U9ETg5HHT4Yz5uUkyjX_xqBvmh505@mail.gmail.com>
	<5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D051@sccorpmail01.mygazoo.com>
	<5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F4BA@sccorpmail01.mygazoo.com>
	<5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D053@sccorpmail01.mygazoo.com>
	<AANLkTilAzw5J1oL8raqTmPtxi9BGpuEwHXimEaUdfOzL@mail.gmail.com>
	<5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F5EB@sccorpmail01.mygazoo.com>
	<AANLkTinDkca4ABkityQiyu9GGX-NddzE0pMkAiIRN9J3@mail.gmail.com>
	<AANLkTimScTowrYvQNcC5-ccxwVlwRyWnOdQ4asBWjNUW@mail.gmail.com>
Date: Tue, 20 Jul 2010 08:56:50 -0700
Message-ID: <AANLkTilCSmQsML8I5ayJZpIA7o0AJ5jb2-IMVxBu3QaN@mail.gmail.com>
Subject: Re: Hadoop and XML
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175ce054b6f510048bd3bbb5
X-Virus-Checked: Checked by ClamAV on apache.org

--0015175ce054b6f510048bd3bbb5
Content-Type: text/plain; charset=ISO-8859-1

Interesting.
String class is able to handle this scenario:

  348       public String(byte[] data, String encoding) throws
UnsupportedEncodingException {
  349           this(data, 0, data.length, encoding);
  350       }



On Tue, Jul 20, 2010 at 6:01 AM, Jeff Bean <jwfbean@cloudera.com> wrote:

> I think the problem is here:
>
> String valueString = new String(valueText.getBytes(), "UTF-8");
>
> Javadoc for Text says:
>
> *getBytes<
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/Text.html#getBytes%28%29
> >
> *()
>          Returns the raw bytes; however, only data up to
> getLength()<
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/Text.html#getLength%28%29
> >is
> valid.
>
> So try getting the length, truncating the byte array at the value returned
> by getLength() and THEN converting it to a String.
>
> Jeff
>
> On Mon, Jul 19, 2010 at 9:08 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > For your initial question on Text.set().
> > Text.setCapacity() allocates new byte array. Since keepData is false, old
> > data wouldn't be copied over.
> >
> > On Mon, Jul 19, 2010 at 8:01 AM, Peter Minearo <
> > Peter.Minearo@reardencommerce.com> wrote:
> >
> > > I am already using XmlInputFormat.  The input into the Map phase is not
> > > the problem.  The problem lays in between the Map and Reduce phase.
> > >
> > > BTW - The article is correct.  DO NOT USE StreamXmlRecordReader.
> > > XmlInputFormat is a lot faster.  From my testing, StreamXmlRecordReader
> > > took 8 minutes to read a 1 GB XML document; where as, XmlInputFormat
> was
> > > under 2 minutes. (Using 2 Core, 8GB machines)
> > >
> > >
> > > -----Original Message-----
> > > From: Ted Yu [mailto:yuzhihong@gmail.com]
> > > Sent: Friday, July 16, 2010 9:44 PM
> > > To: general@hadoop.apache.org
> > > Subject: Re: Hadoop and XML
> > >
> > > From an earlier post:
> > > http://oobaloo.co.uk/articles/2010/1/20/processing-xml-in-hadoop.html
> > >
> > > On Fri, Jul 16, 2010 at 3:07 PM, Peter Minearo <
> > > Peter.Minearo@reardencommerce.com> wrote:
> > >
> > > > Moving the variable to a local variable did not seem to work:
> > > >
> > > >
> > > > </PrivateRateSet>vateRateSet>
> > > >
> > > >
> > > >
> > > > public void map(Object key, Object value, OutputCollector output,
> > > > Reporter
> > > > reporter) throws IOException {
> > > >                Text valueText = (Text)value;
> > > >                String valueString = new String(valueText.getBytes(),
> > > > "UTF-8");
> > > >                String keyString = getXmlKey(valueString);
> > > >                 Text returnKeyText = new Text();
> > > >                Text returnValueText = new Text();
> > > >                returnKeyText.set(keyString);
> > > >                returnValueText.set(valueString);
> > > >                output.collect(returnKeyText, returnValueText); }
> > > >
> > > > -----Original Message-----
> > > > From: Peter Minearo [mailto:Peter.Minearo@Reardencommerce.com]
> > > > Sent: Fri 7/16/2010 2:51 PM
> > > > To: general@hadoop.apache.org
> > > > Subject: RE: Hadoop and XML
> > > >
> > > > Whoops....right after I sent it and someone else made a suggestion; I
> > > > realized what question 2 was about.  I can try that, but wouldn't
> that
> > >
> > > > cause Object bloat?  During the Hadoop training I went through; it
> was
> > >
> > > > mentioned to reuse the returning Key and Value objects to keep the
> > > > number of Objects created down to a minimum.  Is this not really a
> > > > valid point?
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Peter Minearo [mailto:Peter.Minearo@Reardencommerce.com]
> > > > Sent: Friday, July 16, 2010 2:44 PM
> > > > To: general@hadoop.apache.org
> > > > Subject: RE: Hadoop and XML
> > > >
> > > >
> > > > I am not using multi-threaded Map tasks.  Also, if I understand your
> > > > second question correctly:
> > > > "Also can you try creating the output key and values in the map
> > > > method(method lacal) ?"
> > > > In the first code snippet I am doing exactly that.
> > > >
> > > > Below is the class that runs the Job.
> > > >
> > > > public class HadoopJobClient {
> > > >
> > > >        private static final Log LOGGER =
> > > > LogFactory.getLog(Prds.class.getName());
> > > >
> > > >        public static void main(String[] args) {
> > > >                JobConf conf = new JobConf(Prds.class);
> > > >
> > > >                conf.set("xmlinput.start", "<PrivateRateSet>");
> > > >                conf.set("xmlinput.end", "</PrivateRateSet>");
> > > >
> > > >                conf.setJobName("PRDS Parse");
> > > >
> > > >                conf.setOutputKeyClass(Text.class);
> > > >                conf.setOutputValueClass(Text.class);
> > > >
> > > >                conf.setMapperClass(PrdsMapper.class);
> > > >                conf.setReducerClass(PrdsReducer.class);
> > > >
> > > >                conf.setInputFormat(XmlInputFormat.class);
> > > >                conf.setOutputFormat(TextOutputFormat.class);
> > > >
> > > >                FileInputFormat.setInputPaths(conf, new
> Path(args[0]));
> > > >                FileOutputFormat.setOutputPath(conf, new
> > > > Path(args[1]));
> > > >
> > > >                // Run the job
> > > >                try {
> > > >                        JobClient.runJob(conf);
> > > >                } catch (IOException e) {
> > > >                        LOGGER.error(e.getMessage(), e);
> > > >                }
> > > >
> > > >        }
> > > >
> > > >
> > > > }
> > > >
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Soumya Banerjee [mailto:soumya.sbanerjee@gmail.com]
> > > > Sent: Fri 7/16/2010 2:29 PM
> > > > To: general@hadoop.apache.org
> > > > Subject: Re: Hadoop and XML
> > > >
> > > > Hi,
> > > >
> > > > Can you please share the code of the job submission client ?
> > > >
> > > > Also can you try creating the output key and values in the map
> > > > method(method
> > > > lacal) ?
> > > > Make sure you are not using multi threaded map task configuration.
> > > >
> > > > map()
> > > > {
> > > > private Text keyText = new Text();
> > > >  private Text valueText = new Text();
> > > >
> > > > //rest of the code
> > > > }
> > > >
> > > > Soumya.
> > > >
> > > > On Sat, Jul 17, 2010 at 2:30 AM, Peter Minearo <
> > > > Peter.Minearo@reardencommerce.com> wrote:
> > > >
> > > > > I have an XML file that has sparse data in it.  I am running a
> > > > > MapReduce Job that reads in an XML file, pulls out a Key from
> within
> > >
> > > > > the XML snippet and then hands back the Key and the XML snippet (as
> > > > > the Value) to the OutputCollector.  The reason is to sort the file
> > > > back into order.
> > > > > Below is the snippet of code.
> > > > >
> > > > > public class XmlMapper extends MapReduceBase implements Mapper {
> > > > >
> > > > >  private Text keyText = new Text();
> > > > >  private Text valueText = new Text();
> > > > >
> > > > >  @SuppressWarnings("unchecked")
> > > > >  public void map(Object key, Object value, OutputCollector output,
> > > > > Reporter reporter) throws IOException {  Text valueText =
> > > > > (Text)value;
> > > >
> > > > > String valueString = new String(valueText.getBytes(), "UTF-8");
> > > > > String keyString = getXmlKey(valueString);
> > > > > getKeyText().set(keyString);  getValueText().set(valueString);
> > > > > output.collect(getKeyText(), getValueText());  }
> > > > >
> > > > >
> > > > >  public Text getKeyText() {
> > > > >  return keyText;
> > > > >  }
> > > > >
> > > > >
> > > > >  public void setKeyText(Text keyText) {  this.keyText = keyText;  }
> > > > >
> > > > >
> > > > >  public Text getValueText() {
> > > > >  return valueText;
> > > > >  }
> > > > >
> > > > >
> > > > >  public void setValueText(Text valueText) {  this.valueText =
> > > > > valueText;  }
> > > > >
> > > > >
> > > > >  private String getXmlKey(String value) {
> > > > >        // Get the Key from the XML in the value.
> > > > >  }
> > > > >
> > > > > }
> > > > >
> > > > > The XML snippet from the Value is fine when it is passed into the
> > > > > map() method.  I am not changing any data either, just pulling out
> > > > > information for the key.  The problem I am seeing is between the
> Map
> > >
> > > > > phase and the Reduce phase, the XML is getting munged.  For
> Example:
> > > > >
> > > > >  </PrivateRate>
> > > > >  </PrivateRateSet>te>
> > > > >
> > > > > It is my understanding that Hadoop uses the same instance of the
> Key
> > >
> > > > > and Value object when calling the Map method.  What changes is the
> > > > > data within those instances.  So, I ran an experiment where I do
> not
> > >
> > > > > have different Key or Value Text Objects.  I reuse the ones passed
> > > > > into the method, like below:
> > > > >
> > > > > public class XmlMapper extends MapReduceBase implements Mapper {
> > > > >
> > > > >  @SuppressWarnings("unchecked")
> > > > >  public void map(Object key, Object value, OutputCollector output,
> > > > > Reporter reporter) throws IOException {  Text keyText = (Text)key;
> > > > > Text valueText = (Text)value;  String valueString = new
> > > > > String(valueText.getBytes(), "UTF-8");  String keyString =
> > > > > getXmlKey(valueString);  keyText.set(keyString);
> > > > > valueText.set(valueString);  output.collect(keyText, valueText);  }
> > > > >
> > > > >
> > > > >  private String getXmlKey(String value) {
> > > > >        // Get the Key from the XML in the value.
> > > > >  }
> > > > >
> > > > > }
> > > > >
> > > > > What was interesting about this is the fact that the XML was
> getting
> > >
> > > > > munged within the Map Phase.  When I changed over to the code at
> the
> > >
> > > > > top, the Map phase was fine.  However, the Reduce phase picks up
> the
> > >
> > > > > munged XML.  Trying to debug the problem, I came across this method
> > > > > in
> > > >
> > > > > the Text Object:
> > > > >
> > > > > public void set(byte[] utf8, int start, int len) {
> > > > >    setCapacity(len, false);
> > > > >    System.arraycopy(utf8, start, bytes, 0, len);
> > > > >    this.length = len;
> > > > > }
> > > > >
> > > > > If the "bytes" array had a length of 1000 and the "utf8" array has
> a
> > >
> > > > > length of 500; doing a System.arraycopy() would only copy the first
> > > > > 500 from "utf8" to "bytes" but leave the last 500 in "bytes" alone.
> > > > > Could this be the cause of the XML munging?
> > > > >
> > > > > All of this leads me to a few questions:
> > > > >
> > > > > 1) Has anyone successfully used XML snippets as the data format
> > > > > within
> > > >
> > > > > a MapReduce job; not just reading from the file but used during the
> > > > > shuffle?
> > > > > 2) Is anyone seeing this problem with XML or any other format?
> > > > > 3) Does anyone know what is going on?
> > > > > 4) Is this a bug?
> > > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Peter
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> >
>

--0015175ce054b6f510048bd3bbb5--

From general-return-1843-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 16:24:42 2010
Return-Path: <general-return-1843-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87708 invoked from network); 20 Jul 2010 16:24:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 16:24:42 -0000
Received: (qmail 94825 invoked by uid 500); 20 Jul 2010 16:24:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94592 invoked by uid 500); 20 Jul 2010 16:24:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94584 invoked by uid 99); 20 Jul 2010 16:24:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 16:24:40 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 16:24:33 +0000
Received: by iwn37 with SMTP id 37so8515629iwn.35
        for <general@hadoop.apache.org>; Tue, 20 Jul 2010 09:23:12 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.170.144 with SMTP id d16mr7694043ibz.160.1279642992094; 
	Tue, 20 Jul 2010 09:23:12 -0700 (PDT)
Received: by 10.231.184.9 with HTTP; Tue, 20 Jul 2010 09:23:11 -0700 (PDT)
In-Reply-To: <AANLkTilCSmQsML8I5ayJZpIA7o0AJ5jb2-IMVxBu3QaN@mail.gmail.com>
References: <5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F46C@sccorpmail01.mygazoo.com>
	<AANLkTilvuqho0l3U9ETg5HHT4Yz5uUkyjX_xqBvmh505@mail.gmail.com>
	<5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D051@sccorpmail01.mygazoo.com>
	<5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F4BA@sccorpmail01.mygazoo.com>
	<5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D053@sccorpmail01.mygazoo.com>
	<AANLkTilAzw5J1oL8raqTmPtxi9BGpuEwHXimEaUdfOzL@mail.gmail.com>
	<5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F5EB@sccorpmail01.mygazoo.com>
	<AANLkTinDkca4ABkityQiyu9GGX-NddzE0pMkAiIRN9J3@mail.gmail.com>
	<AANLkTimScTowrYvQNcC5-ccxwVlwRyWnOdQ4asBWjNUW@mail.gmail.com>
	<AANLkTilCSmQsML8I5ayJZpIA7o0AJ5jb2-IMVxBu3QaN@mail.gmail.com>
Date: Tue, 20 Jul 2010 09:23:11 -0700
Message-ID: <AANLkTikXmgzQyd_fuD3EN-nvjK-FP60wDqtTn7Z4xcwB@mail.gmail.com>
Subject: Re: Hadoop and XML
From: Jeff Bean <jwfbean@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=005045029894fbb717048bd41979
X-Virus-Checked: Checked by ClamAV on apache.org

--005045029894fbb717048bd41979
Content-Type: text/plain; charset=ISO-8859-1

data.length is the length of the byte array.

Text.getLength() most likely returns a different value than getBytes.length.

Hadoop reuses box class objects like Text, so what it's probably doing is
writing over the byte array, lengthening it as necessary, and just updating
a separate length attribute.

Jeff

On Tue, Jul 20, 2010 at 8:56 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> Interesting.
> String class is able to handle this scenario:
>
>  348       public String(byte[] data, String encoding) throws
> UnsupportedEncodingException {
>  349           this(data, 0, data.length, encoding);
>  350       }
>
>
>
> On Tue, Jul 20, 2010 at 6:01 AM, Jeff Bean <jwfbean@cloudera.com> wrote:
>
> > I think the problem is here:
> >
> > String valueString = new String(valueText.getBytes(), "UTF-8");
> >
> > Javadoc for Text says:
> >
> > *getBytes<
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/Text.html#getBytes%28%29
> > >
> > *()
> >          Returns the raw bytes; however, only data up to
> > getLength()<
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/Text.html#getLength%28%29
> > >is
> > valid.
> >
> > So try getting the length, truncating the byte array at the value
> returned
> > by getLength() and THEN converting it to a String.
> >
> > Jeff
> >
> > On Mon, Jul 19, 2010 at 9:08 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> >
> > > For your initial question on Text.set().
> > > Text.setCapacity() allocates new byte array. Since keepData is false,
> old
> > > data wouldn't be copied over.
> > >
> > > On Mon, Jul 19, 2010 at 8:01 AM, Peter Minearo <
> > > Peter.Minearo@reardencommerce.com> wrote:
> > >
> > > > I am already using XmlInputFormat.  The input into the Map phase is
> not
> > > > the problem.  The problem lays in between the Map and Reduce phase.
> > > >
> > > > BTW - The article is correct.  DO NOT USE StreamXmlRecordReader.
> > > > XmlInputFormat is a lot faster.  From my testing,
> StreamXmlRecordReader
> > > > took 8 minutes to read a 1 GB XML document; where as, XmlInputFormat
> > was
> > > > under 2 minutes. (Using 2 Core, 8GB machines)
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Ted Yu [mailto:yuzhihong@gmail.com]
> > > > Sent: Friday, July 16, 2010 9:44 PM
> > > > To: general@hadoop.apache.org
> > > > Subject: Re: Hadoop and XML
> > > >
> > > > From an earlier post:
> > > >
> http://oobaloo.co.uk/articles/2010/1/20/processing-xml-in-hadoop.html
> > > >
> > > > On Fri, Jul 16, 2010 at 3:07 PM, Peter Minearo <
> > > > Peter.Minearo@reardencommerce.com> wrote:
> > > >
> > > > > Moving the variable to a local variable did not seem to work:
> > > > >
> > > > >
> > > > > </PrivateRateSet>vateRateSet>
> > > > >
> > > > >
> > > > >
> > > > > public void map(Object key, Object value, OutputCollector output,
> > > > > Reporter
> > > > > reporter) throws IOException {
> > > > >                Text valueText = (Text)value;
> > > > >                String valueString = new
> String(valueText.getBytes(),
> > > > > "UTF-8");
> > > > >                String keyString = getXmlKey(valueString);
> > > > >                 Text returnKeyText = new Text();
> > > > >                Text returnValueText = new Text();
> > > > >                returnKeyText.set(keyString);
> > > > >                returnValueText.set(valueString);
> > > > >                output.collect(returnKeyText, returnValueText); }
> > > > >
> > > > > -----Original Message-----
> > > > > From: Peter Minearo [mailto:Peter.Minearo@Reardencommerce.com]
> > > > > Sent: Fri 7/16/2010 2:51 PM
> > > > > To: general@hadoop.apache.org
> > > > > Subject: RE: Hadoop and XML
> > > > >
> > > > > Whoops....right after I sent it and someone else made a suggestion;
> I
> > > > > realized what question 2 was about.  I can try that, but wouldn't
> > that
> > > >
> > > > > cause Object bloat?  During the Hadoop training I went through; it
> > was
> > > >
> > > > > mentioned to reuse the returning Key and Value objects to keep the
> > > > > number of Objects created down to a minimum.  Is this not really a
> > > > > valid point?
> > > > >
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Peter Minearo [mailto:Peter.Minearo@Reardencommerce.com]
> > > > > Sent: Friday, July 16, 2010 2:44 PM
> > > > > To: general@hadoop.apache.org
> > > > > Subject: RE: Hadoop and XML
> > > > >
> > > > >
> > > > > I am not using multi-threaded Map tasks.  Also, if I understand
> your
> > > > > second question correctly:
> > > > > "Also can you try creating the output key and values in the map
> > > > > method(method lacal) ?"
> > > > > In the first code snippet I am doing exactly that.
> > > > >
> > > > > Below is the class that runs the Job.
> > > > >
> > > > > public class HadoopJobClient {
> > > > >
> > > > >        private static final Log LOGGER =
> > > > > LogFactory.getLog(Prds.class.getName());
> > > > >
> > > > >        public static void main(String[] args) {
> > > > >                JobConf conf = new JobConf(Prds.class);
> > > > >
> > > > >                conf.set("xmlinput.start", "<PrivateRateSet>");
> > > > >                conf.set("xmlinput.end", "</PrivateRateSet>");
> > > > >
> > > > >                conf.setJobName("PRDS Parse");
> > > > >
> > > > >                conf.setOutputKeyClass(Text.class);
> > > > >                conf.setOutputValueClass(Text.class);
> > > > >
> > > > >                conf.setMapperClass(PrdsMapper.class);
> > > > >                conf.setReducerClass(PrdsReducer.class);
> > > > >
> > > > >                conf.setInputFormat(XmlInputFormat.class);
> > > > >                conf.setOutputFormat(TextOutputFormat.class);
> > > > >
> > > > >                FileInputFormat.setInputPaths(conf, new
> > Path(args[0]));
> > > > >                FileOutputFormat.setOutputPath(conf, new
> > > > > Path(args[1]));
> > > > >
> > > > >                // Run the job
> > > > >                try {
> > > > >                        JobClient.runJob(conf);
> > > > >                } catch (IOException e) {
> > > > >                        LOGGER.error(e.getMessage(), e);
> > > > >                }
> > > > >
> > > > >        }
> > > > >
> > > > >
> > > > > }
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Soumya Banerjee [mailto:soumya.sbanerjee@gmail.com]
> > > > > Sent: Fri 7/16/2010 2:29 PM
> > > > > To: general@hadoop.apache.org
> > > > > Subject: Re: Hadoop and XML
> > > > >
> > > > > Hi,
> > > > >
> > > > > Can you please share the code of the job submission client ?
> > > > >
> > > > > Also can you try creating the output key and values in the map
> > > > > method(method
> > > > > lacal) ?
> > > > > Make sure you are not using multi threaded map task configuration.
> > > > >
> > > > > map()
> > > > > {
> > > > > private Text keyText = new Text();
> > > > >  private Text valueText = new Text();
> > > > >
> > > > > //rest of the code
> > > > > }
> > > > >
> > > > > Soumya.
> > > > >
> > > > > On Sat, Jul 17, 2010 at 2:30 AM, Peter Minearo <
> > > > > Peter.Minearo@reardencommerce.com> wrote:
> > > > >
> > > > > > I have an XML file that has sparse data in it.  I am running a
> > > > > > MapReduce Job that reads in an XML file, pulls out a Key from
> > within
> > > >
> > > > > > the XML snippet and then hands back the Key and the XML snippet
> (as
> > > > > > the Value) to the OutputCollector.  The reason is to sort the
> file
> > > > > back into order.
> > > > > > Below is the snippet of code.
> > > > > >
> > > > > > public class XmlMapper extends MapReduceBase implements Mapper {
> > > > > >
> > > > > >  private Text keyText = new Text();
> > > > > >  private Text valueText = new Text();
> > > > > >
> > > > > >  @SuppressWarnings("unchecked")
> > > > > >  public void map(Object key, Object value, OutputCollector
> output,
> > > > > > Reporter reporter) throws IOException {  Text valueText =
> > > > > > (Text)value;
> > > > >
> > > > > > String valueString = new String(valueText.getBytes(), "UTF-8");
> > > > > > String keyString = getXmlKey(valueString);
> > > > > > getKeyText().set(keyString);  getValueText().set(valueString);
> > > > > > output.collect(getKeyText(), getValueText());  }
> > > > > >
> > > > > >
> > > > > >  public Text getKeyText() {
> > > > > >  return keyText;
> > > > > >  }
> > > > > >
> > > > > >
> > > > > >  public void setKeyText(Text keyText) {  this.keyText = keyText;
>  }
> > > > > >
> > > > > >
> > > > > >  public Text getValueText() {
> > > > > >  return valueText;
> > > > > >  }
> > > > > >
> > > > > >
> > > > > >  public void setValueText(Text valueText) {  this.valueText =
> > > > > > valueText;  }
> > > > > >
> > > > > >
> > > > > >  private String getXmlKey(String value) {
> > > > > >        // Get the Key from the XML in the value.
> > > > > >  }
> > > > > >
> > > > > > }
> > > > > >
> > > > > > The XML snippet from the Value is fine when it is passed into the
> > > > > > map() method.  I am not changing any data either, just pulling
> out
> > > > > > information for the key.  The problem I am seeing is between the
> > Map
> > > >
> > > > > > phase and the Reduce phase, the XML is getting munged.  For
> > Example:
> > > > > >
> > > > > >  </PrivateRate>
> > > > > >  </PrivateRateSet>te>
> > > > > >
> > > > > > It is my understanding that Hadoop uses the same instance of the
> > Key
> > > >
> > > > > > and Value object when calling the Map method.  What changes is
> the
> > > > > > data within those instances.  So, I ran an experiment where I do
> > not
> > > >
> > > > > > have different Key or Value Text Objects.  I reuse the ones
> passed
> > > > > > into the method, like below:
> > > > > >
> > > > > > public class XmlMapper extends MapReduceBase implements Mapper {
> > > > > >
> > > > > >  @SuppressWarnings("unchecked")
> > > > > >  public void map(Object key, Object value, OutputCollector
> output,
> > > > > > Reporter reporter) throws IOException {  Text keyText =
> (Text)key;
> > > > > > Text valueText = (Text)value;  String valueString = new
> > > > > > String(valueText.getBytes(), "UTF-8");  String keyString =
> > > > > > getXmlKey(valueString);  keyText.set(keyString);
> > > > > > valueText.set(valueString);  output.collect(keyText, valueText);
>  }
> > > > > >
> > > > > >
> > > > > >  private String getXmlKey(String value) {
> > > > > >        // Get the Key from the XML in the value.
> > > > > >  }
> > > > > >
> > > > > > }
> > > > > >
> > > > > > What was interesting about this is the fact that the XML was
> > getting
> > > >
> > > > > > munged within the Map Phase.  When I changed over to the code at
> > the
> > > >
> > > > > > top, the Map phase was fine.  However, the Reduce phase picks up
> > the
> > > >
> > > > > > munged XML.  Trying to debug the problem, I came across this
> method
> > > > > > in
> > > > >
> > > > > > the Text Object:
> > > > > >
> > > > > > public void set(byte[] utf8, int start, int len) {
> > > > > >    setCapacity(len, false);
> > > > > >    System.arraycopy(utf8, start, bytes, 0, len);
> > > > > >    this.length = len;
> > > > > > }
> > > > > >
> > > > > > If the "bytes" array had a length of 1000 and the "utf8" array
> has
> > a
> > > >
> > > > > > length of 500; doing a System.arraycopy() would only copy the
> first
> > > > > > 500 from "utf8" to "bytes" but leave the last 500 in "bytes"
> alone.
> > > > > > Could this be the cause of the XML munging?
> > > > > >
> > > > > > All of this leads me to a few questions:
> > > > > >
> > > > > > 1) Has anyone successfully used XML snippets as the data format
> > > > > > within
> > > > >
> > > > > > a MapReduce job; not just reading from the file but used during
> the
> > > > > > shuffle?
> > > > > > 2) Is anyone seeing this problem with XML or any other format?
> > > > > > 3) Does anyone know what is going on?
> > > > > > 4) Is this a bug?
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Peter
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

--005045029894fbb717048bd41979--

From general-return-1844-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 16:35:51 2010
Return-Path: <general-return-1844-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91211 invoked from network); 20 Jul 2010 16:35:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 16:35:51 -0000
Received: (qmail 15824 invoked by uid 500); 20 Jul 2010 16:35:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15763 invoked by uid 500); 20 Jul 2010 16:35:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15727 invoked by uid 99); 20 Jul 2010 16:35:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 16:35:49 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of Peter.Minearo@reardencommerce.com designates 64.41.141.18 as permitted sender)
Received: from [64.41.141.18] (HELO barracuda.mygazoo.com) (64.41.141.18)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 16:35:42 +0000
X-ASG-Debug-ID: 1279643713-72cf00050002-IjwG86
Received: from sccorpmail01.mygazoo.com ([10.38.20.8]) by barracuda.mygazoo.com with ESMTP id pMc3HBqKIx32FxoD for <general@hadoop.apache.org>; Tue, 20 Jul 2010 09:35:14 -0700 (PDT)
X-Barracuda-Envelope-From: Peter.Minearo@Reardencommerce.com
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/mixed;
	boundary="----_=_NextPart_001_01CB2829.83EDDDB8"
X-ASG-Orig-Subj: RE: Hadoop and XML
Subject: RE: Hadoop and XML
Date: Tue, 20 Jul 2010 09:35:08 -0700
Message-ID: <5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D055@sccorpmail01.mygazoo.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: <5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D055@sccorpmail01.mygazoo.com>
Thread-Topic: Hadoop and XML
Thread-Index: AcsoKBBnjGgnGBwyQKK4n7mbbjMtPwAAHghL
References: <5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F46C@sccorpmail01.mygazoo.com><AANLkTilvuqho0l3U9ETg5HHT4Yz5uUkyjX_xqBvmh505@mail.gmail.com><5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D051@sccorpmail01.mygazoo.com><5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F4BA@sccorpmail01.mygazoo.com><5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D053@sccorpmail01.mygazoo.com><AANLkTilAzw5J1oL8raqTmPtxi9BGpuEwHXimEaUdfOzL@mail.gmail.com><5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F5EB@sccorpmail01.mygazoo.com><AANLkTinDkca4ABkityQiyu9GGX-NddzE0pMkAiIRN9J3@mail.gmail.com><AANLkTimScTowrYvQNcC5-ccxwVlwRyWnOdQ4asBWjNUW@mail.gmail.com><AANLkTilCSmQsML8I5ayJZpIA7o0AJ5jb2-IMVxBu3QaN@mail.gmail.com> <AANLkTikXmgzQyd_fuD3EN-nvjK-FP60wDqtTn7Z4xcwB@mail.gmail.com>
From: "Peter Minearo" <Peter.Minearo@Reardencommerce.com>
To: <general@hadoop.apache.org>
X-Barracuda-Connect: UNKNOWN[10.38.20.8]
X-Barracuda-Start-Time: 1279643714
X-Barracuda-URL: http://10.38.16.14:8000/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at mygazoo.com
X-Virus-Checked: Checked by ClamAV on apache.org

------_=_NextPart_001_01CB2829.83EDDDB8
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

That is exacly what is happening.  This is the code from the Text class.

public void set(String string) {
    try {
      ByteBuffer bb =3D encode(string, true);
      bytes =3D bb.array();
      length =3D bb.limit();
    }catch(CharacterCodingException e) {
      throw new RuntimeException("Should not have happened " + =
e.toString());=20
    }
  }


This sounds like a bug. =20

Let's say you create a Text object and drop in a String that sets the =
byte array length to 200.  Then drop in a a second String that sets the =
byte array length to 500.  Since, the new length is greater than the =
previous length; the byte array length is reset to the longer length.  =
Now, if you drop in a third String that would set the byte array length =
to 350; the Text object does not replace the byte array with a new =
length of 350; it utilizes the greater length of 500 and sets an extra =
variable to track the "real" length. =20

So: Text.getBytes().length !=3D Text.getLength()

This does 2 things:

1. Passes around more data than what is needed
2. Makes the Text object confusing to work with

Text.getBytes().length =3D=3D Text.getLength() - should be the correct =
behavior.



-----Original Message-----
From: Jeff Bean [mailto:jwfbean@cloudera.com]
Sent: Tue 7/20/2010 9:23 AM
To: general@hadoop.apache.org
Subject: Re: Hadoop and XML
=20
data.length is the length of the byte array.

Text.getLength() most likely returns a different value than =
getBytes.length.

Hadoop reuses box class objects like Text, so what it's probably doing =
is
writing over the byte array, lengthening it as necessary, and just =
updating
a separate length attribute.

Jeff

On Tue, Jul 20, 2010 at 8:56 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> Interesting.
> String class is able to handle this scenario:
>
>  348       public String(byte[] data, String encoding) throws
> UnsupportedEncodingException {
>  349           this(data, 0, data.length, encoding);
>  350       }
>
>
>
> On Tue, Jul 20, 2010 at 6:01 AM, Jeff Bean <jwfbean@cloudera.com> =
wrote:
>
> > I think the problem is here:
> >
> > String valueString =3D new String(valueText.getBytes(), "UTF-8");
> >
> > Javadoc for Text says:
> >
> > *getBytes<
> >
> =
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/Tex=
t.html#getBytes%28%29
> > >
> > *()
> >          Returns the raw bytes; however, only data up to
> > getLength()<
> >
> =
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/Tex=
t.html#getLength%28%29
> > >is
> > valid.
> >
> > So try getting the length, truncating the byte array at the value
> returned
> > by getLength() and THEN converting it to a String.
> >
> > Jeff
> >
> > On Mon, Jul 19, 2010 at 9:08 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> >
> > > For your initial question on Text.set().
> > > Text.setCapacity() allocates new byte array. Since keepData is =
false,
> old
> > > data wouldn't be copied over.
> > >
> > > On Mon, Jul 19, 2010 at 8:01 AM, Peter Minearo <
> > > Peter.Minearo@reardencommerce.com> wrote:
> > >
> > > > I am already using XmlInputFormat.  The input into the Map phase =
is
> not
> > > > the problem.  The problem lays in between the Map and Reduce =
phase.
> > > >
> > > > BTW - The article is correct.  DO NOT USE StreamXmlRecordReader.
> > > > XmlInputFormat is a lot faster.  From my testing,
> StreamXmlRecordReader
> > > > took 8 minutes to read a 1 GB XML document; where as, =
XmlInputFormat
> > was
> > > > under 2 minutes. (Using 2 Core, 8GB machines)
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Ted Yu [mailto:yuzhihong@gmail.com]
> > > > Sent: Friday, July 16, 2010 9:44 PM
> > > > To: general@hadoop.apache.org
> > > > Subject: Re: Hadoop and XML
> > > >
> > > > From an earlier post:
> > > >
> http://oobaloo.co.uk/articles/2010/1/20/processing-xml-in-hadoop.html
> > > >
> > > > On Fri, Jul 16, 2010 at 3:07 PM, Peter Minearo <
> > > > Peter.Minearo@reardencommerce.com> wrote:
> > > >
> > > > > Moving the variable to a local variable did not seem to work:
> > > > >
> > > > >
> > > > > </PrivateRateSet>vateRateSet>
> > > > >
> > > > >
> > > > >
> > > > > public void map(Object key, Object value, OutputCollector =
output,
> > > > > Reporter
> > > > > reporter) throws IOException {
> > > > >                Text valueText =3D (Text)value;
> > > > >                String valueString =3D new
> String(valueText.getBytes(),
> > > > > "UTF-8");
> > > > >                String keyString =3D getXmlKey(valueString);
> > > > >                 Text returnKeyText =3D new Text();
> > > > >                Text returnValueText =3D new Text();
> > > > >                returnKeyText.set(keyString);
> > > > >                returnValueText.set(valueString);
> > > > >                output.collect(returnKeyText, returnValueText); =
}
> > > > >
> > > > > -----Original Message-----
> > > > > From: Peter Minearo [mailto:Peter.Minearo@Reardencommerce.com]
> > > > > Sent: Fri 7/16/2010 2:51 PM
> > > > > To: general@hadoop.apache.org
> > > > > Subject: RE: Hadoop and XML
> > > > >
> > > > > Whoops....right after I sent it and someone else made a =
suggestion;
> I
> > > > > realized what question 2 was about.  I can try that, but =
wouldn't
> > that
> > > >
> > > > > cause Object bloat?  During the Hadoop training I went =
through; it
> > was
> > > >
> > > > > mentioned to reuse the returning Key and Value objects to keep =
the
> > > > > number of Objects created down to a minimum.  Is this not =
really a
> > > > > valid point?
> > > > >
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Peter Minearo [mailto:Peter.Minearo@Reardencommerce.com]
> > > > > Sent: Friday, July 16, 2010 2:44 PM
> > > > > To: general@hadoop.apache.org
> > > > > Subject: RE: Hadoop and XML
> > > > >
> > > > >
> > > > > I am not using multi-threaded Map tasks.  Also, if I =
understand
> your
> > > > > second question correctly:
> > > > > "Also can you try creating the output key and values in the =
map
> > > > > method(method lacal) ?"
> > > > > In the first code snippet I am doing exactly that.
> > > > >
> > > > > Below is the class that runs the Job.
> > > > >
> > > > > public class HadoopJobClient {
> > > > >
> > > > >        private static final Log LOGGER =3D
> > > > > LogFactory.getLog(Prds.class.getName());
> > > > >
> > > > >        public static void main(String[] args) {
> > > > >                JobConf conf =3D new JobConf(Prds.class);
> > > > >
> > > > >                conf.set("xmlinput.start", "<PrivateRateSet>");
> > > > >                conf.set("xmlinput.end", "</PrivateRateSet>");
> > > > >
> > > > >                conf.setJobName("PRDS Parse");
> > > > >
> > > > >                conf.setOutputKeyClass(Text.class);
> > > > >                conf.setOutputValueClass(Text.class);
> > > > >
> > > > >                conf.setMapperClass(PrdsMapper.class);
> > > > >                conf.setReducerClass(PrdsReducer.class);
> > > > >
> > > > >                conf.setInputFormat(XmlInputFormat.class);
> > > > >                conf.setOutputFormat(TextOutputFormat.class);
> > > > >
> > > > >                FileInputFormat.setInputPaths(conf, new
> > Path(args[0]));
> > > > >                FileOutputFormat.setOutputPath(conf, new
> > > > > Path(args[1]));
> > > > >
> > > > >                // Run the job
> > > > >                try {
> > > > >                        JobClient.runJob(conf);
> > > > >                } catch (IOException e) {
> > > > >                        LOGGER.error(e.getMessage(), e);
> > > > >                }
> > > > >
> > > > >        }
> > > > >
> > > > >
> > > > > }
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Soumya Banerjee [mailto:soumya.sbanerjee@gmail.com]
> > > > > Sent: Fri 7/16/2010 2:29 PM
> > > > > To: general@hadoop.apache.org
> > > > > Subject: Re: Hadoop and XML
> > > > >
> > > > > Hi,
> > > > >
> > > > > Can you please share the code of the job submission client ?
> > > > >
> > > > > Also can you try creating the output key and values in the map
> > > > > method(method
> > > > > lacal) ?
> > > > > Make sure you are not using multi threaded map task =
configuration.
> > > > >
> > > > > map()
> > > > > {
> > > > > private Text keyText =3D new Text();
> > > > >  private Text valueText =3D new Text();
> > > > >
> > > > > //rest of the code
> > > > > }
> > > > >
> > > > > Soumya.
> > > > >
> > > > > On Sat, Jul 17, 2010 at 2:30 AM, Peter Minearo <
> > > > > Peter.Minearo@reardencommerce.com> wrote:
> > > > >
> > > > > > I have an XML file that has sparse data in it.  I am running =
a
> > > > > > MapReduce Job that reads in an XML file, pulls out a Key =
from
> > within
> > > >
> > > > > > the XML snippet and then hands back the Key and the XML =
snippet
> (as
> > > > > > the Value) to the OutputCollector.  The reason is to sort =
the
> file
> > > > > back into order.
> > > > > > Below is the snippet of code.
> > > > > >
> > > > > > public class XmlMapper extends MapReduceBase implements =
Mapper {
> > > > > >
> > > > > >  private Text keyText =3D new Text();
> > > > > >  private Text valueText =3D new Text();
> > > > > >
> > > > > >  @SuppressWarnings("unchecked")
> > > > > >  public void map(Object key, Object value, OutputCollector
> output,
> > > > > > Reporter reporter) throws IOException {  Text valueText =3D
> > > > > > (Text)value;
> > > > >
> > > > > > String valueString =3D new String(valueText.getBytes(), =
"UTF-8");
> > > > > > String keyString =3D getXmlKey(valueString);
> > > > > > getKeyText().set(keyString);  =
getValueText().set(valueString);
> > > > > > output.collect(getKeyText(), getValueText());  }
> > > > > >
> > > > > >
> > > > > >  public Text getKeyText() {
> > > > > >  return keyText;
> > > > > >  }
> > > > > >
> > > > > >
> > > > > >  public void setKeyText(Text keyText) {  this.keyText =3D =
keyText;
>  }
> > > > > >
> > > > > >
> > > > > >  public Text getValueText() {
> > > > > >  return valueText;
> > > > > >  }
> > > > > >
> > > > > >
> > > > > >  public void setValueText(Text valueText) {  this.valueText =
=3D
> > > > > > valueText;  }
> > > > > >
> > > > > >
> > > > > >  private String getXmlKey(String value) {
> > > > > >        // Get the Key from the XML in the value.
> > > > > >  }
> > > > > >
> > > > > > }
> > > > > >
> > > > > > The XML snippet from the Value is fine when it is passed =
into the
> > > > > > map() method.  I am not changing any data either, just =
pulling
> out
> > > > > > information for the key.  The problem I am seeing is between =
the
> > Map
> > > >
> > > > > > phase and the Reduce phase, the XML is getting munged.  For
> > Example:
> > > > > >
> > > > > >  </PrivateRate>
> > > > > >  </PrivateRateSet>te>
> > > > > >
> > > > > > It is my understanding that Hadoop uses the same instance of =
the
> > Key
> > > >
> > > > > > and Value object when calling the Map method.  What changes =
is
> the
> > > > > > data within those instances.  So, I ran an experiment where =
I do
> > not
> > > >
> > > > > > have different Key or Value Text Objects.  I reuse the ones
> passed
> > > > > > into the method, like below:
> > > > > >
> > > > > > public class XmlMapper extends MapReduceBase implements =
Mapper {
> > > > > >
> > > > > >  @SuppressWarnings("unchecked")
> > > > > >  public void map(Object key, Object value, OutputCollector
> output,
> > > > > > Reporter reporter) throws IOException {  Text keyText =3D
> (Text)key;
> > > > > > Text valueText =3D (Text)value;  String valueString =3D new
> > > > > > String(valueText.getBytes(), "UTF-8");  String keyString =3D
> > > > > > getXmlKey(valueString);  keyText.set(keyString);
> > > > > > valueText.set(valueString);  output.collect(keyText, =
valueText);
>  }
> > > > > >
> > > > > >
> > > > > >  private String getXmlKey(String value) {
> > > > > >        // Get the Key from the XML in the value.
> > > > > >  }
> > > > > >
> > > > > > }
> > > > > >
> > > > > > What was interesting about this is the fact that the XML was
> > getting
> > > >
> > > > > > munged within the Map Phase.  When I changed over to the =
code at
> > the
> > > >
> > > > > > top, the Map phase was fine.  However, the Reduce phase =
picks up
> > the
> > > >
> > > > > > munged XML.  Trying to debug the problem, I came across this
> method
> > > > > > in
> > > > >
> > > > > > the Text Object:
> > > > > >
> > > > > > public void set(byte[] utf8, int start, int len) {
> > > > > >    setCapacity(len, false);
> > > > > >    System.arraycopy(utf8, start, bytes, 0, len);
> > > > > >    this.length =3D len;
> > > > > > }
> > > > > >
> > > > > > If the "bytes" array had a length of 1000 and the "utf8" =
array
> has
> > a
> > > >
> > > > > > length of 500; doing a System.arraycopy() would only copy =
the
> first
> > > > > > 500 from "utf8" to "bytes" but leave the last 500 in "bytes"
> alone.
> > > > > > Could this be the cause of the XML munging?
> > > > > >
> > > > > > All of this leads me to a few questions:
> > > > > >
> > > > > > 1) Has anyone successfully used XML snippets as the data =
format
> > > > > > within
> > > > >
> > > > > > a MapReduce job; not just reading from the file but used =
during
> the
> > > > > > shuffle?
> > > > > > 2) Is anyone seeing this problem with XML or any other =
format?
> > > > > > 3) Does anyone know what is going on?
> > > > > > 4) Is this a bug?
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Peter
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>


------_=_NextPart_001_01CB2829.83EDDDB8--

From general-return-1845-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 16:39:06 2010
Return-Path: <general-return-1845-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 92123 invoked from network); 20 Jul 2010 16:39:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 16:39:06 -0000
Received: (qmail 22327 invoked by uid 500); 20 Jul 2010 16:39:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22247 invoked by uid 500); 20 Jul 2010 16:39:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22239 invoked by uid 99); 20 Jul 2010 16:39:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 16:39:04 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates 209.85.161.176 as permitted sender)
Received: from [209.85.161.176] (HELO mail-gx0-f176.google.com) (209.85.161.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 16:38:58 +0000
Received: by gxk7 with SMTP id 7so4632620gxk.35
        for <general@hadoop.apache.org>; Tue, 20 Jul 2010 09:38:36 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=zuiToMwB6TVyQEvqe1auThYHo+jb4lIE15xnV8nK1l4=;
        b=KtM6lLpqlliEyZGU5t8ocR/n1j4LfU93qI4BLqgetchLsU+lJD/8WTkRn+NgW61ZSm
         GwNY0KkUtBY95swzHHrfTrwaO62zH4Q6NO0IG4l6CaSundfBFzBasQDHegVtUYyomFOl
         QbytfxFROKCyyl+i+JVE7WlvfW+XmH0oPDvmY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Vf9KqcjnopU2c5tJJ72G03rBgOaqyyJVm7lEE41i3C7LoEX573mMdKfHjh42bikj93
         osErUUMPeAjdOGEnFWHg7XyeUrtVPzeKfRNHkmgen7LTx3mPt55uTDI/1F5WWmWmcSn2
         kNWYiysyNLmrfkGWt/N8LlqB7CGMCzrVEcD1U=
MIME-Version: 1.0
Received: by 10.224.79.79 with SMTP id o15mr6391273qak.365.1279643916073; Tue, 
	20 Jul 2010 09:38:36 -0700 (PDT)
Received: by 10.229.75.85 with HTTP; Tue, 20 Jul 2010 09:38:34 -0700 (PDT)
In-Reply-To: <AANLkTikXmgzQyd_fuD3EN-nvjK-FP60wDqtTn7Z4xcwB@mail.gmail.com>
References: <5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F46C@sccorpmail01.mygazoo.com>
	<AANLkTilvuqho0l3U9ETg5HHT4Yz5uUkyjX_xqBvmh505@mail.gmail.com>
	<5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D051@sccorpmail01.mygazoo.com>
	<5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F4BA@sccorpmail01.mygazoo.com>
	<5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D053@sccorpmail01.mygazoo.com>
	<AANLkTilAzw5J1oL8raqTmPtxi9BGpuEwHXimEaUdfOzL@mail.gmail.com>
	<5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F5EB@sccorpmail01.mygazoo.com>
	<AANLkTinDkca4ABkityQiyu9GGX-NddzE0pMkAiIRN9J3@mail.gmail.com>
	<AANLkTimScTowrYvQNcC5-ccxwVlwRyWnOdQ4asBWjNUW@mail.gmail.com>
	<AANLkTilCSmQsML8I5ayJZpIA7o0AJ5jb2-IMVxBu3QaN@mail.gmail.com>
	<AANLkTikXmgzQyd_fuD3EN-nvjK-FP60wDqtTn7Z4xcwB@mail.gmail.com>
Date: Tue, 20 Jul 2010 09:38:34 -0700
Message-ID: <AANLkTinj7obRRn4lywbNtTrXTt6kpDiprBQEw_1MZQO7@mail.gmail.com>
Subject: Re: Hadoop and XML
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00c09f8a4c010e7dbd048bd451b9
X-Virus-Checked: Checked by ClamAV on apache.org

--00c09f8a4c010e7dbd048bd451b9
Content-Type: text/plain; charset=ISO-8859-1

So the correct call should be:
String valueString = new String(valueText.getBytes(), 0,
valueText.getLength(), "UTF-8");

Cheers

On Tue, Jul 20, 2010 at 9:23 AM, Jeff Bean <jwfbean@cloudera.com> wrote:

> data.length is the length of the byte array.
>
> Text.getLength() most likely returns a different value than
> getBytes.length.
>
> Hadoop reuses box class objects like Text, so what it's probably doing is
> writing over the byte array, lengthening it as necessary, and just updating
> a separate length attribute.
>
> Jeff
>
> On Tue, Jul 20, 2010 at 8:56 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > Interesting.
> > String class is able to handle this scenario:
> >
> >  348       public String(byte[] data, String encoding) throws
> > UnsupportedEncodingException {
> >  349           this(data, 0, data.length, encoding);
> >  350       }
> >
> >
> >
> > On Tue, Jul 20, 2010 at 6:01 AM, Jeff Bean <jwfbean@cloudera.com> wrote:
> >
> > > I think the problem is here:
> > >
> > > String valueString = new String(valueText.getBytes(), "UTF-8");
> > >
> > > Javadoc for Text says:
> > >
> > > *getBytes<
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/Text.html#getBytes%28%29
> > > >
> > > *()
> > >          Returns the raw bytes; however, only data up to
> > > getLength()<
> > >
> >
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/Text.html#getLength%28%29
> > > >is
> > > valid.
> > >
> > > So try getting the length, truncating the byte array at the value
> > returned
> > > by getLength() and THEN converting it to a String.
> > >
> > > Jeff
> > >
> > > On Mon, Jul 19, 2010 at 9:08 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> > >
> > > > For your initial question on Text.set().
> > > > Text.setCapacity() allocates new byte array. Since keepData is false,
> > old
> > > > data wouldn't be copied over.
> > > >
> > > > On Mon, Jul 19, 2010 at 8:01 AM, Peter Minearo <
> > > > Peter.Minearo@reardencommerce.com> wrote:
> > > >
> > > > > I am already using XmlInputFormat.  The input into the Map phase is
> > not
> > > > > the problem.  The problem lays in between the Map and Reduce phase.
> > > > >
> > > > > BTW - The article is correct.  DO NOT USE StreamXmlRecordReader.
> > > > > XmlInputFormat is a lot faster.  From my testing,
> > StreamXmlRecordReader
> > > > > took 8 minutes to read a 1 GB XML document; where as,
> XmlInputFormat
> > > was
> > > > > under 2 minutes. (Using 2 Core, 8GB machines)
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Ted Yu [mailto:yuzhihong@gmail.com]
> > > > > Sent: Friday, July 16, 2010 9:44 PM
> > > > > To: general@hadoop.apache.org
> > > > > Subject: Re: Hadoop and XML
> > > > >
> > > > > From an earlier post:
> > > > >
> > http://oobaloo.co.uk/articles/2010/1/20/processing-xml-in-hadoop.html
> > > > >
> > > > > On Fri, Jul 16, 2010 at 3:07 PM, Peter Minearo <
> > > > > Peter.Minearo@reardencommerce.com> wrote:
> > > > >
> > > > > > Moving the variable to a local variable did not seem to work:
> > > > > >
> > > > > >
> > > > > > </PrivateRateSet>vateRateSet>
> > > > > >
> > > > > >
> > > > > >
> > > > > > public void map(Object key, Object value, OutputCollector output,
> > > > > > Reporter
> > > > > > reporter) throws IOException {
> > > > > >                Text valueText = (Text)value;
> > > > > >                String valueString = new
> > String(valueText.getBytes(),
> > > > > > "UTF-8");
> > > > > >                String keyString = getXmlKey(valueString);
> > > > > >                 Text returnKeyText = new Text();
> > > > > >                Text returnValueText = new Text();
> > > > > >                returnKeyText.set(keyString);
> > > > > >                returnValueText.set(valueString);
> > > > > >                output.collect(returnKeyText, returnValueText); }
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Peter Minearo [mailto:Peter.Minearo@Reardencommerce.com]
> > > > > > Sent: Fri 7/16/2010 2:51 PM
> > > > > > To: general@hadoop.apache.org
> > > > > > Subject: RE: Hadoop and XML
> > > > > >
> > > > > > Whoops....right after I sent it and someone else made a
> suggestion;
> > I
> > > > > > realized what question 2 was about.  I can try that, but wouldn't
> > > that
> > > > >
> > > > > > cause Object bloat?  During the Hadoop training I went through;
> it
> > > was
> > > > >
> > > > > > mentioned to reuse the returning Key and Value objects to keep
> the
> > > > > > number of Objects created down to a minimum.  Is this not really
> a
> > > > > > valid point?
> > > > > >
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Peter Minearo [mailto:Peter.Minearo@Reardencommerce.com]
> > > > > > Sent: Friday, July 16, 2010 2:44 PM
> > > > > > To: general@hadoop.apache.org
> > > > > > Subject: RE: Hadoop and XML
> > > > > >
> > > > > >
> > > > > > I am not using multi-threaded Map tasks.  Also, if I understand
> > your
> > > > > > second question correctly:
> > > > > > "Also can you try creating the output key and values in the map
> > > > > > method(method lacal) ?"
> > > > > > In the first code snippet I am doing exactly that.
> > > > > >
> > > > > > Below is the class that runs the Job.
> > > > > >
> > > > > > public class HadoopJobClient {
> > > > > >
> > > > > >        private static final Log LOGGER =
> > > > > > LogFactory.getLog(Prds.class.getName());
> > > > > >
> > > > > >        public static void main(String[] args) {
> > > > > >                JobConf conf = new JobConf(Prds.class);
> > > > > >
> > > > > >                conf.set("xmlinput.start", "<PrivateRateSet>");
> > > > > >                conf.set("xmlinput.end", "</PrivateRateSet>");
> > > > > >
> > > > > >                conf.setJobName("PRDS Parse");
> > > > > >
> > > > > >                conf.setOutputKeyClass(Text.class);
> > > > > >                conf.setOutputValueClass(Text.class);
> > > > > >
> > > > > >                conf.setMapperClass(PrdsMapper.class);
> > > > > >                conf.setReducerClass(PrdsReducer.class);
> > > > > >
> > > > > >                conf.setInputFormat(XmlInputFormat.class);
> > > > > >                conf.setOutputFormat(TextOutputFormat.class);
> > > > > >
> > > > > >                FileInputFormat.setInputPaths(conf, new
> > > Path(args[0]));
> > > > > >                FileOutputFormat.setOutputPath(conf, new
> > > > > > Path(args[1]));
> > > > > >
> > > > > >                // Run the job
> > > > > >                try {
> > > > > >                        JobClient.runJob(conf);
> > > > > >                } catch (IOException e) {
> > > > > >                        LOGGER.error(e.getMessage(), e);
> > > > > >                }
> > > > > >
> > > > > >        }
> > > > > >
> > > > > >
> > > > > > }
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Soumya Banerjee [mailto:soumya.sbanerjee@gmail.com]
> > > > > > Sent: Fri 7/16/2010 2:29 PM
> > > > > > To: general@hadoop.apache.org
> > > > > > Subject: Re: Hadoop and XML
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Can you please share the code of the job submission client ?
> > > > > >
> > > > > > Also can you try creating the output key and values in the map
> > > > > > method(method
> > > > > > lacal) ?
> > > > > > Make sure you are not using multi threaded map task
> configuration.
> > > > > >
> > > > > > map()
> > > > > > {
> > > > > > private Text keyText = new Text();
> > > > > >  private Text valueText = new Text();
> > > > > >
> > > > > > //rest of the code
> > > > > > }
> > > > > >
> > > > > > Soumya.
> > > > > >
> > > > > > On Sat, Jul 17, 2010 at 2:30 AM, Peter Minearo <
> > > > > > Peter.Minearo@reardencommerce.com> wrote:
> > > > > >
> > > > > > > I have an XML file that has sparse data in it.  I am running a
> > > > > > > MapReduce Job that reads in an XML file, pulls out a Key from
> > > within
> > > > >
> > > > > > > the XML snippet and then hands back the Key and the XML snippet
> > (as
> > > > > > > the Value) to the OutputCollector.  The reason is to sort the
> > file
> > > > > > back into order.
> > > > > > > Below is the snippet of code.
> > > > > > >
> > > > > > > public class XmlMapper extends MapReduceBase implements Mapper
> {
> > > > > > >
> > > > > > >  private Text keyText = new Text();
> > > > > > >  private Text valueText = new Text();
> > > > > > >
> > > > > > >  @SuppressWarnings("unchecked")
> > > > > > >  public void map(Object key, Object value, OutputCollector
> > output,
> > > > > > > Reporter reporter) throws IOException {  Text valueText =
> > > > > > > (Text)value;
> > > > > >
> > > > > > > String valueString = new String(valueText.getBytes(), "UTF-8");
> > > > > > > String keyString = getXmlKey(valueString);
> > > > > > > getKeyText().set(keyString);  getValueText().set(valueString);
> > > > > > > output.collect(getKeyText(), getValueText());  }
> > > > > > >
> > > > > > >
> > > > > > >  public Text getKeyText() {
> > > > > > >  return keyText;
> > > > > > >  }
> > > > > > >
> > > > > > >
> > > > > > >  public void setKeyText(Text keyText) {  this.keyText =
> keyText;
> >  }
> > > > > > >
> > > > > > >
> > > > > > >  public Text getValueText() {
> > > > > > >  return valueText;
> > > > > > >  }
> > > > > > >
> > > > > > >
> > > > > > >  public void setValueText(Text valueText) {  this.valueText =
> > > > > > > valueText;  }
> > > > > > >
> > > > > > >
> > > > > > >  private String getXmlKey(String value) {
> > > > > > >        // Get the Key from the XML in the value.
> > > > > > >  }
> > > > > > >
> > > > > > > }
> > > > > > >
> > > > > > > The XML snippet from the Value is fine when it is passed into
> the
> > > > > > > map() method.  I am not changing any data either, just pulling
> > out
> > > > > > > information for the key.  The problem I am seeing is between
> the
> > > Map
> > > > >
> > > > > > > phase and the Reduce phase, the XML is getting munged.  For
> > > Example:
> > > > > > >
> > > > > > >  </PrivateRate>
> > > > > > >  </PrivateRateSet>te>
> > > > > > >
> > > > > > > It is my understanding that Hadoop uses the same instance of
> the
> > > Key
> > > > >
> > > > > > > and Value object when calling the Map method.  What changes is
> > the
> > > > > > > data within those instances.  So, I ran an experiment where I
> do
> > > not
> > > > >
> > > > > > > have different Key or Value Text Objects.  I reuse the ones
> > passed
> > > > > > > into the method, like below:
> > > > > > >
> > > > > > > public class XmlMapper extends MapReduceBase implements Mapper
> {
> > > > > > >
> > > > > > >  @SuppressWarnings("unchecked")
> > > > > > >  public void map(Object key, Object value, OutputCollector
> > output,
> > > > > > > Reporter reporter) throws IOException {  Text keyText =
> > (Text)key;
> > > > > > > Text valueText = (Text)value;  String valueString = new
> > > > > > > String(valueText.getBytes(), "UTF-8");  String keyString =
> > > > > > > getXmlKey(valueString);  keyText.set(keyString);
> > > > > > > valueText.set(valueString);  output.collect(keyText,
> valueText);
> >  }
> > > > > > >
> > > > > > >
> > > > > > >  private String getXmlKey(String value) {
> > > > > > >        // Get the Key from the XML in the value.
> > > > > > >  }
> > > > > > >
> > > > > > > }
> > > > > > >
> > > > > > > What was interesting about this is the fact that the XML was
> > > getting
> > > > >
> > > > > > > munged within the Map Phase.  When I changed over to the code
> at
> > > the
> > > > >
> > > > > > > top, the Map phase was fine.  However, the Reduce phase picks
> up
> > > the
> > > > >
> > > > > > > munged XML.  Trying to debug the problem, I came across this
> > method
> > > > > > > in
> > > > > >
> > > > > > > the Text Object:
> > > > > > >
> > > > > > > public void set(byte[] utf8, int start, int len) {
> > > > > > >    setCapacity(len, false);
> > > > > > >    System.arraycopy(utf8, start, bytes, 0, len);
> > > > > > >    this.length = len;
> > > > > > > }
> > > > > > >
> > > > > > > If the "bytes" array had a length of 1000 and the "utf8" array
> > has
> > > a
> > > > >
> > > > > > > length of 500; doing a System.arraycopy() would only copy the
> > first
> > > > > > > 500 from "utf8" to "bytes" but leave the last 500 in "bytes"
> > alone.
> > > > > > > Could this be the cause of the XML munging?
> > > > > > >
> > > > > > > All of this leads me to a few questions:
> > > > > > >
> > > > > > > 1) Has anyone successfully used XML snippets as the data format
> > > > > > > within
> > > > > >
> > > > > > > a MapReduce job; not just reading from the file but used during
> > the
> > > > > > > shuffle?
> > > > > > > 2) Is anyone seeing this problem with XML or any other format?
> > > > > > > 3) Does anyone know what is going on?
> > > > > > > 4) Is this a bug?
> > > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Peter
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

--00c09f8a4c010e7dbd048bd451b9--

From general-return-1846-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 16:51:34 2010
Return-Path: <general-return-1846-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97590 invoked from network); 20 Jul 2010 16:51:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 16:51:34 -0000
Received: (qmail 47948 invoked by uid 500); 20 Jul 2010 16:51:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47899 invoked by uid 500); 20 Jul 2010 16:51:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47891 invoked by uid 99); 20 Jul 2010 16:51:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 16:51:32 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates 209.85.210.48 as permitted sender)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 16:51:25 +0000
Received: by pzk36 with SMTP id 36so3656170pzk.35
        for <general@hadoop.apache.org>; Tue, 20 Jul 2010 09:50:09 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=SREMvv2S5920s3+1uDQt8eGruk1sLO0xMgJSqeTbAr4=;
        b=YoRCNzMiOUANu0jS4IloLVWIEqkuM20LgFO4b54B0vbdXWHnyeCX6/sa8wXQsaMFIZ
         WCxnqf+gWAY8sDDssqRzgEMXjRAnip6BhbLfr/VGLx5sutSSGeHcaz+nLOeZaQr7dvlf
         3M9a9L5AdcJN26nbpYfO1FoL8zwxVfGlrpuZ8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=gMDkZM1CZnAM/mpe7IPGisXWkJ/kqMlPd2CvTj5hyNUDKyxE63oKTiZRyd5cdGvr+B
         h1AyFJ0r1uQbmnUEAR+inoVpbG3xtBhgXfPh3c8DLcEvodggT+dZMz2t/iuvLXa872qx
         HyHTsNZMUMaZuDC9eFey5GQEXvYm881HxyTG8=
MIME-Version: 1.0
Received: by 10.114.57.13 with SMTP id f13mr9837237waa.116.1279644609360; Tue, 
	20 Jul 2010 09:50:09 -0700 (PDT)
Received: by 10.229.75.85 with HTTP; Tue, 20 Jul 2010 09:50:08 -0700 (PDT)
In-Reply-To: <AANLkTinj7obRRn4lywbNtTrXTt6kpDiprBQEw_1MZQO7@mail.gmail.com>
References: <5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F46C@sccorpmail01.mygazoo.com>
	<AANLkTilvuqho0l3U9ETg5HHT4Yz5uUkyjX_xqBvmh505@mail.gmail.com>
	<5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D051@sccorpmail01.mygazoo.com>
	<5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F4BA@sccorpmail01.mygazoo.com>
	<5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D053@sccorpmail01.mygazoo.com>
	<AANLkTilAzw5J1oL8raqTmPtxi9BGpuEwHXimEaUdfOzL@mail.gmail.com>
	<5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F5EB@sccorpmail01.mygazoo.com>
	<AANLkTinDkca4ABkityQiyu9GGX-NddzE0pMkAiIRN9J3@mail.gmail.com>
	<AANLkTimScTowrYvQNcC5-ccxwVlwRyWnOdQ4asBWjNUW@mail.gmail.com>
	<AANLkTilCSmQsML8I5ayJZpIA7o0AJ5jb2-IMVxBu3QaN@mail.gmail.com>
	<AANLkTikXmgzQyd_fuD3EN-nvjK-FP60wDqtTn7Z4xcwB@mail.gmail.com>
	<AANLkTinj7obRRn4lywbNtTrXTt6kpDiprBQEw_1MZQO7@mail.gmail.com>
Date: Tue, 20 Jul 2010 09:50:08 -0700
Message-ID: <AANLkTimtBpY83iv955N3wPSb-P3mUEqlcrlCn7byoE87@mail.gmail.com>
Subject: Re: Hadoop and XML
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364587226134d0048bd47aa5
X-Virus-Checked: Checked by ClamAV on apache.org

--0016364587226134d0048bd47aa5
Content-Type: text/plain; charset=ISO-8859-1

I also added Peter's comment to the JIRA I logged:
https://issues.apache.org/jira/browse/HADOOP-6868

On Tue, Jul 20, 2010 at 9:38 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> So the correct call should be:
> String valueString = new String(valueText.getBytes(), 0,
> valueText.getLength(), "UTF-8");
>
> Cheers
>
>
> On Tue, Jul 20, 2010 at 9:23 AM, Jeff Bean <jwfbean@cloudera.com> wrote:
>
>> data.length is the length of the byte array.
>>
>> Text.getLength() most likely returns a different value than
>> getBytes.length.
>>
>> Hadoop reuses box class objects like Text, so what it's probably doing is
>> writing over the byte array, lengthening it as necessary, and just
>> updating
>> a separate length attribute.
>>
>> Jeff
>>
>> On Tue, Jul 20, 2010 at 8:56 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>>
>> > Interesting.
>> > String class is able to handle this scenario:
>> >
>> >  348       public String(byte[] data, String encoding) throws
>> > UnsupportedEncodingException {
>> >  349           this(data, 0, data.length, encoding);
>> >  350       }
>> >
>> >
>> >
>> > On Tue, Jul 20, 2010 at 6:01 AM, Jeff Bean <jwfbean@cloudera.com>
>> wrote:
>> >
>> > > I think the problem is here:
>> > >
>> > > String valueString = new String(valueText.getBytes(), "UTF-8");
>> > >
>> > > Javadoc for Text says:
>> > >
>> > > *getBytes<
>> > >
>> >
>> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/Text.html#getBytes%28%29
>> > > >
>> > > *()
>> > >          Returns the raw bytes; however, only data up to
>> > > getLength()<
>> > >
>> >
>> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/Text.html#getLength%28%29
>> > > >is
>> > > valid.
>> > >
>> > > So try getting the length, truncating the byte array at the value
>> > returned
>> > > by getLength() and THEN converting it to a String.
>> > >
>> > > Jeff
>> > >
>> > > On Mon, Jul 19, 2010 at 9:08 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>> > >
>> > > > For your initial question on Text.set().
>> > > > Text.setCapacity() allocates new byte array. Since keepData is
>> false,
>> > old
>> > > > data wouldn't be copied over.
>> > > >
>> > > > On Mon, Jul 19, 2010 at 8:01 AM, Peter Minearo <
>> > > > Peter.Minearo@reardencommerce.com> wrote:
>> > > >
>> > > > > I am already using XmlInputFormat.  The input into the Map phase
>> is
>> > not
>> > > > > the problem.  The problem lays in between the Map and Reduce
>> phase.
>> > > > >
>> > > > > BTW - The article is correct.  DO NOT USE StreamXmlRecordReader.
>> > > > > XmlInputFormat is a lot faster.  From my testing,
>> > StreamXmlRecordReader
>> > > > > took 8 minutes to read a 1 GB XML document; where as,
>> XmlInputFormat
>> > > was
>> > > > > under 2 minutes. (Using 2 Core, 8GB machines)
>> > > > >
>> > > > >
>> > > > > -----Original Message-----
>> > > > > From: Ted Yu [mailto:yuzhihong@gmail.com]
>> > > > > Sent: Friday, July 16, 2010 9:44 PM
>> > > > > To: general@hadoop.apache.org
>> > > > > Subject: Re: Hadoop and XML
>> > > > >
>> > > > > From an earlier post:
>> > > > >
>> > http://oobaloo.co.uk/articles/2010/1/20/processing-xml-in-hadoop.html
>> > > > >
>> > > > > On Fri, Jul 16, 2010 at 3:07 PM, Peter Minearo <
>> > > > > Peter.Minearo@reardencommerce.com> wrote:
>> > > > >
>> > > > > > Moving the variable to a local variable did not seem to work:
>> > > > > >
>> > > > > >
>> > > > > > </PrivateRateSet>vateRateSet>
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > public void map(Object key, Object value, OutputCollector
>> output,
>> > > > > > Reporter
>> > > > > > reporter) throws IOException {
>> > > > > >                Text valueText = (Text)value;
>> > > > > >                String valueString = new
>> > String(valueText.getBytes(),
>> > > > > > "UTF-8");
>> > > > > >                String keyString = getXmlKey(valueString);
>> > > > > >                 Text returnKeyText = new Text();
>> > > > > >                Text returnValueText = new Text();
>> > > > > >                returnKeyText.set(keyString);
>> > > > > >                returnValueText.set(valueString);
>> > > > > >                output.collect(returnKeyText, returnValueText); }
>> > > > > >
>> > > > > > -----Original Message-----
>> > > > > > From: Peter Minearo [mailto:Peter.Minearo@Reardencommerce.com]
>> > > > > > Sent: Fri 7/16/2010 2:51 PM
>> > > > > > To: general@hadoop.apache.org
>> > > > > > Subject: RE: Hadoop and XML
>> > > > > >
>> > > > > > Whoops....right after I sent it and someone else made a
>> suggestion;
>> > I
>> > > > > > realized what question 2 was about.  I can try that, but
>> wouldn't
>> > > that
>> > > > >
>> > > > > > cause Object bloat?  During the Hadoop training I went through;
>> it
>> > > was
>> > > > >
>> > > > > > mentioned to reuse the returning Key and Value objects to keep
>> the
>> > > > > > number of Objects created down to a minimum.  Is this not really
>> a
>> > > > > > valid point?
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > -----Original Message-----
>> > > > > > From: Peter Minearo [mailto:Peter.Minearo@Reardencommerce.com]
>> > > > > > Sent: Friday, July 16, 2010 2:44 PM
>> > > > > > To: general@hadoop.apache.org
>> > > > > > Subject: RE: Hadoop and XML
>> > > > > >
>> > > > > >
>> > > > > > I am not using multi-threaded Map tasks.  Also, if I understand
>> > your
>> > > > > > second question correctly:
>> > > > > > "Also can you try creating the output key and values in the map
>> > > > > > method(method lacal) ?"
>> > > > > > In the first code snippet I am doing exactly that.
>> > > > > >
>> > > > > > Below is the class that runs the Job.
>> > > > > >
>> > > > > > public class HadoopJobClient {
>> > > > > >
>> > > > > >        private static final Log LOGGER =
>> > > > > > LogFactory.getLog(Prds.class.getName());
>> > > > > >
>> > > > > >        public static void main(String[] args) {
>> > > > > >                JobConf conf = new JobConf(Prds.class);
>> > > > > >
>> > > > > >                conf.set("xmlinput.start", "<PrivateRateSet>");
>> > > > > >                conf.set("xmlinput.end", "</PrivateRateSet>");
>> > > > > >
>> > > > > >                conf.setJobName("PRDS Parse");
>> > > > > >
>> > > > > >                conf.setOutputKeyClass(Text.class);
>> > > > > >                conf.setOutputValueClass(Text.class);
>> > > > > >
>> > > > > >                conf.setMapperClass(PrdsMapper.class);
>> > > > > >                conf.setReducerClass(PrdsReducer.class);
>> > > > > >
>> > > > > >                conf.setInputFormat(XmlInputFormat.class);
>> > > > > >                conf.setOutputFormat(TextOutputFormat.class);
>> > > > > >
>> > > > > >                FileInputFormat.setInputPaths(conf, new
>> > > Path(args[0]));
>> > > > > >                FileOutputFormat.setOutputPath(conf, new
>> > > > > > Path(args[1]));
>> > > > > >
>> > > > > >                // Run the job
>> > > > > >                try {
>> > > > > >                        JobClient.runJob(conf);
>> > > > > >                } catch (IOException e) {
>> > > > > >                        LOGGER.error(e.getMessage(), e);
>> > > > > >                }
>> > > > > >
>> > > > > >        }
>> > > > > >
>> > > > > >
>> > > > > > }
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > -----Original Message-----
>> > > > > > From: Soumya Banerjee [mailto:soumya.sbanerjee@gmail.com]
>> > > > > > Sent: Fri 7/16/2010 2:29 PM
>> > > > > > To: general@hadoop.apache.org
>> > > > > > Subject: Re: Hadoop and XML
>> > > > > >
>> > > > > > Hi,
>> > > > > >
>> > > > > > Can you please share the code of the job submission client ?
>> > > > > >
>> > > > > > Also can you try creating the output key and values in the map
>> > > > > > method(method
>> > > > > > lacal) ?
>> > > > > > Make sure you are not using multi threaded map task
>> configuration.
>> > > > > >
>> > > > > > map()
>> > > > > > {
>> > > > > > private Text keyText = new Text();
>> > > > > >  private Text valueText = new Text();
>> > > > > >
>> > > > > > //rest of the code
>> > > > > > }
>> > > > > >
>> > > > > > Soumya.
>> > > > > >
>> > > > > > On Sat, Jul 17, 2010 at 2:30 AM, Peter Minearo <
>> > > > > > Peter.Minearo@reardencommerce.com> wrote:
>> > > > > >
>> > > > > > > I have an XML file that has sparse data in it.  I am running a
>> > > > > > > MapReduce Job that reads in an XML file, pulls out a Key from
>> > > within
>> > > > >
>> > > > > > > the XML snippet and then hands back the Key and the XML
>> snippet
>> > (as
>> > > > > > > the Value) to the OutputCollector.  The reason is to sort the
>> > file
>> > > > > > back into order.
>> > > > > > > Below is the snippet of code.
>> > > > > > >
>> > > > > > > public class XmlMapper extends MapReduceBase implements Mapper
>> {
>> > > > > > >
>> > > > > > >  private Text keyText = new Text();
>> > > > > > >  private Text valueText = new Text();
>> > > > > > >
>> > > > > > >  @SuppressWarnings("unchecked")
>> > > > > > >  public void map(Object key, Object value, OutputCollector
>> > output,
>> > > > > > > Reporter reporter) throws IOException {  Text valueText =
>> > > > > > > (Text)value;
>> > > > > >
>> > > > > > > String valueString = new String(valueText.getBytes(),
>> "UTF-8");
>> > > > > > > String keyString = getXmlKey(valueString);
>> > > > > > > getKeyText().set(keyString);  getValueText().set(valueString);
>> > > > > > > output.collect(getKeyText(), getValueText());  }
>> > > > > > >
>> > > > > > >
>> > > > > > >  public Text getKeyText() {
>> > > > > > >  return keyText;
>> > > > > > >  }
>> > > > > > >
>> > > > > > >
>> > > > > > >  public void setKeyText(Text keyText) {  this.keyText =
>> keyText;
>> >  }
>> > > > > > >
>> > > > > > >
>> > > > > > >  public Text getValueText() {
>> > > > > > >  return valueText;
>> > > > > > >  }
>> > > > > > >
>> > > > > > >
>> > > > > > >  public void setValueText(Text valueText) {  this.valueText =
>> > > > > > > valueText;  }
>> > > > > > >
>> > > > > > >
>> > > > > > >  private String getXmlKey(String value) {
>> > > > > > >        // Get the Key from the XML in the value.
>> > > > > > >  }
>> > > > > > >
>> > > > > > > }
>> > > > > > >
>> > > > > > > The XML snippet from the Value is fine when it is passed into
>> the
>> > > > > > > map() method.  I am not changing any data either, just pulling
>> > out
>> > > > > > > information for the key.  The problem I am seeing is between
>> the
>> > > Map
>> > > > >
>> > > > > > > phase and the Reduce phase, the XML is getting munged.  For
>> > > Example:
>> > > > > > >
>> > > > > > >  </PrivateRate>
>> > > > > > >  </PrivateRateSet>te>
>> > > > > > >
>> > > > > > > It is my understanding that Hadoop uses the same instance of
>> the
>> > > Key
>> > > > >
>> > > > > > > and Value object when calling the Map method.  What changes is
>> > the
>> > > > > > > data within those instances.  So, I ran an experiment where I
>> do
>> > > not
>> > > > >
>> > > > > > > have different Key or Value Text Objects.  I reuse the ones
>> > passed
>> > > > > > > into the method, like below:
>> > > > > > >
>> > > > > > > public class XmlMapper extends MapReduceBase implements Mapper
>> {
>> > > > > > >
>> > > > > > >  @SuppressWarnings("unchecked")
>> > > > > > >  public void map(Object key, Object value, OutputCollector
>> > output,
>> > > > > > > Reporter reporter) throws IOException {  Text keyText =
>> > (Text)key;
>> > > > > > > Text valueText = (Text)value;  String valueString = new
>> > > > > > > String(valueText.getBytes(), "UTF-8");  String keyString =
>> > > > > > > getXmlKey(valueString);  keyText.set(keyString);
>> > > > > > > valueText.set(valueString);  output.collect(keyText,
>> valueText);
>> >  }
>> > > > > > >
>> > > > > > >
>> > > > > > >  private String getXmlKey(String value) {
>> > > > > > >        // Get the Key from the XML in the value.
>> > > > > > >  }
>> > > > > > >
>> > > > > > > }
>> > > > > > >
>> > > > > > > What was interesting about this is the fact that the XML was
>> > > getting
>> > > > >
>> > > > > > > munged within the Map Phase.  When I changed over to the code
>> at
>> > > the
>> > > > >
>> > > > > > > top, the Map phase was fine.  However, the Reduce phase picks
>> up
>> > > the
>> > > > >
>> > > > > > > munged XML.  Trying to debug the problem, I came across this
>> > method
>> > > > > > > in
>> > > > > >
>> > > > > > > the Text Object:
>> > > > > > >
>> > > > > > > public void set(byte[] utf8, int start, int len) {
>> > > > > > >    setCapacity(len, false);
>> > > > > > >    System.arraycopy(utf8, start, bytes, 0, len);
>> > > > > > >    this.length = len;
>> > > > > > > }
>> > > > > > >
>> > > > > > > If the "bytes" array had a length of 1000 and the "utf8" array
>> > has
>> > > a
>> > > > >
>> > > > > > > length of 500; doing a System.arraycopy() would only copy the
>> > first
>> > > > > > > 500 from "utf8" to "bytes" but leave the last 500 in "bytes"
>> > alone.
>> > > > > > > Could this be the cause of the XML munging?
>> > > > > > >
>> > > > > > > All of this leads me to a few questions:
>> > > > > > >
>> > > > > > > 1) Has anyone successfully used XML snippets as the data
>> format
>> > > > > > > within
>> > > > > >
>> > > > > > > a MapReduce job; not just reading from the file but used
>> during
>> > the
>> > > > > > > shuffle?
>> > > > > > > 2) Is anyone seeing this problem with XML or any other format?
>> > > > > > > 3) Does anyone know what is going on?
>> > > > > > > 4) Is this a bug?
>> > > > > > >
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > >
>> > > > > > > Peter
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

--0016364587226134d0048bd47aa5--

From general-return-1847-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 17:36:16 2010
Return-Path: <general-return-1847-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16025 invoked from network); 20 Jul 2010 17:36:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 17:36:16 -0000
Received: (qmail 5926 invoked by uid 500); 20 Jul 2010 17:36:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5747 invoked by uid 500); 20 Jul 2010 17:36:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5739 invoked by uid 99); 20 Jul 2010 17:36:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 17:36:14 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of SRS0=ksgr7x=O2=russellwinterassociates.com=emma@yourhostingaccount.com designates 65.254.253.149 as permitted sender)
Received: from [65.254.253.149] (HELO mailout18.yourhostingaccount.com) (65.254.253.149)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 17:36:04 +0000
Received: from mailscan13.yourhostingaccount.com ([10.1.15.13] helo=mailscan13.yourhostingaccount.com)
	by mailout18.yourhostingaccount.com with esmtp (Exim)
	id 1ObGjH-0006Zp-SZ
	for general@hadoop.apache.org; Tue, 20 Jul 2010 13:35:43 -0400
Received: from impout03.yourhostingaccount.com ([10.1.55.3] helo=impout03.yourhostingaccount.com)
	by mailscan13.yourhostingaccount.com with esmtp (Exim)
	id 1ObGjG-00051J-Sl
	for general@hadoop.apache.org; Tue, 20 Jul 2010 13:35:42 -0400
Received: from authsmtp08.yourhostingaccount.com ([10.1.18.8])
	by impout03.yourhostingaccount.com with NO UCE
	id kHbj1e0050ASqTN0000000; Tue, 20 Jul 2010 13:35:43 -0400
X-EN-OrigOutIP: 10.1.18.8
X-EN-IMPSID: kHbj1e0050ASqTN0000000
Received: from host86-173-2-175.range86-173.btcentralplus.com ([86.173.2.175] helo=D1W03L3J)
	by authsmtp08.yourhostingaccount.com with esmtpa (Exim)
	id 1ObGjG-0002Dg-BN
	for general@hadoop.apache.org; Tue, 20 Jul 2010 13:35:42 -0400
From: "Emma Russell" <emma@russellwinterassociates.com>
To: <general@hadoop.apache.org>
References: <AANLkTimiol0QLyISg4RP54t3GN5AsExoxbli0T7GXMe9@mail.gmail.com>
In-Reply-To: <AANLkTimiol0QLyISg4RP54t3GN5AsExoxbli0T7GXMe9@mail.gmail.com>
Subject: RE: Developers sought for web crawling project
Date: Tue, 20 Jul 2010 18:35:37 +0100
Message-ID: <!&!AAAAAAAAAAAYAAAAAAAAABoNLBJYWzdJtoc4YmLnIn3CgAAAEAAAAJVoIAPoSDNPq/TPKwbulXIBAAAAAA==@russellwinterassociates.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
X-Mailer: Microsoft Office Outlook 12.0
Thread-Index: AcsoG1EYM6NdoV7KSUCa5yN6vOISwQAFlr7w
Content-Language: en-gb
X-EN-UserInfo: 241204cd915fb4df7b3509d8aef24c22:9706facac6a42dca8255befa5a58a182
X-EN-AuthUser: emma_russell@pop.powweb.com
Sender:  "Emma Russell" <emma@russellwinterassociates.com>
X-EN-OrigIP: 86.173.2.175
X-EN-OrigHost: host86-173-2-175.range86-173.btcentralplus.com
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Tim,

Try looking for people on experts exchange. I am a high tech recruiter =
for
the uk and I often look for people on ee as I can see not only the best
answers but also an insight into how the person thinks and which areas =
they
have the most strength

Hope this helps

Emma Russell
SVP Major Accounts
Dir=A0=A0=A0 0208 498 9398
Fax=A0=A0 0208 498 9399
Mob=A0 0778 229 5086
emma@russellwinterassociates.com

20 Roebuck Lane Buckhurst Hill Essex IG9 5QN.=A0=A0=A0 P. 0208 498 =
9398=A0=A0 F. 0208
498 9399=20
Russell Winter Associates Limited is registered in England and Wales at =
2nd
Floor 196 High Road London N22 8HH. Registered Number. =
04896428.=A0=A0=A0=20
=A0
This message and any attachments are solely for the intended recipient =
and
may contain confidential or privileged information. If you are not the
intended recipient, any disclosure, copying, use, or distribution of the
information included in this message and any attachments is =
prohibited.=A0 If
you have received this communication in error, please notify us by reply
e-mail and immediately and permanently delete this message and any
attachments.=A0


-----Original Message-----
From: Tim Robertson [mailto:timrobertson100@gmail.com]=20
Sent: 20 July 2010 15:53
To: general@hadoop.apache.org; user@hbase.apache.org;
hive-user@hadoop.apache.org
Subject: Developers sought for web crawling project

Hi all,

Disclosure: I have been an active member of the Hadoop / HBase / Hive
mailing lists for some time.  I am not a recruiter, but looking to
increase a development team that I lead.  I sincerely apologize if
this message is against mailing list etiquette; I have not seen any
guidelines forbidding this.

We are looking to expand the development team based in Copenhagen,
Denmark.  We run and operate an indexing system that provides access
to biodiversity information, and specifically 100s millions of point
based observations of species occurrence documented in the past
century.  Our live system is outgrowing a MySQL database, and we are
already tentatively using Hive, Hadoop and have run some tests using
HBase.  Over the next 16 months we will rework the whole system,
including custom reporting, custom maps and other visualizations, real
time search indexes, reducing latency, quality control integration,
custom vocabulary work etc.  Our final architecture will likely
include Hive, Hadoop, HBase, Lucene (SOLR / ElasticSearch?), Sqoop and
PostGIS.  If you are familiar with these technologies, and the thought
of some time working in Europe appeals to you, then we'd love to hear
from you.  Guidelines for applications are on the advert:
http://tinyurl.com/gbif-dev-job

Thanks,
Tim


From general-return-1848-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 18:24:44 2010
Return-Path: <general-return-1848-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 41739 invoked from network); 20 Jul 2010 18:24:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 18:24:44 -0000
Received: (qmail 1692 invoked by uid 500); 20 Jul 2010 18:24:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1536 invoked by uid 500); 20 Jul 2010 18:24:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1528 invoked by uid 99); 20 Jul 2010 18:24:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 18:24:42 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [64.78.17.17] (HELO EXHUB018-2.exch018.msoutlookonline.net) (64.78.17.17)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 18:24:34 +0000
Received: from EXVMBX018-1.exch018.msoutlookonline.net ([64.78.17.47]) by
 EXHUB018-2.exch018.msoutlookonline.net ([64.78.17.17]) with mapi; Tue, 20 Jul
 2010 11:23:52 -0700
From: Scott Carey <scott@richrelevance.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 20 Jul 2010 11:24:14 -0700
Subject: Re: Hadoop and XML
Thread-Topic: Hadoop and XML
Thread-Index: AcsoOLRLU5FpYfXmR1SyynHR54PRnQ==
Message-ID: <7FC5996B-8D27-4E79-84A9-F240E47C351C@richrelevance.com>
References: <5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F46C@sccorpmail01.mygazoo.com><AANLkTilvuqho0l3U9ETg5HHT4Yz5uUkyjX_xqBvmh505@mail.gmail.com><5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D051@sccorpmail01.mygazoo.com><5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F4BA@sccorpmail01.mygazoo.com><5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D053@sccorpmail01.mygazoo.com><AANLkTilAzw5J1oL8raqTmPtxi9BGpuEwHXimEaUdfOzL@mail.gmail.com><5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F5EB@sccorpmail01.mygazoo.com><AANLkTinDkca4ABkityQiyu9GGX-NddzE0pMkAiIRN9J3@mail.gmail.com><AANLkTimScTowrYvQNcC5-ccxwVlwRyWnOdQ4asBWjNUW@mail.gmail.com><AANLkTilCSmQsML8I5ayJZpIA7o0AJ5jb2-IMVxBu3QaN@mail.gmail.com>
 <AANLkTikXmgzQyd_fuD3EN-nvjK-FP60wDqtTn7Z4xcwB@mail.gmail.com>
 <5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D055@sccorpmail01.mygazoo.com>
In-Reply-To: <5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D055@sccorpmail01.mygazoo.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


>=20
> This sounds like a bug.
>=20
> Let's say you create a Text object and drop in a String that sets the byt=
e array length to 200.  Then drop in a a second String that sets the byte a=
rray length to 500.  Since, the new length is greater than the previous len=
gth; the byte array length is reset to the longer length.  Now, if you drop=
 in a third String that would set the byte array length to 350; the Text ob=
ject does not replace the byte array with a new length of 350; it utilizes =
the greater length of 500 and sets an extra variable to track the "real" le=
ngth.
>=20
> So: Text.getBytes().length !=3D Text.getLength()
>=20
> This does 2 things:
>=20
> 1. Passes around more data than what is needed
> 2. Makes the Text object confusing to work with
>=20
> Text.getBytes().length =3D=3D Text.getLength() - should be the correct be=
havior.
>=20
>=20

I don't think so.  Passing around byte arrays larger than the valid data is=
 common practice in Java for performance reasons.  Hence, the common method=
 signature containing  (byte[] bytes, int len, int offset) and similar.   C=
reating a new byte array for each resize defeats the purpose of re-using th=
e byte array and the Text object -- lower memory allocation and improved CP=
U cache locality.  The byte array here is a buffer, it does not represent t=
he entire string.


From general-return-1849-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 18:29:17 2010
Return-Path: <general-return-1849-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42709 invoked from network); 20 Jul 2010 18:29:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 18:29:17 -0000
Received: (qmail 6032 invoked by uid 500); 20 Jul 2010 18:29:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5971 invoked by uid 500); 20 Jul 2010 18:29:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5959 invoked by uid 99); 20 Jul 2010 18:29:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 18:29:15 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [64.78.17.18] (HELO EXHUB018-3.exch018.msoutlookonline.net) (64.78.17.18)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 18:29:06 +0000
Received: from EXVMBX018-1.exch018.msoutlookonline.net ([64.78.17.47]) by
 EXHUB018-3.exch018.msoutlookonline.net ([64.78.17.18]) with mapi; Tue, 20 Jul
 2010 11:28:45 -0700
From: Scott Carey <scott@richrelevance.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 20 Jul 2010 11:29:08 -0700
Subject: Re: Hadoop and XML
Thread-Topic: Hadoop and XML
Thread-Index: AcsoOWMktGjIeqx5RDafx1DyrMpVbA==
Message-ID: <EC3823EA-0DB7-4F1F-8996-60A6BCB312D7@richrelevance.com>
References: <5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F46C@sccorpmail01.mygazoo.com><AANLkTilvuqho0l3U9ETg5HHT4Yz5uUkyjX_xqBvmh505@mail.gmail.com><5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D051@sccorpmail01.mygazoo.com><5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F4BA@sccorpmail01.mygazoo.com><5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D053@sccorpmail01.mygazoo.com><AANLkTilAzw5J1oL8raqTmPtxi9BGpuEwHXimEaUdfOzL@mail.gmail.com><5A4D71ACC2708E4BB1BD7B6F87E51C6B0828F5EB@sccorpmail01.mygazoo.com><AANLkTinDkca4ABkityQiyu9GGX-NddzE0pMkAiIRN9J3@mail.gmail.com><AANLkTimScTowrYvQNcC5-ccxwVlwRyWnOdQ4asBWjNUW@mail.gmail.com><AANLkTilCSmQsML8I5ayJZpIA7o0AJ5jb2-IMVxBu3QaN@mail.gmail.com>
 <AANLkTikXmgzQyd_fuD3EN-nvjK-FP60wDqtTn7Z4xcwB@mail.gmail.com>
 <5A4D71ACC2708E4BB1BD7B6F87E51C6B02B6D055@sccorpmail01.mygazoo.com>
 <7FC5996B-8D27-4E79-84A9-F240E47C351C@richrelevance.com>
In-Reply-To: <7FC5996B-8D27-4E79-84A9-F240E47C351C@richrelevance.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


On Jul 20, 2010, at 11:24 AM, Scott Carey wrote:

>=20
>>=20
>> This sounds like a bug.
>>=20
>> Let's say you create a Text object and drop in a String that sets the by=
te array length to 200.  Then drop in a a second String that sets the byte =
array length to 500.  Since, the new length is greater than the previous le=
ngth; the byte array length is reset to the longer length.  Now, if you dro=
p in a third String that would set the byte array length to 350; the Text o=
bject does not replace the byte array with a new length of 350; it utilizes=
 the greater length of 500 and sets an extra variable to track the "real" l=
ength.
>>=20
>> So: Text.getBytes().length !=3D Text.getLength()
>>=20
>> This does 2 things:
>>=20
>> 1. Passes around more data than what is needed
>> 2. Makes the Text object confusing to work with
>>=20
>> Text.getBytes().length =3D=3D Text.getLength() - should be the correct b=
ehavior.
>>=20
>>=20
>=20
> I don't think so.  Passing around byte arrays larger than the valid data =
is common practice in Java for performance reasons.  Hence, the common meth=
od signature containing  (byte[] bytes, int len, int offset) and similar.  =
 Creating a new byte array for each resize defeats the purpose of re-using =
the byte array and the Text object -- lower memory allocation and improved =
CPU cache locality.  The byte array here is a buffer, it does not represent=
 the entire string.
>=20

To be more specific here, shouldn't Text.toString() do the trick?   If Text=
.toString() doesn't work and does something other than what you expect here=
, it should be documented and that class should have another helper method =
that gets you a String from Text.   Calling getBytes() and manually constru=
cting a string means you should know what those bytes represent -- a buffer=
 where the bytes for the string are from index - to Text.getLength().=

From general-return-1850-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 15:32:20 2010
Return-Path: <general-return-1850-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46682 invoked from network); 21 Jul 2010 15:32:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 15:32:19 -0000
Received: (qmail 65557 invoked by uid 500); 21 Jul 2010 15:32:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65474 invoked by uid 500); 21 Jul 2010 15:32:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65466 invoked by uid 99); 21 Jul 2010 15:32:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 15:32:17 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.30.64] (HELO qmta07.emeryville.ca.mail.comcast.net) (76.96.30.64)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 15:32:09 +0000
Received: from omta10.emeryville.ca.mail.comcast.net ([76.96.30.28])
	by qmta07.emeryville.ca.mail.comcast.net with comcast
	id kdNZ1e0040cQ2SLA7fXgHl; Wed, 21 Jul 2010 15:31:40 +0000
Received: from [10.0.0.13] ([98.234.216.58])
	by omta10.emeryville.ca.mail.comcast.net with comcast
	id kfXf1e0061GAoR68WfXfcE; Wed, 21 Jul 2010 15:31:40 +0000
Message-Id: <3BB5B9B0-1D8C-4C32-91B1-51D74A4A8DB5@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=Apple-Mail-24-771388440
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Bay Area HUG tonight
Date: Wed, 21 Jul 2010 08:31:39 -0700
Cc: Susheel Kaushik <susheel@yahoo-inc.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-24-771388440
Content-Type: text/plain;
	charset=WINDOWS-1252;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: quoted-printable

Don't forget that after taking off June for the Hadoop Summit, Yahoo =20
is continuing to host the monthly Bay Area HUG tonight. One =20
organizational note is that Shusheel Kaushik (susheel@yahoo-inc.com) =20
has taken over from Dekel organizing the Bay Area HUGs, so please send =20=

your suggestions for ideas to him.

Tonight's agenda is:
* 6:00 - 6:30 - Socializing and Beers
* 6:30 =96 7:00 =96 Online Content Optimization with Hadoop, Nitin =
Motgi, =20
Yahoo!
We make extensive use of Hadoop technology stack in our content =20
optimization systems. Using Hadoop, we are able to scale to build =20
models for millions of items, and users in near-real time. We leverage =20=

HBase for point lookups/stores of these models. We also use Pig for =20
phrasing our workflows so the map-reduce parallelism is abstracted out =20=

of core processing.
* 7:00 - 7:30 =96 Hadoop at eBay, Anil Madan, eBay
This talk will illustrate how eBay is leveraging its data assets to do =20=

advanced insights and analytics.
Learn how eBay is sourcing huge volumes of data into the cluster and =20
running Click Stream and Transactional data analysis for user =20
behavior, search quality and research use cases.
Anil Madan is the Director of Engineering at eBay responsible for =20
Hadoop cluster build out.
* 7:30 =96 8:00 - Introduction to Avro, Doug Cutting, Cloudera
Avro is a serialization system. It supports interoperable, efficient, =20=

dynamic data storage and RPC.
It's currently implemented in C, C++, Java, Python and Ruby. Support =20
for Map-Reduce over Avro data is being developed, and we expect Hadoop =20=

to eventually move to Avro for its RPC.

You can sign up on meetup: http://bit.ly/9UAnIN

-- Owen=

--Apple-Mail-24-771388440--

From general-return-1851-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 16:08:37 2010
Return-Path: <general-return-1851-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 61847 invoked from network); 21 Jul 2010 16:08:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 16:08:36 -0000
Received: (qmail 19667 invoked by uid 500); 21 Jul 2010 16:08:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19214 invoked by uid 500); 21 Jul 2010 16:08:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 2657 invoked by uid 99); 21 Jul 2010 15:58:08 -0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:x-mimeole:content-class:mime-version:
	content-type:subject:date:message-id:x-ms-has-attach:
	x-ms-tnef-correlator:thread-topic:thread-index:from:to:return-path:x-originalarrivaltime;
	b=BHZI923ZMRa35NxOXiIsGuIOTg+xArCydWt82sUDeHzoZ7NL34GyNztVbIBeHb8f
x-mimeole: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01CB28ED.207B7DAE"
Subject: Hadoop User Group - July Meeting
Date: Wed, 21 Jul 2010 08:55:22 -0700
Message-ID: <C3F64A13EF9E8F4A97A5E60F2D820DE606110F9F@BUR-EXVS03.ds.corp.yahoo.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Hadoop User Group - July Meeting
Thread-Index: AcsocGVLaYiC1TtBTUmICtz3JYC2zw==
From: "Susheel Kaushik" <susheel@yahoo-inc.com>
To: <general@hadoop.apache.org>
X-OriginalArrivalTime: 21 Jul 2010 15:55:45.0410 (UTC) FILETIME=[2DFCE620:01CB28ED]
X-Virus-Checked: Checked by ClamAV on apache.org

------_=_NextPart_001_01CB28ED.207B7DAE
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hello Hadoopers,

=20

Reminder that July Hadoop Users Group is tomorrow.

=20

Agenda:

=20

* 6:00 - 6:30 - Socializing and Beers

* 6:30 - 7:00 - Online Content Optimization with Hadoop, Nitin Motgi,
Yahoo!=20

* 7:00 - 7:30 - Hadoop at eBay, Anil Madan, eBay=20

* 7:30 - 8:00 - Introduction to Avro, Doug Cutting, Cloudera=20

* QnA , Open Discussion

=20

Session details at http://www.meetup.com/hadoop/calendar/13546804/.
(Please register and RSVP in case you have not done so)=20

=20

Looking forward to seeing you there!

=20

Susheel

=20

PS: In case you miss the event, the slides and videos will be posted at
http://developer.yahoo.net/blogs/hadoop/=20

=20


------_=_NextPart_001_01CB28ED.207B7DAE--

From general-return-1852-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 16:27:11 2010
Return-Path: <general-return-1852-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67361 invoked from network); 21 Jul 2010 16:27:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 16:27:11 -0000
Received: (qmail 63806 invoked by uid 500); 21 Jul 2010 16:27:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63709 invoked by uid 500); 21 Jul 2010 16:27:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63701 invoked by uid 99); 21 Jul 2010 16:27:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 16:27:09 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [208.65.144.80] (HELO p02c11o147.mxlogic.net) (208.65.144.80)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 16:27:02 +0000
Received: from unknown [63.251.237.3] (EHLO p02c11o147.mxlogic.net)
	by p02c11o147.mxlogic.net(mxl_mta-6.7.0-0)
	with ESMTP id 2cf174c4.66771940.3105.00-565.7126.p02c11o147.mxlogic.net (envelope-from <wayne@mellanox.com>);
	Wed, 21 Jul 2010 10:26:42 -0600 (MDT)
X-MXL-Hash: 4c471fc24db77a2f-674b1332d8dff54185c5ec7ef0779f9461a82b68
Received: from unknown [63.251.237.3] (EHLO mtiexch01.mti.com)
	by p02c11o147.mxlogic.net(mxl_mta-6.7.0-0)
	with ESMTP id 9fd174c4.0.1745.00-372.3976.p02c11o147.mxlogic.net (envelope-from <wayne@mellanox.com>);
	Wed, 21 Jul 2010 10:20:06 -0600 (MDT)
X-MXL-Hash: 4c471e361d90bed7-b600a4ea528bcb3693d8d4e0386dad5c3f85ecf8
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: RE: Hadoop User Group - July Meeting
Date: Wed, 21 Jul 2010 09:19:05 -0700
Message-ID: <1E3DCD1C63492545881FACB6063A57C1061C54CB@mtiexch01.mti.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Hadoop User Group - July Meeting
Thread-Index: AcsocGVLaYiC1TtBTUmICtz3JYC2zwAf/Uqg
References: <C3F64A13EF9E8F4A97A5E60F2D820DE606110F9F@BUR-EXVS03.ds.corp.yahoo.com>
From: "Wayne Augsburger" <Wayne@Mellanox.com>
To: <general@hadoop.apache.org>
X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010070601)]
X-MAIL-FROM: <wayne@mellanox.com>
X-SOURCE-IP: [63.251.237.3]
X-AnalysisOut: [v=1.0 c=1 a=MtLyulyNObIA:10 a=VphdPIyG4kEA:10 a=kj9zAlcOel]
X-AnalysisOut: [0A:10 a=xupnbh4h0YLOHZnncC45HQ==:17 a=AnZFoEh2AAAA:8 a=mV9]
X-AnalysisOut: [VRH-2AAAA:8 a=kYD52I4EAAAA:8 a=oxCZ_N96AAAA:8 a=bk71vwnq-T]
X-AnalysisOut: [KedbdFk8YA:9 a=6SjYJCSUzWfVMzLBd3C3sFu8QwsA:4 a=CjuIK1q_8u]
X-AnalysisOut: [gA:10 a=-NOWlJdPJ3YA:10]
X-Virus-Checked: Checked by ClamAV on apache.org

Is it tonight or tomorrow?  Owen's email said tonight.

Thanks,
Wayne



-----Original Message-----
From: Susheel Kaushik [mailto:susheel@yahoo-inc.com]=20
Sent: Wednesday, July 21, 2010 8:55 AM
To: general@hadoop.apache.org
Subject: Hadoop User Group - July Meeting

Hello Hadoopers,

=20

Reminder that July Hadoop Users Group is tomorrow.

=20

Agenda:

=20

* 6:00 - 6:30 - Socializing and Beers

* 6:30 - 7:00 - Online Content Optimization with Hadoop, Nitin Motgi,
Yahoo!=20

* 7:00 - 7:30 - Hadoop at eBay, Anil Madan, eBay=20

* 7:30 - 8:00 - Introduction to Avro, Doug Cutting, Cloudera=20

* QnA , Open Discussion

=20

Session details at http://www.meetup.com/hadoop/calendar/13546804/.
(Please register and RSVP in case you have not done so)=20

=20

Looking forward to seeing you there!

=20

Susheel

=20

PS: In case you miss the event, the slides and videos will be posted at
http://developer.yahoo.net/blogs/hadoop/=20

=20


From general-return-1853-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 16:34:09 2010
Return-Path: <general-return-1853-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69417 invoked from network); 21 Jul 2010 16:34:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 16:34:08 -0000
Received: (qmail 76199 invoked by uid 500); 21 Jul 2010 16:34:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76138 invoked by uid 500); 21 Jul 2010 16:34:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76130 invoked by uid 99); 21 Jul 2010 16:34:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 16:34:06 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [209.191.86.156] (HELO n8.bullet.mail.mud.yahoo.com) (209.191.86.156)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 21 Jul 2010 16:33:58 +0000
Received: from [68.142.200.224] by n8.bullet.mail.mud.yahoo.com with NNFMP; 21 Jul 2010 16:32:22 -0000
Received: from [216.252.111.168] by t5.bullet.mud.yahoo.com with NNFMP; 21 Jul 2010 16:32:21 -0000
Received: from [127.0.0.1] by omp103.mail.re3.yahoo.com with NNFMP; 21 Jul 2010 16:31:21 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 848545.61016.bm@omp103.mail.re3.yahoo.com
Received: (qmail 17654 invoked by uid 60001); 21 Jul 2010 16:31:21 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1279729881; bh=Spsj/WQx6KzOxi/+0wcihFAn1gG8UfN619twpwuNNJI=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=pmgzf1qpVNZ42aYGMqf7UdEgFX/lfNRs0jwb3SaYRbLHesX/yYtS7aQCddAlw15ffSeF6zFOFq+llvUvCfunMyRqfRTLWLLpuP3CS57HmblS1iG3fuRgIsYVUIP0ZTt6XCznkTdw8ewDZi+0ePlv0YiHZpKot8wFNFOdpfnTSKw=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=yf3/8u47rAXDSQQ+HFIYtpYbb1Dvfd07F8Gza6ZzDgG/9MtA6Wp3Q8GV2U/kpAv/elI2z5lh5X/eqFNulZebLxAXpSwhxSY58KsgeUk3ZES9Aduf6MSwllsEPYqeR9kFtg2gZ26WYHWsLnW0+ODBpr1ymVq6jKFlB2aesLJ175Y=;
Message-ID: <750244.17617.qm@web56202.mail.re3.yahoo.com>
X-YMail-OSG: Io.8YBsVM1l276xByUXZ65yK0.dwu.l26kJ4K4Om9E4MW6g
 h4_D0icLa1BYzzp1XTg2NUeIV9ltH6DjChqFr6vzkoFgorfOl6zpOUYCSbst
 m_B1CLgJ9xX7x5nAHe2kVYHnpmiVZeh1e6AqbsfElaAl_CyipGIauQtapYN5
 g.XyXYH2opEy40b_PobKdDzJ7C0nnjwMBbXhMMs_MEUH5pJNy9pymyYdVpR6
 kfXqgC4Y4YF243etQDJyl0vKFr3hJo.8iRfvemzmAitXVrye6vPFNpnX.b64
 G0QGNQvYydOmNqW_Qndq6JpghHBxHTdFkrwlvk.G._ATnR.X3kOHAFaz6DUD
 0_h0MbqBbc61qC.MYNQNF
Received: from [216.145.54.158] by web56202.mail.re3.yahoo.com via HTTP; Wed, 21 Jul 2010 09:31:21 PDT
X-Mailer: YahooMailRC/420.4 YahooMailWebService/0.8.104.276605
References: <C3F64A13EF9E8F4A97A5E60F2D820DE606110F9F@BUR-EXVS03.ds.corp.yahoo.com>
Date: Wed, 21 Jul 2010 09:31:21 -0700 (PDT)
From: "Tsz Wo \(Nicholas\), Sze" <s29752-hadoopgeneral@yahoo.com>
Subject: Re: Hadoop User Group - July Meeting
To: general@hadoop.apache.org
In-Reply-To: <C3F64A13EF9E8F4A97A5E60F2D820DE606110F9F@BUR-EXVS03.ds.corp.yahoo.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org

I think you mean to say July Hadoop Users Group is July 21 *today*.

Nicholas



----- Original Message ----
> From: Susheel Kaushik <susheel@yahoo-inc.com>
> To: general@hadoop.apache.org
> Sent: Wed, July 21, 2010 8:55:22 AM
> Subject: Hadoop User Group - July Meeting
> 
> Hello Hadoopers,
> 
> 
> 
> Reminder that July Hadoop Users Group is  tomorrow.
> 
> 
> 
> Agenda:
> 
> 
> 
> * 6:00 - 6:30 - Socializing  and Beers
> 
> * 6:30 - 7:00 - Online Content Optimization with Hadoop, Nitin  Motgi,
> Yahoo! 
> 
> * 7:00 - 7:30 - Hadoop at eBay, Anil Madan, eBay 
> 
> * 7:30 - 8:00 - Introduction to Avro, Doug Cutting, Cloudera 
> 
> *  QnA , Open Discussion
> 
> 
> 
> Session details at  http://www.meetup.com/hadoop/calendar/13546804/.
> (Please register and RSVP in  case you have not done so) 
> 
> 
> 
> Looking forward to seeing you  there!
> 
> 
> 
> Susheel
> 
> 
> 
> PS: In case you miss the event,  the slides and videos will be posted  at
> http://developer.yahoo.net/blogs/hadoop/ 
> 
> 
> 
> 


From general-return-1854-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 16:49:39 2010
Return-Path: <general-return-1854-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73112 invoked from network); 21 Jul 2010 16:49:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 16:49:39 -0000
Received: (qmail 4872 invoked by uid 500); 21 Jul 2010 16:49:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4773 invoked by uid 500); 21 Jul 2010 16:49:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4765 invoked by uid 99); 21 Jul 2010 16:49:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 16:49:37 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.62.17] (HELO qmta10.westchester.pa.mail.comcast.net) (76.96.62.17)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 16:49:27 +0000
Received: from omta07.westchester.pa.mail.comcast.net ([76.96.62.59])
	by qmta10.westchester.pa.mail.comcast.net with comcast
	id kbfj1e0041GhbT85Agp7dS; Wed, 21 Jul 2010 16:49:07 +0000
Received: from [10.0.0.13] ([209.131.62.115])
	by omta07.westchester.pa.mail.comcast.net with comcast
	id kgox1e00R2VBGtd3Tgp0MQ; Wed, 21 Jul 2010 16:49:05 +0000
Message-Id: <5F4B533D-608A-4C3F-929E-9C95B86FCE71@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <1E3DCD1C63492545881FACB6063A57C1061C54CB@mtiexch01.mti.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Hadoop User Group - July Meeting
Date: Wed, 21 Jul 2010 09:48:56 -0700
References: <C3F64A13EF9E8F4A97A5E60F2D820DE606110F9F@BUR-EXVS03.ds.corp.yahoo.com> <1E3DCD1C63492545881FACB6063A57C1061C54CB@mtiexch01.mti.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


On Jul 21, 2010, at 9:19 AM, Wayne Augsburger wrote:

> Is it tonight or tomorrow?  Owen's email said tonight.

Susheel's message got stuck in the Apache servers for a day. I sent  
mine because his hadn't come through.

The meeting is *TONIGHT*.

-- Owen

From general-return-1855-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 17:16:01 2010
Return-Path: <general-return-1855-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84335 invoked from network); 21 Jul 2010 17:16:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 17:16:01 -0000
Received: (qmail 44547 invoked by uid 500); 21 Jul 2010 17:16:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44381 invoked by uid 500); 21 Jul 2010 17:15:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44373 invoked by uid 99); 21 Jul 2010 17:15:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 17:15:59 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mdwasti@hotmail.com designates 65.54.190.29 as permitted sender)
Received: from [65.54.190.29] (HELO bay0-omc1-s18.bay0.hotmail.com) (65.54.190.29)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 17:15:51 +0000
Received: from BAY145-W22 ([65.54.190.61]) by bay0-omc1-s18.bay0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Wed, 21 Jul 2010 10:15:29 -0700
Message-ID: <BAY145-w22E138C921BF8B5E6BE89DBEA10@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_d38e2a41-cd5a-4d55-92e2-7be0be4f394c_"
X-Originating-IP: [208.84.47.48]
From: Syed Wasti <mdwasti@hotmail.com>
To: Hadoop <general@hadoop.apache.org>
Subject: Lazy initialization of Reducers
Date: Wed, 21 Jul 2010 10:15:29 -0700
Importance: Normal
MIME-Version: 1.0
X-OriginalArrivalTime: 21 Jul 2010 17:15:29.0882 (UTC) FILETIME=[51C163A0:01CB28F8]
X-Virus-Checked: Checked by ClamAV on apache.org

--_d38e2a41-cd5a-4d55-92e2-7be0be4f394c_
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable


Hi=2C

I read about this Reducer Lazy initialization in a document found in the be=
low URL.

http://www.scribd.com/doc/23046928/Hadoop-Performance-Tuning



It says =93:In M/R job Reducers are initialized with Mappers at the job ini=
tialization=2C but the reduce method is called in reduce phase when all the=
 maps had been finished. So in large jobs where Reducer loads data (>100 MB=
 for business logic) in-memory on initialization=2C the performance can be =
increased by lazily initializing Reducers i.e. loading data in reduce metho=
d controlled by an initialize flag variable which assures that it is loaded=
 only once. By lazily initializing Reducers which require memory (for busin=
ess logic) on initialization=2C number of maps can be increased.=94



But I did not find any other resource which talks about Reducer Lazy initia=
lization.

Does anyone have experience on this ?

If yes=2C how and where can I set this parameter to get it working.



Thanks for the support.


Regards
Syed Wasti


 		 	   		  =

--_d38e2a41-cd5a-4d55-92e2-7be0be4f394c_--

From general-return-1856-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 17:39:58 2010
Return-Path: <general-return-1856-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94644 invoked from network); 21 Jul 2010 17:39:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 17:39:57 -0000
Received: (qmail 79647 invoked by uid 500); 21 Jul 2010 17:39:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79597 invoked by uid 500); 21 Jul 2010 17:39:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 79589 invoked by uid 99); 21 Jul 2010 17:39:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 17:39:55 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 17:39:49 +0000
Received: by qwd7 with SMTP id 7so3890407qwd.35
        for <general@hadoop.apache.org>; Wed, 21 Jul 2010 10:38:39 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=UWywDlUHJJNttr0Zk7/6OWO/ZBcApppmTQIbpgMNdKw=;
        b=d30tIYd2Pto8l80R0WlNG8/nzXh9rCCmNqCpdz5CMnRPByRDx5jzMUvBzrvL6V3aVS
         JKJ24AYTJxqfvwX/qWkeRHjfPcltILFAxH930Fa9kAc3GW1sSgaORDLF+hINvqDijuVx
         zsdMq4j0WKPlDjsyCqcTlHtigIofJQ0etf+Gw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=HI3FCB0v1kMrwaQLc6pyK9Ej3ybYO4+BFokoldAOUOiWmZ5XDTyXEbDcSs+k16kXt3
         HPYZFpXnomNVc7TJUDHJAgqVhiirOvXRpbpmLNrubdo7sN9feAlzvJJ1yftrw7uTtMKa
         6lmnh5Y9JekqGnClS+WU1mV+9Zv1IMX14K40E=
MIME-Version: 1.0
Received: by 10.224.98.81 with SMTP id p17mr363994qan.290.1279733919522; Wed, 
	21 Jul 2010 10:38:39 -0700 (PDT)
Received: by 10.229.75.85 with HTTP; Wed, 21 Jul 2010 10:38:37 -0700 (PDT)
In-Reply-To: <BAY145-w22E138C921BF8B5E6BE89DBEA10@phx.gbl>
References: <BAY145-w22E138C921BF8B5E6BE89DBEA10@phx.gbl>
Date: Wed, 21 Jul 2010 10:38:37 -0700
Message-ID: <AANLkTikuAFcts2wvUX9u7OL60glOPdKk9AtTHqtPsAjm@mail.gmail.com>
Subject: Re: Lazy initialization of Reducers
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00c09f89923eae2241048be94507
X-Virus-Checked: Checked by ClamAV on apache.org

--00c09f89923eae2241048be94507
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

I don't find such parameter in 0.20.2

Please create such flag in your own class.

On Wed, Jul 21, 2010 at 10:15 AM, Syed Wasti <mdwasti@hotmail.com> wrote:

>
> Hi,
>
> I read about this Reducer Lazy initialization in a document found in the
> below URL.
>
> http://www.scribd.com/doc/23046928/Hadoop-Performance-Tuning
>
>
>
> It says =93:In M/R job Reducers are initialized with Mappers at the job
> initialization, but the reduce method is called in reduce phase when all =
the
> maps had been finished. So in large jobs where Reducer loads data (>100 M=
B
> for business logic) in-memory on initialization, the performance can be
> increased by lazily initializing Reducers i.e. loading data in reduce met=
hod
> controlled by an initialize flag variable which assures that it is loaded
> only once. By lazily initializing Reducers which require memory (for
> business logic) on initialization, number of maps can be increased.=94
>
>
>
> But I did not find any other resource which talks about Reducer Lazy
> initialization.
>
> Does anyone have experience on this ?
>
> If yes, how and where can I set this parameter to get it working.
>
>
>
> Thanks for the support.
>
>
> Regards
> Syed Wasti
>
>
>

--00c09f89923eae2241048be94507--

From general-return-1857-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 18:29:56 2010
Return-Path: <general-return-1857-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22807 invoked from network); 21 Jul 2010 18:29:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 18:29:56 -0000
Received: (qmail 45099 invoked by uid 500); 21 Jul 2010 18:29:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44847 invoked by uid 500); 21 Jul 2010 18:29:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44839 invoked by uid 99); 21 Jul 2010 18:29:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 18:29:54 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of msegel@navteq.com designates 204.120.70.37 as permitted sender)
Received: from [204.120.70.37] (HELO xmailfargo.navteq.com) (204.120.70.37)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 18:29:48 +0000
Received: from fargo-spamfw.navteq.com (navteq-spamfw.navteq.com [10.6.10.131])
	by xmailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id o6LI5AxK030079
	for <general@hadoop.apache.org>; Wed, 21 Jul 2010 13:05:10 -0500
Received: from imailfargo.navteq.com (localhost [127.0.0.1])
	by fargo-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id DF46B4871FEA
	for <general@hadoop.apache.org>; Wed, 21 Jul 2010 13:26:17 -0500 (CDT)
Received: from imailfargo.navteq.com (imailfargo.navteq.com [10.6.10.130]) by fargo-spamfw.navteq.com with ESMTP id 5o4QpWtDWRDAAPX3 for <general@hadoop.apache.org>; Wed, 21 Jul 2010 13:26:17 -0500 (CDT)
Received: from hq-ex-ht02.ad.navteq.com ([10.8.222.172])
	by imailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id o6LIQHiL022080
	for <general@hadoop.apache.org>; Wed, 21 Jul 2010 13:26:17 -0500
Received: from hq-ex-mb02.ad.navteq.com ([fe80::880d:318e:e860:a8db]) by
 hq-ex-ht02.ad.navteq.com ([fe80::24f6:ca50:734d:df6e%14]) with mapi; Wed, 21
 Jul 2010 13:26:17 -0500
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 21 Jul 2010 13:26:15 -0500
Subject: RE: Hadoop User Group - July Meeting
Thread-Topic: Hadoop User Group - July Meeting
Thread-Index: Acso8ouWor92iB1nR82uwL3Tul2hxwAD406w
Message-ID: <ABC24175AFD3BE4DA15F4CD375ED413D0614A94700@hq-ex-mb02.ad.navteq.com>
References: <C3F64A13EF9E8F4A97A5E60F2D820DE606110F9F@BUR-EXVS03.ds.corp.yahoo.com>
 <750244.17617.qm@web56202.mail.re3.yahoo.com>
In-Reply-To: <750244.17617.qm@web56202.mail.re3.yahoo.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1
X-Virus-Checked: Checked by ClamAV on apache.org

Not to confuse everyone but yes today is also Chicago's area Hadoop User =
Group meeting too. ;-)
Ok so it's a shameless plug. :-)

-Mike


-----Original Message-----
From: Tsz Wo (Nicholas), Sze [mailto:s29752-hadoopgeneral@yahoo.com]=20
Sent: Wednesday, July 21, 2010 11:31 AM
To: general@hadoop.apache.org
Subject: Re: Hadoop User Group - July Meeting

I think you mean to say July Hadoop Users Group is July 21 *today*.

Nicholas



----- Original Message ----
> From: Susheel Kaushik <susheel@yahoo-inc.com>
> To: general@hadoop.apache.org
> Sent: Wed, July 21, 2010 8:55:22 AM
> Subject: Hadoop User Group - July Meeting
>=20
> Hello Hadoopers,
>=20
>=20
>=20
> Reminder that July Hadoop Users Group is  tomorrow.
>=20
>=20
>=20
> Agenda:
>=20
>=20
>=20
> * 6:00 - 6:30 - Socializing  and Beers
>=20
> * 6:30 - 7:00 - Online Content Optimization with Hadoop, Nitin  Motgi,
> Yahoo!=20
>=20
> * 7:00 - 7:30 - Hadoop at eBay, Anil Madan, eBay=20
>=20
> * 7:30 - 8:00 - Introduction to Avro, Doug Cutting, Cloudera=20
>=20
> *  QnA , Open Discussion
>=20
>=20
>=20
> Session details at  http://www.meetup.com/hadoop/calendar/13546804/.
> (Please register and RSVP in  case you have not done so)=20
>=20
>=20
>=20
> Looking forward to seeing you  there!
>=20
>=20
>=20
> Susheel
>=20
>=20
>=20
> PS: In case you miss the event,  the slides and videos will be posted  =
at
> http://developer.yahoo.net/blogs/hadoop/=20
>=20
>=20
>=20
>=20



The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.

From general-return-1858-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 22:24:56 2010
Return-Path: <general-return-1858-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19972 invoked from network); 21 Jul 2010 22:24:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 22:24:55 -0000
Received: (qmail 53440 invoked by uid 500); 21 Jul 2010 22:24:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53374 invoked by uid 500); 21 Jul 2010 22:24:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53359 invoked by uid 99); 21 Jul 2010 22:24:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 22:24:52 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 22:24:46 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o6LMN0cR072600;
	Wed, 21 Jul 2010 15:23:00 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:subject:mime-version:date:references:x-mailer;
	b=lL7kn//yEz4k7mGVCK3PfnBnvS9Zd7WUJgvI41vV1CFc7u92WpRXU61NIW/WwnUe
Message-Id: <4C051910-DC55-4D7A-90E6-8B03850630CC@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: mapreduce-user@hadoop.apache.org
In-Reply-To: <BAY145-w22E138C921BF8B5E6BE89DBEA10@phx.gbl>
Content-Type: text/plain; charset=WINDOWS-1252; format=flowed; delsp=yes
Content-Transfer-Encoding: quoted-printable
Subject: Re: Lazy initialization of Reducers
Mime-Version: 1.0 (Apple Message framework v936)
Date: Wed, 21 Jul 2010 15:22:59 -0700
References: <BAY145-w22E138C921BF8B5E6BE89DBEA10@phx.gbl>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

Moving to mapreduce-user@, bcc general@. Please do not use the =20
general@ list for project specific discussions.


On Jul 21, 2010, at 10:15 AM, Syed Wasti wrote:
> It says =93:In M/R job Reducers are initialized with Mappers at the =20=

> job initialization, but the reduce method is called in reduce phase =20=

> when all the maps had been finished. So in large jobs where Reducer =20=

> loads data (>100 MB for business logic) in-memory on initialization, =20=

> the performance can be increased by lazily initializing Reducers =20
> i.e. loading data in reduce method controlled by an initialize flag =20=

> variable which assures that it is loaded only once. By lazily =20
> initializing Reducers which require memory (for business logic) on =20
> initialization, number of maps can be increased.=94

The part about 'loading data in reduce method controlled by an =20
initialize flag variable which assures that it is loaded only once' =20
makes no sense to me.

However, you can 'slowstart' reduces by ensuring sufficient maps are =20
complete before _any_ reduces are launched... from mapred-default.xml:

<property>
   <name>mapred.reduce.slowstart.completed.maps</name>
   <value>0.05</value>
   <description>Fraction of the number of maps in the job which should =20=

be
   complete before reduces are scheduled for the job.
   </description>
</property>

Arun



From general-return-1859-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 22 18:25:08 2010
Return-Path: <general-return-1859-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 3817 invoked from network); 22 Jul 2010 18:25:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jul 2010 18:25:07 -0000
Received: (qmail 59448 invoked by uid 500); 22 Jul 2010 18:25:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59271 invoked by uid 500); 22 Jul 2010 18:25:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59262 invoked by uid 99); 22 Jul 2010 18:25:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 18:25:05 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mdwasti@hotmail.com designates 65.54.190.13 as permitted sender)
Received: from [65.54.190.13] (HELO bay0-omc1-s2.bay0.hotmail.com) (65.54.190.13)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 18:24:57 +0000
Received: from BAY145-W56 ([65.54.190.61]) by bay0-omc1-s2.bay0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Thu, 22 Jul 2010 11:24:36 -0700
Message-ID: <BAY145-w5603FAE32A12B4317DFC97BEA20@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_0ab71581-0189-4388-8db0-6de27b862eae_"
X-Originating-IP: [208.84.47.48]
From: Syed Wasti <mdwasti@hotmail.com>
To: Hadoop <general@hadoop.apache.org>
Subject: TaskTracker Error
Date: Thu, 22 Jul 2010 11:24:36 -0700
Importance: Normal
MIME-Version: 1.0
X-OriginalArrivalTime: 22 Jul 2010 18:24:36.0848 (UTC) FILETIME=[23F3EF00:01CB29CB]

--_0ab71581-0189-4388-8db0-6de27b862eae_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


Hi=2C
I am seeing the below two errors consistently while running my map/reduce j=
obs. This does not stop my script=2C but throws an error on datanode and su=
ccessfully completes the same task on a different datanode. The task could =
fail on any datanode=2C there are no blacklisted datanodes.
Any thoughts on why this is and how can this be resolved ?

This is the whole stack=3B
java.lang.Throwable: Child Error
	at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:472)
Caused by: java.io.IOException: Task process exit with nonzero status of 13=
4.
	at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:459)


Second Error: This is all it gives.

Error: null
Thanks for the support.
Regards
Syed Wasti


 		 	   		  =

--_0ab71581-0189-4388-8db0-6de27b862eae_--

From general-return-1860-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 22 20:54:49 2010
Return-Path: <general-return-1860-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58985 invoked from network); 22 Jul 2010 20:54:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jul 2010 20:54:49 -0000
Received: (qmail 91355 invoked by uid 500); 22 Jul 2010 20:54:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91307 invoked by uid 500); 22 Jul 2010 20:54:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91292 invoked by uid 99); 22 Jul 2010 20:54:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 20:54:47 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mdwasti@hotmail.com designates 65.54.190.161 as permitted sender)
Received: from [65.54.190.161] (HELO bay0-omc3-s23.bay0.hotmail.com) (65.54.190.161)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 20:54:40 +0000
Received: from BAY145-W33 ([65.54.190.189]) by bay0-omc3-s23.bay0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Thu, 22 Jul 2010 13:54:19 -0700
Message-ID: <BAY145-w338720719FD00C11E70824BEA20@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_a25434ea-45bf-4e27-b27d-934be2688525_"
X-Originating-IP: [208.84.47.48]
From: Syed Wasti <mdwasti@hotmail.com>
To: pig user <pig-user@hadoop.apache.org>, Hadoop <general@hadoop.apache.org>
Subject: RE: TaskTracker Error
Date: Thu, 22 Jul 2010 13:54:20 -0700
Importance: Normal
In-Reply-To: <BAY145-w296E5AFF098ECADBDEEA5BEA20@phx.gbl>
References:
 <BAY145-w5603FAE32A12B4317DFC97BEA20@phx.gbl>,<BAY145-w296E5AFF098ECADBDEEA5BEA20@phx.gbl>
MIME-Version: 1.0
X-OriginalArrivalTime: 22 Jul 2010 20:54:19.0630 (UTC) FILETIME=[0E1BB0E0:01CB29E0]

--_a25434ea-45bf-4e27-b27d-934be2688525_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


Found this almost a year old discussion=2C on the same error code and it lo=
oks like a JVM crash.=20
The discussion confirms the crash on JDK 14 and I am using JDK version 18.=
=20



http://markmail.org/message/74yv7zyuq7ucm2z6


From: mdwasti@hotmail.com
To: pig-user@hadoop.apache.org
Subject: TaskTracker Error
Date: Thu=2C 22 Jul 2010 11:45:38 -0700








Hi=2C
I am seeing the below two errors consistently while running my script. This=
 does not stop my script=2C but throws an error on datanode and successfull=
y completes the same task on a different datanode. The task could fail on a=
ny datanode=2C there are no blacklisted datanodes.
Any thoughts on why this is and how can this be resolved ?
=20
This is the whole stack=3B
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:472)
Caused by: java.io.IOException: Task process exit with nonzero status of 13=
4.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:459)
=20
Second Error: This is all it gives.
Error: null

Third Error:
Error: (class: org/apache/pig/builtin/PigStorage=2C method: setLocation sig=
nature: (Ljava/lang/String=3BLorg/apache/hadoop/mapreduce/Job=3B)V)=20


Thanks for the support.
Regards
Syed Wasti
 		 	   		  =

--_a25434ea-45bf-4e27-b27d-934be2688525_--

From general-return-1861-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 14:52:25 2010
Return-Path: <general-return-1861-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19785 invoked from network); 26 Jul 2010 14:52:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 14:52:25 -0000
Received: (qmail 5093 invoked by uid 500); 26 Jul 2010 14:52:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4802 invoked by uid 500); 26 Jul 2010 14:52:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4794 invoked by uid 99); 26 Jul 2010 14:52:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 14:52:23 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [156.148.72.33] (HELO raffaello.crs4.it) (156.148.72.33)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 14:52:16 +0000
Received: from [156.148.71.202] (neuron.crs4.it [156.148.71.202])
	by raffaello.crs4.it (Postfix) with ESMTP id B49906BBA1
	for <general@hadoop.apache.org>; Mon, 26 Jul 2010 16:51:07 +0200 (CEST)
Message-ID: <4C4DA0C2.70208@crs4.it>
Date: Mon, 26 Jul 2010 16:50:42 +0200
From: Simone Leo <simone.leo@crs4.it>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.10) Gecko/20100629 Thunderbird/3.0.5
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Announcing Pydoop 0.3.6_rc2
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 8bit

Hi,

we've just released version 0.3.6_rc2 of Pydoop
(http://pydoop.sourceforge.net). Pydoop is a Python MapReduce and HDFS
API for Hadoop, built upon the C++ Pipes and the C libhdfs APIs, that
allows to write full-fledged MapReduce applications with HDFS access.
Its key features are:

 * access to most MapReduce application components: Mapper, Reducer,
RecordReader, RecordWriter, Partitioner;
 * direct access to JobConf parameters; support for counters and status
messages;
 * CPython implementation: any Python module can be used, either pure
Python or C/C++ extension (note that this is not possible with Jython);
 * Direct HDFS access from Python.

With Pydoop you can write complete applications in Python, using a
programming style that's very similar to the one supported by the Java
and C++ APIs: developers define classes that are instantiated and used
by the framework. This allows for much cleaner and faster [1] code with
respect to the traditional Python + Streaming approach.

See http://pydoop.sourceforge.net/docs/examples for a collection of
Pydoop usage examples, including a complete application that leverages
the Hadoop Distributed Cache to distribute all required Python packages,
including Pydoop itself, to Hadoop cluster nodes.

Pydoop is actively used in production at our site, mostly for
data-intensive biocomputing applications.

The 0.3.6_rc2 release is being used internally in production. We'd
greatly appreciate any kind of feedback before we release it as 0.3.6
(stable), which we expect to do within two weeks or so.

Links:

 * download page: http://sourceforge.net/projects/pydoop/files
 * release notes:
http://sourceforge.net/apps/mediawiki/pydoop/index.php?title=Release_Notes

[1] Simone Leo and Gianluigi Zanetti. "Pydoop: a Python MapReduce and
HDFS API for Hadoop". In Proceedings of the 19th ACM International
Symposium on High Performance Distributed Computing (HPDC 2010), pages
819–825. ACM, 2010.


-- 
Simone Leo
Distributed Computing group
Advanced Computing and Communications program
CRS4
POLARIS - Building #1
Piscina Manna
I-09010 Pula (CA) - Italy
e-mail: simleo@crs4.it
http://www.crs4.it

From general-return-1862-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 16:03:06 2010
Return-Path: <general-return-1862-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78239 invoked from network); 26 Jul 2010 16:03:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 16:03:05 -0000
Received: (qmail 20862 invoked by uid 500); 26 Jul 2010 16:03:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20739 invoked by uid 500); 26 Jul 2010 16:03:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 20731 invoked by uid 99); 26 Jul 2010 16:03:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 16:03:04 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of michael_segel@hotmail.com designates 65.55.34.26 as permitted sender)
Received: from [65.55.34.26] (HELO col0-omc1-s16.col0.hotmail.com) (65.55.34.26)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 16:02:57 +0000
Received: from COL117-W15 ([65.55.34.8]) by col0-omc1-s16.col0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Mon, 26 Jul 2010 09:02:36 -0700
Message-ID: <COL117-W15C85FC4B3465C944794C88FA60@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_d2511a14-1382-42ed-8104-6c2372ba35a2_"
X-Originating-IP: [65.167.11.254]
From: Michael Segel <michael_segel@hotmail.com>
To: <general@hadoop.apache.org>
Subject: Fair Scheduler documentation?
Date: Mon, 26 Jul 2010 11:02:36 -0500
Importance: Normal
MIME-Version: 1.0
X-OriginalArrivalTime: 26 Jul 2010 16:02:36.0504 (UTC) FILETIME=[F7159D80:01CB2CDB]

--_d2511a14-1382-42ed-8104-6c2372ba35a2_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


Hi=2C

Ok so we're setting up the fair scheduler on one of our clusters.=20

The only documentation I could find is the wiki=2C and then one or two powe=
r points...

Is there any source that contains all of the possible options allowed in th=
e allocations.xml file?

I came across something like <maxMaps> somewhere the other day.. couldn't f=
ind it. Not sure if its a valid
option.

Also is there any logged output when the allocations file is read and there=
's a bad xml parameter?

Thx

-Mike

 		 	   		 =20
_________________________________________________________________
Hotmail has tools for the New Busy. Search=2C chat and e-mail from your inb=
ox.
http://www.windowslive.com/campaign/thenewbusy?ocid=3DPID28326::T:WLMTAGL:O=
N:WL:en-US:WM_HMP:042010_1=

--_d2511a14-1382-42ed-8104-6c2372ba35a2_--

From general-return-1863-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 16:14:23 2010
Return-Path: <general-return-1863-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85570 invoked from network); 26 Jul 2010 16:14:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 16:14:23 -0000
Received: (qmail 32366 invoked by uid 500); 26 Jul 2010 16:14:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32293 invoked by uid 500); 26 Jul 2010 16:14:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32285 invoked by uid 99); 26 Jul 2010 16:14:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 16:14:21 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of michael_segel@hotmail.com designates 65.55.34.12 as permitted sender)
Received: from [65.55.34.12] (HELO col0-omc1-s2.col0.hotmail.com) (65.55.34.12)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 16:14:14 +0000
Received: from COL117-W57 ([65.55.34.7]) by col0-omc1-s2.col0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Mon, 26 Jul 2010 09:13:53 -0700
Message-ID: <COL117-W57E09052147A0DB3F567368FA60@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_835800a7-e078-4408-9f09-224a4ced2398_"
X-Originating-IP: [65.167.11.254]
From: Michael Segel <michael_segel@hotmail.com>
To: <general@hadoop.apache.org>
Subject: RE: Fair Scheduler documentation?
Date: Mon, 26 Jul 2010 11:13:53 -0500
Importance: Normal
In-Reply-To: <COL117-W15C85FC4B3465C944794C88FA60@phx.gbl>
References: <COL117-W15C85FC4B3465C944794C88FA60@phx.gbl>
MIME-Version: 1.0
X-OriginalArrivalTime: 26 Jul 2010 16:13:53.0739 (UTC) FILETIME=[8ABF79B0:01CB2CDD]

--_835800a7-e078-4408-9f09-224a4ced2398_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable



> From: michael_segel@hotmail.com
> To: general@hadoop.apache.org
> Subject: Fair Scheduler documentation?
> Date: Mon=2C 26 Jul 2010 11:02:36 -0500
>=20
>=20
> Hi=2C
>=20
> Ok so we're setting up the fair scheduler on one of our clusters.=20
>=20
> The only documentation I could find is the wiki=2C and then one or two po=
wer points...
>=20
> Is there any source that contains all of the possible options allowed in =
the allocations.xml file?
>=20
> I came across something like <maxMaps> somewhere the other day.. couldn't=
 find it. Not sure if its a valid
> option.
[SNIP]




https://issues.apache.org/jira/browse/MAPREDUCE-698
(Found using Yahoo! as my search engine... )

Ok this explains why I can't implement maxMaps on my cloud where I'm testin=
g out the fair scheduler.

Also is there a way to specify the pool to be used at the time the job is l=
aunched?=20

Thx

-Mike

 		 	   		 =20
_________________________________________________________________
The New Busy is not the too busy. Combine all your e-mail accounts with Hot=
mail.
http://www.windowslive.com/campaign/thenewbusy?tile=3Dmultiaccount&ocid=3DP=
ID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_4=

--_835800a7-e078-4408-9f09-224a4ced2398_--

From general-return-1864-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 01:12:58 2010
Return-Path: <general-return-1864-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54149 invoked from network); 28 Jul 2010 01:12:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 01:12:58 -0000
Received: (qmail 64367 invoked by uid 500); 28 Jul 2010 01:12:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 64261 invoked by uid 500); 28 Jul 2010 01:12:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 64253 invoked by uid 99); 28 Jul 2010 01:12:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 01:12:56 +0000
X-ASF-Spam-Status: No, hits=0.6 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.149.205] (HELO na3sys009aog111.obsmtp.com) (74.125.149.205)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 28 Jul 2010 01:12:47 +0000
Received: from source ([209.85.161.41]) by na3sys009aob111.postini.com ([74.125.148.12]) with SMTP
	ID DSNKTE+D+S/pJ5LSiCLLJDZG4l2Fu/Mw1lfP@postini.com; Tue, 27 Jul 2010 18:12:26 PDT
Received: by fxm20 with SMTP id 20so1130999fxm.14
        for <general@hadoop.apache.org>; Tue, 27 Jul 2010 18:12:25 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.239.132.131 with SMTP id 3mr409415hbr.211.1280279543944; Tue, 
	27 Jul 2010 18:12:23 -0700 (PDT)
Received: by 10.239.133.65 with HTTP; Tue, 27 Jul 2010 18:12:23 -0700 (PDT)
Date: Tue, 27 Jul 2010 21:12:23 -0400
Message-ID: <AANLkTimp1bEvRWR7ivOOb2Xkt2w5G_ur-SXuM5qSMn6w@mail.gmail.com>
Subject: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap
From: Sameer Joshi <sameer.joshi@serenesoftware.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f427806e1797048c684fd1
X-Virus-Checked: Checked by ClamAV on apache.org

--001485f427806e1797048c684fd1
Content-Type: text/plain; charset=ISO-8859-1

I installed Java and Hadoop on a GoGrid cloud server using Red Hat
Enterprise Linux Server release 5.1 (Tikanga). Hadoop installed fine and
starts fine, however I get an error (java.lang.NullPointerException at
java.util.concurrent.ConcurrentHashMap) while running the Hadoop wordcount
example. My guess was that this was a localhost or IPv6 issue.

* I have tested replacing 'localhost' with both the local IP, and server IP
addresses (when out of options) in Hadoop conf
* I have disabled IPv6 both in sysctl.conf and hadoop-env.sh (former
followed by a server restart)

Any thoughts?
Thank you.


The output is given below

# bin/hadoop jar hadoop-0.20.2-examples.jar wordcount datasets tests/out7
10/07/27 05:44:59 INFO input.FileInputFormat: Total input paths to process :
1
10/07/27 05:44:59 INFO mapred.JobClient: Running job: job_201007270544_0001
10/07/27 05:45:00 INFO mapred.JobClient: map 0% reduce 0%
10/07/27 05:45:12 INFO mapred.JobClient: map 100% reduce 0%
10/07/27 05:45:17 INFO mapred.JobClient: Task Id :
attempt_201007270544_0001_r_000000_0, Status : FAILED

Error: java.lang.NullPointerException
at java.util.concurrent.ConcurrentHashMap.get(Unknown Source)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)

at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)


10/07/27 05:45:24 INFO mapred.JobClient: Task Id :
attempt_201007270544_0001_r_000000_1, Status : FAILED
Error: java.lang.NullPointerException
at java.util.concurrent.ConcurrentHashMap.get(Unknown Source)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)

at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)


10/07/27 05:45:30 INFO mapred.JobClient: Task Id :
attempt_201007270544_0001_r_000000_2, Status : FAILED
Error: java.lang.NullPointerException
at java.util.concurrent.ConcurrentHashMap.get(Unknown Source)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)

at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)


10/07/27 05:45:39 INFO mapred.JobClient: Job complete: job_201007270544_0001

10/07/27 05:45:39 INFO mapred.JobClient: Counters: 12
10/07/27 05:45:39 INFO mapred.JobClient: Job Counters
10/07/27 05:45:39 INFO mapred.JobClient: Launched reduce tasks=4
10/07/27 05:45:39 INFO mapred.JobClient: Launched map tasks=1
10/07/27 05:45:39 INFO mapred.JobClient: Data-local map tasks=1
10/07/27 05:45:39 INFO mapred.JobClient: Failed reduce tasks=1
10/07/27 05:45:39 INFO mapred.JobClient: FileSystemCounters
10/07/27 05:45:39 INFO mapred.JobClient: HDFS_BYTES_READ=15319
10/07/27 05:45:39 INFO mapred.JobClient: FILE_BYTES_WRITTEN=12847
10/07/27 05:45:39 INFO mapred.JobClient: Map-Reduce Framework
10/07/27 05:45:39 INFO mapred.JobClient: Combine output records=934
10/07/27 05:45:39 INFO mapred.JobClient: Map input records=149
10/07/27 05:45:39 INFO mapred.JobClient: Spilled Records=934
10/07/27 05:45:39 INFO mapred.JobClient: Map output bytes=25346
10/07/27 05:45:39 INFO mapred.JobClient: Combine input records=2541
10/07/27 05:45:39 INFO mapred.JobClient: Map output records=2541

-- 
Dr. Sameer Joshi, Ph.D.
Senior computer scientist,
Serene Software.

--001485f427806e1797048c684fd1--

From general-return-1865-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 04:12:58 2010
Return-Path: <general-return-1865-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97889 invoked from network); 28 Jul 2010 04:12:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 04:12:58 -0000
Received: (qmail 35454 invoked by uid 500); 28 Jul 2010 04:12:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 35086 invoked by uid 500); 28 Jul 2010 04:12:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 35077 invoked by uid 99); 28 Jul 2010 04:12:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 04:12:53 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of sonalgoyal4@gmail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 04:12:48 +0000
Received: by iwn37 with SMTP id 37so2216206iwn.35
        for <general@hadoop.apache.org>; Tue, 27 Jul 2010 21:12:26 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=ivAhL0pD3wNfMChYdZg7A/IVqhUcA3QIYEFmQaonEVQ=;
        b=hSCSz9tBehgSa5ixBvJte3CeuEqB4yIhBThfObsEsd3PHCqOeW1l8G37EOO2svAhXH
         Gx4tLTO+0ByN8ce+om9G3fRckBwSZxKfLiOLLRJdFdYglloVOCGlbve0qSi8HFIEtgr5
         Ad61n1D/5Qwyy5cNEtW/rZhxOESLl8+QM/JRw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=bXNfKquYCxq2XpT26etLnM4gZJgENDR9zjuG669shu/eHvp7W96NEUFLtMT7w+bRMo
         MJfHuc/+KlmR6/qypEDnqPhp4gZzd2xRzDPFUTSkYYKvoauhfUhnE19gTZV07Djzyd/1
         9WH6ifMmtmCHQsSQvJo+yqUwAngPO7TcGjYTM=
MIME-Version: 1.0
Received: by 10.231.166.9 with SMTP id k9mr11595881iby.127.1280290346790; Tue, 
	27 Jul 2010 21:12:26 -0700 (PDT)
Received: by 10.231.12.136 with HTTP; Tue, 27 Jul 2010 21:12:26 -0700 (PDT)
In-Reply-To: <AANLkTimp1bEvRWR7ivOOb2Xkt2w5G_ur-SXuM5qSMn6w@mail.gmail.com>
References: <AANLkTimp1bEvRWR7ivOOb2Xkt2w5G_ur-SXuM5qSMn6w@mail.gmail.com>
Date: Wed, 28 Jul 2010 09:42:26 +0530
Message-ID: <AANLkTi=6bXkcmm7NM5HRCzZgWdjrdvSsNF5yCf5UT=qH@mail.gmail.com>
Subject: Re: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap
From: Sonal Goyal <sonalgoyal4@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=005045013dcc546f4a048c6ad341
X-Virus-Checked: Checked by ClamAV on apache.org

--005045013dcc546f4a048c6ad341
Content-Type: text/plain; charset=ISO-8859-1

Hi Sameer,

For GoGrid, you will have to configure /etc/hosts with the ip and hostname
for master and slaves. If you run hostname and nslookup -sil, you will get
the details for your machines. Make sure /etc/hosts has an entry like:

173.1.45.202 173.1.45.202.reverse.gogrid.com 28684_1_85017_358406

Thanks and Regards,
Sonal
www.meghsoft.com
http://in.linkedin.com/in/sonalgoyal


On Wed, Jul 28, 2010 at 6:42 AM, Sameer Joshi <
sameer.joshi@serenesoftware.com> wrote:

> I installed Java and Hadoop on a GoGrid cloud server using Red Hat
> Enterprise Linux Server release 5.1 (Tikanga). Hadoop installed fine and
> starts fine, however I get an error (java.lang.NullPointerException at
> java.util.concurrent.ConcurrentHashMap) while running the Hadoop wordcount
> example. My guess was that this was a localhost or IPv6 issue.
>
> * I have tested replacing 'localhost' with both the local IP, and server IP
> addresses (when out of options) in Hadoop conf
> * I have disabled IPv6 both in sysctl.conf and hadoop-env.sh (former
> followed by a server restart)
>
> Any thoughts?
> Thank you.
>
>
> The output is given below
>
> # bin/hadoop jar hadoop-0.20.2-examples.jar wordcount datasets tests/out7
> 10/07/27 05:44:59 INFO input.FileInputFormat: Total input paths to process
> :
> 1
> 10/07/27 05:44:59 INFO mapred.JobClient: Running job: job_201007270544_0001
> 10/07/27 05:45:00 INFO mapred.JobClient: map 0% reduce 0%
> 10/07/27 05:45:12 INFO mapred.JobClient: map 100% reduce 0%
> 10/07/27 05:45:17 INFO mapred.JobClient: Task Id :
> attempt_201007270544_0001_r_000000_0, Status : FAILED
>
> Error: java.lang.NullPointerException
> at java.util.concurrent.ConcurrentHashMap.get(Unknown Source)
> at
>
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
>
> at
>
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
>
>
> 10/07/27 05:45:24 INFO mapred.JobClient: Task Id :
> attempt_201007270544_0001_r_000000_1, Status : FAILED
> Error: java.lang.NullPointerException
> at java.util.concurrent.ConcurrentHashMap.get(Unknown Source)
> at
>
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
>
> at
>
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
>
>
> 10/07/27 05:45:30 INFO mapred.JobClient: Task Id :
> attempt_201007270544_0001_r_000000_2, Status : FAILED
> Error: java.lang.NullPointerException
> at java.util.concurrent.ConcurrentHashMap.get(Unknown Source)
> at
>
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
>
> at
>
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
>
>
> 10/07/27 05:45:39 INFO mapred.JobClient: Job complete:
> job_201007270544_0001
>
> 10/07/27 05:45:39 INFO mapred.JobClient: Counters: 12
> 10/07/27 05:45:39 INFO mapred.JobClient: Job Counters
> 10/07/27 05:45:39 INFO mapred.JobClient: Launched reduce tasks=4
> 10/07/27 05:45:39 INFO mapred.JobClient: Launched map tasks=1
> 10/07/27 05:45:39 INFO mapred.JobClient: Data-local map tasks=1
> 10/07/27 05:45:39 INFO mapred.JobClient: Failed reduce tasks=1
> 10/07/27 05:45:39 INFO mapred.JobClient: FileSystemCounters
> 10/07/27 05:45:39 INFO mapred.JobClient: HDFS_BYTES_READ=15319
> 10/07/27 05:45:39 INFO mapred.JobClient: FILE_BYTES_WRITTEN=12847
> 10/07/27 05:45:39 INFO mapred.JobClient: Map-Reduce Framework
> 10/07/27 05:45:39 INFO mapred.JobClient: Combine output records=934
> 10/07/27 05:45:39 INFO mapred.JobClient: Map input records=149
> 10/07/27 05:45:39 INFO mapred.JobClient: Spilled Records=934
> 10/07/27 05:45:39 INFO mapred.JobClient: Map output bytes=25346
> 10/07/27 05:45:39 INFO mapred.JobClient: Combine input records=2541
> 10/07/27 05:45:39 INFO mapred.JobClient: Map output records=2541
>
> --
> Dr. Sameer Joshi, Ph.D.
> Senior computer scientist,
> Serene Software.
>

--005045013dcc546f4a048c6ad341--

From general-return-1866-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 07:55:56 2010
Return-Path: <general-return-1866-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76203 invoked from network); 28 Jul 2010 07:55:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 07:55:55 -0000
Received: (qmail 84156 invoked by uid 500); 28 Jul 2010 07:55:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83852 invoked by uid 500); 28 Jul 2010 07:55:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83840 invoked by uid 99); 28 Jul 2010 07:55:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 07:55:50 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=FREEMAIL_FROM,FROM_LOCAL_NOVOWEL,HK_RANDOM_ENVFROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of sssssssenator@gmail.com designates 74.125.83.176 as permitted sender)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 07:55:44 +0000
Received: by pvc21 with SMTP id 21so1449416pvc.35
        for <general@hadoop.apache.org>; Wed, 28 Jul 2010 00:55:24 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=AE36a3Ka7KEVwSax1G0Teh9BgcOH12eAHwFaAexEv4Q=;
        b=QBUMHlNQwPJdTyJ4QECaCXfD139Pe9WNWeFDK3FirAyHdnljj7rYDTvR7UiZIkkHkj
         l4HJ7guxdAM4RN/HUBKMqLvc1kmJwdh/2Yky9W4j9F5GhxKUNQo6AsT0CRoQLGCwpRc3
         E4v42Hx7KIbUcCTQZgRSplrgf0xnaPcJ5wUeg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=j8m3iulbbmOBonY8EwcIMBEE9P+liKvxbRXAYUNAUMTv3cFG7t95WoqMr6W7WScEEV
         wziQir2MnNoHak2L/WfFSbZbLpAg2wZX4AzXqshGuZnWXpan7z0JmMgz8M2uwDqYMCxD
         FcElolrIgehI32aINy86XbkWQGvEt6d961MUY=
MIME-Version: 1.0
Received: by 10.142.233.21 with SMTP id f21mr780238wfh.284.1280303722000; Wed, 
	28 Jul 2010 00:55:22 -0700 (PDT)
Received: by 10.142.103.17 with HTTP; Wed, 28 Jul 2010 00:55:21 -0700 (PDT)
Date: Wed, 28 Jul 2010 13:25:21 +0530
Message-ID: <AANLkTi=c8p1O9wBjfO6EGi7XcwgfUGPcFhLyR086oxAF@mail.gmail.com>
Subject: Hadoop Cluster Configuration
From: vaibhav negi <sssssssenator@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd2e04c8df273048c6df049

--000e0cd2e04c8df273048c6df049
Content-Type: text/plain; charset=ISO-8859-1

Hi,

While setting hadoop cluster, does configuration files (conf/core-site.xml,
conf/mapred-site.xml,conf/hdfs-site.xml) in every node(name node and data
nodes)  needs to be configured in the same manner?
How does configuration of name node differs from configuration of data
nodes?


Vaibhav Negi

--000e0cd2e04c8df273048c6df049--

From general-return-1867-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 08:02:34 2010
Return-Path: <general-return-1867-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77290 invoked from network); 28 Jul 2010 08:02:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 08:02:33 -0000
Received: (qmail 94484 invoked by uid 500); 28 Jul 2010 08:02:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94353 invoked by uid 500); 28 Jul 2010 08:02:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94345 invoked by uid 99); 28 Jul 2010 08:02:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 08:02:30 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yhemanth@gmail.com designates 209.85.216.176 as permitted sender)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 08:02:23 +0000
Received: by qyk34 with SMTP id 34so4498579qyk.14
        for <general@hadoop.apache.org>; Wed, 28 Jul 2010 01:02:03 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=2UqLgjNPZ2X4iWFt4TnZ0uL1iFm8wGUQ9Dv0RWxlliE=;
        b=b0Nhpic1TQ9F+bieG85MMkAvyQVO7MDXkzdstvZxtYwQT3r3RMfWvmLldMF0Nv7bPj
         lcg/CZeKiZs6ODqnrDuwYkLIzU+JI+T7hZm0dAgMqW6W/W6OQLccGQmC+I5kL0bkKhB4
         3ZVq7TbGzLrOpK5fkEksYNZr9FxkcBpCdfkQY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=syfLIHmeRJc/FV+ZVWzXio/j8JFUXSM0wUmg4XYK23rn+7e9m7swnRUOqRUHJ0vV7l
         L5GuMho6z01cddGr2DJ2iBx+A2l5z+lOZc34VBLYnNn5H9isegEtHyuIvzcRBmSCNfu/
         PiG52TRI+oXT2SHS3lAKg9D959cAIn0/PFcmU=
MIME-Version: 1.0
Received: by 10.224.54.146 with SMTP id q18mr8234101qag.93.1280304122755; Wed, 
	28 Jul 2010 01:02:02 -0700 (PDT)
Received: by 10.220.189.141 with HTTP; Wed, 28 Jul 2010 01:02:02 -0700 (PDT)
In-Reply-To: <AANLkTi=c8p1O9wBjfO6EGi7XcwgfUGPcFhLyR086oxAF@mail.gmail.com>
References: <AANLkTi=c8p1O9wBjfO6EGi7XcwgfUGPcFhLyR086oxAF@mail.gmail.com>
Date: Wed, 28 Jul 2010 13:32:02 +0530
Message-ID: <AANLkTi=pEPNuXOmBrZ-tfhqQJY---A5_sE+VXJf2BB1G@mail.gmail.com>
Subject: Re: Hadoop Cluster Configuration
From: Hemanth Yamijala <yhemanth@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Vaibhav,

> While setting hadoop cluster, does configuration files (conf/core-site.xm=
l,
> conf/mapred-site.xml,conf/hdfs-site.xml) in every node(name node and data
> nodes) =A0needs to be configured in the same manner?

This is a complicated question to answer :-). There are certain
configuration variables that need to be defined to be the same between
the master and the slaves and some that don't need to be. Pre Hadoop
0.21, there is no easy way other than documentation for the variables
(hopefully) to determine if this is the case or not. I think in Hadoop
0.21 and since, we have tried to adopt a convention to include the
daemon name to specify which variables are used by which daemons. And
those that are cluster-wide, that need to be the same throughout all
the nodes will have something like 'cluster' in the name.

Your best bet in any case is possibly to sift through the
documentation of the variables you are interested in. Or else to post
a query here.

> How does configuration of name node differs from configuration of data
> nodes?

Not sure about this one.

Thanks
hemanth

From general-return-1868-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 08:44:33 2010
Return-Path: <general-return-1868-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94272 invoked from network); 28 Jul 2010 08:44:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 08:44:33 -0000
Received: (qmail 31699 invoked by uid 500); 28 Jul 2010 08:44:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31294 invoked by uid 500); 28 Jul 2010 08:44:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31286 invoked by uid 99); 28 Jul 2010 08:44:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 08:44:29 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=FREEMAIL_FROM,FROM_LOCAL_NOVOWEL,HK_RANDOM_ENVFROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of sssssssenator@gmail.com designates 209.85.210.48 as permitted sender)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 08:44:23 +0000
Received: by pzk36 with SMTP id 36so3541127pzk.35
        for <general@hadoop.apache.org>; Wed, 28 Jul 2010 01:44:02 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=OAwVkAbi7pqIUHboPhG6B3s3SC99RwrmYCM8+jZWbyU=;
        b=w3FRENfSW+z0rJ+aKvRiG9YQ8GzDrZc1VwkP5qhMBdS+KJvL5DKjKmvipQxRoNEn+G
         czYyRc9Pgxd3l6W3sXLhZK0C+QkH94bCQjw/iChgHFpd3zUx+cG1T5bLdXjkn2O9B8SF
         kJ8Fr5aPL/09fp1WNz1OVbpWFYb9mmlAiUtvU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=rl3WXvScwcTAsRYm5Df0U25OBwqorQ8GonPyN6DKJV2Nsfu7vxP9HnA6iHFDvsTdpk
         eyQI7RxAC2oBdmBzveaF+1+Gad1TwDEALcS5eSc72btH2xjBIIG4CMfZR29mxbS3Stu/
         LEfcCpQbV2BoMPPIcYV9q7Al3BBUY+AlNssS8=
MIME-Version: 1.0
Received: by 10.142.155.13 with SMTP id c13mr820394wfe.288.1280306642689; Wed, 
	28 Jul 2010 01:44:02 -0700 (PDT)
Received: by 10.142.103.17 with HTTP; Wed, 28 Jul 2010 01:44:02 -0700 (PDT)
In-Reply-To: <AANLkTi=pEPNuXOmBrZ-tfhqQJY---A5_sE+VXJf2BB1G@mail.gmail.com>
References: <AANLkTi=c8p1O9wBjfO6EGi7XcwgfUGPcFhLyR086oxAF@mail.gmail.com>
	<AANLkTi=pEPNuXOmBrZ-tfhqQJY---A5_sE+VXJf2BB1G@mail.gmail.com>
Date: Wed, 28 Jul 2010 14:14:02 +0530
Message-ID: <AANLkTim9qtnHBZfFpTxyFCq37X=TRaQ1Z=LWcv_xTNri@mail.gmail.com>
Subject: Re: Hadoop Cluster Configuration
From: vaibhav negi <sssssssenator@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd2e458a41e52048c6e9ee6

--000e0cd2e458a41e52048c6e9ee6
Content-Type: text/plain; charset=ISO-8859-1

Hi,

I am using hadoop versoion 0.20 . I am trying set up hadoop cluster. What
are the configurations for name node? What are the configurations for data
node?

In documentation all configurations  are given, but, it is not mentioned
what needs to be configured for which node.


Vaibhav Negi


On Wed, Jul 28, 2010 at 1:32 PM, Hemanth Yamijala <yhemanth@gmail.com>wrote:

> Vaibhav,
>
> > While setting hadoop cluster, does configuration files
> (conf/core-site.xml,
> > conf/mapred-site.xml,conf/hdfs-site.xml) in every node(name node and data
> > nodes)  needs to be configured in the same manner?
>
> This is a complicated question to answer :-). There are certain
> configuration variables that need to be defined to be the same between
> the master and the slaves and some that don't need to be. Pre Hadoop
> 0.21, there is no easy way other than documentation for the variables
> (hopefully) to determine if this is the case or not. I think in Hadoop
> 0.21 and since, we have tried to adopt a convention to include the
> daemon name to specify which variables are used by which daemons. And
> those that are cluster-wide, that need to be the same throughout all
> the nodes will have something like 'cluster' in the name.
>
> Your best bet in any case is possibly to sift through the
> documentation of the variables you are interested in. Or else to post
> a query here.
>
> > How does configuration of name node differs from configuration of data
> > nodes?
>
> Not sure about this one.
>
> Thanks
> hemanth
>

--000e0cd2e458a41e52048c6e9ee6--

From general-return-1869-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 17:09:15 2010
Return-Path: <general-return-1869-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 89081 invoked from network); 28 Jul 2010 17:09:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 17:09:15 -0000
Received: (qmail 81405 invoked by uid 500); 28 Jul 2010 17:09:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81343 invoked by uid 500); 28 Jul 2010 17:09:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81335 invoked by uid 99); 28 Jul 2010 17:09:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 17:09:13 +0000
X-ASF-Spam-Status: No, hits=0.6 required=10.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.149.205] (HELO na3sys009aog111.obsmtp.com) (74.125.149.205)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 28 Jul 2010 17:09:08 +0000
Received: from source ([209.85.161.44]) by na3sys009aob111.postini.com ([74.125.148.12]) with SMTP
	ID DSNKTFBkHPCotnptxHtXyMKLNwNSzT2VEsSJ@postini.com; Wed, 28 Jul 2010 10:08:47 PDT
Received: by mail-fx0-f44.google.com with SMTP id 1so1392474fxm.3
        for <general@hadoop.apache.org>; Wed, 28 Jul 2010 10:08:43 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.239.153.145 with SMTP id z17mr871958hbb.5.1280336922598; Wed, 
	28 Jul 2010 10:08:42 -0700 (PDT)
Received: by 10.239.133.65 with HTTP; Wed, 28 Jul 2010 10:08:42 -0700 (PDT)
In-Reply-To: <AANLkTi=6bXkcmm7NM5HRCzZgWdjrdvSsNF5yCf5UT=qH@mail.gmail.com>
References: <AANLkTimp1bEvRWR7ivOOb2Xkt2w5G_ur-SXuM5qSMn6w@mail.gmail.com>
	<AANLkTi=6bXkcmm7NM5HRCzZgWdjrdvSsNF5yCf5UT=qH@mail.gmail.com>
Date: Wed, 28 Jul 2010 13:08:42 -0400
Message-ID: <AANLkTi=kHkc-zRh-bv3Ot-s-+8T2gSEiTF7kUH4Z9+oP@mail.gmail.com>
Subject: Re: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap
From: Sameer Joshi <sameer.joshi@serenesoftware.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f7928676dd4e048c75ab53

--001485f7928676dd4e048c75ab53
Content-Type: text/plain; charset=ISO-8859-1

Hi Sonal,
Thank you for replying.

I'm trying to run it as single node, so I had localhost in all the conf
files and in /etc/hosts. I tried replacing it with 127.0.0.1 in all Hadoop
conf files (read to do that on some forum), but with no change.

I figured I didn't need hostname since I was trying to run it locally, but
still tried replacing localhost with 173.1.86.194. However, as expected, it
could not connect to server.

Any thoughts?

Thanks for your help, I'm new at this and feel much obliged.

Regards,
Sameer

On Wed, Jul 28, 2010 at 12:12 AM, Sonal Goyal <sonalgoyal4@gmail.com> wrote:

> Hi Sameer,
>
> For GoGrid, you will have to configure /etc/hosts with the ip and hostname
> for master and slaves. If you run hostname and nslookup -sil, you will get
> the details for your machines. Make sure /etc/hosts has an entry like:
>
> 173.1.45.202 173.1.45.202.reverse.gogrid.com 28684_1_85017_358406
>
> Thanks and Regards,
> Sonal
> www.meghsoft.com
> http://in.linkedin.com/in/sonalgoyal
>
>
> On Wed, Jul 28, 2010 at 6:42 AM, Sameer Joshi <
> sameer.joshi@serenesoftware.com> wrote:
>
> > I installed Java and Hadoop on a GoGrid cloud server using Red Hat
> > Enterprise Linux Server release 5.1 (Tikanga). Hadoop installed fine and
> > starts fine, however I get an error (java.lang.NullPointerException at
> > java.util.concurrent.ConcurrentHashMap) while running the Hadoop
> wordcount
> > example. My guess was that this was a localhost or IPv6 issue.
> >
> > * I have tested replacing 'localhost' with both the local IP, and server
> IP
> > addresses (when out of options) in Hadoop conf
> > * I have disabled IPv6 both in sysctl.conf and hadoop-env.sh (former
> > followed by a server restart)
> >
> > Any thoughts?
> > Thank you.
> >
> >
> > The output is given below
> >
> > # bin/hadoop jar hadoop-0.20.2-examples.jar wordcount datasets tests/out7
> > 10/07/27 05:44:59 INFO input.FileInputFormat: Total input paths to
> process
> > :
> > 1
> > 10/07/27 05:44:59 INFO mapred.JobClient: Running job:
> job_201007270544_0001
> > 10/07/27 05:45:00 INFO mapred.JobClient: map 0% reduce 0%
> > 10/07/27 05:45:12 INFO mapred.JobClient: map 100% reduce 0%
> > 10/07/27 05:45:17 INFO mapred.JobClient: Task Id :
> > attempt_201007270544_0001_r_000000_0, Status : FAILED
> >
> > Error: java.lang.NullPointerException
> > at java.util.concurrent.ConcurrentHashMap.get(Unknown Source)
> > at
> >
> >
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
> >
> > at
> >
> >
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
> >
> >
> > 10/07/27 05:45:24 INFO mapred.JobClient: Task Id :
> > attempt_201007270544_0001_r_000000_1, Status : FAILED
> > Error: java.lang.NullPointerException
> > at java.util.concurrent.ConcurrentHashMap.get(Unknown Source)
> > at
> >
> >
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
> >
> > at
> >
> >
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
> >
> >
> > 10/07/27 05:45:30 INFO mapred.JobClient: Task Id :
> > attempt_201007270544_0001_r_000000_2, Status : FAILED
> > Error: java.lang.NullPointerException
> > at java.util.concurrent.ConcurrentHashMap.get(Unknown Source)
> > at
> >
> >
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
> >
> > at
> >
> >
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
> >
> >
> > 10/07/27 05:45:39 INFO mapred.JobClient: Job complete:
> > job_201007270544_0001
> >
> > 10/07/27 05:45:39 INFO mapred.JobClient: Counters: 12
> > 10/07/27 05:45:39 INFO mapred.JobClient: Job Counters
> > 10/07/27 05:45:39 INFO mapred.JobClient: Launched reduce tasks=4
> > 10/07/27 05:45:39 INFO mapred.JobClient: Launched map tasks=1
> > 10/07/27 05:45:39 INFO mapred.JobClient: Data-local map tasks=1
> > 10/07/27 05:45:39 INFO mapred.JobClient: Failed reduce tasks=1
> > 10/07/27 05:45:39 INFO mapred.JobClient: FileSystemCounters
> > 10/07/27 05:45:39 INFO mapred.JobClient: HDFS_BYTES_READ=15319
> > 10/07/27 05:45:39 INFO mapred.JobClient: FILE_BYTES_WRITTEN=12847
> > 10/07/27 05:45:39 INFO mapred.JobClient: Map-Reduce Framework
> > 10/07/27 05:45:39 INFO mapred.JobClient: Combine output records=934
> > 10/07/27 05:45:39 INFO mapred.JobClient: Map input records=149
> > 10/07/27 05:45:39 INFO mapred.JobClient: Spilled Records=934
> > 10/07/27 05:45:39 INFO mapred.JobClient: Map output bytes=25346
> > 10/07/27 05:45:39 INFO mapred.JobClient: Combine input records=2541
> > 10/07/27 05:45:39 INFO mapred.JobClient: Map output records=2541
> >
> > --
> > Dr. Sameer Joshi, Ph.D.
> > Senior computer scientist,
> > Serene Software.
> >
>



-- 
Dr. Sameer Joshi, Ph.D.
Senior computer scientist,
Serene Software.

--001485f7928676dd4e048c75ab53--

From general-return-1870-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 17:21:11 2010
Return-Path: <general-return-1870-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94576 invoked from network); 28 Jul 2010 17:21:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 17:21:11 -0000
Received: (qmail 3327 invoked by uid 500); 28 Jul 2010 17:21:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3217 invoked by uid 500); 28 Jul 2010 17:21:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3209 invoked by uid 99); 28 Jul 2010 17:21:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 17:21:09 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of sonalgoyal4@gmail.com designates 209.85.216.169 as permitted sender)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 17:21:03 +0000
Received: by qyk9 with SMTP id 9so4887485qyk.14
        for <general@hadoop.apache.org>; Wed, 28 Jul 2010 10:20:42 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=IRRG5u9pVgI+RuCI1hPiaQDEoCibzOffFZhHsmZBikI=;
        b=TWEnn37YuxfR0yCjyO86FqcTl6jpuaEAF+Cl5vI3j+3n1QCPBzXXOWYcgTzjRs411M
         +AwurOdNZt24dUqLiTb+0KxwQErDeFUO6fpCcR8nXBKqRNgCD8zGv4gCz29tDqqK29Co
         Vy/O9kfM1h5BVqExnDJqVyyBDylg8Zy520V4E=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=CjPpERWKjg1U8cJJFR12bCKi5oOEB7jAVsx/HFg1+qiVhaKYSy0W2TGswdyW0+T9jD
         V1DArVXNafeEdUDQdUsLOdfC+Ola2TJdv5eh1/HTpG4GvxEfiGiKDSw/Hhn4RjQAlQ3F
         dQptlYv/aobs4z++xkCYvx+Fd91IOyoDMHuJM=
MIME-Version: 1.0
Received: by 10.224.56.195 with SMTP id z3mr4938108qag.41.1280337640488; Wed, 
	28 Jul 2010 10:20:40 -0700 (PDT)
Received: by 10.229.232.67 with HTTP; Wed, 28 Jul 2010 10:20:40 -0700 (PDT)
In-Reply-To: <AANLkTi=kHkc-zRh-bv3Ot-s-+8T2gSEiTF7kUH4Z9+oP@mail.gmail.com>
References: <AANLkTimp1bEvRWR7ivOOb2Xkt2w5G_ur-SXuM5qSMn6w@mail.gmail.com>
	<AANLkTi=6bXkcmm7NM5HRCzZgWdjrdvSsNF5yCf5UT=qH@mail.gmail.com>
	<AANLkTi=kHkc-zRh-bv3Ot-s-+8T2gSEiTF7kUH4Z9+oP@mail.gmail.com>
Date: Wed, 28 Jul 2010 22:50:40 +0530
Message-ID: <AANLkTin=Caq5maK=vWQXURhO7fytE-QdFVh=6Ad7pVkf@mail.gmail.com>
Subject: Re: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap
From: Sonal Goyal <sonalgoyal4@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163613797d410028048c75d6df
X-Virus-Checked: Checked by ClamAV on apache.org

--00163613797d410028048c75d6df
Content-Type: text/plain; charset=ISO-8859-1

Hi Sameer,

Do you see all processes up? Can you check the logs and see what is going
wrong? I suspect you will need to make the entries in /etc/hosts anyways.
Somewhere in hadoop code, the ip  hostname mapping is done, I forget the
exact location.

Thanks and Regards,
Sonal
www.meghsoft.com
http://in.linkedin.com/in/sonalgoyal


On Wed, Jul 28, 2010 at 10:38 PM, Sameer Joshi <
sameer.joshi@serenesoftware.com> wrote:

> Hi Sonal,
> Thank you for replying.
>
> I'm trying to run it as single node, so I had localhost in all the conf
> files and in /etc/hosts. I tried replacing it with 127.0.0.1 in all Hadoop
> conf files (read to do that on some forum), but with no change.
>
> I figured I didn't need hostname since I was trying to run it locally, but
> still tried replacing localhost with 173.1.86.194. However, as expected, it
> could not connect to server.
>
> Any thoughts?
>
> Thanks for your help, I'm new at this and feel much obliged.
>
> Regards,
> Sameer
>
> On Wed, Jul 28, 2010 at 12:12 AM, Sonal Goyal <sonalgoyal4@gmail.com>
> wrote:
>
> > Hi Sameer,
> >
> > For GoGrid, you will have to configure /etc/hosts with the ip and
> hostname
> > for master and slaves. If you run hostname and nslookup -sil, you will
> get
> > the details for your machines. Make sure /etc/hosts has an entry like:
> >
> > 173.1.45.202 173.1.45.202.reverse.gogrid.com 28684_1_85017_358406
> >
> > Thanks and Regards,
> > Sonal
> > www.meghsoft.com
> > http://in.linkedin.com/in/sonalgoyal
> >
> >
> > On Wed, Jul 28, 2010 at 6:42 AM, Sameer Joshi <
> > sameer.joshi@serenesoftware.com> wrote:
> >
> > > I installed Java and Hadoop on a GoGrid cloud server using Red Hat
> > > Enterprise Linux Server release 5.1 (Tikanga). Hadoop installed fine
> and
> > > starts fine, however I get an error (java.lang.NullPointerException at
> > > java.util.concurrent.ConcurrentHashMap) while running the Hadoop
> > wordcount
> > > example. My guess was that this was a localhost or IPv6 issue.
> > >
> > > * I have tested replacing 'localhost' with both the local IP, and
> server
> > IP
> > > addresses (when out of options) in Hadoop conf
> > > * I have disabled IPv6 both in sysctl.conf and hadoop-env.sh (former
> > > followed by a server restart)
> > >
> > > Any thoughts?
> > > Thank you.
> > >
> > >
> > > The output is given below
> > >
> > > # bin/hadoop jar hadoop-0.20.2-examples.jar wordcount datasets
> tests/out7
> > > 10/07/27 05:44:59 INFO input.FileInputFormat: Total input paths to
> > process
> > > :
> > > 1
> > > 10/07/27 05:44:59 INFO mapred.JobClient: Running job:
> > job_201007270544_0001
> > > 10/07/27 05:45:00 INFO mapred.JobClient: map 0% reduce 0%
> > > 10/07/27 05:45:12 INFO mapred.JobClient: map 100% reduce 0%
> > > 10/07/27 05:45:17 INFO mapred.JobClient: Task Id :
> > > attempt_201007270544_0001_r_000000_0, Status : FAILED
> > >
> > > Error: java.lang.NullPointerException
> > > at java.util.concurrent.ConcurrentHashMap.get(Unknown Source)
> > > at
> > >
> > >
> >
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
> > >
> > > at
> > >
> > >
> >
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
> > >
> > >
> > > 10/07/27 05:45:24 INFO mapred.JobClient: Task Id :
> > > attempt_201007270544_0001_r_000000_1, Status : FAILED
> > > Error: java.lang.NullPointerException
> > > at java.util.concurrent.ConcurrentHashMap.get(Unknown Source)
> > > at
> > >
> > >
> >
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
> > >
> > > at
> > >
> > >
> >
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
> > >
> > >
> > > 10/07/27 05:45:30 INFO mapred.JobClient: Task Id :
> > > attempt_201007270544_0001_r_000000_2, Status : FAILED
> > > Error: java.lang.NullPointerException
> > > at java.util.concurrent.ConcurrentHashMap.get(Unknown Source)
> > > at
> > >
> > >
> >
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
> > >
> > > at
> > >
> > >
> >
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
> > >
> > >
> > > 10/07/27 05:45:39 INFO mapred.JobClient: Job complete:
> > > job_201007270544_0001
> > >
> > > 10/07/27 05:45:39 INFO mapred.JobClient: Counters: 12
> > > 10/07/27 05:45:39 INFO mapred.JobClient: Job Counters
> > > 10/07/27 05:45:39 INFO mapred.JobClient: Launched reduce tasks=4
> > > 10/07/27 05:45:39 INFO mapred.JobClient: Launched map tasks=1
> > > 10/07/27 05:45:39 INFO mapred.JobClient: Data-local map tasks=1
> > > 10/07/27 05:45:39 INFO mapred.JobClient: Failed reduce tasks=1
> > > 10/07/27 05:45:39 INFO mapred.JobClient: FileSystemCounters
> > > 10/07/27 05:45:39 INFO mapred.JobClient: HDFS_BYTES_READ=15319
> > > 10/07/27 05:45:39 INFO mapred.JobClient: FILE_BYTES_WRITTEN=12847
> > > 10/07/27 05:45:39 INFO mapred.JobClient: Map-Reduce Framework
> > > 10/07/27 05:45:39 INFO mapred.JobClient: Combine output records=934
> > > 10/07/27 05:45:39 INFO mapred.JobClient: Map input records=149
> > > 10/07/27 05:45:39 INFO mapred.JobClient: Spilled Records=934
> > > 10/07/27 05:45:39 INFO mapred.JobClient: Map output bytes=25346
> > > 10/07/27 05:45:39 INFO mapred.JobClient: Combine input records=2541
> > > 10/07/27 05:45:39 INFO mapred.JobClient: Map output records=2541
> > >
> > > --
> > > Dr. Sameer Joshi, Ph.D.
> > > Senior computer scientist,
> > > Serene Software.
> > >
> >
>
>
>
> --
> Dr. Sameer Joshi, Ph.D.
> Senior computer scientist,
> Serene Software.
>

--00163613797d410028048c75d6df--

From general-return-1871-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 06:36:29 2010
Return-Path: <general-return-1871-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93175 invoked from network); 29 Jul 2010 06:36:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 06:36:28 -0000
Received: (qmail 98761 invoked by uid 500); 29 Jul 2010 06:36:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98399 invoked by uid 500); 29 Jul 2010 06:36:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98391 invoked by uid 99); 29 Jul 2010 06:36:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 06:36:23 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 74.125.82.176 is neither permitted nor denied by domain of oded@legolas-media.com)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 06:36:15 +0000
Received: by wyf23 with SMTP id 23so1246wyf.35
        for <general@hadoop.apache.org>; Wed, 28 Jul 2010 23:35:53 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.227.27.197 with SMTP id j5mr10852733wbc.111.1280385352651; 
	Wed, 28 Jul 2010 23:35:52 -0700 (PDT)
Received: by 10.216.172.135 with HTTP; Wed, 28 Jul 2010 23:35:52 -0700 (PDT)
Date: Thu, 29 Jul 2010 09:35:52 +0300
Message-ID: <AANLkTin6to3eE5OqqNKDsaMqyLsvXbgT-=VS89r8sgGO@mail.gmail.com>
Subject: OutOfMemoryError when writing a huge writable
From: Oded Rosen <oded@legolas-media.com>
To: general@hadoop.apache.org
Cc: Yiftah Frechter <yiftahf@legolas-media.com>
Content-Type: multipart/alternative; boundary=0022158bfd911ed5eb048c80f259

--0022158bfd911ed5eb048c80f259
Content-Type: text/plain; charset=ISO-8859-1

Hi,

My job uses 0.18 api (my cluster's hadoop version is 0.20.1), and I have my
own set of nesting writables as input and output, some of them might be
really big (memory wise).
During the finishing steps of my reduce() function, When i call
output.collect() on a particularly large writable, the task failes with a
java heap space error:

2010-07-28 16:08:46,727 FATAL org.apache.hadoop.mapred.TaskTracker:
Error running child : java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:2786)
	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
	at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
	at com.legolas.data.users.EntryWritable.write(EntryWritable.java:339)
	at com.legolas.data.users.UserWritable.write(UserWritable.java:1313)
	at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
	at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
	at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1006)
	at org.apache.hadoop.mapred.SequenceFileOutputFormat$1.write(SequenceFileOutputFormat.java:75)
	at org.apache.hadoop.mapred.lib.MultipleOutputs$1.collect(MultipleOutputs.java:521)
	at com.legolas.data.users.UserReducer.reduce(UserReducer.java:267)
	at com.legolas.data.users.UserReducer.reduce(UserReducer.java:32)
	at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:463)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:411)
	at org.apache.hadoop.mapred.Child.main(Child.java:170)

This failure was during writeLong(), but many other writes were made to the
same stream before that call, on that specific instance of my object (it was
a big one, i can tell because that specific output.collect only happens on
very big objects).
My first guess is that I should flush() the stream periodically during these
big writes, and I want to know if hadoop's DataOutputStream supports
flush(), as many other streams does not really pay attention to flush()
calls (according to a java api on DataOutputStream's flush())

Does anyone know how can I outwit this problem and write by big writables
without failure?

Again, this only happens when I write() a perticularly large writable. Other
write() operations ends without a problem.

-- 
Oded

--0022158bfd911ed5eb048c80f259--

From general-return-1872-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 30 18:55:32 2010
Return-Path: <general-return-1872-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19166 invoked from network); 30 Jul 2010 18:55:31 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jul 2010 18:55:31 -0000
Received: (qmail 72681 invoked by uid 500); 30 Jul 2010 18:55:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72630 invoked by uid 500); 30 Jul 2010 18:55:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72622 invoked by uid 99); 30 Jul 2010 18:55:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 18:55:29 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 209.85.212.176 as permitted sender)
Received: from [209.85.212.176] (HELO mail-px0-f176.google.com) (209.85.212.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 18:55:24 +0000
Received: by pxi11 with SMTP id 11so1144136pxi.35
        for <general@hadoop.apache.org>; Fri, 30 Jul 2010 11:55:04 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        bh=kMSckIhdygn41ppQTPbUjE+/huLNZSTU0+ErZQm9AD8=;
        b=HKYgTVi1uG9H+pyeDMPerSaWgb3xv8PcZaiKzplUr3/06MbYeeYoGRyLan3v0POln3
         9LaQ9i8HvEqpPfkqIB1NJphuVCNLyZfb/mYLfjrBWZItcGQ9EO1Jag9bkUZ4MaCOo020
         UpIGfykLww7dULXgPpPydxWnmtHIN/D9Nk9eQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        b=IJEQ7D4Hp11qBCR4PUbWLN6i8aNc49kscNSf31ISL6IwCHv+IMWWmgT4WxAT2IRIEN
         SGRSWjOxafEJAHVpwmeannaj5Wap73R0h3rSoja9mIXYxlEHBW+ucYVwDCprDj2EAPjx
         JgBWNRXQrHk58ie583OeWzJOJ7bfFCdH2Z2ow=
MIME-Version: 1.0
Received: by 10.142.194.15 with SMTP id r15mr2084472wff.274.1280516104558; 
	Fri, 30 Jul 2010 11:55:04 -0700 (PDT)
Sender: saint.ack@gmail.com
Received: by 10.142.154.6 with HTTP; Fri, 30 Jul 2010 11:55:04 -0700 (PDT)
Date: Fri, 30 Jul 2010 11:55:04 -0700
X-Google-Sender-Auth: kg3Kkqs5XjlTa1OLTuvujmcGifY
Message-ID: <AANLkTi=z_1+fC1wOYio=Pw1HnKNgXexwo2UcMEW8E7m6@mail.gmail.com>
Subject: [ANN] HBase-0.20.6 available for download
From: Stack <stack@duboce.net>
To: user@hbase.apache.org, general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

HBase 0.20.6 is available for download:

 http://hadoop.apache.org/hbase/releases.html

The Release Notes are available here: http://su.pr/2itvaW

We recommend that all users, particularly those running 0.20.4, upgrade.

Thanks to all who contributed to this release.

Yours,
The HBasistas

From general-return-1873-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 31 07:38:12 2010
Return-Path: <general-return-1873-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67210 invoked from network); 31 Jul 2010 07:38:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Jul 2010 07:38:12 -0000
Received: (qmail 82598 invoked by uid 500); 31 Jul 2010 07:38:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82008 invoked by uid 500); 31 Jul 2010 07:38:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82000 invoked by uid 99); 31 Jul 2010 07:38:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 31 Jul 2010 07:38:07 +0000
X-ASF-Spam-Status: No, hits=0.6 required=10.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.149.199] (HELO na3sys009aog108.obsmtp.com) (74.125.149.199)
    by apache.org (qpsmtpd/0.29) with SMTP; Sat, 31 Jul 2010 07:38:00 +0000
Received: from source ([209.85.161.45]) by na3sys009aob108.postini.com ([74.125.148.12]) with SMTP
	ID DSNKTFPSwUBg7ElVmHhw7oratp4F9EUn1ulW@postini.com; Sat, 31 Jul 2010 00:37:39 PDT
Received: by fxm7 with SMTP id 7so441709fxm.18
        for <general@hadoop.apache.org>; Sat, 31 Jul 2010 00:37:36 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.239.165.9 with SMTP id v9mr165185hbd.93.1280561856222; Sat, 31 
	Jul 2010 00:37:36 -0700 (PDT)
Received: by 10.239.156.195 with HTTP; Sat, 31 Jul 2010 00:37:36 -0700 (PDT)
In-Reply-To: <AANLkTin=Caq5maK=vWQXURhO7fytE-QdFVh=6Ad7pVkf@mail.gmail.com>
References: <AANLkTimp1bEvRWR7ivOOb2Xkt2w5G_ur-SXuM5qSMn6w@mail.gmail.com>
	<AANLkTi=6bXkcmm7NM5HRCzZgWdjrdvSsNF5yCf5UT=qH@mail.gmail.com>
	<AANLkTi=kHkc-zRh-bv3Ot-s-+8T2gSEiTF7kUH4Z9+oP@mail.gmail.com>
	<AANLkTin=Caq5maK=vWQXURhO7fytE-QdFVh=6Ad7pVkf@mail.gmail.com>
Date: Sat, 31 Jul 2010 03:37:36 -0400
Message-ID: <AANLkTi=6kTtcJaKS6hXY_kLAmLEg+Pd_oDoWN1nDgQ42@mail.gmail.com>
Subject: Re: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap
From: Sameer Joshi <sameer.joshi@serenesoftware.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f794de8d9698048caa0a6e
X-Virus-Checked: Checked by ClamAV on apache.org

--001485f794de8d9698048caa0a6e
Content-Type: text/plain; charset=ISO-8859-1

Hi Sonal,
Sorry for the late reply, I was traveling. The logs, unfortunately, did not
yield too much information. I'll try some more configurations with etc/hosts
edits, and see if I can't get it to behave.
Thanks so much again for your help.
Sameer

On Wed, Jul 28, 2010 at 1:20 PM, Sonal Goyal <sonalgoyal4@gmail.com> wrote:

> Hi Sameer,
>
> Do you see all processes up? Can you check the logs and see what is going
> wrong? I suspect you will need to make the entries in /etc/hosts anyways.
> Somewhere in hadoop code, the ip  hostname mapping is done, I forget the
> exact location.
>
> Thanks and Regards,
> Sonal
> www.meghsoft.com
> http://in.linkedin.com/in/sonalgoyal
>
>
> On Wed, Jul 28, 2010 at 10:38 PM, Sameer Joshi <
> sameer.joshi@serenesoftware.com> wrote:
>
> > Hi Sonal,
> > Thank you for replying.
> >
> > I'm trying to run it as single node, so I had localhost in all the conf
> > files and in /etc/hosts. I tried replacing it with 127.0.0.1 in all
> Hadoop
> > conf files (read to do that on some forum), but with no change.
> >
> > I figured I didn't need hostname since I was trying to run it locally,
> but
> > still tried replacing localhost with 173.1.86.194. However, as expected,
> it
> > could not connect to server.
> >
> > Any thoughts?
> >
> > Thanks for your help, I'm new at this and feel much obliged.
> >
> > Regards,
> > Sameer
> >
> > On Wed, Jul 28, 2010 at 12:12 AM, Sonal Goyal <sonalgoyal4@gmail.com>
> > wrote:
> >
> > > Hi Sameer,
> > >
> > > For GoGrid, you will have to configure /etc/hosts with the ip and
> > hostname
> > > for master and slaves. If you run hostname and nslookup -sil, you will
> > get
> > > the details for your machines. Make sure /etc/hosts has an entry like:
> > >
> > > 173.1.45.202 173.1.45.202.reverse.gogrid.com 28684_1_85017_358406
> > >
> > > Thanks and Regards,
> > > Sonal
> > > www.meghsoft.com
> > > http://in.linkedin.com/in/sonalgoyal
> > >
> > >
> > > On Wed, Jul 28, 2010 at 6:42 AM, Sameer Joshi <
> > > sameer.joshi@serenesoftware.com> wrote:
> > >
> > > > I installed Java and Hadoop on a GoGrid cloud server using Red Hat
> > > > Enterprise Linux Server release 5.1 (Tikanga). Hadoop installed fine
> > and
> > > > starts fine, however I get an error (java.lang.NullPointerException
> at
> > > > java.util.concurrent.ConcurrentHashMap) while running the Hadoop
> > > wordcount
> > > > example. My guess was that this was a localhost or IPv6 issue.
> > > >
> > > > * I have tested replacing 'localhost' with both the local IP, and
> > server
> > > IP
> > > > addresses (when out of options) in Hadoop conf
> > > > * I have disabled IPv6 both in sysctl.conf and hadoop-env.sh (former
> > > > followed by a server restart)
> > > >
> > > > Any thoughts?
> > > > Thank you.
> > > >
> > > >
> > > > The output is given below
> > > >
> > > > # bin/hadoop jar hadoop-0.20.2-examples.jar wordcount datasets
> > tests/out7
> > > > 10/07/27 05:44:59 INFO input.FileInputFormat: Total input paths to
> > > process
> > > > :
> > > > 1
> > > > 10/07/27 05:44:59 INFO mapred.JobClient: Running job:
> > > job_201007270544_0001
> > > > 10/07/27 05:45:00 INFO mapred.JobClient: map 0% reduce 0%
> > > > 10/07/27 05:45:12 INFO mapred.JobClient: map 100% reduce 0%
> > > > 10/07/27 05:45:17 INFO mapred.JobClient: Task Id :
> > > > attempt_201007270544_0001_r_000000_0, Status : FAILED
> > > >
> > > > Error: java.lang.NullPointerException
> > > > at java.util.concurrent.ConcurrentHashMap.get(Unknown Source)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
> > > >
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
> > > >
> > > >
> > > > 10/07/27 05:45:24 INFO mapred.JobClient: Task Id :
> > > > attempt_201007270544_0001_r_000000_1, Status : FAILED
> > > > Error: java.lang.NullPointerException
> > > > at java.util.concurrent.ConcurrentHashMap.get(Unknown Source)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
> > > >
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
> > > >
> > > >
> > > > 10/07/27 05:45:30 INFO mapred.JobClient: Task Id :
> > > > attempt_201007270544_0001_r_000000_2, Status : FAILED
> > > > Error: java.lang.NullPointerException
> > > > at java.util.concurrent.ConcurrentHashMap.get(Unknown Source)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
> > > >
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
> > > >
> > > >
> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Job complete:
> > > > job_201007270544_0001
> > > >
> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Counters: 12
> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Job Counters
> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Launched reduce tasks=4
> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Launched map tasks=1
> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Data-local map tasks=1
> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Failed reduce tasks=1
> > > > 10/07/27 05:45:39 INFO mapred.JobClient: FileSystemCounters
> > > > 10/07/27 05:45:39 INFO mapred.JobClient: HDFS_BYTES_READ=15319
> > > > 10/07/27 05:45:39 INFO mapred.JobClient: FILE_BYTES_WRITTEN=12847
> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Map-Reduce Framework
> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Combine output records=934
> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Map input records=149
> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Spilled Records=934
> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Map output bytes=25346
> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Combine input records=2541
> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Map output records=2541
> > > >
> > > > --
> > > > Dr. Sameer Joshi, Ph.D.
> > > > Senior computer scientist,
> > > > Serene Software.
> > > >
> > >
> >
> >
> >
> > --
> > Dr. Sameer Joshi, Ph.D.
> > Senior computer scientist,
> > Serene Software.
> >
>



-- 
Dr. Sameer Joshi, Ph.D.
Senior computer scientist,
Serene Software.

--001485f794de8d9698048caa0a6e--

From general-return-204-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri May 01 14:10:29 2009
Return-Path: <general-return-204-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60264 invoked from network); 1 May 2009 14:10:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 May 2009 14:10:29 -0000
Received: (qmail 55523 invoked by uid 500); 1 May 2009 14:10:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55466 invoked by uid 500); 1 May 2009 14:10:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55456 invoked by uid 99); 1 May 2009 14:10:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 May 2009 14:10:28 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of hadoop@anarres.org designates 62.49.9.82 as permitted sender)
Received: from [62.49.9.82] (HELO pink.anarres.org) (62.49.9.82)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 May 2009 14:10:21 +0000
Received: from [192.168.3.16]
	by pink.anarres.org with esmtpsa (SSLv3:AES256-SHA:256)
	(Exim 4.62)
	(envelope-from <hadoop@anarres.org>)
	id 1LztR9-00060Y-Dd
	for general@hadoop.apache.org; Fri, 01 May 2009 15:09:59 +0100
Subject: Re: Implementing compareTo in user-written keys where one extends
	the other is error prone
From: Shevek <hadoop@anarres.org>
To: general@hadoop.apache.org
In-Reply-To: <49FA18B1.8090701@schor.com>
References: <49FA18B1.8090701@schor.com>
Content-Type: text/plain
Date: Fri, 01 May 2009 15:09:29 +0100
Message-Id: <1241186969.13412.27.camel@localhost>
Mime-Version: 1.0
X-Mailer: Evolution 2.12.3 
Content-Transfer-Encoding: 7bit
X-Karma: 999: Found in custom ip4 whitelist
X-Virus-Checked: Checked by ClamAV on apache.org

So, I'll pull out the pertinent bits of your mail, please excuse the
extreme trimming, and explain why this happened at the bottom.

On Thu, 2009-04-30 at 17:31 -0400, Marshall Schor wrote:

A superclass which sorts on a String field:

> public class Super implements WritableComparable<Super> {

A subclass:

> public class Sub extends Super {
>  . . .
>   public int compareTo(Sub o) {

> What actually happened, was that the sort on the boolean value was
> skipped completely, and only the sort on the string was done.

[SNIP]
> ... they are passed via this code in
> WritableComparator:
> 
>  public int compare(WritableComparable a, WritableComparable b) {
>     return a.compareTo(b);
>   }
> 
> Java uses the interface spec for WritableComparable that was declared,
> in this case WritableComparable<Super>, and infers that the arg type for
> the compareTo is Super.  So it "skips" calling the compareTo in Sub, and
> just calls the one in Super.

Right.

> The workaround is to change the signature of Sub's compareTo method to
> match the spec in the interface, namely it has to take the Super as an
> argument, and then cast it to Sub.

Right. (As you'll see in a minute, this is what the JVM does anyway, but
starting from Object.)

> This seems like a very error prone design.  Am I doing something wrong,
> or can this be improved so that this kind of error is avoided?

Yes, you're doing something wrong. You are misunderstanding type erasure
and its implementation in Java.

At the simplest level, this class implements Comparable<Super> but never
implements Comparable<Sub>, and the method compare(Sub o) is NOT AN
OVERRIDE. I haven't tested, but if you make Sub implement
Comparable<Sub>, it might work as you intend.

If in doubt, and in any case where you believe a method to override
another method, USE THE @Override ANNOTATION. In fact, if someone were
to go through every Java 5 codebase in the world and add @Override
everywhere pertinent, it would be an improvement. Even in C++, I don't
think this method combination would be a dynamic override. You'd be
relying on enough static type information to be available to let the
compiler determine the Sub-call, which it probably wouldn't.

With the rant over, let's look at why this particular confusion happens.

Type erasure means that in the class file, Comparable drops its
parameters. Java does not know whether it's a Comparable<A> or a
Comparable<B>, and if it all goes wrong, you'll get a runtime
ClassCastException just as you would have in old-land.

The actual method is still compareTo(Object), and so when we write a
method compareTo(Super), how does it get called?

The answer is that the Java compiler compiles an extra hidden method
compareTo(Object) which forwards to compareTo(Super). I happen to have
the following example lying around, so I'll use it:

abstract class Functor<I,O> {
        public abstract O eval(I x);
}

Remember this is really Object eval(Object x) since Object is the type
bound for both I and O.

class MyFunctor extends Functor<String,String> {
        @Override
        public String eval(String in) { return in; }
}

Now, let's compile both classes and have a look:

Method name:"eval" public Signature: (java.lang.String)java.lang.String
Attribute "Code", length:26, max_stack:1, max_locals:2, code_length:2
  0: aload_1
  1: areturn

That's our expected eval method.

Method name:"eval" public volatile Signature:
(java.lang.Object)java.lang.Object
Attribute "Code", length:33, max_stack:2, max_locals:2, code_length:9
  0: aload_0
  1: aload_1
  2: checkcast <Class java.lang.String>
  5: invokevirtual <Method MyFunctor.eval
(java.lang.String)java.lang.String>
  8: areturn

This is an EXTRA method compiled by javac because the compile time type
information tells it that eval(String) is an implementation of
eval(Object) because of the relationship created by the type variables I
and O. Note the extra checkcast, and that this method has an attribute
of volatile (which is interestingly not documented as a valid method
attribute in the JVM spec).

In the case of Comparable, if you dump your class file, you will find
that compareTo(Object) forwards to compareTo(Super), hence the
a.compareTo(b) call calls compareTo(Super) and at no point does anything
know to call compareTo(Sub).

Either way, an @Override annotation will tell you at compile-time
whether you got it right, and I'm a firm believer in knowing whether
your code will work BEFORE deployment.

S.


From general-return-205-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri May 01 20:30:40 2009
Return-Path: <general-return-205-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57986 invoked from network); 1 May 2009 20:30:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 May 2009 20:30:40 -0000
Received: (qmail 1904 invoked by uid 500); 1 May 2009 20:30:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1862 invoked by uid 500); 1 May 2009 20:30:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 65668 invoked by uid 99); 1 May 2009 20:14:16 -0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received-SPF: unknown (nike.apache.org: error in processing during lookup of asachdeva@attributor.com)
Date: Fri, 1 May 2009 13:13:46 -0700 (PDT)
From: Anshuman Sachdeva <asachdeva@attributor.com>
To: general@hadoop.apache.org
Message-ID: <376870849.6850791241208826390.JavaMail.root@zimbra1.mindcentric.com>
Subject: du command hangs the machine
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-Originating-IP: [99.35.134.13]
X-Mailer: Zimbra 5.0.6_GA_2313.RHEL4_64 (ZimbraWebClient - FF3.0 (Win)/5.0.6_GA_2313.RHEL4_64)
X-Virus-Checked: Checked by ClamAV on apache.org

Hi All,
        My engineers have seen this issue. After every few days the node hangs. we are using xfs file system and when ever hadoop runs "du" command and the data on the file system is really high and it some how locks the directory and we end up rebooting the machine.

A help or guidance from any one will be helpful



Thanks

From general-return-206-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri May 01 20:34:25 2009
Return-Path: <general-return-206-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58431 invoked from network); 1 May 2009 20:34:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 May 2009 20:34:23 -0000
Received: (qmail 4263 invoked by uid 500); 1 May 2009 20:34:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4197 invoked by uid 500); 1 May 2009 20:34:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4186 invoked by uid 99); 1 May 2009 20:34:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 May 2009 20:34:23 +0000
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.62.32] (HELO QMTA03.westchester.pa.mail.comcast.net) (76.96.62.32)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 May 2009 20:34:14 +0000
Received: from OMTA10.westchester.pa.mail.comcast.net ([76.96.62.28])
	by QMTA03.westchester.pa.mail.comcast.net with comcast
	id mENt1b0040cZkys53LZmbt; Fri, 01 May 2009 20:33:46 +0000
Received: from battlerock-lm.corp.yahoo.com ([209.131.62.113])
	by OMTA10.westchester.pa.mail.comcast.net with comcast
	id mLZl1b00E2SbwD53WLZoZU; Fri, 01 May 2009 20:33:52 +0000
Message-Id: <C9F4AB38-6968-4BF6-8827-0FC13E62EE40@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <376870849.6850791241208826390.JavaMail.root@zimbra1.mindcentric.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: Re: du command hangs the machine
Date: Fri, 1 May 2009 13:33:44 -0700
References: <376870849.6850791241208826390.JavaMail.root@zimbra1.mindcentric.com>
X-Mailer: Apple Mail (2.930.3)
X-Virus-Checked: Checked by ClamAV on apache.org


On May 1, 2009, at 1:13 PM, Anshuman Sachdeva wrote:

> Hi All,
>        My engineers have seen this issue. After every few days the  
> node hangs. we are using xfs file system and when ever hadoop runs  
> "du" command and the data on the file system is really high and it  
> some how locks the directory and we end up rebooting the machine.
>
> A help or guidance from any one will be helpful

Probably the best way to solve the problem is:

https://issues.apache.org/jira/browse/HADOOP-4998

It would be great to have native libraries that provide the  
functionality we need without forking at all.

-- Owen

From general-return-207-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri May 01 20:38:54 2009
Return-Path: <general-return-207-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59591 invoked from network); 1 May 2009 20:38:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 May 2009 20:38:54 -0000
Received: (qmail 14044 invoked by uid 500); 1 May 2009 20:38:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13981 invoked by uid 500); 1 May 2009 20:38:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13956 invoked by uid 99); 1 May 2009 20:38:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 May 2009 20:38:46 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of gtsafas.bb@gmail.com designates 209.85.132.243 as permitted sender)
Received: from [209.85.132.243] (HELO an-out-0708.google.com) (209.85.132.243)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 May 2009 20:38:38 +0000
Received: by an-out-0708.google.com with SMTP id c2so1473329anc.29
        for <general@hadoop.apache.org>; Fri, 01 May 2009 13:38:17 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:x-rim-org-msg-ref-id
         :return-receipt-to:message-id:reply-to:x-priority:sensitivity
         :importance:subject:to:from:date:content-type:mime-version;
        bh=7weXhHPFrfRF8u/NT1FoBb0VssXRYWleETVkdO93rGM=;
        b=ILTQNhFOUpyy6q2eK/F6cFkayyzqUDY8qtShtwae3lSie0Qqp3KnCq4BCf1MW/FtlY
         gize5gBt21hLacy8BXGs5yeqDMyM5zvXSQ7b6lFIR3RwZUmnfHPiTtW2eJgzUzI6kee5
         YdnAdnNp7hB1lPnhU8e4W2mvbUuPIa3IGjuyo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=x-rim-org-msg-ref-id:return-receipt-to:message-id:reply-to
         :x-priority:sensitivity:importance:subject:to:from:date:content-type
         :mime-version;
        b=IkDtq73IBLOTyqqMtdnU5377qYpw7i4ofQpJQK2WhEQtGiZmMUbc1N0vjFR3JiEhw4
         peWc+/1hHDbq3EBRUTbisr39YjeOQTTQ9MKZEvadVme7lHjx9Yy6t1V4J8KtGxWMuzOE
         pWBWbjFhdHLAkjhpEEzTzHllcimYMksZBQ+Fs=
Received: by 10.100.171.7 with SMTP id t7mr262563ane.33.1241210297644;
        Fri, 01 May 2009 13:38:17 -0700 (PDT)
Received: from bda739.bisx.prod.on.blackberry (e739.bda.bis.na.blackberry.com [67.223.79.148])
        by mx.google.com with ESMTPS id c28sm371548anc.9.2009.05.01.13.38.16
        (version=SSLv3 cipher=RC4-MD5);
        Fri, 01 May 2009 13:38:16 -0700 (PDT)
X-rim-org-msg-ref-id:1997177994
Message-ID:<1997177994-1241210293-cardhu_decombobulator_blackberry.rim.net-255471184-@bxe1026.bisx.prod.on.blackberry>
Reply-To: gtsafas.bb@gmail.com
X-Priority: Normal
Sensitivity: Normal
Importance: Normal
Subject: Beginer
To: general@hadoop.apache.org
From: gtsafas.bb@gmail.com
Date: Fri, 1 May 2009 20:40:14 +0000
Content-Type: text/plain
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Hello everyone, 

I am new to hadoop. My company is looking into using it. I am trying to get an idea of it but some of the literature is beyond my comprehension. If someone would not mind giving me an easier explanation I would appreciate it.

Thank you.

Sent from my Verizon Wireless BlackBerry

From general-return-208-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri May 01 21:28:30 2009
Return-Path: <general-return-208-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 79233 invoked from network); 1 May 2009 21:28:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 May 2009 21:28:29 -0000
Received: (qmail 88840 invoked by uid 500); 1 May 2009 21:28:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88774 invoked by uid 500); 1 May 2009 21:28:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88764 invoked by uid 99); 1 May 2009 21:28:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 May 2009 21:28:28 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jason.hadoop@gmail.com designates 209.85.198.225 as permitted sender)
Received: from [209.85.198.225] (HELO rv-out-0506.google.com) (209.85.198.225)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 May 2009 21:28:19 +0000
Received: by rv-out-0506.google.com with SMTP id g37so1653115rvb.29
        for <general@hadoop.apache.org>; Fri, 01 May 2009 14:27:57 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=jGPuiK+RLoBaJktWfLukBUdDocmAVvFI6RVDrZvB0sU=;
        b=Tl0cX1T29ZYpN0uajxhq47+yTMDplMpZW+5HOPFmcruEKaDx+U6MrMl5QNvlKGBV9Y
         7odhXPTk28YHvlXYcNBs1Gbwo7oyXgz4O8z5RiMwUP+hXGZf/0BmctcX9Of2Q3i93wEy
         /aERc9j+tCtLqC1U5heLbXe2UrjFS7lG3li48=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Jxzo6gBt0AmJJu0NTRje7b7iG7TqNIINSu76P9T5wd4LB637BnlErJIhOWtRBOj2tR
         h7VQWD/a1OSx844UCZrjIYEjQ0Mn7y6cP4WQHI/xfJZ/aeEyekyL53TvT5ke39c0LwUb
         CgY1ryXBWXg//KcUQeavqBBl55adEQoDtQGdY=
MIME-Version: 1.0
Received: by 10.114.147.1 with SMTP id u1mr2394098wad.115.1241213277859; Fri, 
	01 May 2009 14:27:57 -0700 (PDT)
In-Reply-To: <C9F4AB38-6968-4BF6-8827-0FC13E62EE40@apache.org>
References: <376870849.6850791241208826390.JavaMail.root@zimbra1.mindcentric.com>
	 <C9F4AB38-6968-4BF6-8827-0FC13E62EE40@apache.org>
Date: Fri, 1 May 2009 14:27:57 -0700
Message-ID: <314098690905011427i5abcba80l65f633a7f49482c8@mail.gmail.com>
Subject: Re: du command hangs the machine
From: jason hadoop <jason.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364c5baf8472970468e07c0e
X-Virus-Checked: Checked by ClamAV on apache.org

--0016364c5baf8472970468e07c0e
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

There is a particular problem with XFS that I have only seen on PAE kernels
that can cause XFS deadlocks, on top of the long duration du issue.

We never had a solution but it only seemed to happen on the machines with
the PAE kernels (32 bit kernels with larger than 4gig of physical ram in the
machine).


On Fri, May 1, 2009 at 1:33 PM, Owen O'Malley <omalley@apache.org> wrote:

>
> On May 1, 2009, at 1:13 PM, Anshuman Sachdeva wrote:
>
>  Hi All,
>>       My engineers have seen this issue. After every few days the node
>> hangs. we are using xfs file system and when ever hadoop runs "du" command
>> and the data on the file system is really high and it some how locks the
>> directory and we end up rebooting the machine.
>>
>> A help or guidance from any one will be helpful
>>
>
> Probably the best way to solve the problem is:
>
> https://issues.apache.org/jira/browse/HADOOP-4998
>
> It would be great to have native libraries that provide the functionality
> we need without forking at all.
>
> -- Owen
>



-- 
Alpha Chapters of my book on Hadoop are available
http://www.apress.com/book/view/9781430219422

--0016364c5baf8472970468e07c0e--

From general-return-209-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat May 02 18:30:50 2009
Return-Path: <general-return-209-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84460 invoked from network); 2 May 2009 18:30:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 May 2009 18:30:49 -0000
Received: (qmail 15022 invoked by uid 500); 2 May 2009 18:30:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14945 invoked by uid 500); 2 May 2009 18:30:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 82305 invoked by uid 99); 1 May 2009 22:39:36 -0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of vijay2win@gmail.com designates 209.85.132.246 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :from:date:message-id:subject:to:content-type;
        bh=TQGRypiqU7W7EW3U556BIHyuyTOLW/ZBvCwrV+L94Xo=;
        b=VztvMmZasyrrtLibo0BYp0ab2He6Ym85iIFWdTYe+gtVLgZizXENT7R8ms6DLclabq
         SSFLMcmlVC3p8l/nF2xKNcOpnRimXe7fT7kxGVwM1/RaWCIpmv9QXbYlacYsTZd5hTG8
         97jSm5izr8UHcsrEsVHnR6/p76g90EjOgCXJk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=j3jwU2HKJk0HZ4JpgwW3cGQ4BeKRezqBqP9ijYkF962RnjKPn5M48WjDTaHEiecl5q
         rtxFkyzbxGvpX6gueYhgIT9hyr2XnDciKJB0rUMCbeTUxMam5hBj0klKxyr4e3btkmVS
         3lUlmKJd1r8ImwIOozS4vwgeiojD85proHPvg=
MIME-Version: 1.0
In-Reply-To: <1997177994-1241210293-cardhu_decombobulator_blackberry.rim.net-255471184-@bxe1026.bisx.prod.on.blackberry>
References: <1997177994-1241210293-cardhu_decombobulator_blackberry.rim.net-255471184-@bxe1026.bisx.prod.on.blackberry>
From: Vijay <vijay2win@gmail.com>
Date: Fri, 1 May 2009 15:38:49 -0700
Message-ID: <9b40bc2a0905011538x63342e9dm3cdea5b0ae74aacd@mail.gmail.com>
Subject: Re: Beginer
To: general@hadoop.apache.org, gtsafas.bb@gmail.com
Content-Type: multipart/alternative; boundary=0016e645b83c1bda190468e17b02
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e645b83c1bda190468e17b02
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

Simple way to explain is.....
Think abt it, if you have million things to do to complete a One JOB..... To
get that done you will divide it to 10 * 100 K things.... Send it 100K
server to do the job individually (Map Job)
Get the data back from it and run a reduce job which will combine all the
results and at the end you have the results of those million things....
Doing the same thing without any Loss of data or loss of processing time by
redundancy etc....

What you gain? It is like your car with 4 Cyl or 6 Cyl or even 8 Cyl in
it..... :)

the above is Just a simple over view or a 1min summary :) Well this is my
way of interpreting it.

Regards,
</VJ>




On Fri, May 1, 2009 at 1:40 PM, <gtsafas.bb@gmail.com> wrote:

> Hello everyone,
>
> I am new to hadoop. My company is looking into using it. I am trying to get
> an idea of it but some of the literature is beyond my comprehension. If
> someone would not mind giving me an easier explanation I would appreciate
> it.
>
> Thank you.
>
> Sent from my Verizon Wireless BlackBerry
>

--0016e645b83c1bda190468e17b02--

From general-return-210-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 05 21:15:27 2009
Return-Path: <general-return-210-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56641 invoked from network); 5 May 2009 21:15:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 May 2009 21:15:27 -0000
Received: (qmail 1067 invoked by uid 500); 5 May 2009 21:15:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 690 invoked by uid 500); 5 May 2009 21:15:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 644 invoked by uid 99); 5 May 2009 21:15:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 May 2009 21:15:21 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 May 2009 21:15:09 +0000
Received: from SNV-EXPF01.ds.corp.yahoo.com (snv-expf01.ds.corp.yahoo.com [207.126.227.250])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n45LBSsC074162;
	Tue, 5 May 2009 14:11:28 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:x-mimeole:content-class:mime-version:
	content-type:subject:date:message-id:x-ms-has-attach:
	x-ms-tnef-correlator:thread-topic:thread-index:from:to:x-originalarrivaltime;
	b=AjRq4vAIB8m51XbGXqMJLch5g9NunMb0zUoAFSUTaOEQQh/2sMmJRNel0Rt2/yYU
Received: from SNV-EXVS02.ds.corp.yahoo.com ([216.145.51.196]) by SNV-EXPF01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Tue, 5 May 2009 14:11:27 -0700
x-mimeole: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C9CDC5.EADEC5E7"
Subject: Hadoop Summit 2009 - Open for registration
Date: Tue, 5 May 2009 14:10:28 -0700
Message-ID: <DD27C9769EA63D43BE3CF4CCCC12DD970262AEA8@SNV-EXVS02.ds.corp.yahoo.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Hadoop Summit 2009 - Open for registration
Thread-Index: AcnNxepSpOFHRec4RX+Djty21JFqUg==
From: "Ajay Anand" <aanand@yahoo-inc.com>
To: "core-user@hadoop.apache.org" <'core-user@hadoop.apache.org'>,
        "general@hadoop.apache.org" <'general@hadoop.apache.org'>,
        "zookeeper-user@hadoop.apache.org" <'zookeeper-user@hadoop.apache.org'>,
        "hbase-user@hadoop.apache.org" <'hbase-user@hadoop.apache.org'>,
        <pig-user@hadoop.apache.org>
X-OriginalArrivalTime: 05 May 2009 21:11:27.0991 (UTC) FILETIME=[0E0FE070:01C9CDC6]
X-Virus-Checked: Checked by ClamAV on apache.org

------_=_NextPart_001_01C9CDC5.EADEC5E7
Content-Type: text/plain;
	charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable

This year's Hadoop Summit
(http://developer.yahoo.com/events/hadoopsummit09/) is confirmed for
June 10th at the Santa Clara Marriott, and is now open for registration.

=20

We have a packed agenda, with three tracks - for developers,
administrators, and one focused on new and innovative applications using
Hadoop. The presentations include talks from Amazon, IBM, Sun, Cloudera,
Facebook, HP, Microsoft, and the Yahoo! team, as well as leading
universities including UC Berkeley, CMU, Cornell, U of Maryland, U of
Nebraska and SUNY.

=20

>From our experience last year with the rush for seats, I would encourage
people to register early at http://hadoopsummit09.eventbrite.com/

=20

Looking forward to seeing you at the summit!

=20

Ajay


------_=_NextPart_001_01C9CDC5.EADEC5E7--

From general-return-211-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed May 06 01:42:40 2009
Return-Path: <general-return-211-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 61437 invoked from network); 6 May 2009 01:42:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 May 2009 01:42:40 -0000
Received: (qmail 64482 invoked by uid 500); 6 May 2009 01:42:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 64372 invoked by uid 500); 6 May 2009 01:42:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 54759 invoked by uid 99); 6 May 2009 01:26:19 -0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
Date: Wed, 6 May 2009 10:25:47 +0900
Message-ID: <69035570905051825m2dbb6ddejae7d9e5846c21214@mail.gmail.com>
Subject: Free Training at 2009 Hadoop Summit
From: Christophe Bisciglia <christophe@cloudera.com>
To: core-user@hadoop.apache.org, general@hadoop.apache.org, 
	zookeeper-user@hadoop.apache.org, hbase-user@hadoop.apache.org, 
	pig-user@hadoop.apache.org, hive-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64714c46ca856046934467d
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e64714c46ca856046934467d
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Just wanted to follow up on this and let everyone know that Cloudera and Y!
are teaming up to offer two day-long training sessions for free on the day
after the summit (June 11th).

We'll cover Hadoop basics, Pig, Hive and some new tools Cloudera is
releasing for importing data to Hadoop from existing databases.

http://hadoopsummit09-training.eventbrite.com

Each of these sessions normally runs about $1000 but were taking advantage
of having so much of the Hadoop community in one place and offering this fo=
r
free at the 2009 Hadoop Summit.

Basic training is appropriate for people just getting started with Hadoop,
and the advanced training will focus on augmenting your existing
infrastructure with Hadoop and taking advantage of Hadoop's advanced
features and related projects.

Space is limited, so sign up before time runs out.

Hope to see you there!

Christophe and the Cloduera Team

On Wed, May 6, 2009 at 6:10 AM, Ajay Anand <aanand@yahoo-inc.com> wrote:
> This year=92s Hadoop Summit
> (http://developer.yahoo.com/events/hadoopsummit09/) is confirmed for June
> 10th at the Santa Clara Marriott, and is now open for registration.
>
>
>
> We have a packed agenda, with three tracks =96 for developers,
administrators,
> and one focused on new and innovative applications using Hadoop. The
> presentations include talks from Amazon, IBM, Sun, Cloudera, Facebook, HP=
,
> Microsoft, and the Yahoo! team, as well as leading universities including
UC
> Berkeley, CMU, Cornell, U of Maryland, U of Nebraska and SUNY.
>
>
>
> From our experience last year with the rush for seats, I would encourage
> people to register early at http://hadoopsummit09.eventbrite.com/
>
>
>
> Looking forward to seeing you at the summit!
>
>
>
> Ajay



--=20
get hadoop: cloudera.com/hadoop
online training: cloudera.com/hadoop-training
blog: cloudera.com/blog
twitter: twitter.com/cloudera

--0016e64714c46ca856046934467d--

From general-return-212-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed May 06 07:37:25 2009
Return-Path: <general-return-212-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73921 invoked from network); 6 May 2009 07:37:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 May 2009 07:37:25 -0000
Received: (qmail 33823 invoked by uid 500); 6 May 2009 07:37:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33697 invoked by uid 500); 6 May 2009 07:37:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33678 invoked by uid 99); 6 May 2009 07:37:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 May 2009 07:37:21 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of p0941p@gmail.com designates 74.125.46.29 as permitted sender)
Received: from [74.125.46.29] (HELO yw-out-2324.google.com) (74.125.46.29)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 May 2009 07:37:13 +0000
Received: by yw-out-2324.google.com with SMTP id 9so2893240ywe.29
        for <multiple recipients>; Wed, 06 May 2009 00:36:52 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=Lqz8mCIcYDCj+GY43o/Vjwwh8331Pn3axggOfKqk33s=;
        b=ZEnMIN3d1RPPunkMusgcJn0lbJsJVZlzf/WLBymck+1pKLbASe3Yc4DAa5SSFTBkEe
         649nEzbX4GpFgeoQvytkBFI2smeCo4QOBJLjfOf5gArOzR2Xh3JdwqkDK4dZSlV09pvw
         +Dxy8uMoJWO4qIX36HoM/Qtfa4o5baknAlCyE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=dXqH5BHRR00hdEpsGHp6bTS7H4wj2hd2Q5eUlOTcEsYimfz2FULoAhkW7XmOYxZRo+
         ACtsfFGqH2JkwavniJvnnsl/g4LriH4qUkBJhJtW5YPfypi2zIPV6Itz9JSxvdvRXM5m
         TmrC2pTe70LKLbJOtoclq1S+wm0EAcAgXSFgs=
MIME-Version: 1.0
Received: by 10.151.69.2 with SMTP id w2mr190401ybk.190.1241595411928; Wed, 06 
	May 2009 00:36:51 -0700 (PDT)
Date: Wed, 6 May 2009 00:36:51 -0700
Message-ID: <9c39bdeb0905060036u2f1f03aamd4d7cb532be8ec6b@mail.gmail.com>
Subject: On usig Eclipse IDE
From: George Pang <p0941p@gmail.com>
To: core-user@hadoop.apache.org, general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001e680f1a3c7b957104693975b8
X-Virus-Checked: Checked by ClamAV on apache.org

--001e680f1a3c7b957104693975b8
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Dear Users,

I configure Eclipse Europa according to Yahoo tutorial on hadoop:
http://public.yahoo.com/gogate/hadoop-tutorial/html/module3.html

and in the instruction it goes about creating new DFS Location:

=93=85..Next, click on the =93Advanced=94 tab. There are two settings here =
which
must be changed.

Scroll down to hadoop.job.ugi. It contains your current Windows login
credentials. Highlight the first comma-separated value in this list (your
username) and replace it with hadoop-user.=94

I can=92t find this attribute(hadoop.job.ugi) in the advance list from =93D=
efine
Hadoop location=94 on Eclipse. Do you have an idea?

My hadoop is installed on VM ware.

Thank you, fast reply will be much appreciated.

George

--001e680f1a3c7b957104693975b8--

From general-return-213-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed May 06 17:07:04 2009
Return-Path: <general-return-213-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 51375 invoked from network); 6 May 2009 17:07:04 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 May 2009 17:07:04 -0000
Received: (qmail 69045 invoked by uid 500); 6 May 2009 17:07:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68601 invoked by uid 500); 6 May 2009 17:07:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68484 invoked by uid 99); 6 May 2009 17:06:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 May 2009 17:06:39 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of p0941p@gmail.com designates 74.125.44.29 as permitted sender)
Received: from [74.125.44.29] (HELO yx-out-2324.google.com) (74.125.44.29)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 May 2009 17:06:31 +0000
Received: by yx-out-2324.google.com with SMTP id 8so118199yxm.29
        for <multiple recipients>; Wed, 06 May 2009 10:06:10 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=QFrD38FW8rzRpw5ud8/z2iG/a+J21PGUhb+tRw8F1e4=;
        b=vcmp8ets6JugikdDe513Rv9KqSoyRBTR5C53VLC9XTydvZ1v+xN8eH6hbbGnHnVE6o
         aAvkeMJw+ULykrZhwtzSnzTK/T9iSWjQjYdrrClZmXf/y5uqr7NM8RRqT7IOiTXs1OA8
         kyESoAs3q9bR/mLgcSa8c2xNKDGMa44moe0Do=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=kfqrdhhSLemIeU2lXt5ja+rn/pkPN6Dnbok9pqFjTtlP+SfNtLhExkkV9RezM+WlIZ
         j3nqrJGwDgcm9TxiYNqZaE+N6uDHHzvJqG+kiRF/SlOZeWQqYwIMf/kUE5q5pKeQs9Wu
         OSEt0mkxtN3QapLIXjIRj01vGM1GhNNCFx0xM=
MIME-Version: 1.0
Received: by 10.151.133.9 with SMTP id k9mr1290209ybn.51.1241629570558; Wed, 
	06 May 2009 10:06:10 -0700 (PDT)
In-Reply-To: <9c39bdeb0905060036u2f1f03aamd4d7cb532be8ec6b@mail.gmail.com>
References: <9c39bdeb0905060036u2f1f03aamd4d7cb532be8ec6b@mail.gmail.com>
Date: Wed, 6 May 2009 10:06:10 -0700
Message-ID: <9c39bdeb0905061006j1b315e0fkc91bbaaad50be5c7@mail.gmail.com>
Subject: Re: On usig Eclipse IDE
From: George Pang <p0941p@gmail.com>
To: core-user@hadoop.apache.org, general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001e680f18047eecfa04694169b0
X-Virus-Checked: Checked by ClamAV on apache.org

--001e680f18047eecfa04694169b0
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Or, is there a way to know who is the author of that tutorial from Yahoo on
hadoop / Eclipse?
Thanks

George

2009/5/6 George Pang <p0941p@gmail.com>

> Dear Users,
>
> I configure Eclipse Europa according to Yahoo tutorial on hadoop:
> http://public.yahoo.com/gogate/hadoop-tutorial/html/module3.html
>
> and in the instruction it goes about creating new DFS Location:
>
> =93=85..Next, click on the =93Advanced=94 tab. There are two settings her=
e which
> must be changed.
>
> Scroll down to hadoop.job.ugi. It contains your current Windows login
> credentials. Highlight the first comma-separated value in this list (your
> username) and replace it with hadoop-user.=94
>
> I can=92t find this attribute(hadoop.job.ugi) in the advance list from
> =93Define Hadoop location=94 on Eclipse. Do you have an idea?
>
> My hadoop is installed on VM ware.
>
> Thank you, fast reply will be much appreciated.
>
> George
>

--001e680f18047eecfa04694169b0--

From general-return-214-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri May 08 07:40:45 2009
Return-Path: <general-return-214-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14181 invoked from network); 8 May 2009 07:40:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 May 2009 07:40:45 -0000
Received: (qmail 76671 invoked by uid 500); 8 May 2009 07:40:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76187 invoked by uid 500); 8 May 2009 07:40:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75919 invoked by uid 99); 8 May 2009 07:40:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 May 2009 07:40:42 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of p0941p@gmail.com designates 74.125.46.31 as permitted sender)
Received: from [74.125.46.31] (HELO yw-out-2324.google.com) (74.125.46.31)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 May 2009 07:40:34 +0000
Received: by yw-out-2324.google.com with SMTP id 9so731381ywe.29
        for <multiple recipients>; Fri, 08 May 2009 00:40:13 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=nIR6jqU8qcbDg+OjE3CJaD0SAXb6587Kf/JXCCOaI6A=;
        b=Mc3JglebiQth3oEIU7/WwLBhuyJ224FK6yG+j55biJEgMY3yr2uTlvhSI2K51da5Qa
         OrW6Fic990uwVMOfpFVUs522muDNDQcoBRxatGIalwNfXAZrcHhLeSPtboWIe76g5gp8
         HE1nwqq4hpV+965giZCDZmszABRp3SvHJFRu8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=DzmgAtVMXmr3n9JpWZ8+Z9nvoJ2i223l6ktVVUSelOPNwKcNR71ek/X469EUX7a7S4
         0tDA/UustQpicxJenVX7ROxCqRV+fPXbKo3NdZnxYCX7EoVX2HuVFmba1Gq2AFOMAi0q
         rAaKdzkaQG7xDWjDhqsFaUX68CayaPDv//4cI=
MIME-Version: 1.0
Received: by 10.151.129.8 with SMTP id g8mr6116450ybn.104.1241768413683; Fri, 
	08 May 2009 00:40:13 -0700 (PDT)
Date: Fri, 8 May 2009 00:40:13 -0700
Message-ID: <9c39bdeb0905080040q6cc61652u6135eec4c63b3ff@mail.gmail.com>
Subject: ClassNotFoundException
From: George Pang <p0941p@gmail.com>
To: general@hadoop.apache.org, core-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001e680f10f030e38b046961bd8e
X-Virus-Checked: Checked by ClamAV on apache.org

--001e680f10f030e38b046961bd8e
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Dear  users,
I got "ClassNotFoundException" when run the WordCount example on hadoop
using Eclipse.  Does anyone know where is the problem?

Thank you!

George

--001e680f10f030e38b046961bd8e--

From general-return-215-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri May 08 18:52:47 2009
Return-Path: <general-return-215-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22791 invoked from network); 8 May 2009 18:52:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 May 2009 18:52:47 -0000
Received: (qmail 92844 invoked by uid 500); 8 May 2009 18:52:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92790 invoked by uid 500); 8 May 2009 18:52:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 92780 invoked by uid 99); 8 May 2009 18:52:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 May 2009 18:52:46 +0000
X-ASF-Spam-Status: No, hits=3.7 required=10.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of vijay2win@gmail.com designates 209.85.198.229 as permitted sender)
Received: from [209.85.198.229] (HELO rv-out-0506.google.com) (209.85.198.229)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 May 2009 18:52:36 +0000
Received: by rv-out-0506.google.com with SMTP id g37so1211007rvb.29
        for <general@hadoop.apache.org>; Fri, 08 May 2009 11:52:14 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:from:date:message-id
         :subject:to:content-type;
        bh=YIXa5alnnVr0fy8GKf+KevCEq55e4k5rrvSa7zv9dBA=;
        b=iqRSS41ux/ZiOIOzEuhk3nQNH/kuBoy21yC13bW1hgjYc0Vk8OlI9Gc2escJICNfE7
         QwS+ExFk4aZTQAPAtwYcuhKajGdf7U3lC65dYxxAYeJUCC/XrHf2MwWAekaHjLbDGW4+
         E1tVCT8u7OSlNpgrUvVjjSJIZvn5vfic2M0UA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type;
        b=LgM1q8TaoSgRoDW6JsmssmS0r7vqfzpBhw1KdlyBeSG2N7gEp0ErI+cBt3dsK4dyso
         ITeVFz1k5BWFgAeUjuCBkrHFFXJ+j7RPKuJV1mBQldWEZTKP1wTshJKaTV+YH8XyOzY7
         x/+6KGjnFNk+0dLB51qqi+nTW31+T2cLhBMc8=
MIME-Version: 1.0
Received: by 10.114.131.11 with SMTP id e11mr3642095wad.75.1241808734342; Fri, 
	08 May 2009 11:52:14 -0700 (PDT)
From: Vijay <vijay2win@gmail.com>
Date: Fri, 8 May 2009 11:51:54 -0700
Message-ID: <9b40bc2a0905081151r2a1fd1batdce5ed97f8d337e5@mail.gmail.com>
Subject: File system filling with Logs - Hadoop+ Chukwa
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364c5b2f7d4f3304696b2073
X-Virus-Checked: Checked by ClamAV on apache.org

--0016364c5b2f7d4f3304696b2073
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

Hi,
I am trying out Chukwa on HDFS and the filesystem is been filled up because
of the amount of logs been generated by hadoop filesystem, Can anyone help
me understand why this is happening and how can i reduce the speed atleast?

awapisvr:qawapiall:root > df -m
Filesystem           1M-blocks      Used Available Use% Mounted on
/dev/sda2                19234      7987     10270  44% /
/dev/sda1                  282        24       243   9% /boot
none                      3983         0      3983   0% /dev/shm
/dev/sda3                38459       903     35604   3% /opt
/dev/sda8               338395     43645    277561  14% /spare
/dev/sda7                19234        77     18180   1% /tmp
/dev/sda5                38459      1701     34805   5% /var
64.68.126.215:/vol/vol2/logarchive
                        286720     52678    234043  19% /backup
awapisvr:qawapiall:root > df -m
Filesystem           1M-blocks      Used Available Use% Mounted on
/dev/sda2                19234      7987     10270  44% /
/dev/sda1                  282        24       243   9% /boot
none                      3983         0      3983   0% /dev/shm
/dev/sda3                38459       903     35604   3% /opt
/dev/sda8               338395     43648    277559  14% /spare
/dev/sda7                19234        77     18180   1% /tmp
/dev/sda5                38459      1701     34805   5% /var
64.68.126.215:/vol/vol2/logarchive
                        286720     52678    234043  19% /backup


in matter of millisec

i see the below in the logs...

2009-05-08 18:50:38,351 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=root,root,bin,daemon,sys,adm,disk,wheel     ip=/127.0.0.1
cmd=listStatus  src=/chukwa/dataSinkArchives    dst=nullperm=null
2009-05-08 18:50:38,352 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=root,root,bin,daemon,sys,adm,disk,wheel     ip=/127.0.0.1
cmd=listStatus  src=/chukwa/dataSinkArchives    dst=nullperm=null
2009-05-08 18:50:38,352 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=root,root,bin,daemon,sys,adm,disk,wheel     ip=/127.0.0.1
cmd=listStatus  src=/chukwa/dataSinkArchives    dst=nullperm=null
2009-05-08 18:50:38,352 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=root,root,bin,daemon,sys,adm,disk,wheel     ip=/127.0.0.1
cmd=listStatus  src=/chukwa/dataSinkArchives    dst=nullperm=null
2009-05-08 18:50:38,352 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=root,root,bin,daemon,sys,adm,disk,wheel     ip=/127.0.0.1
cmd=listStatus  src=/chukwa/dataSinkArchives    dst=nullperm=null
2009-05-08 18:50:38,353 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=root,root,bin,daemon,sys,adm,disk,wheel     ip=/127.0.0.1
cmd=listStatus  src=/chukwa/dataSinkArchives    dst=nullperm=null


and

28527974-127.0.0.1-50010-1241721203027, blockid:
blk_-4466028835570444014_1290
2009-05-08 18:48:54,137 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block
blk_-4466028835570444014_1290 terminating
2009-05-08 18:50:33,139 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_-7272993078256778328_1291 src: /127.0.0.1:57067 dest: /127.0.0.1:50010
2009-05-08 18:50:34,222 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
127.0.0.1:57067, dest: /127.0.0.1:50010, bytes: 69046, op: HDFS_WRITE,
cliID: DFSClient_-223712422, srvID:
DS-2128527974-127.0.0.1-50010-1241721203027, blockid:
blk_-7272993078256778328_1291
2009-05-08 18:50:34,223 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block
blk_-7272993078256778328_1291 terminating
2009-05-08 18:50:35,145 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_-2496183264937203840_1292 src: /127.0.0.1:57136 dest: /127.0.0.1:50010
2009-05-08 18:50:41,920 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_2625289997411230424_1240




Regards,
</VJ>

--0016364c5b2f7d4f3304696b2073--

From general-return-216-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat May 09 07:43:32 2009
Return-Path: <general-return-216-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30986 invoked from network); 9 May 2009 07:43:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 May 2009 07:43:32 -0000
Received: (qmail 35639 invoked by uid 500); 9 May 2009 07:43:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 35588 invoked by uid 500); 9 May 2009 07:43:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 35578 invoked by uid 99); 9 May 2009 07:43:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 09 May 2009 07:43:30 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cpbhagtani@gmail.com designates 209.85.142.191 as permitted sender)
Received: from [209.85.142.191] (HELO ti-out-0910.google.com) (209.85.142.191)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 09 May 2009 07:43:21 +0000
Received: by ti-out-0910.google.com with SMTP id 28so170810tif.9
        for <general@hadoop.apache.org>; Sat, 09 May 2009 00:42:59 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=coYLSJcGj6IWBzKz0kxDSiHaYX6mnALU6o5TagV72lo=;
        b=F7H+/TmmJQ5Nd7yoUxlZ7GPIuVqab0BNHPFkwrp7/Ie9mhJzUMZWQAJI3ioPQ2lGDd
         6ZAHIulk1oLuH6xrZlf1YeS9+ivfXYMTDKYHS+sb3p/YAN24tAfOj8srn2QcfNNczvMe
         wtzxCW7ZkNwEkYOKIhN6j9+SuxS6QieV7ovzs=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=oWcdlfSp7+VgznlguDYTqlbGS3LnTsl1HvaMlugVaVa4wHVThqK6741bdIncEd/uwZ
         a8cyIm+zXH9VNJVP3FKomRSXhWufF3cjyIh8s3K99gpcjT5eEhha4SMpbWD5IvRGHYTo
         fRTLI3T8Z0azhk6TORL6Oi3F5A8K0u9NEK38g=
MIME-Version: 1.0
Received: by 10.110.47.17 with SMTP id u17mr312256tiu.41.1241854979532; Sat, 
	09 May 2009 00:42:59 -0700 (PDT)
Date: Sat, 9 May 2009 13:12:59 +0530
Message-ID: <4061df20905090042g1050facfv9074f87799b97df0@mail.gmail.com>
Subject: Problem Configuring Hadoop Eclipse Plugin
From: Chandraprakash Bhagtani <cpbhagtani@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e652f56aeae882046975e411
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e652f56aeae882046975e411
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi,

I am trying to configure hadoop eclipse plugin 0.19.1 but getting the
following exception

An internal error occurred during: "Connecting to DFS Test".
java.lang.IllegalStateException

Please help

-- 
Thanks & Regards,
Chandra Prakash Bhagtani,

--0016e652f56aeae882046975e411--

From general-return-217-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat May 09 18:32:08 2009
Return-Path: <general-return-217-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43750 invoked from network); 9 May 2009 18:32:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 May 2009 18:32:08 -0000
Received: (qmail 12819 invoked by uid 500); 9 May 2009 18:32:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12717 invoked by uid 500); 9 May 2009 18:32:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12707 invoked by uid 99); 9 May 2009 18:32:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 09 May 2009 18:32:07 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [68.142.199.177] (HELO web301.biz.mail.mud.yahoo.com) (68.142.199.177)
    by apache.org (qpsmtpd/0.29) with SMTP; Sat, 09 May 2009 18:31:56 +0000
Received: (qmail 76533 invoked by uid 60001); 9 May 2009 18:31:33 -0000
Message-ID: <672699.72731.qm@web301.biz.mail.mud.yahoo.com>
X-YMail-OSG: 1cVt6gQVM1nxtMjzLo0K9awlLfcCRUe3jr6A5eLSlNBAOefQyT8n8wyHMWy14vJEdIuw4bomEM2nkSCpJ0EzbIyUyl2sBuAAuST2s.onMeO3_W7tnGeaamq5JHFwEVUwqZB6Qj4Hmu.F6O_5cx30.J_3dSOt821CULWtRPq7csnCYmaypl8kS.aBBeKTeJMfHphWGTeTg6UEpxokwlfMTF6A_zOC043637RdaFOnEZ9dLNn44lTtgvO_sBoHLocUqeVUI4.Bldr0VOAUnIYjdQqiuuxhpxKMlt7qFsiT7AZ_71IsolpTuQztF99RZiQnIW02cUmOze9jfVFEEmkKWu4.xo71CN5hZpju296D
Received: from [216.103.210.17] by web301.biz.mail.mud.yahoo.com via HTTP; Sat, 09 May 2009 11:31:33 PDT
X-Mailer: YahooMailClassic/5.3.9 YahooMailWebService/0.7.289.10
Date: Sat, 9 May 2009 11:31:33 -0700 (PDT)
From: Bonesata <bonesata@bonesata.com>
Subject: (event) How Hadoop Enables Big Data for Every Enterprise 
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org


Registration: http://www.meetup.com/CIO-IT-Executives/calendar/10266376/

Do you have too much data? Do you find yourself discarding data due to performance and cost considerations? Have you heard about Hadoop, but find yourself wondering how it fits with the rest of your systems?

Come here the answers to these questions and more at Cloudera's office
on Tuesday May 19th.

This talk will provide a high level, mildly technical overview of Hadoop and existing architectures for working with large volumes of data. We'll pay special attention to how to augment your existing systems with Hadoop so each can focus on what they do best. Working together, they can deliver a more powerful, scalable, and robust solution.

Our speaker: Christophe Bisciglia, Founder, Cloudera

Dinner will be served.

From general-return-218-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 12 05:14:43 2009
Return-Path: <general-return-218-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56090 invoked from network); 12 May 2009 05:14:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 May 2009 05:14:42 -0000
Received: (qmail 27093 invoked by uid 500); 12 May 2009 05:14:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27003 invoked by uid 500); 12 May 2009 05:14:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 25106 invoked by uid 99); 9 May 2009 07:24:44 -0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mingzong.lyu@gmail.com designates 209.85.132.242 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=ythkSnELKcv52qShYOV+YOijGX6oWMWdNH8e2jGtfeM=;
        b=f2vA5dij9ief3f8m4NVfbwTqh6eqfX+J92E/alFv1MSD1AEek/nITS4WSfMCX9MA5z
         6mk/6C75nGceA3C4KhTeCRpSD/L7WhMOA1SBHdbcul6r29b8EjthpadDRwO73bx4yuTc
         XL0cTWJz+YHyfs6R6M4Wm/dWGREtSpErZxYik=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=uvOE2HsS7Bh2xUe/sDaoAbSRJIFQdc/ejZQXv4M0wtBoLaN+1sNv3TuavXJvelsgP5
         j9hfABz30amQGq7Tx46qRlRNTL5VZ4kVYRR49+1I+2O8SoPktyLMT0GmH7UT0ynV68SH
         YfJzztmFrs0xWUw5PccfaMeI3b4InSUflVDHw=
MIME-Version: 1.0
Date: Sat, 9 May 2009 15:24:15 +0800
Message-ID: <11bb374e0905090024r20492097x64fe81da4037bb2a@mail.gmail.com>
Subject: A Question of Hadoop
From: Dylan <mingzong.lyu@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6441f3aedc62b046975a154
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6441f3aedc62b046975a154
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi! My name is Ming-Zong.LYU.
I have setup Hadoop some advanced question.


first question:
Settings where time of death datanode?
Is dfs.heartbeat.interval?
Second question:
How to setup more than two namenode and how do I setup hfs?
How can i do?

Can you help me ?

--0016e6441f3aedc62b046975a154--

From general-return-219-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 12 05:22:29 2009
Return-Path: <general-return-219-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59859 invoked from network); 12 May 2009 05:22:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 May 2009 05:22:28 -0000
Received: (qmail 31966 invoked by uid 500); 12 May 2009 05:22:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31881 invoked by uid 500); 12 May 2009 05:22:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31871 invoked by uid 99); 12 May 2009 05:22:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 May 2009 05:22:27 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of starrysl@gmail.com designates 209.85.142.185 as permitted sender)
Received: from [209.85.142.185] (HELO ti-out-0910.google.com) (209.85.142.185)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 May 2009 05:22:18 +0000
Received: by ti-out-0910.google.com with SMTP id 28so312716tif.9
        for <general@hadoop.apache.org>; Mon, 11 May 2009 22:21:56 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=inf1FNOdlnvCM36JGHfHGL+xQ+g7Amwe9WU3LIRYEZM=;
        b=NMLK6qAnnMa3vIwGBZEXDPSC9YO4H8VcAHDVOuhwUfKx4eA/qRSBWv3POjZx9qA/0P
         2sb2GLOKwA/OnDBxkMon9/kGWXQdCI4nFD2Na4vbqmtxApQFOTTKGQ/6JQga24Aoqidm
         yRGVYkBOPjd3bW9dQ37lRiAW4ThbYqxcoMo5s=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=uYfgWO8+SjrHdAZS8QFucBeIpZSynCxALs2cBt+Yk5BQGyre0qxS6VL2QdMg28MfYg
         eSHm1RvFoFAmtADHcq9nFl+xjleT7OaYTTwNvcDb0qE1IK9k46BTxkfRl0btAYGZA0iH
         uZcmJ+5q1h3DOYKm4ZDXXnUuJQl7Df2A685Sg=
MIME-Version: 1.0
Received: by 10.110.95.11 with SMTP id s11mr512714tib.24.1242105716080; Mon, 
	11 May 2009 22:21:56 -0700 (PDT)
In-Reply-To: <11bb374e0905090024r20492097x64fe81da4037bb2a@mail.gmail.com>
References: <11bb374e0905090024r20492097x64fe81da4037bb2a@mail.gmail.com>
From: Starry SHI <starrysl@gmail.com>
Date: Tue, 12 May 2009 13:21:36 +0800
Message-ID: <a5accba30905112221n168f87b3t641012f82455867@mail.gmail.com>
Subject: Re: A Question of Hadoop
To: general@hadoop.apache.org, mingzong.lyu@gmail.com
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hi, Dylan. About the second question, why do you need more than 2
namenode? I think one node for namenode is enough for processing, or
the interaction and synchronization will be a big problem.

Best regards,
Starry

/* Tomorrow is another day. So is today. */



On Sat, May 9, 2009 at 15:24, Dylan <mingzong.lyu@gmail.com> wrote:
> Hi! My name is Ming-Zong.LYU.
> I have setup Hadoop some advanced question.
>
>
> first question:
> Settings where time of death datanode?
> Is dfs.heartbeat.interval?
> Second question:
> How to setup more than two namenode and how do I setup hfs?
> How can i do?
>
> Can you help me ?
>

From general-return-220-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 12 05:27:59 2009
Return-Path: <general-return-220-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 62569 invoked from network); 12 May 2009 05:27:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 May 2009 05:27:58 -0000
Received: (qmail 34885 invoked by uid 500); 12 May 2009 05:27:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34759 invoked by uid 500); 12 May 2009 05:27:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34730 invoked by uid 99); 12 May 2009 05:27:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 May 2009 05:27:53 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of starrysl@gmail.com designates 209.85.142.190 as permitted sender)
Received: from [209.85.142.190] (HELO ti-out-0910.google.com) (209.85.142.190)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 May 2009 05:27:45 +0000
Received: by ti-out-0910.google.com with SMTP id 28so313084tif.9
        for <multiple recipients>; Mon, 11 May 2009 22:27:23 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:from:date:message-id
         :subject:to:content-type:content-transfer-encoding;
        bh=jlHkLotgiKAwRo088u/nKQ8X0+gdCmAM2XoztEIkXb0=;
        b=w+vRPdJqAvTxtekuoU8arMgWfdMlHhUvoB5z2Xt8es8VyKPb8PD3eJ5LjQqImk5ehg
         edo5rGimEEV+Q8lATHj/5KXwg5wNZ39SWZqw/+8q19DkKdt5UZee4XsBeSOHOswP/kEz
         lExoifApVWYjIWbveOPWIASAz7pfYucN3EYOc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        b=w4FBvgTquLNfSHSwyxvkRz6HRfZHBHsHxx2rOv0iB8LfmLUwcoPrYxO84/3ym3rOnA
         J4as2xf+vSWDQuHHZz1ZYCdEe63ui0IwWF7f7JnGyUbEWvu1+I/qxrVfHhaSEKse2CXc
         GwiCfcYpgfaxx59nRFEB8+BhoVN7V3wYVltFU=
MIME-Version: 1.0
Received: by 10.110.95.3 with SMTP id s3mr521514tib.55.1242106043277; Mon, 11 
	May 2009 22:27:23 -0700 (PDT)
From: Starry SHI <starrysl@gmail.com>
Date: Tue, 12 May 2009 13:27:03 +0800
Message-ID: <a5accba30905112227w5b222521k68465f3a027a4dda@mail.gmail.com>
Subject: hadoop getProtocolVersion and getBuildVersion error
To: core-user@hadoop.apache.org, general@hadoop.apache.org, 
	core-dev@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hi, all. Today I noticed that my hadoop cluster (r0.20.0+jdk1.6) threw
some errors in RPC handling. Below is part of the content of namenode
log file:

2009-05-12 10:27:08,200 INFO org.apache.hadoop.ipc.Server: Error
register getProtocolVersion
java.lang.IllegalArgumentException: Duplicate metricsName:getProtocolVersion
	at org.apache.hadoop.metrics.util.MetricsRegistry.add(MetricsRegistry.java:56)
	at org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:89)
	at org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:99)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:523)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
2009-05-12 10:27:08,200 INFO org.apache.hadoop.ipc.Server: Error
register getProtocolVersion
java.lang.IllegalArgumentException: Duplicate metricsName:getProtocolVersion
	at org.apache.hadoop.metrics.util.MetricsRegistry.add(MetricsRegistry.java:56)
	at org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:89)
	at org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:99)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:523)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
2009-05-12 10:27:08,200 INFO org.apache.hadoop.ipc.Server: Error
register getProtocolVersion
java.lang.IllegalArgumentException: Duplicate metricsName:getProtocolVersion
	at org.apache.hadoop.metrics.util.MetricsRegistry.add(MetricsRegistry.java:56)
	at org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:89)
	at org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:99)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:523)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
2009-05-12 10:27:08,276 INFO org.apache.hadoop.ipc.Server: Error
register getBuildVersion
java.lang.IllegalArgumentException: Duplicate metricsName:getBuildVersion
	at org.apache.hadoop.metrics.util.MetricsRegistry.add(MetricsRegistry.java:56)
	at org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:89)
	at org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:99)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:523)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)

I noticed that the similar error appears on the log in every datanode.
Can anybody tell me how to fix this?

I have patched this:
https://issues.apache.org/jira/browse/HADOOP-5139, but the error still
exist. I really don't know what to do and am expecting for your help!

Best regards,
Starry

/* Tomorrow is another day. So is today. */

From general-return-221-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed May 13 20:14:15 2009
Return-Path: <general-return-221-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 3493 invoked from network); 13 May 2009 20:14:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 May 2009 20:14:15 -0000
Received: (qmail 54374 invoked by uid 500); 13 May 2009 20:14:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53773 invoked by uid 500); 13 May 2009 20:14:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53723 invoked by uid 99); 13 May 2009 20:14:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 May 2009 20:14:09 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 May 2009 20:13:56 +0000
Received: from SNV-EXBH01.ds.corp.yahoo.com (snv-exbh01.ds.corp.yahoo.com [207.126.227.249])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n4DKBvL4096766;
	Wed, 13 May 2009 13:11:57 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:x-mimeole:content-class:mime-version:
	content-type:subject:date:message-id:in-reply-to:x-ms-has-attach:
	x-ms-tnef-correlator:thread-topic:thread-index:references:from:to:x-originalarrivaltime;
	b=nf2SEWW+e8PJVemOi1nsfftEJ6dve9VnoupwJ9LDtsp273PC/QODAu4mny/v8pfR
Received: from SNV-EXVS02.ds.corp.yahoo.com ([216.145.51.196]) by SNV-EXBH01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Wed, 13 May 2009 13:11:57 -0700
x-mimeole: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C9D407.10A6DC71"
Subject: Hadoop User Group (Bay Area) May 20th
Date: Wed, 13 May 2009 13:11:56 -0700
Message-ID: <DD27C9769EA63D43BE3CF4CCCC12DD970262B123@SNV-EXVS02.ds.corp.yahoo.com>
In-Reply-To: <AD0C16AAE446924586B68FC0EB421C9BAE155806@SNV-EXVS02.ds.corp.yahoo.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Hadoop User Group (Bay Area) May 20th
Thread-Index: AcnNxepSpOFHRec4RX+Djty21JFqUgGP/ssQ
References: <AD0C16AAE446924586B68FC0EB421C9BAE155806@SNV-EXVS02.ds.corp.yahoo.com>
From: "Ajay Anand" <aanand@yahoo-inc.com>
To: "core-user@hadoop.apache.org" <'core-user@hadoop.apache.org'>,
        "general@hadoop.apache.org" <'general@hadoop.apache.org'>,
        "zookeeper-user@hadoop.apache.org" <'zookeeper-user@hadoop.apache.org'>,
        "hbase-user@hadoop.apache.org" <'hbase-user@hadoop.apache.org'>,
        <pig-user@hadoop.apache.org>
X-OriginalArrivalTime: 13 May 2009 20:11:57.0496 (UTC) FILETIME=[112F9380:01C9D407]
X-Virus-Checked: Checked by ClamAV on apache.org

------_=_NextPart_001_01C9D407.10A6DC71
Content-Type: text/plain;
	charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable

The next Bay Area Hadoop User Group meeting is scheduled for Wednesday,
May 20th at Yahoo! 700 First Avenue, Sunnyvale, Building E, Class room
10 from 6:00-7:30 pm. Please note that the location has changed from our
usual meeting place in Santa Clara.=20

Registration: http://upcoming.yahoo.com/event/2659418/

=20

Agenda:

- Hadoop Summit 2009 preview (registration open at
http://hadoopsummit09.eventbrite.com/ )
- Hadoop with GPFS - Prasenjit Sarkar, IBM
- SQoop - Cloudera



Look forward to seeing you there!

Ajay

=20

=20


------_=_NextPart_001_01C9D407.10A6DC71--

From general-return-222-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu May 14 12:49:15 2009
Return-Path: <general-return-222-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16700 invoked from network); 14 May 2009 12:49:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 May 2009 12:49:15 -0000
Received: (qmail 43923 invoked by uid 500); 14 May 2009 12:49:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 43483 invoked by uid 500); 14 May 2009 12:49:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 43271 invoked by uid 99); 14 May 2009 12:49:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 May 2009 12:49:10 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of starrysl@gmail.com designates 209.85.142.186 as permitted sender)
Received: from [209.85.142.186] (HELO ti-out-0910.google.com) (209.85.142.186)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 May 2009 12:49:02 +0000
Received: by ti-out-0910.google.com with SMTP id 28so139271tif.9
        for <multiple recipients>; Thu, 14 May 2009 05:48:41 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=jVTM/PcqEf6uDghPhf0kOKLpTVo4TeZL7B9t3N3O1jY=;
        b=J51rCwvqgNo6tNYZZxpsCfxbUJW3Od5QDw3vmE7fdkW8EuxuSEYrcgdrI3AfbXnpY1
         6Z9R3jH1F/GB6XkDTl81914S1IXjsX6mUPFMutwTbYmRO7m5+/GNWxEV3Pze8D5Sj4Mo
         iD07u55ymhfAIQ+u+g8J8zNPE7JrE3LzadGgU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=SHldTbTVH6kSiwKFf67gm2gBJBTn9U0LxFnYB19zb0RIY85ZGpb2oKHFkwXK8jON9g
         CZyh63fvqQFZr3p8Y7Xr1sMZgdMHoZpDBc8ylXAzzKtsrSz8Wefq88Zf46YicQYtbQNU
         7Qgsnbrq7MisIx+HWRMY6RP+b7+5UZEWJtbac=
MIME-Version: 1.0
Received: by 10.110.57.6 with SMTP id f6mr159243tia.25.1242305321054; Thu, 14 
	May 2009 05:48:41 -0700 (PDT)
In-Reply-To: <a5accba30905112227w5b222521k68465f3a027a4dda@mail.gmail.com>
References: <a5accba30905112227w5b222521k68465f3a027a4dda@mail.gmail.com>
From: Starry SHI <starrysl@gmail.com>
Date: Thu, 14 May 2009 20:48:21 +0800
Message-ID: <a5accba30905140548r5bcb5f96hcbd13a07ed487562@mail.gmail.com>
Subject: Re: hadoop getProtocolVersion and getBuildVersion error
To: core-user@hadoop.apache.org, general@hadoop.apache.org, 
	core-dev@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Nobody has encountered with these problems: "Error
register getProtocolVersion" and "Error
register getBuildVersion"?

Starry

/* Tomorrow is another day. So is today. */



On Tue, May 12, 2009 at 13:27, Starry SHI <starrysl@gmail.com> wrote:
> Hi, all. Today I noticed that my hadoop cluster (r0.20.0+jdk1.6) threw
> some errors in RPC handling. Below is part of the content of namenode
> log file:
>
> 2009-05-12 10:27:08,200 INFO org.apache.hadoop.ipc.Server: Error
> register getProtocolVersion
> java.lang.IllegalArgumentException: Duplicate metricsName:getProtocolVers=
ion
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.metrics.util.MetricsRegis=
try.add(MetricsRegistry.java:56)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.metrics.util.MetricsTimeV=
aryingRate.<init>(MetricsTimeVaryingRate.java:89)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.metrics.util.MetricsTimeV=
aryingRate.<init>(MetricsTimeVaryingRate.java:99)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.ipc.RPC$Server.call(RPC.j=
ava:523)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.ipc.Server$Handler$1.run(=
Server.java:959)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.ipc.Server$Handler$1.run(=
Server.java:955)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.security.AccessController.doPrivileged=
(Native Method)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at javax.security.auth.Subject.doAs(Subject.ja=
va:396)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.ipc.Server$Handler.run(Se=
rver.java:953)
> 2009-05-12 10:27:08,200 INFO org.apache.hadoop.ipc.Server: Error
> register getProtocolVersion
> java.lang.IllegalArgumentException: Duplicate metricsName:getProtocolVers=
ion
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.metrics.util.MetricsRegis=
try.add(MetricsRegistry.java:56)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.metrics.util.MetricsTimeV=
aryingRate.<init>(MetricsTimeVaryingRate.java:89)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.metrics.util.MetricsTimeV=
aryingRate.<init>(MetricsTimeVaryingRate.java:99)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.ipc.RPC$Server.call(RPC.j=
ava:523)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.ipc.Server$Handler$1.run(=
Server.java:959)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.ipc.Server$Handler$1.run(=
Server.java:955)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.security.AccessController.doPrivileged=
(Native Method)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at javax.security.auth.Subject.doAs(Subject.ja=
va:396)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.ipc.Server$Handler.run(Se=
rver.java:953)
> 2009-05-12 10:27:08,200 INFO org.apache.hadoop.ipc.Server: Error
> register getProtocolVersion
> java.lang.IllegalArgumentException: Duplicate metricsName:getProtocolVers=
ion
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.metrics.util.MetricsRegis=
try.add(MetricsRegistry.java:56)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.metrics.util.MetricsTimeV=
aryingRate.<init>(MetricsTimeVaryingRate.java:89)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.metrics.util.MetricsTimeV=
aryingRate.<init>(MetricsTimeVaryingRate.java:99)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.ipc.RPC$Server.call(RPC.j=
ava:523)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.ipc.Server$Handler$1.run(=
Server.java:959)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.ipc.Server$Handler$1.run(=
Server.java:955)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.security.AccessController.doPrivileged=
(Native Method)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at javax.security.auth.Subject.doAs(Subject.ja=
va:396)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.ipc.Server$Handler.run(Se=
rver.java:953)
> 2009-05-12 10:27:08,276 INFO org.apache.hadoop.ipc.Server: Error
> register getBuildVersion
> java.lang.IllegalArgumentException: Duplicate metricsName:getBuildVersion
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.metrics.util.MetricsRegis=
try.add(MetricsRegistry.java:56)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.metrics.util.MetricsTimeV=
aryingRate.<init>(MetricsTimeVaryingRate.java:89)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.metrics.util.MetricsTimeV=
aryingRate.<init>(MetricsTimeVaryingRate.java:99)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.ipc.RPC$Server.call(RPC.j=
ava:523)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.ipc.Server$Handler$1.run(=
Server.java:959)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.ipc.Server$Handler$1.run(=
Server.java:955)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.security.AccessController.doPrivileged=
(Native Method)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at javax.security.auth.Subject.doAs(Subject.ja=
va:396)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.ipc.Server$Handler.run(Se=
rver.java:953)
>
> I noticed that the similar error appears on the log in every datanode.
> Can anybody tell me how to fix this?
>
> I have patched this:
> https://issues.apache.org/jira/browse/HADOOP-5139, but the error still
> exist. I really don't know what to do and am expecting for your help!
>
> Best regards,
> Starry
>
> /* Tomorrow is another day. So is today. */
>

From general-return-223-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu May 14 12:55:23 2009
Return-Path: <general-return-223-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 18574 invoked from network); 14 May 2009 12:55:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 May 2009 12:55:23 -0000
Received: (qmail 60149 invoked by uid 500); 14 May 2009 12:55:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60051 invoked by uid 500); 14 May 2009 12:55:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60025 invoked by uid 99); 14 May 2009 12:55:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 May 2009 12:55:22 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of vermaabhishekp@gmail.com designates 209.85.217.167 as permitted sender)
Received: from [209.85.217.167] (HELO mail-gx0-f167.google.com) (209.85.217.167)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 May 2009 12:55:14 +0000
Received: by gxk11 with SMTP id 11so2414072gxk.5
        for <multiple recipients>; Thu, 14 May 2009 05:54:53 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:cc:content-type;
        bh=iJGysT0sohqUPGKg+Tr9vE+/0G1aaIYTpj3S+r46EI0=;
        b=W6ilKmQXJsMPYCtXJEZU1qLGf2FCrsiditOcghUVooIaKIVhER5NtqrK8s9KakyFsh
         zKjvbM5Mf9bWptxPjg5mUEunjiMZXOO1kR1l+R2Ung1TOYPn5j18DGU7W5WzA0+m+8Vy
         MHGugam2xKa9iLYqy4kexUrFD+ciV5CQoQpe8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=iECLtT+lSHmVz6CawrWnS9gs+5wL54JskBkpr+UuCSygVU4BkGH935ll00yu0RnKLg
         ChAgvwCWkyVL/7XoebQ8RlYVDHW8ZsfkLyz1pAZ9Lwn+8efd0WD5cOTJdgOXwy5fVM8H
         OO7i5+ybaKuG1XQ1SdZTCOaWXSBdfnFtfY5Xg=
MIME-Version: 1.0
Received: by 10.151.135.4 with SMTP id m4mr3564395ybn.55.1242305692231; Thu, 
	14 May 2009 05:54:52 -0700 (PDT)
In-Reply-To: <a5accba30905140548r5bcb5f96hcbd13a07ed487562@mail.gmail.com>
References: <a5accba30905112227w5b222521k68465f3a027a4dda@mail.gmail.com>
	 <a5accba30905140548r5bcb5f96hcbd13a07ed487562@mail.gmail.com>
Date: Thu, 14 May 2009 07:54:52 -0500
Message-ID: <3c682ecd0905140554q1b8c5897j703d31a3725198e7@mail.gmail.com>
Subject: Re: hadoop getProtocolVersion and getBuildVersion error
From: Abhishek Verma <vermaabhishekp@gmail.com>
To: general@hadoop.apache.org
Cc: core-user@hadoop.apache.org, core-dev@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001e680f0f047cea7e0469ded5cb
X-Virus-Checked: Checked by ClamAV on apache.org

--001e680f0f047cea7e0469ded5cb
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi Starry,

I noticed the same problem when I copied hadoop-metrics.properties from my
old hadoop-0.19 conf along with the other files. Make sure you are using the
right version of the conf files.

Hope that helps.

-Abhishek.

On Thu, May 14, 2009 at 7:48 AM, Starry SHI <starrysl@gmail.com> wrote:

> Nobody has encountered with these problems: "Error
> register getProtocolVersion" and "Error
> register getBuildVersion"?
>
> Starry
>
> /* Tomorrow is another day. So is today. */
>
>
>
> On Tue, May 12, 2009 at 13:27, Starry SHI <starrysl@gmail.com> wrote:
> > Hi, all. Today I noticed that my hadoop cluster (r0.20.0+jdk1.6) threw
> > some errors in RPC handling. Below is part of the content of namenode
> > log file:
> >
> > 2009-05-12 10:27:08,200 INFO org.apache.hadoop.ipc.Server: Error
> > register getProtocolVersion
> > java.lang.IllegalArgumentException: Duplicate
> metricsName:getProtocolVersion
> >        at
> org.apache.hadoop.metrics.util.MetricsRegistry.add(MetricsRegistry.java:56)
> >        at
> org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:89)
> >        at
> org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:99)
> >        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:523)
> >        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> >        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> >        at java.security.AccessController.doPrivileged(Native Method)
> >        at javax.security.auth.Subject.doAs(Subject.java:396)
> >        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> > 2009-05-12 10:27:08,200 INFO org.apache.hadoop.ipc.Server: Error
> > register getProtocolVersion
> > java.lang.IllegalArgumentException: Duplicate
> metricsName:getProtocolVersion
> >        at
> org.apache.hadoop.metrics.util.MetricsRegistry.add(MetricsRegistry.java:56)
> >        at
> org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:89)
> >        at
> org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:99)
> >        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:523)
> >        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> >        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> >        at java.security.AccessController.doPrivileged(Native Method)
> >        at javax.security.auth.Subject.doAs(Subject.java:396)
> >        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> > 2009-05-12 10:27:08,200 INFO org.apache.hadoop.ipc.Server: Error
> > register getProtocolVersion
> > java.lang.IllegalArgumentException: Duplicate
> metricsName:getProtocolVersion
> >        at
> org.apache.hadoop.metrics.util.MetricsRegistry.add(MetricsRegistry.java:56)
> >        at
> org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:89)
> >        at
> org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:99)
> >        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:523)
> >        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> >        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> >        at java.security.AccessController.doPrivileged(Native Method)
> >        at javax.security.auth.Subject.doAs(Subject.java:396)
> >        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> > 2009-05-12 10:27:08,276 INFO org.apache.hadoop.ipc.Server: Error
> > register getBuildVersion
> > java.lang.IllegalArgumentException: Duplicate metricsName:getBuildVersion
> >        at
> org.apache.hadoop.metrics.util.MetricsRegistry.add(MetricsRegistry.java:56)
> >        at
> org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:89)
> >        at
> org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:99)
> >        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:523)
> >        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> >        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> >        at java.security.AccessController.doPrivileged(Native Method)
> >        at javax.security.auth.Subject.doAs(Subject.java:396)
> >        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> >
> > I noticed that the similar error appears on the log in every datanode.
> > Can anybody tell me how to fix this?
> >
> > I have patched this:
> > https://issues.apache.org/jira/browse/HADOOP-5139, but the error still
> > exist. I really don't know what to do and am expecting for your help!
> >
> > Best regards,
> > Starry
> >
> > /* Tomorrow is another day. So is today. */
> >
>

--001e680f0f047cea7e0469ded5cb--

From general-return-224-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu May 14 14:39:55 2009
Return-Path: <general-return-224-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68971 invoked from network); 14 May 2009 14:39:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 May 2009 14:39:55 -0000
Received: (qmail 73766 invoked by uid 500); 14 May 2009 14:39:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73715 invoked by uid 500); 14 May 2009 14:39:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73705 invoked by uid 99); 14 May 2009 14:39:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 May 2009 14:39:55 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jason.hadoop@gmail.com designates 209.85.216.103 as permitted sender)
Received: from [209.85.216.103] (HELO mail-px0-f103.google.com) (209.85.216.103)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 May 2009 14:39:44 +0000
Received: by pxi1 with SMTP id 1so677325pxi.5
        for <general@hadoop.apache.org>; Thu, 14 May 2009 07:39:24 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=ukq8KsscwiC25hoqk4aBhg8dcpTZQhuvTTYPBGtUN+c=;
        b=h8ubRQwdhIs3LbsGczY+zfqS9Q5XHjHt1iLwpRjEUcvsUp/NL5IucspsUSaKs6TLMz
         upNSePhoMEbaYHKdKxVBFEuf8k7J7k47Wl4xW9cDI/xQBsyyxXjdgIl2DIvtN2Cl0n7y
         alBIEQE7az6l1o0TBhT87Nwdc+g6M5jQoxsP0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=KNYt/6dqjivIg+RTNeC/66AO2Iu5lMMg3G6cg04e54zf9iVCVAt6DtnuiL2nQNNEQU
         JxxiIgcqk42M2kaVVgMlp1OzLrMjKWSQXL+CnElUItNA8Ftrt+aBsFzTV+fieMDRahAD
         cCYV20js2Ma+4V2pr78t4lRmUrdZQwVaT96qQ=
MIME-Version: 1.0
Received: by 10.115.22.1 with SMTP id z1mr2215099wai.202.1242311964539; Thu, 
	14 May 2009 07:39:24 -0700 (PDT)
In-Reply-To: <3c682ecd0905140554q1b8c5897j703d31a3725198e7@mail.gmail.com>
References: <a5accba30905112227w5b222521k68465f3a027a4dda@mail.gmail.com>
	 <a5accba30905140548r5bcb5f96hcbd13a07ed487562@mail.gmail.com>
	 <3c682ecd0905140554q1b8c5897j703d31a3725198e7@mail.gmail.com>
Date: Thu, 14 May 2009 07:39:24 -0700
Message-ID: <314098690905140739y7004923fheb9cf53fd219b651@mail.gmail.com>
Subject: Re: hadoop getProtocolVersion and getBuildVersion error
From: jason hadoop <jason.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e649c0a658bd910469e04b03
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e649c0a658bd910469e04b03
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Most likely someone has compiled in your production hadoop tree.
The default is to include the build subdirectory in the classpath ahead of
the jars, and the compile process changes the version number.

I haven't looked at 0.20 but up through 0.19, src/saveVersion.sh handles
this.

On Thu, May 14, 2009 at 5:54 AM, Abhishek Verma <vermaabhishekp@gmail.com>wrote:

> Hi Starry,
>
> I noticed the same problem when I copied hadoop-metrics.properties from my
> old hadoop-0.19 conf along with the other files. Make sure you are using
> the
> right version of the conf files.
>
> Hope that helps.
>
> -Abhishek.
>
> On Thu, May 14, 2009 at 7:48 AM, Starry SHI <starrysl@gmail.com> wrote:
>
> > Nobody has encountered with these problems: "Error
> > register getProtocolVersion" and "Error
> > register getBuildVersion"?
> >
> > Starry
> >
> > /* Tomorrow is another day. So is today. */
> >
> >
> >
> > On Tue, May 12, 2009 at 13:27, Starry SHI <starrysl@gmail.com> wrote:
> > > Hi, all. Today I noticed that my hadoop cluster (r0.20.0+jdk1.6) threw
> > > some errors in RPC handling. Below is part of the content of namenode
> > > log file:
> > >
> > > 2009-05-12 10:27:08,200 INFO org.apache.hadoop.ipc.Server: Error
> > > register getProtocolVersion
> > > java.lang.IllegalArgumentException: Duplicate
> > metricsName:getProtocolVersion
> > >        at
> >
> org.apache.hadoop.metrics.util.MetricsRegistry.add(MetricsRegistry.java:56)
> > >        at
> >
> org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:89)
> > >        at
> >
> org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:99)
> > >        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:523)
> > >        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> > >        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> > >        at java.security.AccessController.doPrivileged(Native Method)
> > >        at javax.security.auth.Subject.doAs(Subject.java:396)
> > >        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> > > 2009-05-12 10:27:08,200 INFO org.apache.hadoop.ipc.Server: Error
> > > register getProtocolVersion
> > > java.lang.IllegalArgumentException: Duplicate
> > metricsName:getProtocolVersion
> > >        at
> >
> org.apache.hadoop.metrics.util.MetricsRegistry.add(MetricsRegistry.java:56)
> > >        at
> >
> org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:89)
> > >        at
> >
> org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:99)
> > >        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:523)
> > >        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> > >        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> > >        at java.security.AccessController.doPrivileged(Native Method)
> > >        at javax.security.auth.Subject.doAs(Subject.java:396)
> > >        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> > > 2009-05-12 10:27:08,200 INFO org.apache.hadoop.ipc.Server: Error
> > > register getProtocolVersion
> > > java.lang.IllegalArgumentException: Duplicate
> > metricsName:getProtocolVersion
> > >        at
> >
> org.apache.hadoop.metrics.util.MetricsRegistry.add(MetricsRegistry.java:56)
> > >        at
> >
> org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:89)
> > >        at
> >
> org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:99)
> > >        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:523)
> > >        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> > >        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> > >        at java.security.AccessController.doPrivileged(Native Method)
> > >        at javax.security.auth.Subject.doAs(Subject.java:396)
> > >        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> > > 2009-05-12 10:27:08,276 INFO org.apache.hadoop.ipc.Server: Error
> > > register getBuildVersion
> > > java.lang.IllegalArgumentException: Duplicate
> metricsName:getBuildVersion
> > >        at
> >
> org.apache.hadoop.metrics.util.MetricsRegistry.add(MetricsRegistry.java:56)
> > >        at
> >
> org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:89)
> > >        at
> >
> org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:99)
> > >        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:523)
> > >        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> > >        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> > >        at java.security.AccessController.doPrivileged(Native Method)
> > >        at javax.security.auth.Subject.doAs(Subject.java:396)
> > >        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> > >
> > > I noticed that the similar error appears on the log in every datanode.
> > > Can anybody tell me how to fix this?
> > >
> > > I have patched this:
> > > https://issues.apache.org/jira/browse/HADOOP-5139, but the error still
> > > exist. I really don't know what to do and am expecting for your help!
> > >
> > > Best regards,
> > > Starry
> > >
> > > /* Tomorrow is another day. So is today. */
> > >
> >
>



-- 
Alpha Chapters of my book on Hadoop are available
http://www.apress.com/book/view/9781430219422
www.prohadoopbook.com a community for Hadoop Professionals

--0016e649c0a658bd910469e04b03--

From general-return-225-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu May 14 17:00:28 2009
Return-Path: <general-return-225-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 40354 invoked from network); 14 May 2009 17:00:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 May 2009 17:00:27 -0000
Received: (qmail 51056 invoked by uid 500); 14 May 2009 17:00:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50989 invoked by uid 500); 14 May 2009 17:00:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50979 invoked by uid 99); 14 May 2009 17:00:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 May 2009 17:00:26 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of starrysl@gmail.com designates 209.85.142.186 as permitted sender)
Received: from [209.85.142.186] (HELO ti-out-0910.google.com) (209.85.142.186)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 May 2009 17:00:14 +0000
Received: by ti-out-0910.google.com with SMTP id 28so151817tif.9
        for <general@hadoop.apache.org>; Thu, 14 May 2009 09:59:52 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=K9jKMQrxQX+1eT/08ECAniit5AbZMOKWl4K4QFSR5ho=;
        b=MB2kC9nyEkfmu5HZ5e/1dYAL3RyYKCqugS8W4VKcEWMlBIDUV4O+CgnAw1ZqmJShAw
         y12LU3fC50XhMD/p4N9eFjvQobVn7cn8IApwb1DdWA2YRz263S6bwVq9SQtZR2ThltL8
         Q+imtC5aDipzED6rIMt89w/AvQuLg9REDPO/Q=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=MTdBhv3Tqsc6hf3W0xUrSLMHabPdc9sHC+VtYG3V5znQrzDLAl85PToUmhU6364GOJ
         u0cE9icPPDocn2IgK882PIa91rgnokhZDFXuHIKc1DR9bH7v6l43emgoh9cMD1LYLurp
         BpNV/tbq50ll6SVOXBVdwtiWeZZPJ95MKAlRA=
MIME-Version: 1.0
Received: by 10.110.53.19 with SMTP id b19mr170693tia.34.1242320392361; Thu, 
	14 May 2009 09:59:52 -0700 (PDT)
In-Reply-To: <314098690905140739y7004923fheb9cf53fd219b651@mail.gmail.com>
References: <a5accba30905112227w5b222521k68465f3a027a4dda@mail.gmail.com>
	 <a5accba30905140548r5bcb5f96hcbd13a07ed487562@mail.gmail.com>
	 <3c682ecd0905140554q1b8c5897j703d31a3725198e7@mail.gmail.com>
	 <314098690905140739y7004923fheb9cf53fd219b651@mail.gmail.com>
Date: Fri, 15 May 2009 00:59:52 +0800
Message-ID: <a5accba30905140959k324a2555p2cd8bcdbe5d37861@mail.gmail.com>
Subject: Re: hadoop getProtocolVersion and getBuildVersion error
From: Starry SHI <starrysl@gmail.com>
To: general@hadoop.apache.org, jason.hadoop@gmail.com
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hi, Jason. Your advice seems right, because there are several users in
my hadoop cluster. It is possible for some user to re-compile some
files in hadoop. Since I am new to hadoop, could you please tell me
how to restore the version so as to get rid of the exceptions?
Further, can you share with me how to avoid users to modify the hadoop
base files?

Eagering to hear your advice soon!!!

Thank you again for your kind help!

Best regards,
Starry

On 2009-05-14, jason hadoop <jason.hadoop@gmail.com> wrote:
> Most likely someone has compiled in your production hadoop tree.
> The default is to include the build subdirectory in the classpath ahead of
> the jars, and the compile process changes the version number.
>
> I haven't looked at 0.20 but up through 0.19, src/saveVersion.sh handles
> this.
>
> On Thu, May 14, 2009 at 5:54 AM, Abhishek Verma <vermaabhishekp@gmail.com>wrote:
>
> > Hi Starry,
> >
> > I noticed the same problem when I copied hadoop-metrics.properties from my
> > old hadoop-0.19 conf along with the other files. Make sure you are using
> > the
> > right version of the conf files.
> >
> > Hope that helps.
> >
> > -Abhishek.
> >
> > On Thu, May 14, 2009 at 7:48 AM, Starry SHI <starrysl@gmail.com> wrote:
> >
> > > Nobody has encountered with these problems: "Error
> > > register getProtocolVersion" and "Error
> > > register getBuildVersion"?
> > >
> > > Starry
> > >
> > > /* Tomorrow is another day. So is today. */
> > >
> > >
> > >
> > > On Tue, May 12, 2009 at 13:27, Starry SHI <starrysl@gmail.com> wrote:
> > > > Hi, all. Today I noticed that my hadoop cluster (r0.20.0+jdk1.6) threw
> > > > some errors in RPC handling. Below is part of the content of namenode
> > > > log file:
> > > >
> > > > 2009-05-12 10:27:08,200 INFO org.apache.hadoop.ipc.Server: Error
> > > > register getProtocolVersion
> > > > java.lang.IllegalArgumentException: Duplicate
> > > metricsName:getProtocolVersion
> > > >        at
> > >
> > org.apache.hadoop.metrics.util.MetricsRegistry.add(MetricsRegistry.java:56)
> > > >        at
> > >
> > org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:89)
> > > >        at
> > >
> > org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:99)
> > > >        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:523)
> > > >        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> > > >        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> > > >        at java.security.AccessController.doPrivileged(Native Method)
> > > >        at javax.security.auth.Subject.doAs(Subject.java:396)
> > > >        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> > > > 2009-05-12 10:27:08,200 INFO org.apache.hadoop.ipc.Server: Error
> > > > register getProtocolVersion
> > > > java.lang.IllegalArgumentException: Duplicate
> > > metricsName:getProtocolVersion
> > > >        at
> > >
> > org.apache.hadoop.metrics.util.MetricsRegistry.add(MetricsRegistry.java:56)
> > > >        at
> > >
> > org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:89)
> > > >        at
> > >
> > org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:99)
> > > >        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:523)
> > > >        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> > > >        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> > > >        at java.security.AccessController.doPrivileged(Native Method)
> > > >        at javax.security.auth.Subject.doAs(Subject.java:396)
> > > >        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> > > > 2009-05-12 10:27:08,200 INFO org.apache.hadoop.ipc.Server: Error
> > > > register getProtocolVersion
> > > > java.lang.IllegalArgumentException: Duplicate
> > > metricsName:getProtocolVersion
> > > >        at
> > >
> > org.apache.hadoop.metrics.util.MetricsRegistry.add(MetricsRegistry.java:56)
> > > >        at
> > >
> > org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:89)
> > > >        at
> > >
> > org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:99)
> > > >        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:523)
> > > >        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> > > >        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> > > >        at java.security.AccessController.doPrivileged(Native Method)
> > > >        at javax.security.auth.Subject.doAs(Subject.java:396)
> > > >        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> > > > 2009-05-12 10:27:08,276 INFO org.apache.hadoop.ipc.Server: Error
> > > > register getBuildVersion
> > > > java.lang.IllegalArgumentException: Duplicate
> > metricsName:getBuildVersion
> > > >        at
> > >
> > org.apache.hadoop.metrics.util.MetricsRegistry.add(MetricsRegistry.java:56)
> > > >        at
> > >
> > org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:89)
> > > >        at
> > >
> > org.apache.hadoop.metrics.util.MetricsTimeVaryingRate.<init>(MetricsTimeVaryingRate.java:99)
> > > >        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:523)
> > > >        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> > > >        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> > > >        at java.security.AccessController.doPrivileged(Native Method)
> > > >        at javax.security.auth.Subject.doAs(Subject.java:396)
> > > >        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> > > >
> > > > I noticed that the similar error appears on the log in every datanode.
> > > > Can anybody tell me how to fix this?
> > > >
> > > > I have patched this:
> > > > https://issues.apache.org/jira/browse/HADOOP-5139, but the error still
> > > > exist. I really don't know what to do and am expecting for your help!
> > > >
> > > > Best regards,
> > > > Starry
> > > >
> > > > /* Tomorrow is another day. So is today. */
> > > >
> > >
> >
>
>
>
> --
> Alpha Chapters of my book on Hadoop are available
> http://www.apress.com/book/view/9781430219422
> www.prohadoopbook.com a community for Hadoop Professionals
>


-- 
Starry

/* Tomorrow is another day. So is today. */

From general-return-226-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun May 17 08:46:03 2009
Return-Path: <general-return-226-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 64138 invoked from network); 17 May 2009 08:46:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 May 2009 08:46:03 -0000
Received: (qmail 71752 invoked by uid 500); 17 May 2009 08:46:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71691 invoked by uid 500); 17 May 2009 08:46:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71681 invoked by uid 99); 17 May 2009 08:46:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 17 May 2009 08:46:02 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [86.36.128.10] (HELO idc-smtp2-meeza.meeza.com.qa) (86.36.128.10)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 17 May 2009 08:45:52 +0000
X-AuditID: 5624800a-0000131c00000580-7a-4a0fcea95ead 
From: Scott Mackenzie <scott.mackenzie@meeza.com.qa>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Sun, 17 May 2009 11:35:57 +0300
Subject: Question - Remote Logging to Hadoop
Thread-Topic: Question - Remote Logging to Hadoop
Thread-Index: AcnWy8g6Ezdlwg7GSC2WSUzUUie4jA==
Message-ID: <288C701BB8D5334097F77987DD1397B70D6071A873@IDC-EXMB1-MEEZA.meeza.local>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_288C701BB8D5334097F77987DD1397B70D6071A873IDCEXMB1MEEZA_"
MIME-Version: 1.0
X-Brightmail-Tracker: AAAAAA==
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_288C701BB8D5334097F77987DD1397B70D6071A873IDCEXMB1MEEZA_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

All,

Does anyone have a working method to capture system and application logging=
 for remote servers directly into HDFS in real-time?

We have tested a few methods but all seem reliant on a batch process run ev=
ery 5 to 10 minutes.

Any ideas would be appreciated?



Warm Regards,


Scott

________________________________
Disclaimer: This email and any attachments thereto are confidential and it =
may contain information protected by intellectual Property Laws or otherwis=
e legally privileged. It is intended solely for the use of the addressee(s)=
. If you are not the intended recipient, please notify the sender immediate=
ly and delete this email and any attachments from your system and destroy a=
ny printout thereof. Any unauthorized use or disclosure of this email and i=
ts attachments is strictly prohibited and may be unlawful. MEEZA QSTP-LLC (=
MEEZA) does not certify that this email or any attachments are free of viru=
ses or defects and recommends that you undertake your own virus checking pr=
ocedures before opening this e-mail or any attachments. MEEZA does not acce=
pt any liability for any loss or damage resulting from the use of this e-ma=
il or any attachments. The contents of this e-mail and any attachments do n=
ot necessarily represent the views or polices of MEEZA. As the integrity of=
 e-mails across the internet cannot be guaranteed, MEEZA does not accept an=
y liability for any e-mails and attachments which may violate the relevant =
applicable laws.


--_000_288C701BB8D5334097F77987DD1397B70D6071A873IDCEXMB1MEEZA_--

From general-return-227-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 18 02:34:42 2009
Return-Path: <general-return-227-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70324 invoked from network); 18 May 2009 02:34:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 May 2009 02:34:42 -0000
Received: (qmail 27360 invoked by uid 500); 18 May 2009 02:34:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27267 invoked by uid 500); 18 May 2009 02:34:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27257 invoked by uid 99); 18 May 2009 02:34:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 May 2009 02:34:41 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of leitao.guo@gmail.com designates 209.85.221.178 as permitted sender)
Received: from [209.85.221.178] (HELO mail-qy0-f178.google.com) (209.85.221.178)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 May 2009 02:34:33 +0000
Received: by qyk8 with SMTP id 8so7633772qyk.5
        for <general@hadoop.apache.org>; Sun, 17 May 2009 19:34:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=8xyZWMsWXdzw/jQ614D4nNAFrsqB/wDXq1nBZBTWPm4=;
        b=XoW7XRuEjjjz2TDywLqUV/xNtod6tn4YlzqudUDqKyq6TeKHB9aks4OnydTUYnZmx/
         jHL/QboCY7xJqRPyAbR5I08sONoDh22CVF2/ucvVXGIy1z17zPJaK/MIr0XWEFWFE+8b
         Q/CUpoq32vw3A32JqvWNgqtqsEDjNyAm60zTM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=KgrXaMHFs0yg5aF981mRkeFk5M8coTFFrz6NEH3qMQcOFjRPbCmHFyNtvyjF3EgJY/
         7fZ+K6s5zk+PB7K3lCHfwuv6lsbqJRGW/n/U5sttvqMXjFYkrfioraNHfElU3i0ymgsz
         aULY/TZ71FSjav2KNk1VKlPC/vQKeK3Idln3M=
MIME-Version: 1.0
Received: by 10.220.95.194 with SMTP id e2mr6166774vcn.60.1242614052658; Sun, 
	17 May 2009 19:34:12 -0700 (PDT)
In-Reply-To: <288C701BB8D5334097F77987DD1397B70D6071A873@IDC-EXMB1-MEEZA.meeza.local>
References: <288C701BB8D5334097F77987DD1397B70D6071A873@IDC-EXMB1-MEEZA.meeza.local>
Date: Mon, 18 May 2009 10:34:11 +0800
Message-ID: <d0d818960905171934h38f6b570m4921a32fd471a77c@mail.gmail.com>
Subject: Re: Question - Remote Logging to Hadoop
From: Guo Leitao <leitao.guo@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e646477e33b2b3046a26a15d
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e646477e33b2b3046a26a15d
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

have you ever tried X-trace?
http://research.yahoo.com/files/andy_konwinski_x-tracing_hadoop.pdf

2009/5/17 Scott Mackenzie <scott.mackenzie@meeza.com.qa>

> All,
>
> Does anyone have a working method to capture system and application logging
> for remote servers directly into HDFS in real-time?
>
> We have tested a few methods but all seem reliant on a batch process run
> every 5 to 10 minutes.
>
> Any ideas would be appreciated?
>
>
>
> Warm Regards,
>
>
> Scott
>
> ________________________________
> Disclaimer: This email and any attachments thereto are confidential and it
> may contain information protected by intellectual Property Laws or otherwise
> legally privileged. It is intended solely for the use of the addressee(s).
> If you are not the intended recipient, please notify the sender immediately
> and delete this email and any attachments from your system and destroy any
> printout thereof. Any unauthorized use or disclosure of this email and its
> attachments is strictly prohibited and may be unlawful. MEEZA QSTP-LLC
> (MEEZA) does not certify that this email or any attachments are free of
> viruses or defects and recommends that you undertake your own virus checking
> procedures before opening this e-mail or any attachments. MEEZA does not
> accept any liability for any loss or damage resulting from the use of this
> e-mail or any attachments. The contents of this e-mail and any attachments
> do not necessarily represent the views or polices of MEEZA. As the integrity
> of e-mails across the internet cannot be guaranteed, MEEZA does not accept
> any liability for any e-mails and attachments which may violate the relevant
> applicable laws.
>
>

--0016e646477e33b2b3046a26a15d--

From general-return-228-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 18 07:22:36 2009
Return-Path: <general-return-228-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53773 invoked from network); 18 May 2009 07:22:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 May 2009 07:22:36 -0000
Received: (qmail 36888 invoked by uid 500); 18 May 2009 07:22:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36811 invoked by uid 500); 18 May 2009 07:22:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36796 invoked by uid 99); 18 May 2009 07:22:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 May 2009 07:22:28 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [66.51.199.98] (HELO mail10.dslextreme.com) (66.51.199.98)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 18 May 2009 07:22:18 +0000
Received: (qmail 27270 invoked from network); 18 May 2009 07:21:56 -0000
Received: from unknown (HELO [192.168.0.100]) (68.183.71.37)
	by mail10.dslextreme.com with (DHE-RSA-AES256-SHA encrypted) SMTP; Mon, 18 May 2009 00:21:56 -0700
Message-ID: <4A110CB1.6000300@cloudera.com>
Date: Mon, 18 May 2009 00:22:25 -0700
From: Amr Awadallah <aaa@cloudera.com>
User-Agent: Thunderbird 2.0.0.21 (Windows/20090302)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Question - Remote Logging to Hadoop
References: <288C701BB8D5334097F77987DD1397B70D6071A873@IDC-EXMB1-MEEZA.meeza.local>
In-Reply-To: <288C701BB8D5334097F77987DD1397B70D6071A873@IDC-EXMB1-MEEZA.meeza.local>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: quoted-printable
X-MagicMail-UUID: 91943b18-437c-11de-9e8c-00188bf6df8c
X-Virus-Checked: Checked by ClamAV on apache.org

Scribe from Facebook:

http://developers.facebook.com/scribe/

-- amr
> All,
>
> Does anyone have a working method to capture system and application log=
ging for remote servers directly into HDFS in real-time?
>
> We have tested a few methods but all seem reliant on a batch process ru=
n every 5 to 10 minutes.
>
> Any ideas would be appreciated?
>
>
>
> Warm Regards,
>
>
> Scott
>
> ________________________________
> Disclaimer: This email and any attachments thereto are confidential and=
 it may contain information protected by intellectual Property Laws or ot=
herwise legally privileged. It is intended solely for the use of the addr=
essee(s). If you are not the intended recipient, please notify the sender=
 immediately and delete this email and any attachments from your system a=
nd destroy any printout thereof. Any unauthorized use or disclosure of th=
is email and its attachments is strictly prohibited and may be unlawful. =
MEEZA QSTP-LLC (MEEZA) does not certify that this email or any attachment=
s are free of viruses or defects and recommends that you undertake your o=
wn virus checking procedures before opening this e-mail or any attachment=
s. MEEZA does not accept any liability for any loss or damage resulting f=
rom the use of this e-mail or any attachments. The contents of this e-mai=
l and any attachments do not necessarily represent the views or polices o=
f MEEZA. As the integrity of e-mails across the internet cannot be guaran=
teed, MEEZA does not accept any liability for any e-mails and attachments=
 which may violate the relevant applicable laws.
>
>
>  =20



From general-return-229-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 18 07:40:15 2009
Return-Path: <general-return-229-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58073 invoked from network); 18 May 2009 07:40:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 May 2009 07:40:15 -0000
Received: (qmail 56752 invoked by uid 500); 18 May 2009 07:40:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56683 invoked by uid 500); 18 May 2009 07:40:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56673 invoked by uid 99); 18 May 2009 07:40:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 May 2009 07:40:14 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 May 2009 07:40:03 +0000
Received: from [192.168.1.64] (snvvpn1-10-73-153-c10.hq.corp.yahoo.com [10.73.153.10])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n4I7dTga036592
	for <general@hadoop.apache.org>; Mon, 18 May 2009 00:39:29 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=X5eEGIsBVRfYPIhgOlq3F0LdR9mN7iGLux/zv5MYznPRTNRu9oTWhR12h5jqsXo6
Message-Id: <44B831F7-F720-4B81-BF96-E7E02FBB3B6B@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <288C701BB8D5334097F77987DD1397B70D6071A873@IDC-EXMB1-MEEZA.meeza.local>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: Re: Question - Remote Logging to Hadoop
Date: Mon, 18 May 2009 00:39:29 -0700
References: <288C701BB8D5334097F77987DD1397B70D6071A873@IDC-EXMB1-MEEZA.meeza.local>
X-Mailer: Apple Mail (2.930.3)
X-Virus-Checked: Checked by ClamAV on apache.org


On May 17, 2009, at 1:35 AM, Scott Mackenzie wrote:

> All,
>
> Does anyone have a working method to capture system and application  
> logging for remote servers directly into HDFS in real-time?
>
> We have tested a few methods but all seem reliant on a batch process  
> run every 5 to 10 minutes.
>
> Any ideas would be appreciated?
>
>

Not exactly 'real-time', but you might want to take a look at Chukwa (http://svn.apache.org/viewvc/hadoop/chukwa/ 
).

Unfortunately I can't seem to find documentation for it, I'll try and  
push the chukwa-devs.

Arun

>
> Warm Regards,
>
>
> Scott
>
> ________________________________
> Disclaimer: This email and any attachments thereto are confidential  
> and it may contain information protected by intellectual Property  
> Laws or otherwise legally privileged. It is intended solely for the  
> use of the addressee(s). If you are not the intended recipient,  
> please notify the sender immediately and delete this email and any  
> attachments from your system and destroy any printout thereof. Any  
> unauthorized use or disclosure of this email and its attachments is  
> strictly prohibited and may be unlawful. MEEZA QSTP-LLC (MEEZA) does  
> not certify that this email or any attachments are free of viruses  
> or defects and recommends that you undertake your own virus checking  
> procedures before opening this e-mail or any attachments. MEEZA does  
> not accept any liability for any loss or damage resulting from the  
> use of this e-mail or any attachments. The contents of this e-mail  
> and any attachments do not necessarily represent the views or  
> polices of MEEZA. As the integrity of e-mails across the internet  
> cannot be guaranteed, MEEZA does not accept any liability for any e- 
> mails and attachments which may violate the relevant applicable laws.
>


From general-return-230-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 18 07:55:59 2009
Return-Path: <general-return-230-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 61761 invoked from network); 18 May 2009 07:55:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 May 2009 07:55:59 -0000
Received: (qmail 70083 invoked by uid 500); 18 May 2009 07:55:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69989 invoked by uid 500); 18 May 2009 07:55:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69978 invoked by uid 99); 18 May 2009 07:55:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 May 2009 07:55:58 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 74.125.46.29 as permitted sender)
Received: from [74.125.46.29] (HELO yw-out-2324.google.com) (74.125.46.29)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 May 2009 07:55:48 +0000
Received: by yw-out-2324.google.com with SMTP id 9so1778611ywe.29
        for <general@hadoop.apache.org>; Mon, 18 May 2009 00:55:27 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=nDoGUfq4dH0Ezt5WODYrZqk1sI0tmg+rYFiFgi9eOjQ=;
        b=SBVZjQEkaw2gsxSXcf6ClDfwapSiN8s8193dwkF9KY1FYqPb4fVNKy5QG4UUUCWVmo
         rND+3Nt7nMqDp04Hb5WXnOOSlUxqclZiUX27ILoU82YboPffSHc3yXpNk29YHe/QH+sn
         DTK7513KrFlJbZrNMEZ2fKvEYK45oIKHyN4lk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=as6fPrjMUpBgqsr1K8nEeLJT/EKWBIsv/yrMhpU5C3lIzAp2ZMxdTJ3MK68paMoiS5
         T5DKmFJt6F53Y8B/XA5ZJpdOzIavvIWoRgPR5wjxwEN8g5MyrlzvtklcFmjXpf9aAO2U
         b4bkUUZUPlS4RO1MjDXNu9DO2mCY0N4UqFAiU=
MIME-Version: 1.0
Received: by 10.151.98.17 with SMTP id a17mr11802583ybm.262.1242633327409; 
	Mon, 18 May 2009 00:55:27 -0700 (PDT)
In-Reply-To: <4A110CB1.6000300@cloudera.com>
References: <288C701BB8D5334097F77987DD1397B70D6071A873@IDC-EXMB1-MEEZA.meeza.local>
	 <4A110CB1.6000300@cloudera.com>
Date: Mon, 18 May 2009 00:55:27 -0700
Message-ID: <4aa34eb70905180055x3f859a29lee171e5531133549@mail.gmail.com>
Subject: Re: Question - Remote Logging to Hadoop
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001517510b0e10fa22046a2b1e3e
X-Virus-Checked: Checked by ClamAV on apache.org

--001517510b0e10fa22046a2b1e3e
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

We have deployed a pilot for making scribe log directly into a hdfs 0.19
cluster. We will be pushing these changes to scribe to the open-source
location this week. Also, there are a few changes to HDFS that were required
to make this work smoothly, e.g. HADOOP-2757. These changes are being
debated to figure out what will be pushed into hdfs trunk.

 Also, here are some of the configuration settings:

        <property>
                <name>ipc.client.idlethreshold</name>
                <value>10000</value>
                <description>Defines the threshold number of connections
after which
               connections will be inspected for idleness.
               </description>
        </property>
        <property>
          <name>ipc.client.connection.maxidletime</name>
          <value>10000</value>
          <description>The maximum time in msec after which a client will
bring down the
                       connection to the server.
          </description>
        </property>
        <property>
          <name>ipc.client.connect.max.retries</name>
          <value>2</value>
          <description>Indicates the number of retries a client will make to
establish
                       a server connection.
          </description>
        </property>
        <property>
          <name>ipc.server.listen.queue.size</name>
          <value>128</value>
          <description>Indicates the length of the listen queue for servers
accepting
                       client connections.
          </description>
        </property>
      <property>
          <name>ipc.server.tcpnodelay</name>
          <value>true</value>
          <description>Turn on/off Nagle's algorithm for the TCP socket
connection on
          the server. Setting to true disables the algorithm and may
decrease latency
          with a cost of more/smaller packets.
          </description>
        </property>
        <property>
          <name>ipc.client.tcpnodelay</name>
          <value>true</value>
          <description>Turn on/off Nagle's algorithm for the TCP socket
connection on
          the client. Setting to true disables the algorithm and may
decrease latency
          with a cost of more/smaller packets.
          </description>
        </property>
        <property>
          <name>ipc.ping.interval</name>
          <value>5000</value>
          <description>The Client sends a ping message to server every
period. This is helpful
          to detect socket connections that were idle and have been
terminated by a failed server.
          </description>
        </property>
        <property>
          <name>ipc.client.connect.maxwaittime</name>
          <value>5000</value>
          <description>The Client waits for this much time for a socket
connect call to be establised
          with the server.
          </description>
        </property>
        <property>
          <name>dfs.datanode.socket.write.timeout</name>
          <value>20000</value>
          <description>The dfs Client waits for this much time for a socket
write call to the datanode.
          </description>
        </property>
        <property>
          <name>dfs.leaserenewal.timeout</name>
          <value>10000</value>
          <description>The dfs writer waits for this much time after the
last successful lease renewal
          before aborting the write to the file.
          </description>
        </property>


On Mon, May 18, 2009 at 12:22 AM, Amr Awadallah <aaa@cloudera.com> wrote:

> Scribe from Facebook:
>
> http://developers.facebook.com/scribe/
>
> -- amr
>
>> All,
>>
>> Does anyone have a working method to capture system and application
>> logging for remote servers directly into HDFS in real-time?
>>
>> We have tested a few methods but all seem reliant on a batch process run
>> every 5 to 10 minutes.
>>
>> Any ideas would be appreciated?
>>
>>
>>
>> Warm Regards,
>>
>>
>> Scott
>>
>> ________________________________
>> Disclaimer: This email and any attachments thereto are confidential and it
>> may contain information protected by intellectual Property Laws or otherwise
>> legally privileged. It is intended solely for the use of the addressee(s).
>> If you are not the intended recipient, please notify the sender immediately
>> and delete this email and any attachments from your system and destroy any
>> printout thereof. Any unauthorized use or disclosure of this email and its
>> attachments is strictly prohibited and may be unlawful. MEEZA QSTP-LLC
>> (MEEZA) does not certify that this email or any attachments are free of
>> viruses or defects and recommends that you undertake your own virus checking
>> procedures before opening this e-mail or any attachments. MEEZA does not
>> accept any liability for any loss or damage resulting from the use of this
>> e-mail or any attachments. The contents of this e-mail and any attachments
>> do not necessarily represent the views or polices of MEEZA. As the integrity
>> of e-mails across the internet cannot be guaranteed, MEEZA does not accept
>> any liability for any e-mails and attachments which may violate the relevant
>> applicable laws.
>>
>>
>>
>>
>
>
>

--001517510b0e10fa22046a2b1e3e--

From general-return-231-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 18 17:47:02 2009
Return-Path: <general-return-231-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12335 invoked from network); 18 May 2009 17:47:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 May 2009 17:47:01 -0000
Received: (qmail 1948 invoked by uid 500); 18 May 2009 17:47:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1877 invoked by uid 500); 18 May 2009 17:47:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1867 invoked by uid 99); 18 May 2009 17:47:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 May 2009 17:47:00 +0000
X-ASF-Spam-Status: No, hits=4.2 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 May 2009 17:46:49 +0000
Received: from sp1-ex07cas01.ds.corp.yahoo.com (sp1-ex07cas01.ds.corp.yahoo.com [216.252.116.137])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n4IHj5bu033746
	for <general@hadoop.apache.org>; Mon, 18 May 2009 10:45:09 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:in-reply-to:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=rWhEdKXVxPOvwSpQQih9OIgZkn+ccYSfSEHVSU3dPWl0xtoD55zyiJHg/3MoCPI1
Received: from SP1-EX07VS02.ds.corp.yahoo.com ([216.252.116.135]) by
 sp1-ex07cas01.ds.corp.yahoo.com ([216.252.116.137]) with mapi; Mon, 18 May
 2009 10:45:09 -0700
From: Jerome Boulon <jboulon@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 18 May 2009 10:45:08 -0700
Subject: Re: Question - Remote Logging to Hadoop
Thread-Topic: Question - Remote Logging to Hadoop
Thread-Index: AcnXjCMKQ8FzibUeR7ye/O+W1NrMRwAVD9X3
Message-ID: <C636ECB4.1BDA2%jboulon@yahoo-inc.com>
In-Reply-To: <44B831F7-F720-4B81-BF96-E7E02FBB3B6B@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C636ECB41BDA2jboulonyahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C636ECB41BDA2jboulonyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

The documentation for chukwa has not been released yet but you can access i=
t here:
http://people.apache.org/~eyang/docs/r0.1.2/index.html

/Jerome.


On 5/18/09 12:39 AM, "Arun C Murthy" <acm@yahoo-inc.com> wrote:



On May 17, 2009, at 1:35 AM, Scott Mackenzie wrote:

> All,
>
> Does anyone have a working method to capture system and application
> logging for remote servers directly into HDFS in real-time?
>
> We have tested a few methods but all seem reliant on a batch process
> run every 5 to 10 minutes.
>
> Any ideas would be appreciated?
>
>

Not exactly 'real-time', but you might want to take a look at Chukwa (http:=
//svn.apache.org/viewvc/hadoop/chukwa/
).

Unfortunately I can't seem to find documentation for it, I'll try and
push the chukwa-devs.

Arun

>
> Warm Regards,
>
>
> Scott
>
> ________________________________
> Disclaimer: This email and any attachments thereto are confidential
> and it may contain information protected by intellectual Property
> Laws or otherwise legally privileged. It is intended solely for the
> use of the addressee(s). If you are not the intended recipient,
> please notify the sender immediately and delete this email and any
> attachments from your system and destroy any printout thereof. Any
> unauthorized use or disclosure of this email and its attachments is
> strictly prohibited and may be unlawful. MEEZA QSTP-LLC (MEEZA) does
> not certify that this email or any attachments are free of viruses
> or defects and recommends that you undertake your own virus checking
> procedures before opening this e-mail or any attachments. MEEZA does
> not accept any liability for any loss or damage resulting from the
> use of this e-mail or any attachments. The contents of this e-mail
> and any attachments do not necessarily represent the views or
> polices of MEEZA. As the integrity of e-mails across the internet
> cannot be guaranteed, MEEZA does not accept any liability for any e-
> mails and attachments which may violate the relevant applicable laws.
>



--_000_C636ECB41BDA2jboulonyahooinccom_--

From general-return-232-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 19 05:43:00 2009
Return-Path: <general-return-232-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 51534 invoked from network); 19 May 2009 05:42:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 May 2009 05:42:59 -0000
Received: (qmail 60875 invoked by uid 500); 19 May 2009 05:42:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60814 invoked by uid 500); 19 May 2009 05:42:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60804 invoked by uid 99); 19 May 2009 05:42:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 May 2009 05:42:57 +0000
X-ASF-Spam-Status: No, hits=-1.8 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of qiujie@cn.ibm.com designates 202.81.31.142 as permitted sender)
Received: from [202.81.31.142] (HELO e23smtp09.au.ibm.com) (202.81.31.142)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 May 2009 05:42:47 +0000
Received: from d23relay01.au.ibm.com (d23relay01.au.ibm.com [202.81.31.243])
	by e23smtp09.au.ibm.com (8.13.1/8.13.1) with ESMTP id n4J5E8Fa002105
	for <general@hadoop.apache.org>; Tue, 19 May 2009 01:14:08 -0400
Received: from d23av04.au.ibm.com (d23av04.au.ibm.com [9.190.235.139])
	by d23relay01.au.ibm.com (8.13.8/8.13.8/NCO v9.2) with ESMTP id n4J5gIYw454974
	for <general@hadoop.apache.org>; Tue, 19 May 2009 15:42:24 +1000
Received: from d23av04.au.ibm.com (loopback [127.0.0.1])
	by d23av04.au.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id n4J5gIfH014945
	for <general@hadoop.apache.org>; Tue, 19 May 2009 15:42:18 +1000
Received: from d23m0037.cn.ibm.com (d23m0037.cn.ibm.com [9.181.2.105])
	by d23av04.au.ibm.com (8.12.11.20060308/8.12.11) with ESMTP id n4J5gHQb014935
	for <general@hadoop.apache.org>; Tue, 19 May 2009 15:42:18 +1000
To: general@hadoop.apache.org
MIME-Version: 1.0
Subject: avro-dev-subscribe
X-Mailer: Lotus Notes Release 7.0 HF277 June 21, 2006
Message-ID: <OFB92EF861.EE7F731C-ON482575BB.001F3CA7-482575BB.001F576F@cn.ibm.com>
From: Jie Qiu <qiujie@cn.ibm.com>
Date: Tue, 19 May 2009 13:42:23 +0800
X-MIMETrack: Serialize by Router on D23M0037/23/M/IBM(Release 8.0.1|February 07, 2008) at
 05/19/2009 13:42:24,
	Serialize complete at 05/19/2009 13:42:24
Content-Type: multipart/alternative; boundary="=_alternative 001F576E482575BB_="
X-Virus-Checked: Checked by ClamAV on apache.org

--=_alternative 001F576E482575BB_=
Content-Type: text/plain; charset="US-ASCII"

FYI...

==================================================
True insight comes from within!

Qiu Jie

Autonomic Middleware and Service Delivery
IBM China Research Lab.
Tel: 8610-58748086  Tieline: 905-8086
FAX: 8610-58748330 
Add.: Building 19 Zhongguancun Software Park, 8 Dongbeiwang WestRoad, 
Haidian District, Beijing,P.R.C.100094
Email: qiujie@cn.ibm.com
==================================================
--=_alternative 001F576E482575BB_=--

From general-return-233-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 19 21:49:58 2009
Return-Path: <general-return-233-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 40933 invoked from network); 19 May 2009 21:49:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 May 2009 21:49:57 -0000
Received: (qmail 88010 invoked by uid 500); 19 May 2009 21:50:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87923 invoked by uid 500); 19 May 2009 21:50:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87905 invoked by uid 99); 19 May 2009 21:50:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 May 2009 21:50:07 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of p0941p@gmail.com designates 209.85.217.167 as permitted sender)
Received: from [209.85.217.167] (HELO mail-gx0-f167.google.com) (209.85.217.167)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 May 2009 21:49:59 +0000
Received: by gxk11 with SMTP id 11so146294gxk.5
        for <multiple recipients>; Tue, 19 May 2009 14:49:38 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=vJW97xEkGFm05cXH0ZPM+ckjd25dTyn38osoBCHbk1k=;
        b=QkF5yw8RD7gXuj9durVArPFv4ohqLCqB8JBg5Wf1n7TfTCD4hpwi4G6ivyHGRBbN4A
         Gkd/uAFYTWgN2Bvi45KZLKKoSIOOCVPlHdzsfjIZKx3YzZZalh3WZabOxZNMLU2UzArN
         6hzrdaODtchRWF09WVBD31ebTxyD6r3GABDKk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=JOErOMiPMLRO9FP1ICWf/FBC9XuvBDHOoMl7PBOWHCHCsaf0Qj/LelbQXEotjgr+0d
         Rxr3IQhru0D8msKVayU0+CdzEf2uMQhDh7rcBy+4qk6P7aWOsZrbw85BLPuekZwazvhG
         nakThuHnjLEfNwFppEmU+n9wAfdHGTpC8pELE=
MIME-Version: 1.0
Received: by 10.151.74.12 with SMTP id b12mr69424ybl.68.1242769762077; Tue, 19 
	May 2009 14:49:22 -0700 (PDT)
Date: Tue, 19 May 2009 14:49:22 -0700
Message-ID: <9c39bdeb0905191449x256cb2f1pb6d40caee9459934@mail.gmail.com>
Subject: DFS Access Error
From: George Pang <p0941p@gmail.com>
To: general@hadoop.apache.org, core-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001e680f182834d96f046a4ae27e
X-Virus-Checked: Checked by ClamAV on apache.org

--001e680f182834d96f046a4ae27e
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Dear Users,

When I tried to access the DFS, an error message apears:

Error: java.io.IOException: Unknown protocol to job tracker:
org.apache.hadoop.dfs.ClientProtocol
at
org.apache.hadoop.mapred.jobTracker.getProtocolVersion(JobTracker.java:163)
at sun.reflec.NativeMethodAccessorimpl.invoke0(Native Method)
at
sun.reflec.NativeMethodAccessorimpl.invoke0(NativeMethodAccessorimpl.java:39)
at
sun.reflec.NativeMethodAccessorimpl.invoke0(DelegatingMethodAccessorimpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server:java:890)

Hadoop version: 0.18.3

I'd appreciate your opinion!

George

--001e680f182834d96f046a4ae27e--

From general-return-234-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 19 22:34:57 2009
Return-Path: <general-return-234-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6789 invoked from network); 19 May 2009 22:34:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 May 2009 22:34:57 -0000
Received: (qmail 73681 invoked by uid 500); 19 May 2009 22:35:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73589 invoked by uid 500); 19 May 2009 22:35:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73569 invoked by uid 99); 19 May 2009 22:35:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 May 2009 22:35:07 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of p0941p@gmail.com designates 74.125.44.30 as permitted sender)
Received: from [74.125.44.30] (HELO yx-out-2324.google.com) (74.125.44.30)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 May 2009 22:34:57 +0000
Received: by yx-out-2324.google.com with SMTP id 8so55641yxm.29
        for <multiple recipients>; Tue, 19 May 2009 15:34:36 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=nNKtyHdHmnsmVgvHH1E0pcmUqC4StC5Emmezqo01CrQ=;
        b=MrD52LCwJqdIg9secwU9eltC1NpOwN1YOvcOmKNoBaq+wT/9aTVTUDxykucx7s6qU3
         IaqAHTfDO3N/szmr4zvXeIhS8mACqgMv35N2bCCJ3C99cZnLIwr2O3RJIYWSh8AJjuTw
         j3OwoSyCEKwRXcq4Qhw822wOVUC1A1QXGwxaI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=mKFdNJarlUqMkmpHDsfOUFsj9ExLb898m3gA22TqSRUiZjtaURGib9TSQcTk+D4/Nd
         ljopbTQMYr9jWQku8Mvdq66Xc5+83/6Sk9rKmg7PN1bfCwyeQ0bz3sR/5muAA37MpRSJ
         CVEVUnwccI5eKAxBqfHGc4lz+kdaPcm/CILuA=
MIME-Version: 1.0
Received: by 10.151.134.9 with SMTP id l9mr1152208ybn.240.1242772476757; Tue, 
	19 May 2009 15:34:36 -0700 (PDT)
In-Reply-To: <9c39bdeb0905191449x256cb2f1pb6d40caee9459934@mail.gmail.com>
References: <9c39bdeb0905191449x256cb2f1pb6d40caee9459934@mail.gmail.com>
Date: Tue, 19 May 2009 15:34:36 -0700
Message-ID: <9c39bdeb0905191534q5e9234a0hbe388121357b2f7c@mail.gmail.com>
Subject: Re: DFS Access Error
From: George Pang <p0941p@gmail.com>
To: general@hadoop.apache.org, core-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001e680f10e00384f2046a4b84b5
X-Virus-Checked: Checked by ClamAV on apache.org

--001e680f10e00384f2046a4b84b5
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

I made a mistake on misplacing the port no. for MR master with HDFS master.
 After correcting that, it works.
George

2009/5/19 George Pang <p0941p@gmail.com>

> Dear Users,
>
> When I tried to access the DFS, an error message apears:
>
> Error: java.io.IOException: Unknown protocol to job tracker:
> org.apache.hadoop.dfs.ClientProtocol
> at
> org.apache.hadoop.mapred.jobTracker.getProtocolVersion(JobTracker.java:163)
> at sun.reflec.NativeMethodAccessorimpl.invoke0(Native Method)
> at
> sun.reflec.NativeMethodAccessorimpl.invoke0(NativeMethodAccessorimpl.java:39)
> at
> sun.reflec.NativeMethodAccessorimpl.invoke0(DelegatingMethodAccessorimpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server:java:890)
>
> Hadoop version: 0.18.3
>
> I'd appreciate your opinion!
>
> George
>

--001e680f10e00384f2046a4b84b5--

From general-return-235-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed May 20 16:35:04 2009
Return-Path: <general-return-235-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 92904 invoked from network); 20 May 2009 16:35:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 May 2009 16:35:03 -0000
Received: (qmail 46915 invoked by uid 500); 20 May 2009 16:35:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46319 invoked by uid 500); 20 May 2009 16:35:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45925 invoked by uid 99); 20 May 2009 16:35:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 May 2009 16:35:11 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 May 2009 16:34:58 +0000
Received: from SNV-EXBH01.ds.corp.yahoo.com (snv-exbh01.ds.corp.yahoo.com [207.126.227.249])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n4KGSFq5019208;
	Wed, 20 May 2009 09:28:15 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:x-mimeole:content-class:mime-version:
	content-type:subject:date:message-id:in-reply-to:x-ms-has-attach:
	x-ms-tnef-correlator:thread-topic:thread-index:references:from:to:x-originalarrivaltime;
	b=ZuQTc0R8qLfHxu/uMJRKzU/0Wzz/1Jn/XE2XTyQIsUrzDDMzA3L8GgO9QeLWmv1+
Received: from SNV-EXVS02.ds.corp.yahoo.com ([216.145.51.196]) by SNV-EXBH01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Wed, 20 May 2009 09:28:15 -0700
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C9D967.F9169E3D"
Subject: RE: Hadoop User Group (Bay Area) May 20th
Date: Wed, 20 May 2009 09:28:13 -0700
Message-ID: <DD27C9769EA63D43BE3CF4CCCC12DD970262B33A@SNV-EXVS02.ds.corp.yahoo.com>
In-Reply-To: <AD0C16AAE446924586B68FC0EB421C9BAE155864@SNV-EXVS02.ds.corp.yahoo.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Hadoop User Group (Bay Area) May 20th
Thread-Index: AcnNxepSpOFHRec4RX+Djty21JFqUgGP/ssQAVhVC7A=
References: <AD0C16AAE446924586B68FC0EB421C9BAE155806@SNV-EXVS02.ds.corp.yahoo.com> <AD0C16AAE446924586B68FC0EB421C9BAE155864@SNV-EXVS02.ds.corp.yahoo.com>
From: "Ajay Anand" <aanand@yahoo-inc.com>
To: "core-user@hadoop.apache.org" <'core-user@hadoop.apache.org'>,
        "general@hadoop.apache.org" <'general@hadoop.apache.org'>,
        "zookeeper-user@hadoop.apache.org" <'zookeeper-user@hadoop.apache.org'>,
        "hbase-user@hadoop.apache.org" <'hbase-user@hadoop.apache.org'>,
        <pig-user@hadoop.apache.org>
X-OriginalArrivalTime: 20 May 2009 16:28:15.0303 (UTC) FILETIME=[F9D3B570:01C9D967]
X-Virus-Checked: Checked by ClamAV on apache.org

------_=_NextPart_001_01C9D967.F9169E3D
Content-Type: text/plain;
	charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable

Reminder: the Bay Area Hadoop User Group meeting is today at 6 pm at the
Yahoo! Sunnyvale campus - http://upcoming.yahoo.com/event/2659418/

=20

________________________________

From: Ajay Anand=20
Sent: Wednesday, May 13, 2009 1:12 PM
To: 'core-user@hadoop.apache.org'; 'general@hadoop.apache.org';
'zookeeper-user@hadoop.apache.org'; 'hbase-user@hadoop.apache.org';
'pig-user@hadoop.apache.org'
Subject: Hadoop User Group (Bay Area) May 20th

=20

The next Bay Area Hadoop User Group meeting is scheduled for Wednesday,
May 20th at Yahoo! 700 First Avenue, Sunnyvale, Building E, Class room
10 from 6:00-7:30 pm. Please note that the location has changed from our
usual meeting place in Santa Clara.=20

Registration: http://upcoming.yahoo.com/event/2659418/

=20

Agenda:

- Hadoop Summit 2009 preview (registration open at
http://hadoopsummit09.eventbrite.com/ )
- Hadoop with GPFS - Prasenjit Sarkar, IBM
- SQoop - Cloudera

Look forward to seeing you there!

Ajay

=20

=20


------_=_NextPart_001_01C9D967.F9169E3D--

From general-return-236-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu May 21 06:25:51 2009
Return-Path: <general-return-236-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58174 invoked from network); 21 May 2009 06:25:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 May 2009 06:25:51 -0000
Received: (qmail 99454 invoked by uid 500); 21 May 2009 06:26:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98904 invoked by uid 500); 21 May 2009 06:26:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98729 invoked by uid 99); 21 May 2009 06:26:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 May 2009 06:26:01 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of p0941p@gmail.com designates 74.125.44.30 as permitted sender)
Received: from [74.125.44.30] (HELO yx-out-2324.google.com) (74.125.44.30)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 May 2009 06:25:53 +0000
Received: by yx-out-2324.google.com with SMTP id 8so531979yxm.29
        for <multiple recipients>; Wed, 20 May 2009 23:25:32 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=UkKMITEsylGfYU7OQ0jFzVwp3fpxzXdXzYNXwYMQGSA=;
        b=B+t/WxGctmpiBbGMfgDoTG8EXNmV93UXqrUDc7vlkQuVV063DrNu74k/arTZi/jg94
         cj04OVcKlz9t/ut6po9fsZ7KvViCZxqB6XWdpFt0zSrMe9BghLqoxCWqxLGQ21OKV8QD
         OYZcov8tqkIiHFeUd6Eww3ZNLH82LzLKPNYGo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=SEKV3esOwLVEY/jL0bjvvBBCn0c8vXKxGYr//Bjm6Q4tncO021vSiITl/GFb3h0AtO
         tpmOdf1TBj7W1SiJ1yW1wyo1Msohf2Q7wD6M19w5SIivWd6pN1MtkD9E6A5kxjJwhlQ8
         0+mGFvukVqCjOOFFs8Wg7/Tx/4hn/cEy70AP0=
MIME-Version: 1.0
Received: by 10.150.202.11 with SMTP id z11mr4492571ybf.129.1242887132799; 
	Wed, 20 May 2009 23:25:32 -0700 (PDT)
In-Reply-To: <9c39bdeb0905202312u70dae466m108f2e063cb11ac8@mail.gmail.com>
References: <9c39bdeb0905202312u70dae466m108f2e063cb11ac8@mail.gmail.com>
Date: Wed, 20 May 2009 23:25:32 -0700
Message-ID: <9c39bdeb0905202325x5758d9aew11a9304ee6505761@mail.gmail.com>
Subject: Fwd: Error message for PigPen
From: George Pang <p0941p@gmail.com>
To: core-user@hadoop.apache.org, general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd4d35c0be4d2046a6636ca
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd4d35c0be4d2046a6636ca
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Dear users,

When I try to use pig editor on Eclipse (with PigPen plugin), one error
message appears on the console: "*org.apache.hadoop.dfs.DistributedFileSystem
cannot be cast to org.apache.hadoop.fs.FileSystem*"

Does this have something to do with hadoop version?

Thank you!

George

--000e0cd4d35c0be4d2046a6636ca--

From general-return-237-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu May 21 22:16:11 2009
Return-Path: <general-return-237-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56483 invoked from network); 21 May 2009 22:16:10 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 May 2009 22:16:10 -0000
Received: (qmail 2411 invoked by uid 500); 21 May 2009 22:16:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2364 invoked by uid 500); 21 May 2009 22:16:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 63646 invoked by uid 99); 21 May 2009 18:13:58 -0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saidtherobot@gmail.com designates 209.85.217.167 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        bh=uueuy8MPrXIdsO+fB5ZvDv8ou22+w0vTXoVWSXaG0+c=;
        b=r1Wg9COgbzf9jtzu4cB4/yHBHh2DgmpzdXDMGjYi8jKs73DkvUJvsdsHlX+VviQv4A
         mDu9+z4KGwmxFiJFtURbYu3mFcdPZYzk21PDLIdBfP0wEOMjKANHmJ+etJprHpG7M8YF
         2wEVQDm/pZEsG0crZD6LCqYNLM0Bs3d3yJqOU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        b=R8J9p4aagcS9h2d1mfqROE/V0JbXkeM5cuoIGmZqvf6dDRH2jVNbjtBqYQFkADDtRo
         3xro3vW4wN/d+udOdElmWVtMzbAJZlMCp4y98a0EAvg1Gpkojdi8HuHQMuhThwLwc83d
         Id7fyzmKYPmGdzc9JgDBSOG1fW79oOECg3dic=
MIME-Version: 1.0
Sender: saidtherobot@gmail.com
Date: Thu, 21 May 2009 14:13:29 -0400
X-Google-Sender-Auth: 5d95efc6a784d745
Message-ID: <2986c2f30905211113j28a657ffr96e2061334b10e85@mail.gmail.com>
Subject: interface to HDFS
From: Mike Anderson <mike_a@mit.edu>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001e680f10acde5e02046a7019b4
X-Virus-Checked: Checked by ClamAV on apache.org

--001e680f10acde5e02046a7019b4
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hello, I'm working on a hadoop project where my data is comprised of many
HTML files (websites). One aspect of the project involves traditional
MapReduce analysis on the data set, but I would also like to use hadoop as a
sort of "cache server," i.e, having the ability to retrieve the HTML for a
website that I have already been to.

My question is this: what is the best way to interact with HDFS to make
simple existance queries and retrieve specific files for reading. Ideally I
would like to do this at an application level, (most likely written in
Ruby). So far I have explored the option of using one of the FUSE packages
to mount it in the userspace, but, I ran into quite a bit of difficulty
installing either of the two popular packages. My second option seems to be
Hive, but I haven't been able to find any bindings for Ruby or Python, etc.

Any suggestions or advice would be greatly appreciated!

Cheers,
Mike

--001e680f10acde5e02046a7019b4--

From general-return-238-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri May 22 10:24:30 2009
Return-Path: <general-return-238-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 2076 invoked from network); 22 May 2009 10:24:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 May 2009 10:24:29 -0000
Received: (qmail 83492 invoked by uid 500); 22 May 2009 10:24:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83404 invoked by uid 500); 22 May 2009 10:24:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83394 invoked by uid 99); 22 May 2009 10:24:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 May 2009 10:24:41 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.198.236] (HELO rv-out-0506.google.com) (209.85.198.236)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 May 2009 10:24:30 +0000
Received: by rv-out-0506.google.com with SMTP id g37so531661rvb.29
        for <general@hadoop.apache.org>; Fri, 22 May 2009 03:24:10 -0700 (PDT)
Received: by 10.141.162.19 with SMTP id p19mr1648883rvo.245.1242987849139;
        Fri, 22 May 2009 03:24:09 -0700 (PDT)
Received: from ?192.168.42.100? (75-101-123-136.dsl.static.sonic.net [75.101.123.136])
        by mx.google.com with ESMTPS id k2sm2009158rvb.4.2009.05.22.03.24.07
        (version=SSLv3 cipher=RC4-MD5);
        Fri, 22 May 2009 03:24:08 -0700 (PDT)
Message-ID: <4A167D42.5080209@cloudera.com>
Date: Fri, 22 May 2009 03:24:02 -0700
From: Amr Awadallah <aaa@cloudera.com>
Organization: Cloudera, Inc.
User-Agent: Thunderbird 2.0.0.21 (Windows/20090302)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: interface to HDFS
References: <2986c2f30905211113j28a657ffr96e2061334b10e85@mail.gmail.com>
In-Reply-To: <2986c2f30905211113j28a657ffr96e2061334b10e85@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Mike,

webdav should work for you, see:

https://issues.apache.org/jira/browse/HADOOP-496
http://www.hadoop.iponweb.net/Home/hdfs-over-webdav/webdav-server

That said, note that HDFS is not optimized for handling lots of small 
files, it will store them fine, and diskspace will not be wasted, but 
the NameNode will not scale very well (the default block size is 64MB 
and HTML docs are way smaller than that). See this blog post for hints 
on how to work with many small files:

http://www.cloudera.com/blog/2009/02/02/the-small-files-problem/

Cheers,

-- amr

Mike Anderson wrote:
> Hello, I'm working on a hadoop project where my data is comprised of many
> HTML files (websites). One aspect of the project involves traditional
> MapReduce analysis on the data set, but I would also like to use hadoop as a
> sort of "cache server," i.e, having the ability to retrieve the HTML for a
> website that I have already been to.
>
> My question is this: what is the best way to interact with HDFS to make
> simple existance queries and retrieve specific files for reading. Ideally I
> would like to do this at an application level, (most likely written in
> Ruby). So far I have explored the option of using one of the FUSE packages
> to mount it in the userspace, but, I ran into quite a bit of difficulty
> installing either of the two popular packages. My second option seems to be
> Hive, but I haven't been able to find any bindings for Ruby or Python, etc.
>
> Any suggestions or advice would be greatly appreciated!
>
> Cheers,
> Mike
>
>   

From general-return-239-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed May 27 23:08:06 2009
Return-Path: <general-return-239-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78839 invoked from network); 27 May 2009 23:08:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 May 2009 23:08:05 -0000
Received: (qmail 71260 invoked by uid 500); 27 May 2009 23:08:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70807 invoked by uid 500); 27 May 2009 23:08:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70623 invoked by uid 99); 27 May 2009 23:08:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 May 2009 23:08:13 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.146.183] (HELO wa-out-1112.google.com) (209.85.146.183)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 May 2009 23:08:00 +0000
Received: by wa-out-1112.google.com with SMTP id j32so878385waf.29
        for <multiple recipients>; Wed, 27 May 2009 16:07:39 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.114.158.1 with SMTP id g1mr1025423wae.126.1243465659597; Wed, 
	27 May 2009 16:07:39 -0700 (PDT)
Date: Wed, 27 May 2009 16:07:39 -0700
Message-ID: <69035570905271607u745eafd2lf1826ccafa6a3b18@mail.gmail.com>
Subject: Annoucement: RFP Open - Hadoop Summit East - Oct 2, 2009 - NYC
From: Christophe Bisciglia <christophe@cloudera.com>
To: core-user@hadoop.apache.org, pig-user@hadoop.apache.org, 
	hbase-user <hbase-user@hadoop.apache.org>, hive-user@hadoop.apache.org, 
	zookeeper-user@hadoop.apache.org, general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016363ba868ee4c4d046aece87d
X-Virus-Checked: Checked by ClamAV on apache.org

--0016363ba868ee4c4d046aece87d
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hadoop Fans,

Lately, we've been spending a lot of time on the East Coast, and one thing
is clear: Hadoop is everywhere.

Hadoop usage on the East Coast tends to be slightly different. There are
still web companies with armies of tech gurus, but there are also many
"regular" industries and enterprises using and exploring Hadoop. It's time
to get together and learn a thing or two from one other.

The Hadoop Summit East <http://www.cloudera.com/hadoop-summit-east> will
take place on October 2nd, 2009 and focus on two areas of interest to
enterprise users. We've opened requests for proposals at:
http://hadoop-summit-east-rfp.eventbrite.com/

Development and Administration:
* Core Hadoop: Areas for Development, Major Upcoming Contributions,
Functional Deep Dives
* Administration: Large Cluster Overviews, Performance Tips, Resource
Management, High Availability
* Developer: IDEs, QA Best Practices, Sharing Code / Data / Clusters, Higher
Level Abstractions

Hadoop Applications:
* Lessons from the Web: What can traditional industries learn from companies
with web scale data?
* Industry Case Studies: Finance, Telecom / Utilities, Retail, Biotech, etc.
* Integration with Existing Systems: Databases, BI Tools, Message Buses and
other Infrastructure
* New Ideas / Applications: Big Ideas for Hadoop
* Hosted Solutions: Hadoop and the Cloud

We'll close the RFP on July 31st and announce the schedule soon thereafter.
More details including discounted / early registration and sponsorship
info available at: http://www.cloudera.com/hadoop-summit-east

Cheers,
Christophe and the Cloudera Team

-- 
get hadoop: cloudera.com/hadoop
online training: cloudera.com/hadoop-training
blog: cloudera.com/blog
twitter: twitter.com/cloudera

--0016363ba868ee4c4d046aece87d--

From general-return-1874-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 01 02:05:45 2010
Return-Path: <general-return-1874-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98289 invoked from network); 1 Aug 2010 02:05:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Aug 2010 02:05:44 -0000
Received: (qmail 67100 invoked by uid 500); 1 Aug 2010 02:05:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66964 invoked by uid 500); 1 Aug 2010 02:05:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66956 invoked by uid 99); 1 Aug 2010 02:05:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Aug 2010 02:05:42 +0000
X-ASF-Spam-Status: No, hits=0.6 required=10.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.149.75] (HELO na3sys009aog105.obsmtp.com) (74.125.149.75)
    by apache.org (qpsmtpd/0.29) with SMTP; Sun, 01 Aug 2010 02:05:35 +0000
Received: from source ([209.85.161.44]) by na3sys009aob105.postini.com ([74.125.148.12]) with SMTP
	ID DSNKTFTWVxDd40V/VKnDpOBWyNOFb6winrxb@postini.com; Sat, 31 Jul 2010 19:05:14 PDT
Received: by fxm16 with SMTP id 16so448633fxm.17
        for <general@hadoop.apache.org>; Sat, 31 Jul 2010 19:05:11 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.239.152.75 with SMTP id u11mr224391hbb.113.1280628310389; Sat, 
	31 Jul 2010 19:05:10 -0700 (PDT)
Received: by 10.239.156.195 with HTTP; Sat, 31 Jul 2010 19:05:10 -0700 (PDT)
In-Reply-To: <AANLkTi=6kTtcJaKS6hXY_kLAmLEg+Pd_oDoWN1nDgQ42@mail.gmail.com>
References: <AANLkTimp1bEvRWR7ivOOb2Xkt2w5G_ur-SXuM5qSMn6w@mail.gmail.com>
	<AANLkTi=6bXkcmm7NM5HRCzZgWdjrdvSsNF5yCf5UT=qH@mail.gmail.com>
	<AANLkTi=kHkc-zRh-bv3Ot-s-+8T2gSEiTF7kUH4Z9+oP@mail.gmail.com>
	<AANLkTin=Caq5maK=vWQXURhO7fytE-QdFVh=6Ad7pVkf@mail.gmail.com>
	<AANLkTi=6kTtcJaKS6hXY_kLAmLEg+Pd_oDoWN1nDgQ42@mail.gmail.com>
Date: Sat, 31 Jul 2010 22:05:10 -0400
Message-ID: <AANLkTinFpaQ+WWR_L6ua5gvg5zU9yZo0y3yZFHr41F7t@mail.gmail.com>
Subject: Re: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap
From: Sameer Joshi <sameer.joshi@serenesoftware.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485ecf72c87b4c9048cb9836b
X-Virus-Checked: Checked by ClamAV on apache.org

--001485ecf72c87b4c9048cb9836b
Content-Type: text/plain; charset=ISO-8859-1

Hi again Sonal,
Just checking, if I'm running a pseudo-cluster on one node, so the master
and slaves are all localhost. What may the /etc/hosts look like in this
case?

Thanks,
Sameer

On Sat, Jul 31, 2010 at 3:37 AM, Sameer Joshi <
sameer.joshi@serenesoftware.com> wrote:

> Hi Sonal,
> Sorry for the late reply, I was traveling. The logs, unfortunately, did not
> yield too much information. I'll try some more configurations with etc/hosts
> edits, and see if I can't get it to behave.
> Thanks so much again for your help.
> Sameer
>
>
> On Wed, Jul 28, 2010 at 1:20 PM, Sonal Goyal <sonalgoyal4@gmail.com>wrote:
>
>> Hi Sameer,
>>
>> Do you see all processes up? Can you check the logs and see what is going
>> wrong? I suspect you will need to make the entries in /etc/hosts anyways.
>> Somewhere in hadoop code, the ip  hostname mapping is done, I forget the
>> exact location.
>>
>> Thanks and Regards,
>> Sonal
>> www.meghsoft.com
>> http://in.linkedin.com/in/sonalgoyal
>>
>>
>> On Wed, Jul 28, 2010 at 10:38 PM, Sameer Joshi <
>> sameer.joshi@serenesoftware.com> wrote:
>>
>> > Hi Sonal,
>> > Thank you for replying.
>> >
>> > I'm trying to run it as single node, so I had localhost in all the conf
>> > files and in /etc/hosts. I tried replacing it with 127.0.0.1 in all
>> Hadoop
>> > conf files (read to do that on some forum), but with no change.
>> >
>> > I figured I didn't need hostname since I was trying to run it locally,
>> but
>> > still tried replacing localhost with 173.1.86.194. However, as expected,
>> it
>> > could not connect to server.
>> >
>> > Any thoughts?
>> >
>> > Thanks for your help, I'm new at this and feel much obliged.
>> >
>> > Regards,
>> > Sameer
>> >
>> > On Wed, Jul 28, 2010 at 12:12 AM, Sonal Goyal <sonalgoyal4@gmail.com>
>> > wrote:
>> >
>> > > Hi Sameer,
>> > >
>> > > For GoGrid, you will have to configure /etc/hosts with the ip and
>> > hostname
>> > > for master and slaves. If you run hostname and nslookup -sil, you will
>> > get
>> > > the details for your machines. Make sure /etc/hosts has an entry like:
>> > >
>> > > 173.1.45.202 173.1.45.202.reverse.gogrid.com 28684_1_85017_358406
>> > >
>> > > Thanks and Regards,
>> > > Sonal
>> > > www.meghsoft.com
>> > > http://in.linkedin.com/in/sonalgoyal
>> > >
>> > >
>> > > On Wed, Jul 28, 2010 at 6:42 AM, Sameer Joshi <
>> > > sameer.joshi@serenesoftware.com> wrote:
>> > >
>> > > > I installed Java and Hadoop on a GoGrid cloud server using Red Hat
>> > > > Enterprise Linux Server release 5.1 (Tikanga). Hadoop installed fine
>> > and
>> > > > starts fine, however I get an error (java.lang.NullPointerException
>> at
>> > > > java.util.concurrent.ConcurrentHashMap) while running the Hadoop
>> > > wordcount
>> > > > example. My guess was that this was a localhost or IPv6 issue.
>> > > >
>> > > > * I have tested replacing 'localhost' with both the local IP, and
>> > server
>> > > IP
>> > > > addresses (when out of options) in Hadoop conf
>> > > > * I have disabled IPv6 both in sysctl.conf and hadoop-env.sh (former
>> > > > followed by a server restart)
>> > > >
>> > > > Any thoughts?
>> > > > Thank you.
>> > > >
>> > > >
>> > > > The output is given below
>> > > >
>> > > > # bin/hadoop jar hadoop-0.20.2-examples.jar wordcount datasets
>> > tests/out7
>> > > > 10/07/27 05:44:59 INFO input.FileInputFormat: Total input paths to
>> > > process
>> > > > :
>> > > > 1
>> > > > 10/07/27 05:44:59 INFO mapred.JobClient: Running job:
>> > > job_201007270544_0001
>> > > > 10/07/27 05:45:00 INFO mapred.JobClient: map 0% reduce 0%
>> > > > 10/07/27 05:45:12 INFO mapred.JobClient: map 100% reduce 0%
>> > > > 10/07/27 05:45:17 INFO mapred.JobClient: Task Id :
>> > > > attempt_201007270544_0001_r_000000_0, Status : FAILED
>> > > >
>> > > > Error: java.lang.NullPointerException
>> > > > at java.util.concurrent.ConcurrentHashMap.get(Unknown Source)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
>> > > >
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
>> > > >
>> > > >
>> > > > 10/07/27 05:45:24 INFO mapred.JobClient: Task Id :
>> > > > attempt_201007270544_0001_r_000000_1, Status : FAILED
>> > > > Error: java.lang.NullPointerException
>> > > > at java.util.concurrent.ConcurrentHashMap.get(Unknown Source)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
>> > > >
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
>> > > >
>> > > >
>> > > > 10/07/27 05:45:30 INFO mapred.JobClient: Task Id :
>> > > > attempt_201007270544_0001_r_000000_2, Status : FAILED
>> > > > Error: java.lang.NullPointerException
>> > > > at java.util.concurrent.ConcurrentHashMap.get(Unknown Source)
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
>> > > >
>> > > > at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
>> > > >
>> > > >
>> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Job complete:
>> > > > job_201007270544_0001
>> > > >
>> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Counters: 12
>> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Job Counters
>> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Launched reduce tasks=4
>> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Launched map tasks=1
>> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Data-local map tasks=1
>> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Failed reduce tasks=1
>> > > > 10/07/27 05:45:39 INFO mapred.JobClient: FileSystemCounters
>> > > > 10/07/27 05:45:39 INFO mapred.JobClient: HDFS_BYTES_READ=15319
>> > > > 10/07/27 05:45:39 INFO mapred.JobClient: FILE_BYTES_WRITTEN=12847
>> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Map-Reduce Framework
>> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Combine output records=934
>> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Map input records=149
>> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Spilled Records=934
>> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Map output bytes=25346
>> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Combine input records=2541
>> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Map output records=2541
>> > > >
>> > > > --
>> > > > Dr. Sameer Joshi, Ph.D.
>> > > > Senior computer scientist,
>> > > > Serene Software.
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Dr. Sameer Joshi, Ph.D.
>> > Senior computer scientist,
>> > Serene Software.
>> >
>>
>
>
>
> --
> Dr. Sameer Joshi, Ph.D.
> Senior computer scientist,
> Serene Software.
>
>


-- 
Dr. Sameer Joshi, Ph.D.
Senior computer scientist,
Serene Software.

--001485ecf72c87b4c9048cb9836b--

From general-return-1875-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 01 15:00:04 2010
Return-Path: <general-return-1875-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96758 invoked from network); 1 Aug 2010 15:00:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Aug 2010 15:00:03 -0000
Received: (qmail 1948 invoked by uid 500); 1 Aug 2010 15:00:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1813 invoked by uid 500); 1 Aug 2010 15:00:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1805 invoked by uid 99); 1 Aug 2010 15:00:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Aug 2010 15:00:01 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of sonalgoyal4@gmail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Aug 2010 14:59:57 +0000
Received: by iwn37 with SMTP id 37so3016iwn.35
        for <general@hadoop.apache.org>; Sun, 01 Aug 2010 07:59:36 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=c3iUYEWZ9GTbmDWd4aMIcUTpLRW5TWpisNJh7ILrlJE=;
        b=aGG8YVOsQbfLgYL+LUGdfvni9B7oTEB+PuamPQOZoQfHVTunytPSQuNj9ifDo4wIih
         9nreYp88GMkvmRQCILJbeH66eFTc8TSlV+CeqbsaYoUPOgXfjHXJH7hcPJ5ZZTh+w5rU
         L3ua5RR9THOZgWwphUejgeWVa5qC94tlXWJcI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=bsQqEN1yqUfY64dxemqffAxBypm6Ey/cYNPfjmZ/Dxqvp/5M8pygmOhq6/h8I4LytW
         xUZnTsHWgxUBDKiMwkJ8t5srG3+JAZpVTGlh8yTW59OXpPUClGN5sPJnwbF0oW+EbAWq
         71Rbk6kfpNezrNfJ4iFv9PKCuj4vlJ2I8KknQ=
MIME-Version: 1.0
Received: by 10.231.182.9 with SMTP id ca9mr5386415ibb.120.1280674776653; Sun, 
	01 Aug 2010 07:59:36 -0700 (PDT)
Received: by 10.231.12.73 with HTTP; Sun, 1 Aug 2010 07:59:36 -0700 (PDT)
In-Reply-To: <AANLkTinFpaQ+WWR_L6ua5gvg5zU9yZo0y3yZFHr41F7t@mail.gmail.com>
References: <AANLkTimp1bEvRWR7ivOOb2Xkt2w5G_ur-SXuM5qSMn6w@mail.gmail.com>
	<AANLkTi=6bXkcmm7NM5HRCzZgWdjrdvSsNF5yCf5UT=qH@mail.gmail.com>
	<AANLkTi=kHkc-zRh-bv3Ot-s-+8T2gSEiTF7kUH4Z9+oP@mail.gmail.com>
	<AANLkTin=Caq5maK=vWQXURhO7fytE-QdFVh=6Ad7pVkf@mail.gmail.com>
	<AANLkTi=6kTtcJaKS6hXY_kLAmLEg+Pd_oDoWN1nDgQ42@mail.gmail.com>
	<AANLkTinFpaQ+WWR_L6ua5gvg5zU9yZo0y3yZFHr41F7t@mail.gmail.com>
Date: Sun, 1 Aug 2010 20:29:36 +0530
Message-ID: <AANLkTi=YRjz4bsFPhoibbsAajyc7N5GZ=+8NHKqpNyA-@mail.gmail.com>
Subject: Re: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap
From: Sonal Goyal <sonalgoyal4@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364ee28a22a2e2048cc4552b

--0016364ee28a22a2e2048cc4552b
Content-Type: text/plain; charset=ISO-8859-1

173.1.45.202 173.1.45.202.reverse.gogrid.com 28684_1_85017_358406

Change that to whatever your ips/hostnames are.

Thanks and Regards,
Sonal
www.meghsoft.com
http://in.linkedin.com/in/sonalgoyal


On Sun, Aug 1, 2010 at 7:35 AM, Sameer Joshi <
sameer.joshi@serenesoftware.com> wrote:

> Hi again Sonal,
> Just checking, if I'm running a pseudo-cluster on one node, so the master
> and slaves are all localhost. What may the /etc/hosts look like in this
> case?
>
> Thanks,
> Sameer
>
> On Sat, Jul 31, 2010 at 3:37 AM, Sameer Joshi <
> sameer.joshi@serenesoftware.com> wrote:
>
> > Hi Sonal,
> > Sorry for the late reply, I was traveling. The logs, unfortunately, did
> not
> > yield too much information. I'll try some more configurations with
> etc/hosts
> > edits, and see if I can't get it to behave.
> > Thanks so much again for your help.
> > Sameer
> >
> >
> > On Wed, Jul 28, 2010 at 1:20 PM, Sonal Goyal <sonalgoyal4@gmail.com
> >wrote:
> >
> >> Hi Sameer,
> >>
> >> Do you see all processes up? Can you check the logs and see what is
> going
> >> wrong? I suspect you will need to make the entries in /etc/hosts
> anyways.
> >> Somewhere in hadoop code, the ip  hostname mapping is done, I forget the
> >> exact location.
> >>
> >> Thanks and Regards,
> >> Sonal
> >> www.meghsoft.com
> >> http://in.linkedin.com/in/sonalgoyal
> >>
> >>
> >> On Wed, Jul 28, 2010 at 10:38 PM, Sameer Joshi <
> >> sameer.joshi@serenesoftware.com> wrote:
> >>
> >> > Hi Sonal,
> >> > Thank you for replying.
> >> >
> >> > I'm trying to run it as single node, so I had localhost in all the
> conf
> >> > files and in /etc/hosts. I tried replacing it with 127.0.0.1 in all
> >> Hadoop
> >> > conf files (read to do that on some forum), but with no change.
> >> >
> >> > I figured I didn't need hostname since I was trying to run it locally,
> >> but
> >> > still tried replacing localhost with 173.1.86.194. However, as
> expected,
> >> it
> >> > could not connect to server.
> >> >
> >> > Any thoughts?
> >> >
> >> > Thanks for your help, I'm new at this and feel much obliged.
> >> >
> >> > Regards,
> >> > Sameer
> >> >
> >> > On Wed, Jul 28, 2010 at 12:12 AM, Sonal Goyal <sonalgoyal4@gmail.com>
> >> > wrote:
> >> >
> >> > > Hi Sameer,
> >> > >
> >> > > For GoGrid, you will have to configure /etc/hosts with the ip and
> >> > hostname
> >> > > for master and slaves. If you run hostname and nslookup -sil, you
> will
> >> > get
> >> > > the details for your machines. Make sure /etc/hosts has an entry
> like:
> >> > >
> >> > > 173.1.45.202 173.1.45.202.reverse.gogrid.com 28684_1_85017_358406
> >> > >
> >> > > Thanks and Regards,
> >> > > Sonal
> >> > > www.meghsoft.com
> >> > > http://in.linkedin.com/in/sonalgoyal
> >> > >
> >> > >
> >> > > On Wed, Jul 28, 2010 at 6:42 AM, Sameer Joshi <
> >> > > sameer.joshi@serenesoftware.com> wrote:
> >> > >
> >> > > > I installed Java and Hadoop on a GoGrid cloud server using Red Hat
> >> > > > Enterprise Linux Server release 5.1 (Tikanga). Hadoop installed
> fine
> >> > and
> >> > > > starts fine, however I get an error
> (java.lang.NullPointerException
> >> at
> >> > > > java.util.concurrent.ConcurrentHashMap) while running the Hadoop
> >> > > wordcount
> >> > > > example. My guess was that this was a localhost or IPv6 issue.
> >> > > >
> >> > > > * I have tested replacing 'localhost' with both the local IP, and
> >> > server
> >> > > IP
> >> > > > addresses (when out of options) in Hadoop conf
> >> > > > * I have disabled IPv6 both in sysctl.conf and hadoop-env.sh
> (former
> >> > > > followed by a server restart)
> >> > > >
> >> > > > Any thoughts?
> >> > > > Thank you.
> >> > > >
> >> > > >
> >> > > > The output is given below
> >> > > >
> >> > > > # bin/hadoop jar hadoop-0.20.2-examples.jar wordcount datasets
> >> > tests/out7
> >> > > > 10/07/27 05:44:59 INFO input.FileInputFormat: Total input paths to
> >> > > process
> >> > > > :
> >> > > > 1
> >> > > > 10/07/27 05:44:59 INFO mapred.JobClient: Running job:
> >> > > job_201007270544_0001
> >> > > > 10/07/27 05:45:00 INFO mapred.JobClient: map 0% reduce 0%
> >> > > > 10/07/27 05:45:12 INFO mapred.JobClient: map 100% reduce 0%
> >> > > > 10/07/27 05:45:17 INFO mapred.JobClient: Task Id :
> >> > > > attempt_201007270544_0001_r_000000_0, Status : FAILED
> >> > > >
> >> > > > Error: java.lang.NullPointerException
> >> > > > at java.util.concurrent.ConcurrentHashMap.get(Unknown Source)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
> >> > > >
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
> >> > > >
> >> > > >
> >> > > > 10/07/27 05:45:24 INFO mapred.JobClient: Task Id :
> >> > > > attempt_201007270544_0001_r_000000_1, Status : FAILED
> >> > > > Error: java.lang.NullPointerException
> >> > > > at java.util.concurrent.ConcurrentHashMap.get(Unknown Source)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
> >> > > >
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
> >> > > >
> >> > > >
> >> > > > 10/07/27 05:45:30 INFO mapred.JobClient: Task Id :
> >> > > > attempt_201007270544_0001_r_000000_2, Status : FAILED
> >> > > > Error: java.lang.NullPointerException
> >> > > > at java.util.concurrent.ConcurrentHashMap.get(Unknown Source)
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
> >> > > >
> >> > > > at
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
> >> > > >
> >> > > >
> >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Job complete:
> >> > > > job_201007270544_0001
> >> > > >
> >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Counters: 12
> >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Job Counters
> >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Launched reduce tasks=4
> >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Launched map tasks=1
> >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Data-local map tasks=1
> >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Failed reduce tasks=1
> >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: FileSystemCounters
> >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: HDFS_BYTES_READ=15319
> >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: FILE_BYTES_WRITTEN=12847
> >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Map-Reduce Framework
> >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Combine output
> records=934
> >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Map input records=149
> >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Spilled Records=934
> >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Map output bytes=25346
> >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Combine input
> records=2541
> >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Map output records=2541
> >> > > >
> >> > > > --
> >> > > > Dr. Sameer Joshi, Ph.D.
> >> > > > Senior computer scientist,
> >> > > > Serene Software.
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Dr. Sameer Joshi, Ph.D.
> >> > Senior computer scientist,
> >> > Serene Software.
> >> >
> >>
> >
> >
> >
> > --
> > Dr. Sameer Joshi, Ph.D.
> > Senior computer scientist,
> > Serene Software.
> >
> >
>
>
> --
> Dr. Sameer Joshi, Ph.D.
> Senior computer scientist,
> Serene Software.
>

--0016364ee28a22a2e2048cc4552b--

From general-return-1876-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 01 18:32:32 2010
Return-Path: <general-return-1876-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72393 invoked from network); 1 Aug 2010 18:32:31 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Aug 2010 18:32:31 -0000
Received: (qmail 82722 invoked by uid 500); 1 Aug 2010 18:32:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82607 invoked by uid 500); 1 Aug 2010 18:32:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82599 invoked by uid 99); 1 Aug 2010 18:32:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Aug 2010 18:32:29 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Aug 2010 18:32:25 +0000
Received: by wwb18 with SMTP id 18so3722830wwb.29
        for <general@hadoop.apache.org>; Sun, 01 Aug 2010 11:32:03 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.87.208 with SMTP id y58mr3998162wee.82.1280687523373; Sun, 
	01 Aug 2010 11:32:03 -0700 (PDT)
Received: by 10.216.181.204 with HTTP; Sun, 1 Aug 2010 11:32:03 -0700 (PDT)
In-Reply-To: <COL117-W57E09052147A0DB3F567368FA60@phx.gbl>
References: <COL117-W15C85FC4B3465C944794C88FA60@phx.gbl>
	<COL117-W57E09052147A0DB3F567368FA60@phx.gbl>
Date: Sun, 1 Aug 2010 14:32:03 -0400
Message-ID: <AANLkTin8f6QXx6i7sciBWxrbPmqvSUSC_jVQusj55+7B@mail.gmail.com>
Subject: Re: Fair Scheduler documentation?
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d7e352e6223d048cc74cdc

--0016e6d7e352e6223d048cc74cdc
Content-Type: text/plain; charset=ISO-8859-1

The fair scheduler docs are available at:
http://hadoop.apache.org/common/docs/current/fair_scheduler.html

On Mon, Jul 26, 2010 at 12:13 PM, Michael Segel
<michael_segel@hotmail.com>wrote:

>
>
> > From: michael_segel@hotmail.com
> > To: general@hadoop.apache.org
> > Subject: Fair Scheduler documentation?
> > Date: Mon, 26 Jul 2010 11:02:36 -0500
> >
> >
> > Hi,
> >
> > Ok so we're setting up the fair scheduler on one of our clusters.
> >
> > The only documentation I could find is the wiki, and then one or two
> power points...
> >
> > Is there any source that contains all of the possible options allowed in
> the allocations.xml file?
> >
> > I came across something like <maxMaps> somewhere the other day.. couldn't
> find it. Not sure if its a valid
> > option.
> [SNIP]
>
>
>
>
> https://issues.apache.org/jira/browse/MAPREDUCE-698
> (Found using Yahoo! as my search engine... )
>
> Ok this explains why I can't implement maxMaps on my cloud where I'm
> testing out the fair scheduler.
>
> Also is there a way to specify the pool to be used at the time the job is
> launched?
>
> Thx
>
> -Mike
>
>
> _________________________________________________________________
> The New Busy is not the too busy. Combine all your e-mail accounts with
> Hotmail.
>
> http://www.windowslive.com/campaign/thenewbusy?tile=multiaccount&ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_4
>



-- 
Eric Sammer
twitter: esammer
data: www.cloudera.com

--0016e6d7e352e6223d048cc74cdc--

From general-return-1877-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 01 18:39:47 2010
Return-Path: <general-return-1877-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72969 invoked from network); 1 Aug 2010 18:39:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Aug 2010 18:39:47 -0000
Received: (qmail 88506 invoked by uid 500); 1 Aug 2010 18:39:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88440 invoked by uid 500); 1 Aug 2010 18:39:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88432 invoked by uid 99); 1 Aug 2010 18:39:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Aug 2010 18:39:45 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Aug 2010 18:39:40 +0000
Received: by wwb18 with SMTP id 18so3727147wwb.29
        for <general@hadoop.apache.org>; Sun, 01 Aug 2010 11:39:19 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.2.195 with SMTP id 45mr1853881wef.106.1280687958908; Sun, 
	01 Aug 2010 11:39:18 -0700 (PDT)
Received: by 10.216.181.204 with HTTP; Sun, 1 Aug 2010 11:39:18 -0700 (PDT)
In-Reply-To: <AANLkTim9qtnHBZfFpTxyFCq37X=TRaQ1Z=LWcv_xTNri@mail.gmail.com>
References: <AANLkTi=c8p1O9wBjfO6EGi7XcwgfUGPcFhLyR086oxAF@mail.gmail.com>
	<AANLkTi=pEPNuXOmBrZ-tfhqQJY---A5_sE+VXJf2BB1G@mail.gmail.com>
	<AANLkTim9qtnHBZfFpTxyFCq37X=TRaQ1Z=LWcv_xTNri@mail.gmail.com>
Date: Sun, 1 Aug 2010 14:39:18 -0400
Message-ID: <AANLkTikmBnUNJvzfPEzS19KToPYwCh3UdhwfXOPny=aO@mail.gmail.com>
Subject: Re: Hadoop Cluster Configuration
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364c7541dbe04b048cc766c8

--0016364c7541dbe04b048cc766c8
Content-Type: text/plain; charset=ISO-8859-1

Vaibhav:

The configuration for Hadoop is such that the same configuration files can
be distributed to all nodes. Normally, users configure Hadoop appropriate to
their environment and then simply distribute the same configs to all nodes
in the cluster. This is usually much easier than worrying about which config
parameters apply to which daemons.

That said, it is possible to have different configs for different machines,
if necessary. There's currently no documentation that says which parameters
are read by which daemons although it's (usually) possible to figure that
out from the naming convention of the parameters. Parameters in
core-site.xml apply to all daemons, hdfs-site to HDFS, and mapred-site.xml
to the map reduce daemons. As for which parameters are used by which
daemons, dfs.tasktracker.* is used by the task tracker, for instance. Some
are harder to figure out just by their names. Again, I would recommend
distributing the same configs everywhere to make things easier.

On Wed, Jul 28, 2010 at 4:44 AM, vaibhav negi <sssssssenator@gmail.com>wrote:

> Hi,
>
> I am using hadoop versoion 0.20 . I am trying set up hadoop cluster. What
> are the configurations for name node? What are the configurations for data
> node?
>
> In documentation all configurations  are given, but, it is not mentioned
> what needs to be configured for which node.
>
>
> Vaibhav Negi
>
>
> On Wed, Jul 28, 2010 at 1:32 PM, Hemanth Yamijala <yhemanth@gmail.com
> >wrote:
>
> > Vaibhav,
> >
> > > While setting hadoop cluster, does configuration files
> > (conf/core-site.xml,
> > > conf/mapred-site.xml,conf/hdfs-site.xml) in every node(name node and
> data
> > > nodes)  needs to be configured in the same manner?
> >
> > This is a complicated question to answer :-). There are certain
> > configuration variables that need to be defined to be the same between
> > the master and the slaves and some that don't need to be. Pre Hadoop
> > 0.21, there is no easy way other than documentation for the variables
> > (hopefully) to determine if this is the case or not. I think in Hadoop
> > 0.21 and since, we have tried to adopt a convention to include the
> > daemon name to specify which variables are used by which daemons. And
> > those that are cluster-wide, that need to be the same throughout all
> > the nodes will have something like 'cluster' in the name.
> >
> > Your best bet in any case is possibly to sift through the
> > documentation of the variables you are interested in. Or else to post
> > a query here.
> >
> > > How does configuration of name node differs from configuration of data
> > > nodes?
> >
> > Not sure about this one.
> >
> > Thanks
> > hemanth
> >
>



-- 
Eric Sammer
twitter: esammer
data: www.cloudera.com

--0016364c7541dbe04b048cc766c8--

From general-return-1878-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 01 18:41:51 2010
Return-Path: <general-return-1878-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73167 invoked from network); 1 Aug 2010 18:41:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Aug 2010 18:41:50 -0000
Received: (qmail 90791 invoked by uid 500); 1 Aug 2010 18:41:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90692 invoked by uid 500); 1 Aug 2010 18:41:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90684 invoked by uid 99); 1 Aug 2010 18:41:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Aug 2010 18:41:48 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Aug 2010 18:41:42 +0000
Received: by wwb18 with SMTP id 18so3728323wwb.29
        for <general@hadoop.apache.org>; Sun, 01 Aug 2010 11:41:21 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.167.205 with SMTP id i55mr1875024wel.17.1280688081840; 
	Sun, 01 Aug 2010 11:41:21 -0700 (PDT)
Received: by 10.216.181.204 with HTTP; Sun, 1 Aug 2010 11:41:21 -0700 (PDT)
In-Reply-To: <BAY145-w338720719FD00C11E70824BEA20@phx.gbl>
References: <BAY145-w5603FAE32A12B4317DFC97BEA20@phx.gbl>
	<BAY145-w296E5AFF098ECADBDEEA5BEA20@phx.gbl>
	<BAY145-w338720719FD00C11E70824BEA20@phx.gbl>
Date: Sun, 1 Aug 2010 14:41:21 -0400
Message-ID: <AANLkTi==u7wbEkod2YRfNuMES6TkkB6vT6fGD+PwNoxs@mail.gmail.com>
Subject: Re: TaskTracker Error
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Cc: pig user <pig-user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=0016e649c80c2fabf9048cc76ee0
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e649c80c2fabf9048cc76ee0
Content-Type: text/plain; charset=ISO-8859-1

Syed:

JDK u18 has significant known problems. You should definitely switch to
either 1.6.0_16 (tried and true) or _20. These are most commonly the patch
levels we see in production settings. Try that and see if the problem
persists.

On Thu, Jul 22, 2010 at 4:54 PM, Syed Wasti <mdwasti@hotmail.com> wrote:

>
> Found this almost a year old discussion, on the same error code and it
> looks like a JVM crash.
> The discussion confirms the crash on JDK 14 and I am using JDK version 18.
>
>
>
> http://markmail.org/message/74yv7zyuq7ucm2z6
>
>
> From: mdwasti@hotmail.com
> To: pig-user@hadoop.apache.org
> Subject: TaskTracker Error
> Date: Thu, 22 Jul 2010 11:45:38 -0700
>
>
>
>
>
>
>
>
> Hi,
> I am seeing the below two errors consistently while running my script. This
> does not stop my script, but throws an error on datanode and successfully
> completes the same task on a different datanode. The task could fail on any
> datanode, there are no blacklisted datanodes.
> Any thoughts on why this is and how can this be resolved ?
>
> This is the whole stack;
> java.lang.Throwable: Child Error
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:472)
> Caused by: java.io.IOException: Task process exit with nonzero status of
> 134.
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:459)
>
> Second Error: This is all it gives.
> Error: null
>
> Third Error:
> Error: (class: org/apache/pig/builtin/PigStorage, method: setLocation
> signature: (Ljava/lang/String;Lorg/apache/hadoop/mapreduce/Job;)V)
>
>
> Thanks for the support.
> Regards
> Syed Wasti
>
>



-- 
Eric Sammer
twitter: esammer
data: www.cloudera.com

--0016e649c80c2fabf9048cc76ee0--

From general-return-1879-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 16:03:18 2010
Return-Path: <general-return-1879-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 11103 invoked from network); 2 Aug 2010 16:03:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 16:03:18 -0000
Received: (qmail 79647 invoked by uid 500); 2 Aug 2010 16:03:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79538 invoked by uid 500); 2 Aug 2010 16:03:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 79530 invoked by uid 99); 2 Aug 2010 16:03:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 16:03:16 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jdixon@omniti.com designates 8.8.38.6 as permitted sender)
Received: from [8.8.38.6] (HELO edge.omniti.com) (8.8.38.6)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 16:03:08 +0000
DKIM-Signature: v=1; a=rsa-sha256; d=omniti.com; s=s1024; c=relaxed/relaxed;
	q=dns/txt; i=@omniti.com; t=1280764967;
	h=From:Subject:Date:To;
	bh=ZBUdPoTkYjXTQ+PfVQi4OWGgYV50VdOXaTT+oMwJw5E=;
	b=BdLwgziwJlr1cQRTdMg1ucQCWV1ocrdeHmijwPuqKAQcH0HhFzWhn+hNJjjjfKbp
	wB/rU9HQ9gdRQqtUOeA+Ff2aHW/kpdy/4REifys0LH/OdGz7q3hTuQBxFJk2D/lZ
	K15RuRHjcMrEOPodNCDKVpNylfU4wSiU4KBV7FKK/Xg=;
Authentication-Results:  edge smtp.user=jdixon@omniti.com; auth=pass (LOGIN)
Received: from [68.55.0.29] ([68.55.0.29:65314] helo=omniti.com)
	by edge (envelope-from <jdixon@omniti.com>)
	(ecelerity 2.2.3.46 r(37468M)) with ESMTPSA (cipher=AES256-SHA) 
	id 78/D9-03560-72CE65C4; Mon, 02 Aug 2010 12:02:47 -0400
Date: Mon, 2 Aug 2010 12:02:44 -0400
From: Jason Dixon <jdixon@omniti.com>
To: general@hadoop.apache.org
Subject: Register now for Surge 2010
Message-ID: <20100802160244.GV4578@omniti.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.18 (2008-05-17)
X-Virus-Checked: Checked by ClamAV on apache.org

Registration for Surge Scalability Conference 2010 is open for all
attendees!  We have an awesome lineup of leaders from across the various
communities that support highly scalable architectures, as well as the
companies that implement them.  Here's a small sampling from our list of
speakers:

John Allspaw, Etsy
Theo Schlossnagle, OmniTI
Rasmus Lerdorf, creator of PHP
Tom Cook, Facebook
Benjamin Black, fast_ip
Artur Bergman, Wikia
Christopher Brown, Opscode
Bryan Cantrill, Joyent
Baron Schwartz, Percona
Paul Querna, Cloudkick

Surge 2010 focuses on real case studies from production environments;
the lessons learned from failure and how to re-engineer your way to a
successful, highly scalable Internet architecture.  The conference takes
place at the Tremont Grand Historic Venue on Sept 30 and Oct 1, 2010 in
Baltimore, MD.  Register now to enjoy the Early Bird discount and
guarantee your seat to this year's event!

http://omniti.com/surge/2010/register

Thanks,

-- 
Jason Dixon
OmniTI Computer Consulting, Inc.
jdixon@omniti.com
443.325.1357 x.241

From general-return-1880-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 16:53:53 2010
Return-Path: <general-return-1880-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30418 invoked from network); 2 Aug 2010 16:53:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 16:53:52 -0000
Received: (qmail 57011 invoked by uid 500); 2 Aug 2010 16:53:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56896 invoked by uid 500); 2 Aug 2010 16:53:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56888 invoked by uid 99); 2 Aug 2010 16:53:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 16:53:50 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 16:53:44 +0000
Received: by pwj2 with SMTP id 2so2708203pwj.35
        for <general@hadoop.apache.org>; Mon, 02 Aug 2010 09:53:21 -0700 (PDT)
Received: by 10.142.192.2 with SMTP id p2mr5501178wff.222.1280768001600; Mon, 
	02 Aug 2010 09:53:21 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.182.79 with HTTP; Mon, 2 Aug 2010 09:52:51 -0700 (PDT)
X-Originating-IP: [84.108.229.170]
From: Maxim Veksler <maxim@vekslers.org>
Date: Mon, 2 Aug 2010 19:52:51 +0300
Message-ID: <AANLkTik5koLPa0yNxtc+ta4nJ__Oym4ATq5xn=sRgz29@mail.gmail.com>
Subject: Data World Record Falls as Computer Scientists Break Terabyte Sort 
	Barrier
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd2e094c67aed048cda09aa
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd2e094c67aed048cda09aa
Content-Type: text/plain; charset=ISO-8859-1

Hi,

Anyone knows if Hadoop is involved? And if so what is the configuration for
such cluster?

http://ucsdnews.ucsd.edu/newsrel/science/07-27DataWorld.asp


Thank you,
Maxim.

--000e0cd2e094c67aed048cda09aa--

From general-return-1881-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 17:35:23 2010
Return-Path: <general-return-1881-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49709 invoked from network); 2 Aug 2010 17:35:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 17:35:22 -0000
Received: (qmail 9683 invoked by uid 500); 2 Aug 2010 17:35:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 9164 invoked by uid 500); 2 Aug 2010 17:35:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8940 invoked by uid 99); 2 Aug 2010 17:35:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 17:35:20 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of vermaabhishekp@gmail.com designates 74.125.82.42 as permitted sender)
Received: from [74.125.82.42] (HELO mail-ww0-f42.google.com) (74.125.82.42)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 17:35:14 +0000
Received: by wwf26 with SMTP id 26so10730553wwf.5
        for <general@hadoop.apache.org>; Mon, 02 Aug 2010 10:34:51 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=OFKq3l82QnN1h//6vMGKzmJvSUXmQWKaYZnnsAGZPsE=;
        b=J4sFGZl7R3WbYFcIae2gRPiyLHHaY2We+qNW7aZsErJKNKaLgyEDJTcYmKkMw6iY4Y
         UReSqWUbsSny3tXSQntqApSgtJXNaTntkQ9LypCykDtPuotv6HHG3ydT3UCbkNvVa5Vm
         st661/ZlAQKjnL1QhoJ9/Zw14vUyJGd1lQiUM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=vv6fsa0nWouJ36PtxCQE38Btq0OkUdZfq8EFoWfUgup+qZOLWO0SAHyoSjieVLgNIH
         0QbPzPGfyzdIrP10WV5HoE2IbScLL/rJcRIdhzbvpwMHAkdTXp91krLdaDW+8ydY6c38
         PlYgNQctOkw9PbWbJbEJEIFIE3BnU4/u57JS0=
MIME-Version: 1.0
Received: by 10.227.136.69 with SMTP id q5mr2120687wbt.202.1280770486025; Mon, 
	02 Aug 2010 10:34:46 -0700 (PDT)
Received: by 10.227.144.205 with HTTP; Mon, 2 Aug 2010 10:34:45 -0700 (PDT)
In-Reply-To: <AANLkTik5koLPa0yNxtc+ta4nJ__Oym4ATq5xn=sRgz29@mail.gmail.com>
References: <AANLkTik5koLPa0yNxtc+ta4nJ__Oym4ATq5xn=sRgz29@mail.gmail.com>
Date: Mon, 2 Aug 2010 10:34:45 -0700
Message-ID: <AANLkTi=yMdJuOxWAogr5uw0iYbp2B88iujFO-0_M=8mr@mail.gmail.com>
Subject: Re: Data World Record Falls as Computer Scientists Break Terabyte 
	Sort Barrier
From: Abhishek Verma <vermaabhishekp@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6498450dbc039048cda9d02
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6498450dbc039048cda9d02
Content-Type: text/plain; charset=ISO-8859-1

Hi Maxim,

Hadoop was not involved. You can find more details here :
http://sortbenchmark.org/tritonsort_2010_May_15.pdf
and all the records and their information here : http://sortbenchmark.org/

<http://sortbenchmark.org/>-Abhishek.

On Mon, Aug 2, 2010 at 9:52 AM, Maxim Veksler <maxim@vekslers.org> wrote:

> Hi,
>
> Anyone knows if Hadoop is involved? And if so what is the configuration for
> such cluster?
>
> http://ucsdnews.ucsd.edu/newsrel/science/07-27DataWorld.asp
>
>
> Thank you,
> Maxim.
>

--0016e6498450dbc039048cda9d02--

From general-return-1882-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 18:49:30 2010
Return-Path: <general-return-1882-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 86261 invoked from network); 2 Aug 2010 18:49:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 18:49:30 -0000
Received: (qmail 29745 invoked by uid 500); 2 Aug 2010 18:49:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29655 invoked by uid 500); 2 Aug 2010 18:49:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29647 invoked by uid 99); 2 Aug 2010 18:49:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 18:49:28 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 18:49:24 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o72IlDQY036000
	for <general@hadoop.apache.org>; Mon, 2 Aug 2010 11:47:13 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=O5/6TZMGYrh3iHhatDD/w0Zo04Bc9qUpG+Vjl7oXU82fvmTWKIu/m+o5NOHy86h2
Message-Id: <17C47A26-3B02-4BEA-B862-7849391A6D74@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <AANLkTi=yMdJuOxWAogr5uw0iYbp2B88iujFO-0_M=8mr@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Data World Record Falls as Computer Scientists Break Terabyte Sort Barrier
Date: Mon, 2 Aug 2010 11:47:12 -0700
References: <AANLkTik5koLPa0yNxtc+ta4nJ__Oym4ATq5xn=sRgz29@mail.gmail.com> <AANLkTi=yMdJuOxWAogr5uw0iYbp2B88iujFO-0_M=8mr@mail.gmail.com>
X-Mailer: Apple Mail (2.936)

The UCSD results are very impressive, especially given their hardware  
budget.

I may be wrong, but I'm pretty sure there were no Hadoop based entries  
this year - I know we at Yahoo! didn't enter.

Couple of points:
# The Indy category is a benchmark to sort fixed length records, not a  
_general_ sort benchmark i.e. Daytona.
# Our _best_ result missed the deadline by a whisker last year, but we  
eventually did 100Tb sort in 95 mins and a 1000TB (1PB) in 975 mins  
(16.25 hrs) - which worked out to be just over 1.0 TB/min, which was  
nearly twice as fast as the record attributed to us. (http://developer.yahoo.net/blogs/hadoop/2009/05/hadoop_sorts_a_petabyte_in_162.html 
)

Arun

On Aug 2, 2010, at 10:34 AM, Abhishek Verma wrote:

> Hi Maxim,
>
> Hadoop was not involved. You can find more details here :
> http://sortbenchmark.org/tritonsort_2010_May_15.pdf
> and all the records and their information here : http://sortbenchmark.org/
>
> <http://sortbenchmark.org/>-Abhishek.
>
> On Mon, Aug 2, 2010 at 9:52 AM, Maxim Veksler <maxim@vekslers.org>  
> wrote:
>
>> Hi,
>>
>> Anyone knows if Hadoop is involved? And if so what is the  
>> configuration for
>> such cluster?
>>
>> http://ucsdnews.ucsd.edu/newsrel/science/07-27DataWorld.asp
>>
>>
>> Thank you,
>> Maxim.
>>


From general-return-1883-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 20:51:21 2010
Return-Path: <general-return-1883-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30527 invoked from network); 2 Aug 2010 20:51:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 20:51:20 -0000
Received: (qmail 6570 invoked by uid 500); 2 Aug 2010 20:51:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6474 invoked by uid 500); 2 Aug 2010 20:51:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6363 invoked by uid 99); 2 Aug 2010 20:51:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 20:51:19 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of vermaabhishekp@gmail.com designates 74.125.82.179 as permitted sender)
Received: from [74.125.82.179] (HELO mail-wy0-f179.google.com) (74.125.82.179)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 20:51:13 +0000
Received: by wyb42 with SMTP id 42so3567570wyb.38
        for <general@hadoop.apache.org>; Mon, 02 Aug 2010 13:50:53 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=1/VwaGHdj1Fo/3uBF5GvbDxdctUZk8+oPFBUJPd9OFw=;
        b=tSiMFCwjnpu8GLka8BE5wWJ3amr34vzMaEY2kK5d8Rf0jT9R4DP6Jd7zUf8L+NIxS9
         MAvizR6yTmoQCGTZFdqIcEZLDp6cQIz0/axBk4ra3sVC+brL6DZUnVOwO5aKT5kRx3/i
         eqyDEpUMo5XfyxMWzv2JEk4K+cTTQaae5cXUc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=T59p9EYwdElfi8WSXyARpsJ7IOuS5HfWhvnZC7JY0CBLnnd/1ViZ3oNO7rLGWJhNwP
         cw/Py+RjFRA7ED7T7rjT1uS835TYKTkEgwBi7F+sLAJJ+1//QpwOsZey9VJYzm3DucQ+
         Py8Ol6n3iZ7O6AwQ3oMuLX8n6Hpzbzoeqk81k=
MIME-Version: 1.0
Received: by 10.227.136.69 with SMTP id q5mr2314061wbt.202.1280782252977; Mon, 
	02 Aug 2010 13:50:52 -0700 (PDT)
Received: by 10.227.144.205 with HTTP; Mon, 2 Aug 2010 13:50:52 -0700 (PDT)
In-Reply-To: <17C47A26-3B02-4BEA-B862-7849391A6D74@yahoo-inc.com>
References: <AANLkTik5koLPa0yNxtc+ta4nJ__Oym4ATq5xn=sRgz29@mail.gmail.com>
	<AANLkTi=yMdJuOxWAogr5uw0iYbp2B88iujFO-0_M=8mr@mail.gmail.com>
	<17C47A26-3B02-4BEA-B862-7849391A6D74@yahoo-inc.com>
Date: Mon, 2 Aug 2010 13:50:52 -0700
Message-ID: <AANLkTikoAofAVM6Wi2Gj1nh+O4v2RL9VLqbocQMmVV1d@mail.gmail.com>
Subject: Re: Data World Record Falls as Computer Scientists Break Terabyte 
	Sort Barrier
From: Abhishek Verma <vermaabhishekp@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64984503931c5048cdd5bf8
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e64984503931c5048cdd5bf8
Content-Type: text/plain; charset=ISO-8859-1

It shows how further behind Hadoop is in terms of performance. Are there
people working on finding the bottlenecks and making it more efficient? Are
there any JIRA issues related to this?

-Abhishek.

On Mon, Aug 2, 2010 at 11:47 AM, Arun C Murthy <acm@yahoo-inc.com> wrote:

> The UCSD results are very impressive, especially given their hardware
> budget.
>
> I may be wrong, but I'm pretty sure there were no Hadoop based entries this
> year - I know we at Yahoo! didn't enter.
>
> Couple of points:
> # The Indy category is a benchmark to sort fixed length records, not a
> _general_ sort benchmark i.e. Daytona.
> # Our _best_ result missed the deadline by a whisker last year, but we
> eventually did 100Tb sort in 95 mins and a 1000TB (1PB) in 975 mins (16.25
> hrs) - which worked out to be just over 1.0 TB/min, which was nearly twice
> as fast as the record attributed to us. (
> http://developer.yahoo.net/blogs/hadoop/2009/05/hadoop_sorts_a_petabyte_in_162.html
> )
>
> Arun
>
>
> On Aug 2, 2010, at 10:34 AM, Abhishek Verma wrote:
>
>  Hi Maxim,
>>
>> Hadoop was not involved. You can find more details here :
>> http://sortbenchmark.org/tritonsort_2010_May_15.pdf
>> and all the records and their information here :
>> http://sortbenchmark.org/
>>
>> <http://sortbenchmark.org/>-Abhishek.
>>
>> On Mon, Aug 2, 2010 at 9:52 AM, Maxim Veksler <maxim@vekslers.org> wrote:
>>
>>  Hi,
>>>
>>> Anyone knows if Hadoop is involved? And if so what is the configuration
>>> for
>>> such cluster?
>>>
>>> http://ucsdnews.ucsd.edu/newsrel/science/07-27DataWorld.asp
>>>
>>>
>>> Thank you,
>>> Maxim.
>>>
>>>
>

--0016e64984503931c5048cdd5bf8--

From general-return-1884-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 03 00:46:27 2010
Return-Path: <general-return-1884-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23870 invoked from network); 3 Aug 2010 00:46:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Aug 2010 00:46:27 -0000
Received: (qmail 31201 invoked by uid 500); 3 Aug 2010 00:46:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31131 invoked by uid 500); 3 Aug 2010 00:46:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31123 invoked by uid 99); 3 Aug 2010 00:46:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Aug 2010 00:46:25 +0000
X-ASF-Spam-Status: No, hits=0.6 required=10.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_MED,SPF_NEUTRAL,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.149.205] (HELO na3sys009aog111.obsmtp.com) (74.125.149.205)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 03 Aug 2010 00:46:18 +0000
Received: from source ([209.85.161.54]) by na3sys009aob111.postini.com ([74.125.148.12]) with SMTP
	ID DSNKTFdmwcCDz1iSg3Lc7UOyTyNsCa5qOTCB@postini.com; Mon, 02 Aug 2010 17:45:57 PDT
Received: by fxm13 with SMTP id 13so1948960fxm.13
        for <general@hadoop.apache.org>; Mon, 02 Aug 2010 17:45:53 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.239.172.67 with SMTP id z3mr315745hbe.92.1280796352873; Mon, 
	02 Aug 2010 17:45:52 -0700 (PDT)
Received: by 10.239.156.195 with HTTP; Mon, 2 Aug 2010 17:45:52 -0700 (PDT)
In-Reply-To: <AANLkTi=YRjz4bsFPhoibbsAajyc7N5GZ=+8NHKqpNyA-@mail.gmail.com>
References: <AANLkTimp1bEvRWR7ivOOb2Xkt2w5G_ur-SXuM5qSMn6w@mail.gmail.com>
	<AANLkTi=6bXkcmm7NM5HRCzZgWdjrdvSsNF5yCf5UT=qH@mail.gmail.com>
	<AANLkTi=kHkc-zRh-bv3Ot-s-+8T2gSEiTF7kUH4Z9+oP@mail.gmail.com>
	<AANLkTin=Caq5maK=vWQXURhO7fytE-QdFVh=6Ad7pVkf@mail.gmail.com>
	<AANLkTi=6kTtcJaKS6hXY_kLAmLEg+Pd_oDoWN1nDgQ42@mail.gmail.com>
	<AANLkTinFpaQ+WWR_L6ua5gvg5zU9yZo0y3yZFHr41F7t@mail.gmail.com>
	<AANLkTi=YRjz4bsFPhoibbsAajyc7N5GZ=+8NHKqpNyA-@mail.gmail.com>
Date: Mon, 2 Aug 2010 20:45:52 -0400
Message-ID: <AANLkTikj_R_aM+qitDjmryMArKW_W3yOfzzfa+zENzT8@mail.gmail.com>
Subject: Re: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap
From: Sameer Joshi <sameer.joshi@serenesoftware.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364187e3a4873f048ce0a357
X-Virus-Checked: Checked by ClamAV on apache.org

--0016364187e3a4873f048ce0a357
Content-Type: text/plain; charset=ISO-8859-1

Thanks Sonal,
I tried it, and seemed to make some progress as it hung instead of giving
the error. How do you refer to it in the hadoop conf files? For example, in
mapred-site.xml I have it as
<name>mapred.job.tracker</name>
<value>173.1.86.194.reverse.gogrid.com:54311</value>

Am I doing this correctly?
Thanks and regards,
Sameer



On Sun, Aug 1, 2010 at 10:59 AM, Sonal Goyal <sonalgoyal4@gmail.com> wrote:

> 173.1.45.202 173.1.45.202.reverse.gogrid.com 28684_1_85017_358406
>
> Change that to whatever your ips/hostnames are.
>
> Thanks and Regards,
> Sonal
> www.meghsoft.com
> http://in.linkedin.com/in/sonalgoyal
>
>
> On Sun, Aug 1, 2010 at 7:35 AM, Sameer Joshi <
> sameer.joshi@serenesoftware.com> wrote:
>
> > Hi again Sonal,
> > Just checking, if I'm running a pseudo-cluster on one node, so the master
> > and slaves are all localhost. What may the /etc/hosts look like in this
> > case?
> >
> > Thanks,
> > Sameer
> >
> > On Sat, Jul 31, 2010 at 3:37 AM, Sameer Joshi <
> > sameer.joshi@serenesoftware.com> wrote:
> >
> > > Hi Sonal,
> > > Sorry for the late reply, I was traveling. The logs, unfortunately, did
> > not
> > > yield too much information. I'll try some more configurations with
> > etc/hosts
> > > edits, and see if I can't get it to behave.
> > > Thanks so much again for your help.
> > > Sameer
> > >
> > >
> > > On Wed, Jul 28, 2010 at 1:20 PM, Sonal Goyal <sonalgoyal4@gmail.com
> > >wrote:
> > >
> > >> Hi Sameer,
> > >>
> > >> Do you see all processes up? Can you check the logs and see what is
> > going
> > >> wrong? I suspect you will need to make the entries in /etc/hosts
> > anyways.
> > >> Somewhere in hadoop code, the ip  hostname mapping is done, I forget
> the
> > >> exact location.
> > >>
> > >> Thanks and Regards,
> > >> Sonal
> > >> www.meghsoft.com
> > >> http://in.linkedin.com/in/sonalgoyal
> > >>
> > >>
> > >> On Wed, Jul 28, 2010 at 10:38 PM, Sameer Joshi <
> > >> sameer.joshi@serenesoftware.com> wrote:
> > >>
> > >> > Hi Sonal,
> > >> > Thank you for replying.
> > >> >
> > >> > I'm trying to run it as single node, so I had localhost in all the
> > conf
> > >> > files and in /etc/hosts. I tried replacing it with 127.0.0.1 in all
> > >> Hadoop
> > >> > conf files (read to do that on some forum), but with no change.
> > >> >
> > >> > I figured I didn't need hostname since I was trying to run it
> locally,
> > >> but
> > >> > still tried replacing localhost with 173.1.86.194. However, as
> > expected,
> > >> it
> > >> > could not connect to server.
> > >> >
> > >> > Any thoughts?
> > >> >
> > >> > Thanks for your help, I'm new at this and feel much obliged.
> > >> >
> > >> > Regards,
> > >> > Sameer
> > >> >
> > >> > On Wed, Jul 28, 2010 at 12:12 AM, Sonal Goyal <
> sonalgoyal4@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > Hi Sameer,
> > >> > >
> > >> > > For GoGrid, you will have to configure /etc/hosts with the ip and
> > >> > hostname
> > >> > > for master and slaves. If you run hostname and nslookup -sil, you
> > will
> > >> > get
> > >> > > the details for your machines. Make sure /etc/hosts has an entry
> > like:
> > >> > >
> > >> > > 173.1.45.202 173.1.45.202.reverse.gogrid.com 28684_1_85017_358406
> > >> > >
> > >> > > Thanks and Regards,
> > >> > > Sonal
> > >> > > www.meghsoft.com
> > >> > > http://in.linkedin.com/in/sonalgoyal
> > >> > >
> > >> > >
> > >> > > On Wed, Jul 28, 2010 at 6:42 AM, Sameer Joshi <
> > >> > > sameer.joshi@serenesoftware.com> wrote:
> > >> > >
> > >> > > > I installed Java and Hadoop on a GoGrid cloud server using Red
> Hat
> > >> > > > Enterprise Linux Server release 5.1 (Tikanga). Hadoop installed
> > fine
> > >> > and
> > >> > > > starts fine, however I get an error
> > (java.lang.NullPointerException
> > >> at
> > >> > > > java.util.concurrent.ConcurrentHashMap) while running the Hadoop
> > >> > > wordcount
> > >> > > > example. My guess was that this was a localhost or IPv6 issue.
> > >> > > >
> > >> > > > * I have tested replacing 'localhost' with both the local IP,
> and
> > >> > server
> > >> > > IP
> > >> > > > addresses (when out of options) in Hadoop conf
> > >> > > > * I have disabled IPv6 both in sysctl.conf and hadoop-env.sh
> > (former
> > >> > > > followed by a server restart)
> > >> > > >
> > >> > > > Any thoughts?
> > >> > > > Thank you.
> > >> > > >
> > >> > > >
> > >> > > > The output is given below
> > >> > > >
> > >> > > > # bin/hadoop jar hadoop-0.20.2-examples.jar wordcount datasets
> > >> > tests/out7
> > >> > > > 10/07/27 05:44:59 INFO input.FileInputFormat: Total input paths
> to
> > >> > > process
> > >> > > > :
> > >> > > > 1
> > >> > > > 10/07/27 05:44:59 INFO mapred.JobClient: Running job:
> > >> > > job_201007270544_0001
> > >> > > > 10/07/27 05:45:00 INFO mapred.JobClient: map 0% reduce 0%
> > >> > > > 10/07/27 05:45:12 INFO mapred.JobClient: map 100% reduce 0%
> > >> > > > 10/07/27 05:45:17 INFO mapred.JobClient: Task Id :
> > >> > > > attempt_201007270544_0001_r_000000_0, Status : FAILED
> > >> > > >
> > >> > > > Error: java.lang.NullPointerException
> > >> > > > at java.util.concurrent.ConcurrentHashMap.get(Unknown Source)
> > >> > > > at
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
> > >> > > >
> > >> > > > at
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
> > >> > > >
> > >> > > >
> > >> > > > 10/07/27 05:45:24 INFO mapred.JobClient: Task Id :
> > >> > > > attempt_201007270544_0001_r_000000_1, Status : FAILED
> > >> > > > Error: java.lang.NullPointerException
> > >> > > > at java.util.concurrent.ConcurrentHashMap.get(Unknown Source)
> > >> > > > at
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
> > >> > > >
> > >> > > > at
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
> > >> > > >
> > >> > > >
> > >> > > > 10/07/27 05:45:30 INFO mapred.JobClient: Task Id :
> > >> > > > attempt_201007270544_0001_r_000000_2, Status : FAILED
> > >> > > > Error: java.lang.NullPointerException
> > >> > > > at java.util.concurrent.ConcurrentHashMap.get(Unknown Source)
> > >> > > > at
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
> > >> > > >
> > >> > > > at
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
> > >> > > >
> > >> > > >
> > >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Job complete:
> > >> > > > job_201007270544_0001
> > >> > > >
> > >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Counters: 12
> > >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Job Counters
> > >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Launched reduce tasks=4
> > >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Launched map tasks=1
> > >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Data-local map tasks=1
> > >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Failed reduce tasks=1
> > >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: FileSystemCounters
> > >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: HDFS_BYTES_READ=15319
> > >> > > > 10/07/27 05:45:39 INFO mapred.JobClient:
> FILE_BYTES_WRITTEN=12847
> > >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Map-Reduce Framework
> > >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Combine output
> > records=934
> > >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Map input records=149
> > >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Spilled Records=934
> > >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Map output bytes=25346
> > >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Combine input
> > records=2541
> > >> > > > 10/07/27 05:45:39 INFO mapred.JobClient: Map output records=2541
> > >> > > >
> > >> > > > --
> > >> > > > Dr. Sameer Joshi, Ph.D.
> > >> > > > Senior computer scientist,
> > >> > > > Serene Software.
> > >> > > >
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Dr. Sameer Joshi, Ph.D.
> > >> > Senior computer scientist,
> > >> > Serene Software.
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Dr. Sameer Joshi, Ph.D.
> > > Senior computer scientist,
> > > Serene Software.
> > >
> > >
> >
> >
> > --
> > Dr. Sameer Joshi, Ph.D.
> > Senior computer scientist,
> > Serene Software.
> >
>



-- 
Dr. Sameer Joshi, Ph.D.
Senior computer scientist,
Serene Software.

--0016364187e3a4873f048ce0a357--

From general-return-1885-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 03 07:35:53 2010
Return-Path: <general-return-1885-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67136 invoked from network); 3 Aug 2010 07:35:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Aug 2010 07:35:53 -0000
Received: (qmail 46219 invoked by uid 500); 3 Aug 2010 07:35:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45598 invoked by uid 500); 3 Aug 2010 07:35:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45586 invoked by uid 99); 3 Aug 2010 07:35:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Aug 2010 07:35:47 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [203.209.230.21] (HELO n4c.bullet.cnb.yahoo.com) (203.209.230.21)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 03 Aug 2010 07:35:38 +0000
Received: from [203.209.230.74] by n4.bullet.cnb.yahoo.com with NNFMP; 03 Aug 2010 07:35:12 -0000
Received: from [202.165.102.47] by t4.bullet.cnb.yahoo.com with NNFMP; 03 Aug 2010 07:35:12 -0000
Received: from [127.0.0.1] by omp101.mail.cnb.yahoo.com with NNFMP; 03 Aug 2010 07:35:12 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 282670.94265.bm@omp101.mail.cnb.yahoo.com
Received: (qmail 75274 invoked by uid 60001); 3 Aug 2010 07:35:09 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com.cn; s=s1024; t=1280820909; bh=xl535HCVrioMk8ix98bBvf3R/zYznJL/AdV6+ATa9S4=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=re6wQuqRUY11C11Q5HwATzvdV+NFAbeeBbl0wqyc555hJUDnaF1I6QWdTBdXEGRxkZxJ0XrDCnQpvUyDQ83yiklQxpF4gek9OMeJPerBHyInOgRmdfF8LI0BfOK2nh1c+HXNCb+qqd+xXcTds8BmuCBKU2/axmRk0vz0Z6wZ/8g=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com.cn;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=HwApAoR18SjXR8RshRK29zdY1/gBJBGMQXLPCvkWI8m/1IEgLre7VAM1H9/bh/9tlGRvkwzkN7YolJU9Akd0p6/njEVRZS5GhKdp90aZ16zjwLLITS/B2DYFXTgI/rO9z5UCG6brkdFx3NQ+Mjz63m9ZFxn8qizJcqswDlzMn3o=;
Message-ID: <158791.69815.qm@web15201.mail.cnb.yahoo.com>
X-YMail-OSG: kGkf5RwVM1njI38iJpt2H7HEn0QfFwaXhqTNCohNMWyf4zj
 eO8rqlzuU7Pf.v6x_6_PkPruGO5i2eMEDkyTjEV6YUAMdxr.za6o6ecJSi6b
 46zawTKpYAFLmR_6Yhdfi6yHT815zRiCfczqgAToVEevOgBGmaWbQ3stqK24
 WfSExibp70H1018hyqOcux5WJAPkx3yZeKTVUuTtvceyUvYW1.pIF90IN0MB
 zy4e.fj0uG5O1X9KOWmrLTdXDDuIFFe6LUtl2NH8pJI0r7vxm88qj3pMLoZ9
 hNcqv0Iu.lwzwVMQ-
Received: from [202.118.67.200] by web15201.mail.cnb.yahoo.com via HTTP; Tue, 03 Aug 2010 15:35:08 CST
X-Mailer: YahooMailClassic/11.3.2 YahooMailWebService/0.8.105.279950
Date: Tue, 3 Aug 2010 15:35:08 +0800 (CST)
From: Dennis <arsenepark@yahoo.com.cn>
Subject: strange host ip
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-532221473-1280820908=:69815"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-532221473-1280820908=:69815
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Hi, all,

I am trying to install "Pseudo-Distributed" "hadoop 0.20.2" on my Ubuntu 9.=
10. When I tried to "bin/hadoop namenode -format", it told me "STARTUP_MSG:=
=A0=A0 host =3D cloud/192.168.0.199", while my ip was 192.168.3.167, and I =
even could not ping 192.168.0.199. The following is the CLI. And I could no=
t finish the installation.
Any idea?
Thanks
Dennis

cloud@cloud:~/HadoopInstall/hadoop-0.20.2$ bin/hadoop namenode -format
10/08/03 15:23:03 INFO namenode.NameNode: STARTUP_MSG:=20
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:=A0=A0 host =3D cloud/192.168.0.199
STARTUP_MSG:=A0=A0 args =3D [-format]
STARTUP_MSG:=A0=A0 version =3D 0.20.2
STARTUP_MSG:=A0=A0 build =3D https://svn.apache.org/repos/asf/hadoop/common=
/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:=
34 UTC 2010
************************************************************/
10/08/03 15:23:03 INFO namenode.FSNamesystem: fsOwner=3Dcloud,cloud,adm,dia=
lout,cdrom,plugdev,lpadmin,admin,sambashare
10/08/03 15:23:03 INFO namenode.FSNamesystem: supergroup=3Dsupergroup
10/08/03 15:23:03 INFO namenode.FSNamesystem: isPermissionEnabled=3Dtrue
10/08/03 15:23:03 INFO common.Storage: Image file of size 95 saved in 0 sec=
onds.
10/08/03 15:23:04 INFO common.Storage: Storage directory /home/cloud/Hadoop=
Install/tmp/dfs/name has been successfully formatted.
10/08/03 15:23:04 INFO namenode.NameNode: SHUTDOWN_MSG:=20
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at cloud/192.168.0.199
************************************************************/
cloud@cloud:~/HadoopInstall/hadoop-0.20.2$ ping 192.168.0.199
PING 192.168.0.199 (192.168.0.199) 56(84) bytes of data.
>From 192.168.3.254 icmp_seq=3D7 Destination Host Unreachable
>From 192.168.3.254 icmp_seq=3D8 Destination Host Unreachable
^C
--- 192.168.0.199 ping statistics ---
8 packets transmitted, 0 received, +2 errors, 100% packet loss, time 7038ms

cloud@cloud:~/HadoopInstall/hadoop-0.20.2$
=0A=0A=0A      
--0-532221473-1280820908=:69815--


From general-return-1886-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 03 07:39:22 2010
Return-Path: <general-return-1886-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68700 invoked from network); 3 Aug 2010 07:39:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Aug 2010 07:39:22 -0000
Received: (qmail 50196 invoked by uid 500); 3 Aug 2010 07:39:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49886 invoked by uid 500); 3 Aug 2010 07:39:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49849 invoked by uid 99); 3 Aug 2010 07:39:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Aug 2010 07:39:16 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Aug 2010 07:39:08 +0000
Received: by iwn37 with SMTP id 37so2406125iwn.35
        for <general@hadoop.apache.org>; Tue, 03 Aug 2010 00:38:47 -0700 (PDT)
Received: by 10.231.182.9 with SMTP id ca9mr8483233ibb.120.1280821127216; Tue, 
	03 Aug 2010 00:38:47 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.182.79 with HTTP; Tue, 3 Aug 2010 00:38:17 -0700 (PDT)
X-Originating-IP: [80.179.214.229]
In-Reply-To: <158791.69815.qm@web15201.mail.cnb.yahoo.com>
References: <158791.69815.qm@web15201.mail.cnb.yahoo.com>
From: Maxim Veksler <maxim@vekslers.org>
Date: Tue, 3 Aug 2010 10:38:17 +0300
Message-ID: <AANLkTinh-vitcOqAX+nPtf_nb1606NWKA3bbyitfm6k=@mail.gmail.com>
Subject: Re: strange host ip
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364ee28a4efd9a048ce668dd
X-Virus-Checked: Checked by ClamAV on apache.org

--0016364ee28a4efd9a048ce668dd
Content-Type: text/plain; charset=ISO-8859-1

Can it be that your /etc/hosts is mis-configured ?

Please post this file to the list.

On Tue, Aug 3, 2010 at 10:35 AM, Dennis <arsenepark@yahoo.com.cn> wrote:

> Hi, all,
>
> I am trying to install "Pseudo-Distributed" "hadoop 0.20.2" on my Ubuntu
> 9.10. When I tried to "bin/hadoop namenode -format", it told me
> "STARTUP_MSG:   host = cloud/192.168.0.199", while my ip was
> 192.168.3.167, and I even could not ping 192.168.0.199. The following is the
> CLI. And I could not finish the installation.
> Any idea?
> Thanks
> Dennis
>
> cloud@cloud:~/HadoopInstall/hadoop-0.20.2$ bin/hadoop namenode -format
> 10/08/03 15:23:03 INFO namenode.NameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG:   host = cloud/192.168.0.199
> STARTUP_MSG:   args = [-format]
> STARTUP_MSG:   version = 0.20.2
> STARTUP_MSG:   build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 10/08/03 15:23:03 INFO namenode.FSNamesystem:
> fsOwner=cloud,cloud,adm,dialout,cdrom,plugdev,lpadmin,admin,sambashare
> 10/08/03 15:23:03 INFO namenode.FSNamesystem: supergroup=supergroup
> 10/08/03 15:23:03 INFO namenode.FSNamesystem: isPermissionEnabled=true
> 10/08/03 15:23:03 INFO common.Storage: Image file of size 95 saved in 0
> seconds.
> 10/08/03 15:23:04 INFO common.Storage: Storage directory
> /home/cloud/HadoopInstall/tmp/dfs/name has been successfully formatted.
> 10/08/03 15:23:04 INFO namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at cloud/192.168.0.199
> ************************************************************/
> cloud@cloud:~/HadoopInstall/hadoop-0.20.2$ ping 192.168.0.199
> PING 192.168.0.199 (192.168.0.199) 56(84) bytes of data.
> From 192.168.3.254 icmp_seq=7 Destination Host Unreachable
> From 192.168.3.254 icmp_seq=8 Destination Host Unreachable
> ^C
> --- 192.168.0.199 ping statistics ---
> 8 packets transmitted, 0 received, +2 errors, 100% packet loss, time 7038ms
>
> cloud@cloud:~/HadoopInstall/hadoop-0.20.2$
>
>
>
>

--0016364ee28a4efd9a048ce668dd--

From general-return-1887-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 03 07:48:32 2010
Return-Path: <general-return-1887-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 71362 invoked from network); 3 Aug 2010 07:48:31 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Aug 2010 07:48:31 -0000
Received: (qmail 56747 invoked by uid 500); 3 Aug 2010 07:48:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56478 invoked by uid 500); 3 Aug 2010 07:48:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56470 invoked by uid 99); 3 Aug 2010 07:48:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Aug 2010 07:48:27 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [203.209.230.24] (HELO n4f.bullet.cnb.yahoo.com) (203.209.230.24)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 03 Aug 2010 07:48:18 +0000
Received: from [202.43.217.36] by n4.bullet.cnb.yahoo.com with NNFMP; 03 Aug 2010 07:47:56 -0000
Received: from [203.209.230.75] by t1.bullet.cnmail.cnb.yahoo.com with NNFMP; 03 Aug 2010 07:47:55 -0000
Received: from [127.0.0.1] by omp104.mail.cnb.yahoo.com with NNFMP; 03 Aug 2010 07:47:55 -0000
X-Yahoo-Newman-Property: ymail-5
X-Yahoo-Newman-Id: 879276.60215.bm@omp104.mail.cnb.yahoo.com
Received: (qmail 79310 invoked by uid 60001); 3 Aug 2010 07:47:55 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com.cn; s=s1024; t=1280821675; bh=k+G1whznnx+qnfKEsi2Hncr5nVJ4axfGn6Us5dphJ5E=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=1OCxDhk/wYmIKLAQFM7r3JaYzru7qseT0ZU3tKWEVSUIgTupqOZD/UYNFv11Xqlv/tS1sWBv96zaGW+yrCh7LyMIw2LGFf4hSX4rRr/uZew7TvEwrUe3318deHYZoHd/ZiaMaTW7GO/9Y8vTfikPEXG8sL+Yu2ryDpeDiK6/lTI=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com.cn;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=D5kCDHq2oKFsKZag8aQ0bHDAyUVZQ073S8Mz2J/+7pRBI5CIHcL+1HLrq6dOlUc6GJv+WVjnBpei6799Jg3zWf0GKbytqDDrYswEbuXXgyOE97nysPSKhRoiRFHruhDehkTmtgOPZzXblxhIArIv0mdFqU7k5DWT9mIf+r4jUak=;
Message-ID: <774853.79040.qm@web15203.mail.cnb.yahoo.com>
X-YMail-OSG: NyK5Ee4VM1l1.zDgfmG4ZFode4Tn8p_CoGzG5PaaroMCxwG
 XSPJOnK0Ct8vB.sta4lSw4ONMs2X84qZ0a6fKQ8EH5H3voIg3_oGzBctBsKN
 E12M9djeIH63sOY4.iYDSu57UjS0FlmalBmunahktTj88gnoS8OweTR5Imxl
 aeXHv0l65s5wzw8zrqpu4_BkWjn5CIwHZKIF9jMetukPQ3V1dSElr8IhnMb7
 gvg7jNEM4E8nFaHjy45uBGw6CRNfWc.q08OjhKxjG9sz9eTDEwTXnMCRxONU
 dca1TcRwRlyeX6Z4-
Received: from [202.118.67.200] by web15203.mail.cnb.yahoo.com via HTTP; Tue, 03 Aug 2010 15:47:55 CST
X-Mailer: YahooMailClassic/11.3.2 YahooMailWebService/0.8.105.279950
Date: Tue, 3 Aug 2010 15:47:55 +0800 (CST)
From: Dennis <arsenepark@yahoo.com.cn>
Subject: Re: strange host ip
To: general@hadoop.apache.org
In-Reply-To: <AANLkTinh-vitcOqAX+nPtf_nb1606NWKA3bbyitfm6k=@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-138240386-1280821675=:79040"

--0-138240386-1280821675=:79040
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Thanks, Maxim,

You right. My studio is being decorated, and I moved to another room. But I=
 forgot the /etc/hosts.

Many thanks.

Dennis

--- On Tue, 8/3/10, Maxim Veksler <maxim@vekslers.org> wrote:

From: Maxim Veksler <maxim@vekslers.org>
Subject: Re: strange host ip
To: general@hadoop.apache.org
Date: Tuesday, August 3, 2010, 3:38 PM

Can it be that your /etc/hosts is mis-configured ?

Please post this file to the list.

On Tue, Aug 3, 2010 at 10:35 AM, Dennis <arsenepark@yahoo.com.cn> wrote:

> Hi, all,
>
> I am trying to install "Pseudo-Distributed" "hadoop 0.20.2" on my Ubuntu
> 9.10. When I tried to "bin/hadoop namenode -format", it told me
> "STARTUP_MSG:=A0=A0=A0host =3D cloud/192.168.0.199", while my ip was
> 192.168.3.167, and I even could not ping 192.168.0.199. The following is =
the
> CLI. And I could not finish the installation.
> Any idea?
> Thanks
> Dennis
>
> cloud@cloud:~/HadoopInstall/hadoop-0.20.2$ bin/hadoop namenode -format
> 10/08/03 15:23:03 INFO namenode.NameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG:=A0=A0=A0host =3D cloud/192.168.0.199
> STARTUP_MSG:=A0=A0=A0args =3D [-format]
> STARTUP_MSG:=A0=A0=A0version =3D 0.20.2
> STARTUP_MSG:=A0=A0=A0build =3D
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 10/08/03 15:23:03 INFO namenode.FSNamesystem:
> fsOwner=3Dcloud,cloud,adm,dialout,cdrom,plugdev,lpadmin,admin,sambashare
> 10/08/03 15:23:03 INFO namenode.FSNamesystem: supergroup=3Dsupergroup
> 10/08/03 15:23:03 INFO namenode.FSNamesystem: isPermissionEnabled=3Dtrue
> 10/08/03 15:23:03 INFO common.Storage: Image file of size 95 saved in 0
> seconds.
> 10/08/03 15:23:04 INFO common.Storage: Storage directory
> /home/cloud/HadoopInstall/tmp/dfs/name has been successfully formatted.
> 10/08/03 15:23:04 INFO namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at cloud/192.168.0.199
> ************************************************************/
> cloud@cloud:~/HadoopInstall/hadoop-0.20.2$ ping 192.168.0.199
> PING 192.168.0.199 (192.168.0.199) 56(84) bytes of data.
> From 192.168.3.254 icmp_seq=3D7 Destination Host Unreachable
> From 192.168.3.254 icmp_seq=3D8 Destination Host Unreachable
> ^C
> --- 192.168.0.199 ping statistics ---
> 8 packets transmitted, 0 received, +2 errors, 100% packet loss, time 7038=
ms
>
> cloud@cloud:~/HadoopInstall/hadoop-0.20.2$
>
>
>
>
=0A=0A=0A      
--0-138240386-1280821675=:79040--


From general-return-1888-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 03 11:38:55 2010
Return-Path: <general-return-1888-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 62266 invoked from network); 3 Aug 2010 11:38:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Aug 2010 11:38:55 -0000
Received: (qmail 17305 invoked by uid 500); 3 Aug 2010 11:38:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16780 invoked by uid 500); 3 Aug 2010 11:38:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16763 invoked by uid 99); 3 Aug 2010 11:38:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Aug 2010 11:38:49 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of sarah.kho@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Aug 2010 11:38:41 +0000
Received: by wyb35 with SMTP id 35so787164wyb.35
        for <general@hadoop.apache.org>; Tue, 03 Aug 2010 04:38:21 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=PhOy1ncJPbqyocmbXjwIj9tbSI/PrCKyye44/vsoB1g=;
        b=ttPhFVYL4gFGd9WZ79oeZgbCRqkHYeLPlLvw079bx1CttfKKRSPXwsbzEPCweUG4+0
         DQwunIrNizxZXo/U1dYl6Qu3h8J33orcI1uJzUGjPpHg81loT6k5Sjp31zinSuyM/vFQ
         3VgJ/vUUdhbbTKaYjZ/0ylt6TAa2l1T2EvWXg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=OdRXHUDtiR3jfMTiVa46KLeDx2mpXijYKeG9nVOAAr6HzGZpW2AOs6u4n7B5BQo87w
         qkBDiZuFgnGkEbth+bXS+uCZMYU0csZK43kTT2f9Zt98ywjRZRHM5RC0zIbBuagEeBmw
         so3UpXFA6GvBxBlUljVg4++Z4C5nsckgK1Lrc=
MIME-Version: 1.0
Received: by 10.216.0.68 with SMTP id 46mr662301wea.2.1280835501409; Tue, 03 
	Aug 2010 04:38:21 -0700 (PDT)
Received: by 10.216.61.75 with HTTP; Tue, 3 Aug 2010 04:38:21 -0700 (PDT)
Date: Tue, 3 Aug 2010 16:08:21 +0430
Message-ID: <AANLkTikPKNZDsr6bPT625X+wqzXDC+F-isa+Q7HbS-8w@mail.gmail.com>
Subject: How to deploy Hadoop daemons web applications?
From: Sarah kho <sarah.kho@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f6c90413ccf4048ce9c10c
X-Virus-Checked: Checked by ClamAV on apache.org

--001485f6c90413ccf4048ce9c10c
Content-Type: text/plain; charset=ISO-8859-1

Hi,

Is there nay reference about how we can deploy hadoop daemons web
applications?

Thanks.

--001485f6c90413ccf4048ce9c10c--

From general-return-1889-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 04 03:03:16 2010
Return-Path: <general-return-1889-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43949 invoked from network); 4 Aug 2010 03:03:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Aug 2010 03:03:16 -0000
Received: (qmail 1798 invoked by uid 500); 4 Aug 2010 03:03:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1464 invoked by uid 500); 4 Aug 2010 03:03:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1456 invoked by uid 99); 4 Aug 2010 03:03:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 03:03:11 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 03:03:05 +0000
Received: by qwd7 with SMTP id 7so3933750qwd.35
        for <general@hadoop.apache.org>; Tue, 03 Aug 2010 20:02:44 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=FWpiHvRN5XaUj/rY+PiEn/fx07lD1WMGeDbRkqUsMMM=;
        b=xs4DeOobuEF4pj0mQLcimMrhtLr8qf8OyHOXNgzWnP0WIQ6XRJczt3gWdCiJ4OhBXh
         qoZG1VmIl6EpzJgiXwfAK/2Lut8wC8xsTjOszHL3Uo5l2JS8kDQ1I8va4RA/MMyik3WP
         x9soo1CBaKBLnixDe/FIWzSVHCW3wzwMqqGu0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=a7ygaziIMwmDA7fOxQEFdRvx5kNIchx/wiLEa4lPzZWIf87zosdQhCrWIT87sFkLvI
         B5Esj6EeqCEAH5UBInDbTb75JjFsWUeVV9bpqseIr8IVUwaptEdhd7eA7GAcke6r0cRE
         d/iTNB/zHFewEHxILMQKdtkcQVUQCVruOM824=
MIME-Version: 1.0
Received: by 10.229.52.5 with SMTP id f5mr1709923qcg.254.1280890964552; Tue, 
	03 Aug 2010 20:02:44 -0700 (PDT)
Received: by 10.229.48.77 with HTTP; Tue, 3 Aug 2010 20:02:44 -0700 (PDT)
In-Reply-To: <AANLkTikoAofAVM6Wi2Gj1nh+O4v2RL9VLqbocQMmVV1d@mail.gmail.com>
References: <AANLkTik5koLPa0yNxtc+ta4nJ__Oym4ATq5xn=sRgz29@mail.gmail.com>
	<AANLkTi=yMdJuOxWAogr5uw0iYbp2B88iujFO-0_M=8mr@mail.gmail.com>
	<17C47A26-3B02-4BEA-B862-7849391A6D74@yahoo-inc.com>
	<AANLkTikoAofAVM6Wi2Gj1nh+O4v2RL9VLqbocQMmVV1d@mail.gmail.com>
Date: Tue, 3 Aug 2010 20:02:44 -0700
Message-ID: <AANLkTinD2_70JChKVy9wzYjKwJLGDQoWxXgymnmYysby@mail.gmail.com>
Subject: Re: Data World Record Falls as Computer Scientists Break Terabyte 
	Sort Barrier
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364edcccf02c61048cf6aa16
X-Virus-Checked: Checked by ClamAV on apache.org

--0016364edcccf02c61048cf6aa16
Content-Type: text/plain; charset=ISO-8859-1

Maybe this would give us a little more incentive to resolve HDFS-347

Cheers

On Mon, Aug 2, 2010 at 1:50 PM, Abhishek Verma <vermaabhishekp@gmail.com>wrote:

> It shows how further behind Hadoop is in terms of performance. Are there
> people working on finding the bottlenecks and making it more efficient? Are
> there any JIRA issues related to this?
>
> -Abhishek.
>
> On Mon, Aug 2, 2010 at 11:47 AM, Arun C Murthy <acm@yahoo-inc.com> wrote:
>
> > The UCSD results are very impressive, especially given their hardware
> > budget.
> >
> > I may be wrong, but I'm pretty sure there were no Hadoop based entries
> this
> > year - I know we at Yahoo! didn't enter.
> >
> > Couple of points:
> > # The Indy category is a benchmark to sort fixed length records, not a
> > _general_ sort benchmark i.e. Daytona.
> > # Our _best_ result missed the deadline by a whisker last year, but we
> > eventually did 100Tb sort in 95 mins and a 1000TB (1PB) in 975 mins
> (16.25
> > hrs) - which worked out to be just over 1.0 TB/min, which was nearly
> twice
> > as fast as the record attributed to us. (
> >
> http://developer.yahoo.net/blogs/hadoop/2009/05/hadoop_sorts_a_petabyte_in_162.html
> > )
> >
> > Arun
> >
> >
> > On Aug 2, 2010, at 10:34 AM, Abhishek Verma wrote:
> >
> >  Hi Maxim,
> >>
> >> Hadoop was not involved. You can find more details here :
> >> http://sortbenchmark.org/tritonsort_2010_May_15.pdf
> >> and all the records and their information here :
> >> http://sortbenchmark.org/
> >>
> >> <http://sortbenchmark.org/>-Abhishek.
> >>
> >> On Mon, Aug 2, 2010 at 9:52 AM, Maxim Veksler <maxim@vekslers.org>
> wrote:
> >>
> >>  Hi,
> >>>
> >>> Anyone knows if Hadoop is involved? And if so what is the configuration
> >>> for
> >>> such cluster?
> >>>
> >>> http://ucsdnews.ucsd.edu/newsrel/science/07-27DataWorld.asp
> >>>
> >>>
> >>> Thank you,
> >>> Maxim.
> >>>
> >>>
> >
>

--0016364edcccf02c61048cf6aa16--

From general-return-1890-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 05 03:58:55 2010
Return-Path: <general-return-1890-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29710 invoked from network); 5 Aug 2010 03:58:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Aug 2010 03:58:55 -0000
Received: (qmail 73437 invoked by uid 500); 5 Aug 2010 03:58:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72978 invoked by uid 500); 5 Aug 2010 03:58:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72964 invoked by uid 99); 5 Aug 2010 03:58:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 03:58:49 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of xuwenhao2008@gmail.com designates 209.85.213.176 as permitted sender)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 03:58:42 +0000
Received: by yxm34 with SMTP id 34so3587569yxm.35
        for <general@hadoop.apache.org>; Wed, 04 Aug 2010 20:58:21 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:from:date
         :message-id:subject:to:content-type;
        bh=jr4vyj5U2CgU7vpKKbPVOLbkFq9Er3pGQk/CDd62M08=;
        b=AobjiHksTFt6UoyiJRij2FbsZtb3HUcvOkeDr4toSGYlxZqSlleSsfaPunCo6f6ELv
         y/NVk/iAZ1BD98q8ck8IY7nEbGCK3tJQ+gyd7casd1TdtaG8LuaVMqAtvGTIukWQrO1L
         n0AB9MyQNsmWU0IqL1qx0jinBcG6+L5D8gHZQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type;
        b=Mh4A6RLVzDnsszExKDnuNBxFhyI0xejYcs1RfwrgOE92PXEOnybXcbsgukt+KB1GUR
         rxlZ0d9xqQMrwbQMweKeXef+ubO8ULRmxYGPM/Hj0tfpC5L2qqViho62jTQxUQBqHxk+
         HV+6bSErqn/17xNtgh4x3OlA+AWUu0GZpt/i8=
Received: by 10.231.35.77 with SMTP id o13mr11406214ibd.92.1280980701592; Wed, 
	04 Aug 2010 20:58:21 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.158.137 with HTTP; Wed, 4 Aug 2010 20:58:01 -0700 (PDT)
From: Wenhao Xu <xuwenhao2008@gmail.com>
Date: Wed, 4 Aug 2010 20:58:01 -0700
Message-ID: <AANLkTimmxXmE8_8meO4oOg=eXQrAz_U6y_zQAukoaufT@mail.gmail.com>
Subject: is it possible to use other file system/database rather than HDFS
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000325572daaaebe40048d0b8f1a

--000325572daaaebe40048d0b8f1a
Content-Type: text/plain; charset=ISO-8859-1

Hi all,
   Is it possible to configure Hadoop to use other file system or database
rather than HDFS? Thanks!

regards,
W.

-- 
~_~

--000325572daaaebe40048d0b8f1a--

From general-return-1891-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 05 04:11:34 2010
Return-Path: <general-return-1891-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37206 invoked from network); 5 Aug 2010 04:11:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Aug 2010 04:11:34 -0000
Received: (qmail 81277 invoked by uid 500); 5 Aug 2010 04:11:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80992 invoked by uid 500); 5 Aug 2010 04:11:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80984 invoked by uid 99); 5 Aug 2010 04:11:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 04:11:29 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of qwertymaniac@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 04:11:22 +0000
Received: by wwb39 with SMTP id 39so3961304wwb.29
        for <general@hadoop.apache.org>; Wed, 04 Aug 2010 21:11:02 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=38rfquImyYQpE/UroIyuIULmg+N7n7JCIxQWgc82edo=;
        b=ids50vKXOyymsGwI+NVilSgPxO3g0/WJNM7hb5dcqimUvosm1Z48e3qiBcrUIFG6Ni
         mc33Sb9yytJLrpSucqrO7Npwdo2HgX0O4bzZdWE2jf+UTPl64SW5wF/eV+9DXDCOiFSK
         yGR+rrrX641i3iq47GycTRlH+51Zc5zJo1a6k=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=Ya82u3G2WU6+97t8kgsBx5zzGKa9eue8dAHhJQwG6rFzuauNg9cbKLVIdl7FO7QiQy
         gngQ2izmzJ0gwRqOrsi2Jstf6NBYK0aYoz4irY8prBQDw3dxxOPUMNrb+XdeRtvt2GMT
         eG8TrPDEWgtuGgRS3XUc8U1aZniR5xvXLFuxA=
Received: by 10.216.50.18 with SMTP id y18mr2893007web.113.1280981462211; Wed, 
	04 Aug 2010 21:11:02 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.236.197 with HTTP; Wed, 4 Aug 2010 21:10:42 -0700 (PDT)
In-Reply-To: <AANLkTimmxXmE8_8meO4oOg=eXQrAz_U6y_zQAukoaufT@mail.gmail.com>
References: <AANLkTimmxXmE8_8meO4oOg=eXQrAz_U6y_zQAukoaufT@mail.gmail.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Thu, 5 Aug 2010 09:40:42 +0530
Message-ID: <AANLkTimEszOS+wj5_yxO+oTV+u3YwJn8Y3hrNNp+ygvS@mail.gmail.com>
Subject: Re: is it possible to use other file system/database rather than HDFS
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Filesystems - Yes. You can use NFS/Local FSes for MR. Check this link
for an example of a non-HDFS FS in use:
http://old.nabble.com/Hadoop-over-Lustre--td19092864.html (Solution's
down the thread)

DataBases - No. Not directly AFAIK. However you can write custom
sources (and perhaps sinks) for it for use within MR jobs.

On Thu, Aug 5, 2010 at 9:28 AM, Wenhao Xu <xuwenhao2008@gmail.com> wrote:
> Hi all,
> =A0 Is it possible to configure Hadoop to use other file system or databa=
se
> rather than HDFS? Thanks!
>
> regards,
> W.
>
> --
> ~_~
>



--=20
Harsh J
www.harshj.com

From general-return-1892-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 05 12:57:21 2010
Return-Path: <general-return-1892-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 52252 invoked from network); 5 Aug 2010 12:57:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Aug 2010 12:57:21 -0000
Received: (qmail 17017 invoked by uid 500); 5 Aug 2010 12:57:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16600 invoked by uid 500); 5 Aug 2010 12:57:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16589 invoked by uid 99); 5 Aug 2010 12:57:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 12:57:15 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of marcoscba@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 12:57:08 +0000
Received: by wwb39 with SMTP id 39so4419309wwb.29
        for <general@hadoop.apache.org>; Thu, 05 Aug 2010 05:56:48 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=wp3oJ0GENUT0RCs4RA6EiQZxBxqFD7Wmcz6GrPGEr/0=;
        b=PQmDzkCn2zQsuoF40NUVjFapVtFdZcKB4RMBU7EYp6rlvBXddTr3Q4L9OE1G5WFAWU
         0OBOL3IR30LYWQdulLI6UI8aGYXu97HnDOBt7VhexjF3IzVZJl8zGVEIom3tc097RkNP
         Ex6tXlkUX32/YyFpNFBj+MULETsxnUdNb8DhY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=u7C3Ms207xMYdBFDuYrHLJuSzUsif/4fHxx2sXV6zsM5PSzLwzZ2EB4/4nXg2KdKli
         GsAizGpMY4OLcsmsfdnhRpxzsTwxvmA1bwnjcWt/DsLG+P53a/9ERRQMjUChofre6CyT
         JtOFmYkNHAE2X/Ly7LSKAY94X6avRSqS3QzV0=
MIME-Version: 1.0
Received: by 10.227.128.8 with SMTP id i8mr9279519wbs.91.1281012982566; Thu, 
	05 Aug 2010 05:56:22 -0700 (PDT)
Received: by 10.216.166.76 with HTTP; Thu, 5 Aug 2010 05:56:22 -0700 (PDT)
Date: Thu, 5 Aug 2010 09:56:22 -0300
Message-ID: <AANLkTimp9MDCOTdSzpPKGufnqja+bgYagG4fiReYE5qU@mail.gmail.com>
Subject: Problem: when I run a pig's script I got one reduce task
From: Marcos Pinto <marcoscba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636833ed4c74fab048d131384
X-Virus-Checked: Checked by ClamAV on apache.org

--001636833ed4c74fab048d131384
Content-Type: text/plain; charset=ISO-8859-1

Hi guys, how u doing?

I am learning how to use hadoop and I got this problem:
I set up a cluster with 5 nodes( 4 datanode n 1 namenode) and I used the
same configuration for jobtracker n tasktracker.
when I run a pig's script I get many map's( like 15) but just 1 reduce!!!!!
this kills all the parallel processing. For example.
I have a file that has 1 GB and when I run the pig's script in a cluster It
takes about 50 minutes to process. =(

So I really appreciate if someone could help with any tip. Thanks for your
time.

--001636833ed4c74fab048d131384--

From general-return-1893-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 05 13:38:42 2010
Return-Path: <general-return-1893-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66449 invoked from network); 5 Aug 2010 13:38:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Aug 2010 13:38:42 -0000
Received: (qmail 59437 invoked by uid 500); 5 Aug 2010 13:38:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58794 invoked by uid 500); 5 Aug 2010 13:38:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58777 invoked by uid 99); 5 Aug 2010 13:38:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 13:38:35 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [195.232.224.72] (HELO mailout03.vodafone.com) (195.232.224.72)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 13:38:27 +0000
Received: from mailint03 (localhost [127.0.0.1])
	by mailout03 (Postfix) with ESMTP id D073D116278
	for <general@hadoop.apache.org>; Thu,  5 Aug 2010 15:38:04 +0200 (CEST)
Received: from avoexs04.internal.vodafone.com (avoexs04.dc-ratingen.de [145.230.4.198])
	(using TLSv1 with cipher RC4-MD5 (128/128 bits))
	(No client certificate requested)
	by mailint03 (Postfix) with ESMTPS id C52441165B8
	for <general@hadoop.apache.org>; Thu,  5 Aug 2010 15:38:04 +0200 (CEST)
Received: from VF-MBX13.internal.vodafone.com ([145.230.5.24]) by avoexs04.internal.vodafone.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Thu, 5 Aug 2010 15:38:06 +0200
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: RE: Problem: when I run a pig's script I got one reduce task
Date: Thu, 5 Aug 2010 15:38:05 +0200
Message-ID: <219D8244D980254ABF28AB469AD4E98F03C38611@VF-MBX13.internal.vodafone.com>
In-Reply-To: <AANLkTimp9MDCOTdSzpPKGufnqja+bgYagG4fiReYE5qU@mail.gmail.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Problem: when I run a pig's script I got one reduce task
Thread-Index: Acs0ncAfmIYGHN6vSLiTb8Z1HC4GMQABXMTA
References: <AANLkTimp9MDCOTdSzpPKGufnqja+bgYagG4fiReYE5qU@mail.gmail.com>
From: "Gibbon, Robert, VF-Group" <Robert.Gibbon@vodafone.com>
To: <general@hadoop.apache.org>
X-OriginalArrivalTime: 05 Aug 2010 13:38:06.0063 (UTC) FILETIME=[6F3ACFF0:01CB34A3]


Use the PARALLEL clause of course!

PARALLEL n

Increase the parallelism of a job by specifying the number of reduce
tasks, n. The default value for n is 1 (one reduce task). Note the
following:

    * Parallel only affects the number of reduce tasks. Map parallelism
is determined by the input file, one map for each HDFS block.
    * If you don't specify parallel, you still get the same map
parallelism but only one reduce task.

For more information, see the Pig Cookbook.=20


-----Original Message-----
From: Marcos Pinto [mailto:marcoscba@gmail.com]=20
Sent: Donnerstag, 5. August 2010 14:56
To: general@hadoop.apache.org
Subject: Problem: when I run a pig's script I got one reduce task

Hi guys, how u doing?

I am learning how to use hadoop and I got this problem:
I set up a cluster with 5 nodes( 4 datanode n 1 namenode) and I used the
same configuration for jobtracker n tasktracker.
when I run a pig's script I get many map's( like 15) but just 1
reduce!!!!!
this kills all the parallel processing. For example.
I have a file that has 1 GB and when I run the pig's script in a cluster
It takes about 50 minutes to process. =3D(

So I really appreciate if someone could help with any tip. Thanks for
your time.

From general-return-1894-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 05 14:28:53 2010
Return-Path: <general-return-1894-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91525 invoked from network); 5 Aug 2010 14:28:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Aug 2010 14:28:53 -0000
Received: (qmail 33941 invoked by uid 500); 5 Aug 2010 14:28:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33643 invoked by uid 500); 5 Aug 2010 14:28:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33635 invoked by uid 99); 5 Aug 2010 14:28:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 14:28:49 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of marcoscba@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 14:28:42 +0000
Received: by wwb39 with SMTP id 39so4544123wwb.29
        for <general@hadoop.apache.org>; Thu, 05 Aug 2010 07:28:22 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=waEOB3PEg5ThO2JKD/y7ve71ctXGvfRfrvDxDtAe3f4=;
        b=mOR/wZGV6gxwg8H2PzR5UtUSoMK1dFANYVtVIrpRsN8buy/2f8VCnT7fW4UIuVjEdi
         9XltNf6fQyWnW840nZ9UE1h6IOeViqBDVcxD18Ub3iStWDyF9kiIw84XDf4VoS8DmFhd
         PgEiqDIGP7QQB4dcS8cNbyQdpLYOZn2jFVGZQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=vZZypVXZ0WBlny2SLCgbAxl9UMcxTJQIaLc50zKqBTYm4ah6yDWvqQCGFfpB3VgsX4
         BGMfCGkpZQpx75/7CzMVNde2McPxQhnEthXCbzhVKukiDqWvOdiZ6eDXE3Sp9pFD8Eu9
         ev8G0pwuP/GFO/G6Wch9x8TjoDS5RKQ6UL/J4=
MIME-Version: 1.0
Received: by 10.216.3.83 with SMTP id 61mr3602309weg.110.1281018502324; Thu, 
	05 Aug 2010 07:28:22 -0700 (PDT)
Received: by 10.216.166.76 with HTTP; Thu, 5 Aug 2010 07:28:22 -0700 (PDT)
In-Reply-To: <219D8244D980254ABF28AB469AD4E98F03C38611@VF-MBX13.internal.vodafone.com>
References: <AANLkTimp9MDCOTdSzpPKGufnqja+bgYagG4fiReYE5qU@mail.gmail.com>
	<219D8244D980254ABF28AB469AD4E98F03C38611@VF-MBX13.internal.vodafone.com>
Date: Thu, 5 Aug 2010 11:28:22 -0300
Message-ID: <AANLkTimFuNQ=chd0jT8K-z4NGfGZsLa4uz=hiaWzY+M0@mail.gmail.com>
Subject: Re: Problem: when I run a pig's script I got one reduce task
From: Marcos Pinto <marcoscba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64c1e74c82274048d145c02
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e64c1e74c82274048d145c02
Content-Type: text/plain; charset=ISO-8859-1

Oh Man!!! Thanks, your tip comes in handy, =). I had tried to set some
properties but It hadnt worked out but now when I use PARALLEL clause it
works very well. thanks dude. Have nice day.

On Thu, Aug 5, 2010 at 10:38 AM, Gibbon, Robert, VF-Group <
Robert.Gibbon@vodafone.com> wrote:

>
> Use the PARALLEL clause of course!
>
> PARALLEL n
>
> Increase the parallelism of a job by specifying the number of reduce
> tasks, n. The default value for n is 1 (one reduce task). Note the
> following:
>
>    * Parallel only affects the number of reduce tasks. Map parallelism
> is determined by the input file, one map for each HDFS block.
>    * If you don't specify parallel, you still get the same map
> parallelism but only one reduce task.
>
> For more information, see the Pig Cookbook.
>
>
> -----Original Message-----
> From: Marcos Pinto [mailto:marcoscba@gmail.com]
> Sent: Donnerstag, 5. August 2010 14:56
> To: general@hadoop.apache.org
> Subject: Problem: when I run a pig's script I got one reduce task
>
> Hi guys, how u doing?
>
> I am learning how to use hadoop and I got this problem:
> I set up a cluster with 5 nodes( 4 datanode n 1 namenode) and I used the
> same configuration for jobtracker n tasktracker.
> when I run a pig's script I get many map's( like 15) but just 1
> reduce!!!!!
> this kills all the parallel processing. For example.
> I have a file that has 1 GB and when I run the pig's script in a cluster
> It takes about 50 minutes to process. =(
>
> So I really appreciate if someone could help with any tip. Thanks for
> your time.
>

--0016e64c1e74c82274048d145c02--

From general-return-1895-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 05 22:29:19 2010
Return-Path: <general-return-1895-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 13200 invoked from network); 5 Aug 2010 22:29:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Aug 2010 22:29:19 -0000
Received: (qmail 57588 invoked by uid 500); 5 Aug 2010 22:29:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57528 invoked by uid 500); 5 Aug 2010 22:29:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57520 invoked by uid 99); 5 Aug 2010 22:29:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 22:29:16 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 22:29:11 +0000
Received: by wyb35 with SMTP id 35so5076154wyb.35
        for <general@hadoop.apache.org>; Thu, 05 Aug 2010 15:28:50 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.49.212 with SMTP id x62mr9947093web.55.1281047330268; Thu, 
	05 Aug 2010 15:28:50 -0700 (PDT)
Received: by 10.216.20.206 with HTTP; Thu, 5 Aug 2010 15:28:50 -0700 (PDT)
In-Reply-To: <AANLkTinD2_70JChKVy9wzYjKwJLGDQoWxXgymnmYysby@mail.gmail.com>
References: <AANLkTik5koLPa0yNxtc+ta4nJ__Oym4ATq5xn=sRgz29@mail.gmail.com>
	<AANLkTi=yMdJuOxWAogr5uw0iYbp2B88iujFO-0_M=8mr@mail.gmail.com>
	<17C47A26-3B02-4BEA-B862-7849391A6D74@yahoo-inc.com>
	<AANLkTikoAofAVM6Wi2Gj1nh+O4v2RL9VLqbocQMmVV1d@mail.gmail.com>
	<AANLkTinD2_70JChKVy9wzYjKwJLGDQoWxXgymnmYysby@mail.gmail.com>
Date: Thu, 5 Aug 2010 15:28:50 -0700
Message-ID: <AANLkTi=VemHwMj8qcQrBAWA_5XYUdQdmsk5vRS8qco5L@mail.gmail.com>
Subject: Re: Data World Record Falls as Computer Scientists Break Terabyte 
	Sort Barrier
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f5ce5c0fa773048d1b13e4
X-Virus-Checked: Checked by ClamAV on apache.org

--001485f5ce5c0fa773048d1b13e4
Content-Type: text/plain; charset=ISO-8859-1

You might recognize the name of the person who filed HDFS-347 from the press
release about the new sorting world record...

On Tue, Aug 3, 2010 at 8:02 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> Maybe this would give us a little more incentive to resolve HDFS-347
>
> Cheers
>
> On Mon, Aug 2, 2010 at 1:50 PM, Abhishek Verma <vermaabhishekp@gmail.com
> >wrote:
>
> > It shows how further behind Hadoop is in terms of performance. Are there
> > people working on finding the bottlenecks and making it more efficient?
> Are
> > there any JIRA issues related to this?
> >
> > -Abhishek.
> >
> > On Mon, Aug 2, 2010 at 11:47 AM, Arun C Murthy <acm@yahoo-inc.com>
> wrote:
> >
> > > The UCSD results are very impressive, especially given their hardware
> > > budget.
> > >
> > > I may be wrong, but I'm pretty sure there were no Hadoop based entries
> > this
> > > year - I know we at Yahoo! didn't enter.
> > >
> > > Couple of points:
> > > # The Indy category is a benchmark to sort fixed length records, not a
> > > _general_ sort benchmark i.e. Daytona.
> > > # Our _best_ result missed the deadline by a whisker last year, but we
> > > eventually did 100Tb sort in 95 mins and a 1000TB (1PB) in 975 mins
> > (16.25
> > > hrs) - which worked out to be just over 1.0 TB/min, which was nearly
> > twice
> > > as fast as the record attributed to us. (
> > >
> >
> http://developer.yahoo.net/blogs/hadoop/2009/05/hadoop_sorts_a_petabyte_in_162.html
> > > )
> > >
> > > Arun
> > >
> > >
> > > On Aug 2, 2010, at 10:34 AM, Abhishek Verma wrote:
> > >
> > >  Hi Maxim,
> > >>
> > >> Hadoop was not involved. You can find more details here :
> > >> http://sortbenchmark.org/tritonsort_2010_May_15.pdf
> > >> and all the records and their information here :
> > >> http://sortbenchmark.org/
> > >>
> > >> <http://sortbenchmark.org/>-Abhishek.
> > >>
> > >> On Mon, Aug 2, 2010 at 9:52 AM, Maxim Veksler <maxim@vekslers.org>
> > wrote:
> > >>
> > >>  Hi,
> > >>>
> > >>> Anyone knows if Hadoop is involved? And if so what is the
> configuration
> > >>> for
> > >>> such cluster?
> > >>>
> > >>> http://ucsdnews.ucsd.edu/newsrel/science/07-27DataWorld.asp
> > >>>
> > >>>
> > >>> Thank you,
> > >>> Maxim.
> > >>>
> > >>>
> > >
> >
>

--001485f5ce5c0fa773048d1b13e4--

From general-return-1896-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 10:43:53 2010
Return-Path: <general-return-1896-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5512 invoked from network); 6 Aug 2010 10:43:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 10:43:53 -0000
Received: (qmail 96307 invoked by uid 500); 6 Aug 2010 10:43:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95929 invoked by uid 500); 6 Aug 2010 10:43:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95921 invoked by uid 99); 6 Aug 2010 10:43:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 10:43:47 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of qwertymaniac@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 10:43:41 +0000
Received: by wyb35 with SMTP id 35so5710106wyb.35
        for <general@hadoop.apache.org>; Fri, 06 Aug 2010 03:43:21 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=sWjL9FW2ya6ZgDobNWArfZlgmvn4vpzBVSX3DVQ3ePM=;
        b=U2ZmHKGK6O7KpGr/1xhuBzw+FzQkcwcvh5r06KL03foLzobUdqV2XIQoZvMTiX+/pc
         BIzkrf20acdUcdGVd9cB7aZNltFu78nmrGF1O+MuMXajy+CQWpl2vc/obiTh6ISaJuAv
         aIE0YxFNPye8T1DQT8nI6x+Y5jEEARkisNhg0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=NjpORJLcsW2rDKRaK8MduGWuM319gywvThuPZURiODJu55pyYfXkmbTMheTI0mtmBA
         kyAjNkzcQ/+DTE0V8gLH0bCtM+APDf7G54CbAIZdlJehG0zR4o+ZPoByA9iR7/GnhDPI
         m68U3Q3yn1133foUyXrGsaisc/sW4yKt1RqW0=
Received: by 10.227.156.129 with SMTP id x1mr10407737wbw.178.1281091400132; 
	Fri, 06 Aug 2010 03:43:20 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.236.197 with HTTP; Fri, 6 Aug 2010 03:43:00 -0700 (PDT)
In-Reply-To: <AANLkTin_yUUdo3B8OMK9Kglfv3-Yx91rwDP4ZqzGgY1n@mail.gmail.com>
References: <AANLkTin_yUUdo3B8OMK9Kglfv3-Yx91rwDP4ZqzGgY1n@mail.gmail.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Fri, 6 Aug 2010 16:13:00 +0530
Message-ID: <AANLkTi=GE+kVkfPJNxK4N8GRN5AJHOoOzip4PbJSr4t9@mail.gmail.com>
Subject: Re: problem starting cdh3b2 jobtracker
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

java.io.IOException: Cannot create toBeDeleted in /data1/mapred/local

This line points at the solution actually. In earlier versions of CDH
if the list of local mapred directories had false ones (like say the
jobtracker machine not having 2 disks like all the tasktracking
machines and it not being in the slaves list either), it used to
ignore it. Now it doesn't seem to and instead tries to operate things
upon it? Looks like a major bug Cloudera folks! Encountered this using
CDH3 +320. Not using my jobtracker machine to perform tasks as well.

It gets resolved after you validate the mapred local directory list on
the job tracker machine's config alone. However, this would lead to
issues with conf-syncing between nodes if it acts this way forever.

On Fri, Jul 2, 2010 at 8:32 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> We installed cdh3b2 0.20.2+320 and saw some strange error in jobtracker l=
og:
>
> 2010-07-02 01:49:31,977 INFO org.apache.hadoop.mapred.JobTracker: JobTrac=
ker
> up at: 9001
> 2010-07-02 01:49:31,977 INFO org.apache.hadoop.mapred.JobTracker: JobTrac=
ker
> webserver: 50030
> 2010-07-02 01:49:31,988 WARN org.apache.hadoop.mapred.JobTracker: Error
> starting tracker: java.io.IOException: Cannot create toBeDeleted in
> /data1/mapred/local
> =A0 =A0at
> org.apache.hadoop.util.MRAsyncDiskService.<init>(MRAsyncDiskService.java:=
85)
> =A0 =A0at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1688=
)
> =A0 =A0at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.jav=
a:199)
> =A0 =A0at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.jav=
a:191)
> =A0 =A0at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3765)
>
> 2010-07-02 01:49:32,990 INFO org.apache.hadoop.mapred.JobTracker: Schedul=
er
> configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
> limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
> 2010-07-02 01:49:32,991 FATAL org.apache.hadoop.mapred.JobTracker:
> java.net.BindException: Problem binding to
> sjc1-hadoop0.sjc1.ciq.com/10.201.8.204:9001<http://sjc1-hadoop0.sjc1.carr=
ieriq.com/10.201.8.204:9001>:
> Address already in use
> =A0 =A0at org.apache.hadoop.ipc.Server.bind(Server.java:198)
> =A0 =A0at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:261)
> =A0 =A0at org.apache.hadoop.ipc.Server.<init>(Server.java:1043)
> =A0 =A0at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:492)
> =A0 =A0at org.apache.hadoop.ipc.RPC.getServer(RPC.java:454)
> =A0 =A0at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1628=
)
> =A0 =A0at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.jav=
a:199)
> =A0 =A0at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.jav=
a:191)
> =A0 =A0at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3765)
> Caused by: java.net.BindException: Address already in use
> =A0 =A0at sun.nio.ch.Net.bind(Native Method)
> =A0 =A0at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119)
> =A0 =A0at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59=
)
> =A0 =A0at org.apache.hadoop.ipc.Server.bind(Server.java:196)
> =A0 =A0... 8 more
>
> 2010-07-02 01:49:32,992 INFO org.apache.hadoop.mapred.JobTracker:
> SHUTDOWN_MSG:
>
> But 9001 wasn't used:
> [sjc1-hadoop0.sjc1:hadoop 25618]netstat -nta | grep 9001
> [sjc1-hadoop0.sjc1:hadoop 25619]netstat -nta | grep 9000
> tcp =A0 =A0 =A0 =A00 =A0 =A0 =A00 10.201.8.204:9000 =A0 =A0 =A0 =A0 =A0 0=
.0.0.0:*
> LISTEN
> tcp =A0 =A0 =A0 =A00 =A0 =A0 =A00 10.201.8.204:9000 =A0 =A0 =A0 =A0 =A0 1=
0.201.8.214:4223
> ESTABLISHED
> tcp =A0 =A0 =A0 =A00 =A0 =A0 =A00 10.201.8.204:9000 =A0 =A0 =A0 =A0 =A0 1=
0.201.8.212:49074
> ESTABLISHED
> tcp =A0 =A0 =A0 =A00 =A0 =A0 =A00 10.201.8.204:9000 =A0 =A0 =A0 =A0 =A0 1=
0.201.8.206:11910
> ESTABLISHED
> tcp =A0 =A0 =A0 =A00 =A0 =A0 =A00 10.201.8.204:9000 =A0 =A0 =A0 =A0 =A0 1=
0.201.8.210:62611
> ESTABLISHED
> tcp =A0 =A0 =A0 =A00 =A0 =A0 =A00 10.201.8.204:9000 =A0 =A0 =A0 =A0 =A0 1=
0.201.8.213:1299
> ESTABLISHED
> tcp =A0 =A0 =A0 =A00 =A0 =A0 =A00 10.201.8.204:9000 =A0 =A0 =A0 =A0 =A0 1=
0.201.8.205:9756
> ESTABLISHED
> tcp =A0 =A0 =A0 =A00 =A0 =A0 =A00 10.201.8.204:9000 =A0 =A0 =A0 =A0 =A0 1=
0.201.8.207:59207
> ESTABLISHED
>
> Here is output from ifconfig:
> bond0 =A0 =A0 Link encap:Ethernet =A0HWaddr 00:30:48:60:53:94
> =A0 =A0 =A0 =A0 =A0inet addr:10.201.8.204 =A0Bcast:10.201.8.255 =A0Mask:2=
55.255.255.0
> =A0 =A0 =A0 =A0 =A0UP BROADCAST RUNNING MASTER MULTICAST =A0MTU:1500 =A0M=
etric:1
> =A0 =A0 =A0 =A0 =A0RX packets:351496605 errors:0 dropped:1015 overruns:0 =
frame:0
> =A0 =A0 =A0 =A0 =A0TX packets:178144953 errors:0 dropped:0 overruns:0 car=
rier:0
> =A0 =A0 =A0 =A0 =A0collisions:0 txqueuelen:0
> =A0 =A0 =A0 =A0 =A0RX bytes:119420730164 (111.2 GiB) =A0TX bytes:12000212=
3131 (111.7
> GiB)
>
> eth0 =A0 =A0 =A0Link encap:Ethernet =A0HWaddr 00:30:48:60:53:94
> =A0 =A0 =A0 =A0 =A0UP BROADCAST RUNNING SLAVE MULTICAST =A0MTU:1500 =A0Me=
tric:1
> =A0 =A0 =A0 =A0 =A0RX packets:351496605 errors:0 dropped:1015 overruns:0 =
frame:0
> =A0 =A0 =A0 =A0 =A0TX packets:178144953 errors:0 dropped:0 overruns:0 car=
rier:0
> =A0 =A0 =A0 =A0 =A0collisions:0 txqueuelen:1000
> =A0 =A0 =A0 =A0 =A0RX bytes:119420730164 (111.2 GiB) =A0TX bytes:12000212=
3131 (111.7
> GiB)
> =A0 =A0 =A0 =A0 =A0Interrupt:161
>
> eth1 =A0 =A0 =A0Link encap:Ethernet =A0HWaddr 00:30:48:60:53:94
> =A0 =A0 =A0 =A0 =A0UP BROADCAST SLAVE MULTICAST =A0MTU:1500 =A0Metric:1
> =A0 =A0 =A0 =A0 =A0RX packets:0 errors:0 dropped:0 overruns:0 frame:0
> =A0 =A0 =A0 =A0 =A0TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
> =A0 =A0 =A0 =A0 =A0collisions:0 txqueuelen:1000
> =A0 =A0 =A0 =A0 =A0RX bytes:0 (0.0 b) =A0TX bytes:0 (0.0 b)
> =A0 =A0 =A0 =A0 =A0Interrupt:169
>
> Has anyone encountered similar issue ?
>



--=20
Harsh J
www.harshj.com

From general-return-1897-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 17:19:50 2010
Return-Path: <general-return-1897-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98894 invoked from network); 6 Aug 2010 17:19:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 17:19:49 -0000
Received: (qmail 17027 invoked by uid 500); 6 Aug 2010 17:19:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16971 invoked by uid 500); 6 Aug 2010 17:19:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16963 invoked by uid 99); 6 Aug 2010 17:19:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 17:19:47 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 17:19:42 +0000
Received: by qyk34 with SMTP id 34so7809176qyk.14
        for <general@hadoop.apache.org>; Fri, 06 Aug 2010 10:19:21 -0700 (PDT)
Received: by 10.220.127.65 with SMTP id f1mr8538770vcs.94.1281115159310; Fri, 
	06 Aug 2010 10:19:19 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.220.201.197 with HTTP; Fri, 6 Aug 2010 10:18:59 -0700 (PDT)
In-Reply-To: <AANLkTi=GE+kVkfPJNxK4N8GRN5AJHOoOzip4PbJSr4t9@mail.gmail.com>
References: <AANLkTin_yUUdo3B8OMK9Kglfv3-Yx91rwDP4ZqzGgY1n@mail.gmail.com> 
	<AANLkTi=GE+kVkfPJNxK4N8GRN5AJHOoOzip4PbJSr4t9@mail.gmail.com>
From: Aaron Kimball <aaron@cloudera.com>
Date: Fri, 6 Aug 2010 10:18:59 -0700
Message-ID: <AANLkTikK3Oz96d5TDTD2AMoPkJNYbGDDQH6femCm=z1V@mail.gmail.com>
Subject: Re: problem starting cdh3b2 jobtracker
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636e1ee15fcb304048d2add83

--001636e1ee15fcb304048d2add83
Content-Type: text/plain; charset=ISO-8859-1

The IOException in MRAsyncDiskService is being logged with severity level
WARN; I believe that system operation continues normally despite being
unable to clean some of the directories.

Are you experiencing problems where a partial mis-match of mapred.local.dir
configuration and available disks caused a daemon to fail to start? If so,
can you post the configuration and log?

I'm inclined to believe that the fact that the above warning appears in
Ted's log immediately before the fatal problem (trying to bind to a port
already in use) is a red herring. It would be quite surprising to me if the
disk cleanup service affected the socket binding component of the
jobtracker.
- Aaron

On Fri, Aug 6, 2010 at 3:43 AM, Harsh J <qwertymaniac@gmail.com> wrote:

> java.io.IOException: Cannot create toBeDeleted in /data1/mapred/local
>
> This line points at the solution actually. In earlier versions of CDH
> if the list of local mapred directories had false ones (like say the
> jobtracker machine not having 2 disks like all the tasktracking
> machines and it not being in the slaves list either), it used to
> ignore it. Now it doesn't seem to and instead tries to operate things
> upon it? Looks like a major bug Cloudera folks! Encountered this using
> CDH3 +320. Not using my jobtracker machine to perform tasks as well.
>
> It gets resolved after you validate the mapred local directory list on
> the job tracker machine's config alone. However, this would lead to
> issues with conf-syncing between nodes if it acts this way forever.
>
> On Fri, Jul 2, 2010 at 8:32 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> > We installed cdh3b2 0.20.2+320 and saw some strange error in jobtracker
> log:
> >
> > 2010-07-02 01:49:31,977 INFO org.apache.hadoop.mapred.JobTracker:
> JobTracker
> > up at: 9001
> > 2010-07-02 01:49:31,977 INFO org.apache.hadoop.mapred.JobTracker:
> JobTracker
> > webserver: 50030
> > 2010-07-02 01:49:31,988 WARN org.apache.hadoop.mapred.JobTracker: Error
> > starting tracker: java.io.IOException: Cannot create toBeDeleted in
> > /data1/mapred/local
> >    at
> >
> org.apache.hadoop.util.MRAsyncDiskService.<init>(MRAsyncDiskService.java:85)
> >    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1688)
> >    at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:199)
> >    at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:191)
> >    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3765)
> >
> > 2010-07-02 01:49:32,990 INFO org.apache.hadoop.mapred.JobTracker:
> Scheduler
> > configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
> > limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
> > 2010-07-02 01:49:32,991 FATAL org.apache.hadoop.mapred.JobTracker:
> > java.net.BindException: Problem binding to
> > sjc1-hadoop0.sjc1.ciq.com/10.201.8.204:9001<
> http://sjc1-hadoop0.sjc1.carrieriq.com/10.201.8.204:9001>:
> > Address already in use
> >    at org.apache.hadoop.ipc.Server.bind(Server.java:198)
> >    at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:261)
> >    at org.apache.hadoop.ipc.Server.<init>(Server.java:1043)
> >    at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:492)
> >    at org.apache.hadoop.ipc.RPC.getServer(RPC.java:454)
> >    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1628)
> >    at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:199)
> >    at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:191)
> >    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3765)
> > Caused by: java.net.BindException: Address already in use
> >    at sun.nio.ch.Net.bind(Native Method)
> >    at
> > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119)
> >    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
> >    at org.apache.hadoop.ipc.Server.bind(Server.java:196)
> >    ... 8 more
> >
> > 2010-07-02 01:49:32,992 INFO org.apache.hadoop.mapred.JobTracker:
> > SHUTDOWN_MSG:
> >
> > But 9001 wasn't used:
> > [sjc1-hadoop0.sjc1:hadoop 25618]netstat -nta | grep 9001
> > [sjc1-hadoop0.sjc1:hadoop 25619]netstat -nta | grep 9000
> > tcp        0      0 10.201.8.204:9000           0.0.0.0:*
> > LISTEN
> > tcp        0      0 10.201.8.204:9000           10.201.8.214:4223
> > ESTABLISHED
> > tcp        0      0 10.201.8.204:9000           10.201.8.212:49074
> > ESTABLISHED
> > tcp        0      0 10.201.8.204:9000           10.201.8.206:11910
> > ESTABLISHED
> > tcp        0      0 10.201.8.204:9000           10.201.8.210:62611
> > ESTABLISHED
> > tcp        0      0 10.201.8.204:9000           10.201.8.213:1299
> > ESTABLISHED
> > tcp        0      0 10.201.8.204:9000           10.201.8.205:9756
> > ESTABLISHED
> > tcp        0      0 10.201.8.204:9000           10.201.8.207:59207
> > ESTABLISHED
> >
> > Here is output from ifconfig:
> > bond0     Link encap:Ethernet  HWaddr 00:30:48:60:53:94
> >          inet addr:10.201.8.204  Bcast:10.201.8.255  Mask:255.255.255.0
> >          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
> >          RX packets:351496605 errors:0 dropped:1015 overruns:0 frame:0
> >          TX packets:178144953 errors:0 dropped:0 overruns:0 carrier:0
> >          collisions:0 txqueuelen:0
> >          RX bytes:119420730164 (111.2 GiB)  TX bytes:120002123131 (111.7
> > GiB)
> >
> > eth0      Link encap:Ethernet  HWaddr 00:30:48:60:53:94
> >          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
> >          RX packets:351496605 errors:0 dropped:1015 overruns:0 frame:0
> >          TX packets:178144953 errors:0 dropped:0 overruns:0 carrier:0
> >          collisions:0 txqueuelen:1000
> >          RX bytes:119420730164 (111.2 GiB)  TX bytes:120002123131 (111.7
> > GiB)
> >          Interrupt:161
> >
> > eth1      Link encap:Ethernet  HWaddr 00:30:48:60:53:94
> >          UP BROADCAST SLAVE MULTICAST  MTU:1500  Metric:1
> >          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
> >          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
> >          collisions:0 txqueuelen:1000
> >          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
> >          Interrupt:169
> >
> > Has anyone encountered similar issue ?
> >
>
>
>
> --
> Harsh J
> www.harshj.com
>

--001636e1ee15fcb304048d2add83--

From general-return-1898-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 17:48:36 2010
Return-Path: <general-return-1898-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9737 invoked from network); 6 Aug 2010 17:48:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 17:48:35 -0000
Received: (qmail 40282 invoked by uid 500); 6 Aug 2010 17:48:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40192 invoked by uid 500); 6 Aug 2010 17:48:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40184 invoked by uid 99); 6 Aug 2010 17:48:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 17:48:34 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 17:48:29 +0000
Received: from wlanvpn-snve-247-130.corp.yahoo.com (wlanvpn-snve-247-130.corp.yahoo.com [172.21.149.130])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o76HlBHB091658
	for <general@hadoop.apache.org>; Fri, 6 Aug 2010 10:47:12 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=Mu5kWpLWQxV8Tm9lly5k05rIGSEc9RzgWZdEjXG4TyWFLE35I5hJ+o1VR7GsfV4p
Message-Id: <A71B1C9C-0CF4-475A-9F1C-81E143D1B9F0@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <AANLkTi=GE+kVkfPJNxK4N8GRN5AJHOoOzip4PbJSr4t9@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: problem starting cdh3b2 jobtracker
Date: Fri, 6 Aug 2010 10:47:11 -0700
References: <AANLkTin_yUUdo3B8OMK9Kglfv3-Yx91rwDP4ZqzGgY1n@mail.gmail.com> <AANLkTi=GE+kVkfPJNxK4N8GRN5AJHOoOzip4PbJSr4t9@mail.gmail.com>
X-Mailer: Apple Mail (2.936)

Please keep discussions of CDH confined to Cloudera's email lists.

On Aug 6, 2010, at 3:43 AM, Harsh J wrote:

> java.io.IOException: Cannot create toBeDeleted in /data1/mapred/local
>
> This line points at the solution actually. In earlier versions of CDH
> if the list of local mapred directories had false ones (like say the
> jobtracker machine not having 2 disks like all the tasktracking
> machines and it not being in the slaves list either), it used to
> ignore it. Now it doesn't seem to and instead tries to operate things
> upon it? Looks like a major bug Cloudera folks! Encountered this using
> CDH3 +320. Not using my jobtracker machine to perform tasks as well.
>
> It gets resolved after you validate the mapred local directory list on
> the job tracker machine's config alone. However, this would lead to
> issues with conf-syncing between nodes if it acts this way forever.
>
> On Fri, Jul 2, 2010 at 8:32 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>> We installed cdh3b2 0.20.2+320 and saw some strange error in  
>> jobtracker log:
>>
>> 2010-07-02 01:49:31,977 INFO org.apache.hadoop.mapred.JobTracker:  
>> JobTracker
>> up at: 9001
>> 2010-07-02 01:49:31,977 INFO org.apache.hadoop.mapred.JobTracker:  
>> JobTracker
>> webserver: 50030
>> 2010-07-02 01:49:31,988 WARN org.apache.hadoop.mapred.JobTracker:  
>> Error
>> starting tracker: java.io.IOException: Cannot create toBeDeleted in
>> /data1/mapred/local
>>    at
>> org 
>> .apache 
>> .hadoop.util.MRAsyncDiskService.<init>(MRAsyncDiskService.java:85)
>>    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java: 
>> 1688)
>>    at  
>> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:199)
>>    at  
>> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:191)
>>    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3765)
>>
>> 2010-07-02 01:49:32,990 INFO org.apache.hadoop.mapred.JobTracker:  
>> Scheduler
>> configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
>> limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
>> 2010-07-02 01:49:32,991 FATAL org.apache.hadoop.mapred.JobTracker:
>> java.net.BindException: Problem binding to
>> sjc1-hadoop0.sjc1.ciq.com/10.201.8.204:9001<http://sjc1-hadoop0.sjc1.carrieriq.com/10.201.8.204:9001 
>> >:
>> Address already in use
>>    at org.apache.hadoop.ipc.Server.bind(Server.java:198)
>>    at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:261)
>>    at org.apache.hadoop.ipc.Server.<init>(Server.java:1043)
>>    at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:492)
>>    at org.apache.hadoop.ipc.RPC.getServer(RPC.java:454)
>>    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java: 
>> 1628)
>>    at  
>> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:199)
>>    at  
>> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:191)
>>    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3765)
>> Caused by: java.net.BindException: Address already in use
>>    at sun.nio.ch.Net.bind(Native Method)
>>    at
>> sun 
>> .nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java: 
>> 119)
>>    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java: 
>> 59)
>>    at org.apache.hadoop.ipc.Server.bind(Server.java:196)
>>    ... 8 more
>>
>> 2010-07-02 01:49:32,992 INFO org.apache.hadoop.mapred.JobTracker:
>> SHUTDOWN_MSG:
>>
>> But 9001 wasn't used:
>> [sjc1-hadoop0.sjc1:hadoop 25618]netstat -nta | grep 9001
>> [sjc1-hadoop0.sjc1:hadoop 25619]netstat -nta | grep 9000
>> tcp        0      0 10.201.8.204:9000           0.0.0.0:*
>> LISTEN
>> tcp        0      0 10.201.8.204:9000           10.201.8.214:4223
>> ESTABLISHED
>> tcp        0      0 10.201.8.204:9000           10.201.8.212:49074
>> ESTABLISHED
>> tcp        0      0 10.201.8.204:9000           10.201.8.206:11910
>> ESTABLISHED
>> tcp        0      0 10.201.8.204:9000           10.201.8.210:62611
>> ESTABLISHED
>> tcp        0      0 10.201.8.204:9000           10.201.8.213:1299
>> ESTABLISHED
>> tcp        0      0 10.201.8.204:9000           10.201.8.205:9756
>> ESTABLISHED
>> tcp        0      0 10.201.8.204:9000           10.201.8.207:59207
>> ESTABLISHED
>>
>> Here is output from ifconfig:
>> bond0     Link encap:Ethernet  HWaddr 00:30:48:60:53:94
>>          inet addr:10.201.8.204  Bcast:10.201.8.255  Mask: 
>> 255.255.255.0
>>          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
>>          RX packets:351496605 errors:0 dropped:1015 overruns:0  
>> frame:0
>>          TX packets:178144953 errors:0 dropped:0 overruns:0 carrier:0
>>          collisions:0 txqueuelen:0
>>          RX bytes:119420730164 (111.2 GiB)  TX bytes:120002123131  
>> (111.7
>> GiB)
>>
>> eth0      Link encap:Ethernet  HWaddr 00:30:48:60:53:94
>>          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
>>          RX packets:351496605 errors:0 dropped:1015 overruns:0  
>> frame:0
>>          TX packets:178144953 errors:0 dropped:0 overruns:0 carrier:0
>>          collisions:0 txqueuelen:1000
>>          RX bytes:119420730164 (111.2 GiB)  TX bytes:120002123131  
>> (111.7
>> GiB)
>>          Interrupt:161
>>
>> eth1      Link encap:Ethernet  HWaddr 00:30:48:60:53:94
>>          UP BROADCAST SLAVE MULTICAST  MTU:1500  Metric:1
>>          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>>          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>>          collisions:0 txqueuelen:1000
>>          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
>>          Interrupt:169
>>
>> Has anyone encountered similar issue ?
>>
>
>
>
> -- 
> Harsh J
> www.harshj.com


From general-return-1899-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 21:03:09 2010
Return-Path: <general-return-1899-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97665 invoked from network); 6 Aug 2010 21:03:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 21:03:08 -0000
Received: (qmail 91789 invoked by uid 500); 6 Aug 2010 21:03:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91544 invoked by uid 500); 6 Aug 2010 21:03:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91535 invoked by uid 99); 6 Aug 2010 21:03:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 21:03:06 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 06 Aug 2010 21:03:04 +0000
Received: (qmail 97428 invoked by uid 99); 6 Aug 2010 21:02:43 -0000
Received: from localhost.apache.org (HELO mail-iw0-f176.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 21:02:43 +0000
Received: by iwn37 with SMTP id 37so2129601iwn.35
        for <multiple recipients>; Fri, 06 Aug 2010 14:02:42 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.149.207 with SMTP id u15mr14822845ibv.13.1281128562600; 
	Fri, 06 Aug 2010 14:02:42 -0700 (PDT)
Received: by 10.231.144.135 with HTTP; Fri, 6 Aug 2010 14:02:42 -0700 (PDT)
Date: Fri, 6 Aug 2010 14:02:42 -0700
Message-ID: <AANLkTi=GyFLivH8uOQNppUJnVeixpY5HA98=0cJEpaNa@mail.gmail.com>
Subject: [DISCUSSION] Combine committer lists in Common/HDFS/MapReduce
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Cc: private@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hadoop developers tend to specialize in either HDFS or MapReduce, but
given that:

0) Granting karma to Common is routine for a committer in either
space; there are no Common-only committers
1) The majority of committers have been grandfathered into committer
roles in all three projects
2) Many patches to Common require corresponding commits to both HDFS
and MapReduce
3) Review-then-commit is usually sufficient notice for interested
parties to comment
4) There have been few problems with committers pushing in patches
without consulting someone more directly involved
5) Everyone on the PMC gets commit rights to all subprojects, anyway

Perhaps it would make sense to give up on separate committer roles
until the projects are separate TLPs.

On the other hand:

0) Nobody has been independently added to both HDFS and MapReduce
since the projects were separated
1) It could exacerbate the focus on MapReduce in HDFS, at the expense
of other projects (like HBase).
2) HDFS and MapReduce are mostly independent communities and
codebases; expertise in one does not imply fluency in the other
3) Granting veto power across projects can lead to deadlock despite
consensus within that community
4) TLP status for either project may require untangling HDFS/MR roles
that could be distinguished now

Personally, I'm in favor of combining the roles. I trust all six of
the committers made since the project split no less than those made
earlier. Further, version control is sufficient for recovering from
most, foreseeable issues. I have some concerns about "harmless"
commits pushed through without an audit by the subproject's
maintainers (a few in recent memory caused downtime in Y! clusters),
but combining the roles seems like a worthwhile experiment.

Thoughts? -C

From general-return-1900-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 21:48:14 2010
Return-Path: <general-return-1900-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14405 invoked from network); 6 Aug 2010 21:48:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 21:48:14 -0000
Received: (qmail 30721 invoked by uid 500); 6 Aug 2010 21:48:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30649 invoked by uid 500); 6 Aug 2010 21:48:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30641 invoked by uid 99); 6 Aug 2010 21:48:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 21:48:12 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 06 Aug 2010 21:48:10 +0000
Received: (qmail 14341 invoked by uid 99); 6 Aug 2010 21:47:47 -0000
Received: from localhost.apache.org (HELO [192.168.168.147]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 21:47:47 +0000
Message-ID: <4C5C8302.3010008@apache.org>
Date: Fri, 06 Aug 2010 14:47:46 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.11) Gecko/20100713 Thunderbird/3.0.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSSION] Combine committer lists in Common/HDFS/MapReduce
References: <AANLkTi=GyFLivH8uOQNppUJnVeixpY5HA98=0cJEpaNa@mail.gmail.com>
In-Reply-To: <AANLkTi=GyFLivH8uOQNppUJnVeixpY5HA98=0cJEpaNa@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 08/06/2010 02:02 PM, Chris Douglas wrote:
> Thoughts?

To my thinking, a single set of committers should manage a product.  A 
product is something that's released.  Currently, we still branch and 
release Common, HDFS and MapReduce together, so I regard them as a 
single product and hence believe they should have a single set of 
committers.  If/when we start to release them separately then we can 
consider splitting committer lists.  Even then, we don't have to split 
them, since a single set of committers can manage multiple products.  In 
fact, I don't see a strong case for splitting committer lists until 
these become separate TLPs.  (The other Hadoop subprojects with separate 
committer lists ought to become TLPs, but that's a separate discussion.)

Doug

From general-return-1901-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 07 04:45:55 2010
Return-Path: <general-return-1901-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 48123 invoked from network); 7 Aug 2010 04:45:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Aug 2010 04:45:55 -0000
Received: (qmail 95850 invoked by uid 500); 7 Aug 2010 04:45:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95399 invoked by uid 500); 7 Aug 2010 04:45:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95383 invoked by uid 99); 7 Aug 2010 04:45:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 Aug 2010 04:45:48 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 Aug 2010 04:45:40 +0000
Received: by wyb35 with SMTP id 35so6936291wyb.35
        for <multiple recipients>; Fri, 06 Aug 2010 21:45:20 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:cc:content-type;
        bh=v9KS8l08mrHHRigMH8l8ORR434DLQa06xwP0zB88DJk=;
        b=IPtyewzd8B6HK3NEbIOknMNzkXQQlg6Rs1XaAaMa0z9WUYJLWyDP2JLTdCDhy4sbpj
         lBka5hBB+iQ5ocMBuzRpPoHk/12icauWe5ilI2MF2dW5J+Qjp9rKh4SwIHS2iIkLP6lW
         dRdXyY7HCgbI77Xk6uPPeSm310232VLXtGGl8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:cc:content-type;
        b=iGwz9+veTwYgIDYxkoVkvbjCNCiCzzsJai/05MxAHM6bLtNWyDmDmgLosiE52SgmW2
         o3j0weUFy732dgddfjFfwbf5k3TcVemLCudYOxLppak0jjmsLPDHBIqf0XMhd2RAx7IM
         bcJiFD8LeRGsrIcuOv/brIYVx+AG/BLPgBWbw=
MIME-Version: 1.0
Received: by 10.227.40.206 with SMTP id l14mr11381914wbe.139.1281156320400; 
	Fri, 06 Aug 2010 21:45:20 -0700 (PDT)
Sender: saint.ack@gmail.com
Received: by 10.216.168.149 with HTTP; Fri, 6 Aug 2010 21:45:20 -0700 (PDT)
In-Reply-To: <AANLkTi=GyFLivH8uOQNppUJnVeixpY5HA98=0cJEpaNa@mail.gmail.com>
References: <AANLkTi=GyFLivH8uOQNppUJnVeixpY5HA98=0cJEpaNa@mail.gmail.com>
Date: Fri, 6 Aug 2010 21:45:20 -0700
X-Google-Sender-Auth: Yuu5U46U6VwC70Wgn9YKF5XjGUw
Message-ID: <AANLkTi=z3UMZq0iokEkYvHVNtxVfBfNyaPTzFqGGLnTN@mail.gmail.com>
Subject: Re: [DISCUSSION] Combine committer lists in Common/HDFS/MapReduce
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Cc: private@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

+1 on committers getting access to common+hdfs+mr.

Thanks for putting up the comprehensive pros and cons Chris.  It makes
it so I don't have to think.

The only CON that gives me pause is item 3). but my guess is that its
unlikely and that if it should ever happen, as a community, we'd
figure some way around (over/under) the roadblock.

For those who might be strong on HDFS but not on MR (or vice-versa), I
think its OK having 'privileges' though you may not exercise them.  In
being nominated for committer status, a nominee has probably passed
the baseline common sense test that has it that they'll NOT be
applying patches in areas they do not well-understand.

St.Ack

From general-return-1902-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 07 18:41:27 2010
Return-Path: <general-return-1902-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59542 invoked from network); 7 Aug 2010 18:41:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Aug 2010 18:41:26 -0000
Received: (qmail 17225 invoked by uid 500); 7 Aug 2010 18:41:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17055 invoked by uid 500); 7 Aug 2010 18:41:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17047 invoked by uid 99); 7 Aug 2010 18:41:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 Aug 2010 18:41:24 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of renatoj.marroquin@gmail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 Aug 2010 18:41:16 +0000
Received: by iwn35 with SMTP id 35so777385iwn.35
        for <general@hadoop.apache.org>; Sat, 07 Aug 2010 11:40:56 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=MP6IwDVMW/U2GGWpTpv8OWEBcLMe39qwXGvii2G7iAA=;
        b=vJHoATisPIQsAD2DpnlZ0epV2YuuHfbaIQQw/u27G31y+MdhrbeLZ1mVZeSAJBgggT
         5AEHtYepXGSfmfaVhF2kk/WEQHHYcDIxkFNjUWJARuaHll2b+lYI5BDJ2uIneLf7Um9x
         GldBC/OCez1cikMDtqDvvxePb+totZELJnfvo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=AAAhBf8EfMwC0ad0f4gS54Urwbi2gjp2o78J3sptQZ2FYTNbfSNCe6x3+mjLKt5LQE
         cE/o5WhLXqn3j/czGqcZARo4nVSedgykSHNIVlZ9R16Z4dQXFrfj3qswVvo7lGz+yfxW
         XRIynYOSP51Zf3H+dUiDwUnctTPj6ZeMiXti8=
MIME-Version: 1.0
Received: by 10.231.29.33 with SMTP id o33mr15862033ibc.164.1281206454897; 
	Sat, 07 Aug 2010 11:40:54 -0700 (PDT)
Received: by 10.231.15.73 with HTTP; Sat, 7 Aug 2010 11:40:54 -0700 (PDT)
Date: Sat, 7 Aug 2010 13:40:54 -0500
Message-ID: <AANLkTinbKV8nOtDyzNoURrFjJLz=M8gr6bLtg1TeZLqZ@mail.gmail.com>
Subject: Hadoop node's locality
From: =?ISO-8859-1?Q?Renato_Marroqu=EDn_Mogrovejo?= <renatoj.marroquin@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015177408fca0cd36048d401f40
X-Virus-Checked: Checked by ClamAV on apache.org

--0015177408fca0cd36048d401f40
Content-Type: text/plain; charset=ISO-8859-1

Hi everyone,
Does anybody know if Hadoop can take scheduling decisions using nodes positions
within network topology ? Or if there is any work being done on this?
Thanks in advanced.


Renato M.

--0015177408fca0cd36048d401f40--

From general-return-1903-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 08 19:22:56 2010
Return-Path: <general-return-1903-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46578 invoked from network); 8 Aug 2010 19:22:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Aug 2010 19:22:56 -0000
Received: (qmail 20104 invoked by uid 500); 8 Aug 2010 19:22:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20031 invoked by uid 500); 8 Aug 2010 19:22:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 20023 invoked by uid 99); 8 Aug 2010 19:22:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 Aug 2010 19:22:54 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 Aug 2010 19:22:49 +0000
Received: from [192.168.1.71] (snvvpn1-10-72-244-c36.hq.corp.yahoo.com [10.72.244.36])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o78JLZbC038414
	for <general@hadoop.apache.org>; Sun, 8 Aug 2010 12:21:36 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=fxoDCNNat2wvUzZXrqYtSm3ZqqH9kuKjAWfpQVXFmRuh9HiOfeV7ME3i3zEHqaUo
Message-Id: <97E2C026-0C44-4FDC-A569-63E218E03D59@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <4C5C8302.3010008@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSSION] Combine committer lists in Common/HDFS/MapReduce
Date: Sun, 8 Aug 2010 12:21:40 -0700
References: <AANLkTi=GyFLivH8uOQNppUJnVeixpY5HA98=0cJEpaNa@mail.gmail.com> <4C5C8302.3010008@apache.org>
X-Mailer: Apple Mail (2.936)


On Aug 6, 2010, at 2:47 PM, Doug Cutting wrote:

> On 08/06/2010 02:02 PM, Chris Douglas wrote:
>> Thoughts?
>
> To my thinking, a single set of committers should manage a product.  A
> product is something that's released.  Currently, we still branch and
> release Common, HDFS and MapReduce together, so I regard them as a
> single product and hence believe they should have a single set of
> committers.

This of course begs a larger question - should we just merge Common,  
HDFS & Map-Reduce together and be done with?

Arun

From general-return-1904-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 08 19:51:09 2010
Return-Path: <general-return-1904-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53895 invoked from network); 8 Aug 2010 19:51:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Aug 2010 19:51:09 -0000
Received: (qmail 37979 invoked by uid 500); 8 Aug 2010 19:51:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37903 invoked by uid 500); 8 Aug 2010 19:51:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37895 invoked by uid 99); 8 Aug 2010 19:51:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 Aug 2010 19:51:07 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 Aug 2010 19:51:00 +0000
Received: from sp1-ex07cas03.ds.corp.yahoo.com (sp1-ex07cas03.ds.corp.yahoo.com [216.252.116.151])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o78Jo85O039419;
	Sun, 8 Aug 2010 12:50:08 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:cc:date:subject:thread-topic:
	thread-index:message-id:references:in-reply-to:accept-language:
	content-language:x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:
	content-type:content-transfer-encoding:mime-version;
	b=N7W7hnSTvtMP4GV6zxccUeKImvw0JLIFZ9++jl3YftzTebLpFa+AIcYYY7B2PTnN
Received: from SP1-EX07VS02.ds.corp.yahoo.com ([216.252.116.136]) by
 sp1-ex07cas03.ds.corp.yahoo.com ([216.252.116.151]) with mapi; Sun, 8 Aug
 2010 12:50:08 -0700
From: Devaraj Das <ddas@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Sun, 8 Aug 2010 12:50:03 -0700
Subject: Re: [DISCUSSION] Combine committer lists in Common/HDFS/MapReduce
Thread-Topic: [DISCUSSION] Combine committer lists in Common/HDFS/MapReduce
Thread-Index: Acs3Mubv0sxKgAZHSDiKTUumhCQpLw==
Message-ID: <196D22F3-42CE-416F-AB88-33E197FF22ED@yahoo-inc.com>
References: <AANLkTi=GyFLivH8uOQNppUJnVeixpY5HA98=0cJEpaNa@mail.gmail.com>
 <4C5C8302.3010008@apache.org>
 <97E2C026-0C44-4FDC-A569-63E218E03D59@yahoo-inc.com>
In-Reply-To: <97E2C026-0C44-4FDC-A569-63E218E03D59@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0



Sent from my iPhone

On Aug 8, 2010, at 12:23 PM, "Arun C Murthy" <acm@yahoo-inc.com> wrote:

>
> On Aug 6, 2010, at 2:47 PM, Doug Cutting wrote:
>
>> On 08/06/2010 02:02 PM, Chris Douglas wrote:
>>> Thoughts?
>>
>> To my thinking, a single set of committers should manage a =20
>> product.  A
>> product is something that's released.  Currently, we still branch and
>> release Common, HDFS and MapReduce together, so I regard them as a
>> single product and hence believe they should have a single set of
>> committers.
>
> This of course begs a larger question - should we just merge Common,
> HDFS & Map-Reduce together and be done with?

Good point Arun. I am in support of this.

>
> Arun

From general-return-1905-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 09 04:43:32 2010
Return-Path: <general-return-1905-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70374 invoked from network); 9 Aug 2010 04:43:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Aug 2010 04:43:32 -0000
Received: (qmail 10342 invoked by uid 500); 9 Aug 2010 04:43:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 9935 invoked by uid 500); 9 Aug 2010 04:43:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 9927 invoked by uid 99); 9 Aug 2010 04:43:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 04:43:27 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yhemanth@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 04:43:20 +0000
Received: by qwd7 with SMTP id 7so9887019qwd.35
        for <general@hadoop.apache.org>; Sun, 08 Aug 2010 21:42:59 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=qEz3uD89HU3ZsfjA3+v9DKRnci3VBuQxgwEhWosuoGQ=;
        b=XB4EoRpK23SxEz05tBgNNt4LqAQIODpD7SXbPYj4IyhMnOc53Xum5pOnGAyAYl1+r+
         pqDIEwYj3olSS+Se8eeqmyo8SI+3lCa47oPwKqw7YKsM6G2vs/RFYjtdMjnUFy8qAoYB
         +Atd5NA0+TRhI49+2ZGDrsAa/6bunfXakgAZo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=Plm36bSHl5FThBmY0MJeUFKpFeAv3cm2z5XxyVee6kCu8gYUVRa/1hRHg/0cRcycT2
         phoDWGyvYmwYOybVcSTFU2XCTNNl16SCvHXKmo98+fzlcAsEBga9CnjQ9ctF/dnVYzZF
         LumTXxcmUlOaZgt84rSbMvQMByoq1sx+IrV6U=
MIME-Version: 1.0
Received: by 10.220.163.10 with SMTP id y10mr9326988vcx.63.1281328979595; Sun, 
	08 Aug 2010 21:42:59 -0700 (PDT)
Received: by 10.220.178.138 with HTTP; Sun, 8 Aug 2010 21:42:59 -0700 (PDT)
In-Reply-To: <97E2C026-0C44-4FDC-A569-63E218E03D59@yahoo-inc.com>
References: <AANLkTi=GyFLivH8uOQNppUJnVeixpY5HA98=0cJEpaNa@mail.gmail.com>
	<4C5C8302.3010008@apache.org>
	<97E2C026-0C44-4FDC-A569-63E218E03D59@yahoo-inc.com>
Date: Mon, 9 Aug 2010 10:12:59 +0530
Message-ID: <AANLkTim1_=o74n6vQMhb8jyGBAXW-9e9L3vwjQL6z63O@mail.gmail.com>
Subject: Re: [DISCUSSION] Combine committer lists in Common/HDFS/MapReduce
From: Hemanth Yamijala <yhemanth@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

>
> On Aug 6, 2010, at 2:47 PM, Doug Cutting wrote:
>
>> On 08/06/2010 02:02 PM, Chris Douglas wrote:
>>>
>>> Thoughts?
>>
>> To my thinking, a single set of committers should manage a product. =A0A
>> product is something that's released. =A0Currently, we still branch and
>> release Common, HDFS and MapReduce together, so I regard them as a
>> single product and hence believe they should have a single set of
>> committers.
>
> This of course begs a larger question - should we just merge Common, HDFS=
 &
> Map-Reduce together and be done with?

I actually think that would simplify matters *smile*

Thanks
hemanth

From general-return-1906-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 09 11:08:25 2010
Return-Path: <general-return-1906-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24243 invoked from network); 9 Aug 2010 11:08:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Aug 2010 11:08:25 -0000
Received: (qmail 57700 invoked by uid 500); 9 Aug 2010 11:08:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57311 invoked by uid 500); 9 Aug 2010 11:08:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57303 invoked by uid 99); 9 Aug 2010 11:08:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 11:08:19 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 11:08:09 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 223AEB7CBF
	for <general@hadoop.apache.org>; Mon,  9 Aug 2010 12:07:47 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id Irv5n5uI08cG for <general@hadoop.apache.org>;
	Mon,  9 Aug 2010 12:07:47 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 52422B7CB5
	for <general@hadoop.apache.org>; Mon,  9 Aug 2010 12:07:46 +0100 (BST)
MailScanner-NULL-Check: 1281956854.4814@DroRhlhXNmD66w708lK8Dw
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o79B7WHc026182
	for <general@hadoop.apache.org>; Mon, 9 Aug 2010 12:07:32 +0100 (BST)
Message-ID: <4C5FE174.5030702@apache.org>
Date: Mon, 09 Aug 2010 12:07:32 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.11) Gecko/20100713 Thunderbird/3.0.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSSION] Combine committer lists in Common/HDFS/MapReduce
References: <AANLkTi=GyFLivH8uOQNppUJnVeixpY5HA98=0cJEpaNa@mail.gmail.com>	<4C5C8302.3010008@apache.org>	<97E2C026-0C44-4FDC-A569-63E218E03D59@yahoo-inc.com> <AANLkTim1_=o74n6vQMhb8jyGBAXW-9e9L3vwjQL6z63O@mail.gmail.com>
In-Reply-To: <AANLkTim1_=o74n6vQMhb8jyGBAXW-9e9L3vwjQL6z63O@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o79B7WHc026182
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 09/08/10 05:42, Hemanth Yamijala wrote:
>>
>> On Aug 6, 2010, at 2:47 PM, Doug Cutting wrote:
>>
>>> On 08/06/2010 02:02 PM, Chris Douglas wrote:
>>>>
>>>> Thoughts?
>>>
>>> To my thinking, a single set of committers should manage a product.  A
>>> product is something that's released.  Currently, we still branch and
>>> release Common, HDFS and MapReduce together, so I regard them as a
>>> single product and hence believe they should have a single set of
>>> committers.
>>
>> This of course begs a larger question - should we just merge Common, HDFS&
>> Map-Reduce together and be done with?
>
> I actually think that would simplify matters *smile*

I seem to recall the original goal of the split was to make it easier 
for people to stay current with the dev of their particular bit, but I'm 
not sure that goal has been met

-more dev mailing lists, all of which I'm behind on
-more user lists, messages get sent to more of them or its not clear where
-its very hard to push out manageability changes across everything
-no clear push for a desynchronised release process
-testing of everything is very convoluted

common, hdfs and mapreduce are very tightly coupled, in code and in 
releases. I don't see the split as having helped, it just seems to have 
slowed the rate of release, and that's been bad for everyone.

-steve

From general-return-1907-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 09 12:35:20 2010
Return-Path: <general-return-1907-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66395 invoked from network); 9 Aug 2010 12:35:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Aug 2010 12:35:20 -0000
Received: (qmail 67526 invoked by uid 500); 9 Aug 2010 12:35:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66950 invoked by uid 500); 9 Aug 2010 12:35:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66942 invoked by uid 99); 9 Aug 2010 12:35:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 12:35:15 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=FREEMAIL_FROM,FROM_LOCAL_NOVOWEL,HK_RANDOM_ENVFROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of sssssssenator@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 12:35:08 +0000
Received: by wwb39 with SMTP id 39so1647502wwb.29
        for <general@hadoop.apache.org>; Mon, 09 Aug 2010 05:34:47 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=OOaMKhhet+EbwIJGmziaHCRSVl3VeExQWPjoa9f9By8=;
        b=WKLntCSGObKQhIkmRQAkj/BgLMb68JZMdt3XXeJhdf/Cp1UPFgIRiZKsljxpHmgs2j
         +gEOXiMa1V2Rt2fvd+RSM76TLBqZKWHRlMe6cnx8SSiq/VyNP8z+0RR5iQ7ex9Dr5MJV
         OV3nMhLMI5i+mqm1CnotxU9iFkyK0EpZJfjb4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=C2qnxTSeXQPuG27zHOWSdrbJLwTP2KGNm89WqOZvDiBO29clrnQ9Ea6Yi5P/+ZRYKd
         jkd1dLhJVZyNwMgJIEocdLmdFc2beEp2ZeKcwI05+7ZgV9Zm6ZecbPLW34ZKV5iMxWbV
         3JkJa7+QljZz5Ppa64dWb+UBCZSCsDCn2dBdk=
MIME-Version: 1.0
Received: by 10.227.156.202 with SMTP id y10mr13498877wbw.48.1281357284460; 
	Mon, 09 Aug 2010 05:34:44 -0700 (PDT)
Received: by 10.216.139.13 with HTTP; Mon, 9 Aug 2010 05:34:44 -0700 (PDT)
In-Reply-To: <AANLkTinbKV8nOtDyzNoURrFjJLz=M8gr6bLtg1TeZLqZ@mail.gmail.com>
References: <AANLkTinbKV8nOtDyzNoURrFjJLz=M8gr6bLtg1TeZLqZ@mail.gmail.com>
Date: Mon, 9 Aug 2010 18:04:44 +0530
Message-ID: <AANLkTinfnCioatFRCQd2GePuyx6UGKJOx+br5Tj5yqtT@mail.gmail.com>
Subject: Re: Hadoop node's locality
From: vaibhav negi <sssssssenator@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016363b9a58c548ce048d633dc2
X-Virus-Checked: Checked by ClamAV on apache.org

--0016363b9a58c548ce048d633dc2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi ,

I am also a nebie to hadoop. But i think u should read about "rack aware"
concept.
It may solve your query.

Vaibhav Negi


On Sun, Aug 8, 2010 at 12:10 AM, Renato Marroqu=EDn Mogrovejo <
renatoj.marroquin@gmail.com> wrote:

> Hi everyone,
> Does anybody know if Hadoop can take scheduling decisions using nodes
> positions
> within network topology ? Or if there is any work being done on this?
> Thanks in advanced.
>
>
> Renato M.
>

--0016363b9a58c548ce048d633dc2--

From general-return-1908-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 09 12:41:18 2010
Return-Path: <general-return-1908-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68793 invoked from network); 9 Aug 2010 12:41:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Aug 2010 12:41:17 -0000
Received: (qmail 71383 invoked by uid 500); 9 Aug 2010 12:41:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71084 invoked by uid 500); 9 Aug 2010 12:41:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71076 invoked by uid 99); 9 Aug 2010 12:41:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 12:41:12 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [210.210.41.205] (HELO mail1.mkhoj.com) (210.210.41.205)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 12:41:06 +0000
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: Al4FAAOQX0wKDmQH/2dsb2JhbACTUI1lwReFOgSEJIUXgk4
X-IronPort-AV: E=Sophos;i="4.55,341,1278268200"; 
   d="scan'208";a="898080"
Received: from mk-exch-1.mkhoj.com ([10.14.100.7])
  by mail1.mkhoj.com with ESMTP; 09 Aug 2010 18:10:41 +0530
Received: from [10.14.100.125] (10.14.100.125) by MK-EXCH-1.MKHOJ.COM
 (10.14.100.7) with Microsoft SMTP Server id 8.1.340.0; Mon, 9 Aug 2010
 18:07:40 +0530
Message-ID: <4C5FF6C0.5000906@inmobi.com>
Date: Mon, 9 Aug 2010 18:08:24 +0530
From: Rohan Rai <rohan.rai@inmobi.com>
User-Agent: Thunderbird 2.0.0.24 (X11/20100411)
MIME-Version: 1.0
To: <general@hadoop.apache.org>
Subject: Hadoop 0.21
Content-Type: text/plain; charset="ISO-8859-1"; format=flowed
Content-Transfer-Encoding: quoted-printable

Hi

http://people.apache.org/~tomwhite/hadoop-0.21.0-candidate-0/

Do we have any more candidate release after this.

Thanking You
Regards
Rohan

The information contained in this communication is intended solely for the =
use of the individual or entity to whom it is addressed and others authoriz=
ed to receive it. It may contain confidential or legally privileged informa=
tion. If you are not the intended recipient you are hereby notified that an=
y disclosure, copying, distribution or taking any action in reliance on the=
 contents of this information is strictly prohibited and may be unlawful. I=
f you have received this communication in error, please notify us immediate=
ly by responding to this email and then delete it from your system. The fir=
m is neither liable for the proper and complete transmission of the informa=
tion contained in this communication nor for any delay in its receipt.

From general-return-1909-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 09 14:32:34 2010
Return-Path: <general-return-1909-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39688 invoked from network); 9 Aug 2010 14:32:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Aug 2010 14:32:33 -0000
Received: (qmail 44602 invoked by uid 500); 9 Aug 2010 14:32:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44031 invoked by uid 500); 9 Aug 2010 14:32:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44023 invoked by uid 99); 9 Aug 2010 14:32:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 14:32:27 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 14:32:21 +0000
Received: by bwz9 with SMTP id 9so2072776bwz.35
        for <general@hadoop.apache.org>; Mon, 09 Aug 2010 07:32:00 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.49.22 with SMTP id t22mr10767045bkf.188.1281364319394; 
	Mon, 09 Aug 2010 07:31:59 -0700 (PDT)
Received: by 10.204.179.74 with HTTP; Mon, 9 Aug 2010 07:31:59 -0700 (PDT)
In-Reply-To: <4C5FF6C0.5000906@inmobi.com>
References: <4C5FF6C0.5000906@inmobi.com>
Date: Mon, 9 Aug 2010 07:31:59 -0700
Message-ID: <AANLkTi=bRP-viDa7fg97STCceGgOGkVf7VAUYgL-pZ1c@mail.gmail.com>
Subject: Re: Hadoop 0.21
From: Tom White <tom@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hi Rohan,

MAPREDUCE-1920 is blocking the next candidate. I'm going to create a
new candidate once this JIRA is resolved.

Cheers,
Tom

On Mon, Aug 9, 2010 at 5:38 AM, Rohan Rai <rohan.rai@inmobi.com> wrote:
> Hi
>
> http://people.apache.org/~tomwhite/hadoop-0.21.0-candidate-0/
>
> Do we have any more candidate release after this.
>
> Thanking You
> Regards
> Rohan
>
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify us
> immediately by responding to this email and then delete it from your system.
> The firm is neither liable for the proper and complete transmission of the
> information contained in this communication nor for any delay in its
> receipt.
>

From general-return-1910-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 09 16:21:38 2010
Return-Path: <general-return-1910-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 26905 invoked from network); 9 Aug 2010 16:21:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Aug 2010 16:21:38 -0000
Received: (qmail 25865 invoked by uid 500); 9 Aug 2010 16:21:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25807 invoked by uid 500); 9 Aug 2010 16:21:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25799 invoked by uid 99); 9 Aug 2010 16:21:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 16:21:36 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 16:21:29 +0000
Received: by bwz9 with SMTP id 9so2189025bwz.35
        for <general@hadoop.apache.org>; Mon, 09 Aug 2010 09:21:07 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.126.82 with SMTP id b18mr3190902bks.124.1281370865763; 
	Mon, 09 Aug 2010 09:21:05 -0700 (PDT)
Received: by 10.204.179.74 with HTTP; Mon, 9 Aug 2010 09:21:05 -0700 (PDT)
In-Reply-To: <AANLkTi=GyFLivH8uOQNppUJnVeixpY5HA98=0cJEpaNa@mail.gmail.com>
References: <AANLkTi=GyFLivH8uOQNppUJnVeixpY5HA98=0cJEpaNa@mail.gmail.com>
Date: Mon, 9 Aug 2010 09:21:05 -0700
Message-ID: <AANLkTi=LrY0uOou-auud3QCA4BiZW39qNeEUC0Wjh=rR@mail.gmail.com>
Subject: Re: [DISCUSSION] Combine committer lists in Common/HDFS/MapReduce
From: Tom White <tom@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

+1 to having a single committer list for Common, HDFS, and MapReduce.
The projects are currently closely aligned and there have been cases
where a committer couldn't commit what was logically a single patch
because it was split between Common and MapReduce.

I think we should make this change regardless of whether we ultimately
split the projects into TLPs, we merge them back into a single
project, or we keep them as three subprojects within the Hadoop TLP
(as they are now). I would also suggest that the question of what to
do about the project split, although related, probably deserves a
separate discussion.

Tom

On Fri, Aug 6, 2010 at 2:02 PM, Chris Douglas <cdouglas@apache.org> wrote:
> Hadoop developers tend to specialize in either HDFS or MapReduce, but
> given that:
>
> 0) Granting karma to Common is routine for a committer in either
> space; there are no Common-only committers
> 1) The majority of committers have been grandfathered into committer
> roles in all three projects
> 2) Many patches to Common require corresponding commits to both HDFS
> and MapReduce
> 3) Review-then-commit is usually sufficient notice for interested
> parties to comment
> 4) There have been few problems with committers pushing in patches
> without consulting someone more directly involved
> 5) Everyone on the PMC gets commit rights to all subprojects, anyway
>
> Perhaps it would make sense to give up on separate committer roles
> until the projects are separate TLPs.
>
> On the other hand:
>
> 0) Nobody has been independently added to both HDFS and MapReduce
> since the projects were separated
> 1) It could exacerbate the focus on MapReduce in HDFS, at the expense
> of other projects (like HBase).
> 2) HDFS and MapReduce are mostly independent communities and
> codebases; expertise in one does not imply fluency in the other
> 3) Granting veto power across projects can lead to deadlock despite
> consensus within that community
> 4) TLP status for either project may require untangling HDFS/MR roles
> that could be distinguished now
>
> Personally, I'm in favor of combining the roles. I trust all six of
> the committers made since the project split no less than those made
> earlier. Further, version control is sufficient for recovering from
> most, foreseeable issues. I have some concerns about "harmless"
> commits pushed through without an audit by the subproject's
> maintainers (a few in recent memory caused downtime in Y! clusters),
> but combining the roles seems like a worthwhile experiment.
>
> Thoughts? -C
>

From general-return-1911-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 09 16:24:38 2010
Return-Path: <general-return-1911-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29039 invoked from network); 9 Aug 2010 16:24:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Aug 2010 16:24:37 -0000
Received: (qmail 31562 invoked by uid 500); 9 Aug 2010 16:24:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31504 invoked by uid 500); 9 Aug 2010 16:24:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31496 invoked by uid 99); 9 Aug 2010 16:24:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 16:24:36 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.62.80] (HELO qmta08.westchester.pa.mail.comcast.net) (76.96.62.80)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 16:24:27 +0000
Received: from omta23.westchester.pa.mail.comcast.net ([76.96.62.74])
	by qmta08.westchester.pa.mail.comcast.net with comcast
	id sBkl1e0041c6gX858GQ7es; Mon, 09 Aug 2010 16:24:07 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta23.westchester.pa.mail.comcast.net with comcast
	id sGPy1e0052SbwD53jGQ1gS; Mon, 09 Aug 2010 16:24:05 +0000
Message-Id: <86CD5A62-7982-44DD-A03C-D4D1B5546088@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <97E2C026-0C44-4FDC-A569-63E218E03D59@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSSION] Combine committer lists in Common/HDFS/MapReduce
Date: Mon, 9 Aug 2010 09:23:56 -0700
References: <AANLkTi=GyFLivH8uOQNppUJnVeixpY5HA98=0cJEpaNa@mail.gmail.com> <4C5C8302.3010008@apache.org> <97E2C026-0C44-4FDC-A569-63E218E03D59@yahoo-inc.com>
X-Mailer: Apple Mail (2.936)


On Aug 8, 2010, at 12:21 PM, Arun C Murthy wrote:

> This of course begs a larger question - should we just merge Common,  
> HDFS & Map-Reduce together and be done with?

*Sigh* I wish we'd just split the mailing lists, source code, and  
artifacts (jars, documentation) and left it in a single project.  
Basically, we should have set it up as:

http://svn.apache.org/repos/asf/hadoop/trunk/{common,hdfs,mapreduce}

It would certainly make it easier to deal with cross-project patches.

But, that said, I'm not wild about going through another cutoff where  
we lose all of our history in git. (Subversion tracks through the  
split but the svn to git gateway views them as new files.) I really  
wish we could just move to git for commits as well as for reads.

I'm also concerned about spinning for the sake of spinning. Doing more  
major code movement will cause more instability in trunk that I'm not  
sure is a good idea without clear goals and roadmaps.

-- Owen

From general-return-1912-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 09 16:26:43 2010
Return-Path: <general-return-1912-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31260 invoked from network); 9 Aug 2010 16:26:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Aug 2010 16:26:42 -0000
Received: (qmail 38746 invoked by uid 500); 9 Aug 2010 16:26:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38609 invoked by uid 500); 9 Aug 2010 16:26:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 38601 invoked by uid 99); 9 Aug 2010 16:26:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 16:26:40 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 09 Aug 2010 16:26:38 +0000
Received: (qmail 29894 invoked by uid 99); 9 Aug 2010 16:26:16 -0000
Received: from localhost.apache.org (HELO [192.168.168.103]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 16:26:16 +0000
Message-ID: <4C602C27.6060802@apache.org>
Date: Mon, 09 Aug 2010 09:26:15 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.11) Gecko/20100713 Thunderbird/3.0.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSSION] Combine committer lists in Common/HDFS/MapReduce
References: <AANLkTi=GyFLivH8uOQNppUJnVeixpY5HA98=0cJEpaNa@mail.gmail.com> <4C5C8302.3010008@apache.org> <97E2C026-0C44-4FDC-A569-63E218E03D59@yahoo-inc.com>
In-Reply-To: <97E2C026-0C44-4FDC-A569-63E218E03D59@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 08/08/2010 12:21 PM, Arun C Murthy wrote:
> This of course begs a larger question - should we just merge Common,
> HDFS & Map-Reduce together and be done with?

I think there's still a reasonable long-term goal to split MapReduce 
from HDFS, so that they can release separately and are maintained by 
separate teams.  So I believe a strong division of these code trees and 
release artifacts should remain.

I'd like to get rid of Common.  It could either be merged into HDFS or 
gradually whittled away to nothing.  I'd prefer the latter.  If we move 
to different RPC and serialization systems (e.g., Avro) then Common's 
io, and ipc packages might be removed.  Configuration might be 
replaced/merged with Jakarta Commons Configuration 
(http://commons.apache.org/configuration/).  Similarly, the metrics and 
fs packages might be moved to Jakarta Commons.  Such changes might be 
hard to do back-compatibly, however.

I don't see that merging the Jira databases or mailing lists for HDFS 
and MapReduce offers big advantages.  The redundant, coordinated Jira's 
tend to be between Common the others, no?

Doug

From general-return-1913-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 09 20:53:21 2010
Return-Path: <general-return-1913-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 52216 invoked from network); 9 Aug 2010 20:53:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Aug 2010 20:53:21 -0000
Received: (qmail 23046 invoked by uid 500); 9 Aug 2010 20:53:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22937 invoked by uid 500); 9 Aug 2010 20:53:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22780 invoked by uid 99); 9 Aug 2010 20:53:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 20:53:18 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 20:53:13 +0000
Received: by wyb35 with SMTP id 35so9709468wyb.35
        for <general@hadoop.apache.org>; Mon, 09 Aug 2010 13:52:50 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        bh=z0GabpTuhekfwdeINznjkd1QLcQjc086TqIS2+Vi8cA=;
        b=lNbIkTiyT5S0wyqZoOvHVi1bSX32BFG+hbr/ikYNNv3EdLgxCtgILI9NBe8p0qFj/o
         q7cd2ZF1aziWaH2Pu+q8x0s7L5vbXMbb4OWwi0g1fuRkLirYxU472hNipGzVbB28qQ+P
         j60b+7SAQvdap6QMQtLxa46gTzvE1YVTUJ7t0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        b=evK5SNgcOEihRvDuSG+eeDp+4Ff4oD6bdtl7Qjgx3VoXevB7Jczkaese5zSTWZdi6/
         n4+HBNcXyxqYSGvhav9e9h0wzhEcpZC0e/HVJ4JWZ8q64DhVyyjNFYKvxgc6romf7Qim
         ObzzWmnbm6DcguWnF68/QmloMTMk5lYw/ttW4=
MIME-Version: 1.0
Received: by 10.227.138.5 with SMTP id y5mr14275734wbt.204.1281387170686; Mon, 
	09 Aug 2010 13:52:50 -0700 (PDT)
Sender: saint.ack@gmail.com
Received: by 10.216.168.73 with HTTP; Mon, 9 Aug 2010 13:52:50 -0700 (PDT)
Date: Mon, 9 Aug 2010 13:52:50 -0700
X-Google-Sender-Auth: 9mkKDURgH6M7xQcAh2MO9MzcUOs
Message-ID: <AANLkTi=J2gmvO4tm6o0MA3H+spXyO4kj4fyBidx_yt6U@mail.gmail.com>
Subject: [ANN] HBase-0.89.20100726, our second 'developer release', is now 
	available for download
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org, user@hbase.apache.org
Content-Type: text/plain; charset=ISO-8859-1

The HBase Team would like to announce the availability of the second
in our 'developer release' series, hbase-0.89.20100726.  Source binary
and source tar balls are available here:

  http://www.apache.org/dyn/closer.cgi/hbase/

You can browse the documentation here:

 http://hbase.apache.org/docs/r0.89.20100726/

Issues resolved since 0.89.20100621, our first 0.89.x release, are ~80
including our first cut at a working replication: See
http://su.pr/2lZsHk for the complete list.

Our 'developer releases' are for those who can tolerate risk and who
are willing to give feedback in advance of our next major release.
We're not making any guarantees that this is bug free. Its definitely
not for production deploys.  See
http://wiki.apache.org/hadoop/Hbase/HBaseVersions for more on the new
hbase versoning and our developer releases series.

Thanks to all who contributed to this release.

Yours,
St.Ack

From general-return-1914-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 09 21:33:05 2010
Return-Path: <general-return-1914-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72339 invoked from network); 9 Aug 2010 21:33:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Aug 2010 21:33:05 -0000
Received: (qmail 75367 invoked by uid 500); 9 Aug 2010 21:33:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75279 invoked by uid 500); 9 Aug 2010 21:33:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75271 invoked by uid 99); 9 Aug 2010 21:33:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 21:33:03 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of renatoj.marroquin@gmail.com designates 209.85.213.176 as permitted sender)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 21:32:56 +0000
Received: by yxm34 with SMTP id 34so5521735yxm.35
        for <general@hadoop.apache.org>; Mon, 09 Aug 2010 14:32:35 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=XkRa8C5SVF0XdIO1ufiAa8yWueGdNujTH5Wy/xJBRaQ=;
        b=t+csTQ6GavHva2d4RMUYtg6nAaAKV0Q5PORv/RbUeUo9EQEBztDLu0tAk5zfYwjeM5
         B0CGE6mYJihrT9TW/7cIB8z8pYyOALqAlO/rzlGJflXtp7QK/IC2MXG5Aaha19YrPsU8
         /iezG4byp60DmFdafHlP8Eiq/fR1DzcAfJgk0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=d9lI/3+cqxdmC6RBBT3Z4ckvokOHjLr0DuiKKs6TSnR+JLzCqwIObgBIbEQeBUwytN
         kYNTHMKvbBvI0w2WHZZ/7s+INjHYvL6fseP4yMqq6Y3Lmwp8+UTfYBhBbC0wtQhBzGQA
         36vUf0fB9ajG5s5Nhf6hY/7KMS5pmodYlHuOs=
MIME-Version: 1.0
Received: by 10.90.27.19 with SMTP id a19mr2389425aga.54.1281389555065; Mon, 
	09 Aug 2010 14:32:35 -0700 (PDT)
Received: by 10.231.15.73 with HTTP; Mon, 9 Aug 2010 14:32:34 -0700 (PDT)
In-Reply-To: <AANLkTinfnCioatFRCQd2GePuyx6UGKJOx+br5Tj5yqtT@mail.gmail.com>
References: <AANLkTinbKV8nOtDyzNoURrFjJLz=M8gr6bLtg1TeZLqZ@mail.gmail.com>
	<AANLkTinfnCioatFRCQd2GePuyx6UGKJOx+br5Tj5yqtT@mail.gmail.com>
Date: Mon, 9 Aug 2010 16:32:34 -0500
Message-ID: <AANLkTinSRNvC+W01GsoZ9B6b4oyCbCyjS2ZdUU2LXtQh@mail.gmail.com>
Subject: Re: Hadoop node's locality
From: =?ISO-8859-1?Q?Renato_Marroqu=EDn_Mogrovejo?= <renatoj.marroquin@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e65096d63fa3a7048d6ac1f4
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e65096d63fa3a7048d6ac1f4
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi, thanks for answering, but I did find the answer out. I knew Hadoop did
it, but I needed a reference.
I found it in "Data-intensive Text Processing with MapReduce" in the page
25.
I quoted here:

Data/code co-location: . . . "An important optimization here is to prefer
nodes that are on the same rack in the datacenter as the node holding the
relevant data block, since inter-rack bandwidth is significantly less than
intra-rack bandwidth."

Thanks again.


Renato M.


2010/8/9 vaibhav negi <sssssssenator@gmail.com>

> Hi ,
>
> I am also a nebie to hadoop. But i think u should read about "rack aware"
> concept.
> It may solve your query.
>
> Vaibhav Negi
>
>
> On Sun, Aug 8, 2010 at 12:10 AM, Renato Marroqu=EDn Mogrovejo <
> renatoj.marroquin@gmail.com> wrote:
>
> > Hi everyone,
> > Does anybody know if Hadoop can take scheduling decisions using nodes
> > positions
> > within network topology ? Or if there is any work being done on this?
> > Thanks in advanced.
> >
> >
> > Renato M.
> >
>

--0016e65096d63fa3a7048d6ac1f4--

From general-return-1915-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 09 22:16:18 2010
Return-Path: <general-return-1915-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85751 invoked from network); 9 Aug 2010 22:16:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Aug 2010 22:16:18 -0000
Received: (qmail 15786 invoked by uid 500); 9 Aug 2010 22:16:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15538 invoked by uid 500); 9 Aug 2010 22:16:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15530 invoked by uid 99); 9 Aug 2010 22:16:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 22:16:16 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ryanobjc@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 22:16:09 +0000
Received: by wyb35 with SMTP id 35so9822031wyb.35
        for <general@hadoop.apache.org>; Mon, 09 Aug 2010 15:15:48 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=0Pn/GEe7v0kvsEorMuv6ohZKLPCRXaqFNY0mlfWy1DM=;
        b=I1krujtotzrE4wZuJ9ZUlOhMQnUoHe//clYQyJ3dOustr+XSBsF5vr0CElMENpP0PT
         AX5BDLX9Yd2IfdiI9jGwycgH3z+3pbCy/ADd1kjth7HETDGVxAvSpw7NDp3uN4CrumUC
         sg0GRlIVHVroI3ujdzo6kZbOd5QCavSFJvwnQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=U7pvCpvnNrwV5Op0XNt66Exvui3fiqAJTUpHb67zuLYTkfqEfuvR9azVrotnyhTuCk
         rLRrz1wJ5RbkQ6Gzn6pxdph3b2l62VyB/06m5MvupSTMz/4yjwEoASDbxotJIdOMImsR
         83vYxmq3EQOYLa8dI5wK3aJMFW5BfB2vEQoi8=
MIME-Version: 1.0
Received: by 10.216.232.90 with SMTP id m68mr14376757weq.10.1281392148332; 
	Mon, 09 Aug 2010 15:15:48 -0700 (PDT)
Received: by 10.216.160.71 with HTTP; Mon, 9 Aug 2010 15:15:48 -0700 (PDT)
In-Reply-To: <AANLkTinSRNvC+W01GsoZ9B6b4oyCbCyjS2ZdUU2LXtQh@mail.gmail.com>
References: <AANLkTinbKV8nOtDyzNoURrFjJLz=M8gr6bLtg1TeZLqZ@mail.gmail.com>
	<AANLkTinfnCioatFRCQd2GePuyx6UGKJOx+br5Tj5yqtT@mail.gmail.com>
	<AANLkTinSRNvC+W01GsoZ9B6b4oyCbCyjS2ZdUU2LXtQh@mail.gmail.com>
Date: Mon, 9 Aug 2010 15:15:48 -0700
Message-ID: <AANLkTikUy4riTmW2yj+h9kTKf9Yp3+r_QoAEgpADz9c3@mail.gmail.com>
Subject: Re: Hadoop node's locality
From: Ryan Rawson <ryanobjc@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi,

The map-reduce scheduling does take block placement into account and
attempts to schedule map tasks accordingly.  The system is flexible,
and HBase uses it to put maps colocated with the regionservers they
read from.

-ryan

On Mon, Aug 9, 2010 at 2:32 PM, Renato Marroqu=EDn Mogrovejo
<renatoj.marroquin@gmail.com> wrote:
> Hi, thanks for answering, but I did find the answer out. I knew Hadoop di=
d
> it, but I needed a reference.
> I found it in "Data-intensive Text Processing with MapReduce" in the page
> 25.
> I quoted here:
>
> Data/code co-location: . . . "An important optimization here is to prefer
> nodes that are on the same rack in the datacenter as the node holding the
> relevant data block, since inter-rack bandwidth is significantly less tha=
n
> intra-rack bandwidth."
>
> Thanks again.
>
>
> Renato M.
>
>
> 2010/8/9 vaibhav negi <sssssssenator@gmail.com>
>
>> Hi ,
>>
>> I am also a nebie to hadoop. But i think u should read about "rack aware=
"
>> concept.
>> It may solve your query.
>>
>> Vaibhav Negi
>>
>>
>> On Sun, Aug 8, 2010 at 12:10 AM, Renato Marroqu=EDn Mogrovejo <
>> renatoj.marroquin@gmail.com> wrote:
>>
>> > Hi everyone,
>> > Does anybody know if Hadoop can take scheduling decisions using nodes
>> > positions
>> > within network topology ? Or if there is any work being done on this?
>> > Thanks in advanced.
>> >
>> >
>> > Renato M.
>> >
>>
>

From general-return-1916-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 00:41:05 2010
Return-Path: <general-return-1916-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43786 invoked from network); 10 Aug 2010 00:41:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 00:41:05 -0000
Received: (qmail 961 invoked by uid 500); 10 Aug 2010 00:41:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 829 invoked by uid 500); 10 Aug 2010 00:41:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 821 invoked by uid 99); 10 Aug 2010 00:41:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 00:41:03 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 00:40:58 +0000
Received: from [127.0.0.1] (gentlepaint-lx.corp.yahoo.com [10.72.185.127])
	by mrout2.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7A0eFOC057012
	for <general@hadoop.apache.org>; Mon, 9 Aug 2010 17:40:20 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=hYWUMeXmrCuUCfRdYYsqiklhGV1S8c3kl2FX7KYCCoL6IiwknhhkC4CiFP5dP7Dm
Message-ID: <4C609FEE.1050209@yahoo-inc.com>
Date: Mon, 09 Aug 2010 17:40:14 -0700
From: Konstantin Shvachko <shv@yahoo-inc.com>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.8) Gecko/20100802 Lightning/1.0b2 Thunderbird/3.1.2
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSSION] Combine committer lists in Common/HDFS/MapReduce
References: <AANLkTi=GyFLivH8uOQNppUJnVeixpY5HA98=0cJEpaNa@mail.gmail.com> <4C5C8302.3010008@apache.org> <97E2C026-0C44-4FDC-A569-63E218E03D59@yahoo-inc.com> <4C602C27.6060802@apache.org>
In-Reply-To: <4C602C27.6060802@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit



On 8/9/2010 9:26 AM, Doug Cutting wrote:
 > On 08/08/2010 12:21 PM, Arun C Murthy wrote:
 >> This of course begs a larger question - should we just merge Common,
 >> HDFS&  Map-Reduce together and be done with?
 >
 > I think there's still a reasonable long-term goal to split MapReduce
 > from HDFS, so that they can release separately and are maintained by
 > separate teams.  So I believe a strong division of these code trees and
 > release artifacts should remain.

I think that eventually when we solve the backward compatibility issue,
with Avro, we can make MR and HDFS different products.
Even though right now they are not, it would be a step back to merge
them back.

As we look forward to making two different products, shouldn't there be
two separate sets of committers?
Of course, as long as they are not really separated it makes sense to
have single committers pool.

 > I'd like to get rid of Common.  It could either be merged into HDFS or
 > gradually whittled away to nothing.  I'd prefer the latter.  If we move
 > to different RPC and serialization systems (e.g., Avro) then Common's
 > io, and ipc packages might be removed.  Configuration might be
 > replaced/merged with Jakarta Commons Configuration
 > (http://commons.apache.org/configuration/).  Similarly, the metrics and
 > fs packages might be moved to Jakarta Commons.  Such changes might be
 > hard to do back-compatibly, however.

Great idea. We should do this.

 > I don't see that merging the Jira databases or mailing lists for HDFS
 > and MapReduce offers big advantages.  The redundant, coordinated Jira's
 > tend to be between Common the others, no?

Yes.
--Konstantin


From general-return-1917-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 05:26:20 2010
Return-Path: <general-return-1917-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6085 invoked from network); 10 Aug 2010 05:26:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 05:26:19 -0000
Received: (qmail 22788 invoked by uid 500); 10 Aug 2010 05:26:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22354 invoked by uid 500); 10 Aug 2010 05:26:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22346 invoked by uid 99); 10 Aug 2010 05:26:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 05:26:14 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [119.226.208.33] (HELO mail1.inmobi.com) (119.226.208.33)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 05:26:07 +0000
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AkcGABh/YEwKDmQH/2dsb2JhbAChQQbCL4U6BIQkhRaCTg
X-IronPort-AV: E=Sophos;i="4.55,346,1278268200"; 
   d="scan'208";a="900368"
Received: from mk-exch-1.mkhoj.com ([10.14.100.7])
  by mail1.mkhoj.com with ESMTP; 10 Aug 2010 10:55:42 +0530
Received: from [10.14.100.74] (10.14.100.74) by MK-EXCH-1.MKHOJ.COM
 (10.14.100.7) with Microsoft SMTP Server id 8.1.340.0; Tue, 10 Aug 2010
 10:52:42 +0530
Message-ID: <4C60E24F.7000100@inmobi.com>
Date: Tue, 10 Aug 2010 10:53:27 +0530
From: Rohan Rai <rohan.rai@inmobi.com>
User-Agent: Thunderbird 2.0.0.24 (X11/20100411)
MIME-Version: 1.0
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: Hadoop 0.21
References: <4C5FF6C0.5000906@inmobi.com> <AANLkTi=bRP-viDa7fg97STCceGgOGkVf7VAUYgL-pZ1c@mail.gmail.com>
In-Reply-To: <AANLkTi=bRP-viDa7fg97STCceGgOGkVf7VAUYgL-pZ1c@mail.gmail.com>
Content-Type: text/plain; charset="ISO-8859-1"; format=flowed
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Thank You Tom for the information.

Any hints on possible stable release.

Regards
Rohan

Tom White wrote:
> Hi Rohan,
>
> MAPREDUCE-1920 is blocking the next candidate. I'm going to create a
> new candidate once this JIRA is resolved.
>
> Cheers,
> Tom
>
> On Mon, Aug 9, 2010 at 5:38 AM, Rohan Rai <rohan.rai@inmobi.com> wrote:
>
>> Hi
>>
>> http://people.apache.org/~tomwhite/hadoop-0.21.0-candidate-0/
>>
>> Do we have any more candidate release after this.
>>
>> Thanking You
>> Regards
>> Rohan
>>
>> The information contained in this communication is intended solely for t=
he
>> use of the individual or entity to whom it is addressed and others
>> authorized to receive it. It may contain confidential or legally privile=
ged
>> information. If you are not the intended recipient you are hereby notifi=
ed
>> that any disclosure, copying, distribution or taking any action in relia=
nce
>> on the contents of this information is strictly prohibited and may be
>> unlawful. If you have received this communication in error, please notif=
y us
>> immediately by responding to this email and then delete it from your sys=
tem.
>> The firm is neither liable for the proper and complete transmission of t=
he
>> information contained in this communication nor for any delay in its
>> receipt.
>>
>>
>
>


The information contained in this communication is intended solely for the =
use of the individual or entity to whom it is addressed and others authoriz=
ed to receive it. It may contain confidential or legally privileged informa=
tion. If you are not the intended recipient you are hereby notified that an=
y disclosure, copying, distribution or taking any action in reliance on the=
 contents of this information is strictly prohibited and may be unlawful. I=
f you have received this communication in error, please notify us immediate=
ly by responding to this email and then delete it from your system. The fir=
m is neither liable for the proper and complete transmission of the informa=
tion contained in this communication nor for any delay in its receipt.

From general-return-1918-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 08:01:58 2010
Return-Path: <general-return-1918-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63263 invoked from network); 10 Aug 2010 08:01:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 08:01:58 -0000
Received: (qmail 32626 invoked by uid 500); 10 Aug 2010 08:01:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31906 invoked by uid 500); 10 Aug 2010 08:01:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31892 invoked by uid 99); 10 Aug 2010 08:01:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 08:01:53 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [202.43.216.211] (HELO n1b.bullet.cnb.yahoo.com) (202.43.216.211)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 10 Aug 2010 08:01:45 +0000
Received: from [202.43.217.36] by n1.bullet.cnb.yahoo.com with NNFMP; 10 Aug 2010 08:01:21 -0000
Received: from [202.165.102.49] by t1.bullet.cnmail.cnb.yahoo.com with NNFMP; 10 Aug 2010 08:01:21 -0000
Received: from [127.0.0.1] by omp103.mail.cnb.yahoo.com with NNFMP; 10 Aug 2010 08:01:21 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 921665.20330.bm@omp103.mail.cnb.yahoo.com
Received: (qmail 6579 invoked by uid 60001); 10 Aug 2010 08:01:21 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com.cn; s=s1024; t=1281427281; bh=NkFwKDbH9TBKDip5ry6xBNLbnRognzRIxiwjk4pBC18=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=bgM4muMgd05KTqAJYy9+6DGcCOqhonl0ueorSX2blvbXSKaSUp64Jeo5ez0RPL3FOKEqx2UlWQkmtFtS3jCp4ywSDjXyye19IbrGXwZmJXCaaZ7zisJ2w4mjf/7s+pxKQ3+hu0C9Rl/oI2ZuIwa1sJ+hvLMEr1IgvPu1a23cdHo=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com.cn;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=op3pkvoQzpUiB0oVdkNXT6sT/MSrfigU/6BkTG2pDgwP8ywgSx63TYQaGtJukFpeH4dzvOPeEi6MI1JIh5fc1aEqqFufrZBstQcOHodvYPw9j446IVu5Zhl1ZMaWe5mJXAA966PbUYAWKh8ywlj2aKwdQYSzSTMzK21/Xy0uiEg=;
Message-ID: <213553.84377.qm@web15204.mail.cnb.yahoo.com>
X-YMail-OSG: vuOueWgVM1mUaRaNsZEarnQyIPzJglCwrBoVHnjN1oVY3Nv
 HGHvoWor4
Received: from [202.118.67.200] by web15204.mail.cnb.yahoo.com via HTTP; Tue, 10 Aug 2010 16:01:20 CST
X-Mailer: YahooMailClassic/11.3.2 YahooMailWebService/0.8.105.279950
Date: Tue, 10 Aug 2010 16:01:20 +0800 (CST)
From: Dennis <arsenepark@yahoo.com.cn>
Subject: RawComparator
To: mapreduce-user@hadoop.apache.org, general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1336986452-1281427280=:84377"

--0-1336986452-1281427280=:84377
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Hi, guys,

I am using hadoop 0.20.2, and I am trying to run the "SecondarySort" exmapl=
e. The following is the "FirstGroupingComparator" class, and I just cannot =
figure out how "WritableComparator.compareBytes(b1, s1, Integer.SIZE / 8, b=
2, s2, Integer.SIZE / 8)" works. There are really few javadocs of this clas=
s or=A0 this method.
1. Why it is "Integer.SIZE / 8"?
2. If I want to compare two "String" here, how should I write to code?

=A0=A0=A0 public static class FirstGroupingComparator implements
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 RawComparator<IntPair> {
=A0=A0=A0 =A0=A0=A0 @Override
=A0=A0=A0 =A0=A0=A0 public int compare(byte[] b1, int s1, int l1, byte[] b2=
, int s2, int l2) {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 int ret =3D WritableComparator.compareBytes(b=
1, s1, Integer.SIZE / 8,
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 b2, s2, Integer.SIZE / 8)=
;
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return ret;
=A0=A0=A0 =A0=A0=A0 }

=A0=A0=A0 =A0=A0=A0 @Override
=A0=A0=A0 =A0=A0=A0 public int compare(IntPair o1, IntPair o2) {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 int l =3D o1.getFirst();
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 int r =3D o2.getFirst();
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return l =3D=3D r ? 0 : (l < r ? -1 : 1);
=A0=A0=A0 =A0=A0=A0 }
=A0=A0=A0 }

Thanks.
Dennis
=0A=0A=0A      
--0-1336986452-1281427280=:84377--


From general-return-1919-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 12:49:22 2010
Return-Path: <general-return-1919-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70589 invoked from network); 10 Aug 2010 12:49:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 12:49:22 -0000
Received: (qmail 78821 invoked by uid 500); 10 Aug 2010 12:49:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78423 invoked by uid 500); 10 Aug 2010 12:49:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78414 invoked by uid 99); 10 Aug 2010 12:49:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 12:49:17 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [203.209.230.21] (HELO n4c.bullet.cnb.yahoo.com) (203.209.230.21)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 10 Aug 2010 12:49:07 +0000
Received: from [202.43.217.102] by n4.bullet.cnb.yahoo.com with NNFMP; 10 Aug 2010 12:48:43 -0000
Received: from [203.209.230.75] by t2.bullet.cnb.yahoo.com with NNFMP; 10 Aug 2010 12:48:43 -0000
Received: from [127.0.0.1] by omp104.mail.cnb.yahoo.com with NNFMP; 10 Aug 2010 12:48:43 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 31034.4273.bm@omp104.mail.cnb.yahoo.com
Received: (qmail 35790 invoked by uid 60001); 10 Aug 2010 12:48:43 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com.cn; s=s1024; t=1281444522; bh=l/ch6peDsyMnax7NaDhAhcAPHz+x8VXu5YOPtTVMCps=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=5quw8HWRE9kCY+JCO7Dul7DZ/oBStHQi9kFJpWIScmv9tcMc2DFP/4OMUF5ghIlAwTPpQ5l/S7xt/l1U09B/mosqX6YaK24Rwn+l+Si26ql/y6p3bwOK8XVcW4jCw+0xTikxdr8D+egbhJZNUVPruKeOXCqrmqVuf9KTJsWLGPM=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com.cn;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=Hk/6wZ3y/JL5K+oEPxXKsYj1+MScnRZc4up39y4nOm+08qquWY7rgYlVcg1SB4FfqZoLIi6vVl+q5GS79yBjAsvO+u1qGlUR6yvL91ftZfvGcz9IJGqTv+5aK6daBcTCiXYKG6yyd65jXz9aLFXUZb7F0MAV8Wx3xydrR5zzt0Y=;
Message-ID: <918137.34655.qm@web15207.mail.cnb.yahoo.com>
X-YMail-OSG: .Ufyq4MVM1mPgIqHJhToQkI4qrr3mZJdgPmykRlM2YsKYoa
 749SyONVW.rwPX6DS6npzqx9VYrJYMoiq6aoS10nXNLkeEfXY.6KDSGOaqBV
 HzJFvHX3jYkOHrj4tKS8php_A469E.JOdiZFo8f6R0h9vB3_EJILOBBWLiCE
 6KaGlPXSrJJx6OfDZAevgNiHAHWecQJlso5JaLt2FU1iI1qCy3s4j9sU6irS
 4qWBe1Y7.0Xr6dPIHahKp2u.94J.bctITTQAR0PjupQ--
Received: from [202.118.67.200] by web15207.mail.cnb.yahoo.com via HTTP; Tue, 10 Aug 2010 05:48:42 PDT
X-Mailer: YahooMailClassic/11.3.2 YahooMailWebService/0.8.105.279950
Date: Tue, 10 Aug 2010 05:48:42 -0700 (PDT)
From: Dennis <arsenepark@yahoo.com.cn>
Subject: about "RawComparator" class
To: mapreduce-user@hadoop.apache.org, general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-11938642-1281444522=:34655"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-11938642-1281444522=:34655
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Hi, guys,

I am using hadoop 0.20.2, and I am trying to run the "SecondarySort" exmapl=
e. The following is the "FirstGroupingComparator" class, and I just cannot =
figure out how "WritableComparator.compareBytes(b1, s1, Integer.SIZE / 8, b=
2, s2, Integer.SIZE / 8)" works. There are really few javadocs of this clas=
s or=A0 this method.
1. Why it is "Integer.SIZE / 8"?
2. If I want to compare two "String" here, how should I write to code?

=A0=A0=A0 public static class FirstGroupingComparator implements
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 RawComparator<IntPair> {
=A0=A0=A0=A0=A0=A0=A0 @Override
=A0=A0=A0=A0=A0=A0=A0 public int compare(byte[] b1, int s1, int l1, byte[] =
b2, int s2, int l2) {
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 int ret =3D WritableComparator.compareByt=
es(b1, s1, Integer.SIZE / 8,
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 b2, s2, Integer.S=
IZE / 8);
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 return ret;
=A0=A0=A0=A0=A0=A0=A0 }

=A0=A0=A0=A0=A0=A0=A0 @Override
=A0=A0=A0=A0=A0=A0=A0 public int compare(IntPair o1, IntPair o2) {
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 int l =3D o1.getFirst();
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 int r =3D o2.getFirst();
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 return l =3D=3D r ? 0 : (l < r ? -1 : 1);
=A0=A0=A0=A0=A0=A0=A0 }
=A0=A0=A0 }

Thanks.
Dennis=0A=0A=0A      
--0-11938642-1281444522=:34655--


From general-return-1920-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 16:45:31 2010
Return-Path: <general-return-1920-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 71297 invoked from network); 10 Aug 2010 16:45:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 16:45:30 -0000
Received: (qmail 66171 invoked by uid 500); 10 Aug 2010 16:45:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66070 invoked by uid 500); 10 Aug 2010 16:45:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66062 invoked by uid 99); 10 Aug 2010 16:45:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 16:45:28 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 16:45:19 +0000
Received: by wyb35 with SMTP id 35so10955944wyb.35
        for <general@hadoop.apache.org>; Tue, 10 Aug 2010 09:44:59 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.227.27.69 with SMTP id h5mr15122932wbc.210.1281458699096; Tue, 
	10 Aug 2010 09:44:59 -0700 (PDT)
Received: by 10.216.20.206 with HTTP; Tue, 10 Aug 2010 09:44:54 -0700 (PDT)
In-Reply-To: <4C609FEE.1050209@yahoo-inc.com>
References: <AANLkTi=GyFLivH8uOQNppUJnVeixpY5HA98=0cJEpaNa@mail.gmail.com>
	<4C5C8302.3010008@apache.org>
	<97E2C026-0C44-4FDC-A569-63E218E03D59@yahoo-inc.com>
	<4C602C27.6060802@apache.org>
	<4C609FEE.1050209@yahoo-inc.com>
Date: Tue, 10 Aug 2010 09:44:54 -0700
Message-ID: <AANLkTi=_EYrdpZSzAZoDzHwX18gTwifDU6DZ3hhtOdv6@mail.gmail.com>
Subject: Re: [DISCUSSION] Combine committer lists in Common/HDFS/MapReduce
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=002215b01d528dd29a048d7ada75
X-Virus-Checked: Checked by ClamAV on apache.org

--002215b01d528dd29a048d7ada75
Content-Type: text/plain; charset=ISO-8859-1

> > I'd like to get rid of Common.  It could either be merged into HDFS or
> > gradually whittled away to nothing.  I'd prefer the latter.  If we move
> > to different RPC and serialization systems (e.g., Avro) then Common's
> > io, and ipc packages might be removed.  Configuration might be
> > replaced/merged with Jakarta Commons Configuration
> > (http://commons.apache.org/configuration/).  Similarly, the metrics and
> > fs packages might be moved to Jakarta Commons.  Such changes might be
> > hard to do back-compatibly, however.
>
> Great idea. We should do this.
>

Cool, created https://issues.apache.org/jira/browse/HADOOP-6909 to track.

--002215b01d528dd29a048d7ada75--

From general-return-1921-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 17:25:24 2010
Return-Path: <general-return-1921-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88654 invoked from network); 10 Aug 2010 17:25:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 17:25:24 -0000
Received: (qmail 37882 invoked by uid 500); 10 Aug 2010 17:25:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37729 invoked by uid 500); 10 Aug 2010 17:25:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37721 invoked by uid 99); 10 Aug 2010 17:25:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:25:22 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:25:15 +0000
Received: by pvg3 with SMTP id 3so2022813pvg.35
        for <general@hadoop.apache.org>; Tue, 10 Aug 2010 10:24:53 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.142.211.6 with SMTP id j6mr15026130wfg.126.1281461093689; Tue, 
	10 Aug 2010 10:24:53 -0700 (PDT)
Received: by 10.142.179.20 with HTTP; Tue, 10 Aug 2010 10:24:53 -0700 (PDT)
In-Reply-To: <4C602C27.6060802@apache.org>
References: <AANLkTi=GyFLivH8uOQNppUJnVeixpY5HA98=0cJEpaNa@mail.gmail.com>
	<4C5C8302.3010008@apache.org>
	<97E2C026-0C44-4FDC-A569-63E218E03D59@yahoo-inc.com>
	<4C602C27.6060802@apache.org>
Date: Tue, 10 Aug 2010 10:24:53 -0700
Message-ID: <AANLkTikpXTgHN7A1HUfd6cOgrossH4xW3xMQtKee5ayS@mail.gmail.com>
Subject: Re: [DISCUSSION] Combine committer lists in Common/HDFS/MapReduce
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Mon, Aug 9, 2010 at 9:26 AM, Doug Cutting <cutting@apache.org> wrote:
> On 08/08/2010 12:21 PM, Arun C Murthy wrote:
>>
>> This of course begs a larger question - should we just merge Common,
>> HDFS & Map-Reduce together and be done with?
>
> I think there's still a reasonable long-term goal to split MapReduce from
> HDFS, so that they can release separately and are maintained by separate
> teams. =A0So I believe a strong division of these code trees and release
> artifacts should remain.
>
> I'd like to get rid of Common. =A0It could either be merged into HDFS or
> gradually whittled away to nothing. =A0I'd prefer the latter. =A0If we mo=
ve to
> different RPC and serialization systems (e.g., Avro) then Common's io, an=
d
> ipc packages might be removed. =A0Configuration might be replaced/merged =
with
> Jakarta Commons Configuration (http://commons.apache.org/configuration/).
> =A0Similarly, the metrics and fs packages might be moved to Jakarta Commo=
ns.
> =A0Such changes might be hard to do back-compatibly, however.

Merging the o.a.h.fs back into the hdfs repo would be helpful.  It's a
pain to develop a file system with client and server split into
multiple repositories, and the other fs implementations probably do
not want their own repository since they need to get updated when the
clients change as well.

Developing and releasing hadoop post-project split has been a pain,
going back to two repos (merging common and hdfs) or a single repo
would make the life of people developing and releasing easier.  As you
point out, users want to consume hadoop as a single project and not
worry about common, mr and hdfs as separately released and versioned
components, so I'm not sure which community the split is serving.

Thanks,
Eli



>
> I don't see that merging the Jira databases or mailing lists for HDFS and
> MapReduce offers big advantages. =A0The redundant, coordinated Jira's ten=
d to
> be between Common the others, no?
>
> Doug
>

From general-return-1922-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 19:22:30 2010
Return-Path: <general-return-1922-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35863 invoked from network); 10 Aug 2010 19:22:29 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 19:22:29 -0000
Received: (qmail 15068 invoked by uid 500); 10 Aug 2010 19:22:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14724 invoked by uid 500); 10 Aug 2010 19:22:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14709 invoked by uid 99); 10 Aug 2010 19:22:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 19:22:26 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 19:22:21 +0000
Received: by qwd7 with SMTP id 7so12058057qwd.35
        for <multiple recipients>; Tue, 10 Aug 2010 12:21:57 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.224.103.130 with SMTP id k2mr10033090qao.63.1281468110999; 
	Tue, 10 Aug 2010 12:21:50 -0700 (PDT)
Received: by 10.229.238.194 with HTTP; Tue, 10 Aug 2010 12:21:50 -0700 (PDT)
In-Reply-To: <213553.84377.qm@web15204.mail.cnb.yahoo.com>
References: <213553.84377.qm@web15204.mail.cnb.yahoo.com>
Date: Tue, 10 Aug 2010 15:21:50 -0400
Message-ID: <AANLkTinw0_KpBn+B=NEKFTKJNuKQoFt5JB5c_0dexJCo@mail.gmail.com>
Subject: Re: RawComparator
From: Josh Patterson <josh@cloudera.com>
To: mapreduce-user@hadoop.apache.org
Cc: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000feaeadea28d6e28048d7d0b1b

--000feaeadea28d6e28048d7d0b1b
Content-Type: text/plain; charset=ISO-8859-1

Dennis,

On Tue, Aug 10, 2010 at 4:01 AM, Dennis <arsenepark@yahoo.com.cn> wrote:

> Hi, guys,
>
> I am using hadoop 0.20.2, and I am trying to run the "SecondarySort"
> exmaple. The following is the "FirstGroupingComparator" class, and I just
> cannot figure out how "WritableComparator.compareBytes(b1, s1,
> Integer.SIZE / 8, b2, s2, Integer.SIZE / 8)" works. There are really few
> javadocs of this class or  this method.
> 1. Why it is "Integer.SIZE / 8"?
>

That says "take the size of the integer in bits on this system and divide it
by 8" --- which in java on 32 and 64 bit systems should give you 32 / 8 == 4
as afaik the integer bit width doesnt change based on the architecture with
java. So its saying here "compare the first 4 bytes of each byte array" (the
width, in bytes, of the first integer in the composite key) ,whereas
Integer.SIZE gives the number of bits in the datatype.

WritableComparators are useful in the shuffle phase of hadoop; we are
constantly comparing and sorting WritableComparables, and the secondary
sorting mechanics allow us to have a group of data for a key arrive at the
reducer in a certain order (example: time series data, where we want a range
of timestamps in one group, but we also want them in order when they are
processed inside the reducer)


> 2. If I want to compare two "String" here, how should I write to code?
>
>     public static class FirstGroupingComparator implements
>             RawComparator<IntPair> {
>         @Override
>         public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2,
> int l2) {
>             int ret = WritableComparator.compareBytes(b1, s1, Integer.SIZE
> / 8,
>                     b2, s2, Integer.SIZE / 8);
>             return ret;
>         }
>
>         @Override
>         public int compare(IntPair o1, IntPair o2) {
>             int l = o1.getFirst();
>             int r = o2.getFirst();
>             return l == r ? 0 : (l < r ? -1 : 1);
>         }
>     }
>

In the case of the comparison of strings, lets say for example you have a
"composite key" that has two String or Text object members (k1, k2); We
"group by" the first part of the key k1 and we sort by this key as well (we
block ranges together). This is very similar to the example above. Since
with a RawComparator we are looking to only deserialize a portion of the
data to do the comparison, you'll need a strategy for the compare() function
that takes into account that the strings are variable length (which means we
are unable to simply read 4 bytes as in the case of the integer). The
challenge here is to only deserialize the portion of the composite key that
contains the string/text that you want to compare against, which is going to
be a variable number of bytes each time. A good place to start looking at
for ideas would be the Text class in Hadoop and also WritableUtils.

Josh Patterson
Cloudera



> Thanks.
> Dennis
>
>

--000feaeadea28d6e28048d7d0b1b--

From general-return-1923-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 01:07:47 2010
Return-Path: <general-return-1923-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78351 invoked from network); 11 Aug 2010 01:07:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 01:07:47 -0000
Received: (qmail 65635 invoked by uid 500); 11 Aug 2010 01:07:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65518 invoked by uid 500); 11 Aug 2010 01:07:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65491 invoked by uid 99); 11 Aug 2010 01:07:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 01:07:45 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [202.43.217.38] (HELO n1.bullet.cnmail.cnb.yahoo.com) (202.43.217.38)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 11 Aug 2010 01:07:35 +0000
Received: from [203.209.230.73] by n1.bullet.cnmail.cnb.yahoo.com with NNFMP; 11 Aug 2010 01:07:10 -0000
Received: from [202.165.102.49] by t3.bullet.cnb.yahoo.com with NNFMP; 11 Aug 2010 01:07:10 -0000
Received: from [127.0.0.1] by omp103.mail.cnb.yahoo.com with NNFMP; 11 Aug 2010 01:07:10 -0000
X-Yahoo-Newman-Property: ymail-5
X-Yahoo-Newman-Id: 656543.1568.bm@omp103.mail.cnb.yahoo.com
Received: (qmail 22847 invoked by uid 60001); 11 Aug 2010 01:07:10 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com.cn; s=s1024; t=1281488830; bh=XeQyBDp2gBSI5S2SwVI2YvjMsdYjLN7RgnEkSJDFJYc=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=EKuzV0yqBXqUOlTqenlrD/z8/nscWgpZYW1o0lJbwwGFf1G2d1RMR3KhDmyXBXQwumZKjhAaJVyElfZ8TV3uwiCKmiSYJ1OVcpGLVrUz/alZmGjTfrQoKIVnUtNh9EFF/mli0Ckf9S37PnfnTKMMhpRoetz/CF2xqj3zipWm2cQ=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com.cn;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=y5Pm04m+2YfwsgEYth4O+WT4GyUspmbibK3bMD26TizEC1m1lON1dPbJyCY841dSWzJiogE4A7bK/gpIbRd4XjTyglj7W/z5IwncpvSY4+/CZqKV6HICC/eHS8WKPSpaTXV8597sOuT/YXGaBBulptq1TZqnevWP52lBJLjQntw=;
Message-ID: <502939.22759.qm@web15206.mail.cnb.yahoo.com>
X-YMail-OSG: woCd9t0VM1lQkZ4mlJ0IdGj9jVocFLu9mOTpCunnRk9zKMh
 _2p.1wZpOZGhz7ihRehSJYw.EvkwW5lCcYZLv4Eakl67tFlmMqldPg3rcE2.
 ch5iFFyBuxjakscCHGK1lQU0xFz0ZelTeD1NoDbakMMhie.0gua5rSTQZlM3
 n8DrIdG9YOCG7f60NfopdVfFPTai7di6.o6PEBBw_riDpkDHuqwCCYLiihnN
 WnTP5RR.pfDetwu6RCfhCNUF_N0P34_dG_MsaxDrLcTxT5JD3tCF6idRQ9JV
 fKiUn9b5b3K9qUIo-
Received: from [202.118.67.200] by web15206.mail.cnb.yahoo.com via HTTP; Wed, 11 Aug 2010 09:07:10 CST
X-Mailer: YahooMailClassic/11.3.2 YahooMailWebService/0.8.105.279950
Date: Wed, 11 Aug 2010 09:07:10 +0800 (CST)
From: Dennis <arsenepark@yahoo.com.cn>
Subject: Re: RawComparator
To: general@hadoop.apache.org
In-Reply-To: <AANLkTinw0_KpBn+B=NEKFTKJNuKQoFt5JB5c_0dexJCo@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-435010945-1281488830=:22759"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-435010945-1281488830=:22759
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Thanks, Josh, for your professional answers.

The following is my "composite key" class. And in the Q1003InPairComparator=
.compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) method, I fi=
gured out two ways of picking up original String values and comparing. The =
first "using streams" one is using java.io.* package to deserialize the ori=
ginal String values, and the comparision is easy to do. The second "using b=
yte[]" one, I just try to deserialize the byte[] by myself, as it's not dif=
ficult to do this, and I donnot have to "new" any stream classes in java.io=
.*.
So,
1. What do you say about the two methods? Or any better ones?
2. If I use the first "using streams" one, what do I deal with the IOExcept=
ion? If an Exception is thrown, what value should I return?(In the followin=
g code, I simply returned -1. I know it's not smart to do so.)


=A0=A0=A0 public static class Q1003InPair implements WritableComparable<Q10=
03InPair> {
=A0=A0=A0 =A0=A0=A0 private String dateStr;
=A0=A0=A0 =A0=A0=A0 private String str;

=A0=A0=A0 =A0=A0=A0 public void set(String dateStr, String str) {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 this.dateStr =3D dateStr;
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 this.str =3D str;
=A0=A0=A0 =A0=A0=A0 }

=A0=A0=A0 =A0=A0=A0 public String getDateStr() {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return this.dateStr;
=A0=A0=A0 =A0=A0=A0 }

=A0=A0=A0 =A0=A0=A0 public String getStr() {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return this.str;
=A0=A0=A0 =A0=A0=A0 }

=A0=A0=A0 =A0=A0=A0 @Override
=A0=A0=A0 =A0=A0=A0 public void readFields(DataInput in) throws IOException=
 {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 dateStr =3D in.readUTF();
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 str =3D in.readUTF();
=A0=A0=A0 =A0=A0=A0 }

=A0=A0=A0 =A0=A0=A0 @Override
=A0=A0=A0 =A0=A0=A0 public void write(DataOutput out) throws IOException {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 out.writeUTF(dateStr);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 out.writeUTF(str);
=A0=A0=A0 =A0=A0=A0 }

=A0=A0=A0 =A0=A0=A0 @Override
=A0=A0=A0 =A0=A0=A0 public int hashCode() {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 final int prime =3D 31;
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 int result =3D 1;
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 result =3D prime * result
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 + ((dateStr =3D=3D null) =
? 0 : dateStr.hashCode());
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 result =3D prime * result + ((str =3D=3D null=
) ? 0 : str.hashCode());
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return result;
=A0=A0=A0 =A0=A0=A0 }

=A0=A0=A0 =A0=A0=A0 @Override
=A0=A0=A0 =A0=A0=A0 public boolean equals(Object obj) {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 if (this =3D=3D obj)
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return true;
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 if (obj =3D=3D null)
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return false;
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 if (getClass() !=3D obj.getClass())
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return false;
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 final Q1003InPair other =3D (Q1003InPair) obj=
;
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 if (dateStr =3D=3D null) {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 if (other.dateStr !=3D null)
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return false;
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 } else if (!dateStr.equals(other.dateStr))
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return false;
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 if (str =3D=3D null) {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 if (other.str !=3D null)
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return false;
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 } else if (!str.equals(other.str))
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return false;
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return true;
=A0=A0=A0 =A0=A0=A0 }

=A0=A0=A0 =A0=A0=A0 public String toString() {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 StringBuffer sb =3D new StringBuffer();
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 sb.append(this.getDateStr());
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 sb.append(",");
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 sb.append(this.getStr());
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return sb.toString();
=A0=A0=A0 =A0=A0=A0 }

=A0=A0=A0 =A0=A0=A0 public static class Q1003InPairComparator extends Writa=
bleComparator {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 public Q1003InPairComparator() {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 super(Q1003InPair.class);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 }

=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 public int compare(byte[] b1, int s1, int l1,=
 byte[] b2, int s2,
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 int l2) {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // 1. using streams
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // ByteArrayInputStream bais1 =3D n=
ull;
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // ByteArrayInputStream bais2 =3D n=
ull;
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // DataInputStream dis1 =3D null;
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // DataInputStream dis2 =3D null;
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // try {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // bais1 =3D new ByteArrayInputStre=
am(b1);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // dis1 =3D new DataInputStream(bai=
s1);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // dis1.skip(s1);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // String str1 =3D dis1.readUTF();
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // String str3 =3D dis1.readUTF();
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 //=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=
=A0=A0 =A0=A0=A0=20
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // bais2 =3D new ByteArrayInputStre=
am(b2);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // dis2 =3D new DataInputStream(bai=
s2);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // dis2.skip(s2);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // String str2 =3D dis2.readUTF();
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // String str4 =3D dis2.readUTF();
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 //=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=
=A0=A0 =A0=A0=A0=20
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // if (str1.equals(str2)) {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // return str3.compareTo(str4);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // } else {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // return str1.compareTo(str2);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // }
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // } catch(Exception e) {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // } finally {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // try {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // dis1.close();
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // } catch(Exception e) {}
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // try {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // dis2.close();
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // } catch(Exception e) {}
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // try {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // bais1.close();
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // } catch(Exception e) {}
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // try {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // bais2.close();
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // } catch(Exception e) {}
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // }
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // return -1;

=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // 2. using byte[]
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 int strLength1 =3D ((int) b1[s1]) *=
 256 + ((int) b1[s1 + 1]);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 String str1 =3D new String(b1, s1 +=
 2, strLength1);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 s1 +=3D strLength1 + 2;
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 int strLength3 =3D ((int) b1[s1]) *=
 256 + ((int) b1[s1 + 1]);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 String str3 =3D new String(b1, s1 +=
 2, strLength3);

=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 int strLength2 =3D ((int) b2[s2]) *=
 256 + ((int) b2[s2 + 1]);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 String str2 =3D new String(b2, s2 +=
 2, strLength2);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 s2 +=3D strLength2 + 2;
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 int strLength4 =3D ((int) b2[s2]) *=
 256 + ((int) b2[s2 + 1]);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 String str4 =3D new String(b2, s2 +=
 2, strLength4);

=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 if (str1.equals(str2)) {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return str3.compareTo(str=
4);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 } else {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return str1.compareTo(str=
2);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 }
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 }
=A0=A0=A0 =A0=A0=A0 }

=A0=A0=A0 =A0=A0=A0 static {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 WritableComparator.define(Q1003InPair.class,
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 new Q1003InPairComparator=
());
=A0=A0=A0 =A0=A0=A0 }

=A0=A0=A0 =A0=A0=A0 @Override
=A0=A0=A0 =A0=A0=A0 public int compareTo(Q1003InPair o) {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 if (!this.getDateStr().equals(o.getDateStr())=
) {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return this.getDateStr().equals(o.g=
etDateStr()) ? 0 : (this
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 .getDateStr().c=
ompareTo(o.getDateStr()) > 0 ? 1 : -1);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 } else if (!this.getStr().equals(o.getStr()))=
 {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return this.getStr().equals(o.getSt=
r()) ? 0 : (this.getStr()
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 .compareTo(o.ge=
tStr()) > 0 ? 1 : -1);
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 } else {
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return 0;
=A0=A0=A0 =A0=A0=A0 =A0=A0=A0 }
=A0=A0=A0 =A0=A0=A0 }
=A0=A0=A0 }

Thanks again.
Dennis


--- On Wed, 8/11/10, Josh Patterson <josh@cloudera.com> wrote:

From: Josh Patterson <josh@cloudera.com>
Subject: Re: RawComparator
To: mapreduce-user@hadoop.apache.org
Cc: general@hadoop.apache.org
Date: Wednesday, August 11, 2010, 3:21 AM

Dennis,

On Tue, Aug 10, 2010 at 4:01 AM, Dennis <arsenepark@yahoo.com.cn> wrote:

> Hi, guys,
>
> I am using hadoop 0.20.2, and I am trying to run the "SecondarySort"
> exmaple. The following is the "FirstGroupingComparator" class, and I just
> cannot figure out how "WritableComparator.compareBytes(b1, s1,
> Integer.SIZE / 8, b2, s2, Integer.SIZE / 8)" works. There are really few
> javadocs of this class or=A0 this method.
> 1. Why it is "Integer.SIZE / 8"?
>

That says "take the size of the integer in bits on this system and divide i=
t
by 8" --- which in java on 32 and 64 bit systems should give you 32 / 8 =3D=
=3D 4
as afaik the integer bit width doesnt change based on the architecture with
java. So its saying here "compare the first 4 bytes of each byte array" (th=
e
width, in bytes, of the first integer in the composite key) ,whereas
Integer.SIZE gives the number of bits in the datatype.

WritableComparators are useful in the shuffle phase of hadoop; we are
constantly comparing and sorting WritableComparables, and the secondary
sorting mechanics allow us to have a group of data for a key arrive at the
reducer in a certain order (example: time series data, where we want a rang=
e
of timestamps in one group, but we also want them in order when they are
processed inside the reducer)


> 2. If I want to compare two "String" here, how should I write to code?
>
>=A0 =A0=A0=A0public static class FirstGroupingComparator implements
>=A0 =A0 =A0 =A0 =A0 =A0=A0=A0RawComparator<IntPair> {
>=A0 =A0 =A0 =A0=A0=A0@Override
>=A0 =A0 =A0 =A0=A0=A0public int compare(byte[] b1, int s1, int l1, byte[] =
b2, int s2,
> int l2) {
>=A0 =A0 =A0 =A0 =A0 =A0=A0=A0int ret =3D WritableComparator.compareBytes(b=
1, s1, Integer.SIZE
> / 8,
>=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=A0=A0b2, s2, Integer.SIZE / 8);
>=A0 =A0 =A0 =A0 =A0 =A0=A0=A0return ret;
>=A0 =A0 =A0 =A0=A0=A0}
>
>=A0 =A0 =A0 =A0=A0=A0@Override
>=A0 =A0 =A0 =A0=A0=A0public int compare(IntPair o1, IntPair o2) {
>=A0 =A0 =A0 =A0 =A0 =A0=A0=A0int l =3D o1.getFirst();
>=A0 =A0 =A0 =A0 =A0 =A0=A0=A0int r =3D o2.getFirst();
>=A0 =A0 =A0 =A0 =A0 =A0=A0=A0return l =3D=3D r ? 0 : (l < r ? -1 : 1);
>=A0 =A0 =A0 =A0=A0=A0}
>=A0 =A0=A0=A0}
>

In the case of the comparison of strings, lets say for example you have a
"composite key" that has two String or Text object members (k1, k2); We
"group by" the first part of the key k1 and we sort by this key as well (we
block ranges together). This is very similar to the example above. Since
with a RawComparator we are looking to only deserialize a portion of the
data to do the comparison, you'll need a strategy for the compare() functio=
n
that takes into account that the strings are variable length (which means w=
e
are unable to simply read 4 bytes as in the case of the integer). The
challenge here is to only deserialize the portion of the composite key that
contains the string/text that you want to compare against, which is going t=
o
be a variable number of bytes each time. A good place to start looking at
for ideas would be the Text class in Hadoop and also WritableUtils.

Josh Patterson
Cloudera



> Thanks.
> Dennis
>
>
=0A=0A=0A      
--0-435010945-1281488830=:22759--


From general-return-1924-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 04:03:29 2010
Return-Path: <general-return-1924-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33719 invoked from network); 11 Aug 2010 04:03:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 04:03:28 -0000
Received: (qmail 63484 invoked by uid 500); 11 Aug 2010 04:03:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63023 invoked by uid 500); 11 Aug 2010 04:03:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63008 invoked by uid 99); 11 Aug 2010 04:03:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 04:03:23 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.101 as permitted sender)
Received: from [17.148.16.101] (HELO asmtpout026.mac.com) (17.148.16.101)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 04:03:16 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; charset=us-ascii
Received: from [10.0.1.5] (S010600179a276713.ok.shawcable.net [24.71.145.57])
 by asmtp026.mac.com
 (Sun Java(tm) System Messaging Server 6.3-8.01 (built Dec 16 2008; 32bit))
 with ESMTPSA id <0L6Y00CO5Z8U1D30@asmtp026.mac.com> for
 general@hadoop.apache.org; Tue, 10 Aug 2010 21:02:55 -0700 (PDT)
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0
 reason=mlx engine=6.0.2-1004200000 definitions=main-1008100266
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.0.10011,1.0.148,0.0.0000
 definitions=2010-08-11_02:2010-08-11,2010-08-11,1970-01-01 signatures=0
Subject: Re: [DISCUSSION] Combine committer lists in Common/HDFS/MapReduce
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <AANLkTi=GyFLivH8uOQNppUJnVeixpY5HA98=0cJEpaNa@mail.gmail.com>
Date: Tue, 10 Aug 2010 21:02:54 -0700
Message-id: <5C1597A8-3413-4858-A036-0629FCE7D8E9@mac.com>
References: <AANLkTi=GyFLivH8uOQNppUJnVeixpY5HA98=0cJEpaNa@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1081)

+1 on merging the committer lists for now.  I think separating them was trying to solve for problems that we don't actually have in our community.  We can always separate them again later if needed.

n.

On Aug 6, 2010, at 2:02 PM, Chris Douglas wrote:

> Hadoop developers tend to specialize in either HDFS or MapReduce, but
> given that:
> 
> 0) Granting karma to Common is routine for a committer in either
> space; there are no Common-only committers
> 1) The majority of committers have been grandfathered into committer
> roles in all three projects
> 2) Many patches to Common require corresponding commits to both HDFS
> and MapReduce
> 3) Review-then-commit is usually sufficient notice for interested
> parties to comment
> 4) There have been few problems with committers pushing in patches
> without consulting someone more directly involved
> 5) Everyone on the PMC gets commit rights to all subprojects, anyway
> 
> Perhaps it would make sense to give up on separate committer roles
> until the projects are separate TLPs.
> 
> On the other hand:
> 
> 0) Nobody has been independently added to both HDFS and MapReduce
> since the projects were separated
> 1) It could exacerbate the focus on MapReduce in HDFS, at the expense
> of other projects (like HBase).
> 2) HDFS and MapReduce are mostly independent communities and
> codebases; expertise in one does not imply fluency in the other
> 3) Granting veto power across projects can lead to deadlock despite
> consensus within that community
> 4) TLP status for either project may require untangling HDFS/MR roles
> that could be distinguished now
> 
> Personally, I'm in favor of combining the roles. I trust all six of
> the committers made since the project split no less than those made
> earlier. Further, version control is sufficient for recovering from
> most, foreseeable issues. I have some concerns about "harmless"
> commits pushed through without an audit by the subproject's
> maintainers (a few in recent memory caused downtime in Y! clusters),
> but combining the roles seems like a worthwhile experiment.
> 
> Thoughts? -C


From general-return-1925-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 05:05:38 2010
Return-Path: <general-return-1925-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50697 invoked from network); 11 Aug 2010 05:05:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 05:05:38 -0000
Received: (qmail 99030 invoked by uid 500); 11 Aug 2010 05:05:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98872 invoked by uid 500); 11 Aug 2010 05:05:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98864 invoked by uid 99); 11 Aug 2010 05:05:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 05:05:32 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 05:05:26 +0000
Received: from [192.168.1.71] ([10.73.152.253])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7B54p2X080337
	for <general@hadoop.apache.org>; Tue, 10 Aug 2010 22:04:51 -0700 (PDT)
Message-Id: <A7CB92D6-81C3-4550-BF38-FB02FD3516C6@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <4C602C27.6060802@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSSION] Combine committer lists in Common/HDFS/MapReduce
Date: Tue, 10 Aug 2010 22:04:51 -0700
References: <AANLkTi=GyFLivH8uOQNppUJnVeixpY5HA98=0cJEpaNa@mail.gmail.com> <4C5C8302.3010008@apache.org> <97E2C026-0C44-4FDC-A569-63E218E03D59@yahoo-inc.com> <4C602C27.6060802@apache.org>
X-Mailer: Apple Mail (2.936)


On Aug 9, 2010, at 9:26 AM, Doug Cutting wrote:

> On 08/08/2010 12:21 PM, Arun C Murthy wrote:
>> This of course begs a larger question - should we just merge Common,
>> HDFS & Map-Reduce together and be done with?
>
> I think there's still a reasonable long-term goal to split MapReduce
> from HDFS, so that they can release separately and are maintained by
> separate teams.  So I believe a strong division of these code trees  
> and
> release artifacts should remain.

It is clear from the comments on this thread that the move to split  
HDFS and Map-Reduce into separate projects has been a mixed bag -  
maybe a net negative.

It has caused issues we as a community haven't dealt well with.

So, we need to make a decision - do we stick the course, get through  
the painful process and emerge stronger or do we take a step back and  
regress a decision we made back then? The current proposal to merge  
committer lists seems like a regression.

Arun




From general-return-1926-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 05:36:46 2010
Return-Path: <general-return-1926-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57688 invoked from network); 11 Aug 2010 05:36:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 05:36:45 -0000
Received: (qmail 23743 invoked by uid 500); 11 Aug 2010 05:36:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23362 invoked by uid 500); 11 Aug 2010 05:36:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23354 invoked by uid 99); 11 Aug 2010 05:36:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 05:36:41 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 05:36:35 +0000
Received: from [10.66.72.163] (sightbusy-lx.eglbp.corp.yahoo.com [10.66.72.163])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7B5ZQEW031435
	for <general@hadoop.apache.org>; Tue, 10 Aug 2010 22:35:27 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=hxe/EcvTnOQNPB2sQwTbi3OVimVJrmSrxVzU8kgS/v3oHYw2QdYBiLoEbcSOfkCP
Message-ID: <4C62369D.6000703@yahoo-inc.com>
Date: Wed, 11 Aug 2010 11:05:25 +0530
From: Vinod KV <vinodkv@yahoo-inc.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.11) Gecko/20100713 Lightning/1.0b1 Thunderbird/3.0.6
MIME-Version: 1.0
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: [DISCUSSION] Combine committer lists in Common/HDFS/MapReduce
References: <AANLkTi=GyFLivH8uOQNppUJnVeixpY5HA98=0cJEpaNa@mail.gmail.com> <4C5C8302.3010008@apache.org> <97E2C026-0C44-4FDC-A569-63E218E03D59@yahoo-inc.com> <4C602C27.6060802@apache.org> <A7CB92D6-81C3-4550-BF38-FB02FD3516C6@yahoo-inc.com>
In-Reply-To: <A7CB92D6-81C3-4550-BF38-FB02FD3516C6@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org


I know of very few to nearly zero number of patches in mapreduce that 
touch HDFS also. And vice versa. The common case is of patches that 
touch both common and mapreduce or common and hdfs. I am one of those 
who has access to mapreduce project but not to common project and find 
more than 50% of the patches that fall into this category.

(*) May be then we should simply knock off the separate list for common 
project and maintain separate lists for mapreduce and hdfs members of 
which will automatically have karma for the common project.

As for the separation of the repositories, I personally felt separation 
of mapreduce from hdfs helped focusing on things a lot better. The last 
gasp work done for 0.21, mostly by Tom, did help a lot in decoupling the 
projects. Common is the hot point, sure, but as others noted, that is a 
separate discussion.

My few cents.

+vinod


On Wednesday 11 August 2010 10:34 AM, Arun C Murthy wrote:
> On Aug 9, 2010, at 9:26 AM, Doug Cutting wrote:
>
>    
>> On 08/08/2010 12:21 PM, Arun C Murthy wrote:
>>      
>>> This of course begs a larger question - should we just merge Common,
>>> HDFS&  Map-Reduce together and be done with?
>>>        
>> I think there's still a reasonable long-term goal to split MapReduce
>> from HDFS, so that they can release separately and are maintained by
>> separate teams.  So I believe a strong division of these code trees
>> and
>> release artifacts should remain.
>>      
> It is clear from the comments on this thread that the move to split
> HDFS and Map-Reduce into separate projects has been a mixed bag -
> maybe a net negative.
>
> It has caused issues we as a community haven't dealt well with.
>
> So, we need to make a decision - do we stick the course, get through
> the painful process and emerge stronger or do we take a step back and
> regress a decision we made back then? The current proposal to merge
> committer lists seems like a regression.
>
> Arun
>
>
>
>
>    


From general-return-1927-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 15:07:50 2010
Return-Path: <general-return-1927-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 80080 invoked from network); 11 Aug 2010 15:07:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 15:07:49 -0000
Received: (qmail 53800 invoked by uid 500); 11 Aug 2010 15:07:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53547 invoked by uid 500); 11 Aug 2010 15:07:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53539 invoked by uid 99); 11 Aug 2010 15:07:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 15:07:46 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 15:07:39 +0000
Received: by qyk34 with SMTP id 34so298210qyk.14
        for <general@hadoop.apache.org>; Wed, 11 Aug 2010 08:07:18 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.224.20.13 with SMTP id d13mr11016672qab.258.1281539238099; 
	Wed, 11 Aug 2010 08:07:18 -0700 (PDT)
Received: by 10.229.238.194 with HTTP; Wed, 11 Aug 2010 08:07:18 -0700 (PDT)
In-Reply-To: <502939.22759.qm@web15206.mail.cnb.yahoo.com>
References: <AANLkTinw0_KpBn+B=NEKFTKJNuKQoFt5JB5c_0dexJCo@mail.gmail.com>
	<502939.22759.qm@web15206.mail.cnb.yahoo.com>
Date: Wed, 11 Aug 2010 11:07:18 -0400
Message-ID: <AANLkTik4pOa2dFwyxFgV4Gzy4CoVR1g4QbOw+MmPsH9_@mail.gmail.com>
Subject: Re: RawComparator
From: Josh Patterson <josh@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Dennis,
Well, in general, consider this: this method is getting called per K/V
pair potentially billions or even trillions of times. We want to do
the minimum amount of work that we can here and get the most results
to get the best performance, right?

If we think about the lower levels of memory and using the stack, we
want to allocate as little memory as possible since at the rate we'd
be creating new objects on the heap we'd really be tearing through
some memory. Without doing testing myself (which you should do, this
is easy to test, data always dominates these discussions), I'm going
to bet that the byte[] manipulations are going to be more performant.
If you can pull out the number of bytes (from the first byte or two,
like Text does it) from the front of the byte array, then you can
deserialize only the part of the composite key you are interested in.
If you can do this without creating new objects on the heap, I'd say
you are going to get better performance (just like as in the previous
example where we deserialized the first 4 bytes of the integer, that
made up the int pair).

Josh Patterson
Cloudera

On Tue, Aug 10, 2010 at 9:07 PM, Dennis <arsenepark@yahoo.com.cn> wrote:
> Thanks, Josh, for your professional answers.
>
> The following is my "composite key" class. And in the Q1003InPairComparat=
or.compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) method, I =
figured out two ways of picking up original String values and comparing. Th=
e first "using streams" one is using java.io.* package to deserialize the o=
riginal String values, and the comparision is easy to do. The second "using=
 byte[]" one, I just try to deserialize the byte[] by myself, as it's not d=
ifficult to do this, and I donnot have to "new" any stream classes in java.=
io.*.
> So,
> 1. What do you say about the two methods? Or any better ones?
> 2. If I use the first "using streams" one, what do I deal with the IOExce=
ption? If an Exception is thrown, what value should I return?(In the follow=
ing code, I simply returned -1. I know it's not smart to do so.)
>
>
> =A0=A0=A0 public static class Q1003InPair implements WritableComparable<Q=
1003InPair> {
> =A0=A0=A0 =A0=A0=A0 private String dateStr;
> =A0=A0=A0 =A0=A0=A0 private String str;
>
> =A0=A0=A0 =A0=A0=A0 public void set(String dateStr, String str) {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 this.dateStr =3D dateStr;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 this.str =3D str;
> =A0=A0=A0 =A0=A0=A0 }
>
> =A0=A0=A0 =A0=A0=A0 public String getDateStr() {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return this.dateStr;
> =A0=A0=A0 =A0=A0=A0 }
>
> =A0=A0=A0 =A0=A0=A0 public String getStr() {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return this.str;
> =A0=A0=A0 =A0=A0=A0 }
>
> =A0=A0=A0 =A0=A0=A0 @Override
> =A0=A0=A0 =A0=A0=A0 public void readFields(DataInput in) throws IOExcepti=
on {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 dateStr =3D in.readUTF();
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 str =3D in.readUTF();
> =A0=A0=A0 =A0=A0=A0 }
>
> =A0=A0=A0 =A0=A0=A0 @Override
> =A0=A0=A0 =A0=A0=A0 public void write(DataOutput out) throws IOException =
{
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 out.writeUTF(dateStr);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 out.writeUTF(str);
> =A0=A0=A0 =A0=A0=A0 }
>
> =A0=A0=A0 =A0=A0=A0 @Override
> =A0=A0=A0 =A0=A0=A0 public int hashCode() {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 final int prime =3D 31;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 int result =3D 1;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 result =3D prime * result
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 + ((dateStr =3D=3D null=
) ? 0 : dateStr.hashCode());
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 result =3D prime * result + ((str =3D=3D nu=
ll) ? 0 : str.hashCode());
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return result;
> =A0=A0=A0 =A0=A0=A0 }
>
> =A0=A0=A0 =A0=A0=A0 @Override
> =A0=A0=A0 =A0=A0=A0 public boolean equals(Object obj) {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 if (this =3D=3D obj)
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return true;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 if (obj =3D=3D null)
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return false;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 if (getClass() !=3D obj.getClass())
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return false;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 final Q1003InPair other =3D (Q1003InPair) o=
bj;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 if (dateStr =3D=3D null) {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 if (other.dateStr !=3D null)
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return false;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 } else if (!dateStr.equals(other.dateStr))
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return false;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 if (str =3D=3D null) {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 if (other.str !=3D null)
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return false;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 } else if (!str.equals(other.str))
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return false;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return true;
> =A0=A0=A0 =A0=A0=A0 }
>
> =A0=A0=A0 =A0=A0=A0 public String toString() {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 StringBuffer sb =3D new StringBuffer();
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 sb.append(this.getDateStr());
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 sb.append(",");
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 sb.append(this.getStr());
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return sb.toString();
> =A0=A0=A0 =A0=A0=A0 }
>
> =A0=A0=A0 =A0=A0=A0 public static class Q1003InPairComparator extends Wri=
tableComparator {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 public Q1003InPairComparator() {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 super(Q1003InPair.class);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 }
>
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 public int compare(byte[] b1, int s1, int l=
1, byte[] b2, int s2,
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 int l2) {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // 1. using streams
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // ByteArrayInputStream bais1 =3D=
 null;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // ByteArrayInputStream bais2 =3D=
 null;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // DataInputStream dis1 =3D null;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // DataInputStream dis2 =3D null;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // try {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // bais1 =3D new ByteArrayInputSt=
ream(b1);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // dis1 =3D new DataInputStream(b=
ais1);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // dis1.skip(s1);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // String str1 =3D dis1.readUTF()=
;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // String str3 =3D dis1.readUTF()=
;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 //
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // bais2 =3D new ByteArrayInputSt=
ream(b2);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // dis2 =3D new DataInputStream(b=
ais2);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // dis2.skip(s2);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // String str2 =3D dis2.readUTF()=
;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // String str4 =3D dis2.readUTF()=
;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 //
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // if (str1.equals(str2)) {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // return str3.compareTo(str4);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // } else {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // return str1.compareTo(str2);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // }
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // } catch(Exception e) {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // } finally {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // try {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // dis1.close();
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // } catch(Exception e) {}
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // try {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // dis2.close();
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // } catch(Exception e) {}
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // try {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // bais1.close();
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // } catch(Exception e) {}
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // try {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // bais2.close();
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // } catch(Exception e) {}
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // }
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // return -1;
>
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // 2. using byte[]
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 int strLength1 =3D ((int) b1[s1])=
 * 256 + ((int) b1[s1 + 1]);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 String str1 =3D new String(b1, s1=
 + 2, strLength1);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 s1 +=3D strLength1 + 2;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 int strLength3 =3D ((int) b1[s1])=
 * 256 + ((int) b1[s1 + 1]);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 String str3 =3D new String(b1, s1=
 + 2, strLength3);
>
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 int strLength2 =3D ((int) b2[s2])=
 * 256 + ((int) b2[s2 + 1]);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 String str2 =3D new String(b2, s2=
 + 2, strLength2);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 s2 +=3D strLength2 + 2;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 int strLength4 =3D ((int) b2[s2])=
 * 256 + ((int) b2[s2 + 1]);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 String str4 =3D new String(b2, s2=
 + 2, strLength4);
>
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 if (str1.equals(str2)) {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return str3.compareTo(s=
tr4);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 } else {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return str1.compareTo(s=
tr2);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 }
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 }
> =A0=A0=A0 =A0=A0=A0 }
>
> =A0=A0=A0 =A0=A0=A0 static {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 WritableComparator.define(Q1003InPair.class=
,
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 new Q1003InPairComparat=
or());
> =A0=A0=A0 =A0=A0=A0 }
>
> =A0=A0=A0 =A0=A0=A0 @Override
> =A0=A0=A0 =A0=A0=A0 public int compareTo(Q1003InPair o) {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 if (!this.getDateStr().equals(o.getDateStr(=
))) {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return this.getDateStr().equals(o=
.getDateStr()) ? 0 : (this
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 .getDateStr()=
.compareTo(o.getDateStr()) > 0 ? 1 : -1);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 } else if (!this.getStr().equals(o.getStr()=
)) {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return this.getStr().equals(o.get=
Str()) ? 0 : (this.getStr()
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 .compareTo(o.=
getStr()) > 0 ? 1 : -1);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 } else {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return 0;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 }
> =A0=A0=A0 =A0=A0=A0 }
> =A0=A0=A0 }
>
> Thanks again.
> Dennis
>
>
> --- On Wed, 8/11/10, Josh Patterson <josh@cloudera.com> wrote:
>
> From: Josh Patterson <josh@cloudera.com>
> Subject: Re: RawComparator
> To: mapreduce-user@hadoop.apache.org
> Cc: general@hadoop.apache.org
> Date: Wednesday, August 11, 2010, 3:21 AM
>
> Dennis,
>
> On Tue, Aug 10, 2010 at 4:01 AM, Dennis <arsenepark@yahoo.com.cn> wrote:
>
>> Hi, guys,
>>
>> I am using hadoop 0.20.2, and I am trying to run the "SecondarySort"
>> exmaple. The following is the "FirstGroupingComparator" class, and I jus=
t
>> cannot figure out how "WritableComparator.compareBytes(b1, s1,
>> Integer.SIZE / 8, b2, s2, Integer.SIZE / 8)" works. There are really few
>> javadocs of this class or=A0 this method.
>> 1. Why it is "Integer.SIZE / 8"?
>>
>
> That says "take the size of the integer in bits on this system and divide=
 it
> by 8" --- which in java on 32 and 64 bit systems should give you 32 / 8 =
=3D=3D 4
> as afaik the integer bit width doesnt change based on the architecture wi=
th
> java. So its saying here "compare the first 4 bytes of each byte array" (=
the
> width, in bytes, of the first integer in the composite key) ,whereas
> Integer.SIZE gives the number of bits in the datatype.
>
> WritableComparators are useful in the shuffle phase of hadoop; we are
> constantly comparing and sorting WritableComparables, and the secondary
> sorting mechanics allow us to have a group of data for a key arrive at th=
e
> reducer in a certain order (example: time series data, where we want a ra=
nge
> of timestamps in one group, but we also want them in order when they are
> processed inside the reducer)
>
>
>> 2. If I want to compare two "String" here, how should I write to code?
>>
>>=A0 =A0=A0=A0public static class FirstGroupingComparator implements
>>=A0 =A0 =A0 =A0 =A0 =A0=A0=A0RawComparator<IntPair> {
>>=A0 =A0 =A0 =A0=A0=A0@Override
>>=A0 =A0 =A0 =A0=A0=A0public int compare(byte[] b1, int s1, int l1, byte[]=
 b2, int s2,
>> int l2) {
>>=A0 =A0 =A0 =A0 =A0 =A0=A0=A0int ret =3D WritableComparator.compareBytes(=
b1, s1, Integer.SIZE
>> / 8,
>>=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=A0=A0b2, s2, Integer.SIZE / 8);
>>=A0 =A0 =A0 =A0 =A0 =A0=A0=A0return ret;
>>=A0 =A0 =A0 =A0=A0=A0}
>>
>>=A0 =A0 =A0 =A0=A0=A0@Override
>>=A0 =A0 =A0 =A0=A0=A0public int compare(IntPair o1, IntPair o2) {
>>=A0 =A0 =A0 =A0 =A0 =A0=A0=A0int l =3D o1.getFirst();
>>=A0 =A0 =A0 =A0 =A0 =A0=A0=A0int r =3D o2.getFirst();
>>=A0 =A0 =A0 =A0 =A0 =A0=A0=A0return l =3D=3D r ? 0 : (l < r ? -1 : 1);
>>=A0 =A0 =A0 =A0=A0=A0}
>>=A0 =A0=A0=A0}
>>
>
> In the case of the comparison of strings, lets say for example you have a
> "composite key" that has two String or Text object members (k1, k2); We
> "group by" the first part of the key k1 and we sort by this key as well (=
we
> block ranges together). This is very similar to the example above. Since
> with a RawComparator we are looking to only deserialize a portion of the
> data to do the comparison, you'll need a strategy for the compare() funct=
ion
> that takes into account that the strings are variable length (which means=
 we
> are unable to simply read 4 bytes as in the case of the integer). The
> challenge here is to only deserialize the portion of the composite key th=
at
> contains the string/text that you want to compare against, which is going=
 to
> be a variable number of bytes each time. A good place to start looking at
> for ideas would be the Text class in Hadoop and also WritableUtils.
>
> Josh Patterson
> Cloudera
>
>
>
>> Thanks.
>> Dennis
>>
>>
>
>
>
>

From general-return-1928-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 16:52:09 2010
Return-Path: <general-return-1928-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 7854 invoked from network); 11 Aug 2010 16:51:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 16:51:55 -0000
Received: (qmail 46388 invoked by uid 500); 11 Aug 2010 15:51:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46313 invoked by uid 500); 11 Aug 2010 15:51:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46305 invoked by uid 99); 11 Aug 2010 15:51:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 15:51:54 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 15:51:49 +0000
Received: by bwz9 with SMTP id 9so316189bwz.35
        for <general@hadoop.apache.org>; Wed, 11 Aug 2010 08:51:28 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.114.5 with SMTP id c5mr4486447bkq.171.1281541887970; Wed, 
	11 Aug 2010 08:51:27 -0700 (PDT)
Received: by 10.204.179.74 with HTTP; Wed, 11 Aug 2010 08:51:27 -0700 (PDT)
In-Reply-To: <4C62369D.6000703@yahoo-inc.com>
References: <AANLkTi=GyFLivH8uOQNppUJnVeixpY5HA98=0cJEpaNa@mail.gmail.com>
	<4C5C8302.3010008@apache.org>
	<97E2C026-0C44-4FDC-A569-63E218E03D59@yahoo-inc.com>
	<4C602C27.6060802@apache.org>
	<A7CB92D6-81C3-4550-BF38-FB02FD3516C6@yahoo-inc.com>
	<4C62369D.6000703@yahoo-inc.com>
Date: Wed, 11 Aug 2010 08:51:27 -0700
Message-ID: <AANLkTin-1tUGXvMK=dNdvX=bNUseCn=yQvj-rjPK9qeB@mail.gmail.com>
Subject: Re: [DISCUSSION] Combine committer lists in Common/HDFS/MapReduce
From: Tom White <tom@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Aug 10, 2010 at 10:35 PM, Vinod KV <vinodkv@yahoo-inc.com> wrote:
>
> I am one of those who has access to
> mapreduce project but not to common project

This looks like an oversight to me, which we should remedy.

Also, I did a little analysis, and there are only six committers out
of a total of 30 that do not have commit privileges for all three
subprojects. So, practically speaking, it would not be a big change to
give all committers privileges across the projects.

Tom

From general-return-1929-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 16:56:57 2010
Return-Path: <general-return-1929-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 13066 invoked from network); 11 Aug 2010 16:56:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 16:56:35 -0000
Received: (qmail 53311 invoked by uid 500); 11 Aug 2010 15:54:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53242 invoked by uid 500); 11 Aug 2010 15:54:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53234 invoked by uid 99); 11 Aug 2010 15:54:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 15:54:33 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of swhitecross@gmail.com designates 74.125.83.48 as permitted sender)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 15:54:26 +0000
Received: by gwj19 with SMTP id 19so123319gwj.35
        for <general@hadoop.apache.org>; Wed, 11 Aug 2010 08:54:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=S9LjCD+T2etgW/zEAHTVoOxoUJj1FEbzvDgMe7btQs0=;
        b=q/YbtBlmYqYVZZdjei5BSGTxUg+wKLr4Wfs0M2hm0mQ1YMIVtxdeAkAdiAXNePyFT5
         +Sh8uREBd+h059qXjsK7nBFbhECco45FgSg5VgAt5N7l05RLfAzSvjZJo6Nzr1J3HtmP
         9JKUm0OWNw+YskoRrin36pEbeDQ1x5/YhJh34=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=iBRsz4LhunH9PXoIrBbLT9vGhkoIYCLiboJOl+LD4xTq88nhEpvYvogBSMOMOW/G/y
         U1Vj1sEjv6zFnBRJgHI/cF7CYD6rLqWdrWcSXm5+yZz06+gHpeANl2pbg3an5yIBGWHB
         jmr1wZtZ0rCgRhdkMzs7gfNH562gkzybZMPK8=
MIME-Version: 1.0
Received: by 10.101.209.13 with SMTP id l13mr21725024anq.238.1281542045252; 
	Wed, 11 Aug 2010 08:54:05 -0700 (PDT)
Received: by 10.220.128.204 with HTTP; Wed, 11 Aug 2010 08:54:05 -0700 (PDT)
Date: Wed, 11 Aug 2010 11:54:05 -0400
Message-ID: <AANLkTin9vdavJYgHDPYzKg_DriLgp6BrgPkiqXTvayqz@mail.gmail.com>
Subject: Listing Hadoop Job History Statistics
From: Scott Whitecross <swhitecross@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e68bd3fe5f3cd5048d8e42c4
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e68bd3fe5f3cd5048d8e42c4
Content-Type: text/plain; charset=ISO-8859-1

Hi -

What's the best way to list and query information on Hadoop job histories?
 For example, I'd like to see the job names from the past week against a
Hadoop cluster I'm using.   I don't see an API call or a way through the
command line to pull the information.  Is the best way writing a quick
script to process the job history files?

Thanks.
Scott

--0016e68bd3fe5f3cd5048d8e42c4--

From general-return-1930-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 17:13:22 2010
Return-Path: <general-return-1930-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30881 invoked from network); 11 Aug 2010 17:13:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 17:13:17 -0000
Received: (qmail 9465 invoked by uid 500); 11 Aug 2010 17:13:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 9383 invoked by uid 500); 11 Aug 2010 17:13:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 9375 invoked by uid 99); 11 Aug 2010 17:13:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 17:13:15 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [202.43.216.213] (HELO n3b.bullet.cnb.yahoo.com) (202.43.216.213)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 11 Aug 2010 17:13:05 +0000
Received: from [202.43.217.36] by n3.bullet.cnb.yahoo.com with NNFMP; 11 Aug 2010 17:12:42 -0000
Received: from [203.209.230.77] by t1.bullet.cnmail.cnb.yahoo.com with NNFMP; 11 Aug 2010 17:12:42 -0000
Received: from [127.0.0.1] by omp106.mail.cnb.yahoo.com with NNFMP; 11 Aug 2010 17:12:42 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 999475.6109.bm@omp106.mail.cnb.yahoo.com
Received: (qmail 85034 invoked by uid 60001); 11 Aug 2010 17:12:41 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com.cn; s=s1024; t=1281546761; bh=mnjLnEhRl7t4b2CXWnAvoGv3/clio9xyTEGPiz9dl5s=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=QLuyg71h9eehUYIAmog9GkdcOKrgYyrvZ765oe8WfT6RA8Xo8V/Ng+LVLgf/2l1EuR5zPfJSTZIXg7smoI7b1dQwyixtyr6NNvtYL5F3S8VaHoS5eaX4iV5vmNXih+DIsyivreqgBOk9g9q61KryIIS4so59brr/h/Eli+sQKKY=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com.cn;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=4BYl+yL4UvA80nzjimGk9GEa+0IP+fX4ZkMLn+z5A1SPtm76Yv44O/5CBLM/dSLQZbUYMZesG86YRLvV5YIS7xNBPk5sDuvdedf5RbQq/XtGkArJm0+gD+l0BoZ64q5U4kuR34Lxn6v9zhD2Wl7D5nHW/fnV77klgEnyHgDEr5Y=;
Message-ID: <856580.84210.qm@web15202.mail.cnb.yahoo.com>
X-YMail-OSG: bQqPP7QVM1mKHimQykNhrTLZ9VUEBY0sZbz4JkLWnv3lKoa
 A_yi7Qs8O
Received: from [202.118.67.200] by web15202.mail.cnb.yahoo.com via HTTP; Thu, 12 Aug 2010 01:12:41 CST
X-Mailer: YahooMailClassic/11.3.2 YahooMailWebService/0.8.105.279950
Date: Thu, 12 Aug 2010 01:12:41 +0800 (CST)
From: Dennis <arsenepark@yahoo.com.cn>
Subject: Re: RawComparator
To: general@hadoop.apache.org
In-Reply-To: <AANLkTik4pOa2dFwyxFgV4Gzy4CoVR1g4QbOw+MmPsH9_@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-249652734-1281546761=:84210"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-249652734-1281546761=:84210
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Hi, Josh,

Yes, you right. The byte[] manipulations is much faster, as it's not creati=
ng new objects, just like you said.
Thanks again, I've learned a lot from this example, and you've been very he=
lpful.

Dennis

--- On Wed, 8/11/10, Josh Patterson <josh@cloudera.com> wrote:

From: Josh Patterson <josh@cloudera.com>
Subject: Re: RawComparator
To: general@hadoop.apache.org
Date: Wednesday, August 11, 2010, 11:07 PM

Dennis,
Well, in general, consider this: this method is getting called per K/V
pair potentially billions or even trillions of times. We want to do
the minimum amount of work that we can here and get the most results
to get the best performance, right?

If we think about the lower levels of memory and using the stack, we
want to allocate as little memory as possible since at the rate we'd
be creating new objects on the heap we'd really be tearing through
some memory. Without doing testing myself (which you should do, this
is easy to test, data always dominates these discussions), I'm going
to bet that the byte[] manipulations are going to be more performant.
If you can pull out the number of bytes (from the first byte or two,
like Text does it) from the front of the byte array, then you can
deserialize only the part of the composite key you are interested in.
If you can do this without creating new objects on the heap, I'd say
you are going to get better performance (just like as in the previous
example where we deserialized the first 4 bytes of the integer, that
made up the int pair).

Josh Patterson
Cloudera

On Tue, Aug 10, 2010 at 9:07 PM, Dennis <arsenepark@yahoo.com.cn> wrote:
> Thanks, Josh, for your professional answers.
>
> The following is my "composite key" class. And in the Q1003InPairComparat=
or.compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) method, I =
figured out two ways of picking up original String values and comparing. Th=
e first "using streams" one is using java.io.* package to deserialize the o=
riginal String values, and the comparision is easy to do. The second "using=
 byte[]" one, I just try to deserialize the byte[] by myself, as it's not d=
ifficult to do this, and I donnot have to "new" any stream classes in java.=
io.*.
> So,
> 1. What do you say about the two methods? Or any better ones?
> 2. If I use the first "using streams" one, what do I deal with the IOExce=
ption? If an Exception is thrown, what value should I return?(In the follow=
ing code, I simply returned -1. I know it's not smart to do so.)
>
>
> =A0=A0=A0 public static class Q1003InPair implements WritableComparable<Q=
1003InPair> {
> =A0=A0=A0 =A0=A0=A0 private String dateStr;
> =A0=A0=A0 =A0=A0=A0 private String str;
>
> =A0=A0=A0 =A0=A0=A0 public void set(String dateStr, String str) {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 this.dateStr =3D dateStr;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 this.str =3D str;
> =A0=A0=A0 =A0=A0=A0 }
>
> =A0=A0=A0 =A0=A0=A0 public String getDateStr() {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return this.dateStr;
> =A0=A0=A0 =A0=A0=A0 }
>
> =A0=A0=A0 =A0=A0=A0 public String getStr() {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return this.str;
> =A0=A0=A0 =A0=A0=A0 }
>
> =A0=A0=A0 =A0=A0=A0 @Override
> =A0=A0=A0 =A0=A0=A0 public void readFields(DataInput in) throws IOExcepti=
on {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 dateStr =3D in.readUTF();
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 str =3D in.readUTF();
> =A0=A0=A0 =A0=A0=A0 }
>
> =A0=A0=A0 =A0=A0=A0 @Override
> =A0=A0=A0 =A0=A0=A0 public void write(DataOutput out) throws IOException =
{
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 out.writeUTF(dateStr);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 out.writeUTF(str);
> =A0=A0=A0 =A0=A0=A0 }
>
> =A0=A0=A0 =A0=A0=A0 @Override
> =A0=A0=A0 =A0=A0=A0 public int hashCode() {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 final int prime =3D 31;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 int result =3D 1;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 result =3D prime * result
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 + ((dateStr =3D=3D null=
) ? 0 : dateStr.hashCode());
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 result =3D prime * result + ((str =3D=3D nu=
ll) ? 0 : str.hashCode());
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return result;
> =A0=A0=A0 =A0=A0=A0 }
>
> =A0=A0=A0 =A0=A0=A0 @Override
> =A0=A0=A0 =A0=A0=A0 public boolean equals(Object obj) {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 if (this =3D=3D obj)
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return true;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 if (obj =3D=3D null)
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return false;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 if (getClass() !=3D obj.getClass())
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return false;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 final Q1003InPair other =3D (Q1003InPair) o=
bj;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 if (dateStr =3D=3D null) {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 if (other.dateStr !=3D null)
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return false;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 } else if (!dateStr.equals(other.dateStr))
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return false;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 if (str =3D=3D null) {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 if (other.str !=3D null)
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return false;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 } else if (!str.equals(other.str))
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return false;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return true;
> =A0=A0=A0 =A0=A0=A0 }
>
> =A0=A0=A0 =A0=A0=A0 public String toString() {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 StringBuffer sb =3D new StringBuffer();
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 sb.append(this.getDateStr());
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 sb.append(",");
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 sb.append(this.getStr());
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return sb.toString();
> =A0=A0=A0 =A0=A0=A0 }
>
> =A0=A0=A0 =A0=A0=A0 public static class Q1003InPairComparator extends Wri=
tableComparator {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 public Q1003InPairComparator() {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 super(Q1003InPair.class);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 }
>
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 public int compare(byte[] b1, int s1, int l=
1, byte[] b2, int s2,
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 int l2) {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // 1. using streams
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // ByteArrayInputStream bais1 =3D=
 null;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // ByteArrayInputStream bais2 =3D=
 null;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // DataInputStream dis1 =3D null;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // DataInputStream dis2 =3D null;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // try {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // bais1 =3D new ByteArrayInputSt=
ream(b1);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // dis1 =3D new DataInputStream(b=
ais1);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // dis1.skip(s1);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // String str1 =3D dis1.readUTF()=
;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // String str3 =3D dis1.readUTF()=
;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 //
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // bais2 =3D new ByteArrayInputSt=
ream(b2);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // dis2 =3D new DataInputStream(b=
ais2);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // dis2.skip(s2);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // String str2 =3D dis2.readUTF()=
;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // String str4 =3D dis2.readUTF()=
;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 //
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // if (str1.equals(str2)) {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // return str3.compareTo(str4);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // } else {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // return str1.compareTo(str2);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // }
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // } catch(Exception e) {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // } finally {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // try {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // dis1.close();
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // } catch(Exception e) {}
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // try {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // dis2.close();
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // } catch(Exception e) {}
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // try {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // bais1.close();
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // } catch(Exception e) {}
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // try {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // bais2.close();
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // } catch(Exception e) {}
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // }
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // return -1;
>
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 // 2. using byte[]
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 int strLength1 =3D ((int) b1[s1])=
 * 256 + ((int) b1[s1 + 1]);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 String str1 =3D new String(b1, s1=
 + 2, strLength1);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 s1 +=3D strLength1 + 2;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 int strLength3 =3D ((int) b1[s1])=
 * 256 + ((int) b1[s1 + 1]);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 String str3 =3D new String(b1, s1=
 + 2, strLength3);
>
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 int strLength2 =3D ((int) b2[s2])=
 * 256 + ((int) b2[s2 + 1]);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 String str2 =3D new String(b2, s2=
 + 2, strLength2);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 s2 +=3D strLength2 + 2;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 int strLength4 =3D ((int) b2[s2])=
 * 256 + ((int) b2[s2 + 1]);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 String str4 =3D new String(b2, s2=
 + 2, strLength4);
>
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 if (str1.equals(str2)) {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return str3.compareTo(s=
tr4);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 } else {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return str1.compareTo(s=
tr2);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 }
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 }
> =A0=A0=A0 =A0=A0=A0 }
>
> =A0=A0=A0 =A0=A0=A0 static {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 WritableComparator.define(Q1003InPair.class=
,
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 new Q1003InPairComparat=
or());
> =A0=A0=A0 =A0=A0=A0 }
>
> =A0=A0=A0 =A0=A0=A0 @Override
> =A0=A0=A0 =A0=A0=A0 public int compareTo(Q1003InPair o) {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 if (!this.getDateStr().equals(o.getDateStr(=
))) {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return this.getDateStr().equals(o=
.getDateStr()) ? 0 : (this
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 .getDateStr()=
.compareTo(o.getDateStr()) > 0 ? 1 : -1);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 } else if (!this.getStr().equals(o.getStr()=
)) {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return this.getStr().equals(o.get=
Str()) ? 0 : (this.getStr()
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 .compareTo(o.=
getStr()) > 0 ? 1 : -1);
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 } else {
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 return 0;
> =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 }
> =A0=A0=A0 =A0=A0=A0 }
> =A0=A0=A0 }
>
> Thanks again.
> Dennis
>
>
> --- On Wed, 8/11/10, Josh Patterson <josh@cloudera.com> wrote:
>
> From: Josh Patterson <josh@cloudera.com>
> Subject: Re: RawComparator
> To: mapreduce-user@hadoop.apache.org
> Cc: general@hadoop.apache.org
> Date: Wednesday, August 11, 2010, 3:21 AM
>
> Dennis,
>
> On Tue, Aug 10, 2010 at 4:01 AM, Dennis <arsenepark@yahoo.com.cn> wrote:
>
>> Hi, guys,
>>
>> I am using hadoop 0.20.2, and I am trying to run the "SecondarySort"
>> exmaple. The following is the "FirstGroupingComparator" class, and I jus=
t
>> cannot figure out how "WritableComparator.compareBytes(b1, s1,
>> Integer.SIZE / 8, b2, s2, Integer.SIZE / 8)" works. There are really few
>> javadocs of this class or=A0 this method.
>> 1. Why it is "Integer.SIZE / 8"?
>>
>
> That says "take the size of the integer in bits on this system and divide=
 it
> by 8" --- which in java on 32 and 64 bit systems should give you 32 / 8 =
=3D=3D 4
> as afaik the integer bit width doesnt change based on the architecture wi=
th
> java. So its saying here "compare the first 4 bytes of each byte array" (=
the
> width, in bytes, of the first integer in the composite key) ,whereas
> Integer.SIZE gives the number of bits in the datatype.
>
> WritableComparators are useful in the shuffle phase of hadoop; we are
> constantly comparing and sorting WritableComparables, and the secondary
> sorting mechanics allow us to have a group of data for a key arrive at th=
e
> reducer in a certain order (example: time series data, where we want a ra=
nge
> of timestamps in one group, but we also want them in order when they are
> processed inside the reducer)
>
>
>> 2. If I want to compare two "String" here, how should I write to code?
>>
>>=A0 =A0=A0=A0public static class FirstGroupingComparator implements
>>=A0 =A0 =A0 =A0 =A0 =A0=A0=A0RawComparator<IntPair> {
>>=A0 =A0 =A0 =A0=A0=A0@Override
>>=A0 =A0 =A0 =A0=A0=A0public int compare(byte[] b1, int s1, int l1, byte[]=
 b2, int s2,
>> int l2) {
>>=A0 =A0 =A0 =A0 =A0 =A0=A0=A0int ret =3D WritableComparator.compareBytes(=
b1, s1, Integer.SIZE
>> / 8,
>>=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=A0=A0b2, s2, Integer.SIZE / 8);
>>=A0 =A0 =A0 =A0 =A0 =A0=A0=A0return ret;
>>=A0 =A0 =A0 =A0=A0=A0}
>>
>>=A0 =A0 =A0 =A0=A0=A0@Override
>>=A0 =A0 =A0 =A0=A0=A0public int compare(IntPair o1, IntPair o2) {
>>=A0 =A0 =A0 =A0 =A0 =A0=A0=A0int l =3D o1.getFirst();
>>=A0 =A0 =A0 =A0 =A0 =A0=A0=A0int r =3D o2.getFirst();
>>=A0 =A0 =A0 =A0 =A0 =A0=A0=A0return l =3D=3D r ? 0 : (l < r ? -1 : 1);
>>=A0 =A0 =A0 =A0=A0=A0}
>>=A0 =A0=A0=A0}
>>
>
> In the case of the comparison of strings, lets say for example you have a
> "composite key" that has two String or Text object members (k1, k2); We
> "group by" the first part of the key k1 and we sort by this key as well (=
we
> block ranges together). This is very similar to the example above. Since
> with a RawComparator we are looking to only deserialize a portion of the
> data to do the comparison, you'll need a strategy for the compare() funct=
ion
> that takes into account that the strings are variable length (which means=
 we
> are unable to simply read 4 bytes as in the case of the integer). The
> challenge here is to only deserialize the portion of the composite key th=
at
> contains the string/text that you want to compare against, which is going=
 to
> be a variable number of bytes each time. A good place to start looking at
> for ideas would be the Text class in Hadoop and also WritableUtils.
>
> Josh Patterson
> Cloudera
>
>
>
>> Thanks.
>> Dennis
>>
>>
>
>
>
>
=0A=0A=0A      
--0-249652734-1281546761=:84210--


From general-return-1931-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 20:57:12 2010
Return-Path: <general-return-1931-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 40223 invoked from network); 11 Aug 2010 20:57:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 20:57:11 -0000
Received: (qmail 47163 invoked by uid 500); 11 Aug 2010 20:57:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47092 invoked by uid 500); 11 Aug 2010 20:57:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47084 invoked by uid 99); 11 Aug 2010 20:57:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 20:57:09 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 20:57:03 +0000
Received: from SP1-EX07CAS02.ds.corp.yahoo.com (sp1-ex07cas02.ds.corp.yahoo.com [216.252.116.138])
	by mrout2.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7BKuGiT071358
	for <general@hadoop.apache.org>; Wed, 11 Aug 2010 13:56:16 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:accept-language:content-language:x-ms-has-attach:
	x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=2Q3cFU+FhWZ8HQkuXq0R5rc3oxnqAQ+eaOYzRLN2iCGlCrJzOplIjuvwJhQw4y8P
Received: from SP1-EX07VS02.ds.corp.yahoo.com ([216.252.116.136]) by
 SP1-EX07CAS02.ds.corp.yahoo.com ([216.252.116.167]) with mapi; Wed, 11 Aug
 2010 13:56:17 -0700
From: "Sanjay X. Jain" <sanjayja@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 11 Aug 2010 13:56:15 -0700
Subject: A log4jconfig.xml file 
Thread-Topic: A log4jconfig.xml file 
Thread-Index: Acs5l6MiKam7gyep+Ue6uWfWtUOLZg==
Message-ID: <C8885C7F.2D99%sanjayja@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C8885C7F2D99sanjayjayahooinccom_"
MIME-Version: 1.0

--_000_C8885C7F2D99sanjayjayahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

I have a log4jconfig.xml file that is loaded in my hadoop job code as: DOMC=
onfigurator.configure("log4jconfig.xml");
How should I pass this file to the job (and its mapper and reducer)? I have=
 tried to include it in CLASSPATH env variable, and by including the file i=
n job's jar file, both don't seem to work.
Purpose: want to control what is logged where in my hadoop job's classes/pa=
ckages.

Any clues will be really helpful.

Thx!
Sanjay

--_000_C8885C7F2D99sanjayjayahooinccom_--

From general-return-1932-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 12 00:58:04 2010
Return-Path: <general-return-1932-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 36195 invoked from network); 12 Aug 2010 00:58:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Aug 2010 00:58:03 -0000
Received: (qmail 99677 invoked by uid 500); 12 Aug 2010 00:58:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99599 invoked by uid 500); 12 Aug 2010 00:58:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 99591 invoked by uid 99); 12 Aug 2010 00:58:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 00:58:01 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 00:57:53 +0000
Received: from sp1-ex07cas01.ds.corp.yahoo.com (sp1-ex07cas01.ds.corp.yahoo.com [216.252.116.137])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7C0vKSk028999
	for <general@hadoop.apache.org>; Wed, 11 Aug 2010 17:57:20 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:in-reply-to:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=YEIb73VakZqbR5zYvKpLKDIZndKjImJRoADv3RkAw74i0VfFGmnMtImTX46qbi5c
Received: from SP1-EX07VS02.ds.corp.yahoo.com ([216.252.116.136]) by
 sp1-ex07cas01.ds.corp.yahoo.com ([216.252.116.137]) with mapi; Wed, 11 Aug
 2010 17:57:20 -0700
From: "Sanjay X. Jain" <sanjayja@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 11 Aug 2010 17:57:19 -0700
Subject: Re: A log4jconfig.xml file 
Thread-Topic: A log4jconfig.xml file 
Thread-Index: Acs5l6MiKam7gyep+Ue6uWfWtUOLZgAIa05m
Message-ID: <C88894FF.2DAC%sanjayja@yahoo-inc.com>
In-Reply-To: <C8885C7F.2D99%sanjayja@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C88894FF2DACsanjayjayahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C88894FF2DACsanjayjayahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Probably a better question is: In a hadoop job, how can I load my own log4j=
.properties file instead of the default one?

Thx!


On 8/11/10 1:56 PM, "Sanjay Jain" <sanjayja@yahoo-inc.com> wrote:

I have a log4jconfig.xml file that is loaded in my hadoop job code as: DOMC=
onfigurator.configure("log4jconfig.xml");
How should I pass this file to the job (and its mapper and reducer)? I have=
 tried to include it in CLASSPATH env variable, and by including the file i=
n job's jar file, both don't seem to work.
Purpose: want to control what is logged where in my hadoop job's classes/pa=
ckages.

Any clues will be really helpful.

Thx!
Sanjay


--_000_C88894FF2DACsanjayjayahooinccom_--

From general-return-1933-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 12 04:23:57 2010
Return-Path: <general-return-1933-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94418 invoked from network); 12 Aug 2010 04:23:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Aug 2010 04:23:57 -0000
Received: (qmail 15301 invoked by uid 500); 12 Aug 2010 04:23:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14821 invoked by uid 500); 12 Aug 2010 04:23:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14811 invoked by uid 99); 12 Aug 2010 04:23:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 04:23:51 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [96.236.214.3] (HELO dugos.com) (96.236.214.3)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 04:23:44 +0000
Received: (qmail 18836 invoked from network); 12 Aug 2010 04:23:21 -0000
Received: from beast.dugos.com (HELO macpro.dugos.com) (96.236.214.4)
  by dugos.com with SMTP; 12 Aug 2010 04:23:21 -0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1081)
Subject: Re: Listing Hadoop Job History Statistics
From: Doug Balog <doug.hdpgen@dugos.com>
In-Reply-To: <AANLkTin9vdavJYgHDPYzKg_DriLgp6BrgPkiqXTvayqz@mail.gmail.com>
Date: Thu, 12 Aug 2010 00:23:21 -0400
Content-Transfer-Encoding: quoted-printable
Message-Id: <D803216E-61A9-4F8F-80EC-95BF6B8C052B@dugos.com>
References: <AANLkTin9vdavJYgHDPYzKg_DriLgp6BrgPkiqXTvayqz@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1081)

I don't know if this is the best way, but this is how I do it.

Configuration  conf =3D new Configuration();
JobClient jobClient =3D new JobClient(new =
InetSocketAddress("jobTracker",9001),conf);
jobClient.setConf(conf); // Bug in constructor, doesn't set conf.

 for(JobStatus js: jobClient.getAllJobs()){
    // We only care about completed jobs.
                if(!js.isJobComplete()){
                    continue;
                }=20
                // Do stuff on jobStatus.
               :
	       :
 }

You can also scrape info from http://jobtracker:50030/jobhistory.jsp

Or read it from the job's outputDir/_log/ directory.

Cheers,

Doug


On Aug 11, 2010, at 11:54 AM, Scott Whitecross wrote:

> Hi -
>=20
> What's the best way to list and query information on Hadoop job =
histories?
> For example, I'd like to see the job names from the past week against =
a
> Hadoop cluster I'm using.   I don't see an API call or a way through =
the
> command line to pull the information.  Is the best way writing a quick
> script to process the job history files?
>=20
> Thanks.
> Scott


From general-return-1934-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 12 04:53:20 2010
Return-Path: <general-return-1934-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1092 invoked from network); 12 Aug 2010 04:53:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Aug 2010 04:53:19 -0000
Received: (qmail 26644 invoked by uid 500); 12 Aug 2010 04:53:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25849 invoked by uid 500); 12 Aug 2010 04:53:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25827 invoked by uid 99); 12 Aug 2010 04:53:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 04:53:13 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 04:53:07 +0000
Received: from [192.168.1.71] ([10.72.168.227])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7C4qNqH085193;
	Wed, 11 Aug 2010 21:52:24 -0700 (PDT)
Message-Id: <0F9D3E90-4F42-4DFC-B48A-F61A6B476F42@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: mapreduce-user@hadoop.apache.org
In-Reply-To: <AANLkTin9vdavJYgHDPYzKg_DriLgp6BrgPkiqXTvayqz@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Subject: Re: Listing Hadoop Job History Statistics
Mime-Version: 1.0 (Apple Message framework v936)
Date: Wed, 11 Aug 2010 21:52:24 -0700
References: <AANLkTin9vdavJYgHDPYzKg_DriLgp6BrgPkiqXTvayqz@mail.gmail.com>
X-Mailer: Apple Mail (2.936)

Moving to mapreduce-user@, bcc general@.

There isn't a direct way. One possible option is just use the per-job  
job-history file which is on HDFS (See http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html#Job+Submission+and+Monitoring 
  for info on job-history).

Hope that helps.

Arun

On Aug 11, 2010, at 8:54 AM, Scott Whitecross wrote:

> Hi -
>
> What's the best way to list and query information on Hadoop job  
> histories?
> For example, I'd like to see the job names from the past week  
> against a
> Hadoop cluster I'm using.   I don't see an API call or a way through  
> the
> command line to pull the information.  Is the best way writing a quick
> script to process the job history files?
>
> Thanks.
> Scott


From general-return-1935-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 12 11:43:08 2010
Return-Path: <general-return-1935-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56951 invoked from network); 12 Aug 2010 11:43:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Aug 2010 11:43:08 -0000
Received: (qmail 1323 invoked by uid 500); 12 Aug 2010 11:43:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 939 invoked by uid 500); 12 Aug 2010 11:43:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 931 invoked by uid 99); 12 Aug 2010 11:43:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 11:43:03 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 11:42:59 +0000
Received: from [10.66.72.163] (sightbusy-lx.eglbp.corp.yahoo.com [10.66.72.163])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7CBfscT062606;
	Thu, 12 Aug 2010 04:41:55 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:cc:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=VUNErwOK20ZIRwCLYfMf6/WYEHEgLxR/ZpB8ndjqTZMQy5q9+18LkwLyz/Fw6pr8
Message-ID: <4C63DE01.8080303@yahoo-inc.com>
Date: Thu, 12 Aug 2010 17:11:53 +0530
From: Vinod KV <vinodkv@yahoo-inc.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.11) Gecko/20100713 Lightning/1.0b1 Thunderbird/3.0.6
MIME-Version: 1.0
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "Sanjay X. Jain" <sanjayja@yahoo-inc.com>
Subject: Re: A log4jconfig.xml file
References: <C88894FF.2DAC%sanjayja@yahoo-inc.com>
In-Reply-To: <C88894FF.2DAC%sanjayja@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit


The earlier question is easier to answer given you are loading the file 
yourselves :)

You have multiple solutions:
  (1) Put it in the lib folder inside the job.jar. It should be 
automatically available from the task's classpath.
  (2) Set it in distributed cache as a classpath file. See 
DistributedCache.addCacheFile(URI uri, Configuration conf).

+vinod


On Thursday 12 August 2010 06:27 AM, Sanjay X. Jain wrote:
> Probably a better question is: In a hadoop job, how can I load my own log4j.properties file instead of the default one?
>
> Thx!
>
>
> On 8/11/10 1:56 PM, "Sanjay Jain"<sanjayja@yahoo-inc.com>  wrote:
>
> I have a log4jconfig.xml file that is loaded in my hadoop job code as: DOMConfigurator.configure("log4jconfig.xml");
> How should I pass this file to the job (and its mapper and reducer)? I have tried to include it in CLASSPATH env variable, and by including the file in job's jar file, both don't seem to work.
> Purpose: want to control what is logged where in my hadoop job's classes/packages.
>
> Any clues will be really helpful.
>
> Thx!
> Sanjay
>
>
>    


From general-return-1936-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 13 22:18:23 2010
Return-Path: <general-return-1936-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21606 invoked from network); 13 Aug 2010 22:18:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Aug 2010 22:18:22 -0000
Received: (qmail 10713 invoked by uid 500); 13 Aug 2010 22:18:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 10442 invoked by uid 500); 13 Aug 2010 22:18:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 7574 invoked by uid 99); 8 Aug 2010 15:49:26 -0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of oded.rotem.hd@gmail.com designates 74.125.82.176 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:from:to:references
         :in-reply-to:subject:date:message-id:mime-version:content-type
         :content-transfer-encoding:x-mailer:thread-index:content-language
         :x-cr-hashedpuzzle:x-cr-puzzleid;
        bh=wacNFlEhLyqSbJwAhiucVYmgj8KjGNmnvjErsv3Ifow=;
        b=X5SLfDXlGxG80ziU3sJiBj8ESoxskcqcceumezh4kibBFrfL4BI3WNZBtI2fiPTTR5
         sMtog2IbKgISkXlzdE7KFjQ6XEXz2qm95zrbCbLapzxRweJsDTq+yGPl7mFRRZSYlL8s
         824jZODvB3POFVb0x7uEk0cWJe2KG+QmrsI/4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=from:to:references:in-reply-to:subject:date:message-id:mime-version
         :content-type:content-transfer-encoding:x-mailer:thread-index
         :content-language:x-cr-hashedpuzzle:x-cr-puzzleid;
        b=B1Pz0i0PCgPelDk2/F9LU2lD1yuDtn5AWUe16AVpJYmbpaqKapB5bGigdtjo1rMjx4
         cXFJQOKfOHN+EjmbHF4o6rPgKsbQm5cv/Rf5N61IMb6pn441y7MS0z/zH6tzMGBxwkcX
         XPc+VXQ7qs0FBiq3ndKOCsyGHjXR6GX/NVw3c=
From: "Oded Rotem" <oded.rotem.hd@gmail.com>
To: <general@hadoop.apache.org>
References: <AANLkTinbKV8nOtDyzNoURrFjJLz=M8gr6bLtg1TeZLqZ@mail.gmail.com>
In-Reply-To: <AANLkTinbKV8nOtDyzNoURrFjJLz=M8gr6bLtg1TeZLqZ@mail.gmail.com>
Subject: Session analysis using Hadoop
Date: Sun, 8 Aug 2010 18:51:03 +0300
Message-ID: <4c5ed1e7.1207e30a.3749.ffff8be0@mx.google.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Mailer: Microsoft Office Outlook 12.0
Thread-Index: Acs2YCVYTkkJnNi2TjSzpobMLjsqRAAsNbwQ
Content-Language: en-us
x-cr-hashedpuzzle: Auj2 AvwA CMRB C2DL DRGY Ecbq FG6d FxD2 Hk+d H+dG INia IsGL JxJS J7T+ KfOT MPXS;1;ZwBlAG4AZQByAGEAbABAAGgAYQBkAG8AbwBwAC4AYQBwAGEAYwBoAGUALgBvAHIAZwA=;Sosha1_v1;7;{2D2ACB07-0761-4F97-8448-E8508979BD30};bwBkAGUAZAAuAHIAbwB0AGUAbQAuAGgAZABAAGcAbQBhAGkAbAAuAGMAbwBtAA==;Sun, 08 Aug 2010 15:51:00 GMT;UwBlAHMAcwBpAG8AbgAgAGEAbgBhAGwAeQBzAGkAcwAgAHUAcwBpAG4AZwAgAEgAYQBkAG8AbwBwAA==
x-cr-puzzleid: {2D2ACB07-0761-4F97-8448-E8508979BD30}

Hi,

Does anyone have any experience using Hadoop to do sessionizing / session
analysis? I know eBay has something called MQL (Mobius Query Language) for
this, but they have not (yet?...) made it open source.

Any useful pointers here would be appreciated.

Cheers,
Oded


From general-return-1937-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 13 22:18:32 2010
Return-Path: <general-return-1937-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21715 invoked from network); 13 Aug 2010 22:18:31 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Aug 2010 22:18:31 -0000
Received: (qmail 11693 invoked by uid 500); 13 Aug 2010 22:18:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11606 invoked by uid 500); 13 Aug 2010 22:18:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 9467 invoked by uid 99); 9 Aug 2010 18:49:37 -0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Date: Mon, 9 Aug 2010 11:47:53 -0700
From: Konstantin Boudnik <cos@apache.org>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: [DISCUSSION] Combine committer lists in Common/HDFS/MapReduce
Message-ID: <20100809184752.GA94111@goodenter-lm.local>
References: <AANLkTi=GyFLivH8uOQNppUJnVeixpY5HA98=0cJEpaNa@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature"; boundary="FCuugMFkClbJLl1L"
Content-Disposition: inline
In-Reply-To: <AANLkTi=GyFLivH8uOQNppUJnVeixpY5HA98=0cJEpaNa@mail.gmail.com>
X-Organization: Yahoo! Hadoop (Grid computing)
User-Agent: Mutt/1.5.20 (2009-06-14)
X-Virus-Checked: Checked by ClamAV on apache.org

--FCuugMFkClbJLl1L
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

+1 on combining the committer roles in one (as Doug has mentioned elsewhere,
all three are feel like a single project and are released as such, therefore
it doesn't make much sense to split committers). In effect, MR testing that
pulls in "external" HDFS dependencies works as a poor-man integration testi=
ng.

-1 on combining the projects back together. While having them separated mig=
ht
cause an extra maintenance burden, the same separation forces more clean
design decisions.=20

Cos

On Fri, Aug 06, 2010 at 02:02PM, Chris Douglas wrote:
> Hadoop developers tend to specialize in either HDFS or MapReduce, but
> given that:
>=20
> 0) Granting karma to Common is routine for a committer in either
> space; there are no Common-only committers
> 1) The majority of committers have been grandfathered into committer
> roles in all three projects
> 2) Many patches to Common require corresponding commits to both HDFS
> and MapReduce
> 3) Review-then-commit is usually sufficient notice for interested
> parties to comment
> 4) There have been few problems with committers pushing in patches
> without consulting someone more directly involved
> 5) Everyone on the PMC gets commit rights to all subprojects, anyway
>=20
> Perhaps it would make sense to give up on separate committer roles
> until the projects are separate TLPs.
>=20
> On the other hand:
>=20
> 0) Nobody has been independently added to both HDFS and MapReduce
> since the projects were separated
> 1) It could exacerbate the focus on MapReduce in HDFS, at the expense
> of other projects (like HBase).
> 2) HDFS and MapReduce are mostly independent communities and
> codebases; expertise in one does not imply fluency in the other
> 3) Granting veto power across projects can lead to deadlock despite
> consensus within that community
> 4) TLP status for either project may require untangling HDFS/MR roles
> that could be distinguished now
>=20
> Personally, I'm in favor of combining the roles. I trust all six of
> the committers made since the project split no less than those made
> earlier. Further, version control is sufficient for recovering from
> most, foreseeable issues. I have some concerns about "harmless"
> commits pushed through without an audit by the subproject's
> maintainers (a few in recent memory caused downtime in Y! clusters),
> but combining the roles seems like a worthwhile experiment.
>=20
> Thoughts? -C

--FCuugMFkClbJLl1L
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)

iF4EAREIAAYFAkxgTVgACgkQMqXifkwDoaEHNAD/bPBu1ElJJ537qf3UGKm3gOV6
sINlS+OC3O2fh/KizsQA/i3l6ltDfowz9iPbBW2/2VWu/dNaZu53px4oS9jEmqPH
=z2/s
-----END PGP SIGNATURE-----

--FCuugMFkClbJLl1L--

From general-return-1938-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 14 01:53:48 2010
Return-Path: <general-return-1938-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97190 invoked from network); 14 Aug 2010 01:53:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Aug 2010 01:53:48 -0000
Received: (qmail 38638 invoked by uid 500); 14 Aug 2010 01:53:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38561 invoked by uid 500); 14 Aug 2010 01:53:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 38553 invoked by uid 99); 14 Aug 2010 01:53:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Aug 2010 01:53:46 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Aug 2010 01:53:41 +0000
Received: by wwb22 with SMTP id 22so4454885wwb.29
        for <general@hadoop.apache.org>; Fri, 13 Aug 2010 18:53:20 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=MtKOY+V7uZOLBNJZEIKl3zmkQnJAJLfvgxOi+52jYnY=;
        b=b2NAfpsXUtaVGxm8ELQk1qYTca8nPRuGMi3i90+mYJyoOfO9kR9QwsAgkretlw3sXc
         AULqg4jlZEFHMFKbLy9mEMv43CNnIvHJEFwxVznR0iSwbwQATy3wrtKpBRuKi4teixDF
         hdyHW/vCFHxD6rr+powUr+qa0WCsxkAeiw3WE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=hGMSYbsE9xAsUoP0tjP0ffdcGXIMfYrhmHVA7+J3ncxm78hcbRP+k1aLWojDWjaCSA
         bzGWmYGOoq75ce4Qrly8mREFQ7QYzgn94qZMM0rqLfWCyB5R2D/DQwpyJ53bxM+GBDE3
         lc/MR2I6U9xwM9/iNt8snwvg/k49ZITNWxZzg=
MIME-Version: 1.0
Received: by 10.216.176.83 with SMTP id a61mr1953090wem.47.1281750799970; Fri,
 13 Aug 2010 18:53:19 -0700 (PDT)
Received: by 10.216.135.129 with HTTP; Fri, 13 Aug 2010 18:53:19 -0700 (PDT)
In-Reply-To: <20100809184752.GA94111@goodenter-lm.local>
References: <AANLkTi=GyFLivH8uOQNppUJnVeixpY5HA98=0cJEpaNa@mail.gmail.com>
	<20100809184752.GA94111@goodenter-lm.local>
Date: Fri, 13 Aug 2010 18:53:19 -0700
Message-ID: <AANLkTin+66pFVCmwTbigiJ0u3dEDu4xdnt1x29pmzyyE@mail.gmail.com>
Subject: Re: [DISCUSSION] Combine committer lists in Common/HDFS/MapReduce
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e65aeed61f7236048dbedd22

--0016e65aeed61f7236048dbedd22
Content-Type: text/plain; charset=ISO-8859-1

+1 on combining committers list for common + hdfs + mapreduce. It is mostly
the same set of users who contribute to all three projects.

thanks,
dhruba


On Mon, Aug 9, 2010 at 11:47 AM, Konstantin Boudnik <cos@apache.org> wrote:

> +1 on combining the committer roles in one (as Doug has mentioned
> elsewhere,
> all three are feel like a single project and are released as such,
> therefore
> it doesn't make much sense to split committers). In effect, MR testing that
> pulls in "external" HDFS dependencies works as a poor-man integration
> testing.
>
> -1 on combining the projects back together. While having them separated
> might
> cause an extra maintenance burden, the same separation forces more clean
> design decisions.
>
> Cos
>
> On Fri, Aug 06, 2010 at 02:02PM, Chris Douglas wrote:
> > Hadoop developers tend to specialize in either HDFS or MapReduce, but
> > given that:
> >
> > 0) Granting karma to Common is routine for a committer in either
> > space; there are no Common-only committers
> > 1) The majority of committers have been grandfathered into committer
> > roles in all three projects
> > 2) Many patches to Common require corresponding commits to both HDFS
> > and MapReduce
> > 3) Review-then-commit is usually sufficient notice for interested
> > parties to comment
> > 4) There have been few problems with committers pushing in patches
> > without consulting someone more directly involved
> > 5) Everyone on the PMC gets commit rights to all subprojects, anyway
> >
> > Perhaps it would make sense to give up on separate committer roles
> > until the projects are separate TLPs.
> >
> > On the other hand:
> >
> > 0) Nobody has been independently added to both HDFS and MapReduce
> > since the projects were separated
> > 1) It could exacerbate the focus on MapReduce in HDFS, at the expense
> > of other projects (like HBase).
> > 2) HDFS and MapReduce are mostly independent communities and
> > codebases; expertise in one does not imply fluency in the other
> > 3) Granting veto power across projects can lead to deadlock despite
> > consensus within that community
> > 4) TLP status for either project may require untangling HDFS/MR roles
> > that could be distinguished now
> >
> > Personally, I'm in favor of combining the roles. I trust all six of
> > the committers made since the project split no less than those made
> > earlier. Further, version control is sufficient for recovering from
> > most, foreseeable issues. I have some concerns about "harmless"
> > commits pushed through without an audit by the subproject's
> > maintainers (a few in recent memory caused downtime in Y! clusters),
> > but combining the roles seems like a worthwhile experiment.
> >
> > Thoughts? -C
>



-- 
Connect to me at http://www.facebook.com/dhruba

--0016e65aeed61f7236048dbedd22--

From general-return-1939-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 14 02:41:44 2010
Return-Path: <general-return-1939-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6266 invoked from network); 14 Aug 2010 02:41:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Aug 2010 02:41:44 -0000
Received: (qmail 52028 invoked by uid 500); 14 Aug 2010 02:41:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51771 invoked by uid 500); 14 Aug 2010 02:41:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51763 invoked by uid 99); 14 Aug 2010 02:41:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Aug 2010 02:41:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Sat, 14 Aug 2010 02:41:40 +0000
Received: (qmail 6239 invoked by uid 99); 14 Aug 2010 02:41:20 -0000
Received: from localhost.apache.org (HELO mail-gx0-f176.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Aug 2010 02:41:20 +0000
Received: by gxk7 with SMTP id 7so1747313gxk.35
        for <general@hadoop.apache.org>; Fri, 13 Aug 2010 19:41:19 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.157.207 with SMTP id c15mr2242117ibx.143.1281753679003;
 Fri, 13 Aug 2010 19:41:19 -0700 (PDT)
Received: by 10.231.141.11 with HTTP; Fri, 13 Aug 2010 19:41:18 -0700 (PDT)
In-Reply-To: <4C62369D.6000703@yahoo-inc.com>
References: <AANLkTi=GyFLivH8uOQNppUJnVeixpY5HA98=0cJEpaNa@mail.gmail.com>
	<4C5C8302.3010008@apache.org>
	<97E2C026-0C44-4FDC-A569-63E218E03D59@yahoo-inc.com>
	<4C602C27.6060802@apache.org>
	<A7CB92D6-81C3-4550-BF38-FB02FD3516C6@yahoo-inc.com>
	<4C62369D.6000703@yahoo-inc.com>
Date: Fri, 13 Aug 2010 19:41:18 -0700
Message-ID: <AANLkTimVU=SgNADQTgbx9meO6m4QW3oJ7R+ooUX93xsq@mail.gmail.com>
Subject: Re: [DISCUSSION] Combine committer lists in Common/HDFS/MapReduce
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Aug 10, 2010 at 10:35 PM, Vinod KV <vinodkv@yahoo-inc.com> wrote:
>
> I know of very few to nearly zero number of patches in mapreduce that touch
> HDFS also. And vice versa. The common case is of patches that touch both
> common and mapreduce or common and hdfs.

This is a good point, though changes in Common and HDFS interfaces do
break MapReduce, particularly unit tests that take liberties with
interface visibility. It would be convenient if HDFS committers could
push in the fix with the original issue, to shrink the window where MR
is broken, but in practice such changes are usually committed in short
order. After all, most committers have commit access to all three
projects... though this is one of the reasons why the constraint
difficult to justify.

> I am one of those who has access to
> mapreduce project but not to common project and find more than 50% of the
> patches that fall into this category.

This *was* an oversight that should be corrected either through this
vote or independently.

> As for the separation of the repositories, I personally felt separation of
> mapreduce from hdfs helped focusing on things a lot better. The last gasp
> work done for 0.21, mostly by Tom, did help a lot in decoupling the
> projects. Common is the hot point, sure, but as others noted, that is a
> separate discussion.

+1

----

The discussion appears to be dying down. Quick summary of comments so far:

* That all HDFS and MapReduce committers should have commit rights to
Common appears to be undisputed.

* The value of splitting the projects at all is disputed. Some have
argued that it has complicated work without delivering the benefits to
developers it promised, though others have experienced this as
discipline rather than inconvenience. Most of these complaints
reference patches that touch Common, particularly actively-developed
packages like fs. The vote concerns a narrower question than the
project split, though it's fair to assert a lack of unanimity on the
premise of a split Hadoop, let alone the more limited question of
whether to retain a split list of committers.

* Not everyone agrees that combining HDFS and MapReduce committers is
sound. While there is sufficient overlap today to branch all three
together, patch releases could- and likely will- be cut independently.
Not everyone thinks the two projects are developing independent
communities, but none have difficulty imagining TLP status for both.

Please feel free to amend this if I left out an important point.

-----

I think we're ready to vote. Though we have no bylaws to amend, this
would be a modification to them, I guess. The last-proposed set
required a 2/3 majority of the PMC, IIRC. Since adding a committer
requires consensus on the PMC, it's probably fair to require a 2/3
majority to cross-pollenate lists instead of a simple majority.

Though the vote could be conducted on a huge cross-post to
mapreduce-dev@, hdfs-dev@ and common-dev@, it'll be easier to count if
it's run on general@; I'll start it here on Monday if nobody minds. -C

From general-return-1940-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 14 08:26:49 2010
Return-Path: <general-return-1940-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17826 invoked from network); 14 Aug 2010 08:26:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Aug 2010 08:26:48 -0000
Received: (qmail 53621 invoked by uid 500); 14 Aug 2010 08:26:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53351 invoked by uid 500); 14 Aug 2010 08:26:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53343 invoked by uid 99); 14 Aug 2010 08:26:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Aug 2010 08:26:43 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [203.209.230.50] (HELO n6e.bullet.cnb.yahoo.com) (203.209.230.50)
    by apache.org (qpsmtpd/0.29) with SMTP; Sat, 14 Aug 2010 08:26:36 +0000
Received: from [202.43.217.36] by n6.bullet.cnb.yahoo.com with NNFMP; 14 Aug 2010 08:26:13 -0000
Received: from [202.165.102.48] by t1.bullet.cnmail.cnb.yahoo.com with NNFMP; 14 Aug 2010 08:26:12 -0000
Received: from [127.0.0.1] by omp102.mail.cnb.yahoo.com with NNFMP; 14 Aug 2010 08:26:12 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 961170.98851.bm@omp102.mail.cnb.yahoo.com
Received: (qmail 20530 invoked by uid 60001); 14 Aug 2010 08:26:12 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com.cn; s=s1024; t=1281774372; bh=d86D/SKXktLWwMhYs11CTX9A9mVXAxzEqocLDe7EqmQ=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=QOc2f1I9HgFMLwsZb9chJqElTWdcGp+fDoJPlRcCOOMaiv0fWwVLAE25ueTddVG9+tJai513k6PQBaZH6XhIXQmHXIPUgoYqYR3vCQRx/9Q/StRPU+OF+KirH3XIz/HSfEnSE/TMLpO2qMCl4VBPBPzABu6ClaWDbm/j614z3t0=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com.cn;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=VDNfS8qgFnjc1AQ4crkFm3y/qi74zXcJA7D1KWudkRV/4jTKsn1G1H6UobkPUoeh/hBBoW5c6nux4ooQuE3slOXwWJgzrXsiMkX3bSofDYFhvZJJEskTpS2t8O0+Yo47xRQWs06yIK0rkRWLeY1upsuujKFVtMQI49Z3+iVcGpo=;
Message-ID: <768926.9887.qm@web15202.mail.cnb.yahoo.com>
X-YMail-OSG: B0WHZcAVM1ntQBXGA2Y1lEIyCE3uatoCry4WmKBOE_E0cvm
 Eel2eTgd.1wAkvYJRfZsBFQPNXu9y984BX0DbVKoVLo7z3HlSdhFz59Uc.uA
 OFPQ0_VQMDEV1stkQeWQemy04vAmOB_Fz7I7N4oLixZjXYBNPo4KR5m5SKnx
 u2aTKuTLgxvGkL9M6Kx2RvrsQPRfoUYDJbwv_iAodbsDFlexmfzdpDfX6Fv4
 fM8z07TIxrZk3s0pkiQMa1JYf0LIuyEZFU4Y2YMSomg--
Received: from [202.118.67.200] by web15202.mail.cnb.yahoo.com via HTTP; Sat, 14 Aug 2010 16:26:12 CST
X-Mailer: YahooMailClassic/11.3.2 YahooMailWebService/0.8.105.279950
Date: Sat, 14 Aug 2010 16:26:12 +0800 (CST)
From: Dennis <arsenepark@yahoo.com.cn>
Subject: Sort Input
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1923413541-1281774372=:9887"

--0-1923413541-1281774372=:9887
Content-Type: text/plain; charset=us-ascii

Hi, there, 

There is a "Sort" example in hadoop-0.20.2 release. Can someone give an example of the "SequenceFile" example?

Thanks,
Dennis



      
--0-1923413541-1281774372=:9887--


From general-return-1941-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 14 10:18:32 2010
Return-Path: <general-return-1941-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45556 invoked from network); 14 Aug 2010 10:18:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Aug 2010 10:18:32 -0000
Received: (qmail 91895 invoked by uid 500); 14 Aug 2010 10:18:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91428 invoked by uid 500); 14 Aug 2010 10:18:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91420 invoked by uid 99); 14 Aug 2010 10:18:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Aug 2010 10:18:27 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [202.43.216.212] (HELO n2b.bullet.cnb.yahoo.com) (202.43.216.212)
    by apache.org (qpsmtpd/0.29) with SMTP; Sat, 14 Aug 2010 10:18:17 +0000
Received: from [202.43.217.37] by n2.bullet.cnb.yahoo.com with NNFMP; 14 Aug 2010 10:17:51 -0000
Received: from [203.209.230.76] by t2.bullet.cnmail.cnb.yahoo.com with NNFMP; 14 Aug 2010 10:17:51 -0000
Received: from [127.0.0.1] by omp105.mail.cnb.yahoo.com with NNFMP; 14 Aug 2010 10:17:51 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 451237.43276.bm@omp105.mail.cnb.yahoo.com
Received: (qmail 89896 invoked by uid 60001); 14 Aug 2010 10:17:51 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com.cn; s=s1024; t=1281781071; bh=Oaw8fs8yPkCHh8SZ+gxGO5oE4YWgTudlMKeMRHHG0yM=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=eF8EcS0sEI/dQXdQjyDlKMVJlCfZo25cMNwZyFFQjCE5x/TATS4pdOZXqjOm7HMco/ZnKIcrXArhb5cSUYIbs1XZ6iHFmFsvuLSICypfcE4dozNrzA1XS81G6ytkX91y+Inj0IxNKxedGjsBI16BxP2ITfJURYeyD54xvY1HlYs=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com.cn;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=vhsQIt5ptZlsJAPERW9J5Gxua7Bp79LBSiHJ3jbGcOp3bPzDSqN24n+q72ODRJMHDyD8LBJ+rBUg46TIvwqWeLoSrDHxF0UeEz8V/Xu8XBg7/+krTnaNB8eXUEY3shBEZIyWPq4rQzZPyEuQur4P2QYYNPFk1HzOWjfGqx3R7+8=;
Message-ID: <305491.89880.qm@web15202.mail.cnb.yahoo.com>
X-YMail-OSG: egEQcUEVM1lfy.E_naywkLvwD1anZFKi4ueXpNxzFTpaq0K
 KV4LjlfXN0Eb7MH51YHrJLeYy8QWCCwaXA1m629Ru17XlAdRhOwrKJGHPfHF
 F1o6MjUuqUK_mh.nAiSJanLKSEDsMjmxmIh5UkEPNmYdCAdTfea02mbrTXED
 TiHOJVtEJOAgbo4mQ10iCCpWqbkLhIoiA8KxMyD2Z8YMjgtiGqDZzwD.Jvpn
 z7A3p9hvWAKwyalErD9RH.dzllw6WM1yr.mL2z5a3xLnXbAL92UJCyrBkZKc
 AYvltV5YIJdshShk-
Received: from [202.118.67.200] by web15202.mail.cnb.yahoo.com via HTTP; Sat, 14 Aug 2010 18:17:50 CST
X-Mailer: YahooMailClassic/11.3.2 YahooMailWebService/0.8.105.279950
Date: Sat, 14 Aug 2010 18:17:50 +0800 (CST)
From: Dennis <arsenepark@yahoo.com.cn>
Subject: Re: Sort Input
To: general@hadoop.apache.org
In-Reply-To: <768926.9887.qm@web15202.mail.cnb.yahoo.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1871874496-1281781070=:89880"

--0-1871874496-1281781070=:89880
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Just figured that out.
The "org.apache.hadoop.io.SequenceFile" class.


--- On Sat, 8/14/10, Dennis <arsenepark@yahoo.com.cn> wrote:

From: Dennis <arsenepark@yahoo.com.cn>
Subject: Sort Input
To: general@hadoop.apache.org
Date: Saturday, August 14, 2010, 4:26 PM

Hi, there,=20

There is a "Sort" example in hadoop-0.20.2 release. Can someone give an exa=
mple of the "SequenceFile" example?

Thanks,
Dennis



=A0 =A0 =A0 =0A=0A=0A      
--0-1871874496-1281781070=:89880--


From general-return-1942-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 15 03:37:49 2010
Return-Path: <general-return-1942-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47919 invoked from network); 15 Aug 2010 03:37:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Aug 2010 03:37:48 -0000
Received: (qmail 24234 invoked by uid 500); 15 Aug 2010 03:37:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23824 invoked by uid 500); 15 Aug 2010 03:37:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23816 invoked by uid 99); 15 Aug 2010 03:37:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Aug 2010 03:37:44 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of crystaldoll06@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Aug 2010 03:37:36 +0000
Received: by wwb22 with SMTP id 22so5734006wwb.29
        for <general@hadoop.apache.org>; Sat, 14 Aug 2010 20:37:16 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=CzVIJIl4biK+DgDVpKHJ8rR7gDNDY6oJzUucaURMvPs=;
        b=gLp/qOjOhLQ9N5ITTCefLYyVtlonJQuI/o0om//h/KCfXA/1Yhd64KvKbpHk7i5m4e
         yHylvDOtDh1rtwmlYL3z/FT/20Mx4d/wBE8PQbz/bj6Wi/8VX6Gl1CMIBnkK/upNnqgf
         br8PkARTrii1AUsNHl6Iw4k2jQsma2KgtF7UM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=nqDAg9gwzjvIc6wH5N7s+NeBFw278BR+FSe8XUFG7WcHFsNB4nn8OwK19UR+BKoJLf
         MIfpQwgg90FY2cQVJqQaVcZZz0xqjgIXHmO8Sv/PpWuIr8YuBZ6TMuzFQAI/cMW+BskY
         QsGF0+fAJsuP/8E8NhkKJJHIo4Brr87zauGM8=
MIME-Version: 1.0
Received: by 10.216.178.200 with SMTP id f50mr2978251wem.62.1281843436067;
 Sat, 14 Aug 2010 20:37:16 -0700 (PDT)
Received: by 10.216.237.90 with HTTP; Sat, 14 Aug 2010 20:37:15 -0700 (PDT)
Date: Sat, 14 Aug 2010 20:37:15 -0700
Message-ID: <AANLkTimTNY_EU8Eu4ZG3zNi78wOQNCwe0piJYuu_0DQU@mail.gmail.com>
Subject: Hadoop basics
From: Rita Liu <crystaldoll06@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016367fabb1aa1b6d048dd46e79
X-Virus-Checked: Checked by ClamAV on apache.org

--0016367fabb1aa1b6d048dd46e79
Content-Type: text/plain; charset=ISO-8859-1

Hi!

I am a total beginner, but I am very interested in hadoop. I've already
downloaded hadoop 0.19.2 and run on Ubuntu in single-node mode. Now I want
to do two things:

1. Explore how hadoop works internally with one of the example applications
hadoop provides
2. Write an application on my own

Those two things bring me following questions:

a. debugger?
I am stuck since I don't know how to "explore" hadoop. I used to trace
through the code using a debugger, but in this case, I don't know if there
is a good debugger to use; or -- maybe a debugger is not necessary for
hadoop? If not, then how do you trace through the code to either debug or
just gain an understanding about the system? May I know what you,
experienced experts, do? :)

b. Where to run hadoop?
Also -- may I know where you run your hadoop? Do you run on linux, or on VM
-- in particular, Cloudera? I heard that Cloudera is good for writing
mapreduce applications with hadoop itself as a blackbox; is it true? If my
ultimate goal is to understand how hadoop works internally, would it be
better if I directly run it on linux?

c. Single-node or multi-node?
In the beginning (just like my case :p) would it be better to use
single-node or multi-node? If the latter is true, should I obtain more
machines, or should I use more virtual machines to create more nodes?

As a newbie, I am sorry for all those basic (and silly, I know :$)
questions. If possible, please help me out? Any suggestion or advice will be
greatly appreciated. Thank you very much!

Best,
Rita :)

P.S. If my questions are not suitable for this mailing-list, please let me
apologize, and then, could you please direct me to other mailing-lists?
Sorry, and thanks a lot! :)

--0016367fabb1aa1b6d048dd46e79--

From general-return-1943-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 15 03:50:03 2010
Return-Path: <general-return-1943-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49009 invoked from network); 15 Aug 2010 03:50:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Aug 2010 03:50:02 -0000
Received: (qmail 30714 invoked by uid 500); 15 Aug 2010 03:50:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30561 invoked by uid 500); 15 Aug 2010 03:49:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30553 invoked by uid 99); 15 Aug 2010 03:49:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Aug 2010 03:49:58 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of thinke365@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Aug 2010 03:49:50 +0000
Received: by wwb22 with SMTP id 22so5741951wwb.29
        for <general@hadoop.apache.org>; Sat, 14 Aug 2010 20:49:30 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=QkRD85raJYRUQ/rkLIRKESjB5usvrm6gqYr+F/aePgY=;
        b=uBywzI2DulDRFw1ldQyp/ymTzOe59M0Vedm1JPbL0di2SxkidM8V6b4f2N/MJAbZ35
         uakgBN9VqbaYJSolkqCIzxWj4UaWupL9YjTtRo8jIIOtLovXb8jfY0f79NY87hvgSi/+
         25IFfjkNCFKNLPeraNlyP8UQxV0kuhohrYr0s=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Cp7ISrzehndkSRLxTPuCNiQ56M2WDiF9fUraFzj31aUvX/3pAfXPE/AUGKMKIv0Qkx
         QFavPNyWsZfLskYrvMw4u+YoWEtBSqUfa30nbRV2llgvA5AnZudGJDzY89TJdQhIKXV+
         dkRPVXVoqvcsVXI5yvlq3Pc71DetQYGXygNvk=
MIME-Version: 1.0
Received: by 10.216.54.73 with SMTP id h51mr1156998wec.100.1281844170272; Sat,
 14 Aug 2010 20:49:30 -0700 (PDT)
Received: by 10.216.234.7 with HTTP; Sat, 14 Aug 2010 20:49:30 -0700 (PDT)
In-Reply-To: <AANLkTimTNY_EU8Eu4ZG3zNi78wOQNCwe0piJYuu_0DQU@mail.gmail.com>
References: <AANLkTimTNY_EU8Eu4ZG3zNi78wOQNCwe0piJYuu_0DQU@mail.gmail.com>
Date: Sun, 15 Aug 2010 11:49:30 +0800
Message-ID: <AANLkTimr2=-0H0uj0npn7uS4bZFHquswkWq==2ai1cPm@mail.gmail.com>
Subject: Re: Hadoop basics
From: smith jack <thinke365@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

There is still a long way to go before you start debuging hadoop,
the first point is that, you should run hadoop first!
how to make this done.
simple run bin/start-all, then hadoop can response to your request.
you can use FsShell when you begin.
hope you will enjoy it

2010/8/15 Rita Liu <crystaldoll06@gmail.com>:
> Hi!
>
> I am a total beginner, but I am very interested in hadoop. I've already
> downloaded hadoop 0.19.2 and run on Ubuntu in single-node mode. Now I want
> to do two things:
>
> 1. Explore how hadoop works internally with one of the example applications
> hadoop provides
> 2. Write an application on my own
>
> Those two things bring me following questions:
>
> a. debugger?
> I am stuck since I don't know how to "explore" hadoop. I used to trace
> through the code using a debugger, but in this case, I don't know if there
> is a good debugger to use; or -- maybe a debugger is not necessary for
> hadoop? If not, then how do you trace through the code to either debug or
> just gain an understanding about the system? May I know what you,
> experienced experts, do? :)
>
> b. Where to run hadoop?
> Also -- may I know where you run your hadoop? Do you run on linux, or on VM
> -- in particular, Cloudera? I heard that Cloudera is good for writing
> mapreduce applications with hadoop itself as a blackbox; is it true? If my
> ultimate goal is to understand how hadoop works internally, would it be
> better if I directly run it on linux?
>
> c. Single-node or multi-node?
> In the beginning (just like my case :p) would it be better to use
> single-node or multi-node? If the latter is true, should I obtain more
> machines, or should I use more virtual machines to create more nodes?
>
> As a newbie, I am sorry for all those basic (and silly, I know :$)
> questions. If possible, please help me out? Any suggestion or advice will be
> greatly appreciated. Thank you very much!
>
> Best,
> Rita :)
>
> P.S. If my questions are not suitable for this mailing-list, please let me
> apologize, and then, could you please direct me to other mailing-lists?
> Sorry, and thanks a lot! :)
>

From general-return-1944-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 15 03:54:48 2010
Return-Path: <general-return-1944-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49198 invoked from network); 15 Aug 2010 03:54:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Aug 2010 03:54:48 -0000
Received: (qmail 32253 invoked by uid 500); 15 Aug 2010 03:54:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31895 invoked by uid 500); 15 Aug 2010 03:54:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31887 invoked by uid 99); 15 Aug 2010 03:54:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Aug 2010 03:54:43 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of crystaldoll06@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Aug 2010 03:54:37 +0000
Received: by wwb22 with SMTP id 22so5744820wwb.29
        for <general@hadoop.apache.org>; Sat, 14 Aug 2010 20:54:15 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=FGT8EvZA/IUqf0n/vSBjAKohTepBm86rcng0h7r1Lwc=;
        b=czM+3sFfdYBGr64WeOgTcnhgETtogGgTqjK644vbmELP2RiV4THT9yW8h5bpuLbwrT
         zo3nRbmRsqTdcS750Vuwa+LXhbW/Nce/IwoXSGRFNHXzNkWdYK3lwlCVVxzlHS9X6D+K
         VXQJmWp9fuLNsMK0uSwnUx4gIYsR4ONap+7LM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=tw1gBN4VKHSCqOjDNTsha7x1BGAgvBxKx7ZmAL6wA5ONQmM77ecH9HgSyPLG4laN7m
         YQ99WsXnn7S0qJSF8CoaBmbP4S1ulORKzLquxxmFWxnW+wTXDtYHNG3VczM5rFjF5uhw
         lHehS7qtdXTjdJd/vD3Dfclz5An5wnA4cJdzk=
MIME-Version: 1.0
Received: by 10.216.74.82 with SMTP id w60mr1163728wed.106.1281844455461; Sat,
 14 Aug 2010 20:54:15 -0700 (PDT)
Received: by 10.216.237.90 with HTTP; Sat, 14 Aug 2010 20:54:15 -0700 (PDT)
In-Reply-To: <AANLkTimr2=-0H0uj0npn7uS4bZFHquswkWq==2ai1cPm@mail.gmail.com>
References: <AANLkTimTNY_EU8Eu4ZG3zNi78wOQNCwe0piJYuu_0DQU@mail.gmail.com>
	<AANLkTimr2=-0H0uj0npn7uS4bZFHquswkWq==2ai1cPm@mail.gmail.com>
Date: Sat, 14 Aug 2010 20:54:15 -0700
Message-ID: <AANLkTinDaGQDUAKdiTeF0naFwt13CfX2fQ-7tT44TCDw@mail.gmail.com>
Subject: Re: Hadoop basics
From: Rita Liu <crystaldoll06@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502c8a46cd226048dd4ab4d

--00504502c8a46cd226048dd4ab4d
Content-Type: text/plain; charset=ISO-8859-1

Thanks so much! I did do that, and now I want to move to the next step. If
possible, may I know more to move on? Thank you very much!
-Rita :)

On Sat, Aug 14, 2010 at 8:49 PM, smith jack <thinke365@gmail.com> wrote:

> There is still a long way to go before you start debuging hadoop,
> the first point is that, you should run hadoop first!
> how to make this done.
> simple run bin/start-all, then hadoop can response to your request.
> you can use FsShell when you begin.
> hope you will enjoy it
>
> 2010/8/15 Rita Liu <crystaldoll06@gmail.com>:
> > Hi!
> >
> > I am a total beginner, but I am very interested in hadoop. I've already
> > downloaded hadoop 0.19.2 and run on Ubuntu in single-node mode. Now I
> want
> > to do two things:
> >
> > 1. Explore how hadoop works internally with one of the example
> applications
> > hadoop provides
> > 2. Write an application on my own
> >
> > Those two things bring me following questions:
> >
> > a. debugger?
> > I am stuck since I don't know how to "explore" hadoop. I used to trace
> > through the code using a debugger, but in this case, I don't know if
> there
> > is a good debugger to use; or -- maybe a debugger is not necessary for
> > hadoop? If not, then how do you trace through the code to either debug or
> > just gain an understanding about the system? May I know what you,
> > experienced experts, do? :)
> >
> > b. Where to run hadoop?
> > Also -- may I know where you run your hadoop? Do you run on linux, or on
> VM
> > -- in particular, Cloudera? I heard that Cloudera is good for writing
> > mapreduce applications with hadoop itself as a blackbox; is it true? If
> my
> > ultimate goal is to understand how hadoop works internally, would it be
> > better if I directly run it on linux?
> >
> > c. Single-node or multi-node?
> > In the beginning (just like my case :p) would it be better to use
> > single-node or multi-node? If the latter is true, should I obtain more
> > machines, or should I use more virtual machines to create more nodes?
> >
> > As a newbie, I am sorry for all those basic (and silly, I know :$)
> > questions. If possible, please help me out? Any suggestion or advice will
> be
> > greatly appreciated. Thank you very much!
> >
> > Best,
> > Rita :)
> >
> > P.S. If my questions are not suitable for this mailing-list, please let
> me
> > apologize, and then, could you please direct me to other mailing-lists?
> > Sorry, and thanks a lot! :)
> >
>

--00504502c8a46cd226048dd4ab4d--

From general-return-1945-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 15 04:42:59 2010
Return-Path: <general-return-1945-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 62291 invoked from network); 15 Aug 2010 04:42:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Aug 2010 04:42:59 -0000
Received: (qmail 42886 invoked by uid 500); 15 Aug 2010 04:42:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42568 invoked by uid 500); 15 Aug 2010 04:42:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42560 invoked by uid 99); 15 Aug 2010 04:42:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Aug 2010 04:42:54 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of piyushgarg80@gmail.com designates 209.85.212.176 as permitted sender)
Received: from [209.85.212.176] (HELO mail-px0-f176.google.com) (209.85.212.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Aug 2010 04:42:47 +0000
Received: by pxi3 with SMTP id 3so1985679pxi.35
        for <general@hadoop.apache.org>; Sat, 14 Aug 2010 21:42:27 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :x-enigmail-version:content-type;
        bh=2PHLa/TiIOVD3ORKM81PQoALH7Mhc2huMcBIl6Gh6Xg=;
        b=eXXB8uBe0RCvxfVBkFAzb97oqEAxrccPSXzkqhThzbQ57Dg9343pujxxKPTPMwdQZA
         2dgQL0yoIccZ50p+Exl2FJF1Mk3l7e10ktHiNwPTx3+DhAgXCqeLMc56fyNT8nFZ0o28
         lrxf2kmEVgMdkhUZQ6sipeS93Ui1smuk0TwYQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:x-enigmail-version:content-type;
        b=Q0zCCxUnuMqfxe5RBY9fk2wEHAjFz+NHSafXzuTa43zeDe/1+uIJlFbhYH7/0g6Zj+
         JzFwpUkzJFrGZG/si9bgTwLznK5l/A7P8cynR1mjkEo0tpkuEw4NobW7G36z85QCt5+s
         2Wng3ODZp3+q81Sz9qMBWbx1RVsCrdMsV7pmA=
Received: by 10.114.175.14 with SMTP id x14mr4171406wae.152.1281847347226;
        Sat, 14 Aug 2010 21:42:27 -0700 (PDT)
Received: from [192.168.2.3] ([115.99.74.22])
        by mx.google.com with ESMTPS id d38sm8752862wam.20.2010.08.14.21.42.23
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Sat, 14 Aug 2010 21:42:26 -0700 (PDT)
Message-ID: <4C677028.8000502@gmail.com>
Date: Sun, 15 Aug 2010 10:12:16 +0530
From: Piyush Garg <piyushgarg80@gmail.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.11) Gecko/20100713 Lightning/1.0b1 Thunderbird/3.0.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Hadoop basics
References: <AANLkTimTNY_EU8Eu4ZG3zNi78wOQNCwe0piJYuu_0DQU@mail.gmail.com>
In-Reply-To: <AANLkTimTNY_EU8Eu4ZG3zNi78wOQNCwe0piJYuu_0DQU@mail.gmail.com>
X-Enigmail-Version: 1.0.1
Content-Type: multipart/alternative;
 boundary="------------060201020506090909060301"

--------------060201020506090909060301
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi Rita,

I have just started to learn hadoop as well, I know there is a long way
to go.
I found some useful links which I am sharing with you.

Hadoop Tutorial - YDN
<http://developer.yahoo.com/hadoop/tutorial/index.html> excellent
beginners tutorial and well organized.
Running Hadoop On Ubuntu Linux (Single-Node Cluster) - Michael G. Noll
<http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Single-Node_Cluster%29>
Running_Hadoop_On_Ubuntu_Linux_(Multi-Node_Cluster)
<http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Multi-Node_Cluster%29>
The tutorial on the hadoop wiki
<http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html> is
too much for a beginner.

Debugger:
I do not think you can easily do debugging using remote debugger. This
is natural since hadoop is not sequential programming, it would be very
difficult to debug its apps.
The only way to debug is to use traces.

I think you can learn how to setup multi-node cluster, but for practice
session you can use single node setup.

Lets see what the experts say.

Thanks and Regards
Piyush Garg


On Sunday 15 August 2010 09:07 AM, Rita Liu wrote:
> Hi!
>
> I am a total beginner, but I am very interested in hadoop. I've already
> downloaded hadoop 0.19.2 and run on Ubuntu in single-node mode. Now I want
> to do two things:
>
> 1. Explore how hadoop works internally with one of the example applications
> hadoop provides
> 2. Write an application on my own
>
> Those two things bring me following questions:
>
> a. debugger?
> I am stuck since I don't know how to "explore" hadoop. I used to trace
> through the code using a debugger, but in this case, I don't know if there
> is a good debugger to use; or -- maybe a debugger is not necessary for
> hadoop? If not, then how do you trace through the code to either debug or
> just gain an understanding about the system? May I know what you,
> experienced experts, do? :)
>
> b. Where to run hadoop?
> Also -- may I know where you run your hadoop? Do you run on linux, or on VM
> -- in particular, Cloudera? I heard that Cloudera is good for writing
> mapreduce applications with hadoop itself as a blackbox; is it true? If my
> ultimate goal is to understand how hadoop works internally, would it be
> better if I directly run it on linux?
>
> c. Single-node or multi-node?
> In the beginning (just like my case :p) would it be better to use
> single-node or multi-node? If the latter is true, should I obtain more
> machines, or should I use more virtual machines to create more nodes?
>
> As a newbie, I am sorry for all those basic (and silly, I know :$)
> questions. If possible, please help me out? Any suggestion or advice will be
> greatly appreciated. Thank you very much!
>
> Best,
> Rita :)
>
> P.S. If my questions are not suitable for this mailing-list, please let me
> apologize, and then, could you please direct me to other mailing-lists?
> Sorry, and thanks a lot! :)
>
>   

--------------060201020506090909060301--

From general-return-1946-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 15 04:50:38 2010
Return-Path: <general-return-1946-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 62683 invoked from network); 15 Aug 2010 04:50:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Aug 2010 04:50:38 -0000
Received: (qmail 44140 invoked by uid 500); 15 Aug 2010 04:50:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 43978 invoked by uid 500); 15 Aug 2010 04:50:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 43970 invoked by uid 99); 15 Aug 2010 04:50:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Aug 2010 04:50:35 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of crystaldoll06@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Aug 2010 04:50:28 +0000
Received: by wyb35 with SMTP id 35so6002092wyb.35
        for <general@hadoop.apache.org>; Sat, 14 Aug 2010 21:50:08 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=MEkS1oaDT/G7TZ66u2RRt9bvKeKC3tHHz9bbQ0M4VE4=;
        b=TYZjm5IBr0h/dfXNSqpvCp4XcX30ySOhKnJXgyCn/uKlSIWuk/CfOJJfhpC+gfGug1
         Yp8mDkAY4sr17I5eIX5SlOOrE+o7pNNrGCmoNcKng3R0Da84H4e/9Hpd1VEYAGMehv1y
         islagL9G7H39uwqZUe00buDCJgoKD9rG7tDIM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=g+0LIqljPqsOAYhs0YZMobg3XoPjXr1uE+czR5z5XBmrgRdXHOrpy+rwH9bThDSTnT
         p4lQlXqirCNLF6T/zOuSYAsi7/X1/BrNiBcFrfrXrAbr1StF5rZS1+yVU6RfPxe+A43z
         j0w2mt8x/5aTblKivzv0u7vBWI/oz09U5dNtE=
MIME-Version: 1.0
Received: by 10.216.12.8 with SMTP id 8mr1184846wey.107.1281847807546; Sat, 14
 Aug 2010 21:50:07 -0700 (PDT)
Received: by 10.216.237.90 with HTTP; Sat, 14 Aug 2010 21:50:07 -0700 (PDT)
In-Reply-To: <4C677028.8000502@gmail.com>
References: <AANLkTimTNY_EU8Eu4ZG3zNi78wOQNCwe0piJYuu_0DQU@mail.gmail.com>
	<4C677028.8000502@gmail.com>
Date: Sat, 14 Aug 2010 21:50:07 -0700
Message-ID: <AANLkTi=ys3g3VUg+o=NvxCOWo_exPBa-jwNHyJ0671_F@mail.gmail.com>
Subject: Re: Hadoop basics
From: Rita Liu <crystaldoll06@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64c1e00399609048dd573bf
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e64c1e00399609048dd573bf
Content-Type: text/plain; charset=ISO-8859-1

Thank you very much, Piyush! :) May I know more about how to use "traces"?

And -- yes, please teach me if possible, experts! :)

Thanks a lot,
-Rita :))

On Sat, Aug 14, 2010 at 9:42 PM, Piyush Garg <piyushgarg80@gmail.com> wrote:

> Hi Rita,
>
> I have just started to learn hadoop as well, I know there is a long way
> to go.
> I found some useful links which I am sharing with you.
>
> Hadoop Tutorial - YDN
> <http://developer.yahoo.com/hadoop/tutorial/index.html> excellent
> beginners tutorial and well organized.
> Running Hadoop On Ubuntu Linux (Single-Node Cluster) - Michael G. Noll
> <
> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Single-Node_Cluster%29
> >
> Running_Hadoop_On_Ubuntu_Linux_(Multi-Node_Cluster)
> <
> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Multi-Node_Cluster%29
> >
> The tutorial on the hadoop wiki
> <http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html> is
> too much for a beginner.
>
> Debugger:
> I do not think you can easily do debugging using remote debugger. This
> is natural since hadoop is not sequential programming, it would be very
> difficult to debug its apps.
> The only way to debug is to use traces.
>
> I think you can learn how to setup multi-node cluster, but for practice
> session you can use single node setup.
>
> Lets see what the experts say.
>
> Thanks and Regards
> Piyush Garg
>
>
> On Sunday 15 August 2010 09:07 AM, Rita Liu wrote:
> > Hi!
> >
> > I am a total beginner, but I am very interested in hadoop. I've already
> > downloaded hadoop 0.19.2 and run on Ubuntu in single-node mode. Now I
> want
> > to do two things:
> >
> > 1. Explore how hadoop works internally with one of the example
> applications
> > hadoop provides
> > 2. Write an application on my own
> >
> > Those two things bring me following questions:
> >
> > a. debugger?
> > I am stuck since I don't know how to "explore" hadoop. I used to trace
> > through the code using a debugger, but in this case, I don't know if
> there
> > is a good debugger to use; or -- maybe a debugger is not necessary for
> > hadoop? If not, then how do you trace through the code to either debug or
> > just gain an understanding about the system? May I know what you,
> > experienced experts, do? :)
> >
> > b. Where to run hadoop?
> > Also -- may I know where you run your hadoop? Do you run on linux, or on
> VM
> > -- in particular, Cloudera? I heard that Cloudera is good for writing
> > mapreduce applications with hadoop itself as a blackbox; is it true? If
> my
> > ultimate goal is to understand how hadoop works internally, would it be
> > better if I directly run it on linux?
> >
> > c. Single-node or multi-node?
> > In the beginning (just like my case :p) would it be better to use
> > single-node or multi-node? If the latter is true, should I obtain more
> > machines, or should I use more virtual machines to create more nodes?
> >
> > As a newbie, I am sorry for all those basic (and silly, I know :$)
> > questions. If possible, please help me out? Any suggestion or advice will
> be
> > greatly appreciated. Thank you very much!
> >
> > Best,
> > Rita :)
> >
> > P.S. If my questions are not suitable for this mailing-list, please let
> me
> > apologize, and then, could you please direct me to other mailing-lists?
> > Sorry, and thanks a lot! :)
> >
> >
>

--0016e64c1e00399609048dd573bf--

From general-return-1947-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 15 05:06:14 2010
Return-Path: <general-return-1947-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68742 invoked from network); 15 Aug 2010 05:06:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Aug 2010 05:06:14 -0000
Received: (qmail 46938 invoked by uid 500); 15 Aug 2010 05:06:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46800 invoked by uid 500); 15 Aug 2010 05:06:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46792 invoked by uid 99); 15 Aug 2010 05:06:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Aug 2010 05:06:10 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of piyushgarg80@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Aug 2010 05:06:01 +0000
Received: by pwj9 with SMTP id 9so996203pwj.35
        for <general@hadoop.apache.org>; Sat, 14 Aug 2010 22:05:40 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :x-enigmail-version:content-type;
        bh=yIx5+WAJZMrJ6HEoXwHGu0w8S5PzMNeUzR+FWk5qIvE=;
        b=qUIANDasyi290dFzq2FjxiZWZyvh55PVmqJxYQruxrWwtUErx+YmuAj7AkBs6A5aSi
         6/MEQxkD8pD5eBpgG/JK0h7YZA+ZpWA+22sF/H7TQhBkwWxl1tjwNFCZ4pqBd+1fVMDV
         hfz0rLJHMD3VhVvrMRd2FdRNrkhh1MI6ZQoXA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:x-enigmail-version:content-type;
        b=uznYU18ZCOQTh++Oz+kGVx9IwGMJylmCu3rvNAXHLNmj8jUnuy5gj5YAxBLLqsItWP
         BeWmezJfdEBDYlgvBxkNx5y9O5CqHJvO1zGuqYUOdvnS8QG/EVRW2iIupTrvnyh0oFRw
         2nPkMYX0rpEQOb039o+hOwsD2gfezgzla+NQk=
Received: by 10.114.61.8 with SMTP id j8mr4165074waa.228.1281848739270;
        Sat, 14 Aug 2010 22:05:39 -0700 (PDT)
Received: from [192.168.2.3] ([115.99.74.22])
        by mx.google.com with ESMTPS id q6sm8803568waj.10.2010.08.14.22.05.28
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Sat, 14 Aug 2010 22:05:38 -0700 (PDT)
Message-ID: <4C67757B.8050108@gmail.com>
Date: Sun, 15 Aug 2010 10:34:59 +0530
From: Piyush Garg <piyushgarg80@gmail.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.11) Gecko/20100713 Lightning/1.0b1 Thunderbird/3.0.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Hadoop basics
References: <AANLkTimTNY_EU8Eu4ZG3zNi78wOQNCwe0piJYuu_0DQU@mail.gmail.com>	<4C677028.8000502@gmail.com> <AANLkTi=ys3g3VUg+o=NvxCOWo_exPBa-jwNHyJ0671_F@mail.gmail.com>
In-Reply-To: <AANLkTi=ys3g3VUg+o=NvxCOWo_exPBa-jwNHyJ0671_F@mail.gmail.com>
X-Enigmail-Version: 1.0.1
Content-Type: multipart/alternative;
 boundary="------------070308090400000409070802"
X-Virus-Checked: Checked by ClamAV on apache.org

--------------070308090400000409070802
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi Rita,

You can put log4j logger debug statements in the code. log4j library is
part of hadoop framework and there is already a log4j.properties file in
hadoop conf directory and all the output logs are saved in hadoop logs
directory.

Thanks and Regards
Piyush Garg


On Sunday 15 August 2010 10:20 AM, Rita Liu wrote:
> Thank you very much, Piyush! :) May I know more about how to use "traces"?
>
> And -- yes, please teach me if possible, experts! :)
>
> Thanks a lot,
> -Rita :))
>
> On Sat, Aug 14, 2010 at 9:42 PM, Piyush Garg <piyushgarg80@gmail.com> wrote:
>
>   
>> Hi Rita,
>>
>> I have just started to learn hadoop as well, I know there is a long way
>> to go.
>> I found some useful links which I am sharing with you.
>>
>> Hadoop Tutorial - YDN
>> <http://developer.yahoo.com/hadoop/tutorial/index.html> excellent
>> beginners tutorial and well organized.
>> Running Hadoop On Ubuntu Linux (Single-Node Cluster) - Michael G. Noll
>> <
>> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Single-Node_Cluster%29
>>     
>>>       
>> Running_Hadoop_On_Ubuntu_Linux_(Multi-Node_Cluster)
>> <
>> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Multi-Node_Cluster%29
>>     
>>>       
>> The tutorial on the hadoop wiki
>> <http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html> is
>> too much for a beginner.
>>
>> Debugger:
>> I do not think you can easily do debugging using remote debugger. This
>> is natural since hadoop is not sequential programming, it would be very
>> difficult to debug its apps.
>> The only way to debug is to use traces.
>>
>> I think you can learn how to setup multi-node cluster, but for practice
>> session you can use single node setup.
>>
>> Lets see what the experts say.
>>
>> Thanks and Regards
>> Piyush Garg
>>
>>
>> On Sunday 15 August 2010 09:07 AM, Rita Liu wrote:
>>     
>>> Hi!
>>>
>>> I am a total beginner, but I am very interested in hadoop. I've already
>>> downloaded hadoop 0.19.2 and run on Ubuntu in single-node mode. Now I
>>>       
>> want
>>     
>>> to do two things:
>>>
>>> 1. Explore how hadoop works internally with one of the example
>>>       
>> applications
>>     
>>> hadoop provides
>>> 2. Write an application on my own
>>>
>>> Those two things bring me following questions:
>>>
>>> a. debugger?
>>> I am stuck since I don't know how to "explore" hadoop. I used to trace
>>> through the code using a debugger, but in this case, I don't know if
>>>       
>> there
>>     
>>> is a good debugger to use; or -- maybe a debugger is not necessary for
>>> hadoop? If not, then how do you trace through the code to either debug or
>>> just gain an understanding about the system? May I know what you,
>>> experienced experts, do? :)
>>>
>>> b. Where to run hadoop?
>>> Also -- may I know where you run your hadoop? Do you run on linux, or on
>>>       
>> VM
>>     
>>> -- in particular, Cloudera? I heard that Cloudera is good for writing
>>> mapreduce applications with hadoop itself as a blackbox; is it true? If
>>>       
>> my
>>     
>>> ultimate goal is to understand how hadoop works internally, would it be
>>> better if I directly run it on linux?
>>>
>>> c. Single-node or multi-node?
>>> In the beginning (just like my case :p) would it be better to use
>>> single-node or multi-node? If the latter is true, should I obtain more
>>> machines, or should I use more virtual machines to create more nodes?
>>>
>>> As a newbie, I am sorry for all those basic (and silly, I know :$)
>>> questions. If possible, please help me out? Any suggestion or advice will
>>>       
>> be
>>     
>>> greatly appreciated. Thank you very much!
>>>
>>> Best,
>>> Rita :)
>>>
>>> P.S. If my questions are not suitable for this mailing-list, please let
>>>       
>> me
>>     
>>> apologize, and then, could you please direct me to other mailing-lists?
>>> Sorry, and thanks a lot! :)
>>>
>>>
>>>       
>>     
>   

--------------070308090400000409070802--

From general-return-1948-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 15 05:09:37 2010
Return-Path: <general-return-1948-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69511 invoked from network); 15 Aug 2010 05:09:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Aug 2010 05:09:37 -0000
Received: (qmail 49092 invoked by uid 500); 15 Aug 2010 05:09:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48923 invoked by uid 500); 15 Aug 2010 05:09:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48915 invoked by uid 99); 15 Aug 2010 05:09:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Aug 2010 05:09:33 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of thinke365@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Aug 2010 05:09:28 +0000
Received: by wwb22 with SMTP id 22so5793614wwb.29
        for <general@hadoop.apache.org>; Sat, 14 Aug 2010 22:09:07 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=WbUwiXzpuUOH8vVp44SpPue0Z5AItaa69ApvwIcjFdI=;
        b=oT9bG9fHPf9bfMc9JU6pxGTIyhF//NU9vhky3gd/hFH+4bGvw0QfwSmAd7dtcppOuE
         YMlfS61+3YaKGckdnhFkJmtMIAPclod6vAsW/qvk3fFaAm51b2pivpIYiAAWduAL+qdd
         c8mqTSLGWbVyu2eHNoApiS8dqySgDJIwZ2h/E=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=uIicOX8J2tsNvCmmgQbcxH5TvNT21d0sUSfoLAmevsQErTpD/BMAKklPruPhYAH3pp
         H96APxAQ3dzR1ajLrzPCKgd+Qe87gZiNMElmoxDHOznYwjR6fPSjwBoTPXyIkT/2jNKm
         4RKjvpeGCA1E3/RNGYtE1DpoBSeO4RcgBG6TQ=
MIME-Version: 1.0
Received: by 10.216.10.145 with SMTP id 17mr3070804wev.27.1281848946224; Sat,
 14 Aug 2010 22:09:06 -0700 (PDT)
Received: by 10.216.234.7 with HTTP; Sat, 14 Aug 2010 22:09:06 -0700 (PDT)
In-Reply-To: <4C67757B.8050108@gmail.com>
References: <AANLkTimTNY_EU8Eu4ZG3zNi78wOQNCwe0piJYuu_0DQU@mail.gmail.com>
	<4C677028.8000502@gmail.com>
	<AANLkTi=ys3g3VUg+o=NvxCOWo_exPBa-jwNHyJ0671_F@mail.gmail.com>
	<4C67757B.8050108@gmail.com>
Date: Sun, 15 Aug 2010 13:09:06 +0800
Message-ID: <AANLkTinEG-uLPonp8GVw5c_s-ZYhzeo+3wZ2Bv7g2q+G@mail.gmail.com>
Subject: Re: Hadoop basics
From: smith jack <thinke365@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

that means you can only trace by log,
and not possible to debug hadoop using step debug, haha
distributed system always introduce extra complexity and confusing issues.

2010/8/15 Piyush Garg <piyushgarg80@gmail.com>:
> Hi Rita,
>
> You can put log4j logger debug statements in the code. log4j library is
> part of hadoop framework and there is already a log4j.properties file in
> hadoop conf directory and all the output logs are saved in hadoop logs
> directory.
>
> Thanks and Regards
> Piyush Garg
>
>
> On Sunday 15 August 2010 10:20 AM, Rita Liu wrote:
>> Thank you very much, Piyush! :) May I know more about how to use "traces"?
>>
>> And -- yes, please teach me if possible, experts! :)
>>
>> Thanks a lot,
>> -Rita :))
>>
>> On Sat, Aug 14, 2010 at 9:42 PM, Piyush Garg <piyushgarg80@gmail.com> wrote:
>>
>>
>>> Hi Rita,
>>>
>>> I have just started to learn hadoop as well, I know there is a long way
>>> to go.
>>> I found some useful links which I am sharing with you.
>>>
>>> Hadoop Tutorial - YDN
>>> <http://developer.yahoo.com/hadoop/tutorial/index.html> excellent
>>> beginners tutorial and well organized.
>>> Running Hadoop On Ubuntu Linux (Single-Node Cluster) - Michael G. Noll
>>> <
>>> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Single-Node_Cluster%29
>>>
>>>>
>>> Running_Hadoop_On_Ubuntu_Linux_(Multi-Node_Cluster)
>>> <
>>> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Multi-Node_Cluster%29
>>>
>>>>
>>> The tutorial on the hadoop wiki
>>> <http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html> is
>>> too much for a beginner.
>>>
>>> Debugger:
>>> I do not think you can easily do debugging using remote debugger. This
>>> is natural since hadoop is not sequential programming, it would be very
>>> difficult to debug its apps.
>>> The only way to debug is to use traces.
>>>
>>> I think you can learn how to setup multi-node cluster, but for practice
>>> session you can use single node setup.
>>>
>>> Lets see what the experts say.
>>>
>>> Thanks and Regards
>>> Piyush Garg
>>>
>>>
>>> On Sunday 15 August 2010 09:07 AM, Rita Liu wrote:
>>>
>>>> Hi!
>>>>
>>>> I am a total beginner, but I am very interested in hadoop. I've already
>>>> downloaded hadoop 0.19.2 and run on Ubuntu in single-node mode. Now I
>>>>
>>> want
>>>
>>>> to do two things:
>>>>
>>>> 1. Explore how hadoop works internally with one of the example
>>>>
>>> applications
>>>
>>>> hadoop provides
>>>> 2. Write an application on my own
>>>>
>>>> Those two things bring me following questions:
>>>>
>>>> a. debugger?
>>>> I am stuck since I don't know how to "explore" hadoop. I used to trace
>>>> through the code using a debugger, but in this case, I don't know if
>>>>
>>> there
>>>
>>>> is a good debugger to use; or -- maybe a debugger is not necessary for
>>>> hadoop? If not, then how do you trace through the code to either debug or
>>>> just gain an understanding about the system? May I know what you,
>>>> experienced experts, do? :)
>>>>
>>>> b. Where to run hadoop?
>>>> Also -- may I know where you run your hadoop? Do you run on linux, or on
>>>>
>>> VM
>>>
>>>> -- in particular, Cloudera? I heard that Cloudera is good for writing
>>>> mapreduce applications with hadoop itself as a blackbox; is it true? If
>>>>
>>> my
>>>
>>>> ultimate goal is to understand how hadoop works internally, would it be
>>>> better if I directly run it on linux?
>>>>
>>>> c. Single-node or multi-node?
>>>> In the beginning (just like my case :p) would it be better to use
>>>> single-node or multi-node? If the latter is true, should I obtain more
>>>> machines, or should I use more virtual machines to create more nodes?
>>>>
>>>> As a newbie, I am sorry for all those basic (and silly, I know :$)
>>>> questions. If possible, please help me out? Any suggestion or advice will
>>>>
>>> be
>>>
>>>> greatly appreciated. Thank you very much!
>>>>
>>>> Best,
>>>> Rita :)
>>>>
>>>> P.S. If my questions are not suitable for this mailing-list, please let
>>>>
>>> me
>>>
>>>> apologize, and then, could you please direct me to other mailing-lists?
>>>> Sorry, and thanks a lot! :)
>>>>
>>>>
>>>>
>>>
>>
>

From general-return-1949-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 15 05:11:25 2010
Return-Path: <general-return-1949-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70078 invoked from network); 15 Aug 2010 05:11:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Aug 2010 05:11:25 -0000
Received: (qmail 50408 invoked by uid 500); 15 Aug 2010 05:11:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50208 invoked by uid 500); 15 Aug 2010 05:11:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50200 invoked by uid 99); 15 Aug 2010 05:11:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Aug 2010 05:11:21 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of crystaldoll06@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Aug 2010 05:11:13 +0000
Received: by wyb35 with SMTP id 35so6015391wyb.35
        for <general@hadoop.apache.org>; Sat, 14 Aug 2010 22:10:53 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=d7LmToOfPx4RdzPUV4hqka0t3sn5iAU369HADlrbVSk=;
        b=aH5id5+7m4mqeT5MglbMnnLjoKxYExoUxr6OjaeERMJ/r2j/gEUJrpTcWLxlYUV9nb
         CfqFSgqPln7NmDscIIaEo4lB4rD+1j+MQRJ58vYHtprPEBynL9VQsWDNa3f71RIhaE8V
         8VsGPEsHaPLyzG5GFUyhNmyL2bQ23SkW9PFdg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=eeN2/rJrY1ehOy3gbnCO894TBNLTZWbD/VXetIGSJljvL9D0Io3INF/FhJzzyOLVZm
         XL3/rfTsg0k8VO9awkJ7cCMT4g3AvYtyLVsGOORHfr1aAWWWPsSA6B7P8o7eA9ERGCHG
         TrJqVdU3PPx4+sy8uI1FxoF78ZhUAtv6hCP9s=
MIME-Version: 1.0
Received: by 10.216.232.144 with SMTP id n16mr3031218weq.1.1281849052982; Sat,
 14 Aug 2010 22:10:52 -0700 (PDT)
Received: by 10.216.237.90 with HTTP; Sat, 14 Aug 2010 22:10:52 -0700 (PDT)
In-Reply-To: <AANLkTinEG-uLPonp8GVw5c_s-ZYhzeo+3wZ2Bv7g2q+G@mail.gmail.com>
References: <AANLkTimTNY_EU8Eu4ZG3zNi78wOQNCwe0piJYuu_0DQU@mail.gmail.com>
	<4C677028.8000502@gmail.com>
	<AANLkTi=ys3g3VUg+o=NvxCOWo_exPBa-jwNHyJ0671_F@mail.gmail.com>
	<4C67757B.8050108@gmail.com>
	<AANLkTinEG-uLPonp8GVw5c_s-ZYhzeo+3wZ2Bv7g2q+G@mail.gmail.com>
Date: Sat, 14 Aug 2010 22:10:52 -0700
Message-ID: <AANLkTinoHN1-OaytcwmPeTWPnq95CNDzwHc=P1NOZ90n@mail.gmail.com>
Subject: Re: Hadoop basics
From: Rita Liu <crystaldoll06@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00151758ae6c756e2a048dd5bd9e
X-Virus-Checked: Checked by ClamAV on apache.org

--00151758ae6c756e2a048dd5bd9e
Content-Type: text/plain; charset=ISO-8859-1

Thank you very much, Piyush! I'll do as you say :DD Thanks a lot!!

Thanks Smith :) hmm ... I see. ok :)

Please give me more guidance and suggestions if possible, dear experts!
-Rita :))

On Sat, Aug 14, 2010 at 10:09 PM, smith jack <thinke365@gmail.com> wrote:

> that means you can only trace by log,
> and not possible to debug hadoop using step debug, haha
> distributed system always introduce extra complexity and confusing issues.
>
> 2010/8/15 Piyush Garg <piyushgarg80@gmail.com>:
> > Hi Rita,
> >
> > You can put log4j logger debug statements in the code. log4j library is
> > part of hadoop framework and there is already a log4j.properties file in
> > hadoop conf directory and all the output logs are saved in hadoop logs
> > directory.
> >
> > Thanks and Regards
> > Piyush Garg
> >
> >
> > On Sunday 15 August 2010 10:20 AM, Rita Liu wrote:
> >> Thank you very much, Piyush! :) May I know more about how to use
> "traces"?
> >>
> >> And -- yes, please teach me if possible, experts! :)
> >>
> >> Thanks a lot,
> >> -Rita :))
> >>
> >> On Sat, Aug 14, 2010 at 9:42 PM, Piyush Garg <piyushgarg80@gmail.com>
> wrote:
> >>
> >>
> >>> Hi Rita,
> >>>
> >>> I have just started to learn hadoop as well, I know there is a long way
> >>> to go.
> >>> I found some useful links which I am sharing with you.
> >>>
> >>> Hadoop Tutorial - YDN
> >>> <http://developer.yahoo.com/hadoop/tutorial/index.html> excellent
> >>> beginners tutorial and well organized.
> >>> Running Hadoop On Ubuntu Linux (Single-Node Cluster) - Michael G. Noll
> >>> <
> >>>
> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Single-Node_Cluster%29
> >>>
> >>>>
> >>> Running_Hadoop_On_Ubuntu_Linux_(Multi-Node_Cluster)
> >>> <
> >>>
> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Multi-Node_Cluster%29
> >>>
> >>>>
> >>> The tutorial on the hadoop wiki
> >>> <http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html> is
> >>> too much for a beginner.
> >>>
> >>> Debugger:
> >>> I do not think you can easily do debugging using remote debugger. This
> >>> is natural since hadoop is not sequential programming, it would be very
> >>> difficult to debug its apps.
> >>> The only way to debug is to use traces.
> >>>
> >>> I think you can learn how to setup multi-node cluster, but for practice
> >>> session you can use single node setup.
> >>>
> >>> Lets see what the experts say.
> >>>
> >>> Thanks and Regards
> >>> Piyush Garg
> >>>
> >>>
> >>> On Sunday 15 August 2010 09:07 AM, Rita Liu wrote:
> >>>
> >>>> Hi!
> >>>>
> >>>> I am a total beginner, but I am very interested in hadoop. I've
> already
> >>>> downloaded hadoop 0.19.2 and run on Ubuntu in single-node mode. Now I
> >>>>
> >>> want
> >>>
> >>>> to do two things:
> >>>>
> >>>> 1. Explore how hadoop works internally with one of the example
> >>>>
> >>> applications
> >>>
> >>>> hadoop provides
> >>>> 2. Write an application on my own
> >>>>
> >>>> Those two things bring me following questions:
> >>>>
> >>>> a. debugger?
> >>>> I am stuck since I don't know how to "explore" hadoop. I used to trace
> >>>> through the code using a debugger, but in this case, I don't know if
> >>>>
> >>> there
> >>>
> >>>> is a good debugger to use; or -- maybe a debugger is not necessary for
> >>>> hadoop? If not, then how do you trace through the code to either debug
> or
> >>>> just gain an understanding about the system? May I know what you,
> >>>> experienced experts, do? :)
> >>>>
> >>>> b. Where to run hadoop?
> >>>> Also -- may I know where you run your hadoop? Do you run on linux, or
> on
> >>>>
> >>> VM
> >>>
> >>>> -- in particular, Cloudera? I heard that Cloudera is good for writing
> >>>> mapreduce applications with hadoop itself as a blackbox; is it true?
> If
> >>>>
> >>> my
> >>>
> >>>> ultimate goal is to understand how hadoop works internally, would it
> be
> >>>> better if I directly run it on linux?
> >>>>
> >>>> c. Single-node or multi-node?
> >>>> In the beginning (just like my case :p) would it be better to use
> >>>> single-node or multi-node? If the latter is true, should I obtain more
> >>>> machines, or should I use more virtual machines to create more nodes?
> >>>>
> >>>> As a newbie, I am sorry for all those basic (and silly, I know :$)
> >>>> questions. If possible, please help me out? Any suggestion or advice
> will
> >>>>
> >>> be
> >>>
> >>>> greatly appreciated. Thank you very much!
> >>>>
> >>>> Best,
> >>>> Rita :)
> >>>>
> >>>> P.S. If my questions are not suitable for this mailing-list, please
> let
> >>>>
> >>> me
> >>>
> >>>> apologize, and then, could you please direct me to other
> mailing-lists?
> >>>> Sorry, and thanks a lot! :)
> >>>>
> >>>>
> >>>>
> >>>
> >>
> >
>

--00151758ae6c756e2a048dd5bd9e--

From general-return-1950-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 15 05:15:20 2010
Return-Path: <general-return-1950-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 71267 invoked from network); 15 Aug 2010 05:15:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Aug 2010 05:15:20 -0000
Received: (qmail 51725 invoked by uid 500); 15 Aug 2010 05:15:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51497 invoked by uid 500); 15 Aug 2010 05:15:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51489 invoked by uid 99); 15 Aug 2010 05:15:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Aug 2010 05:15:15 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of qwertymaniac@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Aug 2010 05:15:10 +0000
Received: by wwb22 with SMTP id 22so5797598wwb.29
        for <general@hadoop.apache.org>; Sat, 14 Aug 2010 22:14:49 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type;
        bh=K3TNgwzY4JL24uKKW073BuYW6dqIe57Sg7TQRHFgsj4=;
        b=CCuEC7Iqf+X7vB/QstZZ8QuP4JQVULIPe+th0Wiic5mJRmFeULJ2sm4fhxywNkK03c
         PD4PIZzlbRFn6c8DLGIR3yNt96qOQz2k0qeHgka96Yw63LJiQCoJpBqh7nv4qQvJh4K3
         z51UKh3q6uX93lUuntAXI2XwG/8tAWryUs+ZI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=mN4AkOXLk8RBxm9nNB+w8gEsmAa3FX9/+Sls+Iy8Fi+7XmU2q4X5ttxCf5xDxcQZKM
         K3qY4pGJ93HXkRacy6oTHzkVy61bYoJ/wRjnN6bJkThAOTq3vcVSvjxXEqjos1DOXCHY
         R9aRYKnBqnlSMbTiVKdUOhHZ4EdCDhD5+0Kwc=
Received: by 10.216.210.206 with SMTP id u56mr1237107weo.23.1281849289124;
 Sat, 14 Aug 2010 22:14:49 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.236.197 with HTTP; Sat, 14 Aug 2010 22:14:29 -0700 (PDT)
In-Reply-To: <AANLkTinoHN1-OaytcwmPeTWPnq95CNDzwHc=P1NOZ90n@mail.gmail.com>
References: <AANLkTimTNY_EU8Eu4ZG3zNi78wOQNCwe0piJYuu_0DQU@mail.gmail.com>
 <4C677028.8000502@gmail.com> <AANLkTi=ys3g3VUg+o=NvxCOWo_exPBa-jwNHyJ0671_F@mail.gmail.com>
 <4C67757B.8050108@gmail.com> <AANLkTinEG-uLPonp8GVw5c_s-ZYhzeo+3wZ2Bv7g2q+G@mail.gmail.com>
 <AANLkTinoHN1-OaytcwmPeTWPnq95CNDzwHc=P1NOZ90n@mail.gmail.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Sun, 15 Aug 2010 10:44:29 +0530
Message-ID: <AANLkTikCF5o8_FPahb4XDK1U+c5eXpRzKaQ1a=tvQ13=@mail.gmail.com>
Subject: Re: Hadoop basics
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

If you start your job in a single-node cluster with a "local"
configuration (conf.set("mapred.job.tracker", "local") and
conf.set("fs.default.name", "local")), you can _almost_ debug all the
vital parts. I use this method (though its been deprecated) to debug
my Map and Reduce functions locally.

Remote debugging with multiple-nodes would still be cool to have though.

On Sun, Aug 15, 2010 at 10:40 AM, Rita Liu <crystaldoll06@gmail.com> wrote:
> Thank you very much, Piyush! I'll do as you say :DD Thanks a lot!!
>
> Thanks Smith :) hmm ... I see. ok :)
>
> Please give me more guidance and suggestions if possible, dear experts!
> -Rita :))
>
> On Sat, Aug 14, 2010 at 10:09 PM, smith jack <thinke365@gmail.com> wrote:
>
>> that means you can only trace by log,
>> and not possible to debug hadoop using step debug, haha
>> distributed system always introduce extra complexity and confusing issues.
>>
>> 2010/8/15 Piyush Garg <piyushgarg80@gmail.com>:
>> > Hi Rita,
>> >
>> > You can put log4j logger debug statements in the code. log4j library is
>> > part of hadoop framework and there is already a log4j.properties file in
>> > hadoop conf directory and all the output logs are saved in hadoop logs
>> > directory.
>> >
>> > Thanks and Regards
>> > Piyush Garg
>> >
>> >
>> > On Sunday 15 August 2010 10:20 AM, Rita Liu wrote:
>> >> Thank you very much, Piyush! :) May I know more about how to use
>> "traces"?
>> >>
>> >> And -- yes, please teach me if possible, experts! :)
>> >>
>> >> Thanks a lot,
>> >> -Rita :))
>> >>
>> >> On Sat, Aug 14, 2010 at 9:42 PM, Piyush Garg <piyushgarg80@gmail.com>
>> wrote:
>> >>
>> >>
>> >>> Hi Rita,
>> >>>
>> >>> I have just started to learn hadoop as well, I know there is a long way
>> >>> to go.
>> >>> I found some useful links which I am sharing with you.
>> >>>
>> >>> Hadoop Tutorial - YDN
>> >>> <http://developer.yahoo.com/hadoop/tutorial/index.html> excellent
>> >>> beginners tutorial and well organized.
>> >>> Running Hadoop On Ubuntu Linux (Single-Node Cluster) - Michael G. Noll
>> >>> <
>> >>>
>> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Single-Node_Cluster%29
>> >>>
>> >>>>
>> >>> Running_Hadoop_On_Ubuntu_Linux_(Multi-Node_Cluster)
>> >>> <
>> >>>
>> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Multi-Node_Cluster%29
>> >>>
>> >>>>
>> >>> The tutorial on the hadoop wiki
>> >>> <http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html> is
>> >>> too much for a beginner.
>> >>>
>> >>> Debugger:
>> >>> I do not think you can easily do debugging using remote debugger. This
>> >>> is natural since hadoop is not sequential programming, it would be very
>> >>> difficult to debug its apps.
>> >>> The only way to debug is to use traces.
>> >>>
>> >>> I think you can learn how to setup multi-node cluster, but for practice
>> >>> session you can use single node setup.
>> >>>
>> >>> Lets see what the experts say.
>> >>>
>> >>> Thanks and Regards
>> >>> Piyush Garg
>> >>>
>> >>>
>> >>> On Sunday 15 August 2010 09:07 AM, Rita Liu wrote:
>> >>>
>> >>>> Hi!
>> >>>>
>> >>>> I am a total beginner, but I am very interested in hadoop. I've
>> already
>> >>>> downloaded hadoop 0.19.2 and run on Ubuntu in single-node mode. Now I
>> >>>>
>> >>> want
>> >>>
>> >>>> to do two things:
>> >>>>
>> >>>> 1. Explore how hadoop works internally with one of the example
>> >>>>
>> >>> applications
>> >>>
>> >>>> hadoop provides
>> >>>> 2. Write an application on my own
>> >>>>
>> >>>> Those two things bring me following questions:
>> >>>>
>> >>>> a. debugger?
>> >>>> I am stuck since I don't know how to "explore" hadoop. I used to trace
>> >>>> through the code using a debugger, but in this case, I don't know if
>> >>>>
>> >>> there
>> >>>
>> >>>> is a good debugger to use; or -- maybe a debugger is not necessary for
>> >>>> hadoop? If not, then how do you trace through the code to either debug
>> or
>> >>>> just gain an understanding about the system? May I know what you,
>> >>>> experienced experts, do? :)
>> >>>>
>> >>>> b. Where to run hadoop?
>> >>>> Also -- may I know where you run your hadoop? Do you run on linux, or
>> on
>> >>>>
>> >>> VM
>> >>>
>> >>>> -- in particular, Cloudera? I heard that Cloudera is good for writing
>> >>>> mapreduce applications with hadoop itself as a blackbox; is it true?
>> If
>> >>>>
>> >>> my
>> >>>
>> >>>> ultimate goal is to understand how hadoop works internally, would it
>> be
>> >>>> better if I directly run it on linux?
>> >>>>
>> >>>> c. Single-node or multi-node?
>> >>>> In the beginning (just like my case :p) would it be better to use
>> >>>> single-node or multi-node? If the latter is true, should I obtain more
>> >>>> machines, or should I use more virtual machines to create more nodes?
>> >>>>
>> >>>> As a newbie, I am sorry for all those basic (and silly, I know :$)
>> >>>> questions. If possible, please help me out? Any suggestion or advice
>> will
>> >>>>
>> >>> be
>> >>>
>> >>>> greatly appreciated. Thank you very much!
>> >>>>
>> >>>> Best,
>> >>>> Rita :)
>> >>>>
>> >>>> P.S. If my questions are not suitable for this mailing-list, please
>> let
>> >>>>
>> >>> me
>> >>>
>> >>>> apologize, and then, could you please direct me to other
>> mailing-lists?
>> >>>> Sorry, and thanks a lot! :)
>> >>>>
>> >>>>
>> >>>>
>> >>>
>> >>
>> >
>>
>



-- 
Harsh J
www.harshj.com

From general-return-1951-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 15 05:23:22 2010
Return-Path: <general-return-1951-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73244 invoked from network); 15 Aug 2010 05:23:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Aug 2010 05:23:22 -0000
Received: (qmail 56056 invoked by uid 500); 15 Aug 2010 05:23:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55852 invoked by uid 500); 15 Aug 2010 05:23:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55844 invoked by uid 99); 15 Aug 2010 05:23:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Aug 2010 05:23:17 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of piyushgarg80@gmail.com designates 74.125.83.176 as permitted sender)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Aug 2010 05:23:08 +0000
Received: by pvg3 with SMTP id 3so1995985pvg.35
        for <general@hadoop.apache.org>; Sat, 14 Aug 2010 22:22:47 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :x-enigmail-version:content-type;
        bh=raY7cP0FYFDk9S2FNCqxO92reORPI6ZYbYiDHdBDhR0=;
        b=cbdW9VLrlZqLUZzOZnXR1qm/KUF1CVq5wDh56o6NdjKuVueZxCepOEEKMCU4j1QyYR
         MzwR7ZTAyFfUbZ+18uLQasH6LxO4MCW6HbdDYbQ0eDXsvlEu/jd7b+DlA15TwY+40rnA
         nNOX6dDxO7qAkz2hqz0K1VStf6oebhdCNHrVk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:x-enigmail-version:content-type;
        b=Xpx5b7NdgNevZMyXpmpxXqsIe1yKgvlTWne3bk2eX0JWkzu89xFgNd2Pf8nX0U7Ecu
         MZSwdVJGutAr2NJfH/H9o6ro823sP2Snm3j71o/j3qpN7srkLJBYAkgJZInYw1d1g62o
         tEA/3JyiEM4rkWRpSO1IxdCF+1QjO+1KZFX2A=
Received: by 10.114.103.6 with SMTP id a6mr4189885wac.213.1281849765918;
        Sat, 14 Aug 2010 22:22:45 -0700 (PDT)
Received: from [192.168.2.3] ([115.99.74.22])
        by mx.google.com with ESMTPS id d35sm8830884waa.21.2010.08.14.22.22.36
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Sat, 14 Aug 2010 22:22:44 -0700 (PDT)
Message-ID: <4C67798F.4010701@gmail.com>
Date: Sun, 15 Aug 2010 10:52:23 +0530
From: Piyush Garg <piyushgarg80@gmail.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.11) Gecko/20100713 Lightning/1.0b1 Thunderbird/3.0.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Hadoop basics
References: <AANLkTimTNY_EU8Eu4ZG3zNi78wOQNCwe0piJYuu_0DQU@mail.gmail.com>	<4C677028.8000502@gmail.com>	<AANLkTi=ys3g3VUg+o=NvxCOWo_exPBa-jwNHyJ0671_F@mail.gmail.com>	<4C67757B.8050108@gmail.com> <AANLkTinEG-uLPonp8GVw5c_s-ZYhzeo+3wZ2Bv7g2q+G@mail.gmail.com>
In-Reply-To: <AANLkTinEG-uLPonp8GVw5c_s-ZYhzeo+3wZ2Bv7g2q+G@mail.gmail.com>
X-Enigmail-Version: 1.0.1
Content-Type: multipart/alternative;
 boundary="------------070201060301050706070908"
X-Virus-Checked: Checked by ClamAV on apache.org

--------------070201060301050706070908
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi Smith,

step debugging also works in hadoop as with other java applications.
export
HADOOP_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=8000"
'suspend=y' is to let the jvm suspend until the remote debugger is attached.

Thanks and Regards
Piyush Garg


On Sunday 15 August 2010 10:39 AM, smith jack wrote:
> that means you can only trace by log,
> and not possible to debug hadoop using step debug, haha
> distributed system always introduce extra complexity and confusing issues.
>
> 2010/8/15 Piyush Garg <piyushgarg80@gmail.com>:
>   
>> Hi Rita,
>>
>> You can put log4j logger debug statements in the code. log4j library is
>> part of hadoop framework and there is already a log4j.properties file in
>> hadoop conf directory and all the output logs are saved in hadoop logs
>> directory.
>>
>> Thanks and Regards
>> Piyush Garg
>>
>>
>> On Sunday 15 August 2010 10:20 AM, Rita Liu wrote:
>>     
>>> Thank you very much, Piyush! :) May I know more about how to use "traces"?
>>>
>>> And -- yes, please teach me if possible, experts! :)
>>>
>>> Thanks a lot,
>>> -Rita :))
>>>
>>> On Sat, Aug 14, 2010 at 9:42 PM, Piyush Garg <piyushgarg80@gmail.com> wrote:
>>>
>>>
>>>       
>>>> Hi Rita,
>>>>
>>>> I have just started to learn hadoop as well, I know there is a long way
>>>> to go.
>>>> I found some useful links which I am sharing with you.
>>>>
>>>> Hadoop Tutorial - YDN
>>>> <http://developer.yahoo.com/hadoop/tutorial/index.html> excellent
>>>> beginners tutorial and well organized.
>>>> Running Hadoop On Ubuntu Linux (Single-Node Cluster) - Michael G. Noll
>>>> <
>>>> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Single-Node_Cluster%29
>>>>
>>>>         
>>>>>           
>>>> Running_Hadoop_On_Ubuntu_Linux_(Multi-Node_Cluster)
>>>> <
>>>> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Multi-Node_Cluster%29
>>>>
>>>>         
>>>>>           
>>>> The tutorial on the hadoop wiki
>>>> <http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html> is
>>>> too much for a beginner.
>>>>
>>>> Debugger:
>>>> I do not think you can easily do debugging using remote debugger. This
>>>> is natural since hadoop is not sequential programming, it would be very
>>>> difficult to debug its apps.
>>>> The only way to debug is to use traces.
>>>>
>>>> I think you can learn how to setup multi-node cluster, but for practice
>>>> session you can use single node setup.
>>>>
>>>> Lets see what the experts say.
>>>>
>>>> Thanks and Regards
>>>> Piyush Garg
>>>>
>>>>
>>>> On Sunday 15 August 2010 09:07 AM, Rita Liu wrote:
>>>>
>>>>         
>>>>> Hi!
>>>>>
>>>>> I am a total beginner, but I am very interested in hadoop. I've already
>>>>> downloaded hadoop 0.19.2 and run on Ubuntu in single-node mode. Now I
>>>>>
>>>>>           
>>>> want
>>>>
>>>>         
>>>>> to do two things:
>>>>>
>>>>> 1. Explore how hadoop works internally with one of the example
>>>>>
>>>>>           
>>>> applications
>>>>
>>>>         
>>>>> hadoop provides
>>>>> 2. Write an application on my own
>>>>>
>>>>> Those two things bring me following questions:
>>>>>
>>>>> a. debugger?
>>>>> I am stuck since I don't know how to "explore" hadoop. I used to trace
>>>>> through the code using a debugger, but in this case, I don't know if
>>>>>
>>>>>           
>>>> there
>>>>
>>>>         
>>>>> is a good debugger to use; or -- maybe a debugger is not necessary for
>>>>> hadoop? If not, then how do you trace through the code to either debug or
>>>>> just gain an understanding about the system? May I know what you,
>>>>> experienced experts, do? :)
>>>>>
>>>>> b. Where to run hadoop?
>>>>> Also -- may I know where you run your hadoop? Do you run on linux, or on
>>>>>
>>>>>           
>>>> VM
>>>>
>>>>         
>>>>> -- in particular, Cloudera? I heard that Cloudera is good for writing
>>>>> mapreduce applications with hadoop itself as a blackbox; is it true? If
>>>>>
>>>>>           
>>>> my
>>>>
>>>>         
>>>>> ultimate goal is to understand how hadoop works internally, would it be
>>>>> better if I directly run it on linux?
>>>>>
>>>>> c. Single-node or multi-node?
>>>>> In the beginning (just like my case :p) would it be better to use
>>>>> single-node or multi-node? If the latter is true, should I obtain more
>>>>> machines, or should I use more virtual machines to create more nodes?
>>>>>
>>>>> As a newbie, I am sorry for all those basic (and silly, I know :$)
>>>>> questions. If possible, please help me out? Any suggestion or advice will
>>>>>
>>>>>           
>>>> be
>>>>
>>>>         
>>>>> greatly appreciated. Thank you very much!
>>>>>
>>>>> Best,
>>>>> Rita :)
>>>>>
>>>>> P.S. If my questions are not suitable for this mailing-list, please let
>>>>>
>>>>>           
>>>> me
>>>>
>>>>         
>>>>> apologize, and then, could you please direct me to other mailing-lists?
>>>>> Sorry, and thanks a lot! :)
>>>>>
>>>>>
>>>>>
>>>>>           
>>>>         
>>>       
>>     

--------------070201060301050706070908--

From general-return-1952-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 15 05:40:38 2010
Return-Path: <general-return-1952-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 75542 invoked from network); 15 Aug 2010 05:40:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Aug 2010 05:40:38 -0000
Received: (qmail 58547 invoked by uid 500); 15 Aug 2010 05:40:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58236 invoked by uid 500); 15 Aug 2010 05:40:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58228 invoked by uid 99); 15 Aug 2010 05:40:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Aug 2010 05:40:33 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of crystaldoll06@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Aug 2010 05:40:26 +0000
Received: by wwb22 with SMTP id 22so5815018wwb.29
        for <general@hadoop.apache.org>; Sat, 14 Aug 2010 22:40:06 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=3xtfYNN5s+KwRTx8qbTVNgScMjJgIx5738n3HljTBQU=;
        b=VgYmp9n5d2LR1KGIBHGm0K0ZitmYq18M9/V03LAjn4gnMFfZhjNuRJHSF4qxNpGGfE
         kFzueAjqVWwqjzglYWqODtFU9qB80b/LX3JdSadIvxgU2HtE18KZall/rgiSNesXMpSI
         a6asO2qixNnrFj2cunu6r+7Q0Icm5svo1eY98=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=PQfe5zMDtoBIF+tPxCHQGybjgAsAYhj+oqIE2W/wlbMvxGOxBVNC5pNsMyti14btd9
         n2vGMmRqSv9YBfhwBcDQc6N0dI3H9bE1DK7Djmq+HSmI1qWgYf8GhmVEL0gEc7kQ342F
         99FjKdl1DKwRAHMfkIWbciSSRF1u+vxRTXiMw=
MIME-Version: 1.0
Received: by 10.216.188.211 with SMTP id a61mr1249503wen.15.1281850805920;
 Sat, 14 Aug 2010 22:40:05 -0700 (PDT)
Received: by 10.216.237.90 with HTTP; Sat, 14 Aug 2010 22:40:05 -0700 (PDT)
In-Reply-To: <4C67798F.4010701@gmail.com>
References: <AANLkTimTNY_EU8Eu4ZG3zNi78wOQNCwe0piJYuu_0DQU@mail.gmail.com>
	<4C677028.8000502@gmail.com>
	<AANLkTi=ys3g3VUg+o=NvxCOWo_exPBa-jwNHyJ0671_F@mail.gmail.com>
	<4C67757B.8050108@gmail.com>
	<AANLkTinEG-uLPonp8GVw5c_s-ZYhzeo+3wZ2Bv7g2q+G@mail.gmail.com>
	<4C67798F.4010701@gmail.com>
Date: Sat, 14 Aug 2010 22:40:05 -0700
Message-ID: <AANLkTimgrbKdyPPPnDyrcsPH5taioVV9yyqDhHK_2Ors@mail.gmail.com>
Subject: Re: Hadoop basics
From: Rita Liu <crystaldoll06@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64c26a4f12373048dd62522
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e64c26a4f12373048dd62522
Content-Type: text/plain; charset=ISO-8859-1

Hi Harsh and Piyush! Thank you very much. So it seems like it would be best
if I use log4j to trace, and debugging with a debugger is still possible if
I set "mapred.job.tracker" to be "local" and "fs.default.name" to be
"local", in hadoop-site.xml. Plus: in hadoop-env.sh, I should specify
HADOOP_OPTS to be:

"-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=8000" (why
8000? also, what does "-agentlib:jdwp=transport=dt_socket" mean?)

... in order to use a debugger. Is my understanding correct? :)

If so -- then which debugger do you use? May I know? Thanks a lot! I am also
going to try log4j now!

Many thanks,
-Rita :))

On Sat, Aug 14, 2010 at 10:22 PM, Piyush Garg <piyushgarg80@gmail.com>wrote:

> Hi Smith,
>
> step debugging also works in hadoop as with other java applications.
> export
>
> HADOOP_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=8000"
> 'suspend=y' is to let the jvm suspend until the remote debugger is
> attached.
>
> Thanks and Regards
> Piyush Garg
>
>
> On Sunday 15 August 2010 10:39 AM, smith jack wrote:
> > that means you can only trace by log,
> > and not possible to debug hadoop using step debug, haha
> > distributed system always introduce extra complexity and confusing
> issues.
> >
> > 2010/8/15 Piyush Garg <piyushgarg80@gmail.com>:
> >
> >> Hi Rita,
> >>
> >> You can put log4j logger debug statements in the code. log4j library is
> >> part of hadoop framework and there is already a log4j.properties file in
> >> hadoop conf directory and all the output logs are saved in hadoop logs
> >> directory.
> >>
> >> Thanks and Regards
> >> Piyush Garg
> >>
> >>
> >> On Sunday 15 August 2010 10:20 AM, Rita Liu wrote:
> >>
> >>> Thank you very much, Piyush! :) May I know more about how to use
> "traces"?
> >>>
> >>> And -- yes, please teach me if possible, experts! :)
> >>>
> >>> Thanks a lot,
> >>> -Rita :))
> >>>
> >>> On Sat, Aug 14, 2010 at 9:42 PM, Piyush Garg <piyushgarg80@gmail.com>
> wrote:
> >>>
> >>>
> >>>
> >>>> Hi Rita,
> >>>>
> >>>> I have just started to learn hadoop as well, I know there is a long
> way
> >>>> to go.
> >>>> I found some useful links which I am sharing with you.
> >>>>
> >>>> Hadoop Tutorial - YDN
> >>>> <http://developer.yahoo.com/hadoop/tutorial/index.html> excellent
> >>>> beginners tutorial and well organized.
> >>>> Running Hadoop On Ubuntu Linux (Single-Node Cluster) - Michael G. Noll
> >>>> <
> >>>>
> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Single-Node_Cluster%29
> >>>>
> >>>>
> >>>>>
> >>>> Running_Hadoop_On_Ubuntu_Linux_(Multi-Node_Cluster)
> >>>> <
> >>>>
> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Multi-Node_Cluster%29
> >>>>
> >>>>
> >>>>>
> >>>> The tutorial on the hadoop wiki
> >>>> <http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html>
> is
> >>>> too much for a beginner.
> >>>>
> >>>> Debugger:
> >>>> I do not think you can easily do debugging using remote debugger. This
> >>>> is natural since hadoop is not sequential programming, it would be
> very
> >>>> difficult to debug its apps.
> >>>> The only way to debug is to use traces.
> >>>>
> >>>> I think you can learn how to setup multi-node cluster, but for
> practice
> >>>> session you can use single node setup.
> >>>>
> >>>> Lets see what the experts say.
> >>>>
> >>>> Thanks and Regards
> >>>> Piyush Garg
> >>>>
> >>>>
> >>>> On Sunday 15 August 2010 09:07 AM, Rita Liu wrote:
> >>>>
> >>>>
> >>>>> Hi!
> >>>>>
> >>>>> I am a total beginner, but I am very interested in hadoop. I've
> already
> >>>>> downloaded hadoop 0.19.2 and run on Ubuntu in single-node mode. Now I
> >>>>>
> >>>>>
> >>>> want
> >>>>
> >>>>
> >>>>> to do two things:
> >>>>>
> >>>>> 1. Explore how hadoop works internally with one of the example
> >>>>>
> >>>>>
> >>>> applications
> >>>>
> >>>>
> >>>>> hadoop provides
> >>>>> 2. Write an application on my own
> >>>>>
> >>>>> Those two things bring me following questions:
> >>>>>
> >>>>> a. debugger?
> >>>>> I am stuck since I don't know how to "explore" hadoop. I used to
> trace
> >>>>> through the code using a debugger, but in this case, I don't know if
> >>>>>
> >>>>>
> >>>> there
> >>>>
> >>>>
> >>>>> is a good debugger to use; or -- maybe a debugger is not necessary
> for
> >>>>> hadoop? If not, then how do you trace through the code to either
> debug or
> >>>>> just gain an understanding about the system? May I know what you,
> >>>>> experienced experts, do? :)
> >>>>>
> >>>>> b. Where to run hadoop?
> >>>>> Also -- may I know where you run your hadoop? Do you run on linux, or
> on
> >>>>>
> >>>>>
> >>>> VM
> >>>>
> >>>>
> >>>>> -- in particular, Cloudera? I heard that Cloudera is good for writing
> >>>>> mapreduce applications with hadoop itself as a blackbox; is it true?
> If
> >>>>>
> >>>>>
> >>>> my
> >>>>
> >>>>
> >>>>> ultimate goal is to understand how hadoop works internally, would it
> be
> >>>>> better if I directly run it on linux?
> >>>>>
> >>>>> c. Single-node or multi-node?
> >>>>> In the beginning (just like my case :p) would it be better to use
> >>>>> single-node or multi-node? If the latter is true, should I obtain
> more
> >>>>> machines, or should I use more virtual machines to create more nodes?
> >>>>>
> >>>>> As a newbie, I am sorry for all those basic (and silly, I know :$)
> >>>>> questions. If possible, please help me out? Any suggestion or advice
> will
> >>>>>
> >>>>>
> >>>> be
> >>>>
> >>>>
> >>>>> greatly appreciated. Thank you very much!
> >>>>>
> >>>>> Best,
> >>>>> Rita :)
> >>>>>
> >>>>> P.S. If my questions are not suitable for this mailing-list, please
> let
> >>>>>
> >>>>>
> >>>> me
> >>>>
> >>>>
> >>>>> apologize, and then, could you please direct me to other
> mailing-lists?
> >>>>> Sorry, and thanks a lot! :)
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
>

--0016e64c26a4f12373048dd62522--

From general-return-1953-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 06:28:24 2010
Return-Path: <general-return-1953-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91275 invoked from network); 16 Aug 2010 06:28:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 06:28:24 -0000
Received: (qmail 80524 invoked by uid 500); 16 Aug 2010 06:28:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80163 invoked by uid 500); 16 Aug 2010 06:28:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80155 invoked by uid 99); 16 Aug 2010 06:28:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 06:28:18 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of v.amit@verchaska.com designates 203.123.141.155 as permitted sender)
Received: from [203.123.141.155] (HELO mail.verchaska.com) (203.123.141.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 06:28:10 +0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by mail.verchaska.com (Postfix) with ESMTP id 3DE3B11DBBD7
	for <general@hadoop.apache.org>; Mon, 16 Aug 2010 11:56:24 +0530 (IST)
X-Virus-Scanned: amavisd-new at mail.verchaska.com
Received: from mail.verchaska.com ([127.0.0.1])
	by localhost (mail.verchaska.com [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id 9cnoWMjAGLN4 for <general@hadoop.apache.org>;
	Mon, 16 Aug 2010 11:56:19 +0530 (IST)
Received: from [192.168.0.186] (unknown [192.168.0.186])
	by mail.verchaska.com (Postfix) with ESMTP id BB74D11DBBD3
	for <general@hadoop.apache.org>; Mon, 16 Aug 2010 11:56:19 +0530 (IST)
Message-ID: <4C68DA5E.8080504@verchaska.com>
Date: Mon, 16 Aug 2010 11:57:42 +0530
From: amit kumar verma <v.amit@verchaska.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.2.8) Gecko/20100802 Thunderbird/3.1.2
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Hadoop basics
References: <AANLkTimTNY_EU8Eu4ZG3zNi78wOQNCwe0piJYuu_0DQU@mail.gmail.com>	<4C677028.8000502@gmail.com>	<AANLkTi=ys3g3VUg+o=NvxCOWo_exPBa-jwNHyJ0671_F@mail.gmail.com>	<4C67757B.8050108@gmail.com>	<AANLkTinEG-uLPonp8GVw5c_s-ZYhzeo+3wZ2Bv7g2q+G@mail.gmail.com>	<4C67798F.4010701@gmail.com> <AANLkTimgrbKdyPPPnDyrcsPH5taioVV9yyqDhHK_2Ors@mail.gmail.com>
In-Reply-To: <AANLkTimgrbKdyPPPnDyrcsPH5taioVV9yyqDhHK_2Ors@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

  Hi Rita,

If you reached a place where you need to use api like hahoop, forget 
about the debugging the code. Your code must be syntactically and 
logically error free, for rest of the things logging is enough. Try 
log4j only.

Thanks,
Amit Kumar Verma
Verchaska Infotech Pvt. Ltd.



On 08/15/2010 11:10 AM, Rita Liu wrote:
> Hi Harsh and Piyush! Thank you very much. So it seems like it would be best
> if I use log4j to trace, and debugging with a debugger is still possible if
> I set "mapred.job.tracker" to be "local" and "fs.default.name" to be
> "local", in hadoop-site.xml. Plus: in hadoop-env.sh, I should specify
> HADOOP_OPTS to be:
>
> "-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=8000" (why
> 8000? also, what does "-agentlib:jdwp=transport=dt_socket" mean?)
>
> ... in order to use a debugger. Is my understanding correct? :)
>
> If so -- then which debugger do you use? May I know? Thanks a lot! I am also
> going to try log4j now!
>
> Many thanks,
> -Rita :))
>
> On Sat, Aug 14, 2010 at 10:22 PM, Piyush Garg<piyushgarg80@gmail.com>wrote:
>
>> Hi Smith,
>>
>> step debugging also works in hadoop as with other java applications.
>> export
>>
>> HADOOP_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=8000"
>> 'suspend=y' is to let the jvm suspend until the remote debugger is
>> attached.
>>
>> Thanks and Regards
>> Piyush Garg
>>
>>
>> On Sunday 15 August 2010 10:39 AM, smith jack wrote:
>>> that means you can only trace by log,
>>> and not possible to debug hadoop using step debug, haha
>>> distributed system always introduce extra complexity and confusing
>> issues.
>>> 2010/8/15 Piyush Garg<piyushgarg80@gmail.com>:
>>>
>>>> Hi Rita,
>>>>
>>>> You can put log4j logger debug statements in the code. log4j library is
>>>> part of hadoop framework and there is already a log4j.properties file in
>>>> hadoop conf directory and all the output logs are saved in hadoop logs
>>>> directory.
>>>>
>>>> Thanks and Regards
>>>> Piyush Garg
>>>>
>>>>
>>>> On Sunday 15 August 2010 10:20 AM, Rita Liu wrote:
>>>>
>>>>> Thank you very much, Piyush! :) May I know more about how to use
>> "traces"?
>>>>> And -- yes, please teach me if possible, experts! :)
>>>>>
>>>>> Thanks a lot,
>>>>> -Rita :))
>>>>>
>>>>> On Sat, Aug 14, 2010 at 9:42 PM, Piyush Garg<piyushgarg80@gmail.com>
>> wrote:
>>>>>
>>>>>
>>>>>> Hi Rita,
>>>>>>
>>>>>> I have just started to learn hadoop as well, I know there is a long
>> way
>>>>>> to go.
>>>>>> I found some useful links which I am sharing with you.
>>>>>>
>>>>>> Hadoop Tutorial - YDN
>>>>>> <http://developer.yahoo.com/hadoop/tutorial/index.html>  excellent
>>>>>> beginners tutorial and well organized.
>>>>>> Running Hadoop On Ubuntu Linux (Single-Node Cluster) - Michael G. Noll
>>>>>> <
>>>>>>
>> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Single-Node_Cluster%29
>>>>>>
>>>>>> Running_Hadoop_On_Ubuntu_Linux_(Multi-Node_Cluster)
>>>>>> <
>>>>>>
>> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Multi-Node_Cluster%29
>>>>>>
>>>>>> The tutorial on the hadoop wiki
>>>>>> <http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html>
>> is
>>>>>> too much for a beginner.
>>>>>>
>>>>>> Debugger:
>>>>>> I do not think you can easily do debugging using remote debugger. This
>>>>>> is natural since hadoop is not sequential programming, it would be
>> very
>>>>>> difficult to debug its apps.
>>>>>> The only way to debug is to use traces.
>>>>>>
>>>>>> I think you can learn how to setup multi-node cluster, but for
>> practice
>>>>>> session you can use single node setup.
>>>>>>
>>>>>> Lets see what the experts say.
>>>>>>
>>>>>> Thanks and Regards
>>>>>> Piyush Garg
>>>>>>
>>>>>>
>>>>>> On Sunday 15 August 2010 09:07 AM, Rita Liu wrote:
>>>>>>
>>>>>>
>>>>>>> Hi!
>>>>>>>
>>>>>>> I am a total beginner, but I am very interested in hadoop. I've
>> already
>>>>>>> downloaded hadoop 0.19.2 and run on Ubuntu in single-node mode. Now I
>>>>>>>
>>>>>>>
>>>>>> want
>>>>>>
>>>>>>
>>>>>>> to do two things:
>>>>>>>
>>>>>>> 1. Explore how hadoop works internally with one of the example
>>>>>>>
>>>>>>>
>>>>>> applications
>>>>>>
>>>>>>
>>>>>>> hadoop provides
>>>>>>> 2. Write an application on my own
>>>>>>>
>>>>>>> Those two things bring me following questions:
>>>>>>>
>>>>>>> a. debugger?
>>>>>>> I am stuck since I don't know how to "explore" hadoop. I used to
>> trace
>>>>>>> through the code using a debugger, but in this case, I don't know if
>>>>>>>
>>>>>>>
>>>>>> there
>>>>>>
>>>>>>
>>>>>>> is a good debugger to use; or -- maybe a debugger is not necessary
>> for
>>>>>>> hadoop? If not, then how do you trace through the code to either
>> debug or
>>>>>>> just gain an understanding about the system? May I know what you,
>>>>>>> experienced experts, do? :)
>>>>>>>
>>>>>>> b. Where to run hadoop?
>>>>>>> Also -- may I know where you run your hadoop? Do you run on linux, or
>> on
>>>>>>>
>>>>>> VM
>>>>>>
>>>>>>
>>>>>>> -- in particular, Cloudera? I heard that Cloudera is good for writing
>>>>>>> mapreduce applications with hadoop itself as a blackbox; is it true?
>> If
>>>>>>>
>>>>>> my
>>>>>>
>>>>>>
>>>>>>> ultimate goal is to understand how hadoop works internally, would it
>> be
>>>>>>> better if I directly run it on linux?
>>>>>>>
>>>>>>> c. Single-node or multi-node?
>>>>>>> In the beginning (just like my case :p) would it be better to use
>>>>>>> single-node or multi-node? If the latter is true, should I obtain
>> more
>>>>>>> machines, or should I use more virtual machines to create more nodes?
>>>>>>>
>>>>>>> As a newbie, I am sorry for all those basic (and silly, I know :$)
>>>>>>> questions. If possible, please help me out? Any suggestion or advice
>> will
>>>>>>>
>>>>>> be
>>>>>>
>>>>>>
>>>>>>> greatly appreciated. Thank you very much!
>>>>>>>
>>>>>>> Best,
>>>>>>> Rita :)
>>>>>>>
>>>>>>> P.S. If my questions are not suitable for this mailing-list, please
>> let
>>>>>>>
>>>>>> me
>>>>>>
>>>>>>
>>>>>>> apologize, and then, could you please direct me to other
>> mailing-lists?
>>>>>>> Sorry, and thanks a lot! :)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>

From general-return-1954-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 09:57:23 2010
Return-Path: <general-return-1954-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69456 invoked from network); 16 Aug 2010 09:57:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 09:57:22 -0000
Received: (qmail 86995 invoked by uid 500); 16 Aug 2010 09:57:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86563 invoked by uid 500); 16 Aug 2010 09:57:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86536 invoked by uid 99); 16 Aug 2010 09:57:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 09:57:17 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 09:57:04 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 34C81B7CEC
	for <general@hadoop.apache.org>; Mon, 16 Aug 2010 10:56:44 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id mSau-uj5F9zT for <general@hadoop.apache.org>;
	Mon, 16 Aug 2010 10:56:42 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id A9F4BB7CEB
	for <general@hadoop.apache.org>; Mon, 16 Aug 2010 10:56:42 +0100 (BST)
MailScanner-NULL-Check: 1282557390.13827@aYcqjV5Y4bvFrfI3b9dp4w
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o7G9uSoC026108
	for <general@hadoop.apache.org>; Mon, 16 Aug 2010 10:56:28 +0100 (BST)
Message-ID: <4C690B4C.4070906@apache.org>
Date: Mon, 16 Aug 2010 10:56:28 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.11) Gecko/20100713 Thunderbird/3.0.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSSION] Combine committer lists in Common/HDFS/MapReduce
References: <AANLkTi=GyFLivH8uOQNppUJnVeixpY5HA98=0cJEpaNa@mail.gmail.com>	<4C5C8302.3010008@apache.org>	<97E2C026-0C44-4FDC-A569-63E218E03D59@yahoo-inc.com>	<4C602C27.6060802@apache.org>	<4C609FEE.1050209@yahoo-inc.com> <AANLkTi=_EYrdpZSzAZoDzHwX18gTwifDU6DZ3hhtOdv6@mail.gmail.com>
In-Reply-To: <AANLkTi=_EYrdpZSzAZoDzHwX18gTwifDU6DZ3hhtOdv6@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o7G9uSoC026108
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 10/08/10 17:44, Jeff Hammerbacher wrote:
>>> I'd like to get rid of Common.  It could either be merged into HDFS or
>>> gradually whittled away to nothing.  I'd prefer the latter.  If we move
>>> to different RPC and serialization systems (e.g., Avro) then Common's
>>> io, and ipc packages might be removed.  Configuration might be
>>> replaced/merged with Jakarta Commons Configuration
>>> (http://commons.apache.org/configuration/).  Similarly, the metrics and
>>> fs packages might be moved to Jakarta Commons.  Such changes might be
>>> hard to do back-compatibly, however.
>>
>> Great idea. We should do this.
>>
>
> Cool, created https://issues.apache.org/jira/browse/HADOOP-6909 to track.
>

I'm actually against this. You push all your stuff upstream -your 
management foundations- and you are dependent on the other stuff release 
schedules, and you don't have one single place to do your own layers, 
your own indirection.

This doesn't imply anything against commons-config, just that there is 
benefit from having some core stuff, and it's in management that it 
really wins, and in shared utility stuff like in-VM httpd servers

From general-return-1955-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 14:28:29 2010
Return-Path: <general-return-1955-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 13171 invoked from network); 16 Aug 2010 14:28:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 14:28:28 -0000
Received: (qmail 63019 invoked by uid 500); 16 Aug 2010 14:28:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62648 invoked by uid 500); 16 Aug 2010 14:28:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62639 invoked by uid 99); 16 Aug 2010 14:28:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 14:28:23 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 14:28:16 +0000
Received: from [192.168.1.71] ([10.72.244.128])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7GERPN2057711
	for <general@hadoop.apache.org>; Mon, 16 Aug 2010 07:27:25 -0700 (PDT)
Message-Id: <2958FF70-4BBC-47E4-BFB7-656CA3594637@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <4C690B4C.4070906@apache.org>
Content-Type: multipart/alternative; boundary=Apple-Mail-104-866451547
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSSION] Combine committer lists in Common/HDFS/MapReduce
Date: Mon, 16 Aug 2010 07:27:25 -0700
References: <AANLkTi=GyFLivH8uOQNppUJnVeixpY5HA98=0cJEpaNa@mail.gmail.com> <4C5C8302.3010008@apache.org> <97E2C026-0C44-4FDC-A569-63E218E03D59@yahoo-inc.com> <4C602C27.6060802@apache.org>	<4C609FEE.1050209@yahoo-inc.com> <AANLkTi=_EYrdpZSzAZoDzHwX18gTwifDU6DZ3hhtOdv6@mail.gmail.com> <4C690B4C.4070906@apache.org>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-104-866451547
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


On Aug 16, 2010, at 2:56 AM, Steve Loughran wrote:

> On 10/08/10 17:44, Jeff Hammerbacher wrote:
>>
>> Cool, created https://issues.apache.org/jira/browse/HADOOP-6909 to  
>> track.
>>
>
> I'm actually against this. You push all your stuff upstream -your
> management foundations- and you are dependent on the other stuff  
> release
> schedules, and you don't have one single place to do your own layers,
> your own indirection.
>
> This doesn't imply anything against commons-config, just that there is
> benefit from having some core stuff, and it's in management that it
> really wins, and in shared utility stuff like in-VM httpd servers

Agreed. Please comment on HADOOP-6909/HADOOP-6910, let us continue the  
discussion on jira.

There are several other reasons this is infeasible including features  
such as support for 'final' parameters, lack of serialization etc.

Arun
--Apple-Mail-104-866451547--

From general-return-1956-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 21:39:36 2010
Return-Path: <general-return-1956-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67677 invoked from network); 16 Aug 2010 21:39:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 21:39:36 -0000
Received: (qmail 28945 invoked by uid 500); 16 Aug 2010 21:39:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28847 invoked by uid 500); 16 Aug 2010 21:39:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28837 invoked by uid 99); 16 Aug 2010 21:39:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 21:39:34 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.27.212] (HELO qmta14.emeryville.ca.mail.comcast.net) (76.96.27.212)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 21:39:24 +0000
Received: from omta13.emeryville.ca.mail.comcast.net ([76.96.30.52])
	by qmta14.emeryville.ca.mail.comcast.net with comcast
	id v8Uf1e00817UAYkAE9f2AP; Mon, 16 Aug 2010 21:39:02 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta13.emeryville.ca.mail.comcast.net with comcast
	id v9et1e0072SbwD58Z9evt7; Mon, 16 Aug 2010 21:39:00 +0000
Message-Id: <7962BB4E-7176-44EB-A708-64881639E2AF@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: [VOTE] Should we define the Common committers as HDFS + MapReduce committers?
Date: Mon, 16 Aug 2010 14:38:53 -0700
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

Since the project split, it is mostly required that any MapReduce or  
HDFS committers have access to change Common. We haven't had any cases  
where we wanted to nominate any one for just Common. Toward the goal  
of simplifying the structure, I'd propose that we define the Common  
committers as precisely the union of the HDFS committers and MapReduce  
committers.

Clearly, I'm +1.

-- Owen

From general-return-1957-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 21:44:28 2010
Return-Path: <general-return-1957-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68473 invoked from network); 16 Aug 2010 21:44:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 21:44:28 -0000
Received: (qmail 36382 invoked by uid 500); 16 Aug 2010 21:44:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36172 invoked by uid 500); 16 Aug 2010 21:44:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36162 invoked by uid 99); 16 Aug 2010 21:44:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 21:44:27 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 21:44:19 +0000
Received: from wlanvpn-snve-247-101.corp.yahoo.com ([172.21.149.101])
	by mrout1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7GLhgT9003176
	for <general@hadoop.apache.org>; Mon, 16 Aug 2010 14:43:42 -0700 (PDT)
Message-Id: <FE89EB5B-AA88-489C-887A-B8E3048DE415@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <7962BB4E-7176-44EB-A708-64881639E2AF@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Should we define the Common committers as HDFS + MapReduce committers?
Date: Mon, 16 Aug 2010 14:43:42 -0700
References: <7962BB4E-7176-44EB-A708-64881639E2AF@apache.org>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Arun

On Aug 16, 2010, at 2:38 PM, Owen O'Malley wrote:

> Since the project split, it is mostly required that any MapReduce or
> HDFS committers have access to change Common. We haven't had any cases
> where we wanted to nominate any one for just Common. Toward the goal
> of simplifying the structure, I'd propose that we define the Common
> committers as precisely the union of the HDFS committers and MapReduce
> committers.
>
> Clearly, I'm +1.
>
> -- Owen


From general-return-1958-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 21:59:45 2010
Return-Path: <general-return-1958-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 71414 invoked from network); 16 Aug 2010 21:59:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 21:59:45 -0000
Received: (qmail 52407 invoked by uid 500); 16 Aug 2010 21:59:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52366 invoked by uid 500); 16 Aug 2010 21:59:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52358 invoked by uid 99); 16 Aug 2010 21:59:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 21:59:43 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 21:59:37 +0000
Received: from [172.21.148.33] (wlanvpn-snve-246-33.corp.yahoo.com [172.21.148.33])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7GLwvfp005681
	for <general@hadoop.apache.org>; Mon, 16 Aug 2010 14:58:57 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=yvcVNMBnKBkH3rwSqbYX+KB6NdyCsKtDBLhYhGm3G68BbYgEWuUu1TqIjceCSMYW
Message-ID: <4C69B4A1.7060303@yahoo-inc.com>
Date: Mon, 16 Aug 2010 14:58:57 -0700
From: Jakob Homan <jhoman@yahoo-inc.com>
User-Agent: Thunderbird 2.0.0.12 (Macintosh/20080213)
MIME-Version: 1.0
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: [VOTE] Should we define the Common committers as HDFS + MapReduce
 committers?
References: <7962BB4E-7176-44EB-A708-64881639E2AF@apache.org> <FE89EB5B-AA88-489C-887A-B8E3048DE415@yahoo-inc.com>
In-Reply-To: <FE89EB5B-AA88-489C-887A-B8E3048DE415@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

+1
Jakob
Arun C Murthy wrote:
> +1
> 
> Arun
> 
> On Aug 16, 2010, at 2:38 PM, Owen O'Malley wrote:
> 
>> Since the project split, it is mostly required that any MapReduce or
>> HDFS committers have access to change Common. We haven't had any cases
>> where we wanted to nominate any one for just Common. Toward the goal
>> of simplifying the structure, I'd propose that we define the Common
>> committers as precisely the union of the HDFS committers and MapReduce
>> committers.
>>
>> Clearly, I'm +1.
>>
>> -- Owen
> 


From general-return-1959-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 22:37:43 2010
Return-Path: <general-return-1959-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88485 invoked from network); 16 Aug 2010 22:37:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 22:37:42 -0000
Received: (qmail 84563 invoked by uid 500); 16 Aug 2010 22:37:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84523 invoked by uid 500); 16 Aug 2010 22:37:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84515 invoked by uid 99); 16 Aug 2010 22:37:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 22:37:40 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 16 Aug 2010 22:37:40 +0000
Received: (qmail 88440 invoked by uid 99); 16 Aug 2010 22:37:19 -0000
Received: from localhost.apache.org (HELO [192.168.168.104]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 22:37:19 +0000
Message-ID: <4C69BD9E.9030802@apache.org>
Date: Mon, 16 Aug 2010 15:37:18 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.11) Gecko/20100713 Thunderbird/3.0.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Should we define the Common committers as HDFS + MapReduce
 committers?
References: <7962BB4E-7176-44EB-A708-64881639E2AF@apache.org>
In-Reply-To: <7962BB4E-7176-44EB-A708-64881639E2AF@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

+1 I believe that so long as we have synchronized releases then we're 
co-developing a single product and should be a single community.

Doug

On 08/16/2010 02:38 PM, Owen O'Malley wrote:
> Since the project split, it is mostly required that any MapReduce or
> HDFS committers have access to change Common. We haven't had any cases
> where we wanted to nominate any one for just Common. Toward the goal of
> simplifying the structure, I'd propose that we define the Common
> committers as precisely the union of the HDFS committers and MapReduce
> committers.
>
> Clearly, I'm +1.
>
> -- Owen

From general-return-1960-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 23:51:45 2010
Return-Path: <general-return-1960-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24482 invoked from network); 16 Aug 2010 23:51:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 23:51:44 -0000
Received: (qmail 51196 invoked by uid 500); 16 Aug 2010 23:51:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51071 invoked by uid 500); 16 Aug 2010 23:51:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51063 invoked by uid 99); 16 Aug 2010 23:51:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 23:51:42 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.100 as permitted sender)
Received: from [17.148.16.100] (HELO asmtpout025.mac.com) (17.148.16.100)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 23:51:33 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; charset=us-ascii
Received: from [172.10.20.2]
 (166-205-142-204.mobile.mymmode.com [166.205.142.204])
 by asmtp025.mac.com (Sun Java(tm) System Messaging Server 6.3-8.01 (built Dec
 16 2008; 32bit)) with ESMTPSA id <0L79001A9RL44I50@asmtp025.mac.com> for
 general@hadoop.apache.org; Mon, 16 Aug 2010 16:51:07 -0700 (PDT)
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0
 reason=mlx engine=6.0.2-1004200000 definitions=main-1008160212
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.0.10011,1.0.148,0.0.0000
 definitions=2010-08-16_09:2010-08-17,2010-08-16,1970-01-01 signatures=0
Subject: Re: [VOTE] Should we define the Common committers as HDFS + MapReduce
 committers?
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <4C69BD9E.9030802@apache.org>
Date: Mon, 16 Aug 2010 16:51:03 -0700
Message-id: <48C487FD-C99E-461B-996E-4711C786D992@mac.com>
References: <7962BB4E-7176-44EB-A708-64881639E2AF@apache.org>
 <4C69BD9E.9030802@apache.org>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1081)
X-Virus-Checked: Checked by ClamAV on apache.org

Doug, I don't think that's what this vote is doing.  It's not giving all committers access to HDFS + MR + Common.  It seems to be just making committers for Common the union of HDFS and MR.

I'm +1 on the vote.

Nige

On Aug 16, 2010, at 3:37 PM, Doug Cutting wrote:

> +1 I believe that so long as we have synchronized releases then we're co-developing a single product and should be a single community.
> 
> Doug
> 
> On 08/16/2010 02:38 PM, Owen O'Malley wrote:
>> Since the project split, it is mostly required that any MapReduce or
>> HDFS committers have access to change Common. We haven't had any cases
>> where we wanted to nominate any one for just Common. Toward the goal of
>> simplifying the structure, I'd propose that we define the Common
>> committers as precisely the union of the HDFS committers and MapReduce
>> committers.
>> 
>> Clearly, I'm +1.
>> 
>> -- Owen


From general-return-1961-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 02:06:32 2010
Return-Path: <general-return-1961-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93529 invoked from network); 17 Aug 2010 02:06:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 02:06:32 -0000
Received: (qmail 65687 invoked by uid 500); 17 Aug 2010 02:06:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65544 invoked by uid 500); 17 Aug 2010 02:06:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65530 invoked by uid 99); 17 Aug 2010 02:06:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 02:06:30 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of swhitecross@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 02:06:24 +0000
Received: by vws4 with SMTP id 4so5458424vws.35
        for <multiple recipients>; Mon, 16 Aug 2010 19:06:03 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:cc:content-type;
        bh=XnSy1u1za9giTboE2LCeyg8UpCPvJqn+pjFJpLxL09I=;
        b=v9wbLmVHVxGb1KfNhXbvynDhyOIVbHiFXtFFVmVlBwcWB0mkjHIFiSjL+T9iLaaObf
         sd3wwcDZI0mUWz7IuZ3615VdEW2PaSC17lQDrePm8CRpz6yCdVctKMk5nC3l8NOMEu1X
         Y+T6/2dKsLuLrErMxx7r26B5+K97/76Ims+rM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=eVZwLrjbcvuEJ+YF3+jhGO6A2Bw0eqLHg/qBjzD8FjvybrFQ/hadwWGGWGgoN7+VWQ
         7aFGd5PyMBNS6+DYwfwON/vFxPqXzvqNC8jAF7CFYRKqtbkALOHZcw2E0FpQaEmDRYbD
         ei+VnhEB1cMDIWnXlph2ymJbzN6UT9zBKNPEg=
MIME-Version: 1.0
Received: by 10.220.158.9 with SMTP id d9mr3603382vcx.249.1282010763363; Mon,
 16 Aug 2010 19:06:03 -0700 (PDT)
Received: by 10.220.128.204 with HTTP; Mon, 16 Aug 2010 19:06:03 -0700 (PDT)
In-Reply-To: <0F9D3E90-4F42-4DFC-B48A-F61A6B476F42@yahoo-inc.com>
References: <AANLkTin9vdavJYgHDPYzKg_DriLgp6BrgPkiqXTvayqz@mail.gmail.com>
	<0F9D3E90-4F42-4DFC-B48A-F61A6B476F42@yahoo-inc.com>
Date: Mon, 16 Aug 2010 22:06:03 -0400
Message-ID: <AANLkTinjHLf024W-E_7_h0Px=vE4Q-ctX4jiBKUzx9u5@mail.gmail.com>
Subject: Re: Listing Hadoop Job History Statistics
From: Scott Whitecross <swhitecross@gmail.com>
To: general@hadoop.apache.org
Cc: mapreduce-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=e0cb4e8878e3260567048dfb6447

--e0cb4e8878e3260567048dfb6447
Content-Type: text/plain; charset=ISO-8859-1

Thanks for the answers Doug and Arun.   I'm assuming the job-history files
mentioned are in ./hadoop-0.20/logs/history/done/.  The files look like they
were serialized by a class in Hadoop?  (If I can read the files back into
the appropriate class, and then dump them out into a custom format, that'd
be great.)

Thanks.



On Thu, Aug 12, 2010 at 12:52 AM, Arun C Murthy <acm@yahoo-inc.com> wrote:

> Moving to mapreduce-user@, bcc general@.
>
> There isn't a direct way. One possible option is just use the per-job
> job-history file which is on HDFS (See
> http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html#Job+Submission+and+Monitoringfor info on job-history).
>
> Hope that helps.
>
> Arun
>
>
> On Aug 11, 2010, at 8:54 AM, Scott Whitecross wrote:
>
>  Hi -
>>
>> What's the best way to list and query information on Hadoop job histories?
>> For example, I'd like to see the job names from the past week against a
>> Hadoop cluster I'm using.   I don't see an API call or a way through the
>> command line to pull the information.  Is the best way writing a quick
>> script to process the job history files?
>>
>> Thanks.
>> Scott
>>
>
>

--e0cb4e8878e3260567048dfb6447--

From general-return-1962-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 04:30:25 2010
Return-Path: <general-return-1962-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47013 invoked from network); 17 Aug 2010 04:30:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 04:30:24 -0000
Received: (qmail 35502 invoked by uid 500); 17 Aug 2010 04:30:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34668 invoked by uid 500); 17 Aug 2010 04:30:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34653 invoked by uid 99); 17 Aug 2010 04:30:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 04:30:18 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 04:30:13 +0000
Received: from dishnailfind-dx.eglbp.corp.yahoo.com ([10.66.77.50])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7H4TgM1097810;
	Mon, 16 Aug 2010 21:29:43 -0700 (PDT)
Message-ID: <4C6A1036.6080004@yahoo-inc.com>
Date: Tue, 17 Aug 2010 09:59:42 +0530
From: Ranjit Mathew <ranjit@yahoo-inc.com>
Organization: Yahoo
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.9.2.8) Gecko/20100802 Lightning/1.0b2 Thunderbird/3.1.2
MIME-Version: 1.0
To: Scott Whitecross <swhitecross@gmail.com>
CC: "mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>
Subject: Re: Listing Hadoop Job History Statistics
References: <AANLkTin9vdavJYgHDPYzKg_DriLgp6BrgPkiqXTvayqz@mail.gmail.com>	<0F9D3E90-4F42-4DFC-B48A-F61A6B476F42@yahoo-inc.com> <AANLkTinjHLf024W-E_7_h0Px=vE4Q-ctX4jiBKUzx9u5@mail.gmail.com>
In-Reply-To: <AANLkTinjHLf024W-E_7_h0Px=vE4Q-ctX4jiBKUzx9u5@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

[BCC-ing "general" - again.]

On Tuesday 17 August 2010 07:36 AM, Scott Whitecross wrote:
> Thanks for the answers Doug and Arun.   I'm assuming the job-history files
> mentioned are in ./hadoop-0.20/logs/history/done/.  The files look like they
> were serialized by a class in Hadoop?  (If I can read the files back into
> the appropriate class, and then dump them out into a custom format, that'd
> be great.)

Rumen (src/tools/org/apache/hadoop/tools/rumen/) parses Job History files
and creates JSON files that can be either be loaded independently, or via
the API provided by Rumen itself. As an added benefit, it abstracts away
the differences between the 0.20.xx format and the Avro-based format used
in trunk.

There is not much documentation on Rumen right now, but MAPREDUCE-1918
(https://issues.apache.org/jira/browse/MAPREDUCE-1918) attempts to fix
that.

HTH,
Ranjit

> On Thu, Aug 12, 2010 at 12:52 AM, Arun C Murthy<acm@yahoo-inc.com>  wrote:
>
>> Moving to mapreduce-user@, bcc general@.
>>
>> There isn't a direct way. One possible option is just use the per-job
>> job-history file which is on HDFS (See
>> http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html#Job+Submission+and+Monitoringfor info on job-history).
>>
>> Hope that helps.
>>
>> Arun
>>
>>
>> On Aug 11, 2010, at 8:54 AM, Scott Whitecross wrote:
>>
>>   Hi -
>>>
>>> What's the best way to list and query information on Hadoop job histories?
>>> For example, I'd like to see the job names from the past week against a
>>> Hadoop cluster I'm using.   I don't see an API call or a way through the
>>> command line to pull the information.  Is the best way writing a quick
>>> script to process the job history files?
>>>
>>> Thanks.
>>> Scott
>>>
>>
>>


From general-return-1963-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 06:15:06 2010
Return-Path: <general-return-1963-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84158 invoked from network); 17 Aug 2010 06:15:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 06:15:05 -0000
Received: (qmail 97172 invoked by uid 500); 17 Aug 2010 06:15:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96789 invoked by uid 500); 17 Aug 2010 06:15:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96781 invoked by uid 99); 17 Aug 2010 06:15:02 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 06:15:02 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yhemanth@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 06:14:38 +0000
Received: by vws4 with SMTP id 4so5642018vws.35
        for <general@hadoop.apache.org>; Mon, 16 Aug 2010 23:14:18 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=DjOVFZC6SMlDrnUIjNj1zOn1c0wpCpPSxlF5Soq8p3w=;
        b=shCIpERAQ4546LLyYDv7a95Pb06W8rMatqEMW3JOPld9SfqkEInBrAyFT2PoASmqZs
         lFknfRI8k3ovtvixxPlJNOZX2PrQGeUsNvJiGHdfMxU95hZmRg0CkujsRBQbXBu/NuWT
         yUTa1xab+HY8YPm+b7cWckpB0N81s/3d5K444=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=TC8jI2YV5HR3gVomSwuLk0gSPINGvK6eMIpFGtFOFE8L+Fram76iOPYJFcinnRcy5D
         ZBsCK7irH9Yg8Vk889fJS2YnN79qA9dpkDKp0uwvebzPJW4q0ScOPz/aZDXALLcNPiK0
         aV6RznQO4ZaLndvN11t9sEALupWdoVzKaZJHo=
MIME-Version: 1.0
Received: by 10.220.121.206 with SMTP id i14mr3701407vcr.1.1282025657782; Mon,
 16 Aug 2010 23:14:17 -0700 (PDT)
Received: by 10.220.178.135 with HTTP; Mon, 16 Aug 2010 23:14:17 -0700 (PDT)
In-Reply-To: <7962BB4E-7176-44EB-A708-64881639E2AF@apache.org>
References: <7962BB4E-7176-44EB-A708-64881639E2AF@apache.org>
Date: Tue, 17 Aug 2010 11:44:17 +0530
Message-ID: <AANLkTim8LTe2bHUez8xwJwC96yUEtUcWpSckf7ze7OHP@mail.gmail.com>
Subject: Re: [VOTE] Should we define the Common committers as HDFS + MapReduce committers?
From: Hemanth Yamijala <yhemanth@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Thanks
Hemanth

On Tue, Aug 17, 2010 at 3:08 AM, Owen O'Malley <omalley@apache.org> wrote:
> Since the project split, it is mostly required that any MapReduce or HDFS
> committers have access to change Common. We haven't had any cases where we
> wanted to nominate any one for just Common. Toward the goal of simplifying
> the structure, I'd propose that we define the Common committers as precisely
> the union of the HDFS committers and MapReduce committers.
>
> Clearly, I'm +1.
>
> -- Owen
>

From general-return-1964-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 10:16:15 2010
Return-Path: <general-return-1964-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84581 invoked from network); 17 Aug 2010 10:16:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 10:16:15 -0000
Received: (qmail 37419 invoked by uid 500); 17 Aug 2010 10:16:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37139 invoked by uid 500); 17 Aug 2010 10:16:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37131 invoked by uid 99); 17 Aug 2010 10:16:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 10:16:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 17 Aug 2010 10:16:09 +0000
Received: (qmail 84491 invoked by uid 99); 17 Aug 2010 10:15:49 -0000
Received: from localhost.apache.org (HELO mail-gw0-f48.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 10:15:49 +0000
Received: by gwj19 with SMTP id 19so3347713gwj.35
        for <general@hadoop.apache.org>; Tue, 17 Aug 2010 03:15:48 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.162.13 with SMTP id t13mr7248650ibx.160.1282040147936;
 Tue, 17 Aug 2010 03:15:47 -0700 (PDT)
Received: by 10.231.171.75 with HTTP; Tue, 17 Aug 2010 03:15:47 -0700 (PDT)
Date: Tue, 17 Aug 2010 03:15:47 -0700
Message-ID: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com>
Subject: [VOTE] Combine MapReduce/HDFS Committers
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Per the discussion thread: http://s.apache.org/XkY

Should HDFS and MapReduce committers lists be combined and all
subsequent committers on either of these two projects be granted karma
in the other?

If the vote passes, current and future committers to MapReduce and
HDFS will gain commit rights in both projects. Commit rights to Common
are unaffected.

Without bylaws, a 2/3 majority for a committer import seems like a
reasonable bar, given that adding an individual committer requires
consensus.

----

Owen has started a separate voting thread, proposing to define the
Common committer list as the union of HDFS and MapReduce committers
(vote A), so I tried to write this (vote B) so it would not conflict.
As I'm reading it:

A passes, B passes: One can become a committer on HDFS or MapReduce.
Commit to either implies commit on HDFS, MR, and Common.
A passes, B fails: One can become a committer on HDFS or MapReduce.
Commit to either implies commit on Common, only.
A fails, B passes: One can become a committer on HDFS, MapReduce, or
Common. Commit to to HDFS/MR implies converse, but individual
appointments to Common continue.
A fails, B fails: Committers continue to be appointed individually to
HDFS, MapReduce, and Common.

In no scheme would commit rights to Common imply commit rights to
either HDFS or MapReduce, I guess. -C

From general-return-1965-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 10:17:10 2010
Return-Path: <general-return-1965-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84877 invoked from network); 17 Aug 2010 10:17:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 10:17:09 -0000
Received: (qmail 38912 invoked by uid 500); 17 Aug 2010 10:17:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38814 invoked by uid 500); 17 Aug 2010 10:17:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 38792 invoked by uid 99); 17 Aug 2010 10:17:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 10:17:05 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 17 Aug 2010 10:17:05 +0000
Received: (qmail 84775 invoked by uid 99); 17 Aug 2010 10:16:45 -0000
Received: from localhost.apache.org (HELO mail-yx0-f176.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 10:16:45 +0000
Received: by yxm34 with SMTP id 34so3355361yxm.35
        for <general@hadoop.apache.org>; Tue, 17 Aug 2010 03:16:43 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.157.212 with SMTP id c20mr7238512ibx.186.1282040202723;
 Tue, 17 Aug 2010 03:16:42 -0700 (PDT)
Received: by 10.231.171.75 with HTTP; Tue, 17 Aug 2010 03:16:42 -0700 (PDT)
In-Reply-To: <7962BB4E-7176-44EB-A708-64881639E2AF@apache.org>
References: <7962BB4E-7176-44EB-A708-64881639E2AF@apache.org>
Date: Tue, 17 Aug 2010 03:16:42 -0700
Message-ID: <AANLkTi=ZhdN9dfnaNUeKMvSYGMASVeQCizMM3uBdQAYG@mail.gmail.com>
Subject: Re: [VOTE] Should we define the Common committers as HDFS + MapReduce committers?
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

+1

On Mon, Aug 16, 2010 at 2:38 PM, Owen O'Malley <omalley@apache.org> wrote:
> Since the project split, it is mostly required that any MapReduce or HDFS
> committers have access to change Common. We haven't had any cases where we
> wanted to nominate any one for just Common. Toward the goal of simplifying
> the structure, I'd propose that we define the Common committers as precisely
> the union of the HDFS committers and MapReduce committers.
>
> Clearly, I'm +1.
>
> -- Owen
>

From general-return-1966-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 10:27:56 2010
Return-Path: <general-return-1966-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88442 invoked from network); 17 Aug 2010 10:27:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 10:27:55 -0000
Received: (qmail 56534 invoked by uid 500); 17 Aug 2010 10:27:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56136 invoked by uid 500); 17 Aug 2010 10:27:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56128 invoked by uid 99); 17 Aug 2010 10:27:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 10:27:51 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of bjoern@schiessle.org designates 78.46.69.5 as permitted sender)
Received: from [78.46.69.5] (HELO zucker.schokokeks.org) (78.46.69.5)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 10:27:44 +0000
Received: from pcmoholynagy.informatik.uni-stuttgart.de (pcmoholynagy.informatik.uni-stuttgart.de [::ffff:129.69.216.55])
  (AUTH: LOGIN schiesbn@schokokeks.org, TLS: TLSv1/SSLv3,128bits,AES128-SHA)
  by zucker.schokokeks.org with esmtp; Tue, 17 Aug 2010 12:27:23 +0200
  id 0000000000018010.000000004C6A640B.00003F69
Date: Tue, 17 Aug 2010 12:27:19 +0200
From: Bjoern Schiessle <bjoern@schiessle.org>
To: general@hadoop.apache.org
Subject: Status of the Heart proposal?
Message-ID: <20100817122719.1165533d@pcmoholynagy.informatik.uni-stuttgart.de>
X-Mailer: Claws Mail 3.7.6 (GTK+ 2.20.1; i486-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Hi all,

I'm quite new to Hadoop and discover all the possibilities. I read about
the Heart proposal[1] and it sounds really promising.

But it looks a little bit outdated. The link to the subversion
repository[2] is dead and the documentation[3] only links to some
directories.

What is the status of the proposal? Is it still active?

Btw: Are there online archives for the mailing lists like
heart-dev@incubator.apache.org and heart-user@incubator.apache.org? I
couldn't find it.

[1] http://wiki.apache.org/incubator/HeartProposal
[2] https://svn.apache.org/repos/asf/incubator/heart
[3] http://heart.korea.ac.kr/

best wishes,
Bj=F6rn

--=20
Bj=F6rn Schie=DFle
Support Free Software, join FSFE's Fellowship (fellowship.fsfe.org)
Buy books and support Free Software (wiki.fsfe.org/SupportPrograms)

From general-return-1967-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 16:08:23 2010
Return-Path: <general-return-1967-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46973 invoked from network); 17 Aug 2010 16:08:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 16:08:22 -0000
Received: (qmail 60720 invoked by uid 500); 17 Aug 2010 16:08:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60567 invoked by uid 500); 17 Aug 2010 16:08:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60559 invoked by uid 99); 17 Aug 2010 16:08:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 16:08:20 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 16:08:12 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 31452B7D14
	for <general@hadoop.apache.org>; Tue, 17 Aug 2010 17:07:51 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id ungrqCT7fxCg for <general@hadoop.apache.org>;
	Tue, 17 Aug 2010 17:07:50 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 6ABF3B7D13
	for <general@hadoop.apache.org>; Tue, 17 Aug 2010 17:07:49 +0100 (BST)
MailScanner-NULL-Check: 1282666057.38931@vkbqdBkLAHKXbfy4GLOH5w
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o7HG7bsg004692
	for <general@hadoop.apache.org>; Tue, 17 Aug 2010 17:07:37 +0100 (BST)
Message-ID: <4C6AB3C9.3050705@apache.org>
Date: Tue, 17 Aug 2010 17:07:37 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.11) Gecko/20100713 Thunderbird/3.0.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Should we define the Common committers as HDFS + MapReduce
 committers?
References: <7962BB4E-7176-44EB-A708-64881639E2AF@apache.org>
In-Reply-To: <7962BB4E-7176-44EB-A708-64881639E2AF@apache.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o7HG7bsg004692
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 16/08/10 22:38, Owen O'Malley wrote:
> Since the project split, it is mostly required that any MapReduce or
> HDFS committers have access to change Common. We haven't had any cases
> where we wanted to nominate any one for just Common. Toward the goal of
> simplifying the structure, I'd propose that we define the Common
> committers as precisely the union of the HDFS committers and MapReduce
> committers.
>
> Clearly, I'm +1.
>
> -- Owen

+1

steve. Presumably this means I need to read the hdfs-dev and mapred-dev 
lists too ...

From general-return-1968-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 16:10:50 2010
Return-Path: <general-return-1968-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47958 invoked from network); 17 Aug 2010 16:10:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 16:10:50 -0000
Received: (qmail 62968 invoked by uid 500); 17 Aug 2010 16:10:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62904 invoked by uid 500); 17 Aug 2010 16:10:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62896 invoked by uid 99); 17 Aug 2010 16:10:48 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 16:10:48 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 16:10:26 +0000
Received: by bwz14 with SMTP id 14so3562993bwz.35
        for <general@hadoop.apache.org>; Tue, 17 Aug 2010 09:10:05 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.103.199 with SMTP id l7mr4565557bko.18.1282061404888; Tue,
 17 Aug 2010 09:10:04 -0700 (PDT)
Received: by 10.204.179.74 with HTTP; Tue, 17 Aug 2010 09:10:04 -0700 (PDT)
In-Reply-To: <7962BB4E-7176-44EB-A708-64881639E2AF@apache.org>
References: <7962BB4E-7176-44EB-A708-64881639E2AF@apache.org>
Date: Tue, 17 Aug 2010 09:10:04 -0700
Message-ID: <AANLkTinoZYmHnNEXyXhQ4dG8oTLaeWZ5z71arXHp2RV9@mail.gmail.com>
Subject: Re: [VOTE] Should we define the Common committers as HDFS + MapReduce committers?
From: Tom White <tom@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Tom

On Mon, Aug 16, 2010 at 2:38 PM, Owen O'Malley <omalley@apache.org> wrote:
> Since the project split, it is mostly required that any MapReduce or HDFS
> committers have access to change Common. We haven't had any cases where we
> wanted to nominate any one for just Common. Toward the goal of simplifying
> the structure, I'd propose that we define the Common committers as precisely
> the union of the HDFS committers and MapReduce committers.
>
> Clearly, I'm +1.
>
> -- Owen
>

From general-return-1969-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 16:11:25 2010
Return-Path: <general-return-1969-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 48218 invoked from network); 17 Aug 2010 16:11:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 16:11:25 -0000
Received: (qmail 64202 invoked by uid 500); 17 Aug 2010 16:11:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 64116 invoked by uid 500); 17 Aug 2010 16:11:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 64107 invoked by uid 99); 17 Aug 2010 16:11:24 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 16:11:24 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 209.85.214.48 is neither permitted nor denied by domain of tom@cloudera.com)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 16:11:01 +0000
Received: by mail-bw0-f48.google.com with SMTP id 14so3562993bwz.35
        for <general@hadoop.apache.org>; Tue, 17 Aug 2010 09:10:41 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.35.69 with SMTP id o5mr4536490bkd.87.1282061441356; Tue,
 17 Aug 2010 09:10:41 -0700 (PDT)
Received: by 10.204.179.74 with HTTP; Tue, 17 Aug 2010 09:10:41 -0700 (PDT)
In-Reply-To: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com>
References: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com>
Date: Tue, 17 Aug 2010 09:10:41 -0700
Message-ID: <AANLkTin4JOU5YOPDwCJKpVL20d--03LB-ar++PaYmF36@mail.gmail.com>
Subject: Re: [VOTE] Combine MapReduce/HDFS Committers
From: Tom White <tom@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Tue, Aug 17, 2010 at 3:15 AM, Chris Douglas <cdouglas@apache.org> wrote:
> Per the discussion thread: http://s.apache.org/XkY
>
> Should HDFS and MapReduce committers lists be combined and all
> subsequent committers on either of these two projects be granted karma
> in the other?

+1

Tom

From general-return-1970-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 16:13:35 2010
Return-Path: <general-return-1970-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49458 invoked from network); 17 Aug 2010 16:13:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 16:13:35 -0000
Received: (qmail 72988 invoked by uid 500); 17 Aug 2010 16:13:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72808 invoked by uid 500); 17 Aug 2010 16:13:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72800 invoked by uid 99); 17 Aug 2010 16:13:33 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 16:13:33 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 16:13:10 +0000
Received: by pvg3 with SMTP id 3so3380199pvg.35
        for <general@hadoop.apache.org>; Tue, 17 Aug 2010 09:12:49 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.142.128.12 with SMTP id a12mr6068853wfd.9.1282061569354; Tue,
 17 Aug 2010 09:12:49 -0700 (PDT)
Received: by 10.142.177.15 with HTTP; Tue, 17 Aug 2010 09:12:49 -0700 (PDT)
In-Reply-To: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com>
References: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com>
Date: Tue, 17 Aug 2010 09:12:49 -0700
Message-ID: <AANLkTikGxPt44KjKVkV0882LWf2qjnqfCF+E3sF+Y2qb@mail.gmail.com>
Subject: Re: [VOTE] Combine MapReduce/HDFS Committers
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Tue, Aug 17, 2010 at 3:15 AM, Chris Douglas <cdouglas@apache.org> wrote:
> Per the discussion thread: http://s.apache.org/XkY
>
> Should HDFS and MapReduce committers lists be combined and all
> subsequent committers on either of these two projects be granted karma
> in the other?

+1      (non-binding)

Thanks,
Eli

From general-return-1971-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 16:31:35 2010
Return-Path: <general-return-1971-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57580 invoked from network); 17 Aug 2010 16:31:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 16:31:34 -0000
Received: (qmail 97400 invoked by uid 500); 17 Aug 2010 16:31:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97282 invoked by uid 500); 17 Aug 2010 16:31:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97274 invoked by uid 99); 17 Aug 2010 16:31:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 16:31:32 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 17 Aug 2010 16:31:32 +0000
Received: (qmail 57463 invoked by uid 99); 17 Aug 2010 16:31:11 -0000
Received: from localhost.apache.org (HELO [192.168.168.123]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 16:31:11 +0000
Message-ID: <4C6AB94F.8090405@apache.org>
Date: Tue, 17 Aug 2010 09:31:11 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.11) Gecko/20100713 Thunderbird/3.0.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Combine MapReduce/HDFS Committers
References: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com>
In-Reply-To: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

+1

On 08/17/2010 03:15 AM, Chris Douglas wrote:
> Per the discussion thread: http://s.apache.org/XkY
>
> Should HDFS and MapReduce committers lists be combined and all
> subsequent committers on either of these two projects be granted karma
> in the other?
>
> If the vote passes, current and future committers to MapReduce and
> HDFS will gain commit rights in both projects. Commit rights to Common
> are unaffected.
>
> Without bylaws, a 2/3 majority for a committer import seems like a
> reasonable bar, given that adding an individual committer requires
> consensus.
>
> ----
>
> Owen has started a separate voting thread, proposing to define the
> Common committer list as the union of HDFS and MapReduce committers
> (vote A), so I tried to write this (vote B) so it would not conflict.
> As I'm reading it:
>
> A passes, B passes: One can become a committer on HDFS or MapReduce.
> Commit to either implies commit on HDFS, MR, and Common.
> A passes, B fails: One can become a committer on HDFS or MapReduce.
> Commit to either implies commit on Common, only.
> A fails, B passes: One can become a committer on HDFS, MapReduce, or
> Common. Commit to to HDFS/MR implies converse, but individual
> appointments to Common continue.
> A fails, B fails: Committers continue to be appointed individually to
> HDFS, MapReduce, and Common.
>
> In no scheme would commit rights to Common imply commit rights to
> either HDFS or MapReduce, I guess. -C

From general-return-1972-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 17:18:01 2010
Return-Path: <general-return-1972-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 80681 invoked from network); 17 Aug 2010 17:18:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 17:18:00 -0000
Received: (qmail 70341 invoked by uid 500); 17 Aug 2010 17:17:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70223 invoked by uid 500); 17 Aug 2010 17:17:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70206 invoked by uid 99); 17 Aug 2010 17:17:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 17:17:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 17 Aug 2010 17:17:57 +0000
Received: (qmail 80506 invoked by uid 99); 17 Aug 2010 17:17:37 -0000
Received: from localhost.apache.org (HELO mail-yw0-f48.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 17:17:37 +0000
Received: by ywg4 with SMTP id 4so2176856ywg.35
        for <general@hadoop.apache.org>; Tue, 17 Aug 2010 10:17:36 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.191.147 with SMTP id dm19mr7903319ibb.6.1282065455708;
 Tue, 17 Aug 2010 10:17:35 -0700 (PDT)
Received: by 10.231.171.75 with HTTP; Tue, 17 Aug 2010 10:17:35 -0700 (PDT)
In-Reply-To: <7962BB4E-7176-44EB-A708-64881639E2AF@apache.org>
References: <7962BB4E-7176-44EB-A708-64881639E2AF@apache.org>
Date: Tue, 17 Aug 2010 10:17:35 -0700
Message-ID: <AANLkTinMS5D14DKQvVArgy_y38V-V2wXKU9dDXtSC25w@mail.gmail.com>
Subject: Re: [VOTE] Should we define the Common committers as HDFS + MapReduce committers?
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

+1 -C

On Mon, Aug 16, 2010 at 2:38 PM, Owen O'Malley <omalley@apache.org> wrote:
> Since the project split, it is mostly required that any MapReduce or HDFS
> committers have access to change Common. We haven't had any cases where we
> wanted to nominate any one for just Common. Toward the goal of simplifying
> the structure, I'd propose that we define the Common committers as precisely
> the union of the HDFS committers and MapReduce committers.
>
> Clearly, I'm +1.
>
> -- Owen
>

From general-return-1973-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 17:28:48 2010
Return-Path: <general-return-1973-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 86210 invoked from network); 17 Aug 2010 17:28:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 17:28:48 -0000
Received: (qmail 85336 invoked by uid 500); 17 Aug 2010 17:28:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85269 invoked by uid 500); 17 Aug 2010 17:28:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85261 invoked by uid 99); 17 Aug 2010 17:28:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 17:28:46 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [206.190.56.20] (HELO n1.bullet.mail.re4.yahoo.com) (206.190.56.20)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 17 Aug 2010 17:28:37 +0000
Received: from [68.142.237.90] by n1.bullet.re4.yahoo.com with NNFMP; 17 Aug 2010 17:28:16 -0000
Received: from [66.196.97.153] by t6.bullet.re3.yahoo.com with NNFMP; 17 Aug 2010 17:28:16 -0000
Received: from [127.0.0.1] by omp206.mail.re3.yahoo.com with NNFMP; 17 Aug 2010 17:28:16 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 169820.31105.bm@omp206.mail.re3.yahoo.com
Received: (qmail 27824 invoked by uid 60001); 17 Aug 2010 17:28:16 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1282066096; bh=/rFJk6RRiO4ozDmufe7yRkC8423WAlQ/eqlQrnaN9VA=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=vo+iGk58092eQ/sdOfGqV7PuKjAbL32D+jNnDhXv4SJylj0r2WcSZig24/debzQrwm74q8b9dRlpuXvkgE0l5gnflFqmzKjJ57FCUw+L5ctpgY9ckz6H05Prxivcp5gGJi7MioO5rLbYx23dmDAXGqTKfxT7iyfhXeuAN9Abon0=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=YQi5rI2SsJjFbHtoWVOjm0iWpiEsgCq4vP7BpElt1HG0nX3EkgF3YRNtpArthUUM3xAxU3JCxB4T7aF2Oab3k4ObiXT7nC5HmjwkijMOQblXBay+ilH8vRaAiOyVYw4TQwaXjKRjgVzV1GlIQMLAq6NkOPTqA3ObObrNQlJaMYI=;
Message-ID: <54087.26421.qm@web56201.mail.re3.yahoo.com>
X-YMail-OSG: pauG06IVM1kyb6r13V74LdKbyZbWByTJqN8DzAZWWUMvUJX
 CaSLfL3WAFRH1N0UWNNCOTD1AYup0lhtPM4pXF8NEY528npDPFaS2cKVQOkC
 viBy91Srweo5ghEztUzvvPLsrPxKnXUH9gQpFSH_QmnJn9MReXawgw1u1Dko
 81ObdeKMpoTwt5Ovy.rUkVWJZCQQ0DchoRvZAypUtBzgxWAEFP9jKpMJSQnN
 bi_lL139bqKNj.H67t_ok.pqcBjsftJg7rOcP5Pr80SD9i1bHOF.V_EqmMIV
 uYLt0Zkafi71drDlKiegUmxik4o7ZVxmc4rDqYAZC4QW4FIdlyW9fGAwWNX6
 tXauq7LZsxun6uIde7aOP2t07YHB7zEjW72bukPZY8t16Troo
Received: from [209.131.62.113] by web56201.mail.re3.yahoo.com via HTTP; Tue, 17 Aug 2010 10:28:15 PDT
X-Mailer: YahooMailRC/470 YahooMailWebService/0.8.105.279950
References: <7962BB4E-7176-44EB-A708-64881639E2AF@apache.org>
Date: Tue, 17 Aug 2010 10:28:15 -0700 (PDT)
From: "Tsz Wo \(Nicholas\), Sze" <s29752-hadoopgeneral@yahoo.com>
Subject: Re: [VOTE] Should we define the Common committers as HDFS + MapReduce committers?
To: general@hadoop.apache.org
In-Reply-To: <7962BB4E-7176-44EB-A708-64881639E2AF@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii

It is possibly to have contributors who are only interested in RPC, 
serialization, compression, etc.  However, just as Owen mentioned, we don't have 
such cases yet.  We might decide how to accommodate these contributors when we 
have such cases.

+1

Nicholas




----- Original Message ----
> From: Owen O'Malley <omalley@apache.org>
> To: general@hadoop.apache.org
> Sent: Mon, August 16, 2010 2:38:53 PM
> Subject: [VOTE] Should we define the Common committers as HDFS + MapReduce 
>committers?
> 
> Since the project split, it is mostly required that any MapReduce or HDFS  
>committers have access to change Common. We haven't had any cases where we  
>wanted to nominate any one for just Common. Toward the goal of simplifying the  
>structure, I'd propose that we define the Common committers as precisely the  
>union of the HDFS committers and MapReduce committers.
> 
> Clearly, I'm  +1.
> 
> -- Owen
> 


From general-return-1974-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 17:42:11 2010
Return-Path: <general-return-1974-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91072 invoked from network); 17 Aug 2010 17:42:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 17:42:11 -0000
Received: (qmail 3264 invoked by uid 500); 17 Aug 2010 17:42:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3177 invoked by uid 500); 17 Aug 2010 17:42:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3147 invoked by uid 99); 17 Aug 2010 17:42:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 17:42:09 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of teamphy6@gmail.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 17:42:03 +0000
Received: by gyb13 with SMTP id 13so3651872gyb.35
        for <general@hadoop.apache.org>; Tue, 17 Aug 2010 10:41:42 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=ZwoNf4SrOQDhUDHiVBDtVUt/T5vjb7ajOZ/d95GuNsc=;
        b=F1d6xc54sLJShS8RKeD8gKJsKY9nhCxyycXfNCdZ4RwfPgIFtJe/597Hl4NXulWSZ8
         RfGj055AMI1dBi7P6yGVrnJvJ5yojDclSrm+8F6KMF/MPmUTUe7WKS5aLSaZSF+UMWpg
         Ci7JVSG1193YSVV1EUIZrC6XFxdJTubTxXPzE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=ofZl8TOLJMGbPG/YWAfZgpkQhsJ036I4dlFClqaakNT0qLheZGZWVxDws6DLlA7/8E
         FGtSZpJsaczJEKMovShPqp8+/0lg6hvyaB70NY3U4Ny/Hk3sFZl/ZDvQDtfy61FJTL9y
         CZJ2KYpfIh0MuVpoDXEiHGl5+tNiLgGecxVy0=
MIME-Version: 1.0
Received: by 10.100.131.18 with SMTP id e18mr7911335and.219.1282066902402;
 Tue, 17 Aug 2010 10:41:42 -0700 (PDT)
Received: by 10.231.36.9 with HTTP; Tue, 17 Aug 2010 10:41:42 -0700 (PDT)
In-Reply-To: <54087.26421.qm@web56201.mail.re3.yahoo.com>
References: <7962BB4E-7176-44EB-A708-64881639E2AF@apache.org>
	<54087.26421.qm@web56201.mail.re3.yahoo.com>
Date: Tue, 17 Aug 2010 13:41:42 -0400
Message-ID: <AANLkTimwHgcgpjkxizC9xVWfsA8MndHK6KUu+t0pMXUs@mail.gmail.com>
Subject: Re: [VOTE] Should we define the Common committers as HDFS + MapReduce committers?
From: Daniel Jue <teamphy6@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

+1

On Tue, Aug 17, 2010 at 1:28 PM, Tsz Wo (Nicholas), Sze
<s29752-hadoopgeneral@yahoo.com> wrote:
> It is possibly to have contributors who are only interested in RPC,
> serialization, compression, etc. =A0However, just as Owen mentioned, we d=
on't have
> such cases yet. =A0We might decide how to accommodate these contributors =
when we
> have such cases.
>
> +1
>
> Nicholas
>
>
>
>
> ----- Original Message ----
>> From: Owen O'Malley <omalley@apache.org>
>> To: general@hadoop.apache.org
>> Sent: Mon, August 16, 2010 2:38:53 PM
>> Subject: [VOTE] Should we define the Common committers as HDFS + MapRedu=
ce
>>committers?
>>
>> Since the project split, it is mostly required that any MapReduce or HDF=
S
>>committers have access to change Common. We haven't had any cases where w=
e
>>wanted to nominate any one for just Common. Toward the goal of simplifyin=
g the
>>structure, I'd propose that we define the Common committers as precisely =
the
>>union of the HDFS committers and MapReduce committers.
>>
>> Clearly, I'm =A0+1.
>>
>> -- Owen
>>
>
>

From general-return-1975-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 18:04:47 2010
Return-Path: <general-return-1975-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1357 invoked from network); 17 Aug 2010 18:04:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 18:04:46 -0000
Received: (qmail 48880 invoked by uid 500); 17 Aug 2010 18:04:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48787 invoked by uid 500); 17 Aug 2010 18:04:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48779 invoked by uid 99); 17 Aug 2010 18:04:44 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 18:04:44 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 18:04:23 +0000
Received: from localhost (snvvpn4-10-72-168-c110.hq.corp.yahoo.com [10.72.168.110])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7HI3eJI083148
	for <general@hadoop.apache.org>; Tue, 17 Aug 2010 11:03:40 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=date:from:to:subject:message-id:references:mime-version:
	content-type:content-disposition:in-reply-to:x-organization:user-agent;
	b=B/bEgi+esLd/CQN7vexmYZt3Cmx4p/2WwWewG40Zd5T6dXbS7KEWO3xsvzrEj6kH
Date: Tue, 17 Aug 2010 11:03:39 -0700
From: Konstantin Boudnik <cos@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: [VOTE] Should we define the Common committers as HDFS +
 MapReduce committers?
Message-ID: <20100817180338.GC89226@goodenter-lm.local>
References: <7962BB4E-7176-44EB-A708-64881639E2AF@apache.org>
 <AANLkTi=ZhdN9dfnaNUeKMvSYGMASVeQCizMM3uBdQAYG@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature"; boundary="1ccMZA6j1vT5UqiK"
Content-Disposition: inline
In-Reply-To: <AANLkTi=ZhdN9dfnaNUeKMvSYGMASVeQCizMM3uBdQAYG@mail.gmail.com>
X-Organization: Yahoo! Hadoop (Grid computing)
User-Agent: Mutt/1.5.20 (2009-06-14)
X-Virus-Checked: Checked by ClamAV on apache.org

--1ccMZA6j1vT5UqiK
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

+1=20

On Tue, Aug 17, 2010 at 03:16AM, Chris Douglas wrote:
> +1
>=20
> On Mon, Aug 16, 2010 at 2:38 PM, Owen O'Malley <omalley@apache.org> wrote:
> > Since the project split, it is mostly required that any MapReduce or HD=
FS
> > committers have access to change Common. We haven't had any cases where=
 we
> > wanted to nominate any one for just Common. Toward the goal of simplify=
ing
> > the structure, I'd propose that we define the Common committers as prec=
isely
> > the union of the HDFS committers and MapReduce committers.
> >
> > Clearly, I'm +1.
> >
> > -- Owen
> >

--1ccMZA6j1vT5UqiK
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)

iF4EAREIAAYFAkxqzvoACgkQMqXifkwDoaHX7AD/a8u7+gJf4E6NGLLQfVu3lYTJ
2IBr3TxYcc3vnvGuNQkA/3WsF8SOkc01mt4WAY2Dtgk9b4BjINWv8lHBbeaWUWm9
=TJr/
-----END PGP SIGNATURE-----

--1ccMZA6j1vT5UqiK--

From general-return-1976-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 18:24:48 2010
Return-Path: <general-return-1976-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19763 invoked from network); 17 Aug 2010 18:24:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 18:24:48 -0000
Received: (qmail 71628 invoked by uid 500); 17 Aug 2010 18:24:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71454 invoked by uid 500); 17 Aug 2010 18:24:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71434 invoked by uid 99); 17 Aug 2010 18:24:46 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 18:24:46 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of bernd.fondermann@googlemail.com designates 209.85.216.169 as permitted sender)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 18:24:24 +0000
Received: by qyk12 with SMTP id 12so1067255qyk.14
        for <multiple recipients>; Tue, 17 Aug 2010 11:24:03 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:cc:content-type;
        bh=MSXoYpwQf58NR6YsUknwWZ/hYcp3pd91+V/dPYmQa3o=;
        b=QNRVYaafNKEuLK4/sqjgOzOlmNqchFFfq9SO8x0cGnfBwt7a6bxMnFGlukeol69U1o
         XzJt2zbJSuaHAAtWdrtg8BYuBeRxY9eV96RxUF0TpgyT7mPE04gKfecRk2cXChq0mJXW
         MxgsZrcAaCgu7jtMVy/ZpdfB4AYof6br+9CsY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:cc:content-type;
        b=oAeVMi3E7OKCVpP4BJ4xSYqc9oHhzNWHgQ8xGD7mL1FUrMFIji189DADgi13jz+xSw
         lUoYHHd5SSoM4GN8Zl9hWyNZb9P4GyQ+cvdf1TcZrJ2D4imf4XsfMcuDOZJ/W/LyQw7c
         Gb3O0gWtSY+jbI6i3ZFTuZfPd4e38BF6kyPQM=
MIME-Version: 1.0
Received: by 10.224.119.20 with SMTP id x20mr4548426qaq.249.1282069442945;
 Tue, 17 Aug 2010 11:24:02 -0700 (PDT)
Received: by 10.229.236.82 with HTTP; Tue, 17 Aug 2010 11:24:02 -0700 (PDT)
Date: Tue, 17 Aug 2010 20:24:02 +0200
Message-ID: <AANLkTin7bbgJAO-XYC2Pbkdv7M-F6inRdK-S11oT9WL-@mail.gmail.com>
Subject: [ANNOUNCE] Chukwa mailing lists will be moved soon
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: chukwa-dev@hadoop.apache.org, chukwa-user@hadoop.apache.org
Cc: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hi everyone,

All mailing lists,
chukwa-dev@, chukwa-user@ and chukwa-commits@hadoop.apache.org
will be moved over to ...@incubator.apache.org, over the next days.
As a subscriber, there is nothing you have to do. It will "just work",
fingers crossed.
But please be aware to adjust your mailing list filters, eventually.

See
  http://issues.apache.org/jira/browse/INFRA-2934

Thanks for your continued interest in Chukwa,

  Bernd

From general-return-1977-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 19:08:04 2010
Return-Path: <general-return-1977-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43842 invoked from network); 17 Aug 2010 19:08:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 19:08:04 -0000
Received: (qmail 37256 invoked by uid 500); 17 Aug 2010 19:07:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36912 invoked by uid 500); 17 Aug 2010 19:07:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36884 invoked by uid 99); 17 Aug 2010 19:07:58 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 19:07:58 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [85.214.115.60] (HELO h1349259.stratoserver.net) (85.214.115.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 19:07:34 +0000
Received: from g225064220.adsl.alicedsl.de ([92.225.64.220] helo=motokokusanagi.localnet)
	by h1349259.stratoserver.net with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
	(Exim 4.69)
	(envelope-from <isabel@apache.org>)
	id 1OlRVB-0003g3-PU; Tue, 17 Aug 2010 19:07:13 +0000
From: Isabel Drost <isabel@apache.org>
Reply-To: isabel@apache.org
To: community@apache.org
Subject: Apache Dinner DUS (co-located with FSFE fellowship meetup)
Date: Tue, 17 Aug 2010 21:06:58 +0200
User-Agent: KMail/1.13.5 (Linux/2.6.32-5-amd64; KDE/4.4.5; x86_64; ; )
Cc: java-user@lucene.apache.org,
 user@mahout.apache.org,
 general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/signed;
  boundary="nextPart3207924.hMO8L3MZZH";
  protocol="application/pgp-signature";
  micalg=pgp-sha1
Content-Transfer-Encoding: 7bit
Message-Id: <201008172107.08117.isabel@apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

--nextPart3207924.hMO8L3MZZH
Content-Type: Text/Plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


Hello,

the evening after FrOSCon - that is on August 22nd 2010 at 7:30p.m. CEST - =
a=20
combined "FSFE Fellowship meetup/ Apache dinner*" takes place in Tigges in=
=20
D=FCsseldorf (Brunnenstra=DFe 1, at Bilker S-Bahnhof). Given it doesn't rai=
n, we'll=20
be sitting outside.

Would be great to meet you there for tasty food, interesting discussions on=
=20
Apache in general, as well as projects like Lucene, Hadoop or Tomcat in=20
particular. Anyone interested in either the FSFE or Apache is welcome to jo=
in=20
us.

One personal request: Somehow, Rainer (Kersten, FSFE) talked me into prepar=
ing a=20
talk on what the ASF is all about - would be really great to have more peop=
le=20
around share their experience.


See you in D=FCsseldorf,
Isabel=20


* http://blog.isabel-drost.de/index.php/archives/category/events/apache-din=
ner-
berlin

--nextPart3207924.hMO8L3MZZH
Content-Type: application/pgp-signature; name=signature.asc 
Content-Description: This is a digitally signed message part.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEABECAAYFAkxq3dMACgkQPJpCBAwIhbTtLACgrGdZpWH6P5mYBJtsxwDSpiD/
B2gAnRbm4TVlceARdnUU/FfX6j5irkXk
=HqWO
-----END PGP SIGNATURE-----

--nextPart3207924.hMO8L3MZZH--

From general-return-1978-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 19:39:40 2010
Return-Path: <general-return-1978-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63978 invoked from network); 17 Aug 2010 19:39:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 19:39:40 -0000
Received: (qmail 86145 invoked by uid 500); 17 Aug 2010 19:39:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86045 invoked by uid 500); 17 Aug 2010 19:39:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86035 invoked by uid 99); 17 Aug 2010 19:39:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 19:39:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 17 Aug 2010 19:39:38 +0000
Received: (qmail 63914 invoked by uid 99); 17 Aug 2010 19:39:17 -0000
Received: from localhost.apache.org (HELO mail-gy0-f176.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 19:39:17 +0000
Received: by gyb13 with SMTP id 13so3756933gyb.35
        for <general@hadoop.apache.org>; Tue, 17 Aug 2010 12:39:16 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.174.65 with SMTP id s1mr8109815ibz.3.1282073956500; Tue,
 17 Aug 2010 12:39:16 -0700 (PDT)
Received: by 10.231.146.206 with HTTP; Tue, 17 Aug 2010 12:39:16 -0700 (PDT)
In-Reply-To: <4C6AB94F.8090405@apache.org>
References: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com>
	<4C6AB94F.8090405@apache.org>
Date: Tue, 17 Aug 2010 12:39:16 -0700
Message-ID: <AANLkTikyc64Egzbz_bRneUVEW82KTN6NPJRcMv=Q-gJh@mail.gmail.com>
Subject: Re: [VOTE] Combine MapReduce/HDFS Committers
From: "Owen O'Malley" <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8

-1

I think it is clear that separate communities are developing for HDFS
and MapReduce (unlike Common) and will further develop if we allow
them to. There will continue to be significant overlap between those
communities, but I think it is a useful distinction and matches the
way that users think about the projects.

-- Owen

From general-return-1979-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 21:36:37 2010
Return-Path: <general-return-1979-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14089 invoked from network); 17 Aug 2010 21:36:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 21:36:36 -0000
Received: (qmail 45821 invoked by uid 500); 17 Aug 2010 21:36:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45673 invoked by uid 500); 17 Aug 2010 21:36:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45665 invoked by uid 99); 17 Aug 2010 21:36:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 21:36:34 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 21:36:29 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout2.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7HLZbx9088992
	for <general@hadoop.apache.org>; Tue, 17 Aug 2010 14:35:37 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=jm96TRIyA9XoCmpN79xVmTfNs5syOY2TRt5Ee7go8jSelYu4jg+rrkXSwHN2tXyA
Message-Id: <DC5425AB-C9AF-4842-AB9A-818FCE3A4D85@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <AANLkTikyc64Egzbz_bRneUVEW82KTN6NPJRcMv=Q-gJh@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Combine MapReduce/HDFS Committers
Date: Tue, 17 Aug 2010 14:35:37 -0700
References: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com> <4C6AB94F.8090405@apache.org> <AANLkTikyc64Egzbz_bRneUVEW82KTN6NPJRcMv=Q-gJh@mail.gmail.com>
X-Mailer: Apple Mail (2.936)

-1

I agree, we need to allow HDFS and Map-Reduce to further develop into  
coherent, independent communities. This seems like a step backwards.

Arun

On Aug 17, 2010, at 12:39 PM, Owen O'Malley wrote:

> -1
>
> I think it is clear that separate communities are developing for HDFS
> and MapReduce (unlike Common) and will further develop if we allow
> them to. There will continue to be significant overlap between those
> communities, but I think it is a useful distinction and matches the
> way that users think about the projects.
>
> -- Owen


From general-return-1980-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 22:05:00 2010
Return-Path: <general-return-1980-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22309 invoked from network); 17 Aug 2010 22:04:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 22:04:59 -0000
Received: (qmail 79986 invoked by uid 500); 17 Aug 2010 22:04:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79927 invoked by uid 500); 17 Aug 2010 22:04:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 79919 invoked by uid 99); 17 Aug 2010 22:04:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 22:04:57 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.100 as permitted sender)
Received: from [17.148.16.100] (HELO asmtpout025.mac.com) (17.148.16.100)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 22:04:48 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; charset=us-ascii
Received: from [172.10.20.2]
 (166-205-142-080.mobile.mymmode.com [166.205.142.80])
 by asmtp025.mac.com (Sun Java(tm) System Messaging Server 6.3-8.01 (built Dec
 16 2008; 32bit)) with ESMTPSA id <0L7B00AFKHB4ZK70@asmtp025.mac.com> for
 general@hadoop.apache.org; Tue, 17 Aug 2010 15:04:19 -0700 (PDT)
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0
 reason=mlx engine=6.0.2-1004200000 definitions=main-1008170169
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.0.10011,1.0.148,0.0.0000
 definitions=2010-08-17_11:2010-08-17,2010-08-17,1970-01-01 signatures=0
Subject: Re: [VOTE] Combine MapReduce/HDFS Committers
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com>
Date: Tue, 17 Aug 2010 15:04:15 -0700
Message-id: <6ED46D65-0B00-4F9A-BC11-4FB4BA206466@mac.com>
References: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1081)

+1

On Aug 17, 2010, at 3:15 AM, Chris Douglas wrote:

> Per the discussion thread: http://s.apache.org/XkY
> 
> Should HDFS and MapReduce committers lists be combined and all
> subsequent committers on either of these two projects be granted karma
> in the other?
> 
> If the vote passes, current and future committers to MapReduce and
> HDFS will gain commit rights in both projects. Commit rights to Common
> are unaffected.
> 
> Without bylaws, a 2/3 majority for a committer import seems like a
> reasonable bar, given that adding an individual committer requires
> consensus.
> 
> ----
> 
> Owen has started a separate voting thread, proposing to define the
> Common committer list as the union of HDFS and MapReduce committers
> (vote A), so I tried to write this (vote B) so it would not conflict.
> As I'm reading it:
> 
> A passes, B passes: One can become a committer on HDFS or MapReduce.
> Commit to either implies commit on HDFS, MR, and Common.
> A passes, B fails: One can become a committer on HDFS or MapReduce.
> Commit to either implies commit on Common, only.
> A fails, B passes: One can become a committer on HDFS, MapReduce, or
> Common. Commit to to HDFS/MR implies converse, but individual
> appointments to Common continue.
> A fails, B fails: Committers continue to be appointed individually to
> HDFS, MapReduce, and Common.
> 
> In no scheme would commit rights to Common imply commit rights to
> either HDFS or MapReduce, I guess. -C


From general-return-1981-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 22:45:16 2010
Return-Path: <general-return-1981-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44325 invoked from network); 17 Aug 2010 22:45:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 22:45:16 -0000
Received: (qmail 37770 invoked by uid 500); 17 Aug 2010 22:45:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37592 invoked by uid 500); 17 Aug 2010 22:45:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37584 invoked by uid 99); 17 Aug 2010 22:45:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 22:45:14 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of crystaldoll06@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 22:45:07 +0000
Received: by wyb35 with SMTP id 35so10230650wyb.35
        for <general@hadoop.apache.org>; Tue, 17 Aug 2010 15:44:47 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=742xF8SSb2q8q4C6VomKZocNTpa3L/KnNm/hljE4I9g=;
        b=pjPgPQtu3wuvWxvnOz4LD268YZoLFPNKU7hoMraWett6+d4iQHmr3FWMuxmUH3occj
         Ue4uYOINpcLIfIKEnJioy/jsECnvjqzmXTmrqXOPDaZye8tJC9eimy18zXibAkcat3Nw
         XWsQ/ikseGEFaT+SfYpPHq3Oid6KI7XKMb2b4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=bfewY3alN/QtoRbBHWKl1D1r5iVaj49m/4OftJdYJpQujEXQpxJtLKRY997YID/7Dr
         fKyy9m2etMSxr7HRUDapXC8+ktSUlB2VklKVr2FqylNIGsgf4iw6G9h68Lh8IL2lypkT
         aKDek0Z97lwLZ3MV6LwmB/dvgBEfW9JoF4VI8=
MIME-Version: 1.0
Received: by 10.216.150.195 with SMTP id z45mr1327564wej.63.1282085087427;
 Tue, 17 Aug 2010 15:44:47 -0700 (PDT)
Received: by 10.216.237.90 with HTTP; Tue, 17 Aug 2010 15:44:47 -0700 (PDT)
In-Reply-To: <4C68DA5E.8080504@verchaska.com>
References: <AANLkTimTNY_EU8Eu4ZG3zNi78wOQNCwe0piJYuu_0DQU@mail.gmail.com>
	<4C677028.8000502@gmail.com>
	<AANLkTi=ys3g3VUg+o=NvxCOWo_exPBa-jwNHyJ0671_F@mail.gmail.com>
	<4C67757B.8050108@gmail.com>
	<AANLkTinEG-uLPonp8GVw5c_s-ZYhzeo+3wZ2Bv7g2q+G@mail.gmail.com>
	<4C67798F.4010701@gmail.com>
	<AANLkTimgrbKdyPPPnDyrcsPH5taioVV9yyqDhHK_2Ors@mail.gmail.com>
	<4C68DA5E.8080504@verchaska.com>
Date: Tue, 17 Aug 2010 15:44:47 -0700
Message-ID: <AANLkTinT4HB8v32V_V0pOkkN7kx+x=hiMVDYr3ja0iVF@mail.gmail.com>
Subject: Re: Hadoop basics
From: Rita Liu <crystaldoll06@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Piyush and Amit:

Thanks so much for your kind suggestions!! I am trying log4j now :))
Here is a beginner-level question -- where should I put the loggers?

Say I start with a MapReduce application (say, WordCount.java), and I
want to trace the code so that I could know which methods (from which
classes) have been called and what have been done while they are being
called, before hadoop finishes executing the application. In order to
write log files to record those information, I have to know where
(i.e. in which files) to put my loggers. However, without knowing
which methods (from which classes) are called, how do I know where to
put the loggers? If I just put my logger inside the main method of
WordCount.java, it probably doesn't make too much sense ...

Is there any way to trace the call stack so that I would know where to
put my loggers (with log4j)? Or: there might be a smart way for me to
create a logger so that I would get what I need?

Also -- although I know debugging in a distributed system could be a
pain, I wonder if I could just load the whole hadoop project into
Eclipse, say, and trace the code locally without actually running the
application on the cluster. There are many libs and consequent
dependencies in hadoop -- how may I load hadoop so that I could
locally trace it?

If possible, please help me out ... Piyush, Amit, and all the experts
here? Thank you very much!

Best,
Rita :))


On Sun, Aug 15, 2010 at 11:27 PM, amit kumar verma <v.amit@verchaska.com> w=
rote:
>
> =A0Hi Rita,
>
> If you reached a place where you need to use api like hahoop, forget abou=
t the debugging the code. Your code must be syntactically and logically err=
or free, for rest of the things logging is enough. Try log4j only.
>
> Thanks,
> Amit Kumar Verma
> Verchaska Infotech Pvt. Ltd.
>
>
>
> On 08/15/2010 11:10 AM, Rita Liu wrote:
>>
>> Hi Harsh and Piyush! Thank you very much. So it seems like it would be b=
est
>> if I use log4j to trace, and debugging with a debugger is still possible=
 if
>> I set "mapred.job.tracker" to be "local" and "fs.default.name" to be
>> "local", in hadoop-site.xml. Plus: in hadoop-env.sh, I should specify
>> HADOOP_OPTS to be:
>>
>> "-agentlib:jdwp=3Dtransport=3Ddt_socket,server=3Dy,suspend=3Dy,address=
=3D8000" (why
>> 8000? also, what does "-agentlib:jdwp=3Dtransport=3Ddt_socket" mean?)
>>
>> ... in order to use a debugger. Is my understanding correct? :)
>>
>> If so -- then which debugger do you use? May I know? Thanks a lot! I am =
also
>> going to try log4j now!
>>
>> Many thanks,
>> -Rita :))
>>
>> On Sat, Aug 14, 2010 at 10:22 PM, Piyush Garg<piyushgarg80@gmail.com>wro=
te:
>>
>>> Hi Smith,
>>>
>>> step debugging also works in hadoop as with other java applications.
>>> export
>>>
>>> HADOOP_OPTS=3D"-agentlib:jdwp=3Dtransport=3Ddt_socket,server=3Dy,suspen=
d=3Dy,address=3D8000"
>>> 'suspend=3Dy' is to let the jvm suspend until the remote debugger is
>>> attached.
>>>
>>> Thanks and Regards
>>> Piyush Garg
>>>
>>>
>>> On Sunday 15 August 2010 10:39 AM, smith jack wrote:
>>>>
>>>> that means you can only trace by log,
>>>> and not possible to debug hadoop using step debug, haha
>>>> distributed system always introduce extra complexity and confusing
>>>
>>> issues.
>>>>
>>>> 2010/8/15 Piyush Garg<piyushgarg80@gmail.com>:
>>>>
>>>>> Hi Rita,
>>>>>
>>>>> You can put log4j logger debug statements in the code. log4j library =
is
>>>>> part of hadoop framework and there is already a log4j.properties file=
 in
>>>>> hadoop conf directory and all the output logs are saved in hadoop log=
s
>>>>> directory.
>>>>>
>>>>> Thanks and Regards
>>>>> Piyush Garg
>>>>>
>>>>>
>>>>> On Sunday 15 August 2010 10:20 AM, Rita Liu wrote:
>>>>>
>>>>>> Thank you very much, Piyush! :) May I know more about how to use
>>>
>>> "traces"?
>>>>>>
>>>>>> And -- yes, please teach me if possible, experts! :)
>>>>>>
>>>>>> Thanks a lot,
>>>>>> -Rita :))
>>>>>>
>>>>>> On Sat, Aug 14, 2010 at 9:42 PM, Piyush Garg<piyushgarg80@gmail.com>
>>>
>>> wrote:
>>>>>>
>>>>>>
>>>>>>> Hi Rita,
>>>>>>>
>>>>>>> I have just started to learn hadoop as well, I know there is a long
>>>
>>> way
>>>>>>>
>>>>>>> to go.
>>>>>>> I found some useful links which I am sharing with you.
>>>>>>>
>>>>>>> Hadoop Tutorial - YDN
>>>>>>> <http://developer.yahoo.com/hadoop/tutorial/index.html> =A0excellen=
t
>>>>>>> beginners tutorial and well organized.
>>>>>>> Running Hadoop On Ubuntu Linux (Single-Node Cluster) - Michael G. N=
oll
>>>>>>> <
>>>>>>>
>>> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Sing=
le-Node_Cluster%29
>>>>>>>
>>>>>>> Running_Hadoop_On_Ubuntu_Linux_(Multi-Node_Cluster)
>>>>>>> <
>>>>>>>
>>> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Mult=
i-Node_Cluster%29
>>>>>>>
>>>>>>> The tutorial on the hadoop wiki
>>>>>>> <http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html>
>>>
>>> is
>>>>>>>
>>>>>>> too much for a beginner.
>>>>>>>
>>>>>>> Debugger:
>>>>>>> I do not think you can easily do debugging using remote debugger. T=
his
>>>>>>> is natural since hadoop is not sequential programming, it would be
>>>
>>> very
>>>>>>>
>>>>>>> difficult to debug its apps.
>>>>>>> The only way to debug is to use traces.
>>>>>>>
>>>>>>> I think you can learn how to setup multi-node cluster, but for
>>>
>>> practice
>>>>>>>
>>>>>>> session you can use single node setup.
>>>>>>>
>>>>>>> Lets see what the experts say.
>>>>>>>
>>>>>>> Thanks and Regards
>>>>>>> Piyush Garg
>>>>>>>
>>>>>>>
>>>>>>> On Sunday 15 August 2010 09:07 AM, Rita Liu wrote:
>>>>>>>
>>>>>>>
>>>>>>>> Hi!
>>>>>>>>
>>>>>>>> I am a total beginner, but I am very interested in hadoop. I've
>>>
>>> already
>>>>>>>>
>>>>>>>> downloaded hadoop 0.19.2 and run on Ubuntu in single-node mode. No=
w I
>>>>>>>>
>>>>>>>>
>>>>>>> want
>>>>>>>
>>>>>>>
>>>>>>>> to do two things:
>>>>>>>>
>>>>>>>> 1. Explore how hadoop works internally with one of the example
>>>>>>>>
>>>>>>>>
>>>>>>> applications
>>>>>>>
>>>>>>>
>>>>>>>> hadoop provides
>>>>>>>> 2. Write an application on my own
>>>>>>>>
>>>>>>>> Those two things bring me following questions:
>>>>>>>>
>>>>>>>> a. debugger?
>>>>>>>> I am stuck since I don't know how to "explore" hadoop. I used to
>>>
>>> trace
>>>>>>>>
>>>>>>>> through the code using a debugger, but in this case, I don't know =
if
>>>>>>>>
>>>>>>>>
>>>>>>> there
>>>>>>>
>>>>>>>
>>>>>>>> is a good debugger to use; or -- maybe a debugger is not necessary
>>>
>>> for
>>>>>>>>
>>>>>>>> hadoop? If not, then how do you trace through the code to either
>>>
>>> debug or
>>>>>>>>
>>>>>>>> just gain an understanding about the system? May I know what you,
>>>>>>>> experienced experts, do? :)
>>>>>>>>
>>>>>>>> b. Where to run hadoop?
>>>>>>>> Also -- may I know where you run your hadoop? Do you run on linux,=
 or
>>>
>>> on
>>>>>>>>
>>>>>>> VM
>>>>>>>
>>>>>>>
>>>>>>>> -- in particular, Cloudera? I heard that Cloudera is good for writ=
ing
>>>>>>>> mapreduce applications with hadoop itself as a blackbox; is it tru=
e?
>>>
>>> If
>>>>>>>>
>>>>>>> my
>>>>>>>
>>>>>>>
>>>>>>>> ultimate goal is to understand how hadoop works internally, would =
it
>>>
>>> be
>>>>>>>>
>>>>>>>> better if I directly run it on linux?
>>>>>>>>
>>>>>>>> c. Single-node or multi-node?
>>>>>>>> In the beginning (just like my case :p) would it be better to use
>>>>>>>> single-node or multi-node? If the latter is true, should I obtain
>>>
>>> more
>>>>>>>>
>>>>>>>> machines, or should I use more virtual machines to create more nod=
es?
>>>>>>>>
>>>>>>>> As a newbie, I am sorry for all those basic (and silly, I know :$)
>>>>>>>> questions. If possible, please help me out? Any suggestion or advi=
ce
>>>
>>> will
>>>>>>>>
>>>>>>> be
>>>>>>>
>>>>>>>
>>>>>>>> greatly appreciated. Thank you very much!
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Rita :)
>>>>>>>>
>>>>>>>> P.S. If my questions are not suitable for this mailing-list, pleas=
e
>>>
>>> let
>>>>>>>>
>>>>>>> me
>>>>>>>
>>>>>>>
>>>>>>>> apologize, and then, could you please direct me to other
>>>
>>> mailing-lists?
>>>>>>>>
>>>>>>>> Sorry, and thanks a lot! :)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>

From general-return-1982-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 04:53:30 2010
Return-Path: <general-return-1982-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88886 invoked from network); 18 Aug 2010 04:53:29 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 04:53:29 -0000
Received: (qmail 98985 invoked by uid 500); 18 Aug 2010 04:53:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98483 invoked by uid 500); 18 Aug 2010 04:53:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98475 invoked by uid 99); 18 Aug 2010 04:53:24 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 04:53:24 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yhemanth@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 04:53:01 +0000
Received: by vws4 with SMTP id 4so255394vws.35
        for <general@hadoop.apache.org>; Tue, 17 Aug 2010 21:52:40 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=wn6yxBx7/J5ykqgev2rMGQw9xgR5t0oJctd0eHMwyy0=;
        b=JijLVVkLJln8iyQ4i29B83syq1gDA1S1VlbdcxH1nioa9H278qI4T6UfPvKQzgrN3m
         4yWP1mIfd9XB02sNx5Vk9Y5ePS7AWab9ZvWOVN0XVW7W2Qo/DtC1bEVkSS4mNbV0XGIu
         Ipfo46Ll5IlKcvwBrA53dEwrWK3iYMytYpWvY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=lJxw15PVtW7oS4JjNzpKeV2sebAJHB3ySf+MHIfHNzPLB1CVzzSOS1VTsJhFNv6HUL
         BXpIY7GKxMs6gXu2TNjRZBdqJMkgCz2at5fHiy0xG+1d7Nu6vmQ5jYJ1hbeou0emwipY
         N26y5Sfrpfprdc5lmX8h8Vk8Xij6mfc08DtvM=
MIME-Version: 1.0
Received: by 10.220.121.210 with SMTP id i18mr4444629vcr.8.1282107160812; Tue,
 17 Aug 2010 21:52:40 -0700 (PDT)
Received: by 10.220.178.135 with HTTP; Tue, 17 Aug 2010 21:52:40 -0700 (PDT)
In-Reply-To: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com>
References: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com>
Date: Wed, 18 Aug 2010 10:22:40 +0530
Message-ID: <AANLkTimvxO=UzN7+b3KK4a8dZo-+khHprwcxqWK6wArA@mail.gmail.com>
Subject: Re: [VOTE] Combine MapReduce/HDFS Committers
From: Hemanth Yamijala <yhemanth@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

-1. After thinking through, I do feel that there is and will be a
trend where more folks will focus on only one of the two projects. I
certainly know of several Map/Reduce committers who are not active on
HDFS and vice-versa, and therefore couldn't qualify to have commit
rights on both projects.

Thanks
Hemanth

On Tue, Aug 17, 2010 at 3:45 PM, Chris Douglas <cdouglas@apache.org> wrote:
> Per the discussion thread: http://s.apache.org/XkY
>
> Should HDFS and MapReduce committers lists be combined and all
> subsequent committers on either of these two projects be granted karma
> in the other?
>
> If the vote passes, current and future committers to MapReduce and
> HDFS will gain commit rights in both projects. Commit rights to Common
> are unaffected.
>
> Without bylaws, a 2/3 majority for a committer import seems like a
> reasonable bar, given that adding an individual committer requires
> consensus.
>
> ----
>
> Owen has started a separate voting thread, proposing to define the
> Common committer list as the union of HDFS and MapReduce committers
> (vote A), so I tried to write this (vote B) so it would not conflict.
> As I'm reading it:
>
> A passes, B passes: One can become a committer on HDFS or MapReduce.
> Commit to either implies commit on HDFS, MR, and Common.
> A passes, B fails: One can become a committer on HDFS or MapReduce.
> Commit to either implies commit on Common, only.
> A fails, B passes: One can become a committer on HDFS, MapReduce, or
> Common. Commit to to HDFS/MR implies converse, but individual
> appointments to Common continue.
> A fails, B fails: Committers continue to be appointed individually to
> HDFS, MapReduce, and Common.
>
> In no scheme would commit rights to Common imply commit rights to
> either HDFS or MapReduce, I guess. -C
>

From general-return-1983-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 13:43:15 2010
Return-Path: <general-return-1983-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 20818 invoked from network); 18 Aug 2010 13:43:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 13:43:14 -0000
Received: (qmail 495 invoked by uid 500); 18 Aug 2010 13:43:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99619 invoked by uid 500); 18 Aug 2010 13:43:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 99601 invoked by uid 99); 18 Aug 2010 13:43:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 13:43:08 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 13:42:57 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 417C8B7DCA
	for <general@hadoop.apache.org>; Wed, 18 Aug 2010 14:42:37 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id NyUcomHUmsG2 for <general@hadoop.apache.org>;
	Wed, 18 Aug 2010 14:42:36 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 8BCCAB7DC2
	for <general@hadoop.apache.org>; Wed, 18 Aug 2010 14:42:36 +0100 (BST)
MailScanner-NULL-Check: 1282743744.06828@t+vKB6KzaEAvuDT+dYaFiw
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o7IDgNew000346
	for <general@hadoop.apache.org>; Wed, 18 Aug 2010 14:42:23 +0100 (BST)
Message-ID: <4C6BE33F.6000104@apache.org>
Date: Wed, 18 Aug 2010 14:42:23 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.11) Gecko/20100713 Thunderbird/3.0.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Combine MapReduce/HDFS Committers
References: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com>
In-Reply-To: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com>
Content-Type: text/plain; charset=GB2312
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o7IDgNew000346
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 17/08/10 11:15, Chris Douglas wrote:
> Per the discussion thread: http://s.apache.org/XkY
> 
> Should HDFS and MapReduce committers lists be combined and all
> subsequent committers on either of these two projects be granted karma
> in the other?

0

I don't know.

A big driver for the split was the volume of emails, for a dev you had
to track lots of jira issues across the board. Having the split has been
bad for me in that I've got lazy about tracking everything, which is
what you need to do to understand what's going on. Question is -how many
people need to understand all that's going on? And of those people who
are trying to do this, how do they get time for anything else?

I do think having some separate user groups where hdfs and mapreduce
issues can be discussed is good, so some separation there does seem to
help, though it's not clear for new users which list to go to.

-steve

From general-return-1984-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 15:36:09 2010
Return-Path: <general-return-1984-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87674 invoked from network); 18 Aug 2010 15:36:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 15:36:08 -0000
Received: (qmail 39886 invoked by uid 500); 18 Aug 2010 15:36:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39744 invoked by uid 500); 18 Aug 2010 15:36:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39736 invoked by uid 99); 18 Aug 2010 15:36:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 15:36:06 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [68.142.206.147] (HELO n20.bullet.mail.mud.yahoo.com) (68.142.206.147)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 18 Aug 2010 15:35:57 +0000
Received: from [68.142.194.244] by n20.bullet.mail.mud.yahoo.com with NNFMP; 18 Aug 2010 15:35:33 -0000
Received: from [66.196.97.133] by t2.bullet.mud.yahoo.com with NNFMP; 18 Aug 2010 15:32:34 -0000
Received: from [127.0.0.1] by omp106.mail.re3.yahoo.com with NNFMP; 18 Aug 2010 15:31:00 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 355642.23043.bm@omp106.mail.re3.yahoo.com
Received: (qmail 19305 invoked by uid 60001); 18 Aug 2010 15:31:00 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1282145460; bh=MDOIX0v9EKeoU4tUOPI16wO93GZsFhGpxC+A7zLT2O4=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=q7sahUiGhlhkhC7sKvNS8UcShN5J5pZ/eIzJc1VMgbwVGrBpDLjPIdpaDCbKb5pZ5LWa2a8YOuZkso5YP2/d8RNFeCzNiRi9py0DXYW3HzgGksS2CQ52qoJTXBOQ7xuGjO9rK99vOd0bawNRkF97IcxEGspmB9Li7r2jHCUpQwE=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=E9wVrwqXGjerxeuGGwUOgVPnvrMnOjgaw4gHapQjrohGiddDRSZp9Ul/wWj5OrjC3XUCGTzQNJFddypx3HVmBpIHQdMcRmtPsQGET11SqGqO7CwzZ0letNQWGJE25WUyBissOQMQTAhKu/rWwWvDml+a3ehas6g2HgtSEyeGUTE=;
Message-ID: <264147.15518.qm@web56202.mail.re3.yahoo.com>
X-YMail-OSG: zej5Q8gVM1mKt2XH_fbDLGj88SSgDvTm8HEgawgf3qjCFIm
 bzaYGCB2wtDMY3GE2ShbHMxLIl3gDQnDqo0ogLYBP_tRUSHVHEPVpxpcu7Xm
 EELCNPfBgAFPYERttLQMrZK3noZpfRRdsiDIRCydYILapjLdfT4uDM54Xk6w
 nEfWxyj5SezOB5hFcr..yPSBUU1LRrRQKr4c05imXArNuydj5WdR7ZXpdsLp
 QziavWcrAviopmH25rRKid5aiaJhNR_ub9pN41H7O7NqN.uh55SdCTBim5zg
 oC6bCTDh092J69aaFYMv7AzJV.OGZo6v4.gbUUxXomTqi0wVkfRZomLQluVP
 q6iNrFX60VlzRZcWlijtjlbtiJNB54uodCyI-
Received: from [209.131.62.113] by web56202.mail.re3.yahoo.com via HTTP; Wed, 18 Aug 2010 08:31:00 PDT
X-Mailer: YahooMailRC/470 YahooMailWebService/0.8.105.279950
References: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com>
Date: Wed, 18 Aug 2010 08:31:00 -0700 (PDT)
From: "Tsz Wo \(Nicholas\), Sze" <s29752-hadoopgeneral@yahoo.com>
Subject: Re: [VOTE] Combine MapReduce/HDFS Committers
To: general@hadoop.apache.org
In-Reply-To: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii

-1

Most committers work on either HDFS or MapReduce in general.  For the cases that 
contributors may provide patches on both projects, those people will gain 
committer accesses on both projects.  For the occasional cases that some issues 
may require patches to be committed on both projects, it totally makes sense to 
find a committer on each project reviewing and committing those patches.

Nicholas



----- Original Message ----
> From: Chris Douglas <cdouglas@apache.org>
> To: general@hadoop.apache.org
> Sent: Tue, August 17, 2010 3:15:47 AM
> Subject: [VOTE] Combine MapReduce/HDFS Committers
> 
> Per the discussion thread: http://s.apache.org/XkY
> 
> Should HDFS and  MapReduce committers lists be combined and all
> subsequent committers on  either of these two projects be granted karma
> in the other?


From general-return-1985-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 15:57:49 2010
Return-Path: <general-return-1985-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94570 invoked from network); 18 Aug 2010 15:57:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 15:57:49 -0000
Received: (qmail 82596 invoked by uid 500); 18 Aug 2010 15:57:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82484 invoked by uid 500); 18 Aug 2010 15:57:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82476 invoked by uid 99); 18 Aug 2010 15:57:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 15:57:47 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.98 as permitted sender)
Received: from [17.148.16.98] (HELO asmtpout023.mac.com) (17.148.16.98)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 15:57:39 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; charset=us-ascii
Received: from [172.10.20.5]
 (166-205-143-140.mobile.mymmode.com [166.205.143.140])
 by asmtp023.mac.com (Sun Java(tm) System Messaging Server 6.3-8.01 (built Dec
 16 2008; 32bit)) with ESMTPSA id <0L7C00L5ZUZ31Q90@asmtp023.mac.com> for
 general@hadoop.apache.org; Wed, 18 Aug 2010 08:57:06 -0700 (PDT)
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0
 reason=mlx engine=6.0.2-1004200000 definitions=main-1008180100
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.0.10011,1.0.148,0.0.0000
 definitions=2010-08-18_07:2010-08-18,2010-08-18,1970-01-01 signatures=0
Subject: Re: [VOTE] Combine MapReduce/HDFS Committers
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <264147.15518.qm@web56202.mail.re3.yahoo.com>
Date: Wed, 18 Aug 2010 08:57:04 -0700
Message-id: <2A843341-8207-40DA-96B8-192FAE05C3A2@mac.com>
References: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com>
 <264147.15518.qm@web56202.mail.re3.yahoo.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1081)

I have a technical question if this vote fails.  Since we still release MR and HDFS together as Hadoop, does this mean that only committers who can commit to both projects can then make a release candidate of Hadoop?  Do we end up with MR committers, HDFS committers, and Hadoop committers?

Nige

On Aug 18, 2010, at 8:31 AM, Tsz Wo (Nicholas), Sze wrote:

> -1
> 
> Most committers work on either HDFS or MapReduce in general.  For the cases that 
> contributors may provide patches on both projects, those people will gain 
> committer accesses on both projects.  For the occasional cases that some issues 
> may require patches to be committed on both projects, it totally makes sense to 
> find a committer on each project reviewing and committing those patches.
> 
> Nicholas
> 
> 
> 
> ----- Original Message ----
>> From: Chris Douglas <cdouglas@apache.org>
>> To: general@hadoop.apache.org
>> Sent: Tue, August 17, 2010 3:15:47 AM
>> Subject: [VOTE] Combine MapReduce/HDFS Committers
>> 
>> Per the discussion thread: http://s.apache.org/XkY
>> 
>> Should HDFS and  MapReduce committers lists be combined and all
>> subsequent committers on  either of these two projects be granted karma
>> in the other?
> 


From general-return-1986-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 18:01:08 2010
Return-Path: <general-return-1986-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 65605 invoked from network); 18 Aug 2010 18:01:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 18:01:07 -0000
Received: (qmail 55220 invoked by uid 500); 18 Aug 2010 18:01:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55130 invoked by uid 500); 18 Aug 2010 18:01:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55122 invoked by uid 99); 18 Aug 2010 18:01:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 18:01:05 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 18:01:00 +0000
Received: from [127.0.0.1] (gentlepaint-lx.corp.yahoo.com [10.72.185.127])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7IHxDQa031017
	for <general@hadoop.apache.org>; Wed, 18 Aug 2010 10:59:19 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=xq85z/MNwv4ND8Ud59+bvnbDKub2b0YphA/ebbmaZZfO+MGJrG67ISwoK3VTEgYk
Message-ID: <4C6C1F70.4000300@yahoo-inc.com>
Date: Wed, 18 Aug 2010 10:59:12 -0700
From: Konstantin Shvachko <shv@yahoo-inc.com>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.8) Gecko/20100802 Lightning/1.0b2 Thunderbird/3.1.2
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Should we define the Common committers as HDFS + MapReduce
 committers?
References: <7962BB4E-7176-44EB-A708-64881639E2AF@apache.org>
In-Reply-To: <7962BB4E-7176-44EB-A708-64881639E2AF@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

+1

Do I understand correctly, with this proposal there will be no
common only committers? That is, there will be no committers to
common who are not either HDFS or MR committers.

--Konstantin

On 8/16/2010 2:38 PM, Owen O'Malley wrote:
> Since the project split, it is mostly required that any MapReduce or
> HDFS committers have access to change Common. We haven't had any cases
> where we wanted to nominate any one for just Common. Toward the goal
> of simplifying the structure, I'd propose that we define the Common
> committers as precisely the union of the HDFS committers and MapReduce
> committers.
>
> Clearly, I'm +1.
>
> -- Owen
>


From general-return-1987-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 18:19:58 2010
Return-Path: <general-return-1987-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76350 invoked from network); 18 Aug 2010 18:19:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 18:19:57 -0000
Received: (qmail 87321 invoked by uid 500); 18 Aug 2010 18:19:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87195 invoked by uid 500); 18 Aug 2010 18:19:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87182 invoked by uid 99); 18 Aug 2010 18:19:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 18:19:55 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.62.96] (HELO qmta09.westchester.pa.mail.comcast.net) (76.96.62.96)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 18:19:47 +0000
Received: from omta16.westchester.pa.mail.comcast.net ([76.96.62.88])
	by qmta09.westchester.pa.mail.comcast.net with comcast
	id voAh1e0021uE5Es59uKTWV; Wed, 18 Aug 2010 18:19:27 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta16.westchester.pa.mail.comcast.net with comcast
	id vuKH1e00D2SbwD53cuKL3V; Wed, 18 Aug 2010 18:19:25 +0000
Message-Id: <62188BE8-B003-4169-A0A5-C718241FD3A0@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <4C6C1F70.4000300@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Should we define the Common committers as HDFS + MapReduce committers?
Date: Wed, 18 Aug 2010 11:19:16 -0700
References: <7962BB4E-7176-44EB-A708-64881639E2AF@apache.org> <4C6C1F70.4000300@yahoo-inc.com>
X-Mailer: Apple Mail (2.936)


On Aug 18, 2010, at 10:59 AM, Konstantin Shvachko wrote:

> +1
>
> Do I understand correctly, with this proposal there will be no
> common only committers? That is, there will be no committers to
> common who are not either HDFS or MR committers.

Correct. There haven't been any and there doesn't seem to be enough  
stand alone functionality in Common that it seems likely. (ie. a  
Commons only committer wouldn't be able to do enough to be interesting.)

-- Owen

From general-return-1988-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 18:32:45 2010
Return-Path: <general-return-1988-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91213 invoked from network); 18 Aug 2010 18:32:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 18:32:45 -0000
Received: (qmail 21472 invoked by uid 500); 18 Aug 2010 18:32:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21415 invoked by uid 500); 18 Aug 2010 18:32:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21407 invoked by uid 99); 18 Aug 2010 18:32:43 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 18:32:43 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 18:32:20 +0000
Received: from localhost ([10.72.168.47])
	by mrout1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7IIV5mQ064016
	for <general@hadoop.apache.org>; Wed, 18 Aug 2010 11:31:05 -0700 (PDT)
Date: Wed, 18 Aug 2010 11:31:05 -0700
From: Konstantin Boudnik <cos@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: [VOTE] Combine MapReduce/HDFS Committers
Message-ID: <20100818183105.GA9985@goodenter-lm.local>
References: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature"; boundary="EeQfGwPcQSOJBaQU"
Content-Disposition: inline
In-Reply-To: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com>
X-Organization: Yahoo! Hadoop (Grid computing)
User-Agent: Mutt/1.5.20 (2009-06-14)
X-Virus-Checked: Checked by ClamAV on apache.org

--EeQfGwPcQSOJBaQU
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

I tend to +1 on this because most of Hadoop committers wear both hats alrea=
dy.
Most of those who aren't should have for all I know.

The argument about enforcing projects separation by the means of splitting =
dev.
communities doesn't hold the critique IMO. True separation would come from
design independence and well defined contracts between components, not from
the perception of the policies.

Cos

On Tue, Aug 17, 2010 at 03:15AM, Chris Douglas wrote:
> Per the discussion thread: http://s.apache.org/XkY
>=20
> Should HDFS and MapReduce committers lists be combined and all
> subsequent committers on either of these two projects be granted karma
> in the other?
>=20
> If the vote passes, current and future committers to MapReduce and
> HDFS will gain commit rights in both projects. Commit rights to Common
> are unaffected.

--EeQfGwPcQSOJBaQU
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)

iF4EAREIAAYFAkxsJukACgkQMqXifkwDoaHkJAD/YyzOdi1eFd923j6cS9pVlxes
2HP4frd83MMvd6DnbvcA/RzZ5xTry5uUBRgO0H7KsfkQzOITwLIw8bNDcWiSieKa
=5KXR
-----END PGP SIGNATURE-----

--EeQfGwPcQSOJBaQU--

From general-return-1989-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 19 13:41:13 2010
Return-Path: <general-return-1989-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59267 invoked from network); 19 Aug 2010 13:41:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Aug 2010 13:41:12 -0000
Received: (qmail 58519 invoked by uid 500); 19 Aug 2010 13:41:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58041 invoked by uid 500); 19 Aug 2010 13:41:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57991 invoked by uid 99); 19 Aug 2010 13:41:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 13:41:06 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=FREEMAIL_FROM,FROM_LOCAL_NOVOWEL,HK_RANDOM_ENVFROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of sssssssenator@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 13:41:00 +0000
Received: by wwb34 with SMTP id 34so2086299wwb.29
        for <general@hadoop.apache.org>; Thu, 19 Aug 2010 06:40:39 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=OPvohPKilTlNKQ0i6xTwbkvUHtgvWJLA/Gw9JmPL9rI=;
        b=OgSrqf2SAU1qmltWqGQXHi2XJqHhRCqyf/9azznu5ybWn6E8pMVkIQmQhnc2NafbUL
         jC1b34dGe4vSHToLfrDCzW4+ge18XnSU7yVTlHn25TegID+0jKaWtAked9KW0IhUxEdJ
         z4EJs2bTFsOWgwD7dWiSjrBC3SeAoapZqD+HM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=gK/Hx/25vPexYP6TvX4m/gA7fZHXWvUAgQkD4WdLqwxVqazLiqrHZInDhj790HlYXJ
         k6aM/LZctAR5oAzBHxXgjrIHz7mjJPgIZIgUGIGMkTu5GX8j7PI/pAVE90sGYYGlZNrd
         7RFIcpFcH/s7nLoy5CzM+MDzL1dmCBgaQXpOE=
MIME-Version: 1.0
Received: by 10.227.141.77 with SMTP id l13mr8516248wbu.77.1282225237980; Thu,
 19 Aug 2010 06:40:37 -0700 (PDT)
Received: by 10.216.139.17 with HTTP; Thu, 19 Aug 2010 06:40:37 -0700 (PDT)
Date: Thu, 19 Aug 2010 19:10:37 +0530
Message-ID: <AANLkTimi+Yg_i8ELApdeBuNB+GAaVbUD2zubf0kFRzaa@mail.gmail.com>
Subject: Warning : Missing blocks
From: vaibhav negi <sssssssenator@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e659f5e6d4f957048e2d531b

--0016e659f5e6d4f957048e2d531b
Content-Type: text/plain; charset=ISO-8859-1

Hi,

I added a server to my pseudo distributed cluster. But still web viewer
shows only 1 live node and disk space is still same.
I am getting this message :-   WARNING : There are about 18 missing blocks.
Please check the log or run fsck.

Please help.

Vaibhav Negi

--0016e659f5e6d4f957048e2d531b--

From general-return-1990-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 19 18:41:42 2010
Return-Path: <general-return-1990-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47880 invoked from network); 19 Aug 2010 18:41:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Aug 2010 18:41:42 -0000
Received: (qmail 66494 invoked by uid 500); 19 Aug 2010 18:41:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66406 invoked by uid 500); 19 Aug 2010 18:41:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66398 invoked by uid 99); 19 Aug 2010 18:41:40 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 18:41:40 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 18:41:17 +0000
Received: by wwb34 with SMTP id 34so2471157wwb.29
        for <general@hadoop.apache.org>; Thu, 19 Aug 2010 11:40:57 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        bh=ZdEtOrZmc+kj69Ejgify9RHmVYEUGcU/0RJCnGsY5uI=;
        b=VGd07q5ORExf6caGEpKhjpXkPK4t0rIJ7WzIyUCGzsHZgWm7KBPyxxXnyhR7gOS2hP
         3Vbo//HsKjKPwncU9OWEX+8Moe3d+ZS5VaygAyRrzgPry2A/X6wcBxyOm2HdWtraLIG9
         F+lTCe8hvjiVvNE6SRWqp8fqcITIglt/QG/Ko=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=Bt74OiSuocLXShzRLB/SACQMkJ9FrHnootWKNL3VmhetmuX7Xj4O6+LHB+V0NJAc8j
         2s7y/FzOFl5OzF8GQxNFOHQ7YurzYWmntkDhm3A5Uv9f1Luej0gt50Up/DlZDsGwkfJv
         2EcdRGakEdVO0WgCyCvVPYkpeoFKkH0LHDMVI=
MIME-Version: 1.0
Received: by 10.227.133.148 with SMTP id f20mr233709wbt.35.1282243257226; Thu,
 19 Aug 2010 11:40:57 -0700 (PDT)
Sender: saint.ack@gmail.com
Received: by 10.216.168.73 with HTTP; Thu, 19 Aug 2010 11:40:57 -0700 (PDT)
In-Reply-To: <7962BB4E-7176-44EB-A708-64881639E2AF@apache.org>
References: <7962BB4E-7176-44EB-A708-64881639E2AF@apache.org>
Date: Thu, 19 Aug 2010 11:40:57 -0700
X-Google-Sender-Auth: YRKnS9um1C2B9-7LTbt5d3B0IL4
Message-ID: <AANLkTikCW=ZcwMAyfpBaPG_Lbp6b5bqj3oQXDOFpC4Ju@mail.gmail.com>
Subject: Re: [VOTE] Should we define the Common committers as HDFS + MapReduce committers?
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

+1

On Mon, Aug 16, 2010 at 2:38 PM, Owen O'Malley <omalley@apache.org> wrote:
> Since the project split, it is mostly required that any MapReduce or HDFS
> committers have access to change Common. We haven't had any cases where we
> wanted to nominate any one for just Common. Toward the goal of simplifying
> the structure, I'd propose that we define the Common committers as precisely
> the union of the HDFS committers and MapReduce committers.
>
> Clearly, I'm +1.
>
> -- Owen
>

From general-return-1991-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 19 19:49:38 2010
Return-Path: <general-return-1991-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 80401 invoked from network); 19 Aug 2010 19:49:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Aug 2010 19:49:37 -0000
Received: (qmail 82599 invoked by uid 500); 19 Aug 2010 19:49:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82462 invoked by uid 500); 19 Aug 2010 19:49:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82454 invoked by uid 99); 19 Aug 2010 19:49:35 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 19:49:35 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 209.85.215.176 as permitted sender)
Received: from [209.85.215.176] (HELO mail-ey0-f176.google.com) (209.85.215.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 19:49:12 +0000
Received: by eydd26 with SMTP id d26so2099124eyd.35
        for <general@hadoop.apache.org>; Thu, 19 Aug 2010 12:48:52 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        bh=gPrjar46S7c97uif0KKDDLDl2RUKsfeRXrPonzdtl18=;
        b=PnHnEqZwHOWzf0y3X984s2790lrqMu1eC3mU7Pv7GeyDd4KfK2N1lXv6wbqklpES8A
         T+tgSyt+3K2uSYPdO9fNDiIXGxCRd0ub4cBHY6WMjUx9KMOMSaMbiaUiZn6RYTr5YxMf
         RNLVsam0PyGjyRpvjRGqSVSQMfQYXKCKmVydE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=GFm3PNui9xVyXqZ7rSk79aNZpdzMLtJcOyncTLrRGSt5G30v5eDZ6YRUqfSQMeZYPr
         4ZKNjN0kBoVWUpcK5dcpwXzJlHLhg4YRaxK0kDrYCw4FBiiYJqnDOBnIAXWyndQWlHYf
         1szsZcsNXnSoLKgvzA9ehq+6acgB0PKncNzl0=
MIME-Version: 1.0
Received: by 10.216.48.146 with SMTP id v18mr245894web.56.1282247332047; Thu,
 19 Aug 2010 12:48:52 -0700 (PDT)
Sender: saint.ack@gmail.com
Received: by 10.216.168.73 with HTTP; Thu, 19 Aug 2010 12:48:51 -0700 (PDT)
In-Reply-To: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com>
References: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com>
Date: Thu, 19 Aug 2010 12:48:51 -0700
X-Google-Sender-Auth: tNOhdq6VfBPvxklG7LaHUIzlwZc
Message-ID: <AANLkTinbHs2wrzp5c46kfsAP937pfa-mLBnF6Ac3sQEJ@mail.gmail.com>
Subject: Re: [VOTE] Combine MapReduce/HDFS Committers
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

+1

IMO, a combined committer list is less messy given how we currently
release where common+mr+hdfs are all bundled and released together.

Corner cases like the one postulated by Nigel above are the less
likely if contributors have access to all hadoop.

I do not  see how a combined contributor list could act as friction on
the ongoing break-up of the hadoop project -- something I'm in favor
of -- nor get in the way of the development of distinct mr/hdfs
user+dev communities; it seems to me that that project can progress
independent of who can commit where.

St.Ack




On Tue, Aug 17, 2010 at 3:15 AM, Chris Douglas <cdouglas@apache.org> wrote:
> Per the discussion thread: http://s.apache.org/XkY
>
> Should HDFS and MapReduce committers lists be combined and all
> subsequent committers on either of these two projects be granted karma
> in the other?
>
> If the vote passes, current and future committers to MapReduce and
> HDFS will gain commit rights in both projects. Commit rights to Common
> are unaffected.
>
> Without bylaws, a 2/3 majority for a committer import seems like a
> reasonable bar, given that adding an individual committer requires
> consensus.
>
> ----
>
> Owen has started a separate voting thread, proposing to define the
> Common committer list as the union of HDFS and MapReduce committers
> (vote A), so I tried to write this (vote B) so it would not conflict.
> As I'm reading it:
>
> A passes, B passes: One can become a committer on HDFS or MapReduce.
> Commit to either implies commit on HDFS, MR, and Common.
> A passes, B fails: One can become a committer on HDFS or MapReduce.
> Commit to either implies commit on Common, only.
> A fails, B passes: One can become a committer on HDFS, MapReduce, or
> Common. Commit to to HDFS/MR implies converse, but individual
> appointments to Common continue.
> A fails, B fails: Committers continue to be appointed individually to
> HDFS, MapReduce, and Common.
>
> In no scheme would commit rights to Common imply commit rights to
> either HDFS or MapReduce, I guess. -C
>

From general-return-1992-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 19 20:03:40 2010
Return-Path: <general-return-1992-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84858 invoked from network); 19 Aug 2010 20:03:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Aug 2010 20:03:39 -0000
Received: (qmail 12519 invoked by uid 500); 19 Aug 2010 20:03:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12463 invoked by uid 500); 19 Aug 2010 20:03:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12455 invoked by uid 99); 19 Aug 2010 20:03:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 20:03:38 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 19 Aug 2010 20:03:35 +0000
Received: (qmail 84688 invoked by uid 99); 19 Aug 2010 20:03:14 -0000
Received: from localhost.apache.org (HELO [192.168.168.101]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 20:03:14 +0000
Message-ID: <4C6D8E01.6070005@apache.org>
Date: Thu, 19 Aug 2010 13:03:13 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.11) Gecko/20100713 Thunderbird/3.0.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Combine MapReduce/HDFS Committers
References: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com> <AANLkTinbHs2wrzp5c46kfsAP937pfa-mLBnF6Ac3sQEJ@mail.gmail.com>
In-Reply-To: <AANLkTinbHs2wrzp5c46kfsAP937pfa-mLBnF6Ac3sQEJ@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 08/19/2010 12:48 PM, Stack wrote:
> I do not  see how a combined contributor list could act as friction on
> the ongoing break-up of the hadoop project -- something I'm in favor
> of -- nor get in the way of the development of distinct mr/hdfs
> user+dev communities; it seems to me that that project can progress
> independent of who can commit where.

I agree.

Many projects manage such things with trust, not with rigid ACLs.  Some 
committers are trusted to commit just test code, some just 
documentation, and some deep implementation details in particular areas 
of the code.  But should any committer wish to manage a release, or even 
commit a patch outside of their normal domain of expertise (if it's been 
reviewed by someone with expertise in that domain) they can.  Over time 
they can gain trust in new areas of the project.  In such projects, 
adding a committer merely implies that you trust someone not to overstep 
their abilities.  If someone consistently suggests that patches are 
ready for commit which others feel are not, then they won't earn that 
trust and be invited to become a committer.  Thus committership is not 
about deep technical abilities, but personal trust not to violate social 
contracts.  Erecting walls within the community of trust via ACLs 
doesn't seem productive.

Doug

From general-return-1993-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 19 23:01:17 2010
Return-Path: <general-return-1993-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66629 invoked from network); 19 Aug 2010 23:01:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Aug 2010 23:01:17 -0000
Received: (qmail 57651 invoked by uid 500); 19 Aug 2010 23:01:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57575 invoked by uid 500); 19 Aug 2010 23:01:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57567 invoked by uid 99); 19 Aug 2010 23:01:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 23:01:15 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.98 as permitted sender)
Received: from [17.148.16.98] (HELO asmtpout023.mac.com) (17.148.16.98)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 23:01:06 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; charset=us-ascii
Received: from [10.0.1.3] (c-67-180-80-11.hsd1.ca.comcast.net [67.180.80.11])
 by asmtp023.mac.com
 (Sun Java(tm) System Messaging Server 6.3-8.01 (built Dec 16 2008; 32bit))
 with ESMTPSA id <0L7F00IEJ98HDC00@asmtp023.mac.com> for
 general@hadoop.apache.org; Thu, 19 Aug 2010 16:00:17 -0700 (PDT)
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0
 reason=mlx engine=6.0.2-1004200000 definitions=main-1008190199
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.0.10011,1.0.148,0.0.0000
 definitions=2010-08-19_11:2010-08-19,2010-08-19,1970-01-01 signatures=0
Subject: Re: [VOTE] Combine MapReduce/HDFS Committers
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <4C6D8E01.6070005@apache.org>
Date: Thu, 19 Aug 2010 16:00:17 -0700
Message-id: <40952204-AE73-4E5E-97CC-7BAAB3374C9D@mac.com>
References: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com>
 <AANLkTinbHs2wrzp5c46kfsAP937pfa-mLBnF6Ac3sQEJ@mail.gmail.com>
 <4C6D8E01.6070005@apache.org>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1081)
X-Virus-Checked: Checked by ClamAV on apache.org


On Aug 19, 2010, at 1:03 PM, Doug Cutting wrote:

> On 08/19/2010 12:48 PM, Stack wrote:
>> I do not  see how a combined contributor list could act as friction on
>> the ongoing break-up of the hadoop project -- something I'm in favor
>> of -- nor get in the way of the development of distinct mr/hdfs
>> user+dev communities; it seems to me that that project can progress
>> independent of who can commit where.
> 
> I agree.
> 
> Many projects manage such things with trust, not with rigid ACLs.  Some committers are trusted to commit just test code, some just documentation, and some deep implementation details in particular areas of the code.  But should any committer wish to manage a release, or even commit a patch outside of their normal domain of expertise (if it's been reviewed by someone with expertise in that domain) they can.  Over time they can gain trust in new areas of the project.  In such projects, adding a committer merely implies that you trust someone not to overstep their abilities.  If someone consistently suggests that patches are ready for commit which others feel are not, then they won't earn that trust and be invited to become a committer.  Thus committership is not about deep technical abilities, but personal trust not to violate social contracts.  Erecting walls within the community of trust via ACLs doesn't seem productive.
> 
> Doug

Agreed.  Anyone want to reconsider their vote?


From general-return-1994-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 20 00:51:15 2010
Return-Path: <general-return-1994-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12518 invoked from network); 20 Aug 2010 00:51:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Aug 2010 00:51:14 -0000
Received: (qmail 28364 invoked by uid 500); 20 Aug 2010 00:51:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28293 invoked by uid 500); 20 Aug 2010 00:51:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28285 invoked by uid 99); 20 Aug 2010 00:51:12 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 00:51:12 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 00:50:50 +0000
Received: from [127.0.0.1] (gentlepaint-lx.corp.yahoo.com [10.72.185.127])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7K0o3Dj044358
	for <general@hadoop.apache.org>; Thu, 19 Aug 2010 17:50:08 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=Og7/nO8pWf3YQ/EMdyMT1TWVOe87agW1AhCh7yKB4FrJGBDxJTgA9Durfb6PlABc
Message-ID: <4C6DD13B.1060805@yahoo-inc.com>
Date: Thu, 19 Aug 2010 17:50:03 -0700
From: Konstantin Shvachko <shv@yahoo-inc.com>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.8) Gecko/20100802 Lightning/1.0b2 Thunderbird/3.1.2
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Combine MapReduce/HDFS Committers
References: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com>
In-Reply-To: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

0

My thinking is that while MR and HDFS are the same product,
we can have joint committership.
If split happens there is no reason to have common committers,
like nobody is raising a question to merge e.g. tomcat and
subversion committers.
I think nobody does, but I don't know for sure of course.

--Konstantin

On 8/17/2010 3:15 AM, Chris Douglas wrote:
> Per the discussion thread: http://s.apache.org/XkY
>
> Should HDFS and MapReduce committers lists be combined and all
> subsequent committers on either of these two projects be granted karma
> in the other?
>
> If the vote passes, current and future committers to MapReduce and
> HDFS will gain commit rights in both projects. Commit rights to Common
> are unaffected.
>
> Without bylaws, a 2/3 majority for a committer import seems like a
> reasonable bar, given that adding an individual committer requires
> consensus.
>
> ----
>
> Owen has started a separate voting thread, proposing to define the
> Common committer list as the union of HDFS and MapReduce committers
> (vote A), so I tried to write this (vote B) so it would not conflict.
> As I'm reading it:
>
> A passes, B passes: One can become a committer on HDFS or MapReduce.
> Commit to either implies commit on HDFS, MR, and Common.
> A passes, B fails: One can become a committer on HDFS or MapReduce.
> Commit to either implies commit on Common, only.
> A fails, B passes: One can become a committer on HDFS, MapReduce, or
> Common. Commit to to HDFS/MR implies converse, but individual
> appointments to Common continue.
> A fails, B fails: Committers continue to be appointed individually to
> HDFS, MapReduce, and Common.
>
> In no scheme would commit rights to Common imply commit rights to
> either HDFS or MapReduce, I guess. -C
>


From general-return-1995-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 20 05:08:04 2010
Return-Path: <general-return-1995-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87332 invoked from network); 20 Aug 2010 05:08:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Aug 2010 05:08:04 -0000
Received: (qmail 34960 invoked by uid 500); 20 Aug 2010 05:08:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34606 invoked by uid 500); 20 Aug 2010 05:07:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34598 invoked by uid 99); 20 Aug 2010 05:07:59 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 05:07:59 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of dhruba@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 05:07:35 +0000
Received: by pwj2 with SMTP id 2so1449645pwj.35
        for <general@hadoop.apache.org>; Thu, 19 Aug 2010 22:07:14 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:from:to:in-reply-to:x-mailer
         :subject:references:message-id:content-type
         :content-transfer-encoding:mime-version:date:cc;
        bh=PBVa5iHwf64NTJgJRkXfed2bYxbJIKtl3A0YgxXjFmk=;
        b=rpsMpPKOHIosy9nDMsuwhgeYBpt1DMamUIFeJEnFYkiVdqjyZbHv+bKZ/lfwbnj0Yn
         Rdl3Nbbbi7R7inyO6MGw2dgdQuchWPV9kK2RPBkeMx19JrrMD0WCcuKGODxdMvRs+gkd
         h86yKYcRDn3GM8ueilUx04G50PkjiyT0mS/84=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=from:to:in-reply-to:x-mailer:subject:references:message-id
         :content-type:content-transfer-encoding:mime-version:date:cc;
        b=ikQrLgiOnlY8N2Ei4AfjbUp6uZkn4YSvR4Er9Jsdi4tm8NIc1A6FVn52uYXCsX+Qoy
         npV743A6AxAYYm0YLdpjC5w5pJybcp5N2POhSsp5hAS7dXTSLzYKjJ7fsze7DIzP1H/m
         8SDC5nyW4R9iUrarVF4CUUi85DAfn52qxx19A=
Received: by 10.142.108.1 with SMTP id g1mr723684wfc.108.1282280833945;
        Thu, 19 Aug 2010 22:07:13 -0700 (PDT)
Received: from [10.51.171.77] ([166.205.139.165])
        by mx.google.com with ESMTPS id w31sm2692916wfd.8.2010.08.19.22.07.10
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Thu, 19 Aug 2010 22:07:12 -0700 (PDT)
From: Dhruba Borthakur <dhruba@gmail.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <40952204-AE73-4E5E-97CC-7BAAB3374C9D@mac.com>
X-Mailer: iPhone Mail (7E18)
Subject: Re: [VOTE] Combine MapReduce/HDFS Committers
References: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com> <AANLkTinbHs2wrzp5c46kfsAP937pfa-mLBnF6Ac3sQEJ@mail.gmail.com> <4C6D8E01.6070005@apache.org> <40952204-AE73-4E5E-97CC-7BAAB3374C9D@mac.com>
Message-Id: <32A3FA21-69A1-4B2F-9F7F-DBF0473BD1B5@gmail.com>
Content-Type: text/plain;
	charset=us-ascii;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (iPhone Mail 7E18)
Date: Thu, 19 Aug 2010 22:07:02 -0700
Cc: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

For the records, I am +1 on having a single set of committers for  
hdfs , mr and common. For me, all committers do not need to be equally  
good, or have to reach a grand bar; but rather I should trust that  
person to be a team player.

Dhruba

Sent from my iPhone

On Aug 19, 2010, at 4:00 PM, Nigel Daley <ndaley@mac.com> wrote:

>
> On Aug 19, 2010, at 1:03 PM, Doug Cutting wrote:
>
>> On 08/19/2010 12:48 PM, Stack wrote:
>>> I do not  see how a combined contributor list could act as  
>>> friction on
>>> the ongoing break-up of the hadoop project -- something I'm in favor
>>> of -- nor get in the way of the development of distinct mr/hdfs
>>> user+dev communities; it seems to me that that project can progress
>>> independent of who can commit where.
>>
>> I agree.
>>
>> Many projects manage such things with trust, not with rigid ACLs.   
>> Some committers are trusted to commit just test code, some just  
>> documentation, and some deep implementation details in particular  
>> areas of the code.  But should any committer wish to manage a  
>> release, or even commit a patch outside of their normal domain of  
>> expertise (if it's been reviewed by someone with expertise in that  
>> domain) they can.  Over time they can gain trust in new areas of  
>> the project.  In such projects, adding a committer merely implies  
>> that you trust someone not to overstep their abilities.  If someone  
>> consistently suggests that patches are ready for commit which  
>> others feel are not, then they won't earn that trust and be invited  
>> to become a committer.  Thus committership is not about deep  
>> technical abilities, but personal trust not to violate social  
>> contracts.  Erecting walls within the community of trust via ACLs  
>> doesn't seem productive.
>>
>> Doug
>
> Agreed.  Anyone want to reconsider their vote?
>

From general-return-1996-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 20 21:38:58 2010
Return-Path: <general-return-1996-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 3374 invoked from network); 20 Aug 2010 21:38:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Aug 2010 21:38:57 -0000
Received: (qmail 29959 invoked by uid 500); 20 Aug 2010 21:38:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29693 invoked by uid 500); 20 Aug 2010 21:38:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29685 invoked by uid 99); 20 Aug 2010 21:38:55 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 21:38:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 20 Aug 2010 21:38:38 +0000
Received: (qmail 3250 invoked by uid 99); 20 Aug 2010 21:38:15 -0000
Received: from localhost.apache.org (HELO mail-iw0-f176.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 21:38:15 +0000
Received: by iwn9 with SMTP id 9so1236883iwn.35
        for <general@hadoop.apache.org>; Fri, 20 Aug 2010 14:38:14 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.30.193 with SMTP id v1mr2236748ibc.87.1282340294314; Fri,
 20 Aug 2010 14:38:14 -0700 (PDT)
Received: by 10.231.171.75 with HTTP; Fri, 20 Aug 2010 14:38:14 -0700 (PDT)
In-Reply-To: <32A3FA21-69A1-4B2F-9F7F-DBF0473BD1B5@gmail.com>
References: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com>
	<AANLkTinbHs2wrzp5c46kfsAP937pfa-mLBnF6Ac3sQEJ@mail.gmail.com>
	<4C6D8E01.6070005@apache.org>
	<40952204-AE73-4E5E-97CC-7BAAB3374C9D@mac.com>
	<32A3FA21-69A1-4B2F-9F7F-DBF0473BD1B5@gmail.com>
Date: Fri, 20 Aug 2010 14:38:14 -0700
Message-ID: <AANLkTimCWXJ+JzOsQkd9KVKiAsMWNDC4hbzHU=qt_k=A@mail.gmail.com>
Subject: Re: [VOTE] Combine MapReduce/HDFS Committers
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

+1: Tom, Doug, Nigel, Stack, Dhruba, (Eli)
-1: Owen, Arun, Hemanth, Nicholas
+/-0: Konstantin, (Steve)

The vote did not pass.

Thanks to everyone who voted. -C

From general-return-1997-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 20 21:43:10 2010
Return-Path: <general-return-1997-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4195 invoked from network); 20 Aug 2010 21:43:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Aug 2010 21:43:09 -0000
Received: (qmail 39897 invoked by uid 500); 20 Aug 2010 21:43:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39807 invoked by uid 500); 20 Aug 2010 21:43:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39799 invoked by uid 99); 20 Aug 2010 21:43:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 21:43:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 20 Aug 2010 21:43:07 +0000
Received: (qmail 4138 invoked by uid 99); 20 Aug 2010 21:42:47 -0000
Received: from localhost.apache.org (HELO mail-iw0-f176.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 21:42:47 +0000
Received: by iwn9 with SMTP id 9so1243023iwn.35
        for <general@hadoop.apache.org>; Fri, 20 Aug 2010 14:42:46 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.11.69 with SMTP id s5mr643914ibs.38.1282340566759; Fri, 20
 Aug 2010 14:42:46 -0700 (PDT)
Received: by 10.231.171.75 with HTTP; Fri, 20 Aug 2010 14:42:46 -0700 (PDT)
In-Reply-To: <AANLkTimCWXJ+JzOsQkd9KVKiAsMWNDC4hbzHU=qt_k=A@mail.gmail.com>
References: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com>
	<AANLkTinbHs2wrzp5c46kfsAP937pfa-mLBnF6Ac3sQEJ@mail.gmail.com>
	<4C6D8E01.6070005@apache.org>
	<40952204-AE73-4E5E-97CC-7BAAB3374C9D@mac.com>
	<32A3FA21-69A1-4B2F-9F7F-DBF0473BD1B5@gmail.com>
	<AANLkTimCWXJ+JzOsQkd9KVKiAsMWNDC4hbzHU=qt_k=A@mail.gmail.com>
Date: Fri, 20 Aug 2010 14:42:46 -0700
Message-ID: <AANLkTinonSD-tdK-6OuE+tAj_6YBZFLqFT+3On5en6dy@mail.gmail.com>
Subject: Re: [VOTE] Combine MapReduce/HDFS Committers
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Sorry, I missed (Cos, +1) in my count, but the outcome is unchanged. -C

On Fri, Aug 20, 2010 at 2:38 PM, Chris Douglas <cdouglas@apache.org> wrote:
> +1: Tom, Doug, Nigel, Stack, Dhruba, (Eli)
> -1: Owen, Arun, Hemanth, Nicholas
> +/-0: Konstantin, (Steve)
>
> The vote did not pass.
>
> Thanks to everyone who voted. -C
>

From general-return-1998-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 20 21:47:24 2010
Return-Path: <general-return-1998-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6095 invoked from network); 20 Aug 2010 21:47:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Aug 2010 21:47:24 -0000
Received: (qmail 45434 invoked by uid 500); 20 Aug 2010 21:47:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45335 invoked by uid 500); 20 Aug 2010 21:47:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45327 invoked by uid 99); 20 Aug 2010 21:47:22 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 21:47:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 20 Aug 2010 21:47:05 +0000
Received: (qmail 6042 invoked by uid 99); 20 Aug 2010 21:46:43 -0000
Received: from localhost.apache.org (HELO mail-iw0-f176.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 21:46:43 +0000
Received: by iwn9 with SMTP id 9so1247648iwn.35
        for <general@hadoop.apache.org>; Fri, 20 Aug 2010 14:46:42 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.182.204 with SMTP id cd12mr2225735ibb.101.1282340802834;
 Fri, 20 Aug 2010 14:46:42 -0700 (PDT)
Received: by 10.231.171.75 with HTTP; Fri, 20 Aug 2010 14:46:42 -0700 (PDT)
In-Reply-To: <AANLkTikCW=ZcwMAyfpBaPG_Lbp6b5bqj3oQXDOFpC4Ju@mail.gmail.com>
References: <7962BB4E-7176-44EB-A708-64881639E2AF@apache.org>
	<AANLkTikCW=ZcwMAyfpBaPG_Lbp6b5bqj3oQXDOFpC4Ju@mail.gmail.com>
Date: Fri, 20 Aug 2010 14:46:42 -0700
Message-ID: <AANLkTimq=BuN80Uf1B8pjLHQZ+Uncn9Mgxm5PCXd86TS@mail.gmail.com>
Subject: Re: [VOTE] Should we define the Common committers as HDFS + MapReduce committers?
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

+1: Owen, Arun, Doug, Nigel, Hemanth, Chris, Tom, Nicholas,
Konstantin, Stack, (Jakob), (Cos), (Daniel)

The vote passes.

Thanks to everyone who voted. -C

From general-return-1999-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 20 21:53:25 2010
Return-Path: <general-return-1999-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 7858 invoked from network); 20 Aug 2010 21:53:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Aug 2010 21:53:24 -0000
Received: (qmail 54021 invoked by uid 500); 20 Aug 2010 21:53:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53973 invoked by uid 500); 20 Aug 2010 21:53:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53965 invoked by uid 99); 20 Aug 2010 21:53:23 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 21:53:23 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 21:53:00 +0000
Received: by bwz14 with SMTP id 14so3915475bwz.35
        for <general@hadoop.apache.org>; Fri, 20 Aug 2010 14:52:40 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.52.12 with SMTP id f12mr1114277bkg.212.1282341159855; Fri,
 20 Aug 2010 14:52:39 -0700 (PDT)
Received: by 10.204.179.74 with HTTP; Fri, 20 Aug 2010 14:52:39 -0700 (PDT)
In-Reply-To: <AANLkTinonSD-tdK-6OuE+tAj_6YBZFLqFT+3On5en6dy@mail.gmail.com>
References: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com>
	<AANLkTinbHs2wrzp5c46kfsAP937pfa-mLBnF6Ac3sQEJ@mail.gmail.com>
	<4C6D8E01.6070005@apache.org>
	<40952204-AE73-4E5E-97CC-7BAAB3374C9D@mac.com>
	<32A3FA21-69A1-4B2F-9F7F-DBF0473BD1B5@gmail.com>
	<AANLkTimCWXJ+JzOsQkd9KVKiAsMWNDC4hbzHU=qt_k=A@mail.gmail.com>
	<AANLkTinonSD-tdK-6OuE+tAj_6YBZFLqFT+3On5en6dy@mail.gmail.com>
Date: Fri, 20 Aug 2010 14:52:39 -0700
Message-ID: <AANLkTi=jm-jm7jB3W5njFV-_3XJ2ZZsh5-mEfgkqEuiA@mail.gmail.com>
Subject: Re: [VOTE] Combine MapReduce/HDFS Committers
From: Tom White <tom@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Chris,

Didn't you vote +1 too? (If I recall correctly from the discussion.)

Tom

On Fri, Aug 20, 2010 at 2:42 PM, Chris Douglas <cdouglas@apache.org> wrote:
> Sorry, I missed (Cos, +1) in my count, but the outcome is unchanged. -C
>
> On Fri, Aug 20, 2010 at 2:38 PM, Chris Douglas <cdouglas@apache.org> wrote:
>> +1: Tom, Doug, Nigel, Stack, Dhruba, (Eli)
>> -1: Owen, Arun, Hemanth, Nicholas
>> +/-0: Konstantin, (Steve)
>>
>> The vote did not pass.
>>
>> Thanks to everyone who voted. -C
>>
>

From general-return-2000-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 21 00:07:54 2010
Return-Path: <general-return-2000-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53838 invoked from network); 21 Aug 2010 00:07:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Aug 2010 00:07:53 -0000
Received: (qmail 60283 invoked by uid 500); 21 Aug 2010 00:07:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60159 invoked by uid 500); 21 Aug 2010 00:07:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60151 invoked by uid 99); 21 Aug 2010 00:07:52 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Aug 2010 00:07:52 +0000
X-ASF-Spam-Status: No, hits=2.7 required=10.0
	tests=RCVD_ILLEGAL_IP,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of athusoo@facebook.com designates 69.63.178.168 as permitted sender)
Received: from [69.63.178.168] (HELO mx-out.facebook.com) (69.63.178.168)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Aug 2010 00:07:30 +0000
Received: from [10.18.255.125] ([10.18.255.125:21884] helo=mail.thefacebook.com)
	by mta008.snc1.facebook.com (envelope-from <athusoo@facebook.com>)
	(ecelerity 2.2.2.45 r(34067)) with ESMTP
	id DD/76-01699-DA81F6C4; Fri, 20 Aug 2010 17:07:09 -0700
Received: from SC-MBX03.TheFacebook.com ([169.254.2.176]) by
 sc-hub03.TheFacebook.com ([fe80::1cfe:1f6b:8b35:cf7f%11]) with mapi; Fri, 20
 Aug 2010 17:05:55 -0700
From: Ashish Thusoo <athusoo@facebook.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "hive-dev@hadoop.apache.org" <hive-dev@hadoop.apache.org>
Subject: [DISCUSS] Hive as TLP
Thread-Topic: [DISCUSS] Hive as TLP
Thread-Index: ActAxJyTtS6qs0fBTd2lo41dVEUXsA==
Date: Sat, 21 Aug 2010 00:05:49 +0000
Message-ID: <09370B27F829314182C33A219530C1040379C139@sc-mbx03.TheFacebook.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

The Hive subproject has voted to become a TLP

http://bit.ly/9nb4nN

Does the Hadoop community have any questions or concerns on this? I will be=
 calling a more formal vote after this discussion.

The Hive dev community is still dominated by Facebook but the community is =
working hard to diversify the base and hopes to add committers from Yahoo a=
nd Cloudera. We anticipate that we will have a more diversified base by the=
 end of the year modulo contributions from developers at these entities - a=
nd there are a fair bit in the pipeline.

Thanks,
Ashish

From general-return-2001-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 21 00:38:08 2010
Return-Path: <general-return-2001-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68550 invoked from network); 21 Aug 2010 00:38:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Aug 2010 00:38:08 -0000
Received: (qmail 74802 invoked by uid 500); 21 Aug 2010 00:38:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74739 invoked by uid 500); 21 Aug 2010 00:38:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74731 invoked by uid 99); 21 Aug 2010 00:38:06 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Aug 2010 00:38:06 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Sat, 21 Aug 2010 00:37:48 +0000
Received: (qmail 68517 invoked by uid 99); 21 Aug 2010 00:37:27 -0000
Received: from localhost.apache.org (HELO mail-yw0-f48.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Aug 2010 00:37:27 +0000
Received: by ywg4 with SMTP id 4so2054493ywg.35
        for <general@hadoop.apache.org>; Fri, 20 Aug 2010 17:37:26 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.150.202.6 with SMTP id z6mr1995202ybf.290.1282351044092; Fri,
 20 Aug 2010 17:37:24 -0700 (PDT)
Received: by 10.150.184.5 with HTTP; Fri, 20 Aug 2010 17:37:24 -0700 (PDT)
In-Reply-To: <AANLkTi=jm-jm7jB3W5njFV-_3XJ2ZZsh5-mEfgkqEuiA@mail.gmail.com>
References: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com>
	<AANLkTinbHs2wrzp5c46kfsAP937pfa-mLBnF6Ac3sQEJ@mail.gmail.com>
	<4C6D8E01.6070005@apache.org>
	<40952204-AE73-4E5E-97CC-7BAAB3374C9D@mac.com>
	<32A3FA21-69A1-4B2F-9F7F-DBF0473BD1B5@gmail.com>
	<AANLkTimCWXJ+JzOsQkd9KVKiAsMWNDC4hbzHU=qt_k=A@mail.gmail.com>
	<AANLkTinonSD-tdK-6OuE+tAj_6YBZFLqFT+3On5en6dy@mail.gmail.com>
	<AANLkTi=jm-jm7jB3W5njFV-_3XJ2ZZsh5-mEfgkqEuiA@mail.gmail.com>
Date: Fri, 20 Aug 2010 17:37:24 -0700
Message-ID: <AANLkTimmRAn+UUyP=GjnvD=U8NKE6ej1rcu=WN3QJWJ1@mail.gmail.com>
Subject: Re: [VOTE] Combine MapReduce/HDFS Committers
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Fri, Aug 20, 2010 at 2:52 PM, Tom White <tom@cloudera.com> wrote:
> Chris,
>
> Didn't you vote +1 too? (If I recall correctly from the discussion.)

At the start of the discussion I was leaning toward a +1, but elected
to abstain from the vote as I read the responses. I didn't add an
explicit +/-0, as I'd already added all the content I had in the
discussion thread, and would only repeat myself. Since you ask:

Vinod's point, i.e. "Where are the HDFS + MR issues?" gave me pause.
There really aren't that many. The only exception I can think of are
the RAID JIRAs, which aren't MR issues. Removing contrib is a better
fix. The projects really are separating, and I expect this will become
clearer as we cut more releases. If not, we should revisit this topic.

After these votes, "Hadoop committers" are defined as PMC members, or
contributors who independently earn commit status in both projects.
Most members of the current PMC required about two years of full-time
work in Hadoop before they were added. That's unusually steep, and the
reason I didn't vote -1.

There's a lack of data that makes this ambivalence hard to resolve.
The arguments in favor are too abstract (trust, coherency, etc.) to
make a definitive statements about the cost of separate lists.
Examples of the form, "I tried to do X, but was prevented and it
caused a costly, bureaucratic detour that took N days to resolve" (as
Vinod provided for lacking access to Common) are persuasive to me.
Hypothetical contributors that might be prevented from trying
something are compelling, but less so. I'd like to get more experience
working and releasing before I vote one way or the other. -C

From general-return-2002-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 21 00:58:05 2010
Return-Path: <general-return-2002-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70249 invoked from network); 21 Aug 2010 00:58:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Aug 2010 00:58:05 -0000
Received: (qmail 84576 invoked by uid 500); 21 Aug 2010 00:58:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84527 invoked by uid 500); 21 Aug 2010 00:58:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84496 invoked by uid 99); 21 Aug 2010 00:58:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Aug 2010 00:58:03 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Sat, 21 Aug 2010 00:58:02 +0000
Received: (qmail 70167 invoked by uid 99); 21 Aug 2010 00:57:42 -0000
Received: from localhost.apache.org (HELO [192.168.168.106]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Aug 2010 00:57:42 +0000
Message-ID: <4C6F2485.9000509@apache.org>
Date: Fri, 20 Aug 2010 17:57:41 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.11) Gecko/20100713 Thunderbird/3.0.6
MIME-Version: 1.0
To: general@hadoop.apache.org
CC: Ashish Thusoo <athusoo@facebook.com>, 
 "hive-dev@hadoop.apache.org" <hive-dev@hadoop.apache.org>
Subject: Re: [DISCUSS] Hive as TLP
References: <09370B27F829314182C33A219530C1040379C139@sc-mbx03.TheFacebook.com>
In-Reply-To: <09370B27F829314182C33A219530C1040379C139@sc-mbx03.TheFacebook.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hive seems a distinct developer community from Hadoop and thus deserves 
to be a separate project.  It has made releases and appears to have met 
the requirements expected of a TLP.  3+ organizations are represented on 
its PMC, so while diversity is a concern, there seems to be a will to 
address this going forward, so I feel a detour through the Incubator is 
not required.

http://incubator.apache.org/incubation/Incubation_Policy.html#Graduating+from+the+Incubator

Doug


On 08/20/2010 05:05 PM, Ashish Thusoo wrote:
> The Hive subproject has voted to become a TLP
>
> http://bit.ly/9nb4nN
>
> Does the Hadoop community have any questions or concerns on this? I will be calling a more formal vote after this discussion.
>
> The Hive dev community is still dominated by Facebook but the community is working hard to diversify the base and hopes to add committers from Yahoo and Cloudera. We anticipate that we will have a more diversified base by the end of the year modulo contributions from developers at these entities - and there are a fair bit in the pipeline.
>
> Thanks,
> Ashish

From general-return-2003-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 21 15:42:33 2010
Return-Path: <general-return-2003-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43067 invoked from network); 21 Aug 2010 15:42:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Aug 2010 15:42:32 -0000
Received: (qmail 83450 invoked by uid 500); 21 Aug 2010 15:42:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83289 invoked by uid 500); 21 Aug 2010 15:42:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83270 invoked by uid 99); 21 Aug 2010 15:42:30 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Aug 2010 15:42:30 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Aug 2010 15:42:08 +0000
Received: by bwz14 with SMTP id 14so4542558bwz.35
        for <general@hadoop.apache.org>; Sat, 21 Aug 2010 08:41:48 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.51.67 with SMTP id c3mr2074463bkg.69.1282405308341; Sat,
 21 Aug 2010 08:41:48 -0700 (PDT)
Received: by 10.204.179.74 with HTTP; Sat, 21 Aug 2010 08:41:48 -0700 (PDT)
In-Reply-To: <AANLkTimmRAn+UUyP=GjnvD=U8NKE6ej1rcu=WN3QJWJ1@mail.gmail.com>
References: <AANLkTikcnTxtR-C_+Diw9cNFgf6e2t5xwKt8-fQDPO59@mail.gmail.com>
	<AANLkTinbHs2wrzp5c46kfsAP937pfa-mLBnF6Ac3sQEJ@mail.gmail.com>
	<4C6D8E01.6070005@apache.org>
	<40952204-AE73-4E5E-97CC-7BAAB3374C9D@mac.com>
	<32A3FA21-69A1-4B2F-9F7F-DBF0473BD1B5@gmail.com>
	<AANLkTimCWXJ+JzOsQkd9KVKiAsMWNDC4hbzHU=qt_k=A@mail.gmail.com>
	<AANLkTinonSD-tdK-6OuE+tAj_6YBZFLqFT+3On5en6dy@mail.gmail.com>
	<AANLkTi=jm-jm7jB3W5njFV-_3XJ2ZZsh5-mEfgkqEuiA@mail.gmail.com>
	<AANLkTimmRAn+UUyP=GjnvD=U8NKE6ej1rcu=WN3QJWJ1@mail.gmail.com>
Date: Sat, 21 Aug 2010 08:41:48 -0700
Message-ID: <AANLkTinuUyc4N3NmuqH3MWXrrJkJM_yK5k7-qF_0em6t@mail.gmail.com>
Subject: Re: [VOTE] Combine MapReduce/HDFS Committers
From: Tom White <tom@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks for sharing this, Chris.

Cheers,
Tom

On Fri, Aug 20, 2010 at 5:37 PM, Chris Douglas <cdouglas@apache.org> wrote:
> On Fri, Aug 20, 2010 at 2:52 PM, Tom White <tom@cloudera.com> wrote:
>> Chris,
>>
>> Didn't you vote +1 too? (If I recall correctly from the discussion.)
>
> At the start of the discussion I was leaning toward a +1, but elected
> to abstain from the vote as I read the responses. I didn't add an
> explicit +/-0, as I'd already added all the content I had in the
> discussion thread, and would only repeat myself. Since you ask:
>
> Vinod's point, i.e. "Where are the HDFS + MR issues?" gave me pause.
> There really aren't that many. The only exception I can think of are
> the RAID JIRAs, which aren't MR issues. Removing contrib is a better
> fix. The projects really are separating, and I expect this will become
> clearer as we cut more releases. If not, we should revisit this topic.
>
> After these votes, "Hadoop committers" are defined as PMC members, or
> contributors who independently earn commit status in both projects.
> Most members of the current PMC required about two years of full-time
> work in Hadoop before they were added. That's unusually steep, and the
> reason I didn't vote -1.
>
> There's a lack of data that makes this ambivalence hard to resolve.
> The arguments in favor are too abstract (trust, coherency, etc.) to
> make a definitive statements about the cost of separate lists.
> Examples of the form, "I tried to do X, but was prevented and it
> caused a costly, bureaucratic detour that took N days to resolve" (as
> Vinod provided for lacking access to Common) are persuasive to me.
> Hypothetical contributors that might be prevented from trying
> something are compelling, but less so. I'd like to get more experience
> working and releasing before I vote one way or the other. -C
>

From general-return-2004-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 23 17:39:05 2010
Return-Path: <general-return-2004-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 36771 invoked from network); 23 Aug 2010 17:39:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Aug 2010 17:39:05 -0000
Received: (qmail 35500 invoked by uid 500); 23 Aug 2010 17:39:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 35389 invoked by uid 500); 23 Aug 2010 17:39:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 35247 invoked by uid 99); 23 Aug 2010 17:39:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 17:39:03 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 17:38:58 +0000
Received: from pugholehand-lm.corp.yahoo.com (pugholehand-lm.corp.yahoo.com [10.72.115.44])
	by mrout1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7NHcPif038227
	for <general@hadoop.apache.org>; Mon, 23 Aug 2010 10:38:25 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:content-type:
	content-transfer-encoding:mime-version:subject:date:x-mailer;
	b=dH7T1KYCeRkX2ZczbZuwE/c0UJuq7Awj8u3S/tC8M+Lkm1zs4UoAX4NNr9VBacXL
Message-Id: <71D87762-A95B-434D-9FF6-8627AF65790A@yahoo-inc.com>
From: Alan Gates <gates@yahoo-inc.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: [VOTE] Pig to become a TLP
Date: Mon, 23 Aug 2010 10:38:12 -0700
X-Mailer: Apple Mail (2.936)

I propose that Pig become a top level Apache project.

The Pig development community has already voted on and approved this  
proposal.  In summary, the community voted that all current active  
committers (listed at http://hadoop.apache.org/pig/whoweare.html ) as  
of August 18 2010 should become PMC members, and that Olga Natkovich  
should be the PMC chair.  They also agreed that adopting bylaws should  
be among the first tasks of the new PMC.  Full details of the vote can  
be seen at http://bit.ly/c3GDyl

Please vote on sending this proposal to the Apache Board.

Below is a draft resolution to be sent to the Apache Board:

Establish the Apache Pig Project

         WHEREAS, the Board of Directors deems it to be in the best
         interests of the Foundation and consistent with the
         Foundation's purpose to establish a Project Management
         Committee charged with the creation and maintenance of
         open-source software related to parallel analysis of large
         data sets for distribution at no charge to the public.

         NOW, THEREFORE, BE IT RESOLVED, that a Project Management
         Committee (PMC), to be known as the "Apache Pig Project",
         be and hereby is established pursuant to Bylaws of the
         Foundation; and be it further

         RESOLVED, that the Apache Pig Project be and hereby is
         responsible for the creation and maintenance of software
         related to parallel analysis of large data sets; and be
         it further

         RESOLVED, that the office of "Vice President, Apache Pig" be
         and hereby is created, the person holding such office to
         serve at the direction of the Board of Directors as the chair
         of the Apache Pig Project, and to have primary responsibility
         for management of the projects within the scope of
         responsibility of the Apache Pig Project; and be it further

         RESOLVED, that the persons listed immediately below be and
         hereby are appointed to serve as the initial members of the
         Apache Pig Project:

           * Benjamin Reed          <breed@apache.org>
           * Daniel Dai                  <daijy@apache.org>
           * Alan Gates                 <gates@apache.org>
           * Giridharen Kesavan   <gkesavan@apache.org>
           * Olga Natkovich           <olga@apache.org>
           * Pradeep Kamath        <pradeepkth@apache.org>
           * Santhosh Srinivasan  <sms@apache.org>
           * Yan Zhou                   <yanz@apache.org>
           * Jeff Zhang                  <zjffdu@apache.org>
           * Ashutosh Chauhan     <hashutosh@apache.org>
           * Richard Ding               <rding@apache.org>
           * Dmitriy Ryaboy            <dvryaboy@apache.org>
           * Thejas Nair                  <thejas@apache.org>

         NOW, THEREFORE, BE IT FURTHER RESOLVED, that Olga Natkovich
         be appointed to the office of Vice President, Apache Pig, to
         serve in accordance with and subject to the direction of the
         Board of Directors and the Bylaws of the Foundation until
         death, resignation, retirement, removal or disqualification,
         or until a successor is appointed; and be it further

         RESOLVED, that the initial Apache Pig PMC be and hereby is
         tasked with the creation of a set of bylaws intended to
         encourage open development and increased participation in the
         Apache Pig Project; and be it further

         RESOLVED, that the Apache Pig Project be and hereby
         is tasked with the migration and rationalization of the Apache
         Hadoop Pig sub-project; and be it further

         RESOLVED, that all responsibilities pertaining to the Apache
         Hadoop Pig sub-project encumbered upon the
         Apache Hadoop Project are hereafter discharged.





From general-return-2005-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 23 21:41:18 2010
Return-Path: <general-return-2005-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58896 invoked from network); 23 Aug 2010 21:41:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Aug 2010 21:41:18 -0000
Received: (qmail 23479 invoked by uid 500); 23 Aug 2010 21:41:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23397 invoked by uid 500); 23 Aug 2010 21:41:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23389 invoked by uid 99); 23 Aug 2010 21:41:16 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 21:41:16 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.27.243] (HELO qmta13.emeryville.ca.mail.comcast.net) (76.96.27.243)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 21:40:52 +0000
Received: from omta16.emeryville.ca.mail.comcast.net ([76.96.30.72])
	by qmta13.emeryville.ca.mail.comcast.net with comcast
	id xuTY1e0021ZMdJ4ADxgWuG; Mon, 23 Aug 2010 21:40:30 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta16.emeryville.ca.mail.comcast.net with comcast
	id xxgM1e0082SbwD58cxgPgy; Mon, 23 Aug 2010 21:40:28 +0000
Message-Id: <635760BB-1E76-44D5-A363-71EBB1F0CF57@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <AANLkTimq=BuN80Uf1B8pjLHQZ+Uncn9Mgxm5PCXd86TS@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Should we define the Common committers as HDFS + MapReduce committers?
Date: Mon, 23 Aug 2010 14:40:20 -0700
References: <7962BB4E-7176-44EB-A708-64881639E2AF@apache.org> <AANLkTikCW=ZcwMAyfpBaPG_Lbp6b5bqj3oQXDOFpC4Ju@mail.gmail.com> <AANLkTimq=BuN80Uf1B8pjLHQZ+Uncn9Mgxm5PCXd86TS@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


On Aug 20, 2010, at 2:46 PM, Chris Douglas wrote:

> The vote passes.

Ok, I've removed the common committer list and replaced it with the  
union of the HDFS and MapReduce committer lists.

-- Owen

From general-return-2006-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 23 22:19:47 2010
Return-Path: <general-return-2006-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78166 invoked from network); 23 Aug 2010 22:19:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Aug 2010 22:19:46 -0000
Received: (qmail 65846 invoked by uid 500); 23 Aug 2010 22:19:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65090 invoked by uid 500); 23 Aug 2010 22:19:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65051 invoked by uid 99); 23 Aug 2010 22:19:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 22:19:44 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.27.243] (HELO qmta13.emeryville.ca.mail.comcast.net) (76.96.27.243)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 22:19:35 +0000
Received: from omta19.emeryville.ca.mail.comcast.net ([76.96.30.76])
	by qmta13.emeryville.ca.mail.comcast.net with comcast
	id xp661e0031eYJf8ADyKFEk; Mon, 23 Aug 2010 22:19:15 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta19.emeryville.ca.mail.comcast.net with comcast
	id xyK61e0092SbwD501yK81p; Mon, 23 Aug 2010 22:19:13 +0000
Message-Id: <ACC0BDF0-C9F5-4DBA-90C4-5B94A01CD968@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Branching and testing strategy for 0.22
Date: Mon, 23 Aug 2010 15:19:06 -0700
X-Mailer: Apple Mail (2.936)

I'd like to get started testing 0.22.

I plan to start making mini-branches for QA. These branches will be  
snapshots that QA can use for testing with an expected lifetime of two  
weeks each. Only bug fixes that are blocking QA will be applied to the  
mini-branches and every two weeks, the base of the branch will be  
moved to the head of trunk. This will allow QA to test a point in time  
(possibly with required bug fixes) with requiring development to  
continually maintain two branches.

To simplify automated builds, I'll call the branch the final name of  
"branch-0.22." But it will be rebased every two weeks or so.

Are there any concerns?

-- Owen

From general-return-2007-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 23 22:27:21 2010
Return-Path: <general-return-2007-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82291 invoked from network); 23 Aug 2010 22:27:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Aug 2010 22:27:21 -0000
Received: (qmail 74909 invoked by uid 500); 23 Aug 2010 22:27:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74643 invoked by uid 500); 23 Aug 2010 22:27:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74635 invoked by uid 99); 23 Aug 2010 22:27:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 22:27:19 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 22:27:14 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version;
  b=ueaPg+TSWNqVAEVhUS+CXU99sTMVV5o+g/UDpnSiTOuso3dJPjSDh41W
   bdx8flBfOvBRxGHP4urUe+bE2vnUVklVDEt3M8X1XkJJuyb34O6k4W2Pz
   WNtuUikEOx2qxCh;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1282602434; x=1314138434;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20Branching=20and=20testing=20strategy=20
   for=200.22|Date:=20Mon,=2023=20Aug=202010=2022:26:53=20+0
   000|Message-ID:=20<D14F0E67-2B33-4CB2-A184-CEB4FA06A840@l
   inkedin.com>|To:=20"<general@hadoop.apache.org>"=20<gener
   al@hadoop.apache.org>|MIME-Version:=201.0
   |Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<033b2c8e-989c-488f-bce2-21cc95ff0e59>
   |In-Reply-To:=20<ACC0BDF0-C9F5-4DBA-90C4-5B94A01CD968@apa
   che.org>|References:=20<ACC0BDF0-C9F5-4DBA-90C4-5B94A01CD
   968@apache.org>;
  bh=Plm0ZBcRKz2HOmZeIn2kEhUhiyvMELOShf48w4FliLs=;
  b=fGX8y9IKqdlhK0ttQKDMxrBkaEvtn9p4NoeN8nJzQy8VFgHeOIi3a3Tr
   o+yT8BqBKIlrxGqmV483/4bvj8jiRIZLpaNZtHIJNyYbhAoLPC1FBXABz
   eP/zDsUBrhhMqSt;
X-IronPort-AV: E=Sophos;i="4.56,259,1280732400"; 
   d="scan'208";a="14420384"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi; Mon, 23 Aug 2010
 15:26:54 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: Branching and testing strategy for 0.22
Thread-Topic: Branching and testing strategy for 0.22
Thread-Index: AQHLQxFL8Ew1bFLlK0iEzfCmE7PZ1ZLwE8iA
Date: Mon, 23 Aug 2010 22:26:53 +0000
Message-ID: <D14F0E67-2B33-4CB2-A184-CEB4FA06A840@linkedin.com>
References: <ACC0BDF0-C9F5-4DBA-90C4-5B94A01CD968@apache.org>
In-Reply-To: <ACC0BDF0-C9F5-4DBA-90C4-5B94A01CD968@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Content-Type: text/plain; charset="us-ascii"
Content-ID: <033b2c8e-989c-488f-bce2-21cc95ff0e59>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


On Aug 23, 2010, at 3:19 PM, Owen O'Malley wrote:
> Are there any concerns?

Just that 0.21 isn't even out of RC yet and that patches to fix it may get =
missed.


From general-return-2008-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 23 22:27:33 2010
Return-Path: <general-return-2008-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82426 invoked from network); 23 Aug 2010 22:27:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Aug 2010 22:27:33 -0000
Received: (qmail 76980 invoked by uid 500); 23 Aug 2010 22:27:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76903 invoked by uid 500); 23 Aug 2010 22:27:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76895 invoked by uid 99); 23 Aug 2010 22:27:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 22:27:31 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 22:27:25 +0000
Received: by vws4 with SMTP id 4so8182101vws.35
        for <general@hadoop.apache.org>; Mon, 23 Aug 2010 15:27:04 -0700 (PDT)
Received: by 10.220.124.211 with SMTP id v19mr3560424vcr.184.1282602424321;
 Mon, 23 Aug 2010 15:27:04 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.220.96.141 with HTTP; Mon, 23 Aug 2010 15:26:44 -0700 (PDT)
In-Reply-To: <ACC0BDF0-C9F5-4DBA-90C4-5B94A01CD968@apache.org>
References: <ACC0BDF0-C9F5-4DBA-90C4-5B94A01CD968@apache.org>
From: Aaron Kimball <aaron@cloudera.com>
Date: Mon, 23 Aug 2010 15:26:44 -0700
Message-ID: <AANLkTi=80r4NEGuEhhBOdKLYWLCoraVge4WYAy=v_ktn@mail.gmail.com>
Subject: Re: Branching and testing strategy for 0.22
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636ed782ae3cf69048e8525ca

--001636ed782ae3cf69048e8525ca
Content-Type: text/plain; charset=ISO-8859-1

Would it be worthwhile to give branches unique, persistent names?
branch-0.22-qa1, branch-0.22-qa2, etc. Then problems in a later incarnation
of the QA branch could be regression-tested against the previous one.

Your point about automated builds is, however, noted. If this were git,
branch-0.22 could be a "floating" branch which is aliased with the most
recent qa branch name. Can we do something similar with svn?

We could remove all the QA branches when we officially cut for RC, if you're
concerned about clutter.

- Aaron

On Mon, Aug 23, 2010 at 3:19 PM, Owen O'Malley <omalley@apache.org> wrote:

> I'd like to get started testing 0.22.
>
> I plan to start making mini-branches for QA. These branches will be
> snapshots that QA can use for testing with an expected lifetime of two weeks
> each. Only bug fixes that are blocking QA will be applied to the
> mini-branches and every two weeks, the base of the branch will be moved to
> the head of trunk. This will allow QA to test a point in time (possibly with
> required bug fixes) with requiring development to continually maintain two
> branches.
>
> To simplify automated builds, I'll call the branch the final name of
> "branch-0.22." But it will be rebased every two weeks or so.
>
> Are there any concerns?
>
> -- Owen
>

--001636ed782ae3cf69048e8525ca--

From general-return-2009-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 23 22:58:53 2010
Return-Path: <general-return-2009-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87633 invoked from network); 23 Aug 2010 22:58:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Aug 2010 22:58:52 -0000
Received: (qmail 5325 invoked by uid 500); 23 Aug 2010 22:58:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5259 invoked by uid 500); 23 Aug 2010 22:58:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5238 invoked by uid 99); 23 Aug 2010 22:58:51 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 22:58:51 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 23 Aug 2010 22:58:33 +0000
Received: (qmail 87503 invoked by uid 99); 23 Aug 2010 22:58:11 -0000
Received: from localhost.apache.org (HELO [192.168.168.104]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 22:58:11 +0000
Message-ID: <4C72FD03.7070309@apache.org>
Date: Mon, 23 Aug 2010 15:58:11 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.11) Gecko/20100713 Thunderbird/3.0.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Branching and testing strategy for 0.22
References: <ACC0BDF0-C9F5-4DBA-90C4-5B94A01CD968@apache.org>
In-Reply-To: <ACC0BDF0-C9F5-4DBA-90C4-5B94A01CD968@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 08/23/2010 03:19 PM, Owen O'Malley wrote:
> To simplify automated builds, I'll call the branch the final name of
> "branch-0.22." But it will be rebased every two weeks or so.
>
> Are there any concerns?

Since this will be used differently than what we've in the past named 
'branch-${version}", perhaps we should name it differently too, e.g., 
"qa-0.22" or "test-0.22".

Also, since this is branched from trunk, it might just be called 
'trunk-checkpoint' or somesuch rather than any particular version.

Doug

From general-return-2010-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 23 23:17:57 2010
Return-Path: <general-return-2010-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97777 invoked from network); 23 Aug 2010 23:17:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Aug 2010 23:17:57 -0000
Received: (qmail 27150 invoked by uid 500); 23 Aug 2010 23:17:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26958 invoked by uid 500); 23 Aug 2010 23:17:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26950 invoked by uid 99); 23 Aug 2010 23:17:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 23:17:55 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 23:17:49 +0000
Received: from localhost (snvvpn4-10-72-168-c184.hq.corp.yahoo.com [10.72.168.184])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7NNH51X058646
	for <general@hadoop.apache.org>; Mon, 23 Aug 2010 16:17:06 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=date:from:to:subject:message-id:references:mime-version:
	content-type:content-disposition:in-reply-to:x-organization:user-agent;
	b=2JB/DvsyiDVoTTbC7quMuYz2gpFCgtaKJIqMtNbRTFFYJx1GkrWm+sEZj5Gehxjo
Date: Mon, 23 Aug 2010 16:17:05 -0700
From: Konstantin Boudnik <cos@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: Branching and testing strategy for 0.22
Message-ID: <20100823231705.GB26339@goodenter-lm.local>
References: <ACC0BDF0-C9F5-4DBA-90C4-5B94A01CD968@apache.org>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature"; boundary="NDin8bjvE/0mNLFQ"
Content-Disposition: inline
In-Reply-To: <ACC0BDF0-C9F5-4DBA-90C4-5B94A01CD968@apache.org>
X-Organization: Yahoo! Hadoop (Grid computing)
User-Agent: Mutt/1.5.20 (2009-06-14)

--NDin8bjvE/0mNLFQ
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

If I may... why QA need source code branches rather than a sequential builds
=66rom the trunk (as it usually done)? Say:
 - build qa1 is cut
 - QA works on it and finds issues
 - issues are reported against build qa1
 - dev fixes some issues and mark them appropriately
 - build qa2 is cut
 - QA verifies that all issues from qa1 marked as 'fixed in qa2' are fixed =
and
   keep going with their usual cycle.

Is there anything wrong with that model? It also eliminates the need to
maintain two branches at the same time.

Cos

On Mon, Aug 23, 2010 at 03:19PM, Owen O'Malley wrote:
> I'd like to get started testing 0.22.
>=20
> I plan to start making mini-branches for QA. These branches will be =20
> snapshots that QA can use for testing with an expected lifetime of two =
=20
> weeks each. Only bug fixes that are blocking QA will be applied to the =
=20
> mini-branches and every two weeks, the base of the branch will be =20
> moved to the head of trunk. This will allow QA to test a point in time =
=20
> (possibly with required bug fixes) with requiring development to =20
> continually maintain two branches.
>=20
> To simplify automated builds, I'll call the branch the final name of =20
> "branch-0.22." But it will be rebased every two weeks or so.
>=20
> Are there any concerns?
>=20
> -- Owen

--NDin8bjvE/0mNLFQ
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)

iF4EAREIAAYFAkxzAXEACgkQMqXifkwDoaF0jwD/Q2sLFVldm+crAiZKSMRAdwCW
RSWC5c93paDMyi06w9oA/iwYo2YmOo45JQbBXHWubXrzBdPeO4yPENuiZ7x6RTVi
=Nzff
-----END PGP SIGNATURE-----

--NDin8bjvE/0mNLFQ--

From general-return-2011-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 23 23:35:57 2010
Return-Path: <general-return-2011-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4780 invoked from network); 23 Aug 2010 23:35:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Aug 2010 23:35:56 -0000
Received: (qmail 48144 invoked by uid 500); 23 Aug 2010 23:35:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48078 invoked by uid 500); 23 Aug 2010 23:35:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48070 invoked by uid 99); 23 Aug 2010 23:35:55 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 23:35:55 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.30.16] (HELO qmta01.emeryville.ca.mail.comcast.net) (76.96.30.16)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 23:35:30 +0000
Received: from omta05.emeryville.ca.mail.comcast.net ([76.96.30.43])
	by qmta01.emeryville.ca.mail.comcast.net with comcast
	id xskp1e0030vp7WLA1zb8bd; Mon, 23 Aug 2010 23:35:08 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta05.emeryville.ca.mail.comcast.net with comcast
	id xzaz1e0072SbwD58Rzb21A; Mon, 23 Aug 2010 23:35:06 +0000
Message-Id: <38F76180-6DC0-4E1C-88C7-2D4C54539D5B@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <AANLkTi=80r4NEGuEhhBOdKLYWLCoraVge4WYAy=v_ktn@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Branching and testing strategy for 0.22
Date: Mon, 23 Aug 2010 16:34:59 -0700
References: <ACC0BDF0-C9F5-4DBA-90C4-5B94A01CD968@apache.org> <AANLkTi=80r4NEGuEhhBOdKLYWLCoraVge4WYAy=v_ktn@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


On Aug 23, 2010, at 3:26 PM, Aaron Kimball wrote:

> Would it be worthwhile to give branches unique, persistent names?

I don't think so. Of course the old branches will always be there  
under the old revision. But it will be convenient to be able to do  
"svn up" on the given branch and have it track the current qa branch  
without worrying about what the "current" index is and doing a svn  
switch to the new url.

> branch-0.22-qa1, branch-0.22-qa2, etc. Then problems in a later  
> incarnation
> of the QA branch could be regression-tested against the previous one.

These are not intended to be long lived. In particular, any comparison  
testing will be done with the generated artifacts, not the source  
repository.

> Your point about automated builds is, however, noted. If this were  
> git,
> branch-0.22 could be a "floating" branch which is aliased with the  
> most
> recent qa branch name. Can we do something similar with svn?

I don't think so, since branches in subversion act like copies rather  
than markers.

-- Owen

From general-return-2012-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 00:28:31 2010
Return-Path: <general-return-2012-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29175 invoked from network); 24 Aug 2010 00:28:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 00:28:30 -0000
Received: (qmail 78135 invoked by uid 500); 24 Aug 2010 00:28:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78065 invoked by uid 500); 24 Aug 2010 00:28:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78057 invoked by uid 99); 24 Aug 2010 00:28:28 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 00:28:28 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 00:28:08 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7O0REZn088557
	for <general@hadoop.apache.org>; Mon, 23 Aug 2010 17:27:15 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:content-type:
	content-transfer-encoding:mime-version:subject:date:x-mailer;
	b=gmuHD/T0G3zth/n4brO0w3BLIdkU4LJ1GqgTQcIOhIPmXTy9TtmmGidKU90fz8yr
Message-Id: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Date: Mon, 23 Aug 2010 17:27:14 -0700
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

Even with the work on hadoop-0.22 (trunk) starting in earnest it is  
fairly obvious, given our past history, that it will take a while for  
us to get it stable and deployable - for e.g. it took us nearly 6  
months to deploy hadoop-0.20.

In the interim I'd like to propose we push a hadoop-0.20-security  
release off the Yahoo! patchset (http://github.com/yahoo/hadoop- 
common). This will ensure the community benefits from all the work  
done at Yahoo! for over 12 months *now*, and ensures that we do not  
have to wait until hadoop-0.22 which has all of these patches.

Some salient aspects:
a) Full-fledged security implementation deployed at scale (4000 nodes)  
in production.
b) Lots of work on the stabilizing and optimizing the NameNode and  
JobTracker for over 12 months. This has been critical in deploying  
Hadoop at scale i.e. clusters of 4000 nodes. For e.g. we have a 50%  
improvement in CPU utilization on the JobTracker vis-a-vis the  
hadoop-0.20.2 release.
c) Several new features in the scheduler (CapacityScheduler), Map- 
Reduce framework, better support for multi-tenancy etc.
d) Several performance and stability improvements to the system e.g.  
iterative ls, robustness against rogue clients/jobs/users etc.

Also, given the huge number of features and enhancements I'd like to  
propose we create a new 0.20-security branch and commit the Yahoo  
patchset there for the release.

This has been proposed earlier by Doug and did not get far due to  
concerns about the effect this would have on development on trunk.  
However, I believe, we have a case for demonstrable progress on trunk  
now, and it would be useful to have an interim, fully-tested Apache  
Hadoop release available to the community.

  Conceivably, one could imagine a Hadoop Security + Append release  
soon after. At this point a Hadoop Security release alone would add  
tremendous value for the reasons above. Presently we would like to get  
this release out quickly to focus the majority of our efforts on trunk.

Thoughts?

Arun


From general-return-2013-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 03:54:55 2010
Return-Path: <general-return-2013-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 92610 invoked from network); 24 Aug 2010 03:54:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 03:54:55 -0000
Received: (qmail 97681 invoked by uid 500); 24 Aug 2010 03:54:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97306 invoked by uid 500); 24 Aug 2010 03:54:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97298 invoked by uid 99); 24 Aug 2010 03:54:51 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 03:54:51 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 03:54:29 +0000
Received: from [10.66.72.163] (sightbusy-lx.eglbp.corp.yahoo.com [10.66.72.163])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7O3rkW7011104
	for <general@hadoop.apache.org>; Mon, 23 Aug 2010 20:53:47 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=sovV/KcrJX6C1gZZAetjmcYOajwVTsBcmNkBeEr1wdzQ7b19O/+4wyaTesZeDBgR
Message-ID: <4C734249.1060102@yahoo-inc.com>
Date: Tue, 24 Aug 2010 09:23:45 +0530
From: Vinod KV <vinodkv@yahoo-inc.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.11) Gecko/20100713 Lightning/1.0b1 Thunderbird/3.0.6
MIME-Version: 1.0
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: Branching and testing strategy for 0.22
References: <ACC0BDF0-C9F5-4DBA-90C4-5B94A01CD968@apache.org> <20100823231705.GB26339@goodenter-lm.local>
In-Reply-To: <20100823231705.GB26339@goodenter-lm.local>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org


My first thought was also inline with Cos's. Any background we are 
missing about this?

+vinod


On Tuesday 24 August 2010 04:47 AM, Konstantin Boudnik wrote:
> If I may... why QA need source code branches rather than a sequential builds
> from the trunk (as it usually done)? Say:
>   - build qa1 is cut
>   - QA works on it and finds issues
>   - issues are reported against build qa1
>   - dev fixes some issues and mark them appropriately
>   - build qa2 is cut
>   - QA verifies that all issues from qa1 marked as 'fixed in qa2' are fixed and
>     keep going with their usual cycle.
>
> Is there anything wrong with that model? It also eliminates the need to
> maintain two branches at the same time.
>
> Cos
>
> On Mon, Aug 23, 2010 at 03:19PM, Owen O'Malley wrote:
>    
>> I'd like to get started testing 0.22.
>>
>> I plan to start making mini-branches for QA. These branches will be
>> snapshots that QA can use for testing with an expected lifetime of two
>> weeks each. Only bug fixes that are blocking QA will be applied to the
>> mini-branches and every two weeks, the base of the branch will be
>> moved to the head of trunk. This will allow QA to test a point in time
>> (possibly with required bug fixes) with requiring development to
>> continually maintain two branches.
>>
>> To simplify automated builds, I'll call the branch the final name of
>> "branch-0.22." But it will be rebased every two weeks or so.
>>
>> Are there any concerns?
>>
>> -- Owen
>>      


From general-return-2014-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 07:08:44 2010
Return-Path: <general-return-2014-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 62954 invoked from network); 24 Aug 2010 07:08:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 07:08:44 -0000
Received: (qmail 21765 invoked by uid 500); 24 Aug 2010 07:08:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21233 invoked by uid 500); 24 Aug 2010 07:08:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21212 invoked by uid 99); 24 Aug 2010 07:08:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 07:08:38 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.27.227] (HELO qmta12.emeryville.ca.mail.comcast.net) (76.96.27.227)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 07:08:29 +0000
Received: from omta02.emeryville.ca.mail.comcast.net ([76.96.30.19])
	by qmta12.emeryville.ca.mail.comcast.net with comcast
	id y72g1e0010QkzPwAC789zQ; Tue, 24 Aug 2010 07:08:09 +0000
Received: from [10.0.0.13] ([98.234.216.58])
	by omta02.emeryville.ca.mail.comcast.net with comcast
	id y7881e0011GAoR68N788wA; Tue, 24 Aug 2010 07:08:08 +0000
Message-Id: <115EFBD4-93A6-4388-9F72-D6E6E6693E0E@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <20100823231705.GB26339@goodenter-lm.local>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Branching and testing strategy for 0.22
Date: Tue, 24 Aug 2010 00:08:07 -0700
References: <ACC0BDF0-C9F5-4DBA-90C4-5B94A01CD968@apache.org> <20100823231705.GB26339@goodenter-lm.local>
X-Mailer: Apple Mail (2.936)


On Aug 23, 2010, at 4:17 PM, Konstantin Boudnik wrote:

> If I may... why QA need source code branches rather than a  
> sequential builds
> from the trunk (as it usually done)?

For QA to be effective, they can't handle a different artifact every  
day with a new set of features. So the plan is to take a snapshot and  
test that. However, if bugs are found that block testing they need  
precisely those bugs fixed with out any additional features. So you  
need branches to support that. So a sample cycle would look like:

1. create build-1 and test it, discovering bugs B1, B2, and B3. Only  
B1 is blocking further testing.
2. meanwhile, someone has checked in feature F1 into trunk
3. B1 is fixed in the branch and trunk. B2 and B3 are just fixed in  
trunk
4. when build-2 is built it only has B1's fix
5. at the end of the cycle, we rebase the branch to trunk and pickup  
B1, B2, B3 and F1 into build-3.

If you don't have a branch, you would get F1 whether you want it or not.

-- Owen

From general-return-2015-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 09:45:17 2010
Return-Path: <general-return-2015-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 18300 invoked from network); 24 Aug 2010 09:45:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 09:45:17 -0000
Received: (qmail 4044 invoked by uid 500); 24 Aug 2010 09:45:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3628 invoked by uid 500); 24 Aug 2010 09:45:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3603 invoked by uid 99); 24 Aug 2010 09:45:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 09:45:11 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [88.198.2.104] (HELO koch.ro) (88.198.2.104)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 09:45:05 +0000
Received: from [84.72.85.88] (helo=jona.localnet)
	by koch.ro with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
	(Exim 4.69)
	(envelope-from <thomas@koch.ro>)
	id 1Onppv-0002IQ-Bg; Tue, 24 Aug 2010 11:30:31 +0200
From: Thomas Koch <thomas@koch.ro>
Reply-To: thomas@koch.ro
To: general@hadoop.apache.org,
 zookeeper-dev@hadoop.apache.org
Subject: apache commons configuration
Date: Tue, 24 Aug 2010 11:43:45 +0200
User-Agent: KMail/1.13.5 (Linux/2.6.32-4-amd64; KDE/4.4.5; x86_64; ; )
MIME-Version: 1.0
Content-Type: text/plain;
  charset="us-ascii"
Content-Transfer-Encoding: 7bit
Message-Id: <201008241143.46918.thomas@koch.ro>

Hi,

just out of curiosity: Is there any particular reason, why Hadoop projects or 
ZooKeeper do not use apache commons configuration[1]?
I'm just evaluating it for an internal project. Would you recommend it or not?
I imagine, that it could help to make projects configurable either via XML or 
INI files. Some people may prefer one or the other.

[1] http://commons.apache.org/configuration/

Best regards,

Thomas Koch, http://www.koch.ro

From general-return-2016-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 10:25:08 2010
Return-Path: <general-return-2016-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34106 invoked from network); 24 Aug 2010 10:25:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 10:25:07 -0000
Received: (qmail 44910 invoked by uid 500); 24 Aug 2010 10:25:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44517 invoked by uid 500); 24 Aug 2010 10:25:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44509 invoked by uid 99); 24 Aug 2010 10:25:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 10:25:02 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 10:24:54 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 0E5321BA4E3
	for <general@hadoop.apache.org>; Tue, 24 Aug 2010 11:24:32 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id gk2sjMpgmtCA for <general@hadoop.apache.org>;
	Tue, 24 Aug 2010 11:24:31 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by colossus.hpl.hp.com (Postfix) with ESMTPS id 897851BA4C1
	for <general@hadoop.apache.org>; Tue, 24 Aug 2010 11:24:31 +0100 (BST)
MailScanner-NULL-Check: 1283250258.54731@S/aHiNjA678sqbf6f7FTDg
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o7OAOHLh016998
	for <general@hadoop.apache.org>; Tue, 24 Aug 2010 11:24:17 +0100 (BST)
Message-ID: <4C739DD0.8050502@apache.org>
Date: Tue, 24 Aug 2010 11:24:16 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.11) Gecko/20100713 Thunderbird/3.0.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: apache commons configuration
References: <201008241143.46918.thomas@koch.ro>
In-Reply-To: <201008241143.46918.thomas@koch.ro>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o7OAOHLh016998
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 24/08/10 10:43, Thomas Koch wrote:
> Hi,
>
> just out of curiosity: Is there any particular reason, why Hadoop projects or
> ZooKeeper do not use apache commons configuration[1]?
> I'm just evaluating it for an internal project. Would you recommend it or not?
> I imagine, that it could help to make projects configurable either via XML or
> INI files. Some people may prefer one or the other.
>
> [1] http://commons.apache.org/configuration/

Someone's discussed it, I'm against because (a) there's a lot of 
historical configuration stuff in there, (b) ini files are less flexible 
than XML, (c) neither of them are very dynamic. when the NN goes down 
and you want the DNs to reconnect to a different host, how do you do 
that when all data is read at start time?



From general-return-2017-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 16:40:52 2010
Return-Path: <general-return-2017-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5683 invoked from network); 24 Aug 2010 16:40:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 16:40:52 -0000
Received: (qmail 37376 invoked by uid 500); 24 Aug 2010 16:40:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37275 invoked by uid 500); 24 Aug 2010 16:40:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37267 invoked by uid 99); 24 Aug 2010 16:40:49 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 16:40:49 +0000
X-ASF-Spam-Status: No, hits=-0.1 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of swatt@us.ibm.com designates 32.97.182.138 as permitted sender)
Received: from [32.97.182.138] (HELO e8.ny.us.ibm.com) (32.97.182.138)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 16:40:24 +0000
Received: from d01relay04.pok.ibm.com (d01relay04.pok.ibm.com [9.56.227.236])
	by e8.ny.us.ibm.com (8.14.4/8.13.1) with ESMTP id o7OGMBhX020205
	for <general@hadoop.apache.org>; Tue, 24 Aug 2010 12:22:11 -0400
Received: from d03av02.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.195.168])
	by d01relay04.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id o7OGe235088334
	for <general@hadoop.apache.org>; Tue, 24 Aug 2010 12:40:03 -0400
Received: from d03av02.boulder.ibm.com (loopback [127.0.0.1])
	by d03av02.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id o7OGe04q002138
	for <general@hadoop.apache.org>; Tue, 24 Aug 2010 10:40:00 -0600
Received: from d03nm123.boulder.ibm.com (d03nm123.boulder.ibm.com [9.17.195.149])
	by d03av02.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id o7OGdx5F002089
	for <general@hadoop.apache.org>; Tue, 24 Aug 2010 10:39:59 -0600
In-Reply-To: <4C739DD0.8050502@apache.org>
References: <201008241143.46918.thomas@koch.ro> <4C739DD0.8050502@apache.org>
To: general@hadoop.apache.org
MIME-Version: 1.0
Subject: August Austin Hadoop User Group Meeting 
X-KeepSent: 3950A1AF:08B70263-86257789:005B515F;
 type=4; name=$KeepSent
X-Mailer: Lotus Notes Release 8.5 December 05, 2008
Message-ID: <OF3950A1AF.08B70263-ON86257789.005B515F-86257789.005B8D52@us.ibm.com>
From: Stephen Watt <swatt@us.ibm.com>
Date: Tue, 24 Aug 2010 11:39:58 -0500
X-MIMETrack: Serialize by Router on D03NM123/03/M/IBM(Release 8.5.1FP2|March 17, 2010) at
 08/24/2010 10:39:59,
	Serialize complete at 08/24/2010 10:39:59
Content-Type: multipart/alternative; boundary="=_alternative 005B8D5186257789_="
X-Virus-Checked: Checked by ClamAV on apache.org

--=_alternative 005B8D5186257789_=
Content-Type: text/plain; charset="US-ASCII"

Hi Folks 

For those in the Austin, Texas area we'll be meeting again this Thursday 
evening (8/26). Karmasphere and Pentaho will be presenting and I'll also 
be giving a quick introduction to Hadoop for all the new attendees before 
we begin.

Full details are available here - 
http://austinhug.blogspot.com/2010/08/august-meeting-826.html

Regards
Steve Watt 



From:
Steve Loughran <stevel@apache.org>
To:
general@hadoop.apache.org
Date:
08/24/2010 05:25 AM
Subject:
Re: apache commons configuration



On 24/08/10 10:43, Thomas Koch wrote:
> Hi,
>
> just out of curiosity: Is there any particular reason, why Hadoop 
projects or
> ZooKeeper do not use apache commons configuration[1]?
> I'm just evaluating it for an internal project. Would you recommend it 
or not?
> I imagine, that it could help to make projects configurable either via 
XML or
> INI files. Some people may prefer one or the other.
>
> [1] http://commons.apache.org/configuration/

Someone's discussed it, I'm against because (a) there's a lot of 
historical configuration stuff in there, (b) ini files are less flexible 
than XML, (c) neither of them are very dynamic. when the NN goes down 
and you want the DNs to reconnect to a different host, how do you do 
that when all data is read at start time?





--=_alternative 005B8D5186257789_=--

From general-return-2018-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 16:50:23 2010
Return-Path: <general-return-2018-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 8036 invoked from network); 24 Aug 2010 16:50:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 16:50:22 -0000
Received: (qmail 51951 invoked by uid 500); 24 Aug 2010 16:50:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51840 invoked by uid 500); 24 Aug 2010 16:50:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51832 invoked by uid 99); 24 Aug 2010 16:50:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 16:50:21 +0000
X-ASF-Spam-Status: No, hits=-0.1 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of swatt@us.ibm.com designates 32.97.182.142 as permitted sender)
Received: from [32.97.182.142] (HELO e2.ny.us.ibm.com) (32.97.182.142)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 16:50:13 +0000
Received: from d01relay04.pok.ibm.com (d01relay04.pok.ibm.com [9.56.227.236])
	by e2.ny.us.ibm.com (8.14.4/8.13.1) with ESMTP id o7OGZZO8027048
	for <general@hadoop.apache.org>; Tue, 24 Aug 2010 12:35:35 -0400
Received: from d03av05.boulder.ibm.com (d03av05.boulder.ibm.com [9.17.195.85])
	by d01relay04.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id o7OGnq0D102508
	for <general@hadoop.apache.org>; Tue, 24 Aug 2010 12:49:52 -0400
Received: from d03av05.boulder.ibm.com (loopback [127.0.0.1])
	by d03av05.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id o7OGnpj8003693
	for <general@hadoop.apache.org>; Tue, 24 Aug 2010 10:49:51 -0600
Received: from d03nm123.boulder.ibm.com (d03nm123.boulder.ibm.com [9.17.195.149])
	by d03av05.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id o7OGnpUr003661
	for <general@hadoop.apache.org>; Tue, 24 Aug 2010 10:49:51 -0600
In-Reply-To: <OF3950A1AF.08B70263-ON86257789.005B515F-86257789.005B8D52@us.ibm.com>
References: <201008241143.46918.thomas@koch.ro> <4C739DD0.8050502@apache.org> <OF3950A1AF.08B70263-ON86257789.005B515F-86257789.005B8D52@us.ibm.com>
To: general@hadoop.apache.org
MIME-Version: 1.0
Subject: SXSW Interactive Big Data Panel Session - Community Votes required
X-KeepSent: 064977E9:D8D1A29F-86257789:005BBA3A;
 type=4; name=$KeepSent
X-Mailer: Lotus Notes Release 8.5 December 05, 2008
Message-ID: <OF064977E9.D8D1A29F-ON86257789.005BBA3A-86257789.005C7431@us.ibm.com>
From: Stephen Watt <swatt@us.ibm.com>
Date: Tue, 24 Aug 2010 11:49:50 -0500
X-MIMETrack: Serialize by Router on D03NM123/03/M/IBM(Release 8.5.1FP2|March 17, 2010) at
 08/24/2010 10:49:50,
	Serialize complete at 08/24/2010 10:49:50
Content-Type: multipart/alternative; boundary="=_alternative 005C743086257789_="

--=_alternative 005C743086257789_=
Content-Type: text/plain; charset="US-ASCII"

Hi Folks

15,000 people attended SXSW Interactive last year. As such, a couple of 
folks from the Apache Hadoop and Cassandra community have joined up to put 
together a "Big Data for everyone" panel at SXSW next year. The intent is 
to proselytize both the Online and Offline Big Data stack (incl. HBase, 
Pig, Hive, etc.) to the as yet uninformed. There are no agendas here other 
than getting the word out and educating the audience. 

Acceptance to SXSW is partially democratic and community votes for the 
session greatly affect the outcome. 

If you have a few minutes, please sign up and vote for our panel. We'd 
really appreciate it. Voting closes on Friday.

http://panelpicker.sxsw.com/ideas/view/7475

Regards
Steve Watt
--=_alternative 005C743086257789_=--

From general-return-2019-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 18:36:21 2010
Return-Path: <general-return-2019-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59671 invoked from network); 24 Aug 2010 18:36:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 18:36:21 -0000
Received: (qmail 10343 invoked by uid 500); 24 Aug 2010 18:36:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 10291 invoked by uid 500); 24 Aug 2010 18:36:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 10283 invoked by uid 99); 24 Aug 2010 18:36:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 18:36:19 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 18:36:11 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout2.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7OIYa0N033306
	for <general@hadoop.apache.org>; Tue, 24 Aug 2010 11:34:37 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:references:in-reply-to:accept-language:
	content-language:x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:
	content-type:content-transfer-encoding:mime-version;
	b=qRzTjYTGC+67mBqt0yJ1aski85Y81a+PxSKiR110eGXDuJKcfEKvyVswta/RAWuq
Received: from SP2-EX07VS03.ds.corp.yahoo.com ([98.137.59.33]) by
 SP2-EX07CAS01.ds.corp.yahoo.com ([98.137.59.37]) with mapi; Tue, 24 Aug 2010
 11:34:36 -0700
From: Olga Natkovich <olgan@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 24 Aug 2010 11:34:35 -0700
Subject: RE: [VOTE] Pig to become a TLP
Thread-Topic: [VOTE] Pig to become a TLP
Thread-Index: ActDtzub1vRonnzOS9u8A3TUrZJWPAAA41TQ
Message-ID: <C1EDB62C3771D74AA823DBBBBFB3F89D4E5DE391@SP2-EX07VS03.ds.corp.yahoo.com>
References: <71D87762-A95B-434D-9FF6-8627AF65790A@yahoo-inc.com>
 <71E1E01A-5389-48C2-9A43-9A8BD6B87A2A@yahoo-inc.com>
In-Reply-To: <71E1E01A-5389-48C2-9A43-9A8BD6B87A2A@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

+1

> From: Alan Gates <gates@yahoo-inc.com>
> Date: August 23, 2010 10:38:12 AM PDT
> To: "general@hadoop.apache.org" <general@hadoop.apache.org>
> Subject: [VOTE] Pig to become a TLP
> Reply-To: "general@hadoop.apache.org" <general@hadoop.apache.org>
>
> I propose that Pig become a top level Apache project.
>
> The Pig development community has already voted on and approved this
> proposal.  In summary, the community voted that all current active
> committers (listed at http://hadoop.apache.org/pig/whoweare.html ) as
> of August 18 2010 should become PMC members, and that Olga Natkovich
> should be the PMC chair.  They also agreed that adopting bylaws should
> be among the first tasks of the new PMC.  Full details of the vote can
> be seen at http://bit.ly/c3GDyl
>
> Please vote on sending this proposal to the Apache Board.
>
> Below is a draft resolution to be sent to the Apache Board:
>
> Establish the Apache Pig Project
>
>         WHEREAS, the Board of Directors deems it to be in the best
>         interests of the Foundation and consistent with the
>         Foundation's purpose to establish a Project Management
>         Committee charged with the creation and maintenance of
>         open-source software related to parallel analysis of large
>         data sets for distribution at no charge to the public.
>
>         NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>         Committee (PMC), to be known as the "Apache Pig Project",
>         be and hereby is established pursuant to Bylaws of the
>         Foundation; and be it further
>
>         RESOLVED, that the Apache Pig Project be and hereby is
>         responsible for the creation and maintenance of software
>         related to parallel analysis of large data sets; and be
>         it further
>
>         RESOLVED, that the office of "Vice President, Apache Pig" be
>         and hereby is created, the person holding such office to
>         serve at the direction of the Board of Directors as the chair
>         of the Apache Pig Project, and to have primary responsibility
>         for management of the projects within the scope of
>         responsibility of the Apache Pig Project; and be it further
>
>         RESOLVED, that the persons listed immediately below be and
>         hereby are appointed to serve as the initial members of the
>         Apache Pig Project:
>
>           * Benjamin Reed          <breed@apache.org>
>           * Daniel Dai                  <daijy@apache.org>
>           * Alan Gates                 <gates@apache.org>
>           * Giridharen Kesavan   <gkesavan@apache.org>
>           * Olga Natkovich           <olga@apache.org>
>           * Pradeep Kamath        <pradeepkth@apache.org>
>           * Santhosh Srinivasan  <sms@apache.org>
>           * Yan Zhou                   <yanz@apache.org>
>           * Jeff Zhang                  <zjffdu@apache.org>
>           * Ashutosh Chauhan     <hashutosh@apache.org>
>           * Richard Ding               <rding@apache.org>
>           * Dmitriy Ryaboy            <dvryaboy@apache.org>
>           * Thejas Nair                  <thejas@apache.org>
>
>         NOW, THEREFORE, BE IT FURTHER RESOLVED, that Olga Natkovich
>         be appointed to the office of Vice President, Apache Pig, to
>         serve in accordance with and subject to the direction of the
>         Board of Directors and the Bylaws of the Foundation until
>         death, resignation, retirement, removal or disqualification,
>         or until a successor is appointed; and be it further
>
>         RESOLVED, that the initial Apache Pig PMC be and hereby is
>         tasked with the creation of a set of bylaws intended to
>         encourage open development and increased participation in the
>         Apache Pig Project; and be it further
>
>         RESOLVED, that the Apache Pig Project be and hereby
>         is tasked with the migration and rationalization of the Apache
>         Hadoop Pig sub-project; and be it further
>
>         RESOLVED, that all responsibilities pertaining to the Apache
>         Hadoop Pig sub-project encumbered upon the
>         Apache Hadoop Project are hereafter discharged.
>
>
>
>


From general-return-2020-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 18:38:52 2010
Return-Path: <general-return-2020-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60322 invoked from network); 24 Aug 2010 18:38:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 18:38:51 -0000
Received: (qmail 16535 invoked by uid 500); 24 Aug 2010 18:38:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16483 invoked by uid 500); 24 Aug 2010 18:38:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16475 invoked by uid 99); 24 Aug 2010 18:38:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 18:38:50 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.101 as permitted sender)
Received: from [17.148.16.101] (HELO asmtpout026.mac.com) (17.148.16.101)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 18:38:42 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; charset=us-ascii
Received: from [10.0.1.3] (c-67-180-80-11.hsd1.ca.comcast.net [67.180.80.11])
 by asmtp026.mac.com
 (Sun Java(tm) System Messaging Server 6.3-8.01 (built Dec 16 2008; 32bit))
 with ESMTPSA id <0L7O00E1S6F7OKA0@asmtp026.mac.com> for
 general@hadoop.apache.org; Tue, 24 Aug 2010 11:38:09 -0700 (PDT)
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0
 reason=mlx engine=6.0.2-1004200000 definitions=main-1008240117
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.0.10011,1.0.148,0.0.0000
 definitions=2010-08-24_10:2010-08-24,2010-08-24,1970-01-01 signatures=0
Subject: Re: [VOTE] Pig to become a TLP
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <71D87762-A95B-434D-9FF6-8627AF65790A@yahoo-inc.com>
Date: Tue, 24 Aug 2010 11:38:08 -0700
Message-id: <74F969A5-B987-49EB-8547-9A1EAD165599@mac.com>
References: <71D87762-A95B-434D-9FF6-8627AF65790A@yahoo-inc.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1081)
X-Virus-Checked: Checked by ClamAV on apache.org

+1

On Aug 23, 2010, at 10:38 AM, Alan Gates wrote:

> I propose that Pig become a top level Apache project.
> 
> The Pig development community has already voted on and approved this proposal.  In summary, the community voted that all current active committers (listed at http://hadoop.apache.org/pig/whoweare.html ) as of August 18 2010 should become PMC members, and that Olga Natkovich should be the PMC chair.  They also agreed that adopting bylaws should be among the first tasks of the new PMC.  Full details of the vote can be seen at http://bit.ly/c3GDyl
> 
> Please vote on sending this proposal to the Apache Board.
> 
> Below is a draft resolution to be sent to the Apache Board:
> 
> Establish the Apache Pig Project
> 
>        WHEREAS, the Board of Directors deems it to be in the best
>        interests of the Foundation and consistent with the
>        Foundation's purpose to establish a Project Management
>        Committee charged with the creation and maintenance of
>        open-source software related to parallel analysis of large
>        data sets for distribution at no charge to the public.
> 
>        NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>        Committee (PMC), to be known as the "Apache Pig Project",
>        be and hereby is established pursuant to Bylaws of the
>        Foundation; and be it further
> 
>        RESOLVED, that the Apache Pig Project be and hereby is
>        responsible for the creation and maintenance of software
>        related to parallel analysis of large data sets; and be
>        it further
> 
>        RESOLVED, that the office of "Vice President, Apache Pig" be
>        and hereby is created, the person holding such office to
>        serve at the direction of the Board of Directors as the chair
>        of the Apache Pig Project, and to have primary responsibility
>        for management of the projects within the scope of
>        responsibility of the Apache Pig Project; and be it further
> 
>        RESOLVED, that the persons listed immediately below be and
>        hereby are appointed to serve as the initial members of the
>        Apache Pig Project:
> 
>          * Benjamin Reed          <breed@apache.org>
>          * Daniel Dai                  <daijy@apache.org>
>          * Alan Gates                 <gates@apache.org>
>          * Giridharen Kesavan   <gkesavan@apache.org>
>          * Olga Natkovich           <olga@apache.org>
>          * Pradeep Kamath        <pradeepkth@apache.org>
>          * Santhosh Srinivasan  <sms@apache.org>
>          * Yan Zhou                   <yanz@apache.org>
>          * Jeff Zhang                  <zjffdu@apache.org>
>          * Ashutosh Chauhan     <hashutosh@apache.org>
>          * Richard Ding               <rding@apache.org>
>          * Dmitriy Ryaboy            <dvryaboy@apache.org>
>          * Thejas Nair                  <thejas@apache.org>
> 
>        NOW, THEREFORE, BE IT FURTHER RESOLVED, that Olga Natkovich
>        be appointed to the office of Vice President, Apache Pig, to
>        serve in accordance with and subject to the direction of the
>        Board of Directors and the Bylaws of the Foundation until
>        death, resignation, retirement, removal or disqualification,
>        or until a successor is appointed; and be it further
> 
>        RESOLVED, that the initial Apache Pig PMC be and hereby is
>        tasked with the creation of a set of bylaws intended to
>        encourage open development and increased participation in the
>        Apache Pig Project; and be it further
> 
>        RESOLVED, that the Apache Pig Project be and hereby
>        is tasked with the migration and rationalization of the Apache
>        Hadoop Pig sub-project; and be it further
> 
>        RESOLVED, that all responsibilities pertaining to the Apache
>        Hadoop Pig sub-project encumbered upon the
>        Apache Hadoop Project are hereafter discharged.
> 
> 
> 
> 


From general-return-2021-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 18:48:42 2010
Return-Path: <general-return-2021-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 65149 invoked from network); 24 Aug 2010 18:48:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 18:48:42 -0000
Received: (qmail 39811 invoked by uid 500); 24 Aug 2010 18:48:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39751 invoked by uid 500); 24 Aug 2010 18:48:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39742 invoked by uid 99); 24 Aug 2010 18:48:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 18:48:40 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 18:48:34 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7OIlDbc094603
	for <general@hadoop.apache.org>; Tue, 24 Aug 2010 11:47:14 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:in-reply-to:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:
	content-transfer-encoding:mime-version;
	b=YhD3PEI8Xz2d05V7ehO4n4xyZL4AW0iCNNbZLsqnq3DcRmscOeopuLhTYDRSzW+H
Received: from SP2-EX07VS04.ds.corp.yahoo.com ([98.137.59.32]) by
 SP2-EX07CAS01.ds.corp.yahoo.com ([98.137.59.37]) with mapi; Tue, 24 Aug 2010
 11:47:14 -0700
From: Mahadev Konar <mahadev@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 24 Aug 2010 11:47:12 -0700
Subject: Re: [VOTE] Pig to become a TLP
Thread-Topic: [VOTE] Pig to become a TLP
Thread-Index: ActDu7B1GRWe+ZaaTK6akDtuuABldAAARLfr
Message-ID: <C89961C0.3E24E%mahadev@yahoo-inc.com>
In-Reply-To: <74F969A5-B987-49EB-8547-9A1EAD165599@mac.com>
Accept-Language: en, en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en, en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

+1=20

Thanks
mahadev


On 8/24/10 11:38 AM, "Nigel Daley" <ndaley@mac.com> wrote:

> +1
>=20
> On Aug 23, 2010, at 10:38 AM, Alan Gates wrote:
>=20
>> I propose that Pig become a top level Apache project.
>>=20
>> The Pig development community has already voted on and approved this
>> proposal.  In summary, the community voted that all current active commi=
tters
>> (listed at http://hadoop.apache.org/pig/whoweare.html ) as of August 18 =
2010
>> should become PMC members, and that Olga Natkovich should be the PMC cha=
ir.
>> They also agreed that adopting bylaws should be among the first tasks of=
 the
>> new PMC.  Full details of the vote can be seen at http://bit.ly/c3GDyl
>>=20
>> Please vote on sending this proposal to the Apache Board.
>>=20
>> Below is a draft resolution to be sent to the Apache Board:
>>=20
>> Establish the Apache Pig Project
>>=20
>>        WHEREAS, the Board of Directors deems it to be in the best
>>        interests of the Foundation and consistent with the
>>        Foundation's purpose to establish a Project Management
>>        Committee charged with the creation and maintenance of
>>        open-source software related to parallel analysis of large
>>        data sets for distribution at no charge to the public.
>>=20
>>        NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>>        Committee (PMC), to be known as the "Apache Pig Project",
>>        be and hereby is established pursuant to Bylaws of the
>>        Foundation; and be it further
>>=20
>>        RESOLVED, that the Apache Pig Project be and hereby is
>>        responsible for the creation and maintenance of software
>>        related to parallel analysis of large data sets; and be
>>        it further
>>=20
>>        RESOLVED, that the office of "Vice President, Apache Pig" be
>>        and hereby is created, the person holding such office to
>>        serve at the direction of the Board of Directors as the chair
>>        of the Apache Pig Project, and to have primary responsibility
>>        for management of the projects within the scope of
>>        responsibility of the Apache Pig Project; and be it further
>>=20
>>        RESOLVED, that the persons listed immediately below be and
>>        hereby are appointed to serve as the initial members of the
>>        Apache Pig Project:
>>=20
>>          * Benjamin Reed          <breed@apache.org>
>>          * Daniel Dai                  <daijy@apache.org>
>>          * Alan Gates                 <gates@apache.org>
>>          * Giridharen Kesavan   <gkesavan@apache.org>
>>          * Olga Natkovich           <olga@apache.org>
>>          * Pradeep Kamath        <pradeepkth@apache.org>
>>          * Santhosh Srinivasan  <sms@apache.org>
>>          * Yan Zhou                   <yanz@apache.org>
>>          * Jeff Zhang                  <zjffdu@apache.org>
>>          * Ashutosh Chauhan     <hashutosh@apache.org>
>>          * Richard Ding               <rding@apache.org>
>>          * Dmitriy Ryaboy            <dvryaboy@apache.org>
>>          * Thejas Nair                  <thejas@apache.org>
>>=20
>>        NOW, THEREFORE, BE IT FURTHER RESOLVED, that Olga Natkovich
>>        be appointed to the office of Vice President, Apache Pig, to
>>        serve in accordance with and subject to the direction of the
>>        Board of Directors and the Bylaws of the Foundation until
>>        death, resignation, retirement, removal or disqualification,
>>        or until a successor is appointed; and be it further
>>=20
>>        RESOLVED, that the initial Apache Pig PMC be and hereby is
>>        tasked with the creation of a set of bylaws intended to
>>        encourage open development and increased participation in the
>>        Apache Pig Project; and be it further
>>=20
>>        RESOLVED, that the Apache Pig Project be and hereby
>>        is tasked with the migration and rationalization of the Apache
>>        Hadoop Pig sub-project; and be it further
>>=20
>>        RESOLVED, that all responsibilities pertaining to the Apache
>>        Hadoop Pig sub-project encumbered upon the
>>        Apache Hadoop Project are hereafter discharged.
>>=20
>>=20
>>=20
>>=20
>=20
>=20


From general-return-2022-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 19:02:57 2010
Return-Path: <general-return-2022-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70355 invoked from network); 24 Aug 2010 19:02:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 19:02:56 -0000
Received: (qmail 65576 invoked by uid 500); 24 Aug 2010 19:02:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65512 invoked by uid 500); 24 Aug 2010 19:02:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65504 invoked by uid 99); 24 Aug 2010 19:02:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 19:02:54 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 19:02:50 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout2.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7OJ1KgY043032
	for <general@hadoop.apache.org>; Tue, 24 Aug 2010 12:02:03 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=wGaT/IBk17xw/oy7F9Du/IZvELjZScrQBuijS2u5ZUxGAEeD1oZ9lEr0mWkFJDeg
Message-Id: <4ABE6BD0-39E7-404D-B093-1955733D3811@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <71D87762-A95B-434D-9FF6-8627AF65790A@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Pig to become a TLP
Date: Tue, 24 Aug 2010 11:59:47 -0700
References: <71D87762-A95B-434D-9FF6-8627AF65790A@yahoo-inc.com>
X-Mailer: Apple Mail (2.936)

+1

Arun

On Aug 23, 2010, at 10:38 AM, Alan Gates wrote:

> I propose that Pig become a top level Apache project.
>
> The Pig development community has already voted on and approved this
> proposal.  In summary, the community voted that all current active
> committers (listed at http://hadoop.apache.org/pig/whoweare.html ) as
> of August 18 2010 should become PMC members, and that Olga Natkovich
> should be the PMC chair.  They also agreed that adopting bylaws should
> be among the first tasks of the new PMC.  Full details of the vote can
> be seen at http://bit.ly/c3GDyl
>
> Please vote on sending this proposal to the Apache Board.
>
> Below is a draft resolution to be sent to the Apache Board:
>
> Establish the Apache Pig Project
>
>         WHEREAS, the Board of Directors deems it to be in the best
>         interests of the Foundation and consistent with the
>         Foundation's purpose to establish a Project Management
>         Committee charged with the creation and maintenance of
>         open-source software related to parallel analysis of large
>         data sets for distribution at no charge to the public.
>
>         NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>         Committee (PMC), to be known as the "Apache Pig Project",
>         be and hereby is established pursuant to Bylaws of the
>         Foundation; and be it further
>
>         RESOLVED, that the Apache Pig Project be and hereby is
>         responsible for the creation and maintenance of software
>         related to parallel analysis of large data sets; and be
>         it further
>
>         RESOLVED, that the office of "Vice President, Apache Pig" be
>         and hereby is created, the person holding such office to
>         serve at the direction of the Board of Directors as the chair
>         of the Apache Pig Project, and to have primary responsibility
>         for management of the projects within the scope of
>         responsibility of the Apache Pig Project; and be it further
>
>         RESOLVED, that the persons listed immediately below be and
>         hereby are appointed to serve as the initial members of the
>         Apache Pig Project:
>
>           * Benjamin Reed          <breed@apache.org>
>           * Daniel Dai                  <daijy@apache.org>
>           * Alan Gates                 <gates@apache.org>
>           * Giridharen Kesavan   <gkesavan@apache.org>
>           * Olga Natkovich           <olga@apache.org>
>           * Pradeep Kamath        <pradeepkth@apache.org>
>           * Santhosh Srinivasan  <sms@apache.org>
>           * Yan Zhou                   <yanz@apache.org>
>           * Jeff Zhang                  <zjffdu@apache.org>
>           * Ashutosh Chauhan     <hashutosh@apache.org>
>           * Richard Ding               <rding@apache.org>
>           * Dmitriy Ryaboy            <dvryaboy@apache.org>
>           * Thejas Nair                  <thejas@apache.org>
>
>         NOW, THEREFORE, BE IT FURTHER RESOLVED, that Olga Natkovich
>         be appointed to the office of Vice President, Apache Pig, to
>         serve in accordance with and subject to the direction of the
>         Board of Directors and the Bylaws of the Foundation until
>         death, resignation, retirement, removal or disqualification,
>         or until a successor is appointed; and be it further
>
>         RESOLVED, that the initial Apache Pig PMC be and hereby is
>         tasked with the creation of a set of bylaws intended to
>         encourage open development and increased participation in the
>         Apache Pig Project; and be it further
>
>         RESOLVED, that the Apache Pig Project be and hereby
>         is tasked with the migration and rationalization of the Apache
>         Hadoop Pig sub-project; and be it further
>
>         RESOLVED, that all responsibilities pertaining to the Apache
>         Hadoop Pig sub-project encumbered upon the
>         Apache Hadoop Project are hereafter discharged.
>
>
>
>


From general-return-2023-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 19:11:24 2010
Return-Path: <general-return-2023-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77677 invoked from network); 24 Aug 2010 19:11:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 19:11:23 -0000
Received: (qmail 82599 invoked by uid 500); 24 Aug 2010 19:11:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82531 invoked by uid 500); 24 Aug 2010 19:11:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82521 invoked by uid 99); 24 Aug 2010 19:11:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 19:11:21 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 24 Aug 2010 19:11:21 +0000
Received: (qmail 77543 invoked by uid 99); 24 Aug 2010 19:11:01 -0000
Received: from localhost.apache.org (HELO [192.168.168.106]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 19:11:01 +0000
Message-ID: <4C741944.1020907@apache.org>
Date: Tue, 24 Aug 2010 12:11:00 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.11) Gecko/20100713 Thunderbird/3.0.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Pig to become a TLP
References: <71D87762-A95B-434D-9FF6-8627AF65790A@yahoo-inc.com>
In-Reply-To: <71D87762-A95B-434D-9FF6-8627AF65790A@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

+1

Doug

On 08/23/2010 10:38 AM, Alan Gates wrote:
> I propose that Pig become a top level Apache project.
>
> The Pig development community has already voted on and approved this
> proposal. In summary, the community voted that all current active
> committers (listed at http://hadoop.apache.org/pig/whoweare.html ) as of
> August 18 2010 should become PMC members, and that Olga Natkovich should
> be the PMC chair. They also agreed that adopting bylaws should be among
> the first tasks of the new PMC. Full details of the vote can be seen at
> http://bit.ly/c3GDyl
>
> Please vote on sending this proposal to the Apache Board.
>
> Below is a draft resolution to be sent to the Apache Board:
>
> Establish the Apache Pig Project
>
> WHEREAS, the Board of Directors deems it to be in the best
> interests of the Foundation and consistent with the
> Foundation's purpose to establish a Project Management
> Committee charged with the creation and maintenance of
> open-source software related to parallel analysis of large
> data sets for distribution at no charge to the public.
>
> NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> Committee (PMC), to be known as the "Apache Pig Project",
> be and hereby is established pursuant to Bylaws of the
> Foundation; and be it further
>
> RESOLVED, that the Apache Pig Project be and hereby is
> responsible for the creation and maintenance of software
> related to parallel analysis of large data sets; and be
> it further
>
> RESOLVED, that the office of "Vice President, Apache Pig" be
> and hereby is created, the person holding such office to
> serve at the direction of the Board of Directors as the chair
> of the Apache Pig Project, and to have primary responsibility
> for management of the projects within the scope of
> responsibility of the Apache Pig Project; and be it further
>
> RESOLVED, that the persons listed immediately below be and
> hereby are appointed to serve as the initial members of the
> Apache Pig Project:
>
> * Benjamin Reed <breed@apache.org>
> * Daniel Dai <daijy@apache.org>
> * Alan Gates <gates@apache.org>
> * Giridharen Kesavan <gkesavan@apache.org>
> * Olga Natkovich <olga@apache.org>
> * Pradeep Kamath <pradeepkth@apache.org>
> * Santhosh Srinivasan <sms@apache.org>
> * Yan Zhou <yanz@apache.org>
> * Jeff Zhang <zjffdu@apache.org>
> * Ashutosh Chauhan <hashutosh@apache.org>
> * Richard Ding <rding@apache.org>
> * Dmitriy Ryaboy <dvryaboy@apache.org>
> * Thejas Nair <thejas@apache.org>
>
> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Olga Natkovich
> be appointed to the office of Vice President, Apache Pig, to
> serve in accordance with and subject to the direction of the
> Board of Directors and the Bylaws of the Foundation until
> death, resignation, retirement, removal or disqualification,
> or until a successor is appointed; and be it further
>
> RESOLVED, that the initial Apache Pig PMC be and hereby is
> tasked with the creation of a set of bylaws intended to
> encourage open development and increased participation in the
> Apache Pig Project; and be it further
>
> RESOLVED, that the Apache Pig Project be and hereby
> is tasked with the migration and rationalization of the Apache
> Hadoop Pig sub-project; and be it further
>
> RESOLVED, that all responsibilities pertaining to the Apache
> Hadoop Pig sub-project encumbered upon the
> Apache Hadoop Project are hereafter discharged.
>
>
>
>

From general-return-2024-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 19:13:46 2010
Return-Path: <general-return-2024-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78580 invoked from network); 24 Aug 2010 19:13:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 19:13:45 -0000
Received: (qmail 84916 invoked by uid 500); 24 Aug 2010 19:13:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84870 invoked by uid 500); 24 Aug 2010 19:13:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84862 invoked by uid 99); 24 Aug 2010 19:13:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 19:13:44 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of teamphy6@gmail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 19:13:39 +0000
Received: by iwn9 with SMTP id 9so5941068iwn.35
        for <general@hadoop.apache.org>; Tue, 24 Aug 2010 12:13:18 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=RPw494Zcg4y4eyhaxvXaTYSivvJbtRsXTo8Ww+Alr4Y=;
        b=BlfWGXpPhJ6n6aSs+XPCx6LPSX6geF5f+s/T/mdagBheUZXu7HF0zjAXMFcEcKvmAm
         cTMVvVh4x6WTxFYJXbTypHvCpS+EDfnGZcZylbReUZmn1AsLKdSg9imkdgiFASOR15YO
         9aAbYMfzLjiND2m7/HRTOWeSd9SEP4rfMwVA0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=tDKH4gkyhfAiX79BpqdS9jog5ioXiOlHhj+AkoBMdmz2I/51/maFZGYX9iu6kGrWSO
         XGpZBRG/LdUrHfmXB/DOL0MbqwfFhth6sG8qj9F0+HQStIomJX0hqBn/415nx+aU3QVm
         0VB7yQg5OsMDhGqw9I9x3bIlvx6TaxeqWVWzg=
MIME-Version: 1.0
Received: by 10.231.146.141 with SMTP id h13mr8780073ibv.1.1282677198483; Tue,
 24 Aug 2010 12:13:18 -0700 (PDT)
Received: by 10.231.36.9 with HTTP; Tue, 24 Aug 2010 12:13:18 -0700 (PDT)
In-Reply-To: <4C741944.1020907@apache.org>
References: <71D87762-A95B-434D-9FF6-8627AF65790A@yahoo-inc.com>
	<4C741944.1020907@apache.org>
Date: Tue, 24 Aug 2010 15:13:18 -0400
Message-ID: <AANLkTikv7f9O94evzBnE6uKNbWKUJ0hUbm7nhaKVCkTs@mail.gmail.com>
Subject: Re: [VOTE] Pig to become a TLP
From: Daniel Jue <teamphy6@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

+1

On Tue, Aug 24, 2010 at 3:11 PM, Doug Cutting <cutting@apache.org> wrote:
> +1
>
> Doug
>
> On 08/23/2010 10:38 AM, Alan Gates wrote:
>>
>> I propose that Pig become a top level Apache project.
>>
>> The Pig development community has already voted on and approved this
>> proposal. In summary, the community voted that all current active
>> committers (listed at http://hadoop.apache.org/pig/whoweare.html ) as of
>> August 18 2010 should become PMC members, and that Olga Natkovich should
>> be the PMC chair. They also agreed that adopting bylaws should be among
>> the first tasks of the new PMC. Full details of the vote can be seen at
>> http://bit.ly/c3GDyl
>>
>> Please vote on sending this proposal to the Apache Board.
>>
>> Below is a draft resolution to be sent to the Apache Board:
>>
>> Establish the Apache Pig Project
>>
>> WHEREAS, the Board of Directors deems it to be in the best
>> interests of the Foundation and consistent with the
>> Foundation's purpose to establish a Project Management
>> Committee charged with the creation and maintenance of
>> open-source software related to parallel analysis of large
>> data sets for distribution at no charge to the public.
>>
>> NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>> Committee (PMC), to be known as the "Apache Pig Project",
>> be and hereby is established pursuant to Bylaws of the
>> Foundation; and be it further
>>
>> RESOLVED, that the Apache Pig Project be and hereby is
>> responsible for the creation and maintenance of software
>> related to parallel analysis of large data sets; and be
>> it further
>>
>> RESOLVED, that the office of "Vice President, Apache Pig" be
>> and hereby is created, the person holding such office to
>> serve at the direction of the Board of Directors as the chair
>> of the Apache Pig Project, and to have primary responsibility
>> for management of the projects within the scope of
>> responsibility of the Apache Pig Project; and be it further
>>
>> RESOLVED, that the persons listed immediately below be and
>> hereby are appointed to serve as the initial members of the
>> Apache Pig Project:
>>
>> * Benjamin Reed <breed@apache.org>
>> * Daniel Dai <daijy@apache.org>
>> * Alan Gates <gates@apache.org>
>> * Giridharen Kesavan <gkesavan@apache.org>
>> * Olga Natkovich <olga@apache.org>
>> * Pradeep Kamath <pradeepkth@apache.org>
>> * Santhosh Srinivasan <sms@apache.org>
>> * Yan Zhou <yanz@apache.org>
>> * Jeff Zhang <zjffdu@apache.org>
>> * Ashutosh Chauhan <hashutosh@apache.org>
>> * Richard Ding <rding@apache.org>
>> * Dmitriy Ryaboy <dvryaboy@apache.org>
>> * Thejas Nair <thejas@apache.org>
>>
>> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Olga Natkovich
>> be appointed to the office of Vice President, Apache Pig, to
>> serve in accordance with and subject to the direction of the
>> Board of Directors and the Bylaws of the Foundation until
>> death, resignation, retirement, removal or disqualification,
>> or until a successor is appointed; and be it further
>>
>> RESOLVED, that the initial Apache Pig PMC be and hereby is
>> tasked with the creation of a set of bylaws intended to
>> encourage open development and increased participation in the
>> Apache Pig Project; and be it further
>>
>> RESOLVED, that the Apache Pig Project be and hereby
>> is tasked with the migration and rationalization of the Apache
>> Hadoop Pig sub-project; and be it further
>>
>> RESOLVED, that all responsibilities pertaining to the Apache
>> Hadoop Pig sub-project encumbered upon the
>> Apache Hadoop Project are hereafter discharged.
>>
>>
>>
>>
>

From general-return-2025-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 19:19:44 2010
Return-Path: <general-return-2025-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 80714 invoked from network); 24 Aug 2010 19:19:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 19:19:44 -0000
Received: (qmail 95557 invoked by uid 500); 24 Aug 2010 19:19:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95462 invoked by uid 500); 24 Aug 2010 19:19:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95442 invoked by uid 99); 24 Aug 2010 19:19:42 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 19:19:42 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 19:19:19 +0000
Received: by bwz14 with SMTP id 14so9653bwz.35
        for <multiple recipients>; Tue, 24 Aug 2010 12:18:58 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.113.209 with SMTP id b17mr5068020bkq.19.1282677538727;
 Tue, 24 Aug 2010 12:18:58 -0700 (PDT)
Received: by 10.204.179.74 with HTTP; Tue, 24 Aug 2010 12:18:58 -0700 (PDT)
Date: Tue, 24 Aug 2010 12:18:58 -0700
Message-ID: <AANLkTimpwY39vhs577N5gfgufSBdDZsBd-81QHH0+zR4@mail.gmail.com>
Subject: [ANNOUNCE] Apache Hadoop 0.21.0 released
From: Tom White <tom@cloudera.com>
To: general@hadoop.apache.org, common-user <common-user@hadoop.apache.org>, 
	common-dev <common-dev@hadoop.apache.org>
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hi everyone,

I am pleased to announce that Apache Hadoop 0.21.0 is available for
download from http://hadoop.apache.org/common/releases.html.

Over 1300 issues have been addressed since 0.20.2; you can find details at

http://hadoop.apache.org/common/docs/r0.21.0/releasenotes.html
http://hadoop.apache.org/hdfs/docs/r0.21.0/releasenotes.html
http://hadoop.apache.org/mapreduce/docs/r0.21.0/releasenotes.html

Please note that this release has not undergone testing at scale and
should not be considered stable or suitable for production. It is
being classified as a minor release, which means that it should be API
compatible with 0.20.2.

Thanks to all who contributed to this release!

Tom

From general-return-2026-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 20:03:52 2010
Return-Path: <general-return-2026-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94633 invoked from network); 24 Aug 2010 20:03:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 20:03:51 -0000
Received: (qmail 54592 invoked by uid 500); 24 Aug 2010 20:03:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54522 invoked by uid 500); 24 Aug 2010 20:03:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54514 invoked by uid 99); 24 Aug 2010 20:03:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 20:03:49 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 20:03:45 +0000
Received: by bwz14 with SMTP id 14so57835bwz.35
        for <general@hadoop.apache.org>; Tue, 24 Aug 2010 13:03:23 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.27.20 with SMTP id g20mr5076305bkc.114.1282680203541; Tue,
 24 Aug 2010 13:03:23 -0700 (PDT)
Received: by 10.204.179.74 with HTTP; Tue, 24 Aug 2010 13:03:23 -0700 (PDT)
In-Reply-To: <71D87762-A95B-434D-9FF6-8627AF65790A@yahoo-inc.com>
References: <71D87762-A95B-434D-9FF6-8627AF65790A@yahoo-inc.com>
Date: Tue, 24 Aug 2010 13:03:23 -0700
Message-ID: <AANLkTi=+76EVHBU_ZK8T06B5U6dM+wG54o+fy=vT+YNG@mail.gmail.com>
Subject: Re: [VOTE] Pig to become a TLP
From: Tom White <tom@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

+1

Tom

On Mon, Aug 23, 2010 at 10:38 AM, Alan Gates <gates@yahoo-inc.com> wrote:
> I propose that Pig become a top level Apache project.
>
> The Pig development community has already voted on and approved this
> proposal. =A0In summary, the community voted that all current active
> committers (listed at http://hadoop.apache.org/pig/whoweare.html ) as of
> August 18 2010 should become PMC members, and that Olga Natkovich should =
be
> the PMC chair. =A0They also agreed that adopting bylaws should be among t=
he
> first tasks of the new PMC. =A0Full details of the vote can be seen at
> http://bit.ly/c3GDyl
>
> Please vote on sending this proposal to the Apache Board.
>
> Below is a draft resolution to be sent to the Apache Board:
>
> Establish the Apache Pig Project
>
> =A0 =A0 =A0 =A0WHEREAS, the Board of Directors deems it to be in the best
> =A0 =A0 =A0 =A0interests of the Foundation and consistent with the
> =A0 =A0 =A0 =A0Foundation's purpose to establish a Project Management
> =A0 =A0 =A0 =A0Committee charged with the creation and maintenance of
> =A0 =A0 =A0 =A0open-source software related to parallel analysis of large
> =A0 =A0 =A0 =A0data sets for distribution at no charge to the public.
>
> =A0 =A0 =A0 =A0NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> =A0 =A0 =A0 =A0Committee (PMC), to be known as the "Apache Pig Project",
> =A0 =A0 =A0 =A0be and hereby is established pursuant to Bylaws of the
> =A0 =A0 =A0 =A0Foundation; and be it further
>
> =A0 =A0 =A0 =A0RESOLVED, that the Apache Pig Project be and hereby is
> =A0 =A0 =A0 =A0responsible for the creation and maintenance of software
> =A0 =A0 =A0 =A0related to parallel analysis of large data sets; and be
> =A0 =A0 =A0 =A0it further
>
> =A0 =A0 =A0 =A0RESOLVED, that the office of "Vice President, Apache Pig" =
be
> =A0 =A0 =A0 =A0and hereby is created, the person holding such office to
> =A0 =A0 =A0 =A0serve at the direction of the Board of Directors as the ch=
air
> =A0 =A0 =A0 =A0of the Apache Pig Project, and to have primary responsibil=
ity
> =A0 =A0 =A0 =A0for management of the projects within the scope of
> =A0 =A0 =A0 =A0responsibility of the Apache Pig Project; and be it furthe=
r
>
> =A0 =A0 =A0 =A0RESOLVED, that the persons listed immediately below be and
> =A0 =A0 =A0 =A0hereby are appointed to serve as the initial members of th=
e
> =A0 =A0 =A0 =A0Apache Pig Project:
>
> =A0 =A0 =A0 =A0 =A0* Benjamin Reed =A0 =A0 =A0 =A0 =A0<breed@apache.org>
> =A0 =A0 =A0 =A0 =A0* Daniel Dai =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0<daijy=
@apache.org>
> =A0 =A0 =A0 =A0 =A0* Alan Gates =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 <gates@ap=
ache.org>
> =A0 =A0 =A0 =A0 =A0* Giridharen Kesavan =A0 <gkesavan@apache.org>
> =A0 =A0 =A0 =A0 =A0* Olga Natkovich =A0 =A0 =A0 =A0 =A0 <olga@apache.org>
> =A0 =A0 =A0 =A0 =A0* Pradeep Kamath =A0 =A0 =A0 =A0<pradeepkth@apache.org=
>
> =A0 =A0 =A0 =A0 =A0* Santhosh Srinivasan =A0<sms@apache.org>
> =A0 =A0 =A0 =A0 =A0* Yan Zhou =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 <yanz@a=
pache.org>
> =A0 =A0 =A0 =A0 =A0* Jeff Zhang =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0<zjffd=
u@apache.org>
> =A0 =A0 =A0 =A0 =A0* Ashutosh Chauhan =A0 =A0 <hashutosh@apache.org>
> =A0 =A0 =A0 =A0 =A0* Richard Ding =A0 =A0 =A0 =A0 =A0 =A0 =A0 <rding@apac=
he.org>
> =A0 =A0 =A0 =A0 =A0* Dmitriy Ryaboy =A0 =A0 =A0 =A0 =A0 =A0<dvryaboy@apac=
he.org>
> =A0 =A0 =A0 =A0 =A0* Thejas Nair =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0<thej=
as@apache.org>
>
> =A0 =A0 =A0 =A0NOW, THEREFORE, BE IT FURTHER RESOLVED, that Olga Natkovic=
h
> =A0 =A0 =A0 =A0be appointed to the office of Vice President, Apache Pig, =
to
> =A0 =A0 =A0 =A0serve in accordance with and subject to the direction of t=
he
> =A0 =A0 =A0 =A0Board of Directors and the Bylaws of the Foundation until
> =A0 =A0 =A0 =A0death, resignation, retirement, removal or disqualificatio=
n,
> =A0 =A0 =A0 =A0or until a successor is appointed; and be it further
>
> =A0 =A0 =A0 =A0RESOLVED, that the initial Apache Pig PMC be and hereby is
> =A0 =A0 =A0 =A0tasked with the creation of a set of bylaws intended to
> =A0 =A0 =A0 =A0encourage open development and increased participation in =
the
> =A0 =A0 =A0 =A0Apache Pig Project; and be it further
>
> =A0 =A0 =A0 =A0RESOLVED, that the Apache Pig Project be and hereby
> =A0 =A0 =A0 =A0is tasked with the migration and rationalization of the Ap=
ache
> =A0 =A0 =A0 =A0Hadoop Pig sub-project; and be it further
>
> =A0 =A0 =A0 =A0RESOLVED, that all responsibilities pertaining to the Apac=
he
> =A0 =A0 =A0 =A0Hadoop Pig sub-project encumbered upon the
> =A0 =A0 =A0 =A0Apache Hadoop Project are hereafter discharged.
>
>
>
>
>

From general-return-2027-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 20:32:03 2010
Return-Path: <general-return-2027-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9507 invoked from network); 24 Aug 2010 20:32:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 20:32:02 -0000
Received: (qmail 88799 invoked by uid 500); 24 Aug 2010 20:32:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88748 invoked by uid 500); 24 Aug 2010 20:32:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88740 invoked by uid 99); 24 Aug 2010 20:32:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 20:32:00 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 20:31:55 +0000
Received: from localhost (wlanvpn-snve-247-55.corp.yahoo.com [172.21.149.55])
	by mrout2.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7OKVDFv074503
	for <general@hadoop.apache.org>; Tue, 24 Aug 2010 13:31:13 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=date:from:to:subject:message-id:references:mime-version:
	content-type:content-disposition:in-reply-to:x-organization:user-agent;
	b=LJw1hx6WikqBKGw15I1OJINgD5LX0wJ2A9fYBdMwyPTEddJtXRyPX0/KnrwmSyID
Date: Tue, 24 Aug 2010 13:31:13 -0700
From: Konstantin Boudnik <cos@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: Branching and testing strategy for 0.22
Message-ID: <20100824203112.GE27700@wlanvpn-snve-247-55.corp.yahoo.com>
References: <ACC0BDF0-C9F5-4DBA-90C4-5B94A01CD968@apache.org>
 <20100823231705.GB26339@goodenter-lm.local>
 <115EFBD4-93A6-4388-9F72-D6E6E6693E0E@apache.org>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature"; boundary="UugvWAfsgieZRqgk"
Content-Disposition: inline
In-Reply-To: <115EFBD4-93A6-4388-9F72-D6E6E6693E0E@apache.org>
X-Organization: Yahoo! Hadoop (Grid computing)
User-Agent: Mutt/1.5.20 (2009-06-14)

--UugvWAfsgieZRqgk
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

As we have briefly discussed off-line I'll try to reiterate the assumptions
you have expressed:

 - testing cycle on a qa branch won't be reset after a blocker (B1) is fixe=
d.
   The validation is continued from the point where the B1 has been
   discovered. In the real cluster test env. it means that only fresh
   deployment needs to be done.
 - in some cases a judgment call has to be made on whether or not testing
   cycle needs to be reset (e.g. in cases of massive fixes and so on). The
   chances are, however, that critical fixes are usually isolated and don't
   affect the rest of the code base.

It isn't like I disagree with the assumptions above. They are valid... under
two conditions:
 - a test harness in use (i.e. a team of test engineers running on manual
   testing errand or an automated solution similar to Herriot) supports the
   notion of interrupted test cycle
 - there's a sound regression tests suite which can guarantee the a fix B2
   doesn't invalidate fix B1 from before.

Cos

On Tue, Aug 24, 2010 at 12:08AM, Owen O'Malley wrote:
>=20
> On Aug 23, 2010, at 4:17 PM, Konstantin Boudnik wrote:
>=20
> > If I may... why QA need source code branches rather than a =20
> > sequential builds
> > from the trunk (as it usually done)?
>=20
> For QA to be effective, they can't handle a different artifact every =20
> day with a new set of features. So the plan is to take a snapshot and =20
> test that. However, if bugs are found that block testing they need =20
> precisely those bugs fixed with out any additional features. So you =20
> need branches to support that. So a sample cycle would look like:
>=20
> 1. create build-1 and test it, discovering bugs B1, B2, and B3. Only =20
> B1 is blocking further testing.
> 2. meanwhile, someone has checked in feature F1 into trunk
> 3. B1 is fixed in the branch and trunk. B2 and B3 are just fixed in =20
> trunk
> 4. when build-2 is built it only has B1's fix
> 5. at the end of the cycle, we rebase the branch to trunk and pickup =20
> B1, B2, B3 and F1 into build-3.
>=20
> If you don't have a branch, you would get F1 whether you want it or not.
>=20
> -- Owen

--UugvWAfsgieZRqgk
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)

iF4EAREIAAYFAkx0LBAACgkQMqXifkwDoaHNuAD8Cei/kjJXhW1IN15koGg044Tf
UoKLsei+ydMpiyAYBRkA/iSzK9vrO0hkOgCsS4T6ZcbKs7BbdfC3I0Nq8aVyrx4n
=Q8j3
-----END PGP SIGNATURE-----

--UugvWAfsgieZRqgk--

From general-return-2028-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 20:49:26 2010
Return-Path: <general-return-2028-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14571 invoked from network); 24 Aug 2010 20:49:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 20:49:26 -0000
Received: (qmail 6894 invoked by uid 500); 24 Aug 2010 20:49:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6827 invoked by uid 500); 24 Aug 2010 20:49:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6819 invoked by uid 99); 24 Aug 2010 20:49:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 20:49:24 +0000
X-ASF-Spam-Status: No, hits=2.7 required=10.0
	tests=RCVD_ILLEGAL_IP,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of athusoo@facebook.com designates 69.63.184.102 as permitted sender)
Received: from [69.63.184.102] (HELO mx-out.facebook.com) (69.63.184.102)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 20:49:19 +0000
Received: from [10.18.255.134] ([10.18.255.134:4219] helo=mail.thefacebook.com)
	by mta003.ash1.facebook.com (envelope-from <athusoo@facebook.com>)
	(ecelerity 2.2.2.45 r(34067)) with ESMTP
	id AD/18-20904-A30347C4; Tue, 24 Aug 2010 13:48:59 -0700
Received: from SC-MBX03.TheFacebook.com ([169.254.2.176]) by
 sc-hub04.TheFacebook.com ([fe80::8df5:7f90:d4a0:bb9%11]) with mapi; Tue, 24
 Aug 2010 13:48:57 -0700
From: Ashish Thusoo <athusoo@facebook.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: RE: [VOTE] Pig to become a TLP
Thread-Topic: [VOTE] Pig to become a TLP
Thread-Index: AQHLQuoVKzGr9GIS4k+nEBDJK0qH/JLxflSA//+XWeA=
Date: Tue, 24 Aug 2010 20:48:56 +0000
Message-ID: <09370B27F829314182C33A219530C104037A5C47@sc-mbx03.TheFacebook.com>
References: <71D87762-A95B-434D-9FF6-8627AF65790A@yahoo-inc.com>
 <AANLkTi=+76EVHBU_ZK8T06B5U6dM+wG54o+fy=vT+YNG@mail.gmail.com>
In-Reply-To: <AANLkTi=+76EVHBU_ZK8T06B5U6dM+wG54o+fy=vT+YNG@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

+1

Ashish=20

-----Original Message-----
From: Tom White [mailto:tom@cloudera.com]=20
Sent: Tuesday, August 24, 2010 1:03 PM
To: general@hadoop.apache.org
Subject: Re: [VOTE] Pig to become a TLP

+1

Tom

On Mon, Aug 23, 2010 at 10:38 AM, Alan Gates <gates@yahoo-inc.com> wrote:
> I propose that Pig become a top level Apache project.
>
> The Pig development community has already voted on and approved this=20
> proposal. =A0In summary, the community voted that all current active=20
> committers (listed at http://hadoop.apache.org/pig/whoweare.html ) as=20
> of August 18 2010 should become PMC members, and that Olga Natkovich=20
> should be the PMC chair. =A0They also agreed that adopting bylaws should=
=20
> be among the first tasks of the new PMC. =A0Full details of the vote can=
=20
> be seen at http://bit.ly/c3GDyl
>
> Please vote on sending this proposal to the Apache Board.
>
> Below is a draft resolution to be sent to the Apache Board:
>
> Establish the Apache Pig Project
>
> =A0 =A0 =A0 =A0WHEREAS, the Board of Directors deems it to be in the best
> =A0 =A0 =A0 =A0interests of the Foundation and consistent with the
> =A0 =A0 =A0 =A0Foundation's purpose to establish a Project Management
> =A0 =A0 =A0 =A0Committee charged with the creation and maintenance of
> =A0 =A0 =A0 =A0open-source software related to parallel analysis of large
> =A0 =A0 =A0 =A0data sets for distribution at no charge to the public.
>
> =A0 =A0 =A0 =A0NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> =A0 =A0 =A0 =A0Committee (PMC), to be known as the "Apache Pig Project",
> =A0 =A0 =A0 =A0be and hereby is established pursuant to Bylaws of the
> =A0 =A0 =A0 =A0Foundation; and be it further
>
> =A0 =A0 =A0 =A0RESOLVED, that the Apache Pig Project be and hereby is
> =A0 =A0 =A0 =A0responsible for the creation and maintenance of software
> =A0 =A0 =A0 =A0related to parallel analysis of large data sets; and be
> =A0 =A0 =A0 =A0it further
>
> =A0 =A0 =A0 =A0RESOLVED, that the office of "Vice President, Apache Pig" =
be
> =A0 =A0 =A0 =A0and hereby is created, the person holding such office to
> =A0 =A0 =A0 =A0serve at the direction of the Board of Directors as the ch=
air
> =A0 =A0 =A0 =A0of the Apache Pig Project, and to have primary responsibil=
ity
> =A0 =A0 =A0 =A0for management of the projects within the scope of
> =A0 =A0 =A0 =A0responsibility of the Apache Pig Project; and be it furthe=
r
>
> =A0 =A0 =A0 =A0RESOLVED, that the persons listed immediately below be and
> =A0 =A0 =A0 =A0hereby are appointed to serve as the initial members of th=
e
> =A0 =A0 =A0 =A0Apache Pig Project:
>
> =A0 =A0 =A0 =A0 =A0* Benjamin Reed =A0 =A0 =A0 =A0 =A0<breed@apache.org>
> =A0 =A0 =A0 =A0 =A0* Daniel Dai =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0<daijy=
@apache.org>
> =A0 =A0 =A0 =A0 =A0* Alan Gates =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 <gates@ap=
ache.org>
> =A0 =A0 =A0 =A0 =A0* Giridharen Kesavan =A0 <gkesavan@apache.org>
> =A0 =A0 =A0 =A0 =A0* Olga Natkovich =A0 =A0 =A0 =A0 =A0 <olga@apache.org>
> =A0 =A0 =A0 =A0 =A0* Pradeep Kamath =A0 =A0 =A0 =A0<pradeepkth@apache.org=
>
> =A0 =A0 =A0 =A0 =A0* Santhosh Srinivasan =A0<sms@apache.org>
> =A0 =A0 =A0 =A0 =A0* Yan Zhou =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 <yanz@a=
pache.org>
> =A0 =A0 =A0 =A0 =A0* Jeff Zhang =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0<zjffd=
u@apache.org>
> =A0 =A0 =A0 =A0 =A0* Ashutosh Chauhan =A0 =A0 <hashutosh@apache.org>
> =A0 =A0 =A0 =A0 =A0* Richard Ding =A0 =A0 =A0 =A0 =A0 =A0 =A0 <rding@apac=
he.org>
> =A0 =A0 =A0 =A0 =A0* Dmitriy Ryaboy =A0 =A0 =A0 =A0 =A0 =A0<dvryaboy@apac=
he.org>
> =A0 =A0 =A0 =A0 =A0* Thejas Nair =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0<thej=
as@apache.org>
>
> =A0 =A0 =A0 =A0NOW, THEREFORE, BE IT FURTHER RESOLVED, that Olga Natkovic=
h
> =A0 =A0 =A0 =A0be appointed to the office of Vice President, Apache Pig, =
to
> =A0 =A0 =A0 =A0serve in accordance with and subject to the direction of t=
he
> =A0 =A0 =A0 =A0Board of Directors and the Bylaws of the Foundation until
> =A0 =A0 =A0 =A0death, resignation, retirement, removal or disqualificatio=
n,
> =A0 =A0 =A0 =A0or until a successor is appointed; and be it further
>
> =A0 =A0 =A0 =A0RESOLVED, that the initial Apache Pig PMC be and hereby is
> =A0 =A0 =A0 =A0tasked with the creation of a set of bylaws intended to
> =A0 =A0 =A0 =A0encourage open development and increased participation in =
the
> =A0 =A0 =A0 =A0Apache Pig Project; and be it further
>
> =A0 =A0 =A0 =A0RESOLVED, that the Apache Pig Project be and hereby
> =A0 =A0 =A0 =A0is tasked with the migration and rationalization of the Ap=
ache
> =A0 =A0 =A0 =A0Hadoop Pig sub-project; and be it further
>
> =A0 =A0 =A0 =A0RESOLVED, that all responsibilities pertaining to the Apac=
he
> =A0 =A0 =A0 =A0Hadoop Pig sub-project encumbered upon the
> =A0 =A0 =A0 =A0Apache Hadoop Project are hereafter discharged.
>
>
>
>
>

From general-return-2029-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 21:04:53 2010
Return-Path: <general-return-2029-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 18029 invoked from network); 24 Aug 2010 21:04:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 21:04:52 -0000
Received: (qmail 22420 invoked by uid 500); 24 Aug 2010 21:04:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22367 invoked by uid 500); 24 Aug 2010 21:04:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22359 invoked by uid 99); 24 Aug 2010 21:04:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 21:04:50 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 21:04:46 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7OL2xss093502
	for <general@hadoop.apache.org>; Tue, 24 Aug 2010 14:02:59 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=mikU34/rmIfn8ZD2AN7UfTuO9J3a9ipkYaa0XbrqaHhc6XBuuTqUVUSBqqMCj1Rj
Message-Id: <9A06CFFB-56BD-4E1B-A6C3-13A35C6B9447@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <09370B27F829314182C33A219530C1040379C139@sc-mbx03.TheFacebook.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSS] Hive as TLP
Date: Tue, 24 Aug 2010 14:02:58 -0700
References: <09370B27F829314182C33A219530C1040379C139@sc-mbx03.TheFacebook.com>
X-Mailer: Apple Mail (2.936)

I think this is a reasonable direction. Hive clearly has a strong user- 
base and is a distinct community.

+1

Arun

On Aug 20, 2010, at 5:05 PM, Ashish Thusoo wrote:

> The Hive subproject has voted to become a TLP
>
> http://bit.ly/9nb4nN
>
> Does the Hadoop community have any questions or concerns on this? I  
> will be calling a more formal vote after this discussion.
>
> The Hive dev community is still dominated by Facebook but the  
> community is working hard to diversify the base and hopes to add  
> committers from Yahoo and Cloudera. We anticipate that we will have  
> a more diversified base by the end of the year modulo contributions  
> from developers at these entities - and there are a fair bit in the  
> pipeline.
>
> Thanks,
> Ashish


From general-return-2030-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 22:18:56 2010
Return-Path: <general-return-2030-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 62851 invoked from network); 24 Aug 2010 22:18:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 22:18:56 -0000
Received: (qmail 11659 invoked by uid 500); 24 Aug 2010 22:18:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11580 invoked by uid 500); 24 Aug 2010 22:18:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11572 invoked by uid 99); 24 Aug 2010 22:18:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 22:18:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 24 Aug 2010 22:18:53 +0000
Received: (qmail 62707 invoked by uid 99); 24 Aug 2010 22:18:33 -0000
Received: from localhost.apache.org (HELO mail-iw0-f176.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 22:18:33 +0000
Received: by iwn9 with SMTP id 9so6157615iwn.35
        for <general@hadoop.apache.org>; Tue, 24 Aug 2010 15:18:31 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.172.75 with SMTP id k11mr9376616ibz.4.1282688311786; Tue,
 24 Aug 2010 15:18:31 -0700 (PDT)
Received: by 10.231.144.198 with HTTP; Tue, 24 Aug 2010 15:18:31 -0700 (PDT)
In-Reply-To: <71D87762-A95B-434D-9FF6-8627AF65790A@yahoo-inc.com>
References: <71D87762-A95B-434D-9FF6-8627AF65790A@yahoo-inc.com>
Date: Tue, 24 Aug 2010 15:18:31 -0700
Message-ID: <AANLkTi=Cg2gCoq9Eg47bPLXSXdU4YsRC7NWYCTA_-wY2@mail.gmail.com>
Subject: Re: [VOTE] Pig to become a TLP
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

+1 -C

On Mon, Aug 23, 2010 at 10:38 AM, Alan Gates <gates@yahoo-inc.com> wrote:
> I propose that Pig become a top level Apache project.
>
> The Pig development community has already voted on and approved this
> proposal. =A0In summary, the community voted that all current active
> committers (listed at http://hadoop.apache.org/pig/whoweare.html ) as of
> August 18 2010 should become PMC members, and that Olga Natkovich should =
be
> the PMC chair. =A0They also agreed that adopting bylaws should be among t=
he
> first tasks of the new PMC. =A0Full details of the vote can be seen at
> http://bit.ly/c3GDyl
>
> Please vote on sending this proposal to the Apache Board.
>
> Below is a draft resolution to be sent to the Apache Board:
>
> Establish the Apache Pig Project
>
> =A0 =A0 =A0 =A0WHEREAS, the Board of Directors deems it to be in the best
> =A0 =A0 =A0 =A0interests of the Foundation and consistent with the
> =A0 =A0 =A0 =A0Foundation's purpose to establish a Project Management
> =A0 =A0 =A0 =A0Committee charged with the creation and maintenance of
> =A0 =A0 =A0 =A0open-source software related to parallel analysis of large
> =A0 =A0 =A0 =A0data sets for distribution at no charge to the public.
>
> =A0 =A0 =A0 =A0NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> =A0 =A0 =A0 =A0Committee (PMC), to be known as the "Apache Pig Project",
> =A0 =A0 =A0 =A0be and hereby is established pursuant to Bylaws of the
> =A0 =A0 =A0 =A0Foundation; and be it further
>
> =A0 =A0 =A0 =A0RESOLVED, that the Apache Pig Project be and hereby is
> =A0 =A0 =A0 =A0responsible for the creation and maintenance of software
> =A0 =A0 =A0 =A0related to parallel analysis of large data sets; and be
> =A0 =A0 =A0 =A0it further
>
> =A0 =A0 =A0 =A0RESOLVED, that the office of "Vice President, Apache Pig" =
be
> =A0 =A0 =A0 =A0and hereby is created, the person holding such office to
> =A0 =A0 =A0 =A0serve at the direction of the Board of Directors as the ch=
air
> =A0 =A0 =A0 =A0of the Apache Pig Project, and to have primary responsibil=
ity
> =A0 =A0 =A0 =A0for management of the projects within the scope of
> =A0 =A0 =A0 =A0responsibility of the Apache Pig Project; and be it furthe=
r
>
> =A0 =A0 =A0 =A0RESOLVED, that the persons listed immediately below be and
> =A0 =A0 =A0 =A0hereby are appointed to serve as the initial members of th=
e
> =A0 =A0 =A0 =A0Apache Pig Project:
>
> =A0 =A0 =A0 =A0 =A0* Benjamin Reed =A0 =A0 =A0 =A0 =A0<breed@apache.org>
> =A0 =A0 =A0 =A0 =A0* Daniel Dai =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0<daijy=
@apache.org>
> =A0 =A0 =A0 =A0 =A0* Alan Gates =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 <gates@ap=
ache.org>
> =A0 =A0 =A0 =A0 =A0* Giridharen Kesavan =A0 <gkesavan@apache.org>
> =A0 =A0 =A0 =A0 =A0* Olga Natkovich =A0 =A0 =A0 =A0 =A0 <olga@apache.org>
> =A0 =A0 =A0 =A0 =A0* Pradeep Kamath =A0 =A0 =A0 =A0<pradeepkth@apache.org=
>
> =A0 =A0 =A0 =A0 =A0* Santhosh Srinivasan =A0<sms@apache.org>
> =A0 =A0 =A0 =A0 =A0* Yan Zhou =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 <yanz@a=
pache.org>
> =A0 =A0 =A0 =A0 =A0* Jeff Zhang =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0<zjffd=
u@apache.org>
> =A0 =A0 =A0 =A0 =A0* Ashutosh Chauhan =A0 =A0 <hashutosh@apache.org>
> =A0 =A0 =A0 =A0 =A0* Richard Ding =A0 =A0 =A0 =A0 =A0 =A0 =A0 <rding@apac=
he.org>
> =A0 =A0 =A0 =A0 =A0* Dmitriy Ryaboy =A0 =A0 =A0 =A0 =A0 =A0<dvryaboy@apac=
he.org>
> =A0 =A0 =A0 =A0 =A0* Thejas Nair =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0<thej=
as@apache.org>
>
> =A0 =A0 =A0 =A0NOW, THEREFORE, BE IT FURTHER RESOLVED, that Olga Natkovic=
h
> =A0 =A0 =A0 =A0be appointed to the office of Vice President, Apache Pig, =
to
> =A0 =A0 =A0 =A0serve in accordance with and subject to the direction of t=
he
> =A0 =A0 =A0 =A0Board of Directors and the Bylaws of the Foundation until
> =A0 =A0 =A0 =A0death, resignation, retirement, removal or disqualificatio=
n,
> =A0 =A0 =A0 =A0or until a successor is appointed; and be it further
>
> =A0 =A0 =A0 =A0RESOLVED, that the initial Apache Pig PMC be and hereby is
> =A0 =A0 =A0 =A0tasked with the creation of a set of bylaws intended to
> =A0 =A0 =A0 =A0encourage open development and increased participation in =
the
> =A0 =A0 =A0 =A0Apache Pig Project; and be it further
>
> =A0 =A0 =A0 =A0RESOLVED, that the Apache Pig Project be and hereby
> =A0 =A0 =A0 =A0is tasked with the migration and rationalization of the Ap=
ache
> =A0 =A0 =A0 =A0Hadoop Pig sub-project; and be it further
>
> =A0 =A0 =A0 =A0RESOLVED, that all responsibilities pertaining to the Apac=
he
> =A0 =A0 =A0 =A0Hadoop Pig sub-project encumbered upon the
> =A0 =A0 =A0 =A0Apache Hadoop Project are hereafter discharged.
>
>
>
>
>

From general-return-2031-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 23:36:21 2010
Return-Path: <general-return-2031-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93453 invoked from network); 24 Aug 2010 23:36:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 23:36:21 -0000
Received: (qmail 96911 invoked by uid 500); 24 Aug 2010 23:36:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96781 invoked by uid 500); 24 Aug 2010 23:36:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96773 invoked by uid 99); 24 Aug 2010 23:36:19 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 23:36:19 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 23:35:58 +0000
Received: by wwc33 with SMTP id 33so28010wwc.29
        for <general@hadoop.apache.org>; Tue, 24 Aug 2010 16:35:37 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:content-type:content-transfer-encoding;
        bh=E7MzQV2FOEZrDOIdoKp4YuxkO0ckoykHbq+HjHZc6zU=;
        b=Bbyx+OvWWMBlJCqAJGLRi/nlekx7+cxLuJBuwJrElq1tZ+wueOPs2PHOm3DFc+QdKo
         KSWOA35XoeYp3jvtGXVl7GU0JTFVURYr4dDBB1sC3/d6eY9PkPWIO0b7hqvipCMBVM6+
         Dx4uzNVJKhh9dvI/sh7T9EqD0pXiduGk008XU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=l9VKiOJpxOI93oOsQJijoFIv+Zg+LFyzOqcf2kFop3MU0s/Tii2G9m8nmRL1pGLEiI
         6otf9qStOOQatfQcLUO4laHz90l3NSjTfeh/RogjbKFOLPg9DhA/3fHhDeWb+YmZg0i+
         Sc0TkTZUtly8JkA8bm0vqA6LNpiBRIzSihyno=
MIME-Version: 1.0
Received: by 10.227.156.67 with SMTP id v3mr6471973wbw.147.1282692937604; Tue,
 24 Aug 2010 16:35:37 -0700 (PDT)
Sender: saint.ack@gmail.com
Received: by 10.216.131.24 with HTTP; Tue, 24 Aug 2010 16:35:37 -0700 (PDT)
In-Reply-To: <71D87762-A95B-434D-9FF6-8627AF65790A@yahoo-inc.com>
References: <71D87762-A95B-434D-9FF6-8627AF65790A@yahoo-inc.com>
Date: Tue, 24 Aug 2010 16:35:37 -0700
X-Google-Sender-Auth: FJFDyuFotVqUbJpe_n4rrmSaeqw
Message-ID: <AANLkTimEgNJ2+Kqe8VB1BhTNo1T1EAZf=6PpzkrfqNKR@mail.gmail.com>
Subject: Re: [VOTE] Pig to become a TLP
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Sounds like good idea to me.

St.Ack

On Mon, Aug 23, 2010 at 10:38 AM, Alan Gates <gates@yahoo-inc.com> wrote:
> I propose that Pig become a top level Apache project.
>
> The Pig development community has already voted on and approved this
> proposal. =A0In summary, the community voted that all current active
> committers (listed at http://hadoop.apache.org/pig/whoweare.html ) as of
> August 18 2010 should become PMC members, and that Olga Natkovich should =
be
> the PMC chair. =A0They also agreed that adopting bylaws should be among t=
he
> first tasks of the new PMC. =A0Full details of the vote can be seen at
> http://bit.ly/c3GDyl
>
> Please vote on sending this proposal to the Apache Board.
>
> Below is a draft resolution to be sent to the Apache Board:
>
> Establish the Apache Pig Project
>
> =A0 =A0 =A0 =A0WHEREAS, the Board of Directors deems it to be in the best
> =A0 =A0 =A0 =A0interests of the Foundation and consistent with the
> =A0 =A0 =A0 =A0Foundation's purpose to establish a Project Management
> =A0 =A0 =A0 =A0Committee charged with the creation and maintenance of
> =A0 =A0 =A0 =A0open-source software related to parallel analysis of large
> =A0 =A0 =A0 =A0data sets for distribution at no charge to the public.
>
> =A0 =A0 =A0 =A0NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> =A0 =A0 =A0 =A0Committee (PMC), to be known as the "Apache Pig Project",
> =A0 =A0 =A0 =A0be and hereby is established pursuant to Bylaws of the
> =A0 =A0 =A0 =A0Foundation; and be it further
>
> =A0 =A0 =A0 =A0RESOLVED, that the Apache Pig Project be and hereby is
> =A0 =A0 =A0 =A0responsible for the creation and maintenance of software
> =A0 =A0 =A0 =A0related to parallel analysis of large data sets; and be
> =A0 =A0 =A0 =A0it further
>
> =A0 =A0 =A0 =A0RESOLVED, that the office of "Vice President, Apache Pig" =
be
> =A0 =A0 =A0 =A0and hereby is created, the person holding such office to
> =A0 =A0 =A0 =A0serve at the direction of the Board of Directors as the ch=
air
> =A0 =A0 =A0 =A0of the Apache Pig Project, and to have primary responsibil=
ity
> =A0 =A0 =A0 =A0for management of the projects within the scope of
> =A0 =A0 =A0 =A0responsibility of the Apache Pig Project; and be it furthe=
r
>
> =A0 =A0 =A0 =A0RESOLVED, that the persons listed immediately below be and
> =A0 =A0 =A0 =A0hereby are appointed to serve as the initial members of th=
e
> =A0 =A0 =A0 =A0Apache Pig Project:
>
> =A0 =A0 =A0 =A0 =A0* Benjamin Reed =A0 =A0 =A0 =A0 =A0<breed@apache.org>
> =A0 =A0 =A0 =A0 =A0* Daniel Dai =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0<daijy=
@apache.org>
> =A0 =A0 =A0 =A0 =A0* Alan Gates =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 <gates@ap=
ache.org>
> =A0 =A0 =A0 =A0 =A0* Giridharen Kesavan =A0 <gkesavan@apache.org>
> =A0 =A0 =A0 =A0 =A0* Olga Natkovich =A0 =A0 =A0 =A0 =A0 <olga@apache.org>
> =A0 =A0 =A0 =A0 =A0* Pradeep Kamath =A0 =A0 =A0 =A0<pradeepkth@apache.org=
>
> =A0 =A0 =A0 =A0 =A0* Santhosh Srinivasan =A0<sms@apache.org>
> =A0 =A0 =A0 =A0 =A0* Yan Zhou =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 <yanz@a=
pache.org>
> =A0 =A0 =A0 =A0 =A0* Jeff Zhang =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0<zjffd=
u@apache.org>
> =A0 =A0 =A0 =A0 =A0* Ashutosh Chauhan =A0 =A0 <hashutosh@apache.org>
> =A0 =A0 =A0 =A0 =A0* Richard Ding =A0 =A0 =A0 =A0 =A0 =A0 =A0 <rding@apac=
he.org>
> =A0 =A0 =A0 =A0 =A0* Dmitriy Ryaboy =A0 =A0 =A0 =A0 =A0 =A0<dvryaboy@apac=
he.org>
> =A0 =A0 =A0 =A0 =A0* Thejas Nair =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0<thej=
as@apache.org>
>
> =A0 =A0 =A0 =A0NOW, THEREFORE, BE IT FURTHER RESOLVED, that Olga Natkovic=
h
> =A0 =A0 =A0 =A0be appointed to the office of Vice President, Apache Pig, =
to
> =A0 =A0 =A0 =A0serve in accordance with and subject to the direction of t=
he
> =A0 =A0 =A0 =A0Board of Directors and the Bylaws of the Foundation until
> =A0 =A0 =A0 =A0death, resignation, retirement, removal or disqualificatio=
n,
> =A0 =A0 =A0 =A0or until a successor is appointed; and be it further
>
> =A0 =A0 =A0 =A0RESOLVED, that the initial Apache Pig PMC be and hereby is
> =A0 =A0 =A0 =A0tasked with the creation of a set of bylaws intended to
> =A0 =A0 =A0 =A0encourage open development and increased participation in =
the
> =A0 =A0 =A0 =A0Apache Pig Project; and be it further
>
> =A0 =A0 =A0 =A0RESOLVED, that the Apache Pig Project be and hereby
> =A0 =A0 =A0 =A0is tasked with the migration and rationalization of the Ap=
ache
> =A0 =A0 =A0 =A0Hadoop Pig sub-project; and be it further
>
> =A0 =A0 =A0 =A0RESOLVED, that all responsibilities pertaining to the Apac=
he
> =A0 =A0 =A0 =A0Hadoop Pig sub-project encumbered upon the
> =A0 =A0 =A0 =A0Apache Hadoop Project are hereafter discharged.
>
>
>
>
>

From general-return-2032-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 23:38:54 2010
Return-Path: <general-return-2032-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94903 invoked from network); 24 Aug 2010 23:38:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 23:38:54 -0000
Received: (qmail 324 invoked by uid 500); 24 Aug 2010 23:38:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 251 invoked by uid 500); 24 Aug 2010 23:38:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 243 invoked by uid 99); 24 Aug 2010 23:38:52 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 23:38:52 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of daijyc@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 23:38:31 +0000
Received: by bwz14 with SMTP id 14so259353bwz.35
        for <general@hadoop.apache.org>; Tue, 24 Aug 2010 16:38:10 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=igPJguAZLrVqVwYIZaoC75vxFva3qIsozzvY8/P/S6I=;
        b=Y2vQhx3MXp6+mxsOJmteTJCh3DIFOwvlPRu2qzWyxQUB2vPPyfY1JzkZppS5B0dUr0
         qqNQ1BsUA0qsKz6T+0EbS2wTM1gGF/QeoXmB4HfOerVQuSVtKwOeY8vhmhRhRPwA03i9
         K3ZiMzYYR9R+hqLzwIfWEi85webST3Mwx8DUI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=rIgOZZoJW4PKT2Yah24hwj6v78Owjz448AOfp4Q2vPL9jxAslUV7gCmhDAy1+HE+6R
         oIba6UHTQFrmYmWB9rBOb5djGZAWWJOxtlPPRT+mf/AwBe9ghPD6gChEuPOLQap0JDwN
         cUDOA77LpGe7zEu4LFGw0tXPkHhkz+20xsgng=
MIME-Version: 1.0
Received: by 10.204.57.9 with SMTP id a9mr5303311bkh.104.1282693090497; Tue,
 24 Aug 2010 16:38:10 -0700 (PDT)
Received: by 10.204.177.206 with HTTP; Tue, 24 Aug 2010 16:38:10 -0700 (PDT)
In-Reply-To: <C1EDB62C3771D74AA823DBBBBFB3F89D4E5DE391@SP2-EX07VS03.ds.corp.yahoo.com>
References: <71D87762-A95B-434D-9FF6-8627AF65790A@yahoo-inc.com>
	<71E1E01A-5389-48C2-9A43-9A8BD6B87A2A@yahoo-inc.com>
	<C1EDB62C3771D74AA823DBBBBFB3F89D4E5DE391@SP2-EX07VS03.ds.corp.yahoo.com>
Date: Tue, 24 Aug 2010 16:38:10 -0700
Message-ID: <AANLkTi=m+yvriiX3TzisLkyYRqas2mjM97GXXQmpwrRC@mail.gmail.com>
Subject: Re: [VOTE] Pig to become a TLP
From: Daniel Dai <daijyc@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c5b0d703d349048e9a4276
X-Virus-Checked: Checked by ClamAV on apache.org

--001636c5b0d703d349048e9a4276
Content-Type: text/plain; charset=ISO-8859-1

+1

On Tue, Aug 24, 2010 at 11:34 AM, Olga Natkovich <olgan@yahoo-inc.com>wrote:

> +1
>
> > From: Alan Gates <gates@yahoo-inc.com>
> > Date: August 23, 2010 10:38:12 AM PDT
> > To: "general@hadoop.apache.org" <general@hadoop.apache.org>
> > Subject: [VOTE] Pig to become a TLP
> > Reply-To: "general@hadoop.apache.org" <general@hadoop.apache.org>
> >
> > I propose that Pig become a top level Apache project.
> >
> > The Pig development community has already voted on and approved this
> > proposal.  In summary, the community voted that all current active
> > committers (listed at http://hadoop.apache.org/pig/whoweare.html ) as
> > of August 18 2010 should become PMC members, and that Olga Natkovich
> > should be the PMC chair.  They also agreed that adopting bylaws should
> > be among the first tasks of the new PMC.  Full details of the vote can
> > be seen at http://bit.ly/c3GDyl
> >
> > Please vote on sending this proposal to the Apache Board.
> >
> > Below is a draft resolution to be sent to the Apache Board:
> >
> > Establish the Apache Pig Project
> >
> >         WHEREAS, the Board of Directors deems it to be in the best
> >         interests of the Foundation and consistent with the
> >         Foundation's purpose to establish a Project Management
> >         Committee charged with the creation and maintenance of
> >         open-source software related to parallel analysis of large
> >         data sets for distribution at no charge to the public.
> >
> >         NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> >         Committee (PMC), to be known as the "Apache Pig Project",
> >         be and hereby is established pursuant to Bylaws of the
> >         Foundation; and be it further
> >
> >         RESOLVED, that the Apache Pig Project be and hereby is
> >         responsible for the creation and maintenance of software
> >         related to parallel analysis of large data sets; and be
> >         it further
> >
> >         RESOLVED, that the office of "Vice President, Apache Pig" be
> >         and hereby is created, the person holding such office to
> >         serve at the direction of the Board of Directors as the chair
> >         of the Apache Pig Project, and to have primary responsibility
> >         for management of the projects within the scope of
> >         responsibility of the Apache Pig Project; and be it further
> >
> >         RESOLVED, that the persons listed immediately below be and
> >         hereby are appointed to serve as the initial members of the
> >         Apache Pig Project:
> >
> >           * Benjamin Reed          <breed@apache.org>
> >           * Daniel Dai                  <daijy@apache.org>
> >           * Alan Gates                 <gates@apache.org>
> >           * Giridharen Kesavan   <gkesavan@apache.org>
> >           * Olga Natkovich           <olga@apache.org>
> >           * Pradeep Kamath        <pradeepkth@apache.org>
> >           * Santhosh Srinivasan  <sms@apache.org>
> >           * Yan Zhou                   <yanz@apache.org>
> >           * Jeff Zhang                  <zjffdu@apache.org>
> >           * Ashutosh Chauhan     <hashutosh@apache.org>
> >           * Richard Ding               <rding@apache.org>
> >           * Dmitriy Ryaboy            <dvryaboy@apache.org>
> >           * Thejas Nair                  <thejas@apache.org>
> >
> >         NOW, THEREFORE, BE IT FURTHER RESOLVED, that Olga Natkovich
> >         be appointed to the office of Vice President, Apache Pig, to
> >         serve in accordance with and subject to the direction of the
> >         Board of Directors and the Bylaws of the Foundation until
> >         death, resignation, retirement, removal or disqualification,
> >         or until a successor is appointed; and be it further
> >
> >         RESOLVED, that the initial Apache Pig PMC be and hereby is
> >         tasked with the creation of a set of bylaws intended to
> >         encourage open development and increased participation in the
> >         Apache Pig Project; and be it further
> >
> >         RESOLVED, that the Apache Pig Project be and hereby
> >         is tasked with the migration and rationalization of the Apache
> >         Hadoop Pig sub-project; and be it further
> >
> >         RESOLVED, that all responsibilities pertaining to the Apache
> >         Hadoop Pig sub-project encumbered upon the
> >         Apache Hadoop Project are hereafter discharged.
> >
> >
> >
> >
>
>

--001636c5b0d703d349048e9a4276--

From general-return-2033-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 23:40:17 2010
Return-Path: <general-return-2033-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95072 invoked from network); 24 Aug 2010 23:40:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 23:40:17 -0000
Received: (qmail 2327 invoked by uid 500); 24 Aug 2010 23:40:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2280 invoked by uid 500); 24 Aug 2010 23:40:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2272 invoked by uid 99); 24 Aug 2010 23:40:15 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 23:40:15 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 23:39:52 +0000
Received: from EGL-EX07CAS01.ds.corp.yahoo.com (egl-ex07cas01.eglbp.corp.yahoo.com [203.83.248.208])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7ONclm6095687
	for <general@hadoop.apache.org>; Tue, 24 Aug 2010 16:38:48 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:accept-language:content-language:x-ms-has-attach:
	x-ms-tnef-correlator:acceptlanguage:content-type:
	content-transfer-encoding:mime-version;
	b=mIdwmcYdXsfBDw/CpNPzxSZmuWTGFjjlBekkPWUEuuLlLYEnbuzG7h/RPxQAO4BJ
Received: from EGL-EX07VS01.ds.corp.yahoo.com ([203.83.248.205]) by
 EGL-EX07CAS01.ds.corp.yahoo.com ([203.83.248.215]) with mapi; Wed, 25 Aug
 2010 05:08:47 +0530
From: "Giridharan  Kesavan" <gkesavan@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 25 Aug 2010 05:08:46 +0530
Subject: hudson patch test jobs : hadoop pig and zookeeper
Thread-Topic: hudson patch test jobs : hadoop pig and zookeeper
Thread-Index: ActD5X9SSlsAmr/VSnWlyVrES9koHg==
Message-ID: <155916E3-8F8D-4EF7-8992-4C6E3382D241@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

We have a new hudson master hudson.apache.org and hudson.zones.apache.org i=
s retired.
This means that we need to port all our patch test admin jobs for hadoop(co=
mmon,hdfs,mapred), pig and zookeeper to the new hudson master.

I'm working on configuring patch admin jobs with the new hudson master: hud=
son.apache.org. (this is exactly the reason for why the patch test builds a=
re not running at the moment)

Thanks
Giri =

From general-return-2034-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 25 00:17:18 2010
Return-Path: <general-return-2034-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9194 invoked from network); 25 Aug 2010 00:17:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Aug 2010 00:17:17 -0000
Received: (qmail 46265 invoked by uid 500); 25 Aug 2010 00:17:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46166 invoked by uid 500); 25 Aug 2010 00:17:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46158 invoked by uid 99); 25 Aug 2010 00:17:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 00:17:15 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 00:17:11 +0000
Received: from [127.0.0.1] (gentlepaint-lx.corp.yahoo.com [10.72.185.127])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7P0G0ix065931
	for <general@hadoop.apache.org>; Tue, 24 Aug 2010 17:16:06 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=inGukrR+cUkSp8R0nFIOI8QH3unB/k6h+2bJBngMc2xIkAYNUzUn+JNdYgkFkh6o
Message-ID: <4C7460C0.4010004@yahoo-inc.com>
Date: Tue, 24 Aug 2010 17:16:00 -0700
From: Konstantin Shvachko <shv@yahoo-inc.com>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.8) Gecko/20100802 Lightning/1.0b2 Thunderbird/3.1.2
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Pig to become a TLP
References: <71D87762-A95B-434D-9FF6-8627AF65790A@yahoo-inc.com>
In-Reply-To: <71D87762-A95B-434D-9FF6-8627AF65790A@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

+1
hereby pursuant to proposal
--Konstantin

On 8/23/2010 10:38 AM, Alan Gates wrote:
> I propose that Pig become a top level Apache project.
>
> The Pig development community has already voted on and approved this
> proposal.  In summary, the community voted that all current active
> committers (listed at http://hadoop.apache.org/pig/whoweare.html ) as
> of August 18 2010 should become PMC members, and that Olga Natkovich
> should be the PMC chair.  They also agreed that adopting bylaws should
> be among the first tasks of the new PMC.  Full details of the vote can
> be seen at http://bit.ly/c3GDyl
>
> Please vote on sending this proposal to the Apache Board.
>
> Below is a draft resolution to be sent to the Apache Board:
>
> Establish the Apache Pig Project
>
>           WHEREAS, the Board of Directors deems it to be in the best
>           interests of the Foundation and consistent with the
>           Foundation's purpose to establish a Project Management
>           Committee charged with the creation and maintenance of
>           open-source software related to parallel analysis of large
>           data sets for distribution at no charge to the public.
>
>           NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>           Committee (PMC), to be known as the "Apache Pig Project",
>           be and hereby is established pursuant to Bylaws of the
>           Foundation; and be it further
>
>           RESOLVED, that the Apache Pig Project be and hereby is
>           responsible for the creation and maintenance of software
>           related to parallel analysis of large data sets; and be
>           it further
>
>           RESOLVED, that the office of "Vice President, Apache Pig" be
>           and hereby is created, the person holding such office to
>           serve at the direction of the Board of Directors as the chair
>           of the Apache Pig Project, and to have primary responsibility
>           for management of the projects within the scope of
>           responsibility of the Apache Pig Project; and be it further
>
>           RESOLVED, that the persons listed immediately below be and
>           hereby are appointed to serve as the initial members of the
>           Apache Pig Project:
>
>             * Benjamin Reed<breed@apache.org>
>             * Daniel Dai<daijy@apache.org>
>             * Alan Gates<gates@apache.org>
>             * Giridharen Kesavan<gkesavan@apache.org>
>             * Olga Natkovich<olga@apache.org>
>             * Pradeep Kamath<pradeepkth@apache.org>
>             * Santhosh Srinivasan<sms@apache.org>
>             * Yan Zhou<yanz@apache.org>
>             * Jeff Zhang<zjffdu@apache.org>
>             * Ashutosh Chauhan<hashutosh@apache.org>
>             * Richard Ding<rding@apache.org>
>             * Dmitriy Ryaboy<dvryaboy@apache.org>
>             * Thejas Nair<thejas@apache.org>
>
>           NOW, THEREFORE, BE IT FURTHER RESOLVED, that Olga Natkovich
>           be appointed to the office of Vice President, Apache Pig, to
>           serve in accordance with and subject to the direction of the
>           Board of Directors and the Bylaws of the Foundation until
>           death, resignation, retirement, removal or disqualification,
>           or until a successor is appointed; and be it further
>
>           RESOLVED, that the initial Apache Pig PMC be and hereby is
>           tasked with the creation of a set of bylaws intended to
>           encourage open development and increased participation in the
>           Apache Pig Project; and be it further
>
>           RESOLVED, that the Apache Pig Project be and hereby
>           is tasked with the migration and rationalization of the Apache
>           Hadoop Pig sub-project; and be it further
>
>           RESOLVED, that all responsibilities pertaining to the Apache
>           Hadoop Pig sub-project encumbered upon the
>           Apache Hadoop Project are hereafter discharged.
>
>
>
>
>


From general-return-2035-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 25 03:37:34 2010
Return-Path: <general-return-2035-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72952 invoked from network); 25 Aug 2010 03:37:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Aug 2010 03:37:33 -0000
Received: (qmail 63706 invoked by uid 500); 25 Aug 2010 03:37:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62954 invoked by uid 500); 25 Aug 2010 03:37:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62924 invoked by uid 99); 25 Aug 2010 03:37:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 03:37:20 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 03:37:15 +0000
Received: from [10.66.72.163] (sightbusy-lx.eglbp.corp.yahoo.com [10.66.72.163])
	by mrout1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7P3ZYv3020015;
	Tue, 24 Aug 2010 20:35:35 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:cc:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=UJfmhOEVUG5J5xwASJKJzNnuzimw7aCZVtKN1uIqw4zQ3IpjwIjNXQd/yBle+Wzh
Message-ID: <4C748F85.3000202@yahoo-inc.com>
Date: Wed, 25 Aug 2010 09:05:33 +0530
From: Vinod KV <vinodkv@yahoo-inc.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.11) Gecko/20100713 Lightning/1.0b1 Thunderbird/3.0.6
MIME-Version: 1.0
To: "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>
CC: Tom White <tom@cloudera.com>,
        "general@hadoop.apache.org" <general@hadoop.apache.org>,
        common-user <common-user@hadoop.apache.org>
Subject: Re: [ANNOUNCE] Apache Hadoop 0.21.0 released
References: <AANLkTimpwY39vhs577N5gfgufSBdDZsBd-81QHH0+zR4@mail.gmail.com>
In-Reply-To: <AANLkTimpwY39vhs577N5gfgufSBdDZsBd-81QHH0+zR4@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Thanks for the great work leading this effort, Tom!

+Vinod

On Wednesday 25 August 2010 12:48 AM, Tom White wrote:
> Hi everyone,
>
> I am pleased to announce that Apache Hadoop 0.21.0 is available for
> download from http://hadoop.apache.org/common/releases.html.
>
> Over 1300 issues have been addressed since 0.20.2; you can find details at
>
> http://hadoop.apache.org/common/docs/r0.21.0/releasenotes.html
> http://hadoop.apache.org/hdfs/docs/r0.21.0/releasenotes.html
> http://hadoop.apache.org/mapreduce/docs/r0.21.0/releasenotes.html
>
> Please note that this release has not undergone testing at scale and
> should not be considered stable or suitable for production. It is
> being classified as a minor release, which means that it should be API
> compatible with 0.20.2.
>
> Thanks to all who contributed to this release!
>
> Tom
>
>    


From general-return-2036-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 25 07:16:56 2010
Return-Path: <general-return-2036-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42073 invoked from network); 25 Aug 2010 07:16:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Aug 2010 07:16:56 -0000
Received: (qmail 83405 invoked by uid 500); 25 Aug 2010 07:16:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82918 invoked by uid 500); 25 Aug 2010 07:16:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82910 invoked by uid 99); 25 Aug 2010 07:16:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 07:16:50 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [68.142.237.91] (HELO n6.bullet.re3.yahoo.com) (68.142.237.91)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 25 Aug 2010 07:16:43 +0000
Received: from [68.142.237.87] by n6.bullet.re3.yahoo.com with NNFMP; 25 Aug 2010 07:16:21 -0000
Received: from [66.196.97.152] by t3.bullet.re3.yahoo.com with NNFMP; 25 Aug 2010 07:16:21 -0000
Received: from [127.0.0.1] by omp205.mail.re3.yahoo.com with NNFMP; 25 Aug 2010 07:16:21 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 848134.41578.bm@omp205.mail.re3.yahoo.com
Received: (qmail 57132 invoked by uid 60001); 25 Aug 2010 07:16:21 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1282720581; bh=71hQJ796zWjYEC0c5CgvpfNLjePIE9QPvGksrL35V1M=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=zkUGn6HYrBkMZe9ckmaJUUwo0qJdlSiVGL+M6vjqIRhs5o0qisNE7xyFJg6AL1rDSszMN427l9P+IplEl+kmfmieasEErylNWD08zQduzPxlZh4cY7UXZX3pughsMLEDl4U5ga4WgSysSqD99yOk7Ozwp0H7JsM300dFkeqETnM=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=TGX1dIGyrVS0IllNqoqn62ynJERMJParGlngK4ddRVi8rczh4EqLru0LmeRcxJ8Al7hOEeYrQJeferTGE3ZTiOgHwOgEs+tWQiEpOEQiCGX4YYkzBq0aembNUWV5MLK3RXRPkM8jOD5XZYtuwmApFPfEP59wngvE1UKYkyRIbc8=;
Message-ID: <700422.56791.qm@web56604.mail.re3.yahoo.com>
X-YMail-OSG: Ay3naMkVM1l_.hHM_V.Pq_GuLbR7rdQXWNpiRlOZSpJeXFZ
 CM1TJzqv2DWIz2mbp.p9UtkvOFN0wVfXZC593kSCi2m1emaTIJ41fAGovCJp
 03OSFATvvLwnHpUU002FlBETp0P0AN.xscV2sXlREb2RUZ2qTyKfF_FSF2gh
 KnRIcfu3bhDqLXhyl64BBI3dSzxO3CoEuGS83L6UAXtLWHontxGSVlu_YuAp
 tR93D3rveE9BTcSEQeOWYWNjH79NxDU0wCTNdRcUt9vr2KJn88huzpZ.dD8k
 HX4CUZZ2uJ_o5jBGiCcg_pnY_4Ca0ttR3lsvxogKyipq3Dyck1BP2ViiArhE
 -
Received: from [89.212.72.108] by web56604.mail.re3.yahoo.com via HTTP; Wed, 25 Aug 2010 00:16:21 PDT
X-Mailer: YahooMailRC/470 YahooMailWebService/0.8.105.279950
Date: Wed, 25 Aug 2010 00:16:21 -0700 (PDT)
From: "Oleg V. Zhylin" <ovz@yahoo.com>
Subject: Documentation suggestion: Quick start
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii

Hi All,

   The Quick  Start page 
http://hadoop.apache.org/common/docs/current/quickstart.html contains a 
suggestion on how to ssh to localhost w/o a passphrase

Now check that you can ssh to the localhost without a passphrase:
$ ssh localhost 

If you cannot ssh to localhost without a passphrase, execute the following 
commands:
$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa 
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

   I suggest adding a third command to to the snipped

$ chmod 600 ~/.ssh/authorized_keys 

   Otherwise the number of users who wants a quick start and run into an ssh 
issue like one described in 
http://www.linuxquestions.org/questions/linux-newbie-8/ssh-to-localhost-without-password-802067/
 will grow on.  WBR Oleg V. Zhylin   ovz@yahoo.com



      


From general-return-2037-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 25 08:35:09 2010
Return-Path: <general-return-2037-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67407 invoked from network); 25 Aug 2010 08:35:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Aug 2010 08:35:09 -0000
Received: (qmail 55996 invoked by uid 500); 25 Aug 2010 08:35:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55443 invoked by uid 500); 25 Aug 2010 08:35:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55434 invoked by uid 99); 25 Aug 2010 08:35:03 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 08:35:03 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 08:34:40 +0000
Received: by wyf23 with SMTP id 23so466838wyf.35
        for <general@hadoop.apache.org>; Wed, 25 Aug 2010 01:34:20 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.188.135 with SMTP id a7mr519731wen.39.1282725259226; Wed,
 25 Aug 2010 01:34:19 -0700 (PDT)
Received: by 10.216.166.4 with HTTP; Wed, 25 Aug 2010 01:34:19 -0700 (PDT)
In-Reply-To: <4C739DD0.8050502@apache.org>
References: <201008241143.46918.thomas@koch.ro>
	<4C739DD0.8050502@apache.org>
Date: Wed, 25 Aug 2010 01:34:19 -0700
Message-ID: <AANLkTimJ=0guMqK4uqdtykJMKd1DMPxHfqEMD3q-Ane4@mail.gmail.com>
Subject: Re: apache commons configuration
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e65a09286bae0c048ea1bfe8
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e65a09286bae0c048ea1bfe8
Content-Type: text/plain; charset=ISO-8859-1

For some recent discussion, see
https://issues.apache.org/jira/browse/HADOOP-6910

On Tue, Aug 24, 2010 at 3:24 AM, Steve Loughran <stevel@apache.org> wrote:

> On 24/08/10 10:43, Thomas Koch wrote:
>
>> Hi,
>>
>> just out of curiosity: Is there any particular reason, why Hadoop projects
>> or
>> ZooKeeper do not use apache commons configuration[1]?
>> I'm just evaluating it for an internal project. Would you recommend it or
>> not?
>> I imagine, that it could help to make projects configurable either via XML
>> or
>> INI files. Some people may prefer one or the other.
>>
>> [1] http://commons.apache.org/configuration/
>>
>
> Someone's discussed it, I'm against because (a) there's a lot of historical
> configuration stuff in there, (b) ini files are less flexible than XML, (c)
> neither of them are very dynamic. when the NN goes down and you want the DNs
> to reconnect to a different host, how do you do that when all data is read
> at start time?
>
>
>

--0016e65a09286bae0c048ea1bfe8--

From general-return-2038-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 25 17:01:01 2010
Return-Path: <general-return-2038-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78230 invoked from network); 25 Aug 2010 17:01:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Aug 2010 17:01:01 -0000
Received: (qmail 25549 invoked by uid 500); 25 Aug 2010 17:01:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25318 invoked by uid 500); 25 Aug 2010 17:00:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25310 invoked by uid 99); 25 Aug 2010 17:00:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 17:00:59 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.215.176] (HELO mail-ey0-f176.google.com) (209.85.215.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 17:00:53 +0000
Received: by eydd26 with SMTP id d26so592967eyd.35
        for <general@hadoop.apache.org>; Wed, 25 Aug 2010 10:00:31 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.1.17 with SMTP id 17mr1028629wec.99.1282755628245; Wed, 25
 Aug 2010 10:00:28 -0700 (PDT)
Received: by 10.216.4.72 with HTTP; Wed, 25 Aug 2010 10:00:28 -0700 (PDT)
In-Reply-To: <ACC0BDF0-C9F5-4DBA-90C4-5B94A01CD968@apache.org>
References: <ACC0BDF0-C9F5-4DBA-90C4-5B94A01CD968@apache.org>
Date: Wed, 25 Aug 2010 10:00:28 -0700
Message-ID: <AANLkTin8OZ9-hW-O0T9i0fMFNVv-19Le6NjpJjo0hDwZ@mail.gmail.com>
Subject: Re: Branching and testing strategy for 0.22
From: Tom White <tom@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

+1 Thanks for driving this, Owen.

Tom

On Mon, Aug 23, 2010 at 3:19 PM, Owen O'Malley <omalley@apache.org> wrote:
> I'd like to get started testing 0.22.
>
> I plan to start making mini-branches for QA. These branches will be
> snapshots that QA can use for testing with an expected lifetime of two weeks
> each. Only bug fixes that are blocking QA will be applied to the
> mini-branches and every two weeks, the base of the branch will be moved to
> the head of trunk. This will allow QA to test a point in time (possibly with
> required bug fixes) with requiring development to continually maintain two
> branches.
>
> To simplify automated builds, I'll call the branch the final name of
> "branch-0.22." But it will be rebased every two weeks or so.
>
> Are there any concerns?
>
> -- Owen
>

From general-return-2039-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 25 17:02:08 2010
Return-Path: <general-return-2039-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78666 invoked from network); 25 Aug 2010 17:02:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Aug 2010 17:02:08 -0000
Received: (qmail 27607 invoked by uid 500); 25 Aug 2010 17:02:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27452 invoked by uid 500); 25 Aug 2010 17:02:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27444 invoked by uid 99); 25 Aug 2010 17:02:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 17:02:06 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of alex.baranov.v@gmail.com designates 209.85.216.169 as permitted sender)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 17:01:59 +0000
Received: by qyk12 with SMTP id 12so6054184qyk.14
        for <general@hadoop.apache.org>; Wed, 25 Aug 2010 10:01:38 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=l3NCWWNdmAkB9E2VEBonEiyV2gIkwwvltWcwfDZiwPw=;
        b=aeFz1NVztoiqkSK5Cffo//AkwnE6OhUWxq4/A4AyLQLxEjgmZ/vVqwD+s33b7C6q/I
         Kq9vK/3JWxKMOFoVgMR1naELtp+wJmdYKRm6QPr6dYPOp8d92UOOnTFQ9njq2PWB2wNU
         I/DBurmMukZLVWKTo0YRZENxzqZnoR+uYEjpI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=R/eLA2bggYjr59euVOD0ajvto5WmLm4fEtBHA0cB0Rie5nLdldOrD7MLZd56JLZqaw
         wms1J4xAiebnA8D2hNL0MprEvF9tWHSJL53Gt81PQoC9JwYfWItrSnLZYBdARU40vK0e
         FYcjxxoj3kOiaOHbSo0HVBlVRXTL8+ver8TX8=
MIME-Version: 1.0
Received: by 10.229.131.143 with SMTP id x15mr4799861qcs.198.1282755698844;
 Wed, 25 Aug 2010 10:01:38 -0700 (PDT)
Received: by 10.229.51.137 with HTTP; Wed, 25 Aug 2010 10:01:38 -0700 (PDT)
Date: Wed, 25 Aug 2010 20:01:38 +0300
Message-ID: <AANLkTinuE_=Psz+G4m6Z_stJyYuhWex_F8GuD==hq7YZ@mail.gmail.com>
Subject: Searching Hadoop project(s) content
From: Alex Baranau <alex.baranov.v@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175747ecc3641e048ea8d5f9
X-Virus-Checked: Checked by ClamAV on apache.org

--0015175747ecc3641e048ea8d5f9
Content-Type: text/plain; charset=ISO-8859-1

Hello guys,

Over at http://search-hadoop.com we index Hadoop sub-project's mailing
lists, wiki, web site,
source code, javadoc, jira... Including general MLs for Hadoop TLP.

Would the community be interested in a patch that replaces the
Google-powered
search with that from search-hadoop.com by default?

We look into adding this search service for all Hadoop's sub-projects, so we
think it will be good (and consistent) to use this it on Hadoop TLP main
page as well.

Assuming people are for this, any suggestions for how the search should
function by default or any specific instructions for how the search box
should
be modified would be great!

Thank you,
Alex Baranau.

P.S. HBase community already accepted our proposal (please refer to
https://issues.apache.org/jira/browse/HBASE-2886) and new version (0.90)
will include new search box. Also the patch is available for TIKA (we are in
the process of discussing some details now):
https://issues.apache.org/jira/browse/TIKA-488. Hadoop's site looks much
like Avro's for which we also created patch recently (
https://issues.apache.org/jira/browse/AVRO-626).

--0015175747ecc3641e048ea8d5f9--

From general-return-2040-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 25 17:16:44 2010
Return-Path: <general-return-2040-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 89401 invoked from network); 25 Aug 2010 17:16:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Aug 2010 17:16:44 -0000
Received: (qmail 50547 invoked by uid 500); 25 Aug 2010 17:16:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50347 invoked by uid 500); 25 Aug 2010 17:16:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50339 invoked by uid 99); 25 Aug 2010 17:16:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 17:16:42 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 17:16:37 +0000
Received: by ewy10 with SMTP id 10so594047ewy.35
        for <general@hadoop.apache.org>; Wed, 25 Aug 2010 10:16:16 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.174.67 with SMTP id w45mr1056844wel.78.1282756576255; Wed,
 25 Aug 2010 10:16:16 -0700 (PDT)
Received: by 10.216.4.72 with HTTP; Wed, 25 Aug 2010 10:16:15 -0700 (PDT)
In-Reply-To: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
Date: Wed, 25 Aug 2010 10:16:15 -0700
Message-ID: <AANLkTim7csnT2ULSL5Ud5GTM5rAhcAkUts3d4aCuQcMt@mail.gmail.com>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Tom White <tom@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Arun,

I think it would be good to have a shared 0.20 Apache security branch.
Since security isn't in 0.21, and the 0.22 release is a some way off
as you mention, this would be useful for folks who want the security
features sooner (and want to use an Apache release).

Thanks,
Tom

On Mon, Aug 23, 2010 at 5:27 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:
> Even with the work on hadoop-0.22 (trunk) starting in earnest it is fairl=
y
> obvious, given our past history, that it will take a while for us to get =
it
> stable and deployable - for e.g. it took us nearly 6 months to deploy
> hadoop-0.20.
>
> In the interim I'd like to propose we push a hadoop-0.20-security release
> off the Yahoo! patchset (http://github.com/yahoo/hadoop-common). This wil=
l
> ensure the community benefits from all the work done at Yahoo! for over 1=
2
> months *now*, and ensures that we do not have to wait until hadoop-0.22
> which has all of these patches.
>
> Some salient aspects:
> a) Full-fledged security implementation deployed at scale (4000 nodes) in
> production.
> b) Lots of work on the stabilizing and optimizing the NameNode and
> JobTracker for over 12 months. This has been critical in deploying Hadoop=
 at
> scale i.e. clusters of 4000 nodes. For e.g. we have a 50% improvement in =
CPU
> utilization on the JobTracker vis-a-vis the hadoop-0.20.2 release.
> c) Several new features in the scheduler (CapacityScheduler), Map-Reduce
> framework, better support for multi-tenancy etc.
> d) Several performance and stability improvements to the system e.g.
> iterative ls, robustness against rogue clients/jobs/users etc.
>
> Also, given the huge number of features and enhancements I'd like to prop=
ose
> we create a new 0.20-security branch and commit the Yahoo patchset there =
for
> the release.
>
> This has been proposed earlier by Doug and did not get far due to concern=
s
> about the effect this would have on development on trunk. However, I
> believe, we have a case for demonstrable progress on trunk now, and it wo=
uld
> be useful to have an interim, fully-tested Apache Hadoop release availabl=
e
> to the community.
>
> =A0Conceivably, one could imagine a Hadoop Security + Append release soon
> after. At this point a Hadoop Security release alone would add tremendous
> value for the reasons above. Presently we would like to get this release =
out
> quickly to focus the majority of our efforts on trunk.
>
> Thoughts?
>
> Arun
>
>

From general-return-2041-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 25 17:42:07 2010
Return-Path: <general-return-2041-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96417 invoked from network); 25 Aug 2010 17:42:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Aug 2010 17:42:07 -0000
Received: (qmail 78415 invoked by uid 500); 25 Aug 2010 17:42:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78244 invoked by uid 500); 25 Aug 2010 17:42:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78236 invoked by uid 99); 25 Aug 2010 17:42:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 17:42:05 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yhemanth@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 17:42:01 +0000
Received: by vws4 with SMTP id 4so1030312vws.35
        for <general@hadoop.apache.org>; Wed, 25 Aug 2010 10:41:40 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=ScleatGM4wL6/gcazgCW04SZYvT2HgBnmLcJU0+nbQg=;
        b=xkpmziIEQZ+JIwnbuYQz0oQSsjamS/opWoUa6yY6pDSnxqJXqm1gAIUpQsw4RE8oYO
         OnFiBpO9ShzY26z+DXr6vbCj1SzbXFW+Vp0uAUvmyzNLWEnCM5siXhUFWH07D1vXahUx
         JEN4EiJOH/6onskHBFH3aSTFlPmDhIwnDq6po=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=KeDcPILORqfguS6VWZH/Imf8Tr+GtrnOm5sQt5cD/lxRVJstNHfd861oulSFvY53dE
         iDnf9W5FdAUKAsdhuMehzyihv178oscxmbEUdaPRYmRbZ4k0i6d/ONUN/VMuRfT6KLg0
         W5/wek1DEoqLuK1WLWISNhTYG2XBpCCCrv2fY=
MIME-Version: 1.0
Received: by 10.220.62.72 with SMTP id w8mr5539477vch.60.1282758034918; Wed,
 25 Aug 2010 10:40:34 -0700 (PDT)
Received: by 10.220.178.135 with HTTP; Wed, 25 Aug 2010 10:40:34 -0700 (PDT)
In-Reply-To: <71D87762-A95B-434D-9FF6-8627AF65790A@yahoo-inc.com>
References: <71D87762-A95B-434D-9FF6-8627AF65790A@yahoo-inc.com>
Date: Wed, 25 Aug 2010 23:10:34 +0530
Message-ID: <AANLkTimeeWsaJTOju9wiHP-Tc_4Wa9iMxpS7CjdiNVtm@mail.gmail.com>
Subject: Re: [VOTE] Pig to become a TLP
From: Hemanth Yamijala <yhemanth@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

+1

On Mon, Aug 23, 2010 at 11:08 PM, Alan Gates <gates@yahoo-inc.com> wrote:
> I propose that Pig become a top level Apache project.
>
> The Pig development community has already voted on and approved this
> proposal. =A0In summary, the community voted that all current active
> committers (listed at http://hadoop.apache.org/pig/whoweare.html ) as of
> August 18 2010 should become PMC members, and that Olga Natkovich should =
be
> the PMC chair. =A0They also agreed that adopting bylaws should be among t=
he
> first tasks of the new PMC. =A0Full details of the vote can be seen at
> http://bit.ly/c3GDyl
>
> Please vote on sending this proposal to the Apache Board.
>
> Below is a draft resolution to be sent to the Apache Board:
>
> Establish the Apache Pig Project
>
> =A0 =A0 =A0 =A0WHEREAS, the Board of Directors deems it to be in the best
> =A0 =A0 =A0 =A0interests of the Foundation and consistent with the
> =A0 =A0 =A0 =A0Foundation's purpose to establish a Project Management
> =A0 =A0 =A0 =A0Committee charged with the creation and maintenance of
> =A0 =A0 =A0 =A0open-source software related to parallel analysis of large
> =A0 =A0 =A0 =A0data sets for distribution at no charge to the public.
>
> =A0 =A0 =A0 =A0NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> =A0 =A0 =A0 =A0Committee (PMC), to be known as the "Apache Pig Project",
> =A0 =A0 =A0 =A0be and hereby is established pursuant to Bylaws of the
> =A0 =A0 =A0 =A0Foundation; and be it further
>
> =A0 =A0 =A0 =A0RESOLVED, that the Apache Pig Project be and hereby is
> =A0 =A0 =A0 =A0responsible for the creation and maintenance of software
> =A0 =A0 =A0 =A0related to parallel analysis of large data sets; and be
> =A0 =A0 =A0 =A0it further
>
> =A0 =A0 =A0 =A0RESOLVED, that the office of "Vice President, Apache Pig" =
be
> =A0 =A0 =A0 =A0and hereby is created, the person holding such office to
> =A0 =A0 =A0 =A0serve at the direction of the Board of Directors as the ch=
air
> =A0 =A0 =A0 =A0of the Apache Pig Project, and to have primary responsibil=
ity
> =A0 =A0 =A0 =A0for management of the projects within the scope of
> =A0 =A0 =A0 =A0responsibility of the Apache Pig Project; and be it furthe=
r
>
> =A0 =A0 =A0 =A0RESOLVED, that the persons listed immediately below be and
> =A0 =A0 =A0 =A0hereby are appointed to serve as the initial members of th=
e
> =A0 =A0 =A0 =A0Apache Pig Project:
>
> =A0 =A0 =A0 =A0 =A0* Benjamin Reed =A0 =A0 =A0 =A0 =A0<breed@apache.org>
> =A0 =A0 =A0 =A0 =A0* Daniel Dai =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0<daijy=
@apache.org>
> =A0 =A0 =A0 =A0 =A0* Alan Gates =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 <gates@ap=
ache.org>
> =A0 =A0 =A0 =A0 =A0* Giridharen Kesavan =A0 <gkesavan@apache.org>
> =A0 =A0 =A0 =A0 =A0* Olga Natkovich =A0 =A0 =A0 =A0 =A0 <olga@apache.org>
> =A0 =A0 =A0 =A0 =A0* Pradeep Kamath =A0 =A0 =A0 =A0<pradeepkth@apache.org=
>
> =A0 =A0 =A0 =A0 =A0* Santhosh Srinivasan =A0<sms@apache.org>
> =A0 =A0 =A0 =A0 =A0* Yan Zhou =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 <yanz@a=
pache.org>
> =A0 =A0 =A0 =A0 =A0* Jeff Zhang =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0<zjffd=
u@apache.org>
> =A0 =A0 =A0 =A0 =A0* Ashutosh Chauhan =A0 =A0 <hashutosh@apache.org>
> =A0 =A0 =A0 =A0 =A0* Richard Ding =A0 =A0 =A0 =A0 =A0 =A0 =A0 <rding@apac=
he.org>
> =A0 =A0 =A0 =A0 =A0* Dmitriy Ryaboy =A0 =A0 =A0 =A0 =A0 =A0<dvryaboy@apac=
he.org>
> =A0 =A0 =A0 =A0 =A0* Thejas Nair =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0<thej=
as@apache.org>
>
> =A0 =A0 =A0 =A0NOW, THEREFORE, BE IT FURTHER RESOLVED, that Olga Natkovic=
h
> =A0 =A0 =A0 =A0be appointed to the office of Vice President, Apache Pig, =
to
> =A0 =A0 =A0 =A0serve in accordance with and subject to the direction of t=
he
> =A0 =A0 =A0 =A0Board of Directors and the Bylaws of the Foundation until
> =A0 =A0 =A0 =A0death, resignation, retirement, removal or disqualificatio=
n,
> =A0 =A0 =A0 =A0or until a successor is appointed; and be it further
>
> =A0 =A0 =A0 =A0RESOLVED, that the initial Apache Pig PMC be and hereby is
> =A0 =A0 =A0 =A0tasked with the creation of a set of bylaws intended to
> =A0 =A0 =A0 =A0encourage open development and increased participation in =
the
> =A0 =A0 =A0 =A0Apache Pig Project; and be it further
>
> =A0 =A0 =A0 =A0RESOLVED, that the Apache Pig Project be and hereby
> =A0 =A0 =A0 =A0is tasked with the migration and rationalization of the Ap=
ache
> =A0 =A0 =A0 =A0Hadoop Pig sub-project; and be it further
>
> =A0 =A0 =A0 =A0RESOLVED, that all responsibilities pertaining to the Apac=
he
> =A0 =A0 =A0 =A0Hadoop Pig sub-project encumbered upon the
> =A0 =A0 =A0 =A0Apache Hadoop Project are hereafter discharged.
>
>
>
>
>

From general-return-2042-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 25 17:47:11 2010
Return-Path: <general-return-2042-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97545 invoked from network); 25 Aug 2010 17:47:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Aug 2010 17:47:11 -0000
Received: (qmail 83379 invoked by uid 500); 25 Aug 2010 17:47:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83339 invoked by uid 500); 25 Aug 2010 17:47:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83331 invoked by uid 99); 25 Aug 2010 17:47:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 17:47:09 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yhemanth@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 17:47:05 +0000
Received: by vws4 with SMTP id 4so1036649vws.35
        for <general@hadoop.apache.org>; Wed, 25 Aug 2010 10:46:44 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=rER87QH20koP4iEqbdlFwkTuwTDA6ysztv+YcdjQebg=;
        b=cfZ/txCYu4z9lMf8HkVCrw6PIjFIYMpPZTE0HBNPvPufuokcykrMHZ7VurJCN0y/ag
         BoO+0+nBJ39d9VKsKSSdjA8EQQ1C7gP5Nol40082F35yhaVkjwhkSzNTjAeDnEdmRUb2
         sflTtOgOYRGPKgI9QjogMQIQxqn0SgPdiHIqY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=cY4mhZavq16gGJrr+wpEglN5UCnf/qBZ9fY9ribD7DiGk8LjZMlxdliJpIWtRziPF/
         YgMCJXryLyi4FENRA1byaAd6bJCevv5EcJgegJtnnutrbI2pCTtrxwJKXRZoRJ6RYZFM
         T1lDoMSQE+32h+A7P3vUY6RlRs3uuXHD4ioI8=
MIME-Version: 1.0
Received: by 10.220.71.136 with SMTP id h8mr5506255vcj.135.1282758404318; Wed,
 25 Aug 2010 10:46:44 -0700 (PDT)
Received: by 10.220.178.135 with HTTP; Wed, 25 Aug 2010 10:46:43 -0700 (PDT)
In-Reply-To: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
Date: Wed, 25 Aug 2010 23:16:43 +0530
Message-ID: <AANLkTikv9OFf7uX_PPThvxCceq+c_TJATt3QvE_W_T=2@mail.gmail.com>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Hemanth Yamijala <yhemanth@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Arun,

How much time do you think it would take to have a version of 0.20
with the security features in it ready ? In a different thread, Owen
has started discussing plans around 0.22. Do you think this effort
would affect 0.22 release ?

I do agree that this would be very useful for folks who want security
sooner. And the fact that Yahoo! have been running it at scale for a
good while now is also assuring.

Thanks
hemanth

On Tue, Aug 24, 2010 at 5:57 AM, Arun C Murthy <acm@yahoo-inc.com> wrote:
> Even with the work on hadoop-0.22 (trunk) starting in earnest it is fairl=
y
> obvious, given our past history, that it will take a while for us to get =
it
> stable and deployable - for e.g. it took us nearly 6 months to deploy
> hadoop-0.20.
>
> In the interim I'd like to propose we push a hadoop-0.20-security release
> off the Yahoo! patchset (http://github.com/yahoo/hadoop-common). This wil=
l
> ensure the community benefits from all the work done at Yahoo! for over 1=
2
> months *now*, and ensures that we do not have to wait until hadoop-0.22
> which has all of these patches.
>
> Some salient aspects:
> a) Full-fledged security implementation deployed at scale (4000 nodes) in
> production.
> b) Lots of work on the stabilizing and optimizing the NameNode and
> JobTracker for over 12 months. This has been critical in deploying Hadoop=
 at
> scale i.e. clusters of 4000 nodes. For e.g. we have a 50% improvement in =
CPU
> utilization on the JobTracker vis-a-vis the hadoop-0.20.2 release.
> c) Several new features in the scheduler (CapacityScheduler), Map-Reduce
> framework, better support for multi-tenancy etc.
> d) Several performance and stability improvements to the system e.g.
> iterative ls, robustness against rogue clients/jobs/users etc.
>
> Also, given the huge number of features and enhancements I'd like to prop=
ose
> we create a new 0.20-security branch and commit the Yahoo patchset there =
for
> the release.
>
> This has been proposed earlier by Doug and did not get far due to concern=
s
> about the effect this would have on development on trunk. However, I
> believe, we have a case for demonstrable progress on trunk now, and it wo=
uld
> be useful to have an interim, fully-tested Apache Hadoop release availabl=
e
> to the community.
>
> =A0Conceivably, one could imagine a Hadoop Security + Append release soon
> after. At this point a Hadoop Security release alone would add tremendous
> value for the reasons above. Presently we would like to get this release =
out
> quickly to focus the majority of our efforts on trunk.
>
> Thoughts?
>
> Arun
>
>

From general-return-2043-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 25 18:00:16 2010
Return-Path: <general-return-2043-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 2004 invoked from network); 25 Aug 2010 18:00:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Aug 2010 18:00:16 -0000
Received: (qmail 98449 invoked by uid 500); 25 Aug 2010 18:00:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98216 invoked by uid 500); 25 Aug 2010 18:00:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98207 invoked by uid 99); 25 Aug 2010 18:00:13 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 18:00:13 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 17:59:52 +0000
Received: from wlanvpn-snve-246-178.corp.yahoo.com (wlanvpn-snve-246-178.corp.yahoo.com [172.21.148.178])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7PHxBAY007464
	for <general@hadoop.apache.org>; Wed, 25 Aug 2010 10:59:11 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=ZMSGqxMjgizsZThKUMeixo4ELIio3/WSMM+KX6prs3RASDh8K0D8AT5i2tF/uPKY
Message-Id: <809E9407-FBA7-4414-90ED-D3AA166CE1D0@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <AANLkTikv9OFf7uX_PPThvxCceq+c_TJATt3QvE_W_T=2@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Date: Wed, 25 Aug 2010 10:59:10 -0700
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <AANLkTikv9OFf7uX_PPThvxCceq+c_TJATt3QvE_W_T=2@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

On Aug 25, 2010, at 10:46 AM, Hemanth Yamijala wrote:

> Arun,
>
> How much time do you think it would take to have a version of 0.20
> with the security features in it ready ? In a different thread, Owen
> has started discussing plans around 0.22. Do you think this effort
> would affect 0.22 release ?
>

I think it should be fairly trivial to get this release out - most of  
the effort is just the mechanics of committing the patches to an  
Apache branch from the yahoo git repository, creating a release  
candidate and calling it a success! *smile*

I think doing this quickly is critical in ensuring that we do not lose  
focus on 0.22, but I believe this will definitely help the community.

> I do agree that this would be very useful for folks who want security
> sooner. And the fact that Yahoo! have been running it at scale for a
> good while now is also assuring.

Just to clarify - this has security and a bunch of other enhancements  
(which are either in 0.21 or 0.22 or both).

Arun

From general-return-2044-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 25 18:24:55 2010
Return-Path: <general-return-2044-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23371 invoked from network); 25 Aug 2010 18:24:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Aug 2010 18:24:55 -0000
Received: (qmail 48468 invoked by uid 500); 25 Aug 2010 18:24:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48370 invoked by uid 500); 25 Aug 2010 18:24:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48362 invoked by uid 99); 25 Aug 2010 18:24:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 18:24:53 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.103 as permitted sender)
Received: from [17.148.16.103] (HELO asmtpout028.mac.com) (17.148.16.103)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 18:24:47 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; charset=us-ascii
Received: from [10.93.246.90]
 (166-205-136-044.mobile.mymmode.com [166.205.136.44])
 by asmtp028.mac.com (Sun Java(tm) System Messaging Server 6.3-8.01 (built Dec
 16 2008; 32bit)) with ESMTPSA id <0L7Q002720GA0T60@asmtp028.mac.com> for
 general@hadoop.apache.org; Wed, 25 Aug 2010 11:24:26 -0700 (PDT)
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0
 reason=mlx engine=6.0.2-1004200000 definitions=main-1008250135
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.0.10011,1.0.148,0.0.0000
 definitions=2010-08-25_09:2010-08-25,2010-08-25,1970-01-01 signatures=0
References: <AANLkTinuE_=Psz+G4m6Z_stJyYuhWex_F8GuD==hq7YZ@mail.gmail.com>
In-reply-to: <AANLkTinuE_=Psz+G4m6Z_stJyYuhWex_F8GuD==hq7YZ@mail.gmail.com>
Message-id: <09945313-C4F5-49D4-9CB2-72E12F8D78A9@mac.com>
Cc: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: iPhone Mail (8A400)
From: Nigel Daley <ndaley@mac.com>
Subject: Re: Searching Hadoop project(s) content
Date: Wed, 25 Aug 2010 11:23:45 -0700
To: "general@hadoop.apache.org" <general@hadoop.apache.org>

Big +1. This is cool and gr8 that it uses Solr. 

Cheers,
Nige

On Aug 25, 2010, at 10:01 AM, Alex Baranau <alex.baranov.v@gmail.com> wrote:

> Hello guys,
> 
> Over at http://search-hadoop.com we index Hadoop sub-project's mailing
> lists, wiki, web site,
> source code, javadoc, jira... Including general MLs for Hadoop TLP.
> 
> Would the community be interested in a patch that replaces the
> Google-powered
> search with that from search-hadoop.com by default?
> 
> We look into adding this search service for all Hadoop's sub-projects, so we
> think it will be good (and consistent) to use this it on Hadoop TLP main
> page as well.
> 
> Assuming people are for this, any suggestions for how the search should
> function by default or any specific instructions for how the search box
> should
> be modified would be great!
> 
> Thank you,
> Alex Baranau.
> 
> P.S. HBase community already accepted our proposal (please refer to
> https://issues.apache.org/jira/browse/HBASE-2886) and new version (0.90)
> will include new search box. Also the patch is available for TIKA (we are in
> the process of discussing some details now):
> https://issues.apache.org/jira/browse/TIKA-488. Hadoop's site looks much
> like Avro's for which we also created patch recently (
> https://issues.apache.org/jira/browse/AVRO-626).

From general-return-2045-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 25 19:21:22 2010
Return-Path: <general-return-2045-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49164 invoked from network); 25 Aug 2010 19:21:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Aug 2010 19:21:21 -0000
Received: (qmail 43606 invoked by uid 500); 25 Aug 2010 19:21:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 43427 invoked by uid 500); 25 Aug 2010 19:21:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 43419 invoked by uid 99); 25 Aug 2010 19:21:19 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 19:21:19 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [67.195.9.86] (HELO n4-vm1.bullet.mail.gq1.yahoo.com) (67.195.9.86)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 25 Aug 2010 19:20:53 +0000
Received: from [67.195.9.81] by n4.bullet.mail.gq1.yahoo.com with NNFMP; 25 Aug 2010 19:20:31 -0000
Received: from [98.137.27.209] by t1.bullet.mail.gq1.yahoo.com with NNFMP; 25 Aug 2010 19:20:30 -0000
Received: from [127.0.0.1] by omp119.mail.gq1.yahoo.com with NNFMP; 25 Aug 2010 19:20:30 -0000
X-Yahoo-Newman-Property: ymail-5
X-Yahoo-Newman-Id: 929781.65458.bm@omp119.mail.gq1.yahoo.com
Received: (qmail 78875 invoked by uid 60001); 25 Aug 2010 19:20:30 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1282764030; bh=OxeF0tXhbKQAvXtqHRkjaXiczq+ov9kz+7CWOqIvKjc=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=SXopJGWpNXtC2g+JetFu9WQDXUmp9ugU2nnIhBnVhgYTHs4fEjz65izi/FvTUE/tIUzzhoc2yQx+TjNj8slnThuXQ/IC6G+2ye/AcB/MDdV4hAosByQOUS2Q3PvsMy4mO3K3LkVvGBR0KSzz5htoQCFViDO8c1P0aj4oDE/rztY=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=XMFWl05LGxWL1WnNoaAap3OKtYbRh7R+PIqDgbKBZ9hKCgyis/tIAcRfvvgiqXEAXhIS8RtrsnD5EaMQ2hsnd/0ZO8kyyPsqQUYItL/so5kzEqBeHU7f/qf9IRbvOuF2MzrMxMCa/rErkhXo+JrJtU4Y7FvEpNOmi0UWCPCbOTY=;
Message-ID: <715590.78832.qm@web114505.mail.gq1.yahoo.com>
X-YMail-OSG: QfsyK78VM1l5sc2rgCvIkgn6Y83rUTWCy_0VG4D5F55L00k
 w88n.PQt7M1n8XmHNERAhIcixVI3zNo4qMvAJqN3eKuvl5vz5ZsVdzY8Q2Rd
 7I2QGdSKShy0FKEkh4boyvDsDSHSV5Qx058.g7IVraRVO0RJp5d6Ro0FlSTH
 uBshXYWDRfh7pdkrJ7tj1EYikOp48rNpDOvHoud44AyH0Ag_Ea498FB2TG.f
 eTHIGskZ4d_zoInoVvs.ninv.mDftcy8-
Received: from [67.180.86.237] by web114505.mail.gq1.yahoo.com via HTTP; Wed, 25 Aug 2010 12:20:30 PDT
X-Mailer: YahooMailRC/470 YahooMailWebService/0.8.105.279950
Date: Wed, 25 Aug 2010 12:20:30 -0700 (PDT)
From: C J <c.joshi26@yahoo.com>
Subject: Child processes on datanodes/task trackers
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1998256419-1282764030=:78832"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-1998256419-1282764030=:78832
Content-Type: text/plain; charset=us-ascii

Hi,
 
I wanted to know why I see running Child processes on my datanodes even though 
there is no job running at that time. Are these left over from failed attempts?

Is there anything I can do to keep these clean?

Thanks,
Deepika


      
--0-1998256419-1282764030=:78832--


From general-return-2046-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 25 20:25:57 2010
Return-Path: <general-return-2046-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78683 invoked from network); 25 Aug 2010 20:25:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Aug 2010 20:25:56 -0000
Received: (qmail 33684 invoked by uid 500); 25 Aug 2010 20:25:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33577 invoked by uid 500); 25 Aug 2010 20:25:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33569 invoked by uid 99); 25 Aug 2010 20:25:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 20:25:54 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 20:25:48 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version;
  b=n/HLAkFTbvwdHaYWcARurcvXOMCCY8TTGwYu/i/WG+g11O7m3+y0b+IM
   dExR26DQlhsUm+1iZ4gHMchMn0NjS01siisnHA7uxP+CqS/kCy9k/D89J
   fddmaAoFx4GU0ZZ;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1282767948; x=1314303948;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20[DISCUSS]=20Hadoop=20Security=20Release
   =20off=20Yahoo!=20patchset|Date:=20Wed,=2025=20Aug=202010
   =2020:25:27=20+0000|Message-ID:=20<FBE65FB7-3BCF-4541-BA4
   B-5112F1D3385D@linkedin.com>|To:=20"<general@hadoop.apach
   e.org>"=20<general@hadoop.apache.org>|MIME-Version:=201.0
   |Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<8750b3fa-93ee-4f42-a482-2ffcd83af88b>
   |In-Reply-To:=20<AANLkTikv9OFf7uX_PPThvxCceq+c_TJATt3QvE_
   W_T=3D2@mail.gmail.com>|References:=20<516684F5-0052-4381
   -805D-760B61DECB16@yahoo-inc.com>=0D=0A=20<AANLkTikv9OFf7
   uX_PPThvxCceq+c_TJATt3QvE_W_T=3D2@mail.gmail.com>;
  bh=W7lXWo8hmOsN25N5+k6LftaRoIfZyClxTxFXCXRBiYE=;
  b=xnJyZ2voKHSWvtQs+aTIl4cd5m9eV+yFzEkpivJkqBg5+p8ARBG7bj10
   yDHnHsCiyGu/cl5u0XyoItsc0Y5cxYpcIW9riC9KifWEjwnVE3u/YCKTZ
   VDLbvp7e+V9SLlF;
X-IronPort-AV: E=Sophos;i="4.56,270,1280732400"; 
   d="scan'208";a="14472161"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi; Wed, 25 Aug 2010
 13:25:27 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Thread-Topic: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Thread-Index: AQHLQyNHDr2kMsQuFU+y4yuN2iEG2pLy6geAgAAsWoA=
Date: Wed, 25 Aug 2010 20:25:27 +0000
Message-ID: <FBE65FB7-3BCF-4541-BA4B-5112F1D3385D@linkedin.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
 <AANLkTikv9OFf7uX_PPThvxCceq+c_TJATt3QvE_W_T=2@mail.gmail.com>
In-Reply-To: <AANLkTikv9OFf7uX_PPThvxCceq+c_TJATt3QvE_W_T=2@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Content-Type: text/plain; charset="us-ascii"
Content-ID: <8750b3fa-93ee-4f42-a482-2ffcd83af88b>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


On Aug 25, 2010, at 10:46 AM, Hemanth Yamijala wrote:
> I do agree that this would be very useful for folks who want security
> sooner. And the fact that Yahoo! have been running it at scale for a
> good while now is also assuring.

As has been mentioned a few times, part of the security features are depend=
ent upon Yahoo!-type operations.  Those would need to get replaced or a dec=
ision would need to be made that we are removing/regressing certain feature=
s (the cluster-wide start scripts).=

From general-return-2047-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 25 21:14:13 2010
Return-Path: <general-return-2047-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98562 invoked from network); 25 Aug 2010 21:14:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Aug 2010 21:14:13 -0000
Received: (qmail 99040 invoked by uid 500); 25 Aug 2010 21:14:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98928 invoked by uid 500); 25 Aug 2010 21:14:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98920 invoked by uid 99); 25 Aug 2010 21:14:11 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 21:14:11 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 21:13:48 +0000
Received: from SP1-EX07CAS02.ds.corp.yahoo.com (sp1-ex07cas02.ds.corp.yahoo.com [216.252.116.138])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7PLC0DZ013664
	for <general@hadoop.apache.org>; Wed, 25 Aug 2010 14:12:00 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:in-reply-to:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=o9OHf6bPKMev3fNz2p4qN0tgqb0IaUcGtVbqkytibwy5yq1P4E20NBYchwb1aN9U
Received: from SP1-EX07VS02.ds.corp.yahoo.com ([216.252.116.135]) by
 SP1-EX07CAS02.ds.corp.yahoo.com ([216.252.116.167]) with mapi; Wed, 25 Aug
 2010 14:12:00 -0700
From: Devaraj Das <ddas@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 25 Aug 2010 14:11:58 -0700
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Thread-Topic: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Thread-Index: AQHLQyNHDr2kMsQuFU+y4yuN2iEG2pLy6geAgAAsWoD//5emyw==
Message-ID: <C89AD52E.24E2C%ddas@yahoo-inc.com>
In-Reply-To: <FBE65FB7-3BCF-4541-BA4B-5112F1D3385D@linkedin.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C89AD52E24E2Cddasyahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C89AD52E24E2Cddasyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

>As has been mentioned a few times, part of the security features are depen=
dent upon Yahoo!-type operations.

Allen, could you please enlist them here again (for the benefit of the comm=
unity)? Or, are you referring to only the cluster-wide start scripts?




On 8/25/10 1:25 PM, "Allen Wittenauer" <awittenauer@linkedin.com> wrote:



On Aug 25, 2010, at 10:46 AM, Hemanth Yamijala wrote:
> I do agree that this would be very useful for folks who want security
> sooner. And the fact that Yahoo! have been running it at scale for a
> good while now is also assuring.

As has been mentioned a few times, part of the security features are depend=
ent upon Yahoo!-type operations.  Those would need to get replaced or a dec=
ision would need to be made that we are removing/regressing certain feature=
s (the cluster-wide start scripts).


--_000_C89AD52E24E2Cddasyahooinccom_--

From general-return-2048-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 25 23:18:07 2010
Return-Path: <general-return-2048-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50129 invoked from network); 25 Aug 2010 23:18:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Aug 2010 23:18:07 -0000
Received: (qmail 33711 invoked by uid 500); 25 Aug 2010 23:18:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33608 invoked by uid 500); 25 Aug 2010 23:18:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33600 invoked by uid 99); 25 Aug 2010 23:18:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 23:18:05 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com designates 209.85.216.176 as permitted sender)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 23:17:59 +0000
Received: by qyk2 with SMTP id 2so1344880qyk.14
        for <general@hadoop.apache.org>; Wed, 25 Aug 2010 16:17:38 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=py7u67JSDPHIq750riOiV9y7FYKzRqjJ64hDP4rz5IY=;
        b=Jpe1ml32KmUHEt9Bl21IgckcQ7YjgTXpdn9Uy2gcAwGFfsWNE5FmGxEthzVKxWznbE
         ZDrmfufFV2H7RFkgEM17LlWSYEurZrrSKRGT0iySxCtWIrOBO0C6mHORAOhem3e2xCh+
         Rb/0db1oDhUDh6mLrh6RdoBTWGhQl+nQdpw6g=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=aIT1QTyfLqwpEVSpeqi6RVXs1EMWNs6Y76jWWRLPuQAAFPnl5ueorejU++WiRTyYdX
         SjnKObrRXOsjyjoG1TScaLTNX28ITJ+vmKTkWtXC0qcjr+MoUZ2hcr1AmUj2dicZo3AO
         25P09OFr9ATinlhR0TImc0oBzLJKh6qOF8+8Q=
MIME-Version: 1.0
Received: by 10.224.43.197 with SMTP id x5mr6070988qae.153.1282778258722; Wed,
 25 Aug 2010 16:17:38 -0700 (PDT)
Received: by 10.229.48.129 with HTTP; Wed, 25 Aug 2010 16:17:38 -0700 (PDT)
In-Reply-To: <715590.78832.qm@web114505.mail.gq1.yahoo.com>
References: <715590.78832.qm@web114505.mail.gq1.yahoo.com>
Date: Wed, 25 Aug 2010 16:17:38 -0700
Message-ID: <AANLkTinQ_xeerXn9V7+M4=M0_NHEOgqn=ZsUu=xvMGox@mail.gmail.com>
Subject: Re: Child processes on datanodes/task trackers
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00c09f88d3c26fcfa6048eae1655

--00c09f88d3c26fcfa6048eae1655
Content-Type: text/plain; charset=ISO-8859-1

Use jps to find out pid of the Child.
Then use this to find out which job the Child belongs to:
ps aux | grep <pid>

On Wed, Aug 25, 2010 at 12:20 PM, C J <c.joshi26@yahoo.com> wrote:

> Hi,
>
> I wanted to know why I see running Child processes on my datanodes even
> though
> there is no job running at that time. Are these left over from failed
> attempts?
>
> Is there anything I can do to keep these clean?
>
> Thanks,
> Deepika
>
>
>

--00c09f88d3c26fcfa6048eae1655--

From general-return-2049-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 02:29:16 2010
Return-Path: <general-return-2049-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16898 invoked from network); 26 Aug 2010 02:29:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 02:29:15 -0000
Received: (qmail 34589 invoked by uid 500); 26 Aug 2010 02:29:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34467 invoked by uid 500); 26 Aug 2010 02:29:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34458 invoked by uid 99); 26 Aug 2010 02:29:14 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 02:29:14 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [67.195.9.7] (HELO n4-vm0.bullet.mail.gq1.yahoo.com) (67.195.9.7)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 26 Aug 2010 02:28:48 +0000
Received: from [67.195.9.83] by n4.bullet.mail.gq1.yahoo.com with NNFMP; 26 Aug 2010 02:28:26 -0000
Received: from [98.137.27.216] by t3.bullet.mail.gq1.yahoo.com with NNFMP; 26 Aug 2010 02:28:26 -0000
Received: from [127.0.0.1] by omp126.mail.gq1.yahoo.com with NNFMP; 26 Aug 2010 02:28:26 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 301341.60641.bm@omp126.mail.gq1.yahoo.com
Received: (qmail 51981 invoked by uid 60001); 26 Aug 2010 02:28:26 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1282789706; bh=leWTUQEOesPhm6BCfz1Tg54qvWqyFIgWTwqjBXtSMy8=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=OcVQUmiN+nv27QadlK/9p7hvLWQKRMtT5niOPsZp3ltykhwiTMOElFVSsoXChhbxsGZ3YchcslpHVE4j6TL1WSmuK369Wdc3JmZaDQpFPAjl3FJ1PkT7nDwOMz62JABqW3Nj/pTyMifKozs5iNvGNfGQc9o4SvxukO4vM5f0IAk=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=spbhlwmfsDDz2iMeX+bh1K0ika+PPBi/bCgmgyX8XnOLrFm+WZTT+ZSB0HK0BYj+9WvumMfF8v4eIeL2Cu/cBjbtC/fKbKKfi5pBLBMQbz2IQVahhTNnl1FrJO4SLWHxd4viDR55Zpb39spvz5PP9OFFn+Xc/+AR1E/b7Vb0iIM=;
Message-ID: <194169.50990.qm@web114512.mail.gq1.yahoo.com>
X-YMail-OSG: fqvKj6kVM1mQrdgvI1kXhdwO_f2N3krtPLnfzDXCgfKjvjp
 9zeV9kDHUjc.x89eE7XxGKtwu5T9zd6aeGWQ77yVFYoLrK.tgJMkD6cjG0NA
 SIFow_Z6Pm.zLIRLyl0of5wG2kZaCmemuHtFRkY7lHrwQVVWD7rRt2Ahe8k4
 3Dx_sm12aZvzmTqxGBmfy_jJcWa_jarPAt1BMZjWicLJjHTV5PaB869Bmk9B
 JIipliewNbYr7fCAMM5M3t8qUZIlUP8H0nr0bYwgfe7dT8tdPKVsYj0gDyt3
 CCcE01kA9vYr1QYj4GAGy..sMDPbZ
Received: from [67.180.86.237] by web114512.mail.gq1.yahoo.com via HTTP; Wed, 25 Aug 2010 19:28:26 PDT
X-Mailer: YahooMailRC/470 YahooMailWebService/0.8.105.279950
References: <715590.78832.qm@web114505.mail.gq1.yahoo.com> <AANLkTinQ_xeerXn9V7+M4=M0_NHEOgqn=ZsUu=xvMGox@mail.gmail.com>
Date: Wed, 25 Aug 2010 19:28:26 -0700 (PDT)
From: C J <c.joshi26@yahoo.com>
Subject: Re: Child processes on datanodes/task trackers
To: general@hadoop.apache.org
In-Reply-To: <AANLkTinQ_xeerXn9V7+M4=M0_NHEOgqn=ZsUu=xvMGox@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1202581762-1282789706=:50990"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-1202581762-1282789706=:50990
Content-Type: text/plain; charset=us-ascii

Thanks for your reply.

Some of these child tasks belong to successful jobs. I am wondering why they are 
still hanging there for long finished jobs.





________________________________
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Sent: Wed, August 25, 2010 4:17:38 PM
Subject: Re: Child processes on datanodes/task trackers

Use jps to find out pid of the Child.
Then use this to find out which job the Child belongs to:
ps aux | grep <pid>

On Wed, Aug 25, 2010 at 12:20 PM, C J <c.joshi26@yahoo.com> wrote:

> Hi,
>
> I wanted to know why I see running Child processes on my datanodes even
> though
> there is no job running at that time. Are these left over from failed
> attempts?
>
> Is there anything I can do to keep these clean?
>
> Thanks,
> Deepika
>
>
>



      
--0-1202581762-1282789706=:50990--


From general-return-2050-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 02:35:05 2010
Return-Path: <general-return-2050-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17906 invoked from network); 26 Aug 2010 02:35:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 02:35:05 -0000
Received: (qmail 42517 invoked by uid 500); 26 Aug 2010 02:35:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42357 invoked by uid 500); 26 Aug 2010 02:35:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42349 invoked by uid 99); 26 Aug 2010 02:35:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 02:35:02 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 02:34:56 +0000
Received: by qwk3 with SMTP id 3so1457440qwk.35
        for <general@hadoop.apache.org>; Wed, 25 Aug 2010 19:34:35 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=1V3JZF+LRjRsF7V7SW7cpz1UElikjdheQvoeL/FmtXs=;
        b=Oe35mSjzjL4gigXBIrWHNkHePJRur9kIKepjSfAl33sMa0x/YCcwz8QdwZrx/aA7lj
         OaPz/Didd7BBmGSZQhjBPGai6Nt0ZireX6P2qdIPnhJAIvZ+5BDoqKvdTqsxS5gLskO8
         013Uc3/9Pwq71lu7FH3OpMTdGv/Y0eKFkw2DQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=qICnXR0QXwfdRoEX1Znj0v4TXbGDKpxxAJ0ZdRzlgZKfKbW+q+YXrnMlKFaC2kBLU1
         IvTTurZ3FXXEq71yIg7QjSJRxkvK4tvHs57InhUQSSF+QgkjQXPXJAFFEshKo2FRC35K
         nGq2pvlJvL/ql0wIhMjxQfHlSOZXWVT9e2Dxg=
MIME-Version: 1.0
Received: by 10.229.60.200 with SMTP id q8mr5380839qch.209.1282790075193; Wed,
 25 Aug 2010 19:34:35 -0700 (PDT)
Received: by 10.229.48.129 with HTTP; Wed, 25 Aug 2010 19:34:35 -0700 (PDT)
In-Reply-To: <194169.50990.qm@web114512.mail.gq1.yahoo.com>
References: <715590.78832.qm@web114505.mail.gq1.yahoo.com>
	<AANLkTinQ_xeerXn9V7+M4=M0_NHEOgqn=ZsUu=xvMGox@mail.gmail.com>
	<194169.50990.qm@web114512.mail.gq1.yahoo.com>
Date: Wed, 25 Aug 2010 19:34:35 -0700
Message-ID: <AANLkTim9VK5Ee1ZE8dcbbJXcmOrjTO2vu+8pL5bEReA8@mail.gmail.com>
Subject: Re: Child processes on datanodes/task trackers
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001517576cb6c0da8e048eb0d6d4

--001517576cb6c0da8e048eb0d6d4
Content-Type: text/plain; charset=ISO-8859-1

After you obtain pid, you can use jstack to see what the Child process was
doing.

What hadoop version are you using ?

On Wed, Aug 25, 2010 at 7:28 PM, C J <c.joshi26@yahoo.com> wrote:

> Thanks for your reply.
>
> Some of these child tasks belong to successful jobs. I am wondering why
> they are
> still hanging there for long finished jobs.
>
>
>
>
>
> ________________________________
> From: Ted Yu <yuzhihong@gmail.com>
> To: general@hadoop.apache.org
> Sent: Wed, August 25, 2010 4:17:38 PM
> Subject: Re: Child processes on datanodes/task trackers
>
> Use jps to find out pid of the Child.
> Then use this to find out which job the Child belongs to:
> ps aux | grep <pid>
>
> On Wed, Aug 25, 2010 at 12:20 PM, C J <c.joshi26@yahoo.com> wrote:
>
> > Hi,
> >
> > I wanted to know why I see running Child processes on my datanodes even
> > though
> > there is no job running at that time. Are these left over from failed
> > attempts?
> >
> > Is there anything I can do to keep these clean?
> >
> > Thanks,
> > Deepika
> >
> >
> >
>
>
>
>
>

--001517576cb6c0da8e048eb0d6d4--

From general-return-2051-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 03:00:08 2010
Return-Path: <general-return-2051-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25047 invoked from network); 26 Aug 2010 03:00:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 03:00:07 -0000
Received: (qmail 81998 invoked by uid 500); 26 Aug 2010 03:00:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81626 invoked by uid 500); 26 Aug 2010 03:00:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81618 invoked by uid 99); 26 Aug 2010 03:00:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 03:00:05 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [98.136.44.37] (HELO n61.bullet.mail.sp1.yahoo.com) (98.136.44.37)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 26 Aug 2010 02:59:55 +0000
Received: from [216.252.122.216] by n61.bullet.mail.sp1.yahoo.com with NNFMP; 26 Aug 2010 02:59:35 -0000
Received: from [67.195.9.83] by t1.bullet.sp1.yahoo.com with NNFMP; 26 Aug 2010 02:59:35 -0000
Received: from [98.137.27.128] by t3.bullet.mail.gq1.yahoo.com with NNFMP; 26 Aug 2010 02:59:34 -0000
Received: from [127.0.0.1] by omp202.mail.gq1.yahoo.com with NNFMP; 26 Aug 2010 02:59:34 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 918454.70232.bm@omp202.mail.gq1.yahoo.com
Received: (qmail 46932 invoked by uid 60001); 26 Aug 2010 02:59:34 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1282791574; bh=9gr90DBHrdb4z8wKC5dpYR576MeWk3aUHOdXOnKWl/g=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=LVUnd12kuAY5B0hj0lRzW9oFpbMVeC1QPPnbvSY5aQH1c8VRwAWKoMLGvBM+VOMSsRAqiGI8mvdksUclypvaDrEULmCdWhhv5WTrr2WT5WucyCN3rqU+jRTclnQEp7UIT+LOEm8SpcBfPux/yO+ftAP0AvYsQYd0lNebKiwRl5s=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=l37DNp0x7bnsABD+PWkPzOB7TqMMCK27srZdIm8yu0QJDKZHGOTROmr27H0mryQ9qj/3bpYDDbveSg2HoHM52CnkFCuMD+85mDqXKC/nM5nDZMTZfWRGyVk1QL+RogdpW/cChOoxAXKRgvE0M93vnf37yiJU7foif1flGJZQ/m8=;
Message-ID: <698249.46749.qm@web114510.mail.gq1.yahoo.com>
X-YMail-OSG: 43iyqEsVM1kRrmUktws.cMt68Shi9M87nNTBtR.IZ8PFMLC
 xsY1dBxrm8fTsNXB65T9wj7AJjve19dANalcU18NHWadzoSOgpDwROWIfgS7
 OZA86gozLQEwdH5YKDq8bk6BNodTAOBvejPOwzeLPSP.P7nwN9uj3rCEM0fU
 dsx.Dn7u5aVmaaL20AEf9fXxitEz7h1Q7HJmLSeNT1oWCyO1a45BoEyf0Qf.
 tGf8Xh0ot0pZN3N.GfUWosnQBi14RRbKLtUPE.U2GAb_s4VuSmACqWkYV.W4
 Ed8wo8D1RlIZ0ODFdsx1O6NqS4ETaF6oMtbyZLN26pzZ43iyG1bI2tJKz.yn
 TryAt0w--
Received: from [67.180.86.237] by web114510.mail.gq1.yahoo.com via HTTP; Wed, 25 Aug 2010 19:59:34 PDT
X-Mailer: YahooMailRC/470 YahooMailWebService/0.8.105.279950
References: <715590.78832.qm@web114505.mail.gq1.yahoo.com> <AANLkTinQ_xeerXn9V7+M4=M0_NHEOgqn=ZsUu=xvMGox@mail.gmail.com> <194169.50990.qm@web114512.mail.gq1.yahoo.com> <AANLkTim9VK5Ee1ZE8dcbbJXcmOrjTO2vu+8pL5bEReA8@mail.gmail.com>
Date: Wed, 25 Aug 2010 19:59:34 -0700 (PDT)
From: C J <c.joshi26@yahoo.com>
Subject: Re: Child processes on datanodes/task trackers
To: general@hadoop.apache.org
In-Reply-To: <AANLkTim9VK5Ee1ZE8dcbbJXcmOrjTO2vu+8pL5bEReA8@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-793388696-1282791574=:46749"

--0-793388696-1282791574=:46749
Content-Type: text/plain; charset=us-ascii

Thanks Ted!

I did a jstack and it seems there is an issue with ehcache that I am using in 
the mapper task. 


"net.sf.ehcache.CacheManager@57ac3379" daemon prio=10 tid=0x0000000059180800 
nid=0x379e in Object.wait() [0x0000000041506000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00002aaabb0b89a8> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0x00002aaabb0b89a8> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

   Locked ownable synchronizers:
        - None
.
.
.


The hadoop version I am using is 0.20.2.

Thanks.



________________________________
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Sent: Wed, August 25, 2010 7:34:35 PM
Subject: Re: Child processes on datanodes/task trackers

After you obtain pid, you can use jstack to see what the Child process was
doing.

What hadoop version are you using ?

On Wed, Aug 25, 2010 at 7:28 PM, C J <c.joshi26@yahoo.com> wrote:

> Thanks for your reply.
>
> Some of these child tasks belong to successful jobs. I am wondering why
> they are
> still hanging there for long finished jobs.
>
>
>
>
>
> ________________________________
> From: Ted Yu <yuzhihong@gmail.com>
> To: general@hadoop.apache.org
> Sent: Wed, August 25, 2010 4:17:38 PM
> Subject: Re: Child processes on datanodes/task trackers
>
> Use jps to find out pid of the Child.
> Then use this to find out which job the Child belongs to:
> ps aux | grep <pid>
>
> On Wed, Aug 25, 2010 at 12:20 PM, C J <c.joshi26@yahoo.com> wrote:
>
> > Hi,
> >
> > I wanted to know why I see running Child processes on my datanodes even
> > though
> > there is no job running at that time. Are these left over from failed
> > attempts?
> >
> > Is there anything I can do to keep these clean?
> >
> > Thanks,
> > Deepika
> >
> >
> >
>
>
>
>
>



      
--0-793388696-1282791574=:46749--


From general-return-2052-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 03:39:35 2010
Return-Path: <general-return-2052-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39506 invoked from network); 26 Aug 2010 03:39:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 03:39:35 -0000
Received: (qmail 99047 invoked by uid 500); 26 Aug 2010 03:39:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98740 invoked by uid 500); 26 Aug 2010 03:39:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98732 invoked by uid 99); 26 Aug 2010 03:39:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 03:39:29 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com designates 209.85.216.176 as permitted sender)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 03:39:22 +0000
Received: by qyk2 with SMTP id 2so1545792qyk.14
        for <general@hadoop.apache.org>; Wed, 25 Aug 2010 20:39:02 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=whTTiuXx2recTIlCMLDII4FNHLMrvwk96blUQulimo4=;
        b=Xx8NiMh42df0AJERBgE2OyPt44QIkVsHSAcZ2zo1r/9Xw+AOB0cQC3R7JRv8D+V7xb
         pIgSoRT34ofqlf2i/CWrxZWepM83h7zWkHGT58lUWAPXtzx6Vp1tqhd0zwNJ0yJfZQLa
         PmIT81eYW7I5Rw4/X7tLJKXKVWjsEhaN19ZQI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=I0yntf+LRCDnsRD9eWGdUNImHyJ/5r43EGTioyjJwWTfUaNFqbvyRh8Lb2R7MGZ1sz
         jpK8xk2Ys0Otv+5mDZOei18TUo3jR11xOEgaqD10nbqg+pwmjFHtvgOfJpTFLdXs45mE
         +du5D3gdYAIIynih3xIOT1/VLet2B9u6KJpQA=
MIME-Version: 1.0
Received: by 10.229.71.69 with SMTP id g5mr5536014qcj.176.1282793941909; Wed,
 25 Aug 2010 20:39:01 -0700 (PDT)
Received: by 10.229.48.129 with HTTP; Wed, 25 Aug 2010 20:39:01 -0700 (PDT)
In-Reply-To: <698249.46749.qm@web114510.mail.gq1.yahoo.com>
References: <715590.78832.qm@web114505.mail.gq1.yahoo.com>
	<AANLkTinQ_xeerXn9V7+M4=M0_NHEOgqn=ZsUu=xvMGox@mail.gmail.com>
	<194169.50990.qm@web114512.mail.gq1.yahoo.com>
	<AANLkTim9VK5Ee1ZE8dcbbJXcmOrjTO2vu+8pL5bEReA8@mail.gmail.com>
	<698249.46749.qm@web114510.mail.gq1.yahoo.com>
Date: Wed, 25 Aug 2010 20:39:01 -0700
Message-ID: <AANLkTimUdW9eObq8MMF2zPMt=ggNhN3oERmwXEpcc-GE@mail.gmail.com>
Subject: Re: Child processes on datanodes/task trackers
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64bb8043a4317048eb1bd39

--0016e64bb8043a4317048eb1bd39
Content-Type: text/plain; charset=ISO-8859-1

I don't use ehcache.
Did you forget to close CacheManager at the end of your job by any chance ?

On Wed, Aug 25, 2010 at 7:59 PM, C J <c.joshi26@yahoo.com> wrote:

> Thanks Ted!
>
> I did a jstack and it seems there is an issue with ehcache that I am using
> in
> the mapper task.
>
>
> "net.sf.ehcache.CacheManager@57ac3379" daemon prio=10
> tid=0x0000000059180800
> nid=0x379e in Object.wait() [0x0000000041506000]
>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0x00002aaabb0b89a8> (a java.util.TaskQueue)
>        at java.util.TimerThread.mainLoop(Timer.java:509)
>        - locked <0x00002aaabb0b89a8> (a java.util.TaskQueue)
>        at java.util.TimerThread.run(Timer.java:462)
>
>   Locked ownable synchronizers:
>        - None
> .
> .
> .
>
>
> The hadoop version I am using is 0.20.2.
>
> Thanks.
>
>
>
> ________________________________
> From: Ted Yu <yuzhihong@gmail.com>
> To: general@hadoop.apache.org
> Sent: Wed, August 25, 2010 7:34:35 PM
> Subject: Re: Child processes on datanodes/task trackers
>
> After you obtain pid, you can use jstack to see what the Child process was
> doing.
>
> What hadoop version are you using ?
>
> On Wed, Aug 25, 2010 at 7:28 PM, C J <c.joshi26@yahoo.com> wrote:
>
> > Thanks for your reply.
> >
> > Some of these child tasks belong to successful jobs. I am wondering why
> > they are
> > still hanging there for long finished jobs.
> >
> >
> >
> >
> >
> > ________________________________
> > From: Ted Yu <yuzhihong@gmail.com>
> > To: general@hadoop.apache.org
> > Sent: Wed, August 25, 2010 4:17:38 PM
> > Subject: Re: Child processes on datanodes/task trackers
> >
> > Use jps to find out pid of the Child.
> > Then use this to find out which job the Child belongs to:
> > ps aux | grep <pid>
> >
> > On Wed, Aug 25, 2010 at 12:20 PM, C J <c.joshi26@yahoo.com> wrote:
> >
> > > Hi,
> > >
> > > I wanted to know why I see running Child processes on my datanodes even
> > > though
> > > there is no job running at that time. Are these left over from failed
> > > attempts?
> > >
> > > Is there anything I can do to keep these clean?
> > >
> > > Thanks,
> > > Deepika
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>

--0016e64bb8043a4317048eb1bd39--

From general-return-2053-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 09:41:38 2010
Return-Path: <general-return-2053-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44990 invoked from network); 26 Aug 2010 09:41:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 09:41:38 -0000
Received: (qmail 67203 invoked by uid 500); 26 Aug 2010 09:41:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66716 invoked by uid 500); 26 Aug 2010 09:41:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66703 invoked by uid 99); 26 Aug 2010 09:41:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 09:41:31 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 09:41:21 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 616081BA524
	for <general@hadoop.apache.org>; Thu, 26 Aug 2010 10:41:00 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id 98LMn3+kaKgl for <general@hadoop.apache.org>;
	Thu, 26 Aug 2010 10:41:00 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by colossus.hpl.hp.com (Postfix) with ESMTPS id D2EDB1BA4BC
	for <general@hadoop.apache.org>; Thu, 26 Aug 2010 10:40:59 +0100 (BST)
MailScanner-NULL-Check: 1283420447.3031@YJXMMmOsUySvf7rml80hxQ
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o7Q9ekbC013022
	for <general@hadoop.apache.org>; Thu, 26 Aug 2010 10:40:46 +0100 (BST)
Message-ID: <4C76369E.5030702@apache.org>
Date: Thu, 26 Aug 2010 10:40:46 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.11) Gecko/20100713 Thunderbird/3.0.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Child processes on datanodes/task trackers
References: <715590.78832.qm@web114505.mail.gq1.yahoo.com>
In-Reply-To: <715590.78832.qm@web114505.mail.gq1.yahoo.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o7Q9ekbC013022
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 25/08/10 20:20, C J wrote:
> Hi,
>
> I wanted to know why I see running Child processes on my datanodes even though
> there is no job running at that time. Are these left over from failed attempts?
>
> Is there anything I can do to keep these clean?
>
> Thanks,
> Deepika
>
>
>

could be a bug somewhere in Hadoop or your code.

1. jps -v gives you the process ids,

2. jstack -l <pid> gives you the stack dumps and locks for the given 
process, so you can see where the threads are -this will help you track 
down the problem

3. It is, sadly, going to be you who gets to track down the problem. But 
if you conclude that it is hadoop and not your code at fault, those 
stack traces should be attached to the JIRA issue you can create

From general-return-2054-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 12:57:10 2010
Return-Path: <general-return-2054-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 7493 invoked from network); 26 Aug 2010 12:57:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 12:57:09 -0000
Received: (qmail 98603 invoked by uid 500); 26 Aug 2010 12:57:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98180 invoked by uid 500); 26 Aug 2010 12:57:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98172 invoked by uid 99); 26 Aug 2010 12:57:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 12:57:04 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=FREEMAIL_FROM,FROM_LOCAL_NOVOWEL,HK_RANDOM_ENVFROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of sssssssenator@gmail.com designates 209.85.215.176 as permitted sender)
Received: from [209.85.215.176] (HELO mail-ey0-f176.google.com) (209.85.215.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 12:56:58 +0000
Received: by eydd26 with SMTP id d26so1571008eyd.35
        for <general@hadoop.apache.org>; Thu, 26 Aug 2010 05:56:37 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=jNxXetL+JBF06/zS72KD59S1SuqArtNOozNkZ01SFcA=;
        b=UONfuGbzbCCDfjyaCyA3GbZkfWSF5FvgAIViGMnFBpc1CRcXtTBV2L2Sx/vYUznGU0
         kf0AyjVdtmsf3bF3bNy9BgArAnOA98aob0gCCw8JX/HfwueqoUSOpqsZMwyvexR9jJE1
         +MSG5UANcC9Yl/egoR9je7ice8zq0/US/P4nI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=afNM/s8K9xxo4tDjyY1KE+aEDv3M4fvCMBaSmrb5gaSPGGxl6NDp9jenVMuZC/IDlC
         4u5OtUapRO/v01/2gt8uTc5QsV7y0Y3ZlxvyYqDuKmXPoTzAnZuuw9Dd/ngTVoeOTweK
         Lfs6Vvk0z3ir4M6etBucH9qKHlBqfLclyZEDo=
MIME-Version: 1.0
Received: by 10.216.54.202 with SMTP id i52mr8909179wec.40.1282827396940; Thu,
 26 Aug 2010 05:56:36 -0700 (PDT)
Received: by 10.216.154.132 with HTTP; Thu, 26 Aug 2010 05:56:36 -0700 (PDT)
Date: Thu, 26 Aug 2010 18:26:36 +0530
Message-ID: <AANLkTim-U5DaAQif+mzDfhzcg35+cnSYfp553-qrz_10@mail.gmail.com>
Subject: WARNING : There are about 1 missing blocks. Please check the log or
 run fsck.
From: vaibhav negi <sssssssenator@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6de04c44d8472048eb987a9

--0016e6de04c44d8472048eb987a9
Content-Type: text/plain; charset=ISO-8859-1

Hi ,

I am using hadoop version : 0.20.2 . I am getting this error message.

WARNING : There are about 1 missing blocks. Please check the log or run
fsck.

What to do? Is there any problem in the cluster

Thanks and Best Regards

Vaibhav Negi

--0016e6de04c44d8472048eb987a9--

From general-return-2055-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 14:12:56 2010
Return-Path: <general-return-2055-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33292 invoked from network); 26 Aug 2010 14:12:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 14:12:56 -0000
Received: (qmail 89446 invoked by uid 500); 26 Aug 2010 14:12:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89044 invoked by uid 500); 26 Aug 2010 14:12:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89035 invoked by uid 99); 26 Aug 2010 14:12:51 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 14:12:51 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 14:12:26 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 307421BA4BC
	for <general@hadoop.apache.org>; Thu, 26 Aug 2010 15:12:06 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id id0SxtFBOGlb for <general@hadoop.apache.org>;
	Thu, 26 Aug 2010 15:12:05 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by colossus.hpl.hp.com (Postfix) with ESMTPS id B7E8A1BA4A7
	for <general@hadoop.apache.org>; Thu, 26 Aug 2010 15:12:05 +0100 (BST)
MailScanner-NULL-Check: 1283436713.07572@4JPVfvEwdszzcCQNQiPjwQ
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o7QEBqjX020302
	for <general@hadoop.apache.org>; Thu, 26 Aug 2010 15:11:52 +0100 (BST)
Message-ID: <4C767628.7030803@apache.org>
Date: Thu, 26 Aug 2010 15:11:52 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.11) Gecko/20100713 Thunderbird/3.0.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <AANLkTikv9OFf7uX_PPThvxCceq+c_TJATt3QvE_W_T=2@mail.gmail.com> <809E9407-FBA7-4414-90ED-D3AA166CE1D0@yahoo-inc.com>
In-Reply-To: <809E9407-FBA7-4414-90ED-D3AA166CE1D0@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o7QEBqjX020302
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 25/08/10 18:59, Arun C Murthy wrote:
> On Aug 25, 2010, at 10:46 AM, Hemanth Yamijala wrote:
>
>> Arun,
>>
>> How much time do you think it would take to have a version of 0.20
>> with the security features in it ready ? In a different thread, Owen
>> has started discussing plans around 0.22. Do you think this effort
>> would affect 0.22 release ?
>>
>
> I think it should be fairly trivial to get this release out - most of
> the effort is just the mechanics of committing the patches to an Apache
> branch from the yahoo git repository, creating a release candidate and
> calling it a success! *smile*

oh, and testing it..



what scalability patches like HDFS-599 are in?

From general-return-2056-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 16:10:52 2010
Return-Path: <general-return-2056-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 74811 invoked from network); 26 Aug 2010 16:10:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 16:10:52 -0000
Received: (qmail 4455 invoked by uid 500); 26 Aug 2010 16:10:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4290 invoked by uid 500); 26 Aug 2010 16:10:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4282 invoked by uid 99); 26 Aug 2010 16:10:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 16:10:50 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 16:10:44 +0000
Received: from [192.168.1.71] (snvvpn1-10-72-244-c206.hq.corp.yahoo.com [10.72.244.206])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7QG9biQ070318
	for <general@hadoop.apache.org>; Thu, 26 Aug 2010 09:09:38 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=Kev8c3yDqqcd6oB6RCHfvFgFozfzqX4lXwsX9AO8QG+pPHdNdfXhrog6IjsYyJn/
Message-Id: <BBF5AC4B-FAB0-4BB2-AE9F-C7008D4275DF@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <4C767628.7030803@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Date: Thu, 26 Aug 2010 09:09:38 -0700
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <AANLkTikv9OFf7uX_PPThvxCceq+c_TJATt3QvE_W_T=2@mail.gmail.com> <809E9407-FBA7-4414-90ED-D3AA166CE1D0@yahoo-inc.com> <4C767628.7030803@apache.org>
X-Mailer: Apple Mail (2.936)


On Aug 26, 2010, at 7:11 AM, Steve Loughran wrote:

> On 25/08/10 18:59, Arun C Murthy wrote:
>> On Aug 25, 2010, at 10:46 AM, Hemanth Yamijala wrote:
>>
>>> Arun,
>>>
>>> How much time do you think it would take to have a version of 0.20
>>> with the security features in it ready ? In a different thread, Owen
>>> has started discussing plans around 0.22. Do you think this effort
>>> would affect 0.22 release ?
>>>
>>
>> I think it should be fairly trivial to get this release out - most of
>> the effort is just the mechanics of committing the patches to an  
>> Apache
>> branch from the yahoo git repository, creating a release candidate  
>> and
>> calling it a success! *smile*
>
> oh, and testing it..
>

Already is! *smile*
It's running on 4k clusters in production at this point...


From general-return-2057-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 16:41:37 2010
Return-Path: <general-return-2057-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85947 invoked from network); 26 Aug 2010 16:41:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 16:41:37 -0000
Received: (qmail 49502 invoked by uid 500); 26 Aug 2010 16:41:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49441 invoked by uid 500); 26 Aug 2010 16:41:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49433 invoked by uid 99); 26 Aug 2010 16:41:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 16:41:35 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 16:41:27 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 518E1B7D73
	for <general@hadoop.apache.org>; Thu, 26 Aug 2010 17:41:05 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id vkXuVUgRsRxk for <general@hadoop.apache.org>;
	Thu, 26 Aug 2010 17:41:04 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 4F1D9B7D65
	for <general@hadoop.apache.org>; Thu, 26 Aug 2010 17:41:03 +0100 (BST)
MailScanner-NULL-Check: 1283445651.2027@sDp4v9RMaYVLsIFuYYPlEQ
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o7QGenKV023687
	for <general@hadoop.apache.org>; Thu, 26 Aug 2010 17:40:49 +0100 (BST)
Message-ID: <4C769911.4050604@apache.org>
Date: Thu, 26 Aug 2010 17:40:49 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.11) Gecko/20100713 Thunderbird/3.0.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <AANLkTikv9OFf7uX_PPThvxCceq+c_TJATt3QvE_W_T=2@mail.gmail.com> <809E9407-FBA7-4414-90ED-D3AA166CE1D0@yahoo-inc.com> <4C767628.7030803@apache.org> <BBF5AC4B-FAB0-4BB2-AE9F-C7008D4275DF@yahoo-inc.com>
In-Reply-To: <BBF5AC4B-FAB0-4BB2-AE9F-C7008D4275DF@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o7QGenKV023687
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 26/08/10 17:09, Arun C Murthy wrote:
>
> On Aug 26, 2010, at 7:11 AM, Steve Loughran wrote:
>
>> On 25/08/10 18:59, Arun C Murthy wrote:
>>> On Aug 25, 2010, at 10:46 AM, Hemanth Yamijala wrote:
>>>
>>>> Arun,
>>>>
>>>> How much time do you think it would take to have a version of 0.20
>>>> with the security features in it ready ? In a different thread, Owen
>>>> has started discussing plans around 0.22. Do you think this effort
>>>> would affect 0.22 release ?
>>>>
>>>
>>> I think it should be fairly trivial to get this release out - most of
>>> the effort is just the mechanics of committing the patches to an Apache
>>> branch from the yahoo git repository, creating a release candidate and
>>> calling it a success! *smile*
>>
>> oh, and testing it..
>>
>
> Already is! *smile*
> It's running on 4k clusters in production at this point...
>

+1 then, ship it.

From general-return-2058-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 16:42:34 2010
Return-Path: <general-return-2058-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 86530 invoked from network); 26 Aug 2010 16:42:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 16:42:34 -0000
Received: (qmail 51108 invoked by uid 500); 26 Aug 2010 16:42:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51029 invoked by uid 500); 26 Aug 2010 16:42:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51021 invoked by uid 99); 26 Aug 2010 16:42:32 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 16:42:32 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates 209.85.216.169 as permitted sender)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 16:42:09 +0000
Received: by qyk12 with SMTP id 12so7438809qyk.14
        for <general@hadoop.apache.org>; Thu, 26 Aug 2010 09:41:48 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=EHeoJNRe59gS9qxKoFXMjRXNX81fAtCgEXm1LfGr14Y=;
        b=HuwlJLFtxjlaRPXwb4VXcJGqaoEYVFbsR2JnQMTuoYZiicSedGZXY6m1lmDMbzC7yQ
         yU3nKtkfdoFxqOpgSAy3pfBOcWjnIjpQTZYL+t1w0UTJTCiJsnV32l+Vlk0wEpQd6hqV
         aCnb7ukNphWC2uD2XfEFde8t7GjjuPjZEpIck=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=wsATTI1DNFKnDN5p4Qzh2bbpR0WBXzHcUabw5NX9G/6A2lJoboKc0/A+4A6/59izOP
         B6pgTYGfKUbGUZLF+NZdbiB3io+e4rCp6whqhQjWGM/vpSGCtkcEauwXDnPwiQXb4p8T
         mwlZEhTCCDmR13Gnv7/Z6u5VCKGoxI0wELnBo=
MIME-Version: 1.0
Received: by 10.224.119.18 with SMTP id x18mr6788833qaq.302.1282840908695;
 Thu, 26 Aug 2010 09:41:48 -0700 (PDT)
Received: by 10.229.48.129 with HTTP; Thu, 26 Aug 2010 09:41:48 -0700 (PDT)
In-Reply-To: <AANLkTim-U5DaAQif+mzDfhzcg35+cnSYfp553-qrz_10@mail.gmail.com>
References: <AANLkTim-U5DaAQif+mzDfhzcg35+cnSYfp553-qrz_10@mail.gmail.com>
Date: Thu, 26 Aug 2010 09:41:48 -0700
Message-ID: <AANLkTi=pcnv8Gx6+557BdZrLoC33Naim6an1vt-7GQ-Q@mail.gmail.com>
Subject: Re: WARNING : There are about 1 missing blocks. Please check the log
 or run fsck.
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000feae9a993aa89f6048ebcacc9
X-Virus-Checked: Checked by ClamAV on apache.org

--000feae9a993aa89f6048ebcacc9
Content-Type: text/plain; charset=ISO-8859-1

Run fsck.

On Thu, Aug 26, 2010 at 5:56 AM, vaibhav negi <sssssssenator@gmail.com>wrote:

> Hi ,
>
> I am using hadoop version : 0.20.2 . I am getting this error message.
>
> WARNING : There are about 1 missing blocks. Please check the log or run
> fsck.
>
> What to do? Is there any problem in the cluster
>
> Thanks and Best Regards
>
> Vaibhav Negi
>

--000feae9a993aa89f6048ebcacc9--

From general-return-2059-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 17:04:04 2010
Return-Path: <general-return-2059-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93122 invoked from network); 26 Aug 2010 17:04:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 17:04:04 -0000
Received: (qmail 91584 invoked by uid 500); 26 Aug 2010 17:04:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91527 invoked by uid 500); 26 Aug 2010 17:04:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91516 invoked by uid 99); 26 Aug 2010 17:04:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 17:04:02 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.30.24] (HELO qmta02.emeryville.ca.mail.comcast.net) (76.96.30.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 17:03:54 +0000
Received: from omta17.emeryville.ca.mail.comcast.net ([76.96.30.73])
	by qmta02.emeryville.ca.mail.comcast.net with comcast
	id z51v1e0051afHeLA253aXX; Thu, 26 Aug 2010 17:03:34 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta17.emeryville.ca.mail.comcast.net with comcast
	id z53R1e00B2SbwD58d53Tcb; Thu, 26 Aug 2010 17:03:31 +0000
Message-Id: <232F87F7-2B71-43F3-8B6A-BBCC00106B0B@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <71D87762-A95B-434D-9FF6-8627AF65790A@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Pig to become a TLP
Date: Thu, 26 Aug 2010 10:03:25 -0700
References: <71D87762-A95B-434D-9FF6-8627AF65790A@yahoo-inc.com>
X-Mailer: Apple Mail (2.936)


On Aug 23, 2010, at 10:38 AM, Alan Gates wrote:

> I propose that Pig become a top level Apache project.

+1

From general-return-2060-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 17:18:18 2010
Return-Path: <general-return-2060-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1695 invoked from network); 26 Aug 2010 17:18:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 17:18:18 -0000
Received: (qmail 8157 invoked by uid 500); 26 Aug 2010 17:18:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8069 invoked by uid 500); 26 Aug 2010 17:18:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8058 invoked by uid 99); 26 Aug 2010 17:18:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 17:18:16 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 17:18:11 +0000
Received: from pugholehand-lm.corp.yahoo.com (pugholehand-lm.corp.yahoo.com [10.72.115.44])
	by mrout2.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7QHHbsL081750;
	Thu, 26 Aug 2010 10:17:37 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=cc:message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=Zrxj419QDu383L/c2vFEJn1R/HytwsLsMiQsFvh4DgjVeauO3/cwVgYGWGUZGnIV
Cc: pig-user@hadoop.apache.org
Message-Id: <EC416139-6DC5-4C81-80BF-23041D26F56B@yahoo-inc.com>
From: Alan Gates <gates@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <71D87762-A95B-434D-9FF6-8627AF65790A@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Pig to become a TLP
Date: Thu, 26 Aug 2010 10:17:37 -0700
References: <71D87762-A95B-434D-9FF6-8627AF65790A@yahoo-inc.com>
X-Mailer: Apple Mail (2.936)

With 15 +1 votes (14 from PMC members) the proposal passes.  Thanks  
for voting.

Owen, please push this to the Apache board for their consideration.

Alan.

On Aug 23, 2010, at 10:38 AM, Alan Gates wrote:

> I propose that Pig become a top level Apache project.
>
> The Pig development community has already voted on and approved this
> proposal.  In summary, the community voted that all current active
> committers (listed at http://hadoop.apache.org/pig/whoweare.html ) as
> of August 18 2010 should become PMC members, and that Olga Natkovich
> should be the PMC chair.  They also agreed that adopting bylaws should
> be among the first tasks of the new PMC.  Full details of the vote can
> be seen at http://bit.ly/c3GDyl
>
> Please vote on sending this proposal to the Apache Board.
>
> Below is a draft resolution to be sent to the Apache Board:
>
> Establish the Apache Pig Project
>
>         WHEREAS, the Board of Directors deems it to be in the best
>         interests of the Foundation and consistent with the
>         Foundation's purpose to establish a Project Management
>         Committee charged with the creation and maintenance of
>         open-source software related to parallel analysis of large
>         data sets for distribution at no charge to the public.
>
>         NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>         Committee (PMC), to be known as the "Apache Pig Project",
>         be and hereby is established pursuant to Bylaws of the
>         Foundation; and be it further
>
>         RESOLVED, that the Apache Pig Project be and hereby is
>         responsible for the creation and maintenance of software
>         related to parallel analysis of large data sets; and be
>         it further
>
>         RESOLVED, that the office of "Vice President, Apache Pig" be
>         and hereby is created, the person holding such office to
>         serve at the direction of the Board of Directors as the chair
>         of the Apache Pig Project, and to have primary responsibility
>         for management of the projects within the scope of
>         responsibility of the Apache Pig Project; and be it further
>
>         RESOLVED, that the persons listed immediately below be and
>         hereby are appointed to serve as the initial members of the
>         Apache Pig Project:
>
>           * Benjamin Reed          <breed@apache.org>
>           * Daniel Dai                  <daijy@apache.org>
>           * Alan Gates                 <gates@apache.org>
>           * Giridharen Kesavan   <gkesavan@apache.org>
>           * Olga Natkovich           <olga@apache.org>
>           * Pradeep Kamath        <pradeepkth@apache.org>
>           * Santhosh Srinivasan  <sms@apache.org>
>           * Yan Zhou                   <yanz@apache.org>
>           * Jeff Zhang                  <zjffdu@apache.org>
>           * Ashutosh Chauhan     <hashutosh@apache.org>
>           * Richard Ding               <rding@apache.org>
>           * Dmitriy Ryaboy            <dvryaboy@apache.org>
>           * Thejas Nair                  <thejas@apache.org>
>
>         NOW, THEREFORE, BE IT FURTHER RESOLVED, that Olga Natkovich
>         be appointed to the office of Vice President, Apache Pig, to
>         serve in accordance with and subject to the direction of the
>         Board of Directors and the Bylaws of the Foundation until
>         death, resignation, retirement, removal or disqualification,
>         or until a successor is appointed; and be it further
>
>         RESOLVED, that the initial Apache Pig PMC be and hereby is
>         tasked with the creation of a set of bylaws intended to
>         encourage open development and increased participation in the
>         Apache Pig Project; and be it further
>
>         RESOLVED, that the Apache Pig Project be and hereby
>         is tasked with the migration and rationalization of the Apache
>         Hadoop Pig sub-project; and be it further
>
>         RESOLVED, that all responsibilities pertaining to the Apache
>         Hadoop Pig sub-project encumbered upon the
>         Apache Hadoop Project are hereafter discharged.
>
>
>
>


From general-return-2061-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 18:32:16 2010
Return-Path: <general-return-2061-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31439 invoked from network); 26 Aug 2010 18:32:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 18:32:16 -0000
Received: (qmail 19152 invoked by uid 500); 26 Aug 2010 18:32:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18931 invoked by uid 500); 26 Aug 2010 18:32:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 257 invoked by uid 99); 25 Aug 2010 04:46:51 -0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of anant.bw@gmail.com designates 209.85.212.48 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=VTIeQ1uMgj6xzSJ3h7QYZyDFgu7zq20NtfArjLAsOZI=;
        b=knuWVGZ9UzQ+NLRPRi+q6hhk0yHgmjVe7t7TZ+yx7ZbftjTLLWoEI2HK6fTYmrFJPe
         6uJ23IwcBf+DBgjo+wMSXsaPXCj1fAYfVbYPLKBu7f/ietmqVNbtqHVf53sMH05fJviF
         9OXtzpClmKMokxAKDxEMCvqgj0ZFC6g4KkEU0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=nvC9zxshnAAPFlYFsy0J6UERNYknNK7E2mOrpkQ4bSp+nxj19Zwz8rveAT/EstNQ4W
         n6y1s208qpK+1So6gYcOEvJPJ2gwjLN1r01oWdLqDivpWs8hwGJwfRSwlBkM4wD2wRje
         pZ84K4TEfIaqALmYDM5su/M8sCX55f28oJ8po=
MIME-Version: 1.0
In-Reply-To: <C1EDB62C3771D74AA823DBBBBFB3F89D4E5DE391@SP2-EX07VS03.ds.corp.yahoo.com>
References: <71D87762-A95B-434D-9FF6-8627AF65790A@yahoo-inc.com>
	<71E1E01A-5389-48C2-9A43-9A8BD6B87A2A@yahoo-inc.com>
	<C1EDB62C3771D74AA823DBBBBFB3F89D4E5DE391@SP2-EX07VS03.ds.corp.yahoo.com>
Date: Wed, 25 Aug 2010 10:16:25 +0530
Message-ID: <AANLkTin1=Za+kPAy_97O8zcLxbA+U54-nHaje_44yMsj@mail.gmail.com>
Subject: Re: [VOTE] Pig to become a TLP
From: Anant Wadhokar <anant.bw@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c5a6376b2383048e9e902e

--001636c5a6376b2383048e9e902e
Content-Type: text/plain; charset=ISO-8859-1

+1

On Wed, Aug 25, 2010 at 12:04 AM, Olga Natkovich <olgan@yahoo-inc.com>wrote:

> +1
>
> > From: Alan Gates <gates@yahoo-inc.com>
> > Date: August 23, 2010 10:38:12 AM PDT
> > To: "general@hadoop.apache.org" <general@hadoop.apache.org>
> > Subject: [VOTE] Pig to become a TLP
> > Reply-To: "general@hadoop.apache.org" <general@hadoop.apache.org>
> >
> > I propose that Pig become a top level Apache project.
> >
> > The Pig development community has already voted on and approved this
> > proposal.  In summary, the community voted that all current active
> > committers (listed at http://hadoop.apache.org/pig/whoweare.html ) as
> > of August 18 2010 should become PMC members, and that Olga Natkovich
> > should be the PMC chair.  They also agreed that adopting bylaws should
> > be among the first tasks of the new PMC.  Full details of the vote can
> > be seen at http://bit.ly/c3GDyl
> >
> > Please vote on sending this proposal to the Apache Board.
> >
> > Below is a draft resolution to be sent to the Apache Board:
> >
> > Establish the Apache Pig Project
> >
> >         WHEREAS, the Board of Directors deems it to be in the best
> >         interests of the Foundation and consistent with the
> >         Foundation's purpose to establish a Project Management
> >         Committee charged with the creation and maintenance of
> >         open-source software related to parallel analysis of large
> >         data sets for distribution at no charge to the public.
> >
> >         NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> >         Committee (PMC), to be known as the "Apache Pig Project",
> >         be and hereby is established pursuant to Bylaws of the
> >         Foundation; and be it further
> >
> >         RESOLVED, that the Apache Pig Project be and hereby is
> >         responsible for the creation and maintenance of software
> >         related to parallel analysis of large data sets; and be
> >         it further
> >
> >         RESOLVED, that the office of "Vice President, Apache Pig" be
> >         and hereby is created, the person holding such office to
> >         serve at the direction of the Board of Directors as the chair
> >         of the Apache Pig Project, and to have primary responsibility
> >         for management of the projects within the scope of
> >         responsibility of the Apache Pig Project; and be it further
> >
> >         RESOLVED, that the persons listed immediately below be and
> >         hereby are appointed to serve as the initial members of the
> >         Apache Pig Project:
> >
> >           * Benjamin Reed          <breed@apache.org>
> >           * Daniel Dai                  <daijy@apache.org>
> >           * Alan Gates                 <gates@apache.org>
> >           * Giridharen Kesavan   <gkesavan@apache.org>
> >           * Olga Natkovich           <olga@apache.org>
> >           * Pradeep Kamath        <pradeepkth@apache.org>
> >           * Santhosh Srinivasan  <sms@apache.org>
> >           * Yan Zhou                   <yanz@apache.org>
> >           * Jeff Zhang                  <zjffdu@apache.org>
> >           * Ashutosh Chauhan     <hashutosh@apache.org>
> >           * Richard Ding               <rding@apache.org>
> >           * Dmitriy Ryaboy            <dvryaboy@apache.org>
> >           * Thejas Nair                  <thejas@apache.org>
> >
> >         NOW, THEREFORE, BE IT FURTHER RESOLVED, that Olga Natkovich
> >         be appointed to the office of Vice President, Apache Pig, to
> >         serve in accordance with and subject to the direction of the
> >         Board of Directors and the Bylaws of the Foundation until
> >         death, resignation, retirement, removal or disqualification,
> >         or until a successor is appointed; and be it further
> >
> >         RESOLVED, that the initial Apache Pig PMC be and hereby is
> >         tasked with the creation of a set of bylaws intended to
> >         encourage open development and increased participation in the
> >         Apache Pig Project; and be it further
> >
> >         RESOLVED, that the Apache Pig Project be and hereby
> >         is tasked with the migration and rationalization of the Apache
> >         Hadoop Pig sub-project; and be it further
> >
> >         RESOLVED, that all responsibilities pertaining to the Apache
> >         Hadoop Pig sub-project encumbered upon the
> >         Apache Hadoop Project are hereafter discharged.
> >
> >
> >
> >
>
>

--001636c5a6376b2383048e9e902e--

From general-return-2062-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 18:32:45 2010
Return-Path: <general-return-2062-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31832 invoked from network); 26 Aug 2010 18:32:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 18:32:44 -0000
Received: (qmail 20670 invoked by uid 500); 26 Aug 2010 18:32:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20584 invoked by uid 500); 26 Aug 2010 18:32:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 58803 invoked by uid 99); 26 Aug 2010 11:23:28 -0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of vaibhav.negi@one97.net designates 209.85.214.176 as permitted sender)
MIME-Version: 1.0
X-Originating-IP: [125.63.68.99]
Date: Thu, 26 Aug 2010 16:52:52 +0530
Message-ID: <AANLkTimmc4qQ34h1xS4Hzc2TorNDWZeDoMHfJ9kWaPVa@mail.gmail.com>
Subject: WARNING : There are about 1 missing blocks. Please check the log or
 run fsck.
From: Vaibhav Negi <vaibhav.negi@one97.net>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=002215048fb75a8e9b048eb838af

--002215048fb75a8e9b048eb838af
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Hi ,

I am using hadoop version : 0.20.2 . I am getting this error message.

WARNING : There are about 1 missing blocks. Please check the log or run
fsck.

What to do? Is there any problem in the cluster



Regards

Vaibhav Negi
Software Engineer=96 Engineering.
One97 Communications Ltd
P: +91 11 46500197      Extn:461
M: + 91 9582259332

Let=92s Get Talking!

--002215048fb75a8e9b048eb838af--

From general-return-2063-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 18:56:47 2010
Return-Path: <general-return-2063-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 36614 invoked from network); 26 Aug 2010 18:56:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 18:56:47 -0000
Received: (qmail 60510 invoked by uid 500); 26 Aug 2010 18:56:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60447 invoked by uid 500); 26 Aug 2010 18:56:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60439 invoked by uid 99); 26 Aug 2010 18:56:45 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 18:56:45 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of gallowaymd@ornl.gov designates 160.91.4.119 as permitted sender)
Received: from [160.91.4.119] (HELO emroute1.ornl.gov) (160.91.4.119)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 18:56:20 +0000
Received: from emroute1.ornl.gov ([127.0.0.1])
 by emroute1.ornl.gov (PMDF V6.5-x1 #31823)
 with ESMTP id <0L7R00G88WLBIE@emroute1.ornl.gov> for
 general@hadoop.apache.org; Thu, 26 Aug 2010 14:55:59 -0400 (EDT)
Received: from CONVERSION-DAEMON.emroute1.ornl.gov by emroute1.ornl.gov
 (PMDF V6.5-x1 #31823) id <0L7R00I01WLBCC@emroute1.ornl.gov> for
 general@hadoop.apache.org; Thu, 26 Aug 2010 14:55:59 -0400 (EDT)
Received: from exchhub2.ornl.gov (exchhub2.ornl.gov [160.91.2.112])
 by emroute1.ornl.gov (PMDF V6.5-x1 #31823)
 with ESMTPS id <0L7R00E3AWLBLR@emroute1.ornl.gov> for
 general@hadoop.apache.org; Thu, 26 Aug 2010 14:55:59 -0400 (EDT)
Received: from EXCHMB.ornl.gov ([160.91.2.201])
 by exchhub2.ornl.gov ([160.91.2.112]) with mapi; Thu,
 26 Aug 2010 14:55:58 -0400
Date: Thu, 26 Aug 2010 14:55:58 -0400
From: "Galloway, Michael D." <gallowaymd@ornl.gov>
Subject: Re: WARNING : There are about 1 missing blocks. Please check the log
 or run fsck.
In-reply-to: <AANLkTimmc4qQ34h1xS4Hzc2TorNDWZeDoMHfJ9kWaPVa@mail.gmail.com>
To: "'general@hadoop.apache.org'" <general@hadoop.apache.org>
Message-id: <C6E8EB1B59E1E942860E0410E28414FE39B4004E@EXCHMB.ornl.gov>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-language: en-US
Content-transfer-encoding: base64
Thread-Topic: WARNING : There are about 1 missing blocks. Please check the log
 or run fsck.
Thread-Index: ActFTRoYQvQRA0QuRma771CFEIEhogAAzf2u
Accept-Language: en-US
acceptlanguage: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
X-Virus-Checked: Checked by ClamAV on apache.org

QXMgc2FpZCBlYXJsaWVyIHRyeToNCg0KaGFkb29wIGZzY2sgLw0KDQoNCg0KLS0tLS0gT3JpZ2lu
YWwgTWVzc2FnZSAtLS0tLQ0KRnJvbTogVmFpYmhhdiBOZWdpIFttYWlsdG86dmFpYmhhdi5uZWdp
QG9uZTk3Lm5ldF0NClNlbnQ6IFRodXJzZGF5LCBBdWd1c3QgMjYsIDIwMTAgMDc6MjIgQU0KVG86
IGdlbmVyYWxAaGFkb29wLmFwYWNoZS5vcmcgPGdlbmVyYWxAaGFkb29wLmFwYWNoZS5vcmc+DQpT
dWJqZWN0OiBXQVJOSU5HIDogVGhlcmUgYXJlIGFib3V0IDEgbWlzc2luZyBibG9ja3MuIFBsZWFz
ZSBjaGVjayB0aGUgbG9nIG9yIHJ1biBmc2NrLg0KDQpIaSAsDQoNCkkgYW0gdXNpbmcgaGFkb29w
IHZlcnNpb24gOiAwLjIwLjIgLiBJIGFtIGdldHRpbmcgdGhpcyBlcnJvciBtZXNzYWdlLg0KDQpX
QVJOSU5HIDogVGhlcmUgYXJlIGFib3V0IDEgbWlzc2luZyBibG9ja3MuIFBsZWFzZSBjaGVjayB0
aGUgbG9nIG9yIHJ1bg0KZnNjay4NCg0KV2hhdCB0byBkbz8gSXMgdGhlcmUgYW55IHByb2JsZW0g
aW4gdGhlIGNsdXN0ZXINCg0KDQoNClJlZ2FyZHMNCg0KVmFpYmhhdiBOZWdpDQpTb2Z0d2FyZSBF
bmdpbmVlcuKAkyBFbmdpbmVlcmluZy4NCk9uZTk3IENvbW11bmljYXRpb25zIEx0ZA0KUDogKzkx
IDExIDQ2NTAwMTk3ICAgICAgRXh0bjo0NjENCk06ICsgOTEgOTU4MjI1OTMzMg0KDQpMZXTigJlz
IEdldCBUYWxraW5nIQ0K

From general-return-2064-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 19:09:15 2010
Return-Path: <general-return-2064-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42837 invoked from network); 26 Aug 2010 19:09:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 19:09:15 -0000
Received: (qmail 82853 invoked by uid 500); 26 Aug 2010 19:09:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82749 invoked by uid 500); 26 Aug 2010 19:09:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82741 invoked by uid 99); 26 Aug 2010 19:09:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 19:09:13 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 209.85.215.176 as permitted sender)
Received: from [209.85.215.176] (HELO mail-ey0-f176.google.com) (209.85.215.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 19:09:09 +0000
Received: by eydd26 with SMTP id d26so2049036eyd.35
        for <general@hadoop.apache.org>; Thu, 26 Aug 2010 12:08:48 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:content-type:content-transfer-encoding;
        bh=jUijN+3P0bhoadD+Hxrp8U/wYIgAIIcWu+jKY3pFo6Y=;
        b=f7rVni5ZNo0K6XNa39anpk3bTOJgljv8/A1q0ZV7fCYottGlfCa9y+TRSgGqIVR+ZI
         9CNvwghzDtuAcPOIqOxoqLSZL32nL+FTz3eK0RmzxrNSWIJqajoKSscVLOFgJLY9fU4L
         N9m+LwZIuKnLVxweZsKKE46Q9xFSsPW6g4dWY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=cWpG2uOZcROr6S2/91fovzgBE1rRzdqsa48AuySnkYM4NCePlTO0Vhsfi6ZxNw/o0Y
         M6TBfy14iNDIXIZAYeg/JBaqapqKtrr4xRwMw+7IaqPT/pImOovwhY3NELWLPFV8O3ak
         p0fAXoQSAa7yXLHZgUHHGGIeDnV3BOh3+J5MQ=
MIME-Version: 1.0
Received: by 10.216.232.90 with SMTP id m68mr9407875weq.10.1282849728056; Thu,
 26 Aug 2010 12:08:48 -0700 (PDT)
Sender: saint.ack@gmail.com
Received: by 10.216.131.24 with HTTP; Thu, 26 Aug 2010 12:08:47 -0700 (PDT)
In-Reply-To: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
Date: Thu, 26 Aug 2010 12:08:47 -0700
X-Google-Sender-Auth: -q0VLy-3uT0GdofsRxE-5WQ9R-Y
Message-ID: <AANLkTimBw3kDY6S1R-MN77LeqLbrr_96btr5PQitL47s@mail.gmail.com>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Mon, Aug 23, 2010 at 5:27 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:
> In the interim I'd like to propose we push a hadoop-0.20-security release
> off the Yahoo! patchset (http://github.com/yahoo/hadoop-common). This wil=
l
> ensure the community benefits from all the work done at Yahoo! for over 1=
2
> months *now*, and ensures that we do not have to wait until hadoop-0.22
> which has all of these patches.
>

Sounds good to me.  What will this release be called?  hadoop-0.20.3-securi=
ty?

> =A0Conceivably, one could imagine a Hadoop Security + Append release soon
> after.

Well, it'd probably be better if we just did an append release first?
A good few of us have been banging on the 0.20-append branch w/ a
while now and its for sure doing append better than 0.20 did (smile).

St.Ack

From general-return-2065-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 20:03:54 2010
Return-Path: <general-return-2065-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55932 invoked from network); 26 Aug 2010 20:03:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 20:03:53 -0000
Received: (qmail 58600 invoked by uid 500); 26 Aug 2010 20:03:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58505 invoked by uid 500); 26 Aug 2010 20:03:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58497 invoked by uid 99); 26 Aug 2010 20:03:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 20:03:51 +0000
X-ASF-Spam-Status: No, hits=2.7 required=10.0
	tests=RCVD_ILLEGAL_IP,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of athusoo@facebook.com designates 69.63.178.170 as permitted sender)
Received: from [69.63.178.170] (HELO mx-out.facebook.com) (69.63.178.170)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 20:03:46 +0000
Received: from [10.18.255.134] ([10.18.255.134:8416] helo=mail.thefacebook.com)
	by mta012.snc1.facebook.com (envelope-from <athusoo@facebook.com>)
	(ecelerity 2.2.2.45 r(34067)) with ESMTP
	id E7/2B-27680-E88C67C4; Thu, 26 Aug 2010 13:03:26 -0700
Received: from SC-MBX03.TheFacebook.com ([169.254.2.176]) by
 sc-hub04.TheFacebook.com ([fe80::8df5:7f90:d4a0:bb9%11]) with mapi; Thu, 26
 Aug 2010 13:01:20 -0700
From: Ashish Thusoo <athusoo@facebook.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: [VOTE] Hive as a TLP
Thread-Topic: [VOTE] Hive as a TLP
Thread-Index: ActFWXOffgxgalakRSm+ZBxRJ3CyBg==
Date: Thu, 26 Aug 2010 20:01:20 +0000
Message-ID: <09370B27F829314182C33A219530C104037ABC90@sc-mbx03.TheFacebook.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

The Hive development community voted and passed the following resolution. T=
he details of the vote is at

http://www.bit.ly/aJogyU

The PMC will comprise of the current committers on Hive (as of 8/24/2010) w=
ith Namit Jain being the chair.

Please vote on sending this resolution to the Apache Board.

Thanks,
Ashish

Draft Resolution to be sent to the Apache Board
-----------------------------------------------

Establish the Apache Hive Project

         WHEREAS, the Board of Directors deems it to be in the best
         interests of the Foundation and consistent with the
         Foundation's purpose to establish a Project Management
         Committee charged with the creation and maintenance of
         open-source software related to parallel analysis of large
         data sets for distribution at no charge to the public.

         NOW, THEREFORE, BE IT RESOLVED, that a Project Management
         Committee (PMC), to be known as the "Apache Hive Project",
         be and hereby is established pursuant to Bylaws of the
         Foundation; and be it further

         RESOLVED, that the Apache Hive Project be and hereby is
         responsible for the creation and maintenance of software
         related to parallel analysis of large data sets; and be
         it further

         RESOLVED, that the office of "Vice President, Apache Hive" be
         and hereby is created, the person holding such office to
         serve at the direction of the Board of Directors as the chair
         of the Apache Hive Project, and to have primary responsibility
         for management of the projects within the scope of
         responsibility of the Apache Hive Project; and be it further

         RESOLVED, that the persons listed immediately below be and
         hereby are appointed to serve as the initial members of the
         Apache Hive Project:
             * Namit Jain (namit@apache.org)
             * John Sichi (jvs@apache.org)
             * Zheng Shao (zshao@apache.org)
             * Edward Capriolo (appodictic@apache.org)
             * Raghotham Murthy (rsm@apache.org)
             * Ning Zhang (nzhang@apache.org)
             * Paul Yang (pauly@apache.org)
             * He Yongqiang (he yongqiang@apache.org)
             * Prasad Chakka (prasadc@apache.org)
             * Joydeep Sen Sarma (jsensarma@apache.org)
             * Ashish Thusoo (athusoo@apache.org)

         NOW, THEREFORE, BE IT FURTHER RESOLVED, that Namit Jain
         be appointed to the office of Vice President, Apache Hive, to
         serve in accordance with and subject to the direction of the
         Board of Directors and the Bylaws of the Foundation until
         death, resignation, retirement, removal or disqualification,
         or until a successor is appointed; and be it further

         RESOLVED, that the initial Apache Hive PMC be and hereby is
         tasked with the creation of a set of bylaws intended to
         encourage open development and increased participation in the
         Apache Hive Project; and be it further

         RESOLVED, that the Apache Hive Project be and hereby
         is tasked with the migration and rationalization of the Apache
         Hadoop Hive sub-project; and be it further

         RESOLVED, that all responsibilities pertaining to the Apache
         Hive sub-project encumbered upon the
         Apache Hadoop Project are hereafter discharged.

From general-return-2066-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 22:57:06 2010
Return-Path: <general-return-2066-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 13259 invoked from network); 26 Aug 2010 22:57:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 22:57:05 -0000
Received: (qmail 57673 invoked by uid 500); 26 Aug 2010 22:57:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57440 invoked by uid 500); 26 Aug 2010 22:57:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57432 invoked by uid 99); 26 Aug 2010 22:57:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 22:57:04 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.59.227] (HELO qmta12.westchester.pa.mail.comcast.net) (76.96.59.227)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 22:56:55 +0000
Received: from omta01.westchester.pa.mail.comcast.net ([76.96.62.11])
	by qmta12.westchester.pa.mail.comcast.net with comcast
	id z59E1e0070EZKEL5CAwbmy; Thu, 26 Aug 2010 22:56:35 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta01.westchester.pa.mail.comcast.net with comcast
	id zAwS1e00F2SbwD53MAwVVv; Thu, 26 Aug 2010 22:56:33 +0000
Message-Id: <6E66BD35-8825-495B-8DA4-575766CADD46@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <09370B27F829314182C33A219530C104037ABC90@sc-mbx03.TheFacebook.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Hive as a TLP
Date: Thu, 26 Aug 2010 15:56:25 -0700
References: <09370B27F829314182C33A219530C104037ABC90@sc-mbx03.TheFacebook.com>
X-Mailer: Apple Mail (2.936)


On Aug 26, 2010, at 1:01 PM, Ashish Thusoo wrote:

> The Hive development community voted and passed the following  
> resolution.

+1

-- Owen

From general-return-2067-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 23:09:27 2010
Return-Path: <general-return-2067-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19641 invoked from network); 26 Aug 2010 23:09:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 23:09:27 -0000
Received: (qmail 68839 invoked by uid 500); 26 Aug 2010 23:09:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68720 invoked by uid 500); 26 Aug 2010 23:09:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68712 invoked by uid 99); 26 Aug 2010 23:09:25 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 23:09:25 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 23:09:04 +0000
Received: from wlanvpn-snve-246-72.corp.yahoo.com ([172.21.148.72])
	by mrout1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7QN8K5N024353
	for <general@hadoop.apache.org>; Thu, 26 Aug 2010 16:08:20 -0700 (PDT)
Message-Id: <5EB1D979-3F2A-41B2-BE2B-F3062B0CE75B@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <09370B27F829314182C33A219530C104037ABC90@sc-mbx03.TheFacebook.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Hive as a TLP
Date: Thu, 26 Aug 2010 16:08:19 -0700
References: <09370B27F829314182C33A219530C104037ABC90@sc-mbx03.TheFacebook.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


On Aug 26, 2010, at 1:01 PM, Ashish Thusoo wrote:

> The Hive development community voted and passed the following  
> resolution. The details of the vote is at
>
> http://www.bit.ly/aJogyU
>
> The PMC will comprise of the current committers on Hive (as of  
> 8/24/2010) with Namit Jain being the chair.
>
> Please vote on sending this resolution to the Apache Board.

+1

Arun

From general-return-2068-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 23:13:40 2010
Return-Path: <general-return-2068-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 20883 invoked from network); 26 Aug 2010 23:13:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 23:13:39 -0000
Received: (qmail 76450 invoked by uid 500); 26 Aug 2010 23:13:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76389 invoked by uid 500); 26 Aug 2010 23:13:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76376 invoked by uid 99); 26 Aug 2010 23:13:37 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 23:13:37 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 23:13:15 +0000
Received: from SP2-EX07CAS05.ds.corp.yahoo.com (sp2-ex07cas05.corp.sp2.yahoo.com [98.137.59.39])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7QNBht8043958
	for <general@hadoop.apache.org>; Thu, 26 Aug 2010 16:11:43 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:in-reply-to:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:
	content-transfer-encoding:mime-version;
	b=mO/JvCjE3OHWzCcMOC66cYBmJOTquA/RD5g0d/B2S8iTF0JAH0OW44IJC68ojC+Y
Received: from SP2-EX07VS04.ds.corp.yahoo.com ([98.137.59.33]) by
 SP2-EX07CAS05.ds.corp.yahoo.com ([98.137.59.39]) with mapi; Thu, 26 Aug 2010
 18:11:43 -0500
From: Mahadev Konar <mahadev@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 26 Aug 2010 18:11:42 -0500
Subject: Re: [VOTE] Hive as a TLP
Thread-Topic: [VOTE] Hive as a TLP
Thread-Index: ActFc+FQgok0pATjQM+n8DJQ6VC1NAAACoX6
Message-ID: <C89C42BE.3E5F8%mahadev@yahoo-inc.com>
In-Reply-To: <5EB1D979-3F2A-41B2-BE2B-F3062B0CE75B@yahoo-inc.com>
Accept-Language: en, en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en, en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

+1




On 8/26/10 4:08 PM, "Arun C Murthy" <acm@yahoo-inc.com> wrote:

>=20
>=20
> On Aug 26, 2010, at 1:01 PM, Ashish Thusoo wrote:
>=20
>> The Hive development community voted and passed the following
>> resolution. The details of the vote is at
>>=20
>> http://www.bit.ly/aJogyU
>>=20
>> The PMC will comprise of the current committers on Hive (as of
>> 8/24/2010) with Namit Jain being the chair.
>>=20
>> Please vote on sending this resolution to the Apache Board.
>=20
> +1
>=20
> Arun
>=20


From general-return-2069-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 23:24:32 2010
Return-Path: <general-return-2069-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22644 invoked from network); 26 Aug 2010 23:24:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 23:24:32 -0000
Received: (qmail 83796 invoked by uid 500); 26 Aug 2010 23:24:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83734 invoked by uid 500); 26 Aug 2010 23:24:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83726 invoked by uid 99); 26 Aug 2010 23:24:30 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 23:24:30 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 23:24:10 +0000
Received: from wlanvpn-snve-246-72.corp.yahoo.com ([172.21.148.72])
	by mrout1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7QNMcfG029851
	for <general@hadoop.apache.org>; Thu, 26 Aug 2010 16:22:38 -0700 (PDT)
Message-Id: <A83858A0-14B9-4C76-B3FF-ACC6E564BA0B@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <AANLkTimBw3kDY6S1R-MN77LeqLbrr_96btr5PQitL47s@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Date: Thu, 26 Aug 2010 16:22:38 -0700
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <AANLkTimBw3kDY6S1R-MN77LeqLbrr_96btr5PQitL47s@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


On Aug 26, 2010, at 12:08 PM, Stack wrote:

> On Mon, Aug 23, 2010 at 5:27 PM, Arun C Murthy <acm@yahoo-inc.com>  
> wrote:
>> In the interim I'd like to propose we push a hadoop-0.20-security  
>> release
>> off the Yahoo! patchset (http://github.com/yahoo/hadoop-common).  
>> This will
>> ensure the community benefits from all the work done at Yahoo! for  
>> over 12
>> months *now*, and ensures that we do not have to wait until  
>> hadoop-0.22
>> which has all of these patches.
>>
>
> Sounds good to me.  What will this release be called?  hadoop-0.20.3- 
> security?

hadoop-0.20-security. I want to ensure hadoop-0.20 be a separate line,  
so as to not confuse people.


>
>>  Conceivably, one could imagine a Hadoop Security + Append release  
>> soon
>> after.
>
> Well, it'd probably be better if we just did an append release first?
> A good few of us have been banging on the 0.20-append branch w/ a
> while now and its for sure doing append better than 0.20 did (smile).

I think these are orthogonal and both can run their own course.

Arun

From general-return-2070-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 23:29:01 2010
Return-Path: <general-return-2070-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24825 invoked from network); 26 Aug 2010 23:29:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 23:29:00 -0000
Received: (qmail 91248 invoked by uid 500); 26 Aug 2010 23:28:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91161 invoked by uid 500); 26 Aug 2010 23:28:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91153 invoked by uid 99); 26 Aug 2010 23:28:58 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 23:28:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 26 Aug 2010 23:28:41 +0000
Received: (qmail 23939 invoked by uid 99); 26 Aug 2010 23:28:16 -0000
Received: from localhost.apache.org (HELO mail-iw0-f176.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 23:28:16 +0000
Received: by iwn9 with SMTP id 9so2613593iwn.35
        for <general@hadoop.apache.org>; Thu, 26 Aug 2010 16:28:09 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.183.81 with SMTP id cf17mr104856ibb.32.1282865288953; Thu,
 26 Aug 2010 16:28:08 -0700 (PDT)
Received: by 10.231.145.15 with HTTP; Thu, 26 Aug 2010 16:28:08 -0700 (PDT)
In-Reply-To: <AANLkTimBw3kDY6S1R-MN77LeqLbrr_96btr5PQitL47s@mail.gmail.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
	<AANLkTimBw3kDY6S1R-MN77LeqLbrr_96btr5PQitL47s@mail.gmail.com>
Date: Thu, 26 Aug 2010 16:28:08 -0700
Message-ID: <AANLkTimosURHj4it-W97pT8+Yv0cXrS4x7DVqBcM4KWb@mail.gmail.com>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: "Owen O'Malley" <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Thu, Aug 26, 2010 at 12:08 PM, Stack <stack@duboce.net> wrote:
> Sounds good to me. =C2=A0What will this release be called? =C2=A0hadoop-0=
.20.3-security?

It is a new branch, so the question is what is the branch name. I'd
propose calling it 0.20-security and the releases would be
0.20-security.0, etc.

> Well, it'd probably be better if we just did an append release first?

I don't think the ordering maters. 0.20-security is a different branch
that isn't comparable to 0.20-append.

0.20 < 0.20-security < 0.22
0.20 < 0.20-append < 0.21 < 0.22

It does make a bit of a mess.

-- Owen

From general-return-2071-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 23:30:50 2010
Return-Path: <general-return-2071-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25553 invoked from network); 26 Aug 2010 23:30:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 23:30:50 -0000
Received: (qmail 95052 invoked by uid 500); 26 Aug 2010 23:30:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94877 invoked by uid 500); 26 Aug 2010 23:30:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94868 invoked by uid 99); 26 Aug 2010 23:30:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 23:30:48 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com designates 209.85.216.169 as permitted sender)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 23:30:44 +0000
Received: by qyk12 with SMTP id 12so45509qyk.14
        for <general@hadoop.apache.org>; Thu, 26 Aug 2010 16:30:23 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=ThlO/ZJqta+uvJMhmJWRIcNACWBTyPrSYsddUq1sT5A=;
        b=wT4imcuzkhp1IlWTteyI+eQKpubQNzbqfwzzYj9klBmog6WBGMhMKUvVwvzFyKybP+
         gZQgnU7YLgMWVKdByWzJiWm24n9/Yw3D2qzI99J/+bRfsibNqUp+Mgge645AxAXb0F6A
         butpcCsKhkj6i6HhkNLFRm57Lsw47KuYwWnQs=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=JZAWp2F4l/b4HplB5NIBW+2yTI/vQtZdybjbHCBnV/7Tzo83so9v61oGBZMXpwYeyL
         /fjHhkkoa34EZW2nToAWT+BdSNr6AnRyfl/CLNs0JA4OjVr5SYe1xKqq77un24ghf8lR
         NbrY6oFMbLV3FSZFgMW6HopduqqOS38zbASNo=
MIME-Version: 1.0
Received: by 10.224.10.198 with SMTP id q6mr7183882qaq.366.1282865423633; Thu,
 26 Aug 2010 16:30:23 -0700 (PDT)
Received: by 10.229.48.129 with HTTP; Thu, 26 Aug 2010 16:30:23 -0700 (PDT)
In-Reply-To: <A83858A0-14B9-4C76-B3FF-ACC6E564BA0B@yahoo-inc.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
	<AANLkTimBw3kDY6S1R-MN77LeqLbrr_96btr5PQitL47s@mail.gmail.com>
	<A83858A0-14B9-4C76-B3FF-ACC6E564BA0B@yahoo-inc.com>
Date: Thu, 26 Aug 2010 16:30:23 -0700
Message-ID: <AANLkTi=2NjjOfowm4QxBExM8XXJSBhLXWz5Heu7UAb5f@mail.gmail.com>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175ce170dece54048ec261b6

--0015175ce170dece54048ec261b6
Content-Type: text/plain; charset=ISO-8859-1

This would imply hadoop-0.20-security-append or hadoop-0.20-append-security
release be created which contains security and append features.

On Thu, Aug 26, 2010 at 4:22 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:

>
> On Aug 26, 2010, at 12:08 PM, Stack wrote:
>
>  On Mon, Aug 23, 2010 at 5:27 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:
>>
>>> In the interim I'd like to propose we push a hadoop-0.20-security release
>>> off the Yahoo! patchset (http://github.com/yahoo/hadoop-common). This
>>> will
>>> ensure the community benefits from all the work done at Yahoo! for over
>>> 12
>>> months *now*, and ensures that we do not have to wait until hadoop-0.22
>>> which has all of these patches.
>>>
>>>
>> Sounds good to me.  What will this release be called?
>>  hadoop-0.20.3-security?
>>
>
> hadoop-0.20-security. I want to ensure hadoop-0.20 be a separate line, so
> as to not confuse people.
>
>
>
>>   Conceivably, one could imagine a Hadoop Security + Append release soon
>>> after.
>>>
>>
>> Well, it'd probably be better if we just did an append release first?
>> A good few of us have been banging on the 0.20-append branch w/ a
>> while now and its for sure doing append better than 0.20 did (smile).
>>
>
> I think these are orthogonal and both can run their own course.
>
> Arun
>

--0015175ce170dece54048ec261b6--

From general-return-2072-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 23:42:49 2010
Return-Path: <general-return-2072-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 28726 invoked from network); 26 Aug 2010 23:42:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 23:42:48 -0000
Received: (qmail 12296 invoked by uid 500); 26 Aug 2010 23:42:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12225 invoked by uid 500); 26 Aug 2010 23:42:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12217 invoked by uid 99); 26 Aug 2010 23:42:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 23:42:47 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 23:42:40 +0000
Received: from wlanvpn-snve-246-72.corp.yahoo.com ([172.21.148.72])
	by mrout1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7QNfpWr037120
	for <general@hadoop.apache.org>; Thu, 26 Aug 2010 16:41:51 -0700 (PDT)
Message-Id: <EB0EBAF5-AE91-479D-8154-CB5BD072D82B@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <AANLkTi=2NjjOfowm4QxBExM8XXJSBhLXWz5Heu7UAb5f@mail.gmail.com>
Content-Type: multipart/alternative; boundary=Apple-Mail-465--383767883
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Date: Thu, 26 Aug 2010 16:41:49 -0700
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <AANLkTimBw3kDY6S1R-MN77LeqLbrr_96btr5PQitL47s@mail.gmail.com> <A83858A0-14B9-4C76-B3FF-ACC6E564BA0B@yahoo-inc.com> <AANLkTi=2NjjOfowm4QxBExM8XXJSBhLXWz5Heu7UAb5f@mail.gmail.com>
X-Mailer: Apple Mail (2.936)

--Apple-Mail-465--383767883
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


On Aug 26, 2010, at 4:30 PM, Ted Yu wrote:

> This would imply hadoop-0.20-security-append or hadoop-0.20-append- 
> security
> release be created which contains security and append features.

As I mentioned in my initial proposal - it's conceivable, not imminent.
The community might decide that it is a valuable direction and folks  
may work on integrating the two.

At this point, I am signing up to shepherd hadoop-0.20-security. I'd  
like to do it quickly and move on to working on Hadoop trunk, others  
are welcome to take this and run further.

Arun


--Apple-Mail-465--383767883--

From general-return-2073-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 23:47:40 2010
Return-Path: <general-return-2073-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29690 invoked from network); 26 Aug 2010 23:47:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 23:47:39 -0000
Received: (qmail 14955 invoked by uid 500); 26 Aug 2010 23:47:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14869 invoked by uid 500); 26 Aug 2010 23:47:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14861 invoked by uid 99); 26 Aug 2010 23:47:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 23:47:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 26 Aug 2010 23:47:37 +0000
Received: (qmail 29668 invoked by uid 99); 26 Aug 2010 23:47:17 -0000
Received: from localhost.apache.org (HELO mail-iw0-f176.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 23:47:17 +0000
Received: by iwn9 with SMTP id 9so2636924iwn.35
        for <general@hadoop.apache.org>; Thu, 26 Aug 2010 16:47:16 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.157.11 with SMTP id z11mr62647ibw.147.1282866436233; Thu,
 26 Aug 2010 16:47:16 -0700 (PDT)
Received: by 10.231.144.198 with HTTP; Thu, 26 Aug 2010 16:47:16 -0700 (PDT)
In-Reply-To: <09370B27F829314182C33A219530C104037ABC90@sc-mbx03.TheFacebook.com>
References: <09370B27F829314182C33A219530C104037ABC90@sc-mbx03.TheFacebook.com>
Date: Thu, 26 Aug 2010 16:47:16 -0700
Message-ID: <AANLkTikuq+sjE3MhhzvCUDUipkV-Z_SZmFG6tJxK+afg@mail.gmail.com>
Subject: Re: [VOTE] Hive as a TLP
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

+1 -C

On Thu, Aug 26, 2010 at 1:01 PM, Ashish Thusoo <athusoo@facebook.com> wrote=
:
> The Hive development community voted and passed the following resolution.=
 The details of the vote is at
>
> http://www.bit.ly/aJogyU
>
> The PMC will comprise of the current committers on Hive (as of 8/24/2010)=
 with Namit Jain being the chair.
>
> Please vote on sending this resolution to the Apache Board.
>
> Thanks,
> Ashish
>
> Draft Resolution to be sent to the Apache Board
> -----------------------------------------------
>
> Establish the Apache Hive Project
>
> =A0 =A0 =A0 =A0 WHEREAS, the Board of Directors deems it to be in the bes=
t
> =A0 =A0 =A0 =A0 interests of the Foundation and consistent with the
> =A0 =A0 =A0 =A0 Foundation's purpose to establish a Project Management
> =A0 =A0 =A0 =A0 Committee charged with the creation and maintenance of
> =A0 =A0 =A0 =A0 open-source software related to parallel analysis of larg=
e
> =A0 =A0 =A0 =A0 data sets for distribution at no charge to the public.
>
> =A0 =A0 =A0 =A0 NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> =A0 =A0 =A0 =A0 Committee (PMC), to be known as the "Apache Hive Project"=
,
> =A0 =A0 =A0 =A0 be and hereby is established pursuant to Bylaws of the
> =A0 =A0 =A0 =A0 Foundation; and be it further
>
> =A0 =A0 =A0 =A0 RESOLVED, that the Apache Hive Project be and hereby is
> =A0 =A0 =A0 =A0 responsible for the creation and maintenance of software
> =A0 =A0 =A0 =A0 related to parallel analysis of large data sets; and be
> =A0 =A0 =A0 =A0 it further
>
> =A0 =A0 =A0 =A0 RESOLVED, that the office of "Vice President, Apache Hive=
" be
> =A0 =A0 =A0 =A0 and hereby is created, the person holding such office to
> =A0 =A0 =A0 =A0 serve at the direction of the Board of Directors as the c=
hair
> =A0 =A0 =A0 =A0 of the Apache Hive Project, and to have primary responsib=
ility
> =A0 =A0 =A0 =A0 for management of the projects within the scope of
> =A0 =A0 =A0 =A0 responsibility of the Apache Hive Project; and be it furt=
her
>
> =A0 =A0 =A0 =A0 RESOLVED, that the persons listed immediately below be an=
d
> =A0 =A0 =A0 =A0 hereby are appointed to serve as the initial members of t=
he
> =A0 =A0 =A0 =A0 Apache Hive Project:
> =A0 =A0 =A0 =A0 =A0 =A0 * Namit Jain (namit@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * John Sichi (jvs@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * Zheng Shao (zshao@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * Edward Capriolo (appodictic@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * Raghotham Murthy (rsm@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * Ning Zhang (nzhang@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * Paul Yang (pauly@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * He Yongqiang (he yongqiang@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * Prasad Chakka (prasadc@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * Joydeep Sen Sarma (jsensarma@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * Ashish Thusoo (athusoo@apache.org)
>
> =A0 =A0 =A0 =A0 NOW, THEREFORE, BE IT FURTHER RESOLVED, that Namit Jain
> =A0 =A0 =A0 =A0 be appointed to the office of Vice President, Apache Hive=
, to
> =A0 =A0 =A0 =A0 serve in accordance with and subject to the direction of =
the
> =A0 =A0 =A0 =A0 Board of Directors and the Bylaws of the Foundation until
> =A0 =A0 =A0 =A0 death, resignation, retirement, removal or disqualificati=
on,
> =A0 =A0 =A0 =A0 or until a successor is appointed; and be it further
>
> =A0 =A0 =A0 =A0 RESOLVED, that the initial Apache Hive PMC be and hereby =
is
> =A0 =A0 =A0 =A0 tasked with the creation of a set of bylaws intended to
> =A0 =A0 =A0 =A0 encourage open development and increased participation in=
 the
> =A0 =A0 =A0 =A0 Apache Hive Project; and be it further
>
> =A0 =A0 =A0 =A0 RESOLVED, that the Apache Hive Project be and hereby
> =A0 =A0 =A0 =A0 is tasked with the migration and rationalization of the A=
pache
> =A0 =A0 =A0 =A0 Hadoop Hive sub-project; and be it further
>
> =A0 =A0 =A0 =A0 RESOLVED, that all responsibilities pertaining to the Apa=
che
> =A0 =A0 =A0 =A0 Hive sub-project encumbered upon the
> =A0 =A0 =A0 =A0 Apache Hadoop Project are hereafter discharged.
>

From general-return-2074-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 04:36:38 2010
Return-Path: <general-return-2074-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 99046 invoked from network); 27 Aug 2010 04:36:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 04:36:37 -0000
Received: (qmail 91186 invoked by uid 500); 27 Aug 2010 04:36:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90732 invoked by uid 500); 27 Aug 2010 04:36:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90721 invoked by uid 99); 27 Aug 2010 04:36:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 04:36:32 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 04:36:25 +0000
Received: by wyf23 with SMTP id 23so3867048wyf.35
        for <general@hadoop.apache.org>; Thu, 26 Aug 2010 21:36:04 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        bh=yubnqvOxH5wghrCpYkYGbKsY6Wx8HEC3NSs/FNvj738=;
        b=EB33TQQMJk3OmD8LvVxCcLashTf7H5zW9WTJ1uSxsGVM1X2xzK3j4FLT8y13qVzYT8
         iThVljLOiCKF5pN9N5N+tap5qhD8o5CsobIb1Py02dwuDLqriYMcjAA54gc50LlA9KVJ
         541ZJLQU2Aa8DnvCZrp7fhDzK17JDEe5eZy6A=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=s0kxgdGVKoz/KsR0jnvzQXo/+kby4MpDE7MzcTnGKr4fEFAitN4uUFMlgt736znTif
         nfwKXCN8W4XIXDqO0rrTWNeKgHsNL4hFo2uEmJzVQe6s9p3N1Ox6Pp/bDNSuq/C9ff1A
         K0rspA8uflNaeccszPNCW3CYZ4AFl4FPVq8M8=
MIME-Version: 1.0
Received: by 10.216.236.149 with SMTP id w21mr362510weq.65.1282883763984; Thu,
 26 Aug 2010 21:36:03 -0700 (PDT)
Sender: saint.ack@gmail.com
Received: by 10.216.131.24 with HTTP; Thu, 26 Aug 2010 21:36:03 -0700 (PDT)
In-Reply-To: <09370B27F829314182C33A219530C104037ABC90@sc-mbx03.TheFacebook.com>
References: <09370B27F829314182C33A219530C104037ABC90@sc-mbx03.TheFacebook.com>
Date: Thu, 26 Aug 2010 21:36:03 -0700
X-Google-Sender-Auth: kMgnyV0mWk1EHF7ITuSXqcASimo
Message-ID: <AANLkTimniE4Ors5Dc-j4DNdG6eaAu_NdS4B63zFq2nu6@mail.gmail.com>
Subject: Re: [VOTE] Hive as a TLP
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

+1

From general-return-2075-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 04:57:46 2010
Return-Path: <general-return-2075-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1860 invoked from network); 27 Aug 2010 04:57:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 04:57:46 -0000
Received: (qmail 98348 invoked by uid 500); 27 Aug 2010 04:57:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98110 invoked by uid 500); 27 Aug 2010 04:57:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98101 invoked by uid 99); 27 Aug 2010 04:57:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 04:57:40 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [98.136.44.35] (HELO n62.bullet.mail.sp1.yahoo.com) (98.136.44.35)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 27 Aug 2010 04:57:31 +0000
Received: from [69.147.84.145] by n62.bullet.mail.sp1.yahoo.com with NNFMP; 27 Aug 2010 04:57:10 -0000
Received: from [67.195.9.83] by t8.bullet.mail.sp1.yahoo.com with NNFMP; 27 Aug 2010 04:57:09 -0000
Received: from [98.137.27.213] by t3.bullet.mail.gq1.yahoo.com with NNFMP; 27 Aug 2010 04:57:09 -0000
Received: from [127.0.0.1] by omp123.mail.gq1.yahoo.com with NNFMP; 27 Aug 2010 04:57:09 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 775245.97277.bm@omp123.mail.gq1.yahoo.com
Received: (qmail 4617 invoked by uid 60001); 27 Aug 2010 04:50:28 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1282884628; bh=1yI+hc4XTnhJECggri8ntKXJVf6XIC+ITm0bxdV8epI=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=CjWv1B7dapG78ir8tLprDeE1XVWXiR3PiXzy96yTd2elA2506BaNkXLiq1MReg0sM3ne9icV2bMfm7lpD72DBN/VM7BOxf/GfRimz5ehE1qS1Q7uHDwFWdeRR1x/ouiRrhICLWZ/oN7aQc4ENnCJiYdQFBBT4NOb1tDtIs264js=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=vRuxvJanBq/nfM3ky486G8b0ny8TGKLwegNm3uyElsU7dSc1VkHT991GU4tL4EChNxTHWlUe3K5DKHv7nTMbTqkWpXmHlqn4DBAAqj3O+ZsT5sMiUwC6M9WyFAcqWLgYNMxAEo2FPFkN4fJM9/jcjTJHojshMlVYZKzlyKzJQQs=;
Message-ID: <350379.4590.qm@web114518.mail.gq1.yahoo.com>
X-YMail-OSG: btZMeHsVM1nIev0go_f7wKZUb.XBcrjIrPjREguYEbQVxgb
 G_j6XsWb_
Received: from [67.180.86.237] by web114518.mail.gq1.yahoo.com via HTTP; Thu, 26 Aug 2010 21:50:27 PDT
X-Mailer: YahooMailRC/470 YahooMailWebService/0.8.105.279950
References: <715590.78832.qm@web114505.mail.gq1.yahoo.com> <AANLkTinQ_xeerXn9V7+M4=M0_NHEOgqn=ZsUu=xvMGox@mail.gmail.com> <194169.50990.qm@web114512.mail.gq1.yahoo.com> <AANLkTim9VK5Ee1ZE8dcbbJXcmOrjTO2vu+8pL5bEReA8@mail.gmail.com> <698249.46749.qm@web114510.mail.gq1.yahoo.com> <AANLkTimUdW9eObq8MMF2zPMt=ggNhN3oERmwXEpcc-GE@mail.gmail.com>
Date: Thu, 26 Aug 2010 21:50:27 -0700 (PDT)
From: C J <c.joshi26@yahoo.com>
Subject: Re: Child processes on datanodes/task trackers
To: general@hadoop.apache.org
In-Reply-To: <AANLkTimUdW9eObq8MMF2zPMt=ggNhN3oERmwXEpcc-GE@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1426679418-1282884627=:4590"

--0-1426679418-1282884627=:4590
Content-Type: text/plain; charset=us-ascii

Thanks Ted. Yes, I forgot to shutdown the CacheManager.

Appreciate your help.




________________________________
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Sent: Wed, August 25, 2010 8:39:01 PM
Subject: Re: Child processes on datanodes/task trackers

I don't use ehcache.
Did you forget to close CacheManager at the end of your job by any chance ?

On Wed, Aug 25, 2010 at 7:59 PM, C J <c.joshi26@yahoo.com> wrote:

> Thanks Ted!
>
> I did a jstack and it seems there is an issue with ehcache that I am using
> in
> the mapper task.
>
>
> "net.sf.ehcache.CacheManager@57ac3379" daemon prio=10
> tid=0x0000000059180800
> nid=0x379e in Object.wait() [0x0000000041506000]
>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>        at java.lang.Object.wait(Native Method)
>        - waiting on <0x00002aaabb0b89a8> (a java.util.TaskQueue)
>        at java.util.TimerThread.mainLoop(Timer.java:509)
>        - locked <0x00002aaabb0b89a8> (a java.util.TaskQueue)
>        at java.util.TimerThread.run(Timer.java:462)
>
>   Locked ownable synchronizers:
>        - None
> .
> .
> .
>
>
> The hadoop version I am using is 0.20.2.
>
> Thanks.
>
>
>
> ________________________________
> From: Ted Yu <yuzhihong@gmail.com>
> To: general@hadoop.apache.org
> Sent: Wed, August 25, 2010 7:34:35 PM
> Subject: Re: Child processes on datanodes/task trackers
>
> After you obtain pid, you can use jstack to see what the Child process was
> doing.
>
> What hadoop version are you using ?
>
> On Wed, Aug 25, 2010 at 7:28 PM, C J <c.joshi26@yahoo.com> wrote:
>
> > Thanks for your reply.
> >
> > Some of these child tasks belong to successful jobs. I am wondering why
> > they are
> > still hanging there for long finished jobs.
> >
> >
> >
> >
> >
> > ________________________________
> > From: Ted Yu <yuzhihong@gmail.com>
> > To: general@hadoop.apache.org
> > Sent: Wed, August 25, 2010 4:17:38 PM
> > Subject: Re: Child processes on datanodes/task trackers
> >
> > Use jps to find out pid of the Child.
> > Then use this to find out which job the Child belongs to:
> > ps aux | grep <pid>
> >
> > On Wed, Aug 25, 2010 at 12:20 PM, C J <c.joshi26@yahoo.com> wrote:
> >
> > > Hi,
> > >
> > > I wanted to know why I see running Child processes on my datanodes even
> > > though
> > > there is no job running at that time. Are these left over from failed
> > > attempts?
> > >
> > > Is there anything I can do to keep these clean?
> > >
> > > Thanks,
> > > Deepika
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>



      
--0-1426679418-1282884627=:4590--


From general-return-2076-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 05:20:02 2010
Return-Path: <general-return-2076-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 8896 invoked from network); 27 Aug 2010 05:20:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 05:20:01 -0000
Received: (qmail 12721 invoked by uid 500); 27 Aug 2010 05:20:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12414 invoked by uid 500); 27 Aug 2010 05:19:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12401 invoked by uid 99); 27 Aug 2010 05:19:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 05:19:56 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of timrobertson100@gmail.com designates 209.85.215.176 as permitted sender)
Received: from [209.85.215.176] (HELO mail-ey0-f176.google.com) (209.85.215.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 05:19:49 +0000
Received: by eydd26 with SMTP id d26so2396730eyd.35
        for <general@hadoop.apache.org>; Thu, 26 Aug 2010 22:19:28 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=yubnqvOxH5wghrCpYkYGbKsY6Wx8HEC3NSs/FNvj738=;
        b=QcoFSqzQg6z/IInaGr0yOKrk4k+dW6/diNdhd4yX4LvsgmFfW8lpBOs36W4g9Eg3iv
         ivOPwXsOysTIO6jMOj/Lt5GnUf76aA8rh/iLRHX6QukyU5vD0t0etV9zwXKNtbPkxxop
         kFjRqaK22U0Beo1Nc6IHXCva937mg7zyK8zCM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=hmfh+jC4nHk9/CqQWAp4Tvd74IzfwqI4hjztG+MoWSi5lIfA0AnGWxBgGnpnRf+ACU
         cUU9WR4Ub69UMxlCMoVtIaItvwfZOtD2PmeAO4aKO/mLwtfCaBxokFkuJLmYSk42TYF2
         l91kHgivuNOUg0b/KVZSSh1UxcvlO4Nda7PKw=
MIME-Version: 1.0
Received: by 10.213.109.9 with SMTP id h9mr462234ebp.33.1282886368497; Thu, 26
 Aug 2010 22:19:28 -0700 (PDT)
Received: by 10.14.18.68 with HTTP; Thu, 26 Aug 2010 22:19:28 -0700 (PDT)
In-Reply-To: <AANLkTimniE4Ors5Dc-j4DNdG6eaAu_NdS4B63zFq2nu6@mail.gmail.com>
References: <09370B27F829314182C33A219530C104037ABC90@sc-mbx03.TheFacebook.com>
	<AANLkTimniE4Ors5Dc-j4DNdG6eaAu_NdS4B63zFq2nu6@mail.gmail.com>
Date: Fri, 27 Aug 2010 07:19:28 +0200
Message-ID: <AANLkTim3eVq19Q+rwNC88kFf7o7JDSTdWfBhvPKLPTDf@mail.gmail.com>
Subject: Re: [VOTE] Hive as a TLP
From: Tim Robertson <timrobertson100@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

+1

From general-return-2077-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 06:47:57 2010
Return-Path: <general-return-2077-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33534 invoked from network); 27 Aug 2010 06:47:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 06:47:56 -0000
Received: (qmail 71614 invoked by uid 500); 27 Aug 2010 06:47:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71123 invoked by uid 500); 27 Aug 2010 06:47:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71107 invoked by uid 99); 27 Aug 2010 06:47:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 06:47:51 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=FREEMAIL_FROM,FROM_LOCAL_NOVOWEL,HK_RANDOM_ENVFROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of sssssssenator@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 06:47:47 +0000
Received: by wwd20 with SMTP id 20so1148845wwd.29
        for <multiple recipients>; Thu, 26 Aug 2010 23:47:25 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=ddKGJqDpLsb4Igbiv3u/FDIdt1iQD4WheSD2pSXi9rc=;
        b=EqygR/2kEN+5r9bhYDsWH+lOHVo+y01YfQXvBORhMX3IlLuQKfptotj9havtpIPsGx
         gbwXwgdc+nQNIR9UbuOsiT1B2b7MFLN8djzfldGJC0PdFIQ4blBeJ9CLYhwVnpfS3peZ
         yVlp5YoeDlKqxP4WH6EBeY3pybBFOpTyUj6VM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=ITEsaK8d/sWhr2iMoYynlfG/SZoXtJwJZT6XWYuUF//g7ja0qj9ctiGelKG7ez03fi
         Xwur0fBJlMy2EkHu9DmC5kNOc9W9m/MY6ZOi+61fFW6P1ywyH3FqPRLnHwwWkkkDxtU8
         4lR9nbv5efC8LrYptDtUxEHqfcb9AGaJUfrhY=
MIME-Version: 1.0
Received: by 10.216.1.18 with SMTP id 18mr353588wec.24.1282891641880; Thu, 26
 Aug 2010 23:47:21 -0700 (PDT)
Received: by 10.216.154.132 with HTTP; Thu, 26 Aug 2010 23:47:21 -0700 (PDT)
In-Reply-To: <C6E8EB1B59E1E942860E0410E28414FE39B4004E@EXCHMB.ornl.gov>
References: <AANLkTimmc4qQ34h1xS4Hzc2TorNDWZeDoMHfJ9kWaPVa@mail.gmail.com>
	<C6E8EB1B59E1E942860E0410E28414FE39B4004E@EXCHMB.ornl.gov>
Date: Fri, 27 Aug 2010 12:17:21 +0530
Message-ID: <AANLkTimJfLA9ZzDm_6i1GgnY73V4T9g-cprkX_bE-eiu@mail.gmail.com>
Subject: Re: WARNING : There are about 1 missing blocks. Please check the log
 or run fsck.
From: vaibhav negi <sssssssenator@gmail.com>
To: general@hadoop.apache.org, common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364d31739981e9048ec87cfb

--0016364d31739981e9048ec87cfb
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Hi ,

thanks for reply. Msg says that "The filesystem under path '/' is CORRUPT"

Whole msg is :-

[hadoop@vmlinux3 bin]$ hadoop fsck /
.
/home/hadoop/mapred/system/jobtracker.info: CORRUPT block
blk_-4797974583768874015

/home/hadoop/mapred/system/jobtracker.info: MISSING 1 blocks of total size =
4
B.Status: CORRUPT
 Total size:    4 B
 Total dirs:    8
 Total files:   1
 Total blocks (validated):      1 (avg. block size 4 B)
  ********************************
  CORRUPT FILES:        1
  MISSING BLOCKS:       1
  MISSING SIZE:         4 B
  CORRUPT BLOCKS:       1
  ********************************
 Minimally replicated blocks:   0 (0.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    1
 Average block replication:     0.0
 Corrupt blocks:                1
 Missing replicas:              0
 Number of data-nodes:          1
 Number of racks:               1


The filesystem under path '/' is CORRUPT
[hadoop@vmlinux3 bin]$


Vaibhav Negi


On Fri, Aug 27, 2010 at 12:25 AM, Galloway, Michael D.
<gallowaymd@ornl.gov>wrote:

> As said earlier try:
>
> hadoop fsck /
>
>
>
> ----- Original Message -----
> From: Vaibhav Negi [mailto:vaibhav.negi@one97.net]
> Sent: Thursday, August 26, 2010 07:22 AM
> To: general@hadoop.apache.org <general@hadoop.apache.org>
> Subject: WARNING : There are about 1 missing blocks. Please check the log
> or run fsck.
>
> Hi ,
>
> I am using hadoop version : 0.20.2 . I am getting this error message.
>
> WARNING : There are about 1 missing blocks. Please check the log or run
> fsck.
>
> What to do? Is there any problem in the cluster
>
>
>
> Regards
>
> Vaibhav Negi
> Software Engineer=96 Engineering.
> One97 Communications Ltd
> P: +91 11 46500197      Extn:461
> M: + 91 9582259332
>
> Let=92s Get Talking!
>

--0016364d31739981e9048ec87cfb--

From general-return-2078-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 13:56:42 2010
Return-Path: <general-return-2078-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55776 invoked from network); 27 Aug 2010 13:56:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 13:56:41 -0000
Received: (qmail 82528 invoked by uid 500); 27 Aug 2010 13:56:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82094 invoked by uid 500); 27 Aug 2010 13:56:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82086 invoked by uid 99); 27 Aug 2010 13:56:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 13:56:35 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of msegel@navteq.com designates 204.120.70.37 as permitted sender)
Received: from [204.120.70.37] (HELO xmailfargo.navteq.com) (204.120.70.37)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 13:56:29 +0000
Received: from fargo-spamfw.navteq.com (navteq-spamfw.navteq.com [10.6.10.131])
	by xmailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id o7RDXOAq028971
	for <general@hadoop.apache.org>; Fri, 27 Aug 2010 08:33:24 -0500
Received: from imailfargo.navteq.com (localhost [127.0.0.1])
	by fargo-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id 6054031CB6A6
	for <general@hadoop.apache.org>; Fri, 27 Aug 2010 08:56:08 -0500 (CDT)
Received: from imailfargo.navteq.com (imailfargo.navteq.com [10.6.10.130]) by fargo-spamfw.navteq.com with ESMTP id S32JVvUbVRHZFGOc for <general@hadoop.apache.org>; Fri, 27 Aug 2010 08:56:08 -0500 (CDT)
Received: from hq-ex-ht01.ad.navteq.com ([10.8.222.51])
	by imailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id o7RDu7nB021643
	for <general@hadoop.apache.org>; Fri, 27 Aug 2010 08:56:08 -0500
Received: from hq-ex-mb02.ad.navteq.com ([fe80::880d:318e:e860:a8db]) by
 hq-ex-ht01.ad.navteq.com ([fe80::55a6:5b94:5c00:fe39%12]) with mapi; Fri, 27
 Aug 2010 08:56:07 -0500
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 27 Aug 2010 08:56:07 -0500
Subject: RE: [VOTE] Hive as a TLP
Thread-Topic: [VOTE] Hive as a TLP
Thread-Index: ActFoXCxwz6pYLsJQfGVMca97f+FnwATiLfw
Message-ID: <ABC24175AFD3BE4DA15F4CD375ED413D0614EF082B@hq-ex-mb02.ad.navteq.com>
References: <09370B27F829314182C33A219530C104037ABC90@sc-mbx03.TheFacebook.com>
 <AANLkTimniE4Ors5Dc-j4DNdG6eaAu_NdS4B63zFq2nu6@mail.gmail.com>
In-Reply-To: <AANLkTimniE4Ors5Dc-j4DNdG6eaAu_NdS4B63zFq2nu6@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1


+1


The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.

From general-return-2079-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 17:25:04 2010
Return-Path: <general-return-2079-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 15337 invoked from network); 27 Aug 2010 17:25:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 17:25:04 -0000
Received: (qmail 82168 invoked by uid 500); 27 Aug 2010 17:25:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81938 invoked by uid 500); 27 Aug 2010 17:25:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81929 invoked by uid 99); 27 Aug 2010 17:25:02 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 17:25:02 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 17:24:40 +0000
Received: by bwz14 with SMTP id 14so2933719bwz.35
        for <general@hadoop.apache.org>; Fri, 27 Aug 2010 10:24:20 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.131.132 with SMTP id x4mr748037bks.50.1282929860210; Fri,
 27 Aug 2010 10:24:20 -0700 (PDT)
Received: by 10.204.179.74 with HTTP; Fri, 27 Aug 2010 10:24:20 -0700 (PDT)
In-Reply-To: <09370B27F829314182C33A219530C104037ABC90@sc-mbx03.TheFacebook.com>
References: <09370B27F829314182C33A219530C104037ABC90@sc-mbx03.TheFacebook.com>
Date: Fri, 27 Aug 2010 10:24:20 -0700
Message-ID: <AANLkTi=BgaX8tU7j3+LDAkJrRfsTKQ-MzmKV0r7=5EyZ@mail.gmail.com>
Subject: Re: [VOTE] Hive as a TLP
From: Tom White <tom@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Tom

On Thu, Aug 26, 2010 at 1:01 PM, Ashish Thusoo <athusoo@facebook.com> wrote=
:
> The Hive development community voted and passed the following resolution.=
 The details of the vote is at
>
> http://www.bit.ly/aJogyU
>
> The PMC will comprise of the current committers on Hive (as of 8/24/2010)=
 with Namit Jain being the chair.
>
> Please vote on sending this resolution to the Apache Board.
>
> Thanks,
> Ashish
>
> Draft Resolution to be sent to the Apache Board
> -----------------------------------------------
>
> Establish the Apache Hive Project
>
> =A0 =A0 =A0 =A0 WHEREAS, the Board of Directors deems it to be in the bes=
t
> =A0 =A0 =A0 =A0 interests of the Foundation and consistent with the
> =A0 =A0 =A0 =A0 Foundation's purpose to establish a Project Management
> =A0 =A0 =A0 =A0 Committee charged with the creation and maintenance of
> =A0 =A0 =A0 =A0 open-source software related to parallel analysis of larg=
e
> =A0 =A0 =A0 =A0 data sets for distribution at no charge to the public.
>
> =A0 =A0 =A0 =A0 NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> =A0 =A0 =A0 =A0 Committee (PMC), to be known as the "Apache Hive Project"=
,
> =A0 =A0 =A0 =A0 be and hereby is established pursuant to Bylaws of the
> =A0 =A0 =A0 =A0 Foundation; and be it further
>
> =A0 =A0 =A0 =A0 RESOLVED, that the Apache Hive Project be and hereby is
> =A0 =A0 =A0 =A0 responsible for the creation and maintenance of software
> =A0 =A0 =A0 =A0 related to parallel analysis of large data sets; and be
> =A0 =A0 =A0 =A0 it further
>
> =A0 =A0 =A0 =A0 RESOLVED, that the office of "Vice President, Apache Hive=
" be
> =A0 =A0 =A0 =A0 and hereby is created, the person holding such office to
> =A0 =A0 =A0 =A0 serve at the direction of the Board of Directors as the c=
hair
> =A0 =A0 =A0 =A0 of the Apache Hive Project, and to have primary responsib=
ility
> =A0 =A0 =A0 =A0 for management of the projects within the scope of
> =A0 =A0 =A0 =A0 responsibility of the Apache Hive Project; and be it furt=
her
>
> =A0 =A0 =A0 =A0 RESOLVED, that the persons listed immediately below be an=
d
> =A0 =A0 =A0 =A0 hereby are appointed to serve as the initial members of t=
he
> =A0 =A0 =A0 =A0 Apache Hive Project:
> =A0 =A0 =A0 =A0 =A0 =A0 * Namit Jain (namit@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * John Sichi (jvs@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * Zheng Shao (zshao@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * Edward Capriolo (appodictic@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * Raghotham Murthy (rsm@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * Ning Zhang (nzhang@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * Paul Yang (pauly@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * He Yongqiang (he yongqiang@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * Prasad Chakka (prasadc@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * Joydeep Sen Sarma (jsensarma@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * Ashish Thusoo (athusoo@apache.org)
>
> =A0 =A0 =A0 =A0 NOW, THEREFORE, BE IT FURTHER RESOLVED, that Namit Jain
> =A0 =A0 =A0 =A0 be appointed to the office of Vice President, Apache Hive=
, to
> =A0 =A0 =A0 =A0 serve in accordance with and subject to the direction of =
the
> =A0 =A0 =A0 =A0 Board of Directors and the Bylaws of the Foundation until
> =A0 =A0 =A0 =A0 death, resignation, retirement, removal or disqualificati=
on,
> =A0 =A0 =A0 =A0 or until a successor is appointed; and be it further
>
> =A0 =A0 =A0 =A0 RESOLVED, that the initial Apache Hive PMC be and hereby =
is
> =A0 =A0 =A0 =A0 tasked with the creation of a set of bylaws intended to
> =A0 =A0 =A0 =A0 encourage open development and increased participation in=
 the
> =A0 =A0 =A0 =A0 Apache Hive Project; and be it further
>
> =A0 =A0 =A0 =A0 RESOLVED, that the Apache Hive Project be and hereby
> =A0 =A0 =A0 =A0 is tasked with the migration and rationalization of the A=
pache
> =A0 =A0 =A0 =A0 Hadoop Hive sub-project; and be it further
>
> =A0 =A0 =A0 =A0 RESOLVED, that all responsibilities pertaining to the Apa=
che
> =A0 =A0 =A0 =A0 Hive sub-project encumbered upon the
> =A0 =A0 =A0 =A0 Apache Hadoop Project are hereafter discharged.
>

From general-return-2080-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 19:32:26 2010
Return-Path: <general-return-2080-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54102 invoked from network); 27 Aug 2010 19:32:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 19:32:26 -0000
Received: (qmail 45262 invoked by uid 500); 27 Aug 2010 19:32:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45143 invoked by uid 500); 27 Aug 2010 19:32:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45135 invoked by uid 99); 27 Aug 2010 19:32:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 19:32:24 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jdixon@omniti.com designates 8.8.38.6 as permitted sender)
Received: from [8.8.38.6] (HELO edge.omniti.com) (8.8.38.6)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 19:32:18 +0000
DKIM-Signature: v=1; a=rsa-sha256; d=omniti.com; s=s1024; c=relaxed/relaxed;
	q=dns/txt; i=@omniti.com; t=1282937517;
	h=From:Subject:Date:To;
	bh=RMTKPw5KG9HiDTSd83mha4tbgbmrkkMMt+p2rtWdTiw=;
	b=GSPGPfILUnJxRpaU0OwTIzRVaznh3uLnaioM/tn10y/cSjuLyCMoG3sAi2lYWjH0
	M73dFqCp8GOwxrTYdOku0u/U8AMGHvKz67PsluBfUziA7GX6EIDGFbBApTG60xZR
	IkKSrKiU/iFsArb2tn2t87G8OtGdCjMrvY1P14CAz0I=;
Authentication-Results:  edge smtp.user=jdixon@omniti.com; auth=pass (LOGIN)
Received: from [68.55.0.29] ([68.55.0.29:61371] helo=omniti.com)
	by edge (envelope-from <jdixon@omniti.com>)
	(ecelerity 2.2.3.46 r(37468M)) with ESMTPSA (cipher=AES256-SHA) 
	id 6B/37-03409-DA2187C4; Fri, 27 Aug 2010 15:31:57 -0400
Date: Fri, 27 Aug 2010 15:31:53 -0400
From: Jason Dixon <jdixon@omniti.com>
To: general@hadoop.apache.org
Subject: Surge 2010 Early Registration ends Tuesday!
Message-ID: <20100827193153.GW1736@omniti.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.18 (2008-05-17)

Early Bird Registration for Surge Scalability Conference 2010 ends next
Tuesday, August 31.  We have a killer lineup of speakers and architects
from across the Internet.  Listen to experts talk about the newest
methods and technologies for scaling your Web presence.

http://omniti.com/surge/2010/register

This year's event is all about the challenges faced (and overcome) in
real-life production architectures.  Meet the engineering talent from
some of the best and brightest throughout the Internet:

John Allspaw, Etsy
Theo Schlossnagle, OmniTI
Bryan Cantrill, Joyent
Rasmus Lerdorf, creator of PHP
Tom Cook, Facebook
Benjamin Black, fast_ip
Christopher Brown, Opscode
Artur Bergman, Wikia
Baron Schwartz, Percona
Paul Querna, Cloudkick

Surge 2010 takes place at the Tremont Grand Historic Venue on Sept 30
and Oct 1, 2010 in Baltimore, MD.  Register NOW for the Early Bird
discount and guarantee your seat to this year's event!


-- 
Jason Dixon
OmniTI Computer Consulting, Inc.
jdixon@omniti.com
443.325.1357 x.241

From general-return-2081-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 29 00:38:20 2010
Return-Path: <general-return-2081-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66033 invoked from network); 29 Aug 2010 00:38:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Aug 2010 00:38:20 -0000
Received: (qmail 91651 invoked by uid 500); 29 Aug 2010 00:38:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91571 invoked by uid 500); 29 Aug 2010 00:38:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91563 invoked by uid 99); 29 Aug 2010 00:38:18 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 Aug 2010 00:38:18 +0000
X-ASF-Spam-Status: No, hits=-0.1 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of Deepika.Khera@avg.com designates 193.85.188.248 as permitted sender)
Received: from [193.85.188.248] (HELO ms.grisoft.cz) (193.85.188.248)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 Aug 2010 00:37:55 +0000
Received: from deimos.cz.avg.com (unknown [192.168.200.161])
	(using TLSv1 with cipher AES128-SHA (128/128 bits))
	(No client certificate requested)
	by ms.grisoft.cz (Postfix) with ESMTP id 889EC382CA
	for <general@hadoop.apache.org>; Sun, 29 Aug 2010 02:37:34 +0200 (CEST)
Received: from miranda.us.avg.com (10.32.34.13) by deimos.cz.avg.com
 (192.168.200.161) with Microsoft SMTP Server (TLS) id 8.3.83.0; Sun, 29 Aug
 2010 02:37:34 +0200
Received: from miranda.us.avg.com ([10.32.34.13]) by miranda ([10.32.34.13])
 with mapi; Sat, 28 Aug 2010 20:37:32 -0400
From: Deepika Khera <Deepika.Khera@avg.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Sat, 28 Aug 2010 20:37:36 -0400
Subject: Deploying my job jar on hadoop cluster
Thread-Topic: Deploying my job jar on hadoop cluster
Thread-Index: ActHEmC8HENHqb99T9qjQYtdk2eNqg==
Message-ID: <DA852C02397DBC4EBFAA1B7192FBD15062025E5AC2@miranda>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_DA852C02397DBC4EBFAA1B7192FBD15062025E5AC2miranda_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_DA852C02397DBC4EBFAA1B7192FBD15062025E5AC2miranda_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi,

I want to deploy my map reduce job jar on the Hadoop cluster. I've always d=
one that by doing the following -

1. Copying the job jar to all datanodes
2. Having the job jar on the hadoop classpath on all machines.

Isn't hadoop capable of copying over the job jar to all machines in the clu=
ster? This is what I read (that job tracker copies the job jar, etc) , but =
if I don't do the above the task trackers cannot find the job. I know I am =
missing something.

Could someone please let me know how I can run my job without having to cop=
y it over all machines in the cluster?

Thanks,
Deepika

--_000_DA852C02397DBC4EBFAA1B7192FBD15062025E5AC2miranda_--

From general-return-2082-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 29 05:40:43 2010
Return-Path: <general-return-2082-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38789 invoked from network); 29 Aug 2010 05:40:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Aug 2010 05:40:43 -0000
Received: (qmail 72477 invoked by uid 500); 29 Aug 2010 05:40:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71991 invoked by uid 500); 29 Aug 2010 05:40:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71982 invoked by uid 99); 29 Aug 2010 05:40:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 Aug 2010 05:40:37 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cpbhagtani@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 Aug 2010 05:40:31 +0000
Received: by wwd20 with SMTP id 20so948391wwd.29
        for <general@hadoop.apache.org>; Sat, 28 Aug 2010 22:40:11 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=iUUWNngbFZekY1WfiUr87lPjqKK5C1bk3fRuyD9HsiU=;
        b=dqhiIOfgbaBQHv9Lb9mSND02cOsBu6NNsn/DNhQONPUuuI0dAP1R2RlBkJ1S5PMhtG
         cjE40dnky7gOL8Gl/5POU344je9OHh/5goTsKM5/p8BDhmsd1ypDzJzW+1f4C/FfVfBT
         GSvK9+L6PW3ZkIiWjZzEuN3usLW6LA3miIp1w=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=ic7+zyoI78Pb6Pk7M9qdn4C5JbABgFFT2p5PBJzdL+wB1J+PTbTpcq/tRpTJWjMnQk
         7a8DBbM+t/a7Wlv7DXuF1ww1Cnxzu+1ZsEPMysYb9ZoLfP8hD83BtCr4FbEqiCknhE9M
         Oqv9KWATygqHa3u/e2R+8F8wTPibyxazkIQok=
MIME-Version: 1.0
Received: by 10.227.157.70 with SMTP id a6mr3014507wbx.163.1283060411453; Sat,
 28 Aug 2010 22:40:11 -0700 (PDT)
Received: by 10.216.0.66 with HTTP; Sat, 28 Aug 2010 22:40:11 -0700 (PDT)
In-Reply-To: <DA852C02397DBC4EBFAA1B7192FBD15062025E5AC2@miranda>
References: <DA852C02397DBC4EBFAA1B7192FBD15062025E5AC2@miranda>
Date: Sun, 29 Aug 2010 11:10:11 +0530
Message-ID: <AANLkTi=-dcxp8VVLC8ESpOtLb-X9DNdXr0oPw-KQ9WeJ@mail.gmail.com>
Subject: Re: Deploying my job jar on hadoop cluster
From: Chandraprakash Bhagtani <cpbhagtani@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636416ae10cd1ba048eefc879
X-Virus-Checked: Checked by ClamAV on apache.org

--001636416ae10cd1ba048eefc879
Content-Type: text/plain; charset=ISO-8859-1

Deepika,

You just have to run the following command on any of the cluster node

HADOOP_HOME/bin/hadoop jar <job_jar_name> <class_name> <other_args>

this command will automatically copy the jar on all the tasktrackers.

On Sun, Aug 29, 2010 at 6:07 AM, Deepika Khera <Deepika.Khera@avg.com>wrote:

> Hi,
>
> I want to deploy my map reduce job jar on the Hadoop cluster. I've always
> done that by doing the following -
>
> 1. Copying the job jar to all datanodes
> 2. Having the job jar on the hadoop classpath on all machines.
>
> Isn't hadoop capable of copying over the job jar to all machines in the
> cluster? This is what I read (that job tracker copies the job jar, etc) ,
> but if I don't do the above the task trackers cannot find the job. I know I
> am missing something.
>
> Could someone please let me know how I can run my job without having to
> copy it over all machines in the cluster?
>
> Thanks,
> Deepika
>



-- 
Thanks & Regards,
Chandra Prakash Bhagtani,
Nokia India Pvt. Ltd.

--001636416ae10cd1ba048eefc879--

From general-return-2083-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 29 06:16:21 2010
Return-Path: <general-return-2083-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 48705 invoked from network); 29 Aug 2010 06:16:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Aug 2010 06:16:20 -0000
Received: (qmail 86290 invoked by uid 500); 29 Aug 2010 06:16:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85857 invoked by uid 500); 29 Aug 2010 06:16:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85844 invoked by uid 99); 29 Aug 2010 06:16:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 Aug 2010 06:16:16 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of Deepika.Khera@avg.com designates 193.85.188.248 as permitted sender)
Received: from [193.85.188.248] (HELO ms.grisoft.cz) (193.85.188.248)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 Aug 2010 06:16:11 +0000
Received: from deimos.cz.avg.com (unknown [192.168.200.161])
	(using TLSv1 with cipher AES128-SHA (128/128 bits))
	(No client certificate requested)
	by ms.grisoft.cz (Postfix) with ESMTP id DCAA8382C7
	for <general@hadoop.apache.org>; Sun, 29 Aug 2010 08:15:43 +0200 (CEST)
Received: from miranda.us.avg.com (10.32.34.13) by deimos.cz.avg.com
 (192.168.200.161) with Microsoft SMTP Server (TLS) id 8.3.83.0; Sun, 29 Aug
 2010 08:15:43 +0200
Received: from miranda.us.avg.com ([10.32.34.13]) by miranda ([10.32.34.13])
 with mapi; Sun, 29 Aug 2010 02:15:41 -0400
From: Deepika Khera <Deepika.Khera@avg.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Sun, 29 Aug 2010 02:15:45 -0400
Subject: RE: Deploying my job jar on hadoop cluster
Thread-Topic: Deploying my job jar on hadoop cluster
Thread-Index: ActHPLsrt9GRUhcoQ0KmvjiQovbR+gABH7kg
Message-ID: <DA852C02397DBC4EBFAA1B7192FBD15062025E5AC5@miranda>
References: <DA852C02397DBC4EBFAA1B7192FBD15062025E5AC2@miranda>
 <AANLkTi=-dcxp8VVLC8ESpOtLb-X9DNdXr0oPw-KQ9WeJ@mail.gmail.com>
In-Reply-To: <AANLkTi=-dcxp8VVLC8ESpOtLb-X9DNdXr0oPw-KQ9WeJ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Thanks. I had tried running the command as you suggested but got a ClassNot=
FoundException exception(for the mapper class) in the task trackers. I did =
a=20

bin/hadoop jar /root/myjob.jar

(Already had the main class defined in manifest)

There is probably something else that I am doing wrong.

Appreciate your help.

Thanks,
Deepika

-----Original Message-----
From: Chandraprakash Bhagtani [mailto:cpbhagtani@gmail.com]=20
Sent: Saturday, August 28, 2010 10:40 PM
To: general@hadoop.apache.org
Subject: Re: Deploying my job jar on hadoop cluster

Deepika,

You just have to run the following command on any of the cluster node

HADOOP_HOME/bin/hadoop jar <job_jar_name> <class_name> <other_args>

this command will automatically copy the jar on all the tasktrackers.

On Sun, Aug 29, 2010 at 6:07 AM, Deepika Khera <Deepika.Khera@avg.com>wrote=
:

> Hi,
>
> I want to deploy my map reduce job jar on the Hadoop cluster. I've always
> done that by doing the following -
>
> 1. Copying the job jar to all datanodes
> 2. Having the job jar on the hadoop classpath on all machines.
>
> Isn't hadoop capable of copying over the job jar to all machines in the
> cluster? This is what I read (that job tracker copies the job jar, etc) ,
> but if I don't do the above the task trackers cannot find the job. I know=
 I
> am missing something.
>
> Could someone please let me know how I can run my job without having to
> copy it over all machines in the cluster?
>
> Thanks,
> Deepika
>



--=20
Thanks & Regards,
Chandra Prakash Bhagtani,
Nokia India Pvt. Ltd.

From general-return-2084-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 29 07:50:32 2010
Return-Path: <general-return-2084-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83768 invoked from network); 29 Aug 2010 07:50:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Aug 2010 07:50:32 -0000
Received: (qmail 22587 invoked by uid 500); 29 Aug 2010 07:50:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22148 invoked by uid 500); 29 Aug 2010 07:50:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22140 invoked by uid 99); 29 Aug 2010 07:50:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 Aug 2010 07:50:27 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cpbhagtani@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 Aug 2010 07:50:22 +0000
Received: by wwd20 with SMTP id 20so1051054wwd.29
        for <general@hadoop.apache.org>; Sun, 29 Aug 2010 00:49:59 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=34fQlbAX2ijQ3KsMiCbyIOrU8zkoriSgWM+cLc1A/RY=;
        b=UPisZgbnHEfwqbjtMiaLQoBxp6kL1pbzU0YRj1Ejaw/puC8M7mu32v/pzjUDBOix+U
         8Pytl0f7AG8chwxWsggOCCCZG2LMzeCcWfvodHj1+IJJ2qSs2V1gUuBi9yKJPmuHxN3q
         jMzS2jJGJ7ARHEBkyQjvmBz7UWmbg3uLXjbgo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Fd2UsGuR4/FIeU9N3d+Cr8/iqMAjMw9PH5J8ilfqCNFxaGNzMfXrdUTIT2WGWp2cEK
         pFMSmZNeia+CgEj25ZOyaFhcqnpzNIfSq0b3Z03zPTXkyCHIYF34VCL0HRc53aviXxGj
         HOlKi8FFGe2ipllZGJGMB4g6IHPlppWFTcx50=
MIME-Version: 1.0
Received: by 10.216.11.130 with SMTP id 2mr3248874wex.100.1283068198470; Sun,
 29 Aug 2010 00:49:58 -0700 (PDT)
Received: by 10.216.0.66 with HTTP; Sun, 29 Aug 2010 00:49:58 -0700 (PDT)
In-Reply-To: <DA852C02397DBC4EBFAA1B7192FBD15062025E5AC5@miranda>
References: <DA852C02397DBC4EBFAA1B7192FBD15062025E5AC2@miranda>
	<AANLkTi=-dcxp8VVLC8ESpOtLb-X9DNdXr0oPw-KQ9WeJ@mail.gmail.com>
	<DA852C02397DBC4EBFAA1B7192FBD15062025E5AC5@miranda>
Date: Sun, 29 Aug 2010 13:19:58 +0530
Message-ID: <AANLkTi=Oc0MMYGkcBZCs_6Y6VvDtXNGYX6h9QSDwurkS@mail.gmail.com>
Subject: Re: Deploying my job jar on hadoop cluster
From: Chandraprakash Bhagtani <cpbhagtani@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364d1da1314755048ef19871

--0016364d1da1314755048ef19871
Content-Type: text/plain; charset=ISO-8859-1

check whether the user from which you are running the job has permissions on
the jar or the jar is properly generated i.e. the class is present in the
jar or not

On Sun, Aug 29, 2010 at 11:45 AM, Deepika Khera <Deepika.Khera@avg.com>wrote:

> Thanks. I had tried running the command as you suggested but got a
> ClassNotFoundException exception(for the mapper class) in the task trackers.
> I did a
>
> bin/hadoop jar /root/myjob.jar
>
> (Already had the main class defined in manifest)
>
> There is probably something else that I am doing wrong.
>
> Appreciate your help.
>
> Thanks,
> Deepika
>
> -----Original Message-----
> From: Chandraprakash Bhagtani [mailto:cpbhagtani@gmail.com]
> Sent: Saturday, August 28, 2010 10:40 PM
> To: general@hadoop.apache.org
> Subject: Re: Deploying my job jar on hadoop cluster
>
> Deepika,
>
> You just have to run the following command on any of the cluster node
>
> HADOOP_HOME/bin/hadoop jar <job_jar_name> <class_name> <other_args>
>
> this command will automatically copy the jar on all the tasktrackers.
>
> On Sun, Aug 29, 2010 at 6:07 AM, Deepika Khera <Deepika.Khera@avg.com
> >wrote:
>
> > Hi,
> >
> > I want to deploy my map reduce job jar on the Hadoop cluster. I've always
> > done that by doing the following -
> >
> > 1. Copying the job jar to all datanodes
> > 2. Having the job jar on the hadoop classpath on all machines.
> >
> > Isn't hadoop capable of copying over the job jar to all machines in the
> > cluster? This is what I read (that job tracker copies the job jar, etc) ,
> > but if I don't do the above the task trackers cannot find the job. I know
> I
> > am missing something.
> >
> > Could someone please let me know how I can run my job without having to
> > copy it over all machines in the cluster?
> >
> > Thanks,
> > Deepika
> >
>
>
>
> --
> Thanks & Regards,
> Chandra Prakash Bhagtani,
> Nokia India Pvt. Ltd.
>



-- 
Thanks & Regards,
Chandra Prakash Bhagtani,
Nokia India Pvt. Ltd.

--0016364d1da1314755048ef19871--

From general-return-2085-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 06:52:43 2010
Return-Path: <general-return-2085-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34663 invoked from network); 30 Aug 2010 06:52:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 06:52:36 -0000
Received: (qmail 81764 invoked by uid 500); 30 Aug 2010 06:52:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81290 invoked by uid 500); 30 Aug 2010 06:52:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81183 invoked by uid 99); 30 Aug 2010 06:52:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 06:52:28 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=FREEMAIL_FROM,FROM_LOCAL_NOVOWEL,HK_RANDOM_ENVFROM,HTML_MESSAGE,HTML_OBFUSCATE_05_10,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of sssssssenator@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 06:52:22 +0000
Received: by wyf23 with SMTP id 23so8064062wyf.35
        for <multiple recipients>; Sun, 29 Aug 2010 23:52:01 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=iG2/IzRQqw5Eeiqa+Mw44OIHc6k/qKlnn7KUdBkHs/8=;
        b=xIR/cZKauN7DVQ8sLta4OgV99S6wkJJ/JO7Sc3cmwYN+0edDSFQWFYwZL8a4i69kOO
         AWI0gJBja/9/fRg2SNfR+staaTu0xWB1M5ocImFWGUvupBVfBkdRRDIYEGLhV0PVEcMa
         ysfW92WKoTQ7LQ4TBCjIkZs1I3cnzCx+uWHUU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=dW06uN2IvJeTIs6zuBeO9WcRuVQ1sbtwnvaawB+KuHLXaTJTPA6NnRQMuSxs/Dg/bc
         M27Ns3rlbHl8ozzbdHoYWVKlVWya3Gz9sxeuvHRsd45JtszVmQPI50qV1Tgr0MzznqBW
         NlO+iszB3FNsCDA1OTgfOrTf/r8vIuzJHeKvU=
MIME-Version: 1.0
Received: by 10.216.159.195 with SMTP id s45mr4493819wek.43.1283151121358;
 Sun, 29 Aug 2010 23:52:01 -0700 (PDT)
Received: by 10.216.154.132 with HTTP; Sun, 29 Aug 2010 23:52:01 -0700 (PDT)
Date: Mon, 30 Aug 2010 12:22:01 +0530
Message-ID: <AANLkTinGAiSQT+uJQPS-KJ3iy87r-hEwMBGgVP6gQXrk@mail.gmail.com>
Subject: Hadoop Shutdown Problems
From: vaibhav negi <sssssssenator@gmail.com>
To: user@hadoop.apache.org, general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016367b668ac83d2c048f04e6f0

--0016367b668ac83d2c048f04e6f0
Content-Type: text/plain; charset=ISO-8859-1

Hi ,

I am running hadoop 0.20.2 . with 2 node cluster.
I executed script stop-all.sh . But still 2 line logs are getting created
every hour in log directory of name node log directory.
How to completely shutdown hadoop cluster.
Below is the 1 line log.


2010-08-29 00:30:00,018 INFO org.apache.hadoop.hdfs.server.
namenode.FSNamesystem.audit: ugi=root,root,bin,daemon,sys,adm,disk,wheel
ip=/10.0.8.47        cmd=listStatus  src=/user       dst=null
perm=null


Vaibhav Negi

--0016367b668ac83d2c048f04e6f0--

From general-return-2086-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 07:35:50 2010
Return-Path: <general-return-2086-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50108 invoked from network); 30 Aug 2010 07:35:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 07:35:50 -0000
Received: (qmail 19607 invoked by uid 500); 30 Aug 2010 07:35:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19119 invoked by uid 500); 30 Aug 2010 07:35:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19098 invoked by uid 99); 30 Aug 2010 07:35:46 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 07:35:46 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of qwertymaniac@gmail.com designates 209.85.215.176 as permitted sender)
Received: from [209.85.215.176] (HELO mail-ey0-f176.google.com) (209.85.215.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 07:35:23 +0000
Received: by eydd26 with SMTP id d26so3824760eyd.35
        for <general@hadoop.apache.org>; Mon, 30 Aug 2010 00:35:03 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=L7rt3iLg+U91bWBWtUbTaJY5lORs60frxgUodmaIK9c=;
        b=BSuxG6+KLTVRDplIVU1WqpUJl18SpPAUuh8E28F1r9Vl4Ot4LX7m/UPPfKVjXErnlH
         Pn93yV09KI5ccOIL8YL51sYp6BYmmXoyV1gUs9vJEcz13ETxgyYGFS1ubrcuD0qRcs6E
         OSEqrQQn4UGlFeIh+9O53Z4PBUkPXX9vbboWk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=jOnvKljAs7cxlDtkaYXAzojGCtje/1D27RcONP6TPIKtbhEzitnEIloeOFKtZ/rxZ9
         wznFcR9nABEOSRd02RhuoUowUIZBqnKKjVnCrJrNUweBMyQFlWnNPpBxNla6O+6j/f+z
         pYjfiiTn1Qtk6baEd04OkucORTXWcNsFZzQks=
Received: by 10.227.129.149 with SMTP id o21mr4333489wbs.176.1283153702231;
 Mon, 30 Aug 2010 00:35:02 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.236.197 with HTTP; Mon, 30 Aug 2010 00:34:42 -0700 (PDT)
In-Reply-To: <AANLkTinGAiSQT+uJQPS-KJ3iy87r-hEwMBGgVP6gQXrk@mail.gmail.com>
References: <AANLkTinGAiSQT+uJQPS-KJ3iy87r-hEwMBGgVP6gQXrk@mail.gmail.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Mon, 30 Aug 2010 13:04:42 +0530
Message-ID: <AANLkTikBNvh7esmHcUA5RvD9B1MsY+_vPhPAemRBQQ0_@mail.gmail.com>
Subject: Re: Hadoop Shutdown Problems
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Maybe the user who issues stop-all.sh does not have permissions to
terminate the process of NN (and some other, depending on who/what
started it). Check jps listing after stopping and with some ps/top
checks, switch to the proper user and issue a stop-all again?

You can also issue it a SIGTERM I believe.

On Mon, Aug 30, 2010 at 12:22 PM, vaibhav negi <sssssssenator@gmail.com> wr=
ote:
> Hi ,
>
> I am running hadoop 0.20.2 . with 2 node cluster.
> I executed script stop-all.sh . But still 2 line logs are getting created
> every hour in log directory of name node log directory.
> How to completely shutdown hadoop cluster.
> Below is the 1 line log.
>
>
> 2010-08-29 00:30:00,018 INFO org.apache.hadoop.hdfs.server.
> namenode.FSNamesystem.audit: ugi=3Droot,root,bin,daemon,sys,adm,disk,whee=
l
> ip=3D/10.0.8.47 =A0 =A0 =A0 =A0cmd=3DlistStatus =A0src=3D/user =A0 =A0 =
=A0 dst=3Dnull
> perm=3Dnull
>
>
> Vaibhav Negi
>



--=20
Harsh J
www.harshj.com

From general-return-2087-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 08:42:17 2010
Return-Path: <general-return-2087-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66140 invoked from network); 30 Aug 2010 08:42:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 08:42:17 -0000
Received: (qmail 70969 invoked by uid 500); 30 Aug 2010 08:42:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70537 invoked by uid 500); 30 Aug 2010 08:42:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70528 invoked by uid 99); 30 Aug 2010 08:42:11 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 08:42:11 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of angushe@gmail.com designates 209.85.215.176 as permitted sender)
Received: from [209.85.215.176] (HELO mail-ey0-f176.google.com) (209.85.215.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 08:41:50 +0000
Received: by eydd26 with SMTP id d26so3898992eyd.35
        for <general@hadoop.apache.org>; Mon, 30 Aug 2010 01:41:30 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=lxrnzaHVZhuC04Cyo8lCvre+l+Hljddiw+QWFGtv+fg=;
        b=dgA6kMy2PRJXPUE9J4888vGj6Y2ePtynP37lJ0SzMsyJQqEVOgg+cFsguWKvQyWoMM
         7LWZPrbI4YAvtwYIlvAvdcY4eiU88dgwzD5Xz3aAXZODypJZrnzqlyrffzA5pVYoRTV+
         Eq4S/L40j+i+gOm7TRqB9akUFjUkyCSSNYHc0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=vVPpeu0nMOtDM/uAoyjaofASRPzV135N55FP/pcmncl8E3PT7YwgmpqVdaOjwU8wuC
         wZFIShtnKv663ivRqaYJrNHZNVZCPXhnFrhMPP/c3zskAdzdu/wFq/ujRFw/RkCfCWfi
         M182YFJfpfFoglw+CitzdbjEJnTZRZ9Td2e4I=
MIME-Version: 1.0
Received: by 10.227.146.142 with SMTP id h14mr4758181wbv.25.1283157689873;
 Mon, 30 Aug 2010 01:41:29 -0700 (PDT)
Received: by 10.216.159.4 with HTTP; Mon, 30 Aug 2010 01:41:29 -0700 (PDT)
In-Reply-To: <DA852C02397DBC4EBFAA1B7192FBD15062025E5AC5@miranda>
References: <DA852C02397DBC4EBFAA1B7192FBD15062025E5AC2@miranda>
	<AANLkTi=-dcxp8VVLC8ESpOtLb-X9DNdXr0oPw-KQ9WeJ@mail.gmail.com>
	<DA852C02397DBC4EBFAA1B7192FBD15062025E5AC5@miranda>
Date: Mon, 30 Aug 2010 16:41:29 +0800
Message-ID: <AANLkTinydd1fTW=xteCBS-8y+w2zbB+Gjb+NYfvjOuuZ@mail.gmail.com>
Subject: Re: Deploying my job jar on hadoop cluster
From: Angus He <angushe@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Did the Job.setJarByClass get called?  If not, just add it and take
another try.

On Sun, Aug 29, 2010 at 2:15 PM, Deepika Khera <Deepika.Khera@avg.com> wrote:
> Thanks. I had tried running the command as you suggested but got a ClassNotFoundException exception(for the mapper class) in the task trackers. I did a
>
> bin/hadoop jar /root/myjob.jar
>
> (Already had the main class defined in manifest)
>
> There is probably something else that I am doing wrong.
>
> Appreciate your help.
>
> Thanks,
> Deepika
>
> -----Original Message-----
> From: Chandraprakash Bhagtani [mailto:cpbhagtani@gmail.com]
> Sent: Saturday, August 28, 2010 10:40 PM
> To: general@hadoop.apache.org
> Subject: Re: Deploying my job jar on hadoop cluster
>
> Deepika,
>
> You just have to run the following command on any of the cluster node
>
> HADOOP_HOME/bin/hadoop jar <job_jar_name> <class_name> <other_args>
>
> this command will automatically copy the jar on all the tasktrackers.
>
> On Sun, Aug 29, 2010 at 6:07 AM, Deepika Khera <Deepika.Khera@avg.com>wrote:
>
>> Hi,
>>
>> I want to deploy my map reduce job jar on the Hadoop cluster. I've always
>> done that by doing the following -
>>
>> 1. Copying the job jar to all datanodes
>> 2. Having the job jar on the hadoop classpath on all machines.
>>
>> Isn't hadoop capable of copying over the job jar to all machines in the
>> cluster? This is what I read (that job tracker copies the job jar, etc) ,
>> but if I don't do the above the task trackers cannot find the job. I know I
>> am missing something.
>>
>> Could someone please let me know how I can run my job without having to
>> copy it over all machines in the cluster?
>>
>> Thanks,
>> Deepika
>>
>
>
>
> --
> Thanks & Regards,
> Chandra Prakash Bhagtani,
> Nokia India Pvt. Ltd.
>



-- 
Regards
Angus

From general-return-2088-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 09:54:21 2010
Return-Path: <general-return-2088-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85787 invoked from network); 30 Aug 2010 09:54:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 09:54:20 -0000
Received: (qmail 24594 invoked by uid 500); 30 Aug 2010 09:54:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24160 invoked by uid 500); 30 Aug 2010 09:54:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24147 invoked by uid 99); 30 Aug 2010 09:54:15 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 09:54:15 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=FREEMAIL_FROM,FROM_LOCAL_NOVOWEL,HK_RANDOM_ENVFROM,HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of sssssssenator@gmail.com designates 209.85.215.176 as permitted sender)
Received: from [209.85.215.176] (HELO mail-ey0-f176.google.com) (209.85.215.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 09:53:53 +0000
Received: by eydd26 with SMTP id d26so3981327eyd.35
        for <general@hadoop.apache.org>; Mon, 30 Aug 2010 02:53:33 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=ydi6qIvfKs+aWsCjVT+ytR84UIzGFgBsljFdAGr5TgQ=;
        b=xzei54uFGvCVxkH+qxfJy3OW61QOe8CvY8/VbVhQOp5H0yKM0MNlJ1fDOJAY7Ry6RT
         D/B/hbKRAqV3xLPzbVEKw0oQsYltM5SLOQaamqOVsTyFBsRn3rpGS4g4DrnB+5O2nE3S
         Ei/jHchp6f7gkRkfYDatIC9ZfxtnyaX9Y8I3I=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=m4u52H6WvJ6gOwbMhU5cmNH7ezm2kNZHT6MBK40HJrXIRqjz2xhWrmDXass+SZV9F0
         aUPPYksE49An8FvXnsTqufdtk9M4wXLJv6eerzzJVHx/lWc/dnm5UiVX3XCDMjyyriKn
         90DI9WLlX0QaMi/7KfMJySooUu2ltn0q4LwyY=
MIME-Version: 1.0
Received: by 10.227.142.75 with SMTP id p11mr4729092wbu.27.1283162013604; Mon,
 30 Aug 2010 02:53:33 -0700 (PDT)
Received: by 10.216.154.132 with HTTP; Mon, 30 Aug 2010 02:53:33 -0700 (PDT)
In-Reply-To: <AANLkTikBNvh7esmHcUA5RvD9B1MsY+_vPhPAemRBQQ0_@mail.gmail.com>
References: <AANLkTinGAiSQT+uJQPS-KJ3iy87r-hEwMBGgVP6gQXrk@mail.gmail.com>
	<AANLkTikBNvh7esmHcUA5RvD9B1MsY+_vPhPAemRBQQ0_@mail.gmail.com>
Date: Mon, 30 Aug 2010 15:23:33 +0530
Message-ID: <AANLkTimmxQAg6pZ7dic2--DSWjnv7C7dOw6OhM1m4kLX@mail.gmail.com>
Subject: Re: Hadoop Shutdown Problems
From: vaibhav negi <sssssssenator@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e659f7fe029d0c048f07704f
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e659f7fe029d0c048f07704f
Content-Type: text/plain; charset=ISO-8859-1

Hi Harsh,

thanks for reply. actually the problem was that hadoop was not started
correctly but the port was bound. and after making corrections, as the port
was not available, i was getting problems.
 Solved now

Thanks and Regards

Vaibhav Negi


On Mon, Aug 30, 2010 at 1:04 PM, Harsh J <qwertymaniac@gmail.com> wrote:

> Maybe the user who issues stop-all.sh does not have permissions to
> terminate the process of NN (and some other, depending on who/what
> started it). Check jps listing after stopping and with some ps/top
> checks, switch to the proper user and issue a stop-all again?
>
> You can also issue it a SIGTERM I believe.
>
> On Mon, Aug 30, 2010 at 12:22 PM, vaibhav negi <sssssssenator@gmail.com>
> wrote:
> > Hi ,
> >
> > I am running hadoop 0.20.2 . with 2 node cluster.
> > I executed script stop-all.sh . But still 2 line logs are getting created
> > every hour in log directory of name node log directory.
> > How to completely shutdown hadoop cluster.
> > Below is the 1 line log.
> >
> >
> > 2010-08-29 00:30:00,018 INFO org.apache.hadoop.hdfs.server.
> > namenode.FSNamesystem.audit: ugi=root,root,bin,daemon,sys,adm,disk,wheel
> > ip=/10.0.8.47        cmd=listStatus  src=/user       dst=null
> > perm=null
> >
> >
> > Vaibhav Negi
> >
>
>
>
> --
> Harsh J
> www.harshj.com
>

--0016e659f7fe029d0c048f07704f--

From general-return-2089-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 20:42:26 2010
Return-Path: <general-return-2089-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24625 invoked from network); 30 Aug 2010 20:42:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 20:42:26 -0000
Received: (qmail 24569 invoked by uid 500); 30 Aug 2010 20:42:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24507 invoked by uid 500); 30 Aug 2010 20:42:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24499 invoked by uid 99); 30 Aug 2010 20:42:24 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 20:42:24 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of hc.busy@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 20:41:58 +0000
Received: by qwk3 with SMTP id 3so6502916qwk.35
        for <general@hadoop.apache.org>; Mon, 30 Aug 2010 13:41:32 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=Sd7jYb8voezyKURgnoKI8siDaIiv3QodfH8LkVinfAM=;
        b=mLAM/yaL7UxELcsjN1xcjthefxYB1EU+kZut7HW/R5Lyi+41bvumPVpbSX5g12JiDQ
         rvlxQr57NljblNxk9ROKTLmEGxBSrSER5fgOLNPZEcYLPUSSqh22mX6v8V70RYp6VtwC
         wtHITIW62dTwbntktOzqIvP7AkaOLO3xk/pew=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=jVSW3qTqIq41tbztP4EkkAjyIecHNrUZyxq+d7zNcihCnDDSNlt6zrKUmPZV3Lv68P
         oAR7F81WlXQQ/TdeGJpAtETYPDPDey8o147tEThKboxbJTf4ciaGE5P7iMiT+tN/9VER
         +Gor5wRYPdnMaqt8QXhFaax7lYKyRpyNaZevI=
MIME-Version: 1.0
Received: by 10.224.44.144 with SMTP id a16mr3242622qaf.257.1283200891733;
 Mon, 30 Aug 2010 13:41:31 -0700 (PDT)
Received: by 10.229.40.140 with HTTP; Mon, 30 Aug 2010 13:41:31 -0700 (PDT)
In-Reply-To: <AANLkTin1=Za+kPAy_97O8zcLxbA+U54-nHaje_44yMsj@mail.gmail.com>
References: <71D87762-A95B-434D-9FF6-8627AF65790A@yahoo-inc.com>
	<71E1E01A-5389-48C2-9A43-9A8BD6B87A2A@yahoo-inc.com>
	<C1EDB62C3771D74AA823DBBBBFB3F89D4E5DE391@SP2-EX07VS03.ds.corp.yahoo.com>
	<AANLkTin1=Za+kPAy_97O8zcLxbA+U54-nHaje_44yMsj@mail.gmail.com>
Date: Mon, 30 Aug 2010 13:41:31 -0700
Message-ID: <AANLkTikmA=T=0URfdztx7PYVWN-kjxGNCSYMQSh=JRAp@mail.gmail.com>
Subject: Re: [VOTE] Pig to become a TLP
From: hc busy <hc.busy@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00c09f9b09fb53caa9048f107d13
X-Virus-Checked: Checked by ClamAV on apache.org

--00c09f9b09fb53caa9048f107d13
Content-Type: text/plain; charset=ISO-8859-1

+1, Congrads on the near unanimous vote for progress.

Btw, can somebody tell me what the change was from March when this was
discussed and now? At that time, every one seem to have felt that Pig can
only handle Hadoop as backend and doesn't make sense for it to become a TLP
by itself. In fact, I remember a ticket to remove the possibility of a
different backend. What did I miss?


1.) Technically, what developments/or future plans make it a separate
project from Hadoop? (Other backend support?)
2.) What are the non-technical factors that changed since March that made
this vote possible? (Donation from large company?)
3.)  Can I make tax deductible contributions to the Apache Pig Project ?



Alan Gates to pig-dev
show details Mar 19
You have probably heard by now that there is a discussion going on in the
Hadoop PMC as to whether a number of the subprojects (Hbase, Avro,
Zookeeper, Hive, and Pig) should move out from under the Hadoop umbrella and
become top level Apache projects (TLP).  This discussion has picked up
recently since the Apache board has clearly communicated to the Hadoop PMC
that it is concerned that Hadoop is acting as an umbrella project with many
disjoint subprojects underneath it.  They are concerned that this gives
Apache little insight into the health and happenings of the subproject
communities which in turn means Apache cannot properly mentor those
communities.

The purpose of this email is to start a discussion within the Pig community
about this topic.  Let me cover first what becoming TLPwould mean for Pig,
and then I'll go into what options I think we as a community have.

Becoming a TLP would mean that Pig would itself have a PMC that would report
directly to the Apache board.  Who would be on the PMC would be something we
as a community would need to decide.  Common options would be to say all
active committers are on the PMC, or all active committers who have been a
committer for at least a year.  We would also need to elect a chair of the
PMC.  This lucky person would have no additional power, but would have the
additional responsibility of writing quarterly reports on Pig's status for
Apache board meetings, as well as coordinating with Apache to get accounts
for new  committers, etc.  For more information see
http://www.apache.org/foundation/how-it-works.html#roles

Becoming a TLP would not mean that we are ostracized from the Hadoop
community.  We would continue to be invited to Hadoop Summits, HUGs, etc.
 Since all Pig developers and users are by definition Hadoop users, we would
continue to be a strong presence in the Hadoop community.

I see three ways that we as a community can respond to this:

1) Say yes, we want to be a TLP now.
2) Say yes, we want to be a TLP, but not yet.  We feel we need more time to
mature.  If we choose this option we need to be able to clearly articulate
how much time we need and what we hope to see change in that time.
3) Say no, we feel the benefits for us staying with Hadoop outweigh the
drawbacks of being a disjoint subproject.  If we choose this, we need to be
able to say exactly what those benefits are and why we feel they will be
compromised by leaving the Hadoop project.

There may other options that I haven't thought of.  Please feel free to
suggest any you think of.

Questions?  Thoughts?  Let the discussion begin.

Alan.

 Reply
 Forward
Reply
Alan Gates to pig-dev
show details Mar 31
So far I haven't seen any feedback on this.  Apache has asked the Hadoop PMC
to submit input in April on whether some subprojects should be promoted to
TLPs.  We, the Pig community, need to give feedback to the Hadoop PMC on how
we feel about this.  Please make your voice heard.

So now I'll head my own call and give my thoughts on it.

The biggest advantage I see to being a TLP is a direct connection to Apache.
 Right now all of the Pig team's interaction with Apache is through the
Hadoop PMC.  Being directly connected to Apache would benefit Pig team
members who would have a better view into Apache.  It would also raise our
profile in Apache and thus make other projects more aware of us.

However, I am concerned about loosing Pig's explicit connection to Hadoop.
 This concern has a couple of dimensions.  One, Hadoop and MapReduce are the
current flavor of the month in computing.  Given that Pig shares a name with
the common farm animal, it's hard to be sure based on search statistics.
 But Google trends shows that "hadoop" is searched on much more frequently
than "hadoop pig" or "apache pig" (see
http://www.google.com/trends?q=hadoop%2Chadoop+pig).  I am guessing that
most Pig users come from Hadoop users who discover Pig via Hadoop's website.
 Loosing that subproject tab on Hadoop's front page may radically lower the
number of users coming to Pig to check out our project.  I would argue that
this benefits Hadoop as well, since high level languages like Pig Latin have
the potential to greatly extend the user base and usability of Hadoop.

Two, being explicitly connected to Hadoop keeps our two communities aware of
each others needs.  There are features proposed for MR that would greatly
help Pig.  By staying in the Hadoop community Pig is better positioned to
advocate for and help implement and test those features.  The response to
this will be that Pig developers can still subscribe to Hadoop mailing
lists, submit patches, etc.  That is, they can still be part of the Hadoop
community.  Which reinforces my point that it makes more sense to leave Pig
in the Hadoop community since Pig developers will need to be part of that
community anyway.

Finally, philosophically it makes sense to me that projects that are tightly
connected belong together.  It strikes me as strange to have Pig as a
TLP completely
dependent on another TLP.  Hadoop was originally a subproject of Lucene.  It
moved out to be a TLP when it became obvious that Hadoop had become
independent of and useful apart from Lucene.  Pig is not in that position
relative to Hadoop.

So, I'm -1 on Pig moving out.  But this is a soft -1.  I'm open to being
persuaded that I'm wrong or my concerns can be addressed while still having
Pig as a TLP.

Alan.
- Show quoted text -
 Reply
 Forward
Reply
Dmitriy Ryaboy to pig-dev
show details Mar 31
Over time, Pig is increasing its coupling to Hadoop (for good reasons),
rather than decreasing it. If and when Pig becomes a viable entity without
hadoop around, it might make sense as a TLP. As is, I think becoming a TLP
will only introduce unnecessary administrative and bureaucratic headaches.
So my vote is also -1.

-Dmitriy
- Show quoted text -
 Reply
 Forward
 Invite Dmitriy Ryaboy to chat
Reply
Thejas Nair to pig-dev, Dmitriy
show details Apr 2
I agree with Alan and Dmitriy - Pig is tightly coupled with hadoop, and
heavily influenced by its roadmap. I think it makes sense to continue as a
sub-project of hadoop.

-Thejas
- Show quoted text -
 Reply
 Reply to all
 Forward
Reply
Santhosh Srinivasan to pig-dev, Dmitriy
show details Apr 3
I see this as a multi-part question. Looking back at some of the
significant roadmap/existential questions asked in the last 12 months, I
see the following:

1. With the introduction of SQL, what is the philosophy of Pig (I sent
an email about this approximately 9 months ago)
2. What is the approach to support backward compatibility in Pig (Alan
had sent an email about this 3 months ago)
3. Should Pig be a TLP (the current email thread).

Here is my take on answering the aforementioned questions.

The initial philosophy of Pig was to be backend agnostic. It was
designed as a data flow language. Whenever a new language is designed,
the syntax and semantics of the language have to be laid out. The syntax
is usually captured in the form of a BNF grammar. The semantics are
defined by the language creators. Backward compatibility is then a
question of holding true to the syntax and semantics. With Pig, in
addition to the language, the Java APIs were exposed to customers to
implement UDFs (load/store/filter/grouping/row transformation etc),
provision looping since the language does not support looping constructs
and also support a programmatic mode of access. Backward compatibility
in this context is to support API versioning.

Do we still intend to position as a data flow language that is backend
agnostic? If the answer is yes, then there is a strong case for making
Pig a TLP.

Are we influenced by Hadoop? A big YES! The reason Pig chose to become a
Hadoop sub-project was to ride the Hadoop popularity wave. As a
consequence, we chose to be heavily influenced by the Hadoop roadmap.

Like a good lawyer, I also have rebuttals to Alan's questions :)

1. Search engine popularity - We can discuss this with the Hadoop team
and still retain links to TLP's that are coupled (loosely or tightly).
2. Explicit connection to Hadoop - I see this as logical connection v/s
physical connection. Today, we are physically connected as a
sub-project. Becoming a TLP, will not increase/decrease our influence on
the Hadoop community (think Logical, Physical and MR Layers :)
3. Philosophy - I have already talked about this. The tight coupling is
by choice. If Pig continues to be a data flow language with clear syntax
and semantics then someone can implement Pig on top of a different
backend. Do we intend to take this approach?

I just wanted to offer a different opinion to this thread. I strongly
believe that we should think about the original philosophy. Will we have
a Pig standards committee that will decide on the changes to the
language (think C/C++) if there are multiple backend implementations?

I will reserve my vote based on the outcome of the philosophy and
backward compatibility discussions. If we decide that Pig will be
treated and maintained like a true language with clear syntax and
semantics then we have a strong case to make it into a TLP. If not, we
should retain our existing ties to Hadoop and make Pig into a data flow
language for Hadoop.

Santhosh
- Show quoted text -
 Reply
 Reply to all
 Forward
Reply
Ashutosh Chauhan to pig-dev
show details Apr 4
I concur with Santhosh here. I think main question we need to answer
here is how close our ties are with Hadoop currently and how it will
be in future ? When Pig was originally designed the intent was to keep
it backend neutral, so  much so that there was a reference backend
implementation (also known as local engine) which had nothing to do
with Hadoop. But things have changed since then. Hadoop's local mode
is adopted in favor of Pig's own local mode. We have moved from being
backend agnostic to hadoop favoring. And while this was happening, it
seems we tried to keep Pig Latin language independent of hadoop
backend  while Pig runtime started to make use of hadoop concepts.

Apart from design decisions, this move also has a practical impact on
our codebase. Since we adopted Hadoop more closely, we got rid of an
extra layer of abstraction and instead started using similar
abstractions already existing in Hadoop. This has a positive impact
that it simplified the codebase and provides tighter integration with
Hadoop.
So, if we are continuing in a direction where Hadoop is our only
backend (or atleast a favored one), close ties to Hadoop are useful
because of the reasons Alan and Dmitriy pointed out. if not, then I
think moving out to TLP makes sense. Since, there is no efforts which
I am aware of, is trying to plug in a different backend for Pig, I
think maintaining close ties with Hadoop is useful for Pig. In future
when there is a different distributed computing platform comes up
which we want to use as backend, we can revisit our decision. So, as
for things stand today I am -1 to move out of  Hadoop.

And I would also like to reiterate my point that though Pig runtime
may continue to get closer to Hadoop, we shall keep Pig Latin
completely backend agnostic.

Ashutosh
- Show quoted text -
 Reply
 Forward
 Invite Ashutosh Chauhan to chat
Reply
hc busy to pig-dev
show details Apr 5

This is awesome!!! As much as I hate PJM's for wasting time at all the
places that I've worked at, I think formalizing the management group(PMC) to
openly and clearly determine feature roadmap and dev schedule is the best
thing pig can have.

I once commented to my co-worker (also heavy pig user) that pig's
organization (with all due respect to all you hardworking people) is like a
pigsty! documentations all over the place, javadocs from three versions ago,
much of the documentation doesn't match actual features... links to the
download page is broken.

If you look at cascading's website... it's so much cleaner. (Of course... we
still use pig because it works well)

I think as TLP, pig will receive better marketing and better support in a
way that will propel it both in popularity and in the amount of support it
receives.

As a user, that change will be good for me.
- Show quoted text -
 Reply
 Forward
Reply
Pradeep Kamath to pig-dev
show details Apr 5
I agree with Ashutosh and Santhosh. Just based on the current direction of
the project I think we are more closely tied with Hadoop now (with Pig 0.7,
our load/store interfaces are very closely tied with Hadoop) - hence for now
my vote would be a -1 to be a TLP - if there is change in that
direction/philosophy to be really backend agnostic I think we should revisit
this question.

Pradeep

-----Original Message-----
From: Ashutosh Chauhan [mailto:ashutosh.chauhan@gmail.com]
Sent: Sunday, April 04, 2010 11:11 PM
To: pig-dev@hadoop.apache.org
- Show quoted text -
 Reply
 Forward
Reply
Alan Gates to pig-dev
show details Apr 5
I agree that Pig's code documentation is in sad shape.  I think our user
documentation for each release is good, of limited.  I hope that our
documents on wiki (such as PigJournal) help people understand our roadmap.
 Please let us know if you disagree so we can find ways to improve it.

That said, it isn't clear to me how Pig being a TLP will solve that.  The
current committers or some subset thereof (see original message) would
become the PMC.  Other than having expanded powers to vote on releases and
who becomes new committers, the role of these new PMC members would not
change much.  They won't have anymore time to address documentation and
communication issues.  We need to find a way to address those no matter what
governance framework or community Pig is in.

Alan.
- Show quoted text -
 Reply
 Forward
Reply
Alan Gates to pig-dev
show details Apr 5
Prognostication is a difficult business.  Of course I'd love it if someday
there is an ISO Pig Latin committee (with meetings in cool exotic places)
deciding the official standard for Pig Latin.  But that seems like saying in
your start up's business plan, "When we reach Google's size, then we'll do
x".  If there ever is an ISO Pig Latin standard it will be years off.

As others have noted, staying tight to Hadoop now has many advantages, both
in technical and adoption terms.  Hence my advocacy of keeping Pig Latin
Hadoop agnostic while tightly integrating the backend.  Which is to say that
in my view, Pig is Hadoop specific now, but there may come a day when that
is no longer true.   Whether Pig will ever move past just running on Hadoop
to running in other parallel systems won't be known for years to come.
 Given that, do you think it makes sense to say that Pig stays a subproject
for now, but if it someday grows beyond Hadoop only it becomes a TLP?  I
could agree to that stance.

Alan.
- Show quoted text -
 Reply
 Forward
Reply
hc busy to pig-dev
show details Apr 5

I guess this is more of a suggestion for roadmap than TLP discussion, I
think the PMC/committers can create a dedicate position what maintains the
web/doc's. Somebody who yell and screams until the doc's are in sync with
the implementation before the release.

Because TLP is an elevation of status in addition to internal
re-organization. I think it might to create the PR needed to attract the
talents to fill in that job...
- Show quoted text -
 Reply
 Forward
Reply
hc busy to pig-dev
show details Apr 5
> Of course I'd love it if someday there is an ISO Pig Latin committee (with
meetings in cool exotic places) deciding the official standard for Pig
Latin.

haha!!! Some exotic place like Yahoo's  HQ in sunny Sunnyvale California?

I guess it feels like it depends on the roadmap more than roadmap depends on
it. In terms of positioning, a TLP would appear to potential users who are
evaluating alternatives to consider it as _the_ choice as opposed to one of
the choices. If the ambition is to take it there, then TLP, as useless as it
may seem right now, might actually be worth the effort to attain.

I mean, would you rather wait until Hive makes TLP and then play catch up? I
mean, I can kinda see them doing that...
- Show quoted text -
 Reply
 Forward
Reply
Dmitriy Ryaboy to pig-dev
show details Apr 5
The Twitter office is cushier and has more bars within stumbling distance.
Just sayin'.

To the subject at hand -- I don't think TLP standing has the PR value you
think it does... feature set, velocity of development, adoption,
flexibility, etc -- those are far more important.

-Dmitriy
- Show quoted text -
 Reply
 Forward
 Invite Dmitriy Ryaboy to chat
Reply
Santhosh Srinivasan to pig-dev
show details Apr 5
"Given that, do you think it makes
sense to say that Pig stays a subproject for now, but if it someday
grows beyond Hadoop only it becomes a TLP?  I could agree to that
stance."

Bingo!

Santhosh

-----Original Message-----
From: Alan Gates [mailto:gates@yahoo-inc.com]
Sent: Monday, April 05, 2010 11:37 AM
To: pig-dev@hadoop.apache.org
- Show quoted text -
 Reply
 Forward
Reply
hc busy to pig-dev
show details Apr 5
>The Twitter office is cushier and has more bars within stumbling
distance. Just sayin'.

and strip clubs too, I gather there're a couple on Market... near civic bart
stop ;-)

oh... hey, you guys are at a nice place... lot's of night clubs near there
too .


> "Given that, do you think it makes sense to say that Pig stays a
subproject for now, but if it someday grows beyond Hadoop only it becomes a
TLP?  I could agree to that stance."


Oops, I didn't read your whole message... I think TLP could be part of the
roadmap: Planned publicity, like planned pregnancy, is a good thing.

And on the way there, we should add dedicated resource that updates
documentation and links on the website... :-)
- Show quoted text -
 Reply
 Forward
Reply
Daniel Dai to pig-dev
show details Apr 5
I agree with the stance that we remain in Hadoop until we see more
compelling reasons, such as Pig go beyond Hadoop happens. Currently I cannot
fully weight the advantage and disadvantage of becoming a TLP. But provides
this is a point of no return, I don't want to move unless we do have a
strong motivation. We can always choose to become TLP later when we feel
more convinced to that.

Daniel

--------------------------------------------------
From: "Santhosh Srinivasan" <sms@yahoo-inc.com>
Sent: Monday, April 05, 2010 12:22 PM

To: <pig-dev@hadoop.apache.org>
Subject: RE: Begin a discussion about Pig as a top level project
- Show quoted text -

 Reply
 Forward
 Invite Daniel Dai to chat

On Tue, Aug 24, 2010 at 9:46 PM, Anant Wadhokar <anant.bw@gmail.com> wrote:

> +1
>
> On Wed, Aug 25, 2010 at 12:04 AM, Olga Natkovich <olgan@yahoo-inc.com
> >wrote:
>
> > +1
> >
> > > From: Alan Gates <gates@yahoo-inc.com>
> > > Date: August 23, 2010 10:38:12 AM PDT
> > > To: "general@hadoop.apache.org" <general@hadoop.apache.org>
> > > Subject: [VOTE] Pig to become a TLP
> > > Reply-To: "general@hadoop.apache.org" <general@hadoop.apache.org>
> > >
> > > I propose that Pig become a top level Apache project.
> > >
> > > The Pig development community has already voted on and approved this
> > > proposal.  In summary, the community voted that all current active
> > > committers (listed at http://hadoop.apache.org/pig/whoweare.html ) as
> > > of August 18 2010 should become PMC members, and that Olga Natkovich
> > > should be the PMC chair.  They also agreed that adopting bylaws should
> > > be among the first tasks of the new PMC.  Full details of the vote can
> > > be seen at http://bit.ly/c3GDyl
> > >
> > > Please vote on sending this proposal to the Apache Board.
> > >
> > > Below is a draft resolution to be sent to the Apache Board:
> > >
> > > Establish the Apache Pig Project
> > >
> > >         WHEREAS, the Board of Directors deems it to be in the best
> > >         interests of the Foundation and consistent with the
> > >         Foundation's purpose to establish a Project Management
> > >         Committee charged with the creation and maintenance of
> > >         open-source software related to parallel analysis of large
> > >         data sets for distribution at no charge to the public.
> > >
> > >         NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> > >         Committee (PMC), to be known as the "Apache Pig Project",
> > >         be and hereby is established pursuant to Bylaws of the
> > >         Foundation; and be it further
> > >
> > >         RESOLVED, that the Apache Pig Project be and hereby is
> > >         responsible for the creation and maintenance of software
> > >         related to parallel analysis of large data sets; and be
> > >         it further
> > >
> > >         RESOLVED, that the office of "Vice President, Apache Pig" be
> > >         and hereby is created, the person holding such office to
> > >         serve at the direction of the Board of Directors as the chair
> > >         of the Apache Pig Project, and to have primary responsibility
> > >         for management of the projects within the scope of
> > >         responsibility of the Apache Pig Project; and be it further
> > >
> > >         RESOLVED, that the persons listed immediately below be and
> > >         hereby are appointed to serve as the initial members of the
> > >         Apache Pig Project:
> > >
> > >           * Benjamin Reed          <breed@apache.org>
> > >           * Daniel Dai                  <daijy@apache.org>
> > >           * Alan Gates                 <gates@apache.org>
> > >           * Giridharen Kesavan   <gkesavan@apache.org>
> > >           * Olga Natkovich           <olga@apache.org>
> > >           * Pradeep Kamath        <pradeepkth@apache.org>
> > >           * Santhosh Srinivasan  <sms@apache.org>
> > >           * Yan Zhou                   <yanz@apache.org>
> > >           * Jeff Zhang                  <zjffdu@apache.org>
> > >           * Ashutosh Chauhan     <hashutosh@apache.org>
> > >           * Richard Ding               <rding@apache.org>
> > >           * Dmitriy Ryaboy            <dvryaboy@apache.org>
> > >           * Thejas Nair                  <thejas@apache.org>
> > >
> > >         NOW, THEREFORE, BE IT FURTHER RESOLVED, that Olga Natkovich
> > >         be appointed to the office of Vice President, Apache Pig, to
> > >         serve in accordance with and subject to the direction of the
> > >         Board of Directors and the Bylaws of the Foundation until
> > >         death, resignation, retirement, removal or disqualification,
> > >         or until a successor is appointed; and be it further
> > >
> > >         RESOLVED, that the initial Apache Pig PMC be and hereby is
> > >         tasked with the creation of a set of bylaws intended to
> > >         encourage open development and increased participation in the
> > >         Apache Pig Project; and be it further
> > >
> > >         RESOLVED, that the Apache Pig Project be and hereby
> > >         is tasked with the migration and rationalization of the Apache
> > >         Hadoop Pig sub-project; and be it further
> > >
> > >         RESOLVED, that all responsibilities pertaining to the Apache
> > >         Hadoop Pig sub-project encumbered upon the
> > >         Apache Hadoop Project are hereafter discharged.
> > >
> > >
> > >
> > >
> >
> >
>

--00c09f9b09fb53caa9048f107d13--

From general-return-2090-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 21:15:38 2010
Return-Path: <general-return-2090-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35006 invoked from network); 30 Aug 2010 21:15:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 21:15:38 -0000
Received: (qmail 61372 invoked by uid 500); 30 Aug 2010 21:15:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61290 invoked by uid 500); 30 Aug 2010 21:15:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61282 invoked by uid 99); 30 Aug 2010 21:15:36 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 21:15:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 30 Aug 2010 21:15:19 +0000
Received: (qmail 34468 invoked by uid 99); 30 Aug 2010 21:14:57 -0000
Received: from localhost.apache.org (HELO wlan-snve-153-34.corp.yahoo.com) (127.0.0.1)
  (smtp-auth username acmurthy, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 21:14:57 +0000
Message-Id: <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
From: Arun C Murthy <acmurthy@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Date: Mon, 30 Aug 2010 14:14:54 -0700
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


On Aug 23, 2010, at 5:27 PM, Arun C Murthy wrote:
> In the interim I'd like to propose we push a hadoop-0.20-security
> release off the Yahoo! patchset (http://github.com/yahoo/hadoop-
> common). This will ensure the community benefits from all the work
> done at Yahoo! for over 12 months *now*, and ensures that we do not
> have to wait until hadoop-0.22 which has all of these patches.

Since most people seemed to think of it as a reasonable idea, I'm  
going to create the hadoop-0.20-security branch and start the  
necessary work.

thanks,
Arun



From general-return-2091-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 22:30:39 2010
Return-Path: <general-return-2091-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66606 invoked from network); 30 Aug 2010 22:30:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 22:30:39 -0000
Received: (qmail 40259 invoked by uid 500); 30 Aug 2010 22:30:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40004 invoked by uid 500); 30 Aug 2010 22:30:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39983 invoked by uid 99); 30 Aug 2010 22:30:37 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 22:30:37 +0000
X-ASF-Spam-Status: No, hits=2.7 required=10.0
	tests=RCVD_ILLEGAL_IP,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of athusoo@facebook.com designates 69.63.178.179 as permitted sender)
Received: from [69.63.178.179] (HELO mx-out.facebook.com) (69.63.178.179)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 22:30:15 +0000
Received: from [10.18.255.179] ([10.18.255.179:57710] helo=mail.thefacebook.com)
	by mta026.snc1.facebook.com (envelope-from <athusoo@facebook.com>)
	(ecelerity 2.2.2.45 r(34067)) with ESMTP
	id 4D/C0-30338-1E03C7C4; Mon, 30 Aug 2010 15:29:54 -0700
Received: from SC-MBX03.TheFacebook.com ([169.254.2.176]) by
 sc-hub04.TheFacebook.com ([fe80::8df5:7f90:d4a0:bb9%11]) with mapi; Mon, 30
 Aug 2010 15:27:47 -0700
From: Ashish Thusoo <athusoo@facebook.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "hive-dev@hadoop.apache.org" <hive-dev@hadoop.apache.org>,
	"hive-user@hadoop.apache.org" <hive-user@hadoop.apache.org>
Subject: RE: [VOTE] Hive as a TLP
Thread-Topic: [VOTE] Hive as a TLP
Thread-Index: ActFWXOffgxgalakRSm+ZBxRJ3CyBgA7efAAAJK3yoA=
Date: Mon, 30 Aug 2010 22:27:46 +0000
Message-ID: <09370B27F829314182C33A219530C104037B4FA7@sc-mbx03.TheFacebook.com>
References: <09370B27F829314182C33A219530C104037ABC90@sc-mbx03.TheFacebook.com>
 <AANLkTi=BgaX8tU7j3+LDAkJrRfsTKQ-MzmKV0r7=5EyZ@mail.gmail.com>
In-Reply-To: <AANLkTi=BgaX8tU7j3+LDAkJrRfsTKQ-MzmKV0r7=5EyZ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

With 10 +1 votes this vote passes.

Owen,=20

Please forward this to the Apache board.

Thanks,
Ashish=20

-----Original Message-----
From: Tom White [mailto:tom@cloudera.com]=20
Sent: Friday, August 27, 2010 10:24 AM
To: general@hadoop.apache.org
Subject: Re: [VOTE] Hive as a TLP

+1

Tom

On Thu, Aug 26, 2010 at 1:01 PM, Ashish Thusoo <athusoo@facebook.com> wrote=
:
> The Hive development community voted and passed the following=20
> resolution. The details of the vote is at
>
> http://www.bit.ly/aJogyU
>
> The PMC will comprise of the current committers on Hive (as of 8/24/2010)=
 with Namit Jain being the chair.
>
> Please vote on sending this resolution to the Apache Board.
>
> Thanks,
> Ashish
>
> Draft Resolution to be sent to the Apache Board
> -----------------------------------------------
>
> Establish the Apache Hive Project
>
> =A0 =A0 =A0 =A0 WHEREAS, the Board of Directors deems it to be in the bes=
t
> =A0 =A0 =A0 =A0 interests of the Foundation and consistent with the
> =A0 =A0 =A0 =A0 Foundation's purpose to establish a Project Management
> =A0 =A0 =A0 =A0 Committee charged with the creation and maintenance of
> =A0 =A0 =A0 =A0 open-source software related to parallel analysis of larg=
e
> =A0 =A0 =A0 =A0 data sets for distribution at no charge to the public.
>
> =A0 =A0 =A0 =A0 NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> =A0 =A0 =A0 =A0 Committee (PMC), to be known as the "Apache Hive Project"=
,
> =A0 =A0 =A0 =A0 be and hereby is established pursuant to Bylaws of the
> =A0 =A0 =A0 =A0 Foundation; and be it further
>
> =A0 =A0 =A0 =A0 RESOLVED, that the Apache Hive Project be and hereby is
> =A0 =A0 =A0 =A0 responsible for the creation and maintenance of software
> =A0 =A0 =A0 =A0 related to parallel analysis of large data sets; and be
> =A0 =A0 =A0 =A0 it further
>
> =A0 =A0 =A0 =A0 RESOLVED, that the office of "Vice President, Apache Hive=
" be
> =A0 =A0 =A0 =A0 and hereby is created, the person holding such office to
> =A0 =A0 =A0 =A0 serve at the direction of the Board of Directors as the c=
hair
> =A0 =A0 =A0 =A0 of the Apache Hive Project, and to have primary responsib=
ility
> =A0 =A0 =A0 =A0 for management of the projects within the scope of
> =A0 =A0 =A0 =A0 responsibility of the Apache Hive Project; and be it furt=
her
>
> =A0 =A0 =A0 =A0 RESOLVED, that the persons listed immediately below be an=
d
> =A0 =A0 =A0 =A0 hereby are appointed to serve as the initial members of t=
he
> =A0 =A0 =A0 =A0 Apache Hive Project:
> =A0 =A0 =A0 =A0 =A0 =A0 * Namit Jain (namit@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * John Sichi (jvs@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * Zheng Shao (zshao@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * Edward Capriolo (appodictic@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * Raghotham Murthy (rsm@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * Ning Zhang (nzhang@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * Paul Yang (pauly@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * He Yongqiang (he yongqiang@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * Prasad Chakka (prasadc@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * Joydeep Sen Sarma (jsensarma@apache.org)
> =A0 =A0 =A0 =A0 =A0 =A0 * Ashish Thusoo (athusoo@apache.org)
>
> =A0 =A0 =A0 =A0 NOW, THEREFORE, BE IT FURTHER RESOLVED, that Namit Jain
> =A0 =A0 =A0 =A0 be appointed to the office of Vice President, Apache Hive=
, to
> =A0 =A0 =A0 =A0 serve in accordance with and subject to the direction of =
the
> =A0 =A0 =A0 =A0 Board of Directors and the Bylaws of the Foundation until
> =A0 =A0 =A0 =A0 death, resignation, retirement, removal or disqualificati=
on,
> =A0 =A0 =A0 =A0 or until a successor is appointed; and be it further
>
> =A0 =A0 =A0 =A0 RESOLVED, that the initial Apache Hive PMC be and hereby =
is
> =A0 =A0 =A0 =A0 tasked with the creation of a set of bylaws intended to
> =A0 =A0 =A0 =A0 encourage open development and increased participation in=
 the
> =A0 =A0 =A0 =A0 Apache Hive Project; and be it further
>
> =A0 =A0 =A0 =A0 RESOLVED, that the Apache Hive Project be and hereby
> =A0 =A0 =A0 =A0 is tasked with the migration and rationalization of the A=
pache
> =A0 =A0 =A0 =A0 Hadoop Hive sub-project; and be it further
>
> =A0 =A0 =A0 =A0 RESOLVED, that all responsibilities pertaining to the Apa=
che
> =A0 =A0 =A0 =A0 Hive sub-project encumbered upon the
> =A0 =A0 =A0 =A0 Apache Hadoop Project are hereafter discharged.
>

From general-return-2092-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 31 21:42:15 2010
Return-Path: <general-return-2092-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 99206 invoked from network); 31 Aug 2010 21:42:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Aug 2010 21:42:15 -0000
Received: (qmail 64498 invoked by uid 500); 31 Aug 2010 21:42:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 64396 invoked by uid 500); 31 Aug 2010 21:42:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 64388 invoked by uid 99); 31 Aug 2010 21:42:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 21:42:13 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 21:42:08 +0000
Received: from [192.168.0.199] (snvvpn2-10-73-152-c129.hq.corp.yahoo.com [10.73.152.129])
	by mrout2.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7VLfMo7044787
	for <general@hadoop.apache.org>; Tue, 31 Aug 2010 14:41:22 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=AvRZ4tOwng33TO7G2pLgOB/P6Wb3zFLhZ1BGd6VRx+LQPi5t90t4tVLl3wD5i509
Message-Id: <351A8FBE-528C-4352-8D14-3AC913836336@yahoo-inc.com>
From: Alan Gates <gates@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <AANLkTikmA=T=0URfdztx7PYVWN-kjxGNCSYMQSh=JRAp@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Pig to become a TLP
Date: Tue, 31 Aug 2010 14:41:22 -0700
References: <71D87762-A95B-434D-9FF6-8627AF65790A@yahoo-inc.com> <71E1E01A-5389-48C2-9A43-9A8BD6B87A2A@yahoo-inc.com> <C1EDB62C3771D74AA823DBBBBFB3F89D4E5DE391@SP2-EX07VS03.ds.corp.yahoo.com> <AANLkTin1=Za+kPAy_97O8zcLxbA+U54-nHaje_44yMsj@mail.gmail.com> <AANLkTikmA=T=0URfdztx7PYVWN-kjxGNCSYMQSh=JRAp@mail.gmail.com>
X-Mailer: Apple Mail (2.936)


On Aug 30, 2010, at 1:41 PM, hc busy wrote:

> +1, Congrads on the near unanimous vote for progress.
>
> Btw, can somebody tell me what the change was from March when this was
> discussed and now? At that time, every one seem to have felt that  
> Pig can
> only handle Hadoop as backend and doesn't make sense for it to  
> become a TLP
> by itself. In fact, I remember a ticket to remove the possibility of a
> different backend. What did I miss?

I can only speak for myself, but in the original discussion thread on  
pig-dev (http://bit.ly/byD7L8 ) I discussed why I changed my vote over  
the course of the last few months.

>
>
> 1.) Technically, what developments/or future plans make it a separate
> project from Hadoop? (Other backend support?)

While we still want to design Pig and Pig Latin in such a way that  
there may be separate implementations that do not run on Hadoop,  
nothing in this regard has changed in the last few months.  What did  
change is that I (and I think others) realized that being connected  
technically to Hadoop does not mean we need to be connected in our  
governance.

> 2.) What are the non-technical factors that changed since March that  
> made
> this vote possible? (Donation from large company?)

Apache does not require that projects raise any finances.  Pig is (as  
it has been) supported by the contributions of those choose to work on  
it and those who are paid to work on it.

> 3.)  Can I make tax deductible contributions to the Apache Pig  
> Project ?

Wow, that's a huge vote of confidence.  Thanks.  But no.   
Contributions to Pig can be made (as before) in code, documentation,  
user feedback, etc.  You can donate money to Apache itself, which goes  
to support the infrastructure for Pig and all Apache projects.  For  
more info see http://apache.org/foundation/support-apache.html  (I  
have no idea if that is tax deductible or not.)

Alan.


From general-return-2093-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 31 23:42:23 2010
Return-Path: <general-return-2093-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39315 invoked from network); 31 Aug 2010 23:42:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Aug 2010 23:42:23 -0000
Received: (qmail 5536 invoked by uid 500); 31 Aug 2010 23:42:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5464 invoked by uid 500); 31 Aug 2010 23:42:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5456 invoked by uid 99); 31 Aug 2010 23:42:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 23:42:21 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 23:42:14 +0000
Received: from EGL-EX07CAS03.ds.corp.yahoo.com (egl-ex07cas03.eglbp.corp.yahoo.com [203.83.248.219])
	by mrout2.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o7VNf0N0093844
	for <general@hadoop.apache.org>; Tue, 31 Aug 2010 16:41:01 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:references:in-reply-to:accept-language:
	content-language:x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:
	content-type:content-transfer-encoding:mime-version;
	b=DMzvZI+IcQ+P4kT3qOBNPZvZSTyf4vumLJFFN1iNqdOD+HIPAWykKd1+5tunr+YG
Received: from EGL-EX07VS01.ds.corp.yahoo.com ([203.83.248.205]) by
 EGL-EX07CAS03.ds.corp.yahoo.com ([203.83.248.219]) with mapi; Wed, 1 Sep 2010
 05:11:00 +0530
From: "Giridharan  Kesavan" <gkesavan@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 1 Sep 2010 05:10:59 +0530
Subject: Re: hudson patch test jobs : hadoop pig and zookeeper
Thread-Topic: hudson patch test jobs : hadoop pig and zookeeper
Thread-Index: ActJZfd+kSBIfNKIRQqaJf8bsvbOmA==
Message-ID: <2FFA136A-1495-4522-A29D-FA176722F94E@yahoo-inc.com>
References: <155916E3-8F8D-4EF7-8992-4C6E3382D241@yahoo-inc.com>
In-Reply-To: <155916E3-8F8D-4EF7-8992-4C6E3382D241@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


Hudson patch test job is back UP and Running on the new hudson.apache.org m=
achine.

Thanks,
Giri


On Aug 24, 2010, at 4:38 PM, Giridharan Kesavan wrote:

> Hi,
>=20
> We have a new hudson master hudson.apache.org and hudson.zones.apache.org=
 is retired.
> This means that we need to port all our patch test admin jobs for hadoop(=
common,hdfs,mapred), pig and zookeeper to the new hudson master.
>=20
> I'm working on configuring patch admin jobs with the new hudson master: h=
udson.apache.org. (this is exactly the reason for why the patch test builds=
 are not running at the moment)
>=20
> Thanks
> Giri


From general-return-1597-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 04:18:35 2010
Return-Path: <general-return-1597-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88010 invoked from network); 1 Jun 2010 04:18:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 04:18:34 -0000
Received: (qmail 9872 invoked by uid 500); 1 Jun 2010 04:18:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7475 invoked by uid 500); 1 Jun 2010 04:18:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7452 invoked by uid 99); 1 Jun 2010 04:18:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 04:18:28 +0000
X-ASF-Spam-Status: No, hits=0.8 required=10.0
	tests=AWL,NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 04:18:22 +0000
Received: from SP1-EX07CAS02.ds.corp.yahoo.com (sp1-ex07cas02.ds.corp.yahoo.com [216.252.116.138])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o514HsKe037913;
	Mon, 31 May 2010 21:17:55 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:accept-language:content-language:x-ms-has-attach:
	x-ms-tnef-correlator:user-agent:acceptlanguage:content-type:
	content-transfer-encoding:mime-version;
	b=hd77I8bcze5kghUyxPyfBDioCKCGOvfQ5uer9smerU51P0UV452G30KTqZRKBiRm
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 SP1-EX07CAS02.ds.corp.yahoo.com ([216.252.116.167]) with mapi; Mon, 31 May
 2010 21:16:41 -0700
From: Milind A Bhandarkar <milindb@yahoo-inc.com>
To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>,
        "mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>,
        "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 31 May 2010 21:16:38 -0700
Subject: First International Mapreduce Workshop 2010: Paper submission
 deadline July 15 , 2010
Thread-Topic: First International Mapreduce Workshop 2010: Paper submission
 deadline July 15 , 2010
Thread-Index: AcsBQTq6BX2rRUa9dkmwYCyERP5Z/Q==
Message-ID: <C829D7B6.60276%milindb@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
user-agent: Microsoft-Entourage/13.0.0.090609
acceptlanguage: en-US
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

First International Workshop on Theory and Practice of Mapreduce (MapRed
2010)

Indianapolis, USA: Nov 30 - Dec 3, 2010

MapReduce is a programming model, introduced by Google in 2004, to simplify
distributed processing of large datasets on clusters of commodity computers=
.
Currently, there exist several open-source implementations including Hadoop=
.
MapReduce became the model of choice for many web enterprises, very often
being the enabler for cloud services. Recently, it also gained significant
attention in scientific community for parallel data analysis e.g. Rhipe. Th=
e
purpose of the workshop is to discuss both theoretical and practical
advances in MapReduce (or similar systems), identify open issues, present
new related tools and report on scientific or industrial applications.
Therefore, we are glad to invite contributions including, but not limited
to:

=80 Implementation/architecture of MapReduce programming model
=80 Improvements and extensions
=80 Optimizations e.g multi-core, GPU
=80 Fault-tolerance
=80 Security
=80 Job scheduling
=80 Environments and tools
=80 Real-time extensions and applications
=80 Data collection
=80 Data- and Compute-intensive problems
=80 MapReduce as a support for mobile services
=80 Algorithms based on MapReduce paradigm
=80 Applications and lessons learned
=80 Scientific and industrial data analysis (including higher level languag=
es
based on MapReduce)
=80 Cloud computing implementation and applications of MapReduce (e.g. on A=
WS)


Authors are invited to submit full papers of two types:
- research papers at most 8 pages,
- application papers at most 6 pages,
including figures, tables and references using the IEEE format for
conference proceedings.

Accepted papers will appear in the main conference proceedings. The
conference and workshop proceedings will be published by IEEE (pending).
After the conference, extended and revised versions of distinguished papers
will be invited for possible publication in a special issue of the Journals=
:
Concurrency and Computation: Practice and Experience Journal (Wiley,
SCI-indexed)
Computer Science and Technology Journal (Springer, SCI-indexed pending)


Important dates
=80 Submission deadline: July 15th , 2010
=80 Author notification: Aug. 10th, 2010
=80 Camera-ready manuscript: Sept. 1st, 2010
=80 Author registration: Sept. 1st, 2010

For more info: mapred2010@easychair.org

Organized by the Indiana University,

mapreduce.CloudCom.org

Technically sponsored by the IEEE Computer Society

First International Workshop on Theory and Practice of MapReduce
(MAPRED'2010)
Indianapolis, USA, Nov 30 =AD Dec 3, 20


Workshop/Program Chairs

Milind A Bhandarkar, Yahoo!, USA
Tomasz Wiktor Wlodarczyk, University of Stavanger, Norway

Steering Chair

Chunming Rong, University of Stavanger, Norway

Technical Chair

Rui Maximo Esteves, University of Stavanger, Norway

Program Committee

Abdulrahman Azab, University of Stavanger, Norway
Aleksandar Bradic, Vast.com, USA
Alexandre de Assis Bento Lima, Universidade Federal do Rio de Janeiro,
Brazil
Chris K Wensel, Concurrent, USA
Colin Evans, Metaweb Technologies, USA
Eddy Caron, ENS de Lyon, France
Gabriel Antoniu, INRIA, France
Gilles Fedak, INRIA, France
Hidemoto Nakada, National Institute of Advanced Industrial Science and
Technology, Japan
Oleg Lodygensky, Universite Paris Sud 11, France
Rui Maximo, University of Stavanger, Norway
Scott Preece, Yahoo!, USA
Teresa Wingfield, Datameer, USA
Vadim Zaliva, Tristero Consulting, USA
Xuanhua Shi, Huazhong University of Science and Technology, China
Zheng Shao, Facebook, USA

--=20
Milind Bhandarkar
Y!IM: GridSolutions
Tel: 408-203-5213=20
(milindb@yahoo-inc.com)


From general-return-1598-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 19:25:37 2010
Return-Path: <general-return-1598-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84041 invoked from network); 1 Jun 2010 19:25:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 19:25:36 -0000
Received: (qmail 58855 invoked by uid 500); 1 Jun 2010 19:25:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58627 invoked by uid 500); 1 Jun 2010 19:25:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58437 invoked by uid 99); 1 Jun 2010 19:25:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 19:25:34 +0000
X-ASF-Spam-Status: No, hits=0.9 required=10.0
	tests=AWL,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 19:25:29 +0000
Received: by pvc21 with SMTP id 21so694517pvc.35
        for <multiple recipients>; Tue, 01 Jun 2010 12:25:07 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.114.237.21 with SMTP id k21mr5469623wah.141.1275420307486; 
	Tue, 01 Jun 2010 12:25:07 -0700 (PDT)
Received: by 10.143.16.21 with HTTP; Tue, 1 Jun 2010 12:25:07 -0700 (PDT)
In-Reply-To: <AANLkTinbgBYlWwaedkPTwfk8qpvnUgczFojTnoQJr0i0@mail.gmail.com>
References: <AANLkTimeqo7RDWvCIWDqiUM4M7XKoeeXlCDPw8TY1xix@mail.gmail.com>
	<AANLkTinbgBYlWwaedkPTwfk8qpvnUgczFojTnoQJr0i0@mail.gmail.com>
Date: Tue, 1 Jun 2010 12:25:07 -0700
Message-ID: <AANLkTikPRrzHiTpUWsCXeGeNtR16N8dQF3IQPKwkNpZs@mail.gmail.com>
Subject: Re: Contributor Meeting Minutes 05/28/2010
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Cc: mapreduce-dev@hadoop.apache.org, hdfs-dev@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I posted a link to the slides on the wiki:

http://wiki.apache.org/hadoop/HadoopContributorsMeeting20100528

On Fri, May 28, 2010 at 7:59 PM, Eli Collins <eli@cloudera.com> wrote:
> Slides attached. =A0Thanks for taking notes Chris!
>
>
> On Fri, May 28, 2010 at 5:37 PM, Chris Douglas <cdouglas@apache.org> wrot=
e:
>> This month, the MapReduce + HDFS contributor meeting was held at
>> Cloudera Headquarters.
>>
>> Announcements for contributor meetings are here:
>> http://www.meetup.com/Hadoop-Contributors/
>>
>> Minutes follow. No decisions were made at this meeting, but the
>> following issues were discussed and may presage future discussion and
>> decisions on these lists.
>>
>> Eli, I think you have all the slides. Would you mind sending them out? -=
C
>>
>> =3D=3D 0.21 release update =3D=3D
>> * Continuing to close blockers, ping people for updates and suggestions
>> * About 20 open blockers. Many are MapReduce documentation that may be
>> pushed. Speak up if 0.21 is missing anything substantive.
>> * Common/HDFS visibility and annotations are close to consensus;
>> MapReduce annotations are committed to trunk and the 0.21 branch
>>
>> =3D=3D HEP proposal =3D=3D
>> (what follows is the sketch presented at the meeting. A full proposal
>> with concrete details will be circulated on the list)
>>
>> * Based on- and very similar to- the PEP (Python Enhancement Proposal) P=
rocess
>> * Audience is HDFS and MapReduce; not necessarily adopted by other subpr=
ojects
>> =A0- Addresses the perception that there is friction between
>> innovation/experimentation and stability
>> * Not for small enhancements, features, and bug fixes. This should not
>> slow down typical development or impede casual contribution to Hadoop
>> * Primary mechanism for new features, collecting input, documenting
>> design decisions
>> * JIRA is good for details, but not for deciding on wide shifts in direc=
tion
>> * Purpose is for author to build consensus and gather dissenting opinion=
s.
>> =A0- All may comment, but Editors will review incoming HEP material
>> =A0- Editors determine only whether the HEP is complete, not whether
>> they believe it is a sound idea
>> =A0- Editors are appointed by the PMC
>> =A0- Mechanism for appointing Editors and term of service TBD
>> =A0 =A0- Apache Board appoints Shepherds for projects somewhat randomly,
>> to projects. A similar mechanism could work for incoming HEPs
>> =A0- Proposal *may* come with code, but not necessarily.
>> Drafting/baking of the HEP occurs in public on a list dedicated to
>> that particular proposal. Once Editors certify the HEP as complete, it
>> is sent to general@ for wider discussion.
>> =A0 =A0- The discussion phase begins on general@. The mailing list exist=
s
>> to ensure the HEP is complete enough to present to the community.
>> =A0- Some discussion on the difference between posting to general@ and
>> posting to the HEP list. Completeness is, of course, subjective. If
>> the Editor and Author disagree whether the proposal affects an aspect
>> of the framework enough to merit special consideration, it is not
>> entirely clear how to resolve the disagreement.
>> =A0 =A0- In general, the role of the Editor in the community-driven
>> process of Hadoop is not entirely clear. It may be possible to
>> optimize it out.
>> =A0- Once discussion ends, the HEP is passed (or fails to pass) by a
>> vote of the PMC (mechanics undefined). In Python, the result is
>> committed to the repository. A similar practice would make sense in
>> Hadoop.
>> * Which issues require HEPs?
>> =A0- Discussion ranged. Append, backup namenode, edit log rewrite, et
>> al. were examples of features substantial enough to merit a HEP. Pure
>> Java CRC is an example of an enhancement that would not. Whether an
>> explicit process must be in place to determine whether an issue
>> requires a HEP is not clear.
>> =A0- Viewing HEPs as a way of soliciting consensus for an approach
>> might be more accurate. Going through the HEP process should always
>> improve the chances of a successful proposal
>>
>> * Evaluation
>> =A0- The proposal may be rejected if it is redundant with existing
>> functionality, technically unsound, insufficiently motivated, no
>> backwards compatibility story, etc.
>> =A0- Implementation is not necessary, and is lightly discouraged.
>> Feedback is less welcome once code is in hand.
>> =A0- Purpose is to be clear about the acceptance criteria for that
>> issue, e.g. concerns that the proposal may not scale or may harm
>> performance
>> =A0- Dissenting opinions must be recorded accurately. Quoting would be
>> a safe practice for the Author to encourage HEP reviewers not to block
>> the product of the proposal.
>>
>> * The testing burden and completion strategy may be ambiguous
>> =A0- Whether the proposal affects scalability may not be testable by
>> the implementer. Completing the proposal to address all use cases may
>> require considerably more work than the Author is willing or motivated
>> to invest.
>> =A0- The HEP discussion on general@ should explore whether such
>> objections are merited and reasonable. For example, a particularly
>> obscure/esoteric use case could be included as a condition for
>> acceptance if the dissenter is willing to invest the resources to
>> test/validate it. The process is flexible in this regard.
>> =A0 =A0- But it is not infinitely flexible. Backwards compatibility,
>> performance regression, availability, and other considerations need
>> not be called out in every HEP.
>> =A0 =A0- Traditional concerns need to be documented. Acceptance criteria
>> should ideally be automated and reproducible in different
>> organizations
>>
>> =3D=3D Branching =3D=3D
>> * A patch and a branch are isomorphic from a policy perspective. Of
>> course, they are functionally distinct: branches are easier to
>> collaborate on and are, generally, longer-lived than are patches. But
>> special policies need not be derived to account for these differences,
>> which concern the production of the code, not its review and
>> acceptance.
>> * Some developers find branches to be easier to review than very large
>> patches and easier to merge, given a toolchain that supports this.
>> =A0- Subversion currently is difficult to adapt to this model
>> =A0- Could be done on a HEP-by-HEP basis, as a condition for acceptance
>> * Eclipse Labs
>> =A0- Branded version of Google Code (same functionality, w/ Eclipse bran=
d)
>> =A0- Not official Eclipse projects, but associated with Eclipse
>> =A0- Apache/Hadoop may consider a similar strategy
>> =A0- Distinct from Apache Labs, as one need not be a committer, follow
>> its rules for releases, etc.
>>
>> =3D=3D Contrib =3D=3D
>> * Modules (such as fuse-dfs) are not actively maintained in the main
>> repository and would benefit from a release schedule decoupled from
>> the rest of Hadoop
>> * With few exceptions, the contrib modules have smaller, often
>> discrete groups of maintainers. It may be worth exploring whether
>> these projects could live elsewhere
>>
>

From general-return-1599-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 20:19:48 2010
Return-Path: <general-return-1599-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 51417 invoked from network); 1 Jun 2010 20:19:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 20:19:48 -0000
Received: (qmail 34992 invoked by uid 500); 1 Jun 2010 20:19:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34949 invoked by uid 500); 1 Jun 2010 20:19:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34941 invoked by uid 99); 1 Jun 2010 20:19:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 20:19:47 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [98.136.44.37] (HELO n61.bullet.mail.sp1.yahoo.com) (98.136.44.37)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 01 Jun 2010 20:19:37 +0000
Received: from [216.252.122.217] by n61.bullet.mail.sp1.yahoo.com with NNFMP; 01 Jun 2010 20:19:15 -0000
Received: from [67.195.9.83] by t2.bullet.sp1.yahoo.com with NNFMP; 01 Jun 2010 20:19:15 -0000
Received: from [67.195.9.110] by t3.bullet.mail.gq1.yahoo.com with NNFMP; 01 Jun 2010 20:19:15 -0000
Received: from [127.0.0.1] by omp114.mail.gq1.yahoo.com with NNFMP; 01 Jun 2010 20:19:15 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 486020.86320.bm@omp114.mail.gq1.yahoo.com
Received: (qmail 4714 invoked by uid 60001); 1 Jun 2010 20:19:15 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1275423555; bh=jG8SnY+4L3HFLpaeaAheXbEhctwP/+1jHrZJGhAMEhc=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=xX9J73V6jljF5FYF8HWP9itJN3kMyaLyfWtOUUz9vbGdcPOmpC4lshy/6S8xh+I5EaqEf6BvzHz+HZqFGf/U+mZbtU3o/uXi4j4iNVyBlALqKTrdQTRZrCE6izs8lebapX2NtSn2DfH3e85aBuN/RSvJ6rwbdZTuEkSLEiItwek=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=ftPhDnZCdtgWZM8ZjkzNfchMXmM3u5SRgjBjNznJFnB0TAwj549DQIqqClFYraX+nHMEe40bxvMiZo5moMs9VecOtiyJL8fKjsBvv4ngTM3+we9qJUM1gD9pBXcDofgnRaj+FzE3EEYL3Uxt0lFQxvrRvKHAveJAlhtuVKoGl3U=;
Message-ID: <394531.3151.qm@web114518.mail.gq1.yahoo.com>
X-YMail-OSG: PIUAY1IVM1lPuQvcuArOla28Ba0jFbCityJOhkh1935bxWD
 _GsP0YWUc78.Z8TUJenAS99Rid_Y0d3psOQ.BbRSioOhuNqGlKEpejQpuZoR
 tQpPeaDNASmIgONIS4ZA.cb4V_MNdlatvVhJBeO60Zi5uTdG06OtrOMIf.Mm
 50HgVumkO29zAaeSSqc075PlkHbGEZIzWzp7sYncNdsgypdIdPErEsbjdf5Y
 b9ioNHg.dB7Emqsg8Lw6U4YJnMRGa2_OGJwjzeuXdQdAesG_PfDACEcL1iYx
 wP3OWoZH3rTO3hlIrCk8_gtEy5OStTrNoH3UC1cp932AfOHXNKsd16Ts1uuD
 qQbf605xrJ9VvHnZgkBScwGvtbDJ0mYrooL4uiKk9xCgwG7lVAONE7t5yv2K
 t7H2DS8HcjOzhzalwfBwFbJvoHA2giHoEjt24bk.9Efd63Zn7DBf3mN8ZMQJ
 ecywPWleuedccyi1JVUXA4__2.vtAmm3WfTqNl1t2UDYxnffF_ep5_KvXDcc
 1RtnTf1ogyJsZEFo_0fFK8apqxyj60n_gDfeLBCRKosM_b_kT
Received: from [209.116.62.98] by web114518.mail.gq1.yahoo.com via HTTP; Tue, 01 Jun 2010 13:19:15 PDT
X-Mailer: YahooMailRC/374.4 YahooMailWebService/0.8.103.269680
Date: Tue, 1 Jun 2010 13:19:15 -0700 (PDT)
From: C J <c.joshi26@yahoo.com>
Subject: "No datanode to stop " message on stop-dfs.sh
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-421273419-1275423555=:3151"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-421273419-1275423555=:3151
Content-Type: text/plain; charset=us-ascii

Hi,

I have a brand new setup of hadoop machine cluster with 50 machines. I see some weird issues come up with the cluster with time....Things run just fine for a few days and then when I try to run stop-dfs.sh, it says 

no namenode to stop
hadoop-07: no data node to stop
hadoop-08: no data node to stop
.
.
.
hadoop-03: no secondarynamenode to stop

When I go to these machines, the data node is actually running. 

Any idea what can cause issue like this? The last time it happened I killed all the running datanodes manually and then started the dfs. It started fine. After that even the stop-dfs.sh worked as expected. But now it got back to the same situation again.

One more thing I see a lot of left over running "Child" tasks from task attempts on these machines.

Appreciate any help.

Thanks,
C


      
--0-421273419-1275423555=:3151--


From general-return-1600-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 21:28:45 2010
Return-Path: <general-return-1600-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72040 invoked from network); 1 Jun 2010 21:28:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 21:28:45 -0000
Received: (qmail 22699 invoked by uid 500); 1 Jun 2010 21:28:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22640 invoked by uid 500); 1 Jun 2010 21:28:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22632 invoked by uid 99); 1 Jun 2010 21:28:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 21:28:44 +0000
X-ASF-Spam-Status: No, hits=0.9 required=10.0
	tests=AWL,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 21:28:39 +0000
Received: by pvc21 with SMTP id 21so777973pvc.35
        for <general@hadoop.apache.org>; Tue, 01 Jun 2010 14:28:18 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.115.84.8 with SMTP id m8mr5724263wal.9.1275427698380; Tue, 01 
	Jun 2010 14:28:18 -0700 (PDT)
Received: by 10.143.16.21 with HTTP; Tue, 1 Jun 2010 14:28:18 -0700 (PDT)
In-Reply-To: <AANLkTikuyUulssueBs8EQJNeJJN3_NFa7GfflAaeAFgH@mail.gmail.com>
References: <AANLkTimLB-YNWnMwa93YL8Lpg9hJLxrCKbYL3YUr6de8@mail.gmail.com>
	<66BC29ED-35C1-4916-BDFE-C768810C6EA6@mac.com>
	<AANLkTimCkqddJ521jIPp0dVHC__vZ2PcavAIrNl334L0@mail.gmail.com>
	<4BFBFAC3.2040807@apache.org>
	<AANLkTilhcUYlFRwc1vc7-sFybjUw5c_uPosptIGb-Gn1@mail.gmail.com>
	<4BFCF6CF.2000605@apache.org>
	<AANLkTilLOmfMwuIFc2kzsXMQ7yhbXtrA0jhb3pigTEht@mail.gmail.com>
	<AANLkTikuyUulssueBs8EQJNeJJN3_NFa7GfflAaeAFgH@mail.gmail.com>
Date: Tue, 1 Jun 2010 14:28:18 -0700
Message-ID: <AANLkTikZN5RFeP7veaXeZM2ws4ZNBXRBBMBEnMlDOw-3@mail.gmail.com>
Subject: Re: [DISCUSSION] Proposal for making core Hadoop changes
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hey Jeff,

Blueprints (it's a launchpad thing) is more of an issue tracking
system (launchpad doesn't put features/enhancements in their bug
database), eg drizzle has lots of blueprints, and blueprints for
cleaning up code, adding config flags, etc. We'll use jira for that
kind of stuff, the HEP is for larger stuff that needs more upfront
discussion.

Thanks,
Eli

On Mon, May 31, 2010 at 10:16 AM, Jeff Hammerbacher <hammer@cloudera.com> w=
rote:
> A far more lightweight example of multi-issue feature planning in an open
> source project comes from Drizzle and their "blueprints":
> https://blueprints.launchpad.net/drizzle.
>
> Each "spec" has a drafter, an approver, and an assignee; declares the oth=
er
> specs on which it depends; points to the relevant branches in the source
> tree and issues in the issue tracker; and has a priority, definition stat=
e,
> and implementation state.
>
> I don't know how it's working out for them in practice, but on paper it
> looks quite nice.
>
> On Wed, May 26, 2010 at 9:13 AM, Eli Collins <eli@cloudera.com> wrote:
>
>> > No, but I'd estimate the cost of merging at 1-2 days work a week just =
to
>> > pull in the code *and identify why the tests are failing*. Git may be
>> better
>> > at merging in changes, but if Hadoop doesn't work on my machine after =
the
>> > merge, I need to identify whether its my code, the merged code, some
>> machine
>> > quirk, etc. It's the testing that is the problem for me, not the
>> > merge effort. That's the Hadoop own tests any my own functional test
>> suites,
>> > the ones that bring up clusters and push work through. Those are the
>> > troublespots, as they do things that hadoop's own tests don't do, like=
 as
>> > for all the JSP pages.
>>
>> I've lived off a git branch of common/hdfs for half a year with a big
>> uncommitted patch, it's no where near 1-2 days of effort per week to
>> merge in changes from trunk. If the tests are passing on trunk, and
>> they fail after your merge then those are real test failures due to
>> your change (and therefore should require effort). The issues with
>> your internal tests failing due to changes on trunk is the same
>> whether you merge or you just do an update - you have to update before
>> checking in the patch anyway - so that issue is about the state of
>> trunk when you merge or update, rather than about being on a branch.
>>
>> >
>> >> Might find the
>> >> following interesting:
>> >> http://incubator.apache.org/learn/rules-for-revolutionaries.html
>> >
>> > There's a long story behind JDD's paper, I'm glad you have read it, it
>> does
>> > lay out what is effectively the ASF process for effecting significant
>> change
>> > -but it doesn't imply that's the only process for having changes.
>> >
>>
>> Just to be clear I don't mean imply that branches are the only process
>> for making changes. Interesting that this is considered the effective
>> ASF process, it hasn't seemed to me that recent big features on hadoop
>> have used it, only one I'm aware of that was done on a branch was
>> append.
>>
>> > I think gradual evolution in trunk is good, it lets people play with
>> what's
>> > coming in. Having lots of separate branches and everyone's private
>> release
>> > being a merge of many patches that you choose is bad.
>>
>> Agreed. =A0Personally I don't think people should release from branches.
>> And in practice I don't think you'll see lots of branches, people can
>> and would still develop on trunk. Getting changes merged from a branch
>> back to trunk before the whole branch is merged is a good thing, the
>> whole branch may never be merged and that's OK too. Branches are a
>> mechanism, releases are policy.
>>
>> Thanks,
>> Eli
>>
>

From general-return-1601-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 21:30:06 2010
Return-Path: <general-return-1601-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72277 invoked from network); 1 Jun 2010 21:30:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 21:30:05 -0000
Received: (qmail 24145 invoked by uid 500); 1 Jun 2010 21:30:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24084 invoked by uid 500); 1 Jun 2010 21:30:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24075 invoked by uid 99); 1 Jun 2010 21:30:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 21:30:04 +0000
X-ASF-Spam-Status: No, hits=0.2 required=10.0
	tests=AWL,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 21:29:57 +0000
Received: from SNV-EXPF01.ds.corp.yahoo.com (snv-expf01.ds.corp.yahoo.com [207.126.227.250])
	by mrout2.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o51LTElv069777;
	Tue, 1 Jun 2010 14:29:14 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:cc:message-id:
	thread-topic:thread-index:mime-version:content-type:
	content-transfer-encoding:return-path:x-originalarrivaltime;
	b=rUTMKmt8VRspONtay7Lfj3Q1g3/C7E8758zXhVai3Nl3s+bCtoPjyMI9ycTGXTsf
Received: from SNV-EXVS06.ds.corp.yahoo.com ([207.126.227.234]) by SNV-EXPF01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Tue, 1 Jun 2010 14:29:14 -0700
Received: from 10.72.111.153 ([10.72.111.153]) by SNV-EXVS06.ds.corp.yahoo.com ([207.126.227.82]) via Exchange Front-End Server snv-webmail.corp.yahoo.com ([207.126.227.59]) with Microsoft Exchange Server HTTP-DAV ;
 Tue,  1 Jun 2010 21:28:53 +0000
User-Agent: Microsoft-Entourage/12.24.0.100205
Date: Tue, 01 Jun 2010 14:28:53 -0700
Subject: [VOTE] Chukwa as TLP?
From: Eric Yang <eyang@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: Bill Graham <billgraham@gmail.com>, Jiaqi Tan <tanjiaqi@gmail.com>,
        Jerome Boulon <jboulon@gmail.com>, Ariel Rabkin <asrabkin@gmail.com>
Message-ID: <C82AC9A5.8C32%eyang@yahoo-inc.com>
Thread-Topic: [VOTE] Chukwa as TLP?
Thread-Index: AcsB0W7dgQlmbfPShEWXCOQcnrztAA==
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-OriginalArrivalTime: 01 Jun 2010 21:29:14.0036 (UTC) FILETIME=[7B676F40:01CB01D1]

Please vote as to whether you think Chukwa should become a top-level
Apache project.

I've included below a draft board resolution.  It lists all current
active Chukwa committers as initial members of the project management
committee (PMC) and myself, Eric Yang, as the initial chair.

Do the good people of Hadoop support sending this request on to the Hadoop
PMC?

regards,
Eric

------------

   X. Establish the Apache Chukwa Project

      WHEREAS, the Board of Directors deems it to be in the best
      interests of the Foundation and consistent with the
      Foundation's purpose to establish a Project Management
      Committee charged with the creation and maintenance of
      open-source software related to data serialization
      for distribution at no charge to the public.

      NOW, THEREFORE, BE IT RESOLVED, that a Project Management
      Committee (PMC), to be known as the "Apache Chukwa Project",
      be and hereby is established pursuant to Bylaws of the
      Foundation; and be it further

      RESOLVED, that the Apache Chukwa Project be and hereby is
      responsible for the creation and maintenance of software
      related to a distributed database; and be it further

      RESOLVED, that the office of "Vice President, Apache Chukwa" be
      and hereby is created, the person holding such office to
      serve at the direction of the Board of Directors as the chair
      of the Apache Chukwa Project, and to have primary responsibility
      for management of the projects within the scope of
      responsibility of the Apache Chukwa Project; and be it further

      RESOLVED, that the persons listed immediately below be and
      hereby are appointed to serve as the initial members of the
      Apache Chukwa Project:

        * Eric Yang <eyang@apache.org>
        * Ariel Rabkin <asrabkin@apache.org>
        * Jerome Boulon <jboulon@apache.org>
        * Bill Graham <billgraham@gmail.com>
        * Jiaqi Tan <tanjiaqi@gmail.com>

      NOW, THEREFORE, BE IT FURTHER RESOLVED, that Eric Yang
      be appointed to the office of Vice President, Apache Chukwa, to
      serve in accordance with and subject to the direction of the
      Board of Directors and the Bylaws of the Foundation until
      death, resignation, retirement, removal or disqualification,
      or until a successor is appointed; and be it further

      RESOLVED, that the initial Apache Chukwa PMC be and hereby is
      tasked with the creation of a set of bylaws intended to
      encourage open development and increased participation in the
      Apache Chukwa Project; and be it further

      RESOLVED, that the Apache Chukwa Project be and hereby
      is tasked with the migration and rationalization of the Apache
      Hadoop Chukwa sub-project; and be it further

      RESOLVED, that all responsibilities pertaining to the Apache
      Hadoop Chukwa sub-project encumbered upon the
      Apache Hadoop Project are hereafter discharged.


From general-return-1602-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 22:59:46 2010
Return-Path: <general-return-1602-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93763 invoked from network); 1 Jun 2010 22:59:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 22:59:46 -0000
Received: (qmail 44779 invoked by uid 500); 1 Jun 2010 22:59:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44741 invoked by uid 500); 1 Jun 2010 22:59:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44733 invoked by uid 99); 1 Jun 2010 22:59:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 22:59:45 +0000
X-ASF-Spam-Status: No, hits=0.3 required=10.0
	tests=AWL,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.59.212] (HELO qmta14.westchester.pa.mail.comcast.net) (76.96.59.212)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 22:59:36 +0000
Received: from omta03.westchester.pa.mail.comcast.net ([76.96.62.27])
	by qmta14.westchester.pa.mail.comcast.net with comcast
	id QacD1e0020bG4ec5EmzGuu; Tue, 01 Jun 2010 22:59:16 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta03.westchester.pa.mail.comcast.net with comcast
	id Qmyy1e00F2SbwD53Pmz14E; Tue, 01 Jun 2010 22:59:14 +0000
Cc: Bill Graham <billgraham@gmail.com>,
 Jiaqi Tan <tanjiaqi@gmail.com>,
 Jerome Boulon <jboulon@gmail.com>,
 Ariel Rabkin <asrabkin@gmail.com>
Message-Id: <FCDB1BBF-87C9-4274-8FEE-B143C5C0D470@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <C82AC9A5.8C32%eyang@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Chukwa as TLP?
Date: Tue, 1 Jun 2010 15:58:57 -0700
References: <C82AC9A5.8C32%eyang@yahoo-inc.com>
X-Mailer: Apple Mail (2.936)


On Jun 1, 2010, at 2:28 PM, Eric Yang wrote:

> Please vote as to whether you think Chukwa should become a top-level
> Apache project.

-1

In my opinion, the Chukwa PMC doesn't have enough experience with the  
Apache Way (ie. processes) to be a self-sustaining PMC. I'd be +1 for  
moving to Incubator.

-- Owen

From general-return-1603-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 23:34:19 2010
Return-Path: <general-return-1603-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4221 invoked from network); 1 Jun 2010 23:34:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 23:34:19 -0000
Received: (qmail 76964 invoked by uid 500); 1 Jun 2010 23:34:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76922 invoked by uid 500); 1 Jun 2010 23:34:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76910 invoked by uid 99); 1 Jun 2010 23:34:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 23:34:18 +0000
X-ASF-Spam-Status: No, hits=2.5 required=10.0
	tests=AWL,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 23:34:13 +0000
Received: from [10.72.120.91] (tryarticle-lm.corp.yahoo.com [10.72.120.91])
	by mrout1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o51NX52H073308;
	Tue, 1 Jun 2010 16:33:05 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=cc:message-id:from:to:in-reply-to:content-type:
	mime-version:subject:date:references:x-mailer;
	b=lbhqAenj/fPEMLT9MT8B2JmvQ7iNEA2Z+3SBm78njNqy9hBcK73kbeFIlhNugwU/
Cc: general@hadoop.apache.org
Message-Id: <BD9C6F51-44A8-45A3-90C9-C12DA90E4601@yahoo-inc.com>
From: "Owen O'Malley" <oom@yahoo-inc.com>
To: "Owen O'Malley" <omalley@apache.org>
In-Reply-To: <9DD2220F-023D-4E6B-88C5-F08B184B6102@apache.org>
Content-Type: multipart/alternative; boundary=Apple-Mail-8-775241319
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Last call: ApacheCon TechTalks CFP closes today
Date: Tue, 1 Jun 2010 16:33:04 -0700
References: <9DD2220F-023D-4E6B-88C5-F08B184B6102@apache.org>
X-Mailer: Apple Mail (2.936)

--Apple-Mail-8-775241319
Content-Type: text/plain;
	charset=WINDOWS-1252;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: quoted-printable

We've decided to extend the deadline a week to this Friday. As a =20
reminder, all speakers get two nights at the conference hotel and free =20=

registration at ApacheCon.

-- Owen

On May 28, 2010, at 2:40 PM, Owen O'Malley wrote:

> All,
>    As a reminder the CFP for proposals closes today.  The CFP is at =
http://blogs.apache.org/conferences/date/20100428
>
> -- Owen
>
> ApacheCon North America 2010
> 1-5 November 2010 -- Westin Peachtree in Atlanta
>
> Technical Tracks: Call For Participation
> All submissions must be received by Friday, 28 May 2010 at midnight =20=

> Pacific Time.
> The official conference, trainings, and expo of The Apache Software =20=

> Foundation (ASF) returns to Atlanta this November, with dozens of =20
> technical, business, and community-focused sessions at the beginner, =20=

> intermediate, and advanced levels.
>
> Over the past decade, the ASF has gone from strength to strength, =20
> developing and shepherding nearly 150 Top-Level Projects and new =20
> initiatives in the Apache Incubator and Labs. This year's ApacheCon =20=

> celebrates how Apache technologies have sparked creativity, =20
> challenged processes, streamlined development, improved =20
> collaboration, launched businesses, bolstered economies, and =20
> improved lives.
>
> We are proud of our achievements and recognize that the global =20
> Apache community --both developers and users-- are responsible for =20
> the success and popularity of our products.
>
> The ApacheCon Planning Team are soliciting 50-minute technical =20
> presentations for the next conference, which will focus on the theme =20=

> =93Servers, the Cloud, and Innovation=94.
>
> We are particularly interested in highly-relevant, professionally-=20
> directed presentations that demonstrate specific probrlems and real-=20=

> world solutions. Part of the technical program has already been =20
> planned; we welcome proposals based on the following Apache Projects =20=

> and related technical areas:
>
> - Cassandra/NoSQL
> - Content Technologies
> - (Java) Enterprise Development
> - Felix/OSGi
> - Geronimo
> - Hadoop + friends/Cloud Computing
> - Lucene, Mahout + friends/Search
> - Tomcat
> - Tuscany
> Submissions are open to anyone with relevant expertise: ASF =20
> affiliation is not required to present at, attend, or otherwise =20
> participate in ApacheCon.
>
> Please keep in mind that whilst we encourage submissions that the =20
> highlight the use of specific Apache solutions, we are unable to =20
> accept marketing/commercially-oriented presentations.
>
> Other proposals, such as panels, or those longer than 50 minutes in =20=

> duration have been considered in the past. You are welcome to submit =20=

> an alternate presentation, however, such sessions are accepted under =20=

> exceptional circumstances. Please be as descriptive as possible, =20
> including names/bios of proposed panelists and any related details.
>
> All accepted speakers (not co-presenters) qualify for general =20
> conference admission and a minimum of two nights lodging at the =20
> conference hotel. Additional hotel nights and travel assistance are =20=

> possible, depending on the number of presentations given and type of =20=

> assistance needed.
>
> To submit a presentation proposal, please send an email to =20
> submissions AT apachecon DOT com containing the following =20
> information in plaintext (no attachments, please):
>
> 1. Your full name, title, and organization
>
> 2. Contact information, including your address
>
> 3. The name of your proposed session (keep your title simple and =20
> relevant to the topic)
>
> 4. The technical category of the intended presentation (Cassandra/=20
> NoSQL; Content Technologies; (Java) Enterprise Development; Felix/=20
> OSGi; Geronimo; Hadoop + friends/Cloud Computing; Lucene, Mahout + =20
> friends/Search; Tomcat; or Tuscany)
>
> 5. The classification for each presentation (Servers, Cloud, or =20
> Innovation) =96 some presentations may have more than one theme (e.g., =
=20
> a next-generation server can be classified both as "Servers" and =20
> "Innovation"
>
> 6. The intended audience level (beginner, intermediate, advanced)
>
> 7. A 75-200 word overview of your presentation
>
> 8. A 100-200-word speaker bio that includes prior conference =20
> speaking or related experience
>
> 9. Feedback or references (with contact information) on =20
> presentations given within the last three years
>
> To be considered, proposals must be received by Friday, 28 May 2010 =20=

> at midnight Pacific Time. Please email any questions regarding =20
> proposal submissions to cfp AT apachecon DOT com.
>
> Technical Tracks Key Dates
>
> 23 April 2010: Call For Participation Open
> 28 May 2010: Call For Participation Closes
> 11 June 2010: Speaker Acceptance/Rejection Notification
> 1-5 November 2010: ApacheCon NA 2010
> We look forward to seeing you in Atlanta!
>
> For the ApacheCon Planning team,
> Sally Khudairi, Program Lead


--Apple-Mail-8-775241319--

From general-return-1604-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 23:41:21 2010
Return-Path: <general-return-1604-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 10202 invoked from network); 1 Jun 2010 23:41:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 23:41:21 -0000
Received: (qmail 89900 invoked by uid 500); 1 Jun 2010 23:41:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89788 invoked by uid 500); 1 Jun 2010 23:41:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89779 invoked by uid 99); 1 Jun 2010 23:41:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 23:41:19 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 23:41:13 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:Received:Received:From:To:CC:
   Subject:Thread-Topic:Thread-Index:Date:Message-ID:
   References:In-Reply-To:Accept-Language:Content-Language:
   X-MS-Has-Attach:X-MS-TNEF-Correlator:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version:
   Return-Path:X-OriginalArrivalTime;
  b=jBJVm4qNC/V0emYD1aNcauFiZzbqgp+Lip3nney4ILa9CcpUl3/RQrGq
   MJboxscDohHNOh0104bT7YeuTshKStbkws3ARYtJNKVVIRVrc6p3e3AAv
   Im5oClIsHMlQhCc;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1275435673; x=1306971673;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20[VOTE]=20Chukwa=20as=20TLP?|Date:=20Tue
   ,=201=20Jun=202010=2023:40:50=20+0000|Message-ID:=20<FCE8
   6F4B-BF56-49C9-BBC0-1A77DC4C0D66@linkedin.com>|To:=20"<ge
   neral@hadoop.apache.org>"=20<general@hadoop.apache.org>
   |CC:=20Bill=20Graham=20<billgraham@gmail.com>,=20Jiaqi=20
   Tan=20<tanjiaqi@gmail.com>,=20Jerome=0D=0A=20Boulon=20<jb
   oulon@gmail.com>,=20Ariel=20Rabkin=20<asrabkin@gmail.com>
   |MIME-Version:=201.0|Content-Transfer-Encoding:=20quoted-
   printable|Content-ID:=20<b5e6d08a-7414-4131-910a-a3543d64
   40b3>|In-Reply-To:=20<C82AC9A5.8C32%eyang@yahoo-inc.com>
   |References:=20<C82AC9A5.8C32%eyang@yahoo-inc.com>;
  bh=PNG3rY8nUJDvy8p9N5Xq8QiMHSsrXvZbChzJRap7X0A=;
  b=vocRsJdHsB/RYwBAxwEsRASe5Jkkoy0OWEKb/8v98A56ggVk8WQePHec
   ONNFuq2rvoEfUDpwU2Rp0qzc6240A4aTXrtxOih0O+KgQv1rDokiomEyQ
   hjoXNjRutp/p0BY;
X-IronPort-AV: E=Sophos;i="4.53,343,1272870000"; 
   d="scan'208";a="12823054"
Received: from esv4-exctest.linkedin.biz ([172.18.46.60]) by CORP-MAIL.linkedin.biz with Microsoft SMTPSVC(6.0.3790.3959);
	 Tue, 1 Jun 2010 16:40:52 -0700
Received: from ESV4-CAS01.linkedin.biz (172.18.46.140) by
 esv4-exctest.linkedin.biz (172.18.46.60) with Microsoft SMTP Server (TLS) id
 14.0.682.1; Tue, 1 Jun 2010 16:40:51 -0700
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi; Tue, 1 Jun 2010 16:40:51
 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
CC: Bill Graham <billgraham@gmail.com>, Jiaqi Tan <tanjiaqi@gmail.com>, Jerome
 Boulon <jboulon@gmail.com>, Ariel Rabkin <asrabkin@gmail.com>
Subject: Re: [VOTE] Chukwa as TLP?
Thread-Topic: [VOTE] Chukwa as TLP?
Thread-Index: AcsB0W7dgQlmbfPShEWXCOQcnrztAAATRtEA
Date: Tue, 1 Jun 2010 23:40:50 +0000
Message-ID: <FCE86F4B-BF56-49C9-BBC0-1A77DC4C0D66@linkedin.com>
References: <C82AC9A5.8C32%eyang@yahoo-inc.com>
In-Reply-To: <C82AC9A5.8C32%eyang@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Content-Type: text/plain; charset="us-ascii"
Content-ID: <b5e6d08a-7414-4131-910a-a3543d6440b3>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginalArrivalTime: 01 Jun 2010 23:40:52.0348 (UTC) FILETIME=[DF2A33C0:01CB01E3]
X-Virus-Checked: Checked by ClamAV on apache.org


On Jun 1, 2010, at 2:28 PM, Eric Yang wrote:

> Please vote as to whether you think Chukwa should become a top-level
> Apache project.

Wow.  I thought Chukwa was basically dead, much less prepared to be a TLP.=
=20

From general-return-1605-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 23:44:08 2010
Return-Path: <general-return-1605-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 13041 invoked from network); 1 Jun 2010 23:44:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 23:44:08 -0000
Received: (qmail 97571 invoked by uid 500); 1 Jun 2010 23:44:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97520 invoked by uid 500); 1 Jun 2010 23:44:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97508 invoked by uid 99); 1 Jun 2010 23:44:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 23:44:07 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=AWL,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of asrabkin@gmail.com designates 74.125.83.48 as permitted sender)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 23:44:02 +0000
Received: by gwaa18 with SMTP id a18so52841gwa.35
        for <general@hadoop.apache.org>; Tue, 01 Jun 2010 16:43:41 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:cc:content-type
         :content-transfer-encoding;
        bh=reY7CW5yC+MzFHxw9UTQ33s0R+74zrDDWZSsomgG1eI=;
        b=o/NZesOnVcFIGwY1ENLbwYZ8tkO7/OsEo9Bug1sn03jaYxGuOZO6rTDKEQedS2Ezez
         98AedNM/40+eWfawifE/OBOb9rjC/s+R+XWF02q4HXyG8Y99HQWur/vwsSJlGapT+nCU
         lc5Vi74P5fU9nP5DP+S7Hx3tZgEWCUgr4wzV4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type:content-transfer-encoding;
        b=WsA8eU4KHZrWytaLrksKTO3rhY3vDbVm4RtTrEnrZS+fWvXtjYts80iteDKpfMHs0s
         6BVdqJ5NUXLF50PkhlbeKBcQJTnuKKpWj+75wtHuu6N9jKw5fENuX24o4fEZL4c7S1Xt
         qLI5bJPPRDA2WYccQkksrDOpJqIIkKG++4fjs=
MIME-Version: 1.0
Received: by 10.224.96.78 with SMTP id g14mr2974039qan.117.1275435821347; Tue, 
	01 Jun 2010 16:43:41 -0700 (PDT)
Received: by 10.229.100.131 with HTTP; Tue, 1 Jun 2010 16:43:41 -0700 (PDT)
In-Reply-To: <FCE86F4B-BF56-49C9-BBC0-1A77DC4C0D66@linkedin.com>
References: <C82AC9A5.8C32%eyang@yahoo-inc.com>
	<FCE86F4B-BF56-49C9-BBC0-1A77DC4C0D66@linkedin.com>
Date: Tue, 1 Jun 2010 16:43:41 -0700
Message-ID: <AANLkTikqrmcL3Bw9rD40Moa1ap3ZwkKNELsuK2VeBPtr@mail.gmail.com>
Subject: Re: [VOTE] Chukwa as TLP?
From: Ariel Rabkin <asrabkin@gmail.com>
To: Allen Wittenauer <awittenauer@linkedin.com>
Cc: "<general@hadoop.apache.org>" <general@hadoop.apache.org>, Bill Graham <billgraham@gmail.com>, 
	Jiaqi Tan <tanjiaqi@gmail.com>, Jerome Boulon <jboulon@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

We're certainly far from dead.  It's in use at several sites; we're
getting significant code contributions from outside the original
developer group; we've done several releases, and we're seeing
steadily more user mailing list traffic.

But I agree the jump to TLP is probably ill-advised, and would vote
for incubation instead.

--Ari

On Tue, Jun 1, 2010 at 4:40 PM, Allen Wittenauer
<awittenauer@linkedin.com> wrote:
>
> On Jun 1, 2010, at 2:28 PM, Eric Yang wrote:
>
>> Please vote as to whether you think Chukwa should become a top-level
>> Apache project.
>
> Wow. =A0I thought Chukwa was basically dead, much less prepared to be a T=
LP.
>



--=20
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department

From general-return-1606-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 02:17:33 2010
Return-Path: <general-return-1606-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58949 invoked from network); 2 Jun 2010 02:17:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 02:17:33 -0000
Received: (qmail 27306 invoked by uid 500); 2 Jun 2010 02:17:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27183 invoked by uid 500); 2 Jun 2010 02:17:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27175 invoked by uid 99); 2 Jun 2010 02:17:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 02:17:32 +0000
X-ASF-Spam-Status: No, hits=-0.8 required=10.0
	tests=AWL,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of kiyoshi.mizumaru@gmail.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 02:17:26 +0000
Received: by gyf1 with SMTP id 1so5291763gyf.35
        for <general@hadoop.apache.org>; Tue, 01 Jun 2010 19:17:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=Xt2C43j43UStle8VdpqAsXRsz3vFCxl6CKV3w4UYyBk=;
        b=AOXfRlb6RoPHQdzj9ODIfXNrPxp9TEbHE5ar30g7vffvyynNjQQktgnEL6CoPjd2Lb
         NpT0ff/xbAlOHd64pV0HXOl2NffQlOKnIJTXFEZbAHJRMfCJjzuC2k5dXJ3AXuDAEE5b
         q7GIEoLvWH8KHU3Qga9YvaKtWKx8SniC2b3jM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=k/a3H8SMYgwjm4GQd9V5g0qDi/DMOSzYkoPsTzNtdHQwFe1L6Dbot0lhMEhTf5Pds8
         BMQg202EpY3+16244w4T2P2sFVVWdhn3mfMAEVZuJEIAk9aLhSirK59ipgQILIKyUPbC
         OIWaH9L878eoQL0TgOnDMfsR2IpR0EGiCKIII=
MIME-Version: 1.0
Received: by 10.150.48.9 with SMTP id v9mr7614159ybv.288.1275445025567; Tue, 
	01 Jun 2010 19:17:05 -0700 (PDT)
Received: by 10.151.142.18 with HTTP; Tue, 1 Jun 2010 19:17:05 -0700 (PDT)
In-Reply-To: <394531.3151.qm@web114518.mail.gq1.yahoo.com>
References: <394531.3151.qm@web114518.mail.gq1.yahoo.com>
Date: Wed, 2 Jun 2010 11:17:05 +0900
Message-ID: <AANLkTimXJnTmdSyg8alb0l6ggpmg2aX3K9rx6EfC2Oup@mail.gmail.com>
Subject: Re: "No datanode to stop " message on stop-dfs.sh
From: Kiyoshi Mizumaru <kiyoshi.mizumaru@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

> Any idea what can cause issue like this?

I saw this message when I forgot to switch a user to who had started
the Hadoop/HDFS instance and invoked stop-dfs.sh from different user.

# sorry for my poor English.


On Wed, Jun 2, 2010 at 5:19 AM, C J <c.joshi26@yahoo.com> wrote:
> Hi,
>
> I have a brand new setup of hadoop machine cluster with 50 machines. I see some weird issues come up with the cluster with time....Things run just fine for a few days and then when I try to run stop-dfs.sh, it says
>
> no namenode to stop
> hadoop-07: no data node to stop
> hadoop-08: no data node to stop
> .
> .
> .
> hadoop-03: no secondarynamenode to stop
>
> When I go to these machines, the data node is actually running.
>
> Any idea what can cause issue like this? The last time it happened I killed all the running datanodes manually and then started the dfs. It started fine. After that even the stop-dfs.sh worked as expected. But now it got back to the same situation again.
>
> One more thing I see a lot of left over running "Child" tasks from task attempts on these machines.
>
> Appreciate any help.
>
> Thanks,
> C
>
>
>

From general-return-1607-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 04:22:21 2010
Return-Path: <general-return-1607-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85014 invoked from network); 2 Jun 2010 04:22:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 04:22:21 -0000
Received: (qmail 93927 invoked by uid 500); 2 Jun 2010 04:22:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93800 invoked by uid 500); 2 Jun 2010 04:22:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 30877 invoked by uid 99); 2 Jun 2010 02:22:29 -0000
X-ASF-Spam-Status: No, hits=-0.1 required=10.0
	tests=AWL,FREEMAIL_FROM,FREEMAIL_REPLY,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of tanjiaqi@gmail.com designates 209.85.212.176 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:cc:content-type
         :content-transfer-encoding;
        bh=7yS2rxoSdrkFdoQC79gG9jdm1ezGQL1Bi20maLHxWZ4=;
        b=ZDNWdRmk6T6PLUtT4v6tr/kSxPy8HA2CvyRhJ4AYHTwwC1wgwo9EVWu91YoVVDqqqJ
         cNlKe/HUyYsxlEVMPYDUKNS2megg+7RLKZT8AvaDsns5GL5hX0UOxmFJ9M4eDhKaXAbN
         cX88aoH7pAur6q/M6kPp7rsywTTo86NYc4bl4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :cc:content-type:content-transfer-encoding;
        b=dbMzvZc+MX6Jg9RSyZzsWgWmSWdx1YEHSZFbXxVyNITUO8ERunI84hBJaHZ9sgIvvg
         mUOryyR9A1ABQK4hU2aKUE/HYEnk+8dwkCxOi4lTHLfMOX5v/b1pi9/+AruJErQMm+uq
         tOQWLpSN3bn7kSRnr17jYh8fqGbPho203AmRw=
MIME-Version: 1.0
In-Reply-To: <AANLkTikqrmcL3Bw9rD40Moa1ap3ZwkKNELsuK2VeBPtr@mail.gmail.com>
References: <C82AC9A5.8C32%eyang@yahoo-inc.com> <FCE86F4B-BF56-49C9-BBC0-1A77DC4C0D66@linkedin.com> 
	<AANLkTikqrmcL3Bw9rD40Moa1ap3ZwkKNELsuK2VeBPtr@mail.gmail.com>
From: Jiaqi Tan <tanjiaqi@gmail.com>
Date: Wed, 2 Jun 2010 10:21:44 +0800
Message-ID: <AANLkTilhJoipQanxizfkbghOunf0A8MLeqvK7sj3PjLU@mail.gmail.com>
Subject: Re: [VOTE] Chukwa as TLP?
To: Ariel Rabkin <asrabkin@gmail.com>
Cc: Allen Wittenauer <awittenauer@linkedin.com>, 
	"<general@hadoop.apache.org>" <general@hadoop.apache.org>, Bill Graham <billgraham@gmail.com>, 
	Jerome Boulon <jboulon@gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

I would also vote for incubation.

Jiaqi

On Wed, Jun 2, 2010 at 7:43 AM, Ariel Rabkin <asrabkin@gmail.com> wrote:
> We're certainly far from dead. =C2=A0It's in use at several sites; we're
> getting significant code contributions from outside the original
> developer group; we've done several releases, and we're seeing
> steadily more user mailing list traffic.
>
> But I agree the jump to TLP is probably ill-advised, and would vote
> for incubation instead.
>
> --Ari
>
> On Tue, Jun 1, 2010 at 4:40 PM, Allen Wittenauer
> <awittenauer@linkedin.com> wrote:
>>
>> On Jun 1, 2010, at 2:28 PM, Eric Yang wrote:
>>
>>> Please vote as to whether you think Chukwa should become a top-level
>>> Apache project.
>>
>> Wow. =C2=A0I thought Chukwa was basically dead, much less prepared to be=
 a TLP.
>>
>
>
>
> --
> Ari Rabkin asrabkin@gmail.com
> UC Berkeley Computer Science Department
>

From general-return-1608-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 08:03:57 2010
Return-Path: <general-return-1608-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 40236 invoked from network); 2 Jun 2010 08:03:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 08:03:57 -0000
Received: (qmail 53214 invoked by uid 500); 2 Jun 2010 08:03:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52982 invoked by uid 500); 2 Jun 2010 08:03:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52974 invoked by uid 99); 2 Jun 2010 08:03:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 08:03:55 +0000
X-ASF-Spam-Status: No, hits=-0.4 required=10.0
	tests=AWL,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [64.78.17.18] (HELO EXHUB018-3.exch018.msoutlookonline.net) (64.78.17.18)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 08:03:48 +0000
Received: from EXVMBX018-1.exch018.msoutlookonline.net ([64.78.17.47]) by
 EXHUB018-3.exch018.msoutlookonline.net ([64.78.17.18]) with mapi; Wed, 2 Jun
 2010 01:03:27 -0700
From: Scott Carey <scott@richrelevance.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 2 Jun 2010 01:01:44 -0700
Subject: Re: "No datanode to stop " message on stop-dfs.sh
Thread-Topic: "No datanode to stop " message on stop-dfs.sh
Thread-Index: AcsCKhSXtkNq3WHHT6+qA784W0OnKQ==
Message-ID: <77207064-AAA9-49EF-AA16-D926139BFCC2@richrelevance.com>
References: <394531.3151.qm@web114518.mail.gq1.yahoo.com>
In-Reply-To: <394531.3151.qm@web114518.mail.gq1.yahoo.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hadoop by default places its .pid files in /tmp.

On most linux distros, a script will come along and delete stuff from there=
 that hasn't been modified in a while.  The startup / shutdown scripts are =
merely looking for these pid files.  The solution is to move the temp files=
 to somewhere sane.  Maybe /var/run/hadoop/.




On Jun 1, 2010, at 1:19 PM, C J wrote:

> Hi,
>=20
> I have a brand new setup of hadoop machine cluster with 50 machines. I se=
e some weird issues come up with the cluster with time....Things run just f=
ine for a few days and then when I try to run stop-dfs.sh, it says=20
>=20
> no namenode to stop
> hadoop-07: no data node to stop
> hadoop-08: no data node to stop
> .
> .
> .
> hadoop-03: no secondarynamenode to stop
>=20
> When I go to these machines, the data node is actually running.=20
>=20
> Any idea what can cause issue like this? The last time it happened I kill=
ed all the running datanodes manually and then started the dfs. It started =
fine. After that even the stop-dfs.sh worked as expected. But now it got ba=
ck to the same situation again.
>=20
> One more thing I see a lot of left over running "Child" tasks from task a=
ttempts on these machines.
>=20
> Appreciate any help.
>=20
> Thanks,
> C
>=20
>=20


From general-return-1609-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 17:00:04 2010
Return-Path: <general-return-1609-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24733 invoked from network); 2 Jun 2010 17:00:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 17:00:03 -0000
Received: (qmail 7499 invoked by uid 500); 2 Jun 2010 17:00:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7431 invoked by uid 500); 2 Jun 2010 17:00:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7418 invoked by uid 99); 2 Jun 2010 17:00:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 17:00:01 +0000
X-ASF-Spam-Status: No, hits=0.8 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 209.85.222.180 as permitted sender)
Received: from [209.85.222.180] (HELO mail-pz0-f180.google.com) (209.85.222.180)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 16:59:57 +0000
Received: by pzk10 with SMTP id 10so3339315pzk.20
        for <general@hadoop.apache.org>; Wed, 02 Jun 2010 09:59:37 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=3gydMqd2oNT6hGFxmddmWPeNUCRVXzoFhQTsOXs0zxY=;
        b=qwEZzvlW4tuZWOwne3qZ9DXrtm2+geghpTTWqcbeA5e2PzcZs9APSLv6YP/iEhabw4
         d9fzXufiB1fAoVZnNy1T2W26d2cfGQ8AoxXYw3izXpRQdN8JHvZy4Ixzk+ojQ04e5eyW
         p7Hn4EmXUTqIrIAsf8vPaUcMm9lCLQlISFCfI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=nND27mUuAn4iOUCy/b7W5VmhbHqGEWg9M/PDSaMhWQjJsDh8ef5L8L8upKm9D8OX/8
         AAu5bchVz7FITo3xpi3RzUqKL+JtpGhcWyg6wP9wvUPBdbTvw86YnQkgnYt0j+tR6kJw
         FxY4toe9maTMoapNPkFFA5dsdWxrSTCvELmh0=
MIME-Version: 1.0
Received: by 10.142.74.8 with SMTP id w8mr5113530wfa.274.1275497972438; Wed, 
	02 Jun 2010 09:59:32 -0700 (PDT)
Received: by 10.114.174.19 with HTTP; Wed, 2 Jun 2010 09:59:31 -0700 (PDT)
In-Reply-To: <AANLkTinB9Sp7FAQqpD5WBu206G3W-KVQ7KP3gAbf7Kk4@mail.gmail.com>
References: <h2m4aa34eb71005071034ja97e2f4ckca2a016efedee897@mail.gmail.com>
	<w2q7c962aed1005091515y4965fb9dpa345178a39fc9c45@mail.gmail.com>
	<w2p7c962aed1005091526ne7a51896u7165b7019322a0a6@mail.gmail.com>
	<k2i4aa34eb71005092150sdcc4ab8fnac9b152b2788ce38@mail.gmail.com>
	<EA6A5E07-30C9-4209-945D-413DC9F45090@linkedin.com>
	<AANLkTin5amvNCn9apT64DwJNAEmBZWm8v6k1X-6x9_qz@mail.gmail.com>
	<AANLkTil4M5IetREXm0Paj13gxC4zS6U_mXbxHNf_QjJF@mail.gmail.com>
	<COL117-W64E02417250AA7DC1C1E828FF90@phx.gbl>
	<AANLkTimgAcxJr8xOkx_Na6vtOvb7APW1tGPTKNzStEX2@mail.gmail.com>
	<AANLkTinB9Sp7FAQqpD5WBu206G3W-KVQ7KP3gAbf7Kk4@mail.gmail.com>
Date: Wed, 2 Jun 2010 09:59:31 -0700
Message-ID: <AANLkTil5r5AOyzK2chatetwBt-JIcYSLdaXL0hxFiHAe@mail.gmail.com>
Subject: Re: Hadoop support for hbase
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636e90bf78fb04f04880f0335

--001636e90bf78fb04f04880f0335
Content-Type: text/plain; charset=ISO-8859-1

Hello folks,

I created a branch for doing the append/sync support for Hadoop 0.20. You
can fetch the branch via

http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-append/

If you feel that there are some JIRAS that need to go into this branch,
please update the fix-version of those JIRAS with the tag
"0.20-append</jira/secure/IssueNavigator.jspa?reset=true&mode=hide&sorter/order=DESC&sorter/field=priority&resolution=-1&pid=12310942&fixfor=12315103>
".

thanks,
dhruba

On Mon, May 10, 2010 at 11:35 PM, Dhruba Borthakur <dhruba@gmail.com> wrote:


> @Allen: we are definitely behind 0.21 release. Tom White is guiding that
> release and most developers are committed to removing blockers for that
> release. Todd rightly mentions that the work being done for 0.20 benefits
> 0.21 as well.
>
> @Jay: Thanks for summing it up so well. I completely agree with your
> viewpoint.
>
> thanks
> dhruba
>
>
> On Mon, May 10, 2010 at 2:06 PM, Jay Booth <jaybooth@gmail.com> wrote:
>
>> Given that the 0.20-append branch pretty much already exists
>> unofficially, via IRC, IM and email forwarded patchsets, it seems like
>> giving it an official home is just recognizing the status quo.
>> Especially since 0.21 probably won't be getting rolled out into
>> production everywhere the first day it's officially released.  If the
>> work's going on anyways, I don't see how giving people a shared home
>> hurts matters, if anything it gives them a better shared touchpoint
>> for forward-porting bugfixes to 0.21.
>>
>> A case could be made that by making it more painful to run
>> 0.20-append, more momentum is created towards 0.21 but since Tom is
>> already on top of 21 and seemingly doing an excellent job, and since
>> the HBase community will probably be some of the first people to move
>> to 0.21 anyways, I don't see why having 0.20-append will damage 0.21's
>> momentum at this point.
>>
>>
>>
>> On Mon, May 10, 2010 at 4:21 PM, Michael Segel
>> <michael_segel@hotmail.com> wrote:
>> >
>> >
>> >
>> >> From: todd@cloudera.com
>> >> Date: Mon, 10 May 2010 10:45:13 -0700
>> >> Subject: Re: Hadoop support for hbase
>> >> To: general@hadoop.apache.org
>> >>
>> >
>> >> > The above is a fallacious setup.  How does a branch in 0.20 detract
>> >> > from the 0.21 momentum (The append feature that we'd work on in 0.20
>> >> > branch has little relation to how append works in 0.21).
>> >>
>> >> For what it's worth, though, the majority of the size of the 0.20
>> >> append patch is made up of additional unit tests. I have started
>> >> forward-porting these new tests to the trunk append and it's already
>> >> exposed a number of bugs. So while it's tempting to say that the 0.20
>> >> append is "wasted effort", it really is benefiting the entire
>> >> community and the 0.21 release as well.
>> >>
>> >> -Todd
>> >>
>> >
>> > Sometimes you have to slow down to go faster.
>> >
>> >
>> >
>> > _________________________________________________________________
>> > Hotmail is redefining busy with tools for the New Busy. Get more from
>> your inbox.
>> >
>> http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_2
>>
>
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
>



-- 
Connect to me at http://www.facebook.com/dhruba

--001636e90bf78fb04f04880f0335--

From general-return-1610-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 05:45:38 2010
Return-Path: <general-return-1610-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97681 invoked from network); 3 Jun 2010 05:45:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 05:45:38 -0000
Received: (qmail 95694 invoked by uid 500); 3 Jun 2010 05:45:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95506 invoked by uid 500); 3 Jun 2010 05:45:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95492 invoked by uid 99); 3 Jun 2010 05:45:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 05:45:34 +0000
X-ASF-Spam-Status: No, hits=1.6 required=10.0
	tests=AWL,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 05:45:29 +0000
Received: by vws1 with SMTP id 1so3822223vws.35
        for <general@hadoop.apache.org>; Wed, 02 Jun 2010 22:45:08 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.224.79.133 with SMTP id p5mr4323156qak.220.1275543907918; Wed, 
	02 Jun 2010 22:45:07 -0700 (PDT)
Received: by 10.229.94.203 with HTTP; Wed, 2 Jun 2010 22:45:07 -0700 (PDT)
In-Reply-To: <AANLkTikZN5RFeP7veaXeZM2ws4ZNBXRBBMBEnMlDOw-3@mail.gmail.com>
References: <AANLkTimLB-YNWnMwa93YL8Lpg9hJLxrCKbYL3YUr6de8@mail.gmail.com>
	<66BC29ED-35C1-4916-BDFE-C768810C6EA6@mac.com>
	<AANLkTimCkqddJ521jIPp0dVHC__vZ2PcavAIrNl334L0@mail.gmail.com>
	<4BFBFAC3.2040807@apache.org>
	<AANLkTilhcUYlFRwc1vc7-sFybjUw5c_uPosptIGb-Gn1@mail.gmail.com>
	<4BFCF6CF.2000605@apache.org>
	<AANLkTilLOmfMwuIFc2kzsXMQ7yhbXtrA0jhb3pigTEht@mail.gmail.com>
	<AANLkTikuyUulssueBs8EQJNeJJN3_NFa7GfflAaeAFgH@mail.gmail.com>
	<AANLkTikZN5RFeP7veaXeZM2ws4ZNBXRBBMBEnMlDOw-3@mail.gmail.com>
Date: Wed, 2 Jun 2010 22:45:07 -0700
Message-ID: <AANLkTilCjGXhjulK4WdKr0cg1vN9G4zzD7UUH89rQwtw@mail.gmail.com>
Subject: Re: [DISCUSSION] Proposal for making core Hadoop changes
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00c09f97250086e2fa048819b520

--00c09f97250086e2fa048819b520
Content-Type: text/plain; charset=ISO-8859-1

Sure, each project can choose to use the framework in the way they see fit
on Launchpad. I wanted to call out their use of metadata as being
particularly nice. We may want to consider similar fields and applications
of those fields for HEPs.

On Tue, Jun 1, 2010 at 2:28 PM, Eli Collins <eli@cloudera.com> wrote:

> Hey Jeff,
>
> Blueprints (it's a launchpad thing) is more of an issue tracking
> system (launchpad doesn't put features/enhancements in their bug
> database), eg drizzle has lots of blueprints, and blueprints for
> cleaning up code, adding config flags, etc. We'll use jira for that
> kind of stuff, the HEP is for larger stuff that needs more upfront
> discussion.
>
> Thanks,
> Eli
>
> On Mon, May 31, 2010 at 10:16 AM, Jeff Hammerbacher <hammer@cloudera.com>
> wrote:
> > A far more lightweight example of multi-issue feature planning in an open
> > source project comes from Drizzle and their "blueprints":
> > https://blueprints.launchpad.net/drizzle.
> >
> > Each "spec" has a drafter, an approver, and an assignee; declares the
> other
> > specs on which it depends; points to the relevant branches in the source
> > tree and issues in the issue tracker; and has a priority, definition
> state,
> > and implementation state.
> >
> > I don't know how it's working out for them in practice, but on paper it
> > looks quite nice.
> >
> > On Wed, May 26, 2010 at 9:13 AM, Eli Collins <eli@cloudera.com> wrote:
> >
> >> > No, but I'd estimate the cost of merging at 1-2 days work a week just
> to
> >> > pull in the code *and identify why the tests are failing*. Git may be
> >> better
> >> > at merging in changes, but if Hadoop doesn't work on my machine after
> the
> >> > merge, I need to identify whether its my code, the merged code, some
> >> machine
> >> > quirk, etc. It's the testing that is the problem for me, not the
> >> > merge effort. That's the Hadoop own tests any my own functional test
> >> suites,
> >> > the ones that bring up clusters and push work through. Those are the
> >> > troublespots, as they do things that hadoop's own tests don't do, like
> as
> >> > for all the JSP pages.
> >>
> >> I've lived off a git branch of common/hdfs for half a year with a big
> >> uncommitted patch, it's no where near 1-2 days of effort per week to
> >> merge in changes from trunk. If the tests are passing on trunk, and
> >> they fail after your merge then those are real test failures due to
> >> your change (and therefore should require effort). The issues with
> >> your internal tests failing due to changes on trunk is the same
> >> whether you merge or you just do an update - you have to update before
> >> checking in the patch anyway - so that issue is about the state of
> >> trunk when you merge or update, rather than about being on a branch.
> >>
> >> >
> >> >> Might find the
> >> >> following interesting:
> >> >> http://incubator.apache.org/learn/rules-for-revolutionaries.html
> >> >
> >> > There's a long story behind JDD's paper, I'm glad you have read it, it
> >> does
> >> > lay out what is effectively the ASF process for effecting significant
> >> change
> >> > -but it doesn't imply that's the only process for having changes.
> >> >
> >>
> >> Just to be clear I don't mean imply that branches are the only process
> >> for making changes. Interesting that this is considered the effective
> >> ASF process, it hasn't seemed to me that recent big features on hadoop
> >> have used it, only one I'm aware of that was done on a branch was
> >> append.
> >>
> >> > I think gradual evolution in trunk is good, it lets people play with
> >> what's
> >> > coming in. Having lots of separate branches and everyone's private
> >> release
> >> > being a merge of many patches that you choose is bad.
> >>
> >> Agreed.  Personally I don't think people should release from branches.
> >> And in practice I don't think you'll see lots of branches, people can
> >> and would still develop on trunk. Getting changes merged from a branch
> >> back to trunk before the whole branch is merged is a good thing, the
> >> whole branch may never be merged and that's OK too. Branches are a
> >> mechanism, releases are policy.
> >>
> >> Thanks,
> >> Eli
> >>
> >
>

--00c09f97250086e2fa048819b520--

From general-return-1611-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 15:53:23 2010
Return-Path: <general-return-1611-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96061 invoked from network); 3 Jun 2010 15:53:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 15:53:22 -0000
Received: (qmail 86741 invoked by uid 500); 3 Jun 2010 15:53:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86656 invoked by uid 500); 3 Jun 2010 15:53:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86648 invoked by uid 99); 3 Jun 2010 15:53:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 15:53:21 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of fredwang222@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 15:53:13 +0000
Received: by pwj10 with SMTP id 10so204941pwj.35
        for <general@hadoop.apache.org>; Thu, 03 Jun 2010 08:52:52 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=/YxnAKp1z6jsOm5B4HMd+CQydzryCb2AmA5iE4Pxcfk=;
        b=wy9VVCqVHEvHbX28kNgkWWeamFyB+qoTMiTEh0iKO0UkqzwR0PhoINt/ECefwWPzGc
         IIOIA50LPN9cNOPIs/446ABlc/ipfidtUjvqYL8Vfwd4+m9SSTTD07XEWZ+7ba2q6HSt
         BCoCFI0mkcCJR9GGVQIBLFQ3J7TlIKIay4D3A=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=J6+DfTCnwPUgz5//t6UClR3aqnuXo1uFYY444pNQItNojCHJ1z8ooL6x7LCkJ1O3ux
         Aq2GwO7flDaHbu8Eo0a/6ehQTe1M0oz0Gz839MAInQThJRVe6cgpmMKpbMx0Z15Bzslv
         Ye3m6out6I/xUc0wd+p+601LRxuKMS/3ZvZNQ=
MIME-Version: 1.0
Received: by 10.142.8.13 with SMTP id 13mr6730754wfh.210.1275580371881; Thu, 
	03 Jun 2010 08:52:51 -0700 (PDT)
Received: by 10.142.128.6 with HTTP; Thu, 3 Jun 2010 08:52:51 -0700 (PDT)
Date: Thu, 3 Jun 2010 23:52:51 +0800
Message-ID: <AANLkTil7-7JcAlw49Cd5pNWpjg2S5WpqucFJjgY1EB8s@mail.gmail.com>
Subject: how I can do to configure/start a hadoop cluster(pseudo distributed) 
	with the last hadoop trunk code(I have generated .jar file for common, hdfs 
	and mapred)
From: fred wang <fredwang222@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502c09cf2d12d04882232a5
X-Virus-Checked: Checked by ClamAV on apache.org

--00504502c09cf2d12d04882232a5
Content-Type: text/plain; charset=ISO-8859-1

All,
  I have followed the instructions on
http://wiki.apache.org/hadoop/EclipseEnvironment to download the latest
trunk source code and build .jar for common, hdfs and mapred. but how should
I proceed to configure and start a hadoop cluster(psudo distributed)  with
these latest .jar? I knew how to configured/start the hadoop cluster with
formal hadoop package(hadoop-*.tar.gz with all stuff of common, hdfs and
mapred there). I  googled but didn't find the related information, most
information I got after compile is to run unit test.

  Can anyone help?  Thanks for the help.

Best Regards,
Fred

--00504502c09cf2d12d04882232a5--

From general-return-1612-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 23:04:42 2010
Return-Path: <general-return-1612-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47749 invoked from network); 3 Jun 2010 23:04:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 23:04:42 -0000
Received: (qmail 93451 invoked by uid 500); 3 Jun 2010 23:04:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93394 invoked by uid 500); 3 Jun 2010 23:04:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93383 invoked by uid 99); 3 Jun 2010 23:04:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 23:04:40 +0000
X-ASF-Spam-Status: No, hits=-0.6 required=10.0
	tests=AWL,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 23:04:35 +0000
Received: from localhost (wlanvpn-snve-246-61.corp.yahoo.com [172.21.148.61])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o53N3wfO033984
	for <general@hadoop.apache.org>; Thu, 3 Jun 2010 16:03:59 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=date:from:to:subject:message-id:references:mime-version:
	content-type:content-disposition:in-reply-to:x-organization:user-agent;
	b=FSqfiqytk6SxLAQnZ04E/U7gXP2C0o2kCr54F84dUWYau+DP8zT/Ej+ceXLoi/gP
Date: Thu, 3 Jun 2010 16:03:58 -0700
From: Konstantin Boudnik <cos@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: Last call: ApacheCon TechTalks CFP closes today
Message-ID: <20100603230358.GC47625@wlanvpn-snve-246-61.corp.yahoo.com>
References: <9DD2220F-023D-4E6B-88C5-F08B184B6102@apache.org>
 <BD9C6F51-44A8-45A3-90C9-C12DA90E4601@yahoo-inc.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature"; boundary="dkEUBIird37B8yKS"
Content-Disposition: inline
In-Reply-To: <BD9C6F51-44A8-45A3-90C9-C12DA90E4601@yahoo-inc.com>
X-Organization: Yahoo! Hadoop (Grid computing)
User-Agent: Mutt/1.5.20 (2009-06-14)

--dkEUBIird37B8yKS
Content-Type: text/plain; charset=koi8-r
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Hi Owen

Do you think it'd be a good idea to submit Herriot (large scale testing
framework we were working on for some time; see HADOOP-6332 for more info) =
to
this conference?

Please advise, thanks.
  Cos

On Tue, Jun 01, 2010 at 04:33PM, Owen O'Malley wrote:
> We've decided to extend the deadline a week to this Friday. As a =20
> reminder, all speakers get two nights at the conference hotel and free =
=20
> registration at ApacheCon.
>=20
> -- Owen
>=20
> On May 28, 2010, at 2:40 PM, Owen O'Malley wrote:
>=20
> > All,
> >    As a reminder the CFP for proposals closes today.  The CFP is at htt=
p://blogs.apache.org/conferences/date/20100428
> >
> > -- Owen
> >
> > ApacheCon North America 2010
> > 1-5 November 2010 -- Westin Peachtree in Atlanta
> >
> > Technical Tracks: Call For Participation
> > All submissions must be received by Friday, 28 May 2010 at midnight =20
> > Pacific Time.
> > The official conference, trainings, and expo of The Apache Software =20
> > Foundation (ASF) returns to Atlanta this November, with dozens of =20
> > technical, business, and community-focused sessions at the beginner, =
=20
> > intermediate, and advanced levels.
> >
> > Over the past decade, the ASF has gone from strength to strength, =20
> > developing and shepherding nearly 150 Top-Level Projects and new =20
> > initiatives in the Apache Incubator and Labs. This year's ApacheCon =20
> > celebrates how Apache technologies have sparked creativity, =20
> > challenged processes, streamlined development, improved =20
> > collaboration, launched businesses, bolstered economies, and =20
> > improved lives.
> >
> > We are proud of our achievements and recognize that the global =20
> > Apache community --both developers and users-- are responsible for =20
> > the success and popularity of our products.
> >
> > The ApacheCon Planning Team are soliciting 50-minute technical =20
> > presentations for the next conference, which will focus on the theme =
=20
> > ?Servers, the Cloud, and Innovation?.
> >
> > We are particularly interested in highly-relevant, professionally-=20
> > directed presentations that demonstrate specific probrlems and real-=20
> > world solutions. Part of the technical program has already been =20
> > planned; we welcome proposals based on the following Apache Projects =
=20
> > and related technical areas:
> >
> > - Cassandra/NoSQL
> > - Content Technologies
> > - (Java) Enterprise Development
> > - Felix/OSGi
> > - Geronimo
> > - Hadoop + friends/Cloud Computing
> > - Lucene, Mahout + friends/Search
> > - Tomcat
> > - Tuscany
> > Submissions are open to anyone with relevant expertise: ASF =20
> > affiliation is not required to present at, attend, or otherwise =20
> > participate in ApacheCon.
> >
> > Please keep in mind that whilst we encourage submissions that the =20
> > highlight the use of specific Apache solutions, we are unable to =20
> > accept marketing/commercially-oriented presentations.
> >
> > Other proposals, such as panels, or those longer than 50 minutes in =20
> > duration have been considered in the past. You are welcome to submit =
=20
> > an alternate presentation, however, such sessions are accepted under =
=20
> > exceptional circumstances. Please be as descriptive as possible, =20
> > including names/bios of proposed panelists and any related details.
> >
> > All accepted speakers (not co-presenters) qualify for general =20
> > conference admission and a minimum of two nights lodging at the =20
> > conference hotel. Additional hotel nights and travel assistance are =20
> > possible, depending on the number of presentations given and type of =
=20
> > assistance needed.
> >
> > To submit a presentation proposal, please send an email to =20
> > submissions AT apachecon DOT com containing the following =20
> > information in plaintext (no attachments, please):
> >
> > 1. Your full name, title, and organization
> >
> > 2. Contact information, including your address
> >
> > 3. The name of your proposed session (keep your title simple and =20
> > relevant to the topic)
> >
> > 4. The technical category of the intended presentation (Cassandra/=20
> > NoSQL; Content Technologies; (Java) Enterprise Development; Felix/=20
> > OSGi; Geronimo; Hadoop + friends/Cloud Computing; Lucene, Mahout + =20
> > friends/Search; Tomcat; or Tuscany)
> >
> > 5. The classification for each presentation (Servers, Cloud, or =20
> > Innovation) ? some presentations may have more than one theme (e.g., =
=20
> > a next-generation server can be classified both as "Servers" and =20
> > "Innovation"
> >
> > 6. The intended audience level (beginner, intermediate, advanced)
> >
> > 7. A 75-200 word overview of your presentation
> >
> > 8. A 100-200-word speaker bio that includes prior conference =20
> > speaking or related experience
> >
> > 9. Feedback or references (with contact information) on =20
> > presentations given within the last three years
> >
> > To be considered, proposals must be received by Friday, 28 May 2010 =20
> > at midnight Pacific Time. Please email any questions regarding =20
> > proposal submissions to cfp AT apachecon DOT com.
> >
> > Technical Tracks Key Dates
> >
> > 23 April 2010: Call For Participation Open
> > 28 May 2010: Call For Participation Closes
> > 11 June 2010: Speaker Acceptance/Rejection Notification
> > 1-5 November 2010: ApacheCon NA 2010
> > We look forward to seeing you in Atlanta!
> >
> > For the ApacheCon Planning team,
> > Sally Khudairi, Program Lead
>=20

--dkEUBIird37B8yKS
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)

iF4EAREIAAYFAkwINN4ACgkQMqXifkwDoaF5WwD/azv9wuKPEL+vdydP9NiUI9/d
RCrtdLv6FaIt6A93ivQA/i+Cre5xI/6Z2Rzy2VarEH1H4N9GNtldJbM3wZMZBFzr
=2tqv
-----END PGP SIGNATURE-----

--dkEUBIird37B8yKS--

From general-return-1613-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 19:40:56 2010
Return-Path: <general-return-1613-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 218 invoked from network); 4 Jun 2010 19:40:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 19:40:56 -0000
Received: (qmail 92409 invoked by uid 500); 4 Jun 2010 19:40:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92361 invoked by uid 500); 4 Jun 2010 19:40:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 92349 invoked by uid 99); 4 Jun 2010 19:40:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 19:40:54 +0000
X-ASF-Spam-Status: No, hits=2.8 required=10.0
	tests=AWL,HTML_MESSAGE,SPF_NEUTRAL,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 74.125.82.176 is neither permitted nor denied by domain of oded@legolas-media.com)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 19:40:47 +0000
Received: by wyb33 with SMTP id 33so1625209wyb.35
        for <general@hadoop.apache.org>; Fri, 04 Jun 2010 12:40:26 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.85.194 with SMTP id u44mr10878wee.10.1275680422351; Fri, 
	04 Jun 2010 12:40:22 -0700 (PDT)
Received: by 10.216.152.165 with HTTP; Fri, 4 Jun 2010 12:40:22 -0700 (PDT)
Date: Fri, 4 Jun 2010 22:40:22 +0300
Message-ID: <AANLkTimmmqBiF4NFxK6vRid8GQaAp_ZJY0ZwXplA_568@mail.gmail.com>
Subject: Problematic disk in a datanode
From: Oded Rosen <oded@legolas-media.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d7e7446d03020488397e67

--0016e6d7e7446d03020488397e67
Content-Type: text/plain; charset=ISO-8859-1

Hey,

A while ago We've added a new disk (volume) to every datanode in our
cluster.
We have configured the disks in "data.dfs.dir" in hdfs-site both on the job
tracker and on each machine.
This went successfully for all of the machines except one, where the new
disk was not recognized by hadoop.

We can not find out what's wrong with it.

We know that the new disk is not recognized because "http://namenode:50070/"
shows smaller capacity to that machine.
The mapred + hdfs directories on that drive exist, but they are not
identical to the structure of directories in other disks:
In the problematic drive there is no "local" directory under "mapred", and
no "name", "namesecondary" directories under "hdfs".

This problem was not so terrible until now, when the rest of the disks are
full:
The logs started containing errors such as "No space left on device" and
"DiskErrorException: Could not find any valid local directory for
taskTracker/jobcache/".
Some Hadoop jobs fail with the same errors, and the datanode+tasktracker on
that machine crash a lot.

How do we install this disk properly?

Thanks in advance.

Technical info: hadoop-0.20, centos, each machine is datanode and
tasktracker (another machine is jobtracker + namenode).

-- 
Oded

--0016e6d7e7446d03020488397e67--

From general-return-1614-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 05:34:37 2010
Return-Path: <general-return-1614-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 2559 invoked from network); 7 Jun 2010 05:34:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 05:34:37 -0000
Received: (qmail 31567 invoked by uid 500); 7 Jun 2010 05:34:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31479 invoked by uid 500); 7 Jun 2010 05:34:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31469 invoked by uid 99); 7 Jun 2010 05:34:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 05:34:32 +0000
X-ASF-Spam-Status: No, hits=2.5 required=10.0
	tests=AWL,FREEMAIL_FROM,FS_REPLICA,HK_RANDOM_ENVFROM,HK_RANDOM_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of zjffdu@gmail.com designates 209.85.222.180 as permitted sender)
Received: from [209.85.222.180] (HELO mail-pz0-f180.google.com) (209.85.222.180)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 05:34:26 +0000
Received: by pzk10 with SMTP id 10so2253796pzk.20
        for <general@hadoop.apache.org>; Sun, 06 Jun 2010 22:34:06 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=vcWyk8KF0ASy1PqFw+Tx+6369eUSD3A5AeukGW4s5cM=;
        b=v8cSbSFN2SW0kN9JqoyHQ3M5TbqAMC4QPlEYAWg3GHNgmTb+yOcvWyDjSwejM5oams
         FneEbHeGJWryUa+2xjDaJDC6IrpAA7SoMg2lekSnetFnruhCT9K0jgLGkfNO9X2KnZ/M
         x/8VIhKWNbmaSUozIVUZnqkHdHIJlr2Kfl8jc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=lmqgx9EgMEkxWMt9Jv/ZUrNglzvP93kdpurZ0dO/npK6kPIeqe4VT3Ge2tljzt4sR5
         uX3q2s48hZSy6la9CiYIpSEGBtu00KjZ8FtGQzQ1ryKL9s/MF6qqRv9xexjR02+bu1NP
         y7HYmrrqViYGMGnXts6F2mR4CHouJiXz26M9o=
MIME-Version: 1.0
Received: by 10.142.74.6 with SMTP id w6mr10534816wfa.249.1275888846033; Sun, 
	06 Jun 2010 22:34:06 -0700 (PDT)
Received: by 10.142.215.20 with HTTP; Sun, 6 Jun 2010 22:34:05 -0700 (PDT)
Date: Sun, 6 Jun 2010 22:34:05 -0700
Message-ID: <AANLkTimcgE1g6S6Fcn821rK__sOJhemz2K9VyH8lxmyt@mail.gmail.com>
Subject: How to force the replication ?
From: Jeff Zhang <zjffdu@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8

Hi all
I change the replication of file by using command : bin/hadoop setrep
But when I use fsck to check the status of the file, it always shows
that the one replica of this file is missing. I know that setrep
command only change the metadata of NameNode, so when will it affect
the data node eventually ? And Can I force the procedure ?

-- 
Best Regards

Jeff Zhang

From general-return-1615-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 07:37:09 2010
Return-Path: <general-return-1615-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33002 invoked from network); 7 Jun 2010 07:37:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 07:37:09 -0000
Received: (qmail 22921 invoked by uid 500); 7 Jun 2010 07:37:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22739 invoked by uid 500); 7 Jun 2010 07:37:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22731 invoked by uid 99); 7 Jun 2010 07:37:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 07:37:06 +0000
X-ASF-Spam-Status: No, hits=-1493.2 required=10.0
	tests=ALL_TRUSTED,AWL,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 07 Jun 2010 07:37:05 +0000
Received: (qmail 32893 invoked by uid 99); 7 Jun 2010 07:36:45 -0000
Received: from localhost.apache.org (HELO mail-iw0-f176.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 07:36:45 +0000
Received: by iwn9 with SMTP id 9so934558iwn.35
        for <multiple recipients>; Mon, 07 Jun 2010 00:36:44 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.210.19 with SMTP id gi19mr4085552ibb.140.1275896202181; 
	Mon, 07 Jun 2010 00:36:42 -0700 (PDT)
Received: by 10.231.148.65 with HTTP; Mon, 7 Jun 2010 00:36:42 -0700 (PDT)
Date: Mon, 7 Jun 2010 00:36:42 -0700
Message-ID: <AANLkTil8S3zmudjdNaao7D-uxYgKtmv_83-adO9T4IBU@mail.gmail.com>
Subject: New Common/MapReduce committer: Amareshwari Sriramadasu
From: Chris Douglas <cdouglas@apache.org>
To: mapreduce-dev@hadoop.apache.org, mapreduce-user@hadoop.apache.org, 
	general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

The Hadoop PMC has voted to make Amareshwari Sriramadasu a committer
on the Common and MapReduce subprojects.

Congratulations Amareshwari! Thanks for all your past and continued
work on Hadoop. -C

From general-return-1616-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 11:39:29 2010
Return-Path: <general-return-1616-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96910 invoked from network); 7 Jun 2010 11:39:29 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 11:39:29 -0000
Received: (qmail 41247 invoked by uid 500); 7 Jun 2010 11:39:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40336 invoked by uid 500); 7 Jun 2010 11:39:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39975 invoked by uid 99); 7 Jun 2010 11:39:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 11:39:18 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of vidur@students.iiit.ac.in designates 121.242.23.201 as permitted sender)
Received: from [121.242.23.201] (HELO students.iiit.ac.in) (121.242.23.201)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 11:39:10 +0000
MailScanner-NULL-Check: 1276515527.41327@n3j4OK4DHUmyeL6vUc2lFg
Received: from students.iiit.ac.in (localhost.localdomain [127.0.0.1])
	by students.iiit.ac.in (8.13.8/8.13.8) with ESMTP id o57BckSU032261
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Mon, 7 Jun 2010 17:08:46 +0530
Received: (from apache@localhost)
	by students.iiit.ac.in (8.13.8/8.14.1/Submit) id o57BckCI032250;
	Mon, 7 Jun 2010 17:08:46 +0530
X-Authentication-Warning: students.iiit.ac.in: apache set sender to vidur@localhost using -f
Received: from 125.16.17.152 (proxying for unknown)
        (SquirrelMail authenticated user vidur)
        by students.iiit.ac.in with HTTP;
        Mon, 7 Jun 2010 17:08:46 +0530 (IST)
Message-ID: <55997.125.16.17.152.1275910726.squirrel@students.iiit.ac.in>
Date: Mon, 7 Jun 2010 17:08:46 +0530 (IST)
Subject: HDFS Source Code
From: "Vidur Goyal" <vidur@students.iiit.ac.in>
To: general@hadoop.apache.org
User-Agent: SquirrelMail/1.4.8-4.el5
MIME-Version: 1.0
Content-Type: text/plain;charset=iso-8859-1
Content-Transfer-Encoding: 8bit
X-Priority: 3 (Normal)
Importance: Normal
X-Spam-Level: *
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on students.iiit.ac.in
X-yoursite-MailScanner-Information: Please contact the IIIT Server Room  for more information
X-MailScanner-ID: o57BckSU032261
X-yoursite-MailScanner: Found to be clean
X-yoursite-MailScanner-From: vidur@students.iiit.ac.in
X-Virus-Checked: Checked by ClamAV on apache.org
X-Old-Spam-Status: No, score=1.9 required=5.0 tests=ALL_TRUSTED,FH_DATE_PAST_20XX
	autolearn=no version=3.2.5, No

Hi,

I am experimenting with HDFS API's . I was wondering if somebody could
help me understand the source code of hdfs by providing any relevant
documentation or could guide me how to start.

Thanks,
Vidur

-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.


From general-return-1617-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 14:21:10 2010
Return-Path: <general-return-1617-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53100 invoked from network); 7 Jun 2010 14:21:10 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 14:21:10 -0000
Received: (qmail 40280 invoked by uid 500); 7 Jun 2010 14:21:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40195 invoked by uid 500); 7 Jun 2010 14:21:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40187 invoked by uid 99); 7 Jun 2010 14:21:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 14:21:08 +0000
X-ASF-Spam-Status: No, hits=2.3 required=10.0
	tests=AWL,HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 14:21:02 +0000
Received: by wyb29 with SMTP id 29so308968wyb.35
        for <general@hadoop.apache.org>; Mon, 07 Jun 2010 07:20:41 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.227.151.138 with SMTP id c10mr3357246wbw.219.1275920441273; 
	Mon, 07 Jun 2010 07:20:41 -0700 (PDT)
Received: by 10.216.188.6 with HTTP; Mon, 7 Jun 2010 07:20:41 -0700 (PDT)
In-Reply-To: <55997.125.16.17.152.1275910726.squirrel@students.iiit.ac.in>
References: <55997.125.16.17.152.1275910726.squirrel@students.iiit.ac.in>
Date: Mon, 7 Jun 2010 10:20:41 -0400
Message-ID: <AANLkTinmNw953WWV63KVJFyP1TFO7jHRd3FUEEtqbFHN@mail.gmail.com>
Subject: Re: HDFS Source Code
From: Josh Patterson <josh@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163641842da9ee76048871605a

--00163641842da9ee76048871605a
Content-Type: text/plain; charset=ISO-8859-1

Vidur,
Probably a great place to get started would be:

http://hadoop.apache.org/common/docs/current/hdfs_design.html

<http://hadoop.apache.org/common/docs/current/hdfs_design.html>Then after
reading that, you could take a look at the api docs:

<http://hadoop.apache.org/common/docs/r0.20.0/>
http://hadoop.apache.org/common/docs/current/api/

and then a few ways to touch hdfs from other languages:

http://wiki.apache.org/hadoop/HDFS-APIs

Beyond that, I'd download the source from SVN and take a look at

org.apache.hadoop.hdfs.DFSClient.java

to get a feel for how most apps talk to HDFS. After that you could look at:
*
*
*org.apache.hadoop.fs.FsShell*

to get a feel for how the shell system as a java program talks to DFSClient.

Beyond that, just reading more source code and compiling your own
experiments reading and writing to HDFS is the best way to get a feel for
whats going on under the hood.

Josh Patterson

Solutions Architect
Cloudera




On Mon, Jun 7, 2010 at 7:38 AM, Vidur Goyal <vidur@students.iiit.ac.in>wrote:

> Hi,
>
> I am experimenting with HDFS API's . I was wondering if somebody could
> help me understand the source code of hdfs by providing any relevant
> documentation or could guide me how to start.
>
> Thanks,
> Vidur
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>
>

--00163641842da9ee76048871605a--

From general-return-1618-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 16:21:23 2010
Return-Path: <general-return-1618-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4777 invoked from network); 7 Jun 2010 16:21:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 16:21:22 -0000
Received: (qmail 7239 invoked by uid 500); 7 Jun 2010 16:21:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7185 invoked by uid 500); 7 Jun 2010 16:21:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7177 invoked by uid 99); 7 Jun 2010 16:21:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 16:21:21 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=AWL,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ashahzad4@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 16:21:15 +0000
Received: by wyb29 with SMTP id 29so470707wyb.35
        for <general@hadoop.apache.org>; Mon, 07 Jun 2010 09:20:53 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=tjp85BuTr2+6Qmhybq/tWObIMBiamYIuiSFFOiCK5i4=;
        b=Np4iHgOx4eR0PUIPUqbmASMYnie9hQFxMaqzBQSPQuxcm6cAFuxDitTV7wv+u2H0Cx
         UOBYLEmIBdAHNI2T7rKBRCQAAUN7vqroW7Osc7i4CPHtMNz2J/LHwp7h3aRxWJYCL6iS
         9cvrnCEZPI1iygz4a5bRV0/REjFK+pKGvIoP4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=BfAWrVzk+muHimJ/wnRmwawHaHXtktyIhw0qS1oszey1oBs2LCsy3nz4jjMj6eqY0s
         EDz/0T0Jl/pvMBt4/Ej7xmmBuSl3wFJprN7tkuec2rP5qmn8v6kTN5GBIgIOM5dkQzQc
         ETUcuLqNMb5O/eKT+JVl+DqPcY/jLsg3FmslE=
MIME-Version: 1.0
Received: by 10.216.90.138 with SMTP id e10mr2558641wef.51.1275927652998; Mon, 
	07 Jun 2010 09:20:52 -0700 (PDT)
Received: by 10.216.186.19 with HTTP; Mon, 7 Jun 2010 09:20:52 -0700 (PDT)
In-Reply-To: <AANLkTinmNw953WWV63KVJFyP1TFO7jHRd3FUEEtqbFHN@mail.gmail.com>
References: <55997.125.16.17.152.1275910726.squirrel@students.iiit.ac.in>
	<AANLkTinmNw953WWV63KVJFyP1TFO7jHRd3FUEEtqbFHN@mail.gmail.com>
Date: Mon, 7 Jun 2010 18:20:52 +0200
Message-ID: <AANLkTinFxgObDuuK8LBXVhaSAvzW1kZsEDP7PONhuf47@mail.gmail.com>
Subject: Re: HDFS Source Code
From: Ahmad Shahzad <ashahzad4@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d99bed8423880488730e99

--0016e6d99bed8423880488730e99
Content-Type: text/plain; charset=ISO-8859-1

Hi,
    I am also interested in looking what is going on under the hood. Thanks
for sharing this information.

Could you please tell the same about Map Reduce. I mean, how to figure out
that how job tracker and task trackers communicate with each other. Which
classes in hadoop api are responsible for doing that. If i go in detail, i
would like to know that how can i change the communication mechanism of
hadoop map reduce to use my communication library instead of using regular
sockets and http.

Regards,
Ahmad Shahzad

--0016e6d99bed8423880488730e99--

From general-return-1619-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 19:47:55 2010
Return-Path: <general-return-1619-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97853 invoked from network); 7 Jun 2010 19:47:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 19:47:55 -0000
Received: (qmail 31449 invoked by uid 500); 7 Jun 2010 19:47:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31400 invoked by uid 500); 7 Jun 2010 19:47:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31392 invoked by uid 99); 7 Jun 2010 19:47:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 19:47:53 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 19:47:46 +0000
Received: by wyb29 with SMTP id 29so715433wyb.35
        for <general@hadoop.apache.org>; Mon, 07 Jun 2010 12:47:25 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.179.147 with SMTP id h19mr2667063wem.81.1275940045540; 
	Mon, 07 Jun 2010 12:47:25 -0700 (PDT)
Received: by 10.216.188.6 with HTTP; Mon, 7 Jun 2010 12:47:25 -0700 (PDT)
In-Reply-To: <AANLkTinFxgObDuuK8LBXVhaSAvzW1kZsEDP7PONhuf47@mail.gmail.com>
References: <55997.125.16.17.152.1275910726.squirrel@students.iiit.ac.in>
	<AANLkTinmNw953WWV63KVJFyP1TFO7jHRd3FUEEtqbFHN@mail.gmail.com>
	<AANLkTinFxgObDuuK8LBXVhaSAvzW1kZsEDP7PONhuf47@mail.gmail.com>
Date: Mon, 7 Jun 2010 15:47:25 -0400
Message-ID: <AANLkTik7OL3-s0XqQ1b4xuKbpnuZAtnUFqcvVvPJmGBv@mail.gmail.com>
Subject: Re: HDFS Source Code
From: Josh Patterson <josh@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016367faf132b5976048875f1e1
X-Virus-Checked: Checked by ClamAV on apache.org

--0016367faf132b5976048875f1e1
Content-Type: text/plain; charset=ISO-8859-1

Ahmad,
A nice overview of Map Reduce is at:

http://hadoop.apache.org/common/docs/current/mapred_tutorial.html

<http://hadoop.apache.org/common/docs/current/mapred_tutorial.html>In terms
of replacing the entire communication framework under Map Reduce, thats a
considerably different and more complex task that simply talking to hdfs
from java. You have a large number of interlocking classes based on the
communication system of hadoop; replacing this would be like trying to
replace the frame of an automobile --- in other words, you might try, but
you run a high risk of not having a large percentage of the parts not work
correctly with the new automobile frame.

If you were to tackle such a large task, it would be less of "knowing a
certain set of classes to work with" and more of knowing how a large degree
of hadoop works.

Josh

On Mon, Jun 7, 2010 at 12:20 PM, Ahmad Shahzad <ashahzad4@gmail.com> wrote:

> Hi,
>    I am also interested in looking what is going on under the hood. Thanks
> for sharing this information.
>
> Could you please tell the same about Map Reduce. I mean, how to figure out
> that how job tracker and task trackers communicate with each other. Which
> classes in hadoop api are responsible for doing that. If i go in detail, i
> would like to know that how can i change the communication mechanism of
> hadoop map reduce to use my communication library instead of using regular
> sockets and http.
>
> Regards,
> Ahmad Shahzad
>

--0016367faf132b5976048875f1e1--

From general-return-1620-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 19:50:04 2010
Return-Path: <general-return-1620-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98444 invoked from network); 7 Jun 2010 19:50:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 19:50:04 -0000
Received: (qmail 33946 invoked by uid 500); 7 Jun 2010 19:50:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33777 invoked by uid 500); 7 Jun 2010 19:50:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33769 invoked by uid 99); 7 Jun 2010 19:50:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 19:50:03 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.59.228] (HELO qmta15.westchester.pa.mail.comcast.net) (76.96.59.228)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 19:49:53 +0000
Received: from omta15.westchester.pa.mail.comcast.net ([76.96.62.87])
	by qmta15.westchester.pa.mail.comcast.net with comcast
	id T7Yh1e0051swQuc5F7pZSr; Mon, 07 Jun 2010 19:49:33 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta15.westchester.pa.mail.comcast.net with comcast
	id T7pP1e00p2SbwD53b7pSvf; Mon, 07 Jun 2010 19:49:31 +0000
Message-Id: <D7FAD7C1-2C97-4426-B46D-1AFBB5AF8C3A@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <AANLkTik7OL3-s0XqQ1b4xuKbpnuZAtnUFqcvVvPJmGBv@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: HDFS Source Code
Date: Mon, 7 Jun 2010 12:49:22 -0700
References: <55997.125.16.17.152.1275910726.squirrel@students.iiit.ac.in> <AANLkTinmNw953WWV63KVJFyP1TFO7jHRd3FUEEtqbFHN@mail.gmail.com> <AANLkTinFxgObDuuK8LBXVhaSAvzW1kZsEDP7PONhuf47@mail.gmail.com> <AANLkTik7OL3-s0XqQ1b4xuKbpnuZAtnUFqcvVvPJmGBv@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

Please move this discussion over to hdfs-dev@hadoop.apache.org.

Thanks,
    Owen

From general-return-1621-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 08 05:28:08 2010
Return-Path: <general-return-1621-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63976 invoked from network); 8 Jun 2010 05:28:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jun 2010 05:28:08 -0000
Received: (qmail 59891 invoked by uid 500); 8 Jun 2010 05:28:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59844 invoked by uid 500); 8 Jun 2010 05:28:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59836 invoked by uid 99); 8 Jun 2010 05:28:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 05:28:06 +0000
X-ASF-Spam-Status: No, hits=3.6 required=10.0
	tests=AWL,HTML_MESSAGE,SPF_HELO_SOFTFAIL,SPF_SOFTFAIL
X-Spam-Check-By: apache.org
Received-SPF: softfail (athena.apache.org: transitioning domain of maheswaran@ikyaglobal.com does not designate 121.243.47.5 as permitted sender)
Received: from [121.243.47.5] (HELO ikyaglobal.com) (121.243.47.5)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 05:27:59 +0000
Received: (qmail 26017 invoked by uid 109); 8 Jun 2010 05:26:55 -0000
Received: from 122.165.35.199 by mail (envelope-from <maheswaran@ikyaglobal.com>, uid 112) with qmail-scanner-1.25st 
 (perlscan: 1.25st. dmf: ???.  
 Clear:RC:1(122.165.35.199):. 
 Processed in 0.288082 secs); 08 Jun 2010 05:26:55 -0000
Received: from abts-tn-static-199.35.165.122.airtelbroadband.in (HELO CHNDSK029) (Maheswaran@[122.165.35.199])
          (envelope-sender <maheswaran@ikyaglobal.com>)
          by ikyaglobal.com (qmail-ldap-1.03) with SMTP
          for <general@hadoop.apache.org>; 8 Jun 2010 05:26:55 -0000
From: "Maheswaran" <maheswaran@ikyaglobal.com>
To: <general@hadoop.apache.org>
Subject: Regarding Oppurtunity in Hadoop - Bangalore,INDIA Location
Date: Tue, 8 Jun 2010 10:57:42 +0530
Message-ID: <!~!UENERkVCMDkAAQACAAAAAAAAAAAAAAAAABgAAAAAAAAAjW9UT2Lm9keBcNAk3X7fn8KAAAAQAAAA0WrzCxsrdkez4R2vbBSB1AEAAAAA@ikyaglobal.com>
MIME-Version: 1.0
Content-Type: multipart/related;
	boundary="----=_NextPart_000_0058_01CB06F9.6C023B20"
X-Mailer: Microsoft Office Outlook, Build 11.0.5510
Thread-Index: AcsGyoS6U3a1Orc6QNu1nuZJzjd/xwAAMZ4Q
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5931

------=_NextPart_000_0058_01CB06F9.6C023B20
Content-Type: multipart/alternative;
	boundary="----=_NextPart_001_0059_01CB06F9.6C023B20"


------=_NextPart_001_0059_01CB06F9.6C023B20
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit

Dear Developers/Architect,

 

This is Maheswaran from IKYA, Chennai. We have a Hadoop Architect positions
open in Bangalore for one of our client. Kindly let us know if any one is
interested. Will share more details about the position and the company.

 

Thanks & Regards,

 

Maheswaran A | Practice Lead-IT | Recruitment Solutions



 

Ikya Human Capital Solutions Pvt. Ltd.

No. 2, 5th Floor, Alsa Towers

186/187 Poonamalee High Road, 

Kilpauk, Chennai 600010, India

Direct: +91 44 6662 3008 | Mobile: +91 9840 348 330 | www.ikyaglobal.com
<http://www.ikyaglobal.com/> 

 


------=_NextPart_001_0059_01CB06F9.6C023B20
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:st1=3D"urn:schemas-microsoft-com:office:smarttags" =
xmlns=3D"http://www.w3.org/TR/REC-html40">

<head>
<meta http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dus-ascii">
<meta name=3DGenerator content=3D"Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]--><o:SmartTagType
 namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags" =
name=3D"City"
 downloadurl=3D"http://www.5iamas-microsoft-com:office:smarttags"/>
<o:SmartTagType =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags"
 name=3D"country-region" downloadurl=3D"http://www.5iantlavalamp.com/"/>
<o:SmartTagType =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags"
 name=3D"Street" =
downloadurl=3D"http://www.5iantlavalampft-com:office:smarttags"/>
<o:SmartTagType =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags"
 name=3D"address" =
downloadurl=3D"http://www.5iamas-microsoft-com:office:smarttags"/>
<o:SmartTagType =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags"
 name=3D"PlaceType" downloadurl=3D"http://www.5iantlavalamp.com/"/>
<o:SmartTagType =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags"
 name=3D"PlaceName" downloadurl=3D"http://www.5iantlavalamp.com/"/>
<o:SmartTagType =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags"
 name=3D"place" downloadurl=3D"http://www.5iantlavalamp.com/"/>
<!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
 /* Font Definitions */
 @font-face
	{font-family:"Bookman Old Style";
	panose-1:2 5 6 4 5 5 5 2 2 4;}
 /* Style Definitions */
 p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:12.0pt;
	font-family:"Times New Roman";}
a:link, span.MsoHyperlink
	{color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{color:purple;
	text-decoration:underline;}
span.EmailStyle17
	{mso-style-type:personal;
	font-family:"Bookman Old Style";
	color:blue;
	font-weight:normal;
	font-style:normal;
	text-decoration:none none;}
span.EmailStyle19
	{mso-style-type:personal-reply;
	font-family:"Bookman Old Style";
	color:blue;
	font-weight:normal;
	font-style:normal;
	text-decoration:none none;}
@page Section1
	{size:8.5in 11.0in;
	margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
	{page:Section1;}
-->
</style>

</head>

<body lang=3DEN-US link=3Dblue vlink=3Dpurple>

<div class=3DSection1>

<p class=3DMsoNormal><font size=3D2 color=3Dblue face=3D"Bookman Old =
Style"><span
style=3D'font-size:10.0pt;font-family:"Bookman Old =
Style";color:blue'>Dear
Developers/Architect,<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dblue face=3D"Bookman Old =
Style"><span
style=3D'font-size:10.0pt;font-family:"Bookman Old =
Style";color:blue'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dblue face=3D"Bookman Old =
Style"><span
style=3D'font-size:10.0pt;font-family:"Bookman Old =
Style";color:blue'>This is
Maheswaran from IKYA, Chennai. We have a Hadoop Architect positions open =
in <st1:place
w:st=3D"on"><st1:City w:st=3D"on">Bangalore</st1:City></st1:place> for =
one of our
client. Kindly let us know if any one is interested. Will share more =
details
about the position and the company.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 color=3Dblue face=3D"Bookman Old =
Style"><span
style=3D'font-size:10.0pt;font-family:"Bookman Old =
Style";color:blue'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal style=3D'margin-bottom:5.0pt'><strong><b><font =
size=3D1
color=3Dpurple face=3DArial><span lang=3DEN-IN =
style=3D'font-size:8.0pt;font-family:
Arial;color:purple;font-weight:normal'>Thanks &amp; =
Regards,<o:p></o:p></span></font></b></strong></p>

<p class=3DMsoNormal style=3D'margin-bottom:5.0pt'><strong><b><font =
size=3D1
color=3Dpurple face=3DArial><span lang=3DEN-IN =
style=3D'font-size:8.0pt;font-family:
Arial;color:purple'><o:p>&nbsp;</o:p></span></font></b></strong></p>

<p class=3DMsoNormal style=3D'margin-bottom:5.0pt'><strong><b><font =
size=3D1
color=3Dpurple face=3DArial><span lang=3DEN-IN =
style=3D'font-size:8.0pt;font-family:
Arial;color:purple'>Maheswaran A</span></font></b></strong><font =
size=3D1
color=3Dpurple face=3DArial><span lang=3DEN-IN =
style=3D'font-size:8.0pt;font-family:
Arial;color:purple'>&nbsp;|&nbsp;Practice =
Lead-IT&nbsp;|&nbsp;Recruitment
Solutions</span></font><font color=3Dblue><span =
style=3D'color:blue'><o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 color=3D"#595959" face=3DArial><span =
lang=3DEN-IN
style=3D'font-size:8.0pt;font-family:Arial;color:#595959'><img =
width=3D121
height=3D50 id=3D"_x0000_i1025" =
src=3D"cid:image001.jpg@01CB06F8.9E6DC080"><o:p></o:p></span></font></p>

<p class=3DMsoNormal><b><font size=3D1 color=3D"#595959" =
face=3DArial><span lang=3DEN-IN
style=3D'font-size:8.0pt;font-family:Arial;color:#595959;font-weight:bold=
'><o:p>&nbsp;</o:p></span></font></b></p>

<p class=3DMsoNormal style=3D'margin-top:5.0pt'><strong><b><font =
size=3D1
color=3D"#4d4d4d" face=3DArial><span lang=3DEN-IN =
style=3D'font-size:8.0pt;font-family:
Arial;color:#4D4D4D'>Ikya Human Capital Solutions Pvt. =
Ltd.</span></font><font
color=3D"#4d4d4d"><span =
style=3D'color:#4D4D4D'><o:p></o:p></span></font></b></strong></p>

<p class=3DMsoNormal><font size=3D1 color=3D"#4d4d4d" face=3DArial><span =
lang=3DEN-IN
style=3D'font-size:8.0pt;font-family:Arial;color:#4D4D4D'>No. 2, 5th =
Floor,&nbsp;<st1:place
w:st=3D"on"><st1:PlaceName w:st=3D"on">Alsa</st1:PlaceName> =
<st1:PlaceType =
w:st=3D"on">Towers</st1:PlaceType></st1:place></span></font><font
color=3Dblue><span style=3D'color:blue'><o:p></o:p></span></font></p>

<p class=3DMsoNormal><st1:Street w:st=3D"on"><st1:address =
w:st=3D"on"><font size=3D1
  color=3D"#4d4d4d" face=3DArial><span lang=3DEN-IN =
style=3D'font-size:8.0pt;
  font-family:Arial;color:#4D4D4D'>186/187&nbsp;Poonamalee =
High&nbsp;Road</span></font></st1:address></st1:Street><font
size=3D1 color=3D"#4d4d4d" face=3DArial><span lang=3DEN-IN =
style=3D'font-size:8.0pt;
font-family:Arial;color:#4D4D4D'>,&nbsp;</span></font><font size=3D1
color=3D"#595959" face=3DArial><span lang=3DEN-IN =
style=3D'font-size:8.0pt;font-family:
Arial;color:#595959'><o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D1 color=3D"#4d4d4d" face=3DArial><span =
lang=3DEN-IN
style=3D'font-size:8.0pt;font-family:Arial;color:#4D4D4D'>Kilpauk,&nbsp;<=
st1:City
w:st=3D"on">Chennai</st1:City> 600010, <st1:place =
w:st=3D"on"><st1:country-region
 =
w:st=3D"on">India</st1:country-region></st1:place><o:p></o:p></span></fon=
t></p>

<p class=3DMsoNormal style=3D'margin-top:5.0pt'><font size=3D1 =
color=3D"#4d4d4d"
face=3DArial><span lang=3DEN-IN =
style=3D'font-size:8.0pt;font-family:Arial;
color:#4D4D4D'>Direct: +91&nbsp;44 6662&nbsp;3008</span></font><font =
size=3D1
color=3Dpurple face=3DArial><span lang=3DEN-IN =
style=3D'font-size:8.0pt;font-family:
Arial;color:purple'> |</span></font><font size=3D1 color=3D"#4d4d4d" =
face=3DArial><span
lang=3DEN-IN style=3D'font-size:8.0pt;font-family:Arial;color:#4D4D4D'> =
Mobile: +91
9840&nbsp;348&nbsp;330 </span></font><font size=3D1 color=3Dpurple =
face=3DArial><span
lang=3DEN-IN =
style=3D'font-size:8.0pt;font-family:Arial;color:purple'>|</span></font><=
font
size=3D1 color=3D"#4d4d4d" face=3DArial><span lang=3DEN-IN =
style=3D'font-size:8.0pt;
font-family:Arial;color:#4D4D4D'> </span></font><font size=3D1 =
color=3D"#595959"
face=3DArial><span lang=3DEN-IN =
style=3D'font-size:8.0pt;font-family:Arial;
color:#595959'><a =
href=3D"http://www.ikyaglobal.com/">www.ikyaglobal.com</a><o:p></o:p></sp=
an></font></p>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

</div>

</body>

</html>

------=_NextPart_001_0059_01CB06F9.6C023B20--

------=_NextPart_000_0058_01CB06F9.6C023B20--


From general-return-1622-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 08 11:30:38 2010
Return-Path: <general-return-1622-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 74515 invoked from network); 8 Jun 2010 11:30:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jun 2010 11:30:38 -0000
Received: (qmail 13138 invoked by uid 500); 8 Jun 2010 11:30:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12659 invoked by uid 500); 8 Jun 2010 11:30:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12648 invoked by uid 99); 8 Jun 2010 11:30:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 11:30:33 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=AWL,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ashahzad4@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 11:30:26 +0000
Received: by wwb34 with SMTP id 34so975458wwb.35
        for <general@hadoop.apache.org>; Tue, 08 Jun 2010 04:30:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=bl9HunTbfiiaoIiaf6opqsynmvLLhS3RTqmlxHAZXk0=;
        b=JK0Ch4bUpgq3QcGK9jn79CnP6Y1EfNaNQs3gbQc+X+dso0h6rfROvEYbPzeHDc1n6Q
         BF4xl6FVn6hJcJbt81N3nzxmBU+erq84zAq46cgGZQtvR8hM5FWbprXwngWpjJFzsMDT
         k7LyU2XgrBKZYENVmeCUB1d9+WRsdfVz6u0G4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=Muf/rih68KBl6JQZ0th/2Aw21RK7LPtAcak9+S1OhD93+r37oDll3Y+95aVoAQe2t8
         o8BjYDB4Bwu0afx5vnvIK8XGtTCppcddY/w8q0Q4YccD4pZvXrVTsZDljbY0mCTIKWia
         ar42oMesOtWbxbwPjLJyD3JGIRSPlVjlHK1Y4=
MIME-Version: 1.0
Received: by 10.227.137.1 with SMTP id u1mr7445493wbt.68.1275996604926; Tue, 
	08 Jun 2010 04:30:04 -0700 (PDT)
Received: by 10.216.186.19 with HTTP; Tue, 8 Jun 2010 04:30:04 -0700 (PDT)
Date: Tue, 8 Jun 2010 13:30:04 +0200
Message-ID: <AANLkTiksK--b_qosOGjIbNQvKqCZN-33xj4eQWbOs8CK@mail.gmail.com>
Subject: Is it possible ....!!!
From: Ahmad Shahzad <ashahzad4@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e65b55305f10630488831cab

--0016e65b55305f10630488831cab
Content-Type: text/plain; charset=ISO-8859-1

Hi,
    I wanted to ask if it is possible to intercept every communication that
takes place between hadoop's map reduce task i.e between JobTracker and
TaskTracker and make it pass through my own communication library.
So, if JobTracker and TaskTracker talk through http or rpc, i would like to
intercept the call and let it pass through my communication library. If it
is possible can anyone tell me that which set of classes i need to look at
hadoop's distribution.

Similarly, for the hdfs, is it possible to let all the communication that is
happening between namenode and datanode to pass through my communication
library.

Regards,
Ahmad Shahzad

--0016e65b55305f10630488831cab--

From general-return-1623-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 08 13:49:10 2010
Return-Path: <general-return-1623-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 18704 invoked from network); 8 Jun 2010 13:49:10 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jun 2010 13:49:10 -0000
Received: (qmail 72265 invoked by uid 500); 8 Jun 2010 13:49:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72200 invoked by uid 500); 8 Jun 2010 13:49:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72192 invoked by uid 99); 8 Jun 2010 13:49:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 13:49:08 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of msegel@navteq.com designates 204.120.70.37 as permitted sender)
Received: from [204.120.70.37] (HELO xmailfargo.navteq.com) (204.120.70.37)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 13:49:00 +0000
Received: from fargo-spamfw.navteq.com (navteq-spamfw.navteq.com [10.6.10.131])
	by xmailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id o58DTPJf029598
	for <general@hadoop.apache.org>; Tue, 8 Jun 2010 08:29:25 -0500
Received: from imailfargo.navteq.com (localhost [127.0.0.1])
	by fargo-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id 7842A48B688D
	for <general@hadoop.apache.org>; Tue,  8 Jun 2010 08:48:39 -0500 (CDT)
Received: from imailfargo.navteq.com (imailfargo.navteq.com [10.6.10.130]) by fargo-spamfw.navteq.com with ESMTP id 4hKFmfk37k8W2ciQ for <general@hadoop.apache.org>; Tue, 08 Jun 2010 08:48:39 -0500 (CDT)
Received: from hq-ex-ht01.ad.navteq.com ([10.8.222.51])
	by imailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id o58DmdgU008394
	for <general@hadoop.apache.org>; Tue, 8 Jun 2010 08:48:39 -0500
Received: from hq-ex-mb02.ad.navteq.com ([fe80::880d:318e:e860:a8db]) by
 hq-ex-ht01.ad.navteq.com ([fe80::55a6:5b94:5c00:fe39%13]) with mapi; Tue, 8
 Jun 2010 08:48:38 -0500
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 8 Jun 2010 08:48:37 -0500
Subject: RE: Is it possible ....!!!
Thread-Topic: Is it possible ....!!!
Thread-Index: AcsG/hKLo973s1AeSay/G7JOILdSvgAEyjew
Message-ID: <ABC24175AFD3BE4DA15F4CD375ED413D06082089FE@hq-ex-mb02.ad.navteq.com>
References: <AANLkTiksK--b_qosOGjIbNQvKqCZN-33xj4eQWbOs8CK@mail.gmail.com>
In-Reply-To: <AANLkTiksK--b_qosOGjIbNQvKqCZN-33xj4eQWbOs8CK@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1
X-Virus-Checked: Checked by ClamAV on apache.org

Is it possible?
Sure.

But why? What's your use case?


-----Original Message-----
From: Ahmad Shahzad [mailto:ashahzad4@gmail.com]=20
Sent: Tuesday, June 08, 2010 6:30 AM
To: general@hadoop.apache.org
Subject: Is it possible ....!!!

Hi,
    I wanted to ask if it is possible to intercept every communication th=
at
takes place between hadoop's map reduce task i.e between JobTracker and
TaskTracker and make it pass through my own communication library.
So, if JobTracker and TaskTracker talk through http or rpc, i would like =
to
intercept the call and let it pass through my communication library. If i=
t
is possible can anyone tell me that which set of classes i need to look a=
t
hadoop's distribution.

Similarly, for the hdfs, is it possible to let all the communication that=
 is
happening between namenode and datanode to pass through my communication
library.

Regards,
Ahmad Shahzad


The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.

From general-return-1624-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 08 14:25:09 2010
Return-Path: <general-return-1624-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 32443 invoked from network); 8 Jun 2010 14:25:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jun 2010 14:25:09 -0000
Received: (qmail 17033 invoked by uid 500); 8 Jun 2010 14:25:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16963 invoked by uid 500); 8 Jun 2010 14:25:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16955 invoked by uid 99); 8 Jun 2010 14:25:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 14:25:08 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=AWL,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ashahzad4@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 14:25:02 +0000
Received: by wyb29 with SMTP id 29so1614168wyb.35
        for <general@hadoop.apache.org>; Tue, 08 Jun 2010 07:24:41 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=w87NX+WX/hfrrosXE36ZFplFpLGzSWQZvUjKSk/mnRw=;
        b=iOEBQmWrXt1ipWf6np0LxbSj+gvLacll/kYJLoCdgoajFHUGzu6LuEnK/qFGhsou7I
         4OgdFwGF7HA1gYPzYqgBaFMoAZCB0GPs7SbvK2y5B53aGdSlGOYLIjuAcPl1WxlXGA53
         Gr75qnV3zSMgpqFyFU+T201fMcQBMbe0HN0cE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Yk2c2iRPMSV1maAZl6fk4HCtOHyJxg9VWmU2JuRzzcHZi1ZU6Y2NmrNWTkMS5KT5j/
         pYcVeBz0w6kbPKKU7qHsgb9H5vnscEy/JmTnLLsIXzQCTeqsWiwIrbARTbLja7wZrakR
         rzD9qBSO1g2KgAgpommaSeGEy2nQTcZJPz9Eg=
MIME-Version: 1.0
Received: by 10.216.90.199 with SMTP id e49mr3560983wef.38.1276007080996; Tue, 
	08 Jun 2010 07:24:40 -0700 (PDT)
Received: by 10.216.186.19 with HTTP; Tue, 8 Jun 2010 07:24:40 -0700 (PDT)
In-Reply-To: <ABC24175AFD3BE4DA15F4CD375ED413D06082089FE@hq-ex-mb02.ad.navteq.com>
References: <AANLkTiksK--b_qosOGjIbNQvKqCZN-33xj4eQWbOs8CK@mail.gmail.com>
	<ABC24175AFD3BE4DA15F4CD375ED413D06082089FE@hq-ex-mb02.ad.navteq.com>
Date: Tue, 8 Jun 2010 16:24:40 +0200
Message-ID: <AANLkTinYuP9TdaefRwmVX21GVvrmXENJESYT8GDEkPYo@mail.gmail.com>
Subject: Re: Is it possible ....!!!
From: Ahmad Shahzad <ashahzad4@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d9a0facb312e0488858c26

--0016e6d9a0facb312e0488858c26
Content-Type: text/plain; charset=ISO-8859-1

Hi,
    Reason for doing that is that i want all the communication to happen
through a communication library that resolves every communication problem
that we can have e.g firewalls, NAT, non routed paths, multi homing etc etc.
By using that library all the headache of communication will be gone. So, we
will be able to use hadoop quite easily and there will be no communication
problems.

Thats my master's project. So, i want to know how to start and where to look
for.

Regards,
Ahmad Shahzad

On Tue, Jun 8, 2010 at 3:48 PM, Segel, Mike <msegel@navteq.com> wrote:

> Is it possible?
> Sure.
>
> But why? What's your use case?
>
>
> -----Original Message-----
> From: Ahmad Shahzad [mailto:ashahzad4@gmail.com]
> Sent: Tuesday, June 08, 2010 6:30 AM
> To: general@hadoop.apache.org
> Subject: Is it possible ....!!!
>
> Hi,
>    I wanted to ask if it is possible to intercept every communication that
> takes place between hadoop's map reduce task i.e between JobTracker and
> TaskTracker and make it pass through my own communication library.
> So, if JobTracker and TaskTracker talk through http or rpc, i would like to
> intercept the call and let it pass through my communication library. If it
> is possible can anyone tell me that which set of classes i need to look at
> hadoop's distribution.
>
> Similarly, for the hdfs, is it possible to let all the communication that
> is
> happening between namenode and datanode to pass through my communication
> library.
>
> Regards,
> Ahmad Shahzad
>
>
> The information contained in this communication may be CONFIDENTIAL and is
> intended only for the use of the recipient(s) named above.  If you are not
> the intended recipient, you are hereby notified that any dissemination,
> distribution, or copying of this communication, or any of its contents, is
> strictly prohibited.  If you have received this communication in error,
> please notify the sender and delete/destroy the original message and any
> copy of it from your computer or paper files.
>

--0016e6d9a0facb312e0488858c26--

From general-return-1625-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 08 16:49:35 2010
Return-Path: <general-return-1625-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91262 invoked from network); 8 Jun 2010 16:49:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jun 2010 16:49:35 -0000
Received: (qmail 33531 invoked by uid 500); 8 Jun 2010 16:49:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33464 invoked by uid 500); 8 Jun 2010 16:49:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33456 invoked by uid 99); 8 Jun 2010 16:49:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 16:49:33 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.30.96] (HELO qmta09.emeryville.ca.mail.comcast.net) (76.96.30.96)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 16:49:24 +0000
Received: from omta22.emeryville.ca.mail.comcast.net ([76.96.30.89])
	by qmta09.emeryville.ca.mail.comcast.net with comcast
	id TSai1e0041vN32cA9Up2Q3; Tue, 08 Jun 2010 16:49:02 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta22.emeryville.ca.mail.comcast.net with comcast
	id TUot1e00R2SbwD58iUowtt; Tue, 08 Jun 2010 16:49:00 +0000
Message-Id: <CEF6BDC7-2E4A-4E54-84D2-401E034F1A3D@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <AANLkTiksK--b_qosOGjIbNQvKqCZN-33xj4eQWbOs8CK@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Is it possible ....!!!
Date: Tue, 8 Jun 2010 08:13:18 -0700
References: <AANLkTiksK--b_qosOGjIbNQvKqCZN-33xj4eQWbOs8CK@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


On Jun 8, 2010, at 4:30 AM, Ahmad Shahzad wrote:

> Hi,
>    I wanted to ask if it is possible to intercept every  
> communication that
> takes place between hadoop's map reduce task i.e between JobTracker  
> and
> TaskTracker and make it pass through my own communication library.

It is possible to set up socket proxies for setting up rpc  
connections. Please take this thread over to common-user@hadoop.apache.org 
.

-- Owen

From general-return-1626-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 08 18:54:37 2010
Return-Path: <general-return-1626-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54071 invoked from network); 8 Jun 2010 18:54:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jun 2010 18:54:37 -0000
Received: (qmail 32164 invoked by uid 500); 8 Jun 2010 18:54:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31676 invoked by uid 500); 8 Jun 2010 18:54:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31658 invoked by uid 99); 8 Jun 2010 18:54:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 18:54:35 +0000
X-ASF-Spam-Status: No, hits=1.7 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of n.atreju@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 18:54:28 +0000
Received: by wyb29 with SMTP id 29so1956345wyb.35
        for <multiple recipients>; Tue, 08 Jun 2010 11:54:07 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=XCqsVLbsmkGKIFejVv85bnR61R04W2QwbPgMKTfNb0o=;
        b=KhgljOFjvB7mLMubBMs2eQuQ2K+FFK1VMwvEALWTq8W/ZoZt7Yxr4LesNl/fTO4uM5
         1qj//xUwtkeY7hg7tK1remjuYb6/Wyge5npzwoegkjOUrblDbF71asMBe8Btl5wFaeFb
         HMVoIemDSe0sHRzd+CS+wjrcCaRkhMM28gg5I=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=Po4vbqguTeShaFssL5deuezSfuT/sUGqgReUIk5k2lcH5VfGh89z1JfPrM2Lsr7Mjy
         CB02iY9McaIFvjagiwmaSSiHAo6+M8UTzto+MSzYCbDHp4MyyFLEeVfKzeCbl073y5JU
         UwKdK3aTZyVM/qgr4jkITKGrKPUqo/FkejNpE=
MIME-Version: 1.0
Received: by 10.216.86.19 with SMTP id v19mr3785250wee.89.1276023241572; Tue, 
	08 Jun 2010 11:54:01 -0700 (PDT)
Received: by 10.216.170.202 with HTTP; Tue, 8 Jun 2010 11:54:01 -0700 (PDT)
Date: Tue, 8 Jun 2010 11:54:01 -0700
Message-ID: <AANLkTimg-JqY7UhaxGeLjNMhjMcdrYZ5-aPMjneaqxiw@mail.gmail.com>
Subject: How to apply RDBMS table updates and deletes into Hadoop
From: atreju <n.atreju@gmail.com>
To: general@hadoop.apache.org, hive-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d99bfe0a89d404888950bd

--0016e6d99bfe0a89d404888950bd
Content-Type: text/plain; charset=ISO-8859-1

To generate smart output from base data we need to copy some base tables
from relational database into Hadoop. Some of them are big. To dump the
entire table into Hadoop everyday is not an option since there are like 30+
tables and each would take several hours.

The methodology that we approached is to get the entire table dump first.
Then each day or every 4-6 hours get only insert/update/delete since the
last copy from RDBMS (based on a date field in the table). Using Hive do
outer join + union the new data with existing data and write into a new
file. For example, if there are a 100 rows in Hadoop, and in RDBMS 3 records
inserted, 2 records updated and 1 deleted since the last Hadoop copy, then
the Hive query will get 97 of the not changed data + 3 inserts + 2 updates
and write into a new file. The other applications like Pig or Hive will pick
the most recent file to use when selecting/loading data from those base
table data files.

This logic is working fine in lower environments for small size tables. With
production data, for about 30GB size table, the incremental re-generation of
the file in Hadoop is still taking several hours. I tried using zipped
version and it took even longer time. I am not convinced that this is the
best we can do to handle updates and deletes since we had to re-write 29GB
unchanged data of the 30GB file again into a new file. ...and this is not
the biggest table.

I am thinking that this should be problem for many companies. What are the
other approaches to apply updates and deletes on base tables to the
Hadoop data files?

We have 4 data nodes and using version 20.3.

Thanks!

--0016e6d99bfe0a89d404888950bd--

From general-return-1627-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 08 22:59:39 2010
Return-Path: <general-return-1627-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43717 invoked from network); 8 Jun 2010 22:59:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jun 2010 22:59:38 -0000
Received: (qmail 80988 invoked by uid 500); 8 Jun 2010 22:59:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80902 invoked by uid 500); 8 Jun 2010 22:59:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80886 invoked by uid 99); 8 Jun 2010 22:59:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 22:59:37 +0000
X-ASF-Spam-Status: No, hits=1.0 required=10.0
	tests=AWL,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 22:59:32 +0000
Received: by gyf1 with SMTP id 1so5568315gyf.35
        for <multiple recipients>; Tue, 08 Jun 2010 15:59:10 -0700 (PDT)
Received: by 10.229.224.66 with SMTP id in2mr3681898qcb.232.1276037950495; 
	Tue, 08 Jun 2010 15:59:10 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.220.50.24 with HTTP; Tue, 8 Jun 2010 15:58:50 -0700 (PDT)
In-Reply-To: <AANLkTimg-JqY7UhaxGeLjNMhjMcdrYZ5-aPMjneaqxiw@mail.gmail.com>
References: <AANLkTimg-JqY7UhaxGeLjNMhjMcdrYZ5-aPMjneaqxiw@mail.gmail.com>
From: Aaron Kimball <aaron@cloudera.com>
Date: Wed, 9 Jun 2010 00:58:50 +0200
Message-ID: <AANLkTinRMxozdnKOXBlSwHFFW8NIrlg5f4Y31T7HVW2E@mail.gmail.com>
Subject: Re: How to apply RDBMS table updates and deletes into Hadoop
To: hive-user@hadoop.apache.org
Cc: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016363b80dec25d8c04888cbc58

--0016363b80dec25d8c04888cbc58
Content-Type: text/plain; charset=ISO-8859-1

I think that this might be the way to go. In general, folding updates and
deletes into datasets is a difficult problem due to the append-only nature
of datasets.

Something that might help you here is to partition your tables in Hive based
on some well-distributed key. Then if you have a relatively small number of
partitions affected by an incremental import (perhaps more recently-imported
records are more likely to be updated? in this case, partition the tables by
the month/week you imported them?) you can only perform the fold-in of the
new deltas on the affected partitions. This should be much faster than a
full table scan.

Have you seen the Sqoop tool? It handles imports and exports between HDFS
(and Hive) and RDBMS systems --  but currently can only import new records
(and subsequent INSERTs); it can't handle updates/deletes. Sqoop is
available at http://github.com/cloudera/sqoop -- it doesn't run on Apache
0.20.3, but works on CDH (Cloudera's Distribution for Hadoop) and Hadoop
0.21/trunk.

This sort of capability is something I'm really interested in adding to
Sqoop. If you've got a well-run process for doing this, I'd really
appreciate your help adding this feature :) Send me an email off-list if
you're interested. At the very least, I'd urge you to try out the tool.

Cheers,
- Aaron Kimball

On Tue, Jun 8, 2010 at 8:54 PM, atreju <n.atreju@gmail.com> wrote:

> To generate smart output from base data we need to copy some base tables
> from relational database into Hadoop. Some of them are big. To dump the
> entire table into Hadoop everyday is not an option since there are like 30+
> tables and each would take several hours.
>
> The methodology that we approached is to get the entire table dump first.
> Then each day or every 4-6 hours get only insert/update/delete since the
> last copy from RDBMS (based on a date field in the table). Using Hive do
> outer join + union the new data with existing data and write into a new
> file. For example, if there are a 100 rows in Hadoop, and in RDBMS 3 records
> inserted, 2 records updated and 1 deleted since the last Hadoop copy, then
> the Hive query will get 97 of the not changed data + 3 inserts + 2 updates
> and write into a new file. The other applications like Pig or Hive will pick
> the most recent file to use when selecting/loading data from those base
> table data files.
>
> This logic is working fine in lower environments for small size tables.
> With production data, for about 30GB size table, the incremental
> re-generation of the file in Hadoop is still taking several hours. I tried
> using zipped version and it took even longer time. I am not convinced that
> this is the best we can do to handle updates and deletes since we had to
> re-write 29GB unchanged data of the 30GB file again into a new file. ...and
> this is not the biggest table.
>
> I am thinking that this should be problem for many companies. What are the
> other approaches to apply updates and deletes on base tables to the
> Hadoop data files?
>
> We have 4 data nodes and using version 20.3.
>
> Thanks!
>
>

--0016363b80dec25d8c04888cbc58--

From general-return-1628-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 09 10:41:30 2010
Return-Path: <general-return-1628-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 36092 invoked from network); 9 Jun 2010 10:41:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jun 2010 10:41:30 -0000
Received: (qmail 72887 invoked by uid 500); 9 Jun 2010 10:41:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72520 invoked by uid 500); 9 Jun 2010 10:41:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72512 invoked by uid 99); 9 Jun 2010 10:41:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 10:41:25 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [156.148.72.33] (HELO raffaello.crs4.it) (156.148.72.33)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 10:41:17 +0000
Received: from pflip (pflip.crs4.it [156.148.70.62])
	by raffaello.crs4.it (Postfix) with SMTP id 7A4236BB9F
	for <general@hadoop.apache.org>; Wed,  9 Jun 2010 12:40:37 +0200 (CEST)
Received: by pflip (sSMTP sendmail emulation); Wed, 09 Jun 2010 12:40:55 +0200
Subject: [PROPOSAL]  Hadoop Italy User Group
From: Gianluigi Zanetti <gianluigi.zanetti@crs4.it>
Reply-To: gianluigi.zanetti@crs4.it
To: general@hadoop.apache.org
Content-Type: text/plain; charset="ISO-8859-15"
Content-Transfer-Encoding: quoted-printable
Organization: CRS4
Date: Wed, 09 Jun 2010 12:40:55 +0200
Message-ID: <1276080055.7601.41.camel@pflip>
Mime-Version: 1.0
X-Mailer: Evolution 2.28.3.1 
X-Virus-Checked: Checked by ClamAV on apache.org


Here at CSR4  we are actively using Hadoop for multiple purposes,
ranging from bioinformatics applications connected to the analysis of
deep sequencing datasets and large scale Genome-Wide Association Studies
to  exotic applications such as  using hdfs as a scalable file system to
support the deployment and management of (large) HPC virtual clusters.=20

We would like to establish contacts with other group in Italy to be able
to share experiences and ideas. A reasonable mechanism for this could be
an Italian User group. Please e-mail me if you like the idea, so that we
can assess whether there is enough interest in setting this up.

Gianluigi Zanetti
(gianluigi.zanetti@crs4.it)

From general-return-1629-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 09 16:06:13 2010
Return-Path: <general-return-1629-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53956 invoked from network); 9 Jun 2010 16:06:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jun 2010 16:06:13 -0000
Received: (qmail 92156 invoked by uid 500); 9 Jun 2010 16:06:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92075 invoked by uid 500); 9 Jun 2010 16:06:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 92067 invoked by uid 99); 9 Jun 2010 16:06:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 16:06:11 +0000
X-ASF-Spam-Status: No, hits=-0.1 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of swatt@us.ibm.com designates 32.97.182.141 as permitted sender)
Received: from [32.97.182.141] (HELO e1.ny.us.ibm.com) (32.97.182.141)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 16:06:01 +0000
Received: from d01relay01.pok.ibm.com (d01relay01.pok.ibm.com [9.56.227.233])
	by e1.ny.us.ibm.com (8.14.4/8.13.1) with ESMTP id o59FxibU010988
	for <general@hadoop.apache.org>; Wed, 9 Jun 2010 11:59:44 -0400
Received: from d03av04.boulder.ibm.com (d03av04.boulder.ibm.com [9.17.195.170])
	by d01relay01.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id o59G5cDU074364
	for <general@hadoop.apache.org>; Wed, 9 Jun 2010 12:05:39 -0400
Received: from d03av04.boulder.ibm.com (loopback [127.0.0.1])
	by d03av04.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id o59G5Xtb027863
	for <general@hadoop.apache.org>; Wed, 9 Jun 2010 10:05:33 -0600
Received: from d03nm123.boulder.ibm.com (d03nm123.boulder.ibm.com [9.17.195.149])
	by d03av04.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id o59G5XND027802
	for <general@hadoop.apache.org>; Wed, 9 Jun 2010 10:05:33 -0600
In-Reply-To: <1276080055.7601.41.camel@pflip>
References: <1276080055.7601.41.camel@pflip>
To: general@hadoop.apache.org
MIME-Version: 1.0
Subject: Re: [PROPOSAL]  Hadoop Italy User Group
X-KeepSent: 63D07D95:A1B5BC99-8625773D:005820BA;
 type=4; name=$KeepSent
X-Mailer: Lotus Notes Release 8.5 December 05, 2008
Message-ID: <OF63D07D95.A1B5BC99-ON8625773D.005820BA-8625773D.0058661C@us.ibm.com>
From: Stephen Watt <swatt@us.ibm.com>
Date: Wed, 9 Jun 2010 11:05:31 -0500
X-MIMETrack: Serialize by Router on D03NM123/03/M/IBM(Release 8.5.1HF41 | October 22, 2009) at
 06/09/2010 10:05:32,
	Serialize complete at 06/09/2010 10:05:32
Content-Type: multipart/alternative; boundary="=_alternative 0058661B8625773D_="
X-Virus-Checked: Checked by ClamAV on apache.org

--=_alternative 0058661B8625773D_=
Content-Type: text/plain; charset="US-ASCII"

Hi Gianluigi

I'm glad to hear you are starting a new user group. Welcome to the fold. 
If you have any questions about running a group feel free to shoot me a 
note. Dekel Tankel who runs the Bay Area Group helped me out with my 
questions when we started ours. He's a great resource.

Steve Watt - Austin Hadoop User Group Coordinator



From:
Gianluigi Zanetti <gianluigi.zanetti@crs4.it>
To:
general@hadoop.apache.org
Date:
06/09/2010 05:41 AM
Subject:
[PROPOSAL]  Hadoop Italy User Group




Here at CSR4  we are actively using Hadoop for multiple purposes,
ranging from bioinformatics applications connected to the analysis of
deep sequencing datasets and large scale Genome-Wide Association Studies
to  exotic applications such as  using hdfs as a scalable file system to
support the deployment and management of (large) HPC virtual clusters. 

We would like to establish contacts with other group in Italy to be able
to share experiences and ideas. A reasonable mechanism for this could be
an Italian User group. Please e-mail me if you like the idea, so that we
can assess whether there is enough interest in setting this up.

Gianluigi Zanetti
(gianluigi.zanetti@crs4.it)



--=_alternative 0058661B8625773D_=--

From general-return-1630-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 09 22:33:07 2010
Return-Path: <general-return-1630-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 11989 invoked from network); 9 Jun 2010 22:33:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jun 2010 22:33:07 -0000
Received: (qmail 37764 invoked by uid 500); 9 Jun 2010 22:33:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37452 invoked by uid 500); 9 Jun 2010 22:33:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37224 invoked by uid 99); 9 Jun 2010 22:33:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 22:33:05 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of n.atreju@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 22:32:59 +0000
Received: by wwe15 with SMTP id 15so1664263wwe.35
        for <multiple recipients>; Wed, 09 Jun 2010 15:32:38 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:cc:content-type;
        bh=Uht1AzCRNKuI8WwYXrUhG7oB6WRasK0Fb/jnmwkeSMM=;
        b=wsww5s+feyMi7uwAp2Yn/KaDgBwgTWJtDgNS8yl0F0rIYQLOJaK8TmHkNlatW0T/Qc
         fiOg1HWQeB2UBtQp0okCy5A8dl0x8huOLwLY40SZw28hjhLHvaY6WSnAIys90Xc8qcqw
         dp8SYPbqRhDd0bDstfAu0E+Omx7xbRsGqHTO0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=c0y9UPhs/x1lDW/9K9B8uErCZEAybtA/Xba0At8KvtJ9tklfZXZaiEwE8Iha0me6JR
         ylnjfC4iFkytwrtkvQQD2u5Cw2wmQOUoJpKQO2eqUL3lhsgeugNq4NHDvKrX+lrfgjKj
         gHi7GXabMIDFpFnxJuDqbggDrmx7q/DkqQQQY=
MIME-Version: 1.0
Received: by 10.227.147.193 with SMTP id m1mr3377685wbv.23.1276122758639; Wed, 
	09 Jun 2010 15:32:38 -0700 (PDT)
Received: by 10.216.170.202 with HTTP; Wed, 9 Jun 2010 15:32:38 -0700 (PDT)
In-Reply-To: <AANLkTinRMxozdnKOXBlSwHFFW8NIrlg5f4Y31T7HVW2E@mail.gmail.com>
References: <AANLkTimg-JqY7UhaxGeLjNMhjMcdrYZ5-aPMjneaqxiw@mail.gmail.com>
	<AANLkTinRMxozdnKOXBlSwHFFW8NIrlg5f4Y31T7HVW2E@mail.gmail.com>
Date: Wed, 9 Jun 2010 15:32:38 -0700
Message-ID: <AANLkTilrMOA-gTb4D6R1lrrwz1Rtx3dMgdAQiL1O5G9V@mail.gmail.com>
Subject: Re: How to apply RDBMS table updates and deletes into Hadoop
From: atreju <n.atreju@gmail.com>
To: hive-user@hadoop.apache.org
Cc: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636831762b7f3520488a07b75
X-Virus-Checked: Checked by ClamAV on apache.org

--001636831762b7f3520488a07b75
Content-Type: text/plain; charset=ISO-8859-1

As an ideal solution, I have a suggestion to Hive contributors to make it
look like Hive is doing insert/update/delete:


This will require a special type of table creation syntax. I will call it as
"UPDATABLE TABLE". The table will have 3 special columns that are defined in
the create script:
1. The primary key column. (let's say: col_pk)
2. BIGINT type date column that shows the ms from Jan 1st, 1970 to actual
data manipulation date/time in RDMBS. (dml_date)
3. TINYINT or BOOLEAN type column that will store 0 if the record is deleted
and 1 if it is inserted or updated. (dml_action)

This will require the RDBMS table to have PK and last update date column and
deletes recorded in some other table by pk and date.

On Day 1, the entire table is put into Hadoop, with addition of 2 extra
columns: dml_date (bigint) and dml_action.

On Day 2, we first find the max of dml_date from Hive table. Then we query
from RDBMS inserts/updates/deletes since that date/time and write into a
file with the correct dml_date/dml_action. The file goes to the same folder
that our Hive table is in.

Right now, if on Day 1 we had 100 rows and on Day 2, 10 rows a regular Hive
table would show 110 rows. But, since this is a special table (UPDATABLE
TABLE), every time this table is queried in Hive, Hive first run a
map-reduce that would find the most recent (max(dml_date)) row per pk (group
by col_pk) that is not deleted (dml_action!=0) and use that output in the
user's query. That is the big idea!!

Hive can have Insert/Update/Delete commands that would do nothing but create
a file with rows of manipulated data with correct date and action.

There can be a special "flush" kind of command that runs the MR and replaces
all files in the table directory with single file. That can run weekly,
monthly or may be after each time dml data received from RDBMS.

Sqoop can have Hive interface that saves certain table attributes like pk
column, RDBMS connection info,... and with one command from Hive, the Hive
table gets updated from RDBMS....

What do you think?



On Tue, Jun 8, 2010 at 3:58 PM, Aaron Kimball <aaron@cloudera.com> wrote:

> I think that this might be the way to go. In general, folding updates and
> deletes into datasets is a difficult problem due to the append-only nature
> of datasets.
>
> Something that might help you here is to partition your tables in Hive
> based on some well-distributed key. Then if you have a relatively small
> number of partitions affected by an incremental import (perhaps more
> recently-imported records are more likely to be updated? in this case,
> partition the tables by the month/week you imported them?) you can only
> perform the fold-in of the new deltas on the affected partitions. This
> should be much faster than a full table scan.
>
> Have you seen the Sqoop tool? It handles imports and exports between HDFS
> (and Hive) and RDBMS systems --  but currently can only import new records
> (and subsequent INSERTs); it can't handle updates/deletes. Sqoop is
> available at http://github.com/cloudera/sqoop -- it doesn't run on Apache
> 0.20.3, but works on CDH (Cloudera's Distribution for Hadoop) and Hadoop
> 0.21/trunk.
>
> This sort of capability is something I'm really interested in adding to
> Sqoop. If you've got a well-run process for doing this, I'd really
> appreciate your help adding this feature :) Send me an email off-list if
> you're interested. At the very least, I'd urge you to try out the tool.
>
> Cheers,
> - Aaron Kimball
>
>
> On Tue, Jun 8, 2010 at 8:54 PM, atreju <n.atreju@gmail.com> wrote:
>
>> To generate smart output from base data we need to copy some base tables
>> from relational database into Hadoop. Some of them are big. To dump the
>> entire table into Hadoop everyday is not an option since there are like 30+
>> tables and each would take several hours.
>>
>> The methodology that we approached is to get the entire table dump first.
>> Then each day or every 4-6 hours get only insert/update/delete since the
>> last copy from RDBMS (based on a date field in the table). Using Hive do
>> outer join + union the new data with existing data and write into a new
>> file. For example, if there are a 100 rows in Hadoop, and in RDBMS 3 records
>> inserted, 2 records updated and 1 deleted since the last Hadoop copy, then
>> the Hive query will get 97 of the not changed data + 3 inserts + 2 updates
>> and write into a new file. The other applications like Pig or Hive will pick
>> the most recent file to use when selecting/loading data from those base
>> table data files.
>>
>> This logic is working fine in lower environments for small size tables.
>> With production data, for about 30GB size table, the incremental
>> re-generation of the file in Hadoop is still taking several hours. I tried
>> using zipped version and it took even longer time. I am not convinced that
>> this is the best we can do to handle updates and deletes since we had to
>> re-write 29GB unchanged data of the 30GB file again into a new file. ...and
>> this is not the biggest table.
>>
>> I am thinking that this should be problem for many companies. What are the
>> other approaches to apply updates and deletes on base tables to the
>> Hadoop data files?
>>
>> We have 4 data nodes and using version 20.3.
>>
>> Thanks!
>>
>>
>
>

--001636831762b7f3520488a07b75--

From general-return-1631-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 09 23:57:18 2010
Return-Path: <general-return-1631-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39385 invoked from network); 9 Jun 2010 23:57:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jun 2010 23:57:18 -0000
Received: (qmail 3002 invoked by uid 500); 9 Jun 2010 23:57:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2955 invoked by uid 500); 9 Jun 2010 23:57:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2945 invoked by uid 99); 9 Jun 2010 23:57:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 23:57:17 +0000
X-ASF-Spam-Status: No, hits=2.1 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com designates 74.125.83.48 as permitted sender)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 23:57:12 +0000
Received: by gwb17 with SMTP id 17so5834315gwb.35
        for <general@hadoop.apache.org>; Wed, 09 Jun 2010 16:56:51 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=KanpSzdNAIOs0U4BW9fK4LUNk9Z5f6nY7AdHtnGkAC0=;
        b=AF0i53Q96zAkDzC9C79ltiFhdptk0Bh2fmwClhsh4dftYYAt62I1mftIolnVe6/RRj
         D0N4BG1dIWqI7OG2Dk7QH4Ozw2+pZl49FjOfokQHVOcz3rX0f8XU8q99WULvCovu0lf2
         43XuSYng82iReZqmhoJu+nuoRSbZ6Aj1lKeWA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Ak8w+X0kUb8P3MWsbpc8/3ejSvAw95y06P3amyb7YLlN9glke5uYi7tuUV075+Go7c
         1587FsqUs6KoPejMqLQQ1PjdnFmtwoUZ6NTczf1QlLNXyhC4gYoNKCufEvzX7WGQDclA
         ZPgFvV+JMaqf1a1U7uTOdbMg47wY46dicacIE=
MIME-Version: 1.0
Received: by 10.101.128.32 with SMTP id f32mr19067624ann.93.1276127811428; 
	Wed, 09 Jun 2010 16:56:51 -0700 (PDT)
Received: by 10.100.240.9 with HTTP; Wed, 9 Jun 2010 16:56:51 -0700 (PDT)
In-Reply-To: <AANLkTilrMOA-gTb4D6R1lrrwz1Rtx3dMgdAQiL1O5G9V@mail.gmail.com>
References: <AANLkTimg-JqY7UhaxGeLjNMhjMcdrYZ5-aPMjneaqxiw@mail.gmail.com>
	<AANLkTinRMxozdnKOXBlSwHFFW8NIrlg5f4Y31T7HVW2E@mail.gmail.com>
	<AANLkTilrMOA-gTb4D6R1lrrwz1Rtx3dMgdAQiL1O5G9V@mail.gmail.com>
Date: Wed, 9 Jun 2010 16:56:51 -0700
Message-ID: <AANLkTilGKOg7SPP_X1rurnhvWOgTe1O1aO8g4wI_Qgki@mail.gmail.com>
Subject: Re: How to apply RDBMS table updates and deletes into Hadoop
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c5c05fe361910488a1a84c

--001636c5c05fe361910488a1a84c
Content-Type: text/plain; charset=ISO-8859-1

When hive is running the map-reduce job, how do we handle concurrent
update/deletion/insertion ?

On Wed, Jun 9, 2010 at 3:32 PM, atreju <n.atreju@gmail.com> wrote:

> As an ideal solution, I have a suggestion to Hive contributors to make it
> look like Hive is doing insert/update/delete:
>
>
> This will require a special type of table creation syntax. I will call it
> as
> "UPDATABLE TABLE". The table will have 3 special columns that are defined
> in
> the create script:
> 1. The primary key column. (let's say: col_pk)
> 2. BIGINT type date column that shows the ms from Jan 1st, 1970 to actual
> data manipulation date/time in RDMBS. (dml_date)
> 3. TINYINT or BOOLEAN type column that will store 0 if the record is
> deleted
> and 1 if it is inserted or updated. (dml_action)
>
> This will require the RDBMS table to have PK and last update date column
> and
> deletes recorded in some other table by pk and date.
>
> On Day 1, the entire table is put into Hadoop, with addition of 2 extra
> columns: dml_date (bigint) and dml_action.
>
> On Day 2, we first find the max of dml_date from Hive table. Then we query
> from RDBMS inserts/updates/deletes since that date/time and write into a
> file with the correct dml_date/dml_action. The file goes to the same folder
> that our Hive table is in.
>
> Right now, if on Day 1 we had 100 rows and on Day 2, 10 rows a regular Hive
> table would show 110 rows. But, since this is a special table (UPDATABLE
> TABLE), every time this table is queried in Hive, Hive first run a
> map-reduce that would find the most recent (max(dml_date)) row per pk
> (group
> by col_pk) that is not deleted (dml_action!=0) and use that output in the
> user's query. That is the big idea!!
>
> Hive can have Insert/Update/Delete commands that would do nothing but
> create
> a file with rows of manipulated data with correct date and action.
>
> There can be a special "flush" kind of command that runs the MR and
> replaces
> all files in the table directory with single file. That can run weekly,
> monthly or may be after each time dml data received from RDBMS.
>
> Sqoop can have Hive interface that saves certain table attributes like pk
> column, RDBMS connection info,... and with one command from Hive, the Hive
> table gets updated from RDBMS....
>
> What do you think?
>
>
>
> On Tue, Jun 8, 2010 at 3:58 PM, Aaron Kimball <aaron@cloudera.com> wrote:
>
> > I think that this might be the way to go. In general, folding updates and
> > deletes into datasets is a difficult problem due to the append-only
> nature
> > of datasets.
> >
> > Something that might help you here is to partition your tables in Hive
> > based on some well-distributed key. Then if you have a relatively small
> > number of partitions affected by an incremental import (perhaps more
> > recently-imported records are more likely to be updated? in this case,
> > partition the tables by the month/week you imported them?) you can only
> > perform the fold-in of the new deltas on the affected partitions. This
> > should be much faster than a full table scan.
> >
> > Have you seen the Sqoop tool? It handles imports and exports between HDFS
> > (and Hive) and RDBMS systems --  but currently can only import new
> records
> > (and subsequent INSERTs); it can't handle updates/deletes. Sqoop is
> > available at http://github.com/cloudera/sqoop -- it doesn't run on
> Apache
> > 0.20.3, but works on CDH (Cloudera's Distribution for Hadoop) and Hadoop
> > 0.21/trunk.
> >
> > This sort of capability is something I'm really interested in adding to
> > Sqoop. If you've got a well-run process for doing this, I'd really
> > appreciate your help adding this feature :) Send me an email off-list if
> > you're interested. At the very least, I'd urge you to try out the tool.
> >
> > Cheers,
> > - Aaron Kimball
> >
> >
> > On Tue, Jun 8, 2010 at 8:54 PM, atreju <n.atreju@gmail.com> wrote:
> >
> >> To generate smart output from base data we need to copy some base tables
> >> from relational database into Hadoop. Some of them are big. To dump the
> >> entire table into Hadoop everyday is not an option since there are like
> 30+
> >> tables and each would take several hours.
> >>
> >> The methodology that we approached is to get the entire table dump
> first.
> >> Then each day or every 4-6 hours get only insert/update/delete since the
> >> last copy from RDBMS (based on a date field in the table). Using Hive do
> >> outer join + union the new data with existing data and write into a new
> >> file. For example, if there are a 100 rows in Hadoop, and in RDBMS 3
> records
> >> inserted, 2 records updated and 1 deleted since the last Hadoop copy,
> then
> >> the Hive query will get 97 of the not changed data + 3 inserts + 2
> updates
> >> and write into a new file. The other applications like Pig or Hive will
> pick
> >> the most recent file to use when selecting/loading data from those base
> >> table data files.
> >>
> >> This logic is working fine in lower environments for small size tables.
> >> With production data, for about 30GB size table, the incremental
> >> re-generation of the file in Hadoop is still taking several hours. I
> tried
> >> using zipped version and it took even longer time. I am not convinced
> that
> >> this is the best we can do to handle updates and deletes since we had to
> >> re-write 29GB unchanged data of the 30GB file again into a new file.
> ...and
> >> this is not the biggest table.
> >>
> >> I am thinking that this should be problem for many companies. What are
> the
> >> other approaches to apply updates and deletes on base tables to the
> >> Hadoop data files?
> >>
> >> We have 4 data nodes and using version 20.3.
> >>
> >> Thanks!
> >>
> >>
> >
> >
>

--001636c5c05fe361910488a1a84c--

From general-return-1632-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 00:30:08 2010
Return-Path: <general-return-1632-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60734 invoked from network); 10 Jun 2010 00:30:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 00:30:08 -0000
Received: (qmail 27558 invoked by uid 500); 10 Jun 2010 00:30:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27472 invoked by uid 500); 10 Jun 2010 00:30:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27456 invoked by uid 99); 10 Jun 2010 00:30:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 00:30:06 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of n.atreju@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 00:29:58 +0000
Received: by wyb29 with SMTP id 29so3649164wyb.35
        for <multiple recipients>; Wed, 09 Jun 2010 17:29:35 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=K0keWim3Zx7GXNsbPKunmAWwQf0F/VrNuVPymgOsQrc=;
        b=mhaRULLwik3HP/N2ftN6RBPUZ468KZZlHCuVvTmUqFf5Du6Ogbs4WGQE34phiLm+Sr
         dNhFmNGz79M6P8D65B0IQVbnX9ZT3F31EYh2RkEEfZRWPJCemIjXSDgOhf4gwURu9vo5
         CFRgJJhi1GW9juNTwocxFBne2wRDTpb5WXXwQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=r88xoasrydUy20NSwJ7l+w5YbFy338KKgKGUy+Ks7gy9v4F8mFY05My+Kft3Is6oO3
         I/pVDr1uU6xw2zT91Cp7fpDGVGZ0ekMLvf8k+EkeXapSiERsiCvrjh0eIptQAMLM6lCe
         Qt+8IcQmM+waL+knOdvlJ8PYZ5FoUsIIeJ39M=
MIME-Version: 1.0
Received: by 10.227.146.3 with SMTP id f3mr2665995wbv.211.1276129775600; Wed, 
	09 Jun 2010 17:29:35 -0700 (PDT)
Received: by 10.216.170.202 with HTTP; Wed, 9 Jun 2010 17:29:35 -0700 (PDT)
In-Reply-To: <AANLkTilGKOg7SPP_X1rurnhvWOgTe1O1aO8g4wI_Qgki@mail.gmail.com>
References: <AANLkTimg-JqY7UhaxGeLjNMhjMcdrYZ5-aPMjneaqxiw@mail.gmail.com>
	<AANLkTinRMxozdnKOXBlSwHFFW8NIrlg5f4Y31T7HVW2E@mail.gmail.com>
	<AANLkTilrMOA-gTb4D6R1lrrwz1Rtx3dMgdAQiL1O5G9V@mail.gmail.com>
	<AANLkTilGKOg7SPP_X1rurnhvWOgTe1O1aO8g4wI_Qgki@mail.gmail.com>
Date: Wed, 9 Jun 2010 17:29:35 -0700
Message-ID: <AANLkTimnHIknkk7zPmNwnGlz5Tp8Y0u2tSx7e5UMpjxN@mail.gmail.com>
Subject: Re: How to apply RDBMS table updates and deletes into Hadoop
From: atreju <n.atreju@gmail.com>
To: general@hadoop.apache.org, hive-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364ef8cef646180488a21d86
X-Virus-Checked: Checked by ClamAV on apache.org

--0016364ef8cef646180488a21d86
Content-Type: text/plain; charset=ISO-8859-1

Insert/Update/Delete is nothing but "put" command for another file to the
same directory. Only problem is during "flush" that would replace the files.
I assume it would use the similar kind of logic of Hive's "insert overwrite"
(create the file in a temporary space and replace the Hive file(s) when MR
output is ready). Only for that "replace" (move command?) the flush has to
talk to Namenode to wait for currently running MR jobs to finish and put
others on hold until the file is replaced. That is of course the high level
idea. I am not sure if it is practical.


On Wed, Jun 9, 2010 at 4:56 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> When hive is running the map-reduce job, how do we handle concurrent
> update/deletion/insertion ?
>
>

--0016364ef8cef646180488a21d86--

From general-return-1633-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 10:44:26 2010
Return-Path: <general-return-1633-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95132 invoked from network); 10 Jun 2010 10:44:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 10:44:25 -0000
Received: (qmail 6083 invoked by uid 500); 10 Jun 2010 10:44:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5568 invoked by uid 500); 10 Jun 2010 10:44:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5555 invoked by uid 99); 10 Jun 2010 10:44:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 10:44:20 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ashahzad4@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 10:44:13 +0000
Received: by wwe15 with SMTP id 15so2130238wwe.35
        for <general@hadoop.apache.org>; Thu, 10 Jun 2010 03:43:52 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=nwIAwtwy/IYRD2SYCD3usjmQhEYc7dztsnNnpzOY4JA=;
        b=fBj3ROLuWqh1E4HkS6EtfHY/qB2n1TyU8ARAjV7EZkA1IUXFMMd0uB5iZu4zytFlG0
         bOkdfNyv5Dq70nXc1FzUk1IEgbSkwBPvt8JUN65ppgkjZzBD/V5VzxaXLG2FBHpmrLlf
         5xsuv0lzNFfpqEyFABEG3p8z3LnuEyMGWjUxA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Jb5Z60bVMktY3jYRNhFr0W+9WPScgFM8Syi7IbYiAzAad8Tj4GuwRJmwAw3lK7yrKc
         n29JQJLXmzDKFMk434bqm87vlGQnObuxEPyEOR5xBPBNCpSjw5NleTQu/KKUCYmE3vhg
         s1K//hlVGRF/eiUnrED80nS1hxKIZXMI1UBAA=
MIME-Version: 1.0
Received: by 10.227.151.143 with SMTP id c15mr4659050wbw.135.1276166631706; 
	Thu, 10 Jun 2010 03:43:51 -0700 (PDT)
Received: by 10.216.186.19 with HTTP; Thu, 10 Jun 2010 03:43:51 -0700 (PDT)
In-Reply-To: <CEF6BDC7-2E4A-4E54-84D2-401E034F1A3D@apache.org>
References: <AANLkTiksK--b_qosOGjIbNQvKqCZN-33xj4eQWbOs8CK@mail.gmail.com>
	<CEF6BDC7-2E4A-4E54-84D2-401E034F1A3D@apache.org>
Date: Thu, 10 Jun 2010 12:43:51 +0200
Message-ID: <AANLkTilbzvj5fUOTkgH6PBGy56_gsVMnBo0yRHstL99q@mail.gmail.com>
Subject: Re: Is it possible ....!!!
From: Ahmad Shahzad <ashahzad4@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e657b216c1d80b0488aab2be
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e657b216c1d80b0488aab2be
Content-Type: text/plain; charset=ISO-8859-1

Hi OWEN,
                Can you please explain it further. What do you mean by "set
up socket proxies for setting up rpc connections." . I would really
appreciate your reply

Regards,
Ahmad Shahzad

On Tue, Jun 8, 2010 at 5:13 PM, Owen O'Malley <omalley@apache.org> wrote:

>
> On Jun 8, 2010, at 4:30 AM, Ahmad Shahzad wrote:
>
>  Hi,
>>   I wanted to ask if it is possible to intercept every communication that
>> takes place between hadoop's map reduce task i.e between JobTracker and
>> TaskTracker and make it pass through my own communication library.
>>
>
> It is possible to set up socket proxies for setting up rpc connections.
> Please take this thread over to common-user@hadoop.apache.org.
>
> -- Owen
>

--0016e657b216c1d80b0488aab2be--

From general-return-1634-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 17:38:59 2010
Return-Path: <general-return-1634-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35620 invoked from network); 10 Jun 2010 17:38:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 17:38:59 -0000
Received: (qmail 57658 invoked by uid 500); 10 Jun 2010 17:38:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57530 invoked by uid 500); 10 Jun 2010 17:38:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57522 invoked by uid 99); 10 Jun 2010 17:38:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 17:38:57 +0000
X-ASF-Spam-Status: No, hits=2.3 required=10.0
	tests=AWL,HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 17:38:51 +0000
Received: by yxd39 with SMTP id 39so83846yxd.35
        for <general@hadoop.apache.org>; Thu, 10 Jun 2010 10:38:30 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.150.142.19 with SMTP id p19mr2077372ybd.44.1276191510573; Thu, 
	10 Jun 2010 10:38:30 -0700 (PDT)
Received: by 10.231.146.15 with HTTP; Thu, 10 Jun 2010 10:38:30 -0700 (PDT)
X-Originating-IP: [208.252.10.190]
Date: Thu, 10 Jun 2010 13:38:30 -0400
Message-ID: <AANLkTilDQ0iYlh4UcdwUwl2Cp8rbxNp8IUQCNLrMm6_0@mail.gmail.com>
Subject: Using S3 native filesystem? (s3n://)
From: Denis Haskin <denis@haskinferguson.net>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd7547ea735a80488b07d22

--000e0cd7547ea735a80488b07d22
Content-Type: text/plain; charset=ISO-8859-1

I'm having trouble getting the S3 native filesystem working, trying to use
it from my local desktop.  Following http://wiki.apache.org/hadoop/AmazonS3,
I have my hdfs-site.xml as:

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
<property>
  <name>fs.default.name</name>
  <value>s3n://dwh-hdfs-test</value>
</property>
<property>
  <name>fs.s3n.awsAccessKeyId</name>
  <value>..access key...</value>
</property>
<property>
  <name>fs.s3n.awsSecretAccessKey</name>
  <value>...secret access key, with / encoded as %2F..</value>
</property>
</configuration>

"bin/hadoop namenode -format" works; I end up with this structure (as seen
by s3cmd):

2010-06-10 13:11         4
s3://dwh-hdfs-test/tmp/hadoop-dhaskin/mapred/system/jobtracker.info
2010-06-10 13:11         0
s3://dwh-hdfs-test/tmp/hadoop-dhaskin/mapred/system_$folder$
2010-06-10 13:10         0
s3://dwh-hdfs-test/tmp/hadoop-dhaskin/mapred_$folder$
2010-06-10 13:10         0   s3://dwh-hdfs-test/tmp/hadoop-dhaskin_$folder$
2010-06-10 13:10         0   s3://dwh-hdfs-test/tmp_$folder$

but when I try something like "bin/hadoop fs -ls" I get various errors.
 Part of the problem is it's sort of unclear what the format of the
parameter should be.  See what I tried and the various errors down below, at
[1]

Also, when I started up the jobtracker, it fails with:

2010-06-10 13:34:08,521 FATAL org.apache.hadoop.mapred.JobTracker:
org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException:
S3 HEAD request failed for '/tmp%2Fhadoop-dhaskin%2Fmapred%2Fsystem' -
ResponseCode=403, ResponseMessage=Forbidden

Suggestions?  Thanks...


[1] bin/hadoop fs -ls of various attempts:

dwhsix:hadoop-0.20.2 dhaskin$ bin/hadoop fs -ls /
ls: org.jets3t.service.S3ServiceException: S3 GET failed for '/' XML Error
Message: <?xml version="1.0"
encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>The
request signature we calculated does not match the signature you provided.
Check your key and signing method.</Message><StringToSignBytes>47 45 54 0a
0a 0a 54 68 75 2c 20 31 30 20 4a 75 6e 20 32 30 31 30 20 31 37 3a 33 30 3a
34 30 20 47 4d 54 0a 2f 64 77 68 2d 68 64 66 73 2d 74 65 73 74
2f</StringToSignBytes><RequestId>098E7E853DB93FD9</RequestId><HostId>Y8w3ccqhj4VvWY6Ma17o5HF+8cWK3r1kiIywxAsrwSsuR2DqFpxx4+2+9Xgnn+9i</HostId><SignatureProvided>w8vS4vJjm5MpkLpDbnuVcByruxw=</SignatureProvided><StringToSign>GETThu,
10 Jun 2010 17:30:40
GMT/dwh-hdfs-test/</StringToSign><AWSAccessKeyId>AKIAI5UTNXFARAYFZAXQ</AWSAccessKeyId></Error>

dwhsix:hadoop-0.20.2 dhaskin$ bin/hadoop fs -ls s3n://dwh-hdfs-test
ls: Path must be absolute: s3n://dwh-hdfs-test
Usage: java FsShell [-ls <path>]

dwhsix:hadoop-0.20.2 dhaskin$ bin/hadoop fs -ls s3n://dwh-hdfs-test/
ls: org.jets3t.service.S3ServiceException: S3 GET failed for '/' XML Error
Message: <?xml version="1.0"
encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>The
request signature we calculated does not match the signature you provided.
Check your key and signing method.</Message><StringToSignBytes>47 45 54 0a
0a 0a 54 68 75 2c 20 31 30 20 4a 75 6e 20 32 30 31 30 20 31 37 3a 33 30 3a
35 36 20 47 4d 54 0a 2f 64 77 68 2d 68 64 66 73 2d 74 65 73 74
2f</StringToSignBytes><RequestId>2F7004189009A56A</RequestId><HostId>IQzsA5849ZnBaqGuAsFvTIt78u9oDRaBvrY5Xwg5exf85H+7/aAejxK33QPLXCue</HostId><SignatureProvided>ryv+zgWT3YXPbqPcMNr0F4dcWKM=</SignatureProvided><StringToSign>GETThu,
10 Jun 2010 17:30:56
GMT/dwh-hdfs-test/</StringToSign><AWSAccessKeyId>AKIAI5UTNXFARAYFZAXQ</AWSAccessKeyId></Error>

dwhsix:hadoop-0.20.2 dhaskin$ bin/hadoop fs -ls s3n://dwh-hdfs-test/tmp
ls: org.jets3t.service.S3ServiceException: S3 HEAD request failed for '/tmp'
- ResponseCode=403, ResponseMessage=Forbidden

dwhsix:hadoop-0.20.2 dhaskin$ bin/hadoop fs -ls /tmp
ls: org.jets3t.service.S3ServiceException: S3 HEAD request failed for '/tmp'
- ResponseCode=403, ResponseMessage=Forbidden

--000e0cd7547ea735a80488b07d22--

From general-return-1635-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 18:45:10 2010
Return-Path: <general-return-1635-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63048 invoked from network); 10 Jun 2010 18:45:10 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 18:45:10 -0000
Received: (qmail 28695 invoked by uid 500); 10 Jun 2010 18:45:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28611 invoked by uid 500); 10 Jun 2010 18:45:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28596 invoked by uid 99); 10 Jun 2010 18:45:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 18:45:08 +0000
X-ASF-Spam-Status: No, hits=1.8 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of n.atreju@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 18:45:02 +0000
Received: by wwe15 with SMTP id 15so216770wwe.35
        for <multiple recipients>; Thu, 10 Jun 2010 11:44:40 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:cc:content-type;
        bh=QdlBue7x6nqIPDWMV34MalftuSISGqjfREwV4m4lmwA=;
        b=K8hU6s283OQWC4Tk7TPTJZPEl5uysAW1VQK1sT4AXKXEHScHM6JxcmGcexLiZqROQk
         l0184C9nufotTHqZiWg4Frdp8iWzRAhIx6MskL1aQ18+hJmEsc6HPNB2/aB//v01MEHX
         akLa3P6QvcjF3y+xudekM8ehGC9B3t8ASy/vI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=nHb7CGBuSUTr24adkcdgBx2ua8i6yVGuZasPMuFX7r4et8Gxn84I3Vu98+HyrJjxMl
         oPAiRlB3xPXeODpWaOcHgH8qumCLizQBmZ1KtAstQNHaR6tzLEzAWNJe767uF9imi2oK
         MbuF5XD2pVG7F6aKnRy1H/wZ8YZpT7+G8O7Nc=
MIME-Version: 1.0
Received: by 10.216.85.11 with SMTP id t11mr368724wee.55.1276195480313; Thu, 
	10 Jun 2010 11:44:40 -0700 (PDT)
Received: by 10.216.170.202 with HTTP; Thu, 10 Jun 2010 11:44:40 -0700 (PDT)
In-Reply-To: <AANLkTikGDNT5NIMq4NaeWCh_ES0moKqhHpt9ZbqK_iRz@mail.gmail.com>
References: <AANLkTimg-JqY7UhaxGeLjNMhjMcdrYZ5-aPMjneaqxiw@mail.gmail.com>
	<AANLkTinRMxozdnKOXBlSwHFFW8NIrlg5f4Y31T7HVW2E@mail.gmail.com>
	<AANLkTilrMOA-gTb4D6R1lrrwz1Rtx3dMgdAQiL1O5G9V@mail.gmail.com>
	<AANLkTilGKOg7SPP_X1rurnhvWOgTe1O1aO8g4wI_Qgki@mail.gmail.com>
	<AANLkTimnHIknkk7zPmNwnGlz5Tp8Y0u2tSx7e5UMpjxN@mail.gmail.com>
	<AANLkTikGDNT5NIMq4NaeWCh_ES0moKqhHpt9ZbqK_iRz@mail.gmail.com>
Date: Thu, 10 Jun 2010 11:44:40 -0700
Message-ID: <AANLkTikDWxdY1DbEgEyfL8yxMLPxJRGF3trWSz1sget_@mail.gmail.com>
Subject: Re: How to apply RDBMS table updates and deletes into Hadoop
From: atreju <n.atreju@gmail.com>
To: hive-user@hadoop.apache.org
Cc: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d9a25b44a1bd0488b16a95

--0016e6d9a25b44a1bd0488b16a95
Content-Type: text/plain; charset=ISO-8859-1

Thank you for your response. I understand... Just a few points before I
accept that this is too complicated :)

The main idea is to keep different versions of data under the same table,
similar to HBase but this is row level and you don't have to make the other
versions accessible from Hive but only the most recent one. You just need to
create an access layer to work on the most recent version of the row. If you
can think of a different way of uniquely identifying a row to know the
versions of it and timestamp (or counter or version #??) to know the most
recent one, it doesn't have to be the columns that I specified before. It
can be a different file that you create in the background (which can also be
the index file!!).

Oracle has ROWID for physical location of the row and locks it before the
data manipulation. Hadoop has advantage of storage and map-reduce. So why
not use it and keep all versions of changed data and access it via
map-reduce for the most recent one. Accessing the data can get slower over
time when there are many versions. And that can be fixed with flush or full
replication of data time to time in a maintenance window by the end user.

Hive is a great tool to access and manipulate Hadoop files. You are doing an
amazing job. I have no idea what are the complications you face each day.
Just disregard if I am talking nonsense to you keep up the good work!

Cheers!



Atreju,

>
> Your work is great. Personally I would not get too tied up in the
> transactional side of hive. Once you start dealing with locking and
> concurrency the problem becomes tricky.
>
> We hivers have a long time tradition on 'punting' on complicated stuff we
> do not want to deal with. :) Thus we only have 'Insert Overwrite' no 'insert
> update' :)
>
> Again, I think you wrote a really cool application. It would make a great
> use case, blog post, or a stand alone application. Call it HiveMysqlRsync or
> something :). However you mention several requirements that are specific to
> your application timestamp and primary key. If you can abstract all your
> application specific logic it could make it's way into hive. But it might be
> a stand alone program because hive to rdbms replication might be a little
> out of scope.
>
> Edward
>

--0016e6d9a25b44a1bd0488b16a95--

From general-return-1636-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 13 17:50:50 2010
Return-Path: <general-return-1636-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 74475 invoked from network); 13 Jun 2010 17:50:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jun 2010 17:50:49 -0000
Received: (qmail 85312 invoked by uid 500); 13 Jun 2010 17:50:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85246 invoked by uid 500); 13 Jun 2010 17:50:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85238 invoked by uid 99); 13 Jun 2010 17:50:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jun 2010 17:50:47 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jun 2010 17:50:41 +0000
Received: by wwb13 with SMTP id 13so2263656wwb.35
        for <general@hadoop.apache.org>; Sun, 13 Jun 2010 10:50:20 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.86.16 with SMTP id v16mr1508657wee.111.1276451420710; Sun, 
	13 Jun 2010 10:50:20 -0700 (PDT)
Received: by 10.216.162.20 with HTTP; Sun, 13 Jun 2010 10:50:20 -0700 (PDT)
Date: Sun, 13 Jun 2010 13:50:20 -0400
Message-ID: <AANLkTinOxEPyYzUciI0liYPFYvYK79pTDxSTHI-d780J@mail.gmail.com>
Subject: fbus project
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

All:

I just pushed an extremely early stage project called fbus[1] to
github. This project is meant to facilitate moving complete files
between various configurable end points including HDFS. Spring and
Spring Integration are used as the underbelly and allow users to wire
up channels and move files around. Current tested end point wiring:

* Local -> HDFS
* Local -> Local

The plan is to extend this to be bidirectional and incorporate various
other management features. Spring Integration was chosen so that
additional end points may be used (FTP, SSH transports, etc.) and
other nice messaging features can be used (wire tapping,
transformation, content based routing, etc.). Warning: This is very
early stage and not production quality. I wanted to throw this out
there for those who are interested.

Potential fun uses:

* Fire events over JMS or AMQP as files arrive to update management UIs.
* Monitor file movement.
* Implement a standard for file queuing and configuration.
* Batch and aggregate files on a schedule.

This project is for fun. If people find it useful / interesting, I'll
continue it. Feedback, bug reports, feature requests, and help are all welcome.

[1] http://github.com/esammer/fbus
-- 
Eric Sammer
phone: +1-917-287-2675
twitter: esammer
data: www.cloudera.com

From general-return-1637-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 15 05:26:26 2010
Return-Path: <general-return-1637-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 26791 invoked from network); 15 Jun 2010 05:26:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jun 2010 05:26:26 -0000
Received: (qmail 82527 invoked by uid 500); 15 Jun 2010 03:37:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82152 invoked by uid 500); 15 Jun 2010 03:37:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82137 invoked by uid 99); 15 Jun 2010 03:36:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 03:36:59 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jdixon@omniti.com designates 8.8.38.6 as permitted sender)
Received: from [8.8.38.6] (HELO edge.omniti.com) (8.8.38.6)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 03:36:52 +0000
DKIM-Signature: v=1; a=rsa-sha256; d=omniti.com; s=s1024; c=relaxed/relaxed;
	q=dns/txt; i=@omniti.com; t=1276572991;
	h=From:Subject:Date:To;
	bh=fcTp9JvINuMjraDaL8IFq6pQtqaoHIjjqR3Loya4xLU=;
	b=MASlM50JolKvGqYHTVoGJmBVVyNZ9CN2wGx4hm8Cvc2t7mXHBwy7mvRO6LjKpTCv
	TKOGcGp4ci8bxadLJz+Yal0I8xbU7ex0g1aMFDdX/ZZOy7Dqx+TNYwc/0zbMnXtN
	WtcTyG7Gzfr/ClNHHmkXWQPlzEDxi1RGDlT94+PKrLM=;
Authentication-Results:  edge smtp.user=jdixon@omniti.com; auth=pass (LOGIN)
Received: from [68.55.0.29] ([68.55.0.29:53420] helo=omniti.com)
	by edge (envelope-from <jdixon@omniti.com>)
	(ecelerity 2.2.2.35 r(26636M)) with ESMTPSA (cipher=AES256-SHA) 
	id 9E/3C-17327-F35F61C4; Mon, 14 Jun 2010 23:36:31 -0400
Date: Mon, 14 Jun 2010 23:36:27 -0400
From: Jason Dixon <jdixon@omniti.com>
To: general@hadoop.apache.org
Subject: CFP for Surge Scalability Conference 2010
Message-ID: <20100615033627.GA29381@omniti.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.18 (2008-05-17)
X-Virus-Checked: Checked by ClamAV on apache.org

We're excited to announce Surge, the Scalability and Performance
Conference, to be held in Baltimore on Sept 30 and Oct 1, 2010.  The
event focuses on case studies that demonstrate successes (and failures)
in Web applications and Internet architectures.

Our Keynote speakers include John Allspaw and Theo Schlossnagle.  We are
currently accepting submissions for the Call For Papers through July
9th.  You can find more information, including our current list of
speakers, online:

http://omniti.com/surge/2010

If you've been to Velocity, or wanted to but couldn't afford it, then
Surge is just what you've been waiting for.  For more information,
including CFP, sponsorship of the event, or participating as an
exhibitor, please contact us at surge@omniti.com.

Thanks,

-- 
Jason Dixon
OmniTI Computer Consulting, Inc.
jdixon@omniti.com
443.325.1357 x.241

From general-return-1638-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 15 12:32:08 2010
Return-Path: <general-return-1638-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 75936 invoked from network); 15 Jun 2010 12:32:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jun 2010 12:32:07 -0000
Received: (qmail 81524 invoked by uid 500); 15 Jun 2010 12:32:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81054 invoked by uid 500); 15 Jun 2010 12:32:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81045 invoked by uid 99); 15 Jun 2010 12:32:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 12:32:02 +0000
X-ASF-Spam-Status: No, hits=1.8 required=10.0
	tests=AWL,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_HELO_PASS,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [212.227.126.187] (HELO moutng.kundenserver.de) (212.227.126.187)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 12:31:54 +0000
Received: from smarthost.q2web.local (pd95b3aaa.dip0.t-ipconnect.de [217.91.58.170])
	by mrelayeu.kundenserver.de (node=mreu0) with ESMTP (Nemesis)
	id 0MeOiZ-1Obu7o12I3-00Q3fy; Tue, 15 Jun 2010 14:31:31 +0200
Received: from amidalasrv.q2web.local (amidalasrv.q2web.local [192.168.0.2])
	by smarthost.q2web.local (Postfix) with ESMTP id 1C80C6F4E8
	for <general@hadoop.apache.org>; Tue, 15 Jun 2010 15:06:58 +0200 (CEST)
x-mimeole: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
Subject: HDFS file append
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01CB0C86.AE8BFF30"
Date: Tue, 15 Jun 2010 14:31:29 +0200
Message-ID: <A3E731144A0BAE4A9A42C78EDEB9ADB80C4ED3@amidalasrv.q2web.local>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: HDFS file append
Thread-Index: AcsMhq26Nayx56VkR/OCZWnlnUkiUw==
From: =?iso-8859-1?Q?Jan_St=F6cker?= <Jan.Stoecker@q2web.de>
To: <general@hadoop.apache.org>
X-Provags-ID: V01U2FsdGVkX19NEHJ8ZtuvCxDhhDIkQI18+8nhe/TNnhvrN5f
 MRmASLH/gt7o1G7TZqJLgH6y6abE7X7229wGLvDP1ppnfgllW7
 mPDLwW/YBV7KkxrS4ycZw==

------_=_NextPart_001_01CB0C86.AE8BFF30
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Hello,

=20

in the hdfs-default.xml, I found the warning that the property

"dfs.support.append" is false by default, because of the "append

code" containing bugs.

For our test case (which is not very complicated),  I tried it out

nevertheless (in a C program using libhdfs), and I did not

encounter any problems yet. The only thing I noticed is that the

modification date of the file to which I append is not changed.

=20

Could someone indicate which other bugs may occur? And in

which cases? Because the append could be quite useful for me,

and I would like to know the risk.

=20

Thanks in advance

Jan


------_=_NextPart_001_01CB0C86.AE8BFF30--

From general-return-1639-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 15 14:08:01 2010
Return-Path: <general-return-1639-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12561 invoked from network); 15 Jun 2010 14:08:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jun 2010 14:08:01 -0000
Received: (qmail 27023 invoked by uid 500); 15 Jun 2010 14:08:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26936 invoked by uid 500); 15 Jun 2010 14:07:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26928 invoked by uid 99); 15 Jun 2010 14:07:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 14:07:59 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=AWL,SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of vidur@students.iiit.ac.in designates 121.242.23.201 as permitted sender)
Received: from [121.242.23.201] (HELO students.iiit.ac.in) (121.242.23.201)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 14:07:53 +0000
MailScanner-NULL-Check: 1277215648.31262@+q2DNmYv3ZPAVSiB/GgbHA
Received: from students.iiit.ac.in (localhost.localdomain [127.0.0.1])
	by students.iiit.ac.in (8.13.8/8.13.8) with ESMTP id o5FE7R6k014727
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Tue, 15 Jun 2010 19:37:27 +0530
Received: (from apache@localhost)
	by students.iiit.ac.in (8.13.8/8.14.1/Submit) id o5FE7QTE014723;
	Tue, 15 Jun 2010 19:37:26 +0530
X-Authentication-Warning: students.iiit.ac.in: apache set sender to vidur@localhost using -f
Received: from 122.169.128.82
        (SquirrelMail authenticated user vidur)
        by students.iiit.ac.in with HTTP;
        Tue, 15 Jun 2010 19:37:26 +0530 (IST)
Message-ID: <43519.122.169.128.82.1276610846.squirrel@students.iiit.ac.in>
Date: Tue, 15 Jun 2010 19:37:26 +0530 (IST)
Subject: Re: Hadoop error
From: "Vidur Goyal" <vidur@students.iiit.ac.in>
To: general@hadoop.apache.org
User-Agent: SquirrelMail/1.4.8-4.el5
MIME-Version: 1.0
Content-Type: text/plain;charset=iso-8859-1
Content-Transfer-Encoding: 8bit
X-Priority: 3 (Normal)
Importance: Normal
X-Spam-Level: *
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on students.iiit.ac.in
X-yoursite-MailScanner-Information: Please contact the IIIT Server Room  for more information
X-MailScanner-ID: o5FE7R6k014727
X-yoursite-MailScanner: Found to be clean
X-yoursite-MailScanner-From: vidur@students.iiit.ac.in
X-Old-Spam-Status: No, score=1.9 required=5.0 tests=ALL_TRUSTED,FH_DATE_PAST_20XX
	autolearn=no version=3.2.5, No

Hi,

I am trying to set up a development cluster for hadoop 0.20.1 in eclipse.
I used this url
http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.2/ to
check out the build. I compiled "compile , compile-core-test , and
eclipse-files" using ant. Then when I build the project , I am getting
errors in bin/benchmarks directory. I have followed the screencast from
cloudera
http://www.cloudera.com/blog/2009/04/configuring-eclipse-for-hadoop-development-a-screencast/.


The error log is :
Description	Resource	Path	Location	Type
FileInputFormat cannot be
resolved	CombinerJobCreator.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
84	Java Problem
FileOutputFormat cannot be
resolved	CombinerJobCreator.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
86	Java Problem
JobConf cannot be resolved to a
type	CombinerJobCreator.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
64	Java Problem
JobConf cannot be resolved to a
type	CombinerJobCreator.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
65	Java Problem
JobConf cannot be resolved to a
type	CombinerJobCreator.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
65	Java Problem
Mapper cannot be resolved to a
type	CombinerJobCreator.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
33	Java Problem
MapReduceBase cannot be resolved to a
type	CombinerJobCreator.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
32	Java Problem
MapReduceBase cannot be resolved to a
type	CombinerJobCreator.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
50	Java Problem
OutputCollector cannot be resolved to a
type	CombinerJobCreator.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
39	Java Problem
OutputCollector cannot be resolved to a
type	CombinerJobCreator.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
54	Java Problem
Reducer cannot be resolved to a
type	CombinerJobCreator.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
51	Java Problem
Reporter cannot be resolved to a
type	CombinerJobCreator.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
40	Java Problem
Reporter cannot be resolved to a
type	CombinerJobCreator.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
55	Java Problem
The declared package "org.apache.hadoop.mapred" does not match the
expected package
"gridmix2.src.java.org.apache.hadoop.mapred"	CombinerJobCreator.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
19	Java Problem
FileInputFormat cannot be
resolved	GenericMRLoadJobCreator.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
52	Java Problem
FileInputFormat cannot be
resolved	GenericMRLoadJobCreator.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
69	Java Problem
FileOutputFormat cannot be
resolved	GenericMRLoadJobCreator.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
47	Java Problem
JobClient cannot be resolved to a
type	GenericMRLoadJobCreator.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
58	Java Problem
JobClient cannot be resolved to a
type	GenericMRLoadJobCreator.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
58	Java Problem
The declared package "org.apache.hadoop.mapred" does not match the
expected package
"gridmix2.src.java.org.apache.hadoop.mapred"	GenericMRLoadJobCreator.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
19	Java Problem
Counters cannot be resolved to a
type	GridMixRunner.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
496	Java Problem
Counters cannot be resolved to a
type	GridMixRunner.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
501	Java Problem
Counters cannot be resolved to a
type	GridMixRunner.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
503	Java Problem
FileInputFormat cannot be
resolved	GridMixRunner.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
160	Java Problem
FileOutputFormat cannot be
resolved	GridMixRunner.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
161	Java Problem
JobClient cannot be resolved to a
type	GridMixRunner.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
486	Java Problem
JobConf cannot be resolved to a
type	GridMixRunner.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
123	Java Problem
JobConf cannot be resolved to a
type	GridMixRunner.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
145	Java Problem
JobConf cannot be resolved to a
type	GridMixRunner.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
145	Java Problem
JobConf cannot be resolved to a
type	GridMixRunner.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
192	Java Problem
JobConf cannot be resolved to a
type	GridMixRunner.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
220	Java Problem
JobConf cannot be resolved to a
type	GridMixRunner.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
263	Java Problem
JobConf cannot be resolved to a
type	GridMixRunner.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
300	Java Problem
JobConf cannot be resolved to a
type	GridMixRunner.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
487	Java Problem
JobID cannot be
resolved	GridMixRunner.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
495	Java Problem
JobID cannot be
resolved	GridMixRunner.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
509	Java Problem
JobID cannot be
resolved	GridMixRunner.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
511	Java Problem
TaskReport cannot be resolved to a
type	GridMixRunner.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
509	Java Problem
TaskReport cannot be resolved to a
type	GridMixRunner.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
510	Java Problem
TaskReport cannot be resolved to a
type	GridMixRunner.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
522	Java Problem
TaskReport cannot be resolved to a
type	GridMixRunner.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
544	Java Problem
The declared package "org.apache.hadoop.mapred" does not match the
expected package
"gridmix2.src.java.org.apache.hadoop.mapred"	GridMixRunner.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
19	Java Problem
The method createJob(String[]) from the type CombinerJobCreator refers to
the missing type
JobConf	GridMixRunner.java	/hadoop-0.20.3/src/benchmarks/gridmix2/src/java/org/apache/hadoop/mapred	line
220	Java Problem


I am using eclipse galileo.

Thanks,
Vidur


-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.


From general-return-1640-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 15 15:25:20 2010
Return-Path: <general-return-1640-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53917 invoked from network); 15 Jun 2010 15:25:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jun 2010 15:25:19 -0000
Received: (qmail 69203 invoked by uid 500); 15 Jun 2010 15:25:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69123 invoked by uid 500); 15 Jun 2010 15:25:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69115 invoked by uid 99); 15 Jun 2010 15:25:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 15:25:17 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL,T_FRT_STOCK1,T_FRT_STOCK2
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 15:25:10 +0000
Received: by iwn4 with SMTP id 4so811118iwn.35
        for <general@hadoop.apache.org>; Tue, 15 Jun 2010 08:24:49 -0700 (PDT)
Received: by 10.42.0.135 with SMTP id 7mr2611921icc.57.1276615489410; Tue, 15 
	Jun 2010 08:24:49 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.161.141 with HTTP; Tue, 15 Jun 2010 08:24:29 -0700 (PDT)
In-Reply-To: <A3E731144A0BAE4A9A42C78EDEB9ADB80C4ED3@amidalasrv.q2web.local>
References: <A3E731144A0BAE4A9A42C78EDEB9ADB80C4ED3@amidalasrv.q2web.local>
From: Todd Lipcon <todd@cloudera.com>
Date: Tue, 15 Jun 2010 08:24:29 -0700
Message-ID: <AANLkTin0tQJchosVkRSat5xc45RQ7SUzx748AtPgzsh3@mail.gmail.com>
Subject: Re: HDFS file append
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba180eb4c2d7ab04891334c7
X-Virus-Checked: Checked by ClamAV on apache.org

--90e6ba180eb4c2d7ab04891334c7
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Tue, Jun 15, 2010 at 5:31 AM, Jan St=F6cker <Jan.Stoecker@q2web.de> wrot=
e:

> Hello,
>
>
>
> in the hdfs-default.xml, I found the warning that the property
>
> "dfs.support.append" is false by default, because of the "append
>
> code" containing bugs.
>
> For our test case (which is not very complicated),  I tried it out
>
> nevertheless (in a C program using libhdfs), and I did not
>
> encounter any problems yet. The only thing I noticed is that the
>
> modification date of the file to which I append is not changed.
>
>
>
> Could someone indicate which other bugs may occur? And in
>
> which cases? Because the append could be quite useful for me,
>
> and I would like to know the risk.
>

Mostly in failure handling cases, but not entirely. The result is usually
truncated files, sometimes truncated as if your append didn't happen,
sometimes the entire last block going missing. Look for the 0.20-append fix
version on JIRA for a more significantly list.

I'd recommend waiting a couple weeks and using the hadoop-0.20-append branc=
h
which will contain fixes for many of these bugs.

-Todd


--=20
Todd Lipcon
Software Engineer, Cloudera

--90e6ba180eb4c2d7ab04891334c7--

From general-return-1641-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 15 16:53:50 2010
Return-Path: <general-return-1641-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88858 invoked from network); 15 Jun 2010 16:53:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jun 2010 16:53:50 -0000
Received: (qmail 97343 invoked by uid 500); 15 Jun 2010 16:53:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97285 invoked by uid 500); 15 Jun 2010 16:53:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97277 invoked by uid 99); 15 Jun 2010 16:53:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 16:53:48 +0000
X-ASF-Spam-Status: No, hits=0.9 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_FRT_STOCK1,T_FRT_STOCK2,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 209.85.211.193 as permitted sender)
Received: from [209.85.211.193] (HELO mail-yw0-f193.google.com) (209.85.211.193)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 16:53:43 +0000
Received: by ywh31 with SMTP id 31so4033648ywh.20
        for <general@hadoop.apache.org>; Tue, 15 Jun 2010 09:53:23 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=6Gw/O5PHeqZ861mQUl+7RT3EgJUGsBmMLU+Ca6WoP98=;
        b=TThyIL7MbAuKFQ/JjE1oYh5E7x84ZUjYy9VGS8tCm5qNi+CpwRG//N92rZt/1YqAza
         d8PrfeEVoTrVo5dHbnpoySHv6NZ4N0iPB9UxS0C9UJKj3cysAPaITwZw6dV8cpgpJdvK
         n9mqltiMdmbJSfs6vd5kuFy34a3L60km4F4UE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=JCYZc2hdc8FzFK5QG1J4KKDHMZrwfhbC+sURRNG3nOfwF/DmX+mIWdatYRw/BSxVLB
         76YPO3XMbaHFRuV05s7S270wlTcfqJjliNeLJsorqoNaHk3XNaOtDvqjhIZlq5M0+H2a
         wgm5iIrmaOROLWjiqQKEt5zI2QFU055Dd//Sc=
MIME-Version: 1.0
Received: by 10.150.2.1 with SMTP id 1mr9254803ybb.65.1276620802610; Tue, 15 
	Jun 2010 09:53:22 -0700 (PDT)
Received: by 10.151.145.16 with HTTP; Tue, 15 Jun 2010 09:53:22 -0700 (PDT)
In-Reply-To: <AANLkTin0tQJchosVkRSat5xc45RQ7SUzx748AtPgzsh3@mail.gmail.com>
References: <A3E731144A0BAE4A9A42C78EDEB9ADB80C4ED3@amidalasrv.q2web.local>
	<AANLkTin0tQJchosVkRSat5xc45RQ7SUzx748AtPgzsh3@mail.gmail.com>
Date: Tue, 15 Jun 2010 09:53:22 -0700
Message-ID: <AANLkTilx_SdnHlu4XMfKd2J9Y9Uz86WBsIE94l_RW_oV@mail.gmail.com>
Subject: Re: HDFS file append
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd4879074691204891471e2

--000e0cd4879074691204891471e2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

You can start downloading and using the code form the hadoop-0.20-append.

thanks,
dhruba

On Tue, Jun 15, 2010 at 8:24 AM, Todd Lipcon <todd@cloudera.com> wrote:

> On Tue, Jun 15, 2010 at 5:31 AM, Jan St=F6cker <Jan.Stoecker@q2web.de>
> wrote:
>
> > Hello,
> >
> >
> >
> > in the hdfs-default.xml, I found the warning that the property
> >
> > "dfs.support.append" is false by default, because of the "append
> >
> > code" containing bugs.
> >
> > For our test case (which is not very complicated),  I tried it out
> >
> > nevertheless (in a C program using libhdfs), and I did not
> >
> > encounter any problems yet. The only thing I noticed is that the
> >
> > modification date of the file to which I append is not changed.
> >
> >
> >
> > Could someone indicate which other bugs may occur? And in
> >
> > which cases? Because the append could be quite useful for me,
> >
> > and I would like to know the risk.
> >
>
> Mostly in failure handling cases, but not entirely. The result is usually
> truncated files, sometimes truncated as if your append didn't happen,
> sometimes the entire last block going missing. Look for the 0.20-append f=
ix
> version on JIRA for a more significantly list.
>
> I'd recommend waiting a couple weeks and using the hadoop-0.20-append
> branch
> which will contain fixes for many of these bugs.
>
> -Todd
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>



--=20
Connect to me at http://www.facebook.com/dhruba

--000e0cd4879074691204891471e2--

From general-return-1642-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 15 16:56:09 2010
Return-Path: <general-return-1642-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 89152 invoked from network); 15 Jun 2010 16:56:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jun 2010 16:56:08 -0000
Received: (qmail 98932 invoked by uid 500); 15 Jun 2010 16:56:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98878 invoked by uid 500); 15 Jun 2010 16:56:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98868 invoked by uid 99); 15 Jun 2010 16:56:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 16:56:07 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL,T_FRT_STOCK1,T_FRT_STOCK2
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 16:56:00 +0000
Received: by iwn4 with SMTP id 4so943536iwn.35
        for <general@hadoop.apache.org>; Tue, 15 Jun 2010 09:55:39 -0700 (PDT)
Received: by 10.231.193.93 with SMTP id dt29mr8760677ibb.71.1276620939350; 
	Tue, 15 Jun 2010 09:55:39 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.161.141 with HTTP; Tue, 15 Jun 2010 09:55:19 -0700 (PDT)
In-Reply-To: <AANLkTilx_SdnHlu4XMfKd2J9Y9Uz86WBsIE94l_RW_oV@mail.gmail.com>
References: <A3E731144A0BAE4A9A42C78EDEB9ADB80C4ED3@amidalasrv.q2web.local> 
	<AANLkTin0tQJchosVkRSat5xc45RQ7SUzx748AtPgzsh3@mail.gmail.com> 
	<AANLkTilx_SdnHlu4XMfKd2J9Y9Uz86WBsIE94l_RW_oV@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Tue, 15 Jun 2010 09:55:19 -0700
Message-ID: <AANLkTikW1Ca-oSa9CMGkqc2oJRUD6IL7M6E2ELhx6l2W@mail.gmail.com>
Subject: Re: HDFS file append
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502c1af9a5bc204891479ea
X-Virus-Checked: Checked by ClamAV on apache.org

--00504502c1af9a5bc204891479ea
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Tue, Jun 15, 2010 at 9:53 AM, Dhruba Borthakur <dhruba@gmail.com> wrote:

> You can start downloading and using the code form the hadoop-0.20-append.
>
>
Though it's still missing some patches, so I'd recommend SVN checkouts and
watching the branch :)

-Todd


> thanks,
> dhruba
>
> On Tue, Jun 15, 2010 at 8:24 AM, Todd Lipcon <todd@cloudera.com> wrote:
>
> > On Tue, Jun 15, 2010 at 5:31 AM, Jan St=F6cker <Jan.Stoecker@q2web.de>
> > wrote:
> >
> > > Hello,
> > >
> > >
> > >
> > > in the hdfs-default.xml, I found the warning that the property
> > >
> > > "dfs.support.append" is false by default, because of the "append
> > >
> > > code" containing bugs.
> > >
> > > For our test case (which is not very complicated),  I tried it out
> > >
> > > nevertheless (in a C program using libhdfs), and I did not
> > >
> > > encounter any problems yet. The only thing I noticed is that the
> > >
> > > modification date of the file to which I append is not changed.
> > >
> > >
> > >
> > > Could someone indicate which other bugs may occur? And in
> > >
> > > which cases? Because the append could be quite useful for me,
> > >
> > > and I would like to know the risk.
> > >
> >
> > Mostly in failure handling cases, but not entirely. The result is usual=
ly
> > truncated files, sometimes truncated as if your append didn't happen,
> > sometimes the entire last block going missing. Look for the 0.20-append
> fix
> > version on JIRA for a more significantly list.
> >
> > I'd recommend waiting a couple weeks and using the hadoop-0.20-append
> > branch
> > which will contain fixes for many of these bugs.
> >
> > -Todd
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
>
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
>



--=20
Todd Lipcon
Software Engineer, Cloudera

--00504502c1af9a5bc204891479ea--

From general-return-1643-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 15 17:10:23 2010
Return-Path: <general-return-1643-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 99268 invoked from network); 15 Jun 2010 17:10:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jun 2010 17:10:23 -0000
Received: (qmail 47526 invoked by uid 500); 15 Jun 2010 17:10:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47488 invoked by uid 500); 15 Jun 2010 17:10:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47480 invoked by uid 99); 15 Jun 2010 17:10:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 17:10:21 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [137.100.120.43] (HELO mnbm01-relay1.mnb.gd-ais.com) (137.100.120.43)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 17:10:14 +0000
Received: from ([160.207.224.15])
	by mnbm01-relay1.mnb.gd-ais.com with SMTP  id 5202712.271944112;
	Tue, 15 Jun 2010 12:09:50 -0500
Received: from MAPF01-MAIL01.ad.gd-ais.com ([166.16.220.104]) by mnbm01-fes01.ad.gd-ais.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Tue, 15 Jun 2010 12:09:50 -0500
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01CB0CAD.8E2D6654"
Subject: IPC connections failing
Date: Tue, 15 Jun 2010 13:09:46 -0400
Message-ID: <E0C1686B8F45434DA0E0A904672EA1A8065AC962@MAPF01-MAIL01.ad.gd-ais.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: IPC connections failing
Thread-Index: AcsMrY4jFCWs2PEdSUOSR7UUsnezQA==
From: "Kilbride, James P." <James.Kilbride@gd-ais.com>
To: <general@hadoop.apache.org>
X-OriginalArrivalTime: 15 Jun 2010 17:09:50.0724 (UTC) FILETIME=[90BAFC40:01CB0CAD]

------_=_NextPart_001_01CB0CAD.8E2D6654
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

I've tried to move my system from pseudo distributed to full
distributed(with it being the first node in the full cluster I'll be
setting up) but the components don't seem to be talking to each other
and I find logs filled with ipc.Client connection failures.=20

Here's the basic log info:
DataNode:
2010-06-15 12:42:42,824 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:=20
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG:   host =3D centoshadoop.soa.gd-ais.com/10.28.208.118
STARTUP_MSG:   args =3D []
STARTUP_MSG:   version =3D 0.20.2
STARTUP_MSG:   build =3D
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2010-06-15 12:42:49,335 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 0 time(s).
2010-06-15 12:42:50,343 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 1 time(s).
2010-06-15 12:42:51,354 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 2 time(s).
2010-06-15 12:42:53,641 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 3 time(s).
2010-06-15 12:42:54,733 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 4 time(s).
2010-06-15 12:42:55,737 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 5 time(s).
2010-06-15 12:42:56,824 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 6 time(s).
2010-06-15 12:42:57,827 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 7 time(s).
2010-06-15 12:42:58,831 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 8 time(s).
2010-06-15 12:42:59,833 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 9 time(s).
2010-06-15 12:42:59,836 INFO org.apache.hadoop.ipc.RPC: Server at
/10.28.208.118:54310 not available yet, Zzzzz...
2010-06-15 12:43:01,841 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 0 time(s).
2010-06-15 12:43:02,844 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 1 time(s).
2010-06-15 12:43:03,846 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 2 time(s).
2010-06-15 12:43:04,849 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 3 time(s).
2010-06-15 12:43:05,852 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 4 time(s).
2010-06-15 12:43:06,855 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 5 time(s).
2010-06-15 12:43:07,858 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 6 time(s).
2010-06-15 12:43:08,883 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 7 time(s).
2010-06-15 12:43:09,970 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 8 time(s).
2010-06-15 12:43:10,973 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 9 time(s).
2010-06-15 12:43:10,973 INFO org.apache.hadoop.ipc.RPC: Server at
/10.28.208.118:54310 not available yet, Zzzzz...

JobTracker:
2010-06-15 12:42:52,836 INFO org.apache.hadoop.mapred.JobTracker:
STARTUP_MSG:=20
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG:   host =3D centoshadoop.soa.gd-ais.com/10.28.208.118
STARTUP_MSG:   args =3D []
STARTUP_MSG:   version =3D 0.20.2
STARTUP_MSG:   build =3D
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2010-06-15 12:42:53,350 INFO org.apache.hadoop.mapred.JobTracker:
Scheduler configured with (memSizeForMapSlotOnJT,
memSizeForReduceSlotOnJT, limitMaxMemForMapTasks,
limitMaxMemForReduceTasks) (-1, -1, -1, -1)
2010-06-15 12:42:53,815 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=3DJobTracker, port=3D54311
2010-06-15 12:42:53,986 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2010-06-15 12:42:55,381 INFO org.apache.hadoop.http.HttpServer: Port
returned by webServer.getConnectors()[0].getLocalPort() before open() is
-1. Opening the listener on 50030
2010-06-15 12:42:55,400 INFO org.apache.hadoop.http.HttpServer:
listener.getLocalPort() returned 50030
webServer.getConnectors()[0].getLocalPort() returned 50030
2010-06-15 12:42:55,400 INFO org.apache.hadoop.http.HttpServer: Jetty
bound to port 50030
2010-06-15 12:42:55,402 INFO org.mortbay.log: jetty-6.1.14
2010-06-15 12:42:56,802 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:50030
2010-06-15 12:42:56,805 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=3DJobTracker, sessionId=3D
2010-06-15 12:42:56,807 INFO org.apache.hadoop.mapred.JobTracker:
JobTracker up at: 54311
2010-06-15 12:42:56,807 INFO org.apache.hadoop.mapred.JobTracker:
JobTracker webserver: 50030
2010-06-15 12:42:58,076 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 0 time(s).
2010-06-15 12:42:59,080 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 1 time(s).
2010-06-15 12:43:00,083 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 2 time(s).
2010-06-15 12:43:01,086 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 3 time(s).
2010-06-15 12:43:02,089 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 4 time(s).
2010-06-15 12:43:03,091 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 5 time(s).
2010-06-15 12:43:04,095 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 6 time(s).
2010-06-15 12:43:05,174 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 7 time(s).
2010-06-15 12:43:06,177 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 8 time(s).
2010-06-15 12:43:07,180 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 9 time(s).
2010-06-15 12:43:07,185 INFO org.apache.hadoop.mapred.JobTracker:
problem cleaning system directory: null
java.net.ConnectException: Call to /10.28.208.118:54310 failed on
connection exception: java.net.ConnectException: Connection refused
	at org.apache.hadoop.ipc.Client.wrapException(Client.java:767)
	at org.apache.hadoop.ipc.Client.call(Client.java:743)
	at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
	at $Proxy4.getProtocolVersion(Unknown Source)
	at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
	at
org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
	at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileS
ystem.java:82)
	at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
	at
org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
	at
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
	at
org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1665)
	at
org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
	at
org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
	at
org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
	at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.ja
va:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
	at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:304)
	at
org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:860)
	at org.apache.hadoop.ipc.Client.call(Client.java:720)
	... 16 more

NameNode:
2010-06-15 12:42:38,714 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:=20
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host =3D centoshadoop.soa.gd-ais.com/10.28.208.118
STARTUP_MSG:   args =3D []
STARTUP_MSG:   version =3D 0.20.2
STARTUP_MSG:   build =3D
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2010-06-15 12:42:38,922 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=3DNameNode, port=3D54310
2010-06-15 12:42:38,932 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Namenode up at:
centoshadoop/10.28.208.118:54310
2010-06-15 12:42:38,938 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=3DNameNode, sessionId=3Dnull
2010-06-15 12:42:38,941 INFO
org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics:
Initializing NameNodeMeterics using context
object:org.apache.hadoop.metrics.spi.NullContext
2010-06-15 12:42:39,140 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
fsOwner=3Dhadoop,users,hadoop
2010-06-15 12:42:39,141 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
supergroup=3Dsupergroup
2010-06-15 12:42:39,141 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
isPermissionEnabled=3Dtrue
2010-06-15 12:42:39,156 INFO
org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics:
Initializing FSNamesystemMetrics using context
object:org.apache.hadoop.metrics.spi.NullContext
2010-06-15 12:42:39,161 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
FSNamesystemStatusMBean
2010-06-15 12:42:39,381 INFO
org.apache.hadoop.hdfs.server.common.Storage: Number of files =3D 28
2010-06-15 12:42:39,404 INFO
org.apache.hadoop.hdfs.server.common.Storage: Number of files under
construction =3D 0
2010-06-15 12:42:39,404 INFO
org.apache.hadoop.hdfs.server.common.Storage: Image file of size 2780
loaded in 0 seconds.
2010-06-15 12:42:39,429 ERROR
org.apache.hadoop.hdfs.server.namenode.NameNode:
java.lang.NumberFormatException: For input string: ""
	at
java.lang.NumberFormatException.forInputString(NumberFormatException.jav
a:48)
	at java.lang.Long.parseLong(Long.java:431)
	at java.lang.Long.parseLong(Long.java:468)
	at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.readLong(FSEditLog.java
:1273)
	at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.j
ava:670)
	at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:
992)
	at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:
812)
	at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSI
mage.java:364)
	at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirecto
ry.java:87)
	at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesys
tem.java:311)
	at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.
java:292)
	at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java
:201)
	at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279
)
	at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.
java:956)
	at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)

SecondaryNameNode:
2010-06-15 12:42:53,227 INFO
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:=20
/************************************************************
STARTUP_MSG: Starting SecondaryNameNode
STARTUP_MSG:   host =3D centoshadoop.soa.gd-ais.com/10.28.208.118
STARTUP_MSG:   args =3D []
STARTUP_MSG:   version =3D 0.20.2
STARTUP_MSG:   build =3D
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2010-06-15 12:42:53,442 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=3DSecondaryNameNode,
sessionId=3Dnull
2010-06-15 12:42:55,651 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 0 time(s).
2010-06-15 12:42:56,659 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 1 time(s).
2010-06-15 12:42:57,662 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 2 time(s).
2010-06-15 12:42:58,665 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 3 time(s).
2010-06-15 12:42:59,668 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 4 time(s).
2010-06-15 12:43:00,671 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 5 time(s).
2010-06-15 12:43:01,674 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 6 time(s).
2010-06-15 12:43:02,677 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 7 time(s).
2010-06-15 12:43:03,679 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 8 time(s).
2010-06-15 12:43:04,682 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 9 time(s).
2010-06-15 12:43:04,685 INFO org.apache.hadoop.ipc.RPC: Server at
/10.28.208.118:54310 not available yet, Zzzzz...
2010-06-15 12:43:06,691 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 0 time(s).
2010-06-15 12:43:07,694 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 1 time(s).
2010-06-15 12:43:08,697 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 2 time(s).
2010-06-15 12:43:09,700 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 3 time(s).
2010-06-15 12:43:10,703 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 4 time(s).
2010-06-15 12:43:11,706 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 5 time(s).
2010-06-15 12:43:12,709 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 6 time(s).
2010-06-15 12:43:13,712 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 7 time(s).
2010-06-15 12:43:14,714 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 8 time(s).
2010-06-15 12:43:15,717 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 9 time(s).
2010-06-15 12:43:15,718 INFO org.apache.hadoop.ipc.RPC: Server at
/10.28.208.118:54310 not available yet, Zzzzz...

TaskTracker:
2010-06-15 12:42:57,270 INFO org.apache.hadoop.mapred.TaskTracker:
STARTUP_MSG:=20
/************************************************************
STARTUP_MSG: Starting TaskTracker
STARTUP_MSG:   host =3D centoshadoop.soa.gd-ais.com/10.28.208.118
STARTUP_MSG:   args =3D []
STARTUP_MSG:   version =3D 0.20.2
STARTUP_MSG:   build =3D
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2010-06-15 12:42:57,854 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2010-06-15 12:42:58,328 INFO org.apache.hadoop.http.HttpServer: Port
returned by webServer.getConnectors()[0].getLocalPort() before open() is
-1. Opening the listener on 50060
2010-06-15 12:42:58,344 INFO org.apache.hadoop.http.HttpServer:
listener.getLocalPort() returned 50060
webServer.getConnectors()[0].getLocalPort() returned 50060
2010-06-15 12:42:58,344 INFO org.apache.hadoop.http.HttpServer: Jetty
bound to port 50060
2010-06-15 12:42:58,345 INFO org.mortbay.log: jetty-6.1.14
2010-06-15 12:42:58,852 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:50060
2010-06-15 12:42:58,860 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=3DTaskTracker, sessionId=3D
2010-06-15 12:42:58,882 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=3DTaskTracker, port=3D50758
2010-06-15 12:42:58,960 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2010-06-15 12:42:58,962 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 50758: starting
2010-06-15 12:42:58,964 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 2 on 50758: starting
2010-06-15 12:42:58,964 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 50758: starting
2010-06-15 12:42:58,964 INFO org.apache.hadoop.mapred.TaskTracker:
TaskTracker up at: localhost.localdomain/127.0.0.1:50758
2010-06-15 12:42:58,964 INFO org.apache.hadoop.mapred.TaskTracker:
Starting tracker
tracker_centoshadoop:localhost.localdomain/127.0.0.1:50758
2010-06-15 12:42:58,969 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 3 on 50758: starting
2010-06-15 12:42:58,969 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 1 on 50758: starting

The config files are:
core-site.xml:
<?xml version=3D"1.0"?>
<?xml-stylesheet type=3D"text/xsl" href=3D"configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>hadoop.tmp.dir</name>
  <value>/hadoop-0.20.2/tmp/dir/hadoop-${user.name}</value>
  <description> A base for the other temporary
directories.</description>
  <final>true</final>
</property>
<property>
  <name>fs.default.name</name>
  <value>hdfs://10.28.208.118:54310</value>
  <description> The name of the default file system. A URI whose scheme
and auth
ority determine the filesystem implementation. The uri's scheme
determines the c
onfig property (fs.SCHEME.impl) name the FileSystem implementation
class. The ur
i's authority is used to determine the host, port, etc for a
filesystem</description>
  <final>true</final>
</property>

</configuration>

hadoop-env.sh
# Set Hadoop-specific environment variables here.

# The only required environment variable is JAVA_HOME.  All others are
# optional.  When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.

# The java implementation to use.  Required.
export JAVA_HOME=3D/usr/java/default

# The hadoop home directory
export HADOOP_HOME=3D/hadoop-0.20.2

# Extra Java CLASSPATH elements.  Optional.
export
HADOOP_CLASSPATH=3D/hbase-0.20.4/lib/zookeeper-3.2.2.jar:/habase-0.20.4/h=
b
ase-0.20.4-test.jar:/hbase-0.20.4/hbase-0.20.4.jar:/hbase-0.20.4/conf

# The maximum amount of heap to use, in MB. Default is 1000.
# export HADOOP_HEAPSIZE=3D2000

# Extra Java runtime options.  Empty by default.
export HADOOP_OPTS=3D-Djava.net.preferIPv4Stack=3Dtrue

# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS=3D"-Dcom.sun.management.jmxremote
$HADOOP_NAMENODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS=3D"-Dcom.sun.management.jmxremote
$HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_DATANODE_OPTS=3D"-Dcom.sun.management.jmxremote
$HADOOP_DATANODE_OPTS"
export HADOOP_BALANCER_OPTS=3D"-Dcom.sun.management.jmxremote
$HADOOP_BALANCER_OPTS"
export HADOOP_JOBTRACKER_OPTS=3D"-Dcom.sun.management.jmxremote
$HADOOP_JOBTRACKER_OPTS"
# export HADOOP_TASKTRACKER_OPTS=3D
# The following applies to multiple commands (fs, dfs, fsck, distcp etc)
# export HADOOP_CLIENT_OPTS

# Extra ssh options.  Empty by default.
# export HADOOP_SSH_OPTS=3D"-o ConnectTimeout=3D1 -o
SendEnv=3DHADOOP_CONF_DIR"

# Where log files are stored.  $HADOOP_HOME/logs by default.
# export HADOOP_LOG_DIR=3D${HADOOP_HOME}/logs

# File naming remote slave hosts.  $HADOOP_HOME/conf/slaves by default.
# export HADOOP_SLAVES=3D${HADOOP_HOME}/conf/slaves

# host:path where hadoop code should be rsync'd from.  Unset by default.
# export HADOOP_MASTER=3Dmaster:/home/$USER/src/hadoop

# Seconds to sleep between slave commands.  Unset by default.  This
# can be useful in large clusters, where, e.g., slave rsyncs can
# otherwise arrive faster than the master can service them.
# export HADOOP_SLAVE_SLEEP=3D0.1

# The directory where pid files are stored. /tmp by default.
# export HADOOP_PID_DIR=3D/var/hadoop/pids

# A string representing this instance of hadoop. $USER by default.
# export HADOOP_IDENT_STRING=3D$USER

# The scheduling priority for daemon processes.  See 'man nice'.
# export HADOOP_NICENESS=3D10

hdfs-site.xml
<?xml version=3D"1.0"?>
<?xml-stylesheet type=3D"text/xsl" href=3D"configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>dfs.replication</name>
  <value>22</value>
  <description> Default block replication. The actual number of
replications can be specified when the file is created. Default is used
if not specified at creation time.</description>
  <final>true</final>
</property>
<property>
  <name>dfs.data.dir</name>
  <value>/hadoop-0.20.2/hadoopDataStore/data</value>
  <description>DFS Data Directory</description>
  <final>true</final>
</property>

</configuration>

masters:
10.28.208.118

slaves:
10.28.208.118

(And because I know it'll be asked)
hosts:
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1		localhost.localdomain localhost=20
::1		localhost6.localdomain6 localhost6=20
10.28.208.118	centoshadoop centoshadoop.soa.gd-ais.com

Why is this failing to stand up? I'd like to get this one node working
before I stand up any other data nodes.(Firewall, btw, is turned off)

James Kilbride

------_=_NextPart_001_01CB0CAD.8E2D6654--

From general-return-1644-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 15 17:11:38 2010
Return-Path: <general-return-1644-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 99794 invoked from network); 15 Jun 2010 17:11:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jun 2010 17:11:38 -0000
Received: (qmail 52068 invoked by uid 500); 15 Jun 2010 17:11:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51889 invoked by uid 500); 15 Jun 2010 17:11:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51879 invoked by uid 99); 15 Jun 2010 17:11:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 17:11:36 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=AWL,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [192.5.164.99] (HELO camv02-relay2.casc.gd-ais.com) (192.5.164.99)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 17:11:31 +0000
Received: from ([10.73.100.22])
	by camv02-relay2.casc.gd-ais.com with SMTP  id 5203374.36087912;
	Tue, 15 Jun 2010 10:11:09 -0700
Received: from MAPF01-MAIL01.ad.gd-ais.com ([166.16.220.104]) by camv02-fes01.ad.gd-ais.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Tue, 15 Jun 2010 10:11:09 -0700
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: RE: IPC connections failing
Date: Tue, 15 Jun 2010 13:11:07 -0400
Message-ID: <E0C1686B8F45434DA0E0A904672EA1A8065AC965@MAPF01-MAIL01.ad.gd-ais.com>
In-Reply-To: <E0C1686B8F45434DA0E0A904672EA1A8065AC962@MAPF01-MAIL01.ad.gd-ais.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: IPC connections failing
Thread-Index: AcsMrY4jFCWs2PEdSUOSR7UUsnezQAAACltA
References: <E0C1686B8F45434DA0E0A904672EA1A8065AC962@MAPF01-MAIL01.ad.gd-ais.com>
From: "Kilbride, James P." <James.Kilbride@gd-ais.com>
To: <general@hadoop.apache.org>
X-OriginalArrivalTime: 15 Jun 2010 17:11:09.0072 (UTC) FILETIME=[BF6DF100:01CB0CAD]

Forgot to add on mapred-site.xml:

<?xml version=3D"1.0"?>
<?xml-stylesheet type=3D"text/xsl" href=3D"configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>mapred.job.tracker</name>
  <value>hdfs://10.28.208.118:54311</value>
  <description> The host and port that the MapReduce job tracker runs
at. If "local", then the jobs are run in-process as a single map and
reduce task.</description>
  <final>true</final>
</property>

</configuration>

-----Original Message-----
From: Kilbride, James P. [mailto:James.Kilbride@gd-ais.com]=20
Sent: Tuesday, June 15, 2010 1:10 PM
To: general@hadoop.apache.org
Subject: IPC connections failing

I've tried to move my system from pseudo distributed to full
distributed(with it being the first node in the full cluster I'll be
setting up) but the components don't seem to be talking to each other
and I find logs filled with ipc.Client connection failures.=20

Here's the basic log info:
DataNode:
2010-06-15 12:42:42,824 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:=20
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG:   host =3D centoshadoop.soa.gd-ais.com/10.28.208.118
STARTUP_MSG:   args =3D []
STARTUP_MSG:   version =3D 0.20.2
STARTUP_MSG:   build =3D
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2010-06-15 12:42:49,335 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 0 time(s).
2010-06-15 12:42:50,343 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 1 time(s).
2010-06-15 12:42:51,354 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 2 time(s).
2010-06-15 12:42:53,641 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 3 time(s).
2010-06-15 12:42:54,733 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 4 time(s).
2010-06-15 12:42:55,737 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 5 time(s).
2010-06-15 12:42:56,824 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 6 time(s).
2010-06-15 12:42:57,827 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 7 time(s).
2010-06-15 12:42:58,831 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 8 time(s).
2010-06-15 12:42:59,833 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 9 time(s).
2010-06-15 12:42:59,836 INFO org.apache.hadoop.ipc.RPC: Server at
/10.28.208.118:54310 not available yet, Zzzzz...
2010-06-15 12:43:01,841 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 0 time(s).
2010-06-15 12:43:02,844 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 1 time(s).
2010-06-15 12:43:03,846 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 2 time(s).
2010-06-15 12:43:04,849 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 3 time(s).
2010-06-15 12:43:05,852 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 4 time(s).
2010-06-15 12:43:06,855 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 5 time(s).
2010-06-15 12:43:07,858 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 6 time(s).
2010-06-15 12:43:08,883 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 7 time(s).
2010-06-15 12:43:09,970 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 8 time(s).
2010-06-15 12:43:10,973 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 9 time(s).
2010-06-15 12:43:10,973 INFO org.apache.hadoop.ipc.RPC: Server at
/10.28.208.118:54310 not available yet, Zzzzz...

JobTracker:
2010-06-15 12:42:52,836 INFO org.apache.hadoop.mapred.JobTracker:
STARTUP_MSG:=20
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG:   host =3D centoshadoop.soa.gd-ais.com/10.28.208.118
STARTUP_MSG:   args =3D []
STARTUP_MSG:   version =3D 0.20.2
STARTUP_MSG:   build =3D
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2010-06-15 12:42:53,350 INFO org.apache.hadoop.mapred.JobTracker:
Scheduler configured with (memSizeForMapSlotOnJT,
memSizeForReduceSlotOnJT, limitMaxMemForMapTasks,
limitMaxMemForReduceTasks) (-1, -1, -1, -1)
2010-06-15 12:42:53,815 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=3DJobTracker, port=3D54311
2010-06-15 12:42:53,986 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2010-06-15 12:42:55,381 INFO org.apache.hadoop.http.HttpServer: Port
returned by webServer.getConnectors()[0].getLocalPort() before open() is
-1. Opening the listener on 50030
2010-06-15 12:42:55,400 INFO org.apache.hadoop.http.HttpServer:
listener.getLocalPort() returned 50030
webServer.getConnectors()[0].getLocalPort() returned 50030
2010-06-15 12:42:55,400 INFO org.apache.hadoop.http.HttpServer: Jetty
bound to port 50030
2010-06-15 12:42:55,402 INFO org.mortbay.log: jetty-6.1.14
2010-06-15 12:42:56,802 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:50030
2010-06-15 12:42:56,805 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=3DJobTracker, sessionId=3D
2010-06-15 12:42:56,807 INFO org.apache.hadoop.mapred.JobTracker:
JobTracker up at: 54311
2010-06-15 12:42:56,807 INFO org.apache.hadoop.mapred.JobTracker:
JobTracker webserver: 50030
2010-06-15 12:42:58,076 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 0 time(s).
2010-06-15 12:42:59,080 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 1 time(s).
2010-06-15 12:43:00,083 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 2 time(s).
2010-06-15 12:43:01,086 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 3 time(s).
2010-06-15 12:43:02,089 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 4 time(s).
2010-06-15 12:43:03,091 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 5 time(s).
2010-06-15 12:43:04,095 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 6 time(s).
2010-06-15 12:43:05,174 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 7 time(s).
2010-06-15 12:43:06,177 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 8 time(s).
2010-06-15 12:43:07,180 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 9 time(s).
2010-06-15 12:43:07,185 INFO org.apache.hadoop.mapred.JobTracker:
problem cleaning system directory: null
java.net.ConnectException: Call to /10.28.208.118:54310 failed on
connection exception: java.net.ConnectException: Connection refused
	at org.apache.hadoop.ipc.Client.wrapException(Client.java:767)
	at org.apache.hadoop.ipc.Client.call(Client.java:743)
	at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
	at $Proxy4.getProtocolVersion(Unknown Source)
	at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
	at
org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
	at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileS
ystem.java:82)
	at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
	at
org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
	at
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
	at
org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1665)
	at
org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
	at
org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
	at
org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
	at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.ja
va:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
	at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:304)
	at
org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:860)
	at org.apache.hadoop.ipc.Client.call(Client.java:720)
	... 16 more

NameNode:
2010-06-15 12:42:38,714 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:=20
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host =3D centoshadoop.soa.gd-ais.com/10.28.208.118
STARTUP_MSG:   args =3D []
STARTUP_MSG:   version =3D 0.20.2
STARTUP_MSG:   build =3D
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2010-06-15 12:42:38,922 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=3DNameNode, port=3D54310
2010-06-15 12:42:38,932 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Namenode up at:
centoshadoop/10.28.208.118:54310
2010-06-15 12:42:38,938 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=3DNameNode, sessionId=3Dnull
2010-06-15 12:42:38,941 INFO
org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics:
Initializing NameNodeMeterics using context
object:org.apache.hadoop.metrics.spi.NullContext
2010-06-15 12:42:39,140 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
fsOwner=3Dhadoop,users,hadoop
2010-06-15 12:42:39,141 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
supergroup=3Dsupergroup
2010-06-15 12:42:39,141 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
isPermissionEnabled=3Dtrue
2010-06-15 12:42:39,156 INFO
org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics:
Initializing FSNamesystemMetrics using context
object:org.apache.hadoop.metrics.spi.NullContext
2010-06-15 12:42:39,161 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
FSNamesystemStatusMBean
2010-06-15 12:42:39,381 INFO
org.apache.hadoop.hdfs.server.common.Storage: Number of files =3D 28
2010-06-15 12:42:39,404 INFO
org.apache.hadoop.hdfs.server.common.Storage: Number of files under
construction =3D 0
2010-06-15 12:42:39,404 INFO
org.apache.hadoop.hdfs.server.common.Storage: Image file of size 2780
loaded in 0 seconds.
2010-06-15 12:42:39,429 ERROR
org.apache.hadoop.hdfs.server.namenode.NameNode:
java.lang.NumberFormatException: For input string: ""
	at
java.lang.NumberFormatException.forInputString(NumberFormatException.jav
a:48)
	at java.lang.Long.parseLong(Long.java:431)
	at java.lang.Long.parseLong(Long.java:468)
	at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.readLong(FSEditLog.java
:1273)
	at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.j
ava:670)
	at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:
992)
	at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:
812)
	at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSI
mage.java:364)
	at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirecto
ry.java:87)
	at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesys
tem.java:311)
	at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.
java:292)
	at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java
:201)
	at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279
)
	at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.
java:956)
	at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)

SecondaryNameNode:
2010-06-15 12:42:53,227 INFO
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:=20
/************************************************************
STARTUP_MSG: Starting SecondaryNameNode
STARTUP_MSG:   host =3D centoshadoop.soa.gd-ais.com/10.28.208.118
STARTUP_MSG:   args =3D []
STARTUP_MSG:   version =3D 0.20.2
STARTUP_MSG:   build =3D
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2010-06-15 12:42:53,442 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=3DSecondaryNameNode,
sessionId=3Dnull
2010-06-15 12:42:55,651 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 0 time(s).
2010-06-15 12:42:56,659 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 1 time(s).
2010-06-15 12:42:57,662 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 2 time(s).
2010-06-15 12:42:58,665 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 3 time(s).
2010-06-15 12:42:59,668 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 4 time(s).
2010-06-15 12:43:00,671 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 5 time(s).
2010-06-15 12:43:01,674 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 6 time(s).
2010-06-15 12:43:02,677 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 7 time(s).
2010-06-15 12:43:03,679 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 8 time(s).
2010-06-15 12:43:04,682 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 9 time(s).
2010-06-15 12:43:04,685 INFO org.apache.hadoop.ipc.RPC: Server at
/10.28.208.118:54310 not available yet, Zzzzz...
2010-06-15 12:43:06,691 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 0 time(s).
2010-06-15 12:43:07,694 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 1 time(s).
2010-06-15 12:43:08,697 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 2 time(s).
2010-06-15 12:43:09,700 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 3 time(s).
2010-06-15 12:43:10,703 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 4 time(s).
2010-06-15 12:43:11,706 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 5 time(s).
2010-06-15 12:43:12,709 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 6 time(s).
2010-06-15 12:43:13,712 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 7 time(s).
2010-06-15 12:43:14,714 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 8 time(s).
2010-06-15 12:43:15,717 INFO org.apache.hadoop.ipc.Client: Retrying
connect to server: /10.28.208.118:54310. Already tried 9 time(s).
2010-06-15 12:43:15,718 INFO org.apache.hadoop.ipc.RPC: Server at
/10.28.208.118:54310 not available yet, Zzzzz...

TaskTracker:
2010-06-15 12:42:57,270 INFO org.apache.hadoop.mapred.TaskTracker:
STARTUP_MSG:=20
/************************************************************
STARTUP_MSG: Starting TaskTracker
STARTUP_MSG:   host =3D centoshadoop.soa.gd-ais.com/10.28.208.118
STARTUP_MSG:   args =3D []
STARTUP_MSG:   version =3D 0.20.2
STARTUP_MSG:   build =3D
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2010-06-15 12:42:57,854 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2010-06-15 12:42:58,328 INFO org.apache.hadoop.http.HttpServer: Port
returned by webServer.getConnectors()[0].getLocalPort() before open() is
-1. Opening the listener on 50060
2010-06-15 12:42:58,344 INFO org.apache.hadoop.http.HttpServer:
listener.getLocalPort() returned 50060
webServer.getConnectors()[0].getLocalPort() returned 50060
2010-06-15 12:42:58,344 INFO org.apache.hadoop.http.HttpServer: Jetty
bound to port 50060
2010-06-15 12:42:58,345 INFO org.mortbay.log: jetty-6.1.14
2010-06-15 12:42:58,852 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:50060
2010-06-15 12:42:58,860 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=3DTaskTracker, sessionId=3D
2010-06-15 12:42:58,882 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=3DTaskTracker, port=3D50758
2010-06-15 12:42:58,960 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2010-06-15 12:42:58,962 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 50758: starting
2010-06-15 12:42:58,964 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 2 on 50758: starting
2010-06-15 12:42:58,964 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 50758: starting
2010-06-15 12:42:58,964 INFO org.apache.hadoop.mapred.TaskTracker:
TaskTracker up at: localhost.localdomain/127.0.0.1:50758
2010-06-15 12:42:58,964 INFO org.apache.hadoop.mapred.TaskTracker:
Starting tracker
tracker_centoshadoop:localhost.localdomain/127.0.0.1:50758
2010-06-15 12:42:58,969 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 3 on 50758: starting
2010-06-15 12:42:58,969 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 1 on 50758: starting

The config files are:
core-site.xml:
<?xml version=3D"1.0"?>
<?xml-stylesheet type=3D"text/xsl" href=3D"configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>hadoop.tmp.dir</name>
  <value>/hadoop-0.20.2/tmp/dir/hadoop-${user.name}</value>
  <description> A base for the other temporary
directories.</description>
  <final>true</final>
</property>
<property>
  <name>fs.default.name</name>
  <value>hdfs://10.28.208.118:54310</value>
  <description> The name of the default file system. A URI whose scheme
and auth
ority determine the filesystem implementation. The uri's scheme
determines the c
onfig property (fs.SCHEME.impl) name the FileSystem implementation
class. The ur
i's authority is used to determine the host, port, etc for a
filesystem</description>
  <final>true</final>
</property>

</configuration>

hadoop-env.sh
# Set Hadoop-specific environment variables here.

# The only required environment variable is JAVA_HOME.  All others are
# optional.  When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.

# The java implementation to use.  Required.
export JAVA_HOME=3D/usr/java/default

# The hadoop home directory
export HADOOP_HOME=3D/hadoop-0.20.2

# Extra Java CLASSPATH elements.  Optional.
export
HADOOP_CLASSPATH=3D/hbase-0.20.4/lib/zookeeper-3.2.2.jar:/habase-0.20.4/h=
b
ase-0.20.4-test.jar:/hbase-0.20.4/hbase-0.20.4.jar:/hbase-0.20.4/conf

# The maximum amount of heap to use, in MB. Default is 1000.
# export HADOOP_HEAPSIZE=3D2000

# Extra Java runtime options.  Empty by default.
export HADOOP_OPTS=3D-Djava.net.preferIPv4Stack=3Dtrue

# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS=3D"-Dcom.sun.management.jmxremote
$HADOOP_NAMENODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS=3D"-Dcom.sun.management.jmxremote
$HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_DATANODE_OPTS=3D"-Dcom.sun.management.jmxremote
$HADOOP_DATANODE_OPTS"
export HADOOP_BALANCER_OPTS=3D"-Dcom.sun.management.jmxremote
$HADOOP_BALANCER_OPTS"
export HADOOP_JOBTRACKER_OPTS=3D"-Dcom.sun.management.jmxremote
$HADOOP_JOBTRACKER_OPTS"
# export HADOOP_TASKTRACKER_OPTS=3D
# The following applies to multiple commands (fs, dfs, fsck, distcp etc)
# export HADOOP_CLIENT_OPTS

# Extra ssh options.  Empty by default.
# export HADOOP_SSH_OPTS=3D"-o ConnectTimeout=3D1 -o
SendEnv=3DHADOOP_CONF_DIR"

# Where log files are stored.  $HADOOP_HOME/logs by default.
# export HADOOP_LOG_DIR=3D${HADOOP_HOME}/logs

# File naming remote slave hosts.  $HADOOP_HOME/conf/slaves by default.
# export HADOOP_SLAVES=3D${HADOOP_HOME}/conf/slaves

# host:path where hadoop code should be rsync'd from.  Unset by default.
# export HADOOP_MASTER=3Dmaster:/home/$USER/src/hadoop

# Seconds to sleep between slave commands.  Unset by default.  This
# can be useful in large clusters, where, e.g., slave rsyncs can
# otherwise arrive faster than the master can service them.
# export HADOOP_SLAVE_SLEEP=3D0.1

# The directory where pid files are stored. /tmp by default.
# export HADOOP_PID_DIR=3D/var/hadoop/pids

# A string representing this instance of hadoop. $USER by default.
# export HADOOP_IDENT_STRING=3D$USER

# The scheduling priority for daemon processes.  See 'man nice'.
# export HADOOP_NICENESS=3D10

hdfs-site.xml
<?xml version=3D"1.0"?>
<?xml-stylesheet type=3D"text/xsl" href=3D"configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>dfs.replication</name>
  <value>22</value>
  <description> Default block replication. The actual number of
replications can be specified when the file is created. Default is used
if not specified at creation time.</description>
  <final>true</final>
</property>
<property>
  <name>dfs.data.dir</name>
  <value>/hadoop-0.20.2/hadoopDataStore/data</value>
  <description>DFS Data Directory</description>
  <final>true</final>
</property>

</configuration>

masters:
10.28.208.118

slaves:
10.28.208.118

(And because I know it'll be asked)
hosts:
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1		localhost.localdomain localhost=20
::1		localhost6.localdomain6 localhost6=20
10.28.208.118	centoshadoop centoshadoop.soa.gd-ais.com

Why is this failing to stand up? I'd like to get this one node working
before I stand up any other data nodes.(Firewall, btw, is turned off)

James Kilbride

From general-return-1645-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 15 19:00:33 2010
Return-Path: <general-return-1645-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 48068 invoked from network); 15 Jun 2010 19:00:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jun 2010 19:00:32 -0000
Received: (qmail 55641 invoked by uid 500); 15 Jun 2010 19:00:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55457 invoked by uid 500); 15 Jun 2010 19:00:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55448 invoked by uid 99); 15 Jun 2010 19:00:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 19:00:30 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of dms@facebook.com designates 69.63.178.185 as permitted sender)
Received: from [69.63.178.185] (HELO mx-out.facebook.com) (69.63.178.185)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 19:00:24 +0000
Received: from [10.18.255.133] ([10.18.255.133:25872] helo=mail.thefacebook.com)
	by mta005.snc1.facebook.com (envelope-from <dms@facebook.com>)
	(ecelerity 2.2.2.45 r(34067)) with ESMTP
	id 86/1B-23452-0BDC71C4; Tue, 15 Jun 2010 12:00:00 -0700
Received: from sc-hub06.TheFacebook.com (192.168.18.83) by
 sc-hub04.TheFacebook.com (192.168.18.212) with Microsoft SMTP Server (TLS) id
 14.0.694.1; Tue, 15 Jun 2010 12:00:00 -0700
Received: from SC-MBXC1.TheFacebook.com ([192.168.18.100]) by
 sc-hub06.TheFacebook.com ([192.168.18.83]) with mapi; Tue, 15 Jun 2010
 12:00:00 -0700
From: Dmytro Molkov <dms@facebook.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 15 Jun 2010 11:59:59 -0700
Subject: Re: IPC connections failing
Thread-Topic: IPC connections failing
Thread-Index: AcsMvPQdWK0LvVs4T5ugXLeUE9GxOA==
Message-ID: <DA6DE430-9A5A-4C59-A0EA-3D5B7F8CECAD@facebook.com>
References: <E0C1686B8F45434DA0E0A904672EA1A8065AC962@MAPF01-MAIL01.ad.gd-ais.com>
 <E0C1686B8F45434DA0E0A904672EA1A8065AC965@MAPF01-MAIL01.ad.gd-ais.com>
In-Reply-To: <E0C1686B8F45434DA0E0A904672EA1A8065AC965@MAPF01-MAIL01.ad.gd-ais.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Based on the log of the namenode I do not think it started successfully.
Seems like your edits log is corrupted
org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java=
:670)

So all other parts of the system could not connect to the namenode because =
it was down, not because of IPC problems.

On Jun 15, 2010, at 10:11 AM, Kilbride, James P. wrote:

> Forgot to add on mapred-site.xml:
>
> <?xml version=3D"1.0"?>
> <?xml-stylesheet type=3D"text/xsl" href=3D"configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
> <property>
>  <name>mapred.job.tracker</name>
>  <value>hdfs://10.28.208.118:54311</value>
>  <description> The host and port that the MapReduce job tracker runs
> at. If "local", then the jobs are run in-process as a single map and
> reduce task.</description>
>  <final>true</final>
> </property>
>
> </configuration>
>
> -----Original Message-----
> From: Kilbride, James P. [mailto:James.Kilbride@gd-ais.com]
> Sent: Tuesday, June 15, 2010 1:10 PM
> To: general@hadoop.apache.org
> Subject: IPC connections failing
>
> I've tried to move my system from pseudo distributed to full
> distributed(with it being the first node in the full cluster I'll be
> setting up) but the components don't seem to be talking to each other
> and I find logs filled with ipc.Client connection failures.
>
> Here's the basic log info:
> DataNode:
> 2010-06-15 12:42:42,824 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG:   host =3D centoshadoop.soa.gd-ais.com/10.28.208.118
> STARTUP_MSG:   args =3D []
> STARTUP_MSG:   version =3D 0.20.2
> STARTUP_MSG:   build =3D
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2010-06-15 12:42:49,335 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 0 time(s).
> 2010-06-15 12:42:50,343 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 1 time(s).
> 2010-06-15 12:42:51,354 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 2 time(s).
> 2010-06-15 12:42:53,641 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 3 time(s).
> 2010-06-15 12:42:54,733 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 4 time(s).
> 2010-06-15 12:42:55,737 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 5 time(s).
> 2010-06-15 12:42:56,824 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 6 time(s).
> 2010-06-15 12:42:57,827 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 7 time(s).
> 2010-06-15 12:42:58,831 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 8 time(s).
> 2010-06-15 12:42:59,833 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 9 time(s).
> 2010-06-15 12:42:59,836 INFO org.apache.hadoop.ipc.RPC: Server at
> /10.28.208.118:54310 not available yet, Zzzzz...
> 2010-06-15 12:43:01,841 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 0 time(s).
> 2010-06-15 12:43:02,844 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 1 time(s).
> 2010-06-15 12:43:03,846 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 2 time(s).
> 2010-06-15 12:43:04,849 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 3 time(s).
> 2010-06-15 12:43:05,852 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 4 time(s).
> 2010-06-15 12:43:06,855 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 5 time(s).
> 2010-06-15 12:43:07,858 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 6 time(s).
> 2010-06-15 12:43:08,883 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 7 time(s).
> 2010-06-15 12:43:09,970 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 8 time(s).
> 2010-06-15 12:43:10,973 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 9 time(s).
> 2010-06-15 12:43:10,973 INFO org.apache.hadoop.ipc.RPC: Server at
> /10.28.208.118:54310 not available yet, Zzzzz...
>
> JobTracker:
> 2010-06-15 12:42:52,836 INFO org.apache.hadoop.mapred.JobTracker:
> STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting JobTracker
> STARTUP_MSG:   host =3D centoshadoop.soa.gd-ais.com/10.28.208.118
> STARTUP_MSG:   args =3D []
> STARTUP_MSG:   version =3D 0.20.2
> STARTUP_MSG:   build =3D
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2010-06-15 12:42:53,350 INFO org.apache.hadoop.mapred.JobTracker:
> Scheduler configured with (memSizeForMapSlotOnJT,
> memSizeForReduceSlotOnJT, limitMaxMemForMapTasks,
> limitMaxMemForReduceTasks) (-1, -1, -1, -1)
> 2010-06-15 12:42:53,815 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> Initializing RPC Metrics with hostName=3DJobTracker, port=3D54311
> 2010-06-15 12:42:53,986 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2010-06-15 12:42:55,381 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open() is
> -1. Opening the listener on 50030
> 2010-06-15 12:42:55,400 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50030
> webServer.getConnectors()[0].getLocalPort() returned 50030
> 2010-06-15 12:42:55,400 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50030
> 2010-06-15 12:42:55,402 INFO org.mortbay.log: jetty-6.1.14
> 2010-06-15 12:42:56,802 INFO org.mortbay.log: Started
> SelectChannelConnector@0.0.0.0:50030
> 2010-06-15 12:42:56,805 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=3DJobTracker, sessionId=3D
> 2010-06-15 12:42:56,807 INFO org.apache.hadoop.mapred.JobTracker:
> JobTracker up at: 54311
> 2010-06-15 12:42:56,807 INFO org.apache.hadoop.mapred.JobTracker:
> JobTracker webserver: 50030
> 2010-06-15 12:42:58,076 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 0 time(s).
> 2010-06-15 12:42:59,080 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 1 time(s).
> 2010-06-15 12:43:00,083 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 2 time(s).
> 2010-06-15 12:43:01,086 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 3 time(s).
> 2010-06-15 12:43:02,089 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 4 time(s).
> 2010-06-15 12:43:03,091 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 5 time(s).
> 2010-06-15 12:43:04,095 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 6 time(s).
> 2010-06-15 12:43:05,174 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 7 time(s).
> 2010-06-15 12:43:06,177 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 8 time(s).
> 2010-06-15 12:43:07,180 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 9 time(s).
> 2010-06-15 12:43:07,185 INFO org.apache.hadoop.mapred.JobTracker:
> problem cleaning system directory: null
> java.net.ConnectException: Call to /10.28.208.118:54310 failed on
> connection exception: java.net.ConnectException: Connection refused
>        at org.apache.hadoop.ipc.Client.wrapException(Client.java:767)
>        at org.apache.hadoop.ipc.Client.call(Client.java:743)
>        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
>        at $Proxy4.getProtocolVersion(Unknown Source)
>        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
>        at
> org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
>        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
>        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
>        at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileS
> ystem.java:82)
>        at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
>        at
> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
>        at
> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
>        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
>        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
>        at
> org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1665)
>        at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
>        at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
>        at
> org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
> Caused by: java.net.ConnectException: Connection refused
>        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>        at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>        at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.ja
> va:206)
>        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
>        at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:304)
>        at
> org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
>        at org.apache.hadoop.ipc.Client.getConnection(Client.java:860)
>        at org.apache.hadoop.ipc.Client.call(Client.java:720)
>        ... 16 more
>
> NameNode:
> 2010-06-15 12:42:38,714 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG:   host =3D centoshadoop.soa.gd-ais.com/10.28.208.118
> STARTUP_MSG:   args =3D []
> STARTUP_MSG:   version =3D 0.20.2
> STARTUP_MSG:   build =3D
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2010-06-15 12:42:38,922 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> Initializing RPC Metrics with hostName=3DNameNode, port=3D54310
> 2010-06-15 12:42:38,932 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Namenode up at:
> centoshadoop/10.28.208.118:54310
> 2010-06-15 12:42:38,938 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=3DNameNode, sessionId=3Dnull
> 2010-06-15 12:42:38,941 INFO
> org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics:
> Initializing NameNodeMeterics using context
> object:org.apache.hadoop.metrics.spi.NullContext
> 2010-06-15 12:42:39,140 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> fsOwner=3Dhadoop,users,hadoop
> 2010-06-15 12:42:39,141 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> supergroup=3Dsupergroup
> 2010-06-15 12:42:39,141 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> isPermissionEnabled=3Dtrue
> 2010-06-15 12:42:39,156 INFO
> org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics:
> Initializing FSNamesystemMetrics using context
> object:org.apache.hadoop.metrics.spi.NullContext
> 2010-06-15 12:42:39,161 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
> FSNamesystemStatusMBean
> 2010-06-15 12:42:39,381 INFO
> org.apache.hadoop.hdfs.server.common.Storage: Number of files =3D 28
> 2010-06-15 12:42:39,404 INFO
> org.apache.hadoop.hdfs.server.common.Storage: Number of files under
> construction =3D 0
> 2010-06-15 12:42:39,404 INFO
> org.apache.hadoop.hdfs.server.common.Storage: Image file of size 2780
> loaded in 0 seconds.
> 2010-06-15 12:42:39,429 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode:
> java.lang.NumberFormatException: For input string: ""
>        at
> java.lang.NumberFormatException.forInputString(NumberFormatException.jav
> a:48)
>        at java.lang.Long.parseLong(Long.java:431)
>        at java.lang.Long.parseLong(Long.java:468)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.readLong(FSEditLog.java
> :1273)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.j
> ava:670)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:
> 992)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:
> 812)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSI
> mage.java:364)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirecto
> ry.java:87)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesys
> tem.java:311)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.
> java:292)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java
> :201)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279
> )
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.
> java:956)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
>
> SecondaryNameNode:
> 2010-06-15 12:42:53,227 INFO
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting SecondaryNameNode
> STARTUP_MSG:   host =3D centoshadoop.soa.gd-ais.com/10.28.208.118
> STARTUP_MSG:   args =3D []
> STARTUP_MSG:   version =3D 0.20.2
> STARTUP_MSG:   build =3D
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2010-06-15 12:42:53,442 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=3DSecondaryNameNode,
> sessionId=3Dnull
> 2010-06-15 12:42:55,651 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 0 time(s).
> 2010-06-15 12:42:56,659 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 1 time(s).
> 2010-06-15 12:42:57,662 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 2 time(s).
> 2010-06-15 12:42:58,665 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 3 time(s).
> 2010-06-15 12:42:59,668 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 4 time(s).
> 2010-06-15 12:43:00,671 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 5 time(s).
> 2010-06-15 12:43:01,674 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 6 time(s).
> 2010-06-15 12:43:02,677 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 7 time(s).
> 2010-06-15 12:43:03,679 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 8 time(s).
> 2010-06-15 12:43:04,682 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 9 time(s).
> 2010-06-15 12:43:04,685 INFO org.apache.hadoop.ipc.RPC: Server at
> /10.28.208.118:54310 not available yet, Zzzzz...
> 2010-06-15 12:43:06,691 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 0 time(s).
> 2010-06-15 12:43:07,694 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 1 time(s).
> 2010-06-15 12:43:08,697 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 2 time(s).
> 2010-06-15 12:43:09,700 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 3 time(s).
> 2010-06-15 12:43:10,703 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 4 time(s).
> 2010-06-15 12:43:11,706 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 5 time(s).
> 2010-06-15 12:43:12,709 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 6 time(s).
> 2010-06-15 12:43:13,712 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 7 time(s).
> 2010-06-15 12:43:14,714 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 8 time(s).
> 2010-06-15 12:43:15,717 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 9 time(s).
> 2010-06-15 12:43:15,718 INFO org.apache.hadoop.ipc.RPC: Server at
> /10.28.208.118:54310 not available yet, Zzzzz...
>
> TaskTracker:
> 2010-06-15 12:42:57,270 INFO org.apache.hadoop.mapred.TaskTracker:
> STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting TaskTracker
> STARTUP_MSG:   host =3D centoshadoop.soa.gd-ais.com/10.28.208.118
> STARTUP_MSG:   args =3D []
> STARTUP_MSG:   version =3D 0.20.2
> STARTUP_MSG:   build =3D
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2010-06-15 12:42:57,854 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2010-06-15 12:42:58,328 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open() is
> -1. Opening the listener on 50060
> 2010-06-15 12:42:58,344 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50060
> webServer.getConnectors()[0].getLocalPort() returned 50060
> 2010-06-15 12:42:58,344 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50060
> 2010-06-15 12:42:58,345 INFO org.mortbay.log: jetty-6.1.14
> 2010-06-15 12:42:58,852 INFO org.mortbay.log: Started
> SelectChannelConnector@0.0.0.0:50060
> 2010-06-15 12:42:58,860 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=3DTaskTracker, sessionId=3D
> 2010-06-15 12:42:58,882 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> Initializing RPC Metrics with hostName=3DTaskTracker, port=3D50758
> 2010-06-15 12:42:58,960 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
> 2010-06-15 12:42:58,962 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 50758: starting
> 2010-06-15 12:42:58,964 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 2 on 50758: starting
> 2010-06-15 12:42:58,964 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 0 on 50758: starting
> 2010-06-15 12:42:58,964 INFO org.apache.hadoop.mapred.TaskTracker:
> TaskTracker up at: localhost.localdomain/127.0.0.1:50758
> 2010-06-15 12:42:58,964 INFO org.apache.hadoop.mapred.TaskTracker:
> Starting tracker
> tracker_centoshadoop:localhost.localdomain/127.0.0.1:50758
> 2010-06-15 12:42:58,969 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 3 on 50758: starting
> 2010-06-15 12:42:58,969 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 50758: starting
>
> The config files are:
> core-site.xml:
> <?xml version=3D"1.0"?>
> <?xml-stylesheet type=3D"text/xsl" href=3D"configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
> <property>
>  <name>hadoop.tmp.dir</name>
>  <value>/hadoop-0.20.2/tmp/dir/hadoop-${user.name}</value>
>  <description> A base for the other temporary
> directories.</description>
>  <final>true</final>
> </property>
> <property>
>  <name>fs.default.name</name>
>  <value>hdfs://10.28.208.118:54310</value>
>  <description> The name of the default file system. A URI whose scheme
> and auth
> ority determine the filesystem implementation. The uri's scheme
> determines the c
> onfig property (fs.SCHEME.impl) name the FileSystem implementation
> class. The ur
> i's authority is used to determine the host, port, etc for a
> filesystem</description>
>  <final>true</final>
> </property>
>
> </configuration>
>
> hadoop-env.sh
> # Set Hadoop-specific environment variables here.
>
> # The only required environment variable is JAVA_HOME.  All others are
> # optional.  When running a distributed configuration it is best to
> # set JAVA_HOME in this file, so that it is correctly defined on
> # remote nodes.
>
> # The java implementation to use.  Required.
> export JAVA_HOME=3D/usr/java/default
>
> # The hadoop home directory
> export HADOOP_HOME=3D/hadoop-0.20.2
>
> # Extra Java CLASSPATH elements.  Optional.
> export
> HADOOP_CLASSPATH=3D/hbase-0.20.4/lib/zookeeper-3.2.2.jar:/habase-0.20.4/h=
b
> ase-0.20.4-test.jar:/hbase-0.20.4/hbase-0.20.4.jar:/hbase-0.20.4/conf
>
> # The maximum amount of heap to use, in MB. Default is 1000.
> # export HADOOP_HEAPSIZE=3D2000
>
> # Extra Java runtime options.  Empty by default.
> export HADOOP_OPTS=3D-Djava.net.preferIPv4Stack=3Dtrue
>
> # Command specific options appended to HADOOP_OPTS when specified
> export HADOOP_NAMENODE_OPTS=3D"-Dcom.sun.management.jmxremote
> $HADOOP_NAMENODE_OPTS"
> export HADOOP_SECONDARYNAMENODE_OPTS=3D"-Dcom.sun.management.jmxremote
> $HADOOP_SECONDARYNAMENODE_OPTS"
> export HADOOP_DATANODE_OPTS=3D"-Dcom.sun.management.jmxremote
> $HADOOP_DATANODE_OPTS"
> export HADOOP_BALANCER_OPTS=3D"-Dcom.sun.management.jmxremote
> $HADOOP_BALANCER_OPTS"
> export HADOOP_JOBTRACKER_OPTS=3D"-Dcom.sun.management.jmxremote
> $HADOOP_JOBTRACKER_OPTS"
> # export HADOOP_TASKTRACKER_OPTS=3D
> # The following applies to multiple commands (fs, dfs, fsck, distcp etc)
> # export HADOOP_CLIENT_OPTS
>
> # Extra ssh options.  Empty by default.
> # export HADOOP_SSH_OPTS=3D"-o ConnectTimeout=3D1 -o
> SendEnv=3DHADOOP_CONF_DIR"
>
> # Where log files are stored.  $HADOOP_HOME/logs by default.
> # export HADOOP_LOG_DIR=3D${HADOOP_HOME}/logs
>
> # File naming remote slave hosts.  $HADOOP_HOME/conf/slaves by default.
> # export HADOOP_SLAVES=3D${HADOOP_HOME}/conf/slaves
>
> # host:path where hadoop code should be rsync'd from.  Unset by default.
> # export HADOOP_MASTER=3Dmaster:/home/$USER/src/hadoop
>
> # Seconds to sleep between slave commands.  Unset by default.  This
> # can be useful in large clusters, where, e.g., slave rsyncs can
> # otherwise arrive faster than the master can service them.
> # export HADOOP_SLAVE_SLEEP=3D0.1
>
> # The directory where pid files are stored. /tmp by default.
> # export HADOOP_PID_DIR=3D/var/hadoop/pids
>
> # A string representing this instance of hadoop. $USER by default.
> # export HADOOP_IDENT_STRING=3D$USER
>
> # The scheduling priority for daemon processes.  See 'man nice'.
> # export HADOOP_NICENESS=3D10
>
> hdfs-site.xml
> <?xml version=3D"1.0"?>
> <?xml-stylesheet type=3D"text/xsl" href=3D"configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
> <property>
>  <name>dfs.replication</name>
>  <value>22</value>
>  <description> Default block replication. The actual number of
> replications can be specified when the file is created. Default is used
> if not specified at creation time.</description>
>  <final>true</final>
> </property>
> <property>
>  <name>dfs.data.dir</name>
>  <value>/hadoop-0.20.2/hadoopDataStore/data</value>
>  <description>DFS Data Directory</description>
>  <final>true</final>
> </property>
>
> </configuration>
>
> masters:
> 10.28.208.118
>
> slaves:
> 10.28.208.118
>
> (And because I know it'll be asked)
> hosts:
> # Do not remove the following line, or various programs
> # that require network functionality will fail.
> 127.0.0.1               localhost.localdomain localhost
> ::1             localhost6.localdomain6 localhost6
> 10.28.208.118   centoshadoop centoshadoop.soa.gd-ais.com
>
> Why is this failing to stand up? I'd like to get this one node working
> before I stand up any other data nodes.(Firewall, btw, is turned off)
>
> James Kilbride


From general-return-1646-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 15 19:47:00 2010
Return-Path: <general-return-1646-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 71255 invoked from network); 15 Jun 2010 19:47:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jun 2010 19:47:00 -0000
Received: (qmail 20605 invoked by uid 500); 15 Jun 2010 19:46:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20484 invoked by uid 500); 15 Jun 2010 19:46:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 20476 invoked by uid 99); 15 Jun 2010 19:46:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 19:46:58 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [192.5.164.99] (HELO camv02-relay2.casc.gd-ais.com) (192.5.164.99)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 19:46:50 +0000
Received: from ([10.73.100.22])
	by camv02-relay2.casc.gd-ais.com with SMTP  id 5203374.36116650;
	Tue, 15 Jun 2010 12:46:25 -0700
Received: from MAPF01-MAIL01.ad.gd-ais.com ([166.16.220.104]) by camv02-fes01.ad.gd-ais.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Tue, 15 Jun 2010 12:46:25 -0700
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: RE: IPC connections failing
Date: Tue, 15 Jun 2010 15:46:23 -0400
Message-ID: <E0C1686B8F45434DA0E0A904672EA1A8065E6B15@MAPF01-MAIL01.ad.gd-ais.com>
In-Reply-To: <DA6DE430-9A5A-4C59-A0EA-3D5B7F8CECAD@facebook.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: IPC connections failing
Thread-Index: AcsMvPQdWK0LvVs4T5ugXLeUE9GxOAABl81Q
References: <E0C1686B8F45434DA0E0A904672EA1A8065AC962@MAPF01-MAIL01.ad.gd-ais.com> <E0C1686B8F45434DA0E0A904672EA1A8065AC965@MAPF01-MAIL01.ad.gd-ais.com> <DA6DE430-9A5A-4C59-A0EA-3D5B7F8CECAD@facebook.com>
From: "Kilbride, James P." <James.Kilbride@gd-ais.com>
To: <general@hadoop.apache.org>
X-OriginalArrivalTime: 15 Jun 2010 19:46:25.0246 (UTC) FILETIME=[704D47E0:01CB0CC3]
X-Virus-Checked: Checked by ClamAV on apache.org

How do I repair that and prevent it from happening again?

James Kilbride

-----Original Message-----
From: Dmytro Molkov [mailto:dms@facebook.com]=20
Sent: Tuesday, June 15, 2010 3:00 PM
To: general@hadoop.apache.org
Subject: Re: IPC connections failing

Based on the log of the namenode I do not think it started successfully.
Seems like your edits log is corrupted
org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.j
ava:670)

So all other parts of the system could not connect to the namenode
because it was down, not because of IPC problems.

On Jun 15, 2010, at 10:11 AM, Kilbride, James P. wrote:

> Forgot to add on mapred-site.xml:
>
> <?xml version=3D"1.0"?>
> <?xml-stylesheet type=3D"text/xsl" href=3D"configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
> <property>
>  <name>mapred.job.tracker</name>
>  <value>hdfs://10.28.208.118:54311</value>
>  <description> The host and port that the MapReduce job tracker runs
> at. If "local", then the jobs are run in-process as a single map and
> reduce task.</description>
>  <final>true</final>
> </property>
>
> </configuration>
>
> -----Original Message-----
> From: Kilbride, James P. [mailto:James.Kilbride@gd-ais.com]
> Sent: Tuesday, June 15, 2010 1:10 PM
> To: general@hadoop.apache.org
> Subject: IPC connections failing
>
> I've tried to move my system from pseudo distributed to full
> distributed(with it being the first node in the full cluster I'll be
> setting up) but the components don't seem to be talking to each other
> and I find logs filled with ipc.Client connection failures.
>
> Here's the basic log info:
> DataNode:
> 2010-06-15 12:42:42,824 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG:   host =3D centoshadoop.soa.gd-ais.com/10.28.208.118
> STARTUP_MSG:   args =3D []
> STARTUP_MSG:   version =3D 0.20.2
> STARTUP_MSG:   build =3D
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2010-06-15 12:42:49,335 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 0 time(s).
> 2010-06-15 12:42:50,343 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 1 time(s).
> 2010-06-15 12:42:51,354 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 2 time(s).
> 2010-06-15 12:42:53,641 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 3 time(s).
> 2010-06-15 12:42:54,733 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 4 time(s).
> 2010-06-15 12:42:55,737 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 5 time(s).
> 2010-06-15 12:42:56,824 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 6 time(s).
> 2010-06-15 12:42:57,827 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 7 time(s).
> 2010-06-15 12:42:58,831 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 8 time(s).
> 2010-06-15 12:42:59,833 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 9 time(s).
> 2010-06-15 12:42:59,836 INFO org.apache.hadoop.ipc.RPC: Server at
> /10.28.208.118:54310 not available yet, Zzzzz...
> 2010-06-15 12:43:01,841 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 0 time(s).
> 2010-06-15 12:43:02,844 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 1 time(s).
> 2010-06-15 12:43:03,846 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 2 time(s).
> 2010-06-15 12:43:04,849 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 3 time(s).
> 2010-06-15 12:43:05,852 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 4 time(s).
> 2010-06-15 12:43:06,855 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 5 time(s).
> 2010-06-15 12:43:07,858 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 6 time(s).
> 2010-06-15 12:43:08,883 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 7 time(s).
> 2010-06-15 12:43:09,970 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 8 time(s).
> 2010-06-15 12:43:10,973 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 9 time(s).
> 2010-06-15 12:43:10,973 INFO org.apache.hadoop.ipc.RPC: Server at
> /10.28.208.118:54310 not available yet, Zzzzz...
>
> JobTracker:
> 2010-06-15 12:42:52,836 INFO org.apache.hadoop.mapred.JobTracker:
> STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting JobTracker
> STARTUP_MSG:   host =3D centoshadoop.soa.gd-ais.com/10.28.208.118
> STARTUP_MSG:   args =3D []
> STARTUP_MSG:   version =3D 0.20.2
> STARTUP_MSG:   build =3D
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2010-06-15 12:42:53,350 INFO org.apache.hadoop.mapred.JobTracker:
> Scheduler configured with (memSizeForMapSlotOnJT,
> memSizeForReduceSlotOnJT, limitMaxMemForMapTasks,
> limitMaxMemForReduceTasks) (-1, -1, -1, -1)
> 2010-06-15 12:42:53,815 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> Initializing RPC Metrics with hostName=3DJobTracker, port=3D54311
> 2010-06-15 12:42:53,986 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2010-06-15 12:42:55,381 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open()
is
> -1. Opening the listener on 50030
> 2010-06-15 12:42:55,400 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50030
> webServer.getConnectors()[0].getLocalPort() returned 50030
> 2010-06-15 12:42:55,400 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50030
> 2010-06-15 12:42:55,402 INFO org.mortbay.log: jetty-6.1.14
> 2010-06-15 12:42:56,802 INFO org.mortbay.log: Started
> SelectChannelConnector@0.0.0.0:50030
> 2010-06-15 12:42:56,805 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=3DJobTracker, sessionId=3D
> 2010-06-15 12:42:56,807 INFO org.apache.hadoop.mapred.JobTracker:
> JobTracker up at: 54311
> 2010-06-15 12:42:56,807 INFO org.apache.hadoop.mapred.JobTracker:
> JobTracker webserver: 50030
> 2010-06-15 12:42:58,076 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 0 time(s).
> 2010-06-15 12:42:59,080 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 1 time(s).
> 2010-06-15 12:43:00,083 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 2 time(s).
> 2010-06-15 12:43:01,086 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 3 time(s).
> 2010-06-15 12:43:02,089 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 4 time(s).
> 2010-06-15 12:43:03,091 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 5 time(s).
> 2010-06-15 12:43:04,095 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 6 time(s).
> 2010-06-15 12:43:05,174 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 7 time(s).
> 2010-06-15 12:43:06,177 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 8 time(s).
> 2010-06-15 12:43:07,180 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 9 time(s).
> 2010-06-15 12:43:07,185 INFO org.apache.hadoop.mapred.JobTracker:
> problem cleaning system directory: null
> java.net.ConnectException: Call to /10.28.208.118:54310 failed on
> connection exception: java.net.ConnectException: Connection refused
>        at org.apache.hadoop.ipc.Client.wrapException(Client.java:767)
>        at org.apache.hadoop.ipc.Client.call(Client.java:743)
>        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
>        at $Proxy4.getProtocolVersion(Unknown Source)
>        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
>        at
> org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
>        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
>        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
>        at
>
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileS
> ystem.java:82)
>        at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
>        at
> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
>        at
> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
>        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
>        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
>        at
> org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1665)
>        at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
>        at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
>        at
> org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
> Caused by: java.net.ConnectException: Connection refused
>        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>        at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>        at
>
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.ja
> va:206)
>        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
>        at
>
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:304)
>        at
> org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
>        at org.apache.hadoop.ipc.Client.getConnection(Client.java:860)
>        at org.apache.hadoop.ipc.Client.call(Client.java:720)
>        ... 16 more
>
> NameNode:
> 2010-06-15 12:42:38,714 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG:   host =3D centoshadoop.soa.gd-ais.com/10.28.208.118
> STARTUP_MSG:   args =3D []
> STARTUP_MSG:   version =3D 0.20.2
> STARTUP_MSG:   build =3D
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2010-06-15 12:42:38,922 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> Initializing RPC Metrics with hostName=3DNameNode, port=3D54310
> 2010-06-15 12:42:38,932 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Namenode up at:
> centoshadoop/10.28.208.118:54310
> 2010-06-15 12:42:38,938 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=3DNameNode, sessionId=3Dnull
> 2010-06-15 12:42:38,941 INFO
> org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics:
> Initializing NameNodeMeterics using context
> object:org.apache.hadoop.metrics.spi.NullContext
> 2010-06-15 12:42:39,140 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> fsOwner=3Dhadoop,users,hadoop
> 2010-06-15 12:42:39,141 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> supergroup=3Dsupergroup
> 2010-06-15 12:42:39,141 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> isPermissionEnabled=3Dtrue
> 2010-06-15 12:42:39,156 INFO
> org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics:
> Initializing FSNamesystemMetrics using context
> object:org.apache.hadoop.metrics.spi.NullContext
> 2010-06-15 12:42:39,161 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
> FSNamesystemStatusMBean
> 2010-06-15 12:42:39,381 INFO
> org.apache.hadoop.hdfs.server.common.Storage: Number of files =3D 28
> 2010-06-15 12:42:39,404 INFO
> org.apache.hadoop.hdfs.server.common.Storage: Number of files under
> construction =3D 0
> 2010-06-15 12:42:39,404 INFO
> org.apache.hadoop.hdfs.server.common.Storage: Image file of size 2780
> loaded in 0 seconds.
> 2010-06-15 12:42:39,429 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode:
> java.lang.NumberFormatException: For input string: ""
>        at
>
java.lang.NumberFormatException.forInputString(NumberFormatException.jav
> a:48)
>        at java.lang.Long.parseLong(Long.java:431)
>        at java.lang.Long.parseLong(Long.java:468)
>        at
>
org.apache.hadoop.hdfs.server.namenode.FSEditLog.readLong(FSEditLog.java
> :1273)
>        at
>
org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.j
> ava:670)
>        at
>
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:
> 992)
>        at
>
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:
> 812)
>        at
>
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSI
> mage.java:364)
>        at
>
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirecto
> ry.java:87)
>        at
>
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesys
> tem.java:311)
>        at
>
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.
> java:292)
>        at
>
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java
> :201)
>        at
>
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279
> )
>        at
>
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.
> java:956)
>        at
>
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
>
> SecondaryNameNode:
> 2010-06-15 12:42:53,227 INFO
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting SecondaryNameNode
> STARTUP_MSG:   host =3D centoshadoop.soa.gd-ais.com/10.28.208.118
> STARTUP_MSG:   args =3D []
> STARTUP_MSG:   version =3D 0.20.2
> STARTUP_MSG:   build =3D
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2010-06-15 12:42:53,442 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=3DSecondaryNameNode,
> sessionId=3Dnull
> 2010-06-15 12:42:55,651 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 0 time(s).
> 2010-06-15 12:42:56,659 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 1 time(s).
> 2010-06-15 12:42:57,662 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 2 time(s).
> 2010-06-15 12:42:58,665 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 3 time(s).
> 2010-06-15 12:42:59,668 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 4 time(s).
> 2010-06-15 12:43:00,671 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 5 time(s).
> 2010-06-15 12:43:01,674 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 6 time(s).
> 2010-06-15 12:43:02,677 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 7 time(s).
> 2010-06-15 12:43:03,679 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 8 time(s).
> 2010-06-15 12:43:04,682 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 9 time(s).
> 2010-06-15 12:43:04,685 INFO org.apache.hadoop.ipc.RPC: Server at
> /10.28.208.118:54310 not available yet, Zzzzz...
> 2010-06-15 12:43:06,691 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 0 time(s).
> 2010-06-15 12:43:07,694 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 1 time(s).
> 2010-06-15 12:43:08,697 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 2 time(s).
> 2010-06-15 12:43:09,700 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 3 time(s).
> 2010-06-15 12:43:10,703 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 4 time(s).
> 2010-06-15 12:43:11,706 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 5 time(s).
> 2010-06-15 12:43:12,709 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 6 time(s).
> 2010-06-15 12:43:13,712 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 7 time(s).
> 2010-06-15 12:43:14,714 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 8 time(s).
> 2010-06-15 12:43:15,717 INFO org.apache.hadoop.ipc.Client: Retrying
> connect to server: /10.28.208.118:54310. Already tried 9 time(s).
> 2010-06-15 12:43:15,718 INFO org.apache.hadoop.ipc.RPC: Server at
> /10.28.208.118:54310 not available yet, Zzzzz...
>
> TaskTracker:
> 2010-06-15 12:42:57,270 INFO org.apache.hadoop.mapred.TaskTracker:
> STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting TaskTracker
> STARTUP_MSG:   host =3D centoshadoop.soa.gd-ais.com/10.28.208.118
> STARTUP_MSG:   args =3D []
> STARTUP_MSG:   version =3D 0.20.2
> STARTUP_MSG:   build =3D
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2010-06-15 12:42:57,854 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2010-06-15 12:42:58,328 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open()
is
> -1. Opening the listener on 50060
> 2010-06-15 12:42:58,344 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50060
> webServer.getConnectors()[0].getLocalPort() returned 50060
> 2010-06-15 12:42:58,344 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50060
> 2010-06-15 12:42:58,345 INFO org.mortbay.log: jetty-6.1.14
> 2010-06-15 12:42:58,852 INFO org.mortbay.log: Started
> SelectChannelConnector@0.0.0.0:50060
> 2010-06-15 12:42:58,860 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=3DTaskTracker, sessionId=3D
> 2010-06-15 12:42:58,882 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> Initializing RPC Metrics with hostName=3DTaskTracker, port=3D50758
> 2010-06-15 12:42:58,960 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
> 2010-06-15 12:42:58,962 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 50758: starting
> 2010-06-15 12:42:58,964 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 2 on 50758: starting
> 2010-06-15 12:42:58,964 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 0 on 50758: starting
> 2010-06-15 12:42:58,964 INFO org.apache.hadoop.mapred.TaskTracker:
> TaskTracker up at: localhost.localdomain/127.0.0.1:50758
> 2010-06-15 12:42:58,964 INFO org.apache.hadoop.mapred.TaskTracker:
> Starting tracker
> tracker_centoshadoop:localhost.localdomain/127.0.0.1:50758
> 2010-06-15 12:42:58,969 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 3 on 50758: starting
> 2010-06-15 12:42:58,969 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 50758: starting
>
> The config files are:
> core-site.xml:
> <?xml version=3D"1.0"?>
> <?xml-stylesheet type=3D"text/xsl" href=3D"configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
> <property>
>  <name>hadoop.tmp.dir</name>
>  <value>/hadoop-0.20.2/tmp/dir/hadoop-${user.name}</value>
>  <description> A base for the other temporary
> directories.</description>
>  <final>true</final>
> </property>
> <property>
>  <name>fs.default.name</name>
>  <value>hdfs://10.28.208.118:54310</value>
>  <description> The name of the default file system. A URI whose scheme
> and auth
> ority determine the filesystem implementation. The uri's scheme
> determines the c
> onfig property (fs.SCHEME.impl) name the FileSystem implementation
> class. The ur
> i's authority is used to determine the host, port, etc for a
> filesystem</description>
>  <final>true</final>
> </property>
>
> </configuration>
>
> hadoop-env.sh
> # Set Hadoop-specific environment variables here.
>
> # The only required environment variable is JAVA_HOME.  All others are
> # optional.  When running a distributed configuration it is best to
> # set JAVA_HOME in this file, so that it is correctly defined on
> # remote nodes.
>
> # The java implementation to use.  Required.
> export JAVA_HOME=3D/usr/java/default
>
> # The hadoop home directory
> export HADOOP_HOME=3D/hadoop-0.20.2
>
> # Extra Java CLASSPATH elements.  Optional.
> export
>
HADOOP_CLASSPATH=3D/hbase-0.20.4/lib/zookeeper-3.2.2.jar:/habase-0.20.4/h=
b
> ase-0.20.4-test.jar:/hbase-0.20.4/hbase-0.20.4.jar:/hbase-0.20.4/conf
>
> # The maximum amount of heap to use, in MB. Default is 1000.
> # export HADOOP_HEAPSIZE=3D2000
>
> # Extra Java runtime options.  Empty by default.
> export HADOOP_OPTS=3D-Djava.net.preferIPv4Stack=3Dtrue
>
> # Command specific options appended to HADOOP_OPTS when specified
> export HADOOP_NAMENODE_OPTS=3D"-Dcom.sun.management.jmxremote
> $HADOOP_NAMENODE_OPTS"
> export HADOOP_SECONDARYNAMENODE_OPTS=3D"-Dcom.sun.management.jmxremote
> $HADOOP_SECONDARYNAMENODE_OPTS"
> export HADOOP_DATANODE_OPTS=3D"-Dcom.sun.management.jmxremote
> $HADOOP_DATANODE_OPTS"
> export HADOOP_BALANCER_OPTS=3D"-Dcom.sun.management.jmxremote
> $HADOOP_BALANCER_OPTS"
> export HADOOP_JOBTRACKER_OPTS=3D"-Dcom.sun.management.jmxremote
> $HADOOP_JOBTRACKER_OPTS"
> # export HADOOP_TASKTRACKER_OPTS=3D
> # The following applies to multiple commands (fs, dfs, fsck, distcp
etc)
> # export HADOOP_CLIENT_OPTS
>
> # Extra ssh options.  Empty by default.
> # export HADOOP_SSH_OPTS=3D"-o ConnectTimeout=3D1 -o
> SendEnv=3DHADOOP_CONF_DIR"
>
> # Where log files are stored.  $HADOOP_HOME/logs by default.
> # export HADOOP_LOG_DIR=3D${HADOOP_HOME}/logs
>
> # File naming remote slave hosts.  $HADOOP_HOME/conf/slaves by
default.
> # export HADOOP_SLAVES=3D${HADOOP_HOME}/conf/slaves
>
> # host:path where hadoop code should be rsync'd from.  Unset by
default.
> # export HADOOP_MASTER=3Dmaster:/home/$USER/src/hadoop
>
> # Seconds to sleep between slave commands.  Unset by default.  This
> # can be useful in large clusters, where, e.g., slave rsyncs can
> # otherwise arrive faster than the master can service them.
> # export HADOOP_SLAVE_SLEEP=3D0.1
>
> # The directory where pid files are stored. /tmp by default.
> # export HADOOP_PID_DIR=3D/var/hadoop/pids
>
> # A string representing this instance of hadoop. $USER by default.
> # export HADOOP_IDENT_STRING=3D$USER
>
> # The scheduling priority for daemon processes.  See 'man nice'.
> # export HADOOP_NICENESS=3D10
>
> hdfs-site.xml
> <?xml version=3D"1.0"?>
> <?xml-stylesheet type=3D"text/xsl" href=3D"configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
> <property>
>  <name>dfs.replication</name>
>  <value>22</value>
>  <description> Default block replication. The actual number of
> replications can be specified when the file is created. Default is
used
> if not specified at creation time.</description>
>  <final>true</final>
> </property>
> <property>
>  <name>dfs.data.dir</name>
>  <value>/hadoop-0.20.2/hadoopDataStore/data</value>
>  <description>DFS Data Directory</description>
>  <final>true</final>
> </property>
>
> </configuration>
>
> masters:
> 10.28.208.118
>
> slaves:
> 10.28.208.118
>
> (And because I know it'll be asked)
> hosts:
> # Do not remove the following line, or various programs
> # that require network functionality will fail.
> 127.0.0.1               localhost.localdomain localhost
> ::1             localhost6.localdomain6 localhost6
> 10.28.208.118   centoshadoop centoshadoop.soa.gd-ais.com
>
> Why is this failing to stand up? I'd like to get this one node working
> before I stand up any other data nodes.(Firewall, btw, is turned off)
>
> James Kilbride


From general-return-1647-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 16 06:57:48 2010
Return-Path: <general-return-1647-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70253 invoked from network); 16 Jun 2010 06:57:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 06:57:47 -0000
Received: (qmail 73938 invoked by uid 500); 16 Jun 2010 06:57:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73345 invoked by uid 500); 16 Jun 2010 06:57:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73331 invoked by uid 99); 16 Jun 2010 06:57:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 06:57:42 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_HELO_PASS,SPF_NEUTRAL,T_FRT_STOCK1,T_FRT_STOCK2
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [212.227.126.186] (HELO moutng.kundenserver.de) (212.227.126.186)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 06:57:35 +0000
Received: from smarthost.q2web.local (pd95b3aaa.dip0.t-ipconnect.de [217.91.58.170])
	by mrelayeu.kundenserver.de (node=mrbap2) with ESMTP (Nemesis)
	id 0Lmel1-1OywWX3v75-00aAD6; Wed, 16 Jun 2010 08:57:15 +0200
Received: from amidalasrv.q2web.local (amidalasrv.q2web.local [192.168.0.2])
	by smarthost.q2web.local (Postfix) with ESMTP id 9AA9C6F4E8
	for <general@hadoop.apache.org>; Wed, 16 Jun 2010 09:33:01 +0200 (CEST)
x-mimeole: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
Subject: AW: HDFS file append
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Date: Wed, 16 Jun 2010 08:57:11 +0200
Message-ID: <A3E731144A0BAE4A9A42C78EDEB9ADB80C4ED5@amidalasrv.q2web.local>
In-Reply-To: <AANLkTikW1Ca-oSa9CMGkqc2oJRUD6IL7M6E2ELhx6l2W@mail.gmail.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: HDFS file append
Thread-Index: AcsMq7FsvnETETDHTs2tISQPAxC1SwAdT8rQ
References: <A3E731144A0BAE4A9A42C78EDEB9ADB80C4ED3@amidalasrv.q2web.local> <AANLkTin0tQJchosVkRSat5xc45RQ7SUzx748AtPgzsh3@mail.gmail.com> <AANLkTilx_SdnHlu4XMfKd2J9Y9Uz86WBsIE94l_RW_oV@mail.gmail.com> <AANLkTikW1Ca-oSa9CMGkqc2oJRUD6IL7M6E2ELhx6l2W@mail.gmail.com>
From: =?iso-8859-1?Q?Jan_St=F6cker?= <Jan.Stoecker@q2web.de>
To: <general@hadoop.apache.org>
X-Provags-ID: V01U2FsdGVkX18rNi1X51omnjl1w31ay/l10G64M6J648x9SZN
 lS/spebs2kMRFbXeGs40J+7MqQ15tzdc4OCvzmCkUUj3TL/Qyp
 tcXFD7nbfuw6GlPr2DQHRPewcjpJ9aJ
X-Virus-Checked: Checked by ClamAV on apache.org

O.k., thanks for the information & the suggestions!

Jan

-----Urspr=FCngliche Nachricht-----
Von: Todd Lipcon [mailto:todd@cloudera.com]=20
Gesendet: Dienstag, 15. Juni 2010 18:55
An: general@hadoop.apache.org
Betreff: Re: HDFS file append

On Tue, Jun 15, 2010 at 9:53 AM, Dhruba Borthakur <dhruba@gmail.com> =
wrote:

> You can start downloading and using the code form the =
hadoop-0.20-append.
>
>
Though it's still missing some patches, so I'd recommend SVN checkouts =
and
watching the branch :)

-Todd


> thanks,
> dhruba
>
> On Tue, Jun 15, 2010 at 8:24 AM, Todd Lipcon <todd@cloudera.com> =
wrote:
>
> > On Tue, Jun 15, 2010 at 5:31 AM, Jan St=F6cker =
<Jan.Stoecker@q2web.de>
> > wrote:
> >
> > > Hello,
> > >
> > >
> > >
> > > in the hdfs-default.xml, I found the warning that the property
> > >
> > > "dfs.support.append" is false by default, because of the "append
> > >
> > > code" containing bugs.
> > >
> > > For our test case (which is not very complicated),  I tried it out
> > >
> > > nevertheless (in a C program using libhdfs), and I did not
> > >
> > > encounter any problems yet. The only thing I noticed is that the
> > >
> > > modification date of the file to which I append is not changed.
> > >
> > >
> > >
> > > Could someone indicate which other bugs may occur? And in
> > >
> > > which cases? Because the append could be quite useful for me,
> > >
> > > and I would like to know the risk.
> > >
> >
> > Mostly in failure handling cases, but not entirely. The result is =
usually
> > truncated files, sometimes truncated as if your append didn't =
happen,
> > sometimes the entire last block going missing. Look for the =
0.20-append
> fix
> > version on JIRA for a more significantly list.
> >
> > I'd recommend waiting a couple weeks and using the =
hadoop-0.20-append
> > branch
> > which will contain fixes for many of these bugs.
> >
> > -Todd
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
>
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
>



--=20
Todd Lipcon
Software Engineer, Cloudera

From general-return-1648-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 16 17:05:29 2010
Return-Path: <general-return-1648-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 71532 invoked from network); 16 Jun 2010 12:10:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 12:10:43 -0000
Received: (qmail 54516 invoked by uid 500); 16 Jun 2010 12:10:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54000 invoked by uid 500); 16 Jun 2010 12:10:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53990 invoked by uid 99); 16 Jun 2010 12:10:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 12:10:38 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of karan_jindal@students.iiit.ac.in designates 121.242.23.201 as permitted sender)
Received: from [121.242.23.201] (HELO students.iiit.ac.in) (121.242.23.201)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 12:10:30 +0000
MailScanner-NULL-Check: 1277295005.80572@9fjgOk50klZ+5xQFbQScFg
Received: from students.iiit.ac.in (localhost.localdomain [127.0.0.1])
	by students.iiit.ac.in (8.13.8/8.13.8) with ESMTP id o5GCA5IV001366
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Wed, 16 Jun 2010 17:40:05 +0530
Received: (from apache@localhost)
	by students.iiit.ac.in (8.13.8/8.14.1/Submit) id o5GCA5Sp001349;
	Wed, 16 Jun 2010 17:40:05 +0530
X-Authentication-Warning: students.iiit.ac.in: apache set sender to karan_jindal@localhost using -f
Received: from 125.16.17.151 (proxying for unknown)
        (SquirrelMail authenticated user karan_jindal)
        by students.iiit.ac.in with HTTP;
        Wed, 16 Jun 2010 17:40:05 +0530 (IST)
Message-ID: <46220.125.16.17.151.1276690205.squirrel@students.iiit.ac.in>
Date: Wed, 16 Jun 2010 17:40:05 +0530 (IST)
Subject: How many records will be passed to a map function??
From: "Karan Jindal" <karan_jindal@students.iiit.ac.in>
To: general@hadoop.apache.org
User-Agent: SquirrelMail/1.4.8-4.el5
MIME-Version: 1.0
Content-Type: text/plain;charset=iso-8859-1
Content-Transfer-Encoding: 8bit
X-Priority: 3 (Normal)
Importance: Normal
X-Spam-Level: *
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on students.iiit.ac.in
X-yoursite-MailScanner-Information: Please contact the IIIT Server Room  for more information
X-MailScanner-ID: o5GCA5IV001366
X-yoursite-MailScanner: Found to be clean
X-yoursite-MailScanner-From: karan_jindal@students.iiit.ac.in
X-Virus-Checked: Checked by ClamAV on apache.org
X-Old-Spam-Status: No, score=1.9 required=5.0 tests=ALL_TRUSTED,FH_DATE_PAST_20XX
	autolearn=no version=3.2.5, No


Hi all,

Given a scenario in which a input file contains total 1000 records (record
in a line) of total size 12k and I set number of map tasks to 2.
How many records will be passed to each map task? Is it the equal
distribution?

InputFormat = Text
Block size  = default block of hdfs

Hoping for a reply..

Regards
Karan

-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.


From general-return-1649-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 16 17:15:19 2010
Return-Path: <general-return-1649-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6456 invoked from network); 16 Jun 2010 15:29:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 15:29:38 -0000
Received: (qmail 25963 invoked by uid 500); 16 Jun 2010 15:29:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25902 invoked by uid 500); 16 Jun 2010 15:29:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25894 invoked by uid 99); 16 Jun 2010 15:29:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 15:29:36 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jonathan.seidman@gmail.com designates 74.125.83.48 as permitted sender)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 15:29:29 +0000
Received: by gwb17 with SMTP id 17so5771631gwb.35
        for <general@hadoop.apache.org>; Wed, 16 Jun 2010 08:29:08 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=CNlk4BotftuUystZUkl7D6xNWqjydvWNuSrWQ7b9OBY=;
        b=cjd5Y3pdUQXoTp+fcxLgphJuSMR4o9tHxSMVZmjPgtg/tDm4vO/lOFoRcj2uRAovHm
         4Z1SkHPaol+oiWyRyWmRdGgdLHkv7yfS9ynRMotz0DRhrJrDFHO0LZKs3NFj7+8hP5sk
         WtVgbNZZLIfJ+TDyTG82VDc2Bo6uc5p0OI+6Q=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=k2nWtgNdzq1G9RXqOa/Ovecaw+kCXT12/DATbUObMm+TOtjXujlTus5EfdC2DIXbZd
         k6W25kjMKAqzr4vlijcZCq+9Co1wkslDuQe00Vq/Gs8tFiZjyHqYCmeBtvC0v7XqT9pQ
         RiIxb0y8MHu2F2v81YOVHoXUW1bAFsUCFk/es=
MIME-Version: 1.0
Received: by 10.101.10.8 with SMTP id n8mr7471363ani.30.1276702148605; Wed, 16 
	Jun 2010 08:29:08 -0700 (PDT)
Received: by 10.100.46.17 with HTTP; Wed, 16 Jun 2010 08:29:06 -0700 (PDT)
Date: Wed, 16 Jun 2010 10:29:06 -0500
Message-ID: <AANLkTimFH8vFrfwQgFXZcpBijv5ExMuMnyCaYUNuKXqw@mail.gmail.com>
Subject: [ANNOUNCE] Chicago area Hadoop User Group Meeting - June 22nd
From: Jonathan Seidman <jonathan.seidman@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0050450293580d7add0489276249
X-Virus-Checked: Checked by ClamAV on apache.org

--0050450293580d7add0489276249
Content-Type: text/plain; charset=ISO-8859-1

The next meeting of the Chicago area Hadoop User Group will be help on
Tuesday June 22nd from 6:00-8:00PM. The meeting will be held at  Orbitz
Worldwide headquarters located in the Citigroup building at 500 West Madison
in downtown Chicago. The agenda will include a talk on Hadoop and Hive usage
at Orbitz, and an introduction to Sector/Sphere, a high performance DFS and
parallel processing framework.

Full details and RSVP here:
http://www.meetup.com/Chicago-area-Hadoop-User-Group-CHUG/

Thanks, and hope to see you there!

Jonathan

--0050450293580d7add0489276249--

From general-return-1650-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 16 17:27:37 2010
Return-Path: <general-return-1650-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58803 invoked from network); 16 Jun 2010 17:27:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 17:27:37 -0000
Received: (qmail 44039 invoked by uid 500); 16 Jun 2010 16:59:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 43575 invoked by uid 500); 16 Jun 2010 16:59:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 43469 invoked by uid 99); 16 Jun 2010 16:59:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 16:59:35 +0000
X-ASF-Spam-Status: No, hits=0.2 required=10.0
	tests=AWL,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 16:59:28 +0000
Received: from SNV-EXBH01.ds.corp.yahoo.com (snv-exbh01.ds.corp.yahoo.com [207.126.227.249])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o5GGwiBM027895
	for <general@hadoop.apache.org>; Wed, 16 Jun 2010 09:58:45 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:message-id:
	thread-topic:thread-index:mime-version:content-type:
	content-transfer-encoding:return-path:x-originalarrivaltime;
	b=XX7FH1g58TdJutE8PLliHBR7P6l84c8IwZ38gAlAsIYMxhB2LTh7dWdtiiJJTX5I
Received: from SNV-EXVS06.ds.corp.yahoo.com ([207.126.227.234]) by SNV-EXBH01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Wed, 16 Jun 2010 09:58:44 -0700
Received: from 10.72.111.153 ([10.72.111.153]) by SNV-EXVS06.ds.corp.yahoo.com ([207.126.227.82]) via Exchange Front-End Server snv-webmail.corp.yahoo.com ([207.126.227.59]) with Microsoft Exchange Server HTTP-DAV ;
 Wed, 16 Jun 2010 16:58:05 +0000
User-Agent: Microsoft-Entourage/12.25.0.100505
Date: Wed, 16 Jun 2010 09:58:05 -0700
Subject: [VOTE] Move Chukwa to incubator
From: Eric Yang <eyang@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Message-ID: <C83E50AD.908B%eyang@yahoo-inc.com>
Thread-Topic: [VOTE] Move Chukwa to incubator
Thread-Index: AcsNdRZ/YckvT2pHkE+yp3YT7q35yA==
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-OriginalArrivalTime: 16 Jun 2010 16:58:44.0859 (UTC) FILETIME=[2E41CCB0:01CB0D75]

Please vote as to whether you think Chukwa should move to Apache incubator.

The proposal is posted at:

http://wiki.apache.org/incubator/ChukwaProposal

Thanks

Regards,
Eric


From general-return-1651-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 16 17:36:29 2010
Return-Path: <general-return-1651-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 64828 invoked from network); 16 Jun 2010 17:36:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 17:36:28 -0000
Received: (qmail 5230 invoked by uid 500); 16 Jun 2010 17:36:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5156 invoked by uid 500); 16 Jun 2010 17:36:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5148 invoked by uid 99); 16 Jun 2010 17:36:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 17:36:26 +0000
X-ASF-Spam-Status: No, hits=0.4 required=10.0
	tests=AWL,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.27.212] (HELO qmta14.emeryville.ca.mail.comcast.net) (76.96.27.212)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 17:36:18 +0000
Received: from omta13.emeryville.ca.mail.comcast.net ([76.96.30.52])
	by qmta14.emeryville.ca.mail.comcast.net with comcast
	id Wc2s1e00317UAYkAEhbxFM; Wed, 16 Jun 2010 17:35:57 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta13.emeryville.ca.mail.comcast.net with comcast
	id Whbp1e0052SbwD58Zhbrwt; Wed, 16 Jun 2010 17:35:55 +0000
Message-Id: <46916273-1545-4D94-AC8F-93E97EC643B6@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <C83E50AD.908B%eyang@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Move Chukwa to incubator
Date: Wed, 16 Jun 2010 10:35:49 -0700
References: <C83E50AD.908B%eyang@yahoo-inc.com>
X-Mailer: Apple Mail (2.936)


On Jun 16, 2010, at 9:58 AM, Eric Yang wrote:

> Please vote as to whether you think Chukwa should move to Apache  
> incubator.
>
> The proposal is posted at:
>
> http://wiki.apache.org/incubator/ChukwaProposal

Since the goal of Chukwa's incubation is to make it a TLP, the sponsor  
needs to be the board instead of Hadoop. In the background and source  
submission plan you should mention that it is currently a subproject  
of Hadoop. Other than that, it looks good.

-- Owen

From general-return-1652-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 16 17:57:38 2010
Return-Path: <general-return-1652-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76355 invoked from network); 16 Jun 2010 17:57:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 17:57:38 -0000
Received: (qmail 33304 invoked by uid 500); 16 Jun 2010 17:57:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33237 invoked by uid 500); 16 Jun 2010 17:57:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33229 invoked by uid 99); 16 Jun 2010 17:57:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 17:57:36 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.30.24] (HELO qmta02.emeryville.ca.mail.comcast.net) (76.96.30.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 17:57:26 +0000
Received: from omta18.emeryville.ca.mail.comcast.net ([76.96.30.74])
	by qmta02.emeryville.ca.mail.comcast.net with comcast
	id Wdd51e0021bwxycA2hx57m; Wed, 16 Jun 2010 17:57:05 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta18.emeryville.ca.mail.comcast.net with comcast
	id Whww1e0052SbwD58ehwyFA; Wed, 16 Jun 2010 17:57:02 +0000
Message-Id: <9BF08853-8D72-4A4B-95EA-36BAC20BE2BE@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Sanjay Radia joins the Hadoop PMC
Date: Wed, 16 Jun 2010 10:56:56 -0700
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

The Hadoop PMC has voted to add Sanjay Radia to itself.  
Congratulations, Sanjay! Thanks for all of the hard work on Hadoop.

-- Owen

From general-return-1653-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 16 18:20:36 2010
Return-Path: <general-return-1653-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87760 invoked from network); 16 Jun 2010 18:20:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 18:20:36 -0000
Received: (qmail 72158 invoked by uid 500); 16 Jun 2010 18:20:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72096 invoked by uid 500); 16 Jun 2010 18:20:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72088 invoked by uid 99); 16 Jun 2010 18:20:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 18:20:34 +0000
X-ASF-Spam-Status: No, hits=-175.5 required=10.0
	tests=AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 16 Jun 2010 18:20:33 +0000
Received: (qmail 87687 invoked by uid 99); 16 Jun 2010 18:20:13 -0000
Received: from localhost.apache.org (HELO [192.168.42.205]) (127.0.0.1)
  (smtp-auth username phunt, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 18:20:13 +0000
Message-ID: <4C1915DC.9000506@apache.org>
Date: Wed, 16 Jun 2010 11:20:12 -0700
From: Patrick Hunt <phunt@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.1.9) Gecko/20100317 Lightning/1.0b1 Thunderbird/3.0.4
MIME-Version: 1.0
To: general@hadoop.apache.org
CC: Owen O'Malley <omalley@apache.org>
Subject: Re: Sanjay Radia joins the Hadoop PMC
References: <9BF08853-8D72-4A4B-95EA-36BAC20BE2BE@apache.org>
In-Reply-To: <9BF08853-8D72-4A4B-95EA-36BAC20BE2BE@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Congratulations Sanjay!

Patrick

On 06/16/2010 10:56 AM, Owen O'Malley wrote:
> The Hadoop PMC has voted to add Sanjay Radia to itself. Congratulations,
> Sanjay! Thanks for all of the hard work on Hadoop.
>
> -- Owen

From general-return-1654-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 16 18:27:50 2010
Return-Path: <general-return-1654-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95969 invoked from network); 16 Jun 2010 18:27:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 18:27:49 -0000
Received: (qmail 86194 invoked by uid 500); 16 Jun 2010 18:27:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85786 invoked by uid 500); 16 Jun 2010 18:27:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85773 invoked by uid 99); 16 Jun 2010 18:27:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 18:27:44 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of CGartin@taos.com designates 12.238.9.132 as permitted sender)
Received: from [12.238.9.132] (HELO relay.taos.com) (12.238.9.132)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 18:27:36 +0000
X-IronPort-AV: E=Sophos;i="4.53,427,1272870000"; 
   d="scan'208";a="27499947"
Received: from unknown (HELO mantaray.taos.local) ([172.20.100.18])
  by relay.taos.com with ESMTP; 16 Jun 2010 11:27:34 -0700
x-mimeole: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: RE: Sanjay Radia joins the Hadoop PMC
Date: Wed, 16 Jun 2010 11:28:11 -0700
Message-ID: <853C65D5E3860D40BB05F5C627C796EE01A83420@mantaray.taos.local>
In-Reply-To: <4C1915DC.9000506@apache.org>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Sanjay Radia joins the Hadoop PMC
Thread-Index: AcsNgMDBWplPi1NCSnON6xJrnCG6hAAANc7g
References: <9BF08853-8D72-4A4B-95EA-36BAC20BE2BE@apache.org> <4C1915DC.9000506@apache.org>
From: "Courtney Gartin" <CGartin@taos.com>
To: <general@hadoop.apache.org>
Cc: "Owen O'Malley" <omalley@apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

Congratulations Sanjay!!!

Courtney

-----Original Message-----
From: Patrick Hunt [mailto:phunt@apache.org]=20
Sent: Wednesday, June 16, 2010 11:20 AM
To: general@hadoop.apache.org
Cc: Owen O'Malley
Subject: Re: Sanjay Radia joins the Hadoop PMC

Congratulations Sanjay!

Patrick

On 06/16/2010 10:56 AM, Owen O'Malley wrote:
> The Hadoop PMC has voted to add Sanjay Radia to itself.
Congratulations,
> Sanjay! Thanks for all of the hard work on Hadoop.
>
> -- Owen

From general-return-1655-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 16 19:39:54 2010
Return-Path: <general-return-1655-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34556 invoked from network); 16 Jun 2010 19:39:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 19:39:53 -0000
Received: (qmail 99823 invoked by uid 500); 16 Jun 2010 19:39:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99759 invoked by uid 500); 16 Jun 2010 19:39:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 99751 invoked by uid 99); 16 Jun 2010 19:39:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 19:39:51 +0000
X-ASF-Spam-Status: No, hits=1.8 required=10.0
	tests=AWL,HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [137.100.120.43] (HELO mnbm01-relay1.mnb.gd-ais.com) (137.100.120.43)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 19:39:43 +0000
Received: from ([160.207.224.15])
	by mnbm01-relay1.mnb.gd-ais.com with SMTP  id 5202712.272194510;
	Wed, 16 Jun 2010 14:34:57 -0500
Received: from MAPF01-MAIL01.ad.gd-ais.com ([166.16.220.104]) by mnbm01-fes01.ad.gd-ais.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Wed, 16 Jun 2010 14:34:57 -0500
X-MimeOLE: Produced By Microsoft Exchange V6.5
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01CB0D8A.FFCDF70A"
x-cr-hashedpuzzle: cCE= Apj5 BA4z BXpu BYqw BfIP EP3V GBlL GRxJ HnSn HoM8 H7aA KXN9 Kx1B K7YT LNRM;1;ZwBlAG4AZQByAGEAbABAAGgAYQBkAG8AbwBwAC4AYQBwAGEAYwBoAGUALgBvAHIAZwA=;Sosha1_v1;7;{BBDEA61D-7330-4029-B108-D303CD1B8FE1};agBhAG0AZQBzAC4AawBpAGwAYgByAGkAZABlAEAAZwBkAC0AYQBpAHMALgBjAG8AbQA=;Wed, 16 Jun 2010 19:34:54 GMT;SABhAGQAbwBvAHAAIABpAG4AIABXAGUAYgAgAGEAcABwAGwAaQBjAGEAdABpAG8AbgBzAA==
x-cr-puzzleid: {BBDEA61D-7330-4029-B108-D303CD1B8FE1}
Content-class: urn:content-classes:message
Subject: Hadoop in Web applications
Date: Wed, 16 Jun 2010 15:34:54 -0400
Message-ID: <E0C1686B8F45434DA0E0A904672EA1A8065E72F0@MAPF01-MAIL01.ad.gd-ais.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Hadoop in Web applications
Thread-Index: AcsNiv7nHluf/7qmRsCjw9taX/TI/w==
From: "Kilbride, James P." <James.Kilbride@gd-ais.com>
To: <general@hadoop.apache.org>
X-OriginalArrivalTime: 16 Jun 2010 19:34:57.0056 (UTC) FILETIME=[00859200:01CB0D8B]

------_=_NextPart_001_01CB0D8A.FFCDF70A
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

All,

What's the preferred way to use hadoop within java code running as a web
service in something like tomcat? Would you edit the tomcat
configuration to have access to the hadoop directory? Or simply include
the hadoop jar and have the web service set some configuration somehow
before calling submit(assuming the webservice implements the Tool
interface)?

James Kilbride

------_=_NextPart_001_01CB0D8A.FFCDF70A--

From general-return-1656-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 16 19:42:19 2010
Return-Path: <general-return-1656-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35757 invoked from network); 16 Jun 2010 19:42:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 19:42:19 -0000
Received: (qmail 5049 invoked by uid 500); 16 Jun 2010 19:42:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4993 invoked by uid 500); 16 Jun 2010 19:42:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4985 invoked by uid 99); 16 Jun 2010 19:42:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 19:42:17 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 19:42:09 +0000
Received: from SNV-EXBH01.ds.corp.yahoo.com (snv-exbh01.ds.corp.yahoo.com [207.126.227.249])
	by mrout2.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o5GJeXUx097765
	for <general@hadoop.apache.org>; Wed, 16 Jun 2010 12:40:33 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:return-path:x-originalarrivaltime;
	b=G97zrBLaOTzIUAAzaOsaD2AKQRgl2VfJ9ZEpzCxoNYyUeYyP959fEsfMiEncbWWD
Received: from SNV-EXVS06.ds.corp.yahoo.com ([207.126.227.234]) by SNV-EXBH01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Wed, 16 Jun 2010 12:40:33 -0700
Received: from 10.72.111.153 ([10.72.111.153]) by SNV-EXVS06.ds.corp.yahoo.com ([207.126.227.82]) via Exchange Front-End Server snv-webmail.corp.yahoo.com ([207.126.227.59]) with Microsoft Exchange Server HTTP-DAV ;
 Wed, 16 Jun 2010 19:40:22 +0000
User-Agent: Microsoft-Entourage/12.25.0.100505
Date: Wed, 16 Jun 2010 12:40:22 -0700
Subject: Re: [VOTE] Move Chukwa to incubator
From: Eric Yang <eyang@yahoo-inc.com>
To: <general@hadoop.apache.org>
Message-ID: <C83E76B6.90A0%eyang@yahoo-inc.com>
Thread-Topic: [VOTE] Move Chukwa to incubator
Thread-Index: AcsNi8I01RgleKehAEOydxb5Q+cRRQ==
In-Reply-To: <46916273-1545-4D94-AC8F-93E97EC643B6@apache.org>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-OriginalArrivalTime: 16 Jun 2010 19:40:33.0307 (UTC) FILETIME=[C8F166B0:01CB0D8B]
X-Virus-Checked: Checked by ClamAV on apache.org

I have updated the proposal per Owen's request.  I put Incubator in
Sponsoring Entity like other proposals.

Regards,
Eric


On 6/16/10 10:35 AM, "Owen O'Malley" <omalley@apache.org> wrote:

> 
> On Jun 16, 2010, at 9:58 AM, Eric Yang wrote:
> 
>> Please vote as to whether you think Chukwa should move to Apache
>> incubator.
>> 
>> The proposal is posted at:
>> 
>> http://wiki.apache.org/incubator/ChukwaProposal
> 
> Since the goal of Chukwa's incubation is to make it a TLP, the sponsor
> needs to be the board instead of Hadoop. In the background and source
> submission plan you should mention that it is currently a subproject
> of Hadoop. Other than that, it looks good.
> 
> -- Owen


From general-return-1658-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 17 12:01:10 2010
Return-Path: <general-return-1658-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 18084 invoked from network); 17 Jun 2010 12:01:10 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jun 2010 12:01:10 -0000
Received: (qmail 60884 invoked by uid 500); 17 Jun 2010 10:14:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60347 invoked by uid 500); 17 Jun 2010 10:14:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60334 invoked by uid 99); 17 Jun 2010 10:14:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 10:14:24 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 10:14:15 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 62AA91BA4DB
	for <general@hadoop.apache.org>; Thu, 17 Jun 2010 11:13:54 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id VhNyKLbeNadj for <general@hadoop.apache.org>;
	Thu, 17 Jun 2010 11:13:54 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by colossus.hpl.hp.com (Postfix) with ESMTPS id D2F911BA313
	for <general@hadoop.apache.org>; Thu, 17 Jun 2010 11:13:53 +0100 (BST)
MailScanner-NULL-Check: 1277374421.43511@qGuMqkM4p/JA3Dh1QHeisg
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o5HADe9r004039
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Thu, 17 Jun 2010 11:13:41 +0100 (BST)
Message-ID: <4C19F556.2060207@apache.org>
Date: Thu, 17 Jun 2010 11:13:42 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100228)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Hadoop in Web applications
References: <E0C1686B8F45434DA0E0A904672EA1A8065E72F0@MAPF01-MAIL01.ad.gd-ais.com>
In-Reply-To: <E0C1686B8F45434DA0E0A904672EA1A8065E72F0@MAPF01-MAIL01.ad.gd-ais.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o5HADe9r004039
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

Kilbride, James P. wrote:
> All,
> 
> What's the preferred way to use hadoop within java code running as a web
> service in something like tomcat? Would you edit the tomcat
> configuration to have access to the hadoop directory? Or simply include
> the hadoop jar and have the web service set some configuration somehow
> before calling submit(assuming the webservice implements the Tool
> interface)?

1. run hadoop services outside of the web server process,
2. bring up the Hadoop client libraries passing in a jobconf that binds 
to the filesystem/JT in the cluster
3. talk to the things remotely as you desire

there's no need to do the tool interface, just talk using the DfsClient 
and Filesystem client APIs.


e.g to submit work, call JobClient.submitJob(jobConf) on a client you've 
created, to work with a dfs given its URI and a conf file:

             FileSystem dfs = FileSystem.newInstance(uri, conf);
             dfs.initialize(uri, conf);

From general-return-1659-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 17 12:47:56 2010
Return-Path: <general-return-1659-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94637 invoked from network); 17 Jun 2010 12:47:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jun 2010 12:47:55 -0000
Received: (qmail 53773 invoked by uid 500); 17 Jun 2010 12:47:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53289 invoked by uid 500); 17 Jun 2010 12:47:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53280 invoked by uid 99); 17 Jun 2010 12:47:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 12:47:50 +0000
X-ASF-Spam-Status: No, hits=-1508.1 required=10.0
	tests=ALL_TRUSTED,AWL,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 17 Jun 2010 12:47:49 +0000
Received: (qmail 94407 invoked by uid 99); 17 Jun 2010 12:47:28 -0000
Received: from localhost.apache.org (HELO mail-iw0-f176.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 12:47:28 +0000
Received: by iwn4 with SMTP id 4so3821885iwn.35
        for <general@hadoop.apache.org>; Thu, 17 Jun 2010 05:47:28 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.42.0.68 with SMTP id 4mr3487015icb.98.1276769020753; Thu, 17 
	Jun 2010 03:03:40 -0700 (PDT)
Received: by 10.231.144.11 with HTTP; Thu, 17 Jun 2010 03:03:40 -0700 (PDT)
In-Reply-To: <C83E50AD.908B%eyang@yahoo-inc.com>
References: <C83E50AD.908B%eyang@yahoo-inc.com>
Date: Thu, 17 Jun 2010 03:03:40 -0700
Message-ID: <AANLkTin6kkfgFbb8mmmqOFr5nuLkItGE0pJSdkZuXjBi@mail.gmail.com>
Subject: Re: [VOTE] Move Chukwa to incubator
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

+1 -C

On Wed, Jun 16, 2010 at 9:58 AM, Eric Yang <eyang@yahoo-inc.com> wrote:
> Please vote as to whether you think Chukwa should move to Apache incubator.
>
> The proposal is posted at:
>
> http://wiki.apache.org/incubator/ChukwaProposal
>
> Thanks
>
> Regards,
> Eric
>
>

From general-return-1657-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 17 12:59:28 2010
Return-Path: <general-return-1657-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 7374 invoked from network); 17 Jun 2010 12:59:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jun 2010 12:59:28 -0000
Received: (qmail 76540 invoked by uid 500); 17 Jun 2010 05:52:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76200 invoked by uid 500); 17 Jun 2010 05:52:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76189 invoked by uid 99); 17 Jun 2010 05:52:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 05:52:43 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 05:52:36 +0000
Received: from EGL-EX07CAS01.ds.corp.yahoo.com (egl-ex07cas01.eglbp.corp.yahoo.com [203.83.248.208])
	by mrout2.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o5H5pXO2064761
	for <general@hadoop.apache.org>; Wed, 16 Jun 2010 22:51:34 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:in-reply-to:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=pgNY17C8RaB38fjiBxGAnnc4iKerxtWMwqsDondGTke9SivSPICHcJASULDx1/Aw
Received: from EGL-EX07VS02.ds.corp.yahoo.com ([203.83.248.206]) by
 EGL-EX07CAS01.ds.corp.yahoo.com ([203.83.248.215]) with mapi; Thu, 17 Jun
 2010 11:21:33 +0530
From: Venkatesh S <svenkat@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 17 Jun 2010 11:21:06 +0530
Subject: Re: Hadoop in Web applications
Thread-Topic: Hadoop in Web applications
Thread-Index: AcsNiv7nHluf/7qmRsCjw9taX/TI/wAVhTRM
Message-ID: <C83FB5A2.39587%svenkat@yahoo-inc.com>
In-Reply-To: <E0C1686B8F45434DA0E0A904672EA1A8065E72F0@MAPF01-MAIL01.ad.gd-ais.com>
Accept-Language: en, en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en, en-US
Content-Type: multipart/alternative;
	boundary="_000_C83FB5A239587svenkatyahooinccom_"
MIME-Version: 1.0

--_000_C83FB5A239587svenkatyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Latter should work well.

Venkatesh


On 6/17/10 1:04 AM, "Kilbride, James P." <James.Kilbride@gd-ais.com> wrote:

All,

What's the preferred way to use hadoop within java code running as a web
service in something like tomcat? Would you edit the tomcat
configuration to have access to the hadoop directory? Or simply include
the hadoop jar and have the web service set some configuration somehow
before calling submit(assuming the webservice implements the Tool
interface)?

James Kilbride


--_000_C83FB5A239587svenkatyahooinccom_--

From general-return-1660-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 18 11:35:59 2010
Return-Path: <general-return-1660-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83440 invoked from network); 18 Jun 2010 11:35:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Jun 2010 11:35:59 -0000
Received: (qmail 81297 invoked by uid 500); 18 Jun 2010 11:35:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80367 invoked by uid 500); 18 Jun 2010 11:35:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80351 invoked by uid 99); 18 Jun 2010 11:35:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 11:35:52 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of bernd.fondermann@googlemail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 11:35:46 +0000
Received: by wyb29 with SMTP id 29so981983wyb.35
        for <general@hadoop.apache.org>; Fri, 18 Jun 2010 04:35:26 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=fMqbQ4/Zh94iQDCDPZOzKbsJKxWJgfe8LQQBeA/Mf2k=;
        b=Nf67e9G7z309cWJKYv3QNHam7rexl/nYLpSSTpaM+J9Mfoc4z1BJwH7e3LWYx2YEAX
         jk56RadvBNU4hlmuXh8NbrDDKVihdAq3SQBrzRjeuFXD4Jcb7ounyi6+xZZNL4N0XuEU
         dHodCQ+jNhHEy+2pyhp8g6MjGD5k5z2i7aN9E=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=We4gf8XS1Ses9n2BfqhZDgBthxeZrie2E1whYGJ/AY0SfQpKmevvyPuvD/LUtf+c2R
         5XO3CgSd0/aSzR8oS/fz6UPdB8xWXbNXMkLGehMMm9+Fnp00Ac+3jSDAxW2fi0CUW98X
         GxRQcSP4yNzJl0wyXNKmZyuNh0/FrvL9HQua0=
MIME-Version: 1.0
Received: by 10.216.88.6 with SMTP id z6mr686967wee.79.1276860925356; Fri, 18 
	Jun 2010 04:35:25 -0700 (PDT)
Received: by 10.216.156.6 with HTTP; Fri, 18 Jun 2010 04:35:25 -0700 (PDT)
In-Reply-To: <46916273-1545-4D94-AC8F-93E97EC643B6@apache.org>
References: <C83E50AD.908B%eyang@yahoo-inc.com>
	<46916273-1545-4D94-AC8F-93E97EC643B6@apache.org>
Date: Fri, 18 Jun 2010 13:35:25 +0200
Message-ID: <AANLkTine_-4cX57fTM6DMkU4y-Wh2adAo8lKHH6lJi2E@mail.gmail.com>
Subject: Re: [VOTE] Move Chukwa to incubator
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Wed, Jun 16, 2010 at 19:35, Owen O'Malley <omalley@apache.org> wrote:
>
> On Jun 16, 2010, at 9:58 AM, Eric Yang wrote:
>
>> Please vote as to whether you think Chukwa should move to Apache
>> incubator.
>>
>> The proposal is posted at:
>>
>> http://wiki.apache.org/incubator/ChukwaProposal
>
> Since the goal of Chukwa's incubation is to make it a TLP, the sponsor needs
> to be the board instead of Hadoop.

Although the incubation docs say so, this has rarely - if ever - been executed.
None of the current projects has the board as a sponsor, although most
of them can be expected to become TLPs.
I think, Hadoop should be the sponsor. This would be in line with
previous incubations.
But in the end, what's important are the mentors.

  Bernd

From general-return-1661-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 18 17:45:22 2010
Return-Path: <general-return-1661-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38419 invoked from network); 18 Jun 2010 17:45:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Jun 2010 17:45:22 -0000
Received: (qmail 77198 invoked by uid 500); 18 Jun 2010 17:45:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77102 invoked by uid 500); 18 Jun 2010 17:45:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77094 invoked by uid 99); 18 Jun 2010 17:45:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 17:45:20 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 17:45:14 +0000
Received: from [192.168.0.195] (snvvpn4-10-72-168-c149.hq.corp.yahoo.com [10.72.168.149])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o5IHhKBw032027
	for <general@hadoop.apache.org>; Fri, 18 Jun 2010 10:43:20 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=ZxlZRGckauzUcPM85mtzlxje9Kp2og9JZnZJvYB0m5E78ZWAQ3YggmJ1rGw9E9Hv
Message-Id: <7CF105CA-6D75-472B-A0F3-63D66513E575@yahoo-inc.com>
From: Alan Gates <gates@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <C83E50AD.908B%eyang@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Move Chukwa to incubator
Date: Fri, 18 Jun 2010 10:43:20 -0700
References: <C83E50AD.908B%eyang@yahoo-inc.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

+1.

Alan.

On Jun 16, 2010, at 9:58 AM, Eric Yang wrote:

> Please vote as to whether you think Chukwa should move to Apache  
> incubator.
>
> The proposal is posted at:
>
> http://wiki.apache.org/incubator/ChukwaProposal
>
> Thanks
>
> Regards,
> Eric
>


From general-return-1662-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 18 17:52:53 2010
Return-Path: <general-return-1662-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 40207 invoked from network); 18 Jun 2010 17:52:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Jun 2010 17:52:53 -0000
Received: (qmail 93838 invoked by uid 500); 18 Jun 2010 17:52:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93788 invoked by uid 500); 18 Jun 2010 17:52:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93768 invoked by uid 99); 18 Jun 2010 17:52:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 17:52:51 +0000
X-ASF-Spam-Status: No, hits=0.4 required=10.0
	tests=AWL,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.62.16] (HELO qmta01.westchester.pa.mail.comcast.net) (76.96.62.16)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 17:52:42 +0000
Received: from omta10.westchester.pa.mail.comcast.net ([76.96.62.28])
	by qmta01.westchester.pa.mail.comcast.net with comcast
	id XTuj1e0080cZkys51VsNom; Fri, 18 Jun 2010 17:52:23 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta10.westchester.pa.mail.comcast.net with comcast
	id XVsD1e0032SbwD53WVsFGQ; Fri, 18 Jun 2010 17:52:20 +0000
Message-Id: <CE4B61E4-88B1-4B1C-89A1-F489E0D8DBE8@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <C83E50AD.908B%eyang@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Move Chukwa to incubator
Date: Fri, 18 Jun 2010 10:52:11 -0700
References: <C83E50AD.908B%eyang@yahoo-inc.com>
X-Mailer: Apple Mail (2.936)


On Jun 16, 2010, at 9:58 AM, Eric Yang wrote:

> Please vote as to whether you think Chukwa should move to Apache  
> incubator.

+1

From general-return-1663-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 18 18:01:06 2010
Return-Path: <general-return-1663-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 41854 invoked from network); 18 Jun 2010 18:01:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Jun 2010 18:01:06 -0000
Received: (qmail 99537 invoked by uid 500); 18 Jun 2010 18:01:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99489 invoked by uid 500); 18 Jun 2010 18:01:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 99481 invoked by uid 99); 18 Jun 2010 18:01:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 18:01:04 +0000
X-ASF-Spam-Status: No, hits=0.6 required=10.0
	tests=AWL,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 18:00:59 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o5IHwY8w029914
	for <general@hadoop.apache.org>; Fri, 18 Jun 2010 10:58:34 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=hAd+Zm7cjBUMbJxQg6gsXBjfFEfzFUHN5AI7QtKtSaD43LTo1nvmemo3+gPLJMyZ
Message-Id: <A565C79F-7845-4966-80C5-E6355B457F08@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <C83E50AD.908B%eyang@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Move Chukwa to incubator
Date: Fri, 18 Jun 2010 10:58:34 -0700
References: <C83E50AD.908B%eyang@yahoo-inc.com>
X-Mailer: Apple Mail (2.936)

+1

Arun

On Jun 16, 2010, at 9:58 AM, Eric Yang wrote:

> Please vote as to whether you think Chukwa should move to Apache  
> incubator.
>
> The proposal is posted at:
>
> http://wiki.apache.org/incubator/ChukwaProposal
>
> Thanks
>
> Regards,
> Eric
>


From general-return-1664-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 18 20:38:38 2010
Return-Path: <general-return-1664-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91612 invoked from network); 18 Jun 2010 20:38:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Jun 2010 20:38:38 -0000
Received: (qmail 61035 invoked by uid 500); 18 Jun 2010 20:38:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60976 invoked by uid 500); 18 Jun 2010 20:38:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60968 invoked by uid 99); 18 Jun 2010 20:38:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 20:38:36 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 20:38:28 +0000
Received: from SNV-EXBH01.ds.corp.yahoo.com (snv-exbh01.ds.corp.yahoo.com [207.126.227.249])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o5IKbQNE052662
	for <general@hadoop.apache.org>; Fri, 18 Jun 2010 13:37:26 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:return-path:x-originalarrivaltime;
	b=H4gvmfwtckagyjWD1/fr9z5AX8C/sIA/1w1FhTWbdk5TS7Tvwe/nKlcXjbZ5YHHb
Received: from SNV-EXVS10.ds.corp.yahoo.com ([207.126.227.89]) by SNV-EXBH01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Fri, 18 Jun 2010 13:37:26 -0700
Received: from 172.21.148.167 ([172.21.148.167]) by SNV-EXVS10.ds.corp.yahoo.com ([207.126.227.103]) via Exchange Front-End Server snv-webmail.corp.yahoo.com ([207.126.227.59]) with Microsoft Exchange Server HTTP-DAV ;
 Fri, 18 Jun 2010 20:37:25 +0000
User-Agent: Microsoft-Entourage/12.25.0.100505
Date: Fri, 18 Jun 2010 13:37:25 -0700
Subject: Re: [VOTE] Move Chukwa to incubator
From: Amir Youssefi <amiry@yahoo-inc.com>
To: <general@hadoop.apache.org>
Message-ID: <C8412715.23A51%amiry@yahoo-inc.com>
Thread-Topic: [VOTE] Move Chukwa to incubator
Thread-Index: AcsPJg9LALb3jC8aDkGo8wXGbjhsJQ==
In-Reply-To: <CE4B61E4-88B1-4B1C-89A1-F489E0D8DBE8@apache.org>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-OriginalArrivalTime: 18 Jun 2010 20:37:26.0234 (UTC) FILETIME=[10083BA0:01CB0F26]
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Amir
 
> On Jun 16, 2010, at 9:58 AM, Eric Yang wrote:
> 
>> Please vote as to whether you think Chukwa should move to Apache
>> incubator.



From general-return-1665-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 18 20:49:04 2010
Return-Path: <general-return-1665-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93129 invoked from network); 18 Jun 2010 20:49:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Jun 2010 20:49:04 -0000
Received: (qmail 75226 invoked by uid 500); 18 Jun 2010 20:49:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75179 invoked by uid 500); 18 Jun 2010 20:49:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75171 invoked by uid 99); 18 Jun 2010 20:49:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 20:49:02 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 20:48:56 +0000
Received: by bwz6 with SMTP id 6so677150bwz.35
        for <general@hadoop.apache.org>; Fri, 18 Jun 2010 13:48:35 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.85.70 with SMTP id n6mr1235910bkl.189.1276894115461; Fri, 
	18 Jun 2010 13:48:35 -0700 (PDT)
Received: by 10.204.52.70 with HTTP; Fri, 18 Jun 2010 13:48:35 -0700 (PDT)
In-Reply-To: <C83E50AD.908B%eyang@yahoo-inc.com>
References: <C83E50AD.908B%eyang@yahoo-inc.com>
Date: Fri, 18 Jun 2010 13:48:35 -0700
Message-ID: <AANLkTimHgZwMmvu-e_ecf3BnxqWGZRoEUUpz7ZY2oUUr@mail.gmail.com>
Subject: Re: [VOTE] Move Chukwa to incubator
From: Tom White <tom@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Tom

On Wed, Jun 16, 2010 at 9:58 AM, Eric Yang <eyang@yahoo-inc.com> wrote:
> Please vote as to whether you think Chukwa should move to Apache incubator.
>
> The proposal is posted at:
>
> http://wiki.apache.org/incubator/ChukwaProposal
>
> Thanks
>
> Regards,
> Eric
>
>

From general-return-1666-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 18 21:42:58 2010
Return-Path: <general-return-1666-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6627 invoked from network); 18 Jun 2010 21:42:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Jun 2010 21:42:58 -0000
Received: (qmail 22531 invoked by uid 500); 18 Jun 2010 21:42:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22452 invoked by uid 500); 18 Jun 2010 21:42:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22444 invoked by uid 99); 18 Jun 2010 21:42:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 21:42:56 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 21:42:48 +0000
Received: from SNV-EXPF01.ds.corp.yahoo.com (snv-expf01.ds.corp.yahoo.com [207.126.227.250])
	by mrout1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o5ILg5YK006360
	for <general@hadoop.apache.org>; Fri, 18 Jun 2010 14:42:05 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:return-path:x-originalarrivaltime;
	b=uGiQwsHVEutiYpIyT3yf5pBBzFaFt4ypICegngRkPL1Ys04AtRLtUXGg0EfmOdEn
Received: from SNV-EXVS10.ds.corp.yahoo.com ([207.126.227.89]) by SNV-EXPF01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Fri, 18 Jun 2010 14:42:05 -0700
Received: from 10.72.105.112 ([10.72.105.112]) by SNV-EXVS10.ds.corp.yahoo.com ([207.126.227.103]) via Exchange Front-End Server snv-webmail.corp.yahoo.com ([207.126.227.60]) with Microsoft Exchange Server HTTP-DAV ;
 Fri, 18 Jun 2010 21:42:04 +0000
User-Agent: Microsoft-Entourage/12.25.0.100505
Date: Fri, 18 Jun 2010 14:42:03 -0700
Subject: Re: [VOTE] Move Chukwa to incubator
From: Amir Youssefi <amiry@yahoo-inc.com>
To: <general@hadoop.apache.org>
Message-ID: <C841363B.23A6E%amiry@yahoo-inc.com>
Thread-Topic: [VOTE] Move Chukwa to incubator
Thread-Index: AcsPLxbDESfQIfHnoEezdIyYlDSjKw==
In-Reply-To: <A565C79F-7845-4966-80C5-E6355B457F08@yahoo-inc.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-OriginalArrivalTime: 18 Jun 2010 21:42:05.0671 (UTC) FILETIME=[185B5770:01CB0F2F]
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Amir
 
> On Jun 16, 2010, at 9:58 AM, Eric Yang wrote:
> 
>> Please vote as to whether you think Chukwa should move to Apache
>> incubator.
>> 
>> The proposal is posted at:
>> 
>> http://wiki.apache.org/incubator/ChukwaProposal
>> 
>> Thanks
>> 
>> Regards,
>> Eric
>> 
> 


From general-return-1667-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 18 21:45:29 2010
Return-Path: <general-return-1667-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6940 invoked from network); 18 Jun 2010 21:45:29 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Jun 2010 21:45:29 -0000
Received: (qmail 23797 invoked by uid 500); 18 Jun 2010 21:45:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23721 invoked by uid 500); 18 Jun 2010 21:45:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23713 invoked by uid 99); 18 Jun 2010 21:45:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 21:45:27 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of babogorilla@gmail.com designates 209.85.211.199 as permitted sender)
Received: from [209.85.211.199] (HELO mail-yw0-f199.google.com) (209.85.211.199)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 21:45:21 +0000
Received: by ywh37 with SMTP id 37so1709948ywh.2
        for <general@hadoop.apache.org>; Fri, 18 Jun 2010 14:45:00 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:sender:received
         :in-reply-to:references:from:date:x-google-sender-auth:message-id
         :subject:to:content-type;
        bh=IzU1lydZjUvTip0uKp2IdGGy4WCFwymF6A9XOqRFifc=;
        b=OD72HuaMcwcEUZUXHRs3arOTNK5o4GTkPHzQ5l1lfey7uMBKnPYetGZDMpR6LWENXN
         NXhxcxG3/1Ix2N3qNfxzJA1TyXOkmr4Q8Q1cWzEtMfh4igkU1hpLotBR1PB62Vxg2dwH
         52am3WDM8yyvfyT77mRDmh0RilDkh7qSvBJYA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:from:date
         :x-google-sender-auth:message-id:subject:to:content-type;
        b=pykD2MRwndQI4mZIFUg1OW3dqB+SbNzzF4PyCylt4zH2Mi20eQ0BMwiPk+CVHU6sRQ
         2EFLjRbz2fsAc1FEeIYkOaUeykipSkS4DZEXNkTaUvoJfFp7148yfjdgcrfADLGMtqDu
         03LgmxhEjuwegdKDAXKw5yEen0aeCfB1PabBU=
Received: by 10.220.124.38 with SMTP id s38mr671429vcr.161.1276897500196; Fri, 
	18 Jun 2010 14:45:00 -0700 (PDT)
MIME-Version: 1.0
Sender: babogorilla@gmail.com
Received: by 10.220.86.129 with HTTP; Fri, 18 Jun 2010 14:44:40 -0700 (PDT)
In-Reply-To: <C841363B.23A6E%amiry@yahoo-inc.com>
References: <A565C79F-7845-4966-80C5-E6355B457F08@yahoo-inc.com> 
	<C841363B.23A6E%amiry@yahoo-inc.com>
From: Tim Sanchez <tsanchez01@gmail.com>
Date: Fri, 18 Jun 2010 18:44:40 -0300
X-Google-Sender-Auth: Enx7BL71J36vkyOntVhUVPQLzuo
Message-ID: <AANLkTikBr14E_hrnfWgRInxEbzYBHEGbyk4DYxFe7U-0@mail.gmail.com>
Subject: Re: [VOTE] Move Chukwa to incubator
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636d34455e9f02b048954ddc5
X-Virus-Checked: Checked by ClamAV on apache.org

--001636d34455e9f02b048954ddc5
Content-Type: text/plain; charset=ISO-8859-1

+1

On Fri, Jun 18, 2010 at 6:42 PM, Amir Youssefi <amiry@yahoo-inc.com> wrote:

> +1
>
> Amir
>
> > On Jun 16, 2010, at 9:58 AM, Eric Yang wrote:
> >
> >> Please vote as to whether you think Chukwa should move to Apache
> >> incubator.
> >>
> >> The proposal is posted at:
> >>
> >> http://wiki.apache.org/incubator/ChukwaProposal
> >>
> >> Thanks
> >>
> >> Regards,
> >> Eric
> >>
> >
>
>

--001636d34455e9f02b048954ddc5--

From general-return-1668-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 18 22:49:26 2010
Return-Path: <general-return-1668-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22045 invoked from network); 18 Jun 2010 22:49:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Jun 2010 22:49:25 -0000
Received: (qmail 76156 invoked by uid 500); 18 Jun 2010 22:49:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75863 invoked by uid 500); 18 Jun 2010 22:49:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 72035 invoked by uid 99); 16 Jun 2010 21:49:51 -0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=AWL,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
From: Steve Wooledge <Steve.Wooledge@asterdata.com>
To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>,
	"general@hadoop.apache.org" <general@hadoop.apache.org>,
	"core-user@hadoop.apache.org" <core-user@hadoop.apache.org>,
	"hive-user@hadoop.apache.org" <hive-user@hadoop.apache.org>, hbase-user
	<hbase-user@hadoop.apache.org>
Date: Wed, 16 Jun 2010 14:49:24 -0700
Subject: Hadoop Summit Unconference: BigDataCamp - June 28th
Thread-Topic: Hadoop Summit Unconference: BigDataCamp - June 28th
Thread-Index: AcrxX5vzBTWhLG+KR4ip/RciMdDUXgcPTd4g
Message-ID: <0B7A7A7AB36FC54D88B1B312F8209FEDFFB17591@pandora>
References: <AANLkTinLQ-7zITzjnPLbrtcJvAnvyp3Jt-MEffe4kxTr@mail.gmail.com>
In-Reply-To: <AANLkTinLQ-7zITzjnPLbrtcJvAnvyp3Jt-MEffe4kxTr@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hadoop Fans,

BigDataCamp is a free event the evening before Hadoop Summit. Great chance =
to network and grab some food/drinks:  www.bigdatacamp.org=20

It is an unconference for users of Hadoop and related technologies to excha=
nge ideas in a loosely distributed format. Led by CloudCamp's Dave Nielsen,=
 attendees are encouraged to share thoughts in open discussions with pre-de=
fined and majority-vote topics, including best practices in application dev=
elopment and advanced analytics.  The pre-defined topics will be presented =
in a form of a workshop with free AWS credits for Amazon Elastic MapReduce.

Data engineers, enterprise architects, developers, analysts, data mining an=
d business intelligence professionals are encouraged to attend and mix with=
 other members of the community.

Hope to see you there.

Cheers,
Steve Wooledge
Aster Data
www.asterdata.com=20


From general-return-1669-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 18 22:49:41 2010
Return-Path: <general-return-1669-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22632 invoked from network); 18 Jun 2010 22:49:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Jun 2010 22:49:37 -0000
Received: (qmail 77840 invoked by uid 500); 18 Jun 2010 22:49:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77760 invoked by uid 500); 18 Jun 2010 22:49:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 27990 invoked by uid 99); 17 Jun 2010 04:39:31 -0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of nige.daley@gmail.com designates 209.85.160.48 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:content-type:mime-version
         :subject:from:in-reply-to:date:content-transfer-encoding:message-id
         :references:to:x-mailer;
        bh=bShBUN9oPzmL63JxmGwahaZKLQ0GyCzQN7tlCtBlZDU=;
        b=f2FaDoVCfdTYHEuxn/cDwHlogS7a0ePWdgofzYaKdQkMeRZnfgNpGj0mrLJs4tcypW
         ZyG7r4mpWApWRnoDC3FCSgc+1w1oMWBUZSN0o6K6N8uv8Rh8lQP5uGwrfp54IoMluMlj
         X2f3R59mb4++1cNIXDak0eBTYdoyFfg+708tc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=N8/koMa70R2hLGKnlU0HWUhFMlHy/Dx3rGWWxVXkBd7vvxeRbzpx9NAWBzl2qp3QjD
         37MTYwyU+cE1lZlb72hIB54Sgf1i6MCst6sQmvw61Kfxi6F+BaMUcjD67/KH+oXOSwWq
         pPOODeNuOiQajDLW4wG0AQKfJaQ2rQNCwEV5A=
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1078)
Subject: Re: [VOTE] Move Chukwa to incubator
From: Nigel Daley <nige.daley@gmail.com>
In-Reply-To: <C83E50AD.908B%eyang@yahoo-inc.com>
Date: Wed, 16 Jun 2010 21:38:57 -0700
Content-Transfer-Encoding: 7bit
Message-Id: <BBC5F11B-2016-42E3-A19E-38D0CF022B13@apache.org>
References: <C83E50AD.908B%eyang@yahoo-inc.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1078)
X-Virus-Checked: Checked by ClamAV on apache.org

+1

On Jun 16, 2010, at 9:58 AM, Eric Yang wrote:

> Please vote as to whether you think Chukwa should move to Apache incubator.
> 
> The proposal is posted at:
> 
> http://wiki.apache.org/incubator/ChukwaProposal
> 
> Thanks
> 
> Regards,
> Eric
> 


From general-return-1670-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 18 23:28:13 2010
Return-Path: <general-return-1670-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34448 invoked from network); 18 Jun 2010 23:28:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Jun 2010 23:28:12 -0000
Received: (qmail 7543 invoked by uid 500); 18 Jun 2010 23:28:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7486 invoked by uid 500); 18 Jun 2010 23:28:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7478 invoked by uid 99); 18 Jun 2010 23:28:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 23:28:10 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 23:28:04 +0000
Received: by wwc33 with SMTP id 33so833412wwc.35
        for <general@hadoop.apache.org>; Fri, 18 Jun 2010 16:27:44 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.161.9 with SMTP id v9mr1346444wek.20.1276903663441; Fri, 
	18 Jun 2010 16:27:43 -0700 (PDT)
Received: by 10.216.162.20 with HTTP; Fri, 18 Jun 2010 16:27:43 -0700 (PDT)
In-Reply-To: <46220.125.16.17.151.1276690205.squirrel@students.iiit.ac.in>
References: <46220.125.16.17.151.1276690205.squirrel@students.iiit.ac.in>
Date: Fri, 18 Jun 2010 19:27:43 -0400
Message-ID: <AANLkTim253U8xpvkP8fEbeC8I9LNLDmS9C1QOh1Ax6uP@mail.gmail.com>
Subject: Re: How many records will be passed to a map function??
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Karan:

In general, you should let Hadoop pick the number of mappers to use.
In the case of only 1000 records @ 12k, performance will be better
with a single mapper for IO bound jobs. When you force the number of
map tasks, Hadoop will do the following:

(Assuming FileInputFormat#getSplits(conf, numSplits) gets called)

totalSize is sum size of all input files in bytes
goalSize is totalSize / numSplits
minSplitSize is conf value mapred.min.split.size (default 1)

For each input file:
  length =3D file.size()
  while isSplitable(file) and length !=3D 0
    fileBlockSize is the block size of the file
    minOfGoalBlock is min(goalSize, fileBlockSize)
    realSplitSize is max(minSplitSize, minOfGoalBlock)

    length is length minus realSplitSize (give or take)

Note that it's actually more confusing than this, but this is the
general idea. Let's plug in some numbers:

1 file
totalSize =3D 12k file size
blockSize =3D 64MB block
numSplits =3D 2
goalSize =3D 6k (12k / 2)
minSplitSize =3D 1 (for FileInputFormat)

minOfGoalBlock =3D 6k (6k < 64MB)
realSplitSize =3D 6k (6k > 1)

We end up with 2 splits, 6k each. RecordReaders then parse this into record=
s.

Note that this applies to the old APIs. The newer APIs work slightly
different but I think the result is equivalent.

(If anyone wants to double check my summation, I welcome it. This is
some hairy code and these questions frequently come up.)

Hope this helps.

On Wed, Jun 16, 2010 at 8:10 AM, Karan Jindal
<karan_jindal@students.iiit.ac.in> wrote:
>
> Hi all,
>
> Given a scenario in which a input file contains total 1000 records (recor=
d
> in a line) of total size 12k and I set number of map tasks to 2.
> How many records will be passed to each map task? Is it the equal
> distribution?
>
> InputFormat =3D Text
> Block size =A0=3D default block of hdfs
>
> Hoping for a reply..
>
> Regards
> Karan
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>
>



--=20
Eric Sammer
twitter: esammer
data: www.cloudera.com

From general-return-1671-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 19 00:14:21 2010
Return-Path: <general-return-1671-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46982 invoked from network); 19 Jun 2010 00:14:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Jun 2010 00:14:20 -0000
Received: (qmail 32316 invoked by uid 500); 19 Jun 2010 00:14:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32262 invoked by uid 500); 19 Jun 2010 00:14:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32252 invoked by uid 99); 19 Jun 2010 00:14:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 19 Jun 2010 00:14:19 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 19 Jun 2010 00:14:12 +0000
Received: by vws4 with SMTP id 4so231303vws.35
        for <general@hadoop.apache.org>; Fri, 18 Jun 2010 17:13:51 -0700 (PDT)
Received: by 10.220.89.166 with SMTP id e38mr675509vcm.247.1276906431129; Fri, 
	18 Jun 2010 17:13:51 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.220.18.15 with HTTP; Fri, 18 Jun 2010 17:13:30 -0700 (PDT)
In-Reply-To: <AANLkTim253U8xpvkP8fEbeC8I9LNLDmS9C1QOh1Ax6uP@mail.gmail.com>
References: <46220.125.16.17.151.1276690205.squirrel@students.iiit.ac.in> 
	<AANLkTim253U8xpvkP8fEbeC8I9LNLDmS9C1QOh1Ax6uP@mail.gmail.com>
From: Aaron Kimball <aaron@cloudera.com>
Date: Fri, 18 Jun 2010 17:13:30 -0700
Message-ID: <AANLkTinacmnFemOJx6S8XpZgo27N7fScMnxna3BHQfq5@mail.gmail.com>
Subject: Re: How many records will be passed to a map function??
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64766043d2a37048956f2d6
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e64766043d2a37048956f2d6
Content-Type: text/plain; charset=ISO-8859-1

Short answer: FileInputFormat & friends generate splits based on byte
ranges.

Assuming your records are all equally sized, you'll get half your records in
each mapper. If your records have many different sizes represented, then
your mileage may vary.

- Aaron

On Fri, Jun 18, 2010 at 4:27 PM, Eric Sammer <esammer@cloudera.com> wrote:

> Karan:
>
> In general, you should let Hadoop pick the number of mappers to use.
> In the case of only 1000 records @ 12k, performance will be better
> with a single mapper for IO bound jobs. When you force the number of
> map tasks, Hadoop will do the following:
>
> (Assuming FileInputFormat#getSplits(conf, numSplits) gets called)
>
> totalSize is sum size of all input files in bytes
> goalSize is totalSize / numSplits
> minSplitSize is conf value mapred.min.split.size (default 1)
>
> For each input file:
>  length = file.size()
>  while isSplitable(file) and length != 0
>    fileBlockSize is the block size of the file
>    minOfGoalBlock is min(goalSize, fileBlockSize)
>    realSplitSize is max(minSplitSize, minOfGoalBlock)
>
>    length is length minus realSplitSize (give or take)
>
> Note that it's actually more confusing than this, but this is the
> general idea. Let's plug in some numbers:
>
> 1 file
> totalSize = 12k file size
> blockSize = 64MB block
> numSplits = 2
> goalSize = 6k (12k / 2)
> minSplitSize = 1 (for FileInputFormat)
>
> minOfGoalBlock = 6k (6k < 64MB)
> realSplitSize = 6k (6k > 1)
>
> We end up with 2 splits, 6k each. RecordReaders then parse this into
> records.
>
> Note that this applies to the old APIs. The newer APIs work slightly
> different but I think the result is equivalent.
>
> (If anyone wants to double check my summation, I welcome it. This is
> some hairy code and these questions frequently come up.)
>
> Hope this helps.
>
> On Wed, Jun 16, 2010 at 8:10 AM, Karan Jindal
> <karan_jindal@students.iiit.ac.in> wrote:
> >
> > Hi all,
> >
> > Given a scenario in which a input file contains total 1000 records
> (record
> > in a line) of total size 12k and I set number of map tasks to 2.
> > How many records will be passed to each map task? Is it the equal
> > distribution?
> >
> > InputFormat = Text
> > Block size  = default block of hdfs
> >
> > Hoping for a reply..
> >
> > Regards
> > Karan
> >
> > --
> > This message has been scanned for viruses and
> > dangerous content by MailScanner, and is
> > believed to be clean.
> >
> >
>
>
>
> --
> Eric Sammer
> twitter: esammer
> data: www.cloudera.com
>

--0016e64766043d2a37048956f2d6--

From general-return-1672-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 21 09:30:57 2010
Return-Path: <general-return-1672-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95872 invoked from network); 21 Jun 2010 09:30:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jun 2010 09:30:56 -0000
Received: (qmail 5624 invoked by uid 500); 21 Jun 2010 09:30:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4763 invoked by uid 500); 21 Jun 2010 09:30:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4749 invoked by uid 99); 21 Jun 2010 09:30:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Jun 2010 09:30:50 +0000
X-ASF-Spam-Status: No, hits=0.2 required=10.0
	tests=AWL,FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bernd.fondermann@googlemail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Jun 2010 09:30:45 +0000
Received: by wyb29 with SMTP id 29so3164044wyb.35
        for <general@hadoop.apache.org>; Mon, 21 Jun 2010 02:30:24 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=ODOCGwghsuPFEc8qKgR8NOWwZhpb6p3o9baehV1osrs=;
        b=xj8MUAKI5m/q+y6hWs+FyK4a9wAjF1ipSgb1hwc4SmAgA0KSCShMhMifu9uD3GYVQF
         tcdBJaXPFrfvh0bbsL6UiDEaQJmQy/ebRL9cS3OK2hXcJrnm0LA4yIo+8Q0FkXbiNr1C
         iNLX1GtoPrc8Lu191BJRvQEpYC23MxhjeqnJ0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=u6g5N0nJhAp+pxAv446sYLkbbbhydhqcEXfG7gKRKAmYJ+ixzTEk0WC7QSCEJMLUjU
         0jKd8sc0Lb2nK2WEt7H4DQGwzD5Rte0qj/wpOdOWNesDXnFIX/wvkndn2zm6mDCxxjRH
         5ySofrv/cIP57FyP/6oDSKxfEBu0pOSqQ0Dyg=
MIME-Version: 1.0
Received: by 10.216.176.80 with SMTP id a58mr2494297wem.35.1277112623386; Mon, 
	21 Jun 2010 02:30:23 -0700 (PDT)
Received: by 10.216.156.6 with HTTP; Mon, 21 Jun 2010 02:30:23 -0700 (PDT)
Date: Mon, 21 Jun 2010 11:30:23 +0200
Message-ID: <AANLkTikCu3pebRoKFVtuf069BxHM7I-cePj_XiWox_or@mail.gmail.com>
Subject: Another mentor [WAS: Re: [VOTE] Move Chukwa to incubator]
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hi,

anyone mind if I'd add myself as a mentor?

  Bernd

On Thu, Jun 17, 2010 at 06:38, Nigel Daley <nige.daley@gmail.com> wrote:
> +1
>
> On Jun 16, 2010, at 9:58 AM, Eric Yang wrote:
>
>> Please vote as to whether you think Chukwa should move to Apache incubator.
>>
>> The proposal is posted at:
>>
>> http://wiki.apache.org/incubator/ChukwaProposal
>>
>> Thanks
>>
>> Regards,
>> Eric
>>
>
>

From general-return-1673-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 21 16:18:10 2010
Return-Path: <general-return-1673-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 8874 invoked from network); 21 Jun 2010 16:18:10 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jun 2010 16:18:10 -0000
Received: (qmail 55533 invoked by uid 500); 21 Jun 2010 16:18:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55282 invoked by uid 500); 21 Jun 2010 16:18:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55274 invoked by uid 99); 21 Jun 2010 16:18:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Jun 2010 16:18:08 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.30.80] (HELO qmta08.emeryville.ca.mail.comcast.net) (76.96.30.80)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Jun 2010 16:17:58 +0000
Received: from omta06.emeryville.ca.mail.comcast.net ([76.96.30.51])
	by qmta08.emeryville.ca.mail.comcast.net with comcast
	id YbxY1e00516AWCUA8gHcG9; Mon, 21 Jun 2010 16:17:37 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta06.emeryville.ca.mail.comcast.net with comcast
	id YgHH1e0052SbwD58SgHVPG; Mon, 21 Jun 2010 16:17:34 +0000
Message-Id: <B40086F9-7A89-4925-911B-31881935CBC8@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <AANLkTikCu3pebRoKFVtuf069BxHM7I-cePj_XiWox_or@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Another mentor [WAS: Re: [VOTE] Move Chukwa to incubator]
Date: Mon, 21 Jun 2010 09:17:24 -0700
References: <AANLkTikCu3pebRoKFVtuf069BxHM7I-cePj_XiWox_or@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


On Jun 21, 2010, at 2:30 AM, Bernd Fondermann wrote:

> Hi,
>
> anyone mind if I'd add myself as a mentor?

That would be great.

-- Owen

From general-return-1674-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 16:06:08 2010
Return-Path: <general-return-1674-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 86743 invoked from network); 23 Jun 2010 16:06:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 16:06:08 -0000
Received: (qmail 39045 invoked by uid 500); 23 Jun 2010 16:06:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38671 invoked by uid 500); 23 Jun 2010 16:06:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 6948 invoked by uid 99); 23 Jun 2010 15:50:17 -0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 846738.63928.bm@omp301.mail.re3.yahoo.com
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1277308180; bh=4PSz+URDb/GzQePUmdFr0fScZHoi1xAiRcnigVP7ZaI=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=QAALIkJrE8Sj5HyX5cCznYAVKdSOAcRjK03VmmrLz0XqI7+4u1DWfbZdiMr95ZSV5+PZSImo5YK9cQad82UNQULr9BKhAMCERACuDNJrSYcIFoH/6eV9ORa4F31s/JZMzhqzw/U/eCPCdOufX9q8krSD9hj766NkjqfALVKW4y0=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=vV4LAI59+mPC8qVckY1eq2xkkg9za1YsCWDsNlyYl1StWCftmpuvTt/ubFKUhK7rsK2QyJPdJU8j2bBGIVzJIRnw2DEmTbD+durSBQizUbR+jWbskPV1zcGUbLGWPtw3xLqSZXi3AAlX6Ja1i8mqCIJLr5Eq0Nv//uXpurrAIp0=;
Message-ID: <217002.5465.qm@web56201.mail.re3.yahoo.com>
X-YMail-OSG: Oxz_VcMVM1nwCgFLBeQR59XREtikzlMKIqOA.pKrdXbisDP
 .OnLfHWMZGWplpKRhCw1XBic1S24aEagP6HyT4cc2T6bjk7MQV_Vrlrnk_QA
 ZL4dw7ijp67AzeGxeKIrI6_1vvx1YiOqoR5zfMTuJkSeT0y1o9r1.4QHclc1
 6jp5BAX5lVS8HipVLC8iX.5CNNSaIiEcapbBGob5o9RbMLyCWdnvG_rtSueO
 RQCLMlgYXIjjqQfrlgvESeZolKOfW11Dk5QKHOyNbBXN53q5o8US.P09asAO
 WfArL6gcpC06EbJw2T83kcp8WG.Qdq.6mdxCrk7RQS7zPmhLFDlzqVotg33r
 nVdaey6Kv0HH3fFsvwKuhfI4quO8YSP5ew0n8qZLfClbiq6EIUQ--
X-Mailer: YahooMailRC/397.8 YahooMailWebService/0.8.104.274457
References: <20100623153137.16141.23534@eos.apache.org>
Date: Wed, 23 Jun 2010 08:49:39 -0700 (PDT)
From: "Tsz Wo \(Nicholas\), Sze" <s29752-hadoopdev@yahoo.com>
Subject: Re: New attachment added to page download on Hadoop Wiki
To: common-dev@hadoop.apache.org, general@hadoop.apache.org
In-Reply-To: <20100623153137.16141.23534@eos.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii

We are getting more spams in the hadoop wiki page.  How should we deal with it?

Nicholas Sze




----- Original Message ----
> From: Apache Wiki <wikidiffs@apache.org>
> To: Apache Wiki <wikidiffs@apache.org>
> Sent: Wed, June 23, 2010 8:31:37 AM
> Subject: New attachment added to page download on Hadoop Wiki
> 
> Dear Wiki user,

You have subscribed to a wiki page "download" for change 
> notification. An attachment has been added to that page by Eric Pridz. Following 
> detailed information is available:

Attachment name: 
> download-mtv-movie-awards-2010
Attachment size: 3746
Attachment link: 
> http://wiki.apache.org/hadoop/download?action=AttachFile&do=get&target=download-mtv-movie-awards-2010
Page 
> link: http://wiki.apache.org/hadoop/download


From general-return-1675-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 16:06:22 2010
Return-Path: <general-return-1675-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 86934 invoked from network); 23 Jun 2010 16:06:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 16:06:21 -0000
Received: (qmail 40887 invoked by uid 500); 23 Jun 2010 16:06:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40820 invoked by uid 500); 23 Jun 2010 16:06:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 66062 invoked by uid 99); 22 Jun 2010 12:34:56 -0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of shujamughal@gmail.com designates 209.85.210.48 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=6KpLHn0qcXABYf6K5taG5UMUPp+62cHSSZokHw0RUqE=;
        b=NAfU+I1ycOfrPmOQMYYAmM5mOZWz5sAoq32H2izeaxsjIl6jAaFY5vd/nqf8T5ppxA
         geuaO42lH1uhIfrVh1EuJcLFgUab3JBohCzhNGub3IRDC/ZBR8HyLpZdoIfSEbV/X1pR
         Uk26iZfwM3J4/AXfygItfecCQSbhSYMeP4S+g=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=o60n14Id0wWamqIa1jbmrhnCJxuYIAfRsaMMkMAQu7+uFanwFXTqyJWeDTI44o4voh
         DQVLHoVGzrjcxxFgpcVcXT8CrcMZRq2PI5Xd3wyJft/sWXQ3VPTeMfH4lY798JhsoFwB
         XpQbkKnlXJNu/dzUzorOhM5J+5dj5TgXdreDw=
MIME-Version: 1.0
Date: Tue, 22 Jun 2010 17:34:27 +0500
Message-ID: <AANLkTiltiDhs2-pTGtqYThW40acTc_EVgRv1wzj2Dlf3@mail.gmail.com>
Subject: Configure Groovy for map/reduce jobs in hadoop
From: Shuja Rehman <shujamughal@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd32f9662691804899da417
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd32f9662691804899da417
Content-Type: text/plain; charset=ISO-8859-1

Hi
I have write a map/reduce job in groovy and run the job using hadoop
streaming as follow.

hadoop jar
/usr/lib/hadoop-0.20/contrib/streaming/hadoop-0.20.2+228-streaming.jar \
-inputformat StreamInputFormat \
-inputreader "StreamXmlRecordReader,begin=<mdc xmlns:HTML=\"
http://www.w3.org/TR/REC-xml\">,end=</mdc>" \
-input /user/root/telecom/ \
-jobconf mapred.map.tasks=1 \
-output q5 \
-mapper /home/ftpuser1/Nodemapper2.groovy \
-jobconf mapred.reduce.tasks=0 \
-file /home/ftpuser1/Nodemapper2.groovy


The problem is that the output files has 0 byte size. I have checked the log
file. The log file contains the following error.

*/usr/bin/env: groovy: No such file or directory*


The groovy is installed on the system and i verify it by running with
help of following command.

*cat 1.xml | /root/Nodemapper2.groovy *

This command runs fine and shows the output. The Mapper has the
following line in the start.

*#!/usr/bin/env groovy*

Can anybody tell is there any special configurations required to run groovy
map/reduce jobs with hadoop?


-- 
Regards
Shuja-ur-Rehman Baig
_________________________________
MS CS - School of Science and Engineering
Lahore University of Management Sciences (LUMS)
Sector U, DHA, Lahore, 54792, Pakistan
Cell: +92 3214207445

--000e0cd32f9662691804899da417--

From general-return-1676-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 16:31:00 2010
Return-Path: <general-return-1676-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95661 invoked from network); 23 Jun 2010 16:30:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 16:30:59 -0000
Received: (qmail 87416 invoked by uid 500); 23 Jun 2010 16:30:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87369 invoked by uid 500); 23 Jun 2010 16:30:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87361 invoked by uid 99); 23 Jun 2010 16:30:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 16:30:57 +0000
X-ASF-Spam-Status: No, hits=-0.1 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of swatt@us.ibm.com designates 32.97.110.149 as permitted sender)
Received: from [32.97.110.149] (HELO e31.co.us.ibm.com) (32.97.110.149)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 16:30:47 +0000
Received: from d03relay03.boulder.ibm.com (d03relay03.boulder.ibm.com [9.17.195.228])
	by e31.co.us.ibm.com (8.14.4/8.13.1) with ESMTP id o5NGJebG005514
	for <general@hadoop.apache.org>; Wed, 23 Jun 2010 10:19:40 -0600
Received: from d03av02.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.195.168])
	by d03relay03.boulder.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id o5NGUCKW025468
	for <general@hadoop.apache.org>; Wed, 23 Jun 2010 10:30:17 -0600
Received: from d03av02.boulder.ibm.com (loopback [127.0.0.1])
	by d03av02.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id o5NGU8Ro005621
	for <general@hadoop.apache.org>; Wed, 23 Jun 2010 10:30:09 -0600
Received: from d03nm123.boulder.ibm.com (d03nm123.boulder.ibm.com [9.17.195.149])
	by d03av02.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id o5NGU86v005534
	for <general@hadoop.apache.org>; Wed, 23 Jun 2010 10:30:08 -0600
In-Reply-To: <B40086F9-7A89-4925-911B-31881935CBC8@apache.org>
References: <AANLkTikCu3pebRoKFVtuf069BxHM7I-cePj_XiWox_or@mail.gmail.com> <B40086F9-7A89-4925-911B-31881935CBC8@apache.org>
To: general@hadoop.apache.org
MIME-Version: 1.0
Subject: The IBM Distribution of Apache Hadoop now available
X-KeepSent: F68C4536:C7F1E4B5-8625774B:00558405;
 type=4; name=$KeepSent
X-Mailer: Lotus Notes Release 8.5 December 05, 2008
Message-ID: <OFF68C4536.C7F1E4B5-ON8625774B.00558405-8625774B.005AA5CE@us.ibm.com>
From: Stephen Watt <swatt@us.ibm.com>
Date: Wed, 23 Jun 2010 11:30:07 -0500
X-MIMETrack: Serialize by Router on D03NM123/03/M/IBM(Release 8.5.1HF41 | October 22, 2009) at
 06/23/2010 10:30:08,
	Serialize complete at 06/23/2010 10:30:08
Content-Type: multipart/alternative; boundary="=_alternative 005AA5CD8625774B_="
X-Virus-Checked: Checked by ClamAV on apache.org

--=_alternative 005AA5CD8625774B_=
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: base64

SGkgRm9sa3MNCg0K4oCo4oCoSUJNIGhhcyBtYWRlIGF2YWlsYWJsZSBhIHByZXZpZXcgdmVyc2lv
biBvZiB0aGUgSUJNIERpc3RyaWJ1dGlvbiBvZiANCkFwYWNoZSBIYWRvb3AuIFlvdSBjYW4gYWNj
ZXNzIGl0IGhlcmUgLSANCmh0dHA6Ly93d3cuYWxwaGF3b3Jrcy5pYm0uY29tL3RlY2gvaWRhaC8g
LiBUaGlzIGRpc3RyaWJ1dGlvbiBjb250YWlucyANCkFwYWNoZSBIYWRvb3AgMC4yMC4yLCBhIDMy
LWJpdCBMaW51eCB2ZXJzaW9uIG9mIHRoZSBJQk0gU0RLIGZvciBKYXZhIDYgU1IgDQo4LCBhbmQg
YW4gZWFzeSB0byB1c2Ugd2ViIGJhc2VkIHdpemFyZCBmb3IgdGhlIGluc3RhbGxhdGlvbiBhbmQg
DQpjb25maWd1cmF0aW9uIG9mIEhhZG9vcC7igKjigKgNCg0KVGhlIGJlc3QgcGxhY2UgZm9yIHF1
ZXN0aW9ucyBvbiB0aGlzIGRpc3RyaWJ1dGlvbiBhcmUgdGhlIHRocmVhZHMgdW5kZXIgDQp0aGUg
Zm9ydW0gdGFiLCBhY2Nlc3NpYmxlIGZyb20gdGhlIGxpbmsgcHJvdmlkZWQgYWJvdmUuIFdlJ2xs
IGFsc28gYmUgYXQgDQp0aGUgSGFkb29wIFN1bW1pdC7igKjigKgNCg0KUmVnYXJkcw0K4oCoU3Rl
dmUgV2F0dA0KDQoNCg==
--=_alternative 005AA5CD8625774B_=--

From general-return-1677-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 24 20:10:47 2010
Return-Path: <general-return-1677-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43828 invoked from network); 24 Jun 2010 20:10:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Jun 2010 20:10:47 -0000
Received: (qmail 94020 invoked by uid 500); 24 Jun 2010 20:10:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93501 invoked by uid 500); 24 Jun 2010 20:10:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93483 invoked by uid 99); 24 Jun 2010 20:10:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Jun 2010 20:10:43 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Jun 2010 20:10:37 +0000
Received: by gyf1 with SMTP id 1so7271949gyf.35
        for <multiple recipients>; Thu, 24 Jun 2010 13:10:16 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        bh=Xk6jofMRskNfUGhgwcuEtqR0hrA93R93s/PhGcZWmGA=;
        b=RGIMt7Tjsj67T8fk2c8CBRACWG19+Hvo6tl9OLjbN3NbvIDOubRl8pmG7xs3NfIkYz
         vQFv6KTdLE0jhP1cOHsO0miUQL6pgX0WsUSQJt1l8JUXxhC/G2N8pJ0WSbQFqkLYmAzi
         T1wNv9Iqx6AmWxBp+6WSd93NCROWrJ5bHGqkY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        b=tiqyUb5cl5RBfZGYBjJhcw00CzaH5GbdibqeDcEfTusqwBsPShz1QvHMnWqihqB/bH
         m8JgNEqPBF86dFExEJfzOdM3co3IiDNZjnsBMjE7DbZJ9RF8kX8icxgb0cwZpX9TpLnE
         o/K10M7TCrllgqUGNDJ/5d4cnYOhx371JVgJY=
MIME-Version: 1.0
Received: by 10.229.181.21 with SMTP id bw21mr5719269qcb.117.1277410216253; 
	Thu, 24 Jun 2010 13:10:16 -0700 (PDT)
Sender: saint.ack@gmail.com
Received: by 10.229.40.71 with HTTP; Thu, 24 Jun 2010 13:10:16 -0700 (PDT)
Date: Thu, 24 Jun 2010 13:10:16 -0700
X-Google-Sender-Auth: OMSSk1a063uwOw8ptlfZpcJEwFA
Message-ID: <AANLkTilywIxjw2TbnPq028llQcNRCr-0xvv_SPs9t2H-@mail.gmail.com>
Subject: [ANN] HBase-0.20.5 availalble for download
From: Stack <stack@duboce.net>
To: hbase-user@hadoop.apache.org, general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

HBase 0.20.5 is available for download:

  http://hadoop.apache.org/hbase/releases.html

The Release Notes are available here: http://su.pr/1prW9y

We recommend that all users, particularly those running 0.20.4,
upgrade to this release.

Thanks to all who contributed to this release

Yours,
The HBasistas

From general-return-1678-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 25 09:30:38 2010
Return-Path: <general-return-1678-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58301 invoked from network); 25 Jun 2010 09:30:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Jun 2010 09:30:37 -0000
Received: (qmail 86078 invoked by uid 500); 25 Jun 2010 09:30:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85369 invoked by uid 500); 25 Jun 2010 09:30:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85328 invoked by uid 99); 25 Jun 2010 09:30:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 09:30:27 +0000
X-ASF-Spam-Status: No, hits=-1.0 required=10.0
	tests=AWL,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [88.198.2.104] (HELO koch.ro) (88.198.2.104)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 09:30:22 +0000
Received: from 84-72-85-88.dclient.hispeed.ch ([84.72.85.88] helo=jona.localnet)
	by koch.ro with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
	(Exim 4.69)
	(envelope-from <thomas@koch.ro>)
	id 1OS54r-0004sj-U0; Fri, 25 Jun 2010 11:20:02 +0200
From: Thomas Koch <thomas@koch.ro>
Reply-To: thomas@koch.ro
To: general@hadoop.apache.org,
 dev@hbase.apache.org,
 java-dev@lucene.apache.org,
 zookeeper-dev@hadoop.apache.org
Subject: Please get your gpg keys signed!
Date: Fri, 25 Jun 2010 11:29:55 +0200
User-Agent: KMail/1.13.3 (Linux/2.6.32-4-amd64; KDE/4.4.4; x86_64; ; )
MIME-Version: 1.0
Content-Type: text/plain;
  charset="us-ascii"
Content-Transfer-Encoding: 7bit
Message-Id: <201006251129.56899.thomas@koch.ro>

Hi,

I just wanted to package the new HBase version and since I've just recently 
read about a malicious software tarball for some Linux IRC server[1], I got 
back to the habbit of checking signatures. (Yes, I was lazy recently. I'm 
ashamed.)

But checking the signatures of apache software obviously is meaningless, since 
apache developers appears to not have their keys in the web-of-trust. From 
three signature files I had laying around on my hard disc, all three keys had 
zero signatures on the MIT keyserver:

30CD0996 2010-05-03 Michael Stack <stack@duboce.net>
68E327C1 2008-10-22 Patrick Hunt <phunt@apache.org>
FE045966 2009-10-13 Grant Ingersoll <gsingers@apache.org>

So please, when you've your next Hadoop / HBase / Lucene / Apache meetings, 
take your time for a keysigning party[2]. Or just have some snippet with your 
keys fingerprint in your wallet and hand it to every other geek you meet. (And 
make sure he asks you for your ID card to check your identity!) It's also nice 
to have your gpg fingerprint on your business cards!

[1] http://www.sophos.com/blogs/chetw/g/2010/06/12/linux-malware-rears-ugly-
head/
[2] http://en.wikipedia.org/wiki/Key_signing_party

Thank you!

Thomas Koch, http://www.koch.ro

From general-return-1679-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 25 09:48:31 2010
Return-Path: <general-return-1679-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 64177 invoked from network); 25 Jun 2010 09:48:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Jun 2010 09:48:30 -0000
Received: (qmail 7967 invoked by uid 500); 25 Jun 2010 09:48:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7443 invoked by uid 500); 25 Jun 2010 09:48:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7424 invoked by uid 99); 25 Jun 2010 09:48:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 09:48:24 +0000
X-ASF-Spam-Status: No, hits=0.2 required=10.0
	tests=AWL,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.78.25] (HELO ey-out-2122.google.com) (74.125.78.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 09:48:18 +0000
Received: by ey-out-2122.google.com with SMTP id 22so40200eye.23
        for <multiple recipients>; Fri, 25 Jun 2010 02:47:56 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.103.80.8 with SMTP id h8mr105877mul.90.1277459276357; Fri, 25 
	Jun 2010 02:47:56 -0700 (PDT)
Sender: tcurdt@vafer.org
Received: by 10.103.173.3 with HTTP; Fri, 25 Jun 2010 02:47:56 -0700 (PDT)
In-Reply-To: <201006251129.56899.thomas@koch.ro>
References: <201006251129.56899.thomas@koch.ro>
Date: Fri, 25 Jun 2010 11:47:56 +0200
X-Google-Sender-Auth: EKBC1Tf-kObIqwFVtQ3qCfobMeI
Message-ID: <AANLkTik1ug3NNB_nN4uRZmPak4xNCBLRrQfIOIFgkOvL@mail.gmail.com>
Subject: Re: Please get your gpg keys signed!
From: Torsten Curdt <tcurdt@apache.org>
To: general@hadoop.apache.org, thomas@koch.ro
Cc: dev@hbase.apache.org, java-dev@lucene.apache.org, 
	zookeeper-dev@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

> But checking the signatures of apache software obviously is meaningless, since
> apache developers appears to not have their keys in the web-of-trust

Many many do :)

> So please, when you've your next Hadoop / HBase / Lucene / Apache meetings,
> take your time for a keysigning party[2].

We should have done some key signing during the buzzwords conference.
For people in Berlin: I am happy to exchange keys to get them into the
web-of-trust.
Will certainly suggest something like that for our Apache Dinners.

cheers
--
Torsten

From general-return-1680-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 25 16:37:06 2010
Return-Path: <general-return-1680-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 7264 invoked from network); 25 Jun 2010 16:37:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Jun 2010 16:37:06 -0000
Received: (qmail 81075 invoked by uid 500); 25 Jun 2010 16:37:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80956 invoked by uid 500); 25 Jun 2010 16:37:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80945 invoked by uid 99); 25 Jun 2010 16:37:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 16:37:03 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 25 Jun 2010 16:37:01 +0000
Received: (qmail 7137 invoked by uid 99); 25 Jun 2010 16:36:39 -0000
Received: from localhost.apache.org (HELO [192.168.42.205]) (127.0.0.1)
  (smtp-auth username phunt, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 16:36:39 +0000
Message-ID: <4C24DB16.7090505@apache.org>
Date: Fri, 25 Jun 2010 09:36:38 -0700
From: Patrick Hunt <phunt@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.1.10) Gecko/20100512 Lightning/1.0b1 Thunderbird/3.0.5
MIME-Version: 1.0
To: zookeeper-dev@hadoop.apache.org
CC: Thomas Koch <thomas@koch.ro>, general@hadoop.apache.org, 
 dev@hbase.apache.org, java-dev@lucene.apache.org
Subject: Re: Please get your gpg keys signed!
References: <201006251129.56899.thomas@koch.ro>
In-Reply-To: <201006251129.56899.thomas@koch.ro>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Thomas, are you attending the summit? There are a number of contributor 
workshops the day after, all at (or around) the same location. If you 
feel strongly about this consider attending them, seems like great 
opportunity for a "key signing party".
http://www.meetup.com/Hadoop-Contributors/calendar/13771414/
http://www.meetup.com/Hadoop-Contributors/calendar/13750403/
http://www.meetup.com/hbaseusergroup/calendar/13562846/
etc...

Here's some detail on WOT at apache:
http://www.apache.org/dev/release-signing.html#web-of-trust

Patrick

On 06/25/2010 02:29 AM, Thomas Koch wrote:
> Hi,
>
> I just wanted to package the new HBase version and since I've just recently
> read about a malicious software tarball for some Linux IRC server[1], I got
> back to the habbit of checking signatures. (Yes, I was lazy recently. I'm
> ashamed.)
>
> But checking the signatures of apache software obviously is meaningless, since
> apache developers appears to not have their keys in the web-of-trust. From
> three signature files I had laying around on my hard disc, all three keys had
> zero signatures on the MIT keyserver:
>
> 30CD0996 2010-05-03 Michael Stack<stack@duboce.net>
> 68E327C1 2008-10-22 Patrick Hunt<phunt@apache.org>
> FE045966 2009-10-13 Grant Ingersoll<gsingers@apache.org>
>
> So please, when you've your next Hadoop / HBase / Lucene / Apache meetings,
> take your time for a keysigning party[2]. Or just have some snippet with your
> keys fingerprint in your wallet and hand it to every other geek you meet. (And
> make sure he asks you for your ID card to check your identity!) It's also nice
> to have your gpg fingerprint on your business cards!
>
> [1] http://www.sophos.com/blogs/chetw/g/2010/06/12/linux-malware-rears-ugly-
> head/
> [2] http://en.wikipedia.org/wiki/Key_signing_party
>
> Thank you!
>
> Thomas Koch, http://www.koch.ro

From general-return-1681-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 26 06:01:09 2010
Return-Path: <general-return-1681-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82239 invoked from network); 26 Jun 2010 06:01:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jun 2010 06:01:09 -0000
Received: (qmail 87796 invoked by uid 500); 26 Jun 2010 06:01:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87550 invoked by uid 500); 26 Jun 2010 06:01:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87542 invoked by uid 99); 26 Jun 2010 06:01:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Jun 2010 06:01:04 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=AWL,HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Jun 2010 06:00:57 +0000
Received: by iwn38 with SMTP id 38so3778158iwn.35
        for <general@hadoop.apache.org>; Fri, 25 Jun 2010 23:00:37 -0700 (PDT)
Received: by 10.231.140.99 with SMTP id h35mr1817757ibu.147.1277532037092; 
	Fri, 25 Jun 2010 23:00:37 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.161.72 with HTTP; Fri, 25 Jun 2010 23:00:17 -0700 (PDT)
From: Todd Lipcon <todd@cloudera.com>
Date: Fri, 25 Jun 2010 23:00:17 -0700
Message-ID: <AANLkTinMAQK-uIG9_fwF8tAaBdJQXZfubJmoI-0fI7mF@mail.gmail.com>
Subject: [ANN] HBase-0.89.20100621 "development release" available for 
	download
To: user@hbase.apache.org, general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6464eea4296da0489e89b89

--0016e6464eea4296da0489e89b89
Content-Type: text/plain; charset=ISO-8859-1

Dear HBase Community,

The HBase team is pleased to announce the release of HBase 0.89.20100621.
This release is the first of a series of "development releases" that will
lead up to the release of HBase 0.90 later this year. To call out every new
feature in this release would be too much, but a short list of highlights
include fully durable edits, a preview of inter-cluster replication support,
and a much-improved command line shell.

As a development release series, we have not prepared formal release notes;
interested users should inspect the CHANGES.txt and KNOWN-BUGS.txt files
included in the distribution.

For more information about the purpose and version numbering of the 0.89
development release series, as well as how it relates to the stable 0.20
release series, please refer to the following wiki page:

http://wiki.apache.org/hadoop/Hbase/HBaseVersions

The release artifacts may be downloaded from:

http://www.apache.org/dyn/closer.cgi/hbase/hbase-0.89.20100621/
(the release is still propagating to mirrors, so please try again from a
different mirror if the first you try does not work)

Thanks to all who contributed to this release! We're looking forward to the
community's feedback.

-- The HBase Team

-- 
Todd Lipcon
Software Engineer, Cloudera

--0016e6464eea4296da0489e89b89--

From general-return-1682-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 27 15:00:16 2010
Return-Path: <general-return-1682-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84792 invoked from network); 27 Jun 2010 15:00:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jun 2010 15:00:16 -0000
Received: (qmail 64723 invoked by uid 500); 27 Jun 2010 15:00:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 64622 invoked by uid 500); 27 Jun 2010 15:00:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 64614 invoked by uid 99); 27 Jun 2010 15:00:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Jun 2010 15:00:14 +0000
X-ASF-Spam-Status: No, hits=3.9 required=10.0
	tests=AWL,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of zhangguoping96@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Jun 2010 15:00:07 +0000
Received: by fxm11 with SMTP id 11so545043fxm.35
        for <general@hadoop.apache.org>; Sun, 27 Jun 2010 07:59:46 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=Q8bOEDRkyXZuRiuY2P8iLfnbxaFiLj1FdFspM8gnO7k=;
        b=isIFEDvfMIDcYgeS7W+mOx0mgZsCVKHlYttiCoYRgI0Aw/aDuaTFuw0AALvNV7DQgQ
         O1raff5P8/yLUjQXDQrC81fwbxKo3kSPCpmhuG2+o/2NJd43bmQT9m+dMh1xoCytr5GO
         bnTRMgO7aBi2ztmKMH2oaOEkoOwnOiokVXcxc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=ZtRaUkY9RP7DCwYor+khXFfXkzqX9RzhX9Mj6g7PjhGhq/ZwmxxACv8otMNKl7tqzO
         5bABmIHXaYXCWeHAjj7rzW2x9r4MC0PSie8WlOHt+OAxMW87iZITmczcDDEYL1nbo6Qi
         ogDq3mvcQ3qq9OU70Kiwein5YWA5OegvJfJro=
MIME-Version: 1.0
Received: by 10.239.141.65 with SMTP id b1mr217404hba.8.1277650786269; Sun, 27 
	Jun 2010 07:59:46 -0700 (PDT)
Received: by 10.239.153.71 with HTTP; Sun, 27 Jun 2010 07:59:46 -0700 (PDT)
Date: Sun, 27 Jun 2010 22:59:46 +0800
Message-ID: <AANLkTilTKnEH2IA4XCyo_dbNnzED50hN9wIOQ-6IBU9s@mail.gmail.com>
Subject: how to get the current input file in a Mapper
From: zhangguoping zhangguoping <zhangguoping96@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f20c5e433aa2048a044101

--001485f20c5e433aa2048a044101
Content-Type: text/plain; charset=ISO-8859-1

Hi,

how to get the current input file in a Mapper in 0.20.2 ?

Thanks,

--001485f20c5e433aa2048a044101--

From general-return-1683-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 27 16:02:20 2010
Return-Path: <general-return-1683-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94138 invoked from network); 27 Jun 2010 16:02:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jun 2010 16:02:19 -0000
Received: (qmail 79351 invoked by uid 500); 27 Jun 2010 16:02:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79186 invoked by uid 500); 27 Jun 2010 16:02:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 79082 invoked by uid 99); 27 Jun 2010 16:02:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Jun 2010 16:02:17 +0000
X-ASF-Spam-Status: No, hits=-0.1 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of Dikran_Meliksetian@us.ibm.com designates 32.97.182.146 as permitted sender)
Received: from [32.97.182.146] (HELO e6.ny.us.ibm.com) (32.97.182.146)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Jun 2010 16:02:07 +0000
Received: from d01relay07.pok.ibm.com (d01relay07.pok.ibm.com [9.56.227.147])
	by e6.ny.us.ibm.com (8.14.4/8.13.1) with ESMTP id o5RG0TAo002138
	for <general@hadoop.apache.org>; Sun, 27 Jun 2010 12:00:29 -0400
Received: from d01av02.pok.ibm.com (d01av02.pok.ibm.com [9.56.224.216])
	by d01relay07.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id o5RG1j8B1573084
	for <general@hadoop.apache.org>; Sun, 27 Jun 2010 12:01:45 -0400
Received: from d01av02.pok.ibm.com (loopback [127.0.0.1])
	by d01av02.pok.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id o5RG1jVO024252
	for <general@hadoop.apache.org>; Sun, 27 Jun 2010 13:01:45 -0300
Received: from d01ml255.pok.ibm.com (d01ml255.pok.ibm.com [9.56.227.128])
	by d01av02.pok.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id o5RG1jig024249
	for <general@hadoop.apache.org>; Sun, 27 Jun 2010 13:01:45 -0300
Subject: AUTO: Dikran S Meliksetian is out of the office (returning 07/12/2010)
Auto-Submitted: auto-generated
From: Dikran S Meliksetian <Dikran_Meliksetian@us.ibm.com>
To: general@hadoop.apache.org
Message-ID: <OF0C695685.A2047CE5-ON8525774F.00580C79-8525774F.00580C79@us.ibm.com>
Date: Sun, 27 Jun 2010 12:01:43 -0400
X-MIMETrack: Serialize by Router on D01ML255/01/M/IBM(Release 8.0.1|February 07, 2008) at
 06/27/2010 12:01:44
MIME-Version: 1.0
Content-type: multipart/alternative; 
	Boundary="0__=0ABBFDDCDFCB8AE98f9e8a93df938690918c0ABBFDDCDFCB8AE9"
Content-Disposition: inline
X-Virus-Checked: Checked by ClamAV on apache.org

--0__=0ABBFDDCDFCB8AE98f9e8a93df938690918c0ABBFDDCDFCB8AE9
Content-type: text/plain; charset=US-ASCII
Content-transfer-encoding: quoted-printable



I am out of the office until 07/12/2010.




Note: This is an automated response to your message  "how to get the
current input file in a Mapper" sent on 6/27/10 10:59:46.

This is the only notification you will receive while this person is awa=
y.=

--0__=0ABBFDDCDFCB8AE98f9e8a93df938690918c0ABBFDDCDFCB8AE9--


From general-return-1684-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 27 18:18:23 2010
Return-Path: <general-return-1684-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25458 invoked from network); 27 Jun 2010 18:18:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jun 2010 18:18:22 -0000
Received: (qmail 71556 invoked by uid 500); 27 Jun 2010 18:18:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71330 invoked by uid 500); 27 Jun 2010 18:18:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71322 invoked by uid 99); 27 Jun 2010 18:18:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Jun 2010 18:18:21 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Jun 2010 18:18:13 +0000
Received: by vws15 with SMTP id 15so5347142vws.35
        for <general@hadoop.apache.org>; Sun, 27 Jun 2010 11:17:52 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=VGBgwuJ5DiTDTNPOCIW1qi0xhSlWqdsYsGnyj5e6v5o=;
        b=R4xBKrDtzy5I0UnbxZyJbKzaNh997feJXP0bZSe3Heb717wYK6fja2oyVWsFeM0FsB
         LKODubOWkt8dX3qUeBF6FMY+VIVwihZrXOF1IGwUPp3K4EnPJ2Ioz7hwdnvPYuKHgKFg
         5vouPgatmu7YgNYtZ/oxoB+CdbskM3D/8cOlU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=C5bSM8YuJ9BNoiIjhK8kLzyBz9YmJKD7w31Ja9xH/0soX17M/uPm/CrY8lZr98mnGX
         zEREOOjvg04ZsU/Slb34fEB80GPxJfxNmORY9TABMB5o2yA09VAn++WkdMFtpZ/tqm/S
         gg9WXKKQtlz1ABruDgf91YcGpKgk+Qp87N6+U=
MIME-Version: 1.0
Received: by 10.229.249.138 with SMTP id mk10mr1985810qcb.229.1277662672427; 
	Sun, 27 Jun 2010 11:17:52 -0700 (PDT)
Received: by 10.229.212.209 with HTTP; Sun, 27 Jun 2010 11:17:52 -0700 (PDT)
In-Reply-To: <AANLkTilTKnEH2IA4XCyo_dbNnzED50hN9wIOQ-6IBU9s@mail.gmail.com>
References: <AANLkTilTKnEH2IA4XCyo_dbNnzED50hN9wIOQ-6IBU9s@mail.gmail.com>
Date: Sun, 27 Jun 2010 11:17:52 -0700
Message-ID: <AANLkTimk2pALHZloj6Z2yv-UnDGEC292FX6UKH49iBj5@mail.gmail.com>
Subject: Re: how to get the current input file in a Mapper
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364d3017bb9abb048a07058b
X-Virus-Checked: Checked by ClamAV on apache.org

--0016364d3017bb9abb048a07058b
Content-Type: text/plain; charset=ISO-8859-1

How about:
FileSplit fileSplit = (FileSplit) context.getInputSplit();
String sFileName = fileSplit.getPath().getName();


On Sun, Jun 27, 2010 at 7:59 AM, zhangguoping zhangguoping <
zhangguoping96@gmail.com> wrote:

> Hi,
>
> how to get the current input file in a Mapper in 0.20.2 ?
>
> Thanks,
>

--0016364d3017bb9abb048a07058b--

From general-return-1685-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 28 06:54:40 2010
Return-Path: <general-return-1685-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21513 invoked from network); 28 Jun 2010 06:54:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jun 2010 06:54:40 -0000
Received: (qmail 74164 invoked by uid 500); 28 Jun 2010 06:54:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73751 invoked by uid 500); 28 Jun 2010 06:54:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73743 invoked by uid 99); 28 Jun 2010 06:54:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Jun 2010 06:54:35 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of zhangguoping96@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Jun 2010 06:54:27 +0000
Received: by fxm11 with SMTP id 11so792398fxm.35
        for <general@hadoop.apache.org>; Sun, 27 Jun 2010 23:54:07 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=qSJ2ENodCPNgHTRiWkEoPbEPDFKBF9jaQZbJDsn+SkY=;
        b=F3uqEpkSh9K+MSQ72m1mEtp3H3s4BDnEku/VDQF2MUSKw0lW53fMGd99j63NxiGlmZ
         2zcpeZrIP+H7HAJLpo6mir53nPiK04ph9mI8EORTHQOiCbNnJJhZRiohGkUlAXl7P5rC
         HeWCQYyj25uOucJHlnM/v82x/V4fVUcNS5UFM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=vDwy/FLI5bXYevBoHPmKKtUsg72nMi6nL6+J9YQPakuGYtFyMJpm5LmJ0S5pSQVcdV
         KyH6FpXgODJyn85LoEPgC4tnJkqChwzCcjyxjtDZ3CJNKU+QuRqwlxbIpxqnJnUyhEV/
         4rSTZBhfHD6yu05h+xJtfY6rfZCgsfln1hoKs=
MIME-Version: 1.0
Received: by 10.239.180.19 with SMTP id f19mr273042hbg.91.1277708045579; Sun, 
	27 Jun 2010 23:54:05 -0700 (PDT)
Received: by 10.239.153.71 with HTTP; Sun, 27 Jun 2010 23:54:05 -0700 (PDT)
Date: Mon, 28 Jun 2010 14:54:05 +0800
Message-ID: <AANLkTilIsCTycfHPWLkAy8Y8TK2ezUwB6oFGEthcxTm5@mail.gmail.com>
Subject: MultipleInputs.addInputPath throws Exception
From: zhangguoping zhangguoping <zhangguoping96@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f5b10e2ef5c7048a11964f
X-Virus-Checked: Checked by ClamAV on apache.org

--001485f5b10e2ef5c7048a11964f
Content-Type: text/plain; charset=ISO-8859-1

Hi,

When I was calling following in hadoop 0.20.2,

org.apache.hadoop.mapred.lib.MultipleInputs.addInputPath();

I got following IOException:

"mapred.input.format.class is incompatible with new map API mode."


But looks like InputFormat class is mandatory for
MultipleInputs.addInputPath. How to correct it? Is there any alternatives?

Thanks,

--001485f5b10e2ef5c7048a11964f--

From general-return-1686-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 28 09:26:03 2010
Return-Path: <general-return-1686-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68219 invoked from network); 28 Jun 2010 09:26:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jun 2010 09:26:03 -0000
Received: (qmail 94112 invoked by uid 500); 28 Jun 2010 09:26:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93703 invoked by uid 500); 28 Jun 2010 09:25:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93692 invoked by uid 99); 28 Jun 2010 09:25:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Jun 2010 09:25:56 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Jun 2010 09:25:46 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 9867BB7DB8
	for <general@hadoop.apache.org>; Mon, 28 Jun 2010 10:25:25 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id c93vDtJBW4qn for <general@hadoop.apache.org>;
	Mon, 28 Jun 2010 10:25:25 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id CC0D0B7DB7
	for <general@hadoop.apache.org>; Mon, 28 Jun 2010 10:25:24 +0100 (BST)
MailScanner-NULL-Check: 1278321912.06161@Hij5j23TqksbSBoyG9JjGQ
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o5S9PAvr004691
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Mon, 28 Jun 2010 10:25:11 +0100 (BST)
Message-ID: <4C286A77.4090001@apache.org>
Date: Mon, 28 Jun 2010 10:25:11 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100228)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: The IBM Distribution of Apache Hadoop now available
References: <AANLkTikCu3pebRoKFVtuf069BxHM7I-cePj_XiWox_or@mail.gmail.com> <B40086F9-7A89-4925-911B-31881935CBC8@apache.org> <OFF68C4536.C7F1E4B5-ON8625774B.00558405-8625774B.005AA5CE@us.ibm.com>
In-Reply-To: <OFF68C4536.C7F1E4B5-ON8625774B.00558405-8625774B.005AA5CE@us.ibm.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o5S9PAvr004691
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

Stephen Watt wrote:
> Hi Folks
> 
> ??IBM has made available a preview version of the IBM Distribution of 
> Apache Hadoop. You can access it here - 
> http://www.alphaworks.ibm.com/tech/idah/ . This distribution contains 
> Apache Hadoop 0.20.2, a 32-bit Linux version of the IBM SDK for Java 6 SR 
> 8, and an easy to use web based wizard for the installation and 
> configuration of Hadoop.??

What scale have you tested that JVM at?

> 
> The best place for questions on this distribution are the threads under 
> the forum tab, accessible from the link provided above. We'll also be at 
> the Hadoop Summit.??
> 
> Regards
> ?Steve Watt
> 
> 


From general-return-1687-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 28 14:21:37 2010
Return-Path: <general-return-1687-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98691 invoked from network); 28 Jun 2010 14:21:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jun 2010 14:21:37 -0000
Received: (qmail 85887 invoked by uid 500); 28 Jun 2010 14:21:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85775 invoked by uid 500); 28 Jun 2010 14:21:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85767 invoked by uid 99); 28 Jun 2010 14:21:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Jun 2010 14:21:34 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of sarah.kho@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Jun 2010 14:21:27 +0000
Received: by wyb29 with SMTP id 29so4691000wyb.35
        for <general@hadoop.apache.org>; Mon, 28 Jun 2010 07:20:07 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=rZxGMoEbbFpLVsBwdwkiCxm+xH2n/FXoMEtnSZqq0LY=;
        b=RfWUJJBJ/pv6gSSKkad1zO6sK3xEdIvDkeo4SOiqaXaQh8r7cNGxik+XYMyVY2Men2
         +Qx9dxgWzvcVsWn/MaokcgYQNP1aH96kJ3CHAkvTcPdBqzJa+teR/US9iOwNUEcrwFfb
         M/JrJlmPQbGa+R2uNqO6g43TlOVZ/lhNJBTNg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=gAOUhEMdbzChT4cus33BFED2Nyjzz74bxEMbyFJtu8kI1WscGWgMWjqHmRH1Um/C/F
         tBm5tBbdiLj+miahb9kByRgxVm7UihriDEH8SOW1kCbJmZ5lyggrZhClFJG4Xnxr0FCp
         JW4kEPRxU2xpRNkE7/jvZms5jyLqg2G3BOP+U=
MIME-Version: 1.0
Received: by 10.216.91.12 with SMTP id g12mr3762860wef.78.1277734807055; Mon, 
	28 Jun 2010 07:20:07 -0700 (PDT)
Received: by 10.216.55.141 with HTTP; Mon, 28 Jun 2010 07:20:07 -0700 (PDT)
Date: Mon, 28 Jun 2010 18:50:07 +0430
Message-ID: <AANLkTimG4lm57Rwwqd_LW5Qdj95emkHhomBBKNgXTayi@mail.gmail.com>
Subject: Does hadoop need a zookeeper installation to work?
From: Sarah kho <sarah.kho@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d7e32a4aadf6048a17d14f
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d7e32a4aadf6048a17d14f
Content-Type: text/plain; charset=ISO-8859-1

Hi,

I am wondering whether hadoop depends on zookeeper or zookeeper need hadoop
to work.
When I download hadoop distribution from the website, does it have some
zookeeper libraries included?


thanks.

--0016e6d7e32a4aadf6048a17d14f--

From general-return-1688-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 28 14:54:48 2010
Return-Path: <general-return-1688-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6063 invoked from network); 28 Jun 2010 14:54:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jun 2010 14:54:47 -0000
Received: (qmail 30642 invoked by uid 500); 28 Jun 2010 14:54:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30588 invoked by uid 500); 28 Jun 2010 14:54:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30580 invoked by uid 99); 28 Jun 2010 14:54:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Jun 2010 14:54:46 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Jun 2010 14:54:38 +0000
Received: from [192.168.1.71] ([10.72.244.235])
	by mrout1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o5SErS0g068360
	for <general@hadoop.apache.org>; Mon, 28 Jun 2010 07:53:28 -0700 (PDT)
Message-Id: <B87285B3-0711-4422-B2BA-8850A63D6A2C@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <AANLkTimG4lm57Rwwqd_LW5Qdj95emkHhomBBKNgXTayi@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Does hadoop need a zookeeper installation to work?
Date: Mon, 28 Jun 2010 07:53:28 -0700
References: <AANLkTimG4lm57Rwwqd_LW5Qdj95emkHhomBBKNgXTayi@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

Currently, Hadoop (HDFS/Map-Reduce) doesn't depend on ZooKeeper.

Arun

On Jun 28, 2010, at 7:20 AM, Sarah kho wrote:

> Hi,
>
> I am wondering whether hadoop depends on zookeeper or zookeeper need  
> hadoop
> to work.
> When I download hadoop distribution from the website, does it have  
> some
> zookeeper libraries included?
>
>
> thanks.


From general-return-1689-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 28 15:29:54 2010
Return-Path: <general-return-1689-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21261 invoked from network); 28 Jun 2010 15:29:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jun 2010 15:29:54 -0000
Received: (qmail 82285 invoked by uid 500); 28 Jun 2010 15:29:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82195 invoked by uid 500); 28 Jun 2010 15:29:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82186 invoked by uid 99); 28 Jun 2010 15:29:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Jun 2010 15:29:52 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of sarah.kho@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Jun 2010 15:29:45 +0000
Received: by wyb29 with SMTP id 29so4788432wyb.35
        for <general@hadoop.apache.org>; Mon, 28 Jun 2010 08:28:25 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=qXlEdKVQ1Cl4ECdiGugVVt5I1gWVa1QJUvdkcJMhzV4=;
        b=x4ns/8fDT79CPy8EfaZA23EVqLGWvyK2tdB22OrCBTbQYHrCSO9nv+wqg+SYTFZO0B
         qJ0nZ4C74ScAC4yPmr/n5RSiQsbVzqucvqocHp8+1Y0BENdwJ1Ul1M6vZ+MUhJ/n4lfV
         ySMse7ucnmdV/rBrPTqCy0fowfSLGKVF1xCK0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=NDUQPauK04VCPSgAfreMPAda16vUXrhuue1t56lECRur7Lnp/YY1I1/hucS5zX+OOr
         ueRzmuqnbaSfRcS4atl6OFLmleu9VIpu2Wf/11NgBSpweKTJ3qWJPBW1t05nC4orepTX
         xM5dNfTT7cVKXpwQe9sw3y7XHKeD7fGMX+wzw=
MIME-Version: 1.0
Received: by 10.216.89.199 with SMTP id c49mr3859857wef.29.1277738905351; Mon, 
	28 Jun 2010 08:28:25 -0700 (PDT)
Received: by 10.216.55.141 with HTTP; Mon, 28 Jun 2010 08:28:25 -0700 (PDT)
In-Reply-To: <B87285B3-0711-4422-B2BA-8850A63D6A2C@yahoo-inc.com>
References: <AANLkTimG4lm57Rwwqd_LW5Qdj95emkHhomBBKNgXTayi@mail.gmail.com>
	<B87285B3-0711-4422-B2BA-8850A63D6A2C@yahoo-inc.com>
Date: Mon, 28 Jun 2010 19:58:25 +0430
Message-ID: <AANLkTilLJKYnj5qVQLc1WtLy_ffQzscbZt3go39Efddj@mail.gmail.com>
Subject: Re: Does hadoop need a zookeeper installation to work?
From: Sarah kho <sarah.kho@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d64c5b91ba57048a18c58b
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d64c5b91ba57048a18c58b
Content-Type: text/plain; charset=ISO-8859-1

Thanks for your reply.

What about ZooKeeper being depended  on Hadoop? is there something like that
or Hadoop and ZooKeeper are two independent projects?

On Mon, Jun 28, 2010 at 7:23 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:

> Currently, Hadoop (HDFS/Map-Reduce) doesn't depend on ZooKeeper.
>
> Arun
>
>
> On Jun 28, 2010, at 7:20 AM, Sarah kho wrote:
>
>  Hi,
>>
>> I am wondering whether hadoop depends on zookeeper or zookeeper need
>> hadoop
>> to work.
>> When I download hadoop distribution from the website, does it have some
>> zookeeper libraries included?
>>
>>
>> thanks.
>>
>
>

--0016e6d64c5b91ba57048a18c58b--

From general-return-1690-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 28 15:58:36 2010
Return-Path: <general-return-1690-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 28534 invoked from network); 28 Jun 2010 15:58:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jun 2010 15:58:35 -0000
Received: (qmail 26558 invoked by uid 500); 28 Jun 2010 15:58:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26498 invoked by uid 500); 28 Jun 2010 15:58:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26490 invoked by uid 99); 28 Jun 2010 15:58:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Jun 2010 15:58:33 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of michael_segel@hotmail.com designates 65.55.34.22 as permitted sender)
Received: from [65.55.34.22] (HELO col0-omc1-s12.col0.hotmail.com) (65.55.34.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Jun 2010 15:58:25 +0000
Received: from COL117-W63 ([65.55.34.8]) by col0-omc1-s12.col0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Mon, 28 Jun 2010 08:57:42 -0700
Message-ID: <COL117-W632283DC8F389E53740D3B8FCA0@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_44a04635-987c-49cc-bff0-fcd4d656f70f_"
X-Originating-IP: [65.167.11.254]
From: Michael Segel <michael_segel@hotmail.com>
To: <general@hadoop.apache.org>
Subject: RE: The IBM Distribution of Apache Hadoop now available
Date: Mon, 28 Jun 2010 10:57:42 -0500
Importance: Normal
In-Reply-To: <4C286A77.4090001@apache.org>
References: <AANLkTikCu3pebRoKFVtuf069BxHM7I-cePj_XiWox_or@mail.gmail.com>
 <B40086F9-7A89-4925-911B-31881935CBC8@apache.org>
 <OFF68C4536.C7F1E4B5-ON8625774B.00558405-8625774B.005AA5CE@us.ibm.com>,<4C286A77.4090001@apache.org>
MIME-Version: 1.0
X-OriginalArrivalTime: 28 Jun 2010 15:57:42.0596 (UTC) FILETIME=[A455D040:01CB16DA]
X-Virus-Checked: Checked by ClamAV on apache.org

--_44a04635-987c-49cc-bff0-fcd4d656f70f_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable




> Date: Mon=2C 28 Jun 2010 10:25:11 +0100
> From: stevel@apache.org
> To: general@hadoop.apache.org
> Subject: Re: The IBM Distribution of Apache Hadoop now available
>=20
> Stephen Watt wrote:
> > Hi Folks
> >=20
> > ??IBM has made available a preview version of the IBM Distribution of=20
> > Apache Hadoop. You can access it here -=20
> > http://www.alphaworks.ibm.com/tech/idah/ . This distribution contains=20
> > Apache Hadoop 0.20.2=2C a 32-bit Linux version of the IBM SDK for Java =
6 SR=20
> > 8=2C and an easy to use web based wizard for the installation and=20
> > configuration of Hadoop.??
>=20
> What scale have you tested that JVM at?
>=20

My question is why 32-bit? Why not 64?


> >=20
> > The best place for questions on this distribution are the threads under=
=20
> > the forum tab=2C accessible from the link provided above. We'll also be=
 at=20
> > the Hadoop Summit.??
> >=20

Considering that there are no posts in the fourm and I posted the first... =
:-(

I guess today is a travel day to the Hadoop Summit. Unfortunately not all o=
f us are in CA or can travel...

 		 	   		 =20
_________________________________________________________________
Hotmail is redefining busy with tools for the New Busy. Get more from your =
inbox.
http://www.windowslive.com/campaign/thenewbusy?ocid=3DPID28326::T:WLMTAGL:O=
N:WL:en-US:WM_HMP:042010_2=

--_44a04635-987c-49cc-bff0-fcd4d656f70f_--

From general-return-1691-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 28 16:35:32 2010
Return-Path: <general-return-1691-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43842 invoked from network); 28 Jun 2010 16:35:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jun 2010 16:35:32 -0000
Received: (qmail 75770 invoked by uid 500); 28 Jun 2010 16:35:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75672 invoked by uid 500); 28 Jun 2010 16:35:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75664 invoked by uid 99); 28 Jun 2010 16:35:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Jun 2010 16:35:30 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 28 Jun 2010 16:35:28 +0000
Received: (qmail 43683 invoked by uid 99); 28 Jun 2010 16:35:06 -0000
Received: from localhost.apache.org (HELO [10.0.0.164]) (127.0.0.1)
  (smtp-auth username phunt, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Jun 2010 16:35:06 +0000
Message-ID: <4C28CF3C.4050008@apache.org>
Date: Mon, 28 Jun 2010 09:35:08 -0700
From: Patrick Hunt <phunt@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.1.10) Gecko/20100512 Lightning/1.0b1 Thunderbird/3.0.5
MIME-Version: 1.0
To: general@hadoop.apache.org
CC: Sarah kho <sarah.kho@gmail.com>
Subject: Re: Does hadoop need a zookeeper installation to work?
References: <AANLkTimG4lm57Rwwqd_LW5Qdj95emkHhomBBKNgXTayi@mail.gmail.com>	<B87285B3-0711-4422-B2BA-8850A63D6A2C@yahoo-inc.com> <AANLkTilLJKYnj5qVQLc1WtLy_ffQzscbZt3go39Efddj@mail.gmail.com>
In-Reply-To: <AANLkTilLJKYnj5qVQLc1WtLy_ffQzscbZt3go39Efddj@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hbase depends on ZK but it recently moved to TLP status. As of today 
there are no cross dependencies.

Patrick

On 06/28/2010 08:28 AM, Sarah kho wrote:
> Thanks for your reply.
>
> What about ZooKeeper being depended  on Hadoop? is there something like that
> or Hadoop and ZooKeeper are two independent projects?
>
> On Mon, Jun 28, 2010 at 7:23 PM, Arun C Murthy<acm@yahoo-inc.com>  wrote:
>
>> Currently, Hadoop (HDFS/Map-Reduce) doesn't depend on ZooKeeper.
>>
>> Arun
>>
>>
>> On Jun 28, 2010, at 7:20 AM, Sarah kho wrote:
>>
>>   Hi,
>>>
>>> I am wondering whether hadoop depends on zookeeper or zookeeper need
>>> hadoop
>>> to work.
>>> When I download hadoop distribution from the website, does it have some
>>> zookeeper libraries included?
>>>
>>>
>>> thanks.
>>>
>>
>>
>

From general-return-1692-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 29 00:29:50 2010
Return-Path: <general-return-1692-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53248 invoked from network); 29 Jun 2010 00:29:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jun 2010 00:29:49 -0000
Received: (qmail 36345 invoked by uid 500); 29 Jun 2010 00:29:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36259 invoked by uid 500); 29 Jun 2010 00:29:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36250 invoked by uid 99); 29 Jun 2010 00:29:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 00:29:47 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eltonsky9404@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 00:29:39 +0000
Received: by qwd7 with SMTP id 7so1695989qwd.35
        for <general@hadoop.apache.org>; Mon, 28 Jun 2010 17:29:18 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=D2Ul31GMeqQC2v5Um464JdZBGwoU+YMdb92gUoRwVw0=;
        b=voqEANuOteVBne34qI/fFbZM5lCRT77gLKPgDUhU+Seydswv1L55Fb3MwHpjBEY99g
         YeIFeBUlq27ch2Ks2JLjkLT/FRa6fMOsixngbLlGEocP0N6dpBidFPYGxiYPEGR3zog5
         Nwiz+9Q7OO4/o6Y1RjmyNFHsEiHk3OcCHPfGg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=SjvUSKLk5ztyLz22Sv4SMIIbsd73fmDMc0EqvFDGn6b8pcZQTJeNUgBGo7HiUA9f11
         8W68foXbxTRts5xlvL7wIAiaRjnt0MfL0jQ9eruzZqEmT+5DvNqAzSYfy/nllGrG9jqi
         Jbn69hpRgHmb4NjKu3cbIXFsl4tcFJyjXS8zI=
MIME-Version: 1.0
Received: by 10.224.61.69 with SMTP id s5mr3957723qah.189.1277771358504; Mon, 
	28 Jun 2010 17:29:18 -0700 (PDT)
Received: by 10.224.67.6 with HTTP; Mon, 28 Jun 2010 17:29:18 -0700 (PDT)
Date: Tue, 29 Jun 2010 10:29:18 +1000
Message-ID: <AANLkTil1b6fgdChEJO5t17-OcTVl2htVJ__Vpdf2D_L0@mail.gmail.com>
Subject: Can we modify files in HDFS?
From: elton sky <eltonsky9404@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485172205ed88bc048a205335
X-Virus-Checked: Checked by ClamAV on apache.org

--001485172205ed88bc048a205335
Content-Type: text/plain; charset=ISO-8859-1

hello everyone,

After some research I found HDFS only support create new file and append to
exiting file. What if I want to modify some parts of a, say 2 Petabyte,
file.
Do I have to remove it and create it again or we have some alternative way?

--001485172205ed88bc048a205335--

From general-return-1693-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 29 00:53:43 2010
Return-Path: <general-return-1693-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56427 invoked from network); 29 Jun 2010 00:53:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jun 2010 00:53:43 -0000
Received: (qmail 56246 invoked by uid 500); 29 Jun 2010 00:53:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56193 invoked by uid 500); 29 Jun 2010 00:53:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56185 invoked by uid 99); 29 Jun 2010 00:53:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 00:53:41 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 00:53:35 +0000
Received: from [127.0.0.1] (gentlepaint-lx.corp.yahoo.com [10.72.185.127])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o5T0of82061006
	for <general@hadoop.apache.org>; Mon, 28 Jun 2010 17:50:44 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=WYx7qkwgqsVLeVutyzM0NoGti/DnrMl4Sma152t44pwzb1zCd8ljSatu70fR3of5
Message-ID: <4C294361.5090206@yahoo-inc.com>
Date: Mon, 28 Jun 2010 17:50:41 -0700
From: Konstantin Shvachko <shv@yahoo-inc.com>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.10) Gecko/20100512 Thunderbird/3.0.5
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSSION] Proposal for making core Hadoop changes
References: <AANLkTimLB-YNWnMwa93YL8Lpg9hJLxrCKbYL3YUr6de8@mail.gmail.com>
In-Reply-To: <AANLkTimLB-YNWnMwa93YL8Lpg9hJLxrCKbYL3YUr6de8@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Eli,

Just checking on the status of this proposal.

In the past I was hesitant about introducing more formalities.
I now think we really need some mechanism for
new feature and project proposals, also tracking decisions.
For the reasons exactly as you describe in your email.
Whether it is going to be HEP or something else, it is best
if we adopt it soon.

Thanks,
--Konstantin


On 5/21/2010 1:42 PM, Eli Collins wrote:
> As HDFS and MapReduce have matured the cost and complexity of
> introducing features has grown. Each new feature has to consider
> interactions with a growing set of existing features, a growing user
> base (upgrades, backwards compatibility) and additional use cases
> (more and more projects now build on them). At the same time we don't
> want the high bar for contribution to unnecessarily hinder new
> development and releases.
>
> Many projects at a similar stage address this by adopting a more
> formal way to describe, socialize and shepherd enhancements to their
> platforms. Today, new features are often discussed via an umbrella
> jira, which may have an attached design document. There are a number
> of issues with this approach. The design documents vary in format and
> quality, and are often reviewed by a limited audience. They aren't
> version controlled. Sometimes the proposal is only partially
> specified. Jiras are often ignored. Understanding a proposal and it's
> implications through a series of threads in the jira comments is
> difficult. It's hard for contributors and users to find these
> top-level jiras and follow their status.
>
> I'd like to propose that core Hadoop adopts something similar to
> Python's PEP (Python Enhancement Proposal) [1]. A "HEP" would be a
> single primary mechanism for proposing new features, incorporating
> community feedback, and recording decisions. The author of the HEP
> would be responsible for building consensus and moving the feature
> forward. Similarly, some subset of the community would be responsible
> for reviewing HEPs in a timely manner and identifying missing pieces
> in the proposal. Discussion would occur before patches showed up on
> jira. People interested in the core Hadoop roadmap could keep an eye
> on the HEPs without the overhead of following jira traffic.
>
> Why base this on the PEP? The format has proven useful to a
> substantial existing project, and I think the workflow is not too
> heavy-weight, and well-suited to a community such as ours. That being
> said, we could discuss other models (eg Java's JSR).
>
> Before we get into specifics, is this something the community would
> like to adopt in some form? Does adapting the PEP and its workflow to
> our projects, community and bylaws seem reasonable?
>
> Thanks,
> Eli
>
> 1. http://www.python.org/dev/peps/pep-0001
>


From general-return-1694-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 29 01:52:30 2010
Return-Path: <general-return-1694-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69408 invoked from network); 29 Jun 2010 01:52:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jun 2010 01:52:30 -0000
Received: (qmail 87523 invoked by uid 500); 29 Jun 2010 01:52:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87461 invoked by uid 500); 29 Jun 2010 01:52:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87453 invoked by uid 99); 29 Jun 2010 01:52:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 01:52:28 +0000
X-ASF-Spam-Status: No, hits=0.6 required=10.0
	tests=FREEMAIL_FROM,HK_RANDOM_ENVFROM,HK_RANDOM_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of zjffdu@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 01:52:20 +0000
Received: by pwj1 with SMTP id 1so117772pwj.35
        for <general@hadoop.apache.org>; Mon, 28 Jun 2010 18:50:59 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=dtNhndlspBaDFmyd0RO/NM+CfGPhwn3/tpf91eeoQUY=;
        b=I4+6Viy8C3DIOaJavV+0th58XrcqP2GGhpgWC8Q7RGsjC2NAKiiMESB4hGiecOm/Ko
         n46MWujW42h7fNW8fIMczw/Q1bzNsQfyhE9ouI2500o7hJBf2LKyEEZPwpXYAa6CoebC
         yTIKsonqTviWR1dmY7pIw41J+FuzBMjyBhB7c=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=FBMB04xdhRArhnL+E28HHM1M8HZyT7fvIErP+XzWolAkwTMN2+7S2V5IgM+Q6/gwcN
         tPIa7gvIHFp2tmjT5e+psVE3YHLlSXmSHBoMrnMIpX3x1nWUASFuWg8bkma6rhl9RxEu
         +6VtsssdGMWM7d0fsZ2USShf81VR8LmLceaTg=
MIME-Version: 1.0
Received: by 10.142.7.13 with SMTP id 13mr6910264wfg.122.1277776259509; Mon, 
	28 Jun 2010 18:50:59 -0700 (PDT)
Received: by 10.142.231.15 with HTTP; Mon, 28 Jun 2010 18:50:59 -0700 (PDT)
In-Reply-To: <AANLkTil1b6fgdChEJO5t17-OcTVl2htVJ__Vpdf2D_L0@mail.gmail.com>
References: <AANLkTil1b6fgdChEJO5t17-OcTVl2htVJ__Vpdf2D_L0@mail.gmail.com>
Date: Tue, 29 Jun 2010 09:50:59 +0800
Message-ID: <AANLkTimkK7rLbj42lxluTOeqyHAug5_bssuGN5C1jtda@mail.gmail.com>
Subject: Re: Can we modify files in HDFS?
From: Jeff Zhang <zjffdu@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
X-Virus-Checked: Checked by ClamAV on apache.org

You can not modify file in HDFS, HDFS is designed for write once, read
many times.



On Tue, Jun 29, 2010 at 8:29 AM, elton sky <eltonsky9404@gmail.com> wrote:
> hello everyone,
>
> After some research I found HDFS only support create new file and append to
> exiting file. What if I want to modify some parts of a, say 2 Petabyte,
> file.
> Do I have to remove it and create it again or we have some alternative way?
>



-- 
Best Regards

Jeff Zhang

From general-return-1695-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 29 03:05:28 2010
Return-Path: <general-return-1695-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94827 invoked from network); 29 Jun 2010 03:05:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jun 2010 03:05:28 -0000
Received: (qmail 19511 invoked by uid 500); 29 Jun 2010 03:05:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19224 invoked by uid 500); 29 Jun 2010 03:05:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19216 invoked by uid 99); 29 Jun 2010 03:05:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 03:05:24 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.212.176] (HELO mail-px0-f176.google.com) (209.85.212.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 03:05:18 +0000
Received: by pxi11 with SMTP id 11so932713pxi.35
        for <general@hadoop.apache.org>; Mon, 28 Jun 2010 20:03:57 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.142.201.15 with SMTP id y15mr6966095wff.247.1277780635288; 
	Mon, 28 Jun 2010 20:03:55 -0700 (PDT)
Received: by 10.142.110.10 with HTTP; Mon, 28 Jun 2010 20:03:55 -0700 (PDT)
In-Reply-To: <4C294361.5090206@yahoo-inc.com>
References: <AANLkTimLB-YNWnMwa93YL8Lpg9hJLxrCKbYL3YUr6de8@mail.gmail.com>
	<4C294361.5090206@yahoo-inc.com>
Date: Mon, 28 Jun 2010 20:03:55 -0700
Message-ID: <AANLkTik-2g7gl0QJY0f-i1PAdTKsKc1sJMm6WqQ1-Z61@mail.gmail.com>
Subject: Re: [DISCUSSION] Proposal for making core Hadoop changes
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hey Konstantin,

Apologies for the delay, busy with stuff for the summit.  I'll get a
concrete proposal to general based on our discussion at the
contributor's meeting out this week.

Thanks,
Eli

On Mon, Jun 28, 2010 at 5:50 PM, Konstantin Shvachko <shv@yahoo-inc.com> wrote:
> Eli,
>
> Just checking on the status of this proposal.
>
> In the past I was hesitant about introducing more formalities.
> I now think we really need some mechanism for
> new feature and project proposals, also tracking decisions.
> For the reasons exactly as you describe in your email.
> Whether it is going to be HEP or something else, it is best
> if we adopt it soon.
>
> Thanks,
> --Konstantin
>
>
> On 5/21/2010 1:42 PM, Eli Collins wrote:
>>
>> As HDFS and MapReduce have matured the cost and complexity of
>> introducing features has grown. Each new feature has to consider
>> interactions with a growing set of existing features, a growing user
>> base (upgrades, backwards compatibility) and additional use cases
>> (more and more projects now build on them). At the same time we don't
>> want the high bar for contribution to unnecessarily hinder new
>> development and releases.
>>
>> Many projects at a similar stage address this by adopting a more
>> formal way to describe, socialize and shepherd enhancements to their
>> platforms. Today, new features are often discussed via an umbrella
>> jira, which may have an attached design document. There are a number
>> of issues with this approach. The design documents vary in format and
>> quality, and are often reviewed by a limited audience. They aren't
>> version controlled. Sometimes the proposal is only partially
>> specified. Jiras are often ignored. Understanding a proposal and it's
>> implications through a series of threads in the jira comments is
>> difficult. It's hard for contributors and users to find these
>> top-level jiras and follow their status.
>>
>> I'd like to propose that core Hadoop adopts something similar to
>> Python's PEP (Python Enhancement Proposal) [1]. A "HEP" would be a
>> single primary mechanism for proposing new features, incorporating
>> community feedback, and recording decisions. The author of the HEP
>> would be responsible for building consensus and moving the feature
>> forward. Similarly, some subset of the community would be responsible
>> for reviewing HEPs in a timely manner and identifying missing pieces
>> in the proposal. Discussion would occur before patches showed up on
>> jira. People interested in the core Hadoop roadmap could keep an eye
>> on the HEPs without the overhead of following jira traffic.
>>
>> Why base this on the PEP? The format has proven useful to a
>> substantial existing project, and I think the workflow is not too
>> heavy-weight, and well-suited to a community such as ours. That being
>> said, we could discuss other models (eg Java's JSR).
>>
>> Before we get into specifics, is this something the community would
>> like to adopt in some form? Does adapting the PEP and its workflow to
>> our projects, community and bylaws seem reasonable?
>>
>> Thanks,
>> Eli
>>
>> 1. http://www.python.org/dev/peps/pep-0001
>>
>
>

From general-return-1696-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 29 04:50:16 2010
Return-Path: <general-return-1696-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12121 invoked from network); 29 Jun 2010 04:50:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jun 2010 04:50:16 -0000
Received: (qmail 76187 invoked by uid 500); 29 Jun 2010 04:50:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75824 invoked by uid 500); 29 Jun 2010 04:50:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75815 invoked by uid 99); 29 Jun 2010 04:50:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 04:50:12 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eltonsky9404@gmail.com designates 209.85.216.51 as permitted sender)
Received: from [209.85.216.51] (HELO mail-qw0-f51.google.com) (209.85.216.51)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 04:50:05 +0000
Received: by qwk3 with SMTP id 3so1571072qwk.38
        for <general@hadoop.apache.org>; Mon, 28 Jun 2010 21:48:55 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=1jF6aHG1LSPkzWpAvC9cAbdCmQop/FlgSHOm4mbbV30=;
        b=tygV4VVFyW9PJTtS7yihJNsOcXdTJJd+wRCdtFJULOTjiPqHsEQUBBni0HpLsen59Q
         Ks+SS0HcoBn27i3BPgXR9Qeo25pDtthNpTHi/thfvaUNeMpgOP9VUWVfZN5TxycL3W8J
         2gsP9AubgLPQ7GYSPtA5Bx+hzun53Z0rV97kE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=fU58xmKHuK2vlOTADibJbgCHwDSE2BZn9DW/Gf58RO/lyjqgvTGSja0JN1IfwrdgfO
         CKKO5RXwfeZlvsnGE5gwQy5VL9fFhXdaaR97+igOZ+WCPdPaMEAh91sWsllgtYd18CWf
         Y+rvD0LME67spXYVUod9MiawniAF8YNX01n/Y=
MIME-Version: 1.0
Received: by 10.224.113.201 with SMTP id b9mr4083202qaq.66.1277786935310; Mon, 
	28 Jun 2010 21:48:55 -0700 (PDT)
Received: by 10.224.67.6 with HTTP; Mon, 28 Jun 2010 21:48:55 -0700 (PDT)
In-Reply-To: <AANLkTimkK7rLbj42lxluTOeqyHAug5_bssuGN5C1jtda@mail.gmail.com>
References: <AANLkTil1b6fgdChEJO5t17-OcTVl2htVJ__Vpdf2D_L0@mail.gmail.com>
	<AANLkTimkK7rLbj42lxluTOeqyHAug5_bssuGN5C1jtda@mail.gmail.com>
Date: Tue, 29 Jun 2010 14:48:55 +1000
Message-ID: <AANLkTin0LUFfRsx-m0MbtNRIcbsv8gz0luVd5UAGdYig@mail.gmail.com>
Subject: Re: Can we modify files in HDFS?
From: elton sky <eltonsky9404@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00c09f99de4160b9b4048a23f471
X-Virus-Checked: Checked by ClamAV on apache.org

--00c09f99de4160b9b4048a23f471
Content-Type: text/plain; charset=ISO-8859-1

thanx Jeff,

So...it is a significant drawback.
As a matter of fact, there are many cases we need to modify.
I dont understand why Yahoo didn't provoid that functionality. And as I know
no one else is working on this. Why is that?

--00c09f99de4160b9b4048a23f471--

From general-return-1697-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 29 05:03:51 2010
Return-Path: <general-return-1697-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14107 invoked from network); 29 Jun 2010 05:03:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jun 2010 05:03:51 -0000
Received: (qmail 89678 invoked by uid 500); 29 Jun 2010 05:03:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89433 invoked by uid 500); 29 Jun 2010 05:03:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89425 invoked by uid 99); 29 Jun 2010 05:03:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 05:03:46 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 05:03:38 +0000
Received: by iwn38 with SMTP id 38so7433609iwn.35
        for <general@hadoop.apache.org>; Mon, 28 Jun 2010 22:03:16 -0700 (PDT)
Received: by 10.231.150.2 with SMTP id w2mr6210294ibv.37.1277787795268; Mon, 
	28 Jun 2010 22:03:15 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.161.72 with HTTP; Mon, 28 Jun 2010 22:02:55 -0700 (PDT)
In-Reply-To: <AANLkTin0LUFfRsx-m0MbtNRIcbsv8gz0luVd5UAGdYig@mail.gmail.com>
References: <AANLkTil1b6fgdChEJO5t17-OcTVl2htVJ__Vpdf2D_L0@mail.gmail.com> 
	<AANLkTimkK7rLbj42lxluTOeqyHAug5_bssuGN5C1jtda@mail.gmail.com> 
	<AANLkTin0LUFfRsx-m0MbtNRIcbsv8gz0luVd5UAGdYig@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 28 Jun 2010 22:02:55 -0700
Message-ID: <AANLkTikRA6SBqj2yIJG1dF_VlxyL9YtlTDiMIIrm3uNh@mail.gmail.com>
Subject: Re: Can we modify files in HDFS?
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636ed629fa2a6dd048a2427ad
X-Virus-Checked: Checked by ClamAV on apache.org

--001636ed629fa2a6dd048a2427ad
Content-Type: text/plain; charset=ISO-8859-1

Hi Elton,

Typically, large data sets are of the sort that continuously grow, and are
not edited or amended. For example, a common Hadoop use case is the analysis
of log data or other instrumentation from web or application servers. In
these cases, files are simply added, but there is no need to go back and
change entries.

For the ability to have a more table-like random access storage on top of
Hadoop, I would encourage you to look into HBase. It supports random
read/write access with low latency.

-Todd

On Mon, Jun 28, 2010 at 9:48 PM, elton sky <eltonsky9404@gmail.com> wrote:

> thanx Jeff,
>
> So...it is a significant drawback.
> As a matter of fact, there are many cases we need to modify.
> I dont understand why Yahoo didn't provoid that functionality. And as I
> know
> no one else is working on this. Why is that?
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--001636ed629fa2a6dd048a2427ad--

From general-return-1698-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 29 09:57:51 2010
Return-Path: <general-return-1698-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 3023 invoked from network); 29 Jun 2010 09:57:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jun 2010 09:57:50 -0000
Received: (qmail 65443 invoked by uid 500); 29 Jun 2010 09:57:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65299 invoked by uid 500); 29 Jun 2010 09:57:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65290 invoked by uid 99); 29 Jun 2010 09:57:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 09:57:46 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 09:57:36 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id B3E001BA657
	for <general@hadoop.apache.org>; Tue, 29 Jun 2010 10:57:15 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id cJ8x+vrwV2J6 for <general@hadoop.apache.org>;
	Tue, 29 Jun 2010 10:57:15 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by colossus.hpl.hp.com (Postfix) with ESMTPS id 406C71BA5EB
	for <general@hadoop.apache.org>; Tue, 29 Jun 2010 10:57:15 +0100 (BST)
MailScanner-NULL-Check: 1278410222.76672@HuA+Wxde/Ug9BQgu1COKkQ
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o5T9v1bw002567
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Tue, 29 Jun 2010 10:57:02 +0100 (BST)
Message-ID: <4C29C36D.2030806@apache.org>
Date: Tue, 29 Jun 2010 10:57:01 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100228)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Can we modify files in HDFS?
References: <AANLkTil1b6fgdChEJO5t17-OcTVl2htVJ__Vpdf2D_L0@mail.gmail.com>	<AANLkTimkK7rLbj42lxluTOeqyHAug5_bssuGN5C1jtda@mail.gmail.com> <AANLkTin0LUFfRsx-m0MbtNRIcbsv8gz0luVd5UAGdYig@mail.gmail.com>
In-Reply-To: <AANLkTin0LUFfRsx-m0MbtNRIcbsv8gz0luVd5UAGdYig@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o5T9v1bw002567
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

elton sky wrote:
> thanx Jeff,
> 
> So...it is a significant drawback.
> As a matter of fact, there are many cases we need to modify.


When people say "Hadoop filesystems are not posix", this is what they 
mean. No locks, no read/write. seeking discouraged. Even append is 
something that is just stabilising. to be fair though, even NFS is 
quirky and that's been around since Ether-net was considered so cutting 
edge it had a hyphen in the title.

HDFS delivers availability through redundant copies across multiple 
machines: you can read your data on or near any machine with a copy of 
the data. Think what you'd need for full seek and read/write actions


* seek would kill bulk IO perf on classic rotating-disk HDDs, and nobody 
can afford to build a petabyte filestore out of SSDs yet. You should be 
streaming, not seeking.

* to do writes, you'd need to lock out access to the files, which 
implies a distributed lock infrastructure (zookeeper?) or deal with 
conflicting writes.

* if you want immediate update writes you'd need to push out the changes 
to the (existing) nodes, and deal with queueing up pending changes to 
machines that are currently offline in a way that I don't want to think 
about.

* if you want slower-update writes (eventual consistency), then things 
may be slightly simpler -you'd need a lock on writing and each write 
would eventually be pushed out to the readers with a bit better 
bandwidth and CPU scheduling flexibility , but there's still that 
offline node problem. If a node that was down comes up, how does it know 
it's data is out of date and where does it get the data from? What will 
it do if all other nodes that have updated data are offline.


 > I dont understand why Yahoo didn't provoid that functionality. And as 
I know
 > no one else is working on this. Why is that?

It's because it scares us and we are happier writing code to live in a 
world where you don't seek and patch files, but instead add new data and 
delete old stuff. I don't know what the Cassandra and HBase teams do here.

-steve





From general-return-1699-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 29 15:07:35 2010
Return-Path: <general-return-1699-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 10933 invoked from network); 29 Jun 2010 15:07:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jun 2010 15:07:35 -0000
Received: (qmail 2149 invoked by uid 500); 29 Jun 2010 15:07:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2102 invoked by uid 500); 29 Jun 2010 15:07:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2094 invoked by uid 99); 29 Jun 2010 15:07:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 15:07:33 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of sarah.kho@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 15:07:25 +0000
Received: by wwi14 with SMTP id 14so5517741wwi.35
        for <general@hadoop.apache.org>; Tue, 29 Jun 2010 08:07:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=Nkvf2chIW6on2gyQG95S7/NMEmX7GVtHW0BcXvYjpkw=;
        b=muhYZpZNMYwoL4o86yUUIK/4NRR2IQsHBZUBvYP2uy81Asi811QC7+Jco4GTs0bqdT
         cdiY8x3H9NrG3su9mGTh6c9i72Po/mkMBUspj+e2TKQJKaQgltcEqGvfRtRiZxJW5x6t
         F9yoSFyQNx96ZeOsjhaYIuYLsyDoFtURoJ+nQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=bY4jRFQSj/Wv1xaZRApLQt3V7mveJPJHy4ZPXIJrVPaGVfDoigpYva2DzkkbLrKgkT
         oP5HsSER/p6BK5JxbPmpdtOId0rEdo2cMEv8KZXbMa8duEkq36liPsXh7j9ATmHzoHcK
         W1QBWSHYz+PwPOlT2MB/fgG8nFijmxsgE176g=
MIME-Version: 1.0
Received: by 10.216.170.83 with SMTP id o61mr5263312wel.37.1277824025356; Tue, 
	29 Jun 2010 08:07:05 -0700 (PDT)
Received: by 10.216.55.141 with HTTP; Tue, 29 Jun 2010 08:07:05 -0700 (PDT)
Date: Tue, 29 Jun 2010 19:37:05 +0430
Message-ID: <AANLkTikITUTjiAljgzNTxdnFxHhRdoJwOyuyCk59mgZe@mail.gmail.com>
Subject: What are uses of taskTracker and JobTracker services?
From: Sarah kho <sarah.kho@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f423d41de87a048a2c97ad
X-Virus-Checked: Checked by ClamAV on apache.org

--001485f423d41de87a048a2c97ad
Content-Type: text/plain; charset=ISO-8859-1

Hi,

Can you please let me know what are tasks that the taskTracker and
JobTracker performs?

Thanks.

--001485f423d41de87a048a2c97ad--

From general-return-1700-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 29 15:30:35 2010
Return-Path: <general-return-1700-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 15341 invoked from network); 29 Jun 2010 15:30:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jun 2010 15:30:34 -0000
Received: (qmail 46670 invoked by uid 500); 29 Jun 2010 15:30:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46619 invoked by uid 500); 29 Jun 2010 15:30:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46611 invoked by uid 99); 29 Jun 2010 15:30:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 15:30:32 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of bernd.fondermann@googlemail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 15:30:27 +0000
Received: by wwi14 with SMTP id 14so5551300wwi.35
        for <general@hadoop.apache.org>; Tue, 29 Jun 2010 08:29:06 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=6OhOqlPY3cktGe0KlGrf9WzQArsDZpF/Rc4P3GtGqVE=;
        b=DKU58igQRVKxY9xD/bq8gvRgxchHkemMZbAjJyuYnxv2TaS83LYW3kYiYdnSOdEgHA
         90HUkhwu3MKi8rZaxPckcMvreIzIS9HMvYOL8IbLMO7SDQZODKeHX7dd4aJ0zLYuslJd
         W5yskINSlDbyTEc2R9a0pjKHt+QrFyNrWJxt8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=CarF/9kawWiRbaCwKJRcW66+7rRllUZ8lD09jplqrvFbpTs5hD/IvPrS7TzrqmFIrv
         D+P/wyoLNowZmOVxlvRcyqQlPXIcW+rmUzNxYH9jczaFVLYAskaOa1DEyJQsjXhu2Xpe
         Bzc9VAWOuexnboWzYAmgTTlMSsE5Nhtnbdnug=
MIME-Version: 1.0
Received: by 10.227.141.137 with SMTP id m9mr5421312wbu.202.1277825346699; 
	Tue, 29 Jun 2010 08:29:06 -0700 (PDT)
Received: by 10.216.162.6 with HTTP; Tue, 29 Jun 2010 08:29:06 -0700 (PDT)
In-Reply-To: <4C294361.5090206@yahoo-inc.com>
References: <AANLkTimLB-YNWnMwa93YL8Lpg9hJLxrCKbYL3YUr6de8@mail.gmail.com>
	<4C294361.5090206@yahoo-inc.com>
Date: Tue, 29 Jun 2010 17:29:06 +0200
Message-ID: <AANLkTinf7xOzGkzBphh8rPQN8uMS3TMd2rCwnZHgSC6N@mail.gmail.com>
Subject: Re: [DISCUSSION] Proposal for making core Hadoop changes
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Tue, Jun 29, 2010 at 02:50, Konstantin Shvachko <shv@yahoo-inc.com> wrote:
> Eli,
>
> Just checking on the status of this proposal.
>
> In the past I was hesitant about introducing more formalities.
> I now think we really need some mechanism for
> new feature and project proposals, also tracking decisions.

Making and tracking decisions at Apache is done via public ASF mailing
lists, exclusively.
Any other means of communication, including face-to-face, JIRA, IRC
etc, is *not binding*.
Every community member has equal say (only PMC members votes are
binding though).
Committers can veto commits and commit to svn. PMC members have
special rights and duties, too, as described in our Bylaws.

That's about it.

If Hadoop has issues tracking and making decisions, you won't fix that
by introducing any formalities.

  Bernd

From general-return-1701-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 29 17:12:22 2010
Return-Path: <general-return-1701-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47430 invoked from network); 29 Jun 2010 17:12:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jun 2010 17:12:22 -0000
Received: (qmail 90037 invoked by uid 500); 29 Jun 2010 17:12:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89927 invoked by uid 500); 29 Jun 2010 17:12:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89916 invoked by uid 99); 29 Jun 2010 17:12:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 17:12:20 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jaybooth@gmail.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 17:12:14 +0000
Received: by gyg10 with SMTP id 10so1311836gyg.35
        for <general@hadoop.apache.org>; Tue, 29 Jun 2010 10:11:53 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=5iZQNe8iaw5hTZmBxfUEecU3O5khckpdECxaXq10YE8=;
        b=mi3x6AHiA9ExbAudwhJkZL8o9fjPHmAFIUwEjN47cjVo85hp/Q+mcoPCG+jA6NWa8m
         JyfkSOqrspoOgkEnGkQrofQk1N9VADzRld+Lz4oMXszykSnJT/oR1mvpSsK2kEYIDD/u
         7oKZOEO8UqJ40baceO9TjWbjn4vhU77ZoPYvo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=Vrgam/HPYQ2TS25T+1IjiJhcUpSFPlN6GdZFJtJlAbiSHcB3x8balIYmXPHSw9ImNX
         jTPwddHDoJpXgUMubCWzj5wz45HPrkaLL0wuJ7h3wPyGdxxoaHRR45dBtt9962is21LP
         hkoXOcxtJkx3QgDUqMeAMWG0OjNXjsBc+n678=
MIME-Version: 1.0
Received: by 10.224.79.2 with SMTP id n2mr4693221qak.103.1277831498803; Tue, 
	29 Jun 2010 10:11:38 -0700 (PDT)
Received: by 10.229.21.12 with HTTP; Tue, 29 Jun 2010 10:11:38 -0700 (PDT)
In-Reply-To: <AANLkTinf7xOzGkzBphh8rPQN8uMS3TMd2rCwnZHgSC6N@mail.gmail.com>
References: <AANLkTimLB-YNWnMwa93YL8Lpg9hJLxrCKbYL3YUr6de8@mail.gmail.com>
	<4C294361.5090206@yahoo-inc.com>
	<AANLkTinf7xOzGkzBphh8rPQN8uMS3TMd2rCwnZHgSC6N@mail.gmail.com>
Date: Tue, 29 Jun 2010 13:11:38 -0400
Message-ID: <AANLkTikevCdeRR42LcSFz2C8DRGF-inGEE7blrKyvrQr@mail.gmail.com>
Subject: Re: [DISCUSSION] Proposal for making core Hadoop changes
From: Jay Booth <jaybooth@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Well, if people decide that some system more organized than email
threads is better to keep track of major project proposals, it may
help with some aspects of the project.  Or it may not.  The fact that
a pro forma vote by email may also be required at some points to make
something "official" shouldn't be a major reason against such a
system, if it's otherwise a good idea.

On Tue, Jun 29, 2010 at 11:29 AM, Bernd Fondermann
<bernd.fondermann@googlemail.com> wrote:
> On Tue, Jun 29, 2010 at 02:50, Konstantin Shvachko <shv@yahoo-inc.com> wr=
ote:
>> Eli,
>>
>> Just checking on the status of this proposal.
>>
>> In the past I was hesitant about introducing more formalities.
>> I now think we really need some mechanism for
>> new feature and project proposals, also tracking decisions.
>
> Making and tracking decisions at Apache is done via public ASF mailing
> lists, exclusively.
> Any other means of communication, including face-to-face, JIRA, IRC
> etc, is *not binding*.
> Every community member has equal say (only PMC members votes are
> binding though).
> Committers can veto commits and commit to svn. PMC members have
> special rights and duties, too, as described in our Bylaws.
>
> That's about it.
>
> If Hadoop has issues tracking and making decisions, you won't fix that
> by introducing any formalities.
>
> =A0Bernd
>

From general-return-1702-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 29 17:51:01 2010
Return-Path: <general-return-1702-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57716 invoked from network); 29 Jun 2010 17:51:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jun 2010 17:51:01 -0000
Received: (qmail 29942 invoked by uid 500); 29 Jun 2010 17:51:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29861 invoked by uid 500); 29 Jun 2010 17:50:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29852 invoked by uid 99); 29 Jun 2010 17:50:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 17:50:59 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 17:50:53 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version;
  b=P5gLPY4fUgbbqCb4TqPb2emTbfdzYw5Nr+kk/3aO2bNTTBXj1IJJcbp3
   h8GbIVTWj91UbnKHgb6hdtajE/fuxoTFM59B/1OpmIwhkp5YlKQWLEA5C
   RxlgMAXNz8qAXrn;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1277833853; x=1309369853;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20What=20are=20uses=20of=20taskTracker=20
   and=20JobTracker=20services?|Date:=20Tue,=2029=20Jun=2020
   10=2017:50:30=20+0000|Message-ID:=20<D6325871-64CA-4C29-9
   A8A-27B1A78B10C7@linkedin.com>|To:=20"<general@hadoop.apa
   che.org>"=20<general@hadoop.apache.org>|MIME-Version:=201
   .0|Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<138eef54-9b03-4394-b2dc-15aca04d3d20>
   |In-Reply-To:=20<AANLkTikITUTjiAljgzNTxdnFxHhRdoJwOyuyCk5
   9mgZe@mail.gmail.com>|References:=20<AANLkTikITUTjiAljgzN
   TxdnFxHhRdoJwOyuyCk59mgZe@mail.gmail.com>;
  bh=Q0zBiuc+LXt7k9Yp1zqlt+N5NYHHsJJXAtHpLPb23io=;
  b=tMFhqvo7mOB75fRTDJQ7kDVtB7XEa3M5TFsBQo39RmMP3tWJ8CJKUBvp
   42h3a8s9koY5LtIWgSLdg53vUXQfUrfYDL1Vke5JROX1kjL7nryYPFhW4
   o6iY/RvO4O89iBC;
X-IronPort-AV: E=Sophos;i="4.53,506,1272870000"; 
   d="scan'208";a="13743469"
Received: from ESV4-EXC02.linkedin.biz ([fe80::4d74:48bd:e0bd:13ee]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi; Tue, 29 Jun 2010
 10:50:32 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: What are uses of taskTracker and JobTracker services?
Thread-Topic: What are uses of taskTracker and JobTracker services?
Thread-Index: AQHLF5zOfCQ9ryMITkaP1GURVVAGdpKZrUAA
Date: Tue, 29 Jun 2010 17:50:30 +0000
Message-ID: <D6325871-64CA-4C29-9A8A-27B1A78B10C7@linkedin.com>
References: <AANLkTikITUTjiAljgzNTxdnFxHhRdoJwOyuyCk59mgZe@mail.gmail.com>
In-Reply-To: <AANLkTikITUTjiAljgzNTxdnFxHhRdoJwOyuyCk59mgZe@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Content-Type: text/plain; charset="us-ascii"
Content-ID: <138eef54-9b03-4394-b2dc-15aca04d3d20>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


On Jun 29, 2010, at 8:07 AM, Sarah kho wrote:

> Hi,
>=20
> Can you please let me know what are tasks that the taskTracker and
> JobTracker performs?

Pretty much the entirety of the MapReduce framework.  You can think of it t=
his way:


HDFS <--> MR
NameNode <--> JobTracker
DataNode <--> TaskTracker=

From general-return-1703-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 29 18:03:54 2010
Return-Path: <general-return-1703-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60719 invoked from network); 29 Jun 2010 18:03:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jun 2010 18:03:53 -0000
Received: (qmail 49522 invoked by uid 500); 29 Jun 2010 18:03:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49455 invoked by uid 500); 29 Jun 2010 18:03:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49447 invoked by uid 99); 29 Jun 2010 18:03:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 18:03:51 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 18:03:46 +0000
Received: by pvh1 with SMTP id 1so1460526pvh.35
        for <general@hadoop.apache.org>; Tue, 29 Jun 2010 11:02:24 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.142.2.22 with SMTP id 22mr8626960wfb.13.1277834539249; Tue, 29 
	Jun 2010 11:02:19 -0700 (PDT)
Received: by 10.142.110.10 with HTTP; Tue, 29 Jun 2010 11:02:19 -0700 (PDT)
In-Reply-To: <AANLkTinf7xOzGkzBphh8rPQN8uMS3TMd2rCwnZHgSC6N@mail.gmail.com>
References: <AANLkTimLB-YNWnMwa93YL8Lpg9hJLxrCKbYL3YUr6de8@mail.gmail.com>
	<4C294361.5090206@yahoo-inc.com>
	<AANLkTinf7xOzGkzBphh8rPQN8uMS3TMd2rCwnZHgSC6N@mail.gmail.com>
Date: Tue, 29 Jun 2010 11:02:19 -0700
Message-ID: <AANLkTinoIN1WoHIbnAmvG2VDrfF_hEopJfOKeGFpvjbS@mail.gmail.com>
Subject: Re: [DISCUSSION] Proposal for making core Hadoop changes
From: Eli Collins <eli@cloudera.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Tuesday, June 29, 2010, Bernd Fondermann
<bernd.fondermann@googlemail.com> wrote:
> On Tue, Jun 29, 2010 at 02:50, Konstantin Shvachko <shv@yahoo-inc.com> wrote:
>> Eli,
>>
>> Just checking on the status of this proposal.
>>
>> In the past I was hesitant about introducing more formalities.
>> I now think we really need some mechanism for
>> new feature and project proposals, also tracking decisions.
>
> Making and tracking decisions at Apache is done via public ASF mailing
> lists, exclusively.

All proposals will be discussed and voted on the public lists.  Per
the original mail the proposal must be compatible with current bylaws.

Thanks,
Eli

From general-return-1704-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 29 18:05:03 2010
Return-Path: <general-return-1704-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 61137 invoked from network); 29 Jun 2010 18:05:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jun 2010 18:05:02 -0000
Received: (qmail 50941 invoked by uid 500); 29 Jun 2010 18:05:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50870 invoked by uid 500); 29 Jun 2010 18:05:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50854 invoked by uid 99); 29 Jun 2010 18:05:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 18:05:01 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of bernd.fondermann@googlemail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 18:04:54 +0000
Received: by wyb29 with SMTP id 29so6247439wyb.35
        for <general@hadoop.apache.org>; Tue, 29 Jun 2010 11:04:34 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=/LshnbYVTH4fxmUdeBiezxClbGoYdvEM8UJejBiXBc4=;
        b=NqlbpCqW2WbUF+NmG3xMiG61sqao6ULrngzhHN1snNSBervxxSgpiSx8bWD67Fu0m3
         HOsLFll+/PwLO14pWCVnvKrgeVOdV9+qdXgMOmQ5OKP2+skzrY6wgHt7/Ci2P+gM3kXj
         rckmzcvQihopp2ztECa2W9c6qHBs2k5ZqbSJQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=tPnI3nOtMWtVlHsTJo/hGFtPeb8Q/uOdRcNHq8s6+vwj5aJMfNTREyKzTq06rRsde3
         uLT4ydNgSkS+xmE7ojjmKeAKB6MkiSXuPLsNx1Nw6/DKTLYXPeoXyQsymnwUEqaqK3aI
         2YhHUAmbkoUgfrwB06Rk/VtyUpIptmerAPTcc=
MIME-Version: 1.0
Received: by 10.216.160.15 with SMTP id t15mr5446863wek.17.1277834661390; Tue, 
	29 Jun 2010 11:04:21 -0700 (PDT)
Received: by 10.216.162.6 with HTTP; Tue, 29 Jun 2010 11:04:21 -0700 (PDT)
In-Reply-To: <AANLkTikevCdeRR42LcSFz2C8DRGF-inGEE7blrKyvrQr@mail.gmail.com>
References: <AANLkTimLB-YNWnMwa93YL8Lpg9hJLxrCKbYL3YUr6de8@mail.gmail.com>
	<4C294361.5090206@yahoo-inc.com>
	<AANLkTinf7xOzGkzBphh8rPQN8uMS3TMd2rCwnZHgSC6N@mail.gmail.com>
	<AANLkTikevCdeRR42LcSFz2C8DRGF-inGEE7blrKyvrQr@mail.gmail.com>
Date: Tue, 29 Jun 2010 20:04:21 +0200
Message-ID: <AANLkTimmMoIjjBaPrV1YsMEKhsWxZ0wfIMDGMuifHuYl@mail.gmail.com>
Subject: Re: [DISCUSSION] Proposal for making core Hadoop changes
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Tue, Jun 29, 2010 at 19:11, Jay Booth <jaybooth@gmail.com> wrote:
> Well, if people decide that some system more organized than email
> threads is better to keep track of major project proposals, it may
> help with some aspects of the project.

As long as this system is under the control of our infra team, this is ok.

> Or it may not. =A0The fact that
> a pro forma vote by email may also be required at some points to make
> something "official" shouldn't be a major reason against such a
> system, if it's otherwise a good idea.

Discussions must also take place on-list.
There is no such thing as "pro forma" on-list activity.

  Bernd

From general-return-1705-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 29 18:11:00 2010
Return-Path: <general-return-1705-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66219 invoked from network); 29 Jun 2010 18:11:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jun 2010 18:11:00 -0000
Received: (qmail 58648 invoked by uid 500); 29 Jun 2010 18:10:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58607 invoked by uid 500); 29 Jun 2010 18:10:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58599 invoked by uid 99); 29 Jun 2010 18:10:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 18:10:58 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of bernd.fondermann@googlemail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 18:10:52 +0000
Received: by wyb29 with SMTP id 29so6252802wyb.35
        for <general@hadoop.apache.org>; Tue, 29 Jun 2010 11:10:32 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=oP9z0lq30F6ve/gjTV0R8SkXnhaafFU6mK/qj1Elmq4=;
        b=rT87gGnrMkufBbEMZ/G+eok+4WyfInv5SL9xAKdjUnwXaNxF0FEmSIdMdyZoNzw/9L
         ahxPCly5sG8kgeZSla+lHBgPSlk9dsh5ogVWfB+g4HLJ2EpSGwT8EuS99aVBUUEbKMrY
         aFxCmUz+ffwm0Kx7uc7JPkMLse9irOVyBcUgQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=jhPlL1WT84C2VFF4cdOlpOMLiJPo61TjoOXUwceBFZ6E4IW9D9JQmXZnc3JocJt+gY
         1GEDF+Kr2J8GU7W3aS/7VgDNtRsms5VW8o2ga+5ScHxnnhM4RqQVx12H6WS/77OAdo7J
         xAEGFnh6oceOg+8E71yn4gkOxrBmQBiw4n9hU=
MIME-Version: 1.0
Received: by 10.216.88.132 with SMTP id a4mr5564718wef.16.1277835031551; Tue, 
	29 Jun 2010 11:10:31 -0700 (PDT)
Received: by 10.216.162.6 with HTTP; Tue, 29 Jun 2010 11:10:31 -0700 (PDT)
In-Reply-To: <AANLkTinoIN1WoHIbnAmvG2VDrfF_hEopJfOKeGFpvjbS@mail.gmail.com>
References: <AANLkTimLB-YNWnMwa93YL8Lpg9hJLxrCKbYL3YUr6de8@mail.gmail.com>
	<4C294361.5090206@yahoo-inc.com>
	<AANLkTinf7xOzGkzBphh8rPQN8uMS3TMd2rCwnZHgSC6N@mail.gmail.com>
	<AANLkTinoIN1WoHIbnAmvG2VDrfF_hEopJfOKeGFpvjbS@mail.gmail.com>
Date: Tue, 29 Jun 2010 20:10:31 +0200
Message-ID: <AANLkTilk502281vbEbmJZ9_NtlqmEsHWDdGBz_XpkgtV@mail.gmail.com>
Subject: Re: [DISCUSSION] Proposal for making core Hadoop changes
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Tue, Jun 29, 2010 at 20:02, Eli Collins <eli@cloudera.com> wrote:
> On Tuesday, June 29, 2010, Bernd Fondermann
> <bernd.fondermann@googlemail.com> wrote:
>> On Tue, Jun 29, 2010 at 02:50, Konstantin Shvachko <shv@yahoo-inc.com> w=
rote:
>>> Eli,
>>>
>>> Just checking on the status of this proposal.
>>>
>>> In the past I was hesitant about introducing more formalities.
>>> I now think we really need some mechanism for
>>> new feature and project proposals, also tracking decisions.
>>
>> Making and tracking decisions at Apache is done via public ASF mailing
>> lists, exclusively.
>
> All proposals will be discussed and voted on the public lists. =A0Per
> the original mail the proposal must be compatible with current bylaws.

Thanks for the clarification.

  Bernd

From general-return-1706-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 29 18:55:33 2010
Return-Path: <general-return-1706-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 80053 invoked from network); 29 Jun 2010 18:55:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jun 2010 18:55:33 -0000
Received: (qmail 26711 invoked by uid 500); 29 Jun 2010 18:55:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26653 invoked by uid 500); 29 Jun 2010 18:55:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26645 invoked by uid 99); 29 Jun 2010 18:55:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 18:55:31 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of sarah.kho@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 18:55:25 +0000
Received: by wyb29 with SMTP id 29so6290576wyb.35
        for <general@hadoop.apache.org>; Tue, 29 Jun 2010 11:55:04 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=dkGy2UIl6bLl504EFSkuhWNbsEFQQzdQmnsG+qecSmg=;
        b=uBkFZeXn/uNUFe+75P1Scqg6MmePMZOElUcJI1zbymDnB4UijkBHIRD9Cef66iNYWA
         1jvpUR6XcSbjRIPnMBKhvue5cbdw220ufDE8f2GODRSHfJMRQO4yDAVLVTWyUgJARNka
         kLYLt+R83vJqXcY7yaqYQtKskzSBCjkH2aUGo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=UyXiwQLrpL0MJ3Ixb0apJmTqiUDL9jxBr8rJ5QR53TaBAtmbXqJAxRdZWRvOwBbuJV
         n//DuGUt58yEFb8UQyycQhP6kdpDUgGpS0RWUO6FZ10KS9Cx6N4ln8u7t5XrJTSpyS4l
         w7+ogVoJzOU+/GhSwqgQeqx2Sle7NL2O9KkIQ=
MIME-Version: 1.0
Received: by 10.216.158.147 with SMTP id q19mr5532990wek.11.1277837698256; 
	Tue, 29 Jun 2010 11:54:58 -0700 (PDT)
Received: by 10.216.55.141 with HTTP; Tue, 29 Jun 2010 11:54:58 -0700 (PDT)
In-Reply-To: <D6325871-64CA-4C29-9A8A-27B1A78B10C7@linkedin.com>
References: <AANLkTikITUTjiAljgzNTxdnFxHhRdoJwOyuyCk59mgZe@mail.gmail.com>
	<D6325871-64CA-4C29-9A8A-27B1A78B10C7@linkedin.com>
Date: Tue, 29 Jun 2010 23:24:58 +0430
Message-ID: <AANLkTineHKpQsXXbVcvQBUHtfxJ6gB8_rPEDJ5pGM1WV@mail.gmail.com>
Subject: Re: What are uses of taskTracker and JobTracker services?
From: Sarah kho <sarah.kho@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016367fbb53164d82048a2fc6bc
X-Virus-Checked: Checked by ClamAV on apache.org

--0016367fbb53164d82048a2fc6bc
Content-Type: text/plain; charset=ISO-8859-1

Thanks for reply.

But we have different services for each of them, for example a service for
NameNode and a Service for JobTracker, are they doing separate things each
one?



On Tue, Jun 29, 2010 at 10:20 PM, Allen Wittenauer <awittenauer@linkedin.com
> wrote:

>
> On Jun 29, 2010, at 8:07 AM, Sarah kho wrote:
>
> > Hi,
> >
> > Can you please let me know what are tasks that the taskTracker and
> > JobTracker performs?
>
> Pretty much the entirety of the MapReduce framework.  You can think of it
> this way:
>
>
> HDFS <--> MR
> NameNode <--> JobTracker
> DataNode <--> TaskTracker

--0016367fbb53164d82048a2fc6bc--

From general-return-1707-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 29 19:50:41 2010
Return-Path: <general-return-1707-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93471 invoked from network); 29 Jun 2010 19:50:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jun 2010 19:50:40 -0000
Received: (qmail 4034 invoked by uid 500); 29 Jun 2010 19:50:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3961 invoked by uid 500); 29 Jun 2010 19:50:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3951 invoked by uid 99); 29 Jun 2010 19:50:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 19:50:38 +0000
X-ASF-Spam-Status: No, hits=4.7 required=10.0
	tests=FREEMAIL_FROM,FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of michael_segel@hotmail.com designates 65.55.34.18 as permitted sender)
Received: from [65.55.34.18] (HELO col0-omc1-s8.col0.hotmail.com) (65.55.34.18)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 19:50:31 +0000
Received: from COL117-W28 ([65.55.34.9]) by col0-omc1-s8.col0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Tue, 29 Jun 2010 12:49:49 -0700
Message-ID: <COL117-W28B785415F0BE41CD1A5C98FCB0@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_36322d3f-df02-421e-90c5-353c27f5151e_"
X-Originating-IP: [65.167.11.254]
From: Michael Segel <michael_segel@hotmail.com>
To: <general@hadoop.apache.org>
Subject: RE: What are uses of taskTracker and JobTracker services?
Date: Tue, 29 Jun 2010 14:49:49 -0500
Importance: Normal
In-Reply-To: <AANLkTineHKpQsXXbVcvQBUHtfxJ6gB8_rPEDJ5pGM1WV@mail.gmail.com>
References:
 <AANLkTikITUTjiAljgzNTxdnFxHhRdoJwOyuyCk59mgZe@mail.gmail.com>,<D6325871-64CA-4C29-9A8A-27B1A78B10C7@linkedin.com>,<AANLkTineHKpQsXXbVcvQBUHtfxJ6gB8_rPEDJ5pGM1WV@mail.gmail.com>
MIME-Version: 1.0
X-OriginalArrivalTime: 29 Jun 2010 19:49:49.0051 (UTC) FILETIME=[3B8FF4B0:01CB17C4]
X-Virus-Checked: Checked by ClamAV on apache.org

--_36322d3f-df02-421e-90c5-353c27f5151e_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


I think that he was trying to explain that in HDFS=2C you have a name node =
and then your data nodes.
So you have the name node service on the name node and each data node has a=
 data node service.
When you run a map reduce job=2C you have a Job tracker that resides on the=
 name node and controls the overall job.
On each data node=2C where the jobs run in parallel=2C there exists a task =
tracker.



> Date: Tue=2C 29 Jun 2010 23:24:58 +0430
> Subject: Re: What are uses of taskTracker and JobTracker services?
> From: sarah.kho@gmail.com
> To: general@hadoop.apache.org
>=20
> Thanks for reply.
>=20
> But we have different services for each of them=2C for example a service =
for
> NameNode and a Service for JobTracker=2C are they doing separate things e=
ach
> one?
>=20
>=20
>=20
> On Tue=2C Jun 29=2C 2010 at 10:20 PM=2C Allen Wittenauer <awittenauer@lin=
kedin.com
> > wrote:
>=20
> >
> > On Jun 29=2C 2010=2C at 8:07 AM=2C Sarah kho wrote:
> >
> > > Hi=2C
> > >
> > > Can you please let me know what are tasks that the taskTracker and
> > > JobTracker performs?
> >
> > Pretty much the entirety of the MapReduce framework.  You can think of =
it
> > this way:
> >
> >
> > HDFS <--> MR
> > NameNode <--> JobTracker
> > DataNode <--> TaskTracker
 		 	   		 =20
_________________________________________________________________
The New Busy is not the old busy. Search=2C chat and e-mail from your inbox=
.
http://www.windowslive.com/campaign/thenewbusy?ocid=3DPID28326::T:WLMTAGL:O=
N:WL:en-US:WM_HMP:042010_3=

--_36322d3f-df02-421e-90c5-353c27f5151e_--

From general-return-1708-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 29 20:13:37 2010
Return-Path: <general-return-1708-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 2395 invoked from network); 29 Jun 2010 20:13:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jun 2010 20:13:36 -0000
Received: (qmail 32521 invoked by uid 500); 29 Jun 2010 20:13:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32468 invoked by uid 500); 29 Jun 2010 20:13:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32460 invoked by uid 99); 29 Jun 2010 20:13:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Jun 2010 20:13:35 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [68.180.199.179] (HELO web46314.mail.sp1.yahoo.com) (68.180.199.179)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 29 Jun 2010 20:13:26 +0000
Received: (qmail 5429 invoked by uid 60001); 29 Jun 2010 20:13:04 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1277842384; bh=fEz0ndjXwhJQY77o1S3L49O9Vs4ubskcT1gefIoDf5Y=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=CEihizC5fEH1K8SkHIBGh0f+Eb6z6bZ2VJiaFX12//WZc8AMvDVZdYk+wrvIcTWyHLtAwNSmD+3K0EnebCKSWbEfuof6j72DpruBTby3HZlTD63BMDOduCndYU5HJOLoRKx0OKMqSE/vs9KMg/kWGsqP3UK1+SnfX6oaXTKv1r4=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=HPYiEUIONWHtVnaB2VFz3eAVCr5d81RfBEPcWY83NnXkV4sWU/LCp7Cd46D7sfGEijSimYEcwTNmU7sNQkcll/JDOlvHQFYM/k2g8uACFNTF4Wk5lBIl1hHowTwMS4wZdJ1rj4BRIZG5gf8TGeIVv8yBcx8OPBjQkzapNZEznxE=;
Message-ID: <471613.3450.qm@web46314.mail.sp1.yahoo.com>
X-YMail-OSG: io3jwcgVM1lzM0H76tzVdIB73mQQPib5D9n6hann.DJ1ynb
 jxIhSQxqs0d1rwxhBq2D6uQFRV_ZDXl6eqjyPy2NJ79FMrIv.otYLzOyy0uo
 uw9Iw9D_okDgJwWP4PkFlSKqEPYyd5WsS6Af2mVETfQtdNRPiWBuFJ_ar5n2
 ULl1CyCA3zAbYkmKYfawKufrvSVIg2gMEzoAJyVqeukSCkWxX12qW3fD4WF3
 pNBKDfjpjqyjrJ.opkbdOpz27pN3wflH3NjxtfXJf.iJ6rfeEkpQzg2j3Zlz
 PINs1rYuOASM8GJTaIcp8ycK6gfvJp0hRfF1KFxuHtMYoLK8we2opOl1CNBm
 THcxu0Tdlno85NCWSiWx4tPBJsBhB
Received: from [188.74.76.201] by web46314.mail.sp1.yahoo.com via HTTP; Tue, 29 Jun 2010 13:13:04 PDT
X-Mailer: YahooMailRC/397.8 YahooMailWebService/0.8.104.274457
Date: Tue, 29 Jun 2010 13:13:04 -0700 (PDT)
From: abc xyz <fabc_xyz111@yahoo.com>
Subject: Reduce-side join
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-527755146-1277842384=:3450"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-527755146-1277842384=:3450
Content-Type: text/plain; charset=us-ascii

Hello 
everyone,

I am trying to use the hadoop's default 
reduce-side join. I want to use the TaggedMapOutput, DataJoinMapperBase, and DataJoinReducerBase classes. I have added the hadoop's datajoin jar file in my build path but still i get the following runtime error:

java.lang.NoClassDefFoundError: org/apache/hadoop/contrib/utils/join/TaggedMapOutput

Where 
possibly is the error? Please guide me.

Cheers


      
--0-527755146-1277842384=:3450--

From general-return-1709-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 30 04:15:48 2010
Return-Path: <general-return-1709-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33538 invoked from network); 30 Jun 2010 04:15:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jun 2010 04:15:48 -0000
Received: (qmail 16496 invoked by uid 500); 30 Jun 2010 04:15:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16196 invoked by uid 500); 30 Jun 2010 04:15:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16183 invoked by uid 99); 30 Jun 2010 04:15:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Jun 2010 04:15:43 +0000
X-ASF-Spam-Status: No, hits=2.5 required=10.0
	tests=FREEMAIL_FROM,FREEMAIL_REPLY,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yhemanth@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Jun 2010 04:15:37 +0000
Received: by pwj1 with SMTP id 1so187330pwj.35
        for <general@hadoop.apache.org>; Tue, 29 Jun 2010 21:14:15 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=kqTxNFuB2x3Bsz3ApFZOX6swac6Z+lp/z7LztnXu1wo=;
        b=k6TBKTNabad7JekhNGZD7iatmZsAOVhLP/QPiYRznNwobcQTVYcPHaZ0zFsobUabIk
         MEtDsU73Q2wNWou1m/O8VziOoVNBCJqlxpJw2X8BRLIBtTpXA+6RRRiXoU6eZdXcPH37
         0GQ6/ZA5L2BAmnhwcBwMH1hnktD/rKVjOBw78=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=KbMy188/hOpgvH2b21x3BTN3uuXEowW8qFVuzCAjTgI5AN+dg+alhNQCnMklJdTgut
         N0qXIsxKQy5esJxa6fFhmk3UDQ4zJ5IsdK+qIDW8E0hJFdrNdgZFEtaeLxgyPBJaRoyd
         B4AIYbMt9vkGhNY+s0JXY55HaAzg1VShiC534=
MIME-Version: 1.0
Received: by 10.143.153.9 with SMTP id f9mr9295861wfo.29.1277871255018; Tue, 
	29 Jun 2010 21:14:15 -0700 (PDT)
Received: by 10.142.188.19 with HTTP; Tue, 29 Jun 2010 21:14:14 -0700 (PDT)
In-Reply-To: <COL117-W28B785415F0BE41CD1A5C98FCB0@phx.gbl>
References: <AANLkTikITUTjiAljgzNTxdnFxHhRdoJwOyuyCk59mgZe@mail.gmail.com>
	<D6325871-64CA-4C29-9A8A-27B1A78B10C7@linkedin.com>
	<AANLkTineHKpQsXXbVcvQBUHtfxJ6gB8_rPEDJ5pGM1WV@mail.gmail.com>
	<COL117-W28B785415F0BE41CD1A5C98FCB0@phx.gbl>
Date: Wed, 30 Jun 2010 09:44:14 +0530
Message-ID: <AANLkTims03Gf_1wA6irmXmPx3Ca7LeID5SrlS3cv-9JD@mail.gmail.com>
Subject: Re: What are uses of taskTracker and JobTracker services?
From: Hemanth Yamijala <yhemanth@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

> I think that he was trying to explain that in HDFS, you have a name node =
and then your data nodes.
> So you have the name node service on the name node and each data node has=
 a data node service.
> When you run a map reduce job, you have a Job tracker that resides on the=
 name node and controls the overall job.

May or may not be true. In general, for moderately complex cases, it
is best to run the name node and jobtracker on different nodes so both
masters don't fail where only one of them can.

> On each data node, where the jobs run in parallel, there exists a task tr=
acker.

This is almost always true, of course - it helps Hadoop to achieve
data locality by colocating where the task runs with where it has to
read data from.

Thanks
Hemanth

>
>
>
>> Date: Tue, 29 Jun 2010 23:24:58 +0430
>> Subject: Re: What are uses of taskTracker and JobTracker services?
>> From: sarah.kho@gmail.com
>> To: general@hadoop.apache.org
>>
>> Thanks for reply.
>>
>> But we have different services for each of them, for example a service f=
or
>> NameNode and a Service for JobTracker, are they doing separate things ea=
ch
>> one?
>>
>>
>>
>> On Tue, Jun 29, 2010 at 10:20 PM, Allen Wittenauer <awittenauer@linkedin=
.com
>> > wrote:
>>
>> >
>> > On Jun 29, 2010, at 8:07 AM, Sarah kho wrote:
>> >
>> > > Hi,
>> > >
>> > > Can you please let me know what are tasks that the taskTracker and
>> > > JobTracker performs?
>> >
>> > Pretty much the entirety of the MapReduce framework. =A0You can think =
of it
>> > this way:
>> >
>> >
>> > HDFS <--> MR
>> > NameNode <--> JobTracker
>> > DataNode <--> TaskTracker
>
> _________________________________________________________________
> The New Busy is not the old busy. Search, chat and e-mail from your inbox=
.
> http://www.windowslive.com/campaign/thenewbusy?ocid=3DPID28326::T:WLMTAGL=
:ON:WL:en-US:WM_HMP:042010_3

From general-return-1710-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 30 05:48:34 2010
Return-Path: <general-return-1710-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50977 invoked from network); 30 Jun 2010 05:48:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jun 2010 05:48:33 -0000
Received: (qmail 63318 invoked by uid 500); 30 Jun 2010 05:48:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63071 invoked by uid 500); 30 Jun 2010 05:48:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63061 invoked by uid 99); 30 Jun 2010 05:48:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Jun 2010 05:48:27 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of fariha.atta.pk@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Jun 2010 05:48:19 +0000
Received: by fxm16 with SMTP id 16so224090fxm.35
        for <general@hadoop.apache.org>; Tue, 29 Jun 2010 22:46:59 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=yvQPCxjb1SNFAb8VEueYYt9gTXOYoT6gY0a8W5rX5bU=;
        b=tHhd6raNyIOpNwKy9EIx9L0gWA8z+XKuehBOTCLcwuF93sgVmGk7Xambh+u3PfPYxr
         ZESgFdBIMv6IoEZi29VpZM5YW/iFWwt7osMv2cr0hj1J/WdfnT+KyS8lr9M6tcrgl5bK
         HKOimyYF/OTVxvA2aXf8KtCAA7DlH7e3CpE1M=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=pfv3tMk92ADOa4H/0AS4pwXPFOEu3fbAu1L+Gk6DzV6yAeJ1OdwJ6pXjcnH9W44JWa
         nBomAajAfTCKYLC83WWwdsLWgYlTNxBRV/fImMlIKu8P7pVBR/oFV/N3JKusiUHI5bTL
         La6G8QvkWdWspjBNtVxx+Nv0L5wB3pxBQ/X5w=
MIME-Version: 1.0
Received: by 10.223.50.193 with SMTP id a1mr6822581fag.34.1277876819346; Tue, 
	29 Jun 2010 22:46:59 -0700 (PDT)
Received: by 10.223.19.5 with HTTP; Tue, 29 Jun 2010 22:46:59 -0700 (PDT)
In-Reply-To: <AANLkTims03Gf_1wA6irmXmPx3Ca7LeID5SrlS3cv-9JD@mail.gmail.com>
References: <AANLkTikITUTjiAljgzNTxdnFxHhRdoJwOyuyCk59mgZe@mail.gmail.com>
	<D6325871-64CA-4C29-9A8A-27B1A78B10C7@linkedin.com>
	<AANLkTineHKpQsXXbVcvQBUHtfxJ6gB8_rPEDJ5pGM1WV@mail.gmail.com>
	<COL117-W28B785415F0BE41CD1A5C98FCB0@phx.gbl>
	<AANLkTims03Gf_1wA6irmXmPx3Ca7LeID5SrlS3cv-9JD@mail.gmail.com>
Date: Wed, 30 Jun 2010 06:46:59 +0100
Message-ID: <AANLkTimtadjzCJyGkM1mh3-4Q0Ctk2juBJPaSSTjpcDc@mail.gmail.com>
Subject: Re: What are uses of taskTracker and JobTracker services?
From: Fariha Atta <fariha.atta.pk@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015174479ece245ad048a38e194
X-Virus-Checked: Checked by ClamAV on apache.org

--0015174479ece245ad048a38e194
Content-Type: text/plain; charset=ISO-8859-1

*Hi,

NameNode* serves as a directory namespace manager. It keeps the directory
tree of all files in the file system, and tracks where across the cluster
the file data is kept. It does not store the data of these files itself.


whereas a jobtracker manages job-related parameters.

"The JobTracker is the service within Hadoop that farms out
MapReduce<http://wiki.apache.org/hadoop/MapReduce>tasks to specific
nodes in the cluster, ideally the nodes that have the
data, or at least are in the same rack.

   1. Client applications submit jobs to the Job tracker.
   2.

   The JobTracker talks to the
NameNode<http://wiki.apache.org/hadoop/NameNode>to determine the
location of the data
   3.

   The JobTracker locates
TaskTracker<http://wiki.apache.org/hadoop/TaskTracker>nodes with
available slots at or near the data
   4.

   The JobTracker submits the work to the chosen
TaskTracker<http://wiki.apache.org/hadoop/TaskTracker>nodes."

[source: http://wiki.apache.org/hadoop/JobTracker]


Best,
Jonna

--0015174479ece245ad048a38e194--

From general-return-1711-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 30 06:34:46 2010
Return-Path: <general-return-1711-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68791 invoked from network); 30 Jun 2010 06:34:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jun 2010 06:34:46 -0000
Received: (qmail 9959 invoked by uid 500); 30 Jun 2010 06:34:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 9669 invoked by uid 500); 30 Jun 2010 06:34:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 9661 invoked by uid 99); 30 Jun 2010 06:34:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Jun 2010 06:34:42 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eltonsky9404@gmail.com designates 209.85.216.176 as permitted sender)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Jun 2010 06:34:35 +0000
Received: by qyk31 with SMTP id 31so198198qyk.35
        for <general@hadoop.apache.org>; Tue, 29 Jun 2010 23:34:14 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=UEF+5lIMyAelAgpyVuf17EIGdDvLqiouuZNU4Q76gk0=;
        b=HHXWpVfI3ao4GBiRnBJSRH2nLx01eSALPePZsZF9k0RBimcGEYGetdhg6gu8AklGpP
         V4SLx5szq4aJMZKhqaoyVTMYiWl4Ke//wwA1l1D2HsBcm1LO3BXsvWOz3VedNreQB02L
         9CIT+/22WiLuNTSazBoImlsnOGf2qRCfTOfpY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=cswsZa5gMhyU9G6CjKF1S3GI1IPoUsR7xxXvbiEU2HVQNBd7P4x8fVcmhozQRNv4v3
         xpwIhHb60263g7GFxaGrpGlU25MWB/Irmr8oLHF8t0hIGFT83uPQ16QENx+mn0wGrf4y
         sefztHwGS4nhndyVfJVLdROeQvL9Pn5QbAduE=
MIME-Version: 1.0
Received: by 10.224.110.77 with SMTP id m13mr5428016qap.106.1277879653557; 
	Tue, 29 Jun 2010 23:34:13 -0700 (PDT)
Received: by 10.224.89.18 with HTTP; Tue, 29 Jun 2010 23:34:13 -0700 (PDT)
Date: Wed, 30 Jun 2010 16:34:13 +1000
Message-ID: <AANLkTiljKNGMlcFI4tQOiFxIYFutfKxKFFIYS5Ew7OU_@mail.gmail.com>
Subject: single thread read/write in HDFS
From: elton sky <eltonsky9404@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000feaed6430d0e8da048a398ab4
X-Virus-Checked: Checked by ClamAV on apache.org

--000feaed6430d0e8da048a398ab4
Content-Type: text/plain; charset=ISO-8859-1

>From my understanding, HDFS uses a single thread to do read and write.
Why not using multi-thread? Since a file is composed of many blocks and each
block is stored as a file in the underlying FS, we can do some parallelism
among blocks.
Is this right?

Elton

--000feaed6430d0e8da048a398ab4--

From general-return-1712-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 30 06:35:35 2010
Return-Path: <general-return-1712-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68910 invoked from network); 30 Jun 2010 06:35:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jun 2010 06:35:35 -0000
Received: (qmail 10861 invoked by uid 500); 30 Jun 2010 06:35:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 10800 invoked by uid 500); 30 Jun 2010 06:35:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 10792 invoked by uid 99); 30 Jun 2010 06:35:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Jun 2010 06:35:31 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of zhangguoping96@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Jun 2010 06:35:23 +0000
Received: by fxm16 with SMTP id 16so242113fxm.35
        for <general@hadoop.apache.org>; Tue, 29 Jun 2010 23:35:03 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=i+S3jibIY5GvzqBkwU83BsJEdTI1pBq3UxB5owweaP4=;
        b=kfufaS2xQm5wXmPelmj4cHLRYwkFXcmFDGdL7zoHeo2SPbfxXFx1h59I5qjZ7LFUW9
         wniWYxXmAwii6MWhJjvcgc885ceWQXSpv0+WQekfhieMISkGMfwXrDbXDFBOaqBrnNZi
         Jkkgj+jXALAA8JUhcoHYZkVuaNIaoACVBvT6Q=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=rLcRba4k4Tckd86LafcMH3p/H5eYgAnLKFXsBqSa7HsJZEzpC4WRxAdTG/bpQdh+s7
         BEgtVeipO8T6PYQAybgonitmLGMC6A85FuSJyVRZCqBdfMjuONm5+Mg1mEIu6E5ImA6L
         Ek/SIzlvxaPy0LElm0zi4u7NQ8VNpBMmUsOgo=
MIME-Version: 1.0
Received: by 10.239.140.210 with SMTP id y18mr526619hby.196.1277879703334; 
	Tue, 29 Jun 2010 23:35:03 -0700 (PDT)
Received: by 10.239.153.71 with HTTP; Tue, 29 Jun 2010 23:35:00 -0700 (PDT)
In-Reply-To: <AANLkTimk2pALHZloj6Z2yv-UnDGEC292FX6UKH49iBj5@mail.gmail.com>
References: <AANLkTilTKnEH2IA4XCyo_dbNnzED50hN9wIOQ-6IBU9s@mail.gmail.com>
	<AANLkTimk2pALHZloj6Z2yv-UnDGEC292FX6UKH49iBj5@mail.gmail.com>
Date: Wed, 30 Jun 2010 14:35:00 +0800
Message-ID: <AANLkTilmubY14oZ3kALWiM8SBGjI_atodwmXIN1lVrBH@mail.gmail.com>
Subject: Re: how to get the current input file in a Mapper
From: zhangguoping zhangguoping <zhangguoping96@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f19ef4c86f3a048a398dd2
X-Virus-Checked: Checked by ClamAV on apache.org

--001485f19ef4c86f3a048a398dd2
Content-Type: text/plain; charset=ISO-8859-1

It works. Thanks.

2010/6/28 Ted Yu <yuzhihong@gmail.com>

> How about:
> FileSplit fileSplit = (FileSplit) context.getInputSplit();
> String sFileName = fileSplit.getPath().getName();
>
>
> On Sun, Jun 27, 2010 at 7:59 AM, zhangguoping zhangguoping <
> zhangguoping96@gmail.com> wrote:
>
> > Hi,
> >
> > how to get the current input file in a Mapper in 0.20.2 ?
> >
> > Thanks,
> >
>

--001485f19ef4c86f3a048a398dd2--

From general-return-1713-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 30 10:03:39 2010
Return-Path: <general-return-1713-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22263 invoked from network); 30 Jun 2010 10:03:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jun 2010 10:03:39 -0000
Received: (qmail 2660 invoked by uid 500); 30 Jun 2010 10:03:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2186 invoked by uid 500); 30 Jun 2010 10:03:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2163 invoked by uid 99); 30 Jun 2010 10:03:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Jun 2010 10:03:35 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Jun 2010 10:03:25 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id DD91C1BA77F
	for <general@hadoop.apache.org>; Wed, 30 Jun 2010 11:03:04 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id c2vatex3rTTf for <general@hadoop.apache.org>;
	Wed, 30 Jun 2010 11:03:04 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by colossus.hpl.hp.com (Postfix) with ESMTPS id 779021BA798
	for <general@hadoop.apache.org>; Wed, 30 Jun 2010 11:03:04 +0100 (BST)
MailScanner-NULL-Check: 1278496972.15263@+BY2U3TolAzUulr1Y+O8GQ
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o5UA2poc000713
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Wed, 30 Jun 2010 11:02:51 +0100 (BST)
Message-ID: <4C2B164B.20503@apache.org>
Date: Wed, 30 Jun 2010 11:02:51 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100228)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: What are uses of taskTracker and JobTracker services?
References: <AANLkTikITUTjiAljgzNTxdnFxHhRdoJwOyuyCk59mgZe@mail.gmail.com>	<D6325871-64CA-4C29-9A8A-27B1A78B10C7@linkedin.com>	<AANLkTineHKpQsXXbVcvQBUHtfxJ6gB8_rPEDJ5pGM1WV@mail.gmail.com>	<COL117-W28B785415F0BE41CD1A5C98FCB0@phx.gbl> <AANLkTims03Gf_1wA6irmXmPx3Ca7LeID5SrlS3cv-9JD@mail.gmail.com>
In-Reply-To: <AANLkTims03Gf_1wA6irmXmPx3Ca7LeID5SrlS3cv-9JD@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o5UA2poc000713
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

Hemanth Yamijala wrote:
> Hi,
> 
>> I think that he was trying to explain that in HDFS, you have a name node and then your data nodes.
>> So you have the name node service on the name node and each data node has a data node service.
>> When you run a map reduce job, you have a Job tracker that resides on the name node and controls the overall job.
> 
> May or may not be true. In general, for moderately complex cases, it
> is best to run the name node and jobtracker on different nodes so both
> masters don't fail where only one of them can.

More for scale than availability, was my belief; if the NN goes offline, 
your JT locks up until it comes back anyway

>> On each data node, where the jobs run in parallel, there exists a task tracker.
> 
> This is almost always true, of course - it helps Hadoop to achieve
> data locality by colocating where the task runs with where it has to
> read data from.

If you have machines in the room which can come and go without warning 
-doing Hadoop work with spare cycles- then you can make them 
task-tracker only, so you don't store persistent data there, just temp 
files and when the machines get switched to other work you don't lose 
HDFS data. But you do increase network traffic...

From general-return-1714-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 30 15:47:08 2010
Return-Path: <general-return-1714-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39066 invoked from network); 30 Jun 2010 15:47:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jun 2010 15:47:08 -0000
Received: (qmail 78483 invoked by uid 500); 30 Jun 2010 15:47:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78268 invoked by uid 500); 30 Jun 2010 15:47:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78260 invoked by uid 99); 30 Jun 2010 15:47:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Jun 2010 15:47:06 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of renatoj.marroquin@gmail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Jun 2010 15:46:58 +0000
Received: by iwn39 with SMTP id 39so1044394iwn.35
        for <general@hadoop.apache.org>; Wed, 30 Jun 2010 08:46:37 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=7xuQ1fF46YZZtwQCiX4PCtkCK/Lpas24jNtwmHwyS+c=;
        b=tFGElXcyKDMlHAUNu8xu7ODgfGLUmifJ3Y+xaK+TAOQ8On3BtNMCpxps0GdlVVuMoa
         Oeqw3v3317hw6tIvFdRbiPtb9FlnHC7FyQQuBaAFiPhqfvnbPWIjuC7A4BtylXF1CmSj
         bBnoud73hwqbwk9e/pHGg3iP53R1wles9b7yw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=MBti4BNYNByAIDOtOCk8KJZfXSfzVy77qc5JE0wFhTG6Sq/Ct3OmvSFDDUqRSfJgTU
         EyfPccRGUMvjNz4QHI9VoLRpjEpVs+4p2B6tJsOQdBhdjjWholj4BIFAUUXXZkJFLW2U
         KAsQgiv6yH/rroeUSImypWF48JKt++fBEopls=
MIME-Version: 1.0
Received: by 10.42.36.15 with SMTP id s15mr2037946icd.40.1277912797285; Wed, 
	30 Jun 2010 08:46:37 -0700 (PDT)
Received: by 10.231.33.200 with HTTP; Wed, 30 Jun 2010 08:46:37 -0700 (PDT)
Date: Wed, 30 Jun 2010 12:46:37 -0300
Message-ID: <AANLkTine5opXB6KXLejtKYC7zyiRam6a2Qw7RwkPR8fo@mail.gmail.com>
Subject: Hadoop Platform
From: =?ISO-8859-1?Q?Renato_Marroqu=EDn_Mogrovejo?= <renatoj.marroquin@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba1ef92c560f43048a4142d7
X-Virus-Checked: Checked by ClamAV on apache.org

--90e6ba1ef92c560f43048a4142d7
Content-Type: text/plain; charset=ISO-8859-1

Hi everyone,

I read a while ago that maybe IBM or Yahoo! were going to provide a "cloud"
for academic purposes. Does anybody know if those intentions materialised?
or if there any other ( free or not so expensive ) platform to test Hadoop
jobs??? I mean, obviously there would be several limitations on it, but I
don't have access to a cluster and I have some Hadoop jobs I need to test
them, so I can prove to my people the power of it (:
Thanks in advance.


Renato M.

--90e6ba1ef92c560f43048a4142d7--

From general-return-1715-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 30 16:44:26 2010
Return-Path: <general-return-1715-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53847 invoked from network); 30 Jun 2010 16:44:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jun 2010 16:44:25 -0000
Received: (qmail 67868 invoked by uid 500); 30 Jun 2010 16:44:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67754 invoked by uid 500); 30 Jun 2010 16:44:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67736 invoked by uid 99); 30 Jun 2010 16:44:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Jun 2010 16:44:21 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Jun 2010 16:44:13 +0000
Received: from SNV-EXBH01.ds.corp.yahoo.com (snv-exbh01.ds.corp.yahoo.com [207.126.227.249])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o5UGhLk7046350;
	Wed, 30 Jun 2010 09:43:21 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:cc:message-id:
	thread-topic:thread-index:mime-version:content-type:
	content-transfer-encoding:x-originalarrivaltime;
	b=oA+2DS6NQwsAP864u/KIE5XYXL9bzIDAv8VMEixzWqRuG6jUufq7gLNEMph8gfuf
Received: from SNV-EXVS06.ds.corp.yahoo.com ([207.126.227.234]) by SNV-EXBH01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Wed, 30 Jun 2010 09:43:21 -0700
Received: from 10.72.111.153 ([10.72.111.153]) by SNV-EXVS06.ds.corp.yahoo.com ([207.126.227.82]) via Exchange Front-End Server snv-webmail.corp.yahoo.com ([207.126.227.59]) with Microsoft Exchange Server HTTP-DAV ;
 Wed, 30 Jun 2010 16:42:31 +0000
User-Agent: Microsoft-Entourage/12.25.0.100505
Date: Wed, 30 Jun 2010 09:42:27 -0700
Subject: [Result][Vote] Move Chukwa to incubator
From: Eric Yang <eyang@yahoo-inc.com>
To: "general@incubator.apache.org" <general@incubator.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>
Message-ID: <C850C203.9514%eyang@yahoo-inc.com>
Thread-Topic: [Result][Vote] Move Chukwa to incubator
Thread-Index: AcsYczkwPxLacQzrSUugc+uLsSrwSQ==
Mime-version: 1.0
Content-type: text/plain;
	charset="ISO-8859-1"
Content-transfer-encoding: quoted-printable
X-OriginalArrivalTime: 30 Jun 2010 16:43:21.0246 (UTC) FILETIME=[598623E0:01CB1873]
X-Virus-Checked: Checked by ClamAV on apache.org

Original incubator proposal:

http://wiki.apache.org/incubator/ChukwaProposal

Vote put forward to incubator:

1. recommend TLP with guides to help the initial pmc,
2. accept incubating with tlp resource naming, but -incubating release
naming
3. accept incubating requiring all incubator naming conventions, that might
help the incubator simplify this decision.

Result of the vote:

Option 1) Ant Elder, Eric Yang, William A. Rowe Jr.
Option 2) Ari Rabkin, Jerome Boulon, Chris Douglas, Greg Reddin
Option 3) Bernd Fondermann

Owen O'Malley +1 on proposal

I am not sure about Chris Mattmann=B9s position, but he raised the question
about TLP.

Is it ok to keep hadoop naming until we graduate to TLP, hence, we only
rename once?  =B3-incubating=B2 will be added to release artifacts.

What is next?

Regards,
Eric


From general-return-1716-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 30 17:08:02 2010
Return-Path: <general-return-1716-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 62529 invoked from network); 30 Jun 2010 17:08:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jun 2010 17:08:02 -0000
Received: (qmail 1540 invoked by uid 500); 30 Jun 2010 17:08:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1426 invoked by uid 500); 30 Jun 2010 17:08:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1416 invoked by uid 99); 30 Jun 2010 17:08:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Jun 2010 17:08:00 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yhemanth@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Jun 2010 17:07:53 +0000
Received: by pwj1 with SMTP id 1so745333pwj.35
        for <general@hadoop.apache.org>; Wed, 30 Jun 2010 10:06:29 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=iNmpyg42ZTiE0JN3+r7d+xG23NUMoYy3fZBbcy9FDe8=;
        b=ugfd4aD6HIsnE5amSksWkB9LGs0NcVCmL7PY0F0sgnq833ZMPZy2b9PCyHDDMhHvWo
         6YxZ3OFZtRxWq94w1XIMhT52MuGJQdeLL5IuhFfGb0HJpSgfw9AgPUbyByQtc2kxZveu
         mirkbXNN4Wt794Q2QI2H9IgrJ9bk2DciyOPPw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=acNC2jYNlPIZjbn92cLTrzDVFBnpXOVwwMsc0A0jAefPMZIOaDzvwbPQuk8LLNaVPm
         60jmrS5z35oQRDJqXBjI6cNhzVaOZzQQrvK49+ZkYefEXpLoCQXcBEKA6C+LhegeKX/e
         o2/OH180ck8WVcLmeSwazgCnFEHdE0ldoZUL8=
MIME-Version: 1.0
Received: by 10.142.9.15 with SMTP id 15mr10462765wfi.235.1277917588171; Wed, 
	30 Jun 2010 10:06:28 -0700 (PDT)
Received: by 10.142.188.19 with HTTP; Wed, 30 Jun 2010 10:06:28 -0700 (PDT)
In-Reply-To: <AANLkTine5opXB6KXLejtKYC7zyiRam6a2Qw7RwkPR8fo@mail.gmail.com>
References: <AANLkTine5opXB6KXLejtKYC7zyiRam6a2Qw7RwkPR8fo@mail.gmail.com>
Date: Wed, 30 Jun 2010 22:36:28 +0530
Message-ID: <AANLkTil0DwbxxIFB2lIWRwo_p4n71Yf7cDGRPBisvY0u@mail.gmail.com>
Subject: Re: Hadoop Platform
From: Hemanth Yamijala <yhemanth@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Renato,

> I read a while ago that maybe IBM or Yahoo! were going to provide a "cloud"
> for academic purposes. Does anybody know if those intentions materialised?
> or if there any other ( free or not so expensive ) platform to test Hadoop
> jobs??? I mean, obviously there would be several limitations on it, but I
> don't have access to a cluster and I have some Hadoop jobs I need to test
> them, so I can prove to my people the power of it (:
> Thanks in advance.

Can you check if OpenCirrus (https://opencirrus.org/) can fit your
bill ? It seems like some one like a faculty member can request access
by submitting a proposal.

Hemanth

From general-return-1717-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 30 17:55:17 2010
Return-Path: <general-return-1717-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73653 invoked from network); 30 Jun 2010 17:55:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jun 2010 17:55:17 -0000
Received: (qmail 47869 invoked by uid 500); 30 Jun 2010 17:55:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47797 invoked by uid 500); 30 Jun 2010 17:55:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47789 invoked by uid 99); 30 Jun 2010 17:55:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Jun 2010 17:55:15 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [64.18.0.28] (HELO exprod5og114.obsmtp.com) (64.18.0.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Jun 2010 17:55:05 +0000
Received: from source ([4.78.218.129]) (using TLSv1) by exprod5ob114.postini.com ([64.18.4.12]) with SMTP
	ID DSNKTCuEV51MeoVFO3RHpbX7miX1KKPmDCBE@postini.com; Wed, 30 Jun 2010 10:54:44 PDT
Received: from unknown (HELO cinmlef09.e2k.ad.ge.com) ([3.159.213.56])
  by Cinmlip04.e2k.ad.ge.com with ESMTP; 30 Jun 2010 13:52:21 -0400
Received: from CINMLVEM12.e2k.ad.ge.com ([3.159.214.56]) by cinmlef09.e2k.ad.ge.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Wed, 30 Jun 2010 13:52:19 -0400
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Subject: RE: Hadoop Platform
Date: Wed, 30 Jun 2010 13:52:12 -0400
Message-ID: <30B5A69F3E33FE4C83F4C832F5D6E2CF0254F734@CINMLVEM12.e2k.ad.ge.com>
In-Reply-To: <AANLkTine5opXB6KXLejtKYC7zyiRam6a2Qw7RwkPR8fo@mail.gmail.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Hadoop Platform
Thread-Index: AcsYa39L72UJQtVySXKLKuLoewGb0AAEUuVg
References: <AANLkTine5opXB6KXLejtKYC7zyiRam6a2Qw7RwkPR8fo@mail.gmail.com>
From: "Yan, Weizhong (GE, Research)" <yan@ge.com>
To: <general@hadoop.apache.org>
X-OriginalArrivalTime: 30 Jun 2010 17:52:19.0909 (UTC) FILETIME=[FC5BFF50:01CB187C]
X-Virus-Checked: Checked by ClamAV on apache.org

Why don't you just simply try Amazon elastic MapReduce? =
http://aws.amazon.com/elasticmapreduce/

-----Original Message-----
From: Renato Marroqu=EDn Mogrovejo [mailto:renatoj.marroquin@gmail.com]=20
Sent: Wednesday, June 30, 2010 11:47 AM
To: general@hadoop.apache.org
Subject: Hadoop Platform

Hi everyone,

I read a while ago that maybe IBM or Yahoo! were going to provide a =
"cloud"
for academic purposes. Does anybody know if those intentions =
materialised?
or if there any other ( free or not so expensive ) platform to test =
Hadoop jobs??? I mean, obviously there would be several limitations on =
it, but I don't have access to a cluster and I have some Hadoop jobs I =
need to test them, so I can prove to my people the power of it (:
Thanks in advance.


Renato M.

From general-return-1718-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 30 18:24:44 2010
Return-Path: <general-return-1718-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 89438 invoked from network); 30 Jun 2010 18:24:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jun 2010 18:24:44 -0000
Received: (qmail 65017 invoked by uid 500); 30 Jun 2010 18:24:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 64951 invoked by uid 500); 30 Jun 2010 18:24:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 64943 invoked by uid 99); 30 Jun 2010 18:24:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Jun 2010 18:24:42 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.62.64] (HELO qmta07.westchester.pa.mail.comcast.net) (76.96.62.64)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Jun 2010 18:24:34 +0000
Received: from omta04.westchester.pa.mail.comcast.net ([76.96.62.35])
	by qmta07.westchester.pa.mail.comcast.net with comcast
	id cJQC1e0010ldTLk57JQEiF; Wed, 30 Jun 2010 18:24:14 +0000
Received: from wlan-snve-153-216.corp.yahoo.com ([209.131.62.115])
	by omta04.westchester.pa.mail.comcast.net with comcast
	id cJPy1e00P2VBGtd3QJQ1HJ; Wed, 30 Jun 2010 18:24:09 +0000
Cc: "general@hadoop.apache.org" <general@hadoop.apache.org>
Message-Id: <3424DED2-C285-477E-9FA5-80A09A9DA6CC@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@incubator.apache.org
In-Reply-To: <C850D243.1741B%Chris.A.Mattmann@jpl.nasa.gov>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [Result][Vote] Move Chukwa to incubator
Date: Wed, 30 Jun 2010 11:23:57 -0700
References: <C850D243.1741B%Chris.A.Mattmann@jpl.nasa.gov>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

On Jun 30, 2010, at 10:51 AM, Mattmann, Chris A (388J) wrote:

> On 6/30/10 9:42 AM, "Eric Yang" <eyang@yahoo-inc.com> wrote:
>
> Original incubator proposal:
>
> http://wiki.apache.org/incubator/ChukwaProposal
>
> Vote put forward to incubator:
>
> 1. recommend TLP with guides to help the initial pmc,
> 2. accept incubating with tlp resource naming, but -incubating release
> naming
> 3. accept incubating requiring all incubator naming conventions,  
> that might
> help the incubator simplify this decision.
>
> Result of the vote:
>
> Option 1) Ant Elder, Eric Yang, William A. Rowe Jr.
> Option 2) Ari Rabkin, Jerome Boulon, Chris Douglas, Greg Reddin
> Option 3) Bernd Fondermann
>
> Owen O'Malley +1 on proposal

Sorry, I meant to vote for option 2.  I think that it would be good to  
have more visibility in before they become a TLP. They also need to  
figure out how to add new committers to their project.

-- Owen

From general-return-1719-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 30 18:50:31 2010
Return-Path: <general-return-1719-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98357 invoked from network); 30 Jun 2010 18:50:31 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jun 2010 18:50:31 -0000
Received: (qmail 86167 invoked by uid 500); 30 Jun 2010 18:50:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86085 invoked by uid 500); 30 Jun 2010 18:50:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 15873 invoked by uid 99); 29 Jun 2010 17:41:15 -0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1277833182; bh=1D9XYMPw2dVBaz1T56ajpjM+Q9MSPjc3iD3OYmoJkuA=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=vw9+LRj7yj7t58JSHtPC5SilJMywHPTBaV6P+Jm3j5AbrHPhTntfgm4Y3d9fxSiB5x2mRrFj02Va7ITPn19d7hD928NSi1jmIDfl4LscmDi4bzEwUVyVrp/CCjPU/9H7iE+I35B0/Lek+3VKJ/3csHun1eAFXdX7jV/GPBUUf3A=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=1jZw93qjA8LZsgPcVByr8VMbrtXyHr2kh8irasht2Pts8sluF3UrIl3zgeKtkkBWq3eRao8Ormzheu+nz4nvqdMe52tUSUDto/NuU/7qSaBC1yI9/wRDD2F+rRvKNJqfidPvUD4UhEkaCuKLI291cWZ5OtawIvDS2Mc/D85EEYg=;
Message-ID: <529450.31780.qm@web46316.mail.sp1.yahoo.com>
X-YMail-OSG: NijDUEAVM1nhBdydP82LbIWg89uIsjs06zXWtuErbIA11MG
 MQCTv4KLdCNRvCFHIzquPvlJuA8MI0HtZ8_HGVvsUz5yFtpS0sHZquzbzgmj
 Lsmno88T8b1mdb6HyCybkmlJ7tR29FNmi3T3K6C2J9tmF.D4AV69Lxeg2HDv
 VBXuSQnJnwXHXZooKhGI6_KME0WkOqycbIZT69VKvsUmpgeoffSaI1b_jUwB
 Jdzf5U1ycQkjSFW1azxRFK_gfILhaM8etiXVoCoYRdYh3i5QqBD81znRIz5p
 PUbVjBw3n8XgYtnh.iTyhx_YFDwkGCRLbvgxyBWB6AFLC2P8gEZ27FtoGB7K
 6jl_PbyCr6TsFOMBh5Z1ad596cJiI
X-Mailer: YahooMailRC/397.8 YahooMailWebService/0.8.104.274457
Date: Tue, 29 Jun 2010 10:39:42 -0700 (PDT)
From: abc xyz <fabc_xyz111@yahoo.com>
Subject: Reduce-side join
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1728800411-1277833182=:31780"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-1728800411-1277833182=:31780
Content-Type: text/plain; charset=us-ascii

Hello everyone,

I am trying to use the hadoop's default 
reduce-side join. I want to use the TaggedMapOutput, DataJoinMapperBase, and DataJoinReducerBase classes. I have added the hadoop's datajoin jar file in my build path but still i get the following runtime error:

java.lang.NoClassDefFoundError: org/apache/hadoop/contrib/utils/join/TaggedMapOutput

Where possibly is the error? Please guide me.

Cheers



      
--0-1728800411-1277833182=:31780--

From general-return-1720-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 30 18:50:33 2010
Return-Path: <general-return-1720-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98428 invoked from network); 30 Jun 2010 18:50:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jun 2010 18:50:33 -0000
Received: (qmail 87267 invoked by uid 500); 30 Jun 2010 18:50:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87192 invoked by uid 500); 30 Jun 2010 18:50:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 20930 invoked by uid 99); 29 Jun 2010 17:45:14 -0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1277833420; bh=Crb7VJEDBrOgPqQfc6+dzlxNwk4ROapDer2hjCgA9fI=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=xt28wBhFo5f7kYhPitPDDUZagw5c1a9HPZPaqsr5T702nVDV95On+T9RvSyKysWgYdlqNf0UYWMxUUqDANsavrMLn5xszzQ3Eiq9S2TUS6beOFTztxw1p9u4RhAkIo0AEScPFuMml12NyEKGUYQndEu2yxICj2rroAv5oHQEKqM=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=oVBgXX4KfgdGiy3c88yiCAOP3r2bLwcSQsiX4FKgzxJ3jl518W5eezb7SzJn8pe1SRAH3OmNo9Wr8vlyGzaqHJM17ChBBt/obdZ3U4oKq8ko6XX5iwb9l6GDiA0TMxnlFl4xeL19wBGX9b6Ur0Y9qb34DqtD/H0Y7axturzYOvM=;
Message-ID: <608856.6167.qm@web46314.mail.sp1.yahoo.com>
X-YMail-OSG: 9ZuV208VM1k4JT5Va6N3z21yW7_RxvzWvwS2hWpIZPkgY0P
 JWUiDZawS6agj.aiyUfBRvrbfXfVfbxDavZGbrjWNRENI3tJSN4gJOPBhLG_
 QtD2gFP7e3vAYw_ZhO6dOirA386RT3vzbi.AlNKtojNQRIJVInhpwFNk5.xN
 jP2X5eunWZWlkz382ugpoUj7WZZGLa05dRSicESFjeVjbSwfihJOEXqGmyIu
 z65VgvAsnHCEIBV4nDuYYT2UOmP51kuDauFnZDO_e2c39wWnjb5TbLVvSzrA
 69wtv7LHo30I1LBwzcGAFPfuX0JkMz3c.P0wgtSw_jtnljsJk40orcGoKcI7
 qu97bVopIt1M_YHYHmAZsmGjHb.1q
X-Mailer: YahooMailRC/397.8 YahooMailWebService/0.8.104.274457
Date: Tue, 29 Jun 2010 10:43:40 -0700 (PDT)
From: abc xyz <fabc_xyz111@yahoo.com>
Subject: Reduce-side join
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-745025091-1277833420=:6167"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-745025091-1277833420=:6167
Content-Type: text/plain; charset=us-ascii

Hello everyone,

I am trying to use the hadoop's default 
reduce-side join. I want to use the TaggedMapOutput, DataJoinMapperBase, and DataJoinReducerBase classes. I have added the hadoop's datajoin jar file in my build path but still i get the following runtime error:

java.lang.NoClassDefFoundError: org/apache/hadoop/contrib/utils/join/TaggedMapOutput

Where possibly is the error? Please guide me.

Cheers


      
--0-745025091-1277833420=:6167--

From general-return-1721-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 30 23:40:57 2010
Return-Path: <general-return-1721-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88713 invoked from network); 30 Jun 2010 23:40:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jun 2010 23:40:56 -0000
Received: (qmail 37532 invoked by uid 500); 30 Jun 2010 23:40:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37458 invoked by uid 500); 30 Jun 2010 23:40:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37449 invoked by uid 99); 30 Jun 2010 23:40:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Jun 2010 23:40:54 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Jun 2010 23:40:48 +0000
Received: by qwd7 with SMTP id 7so715833qwd.35
        for <general@hadoop.apache.org>; Wed, 30 Jun 2010 16:40:27 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.224.19.100 with SMTP id z36mr6824129qaa.84.1277941227040; Wed, 
	30 Jun 2010 16:40:27 -0700 (PDT)
Received: by 10.229.234.3 with HTTP; Wed, 30 Jun 2010 16:40:26 -0700 (PDT)
In-Reply-To: <30B5A69F3E33FE4C83F4C832F5D6E2CF0254F734@CINMLVEM12.e2k.ad.ge.com>
References: <AANLkTine5opXB6KXLejtKYC7zyiRam6a2Qw7RwkPR8fo@mail.gmail.com>
	<30B5A69F3E33FE4C83F4C832F5D6E2CF0254F734@CINMLVEM12.e2k.ad.ge.com>
Date: Wed, 30 Jun 2010 16:40:26 -0700
Message-ID: <AANLkTilAe0jkMxwf1wg1u6TJr8sWUObgayKU5RJQtMZL@mail.gmail.com>
Subject: Re: Hadoop Platform
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00c09f8e5b59e1afa0048a47e0ab
X-Virus-Checked: Checked by ClamAV on apache.org

--00c09f8e5b59e1afa0048a47e0ab
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

If you want to use an open API and be cloud-provider agnostic, check out
Whirr: http://incubator.apache.org/projects/whirr.html.

On Wed, Jun 30, 2010 at 10:52 AM, Yan, Weizhong (GE, Research)
<yan@ge.com>wrote:

> Why don't you just simply try Amazon elastic MapReduce?
> http://aws.amazon.com/elasticmapreduce/
>
> -----Original Message-----
> From: Renato Marroqu=EDn Mogrovejo [mailto:renatoj.marroquin@gmail.com]
> Sent: Wednesday, June 30, 2010 11:47 AM
> To: general@hadoop.apache.org
> Subject: Hadoop Platform
>
> Hi everyone,
>
> I read a while ago that maybe IBM or Yahoo! were going to provide a "clou=
d"
> for academic purposes. Does anybody know if those intentions materialised=
?
> or if there any other ( free or not so expensive ) platform to test Hadoo=
p
> jobs??? I mean, obviously there would be several limitations on it, but I
> don't have access to a cluster and I have some Hadoop jobs I need to test
> them, so I can prove to my people the power of it (:
> Thanks in advance.
>
>
> Renato M.
>

--00c09f8e5b59e1afa0048a47e0ab--

From general-return-857-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 01 06:57:21 2010
Return-Path: <general-return-857-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77732 invoked from network); 1 Jan 2010 06:57:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jan 2010 06:57:20 -0000
Received: (qmail 92878 invoked by uid 500); 1 Jan 2010 06:57:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92799 invoked by uid 500); 1 Jan 2010 06:57:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 92789 invoked by uid 99); 1 Jan 2010 06:57:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Jan 2010 06:57:18 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.30.17] (HELO QMTA10.emeryville.ca.mail.comcast.net) (76.96.30.17)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Jan 2010 06:57:08 +0000
Received: from OMTA07.emeryville.ca.mail.comcast.net ([76.96.30.59])
	by QMTA10.emeryville.ca.mail.comcast.net with comcast
	id Q6i61d0011GXsucAA6woeC; Fri, 01 Jan 2010 06:56:48 +0000
Received: from [10.108.25.199] ([209.131.62.115])
	by OMTA07.emeryville.ca.mail.comcast.net with comcast
	id Q6wR1d0042VBGtd8T6wVW5; Fri, 01 Jan 2010 06:56:45 +0000
Message-Id: <677239CF-8546-4E94-9D6E-65AA472EF5C0@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <63C88D50-5DD1-4A39-80D0-596BF9FD7E89@noaa.gov>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: What is included in each download
Date: Thu, 31 Dec 2009 22:56:23 -0800
References: <63C88D50-5DD1-4A39-80D0-596BF9FD7E89@noaa.gov>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


On Dec 30, 2009, at 3:20 PM, Roy Mendelssohn wrote:

> Hi All:
>
> I am new to what is in this Apache Project. When I go t http://hadoop.apache.org/index.html 
> , a glaring omission,  IMO, is any description of what contains  
> what, and what I get if I do a download.  Does Hadoop common contain  
> hbase and mapreduce and hdfs, or do I need to install these  
> separately, or what.  I skimmed several of the starting documents,  
> and they do not tell you that either.  Same with the other  
> components listed on that page.
>
> Any information on that would be greatly appreciated,

Ok, it goes like this. After 0.20 Core split into 3 subprojects  
(Common, HDFS, MapReduce). So if you are downloading Core 0.20.1,  
you'll get all 3 of them. HBase, Hive, Pig, and ZooKeeper are all  
separate sub-projects and you need to download them separately.

-- Owen

From general-return-858-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 01 15:25:16 2010
Return-Path: <general-return-858-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78165 invoked from network); 1 Jan 2010 15:25:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jan 2010 15:25:16 -0000
Received: (qmail 30644 invoked by uid 500); 1 Jan 2010 15:25:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30601 invoked by uid 500); 1 Jan 2010 15:25:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30591 invoked by uid 99); 1 Jan 2010 15:25:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Jan 2010 15:25:15 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [63.249.95.37] (HELO mail.cruzio.com) (63.249.95.37)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Jan 2010 15:25:04 +0000
Received: from [10.0.1.84] (dsl-68-170-179-94.dhcp.cruzio.com [68.170.179.94])
	by mail.cruzio.com with ESMTP id o01FOfGI013625
	for <general@hadoop.apache.org>; Fri, 1 Jan 2010 07:24:42 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1077)
Subject: Re: What is included in each download
From: Roy Mendelssohn <Roy.Mendelssohn@noaa.gov>
In-Reply-To: <677239CF-8546-4E94-9D6E-65AA472EF5C0@apache.org>
Date: Fri, 1 Jan 2010 07:24:41 -0800
Content-Transfer-Encoding: quoted-printable
Message-Id: <9B9B6C64-23C4-4D74-8030-5E055318D67C@noaa.gov>
References: <63C88D50-5DD1-4A39-80D0-596BF9FD7E89@noaa.gov> <677239CF-8546-4E94-9D6E-65AA472EF5C0@apache.org>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1077)
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks!

Happy new years,

-Roy M.
On Dec 31, 2009, at 10:56 PM, Owen O'Malley wrote:

>=20
> On Dec 30, 2009, at 3:20 PM, Roy Mendelssohn wrote:
>=20
>> Hi All:
>>=20
>> I am new to what is in this Apache Project. When I go t =
http://hadoop.apache.org/index.html, a glaring omission,  IMO, is any =
description of what contains what, and what I get if I do a download.  =
Does Hadoop common contain hbase and mapreduce and hdfs, or do I need to =
install these separately, or what.  I skimmed several of the starting =
documents, and they do not tell you that either.  Same with the other =
components listed on that page.
>>=20
>> Any information on that would be greatly appreciated,
>=20
> Ok, it goes like this. After 0.20 Core split into 3 subprojects =
(Common, HDFS, MapReduce). So if you are downloading Core 0.20.1, you'll =
get all 3 of them. HBase, Hive, Pig, and ZooKeeper are all separate =
sub-projects and you need to download them separately.
>=20
> -- Owen

**********************
"The contents of this message do not reflect any position of the U.S. =
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@noaa.gov (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"=46rom those who have been given much, much will be expected"=20


From general-return-859-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 05 16:05:46 2010
Return-Path: <general-return-859-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88638 invoked from network); 5 Jan 2010 16:05:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Jan 2010 16:05:46 -0000
Received: (qmail 33396 invoked by uid 500); 5 Jan 2010 16:05:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33304 invoked by uid 500); 5 Jan 2010 16:05:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33294 invoked by uid 99); 5 Jan 2010 16:05:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2010 16:05:44 +0000
X-ASF-Spam-Status: No, hits=-1.8 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [195.232.224.72] (HELO mailout03.vodafone.com) (195.232.224.72)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2010 16:05:34 +0000
Received: from mailint03 (localhost [127.0.0.1])
	by mailout03 (Postfix) with ESMTP id 8B5ED1165B2
	for <general@hadoop.apache.org>; Tue,  5 Jan 2010 17:05:13 +0100 (CET)
Received: from avoexs04.internal.vodafone.com (avoexs04.dc-ratingen.de [145.230.4.198])
	(using TLSv1 with cipher RC4-MD5 (128/128 bits))
	(No client certificate requested)
	by mailint03 (Postfix) with ESMTPS id 803C611659B
	for <general@hadoop.apache.org>; Tue,  5 Jan 2010 17:05:13 +0100 (CET)
Received: from VF-MBX13.internal.vodafone.com ([145.230.5.24]) by avoexs04.internal.vodafone.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Tue, 5 Jan 2010 17:05:14 +0100
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01CA8E20.DDCE3774"
Subject: Hadoop with SELinux?
Date: Tue, 5 Jan 2010 17:05:14 +0100
Message-ID: <219D8244D980254ABF28AB469AD4E98F033F8CFE@VF-MBX13.internal.vodafone.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Hadoop with SELinux?
Thread-Index: AcqOIN2ppXfWnH3TTbWRjg2vdggfUg==
From: "Gibbon, Robert, VF-Group" <Robert.Gibbon@vodafone.com>
To: <general@hadoop.apache.org>
X-OriginalArrivalTime: 05 Jan 2010 16:05:14.0709 (UTC) FILETIME=[DDF04850:01CA8E20]
X-Virus-Checked: Checked by ClamAV on apache.org

------_=_NextPart_001_01CA8E20.DDCE3774
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Hello list

Can someone please tell me if it would be possible to run hadoop with =
SELinux enabled across the cluster? Are there any known issues or =
better, how2's I can be pointed at? Also interested in running iptables =
on the nodes - easy to do?

Many thanks in advance
Robert

Robert Gibbon
Solutions Architect
Integration Design & Solution Engineering

Vodafone Group Service GmbH
Mannesmannufer 2, D-40213 D=FCsseldorf, Germany
Amtsgericht D=FCsseldorf, HRB 53554
Gesch=E4ftsf=FChrung: Helmut Hoffmann, Dr. Joachim Peters



------_=_NextPart_001_01CA8E20.DDCE3774--

From general-return-860-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 05 16:48:52 2010
Return-Path: <general-return-860-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1747 invoked from network); 5 Jan 2010 16:48:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Jan 2010 16:48:52 -0000
Received: (qmail 91705 invoked by uid 500); 5 Jan 2010 16:48:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91628 invoked by uid 500); 5 Jan 2010 16:48:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91618 invoked by uid 99); 5 Jan 2010 16:48:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2010 16:48:51 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_HELO_PASS,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [64.81.36.244] (HELO foobar.kobold.org) (64.81.36.244)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2010 16:48:42 +0000
Received: from owl.kobold.org (owl.kobold.org [192.168.1.7])
	(authenticated bits=0)
	by foobar.kobold.org (8.14.1/8.14.1) with ESMTP id o05Foiix025276
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Tue, 5 Jan 2010 07:50:44 -0800
Message-ID: <4B436D53.2030406@hep.caltech.edu>
Date: Tue, 05 Jan 2010 08:48:19 -0800
From: Michael Thomas <thomas@hep.caltech.edu>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.5) Gecko/20091209 Fedora/3.0-4.fc12 Thunderbird/3.0
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Hadoop with SELinux?
References: <219D8244D980254ABF28AB469AD4E98F033F8CFE@VF-MBX13.internal.vodafone.com>
In-Reply-To: <219D8244D980254ABF28AB469AD4E98F033F8CFE@VF-MBX13.internal.vodafone.com>
Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms060902080204030001040805"
X-Virus-Checked: Checked by ClamAV on apache.org

--------------ms060902080204030001040805
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: quoted-printable

We have used selinux on our large cluster with HDFS (we don't use MR).=20
The only issue I've found is that the mount program does not have=20
permission to execute java, which prohibits you from mounting the fuse=20
filesystem from /etc/fstab.  This is fixed with the policy file below.

require {
         type mount_t;
         type shell_exec_t;
         type proc_net_t;
         type random_device_t;
         type java_exec_t;
         type fusefs_t;
         class process { execstack execmem getsched setrlimit };
         class tcp_socket { accept listen };
         class chr_file read;
         class file { execute read getattr execute_no_trans };
         class dir { read getattr search };
}

#=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D mount_t =3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D
allow mount_t fusefs_t:dir { read getattr };
allow mount_t java_exec_t:file { read getattr execute execute_no_trans };=

allow mount_t proc_net_t:dir search;
allow mount_t proc_net_t:file { read getattr };
allow mount_t random_device_t:chr_file read;
allow mount_t self:process { execstack execmem getsched setrlimit };
allow mount_t self:tcp_socket { accept listen };
allow mount_t shell_exec_t:file { read execute getattr execute_no_trans }=
;


--Mike

On 01/05/2010 08:05 AM, Gibbon, Robert, VF-Group wrote:
> Hello list
>
> Can someone please tell me if it would be possible to run hadoop with S=
ELinux enabled across the cluster? Are there any known issues or better, =
how2's I can be pointed at? Also interested in running iptables on the no=
des - easy to do?
>
> Many thanks in advance
> Robert
>
> Robert Gibbon
> Solutions Architect
> Integration Design&  Solution Engineering
>
> Vodafone Group Service GmbH
> Mannesmannufer 2, D-40213 D=FCsseldorf, Germany
> Amtsgericht D=FCsseldorf, HRB 53554
> Gesch=E4ftsf=FChrung: Helmut Hoffmann, Dr. Joachim Peters
>
>
>



--------------ms060902080204030001040805
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIMEDCC
A/gwggLgoAMCAQICASkwDQYJKoZIhvcNAQEFBQAwdTETMBEGCgmSJomT8ixkARkWA25ldDES
MBAGCgmSJomT8ixkARkWAkVTMQ4wDAYDVQQKEwVFU25ldDEgMB4GA1UECxMXQ2VydGlmaWNh
dGUgQXV0aG9yaXRpZXMxGDAWBgNVBAMTD0VTbmV0IFJvb3QgQ0EgMTAeFw0wMjEyMDUwODAw
MDBaFw0xMzAxMjUwODAwMDBaMGkxEzARBgoJkiaJk/IsZAEZFgNvcmcxGDAWBgoJkiaJk/Is
ZAEZFghET0VHcmlkczEgMB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9yaXRpZXMxFjAUBgNV
BAMTDURPRUdyaWRzIENBIDEwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC09dYj
YaPbCD5mtbiQb7Ka3y1qAm0ZcqKCFciWcfe8Kwcuy9tjHuIsLf9ZItdkDW4xy8sua9nJlx3K
lwjtumTMtOtg35KZCknUd8KM4VGTSFdLVG9AbNayef76caVCGM1+jyF0Lq03kauGOPTcNfZe
1TZa3e1c9rc8ljV5OSWa/mfsCACyS5zFIWu0yIDNyJdf+n0hwaPN53wllpJ30taD+JBjQ7h2
k4xRWzeaznLOb9OztZVRA/1sVze+iczFh2xwa4VdGy0eIIPw1pfvYwxO36rm0S109qvbsNla
roPRbxerPKakQLpKe034Xcx7gBPqUk/FxoRRWin5EWN3rz9LAgMBAAGjgZ4wgZswDgYDVR0P
AQH/BAQDAgGGMBEGCWCGSAGG+EIBAQQEAwIAhzAdBgNVHQ4EFgQUyhkdEo5upDhdQtQxDgjb
2Y0XDV0wHwYDVR0jBBgwFoAUvF1NSC/4NZRZq1yJSz7RsjoUAeowDwYDVR0TAQH/BAUwAwEB
/zAlBgNVHREEHjAcgRpET0VHcmlkcy1DQS0xQGRvZWdyaWRzLm9yZzANBgkqhkiG9w0BAQUF
AAOCAQEAZNVrIDLqe39CEOiJt7Q7EpBPhAihMvDTSf/42u0SMbUmChww4mLmph5DBghZUVF8
Yn59kRZMn1QLOtO1HzLqvAvPITacZVPlJgG2IXzlR636YghZFAycbIUEOJDBHR4vtQO1KDxg
ZwvAbtmKIoxvhUCq2xsfFt9kCBBn+JYtQ6O5LsBJq3PmuubeMcc7mbQAfJZ7h/3QghgkFIhm
E1+LBXPJbkuP8vgfg6h2BKoAf5TFfZECgGZKimfN110tBvfedGZwYYd3/GsJc83B0JN1gny0
gqNVPm392UchXGeBRrHnm2gkhIkr48Oq6EmNGV9/a6XfbplQW/JWbtPVPWkaizCCBAYwggLu
oAMCAQICAnbUMA0GCSqGSIb3DQEBBQUAMGkxEzARBgoJkiaJk/IsZAEZFgNvcmcxGDAWBgoJ
kiaJk/IsZAEZFghET0VHcmlkczEgMB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9yaXRpZXMx
FjAUBgNVBAMTDURPRUdyaWRzIENBIDEwHhcNMDkwMjAzMDUzMzA3WhcNMTAwMjAzMDUzMzA3
WjBgMRMwEQYKCZImiZPyLGQBGRYDb3JnMRgwFgYKCZImiZPyLGQBGRYIZG9lZ3JpZHMxDzAN
BgNVBAsTBlBlb3BsZTEeMBwGA1UEAxMVTWljaGFlbCBUaG9tYXMgNzI1Mzc0MIIBIjANBgkq
hkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA3am27G7W/7Zw5ec/CirtaeSb7yy28gLko9rjSnll
X1B9Yp3ecWlTVME6muVRtAelEnRtxkpFYM04spxwR0YL/0F8WwAOMAthwqIR4p5AA3PzdexY
3WNDvgk/+Z8FWON21NX2N+VMf/okhgR6UVvqhtf/hNcMWywbPpzwSWnSlXZatTpO/m35hlrY
c3spPDz3u7StAd0aZuFZvIX4y+bHtmQ5efcvariWfs3NWC6o5AXNY0AxkrGNnowh/q4iAi7j
kUK8W8/1w9ytZhkNgk/bXi090Yktku/CwWHoNTT+5lJ//2VH8EZVVzjBI0aLXwzeJu/Pdcps
wjNZBZN9OxYFvQIDAQABo4HAMIG9MBEGCWCGSAGG+EIBAQQEAwIF4DAOBgNVHQ8BAf8EBAMC
BPAwGAYDVR0gBBEwDzANBgsqhkiG90wDBwECCjA6BgNVHR8EMzAxMC+gLaArhilodHRwOi8v
cGtpMS5kb2Vncmlkcy5vcmcvQ1JMLzFjM2YyY2E4LmNybDAhBgNVHREEGjAYgRZ0aG9tYXNA
aGVwLmNhbHRlY2guZWR1MB8GA1UdIwQYMBaAFMoZHRKObqQ4XULUMQ4I29mNFw1dMA0GCSqG
SIb3DQEBBQUAA4IBAQCn6RiomxqyDr4lWvxWHtJgmn4B9eEVwlciFI3wkNzS+fRbhUS2W9sa
u7BDeYWAnjDdBR/H2P5HDmtGyQ3FUEgIauPeltQgPZ0OvvjHaAmzqyaVJVCpiCXr9aXOMEil
9AcTZuLbgv8OcnBfZizEhpjO7dFmsv4bovDxx/9UK6WRCPEZRXXctRvo4EtbwmloYh3MGtpQ
tSztL1VlbEhwczG8JXVEz2OUKiC8An9o5av+zTu9i+6lkXaqOLWun6y/weEXTvA+PvZEwBl4
jtXC/aiuWcQGp68VNhQQh1s7drRE+6StvqPWZolWPVRuhMOCDnRJJrB45rD6udgq2BjRpV4D
MIIEBjCCAu6gAwIBAgICdtQwDQYJKoZIhvcNAQEFBQAwaTETMBEGCgmSJomT8ixkARkWA29y
ZzEYMBYGCgmSJomT8ixkARkWCERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRo
b3JpdGllczEWMBQGA1UEAxMNRE9FR3JpZHMgQ0EgMTAeFw0wOTAyMDMwNTMzMDdaFw0xMDAy
MDMwNTMzMDdaMGAxEzARBgoJkiaJk/IsZAEZFgNvcmcxGDAWBgoJkiaJk/IsZAEZFghkb2Vn
cmlkczEPMA0GA1UECxMGUGVvcGxlMR4wHAYDVQQDExVNaWNoYWVsIFRob21hcyA3MjUzNzQw
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDdqbbsbtb/tnDl5z8KKu1p5JvvLLby
AuSj2uNKeWVfUH1ind5xaVNUwTqa5VG0B6USdG3GSkVgzTiynHBHRgv/QXxbAA4wC2HCohHi
nkADc/N17FjdY0O+CT/5nwVY43bU1fY35Ux/+iSGBHpRW+qG1/+E1wxbLBs+nPBJadKVdlq1
Ok7+bfmGWthzeyk8PPe7tK0B3Rpm4Vm8hfjL5se2ZDl59y9quJZ+zc1YLqjkBc1jQDGSsY2e
jCH+riICLuORQrxbz/XD3K1mGQ2CT9teLT3RiS2S78LBYeg1NP7mUn//ZUfwRlVXOMEjRotf
DN4m7891ymzCM1kFk307FgW9AgMBAAGjgcAwgb0wEQYJYIZIAYb4QgEBBAQDAgXgMA4GA1Ud
DwEB/wQEAwIE8DAYBgNVHSAEETAPMA0GCyqGSIb3TAMHAQIKMDoGA1UdHwQzMDEwL6AtoCuG
KWh0dHA6Ly9wa2kxLmRvZWdyaWRzLm9yZy9DUkwvMWMzZjJjYTguY3JsMCEGA1UdEQQaMBiB
FnRob21hc0BoZXAuY2FsdGVjaC5lZHUwHwYDVR0jBBgwFoAUyhkdEo5upDhdQtQxDgjb2Y0X
DV0wDQYJKoZIhvcNAQEFBQADggEBAKfpGKibGrIOviVa/FYe0mCafgH14RXCVyIUjfCQ3NL5
9FuFRLZb2xq7sEN5hYCeMN0FH8fY/kcOa0bJDcVQSAhq496W1CA9nQ6++MdoCbOrJpUlUKmI
Jev1pc4wSKX0BxNm4tuC/w5ycF9mLMSGmM7t0Way/hui8PHH/1QrpZEI8RlFddy1G+jgS1vC
aWhiHcwa2lC1LO0vVWVsSHBzMbwldUTPY5QqILwCf2jlq/7NO72L7qWRdqo4ta6frL/B4RdO
8D4+9kTAGXiO1cL9qK5ZxAanrxU2FBCHWzt2tET7pK2+o9ZmiVY9VG6Ew4IOdEkmsHjmsPq5
2CrYGNGlXgMxggNbMIIDVwIBATBvMGkxEzARBgoJkiaJk/IsZAEZFgNvcmcxGDAWBgoJkiaJ
k/IsZAEZFghET0VHcmlkczEgMB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9yaXRpZXMxFjAU
BgNVBAMTDURPRUdyaWRzIENBIDECAnbUMAkGBSsOAwIaBQCgggHBMBgGCSqGSIb3DQEJAzEL
BgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTEwMDEwNTE2NDgxOVowIwYJKoZIhvcNAQkE
MRYEFKnQFh9pgfky+jikdrvLrfA9LQ0AMF8GCSqGSIb3DQEJDzFSMFAwCwYJYIZIAWUDBAEC
MAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzAN
BggqhkiG9w0DAgIBKDB+BgkrBgEEAYI3EAQxcTBvMGkxEzARBgoJkiaJk/IsZAEZFgNvcmcx
GDAWBgoJkiaJk/IsZAEZFghET0VHcmlkczEgMB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9y
aXRpZXMxFjAUBgNVBAMTDURPRUdyaWRzIENBIDECAnbUMIGABgsqhkiG9w0BCRACCzFxoG8w
aTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERPRUdyaWRzMSAwHgYD
VQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3JpZHMgQ0EgMQIC
dtQwDQYJKoZIhvcNAQEBBQAEggEA0zbC3y7bJLUXK5C36vMXILilvPHvSLc7lwVkVBKvb54l
xQ2G/UsOTR82hXTOeWMj0FHTLQDY8++D3uLlaZzhwN6NzUk/mrGW53nyoJ3y/zau7sigCye9
VKI5FTv8vLeCnPMLlajfGNFe/rmoRlGegE0cLtL2AXeW5Yu7sl8MzMOkfJgJjBcmpJr24X1u
1PyAN3XhzTxdcarhQmYK89HSOjCd4ti9qssbV6dFqYXsATAh9tRL4/a/Y0NSkRsXEApDZRoz
x6nnQy6a4ygFrxPTLwY5kYqCKEblBcWEWM4i9iOcZ6aqbiWycTGT4h8aa/yQ/wxUw144o+56
4sRmZq5J3AAAAAAAAA==
--------------ms060902080204030001040805--

From general-return-861-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 05 16:55:24 2010
Return-Path: <general-return-861-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 3345 invoked from network); 5 Jan 2010 16:55:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Jan 2010 16:55:24 -0000
Received: (qmail 1547 invoked by uid 500); 5 Jan 2010 16:55:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1478 invoked by uid 500); 5 Jan 2010 16:55:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1468 invoked by uid 99); 5 Jan 2010 16:55:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2010 16:55:23 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of psdc1978@gmail.com designates 72.14.220.157 as permitted sender)
Received: from [72.14.220.157] (HELO fg-out-1718.google.com) (72.14.220.157)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2010 16:55:15 +0000
Received: by fg-out-1718.google.com with SMTP id 19so5273466fgg.11
        for <general@hadoop.apache.org>; Tue, 05 Jan 2010 08:54:54 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=Dst4oUSpe+6eKv7P0GFo5sva/PX3DImtiFCGIUXPwNw=;
        b=l8Mc4FSw/K/7CSDadrAGI1irl3QfmxBiqbqc7selspnpxndxOxg17vFkZPsrPVUpfS
         kgEY7Jn8eTSqsHDtcEDkyKcflfbAheSX2DCEBHOy6uhuM+aQGJzDoD3xthaSEG/uQu8Y
         WX9R+zULuezsioblEFBCc4hNBFTGb1C7XiX8w=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=XmGm+5LYrr3MXXcSkYvnVMVdft3aCHeZ7s2PiH5+HGxmyPvtXCjXIHOXhtRB49RGKU
         nvzQHBx0D2BfcJgEzpul6FcXtiBiMD2+LpnonOMR5alexAE2hYHRDlmn27+q29MDWlO6
         tJEqtjKcI8Weiciit+NHHzq12M3Ez7uimJWpk=
MIME-Version: 1.0
Received: by 10.239.170.28 with SMTP id q28mr811898hbe.149.1262710494700; Tue, 
	05 Jan 2010 08:54:54 -0800 (PST)
Date: Tue, 5 Jan 2010 16:54:54 +0000
Message-ID: <61fec3fc1001050854l7436b98ek4c632853e8473ac@mail.gmail.com>
Subject: heck if I'm subscribed to this list
From: psdc1978 <psdc1978@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

It's just to check if I'm subscribed to this list, because I posted
some questions and I haven't got yet a response.

-- 
Pedro

From general-return-862-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 05 18:29:28 2010
Return-Path: <general-return-862-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44931 invoked from network); 5 Jan 2010 18:29:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Jan 2010 18:29:28 -0000
Received: (qmail 13442 invoked by uid 500); 5 Jan 2010 18:29:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13347 invoked by uid 500); 5 Jan 2010 18:29:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 77610 invoked by uid 99); 5 Jan 2010 09:42:36 -0000
X-ASF-Spam-Status: No, hits=2.8 required=10.0
	tests=HTML_FONT_FACE_BAD,HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
From: "qin.wang" <qin.wang@i-soft.com.cn>
To: <general@hadoop.apache.org>
Subject: Three questions about Hadoop
Date: Tue, 5 Jan 2010 17:42:08 +0800
Message-ID: <EFF29D2E4C964443B54746056C36D10B@jszl>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_NextPart_000_0000_01CA8E2E.6842F9A0"
X-Mailer: Microsoft Office Outlook 11
Thread-Index: AcqN60HMFYGT6hAZSHqDno6m9jMjMw==
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5579
X-Virus-Scanned: ClamAV 0.92/10258/Tue Jan  5 03:21:01 2010 on mail.i-soft.com.cn
X-Virus-Status: Clean

------=_NextPart_000_0000_01CA8E2E.6842F9A0
Content-Type: text/plain;
	charset="gb2312"
Content-Transfer-Encoding: quoted-printable

Hi team,

=20

When I try to do some research on Hadoop, I have several high level
questions, if any comments from you it will do great help for me:

=20

1. Hadoop assumes the files are big files, but take Google as an =
example, it
seems the google result for user are small files, so how to understand =
the
big files=A3=BFAnd what=A1=AFs the file content for example?

=20

2. Why are the files write-once and read-many times?

=20

3. How to install other softwares on Hadoop, is there any special
requirements for the software? Do they need to support the Map/Reduce =
module
and then can be installed?=20

=20

It will be very appreciated for your help.

=20

=CD=F5 =C7=D9  Annie.Wang

=20

=C9=CF=BA=A3=CA=D0=D0=EC=BB=E3=C7=F8=B9=F0=C1=D6=C2=B7418=BA=C57=BA=C5=C2=
=A56=C2=A5
Zip code: 200 233
Tel:      +86 21 5497 8666-8004
Fax:     +86 21 5497 7986
Mobile:  +86 137 6108 8369

=20


------=_NextPart_000_0000_01CA8E2E.6842F9A0--


From general-return-863-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 05 18:38:37 2010
Return-Path: <general-return-863-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56380 invoked from network); 5 Jan 2010 18:38:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Jan 2010 18:38:37 -0000
Received: (qmail 27038 invoked by uid 500); 5 Jan 2010 18:38:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26992 invoked by uid 500); 5 Jan 2010 18:38:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26982 invoked by uid 99); 5 Jan 2010 18:38:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2010 18:38:36 +0000
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.59.243] (HELO QMTA13.westchester.pa.mail.comcast.net) (76.96.59.243)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2010 18:38:26 +0000
Received: from OMTA05.westchester.pa.mail.comcast.net ([76.96.62.43])
	by QMTA13.westchester.pa.mail.comcast.net with comcast
	id Robp1d0020vyq2s5Due5NR; Tue, 05 Jan 2010 18:38:05 +0000
Received: from tryarticle-lm.corp.yahoo.com ([209.131.62.113])
	by OMTA05.westchester.pa.mail.comcast.net with comcast
	id Rudw1d0072SbwD53Rudzwq; Tue, 05 Jan 2010 18:38:03 +0000
Message-Id: <800C45C0-4738-4536-B9D9-DEDC03D9C33B@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <61fec3fc0911231410y23a02102xf52f7bed2f2afa1e@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: How compile hadoop-0.20.1
Date: Tue, 5 Jan 2010 10:37:54 -0800
References: <61fec3fc0911231410y23a02102xf52f7bed2f2afa1e@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


On Nov 23, 2009, at 2:10 PM, psdc1978 wrote:

> Hi all,
>
> 1 - I'm trying to compile hadoop-0.20.1 but I'm facing several errors.
> Hadoop includes several subprojects, but these subprojects are all  
> compiled
> in one file called hadoop-0.20.1-core.jar. If I just want to update  
> the code
> in the MapReduce component, do I have to compile all the project?

The project split happened after the 0.20 branch. Common, HDFS, and  
MapReduce were a single project named Core.

> 3 - Is there any tutorial that helps how to compile Hadoop?

http://wiki.apache.org/hadoop/HowToRelease

> 4 - Should I try to compile hadoop in windows with Cygwin, or should I
> compile inside Ubuntu?

I'd strongly suggest Ubuntu. Very few organizations other than  
Microsoft run large Windows clusters. *smile* Of course compiling the  
jars can happen on any operating system, but the native parts (if you  
want them) need to be compiled on the deployment platform.

-- Owen

From general-return-864-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 05 19:30:07 2010
Return-Path: <general-return-864-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72768 invoked from network); 5 Jan 2010 19:30:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Jan 2010 19:30:07 -0000
Received: (qmail 14160 invoked by uid 500); 5 Jan 2010 19:30:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14089 invoked by uid 500); 5 Jan 2010 19:30:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14079 invoked by uid 99); 5 Jan 2010 19:30:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2010 19:30:05 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2010 19:29:58 +0000
Received: by pwi20 with SMTP id 20so11206154pwi.29
        for <general@hadoop.apache.org>; Tue, 05 Jan 2010 11:29:36 -0800 (PST)
MIME-Version: 1.0
Received: by 10.143.154.2 with SMTP id g2mr3856424wfo.18.1262719776268; Tue, 
	05 Jan 2010 11:29:36 -0800 (PST)
From: Todd Lipcon <todd@cloudera.com>
Date: Tue, 5 Jan 2010 11:29:16 -0800
Message-ID: <45f85f71001051129i176c7638ud173ceb82e6b61b8@mail.gmail.com>
Subject: 0.20.2 HDFS incompatible with 0.20.1
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636e0a6c0b72124047c6fdb7a
X-Virus-Checked: Checked by ClamAV on apache.org

--001636e0a6c0b72124047c6fdb7a
Content-Type: text/plain; charset=ISO-8859-1

Hey all,

In a recent discussion, we noticed that the 0.20.2 HDFS client will not be
wire-compatible with 0.20.0 or 0.20.1 due to the inclusion of HDFS-793
(required for HDFS-101). This begs a few questions:

1) Although we certainly do not guarantee wire compatibility between minor
versions (0.20 -> 0.21) have we previously implied wire compatibility
between bugfix releases?

2) Is the above something we *should* be guaranteeing already?

3) If we haven't guaranteed the above, how many users think we have? (ie how
do we correctly call out this fact in the 0.20.2 release notes in such a way
that no one gets surprised). I can imagine plenty of organizations where a
lockstep upgrade between client and server is difficult, and we should make
sure that cluster operators know it will be necessary. Since it wasn't
necessary between 0.20.0 and 0.20.1, or various 0.18 releases, people may
have grown used to non-lockstep upgrades.

4) If the above are problems, would it be worth considering a patch for
branch-20 that provides a client that is compatible with either, based on
the datanode protocol version number of the server? It seems like a bit of
scary complexity, but wanted to throw it out there.

Thanks
-Todd

--001636e0a6c0b72124047c6fdb7a--

From general-return-865-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 05 20:06:56 2010
Return-Path: <general-return-865-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 80678 invoked from network); 5 Jan 2010 20:06:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Jan 2010 20:06:56 -0000
Received: (qmail 60551 invoked by uid 500); 5 Jan 2010 20:06:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60464 invoked by uid 500); 5 Jan 2010 20:06:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60454 invoked by uid 99); 5 Jan 2010 20:06:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2010 20:06:55 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2010 20:06:44 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:User-Agent:Date:Subject:From:To:
   Message-ID:Thread-Topic:Thread-Index:In-Reply-To:
   Mime-version:Content-type:Content-transfer-encoding;
  b=tf3kx6ZGMOQT8L/mngQ43pTO6EeH3p/UsW2IRDgEDUJKi+di/LxvawRZ
   3dj4RHcDhyc7f9PKnU4yFlA95CZ1EAkIaKgMX/AZ63VrQFULbSyIQjTE2
   1+DL7CDlfno89/g;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1262722004; x=1294258004;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=200.20.2=20HDFS=20incompatible=20with=200
   .20.1|Date:=20Tue,=2005=20Jan=202010=2012:06:19=20-0800
   |Message-ID:=20<C768DBBB.21A0%awittenauer@linkedin.com>
   |To:=20<general@hadoop.apache.org>|Mime-version:=201.0
   |Content-transfer-encoding:=207bit|In-Reply-To:=20<45f85f
   71001051129i176c7638ud173ceb82e6b61b8@mail.gmail.com>;
  bh=DfXVbUqNWD1xy19uNVI71ZOJ891xDkSSZtFmvfFHquY=;
  b=kGqN3CXa1RertOQXjHN4TIgJGQFMKuy6jWgJPT7GI6BFJESHP4H6Eqqq
   RKf589MhM+x3hBDa5dbIXH/noJBr+LXDOvB27ENEmArKYuB3htztmvOkI
   GwM9lqYlEf0IGMz;
X-IronPort-AV: E=Sophos;i="4.49,224,1262592000"; 
   d="scan'208";a="10544516"
Received: from 172.16.22.109 ([172.16.22.109]) by CORP-MAIL.linkedin.biz ([172.18.46.135]) via Exchange Front-End Server mail-access.linkedin.biz ([172.18.46.133]) with Microsoft Exchange Server HTTP-DAV ;
 Tue,  5 Jan 2010 20:06:22 +0000
User-Agent: Microsoft-Entourage/12.10.0.080409
Date: Tue, 05 Jan 2010 12:06:19 -0800
Subject: Re: 0.20.2 HDFS incompatible with 0.20.1
From: Allen Wittenauer <awittenauer@linkedin.com>
To: <general@hadoop.apache.org>
Message-ID: <C768DBBB.21A0%awittenauer@linkedin.com>
Thread-Topic: 0.20.2 HDFS incompatible with 0.20.1
Thread-Index: AcqOQotTGyJBfoV1pU22ZYRNM8o24g==
In-Reply-To: <45f85f71001051129i176c7638ud173ceb82e6b61b8@mail.gmail.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 1/5/10 11:29 AM, "Todd Lipcon" <todd@cloudera.com> wrote:
> 1) Although we certainly do not guarantee wire compatibility between minor
> versions (0.20 -> 0.21) have we previously implied wire compatibility
> between bugfix releases?

IIRC, it has been implied and was a goal but not officially written anywhere
public that I know of.

> 2) Is the above something we *should* be guaranteeing already?

A) From an ops perspective, the lack of compatibility between even minor
releases is a pain.

B) Most folks with even slightly complex environments are likely patching
Hadoop.  A good chunk of those are likely in ways that breaks compatibility.
[For example, we're working on a TCP buffer patch for HDFS to fix what we
suspect is a latency problem.  Does it break compat?  Maybe.]

> 3) If we haven't guaranteed the above, how many users think we have? (ie how
> do we correctly call out this fact in the 0.20.2 release notes in such a way
> that no one gets surprised).

I suspect most folks don't even know that micros are incompatible until they
suddenly realize that distcp doesn't work.

> I can imagine plenty of organizations where a
> lockstep upgrade between client and server is difficult, and we should make
> sure that cluster operators know it will be necessary. Since it wasn't
> necessary between 0.20.0 and 0.20.1, or various 0.18 releases, people may
> have grown used to non-lockstep upgrades.

I can easily see many organizations only upgrading when it breaks due to the
Hadoop binaries being spread far and wide  and under the control of many
different departments without any sort of centralized management.

Despite the development model, I doubt few ever do just a mapred or hdfs
upgrade, so any change in the stack will likely trigger a full Hadoop
upgrade.

> 4) If the above are problems, would it be worth considering a patch for
> branch-20 that provides a client that is compatible with either, based on
> the datanode protocol version number of the server? It seems like a bit of
> scary complexity, but wanted to throw it out there.

Everyone knows I don't mind playing devil's advocate :), so let me ask the
obvious question:

Bugs are bad, etc, etc, but is it so critical that it has to be in the 0.20
branch at all?

I'd rather see the community spend cycles on 0.21 than worrying about 0.20
given that we're fast approaching the 1yr birthday of 0.20.0....


From general-return-866-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 05 22:07:14 2010
Return-Path: <general-return-866-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23126 invoked from network); 5 Jan 2010 22:07:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Jan 2010 22:07:14 -0000
Received: (qmail 27988 invoked by uid 500); 5 Jan 2010 22:07:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27902 invoked by uid 500); 5 Jan 2010 22:07:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27892 invoked by uid 99); 5 Jan 2010 22:07:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2010 22:07:13 +0000
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.62.32] (HELO QMTA03.westchester.pa.mail.comcast.net) (76.96.62.32)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2010 22:07:04 +0000
Received: from OMTA10.westchester.pa.mail.comcast.net ([76.96.62.28])
	by QMTA03.westchester.pa.mail.comcast.net with comcast
	id RoTJ1d0070cZkys53y6Zks; Tue, 05 Jan 2010 22:06:33 +0000
Received: from tryarticle-lm.corp.yahoo.com ([209.131.62.113])
	by OMTA10.westchester.pa.mail.comcast.net with comcast
	id Ry6a1d00F2SbwD53Wy6dCR; Tue, 05 Jan 2010 22:06:42 +0000
Message-Id: <8C97CCB1-3EA9-4A8C-9887-2CF4108E347D@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <45f85f71001051129i176c7638ud173ceb82e6b61b8@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: 0.20.2 HDFS incompatible with 0.20.1
Date: Tue, 5 Jan 2010 14:06:33 -0800
References: <45f85f71001051129i176c7638ud173ceb82e6b61b8@mail.gmail.com>
X-Mailer: Apple Mail (2.936)


On Jan 5, 2010, at 11:29 AM, Todd Lipcon wrote:

> 1) Although we certainly do not guarantee wire compatibility between  
> minor
> versions (0.20 -> 0.21) have we previously implied wire compatibility
> between bugfix releases?

Correction. Pre-1.0, the 0.N to 0.N+1 is a major upgrade. After 1.0,  
1.N to 1.N+1 is a minor. In both cases, X.Y.z to X.Y.z+1 is a patch  
release.

I thought we had it documented somewhere, but can't find it. There is  
some discussion of compatibility in HADOOP-5071 that should be pulled  
out into a wiki page.

The standing rules are that you don't incompatibly break APIs or wire  
protocols in patch release. So, this patch violates the rule and  
should have had a vote called before it was applied to branch-0.20.  
(And arguably branch-0.21, although since it hasn't been released, it  
isn't nearly the same level or problem.

> 2) Is the above something we *should* be guaranteeing already?

Patch releases:
   1. Must be backwards API compatible without a client recompile.
   2. Must be on the wire compatible.

Exceptions require a vote of the committers. We should also put a  
notice of any exceptions at the top of the release notes.

> 4) If the above are problems, would it be worth considering a patch  
> for
> branch-20 that provides a client that is compatible with either,  
> based on
> the datanode protocol version number of the server? It seems like a  
> bit of
> scary complexity, but wanted to throw it out there.

I would like Hairong to consider if she could fix the issue in 0.20  
without the incompatible change. If it can not be done (or no one  
wants to do the work), we should vote whether the change should be made.

-- Owen

From general-return-867-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 05 23:05:13 2010
Return-Path: <general-return-867-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46925 invoked from network); 5 Jan 2010 23:05:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Jan 2010 23:05:13 -0000
Received: (qmail 96620 invoked by uid 500); 5 Jan 2010 23:05:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96540 invoked by uid 500); 5 Jan 2010 23:05:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96522 invoked by uid 99); 5 Jan 2010 23:05:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2010 23:05:08 +0000
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2010 23:04:58 +0000
Received: from SP1-EX07CAS02.ds.corp.yahoo.com (sp1-ex07cas02.ds.corp.yahoo.com [216.252.116.138])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o05N3dnx073865;
	Tue, 5 Jan 2010 15:03:39 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:references:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:
	content-transfer-encoding:mime-version;
	b=sM87bamRQ4dC13j8kDOGiuDV9+n4X9hJol9lKr4CbV9uCBCeSnGqU2hkMa9krPHn
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 SP1-EX07CAS02.ds.corp.yahoo.com ([216.252.116.167]) with mapi; Tue, 5 Jan
 2010 15:03:39 -0800
From: Dekel Tankel <dekel@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>,
        "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Date: Tue, 5 Jan 2010 15:03:38 -0800
Subject: Hadoop User Group (Bay Area) - Jan 20th at Yahoo!
Thread-Topic: Hadoop User Group (Bay Area) - Jan 20th at Yahoo!
Thread-Index: AcoSMRRU8X5auVLKR8ufbpFTQi1hLgQXaiyQGvMSDFA=
Message-ID: <46E2672895E8744A97D7577A6110858B4FDA099349@SP1-EX07VS01.ds.corp.yahoo.com>
References: <46E2672895E8744A97D7577A6110858B4E1AC69A91@SP1-EX07VS01.ds.corp.yahoo.com> 
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi all,

Happy new year!

RSVP is now open for the first 2010 Bay Area Hadoop user group at the Yahoo=
! Sunnyvale Campus, planed for Jan 20th.

Registration is available here
http://www.meetup.com/hadoop/calendar/12229988/

Agenda will be posted soon.

Looking forward to seeing you there


Dekel


From general-return-868-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 06 00:40:42 2010
Return-Path: <general-return-868-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73973 invoked from network); 6 Jan 2010 00:40:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2010 00:40:42 -0000
Received: (qmail 66573 invoked by uid 500); 6 Jan 2010 00:40:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66486 invoked by uid 500); 6 Jan 2010 00:40:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66476 invoked by uid 99); 6 Jan 2010 00:40:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 00:40:41 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 00:40:30 +0000
Received: from oceanfarearth-lm.corp.yahoo.com (oceanfarearth-lm.corp.yahoo.com [10.72.113.156])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o060dkGb048520
	for <general@hadoop.apache.org>; Tue, 5 Jan 2010 16:39:46 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:mime-version:
	subject:date:references:x-mailer;
	b=QGAdjRNH0pQL132e5/xptvaVVQP/WQZ3S6WGkxKbETpol/CrrkUjisECUvhKuGK7
Message-Id: <1E4D49F4-98E7-438C-8467-0FE88DCD4517@yahoo-inc.com>
From: Sanjay Radia <sradia@yahoo-inc.com>
To: <general@hadoop.apache.org>
In-Reply-To: <8C97CCB1-3EA9-4A8C-9887-2CF4108E347D@apache.org>
Content-Type: multipart/alternative; boundary=Apple-Mail-3-963344818
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: 0.20.2 HDFS incompatible with 0.20.1
Date: Tue, 5 Jan 2010 16:39:46 -0800
References: <45f85f71001051129i176c7638ud173ceb82e6b61b8@mail.gmail.com> <8C97CCB1-3EA9-4A8C-9887-2CF4108E347D@apache.org>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-3-963344818
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


On Jan 5, 2010, at 2:06 PM, Owen O'Malley wrote:

>
> On Jan 5, 2010, at 11:29 AM, Todd Lipcon wrote:
>
> > 1) Although we certainly do not guarantee wire compatibility between
> > minor
> > versions (0.20 -> 0.21) have we previously implied wire  
> compatibility
> > between bugfix releases?
>
> Correction. Pre-1.0, the 0.N to 0.N+1 is a major upgrade. After 1.0,
> 1.N to 1.N+1 is a minor. In both cases, X.Y.z to X.Y.z+1 is a patch
> release.
>
> I thought we had it documented somewhere, but can't find it. There is
> some discussion of compatibility in HADOOP-5071 that should be pulled
> out into a wiki page.
>
Hadoop-5071 documents the *proposed* post 1.0 rules quite well.

>
> The standing rules are that you don't incompatibly break APIs or wire
> protocols in patch release. So, this patch violates the rule and
> should have had a vote called before it was applied to branch-0.20.
> (And arguably branch-0.21, although since it hasn't been released, it
> isn't nearly the same level or problem.
>
> > 2) Is the above something we *should* be guaranteeing already?
>
> Patch releases:
>    1. Must be backwards API compatible without a client recompile.
>    2. Must be on the wire compatible.
>
> Exceptions require a vote of the committers. We should also put a
> notice of any exceptions at the top of the release notes.
>
This is also my understanding of the current rules.
For patch releases,  the current pre-1.0 rules and the *proposed*  
post-1.0 rules in Hadoop-5071 are the same - no breakage of APIs or  
wire protocol in a patch release.



--Apple-Mail-3-963344818--

From general-return-869-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 06 01:24:44 2010
Return-Path: <general-return-869-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83389 invoked from network); 6 Jan 2010 01:24:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2010 01:24:44 -0000
Received: (qmail 2471 invoked by uid 500); 6 Jan 2010 01:24:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2380 invoked by uid 500); 6 Jan 2010 01:24:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2370 invoked by uid 99); 6 Jan 2010 01:24:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 01:24:43 +0000
X-ASF-Spam-Status: No, hits=-1.8 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of xshu@systemsbiology.org designates 64.18.2.6 as permitted sender)
Received: from [64.18.2.6] (HELO exprod7og117.obsmtp.com) (64.18.2.6)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 01:24:33 +0000
Received: from source ([209.85.223.184]) by exprod7ob117.postini.com ([64.18.6.12]) with SMTP
	ID DSNKS0PmO8CAt6SqjgYcdNGEh+8wP8Y7askn@postini.com; Tue, 05 Jan 2010 17:24:13 PST
Received: by mail-iw0-f184.google.com with SMTP id 14so215922iwn.18
        for <general@hadoop.apache.org>; Tue, 05 Jan 2010 17:24:11 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.167.65 with SMTP id p1mr718860iby.20.1262741051475; Tue, 
	05 Jan 2010 17:24:11 -0800 (PST)
In-Reply-To: <45f85f70912121301x353192du44becd5c16dce181@mail.gmail.com>
References: <28902f870912111919s3f1a86adp75635e8c23d15d69@mail.gmail.com>
	 <45f85f70912111934v2960fb36s1922507c9bc61675@mail.gmail.com>
	 <28902f870912121038g644b72b0vc832ec1343c16687@mail.gmail.com>
	 <45f85f70912121301x353192du44becd5c16dce181@mail.gmail.com>
Date: Tue, 5 Jan 2010 17:24:11 -0800
Message-ID: <28902f871001051724q24d3b173r2a5977d0fa7ab43@mail.gmail.com>
Subject: Re: Which Hadoop product is more appropriate for a quick query on a 
	large data set?
From: Xueling Shu <xshu@systemsbiology.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636d3448dd1069f047c74cf6c
X-Virus-Checked: Checked by ClamAV on apache.org

--001636d3448dd1069f047c74cf6c
Content-Type: text/plain; charset=ISO-8859-1

Hi Todd:

After finishing some tasks I finally get back to HDFS testing.

One question for your last reply to this thread: Are there any code examples
close to your second and third recommendations? Or what APIs I should start
with for my testing?

Thanks.
Xueling

On Sat, Dec 12, 2009 at 1:01 PM, Todd Lipcon <todd@cloudera.com> wrote:

> Hi Xueling,
>
> In that case, I would recommend the following:
>
> 1) Put all of your data on HDFS
> 2) Write a MapReduce job that sorts the data by position of match
> 3) As a second output of this job, you can write a "sparse index" -
> basically a set of entries like this:
>
> <position of match> <offset into file> <number of entries following>
>
> where you're basically giving offsets into every 10K records or so. If
> you index every 10K records, then 5 billion total will mean 100,000
> index entries. Each index entry shouldn't be more than 20 bytes, so
> 100,000 entries will be 2MB. This is super easy to fit into memory.
> (you could probably index every 100th record instead and end up with
> 200MB, still easy to fit in memory)
>
> Then to satisfy your count-range query, you can simply scan your
> in-memory sparse index. Some of the indexed blocks will be completely
> included in the range, in which case you just add up the "number of
> entries following" column. The start and finish block will be
> partially covered, so you can use the file offset info to load that
> file off HDFS, start reading at that offset, and finish the count.
>
> Total time per query should be <100ms no problem.
>
> -Todd
>
> On Sat, Dec 12, 2009 at 10:38 AM, Xueling Shu <xshu@systemsbiology.org>
> wrote:
> > Hi Todd:
> >
> > Thank you for your reply.
> >
> > The datasets wont be updated often. But the query against a data set is
> > frequent. The quicker the query, the better. For example we have done
> > testing on a Mysql database (5 billion records randomly scattered into 24
> > tables) and the slowest query against the biggest table (400,000,000
> > records) is around 12 mins. So if using any Hadoop product can speed up
> the
> > search then the product is what we are looking for.
> >
> > Cheers,
> > Xueling
> >
> > On Fri, Dec 11, 2009 at 7:34 PM, Todd Lipcon <todd@cloudera.com> wrote:
> >
> >> Hi Xueling,
> >>
> >> One important question that can really change the answer:
> >>
> >> How often does the dataset change? Can the changes be merged in in
> >> bulk every once in a while, or do you need to actually update them
> >> randomly very often?
> >>
> >> Also, how fast is "quick"? Do you mean 1 minute, 10 seconds, 1 second,
> or
> >> 10ms?
> >>
> >> Thanks
> >> -Todd
> >>
> >> On Fri, Dec 11, 2009 at 7:19 PM, Xueling Shu <xshu@systemsbiology.org>
> >> wrote:
> >> >  Hi there:
> >> >
> >> > I am researching Hadoop to see which of its products suits our need
> for
> >> > quick queries against large data sets (billions of records per set)
> >> >
> >> > The queries will be performed against chip sequencing data. Each
> record
> >> is
> >> > one line in a file. To be clear below shows a sample record in the
> data
> >> set.
> >> >
> >> >
> >> > one line (record) looks like: 1-1-174-418 TGTGTCCCTTTGTAATGAATCACTATC
> U2
> >> 0 0
> >> > 1 4 *103570835* F .. 23G 24C
> >> >
> >> > The highlighted field is called "position of match" and the query we
> are
> >> > interested in is the # of sequences in a certain range of this
> "position
> >> of
> >> > match". For instance the range can be "position of match" > 200 and
> >> > "position of match" + 36 < 200,000.
> >> >
> >> > Any suggestions on the Hadoop product I should start with to
> accomplish
> >> the
> >> > task? HBase,Pig,Hive, or ...?
> >> >
> >> > Thanks!
> >> >
> >> > Xueling
> >> >
> >>
> >
>

--001636d3448dd1069f047c74cf6c--

From general-return-870-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 06 01:26:55 2010
Return-Path: <general-return-870-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84360 invoked from network); 6 Jan 2010 01:26:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2010 01:26:55 -0000
Received: (qmail 4196 invoked by uid 500); 6 Jan 2010 01:26:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4105 invoked by uid 500); 6 Jan 2010 01:26:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4095 invoked by uid 99); 6 Jan 2010 01:26:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 01:26:54 +0000
X-ASF-Spam-Status: No, hits=-1.8 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of xshu@systemsbiology.org designates 64.18.2.28 as permitted sender)
Received: from [64.18.2.28] (HELO exprod7og125.obsmtp.com) (64.18.2.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 01:26:45 +0000
Received: from source ([209.85.223.182]) by exprod7ob125.postini.com ([64.18.6.12]) with SMTP
	ID DSNKS0PmwI36fdldOwl11C6OocnGIWPGWx9C@postini.com; Tue, 05 Jan 2010 17:26:25 PST
Received: by iwn12 with SMTP id 12so12149731iwn.27
        for <general@hadoop.apache.org>; Tue, 05 Jan 2010 17:26:24 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.24.142 with SMTP id v14mr654636ibb.55.1262741184399; Tue, 
	05 Jan 2010 17:26:24 -0800 (PST)
In-Reply-To: <28902f871001051724q24d3b173r2a5977d0fa7ab43@mail.gmail.com>
References: <28902f870912111919s3f1a86adp75635e8c23d15d69@mail.gmail.com>
	 <45f85f70912111934v2960fb36s1922507c9bc61675@mail.gmail.com>
	 <28902f870912121038g644b72b0vc832ec1343c16687@mail.gmail.com>
	 <45f85f70912121301x353192du44becd5c16dce181@mail.gmail.com>
	 <28902f871001051724q24d3b173r2a5977d0fa7ab43@mail.gmail.com>
Date: Tue, 5 Jan 2010 17:26:24 -0800
Message-ID: <28902f871001051726i1dc69f9fof71ac19d534a70ac@mail.gmail.com>
Subject: Re: Which Hadoop product is more appropriate for a quick query on a 
	large data set?
From: Xueling Shu <xshu@systemsbiology.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001517741884bd4b86047c74d784

--001517741884bd4b86047c74d784
Content-Type: text/plain; charset=ISO-8859-1

Rephrase the sentence "Or what APIs I should start with for my testing?": I
mean "What HDFS APIs I should start to look into for my testing?

Thanks,
Xueling

On Tue, Jan 5, 2010 at 5:24 PM, Xueling Shu <xshu@systemsbiology.org> wrote:

> Hi Todd:
>
> After finishing some tasks I finally get back to HDFS testing.
>
> One question for your last reply to this thread: Are there any code
> examples close to your second and third recommendations? Or what APIs I
> should start with for my testing?
>
> Thanks.
> Xueling
>
>
> On Sat, Dec 12, 2009 at 1:01 PM, Todd Lipcon <todd@cloudera.com> wrote:
>
>> Hi Xueling,
>>
>> In that case, I would recommend the following:
>>
>> 1) Put all of your data on HDFS
>> 2) Write a MapReduce job that sorts the data by position of match
>> 3) As a second output of this job, you can write a "sparse index" -
>> basically a set of entries like this:
>>
>> <position of match> <offset into file> <number of entries following>
>>
>> where you're basically giving offsets into every 10K records or so. If
>> you index every 10K records, then 5 billion total will mean 100,000
>> index entries. Each index entry shouldn't be more than 20 bytes, so
>> 100,000 entries will be 2MB. This is super easy to fit into memory.
>> (you could probably index every 100th record instead and end up with
>> 200MB, still easy to fit in memory)
>>
>> Then to satisfy your count-range query, you can simply scan your
>> in-memory sparse index. Some of the indexed blocks will be completely
>> included in the range, in which case you just add up the "number of
>> entries following" column. The start and finish block will be
>> partially covered, so you can use the file offset info to load that
>> file off HDFS, start reading at that offset, and finish the count.
>>
>> Total time per query should be <100ms no problem.
>>
>> -Todd
>>
>> On Sat, Dec 12, 2009 at 10:38 AM, Xueling Shu <xshu@systemsbiology.org>
>> wrote:
>> > Hi Todd:
>> >
>> > Thank you for your reply.
>> >
>> > The datasets wont be updated often. But the query against a data set is
>> > frequent. The quicker the query, the better. For example we have done
>> > testing on a Mysql database (5 billion records randomly scattered into
>> 24
>> > tables) and the slowest query against the biggest table (400,000,000
>> > records) is around 12 mins. So if using any Hadoop product can speed up
>> the
>> > search then the product is what we are looking for.
>> >
>> > Cheers,
>> > Xueling
>> >
>> > On Fri, Dec 11, 2009 at 7:34 PM, Todd Lipcon <todd@cloudera.com> wrote:
>> >
>> >> Hi Xueling,
>> >>
>> >> One important question that can really change the answer:
>> >>
>> >> How often does the dataset change? Can the changes be merged in in
>> >> bulk every once in a while, or do you need to actually update them
>> >> randomly very often?
>> >>
>> >> Also, how fast is "quick"? Do you mean 1 minute, 10 seconds, 1 second,
>> or
>> >> 10ms?
>> >>
>> >> Thanks
>> >> -Todd
>> >>
>> >> On Fri, Dec 11, 2009 at 7:19 PM, Xueling Shu <xshu@systemsbiology.org>
>> >> wrote:
>> >> >  Hi there:
>> >> >
>> >> > I am researching Hadoop to see which of its products suits our need
>> for
>> >> > quick queries against large data sets (billions of records per set)
>> >> >
>> >> > The queries will be performed against chip sequencing data. Each
>> record
>> >> is
>> >> > one line in a file. To be clear below shows a sample record in the
>> data
>> >> set.
>> >> >
>> >> >
>> >> > one line (record) looks like: 1-1-174-418 TGTGTCCCTTTGTAATGAATCACTATC
>> U2
>> >> 0 0
>> >> > 1 4 *103570835* F .. 23G 24C
>> >> >
>> >> > The highlighted field is called "position of match" and the query we
>> are
>> >> > interested in is the # of sequences in a certain range of this
>> "position
>> >> of
>> >> > match". For instance the range can be "position of match" > 200 and
>> >> > "position of match" + 36 < 200,000.
>> >> >
>> >> > Any suggestions on the Hadoop product I should start with to
>> accomplish
>> >> the
>> >> > task? HBase,Pig,Hive, or ...?
>> >> >
>> >> > Thanks!
>> >> >
>> >> > Xueling
>> >> >
>> >>
>> >
>>
>
>

--001517741884bd4b86047c74d784--

From general-return-871-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 06 01:51:27 2010
Return-Path: <general-return-871-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96946 invoked from network); 6 Jan 2010 01:51:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2010 01:51:27 -0000
Received: (qmail 24882 invoked by uid 500); 6 Jan 2010 01:51:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24812 invoked by uid 500); 6 Jan 2010 01:51:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24801 invoked by uid 99); 6 Jan 2010 01:51:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 01:51:26 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of andrew.wang.1900@gmail.com designates 209.85.219.214 as permitted sender)
Received: from [209.85.219.214] (HELO mail-ew0-f214.google.com) (209.85.219.214)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 01:51:18 +0000
Received: by ewy6 with SMTP id 6so15994207ewy.29
        for <general@hadoop.apache.org>; Tue, 05 Jan 2010 17:50:57 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=QQzOetzRQknVLMtPaQvaFwpm+RVBJvS0a3dpHVZJLFE=;
        b=Hp79pOfI48WsIUIwZ8dhhf2L00O03w+s3/mALuNgj+OEgXr2m/QfibzO3iRRxts4i3
         fTkjEPbf1WPTzIWFu+aHLPHwvVVPsDTAPRTuoeV3T0LeAXh1IxlvD6ncaJhPJpEnIV0l
         /if1a8aMNacSMWQCOJhwiQmYZSGhXDbKnsJ+c=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=iEoaY4zpCGQHsxfH3Oo4u7lwCGjY1pqPvpDMe2ms5tPqBnhV4qLigi8lLCiRlCaMV1
         syjhj6AGISgZzNl80cnf0O1JDchg7tFNMPkO6B6sWl0vajD4Myi71JPra4/fn8B9BW5e
         0UPGr90lGwlCBow0Bvy0yUAd5F2KyRnBAS/fQ=
MIME-Version: 1.0
Received: by 10.216.87.5 with SMTP id x5mr1050444wee.75.1262742657213; Tue, 05 
	Jan 2010 17:50:57 -0800 (PST)
In-Reply-To: <EFF29D2E4C964443B54746056C36D10B@jszl>
References: <EFF29D2E4C964443B54746056C36D10B@jszl>
Date: Wed, 6 Jan 2010 09:50:57 +0800
Message-ID: <995905631001051750x4c2a538cpbd048c2599462e3a@mail.gmail.com>
Subject: Re: Three questions about Hadoop
From: Andrew Wang <andrew.wang.1900@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6db2d7b86a438047c752f3b
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6db2d7b86a438047c752f3b
Content-Type: text/plain; charset=GB2312
Content-Transfer-Encoding: quoted-printable

Hi Annie,

2010/1/5 qin.wang <qin.wang@i-soft.com.cn>

> Hi team,
>
>
>
> When I try to do some research on Hadoop, I have several high level
> questions, if any comments from you it will do great help for me:
>
>
>
> 1. Hadoop assumes the files are big files, but take Google as an example,
> it
> seems the google result for user are small files, so how to understand th=
e
> big files=A3=BFAnd what=A1=AFs the file content for example?
>
> I think the "big files" means very large file (bigger than 64MB). Hadoop
use the HDFS as Distributed filesystem, the user log & web log etc are
stored in HDFS, The engineers can use Hadoop to do analysis on the logs.
Anyway, i don't know whether Google puts it's web pags in the distributed
filesystem like this.

>
> 2. Why are the files write-once and read-many times?
>
>
> As  mentioned in last section, the logs are stored in HDFS, these log are
write-once and alway used by engineers for severl times.

>
> 3. How to install other softwares on Hadoop, is there any special
> requirements for the software? Do they need to support the Map/Reduce
> module
> and then can be installed?
>
> I just don't know what you mean, maybe you would like add additional jar
which used in your application, if so, "distributed cache" in hadoop will
help you.

Good Luck!

>
> It will be very appreciated for your help.
>
>
>
> =CD=F5 =C7=D9  Annie.Wang
>
>
>
> =C9=CF=BA=A3=CA=D0=D0=EC=BB=E3=C7=F8=B9=F0=C1=D6=C2=B7418=BA=C57=BA=C5=C2=
=A56=C2=A5
> Zip code: 200 233
> Tel:      +86 21 5497 8666-8004
> Fax:     +86 21 5497 7986
> Mobile:  +86 137 6108 8369
>
>
>
>


--=20
http://anqiang1900.blog.163.com/

--0016e6db2d7b86a438047c752f3b--

From general-return-872-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 06 02:13:30 2010
Return-Path: <general-return-872-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5507 invoked from network); 6 Jan 2010 02:13:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2010 02:13:30 -0000
Received: (qmail 40304 invoked by uid 500); 6 Jan 2010 02:13:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40218 invoked by uid 500); 6 Jan 2010 02:13:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40208 invoked by uid 99); 6 Jan 2010 02:13:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 02:13:29 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of dashengju@gmail.com designates 209.85.160.50 as permitted sender)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 02:13:20 +0000
Received: by pwi20 with SMTP id 20so11489695pwi.29
        for <general@hadoop.apache.org>; Tue, 05 Jan 2010 18:12:59 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=1bN3OX7unrLmG6ah4nk7t3mQ+XZ/LmaUBQXZCTpz/7o=;
        b=tAjTgc1a33O7AD8VsGmBwmlwIGLRW5wf/K78h8Tq6goe/w26t2/Ec6V0i7enuWvcVr
         PFjrKMfxBexlIAbzZQJv5EC9vLVYHYbXPHqY2lG8X2rAJ5ElrbI+wGdVFtFZJ7nnemwz
         GOrbmnlkEuUT+1urp+3viTn6iGRodBXwQzw3w=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=Ox7P4jTbGdsM6FKq0yH5L9aBGyRjkkPn4GehWG641GeZx1SzGeuXD3iT8aKM8tTVRX
         vt6kBo6c6lWFsFrusWYVNW0fk4gfwH1xyIquwrjBedxKnojAMaJZ15E9XXOYF1al6m+R
         680nJTe5+KHm3WmZkhc8FXRImXaETzjO7vKBs=
MIME-Version: 1.0
Received: by 10.142.209.20 with SMTP id h20mr924806wfg.130.1262743979639; Tue, 
	05 Jan 2010 18:12:59 -0800 (PST)
Date: Wed, 6 Jan 2010 10:12:59 +0800
Message-ID: <1e45bf881001051812h3907c6c0w52aeae27197c1c92@mail.gmail.com>
Subject: Re:Three questions about Hadoop
From: =?UTF-8?B?6Z6g5aSn5Y2H?= <dashengju@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd32e2e594654047c757e76
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd32e2e594654047c757e76
Content-Type: text/plain; charset=GB2312
Content-Transfer-Encoding: quoted-printable

1. At the client side, one user's files are small files; but at the server
side, they will not put one user's file as a sperate file, usually they put
the same type content together, like a database. For example, the webpages
crawled from Internet are small pages, but they put them together as large
webpage data warehouse.

 2. "write-once and read-many times" is usually a charactor for data
warehouse. The webpages crawled from Internet are written to hadoop data
warehouse once, then they use those data to do many other analyse, read man=
y
times by different applications. Not all your data is "write-once and
read-many times".

 3. I do not know.

-----------------------------------
dashengju
+86 13810875910
dashengju@gmail.com

------------------ Original ------------------
  From:  "qin.wang"<qin.wang@i-soft.com.cn>;
 Date:  Tue, Jan 5, 2010 05:42 PM
 To:  "general"<general@hadoop.apache.org>;

 Subject:  Three questions about Hadoop


Hi team,



When I try to do some research on Hadoop, I have several high level
questions, if any comments from you it will do great help for me:



1. Hadoop assumes the files are big files, but take Google as an example, i=
t
seems the google result for user are small files, so how to understand the
big files=A3=BFAnd what=A1=AFs the file content for example?



2. Why are the files write-once and read-many times?



3. How to install other softwares on Hadoop, is there any special
requirements for the software? Do they need to support the Map/Reduce modul=
e
and then can be installed?



It will be very appreciated for your help.



=CD=F5 =C7=D9  Annie.Wang



=C9=CF=BA=A3=CA=D0=D0=EC=BB=E3=C7=F8=B9=F0=C1=D6=C2=B7418=BA=C57=BA=C5=C2=
=A56=C2=A5
Zip code: 200 233
Tel:      +86 21 5497 8666-8004
Fax:     +86 21 5497 7986
Mobile:  +86 137 6108 8369

--000e0cd32e2e594654047c757e76--

From general-return-873-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 06 16:10:44 2010
Return-Path: <general-return-873-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 74111 invoked from network); 6 Jan 2010 16:10:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2010 16:10:44 -0000
Received: (qmail 46399 invoked by uid 500); 6 Jan 2010 16:10:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46313 invoked by uid 500); 6 Jan 2010 16:10:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46303 invoked by uid 99); 6 Jan 2010 16:10:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 16:10:42 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 06 Jan 2010 16:10:40 +0000
Received: (qmail 73874 invoked by uid 99); 6 Jan 2010 16:10:19 -0000
Received: from localhost.apache.org (HELO [192.168.168.102]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 16:10:19 +0000
Message-ID: <4B44B5E9.3010004@apache.org>
Date: Wed, 06 Jan 2010 08:10:17 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: 0.20.2 HDFS incompatible with 0.20.1
References: <45f85f71001051129i176c7638ud173ceb82e6b61b8@mail.gmail.com> <8C97CCB1-3EA9-4A8C-9887-2CF4108E347D@apache.org>
In-Reply-To: <8C97CCB1-3EA9-4A8C-9887-2CF4108E347D@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Owen O'Malley wrote:
> Correction. Pre-1.0, the 0.N to 0.N+1 is a major upgrade. After 1.0, 1.N 
> to 1.N+1 is a minor. In both cases, X.Y.z to X.Y.z+1 is a patch release.
> 
> I thought we had it documented somewhere, but can't find it.

http://wiki.apache.org/hadoop/Roadmap

Doug

From general-return-874-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 06 18:23:45 2010
Return-Path: <general-return-874-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34147 invoked from network); 6 Jan 2010 18:23:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2010 18:23:45 -0000
Received: (qmail 75166 invoked by uid 500); 6 Jan 2010 18:23:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75089 invoked by uid 500); 6 Jan 2010 18:23:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75079 invoked by uid 99); 6 Jan 2010 18:23:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 18:23:44 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jdcryans@gmail.com designates 209.85.210.197 as permitted sender)
Received: from [209.85.210.197] (HELO mail-yx0-f197.google.com) (209.85.210.197)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 18:23:37 +0000
Received: by yxe35 with SMTP id 35so16005179yxe.2
        for <general@hadoop.apache.org>; Wed, 06 Jan 2010 10:23:17 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:message-id:subject:from:to
         :content-type;
        bh=du9/FpyPxgdYd2JhoWcBRgHjyd74OJMMeu3vs6gT0/o=;
        b=NMYvwXmkFe/OL/EEB0rDyNxk/Jfpq80aVnTIpuNHac8OHFNbGzKRv6MK9YT4TD2gjB
         8b7T0dtpHGqHaorZfoU3n+2vfAFpZmofnKGXiQdt/9TfcNXWmD3D7mWPzr64rV9v8OvV
         J5hQfBlmPXe35jpXJIncW2EDqOChlRipYuvng=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=ZmHFMCiaiC/UGtzBZ9wtQMWrr1SFxp5KRiT6i5iUZPJJu59Yio3VWaxvi8X25L744I
         YWZC5caMMm+B45YrzaUaOCNQ6j0P6EOFgNnMt53y5dxr2bNYrK+Di6W/J/yT6jtzymlh
         Xj+ZpBo9j4oBgGh9dW4WCu+lJfhJ9ryLT0UoM=
MIME-Version: 1.0
Sender: jdcryans@gmail.com
Received: by 10.91.159.2 with SMTP id l2mr1187990ago.73.1262802196850; Wed, 06 
	Jan 2010 10:23:16 -0800 (PST)
In-Reply-To: <4B44B5E9.3010004@apache.org>
References: <45f85f71001051129i176c7638ud173ceb82e6b61b8@mail.gmail.com>
	 <8C97CCB1-3EA9-4A8C-9887-2CF4108E347D@apache.org>
	 <4B44B5E9.3010004@apache.org>
Date: Wed, 6 Jan 2010 10:23:16 -0800
X-Google-Sender-Auth: b0fc03bc8e5faa0e
Message-ID: <31a243e71001061023t5ed5d245gb08c26dd51204c64@mail.gmail.com>
Subject: Re: 0.20.2 HDFS incompatible with 0.20.1
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

>From the HBase point of view, we would want to include hadoop
0.20.2-dev in hbase 0.20.3 specifically for HDFS-101 (127 would also
be nice since we could stop patching the jar we distribute). We also
share the same rules as hadoop and we don't want to break
compatibility between point releases (we try).

Our best case scenario would be to have a 0.20 backward compatible
0.20.2 release with HDFS-101 included. Else, we will just stick with
0.20.1

J-D

On Wed, Jan 6, 2010 at 8:10 AM, Doug Cutting <cutting@apache.org> wrote:
> Owen O'Malley wrote:
>>
>> Correction. Pre-1.0, the 0.N to 0.N+1 is a major upgrade. After 1.0, 1.N
>> to 1.N+1 is a minor. In both cases, X.Y.z to X.Y.z+1 is a patch release.
>>
>> I thought we had it documented somewhere, but can't find it.
>
> http://wiki.apache.org/hadoop/Roadmap
>
> Doug
>

From general-return-875-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 06 18:27:11 2010
Return-Path: <general-return-875-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 36095 invoked from network); 6 Jan 2010 18:27:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2010 18:27:11 -0000
Received: (qmail 79913 invoked by uid 500); 6 Jan 2010 18:27:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79860 invoked by uid 500); 6 Jan 2010 18:27:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 79850 invoked by uid 99); 6 Jan 2010 18:27:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 18:27:10 +0000
X-ASF-Spam-Status: No, hits=4.2 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.30.16] (HELO QMTA01.emeryville.ca.mail.comcast.net) (76.96.30.16)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 18:27:01 +0000
Received: from OMTA07.emeryville.ca.mail.comcast.net ([76.96.30.59])
	by QMTA01.emeryville.ca.mail.comcast.net with comcast
	id SEc81d0051GXsucA1JSiGr; Wed, 06 Jan 2010 18:26:42 +0000
Received: from tryarticle-lm.corp.yahoo.com ([209.131.62.113])
	by OMTA07.emeryville.ca.mail.comcast.net with comcast
	id SJSZ1d0092SbwD58TJSceS; Wed, 06 Jan 2010 18:26:40 +0000
Message-Id: <80AB0612-128B-4C3D-8873-3E7CA072FA81@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <8C97CCB1-3EA9-4A8C-9887-2CF4108E347D@apache.org>
Content-Type: multipart/alternative; boundary=Apple-Mail-1-1027351070
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: 0.20.2 HDFS incompatible with 0.20.1
Date: Wed, 6 Jan 2010 10:26:32 -0800
References: <45f85f71001051129i176c7638ud173ceb82e6b61b8@mail.gmail.com> <8C97CCB1-3EA9-4A8C-9887-2CF4108E347D@apache.org>
X-Mailer: Apple Mail (2.936)

--Apple-Mail-1-1027351070
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit

Hairong was having difficulty getting this message through the spam  
filters.

   -- Owen

----------------Start of the message:
 > I would like Hairong to consider if she could fix the issue in 0.20
 > without the incompatible change.

It is possible that I fix the issue in 0.20 without breaking the  
compatibility. But I am worried about the code stability if we take  
this approach. The pipeline code is such a critical and fragile part  
of HDFS. Currently I depend on the extensive pipeline fault injection  
tests created in 0.21 to build the confidence that my change works. If  
the change in 0.20 differs from that in 0.21 and the trunk, it becomes  
harder for us to ensure the pipeline code stability.

I would like to propose that we pull out HDFS-793 and HDFS-101 from  
0.20.

Hairong
--Apple-Mail-1-1027351070--

From general-return-876-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 06 18:31:56 2010
Return-Path: <general-return-876-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42078 invoked from network); 6 Jan 2010 18:31:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2010 18:31:56 -0000
Received: (qmail 89895 invoked by uid 500); 6 Jan 2010 18:31:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89809 invoked by uid 500); 6 Jan 2010 18:31:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89798 invoked by uid 99); 6 Jan 2010 18:31:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 18:31:55 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 209.85.210.197 as permitted sender)
Received: from [209.85.210.197] (HELO mail-yx0-f197.google.com) (209.85.210.197)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 18:31:48 +0000
Received: by yxe35 with SMTP id 35so16013008yxe.2
        for <general@hadoop.apache.org>; Wed, 06 Jan 2010 10:31:27 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=VL0DhsR7LYkiPrDsHGeMbA4FbIZBysf8s53Oxs3JMnc=;
        b=BT6lBtlgwKvoFkiLg02pqnhdAB2RBr/ZAmMxVHlyC6rAjvjxnVefN4Xcl4ne8LxvCe
         Lvlaf2gIWCX8WR8b9nbO8VoFwCzNycVuRgkjYbn04XWk+RSmfk4r6miFRd42/6EOdXdG
         /vwLggHY9MNkDpM4WvMg0oyCZ7RI6fJvhBTwU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=lGu+TEPmKmcShOLP/lnvovKgzHTkWyE0C2ZiAvT8BcOmT5dHLDLBh46GqTrYE2r5s3
         iQ3Hk8dfoWYUkxnhW4qQFuMH+FBrjeBl3W4fKezmggrKjBfRYv7Go/F1Y9mhcHdBoK4q
         9hlhEEMxFtBM/acyK+rwl6o5uZ+KDP2yVcAaY=
MIME-Version: 1.0
Received: by 10.150.17.37 with SMTP id 37mr40284665ybq.285.1262802687470; Wed, 
	06 Jan 2010 10:31:27 -0800 (PST)
In-Reply-To: <80AB0612-128B-4C3D-8873-3E7CA072FA81@apache.org>
References: <45f85f71001051129i176c7638ud173ceb82e6b61b8@mail.gmail.com>
	 <8C97CCB1-3EA9-4A8C-9887-2CF4108E347D@apache.org>
	 <80AB0612-128B-4C3D-8873-3E7CA072FA81@apache.org>
Date: Wed, 6 Jan 2010 10:31:27 -0800
Message-ID: <4aa34eb71001061031m56cf9065q586380342b81a5d1@mail.gmail.com>
Subject: Re: 0.20.2 HDFS incompatible with 0.20.1
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd757e29baca8047c832944

--000e0cd757e29baca8047c832944
Content-Type: text/plain; charset=ISO-8859-1

> I would like to propose that we pull out HDFS-793 and HDFS-101 from 0.20.

+1 to pulling it out. This code is very tricky and is dangerous to change in
a minor release.

thanks,
dhruba

--000e0cd757e29baca8047c832944--

From general-return-877-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 06 18:33:51 2010
Return-Path: <general-return-877-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42829 invoked from network); 6 Jan 2010 18:33:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2010 18:33:51 -0000
Received: (qmail 92126 invoked by uid 500); 6 Jan 2010 18:33:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92031 invoked by uid 500); 6 Jan 2010 18:33:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 92016 invoked by uid 99); 6 Jan 2010 18:33:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 18:33:50 +0000
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.30.17] (HELO QMTA10.emeryville.ca.mail.comcast.net) (76.96.30.17)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 18:33:40 +0000
Received: from OMTA02.emeryville.ca.mail.comcast.net ([76.96.30.19])
	by QMTA10.emeryville.ca.mail.comcast.net with comcast
	id SDXC1d0050QkzPwAAJZLuj; Wed, 06 Jan 2010 18:33:20 +0000
Received: from tryarticle-lm.corp.yahoo.com ([209.131.62.113])
	by OMTA02.emeryville.ca.mail.comcast.net with comcast
	id SJZA1d00H2SbwD58NJZD8w; Wed, 06 Jan 2010 18:33:17 +0000
Message-Id: <9D379087-F473-4D79-A6A0-B278DA04F9F7@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <4aa34eb71001061031m56cf9065q586380342b81a5d1@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: 0.20.2 HDFS incompatible with 0.20.1
Date: Wed, 6 Jan 2010 10:33:09 -0800
References: <45f85f71001051129i176c7638ud173ceb82e6b61b8@mail.gmail.com> <8C97CCB1-3EA9-4A8C-9887-2CF4108E347D@apache.org> <80AB0612-128B-4C3D-8873-3E7CA072FA81@apache.org> <4aa34eb71001061031m56cf9065q586380342b81a5d1@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


On Jan 6, 2010, at 10:31 AM, Dhruba Borthakur wrote:

>> I would like to propose that we pull out HDFS-793 and HDFS-101 from  
>> 0.20.
>
> +1 to pulling it out. This code is very tricky and is dangerous to  
> change in
> a minor release.

+1 to pulling it out.

-- Owen

From general-return-878-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 06 18:42:55 2010
Return-Path: <general-return-878-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53736 invoked from network); 6 Jan 2010 18:42:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2010 18:42:55 -0000
Received: (qmail 10049 invoked by uid 500); 6 Jan 2010 18:42:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 9998 invoked by uid 500); 6 Jan 2010 18:42:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 9970 invoked by uid 99); 6 Jan 2010 18:42:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 18:42:54 +0000
X-ASF-Spam-Status: No, hits=3.5 required=10.0
	tests=SPF_PASS,URIBL_BLACK
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jdcryans@gmail.com designates 209.85.211.187 as permitted sender)
Received: from [209.85.211.187] (HELO mail-yw0-f187.google.com) (209.85.211.187)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 18:42:44 +0000
Received: by ywh17 with SMTP id 17so19776674ywh.2
        for <multiple recipients>; Wed, 06 Jan 2010 10:42:24 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        bh=hrUKsttG/QtaoBaGhEXvayTN+1m8/mxQKSPE/Ms2Xgo=;
        b=E0eNP3cYGY8ku91ReVC4UwtpCJlnYRih7gA2gzi72fsSJZiGorEvEool3AYkwAGmlX
         ZtYOAwXCiovSSeoIVn1VjAIx4uBJL6B0btA6aNYS3HIHWH51pWTaOUrPFUS7/158sv93
         GFUKgpfYgan6Tea3JULFOZn7P24K7ubqJATCE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        b=Z+d8SkPJJ4mPHHtl11D1OTf9Me/aDIMRRXIJoiQ5KtR6SFRNXzGGJiAQSp5J2WYS9e
         Wy7Bx3Jrbh+h0wAk2RViHFJUzXRVBPMpUVirTxM34r6IqHmiSGcDPwkRYMFDd0kGgNnU
         isWbh8gwXCeW4CbpllnG5x5KYxmDHYT6kMybo=
MIME-Version: 1.0
Sender: jdcryans@gmail.com
Received: by 10.91.162.31 with SMTP id p31mr4707495ago.121.1262803343412; Wed, 
	06 Jan 2010 10:42:23 -0800 (PST)
Date: Wed, 6 Jan 2010 10:42:23 -0800
X-Google-Sender-Auth: fc42fb98a01010be
Message-ID: <31a243e71001061042t5da19229ha9ce7ae52868a2d6@mail.gmail.com>
Subject: SF HBase User Group Meetup Jan. 27th @ StumbleUpon
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: hbase-user@hadoop.apache.org, general@hadoop.apache.org, 
	common-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hi all,

This year's first San Francisco HBase User Group meetup takes place on
January 27th at StumbleUpon. The first talk will be about the upcoming
versions, others to be announced.

RSVP at: http://su.pr/6Cldz7

See you there!

J-D

From general-return-879-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 06 18:55:15 2010
Return-Path: <general-return-879-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57648 invoked from network); 6 Jan 2010 18:55:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2010 18:55:15 -0000
Received: (qmail 28188 invoked by uid 500); 6 Jan 2010 18:55:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28109 invoked by uid 500); 6 Jan 2010 18:55:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28099 invoked by uid 99); 6 Jan 2010 18:55:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 18:55:13 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [209.85.216.204] (HELO mail-px0-f204.google.com) (209.85.216.204)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 18:55:06 +0000
Received: by pxi42 with SMTP id 42so12735092pxi.5
        for <general@hadoop.apache.org>; Wed, 06 Jan 2010 10:54:44 -0800 (PST)
MIME-Version: 1.0
Received: by 10.142.8.34 with SMTP id 34mr16708186wfh.103.1262804082128; Wed, 
	06 Jan 2010 10:54:42 -0800 (PST)
In-Reply-To: <4aa34eb71001061031m56cf9065q586380342b81a5d1@mail.gmail.com>
References: <45f85f71001051129i176c7638ud173ceb82e6b61b8@mail.gmail.com> 
	<8C97CCB1-3EA9-4A8C-9887-2CF4108E347D@apache.org> <80AB0612-128B-4C3D-8873-3E7CA072FA81@apache.org> 
	<4aa34eb71001061031m56cf9065q586380342b81a5d1@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Wed, 6 Jan 2010 10:54:22 -0800
Message-ID: <45f85f71001061054v5198e2f2haa766032e824ca48@mail.gmail.com>
Subject: Re: 0.20.2 HDFS incompatible with 0.20.1
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502b146bc7682047c837c9c
X-Virus-Checked: Checked by ClamAV on apache.org

--00504502b146bc7682047c837c9c
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Jan 6, 2010 at 10:31 AM, Dhruba Borthakur <dhruba@gmail.com> wrote:

> > I would like to propose that we pull out HDFS-793 and HDFS-101 from 0.20.
>
> +1 to pulling it out. This code is very tricky and is dangerous to change
> in
> a minor release.
>
>
-0 to pulling it out - I agree that it's very tricky, but I think HDFS-101
is a pretty big bug to knowingly leave in. In my experience this has been
the singular cause behind a lot of HDFS write problems when a cluster has a
couple of "bad egg" nodes.

Given the number of times 0.21 has been pushed back, I think the majority of
users will be on 0.20 for some time to come, so I'd love it to be free of
known issues like this.

As for testing, I'm happy to devote time to testing an alternate version of
HDFS-101 that doesn't break compatibility - I can reproduce that bug
reliably.

That said, Hairong's opinion should carry a lot of weight, and if she thinks
it's too risky, I'm totally willing to agree.

-Todd

--00504502b146bc7682047c837c9c--

From general-return-880-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 06 19:33:26 2010
Return-Path: <general-return-880-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67804 invoked from network); 6 Jan 2010 19:33:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2010 19:33:26 -0000
Received: (qmail 71443 invoked by uid 500); 6 Jan 2010 19:33:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71371 invoked by uid 500); 6 Jan 2010 19:33:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71361 invoked by uid 99); 6 Jan 2010 19:33:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 19:33:25 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.85.216.204] (HELO mail-px0-f204.google.com) (209.85.216.204)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 19:33:18 +0000
Received: by pxi42 with SMTP id 42so12770809pxi.5
        for <general@hadoop.apache.org>; Wed, 06 Jan 2010 11:32:58 -0800 (PST)
MIME-Version: 1.0
Received: by 10.142.121.30 with SMTP id t30mr2082694wfc.283.1262806378146; 
	Wed, 06 Jan 2010 11:32:58 -0800 (PST)
In-Reply-To: <28902f871001051726i1dc69f9fof71ac19d534a70ac@mail.gmail.com>
References: <28902f870912111919s3f1a86adp75635e8c23d15d69@mail.gmail.com> 
	<45f85f70912111934v2960fb36s1922507c9bc61675@mail.gmail.com> 
	<28902f870912121038g644b72b0vc832ec1343c16687@mail.gmail.com> 
	<45f85f70912121301x353192du44becd5c16dce181@mail.gmail.com> 
	<28902f871001051724q24d3b173r2a5977d0fa7ab43@mail.gmail.com> 
	<28902f871001051726i1dc69f9fof71ac19d534a70ac@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Wed, 6 Jan 2010 11:32:38 -0800
Message-ID: <45f85f71001061132p65fde705te5957d679075de3f@mail.gmail.com>
Subject: Re: Which Hadoop product is more appropriate for a quick query on a 
	large data set?
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636e904c696ec5c047c840569

--001636e904c696ec5c047c840569
Content-Type: text/plain; charset=ISO-8859-1

Hi Xueling,

Here's a general outline:

My guess is that your "position of match" field is bounded (perhaps by the
number of base pairs in the human genome?) Given this, you can probably
write a very simple Partitioner implementation that divides this field into
ranges, each with an approximately equal number of records.

Next, write a simple MR job which takes in a line of data, and outputs the
same line, but with the position-of-match as the key. This will get
partitioned by the above function, so you end up with each reducer receiving
all of the records in a given range.

In the reducer, simply output every 1000th position into your "sparse"
output file (along with the non-sparse output file offset), and every
position into the non-sparse output file.

In your realtime query server (not part of Hadoop), load the "sparse" file
into RAM and perform binary search, etc - find the "bins" which the range
endpoints land in, and then open the non-sparse output on HDFS to finish the
count.

Hope that helps.

Thanks
-Todd

On Tue, Jan 5, 2010 at 5:26 PM, Xueling Shu <xshu@systemsbiology.org> wrote:

> Rephrase the sentence "Or what APIs I should start with for my testing?": I
> mean "What HDFS APIs I should start to look into for my testing?
>
> Thanks,
> Xueling
>
> On Tue, Jan 5, 2010 at 5:24 PM, Xueling Shu <xshu@systemsbiology.org>
> wrote:
>
> > Hi Todd:
> >
> > After finishing some tasks I finally get back to HDFS testing.
> >
> > One question for your last reply to this thread: Are there any code
> > examples close to your second and third recommendations? Or what APIs I
> > should start with for my testing?
> >
> > Thanks.
> > Xueling
> >
> >
> > On Sat, Dec 12, 2009 at 1:01 PM, Todd Lipcon <todd@cloudera.com> wrote:
> >
> >> Hi Xueling,
> >>
> >> In that case, I would recommend the following:
> >>
> >> 1) Put all of your data on HDFS
> >> 2) Write a MapReduce job that sorts the data by position of match
> >> 3) As a second output of this job, you can write a "sparse index" -
> >> basically a set of entries like this:
> >>
> >> <position of match> <offset into file> <number of entries following>
> >>
> >> where you're basically giving offsets into every 10K records or so. If
> >> you index every 10K records, then 5 billion total will mean 100,000
> >> index entries. Each index entry shouldn't be more than 20 bytes, so
> >> 100,000 entries will be 2MB. This is super easy to fit into memory.
> >> (you could probably index every 100th record instead and end up with
> >> 200MB, still easy to fit in memory)
> >>
> >> Then to satisfy your count-range query, you can simply scan your
> >> in-memory sparse index. Some of the indexed blocks will be completely
> >> included in the range, in which case you just add up the "number of
> >> entries following" column. The start and finish block will be
> >> partially covered, so you can use the file offset info to load that
> >> file off HDFS, start reading at that offset, and finish the count.
> >>
> >> Total time per query should be <100ms no problem.
> >>
> >> -Todd
> >>
> >> On Sat, Dec 12, 2009 at 10:38 AM, Xueling Shu <xshu@systemsbiology.org>
> >> wrote:
> >> > Hi Todd:
> >> >
> >> > Thank you for your reply.
> >> >
> >> > The datasets wont be updated often. But the query against a data set
> is
> >> > frequent. The quicker the query, the better. For example we have done
> >> > testing on a Mysql database (5 billion records randomly scattered into
> >> 24
> >> > tables) and the slowest query against the biggest table (400,000,000
> >> > records) is around 12 mins. So if using any Hadoop product can speed
> up
> >> the
> >> > search then the product is what we are looking for.
> >> >
> >> > Cheers,
> >> > Xueling
> >> >
> >> > On Fri, Dec 11, 2009 at 7:34 PM, Todd Lipcon <todd@cloudera.com>
> wrote:
> >> >
> >> >> Hi Xueling,
> >> >>
> >> >> One important question that can really change the answer:
> >> >>
> >> >> How often does the dataset change? Can the changes be merged in in
> >> >> bulk every once in a while, or do you need to actually update them
> >> >> randomly very often?
> >> >>
> >> >> Also, how fast is "quick"? Do you mean 1 minute, 10 seconds, 1
> second,
> >> or
> >> >> 10ms?
> >> >>
> >> >> Thanks
> >> >> -Todd
> >> >>
> >> >> On Fri, Dec 11, 2009 at 7:19 PM, Xueling Shu <
> xshu@systemsbiology.org>
> >> >> wrote:
> >> >> >  Hi there:
> >> >> >
> >> >> > I am researching Hadoop to see which of its products suits our need
> >> for
> >> >> > quick queries against large data sets (billions of records per set)
> >> >> >
> >> >> > The queries will be performed against chip sequencing data. Each
> >> record
> >> >> is
> >> >> > one line in a file. To be clear below shows a sample record in the
> >> data
> >> >> set.
> >> >> >
> >> >> >
> >> >> > one line (record) looks like: 1-1-174-418
> TGTGTCCCTTTGTAATGAATCACTATC
> >> U2
> >> >> 0 0
> >> >> > 1 4 *103570835* F .. 23G 24C
> >> >> >
> >> >> > The highlighted field is called "position of match" and the query
> we
> >> are
> >> >> > interested in is the # of sequences in a certain range of this
> >> "position
> >> >> of
> >> >> > match". For instance the range can be "position of match" > 200 and
> >> >> > "position of match" + 36 < 200,000.
> >> >> >
> >> >> > Any suggestions on the Hadoop product I should start with to
> >> accomplish
> >> >> the
> >> >> > task? HBase,Pig,Hive, or ...?
> >> >> >
> >> >> > Thanks!
> >> >> >
> >> >> > Xueling
> >> >> >
> >> >>
> >> >
> >>
> >
> >
>

--001636e904c696ec5c047c840569--

From general-return-881-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 06 19:41:36 2010
Return-Path: <general-return-881-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69874 invoked from network); 6 Jan 2010 19:41:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2010 19:41:36 -0000
Received: (qmail 81673 invoked by uid 500); 6 Jan 2010 19:41:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81594 invoked by uid 500); 6 Jan 2010 19:41:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81584 invoked by uid 99); 6 Jan 2010 19:41:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 19:41:35 +0000
X-ASF-Spam-Status: No, hits=-1.8 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of xshu@systemsbiology.org designates 64.18.2.159 as permitted sender)
Received: from [64.18.2.159] (HELO exprod7og103.obsmtp.com) (64.18.2.159)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 19:41:25 +0000
Received: from source ([209.85.223.195]) by exprod7ob103.postini.com ([64.18.6.12]) with SMTP
	ID DSNKS0TnTw/d8jRVIf9E14QeueWJwZHvK6GR@postini.com; Wed, 06 Jan 2010 11:41:04 PST
Received: by iwn33 with SMTP id 33so4332234iwn.29
        for <general@hadoop.apache.org>; Wed, 06 Jan 2010 11:41:02 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.170.14 with SMTP id b14mr912623ibz.26.1262806862704; Wed, 
	06 Jan 2010 11:41:02 -0800 (PST)
In-Reply-To: <45f85f71001061132p65fde705te5957d679075de3f@mail.gmail.com>
References: <28902f870912111919s3f1a86adp75635e8c23d15d69@mail.gmail.com>
	 <45f85f70912111934v2960fb36s1922507c9bc61675@mail.gmail.com>
	 <28902f870912121038g644b72b0vc832ec1343c16687@mail.gmail.com>
	 <45f85f70912121301x353192du44becd5c16dce181@mail.gmail.com>
	 <28902f871001051724q24d3b173r2a5977d0fa7ab43@mail.gmail.com>
	 <28902f871001051726i1dc69f9fof71ac19d534a70ac@mail.gmail.com>
	 <45f85f71001061132p65fde705te5957d679075de3f@mail.gmail.com>
Date: Wed, 6 Jan 2010 11:41:02 -0800
Message-ID: <28902f871001061141t28f26a37x120357eb90397fdb@mail.gmail.com>
Subject: Re: Which Hadoop product is more appropriate for a quick query on a 
	large data set?
From: Xueling Shu <xshu@systemsbiology.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636d34a9b78b021047c8422d8
X-Virus-Checked: Checked by ClamAV on apache.org

--001636d34a9b78b021047c8422d8
Content-Type: text/plain; charset=ISO-8859-1

Thanks for the information! I will start to try.

Xueling

On Wed, Jan 6, 2010 at 11:32 AM, Todd Lipcon <todd@cloudera.com> wrote:

> Hi Xueling,
>
> Here's a general outline:
>
> My guess is that your "position of match" field is bounded (perhaps by the
> number of base pairs in the human genome?) Given this, you can probably
> write a very simple Partitioner implementation that divides this field into
> ranges, each with an approximately equal number of records.
>
> Next, write a simple MR job which takes in a line of data, and outputs the
> same line, but with the position-of-match as the key. This will get
> partitioned by the above function, so you end up with each reducer
> receiving
> all of the records in a given range.
>
> In the reducer, simply output every 1000th position into your "sparse"
> output file (along with the non-sparse output file offset), and every
> position into the non-sparse output file.
>
> In your realtime query server (not part of Hadoop), load the "sparse" file
> into RAM and perform binary search, etc - find the "bins" which the range
> endpoints land in, and then open the non-sparse output on HDFS to finish
> the
> count.
>
> Hope that helps.
>
> Thanks
> -Todd
>
> On Tue, Jan 5, 2010 at 5:26 PM, Xueling Shu <xshu@systemsbiology.org>
> wrote:
>
> > Rephrase the sentence "Or what APIs I should start with for my testing?":
> I
> > mean "What HDFS APIs I should start to look into for my testing?
> >
> > Thanks,
> > Xueling
> >
> > On Tue, Jan 5, 2010 at 5:24 PM, Xueling Shu <xshu@systemsbiology.org>
> > wrote:
> >
> > > Hi Todd:
> > >
> > > After finishing some tasks I finally get back to HDFS testing.
> > >
> > > One question for your last reply to this thread: Are there any code
> > > examples close to your second and third recommendations? Or what APIs I
> > > should start with for my testing?
> > >
> > > Thanks.
> > > Xueling
> > >
> > >
> > > On Sat, Dec 12, 2009 at 1:01 PM, Todd Lipcon <todd@cloudera.com>
> wrote:
> > >
> > >> Hi Xueling,
> > >>
> > >> In that case, I would recommend the following:
> > >>
> > >> 1) Put all of your data on HDFS
> > >> 2) Write a MapReduce job that sorts the data by position of match
> > >> 3) As a second output of this job, you can write a "sparse index" -
> > >> basically a set of entries like this:
> > >>
> > >> <position of match> <offset into file> <number of entries following>
> > >>
> > >> where you're basically giving offsets into every 10K records or so. If
> > >> you index every 10K records, then 5 billion total will mean 100,000
> > >> index entries. Each index entry shouldn't be more than 20 bytes, so
> > >> 100,000 entries will be 2MB. This is super easy to fit into memory.
> > >> (you could probably index every 100th record instead and end up with
> > >> 200MB, still easy to fit in memory)
> > >>
> > >> Then to satisfy your count-range query, you can simply scan your
> > >> in-memory sparse index. Some of the indexed blocks will be completely
> > >> included in the range, in which case you just add up the "number of
> > >> entries following" column. The start and finish block will be
> > >> partially covered, so you can use the file offset info to load that
> > >> file off HDFS, start reading at that offset, and finish the count.
> > >>
> > >> Total time per query should be <100ms no problem.
> > >>
> > >> -Todd
> > >>
> > >> On Sat, Dec 12, 2009 at 10:38 AM, Xueling Shu <
> xshu@systemsbiology.org>
> > >> wrote:
> > >> > Hi Todd:
> > >> >
> > >> > Thank you for your reply.
> > >> >
> > >> > The datasets wont be updated often. But the query against a data set
> > is
> > >> > frequent. The quicker the query, the better. For example we have
> done
> > >> > testing on a Mysql database (5 billion records randomly scattered
> into
> > >> 24
> > >> > tables) and the slowest query against the biggest table (400,000,000
> > >> > records) is around 12 mins. So if using any Hadoop product can speed
> > up
> > >> the
> > >> > search then the product is what we are looking for.
> > >> >
> > >> > Cheers,
> > >> > Xueling
> > >> >
> > >> > On Fri, Dec 11, 2009 at 7:34 PM, Todd Lipcon <todd@cloudera.com>
> > wrote:
> > >> >
> > >> >> Hi Xueling,
> > >> >>
> > >> >> One important question that can really change the answer:
> > >> >>
> > >> >> How often does the dataset change? Can the changes be merged in in
> > >> >> bulk every once in a while, or do you need to actually update them
> > >> >> randomly very often?
> > >> >>
> > >> >> Also, how fast is "quick"? Do you mean 1 minute, 10 seconds, 1
> > second,
> > >> or
> > >> >> 10ms?
> > >> >>
> > >> >> Thanks
> > >> >> -Todd
> > >> >>
> > >> >> On Fri, Dec 11, 2009 at 7:19 PM, Xueling Shu <
> > xshu@systemsbiology.org>
> > >> >> wrote:
> > >> >> >  Hi there:
> > >> >> >
> > >> >> > I am researching Hadoop to see which of its products suits our
> need
> > >> for
> > >> >> > quick queries against large data sets (billions of records per
> set)
> > >> >> >
> > >> >> > The queries will be performed against chip sequencing data. Each
> > >> record
> > >> >> is
> > >> >> > one line in a file. To be clear below shows a sample record in
> the
> > >> data
> > >> >> set.
> > >> >> >
> > >> >> >
> > >> >> > one line (record) looks like: 1-1-174-418
> > TGTGTCCCTTTGTAATGAATCACTATC
> > >> U2
> > >> >> 0 0
> > >> >> > 1 4 *103570835* F .. 23G 24C
> > >> >> >
> > >> >> > The highlighted field is called "position of match" and the query
> > we
> > >> are
> > >> >> > interested in is the # of sequences in a certain range of this
> > >> "position
> > >> >> of
> > >> >> > match". For instance the range can be "position of match" > 200
> and
> > >> >> > "position of match" + 36 < 200,000.
> > >> >> >
> > >> >> > Any suggestions on the Hadoop product I should start with to
> > >> accomplish
> > >> >> the
> > >> >> > task? HBase,Pig,Hive, or ...?
> > >> >> >
> > >> >> > Thanks!
> > >> >> >
> > >> >> > Xueling
> > >> >> >
> > >> >>
> > >> >
> > >>
> > >
> > >
> >
>

--001636d34a9b78b021047c8422d8--

From general-return-882-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 06 20:00:37 2010
Return-Path: <general-return-882-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76131 invoked from network); 6 Jan 2010 20:00:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2010 20:00:37 -0000
Received: (qmail 7720 invoked by uid 500); 6 Jan 2010 20:00:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7657 invoked by uid 500); 6 Jan 2010 20:00:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7646 invoked by uid 99); 6 Jan 2010 20:00:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 20:00:36 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 20:00:25 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:User-Agent:Date:Subject:From:To:
   Message-ID:Thread-Topic:Thread-Index:In-Reply-To:
   Mime-version:Content-type:Content-transfer-encoding;
  b=aI3ufoWcXueaEKJ3+7ytKSITDczEu2imcSiE/F/BwkDoXM8YaoUL+Lpq
   4gfuRCH7ZqjXsXJM99f0w1mrWAjoONaod6I108pmefZ+DdLb365sAaeb6
   umHtTWmni6FMTj7;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1262808025; x=1294344025;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=200.20.2=20HDFS=20incompatible=20with=200
   .20.1|Date:=20Wed,=2006=20Jan=202010=2012:00:02=20-0800
   |Message-ID:=20<C76A2BC2.21F8%awittenauer@linkedin.com>
   |To:=20<general@hadoop.apache.org>|Mime-version:=201.0
   |Content-transfer-encoding:=207bit|In-Reply-To:=20<45f85f
   71001061054v5198e2f2haa766032e824ca48@mail.gmail.com>;
  bh=iT8z53evzSavynSf3rMZEAT/Se5I6SZUiYafc3XpWww=;
  b=tdbuHRUFshQvObiODY5w5qv2bG4yf/MCRefYnd4e2nbJeRwKJfM2Dmvf
   i9dB2J4KNt5rBMMXPt7xvSihEyjlVVkBFkWSdrGGdb/Mq/xfCxU181p6x
   P4Qzl4M3wtY40sQ;
X-IronPort-AV: E=Sophos;i="4.49,230,1262592000"; 
   d="scan'208";a="10562580"
Received: from 172.16.19.141 ([172.16.19.141]) by CORP-MAIL.linkedin.biz ([172.18.46.135]) via Exchange Front-End Server mail-access.linkedin.biz ([172.18.46.133]) with Microsoft Exchange Server HTTP-DAV ;
 Wed,  6 Jan 2010 20:00:04 +0000
User-Agent: Microsoft-Entourage/12.10.0.080409
Date: Wed, 06 Jan 2010 12:00:02 -0800
Subject: Re: 0.20.2 HDFS incompatible with 0.20.1
From: Allen Wittenauer <awittenauer@linkedin.com>
To: <general@hadoop.apache.org>
Message-ID: <C76A2BC2.21F8%awittenauer@linkedin.com>
Thread-Topic: 0.20.2 HDFS incompatible with 0.20.1
Thread-Index: AcqPCtUIWOGPlr6h70OtUNBEuKm57g==
In-Reply-To: <45f85f71001061054v5198e2f2haa766032e824ca48@mail.gmail.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org




On 1/6/10 10:54 AM, "Todd Lipcon" <todd@cloudera.com> wrote:
> -0 to pulling it out - I agree that it's very tricky, but I think HDFS-101
> is a pretty big bug to knowingly leave in. In my experience this has been
> the singular cause behind a lot of HDFS write problems when a cluster has a
> couple of "bad egg" nodes.
> 
> Given the number of times 0.21 has been pushed back, I think the majority of
> users will be on 0.20 for some time to come, so I'd love it to be free of
> known issues like this.

Due to exactly this problem (0.21 taking 4-ev-er... I think we should hold a
1-yr old birthday party for 0.20 at the appropriate HUGs), I'd like to have
a working 0.20.  I'm sure I'm not the only one willing to forgo back-compat,
but with a caveat: The fact that an upgrade requirement needs to be obvious.
Can we change the message of "protocol violation" or whatever it is to add
"Perhaps you need to upgrade your client?" or something?


From general-return-883-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 06 20:04:50 2010
Return-Path: <general-return-883-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77084 invoked from network); 6 Jan 2010 20:04:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2010 20:04:49 -0000
Received: (qmail 10988 invoked by uid 500); 6 Jan 2010 20:04:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 10904 invoked by uid 500); 6 Jan 2010 20:04:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 10894 invoked by uid 99); 6 Jan 2010 20:04:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 20:04:48 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [195.232.224.72] (HELO mailout03.vodafone.com) (195.232.224.72)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 20:04:38 +0000
Received: from mailint03 (localhost [127.0.0.1])
	by mailout03 (Postfix) with ESMTP id C4C4911643B
	for <general@hadoop.apache.org>; Wed,  6 Jan 2010 21:04:16 +0100 (CET)
Received: from avoexs04.internal.vodafone.com (avoexs04.dc-ratingen.de [145.230.4.198])
	(using TLSv1 with cipher RC4-MD5 (128/128 bits))
	(No client certificate requested)
	by mailint03 (Postfix) with ESMTPS id B5F9511636F
	for <general@hadoop.apache.org>; Wed,  6 Jan 2010 21:04:16 +0100 (CET)
Received: from VF-MBX13.internal.vodafone.com ([145.230.5.24]) by avoexs04.internal.vodafone.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Wed, 6 Jan 2010 21:04:18 +0100
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Subject: RE: Which Hadoop product is more appropriate for a quick query on a large data set?
Date: Wed, 6 Jan 2010 21:00:01 +0100
Message-ID: <219D8244D980254ABF28AB469AD4E98F03469FCB@VF-MBX13.internal.vodafone.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Which Hadoop product is more appropriate for a quick query on a large data set?
Thread-Index: AcqPCERS3vDThaXlTGSsLtgXKGtq4gAApBwn
References: <28902f870912111919s3f1a86adp75635e8c23d15d69@mail.gmail.com><45f85f70912111934v2960fb36s1922507c9bc61675@mail.gmail.com><28902f870912121038g644b72b0vc832ec1343c16687@mail.gmail.com><45f85f70912121301x353192du44becd5c16dce181@mail.gmail.com><28902f871001051724q24d3b173r2a5977d0fa7ab43@mail.gmail.com><28902f871001051726i1dc69f9fof71ac19d534a70ac@mail.gmail.com><45f85f71001061132p65fde705te5957d679075de3f@mail.gmail.com> <28902f871001061141t28f26a37x120357eb90397fdb@mail.gmail.com>
From: "Gibbon, Robert, VF-Group" <Robert.Gibbon@vodafone.com>
To: <general@hadoop.apache.org>,
	<general@hadoop.apache.org>
X-OriginalArrivalTime: 06 Jan 2010 20:04:18.0140 (UTC) FILETIME=[6DB3E9C0:01CA8F0B]
X-Virus-Checked: Checked by ClamAV on apache.org


Isn't this what Hadoop Hbase is supposed to do? The partioning M/R =
implementation - "sharding" in street - is the sideways scaling that =
Hbase is designed to excel at! Also the indexed hbase flavour could =
allow very fast ad-hoc queries for Xueling using the new yet familiar =
HBQL sql-dialect?

Sorry, my 10 pence worth!

-----Original Message-----
From: Xueling Shu [mailto:xshu@systemsbiology.org]
Sent: Wed 1/6/2010 8:41 PM
To: general@hadoop.apache.org
Subject: Re: Which Hadoop product is more appropriate for a quick query =
on a large data set?
=20
Thanks for the information! I will start to try.

Xueling

On Wed, Jan 6, 2010 at 11:32 AM, Todd Lipcon <todd@cloudera.com> wrote:

> Hi Xueling,
>
> Here's a general outline:
>
> My guess is that your "position of match" field is bounded (perhaps by =
the
> number of base pairs in the human genome?) Given this, you can =
probably
> write a very simple Partitioner implementation that divides this field =
into
> ranges, each with an approximately equal number of records.
>
> Next, write a simple MR job which takes in a line of data, and outputs =
the
> same line, but with the position-of-match as the key. This will get
> partitioned by the above function, so you end up with each reducer
> receiving
> all of the records in a given range.
>
> In the reducer, simply output every 1000th position into your "sparse"
> output file (along with the non-sparse output file offset), and every
> position into the non-sparse output file.
>
> In your realtime query server (not part of Hadoop), load the "sparse" =
file
> into RAM and perform binary search, etc - find the "bins" which the =
range
> endpoints land in, and then open the non-sparse output on HDFS to =
finish
> the
> count.
>
> Hope that helps.
>
> Thanks
> -Todd
>
> On Tue, Jan 5, 2010 at 5:26 PM, Xueling Shu <xshu@systemsbiology.org>
> wrote:
>
> > Rephrase the sentence "Or what APIs I should start with for my =
testing?":
> I
> > mean "What HDFS APIs I should start to look into for my testing?
> >
> > Thanks,
> > Xueling
> >
> > On Tue, Jan 5, 2010 at 5:24 PM, Xueling Shu =
<xshu@systemsbiology.org>
> > wrote:
> >
> > > Hi Todd:
> > >
> > > After finishing some tasks I finally get back to HDFS testing.
> > >
> > > One question for your last reply to this thread: Are there any =
code
> > > examples close to your second and third recommendations? Or what =
APIs I
> > > should start with for my testing?
> > >
> > > Thanks.
> > > Xueling
> > >
> > >
> > > On Sat, Dec 12, 2009 at 1:01 PM, Todd Lipcon <todd@cloudera.com>
> wrote:
> > >
> > >> Hi Xueling,
> > >>
> > >> In that case, I would recommend the following:
> > >>
> > >> 1) Put all of your data on HDFS
> > >> 2) Write a MapReduce job that sorts the data by position of match
> > >> 3) As a second output of this job, you can write a "sparse index" =
-
> > >> basically a set of entries like this:
> > >>
> > >> <position of match> <offset into file> <number of entries =
following>
> > >>
> > >> where you're basically giving offsets into every 10K records or =
so. If
> > >> you index every 10K records, then 5 billion total will mean =
100,000
> > >> index entries. Each index entry shouldn't be more than 20 bytes, =
so
> > >> 100,000 entries will be 2MB. This is super easy to fit into =
memory.
> > >> (you could probably index every 100th record instead and end up =
with
> > >> 200MB, still easy to fit in memory)
> > >>
> > >> Then to satisfy your count-range query, you can simply scan your
> > >> in-memory sparse index. Some of the indexed blocks will be =
completely
> > >> included in the range, in which case you just add up the "number =
of
> > >> entries following" column. The start and finish block will be
> > >> partially covered, so you can use the file offset info to load =
that
> > >> file off HDFS, start reading at that offset, and finish the =
count.
> > >>
> > >> Total time per query should be <100ms no problem.
> > >>
> > >> -Todd
> > >>
> > >> On Sat, Dec 12, 2009 at 10:38 AM, Xueling Shu <
> xshu@systemsbiology.org>
> > >> wrote:
> > >> > Hi Todd:
> > >> >
> > >> > Thank you for your reply.
> > >> >
> > >> > The datasets wont be updated often. But the query against a =
data set
> > is
> > >> > frequent. The quicker the query, the better. For example we =
have
> done
> > >> > testing on a Mysql database (5 billion records randomly =
scattered
> into
> > >> 24
> > >> > tables) and the slowest query against the biggest table =
(400,000,000
> > >> > records) is around 12 mins. So if using any Hadoop product can =
speed
> > up
> > >> the
> > >> > search then the product is what we are looking for.
> > >> >
> > >> > Cheers,
> > >> > Xueling
> > >> >
> > >> > On Fri, Dec 11, 2009 at 7:34 PM, Todd Lipcon =
<todd@cloudera.com>
> > wrote:
> > >> >
> > >> >> Hi Xueling,
> > >> >>
> > >> >> One important question that can really change the answer:
> > >> >>
> > >> >> How often does the dataset change? Can the changes be merged =
in in
> > >> >> bulk every once in a while, or do you need to actually update =
them
> > >> >> randomly very often?
> > >> >>
> > >> >> Also, how fast is "quick"? Do you mean 1 minute, 10 seconds, 1
> > second,
> > >> or
> > >> >> 10ms?
> > >> >>
> > >> >> Thanks
> > >> >> -Todd
> > >> >>
> > >> >> On Fri, Dec 11, 2009 at 7:19 PM, Xueling Shu <
> > xshu@systemsbiology.org>
> > >> >> wrote:
> > >> >> >  Hi there:
> > >> >> >
> > >> >> > I am researching Hadoop to see which of its products suits =
our
> need
> > >> for
> > >> >> > quick queries against large data sets (billions of records =
per
> set)
> > >> >> >
> > >> >> > The queries will be performed against chip sequencing data. =
Each
> > >> record
> > >> >> is
> > >> >> > one line in a file. To be clear below shows a sample record =
in
> the
> > >> data
> > >> >> set.
> > >> >> >
> > >> >> >
> > >> >> > one line (record) looks like: 1-1-174-418
> > TGTGTCCCTTTGTAATGAATCACTATC
> > >> U2
> > >> >> 0 0
> > >> >> > 1 4 *103570835* F .. 23G 24C
> > >> >> >
> > >> >> > The highlighted field is called "position of match" and the =
query
> > we
> > >> are
> > >> >> > interested in is the # of sequences in a certain range of =
this
> > >> "position
> > >> >> of
> > >> >> > match". For instance the range can be "position of match" > =
200
> and
> > >> >> > "position of match" + 36 < 200,000.
> > >> >> >
> > >> >> > Any suggestions on the Hadoop product I should start with to
> > >> accomplish
> > >> >> the
> > >> >> > task? HBase,Pig,Hive, or ...?
> > >> >> >
> > >> >> > Thanks!
> > >> >> >
> > >> >> > Xueling
> > >> >> >
> > >> >>
> > >> >
> > >>
> > >
> > >
> >
>


From general-return-884-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 06 21:38:13 2010
Return-Path: <general-return-884-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5011 invoked from network); 6 Jan 2010 21:38:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2010 21:38:13 -0000
Received: (qmail 38048 invoked by uid 500); 6 Jan 2010 21:38:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37968 invoked by uid 500); 6 Jan 2010 21:38:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37958 invoked by uid 99); 6 Jan 2010 21:38:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 21:38:12 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2010 21:38:01 +0000
Received: from oceanfarearth-lm.corp.yahoo.com (oceanfarearth-lm.corp.yahoo.com [10.72.113.156])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o06Lap6x033526
	for <general@hadoop.apache.org>; Wed, 6 Jan 2010 13:36:51 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=iAQYdh2H4F1tYXai0KQLLrI+zcRR67Ui68taUOnoWzsyU+FCr0P8SlL99ESeByXN
Message-Id: <58EECB55-3433-443A-A186-83B3E773C5DA@yahoo-inc.com>
From: Sanjay Radia <sradia@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <80AB0612-128B-4C3D-8873-3E7CA072FA81@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: 0.20.2 HDFS incompatible with 0.20.1
Date: Wed, 6 Jan 2010 13:36:50 -0800
References: <45f85f71001051129i176c7638ud173ceb82e6b61b8@mail.gmail.com> <8C97CCB1-3EA9-4A8C-9887-2CF4108E347D@apache.org> <80AB0612-128B-4C3D-8873-3E7CA072FA81@apache.org>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

>
>
> ----------------Start of the message:
> > I would like Hairong to consider if she could fix the issue in 0.20
> > without the incompatible change.
>
> It is possible that I fix the issue in 0.20 without breaking the  
> compatibility. But I am worried about the code stability if we take  
> this approach. The pipeline code is such a critical and fragile part  
> of HDFS. Currently I depend on the extensive pipeline fault  
> injection tests created in 0.21 to build the confidence that my  
> change works. If the change in 0.20 differs from that in 0.21 and  
> the trunk, it becomes harder for us to ensure the pipeline code  
> stability.
>
> I would like to propose that we pull out HDFS-793 and HDFS-101 from  
> 0.20.
>
> Hairong


+1 to pull it out.
This part of the code is trick to get right and test.

sanjay

From general-return-885-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 07 16:59:39 2010
Return-Path: <general-return-885-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53819 invoked from network); 7 Jan 2010 16:59:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jan 2010 16:59:39 -0000
Received: (qmail 5254 invoked by uid 500); 7 Jan 2010 16:59:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5194 invoked by uid 500); 7 Jan 2010 16:59:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 5818 invoked by uid 99); 7 Jan 2010 13:46:41 -0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of sriramc@ivycomptech.com designates 203.153.210.130 as permitted sender)
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.3790.4073
Content-Class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: Hbase loading error -- Trailer 'header' is  wrong; does the trailer size match content
Date: Thu, 7 Jan 2010 19:19:55 +0530
Message-ID: <49A7BA114AAC6A48B9C44CB06B7B987E089AFCD7@HYDSVWIN004X.ivycomptech.partygaming.local>
In-Reply-To: <31a243e71001061042t5da19229ha9ce7ae52868a2d6@mail.gmail.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Hbase loading error -- Trailer 'header' is  wrong; does the trailer size match content
thread-index: AcqPACQoCNXoT5PcRse71LxjBUltEQAnpBjg
References: <31a243e71001061042t5da19229ha9ce7ae52868a2d6@mail.gmail.com>
From: "Sriram Muthuswamy Chittathoor" <sriramc@ivycomptech.com>
To: <hbase-user@hadoop.apache.org>,<general@hadoop.apache.org>,<common-user@hadoop.apache.org>
CC: <stack@duboce.net>
X-OriginalArrivalTime: 07 Jan 2010 13:46:03.0844 (UTC) FILETIME=[C142E840:01CA8F9F]
X-Virus-Checked: Checked by ClamAV on apache.org

Hi:

I am trying to run a MR job to output HFiles directly containing 10
million records (very simple 1 column family and very small).  The job
completes with some mention about killed jobs (reduce Failed/Killed
Task Attempts > 0) .  Then I use the script loadtable.rb to load my
hfiles into hbase and get the error stack given below.=20

I tried all combinations of settings in the mapred-site.xml. Also tried
the suggestions given in this chain which talks about similar problem.
http://www.mail-archive.com/hbase-user@hadoop.apache.org/msg07668.html


If I reduce the number of records it works. =20

Thanks for any help

Sriram C



org/apache/hadoop/hbase/io/hfile/HFile.java:1335:in `deserialize':
java.io.IOException: Trailer 'header' is  wrong; does the trailer size
match content? (NativeException)
        from org/apache/hadoop/hbase/io/hfile/HFile.java:813:in
`readTrailer'
        from org/apache/hadoop/hbase/io/hfile/HFile.java:758:in
`loadFileInfo'
        from sun/reflect/NativeMethodAccessorImpl.java:-2:in `invoke0'
        from sun/reflect/NativeMethodAccessorImpl.java:39:in `invoke'
        from sun/reflect/DelegatingMethodAccessorImpl.java:25:in
`invoke'
        from java/lang/reflect/Method.java:597:in `invoke'
        from org/jruby/javasupport/JavaMethod.java:298:in
`invokeWithExceptionHandling'
        from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
         ... 19 levels...
        from org/jruby/Main.java:94:in `main'
        from bin/loadtable.rb:83:in `each'
        from bin/loadtable.rb:83
Complete Java stackTrace
java.io.IOException: Trailer 'header' is wrong; does the trailer size
match content?
        at
org.apache.hadoop.hbase.io.hfile.HFile$FixedFileTrailer.deserialize(HFil
e.java:1335)
        at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readTrailer(HFile.java:813
)
        at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.loadFileInfo(HFile.java:75
8)



This email is sent for and on behalf of Ivy Comptech Private Limited. =
Ivy Comptech Private Limited is a limited liability company. =20

This email and any attachments are confidential, and may be legally =
privileged and protected by copyright. If you are not the intended =
recipient dissemination or copying of this email is prohibited. If you =
have received this in error, please notify the sender by replying by =
email and then delete the email completely from your system.=20
Any views or opinions are solely those of the sender.  This =
communication is not intended to form a binding contract on behalf of =
Ivy Comptech Private Limited unless expressly indicated to the contrary =
and properly authorised. Any actions taken on the basis of this email =
are at the recipient's own risk.

Registered office:
Ivy Comptech Private Limited, Cyber Spazio, Road No. 2, Banjara Hills, =
Hyderabad 500 033, Andhra Pradesh, India. Registered number: 37994. =
Registered in India. A list of members' names is available for =
inspection at the registered office.


From general-return-886-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 08 23:25:07 2010
Return-Path: <general-return-886-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76436 invoked from network); 8 Jan 2010 23:25:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jan 2010 23:25:07 -0000
Received: (qmail 17208 invoked by uid 500); 8 Jan 2010 23:25:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17138 invoked by uid 500); 8 Jan 2010 23:25:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17128 invoked by uid 99); 8 Jan 2010 23:25:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Jan 2010 23:25:05 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=HTML_MESSAGE,SPF_PASS,SUBJECT_FUZZY_TION,SUBJ_ALL_CAPS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cabiwan@gmail.com designates 72.14.220.154 as permitted sender)
Received: from [72.14.220.154] (HELO fg-out-1718.google.com) (72.14.220.154)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Jan 2010 23:24:59 +0000
Received: by fg-out-1718.google.com with SMTP id 19so6506625fgg.11
        for <general@hadoop.apache.org>; Fri, 08 Jan 2010 15:24:37 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=lUtaqKfujsX9rhVWnQnRL/tZKr9eyezC/39DqnhS7K8=;
        b=tJR1WFwuFSUfw8BrSCAt23wjVY/NDbnXPOb7af0x65dtLlVtA0ImK6gJJMJcz0496b
         9JeYe1iDsKx9K7fbbuirfOJ3eE/rCYGCTyPNEZrrjNqjDQj8FsTuB2X0yXV+eoP+xyt3
         eHwb7R26xQO4FmUerZZLtuHfQlTx2xPCos08A=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=NLJ1MytBYGD1fPuuy79F7qmmN13gnOSG7JWb5LwhyaUifbrDBZprDOXfksI4G5FUP2
         SYP4AZV1eoGqKv3ZJ8vMquF58P69nkGHhdD8kIgFTlk6iF3LGVfQ37EsJE42tf9yKUTE
         QzO+MBltcvvBpKRPXJiSE/b+CIRe6CSMQMe7g=
MIME-Version: 1.0
Received: by 10.103.126.21 with SMTP id d21mr4337265mun.24.1262993077641; Fri, 
	08 Jan 2010 15:24:37 -0800 (PST)
Date: Sat, 9 Jan 2010 00:24:37 +0100
Message-ID: <309f76d01001081524t45faa45vf54640e503d67b5@mail.gmail.com>
Subject: QUESTION ABOUT PARTITIONERS
From: Alberto Luengo Cabanillas <cabiwan@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e65c8d40bf22f9047caf7d54

--0016e65c8d40bf22f9047caf7d54
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi everyone! First of all, happy new year to all community!. I=B4m actually
developing an Map-Reduce approach to genetic algorithms, and I was
questioning the use and effectivity of Partitioners in the Map-Reduce flow.
I=B4ve been working in pseudo-distributed mode and didn=B4t find any exampl=
e of
usage of Partitioner classes. Does this mean that using another Partitioner
-not the default one- have sense only on a Fully Distributed environment?
Which Partitioner should I implement  with my GA approach?
Thanks a lot in advance!

--=20
Alberto

--0016e65c8d40bf22f9047caf7d54--

From general-return-887-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 08 23:28:02 2010
Return-Path: <general-return-887-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77190 invoked from network); 8 Jan 2010 23:28:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jan 2010 23:28:02 -0000
Received: (qmail 20923 invoked by uid 500); 8 Jan 2010 23:28:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20849 invoked by uid 500); 8 Jan 2010 23:28:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 20839 invoked by uid 99); 8 Jan 2010 23:28:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Jan 2010 23:28:01 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates 209.85.216.204 as permitted sender)
Received: from [209.85.216.204] (HELO mail-px0-f204.google.com) (209.85.216.204)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Jan 2010 23:27:53 +0000
Received: by pxi42 with SMTP id 42so14509615pxi.5
        for <general@hadoop.apache.org>; Fri, 08 Jan 2010 15:27:31 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=MLt0d1mMPVvy8DcY1ACgazbtYjcgGfxmOiLrgs6jk94=;
        b=EU4g9aeLFf0VZ62qnh8ySXnxMxi4czmsS8mhPaN2+0UjKmQG9OlTqwfN2uBxwsN1GW
         UFv7ENW6ECFkS/vIXlewf1HGfwXAjQ70LiJ911DJCKJvWfN2mFgJuWU2bDOdJrpSyTx8
         EXE5pLs+Sk4pJox6aIHyb+ZBuiPIu2Ann91tc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=iKtwXr3mT17m1iXiKHS3dcRAZ4Y5xP6n6UWjMpOXIZO7FjkfPTN0hKGENvzLjg8V3/
         x5iqe08MHhNj1wZ5DsEajPBwX7d60AhPJENuIQiaqUXeW5GpNQJgfltvtbtpre3r0nBQ
         +LtDkzPeTs3dUFFnete7p+A2DOp2shA/DNlmw=
MIME-Version: 1.0
Received: by 10.143.20.36 with SMTP id x36mr1629804wfi.231.1262993251253; Fri, 
	08 Jan 2010 15:27:31 -0800 (PST)
Date: Fri, 8 Jan 2010 15:27:31 -0800
Message-ID: <17e273101001081527y67aaff5dj86da8478dbb78950@mail.gmail.com>
Subject: last call to map()
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636e1f95d184044047caf88f9
X-Virus-Checked: Checked by ClamAV on apache.org

--001636e1f95d184044047caf88f9
Content-Type: text/plain; charset=ISO-8859-1

Hi,
My mapper calls remote client to retrieve certain result.

I plan to introduce asynchronous client calls and buffer results in mapper
before calling output.collect().

Is there a way to detect whether the current call to MapClass.map() is the
last one ?

Thanks

--001636e1f95d184044047caf88f9--

From general-return-888-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 08 23:32:10 2010
Return-Path: <general-return-888-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78062 invoked from network); 8 Jan 2010 23:32:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jan 2010 23:32:09 -0000
Received: (qmail 23008 invoked by uid 500); 8 Jan 2010 23:32:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22922 invoked by uid 500); 8 Jan 2010 23:32:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22898 invoked by uid 99); 8 Jan 2010 23:32:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Jan 2010 23:32:08 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Jan 2010 23:31:59 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o08NVYZj043543;
	Fri, 8 Jan 2010 15:31:34 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=cc:message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=drnWe3feuVP/fOYVJ67RGeQ/pbOAoHJ3PCk0uuYyxgLplWwkG2nn6MfZ/0Ku7QMR
Cc: general@hadoop.apache.org
Message-Id: <3B3421D1-AF67-405C-BC55-AAD79B45BA53@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: mapreduce-user@hadoop.apache.org
In-Reply-To: <17e273101001081527y67aaff5dj86da8478dbb78950@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: last call to map()
Date: Fri, 8 Jan 2010 15:31:34 -0800
References: <17e273101001081527y67aaff5dj86da8478dbb78950@mail.gmail.com>
X-Mailer: Apple Mail (2.936)

Moving to mapreduce-user@

Use the 'close' api.

Arun

On Jan 8, 2010, at 3:27 PM, Ted Yu wrote:

> Hi,
> My mapper calls remote client to retrieve certain result.
>
> I plan to introduce asynchronous client calls and buffer results in  
> mapper
> before calling output.collect().
>
> Is there a way to detect whether the current call to MapClass.map()  
> is the
> last one ?
>
> Thanks


From general-return-889-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jan 09 00:47:35 2010
Return-Path: <general-return-889-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 645 invoked from network); 9 Jan 2010 00:47:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jan 2010 00:47:35 -0000
Received: (qmail 69490 invoked by uid 500); 9 Jan 2010 00:47:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69406 invoked by uid 500); 9 Jan 2010 00:47:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69396 invoked by uid 99); 9 Jan 2010 00:47:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 09 Jan 2010 00:47:34 +0000
X-ASF-Spam-Status: No, hits=2.6 required=10.0
	tests=HTML_MESSAGE,SPF_PASS,SUBJECT_FUZZY_TION
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of vermaabhishekp@gmail.com designates 209.85.220.215 as permitted sender)
Received: from [209.85.220.215] (HELO mail-fx0-f215.google.com) (209.85.220.215)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 09 Jan 2010 00:47:26 +0000
Received: by fxm7 with SMTP id 7so18227708fxm.29
        for <general@hadoop.apache.org>; Fri, 08 Jan 2010 16:47:04 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=dLsTRTsxwyG+sNQnkhUTbXgy/u0UsqgFFqiIKfg7fkg=;
        b=Jsf7iy4ktCgCZ2vIvodGUOYbjUr5ofFC1QQIYtZGL3ldpEen9EPibE91Cj/MBEsLrc
         /mcAQYhLDjyhYoGA2FZ7yzqzLsJgEZ55DUwqmn//ICF4r2pAem9WpZ7T/RUdDcR1ClNU
         bTxT9PNLpuF2meVf2Lc+Quet6rof9h3y3GPus=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=F75E2NQh6sIoj14woSlcEgd6Z8zSJ/HmjydDDjvk5olUTEAJY3nvKp4ok8k0wjQJ2L
         N3S6MKfCCDlQQRGJt3cL0E6AEEKUGVVjIeWXRDhKBc10ldTcsf+8XxOVQfCCVLVI2a/I
         XmICV8RAivquTlaTSwRP+nKiLvLl9LHUHEdUo=
MIME-Version: 1.0
Received: by 10.239.183.81 with SMTP id t17mr2000436hbg.24.1262998023978; Fri, 
	08 Jan 2010 16:47:03 -0800 (PST)
In-Reply-To: <309f76d01001081524t45faa45vf54640e503d67b5@mail.gmail.com>
References: <309f76d01001081524t45faa45vf54640e503d67b5@mail.gmail.com>
Date: Fri, 8 Jan 2010 18:47:03 -0600
Message-ID: <3c682ecd1001081647x14d68f76h9d4eda3ef3502d4c@mail.gmail.com>
Subject: Re: QUESTION ABOUT PARTITIONERS
From: Abhishek Verma <vermaabhishekp@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f7c4c4923e28047cb0a4be

--001485f7c4c4923e28047cb0a4be
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Alberto,

You should be able to override the default partitioner class in pseudo
distributed and the fully distributed mode. I used the same technique for
the paper "Scaling Genetic Algorithms using MapReduce" [
http://verma7.com/wp/wp-content/uploads/2009/10/ISDA09_MR_GA.pdf] that you
might find useful.

-Abhishek.

On Fri, Jan 8, 2010 at 5:24 PM, Alberto Luengo Cabanillas <cabiwan@gmail.co=
m
> wrote:

> Hi everyone! First of all, happy new year to all community!. I=B4m actual=
ly
> developing an Map-Reduce approach to genetic algorithms, and I was
> questioning the use and effectivity of Partitioners in the Map-Reduce flo=
w.
> I=B4ve been working in pseudo-distributed mode and didn=B4t find any exam=
ple of
> usage of Partitioner classes. Does this mean that using another Partition=
er
> -not the default one- have sense only on a Fully Distributed environment?
> Which Partitioner should I implement  with my GA approach?
> Thanks a lot in advance!
>
> --
> Alberto
>

--001485f7c4c4923e28047cb0a4be--

From general-return-890-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jan 09 19:34:39 2010
Return-Path: <general-return-890-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 7257 invoked from network); 9 Jan 2010 19:34:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jan 2010 19:34:39 -0000
Received: (qmail 36583 invoked by uid 500); 9 Jan 2010 19:34:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36494 invoked by uid 500); 9 Jan 2010 19:34:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36483 invoked by uid 99); 9 Jan 2010 19:34:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 09 Jan 2010 19:34:37 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of andrew.wang.1900@gmail.com designates 209.85.219.212 as permitted sender)
Received: from [209.85.219.212] (HELO mail-ew0-f212.google.com) (209.85.219.212)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 09 Jan 2010 19:34:29 +0000
Received: by ewy4 with SMTP id 4so21745833ewy.12
        for <general@hadoop.apache.org>; Sat, 09 Jan 2010 11:34:09 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=MdnBbErf1gCsV2fOzM0ejhZf3k6b7BivT8Mn4SHUKuM=;
        b=mhWQ4nBFo7w6VoqMrsiyMtB1+zmuXQV/uPiJXl0hx4u4Z1kzJIJxfVVgBfCDdm3z+r
         shXSYcZwulp6EK4MOq+hAC/YwGvzKtffswKl6Q0Ioxezj6baX9n9vgkbROCfwWwhjA2X
         rCh7r9z5kxdiPeC7XpilYXiWs0nks9z9KKx8Y=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=hNn7nPdK/GsPBf0X4g1cNWM5MovoaEmJRt1OgrDtF4tADsnmNOraudGPmV9uYMkgAx
         GDRjci4xiQRg61W/pIMYLh3oEhdpqAl+t9DDPuvzv7e/zFb9nRWUfOxFuLNMHGVRVwVx
         sey89DfEvU9oj6dwOK++y3UaSATMJlKUl3hR8=
MIME-Version: 1.0
Received: by 10.216.86.14 with SMTP id v14mr1062035wee.183.1263065648489; Sat, 
	09 Jan 2010 11:34:08 -0800 (PST)
Date: Sun, 10 Jan 2010 03:34:08 +0800
Message-ID: <995905631001091134h39a703a6u1f962d551f7025a@mail.gmail.com>
Subject: about translating Hadoop:The Definitive Guide into chinese one
From: Andrew Wang <andrew.wang.1900@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d7eb684e6519047cc0638f
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d7eb684e6519047cc0638f
Content-Type: text/plain; charset=ISO-8859-1

Hi, you guys

I am interested in Hadoop very much and using it in my current project. I
find this book, Hadoop:The Definitive Guide, is a useful and practical
introduction for new coming Hadoop users. As, there are also many
applications using hadoop here in china, many more people would like to know
what hadoop is and what it can do and how it works. So, maybe translating
one technical book about it into chinese will be helpful.

I am working on with my friend and just finish  for chapter1 and chapter6.
The most important thing i'd like to know is whether you guys think this
job/work will be useful or helpful?

I will appreciate your suggestions very much! Thanks!


-- 
http://anqiang1900.blog.163.com/

--0016e6d7eb684e6519047cc0638f--

From general-return-891-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jan 09 21:46:29 2010
Return-Path: <general-return-891-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35454 invoked from network); 9 Jan 2010 21:46:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jan 2010 21:46:29 -0000
Received: (qmail 18131 invoked by uid 500); 9 Jan 2010 21:46:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18041 invoked by uid 500); 9 Jan 2010 21:46:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18031 invoked by uid 99); 9 Jan 2010 21:46:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 09 Jan 2010 21:46:28 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of andrey.v.kuzmin@gmail.com designates 209.85.220.215 as permitted sender)
Received: from [209.85.220.215] (HELO mail-fx0-f215.google.com) (209.85.220.215)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 09 Jan 2010 21:46:21 +0000
Received: by fxm7 with SMTP id 7so18678345fxm.29
        for <general@hadoop.apache.org>; Sat, 09 Jan 2010 13:46:00 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=gVZhZEgigDs0BuIZunDn3amHynMsewOd+aJjZ40VKlo=;
        b=Kci0015HNWoRrewSWRTb1Gr4w46FswNvAqKqXvsI7CzIuBeg9LiXvayAhxOwd+fMHe
         Gl1xsI4itkKWynewTjPV5IpbTkMgGZcj345vrP+GqORxkPkRd8XCm6JjV9cf5ZWqyUe4
         lOm9c5kSOKSEcTa6foXjacDb6UGxNuXT5pmJU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=P+BHQ/tI9vVQH5EqHAcyQpHTxR7uvXtuco/VJjbhpwR1yn02I6EN4Ee3zy9ebz/iPB
         5BlG9YBtJ78FJT7ymTW6Emu+ThgNzFvOz4lDc14Ffoh1YBQxiX/DqEV9B2X/IHnEwkxd
         +Ah9qx0PHOcwb7XNim5j8Z2k2mORNdzCg9n+s=
MIME-Version: 1.0
Received: by 10.239.186.197 with SMTP id i5mr718370hbh.103.1263073559260; Sat, 
	09 Jan 2010 13:45:59 -0800 (PST)
In-Reply-To: <995905631001091134h39a703a6u1f962d551f7025a@mail.gmail.com>
References: <995905631001091134h39a703a6u1f962d551f7025a@mail.gmail.com>
From: Andrey Kuzmin <andrey.v.kuzmin@gmail.com>
Date: Sun, 10 Jan 2010 00:45:39 +0300
Message-ID: <2a31deca1001091345u6d9c0f67h7ed57a605adce99@mail.gmail.com>
Subject: Re: about translating Hadoop:The Definitive Guide into chinese one
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Same question here with Russian translation.

Regards,
Andrey




On Sat, Jan 9, 2010 at 10:34 PM, Andrew Wang <andrew.wang.1900@gmail.com> w=
rote:
> Hi, you guys
>
> I am interested in Hadoop very much and using it in my current project. I
> find this book, Hadoop:The Definitive Guide, is a useful and practical
> introduction for new coming Hadoop users. As, there are also many
> applications using hadoop here in china, many more people would like to k=
now
> what hadoop is and what it can do and how it works. So, maybe translating
> one technical book about it into chinese will be helpful.
>
> I am working on with my friend and just finish =A0for chapter1 and chapte=
r6.
> The most important thing i'd like to know is whether you guys think this
> job/work will be useful or helpful?
>
> I will appreciate your suggestions very much! Thanks!
>
>
> --
> http://anqiang1900.blog.163.com/
>

From general-return-892-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jan 10 02:21:18 2010
Return-Path: <general-return-892-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5636 invoked from network); 10 Jan 2010 02:21:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jan 2010 02:21:18 -0000
Received: (qmail 21140 invoked by uid 500); 10 Jan 2010 02:21:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21065 invoked by uid 500); 10 Jan 2010 02:21:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21055 invoked by uid 99); 10 Jan 2010 02:21:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Jan 2010 02:21:17 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_SOFTFAIL
X-Spam-Check-By: apache.org
Received-SPF: softfail (nike.apache.org: transitioning domain of tamagawa@osa.att.ne.jp does not designate 211.124.127.8 as permitted sender)
Received: from [211.124.127.8] (HELO ougw02.zaq.ne.jp) (211.124.127.8)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Jan 2010 02:21:06 +0000
Received: from mlrc01.zaq.ne.jp (mlrc01.zaq-back.ne.jp [10.2.92.33])
	by ougw02.zaq.ne.jp with ESMTP id 836141CC003
	for <general@hadoop.apache.org>;
	Sun, 10 Jan 2010 11:20:43 +0900 (JST)
Received: from ouma02.zaq-back.ne.jp (HELO ouma02.zaq.ne.jp) ([10.2.80.34])
  by mlrc01.zaq.ne.jp with ESMTP; 10 Jan 2010 11:20:45 +0900
Received: from [127.0.0.1] (zaqdb73c332.zaq.ne.jp [219.115.195.50])
	by ouma02.zaq.ne.jp with ESMTP id 5CB20724147;
	Sun, 10 Jan 2010 11:20:43 +0900 (JST)
Message-ID: <4B49397B.1050804@osa.att.ne.jp>
Date: Sun, 10 Jan 2010 11:20:43 +0900
From: tamagawa <tamagawa@osa.att.ne.jp>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; ja; rv:1.9.1.5) Gecko/20091204 Thunderbird/3.0
MIME-Version: 1.0
To: general@hadoop.apache.org
CC: Andrew Wang <andrew.wang.1900@gmail.com>
Subject: Re: about translating Hadoop:The Definitive Guide into chinese one
References: <995905631001091134h39a703a6u1f962d551f7025a@mail.gmail.com>
In-Reply-To: <995905631001091134h39a703a6u1f962d551f7025a@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Antivirus: avast! (VPS 100109-1, 2010/01/09), Outbound message
X-Antivirus-Status: Clean
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

I've done the Japanese translation and it will be available from Oreilly 
Japan soon. I believe it is 'Definitely' helpful.

There is an Oreilly office in china:

http://oreilly.com/contact.html

I think you can contact them to see if there is a plan for the 
translation and/or you can work as a translator for the book.

Regards,
--
TAMAGAWA Ryuji


(2010/01/10 4:34), Andrew Wang wrote:
> Hi, you guys
>
> I am interested in Hadoop very much and using it in my current project. I
> find this book, Hadoop:The Definitive Guide, is a useful and practical
> introduction for new coming Hadoop users. As, there are also many
> applications using hadoop here in china, many more people would like to know
> what hadoop is and what it can do and how it works. So, maybe translating
> one technical book about it into chinese will be helpful.
>
> I am working on with my friend and just finish  for chapter1 and chapter6.
> The most important thing i'd like to know is whether you guys think this
> job/work will be useful or helpful?
>
> I will appreciate your suggestions very much! Thanks!
>
>

From general-return-893-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jan 10 06:16:40 2010
Return-Path: <general-return-893-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 48502 invoked from network); 10 Jan 2010 06:16:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jan 2010 06:16:40 -0000
Received: (qmail 87286 invoked by uid 500); 10 Jan 2010 06:16:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87206 invoked by uid 500); 10 Jan 2010 06:16:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87196 invoked by uid 99); 10 Jan 2010 06:16:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Jan 2010 06:16:38 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of zjffdu@gmail.com designates 209.85.160.50 as permitted sender)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Jan 2010 06:16:32 +0000
Received: by pwi20 with SMTP id 20so881705pwi.29
        for <general@hadoop.apache.org>; Sat, 09 Jan 2010 22:16:11 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=gXR/gdfkMzU6cnHk/ZCeLRN0bToH1BV9Gw8+mc4c/Xc=;
        b=guJBQQ7u66pmLbSHWAACsiJTgzWH1j5a8WmCPZjm1mYn9gRYpUYyCrIDnw+RyN94Ye
         S5U/espfkpk2tdHMwdmaxy11PfuiJxU4ygJ1JsVFFVG+sVA7MUdoZP8zxV/rhWSS3nHQ
         s5v57lINlaqZXWAfQ64G+jnQNQ01GZw+ABu/k=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=WmerLVWKrN2fUIqNhVppq1rpSXdCQYcZsm9xE7/akqXw/wKIag+mYZuoxk5+wsyTOk
         jK/FjPVjuVW0ODSgDUQfvGbS5BE2sz5nL39NtfJTbpE4owxtDHkBpPp3o504gFxKVbaQ
         aDB16SH7Dxuhzt4kt/N58qZBoHMWJlQ6Z6DeE=
MIME-Version: 1.0
Received: by 10.142.250.21 with SMTP id x21mr1830998wfh.169.1263104171675; 
	Sat, 09 Jan 2010 22:16:11 -0800 (PST)
In-Reply-To: <995905631001091134h39a703a6u1f962d551f7025a@mail.gmail.com>
References: <995905631001091134h39a703a6u1f962d551f7025a@mail.gmail.com>
Date: Sat, 9 Jan 2010 22:16:11 -0800
Message-ID: <8211a1321001092216l58ff0226he5d4047d2bd9538@mail.gmail.com>
Subject: Re: about translating Hadoop:The Definitive Guide into chinese one
From: Jeff Zhang <zjffdu@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636ed657c778f48047cc95ba6

--001636ed657c778f48047cc95ba6
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi Wang=EF=BC=8C

I am very interested in this. This book really help me a lot when I go deep
into hadoop.
Maybe I can help with chapter 11 Pig. I am now pig's committer.




On Sat, Jan 9, 2010 at 11:34 AM, Andrew Wang <andrew.wang.1900@gmail.com>wr=
ote:

> Hi, you guys
>
> I am interested in Hadoop very much and using it in my current project. I
> find this book, Hadoop:The Definitive Guide, is a useful and practical
> introduction for new coming Hadoop users. As, there are also many
> applications using hadoop here in china, many more people would like to
> know
> what hadoop is and what it can do and how it works. So, maybe translating
> one technical book about it into chinese will be helpful.
>
> I am working on with my friend and just finish  for chapter1 and chapter6=
.
> The most important thing i'd like to know is whether you guys think this
> job/work will be useful or helpful?
>
> I will appreciate your suggestions very much! Thanks!
>
>
> --
> http://anqiang1900.blog.163.com/
>



--=20
Best Regards

Jeff Zhang

--001636ed657c778f48047cc95ba6--

From general-return-894-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jan 10 06:32:38 2010
Return-Path: <general-return-894-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54463 invoked from network); 10 Jan 2010 06:32:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jan 2010 06:32:38 -0000
Received: (qmail 89952 invoked by uid 500); 10 Jan 2010 06:32:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89867 invoked by uid 500); 10 Jan 2010 06:32:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89849 invoked by uid 99); 10 Jan 2010 06:32:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Jan 2010 06:32:37 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of andrew.wang.1900@gmail.com designates 209.85.219.212 as permitted sender)
Received: from [209.85.219.212] (HELO mail-ew0-f212.google.com) (209.85.219.212)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Jan 2010 06:32:29 +0000
Received: by ewy4 with SMTP id 4so22014973ewy.12
        for <general@hadoop.apache.org>; Sat, 09 Jan 2010 22:32:08 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=ujMbdsXIBSqksyIPzPVBqcaPH2KtQh2UFJv1+KS22Rs=;
        b=iPi39H5NDv4+OGKU5AgFE07q5g9cPNCb2HeDxRkvGXmh8pyg/SFTVBof6djYfWW22P
         GxAd9mx/NeUaAhykbQ1WlPq4X0FezcB/eOdPRSB7gSaKtBGnokVBUgincl8cZIkHxyL2
         xuHMDGd0wJPemh2478hns3IdRaog4hD3PvUwE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=jP/X57uWy5XqMWfjS7umjcxeFhs41/s1i++ejYHyAFK5i6TaMkWoAtle+hcUPN3zuY
         lyzCbJdDPaP0jqZn1hv/q3tGeVHKJpk9ZIi+WFZlG7xCtv9dgwarqLk322ThCAThCr/+
         15d6AJdabbuN6JThIbKf/V943fmBt3Qlzrv/8=
MIME-Version: 1.0
Received: by 10.216.87.131 with SMTP id y3mr286190wee.9.1263105127997; Sat, 09 
	Jan 2010 22:32:07 -0800 (PST)
In-Reply-To: <8211a1321001092216l58ff0226he5d4047d2bd9538@mail.gmail.com>
References: <995905631001091134h39a703a6u1f962d551f7025a@mail.gmail.com>
	 <8211a1321001092216l58ff0226he5d4047d2bd9538@mail.gmail.com>
Date: Sun, 10 Jan 2010 14:32:07 +0800
Message-ID: <995905631001092232g7cdd1f0dwe20e09b70d7e1363@mail.gmail.com>
Subject: Re: about translating Hadoop:The Definitive Guide into chinese one
From: Andrew Wang <andrew.wang.1900@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d77e5e77e2df047cc99431
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d77e5e77e2df047cc99431
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi, Zhang

it's nice to meet you. I have seen you in this maillist for serval times. M=
y
friend and i are translating this book together. Glod to have you join us.

However, i am not sure whether we can get the offical authoriation from
Orelly. So, before we get it,  we can share the transilated Chapters with
each other. Is it OK?

What is your opinion?

On Sun, Jan 10, 2010 at 2:16 PM, Jeff Zhang <zjffdu@gmail.com> wrote:

> Hi Wang=EF=BC=8C
>
> I am very interested in this. This book really help me a lot when I go de=
ep
> into hadoop.
> Maybe I can help with chapter 11 Pig. I am now pig's committer.
>
>
>
>
> On Sat, Jan 9, 2010 at 11:34 AM, Andrew Wang <andrew.wang.1900@gmail.com
> >wrote:
>
> > Hi, you guys
> >
> > I am interested in Hadoop very much and using it in my current project.=
 I
> > find this book, Hadoop:The Definitive Guide, is a useful and practical
> > introduction for new coming Hadoop users. As, there are also many
> > applications using hadoop here in china, many more people would like to
> > know
> > what hadoop is and what it can do and how it works. So, maybe translati=
ng
> > one technical book about it into chinese will be helpful.
> >
> > I am working on with my friend and just finish  for chapter1 and
> chapter6.
> > The most important thing i'd like to know is whether you guys think thi=
s
> > job/work will be useful or helpful?
> >
> > I will appreciate your suggestions very much! Thanks!
> >
> >
> > --
> > http://anqiang1900.blog.163.com/
> >
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>



--=20
http://anqiang1900.blog.163.com/

--0016e6d77e5e77e2df047cc99431--

From general-return-895-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jan 10 06:35:56 2010
Return-Path: <general-return-895-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54713 invoked from network); 10 Jan 2010 06:35:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jan 2010 06:35:56 -0000
Received: (qmail 90900 invoked by uid 500); 10 Jan 2010 06:35:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90811 invoked by uid 500); 10 Jan 2010 06:35:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90801 invoked by uid 99); 10 Jan 2010 06:35:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Jan 2010 06:35:55 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of zjffdu@gmail.com designates 209.85.222.195 as permitted sender)
Received: from [209.85.222.195] (HELO mail-pz0-f195.google.com) (209.85.222.195)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Jan 2010 06:35:48 +0000
Received: by pzk33 with SMTP id 33so5983482pzk.2
        for <general@hadoop.apache.org>; Sat, 09 Jan 2010 22:35:28 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=Z5GUZjCds0dU+6unOlXpYb9abgYbuQEc0+mmUe5sRJk=;
        b=ZnPsRfrCxw99Uhr3IhD1bn31SmpM4mXWfQgYR5mjfSNsi+G8L8W+9G++I7XCNN1J0+
         84TmzbUCxIVddjyKN2iUiFXsIb4AcP4eHz3e+mJJSrOBLtND45pW/Kl75PHC58Zzz96v
         SPQUQyvMa1iZvV79fWr8ACVWAjqj92HoaSlPQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=duvSSXSoEG7rjqa0f/2YbbuiyNPEYzJEn8VVyo9FooTxnBUxXI5chNsHf0mVizFzbh
         ID0ARpwAgWHJZaMb59q6kj7ythB+FVN2+J2ahA+5dJjxINtu0Sw9KpLBudq8JK+Z6Vk3
         ztK1IyceivPgYeHgR5mwlRlFNBVTI4U810laY=
MIME-Version: 1.0
Received: by 10.142.60.8 with SMTP id i8mr2693727wfa.310.1263105328449; Sat, 
	09 Jan 2010 22:35:28 -0800 (PST)
In-Reply-To: <995905631001092232g7cdd1f0dwe20e09b70d7e1363@mail.gmail.com>
References: <995905631001091134h39a703a6u1f962d551f7025a@mail.gmail.com>
	 <8211a1321001092216l58ff0226he5d4047d2bd9538@mail.gmail.com>
	 <995905631001092232g7cdd1f0dwe20e09b70d7e1363@mail.gmail.com>
Date: Sat, 9 Jan 2010 22:35:28 -0800
Message-ID: <8211a1321001092235w178fd066n9eb8953c7efbc50@mail.gmail.com>
Subject: Re: about translating Hadoop:The Definitive Guide into chinese one
From: Jeff Zhang <zjffdu@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502af146a89ae047cc9a06b

--00504502af146a89ae047cc9a06b
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi Wang,

This is good, I agree with the plan. And this is msn: zjfzju@hotmail.com
We can talk using IM later.



On Sat, Jan 9, 2010 at 10:32 PM, Andrew Wang <andrew.wang.1900@gmail.com>wr=
ote:

> Hi, Zhang
>
> it's nice to meet you. I have seen you in this maillist for serval times.
> My
> friend and i are translating this book together. Glod to have you join us=
.
>
> However, i am not sure whether we can get the offical authoriation from
> Orelly. So, before we get it,  we can share the transilated Chapters with
> each other. Is it OK?
>
> What is your opinion?
>
> On Sun, Jan 10, 2010 at 2:16 PM, Jeff Zhang <zjffdu@gmail.com> wrote:
>
> > Hi Wang=EF=BC=8C
> >
> > I am very interested in this. This book really help me a lot when I go
> deep
> > into hadoop.
> > Maybe I can help with chapter 11 Pig. I am now pig's committer.
> >
> >
> >
> >
> > On Sat, Jan 9, 2010 at 11:34 AM, Andrew Wang <andrew.wang.1900@gmail.co=
m
> > >wrote:
> >
> > > Hi, you guys
> > >
> > > I am interested in Hadoop very much and using it in my current projec=
t.
> I
> > > find this book, Hadoop:The Definitive Guide, is a useful and practica=
l
> > > introduction for new coming Hadoop users. As, there are also many
> > > applications using hadoop here in china, many more people would like =
to
> > > know
> > > what hadoop is and what it can do and how it works. So, maybe
> translating
> > > one technical book about it into chinese will be helpful.
> > >
> > > I am working on with my friend and just finish  for chapter1 and
> > chapter6.
> > > The most important thing i'd like to know is whether you guys think
> this
> > > job/work will be useful or helpful?
> > >
> > > I will appreciate your suggestions very much! Thanks!
> > >
> > >
> > > --
> > > http://anqiang1900.blog.163.com/
> > >
> >
> >
> >
> > --
> > Best Regards
> >
> > Jeff Zhang
> >
>
>
>
> --
> http://anqiang1900.blog.163.com/
>



--=20
Best Regards

Jeff Zhang

--00504502af146a89ae047cc9a06b--

From general-return-896-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jan 10 06:45:56 2010
Return-Path: <general-return-896-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 61797 invoked from network); 10 Jan 2010 06:45:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jan 2010 06:45:56 -0000
Received: (qmail 93920 invoked by uid 500); 10 Jan 2010 06:45:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93861 invoked by uid 500); 10 Jan 2010 06:45:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93851 invoked by uid 99); 10 Jan 2010 06:45:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Jan 2010 06:45:55 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yeqian.zju@gmail.com designates 209.85.223.193 as permitted sender)
Received: from [209.85.223.193] (HELO mail-iw0-f193.google.com) (209.85.223.193)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Jan 2010 06:45:48 +0000
Received: by iwn31 with SMTP id 31so14717821iwn.5
        for <general@hadoop.apache.org>; Sat, 09 Jan 2010 22:45:28 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=cKzYvrUWlpeU9fZi/lUQsb0jCHf1U7HMNXhyXen+8l4=;
        b=pUAaCAZtZn1aCv1iSFKXXax3ivMgpmHCthIZx/l7tyJL1o+cTTG9F/GBQOJNk84uNg
         JiHqGCYbvJnlbJePb8IY8+TBtITUMjsPmtoheCIxptp231hVNkx1diYnND+7NElYrIhq
         CX+Pt2dLivNsK1rZSBAjONau70r9tA3VBKTOQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=H+Zk8mcJHEXVUYKRQxaghJakW/yHMxYNChh/pv2rHZJOWJvvcJ++45xUXiMw3yW47j
         emKwDK0rUCHKIt9YPmuTY41BW5Rv5n8n7/pkfS8ocp1XEOZhS4RqL0m8rlnOUt2/WwAU
         YoPNIiDY4UBkVNV3tuq3tm3/Jsh9myNgIRlvk=
MIME-Version: 1.0
Received: by 10.231.120.136 with SMTP id d8mr2480017ibr.14.1263105928181; Sat, 
	09 Jan 2010 22:45:28 -0800 (PST)
In-Reply-To: <8211a1321001092216l58ff0226he5d4047d2bd9538@mail.gmail.com>
References: <995905631001091134h39a703a6u1f962d551f7025a@mail.gmail.com>
	 <8211a1321001092216l58ff0226he5d4047d2bd9538@mail.gmail.com>
Date: Sun, 10 Jan 2010 14:45:28 +0800
Message-ID: <c3eaa9191001092245y1eb933e3ifbb9385f6020330b@mail.gmail.com>
Subject: Re: about translating Hadoop:The Definitive Guide into chinese one
From: Qian Ye <yeqian.zju@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e644c18e29bdf1047cc9c45f

--0016e644c18e29bdf1047cc9c45f
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

So am I, although i'm just a novice to hadoop. I worked on a project based
on the Zookeeper these monthes. I would like to make some help for the
Chapter 13, if need.

thx~

On Sun, Jan 10, 2010 at 2:16 PM, Jeff Zhang <zjffdu@gmail.com> wrote:

> Hi Wang=EF=BC=8C
>
> I am very interested in this. This book really help me a lot when I go de=
ep
> into hadoop.
> Maybe I can help with chapter 11 Pig. I am now pig's committer.
>
>
>
>
> On Sat, Jan 9, 2010 at 11:34 AM, Andrew Wang <andrew.wang.1900@gmail.com
> >wrote:
>
> > Hi, you guys
> >
> > I am interested in Hadoop very much and using it in my current project.=
 I
> > find this book, Hadoop:The Definitive Guide, is a useful and practical
> > introduction for new coming Hadoop users. As, there are also many
> > applications using hadoop here in china, many more people would like to
> > know
> > what hadoop is and what it can do and how it works. So, maybe translati=
ng
> > one technical book about it into chinese will be helpful.
> >
> > I am working on with my friend and just finish  for chapter1 and
> chapter6.
> > The most important thing i'd like to know is whether you guys think thi=
s
> > job/work will be useful or helpful?
> >
> > I will appreciate your suggestions very much! Thanks!
> >
> >
> > --
> > http://anqiang1900.blog.163.com/
> >
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>



--=20
With Regards!

Ye, Qian
Made in Zhejiang University

--0016e644c18e29bdf1047cc9c45f--

From general-return-897-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jan 10 07:21:23 2010
Return-Path: <general-return-897-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66286 invoked from network); 10 Jan 2010 07:21:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jan 2010 07:21:23 -0000
Received: (qmail 6402 invoked by uid 500); 10 Jan 2010 07:21:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6322 invoked by uid 500); 10 Jan 2010 07:21:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6312 invoked by uid 99); 10 Jan 2010 07:21:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Jan 2010 07:21:22 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.222.195] (HELO mail-pz0-f195.google.com) (209.85.222.195)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Jan 2010 07:21:08 +0000
Received: by pzk33 with SMTP id 33so5997102pzk.2
        for <general@hadoop.apache.org>; Sat, 09 Jan 2010 23:20:43 -0800 (PST)
Received: by 10.115.66.29 with SMTP id t29mr13810228wak.187.1263108043049;
        Sat, 09 Jan 2010 23:20:43 -0800 (PST)
Received: from pcpcpcda585799 ([168.115.119.118])
        by mx.google.com with ESMTPS id 20sm20218285pzk.13.2010.01.09.23.20.41
        (version=SSLv3 cipher=RC4-MD5);
        Sat, 09 Jan 2010 23:20:42 -0800 (PST)
From: "hadoop" <hadoop@jimey.org>
To: <general@hadoop.apache.org>
References: <995905631001091134h39a703a6u1f962d551f7025a@mail.gmail.com> <8211a1321001092216l58ff0226he5d4047d2bd9538@mail.gmail.com> <c3eaa9191001092245y1eb933e3ifbb9385f6020330b@mail.gmail.com>
In-Reply-To: <c3eaa9191001092245y1eb933e3ifbb9385f6020330b@mail.gmail.com>
Subject: re about translating Hadoop:The Definitive Guide into chinese one
Date: Sun, 10 Jan 2010 16:26:44 +0900
Message-ID: <010b01ca91c6$444c8ca0$cce5a5e0$@org>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Mailer: Microsoft Office Outlook 12.0
Content-Language: zh-cn
Thread-Index: AcqRwI90yZSow5p5QY6+etYLGacCcwABVa3A
X-Virus-Checked: Checked by ClamAV on apache.org

Hi, There is an unreleased Chinese version about this book

Check it http://www.china-pub.com/196200


Best Regards

Jimey Ren







So am I, although i'm just a novice to hadoop. I worked on a project =
based
on the Zookeeper these monthes. I would like to make some help for the
Chapter 13, if need.

thx~

On Sun, Jan 10, 2010 at 2:16 PM, Jeff Zhang <zjffdu@gmail.com> wrote:

> Hi Wang=EF=BC=8C
>
> I am very interested in this. This book really help me a lot when I go =
deep
> into hadoop.
> Maybe I can help with chapter 11 Pig. I am now pig's committer.
>
>
>
>
> On Sat, Jan 9, 2010 at 11:34 AM, Andrew Wang =
<andrew.wang.1900@gmail.com
> >wrote:
>
> > Hi, you guys
> >
> > I am interested in Hadoop very much and using it in my current =
project. I
> > find this book, Hadoop:The Definitive Guide, is a useful and =
practical
> > introduction for new coming Hadoop users. As, there are also many
> > applications using hadoop here in china, many more people would like =
to
> > know
> > what hadoop is and what it can do and how it works. So, maybe =
translating
> > one technical book about it into chinese will be helpful.
> >
> > I am working on with my friend and just finish  for chapter1 and
> chapter6.
> > The most important thing i'd like to know is whether you guys think =
this
> > job/work will be useful or helpful?
> >
> > I will appreciate your suggestions very much! Thanks!
> >
> >
> > --
> > http://anqiang1900.blog.163.com/
> >
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>



--=20
With Regards!

Ye, Qian
Made in Zhejiang University


From general-return-898-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jan 10 08:10:08 2010
Return-Path: <general-return-898-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70909 invoked from network); 10 Jan 2010 08:10:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jan 2010 08:10:07 -0000
Received: (qmail 20924 invoked by uid 500); 10 Jan 2010 08:10:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20859 invoked by uid 500); 10 Jan 2010 08:10:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 20849 invoked by uid 99); 10 Jan 2010 08:10:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Jan 2010 08:10:06 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yangxiao9901@gmail.com designates 209.85.222.195 as permitted sender)
Received: from [209.85.222.195] (HELO mail-pz0-f195.google.com) (209.85.222.195)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Jan 2010 08:09:53 +0000
Received: by pzk33 with SMTP id 33so6010739pzk.2
        for <general@hadoop.apache.org>; Sun, 10 Jan 2010 00:09:32 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=vq2rSBVqemHN8thyY5ZLReZYp+ilCzdtJXFGDx3RHdE=;
        b=sC3j0paEgcnVsvBpiUkqsrqKNniGWhd7khqWQ/54CEb/kWMK4GmCLLKAZ4w+xS+UWr
         TytPAswrNzyeU6p/MzfQst3S6Og/gfNZ45mnrg25o0dYR9sIhpDg4/v/FCu3iO6gdFI7
         J+3Z7RcHMK/W9Qv3CHk0cjPTx8pcSW8BjwwYQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=smWqMG+QGyNgHtdQ2vk4oRfDb60wJBtkpjr2V8Guu2FsaMad/rDNcG/qv/eOI1rMhv
         pGBBbj0W4DyAgn2IzkfSUaGnWeBFTGFmuo+u8HbIa5OR+DKfHGOqrvpXvyf8h/vGhBuN
         X+R7+F4INT1MbbWS+NeKxP+ynTM0xqZL2jg/o=
MIME-Version: 1.0
Received: by 10.140.55.16 with SMTP id d16mr5559294rva.279.1263110971997; Sun, 
	10 Jan 2010 00:09:31 -0800 (PST)
In-Reply-To: <c3eaa9191001092245y1eb933e3ifbb9385f6020330b@mail.gmail.com>
References: <995905631001091134h39a703a6u1f962d551f7025a@mail.gmail.com>
	 <8211a1321001092216l58ff0226he5d4047d2bd9538@mail.gmail.com>
	 <c3eaa9191001092245y1eb933e3ifbb9385f6020330b@mail.gmail.com>
Date: Sun, 10 Jan 2010 16:09:31 +0800
Message-ID: <5a921af21001100009g27cd859dw4aa80a99fac24d3b@mail.gmail.com>
Subject: Re: about translating Hadoop:The Definitive Guide into chinese one
From: xiao yang <yangxiao9901@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

We can create a project on Yeeyan (www.yeeyan.org).

On Sun, Jan 10, 2010 at 2:45 PM, Qian Ye <yeqian.zju@gmail.com> wrote:
> So am I, although i'm just a novice to hadoop. I worked on a project base=
d
> on the Zookeeper these monthes. I would like to make some help for the
> Chapter 13, if need.
>
> thx~
>
> On Sun, Jan 10, 2010 at 2:16 PM, Jeff Zhang <zjffdu@gmail.com> wrote:
>
>> Hi Wang=EF=BC=8C
>>
>> I am very interested in this. This book really help me a lot when I go d=
eep
>> into hadoop.
>> Maybe I can help with chapter 11 Pig. I am now pig's committer.
>>
>>
>>
>>
>> On Sat, Jan 9, 2010 at 11:34 AM, Andrew Wang <andrew.wang.1900@gmail.com
>> >wrote:
>>
>> > Hi, you guys
>> >
>> > I am interested in Hadoop very much and using it in my current project=
. I
>> > find this book, Hadoop:The Definitive Guide, is a useful and practical
>> > introduction for new coming Hadoop users. As, there are also many
>> > applications using hadoop here in china, many more people would like t=
o
>> > know
>> > what hadoop is and what it can do and how it works. So, maybe translat=
ing
>> > one technical book about it into chinese will be helpful.
>> >
>> > I am working on with my friend and just finish =C2=A0for chapter1 and
>> chapter6.
>> > The most important thing i'd like to know is whether you guys think th=
is
>> > job/work will be useful or helpful?
>> >
>> > I will appreciate your suggestions very much! Thanks!
>> >
>> >
>> > --
>> > http://anqiang1900.blog.163.com/
>> >
>>
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>
>
>
> --
> With Regards!
>
> Ye, Qian
> Made in Zhejiang University
>

From general-return-899-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jan 10 17:30:29 2010
Return-Path: <general-return-899-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12610 invoked from network); 10 Jan 2010 17:30:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jan 2010 17:30:29 -0000
Received: (qmail 76812 invoked by uid 500); 10 Jan 2010 17:30:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76743 invoked by uid 500); 10 Jan 2010 17:30:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76733 invoked by uid 99); 10 Jan 2010 17:30:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Jan 2010 17:30:27 +0000
X-ASF-Spam-Status: No, hits=4.0 required=10.0
	tests=HTML_MESSAGE,SPF_PASS,SUBJ_ALL_CAPS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cabiwan@gmail.com designates 209.85.220.215 as permitted sender)
Received: from [209.85.220.215] (HELO mail-fx0-f215.google.com) (209.85.220.215)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Jan 2010 17:30:20 +0000
Received: by fxm7 with SMTP id 7so19055942fxm.29
        for <general@hadoop.apache.org>; Sun, 10 Jan 2010 09:29:59 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=7RIG254y2pRYsMRvA80GnIlTUFMBNlGr3O7NHttGtxQ=;
        b=heD2qk/LzurjojZZMRP/Y5rlpShVmxgj4RP6C+f6emgFDGFzqjDLwITSusSlCxNLE4
         SWECxzdO/xfQBdVUAtW8gKwLFH+rrS+5gUtYYRI0NDBobTtudbIAuP6TjWkOoYwOajRD
         8lQqbHxP92ezuDVDVwruwIyFmfGzx1v+JSfjg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=OKaFWteVkgQbJM8LGtiTIb0U1hIvWxkn1ToZMvZklaU+xcqDpcS45Kl0jPcqOPSasR
         BcuUb6k8WLKh4reJzRUh7e5RStquumH4lyFCm5KgCN8krBWx8DLL7Hf+tKqDNwCNew+m
         zxlWdM3g2MRuioiqhA5pZCWfjRKXgA8PcMOok=
MIME-Version: 1.0
Received: by 10.103.81.35 with SMTP id i35mr1467375mul.42.1263144599591; Sun, 
	10 Jan 2010 09:29:59 -0800 (PST)
Date: Sun, 10 Jan 2010 18:29:59 +0100
Message-ID: <309f76d01001100929v25f9bce8sc7c788755ea8b55b@mail.gmail.com>
Subject: MOVING FILES BETWEEN DIRS IN HDFS
From: Alberto Luengo Cabanillas <cabiwan@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e656b6e4289d56047cd2c50b
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e656b6e4289d56047cd2c50b
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi everyone! I=B4m trying to move a file from a dir in HDFS to another via =
API
(more specifically, with *FileUtil* class). This is because I=B4m launching=
 a
program within Eclipse against a remote HDFS. Is the "copy" method proper
for what I want to do, or is another alternative (like copyTo-copyFrom
sequence)?
Thanks a lot in advance.

--=20
Alberto

--0016e656b6e4289d56047cd2c50b--

From general-return-900-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jan 10 20:20:38 2010
Return-Path: <general-return-900-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81254 invoked from network); 10 Jan 2010 20:20:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jan 2010 20:20:38 -0000
Received: (qmail 9021 invoked by uid 500); 10 Jan 2010 20:20:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8915 invoked by uid 500); 10 Jan 2010 20:20:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8905 invoked by uid 99); 10 Jan 2010 20:20:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Jan 2010 20:20:37 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [166.84.7.136] (HELO vc136.vc.panix.com) (166.84.7.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Jan 2010 20:20:29 +0000
Received: from eric-sammers-macbook-pro.local (cpe-66-108-36-45.nyc.res.rr.com [66.108.36.45])
	by vc136.vc.panix.com (Postfix) with ESMTP id 3B75DDC39C
	for <general@hadoop.apache.org>; Sun, 10 Jan 2010 15:20:07 -0500 (EST)
Message-ID: <4B4A3675.5080901@lifeless.net>
Date: Sun, 10 Jan 2010 15:20:05 -0500
From: Eric Sammer <eric@lifeless.net>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.5) Gecko/20091204 Thunderbird/3.0
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: MOVING FILES BETWEEN DIRS IN HDFS
References: <309f76d01001100929v25f9bce8sc7c788755ea8b55b@mail.gmail.com>
In-Reply-To: <309f76d01001100929v25f9bce8sc7c788755ea8b55b@mail.gmail.com>
X-Enigmail-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit

On 1/10/10 12:29 PM, Alberto Luengo Cabanillas wrote:
> Hi everyone! I´m trying to move a file from a dir in HDFS to another via API
> (more specifically, with *FileUtil* class). This is because I´m launching a
> program within Eclipse against a remote HDFS. Is the "copy" method proper
> for what I want to do, or is another alternative (like copyTo-copyFrom
> sequence)?
> Thanks a lot in advance.
> 

Alberto:

There is no method for moving a file in FileUtil because it's already
just a single simple method call. Use FileSystem#rename(Path, Path)
instead to move files. The copy methods will leave you with two of the
same file (which you probably don't want).

Hope that helps.
-- 
Eric Sammer
eric@lifeless.net
http://esammer.blogspot.com

From general-return-901-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jan 10 21:38:40 2010
Return-Path: <general-return-901-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 2144 invoked from network); 10 Jan 2010 21:38:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jan 2010 21:38:40 -0000
Received: (qmail 62525 invoked by uid 500); 10 Jan 2010 21:38:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62432 invoked by uid 500); 10 Jan 2010 21:38:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62422 invoked by uid 99); 10 Jan 2010 21:38:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Jan 2010 21:38:39 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cabiwan@gmail.com designates 72.14.220.152 as permitted sender)
Received: from [72.14.220.152] (HELO fg-out-1718.google.com) (72.14.220.152)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Jan 2010 21:38:32 +0000
Received: by fg-out-1718.google.com with SMTP id 19so351681fgg.11
        for <general@hadoop.apache.org>; Sun, 10 Jan 2010 13:38:11 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=frY0YvR7oqJGZ5nmO4g/+IJha9zzembt8is3FQpC4jE=;
        b=f71PytddnXewWai3bRwObgQ6b/npFEYt0T59NvC/F9yDpZWCEax21dsVEEdK6LitgE
         acexk2hFs/hFFh2Efrqlg0zFPYtTCINxtKrd9EuxUqUrMV9wSPspQ9eR3GJ3/fzkldks
         wOlvWcvvuGy9eKWP2i0QbOGAH/qzYL/1JF9Dg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=jfcF548j1/i+OuZ67VBjV1HCqFESntZTGNBnxWYQsYWaAT890d5YdR/hP0QfThxcOc
         Vo8GLvzJtoLpoIvwyT21OSsfgG4imJIXfLg+gOU40051tjtAfINgB54oz7b2XXmIWH3t
         sn7S7H1cRCon/Fat9Tu+yagykrrRmUc7fDdEs=
MIME-Version: 1.0
Received: by 10.102.161.7 with SMTP id j7mr6657878mue.111.1263159491121; Sun, 
	10 Jan 2010 13:38:11 -0800 (PST)
In-Reply-To: <4B4A3675.5080901@lifeless.net>
References: <309f76d01001100929v25f9bce8sc7c788755ea8b55b@mail.gmail.com>
	 <4B4A3675.5080901@lifeless.net>
Date: Sun, 10 Jan 2010 22:38:11 +0100
Message-ID: <309f76d01001101338s21f0f3eel828c28addde3bf97@mail.gmail.com>
Subject: Re: MOVING FILES BETWEEN DIRS IN HDFS
From: Alberto Luengo Cabanillas <cabiwan@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364c78fdc3543b047cd63c1e

--0016364c78fdc3543b047cd63c1e
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Thanks a lot for the reply Eric. I=B4ll give it a try.

2010/1/10 Eric Sammer <eric@lifeless.net>

> On 1/10/10 12:29 PM, Alberto Luengo Cabanillas wrote:
> > Hi everyone! I=B4m trying to move a file from a dir in HDFS to another =
via
> API
> > (more specifically, with *FileUtil* class). This is because I=B4m launc=
hing
> a
> > program within Eclipse against a remote HDFS. Is the "copy" method prop=
er
> > for what I want to do, or is another alternative (like copyTo-copyFrom
> > sequence)?
> > Thanks a lot in advance.
> >
>
> Alberto:
>
> There is no method for moving a file in FileUtil because it's already
> just a single simple method call. Use FileSystem#rename(Path, Path)
> instead to move files. The copy methods will leave you with two of the
> same file (which you probably don't want).
>
> Hope that helps.
> --
> Eric Sammer
> eric@lifeless.net
> http://esammer.blogspot.com
>



--=20
Alberto

--0016364c78fdc3543b047cd63c1e--

From general-return-902-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jan 10 23:23:03 2010
Return-Path: <general-return-902-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33526 invoked from network); 10 Jan 2010 23:23:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jan 2010 23:23:03 -0000
Received: (qmail 30878 invoked by uid 500); 10 Jan 2010 23:23:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30802 invoked by uid 500); 10 Jan 2010 23:23:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30792 invoked by uid 99); 10 Jan 2010 23:23:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Jan 2010 23:23:02 +0000
X-ASF-Spam-Status: No, hits=2.6 required=10.0
	tests=HTML_MESSAGE,SPF_PASS,SUBJECT_FUZZY_TION
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cabiwan@gmail.com designates 209.85.220.212 as permitted sender)
Received: from [209.85.220.212] (HELO mail-fx0-f212.google.com) (209.85.220.212)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Jan 2010 23:22:52 +0000
Received: by fxm4 with SMTP id 4so17465972fxm.12
        for <general@hadoop.apache.org>; Sun, 10 Jan 2010 15:22:32 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=tkEE890sE2cuXkkPsbIFtI265us/3ZpCZbUr+pXICjw=;
        b=YAEVAAwMjM7BsJPQLHWdjtsszLd8Hp28TtPGA/YLgY18vKPRGFuBw+d5xcxk325z+Z
         Ys20UDX4JfvuNRs8bdOXY5wQSEkWHXP7WFvr5J2KDAvDpAv+JDyAsO2LzCypjiv+8mxg
         2dbYgG4V9syHtP5DopM6H7//D7zILK1pGXTRQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=mPt46mh38Z6h5OgR74jY5Dz2Zl6uSfg1LTnE0TGul2BV44f730KsYp4NoWXzIL3Vtu
         qaSqeOA/YL3yyFokGE5Rb8p+N3+68zdQNobZr+A7OcAzsyUff2ndeLZBz32xxPo//75y
         mHISi5UfcB6w4PXCQlt9LsVNPwvaJCJC3GFuw=
MIME-Version: 1.0
Received: by 10.102.14.13 with SMTP id 13mr1257905mun.32.1263165752105; Sun, 
	10 Jan 2010 15:22:32 -0800 (PST)
In-Reply-To: <3c682ecd1001081647x14d68f76h9d4eda3ef3502d4c@mail.gmail.com>
References: <309f76d01001081524t45faa45vf54640e503d67b5@mail.gmail.com>
	 <3c682ecd1001081647x14d68f76h9d4eda3ef3502d4c@mail.gmail.com>
Date: Mon, 11 Jan 2010 00:22:32 +0100
Message-ID: <309f76d01001101522y56852912s85975efb8b3cbc58@mail.gmail.com>
Subject: Re: QUESTION ABOUT PARTITIONERS
From: Alberto Luengo Cabanillas <cabiwan@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016363b954af25a0c047cd7b1ec
X-Virus-Checked: Checked by ClamAV on apache.org

--0016363b954af25a0c047cd7b1ec
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Abrishek. Thanks for the quick reply. I=B4ve already read twice your pap=
er
and I find it fantastic. I=B4m working on a similar project for my final wo=
rk
at studies and your work gave me a bunch of ideas. I=B4ll try your approach
and feedback results.

2010/1/9 Abhishek Verma <vermaabhishekp@gmail.com>

> Hi Alberto,
>
> You should be able to override the default partitioner class in pseudo
> distributed and the fully distributed mode. I used the same technique for
> the paper "Scaling Genetic Algorithms using MapReduce" [
> http://verma7.com/wp/wp-content/uploads/2009/10/ISDA09_MR_GA.pdf] that yo=
u
> might find useful.
>
> -Abhishek.
>
> On Fri, Jan 8, 2010 at 5:24 PM, Alberto Luengo Cabanillas <
> cabiwan@gmail.com
> > wrote:
>
> > Hi everyone! First of all, happy new year to all community!. I=B4m actu=
ally
> > developing an Map-Reduce approach to genetic algorithms, and I was
> > questioning the use and effectivity of Partitioners in the Map-Reduce
> flow.
> > I=B4ve been working in pseudo-distributed mode and didn=B4t find any ex=
ample
> of
> > usage of Partitioner classes. Does this mean that using another
> Partitioner
> > -not the default one- have sense only on a Fully Distributed environmen=
t?
> > Which Partitioner should I implement  with my GA approach?
> > Thanks a lot in advance!
> >
> > --
> > Alberto
> >
>



--=20
Alberto

--0016363b954af25a0c047cd7b1ec--

From general-return-903-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 11 10:58:15 2010
Return-Path: <general-return-903-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47850 invoked from network); 11 Jan 2010 10:58:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jan 2010 10:58:15 -0000
Received: (qmail 67397 invoked by uid 500); 11 Jan 2010 10:58:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67313 invoked by uid 500); 11 Jan 2010 10:58:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67300 invoked by uid 99); 11 Jan 2010 10:58:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Jan 2010 10:58:14 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of antonio.mailing@gmail.com designates 209.85.218.223 as permitted sender)
Received: from [209.85.218.223] (HELO mail-bw0-f223.google.com) (209.85.218.223)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Jan 2010 10:58:05 +0000
Received: by bwz23 with SMTP id 23so13772313bwz.29
        for <general@hadoop.apache.org>; Mon, 11 Jan 2010 02:57:44 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=nPoa3lw5UwAdDTpzenEIIEI+pJbOwu+woWm1zsGIS38=;
        b=iHeP6ce8gHMx5yzDRk7/I/3Ocy++Pd32UY/ekfKWTkcZjzC+EfXc/ccHuVFk2U3VgE
         LHNOMEKQQthUTrF1wTdO4DGOWmIY0cg8N37RboR+Iu68ofX+1KrAdpyNSR8CxHRD9E5Y
         So1QujDKV34jil4VmWnrlKr7wM2aa9C5whX6Y=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=kLef5YOjfHOB3+h6Pnu3f2XmE9JXnrqO6ob+UlVkwkDpKY0oo6KmCyTSvsfxz/pCNi
         P398g6o9lPbLqccg8vNzz6is7DNgvq6Dy8z+vguflbb7d87UkJ/9j82H2gkO//50HdU8
         ASmXi5f1OqsK6Oo5weA8uxyuctCn6w6yh5XdQ=
MIME-Version: 1.0
Received: by 10.204.154.69 with SMTP id n5mr1642553bkw.43.1263207464752; Mon, 
	11 Jan 2010 02:57:44 -0800 (PST)
Date: Mon, 11 Jan 2010 11:57:44 +0100
Message-ID: <f414eb3a1001110257l3b278aa3t6ce922e24d1d25ee@mail.gmail.com>
Subject: Maven repository ?
From: Antonio Goncalves <antonio.mailing@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175cfbca36d923047ce1681d
X-Virus-Checked: Checked by ClamAV on apache.org

--0015175cfbca36d923047ce1681d
Content-Type: text/plain; charset=ISO-8859-1

Hi all,

I'm new to Hadoop and I'm starting with some Hello Worlds. I'm used to use
Maven so I've automatically created a pom.xml.... but I could not find any
dependecies. In MvnRepository I found the following :

http://mvnrepository.com/artifact/org.jvnet.hudson.hadoop/hadoop-core

*http://mvnrepository.com/artifact/org.apache.mahout.hadoop/hadoop-core*
But I don't know if there is a problem with the pom.xml or not, but it
doesn't download the hadoop dependencies. I've also read in older mails that
there was not an official Hadoop Maven repository. Do you know if things
have changed ? Is there a repository ? If yes, what is the groupId and
artifactId ?

Thanks,
Antonio

--0015175cfbca36d923047ce1681d--

From general-return-904-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 11 11:10:45 2010
Return-Path: <general-return-904-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50415 invoked from network); 11 Jan 2010 11:10:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jan 2010 11:10:45 -0000
Received: (qmail 73948 invoked by uid 500); 11 Jan 2010 11:10:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73871 invoked by uid 500); 11 Jan 2010 11:10:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73861 invoked by uid 99); 11 Jan 2010 11:10:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Jan 2010 11:10:44 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of antonio.mailing@gmail.com designates 209.85.218.223 as permitted sender)
Received: from [209.85.218.223] (HELO mail-bw0-f223.google.com) (209.85.218.223)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Jan 2010 11:10:37 +0000
Received: by bwz23 with SMTP id 23so13779490bwz.29
        for <general@hadoop.apache.org>; Mon, 11 Jan 2010 03:10:16 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=owQMYHEeiw9JxXEkXbT60kSsv4eSnX48X9XQzYc4xUc=;
        b=UKpDG+6B2hUn1iNR7puLRobOCefMCF1e+Bwze1Hz7e36Szjaod/npWOBr5+cEdAfpg
         pxVxXFBFYrIax54QdadZedyVTz5Z77s8RGT2n9l9YnlKdKrYZce8fQscgVWlrNbiimmf
         bPMKNG4RbyTZ+9ku5kcgbbwgdziKGSKRUfVuM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=KzPiQDaSeDsr3CyyLn7qT6vz0u/Q2RIPNuV2iaBoMSYptfPWoXr6x/4mOMJp8La1sL
         XRTN/2TeQnywHXeXvCGmWDv+C4FfnWWuHADe88DXz6mKEfLo/KzWIVvHw2D0+i18GJ4z
         yMQgFnZJPyQi2cxQU0E64fwkSRGFDbzS4bHj8=
MIME-Version: 1.0
Received: by 10.204.157.2 with SMTP id z2mr3061339bkw.211.1263208216594; Mon, 
	11 Jan 2010 03:10:16 -0800 (PST)
Date: Mon, 11 Jan 2010 12:10:16 +0100
Message-ID: <f414eb3a1001110310i95cfe7bs2408e2f96309da34@mail.gmail.com>
Subject: Running Hadoop across data centers
From: Antonio Goncalves <antonio.mailing@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175cf8ca0708fa047ce195c1

--0015175cf8ca0708fa047ce195c1
Content-Type: text/plain; charset=ISO-8859-1

Hi all,

I'm reading the excellent Tom White Hadoop guide and there is this sentence
written :

"At the time of this writing, Hadoop is not suited for running across data
centers"

Do you know if it's still the case ? Do you know what are the technical
reasons behind that ? We are planning to test MapReduce in several
scenarios, one being multiple data centers (one in France and the other one
in Germany with special network lines). Wouldn't that be possible ?

Thanks,
Antonio

--0015175cf8ca0708fa047ce195c1--

From general-return-905-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 11 17:59:00 2010
Return-Path: <general-return-905-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50846 invoked from network); 11 Jan 2010 17:59:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jan 2010 17:59:00 -0000
Received: (qmail 16331 invoked by uid 500); 11 Jan 2010 17:58:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16268 invoked by uid 500); 11 Jan 2010 17:58:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16258 invoked by uid 99); 11 Jan 2010 17:58:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Jan 2010 17:58:59 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.210.194] (HELO mail-yx0-f194.google.com) (209.85.210.194)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Jan 2010 17:58:49 +0000
Received: by yxe32 with SMTP id 32so20958564yxe.5
        for <general@hadoop.apache.org>; Mon, 11 Jan 2010 09:58:28 -0800 (PST)
MIME-Version: 1.0
Received: by 10.101.134.13 with SMTP id l13mr13961997ann.191.1263232708019; 
	Mon, 11 Jan 2010 09:58:28 -0800 (PST)
In-Reply-To: <f414eb3a1001110257l3b278aa3t6ce922e24d1d25ee@mail.gmail.com>
References: <f414eb3a1001110257l3b278aa3t6ce922e24d1d25ee@mail.gmail.com>
From: Philip Zeyliger <philip@cloudera.com>
Date: Mon, 11 Jan 2010 09:58:06 -0800
Message-ID: <15da8a101001110958n18ba6322jcbe41d3247bad386@mail.gmail.com>
Subject: Re: Maven repository ?
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c5c0ded48689047ce748ee

--001636c5c0ded48689047ce748ee
Content-Type: text/plain; charset=ISO-8859-1

Cloudera publishes a repository for its distribution.  See
http://repository.cloudera.com/artifactory/repo/com.cloudera/ .

You mayalso  want to take a look at
http://github.com/cloudera/repository-example, which is an example of
building wordcount, using Ivy, against the snapshot repository.

-- Philip

On Mon, Jan 11, 2010 at 2:57 AM, Antonio Goncalves <
antonio.mailing@gmail.com> wrote:

> Hi all,
>
> I'm new to Hadoop and I'm starting with some Hello Worlds. I'm used to use
> Maven so I've automatically created a pom.xml.... but I could not find any
> dependecies. In MvnRepository I found the following :
>
> http://mvnrepository.com/artifact/org.jvnet.hudson.hadoop/hadoop-core
>
> *http://mvnrepository.com/artifact/org.apache.mahout.hadoop/hadoop-core*
> But I don't know if there is a problem with the pom.xml or not, but it
> doesn't download the hadoop dependencies. I've also read in older mails
> that
> there was not an official Hadoop Maven repository. Do you know if things
> have changed ? Is there a repository ? If yes, what is the groupId and
> artifactId ?
>
> Thanks,
> Antonio
>

--001636c5c0ded48689047ce748ee--

From general-return-906-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 11 18:16:10 2010
Return-Path: <general-return-906-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63362 invoked from network); 11 Jan 2010 18:16:10 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jan 2010 18:16:10 -0000
Received: (qmail 48050 invoked by uid 500); 11 Jan 2010 18:16:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47968 invoked by uid 500); 11 Jan 2010 18:16:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47958 invoked by uid 99); 11 Jan 2010 18:16:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Jan 2010 18:16:09 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [166.84.7.136] (HELO vc136.vc.panix.com) (166.84.7.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Jan 2010 18:15:59 +0000
Received: from eric-sammers-macbook-pro.local (pool-96-246-67-178.nycmny.east.verizon.net [96.246.67.178])
	by vc136.vc.panix.com (Postfix) with ESMTP id 8B2F9DC39C
	for <general@hadoop.apache.org>; Mon, 11 Jan 2010 13:15:37 -0500 (EST)
Message-ID: <4B4B6AC7.6020801@lifeless.net>
Date: Mon, 11 Jan 2010 13:15:35 -0500
From: Eric Sammer <eric@lifeless.net>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.5) Gecko/20091204 Thunderbird/3.0
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Running Hadoop across data centers
References: <f414eb3a1001110310i95cfe7bs2408e2f96309da34@mail.gmail.com>
In-Reply-To: <f414eb3a1001110310i95cfe7bs2408e2f96309da34@mail.gmail.com>
X-Enigmail-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 1/11/10 6:10 AM, Antonio Goncalves wrote:
> Hi all,
> 
> I'm reading the excellent Tom White Hadoop guide and there is this sentence
> written :
> 
> "At the time of this writing, Hadoop is not suited for running across data
> centers"
> 
> Do you know if it's still the case ? Do you know what are the technical
> reasons behind that ? We are planning to test MapReduce in several
> scenarios, one being multiple data centers (one in France and the other one
> in Germany with special network lines). Wouldn't that be possible ?

If the network connection between said data centers is low latency
enough to not prevent the heartbeat connections, it should work, but
performance will almost definitely not be good.

There are few (major) concerns with doing this. The performance reason
is that the intermediary files produced by mappers could be potentially
shuffled between data centers. If this is a lot of data (likely) you'll
be killing your connections between facilities. This is one of the
primary reasons it's recommended against. Also, you'll (potentially)
have Hadoop (specifically HDFS) making data block replicas across data
centers which is going to be a lot of traffic.

>From a security perspective, Hadoop is assumed to be running in a
trusted environment. It's not hardened or designed to be exposed to
untrusted networks *at all*. If you did over a public network of any
kind, it would absolutely have to be over a VPN which is going to
further exacerbate the performance issues. You'll undoubtedly also want
to encrypt this communication if your data is remotely sensitive...

Another significant reason not to do this (right now) has to do with the
block assignment scheduling and the potential for getting yourself into
a situation where the VPN or inter-data center connection becomes a
significant scaling issue as well as a single point of failure for the
cluster.

I'm sure one could attempt to address these issues with very fancy
network topology scripts for HDFS, redundant data center interconnects,
and a ton of money for high speed connections, but I can't see it being
feasible in terms of cost or maintenance.

...but that's just my opinion. ;)

I'm certainly interested to hear if anyone is actually brave enough to
do this in production, though.

Hope this helps.
-- 
Eric Sammer
eric@lifeless.net
http://esammer.blogspot.com

From general-return-907-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 11 18:36:28 2010
Return-Path: <general-return-907-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73791 invoked from network); 11 Jan 2010 18:36:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jan 2010 18:36:28 -0000
Received: (qmail 74934 invoked by uid 500); 11 Jan 2010 18:36:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74853 invoked by uid 500); 11 Jan 2010 18:36:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74843 invoked by uid 99); 11 Jan 2010 18:36:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Jan 2010 18:36:27 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [15.193.32.61] (HELO g6t0184.atlanta.hp.com) (15.193.32.61)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Jan 2010 18:36:16 +0000
Received: from G3W0630.americas.hpqcorp.net (g3w0630.americas.hpqcorp.net [16.233.58.74])
	(using TLSv1 with cipher RC4-MD5 (128/128 bits))
	(No client certificate requested)
	by g6t0184.atlanta.hp.com (Postfix) with ESMTPS id 5FA04C39F
	for <general@hadoop.apache.org>; Mon, 11 Jan 2010 18:35:53 +0000 (UTC)
Received: from G5W0602.americas.hpqcorp.net (16.228.9.185) by
 G3W0630.americas.hpqcorp.net (16.233.58.74) with Microsoft SMTP Server (TLS)
 id 8.2.176.0; Mon, 11 Jan 2010 18:34:56 +0000
Received: from GVW1114EXC.americas.hpqcorp.net ([16.228.24.170]) by
 G5W0602.americas.hpqcorp.net ([16.228.9.185]) with mapi; Mon, 11 Jan 2010
 18:34:56 +0000
From: "Day, Phil" <philip.day@hp.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 11 Jan 2010 18:34:53 +0000
Subject: RE: Running Hadoop across data centers
Thread-Topic: Running Hadoop across data centers
Thread-Index: AcqS6ichbAIwdaDhR/OjUA1RVkb6TwAAfE9g
Message-ID: <C49B6B58853ACD4E96677F24DD9C446C491137163D@GVW1114EXC.americas.hpqcorp.net>
References: <f414eb3a1001110310i95cfe7bs2408e2f96309da34@mail.gmail.com>
 <4B4B6AC7.6020801@lifeless.net>
In-Reply-To: <4B4B6AC7.6020801@lifeless.net>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Aside from the "Because it might be possible" - I'm curious as to why would=
 you want to do this ?

With no real namenode resilience until 0.21 (and even then you'd have to be=
 quiet smart with the data placement) I can't see a good availability reaso=
n for wanting to do this.

Is it because you want/need a cluster that you can't physically fit into on=
e data centre ?

I'd of thought that you might do better to split your data set into two, ru=
n identical jobs in each DC and then process the combined results on one of=
 the clusters.

Phil

-----Original Message-----
From: Eric Sammer [mailto:eric@lifeless.net]=20
Sent: 11 January 2010 18:16
To: general@hadoop.apache.org
Subject: Re: Running Hadoop across data centers

On 1/11/10 6:10 AM, Antonio Goncalves wrote:
> Hi all,
>=20
> I'm reading the excellent Tom White Hadoop guide and there is this senten=
ce
> written :
>=20
> "At the time of this writing, Hadoop is not suited for running across dat=
a
> centers"
>=20
> Do you know if it's still the case ? Do you know what are the technical
> reasons behind that ? We are planning to test MapReduce in several
> scenarios, one being multiple data centers (one in France and the other o=
ne
> in Germany with special network lines). Wouldn't that be possible ?

If the network connection between said data centers is low latency
enough to not prevent the heartbeat connections, it should work, but
performance will almost definitely not be good.

There are few (major) concerns with doing this. The performance reason
is that the intermediary files produced by mappers could be potentially
shuffled between data centers. If this is a lot of data (likely) you'll
be killing your connections between facilities. This is one of the
primary reasons it's recommended against. Also, you'll (potentially)
have Hadoop (specifically HDFS) making data block replicas across data
centers which is going to be a lot of traffic.

>From a security perspective, Hadoop is assumed to be running in a
trusted environment. It's not hardened or designed to be exposed to
untrusted networks *at all*. If you did over a public network of any
kind, it would absolutely have to be over a VPN which is going to
further exacerbate the performance issues. You'll undoubtedly also want
to encrypt this communication if your data is remotely sensitive...

Another significant reason not to do this (right now) has to do with the
block assignment scheduling and the potential for getting yourself into
a situation where the VPN or inter-data center connection becomes a
significant scaling issue as well as a single point of failure for the
cluster.

I'm sure one could attempt to address these issues with very fancy
network topology scripts for HDFS, redundant data center interconnects,
and a ton of money for high speed connections, but I can't see it being
feasible in terms of cost or maintenance.

...but that's just my opinion. ;)

I'm certainly interested to hear if anyone is actually brave enough to
do this in production, though.

Hope this helps.
--=20
Eric Sammer
eric@lifeless.net
http://esammer.blogspot.com

From general-return-908-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 12 01:10:51 2010
Return-Path: <general-return-908-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39824 invoked from network); 12 Jan 2010 01:10:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2010 01:10:51 -0000
Received: (qmail 20489 invoked by uid 500); 12 Jan 2010 01:10:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20411 invoked by uid 500); 12 Jan 2010 01:10:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 20401 invoked by uid 99); 12 Jan 2010 01:10:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jan 2010 01:10:50 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [213.133.104.83] (HELO www83.your-server.de) (213.133.104.83)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jan 2010 01:10:39 +0000
Received: from [85.178.192.242] (helo=[192.168.178.47])
	by www83.your-server.de with esmtpa (Exim 4.69)
	(envelope-from <o.b.fischer@swe-blog.net>)
	id 1NUVH1-0002EG-7z; Tue, 12 Jan 2010 02:10:19 +0100
Message-ID: <4B4BAFD4.9030507@swe-blog.net>
Date: Tue, 12 Jan 2010 00:10:12 +0100
From: Oliver Fischer <o.b.fischer@swe-blog.net>
User-Agent: Thunderbird 2.0.0.23 (Windows/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Reviewer wanted for german articel on Hadoop
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Content-Transfer-Encoding: 8bit
X-Authenticated-Sender: dialog@sw-blog.net
X-Virus-Scanned: Clear (ClamAV 0.95.1/10284/Tue Jan 12 00:20:00 2010)
X-Virus-Checked: Checked by ClamAV on apache.org

Hello,

I am currently writing an small introductive articel for a german online 
magazine. To ensure the correctness of my article I would like to know 
if someone from Berlin (Germany) here around who is willing to review 
the articel?

Best regards,

Oliver

-- 
Oliver B. Fischer, Schönhauser Allee 64, 10437 Berlin
Tel. +49 30 44793251, Mobil: +49 178 7903538
Mail: o.b.fischer@swe-blog.net Blog: http://www.swe-blog.net


From general-return-909-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 12 01:17:20 2010
Return-Path: <general-return-909-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 41107 invoked from network); 12 Jan 2010 01:17:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2010 01:17:20 -0000
Received: (qmail 27624 invoked by uid 500); 12 Jan 2010 01:17:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27541 invoked by uid 500); 12 Jan 2010 01:17:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27531 invoked by uid 99); 12 Jan 2010 01:17:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jan 2010 01:17:19 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_HELO_PASS,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.208.4.195] (HELO mout.perfora.net) (74.208.4.195)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jan 2010 01:17:11 +0000
Received: from Ralf-Schreijers-MacBook-Pro.local (i59F5365E.versanet.de [89.245.54.94])
	by mrelay.perfora.net (node=mrus0) with ESMTP (Nemesis)
	id 0LwK2Q-1NxUIr2cEI-018F3K; Mon, 11 Jan 2010 20:16:49 -0500
Message-ID: <4B4BCD7D.3040502@sensenetworks.com>
Date: Mon, 11 Jan 2010 20:16:45 -0500
From: Ralf Schreijer <ralf@sensenetworks.com>
User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Reviewer wanted for german articel on Hadoop
References: <4B4BAFD4.9030507@swe-blog.net>
In-Reply-To: <4B4BAFD4.9030507@swe-blog.net>
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Content-Transfer-Encoding: 7bit
X-Provags-ID: V01U2FsdGVkX1+VT4otH1oDEZAtKB1ZgjgoZgpa3Tq+L7zCd+y
 0w0Zq2oKvJ+vGCZgW8gMFgX7Jo2/qtqAXcpfyi0t9DFzsIzSKt
 Aa8jMHUjqbjsKNaWs7jqrvWYatLlDCz

I am not from Berlin but native in German (if you are looking for that), 
send it over!

--Ralf

Oliver Fischer wrote:
> Hello,
> 
> I am currently writing an small introductive articel for a german online 
> magazine. To ensure the correctness of my article I would like to know 
> if someone from Berlin (Germany) here around who is willing to review 
> the articel?
> 
> Best regards,
> 
> Oliver
> 

From general-return-910-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 12 11:02:24 2010
Return-Path: <general-return-910-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50677 invoked from network); 12 Jan 2010 11:02:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2010 11:02:23 -0000
Received: (qmail 25753 invoked by uid 500); 12 Jan 2010 11:02:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25660 invoked by uid 500); 12 Jan 2010 11:02:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25650 invoked by uid 99); 12 Jan 2010 11:02:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jan 2010 11:02:22 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of antonio.mailing@gmail.com designates 209.85.218.223 as permitted sender)
Received: from [209.85.218.223] (HELO mail-bw0-f223.google.com) (209.85.218.223)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jan 2010 11:02:12 +0000
Received: by bwz23 with SMTP id 23so14638722bwz.29
        for <general@hadoop.apache.org>; Tue, 12 Jan 2010 03:01:51 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=+wLcDkvpTjnNPTjsPXc1mLDN7s2miTEkGvym+/3xJ2U=;
        b=oJPPMNi3R7Wgq2fKOz2G5702S0TDYSJL1MXsdhdzyyNkYj7a16knF/fdXHAvav8DqS
         U0/FP75sn4Z7KJuB/EinlPdIiN2RUNOnDAdyn9seCxBguLDNr8aLU1g4e8MlXHaGWHG3
         0y9F+IdCxGDuV0aNDzK/IYWeipBp1dApF+wpc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=dTL7AHfehHXUfShO+Gn0GBA0mdpKxAk+NO5gf2gwPqE2VahaACRUpao5ndJ14EAu7n
         lSxokNGgYJTw7LQ1IZZ6px+RGqx1Qgg8QjDvOa3PRTX6edUQ9krLBxfUIQ09RYIRKpJF
         vFxH6JMFM9hCcBI47aiJS7gXlI5TJJEu2X46o=
MIME-Version: 1.0
Received: by 10.204.33.131 with SMTP id h3mr2553766bkd.53.1263294110729; Tue, 
	12 Jan 2010 03:01:50 -0800 (PST)
In-Reply-To: <C49B6B58853ACD4E96677F24DD9C446C491137163D@GVW1114EXC.americas.hpqcorp.net>
References: <f414eb3a1001110310i95cfe7bs2408e2f96309da34@mail.gmail.com>
	 <4B4B6AC7.6020801@lifeless.net>
	 <C49B6B58853ACD4E96677F24DD9C446C491137163D@GVW1114EXC.americas.hpqcorp.net>
Date: Tue, 12 Jan 2010 12:01:50 +0100
Message-ID: <f414eb3a1001120301x7d6c32eah42ef21dbcefae965@mail.gmail.com>
Subject: Re: Running Hadoop across data centers
From: Antonio Goncalves <antonio.mailing@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000325559a92b78a04047cf59438

--000325559a92b78a04047cf59438
Content-Type: text/plain; charset=ISO-8859-1

Thanks Eric and Phil for your inputs.

We have 80% of our calculation that can be done in one datacenter, but the
rest is heavy calculation. We are using some time consuming algorithm (such
as Monte Carlo for example) that would take too much time in one datacenter.
For this kind of computation we are thinking of using the second datacenter
based in Germany. We haven't done all the study about the data, but I guess
that for the 80% the data will be local to one datacenter, and for the 20%
it would have to be distributed across datacenter. What we haven't worked on
yet, is the size of this distributed data. If looks it would not be that big
(maybe less than a 1Tb but it could grow when doing some calculation based
on archived data).

Antonio

2010/1/11 Day, Phil <philip.day@hp.com>

> Aside from the "Because it might be possible" - I'm curious as to why would
> you want to do this ?
>
> With no real namenode resilience until 0.21 (and even then you'd have to be
> quiet smart with the data placement) I can't see a good availability reason
> for wanting to do this.
>
> Is it because you want/need a cluster that you can't physically fit into
> one data centre ?
>
> I'd of thought that you might do better to split your data set into two,
> run identical jobs in each DC and then process the combined results on one
> of the clusters.
>
> Phil
>
> -----Original Message-----
> From: Eric Sammer [mailto:eric@lifeless.net]
> Sent: 11 January 2010 18:16
> To: general@hadoop.apache.org
> Subject: Re: Running Hadoop across data centers
>
> On 1/11/10 6:10 AM, Antonio Goncalves wrote:
> > Hi all,
> >
> > I'm reading the excellent Tom White Hadoop guide and there is this
> sentence
> > written :
> >
> > "At the time of this writing, Hadoop is not suited for running across
> data
> > centers"
> >
> > Do you know if it's still the case ? Do you know what are the technical
> > reasons behind that ? We are planning to test MapReduce in several
> > scenarios, one being multiple data centers (one in France and the other
> one
> > in Germany with special network lines). Wouldn't that be possible ?
>
> If the network connection between said data centers is low latency
> enough to not prevent the heartbeat connections, it should work, but
> performance will almost definitely not be good.
>
> There are few (major) concerns with doing this. The performance reason
> is that the intermediary files produced by mappers could be potentially
> shuffled between data centers. If this is a lot of data (likely) you'll
> be killing your connections between facilities. This is one of the
> primary reasons it's recommended against. Also, you'll (potentially)
> have Hadoop (specifically HDFS) making data block replicas across data
> centers which is going to be a lot of traffic.
>
> From a security perspective, Hadoop is assumed to be running in a
> trusted environment. It's not hardened or designed to be exposed to
> untrusted networks *at all*. If you did over a public network of any
> kind, it would absolutely have to be over a VPN which is going to
> further exacerbate the performance issues. You'll undoubtedly also want
> to encrypt this communication if your data is remotely sensitive...
>
> Another significant reason not to do this (right now) has to do with the
> block assignment scheduling and the potential for getting yourself into
> a situation where the VPN or inter-data center connection becomes a
> significant scaling issue as well as a single point of failure for the
> cluster.
>
> I'm sure one could attempt to address these issues with very fancy
> network topology scripts for HDFS, redundant data center interconnects,
> and a ton of money for high speed connections, but I can't see it being
> feasible in terms of cost or maintenance.
>
> ...but that's just my opinion. ;)
>
> I'm certainly interested to hear if anyone is actually brave enough to
> do this in production, though.
>
> Hope this helps.
> --
> Eric Sammer
> eric@lifeless.net
> http://esammer.blogspot.com
>



-- 
--
Antonio Goncalves (antonio.goncalves@gmail.com)
Software architect

Web site : www.antoniogoncalves.org
Blog: agoncal.wordpress.com
Feed: feeds2.feedburner.com/AntonioGoncalves
Paris JUG leader : www.parisjug.org
LinkedIn: www.linkedin.com/in/agoncal

--000325559a92b78a04047cf59438--

From general-return-911-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 12 15:55:17 2010
Return-Path: <general-return-911-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78487 invoked from network); 12 Jan 2010 15:55:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2010 15:55:17 -0000
Received: (qmail 74623 invoked by uid 500); 12 Jan 2010 15:55:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74524 invoked by uid 500); 12 Jan 2010 15:55:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74512 invoked by uid 99); 12 Jan 2010 15:55:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jan 2010 15:55:15 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of antonio.mailing@gmail.com designates 209.85.218.223 as permitted sender)
Received: from [209.85.218.223] (HELO mail-bw0-f223.google.com) (209.85.218.223)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jan 2010 15:55:03 +0000
Received: by bwz23 with SMTP id 23so14876050bwz.29
        for <general@hadoop.apache.org>; Tue, 12 Jan 2010 07:54:43 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=yWd4SQCrTb/Z5gQMt3ygmjOXJxclz9guHdMcDbJTne0=;
        b=F6wHA/aqOyWSnmu1HaUUXF/vSYm6MbT+tgJXalVndgMziCepBeAu/Q7Gh8wFMj867C
         6o/bIRdn3CBqJxoEVI9UfuLrwhZHfT9gD+M4LSOQj5WCBphNtLOYrdQHrhLWoQwqT+yw
         vum1bh/v0nVyF/2vRTJSilIJiavNcpsiUlLuc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=fpvnfZ6bdq+WWcziugmjMx5dUN+UO9+8UzbFF2hp6DSPrmallc6l70YiwpkWRZBVO4
         0aVlj7R/O99co3WFModvmoD+Jwk1RSATI8bMjrcBtQlttrvEQsKblkNXyaCKCf021lvq
         6JK53Qw6t/WBYccq5r8NgkPabgD+gdkF7/QSQ=
MIME-Version: 1.0
Received: by 10.204.148.85 with SMTP id o21mr2720175bkv.134.1263311682747; 
	Tue, 12 Jan 2010 07:54:42 -0800 (PST)
In-Reply-To: <15da8a101001110958n18ba6322jcbe41d3247bad386@mail.gmail.com>
References: <f414eb3a1001110257l3b278aa3t6ce922e24d1d25ee@mail.gmail.com>
	 <15da8a101001110958n18ba6322jcbe41d3247bad386@mail.gmail.com>
Date: Tue, 12 Jan 2010 16:54:42 +0100
Message-ID: <f414eb3a1001120754nfc3ea18k5936b354f8f2e33d@mail.gmail.com>
Subject: Re: Maven repository ?
From: Antonio Goncalves <antonio.mailing@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175cd962173cfd047cf9acb6
X-Virus-Checked: Checked by ClamAV on apache.org

--0015175cd962173cfd047cf9acb6
Content-Type: text/plain; charset=ISO-8859-1

Hi Philip,

I've tryied using the Cloudera Maven repository but I can't make it work.
I'm using Maven 2.1 and when I add :

        <repository>
            <id>cloudera</id>
            <name>libs-releases</name>
            <url>http://repository.cloudera.com/artifactory/libs-releases
</url>
        </repository>

Maven looks for :
http://repository.cloudera.com/artifactory/libs-releases/com/cloudera/hadoop-core/0.20.1+152/hadoop-core-0.20.1+152.pom(see
the com/cloudera (slash) instead of the com.cloudera (dot). (note that
the pom.xml is missing in the Cloudera repository)

I thought it was a Maven 1 repository so I've added <layout>legacy</layout>
but it doesn't work either.

Any idea ?
Thanks
Antonio


2010/1/11 Philip Zeyliger <philip@cloudera.com>

> Cloudera publishes a repository for its distribution.  See
> http://repository.cloudera.com/artifactory/repo/com.cloudera/ .
>
> You mayalso  want to take a look at
> http://github.com/cloudera/repository-example, which is an example of
> building wordcount, using Ivy, against the snapshot repository.
>
> -- Philip
>
> On Mon, Jan 11, 2010 at 2:57 AM, Antonio Goncalves <
> antonio.mailing@gmail.com> wrote:
>
> > Hi all,
> >
> > I'm new to Hadoop and I'm starting with some Hello Worlds. I'm used to
> use
> > Maven so I've automatically created a pom.xml.... but I could not find
> any
> > dependecies. In MvnRepository I found the following :
> >
> > http://mvnrepository.com/artifact/org.jvnet.hudson.hadoop/hadoop-core
> >
> > *http://mvnrepository.com/artifact/org.apache.mahout.hadoop/hadoop-core*
> > But I don't know if there is a problem with the pom.xml or not, but it
> > doesn't download the hadoop dependencies. I've also read in older mails
> > that
> > there was not an official Hadoop Maven repository. Do you know if things
> > have changed ? Is there a repository ? If yes, what is the groupId and
> > artifactId ?
> >
> > Thanks,
> > Antonio
> >
>



-- 
--
Antonio Goncalves (antonio.goncalves@gmail.com)
Software architect

Web site : www.antoniogoncalves.org
Blog: agoncal.wordpress.com
Feed: feeds2.feedburner.com/AntonioGoncalves
Paris JUG leader : www.parisjug.org
LinkedIn: www.linkedin.com/in/agoncal

--0015175cd962173cfd047cf9acb6--

From general-return-912-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 12 16:03:58 2010
Return-Path: <general-return-912-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81238 invoked from network); 12 Jan 2010 16:03:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2010 16:03:58 -0000
Received: (qmail 89208 invoked by uid 500); 12 Jan 2010 16:03:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89152 invoked by uid 500); 12 Jan 2010 16:03:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89142 invoked by uid 99); 12 Jan 2010 16:03:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jan 2010 16:03:57 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [166.84.7.136] (HELO vc136.vc.panix.com) (166.84.7.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jan 2010 16:03:47 +0000
Received: from eric-sammers-macbook-pro.local (pool-96-246-67-178.nycmny.east.verizon.net [96.246.67.178])
	by vc136.vc.panix.com (Postfix) with ESMTP id 5405DDC39C
	for <general@hadoop.apache.org>; Tue, 12 Jan 2010 11:03:26 -0500 (EST)
Message-ID: <4B4C9D4D.4030208@lifeless.net>
Date: Tue, 12 Jan 2010 11:03:25 -0500
From: Eric Sammer <eric@lifeless.net>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.5) Gecko/20091204 Thunderbird/3.0
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Running Hadoop across data centers
References: <f414eb3a1001110310i95cfe7bs2408e2f96309da34@mail.gmail.com>	 <4B4B6AC7.6020801@lifeless.net>	 <C49B6B58853ACD4E96677F24DD9C446C491137163D@GVW1114EXC.americas.hpqcorp.net> <f414eb3a1001120301x7d6c32eah42ef21dbcefae965@mail.gmail.com>
In-Reply-To: <f414eb3a1001120301x7d6c32eah42ef21dbcefae965@mail.gmail.com>
X-Enigmail-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 1/12/10 6:01 AM, Antonio Goncalves wrote:
> Thanks Eric and Phil for your inputs.
> 
> We have 80% of our calculation that can be done in one datacenter, but the
> rest is heavy calculation. We are using some time consuming algorithm (such
> as Monte Carlo for example) that would take too much time in one datacenter.
> For this kind of computation we are thinking of using the second datacenter
> based in Germany. We haven't done all the study about the data, but I guess
> that for the 80% the data will be local to one datacenter, and for the 20%
> it would have to be distributed across datacenter. What we haven't worked on
> yet, is the size of this distributed data. If looks it would not be that big
> (maybe less than a 1Tb but it could grow when doing some calculation based
> on archived data).

Antonio:

The point here is that if you build one logical Hadoop cluster across
two data centers, then Hadoop will consider all nodes as candidates for
receiving work regardless of which data center the job is started in.
This means that there's no guarantees about how much data will be
shuffled between data centers (during the shuffle phase). The same is
true for the HDFS layer - there's no promise that data won't be
shuffling between data centers for replication; in fact, it's likely it
will.

If you want to make sure compute intensive jobs run in data center A
you'll probably have better luck creating two logical Hadoop clusters,
each confined to a data center, and then simply making sure the proper
data sets are available in each Hadoop cluster. This way, there will be
no unbounded, uncontrolled data transfer between data centers and job
performance will not suffer.

The down side is that you may have to copy data from one data center to
another if the data for job running in data center A is produced in data
center B.

I would definitely watch the Cloudera Hadoop MapReduce and HDFS training
video[1] and look over the information on the Hadoop wiki and web site
prior to doing anything at all. It may clear a few things up.

Also, I highly recommend the book Hadoop: The Definitive Guide by Tom
White / O'Reilly[2] if you haven't read it. It has tons of excellent
information about the map reduce process and how it works under the hood.

[1] - http://www.cloudera.com/hadoop-training-mapreduce-hdfs
[2] - http://oreilly.com/catalog/9780596521981

-- 
Eric Sammer
eric@lifeless.net
http://esammer.blogspot.com

From general-return-913-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 12 17:05:21 2010
Return-Path: <general-return-913-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4341 invoked from network); 12 Jan 2010 17:05:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2010 17:05:21 -0000
Received: (qmail 99487 invoked by uid 500); 12 Jan 2010 17:05:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99436 invoked by uid 500); 12 Jan 2010 17:05:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 99425 invoked by uid 99); 12 Jan 2010 17:05:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jan 2010 17:05:20 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of antonio.mailing@gmail.com designates 209.85.218.223 as permitted sender)
Received: from [209.85.218.223] (HELO mail-bw0-f223.google.com) (209.85.218.223)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jan 2010 17:05:10 +0000
Received: by bwz23 with SMTP id 23so14952959bwz.29
        for <general@hadoop.apache.org>; Tue, 12 Jan 2010 09:04:49 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=c2kv/wDryHvqfRykhzN9Sc310pW+Emj2CcUnjz8cWAk=;
        b=CMjAYSpniBwqPAJMGfaKER7GQvL1+YExYS5ycN1kqPFSm1rX1o6BUtaB6yukrk/LNf
         d0hT1amksiTEm5f+NCDMUQ7yOKO6r/Mw1Af9zYj0G/3NBf3aoNJZD6D7u5uZ+hSNhfRB
         pE3J4juD2fQhFbK2VYCSJCprvewwFX8gB9HNo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=XI/PjfsKAfhGdDH/xqxdvZPZhhYiEVnSib699zZsDWrvJVXP8msfvfuTtZWKBA71fD
         rbHIk+p3CKzyzyuwNradxAx7oXMyvZefTfx+/DKLEeGd4451WXrhOvCPhl9muYwujk4V
         6n9g54XdRqGGC7tJywLyVakdliaJPOxLsTXQw=
MIME-Version: 1.0
Received: by 10.204.8.220 with SMTP id i28mr163664bki.143.1263315888903; Tue, 
	12 Jan 2010 09:04:48 -0800 (PST)
In-Reply-To: <4B4C9D4D.4030208@lifeless.net>
References: <f414eb3a1001110310i95cfe7bs2408e2f96309da34@mail.gmail.com>
	 <4B4B6AC7.6020801@lifeless.net>
	 <C49B6B58853ACD4E96677F24DD9C446C491137163D@GVW1114EXC.americas.hpqcorp.net>
	 <f414eb3a1001120301x7d6c32eah42ef21dbcefae965@mail.gmail.com>
	 <4B4C9D4D.4030208@lifeless.net>
Date: Tue, 12 Jan 2010 18:04:48 +0100
Message-ID: <f414eb3a1001120904kd477a32m7972f87b4d797f94@mail.gmail.com>
Subject: Re: Running Hadoop across data centers
From: Antonio Goncalves <antonio.mailing@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00151758a57acc158c047cfaa65f

--00151758a57acc158c047cfaa65f
Content-Type: text/plain; charset=ISO-8859-1

Thanks Eric. That's really usefull information. I've started reading the
Hadoop Guide book and I'll watch the video.

Antonio

2010/1/12 Eric Sammer <eric@lifeless.net>

> On 1/12/10 6:01 AM, Antonio Goncalves wrote:
> > Thanks Eric and Phil for your inputs.
> >
> > We have 80% of our calculation that can be done in one datacenter, but
> the
> > rest is heavy calculation. We are using some time consuming algorithm
> (such
> > as Monte Carlo for example) that would take too much time in one
> datacenter.
> > For this kind of computation we are thinking of using the second
> datacenter
> > based in Germany. We haven't done all the study about the data, but I
> guess
> > that for the 80% the data will be local to one datacenter, and for the
> 20%
> > it would have to be distributed across datacenter. What we haven't worked
> on
> > yet, is the size of this distributed data. If looks it would not be that
> big
> > (maybe less than a 1Tb but it could grow when doing some calculation
> based
> > on archived data).
>
> Antonio:
>
> The point here is that if you build one logical Hadoop cluster across
> two data centers, then Hadoop will consider all nodes as candidates for
> receiving work regardless of which data center the job is started in.
> This means that there's no guarantees about how much data will be
> shuffled between data centers (during the shuffle phase). The same is
> true for the HDFS layer - there's no promise that data won't be
> shuffling between data centers for replication; in fact, it's likely it
> will.
>
> If you want to make sure compute intensive jobs run in data center A
> you'll probably have better luck creating two logical Hadoop clusters,
> each confined to a data center, and then simply making sure the proper
> data sets are available in each Hadoop cluster. This way, there will be
> no unbounded, uncontrolled data transfer between data centers and job
> performance will not suffer.
>
> The down side is that you may have to copy data from one data center to
> another if the data for job running in data center A is produced in data
> center B.
>
> I would definitely watch the Cloudera Hadoop MapReduce and HDFS training
> video[1] and look over the information on the Hadoop wiki and web site
> prior to doing anything at all. It may clear a few things up.
>
> Also, I highly recommend the book Hadoop: The Definitive Guide by Tom
> White / O'Reilly[2] if you haven't read it. It has tons of excellent
> information about the map reduce process and how it works under the hood.
>
> [1] - http://www.cloudera.com/hadoop-training-mapreduce-hdfs
> [2] - http://oreilly.com/catalog/9780596521981
>
> --
>  Eric Sammer
> eric@lifeless.net
> http://esammer.blogspot.com
>



-- 
--
Antonio Goncalves (antonio.goncalves@gmail.com)
Software architect

Web site : www.antoniogoncalves.org
Blog: agoncal.wordpress.com
Feed: feeds2.feedburner.com/AntonioGoncalves
Paris JUG leader : www.parisjug.org
LinkedIn: www.linkedin.com/in/agoncal

--00151758a57acc158c047cfaa65f--

From general-return-914-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 12 20:41:45 2010
Return-Path: <general-return-914-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 3748 invoked from network); 12 Jan 2010 20:41:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2010 20:41:45 -0000
Received: (qmail 39696 invoked by uid 500); 12 Jan 2010 20:41:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39540 invoked by uid 500); 12 Jan 2010 20:41:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39471 invoked by uid 99); 12 Jan 2010 20:41:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jan 2010 20:41:40 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [209.85.216.204] (HELO mail-px0-f204.google.com) (209.85.216.204)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jan 2010 20:41:30 +0000
Received: by pxi42 with SMTP id 42so16946922pxi.5
        for <multiple recipients>; Tue, 12 Jan 2010 12:41:08 -0800 (PST)
MIME-Version: 1.0
Received: by 10.114.214.34 with SMTP id m34mr2327443wag.88.1263328868364; Tue, 
	12 Jan 2010 12:41:08 -0800 (PST)
From: Christophe Bisciglia <christophe@cloudera.com>
Date: Tue, 12 Jan 2010 12:40:48 -0800
Message-ID: <69035571001121240l3e1c2275x6b739f9195e8ea57@mail.gmail.com>
Subject: Announcement: Hadoop Training in February (Bay Area) and March (NYC)
To: common-user@hadoop.apache.org, pig-user@hadoop.apache.org, 
	hive-user@hadoop.apache.org, general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64b0bce6ef3ee047cfdac54
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e64b0bce6ef3ee047cfdac54
Content-Type: text/plain; charset=ISO-8859-1

Hadoop Fans, We're looking forward to a full house at our upcoming sessions
in January (Bay Area) and February (NYC). For those of you who have inquired
recently, we're sorry we sold out so early this month! It's amazing to see
the community growing so quickly.
To that end, we're happy to announce two additional sessions:
February 22-24 (SF Bay Area): http://www.eventbrite.com/event/527260049
March 10-12 (NYC): http://www.eventbrite.com/event/534680243

You can get 10% off either session by registering at least 30 days in
advance.
 February's Bay Area Hadoop Training session will again be led by Tom White
(author of O'Reilly's "Hadoop: The Definitive Guide") and will be sponsored
by Yahoo! this time. You'll get to hear from Owen O'Malley, Yahoo! Architect
and Apache VP for Hadoop, about running Hadoop on some of the largest
clusters in the world. We're really excited about the interest we've had
from major technology leaders in sharing their experiences with new Hadoop
users, and we hope to continue taking advantage of this to share that
perspective with the community. For those of you hoping to spend more time
working on web scale data problems, this is a great chance to get certified
and learn more about job opportunities at Yahoo! - and it's not just Yahoo!
that's hiring for Hadoop. Obtaining your certification is a great way to
stand out for all those Hadoop jobs out there. Here's a great graph showing
how interest is rising rapidly amongst employers:
http://www.indeed.com/jobtrends?q=hadoop&l=&relative=1 Christophe and the
Cloudera Team

-- 
get hadoop: cloudera.com/hadoop
online training: cloudera.com/hadoop-training
blog: cloudera.com/blog
twitter: twitter.com/cloudera

--0016e64b0bce6ef3ee047cfdac54--

From general-return-915-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 13 19:09:56 2010
Return-Path: <general-return-915-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17606 invoked from network); 13 Jan 2010 19:09:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jan 2010 19:09:56 -0000
Received: (qmail 23066 invoked by uid 500); 13 Jan 2010 19:09:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22993 invoked by uid 500); 13 Jan 2010 19:09:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22983 invoked by uid 99); 13 Jan 2010 19:09:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Jan 2010 19:09:55 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of andrey.v.kuzmin@gmail.com designates 72.14.220.158 as permitted sender)
Received: from [72.14.220.158] (HELO fg-out-1718.google.com) (72.14.220.158)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Jan 2010 19:09:46 +0000
Received: by fg-out-1718.google.com with SMTP id 16so184380fgg.11
        for <general@hadoop.apache.org>; Wed, 13 Jan 2010 11:09:25 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :from:date:message-id:subject:to:content-type;
        bh=tgKULljTGy+vTS7khE9a1642bhr2Inlmpnr5k3sgWSw=;
        b=H7JGxK+v8IJ3VJYTc8zkjSfK+AppFPGy55GnnYemUm5XHlYMjIkdAg9RMc5NC+6Bcd
         5jif2+FWqEZtzQwwJRFaK6IjESJe7WAwyhUUHk9HOQ8qZDw5rv/dI5Uz2fBAwqkfaQu+
         l1qYEoZLntDD6hbz3j+ldxE+4VqK8rXGbvsJo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=RwO3IwyFB97I3IW0JFxTqIuEdAVY0hgcSYcpQXlToZI14H4cj2jTAFWT57EEyhnnhs
         baLKM5kKx7CNyINySjl+r0tGPnT4w4+mynfanfczwP1e5VnBj0xdmLpi28LHiPTzZroc
         Rpi77AddZfb7WnCNReuybyAQ+IpHQhXryeG+Q=
MIME-Version: 1.0
Received: by 10.239.170.147 with SMTP id s19mr29766hbe.133.1263409765111; Wed, 
	13 Jan 2010 11:09:25 -0800 (PST)
In-Reply-To: <4B4C9D4D.4030208@lifeless.net>
References: <f414eb3a1001110310i95cfe7bs2408e2f96309da34@mail.gmail.com> 
	<4B4B6AC7.6020801@lifeless.net> <C49B6B58853ACD4E96677F24DD9C446C491137163D@GVW1114EXC.americas.hpqcorp.net> 
	<f414eb3a1001120301x7d6c32eah42ef21dbcefae965@mail.gmail.com> 
	<4B4C9D4D.4030208@lifeless.net>
From: Andrey Kuzmin <andrey.v.kuzmin@gmail.com>
Date: Wed, 13 Jan 2010 22:09:05 +0300
Message-ID: <2a31deca1001131109s3386916ne4c8f6a45b02ad96@mail.gmail.com>
Subject: Re: Running Hadoop across data centers
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Jan 12, 2010 at 7:03 PM, Eric Sammer <eric@lifeless.net> wrote:
> On 1/12/10 6:01 AM, Antonio Goncalves wrote:
>> Thanks Eric and Phil for your inputs.
>>
>> We have 80% of our calculation that can be done in one datacenter, but the
>> rest is heavy calculation. We are using some time consuming algorithm (such
>> as Monte Carlo for example) that would take too much time in one datacenter.
>> For this kind of computation we are thinking of using the second datacenter
>> based in Germany. We haven't done all the study about the data, but I guess
>> that for the 80% the data will be local to one datacenter, and for the 20%
>> it would have to be distributed across datacenter. What we haven't worked on
>> yet, is the size of this distributed data. If looks it would not be that big
>> (maybe less than a 1Tb but it could grow when doing some calculation based
>> on archived data).
>
> Antonio:
>
> The point here is that if you build one logical Hadoop cluster across
> two data centers, then Hadoop will consider all nodes as candidates for
> receiving work regardless of which data center the job is started in.

Is Hadoop's job scheduler totally NUMA-unaware? The question holds for
both cross-data center scenario being discussed and or single data
center as well: just imagine the usual within-rack or between-racks
scheduling decision.

Regards,
Andrey

> This means that there's no guarantees about how much data will be
> shuffled between data centers (during the shuffle phase). The same is
> true for the HDFS layer - there's no promise that data won't be
> shuffling between data centers for replication; in fact, it's likely it
> will.
>
> If you want to make sure compute intensive jobs run in data center A
> you'll probably have better luck creating two logical Hadoop clusters,
> each confined to a data center, and then simply making sure the proper
> data sets are available in each Hadoop cluster. This way, there will be
> no unbounded, uncontrolled data transfer between data centers and job
> performance will not suffer.
>
> The down side is that you may have to copy data from one data center to
> another if the data for job running in data center A is produced in data
> center B.
>
> I would definitely watch the Cloudera Hadoop MapReduce and HDFS training
> video[1] and look over the information on the Hadoop wiki and web site
> prior to doing anything at all. It may clear a few things up.
>
> Also, I highly recommend the book Hadoop: The Definitive Guide by Tom
> White / O'Reilly[2] if you haven't read it. It has tons of excellent
> information about the map reduce process and how it works under the hood.
>
> [1] - http://www.cloudera.com/hadoop-training-mapreduce-hdfs
> [2] - http://oreilly.com/catalog/9780596521981
>
> --
> Eric Sammer
> eric@lifeless.net
> http://esammer.blogspot.com
>

From general-return-916-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 13 19:31:39 2010
Return-Path: <general-return-916-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24080 invoked from network); 13 Jan 2010 19:31:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jan 2010 19:31:39 -0000
Received: (qmail 66367 invoked by uid 500); 13 Jan 2010 19:31:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66306 invoked by uid 500); 13 Jan 2010 19:31:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66296 invoked by uid 99); 13 Jan 2010 19:31:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Jan 2010 19:31:38 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [166.84.7.136] (HELO vc136.vc.panix.com) (166.84.7.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Jan 2010 19:31:30 +0000
Received: from eric-sammers-macbook-pro.local (pool-96-246-67-178.nycmny.east.verizon.net [96.246.67.178])
	by vc136.vc.panix.com (Postfix) with ESMTP id D048DDC380
	for <general@hadoop.apache.org>; Wed, 13 Jan 2010 14:31:08 -0500 (EST)
Message-ID: <4B4E1F7A.5060306@lifeless.net>
Date: Wed, 13 Jan 2010 14:31:06 -0500
From: Eric Sammer <eric@lifeless.net>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.5) Gecko/20091204 Thunderbird/3.0
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Running Hadoop across data centers
References: <f414eb3a1001110310i95cfe7bs2408e2f96309da34@mail.gmail.com> 	<4B4B6AC7.6020801@lifeless.net> <C49B6B58853ACD4E96677F24DD9C446C491137163D@GVW1114EXC.americas.hpqcorp.net> 	<f414eb3a1001120301x7d6c32eah42ef21dbcefae965@mail.gmail.com> 	<4B4C9D4D.4030208@lifeless.net> <2a31deca1001131109s3386916ne4c8f6a45b02ad96@mail.gmail.com>
In-Reply-To: <2a31deca1001131109s3386916ne4c8f6a45b02ad96@mail.gmail.com>
X-Enigmail-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On 1/13/10 2:09 PM, Andrey Kuzmin wrote:
> On Tue, Jan 12, 2010 at 7:03 PM, Eric Sammer <eric@lifeless.net> wrote:
>> On 1/12/10 6:01 AM, Antonio Goncalves wrote:
>>> Thanks Eric and Phil for your inputs.
>>>
>>> We have 80% of our calculation that can be done in one datacenter, but the
>>> rest is heavy calculation. We are using some time consuming algorithm (such
>>> as Monte Carlo for example) that would take too much time in one datacenter.
>>> For this kind of computation we are thinking of using the second datacenter
>>> based in Germany. We haven't done all the study about the data, but I guess
>>> that for the 80% the data will be local to one datacenter, and for the 20%
>>> it would have to be distributed across datacenter. What we haven't worked on
>>> yet, is the size of this distributed data. If looks it would not be that big
>>> (maybe less than a 1Tb but it could grow when doing some calculation based
>>> on archived data).
>>
>> Antonio:
>>
>> The point here is that if you build one logical Hadoop cluster across
>> two data centers, then Hadoop will consider all nodes as candidates for
>> receiving work regardless of which data center the job is started in.
> 
> Is Hadoop's job scheduler totally NUMA-unaware? The question holds for
> both cross-data center scenario being discussed and or single data
> center as well: just imagine the usual within-rack or between-racks
> scheduling decision.

Andrey:

Take a look at how replicas are assigned to data nodes[1] to see how
blocks are distributed. During M/R the job tracker will assign a map
task to a tracker where the input split is "as close to the data as
possible." Close is either data local (where the task tracker is running
on the same machine as the data), rack local (task tracker on a machine
in the same rack as the data), or not local at all. What I was saying is
that the third case is always possible (and undesirable).

[1]
http://hadoop.apache.org/common/docs/current/hdfs_design.html#Replica+Placement%3A+The+First+Baby+Steps

-- 
Eric Sammer
eric@lifeless.net
http://esammer.blogspot.com

From general-return-917-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 13 19:35:06 2010
Return-Path: <general-return-917-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24937 invoked from network); 13 Jan 2010 19:35:06 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jan 2010 19:35:06 -0000
Received: (qmail 72079 invoked by uid 500); 13 Jan 2010 19:35:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72020 invoked by uid 500); 13 Jan 2010 19:35:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72010 invoked by uid 99); 13 Jan 2010 19:35:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Jan 2010 19:35:05 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [166.84.7.136] (HELO vc136.vc.panix.com) (166.84.7.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Jan 2010 19:34:56 +0000
Received: from eric-sammers-macbook-pro.local (pool-96-246-67-178.nycmny.east.verizon.net [96.246.67.178])
	by vc136.vc.panix.com (Postfix) with ESMTP id 876A7DC380
	for <general@hadoop.apache.org>; Wed, 13 Jan 2010 14:34:35 -0500 (EST)
Message-ID: <4B4E204A.2090306@lifeless.net>
Date: Wed, 13 Jan 2010 14:34:34 -0500
From: Eric Sammer <eric@lifeless.net>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.5) Gecko/20091204 Thunderbird/3.0
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Running Hadoop across data centers
References: <f414eb3a1001110310i95cfe7bs2408e2f96309da34@mail.gmail.com> 	<4B4B6AC7.6020801@lifeless.net> <C49B6B58853ACD4E96677F24DD9C446C491137163D@GVW1114EXC.americas.hpqcorp.net> 	<f414eb3a1001120301x7d6c32eah42ef21dbcefae965@mail.gmail.com> 	<4B4C9D4D.4030208@lifeless.net> <2a31deca1001131109s3386916ne4c8f6a45b02ad96@mail.gmail.com> <4B4E1F7A.5060306@lifeless.net>
In-Reply-To: <4B4E1F7A.5060306@lifeless.net>
X-Enigmail-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 1/13/10 2:31 PM, Eric Sammer wrote:
> During M/R the job tracker will assign a map
> task to a tracker where the input split is "as close to the data as
> possible."

That should be:

During M/R the job tracker will assign a map task to a tracker where the
data as possible."

Sorry.
-- 
Eric Sammer
eric@lifeless.net
http://esammer.blogspot.com

From general-return-918-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 13 19:44:58 2010
Return-Path: <general-return-918-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27916 invoked from network); 13 Jan 2010 19:44:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jan 2010 19:44:57 -0000
Received: (qmail 86195 invoked by uid 500); 13 Jan 2010 19:44:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86118 invoked by uid 500); 13 Jan 2010 19:44:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86108 invoked by uid 99); 13 Jan 2010 19:44:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Jan 2010 19:44:56 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of andrey.v.kuzmin@gmail.com designates 209.85.220.213 as permitted sender)
Received: from [209.85.220.213] (HELO mail-fx0-f213.google.com) (209.85.220.213)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Jan 2010 19:44:48 +0000
Received: by fxm5 with SMTP id 5so2185601fxm.29
        for <general@hadoop.apache.org>; Wed, 13 Jan 2010 11:44:28 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :from:date:message-id:subject:to:content-type;
        bh=n77ixxCB84twK0oxg/NyJ1w1I2mBQ/s3qRpV865LvO4=;
        b=gCKy6wO7YEoIN7qQvI5cfjk57l0nqdy6gog1lmT4uuPj7E8bKK1Mikjthopr67h76c
         q9hVjnsq8UASA5LaeZTTGS+3nKENyFPwZb7k9RYuiF5L2/R3+I45XTpUEfLTOoYdygZm
         BUYsALyxf93d3GUEQ2cGivHs4dYpbZg7j5Iyo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=TVf8i+r90Tb2dFNb1ZC3FtLXQOywcoS9heb3s51WW7lyT1paHSU71jSbiFut027iEQ
         rFqAR6RVwaCi7RmCnSp0sjTG2zqVSmY2r47F0G0+om+M+/5WA+WcahmFIf3LpsLmkfve
         BQRpsuQ1oZ9/VU0KBdOm20uThQTyDmUks8q+4=
MIME-Version: 1.0
Received: by 10.239.165.9 with SMTP id v9mr127048hbd.77.1263411868105; Wed, 13 
	Jan 2010 11:44:28 -0800 (PST)
In-Reply-To: <4B4E1F7A.5060306@lifeless.net>
References: <f414eb3a1001110310i95cfe7bs2408e2f96309da34@mail.gmail.com> 
	<4B4B6AC7.6020801@lifeless.net> <C49B6B58853ACD4E96677F24DD9C446C491137163D@GVW1114EXC.americas.hpqcorp.net> 
	<f414eb3a1001120301x7d6c32eah42ef21dbcefae965@mail.gmail.com> 
	<4B4C9D4D.4030208@lifeless.net> <2a31deca1001131109s3386916ne4c8f6a45b02ad96@mail.gmail.com> 
	<4B4E1F7A.5060306@lifeless.net>
From: Andrey Kuzmin <andrey.v.kuzmin@gmail.com>
Date: Wed, 13 Jan 2010 22:44:08 +0300
Message-ID: <2a31deca1001131144x22b02c91s924da03543bc5995@mail.gmail.com>
Subject: Re: Running Hadoop across data centers
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Regards,
Andrey




On Wed, Jan 13, 2010 at 10:31 PM, Eric Sammer <eric@lifeless.net> wrote:
> On 1/13/10 2:09 PM, Andrey Kuzmin wrote:
>> On Tue, Jan 12, 2010 at 7:03 PM, Eric Sammer <eric@lifeless.net> wrote:
>>> On 1/12/10 6:01 AM, Antonio Goncalves wrote:
>>>> Thanks Eric and Phil for your inputs.
>>>>
>>>> We have 80% of our calculation that can be done in one datacenter, but the
>>>> rest is heavy calculation. We are using some time consuming algorithm (such
>>>> as Monte Carlo for example) that would take too much time in one datacenter.
>>>> For this kind of computation we are thinking of using the second datacenter
>>>> based in Germany. We haven't done all the study about the data, but I guess
>>>> that for the 80% the data will be local to one datacenter, and for the 20%
>>>> it would have to be distributed across datacenter. What we haven't worked on
>>>> yet, is the size of this distributed data. If looks it would not be that big
>>>> (maybe less than a 1Tb but it could grow when doing some calculation based
>>>> on archived data).
>>>
>>> Antonio:
>>>
>>> The point here is that if you build one logical Hadoop cluster across
>>> two data centers, then Hadoop will consider all nodes as candidates for
>>> receiving work regardless of which data center the job is started in.
>>
>> Is Hadoop's job scheduler totally NUMA-unaware? The question holds for
>> both cross-data center scenario being discussed and or single data
>> center as well: just imagine the usual within-rack or between-racks
>> scheduling decision.
>
> Andrey:
>
> Take a look at how replicas are assigned to data nodes[1] to see how
> blocks are distributed. During M/R the job tracker will assign a map
> task to a tracker where the input split is "as close to the data as
> possible." Close is either data local (where the task tracker is running
> on the same machine as the data), rack local (task tracker on a machine
> in the same rack as the data), or not local at all. What I was saying is
> that the third case is always possible (and undesirable).

Right, but this also means that, with NUMA-aware scheduler, a job
distributed between data centers will run as good as it can be achieve
wrt locality. Hence, theoretically there could be computation models
where cross-data center jobs are feasible (disregarding singe
name-node issue) rather than "undesirable".

Regards,
Andrey

>
> [1]
> http://hadoop.apache.org/common/docs/current/hdfs_design.html#Replica+Placement%3A+The+First+Baby+Steps
>
> --
> Eric Sammer
> eric@lifeless.net
> http://esammer.blogspot.com
>

From general-return-919-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 13 20:25:15 2010
Return-Path: <general-return-919-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39607 invoked from network); 13 Jan 2010 20:25:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jan 2010 20:25:15 -0000
Received: (qmail 54581 invoked by uid 500); 13 Jan 2010 20:25:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54531 invoked by uid 500); 13 Jan 2010 20:25:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54521 invoked by uid 99); 13 Jan 2010 20:25:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Jan 2010 20:25:14 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [209.85.222.198] (HELO mail-pz0-f198.google.com) (209.85.222.198)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Jan 2010 20:25:06 +0000
Received: by pzk36 with SMTP id 36so15913146pzk.5
        for <general@hadoop.apache.org>; Wed, 13 Jan 2010 12:24:45 -0800 (PST)
MIME-Version: 1.0
Received: by 10.142.3.4 with SMTP id 4mr384514wfc.74.1263414285133; Wed, 13 
	Jan 2010 12:24:45 -0800 (PST)
From: Todd Lipcon <todd@cloudera.com>
Date: Wed, 13 Jan 2010 12:24:25 -0800
Message-ID: <45f85f71001131224id806220j8100c7fd7c9c1d25@mail.gmail.com>
Subject: Revote: HDFS 0.20.1/HDFS-0.20.2 compatibility
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636b2be27ab66b9047d118fed
X-Virus-Checked: Checked by ClamAV on apache.org

--001636b2be27ab66b9047d118fed
Content-Type: text/plain; charset=ISO-8859-1

Hi all,

Last week we had a vote regarding the compatibility problem introduced in
branch-0.20 by the backport of HDFS-793, necessary for HDFS-101, which fixes
a large bug in the write pipeline recovery code. The majority of people
seemed to indicate that this incompatibility was unacceptable, and thus we
should pull it out.

However, I think everyone agrees that the bug itself is pretty critical, and
it would be good to have it fixed - Hairong indicated that it's likely going
to go to Yahoo's internal customers, and Cloudera would like to include it
as well. In our experience we've run into it several times - whenever there
are a few "bad apple" nodes in a cluster that haven't failed hard, it causes
a lot of write pipeline failures (particularly, any pipeline that picks a
bad node as the first node will not recover). For MapReduce it's not a huge
deal, since the tasks will rerun elsewhere and usually succeed, but for
applications like HBase or continuous logging to HDFS, it's a big problem.

I have taken the time to develop and test a patch for branch-0.20 which goes
on top of HDFS-793 and HDFS-101 but maintains compatibility with 0.20.1.
I've posted this patch and a summary of my testing to HDFS-872. Although
this code is tricky to get right, the hardest parts are with the thread
communication and understanding the correct semantics, which I've not
touched at all. I think as long as there's a good review of my patch, we
should feel comfortable introducing it into branch-0.20.

Thanks
-Todd

--001636b2be27ab66b9047d118fed--

From general-return-920-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 13 20:42:19 2010
Return-Path: <general-return-920-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44781 invoked from network); 13 Jan 2010 20:42:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jan 2010 20:42:19 -0000
Received: (qmail 75352 invoked by uid 500); 13 Jan 2010 20:42:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75231 invoked by uid 500); 13 Jan 2010 20:42:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75170 invoked by uid 99); 13 Jan 2010 20:42:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Jan 2010 20:42:17 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Jan 2010 20:42:10 +0000
Received: by pwi20 with SMTP id 20so3400744pwi.29
        for <general@hadoop.apache.org>; Wed, 13 Jan 2010 12:41:50 -0800 (PST)
MIME-Version: 1.0
Received: by 10.142.55.20 with SMTP id d20mr477882wfa.322.1263415310215; Wed, 
	13 Jan 2010 12:41:50 -0800 (PST)
In-Reply-To: <45f85f71001131224id806220j8100c7fd7c9c1d25@mail.gmail.com>
References: <45f85f71001131224id806220j8100c7fd7c9c1d25@mail.gmail.com>
Date: Wed, 13 Jan 2010 12:41:50 -0800
Message-ID: <dfe484f01001131241q17468520q63efc7ff0d623173@mail.gmail.com>
Subject: Re: Revote: HDFS 0.20.1/HDFS-0.20.2 compatibility
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Jan 13, 2010 at 12:24 PM, Todd Lipcon <todd@cloudera.com> wrote:
> Hi all,
>
> Last week we had a vote regarding the compatibility problem introduced in
> branch-0.20 by the backport of HDFS-793, necessary for HDFS-101, which fixes
> a large bug in the write pipeline recovery code. The majority of people
> seemed to indicate that this incompatibility was unacceptable, and thus we
> should pull it out.
>
> However, I think everyone agrees that the bug itself is pretty critical, and
> it would be good to have it fixed - Hairong indicated that it's likely going
> to go to Yahoo's internal customers, and Cloudera would like to include it
> as well. In our experience we've run into it several times - whenever there
> are a few "bad apple" nodes in a cluster that haven't failed hard, it causes
> a lot of write pipeline failures (particularly, any pipeline that picks a
> bad node as the first node will not recover). For MapReduce it's not a huge
> deal, since the tasks will rerun elsewhere and usually succeed, but for
> applications like HBase or continuous logging to HDFS, it's a big problem.
>
> I have taken the time to develop and test a patch for branch-0.20 which goes
> on top of HDFS-793 and HDFS-101 but maintains compatibility with 0.20.1.
> I've posted this patch and a summary of my testing to HDFS-872. Although
> this code is tricky to get right, the hardest parts are with the thread
> communication and understanding the correct semantics, which I've not
> touched at all. I think as long as there's a good review of my patch, we
> should feel comfortable introducing it into branch-0.20.
>
> Thanks
> -Todd
>

+1

This is a critical bug, it would have been a blocker for 20 had we
known about it. Assuming your change that resolves the protocol
incompatibility is reviewed and tested to people's liking I think we
should put it in 20.

Thanks,
Eli

From general-return-921-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 13 21:59:41 2010
Return-Path: <general-return-921-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85588 invoked from network); 13 Jan 2010 21:59:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jan 2010 21:59:41 -0000
Received: (qmail 23378 invoked by uid 500); 13 Jan 2010 21:59:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23313 invoked by uid 500); 13 Jan 2010 21:59:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23295 invoked by uid 99); 13 Jan 2010 21:59:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Jan 2010 21:59:40 +0000
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Jan 2010 21:59:28 +0000
Received: from sp1-ex07cas01.ds.corp.yahoo.com (sp1-ex07cas01.ds.corp.yahoo.com [216.252.116.137])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o0DLwuKd062855;
	Wed, 13 Jan 2010 13:58:56 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:references:in-reply-to:accept-language:
	content-language:x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:
	content-type:content-transfer-encoding:mime-version;
	b=TfFTPnLbV1kXYjR2gMEt93go4EsUqGqlDUU6V7VtjBSFJAoDxr/f5FhzHes5bzPe
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas01.ds.corp.yahoo.com ([216.252.116.137]) with mapi; Wed, 13 Jan
 2010 13:58:56 -0800
From: Dekel Tankel <dekel@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>,
        "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Date: Wed, 13 Jan 2010 13:58:55 -0800
Subject: RE: Hadoop User Group (Bay Area) - Jan 20th at Yahoo!
Thread-Topic: Hadoop User Group (Bay Area) - Jan 20th at Yahoo!
Thread-Index: AcoSMRRU8X5auVLKR8ufbpFTQi1hLgQXaiyQGvMSDFABkB1/QA==
Message-ID: <46E2672895E8744A97D7577A6110858B4FDA1C5F26@SP1-EX07VS01.ds.corp.yahoo.com>
References: <46E2672895E8744A97D7577A6110858B4E1AC69A91@SP1-EX07VS01.ds.corp.yahoo.com>
  <73DDD9AE0462F94091EB137F8F89948956B5ABD1FE@SP1-EX07VS01.ds.corp.yahoo.com>
In-Reply-To: <73DDD9AE0462F94091EB137F8F89948956B5ABD1FE@SP1-EX07VS01.ds.corp.yahoo.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Hi=20

Agenda is now posted for next week's HUG

http://www.meetup.com/hadoop/calendar/12229988/


hope to see you all there.

	dekel

-----Original Message-----
From: Dekel Tankel=20
Sent: Tuesday, January 05, 2010 3:04 PM
To: 'general@hadoop.apache.org'; 'common-user@hadoop.apache.org'
Subject: Hadoop User Group (Bay Area) - Jan 20th at Yahoo!

Hi all,

Happy new year!

RSVP is now open for the first 2010 Bay Area Hadoop user group at the Yahoo=
! Sunnyvale Campus, planed for Jan 20th.

Registration is available here
http://www.meetup.com/hadoop/calendar/12229988/

Agenda will be posted soon.

Looking forward to seeing you there


Dekel


From general-return-922-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 13 23:36:33 2010
Return-Path: <general-return-922-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33468 invoked from network); 13 Jan 2010 23:36:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jan 2010 23:36:33 -0000
Received: (qmail 18211 invoked by uid 500); 13 Jan 2010 23:36:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18158 invoked by uid 500); 13 Jan 2010 23:36:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18147 invoked by uid 99); 13 Jan 2010 23:36:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Jan 2010 23:36:31 +0000
X-ASF-Spam-Status: No, hits=4.0 required=10.0
	tests=HTML_MESSAGE,SPF_PASS,SUBJ_ALL_CAPS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cabiwan@gmail.com designates 209.85.220.213 as permitted sender)
Received: from [209.85.220.213] (HELO mail-fx0-f213.google.com) (209.85.220.213)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Jan 2010 23:36:25 +0000
Received: by fxm5 with SMTP id 5so2406847fxm.29
        for <general@hadoop.apache.org>; Wed, 13 Jan 2010 15:36:04 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=z4zXGZ9luw6+eF7TXa+s+cG5ly8j2+aTcjgJ69T3wGQ=;
        b=hux2MOIUPC3E+B99Oon+/bXncvTNsGxNbe/cCiDAKf2RWqIEiTEakcMthfyOCaZJTM
         /UZ3XBtQ5XiKeZt67f8BKov7sT3aLN0bgsllxgv10S9fQ1NgsYAKqjglFR8FX/NuypPm
         qd0m5KiOoI2NU3hhNF7NQBY2/SOv3larcDrv0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=XNxaBBf8mWLDRedoV18bch11IUtFL40wQFMI38yr8FUA6HTumrtfC8rFYDavnWn+pn
         A095Du4io/6lV3QJSdI2uIkSl3oH8rL1CfYakx1RqWDGIcPQWBXZizKFZsWd+w8E5tE7
         HTVsO1zkmCXTHauF4TlfN7c9F6gWz44f1gU34=
MIME-Version: 1.0
Received: by 10.103.85.28 with SMTP id n28mr386075mul.121.1263425763719; Wed, 
	13 Jan 2010 15:36:03 -0800 (PST)
Date: Thu, 14 Jan 2010 00:36:03 +0100
Message-ID: <309f76d01001131536x5c4d3e66xa9a67457719967ac@mail.gmail.com>
Subject: SCALING GENETIC ALGORITHMS USING MAPREDUCE
From: Alberto Luengo Cabanillas <cabiwan@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e65b62a6d8bc98047d143b09

--0016e65b62a6d8bc98047d143b09
Content-Type: text/plain; charset=ISO-8859-1

Hi everyone! For the last six months. my work with Hadoop is being focused
in developing a stable MRPGA. Last paper I read ("Scaling Genetic Algorithms
Using MapReduce") was a fantastic job and gave me a bunch of ideas; but I
have some questions relative to this paper and I think they may be useful
for community:

Anywhere in the paper talks about elitism rate nor mutation rate. It only
talks about selection and crossover. In fact, this part (page 3 and so)
talks about an INDIVIDUALREPRESENTATION(key) function, which I suppose is
used to represent the key part of the par (i.e., if it is Text, its
representation is a String). Also there are TOURN(tournArray) (?) and
CROSSOVER(crossArray), which I think is related to mutation.

How it is supposed to be implemented the mutation part in the process?. Have
you considered some kind of elitism rate for chossing population?.

Thanks a lot in advance.


-- 
Alberto

--0016e65b62a6d8bc98047d143b09--

From general-return-923-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 13 23:44:56 2010
Return-Path: <general-return-923-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37337 invoked from network); 13 Jan 2010 23:44:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jan 2010 23:44:56 -0000
Received: (qmail 31958 invoked by uid 500); 13 Jan 2010 23:44:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31895 invoked by uid 500); 13 Jan 2010 23:44:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31885 invoked by uid 99); 13 Jan 2010 23:44:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Jan 2010 23:44:55 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of vermaabhishekp@gmail.com designates 209.85.220.213 as permitted sender)
Received: from [209.85.220.213] (HELO mail-fx0-f213.google.com) (209.85.220.213)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Jan 2010 23:44:48 +0000
Received: by fxm5 with SMTP id 5so2412203fxm.29
        for <general@hadoop.apache.org>; Wed, 13 Jan 2010 15:44:27 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=xSJ9i07NDs/8S2dmIgDEFwgtr0Yz01xhULsR3jaQTnE=;
        b=L8wHvuy58yMmlZ/FDwIJJ86WYuFAfV74Ky2BQKiUXRwNYHQ4BWcOTLaF2tuZhSfQ8p
         BcD9W3+anXHwKqbr47qtz20PRyU88bEwB/0VRW5h1NAC2GcS3O6aBlqhySdoH4L8oCq7
         Wq6oBFLLMGs8si69QtG2MwlJTp2iUyQ1Mm/2g=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=PTP5vlO+8zxiFU4pszWDhI1/mQMyJQnZBoLEzma5ShAeUDuWUocZMu+FAchl/Y3HO1
         YdxzuGIW86WvoL5Wz/t177rVTvbaxWvbqNAnyv5YJvZEN5mIoK0P2TCaeS52rpl5MIal
         mKKBiE/2HtMnKWnr4/pNK8WU5rDUFgFW4zIfw=
MIME-Version: 1.0
Received: by 10.239.186.199 with SMTP id i7mr777665hbh.44.1263426265597; Wed, 
	13 Jan 2010 15:44:25 -0800 (PST)
In-Reply-To: <309f76d01001131536x5c4d3e66xa9a67457719967ac@mail.gmail.com>
References: <309f76d01001131536x5c4d3e66xa9a67457719967ac@mail.gmail.com>
Date: Wed, 13 Jan 2010 17:44:25 -0600
Message-ID: <3c682ecd1001131544x43180d70i1992a0ab237dd8f1@mail.gmail.com>
Subject: Re: SCALING GENETIC ALGORITHMS USING MAPREDUCE
From: Abhishek Verma <vermaabhishekp@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f92230c2feed047d145985

--001485f92230c2feed047d145985
Content-Type: text/plain; charset=ISO-8859-1

Hi Alberto,

The paper considers only selecto-recombinative genetic algorithms as
mentioned in Section III.A. The mutation operators could be done on the
reduce  after the crossover or before it as required. Elitism can be
implemented by emitting the individual with a different value in the map and
then directly writing it to context in the reduce.

If you are up for it, we could collaborate together to work on your problem.

Hope this helps.
-Abhishek.

On Wed, Jan 13, 2010 at 5:36 PM, Alberto Luengo Cabanillas <
cabiwan@gmail.com> wrote:

> Hi everyone! For the last six months. my work with Hadoop is being focused
> in developing a stable MRPGA. Last paper I read ("Scaling Genetic
> Algorithms
> Using MapReduce") was a fantastic job and gave me a bunch of ideas; but I
> have some questions relative to this paper and I think they may be useful
> for community:
>
> Anywhere in the paper talks about elitism rate nor mutation rate. It only
> talks about selection and crossover. In fact, this part (page 3 and so)
> talks about an INDIVIDUALREPRESENTATION(key) function, which I suppose is
> used to represent the key part of the par (i.e., if it is Text, its
> representation is a String). Also there are TOURN(tournArray) (?) and
> CROSSOVER(crossArray), which I think is related to mutation.
>
> How it is supposed to be implemented the mutation part in the process?.
> Have
> you considered some kind of elitism rate for chossing population?.
>
> Thanks a lot in advance.
>
>
> --
> Alberto
>

--001485f92230c2feed047d145985--

From general-return-924-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 14 00:30:12 2010
Return-Path: <general-return-924-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56511 invoked from network); 14 Jan 2010 00:30:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2010 00:30:12 -0000
Received: (qmail 7782 invoked by uid 500); 14 Jan 2010 00:30:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7699 invoked by uid 500); 14 Jan 2010 00:30:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7679 invoked by uid 99); 14 Jan 2010 00:30:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Jan 2010 00:30:11 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Jan 2010 00:29:59 +0000
Received: from SNV-EXBH01.ds.corp.yahoo.com (snv-exbh01.ds.corp.yahoo.com [207.126.227.249])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o0E0T4jn092298
	for <general@hadoop.apache.org>; Wed, 13 Jan 2010 16:29:04 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:x-originalarrivaltime;
	b=ULtCP7mLjk0+K2dkzoQKqSV/NSepdTZgakP+3bqb2nNMH0c/+NcxnK6sFrfHgKOF
Received: from SNV-EXVS09.ds.corp.yahoo.com ([207.126.227.86]) by SNV-EXBH01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Wed, 13 Jan 2010 16:29:04 -0800
Received: from 10.72.112.100 ([10.72.112.100]) by SNV-EXVS09.ds.corp.yahoo.com ([207.126.227.84]) via Exchange Front-End Server snv-webmail.corp.yahoo.com ([207.126.227.60]) with Microsoft Exchange Server HTTP-DAV ;
 Thu, 14 Jan 2010 00:28:49 +0000
User-Agent: Microsoft-Entourage/12.23.0.091001
Date: Wed, 13 Jan 2010 16:28:47 -0800
Subject: Re: Revote: HDFS 0.20.1/HDFS-0.20.2 compatibility
From: Hairong Kuang <hairong@yahoo-inc.com>
To: <general@hadoop.apache.org>
Message-ID: <C773A53F.1A4FB%hairong@yahoo-inc.com>
Thread-Topic: Revote: HDFS 0.20.1/HDFS-0.20.2 compatibility
Thread-Index: AcqUsIks2tQGzoCHOkC4T6rEj1F2/w==
In-Reply-To: <dfe484f01001131241q17468520q63efc7ff0d623173@mail.gmail.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-OriginalArrivalTime: 14 Jan 2010 00:29:04.0434 (UTC) FILETIME=[93904120:01CA94B0]
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks Todd for taking the initiative to get the compatibility issue
resolved. I will review the patch.

+1 Todd's proposal on the condition that HDFS-101, 793, and the
compatibility fix are well tested.

Hairong

On 1/13/10 12:41 PM, "Eli Collins" <eli@cloudera.com> wrote:

> On Wed, Jan 13, 2010 at 12:24 PM, Todd Lipcon <todd@cloudera.com> wrote:
>> Hi all,
>> 
>> Last week we had a vote regarding the compatibility problem introduced in
>> branch-0.20 by the backport of HDFS-793, necessary for HDFS-101, which fixes
>> a large bug in the write pipeline recovery code. The majority of people
>> seemed to indicate that this incompatibility was unacceptable, and thus we
>> should pull it out.
>> 
>> However, I think everyone agrees that the bug itself is pretty critical, and
>> it would be good to have it fixed - Hairong indicated that it's likely going
>> to go to Yahoo's internal customers, and Cloudera would like to include it
>> as well. In our experience we've run into it several times - whenever there
>> are a few "bad apple" nodes in a cluster that haven't failed hard, it causes
>> a lot of write pipeline failures (particularly, any pipeline that picks a
>> bad node as the first node will not recover). For MapReduce it's not a huge
>> deal, since the tasks will rerun elsewhere and usually succeed, but for
>> applications like HBase or continuous logging to HDFS, it's a big problem.
>> 
>> I have taken the time to develop and test a patch for branch-0.20 which goes
>> on top of HDFS-793 and HDFS-101 but maintains compatibility with 0.20.1.
>> I've posted this patch and a summary of my testing to HDFS-872. Although
>> this code is tricky to get right, the hardest parts are with the thread
>> communication and understanding the correct semantics, which I've not
>> touched at all. I think as long as there's a good review of my patch, we
>> should feel comfortable introducing it into branch-0.20.
>> 
>> Thanks
>> -Todd
>> 
> 
> +1
> 
> This is a critical bug, it would have been a blocker for 20 had we
> known about it. Assuming your change that resolves the protocol
> incompatibility is reviewed and tested to people's liking I think we
> should put it in 20.
> 
> Thanks,
> Eli


From general-return-925-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 14 05:16:38 2010
Return-Path: <general-return-925-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37977 invoked from network); 14 Jan 2010 05:16:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2010 05:16:37 -0000
Received: (qmail 3020 invoked by uid 500); 14 Jan 2010 05:16:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2942 invoked by uid 500); 14 Jan 2010 05:16:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2932 invoked by uid 99); 14 Jan 2010 05:16:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Jan 2010 05:16:35 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.219.217] (HELO mail-ew0-f217.google.com) (209.85.219.217)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Jan 2010 05:16:26 +0000
Received: by ewy9 with SMTP id 9so7994653ewy.11
        for <general@hadoop.apache.org>; Wed, 13 Jan 2010 21:16:04 -0800 (PST)
MIME-Version: 1.0
Received: by 10.216.88.14 with SMTP id z14mr111922wee.25.1263446163308; Wed, 
	13 Jan 2010 21:16:03 -0800 (PST)
In-Reply-To: <5a921af21001100009g27cd859dw4aa80a99fac24d3b@mail.gmail.com>
References: <995905631001091134h39a703a6u1f962d551f7025a@mail.gmail.com>
	 <8211a1321001092216l58ff0226he5d4047d2bd9538@mail.gmail.com>
	 <c3eaa9191001092245y1eb933e3ifbb9385f6020330b@mail.gmail.com>
	 <5a921af21001100009g27cd859dw4aa80a99fac24d3b@mail.gmail.com>
Date: Wed, 13 Jan 2010 21:16:03 -0800
Message-ID: <ac79ea401001132116k33cfe7d2v7b27ab30b80d9a6d@mail.gmail.com>
Subject: Re: about translating Hadoop:The Definitive Guide into chinese one
From: Tom White <tom@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Thanks everyone for the interest in translations of the book. There is
a Chinese translation coming out very soon.

Cheers,
Tom

On Sun, Jan 10, 2010 at 12:09 AM, xiao yang <yangxiao9901@gmail.com> wrote:
> We can create a project on Yeeyan (www.yeeyan.org).
>
> On Sun, Jan 10, 2010 at 2:45 PM, Qian Ye <yeqian.zju@gmail.com> wrote:
>> So am I, although i'm just a novice to hadoop. I worked on a project bas=
ed
>> on the Zookeeper these monthes. I would like to make some help for the
>> Chapter 13, if need.
>>
>> thx~
>>
>> On Sun, Jan 10, 2010 at 2:16 PM, Jeff Zhang <zjffdu@gmail.com> wrote:
>>
>>> Hi Wang=EF=BC=8C
>>>
>>> I am very interested in this. This book really help me a lot when I go =
deep
>>> into hadoop.
>>> Maybe I can help with chapter 11 Pig. I am now pig's committer.
>>>
>>>
>>>
>>>
>>> On Sat, Jan 9, 2010 at 11:34 AM, Andrew Wang <andrew.wang.1900@gmail.co=
m
>>> >wrote:
>>>
>>> > Hi, you guys
>>> >
>>> > I am interested in Hadoop very much and using it in my current projec=
t. I
>>> > find this book, Hadoop:The Definitive Guide, is a useful and practica=
l
>>> > introduction for new coming Hadoop users. As, there are also many
>>> > applications using hadoop here in china, many more people would like =
to
>>> > know
>>> > what hadoop is and what it can do and how it works. So, maybe transla=
ting
>>> > one technical book about it into chinese will be helpful.
>>> >
>>> > I am working on with my friend and just finish =C2=A0for chapter1 and
>>> chapter6.
>>> > The most important thing i'd like to know is whether you guys think t=
his
>>> > job/work will be useful or helpful?
>>> >
>>> > I will appreciate your suggestions very much! Thanks!
>>> >
>>> >
>>> > --
>>> > http://anqiang1900.blog.163.com/
>>> >
>>>
>>>
>>>
>>> --
>>> Best Regards
>>>
>>> Jeff Zhang
>>>
>>
>>
>>
>> --
>> With Regards!
>>
>> Ye, Qian
>> Made in Zhejiang University
>>
>

From general-return-926-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jan 16 10:48:50 2010
Return-Path: <general-return-926-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45540 invoked from network); 16 Jan 2010 10:48:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jan 2010 10:48:49 -0000
Received: (qmail 5517 invoked by uid 500); 16 Jan 2010 10:48:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5421 invoked by uid 500); 16 Jan 2010 10:48:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5411 invoked by uid 99); 16 Jan 2010 10:48:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 16 Jan 2010 10:48:47 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cabiwan@gmail.com designates 209.85.220.213 as permitted sender)
Received: from [209.85.220.213] (HELO mail-fx0-f213.google.com) (209.85.220.213)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 16 Jan 2010 10:48:39 +0000
Received: by fxm5 with SMTP id 5so901029fxm.29
        for <general@hadoop.apache.org>; Sat, 16 Jan 2010 02:48:19 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=+W5ZvZr/fZm5mFcHg9QpO9RJkgJC7WhfFg0u21VsYNQ=;
        b=n+vR/Gbpzw9yVe+B+xRwN1+moLy3LP/3z4rdrzOwLwLPUt1iH5YaOVaEAgPkhWq1rB
         k2REwIqJqSFpYKHljyimWkM+DKPrsIOS6mOrRNE/E4vOaSFOrf1xE58pZUNPXs4sJdMj
         aj8SMQmorwqximXAuWYMtmEd66lVGmY8avaNM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=uM1cZ/5hKkh+I3zGnMCof81xiPPbgrzuTkm01Sa5nVzGJgTeNUi4kkFqXy4UxwWzrE
         ESqUDZ8dHKOug84I2Iho0cNrLzCtZ7QH3ufsEsuSz+R87BmbIpXKxyoGYAlvzgrnZkLQ
         CYaUCltCSnkUFDTITK6U1yMPSh6c9EBQPvluI=
MIME-Version: 1.0
Received: by 10.103.84.27 with SMTP id m27mr1690447mul.19.1263638899096; Sat, 
	16 Jan 2010 02:48:19 -0800 (PST)
In-Reply-To: <3c682ecd1001131544x43180d70i1992a0ab237dd8f1@mail.gmail.com>
References: <309f76d01001131536x5c4d3e66xa9a67457719967ac@mail.gmail.com>
	 <3c682ecd1001131544x43180d70i1992a0ab237dd8f1@mail.gmail.com>
Date: Sat, 16 Jan 2010 11:48:19 +0100
Message-ID: <309f76d01001160248y7a2a605fua4009c7af1ad8932@mail.gmail.com>
Subject: Re: SCALING GENETIC ALGORITHMS USING MAPREDUCE
From: Alberto Luengo Cabanillas <cabiwan@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e65aee2ab47da9047d45db14
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e65aee2ab47da9047d45db14
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Abhishek Verma. Your help would be very appreciated. I=B4ll try to
implement your suggestions and feed them back if successful.
Regards.

2010/1/14 Abhishek Verma <vermaabhishekp@gmail.com>

> Hi Alberto,
>
> The paper considers only selecto-recombinative genetic algorithms as
> mentioned in Section III.A. The mutation operators could be done on the
> reduce  after the crossover or before it as required. Elitism can be
> implemented by emitting the individual with a different value in the map
> and
> then directly writing it to context in the reduce.
>
> If you are up for it, we could collaborate together to work on your
> problem.
>
> Hope this helps.
> -Abhishek.
>
> On Wed, Jan 13, 2010 at 5:36 PM, Alberto Luengo Cabanillas <
> cabiwan@gmail.com> wrote:
>
> > Hi everyone! For the last six months. my work with Hadoop is being
> focused
> > in developing a stable MRPGA. Last paper I read ("Scaling Genetic
> > Algorithms
> > Using MapReduce") was a fantastic job and gave me a bunch of ideas; but=
 I
> > have some questions relative to this paper and I think they may be usef=
ul
> > for community:
> >
> > Anywhere in the paper talks about elitism rate nor mutation rate. It on=
ly
> > talks about selection and crossover. In fact, this part (page 3 and so)
> > talks about an INDIVIDUALREPRESENTATION(key) function, which I suppose =
is
> > used to represent the key part of the par (i.e., if it is Text, its
> > representation is a String). Also there are TOURN(tournArray) (?) and
> > CROSSOVER(crossArray), which I think is related to mutation.
> >
> > How it is supposed to be implemented the mutation part in the process?.
> > Have
> > you considered some kind of elitism rate for chossing population?.
> >
> > Thanks a lot in advance.
> >
> >
> > --
> > Alberto
> >
>



--=20
Alberto

--0016e65aee2ab47da9047d45db14--

From general-return-927-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jan 17 08:58:06 2010
Return-Path: <general-return-927-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53145 invoked from network); 17 Jan 2010 08:58:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jan 2010 08:58:05 -0000
Received: (qmail 74522 invoked by uid 500); 17 Jan 2010 08:58:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74448 invoked by uid 500); 17 Jan 2010 08:58:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74438 invoked by uid 99); 17 Jan 2010 08:58:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 17 Jan 2010 08:58:03 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of alex.baranov.v@gmail.com designates 209.85.217.216 as permitted sender)
Received: from [209.85.217.216] (HELO mail-gx0-f216.google.com) (209.85.217.216)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 17 Jan 2010 08:57:55 +0000
Received: by gxk8 with SMTP id 8so1878400gxk.11
        for <general@hadoop.apache.org>; Sun, 17 Jan 2010 00:57:34 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=/+yOIIrIVkCVH9TxSrksinAUPMtDXDdmdqnB6iFc37Y=;
        b=uzBlWH9ZQEd/LnxrDtfcCVjwil+K1/9By1S4OQZDrNFgrV3dq4kbiuu7FrqJf2axP4
         s1BOgXHIwkSDIGhWZGaiInY40s2SgPQmQu773T0ihyTPPMWmexxW7JhT5yND1uxA6r0f
         MyqA0OqxfRWCfIH1pJaJLtSNN36k6o+rWrtE4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=XWeYbpRoXv6UaI62yTH5FJ5bA2n6Uckh712YPsKicV8ylc+roP+6qpfT03MIMP75i9
         DvmdMoDXSP4orB9k4GygkDjBQD8YH3yjcHJgguHEnFWzCMNHURayh3XFnPNLFoAb8SFF
         LMWxx5yZ545+JokZBxkI9fAONx7ig++wapIB0=
MIME-Version: 1.0
Received: by 10.101.147.27 with SMTP id z27mr7521729ann.62.1263718654356; Sun, 
	17 Jan 2010 00:57:34 -0800 (PST)
In-Reply-To: <309f76d01001131536x5c4d3e66xa9a67457719967ac@mail.gmail.com>
References: <309f76d01001131536x5c4d3e66xa9a67457719967ac@mail.gmail.com>
Date: Sun, 17 Jan 2010 10:57:34 +0200
Message-ID: <35dbe9101001170057l4e3fb0f8yc1ca9ebf4f1c7c7c@mail.gmail.com>
Subject: Re: SCALING GENETIC ALGORITHMS USING MAPREDUCE
From: Alex Baranov <alex.baranov.v@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e68debd67d2dbb047d586dc0
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e68debd67d2dbb047d586dc0
Content-Type: text/plain; charset=ISO-8859-1

Hello,

I've read the paper and here is my question:

Why not just produce pairs (random int, individual with fitness) from Map
function? Thus individuals will be shuffled randomly after Map phase and
there won't be the need to override the partitioner.

Alex.

P.S. Mentioning GFS in "An accompanying distributed file system like GFS [8]
makes the data management scalable and fault tolerant."  can confuse some
readers because the paper is based on Hadoop family (and further HDFS name
used).

On Thu, Jan 14, 2010 at 1:36 AM, Alberto Luengo Cabanillas <
cabiwan@gmail.com> wrote:

> Hi everyone! For the last six months. my work with Hadoop is being focused
> in developing a stable MRPGA. Last paper I read ("Scaling Genetic
> Algorithms
> Using MapReduce") was a fantastic job and gave me a bunch of ideas; but I
> have some questions relative to this paper and I think they may be useful
> for community:
>
> Anywhere in the paper talks about elitism rate nor mutation rate. It only
> talks about selection and crossover. In fact, this part (page 3 and so)
> talks about an INDIVIDUALREPRESENTATION(key) function, which I suppose is
> used to represent the key part of the par (i.e., if it is Text, its
> representation is a String). Also there are TOURN(tournArray) (?) and
> CROSSOVER(crossArray), which I think is related to mutation.
>
> How it is supposed to be implemented the mutation part in the process?.
> Have
> you considered some kind of elitism rate for chossing population?.
>
> Thanks a lot in advance.
>
>
> --
> Alberto
>

--0016e68debd67d2dbb047d586dc0--

From general-return-928-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 18 06:03:07 2010
Return-Path: <general-return-928-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57770 invoked from network); 18 Jan 2010 06:03:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2010 06:03:07 -0000
Received: (qmail 6307 invoked by uid 500); 18 Jan 2010 06:03:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6138 invoked by uid 500); 18 Jan 2010 06:03:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6128 invoked by uid 99); 18 Jan 2010 06:03:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Jan 2010 06:03:04 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of zhangyf14@gmail.com designates 209.85.222.195 as permitted sender)
Received: from [209.85.222.195] (HELO mail-pz0-f195.google.com) (209.85.222.195)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Jan 2010 06:02:57 +0000
Received: by pzk33 with SMTP id 33so1997268pzk.2
        for <general@hadoop.apache.org>; Sun, 17 Jan 2010 22:02:37 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=nXYd/zztNGa4XkoiuEtntzcQ82ys9L4BDRx0RPqlDHk=;
        b=WA/ZNlNriyuyZHyVJEypDvkQnrxM2UGefJOvnqY8xp1QhdBCHcGqpSyqGkuKnCMX78
         /1ghkPLqUvNOeoAgd7FVGghE5dIAtP6gnJsKI8BxXm7KSUPeFeuVMU53WLGjCSiM0rKP
         lY8c0DovECPAcPbIkBAKILvFPEm7mipCRNcoA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=rDEZ3WTTFafZWCE2qV87AaNxUD54ojSsjcN1AU+w+T25WZI946MP3l+Jq09Ff1qkCl
         4mGwcSSih21wCLq046RLI0809uN5JvjdJXpxBJ5+uYhxQBKZVfXm/x8pa8Boio0H9chO
         iDEX2c6yqGB8Uyc0K0n2fDisoY1cOrij6cZvg=
MIME-Version: 1.0
Received: by 10.115.133.4 with SMTP id k4mr433540wan.4.1263794557113; Sun, 17 
	Jan 2010 22:02:37 -0800 (PST)
Date: Mon, 18 Jan 2010 01:02:37 -0500
Message-ID: <bd7d2ecd1001172202g1f4b8979xa1090d395b86c7f6@mail.gmail.com>
Subject: add worker nodes when job is running
From: Yanfeng Zhang <zhangyf14@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e648be3aa5557a047d6a191c

--0016e648be3aa5557a047d6a191c
Content-Type: text/plain; charset=ISO-8859-1

Hello, all

I want to implement this.
1. In a 10-nodes cluster, there is a job J1 running on 5 nodes, at the same
time there is another job J2  running on the other 5 nodes.
2. Then J1 finishes earlier and J1's 5 nodes join to work on job J2, so we
have 10 nodes working for J2.

Could anyone tell me how to implement it?

Thank you very much!


Regards,

--0016e648be3aa5557a047d6a191c--

From general-return-929-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 18 06:10:22 2010
Return-Path: <general-return-929-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59895 invoked from network); 18 Jan 2010 06:10:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2010 06:10:22 -0000
Received: (qmail 11939 invoked by uid 500); 18 Jan 2010 06:10:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11829 invoked by uid 500); 18 Jan 2010 06:10:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11819 invoked by uid 99); 18 Jan 2010 06:10:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Jan 2010 06:10:19 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of zjffdu@gmail.com designates 209.85.222.195 as permitted sender)
Received: from [209.85.222.195] (HELO mail-pz0-f195.google.com) (209.85.222.195)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Jan 2010 06:10:11 +0000
Received: by pzk33 with SMTP id 33so2000819pzk.2
        for <general@hadoop.apache.org>; Sun, 17 Jan 2010 22:09:50 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=vh0jlCe90fehbKy83TH3/iyye+WdWUH2Ou80+Bla59E=;
        b=hkCpisgoH66wa7rMgRZ4LhbmfFa5H8FAIYOFX3xgiK1EQ2qDB/2HPYeudj0Y7K89wa
         iN5OJ9/E0AIvVJo03Eo3aA5PHVQbAHO1FGVAq62t78dSoIBpX+VQ1D86bkBPUdWrCGrM
         sAxVjOEYcyBqPh0UcRxqc/GhXKU/na8Y6frn8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=J7OlGHyasC+cBV3GgB3TaprKOCmBaUjDSjIiyGv3CLwjHUZtdnEscH0iOwR4Yd6x6N
         lGeXmJw+DPcgKE/GubvFmPx6oneaT5eyMmgdoF/v/qX0DfoYU1T+AYac2GBUGAe0CVLD
         +pq06Rhm/Ltn9PRYCaGFlJpLLXKkQV+TaudFk=
MIME-Version: 1.0
Received: by 10.142.209.20 with SMTP id h20mr3874861wfg.130.1263794990419; 
	Sun, 17 Jan 2010 22:09:50 -0800 (PST)
In-Reply-To: <bd7d2ecd1001172202g1f4b8979xa1090d395b86c7f6@mail.gmail.com>
References: <bd7d2ecd1001172202g1f4b8979xa1090d395b86c7f6@mail.gmail.com>
Date: Mon, 18 Jan 2010 14:09:50 +0800
Message-ID: <8211a1321001172209x45d46cc2k4c1b9895cd0ac3e6@mail.gmail.com>
Subject: Re: add worker nodes when job is running
From: Jeff Zhang <zjffdu@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd32e2e790df9047d6a338d
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd32e2e790df9047d6a338d
Content-Type: text/plain; charset=UTF-8

I don't think you need to do extra anything, the Sheduler on JT will do it
for you. When there's node becoming free, Scheduler will assign new task to
these nodes.


On Mon, Jan 18, 2010 at 2:02 PM, Yanfeng Zhang <zhangyf14@gmail.com> wrote:

> Hello, all
>
> I want to implement this.
> 1. In a 10-nodes cluster, there is a job J1 running on 5 nodes, at the same
> time there is another job J2  running on the other 5 nodes.
> 2. Then J1 finishes earlier and J1's 5 nodes join to work on job J2, so we
> have 10 nodes working for J2.
>
> Could anyone tell me how to implement it?
>
> Thank you very much!
>
>
> Regards,
>



-- 
Best Regards

Jeff Zhang

--000e0cd32e2e790df9047d6a338d--

From general-return-930-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 18 07:21:44 2010
Return-Path: <general-return-930-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88278 invoked from network); 18 Jan 2010 07:21:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2010 07:21:44 -0000
Received: (qmail 50450 invoked by uid 500); 18 Jan 2010 07:21:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49816 invoked by uid 500); 18 Jan 2010 07:21:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49797 invoked by uid 99); 18 Jan 2010 07:21:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Jan 2010 07:21:41 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mailinglists19@gmail.com designates 209.85.219.214 as permitted sender)
Received: from [209.85.219.214] (HELO mail-ew0-f214.google.com) (209.85.219.214)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Jan 2010 07:21:33 +0000
Received: by ewy6 with SMTP id 6so3138364ewy.29
        for <multiple recipients>; Sun, 17 Jan 2010 23:21:13 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=YX8XZz07ujvTypACOrE7l4kNtibbv+Gf5vF3JPggM/Y=;
        b=SpgSTkviDJX0ztMWQQ0JXkDH0E+7uHDfafooTmMex9yv7qZb5bvM+2ZRbodB1P1Ymo
         u1rY19VjUoV+ieUkvj7nCpoz4EkyDTdQ8wzto5IH2+DtYsI78kjPI3jAMKpcXkE+K76Y
         uWkzPJ5Eot8HzRgjbeT4x3qLdw+qyI8VU+FM8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=WCnlqgd3cpqY2RF+m8wXnSDDHLkIYkfoVKD8d1VeBE+UlEgxdzkH/Ec+pYSjfg+RRo
         ei7vsu0LzlWx4quG1EAK+I2sPvCUDcQdEuL9dBuStlx6dpygRpO9aMQj9EI2DlAWXYiy
         HCCrJHFC+3V2DT0wEqy/DKzs/4vrYyFW6IOFU=
MIME-Version: 1.0
Received: by 10.216.90.141 with SMTP id e13mr1749916wef.166.1263799273090; 
	Sun, 17 Jan 2010 23:21:13 -0800 (PST)
Date: Sun, 17 Jan 2010 23:21:13 -0800
Message-ID: <1eabbac31001172321m5a444625vdf13937e12a23552@mail.gmail.com>
Subject: How do I trigger multiple Mapper tasks?
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org, hbase-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6db2d81bd6eb7047d6b3258
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6db2d81bd6eb7047d6b3258
Content-Type: text/plain; charset=ISO-8859-1

Hello,

I read the documentation about running multiple Mapper tasks, but I can't
get multiple Mappers to work.  I am running under EC2 with 10 nodes.

Here's what I know:

1)   I guess, by default, No. of Mapper tasks will be decided by DFS block
size, but I would like to override that.  My file is small, but each line
triggers fairly long running complicated calculations that should be run in
parallel.

2)  I tried setting the following property in the mapred-site.xml (only on
Master), but that doesn't seem to help:

<property>
  <name>mapred.map.tasks</name>
  <value>10</value>
</property>

I still see the following message:

10/01/18 01:56:34 INFO mapred.JobClient:     Launched map tasks=1
10/01/18 01:56:34 INFO mapred.JobClient:     Data-local map tasks=1

(Also, I know for fact that multiple mappers are not running!)


3) I read somewhere that JobConf has a method called setNumMapTasks, but
this class has been deprecated, and as such I am not using.  Besides this
method just provides a hint to Hadoop, I heard.

So how do I trigger multiple Mapper tasks?  Please let me know.  Thanks.

--0016e6db2d81bd6eb7047d6b3258--

From general-return-931-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 18 07:30:18 2010
Return-Path: <general-return-931-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 89694 invoked from network); 18 Jan 2010 07:30:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2010 07:30:18 -0000
Received: (qmail 57259 invoked by uid 500); 18 Jan 2010 07:30:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57169 invoked by uid 500); 18 Jan 2010 07:30:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57151 invoked by uid 99); 18 Jan 2010 07:30:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Jan 2010 07:30:16 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cpbhagtani@gmail.com designates 209.85.160.50 as permitted sender)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Jan 2010 07:30:10 +0000
Received: by pwi20 with SMTP id 20so1678447pwi.29
        for <multiple recipients>; Sun, 17 Jan 2010 23:29:50 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:cc:content-type;
        bh=4pPMypGVKg3h9ai2DLOyu4nODciTuQFMxEULUhVp7Uc=;
        b=xcrRBdukcEJLywS295U169oKmrieeI2qB1bAvZF34polsq+q73z/VPgbvD19YZpoh0
         e65d7Kz1Bx2Zn2l6OJYjScQU8Nnq9ur8CJOCmk17JdY/fHU4xUjyCNd6Y3cfYS6broNv
         5F6GBI/HFeOKeYPPAqCyOx4XWhj4WARR+B9mc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=ovXvdNXcxmiQAubQlGjNTS0z4XbWAHoCbesRuoaYCP409TB+EPZpMEFci2zCGTK9Nq
         Ezn1VdfMr9pb4PMFMy9TGdWlxYK/DA+1tdd/SCunpg4bDueg28fK2tn2wBWa4ToTLyEu
         WnswrnRbv9/OfurHUDcBxCUX8lGHgTlXtS8Z0=
MIME-Version: 1.0
Received: by 10.142.5.30 with SMTP id 30mr3873444wfe.115.1263799789821; Sun, 
	17 Jan 2010 23:29:49 -0800 (PST)
In-Reply-To: <1eabbac31001172321m5a444625vdf13937e12a23552@mail.gmail.com>
References: <1eabbac31001172321m5a444625vdf13937e12a23552@mail.gmail.com>
Date: Mon, 18 Jan 2010 12:59:49 +0530
Message-ID: <4061df21001172329l4242810ag2a278fd02d01a59@mail.gmail.com>
Subject: Re: How do I trigger multiple Mapper tasks?
From: Chandraprakash Bhagtani <cpbhagtani@gmail.com>
To: general@hadoop.apache.org
Cc: hbase-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502b09c8a1e30047d6b51b6

--00504502b09c8a1e30047d6b51b6
Content-Type: text/plain; charset=ISO-8859-1

you can set *mapred.max.split.size* property in mapred-site.xml to create
more splits and map tasks.

On Mon, Jan 18, 2010 at 12:51 PM, Something Something <
mailinglists19@gmail.com> wrote:

> Hello,
>
> I read the documentation about running multiple Mapper tasks, but I can't
> get multiple Mappers to work.  I am running under EC2 with 10 nodes.
>
> Here's what I know:
>
> 1)   I guess, by default, No. of Mapper tasks will be decided by DFS block
> size, but I would like to override that.  My file is small, but each line
> triggers fairly long running complicated calculations that should be run in
> parallel.
>
> 2)  I tried setting the following property in the mapred-site.xml (only on
> Master), but that doesn't seem to help:
>
> <property>
>  <name>mapred.map.tasks</name>
>  <value>10</value>
> </property>
>
> I still see the following message:
>
> 10/01/18 01:56:34 INFO mapred.JobClient:     Launched map tasks=1
> 10/01/18 01:56:34 INFO mapred.JobClient:     Data-local map tasks=1
>
> (Also, I know for fact that multiple mappers are not running!)
>
>
> 3) I read somewhere that JobConf has a method called setNumMapTasks, but
> this class has been deprecated, and as such I am not using.  Besides this
> method just provides a hint to Hadoop, I heard.
>
> So how do I trigger multiple Mapper tasks?  Please let me know.  Thanks.
>



-- 
Thanks & Regards,
Chandra Prakash Bhagtani,
Impetus Infotech (india) Pvt Ltd.

--00504502b09c8a1e30047d6b51b6--

From general-return-932-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 18 07:46:36 2010
Return-Path: <general-return-932-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 92486 invoked from network); 18 Jan 2010 07:46:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2010 07:46:36 -0000
Received: (qmail 67988 invoked by uid 500); 18 Jan 2010 07:46:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67876 invoked by uid 500); 18 Jan 2010 07:46:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67853 invoked by uid 99); 18 Jan 2010 07:46:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Jan 2010 07:46:34 +0000
X-ASF-Spam-Status: No, hits=4.2 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Jan 2010 07:46:23 +0000
Received: from EGL-EX07CAS02.ds.corp.yahoo.com (egl-ex07cas02.eglbp.corp.yahoo.com [203.83.248.209])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o0I7joG7028263;
	Sun, 17 Jan 2010 23:45:51 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:cc:date:subject:thread-topic:
	thread-index:message-id:in-reply-to:accept-language:
	content-language:x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:
	content-type:mime-version;
	b=GYarWCAzIQHs0sBukN+TfbuXPpN0X0X2xTvLFnCV2F4dMB3WMyE7ltbmm0uvTATd
Received: from EGL-EX07VS01.ds.corp.yahoo.com ([203.83.248.205]) by
 EGL-EX07CAS02.ds.corp.yahoo.com ([203.83.248.216]) with mapi; Mon, 18 Jan
 2010 13:15:49 +0530
From: Amareshwari Sri Ramadasu <amarsri@yahoo-inc.com>
To: "mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>,
        "hbase-user@hadoop.apache.org" <hbase-user@hadoop.apache.org>
Date: Mon, 18 Jan 2010 13:15:48 +0530
Subject: Re: How do I trigger multiple Mapper tasks?
Thread-Topic: How do I trigger multiple Mapper tasks?
Thread-Index: AcqYDvS+z4IR3NCwRwm8GkvzEEqB8QAA0sFm
Message-ID: <C77A0F84.FA43%amarsri@yahoo-inc.com>
In-Reply-To: <1eabbac31001172321m5a444625vdf13937e12a23552@mail.gmail.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C77A0F84FA43amarsriyahooinccom_"
MIME-Version: 1.0

--_000_C77A0F84FA43amarsriyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


Changing the audience to mapreduce-user.

Setting the number of map tasks (mapred.map.tasks or JobConf.setNumMapTasks=
()) does not guarantee that number of maps in the job will be set to that. =
It will only be used as a hint. Number of maps is decided by your InputForm=
at. You should implement InputFormat.getSplits() to define how the input sh=
ould be split. The fact is "number of splits is equal to the number of maps=
".
If you are using default InputFormat (i.e. TextInputFormat), number of maps=
 is decided by DFS block size. If you use NLineInputFormat with mapred.line=
.input.format.linespermap=3D1, number of maps will be number of lines in th=
e file.
More details @
http://hadoop.apache.org/common/docs/r0.20.0/api/org/apache/hadoop/mapred/J=
obConf.html#setNumMapTasks%28int%29

Thanks
Amareshwari
On 1/18/10 12:51 PM, "Something Something" <mailinglists19@gmail.com> wrote=
:

Hello,

I read the documentation about running multiple Mapper tasks, but I can't
get multiple Mappers to work.  I am running under EC2 with 10 nodes.

Here's what I know:

1)   I guess, by default, No. of Mapper tasks will be decided by DFS block
size, but I would like to override that.  My file is small, but each line
triggers fairly long running complicated calculations that should be run in
parallel.

2)  I tried setting the following property in the mapred-site.xml (only on
Master), but that doesn't seem to help:

<property>
  <name>mapred.map.tasks</name>
  <value>10</value>
</property>

I still see the following message:

10/01/18 01:56:34 INFO mapred.JobClient:     Launched map tasks=3D1
10/01/18 01:56:34 INFO mapred.JobClient:     Data-local map tasks=3D1

(Also, I know for fact that multiple mappers are not running!)


3) I read somewhere that JobConf has a method called setNumMapTasks, but
this class has been deprecated, and as such I am not using.  Besides this
method just provides a hint to Hadoop, I heard.

So how do I trigger multiple Mapper tasks?  Please let me know.  Thanks.


--_000_C77A0F84FA43amarsriyahooinccom_--

From general-return-933-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 18 13:56:37 2010
Return-Path: <general-return-933-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 11418 invoked from network); 18 Jan 2010 13:56:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2010 13:56:37 -0000
Received: (qmail 26045 invoked by uid 500); 18 Jan 2010 13:56:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25964 invoked by uid 500); 18 Jan 2010 13:56:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25954 invoked by uid 99); 18 Jan 2010 13:56:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Jan 2010 13:56:35 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [15.193.32.62] (HELO g6t0185.atlanta.hp.com) (15.193.32.62)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Jan 2010 13:56:25 +0000
Received: from G6W0640.americas.hpqcorp.net (g6w0640.atlanta.hp.com [16.230.34.76])
	(using TLSv1 with cipher RC4-MD5 (128/128 bits))
	(No client certificate requested)
	by g6t0185.atlanta.hp.com (Postfix) with ESMTPS id 66D3E241A8
	for <general@hadoop.apache.org>; Mon, 18 Jan 2010 13:56:03 +0000 (UTC)
Received: from G5W0325.americas.hpqcorp.net (16.228.8.67) by
 G6W0640.americas.hpqcorp.net (16.230.34.76) with Microsoft SMTP Server (TLS)
 id 8.2.176.0; Mon, 18 Jan 2010 13:55:30 +0000
Received: from GVW1114EXC.americas.hpqcorp.net ([16.228.24.170]) by
 G5W0325.americas.hpqcorp.net ([16.228.8.67]) with mapi; Mon, 18 Jan 2010
 13:55:30 +0000
From: "Day, Phil" <philip.day@hp.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 18 Jan 2010 13:55:29 +0000
Subject: DFS Recovery with fsck
Thread-Topic: DFS Recovery with fsck
Thread-Index: AcqYA+hoCHROLS2nRlKZiHU1Zl5jWAAQNjpg
Message-ID: <C49B6B58853ACD4E96677F24DD9C446C491165C2A1@GVW1114EXC.americas.hpqcorp.net>
References: <bd7d2ecd1001172202g1f4b8979xa1090d395b86c7f6@mail.gmail.com>
In-Reply-To: <bd7d2ecd1001172202g1f4b8979xa1090d395b86c7f6@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Hi All,

Can anyone help me with the following please:

I have a 20.1 cluster where I've been doing some testing on recovering from=
 various namenode failure scenarios.

The current problem I've managed to create is where some directories and th=
e files within them were deleted, the cluster then stopped, and the edits f=
ile lost.

On restart dfs stays in safemode as there are blocks missing (the image kno=
ws about the directories, the datanodes don't have the blocks for them).   =
Fsck correctly identifies the missing blocks.

I then take dfs out of safe mode and run "fsck -delete" (to get rid of the =
corrupt files).   After that a further fsck run reports the filesystem as h=
ealth (and a ls shows the directories as empty).

However if I now stop the cluster and restart it, it comes back into the sa=
me state.   It's as if the results of the "fsck -delete" aren't persisted.

Any thought on what's happening, and what I need to do to tidy up, would be=
 very welcome.

Thanks,
Phil

From general-return-934-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 18 17:27:51 2010
Return-Path: <general-return-934-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 89337 invoked from network); 18 Jan 2010 17:27:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2010 17:27:51 -0000
Received: (qmail 57205 invoked by uid 500); 18 Jan 2010 17:27:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57064 invoked by uid 500); 18 Jan 2010 17:27:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57038 invoked by uid 99); 18 Jan 2010 17:27:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Jan 2010 17:27:48 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mailinglists19@gmail.com designates 74.125.78.26 as permitted sender)
Received: from [74.125.78.26] (HELO ey-out-2122.google.com) (74.125.78.26)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Jan 2010 17:27:42 +0000
Received: by ey-out-2122.google.com with SMTP id 22so488981eye.23
        for <multiple recipients>; Mon, 18 Jan 2010 09:27:20 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:cc:content-type;
        bh=xBz4Gl1mgusFDqSmkEP/730U5I85/K4dU2jCNakybjE=;
        b=UeSzryuQO0C04zpD1CzpPat+fnj31tWckmIXXksY+JElk7LTsC4JcJIupRg/tyMgnB
         JAnco/2zO5+c7kI+TR4FRMHlRbrGmjEStlxNqr04uzMg6AvvV+ZSO/03/1N57yhtyEZk
         B28nJli2kD2G1sUkZ+oHTwP64Qo5VR5+65unk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=ffDZXAIsmO0r0e/B4CPciz+hbGB04yKwyUqx69OYgk6AzCfbpwMdpTtwI7vIcN5nKJ
         0cHu7qNVsqgobosCBR4YiXzJ4qHSqKYaJrlKsVx7ELP+/zo7FBv1bKGthv3q3Kk1corR
         1erBQPHmP6w/RtfuzhpUGgQhXjrbG5utOxQtE=
MIME-Version: 1.0
Received: by 10.216.86.16 with SMTP id v16mr80336wee.162.1263835639993; Mon, 
	18 Jan 2010 09:27:19 -0800 (PST)
In-Reply-To: <C77A0F84.FA43%amarsri@yahoo-inc.com>
References: <1eabbac31001172321m5a444625vdf13937e12a23552@mail.gmail.com>
	 <C77A0F84.FA43%amarsri@yahoo-inc.com>
Date: Mon, 18 Jan 2010 09:27:19 -0800
Message-ID: <1eabbac31001180927y73a1a8c9n8ef7058023bfc48a@mail.gmail.com>
Subject: Re: How do I trigger multiple Mapper tasks?
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org
Cc: "mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>, 
	"hbase-user@hadoop.apache.org" <hbase-user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=0016e6d588b960561a047d73aa17

--0016e6d588b960561a047d73aa17
Content-Type: text/plain; charset=ISO-8859-1

Thanks for the replies.  The NLineInputFormat uses JobConf which has been
deprecated so I would rather not use that class.  But I looked at the
FileInputFormat which has the following method:

    FileInputFormat.setMinInputSplitSize(job, 100);

I thought if I set InputSplitSize to 100, for every 100 lines in the input
file a Mapper would be triggered.  My input file has 500 lines, so I was
expecting to see 5 Mappers, but only one Mapper is triggered.

Please help.  Thanks.


On Sun, Jan 17, 2010 at 11:45 PM, Amareshwari Sri Ramadasu <
amarsri@yahoo-inc.com> wrote:

>
> Changing the audience to mapreduce-user.
>
> Setting the number of map tasks (mapred.map.tasks or
> JobConf.setNumMapTasks()) does not guarantee that number of maps in the job
> will be set to that. It will only be used as a hint. Number of maps is
> decided by your InputFormat. You should implement InputFormat.getSplits() to
> define how the input should be split. The fact is "number of splits is equal
> to the number of maps".
> If you are using default InputFormat (i.e. TextInputFormat), number of maps
> is decided by DFS block size. If you use NLineInputFormat with
> mapred.line.input.format.linespermap=1, number of maps will be number of
> lines in the file.
> More details @
>
> http://hadoop.apache.org/common/docs/r0.20.0/api/org/apache/hadoop/mapred/JobConf.html#setNumMapTasks%28int%29
>
> Thanks
> Amareshwari
> On 1/18/10 12:51 PM, "Something Something" <mailinglists19@gmail.com>
> wrote:
>
> Hello,
>
> I read the documentation about running multiple Mapper tasks, but I can't
> get multiple Mappers to work.  I am running under EC2 with 10 nodes.
>
> Here's what I know:
>
> 1)   I guess, by default, No. of Mapper tasks will be decided by DFS block
> size, but I would like to override that.  My file is small, but each line
> triggers fairly long running complicated calculations that should be run in
> parallel.
>
> 2)  I tried setting the following property in the mapred-site.xml (only on
> Master), but that doesn't seem to help:
>
> <property>
>  <name>mapred.map.tasks</name>
>  <value>10</value>
> </property>
>
> I still see the following message:
>
> 10/01/18 01:56:34 INFO mapred.JobClient:     Launched map tasks=1
> 10/01/18 01:56:34 INFO mapred.JobClient:     Data-local map tasks=1
>
> (Also, I know for fact that multiple mappers are not running!)
>
>
> 3) I read somewhere that JobConf has a method called setNumMapTasks, but
> this class has been deprecated, and as such I am not using.  Besides this
> method just provides a hint to Hadoop, I heard.
>
> So how do I trigger multiple Mapper tasks?  Please let me know.  Thanks.
>
>

--0016e6d588b960561a047d73aa17--

From general-return-935-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 19 12:35:23 2010
Return-Path: <general-return-935-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9218 invoked from network); 19 Jan 2010 12:35:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jan 2010 12:35:23 -0000
Received: (qmail 77398 invoked by uid 500); 19 Jan 2010 12:35:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77307 invoked by uid 500); 19 Jan 2010 12:35:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77297 invoked by uid 99); 19 Jan 2010 12:35:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jan 2010 12:35:21 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [66.163.179.145] (HELO web35606.mail.mud.yahoo.com) (66.163.179.145)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 19 Jan 2010 12:35:12 +0000
Received: (qmail 25362 invoked by uid 60001); 19 Jan 2010 12:34:50 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1263904490; bh=Y2rD6bbWywQ6hQby+6ac3s6aa3xLq8oBeQOQxDbHHZk=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=Cal5EfZWI0Krvz/fFExAjnX4u2Mr814DvGwPanNBZjBlFkDhe2IYvVKxaNAiLGAz1NYlD2zerMfIDzGh6OW9XkCjz1EKuE/EU1+K1YotllZxSaNT0A8XBUmaEU3PcefiVUB1wv7TTwk7A9flG3z3PqlqJfZJ6JbKsNfnbzl9K78=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=N+hdVbxRekbFavSu+3I6p/3wPCOhntXQksirmY1JByMTO2O90StivIkR9LbBwv7/cGPLW1wYBtuHaSBV5JiiebYAFQi4dY3C2zKxh9u3UZENdD5kdFT+KIykHSZeboOTgJ+swFdGpOuoBbIagcvTu43fNNOIji5Uc11BGJg3I1Q=;
Message-ID: <573443.25346.qm@web35606.mail.mud.yahoo.com>
X-YMail-OSG: TskDoGwVM1lK1lklfZmGerwXJ8u2101Dh3hcEv5GKGxYhJOf3INLX7ptY7EAfoe.soBHxDJOKeNPG2eV3GYyGKWCkjTNykNMNU_XOPMlLPSd2wv6l4Lb6yksuTxN_vqJLJpYA9KGv7PbHjLi3nMiiafUk_nyGp51BkA7iDq9g9o2kTtE1zATLiLGGrRS8Ln23EUqC2ZoHF0ktNCJ.Av5HaGjiY.60ZzkokrmCrc2f7f3UiUb6xYCeirlR_YImOXjNBT2bEpaSGXUAQU5NvDGLW4MCu8qjBL1frD63AFJn0B0bhDae4.UwBtaYrizi96G8aZD_kj8K6vKgeS9vBI0jULbob5yGOpamOW6nEtsOjnmFjFhneqz8fxyHQ8F0WoO8_PFBuN.Ou0DBfB26zXYBJr9fFJbxkMplRoblvrQE1Kw_BaO9NmhF6v0YVEZMzywxiEZTEyfxgwj5MnCH8BuVC_qDhjViPlDDxRFxkYxjhBP9VGGXBcJsn023M73HO_M_Epb76pU8jpEoBnc2cyFHQtI3F1rgS3KQP2QlJhobP._W9zf5tVWUhyor9xoUnyMdAMdxo0d4J8Uk1fhFRABn2.ciA8YREgVYSKZN.z9
Received: from [115.186.16.168] by web35606.mail.mud.yahoo.com via HTTP; Tue, 19 Jan 2010 04:34:50 PST
X-Mailer: YahooMailRC/272.7 YahooMailWebService/0.8.100.260964
Date: Tue, 19 Jan 2010 04:34:50 -0800 (PST)
From: ZUBAIR MIAN <mian_zubair@yahoo.com>
Subject: Problem with datanode(slave machine)
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-754832029-1263904490=:25346"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-754832029-1263904490=:25346
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Dear all=0A=0AI=A0am facing problem wrt the datanode(at slave). In the log =
file of slave i am getting following message: =0A=A0=A0=A0 =A0=A0=A0 =A0=A0=
=A0 "Call to master/192.168.0.1:54310 failed on local exception: java.net.N=
oRouteToHostException: =A0=A0=A0 =A0=A0=A0 =A0=A0=A0 No route to host=A0at =
org.apache.hadoop.ipc.Client.wrapException"=0AI am currently working on two=
 node cluster. The network is working fine. Both machines can ping each oth=
er.=0A=0AAnd Following is the log file of datanode.=0A=0A=0A=A0STARTUP_MSG:=
 Starting DataNode=0ASTARTUP_MSG:=A0=A0 host =3D slave/192.168.0.2=0ASTARTU=
P_MSG:=A0=A0 args =3D []=0ASTARTUP_MSG:=A0=A0 version =3D 0.20.1=0ASTARTUP_=
MSG:=A0=A0 build =3D http://svn.apache.org/repos/asf/hadoop/common/tags/rel=
ease-0.20.1-rc1 -r 810220; compiled by 'oom' on Tue Sep=A0 1 20:55:56 UTC 2=
009=0A************************************************************/=0A2010-=
01-15 18:50:56,435 INFO org.apache.hadoop.ipc.Client: Retrying connect to s=
erver: master/192.168.0.1:54310. Already tried 0 time(s).=0A2010-01-15 18:5=
0:57,436 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: mas=
ter/192.168.0.1:54310. Already tried 1 time(s).=0A2010-01-15 18:50:58,437 I=
NFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/192.16=
8.0.1:54310. Already tried 2 time(s).=0A2010-01-15 18:50:59,438 INFO org.ap=
ache.hadoop.ipc.Client: Retrying connect to server: master/192.168.0.1:5431=
0. Already tried 3 time(s).=0A2010-01-15 18:51:00,439 INFO org.apache.hadoo=
p.ipc.Client: Retrying connect to server: master/192.168.0.1:54310. Already=
 tried 4 time(s).=0A2010-01-15 18:51:01,440 INFO org.apache.hadoop.ipc.Clie=
nt: Retrying connect to server: master/192.168.0.1:54310. Already tried 5 t=
ime(s).=0A2010-01-15 18:51:02,441 INFO org.apache.hadoop.ipc.Client: Retryi=
ng connect to server: master/192.168.0.1:54310. Already tried 6 time(s).=0A=
2010-01-15 18:51:03,443 INFO org.apache.hadoop.ipc.Client: Retrying connect=
 to server: master/192.168.0.1:54310. Already tried 7 time(s).=0A2010-01-15=
 18:51:04,444 INFO org.apache.hadoop.ipc.Client: Retrying connect to server=
: master/192.168.0.1:54310. Already tried 8 time(s).=0A2010-01-15 18:51:05,=
445 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: master/1=
92.168.0.1:54310. Already tried 9 time(s).=0A2010-01-15 18:51:05,449 ERROR =
org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Call =
to master/192.168.0.1:54310 failed on local exception: java.net.NoRouteToHo=
stException: No route to host=0A=A0at org.apache.hadoop.ipc.Client.wrapExce=
ption(Client.java:774)=0A=A0at org.apache.hadoop.ipc.Client.call(Client.jav=
a:742)=0A=A0at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)=0A=A0=
at $Proxy4.getProtocolVersion(Unknown Source)=0A=A0at org.apache.hadoop.ipc=
.RPC.getProxy(RPC.java:359)=0A=A0at org.apache.hadoop.ipc.RPC.getProxy(RPC.=
java:346)=0A=A0at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:383)=0A=A0at =
org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:314)=0A=A0at org.apache.had=
oop.ipc.RPC.waitForProxy(RPC.java:291)=0A=A0at org.apache.hadoop.hdfs.serve=
r.datanode.DataNode.startDataNode(DataNode.java:269)=0A=A0at org.apache.had=
oop.hdfs.server.datanode.DataNode.<init>(DataNode.java:216)=0A=A0at org.apa=
che.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1283)=
=0A=A0at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNod=
e(DataNode.java:1238)=0A=A0at org.apache.hadoop.hdfs.server.datanode.DataNo=
de.createDataNode(DataNode.java:1246)=0A=A0at org.apache.hadoop.hdfs.server=
.datanode.DataNode.main(DataNode.java:1368)=0ACaused by: java.net.NoRouteTo=
HostException: No route to host=0A=A0at sun.nio.ch.SocketChannelImpl.checkC=
onnect(Native Method)=0A=A0at sun.nio.ch.SocketChannelImpl.finishConnect(So=
cketChannelImpl.java:592)=0A=A0at org.apache.hadoop.net.SocketIOWithTimeout=
.connect(SocketIOWithTimeout.java:206)=0A=A0at org.apache.hadoop.net.NetUti=
ls.connect(NetUtils.java:404)=0A=A0at org.apache.hadoop.ipc.Client$Connecti=
on.setupIOstreams(Client.java:304)=0A=A0at org.apache.hadoop.ipc.Client$Con=
nection.access$1700(Client.java:176)=0A=A0at org.apache.hadoop.ipc.Client.g=
etConnection(Client.java:859)=0A=A0at org.apache.hadoop.ipc.Client.call(Cli=
ent.java:719)=0A=A0... 13 more=0A2010-01-15 18:51:05,451 INFO org.apache.ha=
doop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: =0A/*********************=
***************************************=0ASHUTDOWN_MSG: Shutting down DataN=
ode at slave/192.168.0.2=0A************************************************=
************/=0A=0APlease help me asap.=0A=0Aregards=0Amaria=0A=0A=0A=0A   =
   
--0-754832029-1263904490=:25346--

From general-return-936-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 19 13:10:29 2010
Return-Path: <general-return-936-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24737 invoked from network); 19 Jan 2010 13:10:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jan 2010 13:10:29 -0000
Received: (qmail 18763 invoked by uid 500); 19 Jan 2010 13:10:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18701 invoked by uid 500); 19 Jan 2010 13:10:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18691 invoked by uid 99); 19 Jan 2010 13:10:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jan 2010 13:10:28 +0000
X-ASF-Spam-Status: No, hits=3.7 required=10.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yangxiao9901@gmail.com designates 209.85.216.204 as permitted sender)
Received: from [209.85.216.204] (HELO mail-px0-f204.google.com) (209.85.216.204)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jan 2010 13:10:19 +0000
Received: by pxi42 with SMTP id 42so4911264pxi.5
        for <general@hadoop.apache.org>; Tue, 19 Jan 2010 05:09:57 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=uDuH40U90U/mMuPOo+Fhjm58vgVyV5qeNwy71ziPESE=;
        b=AxpSq2JxnrHxierBFAudBE+LBAEvzdR+JTL/atCL4pMuJlayQCil7HqZjYrxLru2Ou
         bEUh3+YjWFyJWnObaqvAO4zVJyqK3j+8+JEPSPtxfRfZgu7TCi9GkGXrZP7eX+OHKvCv
         kocagz+obtjWL3sQdBSgzaihzh0jNFa5FFPRk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=am9Zxl25ywjfLHQ6IqQ+KpPQ96fJa5rbqfBPYmYMpuLQe0z7AR50DJHRsduO+qN66A
         8My4eKttjXLdyqIteZ6ixFLZUC+QQdnHdHUe8vS25ryE2CPQsD4hgCXACmiQplF/x32S
         fctim/edgABrE0Fnnmr/mYbTtmK19pXLZWceM=
MIME-Version: 1.0
Received: by 10.141.107.3 with SMTP id j3mr2130902rvm.271.1263906597783; Tue, 
	19 Jan 2010 05:09:57 -0800 (PST)
In-Reply-To: <573443.25346.qm@web35606.mail.mud.yahoo.com>
References: <573443.25346.qm@web35606.mail.mud.yahoo.com>
Date: Tue, 19 Jan 2010 21:09:57 +0800
Message-ID: <5a921af21001190509hec85921u5d8be7f84b9cbe@mail.gmail.com>
Subject: Re: Problem with datanode(slave machine)
From: xiao yang <yangxiao9901@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd137beca4b43047d842fa9
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd137beca4b43047d842fa9
Content-Type: text/plain; charset=ISO-8859-1

Add hosts and ip address in /etc/hosts.
It may work.

On Tue, Jan 19, 2010 at 8:34 PM, ZUBAIR MIAN <mian_zubair@yahoo.com> wrote:

> Dear all
>
> I am facing problem wrt the datanode(at slave). In the log file of slave i
> am getting following message:
>             "Call to master/192.168.0.1:54310 failed on local exception:
> java.net.NoRouteToHostException:             No route to host at
> org.apache.hadoop.ipc.Client.wrapException"
> I am currently working on two node cluster. The network is working fine.
> Both machines can ping each other.
>
> And Following is the log file of datanode.
>
>
>  STARTUP_MSG: Starting DataNode
> STARTUP_MSG:   host = slave/192.168.0.2
> STARTUP_MSG:   args = []
> STARTUP_MSG:   version = 0.20.1
> STARTUP_MSG:   build =
> http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc1 -r
> 810220; compiled by 'oom' on Tue Sep  1 20:55:56 UTC 2009
> ************************************************************/
> 2010-01-15 18:50:56,435 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/192.168.0.1:54310. Already tried 0 time(s).
> 2010-01-15 18:50:57,436 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/192.168.0.1:54310. Already tried 1 time(s).
> 2010-01-15 18:50:58,437 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/192.168.0.1:54310. Already tried 2 time(s).
> 2010-01-15 18:50:59,438 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/192.168.0.1:54310. Already tried 3 time(s).
> 2010-01-15 18:51:00,439 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/192.168.0.1:54310. Already tried 4 time(s).
> 2010-01-15 18:51:01,440 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/192.168.0.1:54310. Already tried 5 time(s).
> 2010-01-15 18:51:02,441 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/192.168.0.1:54310. Already tried 6 time(s).
> 2010-01-15 18:51:03,443 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/192.168.0.1:54310. Already tried 7 time(s).
> 2010-01-15 18:51:04,444 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/192.168.0.1:54310. Already tried 8 time(s).
> 2010-01-15 18:51:05,445 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: master/192.168.0.1:54310. Already tried 9 time(s).
> 2010-01-15 18:51:05,449 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Call
> to master/192.168.0.1:54310 failed on local exception:
> java.net.NoRouteToHostException: No route to host
>  at org.apache.hadoop.ipc.Client.wrapException(Client.java:774)
>  at org.apache.hadoop.ipc.Client.call(Client.java:742)
>  at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
>  at $Proxy4.getProtocolVersion(Unknown Source)
>  at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
>  at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:346)
>  at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:383)
>  at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:314)
>  at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:291)
>  at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:269)
>  at
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:216)
>  at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1283)
>  at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1238)
>  at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1246)
>  at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1368)
> Caused by: java.net.NoRouteToHostException: No route to host
>  at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>  at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
>  at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
>  at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
>  at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:304)
>  at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
>  at org.apache.hadoop.ipc.Client.getConnection(Client.java:859)
>  at org.apache.hadoop.ipc.Client.call(Client.java:719)
>  ... 13 more
> 2010-01-15 18:51:05,451 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down DataNode at slave/192.168.0.2
> ************************************************************/
>
> Please help me asap.
>
> regards
> maria
>
>
>
>

--000e0cd137beca4b43047d842fa9--

From general-return-937-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 19 14:18:30 2010
Return-Path: <general-return-937-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59536 invoked from network); 19 Jan 2010 14:18:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jan 2010 14:18:30 -0000
Received: (qmail 12386 invoked by uid 500); 19 Jan 2010 14:18:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12301 invoked by uid 500); 19 Jan 2010 14:18:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12291 invoked by uid 99); 19 Jan 2010 14:18:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jan 2010 14:18:28 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [66.163.179.141] (HELO web35602.mail.mud.yahoo.com) (66.163.179.141)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 19 Jan 2010 14:18:21 +0000
Received: (qmail 17289 invoked by uid 60001); 19 Jan 2010 14:17:59 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1263910679; bh=DdXwBaqdI+Cu1tG+33dodwpYNrmhnh07XZ2lh572YW0=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=w/vW7K50caddags3Q+IoA1iw6FvHJPq12swcxO6QbGSWxStVWaJvfuYYWkkFyhmehEt7spk+XJZkbXF68OmoB9Y/BKGXt0U5Zqhl6ad3KySoPy1K0alKvqjpt6W3s352AQujYmMq2sW4EKVlt0tV8XhCzO3cuNAMPqNvze09uS8=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=51SjLEWGkT9aSuqIzyMmfafC+7C2BCjcGTqEFmCia40J3btGQcOU2O9jDHLmRnwk93Tq9EK8Dk9H8HzhCLFG7MR8Ahkyh2y2jEqWCOGn3jnCKINr0Ml8/HQGijVzoKmNqSejU6SVpNyzq+D5SKdlpE7v4+UJ5SoI441V83bCeXM=;
Message-ID: <364995.16459.qm@web35602.mail.mud.yahoo.com>
X-YMail-OSG: g5MbopsVM1m4Ms58SeHuIH2XEgokFz73yVv0zQo5D2buyF2muAu3LyEx3G2MWRdKsWPz1i5jIfMpAx0_uRxEIC0nt.6EwhqkggqmHOjxgChHDeXR.pJHdEdF1v853rSCJmeS1i9dIzutICIQ_Ie_0ssealwcgBQRVue7xJsTUuhiU3iz9JEfvfXdkeqSAyGJhpz8i9fmhcgQSCoV08jOUpelpYktL0iKDLWKiTgxF3ViPPzTj38YBSpZUxBk7sF8JkMrcPLtZ0c4lRLZCMQoF.wdb0b98p9N5dW8d3WzUTjst40tuTyC1CDi3eb36R.7G7AsmxVly8GNrXqqijwLbYcWy7FSjZbmzRXL1cDWxnhoT2zSRNkXUeQpT4nN.BbQ4xFKiZzxDjeP7yLCtrU1nknKTWnH_MlPXukglwzMV581NpPxxnaI6xgvbKGdJqdOyJWxmX5xYQj9OrpH6t7XWgM5bdLmcC5OcUhJgwWVmPnRbbBfyW2VJQnPqRMZmlYPZJgeAx6Lqyx5cJFa2wbDdvSG_lOgUKuKUhtjahP9z75fF19oJBNfvv.xPZIHBcgDc._2TatmFOMF.5U4GmoGB6isGRfuDLT1lWdS.GKw9nhA.ElYRnJkLV13E0ImjgVZGdgMTX9bwyBLOmjde14xS82NavKZjF4EvZSjQO1rI2aq07SY_ykSM1ONoAfUGzVH9TJjGLNekYhCRpo.vmqW8ZEoCUHREUDAcyFSxiFFaMwihEeY_gekJnSxTAaYEzqan9hpePy4GRvF65vYxX.bGtcBTGaWS7Jue2GTqAI2OjJtDpkumM6ZLN0_LYBFY0JShyhdjiffbBqB02bEiTZ5YB97nLmk_AGmiuEAjshaTHVpr0uay5oOr03xAtlWgr6pzLTAWpvpDa2UFavYh1Pu.piNwkHSZgGDMot_TS9o1LmkfauUuxkudOvzlfIJ_qJ2gqo.yHdzl9JQiiKkH
 x95NWHybBT1JO_ZBrg-
Received: from [115.186.13.188] by web35602.mail.mud.yahoo.com via HTTP; Tue, 19 Jan 2010 06:17:59 PST
X-Mailer: YahooMailRC/272.7 YahooMailWebService/0.8.100.260964
References: <573443.25346.qm@web35606.mail.mud.yahoo.com> <5a921af21001190509hec85921u5d8be7f84b9cbe@mail.gmail.com>
Date: Tue, 19 Jan 2010 06:17:59 -0800 (PST)
From: ZUBAIR MIAN <mian_zubair@yahoo.com>
Subject: Re: Problem with datanode(slave machine)
To: general@hadoop.apache.org
In-Reply-To: <5a921af21001190509hec85921u5d8be7f84b9cbe@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-902127744-1263910679=:16459"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-902127744-1263910679=:16459
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Have already done that.=0A=0A=0A=0A=0A________________________________=0AFr=
om: xiao yang <yangxiao9901@gmail.com>=0ATo: general@hadoop.apache.org=0ASe=
nt: Tue, January 19, 2010 6:09:57 PM=0ASubject: Re: Problem with datanode(s=
lave machine)=0A=0AAdd hosts and ip address in /etc/hosts.=0AIt may work.=
=0A=0AOn Tue, Jan 19, 2010 at 8:34 PM, ZUBAIR MIAN <mian_zubair@yahoo.com> =
wrote:=0A=0A> Dear all=0A>=0A> I am facing problem wrt the datanode(at slav=
e). In the log file of slave i=0A> am getting following message:=0A>=A0 =A0=
 =A0 =A0 =A0 =A0 "Call to master/192.168.0.1:54310 failed on local exceptio=
n:=0A> java.net.NoRouteToHostException:=A0 =A0 =A0 =A0 =A0 =A0 No route to =
host at=0A> org.apache.hadoop.ipc.Client.wrapException"=0A> I am currently =
working on two node cluster. The network is working fine.=0A> Both machines=
 can ping each other.=0A>=0A> And Following is the log file of datanode.=0A=
>=0A>=0A>=A0 STARTUP_MSG: Starting DataNode=0A> STARTUP_MSG:=A0 host =3D sl=
ave/192.168.0.2=0A> STARTUP_MSG:=A0 args =3D []=0A> STARTUP_MSG:=A0 version=
 =3D 0.20.1=0A> STARTUP_MSG:=A0 build =3D=0A> http://svn.apache.org/repos/a=
sf/hadoop/common/tags/release-0.20.1-rc1 -r=0A> 810220; compiled by 'oom' o=
n Tue Sep=A0 1 20:55:56 UTC 2009=0A> **************************************=
**********************/=0A> 2010-01-15 18:50:56,435 INFO org.apache.hadoop.=
ipc.Client: Retrying connect=0A> to server: master/192.168.0.1:54310. Alrea=
dy tried 0 time(s).=0A> 2010-01-15 18:50:57,436 INFO org.apache.hadoop.ipc.=
Client: Retrying connect=0A> to server: master/192.168.0.1:54310. Already t=
ried 1 time(s).=0A> 2010-01-15 18:50:58,437 INFO org.apache.hadoop.ipc.Clie=
nt: Retrying connect=0A> to server: master/192.168.0.1:54310. Already tried=
 2 time(s).=0A> 2010-01-15 18:50:59,438 INFO org.apache.hadoop.ipc.Client: =
Retrying connect=0A> to server: master/192.168.0.1:54310. Already tried 3 t=
ime(s).=0A> 2010-01-15 18:51:00,439 INFO org.apache.hadoop.ipc.Client: Retr=
ying connect=0A> to server: master/192.168.0.1:54310. Already tried 4 time(=
s).=0A> 2010-01-15 18:51:01,440 INFO org.apache.hadoop.ipc.Client: Retrying=
 connect=0A> to server: master/192.168.0.1:54310. Already tried 5 time(s).=
=0A> 2010-01-15 18:51:02,441 INFO org.apache.hadoop.ipc.Client: Retrying co=
nnect=0A> to server: master/192.168.0.1:54310. Already tried 6 time(s).=0A>=
 2010-01-15 18:51:03,443 INFO org.apache.hadoop.ipc.Client: Retrying connec=
t=0A> to server: master/192.168.0.1:54310. Already tried 7 time(s).=0A> 201=
0-01-15 18:51:04,444 INFO org.apache.hadoop.ipc.Client: Retrying connect=0A=
> to server: master/192.168.0.1:54310. Already tried 8 time(s).=0A> 2010-01=
-15 18:51:05,445 INFO org.apache.hadoop.ipc.Client: Retrying connect=0A> to=
 server: master/192.168.0.1:54310. Already tried 9 time(s).=0A> 2010-01-15 =
18:51:05,449 ERROR=0A> org.apache.hadoop.hdfs.server.datanode.DataNode: jav=
a.io.IOException: Call=0A> to master/192.168.0.1:54310 failed on local exce=
ption:=0A> java.net.NoRouteToHostException: No route to host=0A>=A0 at org.=
apache.hadoop.ipc.Client.wrapException(Client.java:774)=0A>=A0 at org.apach=
e.hadoop.ipc.Client.call(Client.java:742)=0A>=A0 at org.apache.hadoop.ipc.R=
PC$Invoker.invoke(RPC.java:220)=0A>=A0 at $Proxy4.getProtocolVersion(Unknow=
n Source)=0A>=A0 at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)=0A>=A0=
 at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:346)=0A>=A0 at org.apache.h=
adoop.ipc.RPC.getProxy(RPC.java:383)=0A>=A0 at org.apache.hadoop.ipc.RPC.wa=
itForProxy(RPC.java:314)=0A>=A0 at org.apache.hadoop.ipc.RPC.waitForProxy(R=
PC.java:291)=0A>=A0 at=0A> org.apache.hadoop.hdfs.server.datanode.DataNode.=
startDataNode(DataNode.java:269)=0A>=A0 at=0A> org.apache.hadoop.hdfs.serve=
r.datanode.DataNode.<init>(DataNode.java:216)=0A>=A0 at=0A> org.apache.hado=
op.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1283)=0A>=A0 at=
=0A> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(Da=
taNode.java:1238)=0A>=A0 at=0A> org.apache.hadoop.hdfs.server.datanode.Data=
Node.createDataNode(DataNode.java:1246)=0A>=A0 at=0A> org.apache.hadoop.hdf=
s.server.datanode.DataNode.main(DataNode.java:1368)=0A> Caused by: java.net=
.NoRouteToHostException: No route to host=0A>=A0 at sun.nio.ch.SocketChanne=
lImpl.checkConnect(Native Method)=0A>=A0 at sun.nio.ch.SocketChannelImpl.fi=
nishConnect(SocketChannelImpl.java:592)=0A>=A0 at=0A> org.apache.hadoop.net=
.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)=0A>=A0 at org.ap=
ache.hadoop.net.NetUtils.connect(NetUtils.java:404)=0A>=A0 at org.apache.ha=
doop.ipc.Client$Connection.setupIOstreams(Client.java:304)=0A>=A0 at org.ap=
ache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)=0A>=A0 at or=
g.apache.hadoop.ipc.Client.getConnection(Client.java:859)=0A>=A0 at org.apa=
che.hadoop.ipc.Client.call(Client.java:719)=0A>=A0 ... 13 more=0A> 2010-01-=
15 18:51:05,451 INFO=0A> org.apache.hadoop.hdfs.server.datanode.DataNode: S=
HUTDOWN_MSG:=0A> /*********************************************************=
***=0A> SHUTDOWN_MSG: Shutting down DataNode at slave/192.168.0.2=0A> *****=
*******************************************************/=0A>=0A> Please hel=
p me asap.=0A>=0A> regards=0A> maria=0A>=0A>=0A>=0A>=0A=0A=0A=0A      
--0-902127744-1263910679=:16459--

From general-return-938-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 19 18:41:52 2010
Return-Path: <general-return-938-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5967 invoked from network); 19 Jan 2010 18:41:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jan 2010 18:41:52 -0000
Received: (qmail 13764 invoked by uid 500); 19 Jan 2010 18:41:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13703 invoked by uid 500); 19 Jan 2010 18:41:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13693 invoked by uid 99); 19 Jan 2010 18:41:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jan 2010 18:41:51 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.85.216.204] (HELO mail-px0-f204.google.com) (209.85.216.204)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jan 2010 18:41:44 +0000
Received: by pxi42 with SMTP id 42so5400318pxi.5
        for <general@hadoop.apache.org>; Tue, 19 Jan 2010 10:41:24 -0800 (PST)
MIME-Version: 1.0
Received: by 10.143.21.10 with SMTP id y10mr964622wfi.114.1263926484095; Tue, 
	19 Jan 2010 10:41:24 -0800 (PST)
In-Reply-To: <C49B6B58853ACD4E96677F24DD9C446C491165C2A1@GVW1114EXC.americas.hpqcorp.net>
References: <bd7d2ecd1001172202g1f4b8979xa1090d395b86c7f6@mail.gmail.com>
	 <C49B6B58853ACD4E96677F24DD9C446C491165C2A1@GVW1114EXC.americas.hpqcorp.net>
Date: Tue, 19 Jan 2010 10:41:24 -0800
Message-ID: <dfe484f01001191041m73fa2799j30ecf50270c4b7bf@mail.gmail.com>
Subject: Re: DFS Recovery with fsck
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Mon, Jan 18, 2010 at 5:55 AM, Day, Phil <philip.day@hp.com> wrote:
> Hi All,
>
> Can anyone help me with the following please:
>
> I have a 20.1 cluster where I've been doing some testing on recovering fr=
om various namenode failure scenarios.
>
> The current problem I've managed to create is where some directories and =
the files within them were deleted, the cluster then stopped, and the edits=
 file lost.
>
> On restart dfs stays in safemode as there are blocks missing (the image k=
nows about the directories, the datanodes don't have the blocks for them). =
=A0 Fsck correctly identifies the missing blocks.
>
> I then take dfs out of safe mode and run "fsck -delete" (to get rid of th=
e corrupt files). =A0 After that a further fsck run reports the filesystem =
as health (and a ls shows the directories as empty).
>
> However if I now stop the cluster and restart it, it comes back into the =
same state. =A0 It's as if the results of the "fsck -delete" aren't persist=
ed.
>
> Any thought on what's happening, and what I need to do to tidy up, would =
be very welcome.

Hey Phil,

Just to confirm, after restarting after the fsck -delete you can
hadoop fs -ls the directory you deleted and the files have returned?
It sounds like the edits generated by fsck -delete are getting lost.

When you shutdown after the fsck -delete are all copes of the fsimage
and edits the same (assuming you've got multiple fs.name.dirs)?

Seeing the namenode edit log before/during the fsck -delete and on
restart would be helpful.

Thanks,
Eli

From general-return-939-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 19 18:43:13 2010
Return-Path: <general-return-939-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6738 invoked from network); 19 Jan 2010 18:43:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jan 2010 18:43:13 -0000
Received: (qmail 15639 invoked by uid 500); 19 Jan 2010 18:43:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15553 invoked by uid 500); 19 Jan 2010 18:43:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15536 invoked by uid 99); 19 Jan 2010 18:43:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jan 2010 18:43:12 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jan 2010 18:43:04 +0000
Received: by pwi20 with SMTP id 20so2789756pwi.29
        for <general@hadoop.apache.org>; Tue, 19 Jan 2010 10:42:42 -0800 (PST)
MIME-Version: 1.0
Received: by 10.142.4.22 with SMTP id 22mr2096068wfd.127.1263926562443; Tue, 
	19 Jan 2010 10:42:42 -0800 (PST)
In-Reply-To: <dfe484f01001191041m73fa2799j30ecf50270c4b7bf@mail.gmail.com>
References: <bd7d2ecd1001172202g1f4b8979xa1090d395b86c7f6@mail.gmail.com>
	 <C49B6B58853ACD4E96677F24DD9C446C491165C2A1@GVW1114EXC.americas.hpqcorp.net>
	 <dfe484f01001191041m73fa2799j30ecf50270c4b7bf@mail.gmail.com>
Date: Tue, 19 Jan 2010 10:42:42 -0800
Message-ID: <dfe484f01001191042j65919266ice831955d48715c9@mail.gmail.com>
Subject: Re: DFS Recovery with fsck
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

> Seeing the namenode edit log before/during the fsck -delete and on
> restart would be helpful.

Sorry, meant to say "namenode log" not "namenode edit log"

Thanks,
Eli

From general-return-940-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 19 19:39:20 2010
Return-Path: <general-return-940-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21852 invoked from network); 19 Jan 2010 19:39:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jan 2010 19:39:20 -0000
Received: (qmail 82636 invoked by uid 500); 19 Jan 2010 19:39:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82564 invoked by uid 500); 19 Jan 2010 19:39:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82554 invoked by uid 99); 19 Jan 2010 19:39:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jan 2010 19:39:19 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates 209.85.160.50 as permitted sender)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jan 2010 19:39:11 +0000
Received: by pwi20 with SMTP id 20so2830130pwi.29
        for <general@hadoop.apache.org>; Tue, 19 Jan 2010 11:38:50 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=UfWacV5H52JSp0hNAO3dmeiYZa73+RTyrt6cWBhZG/U=;
        b=qyO0drBBMCz2XEf+05dlbufffpd0jpZfMJSxxL+cZ3qf234ehFEsZqoILhxO6Dw0Sw
         UPTEo7uT5K+4HHfFO460Neera5NKRp8jPmd8SgJr0tgPMJFaTGzwgaf6xFXmisctDCah
         Mjhn/wv1OzllSB1BZKxRRTAj57yIxK9mDVknU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=hvDADJ6DgrtvYg9rNxvGUtPC8AehFAgWgAUPTGkcgW/XU4h/hDS7rDTkYacYnSeT1i
         OEfDisZmUeBmk1ku2vAbwwCTfAGQ5p5+q+euTZUGcBlGPMfuHlWbnxlRpWk8/TTkNeio
         Q00sf6yo48180dMmqw1Bo4QflnLLHGt0mGmDU=
MIME-Version: 1.0
Received: by 10.142.247.33 with SMTP id u33mr4903663wfh.113.1263929930055; 
	Tue, 19 Jan 2010 11:38:50 -0800 (PST)
Date: Tue, 19 Jan 2010 11:38:50 -0800
Message-ID: <17e273101001191138w4e358d1cja26bc20793d30e26@mail.gmail.com>
Subject: splittable encrypted compressed files
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502cc62808096047d899e17
X-Virus-Checked: Checked by ClamAV on apache.org

--00504502cc62808096047d899e17
Content-Type: text/plain; charset=ISO-8859-1

Hi,
I experimented with hadoop-lzo and liked it.

Our requirement is changing and we may need to encrypt data files. Does
anyone know of splittable encrypted compression scheme that map task can use
?

Thanks

--00504502cc62808096047d899e17--

From general-return-941-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 19 19:55:56 2010
Return-Path: <general-return-941-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25433 invoked from network); 19 Jan 2010 19:55:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jan 2010 19:55:55 -0000
Received: (qmail 1571 invoked by uid 500); 19 Jan 2010 19:55:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1493 invoked by uid 500); 19 Jan 2010 19:55:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1483 invoked by uid 99); 19 Jan 2010 19:55:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jan 2010 19:55:54 +0000
X-ASF-Spam-Status: No, hits=-1.8 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of Joseph.Travaglini@fmr.com designates 192.223.198.27 as permitted sender)
Received: from [192.223.198.27] (HELO maillnx-us112.fmr.com) (192.223.198.27)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jan 2010 19:55:43 +0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=fmr.com; i=Joseph.Travaglini@fmr.com; l=5734;
  q=dns/txt; s=2009-03-17; t=1263930942; x=1295466942;
  h=x-mimeole:content-class:mime-version:content-type:
   subject:date:message-id:x-ms-has-attach:
   x-ms-tnef-correlator:thread-topic:thread-index:from:to:
   return-path:x-originalarrivaltime:x-filenames;
  z=x-mimeole:=20Produced=20By=20Microsoft=20Exchange=20V6.0
   .6619.12|content-class:=20urn:content-classes:message
   |MIME-Version:=201.0|Content-Type:=20multipart/alternativ
   e=3B=0D=0A=09boundary=3D"----_=3D_NextPart_001_01CA9941.5
   4388090"|Subject:=20Google=20Patent=20on=20MapReduce
   |Date:=20Tue,=2019=20Jan=202010=2014:55:19=20-0500
   |Message-ID:=20<CA7285AEA035E041BCF11DBDA646D000048183A9@
   MSGMROCLN2WIN.DMN1.FMR.COM>|X-MS-Has-Attach:=20
   |X-MS-TNEF-Correlator:=20|Thread-Topic:=20Google=20Patent
   =20on=20MapReduce|Thread-Index:=20AcqZQVUvwewdiv0KTKGANU9
   yX2ox5A=3D=3D|From:=20"Travaglini,=20Joseph"=20<Joseph.Tr
   avaglini@FMR.com>|To:=20<general@hadoop.apache.org>
   |Return-Path:=20Joseph.Travaglini@FMR.com
   |X-OriginalArrivalTime:=2019=20Jan=202010=2019:55:19.0953
   =20(UTC)=20FILETIME=3D[5449D810:01CA9941]|X-filenames:=20
   None;
  bh=j60EYsfLgkcAePWFORvvHjlga2UBpgpjSYCwr5FGYHU=;
  b=qT/OZOpktZfg49TR/79giElFUj5TB1AXUcazMTEE6wwjTH/UFyM02NWA
   C2G8aLQpQk4iqki9Ppxa8bgNKInaZ0smkxCOMdA1A8rEhEc9Tu1kQfS+R
   zWzLqu7/3EQsqmk1JCawj++htC3EWIPlQXiEZv7SBhCxwJoRMlnw+CM7J
   Q=;
X-filenames: None
Received: from msgmrosm01win.dmn1.fmr.com ([172.26.7.127])
  by maillnx-us112.fmr.com with SMTP; 19 Jan 2010 14:55:20 -0500
Received: from MSGMROIV01WIN.DMN1.FMR.COM (172.26.31.106)
	by MSGMROSM01WIN.dmn1.fmr.com (Sigaba Gateway v4.1)
	with ESMTP id 302009737; Tue, 19 Jan 2010 14:55:20 -0500
Received: from MSGMMKIM01WIN.DMN1.FMR.COM ([172.25.108.46]) by MSGMROIV01WIN.DMN1.FMR.COM with SMTP_server; Tue, 19 Jan 2010 14:55:20 -0500
Received: from msgmrocln2win.DMN1.FMR.COM ([10.37.181.15]) by MSGMMKIM01WIN.DMN1.FMR.COM with Microsoft SMTPSVC(5.0.2195.6713);
	 Tue, 19 Jan 2010 14:55:19 -0500
x-mimeole: Produced By Microsoft Exchange V6.0.6619.12
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01CA9941.54388090"
Subject: Google Patent on MapReduce
Date: Tue, 19 Jan 2010 14:55:19 -0500
Message-ID: <CA7285AEA035E041BCF11DBDA646D000048183A9@MSGMROCLN2WIN.DMN1.FMR.COM>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Google Patent on MapReduce
Thread-Index: AcqZQVUvwewdiv0KTKGANU9yX2ox5A==
From: "Travaglini, Joseph" <Joseph.Travaglini@FMR.com>
To: <general@hadoop.apache.org>
X-OriginalArrivalTime: 19 Jan 2010 19:55:19.0953 (UTC) FILETIME=[5449D810:01CA9941]
X-Virus-Checked: Checked by ClamAV on apache.org

------_=_NextPart_001_01CA9941.54388090
Content-Type: text/plain;
	charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable

http://patft.uspto.gov/netacgi/nph-Parser?Sect1=3DPTO1&Sect2=3DHITOFF&d=3D=
PALL
&p=3D1&u=3D%2Fnetahtml%2FPTO%2Fsrchnum.htm&r=3D1&f=3DG&l=3D50&s1=3D7,650,=
331.PN.&OS=3D
PN/7,650,331&RS=3DPN/7,650,331

=20

The USPTO granted a patent to Google today for MapReduce, after several
rejections.  I'm no patent expert; how does this affect Hadoop?


------_=_NextPart_001_01CA9941.54388090--

From general-return-942-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 19 20:43:57 2010
Return-Path: <general-return-942-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39284 invoked from network); 19 Jan 2010 20:43:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jan 2010 20:43:57 -0000
Received: (qmail 81621 invoked by uid 500); 19 Jan 2010 20:43:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81542 invoked by uid 500); 19 Jan 2010 20:43:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81532 invoked by uid 99); 19 Jan 2010 20:43:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jan 2010 20:43:56 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 209.85.216.204 is neither permitted nor denied by domain of peter@motally.com)
Received: from [209.85.216.204] (HELO mail-px0-f204.google.com) (209.85.216.204)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jan 2010 20:43:46 +0000
Received: by pxi42 with SMTP id 42so5583302pxi.5
        for <general@hadoop.apache.org>; Tue, 19 Jan 2010 12:43:25 -0800 (PST)
MIME-Version: 1.0
Received: by 10.141.124.7 with SMTP id b7mr5723556rvn.8.1263933804465; Tue, 19 
	Jan 2010 12:43:24 -0800 (PST)
In-Reply-To: <CA7285AEA035E041BCF11DBDA646D000048183A9@MSGMROCLN2WIN.DMN1.FMR.COM>
References: <CA7285AEA035E041BCF11DBDA646D000048183A9@MSGMROCLN2WIN.DMN1.FMR.COM>
Date: Tue, 19 Jan 2010 12:43:24 -0800
Message-ID: <2c44f6691001191243t418989cav28e1d072e59e6e21@mail.gmail.com>
Subject: Re: Google Patent on MapReduce
From: Peter Sankauskas <peter@motally.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000325564d6a6f4f0f047d8a8591
X-Virus-Checked: Checked by ClamAV on apache.org

--000325564d6a6f4f0f047d8a8591
Content-Type: text/plain; charset=UTF-8

I have heard that Google generally get patents to protect themselves rather
than to enforce them. Google don't have any publicly available way to make
money off of map-reduce (even Google App Engine does not map-reduce
capability yet) so I would imagine they would be fine with Hadoop. Fingers
crossed.

Kind regards,
Peter Sankauskas

Motally, Inc
Office: +1 (415) 932-6898


On Tue, Jan 19, 2010 at 11:55 AM, Travaglini, Joseph <
Joseph.Travaglini@fmr.com> wrote:

> http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PALL
> &p=1&u=%2Fnetahtml%2FPTO%2Fsrchnum.htm&r=1&f=G&l=50&s1=7,650,331.PN.&OS=
> PN/7,650,331&RS=PN/7,650,331<http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PALL%0A&p=1&u=%2Fnetahtml%2FPTO%2Fsrchnum.htm&r=1&f=G&l=50&s1=7,650,331.PN.&OS=%0APN/7,650,331&RS=PN/7,650,331>
>
>
>
> The USPTO granted a patent to Google today for MapReduce, after several
> rejections.  I'm no patent expert; how does this affect Hadoop?
>
>

--000325564d6a6f4f0f047d8a8591--

From general-return-943-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 19 22:58:47 2010
Return-Path: <general-return-943-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 738 invoked from network); 19 Jan 2010 22:58:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jan 2010 22:58:46 -0000
Received: (qmail 56482 invoked by uid 500); 19 Jan 2010 22:58:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56399 invoked by uid 500); 19 Jan 2010 22:58:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56389 invoked by uid 99); 19 Jan 2010 22:58:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jan 2010 22:58:45 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of vermaabhishekp@gmail.com designates 74.125.78.24 as permitted sender)
Received: from [74.125.78.24] (HELO ey-out-2122.google.com) (74.125.78.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jan 2010 22:58:38 +0000
Received: by ey-out-2122.google.com with SMTP id 22so866791eye.5
        for <general@hadoop.apache.org>; Tue, 19 Jan 2010 14:58:17 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=NzUowKAOVU4/AK/8TEvjnKImpgCtXuGj18BM8a5ugxk=;
        b=X/CbL4IppUJS0oC3KHlztG9lKsyU4WjDnDvu1gOxRhF/osHGy6M/B7rZaBu56KghYi
         G2F9+rt2vG9rkLmbH8BmnmrF9Ck3mqGeSm1DUX5naC4h/QqK4dFAXKbbIkosTRLBnzzE
         CHZJzXYoAckazJxwlL3sUQp+6XKqXpT0l0tKY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=g2JLJYq17weUor1BO6DTV7lQgRnEUMtBdsL/wtaFoiqDO2d/F5KSskRwL6qtUv+6mr
         MUNeoX9UQQ6uiWF6r86Kdn+wyOirYoz/0qy+Py4rjnB0wtlo/3GBtA2aeaibjL+uKna4
         +iEBv6XUsEkrWy2JWA62G9PdKnZDtXnHo9ex8=
MIME-Version: 1.0
Received: by 10.213.107.13 with SMTP id z13mr2677951ebo.68.1263941897453; Tue, 
	19 Jan 2010 14:58:17 -0800 (PST)
In-Reply-To: <309f76d01001160248y7a2a605fua4009c7af1ad8932@mail.gmail.com>
References: <309f76d01001131536x5c4d3e66xa9a67457719967ac@mail.gmail.com>
	 <3c682ecd1001131544x43180d70i1992a0ab237dd8f1@mail.gmail.com>
	 <309f76d01001160248y7a2a605fua4009c7af1ad8932@mail.gmail.com>
Date: Tue, 19 Jan 2010 16:58:17 -0600
Message-ID: <3c682ecd1001191458w1299b99bhe68acbd57e2c268e@mail.gmail.com>
Subject: Re: SCALING GENETIC ALGORITHMS USING MAPREDUCE
From: Abhishek Verma <vermaabhishekp@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502d2a1d082ae047d8c67c9

--00504502d2a1d082ae047d8c67c9
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Alberto,

If you need it, I can send you the source code that I used for the paper I
wrote. Do you have any conference in mind for this idea?

Thanks,
-Abhishek.

On Sat, Jan 16, 2010 at 4:48 AM, Alberto Luengo Cabanillas <
cabiwan@gmail.com> wrote:

> Hi Abhishek Verma. Your help would be very appreciated. I=B4ll try to
> implement your suggestions and feed them back if successful.
> Regards.
>
> 2010/1/14 Abhishek Verma <vermaabhishekp@gmail.com>
>
> > Hi Alberto,
> >
> > The paper considers only selecto-recombinative genetic algorithms as
> > mentioned in Section III.A. The mutation operators could be done on the
> > reduce  after the crossover or before it as required. Elitism can be
> > implemented by emitting the individual with a different value in the ma=
p
> > and
> > then directly writing it to context in the reduce.
> >
> > If you are up for it, we could collaborate together to work on your
> > problem.
> >
> > Hope this helps.
> > -Abhishek.
> >
> > On Wed, Jan 13, 2010 at 5:36 PM, Alberto Luengo Cabanillas <
> > cabiwan@gmail.com> wrote:
> >
> > > Hi everyone! For the last six months. my work with Hadoop is being
> > focused
> > > in developing a stable MRPGA. Last paper I read ("Scaling Genetic
> > > Algorithms
> > > Using MapReduce") was a fantastic job and gave me a bunch of ideas; b=
ut
> I
> > > have some questions relative to this paper and I think they may be
> useful
> > > for community:
> > >
> > > Anywhere in the paper talks about elitism rate nor mutation rate. It
> only
> > > talks about selection and crossover. In fact, this part (page 3 and s=
o)
> > > talks about an INDIVIDUALREPRESENTATION(key) function, which I suppos=
e
> is
> > > used to represent the key part of the par (i.e., if it is Text, its
> > > representation is a String). Also there are TOURN(tournArray) (?) and
> > > CROSSOVER(crossArray), which I think is related to mutation.
> > >
> > > How it is supposed to be implemented the mutation part in the process=
?.
> > > Have
> > > you considered some kind of elitism rate for chossing population?.
> > >
> > > Thanks a lot in advance.
> > >
> > >
> > > --
> > > Alberto
> > >
> >
>
>
>
> --
> Alberto
>

--00504502d2a1d082ae047d8c67c9--

From general-return-944-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 19 23:00:32 2010
Return-Path: <general-return-944-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1349 invoked from network); 19 Jan 2010 23:00:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jan 2010 23:00:31 -0000
Received: (qmail 58826 invoked by uid 500); 19 Jan 2010 23:00:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58756 invoked by uid 500); 19 Jan 2010 23:00:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58746 invoked by uid 99); 19 Jan 2010 23:00:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jan 2010 23:00:30 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of vermaabhishekp@gmail.com designates 74.125.78.27 as permitted sender)
Received: from [74.125.78.27] (HELO ey-out-2122.google.com) (74.125.78.27)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jan 2010 23:00:24 +0000
Received: by ey-out-2122.google.com with SMTP id 22so807501eye.23
        for <general@hadoop.apache.org>; Tue, 19 Jan 2010 15:00:02 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=g67DCjd/KjloQEyDDeQJWLZBHfu3GymVrxafnLfl4zI=;
        b=xImC56nCVpnLgJsXt607FYNcEJFqbBodMVf8rfePNjbwkxM2NQuuBENSg9ztRiPjEf
         WiQHwz/4h2fdcQs/RGsZqVgwas922zYhGkhJ9W3JXHNYCKdm7lbAWXZw8OBSRz9QE73L
         ylhBjMyKfdEWMEWDgqvOaJg/6fnDqRRjKNIW4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=PCTq7xozWuMR5O9JljES9F+//F5fmYTtQOetDHIwjj1v3ePJZ+dQ98pAkB5JtDXfQl
         Hfg1ei4h1JIJhDIyHPv0Lw1opDiYbpXnsae/AH+6NFmYxywbKzwb4ofImIF68hpR2o1l
         FwmtQq/LKXUoA7lyatC5e1xZ/Mdgt95kYEHfs=
MIME-Version: 1.0
Received: by 10.213.97.83 with SMTP id k19mr105738ebn.29.1263942002437; Tue, 
	19 Jan 2010 15:00:02 -0800 (PST)
In-Reply-To: <3c682ecd1001191458w1299b99bhe68acbd57e2c268e@mail.gmail.com>
References: <309f76d01001131536x5c4d3e66xa9a67457719967ac@mail.gmail.com>
	 <3c682ecd1001131544x43180d70i1992a0ab237dd8f1@mail.gmail.com>
	 <309f76d01001160248y7a2a605fua4009c7af1ad8932@mail.gmail.com>
	 <3c682ecd1001191458w1299b99bhe68acbd57e2c268e@mail.gmail.com>
Date: Tue, 19 Jan 2010 17:00:02 -0600
Message-ID: <3c682ecd1001191500u205b529ekfe34a7b329420b4@mail.gmail.com>
Subject: Re: SCALING GENETIC ALGORITHMS USING MAPREDUCE
From: Abhishek Verma <vermaabhishekp@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0ce00482126cdf047d8c6eb0

--000e0ce00482126cdf047d8c6eb0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Apologies for spamming everybody.

-Abhishek.

On Tue, Jan 19, 2010 at 4:58 PM, Abhishek Verma <vermaabhishekp@gmail.com>w=
rote:

> Hi Alberto,
>
> If you need it, I can send you the source code that I used for the paper =
I
> wrote. Do you have any conference in mind for this idea?
>
> Thanks,
> -Abhishek.
>
>
> On Sat, Jan 16, 2010 at 4:48 AM, Alberto Luengo Cabanillas <
> cabiwan@gmail.com> wrote:
>
>> Hi Abhishek Verma. Your help would be very appreciated. I=B4ll try to
>> implement your suggestions and feed them back if successful.
>> Regards.
>>
>> 2010/1/14 Abhishek Verma <vermaabhishekp@gmail.com>
>>
>> > Hi Alberto,
>> >
>> > The paper considers only selecto-recombinative genetic algorithms as
>> > mentioned in Section III.A. The mutation operators could be done on th=
e
>> > reduce  after the crossover or before it as required. Elitism can be
>> > implemented by emitting the individual with a different value in the m=
ap
>> > and
>> > then directly writing it to context in the reduce.
>> >
>> > If you are up for it, we could collaborate together to work on your
>> > problem.
>> >
>> > Hope this helps.
>> > -Abhishek.
>> >
>> > On Wed, Jan 13, 2010 at 5:36 PM, Alberto Luengo Cabanillas <
>> > cabiwan@gmail.com> wrote:
>> >
>> > > Hi everyone! For the last six months. my work with Hadoop is being
>> > focused
>> > > in developing a stable MRPGA. Last paper I read ("Scaling Genetic
>> > > Algorithms
>> > > Using MapReduce") was a fantastic job and gave me a bunch of ideas;
>> but I
>> > > have some questions relative to this paper and I think they may be
>> useful
>> > > for community:
>> > >
>> > > Anywhere in the paper talks about elitism rate nor mutation rate. It
>> only
>> > > talks about selection and crossover. In fact, this part (page 3 and
>> so)
>> > > talks about an INDIVIDUALREPRESENTATION(key) function, which I suppo=
se
>> is
>> > > used to represent the key part of the par (i.e., if it is Text, its
>> > > representation is a String). Also there are TOURN(tournArray) (?) an=
d
>> > > CROSSOVER(crossArray), which I think is related to mutation.
>> > >
>> > > How it is supposed to be implemented the mutation part in the
>> process?.
>> > > Have
>> > > you considered some kind of elitism rate for chossing population?.
>> > >
>> > > Thanks a lot in advance.
>> > >
>> > >
>> > > --
>> > > Alberto
>> > >
>> >
>>
>>
>>
>> --
>> Alberto
>>
>
>

--000e0ce00482126cdf047d8c6eb0--

From general-return-945-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 19 23:12:17 2010
Return-Path: <general-return-945-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 3748 invoked from network); 19 Jan 2010 23:12:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jan 2010 23:12:17 -0000
Received: (qmail 68071 invoked by uid 500); 19 Jan 2010 23:12:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67988 invoked by uid 500); 19 Jan 2010 23:12:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67978 invoked by uid 99); 19 Jan 2010 23:12:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jan 2010 23:12:16 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of vermaabhishekp@gmail.com designates 74.125.78.24 as permitted sender)
Received: from [74.125.78.24] (HELO ey-out-2122.google.com) (74.125.78.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jan 2010 23:12:06 +0000
Received: by ey-out-2122.google.com with SMTP id 22so809826eye.23
        for <general@hadoop.apache.org>; Tue, 19 Jan 2010 15:11:46 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=8uylb579N+7VYqn6gI1M2fKgJDnsq/mVDc5FaGdtlxA=;
        b=rrGB968GyxApjTV70KmbCCwQJCXgRYVMqEzTLVTG0tljYl3DwwXzsHt7Uvk/INtyH0
         NlnF5jHPEwxABfvto4c7PNb+C0hRomc/ItW7P7jjWRbev1aFFQ39LuWPcmRbnTzUomrB
         ncXbAFHG25jckXOxLTo8KblR27SP8DnIfqD48=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=G8wzNZaradw+zZ4/0hJfoiU9YBvfh5YJL3xozd8TaWOfKkJWvQOa2+LUf+TwU8b4z5
         jWyONzWRSI+E7pOOCOFmWAZh80cphEfQjsWdlnbaJNPGKDwEziCKsatyik3SB7Pj5yfE
         UHn0DeBjHCq0TJBJmunPpid9iPUqdxHmrpM7E=
MIME-Version: 1.0
Received: by 10.213.100.153 with SMTP id y25mr148172ebn.76.1263942705646; Tue, 
	19 Jan 2010 15:11:45 -0800 (PST)
In-Reply-To: <35dbe9101001170057l4e3fb0f8yc1ca9ebf4f1c7c7c@mail.gmail.com>
References: <309f76d01001131536x5c4d3e66xa9a67457719967ac@mail.gmail.com>
	 <35dbe9101001170057l4e3fb0f8yc1ca9ebf4f1c7c7c@mail.gmail.com>
Date: Tue, 19 Jan 2010 17:11:45 -0600
Message-ID: <3c682ecd1001191511w271e08e4x3cad9f73cbbe61e3@mail.gmail.com>
Subject: Re: SCALING GENETIC ALGORITHMS USING MAPREDUCE
From: Abhishek Verma <vermaabhishekp@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c5b640fc8d68047d8c973a
X-Virus-Checked: Checked by ClamAV on apache.org

--001636c5b640fc8d68047d8c973a
Content-Type: text/plain; charset=ISO-8859-1

Hi Alex,

On Sun, Jan 17, 2010 at 2:57 AM, Alex Baranov <alex.baranov.v@gmail.com>wrote:

> Hello,
>
> I've read the paper and here is my question:
>
> Why not just produce pairs (random int, individual with fitness) from Map
> function? Thus individuals will be shuffled randomly after Map phase and
> there won't be the need to override the partitioner.
>
That is a neat trick and would work.

P.S. Mentioning GFS in "An accompanying distributed file system like GFS [8]
> makes the data management scalable and fault tolerant."  can confuse some
> readers because the paper is based on Hadoop family (and further HDFS name
> used).

I tend to use MapReduce and GFS through out the paper and then mention
Hadoop and HDFS in the implementation section as a concrete example. I
apologize if the paper didn't make that clear.

-- 
-Abhishek.
http://verma7.com

--001636c5b640fc8d68047d8c973a--

From general-return-946-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 20 08:51:37 2010
Return-Path: <general-return-946-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 80523 invoked from network); 20 Jan 2010 08:51:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jan 2010 08:51:36 -0000
Received: (qmail 15471 invoked by uid 500); 20 Jan 2010 08:51:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15385 invoked by uid 500); 20 Jan 2010 08:51:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15375 invoked by uid 99); 20 Jan 2010 08:51:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Jan 2010 08:51:35 +0000
X-ASF-Spam-Status: No, hits=3.7 required=10.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yangxiao9901@gmail.com designates 209.85.222.175 as permitted sender)
Received: from [209.85.222.175] (HELO mail-pz0-f175.google.com) (209.85.222.175)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Jan 2010 08:51:26 +0000
Received: by pzk5 with SMTP id 5so387454pzk.30
        for <general@hadoop.apache.org>; Wed, 20 Jan 2010 00:51:04 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=RlqJBdgXRztWyHHTPrkry40UUYGU+zBh4fmPE6XP8W0=;
        b=dTR9EcXWtS6gAbN5Pa/BtjKflHR1hJA39f3LF51akY9tMTANcgTW/KWh30Yu/QAU6O
         Une9hUJWrSbAF/Wveyqmgvw+P0tC7vEltvWZp4XEk2TzRkcsfjbFr2A33pg9SdbqgOkl
         OuBPt4XUeN15Aw0FRDlmvoUmqqy8KkgwY1gPo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=DpxHDlgcC1xfl8Wq317xWPdxedcqSId/zOVlMATBRa8aQqRcolhpTpowDoV1wNbcjy
         jnaCwIlJnyuwbwMOkMXsfN2sBZDagynWyFuSF4qdb+SVEGK2+PkmriJxa6uiZY9hQEyD
         OK7IRSWGuBoUaN+Td83Ew4l+pcMXsum9xvgCI=
MIME-Version: 1.0
Received: by 10.141.101.21 with SMTP id d21mr2642362rvm.41.1263977464033; Wed, 
	20 Jan 2010 00:51:04 -0800 (PST)
In-Reply-To: <364995.16459.qm@web35602.mail.mud.yahoo.com>
References: <573443.25346.qm@web35606.mail.mud.yahoo.com>
	 <5a921af21001190509hec85921u5d8be7f84b9cbe@mail.gmail.com>
	 <364995.16459.qm@web35602.mail.mud.yahoo.com>
Date: Wed, 20 Jan 2010 16:51:04 +0800
Message-ID: <5a921af21001200051p509b27f0jce1a11663f795154@mail.gmail.com>
Subject: Re: Problem with datanode(slave machine)
From: xiao yang <yangxiao9901@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd1383ebf729a047d94af10
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd1383ebf729a047d94af10
Content-Type: text/plain; charset=ISO-8859-1

Maybe the firewall blocked this port.
Try:
$telnet 192.168.0.1 54310

On Tue, Jan 19, 2010 at 10:17 PM, ZUBAIR MIAN <mian_zubair@yahoo.com> wrote:

> Have already done that.
>
>
>
>
> ________________________________
> From: xiao yang <yangxiao9901@gmail.com>
> To: general@hadoop.apache.org
> Sent: Tue, January 19, 2010 6:09:57 PM
> Subject: Re: Problem with datanode(slave machine)
>
> Add hosts and ip address in /etc/hosts.
> It may work.
>
> On Tue, Jan 19, 2010 at 8:34 PM, ZUBAIR MIAN <mian_zubair@yahoo.com>
> wrote:
>
> > Dear all
> >
> > I am facing problem wrt the datanode(at slave). In the log file of slave
> i
> > am getting following message:
> >            "Call to master/192.168.0.1:54310 failed on local exception:
> > java.net.NoRouteToHostException:            No route to host at
> > org.apache.hadoop.ipc.Client.wrapException"
> > I am currently working on two node cluster. The network is working fine.
> > Both machines can ping each other.
> >
> > And Following is the log file of datanode.
> >
> >
> >  STARTUP_MSG: Starting DataNode
> > STARTUP_MSG:  host = slave/192.168.0.2
> > STARTUP_MSG:  args = []
> > STARTUP_MSG:  version = 0.20.1
> > STARTUP_MSG:  build =
> > http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc1 -r
> > 810220; compiled by 'oom' on Tue Sep  1 20:55:56 UTC 2009
> > ************************************************************/
> > 2010-01-15 18:50:56,435 INFO org.apache.hadoop.ipc.Client: Retrying
> connect
> > to server: master/192.168.0.1:54310. Already tried 0 time(s).
> > 2010-01-15 18:50:57,436 INFO org.apache.hadoop.ipc.Client: Retrying
> connect
> > to server: master/192.168.0.1:54310. Already tried 1 time(s).
> > 2010-01-15 18:50:58,437 INFO org.apache.hadoop.ipc.Client: Retrying
> connect
> > to server: master/192.168.0.1:54310. Already tried 2 time(s).
> > 2010-01-15 18:50:59,438 INFO org.apache.hadoop.ipc.Client: Retrying
> connect
> > to server: master/192.168.0.1:54310. Already tried 3 time(s).
> > 2010-01-15 18:51:00,439 INFO org.apache.hadoop.ipc.Client: Retrying
> connect
> > to server: master/192.168.0.1:54310. Already tried 4 time(s).
> > 2010-01-15 18:51:01,440 INFO org.apache.hadoop.ipc.Client: Retrying
> connect
> > to server: master/192.168.0.1:54310. Already tried 5 time(s).
> > 2010-01-15 18:51:02,441 INFO org.apache.hadoop.ipc.Client: Retrying
> connect
> > to server: master/192.168.0.1:54310. Already tried 6 time(s).
> > 2010-01-15 18:51:03,443 INFO org.apache.hadoop.ipc.Client: Retrying
> connect
> > to server: master/192.168.0.1:54310. Already tried 7 time(s).
> > 2010-01-15 18:51:04,444 INFO org.apache.hadoop.ipc.Client: Retrying
> connect
> > to server: master/192.168.0.1:54310. Already tried 8 time(s).
> > 2010-01-15 18:51:05,445 INFO org.apache.hadoop.ipc.Client: Retrying
> connect
> > to server: master/192.168.0.1:54310. Already tried 9 time(s).
> > 2010-01-15 18:51:05,449 ERROR
> > org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
> Call
> > to master/192.168.0.1:54310 failed on local exception:
> > java.net.NoRouteToHostException: No route to host
> >  at org.apache.hadoop.ipc.Client.wrapException(Client.java:774)
> >  at org.apache.hadoop.ipc.Client.call(Client.java:742)
> >  at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> >  at $Proxy4.getProtocolVersion(Unknown Source)
> >  at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
> >  at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:346)
> >  at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:383)
> >  at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:314)
> >  at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:291)
> >  at
> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:269)
> >  at
> > org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:216)
> >  at
> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1283)
> >  at
> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1238)
> >  at
> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1246)
> >  at
> > org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1368)
> > Caused by: java.net.NoRouteToHostException: No route to host
> >  at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >  at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> >  at
> >
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> >  at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
> >  at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:304)
> >  at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
> >  at org.apache.hadoop.ipc.Client.getConnection(Client.java:859)
> >  at org.apache.hadoop.ipc.Client.call(Client.java:719)
> >  ... 13 more
> > 2010-01-15 18:51:05,451 INFO
> > org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
> > /************************************************************
> > SHUTDOWN_MSG: Shutting down DataNode at slave/192.168.0.2
> > ************************************************************/
> >
> > Please help me asap.
> >
> > regards
> > maria
> >
> >
> >
> >
>
>
>
>
>

--000e0cd1383ebf729a047d94af10--

From general-return-947-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 20 09:06:39 2010
Return-Path: <general-return-947-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85028 invoked from network); 20 Jan 2010 09:06:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jan 2010 09:06:38 -0000
Received: (qmail 29618 invoked by uid 500); 20 Jan 2010 09:06:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29540 invoked by uid 500); 20 Jan 2010 09:06:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29530 invoked by uid 99); 20 Jan 2010 09:06:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Jan 2010 09:06:37 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=HTML_IMAGE_ONLY_24,HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of naveenkumarp@huawei.com designates 119.145.14.66 as permitted sender)
Received: from [119.145.14.66] (HELO szxga03-in.huawei.com) (119.145.14.66)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Jan 2010 09:06:27 +0000
Received: from huawei.com (szxga03-in [172.24.2.9])
 by szxga03-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug
 8 2006)) with ESMTP id <0KWJ009QNFWG2M@szxga03-in.huawei.com> for
 general@hadoop.apache.org; Wed, 20 Jan 2010 17:05:04 +0800 (CST)
Received: from huawei.com ([172.24.2.119])
 by szxga03-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug
 8 2006)) with ESMTP id <0KWJ00LKMFWFAF@szxga03-in.huawei.com> for
 general@hadoop.apache.org; Wed, 20 Jan 2010 17:05:03 +0800 (CST)
Received: from BLRNSHTIPL6NC ([10.18.1.36])
 by szxml06-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug
 8 2006)) with ESMTPA id <0KWJ004Z4FWFLH@szxml06-in.huawei.com> for
 general@hadoop.apache.org; Wed, 20 Jan 2010 17:05:03 +0800 (CST)
Date: Wed, 20 Jan 2010 14:35:02 +0530
From: Naveen Kumar Prasad <naveenkumarp@huawei.com>
Subject: debug MapReduce task running on a cluster
To: general@hadoop.apache.org
Reply-to: naveenkumarp@huawei.com
Message-id: <002901ca99af$a73cb2e0$2401120a@china.huawei.com>
Organization: Htipl
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.3790.3168
X-Mailer: Microsoft Office Outlook 11
Content-type: multipart/alternative;
 boundary="Boundary_(ID_fuIU1XZiO0UTndWMWvaoPw)"
Thread-index: AcqZr6ZeddEW+lU+QFO7l6MAv9phhA==
X-Virus-Checked: Checked by ClamAV on apache.org

--Boundary_(ID_fuIU1XZiO0UTndWMWvaoPw)
Content-type: text/plain; charset=US-ASCII
Content-transfer-encoding: 7BIT

Hi All,
 
I am new to hadoop/Mapreduce usage.
I was trying to find ways to debug a MapReduce task running on a cluster.
However, I was not able to get something concrete and precise.
 
It would be great if anyone can help me on the same.
 
Regards,
Naveen Kumar
HUAWEI TECHNOLOGIES CO.,LTD.  huawei_logo
<outbind://24-000000004ED456449CB8724C859F40771D413A7584132000/cid:919325906
@21052009-015A> 


Address: Huawei Industrial Base
Bantian Longgang
Shenzhen 518129, P.R.China
www.huawei.com
----------------------------------------------------------------------------
-------------------------------------
This e-mail and its attachments contain confidential information from
HUAWEI, which is intended only for the person or entity whose address is
listed above. Any use of the information contained herein in any way
(including, but not limited to, total or partial disclosure, reproduction,
or dissemination) by persons other than the intended recipient(s) is
prohibited. If you receive this e-mail in error, please notify the sender by
phone or email immediately and delete it!
 

--Boundary_(ID_fuIU1XZiO0UTndWMWvaoPw)--

From general-return-948-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 20 14:36:20 2010
Return-Path: <general-return-948-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16549 invoked from network); 20 Jan 2010 14:36:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jan 2010 14:36:19 -0000
Received: (qmail 42268 invoked by uid 500); 20 Jan 2010 14:36:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42164 invoked by uid 500); 20 Jan 2010 14:36:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42154 invoked by uid 99); 20 Jan 2010 14:36:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Jan 2010 14:36:18 +0000
X-ASF-Spam-Status: No, hits=4.2 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Jan 2010 14:36:06 +0000
Received: from EGL-EX07CAS02.ds.corp.yahoo.com (egl-ex07cas02.eglbp.corp.yahoo.com [203.83.248.209])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o0KEYdai040886;
	Wed, 20 Jan 2010 06:34:39 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:in-reply-to:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=jHsIXx8Aw8XVHboDtIbOBVVYmQKmuOPcCTGu9NapSsktAZw02JsnHzl/P4L9Do+M
Received: from EGL-EX07VS01.ds.corp.yahoo.com ([203.83.248.205]) by
 EGL-EX07CAS02.ds.corp.yahoo.com ([203.83.248.216]) with mapi; Wed, 20 Jan
 2010 20:04:38 +0530
From: Amareshwari Sri Ramadasu <amarsri@YAHOO-INC.COM>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>,
        "naveenkumarp@huawei.com" <naveenkumarp@huawei.com>
Date: Wed, 20 Jan 2010 20:04:35 +0530
Subject: Re: debug MapReduce task running on a cluster
Thread-Topic: debug MapReduce task running on a cluster
Thread-Index: AcqZr6ZeddEW+lU+QFO7l6MAv9phhAALglzb
Message-ID: <C77D1253.FC17%amarsri@yahoo-inc.com>
In-Reply-To: <002901ca99af$a73cb2e0$2401120a@china.huawei.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C77D1253FC17amarsriyahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C77D1253FC17amarsriyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


See http://wiki.apache.org/hadoop/HowToDebugMapReducePrograms
http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html#Isolation=
Runner
and
http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html#Debugging

Thanks
Amareshwari

On 1/20/10 2:35 PM, "Naveen Kumar Prasad" <naveenkumarp@huawei.com> wrote:

Hi All,

I am new to hadoop/Mapreduce usage.
I was trying to find ways to debug a MapReduce task running on a cluster.
However, I was not able to get something concrete and precise.

It would be great if anyone can help me on the same.

Regards,
Naveen Kumar
HUAWEI TECHNOLOGIES CO.,LTD.  huawei_logo
<outbind://24-000000004ED456449CB8724C859F40771D413A7584132000/cid:91932590=
6
@21052009-015A>


Address: Huawei Industrial Base
Bantian Longgang
Shenzhen 518129, P.R.China
www.huawei.com
---------------------------------------------------------------------------=
-
-------------------------------------
This e-mail and its attachments contain confidential information from
HUAWEI, which is intended only for the person or entity whose address is
listed above. Any use of the information contained herein in any way
(including, but not limited to, total or partial disclosure, reproduction,
or dissemination) by persons other than the intended recipient(s) is
prohibited. If you receive this e-mail in error, please notify the sender b=
y
phone or email immediately and delete it!



--_000_C77D1253FC17amarsriyahooinccom_--

From general-return-949-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 20 17:52:51 2010
Return-Path: <general-return-949-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96962 invoked from network); 20 Jan 2010 17:52:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jan 2010 17:52:51 -0000
Received: (qmail 24398 invoked by uid 500); 20 Jan 2010 17:52:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24356 invoked by uid 500); 20 Jan 2010 17:52:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24346 invoked by uid 99); 20 Jan 2010 17:52:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Jan 2010 17:52:50 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [66.163.179.140] (HELO web35601.mail.mud.yahoo.com) (66.163.179.140)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 20 Jan 2010 17:52:44 +0000
Received: (qmail 28953 invoked by uid 60001); 20 Jan 2010 17:52:23 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1264009942; bh=fmnUpNAfCeKdtclWPfQPYm6VgQJ58gRpjYpCE9N9gaM=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=k0JXp1HHUCZsGJsmI23wIrLEGm4i298loXja06YDnqw0qnlH0xw/tuVpN2/+FZBsoX2iFJl+q0dhr9eKtlQepc/+QGYxxDepaNVJzi4w+bwVlXEmM8h4Gj2CUM/+CSuo9DsSXnvfaDfcN7Egdg/CP4dXaN4AnNK4WOD6c1ppbiM=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=3vWePvizn+zmF9PaBOR0HRMN9laJ+94m3iU10PilhaH58CRihtbae7hzEnHL80dA5Mg70nf2BCgqhbQnm3MhTv0vIHckywCdHNsMVZY3rbEydAsKUVq5Kf3oCiDH/ROKZBWHpCjpbO3D0anGoh7uVY4F+oiZq3TGeSkNd4403sI=;
Message-ID: <987434.28638.qm@web35601.mail.mud.yahoo.com>
X-YMail-OSG: lpozDKYVM1n9fsdYMW8iQLPlsxlSyTyUU9zcKIRV7HY2NQ12QvexlBivazFPHKDl4Avbz9wPbCWWV19jQA3KtalC9MS_z7rzYIt3PPEf1p8TdNXRAhBAD.Ge4UX5GhHtcxL2ByL3QUsNpTbmv3aunL45JaHMko2t59ef3OoFqZtAccprQBEJKbOHlah4oseWOhkhDRulBilC1H1hoM8H80u2tgCBV8Tdr2FSYIfPLo_pwX__CPXxrBGMyupX.bjWmbCbl4OuicJ5YXPrE4h4RyBxmqJ4poLMN70Fm5pVNDiHAGhhM3WNoGnKPbywVGrmbY6eS5PRFnponRj.xZUp1Sujr0NrX.79_3vsJoEMnkkUYGRFVBMLhzlsPsvuiWlo2v12ChQmRCWXLPbIX9KQgU.EOTXB3a1Sk4LdwoOtIhYwAj7jyCMvVRKiL9d0TDoZYAWAICZHqMdatWLRTlXajN.uwS1gd.ww75ka6ljznWDYM.otuRUoVFEBMfWx_8viGaXqbyluDauNJnzs3BmCcXSM79KQAwUzZWkLZ4ZnRTtWlLzuYIv.1HRlD.n1eymCQ4GRqsKm172m0wetomkdXZ9JMQdr_Ax.wjIGp8aFWpv.rO0KgHYHhTJ0MmQ8jruP2S0ushiAqmXhfH90DnF9NzI_9jsQ5hR9Eu620iusaKeKvwsxBDJkYY8CpYtOH6TF8xX7hSlXtTm4gEo6y.QDmaud3m3sxTaf5E77qgT9lOFP6qSGw4x0mzboluTqf.v4KwsAfJ9VlA5UDYhyvRCn_C3.6GNLFOfWY_0VlVC77RudA2SrLm7tvCEyj0fCR7lHZ0NR7s.RpMoFSChXV4e9XrvZaBv8mbarK.LIpUdaX3PkAuPg7sP3JWmeBpgEzBOZAeIJkVdYuRybIXs3Rh6.otJ66FIQ224k1rDdHXEUXbfSF3tRpISwn9n67LY45tWh0t1vSPk9_72ObOxrC
 LTmoXGYk3JqG0MVQ7A-
Received: from [115.186.6.43] by web35601.mail.mud.yahoo.com via HTTP; Wed, 20 Jan 2010 09:52:22 PST
X-Mailer: YahooMailRC/272.7 YahooMailWebService/0.8.100.260964
References: <573443.25346.qm@web35606.mail.mud.yahoo.com>  <5a921af21001190509hec85921u5d8be7f84b9cbe@mail.gmail.com>  <364995.16459.qm@web35602.mail.mud.yahoo.com> <5a921af21001200051p509b27f0jce1a11663f795154@mail.gmail.com>
Date: Wed, 20 Jan 2010 09:52:22 -0800 (PST)
From: ZUBAIR MIAN <mian_zubair@yahoo.com>
Subject: Re: Problem with datanode(slave machine)
To: general@hadoop.apache.org
In-Reply-To: <5a921af21001200051p509b27f0jce1a11663f795154@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-28795546-1264009942=:28638"

--0-28795546-1264009942=:28638
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Still giving me the same error=0A=0A=0A=0A=0A______________________________=
__=0AFrom: xiao yang <yangxiao9901@gmail.com>=0ATo: general@hadoop.apache.o=
rg=0ASent: Wed, January 20, 2010 1:51:04 PM=0ASubject: Re: Problem with dat=
anode(slave machine)=0A=0AMaybe the firewall blocked this port.=0ATry:=0A$t=
elnet 192.168.0.1 54310=0A=0AOn Tue, Jan 19, 2010 at 10:17 PM, ZUBAIR MIAN =
<mian_zubair@yahoo.com> wrote:=0A=0A> Have already done that.=0A>=0A>=0A>=
=0A>=0A> ________________________________=0A> From: xiao yang <yangxiao9901=
@gmail.com>=0A> To: general@hadoop.apache.org=0A> Sent: Tue, January 19, 20=
10 6:09:57 PM=0A> Subject: Re: Problem with datanode(slave machine)=0A>=0A>=
 Add hosts and ip address in /etc/hosts.=0A> It may work.=0A>=0A> On Tue, J=
an 19, 2010 at 8:34 PM, ZUBAIR MIAN <mian_zubair@yahoo.com>=0A> wrote:=0A>=
=0A> > Dear all=0A> >=0A> > I am facing problem wrt the datanode(at slave).=
 In the log file of slave=0A> i=0A> > am getting following message:=0A> >=
=A0 =A0 =A0 =A0 =A0 =A0 "Call to master/192.168.0.1:54310 failed on local e=
xception:=0A> > java.net.NoRouteToHostException:=A0 =A0 =A0 =A0 =A0 =A0 No =
route to host at=0A> > org.apache.hadoop.ipc.Client.wrapException"=0A> > I =
am currently working on two node cluster. The network is working fine.=0A> =
> Both machines can ping each other.=0A> >=0A> > And Following is the log f=
ile of datanode.=0A> >=0A> >=0A> >=A0 STARTUP_MSG: Starting DataNode=0A> > =
STARTUP_MSG:=A0 host =3D slave/192.168.0.2=0A> > STARTUP_MSG:=A0 args =3D [=
]=0A> > STARTUP_MSG:=A0 version =3D 0.20.1=0A> > STARTUP_MSG:=A0 build =3D=
=0A> > http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc=
1 -r=0A> > 810220; compiled by 'oom' on Tue Sep=A0 1 20:55:56 UTC 2009=0A> =
> ************************************************************/=0A> > 2010-=
01-15 18:50:56,435 INFO org.apache.hadoop.ipc.Client: Retrying=0A> connect=
=0A> > to server: master/192.168.0.1:54310. Already tried 0 time(s).=0A> > =
2010-01-15 18:50:57,436 INFO org.apache.hadoop.ipc.Client: Retrying=0A> con=
nect=0A> > to server: master/192.168.0.1:54310. Already tried 1 time(s).=0A=
> > 2010-01-15 18:50:58,437 INFO org.apache.hadoop.ipc.Client: Retrying=0A>=
 connect=0A> > to server: master/192.168.0.1:54310. Already tried 2 time(s)=
.=0A> > 2010-01-15 18:50:59,438 INFO org.apache.hadoop.ipc.Client: Retrying=
=0A> connect=0A> > to server: master/192.168.0.1:54310. Already tried 3 tim=
e(s).=0A> > 2010-01-15 18:51:00,439 INFO org.apache.hadoop.ipc.Client: Retr=
ying=0A> connect=0A> > to server: master/192.168.0.1:54310. Already tried 4=
 time(s).=0A> > 2010-01-15 18:51:01,440 INFO org.apache.hadoop.ipc.Client: =
Retrying=0A> connect=0A> > to server: master/192.168.0.1:54310. Already tri=
ed 5 time(s).=0A> > 2010-01-15 18:51:02,441 INFO org.apache.hadoop.ipc.Clie=
nt: Retrying=0A> connect=0A> > to server: master/192.168.0.1:54310. Already=
 tried 6 time(s).=0A> > 2010-01-15 18:51:03,443 INFO org.apache.hadoop.ipc.=
Client: Retrying=0A> connect=0A> > to server: master/192.168.0.1:54310. Alr=
eady tried 7 time(s).=0A> > 2010-01-15 18:51:04,444 INFO org.apache.hadoop.=
ipc.Client: Retrying=0A> connect=0A> > to server: master/192.168.0.1:54310.=
 Already tried 8 time(s).=0A> > 2010-01-15 18:51:05,445 INFO org.apache.had=
oop.ipc.Client: Retrying=0A> connect=0A> > to server: master/192.168.0.1:54=
310. Already tried 9 time(s).=0A> > 2010-01-15 18:51:05,449 ERROR=0A> > org=
.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:=0A> Call=
=0A> > to master/192.168.0.1:54310 failed on local exception:=0A> > java.ne=
t.NoRouteToHostException: No route to host=0A> >=A0 at org.apache.hadoop.ip=
c.Client.wrapException(Client.java:774)=0A> >=A0 at org.apache.hadoop.ipc.C=
lient.call(Client.java:742)=0A> >=A0 at org.apache.hadoop.ipc.RPC$Invoker.i=
nvoke(RPC.java:220)=0A> >=A0 at $Proxy4.getProtocolVersion(Unknown Source)=
=0A> >=A0 at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)=0A> >=A0 at o=
rg.apache.hadoop.ipc.RPC.getProxy(RPC.java:346)=0A> >=A0 at org.apache.hado=
op.ipc.RPC.getProxy(RPC.java:383)=0A> >=A0 at org.apache.hadoop.ipc.RPC.wai=
tForProxy(RPC.java:314)=0A> >=A0 at org.apache.hadoop.ipc.RPC.waitForProxy(=
RPC.java:291)=0A> >=A0 at=0A> >=0A> org.apache.hadoop.hdfs.server.datanode.=
DataNode.startDataNode(DataNode.java:269)=0A> >=A0 at=0A> > org.apache.hado=
op.hdfs.server.datanode.DataNode.<init>(DataNode.java:216)=0A> >=A0 at=0A> =
>=0A> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode=
.java:1283)=0A> >=A0 at=0A> >=0A> org.apache.hadoop.hdfs.server.datanode.Da=
taNode.instantiateDataNode(DataNode.java:1238)=0A> >=A0 at=0A> >=0A> org.ap=
ache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1246=
)=0A> >=A0 at=0A> > org.apache.hadoop.hdfs.server.datanode.DataNode.main(Da=
taNode.java:1368)=0A> > Caused by: java.net.NoRouteToHostException: No rout=
e to host=0A> >=A0 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Meth=
od)=0A> >=A0 at=0A> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChanne=
lImpl.java:592)=0A> >=A0 at=0A> >=0A> org.apache.hadoop.net.SocketIOWithTim=
eout.connect(SocketIOWithTimeout.java:206)=0A> >=A0 at org.apache.hadoop.ne=
t.NetUtils.connect(NetUtils.java:404)=0A> >=A0 at=0A> org.apache.hadoop.ipc=
.Client$Connection.setupIOstreams(Client.java:304)=0A> >=A0 at org.apache.h=
adoop.ipc.Client$Connection.access$1700(Client.java:176)=0A> >=A0 at org.ap=
ache.hadoop.ipc.Client.getConnection(Client.java:859)=0A> >=A0 at org.apach=
e.hadoop.ipc.Client.call(Client.java:719)=0A> >=A0 ... 13 more=0A> > 2010-0=
1-15 18:51:05,451 INFO=0A> > org.apache.hadoop.hdfs.server.datanode.DataNod=
e: SHUTDOWN_MSG:=0A> > /***************************************************=
*********=0A> > SHUTDOWN_MSG: Shutting down DataNode at slave/192.168.0.2=
=0A> > ************************************************************/=0A> >=
=0A> > Please help me asap.=0A> >=0A> > regards=0A> > maria=0A> >=0A> >=0A>=
 >=0A> >=0A>=0A>=0A>=0A>=0A>=0A=0A=0A=0A      
--0-28795546-1264009942=:28638--

From general-return-950-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 20 17:54:11 2010
Return-Path: <general-return-950-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98541 invoked from network); 20 Jan 2010 17:54:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jan 2010 17:54:11 -0000
Received: (qmail 29253 invoked by uid 500); 20 Jan 2010 17:54:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29176 invoked by uid 500); 20 Jan 2010 17:54:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29166 invoked by uid 99); 20 Jan 2010 17:54:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Jan 2010 17:54:10 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [66.163.179.140] (HELO web35601.mail.mud.yahoo.com) (66.163.179.140)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 20 Jan 2010 17:54:03 +0000
Received: (qmail 29723 invoked by uid 60001); 20 Jan 2010 17:53:40 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1264010020; bh=+Ib424HlhcFFDM5QCCIQJnVR2hAzqYqIqcm5pmGeFRM=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=vzjHxOzOAQDchpR3QFKWVlLYe/Ym6sqwRJVtFCC5bglVdlwenpFuVA0D16qQQ80WjnKRersaSL53bSVGhry5XUFcbeJvR187Bif1P9Vqu3mqE6WNMeaPCjERpdTLFCMja7MNx2HjyuRyfDsmAE5t0mcY2mcIFwH5NBsbPYbB720=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=f6LsXsZbhHmYD/8bMixl4g5Dp8AW0ZHuoto5piuhq10eH6hA2IyffQUMxfXueDvaHsfvIpnx8PsO1r/ezvDZIrr7xtoz/HpkJpmFX+Ehg7qlqxr+Vy1gmw4I3zPaCB0n9XTvf51JNshB0DzHr5aYvcQRW+hfgoBVRPu0lExEZqU=;
Message-ID: <586024.28687.qm@web35601.mail.mud.yahoo.com>
X-YMail-OSG: Z_XcLREVM1lEQfjL1JXEPg08A2K4YYduabmLZdlHCVwjm8qWUsAb9Oly6a95lI8VV1dRY3muY3h3Dy4Ao2C8VYD3.nRVPjPO_lKIIs1YtMgoAjng45PFj5n5LHbnRlRLnA5s4ugc9u85We3P1xyZ9h30hjFC1u_2mJ9Z4IeS.Pj3SsI.h_t9w1bpWILcfy2TE6C5254aM1ZrSbuyml02FcbEEXMlAxqOInvBQBhvAOqX3LqlyjxEFMYt8yrTm34G8iezSQEga.aLT0P8eau2Nu1jLPMCbtDjVdpMfXHcjRMA4eSrFRBgUC7IIHLnZMzgKLrxUHaVVt.qH7cPiIOAZ3Ocp9qwgY8mwB6JIrY0tLRZHMUC91GSTEhacz3iqhfo8hfxZ37dzsYZhzWBsDasMxojmE3oBuE8RVS8NuotQP4lkoZgMCPlxW1Tb6hWMMZIxUUMGpzYPzRVfyz1sbOtLaYXos9moNPGkiJGdjSRYDDTofSL2mxyZQgRUfHH5SK80fjH9IBDc6RFz6K76v2XYC.LERj4epmvTuDQo_xa8I9M_c0fwGCTQ7fDVQY1uYdIhpKh0WxHfEOTi5ayb0mP4aiuzuVuLBVe.6qaXroaSy805Cb8k9aHhF3DN5FaxuMd8CglZbOSTsZA7_IQ94J_VBCzRKm.TJGnaTuxfUavs94PVptxHizdX.WwA2xEvU3uR9J2Ijg3lQd_2eJ1l3PRCI9v7I4UHz5hIgTGR.0BGGqzndnL07NQ0NIcE_FsCIEpkOa6aA7XS05ElG8ihaYxn53SlL3zT2GLVsxufHRSbBAw4vu73DjOUEN0aF0c1MY3KjdHUhJKHLcLbxzw.EAEaiq3g3muA7m.Jw_Rx0AOIn1QVAr7WpV_hiq7tOVfvLtLEtcehLIiWHMJqPN.hqGUMChWyA8lmqO8B6lyzUjl830LWi3OovVaa8T_KgDDX1FZ407WthIC
Received: from [115.186.6.43] by web35601.mail.mud.yahoo.com via HTTP; Wed, 20 Jan 2010 09:53:40 PST
X-Mailer: YahooMailRC/272.7 YahooMailWebService/0.8.100.260964
References: <573443.25346.qm@web35606.mail.mud.yahoo.com>  <5a921af21001190509hec85921u5d8be7f84b9cbe@mail.gmail.com>  <364995.16459.qm@web35602.mail.mud.yahoo.com> <5a921af21001200051p509b27f0jce1a11663f795154@mail.gmail.com>
Date: Wed, 20 Jan 2010 09:53:40 -0800 (PST)
From: ZUBAIR MIAN <mian_zubair@yahoo.com>
Subject: Re: Problem with datanode(slave machine)
To: general@hadoop.apache.org
In-Reply-To: <5a921af21001200051p509b27f0jce1a11663f795154@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-993381238-1264010020=:28687"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-993381238-1264010020=:28687
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Still giving me the same error=0A=0A=0A=0A=0A______________________________=
__=0AFrom: xiao yang <yangxiao9901@gmail.com>=0ATo: general@hadoop.apache.o=
rg=0ASent: Wed, January 20, 2010 1:51:04 PM=0ASubject: Re: Problem with dat=
anode(slave machine)=0A=0AMaybe the firewall blocked this port.=0ATry:=0A$t=
elnet 192.168.0.1 54310=0A=0AOn Tue, Jan 19, 2010 at 10:17 PM, ZUBAIR MIAN =
<mian_zubair@yahoo.com> wrote:=0A=0A> Have already done that.=0A>=0A>=0A>=
=0A>=0A> ________________________________=0A> From: xiao yang <yangxiao9901=
@gmail.com>=0A> To: general@hadoop.apache.org=0A> Sent: Tue, January 19, 20=
10 6:09:57 PM=0A> Subject: Re: Problem with datanode(slave machine)=0A>=0A>=
 Add hosts and ip address in /etc/hosts.=0A> It may work.=0A>=0A> On Tue, J=
an 19, 2010 at 8:34 PM, ZUBAIR MIAN <mian_zubair@yahoo.com>=0A> wrote:=0A>=
=0A> > Dear all=0A> >=0A> > I am facing problem wrt the datanode(at slave).=
 In the log file of slave=0A> i=0A> > am getting following message:=0A> >=
=A0 =A0 =A0 =A0 =A0 =A0 "Call to master/192.168.0.1:54310 failed on local e=
xception:=0A> > java.net.NoRouteToHostException:=A0 =A0 =A0 =A0 =A0 =A0 No =
route to host at=0A> > org.apache.hadoop.ipc.Client.wrapException"=0A> > I =
am currently working on two node cluster. The network is working fine.=0A> =
> Both machines can ping each other.=0A> >=0A> > And Following is the log f=
ile of datanode.=0A> >=0A> >=0A> >=A0 STARTUP_MSG: Starting DataNode=0A> > =
STARTUP_MSG:=A0 host =3D slave/192.168.0.2=0A> > STARTUP_MSG:=A0 args =3D [=
]=0A> > STARTUP_MSG:=A0 version =3D 0.20.1=0A> > STARTUP_MSG:=A0 build =3D=
=0A> > http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc=
1 -r=0A> > 810220; compiled by 'oom' on Tue Sep=A0 1 20:55:56 UTC 2009=0A> =
> ************************************************************/=0A> > 2010-=
01-15 18:50:56,435 INFO org.apache.hadoop.ipc.Client: Retrying=0A> connect=
=0A> > to server: master/192.168.0.1:54310. Already tried 0 time(s).=0A> > =
2010-01-15 18:50:57,436 INFO org.apache.hadoop.ipc.Client: Retrying=0A> con=
nect=0A> > to server: master/192.168.0.1:54310. Already tried 1 time(s).=0A=
> > 2010-01-15 18:50:58,437 INFO org.apache.hadoop.ipc.Client: Retrying=0A>=
 connect=0A> > to server: master/192.168.0.1:54310. Already tried 2 time(s)=
.=0A> > 2010-01-15 18:50:59,438 INFO org.apache.hadoop.ipc.Client: Retrying=
=0A> connect=0A> > to server: master/192.168.0.1:54310. Already tried 3 tim=
e(s).=0A> > 2010-01-15 18:51:00,439 INFO org.apache.hadoop.ipc.Client: Retr=
ying=0A> connect=0A> > to server: master/192.168.0.1:54310. Already tried 4=
 time(s).=0A> > 2010-01-15 18:51:01,440 INFO org.apache.hadoop.ipc.Client: =
Retrying=0A> connect=0A> > to server: master/192.168.0.1:54310. Already tri=
ed 5 time(s).=0A> > 2010-01-15 18:51:02,441 INFO org.apache.hadoop.ipc.Clie=
nt: Retrying=0A> connect=0A> > to server: master/192.168.0.1:54310. Already=
 tried 6 time(s).=0A> > 2010-01-15 18:51:03,443 INFO org.apache.hadoop.ipc.=
Client: Retrying=0A> connect=0A> > to server: master/192.168.0.1:54310. Alr=
eady tried 7 time(s).=0A> > 2010-01-15 18:51:04,444 INFO org.apache.hadoop.=
ipc.Client: Retrying=0A> connect=0A> > to server: master/192.168.0.1:54310.=
 Already tried 8 time(s).=0A> > 2010-01-15 18:51:05,445 INFO org.apache.had=
oop.ipc.Client: Retrying=0A> connect=0A> > to server: master/192.168.0.1:54=
310. Already tried 9 time(s).=0A> > 2010-01-15 18:51:05,449 ERROR=0A> > org=
.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:=0A> Call=
=0A> > to master/192.168.0.1:54310 failed on local exception:=0A> > java.ne=
t.NoRouteToHostException: No route to host=0A> >=A0 at org.apache.hadoop.ip=
c.Client.wrapException(Client.java:774)=0A> >=A0 at org.apache.hadoop.ipc.C=
lient.call(Client.java:742)=0A> >=A0 at org.apache.hadoop.ipc.RPC$Invoker.i=
nvoke(RPC.java:220)=0A> >=A0 at $Proxy4.getProtocolVersion(Unknown Source)=
=0A> >=A0 at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)=0A> >=A0 at o=
rg.apache.hadoop.ipc.RPC.getProxy(RPC.java:346)=0A> >=A0 at org.apache.hado=
op.ipc.RPC.getProxy(RPC.java:383)=0A> >=A0 at org.apache.hadoop.ipc.RPC.wai=
tForProxy(RPC.java:314)=0A> >=A0 at org.apache.hadoop.ipc.RPC.waitForProxy(=
RPC.java:291)=0A> >=A0 at=0A> >=0A> org.apache.hadoop.hdfs.server.datanode.=
DataNode.startDataNode(DataNode.java:269)=0A> >=A0 at=0A> > org.apache.hado=
op.hdfs.server.datanode.DataNode.<init>(DataNode.java:216)=0A> >=A0 at=0A> =
>=0A> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode=
.java:1283)=0A> >=A0 at=0A> >=0A> org.apache.hadoop.hdfs.server.datanode.Da=
taNode.instantiateDataNode(DataNode.java:1238)=0A> >=A0 at=0A> >=0A> org.ap=
ache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1246=
)=0A> >=A0 at=0A> > org.apache.hadoop.hdfs.server.datanode.DataNode.main(Da=
taNode.java:1368)=0A> > Caused by: java.net.NoRouteToHostException: No rout=
e to host=0A> >=A0 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Meth=
od)=0A> >=A0 at=0A> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChanne=
lImpl.java:592)=0A> >=A0 at=0A> >=0A> org.apache.hadoop.net.SocketIOWithTim=
eout.connect(SocketIOWithTimeout.java:206)=0A> >=A0 at org.apache.hadoop.ne=
t.NetUtils.connect(NetUtils.java:404)=0A> >=A0 at=0A> org.apache.hadoop.ipc=
.Client$Connection.setupIOstreams(Client.java:304)=0A> >=A0 at org.apache.h=
adoop.ipc.Client$Connection.access$1700(Client.java:176)=0A> >=A0 at org.ap=
ache.hadoop.ipc.Client.getConnection(Client.java:859)=0A> >=A0 at org.apach=
e.hadoop.ipc.Client.call(Client.java:719)=0A> >=A0 ... 13 more=0A> > 2010-0=
1-15 18:51:05,451 INFO=0A> > org.apache.hadoop.hdfs.server.datanode.DataNod=
e: SHUTDOWN_MSG:=0A> > /***************************************************=
*********=0A> > SHUTDOWN_MSG: Shutting down DataNode at slave/192.168.0.2=
=0A> > ************************************************************/=0A> >=
=0A> > Please help me asap.=0A> >=0A> > regards=0A> > maria=0A> >=0A> >=0A>=
 >=0A> >=0A>=0A>=0A>=0A>=0A>=0A=0A=0A=0A      
--0-993381238-1264010020=:28687--

From general-return-951-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 20 20:07:47 2010
Return-Path: <general-return-951-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70286 invoked from network); 20 Jan 2010 20:07:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jan 2010 20:07:47 -0000
Received: (qmail 66576 invoked by uid 500); 20 Jan 2010 20:07:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66524 invoked by uid 500); 20 Jan 2010 20:07:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66513 invoked by uid 99); 20 Jan 2010 20:07:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Jan 2010 20:07:46 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mailinglists19@gmail.com designates 74.125.78.25 as permitted sender)
Received: from [74.125.78.25] (HELO ey-out-2122.google.com) (74.125.78.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Jan 2010 20:07:35 +0000
Received: by ey-out-2122.google.com with SMTP id 22so1027432eye.23
        for <general@hadoop.apache.org>; Wed, 20 Jan 2010 12:07:15 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=68nCk0QJnIcVQt/J0VixyUDx/kksPfGk777+SI8pUhI=;
        b=Q20RZWQAf9jiMBqwj/Ny6dAp0PBb86TAlR0Wk+yEjcKhMVE9iCKMbsLwGj0UaPD4HM
         zl9LdYjVoM9m6Lw3H4NPL4rQM7NuUARu6MFTs8NRdiujLUw/zxMVh/UEjyMENYPm3RMg
         QRyo0CSBU0iyQwm2Gu1fE8u6libPPlaPKiNoQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=INqql4n5+bO+EKb4zkHZXmRtQj+DvSdN0F5ef6pv4WVyyrMDOSzkaZ0QX1PTdccKXg
         iHjSIWO+ls7S/VgUVBUd0EbFtw21d3kSsvOzlvZOV3+pYtnviNcfrQj/fiQ+lMDv/A7u
         VTWUDfqmK/5Y+FHTw/hFZI727c0UNNVhUb4lE=
MIME-Version: 1.0
Received: by 10.216.86.203 with SMTP id w53mr165157wee.58.1264018035364; Wed, 
	20 Jan 2010 12:07:15 -0800 (PST)
In-Reply-To: <2c44f6691001191243t418989cav28e1d072e59e6e21@mail.gmail.com>
References: <CA7285AEA035E041BCF11DBDA646D000048183A9@MSGMROCLN2WIN.DMN1.FMR.COM>
	 <2c44f6691001191243t418989cav28e1d072e59e6e21@mail.gmail.com>
Date: Wed, 20 Jan 2010 12:07:15 -0800
Message-ID: <1eabbac31001201207s7e2c67d7w99f9598c8868f4df@mail.gmail.com>
Subject: Re: Google Patent on MapReduce
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d7f07afce09a047d9e2114
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d7f07afce09a047d9e2114
Content-Type: text/plain; charset=ISO-8859-1

Here's some more info:

http://arstechnica.com/open-source/news/2010/01/googles-mapreduce-patent-what-does-it-mean-for-hadoop.ars

It will be nice to get some official announcement from Apache Hadoop team
regarding future of Hadoop.

- A very scared Hadoop Fan :(


On Tue, Jan 19, 2010 at 12:43 PM, Peter Sankauskas <peter@motally.com>wrote:

> I have heard that Google generally get patents to protect themselves rather
> than to enforce them. Google don't have any publicly available way to make
> money off of map-reduce (even Google App Engine does not map-reduce
> capability yet) so I would imagine they would be fine with Hadoop. Fingers
> crossed.
>
> Kind regards,
> Peter Sankauskas
>
> Motally, Inc
> Office: +1 (415) 932-6898
>
>
> On Tue, Jan 19, 2010 at 11:55 AM, Travaglini, Joseph <
> Joseph.Travaglini@fmr.com> wrote:
>
> > http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PALL
> > &p=1&u=%2Fnetahtml%2FPTO%2Fsrchnum.htm&r=1&f=G&l=50&s1=7,650,331.PN.&OS=
> > PN/7,650,331&RS=PN/7,650,331<
> http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PALL%0A&p=1&u=%2Fnetahtml%2FPTO%2Fsrchnum.htm&r=1&f=G&l=50&s1=7,650,331.PN.&OS=%0APN/7,650,331&RS=PN/7,650,331
> >
> >
> >
> >
> > The USPTO granted a patent to Google today for MapReduce, after several
> > rejections.  I'm no patent expert; how does this affect Hadoop?
> >
> >
>

--0016e6d7f07afce09a047d9e2114--

From general-return-952-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 20 22:07:19 2010
Return-Path: <general-return-952-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 26105 invoked from network); 20 Jan 2010 22:07:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jan 2010 22:07:19 -0000
Received: (qmail 42453 invoked by uid 500); 20 Jan 2010 22:07:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42369 invoked by uid 500); 20 Jan 2010 22:07:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42359 invoked by uid 99); 20 Jan 2010 22:07:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Jan 2010 22:07:17 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of robertburrelldonkin@gmail.com designates 209.85.218.220 as permitted sender)
Received: from [209.85.218.220] (HELO mail-bw0-f220.google.com) (209.85.218.220)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Jan 2010 22:07:09 +0000
Received: by bwz20 with SMTP id 20so4838711bwz.34
        for <general@hadoop.apache.org>; Wed, 20 Jan 2010 14:06:48 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=IqVwpFTJ2+Y8rgivFJgdrc4KWH0nj6rcGkPDTLSYStQ=;
        b=MaIV+nJiTcqxGXxBkauphZHk6/yh+sAz5QdEt88Z0ZzgkFZryWDfJgPZXzgC0V3pu3
         /9tnfygT8re+u+cySh1tzdW6s5K2AYNmpOS8G6rnKA9jGY+/+RCvqJfXXJb1fSo6lBrW
         iuLXFhyIqR1MILPHe11mW+DpjqfUXk51c4uqA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=CXDe/r+gf+tZ53f2CyzV5V0jKEAqiUEE5j6o3E4KaFmB11jQXboO69wo7y7SnWIhXB
         jEG7WtWdZEQD185FgrO9/yuA4T4vjBQ4yd22vZX/XI3cmsk+v4VgQ3QDed2NGSvnnmI+
         akN6HNBBnJs1nYOi57F5ZlIN/+KD+p6dPKiw8=
MIME-Version: 1.0
Received: by 10.204.36.209 with SMTP id u17mr297940bkd.189.1264025208544; Wed, 
	20 Jan 2010 14:06:48 -0800 (PST)
In-Reply-To: <1eabbac31001201207s7e2c67d7w99f9598c8868f4df@mail.gmail.com>
References: <CA7285AEA035E041BCF11DBDA646D000048183A9@MSGMROCLN2WIN.DMN1.FMR.COM>
	 <2c44f6691001191243t418989cav28e1d072e59e6e21@mail.gmail.com>
	 <1eabbac31001201207s7e2c67d7w99f9598c8868f4df@mail.gmail.com>
Date: Wed, 20 Jan 2010 22:06:48 +0000
Message-ID: <f470f68e1001201406y17a985aeu36fa5b7b23a05ccb@mail.gmail.com>
Subject: Re: Google Patent on MapReduce
From: Robert Burrell Donkin <robertburrelldonkin@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Jan 20, 2010 at 8:07 PM, Something Something
<mailinglists19@gmail.com> wrote:
> Here's some more info:
>
> http://arstechnica.com/open-source/news/2010/01/googles-mapreduce-patent-what-does-it-mean-for-hadoop.ars
>
> It will be nice to get some official announcement from Apache Hadoop team
> regarding future of Hadoop.
>
> - A very scared Hadoop Fan :(

don't worry

every substantial application has a very large probability of patent
infringement

sooner or later, software patents will have to be abolished

- robert

From general-return-953-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 20 22:10:44 2010
Return-Path: <general-return-953-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 28929 invoked from network); 20 Jan 2010 22:10:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jan 2010 22:10:44 -0000
Received: (qmail 52906 invoked by uid 500); 20 Jan 2010 22:10:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52835 invoked by uid 500); 20 Jan 2010 22:10:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52825 invoked by uid 99); 20 Jan 2010 22:10:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Jan 2010 22:10:43 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of robertburrelldonkin@gmail.com designates 209.85.218.220 as permitted sender)
Received: from [209.85.218.220] (HELO mail-bw0-f220.google.com) (209.85.218.220)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Jan 2010 22:10:35 +0000
Received: by bwz20 with SMTP id 20so4841917bwz.34
        for <general@hadoop.apache.org>; Wed, 20 Jan 2010 14:10:14 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=3wFE4TtVrDlbnbtGakseYqaFJdcJrZLwN3eV671uuk0=;
        b=AXDMVUUQ4J8WIunhrJapNKwI6EoOxWj3PsKdPG3n4Poiiu18q9AYRhNLyI1CaaenWU
         AQHzCqs22DDg9FKIipbOt3qj9WL8fRypDdTy80ihv+H+4zpQ6Doy4ADW1AycBZ5AAQJf
         /iabM5b1xak2vKG3z22iH7rytV5pRGFot1qk0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=oWL19PEjA32XfOUa0nv0N7IdnOZjLDppa3NaykPnayWKx/dKbVcoV7ONbmJ1wCRYsU
         KzroR1CyKdtyrp1ttAIu5JAf4BHqB3+jdcJsmUhyjzRNDTHauycqqbCf24mo5eYZXNJn
         NqWOMELy1oZd6ypbqKRaawRyOTKVyy07JfRjU=
MIME-Version: 1.0
Received: by 10.204.24.130 with SMTP id v2mr317718bkb.120.1264025414060; Wed, 
	20 Jan 2010 14:10:14 -0800 (PST)
In-Reply-To: <f470f68e1001201406y17a985aeu36fa5b7b23a05ccb@mail.gmail.com>
References: <CA7285AEA035E041BCF11DBDA646D000048183A9@MSGMROCLN2WIN.DMN1.FMR.COM>
	 <2c44f6691001191243t418989cav28e1d072e59e6e21@mail.gmail.com>
	 <1eabbac31001201207s7e2c67d7w99f9598c8868f4df@mail.gmail.com>
	 <f470f68e1001201406y17a985aeu36fa5b7b23a05ccb@mail.gmail.com>
Date: Wed, 20 Jan 2010 22:10:13 +0000
Message-ID: <f470f68e1001201410y4fd86172j53b11bba884e39e6@mail.gmail.com>
Subject: Re: Google Patent on MapReduce
From: Robert Burrell Donkin <robertburrelldonkin@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Jan 20, 2010 at 10:06 PM, Robert Burrell Donkin
<robertburrelldonkin@gmail.com> wrote:
> On Wed, Jan 20, 2010 at 8:07 PM, Something Something
> <mailinglists19@gmail.com> wrote:
>> Here's some more info:
>>
>> http://arstechnica.com/open-source/news/2010/01/googles-mapreduce-patent-what-does-it-mean-for-hadoop.ars
>>
>> It will be nice to get some official announcement from Apache Hadoop team
>> regarding future of Hadoop.
>>
>> - A very scared Hadoop Fan :(
>
> don't worry
>
> every substantial application has a very large probability of patent
> infringement

basically, no company that uses software can enforce a software patent
against any tech major due to mutually-assured destruction through
software patent armageddon. lots of tech majors use hadoop ergo google
would be MAD to use the patent.

- robert

From general-return-954-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 20 22:13:20 2010
Return-Path: <general-return-954-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30458 invoked from network); 20 Jan 2010 22:13:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jan 2010 22:13:20 -0000
Received: (qmail 56916 invoked by uid 500); 20 Jan 2010 22:13:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56824 invoked by uid 500); 20 Jan 2010 22:13:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56778 invoked by uid 99); 20 Jan 2010 22:13:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Jan 2010 22:13:18 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [145.100.8.32] (HELO planck.ka.sara.nl) (145.100.8.32)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Jan 2010 22:13:08 +0000
Received: from planck.ka.sara.nl ([145.100.8.32]) by planck.ka.sara.nl
 ([145.100.8.32]) with mapi; Wed, 20 Jan 2010 23:12:48 +0100
From: Evert Lammerts <Evert.Lammerts@sara.nl>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 20 Jan 2010 23:11:53 +0100
Subject: RE: Google Patent on MapReduce
Thread-Topic: Google Patent on MapReduce
Thread-Index: AcqaHW612ILTQXq4T5uL4UKdQqCpPgAACPyB
Message-ID: <ADF94D8555C7A246B86A633685E0178AB867BBD1C4@planck.ka.sara.nl>
References: <CA7285AEA035E041BCF11DBDA646D000048183A9@MSGMROCLN2WIN.DMN1.FMR.COM>
	 <2c44f6691001191243t418989cav28e1d072e59e6e21@mail.gmail.com>
	 <1eabbac31001201207s7e2c67d7w99f9598c8868f4df@mail.gmail.com>
	 <f470f68e1001201406y17a985aeu36fa5b7b23a05ccb@mail.gmail.com>,<f470f68e1001201410y4fd86172j53b11bba884e39e6@mail.gmail.com>
In-Reply-To: <f470f68e1001201410y4fd86172j53b11bba884e39e6@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

I tend to second that - nothing to worry about: http://gigaom.com/2010/01/1=
9/why-hadoop-users-shouldnt-fear-googles-new-mapreduce-patent/.

Evert Lammerts
________________________________________
From: Robert Burrell Donkin [robertburrelldonkin@gmail.com]
Sent: Wednesday, January 20, 2010 11:10 PM
To: general@hadoop.apache.org
Subject: Re: Google Patent on MapReduce

On Wed, Jan 20, 2010 at 10:06 PM, Robert Burrell Donkin
<robertburrelldonkin@gmail.com> wrote:
> On Wed, Jan 20, 2010 at 8:07 PM, Something Something
> <mailinglists19@gmail.com> wrote:
>> Here's some more info:
>>
>> http://arstechnica.com/open-source/news/2010/01/googles-mapreduce-patent=
-what-does-it-mean-for-hadoop.ars
>>
>> It will be nice to get some official announcement from Apache Hadoop tea=
m
>> regarding future of Hadoop.
>>
>> - A very scared Hadoop Fan :(
>
> don't worry
>
> every substantial application has a very large probability of patent
> infringement

basically, no company that uses software can enforce a software patent
against any tech major due to mutually-assured destruction through
software patent armageddon. lots of tech majors use hadoop ergo google
would be MAD to use the patent.

- robert

From general-return-955-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 21 04:30:20 2010
Return-Path: <general-return-955-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73016 invoked from network); 21 Jan 2010 04:30:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Jan 2010 04:30:20 -0000
Received: (qmail 93836 invoked by uid 500); 21 Jan 2010 04:30:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93543 invoked by uid 500); 21 Jan 2010 04:30:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93533 invoked by uid 99); 21 Jan 2010 04:30:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Jan 2010 04:30:18 +0000
X-ASF-Spam-Status: No, hits=3.7 required=10.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yangxiao9901@gmail.com designates 209.85.160.50 as permitted sender)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Jan 2010 04:30:11 +0000
Received: by pwi20 with SMTP id 20so3951261pwi.29
        for <general@hadoop.apache.org>; Wed, 20 Jan 2010 20:29:50 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=WGxielrfR26jpzKTnm/YIQnAynOBgNyFSyYva1goM/8=;
        b=QoEgqHRXj4PcMgQniQI585do4blT+VrL7KL7ARy4W7aCBPdP5bctKOdaImUjdM4NuT
         qxtvy49ker7dbnQcBebwzLzfkM95Fd00Mq0pfv2h+GYJB8PAL4m2quYoylA8dM+/f50g
         Z4h9jZfMeiaYRJD7xUSUA2PukppMCu+R/TA8Q=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=u8qmPHX2KwUCsqVi6a1L2GDmvcWCmyMfw3iOtacES2u4y/he258swuedtqSkYPoPrH
         EbUMgIPWPx/YgGnbgApZ7i2OS+SxT5UTwqmkXKf94RWJopQRqjMEcaW/iqfFOU1IlYsa
         7zcg2jmXg/Be/eJ5fZAh0o/p5Pfa40gbR9zMI=
MIME-Version: 1.0
Received: by 10.140.248.8 with SMTP id v8mr644544rvh.228.1264048190554; Wed, 
	20 Jan 2010 20:29:50 -0800 (PST)
In-Reply-To: <586024.28687.qm@web35601.mail.mud.yahoo.com>
References: <573443.25346.qm@web35606.mail.mud.yahoo.com>
	 <5a921af21001190509hec85921u5d8be7f84b9cbe@mail.gmail.com>
	 <364995.16459.qm@web35602.mail.mud.yahoo.com>
	 <5a921af21001200051p509b27f0jce1a11663f795154@mail.gmail.com>
	 <586024.28687.qm@web35601.mail.mud.yahoo.com>
Date: Thu, 21 Jan 2010 12:29:50 +0800
Message-ID: <5a921af21001202029r14b7ae8ew7d1ba0ca4a91c613@mail.gmail.com>
Subject: Re: Problem with datanode(slave machine)
From: xiao yang <yangxiao9901@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd116ba60852c047da5272a

--000e0cd116ba60852c047da5272a
Content-Type: text/plain; charset=ISO-8859-1

What's the error message of telnet?

On Thu, Jan 21, 2010 at 1:53 AM, ZUBAIR MIAN <mian_zubair@yahoo.com> wrote:

> Still giving me the same error
>
>
>
>
> ________________________________
> From: xiao yang <yangxiao9901@gmail.com>
> To: general@hadoop.apache.org
> Sent: Wed, January 20, 2010 1:51:04 PM
> Subject: Re: Problem with datanode(slave machine)
>
> Maybe the firewall blocked this port.
> Try:
> $telnet 192.168.0.1 54310
>
> On Tue, Jan 19, 2010 at 10:17 PM, ZUBAIR MIAN <mian_zubair@yahoo.com>
> wrote:
>
> > Have already done that.
> >
> >
> >
> >
> > ________________________________
> > From: xiao yang <yangxiao9901@gmail.com>
> > To: general@hadoop.apache.org
> > Sent: Tue, January 19, 2010 6:09:57 PM
> > Subject: Re: Problem with datanode(slave machine)
> >
> > Add hosts and ip address in /etc/hosts.
> > It may work.
> >
> > On Tue, Jan 19, 2010 at 8:34 PM, ZUBAIR MIAN <mian_zubair@yahoo.com>
> > wrote:
> >
> > > Dear all
> > >
> > > I am facing problem wrt the datanode(at slave). In the log file of
> slave
> > i
> > > am getting following message:
> > >            "Call to master/192.168.0.1:54310 failed on local
> exception:
> > > java.net.NoRouteToHostException:            No route to host at
> > > org.apache.hadoop.ipc.Client.wrapException"
> > > I am currently working on two node cluster. The network is working
> fine.
> > > Both machines can ping each other.
> > >
> > > And Following is the log file of datanode.
> > >
> > >
> > >  STARTUP_MSG: Starting DataNode
> > > STARTUP_MSG:  host = slave/192.168.0.2
> > > STARTUP_MSG:  args = []
> > > STARTUP_MSG:  version = 0.20.1
> > > STARTUP_MSG:  build =
> > > http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc1-r
> > > 810220; compiled by 'oom' on Tue Sep  1 20:55:56 UTC 2009
> > > ************************************************************/
> > > 2010-01-15 18:50:56,435 INFO org.apache.hadoop.ipc.Client: Retrying
> > connect
> > > to server: master/192.168.0.1:54310. Already tried 0 time(s).
> > > 2010-01-15 18:50:57,436 INFO org.apache.hadoop.ipc.Client: Retrying
> > connect
> > > to server: master/192.168.0.1:54310. Already tried 1 time(s).
> > > 2010-01-15 18:50:58,437 INFO org.apache.hadoop.ipc.Client: Retrying
> > connect
> > > to server: master/192.168.0.1:54310. Already tried 2 time(s).
> > > 2010-01-15 18:50:59,438 INFO org.apache.hadoop.ipc.Client: Retrying
> > connect
> > > to server: master/192.168.0.1:54310. Already tried 3 time(s).
> > > 2010-01-15 18:51:00,439 INFO org.apache.hadoop.ipc.Client: Retrying
> > connect
> > > to server: master/192.168.0.1:54310. Already tried 4 time(s).
> > > 2010-01-15 18:51:01,440 INFO org.apache.hadoop.ipc.Client: Retrying
> > connect
> > > to server: master/192.168.0.1:54310. Already tried 5 time(s).
> > > 2010-01-15 18:51:02,441 INFO org.apache.hadoop.ipc.Client: Retrying
> > connect
> > > to server: master/192.168.0.1:54310. Already tried 6 time(s).
> > > 2010-01-15 18:51:03,443 INFO org.apache.hadoop.ipc.Client: Retrying
> > connect
> > > to server: master/192.168.0.1:54310. Already tried 7 time(s).
> > > 2010-01-15 18:51:04,444 INFO org.apache.hadoop.ipc.Client: Retrying
> > connect
> > > to server: master/192.168.0.1:54310. Already tried 8 time(s).
> > > 2010-01-15 18:51:05,445 INFO org.apache.hadoop.ipc.Client: Retrying
> > connect
> > > to server: master/192.168.0.1:54310. Already tried 9 time(s).
> > > 2010-01-15 18:51:05,449 ERROR
> > > org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
> > Call
> > > to master/192.168.0.1:54310 failed on local exception:
> > > java.net.NoRouteToHostException: No route to host
> > >  at org.apache.hadoop.ipc.Client.wrapException(Client.java:774)
> > >  at org.apache.hadoop.ipc.Client.call(Client.java:742)
> > >  at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> > >  at $Proxy4.getProtocolVersion(Unknown Source)
> > >  at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
> > >  at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:346)
> > >  at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:383)
> > >  at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:314)
> > >  at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:291)
> > >  at
> > >
> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:269)
> > >  at
> > >
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:216)
> > >  at
> > >
> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1283)
> > >  at
> > >
> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1238)
> > >  at
> > >
> >
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1246)
> > >  at
> > >
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1368)
> > > Caused by: java.net.NoRouteToHostException: No route to host
> > >  at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > >  at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> > >  at
> > >
> >
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> > >  at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
> > >  at
> > org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:304)
> > >  at
> org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
> > >  at org.apache.hadoop.ipc.Client.getConnection(Client.java:859)
> > >  at org.apache.hadoop.ipc.Client.call(Client.java:719)
> > >  ... 13 more
> > > 2010-01-15 18:51:05,451 INFO
> > > org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
> > > /************************************************************
> > > SHUTDOWN_MSG: Shutting down DataNode at slave/192.168.0.2
> > > ************************************************************/
> > >
> > > Please help me asap.
> > >
> > > regards
> > > maria
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>

--000e0cd116ba60852c047da5272a--

From general-return-956-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 21 10:21:02 2010
Return-Path: <general-return-956-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95290 invoked from network); 21 Jan 2010 10:21:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Jan 2010 10:21:02 -0000
Received: (qmail 8550 invoked by uid 500); 21 Jan 2010 10:21:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8476 invoked by uid 500); 21 Jan 2010 10:21:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8466 invoked by uid 99); 21 Jan 2010 10:21:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Jan 2010 10:21:00 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [66.163.179.142] (HELO web35603.mail.mud.yahoo.com) (66.163.179.142)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 21 Jan 2010 10:20:52 +0000
Received: (qmail 2248 invoked by uid 60001); 21 Jan 2010 10:20:31 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1264069231; bh=ZdClqcKwTKjAKwf+gdJjwc4G7Lv5SjGAHfgNjTUBWF4=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=i2gIg2MeCMLBZrUWvkujResn2IJi8Y2H0OnwagVzxGVAsqe3owZ4fu1fMt7ibqgXlErlK9+5c8j09hqhyz5p/AKRbbdtQGrnO89X1lW/cMcAjN2cX11IfL0WCH9nbBUvxWRyLCHwO33gjAkSsXwPF2aGso42ICwLFFzJnUo2/vA=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=bAr7lJ5L0aqtJBTIQSxDRry6HFdumrBkbTNWsgcaJbTbtrQMC7pVJkQOMSEbqAkvv+6siQRwrv0wLWnfgXWyGpzv5eqnrD4MwqsjWxqYUvvHZkHbymq8ZBML2e7wGdG8Tl0SmCQquXSEGQV3p10jgrSb8vDhUnK934rhUMliibE=;
Message-ID: <82033.1447.qm@web35603.mail.mud.yahoo.com>
X-YMail-OSG: w9p9EDYVM1nt__mTzQfj_BkNdCCFBOW617AFVjagw.RnnLKIJv8RyCuCVDQtv.KK..NuOr5mYjDKqky_pt5qsFLnoKtHqbu27NgjanTJnuOs5seeG9pjBsNDh_HRz.EKaPkFJlnQc6chKzx.9F2B5kdFlEj0tRgA30UrF1CvLQOg0D1D7I8bZhmTYPTFFKSGLFUw2uzJcUTSDH_UmhvNLIk673ltV42BonEtmJS.z39Mu699goYkUZHg.a4E4t_8cfExhvDlu6MGuo7Dq8hrMHqgLZrMMD020d703YZHXSFG_kZ7NyvEV.XLCF8.n4p1WS2aoZC.tGYf54LPOmR5NT4VIcB4UiSbKorvWd288qdli4hvMnxKT8CJnTG0bgLYeqaaFS4c.jIOWtmC.giU7LVEbPPmVvdCP.eaCWG5HKLWhJrby.vFaSv0WcwOzyAx89PF_ebChDc9Spjro083a3UQX8WjWvZq4TRry6wda.unl4yjoLGSzHLbSfqsih2ThFlJ5B5PpQgV5rYhKCKk8GfN1oic4YzerNIAuV.PCVpU5Ga.ApCHaHSZ.7AVEEQ06wIIqW_dcJ0wtPcpNjHj2bBInpRwKl7283NKy9toZSGyIat9wSVXHU1.IjPgvJhS873D6ARvyWbiggYTt.9rs1E4PDB4l4wT0N19IfyDDFeMbGpAxJpI0RGYEDdnnPEQO6.Hq8CQTKKVuwERhWN2cTvHOQuDGwuEg5.jkQY6lGmNfGjRHoQqxR9j6d6BCnFHvd8o3ZihuKvV8r8pZJ8UpQM5n1RmuebAt8lF9C9uGRFsWttB9DUOo1i.PryADieUMJd6A7pMz.MqHh0L.eVQrcyGIt66uHRjWhxXF.kE1RNlYMWY4ko9hJRrrVb4bAhK3OrLaIngXIWVipqf8zPnClaepBrOEGOEUWe1dH5bx1kQicCBbFRKQfFfgLwyKdm1.4DL6bUigy58Lx_QF
 h8lBGMrQcNioDXXWYOtSfNClQ1SHsYain8xw_.hSQ4pcIxsXIYGbpBRzAo_e01eT5RBJ9b1MoWn.yVQaR4iG1sHNQcgS_eMOFWzrA--
Received: from [115.186.10.32] by web35603.mail.mud.yahoo.com via HTTP; Thu, 21 Jan 2010 02:20:30 PST
X-Mailer: YahooMailRC/272.7 YahooMailWebService/0.8.100.260964
References: <573443.25346.qm@web35606.mail.mud.yahoo.com>  <5a921af21001190509hec85921u5d8be7f84b9cbe@mail.gmail.com>  <364995.16459.qm@web35602.mail.mud.yahoo.com>  <5a921af21001200051p509b27f0jce1a11663f795154@mail.gmail.com>  <586024.28687.qm@web35601.mail.mud.yahoo.com> <5a921af21001202029r14b7ae8ew7d1ba0ca4a91c613@mail.gmail.com>
Date: Thu, 21 Jan 2010 02:20:30 -0800 (PST)
From: ZUBAIR MIAN <mian_zubair@yahoo.com>
Subject: Re: Problem with datanode(slave machine)
To: general@hadoop.apache.org
In-Reply-To: <5a921af21001202029r14b7ae8ew7d1ba0ca4a91c613@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-762101944-1264069230=:1447"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-762101944-1264069230=:1447
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

the error msg is "no route to host" if i do "telnet 192.168.0.2 54311" and =
"Connection failed" with "telnet 192.168.0.1 54310"=0A=0A=0A=0A=0A_________=
_______________________=0AFrom: xiao yang <yangxiao9901@gmail.com>=0ATo: ge=
neral@hadoop.apache.org=0ASent: Thu, January 21, 2010 9:29:50 AM=0ASubject:=
 Re: Problem with datanode(slave machine)=0A=0AWhat's the error message of =
telnet?=0A=0AOn Thu, Jan 21, 2010 at 1:53 AM, ZUBAIR MIAN <mian_zubair@yaho=
o.com> wrote:=0A=0A> Still giving me the same error=0A>=0A>=0A>=0A>=0A> ___=
_____________________________=0A> From: xiao yang <yangxiao9901@gmail.com>=
=0A> To: general@hadoop.apache.org=0A> Sent: Wed, January 20, 2010 1:51:04 =
PM=0A> Subject: Re: Problem with datanode(slave machine)=0A>=0A> Maybe the =
firewall blocked this port.=0A> Try:=0A> $telnet 192.168.0.1 54310=0A>=0A> =
On Tue, Jan 19, 2010 at 10:17 PM, ZUBAIR MIAN <mian_zubair@yahoo.com>=0A> w=
rote:=0A>=0A> > Have already done that.=0A> >=0A> >=0A> >=0A> >=0A> > _____=
___________________________=0A> > From: xiao yang <yangxiao9901@gmail.com>=
=0A> > To: general@hadoop.apache.org=0A> > Sent: Tue, January 19, 2010 6:09=
:57 PM=0A> > Subject: Re: Problem with datanode(slave machine)=0A> >=0A> > =
Add hosts and ip address in /etc/hosts.=0A> > It may work.=0A> >=0A> > On T=
ue, Jan 19, 2010 at 8:34 PM, ZUBAIR MIAN <mian_zubair@yahoo.com>=0A> > wrot=
e:=0A> >=0A> > > Dear all=0A> > >=0A> > > I am facing problem wrt the datan=
ode(at slave). In the log file of=0A> slave=0A> > i=0A> > > am getting foll=
owing message:=0A> > >=A0 =A0 =A0 =A0 =A0 =A0 "Call to master/192.168.0.1:5=
4310 failed on local=0A> exception:=0A> > > java.net.NoRouteToHostException=
:=A0 =A0 =A0 =A0 =A0 =A0 No route to host at=0A> > > org.apache.hadoop.ipc.=
Client.wrapException"=0A> > > I am currently working on two node cluster. T=
he network is working=0A> fine.=0A> > > Both machines can ping each other.=
=0A> > >=0A> > > And Following is the log file of datanode.=0A> > >=0A> > >=
=0A> > >=A0 STARTUP_MSG: Starting DataNode=0A> > > STARTUP_MSG:=A0 host =3D=
 slave/192.168.0.2=0A> > > STARTUP_MSG:=A0 args =3D []=0A> > > STARTUP_MSG:=
=A0 version =3D 0.20.1=0A> > > STARTUP_MSG:=A0 build =3D=0A> > > http://svn=
.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc1-r=0A> > > 81022=
0; compiled by 'oom' on Tue Sep=A0 1 20:55:56 UTC 2009=0A> > > ************=
************************************************/=0A> > > 2010-01-15 18:50:=
56,435 INFO org.apache.hadoop.ipc.Client: Retrying=0A> > connect=0A> > > to=
 server: master/192.168.0.1:54310. Already tried 0 time(s).=0A> > > 2010-01=
-15 18:50:57,436 INFO org.apache.hadoop.ipc.Client: Retrying=0A> > connect=
=0A> > > to server: master/192.168.0.1:54310. Already tried 1 time(s).=0A> =
> > 2010-01-15 18:50:58,437 INFO org.apache.hadoop.ipc.Client: Retrying=0A>=
 > connect=0A> > > to server: master/192.168.0.1:54310. Already tried 2 tim=
e(s).=0A> > > 2010-01-15 18:50:59,438 INFO org.apache.hadoop.ipc.Client: Re=
trying=0A> > connect=0A> > > to server: master/192.168.0.1:54310. Already t=
ried 3 time(s).=0A> > > 2010-01-15 18:51:00,439 INFO org.apache.hadoop.ipc.=
Client: Retrying=0A> > connect=0A> > > to server: master/192.168.0.1:54310.=
 Already tried 4 time(s).=0A> > > 2010-01-15 18:51:01,440 INFO org.apache.h=
adoop.ipc.Client: Retrying=0A> > connect=0A> > > to server: master/192.168.=
0.1:54310. Already tried 5 time(s).=0A> > > 2010-01-15 18:51:02,441 INFO or=
g.apache.hadoop.ipc.Client: Retrying=0A> > connect=0A> > > to server: maste=
r/192.168.0.1:54310. Already tried 6 time(s).=0A> > > 2010-01-15 18:51:03,4=
43 INFO org.apache.hadoop.ipc.Client: Retrying=0A> > connect=0A> > > to ser=
ver: master/192.168.0.1:54310. Already tried 7 time(s).=0A> > > 2010-01-15 =
18:51:04,444 INFO org.apache.hadoop.ipc.Client: Retrying=0A> > connect=0A> =
> > to server: master/192.168.0.1:54310. Already tried 8 time(s).=0A> > > 2=
010-01-15 18:51:05,445 INFO org.apache.hadoop.ipc.Client: Retrying=0A> > co=
nnect=0A> > > to server: master/192.168.0.1:54310. Already tried 9 time(s).=
=0A> > > 2010-01-15 18:51:05,449 ERROR=0A> > > org.apache.hadoop.hdfs.serve=
r.datanode.DataNode: java.io.IOException:=0A> > Call=0A> > > to master/192.=
168.0.1:54310 failed on local exception:=0A> > > java.net.NoRouteToHostExce=
ption: No route to host=0A> > >=A0 at org.apache.hadoop.ipc.Client.wrapExce=
ption(Client.java:774)=0A> > >=A0 at org.apache.hadoop.ipc.Client.call(Clie=
nt.java:742)=0A> > >=A0 at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.jav=
a:220)=0A> > >=A0 at $Proxy4.getProtocolVersion(Unknown Source)=0A> > >=A0 =
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)=0A> > >=A0 at org.apach=
e.hadoop.ipc.RPC.getProxy(RPC.java:346)=0A> > >=A0 at org.apache.hadoop.ipc=
.RPC.getProxy(RPC.java:383)=0A> > >=A0 at org.apache.hadoop.ipc.RPC.waitFor=
Proxy(RPC.java:314)=0A> > >=A0 at org.apache.hadoop.ipc.RPC.waitForProxy(RP=
C.java:291)=0A> > >=A0 at=0A> > >=0A> >=0A> org.apache.hadoop.hdfs.server.d=
atanode.DataNode.startDataNode(DataNode.java:269)=0A> > >=A0 at=0A> > >=0A>=
 org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:216)=
=0A> > >=A0 at=0A> > >=0A> >=0A> org.apache.hadoop.hdfs.server.datanode.Dat=
aNode.makeInstance(DataNode.java:1283)=0A> > >=A0 at=0A> > >=0A> >=0A> org.=
apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.ja=
va:1238)=0A> > >=A0 at=0A> > >=0A> >=0A> org.apache.hadoop.hdfs.server.data=
node.DataNode.createDataNode(DataNode.java:1246)=0A> > >=A0 at=0A> > >=0A> =
org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1368)=0A=
> > > Caused by: java.net.NoRouteToHostException: No route to host=0A> > >=
=A0 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)=0A> > >=A0 =
at=0A> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:=
592)=0A> > >=A0 at=0A> > >=0A> >=0A> org.apache.hadoop.net.SocketIOWithTime=
out.connect(SocketIOWithTimeout.java:206)=0A> > >=A0 at org.apache.hadoop.n=
et.NetUtils.connect(NetUtils.java:404)=0A> > >=A0 at=0A> > org.apache.hadoo=
p.ipc.Client$Connection.setupIOstreams(Client.java:304)=0A> > >=A0 at=0A> o=
rg.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)=0A> > >=
=A0 at org.apache.hadoop.ipc.Client.getConnection(Client.java:859)=0A> > >=
=A0 at org.apache.hadoop.ipc.Client.call(Client.java:719)=0A> > >=A0 ... 13=
 more=0A> > > 2010-01-15 18:51:05,451 INFO=0A> > > org.apache.hadoop.hdfs.s=
erver.datanode.DataNode: SHUTDOWN_MSG:=0A> > > /***************************=
*********************************=0A> > > SHUTDOWN_MSG: Shutting down DataN=
ode at slave/192.168.0.2=0A> > > ******************************************=
******************/=0A> > >=0A> > > Please help me asap.=0A> > >=0A> > > re=
gards=0A> > > maria=0A> > >=0A> > >=0A> > >=0A> > >=0A> >=0A> >=0A> >=0A> >=
=0A> >=0A>=0A>=0A>=0A>=0A>=0A=0A=0A=0A      
--0-762101944-1264069230=:1447--

From general-return-957-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 21 10:47:46 2010
Return-Path: <general-return-957-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 8303 invoked from network); 21 Jan 2010 10:47:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Jan 2010 10:47:46 -0000
Received: (qmail 38533 invoked by uid 500); 21 Jan 2010 10:47:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38466 invoked by uid 500); 21 Jan 2010 10:47:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 38456 invoked by uid 99); 21 Jan 2010 10:47:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Jan 2010 10:47:45 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Jan 2010 10:47:35 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id F0088B7F03
	for <general@hadoop.apache.org>; Thu, 21 Jan 2010 10:47:14 +0000 (GMT)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id 7qJy6J5XCb-I for <general@hadoop.apache.org>;
	Thu, 21 Jan 2010 10:47:08 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 50DF7B7EE8
	for <general@hadoop.apache.org>; Thu, 21 Jan 2010 10:47:08 +0000 (GMT)
MailScanner-NULL-Check: 1264675617.93711@v62GQe6TCwtk11UymIQ+tA
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o0LAkuoK019582
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Thu, 21 Jan 2010 10:46:57 GMT
Message-ID: <4B5830A0.1020801@apache.org>
Date: Thu, 21 Jan 2010 10:46:56 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Problem with datanode(slave machine)
References: <573443.25346.qm@web35606.mail.mud.yahoo.com>  <5a921af21001190509hec85921u5d8be7f84b9cbe@mail.gmail.com>  <364995.16459.qm@web35602.mail.mud.yahoo.com>  <5a921af21001200051p509b27f0jce1a11663f795154@mail.gmail.com>  <586024.28687.qm@web35601.mail.mud.yahoo.com> <5a921af21001202029r14b7ae8ew7d1ba0ca4a91c613@mail.gmail.com> <82033.1447.qm@web35603.mail.mud.yahoo.com>
In-Reply-To: <82033.1447.qm@web35603.mail.mud.yahoo.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o0LAkuoK019582
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

ZUBAIR MIAN wrote:
> the error msg is "no route to host" if i do "telnet 192.168.0.2 54311" and "Connection failed" with "telnet 192.168.0.1 54310"
> 

http://wiki.apache.org/hadoop/NoRouteToHost


From general-return-958-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 22 03:52:59 2010
Return-Path: <general-return-958-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72892 invoked from network); 22 Jan 2010 03:52:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jan 2010 03:52:59 -0000
Received: (qmail 80771 invoked by uid 500); 22 Jan 2010 03:52:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80621 invoked by uid 500); 22 Jan 2010 03:52:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80602 invoked by uid 99); 22 Jan 2010 03:52:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Jan 2010 03:52:57 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of naveenkumarp@huawei.com designates 119.145.14.66 as permitted sender)
Received: from [119.145.14.66] (HELO szxga03-in.huawei.com) (119.145.14.66)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Jan 2010 03:52:48 +0000
Received: from huawei.com (szxga03-in [172.24.2.9])
 by szxga03-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug
 8 2006)) with ESMTP id <0KWM00M4EQRE16@szxga03-in.huawei.com> for
 general@hadoop.apache.org; Fri, 22 Jan 2010 11:52:27 +0800 (CST)
Received: from huawei.com ([172.24.2.119])
 by szxga03-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug
 8 2006)) with ESMTP id <0KWM00D92QRET0@szxga03-in.huawei.com> for
 general@hadoop.apache.org; Fri, 22 Jan 2010 11:52:26 +0800 (CST)
Received: from BLRNSHTIPL6NC ([10.18.1.36])
 by szxml06-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug
 8 2006)) with ESMTPA id <0KWM00K7PQRE49@szxml06-in.huawei.com> for
 general@hadoop.apache.org; Fri, 22 Jan 2010 11:52:26 +0800 (CST)
Date: Fri, 22 Jan 2010 09:22:24 +0530
From: Naveen Kumar Prasad <naveenkumarp@huawei.com>
Subject: RE: debug MapReduce task running on a cluster
In-reply-to: <C77D1253.FC17%amarsri@yahoo-inc.com>
To: general@hadoop.apache.org
Reply-to: naveenkumarp@huawei.com
Message-id: <000b01ca9b16$4f79f960$2401120a@china.huawei.com>
Organization: Htipl
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.3790.3168
X-Mailer: Microsoft Office Outlook 11
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
Thread-index: AcqZr6ZeddEW+lU+QFO7l6MAv9phhAALglzbAE4muyA=

Thanks.. 


Regards,
Naveen Kumar
HUAWEI TECHNOLOGIES CO.,LTD.  huawei_logo


Address: Huawei Industrial Base
Bantian Longgang
Shenzhen 518129, P.R.China
www.huawei.com
----------------------------------------------------------------------------
-------------------------------------
This e-mail and its attachments contain confidential information from
HUAWEI, which is intended only for the person or entity whose address is
listed above. Any use of the information contained herein in any way
(including, but not limited to, total or partial disclosure, reproduction,
or dissemination) by persons other than the intended recipient(s) is
prohibited. If you receive this e-mail in error, please notify the sender by
phone or email immediately and delete it!

-----Original Message-----
From: Amareshwari Sri Ramadasu [mailto:amarsri@YAHOO-INC.COM] 
Sent: Wednesday, January 20, 2010 8:05 PM
To: general@hadoop.apache.org; naveenkumarp@huawei.com
Subject: Re: debug MapReduce task running on a cluster


See http://wiki.apache.org/hadoop/HowToDebugMapReducePrograms
http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html#IsolationR
unner
and
http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html#Debugging

Thanks
Amareshwari

On 1/20/10 2:35 PM, "Naveen Kumar Prasad" <naveenkumarp@huawei.com> wrote:

Hi All,

I am new to hadoop/Mapreduce usage.
I was trying to find ways to debug a MapReduce task running on a cluster.
However, I was not able to get something concrete and precise.

It would be great if anyone can help me on the same.

Regards,
Naveen Kumar
HUAWEI TECHNOLOGIES CO.,LTD.  huawei_logo
<outbind://24-000000004ED456449CB8724C859F40771D413A7584132000/cid:919325906
@21052009-015A>


Address: Huawei Industrial Base
Bantian Longgang
Shenzhen 518129, P.R.China
www.huawei.com
----------------------------------------------------------------------------
-------------------------------------
This e-mail and its attachments contain confidential information from
HUAWEI, which is intended only for the person or entity whose address is
listed above. Any use of the information contained herein in any way
(including, but not limited to, total or partial disclosure, reproduction,
or dissemination) by persons other than the intended recipient(s) is
prohibited. If you receive this e-mail in error, please notify the sender by
phone or email immediately and delete it!




From general-return-959-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 22 03:54:44 2010
Return-Path: <general-return-959-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73350 invoked from network); 22 Jan 2010 03:54:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jan 2010 03:54:44 -0000
Received: (qmail 82582 invoked by uid 500); 22 Jan 2010 03:54:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82433 invoked by uid 500); 22 Jan 2010 03:54:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82423 invoked by uid 99); 22 Jan 2010 03:54:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Jan 2010 03:54:43 +0000
X-ASF-Spam-Status: No, hits=3.7 required=10.0
	tests=HTML_IMAGE_ONLY_28,HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of naveenkumarp@huawei.com designates 119.145.14.67 as permitted sender)
Received: from [119.145.14.67] (HELO szxga04-in.huawei.com) (119.145.14.67)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Jan 2010 03:54:34 +0000
Received: from huawei.com (szxga04-in [172.24.2.12])
 by szxga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug
 8 2006)) with ESMTP id <0KWM00HD7QU9DW@szxga04-in.huawei.com> for
 general@hadoop.apache.org; Fri, 22 Jan 2010 11:54:09 +0800 (CST)
Received: from huawei.com ([172.24.2.119])
 by szxga04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug
 8 2006)) with ESMTP id <0KWM001WSQU948@szxga04-in.huawei.com> for
 general@hadoop.apache.org; Fri, 22 Jan 2010 11:54:09 +0800 (CST)
Received: from BLRNSHTIPL6NC ([10.18.1.36])
 by szxml06-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug
 8 2006)) with ESMTPA id <0KWM002RKQU893@szxml06-in.huawei.com> for
 general@hadoop.apache.org; Fri, 22 Jan 2010 11:54:09 +0800 (CST)
Date: Fri, 22 Jan 2010 09:24:07 +0530
From: Naveen Kumar Prasad <naveenkumarp@huawei.com>
Subject: encoding types supported by Hadoop
To: general@hadoop.apache.org
Reply-to: naveenkumarp@huawei.com
Message-id: <000c01ca9b16$8cb5f040$2401120a@china.huawei.com>
Organization: Htipl
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.3790.3168
X-Mailer: Microsoft Office Outlook 11
Content-type: multipart/alternative;
 boundary="Boundary_(ID_dhFYSgxhEFHkMcPN0BLErQ)"
Thread-index: AcqbFowrx2zkXZwmTZWgQo2llOJyOA==

--Boundary_(ID_dhFYSgxhEFHkMcPN0BLErQ)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT

Hi All,
 
I am new to hadoop/Mapreduce usage.
 
Can anyone tell me how to write a simple MapReduce implementation to just
read some files from the <input> directory 
and write to <output> directory.
 
Also I wanted to know which all encoding types are supported by Hadoop and
how to configure and use various encoding types.
 
Regards,
Naveen Kumar
HUAWEI TECHNOLOGIES CO.,LTD.  huawei_logo
<outbind://24-000000004ED456449CB8724C859F40771D413A7584132000/cid:919325906
@21052009-015A> 


Address: Huawei Industrial Base
Bantian Longgang
Shenzhen 518129, P.R.China
www.huawei.com
----------------------------------------------------------------------------
-------------------------------------
This e-mail and its attachments contain confidential information from
HUAWEI, which is intended only for the person or entity whose address is
listed above. Any use of the information contained herein in any way
(including, but not limited to, total or partial disclosure, reproduction,
or dissemination) by persons other than the intended recipient(s) is
prohibited. If you receive this e-mail in error, please notify the sender by
phone or email immediately and delete it!
 

--Boundary_(ID_dhFYSgxhEFHkMcPN0BLErQ)--

From general-return-960-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 22 04:46:45 2010
Return-Path: <general-return-960-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84204 invoked from network); 22 Jan 2010 04:46:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jan 2010 04:46:45 -0000
Received: (qmail 15186 invoked by uid 500); 22 Jan 2010 04:46:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15015 invoked by uid 500); 22 Jan 2010 04:46:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15005 invoked by uid 99); 22 Jan 2010 04:46:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Jan 2010 04:46:43 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.85.216.201] (HELO mail-px0-f201.google.com) (209.85.216.201)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Jan 2010 04:46:36 +0000
Received: by pxi39 with SMTP id 39so526605pxi.2
        for <general@hadoop.apache.org>; Thu, 21 Jan 2010 20:46:16 -0800 (PST)
MIME-Version: 1.0
Received: by 10.142.55.6 with SMTP id d6mr1626821wfa.303.1264135576189; Thu, 
	21 Jan 2010 20:46:16 -0800 (PST)
In-Reply-To: <000c01ca9b16$8cb5f040$2401120a@china.huawei.com>
References: <000c01ca9b16$8cb5f040$2401120a@china.huawei.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Thu, 21 Jan 2010 20:45:56 -0800
Message-ID: <45f85f71001212045k49edcd4fhd2ebee0fac798a9b@mail.gmail.com>
Subject: Re: encoding types supported by Hadoop
To: general@hadoop.apache.org, naveenkumarp@huawei.com
Content-Type: multipart/alternative; boundary=0016368e2729f77aed047db97f46

--0016368e2729f77aed047db97f46
Content-Type: text/plain; charset=ISO-8859-1

Hi Naveen,

On Thu, Jan 21, 2010 at 7:54 PM, Naveen Kumar Prasad <
naveenkumarp@huawei.com> wrote:

> Hi All,
>
> I am new to hadoop/Mapreduce usage.
>
> Can anyone tell me how to write a simple MapReduce implementation to just
> read some files from the <input> directory
> and write to <output> directory.
>

It sounds like what you want is the distcp job. Just run "hadoop distcp" and
it will print some usage information for you.


>
> Also I wanted to know which all encoding types are supported by Hadoop and
> how to configure and use various encoding types.
>
>
I'm not sure what you mean here by encoding. Could you elaborate on this
question, please?

Thanks
-Todd

--0016368e2729f77aed047db97f46--

From general-return-961-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 22 05:02:05 2010
Return-Path: <general-return-961-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87274 invoked from network); 22 Jan 2010 05:02:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jan 2010 05:02:05 -0000
Received: (qmail 23600 invoked by uid 500); 22 Jan 2010 05:02:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23481 invoked by uid 500); 22 Jan 2010 05:02:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23470 invoked by uid 99); 22 Jan 2010 05:02:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Jan 2010 05:02:03 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of naveenkumarp@huawei.com designates 119.145.14.64 as permitted sender)
Received: from [119.145.14.64] (HELO szxga01-in.huawei.com) (119.145.14.64)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Jan 2010 05:01:53 +0000
Received: from huawei.com (szxga01-in [172.24.2.3])
 by szxga01-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug
 8 2006)) with ESMTP id <0KWM004TPTYJMB@szxga01-in.huawei.com> for
 general@hadoop.apache.org; Fri, 22 Jan 2010 13:01:31 +0800 (CST)
Received: from huawei.com ([172.24.2.119])
 by szxga01-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug
 8 2006)) with ESMTP id <0KWM00I37TYJGB@szxga01-in.huawei.com> for
 general@hadoop.apache.org; Fri, 22 Jan 2010 13:01:31 +0800 (CST)
Received: from BLRNSHTIPL6NC ([10.18.1.36])
 by szxml06-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug
 8 2006)) with ESMTPA id <0KWM00L7DTYHOI@szxml06-in.huawei.com> for
 general@hadoop.apache.org; Fri, 22 Jan 2010 13:01:31 +0800 (CST)
Date: Fri, 22 Jan 2010 10:31:27 +0530
From: Naveen Kumar Prasad <naveenkumarp@huawei.com>
Subject: RE: encoding types supported by Hadoop
In-reply-to: <45f85f71001212045k49edcd4fhd2ebee0fac798a9b@mail.gmail.com>
To: general@hadoop.apache.org
Cc: todd@cloudera.com
Reply-to: naveenkumarp@huawei.com
Message-id: <002201ca9b1f$f5b82d20$2401120a@china.huawei.com>
Organization: Htipl
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.3790.3168
X-Mailer: Microsoft Office Outlook 11
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
Thread-index: AcqbHeffaDmIaxFdSuaRfWNb5zhZaAAAS7og
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Todd,

To elaborate more on the encoding query : 

Actually the input file we use while working with Hadoop, may have different
encoding types,
Like : encoding="UTF-8" (UTF-16, GBK, etc)
So I want to know which all encoding types are supported by Hadoop.

User Scenario : 
I want to read from a input text file (suppose file01.txt) which has chinese
characters 
And write it to a output text file (suppose fileo2.txt) and verify whether
the chinese characters 
are coming properly in the output file (and not as junk characters).

{ It would be appreciable if u cud tell me how to verify this. ) 


Regards,
Naveen Kumar
HUAWEI TECHNOLOGIES CO.,LTD.  huawei_logo


Address: Huawei Industrial Base
Bantian Longgang
Shenzhen 518129, P.R.China
www.huawei.com
----------------------------------------------------------------------------
-------------------------------------
This e-mail and its attachments contain confidential information from
HUAWEI, which is intended only for the person or entity whose address is
listed above. Any use of the information contained herein in any way
(including, but not limited to, total or partial disclosure, reproduction,
or dissemination) by persons other than the intended recipient(s) is
prohibited. If you receive this e-mail in error, please notify the sender by
phone or email immediately and delete it!

-----Original Message-----
From: Todd Lipcon [mailto:todd@cloudera.com] 
Sent: Friday, January 22, 2010 10:16 AM
To: general@hadoop.apache.org; naveenkumarp@huawei.com
Subject: Re: encoding types supported by Hadoop

Hi Naveen,

On Thu, Jan 21, 2010 at 7:54 PM, Naveen Kumar Prasad <
naveenkumarp@huawei.com> wrote:

> Hi All,
>
> I am new to hadoop/Mapreduce usage.
>
> Can anyone tell me how to write a simple MapReduce implementation to 
> just read some files from the <input> directory and write to <output> 
> directory.
>

It sounds like what you want is the distcp job. Just run "hadoop distcp" and
it will print some usage information for you.


>
> Also I wanted to know which all encoding types are supported by Hadoop 
> and how to configure and use various encoding types.
>
>
I'm not sure what you mean here by encoding. Could you elaborate on this
question, please?

Thanks
-Todd


From general-return-962-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 22 05:37:15 2010
Return-Path: <general-return-962-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 279 invoked from network); 22 Jan 2010 05:37:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jan 2010 05:37:15 -0000
Received: (qmail 56493 invoked by uid 500); 22 Jan 2010 05:37:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56394 invoked by uid 500); 22 Jan 2010 05:37:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56384 invoked by uid 99); 22 Jan 2010 05:37:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Jan 2010 05:37:13 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of andrew.wang.1900@gmail.com designates 74.125.78.27 as permitted sender)
Received: from [74.125.78.27] (HELO ey-out-2122.google.com) (74.125.78.27)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Jan 2010 05:37:05 +0000
Received: by ey-out-2122.google.com with SMTP id 22so192854eye.23
        for <general@hadoop.apache.org>; Thu, 21 Jan 2010 21:36:43 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=g9E/Ki5afAAI9gE39FMAZ1O4BwGnOxToIsnpSjHeWpY=;
        b=BL0qAzeMXoyJcux8qnAusLENxeI/3LBohe1JNZHphnge1+yNldjzjjzLGTSE7pbMoj
         l9x/3hOvPyCLifZggoNiwyrg8DstVi0IcqrDUv2XcBhmrj1vLLPtEuv6Pm4zBHN109lT
         APnVzDiV0UuJy8kKBQiaJRFIt/g9ohhg0RaRU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=uuyyv4yOjZD994d2VnLiTFR3fDOVhTNys3xUhu7rQ4Ph5VME+0adpO22twPjFzMKZt
         dvxiIO/VLJqCJIn/PjUtyXJc8qz9BfmGxPRykVZZAGkeUAkdT2z580OdVwZdZ0lmub4Z
         ktiqibW6iNdzZ0+bLf475NS2ZnjN5Ve9zJ3M4=
MIME-Version: 1.0
Received: by 10.216.86.65 with SMTP id v43mr950437wee.118.1264138603802; Thu, 
	21 Jan 2010 21:36:43 -0800 (PST)
In-Reply-To: <000c01ca9b16$8cb5f040$2401120a@china.huawei.com>
References: <000c01ca9b16$8cb5f040$2401120a@china.huawei.com>
Date: Fri, 22 Jan 2010 13:36:43 +0800
Message-ID: <995905631001212136r4e723601i7e80234adcc3196c@mail.gmail.com>
Subject: Re: encoding types supported by Hadoop
From: Andrew Wang <andrew.wang.1900@gmail.com>
To: general@hadoop.apache.org, naveenkumarp@huawei.com
Content-Type: multipart/alternative; boundary=0016e6d784ee6d31ac047dba3467

--0016e6d784ee6d31ac047dba3467
Content-Type: text/plain; charset=ISO-8859-1

Hi,Naveen

The default encoding type supported by Hadoop is UTF-8, so, if you'd like to
use other types, you have to custom the FileInputFormat and
FileOutPutFormat.

For me, i like to convert the input content to some special encoding type,
etc: String line = new String(value.getBytes(), 0, value.getLength(),"GBK");


At same time, i implement one custom FileOutputFormat,
namely,GbkOutputFormat, to support output with GBK type.

On Fri, Jan 22, 2010 at 11:54 AM, Naveen Kumar Prasad <
naveenkumarp@huawei.com> wrote:

> Hi All,
>
> I am new to hadoop/Mapreduce usage.
>
> Can anyone tell me how to write a simple MapReduce implementation to just
> read some files from the <input> directory
> and write to <output> directory.
>
> Also I wanted to know which all encoding types are supported by Hadoop and
> how to configure and use various encoding types.
>
> Regards,
> Naveen Kumar
> HUAWEI TECHNOLOGIES CO.,LTD.  huawei_logo
>
> <outbind://24-000000004ED456449CB8724C859F40771D413A7584132000/cid:919325906
> @21052009-015A>
>
>
> Address: Huawei Industrial Base
> Bantian Longgang
> Shenzhen 518129, P.R.China
> www.huawei.com
>
> ----------------------------------------------------------------------------
> -------------------------------------
> This e-mail and its attachments contain confidential information from
> HUAWEI, which is intended only for the person or entity whose address is
> listed above. Any use of the information contained herein in any way
> (including, but not limited to, total or partial disclosure, reproduction,
> or dissemination) by persons other than the intended recipient(s) is
> prohibited. If you receive this e-mail in error, please notify the sender
> by
> phone or email immediately and delete it!
>
>


-- 
http://anqiang1900.blog.163.com/

--0016e6d784ee6d31ac047dba3467--

From general-return-963-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 22 07:49:04 2010
Return-Path: <general-return-963-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45543 invoked from network); 22 Jan 2010 07:49:04 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jan 2010 07:49:04 -0000
Received: (qmail 37367 invoked by uid 500); 22 Jan 2010 07:49:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37285 invoked by uid 500); 22 Jan 2010 07:49:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37271 invoked by uid 99); 22 Jan 2010 07:49:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Jan 2010 07:49:02 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of hkumar.arora@gmail.com designates 209.85.223.195 as permitted sender)
Received: from [209.85.223.195] (HELO mail-iw0-f195.google.com) (209.85.223.195)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Jan 2010 07:48:55 +0000
Received: by iwn33 with SMTP id 33so789979iwn.29
        for <multiple recipients>; Thu, 21 Jan 2010 23:48:35 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:cc:content-type;
        bh=mrl7arPJEbtvUjjG9PP0vweZbT+NwSWPS4sySN5Ji/Y=;
        b=fNV5Zx6nq+JzIfzR1OmFLXcbWrJpd/oz424NxfLbrPjPUYjn2eA16wRW5AgmArnWQs
         fWzIDnHKFbXuCVaaHMSi6T+/J2nmN/5YR8bO/zQ4jQirBgmRPVxRuLcnz0stfu5K0XXM
         yCSDmdj3QXc4kOcY6LqWjaHHv8TR7B/h/EC8A=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:cc:content-type;
        b=hSnX8NTbrxuY2nm4JaxKC3dv0uZXFPFfcjnR/l+ipn+ofRercFr5JzvJpJmM6LfxzQ
         qsqwnfYW56mcwtAGTAnOHuXIsJULdDGtBsrsVGEjq9vJXGVqoj8hx5ZYPvmHn69b5Ciw
         4mmV8Buz74CNR4GpHkL8l6BVRtDZYd73JqMD0=
MIME-Version: 1.0
Received: by 10.231.145.1 with SMTP id b1mr1734358ibv.62.1264146515096; Thu, 
	21 Jan 2010 23:48:35 -0800 (PST)
Date: Fri, 22 Jan 2010 16:48:35 +0900
Message-ID: <e497591c1001212348o15da64y302667a412b75745@mail.gmail.com>
Subject: attempt_201001221636_0001_m_000003_0, Status : FAILED 
	java.io.FileNotFoundException:
From: Harshit Kumar <hkumar.arora@gmail.com>
To: general@hadoop.apache.org
Cc: mapreduce-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485eba798f9f53f047dbc0be4

--001485eba798f9f53f047dbc0be4
Content-Type: text/plain; charset=UTF-8

Hi

I will appreciate if someone can help resolve the following error. i am
going nuts over this.

$ bin/hadoop jar sparqlcloud.jar org.bike.MainClass
10/01/22 16:38:27 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
10/01/22 16:38:27 INFO mapred.FileInputFormat: Total input paths to process
: 1
10/01/22 16:38:28 INFO mapred.JobClient: Running job: job_201001221636_0001
10/01/22 16:38:29 INFO mapred.JobClient:  map 0% reduce 0%
10/01/22 16:38:37 INFO mapred.JobClient: Task Id :
attempt_201001221636_0001_m_000003_0, Status : FAILED
*java.io.FileNotFoundException: File
C:/tmp/hadoop-SYSTEM/mapred/local/taskTracker/jobcache/job_201001221636_0001/attempt_201001221636_0001_m_000003_0/work/tmp
does not exist.*
        at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:420)
        at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:244)
        at
org.apache.hadoop.mapred.TaskRunner.setupWorkDir(TaskRunner.java:520)
        at org.apache.hadoop.mapred.Child.main(Child.java:143)


===============
 working under CYGWIN and using hadoop-0.19.2

I have tried the following solutions,
1. bin/hadoop fsck / - output - The file system under / is healthy
2. formatted the hdfs bin/hadoop namenode -format - but error does not go.
3. change the value tmp.dir property in conf/hadoop-default.xml to some
other directory but still the error doesnot go.

I will appreciate if some one can help resolve the error.


Thanks
H. Kumar

--001485eba798f9f53f047dbc0be4--

From general-return-964-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 22 08:49:31 2010
Return-Path: <general-return-964-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60512 invoked from network); 22 Jan 2010 08:49:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jan 2010 08:49:31 -0000
Received: (qmail 6150 invoked by uid 500); 22 Jan 2010 08:49:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6082 invoked by uid 500); 22 Jan 2010 08:49:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6072 invoked by uid 99); 22 Jan 2010 08:49:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Jan 2010 08:49:30 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of hkumar.arora@gmail.com designates 209.85.223.195 as permitted sender)
Received: from [209.85.223.195] (HELO mail-iw0-f195.google.com) (209.85.223.195)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Jan 2010 08:49:20 +0000
Received: by iwn33 with SMTP id 33so820147iwn.29
        for <general@hadoop.apache.org>; Fri, 22 Jan 2010 00:48:59 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=XD+Ux/jpmydDgFy4dxyOT8yY4Vk9Z/Go8lJWRnb8l9I=;
        b=D8vXsAc9cU7gyKLMslfiiz6w5vQvQZLCO77KEgPxCZtTPS0f8nxsGvcnR14vPD3zJy
         BO39sR5Z8FWM4JHzywZG1i5AcT5u2NSFqbsPOOxJO030KzYuZ2G6HbOqMdTTcEftRvJa
         t4NciMGYmw6Q2yQOjwDS9XnnBPpJrzXyPCXCQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=bjo7wcyMZV2zJI1zIp+mZEXE1pRA/cGfdI+/hc9e6E2fFB6nSFj743EexV5zblr1Pt
         7f/c2t+WD9SbtJektwQVHktauZ3HCxL/tMBFOqf7j51vkyzlVpa5gUnNdLAyQ2HAl8nP
         JwItR1nTnjj0E6M8s6GGMu/6QRLa98ujR5tuw=
MIME-Version: 1.0
Received: by 10.231.148.13 with SMTP id n13mr4114068ibv.67.1264150139074; Fri, 
	22 Jan 2010 00:48:59 -0800 (PST)
In-Reply-To: <e497591c1001212348o15da64y302667a412b75745@mail.gmail.com>
References: <e497591c1001212348o15da64y302667a412b75745@mail.gmail.com>
Date: Fri, 22 Jan 2010 17:48:59 +0900
Message-ID: <e497591c1001220048w18fa6088yfeb8f4f4b5ebc565@mail.gmail.com>
Subject: Re: attempt_201001221636_0001_m_000003_0, Status : FAILED 
	java.io.FileNotFoundException:
From: Harshit Kumar <hkumar.arora@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64079aefb7b91047dbce3fa
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e64079aefb7b91047dbce3fa
Content-Type: text/plain; charset=UTF-8

I have just updated to cygwin 1.7 and if I try to format the namenode, i get
the following warning by cygwin.

$ bin/hadoop namenode -format
cygwin warning:
  MS-DOS style path detected: C:\cygwin\home\bike\HADOOP~1.2\/build/native
  Preferred POSIX equivalent is: /home/bike/HADOOP~1.2/build/native
  CYGWIN environment variable option "nodosfilewarning" turns off this
warning.
  Consult the user's guide for more details about POSIX paths:
    http://cygwin.com/cygwin-ug-net/using.html#using-pathnames
10/01/22 17:40:59 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = bike-3da7ab8ed1/147.46.166.184
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 0.19.2
STARTUP_MSG:   build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.19 -r
789657; compiled by 'root' on Tue Jun 30 12:40:50 EDT 2009
************************************************************/
Re-format filesystem in \tmp\hadoop-bike\dfs\name ? (Y or N)



---------------------------

Just wondering, is it possible that the error is because namenode format it
according to windows namespace so hadoop cannot find /work/temp folder?

would like to hear some feedback from the community

thanks
H. Kumar



2010/1/22 Harshit Kumar <hkumar.arora@gmail.com>

> Hi
>
> I will appreciate if someone can help resolve the following error. i am
> going nuts over this.
>
> $ bin/hadoop jar sparqlcloud.jar org.bike.MainClass
> 10/01/22 16:38:27 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
> 10/01/22 16:38:27 INFO mapred.FileInputFormat: Total input paths to process
> : 1
> 10/01/22 16:38:28 INFO mapred.JobClient: Running job: job_201001221636_0001
> 10/01/22 16:38:29 INFO mapred.JobClient:  map 0% reduce 0%
> 10/01/22 16:38:37 INFO mapred.JobClient: Task Id :
> attempt_201001221636_0001_m_000003_0, Status : FAILED
> *java.io.FileNotFoundException: File
> C:/tmp/hadoop-SYSTEM/mapred/local/taskTracker/jobcache/job_201001221636_0001/attempt_201001221636_0001_m_000003_0/work/tmp
> does not exist.*
>         at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:420)
>         at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:244)
>         at
> org.apache.hadoop.mapred.TaskRunner.setupWorkDir(TaskRunner.java:520)
>         at org.apache.hadoop.mapred.Child.main(Child.java:143)
>
>
> ===============
>  working under CYGWIN and using hadoop-0.19.2
>
> I have tried the following solutions,
> 1. bin/hadoop fsck / - output - The file system under / is healthy
> 2. formatted the hdfs bin/hadoop namenode -format - but error does not go.
> 3. change the value tmp.dir property in conf/hadoop-default.xml to some
> other directory but still the error doesnot go.
>
> I will appreciate if some one can help resolve the error.
>
>
> Thanks
> H. Kumar
>
>

--0016e64079aefb7b91047dbce3fa--

From general-return-965-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 22 19:30:23 2010
Return-Path: <general-return-965-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54467 invoked from network); 22 Jan 2010 19:30:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jan 2010 19:30:23 -0000
Received: (qmail 72347 invoked by uid 500); 22 Jan 2010 19:30:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72255 invoked by uid 500); 22 Jan 2010 19:30:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72245 invoked by uid 99); 22 Jan 2010 19:30:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Jan 2010 19:30:22 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of huan.liu@accenture.com designates 170.252.248.70 as permitted sender)
Received: from [170.252.248.70] (HELO amrmr1001.accenture.com) (170.252.248.70)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Jan 2010 19:30:11 +0000
Received: from AMRXV1003.dir.svc.accenture.com (amrxv1003.dir.svc.accenture.com [10.10.160.63])
	by amrmr1001.accenture.com (8.13.8/8.13.8) with ESMTP id o0MJTExu012360
	for <general@hadoop.apache.org>; Fri, 22 Jan 2010 13:29:50 -0600 (CST)
Received: from AMRXH3002.dir.svc.accenture.com ([10.63.34.24]) by AMRXV1003.dir.svc.accenture.com with Microsoft SMTPSVC(6.0.3790.1830); Fri, 22 Jan 2010 13:28:02 -0600
Received: from AMRXM3124.dir.svc.accenture.com ([10.63.34.16]) by AMRXH3002.dir.svc.accenture.com ([10.63.34.24]) with mapi; Fri, 22 Jan 2010 14:28:02 -0500
From: <huan.liu@accenture.com>
Content-Class: urn:content-classes:message
Importance: normal
Priority: normal
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.3790.3168
To: <general@hadoop.apache.org>
Date: Fri, 22 Jan 2010 14:27:51 -0500
Subject: RE: Google Patent on MapReduce
Thread-Topic: Google Patent on MapReduce
thread-index: AcqaHW612ILTQXq4T5uL4UKdQqCpPgAACPyBAF6zPgA=
Message-ID: <BD19120596A4D041940290E71BD0097E6503F59883@AMRXM3124.dir.svc.accenture.com>
References: <CA7285AEA035E041BCF11DBDA646D000048183A9@MSGMROCLN2WIN.DMN1.FMR.COM> <2c44f6691001191243t418989cav28e1d072e59e6e21@mail.gmail.com> <1eabbac31001201207s7e2c67d7w99f9598c8868f4df@mail.gmail.com> <f470f68e1001201406y17a985aeu36fa5b7b23a05ccb@mail.gmail.com>,<f470f68e1001201410y4fd86172j53b11bba884e39e6@mail.gmail.com> <ADF94D8555C7A246B86A633685E0178AB867BBD1C4@planck.ka.sara.nl>
In-Reply-To: <ADF94D8555C7A246B86A633685E0178AB867BBD1C4@planck.ka.sara.nl>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
x-ems-proccessed: vrAiQuOOcsXVFhS7ec6D4A==
x-ems-stamp: 09adAobbHVBoiwi/p+E32w==
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginalArrivalTime: 22 Jan 2010 19:28:02.0436 (UTC) FILETIME=[037DD440:01CA9B99]
X-Virus-Checked: Checked by ClamAV on apache.org

The patent covers only the implementation, but not the programming =
model. If Google enforces, Hadoop could change its implementation to get =
around. The independent claims 1 and 9 have too many "wherein" clauses =
that, if Hadoop just changes one aspect, it can get around the claim =
language.=20

http://huanliu.wordpress.com/2010/01/22/googles-mapreduce-patent-and-its-=
impact-on-hadoop-and-cloud-mapreduce/

-Huan

> -----Original Message-----
> From: Evert Lammerts [mailto:Evert.Lammerts@sara.nl]
> Sent: Wednesday, January 20, 2010 2:12 PM
> To: general@hadoop.apache.org
> Subject: RE: Google Patent on MapReduce
>=20
> I tend to second that - nothing to worry about:
> http://gigaom.com/2010/01/19/why-hadoop-users-shouldnt-fear-googles-
> new-mapreduce-patent/.
>=20
> Evert Lammerts
> ________________________________________
> From: Robert Burrell Donkin [robertburrelldonkin@gmail.com]
> Sent: Wednesday, January 20, 2010 11:10 PM
> To: general@hadoop.apache.org
> Subject: Re: Google Patent on MapReduce
>=20
> On Wed, Jan 20, 2010 at 10:06 PM, Robert Burrell Donkin
> <robertburrelldonkin@gmail.com> wrote:
> > On Wed, Jan 20, 2010 at 8:07 PM, Something Something
> > <mailinglists19@gmail.com> wrote:
> >> Here's some more info:
> >>
> >> http://arstechnica.com/open-source/news/2010/01/googles-mapreduce-
> patent-what-does-it-mean-for-hadoop.ars
> >>
> >> It will be nice to get some official announcement from Apache =
Hadoop
> team
> >> regarding future of Hadoop.
> >>
> >> - A very scared Hadoop Fan :(
> >
> > don't worry
> >
> > every substantial application has a very large probability of patent
> > infringement
>=20
> basically, no company that uses software can enforce a software patent
> against any tech major due to mutually-assured destruction through
> software patent armageddon. lots of tech majors use hadoop ergo google
> would be MAD to use the patent.
>=20
> - robert



This message is for the designated recipient only and may contain =
privileged, proprietary, or otherwise private information.  If you have =
received it in error, please notify the sender immediately and delete =
the original.  Any other use of the email by you is prohibited.

From general-return-966-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 22 19:32:03 2010
Return-Path: <general-return-966-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54909 invoked from network); 22 Jan 2010 19:32:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jan 2010 19:32:03 -0000
Received: (qmail 74508 invoked by uid 500); 22 Jan 2010 19:32:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74446 invoked by uid 500); 22 Jan 2010 19:32:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74436 invoked by uid 99); 22 Jan 2010 19:32:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Jan 2010 19:32:02 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [67.59.59.114] (HELO trironport1.altair.com) (67.59.59.114)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Jan 2010 19:31:51 +0000
X-IronPort-AV: E=Sophos;i="4.49,325,1262581200"; 
   d="scan'208";a="10005944"
Received: from unknown (HELO TR-EXCH07.prog.altair.com) ([204.235.24.175])
  by trironport1.altair.com with ESMTP; 22 Jan 2010 14:31:30 -0500
Received: from TR-EXCH07.prog.altair.com ([204.235.24.175]) by
 TR-EXCH07.prog.altair.com ([204.235.24.175]) with mapi; Fri, 22 Jan 2010
 14:32:13 -0500
From: Lori Ann Martin <lmartin@altair.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 22 Jan 2010 14:31:27 -0500
Subject: RE: Google Patent on MapReduce
Thread-Topic: Google Patent on MapReduce
Thread-Index: AcqaHW612ILTQXq4T5uL4UKdQqCpPgAACPyBAF6zPgAAAEYNEA==
Message-ID: <7232A3A7C53F634B8E91E65BC7833C5112D153D857@TR-EXCH07.prog.altair.com>
References: <CA7285AEA035E041BCF11DBDA646D000048183A9@MSGMROCLN2WIN.DMN1.FMR.COM>
 <2c44f6691001191243t418989cav28e1d072e59e6e21@mail.gmail.com>
 <1eabbac31001201207s7e2c67d7w99f9598c8868f4df@mail.gmail.com>
 <f470f68e1001201406y17a985aeu36fa5b7b23a05ccb@mail.gmail.com>,<f470f68e1001201410y4fd86172j53b11bba884e39e6@mail.gmail.com>
 <ADF94D8555C7A246B86A633685E0178AB867BBD1C4@planck.ka.sara.nl>
 <BD19120596A4D041940290E71BD0097E6503F59883@AMRXM3124.dir.svc.accenture.com>
In-Reply-To: <BD19120596A4D041940290E71BD0097E6503F59883@AMRXM3124.dir.svc.accenture.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Are you in Los Angeles?

-----Original Message-----
From: huan.liu@accenture.com [mailto:huan.liu@accenture.com]=20
Sent: Friday, January 22, 2010 11:28 AM
To: general@hadoop.apache.org
Subject: RE: Google Patent on MapReduce

The patent covers only the implementation, but not the programming model. I=
f Google enforces, Hadoop could change its implementation to get around. Th=
e independent claims 1 and 9 have too many "wherein" clauses that, if Hadoo=
p just changes one aspect, it can get around the claim language.=20

http://huanliu.wordpress.com/2010/01/22/googles-mapreduce-patent-and-its-im=
pact-on-hadoop-and-cloud-mapreduce/

-Huan

> -----Original Message-----
> From: Evert Lammerts [mailto:Evert.Lammerts@sara.nl]
> Sent: Wednesday, January 20, 2010 2:12 PM
> To: general@hadoop.apache.org
> Subject: RE: Google Patent on MapReduce
>=20
> I tend to second that - nothing to worry about:
> http://gigaom.com/2010/01/19/why-hadoop-users-shouldnt-fear-googles-
> new-mapreduce-patent/.
>=20
> Evert Lammerts
> ________________________________________
> From: Robert Burrell Donkin [robertburrelldonkin@gmail.com]
> Sent: Wednesday, January 20, 2010 11:10 PM
> To: general@hadoop.apache.org
> Subject: Re: Google Patent on MapReduce
>=20
> On Wed, Jan 20, 2010 at 10:06 PM, Robert Burrell Donkin
> <robertburrelldonkin@gmail.com> wrote:
> > On Wed, Jan 20, 2010 at 8:07 PM, Something Something
> > <mailinglists19@gmail.com> wrote:
> >> Here's some more info:
> >>
> >> http://arstechnica.com/open-source/news/2010/01/googles-mapreduce-
> patent-what-does-it-mean-for-hadoop.ars
> >>
> >> It will be nice to get some official announcement from Apache Hadoop
> team
> >> regarding future of Hadoop.
> >>
> >> - A very scared Hadoop Fan :(
> >
> > don't worry
> >
> > every substantial application has a very large probability of patent
> > infringement
>=20
> basically, no company that uses software can enforce a software patent
> against any tech major due to mutually-assured destruction through
> software patent armageddon. lots of tech majors use hadoop ergo google
> would be MAD to use the patent.
>=20
> - robert



This message is for the designated recipient only and may contain privilege=
d, proprietary, or otherwise private information.  If you have received it =
in error, please notify the sender immediately and delete the original.  An=
y other use of the email by you is prohibited.

From general-return-967-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 22 21:26:42 2010
Return-Path: <general-return-967-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 86751 invoked from network); 22 Jan 2010 21:26:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jan 2010 21:26:42 -0000
Received: (qmail 1887 invoked by uid 500); 22 Jan 2010 21:26:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1817 invoked by uid 500); 22 Jan 2010 21:26:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1807 invoked by uid 99); 22 Jan 2010 21:26:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Jan 2010 21:26:40 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [195.232.224.70] (HELO mailout01.vodafone.com) (195.232.224.70)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Jan 2010 21:26:30 +0000
Received: from mailint01 (localhost [127.0.0.1])
	by mailout01 (Postfix) with ESMTP id 41674432A
	for <general@hadoop.apache.org>; Fri, 22 Jan 2010 22:26:07 +0100 (CET)
Received: from avoexs04.internal.vodafone.com (avoexs04.dc-ratingen.de [145.230.4.198])
	(using TLSv1 with cipher RC4-MD5 (128/128 bits))
	(No client certificate requested)
	by mailint01 (Postfix) with ESMTPS id 32765428D
	for <general@hadoop.apache.org>; Fri, 22 Jan 2010 22:26:07 +0100 (CET)
Received: from VF-MBX13.internal.vodafone.com ([145.230.5.25]) by avoexs04.internal.vodafone.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Fri, 22 Jan 2010 22:26:08 +0100
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/mixed;
	boundary="----_=_NextPart_001_01CA9BA9.834D7604"
Subject: RE: Google Patent on MapReduce
Date: Fri, 22 Jan 2010 22:25:53 +0100
Message-ID: <219D8244D980254ABF28AB469AD4E98F03469FDA@VF-MBX13.internal.vodafone.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: <219D8244D980254ABF28AB469AD4E98F03469FDA@VF-MBX13.internal.vodafone.com>
Thread-Topic: Google Patent on MapReduce
Thread-Index: AcqaHW612ILTQXq4T5uL4UKdQqCpPgAACPyBAF6zPgAAAEYNEAAEAKf3
References: <CA7285AEA035E041BCF11DBDA646D000048183A9@MSGMROCLN2WIN.DMN1.FMR.COM> <2c44f6691001191243t418989cav28e1d072e59e6e21@mail.gmail.com> <1eabbac31001201207s7e2c67d7w99f9598c8868f4df@mail.gmail.com> <f470f68e1001201406y17a985aeu36fa5b7b23a05ccb@mail.gmail.com>,<f470f68e1001201410y4fd86172j53b11bba884e39e6@mail.gmail.com> <ADF94D8555C7A246B86A633685E0178AB867BBD1C4@planck.ka.sara.nl> <BD19120596A4D041940290E71BD0097E6503F59883@AMRXM3124.dir.svc.accenture.com> <7232A3A7C53F634B8E91E65BC7833C5112D153D857@TR-EXCH07.prog.altair.com>
From: "Gibbon, Robert, VF-Group" <Robert.Gibbon@vodafone.com>
To: <general@hadoop.apache.org>
X-OriginalArrivalTime: 22 Jan 2010 21:26:08.0638 (UTC) FILETIME=[833271E0:01CA9BA9]

------_=_NextPart_001_01CA9BA9.834D7604
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


Software patents are not recognised nor enforcable in European Union =
member states (UK, France, Germany, Belgium etc.). The Apache license =
v.2.0 states:

"This License shall be governed by the law of the jurisdiction specified =
in a notice contained within the Original Software..."

Do you know which jurisdiction you chose for the Hadoop MapReduce =
implementation - seems to me to be relevant? I don't find a reference to =
a jurisdiction in release 0.20.1?

-----Original Message-----
From: Lori Ann Martin [mailto:lmartin@altair.com]
Sent: Fri 1/22/2010 8:31 PM
To: general@hadoop.apache.org
Subject: RE: Google Patent on MapReduce
=20
Are you in Los Angeles?

-----Original Message-----
From: huan.liu@accenture.com [mailto:huan.liu@accenture.com]=20
Sent: Friday, January 22, 2010 11:28 AM
To: general@hadoop.apache.org
Subject: RE: Google Patent on MapReduce

The patent covers only the implementation, but not the programming =
model. If Google enforces, Hadoop could change its implementation to get =
around. The independent claims 1 and 9 have too many "wherein" clauses =
that, if Hadoop just changes one aspect, it can get around the claim =
language.=20

http://huanliu.wordpress.com/2010/01/22/googles-mapreduce-patent-and-its-=
impact-on-hadoop-and-cloud-mapreduce/

-Huan

> -----Original Message-----
> From: Evert Lammerts [mailto:Evert.Lammerts@sara.nl]
> Sent: Wednesday, January 20, 2010 2:12 PM
> To: general@hadoop.apache.org
> Subject: RE: Google Patent on MapReduce
>=20
> I tend to second that - nothing to worry about:
> http://gigaom.com/2010/01/19/why-hadoop-users-shouldnt-fear-googles-
> new-mapreduce-patent/.
>=20
> Evert Lammerts
> ________________________________________
> From: Robert Burrell Donkin [robertburrelldonkin@gmail.com]
> Sent: Wednesday, January 20, 2010 11:10 PM
> To: general@hadoop.apache.org
> Subject: Re: Google Patent on MapReduce
>=20
> On Wed, Jan 20, 2010 at 10:06 PM, Robert Burrell Donkin
> <robertburrelldonkin@gmail.com> wrote:
> > On Wed, Jan 20, 2010 at 8:07 PM, Something Something
> > <mailinglists19@gmail.com> wrote:
> >> Here's some more info:
> >>
> >> http://arstechnica.com/open-source/news/2010/01/googles-mapreduce-
> patent-what-does-it-mean-for-hadoop.ars
> >>
> >> It will be nice to get some official announcement from Apache =
Hadoop
> team
> >> regarding future of Hadoop.
> >>
> >> - A very scared Hadoop Fan :(
> >
> > don't worry
> >
> > every substantial application has a very large probability of patent
> > infringement
>=20
> basically, no company that uses software can enforce a software patent
> against any tech major due to mutually-assured destruction through
> software patent armageddon. lots of tech majors use hadoop ergo google
> would be MAD to use the patent.
>=20
> - robert



This message is for the designated recipient only and may contain =
privileged, proprietary, or otherwise private information.  If you have =
received it in error, please notify the sender immediately and delete =
the original.  Any other use of the email by you is prohibited.


------_=_NextPart_001_01CA9BA9.834D7604--

From general-return-968-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jan 24 22:35:33 2010
Return-Path: <general-return-968-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 52042 invoked from network); 24 Jan 2010 22:35:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jan 2010 22:35:33 -0000
Received: (qmail 12618 invoked by uid 500); 24 Jan 2010 22:35:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12552 invoked by uid 500); 24 Jan 2010 22:35:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12542 invoked by uid 99); 24 Jan 2010 22:35:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 24 Jan 2010 22:35:30 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of gautam.singaraju@gmail.com designates 209.85.211.187 as permitted sender)
Received: from [209.85.211.187] (HELO mail-yw0-f187.google.com) (209.85.211.187)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 24 Jan 2010 22:35:24 +0000
Received: by ywh17 with SMTP id 17so2584483ywh.2
        for <general@hadoop.apache.org>; Sun, 24 Jan 2010 14:35:03 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=qoYXyDRk+Z5B2deuUhMimwAhXWzzNmWMi69aBXa2IFA=;
        b=hgCD0Vy0PFeCRpJSwqlKxTCQC+sQRcvIg+47IS+cDumYi5OynZsIclfCD0iR3d9JRp
         dFC5Z8WWWHKCHOSdMLxsBWE+32Ar03gU4nfWfVyLCvKZlXGH/J0Jr64JS3tbbZ3Kvdw/
         7rgS210nd2k9oQvbCRFDY4DHNRbTgTwjwBZfc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=U9OfPNjdcJy2QIr21FEEgKBHanUaGI32J6x/gkxdwYOhLav6SIDiZzoaTSRCR/IHgd
         MovE2qWR/qCpqFNFq80l/ejn8mz5xY4gIosVqkVCt3ermFluB8qNp8f5EHS1/S7VEBcL
         jpaVSsKuJ6jnAZsZOxx8OUPRhEYyq8Km8PDoY=
MIME-Version: 1.0
Received: by 10.150.130.36 with SMTP id c36mr7513615ybd.34.1264372499547; Sun, 
	24 Jan 2010 14:34:59 -0800 (PST)
In-Reply-To: <17e273101001191138w4e358d1cja26bc20793d30e26@mail.gmail.com>
References: <17e273101001191138w4e358d1cja26bc20793d30e26@mail.gmail.com>
Date: Sun, 24 Jan 2010 17:34:59 -0500
Message-ID: <f8b2ccd41001241434p6e84aec5od25ad7211d023fa0@mail.gmail.com>
Subject: Re: splittable encrypted compressed files
From: Gautam Singaraju <gautam.singaraju@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

A encryption/decryption mechanism requires the secret key to be
securely distributed. As far as moving data on the wire is concerned,
SSH is already being used. I am wondering about the performance hit
when encryption and then again decryption is applied to the data and
what value that might add.

---
Gautam



On Tue, Jan 19, 2010 at 2:38 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> Hi,
> I experimented with hadoop-lzo and liked it.
>
> Our requirement is changing and we may need to encrypt data files. Does
> anyone know of splittable encrypted compression scheme that map task can use
> ?
>
> Thanks
>

From general-return-969-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jan 24 22:49:36 2010
Return-Path: <general-return-969-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53677 invoked from network); 24 Jan 2010 22:49:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jan 2010 22:49:36 -0000
Received: (qmail 16727 invoked by uid 500); 24 Jan 2010 22:49:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16643 invoked by uid 500); 24 Jan 2010 22:49:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16633 invoked by uid 99); 24 Jan 2010 22:49:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 24 Jan 2010 22:49:34 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com designates 209.85.222.175 as permitted sender)
Received: from [209.85.222.175] (HELO mail-pz0-f175.google.com) (209.85.222.175)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 24 Jan 2010 22:49:28 +0000
Received: by pzk5 with SMTP id 5so393700pzk.30
        for <general@hadoop.apache.org>; Sun, 24 Jan 2010 14:49:07 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=ez2mQ3mu+ULUMFKCivmKiCJb1qTVLpvyQiwz0r4wWCk=;
        b=sxP/IHABgFAYTl8erk668z2f24UbWQE1SUCy5jdlYhDlyegn9EUlqs0/54TwM/ACzd
         qQfOHO9wtxxZVFbeUaLzYvyx3xze8ZapmFKa65KVpZmVsKxVAiq1ePAcJw6WgvPvZVaY
         h2g3jybGvRDxoCAtWZCYIuqHX8+qvSZhFWU4U=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=j8YXS8vyYDDnkiSYSs6PEaxsg4fSPL4wwuY+FxaUtHDKjKn22aVSB7lK1UTNh9KvYJ
         sunMslFktrHoZes9f69E/L+MxIPzi1Z7eNFrKfvQKF4wICBX3NNE/o6RJFvdwezinJgc
         RHD21VDlFNhuMnH6Izcs3+HWElsSSJv+A5Ysk=
MIME-Version: 1.0
Received: by 10.142.9.10 with SMTP id 10mr4008086wfi.177.1264373347652; Sun, 
	24 Jan 2010 14:49:07 -0800 (PST)
In-Reply-To: <f8b2ccd41001241434p6e84aec5od25ad7211d023fa0@mail.gmail.com>
References: <17e273101001191138w4e358d1cja26bc20793d30e26@mail.gmail.com>
	 <f8b2ccd41001241434p6e84aec5od25ad7211d023fa0@mail.gmail.com>
Date: Sun, 24 Jan 2010 14:49:07 -0800
Message-ID: <17e273101001241449u37fe9607ya1576324a2cb5f09@mail.gmail.com>
Subject: Re: splittable encrypted compressed files
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502b49840158c047df0dca5

--00504502b49840158c047df0dca5
Content-Type: text/plain; charset=ISO-8859-1

A bit more background: the encryption is required by third party. Data
coming from third party may come in encrypted .gz format.

Cheers

On Sun, Jan 24, 2010 at 2:34 PM, Gautam Singaraju <
gautam.singaraju@gmail.com> wrote:

> A encryption/decryption mechanism requires the secret key to be
> securely distributed. As far as moving data on the wire is concerned,
> SSH is already being used. I am wondering about the performance hit
> when encryption and then again decryption is applied to the data and
> what value that might add.
>
> ---
> Gautam
>
>
>
> On Tue, Jan 19, 2010 at 2:38 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> > Hi,
> > I experimented with hadoop-lzo and liked it.
> >
> > Our requirement is changing and we may need to encrypt data files. Does
> > anyone know of splittable encrypted compression scheme that map task can
> use
> > ?
> >
> > Thanks
> >
>

--00504502b49840158c047df0dca5--

From general-return-970-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 25 06:34:09 2010
Return-Path: <general-return-970-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85346 invoked from network); 25 Jan 2010 06:34:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Jan 2010 06:34:09 -0000
Received: (qmail 51066 invoked by uid 500); 25 Jan 2010 06:34:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50915 invoked by uid 500); 25 Jan 2010 06:34:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50902 invoked by uid 99); 25 Jan 2010 06:34:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Jan 2010 06:34:08 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=HTML_IMAGE_ONLY_24,HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of naveenkumarp@huawei.com designates 119.145.14.64 as permitted sender)
Received: from [119.145.14.64] (HELO szxga01-in.huawei.com) (119.145.14.64)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Jan 2010 06:33:59 +0000
Received: from huawei.com (szxga01-in [172.24.2.3])
 by szxga01-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug
 8 2006)) with ESMTP id <0KWS00BTHI816C@szxga01-in.huawei.com> for
 general@hadoop.apache.org; Mon, 25 Jan 2010 14:33:38 +0800 (CST)
Received: from huawei.com ([172.24.2.119])
 by szxga01-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug
 8 2006)) with ESMTP id <0KWS00MBWI81MN@szxga01-in.huawei.com> for
 general@hadoop.apache.org; Mon, 25 Jan 2010 14:33:37 +0800 (CST)
Received: from BLRNSHTIPL6NC ([10.18.1.36])
 by szxml04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug
 8 2006)) with ESMTPA id <0KWS00LX4I7W0B@szxml04-in.huawei.com> for
 general@hadoop.apache.org; Mon, 25 Jan 2010 14:33:37 +0800 (CST)
Date: Mon, 25 Jan 2010 12:03:32 +0530
From: Naveen Kumar Prasad <naveenkumarp@huawei.com>
Subject: set how much CPU to be utilised by a MapReduce job
To: general@hadoop.apache.org
Reply-to: naveenkumarp@huawei.com
Message-id: <001801ca9d88$5371f210$2401120a@china.huawei.com>
Organization: Htipl
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.3790.3168
X-Mailer: Microsoft Office Outlook 11
Content-type: multipart/alternative;
 boundary="Boundary_(ID_iuwFz3VudlX6PZbkHOlUsg)"
Thread-index: AcqdiFBj+hDr2AfOQVCJLWla69wqZg==

--Boundary_(ID_iuwFz3VudlX6PZbkHOlUsg)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT

Hi All,
 
If many jobs are running concurrently in Hadoop, how can we set CPU usage
for individual tasks.
 
The purpose here is to avoid one job affecting the main job's execution.
 
Regards,
Naveen Kumar
HUAWEI TECHNOLOGIES CO.,LTD.  huawei_logo
<outbind://24-000000004ED456449CB8724C859F40771D413A7584132000/cid:919325906
@21052009-015A> 


Address: Huawei Industrial Base
Bantian Longgang
Shenzhen 518129, P.R.China
www.huawei.com
----------------------------------------------------------------------------
-------------------------------------
This e-mail and its attachments contain confidential information from
HUAWEI, which is intended only for the person or entity whose address is
listed above. Any use of the information contained herein in any way
(including, but not limited to, total or partial disclosure, reproduction,
or dissemination) by persons other than the intended recipient(s) is
prohibited. If you receive this e-mail in error, please notify the sender by
phone or email immediately and delete it!
 

--Boundary_(ID_iuwFz3VudlX6PZbkHOlUsg)--

From general-return-971-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 25 07:14:40 2010
Return-Path: <general-return-971-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98895 invoked from network); 25 Jan 2010 07:14:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Jan 2010 07:14:40 -0000
Received: (qmail 77565 invoked by uid 500); 25 Jan 2010 07:14:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77465 invoked by uid 500); 25 Jan 2010 07:14:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77455 invoked by uid 99); 25 Jan 2010 07:14:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Jan 2010 07:14:38 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Jan 2010 07:14:28 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:User-Agent:Date:Subject:From:To:
   Message-ID:Thread-Topic:Thread-Index:In-Reply-To:
   Mime-version:Content-type:Content-transfer-encoding;
  b=r7y+XZcE5OsZr21Fb7fLLCLCYe4cbO75SGCE4eZb3BfsTd8BheMHMioN
   ST+uyqK9mfpAhVjcF4V91YbhDGEVaaxie7ByfYI6HeiRkHhA2A2tu6K3T
   oq19wfhGM/nYJRX;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1264403668; x=1295939668;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20set=20how=20much=20CPU=20to=20be=20util
   ised=20by=20a=20MapReduce=20job|Date:=20Sun,=2024=20Jan
   =202010=2023:14:06=20-0800|Message-ID:=20<C78284BE.27AB%a
   wittenauer@linkedin.com>|To:=20<general@hadoop.apache.org
   >|Mime-version:=201.0|Content-transfer-encoding:=207bit
   |In-Reply-To:=20<001801ca9d88$5371f210$2401120a@china.hua
   wei.com>;
  bh=OFBXWesa4kSR1ypLFGx/ZDsNDHteOlivIiVKiadBDFY=;
  b=ZPaEPXKx2ZVihfNZN82Vd5R89a2QFjbahjL3nR0pq5s1/pvJrWplXlZa
   CKxD3w9whYjnzu4mkrkl5JHUG7PaggI88FFxpv4mu4MrCgCuoRE0mBxMh
   j0ycoSsgdRK1LV2;
X-IronPort-AV: E=Sophos;i="4.49,337,1262592000"; 
   d="scan'208";a="10523408"
Received: from 172.18.36.37 ([172.18.36.37]) by CORP-MAIL.linkedin.biz ([172.18.46.135]) via Exchange Front-End Server mail-access.linkedin.biz ([172.18.46.133]) with Microsoft Exchange Server HTTP-DAV ;
 Mon, 25 Jan 2010 07:14:07 +0000
User-Agent: Microsoft-Entourage/12.10.0.080409
Date: Sun, 24 Jan 2010 23:14:06 -0800
Subject: Re: set how much CPU to be utilised by a MapReduce job
From: Allen Wittenauer <awittenauer@linkedin.com>
To: <general@hadoop.apache.org>
Message-ID: <C78284BE.27AB%awittenauer@linkedin.com>
Thread-Topic: set how much CPU to be utilised by a MapReduce job
Thread-Index: AcqdiFBj+hDr2AfOQVCJLWla69wqZgABaqU/
In-Reply-To: <001801ca9d88$5371f210$2401120a@china.huawei.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit

On 1/24/10 10:33 PM, "Naveen Kumar Prasad" <naveenkumarp@huawei.com> wrote:
> If many jobs are running concurrently in Hadoop, how can we set CPU usage
> for individual tasks.

That functionality does not exist.


From general-return-972-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 25 08:23:57 2010
Return-Path: <general-return-972-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 10088 invoked from network); 25 Jan 2010 08:23:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Jan 2010 08:23:57 -0000
Received: (qmail 18761 invoked by uid 500); 25 Jan 2010 08:23:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18684 invoked by uid 500); 25 Jan 2010 08:23:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18674 invoked by uid 99); 25 Jan 2010 08:23:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Jan 2010 08:23:56 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of naveenkumarp@huawei.com designates 119.145.14.66 as permitted sender)
Received: from [119.145.14.66] (HELO szxga03-in.huawei.com) (119.145.14.66)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Jan 2010 08:23:47 +0000
Received: from huawei.com (szxga03-in [172.24.2.9])
 by szxga03-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug
 8 2006)) with ESMTP id <0KWS009EDNAZ08@szxga03-in.huawei.com> for
 general@hadoop.apache.org; Mon, 25 Jan 2010 16:23:23 +0800 (CST)
Received: from huawei.com ([172.24.2.119])
 by szxga03-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug
 8 2006)) with ESMTP id <0KWS00HT3NAZSU@szxga03-in.huawei.com> for
 general@hadoop.apache.org; Mon, 25 Jan 2010 16:23:23 +0800 (CST)
Received: from BLRNSHTIPL6NC ([10.18.1.36])
 by szxml04-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug
 8 2006)) with ESMTPA id <0KWS003EDNAYJ5@szxml04-in.huawei.com> for
 general@hadoop.apache.org; Mon, 25 Jan 2010 16:23:23 +0800 (CST)
Date: Mon, 25 Jan 2010 13:53:21 +0530
From: Naveen Kumar Prasad <naveenkumarp@huawei.com>
Subject: RE: set how much CPU to be utilised by a MapReduce job
In-reply-to: <C78284BE.27AB%awittenauer@linkedin.com>
To: general@hadoop.apache.org
Reply-to: naveenkumarp@huawei.com
Message-id: <002001ca9d97$a8ba13b0$2401120a@china.huawei.com>
Organization: Htipl
MIME-version: 1.0
X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.3790.3168
X-Mailer: Microsoft Office Outlook 11
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT
Thread-index: AcqdiFBj+hDr2AfOQVCJLWla69wqZgABaqU/AAJPoCA=


This functionality may not be readily available with Hadoop.

But it would be appreciable if anyone can help me in understanding how to go
about developing this feature. 


Regards,
Naveen Kumar
HUAWEI TECHNOLOGIES CO.,LTD.  huawei_logo


Address: Huawei Industrial Base
Bantian Longgang
Shenzhen 518129, P.R.China
www.huawei.com
----------------------------------------------------------------------------
-------------------------------------
This e-mail and its attachments contain confidential information from
HUAWEI, which is intended only for the person or entity whose address is
listed above. Any use of the information contained herein in any way
(including, but not limited to, total or partial disclosure, reproduction,
or dissemination) by persons other than the intended recipient(s) is
prohibited. If you receive this e-mail in error, please notify the sender by
phone or email immediately and delete it!

-----Original Message-----
From: Allen Wittenauer [mailto:awittenauer@linkedin.com] 
Sent: Monday, January 25, 2010 12:44 PM
To: general@hadoop.apache.org
Subject: Re: set how much CPU to be utilised by a MapReduce job

On 1/24/10 10:33 PM, "Naveen Kumar Prasad" <naveenkumarp@huawei.com> wrote:
> If many jobs are running concurrently in Hadoop, how can we set CPU 
> usage for individual tasks.

That functionality does not exist.


From general-return-973-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 25 09:13:52 2010
Return-Path: <general-return-973-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21742 invoked from network); 25 Jan 2010 09:13:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Jan 2010 09:13:52 -0000
Received: (qmail 61710 invoked by uid 500); 25 Jan 2010 09:13:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61630 invoked by uid 500); 25 Jan 2010 09:13:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61620 invoked by uid 99); 25 Jan 2010 09:13:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Jan 2010 09:13:51 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ryanobjc@gmail.com designates 209.85.222.195 as permitted sender)
Received: from [209.85.222.195] (HELO mail-pz0-f195.google.com) (209.85.222.195)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Jan 2010 09:13:41 +0000
Received: by pzk33 with SMTP id 33so2609314pzk.2
        for <general@hadoop.apache.org>; Mon, 25 Jan 2010 01:13:19 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=LFbbyInGPQ212PZx5TUEEK1WARpG6SQH5rpzWIuaLl8=;
        b=kDcy8BKwjNx4FlGQD0G6KG1iRI9vY5nTDkZlLCpm9gkzNr38ebjh1p9MXPxOgc4ikG
         dJFlu4mGoGRw1eFR4PaDtKpJQT3oNdyyUFiTiTeQF8xZERtQBVwSTp+Zwk8oBGU8eGZH
         aJ79PMO80BwWBfe6698yWJBNZX+CrUxmjZRM4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=pK65GKvP9MFfj7GIl8o04X0+m/vfIC96bOCG1nMA8mL2HkNgVTC1VxJn4HHgnPJ8ou
         6Ciu75GEA07Rzv6iSvdtofu0Nmb4UfuX4rk/cb2l3CJ+OLpvld6Pv6exxbXIdkhu7zDR
         sQDCQTVueQDUPOJ1jOrDblHMHoIhRQf3eNjTw=
MIME-Version: 1.0
Received: by 10.115.65.6 with SMTP id s6mr4362884wak.53.1264410799473; Mon, 25 
	Jan 2010 01:13:19 -0800 (PST)
In-Reply-To: <78568af11001250112v3b610478w49703d72306d4a2f@mail.gmail.com>
References: <C78284BE.27AB%awittenauer@linkedin.com>
	 <002001ca9d97$a8ba13b0$2401120a@china.huawei.com>
	 <78568af11001250112v3b610478w49703d72306d4a2f@mail.gmail.com>
Date: Mon, 25 Jan 2010 01:13:19 -0800
Message-ID: <78568af11001250113k41f303ffq8d0eb4ea5bc33523@mail.gmail.com>
Subject: Re: RE: set how much CPU to be utilised by a MapReduce job
From: Ryan Rawson <ryanobjc@gmail.com>
To: general@hadoop.apache.org, naveenkumarp@huawei.com
Content-Type: multipart/alternative; boundary=0016e64ce5848d8791047df994a7
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e64ce5848d8791047df994a7
Content-Type: text/plain; charset=ISO-8859-1

The only thing you could do is to have the tasktracker nice the child when
its exceeding its reservation. Aside from that, its hard to limit without
killing a process.

On Jan 25, 2010 12:23 AM, "Naveen Kumar Prasad" <naveenkumarp@huawei.com>
wrote:


This functionality may not be readily available with Hadoop.

But it would be appreciable if anyone can help me in understanding how to go
about developing this feature.

Regards, Naveen Kumar HUAWEI TECHNOLOGIES CO.,LTD. huawei_logo

Address: Huawei Industrial Base Bantian Longgang Shenzhen 518129, P.R.China
www.huawei.com ---------...

-----Original Message----- From: Allen Wittenauer [mailto:
awittenauer@linkedin.com] Sent: Monday, J...

--0016e64ce5848d8791047df994a7--

From general-return-974-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 25 16:39:38 2010
Return-Path: <general-return-974-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78470 invoked from network); 25 Jan 2010 16:39:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Jan 2010 16:39:38 -0000
Received: (qmail 66726 invoked by uid 500); 25 Jan 2010 16:39:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66654 invoked by uid 500); 25 Jan 2010 16:39:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66636 invoked by uid 99); 25 Jan 2010 16:39:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Jan 2010 16:39:37 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Jan 2010 16:39:26 +0000
Received: by pwi20 with SMTP id 20so2551501pwi.29
        for <general@hadoop.apache.org>; Mon, 25 Jan 2010 08:39:05 -0800 (PST)
MIME-Version: 1.0
Received: by 10.142.4.41 with SMTP id 41mr580085wfd.123.1264437544942; Mon, 25 
	Jan 2010 08:39:04 -0800 (PST)
In-Reply-To: <78568af11001250113k41f303ffq8d0eb4ea5bc33523@mail.gmail.com>
References: <C78284BE.27AB%awittenauer@linkedin.com> <002001ca9d97$a8ba13b0$2401120a@china.huawei.com> 
	<78568af11001250112v3b610478w49703d72306d4a2f@mail.gmail.com> 
	<78568af11001250113k41f303ffq8d0eb4ea5bc33523@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 25 Jan 2010 08:38:43 -0800
Message-ID: <45f85f71001250838s35572132jc84b220379549092@mail.gmail.com>
Subject: Re: RE: set how much CPU to be utilised by a MapReduce job
To: general@hadoop.apache.org
Cc: naveenkumarp@huawei.com
Content-Type: multipart/alternative; boundary=00504502b443b4ff64047dffce97
X-Virus-Checked: Checked by ClamAV on apache.org

--00504502b443b4ff64047dffce97
Content-Type: text/plain; charset=ISO-8859-1

If you can require a recent kernel, you could use cgroups:

http://broadcast.oreilly.com/2009/06/manage-your-performance-with-cgroups-and-projects.html

No one has integrated this with hadoop yet as it's still pretty new, and
Hadoop clusters are meant to be run on unshared hardware.

-Todd

On Mon, Jan 25, 2010 at 1:13 AM, Ryan Rawson <ryanobjc@gmail.com> wrote:

> The only thing you could do is to have the tasktracker nice the child when
> its exceeding its reservation. Aside from that, its hard to limit without
> killing a process.
>
> On Jan 25, 2010 12:23 AM, "Naveen Kumar Prasad" <naveenkumarp@huawei.com>
> wrote:
>
>
> This functionality may not be readily available with Hadoop.
>
> But it would be appreciable if anyone can help me in understanding how to
> go
> about developing this feature.
>
> Regards, Naveen Kumar HUAWEI TECHNOLOGIES CO.,LTD. huawei_logo
>
> Address: Huawei Industrial Base Bantian Longgang Shenzhen 518129, P.R.China
> www.huawei.com ---------...
>
> -----Original Message----- From: Allen Wittenauer [mailto:
> awittenauer@linkedin.com] Sent: Monday, J...
>

--00504502b443b4ff64047dffce97--

From general-return-975-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 25 19:57:00 2010
Return-Path: <general-return-975-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69433 invoked from network); 25 Jan 2010 19:57:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Jan 2010 19:57:00 -0000
Received: (qmail 4410 invoked by uid 500); 25 Jan 2010 19:56:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4281 invoked by uid 500); 25 Jan 2010 19:56:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4247 invoked by uid 99); 25 Jan 2010 19:56:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Jan 2010 19:56:56 +0000
X-ASF-Spam-Status: No, hits=2.5 required=10.0
	tests=PLING_QUERY,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Jan 2010 19:56:46 +0000
Received: by pwi20 with SMTP id 20so2704682pwi.29
        for <multiple recipients>; Mon, 25 Jan 2010 11:56:26 -0800 (PST)
MIME-Version: 1.0
Received: by 10.114.248.39 with SMTP id v39mr4833221wah.39.1264449386130; Mon, 
	25 Jan 2010 11:56:26 -0800 (PST)
From: Christophe Bisciglia <christophe@cloudera.com>
Date: Mon, 25 Jan 2010 11:56:06 -0800
Message-ID: <69035571001251156m17c0fcd5k9c0b4ba86fbcae2a@mail.gmail.com>
Subject: Interested in Hadoop Training outside the US? Let us know!
To: general@hadoop.apache.org, common-user@hadoop.apache.org, 
	hive-user@hadoop.apache.org, pig-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

NOTE: Please forward this message to local use groups, especially
those that might be less active on the core Apache mailing lists. Feel
free to translate into local languages.

Hadoop Fans,

Over the next year, you'll see new options for Hadoop training and
certification from Cloudera. One of the first things you'll see will
be live sessions outside the US, tentatively planned for the April /
May time frame.

We've seen strong interest in Hadoop on all of our international
trips, so we'd like to ask for community input as we decide exactly
which cities to visit next. For cities we come to, we'll offer our 3
day developer training + certification, and with sufficient interest,
we may also include a 1 day training + certification program for
system administrators.

If you are interested in attending one or both of these sessions,
please fill out a brief survey (link below). If you're using Hadoop at
work, and it's time to train more of your team, you can let us know
how large of a group you have. Survey responses aren't a commitment to
attend, but we may reach out to respondents before we schedule a
session to get a better understanding of actual attendance.

You can fill out survey here: http://www.surveymonkey.com/s/MKGZHG9

If you have any trouble with the survey, or are interested in a
private training session, please don't hesitate to reach out directly.

Cheers,
Christophe
--
get hadoop: cloudera.com/hadoop
online training: cloudera.com/hadoop-training
blog: cloudera.com/blog
twitter: twitter.com/cloudera

From general-return-976-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 25 23:23:01 2010
Return-Path: <general-return-976-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70171 invoked from network); 25 Jan 2010 23:23:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Jan 2010 23:23:01 -0000
Received: (qmail 63300 invoked by uid 500); 25 Jan 2010 23:23:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63199 invoked by uid 500); 25 Jan 2010 23:22:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63098 invoked by uid 99); 25 Jan 2010 23:22:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Jan 2010 23:22:59 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mailinglists19@gmail.com designates 209.85.219.216 as permitted sender)
Received: from [209.85.219.216] (HELO mail-ew0-f216.google.com) (209.85.219.216)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Jan 2010 23:22:52 +0000
Received: by ewy8 with SMTP id 8so4812668ewy.29
        for <multiple recipients>; Mon, 25 Jan 2010 15:22:31 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=UILOZnKiYISJoYySYqAB7jgRcwmwi2ZjcveCvFOV2LI=;
        b=vUdPlVG56gNwwDE4yZ2BEOwYCyzdi/9RU0j6Hr0P5ZmUEOU1Zv9Le18ey1Wk03NzLI
         iY69jT2CUd2cvebCre2WRzxRVV7xWF/v6w2vhZrFgXhy42GyxMxIygKnuOa27Qyc7gVC
         VWKJK0WyVLfxt3zm7+DRhVGtLr7PbdLgw+bAY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=IVhClKHqw6trpySxwluFPZmsWL2+oKlRJdgWOYDRJNonBP/9A+6OCwUU+9e18bxwJt
         fthkfuT57jnSVJSue+kn+pw3ETev6WdG4+eIRxH+RsKCQXlal1W+QZEZlzCFKFcLqq/5
         FTYKhlwWxRwdwelJ/r2gXvCJa7wiicGSqGSkk=
MIME-Version: 1.0
Received: by 10.216.89.194 with SMTP id c44mr1429533wef.199.1264461750831; 
	Mon, 25 Jan 2010 15:22:30 -0800 (PST)
Date: Mon, 25 Jan 2010 15:22:30 -0800
Message-ID: <1eabbac31001251522s2301f263rb959c768c2ee4d1@mail.gmail.com>
Subject: setNumReduceTasks(1)
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org, hbase-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d9a3ac7d8cbd047e05713d

--0016e6d9a3ac7d8cbd047e05713d
Content-Type: text/plain; charset=ISO-8859-1

If I set # of reduce tasks to 1 using setNumReduceTasks(1), would the class
be instantiated only on one machine.. always?  I mean if I have a cluster of
say 1 master, 10 workers & 3 zookeepers, is the Reducer class guaranteed to
be instantiated only on 1 machine?

If answer is yes, then I will use static variable as a counter to see how
may rows have been added to my HBase table so far.  In my use case, I want
to write only N number of rows to a table.  Is there a better way to do
this?  Please let me know.  Thanks.

--0016e6d9a3ac7d8cbd047e05713d--

From general-return-977-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 26 00:10:25 2010
Return-Path: <general-return-977-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82886 invoked from network); 26 Jan 2010 00:10:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Jan 2010 00:10:24 -0000
Received: (qmail 9052 invoked by uid 500); 26 Jan 2010 00:10:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8972 invoked by uid 500); 26 Jan 2010 00:10:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8953 invoked by uid 99); 26 Jan 2010 00:10:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Jan 2010 00:10:22 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of zjffdu@gmail.com designates 209.85.160.50 as permitted sender)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Jan 2010 00:10:15 +0000
Received: by pwi20 with SMTP id 20so2863656pwi.29
        for <multiple recipients>; Mon, 25 Jan 2010 16:09:55 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:cc:content-type;
        bh=0BK6F5oAvmezgjLtf2pjYfqCAvvLw5/iRv153tW4xyQ=;
        b=OcjgDWSms3X/5Nkwn16an571OOXRblWL4/azb97oL+XgDygIb8iH3qeOhNOP/gAJO+
         c6cgIN03kEM6AeiRmREh7A0LrS6/6S6vCwEzRrl5lKIddVYmxMz9FWqQTEUcjlc7cEFG
         gh1dO6uriy98oF51xsQVCIEwObpPTrK8HLdts=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=mVSYGzgyS7CBo2beMi01g1MH21jx+j5wMZUlvhJ4ID7CDxDW0YRSIQfTV4IEf9wOXg
         2mguDl3x54A/xXMX6BNGwxc/ajAbhSKroAtYwZ+vy/9c9BWECYA1K3YetDVDZ64QTEqj
         s3D9vQavQLsnVfxQwyHZX+CR7MsHh2QRtp+5o=
MIME-Version: 1.0
Received: by 10.142.1.22 with SMTP id 22mr4954652wfa.331.1264464594964; Mon, 
	25 Jan 2010 16:09:54 -0800 (PST)
In-Reply-To: <1eabbac31001251522s2301f263rb959c768c2ee4d1@mail.gmail.com>
References: <1eabbac31001251522s2301f263rb959c768c2ee4d1@mail.gmail.com>
Date: Mon, 25 Jan 2010 16:09:54 -0800
Message-ID: <8211a1321001251609r19e737e2s2fdd27f379aeea53@mail.gmail.com>
Subject: Re: setNumReduceTasks(1)
From: Jeff Zhang <zjffdu@gmail.com>
To: hbase-user@hadoop.apache.org
Cc: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502b672039198047e061b16

--00504502b672039198047e061b16
Content-Type: text/plain; charset=UTF-8

*See my comments below*

On Mon, Jan 25, 2010 at 3:22 PM, Something Something <
mailinglists19@gmail.com> wrote:

> If I set # of reduce tasks to 1 using setNumReduceTasks(1), would the class
> be instantiated only on one machine.. always?  I mean if I have a cluster
> of
> say 1 master, 10 workers & 3 zookeepers, is the Reducer class guaranteed to
> be instantiated only on 1 machine?
>
> *--Yes*


> If answer is yes, then I will use static variable as a counter to see how
> may rows have been added to my HBase table so far.  In my use case, I want
> to write only N number of rows to a table.  Is there a better way to do
> this?  Please let me know.  Thanks.
>

*--Maybe you can use Counter to track the number of rows you add to HBase,
then you do not need to limit the reduce task as 1*


-- 
Best Regards

Jeff Zhang

--00504502b672039198047e061b16--

From general-return-978-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 26 05:03:56 2010
Return-Path: <general-return-978-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66432 invoked from network); 26 Jan 2010 05:03:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Jan 2010 05:03:55 -0000
Received: (qmail 84724 invoked by uid 500); 26 Jan 2010 05:03:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84542 invoked by uid 500); 26 Jan 2010 05:03:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84532 invoked by uid 99); 26 Jan 2010 05:03:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Jan 2010 05:03:54 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of gautam.singaraju@gmail.com designates 209.85.211.187 as permitted sender)
Received: from [209.85.211.187] (HELO mail-yw0-f187.google.com) (209.85.211.187)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Jan 2010 05:03:44 +0000
Received: by ywh17 with SMTP id 17so3778491ywh.2
        for <general@hadoop.apache.org>; Mon, 25 Jan 2010 21:03:23 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:cc:content-type;
        bh=oRku0ZSVTXDikzJVW1Y2JBe/04c22IH1TKR6u4W449U=;
        b=MBFe0K2yM8PPQmtnF9SN+JAtjUVxlHgwlizAe+RSphtGlAQdOPhshRSjr9A22SO4Et
         mAvtCSbrpzbomMoQ6VsebqKz4LxxHiPp27HBFFukWq+gZMEwcQ1UI6mWBhQ/fYDeH9kg
         npJjyeKsqLyLc0cn4TemOtupJC3f5N9Pxt/IY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=FQvPXfRPYfc+feIey+63PFxWRg5XIk5aTfA8MUzaPzIjhmbX9wF9/a8l0wrgd+YAvo
         Onspib3AP00ih0f7pHbDy/L8iwAsjlMiekR9hVhH5h7smlbm6deg1dMOrXshrKU4cI+8
         kTxbB3jPwlpCjK5U5DbtC5vBfqgQZIqlR9Pu0=
MIME-Version: 1.0
Received: by 10.150.243.12 with SMTP id q12mr9777239ybh.233.1264482203207; 
	Mon, 25 Jan 2010 21:03:23 -0800 (PST)
In-Reply-To: <45f85f71001250838s35572132jc84b220379549092@mail.gmail.com>
References: <C78284BE.27AB%awittenauer@linkedin.com>
	 <002001ca9d97$a8ba13b0$2401120a@china.huawei.com>
	 <78568af11001250112v3b610478w49703d72306d4a2f@mail.gmail.com>
	 <78568af11001250113k41f303ffq8d0eb4ea5bc33523@mail.gmail.com>
	 <45f85f71001250838s35572132jc84b220379549092@mail.gmail.com>
Date: Tue, 26 Jan 2010 00:03:23 -0500
Message-ID: <f8b2ccd41001252103wd5e54e6g43dbc4f4f755ac3c@mail.gmail.com>
Subject: Re: RE: set how much CPU to be utilised by a MapReduce job
From: Gautam Singaraju <gautam.singaraju@gmail.com>
To: general@hadoop.apache.org
Cc: naveenkumarp@huawei.com
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

One thing that you might want to consider is to increase the
replication factor. This might take a lot of disk space, but also
might increase the performance.

You might also want to check out: Sun Grid Engine Hadoop Integration
http://blogs.sun.com/templedf/entry/beta_testing_the_sun_grid
---
Gautam



On Mon, Jan 25, 2010 at 11:38 AM, Todd Lipcon <todd@cloudera.com> wrote:
> If you can require a recent kernel, you could use cgroups:
>
> http://broadcast.oreilly.com/2009/06/manage-your-performance-with-cgroups-and-projects.html
>
> No one has integrated this with hadoop yet as it's still pretty new, and
> Hadoop clusters are meant to be run on unshared hardware.
>
> -Todd
>
> On Mon, Jan 25, 2010 at 1:13 AM, Ryan Rawson <ryanobjc@gmail.com> wrote:
>
>> The only thing you could do is to have the tasktracker nice the child when
>> its exceeding its reservation. Aside from that, its hard to limit without
>> killing a process.
>>
>> On Jan 25, 2010 12:23 AM, "Naveen Kumar Prasad" <naveenkumarp@huawei.com>
>> wrote:
>>
>>
>> This functionality may not be readily available with Hadoop.
>>
>> But it would be appreciable if anyone can help me in understanding how to
>> go
>> about developing this feature.
>>
>> Regards, Naveen Kumar HUAWEI TECHNOLOGIES CO.,LTD. huawei_logo
>>
>> Address: Huawei Industrial Base Bantian Longgang Shenzhen 518129, P.R.China
>> www.huawei.com ---------...
>>
>> -----Original Message----- From: Allen Wittenauer [mailto:
>> awittenauer@linkedin.com] Sent: Monday, J...
>>
>

From general-return-979-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 26 16:48:27 2010
Return-Path: <general-return-979-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39245 invoked from network); 26 Jan 2010 16:48:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Jan 2010 16:48:27 -0000
Received: (qmail 69803 invoked by uid 500); 26 Jan 2010 16:48:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69082 invoked by uid 500); 26 Jan 2010 16:48:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69007 invoked by uid 99); 26 Jan 2010 16:48:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Jan 2010 16:48:25 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mailinglists19@gmail.com designates 209.85.219.213 as permitted sender)
Received: from [209.85.219.213] (HELO mail-ew0-f213.google.com) (209.85.219.213)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Jan 2010 16:48:18 +0000
Received: by ewy5 with SMTP id 5so5618159ewy.12
        for <multiple recipients>; Tue, 26 Jan 2010 08:47:57 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=TJd2g1WXOWQtpWIRLwje33fy/S/7l9tkTpaPMP5pv3o=;
        b=AvfShjRtDjGWokbqvLXyRRS8UzgGrtxf4ojxeZ568W7hgR6v/b9j5ASBdRp/uPOkMs
         N+6Re0705OY6uTwfUBqAjbNWGUdVMN/pn1FLyWfGmdpfwFzFO1+AfLCrtAPcPCA9ZzZX
         EMoAmRXuSbyLQDqHDn+RvO5Vr8lCFOpgnACKM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=V4EM9bJhM8xEA50J4dKYyDqnd0gtEXscUgOR/bPR2kcQclaROpIG9+vlUw/cS8o/u2
         H2LcNeCaJo8pdsXSrc5Wpwrg8jUGzy/Xzvt+NWydwFfhLbd0Sw46FuOUON7zw7XHdYQ5
         bP8/4VZRZHakU01AJbV+Hy6tV3kT9B6YzRDfc=
MIME-Version: 1.0
Received: by 10.216.89.80 with SMTP id b58mr472883wef.73.1264524477092; Tue, 
	26 Jan 2010 08:47:57 -0800 (PST)
Date: Tue, 26 Jan 2010 08:47:57 -0800
Message-ID: <1eabbac31001260847g3a726f62o3a89ee2d74096e6e@mail.gmail.com>
Subject: Performance of EC2
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org, hbase-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6dab03444533c047e140c76

--0016e6dab03444533c047e140c76
Content-Type: text/plain; charset=ISO-8859-1

I have noticed some strange performance numbers on EC2.  If someone can give
me some hints to improve performance that would be greatly appreciated.
 Here are the details:

I have a process that runs a series of Jobs under Hadoop 0.20.1 & Hbase
0.20.2  I ran the *exact* same process with following configurations:

1) 1 Master & 4 Workers (*c1.xlarge* instances) & 1 Zookeeper (*c1.medium*)
with *8 Reducers *for every Reduce task.  The process completed in *849*
 seconds.

2) 1 Master, 4 Workers & 1 Zookeeper  *ALL m1.small* instances with *8
Reducers *for every Reduce task.  The process completed in *906* seconds.

3) 1 Master, *11* Workers & *3* Zookeepers  *ALL m1.small* instances with *20
Reducers *for every Reduce task.  The process completed in *984* seconds!


Two main questions:

1)  It's totally surprising that when I have 11 workers with 20 Reducers it
runs slower than when I have exactly same type of fewer machines with fewer
reducers..
2)  As expected it runs faster on c1.xlarge, but the performance improvement
doesn't justify the high cost difference.  I must not be utilizing the
machine power, but I don't know how to do that.

Here are some of the performance improvements tricks that I have learnt from
this mailing list in the past that I am using:

1)  conf.set("hbase.client.scanner.caching", "30");   I have this for all
jobs.

2)  Using the following code every time I open a HTable:
        this.table = new HTable(new HBaseConfiguration(), "tablenameXYZ");
        table.setAutoFlush(false);
        table.setWriteBufferSize(1024 * 1024 * 12);

3)  For every Put I do this:
          Put put = new Put(Bytes.toBytes(out));
          put.setWriteToWAL(false);

4)  Change the No. of Reducers as per the No. of Workers.  I believe the
formula is:  # of workers * 1.75.

Any other hints?  As always, greatly appreciate the help.  Thanks.

--0016e6dab03444533c047e140c76--

From general-return-980-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 26 18:04:57 2010
Return-Path: <general-return-980-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 80801 invoked from network); 26 Jan 2010 18:04:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Jan 2010 18:04:56 -0000
Received: (qmail 97881 invoked by uid 500); 26 Jan 2010 18:04:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97350 invoked by uid 500); 26 Jan 2010 18:04:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97200 invoked by uid 99); 26 Jan 2010 18:04:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Jan 2010 18:04:54 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 209.85.221.188 as permitted sender)
Received: from [209.85.221.188] (HELO mail-qy0-f188.google.com) (209.85.221.188)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Jan 2010 18:04:46 +0000
Received: by qyk26 with SMTP id 26so2774131qyk.5
        for <multiple recipients>; Tue, 26 Jan 2010 10:04:22 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:message-id:subject:from:to:cc
         :content-type:content-transfer-encoding;
        bh=a3bnVL2qcTNkjdX2GLerdV4MIMVeBj1Iru1cyXIvba0=;
        b=vCMCuAav2CHqPHKAVXmcdmeSD261Il4w8WBIO0AQEgkXtoyFUzSKz326niJg8oTmQ6
         XXz8DlgLlxBX9sICRg8we4MS4dw9SOIFdW5TG/RQvNrD8axSNotRma+bz2mrTpVDmMap
         KsptcAELurRbn50WTArnVtGnL32CbTf9pRoKo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:cc:content-type
         :content-transfer-encoding;
        b=pWSA2dasxwUnbo72Qb2+tYZRVCU6uPyUJVK9JBPhGJjdsKgYIkiqDMmqxgM4R9b2VC
         rLvnCJqQc0deLEL4u8GsG1grl0GIYqdF06MyaPoZeX0GuZCsw9Aecvo2KdmVaRusLqL8
         B8aMYvduoQPi8cGWc8i0lws7PObwHuYlI7lHg=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
Received: by 10.229.59.203 with SMTP id m11mr4591548qch.94.1264529062607; Tue, 
	26 Jan 2010 10:04:22 -0800 (PST)
In-Reply-To: <1eabbac31001260847g3a726f62o3a89ee2d74096e6e@mail.gmail.com>
References: <1eabbac31001260847g3a726f62o3a89ee2d74096e6e@mail.gmail.com>
Date: Tue, 26 Jan 2010 10:04:22 -0800
X-Google-Sender-Auth: 0345e7bfa062f0d1
Message-ID: <7c962aed1001261004j22bd5af4vf937f3dd970d0c94@mail.gmail.com>
Subject: Re: Performance of EC2
From: Stack <stack@duboce.net>
To: hbase-user@hadoop.apache.org
Cc: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Tue, Jan 26, 2010 at 8:47 AM, Something Something
<mailinglists19@gmail.com> wrote:
> I have noticed some strange performance numbers on EC2. =A0If someone can=
 give
> me some hints to improve performance that would be greatly appreciated.
> =A0Here are the details:
>
> I have a process that runs a series of Jobs under Hadoop 0.20.1 & Hbase
> 0.20.2 =A0I ran the *exact* same process with following configurations:
>
> 1) 1 Master & 4 Workers (*c1.xlarge* instances) & 1 Zookeeper (*c1.medium=
*)
> with *8 Reducers *for every Reduce task. =A0The process completed in *849=
*
> =A0seconds.

How many concurrent reducers run on each node?  Default two?

>
> 2) 1 Master, 4 Workers & 1 Zookeeper =A0*ALL m1.small* instances with *8
> Reducers *for every Reduce task. =A0The process completed in *906* second=
s.
>
> 3) 1 Master, *11* Workers & *3* Zookeepers =A0*ALL m1.small* instances wi=
th *20
> Reducers *for every Reduce task. =A0The process completed in *984* second=
s!
>

How much of this overall time is spent in reduce phase, in particular
the time spent inserting into hbase? (Starts at 66% IIRC)

>
> Two main questions:
>
> 1) =A0It's totally surprising that when I have 11 workers with 20 Reducer=
s it
> runs slower than when I have exactly same type of fewer machines with few=
er
> reducers..


Yes.  My guess is that on the small instances, that if you ran the job
multiple times that there would be large variance in how long it takes
to complete.


> 2) =A0As expected it runs faster on c1.xlarge, but the performance improv=
ement
> doesn't justify the high cost difference. =A0I must not be utilizing the
> machine power, but I don't know how to do that.
>

The main reason for xlarge is that the platform is more predictable in
its performance profile than small sized instances.  I'm a little
surprised that all worked on the small instances, that your jobs
completed.

I'd suggest you spend a bit of time figuring where your MR jobs are
spending their time?  Is it all doing hbase inserts?  Are inserts to a
new table?


> Here are some of the performance improvements tricks that I have learnt f=
rom
> this mailing list in the past that I am using:
>
> 1) =A0conf.set("hbase.client.scanner.caching", "30"); =A0 I have this for=
 all
> jobs


FYI, you can set this on the Scan instance rather than globally in the
conf.  Just FYI.


>
> 2) =A0Using the following code every time I open a HTable:
> =A0 =A0 =A0 =A0this.table =3D new HTable(new HBaseConfiguration(), "table=
nameXYZ");
> =A0 =A0 =A0 =A0table.setAutoFlush(false);
> =A0 =A0 =A0 =A0table.setWriteBufferSize(1024 * 1024 * 12);

Are you opening a new table inside each task or once up in the config?

>
> 4) =A0Change the No. of Reducers as per the No. of Workers. =A0I believe =
the
> formula is: =A0# of workers * 1.75.

You have to temper the above general rule with the fact that
tasktrackers and datanodes running on the same node can impinge upon
each other, often to the regionservers detriment.

Thats enough for now.  I'm sure others on list have opinions on the above.

St.Ack

From general-return-981-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 26 18:15:26 2010
Return-Path: <general-return-981-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88818 invoked from network); 26 Jan 2010 18:15:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Jan 2010 18:15:26 -0000
Received: (qmail 17596 invoked by uid 500); 26 Jan 2010 18:15:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17533 invoked by uid 500); 26 Jan 2010 18:15:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17523 invoked by uid 99); 26 Jan 2010 18:15:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Jan 2010 18:15:25 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 26 Jan 2010 18:15:23 +0000
Received: (qmail 88530 invoked by uid 99); 26 Jan 2010 18:15:01 -0000
Received: from localhost.apache.org (HELO [172.16.61.34]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Jan 2010 18:15:01 +0000
Message-ID: <4B5F3124.4090004@apache.org>
Date: Tue, 26 Jan 2010 10:15:00 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: avro in mapreduce
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

I would like to call folks attention to MAPREDUCE-1126.

https://issues.apache.org/jira/browse/MAPREDUCE-1126

This is a key link in a series of issues involved in integrating Avro in
Mapreduce.  Aaron proposed a design in early December, building on the
design Tom developed last summer and committed in September in
HADOOP-6165.  Aaron's design was approved, and, after several rounds of
reviews, I committed Aaron's patch on 11 January.

On 15 January Owen reverted this commit without warning.  It seems that
Owen objects to the path initiated last July in HADOOP-6165.

Aaron has also contributed MAPREDUCE-815, which permits one to use Avro
for all phases of Mapreduce.  When that issue is committed, the primary
chain of Avro integration into Mapreduce will be complete.

Can others please take the time to read this issue and express their
opinions?

Thank you,

Doug


From general-return-982-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 26 18:33:45 2010
Return-Path: <general-return-982-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1468 invoked from network); 26 Jan 2010 18:33:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Jan 2010 18:33:45 -0000
Received: (qmail 46824 invoked by uid 500); 26 Jan 2010 18:33:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46753 invoked by uid 500); 26 Jan 2010 18:33:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46743 invoked by uid 99); 26 Jan 2010 18:33:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Jan 2010 18:33:44 +0000
X-ASF-Spam-Status: No, hits=3.5 required=10.0
	tests=SPF_PASS,URIBL_BLACK
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jdcryans@gmail.com designates 209.85.217.228 as permitted sender)
Received: from [209.85.217.228] (HELO mail-gx0-f228.google.com) (209.85.217.228)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Jan 2010 18:33:34 +0000
Received: by gxk28 with SMTP id 28so4124806gxk.9
        for <general@hadoop.apache.org>; Tue, 26 Jan 2010 10:33:14 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        bh=K0hKScuw2+Otq1EG24yZYxpbE5XNL6aVDekv9nA3g34=;
        b=Lkk4cf9PRfYoiC93JEnqIRmqQUHy0hgf4FmZzCuBitfBPyckRs7c4Yq0I6Nv2WMWA0
         FzEcFwHeyYrMbYrrL6rBNNdTvQAPbrTOWvr486HyONG5pfz2eLQcH2bNGhngvM9xVcyP
         TYHO0FwiVPz2wAojzqP3UaMDC8WvGZm6FHx3I=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=aFw21AvAlu39ACG7nwZPEVRSWXJtyS0Rqui1XRvo7nyD9WfdGiMjzOYj/unF2eqpTh
         S0JFHiXirtlxfv14OGnmT5fyzsY2alLMfOKMAV+Q2VcuKOa0tKeNHPhKhDRzWzO+puiN
         4vHVBHCYukK3Q+hFM4d8AWDOzC0zXhurKXmuc=
MIME-Version: 1.0
Sender: jdcryans@gmail.com
Received: by 10.91.142.16 with SMTP id u16mr948491agn.11.1264530792497; Tue, 
	26 Jan 2010 10:33:12 -0800 (PST)
In-Reply-To: <31a243e71001261026v3de25f31i2ef741d430787279@mail.gmail.com>
References: <31a243e71001261026v3de25f31i2ef741d430787279@mail.gmail.com>
Date: Tue, 26 Jan 2010 10:33:12 -0800
X-Google-Sender-Auth: 2d6182eee9871b01
Message-ID: <31a243e71001261033qcd76c5ahf69ddcb32e1d3fbd@mail.gmail.com>
Subject: ANN: hbase 0.20.3 is available for download
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi everyone!

We are proud to announce that hbase 0.20.3 is available for download:
http://hbase.org/releases.html. Eighty four issues have been addressed
since hbase 0.20.2. =A0The release notes are available here:
http://su.pr/16X1Cr.=A0Please upgrade at your convenience. A rolling
upgrade is possible from 0.20.1 and 0.20.2 but not from 0.20.0.

This is the third bug fix release in the 0.20 branch and it also
contains new EC2 scripts, a new indexation contribution called IHBase,
better JMX metrics and much more.

Thanks to all who contributed to this release!

Yours,
The HBase Team

From general-return-983-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 26 18:37:21 2010
Return-Path: <general-return-983-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9918 invoked from network); 26 Jan 2010 18:37:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Jan 2010 18:37:21 -0000
Received: (qmail 53469 invoked by uid 500); 26 Jan 2010 18:37:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53370 invoked by uid 500); 26 Jan 2010 18:37:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53349 invoked by uid 99); 26 Jan 2010 18:37:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Jan 2010 18:37:19 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jdcryans@gmail.com designates 209.85.217.228 as permitted sender)
Received: from [209.85.217.228] (HELO mail-gx0-f228.google.com) (209.85.217.228)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Jan 2010 18:37:10 +0000
Received: by gxk28 with SMTP id 28so4128174gxk.9
        for <multiple recipients>; Tue, 26 Jan 2010 10:36:50 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:message-id:subject:from:to:cc
         :content-type:content-transfer-encoding;
        bh=SHgQNBDqqSGLj7S4fPMYNbav4hVeBPeBnQbAE0QjwHg=;
        b=uc3FWUXLUZTTjDFyDP1VV189ypCj2ALwzoLdBJok7ZHHN19zaqJnsrUDB5/Q3PacqO
         HSSRx8I8TiBO/sA93MbtBrSpGHRo5RrDXmpw1poqrCbs8jRqHyXZ+jK/1zpWu92hy2Oi
         UR9zNX/57Kf/ufacKJMCJX/eNqnHZ2xUYAFUs=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:cc:content-type
         :content-transfer-encoding;
        b=YhyeJDL76iG8fhacHvzQCBcGyQDAk9SqvDvSMa8hT8Vwx5bnA8LbDlSrQdrN7N04zN
         ZuCMG0JRL9osoTR3Q9dQRcr0AXQVJ+fdVXYz4BB6TjMIOcKCfwaau6WEQ2tj7H0Kct+W
         h5Tl+cNNCYG9GFtqj5Fkl3QjT0lvjoJ9BVrMU=
MIME-Version: 1.0
Sender: jdcryans@gmail.com
Received: by 10.90.13.27 with SMTP id 27mr1789212agm.28.1264531000746; Tue, 26 
	Jan 2010 10:36:40 -0800 (PST)
In-Reply-To: <1eabbac31001260847g3a726f62o3a89ee2d74096e6e@mail.gmail.com>
References: <1eabbac31001260847g3a726f62o3a89ee2d74096e6e@mail.gmail.com>
Date: Tue, 26 Jan 2010 10:36:40 -0800
X-Google-Sender-Auth: 863795c5d95ea82e
Message-ID: <31a243e71001261036t7d47bb7fo387e079cc8ed0974@mail.gmail.com>
Subject: Re: Performance of EC2
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: hbase-user@hadoop.apache.org
Cc: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

How big is your dataset?

J-D

On Tue, Jan 26, 2010 at 8:47 AM, Something Something
<mailinglists19@gmail.com> wrote:
> I have noticed some strange performance numbers on EC2. =A0If someone can=
 give
> me some hints to improve performance that would be greatly appreciated.
> =A0Here are the details:
>
> I have a process that runs a series of Jobs under Hadoop 0.20.1 & Hbase
> 0.20.2 =A0I ran the *exact* same process with following configurations:
>
> 1) 1 Master & 4 Workers (*c1.xlarge* instances) & 1 Zookeeper (*c1.medium=
*)
> with *8 Reducers *for every Reduce task. =A0The process completed in *849=
*
> =A0seconds.
>
> 2) 1 Master, 4 Workers & 1 Zookeeper =A0*ALL m1.small* instances with *8
> Reducers *for every Reduce task. =A0The process completed in *906* second=
s.
>
> 3) 1 Master, *11* Workers & *3* Zookeepers =A0*ALL m1.small* instances wi=
th *20
> Reducers *for every Reduce task. =A0The process completed in *984* second=
s!
>
>
> Two main questions:
>
> 1) =A0It's totally surprising that when I have 11 workers with 20 Reducer=
s it
> runs slower than when I have exactly same type of fewer machines with few=
er
> reducers..
> 2) =A0As expected it runs faster on c1.xlarge, but the performance improv=
ement
> doesn't justify the high cost difference. =A0I must not be utilizing the
> machine power, but I don't know how to do that.
>
> Here are some of the performance improvements tricks that I have learnt f=
rom
> this mailing list in the past that I am using:
>
> 1) =A0conf.set("hbase.client.scanner.caching", "30"); =A0 I have this for=
 all
> jobs.
>
> 2) =A0Using the following code every time I open a HTable:
> =A0 =A0 =A0 =A0this.table =3D new HTable(new HBaseConfiguration(), "table=
nameXYZ");
> =A0 =A0 =A0 =A0table.setAutoFlush(false);
> =A0 =A0 =A0 =A0table.setWriteBufferSize(1024 * 1024 * 12);
>
> 3) =A0For every Put I do this:
> =A0 =A0 =A0 =A0 =A0Put put =3D new Put(Bytes.toBytes(out));
> =A0 =A0 =A0 =A0 =A0put.setWriteToWAL(false);
>
> 4) =A0Change the No. of Reducers as per the No. of Workers. =A0I believe =
the
> formula is: =A0# of workers * 1.75.
>
> Any other hints? =A0As always, greatly appreciate the help. =A0Thanks.
>

From general-return-984-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 26 19:20:41 2010
Return-Path: <general-return-984-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25686 invoked from network); 26 Jan 2010 19:20:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Jan 2010 19:20:41 -0000
Received: (qmail 39748 invoked by uid 500); 26 Jan 2010 19:20:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39630 invoked by uid 500); 26 Jan 2010 19:20:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39611 invoked by uid 99); 26 Jan 2010 19:20:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Jan 2010 19:20:38 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mailinglists19@gmail.com designates 209.85.220.221 as permitted sender)
Received: from [209.85.220.221] (HELO mail-fx0-f221.google.com) (209.85.220.221)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Jan 2010 19:20:32 +0000
Received: by fxm21 with SMTP id 21so901466fxm.29
        for <multiple recipients>; Tue, 26 Jan 2010 11:20:10 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:cc:content-type;
        bh=VFytQRjpRzI3V/rj5unGfz+04mz0/xgNeOD45JTOGSU=;
        b=AYI66ly4dpgm9YOsb2ZR2Jhc6xdSkZvGIbW7y+eShG4Gg9C/B/jTpjKmepnBFngjA1
         USVaL5Go6Tb2GBUwyLr7UlOBlLxolvIqxGFaVTOojoweWAcrK6A2DDB0voWbjwXbpYn5
         dR3uBWsd7OZfGsoTe1LtEkTDZ2lA89PqTRe9A=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=tIwBhQUAK9+D/EL00rgYE2oEDV7ZbRhcz8n3JXkeyMMYxv8GfhHYpvgR0yubfptVA+
         3P9xs4/vHHARjTsznva7rn6NURbkqMmdJZAD2aYPIESwmZDxViZFbDrbBfYLfbnCQyzw
         0rBYM4J5h9ieBs3Vk9Ms+MG1et6QmXumuY0bI=
MIME-Version: 1.0
Received: by 10.216.89.205 with SMTP id c55mr1434407wef.186.1264533603897; 
	Tue, 26 Jan 2010 11:20:03 -0800 (PST)
In-Reply-To: <31a243e71001261036t7d47bb7fo387e079cc8ed0974@mail.gmail.com>
References: <1eabbac31001260847g3a726f62o3a89ee2d74096e6e@mail.gmail.com>
	 <31a243e71001261036t7d47bb7fo387e079cc8ed0974@mail.gmail.com>
Date: Tue, 26 Jan 2010 11:20:03 -0800
Message-ID: <1eabbac31001261120m74ee4817x3751592c930fd98b@mail.gmail.com>
Subject: Re: Performance of EC2
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org
Cc: hbase-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d56652445117047e162cf9

--0016e6d56652445117047e162cf9
Content-Type: text/plain; charset=ISO-8859-1

Here are some of the answers:

>>  How many concurrent reducers run on each node?  Default two?
I was assuming 2 on each node would be the default.  If not, this could be a
problem.  Please let me know.

>>'d suggest you spend a bit of time figuring where your MR jobs
are spending their time?
I agree.  Will do some more research :)

>>How much of this overall time is spent in reduce phase?
Mostly time is spent in the Reduce phases, because that's where most of the
critical code is.

>>Are inserts to a new table?
Yes, all inserts will always be in a new table.  In fact, I disable/drop
HTables during this process.  Not using any special indexes, should I be?

>>I'm a little surprised that all worked on the small instances, that your
jobs completed.
But, really, shouldn't Amazon guarantee predictability :)  After all I am
paying for these instances.. albeit a small amount!

>>Are you opening a new table inside each task or once up in the config?
I open HTable in the 'setup' method for each mapper/reducer, and close table
in the 'cleanup' method.

>>You have to temper the above general rule with the fact that...
I will try a few combinations.

>>How big is your dataset?
This one in particular is not big, but the real production ones will be.
 Here's approximately how many rows get processed:
Phase 1:  300 rows
Phase 2 thru 8:  100 rows.
(Note:  Each phase does complex calculations on the row.)

Thanks for the help.


On Tue, Jan 26, 2010 at 10:36 AM, Jean-Daniel Cryans <jdcryans@apache.org>wrote:

> How big is your dataset?
>
> J-D
>
> On Tue, Jan 26, 2010 at 8:47 AM, Something Something
> <mailinglists19@gmail.com> wrote:
> > I have noticed some strange performance numbers on EC2.  If someone can
> give
> > me some hints to improve performance that would be greatly appreciated.
> >  Here are the details:
> >
> > I have a process that runs a series of Jobs under Hadoop 0.20.1 & Hbase
> > 0.20.2  I ran the *exact* same process with following configurations:
> >
> > 1) 1 Master & 4 Workers (*c1.xlarge* instances) & 1 Zookeeper
> (*c1.medium*)
> > with *8 Reducers *for every Reduce task.  The process completed in *849*
> >  seconds.
> >
> > 2) 1 Master, 4 Workers & 1 Zookeeper  *ALL m1.small* instances with *8
> > Reducers *for every Reduce task.  The process completed in *906* seconds.
> >
> > 3) 1 Master, *11* Workers & *3* Zookeepers  *ALL m1.small* instances with
> *20
> > Reducers *for every Reduce task.  The process completed in *984* seconds!
> >
> >
> > Two main questions:
> >
> > 1)  It's totally surprising that when I have 11 workers with 20 Reducers
> it
> > runs slower than when I have exactly same type of fewer machines with
> fewer
> > reducers..
> > 2)  As expected it runs faster on c1.xlarge, but the performance
> improvement
> > doesn't justify the high cost difference.  I must not be utilizing the
> > machine power, but I don't know how to do that.
> >
> > Here are some of the performance improvements tricks that I have learnt
> from
> > this mailing list in the past that I am using:
> >
> > 1)  conf.set("hbase.client.scanner.caching", "30");   I have this for all
> > jobs.
> >
> > 2)  Using the following code every time I open a HTable:
> >        this.table = new HTable(new HBaseConfiguration(), "tablenameXYZ");
> >        table.setAutoFlush(false);
> >        table.setWriteBufferSize(1024 * 1024 * 12);
> >
> > 3)  For every Put I do this:
> >          Put put = new Put(Bytes.toBytes(out));
> >          put.setWriteToWAL(false);
> >
> > 4)  Change the No. of Reducers as per the No. of Workers.  I believe the
> > formula is:  # of workers * 1.75.
> >
> > Any other hints?  As always, greatly appreciate the help.  Thanks.
> >
>

--0016e6d56652445117047e162cf9--

From general-return-985-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 26 19:44:52 2010
Return-Path: <general-return-985-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31960 invoked from network); 26 Jan 2010 19:44:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Jan 2010 19:44:52 -0000
Received: (qmail 79047 invoked by uid 500); 26 Jan 2010 19:44:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78955 invoked by uid 500); 26 Jan 2010 19:44:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78932 invoked by uid 99); 26 Jan 2010 19:44:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Jan 2010 19:44:49 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Jan 2010 19:44:39 +0000
Received: from [10.73.135.243] (wifi-e-135-243.corp.yahoo.com [10.73.135.243])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o0QJiF9W060174;
	Tue, 26 Jan 2010 11:44:15 -0800 (PST)
Message-ID: <4B5F4610.8050803@apache.org>
Date: Tue, 26 Jan 2010 11:44:16 -0800
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
CC: hbase-user@hadoop.apache.org
Subject: Re: Performance of EC2
References: <1eabbac31001260847g3a726f62o3a89ee2d74096e6e@mail.gmail.com>	 <31a243e71001261036t7d47bb7fo387e079cc8ed0974@mail.gmail.com> <1eabbac31001261120m74ee4817x3751592c930fd98b@mail.gmail.com>
In-Reply-To: <1eabbac31001261120m74ee4817x3751592c930fd98b@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Re "Amazon predictability", did you guys see this recent paper:
http://people.csail.mit.edu/tromer/cloudsec/

Also some addl background on "noisy neighbor effects":
http://bit.ly/4O7dHx
http://bit.ly/8zPvQd

Some interesting bits of information in there.

Patrick

Something Something wrote:
> Here are some of the answers:
> 
>>>  How many concurrent reducers run on each node?  Default two?
> I was assuming 2 on each node would be the default.  If not, this could be a
> problem.  Please let me know.
> 
>>> 'd suggest you spend a bit of time figuring where your MR jobs
> are spending their time?
> I agree.  Will do some more research :)
> 
>>> How much of this overall time is spent in reduce phase?
> Mostly time is spent in the Reduce phases, because that's where most of the
> critical code is.
> 
>>> Are inserts to a new table?
> Yes, all inserts will always be in a new table.  In fact, I disable/drop
> HTables during this process.  Not using any special indexes, should I be?
> 
>>> I'm a little surprised that all worked on the small instances, that your
> jobs completed.
> But, really, shouldn't Amazon guarantee predictability :)  After all I am
> paying for these instances.. albeit a small amount!
> 
>>> Are you opening a new table inside each task or once up in the config?
> I open HTable in the 'setup' method for each mapper/reducer, and close table
> in the 'cleanup' method.
> 
>>> You have to temper the above general rule with the fact that...
> I will try a few combinations.
> 
>>> How big is your dataset?
> This one in particular is not big, but the real production ones will be.
>  Here's approximately how many rows get processed:
> Phase 1:  300 rows
> Phase 2 thru 8:  100 rows.
> (Note:  Each phase does complex calculations on the row.)
> 
> Thanks for the help.
> 
> 
> On Tue, Jan 26, 2010 at 10:36 AM, Jean-Daniel Cryans <jdcryans@apache.org>wrote:
> 
>> How big is your dataset?
>>
>> J-D
>>
>> On Tue, Jan 26, 2010 at 8:47 AM, Something Something
>> <mailinglists19@gmail.com> wrote:
>>> I have noticed some strange performance numbers on EC2.  If someone can
>> give
>>> me some hints to improve performance that would be greatly appreciated.
>>>  Here are the details:
>>>
>>> I have a process that runs a series of Jobs under Hadoop 0.20.1 & Hbase
>>> 0.20.2  I ran the *exact* same process with following configurations:
>>>
>>> 1) 1 Master & 4 Workers (*c1.xlarge* instances) & 1 Zookeeper
>> (*c1.medium*)
>>> with *8 Reducers *for every Reduce task.  The process completed in *849*
>>>  seconds.
>>>
>>> 2) 1 Master, 4 Workers & 1 Zookeeper  *ALL m1.small* instances with *8
>>> Reducers *for every Reduce task.  The process completed in *906* seconds.
>>>
>>> 3) 1 Master, *11* Workers & *3* Zookeepers  *ALL m1.small* instances with
>> *20
>>> Reducers *for every Reduce task.  The process completed in *984* seconds!
>>>
>>>
>>> Two main questions:
>>>
>>> 1)  It's totally surprising that when I have 11 workers with 20 Reducers
>> it
>>> runs slower than when I have exactly same type of fewer machines with
>> fewer
>>> reducers..
>>> 2)  As expected it runs faster on c1.xlarge, but the performance
>> improvement
>>> doesn't justify the high cost difference.  I must not be utilizing the
>>> machine power, but I don't know how to do that.
>>>
>>> Here are some of the performance improvements tricks that I have learnt
>> from
>>> this mailing list in the past that I am using:
>>>
>>> 1)  conf.set("hbase.client.scanner.caching", "30");   I have this for all
>>> jobs.
>>>
>>> 2)  Using the following code every time I open a HTable:
>>>        this.table = new HTable(new HBaseConfiguration(), "tablenameXYZ");
>>>        table.setAutoFlush(false);
>>>        table.setWriteBufferSize(1024 * 1024 * 12);
>>>
>>> 3)  For every Put I do this:
>>>          Put put = new Put(Bytes.toBytes(out));
>>>          put.setWriteToWAL(false);
>>>
>>> 4)  Change the No. of Reducers as per the No. of Workers.  I believe the
>>> formula is:  # of workers * 1.75.
>>>
>>> Any other hints?  As always, greatly appreciate the help.  Thanks.
>>>
> 

From general-return-986-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 26 20:50:07 2010
Return-Path: <general-return-986-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53849 invoked from network); 26 Jan 2010 20:50:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Jan 2010 20:50:07 -0000
Received: (qmail 65548 invoked by uid 500); 26 Jan 2010 20:50:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65466 invoked by uid 500); 26 Jan 2010 20:50:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65447 invoked by uid 99); 26 Jan 2010 20:50:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Jan 2010 20:50:05 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mailinglists19@gmail.com designates 209.85.219.216 as permitted sender)
Received: from [209.85.219.216] (HELO mail-ew0-f216.google.com) (209.85.219.216)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Jan 2010 20:49:54 +0000
Received: by ewy8 with SMTP id 8so5929674ewy.29
        for <multiple recipients>; Tue, 26 Jan 2010 12:49:31 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:cc:content-type;
        bh=xdAiVdUApHFyU5LjF180+W7VLlx8csugxApFVC6Jly8=;
        b=SiJZFDDMcvu86jReDl86Nt1b4AykCUpwB1s5bROFqN/4uvdeCqqPDaFV9+UDV05kv/
         WJGvHo+ESENP3KU/TGSmdn3SiZxGLz3OjMpdylLLYOAYR+We/VqaeVL54yvU/J0au5AL
         R3g6x6WtLFXw/fxLbjroc/TXdn6ckh2S6Wm2c=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=WDrfgaGsqtfZNu6nmp10Iw6YrblkyK5h/qTQfJUMM/CMOg2wjsz0HnFRRhlfjUo4Kr
         HhHb41GYRSVuH1lBfqeHly1CEjBKuk+hCLRkKMDGQjerBGsiMSJMSzrWR19sGNyFeX5d
         5ZndL0GNCja+x0EyYO7qqzeIT9RBGjNXE89ug=
MIME-Version: 1.0
Received: by 10.216.87.11 with SMTP id x11mr166527wee.16.1264538971642; Tue, 
	26 Jan 2010 12:49:31 -0800 (PST)
In-Reply-To: <4B5F4610.8050803@apache.org>
References: <1eabbac31001260847g3a726f62o3a89ee2d74096e6e@mail.gmail.com>
	 <31a243e71001261036t7d47bb7fo387e079cc8ed0974@mail.gmail.com>
	 <1eabbac31001261120m74ee4817x3751592c930fd98b@mail.gmail.com>
	 <4B5F4610.8050803@apache.org>
Date: Tue, 26 Jan 2010 12:49:31 -0800
Message-ID: <1eabbac31001261249s4b09fbdfyb886927b36c2953e@mail.gmail.com>
Subject: Re: Performance of EC2
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org
Cc: hbase-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d6453a359a7a047e176c6d
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d6453a359a7a047e176c6d
Content-Type: text/plain; charset=ISO-8859-1

Wow.. how naive I am to think that I could trust Amazon.  Thanks for
forwarding the links, Patrick.  Seems like Amazon's reliability has gone
down considerably over the past few months.  (Occasionally my instances fail
on startup or die in the middle for no apparent reason, and I used to think
I was doing something dumb!)

But what I don't understand is this... if I *reserve* an instance then I
wouldn't be sharing its CPU with anyone, right?  The blog seems to indicate
otherwise.

I guess, I will have to look for alternatives to Amazon EC2.  Any one has
any recommendations?  Thanks again.


On Tue, Jan 26, 2010 at 11:44 AM, Patrick Hunt <phunt@apache.org> wrote:

> Re "Amazon predictability", did you guys see this recent paper:
> http://people.csail.mit.edu/tromer/cloudsec/
>
> Also some addl background on "noisy neighbor effects":
> http://bit.ly/4O7dHx
> http://bit.ly/8zPvQd
>
> Some interesting bits of information in there.
>
> Patrick
>
>
> Something Something wrote:
>
>> Here are some of the answers:
>>
>>   How many concurrent reducers run on each node?  Default two?
>>>>
>>> I was assuming 2 on each node would be the default.  If not, this could
>> be a
>> problem.  Please let me know.
>>
>>  'd suggest you spend a bit of time figuring where your MR jobs
>>>>
>>> are spending their time?
>> I agree.  Will do some more research :)
>>
>>  How much of this overall time is spent in reduce phase?
>>>>
>>> Mostly time is spent in the Reduce phases, because that's where most of
>> the
>> critical code is.
>>
>>  Are inserts to a new table?
>>>>
>>> Yes, all inserts will always be in a new table.  In fact, I disable/drop
>> HTables during this process.  Not using any special indexes, should I be?
>>
>>  I'm a little surprised that all worked on the small instances, that your
>>>>
>>> jobs completed.
>> But, really, shouldn't Amazon guarantee predictability :)  After all I am
>> paying for these instances.. albeit a small amount!
>>
>>  Are you opening a new table inside each task or once up in the config?
>>>>
>>> I open HTable in the 'setup' method for each mapper/reducer, and close
>> table
>> in the 'cleanup' method.
>>
>>  You have to temper the above general rule with the fact that...
>>>>
>>> I will try a few combinations.
>>
>>  How big is your dataset?
>>>>
>>> This one in particular is not big, but the real production ones will be.
>>  Here's approximately how many rows get processed:
>> Phase 1:  300 rows
>> Phase 2 thru 8:  100 rows.
>> (Note:  Each phase does complex calculations on the row.)
>>
>> Thanks for the help.
>>
>>
>> On Tue, Jan 26, 2010 at 10:36 AM, Jean-Daniel Cryans <jdcryans@apache.org
>> >wrote:
>>
>>  How big is your dataset?
>>>
>>> J-D
>>>
>>> On Tue, Jan 26, 2010 at 8:47 AM, Something Something
>>> <mailinglists19@gmail.com> wrote:
>>>
>>>> I have noticed some strange performance numbers on EC2.  If someone can
>>>>
>>> give
>>>
>>>> me some hints to improve performance that would be greatly appreciated.
>>>>  Here are the details:
>>>>
>>>> I have a process that runs a series of Jobs under Hadoop 0.20.1 & Hbase
>>>> 0.20.2  I ran the *exact* same process with following configurations:
>>>>
>>>> 1) 1 Master & 4 Workers (*c1.xlarge* instances) & 1 Zookeeper
>>>>
>>> (*c1.medium*)
>>>
>>>> with *8 Reducers *for every Reduce task.  The process completed in *849*
>>>>  seconds.
>>>>
>>>> 2) 1 Master, 4 Workers & 1 Zookeeper  *ALL m1.small* instances with *8
>>>> Reducers *for every Reduce task.  The process completed in *906*
>>>> seconds.
>>>>
>>>> 3) 1 Master, *11* Workers & *3* Zookeepers  *ALL m1.small* instances
>>>> with
>>>>
>>> *20
>>>
>>>> Reducers *for every Reduce task.  The process completed in *984*
>>>> seconds!
>>>>
>>>>
>>>> Two main questions:
>>>>
>>>> 1)  It's totally surprising that when I have 11 workers with 20 Reducers
>>>>
>>> it
>>>
>>>> runs slower than when I have exactly same type of fewer machines with
>>>>
>>> fewer
>>>
>>>> reducers..
>>>> 2)  As expected it runs faster on c1.xlarge, but the performance
>>>>
>>> improvement
>>>
>>>> doesn't justify the high cost difference.  I must not be utilizing the
>>>> machine power, but I don't know how to do that.
>>>>
>>>> Here are some of the performance improvements tricks that I have learnt
>>>>
>>> from
>>>
>>>> this mailing list in the past that I am using:
>>>>
>>>> 1)  conf.set("hbase.client.scanner.caching", "30");   I have this for
>>>> all
>>>> jobs.
>>>>
>>>> 2)  Using the following code every time I open a HTable:
>>>>       this.table = new HTable(new HBaseConfiguration(), "tablenameXYZ");
>>>>       table.setAutoFlush(false);
>>>>       table.setWriteBufferSize(1024 * 1024 * 12);
>>>>
>>>> 3)  For every Put I do this:
>>>>         Put put = new Put(Bytes.toBytes(out));
>>>>         put.setWriteToWAL(false);
>>>>
>>>> 4)  Change the No. of Reducers as per the No. of Workers.  I believe the
>>>> formula is:  # of workers * 1.75.
>>>>
>>>> Any other hints?  As always, greatly appreciate the help.  Thanks.
>>>>
>>>>
>>

--0016e6d6453a359a7a047e176c6d--

From general-return-987-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 27 22:13:01 2010
Return-Path: <general-return-987-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25910 invoked from network); 27 Jan 2010 22:13:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jan 2010 22:13:01 -0000
Received: (qmail 55532 invoked by uid 500); 27 Jan 2010 22:13:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55448 invoked by uid 500); 27 Jan 2010 22:12:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55438 invoked by uid 99); 27 Jan 2010 22:12:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Jan 2010 22:12:59 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of owen.omalley@gmail.com designates 72.14.220.155 as permitted sender)
Received: from [72.14.220.155] (HELO fg-out-1718.google.com) (72.14.220.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Jan 2010 22:12:49 +0000
Received: by fg-out-1718.google.com with SMTP id e21so45023fga.11
        for <general@hadoop.apache.org>; Wed, 27 Jan 2010 14:12:29 -0800 (PST)
Received: by 10.87.55.18 with SMTP id h18mr2085691fgk.65.1264630342362;
        Wed, 27 Jan 2010 14:12:22 -0800 (PST)
Received: from wlan-snve-152-48.corp.yahoo.com (nat-dip11.cfw-b-gci.corp.yahoo.com [209.131.62.146])
        by mx.google.com with ESMTPS id l12sm1068216fgb.10.2010.01.27.14.12.20
        (version=SSLv3 cipher=RC4-MD5);
        Wed, 27 Jan 2010 14:12:21 -0800 (PST)
Message-Id: <109E2151-6161-45C3-972F-2FA857CF8AA6@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <4B5F3124.4090004@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: avro in mapreduce
Date: Wed, 27 Jan 2010 14:12:17 -0800
References: <4B5F3124.4090004@apache.org>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

On Jan 26, 2010, at 10:15 AM, Doug Cutting wrote:

> This is a key link in a series of issues involved in integrating  
> Avro in
> Mapreduce.

Getting Avro types passing through MapReduce is a good goal.

I apologize for not seeing the issue before it was committed. I accept  
some of the blame for that because I've buried in Hadoop emails. That  
said, it is important to realize that with changes that radically  
change the user's interaction with the framework require a lot of  
discussion. This jira, as you've admitted, had a very unrepresentative  
subject and description, which means that very few people were  
following it. Additionally, there has been no design document on the  
change to the MapReduce framework's paradigm, so it wasn't clear what  
you were doing until this patch was committed.  Such a large change  
should have been highlighted on the public dev lists. In the future, I  
would strongly suggest all developers planning on making massive  
incompatible changes to post a design document on the public dev lists  
outside of Jira to ensure the discussion happens before instead of  
after the patch has been committed.

In terms of reverting the patch, I had fundamental issues with the  
changes and felt that we needed more time to discuss them. Allowing  
the patch to stay in trunk would bake it further and further in and  
make reverting it much harder.

I've listed my issues on the jira, but at a high level my concerns are:

1. Changing API compatibility is very expensive.
2. Changing the semantics is even more expensive.
3. We are discussing several alternatives on the jira.

Unlike Python and Linux, Apache has a democratic process and we have  
to work together to build consensus. The Apache rules are that a  
single -1 from a committer blocks the change from being made.  
Occasionally that has cost us a lot of time. For example, a single -1  
from a committer on an implementation detail of the symlink patch  
blocked it for more than a year. We need to work together to find a  
solution that everyone can live with.

-- Owen

From general-return-988-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 27 23:23:26 2010
Return-Path: <general-return-988-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50287 invoked from network); 27 Jan 2010 23:23:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jan 2010 23:23:26 -0000
Received: (qmail 34085 invoked by uid 500); 27 Jan 2010 23:23:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34020 invoked by uid 500); 27 Jan 2010 23:23:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34009 invoked by uid 99); 27 Jan 2010 23:23:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Jan 2010 23:23:25 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cutting@gmail.com designates 209.85.160.50 as permitted sender)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Jan 2010 23:23:17 +0000
Received: by pwi20 with SMTP id 20so66694pwi.29
        for <general@hadoop.apache.org>; Wed, 27 Jan 2010 15:22:57 -0800 (PST)
Received: by 10.140.87.23 with SMTP id k23mr7049215rvb.292.1264634577316;
        Wed, 27 Jan 2010 15:22:57 -0800 (PST)
Received: from ?10.32.99.70? ([166.205.139.157])
        by mx.google.com with ESMTPS id 21sm280250pzk.7.2010.01.27.15.22.54
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Wed, 27 Jan 2010 15:22:56 -0800 (PST)
Message-Id: <25503BF5-32A5-4A0D-BB0A-5CD34D5F163A@apache.org>
From: Doug Cutting <cutting@apache.org>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <109E2151-6161-45C3-972F-2FA857CF8AA6@apache.org>
Content-Type: text/plain;
	charset=us-ascii;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit
X-Mailer: iPhone Mail (7D11)
Mime-Version: 1.0 (iPhone Mail 7D11)
Subject: Re: avro in mapreduce
Date: Wed, 27 Jan 2010 15:22:48 -0800
References: <4B5F3124.4090004@apache.org> <109E2151-6161-45C3-972F-2FA857CF8AA6@apache.org>

Since you raise it, my veto on the symlink patch was to one aspect for  
which implemented alternatives existed that no one vetoed. My  
understanding is that the reason this was delayed a year was primarily  
the absence of good tests. The issue was rapidly revived once Eli  
started addressing the tests. A compromise to the vetoed issue was  
rapidly found. So I don't see that my veto was the primary source of  
delay on that issue. Also, that veto was of one point in a large patch  
that had not yet been committed that I had been actively involved in  
the design of. Plus Dhruba, its primary author, while he did not veto,  
did agree with my view, so I was not alone in it.

[Sent from mobile]

On Jan 27, 2010, at 2:12 PM, Owen O'Malley <omalley@apache.org> wrote:

> On Jan 26, 2010, at 10:15 AM, Doug Cutting wrote:
>
>> This is a key link in a series of issues involved in integrating  
>> Avro in
>> Mapreduce.
>
> Getting Avro types passing through MapReduce is a good goal.
>
> I apologize for not seeing the issue before it was committed. I  
> accept some of the blame for that because I've buried in Hadoop  
> emails. That said, it is important to realize that with changes that  
> radically change the user's interaction with the framework require a  
> lot of discussion. This jira, as you've admitted, had a very  
> unrepresentative subject and description, which means that very few  
> people were following it. Additionally, there has been no design  
> document on the change to the MapReduce framework's paradigm, so it  
> wasn't clear what you were doing until this patch was committed.   
> Such a large change should have been highlighted on the public dev  
> lists. In the future, I would strongly suggest all developers  
> planning on making massive incompatible changes to post a design  
> document on the public dev lists outside of Jira to ensure the  
> discussion happens before instead of after the patch has been  
> committed.
>
> In terms of reverting the patch, I had fundamental issues with the  
> changes and felt that we needed more time to discuss them. Allowing  
> the patch to stay in trunk would bake it further and further in and  
> make reverting it much harder.
>
> I've listed my issues on the jira, but at a high level my concerns  
> are:
>
> 1. Changing API compatibility is very expensive.
> 2. Changing the semantics is even more expensive.
> 3. We are discussing several alternatives on the jira.
>
> Unlike Python and Linux, Apache has a democratic process and we have  
> to work together to build consensus. The Apache rules are that a  
> single -1 from a committer blocks the change from being made.  
> Occasionally that has cost us a lot of time. For example, a single  
> -1 from a committer on an implementation detail of the symlink patch  
> blocked it for more than a year. We need to work together to find a  
> solution that everyone can live with.
>
> -- Owen

From general-return-989-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 28 06:04:53 2010
Return-Path: <general-return-989-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83111 invoked from network); 28 Jan 2010 06:04:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Jan 2010 06:04:53 -0000
Received: (qmail 2348 invoked by uid 500); 28 Jan 2010 06:04:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2149 invoked by uid 500); 28 Jan 2010 06:04:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2120 invoked by uid 99); 28 Jan 2010 06:04:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Jan 2010 06:04:51 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of alex.baranov.v@gmail.com designates 209.85.210.199 as permitted sender)
Received: from [209.85.210.199] (HELO mail-yx0-f199.google.com) (209.85.210.199)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Jan 2010 06:04:45 +0000
Received: by yxe37 with SMTP id 37so330738yxe.31
        for <multiple recipients>; Wed, 27 Jan 2010 22:04:24 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:cc:content-type;
        bh=gz5H5r0WkNRc+0LxJn3eaNuB0XSWPrLCGPbECcHWjr8=;
        b=qWB1qT1JnSjSvJWvyFC5ZrXL0pc/SvOytIzUwEPikxFgnk7BmTvbNxpnuu3WH65vL0
         s53/+ItUiTA2NPnDtT0FoyISPogkDcjo5zq+y678TzY4ygj8PUz9KF3DgMLC13Jykyg/
         ZuWBmcyG67qxtxaxlDs97DJrmXospS573bYB4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=FJbxRDnI5MF1+lQd7R/5CcDtIX5c9DVWoioE5TZ1ZAcVeQX+lJbN8uPz+Eox0e76GM
         s4ecF3AtZUTKAT6z3xa3LVyw0umY+Ebj5oQkITv/53RNVtSMfUiYhwFMgMiGXSevv8Uw
         QGkwv3snnm9C0EBZ28EfdU10/I6/dHA4uYh5Y=
MIME-Version: 1.0
Received: by 10.101.11.40 with SMTP id o40mr6091634ani.143.1264658664093; Wed, 
	27 Jan 2010 22:04:24 -0800 (PST)
In-Reply-To: <1eabbac31001251522s2301f263rb959c768c2ee4d1@mail.gmail.com>
References: <1eabbac31001251522s2301f263rb959c768c2ee4d1@mail.gmail.com>
Date: Thu, 28 Jan 2010 08:04:24 +0200
Message-ID: <35dbe9101001272204x2df5c911xcd5d75703866f56@mail.gmail.com>
Subject: Re: setNumReduceTasks(1)
From: Alex Baranov <alex.baranov.v@gmail.com>
To: hbase-user@hadoop.apache.org
Cc: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=005045029fe06f7756047e334a5d

--005045029fe06f7756047e334a5d
Content-Type: text/plain; charset=ISO-8859-1

Since MapReduce programming model defines only one "communication" point
between jobs - the one that occurs after all Map tasks are done and before
Reduce tasks begin I believe that anyway the solution to your problem will
come at a price of lower performance.

Although I don't think that while having 10 workers you should use
"setNumReduceTasks(1)" tactics since the performance will be very degraded.
Of course this depends on N number a lot: if N is quite small and everything
you're going to output from your reduce task is going to lay down in one
datanode then *may be* your strategy can be considered.

If N is really very big and there is lot of work to do before Reducers
should stop, then I'd consider communication throught storing the info about
a progress in DFS (implementation will not be straightforward though, since
we don't want to affect performance a lot).

Alex.

On Tue, Jan 26, 2010 at 1:22 AM, Something Something <
mailinglists19@gmail.com> wrote:

> If I set # of reduce tasks to 1 using setNumReduceTasks(1), would the class
> be instantiated only on one machine.. always?  I mean if I have a cluster
> of
> say 1 master, 10 workers & 3 zookeepers, is the Reducer class guaranteed to
> be instantiated only on 1 machine?
>
> If answer is yes, then I will use static variable as a counter to see how
> may rows have been added to my HBase table so far.  In my use case, I want
> to write only N number of rows to a table.  Is there a better way to do
> this?  Please let me know.  Thanks.
>

--005045029fe06f7756047e334a5d--

From general-return-990-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 28 07:27:56 2010
Return-Path: <general-return-990-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12341 invoked from network); 28 Jan 2010 07:27:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Jan 2010 07:27:56 -0000
Received: (qmail 59064 invoked by uid 500); 28 Jan 2010 07:27:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58976 invoked by uid 500); 28 Jan 2010 07:27:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 74308 invoked by uid 99); 26 Jan 2010 07:09:39 -0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:cc:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=zJSpUPD1Y498S5YIcPwQw3vu7ka3avy65E0TWn9Sl6D0MNEWW6uE6glv8ABzNLuH
Message-ID: <4B5E950A.1040306@yahoo-inc.com>
Date: Tue, 26 Jan 2010 12:38:58 +0530
From: Mridul Muralidharan <mridulm@yahoo-inc.com>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: "hbase-user@hadoop.apache.org" <hbase-user@hadoop.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: setNumReduceTasks(1)
References: <1eabbac31001251522s2301f263rb959c768c2ee4d1@mail.gmail.com> <8211a1321001251609r19e737e2s2fdd27f379aeea53@mail.gmail.com>
In-Reply-To: <8211a1321001251609r19e737e2s2fdd27f379aeea53@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Jeff Zhang wrote:
> *See my comments below*
> 
> On Mon, Jan 25, 2010 at 3:22 PM, Something Something <
> mailinglists19@gmail.com> wrote:
> 
>> If I set # of reduce tasks to 1 using setNumReduceTasks(1), would the class
>> be instantiated only on one machine.. always?  I mean if I have a cluster
>> of
>> say 1 master, 10 workers & 3 zookeepers, is the Reducer class guaranteed to
>> be instantiated only on 1 machine?
>>
>> *--Yes*
> 
> 
>> If answer is yes, then I will use static variable as a counter to see how
>> may rows have been added to my HBase table so far.  In my use case, I want
>> to write only N number of rows to a table.  Is there a better way to do
>> this?  Please let me know.  Thanks.
>>
> 
> *--Maybe you can use Counter to track the number of rows you add to HBase,
> then you do not need to limit the reduce task as 1*
> 
> 

Counter's are not synchronized in 'real-time' : so you cant use that to 
limit at addition time imo.
It is more for aggregation, not realtime messaging.

- Mridul

From general-return-991-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 28 07:28:07 2010
Return-Path: <general-return-991-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12424 invoked from network); 28 Jan 2010 07:28:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Jan 2010 07:28:07 -0000
Received: (qmail 59963 invoked by uid 500); 28 Jan 2010 07:28:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59903 invoked by uid 500); 28 Jan 2010 07:28:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 66150 invoked by uid 99); 27 Jan 2010 01:02:08 -0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 221941.18932.bm@omp124.mail.ac4.yahoo.com
Message-ID: <973807.67980.qm@web65509.mail.ac4.yahoo.com>
X-YMail-OSG: ZbwDR64VM1lwZu04CK.ug_58pfnXjqcfC7_.bmAsCygu_0_kRBLvlIgKEXX5hlENTPbMX44wDL0JG7f7TgK5Wx1IiTMUoX6j2ZlRqq2XT_YTVW5ip3hGkxy4iVlu6M0sBgQ_pxqMg5RDUN.gFzuMpKfUMwp75ho33GBTWllyr2eC99Pc1KHleYrnIBY8wQMQQ06wex2_qBYfipa75_QHdEdYB7fBk0imimEFFnbnr5AHGQkx3Q_9CbbqUdHxf7yqmdz8rASbrEcLHcp1EAXm7Xfu4.KUjjEGf54wGJNi8L2YwjwGFKpYHPfd8ECb03yR.TQFsb4KftFvuFIWTGXEqt0RctoetSMikXxOE7piRtndgcjCjiLVc0LkTrys7BOXJJth.fizbuDWXRcq65KKjUpWCK.kOMkdIv9niYX8qqnEV7351nJ7S7_7sJBDTUmVvBm6L0bIZvoSWXI-
X-RocketYMMF: apurtell
X-Mailer: YahooMailRC/272.7 YahooMailWebService/0.8.100.260964
References: <1eabbac31001260847g3a726f62o3a89ee2d74096e6e@mail.gmail.com>  <31a243e71001261036t7d47bb7fo387e079cc8ed0974@mail.gmail.com> <1eabbac31001261120m74ee4817x3751592c930fd98b@mail.gmail.com> <4B5F4610.8050803@apache.org>
Date: Tue, 26 Jan 2010 17:01:36 -0800 (PST)
From: Andrew Purtell <apurtell@apache.org>
Subject: Re: Performance of EC2
To: hbase-user@hadoop.apache.org, general@hadoop.apache.org
Cc: hbase-user@hadoop.apache.org
In-Reply-To: <4B5F4610.8050803@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii

I have observed "noisy neighbor" effects. 

If you are using HBase EC2 scripts, which run HBase region servers on all of
the slaves colocated with tasktrackers and user tasks, I do not recommend using
other than c1.xlarge instances. 

Our scripts use c1.medium instances for the separate Zookeeper quorum ensemble
as they need fewer resources in terms of RAM but are still sensitive to io and
cpu scheduling latencies. 

   - Andy



----- Original Message ----
> From: Patrick Hunt <phunt@apache.org>
> To: general@hadoop.apache.org
> Cc: hbase-user@hadoop.apache.org
> Sent: Wed, January 27, 2010 3:44:16 AM
> Subject: Re: Performance of EC2
> 
> Re "Amazon predictability", did you guys see this recent paper:
> http://people.csail.mit.edu/tromer/cloudsec/
> 
> Also some addl background on "noisy neighbor effects":
> http://bit.ly/4O7dHx
> http://bit.ly/8zPvQd
> 
> Some interesting bits of information in there.
> 
> Patrick
> 
> Something Something wrote:
> > Here are some of the answers:
> > 
> >>>  How many concurrent reducers run on each node?  Default two?
> > I was assuming 2 on each node would be the default.  If not, this could be a
> > problem.  Please let me know.
> > 
> >>> 'd suggest you spend a bit of time figuring where your MR jobs
> > are spending their time?
> > I agree.  Will do some more research :)
> > 
> >>> How much of this overall time is spent in reduce phase?
> > Mostly time is spent in the Reduce phases, because that's where most of the
> > critical code is.
> > 
> >>> Are inserts to a new table?
> > Yes, all inserts will always be in a new table.  In fact, I disable/drop
> > HTables during this process.  Not using any special indexes, should I be?
> > 
> >>> I'm a little surprised that all worked on the small instances, that your
> > jobs completed.
> > But, really, shouldn't Amazon guarantee predictability :)  After all I am
> > paying for these instances.. albeit a small amount!
> > 
> >>> Are you opening a new table inside each task or once up in the config?
> > I open HTable in the 'setup' method for each mapper/reducer, and close table
> > in the 'cleanup' method.
> > 
> >>> You have to temper the above general rule with the fact that...
> > I will try a few combinations.
> > 
> >>> How big is your dataset?
> > This one in particular is not big, but the real production ones will be.
> >  Here's approximately how many rows get processed:
> > Phase 1:  300 rows
> > Phase 2 thru 8:  100 rows.
> > (Note:  Each phase does complex calculations on the row.)
> > 
> > Thanks for the help.
> > 
> > 
> > On Tue, Jan 26, 2010 at 10:36 AM, Jean-Daniel Cryans 
> wrote:
> > 
> >> How big is your dataset?
> >>
> >> J-D
> >>
> >> On Tue, Jan 26, 2010 at 8:47 AM, Something Something
> >> wrote:
> >>> I have noticed some strange performance numbers on EC2.  If someone can
> >> give
> >>> me some hints to improve performance that would be greatly appreciated.
> >>>  Here are the details:
> >>>
> >>> I have a process that runs a series of Jobs under Hadoop 0.20.1 & Hbase
> >>> 0.20.2  I ran the *exact* same process with following configurations:
> >>>
> >>> 1) 1 Master & 4 Workers (*c1.xlarge* instances) & 1 Zookeeper
> >> (*c1.medium*)
> >>> with *8 Reducers *for every Reduce task.  The process completed in *849*
> >>>  seconds.
> >>>
> >>> 2) 1 Master, 4 Workers & 1 Zookeeper  *ALL m1.small* instances with *8
> >>> Reducers *for every Reduce task.  The process completed in *906* seconds.
> >>>
> >>> 3) 1 Master, *11* Workers & *3* Zookeepers  *ALL m1.small* instances with
> >> *20
> >>> Reducers *for every Reduce task.  The process completed in *984* seconds!
> >>>
> >>>
> >>> Two main questions:
> >>>
> >>> 1)  It's totally surprising that when I have 11 workers with 20 Reducers
> >> it
> >>> runs slower than when I have exactly same type of fewer machines with
> >> fewer
> >>> reducers..
> >>> 2)  As expected it runs faster on c1.xlarge, but the performance
> >> improvement
> >>> doesn't justify the high cost difference.  I must not be utilizing the
> >>> machine power, but I don't know how to do that.
> >>>
> >>> Here are some of the performance improvements tricks that I have learnt
> >> from
> >>> this mailing list in the past that I am using:
> >>>
> >>> 1)  conf.set("hbase.client.scanner.caching", "30");   I have this for all
> >>> jobs.
> >>>
> >>> 2)  Using the following code every time I open a HTable:
> >>>        this.table = new HTable(new HBaseConfiguration(), "tablenameXYZ");
> >>>        table.setAutoFlush(false);
> >>>        table.setWriteBufferSize(1024 * 1024 * 12);
> >>>
> >>> 3)  For every Put I do this:
> >>>          Put put = new Put(Bytes.toBytes(out));
> >>>          put.setWriteToWAL(false);
> >>>
> >>> 4)  Change the No. of Reducers as per the No. of Workers.  I believe the
> >>> formula is:  # of workers * 1.75.
> >>>
> >>> Any other hints?  As always, greatly appreciate the help.  Thanks.
> >>>
> > 



      


From general-return-992-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 28 07:28:18 2010
Return-Path: <general-return-992-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12493 invoked from network); 28 Jan 2010 07:28:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Jan 2010 07:28:18 -0000
Received: (qmail 60820 invoked by uid 500); 28 Jan 2010 07:28:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60772 invoked by uid 500); 28 Jan 2010 07:28:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 47671 invoked by uid 99); 28 Jan 2010 07:09:57 -0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:cc:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=yDglIuIi0FHTucntkt8aGMnH4AakCWkzoLeK6OE4NnfoLrMJJ5EpduIZK7Ifm8G0
Message-ID: <4B61381E.6010805@yahoo-inc.com>
Date: Thu, 28 Jan 2010 12:39:18 +0530
From: Mridul Muralidharan <mridulm@yahoo-inc.com>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: "hbase-user@hadoop.apache.org" <hbase-user@hadoop.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: setNumReduceTasks(1)
References: <1eabbac31001251522s2301f263rb959c768c2ee4d1@mail.gmail.com>
In-Reply-To: <1eabbac31001251522s2301f263rb959c768c2ee4d1@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit


A possible solution is to emit only N rows from each mapper and then use 
1 reduce task [*] - if value of N is not very high.
So you end up with utmost m * N rows on reducer instead of full inputset 
- and so the limit can be done easier.


If you ok with some sort of variance in the number of rows inserted (and 
if value of N is very high), you can do more interesting things like 
N/m' rows per mapper - and multiple reducers (r) : with assumtion that 
each reducer will see atleast N/r rows - and so you can limit to N/r per 
reducer : ofcourse, there is a possible error that gets introduced here ...


Regards,
Mridul

[*] Assuming you just want simple limit - nothing else.
Also note, each mapper might want to emit N rows instead of 'tweaks' 
like N/m rows, since it is possible that multiple mappers might have 
less than N/m rows to emit to begin with !


Something Something wrote:
> If I set # of reduce tasks to 1 using setNumReduceTasks(1), would the class
> be instantiated only on one machine.. always?  I mean if I have a cluster of
> say 1 master, 10 workers & 3 zookeepers, is the Reducer class guaranteed to
> be instantiated only on 1 machine?
> 
> If answer is yes, then I will use static variable as a counter to see how
> may rows have been added to my HBase table so far.  In my use case, I want
> to write only N number of rows to a table.  Is there a better way to do
> this?  Please let me know.  Thanks.


From general-return-993-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 28 08:30:46 2010
Return-Path: <general-return-993-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 26231 invoked from network); 28 Jan 2010 08:30:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Jan 2010 08:30:46 -0000
Received: (qmail 34800 invoked by uid 500); 28 Jan 2010 08:30:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34709 invoked by uid 500); 28 Jan 2010 08:30:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34563 invoked by uid 99); 28 Jan 2010 08:30:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Jan 2010 08:30:44 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Jan 2010 08:30:35 +0000
Received: from [10.72.168.35] (snvvpn4-10-72-168-c35.hq.corp.yahoo.com [10.72.168.35])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o0S8Tc0S012795;
	Thu, 28 Jan 2010 00:29:38 -0800 (PST)
Message-ID: <4B614AF2.2040401@apache.org>
Date: Thu, 28 Jan 2010 00:29:38 -0800
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
CC: hbase-user@hadoop.apache.org
Subject: Re: Performance of EC2
References: <1eabbac31001260847g3a726f62o3a89ee2d74096e6e@mail.gmail.com>	 <31a243e71001261036t7d47bb7fo387e079cc8ed0974@mail.gmail.com>	 <1eabbac31001261120m74ee4817x3751592c930fd98b@mail.gmail.com>	 <4B5F4610.8050803@apache.org> <1eabbac31001261249s4b09fbdfyb886927b36c2953e@mail.gmail.com>
In-Reply-To: <1eabbac31001261249s4b09fbdfyb886927b36c2953e@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

FYI, just noticed this one:

Rackspace Cloud Servers versus Amazon EC2: Performance Analysis 
http://bit.ly/bkG1AB

Patrick

Something Something wrote:
> Wow.. how naive I am to think that I could trust Amazon.  Thanks for
> forwarding the links, Patrick.  Seems like Amazon's reliability has gone
> down considerably over the past few months.  (Occasionally my instances fail
> on startup or die in the middle for no apparent reason, and I used to think
> I was doing something dumb!)
> 
> But what I don't understand is this... if I *reserve* an instance then I
> wouldn't be sharing its CPU with anyone, right?  The blog seems to indicate
> otherwise.
> 
> I guess, I will have to look for alternatives to Amazon EC2.  Any one has
> any recommendations?  Thanks again.
> 
> 
> On Tue, Jan 26, 2010 at 11:44 AM, Patrick Hunt <phunt@apache.org> wrote:
> 
>> Re "Amazon predictability", did you guys see this recent paper:
>> http://people.csail.mit.edu/tromer/cloudsec/
>>
>> Also some addl background on "noisy neighbor effects":
>> http://bit.ly/4O7dHx
>> http://bit.ly/8zPvQd
>>
>> Some interesting bits of information in there.
>>
>> Patrick
>>
>>
>> Something Something wrote:
>>
>>> Here are some of the answers:
>>>
>>>   How many concurrent reducers run on each node?  Default two?
>>>> I was assuming 2 on each node would be the default.  If not, this could
>>> be a
>>> problem.  Please let me know.
>>>
>>>  'd suggest you spend a bit of time figuring where your MR jobs
>>>> are spending their time?
>>> I agree.  Will do some more research :)
>>>
>>>  How much of this overall time is spent in reduce phase?
>>>> Mostly time is spent in the Reduce phases, because that's where most of
>>> the
>>> critical code is.
>>>
>>>  Are inserts to a new table?
>>>> Yes, all inserts will always be in a new table.  In fact, I disable/drop
>>> HTables during this process.  Not using any special indexes, should I be?
>>>
>>>  I'm a little surprised that all worked on the small instances, that your
>>>> jobs completed.
>>> But, really, shouldn't Amazon guarantee predictability :)  After all I am
>>> paying for these instances.. albeit a small amount!
>>>
>>>  Are you opening a new table inside each task or once up in the config?
>>>> I open HTable in the 'setup' method for each mapper/reducer, and close
>>> table
>>> in the 'cleanup' method.
>>>
>>>  You have to temper the above general rule with the fact that...
>>>> I will try a few combinations.
>>>  How big is your dataset?
>>>> This one in particular is not big, but the real production ones will be.
>>>  Here's approximately how many rows get processed:
>>> Phase 1:  300 rows
>>> Phase 2 thru 8:  100 rows.
>>> (Note:  Each phase does complex calculations on the row.)
>>>
>>> Thanks for the help.
>>>
>>>
>>> On Tue, Jan 26, 2010 at 10:36 AM, Jean-Daniel Cryans <jdcryans@apache.org
>>>> wrote:
>>>  How big is your dataset?
>>>> J-D
>>>>
>>>> On Tue, Jan 26, 2010 at 8:47 AM, Something Something
>>>> <mailinglists19@gmail.com> wrote:
>>>>
>>>>> I have noticed some strange performance numbers on EC2.  If someone can
>>>>>
>>>> give
>>>>
>>>>> me some hints to improve performance that would be greatly appreciated.
>>>>>  Here are the details:
>>>>>
>>>>> I have a process that runs a series of Jobs under Hadoop 0.20.1 & Hbase
>>>>> 0.20.2  I ran the *exact* same process with following configurations:
>>>>>
>>>>> 1) 1 Master & 4 Workers (*c1.xlarge* instances) & 1 Zookeeper
>>>>>
>>>> (*c1.medium*)
>>>>
>>>>> with *8 Reducers *for every Reduce task.  The process completed in *849*
>>>>>  seconds.
>>>>>
>>>>> 2) 1 Master, 4 Workers & 1 Zookeeper  *ALL m1.small* instances with *8
>>>>> Reducers *for every Reduce task.  The process completed in *906*
>>>>> seconds.
>>>>>
>>>>> 3) 1 Master, *11* Workers & *3* Zookeepers  *ALL m1.small* instances
>>>>> with
>>>>>
>>>> *20
>>>>
>>>>> Reducers *for every Reduce task.  The process completed in *984*
>>>>> seconds!
>>>>>
>>>>>
>>>>> Two main questions:
>>>>>
>>>>> 1)  It's totally surprising that when I have 11 workers with 20 Reducers
>>>>>
>>>> it
>>>>
>>>>> runs slower than when I have exactly same type of fewer machines with
>>>>>
>>>> fewer
>>>>
>>>>> reducers..
>>>>> 2)  As expected it runs faster on c1.xlarge, but the performance
>>>>>
>>>> improvement
>>>>
>>>>> doesn't justify the high cost difference.  I must not be utilizing the
>>>>> machine power, but I don't know how to do that.
>>>>>
>>>>> Here are some of the performance improvements tricks that I have learnt
>>>>>
>>>> from
>>>>
>>>>> this mailing list in the past that I am using:
>>>>>
>>>>> 1)  conf.set("hbase.client.scanner.caching", "30");   I have this for
>>>>> all
>>>>> jobs.
>>>>>
>>>>> 2)  Using the following code every time I open a HTable:
>>>>>       this.table = new HTable(new HBaseConfiguration(), "tablenameXYZ");
>>>>>       table.setAutoFlush(false);
>>>>>       table.setWriteBufferSize(1024 * 1024 * 12);
>>>>>
>>>>> 3)  For every Put I do this:
>>>>>         Put put = new Put(Bytes.toBytes(out));
>>>>>         put.setWriteToWAL(false);
>>>>>
>>>>> 4)  Change the No. of Reducers as per the No. of Workers.  I believe the
>>>>> formula is:  # of workers * 1.75.
>>>>>
>>>>> Any other hints?  As always, greatly appreciate the help.  Thanks.
>>>>>
>>>>>
> 

From general-return-994-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 28 08:45:15 2010
Return-Path: <general-return-994-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30670 invoked from network); 28 Jan 2010 08:45:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Jan 2010 08:45:15 -0000
Received: (qmail 61914 invoked by uid 500); 28 Jan 2010 08:45:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61836 invoked by uid 500); 28 Jan 2010 08:45:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61826 invoked by uid 99); 28 Jan 2010 08:45:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Jan 2010 08:45:14 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.13.13.73] (HELO n3b.bullet.mail.ac4.yahoo.com) (76.13.13.73)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 28 Jan 2010 08:45:05 +0000
Received: from [76.13.13.25] by n3.bullet.mail.ac4.yahoo.com with NNFMP; 28 Jan 2010 08:44:43 -0000
Received: from [76.13.10.163] by t4.bullet.mail.ac4.yahoo.com with NNFMP; 28 Jan 2010 08:44:43 -0000
Received: from [127.0.0.1] by omp104.mail.ac4.yahoo.com with NNFMP; 28 Jan 2010 08:44:43 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 797261.72291.bm@omp104.mail.ac4.yahoo.com
Received: (qmail 71464 invoked by uid 60001); 28 Jan 2010 08:44:43 -0000
Message-ID: <547544.70925.qm@web65512.mail.ac4.yahoo.com>
X-YMail-OSG: xyT17zMVM1npPUZbaFJYDK2kxNHAPwQ7TkFhZ61B6CXIsNqNahX5wYSEzWp3TpxQf2FKvl.TLQWWBrYRrLfXQJb0lpq1N4m009MSb.F6STU5_sgwWOMRMHT8pJ9aqu2vQHkcXVzNJ.FGajqmv5PiqxvZyIs838JQGQtcG2wHqNs6VZ8mRe4R4CB2HwEE0_t3LibuDiDn7THzJxOChR8Pm9QtyzgU0_gYDwh78SMckz0JkJvYK.Uanq_kqX9tyxnhjQyt7CyhjgzOFfKEZ8trduoFEKgYpW3buu0jdV6_FX.EUEGYNPLo9ILDKfQV.RMvVPabGYLCpx3XAeDHj3pJcW6iluczVLjriwP3QQ--
Received: from [66.135.63.56] by web65512.mail.ac4.yahoo.com via HTTP; Thu, 28 Jan 2010 00:44:43 PST
X-RocketYMMF: apurtell
X-Mailer: YahooMailRC/272.7 YahooMailWebService/0.8.100.260964
References: <1eabbac31001260847g3a726f62o3a89ee2d74096e6e@mail.gmail.com>  <31a243e71001261036t7d47bb7fo387e079cc8ed0974@mail.gmail.com>  <1eabbac31001261120m74ee4817x3751592c930fd98b@mail.gmail.com>  <4B5F4610.8050803@apache.org> <1eabbac31001261249s4b09fbdfyb886927b36c2953e@mail.gmail.com> <4B614AF2.2040401@apache.org>
Date: Thu, 28 Jan 2010 00:44:43 -0800 (PST)
From: Andrew Purtell <apurtell@apache.org>
Subject: Re: Performance of EC2
To: hbase-user@hadoop.apache.org, general@hadoop.apache.org
Cc: hbase-user@hadoop.apache.org
In-Reply-To: <4B614AF2.2040401@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii

> > But what I don't understand is this... if I *reserve* an instance then I
> > wouldn't be sharing its CPU with anyone, right?  The blog seems to indicate
> > otherwise.

No, that is not how EC2 and similar cloud infrastructure platforms work. You are
given a virtual machine. The hypervisor multiplexes many virtual machines on a
single physical server. The instance types purport to guarantee a certain level
of performance measured in virtual CPU units. The EC2 website describes in detail
how Amazon defines a virtual CPU unit. 


> > (Occasionally my instances fail
> > on startup or die in the middle for no apparent reason, and I used to think
> > I was doing something dumb!)

That is also a misunderstanding of how to use cloud infrastructure. No single
instance is reliable. The platform is a commodity, cheap, with basic usability.
Durable services are built on top of these building blocks with service
architecture that does not rely on the stability of any single instance. 

   - Andy



      


From general-return-995-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 28 12:05:19 2010
Return-Path: <general-return-995-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 13172 invoked from network); 28 Jan 2010 12:05:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Jan 2010 12:05:19 -0000
Received: (qmail 39121 invoked by uid 500); 28 Jan 2010 12:05:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39040 invoked by uid 500); 28 Jan 2010 12:05:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39030 invoked by uid 99); 28 Jan 2010 12:05:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Jan 2010 12:05:17 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of lars.george@gmail.com designates 209.85.219.216 as permitted sender)
Received: from [209.85.219.216] (HELO mail-ew0-f216.google.com) (209.85.219.216)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Jan 2010 12:05:08 +0000
Received: by ewy8 with SMTP id 8so649156ewy.29
        for <general@hadoop.apache.org>; Thu, 28 Jan 2010 04:04:47 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type:content-transfer-encoding;
        bh=8nOB84QY2/SBebWDGAof07SnNJf82imO19g61M3kXcU=;
        b=EdryEgXHeHVxhpPO4RalI2d2eL+m3ZSZA8MviGdei503s2+3aKmBmq3y+6FsTLcejl
         E69hWhungRnbxR6/RoNt3ZiX93fRYxbBy4BgbtjAn1KKox4a9if8DpxfEhRcCVsSSCXQ
         4oYNMbdIokD0atOZZQ1hxT+KME0W34eKJBejk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=Q/JG6rgt8cAvkLQQVojJqY4MxOe0i+dWCcTd/GDodiLHFVBxWvUIb4sdveHm+tHw3c
         NUYdSy/FF94PnK6io2oKezeC8fYJFSSp89CT0TMZaJ/xfCLod8xCvAXMq4nhsnTSj2yd
         EZ+OWUAUBMG46zj/llpBkh496NOrDoM4qARSs=
MIME-Version: 1.0
Received: by 10.213.0.148 with SMTP id 20mr2425191ebb.51.1264680286827; Thu, 
	28 Jan 2010 04:04:46 -0800 (PST)
Date: Thu, 28 Jan 2010 13:04:46 +0100
Message-ID: <61770b881001280404j28496d90s9aba0e617e13d62a@mail.gmail.com>
Subject: 2nd Munich OpenHUG Meeting
From: Lars George <lars.george@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

2nd Munich OpenHUG Meeting

At the end of last year we had a first meeting in Munich to allow
everyone interested in all things Hadoop and related technologies to
gather, get to know each other and exchange their knowledge,
experiences and information. We would like to invite for the second
Munich OpenHUG Meeting and hope to see you all again and meet even
more new enthusiasts there. We would also be thrilled if those
attending would be willing to share their story so that we can learn
about other projects and how people are using the exciting NoSQL
related technologies. No pressure though, come along and simply listen
if you prefer, we welcome anyone and everybody!

When: Thursday February 25, 2009 at 5:30pm open end
Where: eCircle AG, Nymphenburger Stra=DFe 86, 80636 M=FCnchen ["Bruckmann"
Building, "U1 Mailinger Str", map
http://www.ecircle.com/de/kontakt/anfahrt.html (in German) and look
for the signs]

Thanks again to Bob Schulze from eCircle to provide the location and
projector. So far we have a talk scheduled by Christoph Rupp about
HamsterDB (http://hamsterdb.com/). We are still looking for volunteers
who would like to present on any related topic (please contact me at
info@larsgeorge.com)! Otherwise we will have an open discussion about
whatever is brought up by the attendees.

Last but not least there will be something to drink and we will get
pizzas in. Since we do not know how many of you will come we simply
stay at the events location and continue our chats over food.

Looking forward to seeing you there!

Please RSVP here:

http://upcoming.yahoo.com/event/5279322/BY/Mnchen/2nd-Munich-OpenHUG-Meetin=
g/eCircle-AG
https://www.xing.com/events/2nd-munich-openhug-meeting-458852

Best regards,

Lars George

From general-return-996-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 28 12:56:14 2010
Return-Path: <general-return-996-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34800 invoked from network); 28 Jan 2010 12:56:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Jan 2010 12:56:14 -0000
Received: (qmail 98870 invoked by uid 500); 28 Jan 2010 12:56:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98775 invoked by uid 500); 28 Jan 2010 12:56:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98756 invoked by uid 99); 28 Jan 2010 12:56:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Jan 2010 12:56:12 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of lars.george@gmail.com designates 209.85.219.216 as permitted sender)
Received: from [209.85.219.216] (HELO mail-ew0-f216.google.com) (209.85.219.216)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Jan 2010 12:56:01 +0000
Received: by ewy8 with SMTP id 8so703277ewy.29
        for <multiple recipients>; Thu, 28 Jan 2010 04:55:40 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=dV7cqSXmRseUNTlTQjd4XVJ17tO3CEzARaQQFulCfzM=;
        b=BQiN86vTFCraVlKk3kZD4yfrl1HTbbk+58HT4fs87zDQdg0kkd/XERdYy0nKon0y7W
         iV9hr7Sreoj3ZdOlYW2HYz3LMmH9s54OH+r4DWKnUFWNYLAp/G3HIuhUpBz/OFRIprtM
         9iCF51HmdWFXFDwT5fUem62ZYscNREPOOwpjE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=MO5Nvm15m4Uu8VyL4KhomxSfr3PNASgHxNb3XKSZlYr94RFcHyCOUBpSmuQkoVErhW
         Y0IcG4oRsevQDHhgS65Vi6fmxOwIZxjL1vFu/dZt4esm5EtbA/52RrZvhlNmmJwPN2yZ
         fPnLnReelQ7ntHPE3WNUypGG9x7jiFbbHNTFk=
MIME-Version: 1.0
Received: by 10.213.25.67 with SMTP id y3mr7936995ebb.73.1264683339676; Thu, 
	28 Jan 2010 04:55:39 -0800 (PST)
In-Reply-To: <61770b881001280403i59c6b60bs972a041c276acdd0@mail.gmail.com>
References: <61770b881001280403i59c6b60bs972a041c276acdd0@mail.gmail.com>
Date: Thu, 28 Jan 2010 13:55:39 +0100
Message-ID: <61770b881001280455s1069ad0w24304f0d06ccb2d2@mail.gmail.com>
Subject: Re: [ANN] 2nd Munich OpenHUG Meeting
From: Lars George <lars.george@gmail.com>
To: hbase-user@hadoop.apache.org, nosql-discussion@googlegroups.com, 
	general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

Sorry, the year should of course say 2010, so the meeting is Feb 25th
2010. Sorry for the typo.

Lars

On Thu, Jan 28, 2010 at 1:03 PM, Lars George <lars.george@gmail.com> wrote:
> 2nd Munich OpenHUG Meeting
>
> At the end of last year we had a first meeting in Munich to allow
> everyone interested in all things Hadoop and related technologies to
> gather, get to know each other and exchange their knowledge,
> experiences and information. We would like to invite for the second
> Munich OpenHUG Meeting and hope to see you all again and meet even
> more new enthusiasts there. We would also be thrilled if those
> attending would be willing to share their story so that we can learn
> about other projects and how people are using the exciting NoSQL
> related technologies. No pressure though, come along and simply listen
> if you prefer, we welcome anyone and everybody!
>
> When: Thursday February 25, 2009 at 5:30pm open end
> Where: eCircle AG, Nymphenburger Stra=DFe 86, 80636 M=FCnchen ["Bruckmann=
"
> Building, "U1 Mailinger Str", map
> http://www.ecircle.com/de/kontakt/anfahrt.html (in German) and look
> for the signs]
>
> Thanks again to Bob Schulze from eCircle to provide the location and
> projector. So far we have a talk scheduled by Christoph Rupp about
> HamsterDB (http://hamsterdb.com/). We are still looking for volunteers
> who would like to present on any related topic (please contact me at
> info@larsgeorge.com)! Otherwise we will have an open discussion about
> whatever is brought up by the attendees.
>
> Last but not least there will be something to drink and we will get
> pizzas in. Since we do not know how many of you will come we simply
> stay at the events location and continue our chats over food.
>
> Looking forward to seeing you there!
>
> Please RSVP here:
>
> http://upcoming.yahoo.com/event/5279322/BY/Mnchen/2nd-Munich-OpenHUG-Meet=
ing/eCircle-AG
> https://www.xing.com/events/2nd-munich-openhug-meeting-458852
>
> Best regards,
>
> Lars George
>

From general-return-997-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 29 06:12:27 2010
Return-Path: <general-return-997-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53508 invoked from network); 29 Jan 2010 06:12:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Jan 2010 06:12:27 -0000
Received: (qmail 82531 invoked by uid 500); 29 Jan 2010 06:12:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82367 invoked by uid 500); 29 Jan 2010 06:12:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 12569 invoked by uid 99); 29 Jan 2010 02:35:55 -0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1264732522; bh=BnB+aO7jYFe07bMKbBAtCkZ6YsBHIR2V3Uq4DYkum3k=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=MplwP024rvGoRx6dOThC03YOjqtwVsp3c/WOI8+TM3qnGCls7kD5a2FhO5IkwZ8Y2YWT18Y3oiI449V6dYOludSU5EfJqY+h0BGSrgi2jCdE5/vpzpyxrH4OFdDm27c1nWgRyxHfQZAwFGmFDas9eKG7ZW39Nqbz+yzY1Jt4MMQ=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=3XePBE9kgcNgaYtqhFdiLpPgo/V8SdSWQHCBTf09mw64GcUqCc/49d/ti6ARRNoC7qKcwXQsm5Nqe7CYrg2A3UPzLhByqEdz4lvUFezVoJ2vW4IN0SSahnuC3AGVMCQYfjtL+NdJvAZqIO6gcpRAOe5tC1uyVzC9mUUoZRSZhJE=;
Message-ID: <397063.76800.qm@web50302.mail.re2.yahoo.com>
X-YMail-OSG: WvBJAOQVM1kd8w7hAeEyuX1KXvZK6TxQtp3LH76kSunZqD0F.Zpb0big1gcwN.LLlP6N8RMPQupKI_0Rdop3G_74LDDC7a5lQ8ewPVsPA9ztHQUCJ9ftwzCnoj4.ksZIf.6FYOuKJ7fu0sWMx6AmAvda8tfOPFnrGRQB0hH6cTP4lkIyjncYaJqvorCqN6K4hM0SN7EfSj0y3yvB1B6c.zKa32A9TfKXxK0ivl3e5Y4oDXnUnFOZjyCdGy_44FuNlVIbMSH9r6kb2abSkTXVjZUdplEprF1WzdUfEZ2NBojIj70vijJcQOPXGdJxo9Ie0h5YsWQPH4MhzR0VYunL4cTsG5qy6jyn4zGKHkd.C08yF6V_SHNt5QgYKRaDXtU_zX0YuFX.uvHylcva2bzacFJ8r9ZHq4KhK2miGpt_woIcFW17xQ8haU3c0UvFsqENAJvA6y7iWXMmwwb_elO1CNyit9z3kgZ_J2LvHVDMUcKMizB3NisHiWAoa9xC61WrCe0ga.LFQQ8bamTgIfeNN34klRqirX4uIHNboR6f395pzXWvmI4Zu6DJK6BAWjTCagimuW2BGWjji5xMkNA-
X-Mailer: YahooMailRC/272.7 YahooMailWebService/0.8.100.260964
References: <1eabbac31001260847g3a726f62o3a89ee2d74096e6e@mail.gmail.com>  <31a243e71001261036t7d47bb7fo387e079cc8ed0974@mail.gmail.com>  <1eabbac31001261120m74ee4817x3751592c930fd98b@mail.gmail.com>  <4B5F4610.8050803@apache.org> <1eabbac31001261249s4b09fbdfyb886927b36c2953e@mail.gmail.com>
Date: Thu, 28 Jan 2010 18:35:22 -0800 (PST)
From: Otis Gospodnetic <otis_gospodnetic@yahoo.com>
Subject: Re: Performance of EC2
To: hbase-user@hadoop.apache.org, general@hadoop.apache.org
In-Reply-To: <1eabbac31001261249s4b09fbdfyb886927b36c2953e@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org

I think the reserved EC2 instances just give you a better deal price-wise in exchange for an advanced payment and, essentially, a contract.  I didn't see any mentions of reserved instances mean no sharing.  If AWS did that, they'd be nothing more than a regular hosting service.

 Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Hadoop ecosystem search :: http://search-hadoop.com/



----- Original Message ----
> From: Something Something <mailinglists19@gmail.com>
> To: general@hadoop.apache.org
> Cc: hbase-user@hadoop.apache.org
> Sent: Tue, January 26, 2010 3:49:31 PM
> Subject: Re: Performance of EC2
> 
> Wow.. how naive I am to think that I could trust Amazon.  Thanks for
> forwarding the links, Patrick.  Seems like Amazon's reliability has gone
> down considerably over the past few months.  (Occasionally my instances fail
> on startup or die in the middle for no apparent reason, and I used to think
> I was doing something dumb!)
> 
> But what I don't understand is this... if I *reserve* an instance then I
> wouldn't be sharing its CPU with anyone, right?  The blog seems to indicate
> otherwise.
> 
> I guess, I will have to look for alternatives to Amazon EC2.  Any one has
> any recommendations?  Thanks again.
> 
> 
> On Tue, Jan 26, 2010 at 11:44 AM, Patrick Hunt wrote:
> 
> > Re "Amazon predictability", did you guys see this recent paper:
> > http://people.csail.mit.edu/tromer/cloudsec/
> >
> > Also some addl background on "noisy neighbor effects":
> > http://bit.ly/4O7dHx
> > http://bit.ly/8zPvQd
> >
> > Some interesting bits of information in there.
> >
> > Patrick
> >
> >
> > Something Something wrote:
> >
> >> Here are some of the answers:
> >>
> >>   How many concurrent reducers run on each node?  Default two?
> >>>>
> >>> I was assuming 2 on each node would be the default.  If not, this could
> >> be a
> >> problem.  Please let me know.
> >>
> >>  'd suggest you spend a bit of time figuring where your MR jobs
> >>>>
> >>> are spending their time?
> >> I agree.  Will do some more research :)
> >>
> >>  How much of this overall time is spent in reduce phase?
> >>>>
> >>> Mostly time is spent in the Reduce phases, because that's where most of
> >> the
> >> critical code is.
> >>
> >>  Are inserts to a new table?
> >>>>
> >>> Yes, all inserts will always be in a new table.  In fact, I disable/drop
> >> HTables during this process.  Not using any special indexes, should I be?
> >>
> >>  I'm a little surprised that all worked on the small instances, that your
> >>>>
> >>> jobs completed.
> >> But, really, shouldn't Amazon guarantee predictability :)  After all I am
> >> paying for these instances.. albeit a small amount!
> >>
> >>  Are you opening a new table inside each task or once up in the config?
> >>>>
> >>> I open HTable in the 'setup' method for each mapper/reducer, and close
> >> table
> >> in the 'cleanup' method.
> >>
> >>  You have to temper the above general rule with the fact that...
> >>>>
> >>> I will try a few combinations.
> >>
> >>  How big is your dataset?
> >>>>
> >>> This one in particular is not big, but the real production ones will be.
> >>  Here's approximately how many rows get processed:
> >> Phase 1:  300 rows
> >> Phase 2 thru 8:  100 rows.
> >> (Note:  Each phase does complex calculations on the row.)
> >>
> >> Thanks for the help.
> >>
> >>
> >> On Tue, Jan 26, 2010 at 10:36 AM, Jean-Daniel Cryans 
> >> >wrote:
> >>
> >>  How big is your dataset?
> >>>
> >>> J-D
> >>>
> >>> On Tue, Jan 26, 2010 at 8:47 AM, Something Something
> >>> wrote:
> >>>
> >>>> I have noticed some strange performance numbers on EC2.  If someone can
> >>>>
> >>> give
> >>>
> >>>> me some hints to improve performance that would be greatly appreciated.
> >>>>  Here are the details:
> >>>>
> >>>> I have a process that runs a series of Jobs under Hadoop 0.20.1 & Hbase
> >>>> 0.20.2  I ran the *exact* same process with following configurations:
> >>>>
> >>>> 1) 1 Master & 4 Workers (*c1.xlarge* instances) & 1 Zookeeper
> >>>>
> >>> (*c1.medium*)
> >>>
> >>>> with *8 Reducers *for every Reduce task.  The process completed in *849*
> >>>>  seconds.
> >>>>
> >>>> 2) 1 Master, 4 Workers & 1 Zookeeper  *ALL m1.small* instances with *8
> >>>> Reducers *for every Reduce task.  The process completed in *906*
> >>>> seconds.
> >>>>
> >>>> 3) 1 Master, *11* Workers & *3* Zookeepers  *ALL m1.small* instances
> >>>> with
> >>>>
> >>> *20
> >>>
> >>>> Reducers *for every Reduce task.  The process completed in *984*
> >>>> seconds!
> >>>>
> >>>>
> >>>> Two main questions:
> >>>>
> >>>> 1)  It's totally surprising that when I have 11 workers with 20 Reducers
> >>>>
> >>> it
> >>>
> >>>> runs slower than when I have exactly same type of fewer machines with
> >>>>
> >>> fewer
> >>>
> >>>> reducers..
> >>>> 2)  As expected it runs faster on c1.xlarge, but the performance
> >>>>
> >>> improvement
> >>>
> >>>> doesn't justify the high cost difference.  I must not be utilizing the
> >>>> machine power, but I don't know how to do that.
> >>>>
> >>>> Here are some of the performance improvements tricks that I have learnt
> >>>>
> >>> from
> >>>
> >>>> this mailing list in the past that I am using:
> >>>>
> >>>> 1)  conf.set("hbase.client.scanner.caching", "30");   I have this for
> >>>> all
> >>>> jobs.
> >>>>
> >>>> 2)  Using the following code every time I open a HTable:
> >>>>       this.table = new HTable(new HBaseConfiguration(), "tablenameXYZ");
> >>>>       table.setAutoFlush(false);
> >>>>       table.setWriteBufferSize(1024 * 1024 * 12);
> >>>>
> >>>> 3)  For every Put I do this:
> >>>>         Put put = new Put(Bytes.toBytes(out));
> >>>>         put.setWriteToWAL(false);
> >>>>
> >>>> 4)  Change the No. of Reducers as per the No. of Workers.  I believe the
> >>>> formula is:  # of workers * 1.75.
> >>>>
> >>>> Any other hints?  As always, greatly appreciate the help.  Thanks.
> >>>>
> >>>>
> >>


From general-return-998-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 29 06:12:37 2010
Return-Path: <general-return-998-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53632 invoked from network); 29 Jan 2010 06:12:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Jan 2010 06:12:37 -0000
Received: (qmail 83292 invoked by uid 500); 29 Jan 2010 06:12:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83229 invoked by uid 500); 29 Jan 2010 06:12:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 26469 invoked by uid 99); 29 Jan 2010 05:05:32 -0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of arya_14@tce.edu designates 210.212.252.72 as permitted sender)
Message-ID: <31ca9cdc8c4fa7a3328c796fbc105e60.squirrel@mail.tce.edu>
Date: Fri, 29 Jan 2010 10:34:59 +0530
Subject: How to obtain dynamic bandwidth 
From: "Arya" <arya_14@tce.edu>
To: common-dev@hadoop.apache.org,
 general@hadoop.apache.org
User-Agent: SquirrelMail/1.4.19
MIME-Version: 1.0
Content-Type: text/plain;charset=iso-8859-1
Content-Transfer-Encoding: 8bit
X-Priority: 3 (Normal)
Importance: Normal
X-XheaderVersion: 1.1
X-UserAgent: 
X-Spam-Score: 1.9 (+)
X-Old-Spam-Status: No


hi all,

       Is there any way to obtain the dynamic bandwidth?. Is any command
available to get the dynamic bandwidth?.


Regards,
Arya





-----------------------------------------
This email was sent using TCEMail Service.
Thiagarajar College of Engineering
Madurai-625 015, India


From general-return-999-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 29 08:33:54 2010
Return-Path: <general-return-999-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94926 invoked from network); 29 Jan 2010 08:33:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Jan 2010 08:33:54 -0000
Received: (qmail 5670 invoked by uid 500); 29 Jan 2010 08:33:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5568 invoked by uid 500); 29 Jan 2010 08:33:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5558 invoked by uid 99); 29 Jan 2010 08:33:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Jan 2010 08:33:52 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bernd.fondermann@googlemail.com designates 209.85.160.50 as permitted sender)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Jan 2010 08:33:43 +0000
Received: by pwi20 with SMTP id 20so1234061pwi.29
        for <general@hadoop.apache.org>; Fri, 29 Jan 2010 00:33:22 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=LqRtrwl1iaP27du5eqMAcFwHA93x6MTllaYQf7Qy1R8=;
        b=BB4Qf2Fd6wn1+jeNRoWKSO8AwavNPLVua0kztpUJ0ONF5QeZ9I7Yy/mE0mpGj+qim2
         c31HkWprIvJfM8CgJLJo13kiy/3W+ZNKne+hC0be0YHBVQ4Qvft5UiglyzLFLnjg7CCR
         IidvzlCSNcs6mRPMx73i8c6unlHRx5DeyvlLI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=pDKA45wUtdmxXBTEUjKVNhzZppfC18lEXOqfyJ8xkIS9x8I+R4sJsuFuNc8RmN3Psr
         5nTaCPsSA/lt0NS/lMSBn65nkTulGCqbHG/IetaN58ekAjgcU1ORiAEpqvwT8fLm4zpI
         Vn4Ew9HVapxAzEQ0zezJtz3YyqWYFKVJL17Ys=
MIME-Version: 1.0
Received: by 10.143.25.17 with SMTP id c17mr404883wfj.13.1264754002402; Fri, 
	29 Jan 2010 00:33:22 -0800 (PST)
In-Reply-To: <109E2151-6161-45C3-972F-2FA857CF8AA6@apache.org>
References: <4B5F3124.4090004@apache.org>
	 <109E2151-6161-45C3-972F-2FA857CF8AA6@apache.org>
Date: Fri, 29 Jan 2010 09:33:22 +0100
Message-ID: <88f6e29a1001290033w3082734an480417a5bc53d45f@mail.gmail.com>
Subject: Re: avro in mapreduce
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Wed, Jan 27, 2010 at 23:12, Owen O'Malley <omalley@apache.org> wrote:
> On Jan 26, 2010, at 10:15 AM, Doug Cutting wrote:
>
>> This is a key link in a series of issues involved in integrating Avro in
>> Mapreduce.
>
> Getting Avro types passing through MapReduce is a good goal.
>
> I apologize for not seeing the issue before it was committed. I accept so=
me
> of the blame for that because I've buried in Hadoop emails. That said, it=
 is
> important to realize that with changes that radically change the user's
> interaction with the framework require a lot of discussion. This jira, as
> you've admitted, had a very unrepresentative subject and description, whi=
ch
> means that very few people were following it. Additionally, there has bee=
n
> no design document on the change to the MapReduce framework's paradigm, s=
o
> it wasn't clear what you were doing until this patch was committed. =A0Su=
ch a
> large change should have been highlighted on the public dev lists. In the
> future, I would strongly suggest all developers planning on making massiv=
e
> incompatible changes to post a design document on the public dev lists
> outside of Jira to ensure the discussion happens before instead of after =
the
> patch has been committed.
>
> In terms of reverting the patch, I had fundamental issues with the change=
s
> and felt that we needed more time to discuss them. Allowing the patch to
> stay in trunk would bake it further and further in and make reverting it
> much harder.
>
> I've listed my issues on the jira, but at a high level my concerns are:
>
> 1. Changing API compatibility is very expensive.
> 2. Changing the semantics is even more expensive.
> 3. We are discussing several alternatives on the jira.
>
> Unlike Python and Linux, Apache has a democratic process

Democratic? Not exactly. Some people have more rights than other's
(committers have more rights than users, PMC members more rights than
committers, PMC chairs have more rights than... oh wait, no - PMC
chairs are just plain-old PMC members.). That's an important
difference to make. Meriocratic would be the right term.

> and we have to work
> together to build consensus. The Apache rules are that a single -1 from a
> committer blocks the change from being made. Occasionally that has cost u=
s a
> lot of time. For example, a single -1 from a committer on an implementati=
on
> detail of the symlink patch blocked it for more than a year.

Vetos are very undemocratic. They exist to help the minority to not be
overruled by the majority.
But vetos have to be formally cast and justified. Simply reverting a
patch is not vetoing.

BTW, I cannot find the MAPREDUCE-1126 related discussion on any public
ML - what am I missing?
Which list do MAPREDUCE JIRA comments go to?

   Bernd

> We need to work
> together to find a solution that everyone can live with.
>
> -- Owen
>

From general-return-1000-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 29 12:01:32 2010
Return-Path: <general-return-1000-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68815 invoked from network); 29 Jan 2010 12:01:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Jan 2010 12:01:32 -0000
Received: (qmail 27789 invoked by uid 500); 29 Jan 2010 12:01:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27741 invoked by uid 500); 29 Jan 2010 12:01:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27731 invoked by uid 99); 29 Jan 2010 12:01:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Jan 2010 12:01:30 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Jan 2010 12:01:20 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 9E3D4B7DE9
	for <general@hadoop.apache.org>; Fri, 29 Jan 2010 12:00:59 +0000 (GMT)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id 1GmY3q268hmH for <general@hadoop.apache.org>;
	Fri, 29 Jan 2010 12:00:53 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id EEB8DB7DE2
	for <general@hadoop.apache.org>; Fri, 29 Jan 2010 12:00:52 +0000 (GMT)
MailScanner-NULL-Check: 1265371242.38932@Yco8vohyeaXfvn362/yGFw
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o0TC0e2D020896
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Fri, 29 Jan 2010 12:00:41 GMT
Message-ID: <4B62CDE7.3050508@apache.org>
Date: Fri, 29 Jan 2010 12:00:39 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Performance of EC2
References: <1eabbac31001260847g3a726f62o3a89ee2d74096e6e@mail.gmail.com>	 <31a243e71001261036t7d47bb7fo387e079cc8ed0974@mail.gmail.com>	 <1eabbac31001261120m74ee4817x3751592c930fd98b@mail.gmail.com>	 <4B5F4610.8050803@apache.org> <1eabbac31001261249s4b09fbdfyb886927b36c2953e@mail.gmail.com>
In-Reply-To: <1eabbac31001261249s4b09fbdfyb886927b36c2953e@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o0TC0e2D020896
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

Something Something wrote:
> Wow.. how naive I am to think that I could trust Amazon.  Thanks for
> forwarding the links, Patrick.  Seems like Amazon's reliability has gone
> down considerably over the past few months.  (Occasionally my instances fail
> on startup or die in the middle for no apparent reason, and I used to think
> I was doing something dumb!)

That's unfair. Large datacentres are inherently unreliable, because we 
build out them out of "normal availability" stuff rather than HA 
hardware. This then pushes the problem of availability down to the 
applications, to you.


* Most of the problems people have been discussing are bandwidth issues; 
it may be that AWS is coming under some massive DDoS attack and you are 
seeing the fringes of it. It could be that your neighbours are noisy 
-but if you are running big Hadoop jobs, you are the noisy neighbour.

* A more likely problem for you is where your machines are placed. If 
they all share a single switch, very high bandwidth. But if they are on 
different racks, the network becomes the bottleneck.


> But what I don't understand is this... if I *reserve* an instance then I
> wouldn't be sharing its CPU with anyone, right?  The blog seems to indicate
> otherwise.

I think you only get exclusive use of a CPU when you rent an XL node. 
Reservations are a form of capacity planning, may or may not help with 
scheduling at all.



From general-return-1001-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 29 16:56:44 2010
Return-Path: <general-return-1001-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68597 invoked from network); 29 Jan 2010 16:56:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Jan 2010 16:56:43 -0000
Received: (qmail 90057 invoked by uid 500); 29 Jan 2010 16:56:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90005 invoked by uid 500); 29 Jan 2010 16:56:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 6777 invoked by uid 99); 29 Jan 2010 15:21:39 -0000
X-ASF-Spam-Status: No, hits=0.2 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Subject: Re: How to obtain dynamic bandwidth 
Mime-Version: 1.0 (Apple Message framework v1077)
Content-Type: multipart/signed; boundary=Apple-Mail-281-855929077; protocol="application/pkcs7-signature"; micalg=sha1
From: Brian Bockelman <bbockelm@cse.unl.edu>
X-Priority: 3 (Normal)
In-Reply-To: <31ca9cdc8c4fa7a3328c796fbc105e60.squirrel@mail.tce.edu>
Date: Fri, 29 Jan 2010 09:20:54 -0600
Cc: general@hadoop.apache.org
Message-Id: <7B8AEBB3-549C-41E2-9AD4-61C3FBFAC4AF@cse.unl.edu>
References: <31ca9cdc8c4fa7a3328c796fbc105e60.squirrel@mail.tce.edu>
To: common-dev@hadoop.apache.org
X-Mailer: Apple Mail (2.1077)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-281-855929077
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

Hey Arya,

Try running Ganglia on your cluster:

http://ganglia.sourceforge.net/

One of the statistics it collects is the network I/O rate.

Also - for the future, you probably want to use the =
common-user@hadoop.apache.org mailing list for these sort of general =
questions.

Brian

On Jan 28, 2010, at 11:04 PM, Arya wrote:

>=20
> hi all,
>=20
>       Is there any way to obtain the dynamic bandwidth?. Is any =
command
> available to get the dynamic bandwidth?.
>=20
>=20
> Regards,
> Arya
>=20
>=20
>=20
>=20
>=20
> -----------------------------------------
> This email was sent using TCEMail Service.
> Thiagarajar College of Engineering
> Madurai-625 015, India


--Apple-Mail-281-855929077
Content-Disposition: attachment;
	filename=smime.p7s
Content-Type: application/pkcs7-signature;
	name=smime.p7s
Content-Transfer-Encoding: base64

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIICjCCA/gw
ggLgoAMCAQICASkwDQYJKoZIhvcNAQEFBQAwdTETMBEGCgmSJomT8ixkARkWA25ldDESMBAGCgmS
JomT8ixkARkWAkVTMQ4wDAYDVQQKEwVFU25ldDEgMB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9y
aXRpZXMxGDAWBgNVBAMTD0VTbmV0IFJvb3QgQ0EgMTAeFw0wMjEyMDUwODAwMDBaFw0xMzAxMjUw
ODAwMDBaMGkxEzARBgoJkiaJk/IsZAEZFgNvcmcxGDAWBgoJkiaJk/IsZAEZFghET0VHcmlkczEg
MB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9yaXRpZXMxFjAUBgNVBAMTDURPRUdyaWRzIENBIDEw
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC09dYjYaPbCD5mtbiQb7Ka3y1qAm0ZcqKC
FciWcfe8Kwcuy9tjHuIsLf9ZItdkDW4xy8sua9nJlx3KlwjtumTMtOtg35KZCknUd8KM4VGTSFdL
VG9AbNayef76caVCGM1+jyF0Lq03kauGOPTcNfZe1TZa3e1c9rc8ljV5OSWa/mfsCACyS5zFIWu0
yIDNyJdf+n0hwaPN53wllpJ30taD+JBjQ7h2k4xRWzeaznLOb9OztZVRA/1sVze+iczFh2xwa4Vd
Gy0eIIPw1pfvYwxO36rm0S109qvbsNlaroPRbxerPKakQLpKe034Xcx7gBPqUk/FxoRRWin5EWN3
rz9LAgMBAAGjgZ4wgZswDgYDVR0PAQH/BAQDAgGGMBEGCWCGSAGG+EIBAQQEAwIAhzAdBgNVHQ4E
FgQUyhkdEo5upDhdQtQxDgjb2Y0XDV0wHwYDVR0jBBgwFoAUvF1NSC/4NZRZq1yJSz7RsjoUAeow
DwYDVR0TAQH/BAUwAwEB/zAlBgNVHREEHjAcgRpET0VHcmlkcy1DQS0xQGRvZWdyaWRzLm9yZzAN
BgkqhkiG9w0BAQUFAAOCAQEAZNVrIDLqe39CEOiJt7Q7EpBPhAihMvDTSf/42u0SMbUmChww4mLm
ph5DBghZUVF8Yn59kRZMn1QLOtO1HzLqvAvPITacZVPlJgG2IXzlR636YghZFAycbIUEOJDBHR4v
tQO1KDxgZwvAbtmKIoxvhUCq2xsfFt9kCBBn+JYtQ6O5LsBJq3PmuubeMcc7mbQAfJZ7h/3Qghgk
FIhmE1+LBXPJbkuP8vgfg6h2BKoAf5TFfZECgGZKimfN110tBvfedGZwYYd3/GsJc83B0JN1gny0
gqNVPm392UchXGeBRrHnm2gkhIkr48Oq6EmNGV9/a6XfbplQW/JWbtPVPWkaizCCBAowggLyoAMC
AQICAwCB+zANBgkqhkiG9w0BAQUFADBpMRMwEQYKCZImiZPyLGQBGRYDb3JnMRgwFgYKCZImiZPy
LGQBGRYIRE9FR3JpZHMxIDAeBgNVBAsTF0NlcnRpZmljYXRlIEF1dGhvcml0aWVzMRYwFAYDVQQD
Ew1ET0VHcmlkcyBDQSAxMB4XDTA5MDYwMjE5NDExM1oXDTEwMDYwMjE5NDExM1owYTETMBEGCgmS
JomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCGRvZWdyaWRzMQ8wDQYDVQQLEwZQZW9wbGUx
HzAdBgNVBAMTFkJyaWFuIEJvY2tlbG1hbiA1MDQzMDcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw
ggEKAoIBAQDPWEl7hBiuFRVBSY4SwvG0HpkCZi74a0BeD0tNARgxoQVJ7jhJjR3G4y8ino0/5axt
2EEfIWUE+DVpV37IWOQl8q/wdvicnhbfjByxBbq4sfWPLepU7+Kd8k1FKHRHermARn9VxEkFLrLB
Gp7O5EX4mFHDaQy+Vv0thtA+m4qKoM+DA/8cOkJA5Rn6ZS/v/vtBzJh9HimVnhBx4+rw2cvKN+7r
lKsm7qTn9TCZmrQ97CvBEXSkHS11m8vYF6ZwcTgSCJM0M9nnX5JilupQO1vDICXSUZeWX2xpsqeL
x1PFGWgDaYXxFGtTRt2Qc9EPwf9Dr72xGPbKN8u5HylpOMDnAgMBAAGjgcIwgb8wEQYJYIZIAYb4
QgEBBAQDAgWgMA4GA1UdDwEB/wQEAwIF4DAfBgNVHSMEGDAWgBTKGR0Sjm6kOF1C1DEOCNvZjRcN
XTAYBgNVHSAEETAPMA0GCyqGSIb3TAMHAQMAMD4GA1UdHwQ3MDUwM6AxoC+GLWh0dHA6Ly9jcmwu
ZG9lZ3JpZHMub3JnLzFjM2YyY2E4LzFjM2YyY2E4LmNybDAfBgNVHREEGDAWgRRiYm9ja2VsbUBj
c2UudW5sLmVkdTANBgkqhkiG9w0BAQUFAAOCAQEAp6KjcWnfnH/MGlUkUWstE9gtPeymHp+2r4zI
w8JXigncJh/8qpSZqBcVhD24WFowI95otblrKYNZKW9f2G/hWwDSxZFqHhCDxFO12vDthrzOc3EH
CwypJPvIlZPt/E/x93XruzPxJwPz84DKKuPoJAMeNlADbd+92YtRr2y+VuMpgZaebMAoeCdWH8Cq
Y8xheNMajf8uiImBbatDuCu7qRvhwgxsMNLHEt4h853K1Zc181RlFGXG1+uL/Q/8VeKiASiCu+7L
1zpfLg7OCr6rJHb5S7wU+CeAvzSqmyy0fd2mwPeiX7huK+Cw4UjaB3yGKItzWT+KQJnV//wcSrzZ
dTGCAv0wggL5AgEBMHAwaTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERP
RUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3Jp
ZHMgQ0EgMQIDAIH7MAkGBSsOAwIaBQCgggFiMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJ
KoZIhvcNAQkFMQ8XDTEwMDEyOTE1MjA1NFowIwYJKoZIhvcNAQkEMRYEFNOxhbv1AtTrpifP64i9
M3Q17jSJMH8GCSsGAQQBgjcQBDFyMHAwaTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT
8ixkARkWCERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UE
AxMNRE9FR3JpZHMgQ0EgMQIDAIH7MIGBBgsqhkiG9w0BCRACCzFyoHAwaTETMBEGCgmSJomT8ixk
ARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBB
dXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3JpZHMgQ0EgMQIDAIH7MA0GCSqGSIb3DQEBAQUABIIB
AMTiLQPK00js1Z3nIuflBWs2ZQYF42srrBJ2/2snZzXKn1VM1YusCn/aC70pXew2m8JsmRC4dwcw
3973KeCjLH7cABpr+xWWFhdRvZfQjr4ZeNVOcIIBJaLkIIP2NhWjqrVMdVxqc6eTMgOJQenczCHf
Et+sIFgdLQuk6/8aoW9p0euPFtyNUulIOdSrHFgGZs4EyHkIbxrlAIBsAcNe5PzkdtC+Ty+DkDOZ
4GkdzddKYOm4WQ/EwWQoU6RHBGWaVnfwSjJAWIQkampMbZ2Aw8ZUAfRA4859N7iFVxLvrB9u5EFW
espLriL3waVHCSZwd58pIBGO6MtetclDS6W+W7cAAAAAAAA=

--Apple-Mail-281-855929077--

From general-return-1002-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 29 17:27:18 2010
Return-Path: <general-return-1002-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 80222 invoked from network); 29 Jan 2010 17:27:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Jan 2010 17:27:18 -0000
Received: (qmail 40650 invoked by uid 500); 29 Jan 2010 17:27:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40560 invoked by uid 500); 29 Jan 2010 17:27:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40550 invoked by uid 99); 29 Jan 2010 17:27:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Jan 2010 17:27:17 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mailinglists19@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Jan 2010 17:27:10 +0000
Received: by wwb24 with SMTP id 24so173992wwb.35
        for <general@hadoop.apache.org>; Fri, 29 Jan 2010 09:26:49 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=ck7GI3m8PlQxULfynx5EvOE2a/m3eJTqPZP61lTC2u8=;
        b=dJr8mdP391bJfsbQ+uXiztIy13DW64joL7JK7T9bW7lygwYMHxSHugtmQ8TabLrjn2
         IubK1GQYB9CwF7EebEMF3tGkGsLpfw9Fe6Xow5yja/8+dQONhKYEUTnVfafrQ7P9QyIS
         QS+GUK1nvAE7X8GnytVANjlNGYfkqFZvSQPZQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=RwU/JQgZYelU9a80xx34D7YXI63D4cP07LF6AaETft9XlHmBoOgytZ30skwZpceW0o
         uKVNyFMcamky1wbmCtLDrgf9pPWfImsJP7yUuEoffZxqzu3s7Ofae3VNeqLskyHNvYLf
         uLgNQWw5dcT3qEmezw9t1Y3KrJsjifoe9fjpo=
MIME-Version: 1.0
Received: by 10.216.91.73 with SMTP id g51mr609496wef.68.1264786009318; Fri, 
	29 Jan 2010 09:26:49 -0800 (PST)
In-Reply-To: <4B62CDE7.3050508@apache.org>
References: <1eabbac31001260847g3a726f62o3a89ee2d74096e6e@mail.gmail.com>
	 <31a243e71001261036t7d47bb7fo387e079cc8ed0974@mail.gmail.com>
	 <1eabbac31001261120m74ee4817x3751592c930fd98b@mail.gmail.com>
	 <4B5F4610.8050803@apache.org>
	 <1eabbac31001261249s4b09fbdfyb886927b36c2953e@mail.gmail.com>
	 <4B62CDE7.3050508@apache.org>
Date: Fri, 29 Jan 2010 09:26:49 -0800
Message-ID: <1eabbac31001290926u6166d9dck69303906e6227b62@mail.gmail.com>
Subject: Re: Performance of EC2
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d58cb2cd6633047e50f024

--0016e6d58cb2cd6633047e50f024
Content-Type: text/plain; charset=ISO-8859-1

Thanks everyone for the replies.  I agree I was being a bit unfair to
Amazon.  I apologize.

On Fri, Jan 29, 2010 at 4:00 AM, Steve Loughran <stevel@apache.org> wrote:

> Something Something wrote:
>
>> Wow.. how naive I am to think that I could trust Amazon.  Thanks for
>> forwarding the links, Patrick.  Seems like Amazon's reliability has gone
>> down considerably over the past few months.  (Occasionally my instances
>> fail
>> on startup or die in the middle for no apparent reason, and I used to
>> think
>> I was doing something dumb!)
>>
>
> That's unfair. Large datacentres are inherently unreliable, because we
> build out them out of "normal availability" stuff rather than HA hardware.
> This then pushes the problem of availability down to the applications, to
> you.
>
>
> * Most of the problems people have been discussing are bandwidth issues; it
> may be that AWS is coming under some massive DDoS attack and you are seeing
> the fringes of it. It could be that your neighbours are noisy -but if you
> are running big Hadoop jobs, you are the noisy neighbour.
>
> * A more likely problem for you is where your machines are placed. If they
> all share a single switch, very high bandwidth. But if they are on different
> racks, the network becomes the bottleneck.
>
>
>
>  But what I don't understand is this... if I *reserve* an instance then I
>> wouldn't be sharing its CPU with anyone, right?  The blog seems to
>> indicate
>> otherwise.
>>
>
> I think you only get exclusive use of a CPU when you rent an XL node.
> Reservations are a form of capacity planning, may or may not help with
> scheduling at all.
>
>
>

--0016e6d58cb2cd6633047e50f024--

From general-return-1003-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 29 17:36:33 2010
Return-Path: <general-return-1003-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84772 invoked from network); 29 Jan 2010 17:36:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Jan 2010 17:36:33 -0000
Received: (qmail 52645 invoked by uid 500); 29 Jan 2010 17:36:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52546 invoked by uid 500); 29 Jan 2010 17:36:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52526 invoked by uid 99); 29 Jan 2010 17:36:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Jan 2010 17:36:31 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mailinglists19@gmail.com designates 72.14.220.157 as permitted sender)
Received: from [72.14.220.157] (HELO fg-out-1718.google.com) (72.14.220.157)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Jan 2010 17:36:22 +0000
Received: by fg-out-1718.google.com with SMTP id 16so117967fgg.11
        for <multiple recipients>; Fri, 29 Jan 2010 09:36:02 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:cc:content-type;
        bh=k7WYJGL8Q2qbBlqQzD9Uulzpg5GYb6F6dsjKi4QFSHg=;
        b=UQHlHAVpnGpHf45MHHr53oOXjzQF0G8R2HeS5FAYttFK7eGO25ulB6CWjMWrGGSJ3n
         Frc6enq3+Vbh6sU5iDpzxlIMpjOgAQjVqE3RZJFIaqQadPOgb++kR5RZTyuuy0cU7Ndr
         ENxJIHIZsjx94g/3qC9f1h5GI06D6T5uboyeQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=UiCKgWcPVcrD4BBGgu2ZYOntp5fJBU0PFWF/3zziYwSQvos0BjdbniJtkXIkPmb99O
         R8vSfs6za24qWFwPeuPZYIEtgMgb0mXVolbpQHkgIAzVScw1SYtcd2Lb54TRwqdydscz
         R8SRBXwG26AzrQ4TQGJQI83hYAOPLJ118tYSk=
MIME-Version: 1.0
Received: by 10.216.90.81 with SMTP id d59mr686280wef.29.1264786560730; Fri, 
	29 Jan 2010 09:36:00 -0800 (PST)
In-Reply-To: <4B61381E.6010805@yahoo-inc.com>
References: <1eabbac31001251522s2301f263rb959c768c2ee4d1@mail.gmail.com>
	 <4B61381E.6010805@yahoo-inc.com>
Date: Fri, 29 Jan 2010 09:36:00 -0800
Message-ID: <1eabbac31001290936n17c8336eqa1b9b9c4d0ba2b76@mail.gmail.com>
Subject: Re: setNumReduceTasks(1)
From: Something Something <mailinglists19@gmail.com>
To: hbase-user@hadoop.apache.org
Cc: "general@hadoop.apache.org" <general@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=0016e6d7df76ab473a047e5111ff
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d7df76ab473a047e5111ff
Content-Type: text/plain; charset=ISO-8859-1

I am sorry, but I forgot to add one important piece of information.

I don't want to write any random N rows to the table.  I want to write the
*top* N rows - meaning - I want to write the "key" values of the Reducer in
descending order.  Does this make sense?  Sorry for the confusion.

On Wed, Jan 27, 2010 at 11:09 PM, Mridul Muralidharan <mridulm@yahoo-inc.com
> wrote:

>
> A possible solution is to emit only N rows from each mapper and then use 1
> reduce task [*] - if value of N is not very high.
> So you end up with utmost m * N rows on reducer instead of full inputset -
> and so the limit can be done easier.
>
>
> If you ok with some sort of variance in the number of rows inserted (and if
> value of N is very high), you can do more interesting things like N/m' rows
> per mapper - and multiple reducers (r) : with assumtion that each reducer
> will see atleast N/r rows - and so you can limit to N/r per reducer :
> ofcourse, there is a possible error that gets introduced here ...
>
>
> Regards,
> Mridul
>
> [*] Assuming you just want simple limit - nothing else.
> Also note, each mapper might want to emit N rows instead of 'tweaks' like
> N/m rows, since it is possible that multiple mappers might have less than
> N/m rows to emit to begin with !
>
>
>
> Something Something wrote:
>
>> If I set # of reduce tasks to 1 using setNumReduceTasks(1), would the
>> class
>> be instantiated only on one machine.. always?  I mean if I have a cluster
>> of
>> say 1 master, 10 workers & 3 zookeepers, is the Reducer class guaranteed
>> to
>> be instantiated only on 1 machine?
>>
>> If answer is yes, then I will use static variable as a counter to see how
>> may rows have been added to my HBase table so far.  In my use case, I want
>> to write only N number of rows to a table.  Is there a better way to do
>> this?  Please let me know.  Thanks.
>>
>
>

--0016e6d7df76ab473a047e5111ff--

From general-return-1004-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 29 17:36:46 2010
Return-Path: <general-return-1004-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84980 invoked from network); 29 Jan 2010 17:36:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Jan 2010 17:36:46 -0000
Received: (qmail 54253 invoked by uid 500); 29 Jan 2010 17:36:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54185 invoked by uid 500); 29 Jan 2010 17:36:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54174 invoked by uid 99); 29 Jan 2010 17:36:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Jan 2010 17:36:45 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.210.199] (HELO mail-yx0-f199.google.com) (209.85.210.199)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Jan 2010 17:36:34 +0000
Received: by yxe37 with SMTP id 37so1930648yxe.31
        for <general@hadoop.apache.org>; Fri, 29 Jan 2010 09:36:13 -0800 (PST)
MIME-Version: 1.0
Received: by 10.101.80.9 with SMTP id h9mr1489219anl.23.1264786572104; Fri, 29 
	Jan 2010 09:36:12 -0800 (PST)
In-Reply-To: <88f6e29a1001290033w3082734an480417a5bc53d45f@mail.gmail.com>
References: <4B5F3124.4090004@apache.org> <109E2151-6161-45C3-972F-2FA857CF8AA6@apache.org> 
	<88f6e29a1001290033w3082734an480417a5bc53d45f@mail.gmail.com>
From: Philip Zeyliger <philip@cloudera.com>
Date: Fri, 29 Jan 2010 09:35:52 -0800
Message-ID: <15da8a101001290935s3aff835bn1833f435a59cc27b@mail.gmail.com>
Subject: Re: avro in mapreduce
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636ed718758d562047e5112f2
X-Virus-Checked: Checked by ClamAV on apache.org

--001636ed718758d562047e5112f2
Content-Type: text/plain; charset=ISO-8859-1

>
>
> Which list do MAPREDUCE JIRA comments go to?
>

mapreduce-issues@hadoop.apache.org

Some web sites archive it:

http://www.mail-archive.com/search?l=mapreduce-issues@hadoop.apache.org&q=mapreduce-1126

--001636ed718758d562047e5112f2--

From general-return-1005-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 29 18:07:53 2010
Return-Path: <general-return-1005-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 958 invoked from network); 29 Jan 2010 18:07:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Jan 2010 18:07:52 -0000
Received: (qmail 7075 invoked by uid 500); 29 Jan 2010 18:07:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7004 invoked by uid 500); 29 Jan 2010 18:07:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6994 invoked by uid 99); 29 Jan 2010 18:07:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Jan 2010 18:07:51 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.27.243] (HELO qmta13.emeryville.ca.mail.comcast.net) (76.96.27.243)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Jan 2010 18:07:41 +0000
Received: from omta13.emeryville.ca.mail.comcast.net ([76.96.30.52])
	by qmta13.emeryville.ca.mail.comcast.net with comcast
	id bQX71d00617UAYkADW7ME2; Fri, 29 Jan 2010 18:07:21 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta13.emeryville.ca.mail.comcast.net with comcast
	id bW7C1d00G2SbwD58ZW7FbY; Fri, 29 Jan 2010 18:07:19 +0000
Message-Id: <62ADCD97-4476-4C7F-8915-8E442299FAB6@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <88f6e29a1001290033w3082734an480417a5bc53d45f@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: avro in mapreduce
Date: Fri, 29 Jan 2010 10:07:11 -0800
References: <4B5F3124.4090004@apache.org> <109E2151-6161-45C3-972F-2FA857CF8AA6@apache.org> <88f6e29a1001290033w3082734an480417a5bc53d45f@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

On Jan 29, 2010, at 12:33 AM, Bernd Fondermann wrote:

> Democratic? Not exactly.

It is completely democratic within the group that gets to vote. Python  
and Linux let the person who started the project have an absolutely  
final say, which is a very different model.

>  PMC chairs have more rights than.

Actually, PMC chairs do have more rights, but they aren't very  
interesting rights. *smile* They are things like granting subversion  
karma to new committers that have been voted in by the PMC and writing  
the quarterly report on the state of the project for the Apache board.

> But vetos have to be formally cast and justified. Simply reverting a
> patch is not vetoing.

I did veto the patch and gave my justification before I reverted it.  
For better or worse, the Apache rules say that -1 votes on code  
changes are binding. Trust me, I've been on the wrong side of a -1 and  
we had to work through to a compromise. It takes time, but the hope is  
that it leads to better software, which is the real goal.

-- Owen

From general-return-1006-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 29 18:21:15 2010
Return-Path: <general-return-1006-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 7559 invoked from network); 29 Jan 2010 18:21:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Jan 2010 18:21:15 -0000
Received: (qmail 29939 invoked by uid 500); 29 Jan 2010 18:21:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29881 invoked by uid 500); 29 Jan 2010 18:21:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29871 invoked by uid 99); 29 Jan 2010 18:21:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Jan 2010 18:21:14 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [166.84.7.136] (HELO vc136.vc.panix.com) (166.84.7.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Jan 2010 18:21:05 +0000
Received: from localhost (localhost [127.0.0.1])
	by vc136.vc.panix.com (Postfix) with ESMTP id 18455DC3B8
	for <general@hadoop.apache.org>; Fri, 29 Jan 2010 13:20:43 -0500 (EST)
Received: from vc136.vc.panix.com ([127.0.0.1])
	by localhost (vc136.vc.panix.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id PHREbpZesMjO for <general@hadoop.apache.org>;
	Fri, 29 Jan 2010 13:20:36 -0500 (EST)
Received: from eric-sammers-macbook-pro.local (pool-96-246-67-178.nycmny.east.verizon.net [96.246.67.178])
	by vc136.vc.panix.com (Postfix) with ESMTP id ACCCFDC0A6
	for <general@hadoop.apache.org>; Fri, 29 Jan 2010 13:20:36 -0500 (EST)
Message-ID: <4B6326F5.4050307@lifeless.net>
Date: Fri, 29 Jan 2010 13:20:37 -0500
From: Eric Sammer <eric@lifeless.net>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.7) Gecko/20100111 Thunderbird/3.0.1
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: avro in mapreduce
References: <4B5F3124.4090004@apache.org>
In-Reply-To: <4B5F3124.4090004@apache.org>
X-Enigmail-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 1/26/10 1:15 PM, Doug Cutting wrote:
> I would like to call folks attention to MAPREDUCE-1126.
> 
> https://issues.apache.org/jira/browse/MAPREDUCE-1126
> 
> This is a key link in a series of issues involved in integrating Avro in
> Mapreduce.  Aaron proposed a design in early December, building on the
> design Tom developed last summer and committed in September in
> HADOOP-6165.  Aaron's design was approved, and, after several rounds of
> reviews, I committed Aaron's patch on 11 January.
> 
> On 15 January Owen reverted this commit without warning.  It seems that
> Owen objects to the path initiated last July in HADOOP-6165.
> 
> Aaron has also contributed MAPREDUCE-815, which permits one to use Avro
> for all phases of Mapreduce.  When that issue is committed, the primary
> chain of Avro integration into Mapreduce will be complete.
> 
> Can others please take the time to read this issue and express their
> opinions?

I'm coming at this from a place of far less information and commitment
to the APIs but as it stands, the existing relationship between formats
and types they support strikes me as odd.

Currently mappers and reducers are written to a contract of a particular
K/V type pair. Input and Output formats are currently aware of those
types directly and are not sanity checked until runtime given the way
format classes are specified to the job configuration. This means that a
given Mapper / Reducer implementation can be used with any format of a
particular "family" (i.e. Writable vs. Avro vs. etc.) although this is
not strongly enforced. It seems to me that if we already have this
dependency relationship that M / R implementations would, instead, be
parameterized by a serialization input / output pair rather than four
individual types and i/o formats would / could be parameterized with
serialization types.

i.e.

public class MyMapper extends
  Mapper<AvroReader<String, AvroMyType>,
    WritableWriter<LongWritable, Text>> {

  public void map(AvroReader<String, AvroMyType> reader,
    Context<WritableWriter<LongWritable, Text>> context) {

    String key = reader.getKey();

    context.write(
      new LongWritable(Long.valueOf(key)),
      new Text(reader.getValue().toString())
    );
}

(I actually don't really like the way I'm using Context here, but you
get the idea. It would be nice if the Reader / Writer thing was
represented symmetrically.)

This means we have a tangible single entity that encapsulates the
relationship between a type and how it is serialized and allows for
nicer parameterization of shared types between serialization formats.

Maybe Avro{Reader,Writer} and Writable{Reader,Writer} extend or
implement a common Serialization{Reader,Writer} that formats can
reference and to which they can defer the process of actual
serialization. This decouples formats from specific serialization
schemes and makes map / reduce implementation far more explicit (as well
as deals with Doug's case of a type that implements multiple
serialization schemes as well as the case where a serializable type need
not implement any known interface such as standard Java types).

Of course, it's a huge API change, comparatively, so I don't think this
does much for Owen's concerns. How best this is conveyed during
Configuration is unclear to me. This is why I'm raising this on list
rather than in the JIRA issue. I don't know that this solves the core
concerns and could be tangential and intractable given the internals /
effect on user facing APIs.

Just my $0.02USD (subject to market fluctuation). Flame away.

Regards.
-- 
Eric Sammer
eric@lifeless.net
http://esammer.blogspot.com

From general-return-809-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 01 08:35:29 2009
Return-Path: <general-return-809-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 11015 invoked from network); 1 Dec 2009 08:35:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Dec 2009 08:35:29 -0000
Received: (qmail 44590 invoked by uid 500); 1 Dec 2009 08:35:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44259 invoked by uid 500); 1 Dec 2009 08:35:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44249 invoked by uid 99); 1 Dec 2009 08:35:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Dec 2009 08:35:26 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of schukanov@rbc.ru designates 80.68.240.91 as permitted sender)
Received: from [80.68.240.91] (HELO hermes.hw.ru) (80.68.240.91)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Dec 2009 08:35:14 +0000
Received: from [80.68.253.76] (HELO chukanovs)
  by hermes.hw.ru (CommuniGate Pro SMTP 5.2.6)
  with ESMTP id 406399458 for general@hadoop.apache.org; Tue, 01 Dec 2009 11:34:53 +0300
From: =?koi8-r?B?/tXLwc7P1yDzxdLHxco=?= <schukanov@rbc.ru>
To: <general@hadoop.apache.org>
Subject: FSNamesystem initialization failed.
Date: Tue, 1 Dec 2009 11:34:53 +0300
MIME-Version: 1.0
Message-ID: <FF159CAD6222408FB0A5B7FC649D14C7@users.rbc.ru>
X-Mailer: Microsoft Office Outlook 11
Content-Type: multipart/signed;
	protocol="application/x-pkcs7-signature";
	micalg=SHA1;
	boundary="----=_NextPart_000_00FC_01CA727A.4CEC4690"
Thread-Index: AcpyYSeam8FeFz5wR+GvfHO+KeCdVw==
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5579
X-Virus-Checked: Checked by ClamAV on apache.org

------=_NextPart_000_00FC_01CA727A.4CEC4690
Content-Type: multipart/alternative;
	boundary="----=_NextPart_001_00FD_01CA727A.4CEC4690"


------=_NextPart_001_00FD_01CA727A.4CEC4690
Content-Type: text/plain;
	charset="koi8-r"
Content-Transfer-Encoding: 7bit

Hi, suddenly I've got problem with starting Namenode:

 

[hadoop@hadoop1 hadoop]$ bin/hadoop namenode

09/12/01 11:26:51 INFO namenode.NameNode: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG:   host = hadoop1.bcr.ru/81.68.243.18

STARTUP_MSG:   args = []

STARTUP_MSG:   version = 0.19.2-dev

STARTUP_MSG:   build =
http://svn.apache.org/repos/asf/hadoop/core/branches/bran

ch-0.19 -r 755955; compiled by 'maksim07' on Sun Mar 22 14:29:37 MSK 2009

************************************************************/

09/12/01 11:26:51 INFO metrics.RpcMetrics: Initializing RPC Metrics with
hostName=NameNode, port=9000

09/12/01 11:26:51 INFO namenode.NameNode: Namenode up at:
hadoop1.bcr.ru/81.68.243.18:9000

09/12/01 11:26:51 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=NameNode, sessionId=null

09/12/01 11:26:51 INFO metrics.NameNodeMetrics: Initializing
NameNodeMeterics using context
object:org.apache.hadoop.metrics.spi.NullContext

09/12/01 11:26:52 INFO namenode.FSNamesystem: fsOwner=hadoop,hadoop,dba

09/12/01 11:26:52 INFO namenode.FSNamesystem: supergroup=supergroup

09/12/01 11:26:52 INFO namenode.FSNamesystem: isPermissionEnabled=true

09/12/01 11:26:52 INFO metrics.FSNamesystemMetrics: Initializing
FSNamesystemMet

rics using context object:org.apache.hadoop.metrics.spi.NullContext

09/12/01 11:26:52 INFO namenode.FSNamesystem: Registered
FSNamesystemStatusMBean

09/12/01 11:26:52 INFO common.Storage: Number of files = 30079

09/12/01 11:26:53 INFO common.Storage: Number of files under construction =
0

09/12/01 11:26:53 INFO common.Storage: Image file of size 4451066 loaded in
1 seconds.

09/12/01 11:27:11 ERROR namenode.FSNamesystem: FSNamesystem initialization
failed.

java.io.IOException: Incorrect data format. logVersion is -18 but
writables.length is 0.

        at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:
542)

        at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:973)

        at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:793)

        at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage
.java:352)

        at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.j
ava:87)

        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.
java:309)

        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java
:288)

        at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:163
)

        at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:208)

        at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:194)

        at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java
:859)

        at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:868)

09/12/01 11:27:11 INFO ipc.Server: Stopping server on 9000

09/12/01 11:27:11 ERROR namenode.NameNode: java.io.IOException: Incorrect
data f

ormat. logVersion is -18 but writables.length is 0.

        at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:
542)

        at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:973)

        at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:793)

        at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage
.java:352)

        at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.j
ava:87)

        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.
java:309)

        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java
:288)

        at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:163
)

        at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:208)

        at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:194)

        at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java
:859)

        at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:868)

09/12/01 11:27:11 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at hadoop1.bcr.ru/81.68.243.18

************************************************************/

 

Any ideas?

 

Thanks!

 


------=_NextPart_001_00FD_01CA727A.4CEC4690
Content-Type: text/html;
	charset="koi8-r"
Content-Transfer-Encoding: quoted-printable

<html xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns=3D"http://www.w3.org/TR/REC-html40">

<head>
<meta http-equiv=3DContent-Type content=3D"text/html; charset=3Dkoi8-r">
<meta name=3DGenerator content=3D"Microsoft Word 11 (filtered medium)">
<style>
<!--
 /* Style Definitions */
 p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0cm;
	margin-bottom:.0001pt;
	font-size:12.0pt;
	font-family:"Times New Roman";}
a:link, span.MsoHyperlink
	{color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{color:purple;
	text-decoration:underline;}
span.EmailStyle17
	{mso-style-type:personal-compose;
	font-family:Arial;
	color:windowtext;}
@page Section1
	{size:595.3pt 841.9pt;
	margin:2.0cm 42.5pt 2.0cm 3.0cm;}
div.Section1
	{page:Section1;}
-->
</style>

</head>

<body lang=3DRU link=3Dblue vlink=3Dpurple>

<div class=3DSection1>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>Hi, suddenly I&#8217;ve got problem with =
starting
Namenode:<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>[hadoop@hadoop1 hadoop]$ bin/hadoop =
namenode<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>09/12/01 11:26:51 INFO namenode.NameNode:
STARTUP_MSG:<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>/**********************************************=
**************<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>STARTUP_MSG: Starting =
NameNode<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>STARTUP_MSG:=9A=9A host =3D =
hadoop1.bcr.ru/81.68.243.18<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>STARTUP_MSG:=9A=9A args =3D =
[]<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>STARTUP_MSG:=9A=9A version =3D =
0.19.2-dev<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>STARTUP_MSG:=9A=9A build =3D
http://svn.apache.org/repos/asf/hadoop/core/branches/bran<o:p></o:p></spa=
n></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>ch-0.19 -r 755955; compiled by 'maksim07' on =
Sun Mar
22 14:29:37 MSK 2009<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>***********************************************=
*************/<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>09/12/01 11:26:51 INFO metrics.RpcMetrics:
Initializing RPC Metrics with hostName=3DNameNode, =
port=3D9000<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>09/12/01 11:26:51 INFO namenode.NameNode: =
Namenode up
at: hadoop1.bcr.ru/81.68.243.18:9000<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>09/12/01 11:26:51 INFO jvm.JvmMetrics: =
Initializing
JVM Metrics with processName=3DNameNode, =
sessionId=3Dnull<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>09/12/01 11:26:51 INFO =
metrics.NameNodeMetrics:
Initializing NameNodeMeterics using context
object:org.apache.hadoop.metrics.spi.NullContext<o:p></o:p></span></font>=
</p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>09/12/01 11:26:52 INFO namenode.FSNamesystem:
fsOwner=3Dhadoop,hadoop,dba<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>09/12/01 11:26:52 INFO namenode.FSNamesystem:
supergroup=3Dsupergroup<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>09/12/01 11:26:52 INFO namenode.FSNamesystem:
isPermissionEnabled=3Dtrue<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>09/12/01 11:26:52 INFO =
metrics.FSNamesystemMetrics:
Initializing FSNamesystemMet<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>rics using context
object:org.apache.hadoop.metrics.spi.NullContext<o:p></o:p></span></font>=
</p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>09/12/01 11:26:52 INFO namenode.FSNamesystem:
Registered FSNamesystemStatusMBean<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>09/12/01 11:26:52 INFO common.Storage: Number =
of
files =3D 30079<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>09/12/01 11:26:53 INFO common.Storage: Number =
of
files under construction =3D 0<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>09/12/01 11:26:53 INFO common.Storage: Image =
file of
size 4451066 loaded in 1 seconds.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>09/12/01 11:27:11 ERROR namenode.FSNamesystem:
FSNamesystem initialization failed.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>java.io.IOException: Incorrect data format.
logVersion is -18 but writables.length is =
0.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>=9A=9A=9A=9A=9A=9A=9A at =
org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.ja=
va:542)<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>=9A=9A=9A=9A=9A=9A=9A at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:9=
73)<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>=9A=9A=9A=9A=9A=9A=9A at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:7=
93)<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>=9A=9A=9A=9A=9A=9A=9A at =
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSIm=
age.java:352)<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>=9A=9A=9A=9A=9A=9A=9A at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirector=
y.java:87)<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>=9A=9A=9A=9A=9A=9A=9A at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesyst=
em.java:309)<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>=9A=9A=9A=9A=9A=9A=9A at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.&lt;init&gt;(FSNamesy=
stem.java:288)<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>=9A=9A=9A=9A=9A=9A=9A at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:=
163)<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>=9A=9A=9A=9A=9A=9A=9A at
org.apache.hadoop.hdfs.server.namenode.NameNode.&lt;init&gt;(NameNode.jav=
a:208)<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>=9A=9A=9A=9A=9A=9A=9A at
org.apache.hadoop.hdfs.server.namenode.NameNode.&lt;init&gt;(NameNode.jav=
a:194)<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>=9A=9A=9A=9A=9A=9A=9A at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.j=
ava:859)<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>=9A=9A=9A=9A=9A=9A=9A at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:868)<o=
:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>09/12/01 11:27:11 INFO ipc.Server: Stopping =
server on
9000<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>09/12/01 11:27:11 ERROR namenode.NameNode:
java.io.IOException: Incorrect data f<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>ormat. logVersion is -18 but writables.length =
is 0.<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>=9A=9A=9A=9A=9A=9A=9A at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.ja=
va:542)<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>=9A=9A=9A=9A=9A=9A=9A at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:9=
73)<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>=9A=9A=9A=9A=9A=9A=9A at =
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:7=
93)<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>=9A=9A=9A=9A=9A=9A=9A at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSIm=
age.java:352)<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>=9A=9A=9A=9A=9A=9A=9A at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirector=
y.java:87)<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>=9A=9A=9A=9A=9A=9A=9A at =
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesyst=
em.java:309)<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>=9A=9A=9A=9A=9A=9A=9A at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.&lt;init&gt;(FSNamesy=
stem.java:288)<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>=9A=9A=9A=9A=9A=9A=9A at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:=
163)<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>=9A =9A=9A=9A=9A=9A=9Aat
org.apache.hadoop.hdfs.server.namenode.NameNode.&lt;init&gt;(NameNode.jav=
a:208)<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>=9A=9A=9A=9A=9A=9A=9A at
org.apache.hadoop.hdfs.server.namenode.NameNode.&lt;init&gt;(NameNode.jav=
a:194)<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>=9A=9A=9A=9A=9A=9A=9A at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.j=
ava:859)<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>=9A=9A=9A=9A=9A=9A=9A at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:868)<o=
:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>09/12/01 11:27:11 INFO namenode.NameNode:
SHUTDOWN_MSG:<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>/**********************************************=
**************<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>SHUTDOWN_MSG: Shutting down NameNode at =
hadoop1.bcr.ru/81.68.243.18<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>***********************************************=
*************/</span></font><font
size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;font-family:Arial'><o:p></o:p></span></font></p=
>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span =
style=3D'font-size:10.0pt;
font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>Any ideas?<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'><o:p>&nbsp;</o:p></span></font></p>

<p class=3DMsoNormal><font size=3D2 face=3DArial><span lang=3DEN-US =
style=3D'font-size:
10.0pt;font-family:Arial'>Thanks!<o:p></o:p></span></font></p>

<p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span =
style=3D'font-size:
12.0pt'><o:p>&nbsp;</o:p></span></font></p>

</div>

</body>

</html>

------=_NextPart_001_00FD_01CA727A.4CEC4690--

------=_NextPart_000_00FC_01CA727A.4CEC4690
Content-Type: application/x-pkcs7-signature;
	name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
	filename="smime.p7s"

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIPRTCCBDIw
ggMaoAMCAQICAQEwDQYJKoZIhvcNAQEFBQAwezELMAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0
ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0
ZWQxITAfBgNVBAMMGEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczAeFw0wNDAxMDEwMDAwMDBaFw0y
ODEyMzEyMzU5NTlaMHsxCzAJBgNVBAYTAkdCMRswGQYDVQQIDBJHcmVhdGVyIE1hbmNoZXN0ZXIx
EDAOBgNVBAcMB1NhbGZvcmQxGjAYBgNVBAoMEUNvbW9kbyBDQSBMaW1pdGVkMSEwHwYDVQQDDBhB
QUEgQ2VydGlmaWNhdGUgU2VydmljZXMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC+
QJ30buHqdoccTUVEjr5GyIMGncEq/hgfjuQC+vOrXVCKFjELmgbQxXAizUktVGPMtm5oRgtT6stM
JMC8ck7q8RWu9FSaEgrDerIzYOLaiVXzIljz3tzP74OGooyUT59o8piQRoQnx3a/48w1LIteB2Rl
gsBIsKiR+WGfdiBQqJHHZrXreGIDVvCKGhPqMaMeoJn9OPb2JzJYbwf1a7j7FCuvt6rM1mNfc4za
BZmoOKjLF3g2UazpnvR4Oo3PD9lC4pgMqy+fDgHe75+ZSfEt36x0TRuYtUfF5SnR+ZAYx2KcvoPH
Jns+iiXHwN2d5jVoECCdj9je0sOEnA1e6C/JAgMBAAGjgcAwgb0wHQYDVR0OBBYEFKARCiM+lvEH
7OKvKe+CpX/QMKS0MA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MHsGA1UdHwR0MHIw
OKA2oDSGMmh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0FBQUNlcnRpZmljYXRlU2VydmljZXMuY3Js
MDagNKAyhjBodHRwOi8vY3JsLmNvbW9kby5uZXQvQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmww
DQYJKoZIhvcNAQEFBQADggEBAAhW/ALwm+j/pPrWe8ZEgM5PxMX2AFjMpra8FEloBHbo5u5d7AIP
YNaNUBhPJk4B4+awpe6/vHRUQb/9/BK4x09a9IlgBX9gtwVK8/bxwr/EuXSGti19a8zS80bdL8bg
asPDNAMsfZbdWsIOpwqZwQWLqwwv81w6z2w3VQmH3lNAbFjv/LarZW4E9hvcPOBaFcae2fFZSDAh
ZQNs7Okhc+ybA6HgN62gFRiP+roCzqcsqRATLNTlCCarIpdg+JBedNSimlO98qlo4KJuwtdssaMP
nr/raOdW8q7y4ys4OgmBtWuF174t7T8at7Jj4vViLILUagBBUPE5g5+V6TaWmG4wggTdMIIDxaAD
AgECAhBxkvvmGV+sTRKFdHE0ohinMA0GCSqGSIb3DQEBBQUAMHsxCzAJBgNVBAYTAkdCMRswGQYD
VQQIDBJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcMB1NhbGZvcmQxGjAYBgNVBAoMEUNvbW9k
byBDQSBMaW1pdGVkMSEwHwYDVQQDDBhBQUEgQ2VydGlmaWNhdGUgU2VydmljZXMwHhcNMDQwMTAx
MDAwMDAwWhcNMjgxMjMxMjM1OTU5WjCBrjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYD
VQQHEw5TYWx0IExha2UgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYD
VQQLExhodHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1VU0VSRmlyc3QtQ2xp
ZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBFbWFpbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBALI5haTyfatBO2JGN67NwWB1vDll+UoaR6K5zEjMapjVTTUZuaRC5c5J4oovHnzSMQfHTrSD
ZJ0uKdWiZMSFvYVRNXmkTmiQexx6pJKoF/KYFfKTzMmkMpW7DE8wvZigC4vlbhuiRvp4vKJvq1le
pS/Pytptqi/rrKGzaqq3Lmc1i3nhHmmI4uZGzaCl6r4LznY6eg6b6vzaJ1s9cx8i5khhxkzzabGo
Lhu21DEgLLyCio6kDqXXiUP8FlqvHXHXEVnauocNr/rz4cLwpMVnjNbWVDreCqS6A3ezZcj9HtN0
YqoYymiTHqGFfvVHZcv4TVcodNI0/zC27vZiMBSMLOsCAwEAAaOCAScwggEjMB8GA1UdIwQYMBaA
FKARCiM+lvEH7OKvKe+CpX/QMKS0MB0GA1UdDgQWBBSJgmd9xJ0mcABLtFBIfN49rgRufTAOBgNV
HQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUH
AwQwEQYDVR0gBAowCDAGBgRVHSAAMHsGA1UdHwR0MHIwOKA2oDSGMmh0dHA6Ly9jcmwuY29tb2Rv
Y2EuY29tL0FBQUNlcnRpZmljYXRlU2VydmljZXMuY3JsMDagNKAyhjBodHRwOi8vY3JsLmNvbW9k
by5uZXQvQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwEQYJYIZIAYb4QgEBBAQDAgEGMA0GCSqG
SIb3DQEBBQUAA4IBAQCdlcs8uH6lCcQevwvCx3aOOTyUxhCqTwzJ4KuEXYlU4GU7820cfDcsJVRf
liH8N4SRnRXcFE+Bz1Qda2xFYMct+ZdRTPlmyjyggoymyPDi6dRK+ew/VsnddozDggFPbADzHhph
dARHA6nGQFeRvGUixSdnT1fbZFrZjR+6hi/0Bq6cae3p9M8pF9jgSp8aIC+XTFG7RgfEijdOIOMJ
MWjHnsSLneh+EbwyaBCWEZhE2CpRYE2I63Q630MGMsg5Vow6EVLTQaRDA/Tt7zMn2zngFE4mydj1
OeKJuJNdtykmQeqzm66D/Hd1yujKtf7iZUpjPkTE0MNeh3OpmByvfxV/MIIGKjCCBRKgAwIBAgIR
AKpB7AqwvGMRYk72IFbTyEUwDQYJKoZIhvcNAQEFBQAwga4xCzAJBgNVBAYTAlVTMQswCQYDVQQI
EwJVVDEXMBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNVBAoTFVRoZSBVU0VSVFJVU1QgTmV0
d29yazEhMB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3QuY29tMTYwNAYDVQQDEy1VVE4tVVNF
UkZpcnN0LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1haWwwHhcNMDkxMTExMDAwMDAwWhcN
MTAxMTExMjM1OTU5WjCB2zE1MDMGA1UECxMsQ29tb2RvIFRydXN0IE5ldHdvcmsgLSBQRVJTT05B
IE5PVCBWQUxJREFURUQxRjBEBgNVBAsTPVRlcm1zIGFuZCBDb25kaXRpb25zIG9mIHVzZTogaHR0
cDovL3d3dy5jb21vZG8ubmV0L3JlcG9zaXRvcnkxHzAdBgNVBAsTFihjKTIwMDMgQ29tb2RvIExp
bWl0ZWQxGDAWBgNVBAMTD1NlcmdleSBDaHVrYW5vdjEfMB0GCSqGSIb3DQEJARYQc2NodWthbm92
QHJiYy5ydTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANDwTpcVhe3t+xMpzHVvHUOh
ebcMtACt/oNu0GkYpZuJ+kT9zwsbF8fGlGOX5bvF4smKI6J/ciifGjj8p/Rl6+IS0zWHAmNXPv3Z
RuNLbuT5Xb741nmg5X7/LprdNeLLmomqJo3LPQgpME6qG6ro9npnPSIS80hiu4FI8npqTyZzYBQK
B9SZUrXVkDkjnHlbtt0clV+/E2LhA5xCE42ljhEFK9c9EhJsIo3xrINl0cPw26kObIbmlxljSfMO
5qHzcLPG5cz1RRSHXl1woGNBfZP/7IxePj2+pC5hRTz9+VOhMlYxEtGwD4kD5unmgKWmhRQexsyi
bkXK8Dn8YZWEHDsCAwEAAaOCAhIwggIOMB8GA1UdIwQYMBaAFImCZ33EnSZwAEu0UEh83j2uBG59
MB0GA1UdDgQWBBT/ENNZ/NAHgMOvMGYarhadXD2zDjAOBgNVHQ8BAf8EBAMCBaAwDAYDVR0TAQH/
BAIwADAgBgNVHSUEGTAXBggrBgEFBQcDBAYLKwYBBAGyMQEDBQIwEQYJYIZIAYb4QgEBBAQDAgUg
MEYGA1UdIAQ/MD0wOwYMKwYBBAGyMQECAQEBMCswKQYIKwYBBQUHAgEWHWh0dHBzOi8vc2VjdXJl
LmNvbW9kby5uZXQvQ1BTMIGlBgNVHR8EgZ0wgZowTKBKoEiGRmh0dHA6Ly9jcmwuY29tb2RvY2Eu
Y29tL1VUTi1VU0VSRmlyc3QtQ2xpZW50QXV0aGVudGljYXRpb25hbmRFbWFpbC5jcmwwSqBIoEaG
RGh0dHA6Ly9jcmwuY29tb2RvLm5ldC9VVE4tVVNFUkZpcnN0LUNsaWVudEF1dGhlbnRpY2F0aW9u
YW5kRW1haWwuY3JsMGwGCCsGAQUFBwEBBGAwXjA2BggrBgEFBQcwAoYqaHR0cDovL2NydC5jb21v
ZG9jYS5jb20vVVROQUFBQ2xpZW50Q0EuY3J0MCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5jb21v
ZG9jYS5jb20wGwYDVR0RBBQwEoEQc2NodWthbm92QHJiYy5ydTANBgkqhkiG9w0BAQUFAAOCAQEA
BlpMmvXGu2lQxIgsoCQKxof39m0s4oKtyDhrAijZRxFrgcF+e7SFqqHiLMYdSvolF462oB37lHu0
KYzuQ2xftB8Mt+Ve0KAMNUksnC0VizcAWBjuo1FLmtwRPVBmvrX3wtrHdW4y1CP+FFMJa/j2XGtN
NE91ymab0XjENjCKR/E2iGSdS9cP/SSnsKD7KhW9O5yqu8qHbLZ8SMOYdnK5FuQdlbYNu5Utm+zj
WJuH6n/MFecuzi00VcELkK/HhL7YcGkR8bfWwEduHxJwINnhtcacxDl6M+lTVquc0n4rUjKwtT68
MxKc1c036Muwr5GWHZLB5ytdOMtuQJlbNI0v5zGCBGgwggRkAgEBMIHEMIGuMQswCQYDVQQGEwJV
UzELMAkGA1UECBMCVVQxFzAVBgNVBAcTDlNhbHQgTGFrZSBDaXR5MR4wHAYDVQQKExVUaGUgVVNF
UlRSVVNUIE5ldHdvcmsxITAfBgNVBAsTGGh0dHA6Ly93d3cudXNlcnRydXN0LmNvbTE2MDQGA1UE
AxMtVVROLVVTRVJGaXJzdC1DbGllbnQgQXV0aGVudGljYXRpb24gYW5kIEVtYWlsAhEAqkHsCrC8
YxFiTvYgVtPIRTAJBgUrDgMCGgUAoIICeDAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqG
SIb3DQEJBTEPFw0wOTEyMDEwODM0NTNaMCMGCSqGSIb3DQEJBDEWBBRZ5Png6bSSmkao+dQNEDEv
cDrjtjBnBgkqhkiG9w0BCQ8xWjBYMAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggqhkiG
9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG9w0DAgIBKDAHBgUrDgMCGjAKBggqhkiG9w0CBTCB1QYJ
KwYBBAGCNxAEMYHHMIHEMIGuMQswCQYDVQQGEwJVUzELMAkGA1UECBMCVVQxFzAVBgNVBAcTDlNh
bHQgTGFrZSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxITAfBgNVBAsTGGh0
dHA6Ly93d3cudXNlcnRydXN0LmNvbTE2MDQGA1UEAxMtVVROLVVTRVJGaXJzdC1DbGllbnQgQXV0
aGVudGljYXRpb24gYW5kIEVtYWlsAhEAqkHsCrC8YxFiTvYgVtPIRTCB1wYLKoZIhvcNAQkQAgsx
gceggcQwga4xCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJVVDEXMBUGA1UEBxMOU2FsdCBMYWtlIENp
dHkxHjAcBgNVBAoTFVRoZSBVU0VSVFJVU1QgTmV0d29yazEhMB8GA1UECxMYaHR0cDovL3d3dy51
c2VydHJ1c3QuY29tMTYwNAYDVQQDEy1VVE4tVVNFUkZpcnN0LUNsaWVudCBBdXRoZW50aWNhdGlv
biBhbmQgRW1haWwCEQCqQewKsLxjEWJO9iBW08hFMA0GCSqGSIb3DQEBAQUABIIBAMiOZKXBaUMT
AieGJp5Fk/efh2NFF9pkqTHIrLIBRyh6qK7HNZbRDpB6+q5RGhPpz9hkeQ/Y2VsyEz4WnJiBIMfG
9UMNBbt1cO8rHpCZqpW7GZCU6LuAjfJtNZSL/x7mdxVRNoI0l3Ma3Qq8ly2DWpPSr1V1LKfAN8TR
7mw2Jev6v03WyOeHOouObGvFSDuu3idpTHjQRw+IXgvPK0Sxno6tjiUFYO6/aPMQy2Jj2k9GkoFq
yePT7NDTdljqXL9Xa3A5L3sCoQ71phfzLFilR4UsEeEvXv4Kybmt8oW0SIdSUIWHiXSyAjJxdHoo
pEfEOoUx3oawmE0QD2l7+wTBPkYAAAAAAAA=

------=_NextPart_000_00FC_01CA727A.4CEC4690--


From general-return-810-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 01 18:38:17 2009
Return-Path: <general-return-810-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 65169 invoked from network); 1 Dec 2009 18:38:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Dec 2009 18:38:16 -0000
Received: (qmail 29133 invoked by uid 500); 1 Dec 2009 18:38:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29033 invoked by uid 500); 1 Dec 2009 18:38:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29023 invoked by uid 99); 1 Dec 2009 18:38:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Dec 2009 18:38:14 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_SOFTFAIL,UNPARSEABLE_RELAY
X-Spam-Check-By: apache.org
Received-SPF: softfail (nike.apache.org: transitioning domain of benjamin.cotton@lehman.com does not designate 69.147.102.67 as permitted sender)
Received: from [69.147.102.67] (HELO smtp104.plus.mail.re1.yahoo.com) (69.147.102.67)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 01 Dec 2009 18:38:06 +0000
Received: (qmail 43181 invoked from network); 1 Dec 2009 18:37:44 -0000
Received: from dsl027-139-111.nyc1.dsl.speakeasy.net (benjamin.cotton@216.27.139.111 with plain)
        by smtp104.plus.mail.re1.yahoo.com with SMTP; 01 Dec 2009 10:37:44 -0800 PST
X-Yahoo-SMTP: Fl3kciCswBCpO0rt4AQpoUBtfUKa8rc-
X-YMail-OSG: Gbb1JfUVM1nNhYqOKhZGigbka_fWHJnpIJUxXkT9eZK0g5XAiX2dmQsC4hKvlXJPkeXjebMnhTxPZWemVNQM0K2MpSZYD8fxpF1xet8G_QcfkiNi14U5tgKMy8szqaPKjXvqLjgDjGJuvGU4HQVjQ8pv0JjRfnWEIO25i1RNciCNsU5PsZk1X2nA7_AIs9ORDTWgkeA6GI3Pah1JrsBiIin.0GKOMQrSnTmAOX8JVAzQInisZREEzeIil.NgwoirS8O1D8SYrf.6yRriZ71VIXCaEKB2gBQLmfbaa8Nb4YBYQrk7wh9IhR8uJucAJHiHmzC48PuG7WNr07.vUj9xIEUYQUruP4mmRlHEZ7ON21kb1C4l7gxxhWXbUW.pVdvbddpkrk1cOVGJdLnrc.d_2yQRAzIWjp1DJdoqwfSM2Xr1_jEfwLKw.5Y-
X-Yahoo-Newman-Property: ymail-3
Message-ID: <4B156277.20404@lehman.com>
Date: Tue, 01 Dec 2009 13:37:43 -0500
From: "ben.cotton@lehman.com" <benjamin.cotton@lehman.com>
Reply-To: ben.cotton@lehman.com
User-Agent: Thunderbird 2.0.0.23 (Windows/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: How to configure read/write/execute ACLs ?
Content-Type: multipart/alternative;
 boundary="------------040007070207050107020904"
X-Virus-Checked: Checked by ClamAV on apache.org

--------------040007070207050107020904
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

How do I configure Hadoop ACLs to specify a uid's  read/write/execute 
privileges?

--------------040007070207050107020904--

From general-return-811-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 01 18:51:44 2009
Return-Path: <general-return-811-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69310 invoked from network); 1 Dec 2009 18:51:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Dec 2009 18:51:44 -0000
Received: (qmail 59500 invoked by uid 500); 1 Dec 2009 18:51:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59402 invoked by uid 500); 1 Dec 2009 18:51:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59392 invoked by uid 99); 1 Dec 2009 18:51:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Dec 2009 18:51:42 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.201] (HELO mail-px0-f201.google.com) (209.85.216.201)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Dec 2009 18:51:31 +0000
Received: by pxi39 with SMTP id 39so4069578pxi.2
        for <general@hadoop.apache.org>; Tue, 01 Dec 2009 10:51:10 -0800 (PST)
MIME-Version: 1.0
Received: by 10.142.247.6 with SMTP id u6mr699636wfh.181.1259693469304; Tue, 
	01 Dec 2009 10:51:09 -0800 (PST)
In-Reply-To: <FF159CAD6222408FB0A5B7FC649D14C7@users.rbc.ru>
References: <FF159CAD6222408FB0A5B7FC649D14C7@users.rbc.ru>
From: Todd Lipcon <todd@cloudera.com>
Date: Tue, 1 Dec 2009 10:50:49 -0800
Message-ID: <45f85f70912011050r79e9c494j6eb8c8f05ae09b09@mail.gmail.com>
Subject: Re: FSNamesystem initialization failed.
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502c5fcc386ac0479af3dc9
X-Virus-Checked: Checked by ClamAV on apache.org

--00504502c5fcc386ac0479af3dc9
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi Sergey,

I replied to your post in common-user - please don't double-post in the
future - just makes the threads harder to follow for others who might have
the same problem.

Thanks
-Todd

2009/12/1 =D0=A7=D1=83=D0=BA=D0=B0=D0=BD=D0=BE=D0=B2 =D0=A1=D0=B5=D1=80=D0=
=B3=D0=B5=D0=B9 <schukanov@rbc.ru>

>  Hi, suddenly I=E2=80=99ve got problem with starting Namenode:
>
>
>
> [hadoop@hadoop1 hadoop]$ bin/hadoop namenode
>
> 09/12/01 11:26:51 INFO namenode.NameNode: STARTUP_MSG:
>
> /************************************************************
>
> STARTUP_MSG: Starting NameNode
>
> STARTUP_MSG:   host =3D hadoop1.bcr.ru/81.68.243.18
>
> STARTUP_MSG:   args =3D []
>
> STARTUP_MSG:   version =3D 0.19.2-dev
>
> STARTUP_MSG:   build =3D
> http://svn.apache.org/repos/asf/hadoop/core/branches/bran
>
> ch-0.19 -r 755955; compiled by 'maksim07' on Sun Mar 22 14:29:37 MSK 2009
>
> ************************************************************/
>
> 09/12/01 11:26:51 INFO metrics.RpcMetrics: Initializing RPC Metrics with
> hostName=3DNameNode, port=3D9000
>
> 09/12/01 11:26:51 INFO namenode.NameNode: Namenode up at:
> hadoop1.bcr.ru/81.68.243.18:9000
>
> 09/12/01 11:26:51 INFO jvm.JvmMetrics: Initializing JVM Metrics with
> processName=3DNameNode, sessionId=3Dnull
>
> 09/12/01 11:26:51 INFO metrics.NameNodeMetrics: Initializing
> NameNodeMeterics using context
> object:org.apache.hadoop.metrics.spi.NullContext
>
> 09/12/01 11:26:52 INFO namenode.FSNamesystem: fsOwner=3Dhadoop,hadoop,dba
>
> 09/12/01 11:26:52 INFO namenode.FSNamesystem: supergroup=3Dsupergroup
>
> 09/12/01 11:26:52 INFO namenode.FSNamesystem: isPermissionEnabled=3Dtrue
>
> 09/12/01 11:26:52 INFO metrics.FSNamesystemMetrics: Initializing
> FSNamesystemMet
>
> rics using context object:org.apache.hadoop.metrics.spi.NullContext
>
> 09/12/01 11:26:52 INFO namenode.FSNamesystem: Registered
> FSNamesystemStatusMBean
>
> 09/12/01 11:26:52 INFO common.Storage: Number of files =3D 30079
>
> 09/12/01 11:26:53 INFO common.Storage: Number of files under construction=
 =3D
> 0
>
> 09/12/01 11:26:53 INFO common.Storage: Image file of size 4451066 loaded =
in
> 1 seconds.
>
> 09/12/01 11:27:11 ERROR namenode.FSNamesystem: FSNamesystem initializatio=
n
> failed.
>
> java.io.IOException: Incorrect data format. logVersion is -18 but
> writables.length is 0.
>
>         at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.ja=
va:542)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:9=
73)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:7=
93)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSIm=
age.java:352)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirector=
y.java:87)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesyst=
em.java:309)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.j=
ava:288)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:=
163)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:208)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:194)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.j=
ava:859)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:868)
>
> 09/12/01 11:27:11 INFO ipc.Server: Stopping server on 9000
>
> 09/12/01 11:27:11 ERROR namenode.NameNode: java.io.IOException: Incorrect
> data f
>
> ormat. logVersion is -18 but writables.length is 0.
>
>         at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.ja=
va:542)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:9=
73)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:7=
93)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSIm=
age.java:352)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirector=
y.java:87)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesyst=
em.java:309)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.j=
ava:288)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:=
163)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:208)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:194)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.j=
ava:859)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:868)
>
> 09/12/01 11:27:11 INFO namenode.NameNode: SHUTDOWN_MSG:
>
> /************************************************************
>
> SHUTDOWN_MSG: Shutting down NameNode at hadoop1.bcr.ru/81.68.243.18
>
> ************************************************************/
>
>
>
> Any ideas?
>
>
>
> Thanks!
>
>
>

--00504502c5fcc386ac0479af3dc9--

From general-return-812-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 01 22:18:27 2009
Return-Path: <general-return-812-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45864 invoked from network); 1 Dec 2009 22:18:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Dec 2009 22:18:27 -0000
Received: (qmail 65815 invoked by uid 500); 1 Dec 2009 22:18:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65733 invoked by uid 500); 1 Dec 2009 22:18:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65723 invoked by uid 99); 1 Dec 2009 22:18:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Dec 2009 22:18:25 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Dec 2009 22:18:14 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:User-Agent:Date:Subject:From:To:
   Message-ID:Thread-Topic:Thread-Index:In-Reply-To:
   Mime-version:Content-type:Content-transfer-encoding;
  b=V7wYNfPa7Ac44pM4qFdF52X1rdVj/QpbIjOmBFhWHdU1nLg0u7RH8cvr
   M3mrBXPA7ft8ATSp4PtG6FYrsw20VoWc5Zq0au/W8/zYvhxs0DUblweun
   67ACZYqRlI8T06x;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1259705894; x=1291241894;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20How=20to=20configure=20read/write/execu
   te=20ACLs=20?|Date:=20Tue,=2001=20Dec=202009=2014:17:40
   =20-0800|Message-ID:=20<C73AD604.1A1C%awittenauer@linkedi
   n.com>|To:=20<general@hadoop.apache.org>|Mime-version:=20
   1.0|Content-transfer-encoding:=207bit|In-Reply-To:=20<4B1
   56277.20404@lehman.com>;
  bh=NEQd0SClfNXJIqqqvksTd6Rz+SpmTyXpG1t1ZhKBkfs=;
  b=ZYmaHMuPzfkxVoh9oEYsZSRywqITfUFAAZE0Yr+WCl1ENdS3S+11z99Z
   46Ar63A0GWbk9ofjEH+A8ApGdKj42dUWZNE9by4vGk0bnJeGgo7Ehprva
   ch/8CkEWjHnzv9J;
X-IronPort-AV: E=Sophos;i="4.47,323,1257148800"; 
   d="scan'208";a="9759504"
Received: from 172.16.19.135 ([172.16.19.135]) by CORP-MAIL.linkedin.biz ([172.18.46.135]) via Exchange Front-End Server mail-access.linkedin.biz ([172.18.46.133]) with Microsoft Exchange Server HTTP-DAV ;
 Tue,  1 Dec 2009 22:17:43 +0000
User-Agent: Microsoft-Entourage/12.10.0.080409
Date: Tue, 01 Dec 2009 14:17:40 -0800
Subject: Re: How to configure read/write/execute ACLs ?
From: Allen Wittenauer <awittenauer@linkedin.com>
To: <general@hadoop.apache.org>
Message-ID: <C73AD604.1A1C%awittenauer@linkedin.com>
Thread-Topic: How to configure read/write/execute ACLs ?
Thread-Index: Acpy1BhP+rsxKr0PR0e+1vtdldXPfg==
In-Reply-To: <4B156277.20404@lehman.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org




On 12/1/09 10:37 AM, "ben.cotton@lehman.com" <benjamin.cotton@lehman.com>
wrote:

> How do I configure Hadoop ACLs to specify a uid's  read/write/execute
> privileges?

If I parse your question correctly, you want to limit certain uids to have
only be able to read or write certain data?  That functionality doesn't
exist.  The permission system is mainly to prevent accidents; it is not real
security. [Uids are trivial to forge. See my "Hadoop 24/7" slide deck from
Apachecon EU for more on Hadoop's busted security model.]



From general-return-813-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 01 23:14:46 2009
Return-Path: <general-return-813-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66068 invoked from network); 1 Dec 2009 23:14:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Dec 2009 23:14:46 -0000
Received: (qmail 38254 invoked by uid 500); 1 Dec 2009 23:14:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38169 invoked by uid 500); 1 Dec 2009 23:14:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 38159 invoked by uid 99); 1 Dec 2009 23:14:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Dec 2009 23:14:45 +0000
X-ASF-Spam-Status: No, hits=-4.1 required=5.0
	tests=AWL,BAYES_00,MIME_QP_LONG_LINE,RCVD_IN_DNSWL_MED
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of vogt7@llnl.gov designates 128.115.41.81 as permitted sender)
Received: from [128.115.41.81] (HELO nspiron-1.llnl.gov) (128.115.41.81)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Dec 2009 23:14:43 +0000
X-Attachments: None
Received: from poisonivy.llnl.gov (HELO [128.115.210.109]) ([128.115.210.109])
  by nspiron-1.llnl.gov with ESMTP; 01 Dec 2009 15:14:22 -0800
User-Agent: Microsoft-Entourage/12.23.0.091001
Date: Tue, 01 Dec 2009 15:14:22 -0800
Subject: Re: Scribe vs. Chukwa
From: Kim Vogt <vogt7@llnl.gov>
To: <general@hadoop.apache.org>
Message-ID: <C73AE34E.4549%vogt7@llnl.gov>
Thread-Topic: Scribe vs. Chukwa
Thread-Index: Acpx9u50+HtKxaEtHUuAzUfAGnnRsgAEPdkXADUHjZE=
In-Reply-To: <C7397F6E.53EA%eyang@yahoo-inc.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="ISO-8859-1"
Content-transfer-encoding: quoted-printable

Hi Eric,

Thanks for the info.  It doesn't look like Scribe comes with a log file
viewer, but I could be wrong?

I'm attempting to install Scribe now, and hopefully if I have time, do the
same with Chukwa to get a better feel for both.  I'm interested in checking
out the reporting tools Chukwa offers.

-Kim

On 11/30/09 1:55 PM, "Eric Yang" <eyang@yahoo-inc.com> wrote:

> Hi Kim,
>=20
> Scribe works well for simple deployment.  The complexity increases when
> "central scribe server" is multi-machines deployment.  Basically, it
> requires a reverse proxy to load balance the data collection.  (
> http://*www.*cloudera.com/blog/2008/11/02/configuring-and-using-scribe-fo=
r-had
> oop-log-collection/ )  I have not used scribe personally, therefore someo=
ne
> else could fill in the experience.
>=20
> Chukwa was designed to be fault tolerant log collection/analytics platfor=
m.
> Each chukwa agent automatically creates it's own routing table to chukwa
> collectors.  Therefore, Chukwa does not require a reverse proxy.  However=
,
> Chukwa Agent requires knowledge of all collector addresses, hence the
> initial deployment complexity may be a little higher than scribe.  The
> largest test for Chukwa deployment was 50 Chukwa collectors running on to=
p
> of 100 dedicated hadoop nodes to process log files from a data center.
> (Which was decommissioned due to lack of log files)  Base on my experienc=
e,
> a single collector with 8GB allocated RAM could handle all log files from
> 2000 hadoop nodes + System Metrics (top, df, sar, iostat, netstat, vmstat
> output). =20
>=20
> Chukwa does not have a direct log file viewer, instead, it has an analyti=
cs
> engine which computes various facts and provide reports.  There are frequ=
ent
> requests about log file viewer but it hasn't been implemented.  We only h=
ave
> command line utility to dump the log files because it is difficult to vie=
w
> terabytes of log file.  At some point in the future, when a full body ind=
ex
> engine is implemented, then we will provide log file search.
>=20
> In essence, it depends on what you are looking for.  If you are looking f=
or
> simple log collection and viewer, Scribe is probably a good tool.  If you
> are looking for log collection and reporting platform, Chukwa is a good
> solution.
>=20
> Regards,
> Eric
>=20
> On 11/30/09 11:54 AM, "Kim Vogt" <vogt7@llnl.gov> wrote:
>=20
>> Hi,
>>=20
>> My team is looking into using Scribe or Chukwa for hadoop log collection=
.  I
>> was wondering if anyone had any opinions about one vs. the other?  I
>> apologize if this topic was covered before, but I don=B9t see a link to th=
e
>> archives for this mailing list.
>>=20
>> Thanks,
>>=20
>> Kim
>=20
>=20



From general-return-814-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 02 07:40:19 2009
Return-Path: <general-return-814-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12345 invoked from network); 2 Dec 2009 07:40:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Dec 2009 07:40:19 -0000
Received: (qmail 81370 invoked by uid 500); 2 Dec 2009 07:40:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81283 invoked by uid 500); 2 Dec 2009 07:40:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81273 invoked by uid 99); 2 Dec 2009 07:40:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Dec 2009 07:40:16 +0000
X-ASF-Spam-Status: No, hits=-1.5 required=5.0
	tests=AWL,BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.30.24] (HELO QMTA02.emeryville.ca.mail.comcast.net) (76.96.30.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Dec 2009 07:40:14 +0000
Received: from OMTA08.emeryville.ca.mail.comcast.net ([76.96.30.12])
	by QMTA02.emeryville.ca.mail.comcast.net with comcast
	id C7Pi1d0060FhH24A27fvXz; Wed, 02 Dec 2009 07:39:55 +0000
Received: from [10.0.0.13] ([98.234.216.58])
	by OMTA08.emeryville.ca.mail.comcast.net with comcast
	id C7fu1d0011GAoR68U7fu4k; Wed, 02 Dec 2009 07:39:54 +0000
Message-Id: <57989512-60C3-4AD7-8AD5-3A3F0DCD4F91@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <7b3355cb0911270941s219c492w63d8ce6c83eeb96b@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Introducing Cloud MapReduce
Date: Tue, 1 Dec 2009 23:39:52 -0800
References: <7b3355cb0911270941s219c492w63d8ce6c83eeb96b@mail.gmail.com>
X-Mailer: Apple Mail (2.936)


On Nov 27, 2009, at 9:41 AM, Bruce Snyder wrote:

> 1) It is faster than other implementations (e.g., 60 times faster than
> Hadoop in one case. Speedup depends on the application and data.).

Based on your report, it looks like your comparison was done using  
1.2GB of data in 92,000 files. If you took the default configuration,  
you'll end up with 92,000 maps, each of them processing a very small  
input. You either need to use MultiFileInputFormat or use Hadoop  
Archives to bundle your tiny files into reasonable size. For input  
data that small you should not have more than 8 maps unless each map  
is doing a lot of CPU work.

In your reduce push iterator model, you give up a lot of determinism  
in your application. Many applications that run in Hadoop depend on  
the reduces getting keys in sorted order. Getting keys in a random  
order won't work. Furthermore, I don't see any way that you can scale  
up your approach. You effectively need to hold open all of the reduce  
states for the keys you've seen and can't close them until it is done.  
Again, that will quickly exhaust memory. So the push model will work  
for something like word count, but fail on large problems.

When you add types in, you should add them in as:

K1, V1 -> map -> K2,V2 -> reduce -> K3,V3

with the combiner taking:

K2,V2 -> combiner -> K2,V2

Of course the types are often not primitives and using something like  
Avro is a good thing.

-- Owen

From general-return-815-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 02 08:46:14 2009
Return-Path: <general-return-815-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27363 invoked from network); 2 Dec 2009 08:46:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Dec 2009 08:46:14 -0000
Received: (qmail 60488 invoked by uid 500); 2 Dec 2009 08:46:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60407 invoked by uid 500); 2 Dec 2009 08:46:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60397 invoked by uid 99); 2 Dec 2009 08:46:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Dec 2009 08:46:12 +0000
X-ASF-Spam-Status: No, hits=-2.2 required=5.0
	tests=AWL,BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yangxiao9901@gmail.com designates 209.85.222.195 as permitted sender)
Received: from [209.85.222.195] (HELO mail-pz0-f195.google.com) (209.85.222.195)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Dec 2009 08:46:09 +0000
Received: by pzk33 with SMTP id 33so4306411pzk.2
        for <general@hadoop.apache.org>; Wed, 02 Dec 2009 00:45:49 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=Fz+x9Erk0PtJEnepfHCb8q9QE87/eXpNMD+IzOin/HY=;
        b=wBxrKJ41xA6o0nkY+VdVvfGkwjsW7I8ioAVhqKzN/nccgfVDdFHyANZO9kRRiFuYT4
         3g5k+r/MTmSFUTW/ax7RAOW0h6jNVJbWW8LAvjWOLgAwCqP1sIVj0ipOGGW7abROb5eW
         cx2AX+vTIMV7xxoszY0LWQ5hjYbQRfCZZI/ZU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=QCLsAIBsfjs8nUDLM7JnOvt185edwttu2US8nG/LlELyfXroL3qSNXXA/9Qr0H7Kfo
         EKbLs1/rFtVYNoTCLaSXcfr5IV3hF4T6oy7bftYdz6DISlQ9oHg4dwuDXBEA+yIN1c+M
         2WsxstCa3tk9g+7LqvJR0usr+4OiiFpeEHtbQ=
MIME-Version: 1.0
Received: by 10.140.164.9 with SMTP id m9mr407624rve.239.1259743549720; Wed, 
	02 Dec 2009 00:45:49 -0800 (PST)
Date: Wed, 2 Dec 2009 16:45:49 +0800
Message-ID: <5a921af20912020045q72334148qeeee55e601961343@mail.gmail.com>
Subject: What if I want do random write in Hadoop?
From: xiao yang <yangxiao9901@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

FSDataInputStream is seekable, but FSDataOutputStream is not?
Why? What are the difficulties to support random write?

Thanks!
Xiao

From general-return-816-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 02 09:48:17 2009
Return-Path: <general-return-816-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53614 invoked from network); 2 Dec 2009 09:48:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Dec 2009 09:48:17 -0000
Received: (qmail 61414 invoked by uid 500); 2 Dec 2009 09:48:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61332 invoked by uid 500); 2 Dec 2009 09:48:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61321 invoked by uid 99); 2 Dec 2009 09:48:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Dec 2009 09:48:15 +0000
X-ASF-Spam-Status: No, hits=-1.5 required=5.0
	tests=AWL,BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bernd.fondermann@googlemail.com designates 209.85.212.174 as permitted sender)
Received: from [209.85.212.174] (HELO mail-vw0-f174.google.com) (209.85.212.174)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Dec 2009 09:48:12 +0000
Received: by vws4 with SMTP id 4so5722vws.2
        for <general@hadoop.apache.org>; Wed, 02 Dec 2009 01:47:52 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=u1BLMQ+WDVLc+68jkh3c8OM8ShtAoATtdxneNBUnpls=;
        b=jEJf/03WHa+epXFOK0WcStSI+9n0nafZrFhiEaK7JmkilR2THocvDVwIYMiFV28Clx
         f3sWHcexaelT59ayJpoNFLZQy0C0rjA8O3BwfqLLPy8nE7XGtVW/bZP9xPbYPqAatsiL
         3mDNE6J13jsqAitFX9cfVclZLT0p+XSXeb/nU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=jXQhUXIrlAuQfn3MDvRdOavcemVPErZ2QhqNtf485G9meIQ3rfR8hd3Z3VodWhqx5X
         F9wKK1NS4tFnHsL730v6N+9YTeJozSVOYixSP7XJEOKe7EU6Rc4ajggxyZ5TiS/pYr4u
         hdOoNOQMzkuMQlRql6neqcGPfPtYnoJhDz0iw=
MIME-Version: 1.0
Received: by 10.220.127.36 with SMTP id e36mr8518749vcs.4.1259747271883; Wed, 
	02 Dec 2009 01:47:51 -0800 (PST)
In-Reply-To: <5a921af20912020045q72334148qeeee55e601961343@mail.gmail.com>
References: <5a921af20912020045q72334148qeeee55e601961343@mail.gmail.com>
Date: Wed, 2 Dec 2009 10:47:51 +0100
Message-ID: <88f6e29a0912020147med0be70ka7ecd1c242d5bc86@mail.gmail.com>
Subject: Re: What if I want do random write in Hadoop?
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Dec 2, 2009 at 09:45, xiao yang <yangxiao9901@gmail.com> wrote:
> FSDataInputStream is seekable, but FSDataOutputStream is not?
> Why? What are the difficulties to support random write?

From
http://hadoop.apache.org/common/docs/r0.20.0/hdfs_design.html#Assumptions+and+Goals
>>>>
 Simple Coherency Model

HDFS applications need a write-once-read-many access model for files.
A file once created, written, and closed need not be changed. This
assumption simplifies data coherency issues and enables high
throughput data access. A Map/Reduce application or a web crawler
application fits perfectly with this model. There is a plan to support
appending-writes to files in the future.
<<<<

Data in HDFS is replicated (potentially across data center
boundaries). While changing a file, old copies of the data remain.
This results in consistency problems when massively reading in
parallel, one of the strength of HDFS. To avoid these complications,
changing written data is not possible.

Other distributed systems, like for example Apache CouchDB, have
different consistency models.

  Bernd

From general-return-817-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 02 13:07:04 2009
Return-Path: <general-return-817-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 15007 invoked from network); 2 Dec 2009 13:07:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Dec 2009 13:07:03 -0000
Received: (qmail 3738 invoked by uid 500); 2 Dec 2009 13:07:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3636 invoked by uid 500); 2 Dec 2009 13:07:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3625 invoked by uid 99); 2 Dec 2009 13:07:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Dec 2009 13:07:01 +0000
X-ASF-Spam-Status: No, hits=3.7 required=10.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of fredwang222@gmail.com designates 209.85.160.50 as permitted sender)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Dec 2009 13:06:52 +0000
Received: by pwi19 with SMTP id 19so110610pwi.29
        for <general@hadoop.apache.org>; Wed, 02 Dec 2009 05:06:31 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=iEhnjiaEOSK4Z3uOOfYjCyH55uuifYeDDhnBoNsbp8g=;
        b=mOvb1UqMb8r+FIi/iC8gocVb5+xYaR2hzuMNDfWwHRHTRRZq60naaOHfMmjmfc3r+t
         LNsTCdNn0X6y5KjXQjethF6uvHe6jQboXccXmeMT0A6X8p5XNHWIucoyHkfc84iCZ/4g
         7sjlZz1K8k+8LF9ugv0iiuXQ6vOhRNCfHq5gQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=DDunLjjKxP4Q099AQQ1MT8kbgpVLtEH0GxnMrm3PxtG/XkMr6h2IivE0SZqTA8Zfk9
         Q1GVQxN2sGINd0gUlqdgxKO83X2pa0HNZD73XubuhdWRpoiC5/+9BEvBQ3k74XbsWPhi
         JTC/06tlWHqL54tPW4/jQeOhrYwpoDXjCdWo0=
MIME-Version: 1.0
Received: by 10.143.137.2 with SMTP id p2mr8404wfn.136.1259759191351; Wed, 02 
	Dec 2009 05:06:31 -0800 (PST)
Date: Wed, 2 Dec 2009 21:06:31 +0800
Message-ID: <702261350912020506q49c66b0brf459a875df3a21a1@mail.gmail.com>
Subject: help needed: mapred Taskrunner(hadoop 0.20.1) was marked as 
	successful but with commuication error at random
From: fred wang <fredwang222@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd4ce921a63880479be8b03
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd4ce921a63880479be8b03
Content-Type: text/plain; charset=ISO-8859-1

It occures in our hadoop cluster)(0.20.1 version) from time to time(the log
is below). Anyone could help? what hadoop is doing in the course of(after)
committing before the map task is done? joWhat kind of reasons could make
communication error. The job is still regarded as succeed even with this
communication error. Thanks for the help.


2009-11-25 01:21:30,476 WARN org.apache.hadoop.fs.FileSystem: "master1:9000"
is a deprecated filesystem name. Use "hdfs://master1:9000/" instead.
2009-11-25 01:21:30,541 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=MAP, sessionId=
2009-11-25 01:21:30,559 WARN org.apache.hadoop.fs.FileSystem: "master1:9000"
is a deprecated filesystem name. Use "hdfs://master1:9000/" instead.
...
2009-11-25 01:21:31,616 INFO org.apache.hadoop.mapred.MapTask:
numReduceTasks: 27
2009-11-25 01:21:31,624 WARN org.apache.hadoop.fs.FileSystem: "master1:9000"
is a deprecated filesystem name. Use "hdfs://master1:9000/" instead.
2009-11-25 01:21:31,627 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb =
100
2009-11-25 01:21:31,799 INFO org.apache.hadoop.mapred.MapTask: data buffer =
79691776/99614720
2009-11-25 01:21:31,799 INFO org.apache.hadoop.mapred.MapTask: record buffer
= 262144/327680
2009-11-25 01:21:33,266 INFO org.apache.hadoop.mapred.MapTask: Starting
flush of map output
2009-11-25 01:21:33,277 WARN org.apache.hadoop.fs.FileSystem: "master1:9000"
is a deprecated filesystem name. Use "hdfs://master1:9000/" instead.
2009-11-25 01:21:37,187 INFO org.apache.hadoop.mapred.MapTask: Finished
spill 0
2009-11-25 01:21:37,236 WARN org.apache.hadoop.fs.FileSystem: "master1:9000"
is a deprecated filesystem name. Use "hdfs://master1:9000/" instead.
2009-11-25 01:21:37,271 INFO org.apache.hadoop.mapred.TaskRunner:
Task:attempt_200911171829_0181_m_000008_0 is done. And is in the process of
commiting
2009-11-25 01:22:37,323 INFO org.apache.hadoop.mapred.TaskRunner:
Communication exception: java.io.IOException: Call to
/127.0.0.1:53674failed on local exception:
java.nio.channels.ClosedByInterruptException
at org.apache.hadoop.ipc.Client.wrapException(Client.java:774)
at org.apache.hadoop.ipc.Client.call(Client.java:742)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at org.apache.hadoop.mapred.$Proxy0.statusUpdate(Unknown Source)
at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:543)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.nio.channels.ClosedByInterruptException
at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:341)
at
org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:480)
at org.apache.hadoop.ipc.Client.call(Client.java:720)
... 4 more
2009-11-25 01:22:37,338 INFO org.apache.hadoop.mapred.TaskRunner: Task
'attempt_200911171829_0181_m_000008_0' done.

--000e0cd4ce921a63880479be8b03--

From general-return-818-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 02 14:47:47 2009
Return-Path: <general-return-818-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60157 invoked from network); 2 Dec 2009 14:47:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Dec 2009 14:47:47 -0000
Received: (qmail 28452 invoked by uid 500); 2 Dec 2009 14:47:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28381 invoked by uid 500); 2 Dec 2009 14:47:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28371 invoked by uid 99); 2 Dec 2009 14:47:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Dec 2009 14:47:45 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE,UNPARSEABLE_RELAY
X-Spam-Check-By: apache.org
Received-SPF: softfail (athena.apache.org: transitioning domain of benjamin.cotton@lehman.com does not designate 69.147.102.75 as permitted sender)
Received: from [69.147.102.75] (HELO smtp112.plus.mail.re1.yahoo.com) (69.147.102.75)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 02 Dec 2009 14:47:43 +0000
Received: (qmail 25639 invoked from network); 2 Dec 2009 14:47:22 -0000
Received: from dsl027-139-111.nyc1.dsl.speakeasy.net (benjamin.cotton@216.27.139.111 with plain)
        by smtp112.plus.mail.re1.yahoo.com with SMTP; 02 Dec 2009 06:47:21 -0800 PST
X-Yahoo-SMTP: Fl3kciCswBCpO0rt4AQpoUBtfUKa8rc-
X-Yahoo-Newman-Property: ymail-3
Message-ID: <4B167DF8.6090800@lehman.com>
Date: Wed, 02 Dec 2009 09:47:20 -0500
From: "ben.cotton@lehman.com" <benjamin.cotton@lehman.com>
Reply-To: ben.cotton@lehman.com
User-Agent: Thunderbird 2.0.0.23 (Windows/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: How to configure read/write/execute ACLs ?
References: <4D22359F6247DA4D8672EDA1CC864E6C6B521B@dnpcor1exum001.aur.alservices.com>
In-Reply-To: <4D22359F6247DA4D8672EDA1CC864E6C6B521B@dnpcor1exum001.aur.alservices.com>
Content-Type: multipart/alternative;
 boundary="------------010006020004060609020704"

--------------010006020004060609020704
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit


Thanks for this response, and the reference to your ApacheCon 
presentation on Hadoop Security (lack of).   It would seem that the 
conventions being used in conf/hadoop-policy.xml might be a basis to 
provide a user/role level ACL capability in Hadoop.  In the meantime, we 
will follow your suggested strategy to provision users only on grids 
with data they can use.


>   
>> How do I configure Hadoop ACLs to specify a uid's  read/write/execute
>> privileges?
>>     
>
> If I parse your question correctly, you want to limit certain uids to
> have only be able to read or write certain data?  That functionality doesn't exist.  The permission system is mainly to prevent accidents; it is not real security. [Uids are trivial to forge. See my "Hadoop 24/7" slide deck from Apachecon EU for more on Hadoop's busted security model.]
>
>
>   

--------------010006020004060609020704--

From general-return-819-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 03 17:41:33 2009
Return-Path: <general-return-819-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 92635 invoked from network); 3 Dec 2009 17:41:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Dec 2009 17:41:32 -0000
Received: (qmail 27842 invoked by uid 500); 3 Dec 2009 17:41:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27775 invoked by uid 500); 3 Dec 2009 17:41:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27765 invoked by uid 99); 3 Dec 2009 17:41:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Dec 2009 17:41:30 +0000
X-ASF-Spam-Status: No, hits=-6.6 required=5.0
	tests=AWL,BAYES_00,RCVD_IN_DNSWL_MED
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of huan.liu@accenture.com designates 170.252.248.71 as permitted sender)
Received: from [170.252.248.71] (HELO amrmr1002.accenture.com) (170.252.248.71)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Dec 2009 17:41:28 +0000
Received: from AMRXV1003.dir.svc.accenture.com (amrxv1003.dir.svc.accenture.com [10.10.160.63])
	by amrmr1002.accenture.com (8.13.8/8.13.8) with ESMTP id nB3Hex5s018431
	for <general@hadoop.apache.org>; Thu, 3 Dec 2009 11:41:03 -0600 (CST)
Received: from AMRXH3001.dir.svc.accenture.com ([10.63.34.23]) by AMRXV1003.dir.svc.accenture.com with Microsoft SMTPSVC(6.0.3790.1830); Thu, 3 Dec 2009 11:40:21 -0600
Received: from AMRXM3124.dir.svc.accenture.com ([10.63.34.14]) by AMRXH3001.dir.svc.accenture.com ([10.63.34.23]) with mapi; Thu, 3 Dec 2009 12:40:21 -0500
From: <huan.liu@accenture.com>
To: <general@hadoop.apache.org>
Content-Class: urn:content-classes:message
Importance: normal
Priority: normal
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.3790.3168
Date: Thu, 3 Dec 2009 12:39:41 -0500
Subject: RE: what is the major difference between Hadoop and cloudMapReduce?
Thread-Topic: what is the major difference between Hadoop and cloudMapReduce?
thread-index: Acpx2TCCt6fgMYqwSBaHD9D34V0K+ACZOx1A
Message-ID: <BD19120596A4D041940290E71BD0097E62E44ED33F@AMRXM3124.dir.svc.accenture.com>
References: <c28cbc140911281944j4925292chfc8f87ce1e4c3c28@mail.gmail.com> 	<c28cbc140911281955j162fcf10w3aade804544a0938@mail.gmail.com> 	<7b3355cb0911282036x394f0acft21da05803b16a01@mail.gmail.com> 	<32120a6a0911290213m5563e1fge8bcc4413909a505@mail.gmail.com> 	<7b3355cb0911290816h1dc160e4ua30054c438702215@mail.gmail.com> 	<45f85f70911291015m5165421fsf75f28bc698f28ed@mail.gmail.com> 	<BD19120596A4D041940290E71BD0097E62E4411DDC@AMRXM3124.dir.svc.accenture.com> <45f85f70911300815q2e86c64r92e28654af38840d@mail.gmail.com>
In-Reply-To: <45f85f70911300815q2e86c64r92e28654af38840d@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-cr-puzzleid: {4102CD52-8C8F-484B-8B0C-9AE980C13122}
x-cr-hashedpuzzle: BTyG B4EY CQkL CQsI C5d3 DJfi DX9a Em1C FUD9 FVOi F7y5 Ger/ Gew0 H/0F IKdM IvOJ;1;ZwBlAG4AZQByAGEAbABAAGgAYQBkAG8AbwBwAC4AYQBwAGEAYwBoAGUALgBvAHIAZwA=;Sosha1_v1;7;{4102CD52-8C8F-484B-8B0C-9AE980C13122};aAB1AGEAbgAuAGwAaQB1AEAAYQBjAGMAZQBuAHQAdQByAGUALgBjAG8AbQA=;Thu, 03 Dec 2009 17:39:41 GMT;UgBFADoAIAB3AGgAYQB0ACAAaQBzACAAdABoAGUAIABtAGEAagBvAHIAIABkAGkAZgBmAGUAcgBlAG4AYwBlACAAYgBlAHQAdwBlAGUAbgAgAEgAYQBkAG8AbwBwACAAYQBuAGQAIABjAGwAbwB1AGQATQBhAHAAUgBlAGQAdQBjAGUAPwA=
acceptlanguage: en-US
x-ems-proccessed: vrAiQuOOcsXVFhS7ec6D4A==
x-ems-stamp: 6Fmx78oxJnUgeanTthCC0Q==
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginalArrivalTime: 03 Dec 2009 17:40:21.0466 (UTC) FILETIME=[AFCC8BA0:01CA743F]

> -----Original Message-----
> From: Todd Lipcon [mailto:todd@cloudera.com]
> Sent: Monday, November 30, 2009 8:15 AM
> To: general@hadoop.apache.org
> Subject: Re: what is the major difference between Hadoop and
> cloudMapReduce?
>=20
> On Mon, Nov 30, 2009 at 1:48 AM, <huan.liu@accenture.com> wrote:
>=20
> > Todd,
> >
> > We do not keep all values for a key in memory. Instead, we only keep
> the
> > partial reduce result in memory, but throw away the value as soon as
> it is
> > used. The point you raised is still very valid if the reduce state
> > maintained per key is large, which I hope is a very rare case. If =
you
> have
> > some concrete workload examples, it will help us prioritize the
> development
> > effort. I can definitely see the benefits of introducing a paging
> mechanism
> > to spill partial reduce results to the output SQS queue in the =
future.
> > Thanks.
> >
>=20
> Hi Huan,
>=20
> I guess I misremembered or misread the paper.
>=20
> Given this technique, doesn't it mean that reducers can only work when
> commutative and associative?
>=20
> -Todd

Todd,=20

I do not see how it is different from Hadoop's iterator interface, =
unless the reduce function relies on the fact that the values are sorted =
in a particular order when fed by the iterator one at a time.=20

If there is no assumption on the value ordering, or the ordering =
expected is different from what the iterator presents, the reduce =
function has to read in all values from the iterator first (page to disk =
if necessary), rearrange them as necessary, then process based on that =
new ordering. This will be the same as what we will do in our iterator =
interface. In the next() function, our reduce function can read in all =
values from the iterator (page to disk if necessary), then in the =
finish() function, our reduce function rearranges the ordering and =
process based on the new ordering.=20

Apologies for the late reply. Traveling in china with no reliable =
network connection this week. Thanks.

-Huan


This message is for the designated recipient only and may contain =
privileged, proprietary, or otherwise private information.  If you have =
received it in error, please notify the sender immediately and delete =
the original.  Any other use of the email by you is prohibited.

From general-return-820-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 03 17:46:05 2009
Return-Path: <general-return-820-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97563 invoked from network); 3 Dec 2009 17:46:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Dec 2009 17:46:05 -0000
Received: (qmail 36430 invoked by uid 500); 3 Dec 2009 17:46:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36347 invoked by uid 500); 3 Dec 2009 17:46:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36337 invoked by uid 99); 3 Dec 2009 17:46:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Dec 2009 17:46:04 +0000
X-ASF-Spam-Status: No, hits=-2.5 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.85.222.204] (HELO mail-pz0-f204.google.com) (209.85.222.204)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Dec 2009 17:46:02 +0000
Received: by pzk42 with SMTP id 42so1623843pzk.31
        for <general@hadoop.apache.org>; Thu, 03 Dec 2009 09:45:40 -0800 (PST)
MIME-Version: 1.0
Received: by 10.142.55.20 with SMTP id d20mr209492wfa.322.1259862340165; Thu, 
	03 Dec 2009 09:45:40 -0800 (PST)
In-Reply-To: <BD19120596A4D041940290E71BD0097E62E44ED33F@AMRXM3124.dir.svc.accenture.com>
References: <c28cbc140911281944j4925292chfc8f87ce1e4c3c28@mail.gmail.com> 
	<c28cbc140911281955j162fcf10w3aade804544a0938@mail.gmail.com> 
	<7b3355cb0911282036x394f0acft21da05803b16a01@mail.gmail.com> 
	<32120a6a0911290213m5563e1fge8bcc4413909a505@mail.gmail.com> 
	<7b3355cb0911290816h1dc160e4ua30054c438702215@mail.gmail.com> 
	<45f85f70911291015m5165421fsf75f28bc698f28ed@mail.gmail.com> 
	<BD19120596A4D041940290E71BD0097E62E4411DDC@AMRXM3124.dir.svc.accenture.com> 
	<45f85f70911300815q2e86c64r92e28654af38840d@mail.gmail.com> 
	<BD19120596A4D041940290E71BD0097E62E44ED33F@AMRXM3124.dir.svc.accenture.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Thu, 3 Dec 2009 09:45:20 -0800
Message-ID: <45f85f70912030945r9613e95p26832cab954e7300@mail.gmail.com>
Subject: Re: what is the major difference between Hadoop and cloudMapReduce?
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0050450296254062460479d68fd4

--0050450296254062460479d68fd4
Content-Type: text/plain; charset=ISO-8859-1

On Thu, Dec 3, 2009 at 9:39 AM, <huan.liu@accenture.com> wrote:

> > -----Original Message-----
> > From: Todd Lipcon [mailto:todd@cloudera.com]
> > Sent: Monday, November 30, 2009 8:15 AM
> > To: general@hadoop.apache.org
> > Subject: Re: what is the major difference between Hadoop and
> > cloudMapReduce?
> >
> > On Mon, Nov 30, 2009 at 1:48 AM, <huan.liu@accenture.com> wrote:
> >
> > > Todd,
> > >
> > > We do not keep all values for a key in memory. Instead, we only keep
> > the
> > > partial reduce result in memory, but throw away the value as soon as
> > it is
> > > used. The point you raised is still very valid if the reduce state
> > > maintained per key is large, which I hope is a very rare case. If you
> > have
> > > some concrete workload examples, it will help us prioritize the
> > development
> > > effort. I can definitely see the benefits of introducing a paging
> > mechanism
> > > to spill partial reduce results to the output SQS queue in the future.
> > > Thanks.
> > >
> >
> > Hi Huan,
> >
> > I guess I misremembered or misread the paper.
> >
> > Given this technique, doesn't it mean that reducers can only work when
> > commutative and associative?
> >
> > -Todd
>
> Todd,
>
> I do not see how it is different from Hadoop's iterator interface, unless
> the reduce function relies on the fact that the values are sorted in a
> particular order when fed by the iterator one at a time.
>
> If there is no assumption on the value ordering, or the ordering expected
> is different from what the iterator presents, the reduce function has to
> read in all values from the iterator first (page to disk if necessary),
> rearrange them as necessary, then process based on that new ordering. This
> will be the same as what we will do in our iterator interface. In the next()
> function, our reduce function can read in all values from the iterator (page
> to disk if necessary), then in the finish() function, our reduce function
> rearranges the ordering and process based on the new ordering.
>
>
If you want sorted values, you can get that in Hadoop, though it's not on by
default.

Also, the reducer only needs to keep all the values for a single key in RAM
in this case. In your case, since the keys come in any order, the reducer
would have to keep all the values for every key in that partition in RAM. I
guess you're suggesting that you could have enough partitions that "every
key in that partition" is only one or two, but for large datasets this just
doesn't soudn feasible (can you get millions of SQS queues?)

Sure, you can implement your own paging to disk, and then an external sort,
and then read them back in the more convenient sorted order. But then you're
just implementing what Hadoop already does for you :)

-Todd


>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise private information.  If you have
> received it in error, please notify the sender immediately and delete the
> original.  Any other use of the email by you is prohibited.
>

--0050450296254062460479d68fd4--

From general-return-821-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 03 18:00:21 2009
Return-Path: <general-return-821-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6876 invoked from network); 3 Dec 2009 18:00:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Dec 2009 18:00:21 -0000
Received: (qmail 64054 invoked by uid 500); 3 Dec 2009 18:00:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63994 invoked by uid 500); 3 Dec 2009 18:00:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63984 invoked by uid 99); 3 Dec 2009 18:00:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Dec 2009 18:00:20 +0000
X-ASF-Spam-Status: No, hits=-6.6 required=5.0
	tests=AWL,BAYES_00,RCVD_IN_DNSWL_MED
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of huan.liu@accenture.com designates 170.252.248.70 as permitted sender)
Received: from [170.252.248.70] (HELO amrmr1001.accenture.com) (170.252.248.70)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Dec 2009 18:00:17 +0000
Received: from AMRXV1001.dir.svc.accenture.com (amrxv1001.dir.svc.accenture.com [10.10.160.61])
	by amrmr1001.accenture.com (8.13.8/8.13.8) with ESMTP id nB3HxdCv019254
	for <general@hadoop.apache.org>; Thu, 3 Dec 2009 11:59:56 -0600 (CST)
Received: from AMRXH3004.dir.svc.accenture.com ([10.63.34.26]) by AMRXV1001.dir.svc.accenture.com with Microsoft SMTPSVC(6.0.3790.1830); Thu, 3 Dec 2009 11:59:45 -0600
Received: from AMRXM3124.dir.svc.accenture.com ([10.63.34.14]) by AMRXH3004.dir.svc.accenture.com ([10.63.34.26]) with mapi; Thu, 3 Dec 2009 12:59:45 -0500
From: <huan.liu@accenture.com>
Content-Class: urn:content-classes:message
Importance: normal
Priority: normal
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.3790.3168
To: <general@hadoop.apache.org>
Date: Thu, 3 Dec 2009 12:59:09 -0500
Subject: RE: Introducing Cloud MapReduce
Thread-Topic: Introducing Cloud MapReduce
thread-index: AcpzIrYNIrY9CQ7MSD+m/y/SKs8qdABHSygQ
Message-ID: <BD19120596A4D041940290E71BD0097E62E44ED36E@AMRXM3124.dir.svc.accenture.com>
References: <7b3355cb0911270941s219c492w63d8ce6c83eeb96b@mail.gmail.com> <57989512-60C3-4AD7-8AD5-3A3F0DCD4F91@apache.org>
In-Reply-To: <57989512-60C3-4AD7-8AD5-3A3F0DCD4F91@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
x-ems-proccessed: vrAiQuOOcsXVFhS7ec6D4A==
x-ems-stamp: 8lW225mj7W/AjgdadZmXTg==
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginalArrivalTime: 03 Dec 2009 17:59:45.0801 (UTC) FILETIME=[65CBF790:01CA7442]

Owen,=20

Great questions. You are hitting some key points, answers are inline.

> > 1) It is faster than other implementations (e.g., 60 times faster
> than
> > Hadoop in one case. Speedup depends on the application and data.).
>=20
> Based on your report, it looks like your comparison was done using
> 1.2GB of data in 92,000 files. If you took the default configuration,
> you'll end up with 92,000 maps, each of them processing a very small
> input. You either need to use MultiFileInputFormat or use Hadoop
> Archives to bundle your tiny files into reasonable size. For input
> data that small you should not have more than 8 maps unless each map
> is doing a lot of CPU work.

We used default because we were not sure how to optimize it. We can =
certainly rerun the test with your suggestion. The point is not about =
the 60x speedup, it is about a potential bottleneck in the master/slave =
architecture. When you scale up the number of slaves nodes and the =
number of tasks, you will run into the same problem even if you use =
larger chunk size. Due to the lack of access to a large cluster, we are =
not able to run an experiment to show that the master node will choke at =
some point. This is essentially a scaled-down version of the same =
large-scale test.

> In your reduce push iterator model, you give up a lot of determinism
> in your application. Many applications that run in Hadoop depend on
> the reduces getting keys in sorted order. Getting keys in a random
> order won't work.=20

Sorting at the end after the values have been reduced is much cheaper =
than sorting the whole set of intermediate key-value pairs. We will add =
a function in our implementation to sort at the end, for those =
applications requiring ordered output.=20

> Furthermore, I don't see any way that you can scale
> up your approach. You effectively need to hold open all of the reduce
> states for the keys you've seen and can't close them until it is done.
> Again, that will quickly exhaust memory. So the push model will work
> for something like word count, but fail on large problems.

Our strategy is to have a large number of partitions to limit the number =
of keys in each partition. Having a large number of partitions would not =
overload our system, so it is easy for us. As I expressed in a related =
thread answering Todd's question, I believe the state each key holds is =
small for most applications. Loved to be proved wrong, so that we have =
real motivations to refine the architecture to handle that case.

> When you add types in, you should add them in as:
>=20
> K1, V1 -> map -> K2,V2 -> reduce -> K3,V3
>=20
> with the combiner taking:
>=20
> K2,V2 -> combiner -> K2,V2
>=20
> Of course the types are often not primitives and using something like
> Avro is a good thing.

Thanks for clarifying. Adding types is in the roadmap now.=20

Regards.

-Huan


This message is for the designated recipient only and may contain =
privileged, proprietary, or otherwise private information.  If you have =
received it in error, please notify the sender immediately and delete =
the original.  Any other use of the email by you is prohibited.

From general-return-822-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 03 19:29:28 2009
Return-Path: <general-return-822-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 86089 invoked from network); 3 Dec 2009 19:29:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Dec 2009 19:29:28 -0000
Received: (qmail 86793 invoked by uid 500); 3 Dec 2009 19:29:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86033 invoked by uid 500); 3 Dec 2009 19:29:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85640 invoked by uid 99); 3 Dec 2009 19:29:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Dec 2009 19:29:25 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of goutham.patnaik@gmail.com designates 209.85.217.228 as permitted sender)
Received: from [209.85.217.228] (HELO mail-gx0-f228.google.com) (209.85.217.228)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Dec 2009 19:29:23 +0000
Received: by gxk28 with SMTP id 28so1486227gxk.9
        for <general@hadoop.apache.org>; Thu, 03 Dec 2009 11:29:02 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :from:date:message-id:subject:to:content-type;
        bh=GaxEhyVsjZBHAkAM1HOIH66CcjsKkZujpNE1LwbpqAM=;
        b=g6x22rMcUufIVA9QmyIn6sC4XNg26uO8Q06gfPCj6ryBcvmDno4wQG5Hy754mYM/6q
         88D7/dIhvIJonjKfnidAWqaMgIOtx526tbTsqlj5Ek2PMoPpxl6oW8lMvvlhKHocOW/a
         UfXvpBBaHyQBg4aBhXpHIN1sOFWzB5w+KM5n4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=QWnZJi7YQsNzoJiCSLGUSoolgtMgcDg3GR3FnHHBk7Fl4tDbTtbJH55GlOEwyGG9vM
         nRp1nDpXHUuMevm7x4t3iRq+jk3AAI/IF6tQHVKYgI2IhH0jTgJgwRcsEoIM4uYfPqoZ
         xraFvHinxuxwCh/PsIwMRGzzMizTP9Z2gRMUg=
MIME-Version: 1.0
Received: by 10.151.29.20 with SMTP id g20mr3582522ybj.189.1259868542163; Thu, 
	03 Dec 2009 11:29:02 -0800 (PST)
In-Reply-To: <BD19120596A4D041940290E71BD0097E62E44ED36E@AMRXM3124.dir.svc.accenture.com>
References: <7b3355cb0911270941s219c492w63d8ce6c83eeb96b@mail.gmail.com> 
	<57989512-60C3-4AD7-8AD5-3A3F0DCD4F91@apache.org> <BD19120596A4D041940290E71BD0097E62E44ED36E@AMRXM3124.dir.svc.accenture.com>
From: goutham patnaik <goutham.patnaik@gmail.com>
Date: Thu, 3 Dec 2009 11:28:42 -0800
Message-ID: <1b3669280912031128i273a3ee5p6f4ccfd793f60e2a@mail.gmail.com>
Subject: Re: Introducing Cloud MapReduce
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd23bb8eb5c310479d80064

--000e0cd23bb8eb5c310479d80064
Content-Type: text/plain; charset=ISO-8859-1

see inline

On Thu, Dec 3, 2009 at 9:59 AM, <huan.liu@accenture.com> wrote:

> Owen,
>
> Great questions. You are hitting some key points, answers are inline.
>
> > > 1) It is faster than other implementations (e.g., 60 times faster
> > than
> > > Hadoop in one case. Speedup depends on the application and data.).
> >
> > Based on your report, it looks like your comparison was done using
> > 1.2GB of data in 92,000 files. If you took the default configuration,
> > you'll end up with 92,000 maps, each of them processing a very small
> > input. You either need to use MultiFileInputFormat or use Hadoop
> > Archives to bundle your tiny files into reasonable size. For input
> > data that small you should not have more than 8 maps unless each map
> > is doing a lot of CPU work.
>
> We used default because we were not sure how to optimize it. We can
> certainly rerun the test with your suggestion. The point is not about the
> 60x speedup, it is about a potential bottleneck in the master/slave
> architecture. When you scale up the number of slaves nodes and the number of
> tasks, you will run into the same problem even if you use larger chunk size.
> Due to the lack of access to a large cluster, we are not able to run an
> experiment to show that the master node will choke at some point. This is
> essentially a scaled-down version of the same large-scale test.
>
> > In your reduce push iterator model, you give up a lot of determinism
> > in your application. Many applications that run in Hadoop depend on
> > the reduces getting keys in sorted order. Getting keys in a random
> > order won't work.
>
> Sorting at the end after the values have been reduced is much cheaper than
> sorting the whole set of intermediate key-value pairs. We will add a
> function in our implementation to sort at the end, for those applications
> requiring ordered output.
>

Owen is right though - there are some applications that require the keys to
be sorted when they reach a reducer - secondary sorting is one such example

>
> > Furthermore, I don't see any way that you can scale
> > up your approach. You effectively need to hold open all of the reduce
> > states for the keys you've seen and can't close them until it is done.
> > Again, that will quickly exhaust memory. So the push model will work
> > for something like word count, but fail on large problems.
>
> Our strategy is to have a large number of partitions to limit the number of
> keys in each partition. Having a large number of partitions would not
> overload our system, so it is easy for us. As I expressed in a related
> thread answering Todd's question, I believe the state each key holds is
> small for most applications. Loved to be proved wrong, so that we have real
> motivations to refine the architecture to handle that case.
>
> > When you add types in, you should add them in as:
> >
> > K1, V1 -> map -> K2,V2 -> reduce -> K3,V3
> >
> > with the combiner taking:
> >
> > K2,V2 -> combiner -> K2,V2
> >
> > Of course the types are often not primitives and using something like
> > Avro is a good thing.
>
> Thanks for clarifying. Adding types is in the roadmap now.
>
> Regards.
>
> -Huan
>
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise private information.  If you have
> received it in error, please notify the sender immediately and delete the
> original.  Any other use of the email by you is prohibited.
>

--000e0cd23bb8eb5c310479d80064--

From general-return-823-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Dec 04 01:21:45 2009
Return-Path: <general-return-823-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 28897 invoked from network); 4 Dec 2009 01:21:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Dec 2009 01:21:44 -0000
Received: (qmail 87883 invoked by uid 500); 4 Dec 2009 01:21:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87704 invoked by uid 500); 4 Dec 2009 01:21:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87694 invoked by uid 99); 4 Dec 2009 01:21:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Dec 2009 01:21:42 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of huan.liu@accenture.com designates 170.252.248.72 as permitted sender)
Received: from [170.252.248.72] (HELO amrmr1003.accenture.com) (170.252.248.72)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Dec 2009 01:21:32 +0000
Received: from AMRXV1003.dir.svc.accenture.com (amrxv1003.dir.svc.accenture.com [10.10.160.63])
	by amrmr1003.accenture.com (8.13.8/8.13.8) with ESMTP id nB41L9dX027356
	for <general@hadoop.apache.org>; Thu, 3 Dec 2009 19:21:10 -0600 (CST)
Received: from AMRXH3001.dir.svc.accenture.com ([10.63.34.23]) by AMRXV1003.dir.svc.accenture.com with Microsoft SMTPSVC(6.0.3790.1830); Thu, 3 Dec 2009 19:20:33 -0600
Received: from AMRXM3124.dir.svc.accenture.com ([10.63.34.14]) by AMRXH3001.dir.svc.accenture.com ([10.63.34.23]) with mapi; Thu, 3 Dec 2009 20:20:33 -0500
From: <huan.liu@accenture.com>
To: <general@hadoop.apache.org>
Content-Class: urn:content-classes:message
Importance: normal
Priority: normal
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.3790.3168
Date: Thu, 3 Dec 2009 20:19:54 -0500
Subject: RE: what is the major difference between Hadoop and cloudMapReduce?
Thread-Topic: what is the major difference between Hadoop and cloudMapReduce?
thread-index: Acp0QIQp2KJG00HnR3q82VQWCh9lYAAPmlMA
Message-ID: <BD19120596A4D041940290E71BD0097E62E454D001@AMRXM3124.dir.svc.accenture.com>
References: <c28cbc140911281944j4925292chfc8f87ce1e4c3c28@mail.gmail.com> 	<c28cbc140911281955j162fcf10w3aade804544a0938@mail.gmail.com> 	<7b3355cb0911282036x394f0acft21da05803b16a01@mail.gmail.com> 	<32120a6a0911290213m5563e1fge8bcc4413909a505@mail.gmail.com> 	<7b3355cb0911290816h1dc160e4ua30054c438702215@mail.gmail.com> 	<45f85f70911291015m5165421fsf75f28bc698f28ed@mail.gmail.com> 	<BD19120596A4D041940290E71BD0097E62E4411DDC@AMRXM3124.dir.svc.accenture.com> 	<45f85f70911300815q2e86c64r92e28654af38840d@mail.gmail.com> 	<BD19120596A4D041940290E71BD0097E62E44ED33F@AMRXM3124.dir.svc.accenture.com> <45f85f70912030945r9613e95p26832cab954e7300@mail.gmail.com>
In-Reply-To: <45f85f70912030945r9613e95p26832cab954e7300@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-cr-puzzleid: {53011AAE-0023-42F9-BD79-04951CEB315C}
x-cr-hashedpuzzle: BW+d Bibm BzWt CPDH Chr5 FTsR FUU2 FVBu FViZ Fvei IDiP I/Ol JDmw Jzjb KjFq KsPH;1;ZwBlAG4AZQByAGEAbABAAGgAYQBkAG8AbwBwAC4AYQBwAGEAYwBoAGUALgBvAHIAZwA=;Sosha1_v1;7;{53011AAE-0023-42F9-BD79-04951CEB315C};aAB1AGEAbgAuAGwAaQB1AEAAYQBjAGMAZQBuAHQAdQByAGUALgBjAG8AbQA=;Fri, 04 Dec 2009 01:19:54 GMT;UgBFADoAIAB3AGgAYQB0ACAAaQBzACAAdABoAGUAIABtAGEAagBvAHIAIABkAGkAZgBmAGUAcgBlAG4AYwBlACAAYgBlAHQAdwBlAGUAbgAgAEgAYQBkAG8AbwBwACAAYQBuAGQAIABjAGwAbwB1AGQATQBhAHAAUgBlAGQAdQBjAGUAPwA=
acceptlanguage: en-US
x-ems-proccessed: vrAiQuOOcsXVFhS7ec6D4A==
x-ems-stamp: TVunI3HmpiPt4bm1WpVhxA==
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginalArrivalTime: 04 Dec 2009 01:20:33.0300 (UTC) FILETIME=[F9BC0D40:01CA747F]
X-Virus-Checked: Checked by ClamAV on apache.org

Todd,

Yes, the difference is that, if special ordering is required, we hold a =
few keys' worth of values in memory v.s. Hadoop holding one key's worth =
of values.=20

SQS has no limit on the number of queues you can create and the number =
of messages you can put in a queue. In practice, we have run an =
experiment in which we created 4,000 queues with 100 threads, and it =
finished successfully in a few seconds. Have not tried to stress test to =
millions of queues though. But even if the test fails with millions of =
queues, it will be a bug with SQS to be fixed since Amazon advertises =
that there is no limit. Thanks.

-Huan

> -----Original Message-----
> From: Todd Lipcon [mailto:todd@cloudera.com]
> Sent: Thursday, December 03, 2009 9:45 AM
> To: general@hadoop.apache.org
> Subject: Re: what is the major difference between Hadoop and
> cloudMapReduce?
>=20
> On Thu, Dec 3, 2009 at 9:39 AM, <huan.liu@accenture.com> wrote:
>=20
> > > -----Original Message-----
> > > From: Todd Lipcon [mailto:todd@cloudera.com]
> > > Sent: Monday, November 30, 2009 8:15 AM
> > > To: general@hadoop.apache.org
> > > Subject: Re: what is the major difference between Hadoop and
> > > cloudMapReduce?
> > >
> > > On Mon, Nov 30, 2009 at 1:48 AM, <huan.liu@accenture.com> wrote:
> > >
> > > > Todd,
> > > >
> > > > We do not keep all values for a key in memory. Instead, we only
> keep
> > > the
> > > > partial reduce result in memory, but throw away the value as =
soon
> as
> > > it is
> > > > used. The point you raised is still very valid if the reduce
> state
> > > > maintained per key is large, which I hope is a very rare case. =
If
> you
> > > have
> > > > some concrete workload examples, it will help us prioritize the
> > > development
> > > > effort. I can definitely see the benefits of introducing a =
paging
> > > mechanism
> > > > to spill partial reduce results to the output SQS queue in the
> future.
> > > > Thanks.
> > > >
> > >
> > > Hi Huan,
> > >
> > > I guess I misremembered or misread the paper.
> > >
> > > Given this technique, doesn't it mean that reducers can only work
> when
> > > commutative and associative?
> > >
> > > -Todd
> >
> > Todd,
> >
> > I do not see how it is different from Hadoop's iterator interface,
> unless
> > the reduce function relies on the fact that the values are sorted in
> a
> > particular order when fed by the iterator one at a time.
> >
> > If there is no assumption on the value ordering, or the ordering
> expected
> > is different from what the iterator presents, the reduce function =
has
> to
> > read in all values from the iterator first (page to disk if
> necessary),
> > rearrange them as necessary, then process based on that new =
ordering.
> This
> > will be the same as what we will do in our iterator interface. In =
the
> next()
> > function, our reduce function can read in all values from the
> iterator (page
> > to disk if necessary), then in the finish() function, our reduce
> function
> > rearranges the ordering and process based on the new ordering.
> >
> >
> If you want sorted values, you can get that in Hadoop, though it's not
> on by
> default.
>=20
> Also, the reducer only needs to keep all the values for a single key =
in
> RAM
> in this case. In your case, since the keys come in any order, the
> reducer
> would have to keep all the values for every key in that partition in
> RAM. I
> guess you're suggesting that you could have enough partitions that
> "every
> key in that partition" is only one or two, but for large datasets this
> just
> doesn't soudn feasible (can you get millions of SQS queues?)
>=20
> Sure, you can implement your own paging to disk, and then an external
> sort,
> and then read them back in the more convenient sorted order. But then
> you're
> just implementing what Hadoop already does for you :)
>=20
> -Todd
>=20
>=20
> >
> > This message is for the designated recipient only and may contain
> > privileged, proprietary, or otherwise private information.  If you
> have
> > received it in error, please notify the sender immediately and =
delete
> the
> > original.  Any other use of the email by you is prohibited.
> >


This message is for the designated recipient only and may contain =
privileged, proprietary, or otherwise private information.  If you have =
received it in error, please notify the sender immediately and delete =
the original.  Any other use of the email by you is prohibited.

From general-return-824-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Dec 04 22:10:37 2009
Return-Path: <general-return-824-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 75590 invoked from network); 4 Dec 2009 22:10:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Dec 2009 22:10:37 -0000
Received: (qmail 23937 invoked by uid 500); 4 Dec 2009 22:10:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23728 invoked by uid 500); 4 Dec 2009 22:10:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23700 invoked by uid 99); 4 Dec 2009 22:10:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Dec 2009 22:10:34 +0000
X-ASF-Spam-Status: No, hits=-2.5 required=5.0
	tests=AWL,BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Dec 2009 22:10:33 +0000
Received: from [10.73.152.131] (snvvpn2-10-73-152-c131.hq.corp.yahoo.com [10.73.152.131])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id nB4M9tgS059470;
	Fri, 4 Dec 2009 14:09:55 -0800 (PST)
Message-ID: <4B1988B3.1050407@apache.org>
Date: Fri, 04 Dec 2009 14:09:55 -0800
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org,
        "zookeeper-dev@hadoop.apache.org" <zookeeper-dev@hadoop.apache.org>,
        Henry Robinson <henry@apache.org>
Subject: Welcome Henry as ZooKeeper committer
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

The ZooKeeper team and the Hadoop PMC would like to welcome Henry
Robinson as the newest committer on ZooKeeper! Henry, please add
yourself to the ZooKeeper website's list of committers.

Thanks for your hard work on behalf of ZooKeeper, Hadoop and the ASF,

Patrick


From general-return-825-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Dec 06 11:33:14 2009
Return-Path: <general-return-825-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53947 invoked from network); 6 Dec 2009 11:33:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Dec 2009 11:33:14 -0000
Received: (qmail 68372 invoked by uid 500); 6 Dec 2009 11:33:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68264 invoked by uid 500); 6 Dec 2009 11:33:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68254 invoked by uid 99); 6 Dec 2009 11:33:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Dec 2009 11:33:11 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yangxiao9901@gmail.com designates 209.85.160.50 as permitted sender)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Dec 2009 11:33:03 +0000
Received: by pwi20 with SMTP id 20so652484pwi.29
        for <general@hadoop.apache.org>; Sun, 06 Dec 2009 03:32:42 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=sJwMK7R82CzRVWL/PKrldmJuVbmNJJUf7jGeo6oF68k=;
        b=tWxiiZqrXbVAnsfVli1v7+34ilBt1nvfLl2KeZm4h7LW5NK2bky9Pvx55ddA5pGe2p
         jkeaCuIFsuthBqFSqhcZqaQGXmjGV3mXZzEYm6HxxA6k8PK+22iBjZAqyr8Aw7mcb33S
         ORFCTNj32V9K/NR4kP5DzJyRimxxj8SsccTeo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=yDuVCm0JpNva4lLjmy+8ObDBZwgstmK/TMXKJyk47ieuvs/bvfeDOHFDAz9Ay+p/Wh
         uNr/CK+tgujMH+1cbeBE1ILYgMdS+Ir0ibboe4aELWaLiiK3j/onNJ1lG3iypqwvZrwn
         aFjTAUqmSYLoMUc63yRUvdTi5NpgOQaf4+Geo=
MIME-Version: 1.0
Received: by 10.140.164.1 with SMTP id m1mr309636rve.134.1260099162425; Sun, 
	06 Dec 2009 03:32:42 -0800 (PST)
Date: Sun, 6 Dec 2009 19:32:42 +0800
Message-ID: <5a921af20912060332k9ddacf1u48d14d0449a68444@mail.gmail.com>
Subject: How to pause a job?
From: xiao yang <yangxiao9901@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hi, all

I'm running a job on a 10-nodes cluster.
Now I want to add another node.
I have to reconfigure the dfs, and restart it, but I don't want to
stop the running job. It run for 1 week already.
What should I do? Is there a way to pause a job, and resume it after
dfs restart.

Thanks!
Xiao

From general-return-826-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Dec 07 03:37:56 2009
Return-Path: <general-return-826-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6685 invoked from network); 7 Dec 2009 03:37:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Dec 2009 03:37:56 -0000
Received: (qmail 25131 invoked by uid 500); 7 Dec 2009 03:37:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24948 invoked by uid 500); 7 Dec 2009 03:37:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24938 invoked by uid 99); 7 Dec 2009 03:37:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Dec 2009 03:37:53 +0000
X-ASF-Spam-Status: No, hits=4.2 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Dec 2009 03:37:41 +0000
Received: from sp1-ex07cas03.ds.corp.yahoo.com (sp1-ex07cas03.ds.corp.yahoo.com [216.252.116.151])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id nB73b5pv022358
	for <general@hadoop.apache.org>; Sun, 6 Dec 2009 19:37:05 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:in-reply-to:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=tjjyzcNoPwFUMd1tsUS1WpKjJ7N3vrmkVONvwgdC7macGb203KxUYW9o3Rr/jJrj
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas03.ds.corp.yahoo.com ([216.252.116.151]) with mapi; Sun, 6 Dec
 2009 19:37:04 -0800
From: Rekha Joshi <rekhajos@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Sun, 6 Dec 2009 19:36:54 -0800
Subject: Re: How to pause a job?
Thread-Topic: How to pause a job?
Thread-Index: Acp2Z+nE1DecgjAsTBaVOz06zpmPQQAhptKt
Message-ID: <C742762E.52E1%rekhajos@yahoo-inc.com>
In-Reply-To: <5a921af20912060332k9ddacf1u48d14d0449a68444@mail.gmail.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C742762E52E1rekhajosyahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C742762E52E1rekhajosyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


If you have workflow engine around your job, that can take care of restarti=
ng the jobs. For direct hadoop execution,  please refer MAPREDUCE-227/ MAPR=
EDUCE-828

Not sure if it would work, you might try (effectively) blocking this job ru=
n by -set-priority on this job to very low and have other VHP jobs running.=
. Thanks!

On 12/6/09 5:02 PM, "xiao yang" <yangxiao9901@gmail.com> wrote:

Hi, all

I'm running a job on a 10-nodes cluster.
Now I want to add another node.
I have to reconfigure the dfs, and restart it, but I don't want to
stop the running job. It run for 1 week already.
What should I do? Is there a way to pause a job, and resume it after
dfs restart.

Thanks!
Xiao


--_000_C742762E52E1rekhajosyahooinccom_--

From general-return-827-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 08 05:53:05 2009
Return-Path: <general-return-827-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5880 invoked from network); 8 Dec 2009 05:53:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Dec 2009 05:53:05 -0000
Received: (qmail 69799 invoked by uid 500); 8 Dec 2009 05:53:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69722 invoked by uid 500); 8 Dec 2009 05:53:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69709 invoked by uid 99); 8 Dec 2009 05:53:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Dec 2009 05:53:02 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [206.190.38.58] (HELO web50304.mail.re2.yahoo.com) (206.190.38.58)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 08 Dec 2009 05:52:52 +0000
Received: (qmail 18729 invoked by uid 60001); 8 Dec 2009 05:52:31 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1260251551; bh=lhqgTuIaKshn89fDQEll2T3TdTafsRSJJOHI8NJltCQ=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=cWSERQi3VAW22zD2jQPicXJMvrZi0L0tVX+EESIW7xvqAktOUBsH71lZ4KjQIs7XzBAgBdK7gFv0fLpoSQYSaCS/Ngeb4iGLpQSXsBxviEhNN2lwyb9pxCud26l2FplWlOArzEDpLHqUO1TGcGPRDsrRzDi5nXI+YVN7W3aSsU8=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=OMUu9C5tMag8dvtvn7vbjfaNRaAE5Bx5MxJqdaHXkt3ejoYm7EguyrWpd0B4TvDG03MBnz2JxoupcfwyGo+qTLcwAoFdxEdVylvzJejUn2zITS9EAda6aJBJbD/ttjQ7+VzLHdncJxvjzMboFx3uu3gghnmcHVwHpMf5r578CB8=;
Message-ID: <544693.18554.qm@web50304.mail.re2.yahoo.com>
X-YMail-OSG: 7tJaE8UVM1lRik8n6_8cR8EeGzCjUrQDy3UeWXBPQ8cUe2E3o.wKsxMvP80X9sP_xsdJ16uoOLi6t336lbA2mQDPi9C4vl_WcDlyljO60w3vGfi4_rqfuAnxqF4SYRdaFY5JVGV1mXf4beR8gJywuvV2Z00vSeqZn1gO9He3a1.7hWBf3cLbG5vtnBelwuOLHXiz5kvnwHNJrCHbfjY9dyAn1yr1IZHhRzVEkVTKLVzVO3RKLyF58N8lA.xa1ji_jAjj4_NFNSuEpsJdC8zBvVsIi4q.HEIdCSKV1VX5esdASev79Jt2i8eNP5QP1Ii4iNg3oE_JYdSr1c8auNRes2r081Lvx8EJ01oHmvSwQsRMeAu53zUvi_svunhxnnSSN2.5CCA2rh3cDKX7OH6YkAFf5lC22FNkbXDS7VqHequU4_PpuidMZDqRkCcGJLCjNdDSMDgg
Received: from [74.73.1.126] by web50304.mail.re2.yahoo.com via HTTP; Mon, 07 Dec 2009 21:52:31 PST
X-Mailer: YahooMailRC/240.3 YahooMailWebService/0.8.100.260964
Date: Mon, 7 Dec 2009 21:52:31 -0800 (PST)
From: Otis Gospodnetic <otis_gospodnetic@yahoo.com>
Subject: Subproject for core-user/-dev ?
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org

Hello,

I was hoping somebody could help me understand where core-user/-dev lists belong in Hadoop-land.

Each Hadoop subproject has its own -user and -dev list.  There is no "Core" subproject listed on hadoop.apache.org, yet we have core-user and core-dev.

Moreover, there is a problem with core-user/-dev archives over on:
 http://mail-archives.apache.org/mod_mbox/ .

Short version: core-user/-dev are not being archived.
I described this problem the other day:
  https://issues.apache.org/jira/browse/INFRA-2363

Thanks,
Otis
--
Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch


From general-return-828-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 08 06:22:14 2009
Return-Path: <general-return-828-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53894 invoked from network); 8 Dec 2009 06:22:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Dec 2009 06:22:14 -0000
Received: (qmail 97686 invoked by uid 500); 8 Dec 2009 06:22:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97588 invoked by uid 500); 8 Dec 2009 06:22:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97578 invoked by uid 99); 8 Dec 2009 06:22:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Dec 2009 06:22:13 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yangxiao9901@gmail.com designates 209.85.216.199 as permitted sender)
Received: from [209.85.216.199] (HELO mail-px0-f199.google.com) (209.85.216.199)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Dec 2009 06:22:05 +0000
Received: by pxi37 with SMTP id 37so1784835pxi.20
        for <general@hadoop.apache.org>; Mon, 07 Dec 2009 22:21:44 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=DDGvQd9mBPJN8hILl7qrgHI6fS2GKNeqN8CGxK7CkDc=;
        b=DJlSWyfNqdo6tZGrjfjaw/9XWfdQCDry+Y/fmT1C/CWGhTd0tU5d/TKNjEeXnqjJyj
         uGJLbRQDTVB4/JoSKj1mPt+U6e29sPj/8fgobl2eJqn4UajOtddN/7oKRukEiRvdHJHY
         GCcrA7xtQDcinVkL7qx+H53OO2m2oOwZJ77U8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=wUZeOUzUmflNcq5RM9Tew1UIfcqxPryrMYVS5pRzSgtNYfdg94d5saikvZU1nBMzp4
         hisq3MMztCfU9qpoEQhvhfUO87hZQJoK6wjdnnvk4AlgPYn2Yh0XRilh2fXJryeG6WGW
         +UoUlwMPxKUISX6HVryQA+83JapqmMe6wH2js=
MIME-Version: 1.0
Received: by 10.141.35.18 with SMTP id n18mr448904rvj.245.1260253304523; Mon, 
	07 Dec 2009 22:21:44 -0800 (PST)
In-Reply-To: <C742762E.52E1%rekhajos@yahoo-inc.com>
References: <5a921af20912060332k9ddacf1u48d14d0449a68444@mail.gmail.com>
	 <C742762E.52E1%rekhajos@yahoo-inc.com>
Date: Tue, 8 Dec 2009 14:21:44 +0800
Message-ID: <5a921af20912072221g75207fd8yc3086763ecbfb87a@mail.gmail.com>
Subject: Re: How to pause a job?
From: xiao yang <yangxiao9901@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks for reply. The job finally failed. It seems that I should make
sure the job won't take too long next time.

On 12/7/09, Rekha Joshi <rekhajos@yahoo-inc.com> wrote:
>
> If you have workflow engine around your job, that can take care of
> restarting the jobs. For direct hadoop execution,  please refer
> MAPREDUCE-227/ MAPREDUCE-828
>
> Not sure if it would work, you might try (effectively) blocking this job run
> by -set-priority on this job to very low and have other VHP jobs running..
> Thanks!
>
> On 12/6/09 5:02 PM, "xiao yang" <yangxiao9901@gmail.com> wrote:
>
> Hi, all
>
> I'm running a job on a 10-nodes cluster.
> Now I want to add another node.
> I have to reconfigure the dfs, and restart it, but I don't want to
> stop the running job. It run for 1 week already.
> What should I do? Is there a way to pause a job, and resume it after
> dfs restart.
>
> Thanks!
> Xiao
>
>

From general-return-829-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 08 07:06:04 2009
Return-Path: <general-return-829-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 74420 invoked from network); 8 Dec 2009 07:06:04 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Dec 2009 07:06:04 -0000
Received: (qmail 52174 invoked by uid 500); 8 Dec 2009 07:06:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52092 invoked by uid 500); 8 Dec 2009 07:06:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52082 invoked by uid 99); 8 Dec 2009 07:06:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Dec 2009 07:06:03 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of owen.omalley@gmail.com designates 209.85.216.199 as permitted sender)
Received: from [209.85.216.199] (HELO mail-px0-f199.google.com) (209.85.216.199)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Dec 2009 07:06:01 +0000
Received: by pxi37 with SMTP id 37so1811091pxi.20
        for <general@hadoop.apache.org>; Mon, 07 Dec 2009 23:05:41 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=wbm2KFAMa876OnZKJhF29FIxExTbrvT3x3mqwDJpgB0=;
        b=aaaaMNGA68KyelGmdalTe/xf7etK+ZEfFSPBCPD/eN/Lku3F+TPJrbxhw/Cbs7bnG1
         pVae2WgFerQTR+lwzPwNTLul6yH9+oZvlebjddCdUuA1jhAWGWYP3D0/+3ihbv3rYDnI
         9TAiiq3pWfS58yuyGV/2LuS3VE0kHDFwfKuWg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=GHqqVuwQfL4cN0UUYobQxz0hO1tiwfB0IIAWAbHxNoJ5R4NkJTaNtCB0Sm24Q1w/d2
         qqnU57HyDrlb00gFc00jwDT1MqNv7EWOj6bZeZjPetrZiYGz1WKwJDGb813G8bdS+L1Y
         dcZ3IPRi09sw2Dd+1fovRpmhiSSzYs8COahXI=
MIME-Version: 1.0
Received: by 10.140.128.9 with SMTP id a9mr470884rvd.146.1260255940725; Mon, 
	07 Dec 2009 23:05:40 -0800 (PST)
In-Reply-To: <544693.18554.qm@web50304.mail.re2.yahoo.com>
References: <544693.18554.qm@web50304.mail.re2.yahoo.com>
Date: Mon, 7 Dec 2009 23:05:40 -0800
Message-ID: <5f7f740b0912072305kca01557q63a7ceced6e60821@mail.gmail.com>
Subject: Re: Subproject for core-user/-dev ?
From: "Owen O'Malley" <owen.omalley@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd29600ac4a9a047a3233c0

--000e0cd29600ac4a9a047a3233c0
Content-Type: text/plain; charset=UTF-8

Core became Common when we moved out HDFS and MapReduce. So core-dev ==
common-dev, core-user == common-user, and so on.

 -- Owen

--000e0cd29600ac4a9a047a3233c0--

From general-return-830-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 08 07:48:49 2009
Return-Path: <general-return-830-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81853 invoked from network); 8 Dec 2009 07:48:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Dec 2009 07:48:49 -0000
Received: (qmail 83610 invoked by uid 500); 8 Dec 2009 07:48:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83524 invoked by uid 500); 8 Dec 2009 07:48:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83514 invoked by uid 99); 8 Dec 2009 07:48:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Dec 2009 07:48:48 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of chillycreator@gmail.com designates 209.85.160.50 as permitted sender)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Dec 2009 07:48:40 +0000
Received: by pwi20 with SMTP id 20so1789321pwi.29
        for <general@hadoop.apache.org>; Mon, 07 Dec 2009 23:48:19 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=jU79l1HztZtRyaNNle4rMVGoXq5A3vRiSHgvKHcmpg0=;
        b=kQ+IdwVDTae/+jF4vbr/HXlwfeHWXOnIQ4+bBKHtrTrjaLe29BYQhMXhkKZqf32Z8R
         LxLl8xteRSNit2y9X375C6rAgNWuDtbuRhtLf7v/lYvQr7Wqgr/ijr8bA2ReAoZ9YMn5
         8b8OCzGsHDj41kezfJwhdMxrWKr7IR9cgFxzg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=JR99Nq0vrlK+6wieCxKpmFLfuWqfCY4B75fdqT1GNQ3tZRPH+eP73toSfW2BCjaqV8
         /6n1htE9pDiIawyL6EBZE9zExUtg6MLHKp+8PvxqwFMbMFBMBK33RLPgbimc2xAlbDaD
         JHaMxi9/n9dY/brWNY6SfHOtXTh/ji2WE62GE=
MIME-Version: 1.0
Received: by 10.142.62.35 with SMTP id k35mr862099wfa.129.1260258499214; Mon, 
	07 Dec 2009 23:48:19 -0800 (PST)
Date: Tue, 8 Dec 2009 15:48:19 +0800
Message-ID: <50e3be80912072348s1ee86294h83425bd64fea5b31@mail.gmail.com>
Subject: Hbase & BigTable & RDBMS
From: chico CHEN <chillycreator@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636e0a9be2bc039047a32ccb4
X-Virus-Checked: Checked by ClamAV on apache.org

--001636e0a9be2bc039047a32ccb4
Content-Type: text/plain; charset=ISO-8859-1

Hi guys,
   I am a newer here. Recently, I am thinking about the data model of
Bigtable-like and that of  RDBMS. I feel confused about the difference of
the two. Take the blog-comment relation as the example. If we use RDBMS,
there will be three tables: one is blogs; one is comments; and the last one
is blog-comments relational table. If we use HBase or Bigtable, there will
be one table like:
blogID   timer   blog:content     blog:comments. If our tables in RDBMS is
denormal, we can put comments into blogs and combine them as one table.
  Is there something wrong in my example or my thoughts? Or else, where are
the differences in the two data models? And why did somebody say "RDBMS is
not suitable for cloud computing"?
  Thank you for taking some minutes to answer my questions.
-chico CHEN

--001636e0a9be2bc039047a32ccb4--

From general-return-831-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 08 18:30:20 2009
Return-Path: <general-return-831-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47789 invoked from network); 8 Dec 2009 18:30:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Dec 2009 18:30:20 -0000
Received: (qmail 34792 invoked by uid 500); 8 Dec 2009 18:30:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34668 invoked by uid 500); 8 Dec 2009 18:30:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34658 invoked by uid 99); 8 Dec 2009 18:30:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Dec 2009 18:30:17 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [206.190.38.61] (HELO web50307.mail.re2.yahoo.com) (206.190.38.61)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 08 Dec 2009 18:30:07 +0000
Received: (qmail 80536 invoked by uid 60001); 8 Dec 2009 18:29:45 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1260296985; bh=7hHoAK339WKi+EuTvwz6BjUfijMoguBWwJTkYD/g/tQ=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=rtryGvagUSv/g55wnpQ9MgxYWDrC2nbyDQJ7lUsNsi2d1jmhOLpRCW5qOcClmsElnerHndDMqlD/DjnKNF1p+nnmcr2vE49e8lAc2joVf0fSeNeHhkpslnHir75F9QPZGEoZ13ZMdzdOngNpEs0AD1uTnykYJT+nq6jwsbOdedY=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=P6G8bzYFWYIppeM+S2FLxEj4UwFs0i8GxQMSN9ZY5AOW7a+oMClGA2UV+uEJEoFq0z5ziS/I2Rko4lHo64OEXFPQzDu/4Fur7INPQ7ixyGAC5WqLh4XteD68YSjQjPh58p67kejzlWjgNETw5eWoaoI7e4eKk/dXt1siP0CkAhY=;
Message-ID: <906133.80299.qm@web50307.mail.re2.yahoo.com>
X-YMail-OSG: a2tcveIVM1lr_EmNfeSMwUPLz4g6DmEuduZlkyVmTIxTfzAwkT6JnzeAYelXyu5BrA8DemzDcdCn2ugTw9fNM9lPheMd1L_u7HgbuH_YJcTzYeHxmDRSr_lzFel7N4r_Bqq59SzK4QaMY7Xmhk4uAZstKZxDQAiZkHcjst5nrP69b662iA8tziWtJC46lwh8gg._tx.gys5VWD7mrsS2mz2g_jK9sts4vrYRRyqb6sucW9nESy_qafHi..RXsxueTreSNrvmR0WWDtnfkQFKCNgoT.mq3xlLd64znfu5vNosjSibhzags0TZnE.qOEP8LXkmq.Cc3OZqyt3ZBDvp3KjthPOiGptN1L_6tAC8yzNrwFGuoEuCuqgKufRJn6r4st9hJfnsyo_zr4l.FIpLvWEKlCdnSnubQSG7k5j5yL_eBWC8eBA3
Received: from [167.206.188.3] by web50307.mail.re2.yahoo.com via HTTP; Tue, 08 Dec 2009 10:29:45 PST
X-Mailer: YahooMailRC/240.3 YahooMailWebService/0.8.100.260964
References: <544693.18554.qm@web50304.mail.re2.yahoo.com> <5f7f740b0912072305kca01557q63a7ceced6e60821@mail.gmail.com>
Date: Tue, 8 Dec 2009 10:29:45 -0800 (PST)
From: Otis Gospodnetic <otis_gospodnetic@yahoo.com>
Subject: Re: Subproject for core-user/-dev ?
To: general@hadoop.apache.org
In-Reply-To: <5f7f740b0912072305kca01557q63a7ceced6e60821@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org

Aha, I see:
  http://mail-archives.apache.org/mod_mbox/hadoop-core-user/
Became:
  http://mail-archives.apache.org/mod_mbox/hadoop-common-user/

 
So https://issues.apache.org/jira/browse/INFRA-2363 could really  be closed as "Won't Fix", ha?

Otis
--
Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch



----- Original Message ----
> From: Owen O'Malley <owen.omalley@gmail.com>
> To: general@hadoop.apache.org
> Sent: Tue, December 8, 2009 2:05:40 AM
> Subject: Re: Subproject for core-user/-dev ?
> 
> Core became Common when we moved out HDFS and MapReduce. So core-dev ==
> common-dev, core-user == common-user, and so on.
> 
> -- Owen


From general-return-832-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Dec 11 15:32:12 2009
Return-Path: <general-return-832-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34382 invoked from network); 11 Dec 2009 15:32:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Dec 2009 15:32:11 -0000
Received: (qmail 70127 invoked by uid 500); 11 Dec 2009 15:32:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69090 invoked by uid 500); 11 Dec 2009 15:32:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69075 invoked by uid 99); 11 Dec 2009 15:32:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Dec 2009 15:32:07 +0000
X-ASF-Spam-Status: No, hits=-9.5 required=5.0
	tests=AWL,BAYES_00,RCVD_IN_DNSWL_HI
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 11 Dec 2009 15:32:04 +0000
Received: (qmail 34051 invoked by uid 99); 11 Dec 2009 15:31:44 -0000
Received: from localhost.apache.org (HELO mail-ew0-f212.google.com) (127.0.0.1)
  (smtp-auth username edwardyoon, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Dec 2009 15:31:44 +0000
Received: by ewy4 with SMTP id 4so1211345ewy.12
        for <multiple recipients>; Fri, 11 Dec 2009 07:31:42 -0800 (PST)
MIME-Version: 1.0
Received: by 10.216.88.195 with SMTP id a45mr600348wef.63.1260545502693; Fri, 
	11 Dec 2009 07:31:42 -0800 (PST)
Date: Sat, 12 Dec 2009 00:31:42 +0900
Message-ID: <eb4706e0912110731j4f5f7ac7la40d9a9baf93ed76@mail.gmail.com>
Subject: Announcement of the BSP (Bulk Synchronous Parallel) package on Hadoop
From: "Edward J. Yoon" <edwardyoon@apache.org>
To: hama-user@incubator.apache.org, general@hadoop.apache.org, 
	common-user@hadoop.apache.org, mahout-user@lucene.apache.org, 
	mapreduce-user@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8

Hello communities,

Now I'm happy to announce that we've developed new computing model
called BSP (Bulk Synchronous Parallel) on top of Hadoop.

Here are the slides for the topic "Apache HAMA: An Introduction to
Bulk Synchronization Parallel on Hadoop":

Download a PDF file:
http://wiki.apache.org/hama/BulkSynchronizationParallel?action=AttachFile&do=get&target=Apache_HAMA_BSP.pdf
or
Google Docs: http://docs.google.com/present/view?id=dc8qtchr_1f44rjccd

Any advices and questions are welcome.

Thanks,
Hama Team
-- 
Best Regards, Edward J. Yoon @ NHN, corp.
edwardyoon@apache.org
http://blog.udanax.org

From general-return-833-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Dec 12 03:19:55 2009
Return-Path: <general-return-833-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42244 invoked from network); 12 Dec 2009 03:19:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Dec 2009 03:19:54 -0000
Received: (qmail 32426 invoked by uid 500); 12 Dec 2009 03:19:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32218 invoked by uid 500); 12 Dec 2009 03:19:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32208 invoked by uid 99); 12 Dec 2009 03:19:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Dec 2009 03:19:51 +0000
X-ASF-Spam-Status: No, hits=-1.8 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of xshu@systemsbiology.org designates 64.18.2.24 as permitted sender)
Received: from [64.18.2.24] (HELO exprod7og123.obsmtp.com) (64.18.2.24)
    by apache.org (qpsmtpd/0.29) with SMTP; Sat, 12 Dec 2009 03:19:42 +0000
Received: from source ([209.85.223.172]) by exprod7ob123.postini.com ([64.18.6.12]) with SMTP
	ID DSNKSyMLuBq84GJSiUnEXU/SqY051vaR1tSy@postini.com; Fri, 11 Dec 2009 19:19:21 PST
Received: by iwn2 with SMTP id 2so1014451iwn.16
        for <general@hadoop.apache.org>; Fri, 11 Dec 2009 19:19:19 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.10.23 with SMTP id n23mr2151517ibn.4.1260587955020; Fri, 
	11 Dec 2009 19:19:15 -0800 (PST)
Date: Fri, 11 Dec 2009 19:19:14 -0800
Message-ID: <28902f870912111919s3f1a86adp75635e8c23d15d69@mail.gmail.com>
Subject: Which Hadoop product is more appropriate for a quick query on a large 
	data set?
From: Xueling Shu <xshu@systemsbiology.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000325579eb2446235047a7f81fd
X-Virus-Checked: Checked by ClamAV on apache.org

--000325579eb2446235047a7f81fd
Content-Type: text/plain; charset=ISO-8859-1

 Hi there:

I am researching Hadoop to see which of its products suits our need for
quick queries against large data sets (billions of records per set)

The queries will be performed against chip sequencing data. Each record is
one line in a file. To be clear below shows a sample record in the data set.


one line (record) looks like: 1-1-174-418 TGTGTCCCTTTGTAATGAATCACTATC U2 0 0
1 4 *103570835* F .. 23G 24C

The highlighted field is called "position of match" and the query we are
interested in is the # of sequences in a certain range of this "position of
match". For instance the range can be "position of match" > 200 and
"position of match" + 36 < 200,000.

Any suggestions on the Hadoop product I should start with to accomplish the
task? HBase,Pig,Hive, or ...?

Thanks!

Xueling

--000325579eb2446235047a7f81fd--

From general-return-834-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Dec 12 03:35:34 2009
Return-Path: <general-return-834-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49771 invoked from network); 12 Dec 2009 03:35:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Dec 2009 03:35:34 -0000
Received: (qmail 40675 invoked by uid 500); 12 Dec 2009 03:35:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40478 invoked by uid 500); 12 Dec 2009 03:35:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40468 invoked by uid 99); 12 Dec 2009 03:35:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Dec 2009 03:35:31 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.204] (HELO mail-px0-f204.google.com) (209.85.216.204)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Dec 2009 03:35:23 +0000
Received: by pxi42 with SMTP id 42so1102457pxi.5
        for <general@hadoop.apache.org>; Fri, 11 Dec 2009 19:35:01 -0800 (PST)
MIME-Version: 1.0
Received: by 10.143.25.9 with SMTP id c9mr1325962wfj.7.1260588901365; Fri, 11 
	Dec 2009 19:35:01 -0800 (PST)
In-Reply-To: <28902f870912111919s3f1a86adp75635e8c23d15d69@mail.gmail.com>
References: <28902f870912111919s3f1a86adp75635e8c23d15d69@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Fri, 11 Dec 2009 19:34:41 -0800
Message-ID: <45f85f70912111934v2960fb36s1922507c9bc61675@mail.gmail.com>
Subject: Re: Which Hadoop product is more appropriate for a quick query on a 
	large data set?
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Xueling,

One important question that can really change the answer:

How often does the dataset change? Can the changes be merged in in
bulk every once in a while, or do you need to actually update them
randomly very often?

Also, how fast is "quick"? Do you mean 1 minute, 10 seconds, 1 second, or 1=
0ms?

Thanks
-Todd

On Fri, Dec 11, 2009 at 7:19 PM, Xueling Shu <xshu@systemsbiology.org> wrot=
e:
> =A0Hi there:
>
> I am researching Hadoop to see which of its products suits our need for
> quick queries against large data sets (billions of records per set)
>
> The queries will be performed against chip sequencing data. Each record i=
s
> one line in a file. To be clear below shows a sample record in the data s=
et.
>
>
> one line (record) looks like: 1-1-174-418 TGTGTCCCTTTGTAATGAATCACTATC U2 =
0 0
> 1 4 *103570835* F .. 23G 24C
>
> The highlighted field is called "position of match" and the query we are
> interested in is the # of sequences in a certain range of this "position =
of
> match". For instance the range can be "position of match" > 200 and
> "position of match" + 36 < 200,000.
>
> Any suggestions on the Hadoop product I should start with to accomplish t=
he
> task? HBase,Pig,Hive, or ...?
>
> Thanks!
>
> Xueling
>

From general-return-835-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Dec 12 18:38:28 2009
Return-Path: <general-return-835-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39286 invoked from network); 12 Dec 2009 18:38:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Dec 2009 18:38:27 -0000
Received: (qmail 21413 invoked by uid 500); 12 Dec 2009 18:38:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21325 invoked by uid 500); 12 Dec 2009 18:38:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21315 invoked by uid 99); 12 Dec 2009 18:38:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Dec 2009 18:38:25 +0000
X-ASF-Spam-Status: No, hits=-6.6 required=5.0
	tests=BAYES_00,HTML_MESSAGE,RCVD_IN_DNSWL_MED
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of xshu@systemsbiology.org designates 64.18.2.215 as permitted sender)
Received: from [64.18.2.215] (HELO exprod7og114.obsmtp.com) (64.18.2.215)
    by apache.org (qpsmtpd/0.29) with SMTP; Sat, 12 Dec 2009 18:38:23 +0000
Received: from source ([209.85.223.199]) by exprod7ob114.postini.com ([64.18.6.12]) with SMTP
	ID DSNKSyPjCtTzYta+I3eRYssybrCtsjGurlvU@postini.com; Sat, 12 Dec 2009 10:38:03 PST
Received: by iwn37 with SMTP id 37so1381994iwn.30
        for <general@hadoop.apache.org>; Sat, 12 Dec 2009 10:38:01 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.5.23 with SMTP id 23mr2957598ibt.45.1260643081744; Sat, 12 
	Dec 2009 10:38:01 -0800 (PST)
In-Reply-To: <45f85f70912111934v2960fb36s1922507c9bc61675@mail.gmail.com>
References: <28902f870912111919s3f1a86adp75635e8c23d15d69@mail.gmail.com>
	 <45f85f70912111934v2960fb36s1922507c9bc61675@mail.gmail.com>
Date: Sat, 12 Dec 2009 10:38:01 -0800
Message-ID: <28902f870912121038g644b72b0vc832ec1343c16687@mail.gmail.com>
Subject: Re: Which Hadoop product is more appropriate for a quick query on a 
	large data set?
From: Xueling Shu <xshu@systemsbiology.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00151773da30137215047a8c57a7

--00151773da30137215047a8c57a7
Content-Type: text/plain; charset=ISO-8859-1

Hi Todd:

Thank you for your reply.

The datasets wont be updated often. But the query against a data set is
frequent. The quicker the query, the better. For example we have done
testing on a Mysql database (5 billion records randomly scattered into 24
tables) and the slowest query against the biggest table (400,000,000
records) is around 12 mins. So if using any Hadoop product can speed up the
search then the product is what we are looking for.

Cheers,
Xueling

On Fri, Dec 11, 2009 at 7:34 PM, Todd Lipcon <todd@cloudera.com> wrote:

> Hi Xueling,
>
> One important question that can really change the answer:
>
> How often does the dataset change? Can the changes be merged in in
> bulk every once in a while, or do you need to actually update them
> randomly very often?
>
> Also, how fast is "quick"? Do you mean 1 minute, 10 seconds, 1 second, or
> 10ms?
>
> Thanks
> -Todd
>
> On Fri, Dec 11, 2009 at 7:19 PM, Xueling Shu <xshu@systemsbiology.org>
> wrote:
> >  Hi there:
> >
> > I am researching Hadoop to see which of its products suits our need for
> > quick queries against large data sets (billions of records per set)
> >
> > The queries will be performed against chip sequencing data. Each record
> is
> > one line in a file. To be clear below shows a sample record in the data
> set.
> >
> >
> > one line (record) looks like: 1-1-174-418 TGTGTCCCTTTGTAATGAATCACTATC U2
> 0 0
> > 1 4 *103570835* F .. 23G 24C
> >
> > The highlighted field is called "position of match" and the query we are
> > interested in is the # of sequences in a certain range of this "position
> of
> > match". For instance the range can be "position of match" > 200 and
> > "position of match" + 36 < 200,000.
> >
> > Any suggestions on the Hadoop product I should start with to accomplish
> the
> > task? HBase,Pig,Hive, or ...?
> >
> > Thanks!
> >
> > Xueling
> >
>

--00151773da30137215047a8c57a7--

From general-return-836-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Dec 12 21:02:24 2009
Return-Path: <general-return-836-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63791 invoked from network); 12 Dec 2009 21:02:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Dec 2009 21:02:24 -0000
Received: (qmail 3556 invoked by uid 500); 12 Dec 2009 21:02:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3464 invoked by uid 500); 12 Dec 2009 21:02:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3453 invoked by uid 99); 12 Dec 2009 21:02:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Dec 2009 21:02:22 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Dec 2009 21:02:19 +0000
Received: by pwi20 with SMTP id 20so1365929pwi.29
        for <general@hadoop.apache.org>; Sat, 12 Dec 2009 13:01:59 -0800 (PST)
MIME-Version: 1.0
Received: by 10.142.3.37 with SMTP id 37mr1842060wfc.146.1260651719142; Sat, 
	12 Dec 2009 13:01:59 -0800 (PST)
In-Reply-To: <28902f870912121038g644b72b0vc832ec1343c16687@mail.gmail.com>
References: <28902f870912111919s3f1a86adp75635e8c23d15d69@mail.gmail.com> 
	<45f85f70912111934v2960fb36s1922507c9bc61675@mail.gmail.com> 
	<28902f870912121038g644b72b0vc832ec1343c16687@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Sat, 12 Dec 2009 13:01:39 -0800
Message-ID: <45f85f70912121301x353192du44becd5c16dce181@mail.gmail.com>
Subject: Re: Which Hadoop product is more appropriate for a quick query on a 
	large data set?
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Xueling,

In that case, I would recommend the following:

1) Put all of your data on HDFS
2) Write a MapReduce job that sorts the data by position of match
3) As a second output of this job, you can write a "sparse index" -
basically a set of entries like this:

<position of match> <offset into file> <number of entries following>

where you're basically giving offsets into every 10K records or so. If
you index every 10K records, then 5 billion total will mean 100,000
index entries. Each index entry shouldn't be more than 20 bytes, so
100,000 entries will be 2MB. This is super easy to fit into memory.
(you could probably index every 100th record instead and end up with
200MB, still easy to fit in memory)

Then to satisfy your count-range query, you can simply scan your
in-memory sparse index. Some of the indexed blocks will be completely
included in the range, in which case you just add up the "number of
entries following" column. The start and finish block will be
partially covered, so you can use the file offset info to load that
file off HDFS, start reading at that offset, and finish the count.

Total time per query should be <100ms no problem.

-Todd

On Sat, Dec 12, 2009 at 10:38 AM, Xueling Shu <xshu@systemsbiology.org> wro=
te:
> Hi Todd:
>
> Thank you for your reply.
>
> The datasets wont be updated often. But the query against a data set is
> frequent. The quicker the query, the better. For example we have done
> testing on a Mysql database (5 billion records randomly scattered into 24
> tables) and the slowest query against the biggest table (400,000,000
> records) is around 12 mins. So if using any Hadoop product can speed up t=
he
> search then the product is what we are looking for.
>
> Cheers,
> Xueling
>
> On Fri, Dec 11, 2009 at 7:34 PM, Todd Lipcon <todd@cloudera.com> wrote:
>
>> Hi Xueling,
>>
>> One important question that can really change the answer:
>>
>> How often does the dataset change? Can the changes be merged in in
>> bulk every once in a while, or do you need to actually update them
>> randomly very often?
>>
>> Also, how fast is "quick"? Do you mean 1 minute, 10 seconds, 1 second, o=
r
>> 10ms?
>>
>> Thanks
>> -Todd
>>
>> On Fri, Dec 11, 2009 at 7:19 PM, Xueling Shu <xshu@systemsbiology.org>
>> wrote:
>> > =A0Hi there:
>> >
>> > I am researching Hadoop to see which of its products suits our need fo=
r
>> > quick queries against large data sets (billions of records per set)
>> >
>> > The queries will be performed against chip sequencing data. Each recor=
d
>> is
>> > one line in a file. To be clear below shows a sample record in the dat=
a
>> set.
>> >
>> >
>> > one line (record) looks like: 1-1-174-418 TGTGTCCCTTTGTAATGAATCACTATC =
U2
>> 0 0
>> > 1 4 *103570835* F .. 23G 24C
>> >
>> > The highlighted field is called "position of match" and the query we a=
re
>> > interested in is the # of sequences in a certain range of this "positi=
on
>> of
>> > match". For instance the range can be "position of match" > 200 and
>> > "position of match" + 36 < 200,000.
>> >
>> > Any suggestions on the Hadoop product I should start with to accomplis=
h
>> the
>> > task? HBase,Pig,Hive, or ...?
>> >
>> > Thanks!
>> >
>> > Xueling
>> >
>>
>

From general-return-837-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Dec 12 22:51:24 2009
Return-Path: <general-return-837-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 3945 invoked from network); 12 Dec 2009 22:51:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Dec 2009 22:51:24 -0000
Received: (qmail 66494 invoked by uid 500); 12 Dec 2009 22:51:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66399 invoked by uid 500); 12 Dec 2009 22:51:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66389 invoked by uid 99); 12 Dec 2009 22:51:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Dec 2009 22:51:23 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 209.85.221.188 as permitted sender)
Received: from [209.85.221.188] (HELO mail-qy0-f188.google.com) (209.85.221.188)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Dec 2009 22:51:15 +0000
Received: by qyk26 with SMTP id 26so1048786qyk.5
        for <general@hadoop.apache.org>; Sat, 12 Dec 2009 14:50:54 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:message-id:subject:from:to
         :content-type;
        bh=O0gIwrAbfhoo3fF28mWQn7cV4cQx/FCwD++ElnuLxvk=;
        b=Ja6ozZY36oCRJLYF7aFF3URPjxnfRUvEnOrqYqWFWyo65RIBFaoBPBJHYQF9VEdaeF
         dBytv2pd6/79+CVaMEVHuG6BEgRfCQdtOLLcgmL5pT5yPhfGUYU99WAKGuamIBt0oskv
         fvJ0g6162EUf2vEeJEDvkuD7bNang8EocS/jg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=mmVtcSO5rRBksEpksqzfIIZIiYQ2oYOCZ9Ca0maKPaZhz+g/G8j3+eULXnSIwFYmXn
         TjUhxYlyGkdIwFqpjJfBYJdsIXQmMX/jpRHY9UVhPmGypXKapyiK5om7phVnfqbr1zjU
         hs/elCLhc3EfbfcypeOUT68fhhcPRbXPCI5sw=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
Received: by 10.229.28.16 with SMTP id k16mr1595250qcc.70.1260658253899; Sat, 
	12 Dec 2009 14:50:53 -0800 (PST)
In-Reply-To: <45f85f70912121301x353192du44becd5c16dce181@mail.gmail.com>
References: <28902f870912111919s3f1a86adp75635e8c23d15d69@mail.gmail.com>
	 <45f85f70912111934v2960fb36s1922507c9bc61675@mail.gmail.com>
	 <28902f870912121038g644b72b0vc832ec1343c16687@mail.gmail.com>
	 <45f85f70912121301x353192du44becd5c16dce181@mail.gmail.com>
Date: Sat, 12 Dec 2009 14:50:53 -0800
X-Google-Sender-Auth: cbcf129e23be6231
Message-ID: <7c962aed0912121450y30f8b86cm54e652002bd14d6f@mail.gmail.com>
Subject: Re: Which Hadoop product is more appropriate for a quick query on a 
	large data set?
From: stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636426bef682ad2047a8fdf5c
X-Virus-Checked: Checked by ClamAV on apache.org

--001636426bef682ad2047a8fdf5c
Content-Type: text/plain; charset=ISO-8859-1

You might also consider hbase, particularly If you find that your data is
being updated with some regularity, particularly if the updates are randomly
distributed over the data set.  See
http://hadoop.apache.org/hbase/docs/r0.20.2/api/org/apache/hadoop/hbase/mapreduce/package-summary.html#bulkfor
how to do a fast bulk load of your billiions of rows of data.

Yours,
St.Ack


On Sat, Dec 12, 2009 at 1:01 PM, Todd Lipcon <todd@cloudera.com> wrote:

> Hi Xueling,
>
> In that case, I would recommend the following:
>
> 1) Put all of your data on HDFS
> 2) Write a MapReduce job that sorts the data by position of match
> 3) As a second output of this job, you can write a "sparse index" -
> basically a set of entries like this:
>
> <position of match> <offset into file> <number of entries following>
>
> where you're basically giving offsets into every 10K records or so. If
> you index every 10K records, then 5 billion total will mean 100,000
> index entries. Each index entry shouldn't be more than 20 bytes, so
> 100,000 entries will be 2MB. This is super easy to fit into memory.
> (you could probably index every 100th record instead and end up with
> 200MB, still easy to fit in memory)
>
> Then to satisfy your count-range query, you can simply scan your
> in-memory sparse index. Some of the indexed blocks will be completely
> included in the range, in which case you just add up the "number of
> entries following" column. The start and finish block will be
> partially covered, so you can use the file offset info to load that
> file off HDFS, start reading at that offset, and finish the count.
>
> Total time per query should be <100ms no problem.
>
> -Todd
>
> On Sat, Dec 12, 2009 at 10:38 AM, Xueling Shu <xshu@systemsbiology.org>
> wrote:
> > Hi Todd:
> >
> > Thank you for your reply.
> >
> > The datasets wont be updated often. But the query against a data set is
> > frequent. The quicker the query, the better. For example we have done
> > testing on a Mysql database (5 billion records randomly scattered into 24
> > tables) and the slowest query against the biggest table (400,000,000
> > records) is around 12 mins. So if using any Hadoop product can speed up
> the
> > search then the product is what we are looking for.
> >
> > Cheers,
> > Xueling
> >
> > On Fri, Dec 11, 2009 at 7:34 PM, Todd Lipcon <todd@cloudera.com> wrote:
> >
> >> Hi Xueling,
> >>
> >> One important question that can really change the answer:
> >>
> >> How often does the dataset change? Can the changes be merged in in
> >> bulk every once in a while, or do you need to actually update them
> >> randomly very often?
> >>
> >> Also, how fast is "quick"? Do you mean 1 minute, 10 seconds, 1 second,
> or
> >> 10ms?
> >>
> >> Thanks
> >> -Todd
> >>
> >> On Fri, Dec 11, 2009 at 7:19 PM, Xueling Shu <xshu@systemsbiology.org>
> >> wrote:
> >> >  Hi there:
> >> >
> >> > I am researching Hadoop to see which of its products suits our need
> for
> >> > quick queries against large data sets (billions of records per set)
> >> >
> >> > The queries will be performed against chip sequencing data. Each
> record
> >> is
> >> > one line in a file. To be clear below shows a sample record in the
> data
> >> set.
> >> >
> >> >
> >> > one line (record) looks like: 1-1-174-418 TGTGTCCCTTTGTAATGAATCACTATC
> U2
> >> 0 0
> >> > 1 4 *103570835* F .. 23G 24C
> >> >
> >> > The highlighted field is called "position of match" and the query we
> are
> >> > interested in is the # of sequences in a certain range of this
> "position
> >> of
> >> > match". For instance the range can be "position of match" > 200 and
> >> > "position of match" + 36 < 200,000.
> >> >
> >> > Any suggestions on the Hadoop product I should start with to
> accomplish
> >> the
> >> > task? HBase,Pig,Hive, or ...?
> >> >
> >> > Thanks!
> >> >
> >> > Xueling
> >> >
> >>
> >
>

--001636426bef682ad2047a8fdf5c--

From general-return-838-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Dec 12 22:56:27 2009
Return-Path: <general-return-838-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4677 invoked from network); 12 Dec 2009 22:56:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Dec 2009 22:56:27 -0000
Received: (qmail 69606 invoked by uid 500); 12 Dec 2009 22:56:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69519 invoked by uid 500); 12 Dec 2009 22:56:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69509 invoked by uid 99); 12 Dec 2009 22:56:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Dec 2009 22:56:26 +0000
X-ASF-Spam-Status: No, hits=-6.6 required=5.0
	tests=BAYES_00,HTML_MESSAGE,RCVD_IN_DNSWL_MED
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of xshu@systemsbiology.org designates 64.18.2.217 as permitted sender)
Received: from [64.18.2.217] (HELO exprod7og115.obsmtp.com) (64.18.2.217)
    by apache.org (qpsmtpd/0.29) with SMTP; Sat, 12 Dec 2009 22:56:24 +0000
Received: from source ([209.85.223.182]) by exprod7ob115.postini.com ([64.18.6.12]) with SMTP
	ID DSNKSyQfglSNwpnNQymIUG+njpl9H+ooTW19@postini.com; Sat, 12 Dec 2009 14:56:04 PST
Received: by iwn12 with SMTP id 12so1479766iwn.27
        for <general@hadoop.apache.org>; Sat, 12 Dec 2009 14:56:02 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.123.216 with SMTP id q24mr1851003ibr.43.1260658562277; 
	Sat, 12 Dec 2009 14:56:02 -0800 (PST)
In-Reply-To: <45f85f70912121301x353192du44becd5c16dce181@mail.gmail.com>
References: <28902f870912111919s3f1a86adp75635e8c23d15d69@mail.gmail.com>
	 <45f85f70912111934v2960fb36s1922507c9bc61675@mail.gmail.com>
	 <28902f870912121038g644b72b0vc832ec1343c16687@mail.gmail.com>
	 <45f85f70912121301x353192du44becd5c16dce181@mail.gmail.com>
Date: Sat, 12 Dec 2009 14:56:02 -0800
Message-ID: <28902f870912121456g116054b3re1e42d317b12e083@mail.gmail.com>
Subject: Re: Which Hadoop product is more appropriate for a quick query on a 
	large data set?
From: Xueling Shu <xshu@systemsbiology.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64698e4c9b02b047a8ff15e

--0016e64698e4c9b02b047a8ff15e
Content-Type: text/plain; charset=ISO-8859-1

Great information! Thank you for your help, Todd.

Xueling

On Sat, Dec 12, 2009 at 1:01 PM, Todd Lipcon <todd@cloudera.com> wrote:

> Hi Xueling,
>
> In that case, I would recommend the following:
>
> 1) Put all of your data on HDFS
> 2) Write a MapReduce job that sorts the data by position of match
> 3) As a second output of this job, you can write a "sparse index" -
> basically a set of entries like this:
>
> <position of match> <offset into file> <number of entries following>
>
> where you're basically giving offsets into every 10K records or so. If
> you index every 10K records, then 5 billion total will mean 100,000
> index entries. Each index entry shouldn't be more than 20 bytes, so
> 100,000 entries will be 2MB. This is super easy to fit into memory.
> (you could probably index every 100th record instead and end up with
> 200MB, still easy to fit in memory)
>
> Then to satisfy your count-range query, you can simply scan your
> in-memory sparse index. Some of the indexed blocks will be completely
> included in the range, in which case you just add up the "number of
> entries following" column. The start and finish block will be
> partially covered, so you can use the file offset info to load that
> file off HDFS, start reading at that offset, and finish the count.
>
> Total time per query should be <100ms no problem.
>
> -Todd
>
> On Sat, Dec 12, 2009 at 10:38 AM, Xueling Shu <xshu@systemsbiology.org>
> wrote:
> > Hi Todd:
> >
> > Thank you for your reply.
> >
> > The datasets wont be updated often. But the query against a data set is
> > frequent. The quicker the query, the better. For example we have done
> > testing on a Mysql database (5 billion records randomly scattered into 24
> > tables) and the slowest query against the biggest table (400,000,000
> > records) is around 12 mins. So if using any Hadoop product can speed up
> the
> > search then the product is what we are looking for.
> >
> > Cheers,
> > Xueling
> >
> > On Fri, Dec 11, 2009 at 7:34 PM, Todd Lipcon <todd@cloudera.com> wrote:
> >
> >> Hi Xueling,
> >>
> >> One important question that can really change the answer:
> >>
> >> How often does the dataset change? Can the changes be merged in in
> >> bulk every once in a while, or do you need to actually update them
> >> randomly very often?
> >>
> >> Also, how fast is "quick"? Do you mean 1 minute, 10 seconds, 1 second,
> or
> >> 10ms?
> >>
> >> Thanks
> >> -Todd
> >>
> >> On Fri, Dec 11, 2009 at 7:19 PM, Xueling Shu <xshu@systemsbiology.org>
> >> wrote:
> >> >  Hi there:
> >> >
> >> > I am researching Hadoop to see which of its products suits our need
> for
> >> > quick queries against large data sets (billions of records per set)
> >> >
> >> > The queries will be performed against chip sequencing data. Each
> record
> >> is
> >> > one line in a file. To be clear below shows a sample record in the
> data
> >> set.
> >> >
> >> >
> >> > one line (record) looks like: 1-1-174-418 TGTGTCCCTTTGTAATGAATCACTATC
> U2
> >> 0 0
> >> > 1 4 *103570835* F .. 23G 24C
> >> >
> >> > The highlighted field is called "position of match" and the query we
> are
> >> > interested in is the # of sequences in a certain range of this
> "position
> >> of
> >> > match". For instance the range can be "position of match" > 200 and
> >> > "position of match" + 36 < 200,000.
> >> >
> >> > Any suggestions on the Hadoop product I should start with to
> accomplish
> >> the
> >> > task? HBase,Pig,Hive, or ...?
> >> >
> >> > Thanks!
> >> >
> >> > Xueling
> >> >
> >>
> >
>

--0016e64698e4c9b02b047a8ff15e--

From general-return-839-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Dec 12 23:31:51 2009
Return-Path: <general-return-839-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 11864 invoked from network); 12 Dec 2009 23:31:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Dec 2009 23:31:50 -0000
Received: (qmail 91037 invoked by uid 500); 12 Dec 2009 23:31:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90943 invoked by uid 500); 12 Dec 2009 23:31:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90933 invoked by uid 99); 12 Dec 2009 23:31:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Dec 2009 23:31:49 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of fzappert@gmail.com designates 209.85.221.188 as permitted sender)
Received: from [209.85.221.188] (HELO mail-qy0-f188.google.com) (209.85.221.188)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Dec 2009 23:31:41 +0000
Received: by qyk26 with SMTP id 26so1057011qyk.5
        for <general@hadoop.apache.org>; Sat, 12 Dec 2009 15:31:20 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=he9TQqttFeHkrsUsM/yknLg5kLomPymjuD2bsXI4tlk=;
        b=P0nyVqne0ETfWKBrNRtBvFsvn2U71rd7l+9CYbJxa89RC5h1716goJMyKfQRUVs8As
         aIOgEvxd1njC7Dpc5JVd+hGvFReFCLw+hFYutXPlZedZkIR7RzXC5rVFdL9x+j7gkmMm
         0HbyrWymGvyg49b0ABKPosztptCphyN7Kohy8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=jGJ3683T0vRPrJt6RCPbQNFY5VTzjAVlEgA2YuXtz2wQXE3B2jr1lz5C8GG7QXf5AO
         2MT3cwssKo6gc29Dchw0OC5zrNK5yNG1jrubPPDPlkm8fs1DRBVy9uaXI+ULOwdSimiE
         yCayEJ0IcLYPW7a9YlqQgjpXcPp/oKFrfREO4=
MIME-Version: 1.0
Received: by 10.229.101.165 with SMTP id c37mr1694312qco.5.1260660679551; Sat, 
	12 Dec 2009 15:31:19 -0800 (PST)
In-Reply-To: <28902f870912121456g116054b3re1e42d317b12e083@mail.gmail.com>
References: <28902f870912111919s3f1a86adp75635e8c23d15d69@mail.gmail.com>
	 <45f85f70912111934v2960fb36s1922507c9bc61675@mail.gmail.com>
	 <28902f870912121038g644b72b0vc832ec1343c16687@mail.gmail.com>
	 <45f85f70912121301x353192du44becd5c16dce181@mail.gmail.com>
	 <28902f870912121456g116054b3re1e42d317b12e083@mail.gmail.com>
Date: Sat, 12 Dec 2009 15:31:19 -0800
Message-ID: <14048aeb0912121531p78d02705vf4402a041658c668@mail.gmail.com>
Subject: Re: Which Hadoop product is more appropriate for a quick query on a 
	large data set?
From: Fred Zappert <fzappert@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364ee4c8fcacfb047a906fb0
X-Virus-Checked: Checked by ClamAV on apache.org

--0016364ee4c8fcacfb047a906fb0
Content-Type: text/plain; charset=ISO-8859-1

+1 for hbase

On Sat, Dec 12, 2009 at 2:56 PM, Xueling Shu <xshu@systemsbiology.org>wrote:

> Great information! Thank you for your help, Todd.
>
> Xueling
>
> On Sat, Dec 12, 2009 at 1:01 PM, Todd Lipcon <todd@cloudera.com> wrote:
>
> > Hi Xueling,
> >
> > In that case, I would recommend the following:
> >
> > 1) Put all of your data on HDFS
> > 2) Write a MapReduce job that sorts the data by position of match
> > 3) As a second output of this job, you can write a "sparse index" -
> > basically a set of entries like this:
> >
> > <position of match> <offset into file> <number of entries following>
> >
> > where you're basically giving offsets into every 10K records or so. If
> > you index every 10K records, then 5 billion total will mean 100,000
> > index entries. Each index entry shouldn't be more than 20 bytes, so
> > 100,000 entries will be 2MB. This is super easy to fit into memory.
> > (you could probably index every 100th record instead and end up with
> > 200MB, still easy to fit in memory)
> >
> > Then to satisfy your count-range query, you can simply scan your
> > in-memory sparse index. Some of the indexed blocks will be completely
> > included in the range, in which case you just add up the "number of
> > entries following" column. The start and finish block will be
> > partially covered, so you can use the file offset info to load that
> > file off HDFS, start reading at that offset, and finish the count.
> >
> > Total time per query should be <100ms no problem.
> >
> > -Todd
> >
> > On Sat, Dec 12, 2009 at 10:38 AM, Xueling Shu <xshu@systemsbiology.org>
> > wrote:
> > > Hi Todd:
> > >
> > > Thank you for your reply.
> > >
> > > The datasets wont be updated often. But the query against a data set is
> > > frequent. The quicker the query, the better. For example we have done
> > > testing on a Mysql database (5 billion records randomly scattered into
> 24
> > > tables) and the slowest query against the biggest table (400,000,000
> > > records) is around 12 mins. So if using any Hadoop product can speed up
> > the
> > > search then the product is what we are looking for.
> > >
> > > Cheers,
> > > Xueling
> > >
> > > On Fri, Dec 11, 2009 at 7:34 PM, Todd Lipcon <todd@cloudera.com>
> wrote:
> > >
> > >> Hi Xueling,
> > >>
> > >> One important question that can really change the answer:
> > >>
> > >> How often does the dataset change? Can the changes be merged in in
> > >> bulk every once in a while, or do you need to actually update them
> > >> randomly very often?
> > >>
> > >> Also, how fast is "quick"? Do you mean 1 minute, 10 seconds, 1 second,
> > or
> > >> 10ms?
> > >>
> > >> Thanks
> > >> -Todd
> > >>
> > >> On Fri, Dec 11, 2009 at 7:19 PM, Xueling Shu <xshu@systemsbiology.org
> >
> > >> wrote:
> > >> >  Hi there:
> > >> >
> > >> > I am researching Hadoop to see which of its products suits our need
> > for
> > >> > quick queries against large data sets (billions of records per set)
> > >> >
> > >> > The queries will be performed against chip sequencing data. Each
> > record
> > >> is
> > >> > one line in a file. To be clear below shows a sample record in the
> > data
> > >> set.
> > >> >
> > >> >
> > >> > one line (record) looks like: 1-1-174-418
> TGTGTCCCTTTGTAATGAATCACTATC
> > U2
> > >> 0 0
> > >> > 1 4 *103570835* F .. 23G 24C
> > >> >
> > >> > The highlighted field is called "position of match" and the query we
> > are
> > >> > interested in is the # of sequences in a certain range of this
> > "position
> > >> of
> > >> > match". For instance the range can be "position of match" > 200 and
> > >> > "position of match" + 36 < 200,000.
> > >> >
> > >> > Any suggestions on the Hadoop product I should start with to
> > accomplish
> > >> the
> > >> > task? HBase,Pig,Hive, or ...?
> > >> >
> > >> > Thanks!
> > >> >
> > >> > Xueling
> > >> >
> > >>
> > >
> >
>

--0016364ee4c8fcacfb047a906fb0--

From general-return-840-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Dec 14 21:34:50 2009
Return-Path: <general-return-840-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97457 invoked from network); 14 Dec 2009 21:34:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Dec 2009 21:34:50 -0000
Received: (qmail 78861 invoked by uid 500); 14 Dec 2009 21:34:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78485 invoked by uid 500); 14 Dec 2009 21:34:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78435 invoked by uid 99); 14 Dec 2009 21:34:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Dec 2009 21:34:45 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Dec 2009 21:34:43 +0000
Received: from [10.72.168.6] (snvvpn4-10-72-168-c6.hq.corp.yahoo.com [10.72.168.6])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id nBELYBmR057375;
	Mon, 14 Dec 2009 13:34:12 -0800 (PST)
Message-ID: <4B26AF53.9070604@apache.org>
Date: Mon, 14 Dec 2009 13:34:11 -0800
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: announce@apache.org,
        "zookeeper-user@hadoop.apache.org" <zookeeper-user@hadoop.apache.org>,
        "zookeeper-dev@hadoop.apache.org" <zookeeper-dev@hadoop.apache.org>,
        general@hadoop.apache.org, private@hadoop.apache.org
Subject: [ANNOUNCE] Apache ZooKeeper 3.1.2
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

The Apache ZooKeeper team is proud to announce Apache ZooKeeper version 
3.1.2.

ZooKeeper is a high-performance coordination service for distributed 
applications. It exposes common services - such as naming, configuration 
management, synchronization, and group services - in a simple interface 
so you don't have to write them from scratch. You can use it 
off-the-shelf to implement consensus, group management, leader election, 
and presence protocols. And you can build on it for your own, specific 
needs.

If you are upgrading from version 2.2.1 on SourceForge be sure to review 
the 3.0.1 release notes for migration instructions.

For ZooKeeper release details and downloads, visit:
http://hadoop.apache.org/zookeeper/releases.html

ZooKeeper 3.1.2 Release Notes are at:
http://hadoop.apache.org/zookeeper/docs/r3.1.2/releasenotes.html

Regards,

The ZooKeeper Team



From general-return-841-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Dec 14 21:35:17 2009
Return-Path: <general-return-841-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97786 invoked from network); 14 Dec 2009 21:35:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Dec 2009 21:35:16 -0000
Received: (qmail 80055 invoked by uid 500); 14 Dec 2009 21:35:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79875 invoked by uid 500); 14 Dec 2009 21:35:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 79822 invoked by uid 99); 14 Dec 2009 21:35:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Dec 2009 21:35:14 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Dec 2009 21:35:12 +0000
Received: from [10.72.168.6] (snvvpn4-10-72-168-c6.hq.corp.yahoo.com [10.72.168.6])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id nBELYj3O057690;
	Mon, 14 Dec 2009 13:34:45 -0800 (PST)
Message-ID: <4B26AF74.7020509@apache.org>
Date: Mon, 14 Dec 2009 13:34:44 -0800
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: announce@apache.org,
        "zookeeper-user@hadoop.apache.org" <zookeeper-user@hadoop.apache.org>,
        "zookeeper-dev@hadoop.apache.org" <zookeeper-dev@hadoop.apache.org>,
        general@hadoop.apache.org, private@hadoop.apache.org
Subject: [ANNOUNCE] Apache ZooKeeper 3.2.2
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

The Apache ZooKeeper team is proud to announce Apache ZooKeeper version 
3.2.2.

ZooKeeper is a high-performance coordination service for distributed 
applications. It exposes common services - such as naming, configuration 
management, synchronization, and group services - in a simple interface 
so you don't have to write them from scratch. You can use it 
off-the-shelf to implement consensus, group management, leader election, 
and presence protocols. And you can build on it for your own, specific 
needs.

If you are upgrading from version 2.2.1 on SourceForge be sure to review 
the 3.0.1 release notes for migration instructions.

For ZooKeeper release details and downloads, visit:
http://hadoop.apache.org/zookeeper/releases.html

ZooKeeper 3.2.2 Release Notes are at:
http://hadoop.apache.org/zookeeper/docs/r3.2.2/releasenotes.html

Regards,

The ZooKeeper Team



From general-return-842-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 15 18:06:10 2009
Return-Path: <general-return-842-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 40982 invoked from network); 15 Dec 2009 18:06:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Dec 2009 18:06:07 -0000
Received: (qmail 1757 invoked by uid 500); 15 Dec 2009 18:06:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1565 invoked by uid 500); 15 Dec 2009 18:06:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1545 invoked by uid 99); 15 Dec 2009 18:06:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Dec 2009 18:06:01 +0000
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Dec 2009 18:05:51 +0000
Received: from SP1-EX07CAS02.ds.corp.yahoo.com (sp1-ex07cas02.ds.corp.yahoo.com [216.252.116.138])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id nBFI4u5X009971;
	Tue, 15 Dec 2009 10:04:56 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:references:in-reply-to:accept-language:
	content-language:x-ms-has-attach:x-ms-tnef-correlator:
	x-cr-hashedpuzzle:x-cr-puzzleid:acceptlanguage:content-type:
	content-transfer-encoding:mime-version;
	b=QULAQWy2LXfZf38tOhcf2imB6pTraKqGM6mJYsN3Sc4FPSOmIAxxHv8tO8Qf2ZoG
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 SP1-EX07CAS02.ds.corp.yahoo.com ([216.252.116.167]) with mapi; Tue, 15 Dec
 2009 10:04:55 -0800
From: Dekel Tankel <dekel@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>,
        "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Date: Tue, 15 Dec 2009 10:04:47 -0800
Subject: Hadoop User Group (Bay Area) - Video recording available for Nov
  18th meeting
Thread-Topic: Hadoop User Group (Bay Area) - Video recording available for
 Nov  18th meeting
Thread-Index: AcoSMRRU8X5auVLKR8ufbpFTQi1hLgQXaiyQCpfO4pAMMLNs0A==
Message-ID: <46E2672895E8744A97D7577A6110858B4FD9A702CE@SP1-EX07VS01.ds.corp.yahoo.com>
References: <46E2672895E8744A97D7577A6110858B4E1AC69A91@SP1-EX07VS01.ds.corp.yahoo.com>
  <46E2672895E8744A97D7577A6110858B4FD39F6EDF@SP1-EX07VS01.ds.corp.yahoo.com>
In-Reply-To: <46E2672895E8744A97D7577A6110858B4FD39F6EDF@SP1-EX07VS01.ds.corp.yahoo.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-cr-hashedpuzzle: +E4= AN5+ ByUA DNB7 FWe6 J/2l Kx2W M17g M/Eh Q+SY RVD9
 TiBh VfXU WaPd WfHj
 Xkg3;2;YwBvAG0AbQBvAG4ALQB1AHMAZQByAEAAaABhAGQAbwBvAHAALgBhAHAAYQBjAGgAZQAuAG8AcgBnADsAZwBlAG4AZQByAGEAbABAAGgAYQBkAG8AbwBwAC4AYQBwAGEAYwBoAGUALgBvAHIAZwA=;Sosha1_v1;7;{382AC0F7-E38B-4AE9-9007-0657061F41C1};ZABlAGsAZQBsAEAAeQBhAGgAbwBvAC0AaQBuAGMALgBjAG8AbQA=;Tue,
 15 Dec 2009 18:04:47
 GMT;SABhAGQAbwBvAHAAIABVAHMAZQByACAARwByAG8AdQBwACAAKABCAGEAeQAgAEEAcgBlAGEAKQAgAC0AIABWAGkAZABlAG8AIAByAGUAYwBvAHIAZABpAG4AZwAgAGEAdgBhAGkAbABhAGIAbABlACAAZgBvAHIAIABOAG8AdgAgACAAMQA4AHQAaAAgAG0AZQBlAHQAaQBuAGcA
x-cr-puzzleid: {382AC0F7-E38B-4AE9-9007-0657061F41C1}
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Hi all,

For those of you who were not able to join us in person for the Nov 18th me=
eting at Yahoo!, please see the blog post below for video recording and sli=
de-share of the interesting presentations held in this event.

http://feeds.developer.yahoo.net/~r/YDNHadoop/~3/_7yYod3aht4/1118_hadoop_ba=
y_area_user_grou.html


Happy holidays

	Dekel


From general-return-843-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 16 15:25:46 2009
Return-Path: <general-return-843-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19878 invoked from network); 16 Dec 2009 15:25:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Dec 2009 15:25:45 -0000
Received: (qmail 11532 invoked by uid 500); 16 Dec 2009 15:25:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11368 invoked by uid 500); 16 Dec 2009 15:25:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11350 invoked by uid 99); 16 Dec 2009 15:25:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Dec 2009 15:25:43 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [156.148.72.33] (HELO raffaello.crs4.it) (156.148.72.33)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Dec 2009 15:25:33 +0000
Received: from [156.148.71.202] (neuron.crs4.it [156.148.71.202])
	by raffaello.crs4.it (Postfix) with ESMTP id B99D66BBAA
	for <general@hadoop.apache.org>; Wed, 16 Dec 2009 16:24:15 +0100 (CET)
Message-ID: <4B28FBD8.5020001@crs4.it>
Date: Wed, 16 Dec 2009 16:25:12 +0100
From: Simone Leo <simone.leo@crs4.it>
User-Agent: Thunderbird 2.0.0.23 (X11/20091007)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: First Italian HUG Meeting
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

We are wondering whether there is interest in a Hadoop User Group
meeting in Italy. Our main area of interest is bioinformatics
applications, specifically deep sequencing data analysis, but anything
MapReduce-related would be great for sharing ideas.

The event could be hosted, let's say this upcoming Spring, by us at CRS4
-- Distributed Computing (http://dc.crs4.it). It could be a "birds of a
feather" meeting where people discuss what they are doing with Hadoop
and how, in an informal fashion, and maybe give some tutorial
presentations. For instance, we could talk about our experience in
Python MapReduce programming and integration with scheduling systems
(Sun Grid Engine).

Please e-mail me if you like the idea, so that we can assess whether
there is enough interest in having the meeting.

-- 
Simone Leo
Distributed Computing group
Advanced Computing and Communications program
CRS4
POLARIS - Building #1
Piscina Manna
I-09010 Pula (CA) - Italy
e-mail: simleo@crs4.it
http://dc.crs4.it

From general-return-844-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 16 21:26:33 2009
Return-Path: <general-return-844-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 86068 invoked from network); 16 Dec 2009 21:26:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Dec 2009 21:26:32 -0000
Received: (qmail 30123 invoked by uid 500); 16 Dec 2009 21:26:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29996 invoked by uid 500); 16 Dec 2009 21:26:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29977 invoked by uid 99); 16 Dec 2009 21:26:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Dec 2009 21:26:30 +0000
X-ASF-Spam-Status: No, hits=-2.4 required=5.0
	tests=AWL,BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of psdc1978@gmail.com designates 209.85.220.215 as permitted sender)
Received: from [209.85.220.215] (HELO mail-fx0-f215.google.com) (209.85.220.215)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Dec 2009 21:26:27 +0000
Received: by fxm7 with SMTP id 7so1410444fxm.29
        for <multiple recipients>; Wed, 16 Dec 2009 13:26:06 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=jtby23HK6ipaX9zrX8s0MCPNs8U83I/O1D8hVa38DT4=;
        b=FfTaDtEo6wWZzKZZ+DOX54U8Q0c2nfxUxwIKOXqbWpy69m/v4NZWLsoX4SP2Z6lE30
         eBIScj1zLjiBTXSX+LYXPWWXykAULAozCL/OGJ8I3L9EVDSHsXZvqOQukAtMBjG9Hj9h
         e5Y8WrsmRtqw4j+Wjhi6ATjbEHn1mSXI95Fzo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=pKZnhKtLIU2b50eQprY1zuTqgpcSwaQdotdtiA3LCUumUPsh9uEZavybXAUBC5TBtQ
         /zbtMQZ/fEUoR3PVIj6MOKq3akEUmj1MAv+epGBDfZtwDzJ/Cd8PUsZNN9jxY/WkEf7t
         SNokbufXPYBzs76sfbl+oLIhVzZq747hSrbzk=
MIME-Version: 1.0
Received: by 10.239.130.31 with SMTP id 31mr159163hbh.134.1260998764067; Wed, 
	16 Dec 2009 13:26:04 -0800 (PST)
Date: Wed, 16 Dec 2009 21:26:04 +0000
Message-ID: <61fec3fc0912161326k63debf43me9361c3b30182d03@mail.gmail.com>
Subject: Defining the number of map tasks
From: psdc1978 <psdc1978@gmail.com>
To: common-user@hadoop.apache.org, general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hi,

I would like to have several Map tasks that execute the same tasks.
For example, I've 3 map tasks (M1, M2 and M3) and a 1Gb of input data
to be read by each map. Each map should read the same input data and
send the result to the same Reduce. At the end, the reduce should
produce the same 3 results.

Put in conf/slaves file 3 instances of the same machine

<file>
localhost
localhost
localhost
</file>

does it solve the problem?


How I define the number of map tasks to run?



Best regards,
-- 
xeon

From general-return-845-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Dec 18 13:43:28 2009
Return-Path: <general-return-845-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25205 invoked from network); 18 Dec 2009 13:43:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Dec 2009 13:43:27 -0000
Received: (qmail 69641 invoked by uid 500); 18 Dec 2009 13:43:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69562 invoked by uid 500); 18 Dec 2009 13:43:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69552 invoked by uid 99); 18 Dec 2009 13:43:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Dec 2009 13:43:25 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of admin@jobomix.com designates 209.85.218.223 as permitted sender)
Received: from [209.85.218.223] (HELO mail-bw0-f223.google.com) (209.85.218.223)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Dec 2009 13:43:15 +0000
Received: by bwz23 with SMTP id 23so2200686bwz.29
        for <general@hadoop.apache.org>; Fri, 18 Dec 2009 05:42:55 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.154.68 with SMTP id n4mr2377655bkw.54.1261143774719; Fri, 
	18 Dec 2009 05:42:54 -0800 (PST)
Date: Fri, 18 Dec 2009 13:42:54 +0000
Message-ID: <9bb662ec0912180542o3753e927t48c64eb4cfed282a@mail.gmail.com>
Subject: Encoding Hell
From: Bruno Abitbol <bruno.abitbol@jobomix.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175d0064b3f18b047b00eac1
X-Virus-Checked: Checked by ClamAV on apache.org

--0015175d0064b3f18b047b00eac1
Content-Type: text/plain; charset=ISO-8859-1

Hi,

I have been playing around for two days trying to figure out an issue
related to the default charset:


   - When I run a very dummy job which just displays the default charset on
   hadoop using the pseudo connected mode, I obtain US-ASCII. When I display
   the java property file.encoding I obtain ANSI_X3.4-1968


   - When I run the same job under Eclipse in locale mode I obtain UTF-8
   (which is the one I expect).

I use a Linux Gentoo distribution, the locale env variables are the
following:

LANG=en_GB.UTF-8
LC_CTYPE="en_GB.UTF-8"
LC_NUMERIC="en_GB.UTF-8"
LC_TIME="en_GB.UTF-8"
LC_COLLATE="en_GB.UTF-8"
LC_MONETARY="en_GB.UTF-8"
LC_MESSAGES="en_GB.UTF-8"
LC_PAPER="en_GB.UTF-8"
LC_NAME="en_GB.UTF-8"
LC_ADDRESS="en_GB.UTF-8"
LC_TELEPHONE="en_GB.UTF-8"
LC_MEASUREMENT="en_GB.UTF-8"
LC_IDENTIFICATION="en_GB.UTF-8"
LC_ALL=en_GB.UTF-8

I have tried to set the file.encoding property to UTF-8 but it doesn't work.
Any help would be greatly appreciated.

Thank you.

-- 
Bruno Abitbol
bruno.abitbol@jobomix.com
http://www.jobomix.fr

--0015175d0064b3f18b047b00eac1--

From general-return-846-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Dec 21 17:43:20 2009
Return-Path: <general-return-846-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 10134 invoked from network); 21 Dec 2009 17:43:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Dec 2009 17:43:19 -0000
Received: (qmail 98039 invoked by uid 500); 21 Dec 2009 17:43:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97938 invoked by uid 500); 21 Dec 2009 17:43:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97928 invoked by uid 99); 21 Dec 2009 17:43:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Dec 2009 17:43:17 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mailinglists19@gmail.com designates 72.14.220.157 as permitted sender)
Received: from [72.14.220.157] (HELO fg-out-1718.google.com) (72.14.220.157)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Dec 2009 17:43:10 +0000
Received: by fg-out-1718.google.com with SMTP id 19so1911913fgg.11
        for <general@hadoop.apache.org>; Mon, 21 Dec 2009 09:42:49 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=OxX9qiGu93v7HHDcNexvZG9T7IjeSihByWT2LX5uaqs=;
        b=LPvelHHc8DdqEFA3THNA1i7EiidGW5/txdRyuqGh/pQ/JKyx6ft6ysrMW50erVRZHL
         W/hHqDslnYMA0caiboCThyDBlKaPyu5f7CX8bZs4EIVD1XCF9VXs/3xZAzjA4WIycAQM
         pHVkJ8gaf44263y517/gucWFjTWgRBI2zBWEE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=Plt1O/GVmdV2ACR2U311PxlmbT6pdpZ/ksAV+erpkhBjU+WOxda8aafOvysIAnyAeE
         T19TnHSV7juSkIrzBXq6UCpw5KPvLXqicdUhLt7g33xuyov5ZUzmI2lJHphQSMZdyG1l
         aV+29lQyaxn7mQT6xbeTQu/nT9E3eD37hAaQQ=
MIME-Version: 1.0
Received: by 10.216.90.133 with SMTP id e5mr2606832wef.23.1261417369362; Mon, 
	21 Dec 2009 09:42:49 -0800 (PST)
Date: Mon, 21 Dec 2009 09:42:49 -0800
Message-ID: <1eabbac30912210942w714b1ab7x3aa4b1d4207f667f@mail.gmail.com>
Subject: InputFormat related question...
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6dab05236e25d047b409e92

--0016e6dab05236e25d047b409e92
Content-Type: text/plain; charset=ISO-8859-1

In my application I have a file in this format:

The first line of the file contains the data to be processed, and *each* of
the remaining lines contain parameters that will be used to slice & dice the
data in various ways.  In other words, each mapper needs two lines - the 1st
line from this file that contains data and another line that contains
parameters.

I looked at NLineInputFormat which can be used for "parameter sweeps", but
it's not quite what I want.  I believe this format returns N no. of
consecutive lines to the mapper, correct?

What's the best way to handle this case?  Do I have to write a special
InputFormat class?  Please help.  Thanks.

--0016e6dab05236e25d047b409e92--

From general-return-847-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Dec 21 21:07:23 2009
Return-Path: <general-return-847-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93741 invoked from network); 21 Dec 2009 21:07:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Dec 2009 21:07:23 -0000
Received: (qmail 67974 invoked by uid 500); 21 Dec 2009 21:07:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67524 invoked by uid 500); 21 Dec 2009 21:07:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67488 invoked by uid 99); 21 Dec 2009 21:07:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Dec 2009 21:07:18 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.210.194] (HELO mail-yx0-f194.google.com) (209.85.210.194)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Dec 2009 21:07:08 +0000
Received: by yxe32 with SMTP id 32so5614083yxe.5
        for <multiple recipients>; Mon, 21 Dec 2009 13:06:46 -0800 (PST)
MIME-Version: 1.0
Received: by 10.91.129.20 with SMTP id g20mr222575agn.7.1261429606731; Mon, 21 
	Dec 2009 13:06:46 -0800 (PST)
From: Christophe Bisciglia <christophe@cloudera.com>
Date: Mon, 21 Dec 2009 13:06:22 -0800
Message-ID: <69035570912211306k1eccbf79rcc02d5b6ecc4104c@mail.gmail.com>
Subject: Announcement: Cloudera Hadoop Training - Featuring Tom White and the 
	Facebook Data Team
To: common-user@hadoop.apache.org, hive-user@hadoop.apache.org, 
	pig-user@hadoop.apache.org, general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hadoop Fans, it's been a few weeks since we've hosted public training
sessions, and now we're happy to announce three sessions in three
cities over the next three months. These sessions are all targeted at
developers, and we'll be adding more sessions in the coming weeks
based on demand. You can always find a full list at
http://www.cloudera.com/hadoop-training

January 19-21, Bay Area, CA - http://www.eventbrite.com/event/511080656
February 3-5, New York, NY - http://www.eventbrite.com/event/511180956
March 10-12, Washington DC Area - http://www.eventbrite.com/event/517223028

All of these sessions offer a 10% discount for booking in advance
(timeframe varies by session), so if you register before the holidays,
you'll save a few bucks.

As some of you may have heard, Tom White, author of O'Reilly's
"Hadoop: The Definitive Guide," has recently relocated to the Bay
Area. We couldn't be happier to have him in the Cloudera office more,
and are excited to have him leading more of our local training
sessions. Tom will be leading the January 19-21 session, so if you've
found his book helpful, this is a great chance to learn more from the
source.

We're also thrilled to be working with Facebook. Many of our customers
have asked us for more exposure to how Hadoop and related technologies
are used in the "real world." As one of the largest Hadoop users, and
surely the largest Hive user, Facebook offers a unique perspective on
how to leverage huge volumes of data across an organization. During
next month's session in the Bay Area, Facebook will give a tech talk
(over lunch) on how they use Hadoop and Hive, and go into detail about
some of the cool things their data team does with the technology.
Lastly, if you're interested in working at Facebook, this is a great
chance to get more information, meet some of their most avid Hadoop
users and demonstrate you skills by taking the Cloudera Certified
Hadoop Developer exam.

Cheers,
Christophe

-- 
get hadoop: cloudera.com/hadoop
online training: cloudera.com/hadoop-training
blog: cloudera.com/blog
twitter: twitter.com/cloudera

From general-return-848-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 22 04:33:01 2009
Return-Path: <general-return-848-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55936 invoked from network); 22 Dec 2009 04:33:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Dec 2009 04:33:01 -0000
Received: (qmail 58066 invoked by uid 500); 22 Dec 2009 04:32:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58013 invoked by uid 500); 22 Dec 2009 04:32:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58003 invoked by uid 99); 22 Dec 2009 04:32:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Dec 2009 04:32:43 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE,NO_RDNS_DOTCOM_HELO
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Dec 2009 04:32:32 +0000
Received: from EGL-EX07CAS01.ds.corp.yahoo.com (egl-ex07cas01.eglbp.corp.yahoo.com [203.83.248.208])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id nBM4VrEV018253
	for <general@hadoop.apache.org>; Mon, 21 Dec 2009 20:31:54 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:in-reply-to:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=1y5HrgROHPxKhdKv/SIMby5c+FL+Q36MBhPeK0t/uBT4oInBTFqqwNPjFhvHXAdM
Received: from EGL-EX07VS01.ds.corp.yahoo.com ([203.83.248.205]) by
 EGL-EX07CAS01.ds.corp.yahoo.com ([203.83.248.215]) with mapi; Tue, 22 Dec
 2009 10:01:52 +0530
From: Amareshwari Sri Ramadasu <amarsri@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 22 Dec 2009 10:01:52 +0530
Subject: Re: InputFormat related question...
Thread-Topic: InputFormat related question...
Thread-Index: AcqCZTE+Kk7yIDiHRs+vcGmey7IrsgAWn3E1
Message-ID: <C7564990.EA0B%amarsri@yahoo-inc.com>
In-Reply-To: <1eabbac30912210942w714b1ab7x3aa4b1d4207f667f@mail.gmail.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C7564990EA0Bamarsriyahooinccom_"
MIME-Version: 1.0

--_000_C7564990EA0Bamarsriyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Hi,

If you want map task to process two lines at a time, you need to write a Re=
cordReader which constructs two lines per record. LineRecordReader makes on=
e line as one record.
You can extend NLineInputFormat for generating splits and return your new R=
ecordReader for reading records from split.
Hope this helps you.

Thanks
Amareshwari

On 12/21/09 11:12 PM, "Something Something" <mailinglists19@gmail.com> wrot=
e:

In my application I have a file in this format:

The first line of the file contains the data to be processed, and *each* of
the remaining lines contain parameters that will be used to slice & dice th=
e
data in various ways.  In other words, each mapper needs two lines - the 1s=
t
line from this file that contains data and another line that contains
parameters.

I looked at NLineInputFormat which can be used for "parameter sweeps", but
it's not quite what I want.  I believe this format returns N no. of
consecutive lines to the mapper, correct?

What's the best way to handle this case?  Do I have to write a special
InputFormat class?  Please help.  Thanks.



--_000_C7564990EA0Bamarsriyahooinccom_--

From general-return-849-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 24 00:22:48 2009
Return-Path: <general-return-849-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83705 invoked from network); 24 Dec 2009 00:22:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Dec 2009 00:22:48 -0000
Received: (qmail 52865 invoked by uid 500); 24 Dec 2009 00:22:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52781 invoked by uid 500); 24 Dec 2009 00:22:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52771 invoked by uid 99); 24 Dec 2009 00:22:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Dec 2009 00:22:46 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mailinglists19@gmail.com designates 74.125.78.24 as permitted sender)
Received: from [74.125.78.24] (HELO ey-out-2122.google.com) (74.125.78.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Dec 2009 00:22:38 +0000
Received: by ey-out-2122.google.com with SMTP id 25so559110eya.23
        for <general@hadoop.apache.org>; Wed, 23 Dec 2009 16:22:18 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=Qm9vLphRCDt8yVIhJ9QW7//mzdoST2QrREMHxRxXPKg=;
        b=XxozsfNiGT7cBsv5cdxJ9FXtfbKrEZMzPPfvxGRwrwQSKVUK1IRwaYeMGSm7R6dtW2
         7mgBf5wYmILhTUUTlXT3NVjeqURWa8/NlMEG49f9I1fCSenyLyod0wX9VywIWDiyCNIM
         3g6oDPquzHZvPjMo4nWEQojmMlH4zef1W5XUg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=xFB/mm1bRzmjthpjZxjHTQjNfx0HT45jSJdXkMYTP9IKmDFdeNpVnlCNBZG0N5RgPI
         glJJw0KN5cAywFqDo77sP0Cy73Cx2YK2QRtLc7wMHqSDAo4SjOX7h1c0PJ9ZkrqDSLsQ
         97A7fy1/EusMeNEdUED1OqR4Xsz3rHpLapKWs=
MIME-Version: 1.0
Received: by 10.216.93.69 with SMTP id k47mr3383570wef.179.1261614137812; Wed, 
	23 Dec 2009 16:22:17 -0800 (PST)
Date: Wed, 23 Dec 2009 16:22:17 -0800
Message-ID: <1eabbac30912231622v3c40e2b6kbf1de21d7755b67b@mail.gmail.com>
Subject: Type mismatch in key from map
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d78423872e92047b6e6e17
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d78423872e92047b6e6e17
Content-Type: text/plain; charset=ISO-8859-1

I would like to feed a file created by one job as an input to the next job.
 When I do that, I get:

java.io.IOException: Type mismatch in key from map: expected
org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable
 at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:807)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:504)
 at
org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124)
 at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)


The first job does:  context.write(key, value) - in a loop.  This creates a
file (<output dir>/part-r-00000) that contains something like this...

1 1,2,4*6*,1**
1 2,2,6*,4**
2 1,6,2*3*5*6*7*8*,1**
2 2,6,3*5*6*7*8*,2**
& so on...

Now in my second job I do:

FileInputFormat.addInputPath(job, new Path(inFile));

Where inFile is set to the one created above (<output dir>/part-r-00000)


What am I doing wrong?  Please help.  Thanks.

--0016e6d78423872e92047b6e6e17--

From general-return-850-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 24 02:16:40 2009
Return-Path: <general-return-850-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33176 invoked from network); 24 Dec 2009 02:16:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Dec 2009 02:16:40 -0000
Received: (qmail 8423 invoked by uid 500); 24 Dec 2009 02:16:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8332 invoked by uid 500); 24 Dec 2009 02:16:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8322 invoked by uid 99); 24 Dec 2009 02:16:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Dec 2009 02:16:39 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of zjffdu@gmail.com designates 209.85.216.201 as permitted sender)
Received: from [209.85.216.201] (HELO mail-px0-f201.google.com) (209.85.216.201)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Dec 2009 02:16:31 +0000
Received: by pxi39 with SMTP id 39so5654310pxi.2
        for <general@hadoop.apache.org>; Wed, 23 Dec 2009 18:16:10 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=D09udwOA1c4Lib02feHvDJxFMBBj8YJJ1SHUofsAr18=;
        b=wvRFIBQAxSVbtQ+KqOeUiATkiMDa5YyiHEBIH4LBnbn8uTRrHsGrStvV9en0GoEv83
         ZFpyzkLVwNUiVQLRlevSHeCyExLxXnoIyws7ZTqo0agzryDvfEjLOH+/EOylqHDSOLWj
         C7FX2n5QCtUOWWi93xzgA+uNb2FdMTdnegrgg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Iw4AFkJZWu51YZd0orgzk4R2fHo+QXzE8502bg41l+EyJv/hFqNbTKD/pmt2xBxcen
         INFANVkw6yrIbX9nDiVvpvOl4vUd+n8wcl/ogptf9mfyjlviSZSLZ8KBBWT/T7c9or8r
         56of3BufGws+mb7m90iCspyxCqWA3me5dNP7Q=
MIME-Version: 1.0
Received: by 10.142.2.24 with SMTP id 24mr7201081wfb.139.1261620969942; Wed, 
	23 Dec 2009 18:16:09 -0800 (PST)
In-Reply-To: <1eabbac30912231622v3c40e2b6kbf1de21d7755b67b@mail.gmail.com>
References: <1eabbac30912231622v3c40e2b6kbf1de21d7755b67b@mail.gmail.com>
Date: Wed, 23 Dec 2009 18:16:09 -0800
Message-ID: <8211a1320912231816s58cb3f94l6b2927e82cb48cd4@mail.gmail.com>
Subject: Re: Type mismatch in key from map
From: Jeff Zhang <zjffdu@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502a31cc13072047b700565
X-Virus-Checked: Checked by ClamAV on apache.org

--00504502a31cc13072047b700565
Content-Type: text/plain; charset=UTF-8

It seems the value type of your first job's output is Text, but I guess your
second job's InputFormat is TextInputFormat, the key type of TextInputFormat
is LongWritable. So you will get the Type mismatch error message. I suggest
you use KeyValueInputFormat as your second job's InputFormat.


Jeff Zhang


On Wed, Dec 23, 2009 at 4:22 PM, Something Something <
mailinglists19@gmail.com> wrote:

> I would like to feed a file created by one job as an input to the next job.
>  When I do that, I get:
>
> java.io.IOException: Type mismatch in key from map: expected
> org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable
>  at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:807)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:504)
>  at
>
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124)
>  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>
>
> The first job does:  context.write(key, value) - in a loop.  This creates a
> file (<output dir>/part-r-00000) that contains something like this...
>
> 1 1,2,4*6*,1**
> 1 2,2,6*,4**
> 2 1,6,2*3*5*6*7*8*,1**
> 2 2,6,3*5*6*7*8*,2**
> & so on...
>
> Now in my second job I do:
>
> FileInputFormat.addInputPath(job, new Path(inFile));
>
> Where inFile is set to the one created above (<output dir>/part-r-00000)
>
>
> What am I doing wrong?  Please help.  Thanks.
>

--00504502a31cc13072047b700565--

From general-return-851-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 24 03:31:40 2009
Return-Path: <general-return-851-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55571 invoked from network); 24 Dec 2009 03:31:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Dec 2009 03:31:40 -0000
Received: (qmail 33531 invoked by uid 500); 24 Dec 2009 03:31:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33367 invoked by uid 500); 24 Dec 2009 03:31:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33357 invoked by uid 99); 24 Dec 2009 03:31:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Dec 2009 03:31:38 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mailinglists19@gmail.com designates 209.85.219.214 as permitted sender)
Received: from [209.85.219.214] (HELO mail-ew0-f214.google.com) (209.85.219.214)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Dec 2009 03:31:30 +0000
Received: by ewy6 with SMTP id 6so5933098ewy.29
        for <general@hadoop.apache.org>; Wed, 23 Dec 2009 19:31:10 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=kxAkGucRUUzi68+1sauZ+l4Cj7Tg+qvyVdx07dX+3wM=;
        b=hcJ2pWW5S/V+2aKxPK9qTY7kUz4X19O3ubqgbGXGG99O8ce3a0sPiw83rGwFbK6va4
         zKjfXISpm3lgEu2R+ZDe0Y+GWYlUsSzn2qIqdKAnbcfKZC7dAWDT89Dy3oTCmN1QuZJ1
         +B4jpdw38v55So91znS7oiYZXIvM6tazgR9rQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=DJP+IEMGDsgCa0fwiWPMsl862ZGoD7WIuK7Jm9gpwPlFHD0rHtLM/sKAvgYGfbBRWa
         oHxUYrTuHsVmOekiE3f/jApD9sjMwBX2XmPeVYpckCH3qCZnFZ2XvFYJjJbplK2O3Z1R
         5bDsgqhVrkkznAIduDsrEDjvTEYeWiVs7OQ1E=
MIME-Version: 1.0
Received: by 10.216.90.137 with SMTP id e9mr3851494wef.141.1261625470200; Wed, 
	23 Dec 2009 19:31:10 -0800 (PST)
In-Reply-To: <8211a1320912231816s58cb3f94l6b2927e82cb48cd4@mail.gmail.com>
References: <1eabbac30912231622v3c40e2b6kbf1de21d7755b67b@mail.gmail.com>
	 <8211a1320912231816s58cb3f94l6b2927e82cb48cd4@mail.gmail.com>
Date: Wed, 23 Dec 2009 19:31:10 -0800
Message-ID: <1eabbac30912231931m19ef267bk7f5442637f9b6fe2@mail.gmail.com>
Subject: Re: Type mismatch in key from map
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d99cd7fdad23047b71119b
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d99cd7fdad23047b71119b
Content-Type: text/plain; charset=ISO-8859-1

I think you meant.. KeyValueTextInputFormat.  This is in a deprecated
package.  It even uses JobConf that's been deprecated.  Is there an
equivalent new class that's not deprecated?  Otherwise, I will have to
create the JobConf object just to use this class.

In any case, I tried using it as follows (also a few other variations...)

    JobConf jobConf = new JobConf();
    jobConf.setWorkingDirectory(new Path("myapp/output"));
    KeyValueTextInputFormat.addInputPath(jobConf, new
Path("./part-r-00000"));

But it keeps throwing this exception...

Exception in thread "main" java.io.IOException: No input paths specified in
job
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:186)
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:241)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:432)


What's the right way to use this class?  Thanks for your help.


On Wed, Dec 23, 2009 at 6:16 PM, Jeff Zhang <zjffdu@gmail.com> wrote:

> It seems the value type of your first job's output is Text, but I guess
> your
> second job's InputFormat is TextInputFormat, the key type of
> TextInputFormat
> is LongWritable. So you will get the Type mismatch error message. I suggest
> you use KeyValueInputFormat as your second job's InputFormat.
>
>
> Jeff Zhang
>
>
> On Wed, Dec 23, 2009 at 4:22 PM, Something Something <
> mailinglists19@gmail.com> wrote:
>
> > I would like to feed a file created by one job as an input to the next
> job.
> >  When I do that, I get:
> >
> > java.io.IOException: Type mismatch in key from map: expected
> > org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable
> >  at
> >
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:807)
> > at
> >
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:504)
> >  at
> >
> >
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> > at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124)
> >  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
> >  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> >
> >
> > The first job does:  context.write(key, value) - in a loop.  This creates
> a
> > file (<output dir>/part-r-00000) that contains something like this...
> >
> > 1 1,2,4*6*,1**
> > 1 2,2,6*,4**
> > 2 1,6,2*3*5*6*7*8*,1**
> > 2 2,6,3*5*6*7*8*,2**
> > & so on...
> >
> > Now in my second job I do:
> >
> > FileInputFormat.addInputPath(job, new Path(inFile));
> >
> > Where inFile is set to the one created above (<output dir>/part-r-00000)
> >
> >
> > What am I doing wrong?  Please help.  Thanks.
> >
>

--0016e6d99cd7fdad23047b71119b--

From general-return-852-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 24 04:01:57 2009
Return-Path: <general-return-852-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58513 invoked from network); 24 Dec 2009 04:01:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Dec 2009 04:01:57 -0000
Received: (qmail 40270 invoked by uid 500); 24 Dec 2009 04:01:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40181 invoked by uid 500); 24 Dec 2009 04:01:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40171 invoked by uid 99); 24 Dec 2009 04:01:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Dec 2009 04:01:55 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of zjffdu@gmail.com designates 209.85.160.50 as permitted sender)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Dec 2009 04:01:47 +0000
Received: by pwi20 with SMTP id 20so5208983pwi.29
        for <general@hadoop.apache.org>; Wed, 23 Dec 2009 20:01:26 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=/7PrIn7pw4nSdVX7cKRi9o24DOa0zm/Y+05jnw3OsQw=;
        b=pkGLjMtL19p//NvUij4wBF5vUjCDPOVOeUzyl+NmQLjoEozWbxphtnpdqWAyG1b8VJ
         ZWYZGzErnP86bkdfgizKljcfw2X9bjpmdLYKub2RGUKgV3cHn5Q+VUmFRPUAL5YMGKlq
         tMLQlIDd0Scg5JrnvaYK54eLvqkld7HK4bzZg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=eVnM3kMuE9hWyXiqx15+z+EE9XJjOcdnIcLkBM9kntrcXHkl7glWYcHfp6f5FqkCwS
         VrG6auafTPw2Cu2s0QwWDUzCJLLY8xzgRP33IpCswug69perHdK9RmxxsSjDpWi2xSmy
         NWm43tPKeTzjJmEGmfvZR31eB2Z4Hb++2j+zQ=
MIME-Version: 1.0
Received: by 10.142.61.33 with SMTP id j33mr6825136wfa.236.1261627285826; Wed, 
	23 Dec 2009 20:01:25 -0800 (PST)
In-Reply-To: <1eabbac30912231931m19ef267bk7f5442637f9b6fe2@mail.gmail.com>
References: <1eabbac30912231622v3c40e2b6kbf1de21d7755b67b@mail.gmail.com>
	 <8211a1320912231816s58cb3f94l6b2927e82cb48cd4@mail.gmail.com>
	 <1eabbac30912231931m19ef267bk7f5442637f9b6fe2@mail.gmail.com>
Date: Wed, 23 Dec 2009 20:01:25 -0800
Message-ID: <8211a1320912232001s39bd8bc1t920358d6dd9c439b@mail.gmail.com>
Subject: Re: Type mismatch in key from map
From: Jeff Zhang <zjffdu@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636e1f93d35f08f047b717e04
X-Virus-Checked: Checked by ClamAV on apache.org

--001636e1f93d35f08f047b717e04
Content-Type: text/plain; charset=UTF-8

The KeyValueTextInputFormat is deprecated in hadoop 0.20.1, and there's a
new one in trunk for new api.
I think you should use part-00000 rather than part-r-00000, then you will
get no IOException, there's output file name change between different hadoop
versions.


Jeff Zhang

On Wed, Dec 23, 2009 at 7:31 PM, Something Something <
mailinglists19@gmail.com> wrote:

> I think you meant.. KeyValueTextInputFormat.  This is in a deprecated
> package.  It even uses JobConf that's been deprecated.  Is there an
> equivalent new class that's not deprecated?  Otherwise, I will have to
> create the JobConf object just to use this class.
>
> In any case, I tried using it as follows (also a few other variations...)
>
>    JobConf jobConf = new JobConf();
>    jobConf.setWorkingDirectory(new Path("myapp/output"));
>    KeyValueTextInputFormat.addInputPath(jobConf, new
> Path("./part-r-00000"));
>
> But it keeps throwing this exception...
>
> Exception in thread "main" java.io.IOException: No input paths specified in
> job
> at
>
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:186)
> at
>
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:241)
> at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:432)
>
>
> What's the right way to use this class?  Thanks for your help.
>
>
> On Wed, Dec 23, 2009 at 6:16 PM, Jeff Zhang <zjffdu@gmail.com> wrote:
>
> > It seems the value type of your first job's output is Text, but I guess
> > your
> > second job's InputFormat is TextInputFormat, the key type of
> > TextInputFormat
> > is LongWritable. So you will get the Type mismatch error message. I
> suggest
> > you use KeyValueInputFormat as your second job's InputFormat.
> >
> >
> > Jeff Zhang
> >
> >
> > On Wed, Dec 23, 2009 at 4:22 PM, Something Something <
> > mailinglists19@gmail.com> wrote:
> >
> > > I would like to feed a file created by one job as an input to the next
> > job.
> > >  When I do that, I get:
> > >
> > > java.io.IOException: Type mismatch in key from map: expected
> > > org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable
> > >  at
> > >
> >
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:807)
> > > at
> > >
> >
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:504)
> > >  at
> > >
> > >
> >
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> > > at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124)
> > >  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> > > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
> > >  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> > > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > >
> > >
> > > The first job does:  context.write(key, value) - in a loop.  This
> creates
> > a
> > > file (<output dir>/part-r-00000) that contains something like this...
> > >
> > > 1 1,2,4*6*,1**
> > > 1 2,2,6*,4**
> > > 2 1,6,2*3*5*6*7*8*,1**
> > > 2 2,6,3*5*6*7*8*,2**
> > > & so on...
> > >
> > > Now in my second job I do:
> > >
> > > FileInputFormat.addInputPath(job, new Path(inFile));
> > >
> > > Where inFile is set to the one created above (<output
> dir>/part-r-00000)
> > >
> > >
> > > What am I doing wrong?  Please help.  Thanks.
> > >
> >
>

--001636e1f93d35f08f047b717e04--

From general-return-853-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Dec 28 07:34:48 2009
Return-Path: <general-return-853-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9925 invoked from network); 28 Dec 2009 07:34:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Dec 2009 07:34:48 -0000
Received: (qmail 63651 invoked by uid 500); 28 Dec 2009 07:34:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63547 invoked by uid 500); 28 Dec 2009 07:34:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 12832 invoked by uid 99); 28 Dec 2009 05:31:06 -0000
X-ASF-Spam-Status: No, hits=-0.8 required=5.0
	tests=BAYES_00,HTML_MESSAGE,HTML_NONELEMENT_30_40,MIME_BASE64_TEXT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of zhaohui821106@qq.com designates 119.147.10.235 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=qq.com; s=s0907;
	t=1261978234; bh=hvDP5vrsbk/HuEaRYeBAznQzxEcMZBASwCu9BCMyWxo=;
	h=X-QQ-ThreadID:X-Originating-IP:X-QQ-mid:X-QQ-STYLE:From:To:Sender:
	 Subject:Mime-Version:Content-Type:Content-Transfer-Encoding:Date:
	 X-Priority:Message-ID:X-QQ-MIME:X-Mailer:X-QQ-Mailer;
	b=hbiCikVLUkGnwRZPMdcqA41bNtrr+QYi3f83sd0Lod6+DM/zNJwr/JYRIapAFmiY3
	 Y3qgZyAKj0b4flXrxLn26Uv2j4a/5a41ftjskm4Zq0GNN6t6v+n13MqsvYjqJxD
X-QQ-ThreadID:WZR33xsnvE,0
X-Originating-IP: 61.187.225.243
X-QQ-mid:webmail374t1261978233t9717
X-QQ-STYLE:
From: "=?gbk?B?z/LI1b/7?=" <zhaohui821106@qq.com>
To: "=?gbk?B?Z2VuZXJhbEBoYWRvb3AuYXBhY2hlLm9yZw==?=" <general@hadoop.apache.org>
Sender: zhaohui821106@qq.com
Subject: about the big file!
Mime-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_NextPart_4B384279_09602518_0FA400AC"
Content-Transfer-Encoding: 8Bit
Date: Mon, 28 Dec 2009 13:30:33 +0800
X-Priority: 3
Message-ID: <tencent_71361BC02CE0362C1E33933B@qq.com>
X-QQ-MIME: TCMime 1.0 by Tencent
X-Mailer: QQMail 2.x
X-QQ-Mailer: QQMail 2.x

------=_NextPart_4B384279_09602518_0FA400AC
Content-Type: text/plain;
	charset="gbk"
Content-Transfer-Encoding: base64

LS0tLS0tLS0tLS0tLS0tLS0tINStyrzTyrz+IC0tLS0tLS0tLS0tLS0tLS0tLQ0KICC3orz+
yMs6ICLP8sjVv/siPDI2ODYyNjE1QHFxLmNvbT47DQogt6LLzcqxvOQ6IDIwMDnE6jEy1MIy
OMjVKNDHxtrSuykg1tDO5zE6MjgNCiDK1bz+yMs6ICJnZW5lcmFsIjxnZW5lcmFsQGhhZG9v
cC5hcGFjaGUub3JnPjsgDQogDQog1vfM4jogYWJvdXQgdGhlIGJpZyBmaWxlIQ0KDQogIA0K
IGhlbGxvLGkgaGF2ZSBhIHF1ZXN0aW9uIGFib3V0IHRoZSBiaWcgZmlsZXMhDQogaGRmcyBp
cyBhaW1lZCBhdCBwcm9jZXNzaW5nIGJpZyBmaWxlIUJ1dCBtb3N0IHdlYiBmaWxlIGlzIHNt
YWxsLG9mZmVuIGxlc3MgdGhhbiBhIHR5cGljYWwgYmxvY2sgc2l6ZSB3aGljaCBpcyA2NE1C
LHNvIGkgd2FudCB0byBrbm93IHdoZXJlIGRvIHRoZSBiaWcgZmlsZXMgY29tZSBmcm9tIUlm
IHRoZXkgYXJlIG1lcmdlZCBmcm9tIG1hbnkgc21hbGwgZmlsZXMgb3IgdGhlIGFjaGl2ZWQg
ZmlsZXM/DQogSSB3aXNoIHRvIGdldCB5b3VyIHJlcGx5IGFzIHNvb24gYXMgcG9zc2libGUh
ICAgVGhhbmtzIHZlcnkgbXVjaCE=

------=_NextPart_4B384279_09602518_0FA400AC--


From general-return-854-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Dec 28 07:36:29 2009
Return-Path: <general-return-854-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 10150 invoked from network); 28 Dec 2009 07:36:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Dec 2009 07:36:28 -0000
Received: (qmail 66235 invoked by uid 500); 28 Dec 2009 07:36:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66174 invoked by uid 500); 28 Dec 2009 07:36:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66164 invoked by uid 99); 28 Dec 2009 07:36:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Dec 2009 07:36:27 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ryanobjc@gmail.com designates 209.85.216.201 as permitted sender)
Received: from [209.85.216.201] (HELO mail-px0-f201.google.com) (209.85.216.201)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Dec 2009 07:36:21 +0000
Received: by pxi39 with SMTP id 39so7577708pxi.2
        for <general@hadoop.apache.org>; Sun, 27 Dec 2009 23:36:00 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=gp9VH4kRKBvSNGhPE+YxwjX7r2+aGvxyi+q84HBevuw=;
        b=QR0gYQSrzMuPpx96kBgxs/uDM+9VF9xnbOcisSJGIkd1j7/HdtPheuAsIxaannu9a9
         H3ymhP3dMg3ZjyimohNadN0jLoTISWDsn1UktSbC6YHgf5TXmpxN3XQ0lK1bWUb2n5Wi
         SjoPHb9xleB5Z4oUGNxzNDTh3+qhU+tI/wpBo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=T3HRszMWr1cS0tS12aJa/do9/+P+uzaxIax62kTI72v1XxjwyPX2fkVWFJhSMf4SdZ
         h6dBbcZoerd02kf2YLHIZtKOvnxV0wUH98VFdOcsY741DI9WPm+39jdqznakLpCxAsPR
         Q7PAP0m4P9ouzbYRLBG8xpERNMfhKLfSotJCM=
MIME-Version: 1.0
Received: by 10.115.112.5 with SMTP id p5mr9636511wam.223.1261985760458; Sun, 
	27 Dec 2009 23:36:00 -0800 (PST)
In-Reply-To: <tencent_71361BC02CE0362C1E33933B@qq.com>
References: <tencent_71361BC02CE0362C1E33933B@qq.com>
Date: Sun, 27 Dec 2009 23:36:00 -0800
Message-ID: <78568af10912272336w514c1c12n7f95efc28179bc9c@mail.gmail.com>
Subject: Re: about the big file!
From: Ryan Rawson <ryanobjc@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=GB2312
Content-Transfer-Encoding: quoted-printable

Think... logfiles.



2009/12/27 =CF=F2=C8=D5=BF=FB <zhaohui821106@qq.com>:
> ------------------ =D4=AD=CA=BC=D3=CA=BC=FE ------------------
>  =B7=A2=BC=FE=C8=CB: "=CF=F2=C8=D5=BF=FB"<26862615@qq.com>;
>  =B7=A2=CB=CD=CA=B1=BC=E4: 2009=C4=EA12=D4=C228=C8=D5(=D0=C7=C6=DA=D2=BB)=
 =D6=D0=CE=E71:28
>  =CA=D5=BC=FE=C8=CB: "general"<general@hadoop.apache.org>;
>
>  =D6=F7=CC=E2: about the big file!
>
>
>  hello,i have a question about the big files!
>  hdfs is aimed at processing big file!But most web file is small,offen le=
ss than a typical block size which is 64MB,so i want to know where do the b=
ig files come from!If they are merged from many small files or the achived =
files?
>  I wish to get your reply as soon as possible!   Thanks very much!

From general-return-855-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 30 23:21:31 2009
Return-Path: <general-return-855-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 92630 invoked from network); 30 Dec 2009 23:21:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Dec 2009 23:21:31 -0000
Received: (qmail 24153 invoked by uid 500); 30 Dec 2009 23:21:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23873 invoked by uid 500); 30 Dec 2009 23:21:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23856 invoked by uid 99); 30 Dec 2009 23:21:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Dec 2009 23:21:15 +0000
X-ASF-Spam-Status: No, hits=-5.7 required=5.0
	tests=AWL,BAYES_00,RCVD_IN_DNSWL_MED
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [137.110.142.3] (HELO stonefish.nmfs.noaa.gov) (137.110.142.3)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Dec 2009 23:21:05 +0000
Received: from [161.55.17.81] by stonefish.nmfs.noaa.gov
 (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005))
 with ESMTPSA id <0KVH00D1SNIKHL30@stonefish.nmfs.noaa.gov> for
 general@hadoop.apache.org; Wed, 30 Dec 2009 15:20:45 -0800 (PST)
Date: Wed, 30 Dec 2009 15:20:43 -0800
From: Roy Mendelssohn <Roy.Mendelssohn@noaa.gov>
Subject: What is included in each download
To: general@hadoop.apache.org
Message-id: <63C88D50-5DD1-4A39-80D0-596BF9FD7E89@noaa.gov>
MIME-version: 1.0
X-Mailer: Apple Mail (2.1077)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT

Hi All:

I am new to what is in this Apache Project. When I go t http://hadoop.apache.org/index.html, a glaring omission,  IMO, is any description of what contains what, and what I get if I do a download.  Does Hadoop common contain hbase and mapreduce and hdfs, or do I need to install these separately, or what.  I skimmed several of the starting documents, and they do not tell you that either.  Same with the other components listed on that page.

Any information on that would be greatly appreciated,

-Roy M.

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@noaa.gov (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected" 


From general-return-856-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 31 19:18:34 2009
Return-Path: <general-return-856-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 18405 invoked from network); 31 Dec 2009 19:18:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Dec 2009 19:18:34 -0000
Received: (qmail 19310 invoked by uid 500); 31 Dec 2009 19:18:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19226 invoked by uid 500); 31 Dec 2009 19:18:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19216 invoked by uid 99); 31 Dec 2009 19:18:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Dec 2009 19:18:32 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mailinglists19@gmail.com designates 74.125.78.26 as permitted sender)
Received: from [74.125.78.26] (HELO ey-out-2122.google.com) (74.125.78.26)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Dec 2009 19:18:25 +0000
Received: by ey-out-2122.google.com with SMTP id 25so1749807eya.23
        for <general@hadoop.apache.org>; Thu, 31 Dec 2009 11:18:04 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=3Z3Gjq/FYnCom1vI5IJJz7jiFiPzO+e+FcVXdcuLlXo=;
        b=btJqjgnYoRxztbEDhKqDiPABhCkrpfQV9K3KoSf4CHPzahaJ3194xuW4L0aU/da4Gw
         fvhnj80NP8ANBN5wa5IhzrQRCdyfYupuYgze9+aOupUsyIcsRXdvjs/PscBNziZYYdyC
         WKuq+DoJ0Y3FFCLP0+TLgmGEEmUd+NZbzgq/E=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=holbw8H6SoS8s2FuP4ySXpivvQahjcz1jmFFvIxEPJVnjX6LH/OBizBJJlji/veUsn
         vERLD/1R4Iut+uYlSQd7+eepcpu16qqu4aqzxdaeK984RjdjwSyM38EWjUUtw1RYyT3y
         TzjJYm2D/Ekyt/iTMELMC0taoYuUxlm/D0zQ0=
MIME-Version: 1.0
Received: by 10.216.85.7 with SMTP id t7mr128618wee.122.1262287084025; Thu, 31 
	Dec 2009 11:18:04 -0800 (PST)
In-Reply-To: <8211a1320912232001s39bd8bc1t920358d6dd9c439b@mail.gmail.com>
References: <1eabbac30912231622v3c40e2b6kbf1de21d7755b67b@mail.gmail.com>
	 <8211a1320912231816s58cb3f94l6b2927e82cb48cd4@mail.gmail.com>
	 <1eabbac30912231931m19ef267bk7f5442637f9b6fe2@mail.gmail.com>
	 <8211a1320912232001s39bd8bc1t920358d6dd9c439b@mail.gmail.com>
Date: Thu, 31 Dec 2009 11:18:03 -0800
Message-ID: <1eabbac30912311118n484c1e73xf6139dace6f3ee27@mail.gmail.com>
Subject: Re: Type mismatch in key from map
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d77cce3f7a82047c0b1d16

--0016e6d77cce3f7a82047c0b1d16
Content-Type: text/plain; charset=ISO-8859-1

I tried using KeyValueTextInputFormat from the 'trunk' but I am getting the
same error message (Type mismatch in key from map: expected
org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable)

Not sure what you mean by "use part-00000".  There's no file called
part-00000.

Anyway, for now I will just workaround this issue.  New plan is this....
 the first job will write to a new table in HBase and the 2nd job will use
TableMapper to go thru each row.  I was trying to avoid this because
something tells me that this might be a bit slower than just using HDFS
directly, but we shall see.  Thanks.



On Wed, Dec 23, 2009 at 8:01 PM, Jeff Zhang <zjffdu@gmail.com> wrote:

> The KeyValueTextInputFormat is deprecated in hadoop 0.20.1, and there's a
> new one in trunk for new api.
> I think you should use part-00000 rather than part-r-00000, then you will
> get no IOException, there's output file name change between different
> hadoop
> versions.
>
>
> Jeff Zhang
>
> On Wed, Dec 23, 2009 at 7:31 PM, Something Something <
> mailinglists19@gmail.com> wrote:
>
> > I think you meant.. KeyValueTextInputFormat.  This is in a deprecated
> > package.  It even uses JobConf that's been deprecated.  Is there an
> > equivalent new class that's not deprecated?  Otherwise, I will have to
> > create the JobConf object just to use this class.
> >
> > In any case, I tried using it as follows (also a few other variations...)
> >
> >    JobConf jobConf = new JobConf();
> >    jobConf.setWorkingDirectory(new Path("myapp/output"));
> >    KeyValueTextInputFormat.addInputPath(jobConf, new
> > Path("./part-r-00000"));
> >
> > But it keeps throwing this exception...
> >
> > Exception in thread "main" java.io.IOException: No input paths specified
> in
> > job
> > at
> >
> >
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:186)
> > at
> >
> >
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:241)
> > at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
> > at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
> > at org.apache.hadoop.mapreduce.Job.submit(Job.java:432)
> >
> >
> > What's the right way to use this class?  Thanks for your help.
> >
> >
> > On Wed, Dec 23, 2009 at 6:16 PM, Jeff Zhang <zjffdu@gmail.com> wrote:
> >
> > > It seems the value type of your first job's output is Text, but I guess
> > > your
> > > second job's InputFormat is TextInputFormat, the key type of
> > > TextInputFormat
> > > is LongWritable. So you will get the Type mismatch error message. I
> > suggest
> > > you use KeyValueInputFormat as your second job's InputFormat.
> > >
> > >
> > > Jeff Zhang
> > >
> > >
> > > On Wed, Dec 23, 2009 at 4:22 PM, Something Something <
> > > mailinglists19@gmail.com> wrote:
> > >
> > > > I would like to feed a file created by one job as an input to the
> next
> > > job.
> > > >  When I do that, I get:
> > > >
> > > > java.io.IOException: Type mismatch in key from map: expected
> > > > org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable
> > > >  at
> > > >
> > >
> >
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:807)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:504)
> > > >  at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> > > > at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124)
> > > >  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> > > > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
> > > >  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> > > > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > > >
> > > >
> > > > The first job does:  context.write(key, value) - in a loop.  This
> > creates
> > > a
> > > > file (<output dir>/part-r-00000) that contains something like this...
> > > >
> > > > 1 1,2,4*6*,1**
> > > > 1 2,2,6*,4**
> > > > 2 1,6,2*3*5*6*7*8*,1**
> > > > 2 2,6,3*5*6*7*8*,2**
> > > > & so on...
> > > >
> > > > Now in my second job I do:
> > > >
> > > > FileInputFormat.addInputPath(job, new Path(inFile));
> > > >
> > > > Where inFile is set to the one created above (<output
> > dir>/part-r-00000)
> > > >
> > > >
> > > > What am I doing wrong?  Please help.  Thanks.
> > > >
> > >
> >
>

--0016e6d77cce3f7a82047c0b1d16--

From general-return-2094-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Sep 01 00:05:28 2010
Return-Path: <general-return-2094-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 71580 invoked from network); 1 Sep 2010 00:05:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Sep 2010 00:05:28 -0000
Received: (qmail 36314 invoked by uid 500); 1 Sep 2010 00:05:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36173 invoked by uid 500); 1 Sep 2010 00:05:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36165 invoked by uid 99); 1 Sep 2010 00:05:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Sep 2010 00:05:26 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of hc.busy@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Sep 2010 00:05:22 +0000
Received: by qwk3 with SMTP id 3so7196943qwk.35
        for <general@hadoop.apache.org>; Tue, 31 Aug 2010 17:05:01 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=53xkH/W1LHy8bL9hJxGiOdIDWbcXUj8Y+/2WwRDWRQ8=;
        b=l4lzgVZKUVmlYZxR+cjzIdnJSvH7CDkH4+NKw2m5l0YVrr9RS1Pk/be69NclNkIBl9
         FZ/BpgqCxMmsBaMe7so46ejPaTq3gwqV0VND+7xfegQoDFnUyhLKebn8q8PY1ILVy0s8
         z2LVLRcj1z49MfOANiMXlcoAmyEoiz0AIu59c=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=cHwvu04fdXHpGcNFnVEu3yTLeD6RVRrGgqV0xvJHOt9YP7Jreg/7eEp0SawwSyv5sJ
         uGsswaCPHi3XD1XyD33Ip5pgysEnjNNynAC3WjHhun9GR/oXCTkgu/bMcALx8axvDk+Z
         Y9tQqGOyYVfn/q8KT6yHEUQh552W1EcyD01g8=
MIME-Version: 1.0
Received: by 10.229.223.198 with SMTP id il6mr4789379qcb.50.1283299501241;
 Tue, 31 Aug 2010 17:05:01 -0700 (PDT)
Received: by 10.229.40.140 with HTTP; Tue, 31 Aug 2010 17:05:01 -0700 (PDT)
In-Reply-To: <351A8FBE-528C-4352-8D14-3AC913836336@yahoo-inc.com>
References: <71D87762-A95B-434D-9FF6-8627AF65790A@yahoo-inc.com>
	<71E1E01A-5389-48C2-9A43-9A8BD6B87A2A@yahoo-inc.com>
	<C1EDB62C3771D74AA823DBBBBFB3F89D4E5DE391@SP2-EX07VS03.ds.corp.yahoo.com>
	<AANLkTin1=Za+kPAy_97O8zcLxbA+U54-nHaje_44yMsj@mail.gmail.com>
	<AANLkTikmA=T=0URfdztx7PYVWN-kjxGNCSYMQSh=JRAp@mail.gmail.com>
	<351A8FBE-528C-4352-8D14-3AC913836336@yahoo-inc.com>
Date: Tue, 31 Aug 2010 17:05:01 -0700
Message-ID: <AANLkTinHsBCjC9zpaN6RfxyREBXnSzas3YTGT4wx0a8M@mail.gmail.com>
Subject: Re: [VOTE] Pig to become a TLP
From: hc busy <hc.busy@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163630eb25e97b8c048f2772e3

--00163630eb25e97b8c048f2772e3
Content-Type: text/plain; charset=ISO-8859-1

Neat!

Congratulations, again, on the solidarity !

I feel that Pig is one of the more reliable and mature ways to program
Hadoop. I look forward to seeing more a$$ kicking of all the other systems.

:-)

On Tue, Aug 31, 2010 at 2:41 PM, Alan Gates <gates@yahoo-inc.com> wrote:

>
> On Aug 30, 2010, at 1:41 PM, hc busy wrote:
>
>  +1, Congrads on the near unanimous vote for progress.
>>
>> Btw, can somebody tell me what the change was from March when this was
>> discussed and now? At that time, every one seem to have felt that Pig can
>> only handle Hadoop as backend and doesn't make sense for it to become a
>> TLP
>> by itself. In fact, I remember a ticket to remove the possibility of a
>> different backend. What did I miss?
>>
>
> I can only speak for myself, but in the original discussion thread on
> pig-dev (http://bit.ly/byD7L8 ) I discussed why I changed my vote over the
> course of the last few months.
>
>
>
>>
>> 1.) Technically, what developments/or future plans make it a separate
>> project from Hadoop? (Other backend support?)
>>
>
> While we still want to design Pig and Pig Latin in such a way that there
> may be separate implementations that do not run on Hadoop, nothing in this
> regard has changed in the last few months.  What did change is that I (and I
> think others) realized that being connected technically to Hadoop does not
> mean we need to be connected in our governance.
>
>
>  2.) What are the non-technical factors that changed since March that made
>> this vote possible? (Donation from large company?)
>>
>
> Apache does not require that projects raise any finances.  Pig is (as it
> has been) supported by the contributions of those choose to work on it and
> those who are paid to work on it.
>
>
>  3.)  Can I make tax deductible contributions to the Apache Pig Project ?
>>
>
> Wow, that's a huge vote of confidence.  Thanks.  But no.  Contributions to
> Pig can be made (as before) in code, documentation, user feedback, etc.  You
> can donate money to Apache itself, which goes to support the infrastructure
> for Pig and all Apache projects.  For more info see
> http://apache.org/foundation/support-apache.html  (I have no idea if that
> is tax deductible or not.)
>
> Alan.
>
>

--00163630eb25e97b8c048f2772e3--

From general-return-2095-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Sep 01 02:17:56 2010
Return-Path: <general-return-2095-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 15510 invoked from network); 1 Sep 2010 02:17:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Sep 2010 02:17:55 -0000
Received: (qmail 18871 invoked by uid 500); 1 Sep 2010 02:17:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18761 invoked by uid 500); 1 Sep 2010 02:17:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18751 invoked by uid 99); 1 Sep 2010 02:17:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Sep 2010 02:17:54 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of Deepika.Khera@avg.com designates 193.85.188.248 as permitted sender)
Received: from [193.85.188.248] (HELO ms.grisoft.cz) (193.85.188.248)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Sep 2010 02:17:48 +0000
Received: from deimos.cz.avg.com (unknown [192.168.200.161])
	(using TLSv1 with cipher AES128-SHA (128/128 bits))
	(No client certificate requested)
	by ms.grisoft.cz (Postfix) with ESMTP id 5E66F382C8
	for <general@hadoop.apache.org>; Wed,  1 Sep 2010 04:17:27 +0200 (CEST)
Received: from miranda.us.avg.com (10.32.34.13) by deimos.cz.avg.com
 (192.168.200.161) with Microsoft SMTP Server (TLS) id 8.3.83.0; Wed, 1 Sep
 2010 04:17:27 +0200
Received: from miranda.us.avg.com ([10.32.34.13]) by miranda ([10.32.34.13])
 with mapi; Tue, 31 Aug 2010 22:17:25 -0400
From: Deepika Khera <Deepika.Khera@avg.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 31 Aug 2010 22:17:26 -0400
Subject: RE: Deploying my job jar on hadoop cluster
Thread-Topic: Deploying my job jar on hadoop cluster
Thread-Index: ActIH0J+vIZx/8HOQvuTgGChZ4b1ZABXHc2w
Message-ID: <DA852C02397DBC4EBFAA1B7192FBD15062025E5C78@miranda>
References: <DA852C02397DBC4EBFAA1B7192FBD15062025E5AC2@miranda>
	<AANLkTi=-dcxp8VVLC8ESpOtLb-X9DNdXr0oPw-KQ9WeJ@mail.gmail.com>
	<DA852C02397DBC4EBFAA1B7192FBD15062025E5AC5@miranda>
 <AANLkTinydd1fTW=xteCBS-8y+w2zbB+Gjb+NYfvjOuuZ@mail.gmail.com>
In-Reply-To: <AANLkTinydd1fTW=xteCBS-8y+w2zbB+Gjb+NYfvjOuuZ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks! Added the line and it worked :)

-----Original Message-----
From: Angus He [mailto:angushe@gmail.com]=20
Sent: Monday, August 30, 2010 1:41 AM
To: general@hadoop.apache.org
Subject: Re: Deploying my job jar on hadoop cluster

Did the Job.setJarByClass get called?  If not, just add it and take
another try.

On Sun, Aug 29, 2010 at 2:15 PM, Deepika Khera <Deepika.Khera@avg.com> wrot=
e:
> Thanks. I had tried running the command as you suggested but got a ClassN=
otFoundException exception(for the mapper class) in the task trackers. I di=
d a
>
> bin/hadoop jar /root/myjob.jar
>
> (Already had the main class defined in manifest)
>
> There is probably something else that I am doing wrong.
>
> Appreciate your help.
>
> Thanks,
> Deepika
>
> -----Original Message-----
> From: Chandraprakash Bhagtani [mailto:cpbhagtani@gmail.com]
> Sent: Saturday, August 28, 2010 10:40 PM
> To: general@hadoop.apache.org
> Subject: Re: Deploying my job jar on hadoop cluster
>
> Deepika,
>
> You just have to run the following command on any of the cluster node
>
> HADOOP_HOME/bin/hadoop jar <job_jar_name> <class_name> <other_args>
>
> this command will automatically copy the jar on all the tasktrackers.
>
> On Sun, Aug 29, 2010 at 6:07 AM, Deepika Khera <Deepika.Khera@avg.com>wro=
te:
>
>> Hi,
>>
>> I want to deploy my map reduce job jar on the Hadoop cluster. I've alway=
s
>> done that by doing the following -
>>
>> 1. Copying the job jar to all datanodes
>> 2. Having the job jar on the hadoop classpath on all machines.
>>
>> Isn't hadoop capable of copying over the job jar to all machines in the
>> cluster? This is what I read (that job tracker copies the job jar, etc) =
,
>> but if I don't do the above the task trackers cannot find the job. I kno=
w I
>> am missing something.
>>
>> Could someone please let me know how I can run my job without having to
>> copy it over all machines in the cluster?
>>
>> Thanks,
>> Deepika
>>
>
>
>
> --
> Thanks & Regards,
> Chandra Prakash Bhagtani,
> Nokia India Pvt. Ltd.
>



--=20
Regards
Angus

From general-return-2096-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Sep 02 18:54:28 2010
Return-Path: <general-return-2096-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16141 invoked from network); 2 Sep 2010 18:54:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Sep 2010 18:54:28 -0000
Received: (qmail 3038 invoked by uid 500); 2 Sep 2010 18:54:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2932 invoked by uid 500); 2 Sep 2010 18:54:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2924 invoked by uid 99); 2 Sep 2010 18:54:26 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Sep 2010 18:54:26 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of palobob@gmail.com designates 209.85.210.48 as permitted sender)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Sep 2010 18:54:04 +0000
Received: by pzk4 with SMTP id 4so449163pzk.35
        for <general@hadoop.apache.org>; Thu, 02 Sep 2010 11:53:42 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=BsbMVB+GzVw7JnDS76/ZLXPznNj924W9ufiGl05LG8s=;
        b=ZHxZ0fdUYlN/qI9eixqyvrqA2M7VE4CirBW4X3FoJ2BSKvokI802jvsB/PL/kpeZdF
         KQ3g5E2bFShp2uU4tPn2YKPZiebyaydAW/cZJauCK5Fa7qaiXMz3cBnenCrFRsxv1eEL
         c2QUQNWlHcbKz5qbeI0Y41h48WNlGGbD/Yj3M=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=cAFcKNWddzXUvqZjLjaVHEBTL66cxxCc419pGh8u0YfY29EIBeWzki3MJrZUw790mP
         tTiIhSEZBifrza/KpRRB5UjYH0nilku8XtVOB4/+sEyLlUwlaN8wZjS4ZAkjR6cOhbi4
         8n8ssVX7+QgoXcerRgaBrMDphYFWgk6ykGq7k=
MIME-Version: 1.0
Received: by 10.114.131.6 with SMTP id e6mr85664wad.90.1283453610502; Thu, 02
 Sep 2010 11:53:30 -0700 (PDT)
Received: by 10.142.106.7 with HTTP; Thu, 2 Sep 2010 11:53:30 -0700 (PDT)
In-Reply-To: <AANLkTimJ=0guMqK4uqdtykJMKd1DMPxHfqEMD3q-Ane4@mail.gmail.com>
References: <201008241143.46918.thomas@koch.ro>
	<4C739DD0.8050502@apache.org>
	<AANLkTimJ=0guMqK4uqdtykJMKd1DMPxHfqEMD3q-Ane4@mail.gmail.com>
Date: Thu, 2 Sep 2010 14:53:30 -0400
Message-ID: <AANLkTi=voOoAxYAp2FpOWohqwSpwzMXVv-m_5zfcFfy=@mail.gmail.com>
Subject: Re: apache commons configuration
From: Bob Li <palobob@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364c5c0d8afffe048f4b5407
X-Virus-Checked: Checked by ClamAV on apache.org

--0016364c5c0d8afffe048f4b5407
Content-Type: text/plain; charset=ISO-8859-1

Hi experts,

For hadoop 0.20.2, we have following task tracker configuration
 <property>
    <name>mapred.local.dir</name>
    <value>/data/mapred/local</value>
  </property>

It seems not what I expected -- the directory /tmp/hadoop-user/mapred/local
was used.

Any suggestion?

Thanks,
Bob

On Wed, Aug 25, 2010 at 4:34 AM, Jeff Hammerbacher <hammer@cloudera.com>wrote:

> For some recent discussion, see
> https://issues.apache.org/jira/browse/HADOOP-6910
>
> On Tue, Aug 24, 2010 at 3:24 AM, Steve Loughran <stevel@apache.org> wrote:
>
> > On 24/08/10 10:43, Thomas Koch wrote:
> >
> >> Hi,
> >>
> >> just out of curiosity: Is there any particular reason, why Hadoop
> projects
> >> or
> >> ZooKeeper do not use apache commons configuration[1]?
> >> I'm just evaluating it for an internal project. Would you recommend it
> or
> >> not?
> >> I imagine, that it could help to make projects configurable either via
> XML
> >> or
> >> INI files. Some people may prefer one or the other.
> >>
> >> [1] http://commons.apache.org/configuration/
> >>
> >
> > Someone's discussed it, I'm against because (a) there's a lot of
> historical
> > configuration stuff in there, (b) ini files are less flexible than XML,
> (c)
> > neither of them are very dynamic. when the NN goes down and you want the
> DNs
> > to reconnect to a different host, how do you do that when all data is
> read
> > at start time?
> >
> >
> >
>

--0016364c5c0d8afffe048f4b5407--

From general-return-2097-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Sep 02 19:03:28 2010
Return-Path: <general-return-2097-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 18274 invoked from network); 2 Sep 2010 19:03:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Sep 2010 19:03:28 -0000
Received: (qmail 13085 invoked by uid 500); 2 Sep 2010 19:03:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13027 invoked by uid 500); 2 Sep 2010 19:03:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13017 invoked by uid 99); 2 Sep 2010 19:03:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Sep 2010 19:03:26 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Sep 2010 19:03:22 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version;
  b=QWlVDSxM5NrbKFRB9vuSGD637/HQChPMGJ+v8toiRmrFq7eAOv9PzX6V
   B9wns6vnGy0/pOxvKK/LwcEX4df+SoppPuFRT6VOVUMpO5+f4XTghQxyq
   4TVGLZeWhxDTW6c;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1283454202; x=1314990202;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20apache=20commons=20configuration|Date:
   =20Thu,=202=20Sep=202010=2019:03:00=20+0000|Message-ID:
   =20<F01C7F56-C2F2-4E96-A6C7-E59D79693152@linkedin.com>
   |To:=20"<general@hadoop.apache.org>"=20<general@hadoop.ap
   ache.org>|MIME-Version:=201.0|Content-Transfer-Encoding:
   =20quoted-printable|Content-ID:=20<523ae60e-5eb4-44b4-bd8
   2-691f62891255>|In-Reply-To:=20<AANLkTi=3DvoOoAxYAp2FpOWo
   hqwSpwzMXVv-m_5zfcFfy=3D@mail.gmail.com>|References:=20<2
   01008241143.46918.thomas@koch.ro>=0D=0A=20<4C739DD0.80505
   02@apache.org>=0D=0A=20<AANLkTimJ=3D0guMqK4uqdtykJMKd1DMP
   xHfqEMD3q-Ane4@mail.gmail.com>=0D=0A=20<AANLkTi=3DvoOoAxY
   Ap2FpOWohqwSpwzMXVv-m_5zfcFfy=3D@mail.gmail.com>;
  bh=zjbU8KKqcb+aeGJ4qNZUAJBdhPfGZb1ckt+V8UlRSQc=;
  b=r2YJwvKThcv6qOMwQCPsr0iSKpsziLtNvH4GUcTovi+VDtTh7fETD1it
   RXwe7GudHPmbQfjhj+oc6OrvVFtPkIDGPVrRftCmDWJ3iFBJC4ozICJpp
   wva3uJcBMQjgSEP;
X-IronPort-AV: E=Sophos;i="4.56,309,1280732400"; 
   d="scan'208";a="15106475"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi; Thu, 2 Sep 2010 12:03:01
 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: apache commons configuration
Thread-Topic: apache commons configuration
Thread-Index: AQHLQ3EZ+5DRkOEu4Uyw2bXTXlOm8JLw23gAgAFznYCADT+mAIAAAqcA
Date: Thu, 2 Sep 2010 19:03:00 +0000
Message-ID: <F01C7F56-C2F2-4E96-A6C7-E59D79693152@linkedin.com>
References: <201008241143.46918.thomas@koch.ro>
 <4C739DD0.8050502@apache.org>
 <AANLkTimJ=0guMqK4uqdtykJMKd1DMPxHfqEMD3q-Ane4@mail.gmail.com>
 <AANLkTi=voOoAxYAp2FpOWohqwSpwzMXVv-m_5zfcFfy=@mail.gmail.com>
In-Reply-To: <AANLkTi=voOoAxYAp2FpOWohqwSpwzMXVv-m_5zfcFfy=@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Content-Type: text/plain; charset="us-ascii"
Content-ID: <523ae60e-5eb4-44b4-bd82-691f62891255>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


On Sep 2, 2010, at 11:53 AM, Bob Li wrote:

> Hi experts,
>=20
> For hadoop 0.20.2, we have following task tracker configuration
> <property>
>    <name>mapred.local.dir</name>
>    <value>/data/mapred/local</value>
>  </property>
>=20
> It seems not what I expected -- the directory /tmp/hadoop-user/mapred/loc=
al
> was used.
>=20
> Any suggestion?


Config problems like this seem to usually be traced back to broken XML form=
atting prior to this entry or the property being in the wrong file (in this=
 case, it should be in mapred-site.xml).




From general-return-2098-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 07 13:53:56 2010
Return-Path: <general-return-2098-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76254 invoked from network); 7 Sep 2010 13:53:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Sep 2010 13:53:56 -0000
Received: (qmail 68830 invoked by uid 500); 7 Sep 2010 13:53:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68459 invoked by uid 500); 7 Sep 2010 13:53:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68451 invoked by uid 99); 7 Sep 2010 13:53:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Sep 2010 13:53:51 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [156.148.72.33] (HELO raffaello.crs4.it) (156.148.72.33)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Sep 2010 13:53:45 +0000
Received: from [156.148.71.202] (neuron.crs4.it [156.148.71.202])
	by raffaello.crs4.it (Postfix) with ESMTP id 6A74F6BB9F
	for <general@hadoop.apache.org>; Tue,  7 Sep 2010 15:51:42 +0200 (CEST)
Message-ID: <4C86433A.6020703@crs4.it>
Date: Tue, 07 Sep 2010 15:50:50 +0200
From: Simone Leo <simone.leo@crs4.it>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.7) Gecko/20100727 Thunderbird/3.1.1
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Pydoop 0.3.6 released
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit

Hello everybody,

We are pleased to announce that Pydoop (http://pydoop.sourceforge.net)
0.3.6 is out.

Pydoop is a Python MapReduce and HDFS API for Hadoop, built upon the C++
Pipes and the C libhdfs APIs, that allows to write full-fledged
MapReduce applications with HDFS access.

The main changes with respect to 0.3.6_rc2 are the following:

 * Pydoop is now released under the Apache License, Version 2.0
 * A "Pydoop for Dumbo Users" section has been added to the docs
 * Several bugs have been fixed

Links:

 * download page: http://sourceforge.net/projects/pydoop/files
 * full release notes:
http://sourceforge.net/apps/mediawiki/pydoop/index.php?title=Release_Notes

Happy pydooping,

Simone

-- 
Simone Leo
Distributed Computing group
Advanced Computing and Communications program
CRS4
POLARIS - Building #1
Piscina Manna
I-09010 Pula (CA) - Italy
e-mail: simleo@crs4.it
http://www.crs4.it

From general-return-2099-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Sep 08 21:49:37 2010
Return-Path: <general-return-2099-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 65748 invoked from network); 8 Sep 2010 21:49:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Sep 2010 21:49:36 -0000
Received: (qmail 43680 invoked by uid 500); 8 Sep 2010 21:49:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 43526 invoked by uid 500); 8 Sep 2010 21:49:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 43518 invoked by uid 99); 8 Sep 2010 21:49:34 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Sep 2010 21:49:34 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [212.122.43.183] (HELO metis.rz1.connectline.net) (212.122.43.183)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Sep 2010 21:49:10 +0000
Received: from metis.rz1.connectline.net (localhost [127.0.0.1])
	by metis.rz1.connectline.net (Postfix) with ESMTP id 536ECE128
	for <general@hadoop.apache.org>; Wed,  8 Sep 2010 23:48:50 +0200 (CEST)
Received: from zeus.rz1.connectline.net (zeus.rz1.connectline.net [212.122.43.181])
	by metis.rz1.connectline.net (Postfix) with ESMTP
	for <general@hadoop.apache.org>; Wed,  8 Sep 2010 23:48:50 +0200 (CEST)
Received: from mail.connectline.de (mail.connectline.de [212.122.43.178])
	by zeus.rz1.connectline.net (Postfix) with ESMTP id 5E432380014
	for <general@hadoop.apache.org>; Wed,  8 Sep 2010 23:48:49 +0200 (CEST)
From: Teresa Wingfield <tw@datameer.com>
Content-Type: multipart/alternative; boundary=Apple-Mail-19-732647816
Subject: Support Engineer Job Opening at Datameer
Date: Wed, 8 Sep 2010 14:48:45 -0700
Message-Id: <8FB856F1-B447-41E0-9C3C-E3C67EDBA553@datameer.com>
To: general@hadoop.apache.org
Mime-Version: 1.0 (Apple Message framework v1081)
X-Mailer: Apple Mail (2.1081)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-19-732647816
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

Datameer, a provider of big data analytics built on Apache Hadoop, is =
looking for a Support Engineer to provide technical assistance to =
customers and to help diagnose and resolve customer support requests. If =
you're interested in big data analytics and Hadoop, have strong =
technical, communication and troubleshooting skills, and are passionate =
about making customers successful, we would like to talk to you.

This position is located in San Mateo, CA.  If interested, please visit =
http://www.datameer.com/about/careers.html to obtain a detailed job =
description and application information.=20


Sincerely,
Teresa Wingfield
Vice President, Marketing
Datameer
650-286-9100=

--Apple-Mail-19-732647816--

From general-return-2100-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 13 14:07:56 2010
Return-Path: <general-return-2100-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1220 invoked from network); 13 Sep 2010 14:07:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Sep 2010 14:07:55 -0000
Received: (qmail 81524 invoked by uid 500); 13 Sep 2010 14:07:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81026 invoked by uid 500); 13 Sep 2010 14:07:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81005 invoked by uid 99); 13 Sep 2010 14:07:50 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Sep 2010 14:07:50 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [85.214.115.60] (HELO h1349259.stratoserver.net) (85.214.115.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Sep 2010 14:07:26 +0000
Received: from crt-01-tr.neofonie.de ([91.213.91.28] helo=essen.neofonie.priv)
	by h1349259.stratoserver.net with esmtpsa (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16)
	(Exim 4.69)
	(envelope-from <isabel@apache.org>)
	id 1Ov9gX-0003Bd-Ap
	for general@hadoop.apache.org; Mon, 13 Sep 2010 14:07:05 +0000
Date: Mon, 13 Sep 2010 16:01:49 +0200
From: Isabel Drost <isabel@apache.org>
To: general@hadoop.apache.org
Subject: Re: Support Engineer Job Opening at Datameer
Message-ID: <20100913160149.77251ea3@essen.neofonie.priv>
In-Reply-To: <8FB856F1-B447-41E0-9C3C-E3C67EDBA553@datameer.com>
References: <8FB856F1-B447-41E0-9C3C-E3C67EDBA553@datameer.com>
X-Mailer: Claws Mail 3.7.2 (GTK+ 2.14.4; x86_64-http://packman.links2linux.de-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org


Hello Teresa,


On Wed, 8 Sep 2010 Teresa Wingfield <tw@datameer.com> wrote:
> Datameer, a provider of big data analytics built on Apache Hadoop, is
> looking for a Support Engineer to provide technical assistance to
> customers and to help diagnose and resolve customer support requests.
> If you're interested in big data analytics and Hadoop, have strong
> technical, communication and troubleshooting skills, and are
> passionate about making customers successful, we would like to talk
> to you.

Just a hint - are you aware of the jobs@apache.org mailing list? Maybe
it makes sense to post the job offer there as well?


Cheers,
Isabel

From general-return-2101-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 13 14:25:25 2010
Return-Path: <general-return-2101-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5933 invoked from network); 13 Sep 2010 14:25:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Sep 2010 14:25:24 -0000
Received: (qmail 13008 invoked by uid 500); 13 Sep 2010 14:25:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12558 invoked by uid 500); 13 Sep 2010 14:25:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12550 invoked by uid 99); 13 Sep 2010 14:25:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Sep 2010 14:25:20 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of david.g.boney@gmail.com designates 209.85.213.48 as permitted sender)
Received: from [209.85.213.48] (HELO mail-yw0-f48.google.com) (209.85.213.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Sep 2010 14:25:15 +0000
Received: by ywf9 with SMTP id 9so2749076ywf.35
        for <general@hadoop.apache.org>; Mon, 13 Sep 2010 07:24:54 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:content-type:mime-version
         :subject:from:in-reply-to:date:content-transfer-encoding:message-id
         :references:to:x-mailer;
        bh=rUZjowNgaLm8zU+pAemgQ9km+HnbrGFVJQVFq6Ix4rg=;
        b=TXkJiNk0n9OgtDvouXAhxWYuhhajdXpw/ULf7khWQmLDoropp75uIrGgMF5Ui7cgx5
         5vHEUSPTUXLPqB7nK4BnbzgeKhVLJDyZ1RTPR9QzN7mAheNtCD/U+S6P+oGgYF0m035c
         d5zHYLB4NywWgbooQ/WVaDCeCpn/XEq1UQdz0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=mJZRZekJaH8GwyICfm+Z4Tr3l8U8PR6PY/MiDkCFtBCAGECmLmsTrCpJ0oENZoMaYg
         T8vz1GYiKuDKrYfhKC31/E/VOGJCXGa0ZE6uEz1XD65NvbyCSs00W2oz5EmiP06iD554
         WprFZLTCkSwxAx979hhdLdNALb9fFd0d/iPDg=
Received: by 10.151.148.18 with SMTP id a18mr1299276ybo.299.1284387894054;
        Mon, 13 Sep 2010 07:24:54 -0700 (PDT)
Received: from [192.168.0.230] ([64.129.157.190])
        by mx.google.com with ESMTPS id w24sm3429250ybk.1.2010.09.13.07.24.52
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Mon, 13 Sep 2010 07:24:53 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1081)
Subject: Re: Support Engineer Job Opening at Datameer
From: "David G. Boney" <david.g.boney@gmail.com>
In-Reply-To: <20100913160149.77251ea3@essen.neofonie.priv>
Date: Mon, 13 Sep 2010 09:24:50 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <CE7ADB2E-312E-4134-9292-7DA567C18A3D@gmail.com>
References: <8FB856F1-B447-41E0-9C3C-E3C67EDBA553@datameer.com> <20100913160149.77251ea3@essen.neofonie.priv>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1081)

Would it be possible to have a mailing list for hadoop jobs, a =
jobs@hadoop.apache.org? My observation is that the skills for using =
Hadoop for big data analysis are more specialized, particularly in math, =
stat, and machine learning.
-------------
Sincerely,
David G. Boney
david.g.boney@gmail.com
http://www.sonicartifacts.com




On Sep 13, 2010, at 9:01 AM, Isabel Drost wrote:

>=20
> Hello Teresa,
>=20
>=20
> On Wed, 8 Sep 2010 Teresa Wingfield <tw@datameer.com> wrote:
>> Datameer, a provider of big data analytics built on Apache Hadoop, is
>> looking for a Support Engineer to provide technical assistance to
>> customers and to help diagnose and resolve customer support requests.
>> If you're interested in big data analytics and Hadoop, have strong
>> technical, communication and troubleshooting skills, and are
>> passionate about making customers successful, we would like to talk
>> to you.
>=20
> Just a hint - are you aware of the jobs@apache.org mailing list? Maybe
> it makes sense to post the job offer there as well?
>=20
>=20
> Cheers,
> Isabel


From general-return-2102-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 13 16:09:04 2010
Return-Path: <general-return-2102-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53940 invoked from network); 13 Sep 2010 16:09:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Sep 2010 16:09:03 -0000
Received: (qmail 88582 invoked by uid 500); 13 Sep 2010 16:09:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88503 invoked by uid 500); 13 Sep 2010 16:09:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88495 invoked by uid 99); 13 Sep 2010 16:09:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Sep 2010 16:09:01 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Sep 2010 16:08:55 +0000
Received: by yxn22 with SMTP id 22so2799945yxn.35
        for <general@hadoop.apache.org>; Mon, 13 Sep 2010 09:08:34 -0700 (PDT)
Received: by 10.100.232.1 with SMTP id e1mr4585033anh.15.1284394113857; Mon,
 13 Sep 2010 09:08:33 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.156.212 with HTTP; Mon, 13 Sep 2010 09:08:13 -0700 (PDT)
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 13 Sep 2010 09:08:13 -0700
Message-ID: <AANLkTiknJ_SzRux7KhjhxVfUU9FBkNgvYnkpbz3G_+a4@mail.gmail.com>
Subject: hadoop.job.ugi backwards compatibility
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016368e1f63e877560490264e59

--0016368e1f63e877560490264e59
Content-Type: text/plain; charset=ISO-8859-1

Hi all,

I wanted to start a (hopefully short) discussion around the treatment of the
hadoop.job.ugi configuration in Hadoop 0.22 and beyond (as well as the
secure 0.20 branch). In the current security implementation, the following
incompatible changes have been made even for users who are sticking with
"simple" security.

1) Groups resolution happens on the server side, where it used to happen on
the client. Thus, all Hadoop users must exist on the NN/JT machines in order
for group mapping to succeed (or the user must write a custom group mapper).
2) The hadoop.job.ugi parameter is ignored - instead the user has to use the
new UGI.createRemoteUser("foo").doAs() API, even in simple security.

I'm curious whether the general user community feels these are acceptable
breaking changes. The potential solutions I can see are:

For 1) Add a configuration like hadoop.security.simple.groupmappinglocation
-> "client" or "server". If it's set to "client", the group mapping would
continue to happen as it does in prior versions on the client side.
For 2) If security is "simple", we can have the FileSystem and JobClient
constructors check for this parameter. If it's set, and there is no Subject
object associated with the current AccessControlContext, wrap the creation
of the RPC proxy with the correct doAs() call.

Although security is obviously an absolute necessity for many organizations,
I know of a lot of people who have small clusters and small teams who don't
have any plans to deploy it. For these people, I imagine the above
backward-compatibility layer may be very helpful as they adopt the next
releases of Hadoop. If we don't want to support these options going forward,
we can of course emit deprecation warnings when they are in effect and
remove the compatibility layer in the next major release.

Any thoughts here? Do people often make use of the hadoop.job.ugi variable
to such an extent that this breaking change would block your organization
from upgrading?

Thanks
-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

--0016368e1f63e877560490264e59--

From general-return-2103-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 13 16:32:24 2010
Return-Path: <general-return-2103-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 61092 invoked from network); 13 Sep 2010 16:32:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Sep 2010 16:32:24 -0000
Received: (qmail 23442 invoked by uid 500); 13 Sep 2010 16:32:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23146 invoked by uid 500); 13 Sep 2010 16:32:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23138 invoked by uid 99); 13 Sep 2010 16:32:22 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Sep 2010 16:32:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 13 Sep 2010 16:31:59 +0000
Received: (qmail 60870 invoked by uid 99); 13 Sep 2010 16:31:37 -0000
Received: from localhost.apache.org (HELO mail-pv0-f176.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Sep 2010 16:31:37 +0000
Received: by pvc22 with SMTP id 22so2343035pvc.35
        for <multiple recipients>; Mon, 13 Sep 2010 09:31:37 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.114.131.8 with SMTP id e8mr209305wad.43.1284395496654; Mon, 13
 Sep 2010 09:31:36 -0700 (PDT)
Received: by 10.231.147.2 with HTTP; Mon, 13 Sep 2010 09:31:35 -0700 (PDT)
In-Reply-To: <AANLkTiknJ_SzRux7KhjhxVfUU9FBkNgvYnkpbz3G_+a4@mail.gmail.com>
References: <AANLkTiknJ_SzRux7KhjhxVfUU9FBkNgvYnkpbz3G_+a4@mail.gmail.com>
Date: Mon, 13 Sep 2010 09:31:35 -0700
Message-ID: <AANLkTinwmb=ntf+RTes-f-WW3ZjxDHEOw_rntUoxgU5n@mail.gmail.com>
Subject: Re: hadoop.job.ugi backwards compatibility
From: "Owen O'Malley" <omalley@apache.org>
To: mapreduce-dev@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
X-Virus-Checked: Checked by ClamAV on apache.org

Moving the discussion over to the more appropriate mapreduce-dev.

On Mon, Sep 13, 2010 at 9:08 AM, Todd Lipcon <todd@cloudera.com> wrote:

> 1) Groups resolution happens on the server side, where it used to happen on
> the client. Thus, all Hadoop users must exist on the NN/JT machines in order
> for group mapping to succeed (or the user must write a custom group mapper).

There is a plugin that performs the group lookup. See HADOOP-4656.
There is no requirement for having the user accounts on the NN/JT
although that is the easiest approach. It is not recommended that the
users be allowed to login.

I think it is important that turning security on and off doesn't
drastically change the semantics or protocols. That will become much
much harder to support downstream.

> 2) The hadoop.job.ugi parameter is ignored - instead the user has to use the
> new UGI.createRemoteUser("foo").doAs() API, even in simple security.

User code that counts on hadoop.job.ugi working will be horribly
broken once you turn on security. Turning on and off security should
not involve testing all of your applications. It is unfortunate that
we ever used the configuration value as the user, but continuing to
support it will make our user's code much much more brittle.

-- Owen

From general-return-2104-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 13 21:33:33 2010
Return-Path: <general-return-2104-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73635 invoked from network); 13 Sep 2010 21:33:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Sep 2010 21:33:32 -0000
Received: (qmail 38875 invoked by uid 500); 13 Sep 2010 21:33:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38791 invoked by uid 500); 13 Sep 2010 21:33:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 38777 invoked by uid 99); 13 Sep 2010 21:33:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Sep 2010 21:33:30 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of rmalviya@apple.com designates 17.254.13.23 as permitted sender)
Received: from [17.254.13.23] (HELO mail-out4.apple.com) (17.254.13.23)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Sep 2010 21:33:22 +0000
Received: from relay16.apple.com (relay16.apple.com [17.128.113.55])
	by mail-out4.apple.com (Postfix) with ESMTP id 50A81AF325F7
	for <general@hadoop.apache.org>; Mon, 13 Sep 2010 14:33:02 -0700 (PDT)
X-AuditID: 11807137-b7b43ae00000547d-61-4c8e988ebd7a
Received: from isdevmars002.apple.com (isdevmars002.apple.com [17.216.20.69])
	(using TLS with cipher AES128-SHA (AES128-SHA/128 bits))
	(Client did not present a certificate)
	by relay16.apple.com (Apple SCV relay) with SMTP id BA.0F.21629.E889E8C4; Mon, 13 Sep 2010 14:33:02 -0700 (PDT)
From: Rahul Malviya <rmalviya@apple.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: Multiples Jobs
Date: Mon, 13 Sep 2010 14:33:02 -0700
Message-Id: <5CE1636B-E0F6-4B43-B9E4-FEA435B78F9B@apple.com>
To: general@hadoop.apache.org
Mime-Version: 1.0 (Apple Message framework v1078)
X-Mailer: Apple Mail (2.1078)
X-Brightmail-Tracker: AAAAAA==

Hi,

I am running Pig jobs on Hadoop cluster.

I just wanted to know whether I can run multiple jobs on hadoop cluster =
simultaneously.

Currently when i start two jobs on hadoop they run in a serial fashion.

Is there a way to run N jobs simultaneously on hadoop ?

Thanks,
Rahul=

From general-return-2105-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 13 22:51:25 2010
Return-Path: <general-return-2105-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97399 invoked from network); 13 Sep 2010 22:51:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Sep 2010 22:51:25 -0000
Received: (qmail 57834 invoked by uid 500); 13 Sep 2010 22:51:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57742 invoked by uid 500); 13 Sep 2010 22:51:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57734 invoked by uid 99); 13 Sep 2010 22:51:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Sep 2010 22:51:23 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of anthony.urso@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Sep 2010 22:51:16 +0000
Received: by fxm17 with SMTP id 17so4736456fxm.35
        for <general@hadoop.apache.org>; Mon, 13 Sep 2010 15:50:55 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        bh=qiOyA7tfOlYikcfsRuX8+KjdZIKjsLkCUVt8ZEZOsAI=;
        b=JJKdE7dfJzKhl7aLnO8LFZZKlC7OFG4gCUDLSonk9kq/xBImnNQjPzvOwKTzva09kB
         ffwhYn971y6eX1z4/vAIPbC0w0Kn6UwMmndqS39qgaqnumYsff7bCcGxngp9uH/ERgbA
         8L8F4F7sYXLnLS3tCVHicbKo0iysvUcLny6KM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=uETrnsGehY+Fv+4XOjT+3DuZ3+460U7OpPXPrO1c4yX6wuYuEmtXho0ZyGstMNk/uF
         gG+eD6cliP0WfHjWcHgbqUxZ2E1xQMCrwsJ/X8vwUi+EK8Lk1l3F5vwXOPL2c9vnBGpH
         ZXX0tA0V0PvJGoTlz4tMikoKrGioehiAFxnwo=
MIME-Version: 1.0
Received: by 10.223.126.79 with SMTP id b15mr3414130fas.88.1284418255600; Mon,
 13 Sep 2010 15:50:55 -0700 (PDT)
Sender: anthony.urso@gmail.com
Received: by 10.223.103.83 with HTTP; Mon, 13 Sep 2010 15:50:55 -0700 (PDT)
In-Reply-To: <5CE1636B-E0F6-4B43-B9E4-FEA435B78F9B@apple.com>
References: <5CE1636B-E0F6-4B43-B9E4-FEA435B78F9B@apple.com>
Date: Mon, 13 Sep 2010 15:50:55 -0700
X-Google-Sender-Auth: pWdLnGaPJWfJ03BEppy719POhQU
Message-ID: <AANLkTingLhp_Pri_rPMbQHutnDFZhdfmpn4Pg8M8JAMn@mail.gmail.com>
Subject: Re: Multiples Jobs
From: Anthony Urso <anthonyu@cs.ucla.edu>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Try the fair scheduler, it will seem more simultaneous than the
default scheduler.

http://hadoop.apache.org/mapreduce/docs/r0.21.0/fair_scheduler.html

On Mon, Sep 13, 2010 at 2:33 PM, Rahul Malviya <rmalviya@apple.com> wrote:
> Hi,
>
> I am running Pig jobs on Hadoop cluster.
>
> I just wanted to know whether I can run multiple jobs on hadoop cluster simultaneously.
>
> Currently when i start two jobs on hadoop they run in a serial fashion.
>
> Is there a way to run N jobs simultaneously on hadoop ?
>
> Thanks,
> Rahul

From general-return-2106-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 14 17:25:11 2010
Return-Path: <general-return-2106-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 13937 invoked from network); 14 Sep 2010 17:25:10 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Sep 2010 17:25:10 -0000
Received: (qmail 89540 invoked by uid 500); 14 Sep 2010 17:25:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89394 invoked by uid 500); 14 Sep 2010 17:25:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89386 invoked by uid 99); 14 Sep 2010 17:25:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Sep 2010 17:25:08 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of rmalviya@apple.com designates 17.254.13.22 as permitted sender)
Received: from [17.254.13.22] (HELO mail-out3.apple.com) (17.254.13.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Sep 2010 17:25:00 +0000
Received: from relay13.apple.com (relay13.apple.com [17.128.113.29])
	by mail-out3.apple.com (Postfix) with ESMTP id 27BF1A88E49F
	for <general@hadoop.apache.org>; Tue, 14 Sep 2010 10:24:40 -0700 (PDT)
X-AuditID: 1180711d-b7b8eae0000035ac-40-4c8fafd8df63
Received: from isdevmars002.apple.com (isdevmars002.apple.com [17.216.20.69])
	(using TLS with cipher AES128-SHA (AES128-SHA/128 bits))
	(Client did not present a certificate)
	by relay13.apple.com (Apple SCV relay) with SMTP id BB.E5.13740.8DFAF8C4; Tue, 14 Sep 2010 10:24:40 -0700 (PDT)
From: Rahul Malviya <rmalviya@apple.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: Number of Mappers
Date: Tue, 14 Sep 2010 10:24:39 -0700
Message-Id: <CA7CFEB8-B186-4575-BABD-335EFA370B2B@apple.com>
To: general@hadoop.apache.org
Mime-Version: 1.0 (Apple Message framework v1078)
X-Mailer: Apple Mail (2.1078)
X-Brightmail-Tracker: AAAAAA==

Hi=20

I am using Pig jobs to run on Hadoop but always it runs 4 mappers =
simultaneously.=20

How can I increase the number of simultaneous mappers to run ?

What config do I have to change ?

Thanks,
Rahul=

From general-return-2107-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 14 19:34:21 2010
Return-Path: <general-return-2107-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 64371 invoked from network); 14 Sep 2010 19:34:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Sep 2010 19:34:21 -0000
Received: (qmail 59489 invoked by uid 500); 14 Sep 2010 19:34:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59357 invoked by uid 500); 14 Sep 2010 19:34:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59349 invoked by uid 99); 14 Sep 2010 19:34:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Sep 2010 19:34:19 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,UNPARSEABLE_RELAY
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of peter_raeth@juno.com designates 64.136.55.15 as permitted sender)
Received: from [64.136.55.15] (HELO outbound-mail.vgs.untd.com) (64.136.55.15)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 14 Sep 2010 19:34:10 +0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=juno.com; s=alpha;
	t=1284492828; bh=47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=; l=0;
	h=From:Date:To:Subject:Message-Id:Content-Type;
	b=f5OndazxxCBXjHh+cUpXlWL/VcbGSRz30jBm2tc9nUU45lpTGUb7CknBYJykJSTy5
	 OHaAqYvBTKdX0ShFECyLmueQ7aFH4xhutGA7BmSsw5Nw4g2IQoaCA7r508s01kJFxV
	 /KbGD20YrEHFbXLJ0lObFvyNYRDSJTJ6YFVKqMrE=
X-UOL-TAGLINE: true
Received: from outbound-bu1.vgs.untd.com (webmail05.vgs.untd.com [10.181.12.145])
	by smtpout03.vgs.untd.com with SMTP id AABGJ9VRTAEAKU7A
	for <general@hadoop.apache.org> (sender <peter_raeth@juno.com>);
	Tue, 14 Sep 2010 12:33:05 -0700 (PDT)
X-UNTD-OriginStamp: IXkAfFukJf2tQJaX29ekrgrTDOH9knyenCwaJABJqMo9jwVrxl5+4Q==
Received: (from peter_raeth@juno.com) 
 by webmail05.vgs.untd.com (jqueuemail) id QPTJT496; Tue, 14 Sep 2010 12:32:59 PDT
Received: from [140.31.142.102] by webmail05.vgs.untd.com with HTTP:
	Tue, 14 Sep 2010 19:31:45 GMT
X-Originating-IP: [140.31.142.102]
Mime-Version: 1.0
From: "peter_raeth@juno.com" <peter_raeth@juno.com>
Date: Tue, 14 Sep 2010 19:31:45 GMT
To: general@hadoop.apache.org
Subject: C++ Record Readers and Writers
X-Mailer: Webmail Version 4.0
Message-Id: <20100914.153145.10299.1@webmail05.vgs.untd.com>
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
Content-Type: text/plain; charset=ISO-8859-1
X-UNTD-BodySize: 228
X-ContentStamp: 2:1:1458885665
X-UNTD-Peer-Info: 10.181.12.145|webmail05.vgs.untd.com|outbound-bu1.vgs.untd.com|peter_raeth@juno.com
X-Virus-Checked: Checked by ClamAV on apache.org

Am searching for guidance on writing record readers and writers in C++. =
All the books and Googled examples that I can find write examples in Jav=
a. Can anyone give me a link to documentation on this topic?

Thanks,

Peter.


____________________________________________________________
Mortgage Rates Hit 3.25%
If you owe under $729k you probably qualify for Obama's Refi Program
http://thirdpartyoffers.juno.com/TGL3131/4c8fcdf133f4ebb5618st03vuc

From general-return-2108-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Sep 15 01:22:51 2010
Return-Path: <general-return-2108-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70485 invoked from network); 15 Sep 2010 01:22:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Sep 2010 01:22:51 -0000
Received: (qmail 41367 invoked by uid 500); 15 Sep 2010 01:22:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 41299 invoked by uid 500); 15 Sep 2010 01:22:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 41291 invoked by uid 99); 15 Sep 2010 01:22:49 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Sep 2010 01:22:49 +0000
X-ASF-Spam-Status: No, hits=0.6 required=10.0
	tests=FREEMAIL_FROM,HK_RANDOM_ENVFROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of zjffdu@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Sep 2010 01:22:42 +0000
Received: by wwc33 with SMTP id 33so350400wwc.29
        for <general@hadoop.apache.org>; Tue, 14 Sep 2010 18:22:22 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=LOx82NW9o3UM0hB7kbS16e0cUWxVSahsmZ3QbE87FYY=;
        b=wY65RwEihmnXm1g0cTgzcP/7iK82GIXk6yhnbvj0E1kgaOAWfGOfltns6XSik4Qce+
         rD5R00w9K8fNPGw17+/cRPDbZSL/mZnO0mYzQaR6o0NfFi9YEISKiNF1KL0F6ynZV4rj
         gYHA+AXNci2NhI2DOX2W0bGKa5JAdXV/a2hmM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Yg5Z5it08nVimFD2CPKXTkbtOEGWi/dVz7jgDCQT+/zahQQRiX2XPuOstzLtzcChl0
         9OJtcJIsOfsyuPwLPEGe8S1E1jKgK7euxRzmCY22307El367rVdKdxRNf83a9heZZiGq
         ReNgxnYd6jEQ4xnbeJ5mpW2QnF+zIZGXmlZOA=
MIME-Version: 1.0
Received: by 10.216.235.104 with SMTP id t82mr4568473weq.103.1284513742078;
 Tue, 14 Sep 2010 18:22:22 -0700 (PDT)
Received: by 10.216.163.5 with HTTP; Tue, 14 Sep 2010 18:22:22 -0700 (PDT)
In-Reply-To: <CA7CFEB8-B186-4575-BABD-335EFA370B2B@apple.com>
References: <CA7CFEB8-B186-4575-BABD-335EFA370B2B@apple.com>
Date: Wed, 15 Sep 2010 09:22:22 +0800
Message-ID: <AANLkTi=c+HSMNOKVKPQC98aOFi=PMoivWfVJ186ZWjd1@mail.gmail.com>
Subject: Re: Number of Mappers
From: Jeff Zhang <zjffdu@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Rahul,

You can not determined set the mapper number as reducer number. The
mapper number is determined by the your InputFormat. If you are using
PigStorage, then the InputFormat is PigTextInputFormat which is a
subclass of TextInputFormat. You can refer TextInputFormat.getSplits()
to see how it split the input.



On Wed, Sep 15, 2010 at 1:24 AM, Rahul Malviya <rmalviya@apple.com> wrote:
> Hi
>
> I am using Pig jobs to run on Hadoop but always it runs 4 mappers simultaneously.
>
> How can I increase the number of simultaneous mappers to run ?
>
> What config do I have to change ?
>
> Thanks,
> Rahul



-- 
Best Regards

Jeff Zhang

From general-return-2109-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Sep 15 07:50:46 2010
Return-Path: <general-return-2109-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82867 invoked from network); 15 Sep 2010 07:50:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Sep 2010 07:50:46 -0000
Received: (qmail 50437 invoked by uid 500); 15 Sep 2010 07:50:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49967 invoked by uid 500); 15 Sep 2010 07:50:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49937 invoked by uid 99); 15 Sep 2010 07:50:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Sep 2010 07:50:42 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [85.214.115.60] (HELO h1349259.stratoserver.net) (85.214.115.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Sep 2010 07:50:35 +0000
Received: from crt-01-tr.neofonie.de ([91.213.91.28] helo=essen.neofonie.priv)
	by h1349259.stratoserver.net with esmtpsa (TLS1.0:DHE_RSA_AES_128_CBC_SHA1:16)
	(Exim 4.69)
	(envelope-from <isabel@apache.org>)
	id 1Ovmks-00032h-FG; Wed, 15 Sep 2010 07:50:10 +0000
Date: Wed, 15 Sep 2010 09:44:45 +0200
From: Isabel Drost <isabel@apache.org>
To: hadoopberlin@isabel-drost.de, user@mahout.apache.org,
 java-user@lucene.apache.org, general@hadoop.apache.org,
 katta-developer@lists.sourceforge.net, solr-user@lucene.apache.org
Cc: Martin Schmidt <msc@newthinking.de>
Subject: Apache Hadoop Get Together Berlin October 2010 - this time with a
 huge Mahout focus
Message-ID: <20100915094445.58115636@essen.neofonie.priv>
X-Mailer: Claws Mail 3.7.2 (GTK+ 2.14.4; x86_64-http://packman.links2linux.de-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org


Hello,

this is to announce the next Apache Hadoop Get Together sponsored by
JTeam (http://www.jteam.nl) that will take place in newthinking store
in Berlin.

When: October 7th, 5p.m.
Where: Newthinking store Berlin

As always there will be slots of 30min each for talks on your Hadoop
topic. After each talk there will be a lot time to discuss. You can
order drinks directly at the bar in the newthinking store. If you like,
you can order pizza. We will go to Cafe Aufsturz after the event for
some beer and something to eat.

Talks scheduled so far:

Max Heimel: "Hidden Markov Models for Apache Mahout"

Abstract: In this talk I will present and discuss an implementation of
a powerful statistical tool called Hidden Markov Models for the Apache
Mahout project. Hidden Markov models allow to mathematically deduce the
structure of an underlying - and unobservable - process based on the
structure of the produced data. Hidden Markov Models are thus
frequently applied in pattern recognition to deduce structures that are
not directly observable. Examples for applications of Hidden Markov
Models include the recognition of syllables in speech recordings,
handwritten letter recognition and part-of-speech tagging.

Sebastian Schelter: Distributed Itembased Collaborative Filtering with
Apache Mahout"

Abstract: Recommendation Mining helps users find items they like. A
very popular way to implement this is by using Collaborative Filtering.
This talk will give an introduction to an approach called Itembased
Collaborative Filtering and explain Mahout's Map/Reduce based
implementation of it.

Please do indicate on Upcoming
<http://upcoming.yahoo.com/event/6792156> or on Xing
<https://www.xing.com/events/apache-hadoop-berlin-october-2010-564265>
if you are coming so we can more safely plan capacities. Updates to the
event, a brief summary and videos will be posted on
http://isabel-drost.de/hadoop

JTeam is looking for Java developers and search enthusiasts. Check out
their jobs page (http://www.jteam.nl/Jobs/Jobs.html) for more info!

As always a big Thank You goes to newthinking store for providing the
venue for free for our event.

Looking forward to seeing you in Berlin as well,
Isabel

From general-return-2110-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Sep 16 07:34:44 2010
Return-Path: <general-return-2110-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60645 invoked from network); 16 Sep 2010 07:34:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Sep 2010 07:34:43 -0000
Received: (qmail 82226 invoked by uid 500); 16 Sep 2010 07:34:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81765 invoked by uid 500); 16 Sep 2010 07:34:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81755 invoked by uid 99); 16 Sep 2010 07:34:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Sep 2010 07:34:37 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [88.198.2.104] (HELO koch.ro) (88.198.2.104)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Sep 2010 07:34:31 +0000
Received: from 246-0.79-83.cust.bluewin.ch ([83.79.0.246] helo=jona.localnet)
	by koch.ro with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
	(Exim 4.69)
	(envelope-from <thomas@koch.ro>)
	id 1Ow8yq-0003MA-Ud; Thu, 16 Sep 2010 09:34:05 +0200
From: Thomas Koch <thomas@koch.ro>
Reply-To: thomas@koch.ro
To: hadoopworld@cloudera.com,
 general@hadoop.apache.org
Subject: Keysigning party at Hadoop World
Date: Thu, 16 Sep 2010 09:33:41 +0200
User-Agent: KMail/1.13.5 (Linux/2.6.32-4-amd64; KDE/4.4.5; x86_64; ; )
MIME-Version: 1.0
Content-Type: multipart/signed;
  boundary="nextPart2334805.d7GfjuynQE";
  protocol="application/pgp-signature";
  micalg=pgp-sha256
Content-Transfer-Encoding: 7bit
Message-Id: <201009160934.01127.thomas@koch.ro>

--nextPart2334805.d7GfjuynQE
Content-Type: Text/Plain;
  charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi,

I'm the Debian maintainer of Hadoop, HBase and ZooKeeper. It's an issue for=
=20
me, that the gpg keys used to sign the releases of these projects are norma=
lly=20
not signed by anyone.
So it is the same as if the releases would not be signed at all.

Therefor I'd kindly like to ask you to offer a key signing party[1] at Hado=
op=20
World. Unfortunately I'm from Switzerland and can't come to Hadoop World,=20
otherwise I'd have offered my assistance.

[1] http://en.wikipedia.org/wiki/Key_signing_party

Best regards,

Thomas Koch, http://www.koch.ro

--nextPart2334805.d7GfjuynQE
Content-Type: application/pgp-signature; name=signature.asc 
Content-Description: This is a digitally signed message part.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iQIcBAABCAAGBQJMkchfAAoJEAf8SJEEK6ZabQoP+gJddx8qZ9CGLiPi75s39Yz2
iRT9BaIjgo3V3DW8R8teUgGdFiiGmaG0IIBwPf1s0cbC7sqmVVONGVS8GD49zGNT
q0dNMZePwBucybiOlVb65GJz/GBfLyTPLsfmS6LDWi05B1bYGmjZuVQ+6gB9268N
0Mung+fN5ShzX4CGx6Ptf7qd87/mUsOWQc47O9iAKomp4suiM+l8GkOGpCr2ciS2
/64F1IKHaHEafqq0bTnyU7ijnuTta2Y8kqPkN6adjb2d/B/yIIAUtieNbUS42/gp
BYweyfI7DDfvUZlAuCNmfdnGjFE2/hA76zWHscbBLxfyjvSxDi4R73CeL4xMEd7l
qvSJbLNoEw6k2j9ubGCxsA5z73QhelWnD9A9YMqMHQzEKufWkMHcpplPaYvFEfqn
yMkN2Tty3eOZe3uOR2E9nDNWhp+ro2zjws+DUHu77IJoXjM5N8Q0pAgt2RQSGnKm
e+VT4CFIQuhncWDWujXRb8dFRDeLAFZADY600e+w7VZ0vlAnmxqvgQUlpNlRepb5
zpcfazr6uKAlb9QH6IcIcoxI6gdRRt/UTX9UGnFxDTh73nVTgOScok+U09kjX7Gi
ga50R1OgLu6ygQNY7W9JUCQhOt/HXYge9bYG3mZFRcExBSmuzjKZSIuVcoe4jz2g
IWQm0I5ruwR1Qj99203h
=j13B
-----END PGP SIGNATURE-----

--nextPart2334805.d7GfjuynQE--

From general-return-2111-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Sep 16 07:51:57 2010
Return-Path: <general-return-2111-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63890 invoked from network); 16 Sep 2010 07:51:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Sep 2010 07:51:57 -0000
Received: (qmail 4289 invoked by uid 500); 16 Sep 2010 07:51:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3945 invoked by uid 500); 16 Sep 2010 07:51:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3937 invoked by uid 99); 16 Sep 2010 07:51:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Sep 2010 07:51:51 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of clb991917@gmail.com designates 209.85.216.176 as permitted sender)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Sep 2010 07:51:45 +0000
Received: by qyk2 with SMTP id 2so1235608qyk.14
        for <general@hadoop.apache.org>; Thu, 16 Sep 2010 00:51:24 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=hwHVl1mKv7o+LU1QCktLD23fwa+T2/uqafUOsp0+RY4=;
        b=gDNxQDkenPn7avyXQEGU++7Yc98/kvV2bNxQcH5BOEFSQzLTnSJ5CHgDmwvMpJ3kZ7
         1BbR1bOoDDJepJAFN1bYCNjmRjVSldq8Q3EnwR9zcg+lmoCp10T0yz0OT2UWhbO9NBBn
         BgdtFe1JGtpi6F+EMaSBplpScjbvKxScOxyKQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=vWgDiLgjc/AaPUQ/c9ha/vnojJODUKzgk8o3teyUgwuu+ecoZ5l95oaM5SA1YsQ7Me
         xfUiQRH7iaYajFO9U0Etp9SO3sHO2fVL3YnmQPrbQ+jmz0SwqoCw5XRoPa4w4GMQ/4rO
         Mp1N3e2+De9PjTj8qK1Ia0jWJSbWzAf+tgPq8=
MIME-Version: 1.0
Received: by 10.224.11.20 with SMTP id r20mr1959352qar.29.1284623484771; Thu,
 16 Sep 2010 00:51:24 -0700 (PDT)
Received: by 10.229.74.18 with HTTP; Thu, 16 Sep 2010 00:51:24 -0700 (PDT)
Date: Thu, 16 Sep 2010 00:51:24 -0700
Message-ID: <AANLkTimwdJgNdPZYcwH=iOSTZe0yS5gz-ZNb2K9a9Pb4@mail.gmail.com>
Subject: question about zip and extract jobs on hadoop
From: vincent cai <clb991917@gmail.com>
To: hadoopworld@cloudera.com, general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175cdd647adec004905bb696

--0015175cdd647adec004905bb696
Content-Type: text/plain; charset=ISO-8859-1

Hi all

   I'm just thinking about the elt extract jobs.

   is it possible to deploy that on hadoop cluster?

   if zip or unzip command can be run on hadoop datanodes , the network
bandwidth will be the only bottleneck.

   Millions of zip files distributed to datanodes and zip or unzip. if
possible make the super datanode by VM and SAN.

   That could be a "Super fast SAN"

   looks like the FileUtil class is containing some methods calling the
linux gzip or untar command, but the hadoop fs manual is not providing that

   pls let me know if you have any comments .

   I'm just thinking about the possibility.

   thanks

Best Regards
Vincent
Skype , cailibing1

--0015175cdd647adec004905bb696--

From general-return-2112-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Sep 16 15:38:47 2010
Return-Path: <general-return-2112-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27051 invoked from network); 16 Sep 2010 15:38:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Sep 2010 15:38:46 -0000
Received: (qmail 45938 invoked by uid 500); 16 Sep 2010 15:38:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45574 invoked by uid 500); 16 Sep 2010 15:38:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45566 invoked by uid 99); 16 Sep 2010 15:38:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Sep 2010 15:38:35 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 16 Sep 2010 15:38:33 +0000
Received: (qmail 26833 invoked by uid 99); 16 Sep 2010 15:38:11 -0000
Received: from localhost.apache.org (HELO mail-gx0-f176.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Sep 2010 15:38:11 +0000
Received: by gxk7 with SMTP id 7so653577gxk.35
        for <general@hadoop.apache.org>; Thu, 16 Sep 2010 08:38:10 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.100.153.11 with SMTP id a11mr3814373ane.200.1284651490525;
 Thu, 16 Sep 2010 08:38:10 -0700 (PDT)
Received: by 10.231.147.2 with HTTP; Thu, 16 Sep 2010 08:38:09 -0700 (PDT)
In-Reply-To: <201009160934.01127.thomas@koch.ro>
References: <201009160934.01127.thomas@koch.ro>
Date: Thu, 16 Sep 2010 08:38:09 -0700
Message-ID: <AANLkTin3TvNCZ+Q0VhyLV9oOwJ4sMGe-avC155YMFMjp@mail.gmail.com>
Subject: Re: Keysigning party at Hadoop World
From: "Owen O'Malley" <omalley@apache.org>
To: general@hadoop.apache.org, thomas@koch.ro
Content-Type: text/plain; charset=UTF-8
X-Virus-Checked: Checked by ClamAV on apache.org

On Thu, Sep 16, 2010 at 12:33 AM, Thomas Koch <thomas@koch.ro> wrote:

> I'm the Debian maintainer of Hadoop, HBase and ZooKeeper. It's an issue for
> me, that the gpg keys used to sign the releases of these projects are normally
> not signed by anyone.
> So it is the same as if the releases would not be signed at all.

That has bothered me too. Since all of the developers who have rolled
the releases are all in the Bay Area we should just use the upcoming
HDFS & MapReduce contributor meeting to do this. That will at least
show we trust each other. *grin* If there is a key signing party at
Apache World, we can get into the larger Apache web of trust.

In terms of getting into your web of trust, I'll contact you off list
so that you can sign my key. For the record, my key is 3D0C92B9, which
is visible in http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
and the MIT key servers.

> Therefor I'd kindly like to ask you to offer a key signing party[1] at Hadoop
> World. Unfortunately I'm from Switzerland and can't come to Hadoop World,
> otherwise I'd have offered my assistance.

Let's get it done before then.

-- Owen

From general-return-2113-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Sep 16 19:03:00 2010
Return-Path: <general-return-2113-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4375 invoked from network); 16 Sep 2010 19:03:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Sep 2010 19:03:00 -0000
Received: (qmail 33735 invoked by uid 500); 16 Sep 2010 19:02:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33461 invoked by uid 500); 16 Sep 2010 19:02:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33444 invoked by uid 99); 16 Sep 2010 19:02:56 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Sep 2010 19:02:56 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of vicaya@gmail.com designates 209.85.213.48 as permitted sender)
Received: from [209.85.213.48] (HELO mail-yw0-f48.google.com) (209.85.213.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Sep 2010 19:02:33 +0000
Received: by ywf9 with SMTP id 9so754875ywf.35
        for <multiple recipients>; Thu, 16 Sep 2010 12:02:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:cc:content-type;
        bh=blHpcP5sjRCmWvktvmgvzx+sLPRji8hwOmikWsRPsWQ=;
        b=ow/r+ca1gYE7z2AwKcKtOXJ3ny6NjcbGJYaP0yfTQJsrH6ejQ80HyKIXkGzJhuvnz+
         u2DEmR2A9ib0jlx8LvDlivCzai2TSx/qUtRoLg5YpT74eZNhvtj0w8e6GCtlMEu27xvB
         +pyiNSUE+NkLr+qNefe/6cPqx58EdaeSsXWbg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:cc:content-type;
        b=TMAiJxu1n/aFwStM1x/L1A7pP3pSs3QpfXEQ+QQs5APsYcfYlyVN83tWd9qqWG1IYR
         09X4SRlEqpd3nIAhq77bz416dbs5Wsjsdrnt/1+t69xIAtq87dqEjcVMMQOhCi+wca29
         4jU0b1NYZ18ZmaqhTvWA26kZLQHndTtlGev5Q=
MIME-Version: 1.0
Received: by 10.103.251.7 with SMTP id d7mr312234mus.80.1284663731342; Thu, 16
 Sep 2010 12:02:11 -0700 (PDT)
Received: by 10.220.92.81 with HTTP; Thu, 16 Sep 2010 12:02:10 -0700 (PDT)
Date: Thu, 16 Sep 2010 12:02:10 -0700
Message-ID: <AANLkTikJEJB3=ztmbcLxSRjxh-CQ7VQyAdFwF1VDe9H8@mail.gmail.com>
Subject: Paths of evolution for Hadoop metrics
From: Luke <vicaya@gmail.com>
To: common-dev@hadoop.apache.org
Cc: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Working on the new metrics framework
(https://issues.apache.org/jira/browse/HADOOP-6728 and its sub-tasks),
I'd like to invite some feedback from the community on how to evolve
Hadoop metrics. Here is a quick summary of changes:

1. Simplified metrics instrumentation
    - Declarative annotations for common use cases
    - Narrower interfaces for advanced use cases
    - Automatic JMX access to all metrics
    - User code reduction in all cases
2. Extensive configuration and filtering options
3. Fix broken design/implementation for parallel/multi-backend use cases
4. More efficient (both space and time) design and implementation

These benefits come with a necessity to break backward compatibility
in both end-user (ops) visible config file changes and API changes for
plugin developers.

It seems to me that we have the following paths (targeting 0.22) to
transition to the new metrics framework (no major issues found in Y's
scale (a few thousand nodes) testing and being deployed to
production.)

1. Remove the current o.a.h.metrics package completely and upgrade all
metrics in Hadoop to use the new framework.
    - The current metrics APIs are marked as "Public/Evolving".
    - There are a few uses (notably HBase) outside Hadoop core
(common, hdfs, mapreduce)
    - Upgrade/fix needs will be immediately apparent.
       - A simple band-aid would be an old hadoop metrics jar for
external projects

2. Deprecate the current metrics package and upgrade all metrics in
Hadoop to use the new framework.
    - Hadoop core would need to use new config files
    - External packages that depend on the old metrics framework
continue working
       with old config files; without the benefits of the new framework
    - Potential config/api confusions and support issues

3. Depreciate the current metrics package and make all metrics work
with both frameworks.
    - Framework switchable via config variable at startup time
    - External dependencies can work without modification
    - More dev work to support dual framework setup
    - More potential config/api confusions and support issues

3a. Same as 3. except with an option to run both framework at the same
time with obviously more runtime overhead.

I prefer the first path, obviously. OTOH, we could be convinced for
other paths if there are real compelling reasons as well.

__Luke

From general-return-2114-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Sep 17 06:44:38 2010
Return-Path: <general-return-2114-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 26420 invoked from network); 17 Sep 2010 06:44:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Sep 2010 06:44:38 -0000
Received: (qmail 54533 invoked by uid 500); 17 Sep 2010 06:44:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52423 invoked by uid 500); 17 Sep 2010 06:44:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52409 invoked by uid 99); 17 Sep 2010 06:44:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Sep 2010 06:44:30 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [211.103.157.132] (HELO mail.oraro.net) (211.103.157.132)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Sep 2010 06:44:20 +0000
Received: from WangssPC ([222.182.25.188])
	(authenticated user wangshisen@oraro.net)
	by mail.oraro.net;
	Fri, 17 Sep 2010 14:43:06 +0800
From: =?gb2312?B?zfXKwMmt?= <wangshisen@oraro.net>
To: <general@hadoop.apache.org>
Cc: <common-dev@hadoop.apache.org>
Subject: how to view the result of kmeans
Date: Fri, 17 Sep 2010 14:42:31 +0800
Message-ID: <000301cb5633$94ae68d0$be0b3a70$@net>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_NextPart_000_0004_01CB5676.A2D1A8D0"
X-Mailer: Microsoft Office Outlook 12.0
Thread-Index: ActWM38X7d60YnG8SY6kWVFG2qA7pQ==
Content-Language: zh-cn

------=_NextPart_000_0004_01CB5676.A2D1A8D0
Content-Type: text/plain;
	charset="gb2312"
Content-Transfer-Encoding: 7bit

hi,

 

I install the mahout and hadoop on SUSE Linux, and it works. 

I run the kmeans sample use command:

hadoop fs -put synthetic_control.data testdata

mahout org.apache.mahout.clustering.syntheticcontrol.kmeans.Job

 

it run successfully here is the output files

drwxr-xr-x   - root supergroup          0 2010-09-17 10:59
/user/root/output/canopies

drwxr-xr-x   - root supergroup          0 2010-09-17 10:59
/user/root/output/canopies/_logs

drwxr-xr-x   - root supergroup          0 2010-09-17 10:59
/user/root/output/canopies/_logs/history

-rw-r--r--   1 root supergroup      17229 2010-09-17 10:59
/user/root/output/canopies/_logs/history/localhost_1284692116007_job_2010091
71055_0002_conf.xml

-rw-r--r--   1 root supergroup       9165 2010-09-17 10:59
/user/root/output/canopies/_logs/history/localhost_1284692116007_job_2010091
71055_0002_root_mahout-examples-0.3.job

-rw-r--r--   1 root supergroup       5749 2010-09-17 10:59
/user/root/output/canopies/part-00000

drwxr-xr-x   - root supergroup          0 2010-09-17 11:00
/user/root/output/clusters-0

drwxr-xr-x   - root supergroup          0 2010-09-17 10:59
/user/root/output/clusters-0/_logs

drwxr-xr-x   - root supergroup          0 2010-09-17 10:59
/user/root/output/clusters-0/_logs/history

-rw-r--r--   1 root supergroup      17479 2010-09-17 10:59
/user/root/output/clusters-0/_logs/history/localhost_1284692116007_job_20100
9171055_0003_conf.xml

-rw-r--r--   1 root supergroup       9283 2010-09-17 10:59
/user/root/output/clusters-0/_logs/history/localhost_1284692116007_job_20100
9171055_0003_root_mahout-examples-0.3.job

-rw-r--r--   1 root supergroup       5757 2010-09-17 11:00
/user/root/output/clusters-0/part-00000

drwxr-xr-x   - root supergroup          0 2010-09-17 11:00
/user/root/output/clusters-1

drwxr-xr-x   - root supergroup          0 2010-09-17 11:00
/user/root/output/clusters-1/_logs

drwxr-xr-x   - root supergroup          0 2010-09-17 11:00
/user/root/output/clusters-1/_logs/history

-rw-r--r--   1 root supergroup      17481 2010-09-17 11:00
/user/root/output/clusters-1/_logs/history/localhost_1284692116007_job_20100
9171055_0004_conf.xml

-rw-r--r--   1 root supergroup       9283 2010-09-17 11:00
/user/root/output/clusters-1/_logs/history/localhost_1284692116007_job_20100
9171055_0004_root_mahout-examples-0.3.job

-rw-r--r--   1 root supergroup       5757 2010-09-17 11:00
/user/root/output/clusters-1/part-00000

drwxr-xr-x   - root supergroup          0 2010-09-17 11:01
/user/root/output/clusters-2

drwxr-xr-x   - root supergroup          0 2010-09-17 11:00
/user/root/output/clusters-2/_logs

drwxr-xr-x   - root supergroup          0 2010-09-17 11:00
/user/root/output/clusters-2/_logs/history

-rw-r--r--   1 root supergroup      17481 2010-09-17 11:00
/user/root/output/clusters-2/_logs/history/localhost_1284692116007_job_20100
9171055_0005_conf.xml

-rw-r--r--   1 root supergroup       9496 2010-09-17 11:00
/user/root/output/clusters-2/_logs/history/localhost_1284692116007_job_20100
9171055_0005_root_mahout-examples-0.3.job

-rw-r--r--   1 root supergroup       5757 2010-09-17 11:01
/user/root/output/clusters-2/part-00000

drwxr-xr-x   - root supergroup          0 2010-09-17 11:01
/user/root/output/clusters-3

drwxr-xr-x   - root supergroup          0 2010-09-17 11:01
/user/root/output/clusters-3/_logs

drwxr-xr-x   - root supergroup          0 2010-09-17 11:01
/user/root/output/clusters-3/_logs/history

-rw-r--r--   1 root supergroup      17481 2010-09-17 11:01
/user/root/output/clusters-3/_logs/history/localhost_1284692116007_job_20100
9171055_0006_conf.xml

-rw-r--r--   1 root supergroup       9496 2010-09-17 11:01
/user/root/output/clusters-3/_logs/history/localhost_1284692116007_job_20100
9171055_0006_root_mahout-examples-0.3.job

-rw-r--r--   1 root supergroup       5757 2010-09-17 11:01
/user/root/output/clusters-3/part-00000

drwxr-xr-x   - root supergroup          0 2010-09-17 11:02
/user/root/output/clusters-4

drwxr-xr-x   - root supergroup          0 2010-09-17 11:01
/user/root/output/clusters-4/_logs

drwxr-xr-x   - root supergroup          0 2010-09-17 11:01
/user/root/output/clusters-4/_logs/history

-rw-r--r--   1 root supergroup      17481 2010-09-17 11:01
/user/root/output/clusters-4/_logs/history/localhost_1284692116007_job_20100
9171055_0007_conf.xml

-rw-r--r--   1 root supergroup       9496 2010-09-17 11:01
/user/root/output/clusters-4/_logs/history/localhost_1284692116007_job_20100
9171055_0007_root_mahout-examples-0.3.job

-rw-r--r--   1 root supergroup       5757 2010-09-17 11:02
/user/root/output/clusters-4/part-00000

drwxr-xr-x   - root supergroup          0 2010-09-17 11:02
/user/root/output/clusters-5

drwxr-xr-x   - root supergroup          0 2010-09-17 11:02
/user/root/output/clusters-5/_logs

drwxr-xr-x   - root supergroup          0 2010-09-17 11:02
/user/root/output/clusters-5/_logs/history

-rw-r--r--   1 root supergroup      17481 2010-09-17 11:02
/user/root/output/clusters-5/_logs/history/localhost_1284692116007_job_20100
9171055_0008_conf.xml

-rw-r--r--   1 root supergroup       9496 2010-09-17 11:02
/user/root/output/clusters-5/_logs/history/localhost_1284692116007_job_20100
9171055_0008_root_mahout-examples-0.3.job

-rw-r--r--   1 root supergroup       5757 2010-09-17 11:02
/user/root/output/clusters-5/part-00000

drwxr-xr-x   - root supergroup          0 2010-09-17 11:03
/user/root/output/clusters-6

drwxr-xr-x   - root supergroup          0 2010-09-17 11:02
/user/root/output/clusters-6/_logs

drwxr-xr-x   - root supergroup          0 2010-09-17 11:02
/user/root/output/clusters-6/_logs/history

-rw-r--r--   1 root supergroup      17481 2010-09-17 11:02
/user/root/output/clusters-6/_logs/history/localhost_1284692116007_job_20100
9171055_0009_conf.xml

-rw-r--r--   1 root supergroup       9496 2010-09-17 11:02
/user/root/output/clusters-6/_logs/history/localhost_1284692116007_job_20100
9171055_0009_root_mahout-examples-0.3.job

-rw-r--r--   1 root supergroup       5757 2010-09-17 11:03
/user/root/output/clusters-6/part-00000

drwxr-xr-x   - root supergroup          0 2010-09-17 11:03
/user/root/output/clusters-7

drwxr-xr-x   - root supergroup          0 2010-09-17 11:03
/user/root/output/clusters-7/_logs

drwxr-xr-x   - root supergroup          0 2010-09-17 11:03
/user/root/output/clusters-7/_logs/history

-rw-r--r--   1 root supergroup      17481 2010-09-17 11:03
/user/root/output/clusters-7/_logs/history/localhost_1284692116007_job_20100
9171055_0010_conf.xml

-rw-r--r--   1 root supergroup       9496 2010-09-17 11:03
/user/root/output/clusters-7/_logs/history/localhost_1284692116007_job_20100
9171055_0010_root_mahout-examples-0.3.job

-rw-r--r--   1 root supergroup       5757 2010-09-17 11:03
/user/root/output/clusters-7/part-00000

drwxr-xr-x   - root supergroup          0 2010-09-17 11:04
/user/root/output/clusters-8

drwxr-xr-x   - root supergroup          0 2010-09-17 11:03
/user/root/output/clusters-8/_logs

drwxr-xr-x   - root supergroup          0 2010-09-17 11:03
/user/root/output/clusters-8/_logs/history

-rw-r--r--   1 root supergroup      17481 2010-09-17 11:03
/user/root/output/clusters-8/_logs/history/localhost_1284692116007_job_20100
9171055_0011_conf.xml

-rw-r--r--   1 root supergroup       9496 2010-09-17 11:03
/user/root/output/clusters-8/_logs/history/localhost_1284692116007_job_20100
9171055_0011_root_mahout-examples-0.3.job

-rw-r--r--   1 root supergroup       5757 2010-09-17 11:03
/user/root/output/clusters-8/part-00000

drwxr-xr-x   - root supergroup          0 2010-09-17 11:04
/user/root/output/clusters-9

drwxr-xr-x   - root supergroup          0 2010-09-17 11:04
/user/root/output/clusters-9/_logs

drwxr-xr-x   - root supergroup          0 2010-09-17 11:04
/user/root/output/clusters-9/_logs/history

-rw-r--r--   1 root supergroup      17481 2010-09-17 11:04
/user/root/output/clusters-9/_logs/history/localhost_1284692116007_job_20100
9171055_0012_conf.xml

-rw-r--r--   1 root supergroup       9496 2010-09-17 11:04
/user/root/output/clusters-9/_logs/history/localhost_1284692116007_job_20100
9171055_0012_root_mahout-examples-0.3.job

-rw-r--r--   1 root supergroup       5757 2010-09-17 11:04
/user/root/output/clusters-9/part-00000

drwxr-xr-x   - root supergroup          0 2010-09-17 10:59
/user/root/output/data

drwxr-xr-x   - root supergroup          0 2010-09-17 10:58
/user/root/output/data/_logs

drwxr-xr-x   - root supergroup          0 2010-09-17 10:58
/user/root/output/data/_logs/history

-rw-r--r--   1 root supergroup      16671 2010-09-17 10:58
/user/root/output/data/_logs/history/localhost_1284692116007_job_20100917105
5_0001_conf.xml

-rw-r--r--   1 root supergroup       6148 2010-09-17 10:58
/user/root/output/data/_logs/history/localhost_1284692116007_job_20100917105
5_0001_root_mahout-examples-0.3.job

-rw-r--r--   1 root supergroup     240672 2010-09-17 10:59
/user/root/output/data/part-00000

-rw-r--r--   1 root supergroup     242288 2010-09-17 10:59
/user/root/output/data/part-00001

drwxr-xr-x   - root supergroup          0 2010-09-17 11:04
/user/root/output/points

drwxr-xr-x   - root supergroup          0 2010-09-17 11:04
/user/root/output/points/_logs

drwxr-xr-x   - root supergroup          0 2010-09-17 11:04
/user/root/output/points/_logs/history

-rw-r--r--   1 root supergroup      17109 2010-09-17 11:04
/user/root/output/points/_logs/history/localhost_1284692116007_job_201009171
055_0013_conf.xml

-rw-r--r--   1 root supergroup       6124 2010-09-17 11:04
/user/root/output/points/_logs/history/localhost_1284692116007_job_201009171
055_0013_root_mahout-examples-0.3.job

-rw-r--r--   1 root supergroup     515948 2010-09-17 11:04
/user/root/output/points/part-00000

-rw-r--r--   1 root supergroup     512632 2010-09-17 11:04
/user/root/output/points/part-00001

 

Then i download the output file to local use "hadoop fs -get output output"

but when i view the result that kmeans created used the command: mahout
clusterdump -s output/clusters-9 -p output/points

it throws a exception:

running on hadoop, using HADOOP_HOME=/usr/hadoop-0.20.2 and
HADOOP_CONF_DIR=/usr/hadoop-0.20.2/conf

10/09/17 14:33:29 ERROR driver.MahoutDriver: MahoutDriver failed with args:
[-s, output/clusters-9, -p, output/points, null]

File does not exist: /usr/mahout-0.3/output/points/part-00000

Exception in thread "main" java.io.FileNotFoundException: File does not
exist: /usr/mahout-0.3/output/points/part-00000

        at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSy
stem.java:457)

        at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:676)

        at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417)

        at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412)

        at
org.apache.mahout.utils.clustering.ClusterDumper.readPoints(ClusterDumper.ja
va:330)

        at
org.apache.mahout.utils.clustering.ClusterDumper.init(ClusterDumper.java:85)

        at
org.apache.mahout.utils.clustering.ClusterDumper.<init>(ClusterDumper.java:7
8)

        at
org.apache.mahout.utils.clustering.ClusterDumper.main(ClusterDumper.java:276
)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver
.java:68)

        at
org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)

        at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:172)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

 

so, how to view the result of kmeans?

 

Best Wishes

 

Wangss


------=_NextPart_000_0004_01CB5676.A2D1A8D0--



From general-return-2115-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Sep 17 21:20:37 2010
Return-Path: <general-return-2115-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24983 invoked from network); 17 Sep 2010 21:20:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Sep 2010 21:20:37 -0000
Received: (qmail 33298 invoked by uid 500); 17 Sep 2010 21:20:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33050 invoked by uid 500); 17 Sep 2010 21:20:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33042 invoked by uid 99); 17 Sep 2010 21:20:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Sep 2010 21:20:35 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Sep 2010 21:20:30 +0000
Received: by qwk3 with SMTP id 3so2843232qwk.35
        for <general@hadoop.apache.org>; Fri, 17 Sep 2010 14:20:10 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.126.222 with SMTP id d30mr3196641qcs.223.1284758409938;
 Fri, 17 Sep 2010 14:20:09 -0700 (PDT)
Received: by 10.229.71.197 with HTTP; Fri, 17 Sep 2010 14:20:09 -0700 (PDT)
Date: Fri, 17 Sep 2010 14:20:09 -0700
Message-ID: <AANLkTi=hSVG3B_bGp6A2kr9NdYi+zE6uOVH0SWtg9F3V@mail.gmail.com>
Subject: What's Happening Around Hadoop World?
From: Jon Zuanich <jon.zuanich@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd66eaca5867204907b203f

--000e0cd66eaca5867204907b203f
Content-Type: text/plain; charset=ISO-8859-1

If anyone knows of other events happening around Hadoop World please let us
know and we will include them in our calendar.

These are items Cloudera has happening around Hadoop World:

*Oct 11:*
Training - Intro to Hadoop
Training - Hadoop Essentials for Managers
Training - Cloudera HUE SDK
6: Customer Appreciation Dinner
*Oct 12:*
6 - 7:30: Networking Reception
*Oct 13:*
Training - Developer Training & Certification
Training - Administrator Training & Certification
Training - Analyzing Data with Hive and Pig
*Oct 14:*
Training(cont.) - Developer Training & Certification
Training(cont.) - Administrator Training & Certification
Training(cont.) - Analyzing Data with Hive and Pig
*Oct 15:*
Training - HBase

Regards,
Jon

-- 
Jon Zuanich
Marketing Intern
Cloudera
jon.zuanich@cloudera.com
blog: cloudera.com/blog
twitter: twitter.com/cloudera
web: www.cloudera.com

*Hadoop World 2010 -- October 12 | New York -- *Register
today!<http://www.cloudera.com/company/press-center/hadoop-world-nyc/>

--000e0cd66eaca5867204907b203f--

From general-return-2116-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Sep 17 21:22:26 2010
Return-Path: <general-return-2116-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25419 invoked from network); 17 Sep 2010 21:22:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Sep 2010 21:22:25 -0000
Received: (qmail 35945 invoked by uid 500); 17 Sep 2010 21:22:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 35583 invoked by uid 500); 17 Sep 2010 21:22:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 35528 invoked by uid 99); 17 Sep 2010 21:22:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Sep 2010 21:22:23 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ryanobjc@gmail.com designates 209.85.161.176 as permitted sender)
Received: from [209.85.161.176] (HELO mail-gx0-f176.google.com) (209.85.161.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Sep 2010 21:22:18 +0000
Received: by gxk7 with SMTP id 7so1327503gxk.35
        for <general@hadoop.apache.org>; Fri, 17 Sep 2010 14:21:58 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=+THS3DEhq4WvYqckKURvgdok8YjSdxZMT1gMsUSIyOo=;
        b=yEGfcu28swY2SvQTnkvYsPS28yv8guMQSSsQx8eM8uGPEHS6m7/MOFZI7oJPy1/1Gi
         gpvnzdKGm1Zwy9EoQpTuBLGd3JBIcPiiU5sqJIXtbeuBNfVSPJEKLZQJ4JHyipgoiVwu
         j5foyLhIgvqlhqJs4KnCtC8KID5+fUvqfE8N8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=lm3/cbJbdfaNn64ryU/Sw5apw/V9RkrXwPT5aUXTg7eu/aRWl1SxlffqdcJcG+7op8
         2ftXNfOoK8YTmPxbCDIjteAMObuzBAfTKYyZX498nEzTgGWDU/GVwA4FxmfLP4WzeIFI
         uMoBjC49iOmkAxfBFNmuH75RJjN8G12pO2gzI=
MIME-Version: 1.0
Received: by 10.90.51.2 with SMTP id y2mr3852226agy.107.1284758518019; Fri, 17
 Sep 2010 14:21:58 -0700 (PDT)
Received: by 10.231.199.138 with HTTP; Fri, 17 Sep 2010 14:21:57 -0700 (PDT)
In-Reply-To: <AANLkTi=hSVG3B_bGp6A2kr9NdYi+zE6uOVH0SWtg9F3V@mail.gmail.com>
References: <AANLkTi=hSVG3B_bGp6A2kr9NdYi+zE6uOVH0SWtg9F3V@mail.gmail.com>
Date: Fri, 17 Sep 2010 14:21:57 -0700
Message-ID: <AANLkTim_Ra=r=1sSYdf=mMkCUy8nONkrdwD1NK92JBPJ@mail.gmail.com>
Subject: Re: What's Happening Around Hadoop World?
From: Ryan Rawson <ryanobjc@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

There is this HUG for HBase the night before:

http://www.meetup.com/hbaseusergroup/calendar/14606174/

-ryan

On Fri, Sep 17, 2010 at 2:20 PM, Jon Zuanich <jon.zuanich@cloudera.com> wrote:
> If anyone knows of other events happening around Hadoop World please let us
> know and we will include them in our calendar.
>
> These are items Cloudera has happening around Hadoop World:
>
> *Oct 11:*
> Training - Intro to Hadoop
> Training - Hadoop Essentials for Managers
> Training - Cloudera HUE SDK
> 6: Customer Appreciation Dinner
> *Oct 12:*
> 6 - 7:30: Networking Reception
> *Oct 13:*
> Training - Developer Training & Certification
> Training - Administrator Training & Certification
> Training - Analyzing Data with Hive and Pig
> *Oct 14:*
> Training(cont.) - Developer Training & Certification
> Training(cont.) - Administrator Training & Certification
> Training(cont.) - Analyzing Data with Hive and Pig
> *Oct 15:*
> Training - HBase
>
> Regards,
> Jon
>
> --
> Jon Zuanich
> Marketing Intern
> Cloudera
> jon.zuanich@cloudera.com
> blog: cloudera.com/blog
> twitter: twitter.com/cloudera
> web: www.cloudera.com
>
> *Hadoop World 2010 -- October 12 | New York -- *Register
> today!<http://www.cloudera.com/company/press-center/hadoop-world-nyc/>
>

From general-return-2117-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Sep 17 22:10:02 2010
Return-Path: <general-return-2117-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39741 invoked from network); 17 Sep 2010 22:10:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Sep 2010 22:10:01 -0000
Received: (qmail 97864 invoked by uid 500); 17 Sep 2010 22:10:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97800 invoked by uid 500); 17 Sep 2010 22:10:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97792 invoked by uid 99); 17 Sep 2010 22:10:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Sep 2010 22:10:00 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Sep 2010 22:09:55 +0000
Received: by qyk2 with SMTP id 2so3649100qyk.14
        for <general@hadoop.apache.org>; Fri, 17 Sep 2010 15:09:34 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.246.194 with SMTP id lz2mr3959820qcb.216.1284761372477;
 Fri, 17 Sep 2010 15:09:32 -0700 (PDT)
Received: by 10.229.71.197 with HTTP; Fri, 17 Sep 2010 15:09:32 -0700 (PDT)
In-Reply-To: <AANLkTim_Ra=r=1sSYdf=mMkCUy8nONkrdwD1NK92JBPJ@mail.gmail.com>
References: <AANLkTi=hSVG3B_bGp6A2kr9NdYi+zE6uOVH0SWtg9F3V@mail.gmail.com>
	<AANLkTim_Ra=r=1sSYdf=mMkCUy8nONkrdwD1NK92JBPJ@mail.gmail.com>
Date: Fri, 17 Sep 2010 15:09:32 -0700
Message-ID: <AANLkTimaJPG4OqOhqLOqXR=G0X7bX3HYFQBfBi-_UAqc@mail.gmail.com>
Subject: Re: What's Happening Around Hadoop World?
From: Jon Zuanich <jon.zuanich@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64f92003a497504907bd1c1

--0016e64f92003a497504907bd1c1
Content-Type: text/plain; charset=ISO-8859-1

Thanks Ryan,

I'll be sure to include it.

-Jon

On Fri, Sep 17, 2010 at 2:21 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:

> There is this HUG for HBase the night before:
>
> http://www.meetup.com/hbaseusergroup/calendar/14606174/
>
> -ryan
>
> On Fri, Sep 17, 2010 at 2:20 PM, Jon Zuanich <jon.zuanich@cloudera.com>
> wrote:
> > If anyone knows of other events happening around Hadoop World please let
> us
> > know and we will include them in our calendar.
> >
> > These are items Cloudera has happening around Hadoop World:
> >
> > *Oct 11:*
> > Training - Intro to Hadoop
> > Training - Hadoop Essentials for Managers
> > Training - Cloudera HUE SDK
> > 6: Customer Appreciation Dinner
> > *Oct 12:*
> > 6 - 7:30: Networking Reception
> > *Oct 13:*
> > Training - Developer Training & Certification
> > Training - Administrator Training & Certification
> > Training - Analyzing Data with Hive and Pig
> > *Oct 14:*
> > Training(cont.) - Developer Training & Certification
> > Training(cont.) - Administrator Training & Certification
> > Training(cont.) - Analyzing Data with Hive and Pig
> > *Oct 15:*
> > Training - HBase
> >
> > Regards,
> > Jon
> >
> > --
> > Jon Zuanich
> > Marketing Intern
> > Cloudera
> > jon.zuanich@cloudera.com
> > blog: cloudera.com/blog
> > twitter: twitter.com/cloudera
> > web: www.cloudera.com
> >
> > *Hadoop World 2010 -- October 12 | New York -- *Register
> > today!<http://www.cloudera.com/company/press-center/hadoop-world-nyc/>
> >
>



-- 
Jon Zuanich
Marketing Intern
Cloudera
jon.zuanich@cloudera.com
blog: cloudera.com/blog
twitter: twitter.com/cloudera
web: www.cloudera.com

*Hadoop World 2010 -- October 12 | New York -- *Register
today!<http://www.cloudera.com/company/press-center/hadoop-world-nyc/>

--0016e64f92003a497504907bd1c1--

From general-return-2118-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Sep 17 22:37:53 2010
Return-Path: <general-return-2118-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45313 invoked from network); 17 Sep 2010 22:37:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Sep 2010 22:37:53 -0000
Received: (qmail 20280 invoked by uid 500); 17 Sep 2010 22:37:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20234 invoked by uid 500); 17 Sep 2010 22:37:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 20218 invoked by uid 99); 17 Sep 2010 22:37:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Sep 2010 22:37:51 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ghelmling@gmail.com designates 209.85.210.48 as permitted sender)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Sep 2010 22:37:43 +0000
Received: by pzk4 with SMTP id 4so1246178pzk.35
        for <multiple recipients>; Fri, 17 Sep 2010 15:37:22 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:cc:content-type;
        bh=Iw3daIibIIaQXCiQg4j/dJ2RldYzfjl78CH0fPLlNeE=;
        b=Tu4V6Bz45S3PdJYS15ge91z1xTr+2j4D3uIZZzq+iGLHIL6It9XM4w4DtH4NkIECos
         /Um2NCJI0z6XFXGHPixOduEj/dVaTtm82CsVfDYv2S8WjXrpoQNMV2My/Q64lDN4q+bd
         ha0aOrVnxYQcOKfbtU0Wlp3v2nT0OkzSBbc3E=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=ILlDAGQSn4SgG84zSjK70Pw5xPGnhOJRWk6tegkGfIU4amcarKMOUa/7wJhM6JEVXG
         tA/ktJaCR9eJa4R8BmiNj20BFuvf6ShoIgrp45ByNzzDom482TkhbmkURHnO8O4IzJ5T
         J+SzpjpdiOBmTfuBJznZOOyBxKYFl1QJnAkVY=
MIME-Version: 1.0
Received: by 10.142.118.9 with SMTP id q9mr823260wfc.87.1284763042181; Fri, 17
 Sep 2010 15:37:22 -0700 (PDT)
Received: by 10.143.160.4 with HTTP; Fri, 17 Sep 2010 15:37:22 -0700 (PDT)
In-Reply-To: <AANLkTikJEJB3=ztmbcLxSRjxh-CQ7VQyAdFwF1VDe9H8@mail.gmail.com>
References: <AANLkTikJEJB3=ztmbcLxSRjxh-CQ7VQyAdFwF1VDe9H8@mail.gmail.com>
Date: Fri, 17 Sep 2010 15:37:22 -0700
Message-ID: <AANLkTinrwq1JNvkwcFgCr-6vmubHoX35VC8mHUGTZd8r@mail.gmail.com>
Subject: Re: Paths of evolution for Hadoop metrics
From: Gary Helmling <ghelmling@gmail.com>
To: general@hadoop.apache.org
Cc: common-dev@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636e0b854bff17504907c3451
X-Virus-Checked: Checked by ClamAV on apache.org

--001636e0b854bff17504907c3451
Content-Type: text/plain; charset=ISO-8859-1

>From the HBase perspective, life is complicated by the need to support
running on multiple Hadoop versions.  Current HBase trunk is really targeted
at 0.20-append, but I'm also working to add 0.20-security (or
0.20-append-security) to that.  And there have been questions (though I'm
not sure a clear decision) on support for 0.21 now that it's released.  And
obviously I can't predict where things will stand in the 0.22 timeframe.

Sounds like the new framework will have some great stuff allowing us to
strip out some of the HBase code we've added for metrics (built-on JMX
support), which would be great to make use of.  But until HBase is targeting
only 0.22+ I don't think we'll fully be able to convert to it.

So I'd vote for option #2 to allow us to make a phased transition.

--gh

--001636e0b854bff17504907c3451--

From general-return-2119-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Sep 19 17:12:24 2010
Return-Path: <general-return-2119-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94629 invoked from network); 19 Sep 2010 17:12:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Sep 2010 17:12:24 -0000
Received: (qmail 19047 invoked by uid 500); 19 Sep 2010 17:12:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18918 invoked by uid 500); 19 Sep 2010 17:12:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18910 invoked by uid 99); 19 Sep 2010 17:12:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Sep 2010 17:12:21 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of raks.mail@gmail.com designates 209.85.216.169 as permitted sender)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Sep 2010 17:12:15 +0000
Received: by qyk8 with SMTP id 8so2693372qyk.14
        for <general@hadoop.apache.org>; Sun, 19 Sep 2010 10:11:54 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=E3fELYS2j7fTmdTSVg2GZBlr8sK/XukRKm69DoQzDnA=;
        b=Jv1A5NiqSjHleZpsoiIm7WXGdRg1IZvyUiHTrUVwm9pXMZk4+V97fvOmSZUcdtmrfn
         Sj2YJrePTvQkFcYtXxh/TBAxGx0nyLxPBB8/Ie6yPklvU3VfTkgRxX0SjIoyqUBHxwLw
         r55mCFS+2v3nh5MkoI0rlE6TOyiUTfbeW7rX4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=a6+iiONCBtIsSictHL1WznxcktYr28FYi8VL6YS69A4ZkYV5h9zd6fJEyUus3kG2Be
         oJ6v8xqmFzz5TUDF+vD2JcQAffrBDFD0IQ0oRmwDzCD+UmHtufKWGjbKvAVcgTLulXQX
         c9q5vxeTl39Ov6Epmogb2YMknCMIvmhwwh55k=
MIME-Version: 1.0
Received: by 10.229.2.19 with SMTP id 19mr5038027qch.283.1284916313847; Sun,
 19 Sep 2010 10:11:53 -0700 (PDT)
Received: by 10.229.182.69 with HTTP; Sun, 19 Sep 2010 10:11:53 -0700 (PDT)
Date: Sun, 19 Sep 2010 12:11:53 -0500
Message-ID: <AANLkTin0K5nLphzJU-MNWxUZR_ZRA0rUzsGhVRkz=Mca@mail.gmail.com>
Subject: readFields() throws a NullPointerException
From: Rakesh Ramakrishnan <raks.mail@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00148539257a73f74b04909fe465

--00148539257a73f74b04909fe465
Content-Type: text/plain; charset=ISO-8859-1

I have a simple map-reduce program in which my map and reduce primitives
look like this

map(K,V) = (Text, OutputAggregator)
reduce(Text, OutputAggregator) = (Text,Text)

The important point is that from my map function I emit an object of type
OutputAggregator which is a custom class that implements the Writable
interface. However, my reduce fails with the following exception. More
specifically, the readFieds() function is throwing an exception. Any clue
why ? I use hadoop 0.18.3


10/09/19 04:04:59 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
10/09/19 04:04:59 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the
same.
10/09/19 04:04:59 INFO mapred.FileInputFormat: Total input paths to process : 1
10/09/19 04:04:59 INFO mapred.FileInputFormat: Total input paths to process : 1
10/09/19 04:04:59 INFO mapred.FileInputFormat: Total input paths to process : 1
10/09/19 04:04:59 INFO mapred.FileInputFormat: Total input paths to process : 1
10/09/19 04:04:59 INFO mapred.JobClient: Running job: job_local_0001
10/09/19 04:04:59 INFO mapred.MapTask: numReduceTasks: 1
10/09/19 04:04:59 INFO mapred.MapTask: io.sort.mb = 100
10/09/19 04:04:59 INFO mapred.MapTask: data buffer = 79691776/99614720
10/09/19 04:04:59 INFO mapred.MapTask: record buffer = 262144/327680
Length = 10
10
10/09/19 04:04:59 INFO mapred.MapTask: Starting flush of map output
10/09/19 04:04:59 INFO mapred.MapTask: bufstart = 0; bufend = 231;
bufvoid = 99614720
10/09/19 04:04:59 INFO mapred.MapTask: kvstart = 0; kvend = 10; length = 327680
gl_books
10/09/19 04:04:59 WARN mapred.LocalJobRunner: job_local_0001
java.lang.NullPointerException
 at org.myorg.OutputAggregator.readFields(OutputAggregator.java:46)
 at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
 at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
 at org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:751)
 at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:691)
 at org.apache.hadoop.mapred.Task$CombineValuesIterator.next(Task.java:770)
 at org.myorg.xxxParallelizer$Reduce.reduce(xxxParallelizer.java:117)
 at org.myorg.xxxParallelizer$Reduce.reduce(xxxParallelizer.java:1)
 at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.combineAndSpill(MapTask.java:904)
 at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:785)
 at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:698)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:228)
 at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:157)
java.io.IOException: Job failed!
 at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1113)
 at org.myorg.xxxParallelizer.main(xxxParallelizer.java:145)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
 at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
 at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)

--00148539257a73f74b04909fe465--

From general-return-2120-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Sep 19 17:23:13 2010
Return-Path: <general-return-2120-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97268 invoked from network); 19 Sep 2010 17:23:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Sep 2010 17:23:13 -0000
Received: (qmail 26105 invoked by uid 500); 19 Sep 2010 17:23:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26038 invoked by uid 500); 19 Sep 2010 17:23:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26030 invoked by uid 99); 19 Sep 2010 17:23:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Sep 2010 17:23:11 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [65.55.88.14] (HELO TX2EHSOBE007.bigfish.com) (65.55.88.14)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Sep 2010 17:23:02 +0000
Received: from mail41-tx2-R.bigfish.com (10.9.14.241) by
 TX2EHSOBE007.bigfish.com (10.9.40.27) with Microsoft SMTP Server id
 8.1.340.0; Sun, 19 Sep 2010 17:22:40 +0000
Received: from mail41-tx2 (localhost.localdomain [127.0.0.1])	by
 mail41-tx2-R.bigfish.com (Postfix) with ESMTP id 948E4B06FB	for
 <general@hadoop.apache.org>; Sun, 19 Sep 2010 17:22:40 +0000 (UTC)
X-SpamScore: -25
X-BigFish: VPS-25(zzbb2dKbb2cK542N328cM9371Pzz1202hzz8275dhz2dh2a8h43h)
Received: from mail41-tx2 (localhost.localdomain [127.0.0.1]) by mail41-tx2
 (MessageSwitch) id 128491696075281_24431; Sun, 19 Sep 2010 17:22:40 +0000
 (UTC)
Received: from TX2EHSMHS017.bigfish.com (unknown [10.9.14.241])	by
 mail41-tx2.bigfish.com (Postfix) with ESMTP id DA6C715D004F	for
 <general@hadoop.apache.org>; Sun, 19 Sep 2010 17:22:39 +0000 (UTC)
Received: from us-voo-smtp05.internal.sungard.corp (216.83.166.46) by
 TX2EHSMHS017.bigfish.com (10.9.99.117) with Microsoft SMTP Server (TLS) id
 14.0.482.44; Sun, 19 Sep 2010 17:22:39 +0000
Received: from us-voo-smtp12.internal.sungard.corp ([168.162.128.54]) by
 us-voo-smtp05.internal.sungard.corp with Microsoft SMTPSVC(6.0.3790.4675);
	 Sun, 19 Sep 2010 13:22:38 -0400
Received: from VOO-EXCHANGE07.internal.sungard.corp ([168.162.128.66]) by
 us-voo-smtp12.internal.sungard.corp with Microsoft SMTPSVC(6.0.3790.4675);
	 Sun, 19 Sep 2010 13:22:38 -0400
Received: from VOO-EXCHANGE04.internal.sungard.corp ([168.162.128.74]) by
 VOO-EXCHANGE07.internal.sungard.corp with Microsoft SMTPSVC(6.0.3790.4675);
	 Sun, 19 Sep 2010 13:22:38 -0400
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-Class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: RE: readFields() throws a NullPointerException
Date: Sun, 19 Sep 2010 13:21:56 -0400
Message-ID: <A7F1BE12AEBC6F4183969A7A00CEA67506289628@VOO-EXCHANGE04.internal.sungard.corp>
In-Reply-To: <AANLkTin0K5nLphzJU-MNWxUZR_ZRA0rUzsGhVRkz=Mca@mail.gmail.com>
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Thread-Topic: readFields() throws a NullPointerException
Thread-Index: ActYHdTtKb3orSj+T+aMbast0ZsLvAAADZ/g
References: <AANLkTin0K5nLphzJU-MNWxUZR_ZRA0rUzsGhVRkz=Mca@mail.gmail.com>
From: <Christopher.Shain@sungard.com>
To: <general@hadoop.apache.org>
X-OriginalArrivalTime: 19 Sep 2010 17:22:38.0648 (UTC) FILETIME=[421AE380:01CB581F]
X-Reverse-DNS: unknown

How are you constructing your WritableDeserializer?  The reason that I
ask is that on the line you are seeing an error:

writable.readFields(dataIn);

The only thing that could throw a null pointer exception is if writable
was null.  Writable is constructed thusly:

    public Writable deserialize(Writable w) throws IOException {
      Writable writable;
      if (w =3D=3D null) {
        writable=20
          =3D (Writable) ReflectionUtils.newInstance(writableClass,
getConf());
      } else {
        writable =3D w;
      }
      writable.readFields(dataIn);
      return writable;
    }

So I suspect that you are passing null as the argument to Deserialize
and when constructing your WritableDeserializer, you are passing null as
the second argument (Class<?> c).  This would result in writable being
null, and you'd see that error.

In one of these two cases you must define your Writable class.

Hope this helps,

Chris


-----Original Message-----
From: Rakesh Ramakrishnan [mailto:raks.mail@gmail.com]=20
Sent: Sunday, September 19, 2010 1:12 PM
To: general@hadoop.apache.org
Subject: readFields() throws a NullPointerException

I have a simple map-reduce program in which my map and reduce primitives
look like this

map(K,V) =3D (Text, OutputAggregator)
reduce(Text, OutputAggregator) =3D (Text,Text)

The important point is that from my map function I emit an object of
type
OutputAggregator which is a custom class that implements the Writable
interface. However, my reduce fails with the following exception. More
specifically, the readFieds() function is throwing an exception. Any
clue
why ? I use hadoop 0.18.3


10/09/19 04:04:59 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=3DJobTracker, sessionId=3D
10/09/19 04:04:59 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the
same.
10/09/19 04:04:59 INFO mapred.FileInputFormat: Total input paths to
process : 1
10/09/19 04:04:59 INFO mapred.FileInputFormat: Total input paths to
process : 1
10/09/19 04:04:59 INFO mapred.FileInputFormat: Total input paths to
process : 1
10/09/19 04:04:59 INFO mapred.FileInputFormat: Total input paths to
process : 1
10/09/19 04:04:59 INFO mapred.JobClient: Running job: job_local_0001
10/09/19 04:04:59 INFO mapred.MapTask: numReduceTasks: 1
10/09/19 04:04:59 INFO mapred.MapTask: io.sort.mb =3D 100
10/09/19 04:04:59 INFO mapred.MapTask: data buffer =3D 79691776/99614720
10/09/19 04:04:59 INFO mapred.MapTask: record buffer =3D 262144/327680
Length =3D 10
10
10/09/19 04:04:59 INFO mapred.MapTask: Starting flush of map output
10/09/19 04:04:59 INFO mapred.MapTask: bufstart =3D 0; bufend =3D 231;
bufvoid =3D 99614720
10/09/19 04:04:59 INFO mapred.MapTask: kvstart =3D 0; kvend =3D 10; =
length =3D
327680
gl_books
10/09/19 04:04:59 WARN mapred.LocalJobRunner: job_local_0001
java.lang.NullPointerException
 at org.myorg.OutputAggregator.readFields(OutputAggregator.java:46)
 at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializ
er.deserialize(WritableSerialization.java:67)
 at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializ
er.deserialize(WritableSerialization.java:40)
 at
org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:751
)
 at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:691)
 at
org.apache.hadoop.mapred.Task$CombineValuesIterator.next(Task.java:770)
 at org.myorg.xxxParallelizer$Reduce.reduce(xxxParallelizer.java:117)
 at org.myorg.xxxParallelizer$Reduce.reduce(xxxParallelizer.java:1)
 at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.combineAndSpill(MapTask
.java:904)
 at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.ja
va:785)
 at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:698)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:228)
 at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:157)
java.io.IOException: Job failed!
 at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1113)
 at org.myorg.xxxParallelizer.main(xxxParallelizer.java:145)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
 at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
 at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)


From general-return-2121-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Sep 19 21:04:40 2010
Return-Path: <general-return-2121-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72201 invoked from network); 19 Sep 2010 21:04:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Sep 2010 21:04:40 -0000
Received: (qmail 55801 invoked by uid 500); 19 Sep 2010 21:04:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55707 invoked by uid 500); 19 Sep 2010 21:04:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55699 invoked by uid 99); 19 Sep 2010 21:04:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Sep 2010 21:04:38 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [85.214.115.60] (HELO h1349259.stratoserver.net) (85.214.115.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Sep 2010 21:04:29 +0000
Received: from [188.106.135.35] (helo=motokokusanagi.localnet)
	by h1349259.stratoserver.net with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
	(Exim 4.69)
	(envelope-from <isabel@apache.org>)
	id 1OxR3Q-0004kD-P7
	for general@hadoop.apache.org; Sun, 19 Sep 2010 21:04:08 +0000
From: Isabel Drost <isabel@apache.org>
Reply-To: isabel@apache.org
To: general@hadoop.apache.org
Subject: Re: Keysigning party at Hadoop World
Date: Sun, 19 Sep 2010 23:04:00 +0200
User-Agent: KMail/1.13.5 (Linux/2.6.32-5-amd64; KDE/4.4.5; x86_64; ; )
References: <201009160934.01127.thomas@koch.ro> <AANLkTin3TvNCZ+Q0VhyLV9oOwJ4sMGe-avC155YMFMjp@mail.gmail.com>
In-Reply-To: <AANLkTin3TvNCZ+Q0VhyLV9oOwJ4sMGe-avC155YMFMjp@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed;
  boundary="nextPart1830513.mhGV8cDS4x";
  protocol="application/pgp-signature";
  micalg=pgp-sha1
Content-Transfer-Encoding: 7bit
Message-Id: <201009192304.00817.isabel@apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

--nextPart1830513.mhGV8cDS4x
Content-Type: Text/Plain;
  charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

On 16.09.2010 Owen O'Malley wrote:
> show we trust each other. *grin* If there is a key signing party at
> Apache World, we can get into the larger Apache web of trust.

Apache World? Do you mean Hadoop World or Apache Con NA? As for the latter =
one,=20
there usually is a key signing party at Apache Cons, so getting the Hadoop =
devs'=20
keys signed by other Apache committers shouldn't be a huge problem.

Isabel

--nextPart1830513.mhGV8cDS4x
Content-Type: application/pgp-signature; name=signature.asc 
Content-Description: This is a digitally signed message part.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEUEABECAAYFAkyWesAACgkQPJpCBAwIhbSb+gCePnDgh+Cp14oGTA1RwKY48nF4
oZ4AmKYMN2zisYyynf0s534iuPxVaQs=
=DrFv
-----END PGP SIGNATURE-----

--nextPart1830513.mhGV8cDS4x--

From general-return-2122-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Sep 19 21:28:18 2010
Return-Path: <general-return-2122-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 79320 invoked from network); 19 Sep 2010 21:28:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Sep 2010 21:28:17 -0000
Received: (qmail 68040 invoked by uid 500); 19 Sep 2010 21:28:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67811 invoked by uid 500); 19 Sep 2010 21:28:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67803 invoked by uid 99); 19 Sep 2010 21:28:15 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Sep 2010 21:28:15 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,MIME_QP_LONG_LINE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of owen.omalley@gmail.com designates 209.85.212.176 as permitted sender)
Received: from [209.85.212.176] (HELO mail-px0-f176.google.com) (209.85.212.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Sep 2010 21:27:51 +0000
Received: by pxi3 with SMTP id 3so1816473pxi.35
        for <general@hadoop.apache.org>; Sun, 19 Sep 2010 14:27:30 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:references:in-reply-to
         :mime-version:content-transfer-encoding:content-type:message-id:cc
         :x-mailer:from:subject:date:to;
        bh=sz7jmxPT+EFdOwEZd6ZdQoTiGFcTeMaKmMY2NXKg9Qo=;
        b=qkf4DISWFxT2dsLEeigS/r4tqPCSxC9m5n80uIeXqq017Lv5LuiUfeZ5SH0lq48HxZ
         eEg2Iu+h2LlmWcvwh4DQ5JKvdekSY+OL33tIwBhoclTGw7qrpq61KOYpAcCVEO5dNfRv
         kXlgJm5gCcYsCWsrm15AxhXpx8NW5ecSe4V3c=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=references:in-reply-to:mime-version:content-transfer-encoding
         :content-type:message-id:cc:x-mailer:from:subject:date:to;
        b=aHwLAsS6na2v1C3FEqN3iB3S6vDZE01bjxH00Lt/9y1J4YHpPVGMWcKyqBc8T4Veik
         pJmHiAXGt4UkQqCjACCu8YBFBcCfv8Vn+U6mdyc3zvgTtXGLOrgNjZdHswbB8eIYAfsv
         8SSsvtL8Vt8noUil9XXaG8eZmSmrWv8zT6bxU=
Received: by 10.142.218.2 with SMTP id q2mr6825958wfg.246.1284931650420;
        Sun, 19 Sep 2010 14:27:30 -0700 (PDT)
Received: from [10.8.190.144] ([166.205.137.185])
        by mx.google.com with ESMTPS id i20sm7643138wff.5.2010.09.19.14.27.27
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Sun, 19 Sep 2010 14:27:29 -0700 (PDT)
References: <201009160934.01127.thomas@koch.ro> <AANLkTin3TvNCZ+Q0VhyLV9oOwJ4sMGe-avC155YMFMjp@mail.gmail.com> <201009192304.00817.isabel@apache.org>
In-Reply-To: <201009192304.00817.isabel@apache.org>
Mime-Version: 1.0 (iPhone Mail 8A306)
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii
Message-Id: <9BA8FD19-6CA7-46AC-8C71-F30F92222006@gmail.com>
Cc: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: iPhone Mail (8A306)
From: Owen O'Malley <owen.omalley@gmail.com>
Subject: Re: Keysigning party at Hadoop World
Date: Sun, 19 Sep 2010 14:27:31 -0700
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

I meant ApacheCon, which with a few key signatures will tie us into the Apac=
he web of trust. I got my key signed by a Debian committer last week and man=
y of us will cross sign our keys at the Hadoop contributors meeting this wee=
k.=20

-- Owen=

From general-return-2123-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 20 01:34:37 2010
Return-Path: <general-return-2123-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 40550 invoked from network); 20 Sep 2010 01:34:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Sep 2010 01:34:36 -0000
Received: (qmail 56946 invoked by uid 500); 20 Sep 2010 01:34:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56865 invoked by uid 500); 20 Sep 2010 01:34:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56857 invoked by uid 99); 20 Sep 2010 01:34:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Sep 2010 01:34:34 +0000
X-ASF-Spam-Status: No, hits=0.6 required=10.0
	tests=FREEMAIL_FROM,HK_RANDOM_ENVFROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of zjffdu@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Sep 2010 01:34:29 +0000
Received: by wyf23 with SMTP id 23so5682145wyf.35
        for <general@hadoop.apache.org>; Sun, 19 Sep 2010 18:34:08 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=Ha6LF3O2GorIQR9E/DE0txiNtZBELVgA/01t8w58QT0=;
        b=I6XStFg0QFn0N0i3xEvpy8BwPjPWQ9zYdyX9onQoowDUHoN1ko9cdB0eO08KxSvIWQ
         ku7LT2cForfneliqM2WO70MsPr5Ohd15Zpg23XqXIIfB3c97In+c9xxhkXD+ibK2Te/l
         x5/hYvhq2FsYws5jOqaRwy/gxl8rSfNek4g+c=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=LlE8q72hVm+USLxxbi+0JkWLmo4E8xTOzmeK96gHSs24IhfYzYwNjlAIVTM/grQTQT
         +oQ+HPLIljuI550FgcgeEKmFjLXFyuFkAhqmMhkzQ3+sYV8b6SvHLSFy8p1DX7aflxW1
         x6AF0ckoezIm72MCDHeiZkdY+wGCe6gbJFvSw=
MIME-Version: 1.0
Received: by 10.216.23.199 with SMTP id v49mr3737687wev.43.1284946448006; Sun,
 19 Sep 2010 18:34:08 -0700 (PDT)
Received: by 10.216.175.70 with HTTP; Sun, 19 Sep 2010 18:34:07 -0700 (PDT)
In-Reply-To: <A7F1BE12AEBC6F4183969A7A00CEA67506289628@VOO-EXCHANGE04.internal.sungard.corp>
References: <AANLkTin0K5nLphzJU-MNWxUZR_ZRA0rUzsGhVRkz=Mca@mail.gmail.com>
	<A7F1BE12AEBC6F4183969A7A00CEA67506289628@VOO-EXCHANGE04.internal.sungard.corp>
Date: Mon, 20 Sep 2010 09:34:07 +0800
Message-ID: <AANLkTingCq6RKK-e-ZCE6Fhyb+dFLOaMDm23x_XSJv5m@mail.gmail.com>
Subject: Re: readFields() throws a NullPointerException
From: Jeff Zhang <zjffdu@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Do you have a no-argument constructor for your customer class and in
the constructor do you do initialize the members ? Otherwise these
members in your customer class will be null.


On Mon, Sep 20, 2010 at 1:21 AM,  <Christopher.Shain@sungard.com> wrote:
> How are you constructing your WritableDeserializer? =C2=A0The reason that=
 I
> ask is that on the line you are seeing an error:
>
> writable.readFields(dataIn);
>
> The only thing that could throw a null pointer exception is if writable
> was null. =C2=A0Writable is constructed thusly:
>
> =C2=A0 =C2=A0public Writable deserialize(Writable w) throws IOException {
> =C2=A0 =C2=A0 =C2=A0Writable writable;
> =C2=A0 =C2=A0 =C2=A0if (w =3D=3D null) {
> =C2=A0 =C2=A0 =C2=A0 =C2=A0writable
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=3D (Writable) ReflectionUtils.newInsta=
nce(writableClass,
> getConf());
> =C2=A0 =C2=A0 =C2=A0} else {
> =C2=A0 =C2=A0 =C2=A0 =C2=A0writable =3D w;
> =C2=A0 =C2=A0 =C2=A0}
> =C2=A0 =C2=A0 =C2=A0writable.readFields(dataIn);
> =C2=A0 =C2=A0 =C2=A0return writable;
> =C2=A0 =C2=A0}
>
> So I suspect that you are passing null as the argument to Deserialize
> and when constructing your WritableDeserializer, you are passing null as
> the second argument (Class<?> c). =C2=A0This would result in writable bei=
ng
> null, and you'd see that error.
>
> In one of these two cases you must define your Writable class.
>
> Hope this helps,
>
> Chris
>
>
> -----Original Message-----
> From: Rakesh Ramakrishnan [mailto:raks.mail@gmail.com]
> Sent: Sunday, September 19, 2010 1:12 PM
> To: general@hadoop.apache.org
> Subject: readFields() throws a NullPointerException
>
> I have a simple map-reduce program in which my map and reduce primitives
> look like this
>
> map(K,V) =3D (Text, OutputAggregator)
> reduce(Text, OutputAggregator) =3D (Text,Text)
>
> The important point is that from my map function I emit an object of
> type
> OutputAggregator which is a custom class that implements the Writable
> interface. However, my reduce fails with the following exception. More
> specifically, the readFieds() function is throwing an exception. Any
> clue
> why ? I use hadoop 0.18.3
>
>
> 10/09/19 04:04:59 INFO jvm.JvmMetrics: Initializing JVM Metrics with
> processName=3DJobTracker, sessionId=3D
> 10/09/19 04:04:59 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the
> same.
> 10/09/19 04:04:59 INFO mapred.FileInputFormat: Total input paths to
> process : 1
> 10/09/19 04:04:59 INFO mapred.FileInputFormat: Total input paths to
> process : 1
> 10/09/19 04:04:59 INFO mapred.FileInputFormat: Total input paths to
> process : 1
> 10/09/19 04:04:59 INFO mapred.FileInputFormat: Total input paths to
> process : 1
> 10/09/19 04:04:59 INFO mapred.JobClient: Running job: job_local_0001
> 10/09/19 04:04:59 INFO mapred.MapTask: numReduceTasks: 1
> 10/09/19 04:04:59 INFO mapred.MapTask: io.sort.mb =3D 100
> 10/09/19 04:04:59 INFO mapred.MapTask: data buffer =3D 79691776/99614720
> 10/09/19 04:04:59 INFO mapred.MapTask: record buffer =3D 262144/327680
> Length =3D 10
> 10
> 10/09/19 04:04:59 INFO mapred.MapTask: Starting flush of map output
> 10/09/19 04:04:59 INFO mapred.MapTask: bufstart =3D 0; bufend =3D 231;
> bufvoid =3D 99614720
> 10/09/19 04:04:59 INFO mapred.MapTask: kvstart =3D 0; kvend =3D 10; lengt=
h =3D
> 327680
> gl_books
> 10/09/19 04:04:59 WARN mapred.LocalJobRunner: job_local_0001
> java.lang.NullPointerException
> =C2=A0at org.myorg.OutputAggregator.readFields(OutputAggregator.java:46)
> =C2=A0at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializ
> er.deserialize(WritableSerialization.java:67)
> =C2=A0at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializ
> er.deserialize(WritableSerialization.java:40)
> =C2=A0at
> org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:751
> )
> =C2=A0at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:691)
> =C2=A0at
> org.apache.hadoop.mapred.Task$CombineValuesIterator.next(Task.java:770)
> =C2=A0at org.myorg.xxxParallelizer$Reduce.reduce(xxxParallelizer.java:117=
)
> =C2=A0at org.myorg.xxxParallelizer$Reduce.reduce(xxxParallelizer.java:1)
> =C2=A0at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.combineAndSpill(MapTask
> .java:904)
> =C2=A0at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.ja
> va:785)
> =C2=A0at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:698)
> =C2=A0at org.apache.hadoop.mapred.MapTask.run(MapTask.java:228)
> =C2=A0at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:157)
> java.io.IOException: Job failed!
> =C2=A0at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1113)
> =C2=A0at org.myorg.xxxParallelizer.main(xxxParallelizer.java:145)
> =C2=A0at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> =C2=A0at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> =C2=A0at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> =C2=A0at java.lang.reflect.Method.invoke(Unknown Source)
> =C2=A0at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
> =C2=A0at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
> =C2=A0at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> =C2=A0at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> =C2=A0at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
>
>



--=20
Best Regards

Jeff Zhang

From general-return-2124-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 20 01:37:17 2010
Return-Path: <general-return-2124-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 41125 invoked from network); 20 Sep 2010 01:37:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Sep 2010 01:37:17 -0000
Received: (qmail 59088 invoked by uid 500); 20 Sep 2010 01:37:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59023 invoked by uid 500); 20 Sep 2010 01:37:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59015 invoked by uid 99); 20 Sep 2010 01:37:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Sep 2010 01:37:15 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of raks.mail@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Sep 2010 01:37:10 +0000
Received: by qwk3 with SMTP id 3so3943743qwk.35
        for <general@hadoop.apache.org>; Sun, 19 Sep 2010 18:36:49 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=W5g2ypvSEjjCYZ3qtfNoshZml6nkbVg8PglxksaUvxs=;
        b=x6N3Cvu/bWusuoG8ejv1L5s26dgyHlwi9dafFxeDeoA8dcX3klSpMR8G1DRKF84ez0
         GZ28JMQPMHnAf5rUVO2CnPBjpR4L0wSZWeWrRUjG8Oh66tkxuRkeeNupI6cVPvEYLRLy
         Z0iv4rE85XUF8DwOnBjZWd8gDBx1hZL33ka40=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=nFgsXa/KQotEXGjAl2csOauuQj7LYqfAxJNFkt2x7bYczLOO09/8P7tPaHSDvI1u5M
         nQNxP47McdbsO0YaS7XEtBBVMQ3n8iroeEGyd+p110T1S5ScXmEK6B2vh/2+Wdx909eI
         /Ax6yzOLtkyomEkx8xV3qKp12VYGNQSqzVTp8=
MIME-Version: 1.0
Received: by 10.224.28.139 with SMTP id m11mr3937756qac.350.1284946609614;
 Sun, 19 Sep 2010 18:36:49 -0700 (PDT)
Received: by 10.229.182.69 with HTTP; Sun, 19 Sep 2010 18:36:49 -0700 (PDT)
In-Reply-To: <AANLkTingCq6RKK-e-ZCE6Fhyb+dFLOaMDm23x_XSJv5m@mail.gmail.com>
References: <AANLkTin0K5nLphzJU-MNWxUZR_ZRA0rUzsGhVRkz=Mca@mail.gmail.com>
	<A7F1BE12AEBC6F4183969A7A00CEA67506289628@VOO-EXCHANGE04.internal.sungard.corp>
	<AANLkTingCq6RKK-e-ZCE6Fhyb+dFLOaMDm23x_XSJv5m@mail.gmail.com>
Date: Sun, 19 Sep 2010 18:36:49 -0700
Message-ID: <AANLkTi=HN+-2pf=xTRmrsRhkEEeo13+HK10W0VZ2R+Ne@mail.gmail.com>
Subject: Re: readFields() throws a NullPointerException
From: Rakesh Ramakrishnan <raks.mail@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175cdfac38ae640490a6f261

--0015175cdfac38ae640490a6f261
Content-Type: text/plain; charset=ISO-8859-1

Thanks Jeff. Yes, I was not initializing the members. It works fine now.

On Sun, Sep 19, 2010 at 6:34 PM, Jeff Zhang <zjffdu@gmail.com> wrote:

> Do you have a no-argument constructor for your customer class and in
> the constructor do you do initialize the members ? Otherwise these
> members in your customer class will be null.
>
>
> On Mon, Sep 20, 2010 at 1:21 AM,  <Christopher.Shain@sungard.com> wrote:
> > How are you constructing your WritableDeserializer?  The reason that I
> > ask is that on the line you are seeing an error:
> >
> > writable.readFields(dataIn);
> >
> > The only thing that could throw a null pointer exception is if writable
> > was null.  Writable is constructed thusly:
> >
> >    public Writable deserialize(Writable w) throws IOException {
> >      Writable writable;
> >      if (w == null) {
> >        writable
> >          = (Writable) ReflectionUtils.newInstance(writableClass,
> > getConf());
> >      } else {
> >        writable = w;
> >      }
> >      writable.readFields(dataIn);
> >      return writable;
> >    }
> >
> > So I suspect that you are passing null as the argument to Deserialize
> > and when constructing your WritableDeserializer, you are passing null as
> > the second argument (Class<?> c).  This would result in writable being
> > null, and you'd see that error.
> >
> > In one of these two cases you must define your Writable class.
> >
> > Hope this helps,
> >
> > Chris
> >
> >
> > -----Original Message-----
> > From: Rakesh Ramakrishnan [mailto:raks.mail@gmail.com]
> > Sent: Sunday, September 19, 2010 1:12 PM
> > To: general@hadoop.apache.org
> > Subject: readFields() throws a NullPointerException
> >
> > I have a simple map-reduce program in which my map and reduce primitives
> > look like this
> >
> > map(K,V) = (Text, OutputAggregator)
> > reduce(Text, OutputAggregator) = (Text,Text)
> >
> > The important point is that from my map function I emit an object of
> > type
> > OutputAggregator which is a custom class that implements the Writable
> > interface. However, my reduce fails with the following exception. More
> > specifically, the readFieds() function is throwing an exception. Any
> > clue
> > why ? I use hadoop 0.18.3
> >
> >
> > 10/09/19 04:04:59 INFO jvm.JvmMetrics: Initializing JVM Metrics with
> > processName=JobTracker, sessionId=
> > 10/09/19 04:04:59 WARN mapred.JobClient: Use GenericOptionsParser for
> > parsing the arguments. Applications should implement Tool for the
> > same.
> > 10/09/19 04:04:59 INFO mapred.FileInputFormat: Total input paths to
> > process : 1
> > 10/09/19 04:04:59 INFO mapred.FileInputFormat: Total input paths to
> > process : 1
> > 10/09/19 04:04:59 INFO mapred.FileInputFormat: Total input paths to
> > process : 1
> > 10/09/19 04:04:59 INFO mapred.FileInputFormat: Total input paths to
> > process : 1
> > 10/09/19 04:04:59 INFO mapred.JobClient: Running job: job_local_0001
> > 10/09/19 04:04:59 INFO mapred.MapTask: numReduceTasks: 1
> > 10/09/19 04:04:59 INFO mapred.MapTask: io.sort.mb = 100
> > 10/09/19 04:04:59 INFO mapred.MapTask: data buffer = 79691776/99614720
> > 10/09/19 04:04:59 INFO mapred.MapTask: record buffer = 262144/327680
> > Length = 10
> > 10
> > 10/09/19 04:04:59 INFO mapred.MapTask: Starting flush of map output
> > 10/09/19 04:04:59 INFO mapred.MapTask: bufstart = 0; bufend = 231;
> > bufvoid = 99614720
> > 10/09/19 04:04:59 INFO mapred.MapTask: kvstart = 0; kvend = 10; length =
> > 327680
> > gl_books
> > 10/09/19 04:04:59 WARN mapred.LocalJobRunner: job_local_0001
> > java.lang.NullPointerException
> >  at org.myorg.OutputAggregator.readFields(OutputAggregator.java:46)
> >  at
> > org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializ
> > er.deserialize(WritableSerialization.java:67)
> >  at
> > org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializ
> > er.deserialize(WritableSerialization.java:40)
> >  at
> > org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:751
> > )
> >  at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:691)
> >  at
> > org.apache.hadoop.mapred.Task$CombineValuesIterator.next(Task.java:770)
> >  at org.myorg.xxxParallelizer$Reduce.reduce(xxxParallelizer.java:117)
> >  at org.myorg.xxxParallelizer$Reduce.reduce(xxxParallelizer.java:1)
> >  at
> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.combineAndSpill(MapTask
> > .java:904)
> >  at
> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.ja
> > va:785)
> >  at
> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:698)
> >  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:228)
> >  at
> > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:157)
> > java.io.IOException: Job failed!
> >  at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1113)
> >  at org.myorg.xxxParallelizer.main(xxxParallelizer.java:145)
> >  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >  at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> >  at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> >  at java.lang.reflect.Method.invoke(Unknown Source)
> >  at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
> >  at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
> >  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> >  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> >  at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
> >
> >
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>

--0015175cdfac38ae640490a6f261--

From general-return-2125-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 20 03:05:27 2010
Return-Path: <general-return-2125-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 74558 invoked from network); 20 Sep 2010 03:05:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Sep 2010 03:05:26 -0000
Received: (qmail 13766 invoked by uid 500); 20 Sep 2010 03:05:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13420 invoked by uid 500); 20 Sep 2010 03:05:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13407 invoked by uid 99); 20 Sep 2010 03:05:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Sep 2010 03:05:23 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of kengoou@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Sep 2010 03:05:14 +0000
Received: by qwk3 with SMTP id 3so3985325qwk.35
        for <general@hadoop.apache.org>; Sun, 19 Sep 2010 20:04:53 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:received:dkim-signature
         :domainkey-signature:received:mime-version:received:from:date
         :message-id:subject:to:content-type;
        bh=h04NWuAhZjmGfoLLP39GrB7Y/GlHmbgekYQVx/qdCyg=;
        b=Nn1dbH6dJ4o4JWBSDNcO+qqNOerIXMw7I7BR5SYZcSCesSRSYzJ0Kw1IyLHOyoxZtg
         Q9GgXumlIq77RRRfyRxskvwK6qSq9n7dWdPi1mwtCxXeDteEeY+WbmP7nRV3MwYS+I3m
         Gt18caMUYnjcDbrhFu7iTCs1SLnFOHDIKzUB4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=dkim-signature:domainkey-signature:mime-version:from:date
         :message-id:subject:to:content-type;
        b=ZcG1ABp8D677ZXWEzqWbH3alkdKGbUUyCOxhHvsu7UScRUqB0tWSuNLOJGp6XNkDAV
         NMCaD6dqXH+x8OQn7OMyab7cmED/7uSylEEZN5bo8iJdJ+whAM2kMyJd39Q0AYLD3Ffd
         bznWR4KsEDy6Qx5l0gRw/dVZsZyyNaH+H7M8E=
Received: by 10.229.79.76 with SMTP id o12mr5784325qck.40.1284951892989;
        Sun, 19 Sep 2010 20:04:52 -0700 (PDT)
Received: from mail-qy0-f169.google.com (mail-qy0-f169.google.com [209.85.216.169])
        by mx.google.com with ESMTPS id e6sm7072348qcr.29.2010.09.19.20.04.51
        (version=SSLv3 cipher=RC4-MD5);
        Sun, 19 Sep 2010 20:04:52 -0700 (PDT)
Received: by qyk8 with SMTP id 8so2986269qyk.14
        for <general@hadoop.apache.org>; Sun, 19 Sep 2010 20:04:51 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:from:date
         :message-id:subject:to:content-type;
        bh=h04NWuAhZjmGfoLLP39GrB7Y/GlHmbgekYQVx/qdCyg=;
        b=UYIQCocxGLVTKW3jF4Y71GHDHvDV/hx0TJXDyvQJcft8mZzeuMrGfb3C2TIS/y7Txq
         IQtn0b+he8QhPhJQ91gzZ9WLjmyQVDHnNAbruP5c6UI1miv28eM8cJvomwaQouIMOlAq
         220nTfEBWkklnOw0EVBcchdHI133dLhBMhs3A=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type;
        b=k3TwTaBM+W9s+OynWgS5wyff3Tn6Vqx5hgnS1dK8JZn3XKy4mg/i7VgChGH591TToX
         +ksBREaNlVnWL+EW7JU4BKgZlzXQJ3K/sxkaWPPfV3eH4FrEbwgp9xtDpfLNfuRYbmxX
         AYmPqJo0CFveEqeTR0h5A3WgeDdq/JvW3tVds=
Received: by 10.224.69.14 with SMTP id x14mr5415131qai.212.1284951891113; Sun,
 19 Sep 2010 20:04:51 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.232.144 with HTTP; Sun, 19 Sep 2010 20:04:31 -0700 (PDT)
From: KENGOOU <kengoou@gmail.com>
Date: Mon, 20 Sep 2010 12:04:31 +0900
Message-ID: <AANLkTingU1kPiRRDFabu8txyimT67jRkH84obbDLmEtZ@mail.gmail.com>
Subject: about "Hadoop in Action"
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Nice to meet you.
I am KENGOOU.

Hadoop is researched in Japan.
Recently, it began to read "Hadoop in Action".
It wants to make this book a Japanese translation and to publish it in
Japan. I am embarrassed because there is no report method though I
want to take the permission of a Japanese translation in author's
Dr.Chuck Lam.

The answer was not able to be gotten though it E-mailed to Manning of
the publisher.

Could you teach if there is something a good method to publish a
Japanese translation?

Please my best regards.

KENGOOOU

From general-return-2126-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 20 13:23:33 2010
Return-Path: <general-return-2126-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95783 invoked from network); 20 Sep 2010 13:23:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Sep 2010 13:23:32 -0000
Received: (qmail 67606 invoked by uid 500); 20 Sep 2010 13:23:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67156 invoked by uid 500); 20 Sep 2010 13:23:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67145 invoked by uid 99); 20 Sep 2010 13:23:27 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Sep 2010 13:23:27 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Sep 2010 13:23:02 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 04425B7D9A
	for <general@hadoop.apache.org>; Mon, 20 Sep 2010 14:22:41 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id oqalsHxN2OKh for <general@hadoop.apache.org>;
	Mon, 20 Sep 2010 14:22:41 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 3C15AB7D97
	for <general@hadoop.apache.org>; Mon, 20 Sep 2010 14:22:41 +0100 (BST)
MailScanner-NULL-Check: 1285593747.47099@9gkL+qoW2zQ4t2fokcL8pw
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o8KDMQte024039
	for <general@hadoop.apache.org>; Mon, 20 Sep 2010 14:22:26 +0100 (BST)
Message-ID: <4C976012.1070806@apache.org>
Date: Mon, 20 Sep 2010 14:22:26 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.12) Gecko/20100826 Thunderbird/3.0.7
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: about "Hadoop in Action"
References: <AANLkTingU1kPiRRDFabu8txyimT67jRkH84obbDLmEtZ@mail.gmail.com>
In-Reply-To: <AANLkTingU1kPiRRDFabu8txyimT67jRkH84obbDLmEtZ@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o8KDMQte024039
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 20/09/10 04:04, KENGOOU wrote:
> Nice to meet you.
> I am KENGOOU.
>
> Hadoop is researched in Japan.
> Recently, it began to read "Hadoop in Action".
> It wants to make this book a Japanese translation and to publish it in
> Japan. I am embarrassed because there is no report method though I
> want to take the permission of a Japanese translation in author's
> Dr.Chuck Lam.
>
> The answer was not able to be gotten though it E-mailed to Manning of
> the publisher.
>
> Could you teach if there is something a good method to publish a
> Japanese translation?
>
> Please my best regards.
>
> KENGOOOU

Hi, I'll put you in touch with the publisher, you can talk to them

From general-return-2127-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Sep 23 16:45:08 2010
Return-Path: <general-return-2127-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24503 invoked from network); 23 Sep 2010 16:45:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Sep 2010 16:45:08 -0000
Received: (qmail 74793 invoked by uid 500); 23 Sep 2010 16:45:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74651 invoked by uid 500); 23 Sep 2010 16:45:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74642 invoked by uid 99); 23 Sep 2010 16:45:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Sep 2010 16:45:06 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of swhitecross@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Sep 2010 16:44:59 +0000
Received: by pwj8 with SMTP id 8so842371pwj.35
        for <general@hadoop.apache.org>; Thu, 23 Sep 2010 09:44:39 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=y0owZQIlHYMQ1nBobXrT1o4Wl3XcX7sRFI/PToPlmKw=;
        b=lebQ6edJ6Vr3HcxaLmtfgpSv6HwpXmEmC/CoCpEDN3CCSaodhwsK7kXhm34bC2rBHB
         0WhJUQV9nUSkxHFvsP8Q+jlJ/lwLRvqbLDxwkrFhqqeojCB24jR9uqJEgEiyirQWTCw+
         en4f28Ew8icX/UtrgF7ibXbq9ybcN1qtIMHF8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=TgqsYMVLRMAIi57J2lNm3pbjlcruM54678BUzMo+GyAAuVvTQnycSrafLfj27fM218
         vzpVKYDMKgwsnlOwEx+Bg+o7Zi2eLoxlJnBvVTLoRLuBCGKPrrLteNgg/9nw3Ha+VrT1
         kulTb8eoKtX9+dD1jA9KNdPLSd0enG7P8TeZ0=
MIME-Version: 1.0
Received: by 10.114.27.4 with SMTP id a4mr2134411waa.44.1285260279437; Thu, 23
 Sep 2010 09:44:39 -0700 (PDT)
Received: by 10.220.12.138 with HTTP; Thu, 23 Sep 2010 09:44:39 -0700 (PDT)
Date: Thu, 23 Sep 2010 12:44:39 -0400
Message-ID: <AANLkTimr2bWXh8zxGXyWeA=6ac4aDikTxk+XAW17kp6L@mail.gmail.com>
Subject: Common EOFException Causes?
From: Scott Whitecross <swhitecross@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636417f7d665bae0490effa43

--001636417f7d665bae0490effa43
Content-Type: text/plain; charset=ISO-8859-1

Hi all -

What are the typical causes for EOFExceptions during a Hadoop job?  I've
seen references to not enough xcievers, but I can't find any logs
complaining explicitly about this case.

Thanks.

--001636417f7d665bae0490effa43--

From general-return-2128-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Sep 23 21:52:17 2010
Return-Path: <general-return-2128-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55300 invoked from network); 23 Sep 2010 21:52:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Sep 2010 21:52:16 -0000
Received: (qmail 33198 invoked by uid 500); 23 Sep 2010 21:52:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32965 invoked by uid 500); 23 Sep 2010 21:52:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32957 invoked by uid 99); 23 Sep 2010 21:52:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Sep 2010 21:52:14 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of kengoou@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Sep 2010 21:52:08 +0000
Received: by qwk3 with SMTP id 3so1806922qwk.35
        for <general@hadoop.apache.org>; Thu, 23 Sep 2010 14:51:47 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:received:dkim-signature
         :domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type;
        bh=P/x1jG45FyuoPakgZr9Vl41MJQrXoPWobLK8b3WaU/8=;
        b=GmNP+H+UygZQOyn0GH33lSynBdejJW2utHXp66+tjGvxVc9nV0ZRGUGkZGBzXIOBnx
         J3f28Lv3hWjTuNKUBxtb37TEEMz2lazt3ZqabkaytPcLJwjgxajJlm+sg0Yc1hvHs168
         0kE+9FQwMSha8QX6pCOyBgHTLvgs2q2vYXQSw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=dkim-signature:domainkey-signature:mime-version:in-reply-to
         :references:from:date:message-id:subject:to:content-type;
        b=Eu+/wKS5NnI/cv78ti007NdzqsvnQdoy/FUSHg3ECkT28doOZ46KyCM7uM7aiuvIDc
         3w0NeVFS8PidBW4jc3fpjNrHulHni+OybktEAuadzfuNBO5jB9tVwqQP91SN4fVmvkhi
         are0A7rxjQGsnQ1k6Dd/2O9KnKV7Evjfs4b+o=
Received: by 10.224.71.143 with SMTP id h15mr1735248qaj.217.1285278706919;
        Thu, 23 Sep 2010 14:51:46 -0700 (PDT)
Received: from mail-qw0-f48.google.com (mail-qw0-f48.google.com [209.85.216.48])
        by mx.google.com with ESMTPS id t1sm1449234qcs.9.2010.09.23.14.51.45
        (version=SSLv3 cipher=RC4-MD5);
        Thu, 23 Sep 2010 14:51:46 -0700 (PDT)
Received: by qwk3 with SMTP id 3so1806900qwk.35
        for <general@hadoop.apache.org>; Thu, 23 Sep 2010 14:51:45 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type;
        bh=P/x1jG45FyuoPakgZr9Vl41MJQrXoPWobLK8b3WaU/8=;
        b=h3WG/eLaINEh1sIigSlvPKdZffAyF+d/e93b8msQcqB83dE6yWQHdNzDTsz6YvGoyd
         uVozf/DjZNlseOPFx13Qv2gP4HMbL+Cg1SSn98X/ipH2gupQtH63ghV6c32wdl8+YWOz
         glEBaBwBhKxVTOl1zcHPCSBRGwABrbX8/4MK8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=KNZdgagtxbzDzRkNcVfiTK8BGl3PWYErb7bS3U2nHXDmFTqtehX1Th7FBkiv12H2Ms
         bsK/C8mL52/91MrYxKC0Uze5Nzife8OwHEJ/TqKPdCZWjkz70dJwpdYoKsP1M0tbyKGM
         Ywl4d9hYGcRtr9ku1LNLQ2MHjA1c9QzoBo1jE=
Received: by 10.224.28.207 with SMTP id n15mr1811534qac.48.1285278705029; Thu,
 23 Sep 2010 14:51:45 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.222.130 with HTTP; Thu, 23 Sep 2010 14:51:24 -0700 (PDT)
In-Reply-To: <4C976012.1070806@apache.org>
References: <AANLkTingU1kPiRRDFabu8txyimT67jRkH84obbDLmEtZ@mail.gmail.com> <4C976012.1070806@apache.org>
From: KENGOOU <kengoou@gmail.com>
Date: Fri, 24 Sep 2010 06:51:24 +0900
Message-ID: <AANLkTi=sRA87Fcps7ruhepoNuq2sj9e70U1mHVRH3xkU@mail.gmail.com>
Subject: Re: about "Hadoop in Action"
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Thank you Mr,Steve

KENGOOU

>
> Hi, I'll put you in touch with the publisher, you can talk to them
>

From general-return-2129-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 28 14:38:07 2010
Return-Path: <general-return-2129-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 90734 invoked from network); 28 Sep 2010 14:38:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Sep 2010 14:38:06 -0000
Received: (qmail 68465 invoked by uid 500); 28 Sep 2010 14:38:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67924 invoked by uid 500); 28 Sep 2010 14:38:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67908 invoked by uid 99); 28 Sep 2010 14:38:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Sep 2010 14:38:00 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=FREEMAIL_FROM,FROM_LOCAL_NOVOWEL,HK_RANDOM_ENVFROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of sssssssenator@gmail.com designates 74.125.82.42 as permitted sender)
Received: from [74.125.82.42] (HELO mail-ww0-f42.google.com) (74.125.82.42)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Sep 2010 14:37:52 +0000
Received: by wwi18 with SMTP id 18so177462wwi.5
        for <multiple recipients>; Tue, 28 Sep 2010 07:37:32 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=4zebXU42Kgy8uM02TF8Ncl7uUMLDbxDGqgr7SLEeKAM=;
        b=pEpF2owki0RM9sMoIIxoBOzPSz3G6QWhBrG0hmBRpAdVkwodxsglsnYd5tanQpNoiE
         cWr/VrtIp4LTZDiKGQChb0rJBmwwrsFy9Z95EQOpFbSfzWkrwW7+qjVsBACxO4lqIn2P
         H8JOsGgVaunaC8SiIAxsGS+mUYG7KXsb+7bn8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=J01BB5YWsCYzyFXM8+tj7WMqyk/qcoc0XKAjRKmeo8rwpd5DCscKq6kLbuPKTRlZtE
         aVjC4Zf33qRAMf1q4V5lw1O0ELL6aGPT6siI8D28xnAkz3rIDwMeRa0MJTCcwAtTzHyg
         KONTBka2yur3a/5t+EtW990ubYN2gkS4fWIIA=
MIME-Version: 1.0
Received: by 10.216.159.6 with SMTP id r6mr1153706wek.55.1285684651770; Tue,
 28 Sep 2010 07:37:31 -0700 (PDT)
Received: by 10.216.170.202 with HTTP; Tue, 28 Sep 2010 07:37:31 -0700 (PDT)
Date: Tue, 28 Sep 2010 20:07:31 +0530
Message-ID: <AANLkTimgLyuOc1T+ytb-0ESSC5ENx+ZpJGtawZs=w_ZZ@mail.gmail.com>
Subject: Hive Error: while executing select count(*)
From: vaibhav negi <sssssssenator@gmail.com>
To: hive-user@hadoop.apache.org, general@hadoop.apache.org, 
	user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f44d18f64330049152c8df
X-Virus-Checked: Checked by ClamAV on apache.org

--001485f44d18f64330049152c8df
Content-Type: text/plain; charset=ISO-8859-1

Hi,

I am using hadoop version 0.20.2

While running query "select count(*) from table;" on Hive,i am getting this
error

FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.MapRedTask

Error in task tracker logs says :-

Error: java.lang.NullPointerException
 at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)

On debugging i found that ReduceTask.java:2683  line is using host which is
coming null

ReduceTask.java:2683 ---
 List loc = mapLocations.get(host);

One user in Hadoop Forum says HADOOP-4744 should be included in hadoop. Is
this patch required for hadoop 0.20.2 ?
How to add patch?

Is this an issue with /etc/hosts file?

Please help

Regards
Vaibhav Negi

--001485f44d18f64330049152c8df--

From general-return-2130-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Sep 29 12:01:24 2010
Return-Path: <general-return-2130-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 28653 invoked from network); 29 Sep 2010 12:01:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Sep 2010 12:01:23 -0000
Received: (qmail 54906 invoked by uid 500); 29 Sep 2010 12:01:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54107 invoked by uid 500); 29 Sep 2010 12:01:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54091 invoked by uid 99); 29 Sep 2010 12:01:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Sep 2010 12:01:17 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=FREEMAIL_FROM,FROM_LOCAL_NOVOWEL,HK_RANDOM_ENVFROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of sssssssenator@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Sep 2010 12:01:10 +0000
Received: by wyg36 with SMTP id 36so771888wyg.35
        for <multiple recipients>; Wed, 29 Sep 2010 05:00:49 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=/e58WF9lh6AUjUfYRO7O+X/19CJqK6+fipZ2ZDDpWZU=;
        b=V8ysTnCkZ2WtISLzvCX7R3HyJwJOTlUHVaxLtNktbbO4MEwQLxxEC0FQR5adUHW3py
         oWHnjwL2KyUpNDTha/lyMxTIeh2r9nliFW/Ll/rolwRGeGYhX857SsuVEofZN6xoKIxz
         qj8h4JTuSlT0j1OkyBS0jQ+dHVl1dFrNkFZ0s=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=drR6rDxY5NNUcAJOqzHr03/ghZdQ9aqsfSmjgYz1pR/ixQi1gRuvMkcJCwIK0+Sf71
         Nx3eHWWtil8QrkkyMIc94mkET/qCoN1ExhU0XEBCxvHmZ7G0FyMn3ve3JxZdAwwWt5mz
         35D7QVhZzGdSRFxtV9qm4N/qSkZBv5ir8Ng4A=
MIME-Version: 1.0
Received: by 10.216.234.93 with SMTP id r71mr2393631weq.104.1285761649354;
 Wed, 29 Sep 2010 05:00:49 -0700 (PDT)
Received: by 10.216.170.202 with HTTP; Wed, 29 Sep 2010 05:00:49 -0700 (PDT)
Date: Wed, 29 Sep 2010 17:30:49 +0530
Message-ID: <AANLkTikxJpWFSrkQMrY9AHeNG+DuD+Enc7qOCXGK_+T_@mail.gmail.com>
Subject: Error: java.lang.NullPointerException
From: vaibhav negi <sssssssenator@gmail.com>
To: hive-user@hadoop.apache.org, general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00151758a95e6028a6049164b610

--00151758a95e6028a6049164b610
Content-Type: text/plain; charset=ISO-8859-1

Hi,

I am using hadoop version 0.20.2

While running query "select count(*) from table;" on Hive,i am getting this
error

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.
exec.MapRedTask

Error in task tracker logs says :-

Error: java.lang.NullPointerException
 at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)


How to debug and correct it??


Vaibhav Negi

--00151758a95e6028a6049164b610--

From general-return-2131-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Sep 30 11:54:04 2010
Return-Path: <general-return-2131-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85719 invoked from network); 30 Sep 2010 11:54:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Sep 2010 11:54:04 -0000
Received: (qmail 81954 invoked by uid 500); 30 Sep 2010 11:54:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80987 invoked by uid 500); 30 Sep 2010 11:53:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80952 invoked by uid 99); 30 Sep 2010 11:53:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Sep 2010 11:53:56 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=FREEMAIL_FROM,FROM_LOCAL_NOVOWEL,HK_RANDOM_ENVFROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of sssssssenator@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Sep 2010 11:53:50 +0000
Received: by wwb22 with SMTP id 22so1230988wwb.29
        for <multiple recipients>; Thu, 30 Sep 2010 04:53:28 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=dYKgdJibQj/QG4bEbvTrHC8MchG+lP/v1vKqvX7pDHE=;
        b=bWE44+uODDRw+468WGq1kl7iXZ88UopwM3VGL4hbpJcDkMf8Gmz32t8W5oEVCKvI+8
         N3JHFv3p13k+11JRSPeG60j5/LEV8YFXoFN3zfw/jrTEyGXMg1ucgtZ6ZK8/CGEOYcaP
         +4Q8R9stvjHD8lnhfcC28IE3AzdELzj3wG95E=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=gHQj+EW/qsleTe/QJ/+xF0rXWc1Er/7euGOfKum4pkaBLBzsuT7M2ppf9w8ErFNzTf
         eoLxAkbkg1pg4gIdcHs5H8msvYIUXpjP5FEfHtBndoOpdrVDSVhCLHpHdYvn5qrEoQ+w
         hOV6iIFEYdTFJ3AUxeLNfDsato2ToPlJvDB6g=
MIME-Version: 1.0
Received: by 10.216.91.16 with SMTP id g16mr2848306wef.78.1285847114804; Thu,
 30 Sep 2010 04:45:14 -0700 (PDT)
Received: by 10.216.170.202 with HTTP; Thu, 30 Sep 2010 04:45:14 -0700 (PDT)
In-Reply-To: <AANLkTikxJpWFSrkQMrY9AHeNG+DuD+Enc7qOCXGK_+T_@mail.gmail.com>
References: <AANLkTikxJpWFSrkQMrY9AHeNG+DuD+Enc7qOCXGK_+T_@mail.gmail.com>
Date: Thu, 30 Sep 2010 17:15:14 +0530
Message-ID: <AANLkTimn+5k4pYmSLV+EXCiPtf5tVhCTWS1DUQiKLfr6@mail.gmail.com>
Subject: Re: Error: java.lang.NullPointerException
From: vaibhav negi <sssssssenator@gmail.com>
To: hive-user@hadoop.apache.org, general@hadoop.apache.org, 
	hive-dev@hadoop.apache.org, mapreduce-user@hadoop.apache.org, 
	hdfs-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d7ec6c836df70491789cb0

--0016e6d7ec6c836df70491789cb0
Content-Type: text/plain; charset=ISO-8859-1

Hello,

I am stuck at it. Please help

Regards
Vaibhav Negi


On Wed, Sep 29, 2010 at 5:30 PM, vaibhav negi <sssssssenator@gmail.com>wrote:

> Hi,
>
> I am using hadoop version 0.20.2
>
> While running query "select count(*) from table;" on Hive,i am getting this
> error
>
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.
> exec.MapRedTask
>
> Error in task tracker logs says :-
>
> Error: java.lang.NullPointerException
>  at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683)
> at
> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605)
>
>
> How to debug and correct it??
>
>
> Vaibhav Negi
>

--0016e6d7ec6c836df70491789cb0--

From general-return-2132-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Sep 30 20:14:16 2010
Return-Path: <general-return-2132-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 75242 invoked from network); 30 Sep 2010 20:14:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Sep 2010 20:14:15 -0000
Received: (qmail 33976 invoked by uid 500); 30 Sep 2010 20:14:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33920 invoked by uid 500); 30 Sep 2010 20:14:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33912 invoked by uid 99); 30 Sep 2010 20:14:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Sep 2010 20:14:13 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Sep 2010 20:14:09 +0000
Received: by qyk7 with SMTP id 7so2678397qyk.14
        for <general@hadoop.apache.org>; Thu, 30 Sep 2010 13:13:47 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.224.89.80 with SMTP id d16mr2940966qam.169.1285877622726; Thu,
 30 Sep 2010 13:13:42 -0700 (PDT)
Received: by 10.229.62.140 with HTTP; Thu, 30 Sep 2010 13:13:42 -0700 (PDT)
Date: Thu, 30 Sep 2010 13:13:42 -0700
Message-ID: <AANLkTinWoofczj=CxvqGqKO-oQ4R=c4sdOYU9ofk8yFa@mail.gmail.com>
Subject: New Content Added @ Hadoop World
From: Jon Zuanich <jon.zuanich@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175cb29aed5fac04917fb628

--0015175cb29aed5fac04917fb628
Content-Type: text/plain; charset=ISO-8859-1

We would like to inform you that the program guide <http://bit.ly/cHkRI2> for
Hadoop World was released this morning. Feel free to browse and plan the
path you would like take at the conference.

We have received so much great content from interesting companies within the
Hadoop community that we decided to add 5 new breakout sessions to the Hadoop
World Agenda <http://bit.ly/9Mp3kT>.

Looking forward to seeing you on October 12th in New York City!

Cheers,
Jon

-- 
Reminder: Hadoop World Hash Tag>> #hw2010

Jon Zuanich
Marketing Intern
Cloudera
jon.zuanich@cloudera.com
blog: cloudera.com/blog
twitter: twitter.com/cloudera
web: www.cloudera.com

*Hadoop World 2010 -- October 12 | New York -- *Register
today!<http://www.cloudera.com/company/press-center/hadoop-world-nyc/>

--0015175cb29aed5fac04917fb628--

From general-return-672-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Nov 01 16:56:20 2009
Return-Path: <general-return-672-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55948 invoked from network); 1 Nov 2009 16:56:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Nov 2009 16:56:19 -0000
Received: (qmail 46182 invoked by uid 500); 1 Nov 2009 16:56:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46095 invoked by uid 500); 1 Nov 2009 16:56:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46085 invoked by uid 99); 1 Nov 2009 16:56:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Nov 2009 16:56:17 +0000
X-ASF-Spam-Status: No, hits=-0.5 required=5.0
	tests=BAYES_00,HTML_MESSAGE,SUBJ_ALL_CAPS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cabiwan@gmail.com designates 72.14.220.156 as permitted sender)
Received: from [72.14.220.156] (HELO fg-out-1718.google.com) (72.14.220.156)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Nov 2009 16:56:15 +0000
Received: by fg-out-1718.google.com with SMTP id 16so139674fgg.11
        for <general@hadoop.apache.org>; Sun, 01 Nov 2009 08:55:53 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=OUwVrJyuT44K79VOwoKfhMj92Yti8GMRvRZ2ksFjB3k=;
        b=Vdv0lqjSBYZT4Ac70KdUNCx1dkE+FN7HXFIMDh/2squ5eNVk6ji2J8S+yUJu4Qrj1n
         NX0yHsRmM7AVEfTfbvkC/ZUQtDEkMSZUkFz0wjdV8OHN+S3FzGfIyD2jEMKFchpJo5XP
         HOfPpubc+yVcYPugCr2EvJuE4BsRtyeB8lX8w=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=vdYc1GlxL0UTqYPcfaS346BuxzkV0mprWABVgpGgcv1BF3Gp2G9yfPMJVLGZWP8WoM
         npQ81yBQLjm2TdBYaHBjSAKgWiNxbrXosLYWHx6F4VO8X0lczLVM6bxzKSluSip08qXj
         VYqBcMZMVVsYSaNuVHX9MtbgxyQcZmr6KtFUM=
MIME-Version: 1.0
Received: by 10.103.48.19 with SMTP id a19mr1657915muk.136.1257094553854; Sun, 
	01 Nov 2009 08:55:53 -0800 (PST)
Date: Sun, 1 Nov 2009 17:55:53 +0100
Message-ID: <309f76d00911010855q155d8efdw12eab526feac9432@mail.gmail.com>
Subject: TWO QUESTIONS ABOUT LOGS & DEBUGGING IN HADOOP
From: Alberto Luengo Cabanillas <cabiwan@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e659f40454e43a0477522252

--0016e659f40454e43a0477522252
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi everyone! In a project involving Hadoop (0.20.1) I=B4m working, I=B4d ne=
ed to
know if I=B4m reading properly the content of a file located in the HDFS.
Beside the try...catch block, is there any other option for printing values
in the console (in a similar way standard IO Java mechanisms do, like
System.out.println("the variable is ", var))?
Another thing, is there a simply way to debug Hadoop applications? (actuall=
y
I=B4m working in pseudo-distributed mode, for developing purposes).
Thanks a lot in advance.

--=20
Alberto

--0016e659f40454e43a0477522252--

From general-return-673-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 02 14:25:25 2009
Return-Path: <general-return-673-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 32397 invoked from network); 2 Nov 2009 14:25:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Nov 2009 14:25:25 -0000
Received: (qmail 48178 invoked by uid 500); 2 Nov 2009 14:25:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47659 invoked by uid 500); 2 Nov 2009 14:25:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47549 invoked by uid 99); 2 Nov 2009 14:25:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Nov 2009 14:25:23 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of abhilaash_cse@tce.edu designates 210.212.252.72 as permitted sender)
Received: from [210.212.252.72] (HELO tce.edu) (210.212.252.72)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Nov 2009 14:25:20 +0000
Received: from [127.0.0.1] (helo=mail.tce.edu)
	by tce.edu with esmtp (Exim 4.63)
	(envelope-from <abhilaash_cse@tce.edu>)
	id 1N4xpu-00051L-5k
	for general@hadoop.apache.org; Mon, 02 Nov 2009 19:54:46 +0530
Received: from 59.92.125.114
        (SquirrelMail authenticated user abhilaash_cse)
        by mail.tce.edu with HTTP;
        Mon, 2 Nov 2009 19:54:46 +0530
Message-ID: <f9d76217185ad6ffef5d3414f2390152.squirrel@mail.tce.edu>
Date: Mon, 2 Nov 2009 19:54:46 +0530
Subject: Http 404 Error while viewing tasktracker in Browser
From: "Abhilaash" <abhilaash_cse@tce.edu>
To: general@hadoop.apache.org
User-Agent: SquirrelMail/1.4.19
MIME-Version: 1.0
Content-Type: text/plain;charset=iso-8859-1
Content-Transfer-Encoding: 8bit
X-Priority: 3 (Normal)
Importance: Normal
X-XheaderVersion: 1.1
X-UserAgent: 
X-Spam-Score: -1.4 (-)
X-Old-Spam-Status: No


Hi all,

      I have downloaded hadoop and configured it in my local system.I have
run the wordcount program with some files as dataset .It run
successfully but when I opened the browser to see GUI for
tasktracker and job tracker , it shows me the following error:

HTTP ERROR: 404

/jobtracker.jsp

RequestURI=/jobtracker.jsp

Powered by Jetty://


Can anyone please help me and tell how to see results in GUI?


Thank you in advance,

R.Abhilaash.


-- 
Nothing is impossible because impossible says i'm possible...


-----------------------------------------
This email was sent using TCEMail Service.
Thiagarajar College of Engineering
Madurai-625 015, India


From general-return-674-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 02 19:52:47 2009
Return-Path: <general-return-674-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 74219 invoked from network); 2 Nov 2009 19:52:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Nov 2009 19:52:47 -0000
Received: (qmail 84256 invoked by uid 500); 2 Nov 2009 19:52:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84168 invoked by uid 500); 2 Nov 2009 19:52:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84156 invoked by uid 99); 2 Nov 2009 19:52:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Nov 2009 19:52:45 +0000
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.27.243] (HELO QMTA13.emeryville.ca.mail.comcast.net) (76.96.27.243)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Nov 2009 19:52:34 +0000
Received: from OMTA09.emeryville.ca.mail.comcast.net ([76.96.30.20])
	by QMTA13.emeryville.ca.mail.comcast.net with comcast
	id 0Jkj1d00d0S2fkCADKsDre; Mon, 02 Nov 2009 19:52:13 +0000
Received: from tryarticle-lm.corp.yahoo.com ([209.131.62.113])
	by OMTA09.emeryville.ca.mail.comcast.net with comcast
	id 0Ks41d0082SbwD58VKs6Sl; Mon, 02 Nov 2009 19:52:10 +0000
Message-Id: <284E0999-F935-47A2-B051-7241BF8A4B39@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <4AEB7728.3000008@yahoo-inc.com>
Content-Type: multipart/mixed; boundary=Apple-Mail-11--288551153
Mime-Version: 1.0 (Apple Message framework v936)
Subject: [VOTE] Proposed bylaws for the Hadoop project
Date: Mon, 2 Nov 2009 11:52:02 -0800
References: <b23e01b50909291345g47ffb267l76fe193ea72652a3@mail.gmail.com>	 <FE14B90F-1F11-4F68-8A6B-3F00DB99D050@apache.org>	 <4ACCBEE7.3070105@apache.org>	 <DDB254D5-1F8E-4B75-B8C1-3B3D095BBBC3@yahoo-inc.com>	 <4ACE1292.6000702@apache.org>	 <b23e01b50910140711y545fa031u7fb199594db33bd6@mail.gmail.com>	 <4ADE4055.3020002@yahoo-inc.com>	 <5764186A-DA78-412B-ADD4-5A9F50866732@yahoo-inc.com> <b23e01b50910291356w17c98e18md0239f30c2aa0a5@mail.gmail.com> <4AEB7728.3000008@yahoo-inc.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-11--288551153
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit

We should have created bylaws when we were established as a PMC back  
in Jan 2008, but it has become clear that we need to define them. I  
looked around and the Ant project had bylaws that were both clear and  
fairly complete. I made some minor edits to match what we do. Please  
look them over and vote. In a recursive usage, let's use 2/3 majority  
to approve these bylaws.

-- Owen


--Apple-Mail-11--288551153
Content-Disposition: attachment;
	filename=bylaws.xml
Content-Type: text/xml;
	x-unix-mode=0644;
	name="bylaws.xml"
Content-Transfer-Encoding: 7bit

<?xml version="1.0"?>
<!--
     Licensed to the Apache Software Foundation (ASF) under one or more
   contributor license agreements.  See the NOTICE file distributed with
   this work for additional information regarding copyright ownership.
   The ASF licenses this file to You under the Apache License, Version 2.0
   (the "License"); you may not use this file except in compliance with
   the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
-->
<document>

  <properties>
    <title>Project Bylaws</title>
    <author email="">Apache Hadoop PMC</author>
  </properties>

<body>
  <section name="Apache Hadoop Project Bylaws">
    <p>
      This document defines the bylaws under which the Apache Hadoop project
      operates. It defines the roles and responsibilities of the
      project, who may vote, how voting works, how conflicts are resolved,
      etc.
    </p>

    <p>
      Hadoop is a project of the
      <a href="http://www.apache.org/foundation/">Apache Software
      Foundation</a>.  The foundation holds the copyright on Apache
      code including the code in the Hadoop codebase. The
      <a href="http://www.apache.org/foundation/faq.html">foundation FAQ</a>
      explains the operation and background of the foundation.
    </p>

    <p>
      Hadoop is typical of Apache projects in that it operates under a set of
      principles, known collectively as the &quot;Apache Way&quot;. If you are
      new to Apache development, please refer to the
      <a href="http://incubator.apache.org">Incubator project</a>
      for more information on how Apache projects operate.
    </p>

    <ul>
        <li><a href="#Roles and Responsibilities">Roles and Responsibilities</a></li>
        <li><a href="#Decision Making">How decisions are made</a></li>
    </ul>

  </section>

  <section name="Roles and Responsibilities">
    <p>
      Apache projects define a set of roles with associated rights and
      responsibilities. These roles govern what tasks an individual may perform
      within the project. The roles are defined in the following sections
    </p>

    <ul>
      <li><a href="#Users">Users</a></li>
      <li><a href="#Developers">Developers</a></li>
      <li><a href="#Committers">Committers</a></li>
      <li><a href="#Project Management Committee">
        Project Management Committee (PMC)</a>
      </li>
    </ul>

    <subsection name="Users">
      <p>
        The most important participants in the project are people who use our
        software. The majority of our developers start out as users and guide
        their development efforts from the user's perspective.
      </p>

      <p>
        Users contribute to the Apache projects by providing feedback to
        developers in the form of bug reports and feature suggestions. As
        well, users participate in the Apache community by helping other users
        on mailing lists and user support forums.
      </p>

    </subsection>

    <subsection name="Developers">
      <p>
        All of the volunteers who are contributing time, code, documentation,
        or resources to the Hadoop Project. A developer that makes sustained,
        welcome contributions to the project may be invited to become a
        Committer, though the exact timing of such invitations depends on many
        factors.
      </p>
    </subsection>

    <subsection name="Committers">
      <p>
        The project's Committers are responsible for the project's
        technical management. Committers have access to a specified
        set of subproject's subversion repositories. Committers on
        subprojects may cast binding votes on any technical discussion
        regarding that subproject.
      </p>

      <p>
        Committer access is by invitation only and must be approved by lazy
        consensus of the active PMC members. A Committer is considered emeritus
        by their own declaration or by not contributing in any form to the
        project for over six months. An emeritus committer may request
        reinstatement of commit access from the PMC. Such reinstatement is
        subject to lazy consensus of active PMC members.
      </p>

      <p>
        Commit access can be revoked by a unanimous vote of all the
        active PMC members (except the committer in question if they
        are also a PMC member).
      </p>

      <p>
        All Apache committers are required to have a signed Contributor License
        Agreement (CLA) on file with the Apache Software Foundation. There is a
        <a href="http://www.apache.org/dev/committers.html">Committer FAQ</a>
        which provides more details on the requirements for Committers
      </p>

      <p>
        A committer who makes a sustained contribution to the project may be
        invited to become a member of the PMC. The form of contribution is
        not limited to code. It can also include code review, helping out
        users on the mailing lists, documentation, etc.
      </p>
    </subsection>
    <subsection name="Project Management Committee">
      <p>
        The Project Management Committee (PMC) for Apache Hadoop was
        created by the Apache Board in January 2008 when Hadoop moved
        out of Lucene and became a top level project at Apache. The
        PMC is responsible to the board and the ASF for the management
        and oversight of the Apache Hadoop codebase. The responsibilities
        of the PMC include
      </p>

      <ul>
        <li>Deciding what is distributed as products of the Apache
        Hadoop project.  In particular all releases must be approved
        by the PMC
        </li>
        <li>Maintaining the project's shared resources, including the codebase
            repository, mailing lists, websites.
        </li>
        <li>Speaking on behalf of the project.
        </li>
        <li>Resolving license disputes regarding products of the project
        </li>
        <li>Nominating new PMC members and committers
        </li>
        <li>Maintaining these bylaws and other guidelines of the project
        </li>
      </ul>

      <p>
        Membership of the PMC is by invitation only and must be approved by a
        lazy consensus of active PMC members. A PMC member is considered
        &quot;emeritus&quot; by their own declaration or by not contributing in
        any form to the project for over six months. An emeritus member may
        request reinstatement to the PMC. Such reinstatement is subject to lazy
        consensus of the active PMC members. Membership of the PMC can be
        revoked by an unanimous vote of all the active PMC members other than
        the member in question.
      </p>

      <p>
        The chair of the PMC is appointed by the ASF board. The chair
        is an office holder of the Apache Software Foundation (Vice
        President, Apache Hadoop) and has primary responsibility to
        the board for the management of the projects within the scope
        of the Hadoop PMC. The chair reports to the board quarterly on
        developments within the Hadoop project.
      </p>

      <p>
	When the current chair of the PMC resigns, the PMC votes to
	recommend a new chair using lazy consensus, but the decision
	must be ratified by the Apache board.
      </p>
    </subsection>
  </section>

  <section name="Decision Making">
    <p>
      Within the Hadoop project, different types of decisions require different
      forms of approval. For example, the
      <a href="#Roles and Responsibilities">previous section</a> describes
      several decisions which require &quot;lazy consensus&quot; approval. This
      section defines how voting is performed, the types of approvals, and which
      types of decision require which type of approval.
    </p>

    <subsection name="Voting">
      <p>
        Decisions regarding the project are made by votes on the
        primary project development mailing list
        (general@hadoop.apache.org).  Where necessary, PMC voting may
        take place on the private Hadoop PMC mailing list.  Votes are
        clearly indicated by subject line starting with [VOTE].  Votes
        may contain multiple items for approval and these should be
        clearly separated. Voting is carried out by replying to the
        vote mail. Voting may take four flavours
      </p>

      <table>
        <tr>
          <td><strong>+1</strong></td>
          <td>
            &quot;Yes,&quot; &quot;Agree,&quot; or &quot;the action should be
            performed.&quot; In general, this vote also indicates a willingness
            on the behalf of the voter in &quot;making it happen&quot;
          </td>
        </tr>

        <tr>
          <td><strong>+0</strong></td>
          <td>
            This vote indicates a willingness for the action under
            consideration to go ahead. The voter, however will not be able
            to help.
          </td>
        </tr>

        <tr>
          <td><strong>-0</strong></td>
          <td>
            This vote indicates that the voter does not, in general, agree with
            the proposed action but is not concerned enough to prevent the
            action going ahead.
          </td>
        </tr>

        <tr>
          <td><strong>-1</strong></td>
          <td>
            This is a negative vote. On issues where consensus is required,
            this vote counts as a <strong>veto</strong>. All vetoes must
            contain an explanation of why the veto is appropriate. Vetoes with
            no explanation are void. It may also be appropriate for a -1 vote
            to include an alternative course of action.
          </td>
        </tr>
      </table>

      <p>
        All participants in the Hadoop project are encouraged to show their
        agreement with or against a particular action by voting. For technical
        decisions, only the votes of active committers are binding. Non binding
        votes are still useful for those with binding votes to understand the
        perception of an action in the wider Hadoop community. For PMC decisions,
        only the votes of PMC members are binding.
      </p>

      <p>
        Voting can also be applied to changes made to the Hadoop codebase. These
        typically take the form of a veto (-1) in reply to the commit message
        sent when the commit is made.
      </p>
    </subsection>

    <subsection name="Approvals">
      <p>
        These are the types of approvals that can be sought. Different actions
        require different types of approvals
      </p>

      <table>
        <tr>
          <td><strong>Consensus</strong></td>
          <td>
            For this to pass, all voters with binding votes must vote and there
            can be no binding vetoes (-1). Consensus votes are rarely required
            due to the impracticality of getting all eligible voters to cast a
            vote.
          </td>
        </tr>

        <tr>
          <td><strong>Lazy Consensus</strong></td>
          <td>
            Lazy consensus requires 3 binding +1 votes and no binding vetoes.
          </td>
        </tr>

        <tr>
          <td><strong>Lazy Majority</strong></td>
          <td>
            A lazy majority vote requires 3 binding +1 votes and more binding +1
            votes that -1 votes.
          </td>
        </tr>

        <tr>
          <td><strong>Lazy Approval</strong></td>
          <td>
            An action with lazy approval is implicitly allowed unless a -1 vote
            is received, at which time, depending on the type of action, either
            lazy majority or lazy consensus approval must be obtained.
          </td>
        </tr>

        <tr>
          <td><strong>2/3 Majority</strong></td>
          <td>
            Some actions require a 2/3 majority of active committers or PMC
            members to pass. Such actions typically affect the foundation
            of the project (e.g. adopting a new codebase to replace an existing
            product). The higher threshold is designed to ensure such changes
            are strongly supported. To pass this vote requires at least 2/3 of
            binding vote holders to vote +1
          </td>
        </tr>
      </tablex>
    </subsection>

    <subsection name="Vetoes">
      <p>
        A valid, binding veto cannot be overruled. If a veto is cast, it must be
        accompanied by a valid reason explaining the reasons for the veto. The
        validity of a veto, if challenged, can be confirmed by anyone who has
        a binding vote. This does not necessarily  signify agreement with the
        veto - merely that the veto is valid.
      </p>

      <p>
        If you disagree with a valid veto, you must lobby the person casting
        the veto to withdraw their veto. If a veto is not withdrawn, the action
        that has been vetoed must be reversed in a timely manner.
      </p>
    </subsection>

    <subsection name="Actions">
      <p>
        This section describes the various actions which are undertaken within
        the project, the corresponding approval required for that action and
        those who have binding votes over the action.
      </p>

      <table>
        <tr>
          <th>Action</th>
          <th>Description</th>
          <th>Approval</th>
          <th>Binding Votes</th>
        </tr>
        <tr>
          <td><strong>Code Change</strong></td>
          <td>
            A change made to a codebase of the project and committed
            by a committer. This includes source code, documentation, website
            content, etc.
          </td>
          <td>
            Lazy approval and then lazy consensus.
          </td>
          <td>
            Active committers.
          </td>
        </tr>
        <tr>
          <td><strong>Release Plan</strong></td>
          <td>
            Defines the timetable and actions for a release. The plan also
            nominates a Release Manager.
          </td>
          <td>
            Lazy majority
          </td>
          <td>
            Active committers
          </td>
        </tr>
        <tr>
          <td><strong>Product Release</strong></td>
          <td>
            When a release of one of the project's products is ready, a vote is
            required to accept the release as an official release of the
            project.
          </td>
          <td>
            Lazy Majority
          </td>
          <td>
            Active PMC members
          </td>
        </tr>
        <tr>
          <td><strong>Adoption of New Codebase</strong></td>
          <td>
            <p>
            When the codebase for an existing, released product is to be
            replaced with an alternative codebase. If such a vote fails to
            gain approval, the existing code base will continue.
            </p>

            <p>
            This also covers the creation of new sub-projects
            within the project
            </p>
          </td>
          <td>
            2/3 majority
          </td>
          <td>
            Active committers
          </td>
        </tr>
        <tr>
          <td><strong>New Committer</strong></td>
          <td>
            When a new committer is proposed for the project
          </td>
          <td>
            Lazy consensus
          </td>
          <td>
            Active PMC members
          </td>
        </tr>
        <tr>
          <td><strong>New PMC Member</strong></td>
          <td>
            When a committer is proposed for the PMC
          </td>
          <td>
            Lazy consensus
          </td>
          <td>
            Active PMC members
          </td>
        </tr>
        <tr>
          <td><strong>Committer Removal</strong></td>
          <td>
            <p>When removal of commit privileges is sought.</p>
            <p><b>Note: </b> Such actions will also be referred to the ASF
            board by the PMC chair</p>
          </td>
          <td>
            Consensus
          </td>
          <td>
            Active PMC members (excluding the committer in question if a
            member of the PMC).
          </td>
        </tr>
        <tr>
          <td><strong>PMC Member Removal</strong></td>
          <td>
            <p>When removal of a PMC member is sought.</p>
            <p><b>Note: </b> Such actions will also be referred to the
            ASF board by the PMC chair</p>
          </td>
          <td>
            Consensus
          </td>
          <td>
            Active PMC members (excluding the member in question).
          </td>
        </tr>
        <tr>
          <td><strong>Modifying Bylaws</strong></td>
          <td>
            <p>Modifying this document.</p>
          </td>
          <td>
            2/3 majority
          </td>
          <td>
            Active PMC members
          </td>
        </tr>
      </table>
    </subsection>
    <subsection name="Voting Timeframes">
      <p>
        Votes are open for a period of 3 work days to allow all active voters
        time to consider the vote. Votes relating to code changes are not
        subject to a strict timetable but should be made as timely as possible.
      </p>
    </subsection>
  </section>
</body>
</document>

--Apple-Mail-11--288551153
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed
Content-Transfer-Encoding: 7bit



--Apple-Mail-11--288551153--

From general-return-675-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 02 20:41:51 2009
Return-Path: <general-return-675-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1557 invoked from network); 2 Nov 2009 20:41:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Nov 2009 20:41:51 -0000
Received: (qmail 69130 invoked by uid 500); 2 Nov 2009 20:41:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69025 invoked by uid 500); 2 Nov 2009 20:41:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69015 invoked by uid 99); 2 Nov 2009 20:41:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Nov 2009 20:41:49 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 02 Nov 2009 20:41:47 +0000
Received: (qmail 1417 invoked by uid 99); 2 Nov 2009 20:41:25 -0000
Received: from localhost.apache.org (HELO [192.168.168.107]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Nov 2009 20:41:25 +0000
Message-ID: <4AEF43F3.5070703@apache.org>
Date: Mon, 02 Nov 2009 12:41:23 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Proposed bylaws for the Hadoop project
References: <b23e01b50909291345g47ffb267l76fe193ea72652a3@mail.gmail.com>	 <FE14B90F-1F11-4F68-8A6B-3F00DB99D050@apache.org>	 <4ACCBEE7.3070105@apache.org>	 <DDB254D5-1F8E-4B75-B8C1-3B3D095BBBC3@yahoo-inc.com>	 <4ACE1292.6000702@apache.org>	 <b23e01b50910140711y545fa031u7fb199594db33bd6@mail.gmail.com>	 <4ADE4055.3020002@yahoo-inc.com>	 <5764186A-DA78-412B-ADD4-5A9F50866732@yahoo-inc.com> <b23e01b50910291356w17c98e18md0239f30c2aa0a5@mail.gmail.com> <4AEB7728.3000008@yahoo-inc.com> <284E0999-F935-47A2-B051-7241BF8A4B39@apache.org>
In-Reply-To: <284E0999-F935-47A2-B051-7241BF8A4B39@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

These look good to me.  +1

A few minor changes I'd prefer but not insist on:

  - Replace the term "developer" with "contributor".  This seems more 
precise and consistent with our wiki documentation.

  - Emeritus timeout should be 12 months instead of 6.  Perhaps this 
doesn't matter, since we're not bound to make someone emeritus after six 
months, but events might reasonably keep someone away for six, and I 
think we've generally not bothered to make anyone emeritus until they've 
been idle for 12 months, so 12 months seems more descriptive of our 
actual practice.

Thanks for proposing this!

Doug

Owen O'Malley wrote:
> We should have created bylaws when we were established as a PMC back in 
> Jan 2008, but it has become clear that we need to define them. I looked 
> around and the Ant project had bylaws that were both clear and fairly 
> complete. I made some minor edits to match what we do. Please look them 
> over and vote. In a recursive usage, let's use 2/3 majority to approve 
> these bylaws.
> 
> -- Owen
> 
> 
> ------------------------------------------------------------------------
> 
> 
> 

From general-return-676-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 02 21:06:37 2009
Return-Path: <general-return-676-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 11138 invoked from network); 2 Nov 2009 21:06:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Nov 2009 21:06:36 -0000
Received: (qmail 9404 invoked by uid 500); 2 Nov 2009 21:06:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 9319 invoked by uid 500); 2 Nov 2009 21:06:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 9309 invoked by uid 99); 2 Nov 2009 21:06:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Nov 2009 21:06:35 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.223.171] (HELO mail-iw0-f171.google.com) (209.85.223.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Nov 2009 21:06:33 +0000
Received: by iwn1 with SMTP id 1so3869940iwn.2
        for <general@hadoop.apache.org>; Mon, 02 Nov 2009 13:06:12 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.153.69 with SMTP id j5mr777019ibw.33.1257195971155; Mon, 
	02 Nov 2009 13:06:11 -0800 (PST)
In-Reply-To: <4AEF43F3.5070703@apache.org>
References: <b23e01b50909291345g47ffb267l76fe193ea72652a3@mail.gmail.com> 
	<DDB254D5-1F8E-4B75-B8C1-3B3D095BBBC3@yahoo-inc.com> <4ACE1292.6000702@apache.org> 
	<b23e01b50910140711y545fa031u7fb199594db33bd6@mail.gmail.com> 
	<4ADE4055.3020002@yahoo-inc.com> <5764186A-DA78-412B-ADD4-5A9F50866732@yahoo-inc.com> 
	<b23e01b50910291356w17c98e18md0239f30c2aa0a5@mail.gmail.com> 
	<4AEB7728.3000008@yahoo-inc.com> <284E0999-F935-47A2-B051-7241BF8A4B39@apache.org> 
	<4AEF43F3.5070703@apache.org>
From: Philip Zeyliger <philip@cloudera.com>
Date: Mon, 2 Nov 2009 13:05:51 -0800
Message-ID: <15da8a100911021305x7201ae99gf2098768c8ea5cf2@mail.gmail.com>
Subject: Re: [VOTE] Proposed bylaws for the Hadoop project
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636d33c7c464b4f047769bfcf

--001636d33c7c464b4f047769bfcf
Content-Type: text/plain; charset=ISO-8859-1

+1.  (I'm not a committer, so not to be counted in the final vote count)

The following quoted section felt a bit inexact.  Vetoes often come in
before the action is "done", so there's nothing to reverse.  The spirit is
clear, of course, but one could change it to something like "If a veto is
not withdrawn but the action has been done, the vetoed action must be
reversed in a timely manner."

        If you disagree with a valid veto, you must lobby the person casting
        the veto to withdraw their veto. If a veto is not withdrawn, the
action
        that has been vetoed must be reversed in a timely manner.


-- Philip

--001636d33c7c464b4f047769bfcf--

From general-return-677-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 02 21:17:36 2009
Return-Path: <general-return-677-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16215 invoked from network); 2 Nov 2009 21:17:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Nov 2009 21:17:36 -0000
Received: (qmail 29517 invoked by uid 500); 2 Nov 2009 21:17:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29440 invoked by uid 500); 2 Nov 2009 21:17:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29430 invoked by uid 99); 2 Nov 2009 21:17:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Nov 2009 21:17:34 +0000
X-ASF-Spam-Status: No, hits=-9.3 required=5.0
	tests=AWL,BAYES_00,RCVD_IN_DNSWL_HI
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 02 Nov 2009 21:17:32 +0000
Received: (qmail 16154 invoked by uid 99); 2 Nov 2009 21:17:12 -0000
Received: from localhost.apache.org (HELO [192.168.168.107]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Nov 2009 21:17:12 +0000
Message-ID: <4AEF4C57.1070109@apache.org>
Date: Mon, 02 Nov 2009 13:17:11 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Proposed bylaws for the Hadoop project
References: <b23e01b50909291345g47ffb267l76fe193ea72652a3@mail.gmail.com> 	<DDB254D5-1F8E-4B75-B8C1-3B3D095BBBC3@yahoo-inc.com> <4ACE1292.6000702@apache.org> 	<b23e01b50910140711y545fa031u7fb199594db33bd6@mail.gmail.com> 	<4ADE4055.3020002@yahoo-inc.com> <5764186A-DA78-412B-ADD4-5A9F50866732@yahoo-inc.com> 	<b23e01b50910291356w17c98e18md0239f30c2aa0a5@mail.gmail.com> 	<4AEB7728.3000008@yahoo-inc.com> <284E0999-F935-47A2-B051-7241BF8A4B39@apache.org> 	<4AEF43F3.5070703@apache.org> <15da8a100911021305x7201ae99gf2098768c8ea5cf2@mail.gmail.com>
In-Reply-To: <15da8a100911021305x7201ae99gf2098768c8ea5cf2@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Philip Zeyliger wrote:
> If a veto is not withdrawn, the action that has been vetoed must be reversed in a timely manner.

A simple change might be to say "any action" instead of "the action".

Doug

From general-return-678-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 03 01:11:58 2009
Return-Path: <general-return-678-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5890 invoked from network); 3 Nov 2009 01:11:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Nov 2009 01:11:58 -0000
Received: (qmail 87956 invoked by uid 500); 3 Nov 2009 01:11:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87851 invoked by uid 500); 3 Nov 2009 01:11:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87840 invoked by uid 99); 3 Nov 2009 01:11:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 01:11:57 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 01:11:45 +0000
Received: from wlanvpn-mc2e-247-5.corp.yahoo.com (socks2.corp.yahoo.com [216.145.54.7])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id nA31ADMN043963
	for <general@hadoop.apache.org>; Mon, 2 Nov 2009 17:10:13 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=fb/5K3ecrNequQK28urjJASBRmtlqGQouG5beUjkvYaExtYaEnVhP7k0OKFtlwv/
Message-Id: <9033CE01-58E7-4753-85ED-E6DA34FF9D50@yahoo-inc.com>
From: Alan Gates <gates@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <4AEF43F3.5070703@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Proposed bylaws for the Hadoop project
Date: Mon, 2 Nov 2009 17:10:13 -0800
References: <b23e01b50909291345g47ffb267l76fe193ea72652a3@mail.gmail.com>	 <FE14B90F-1F11-4F68-8A6B-3F00DB99D050@apache.org>	 <4ACCBEE7.3070105@apache.org>	 <DDB254D5-1F8E-4B75-B8C1-3B3D095BBBC3@yahoo-inc.com>	 <4ACE1292.6000702@apache.org>	 <b23e01b50910140711y545fa031u7fb199594db33bd6@mail.gmail.com>	 <4ADE4055.3020002@yahoo-inc.com>	 <5764186A-DA78-412B-ADD4-5A9F50866732@yahoo-inc.com> <b23e01b50910291356w17c98e18md0239f30c2aa0a5@mail.gmail.com> <4AEB7728.3000008@yahoo-inc.com> <284E0999-F935-47A2-B051-7241BF8A4B39@apache.org> <4AEF43F3.5070703@apache.org>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

+1 for adoption, and +1 for the two minor changes Doug proposes.

Alan.

On Nov 2, 2009, at 12:41 PM, Doug Cutting wrote:

> These look good to me.  +1
>
> A few minor changes I'd prefer but not insist on:
>
> - Replace the term "developer" with "contributor".  This seems more  
> precise and consistent with our wiki documentation.
>
> - Emeritus timeout should be 12 months instead of 6.  Perhaps this  
> doesn't matter, since we're not bound to make someone emeritus after  
> six months, but events might reasonably keep someone away for six,  
> and I think we've generally not bothered to make anyone emeritus  
> until they've been idle for 12 months, so 12 months seems more  
> descriptive of our actual practice.
>
> Thanks for proposing this!
>
> Doug
>
> Owen O'Malley wrote:
>> We should have created bylaws when we were established as a PMC  
>> back in Jan 2008, but it has become clear that we need to define  
>> them. I looked around and the Ant project had bylaws that were both  
>> clear and fairly complete. I made some minor edits to match what we  
>> do. Please look them over and vote. In a recursive usage, let's use  
>> 2/3 majority to approve these bylaws.
>> -- Owen
>> ------------------------------------------------------------------------


From general-return-679-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 03 02:15:19 2009
Return-Path: <general-return-679-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 41839 invoked from network); 3 Nov 2009 02:15:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Nov 2009 02:15:19 -0000
Received: (qmail 46554 invoked by uid 500); 3 Nov 2009 02:15:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46473 invoked by uid 500); 3 Nov 2009 02:15:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46463 invoked by uid 99); 3 Nov 2009 02:15:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 02:15:18 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of hkumar.arora@gmail.com designates 209.85.223.171 as permitted sender)
Received: from [209.85.223.171] (HELO mail-iw0-f171.google.com) (209.85.223.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 02:15:15 +0000
Received: by iwn1 with SMTP id 1so4056813iwn.2
        for <general@hadoop.apache.org>; Mon, 02 Nov 2009 18:14:55 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=jIJ5ittZtZKDrjOv01iLZv7tReQVP3QrErCSDPCDB7o=;
        b=Lr9C+z/bZftybAXJiRKPMNKCKpoPI8LypcZ3m/XXGTOY12r4P58/C5QVkSjeVAEBI/
         tcUGRGJ3llrkbu9MsUK6L4Bpn+5HVhLHcM3tFKycij3tYgyFAsWDs5rAKGu5FlUyon1R
         m5go0zI5TmntfSwHR6ineRd2iA44NOblaslzc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=ebpbtU2nMkQiIT3c61d5E7cwfya4ejaHehrvyrpIEA0qxTebvu1ixGMs48mLB4NW6E
         685MHggS7i42EiqeCfJfKuFqy0OsU0+8wA6YLzfMrHR3iSgfyZIo0EXsoR2phYVOUv2N
         SqnKtvvTLoz16KeeAvaKCyLqXUu/1w41mEDRg=
MIME-Version: 1.0
Received: by 10.231.5.23 with SMTP id 23mr7249494ibt.45.1257214494882; Mon, 02 
	Nov 2009 18:14:54 -0800 (PST)
In-Reply-To: <f9d76217185ad6ffef5d3414f2390152.squirrel@mail.tce.edu>
References: <f9d76217185ad6ffef5d3414f2390152.squirrel@mail.tce.edu>
Date: Tue, 3 Nov 2009 11:14:54 +0900
Message-ID: <e497591c0911021814r3d27ca5fy53581c1492388fee@mail.gmail.com>
Subject: Re: Http 404 Error while viewing tasktracker in Browser
From: Harshit Kumar <hkumar.arora@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00151773da305fc77804776e0fab

--00151773da305fc77804776e0fab
Content-Type: text/plain; charset=UTF-8

Can you please share the URL that you typed. Also, may be the port is
blocked by the firewall, check the open ports on your system.

Cheers
H. Kumar
skype: harshit900
Blog: http://harshitkumar.wordpress.com
Website: http:/kumarharmuscat.tripod.com


2009/11/2 Abhilaash <abhilaash_cse@tce.edu>

>
> Hi all,
>
>      I have downloaded hadoop and configured it in my local system.I have
> run the wordcount program with some files as dataset .It run
> successfully but when I opened the browser to see GUI for
> tasktracker and job tracker , it shows me the following error:
>
> HTTP ERROR: 404
>
> /jobtracker.jsp
>
> RequestURI=/jobtracker.jsp
>
> Powered by Jetty://
>
>
> Can anyone please help me and tell how to see results in GUI?
>
>
> Thank you in advance,
>
> R.Abhilaash.
>
>
> --
> Nothing is impossible because impossible says i'm possible...
>
>
> -----------------------------------------
> This email was sent using TCEMail Service.
> Thiagarajar College of Engineering
> Madurai-625 015, India
>
>

--00151773da305fc77804776e0fab--

From general-return-680-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 03 03:21:42 2009
Return-Path: <general-return-680-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58953 invoked from network); 3 Nov 2009 03:21:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Nov 2009 03:21:42 -0000
Received: (qmail 91137 invoked by uid 500); 3 Nov 2009 03:21:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90904 invoked by uid 500); 3 Nov 2009 03:21:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90894 invoked by uid 99); 3 Nov 2009 03:21:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 03:21:39 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of goutham.patnaik@gmail.com designates 209.85.211.185 as permitted sender)
Received: from [209.85.211.185] (HELO mail-yw0-f185.google.com) (209.85.211.185)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 03:21:31 +0000
Received: by ywh15 with SMTP id 15so6763175ywh.5
        for <general@hadoop.apache.org>; Mon, 02 Nov 2009 19:21:10 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :from:date:message-id:subject:to:content-type;
        bh=VdSHMgLq4bu478AYbdAfmeLMKkVcr74lfpKcAwKFnMY=;
        b=Avz4j+eXxzNyfl+yB3s/COGh1sNT8UHUwg9kAOBsICkHlqCC6w5w+K4mOUwyQ/npry
         yMG81UV0Mgf+ojZxdB35w9EXbfA9xcAZLoZOxsy0/tkDKz/8bR5m3JQlIcMkWg2ntLju
         pMU0WtqfY3taA5gyATr8FbgRENF4HjniNQolU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=sCAvggpbTP66odEVeFpZJ2bu2RTb0YXwwC1b718jGrrN+seSIgjdAkHHFIhgrXUUZP
         U3MXKR1+lPcXlFt8p9zGT5j6U7KH+Xinwc4G7XeiBaNkq2eJgj/ijoGRP18bGhRccSfT
         gsclXGnRQD8GS78U7EptODMifWIeni7SgRKXg=
MIME-Version: 1.0
Received: by 10.151.94.1 with SMTP id w1mr9410174ybl.214.1257218470362; Mon, 
	02 Nov 2009 19:21:10 -0800 (PST)
In-Reply-To: <309f76d00911010855q155d8efdw12eab526feac9432@mail.gmail.com>
References: <309f76d00911010855q155d8efdw12eab526feac9432@mail.gmail.com>
From: goutham patnaik <goutham.patnaik@gmail.com>
Date: Mon, 2 Nov 2009 19:20:50 -0800
Message-ID: <1b3669280911021920o3b2d61c6y9bda8b69ee11f406@mail.gmail.com>
Subject: Re: TWO QUESTIONS ABOUT LOGS & DEBUGGING IN HADOOP
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd6b2ec54bfab04776efc4c
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd6b2ec54bfab04776efc4c
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Alberto,

Are you looking to write some java code that prints the contents of a file
in HDFS ? If so, take a look at IOUtils.copyBytes (IOUtils is in the packag=
e
hadoop.io.IOUtils) As an output stream, you could just use System.out

heres documentation on IOUtils:

http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/IOUti=
ls.html


hope that helps

Goutham

You can print the content of a file

On Sun, Nov 1, 2009 at 8:55 AM, Alberto Luengo Cabanillas <cabiwan@gmail.co=
m
> wrote:

> Hi everyone! In a project involving Hadoop (0.20.1) I=B4m working, I=B4d =
need
> to
> know if I=B4m reading properly the content of a file located in the HDFS.
> Beside the try...catch block, is there any other option for printing valu=
es
> in the console (in a similar way standard IO Java mechanisms do, like
> System.out.println("the variable is ", var))?
> Another thing, is there a simply way to debug Hadoop applications?
> (actually
> I=B4m working in pseudo-distributed mode, for developing purposes).
> Thanks a lot in advance.
>
> --
> Alberto
>

--000e0cd6b2ec54bfab04776efc4c--

From general-return-681-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 03 08:21:41 2009
Return-Path: <general-return-681-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29648 invoked from network); 3 Nov 2009 08:21:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Nov 2009 08:21:41 -0000
Received: (qmail 95634 invoked by uid 500); 3 Nov 2009 08:21:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95544 invoked by uid 500); 3 Nov 2009 08:21:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95533 invoked by uid 99); 3 Nov 2009 08:21:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 08:21:39 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 08:21:36 +0000
Received: from SNV-EXPF01.ds.corp.yahoo.com (snv-expf01.ds.corp.yahoo.com [207.126.227.250])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id nA38KT4b052710
	for <general@hadoop.apache.org>; Tue, 3 Nov 2009 00:20:31 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:return-path:x-originalarrivaltime;
	b=quHtDxZQbr6sfkQUhmRdZgYhbK9/r/qUY3KIlWMxc0McZ0zptNqTpHOBWpEOAvvr
Received: from SNV-EXVS06.ds.corp.yahoo.com ([207.126.227.233]) by SNV-EXPF01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Tue, 3 Nov 2009 00:20:29 -0800
Received: from 10.66.79.68 ([10.66.79.68]) by SNV-EXVS06.ds.corp.yahoo.com ([207.126.227.82]) via Exchange Front-End Server egl-webmail.corp.yahoo.com ([203.83.248.208]) with Microsoft Exchange Server HTTP-DAV ;
 Tue,  3 Nov 2009 08:12:48 +0000
User-Agent: Microsoft-Entourage/12.20.0.090605
Date: Tue, 03 Nov 2009 13:42:47 +0530
Subject: Re: [VOTE] Proposed bylaws for the Hadoop project
From: Hemanth Yamijala <yhemanth@yahoo-inc.com>
To: <general@hadoop.apache.org>
Message-ID: <C715E3D7.2F176%yhemanth@yahoo-inc.com>
Thread-Topic: [VOTE] Proposed bylaws for the Hadoop project
Thread-Index: AcpcXW1cFG35dvR8o02FITQhxy4e3Q==
In-Reply-To: <284E0999-F935-47A2-B051-7241BF8A4B39@apache.org>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-OriginalArrivalTime: 03 Nov 2009 08:20:29.0566 (UTC) FILETIME=[8112E1E0:01CA5C5E]

Owen

Some clarifications:

In the section on "Code Change" when you mention approval as "Lazy approval
and then lazy consensus", did you mean the consensus will need to come into
play if a veto is received ?

Also, very, very minor nit: In the section describing "Lazy Majority"
" A lazy majority vote requires 3 binding +1 votes and more binding +1
  votes that -1 votes."
"that" must be "than"

In general, +1, including Doug's suggested changes.

Thanks
hemanth

On 11/3/09 1:22 AM, "Owen O'Malley" <omalley@apache.org> wrote:

> We should have created bylaws when we were established as a PMC back
> in Jan 2008, but it has become clear that we need to define them. I
> looked around and the Ant project had bylaws that were both clear and
> fairly complete. I made some minor edits to match what we do. Please
> look them over and vote. In a recursive usage, let's use 2/3 majority
> to approve these bylaws.
> 
> -- Owen
> 
> 


From general-return-682-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 03 10:05:30 2009
Return-Path: <general-return-682-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81818 invoked from network); 3 Nov 2009 10:05:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Nov 2009 10:05:30 -0000
Received: (qmail 47637 invoked by uid 500); 3 Nov 2009 10:05:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47548 invoked by uid 500); 3 Nov 2009 10:05:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47538 invoked by uid 99); 3 Nov 2009 10:05:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 10:05:28 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=SPF_HELO_PASS,SPF_PASS,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of abhilaash_cse@tce.edu designates 210.212.252.72 as permitted sender)
Received: from [210.212.252.72] (HELO tce.edu) (210.212.252.72)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 10:05:17 +0000
Received: from [127.0.0.1] (helo=mail.tcenet)
	by tce.edu with esmtp (Exim 4.63)
	(envelope-from <abhilaash_cse@tce.edu>)
	id 1N5GFm-0003p7-F4
	for general@hadoop.apache.org; Tue, 03 Nov 2009 15:34:48 +0530
Received: from 10.2.1.250
        (SquirrelMail authenticated user abhilaash_cse)
        by mail.tcenet with HTTP;
        Tue, 3 Nov 2009 15:34:42 +0530
Message-ID: <98b459018b7877975c9f42e0d05dc1ab.squirrel@mail.tcenet>
In-Reply-To: <e497591c0911021814r3d27ca5fy53581c1492388fee@mail.gmail.com>
References: <f9d76217185ad6ffef5d3414f2390152.squirrel@mail.tce.edu>
    <e497591c0911021814r3d27ca5fy53581c1492388fee@mail.gmail.com>
Date: Tue, 3 Nov 2009 15:34:42 +0530
Subject: Re: Http 404 Error while viewing tasktracker in Browser
From: "Abhilaash" <abhilaash_cse@tce.edu>
To: general@hadoop.apache.org
User-Agent: SquirrelMail/1.4.19
MIME-Version: 1.0
Content-Type: text/plain;charset=iso-8859-1
Content-Transfer-Encoding: 8bit
X-Priority: 3 (Normal)
Importance: Normal
X-XheaderVersion: 1.1
X-UserAgent: 
X-Spam-Score: 0.1 (/)
X-Virus-Checked: Checked by ClamAV on apache.org
X-Old-Spam-Status: No




Hi,


 The URL I typed was http://localhost:50030/jobtracker.jsp and I'll check
for the open ports.


Thank you,
Abhilaash.


> Can you please share the URL that you typed. Also, may be the port is
> blocked by the firewall, check the open ports on your system.
>
> Cheers
> H. Kumar
> skype: harshit900
> Blog: http://harshitkumar.wordpress.com
> Website: http:/kumarharmuscat.tripod.com
>
>
> 2009/11/2 Abhilaash <abhilaash_cse@tce.edu>
>
>>
>> Hi all,
>>
>>      I have downloaded hadoop and configured it in my local system.I
>> have
>> run the wordcount program with some files as dataset .It run
>> successfully but when I opened the browser to see GUI for
>> tasktracker and job tracker , it shows me the following error:
>>
>> HTTP ERROR: 404
>>
>> /jobtracker.jsp
>>
>> RequestURI=/jobtracker.jsp
>>
>> Powered by Jetty://
>>
>>
>> Can anyone please help me and tell how to see results in GUI?
>>
>>
>> Thank you in advance,
>>
>> R.Abhilaash.
>>
>>
>> --
>> Nothing is impossible because impossible says i'm possible...
>>
>>
>> -----------------------------------------
>> This email was sent using TCEMail Service.
>> Thiagarajar College of Engineering
>> Madurai-625 015, India
>>
>>
>


-- 
Nothing is impossible because impossible says i'm possible...


-----------------------------------------
This email was sent using TCEMail Service.
Thiagarajar College of Engineering
Madurai-625 015, India


From general-return-683-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 03 11:04:18 2009
Return-Path: <general-return-683-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 18069 invoked from network); 3 Nov 2009 11:04:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Nov 2009 11:04:18 -0000
Received: (qmail 62567 invoked by uid 500); 3 Nov 2009 11:04:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62483 invoked by uid 500); 3 Nov 2009 11:04:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62472 invoked by uid 99); 3 Nov 2009 11:04:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 11:04:17 +0000
X-ASF-Spam-Status: No, hits=-5.8 required=5.0
	tests=AWL,BAYES_00,RCVD_IN_DNSWL_MED
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 11:04:14 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 2AAE8B7EB3
	for <general@hadoop.apache.org>; Tue,  3 Nov 2009 11:03:52 +0000 (GMT)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id Lf5YLxyrUIPT for <general@hadoop.apache.org>;
	Tue,  3 Nov 2009 11:03:46 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id A2964B7EB0
	for <general@hadoop.apache.org>; Tue,  3 Nov 2009 11:03:46 +0000 (GMT)
MailScanner-NULL-Check: 1257851016.27367@hzejjECqm+URG7gonP+zPg
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id nA3B3ZKK014185
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Tue, 3 Nov 2009 11:03:35 GMT
Message-ID: <4AF00E07.8050701@apache.org>
Date: Tue, 03 Nov 2009 11:03:35 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Proposed bylaws for the Hadoop project
References: <b23e01b50909291345g47ffb267l76fe193ea72652a3@mail.gmail.com>	 <FE14B90F-1F11-4F68-8A6B-3F00DB99D050@apache.org>	 <4ACCBEE7.3070105@apache.org>	 <DDB254D5-1F8E-4B75-B8C1-3B3D095BBBC3@yahoo-inc.com>	 <4ACE1292.6000702@apache.org>	 <b23e01b50910140711y545fa031u7fb199594db33bd6@mail.gmail.com>	 <4ADE4055.3020002@yahoo-inc.com>	 <5764186A-DA78-412B-ADD4-5A9F50866732@yahoo-inc.com> <b23e01b50910291356w17c98e18md0239f30c2aa0a5@mail.gmail.com> <4AEB7728.3000008@yahoo-inc.com> <284E0999-F935-47A2-B051-7241BF8A4B39@apache.org>
In-Reply-To: <284E0999-F935-47A2-B051-7241BF8A4B39@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: nA3B3ZKK014185
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

Owen O'Malley wrote:
> We should have created bylaws when we were established as a PMC back in 
> Jan 2008, but it has become clear that we need to define them. I looked 
> around and the Ant project had bylaws that were both clear and fairly 
> complete. I made some minor edits to match what we do. Please look them 
> over and vote. In a recursive usage, let's use 2/3 majority to approve 
> these bylaws.
> 

silly Q, but what are "work days" in this context? Were I to propose 
something on, say thankgsiviing thursday day, with all the US off work 
until monday, we in rest of the world would leave you US folk with only 
monday to veto our attempts to vote something through.

-steve

From general-return-684-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 03 14:39:02 2009
Return-Path: <general-return-684-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81430 invoked from network); 3 Nov 2009 14:39:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Nov 2009 14:39:02 -0000
Received: (qmail 44659 invoked by uid 500); 3 Nov 2009 14:39:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44560 invoked by uid 500); 3 Nov 2009 14:39:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44550 invoked by uid 99); 3 Nov 2009 14:39:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 14:39:00 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cabiwan@gmail.com designates 209.85.210.194 as permitted sender)
Received: from [209.85.210.194] (HELO mail-yx0-f194.google.com) (209.85.210.194)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 14:38:52 +0000
Received: by yxe32 with SMTP id 32so7255855yxe.5
        for <general@hadoop.apache.org>; Tue, 03 Nov 2009 06:38:31 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=LB5/KkkzfmcbQ461cngkLyNZhaLYJM5faI9L+sLa+us=;
        b=Uv6YC1zBBeRTUOCINJhwXf8TcDUef/w8L+UYPVOegfzhjHtqw0vZCqL2NY89rQjTyK
         QeG2EP6wtAu++75X3Vp8S0RMSyNrdqwXnSG5l/mK/YT5P/9nOY7Iy9vieT8qmhFbbu7C
         fHSCwHn7Ax0CIA5pgZMNSWKPm2JeputSPLT3Q=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=YgNgpQfGlB5CzUjvbyhXTcrJPsgUY6N8yGocBvLvsQYpa7an2poByB5cZtgKDJNouV
         lQcR7ir2B9a5llzXszjQXsMSjyg3YgrV8jgkFHV1ZRslXPhDu+o0cH9wW+HHvk56AxHa
         zh+gVvF13Y3lLYWm3msY7tE+1eFydcu+IN/RY=
MIME-Version: 1.0
Received: by 10.103.81.20 with SMTP id i20mr10646mul.127.1257259110818; Tue, 
	03 Nov 2009 06:38:30 -0800 (PST)
In-Reply-To: <1b3669280911021920o3b2d61c6y9bda8b69ee11f406@mail.gmail.com>
References: <309f76d00911010855q155d8efdw12eab526feac9432@mail.gmail.com>
	 <1b3669280911021920o3b2d61c6y9bda8b69ee11f406@mail.gmail.com>
Date: Tue, 3 Nov 2009 15:38:30 +0100
Message-ID: <309f76d00911030638y4d9c7b5dxe737f4f13952fd34@mail.gmail.com>
Subject: Re: TWO QUESTIONS ABOUT LOGS & DEBUGGING IN HADOOP
From: Alberto Luengo Cabanillas <cabiwan@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e65b60a8b0e4b004777872f0
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e65b60a8b0e4b004777872f0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Thanks for the reply,. I=B4ll try your solution.

2009/11/3 goutham patnaik <goutham.patnaik@gmail.com>

> Alberto,
>
> Are you looking to write some java code that prints the contents of a fil=
e
> in HDFS ? If so, take a look at IOUtils.copyBytes (IOUtils is in the
> package
> hadoop.io.IOUtils) As an output stream, you could just use System.out
>
> heres documentation on IOUtils:
>
>
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/IOU=
tils.html
>
>
> hope that helps
>
> Goutham
>
> You can print the content of a file
>
> On Sun, Nov 1, 2009 at 8:55 AM, Alberto Luengo Cabanillas <
> cabiwan@gmail.com
> > wrote:
>
> > Hi everyone! In a project involving Hadoop (0.20.1) I=B4m working, I=B4=
d need
> > to
> > know if I=B4m reading properly the content of a file located in the HDF=
S.
> > Beside the try...catch block, is there any other option for printing
> values
> > in the console (in a similar way standard IO Java mechanisms do, like
> > System.out.println("the variable is ", var))?
> > Another thing, is there a simply way to debug Hadoop applications?
> > (actually
> > I=B4m working in pseudo-distributed mode, for developing purposes).
> > Thanks a lot in advance.
> >
> > --
> > Alberto
> >
>



--=20
Alberto

--0016e65b60a8b0e4b004777872f0--

From general-return-685-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 03 16:02:40 2009
Return-Path: <general-return-685-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4365 invoked from network); 3 Nov 2009 16:02:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Nov 2009 16:02:40 -0000
Received: (qmail 77138 invoked by uid 500); 3 Nov 2009 16:02:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77059 invoked by uid 500); 3 Nov 2009 16:02:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77049 invoked by uid 99); 3 Nov 2009 16:02:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 16:02:38 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of owen.omalley@gmail.com designates 209.85.222.195 as permitted sender)
Received: from [209.85.222.195] (HELO mail-pz0-f195.google.com) (209.85.222.195)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 16:02:30 +0000
Received: by pzk33 with SMTP id 33so4319202pzk.2
        for <general@hadoop.apache.org>; Tue, 03 Nov 2009 08:02:09 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=smiTmVKcnDeMXzrwH26nA0rrG00T83gP6cWR+kX9T2w=;
        b=KWtNYSRoyam6GuSyPKgPxv+Sg3z4745DjwPoP11OtRfJEhTqEQP/sfm62oJz8IPLNT
         5KHh4FqiDpTEXou/AT0VOZm4wR7oEzV2v/5CokikLEK7IJMlIVzNXlYOfPA7QhKXbckE
         4vBIlOXQbo5LCGWJg6zTsGEKByiUlOBLrOmok=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=SswJd3JjunB+DINcH7F/Vni2J71jspwXXitH38xFddDm6hxckgSqESVkeewRQ90iDE
         0AxJQEpznfosEcZtfaQdfR9aGij6KFVBFy2BOfo8KiQrk4snPf7d92nWI8Jc8UF1XGz5
         Nb+QThD9kd1c6uqA6ZWIx4GgDum0F63dluS8E=
MIME-Version: 1.0
Received: by 10.141.22.21 with SMTP id z21mr9151rvi.170.1257264129362; Tue, 03 
	Nov 2009 08:02:09 -0800 (PST)
In-Reply-To: <4AEF43F3.5070703@apache.org>
References: <b23e01b50909291345g47ffb267l76fe193ea72652a3@mail.gmail.com>
	 <DDB254D5-1F8E-4B75-B8C1-3B3D095BBBC3@yahoo-inc.com>
	 <4ACE1292.6000702@apache.org>
	 <b23e01b50910140711y545fa031u7fb199594db33bd6@mail.gmail.com>
	 <4ADE4055.3020002@yahoo-inc.com>
	 <5764186A-DA78-412B-ADD4-5A9F50866732@yahoo-inc.com>
	 <b23e01b50910291356w17c98e18md0239f30c2aa0a5@mail.gmail.com>
	 <4AEB7728.3000008@yahoo-inc.com>
	 <284E0999-F935-47A2-B051-7241BF8A4B39@apache.org>
	 <4AEF43F3.5070703@apache.org>
Date: Tue, 3 Nov 2009 08:02:09 -0800
Message-ID: <5f7f740b0911030802l46d227bal606266ccee4a2888@mail.gmail.com>
Subject: Re: [VOTE] Proposed bylaws for the Hadoop project
From: "Owen O'Malley" <owen.omalley@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd1b430d1ce6f0477799d05
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd1b430d1ce6f0477799d05
Content-Type: text/plain; charset=UTF-8

On Mon, Nov 2, 2009 at 12:41 PM, Doug Cutting <cutting@apache.org> wrote:

>  - Replace the term "developer" with "contributor".  This seems more
> precise and consistent with our wiki documentation.
>

+1


>   - Emeritus timeout should be 12 months instead of 6.


I considered it, but since we are setting up rules that require actual
majorities, I think that if someone hasn't been responding on the list for 6
months, it is reasonable to make them emeritus if we need to.

-- Owen

--000e0cd1b430d1ce6f0477799d05--

From general-return-686-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 03 16:08:44 2009
Return-Path: <general-return-686-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6628 invoked from network); 3 Nov 2009 16:08:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Nov 2009 16:08:43 -0000
Received: (qmail 87166 invoked by uid 500); 3 Nov 2009 16:08:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87112 invoked by uid 500); 3 Nov 2009 16:08:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87102 invoked by uid 99); 3 Nov 2009 16:08:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 16:08:42 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of owen.omalley@gmail.com designates 209.85.160.50 as permitted sender)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 16:08:34 +0000
Received: by pwi4 with SMTP id 4so2736794pwi.29
        for <general@hadoop.apache.org>; Tue, 03 Nov 2009 08:08:13 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=CLsZLUXFgHqE6kn0OfARSISWLrNSslrbU/rPmocNLKc=;
        b=SmIsKYnGFiDW7+hfAECfEj4tm7q+bxC1LnS2W6lirvCmJ5lMYTFCIx5p9WJupVVjA8
         VdejFYXHgj4n4RgZKuzYmG618uuKxLUUeoYEDmMOmuazTndSKqZpYlJlSS/BXCPm020I
         DX7U+3n9XL2bAXq6N0+n3/Vcj/5D+XGeKvhy0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=ilO0884RHS6bQeG/6wsUpFikqj/WvaQ/us1n1Un5MDByQDxcr7iABf2zrJY8Yi9x1s
         n6UMEUUuNTBVsonvCJQqQAwTvUad9ByyXo3Ciz/KGtR+51my839MI9y9PEolToNRqFj/
         C5qmOU0CiMq3LX2q+DZdIYwpZ6DAD+UzHSf9A=
MIME-Version: 1.0
Received: by 10.140.170.13 with SMTP id s13mr9366rve.117.1257264493142; Tue, 
	03 Nov 2009 08:08:13 -0800 (PST)
In-Reply-To: <C715E3D7.2F176%yhemanth@yahoo-inc.com>
References: <284E0999-F935-47A2-B051-7241BF8A4B39@apache.org>
	 <C715E3D7.2F176%yhemanth@yahoo-inc.com>
Date: Tue, 3 Nov 2009 08:08:13 -0800
Message-ID: <5f7f740b0911030808m3ac21126h1271b39d7ae30cac@mail.gmail.com>
Subject: Re: [VOTE] Proposed bylaws for the Hadoop project
From: "Owen O'Malley" <owen.omalley@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd2999e80a294047779b33b
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd2999e80a294047779b33b
Content-Type: text/plain; charset=UTF-8

On Tue, Nov 3, 2009 at 12:12 AM, Hemanth Yamijala <yhemanth@yahoo-inc.com>wrote:

> Owen
>
> Some clarifications:
>
> In the section on "Code Change" when you mention approval as "Lazy approval
> and then lazy consensus", did you mean the consensus will need to come into
> play if a veto is received ?
>

Hmm... I copied that section from Ant's bylaws. Our current practice is
probably actually lazy consensus since we require a +1 before it can be
committed.

-- Owen

--000e0cd2999e80a294047779b33b--

From general-return-687-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 03 16:10:26 2009
Return-Path: <general-return-687-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6930 invoked from network); 3 Nov 2009 16:10:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Nov 2009 16:10:26 -0000
Received: (qmail 91578 invoked by uid 500); 3 Nov 2009 16:10:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91518 invoked by uid 500); 3 Nov 2009 16:10:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91508 invoked by uid 99); 3 Nov 2009 16:10:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 16:10:25 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of owen.omalley@gmail.com designates 209.85.160.50 as permitted sender)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 16:10:19 +0000
Received: by pwi4 with SMTP id 4so2737954pwi.29
        for <general@hadoop.apache.org>; Tue, 03 Nov 2009 08:09:59 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=WiR1exqWNH32WMutiGmcxZfWMumMk+0foJxbLxA/2SM=;
        b=mr5ONmovvpgoHXjTd7D0wVGwLPaoyCFW4iyZm4HJLIPD9JvvFzeN6UL1T/yoJ++jAc
         ofqxgl1+Ycq3eaeknur/PDc33/gw6QP9744kMY4e95D6XAkjdxgo16A9nbzVcBVUKh+O
         26yOjJ6qOTX8wTPzd2aB53BOdo2DeDepnOx/Y=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=DmydP8MXqqmNGz3r6ljzluoDTg48jogd6CLMqnYFM792v3md+PY4Ein0HevMkxdxxk
         eNeloq18gt+7Gh/5n/ZXXeukrcEyvVqn3ighfIkJ/KyT1SNcHTMa7twRfENFZpRDryMP
         zVAtQHwzWll1C3Yvuc9FdPW8yhNob2Peoiulo=
MIME-Version: 1.0
Received: by 10.140.194.12 with SMTP id r12mr9842rvf.65.1257264599590; Tue, 03 
	Nov 2009 08:09:59 -0800 (PST)
In-Reply-To: <4AF00E07.8050701@apache.org>
References: <b23e01b50909291345g47ffb267l76fe193ea72652a3@mail.gmail.com>
	 <DDB254D5-1F8E-4B75-B8C1-3B3D095BBBC3@yahoo-inc.com>
	 <4ACE1292.6000702@apache.org>
	 <b23e01b50910140711y545fa031u7fb199594db33bd6@mail.gmail.com>
	 <4ADE4055.3020002@yahoo-inc.com>
	 <5764186A-DA78-412B-ADD4-5A9F50866732@yahoo-inc.com>
	 <b23e01b50910291356w17c98e18md0239f30c2aa0a5@mail.gmail.com>
	 <4AEB7728.3000008@yahoo-inc.com>
	 <284E0999-F935-47A2-B051-7241BF8A4B39@apache.org>
	 <4AF00E07.8050701@apache.org>
Date: Tue, 3 Nov 2009 08:09:59 -0800
Message-ID: <5f7f740b0911030809s2e57cd57gb9e91ecb30e28cab@mail.gmail.com>
Subject: Re: [VOTE] Proposed bylaws for the Hadoop project
From: "Owen O'Malley" <owen.omalley@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd2114ad8ea13047779b986

--000e0cd2114ad8ea13047779b986
Content-Type: text/plain; charset=UTF-8

On Tue, Nov 3, 2009 at 3:03 AM, Steve Loughran <stevel@apache.org> wrote:

>
> silly Q, but what are "work days" in this context? Were I to propose
> something on, say thankgsiviing thursday day, with all the US off work until
> monday, we in rest of the world would leave you US folk with only monday to
> veto our attempts to vote something through.
>

Good point! It is what we've done. The Ant bylaws actually use a week. Let's
change it to a week, since that is roughly equivalent and handles all of the
special cases of holidays.

-- Owen

--000e0cd2114ad8ea13047779b986--

From general-return-688-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 03 16:55:19 2009
Return-Path: <general-return-688-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22404 invoked from network); 3 Nov 2009 16:55:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Nov 2009 16:55:19 -0000
Received: (qmail 81792 invoked by uid 500); 3 Nov 2009 16:55:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81695 invoked by uid 500); 3 Nov 2009 16:55:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 60145 invoked by uid 99); 1 Nov 2009 17:30:32 -0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of harsh.107@gmail.com designates 209.85.210.194 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=KFMA4b4WMNKruxNHu08mlFu8qWFzgGJkgdCtTkNcJS8=;
        b=syEz6JYbKug/IJ8SBvG4TjzCsLxEuut50Ttme5vO4WsaoOxZqVlH77lsAymsiUSlMn
         xtM5IEyskKYq7IH473QLV1QgsNq8a3PkJFuPGXKITx3lqa5FO7DGc2EnTI0tiBJi4lIj
         7Umbz0m/EQ34pMbFvT2X+jO1d/4EI8i3fSHfc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=AcWW0jhaiEURGVY2WiiaNfyWGy/muvVGz+d5SlwT6NRx2EZ0lPhs8gCkM9obdc1wbv
         vxgEAh0yOcW2C6URRYs/4Ks28ys5OL15yW/ZanOJqYnN3XNvMP15bwKG3maySKv71KpT
         H3H4bC49YiyYyxhJNMGzW511NyQL6WbLat8MA=
MIME-Version: 1.0
In-Reply-To: <309f76d00911010855q155d8efdw12eab526feac9432@mail.gmail.com>
References: <309f76d00911010855q155d8efdw12eab526feac9432@mail.gmail.com>
Date: Sun, 1 Nov 2009 23:00:03 +0530
Message-ID: <2e63a6a90911010930l794e04aag7e006d7caaf1077f@mail.gmail.com>
Subject: Re: TWO QUESTIONS ABOUT LOGS & DEBUGGING IN HADOOP
From: Harshad Shrikhande <harsh.107@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cdf195280ea0f0477529ce5
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cdf195280ea0f0477529ce5
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi,
     You can read the content of a file in Hadoop Distributed File System
using
  bin/hadoop dfs -cat file name

What else do you want to do . It's simple

Bye
Harshad Shrikhande


On Sun, Nov 1, 2009 at 10:25 PM, Alberto Luengo Cabanillas <
cabiwan@gmail.com> wrote:

> Hi everyone! In a project involving Hadoop (0.20.1) I=B4m working, I=B4d =
need
> to
> know if I=B4m reading properly the content of a file located in the HDFS.
> Beside the try...catch block, is there any other option for printing valu=
es
> in the console (in a similar way standard IO Java mechanisms do, like
> System.out.println("the variable is ", var))?
> Another thing, is there a simply way to debug Hadoop applications?
> (actually
> I=B4m working in pseudo-distributed mode, for developing purposes).
> Thanks a lot in advance.
>
> --
> Alberto
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>
>

--000e0cdf195280ea0f0477529ce5--

From general-return-689-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 03 16:55:28 2009
Return-Path: <general-return-689-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22511 invoked from network); 3 Nov 2009 16:55:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Nov 2009 16:55:28 -0000
Received: (qmail 82731 invoked by uid 500); 3 Nov 2009 16:55:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82661 invoked by uid 500); 3 Nov 2009 16:55:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 68606 invoked by uid 99); 2 Nov 2009 03:26:13 -0000
X-ASF-Spam-Status: No, hits=0.2 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=messagingengine.com; h=date:from:to:subject:message-id:mime-version:content-type; s=smtpout; bh=lLnYQJxbBHHUIRtZ2+8HJrrTRdM=; b=XllNyfQT4H8WwnJPIeClluVzQSkzQQMGX0engNP5cwZgUfKceAGA8Hgw27LJe916IpCobBFfelmm0ngrgdLSM8TZ/d98KPMgTlyNI0rbhheUxm/w73pSnwO6l9AGTgfjRlX3HAsK6/jUL7nq105b9RrXq+rSxKrR/T8fe2ptqJo=
X-Sasl-enc: PjGho2UXsAmJnKcY3tqvkhwtaYJQNgpCLtgshU9oSPDh 1257132340
Date: Mon, 2 Nov 2009 14:25:37 +1100
From: David Crossley <crossley@apache.org>
To: general@hadoop.apache.org
Subject: assistance with Apache Forrest
Message-ID: <20091102032537.GD15843@igg.indexgeo.com.au>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.4.2.3i
X-Virus-Checked: Checked by ClamAV on apache.org

I did some research to find the ASF projects that
manage their websites with Apache Forrest, and am
sending similar email to each project's dev mail list.
Sent this to your general list, hoping that it will
reach sub-project people too.

The purposes of this email are to remind people
about some of the useful facilities of Forrest,
and also alert them to discussion about the status
and future directions of Forrest, and to appeal for
people to assist Forrest.

--- oOo ---

These are useful facilities to assist with developing
and managing a Forrest solution for your project's website.

"How to deploy documentation with the Forrestbot
svn workstage"
This explains how the Forrest project manages our
own documentation.
http://forrest.apache.org/howto-forrestbot-svn.html

"Generate an ASF mirrors page using interactive web form"
http://forrest.apache.org/docs/dev/howto/howto-asf-mirror.html

"ForrestBar - Firefox toolbar to ease navigation
and search of Forrest resources"
http://forrest.apache.org/tools/forrestbar.html

"How to do development with Apache Forrest"
http://forrest.apache.org/howto-dev.html

"Frequently Asked Questions"
http://forrest.apache.org/faq.html

"The Anakia output plugin"
This was developed to assist the old Incubator
website to stop using Forrest and export all content
to an Anakia "xdoc" format. From there it could used
by an Anakia-based build system, or be further transformed.
http://forrest.apache.org/pluginDocs/plugins_0_80/org.apache.forrest.plugin.output.Anakia/

As usual, if you need further assistance with anything
then please ask on the Forrest mail lists.

--- oOo ---

There is discussion currently underway on the Forrest
dev mail list about the current status and future
direction of Forrest.
http://thread.gmane.org/gmane.text.xml.forrest.devel/27325

If anyone can assist Forrest, in any capacity, then
please do.

-David

From general-return-690-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 03 16:55:59 2009
Return-Path: <general-return-690-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22647 invoked from network); 3 Nov 2009 16:55:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Nov 2009 16:55:59 -0000
Received: (qmail 83534 invoked by uid 500); 3 Nov 2009 16:55:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83447 invoked by uid 500); 3 Nov 2009 16:55:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 51386 invoked by uid 99); 3 Nov 2009 05:05:31 -0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of chris.douglas@gmail.com designates 209.85.218.223 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=ahQrwEdTsUcZ8T4L4/UDg+9Y1rnMfRlGhAb4mupTPp4=;
        b=l4iG64K6jXsahGAgrHSNtK7y8PmdfC/9EPTknbGkRDeNOV2RsCIM11PjTfuCS/Nqvx
         PkoJWnloVQO6gRABeWlzE3hA+7WeRv+zoZfpe6chrZfwiYwFuNmdPeb02YYtofVPXtDT
         Bnw8W6EfX1P7sFq6bbcT/khxJB6fRkzLv4cg8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=XbeHGwBMxC43vKldfh5et8HA24v5qMwR6lXqMvl+ne4bhtWF3+adChKnXFud/Gdt0B
         UToK6z8CBvxicXu2+RlWpsVPSfsk4mhuNhJoT9uaUto65XFFjfozwW2Fn0aGxL+9y32B
         M5knpWx/SovcSnqbG/NHgDCybiQcxmyQ9IK6c=
MIME-Version: 1.0
In-Reply-To: <9033CE01-58E7-4753-85ED-E6DA34FF9D50@yahoo-inc.com>
References: <b23e01b50909291345g47ffb267l76fe193ea72652a3@mail.gmail.com>
	 <4ACE1292.6000702@apache.org>
	 <b23e01b50910140711y545fa031u7fb199594db33bd6@mail.gmail.com>
	 <4ADE4055.3020002@yahoo-inc.com>
	 <5764186A-DA78-412B-ADD4-5A9F50866732@yahoo-inc.com>
	 <b23e01b50910291356w17c98e18md0239f30c2aa0a5@mail.gmail.com>
	 <4AEB7728.3000008@yahoo-inc.com>
	 <284E0999-F935-47A2-B051-7241BF8A4B39@apache.org>
	 <4AEF43F3.5070703@apache.org>
	 <9033CE01-58E7-4753-85ED-E6DA34FF9D50@yahoo-inc.com>
Date: Mon, 2 Nov 2009 22:05:03 -0700
Message-ID: <1267dd3b0911022105ibc7c765r58bbf5dfb7b3ceaf@mail.gmail.com>
Subject: Re: [VOTE] Proposed bylaws for the Hadoop project
From: Chris Douglas <chris.douglas@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

This will be helpful. Random questions/thoughts:

* The intro should also mention that Apache owns the trademark for
"Hadoop", in addition to the copyright for its source.

* What does it mean for code changes to have "Lazy approval and then
lazy consensus"?

* Most votes- particularly on releases- take longer than 3 days in
practice. Late votes are usually permitted, as are vetos. Perhaps no
vote can pass in fewer than 3 full business days?

* Does a vetoed proposal require a new vote, or does a vote pass when
all vetos are recanted? Can new objections/vetos be raised during that
debate?

* With consensus and 2/3 majorities, a 6 month emeritus period seems
reasonable to me. However, the actions requiring them in the current
document are rare (I hope), so arguing over these details may not be
worthwhile. Surely we wouldn't move someone into that category without
first asking them if they plan to become more active in the near term.
It's important that this not be used to push people out, particularly
since the 2/3 and consensus votes may be swayed by "noticing" inactive
members.

-C

On Mon, Nov 2, 2009 at 6:10 PM, Alan Gates <gates@yahoo-inc.com> wrote:
> +1 for adoption, and +1 for the two minor changes Doug proposes.
>
> Alan.
>
> On Nov 2, 2009, at 12:41 PM, Doug Cutting wrote:
>
>> These look good to me. =A0+1
>>
>> A few minor changes I'd prefer but not insist on:
>>
>> - Replace the term "developer" with "contributor". =A0This seems more
>> precise and consistent with our wiki documentation.
>>
>> - Emeritus timeout should be 12 months instead of 6. =A0Perhaps this doe=
sn't
>> matter, since we're not bound to make someone emeritus after six months,=
 but
>> events might reasonably keep someone away for six, and I think we've
>> generally not bothered to make anyone emeritus until they've been idle f=
or
>> 12 months, so 12 months seems more descriptive of our actual practice.
>>
>> Thanks for proposing this!
>>
>> Doug
>>
>> Owen O'Malley wrote:
>>>
>>> We should have created bylaws when we were established as a PMC back in
>>> Jan 2008, but it has become clear that we need to define them. I looked
>>> around and the Ant project had bylaws that were both clear and fairly
>>> complete. I made some minor edits to match what we do. Please look them=
 over
>>> and vote. In a recursive usage, let's use 2/3 majority to approve these
>>> bylaws.
>>> -- Owen
>>> -----------------------------------------------------------------------=
-
>
>

From general-return-691-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 03 17:29:46 2009
Return-Path: <general-return-691-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33548 invoked from network); 3 Nov 2009 17:29:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Nov 2009 17:29:46 -0000
Received: (qmail 26766 invoked by uid 500); 3 Nov 2009 17:29:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26681 invoked by uid 500); 3 Nov 2009 17:29:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26671 invoked by uid 99); 3 Nov 2009 17:29:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 17:29:44 +0000
X-ASF-Spam-Status: No, hits=-10.6 required=5.0
	tests=BAYES_00,RCVD_IN_DNSWL_HI
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 03 Nov 2009 17:29:42 +0000
Received: (qmail 33523 invoked by uid 99); 3 Nov 2009 17:29:22 -0000
Received: from localhost.apache.org (HELO [198.18.166.176]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 17:29:22 +0000
Message-ID: <4AF06870.7060102@apache.org>
Date: Tue, 03 Nov 2009 09:29:20 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Proposed bylaws for the Hadoop project
References: <b23e01b50909291345g47ffb267l76fe193ea72652a3@mail.gmail.com>	 <4ACE1292.6000702@apache.org>	 <b23e01b50910140711y545fa031u7fb199594db33bd6@mail.gmail.com>	 <4ADE4055.3020002@yahoo-inc.com>	 <5764186A-DA78-412B-ADD4-5A9F50866732@yahoo-inc.com>	 <b23e01b50910291356w17c98e18md0239f30c2aa0a5@mail.gmail.com>	 <4AEB7728.3000008@yahoo-inc.com>	 <284E0999-F935-47A2-B051-7241BF8A4B39@apache.org>	 <4AEF43F3.5070703@apache.org>	 <9033CE01-58E7-4753-85ED-E6DA34FF9D50@yahoo-inc.com> <1267dd3b0911022105ibc7c765r58bbf5dfb7b3ceaf@mail.gmail.com>
In-Reply-To: <1267dd3b0911022105ibc7c765r58bbf5dfb7b3ceaf@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Chris Douglas wrote:
> * Does a vetoed proposal require a new vote, or does a vote pass when
> all vetos are recanted?

It passes when all vetoes are recanted, in my experience.  I don't see 
advantages to restarting the vote.

> Can new objections/vetos be raised during that
> debate?

I think that as long as a veto is outstanding and discussion continues, 
the vote should be open and folks can change their votes.  Once both a 
week has passed since voting was started and there are no outstanding 
vetos, the vote passes.  A vote can be cancelled at any time.  If there 
are outstanding vetoes, and a week has passed since the vote was called, 
and discussion has stalled, then the vote should be declared as failed.

Should we be more precise about what constitutes ongoing discussion? 
Should we say, e.g., that, if there are no messages on the voting thread 
for a week and the vote has not passed, then the vote implicitly fails, 
and voting must be restarted if this issue is to be pursued further?

> * With consensus and 2/3 majorities, a 6 month emeritus period seems
> reasonable to me. However, the actions requiring them in the current
> document are rare (I hope), so arguing over these details may not be
> worthwhile. Surely we wouldn't move someone into that category without
> first asking them if they plan to become more active in the near term.
> It's important that this not be used to push people out, particularly
> since the 2/3 and consensus votes may be swayed by "noticing" inactive
> members.

+1 I'm okay with 6 months in the bylaws.  12 months seems to be what we 
do in practice, and we can continue to do that even if our rules say 6 
months.

Doug

From general-return-692-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 03 19:50:00 2009
Return-Path: <general-return-692-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93413 invoked from network); 3 Nov 2009 19:50:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Nov 2009 19:50:00 -0000
Received: (qmail 29027 invoked by uid 500); 3 Nov 2009 19:49:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28947 invoked by uid 500); 3 Nov 2009 19:49:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28937 invoked by uid 99); 3 Nov 2009 19:49:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 19:49:58 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of bernd.fondermann@googlemail.com designates 209.85.219.226 as permitted sender)
Received: from [209.85.219.226] (HELO mail-ew0-f226.google.com) (209.85.219.226)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 19:49:48 +0000
Received: by ewy26 with SMTP id 26so6065725ewy.29
        for <general@hadoop.apache.org>; Tue, 03 Nov 2009 11:49:28 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=73w2vGxYu8CSzIjC2yOyw5eo8ijOj1Sdka4IL22pr60=;
        b=IKxpu8sbvhUGXCwXqsXf9YGzcNQqqCHvWM1ZlQSc1Y3O0S9cQz6RbWGmYYKDtZNLrD
         zOzbpQvwKNm7KdXpfzjqKqiCQ1yYv0USORAVua0Ep0tElsnAlVbYhDCNSLrKhu6Eutrj
         Xmt0smm4MhSiBkUQO2yTPdtizABLQAGHY2HyI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=PPtJFD7TwBmoUc1ta0hEcNY3uRuKbhTlqGtZn0tS8DxVfQfscBwRL2aDFPe5LoAIcH
         qvLIVsK9sJp330cStFIeT85XYWfbUi+YwVt5+nBKNDBzDcQpX6ioBMDSbYE6De9pkTew
         P343QhN4K+/f60irXo2/9jLb+ROsAkhU5x6lM=
MIME-Version: 1.0
Received: by 10.216.90.208 with SMTP id e58mr151638wef.57.1257277768058; Tue, 
	03 Nov 2009 11:49:28 -0800 (PST)
In-Reply-To: <284E0999-F935-47A2-B051-7241BF8A4B39@apache.org>
References: <b23e01b50909291345g47ffb267l76fe193ea72652a3@mail.gmail.com>
	 <4ACCBEE7.3070105@apache.org>
	 <DDB254D5-1F8E-4B75-B8C1-3B3D095BBBC3@yahoo-inc.com>
	 <4ACE1292.6000702@apache.org>
	 <b23e01b50910140711y545fa031u7fb199594db33bd6@mail.gmail.com>
	 <4ADE4055.3020002@yahoo-inc.com>
	 <5764186A-DA78-412B-ADD4-5A9F50866732@yahoo-inc.com>
	 <b23e01b50910291356w17c98e18md0239f30c2aa0a5@mail.gmail.com>
	 <4AEB7728.3000008@yahoo-inc.com>
	 <284E0999-F935-47A2-B051-7241BF8A4B39@apache.org>
Date: Tue, 3 Nov 2009 20:49:27 +0100
Message-ID: <88f6e29a0911031149x5543cc6el7f8cce9bdf1212f0@mail.gmail.com>
Subject: Re: [VOTE] Proposed bylaws for the Hadoop project
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Mon, Nov 2, 2009 at 20:52, Owen O'Malley <omalley@apache.org> wrote:
> We should have created bylaws when we were established as a PMC back in Jan
> 2008, but it has become clear that we need to define them. I looked around
> and the Ant project had bylaws that were both clear and fairly complete. I
> made some minor edits to match what we do. Please look them over and vote.
> In a recursive usage, let's use 2/3 majority to approve these bylaws.

A VOTE thread without having a discussion thread first is unlikely to
not be hijacked by detailed discussion.
I think it will be more successful to collect input first and then
vote on something where all important input is featured in.
As soon as discussions start to evolve on a VOTE thread, nobody knows
what he is/has been voting for, really.

  Bernd

From general-return-693-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 03 22:45:44 2009
Return-Path: <general-return-693-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 51427 invoked from network); 3 Nov 2009 22:45:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Nov 2009 22:45:44 -0000
Received: (qmail 60490 invoked by uid 500); 3 Nov 2009 22:45:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60422 invoked by uid 500); 3 Nov 2009 22:45:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60412 invoked by uid 99); 3 Nov 2009 22:45:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 22:45:42 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 22:45:40 +0000
Received: from [127.0.0.1] (gentlepaint-lx.corp.yahoo.com [10.72.185.127])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id nA3MipIE086365
	for <general@hadoop.apache.org>; Tue, 3 Nov 2009 14:44:54 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=Pa/Fxf3hIe3o12TV64onXgLVn2uLXd+goxpdzL79JpsAHxbptKgjkc9WA4BhiXJ+
Message-ID: <4AF0B263.5000204@yahoo-inc.com>
Date: Tue, 03 Nov 2009 14:44:51 -0800
From: Konstantin Shvachko <shv@yahoo-inc.com>
User-Agent: Thunderbird 2.0.0.23 (Windows/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Proposed bylaws for the Hadoop project
References: <b23e01b50909291345g47ffb267l76fe193ea72652a3@mail.gmail.com>	 <FE14B90F-1F11-4F68-8A6B-3F00DB99D050@apache.org>	 <4ACCBEE7.3070105@apache.org>	 <DDB254D5-1F8E-4B75-B8C1-3B3D095BBBC3@yahoo-inc.com>	 <4ACE1292.6000702@apache.org>	 <b23e01b50910140711y545fa031u7fb199594db33bd6@mail.gmail.com>	 <4ADE4055.3020002@yahoo-inc.com>	 <5764186A-DA78-412B-ADD4-5A9F50866732@yahoo-inc.com> <b23e01b50910291356w17c98e18md0239f30c2aa0a5@mail.gmail.com> <4AEB7728.3000008@yahoo-inc.com> <284E0999-F935-47A2-B051-7241BF8A4B39@apache.org> <4AEF43F3.5070703@apache.org>
In-Reply-To: <4AEF43F3.5070703@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Good idea. +1 for this as a draft version.

1. Seems that Doug's corrections are more in line with the Apache Way, and current hadoop practices +1.

2. I like the definition and phrasing for emeritus in here:
http://portals.apache.org/roles.html
"Emeritus Committers"
which says
- "Any committer that has not participated in the sub-project in the last year
(ie not committed anything in the sub-project repository and not posted anything in the mailing-lists)
will automatically be converted to Emeritus Committer of this sub-project."
- "Any Emeritus Committer of a subproject can ask to become full Committer on the sub-project development list.
He or she will automatically be accepted as Committer without vote."

I propose to use this exact phrasing in Hadoop bylaws.

3. Propose to add "testing" here
- users on the mailing lists, documentation, etc.
+ users on the mailing lists, documentation, testing, etc.

4. Adoption of New Codebase: 2/3 majority; Active committers
This will make very hard to create sub-projects.
Should we replace it with    Lazy Consensus; Active PMC members?
Maintaining codebase is the 1st responsibility of PMC.

--Konstantin


Doug Cutting wrote:
> These look good to me.  +1
> 
> A few minor changes I'd prefer but not insist on:
> 
>  - Replace the term "developer" with "contributor".  This seems more 
> precise and consistent with our wiki documentation.
> 
>  - Emeritus timeout should be 12 months instead of 6.  Perhaps this 
> doesn't matter, since we're not bound to make someone emeritus after six 
> months, but events might reasonably keep someone away for six, and I 
> think we've generally not bothered to make anyone emeritus until they've 
> been idle for 12 months, so 12 months seems more descriptive of our 
> actual practice.
> 
> Thanks for proposing this!
> 
> Doug
> 
> Owen O'Malley wrote:
>> We should have created bylaws when we were established as a PMC back 
>> in Jan 2008, but it has become clear that we need to define them. I 
>> looked around and the Ant project had bylaws that were both clear and 
>> fairly complete. I made some minor edits to match what we do. Please 
>> look them over and vote. In a recursive usage, let's use 2/3 majority 
>> to approve these bylaws.
>>
>> -- Owen
>>
>>
>> ------------------------------------------------------------------------
>>
>>
>>
> 


From general-return-694-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 03 23:13:39 2009
Return-Path: <general-return-694-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60891 invoked from network); 3 Nov 2009 23:13:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Nov 2009 23:13:39 -0000
Received: (qmail 95576 invoked by uid 500); 3 Nov 2009 23:13:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95506 invoked by uid 500); 3 Nov 2009 23:13:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95496 invoked by uid 99); 3 Nov 2009 23:13:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 23:13:37 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 23:13:35 +0000
Received: from [10.35.1.231] (snvvpn1-10-72-244-c90.hq.corp.yahoo.com [10.72.244.90])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id nA3NCnE6033024
	for <general@hadoop.apache.org>; Tue, 3 Nov 2009 15:12:49 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=pctZz5Vq8hNHRcD2kgPTb6rU6Il244wtsd147iMTTRj8M6cpNsEQdKfK6fOb6ufn
Message-Id: <2E38916E-FB50-4AD5-908A-27AD79B4E189@yahoo-inc.com>
From: Nigel Daley <ndaley@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <5f7f740b0911030809s2e57cd57gb9e91ecb30e28cab@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Proposed bylaws for the Hadoop project
Date: Tue, 3 Nov 2009 15:12:48 -0800
References: <b23e01b50909291345g47ffb267l76fe193ea72652a3@mail.gmail.com> <DDB254D5-1F8E-4B75-B8C1-3B3D095BBBC3@yahoo-inc.com> <4ACE1292.6000702@apache.org> <b23e01b50910140711y545fa031u7fb199594db33bd6@mail.gmail.com> <4ADE4055.3020002@yahoo-inc.com> <5764186A-DA78-412B-ADD4-5A9F50866732@yahoo-inc.com> <b23e01b50910291356w17c98e18md0239f30c2aa0a5@mail.gmail.com> <4AEB7728.3000008@yahoo-inc.com> <284E0999-F935-47A2-B051-7241BF8A4B39@apache.org> <4AF00E07.8050701@apache.org> <5f7f740b0911030809s2e57cd57gb9e91ecb30e28cab@mail.gmail.com>
X-Mailer: Apple Mail (2.936)


On Nov 3, 2009, at 8:09 AM, Owen O'Malley wrote:

> On Tue, Nov 3, 2009 at 3:03 AM, Steve Loughran <stevel@apache.org>  
> wrote:
>
>>
>> silly Q, but what are "work days" in this context? Were I to propose
>> something on, say thankgsiviing thursday day, with all the US off  
>> work until
>> monday, we in rest of the world would leave you US folk with only  
>> monday to
>> veto our attempts to vote something through.
>>
>
> Good point! It is what we've done. The Ant bylaws actually use a  
> week. Let's
> change it to a week, since that is roughly equivalent and handles  
> all of the
> special cases of holidays.

Does this mean release votes will now take 1 week too?

Nige

From general-return-695-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 03 23:34:25 2009
Return-Path: <general-return-695-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69278 invoked from network); 3 Nov 2009 23:34:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Nov 2009 23:34:25 -0000
Received: (qmail 19636 invoked by uid 500); 3 Nov 2009 23:34:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19532 invoked by uid 500); 3 Nov 2009 23:34:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19522 invoked by uid 99); 3 Nov 2009 23:34:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 23:34:23 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cabiwan@gmail.com designates 209.85.218.223 as permitted sender)
Received: from [209.85.218.223] (HELO mail-bw0-f223.google.com) (209.85.218.223)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 23:34:21 +0000
Received: by bwz23 with SMTP id 23so8248826bwz.29
        for <general@hadoop.apache.org>; Tue, 03 Nov 2009 15:34:00 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=TJQDxMvy9kkP5UhP/QA9br18SElic83ee9KQwIRUxdA=;
        b=UGlKnS/E/4e4waP4bypEvz+/F6iAMNlcO49Ir2sWOBeme3auqIRd6vu2wU+fsPZUxJ
         1g6SS7891tJY55lLbkA5yWBMZO9mjXjgNuqhqI7pnwdfAwxf2Iiucs2Aq7nHMyR7Ud1t
         2I03LATKQeUcAeT2yl3vRztJRs7eEIpoEwzD0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=fUf2tW0ydodOheg3Dzz87/yrgb2dcUjwjyTOE++WDYdqWNk2GMtq/8M23SgbIOIWF4
         Dav2IKQi2A4uRtrSILOv4cvVAckcUu3jmzJTZmmzYe/xgvlBRJF0lyMECOjeoK7jl1pk
         hAUnK1qogLm4pVEjOpJKTlokPtdIe5An1O0Ao=
MIME-Version: 1.0
Received: by 10.102.178.11 with SMTP id a11mr246892muf.129.1257291240034; Tue, 
	03 Nov 2009 15:34:00 -0800 (PST)
In-Reply-To: <2e63a6a90911010930l794e04aag7e006d7caaf1077f@mail.gmail.com>
References: <309f76d00911010855q155d8efdw12eab526feac9432@mail.gmail.com>
	 <2e63a6a90911010930l794e04aag7e006d7caaf1077f@mail.gmail.com>
Date: Wed, 4 Nov 2009 00:33:59 +0100
Message-ID: <309f76d00911031533x67ee3e3fs32ff854a16ff4c2f@mail.gmail.com>
Subject: Re: TWO QUESTIONS ABOUT LOGS & DEBUGGING IN HADOOP
From: Alberto Luengo Cabanillas <cabiwan@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163641797dbdd2e104777fedd2

--00163641797dbdd2e104777fedd2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Harshad, the point is that I want to read the content of a file in HDFS
from inside my Java code to use the information (i.e. I have some file that
has one number per line, so the first one is the number of iterations, the
second is a loop limit, etc, and I need to read them to store them in a
variable). goutham suggested "IOUtils"...I=B4ll take a look.
Thanks for the reply.

2009/11/1 Harshad Shrikhande <harsh.107@gmail.com>

> Hi,
>     You can read the content of a file in Hadoop Distributed File System
> using
>  bin/hadoop dfs -cat file name
>
> What else do you want to do . It's simple
>
> Bye
> Harshad Shrikhande
>
>
> On Sun, Nov 1, 2009 at 10:25 PM, Alberto Luengo Cabanillas <
> cabiwan@gmail.com> wrote:
>
> > Hi everyone! In a project involving Hadoop (0.20.1) I=B4m working, I=B4=
d need
> > to
> > know if I=B4m reading properly the content of a file located in the HDF=
S.
> > Beside the try...catch block, is there any other option for printing
> values
> > in the console (in a similar way standard IO Java mechanisms do, like
> > System.out.println("the variable is ", var))?
> > Another thing, is there a simply way to debug Hadoop applications?
> > (actually
> > I=B4m working in pseudo-distributed mode, for developing purposes).
> > Thanks a lot in advance.
> >
> > --
> > Alberto
> >
> > --
> > This message has been scanned for viruses and
> > dangerous content by MailScanner, and is
> > believed to be clean.
> >
> >
>



--=20
Alberto

--00163641797dbdd2e104777fedd2--

From general-return-696-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 04 01:54:26 2009
Return-Path: <general-return-696-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 52364 invoked from network); 4 Nov 2009 01:54:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Nov 2009 01:54:25 -0000
Received: (qmail 76450 invoked by uid 500); 4 Nov 2009 01:54:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76372 invoked by uid 500); 4 Nov 2009 01:54:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76362 invoked by uid 99); 4 Nov 2009 01:54:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Nov 2009 01:54:24 +0000
X-ASF-Spam-Status: No, hits=-2.4 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of hkumar.arora@gmail.com designates 209.85.223.171 as permitted sender)
Received: from [209.85.223.171] (HELO mail-iw0-f171.google.com) (209.85.223.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Nov 2009 01:54:20 +0000
Received: by iwn1 with SMTP id 1so4884546iwn.2
        for <general@hadoop.apache.org>; Tue, 03 Nov 2009 17:53:59 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=VTwCCw6Bkkb927IOFcn2NPC92fK3+T5SmhQAiByEVYw=;
        b=OaQYoE6XMISRJdyJl5LoxJKyGjdikOYCZSJURwNehT7Xy4R4Nhqa+9kv3uOvxaQ8Tr
         3FM+f8nEQUp26F30JhixciQAyHHgpz4FpYGp9aKlx+NCCp1LnPaJvVpLuiMkSvRvULUB
         RjIjsbjfRUfbdQUs8uY5x0+ZFUy4EjMD4Nr2E=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=DjELgyvBLdWoc9Tm1TMCeE/EDaR++MjJ2G7c6VMX/O3H9ec3MfeHaNJoHgMHjxj1mU
         DCJNzuvbbeNfalnJpmKV2l0O1+ZyszVwudtnjgwxk+HWGYLymbc2N9BreAWYMJ/uJLpP
         +Wep7qRq3S0YbsTxeYg/fdLdp1HSvvNIECzPY=
MIME-Version: 1.0
Received: by 10.231.170.201 with SMTP id e9mr1337905ibz.17.1257299639671; Tue, 
	03 Nov 2009 17:53:59 -0800 (PST)
In-Reply-To: <98b459018b7877975c9f42e0d05dc1ab.squirrel@mail.tcenet>
References: <f9d76217185ad6ffef5d3414f2390152.squirrel@mail.tce.edu>
	 <e497591c0911021814r3d27ca5fy53581c1492388fee@mail.gmail.com>
	 <98b459018b7877975c9f42e0d05dc1ab.squirrel@mail.tcenet>
Date: Wed, 4 Nov 2009 10:53:59 +0900
Message-ID: <e497591c0911031753kfb02230p2add2e92737af727@mail.gmail.com>
Subject: Re: Http 404 Error while viewing tasktracker in Browser
From: Harshit Kumar <hkumar.arora@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=005045018102661f17047781e2a3

--005045018102661f17047781e2a3
Content-Type: text/plain; charset=UTF-8

Hi

Other reasons can be hadoop jobtracker failed to start, check if namenode,
secondarynamenode, task tracker, job tracker are all up or not.

Cheers
H. Kumar
Phone(Mobile): +82-10-2892-9663
Phone(Office): +82-31-
skype: harshit900
Blog: http://harshitkumar.wordpress.com
Website: http:/kumarharmuscat.tripod.com


2009/11/3 Abhilaash <abhilaash_cse@tce.edu>

>
>
>
> Hi,
>
>
>  The URL I typed was http://localhost:50030/jobtracker.jsp and I'll check
> for the open ports.
>
>
> Thank you,
> Abhilaash.
>
>
> > Can you please share the URL that you typed. Also, may be the port is
> > blocked by the firewall, check the open ports on your system.
> >
> > Cheers
> > H. Kumar
> > skype: harshit900
> > Blog: http://harshitkumar.wordpress.com
> > Website: http:/kumarharmuscat.tripod.com
> >
> >
> > 2009/11/2 Abhilaash <abhilaash_cse@tce.edu>
> >
> >>
> >> Hi all,
> >>
> >>      I have downloaded hadoop and configured it in my local system.I
> >> have
> >> run the wordcount program with some files as dataset .It run
> >> successfully but when I opened the browser to see GUI for
> >> tasktracker and job tracker , it shows me the following error:
> >>
> >> HTTP ERROR: 404
> >>
> >> /jobtracker.jsp
> >>
> >> RequestURI=/jobtracker.jsp
> >>
> >> Powered by Jetty://
> >>
> >>
> >> Can anyone please help me and tell how to see results in GUI?
> >>
> >>
> >> Thank you in advance,
> >>
> >> R.Abhilaash.
> >>
> >>
> >> --
> >> Nothing is impossible because impossible says i'm possible...
> >>
> >>
> >> -----------------------------------------
> >> This email was sent using TCEMail Service.
> >> Thiagarajar College of Engineering
> >> Madurai-625 015, India
> >>
> >>
> >
>
>
> --
> Nothing is impossible because impossible says i'm possible...
>
>
> -----------------------------------------
> This email was sent using TCEMail Service.
> Thiagarajar College of Engineering
> Madurai-625 015, India
>
>

--005045018102661f17047781e2a3--

From general-return-697-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 04 04:51:24 2009
Return-Path: <general-return-697-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 90851 invoked from network); 4 Nov 2009 04:51:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Nov 2009 04:51:24 -0000
Received: (qmail 78052 invoked by uid 500); 4 Nov 2009 04:51:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77740 invoked by uid 500); 4 Nov 2009 04:51:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77727 invoked by uid 99); 4 Nov 2009 04:51:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Nov 2009 04:51:20 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of harshad@students.iiit.ac.in designates 121.242.23.201 as permitted sender)
Received: from [121.242.23.201] (HELO students.iiit.ac.in) (121.242.23.201)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Nov 2009 04:51:18 +0000
MailScanner-NULL-Check: 1257915022.6916@M8YWMux39yo3ILIRx3F3pA
Received: from students.iiit.ac.in (localhost.localdomain [127.0.0.1])
	by students.iiit.ac.in (8.13.8/8.13.8) with ESMTP id nA44oBA2011677
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Wed, 4 Nov 2009 10:20:21 +0530
Received: (from apache@localhost)
	by students.iiit.ac.in (8.13.8/8.14.1/Submit) id nA44nwdR011567;
	Wed, 4 Nov 2009 10:19:58 +0530
From: harshad@students.iiit.ac.in
X-Authentication-Warning: students.iiit.ac.in: apache set sender to harshad@localhost using -f
Received: from 172.17.11.61
        (SquirrelMail authenticated user harshad)
        by students.iiit.ac.in with HTTP;
        Wed, 4 Nov 2009 10:19:58 +0530 (IST)
Message-ID: <36701.172.17.11.61.1257310198.squirrel@students.iiit.ac.in>
Date: Wed, 4 Nov 2009 10:19:58 +0530 (IST)
Subject: General doubts about Google App Engine
To: general@hadoop.apache.org
User-Agent: SquirrelMail/1.4.8-4.el5
MIME-Version: 1.0
Content-Type: text/plain;charset=iso-8859-1
Content-Transfer-Encoding: 8bit
X-Priority: 3 (Normal)
Importance: Normal
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on students.iiit.ac.in
X-yoursite-MailScanner-Information: Please contact the IIIT Server Room  for more information
X-MailScanner-ID: nA44oBA2011677
X-yoursite-MailScanner: Found to be clean
X-yoursite-MailScanner-From: harshad@students.iiit.ac.in
X-Old-Spam-Status: No, score=-1.4 required=5.0 tests=ALL_TRUSTED autolearn=failed
	version=3.2.5, No

Hi,
   Is there any possibility of Google app Engine running Map Reduce behind
its framework?. I am doing a project on Google App Engine which has to
expose it into different nodes i.e into TaskTrackers, Jobtrackers.
   Basically I have to balance the load which exists while running any
application behind the scenes. I have to assign different TaskTrackers
on different machines. Does Google App Engine provide support for this?

Thank You.
Harshad Shrikhande


-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.


From general-return-698-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 04 05:04:59 2009
Return-Path: <general-return-698-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93139 invoked from network); 4 Nov 2009 05:04:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Nov 2009 05:04:59 -0000
Received: (qmail 86816 invoked by uid 500); 4 Nov 2009 05:04:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86682 invoked by uid 500); 4 Nov 2009 05:04:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86672 invoked by uid 99); 4 Nov 2009 05:04:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Nov 2009 05:04:57 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of matei@eecs.berkeley.edu designates 169.229.60.87 as permitted sender)
Received: from [169.229.60.87] (HELO gateway0.EECS.Berkeley.EDU) (169.229.60.87)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Nov 2009 05:04:48 +0000
Received: from [192.168.1.4] (c-69-181-219-249.hsd1.ca.comcast.net [69.181.219.249])
	(authenticated bits=0)
	by gateway0.EECS.Berkeley.EDU (8.14.3/8.13.5) with ESMTP id nA454OOa025050
	(version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT)
	for <general@hadoop.apache.org>; Tue, 3 Nov 2009 21:04:25 -0800 (PST)
Subject: Re: General doubts about Google App Engine
Mime-Version: 1.0 (Apple Message framework v1076)
Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes
From: Matei Zaharia <matei@eecs.berkeley.edu>
X-Priority: 3 (Normal)
In-Reply-To: <36701.172.17.11.61.1257310198.squirrel@students.iiit.ac.in>
Date: Tue, 3 Nov 2009 21:04:23 -0800
Content-Transfer-Encoding: 7bit
Message-Id: <C93E7356-3C85-4AF6-A35C-D2A3BD639226@eecs.berkeley.edu>
References: <36701.172.17.11.61.1257310198.squirrel@students.iiit.ac.in>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1076)
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Harshad,

Is there any reason you're using App Engine instead of EC2? App Engine  
is designed for web sites, so in general, I think you'd have to do a  
lot of work to make it run MapReduce, and it wouldn't be particularly  
good at it.

On Nov 3, 2009, at 8:49 PM, harshad@students.iiit.ac.in wrote:

> Hi,
>   Is there any possibility of Google app Engine running Map Reduce  
> behind
> its framework?. I am doing a project on Google App Engine which has to
> expose it into different nodes i.e into TaskTrackers, Jobtrackers.
>   Basically I have to balance the load which exists while running any
> application behind the scenes. I have to assign different TaskTrackers
> on different machines. Does Google App Engine provide support for  
> this?
>
> Thank You.
> Harshad Shrikhande
>
>
> -- 
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>


From general-return-699-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 04 05:19:14 2009
Return-Path: <general-return-699-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 8687 invoked from network); 4 Nov 2009 05:19:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Nov 2009 05:19:14 -0000
Received: (qmail 99455 invoked by uid 500); 4 Nov 2009 05:19:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99389 invoked by uid 500); 4 Nov 2009 05:19:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 99379 invoked by uid 99); 4 Nov 2009 05:19:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Nov 2009 05:19:13 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of harshad@students.iiit.ac.in designates 121.242.23.201 as permitted sender)
Received: from [121.242.23.201] (HELO students.iiit.ac.in) (121.242.23.201)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Nov 2009 05:19:05 +0000
MailScanner-NULL-Check: 1257916710.27959@KAqVIuMKThiIR5kpDPgS9w
Received: from students.iiit.ac.in (localhost.localdomain [127.0.0.1])
	by students.iiit.ac.in (8.13.8/8.13.8) with ESMTP id nA45ITRv021644
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Wed, 4 Nov 2009 10:48:29 +0530
Received: (from apache@localhost)
	by students.iiit.ac.in (8.13.8/8.14.1/Submit) id nA45ITdS021643;
	Wed, 4 Nov 2009 10:48:29 +0530
From: harshad@students.iiit.ac.in
X-Authentication-Warning: students.iiit.ac.in: apache set sender to harshad@localhost using -f
Received: from 172.17.11.61
        (SquirrelMail authenticated user harshad)
        by students.iiit.ac.in with HTTP;
        Wed, 4 Nov 2009 10:48:29 +0530 (IST)
Message-ID: <55737.172.17.11.61.1257311909.squirrel@students.iiit.ac.in>
In-Reply-To: <C93E7356-3C85-4AF6-A35C-D2A3BD639226@eecs.berkeley.edu>
References: <36701.172.17.11.61.1257310198.squirrel@students.iiit.ac.in>
    <C93E7356-3C85-4AF6-A35C-D2A3BD639226@eecs.berkeley.edu>
Date: Wed, 4 Nov 2009 10:48:29 +0530 (IST)
Subject: Re: General doubts about Google App Engine
To: general@hadoop.apache.org
User-Agent: SquirrelMail/1.4.8-4.el5
MIME-Version: 1.0
Content-Type: text/plain;charset=iso-8859-1
Content-Transfer-Encoding: 8bit
X-Priority: 3 (Normal)
Importance: Normal
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on students.iiit.ac.in
X-yoursite-MailScanner-Information: Please contact the IIIT Server Room  for more information
X-MailScanner-ID: nA45ITRv021644
X-yoursite-MailScanner: Found to be clean
X-yoursite-MailScanner-From: harshad@students.iiit.ac.in
X-Virus-Checked: Checked by ClamAV on apache.org
X-Old-Spam-Status: No, score=-1.4 required=5.0 tests=ALL_TRUSTED autolearn=failed
	version=3.2.5, No

Hi,
   It's the sole reason I am using App Engine. I have to deploy a Web
application on it using Java. The whole point is that I have to make
the running of application look like it's being run on a single
TaskTracker but actually, it runs on many. Thus , in short I have to
expose Google App Engine into many nodes.


Thank You.
Harshad

> Hi Harshad,
>
> Is there any reason you're using App Engine instead of EC2? App Engine
> is designed for web sites, so in general, I think you'd have to do a
> lot of work to make it run MapReduce, and it wouldn't be particularly
> good at it.
>
> On Nov 3, 2009, at 8:49 PM, harshad@students.iiit.ac.in wrote:
>
>> Hi,
>>   Is there any possibility of Google app Engine running Map Reduce
>> behind
>> its framework?. I am doing a project on Google App Engine which has to
>> expose it into different nodes i.e into TaskTrackers, Jobtrackers.
>>   Basically I have to balance the load which exists while running any
>> application behind the scenes. I have to assign different TaskTrackers
>> on different machines. Does Google App Engine provide support for
>> this?
>>
>> Thank You.
>> Harshad Shrikhande
>>
>>
>> --
>> This message has been scanned for viruses and
>> dangerous content by MailScanner, and is
>> believed to be clean.
>>
>
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>


-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.


From general-return-700-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 04 08:50:20 2009
Return-Path: <general-return-700-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58578 invoked from network); 4 Nov 2009 08:50:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Nov 2009 08:50:20 -0000
Received: (qmail 52690 invoked by uid 500); 4 Nov 2009 08:50:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52602 invoked by uid 500); 4 Nov 2009 08:50:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52592 invoked by uid 99); 4 Nov 2009 08:50:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Nov 2009 08:50:18 +0000
X-ASF-Spam-Status: No, hits=-2.2 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of avsrit2005@gmail.com designates 209.85.216.198 as permitted sender)
Received: from [209.85.216.198] (HELO mail-px0-f198.google.com) (209.85.216.198)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Nov 2009 08:50:14 +0000
Received: by pxi36 with SMTP id 36so585086pxi.2
        for <general@hadoop.apache.org>; Wed, 04 Nov 2009 00:49:54 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=xqDtmMvWZQcgDhWkTdQ2wINfq0f5BFnCyKf3z4LKVUw=;
        b=EkZ/sfAzgkQggDkgpm1GNazNqyIQshnHBTEjmxg0ey401/JXhkhOcKGuTW2b1NAKIm
         HBg8F3x5xnDTD4Kr/BkSYIe0N/bqPdMOTRbUVlLcbv0YMpO4d7u/kGgSsq8KiZyvNys4
         ePPd4g7WRVlZiHp9QvskXhmGCjhS0qcBTGFC4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=l+RMacbQYC2jWqEjLGH+yETgPel9i6np47xIpGxYkNPOSe2d94Cb5TDGask3mF5erj
         DRXtr9XCYObSIGT7SwhrFB0R/YI7X6x7waInXV5cDOhzFhHx4WcjHUR4mOODemRSy7I7
         p45qxHJ42MUuvYoFUstsjFpNDCMxIAweHD+0g=
MIME-Version: 1.0
Received: by 10.140.148.11 with SMTP id v11mr67349rvd.58.1257324594734; Wed, 
	04 Nov 2009 00:49:54 -0800 (PST)
Date: Wed, 4 Nov 2009 14:19:54 +0530
Message-ID: <da0a2a330911040049j455a4ffcm52068007c6821bd2@mail.gmail.com>
Subject: Too many fetch failures
From: venkata subbarayudu <avsrit2005@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd22936d626a9047787b1c3

--000e0cd22936d626a9047787b1c3
Content-Type: text/plain; charset=ISO-8859-1

Hi All,
          I was setup a single node hadoop (hadoop-0.20.0) and is able to
run that have only Map Tasks, but If a task have Map & Reduce tasks, then
MapTask is thorwing exceptions saying "Too many fetch failures" and the
reduceTask is not starting at all. can some one please guide whats the error
is, and how to resolve this..
         One thing I noted is eventhough I setup hadoop as localhost (in
config files), I am seeing the IP of the machine on tasktracker,namenode
logs saying TaskTracker starting on server : <ip of the machine>, instead of
localhost or 127.0.0.1. and so I am understanding from where hadoop is
picking up the IP of the machine.

                               Thanks for you help in advance,
Thanks,
Rayudu.

--000e0cd22936d626a9047787b1c3--

From general-return-701-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 04 10:55:08 2009
Return-Path: <general-return-701-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87418 invoked from network); 4 Nov 2009 10:55:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Nov 2009 10:55:08 -0000
Received: (qmail 88994 invoked by uid 500); 4 Nov 2009 10:55:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88938 invoked by uid 500); 4 Nov 2009 10:55:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88928 invoked by uid 99); 4 Nov 2009 10:55:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Nov 2009 10:55:07 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ryanobjc@gmail.com designates 209.85.222.195 as permitted sender)
Received: from [209.85.222.195] (HELO mail-pz0-f195.google.com) (209.85.222.195)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Nov 2009 10:55:03 +0000
Received: by pzk33 with SMTP id 33so4918838pzk.2
        for <general@hadoop.apache.org>; Wed, 04 Nov 2009 02:54:43 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=2VMWqj2R/pKIfj37kHjSKhOBf2qCH+tYOy5O3XtB56o=;
        b=Dh210Mh7xb8lEfg4baWDLiRmgM9hy3dQdtKdYvN0WkySHo/RieYhYhCUUl8D7MkZxr
         6H6yPcgfJy+Y9zC6HG/viadJQmSF3NmB09B6Yy2Ep5KDzFost9lYYKMqfU5WE68HXb2Y
         EpkE1RjGABu9Fv2afl5R6+jcBSUlvDvqACyHc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=E35SnTq3KPsYlpYez45hAkUvuLUPpVEnLOIWnUPPT3uvTG6YI3uc7Me7U8wp0MAEjZ
         QbsbhEqbVtpq/avUJgAw+3d+Gm8qNVIvXrPneXO5uOIBZNsBNdZf1pdTheESSt2hL5TM
         c3NG5O87hiwU4LSL6LAO2UqmoTUYOHGh95O2g=
MIME-Version: 1.0
Received: by 10.114.3.29 with SMTP id 29mr1747142wac.208.1257332083461; Wed, 
	04 Nov 2009 02:54:43 -0800 (PST)
In-Reply-To: <55737.172.17.11.61.1257311909.squirrel@students.iiit.ac.in>
References: <36701.172.17.11.61.1257310198.squirrel@students.iiit.ac.in>
	 <C93E7356-3C85-4AF6-A35C-D2A3BD639226@eecs.berkeley.edu>
	 <55737.172.17.11.61.1257311909.squirrel@students.iiit.ac.in>
Date: Wed, 4 Nov 2009 02:54:43 -0800
Message-ID: <78568af10911040254h10f4cb31yd22ddb10ce85c65f@mail.gmail.com>
Subject: Re: General doubts about Google App Engine
From: Ryan Rawson <ryanobjc@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Google app engine has the model whereby all computation is done in the
context of a HTTP request, with fairly hard limits on CPU, memory and
thread resources.  There is no way to create a 'daemon', or long
running thread. That would defeat the horizontal scalability benefits
of a system like GAE.

Like the other requester said, EC2 might be more your thing.

On Tue, Nov 3, 2009 at 9:18 PM,  <harshad@students.iiit.ac.in> wrote:
> Hi,
> =A0 It's the sole reason I am using App Engine. I have to deploy a Web
> application on it using Java. The whole point is that I have to make
> the running of application look like it's being run on a single
> TaskTracker but actually, it runs on many. Thus , in short I have to
> expose Google App Engine into many nodes.
>
>
> Thank You.
> Harshad
>
>> Hi Harshad,
>>
>> Is there any reason you're using App Engine instead of EC2? App Engine
>> is designed for web sites, so in general, I think you'd have to do a
>> lot of work to make it run MapReduce, and it wouldn't be particularly
>> good at it.
>>
>> On Nov 3, 2009, at 8:49 PM, harshad@students.iiit.ac.in wrote:
>>
>>> Hi,
>>> =A0 Is there any possibility of Google app Engine running Map Reduce
>>> behind
>>> its framework?. I am doing a project on Google App Engine which has to
>>> expose it into different nodes i.e into TaskTrackers, Jobtrackers.
>>> =A0 Basically I have to balance the load which exists while running any
>>> application behind the scenes. I have to assign different TaskTrackers
>>> on different machines. Does Google App Engine provide support for
>>> this?
>>>
>>> Thank You.
>>> Harshad Shrikhande
>>>
>>>
>>> --
>>> This message has been scanned for viruses and
>>> dangerous content by MailScanner, and is
>>> believed to be clean.
>>>
>>
>>
>> --
>> This message has been scanned for viruses and
>> dangerous content by MailScanner, and is
>> believed to be clean.
>>
>
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>
>

From general-return-702-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 04 14:38:47 2009
Return-Path: <general-return-702-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 52539 invoked from network); 4 Nov 2009 14:38:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Nov 2009 14:38:47 -0000
Received: (qmail 28863 invoked by uid 500); 4 Nov 2009 14:38:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28789 invoked by uid 500); 4 Nov 2009 14:38:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28778 invoked by uid 99); 4 Nov 2009 14:38:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Nov 2009 14:38:45 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of fzappert@gmail.com designates 209.85.222.195 as permitted sender)
Received: from [209.85.222.195] (HELO mail-pz0-f195.google.com) (209.85.222.195)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Nov 2009 14:38:40 +0000
Received: by pzk33 with SMTP id 33so5028900pzk.2
        for <general@hadoop.apache.org>; Wed, 04 Nov 2009 06:38:20 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:from:to
         :in-reply-to:content-type:content-transfer-encoding:x-mailer
         :mime-version:subject:date:references;
        bh=MLav0GMBu6KLzm39aVblxngZ/PVZR4ASS+Br0cy3gJ8=;
        b=ZicDYyi1GN2MGGHLTC7nglnY+iLigtMbJdZ6BTiSLIci2p6OEKneMslj4Ms5b1RNOe
         AvUom+B31hx7qf4nKe3WH71E3XXT00C1h8BSNIPRG1OgEAi510OuFMXgdZTnaNfeeNB/
         WFvF2rSYiTKdm5On7tGGWkTx87rDVjCqWwnRw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:from:to:in-reply-to:content-type
         :content-transfer-encoding:x-mailer:mime-version:subject:date
         :references;
        b=BfK8d0ZhdLzKmYAIskDyNaocfKn9mhWIXWcj+Df9yjeeQP+A4bJN8eT+TlgtzpJcXP
         uWHGPa30VdQ3YO8rAC94r0OBRxFNny9ibI/HniBzfJlkX0rQmD4yxyMzV+XT4v6uENgu
         CqMO8UOyOU7M11B4nDtXXfPewsR1mhwuFYK+U=
Received: by 10.115.115.14 with SMTP id s14mr2128487wam.189.1257345499991;
        Wed, 04 Nov 2009 06:38:19 -0800 (PST)
Received: from ?10.164.88.93? ([166.205.130.210])
        by mx.google.com with ESMTPS id 23sm648550pxi.5.2009.11.04.06.38.17
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Wed, 04 Nov 2009 06:38:19 -0800 (PST)
Message-Id: <F479B66D-EC66-4963-9279-43F442B31E59@gmail.com>
From: Fred Zappert <fzappert@gmail.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <55737.172.17.11.61.1257311909.squirrel@students.iiit.ac.in>
Content-Type: text/plain;
	charset=us-ascii;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit
X-Mailer: iPhone Mail (7D11)
Mime-Version: 1.0 (iPhone Mail 7D11)
Subject: Re: General doubts about Google App Engine
Date: Wed, 4 Nov 2009 06:38:11 -0800
References: <36701.172.17.11.61.1257310198.squirrel@students.iiit.ac.in> <C93E7356-3C85-4AF6-A35C-D2A3BD639226@eecs.berkeley.edu> <55737.172.17.11.61.1257311909.squirrel@students.iiit.ac.in>

Hi,

You can run any Java app server you like on EC2, such as Tomcat or  
Resin, and also take advantage of the AWS MapReduce framework.

HTH,

Fred

Sent from my iPhone

On Nov 3, 2009, at 9:18 PM, harshad@students.iiit.ac.in wrote:

> Hi,
>   It's the sole reason I am using App Engine. I have to deploy a Web
> application on it using Java. The whole point is that I have to make
> the running of application look like it's being run on a single
> TaskTracker but actually, it runs on many. Thus , in short I have to
> expose Google App Engine into many nodes.
>
>
> Thank You.
> Harshad
>
>> Hi Harshad,
>>
>> Is there any reason you're using App Engine instead of EC2? App  
>> Engine
>> is designed for web sites, so in general, I think you'd have to do a
>> lot of work to make it run MapReduce, and it wouldn't be particularly
>> good at it.
>>
>> On Nov 3, 2009, at 8:49 PM, harshad@students.iiit.ac.in wrote:
>>
>>> Hi,
>>>  Is there any possibility of Google app Engine running Map Reduce
>>> behind
>>> its framework?. I am doing a project on Google App Engine which  
>>> has to
>>> expose it into different nodes i.e into TaskTrackers, Jobtrackers.
>>>  Basically I have to balance the load which exists while running any
>>> application behind the scenes. I have to assign different  
>>> TaskTrackers
>>> on different machines. Does Google App Engine provide support for
>>> this?
>>>
>>> Thank You.
>>> Harshad Shrikhande
>>>
>>>
>>> --
>>> This message has been scanned for viruses and
>>> dangerous content by MailScanner, and is
>>> believed to be clean.
>>>
>>
>>
>> --
>> This message has been scanned for viruses and
>> dangerous content by MailScanner, and is
>> believed to be clean.
>>
>
>
> -- 
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>

From general-return-703-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 04 15:40:48 2009
Return-Path: <general-return-703-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73415 invoked from network); 4 Nov 2009 15:40:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Nov 2009 15:40:48 -0000
Received: (qmail 45040 invoked by uid 500); 4 Nov 2009 15:40:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44977 invoked by uid 500); 4 Nov 2009 15:40:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44967 invoked by uid 99); 4 Nov 2009 15:40:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Nov 2009 15:40:47 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of abhilaash_cse@tce.edu designates 210.212.252.72 as permitted sender)
Received: from [210.212.252.72] (HELO tce.edu) (210.212.252.72)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Nov 2009 15:40:44 +0000
Received: from [127.0.0.1] (helo=mail.tce.edu)
	by tce.edu with esmtp (Exim 4.63)
	(envelope-from <abhilaash_cse@tce.edu>)
	id 1N5hxa-0006Op-3s
	for general@hadoop.apache.org; Wed, 04 Nov 2009 21:09:58 +0530
Received: from 59.92.117.60
        (SquirrelMail authenticated user abhilaash_cse)
        by mail.tce.edu with HTTP;
        Wed, 4 Nov 2009 21:09:46 +0530
Message-ID: <a2ead6540e86acc058af817fb1703a7d.squirrel@mail.tce.edu>
In-Reply-To: <e497591c0911031753kfb02230p2add2e92737af727@mail.gmail.com>
References: <f9d76217185ad6ffef5d3414f2390152.squirrel@mail.tce.edu>
    <e497591c0911021814r3d27ca5fy53581c1492388fee@mail.gmail.com>
    <98b459018b7877975c9f42e0d05dc1ab.squirrel@mail.tcenet>
    <e497591c0911031753kfb02230p2add2e92737af727@mail.gmail.com>
Date: Wed, 4 Nov 2009 21:09:46 +0530
Subject: Re: Http 404 Error while viewing tasktracker in Browser
From: "Abhilaash" <abhilaash_cse@tce.edu>
To: general@hadoop.apache.org
User-Agent: SquirrelMail/1.4.19
MIME-Version: 1.0
Content-Type: text/plain;charset=iso-8859-1
Content-Transfer-Encoding: 8bit
X-Priority: 3 (Normal)
Importance: Normal
X-XheaderVersion: 1.1
X-UserAgent: 
X-Spam-Score: 0.1 (/)
X-Old-Spam-Status: No



Hi,

The Jobtracker has failed to start so only I had that error.Now it is
working and I can view the GUI.

Thank you,

R.Abhilaash.

> Hi
>
> Other reasons can be hadoop jobtracker failed to start, check if namenode,
> secondarynamenode, task tracker, job tracker are all up or not.
>
> Cheers
> H. Kumar
> Phone(Mobile): +82-10-2892-9663
> Phone(Office): +82-31-
> skype: harshit900
> Blog: http://harshitkumar.wordpress.com
> Website: http:/kumarharmuscat.tripod.com
>
>
> 2009/11/3 Abhilaash <abhilaash_cse@tce.edu>
>
>>
>>
>>
>> Hi,
>>
>>
>>  The URL I typed was http://localhost:50030/jobtracker.jsp and I'll
>> check
>> for the open ports.
>>
>>
>> Thank you,
>> Abhilaash.
>>
>>
>> > Can you please share the URL that you typed. Also, may be the port is
>> > blocked by the firewall, check the open ports on your system.
>> >
>> > Cheers
>> > H. Kumar
>> > skype: harshit900
>> > Blog: http://harshitkumar.wordpress.com
>> > Website: http:/kumarharmuscat.tripod.com
>> >
>> >
>> > 2009/11/2 Abhilaash <abhilaash_cse@tce.edu>
>> >
>> >>
>> >> Hi all,
>> >>
>> >>      I have downloaded hadoop and configured it in my local system.I
>> >> have
>> >> run the wordcount program with some files as dataset .It run
>> >> successfully but when I opened the browser to see GUI for
>> >> tasktracker and job tracker , it shows me the following error:
>> >>
>> >> HTTP ERROR: 404
>> >>
>> >> /jobtracker.jsp
>> >>
>> >> RequestURI=/jobtracker.jsp
>> >>
>> >> Powered by Jetty://
>> >>
>> >>
>> >> Can anyone please help me and tell how to see results in GUI?
>> >>
>> >>
>> >> Thank you in advance,
>> >>
>> >> R.Abhilaash.
>> >>
>> >>
>> >> --
>> >> Nothing is impossible because impossible says i'm possible...
>> >>
>> >>
>> >> -----------------------------------------
>> >> This email was sent using TCEMail Service.
>> >> Thiagarajar College of Engineering
>> >> Madurai-625 015, India
>> >>
>> >>
>> >
>>
>>
>> --
>> Nothing is impossible because impossible says i'm possible...
>>
>>
>> -----------------------------------------
>> This email was sent using TCEMail Service.
>> Thiagarajar College of Engineering
>> Madurai-625 015, India
>>
>>
>


-- 
Nothing is impossible because impossible says i'm possible...


-----------------------------------------
This email was sent using TCEMail Service.
Thiagarajar College of Engineering
Madurai-625 015, India


From general-return-704-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 04 23:28:44 2009
Return-Path: <general-return-704-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9232 invoked from network); 4 Nov 2009 22:40:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Nov 2009 22:40:50 -0000
Received: (qmail 92466 invoked by uid 500); 4 Nov 2009 22:40:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92373 invoked by uid 500); 4 Nov 2009 22:40:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 68335 invoked by uid 99); 4 Nov 2009 20:08:25 -0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ptarjan@gmail.com designates 209.85.222.195 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        bh=cUpVM+eHOa22BxEBFFOjKzh+0+FSCbA/+yk38Gca644=;
        b=RXe36AIsziXwwGQquO43Wi6HxGx9SoYMJ2AL+u3EYNwLWELy7fefVZvnSvXmtjRPjc
         5VM3MNkKqpvTFznEhZgXx57zyhkHJDvKu8PvmqW47QiHvL2H+v5vCmPcJbgxMS5zeCcW
         5wSAbWobJoX9QIF+vHInzf+PS8B9YPuzPDaac=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        b=rW/o30ODs/2Rw2udml6Cr3qx8K92gX0mqVnYwo1WYpWsOj8b2S7INqn+EiHYVQZUcy
         ek6FybgCt0BrE+o8vVcvdAF8CFrqG4ecQPHZp9/KYa2ej2yQx7Qay5CcisAmB3cD9NNc
         RWVVoBrAoJaVYvEAtxVNip2at2tt1MgySwdYg=
MIME-Version: 1.0
Sender: ptarjan@gmail.com
Date: Wed, 4 Nov 2009 12:07:53 -0800
X-Google-Sender-Auth: 1075ee41898a28bb
Message-ID: <815c8c330911041207p1689d6b2u378d52c0142afe84@mail.gmail.com>
Subject: Python CSV record reader
From: Paul Tarjan <paul@paulisageek.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

I wrote a CSV Jute record parser in python, and thought some people on
the list might also be interested.

http://github.com/ptarjan/hadoop_record

You can use it in your streaming jobs with

-inputformat SequenceFileAsTextInputFormat -file hadoop_record.mod

And just showing some features:

>>> from hadoop_record import csv
>>> csv("T")
True
>>> csv(";-1234")
-1234
>>> csv("1.0E-10")
1e-10
>>> csv("s{T,F}")
[True, False]
>>> csv("v{T,F}")
[True, False]
>>> csv("v{s{T,F}}")
[[True, False]]
>>> csv("m{'don't,#73746f70}")
{LazyString("don't"): LazyString('stop')}
>>> csv("'\xe2\x98\x83")
LazyString('\xe2\x98\x83')
>>> str(csv("'\xe2\x98\x83"))
'\xe2\x98\x83'
>>> unicode(csv("'\xe2\x98\x83"))
u'\u2603'
>>> csv("'%00%0a%25%2c")
LazyString('\x00\n%,')

The LazyString was needed because I was spending most of my CPU just
decoding data from the Jute record that I didn't care about. It
shouldn't get in your way too much, as long as you cast it to a str
first.

So let me know what you think. For bugs, fork, fix, and then send me a
pull request (or use the issues tracker).

Paul

From general-return-705-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 05 10:48:25 2009
Return-Path: <general-return-705-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17274 invoked from network); 5 Nov 2009 10:48:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Nov 2009 10:48:25 -0000
Received: (qmail 713 invoked by uid 500); 5 Nov 2009 10:48:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 632 invoked by uid 500); 5 Nov 2009 10:48:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 622 invoked by uid 99); 5 Nov 2009 10:48:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Nov 2009 10:48:23 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Nov 2009 10:48:13 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 6B140B7F9D
	for <general@hadoop.apache.org>; Thu,  5 Nov 2009 10:47:53 +0000 (GMT)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id K5V69-IJq7tY for <general@hadoop.apache.org>;
	Thu,  5 Nov 2009 10:47:46 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id A84DAB7F9A
	for <general@hadoop.apache.org>; Thu,  5 Nov 2009 10:47:46 +0000 (GMT)
MailScanner-NULL-Check: 1258022856.62668@yp6pJuZeicLvwGZHqPVGRA
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id nA5AlZdL026346
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Thu, 5 Nov 2009 10:47:35 GMT
Message-ID: <4AF2AD47.40908@apache.org>
Date: Thu, 05 Nov 2009 10:47:35 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Proposed bylaws for the Hadoop project
References: <b23e01b50909291345g47ffb267l76fe193ea72652a3@mail.gmail.com>	 <FE14B90F-1F11-4F68-8A6B-3F00DB99D050@apache.org>	 <4ACCBEE7.3070105@apache.org>	 <DDB254D5-1F8E-4B75-B8C1-3B3D095BBBC3@yahoo-inc.com>	 <4ACE1292.6000702@apache.org>	 <b23e01b50910140711y545fa031u7fb199594db33bd6@mail.gmail.com>	 <4ADE4055.3020002@yahoo-inc.com>	 <5764186A-DA78-412B-ADD4-5A9F50866732@yahoo-inc.com> <b23e01b50910291356w17c98e18md0239f30c2aa0a5@mail.gmail.com> <4AEB7728.3000008@yahoo-inc.com> <284E0999-F935-47A2-B051-7241BF8A4B39@apache.org> <4AEF43F3.5070703@apache.org> <4AF0B263.5000204@yahoo-inc.com>
In-Reply-To: <4AF0B263.5000204@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: nA5AlZdL026346
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

Konstantin Shvachko wrote:

> 3. Propose to add "testing" here
> - users on the mailing lists, documentation, etc.
> + users on the mailing lists, documentation, testing, etc.

and bugreps

> 
> 4. Adoption of New Codebase: 2/3 majority; Active committers
> This will make very hard to create sub-projects.
> Should we replace it with    Lazy Consensus; Active PMC members?
> Maintaining codebase is the 1st responsibility of PMC.
> 

good point, this is probably PMC level issues, maybe 2/3 PMC -it's a 
fairly big decision and assuming the PMC is small enough, majority isn't 
that hard to organise. It's harder with committers, who can come and go 
based on personal whim/work committments

From general-return-706-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 05 11:07:55 2009
Return-Path: <general-return-706-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21631 invoked from network); 5 Nov 2009 11:07:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Nov 2009 11:07:55 -0000
Received: (qmail 28065 invoked by uid 500); 5 Nov 2009 11:07:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28015 invoked by uid 500); 5 Nov 2009 11:07:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28005 invoked by uid 99); 5 Nov 2009 11:07:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Nov 2009 11:07:53 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Nov 2009 11:07:43 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 020FAB7F9F
	for <general@hadoop.apache.org>; Thu,  5 Nov 2009 11:07:22 +0000 (GMT)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id vGV3WSPFSxa0 for <general@hadoop.apache.org>;
	Thu,  5 Nov 2009 11:07:16 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 257DDB7FA0
	for <general@hadoop.apache.org>; Thu,  5 Nov 2009 11:07:15 +0000 (GMT)
MailScanner-NULL-Check: 1258024025.84772@wPRGB82aTRlReyrwSUgAGw
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id nA5B73LT026872
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Thu, 5 Nov 2009 11:07:03 GMT
Message-ID: <4AF2B1D7.7090907@apache.org>
Date: Thu, 05 Nov 2009 11:07:03 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Proposed bylaws for the Hadoop project
References: <284E0999-F935-47A2-B051-7241BF8A4B39@apache.org>	 <C715E3D7.2F176%yhemanth@yahoo-inc.com> <5f7f740b0911030808m3ac21126h1271b39d7ae30cac@mail.gmail.com>
In-Reply-To: <5f7f740b0911030808m3ac21126h1271b39d7ae30cac@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: nA5B73LT026872
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

Owen O'Malley wrote:
> On Tue, Nov 3, 2009 at 12:12 AM, Hemanth Yamijala <yhemanth@yahoo-inc.com>wrote:
> 
>> Owen
>>
>> Some clarifications:
>>
>> In the section on "Code Change" when you mention approval as "Lazy approval
>> and then lazy consensus", did you mean the consensus will need to come into
>> play if a veto is received ?
>>
> 
> Hmm... I copied that section from Ant's bylaws. Our current practice is
> probably actually lazy consensus since we require a +1 before it can be
> committed.
> 

Ant works with commit-then-discuss, nobody works on it for a living, 
it's all voluntary, and the rate of change of the codebase is fairly 
low. There's little bureaucracy overhead to encourage code contribs -and 
every deverloper is expected to look at the commit log to see what is 
going on.

what's not spelled out in Ant's docs are that the architecture of the 
system is very compartmentalised, where it's obvious from  the source 
files what's going to break, so just by looking at the source filenames 
you can guess what the issues will be

-optional obscure SCM tasks: any changes here are welcome
-other optional stuff: those classes and subclasses may break, if they 
are used a lot review carefully
-junit : all the CI servers, lots of hate mail
-java, exec, : how code runs, lots of CI builds fail, many complaints, 
the IDE teams stop sending you xmas cards.
-<fileset> et al. complex, discuss first, tread carefully and worry 
about memory consumption
-IntrospectionHelper, TaskAdapter, the code that maps from XML to java; 
you have to really know what you are doing before going near this code, 
and unless the rest of the team trusts you, you won't get a lookin. 
Nobody touches this stuff without lots of discussion, more tests etc as 
it breaks everything
-Classpath work. Everyone worries about this, nobody really understands 
it. Anyone who says they do is probably misguided.

Because of that hierarchy, you can get a vague idea of where trouble 
will be without even reading the diffs, just looking at the subject 
lines, and if you get people with problems on optional tasks to start 
providing tests then patches, you get them fixing the problem and sucked 
into the task of codebase maintenance.

-steve


From general-return-707-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 05 12:28:30 2009
Return-Path: <general-return-707-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 36913 invoked from network); 5 Nov 2009 12:28:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Nov 2009 12:28:29 -0000
Received: (qmail 16329 invoked by uid 500); 5 Nov 2009 12:28:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16249 invoked by uid 500); 5 Nov 2009 12:28:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16233 invoked by uid 99); 5 Nov 2009 12:28:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Nov 2009 12:28:29 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of avsrit2005@gmail.com designates 209.85.160.50 as permitted sender)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Nov 2009 12:28:19 +0000
Received: by pwi4 with SMTP id 4so4093158pwi.29
        for <multiple recipients>; Thu, 05 Nov 2009 04:27:57 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=eRPjAChzi/CdxkJ6KyYvPBChrx3MdMLXYnE4WZFuZVU=;
        b=Skao9BxXNUL2vMYhP90o3AV8uNEdmKjoOSqyrG3ZtDvKJRvrKUSlrBvUe4CS/97opW
         rGWR/suboqUsVu78Rj4mVS5Ro6nBQf2Y+vhzH3SCkgUnYKmnieOhNGuoRdy5bidYSbbG
         aUCu/sjf09TrXcYMLajw00EC9sl02NgFEN/X8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=HLX1eJOILeqf6UB96msgU8QVJTVypPXXCxFUG9TO1RQdfO3JU5PmKk3aXQlUxSw8Wx
         XxvcyFKsDVEFTLR7xADMoMjeOXwcy4S6Zcx8Pv9PHg0jx+/sB4SlLQ/NvFd0IUqNjLNs
         zjoHizRbxDzMqboxhlwJ/cnPuUx+3YOhtBxOc=
MIME-Version: 1.0
Received: by 10.141.3.4 with SMTP id f4mr164506rvi.104.1257424077440; Thu, 05 
	Nov 2009 04:27:57 -0800 (PST)
Date: Thu, 5 Nov 2009 17:57:57 +0530
Message-ID: <da0a2a330911050427h5a273f8x6600e771536f7052@mail.gmail.com>
Subject: Hadoop : Too many fetch failures -- Reducer doesn't start
From: venkata subbarayudu <avsrit2005@gmail.com>
To: general@hadoop.apache.org, mapreduce-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd1136277c71604779edbef
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd1136277c71604779edbef
Content-Type: text/plain; charset=ISO-8859-1

Hi All
- I have setup a 'single node hadoop' (hadoop-version 0.20.0)  one localhost
by following the instructions from '
http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Single-Node_Cluster)'
.  (i.e. master&slave nodes are 'localhost'.)

I was able to run MapReduce tasks with no problems in a standalone system,
and
now I am trying to setup the same on a different system, and this system has
two IP addresses, the new setup looks fine (because all the hadoop processes
like datanode, namenode, secondarynamenode, tasktracker, jobtracker were
started). and I'm able to run Hadoop Jobs that have only Mapper Tasks. but
for Jobs that have Map&Reduce tasks, Hadoop Map Task is throwing '*Too Many
fetch failures*', exception.


Can Somebody please suggest and give any insights on how the problem can be
resolved.
---------------------------------------------------------------------------------------------------------------------------------------------------
I think, the reason for this exception is that, some how the Reducer task is
not able to read o/p of MapperTask, and this communication happens over a
dns that was specified by the following properties in hadoop-spec [Please
see the config file below].

           *hdfs-site.xml*

           <property>
           <name>dfs.datanode.dns.interface</name>
                     <value>default</value>
                     <description>The name of the Network Interface from
which a data node should  report its IP address. </description>
           </property>

         <property>
                 <name>dfs.datanode.dns.nameserver</name>
                 <value>default</value>
                 <description>The host name or IP address of the name server
(DNS)
                      which a DataNode should use to determine the host name
used by the
                      NameNode for communication and display purposes.
                </description>
         </property>

        *mapred-site.xml*

        <property>
                 <name>mapred.tasktracker.dns.interface</name>
                 <value>default</value>
                <description>The name of the Network Interface from which a
task tracker should report its IP address. </description>
        </property>

        <property>
              <name>mapred.tasktracker.dns.nameserver</name>
              <value>default</value>
              <description>The host name or IP address of the name server
(DNS)
                          which a TaskTracker should use to determine the
host name used by
                          the JobTracker for communication and display
purposes.
             </description>
       </property>
-----------------------------------------------------------------------------------------------------------------------------------------------------------

Our Server has 2 IP addresses and the hadoop-user can access the system
using 2nd IP. I believe, As the value specified above is "default" -->
hadoop gets the localhost hostname and uses this(IP1) for further
communication.As the hadoop-user is not allowed to access the system using
the IP1, the communication fails, and the Map/Reduce Task is not able to
report/read the hdfs-data. which caused Hadoop to throw 'Too Many Fetch
failures' exception.

please correct me If the above explanation is incorrect, quick reply is much
appreciated.

Thanks,
Rayudu.

--000e0cd1136277c71604779edbef--

From general-return-708-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 05 20:31:49 2009
Return-Path: <general-return-708-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19858 invoked from network); 5 Nov 2009 20:31:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Nov 2009 20:31:48 -0000
Received: (qmail 1703 invoked by uid 500); 5 Nov 2009 20:31:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1615 invoked by uid 500); 5 Nov 2009 20:31:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1605 invoked by uid 99); 5 Nov 2009 20:31:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Nov 2009 20:31:48 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of dhruba@gmail.com designates 209.85.210.194 as permitted sender)
Received: from [209.85.210.194] (HELO mail-yx0-f194.google.com) (209.85.210.194)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Nov 2009 20:31:37 +0000
Received: by yxe32 with SMTP id 32so406885yxe.5
        for <general@hadoop.apache.org>; Thu, 05 Nov 2009 12:31:16 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=cs460NudNuCQJBgyf/4s/iV2GvavoCEhykKj0/fg+S8=;
        b=S1LHFOYLb2jTLzK0hrvfKIRDazP7GdsW89q+ufYT5afwEY0d2wR/ivQs3PeSjtirxW
         WNYBtIveF37zGtHfT8JUSkf5YKiaWHC2QZVBz69tl/gI5tO/cyreN0EV9VZrm1BLqEWO
         yUv/jGmfjXPRCDSUzitMDXVKt5YpRYsq9hr8U=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Rnz+a5MCGIbmGmuFQ9R0awT0sc+j1Z867KwdecZFqeT8t3h2f63mb51evvmh3XzG1B
         9Mao+13OFxSMWhiEQ0d6f0RIsuHIuxLA1AJgYnB99osYkLjWJsUMYZCyQqjKd1C9uExU
         GBMry/NTBDiwghmnjLa3GXvoYrbsYKB3uyPAw=
MIME-Version: 1.0
Received: by 10.151.88.18 with SMTP id q18mr6001405ybl.307.1257453073696; Thu, 
	05 Nov 2009 12:31:13 -0800 (PST)
In-Reply-To: <4AF2AD47.40908@apache.org>
References: <b23e01b50909291345g47ffb267l76fe193ea72652a3@mail.gmail.com>
	 <b23e01b50910140711y545fa031u7fb199594db33bd6@mail.gmail.com>
	 <4ADE4055.3020002@yahoo-inc.com>
	 <5764186A-DA78-412B-ADD4-5A9F50866732@yahoo-inc.com>
	 <b23e01b50910291356w17c98e18md0239f30c2aa0a5@mail.gmail.com>
	 <4AEB7728.3000008@yahoo-inc.com>
	 <284E0999-F935-47A2-B051-7241BF8A4B39@apache.org>
	 <4AEF43F3.5070703@apache.org> <4AF0B263.5000204@yahoo-inc.com>
	 <4AF2AD47.40908@apache.org>
Date: Thu, 5 Nov 2009 12:31:13 -0800
Message-ID: <4aa34eb70911051231h620d2e82od0b4fd5f23306916@mail.gmail.com>
Subject: Re: [VOTE] Proposed bylaws for the Hadoop project
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd648ccc78b0e0477a59b2a
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd648ccc78b0e0477a59b2a
Content-Type: text/plain; charset=ISO-8859-1

>From my understanding, the section related to "Adoption of a new CodeBase"
refer to wholesale replacement of existing codebase with a new one. For that
operation, this document proposes to have a 2/3 majority. It does not apply
to creating new "contrib" modules inside an existing project. Creating new
contrib modules within an existing project has the same requirements as
needed by any bug-fix (i.e. "Lazy approval")

Just for fun (no offense to anybody), does anybody know the procedure to
impeach the Chair of the PMC :-)?

thanks,
dhruba


>
>> 4. Adoption of New Codebase: 2/3 majority; Active committers
>> This will make very hard to create sub-projects.
>> Should we replace it with    Lazy Consensus; Active PMC members?
>> Maintaining codebase is the 1st responsibility of PMC.
>>
>>
>

-- 
Connect to me at http://www.facebook.com/dhruba

--000e0cd648ccc78b0e0477a59b2a--

From general-return-709-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 05 23:25:20 2009
Return-Path: <general-return-709-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 89567 invoked from network); 5 Nov 2009 23:25:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Nov 2009 23:25:20 -0000
Received: (qmail 90232 invoked by uid 500); 5 Nov 2009 23:25:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90151 invoked by uid 500); 5 Nov 2009 23:25:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90141 invoked by uid 99); 5 Nov 2009 23:25:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Nov 2009 23:25:19 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.59.243] (HELO QMTA13.westchester.pa.mail.comcast.net) (76.96.59.243)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Nov 2009 23:25:09 +0000
Received: from OMTA05.westchester.pa.mail.comcast.net ([76.96.62.43])
	by QMTA13.westchester.pa.mail.comcast.net with comcast
	id 1WJv1d0060vyq2s5DbQppL; Thu, 05 Nov 2009 23:24:49 +0000
Received: from [10.12.10.255] ([216.75.233.114])
	by OMTA05.westchester.pa.mail.comcast.net with comcast
	id 1bQg1d0062UlRY83RbQj2E; Thu, 05 Nov 2009 23:24:47 +0000
Message-Id: <CE15A3C6-FA1B-4C21-B002-D3CE91CDF8F9@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <4aa34eb70911051231h620d2e82od0b4fd5f23306916@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Proposed bylaws for the Hadoop project
Date: Thu, 5 Nov 2009 15:24:37 -0800
References: <b23e01b50909291345g47ffb267l76fe193ea72652a3@mail.gmail.com> <b23e01b50910140711y545fa031u7fb199594db33bd6@mail.gmail.com> <4ADE4055.3020002@yahoo-inc.com> <5764186A-DA78-412B-ADD4-5A9F50866732@yahoo-inc.com> <b23e01b50910291356w17c98e18md0239f30c2aa0a5@mail.gmail.com> <4AEB7728.3000008@yahoo-inc.com> <284E0999-F935-47A2-B051-7241BF8A4B39@apache.org> <4AEF43F3.5070703@apache.org> <4AF0B263.5000204@yahoo-inc.com> <4AF2AD47.40908@apache.org> <4aa34eb70911051231h620d2e82od0b4fd5f23306916@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


On Nov 5, 2009, at 12:31 PM, Dhruba Borthakur wrote:

> Just for fun (no offense to anybody), does anybody know the  
> procedure to
> impeach the Chair of the PMC :-)?

The same way we remove any uppity PMC members. *smile* (Consensus - 1)

-- Owen

From general-return-710-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 06 02:04:57 2009
Return-Path: <general-return-710-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39682 invoked from network); 6 Nov 2009 02:04:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Nov 2009 02:04:57 -0000
Received: (qmail 67287 invoked by uid 500); 6 Nov 2009 02:04:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67217 invoked by uid 500); 6 Nov 2009 02:04:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67207 invoked by uid 99); 6 Nov 2009 02:04:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Nov 2009 02:04:56 +0000
X-ASF-Spam-Status: No, hits=-6.7 required=5.0
	tests=AWL,BAYES_00,RCVD_IN_DNSWL_HI
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 06 Nov 2009 02:04:54 +0000
Received: (qmail 39588 invoked by uid 99); 6 Nov 2009 02:04:33 -0000
Received: from localhost.apache.org (HELO mail-bw0-f223.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Nov 2009 02:04:33 +0000
Received: by bwz23 with SMTP id 23so697418bwz.29
        for <general@hadoop.apache.org>; Thu, 05 Nov 2009 18:04:31 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.154.203 with SMTP id p11mr3874583bkw.180.1257473071578; 
	Thu, 05 Nov 2009 18:04:31 -0800 (PST)
Date: Thu, 5 Nov 2009 18:04:31 -0800
Message-ID: <1267dd3b0911051804jc8d915bqba0b2d235a8e289e@mail.gmail.com>
Subject: Re: Proposed bylaws for the Hadoop project (was: [VOTE])
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I think Bernd's prediction has come true: it's no longer clear what
form of the bylaws we're voting on. Quick summary:

Proposed, uncontroversial (or at least seconded) modifications to the origi=
nal:
* Change "3 business days" to "1 week" for votes
* Change "Developer" to "Contributor"
* Change "Lazy approval then lazy consensus" to simply "Lazy
consensus" for source modifications
* Add "testing" and "bug reports" to explicit forms of contributions from U=
sers

Still open/vague:
* State changes to/from emeritus status for committers and PMC members
  * Inactive status for 6 months vs 1 year
  * Vote of the PMC required to restore status to a) Committer b) PMC
* Definition of "adopt a new codebase"
* When to close a vote that has been vetoed

---

I agree with Dhruba's definition of "new codebase" as it concerns
contrib and second Konstantin's interpretation requiring a 2/3 vote of
the PMC for new subprojects.

Concerning vetoes: the advantages to restarting the vote would be to
explicitly reassert its content (as in this thread) or to notify those
who withheld their objections because they were redundant for the
outcome. I don't know if it needs to be written into the bylaws, but a
few days after a veto has been recanted (assuming its premise wasn't
mistaken) seems appropriate to me. The context matters a great deal,
so I hope we can trust in the discretion of the person who called the
vote.

Restoring a committer/PMC member from emeritus should always require a
vote, with the same rules as those new to the role. Asking the person
after an unexplained, idle 6 months and- on request- extending the
grace period to 12 before automatically moving them sounds fair. That
leaves "no response" as a reasonable criterion for moving an inactive
member, so decisions are not blocked for lapses in participation. -C

On Tue, Nov 3, 2009 at 11:49 AM, Bernd Fondermann
<bernd.fondermann@googlemail.com> wrote:
> On Mon, Nov 2, 2009 at 20:52, Owen O'Malley <omalley@apache.org> wrote:
>> We should have created bylaws when we were established as a PMC back in =
Jan
>> 2008, but it has become clear that we need to define them. I looked arou=
nd
>> and the Ant project had bylaws that were both clear and fairly complete.=
 I
>> made some minor edits to match what we do. Please look them over and vot=
e.
>> In a recursive usage, let's use 2/3 majority to approve these bylaws.
>
> A VOTE thread without having a discussion thread first is unlikely to
> not be hijacked by detailed discussion.
> I think it will be more successful to collect input first and then
> vote on something where all important input is featured in.
> As soon as discussions start to evolve on a VOTE thread, nobody knows
> what he is/has been voting for, really.
>
> =A0Bernd
>

From general-return-711-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 06 18:43:43 2009
Return-Path: <general-return-711-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 40125 invoked from network); 6 Nov 2009 18:43:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Nov 2009 18:43:42 -0000
Received: (qmail 52662 invoked by uid 500); 6 Nov 2009 18:43:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52263 invoked by uid 500); 6 Nov 2009 18:43:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 28098 invoked by uid 99); 6 Nov 2009 17:21:01 -0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Message-ID: <4AF45AE4.7090506@crs4.it>
Date: Fri, 06 Nov 2009 18:20:36 +0100
From: Simone Leo <simone.leo@crs4.it>
User-Agent: Thunderbird 2.0.0.23 (X11/20091007)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: pydoop -- Python MapReduce and HDFS API for Hadoop
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hello everybody,

we recently released pydoop, a Python MapReduce and HDFS API for Hadoop:

http://pydoop.sourceforge.net

It is implemented as a Boost.Python wrapper around the C++ code (pipes
and libhdfs). It allows you to write complete MapReduce application in
CPython, with the same capabilities as the C++ API. Here is a minimal
wordcount example:


from pydoop.pipes import Mapper, Reducer, Factory, runTask

class WordCountMapper(Mapper):

  def __init__(self, context):
    super(WordCountMapper, self).__init__(context)

  def map(self, context):
    words = context.getInputValue().split()
    for w in words:
      context.emit(w, "1")

class WordCountReducer(Reducer):

  def __init__(self, context):
    super(WordCountReducer, self).__init__(context)

  def reduce(self, context):
    s = 0
    while context.nextValue():
      s += int(context.getInputValue())
    context.emit(context.getInputKey(), str(s))

runTask(Factory(WordCountMapper, WordCountReducer))


Any feedback would be greatly appreciated.

-- 
Simone Leo
Distributed Computing group
Advanced Computing and Communications program
CRS4
POLARIS - Building #1
Piscina Manna
I-09010 Pula (CA) - Italy
e-mail: simleo@crs4.it
http://www.crs4.it

From general-return-712-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 06 18:57:45 2009
Return-Path: <general-return-712-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 52729 invoked from network); 6 Nov 2009 18:57:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Nov 2009 18:57:45 -0000
Received: (qmail 84213 invoked by uid 500); 6 Nov 2009 18:57:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84140 invoked by uid 500); 6 Nov 2009 18:57:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84130 invoked by uid 99); 6 Nov 2009 18:57:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Nov 2009 18:57:44 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 06 Nov 2009 18:57:41 +0000
Received: (qmail 52675 invoked by uid 99); 6 Nov 2009 18:57:20 -0000
Received: from localhost.apache.org (HELO [192.168.168.105]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Nov 2009 18:57:20 +0000
Message-ID: <4AF4717D.4090106@apache.org>
Date: Fri, 06 Nov 2009 10:57:01 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Proposed bylaws for the Hadoop project
References: <b23e01b50909291345g47ffb267l76fe193ea72652a3@mail.gmail.com>	 <FE14B90F-1F11-4F68-8A6B-3F00DB99D050@apache.org>	 <4ACCBEE7.3070105@apache.org>	 <DDB254D5-1F8E-4B75-B8C1-3B3D095BBBC3@yahoo-inc.com>	 <4ACE1292.6000702@apache.org>	 <b23e01b50910140711y545fa031u7fb199594db33bd6@mail.gmail.com>	 <4ADE4055.3020002@yahoo-inc.com>	 <5764186A-DA78-412B-ADD4-5A9F50866732@yahoo-inc.com> <b23e01b50910291356w17c98e18md0239f30c2aa0a5@mail.gmail.com> <4AEB7728.3000008@yahoo-inc.com> <284E0999-F935-47A2-B051-7241BF8A4B39@apache.org>
In-Reply-To: <284E0999-F935-47A2-B051-7241BF8A4B39@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

I'm concerned about the use of majority voting.  For example, we've 
always conducted release votes as if by consensus: 3 +1 votes and no -1 
votes are required.  This proposes to change this to "Lazy majority". 
Changing anything from consensus to majority risks a tyranny of the 
majority.  Currently about 2/3 of the Hadoop PMC share an employer.

Issues that I believe we've previously handled with consensus that would 
move to some kind of majority are:
   - Release Plan
   - Release
   - Adoption of New Codebase (e.g. subprojects)
   - Modifying Bylaws

I'd prefer we keep these consensus or consensus-but-one.  As long as we 
maintain sufficient PMC diversity (at least three legally independent 
parties) consensus-but-one would prevent any single entity from 
controlling a vote.

As long as we keep PMC removal consensus-but-one, everything else could 
be consensus: if an individual is consistently blocking project progress 
with vetos, then they can be removed from the PMC, unless their removal 
would create a PMC with fewer than three legally independent entities.

Doug

Owen O'Malley wrote:
> We should have created bylaws when we were established as a PMC back in 
> Jan 2008, but it has become clear that we need to define them. I looked 
> around and the Ant project had bylaws that were both clear and fairly 
> complete. I made some minor edits to match what we do. Please look them 
> over and vote. In a recursive usage, let's use 2/3 majority to approve 
> these bylaws.
> 
> -- Owen
> 
> 
> ------------------------------------------------------------------------
> 
> 
> 

From general-return-713-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 06 19:17:15 2009
Return-Path: <general-return-713-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63083 invoked from network); 6 Nov 2009 19:17:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Nov 2009 19:17:15 -0000
Received: (qmail 19525 invoked by uid 500); 6 Nov 2009 19:17:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19438 invoked by uid 500); 6 Nov 2009 19:17:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19428 invoked by uid 99); 6 Nov 2009 19:17:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Nov 2009 19:17:14 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Nov 2009 19:17:12 +0000
Received: from SNV-EXPF01.ds.corp.yahoo.com (snv-expf01.ds.corp.yahoo.com [207.126.227.250])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id nA6JG5Ce001050
	for <general@hadoop.apache.org>; Fri, 6 Nov 2009 11:16:13 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:x-originalarrivaltime;
	b=L7tkSsYbFEJafd76Hmnu88ZVZ2u85Ajxq+JrcrywsSL3G9C+8OKoZfgo92Lt13ca
Received: from SNV-EXVS01.ds.corp.yahoo.com ([216.145.51.185]) by SNV-EXPF01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Fri, 6 Nov 2009 11:16:07 -0800
Received: from 10.72.106.176 ([10.72.106.176]) by SNV-EXVS01.ds.corp.yahoo.com ([216.145.51.166]) via Exchange Front-End Server snv-webmail.corp.yahoo.com ([207.126.227.248]) with Microsoft Exchange Server HTTP-DAV ;
 Fri,  6 Nov 2009 19:15:10 +0000
User-Agent: Microsoft-Entourage/12.20.0.090605
Date: Fri, 06 Nov 2009 11:15:04 -0800
Subject: Re: HTTP transport?
From: Kan Zhang <kan@yahoo-inc.com>
To: <general@hadoop.apache.org>
Message-ID: <C719B5B8.1A881%kan@yahoo-inc.com>
Thread-Topic: HTTP transport?
Thread-Index: AcpfFXGzHDDYZuExJk2eFgwdaqI7ag==
In-Reply-To: <4AD5FE4E.4040100@apache.org>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-OriginalArrivalTime: 06 Nov 2009 19:16:07.0492 (UTC) FILETIME=[978B4840:01CA5F15]




On 10/14/09 9:37 AM, "Doug Cutting" <cutting@apache.org> wrote:

> Kan Zhang wrote:
>> One problem I see with using HTTP is that it's expensive to provide data
>> encryption. We're currently adding 2 authentication mechanisms (Kerberos and
>> DIGEST-MD5) to our existing RPC. Both of them can provide data encryption
>> for subsequent communication over the authenticated channel. However, when
>> similar authentication mechanisms are specified for HTTP (SPNEGO and HTTP
>> DIGEST, respectively), they don't provide data encryption (correct me if I'm
>> wrong). For data encryption over HTTP, one has to use SSL, which is
>> expensive.
> 
> Java supports using Kerberos-based encryption for TLS (nee SSL):
> 
> http://java.sun.com/j2se/1.5.0/docs/guide/security/jsse/JSSERefGuide.html#KRB
> 
> http://tools.ietf.org/html/rfc2712
> 
Thanks for pointing this out. I did a little testing on it. It seems that
when you use Kerberos cipher suites with SSL, the Kerberos service name for
a TLS server has to be literally "host." For example, a TLS server running
on the machine mach1.imc.org in the Kerberos realm IMC.ORG must use
host/mach1.imc.org@IMC.ORG as its Kerberos principal name. I couldn't find a
way to specify a different service name. Can someone confirm this? This can
be a limitation since we typically run DN and TT on the same set of nodes.

Kan


From general-return-714-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 06 19:55:53 2009
Return-Path: <general-return-714-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78681 invoked from network); 6 Nov 2009 19:55:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Nov 2009 19:55:52 -0000
Received: (qmail 96993 invoked by uid 500); 6 Nov 2009 19:55:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96916 invoked by uid 500); 6 Nov 2009 19:55:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96906 invoked by uid 99); 6 Nov 2009 19:55:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Nov 2009 19:55:51 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ptarjan@gmail.com designates 72.14.220.153 as permitted sender)
Received: from [72.14.220.153] (HELO fg-out-1718.google.com) (72.14.220.153)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Nov 2009 19:55:40 +0000
Received: by fg-out-1718.google.com with SMTP id e12so390313fga.11
        for <general@hadoop.apache.org>; Fri, 06 Nov 2009 11:55:20 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:user-agent:date:subject:from
         :to:message-id:thread-topic:thread-index:in-reply-to:mime-version
         :content-type:content-transfer-encoding;
        bh=O7s8IDrsujK3rKIcaSDgqZzQWMQiDmBtRwhAoohGfBI=;
        b=YG6PdfycOmCBp48n1zZxkrwCch2JuU/bZEFcLIN5V3vFmsyczKt78CqTeBxdX35O88
         I1tkOU2BlRWvO2L4coPUgdGPUtI4zi6R48KrY+7+XKCX279QJCaBtmvnUiJ1RXkIDMtO
         SaZWGOffekjgGMxaDkE1OdwkwW2PzlGMgytP4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=user-agent:date:subject:from:to:message-id:thread-topic
         :thread-index:in-reply-to:mime-version:content-type
         :content-transfer-encoding;
        b=Yzo3lizFsqlHurgmNh7yZaSdeYopaopH3khhsGb304f1QYWp+B7+iuUpTRqt4LXESp
         YkCcH8Alxz+7YvHCoz88Pya2B4ZJDPDNAjumuZyzke6v1uvV3tWfKeN4Xlt3bX2YpOvw
         q1K4FqPcSqzwoOK9pMiK2Z8+U5A0BL9kw9HtY=
Received: by 10.86.106.21 with SMTP id e21mr6010683fgc.67.1257537319767;
        Fri, 06 Nov 2009 11:55:19 -0800 (PST)
Received: from ?10.72.115.102? (nat-dip4.cfw-a-gci.corp.yahoo.com [209.131.62.113])
        by mx.google.com with ESMTPS id l12sm1026309fgb.2.2009.11.06.11.55.17
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Fri, 06 Nov 2009 11:55:18 -0800 (PST)
User-Agent: Microsoft-Entourage/12.20.0.090605
Date: Fri, 06 Nov 2009 11:55:13 -0800
Subject: Re: pydoop -- Python MapReduce and HDFS API for Hadoop
From: Paul Tarjan <ptarjan@gmail.com>
To: <general@hadoop.apache.org>
Message-ID: <C719BF21.17B8C%ptarjan@gmail.com>
Thread-Topic: pydoop -- Python MapReduce and HDFS API for Hadoop
Thread-Index: AcpfGw2TGX7xA0ocXU+hQj7Pt0UGNg==
In-Reply-To: <4AF45AE4.7090506@crs4.it>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Cool. Try reading a CSV Jute record with

http://github.com/ptarjan/hadoop_record

And let me know if it works in CPython in your framework.

Paul

On 11/6/09 9:20 AM, "Simone Leo" <simone.leo@crs4.it> wrote:

> Hello everybody,
> 
> we recently released pydoop, a Python MapReduce and HDFS API for Hadoop:
> 
> http://pydoop.sourceforge.net
> 
> It is implemented as a Boost.Python wrapper around the C++ code (pipes
> and libhdfs). It allows you to write complete MapReduce application in
> CPython, with the same capabilities as the C++ API. Here is a minimal
> wordcount example:
> 
> 
> from pydoop.pipes import Mapper, Reducer, Factory, runTask
> 
> class WordCountMapper(Mapper):
> 
>   def __init__(self, context):
>     super(WordCountMapper, self).__init__(context)
> 
>   def map(self, context):
>     words = context.getInputValue().split()
>     for w in words:
>       context.emit(w, "1")
> 
> class WordCountReducer(Reducer):
> 
>   def __init__(self, context):
>     super(WordCountReducer, self).__init__(context)
> 
>   def reduce(self, context):
>     s = 0
>     while context.nextValue():
>       s += int(context.getInputValue())
>     context.emit(context.getInputKey(), str(s))
> 
> runTask(Factory(WordCountMapper, WordCountReducer))
> 
> 
> Any feedback would be greatly appreciated.



From general-return-715-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 06 21:07:04 2009
Return-Path: <general-return-715-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12140 invoked from network); 6 Nov 2009 21:07:04 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Nov 2009 21:07:04 -0000
Received: (qmail 85776 invoked by uid 500); 6 Nov 2009 21:07:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85670 invoked by uid 500); 6 Nov 2009 21:07:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85660 invoked by uid 99); 6 Nov 2009 21:07:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Nov 2009 21:07:03 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 06 Nov 2009 21:06:52 +0000
Received: (qmail 11977 invoked by uid 99); 6 Nov 2009 21:06:30 -0000
Received: from localhost.apache.org (HELO [192.168.168.105]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Nov 2009 21:06:30 +0000
Message-ID: <4AF48FD5.6070307@apache.org>
Date: Fri, 06 Nov 2009 13:06:29 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: HTTP transport?
References: <C719B5B8.1A881%kan@yahoo-inc.com>
In-Reply-To: <C719B5B8.1A881%kan@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Kan Zhang wrote:
> Thanks for pointing this out. I did a little testing on it. It seems that
> when you use Kerberos cipher suites with SSL, the Kerberos service name for
> a TLS server has to be literally "host." For example, a TLS server running
> on the machine mach1.imc.org in the Kerberos realm IMC.ORG must use
> host/mach1.imc.org@IMC.ORG as its Kerberos principal name. I couldn't find a
> way to specify a different service name. Can someone confirm this? This can
> be a limitation since we typically run DN and TT on the same set of nodes.

This is unfortunate.  It looks to be part of the specification.

BTW, I found an approach to Kerberos over HTTP bypassing SPNEGO:

http://beamdocs.fnal.gov/DocDB/0019/001987/001/KMJ3_1-guide.pdf

Starting on page 13, he suggests having an applet that the browser loads 
to create a ticket.  The ticket is created by the user's browser talking 
directly to Kerberos.  Then the ticket can be used in subsequent 
requests to identify the user.  An application using HTTP could 
similarly contact Kerberos directly to create tickets that are sent with 
requests.  No multi-step HTTP handshake is thus required.

Doug

From general-return-716-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 06 21:26:41 2009
Return-Path: <general-return-716-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17815 invoked from network); 6 Nov 2009 21:26:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Nov 2009 21:26:41 -0000
Received: (qmail 8986 invoked by uid 500); 6 Nov 2009 21:26:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8908 invoked by uid 500); 6 Nov 2009 21:26:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8898 invoked by uid 99); 6 Nov 2009 21:26:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Nov 2009 21:26:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 06 Nov 2009 21:26:37 +0000
Received: (qmail 17762 invoked by uid 99); 6 Nov 2009 21:26:16 -0000
Received: from localhost.apache.org (HELO mail-bw0-f223.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Nov 2009 21:26:16 +0000
Received: by bwz23 with SMTP id 23so1650642bwz.29
        for <general@hadoop.apache.org>; Fri, 06 Nov 2009 13:26:14 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.156.25 with SMTP id u25mr5147785bkw.129.1257542774071; 
	Fri, 06 Nov 2009 13:26:14 -0800 (PST)
In-Reply-To: <1267dd3b0911051804jc8d915bqba0b2d235a8e289e@mail.gmail.com>
References: <1267dd3b0911051804jc8d915bqba0b2d235a8e289e@mail.gmail.com>
Date: Fri, 6 Nov 2009 13:26:14 -0800
Message-ID: <1267dd3b0911061326h32448ff6r28e383be345264a4@mail.gmail.com>
Subject: Proposed bylaws for the Hadoop project (was: [VOTE])
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Sent from the wrong address. -C

---------- Forwarded message ----------
From: Chris Douglas <cdouglas@apache.org>
Date: Thu, Nov 5, 2009 at 6:04 PM
Subject: Re: Proposed bylaws for the Hadoop project (was: [VOTE])
To: general@hadoop.apache.org


I think Bernd's prediction has come true: it's no longer clear what
form of the bylaws we're voting on. Quick summary:

Proposed, uncontroversial (or at least seconded) modifications to the origi=
nal:
* Change "3 business days" to "1 week" for votes
* Change "Developer" to "Contributor"
* Change "Lazy approval then lazy consensus" to simply "Lazy
consensus" for source modifications
* Add "testing" and "bug reports" to explicit forms of contributions from U=
sers

Still open/vague:
* State changes to/from emeritus status for committers and PMC members
=A0* Inactive status for 6 months vs 1 year
=A0* Vote of the PMC required to restore status to a) Committer b) PMC
* Definition of "adopt a new codebase"
* When to close a vote that has been vetoed

---

I agree with Dhruba's definition of "new codebase" as it concerns
contrib and second Konstantin's interpretation requiring a 2/3 vote of
the PMC for new subprojects.

Concerning vetoes: the advantages to restarting the vote would be to
explicitly reassert its content (as in this thread) or to notify those
who withheld their objections because they were redundant for the
outcome. I don't know if it needs to be written into the bylaws, but a
few days after a veto has been recanted (assuming its premise wasn't
mistaken) seems appropriate to me. The context matters a great deal,
so I hope we can trust in the discretion of the person who called the
vote.

Restoring a committer/PMC member from emeritus should always require a
vote, with the same rules as those new to the role. Asking the person
after an unexplained, idle 6 months and- on request- extending the
grace period to 12 before automatically moving them sounds fair. That
leaves "no response" as a reasonable criterion for moving an inactive
member, so decisions are not blocked for lapses in participation. -C

On Tue, Nov 3, 2009 at 11:49 AM, Bernd Fondermann
<bernd.fondermann@googlemail.com> wrote:
> On Mon, Nov 2, 2009 at 20:52, Owen O'Malley <omalley@apache.org> wrote:
>> We should have created bylaws when we were established as a PMC back in =
Jan
>> 2008, but it has become clear that we need to define them. I looked arou=
nd
>> and the Ant project had bylaws that were both clear and fairly complete.=
 I
>> made some minor edits to match what we do. Please look them over and vot=
e.
>> In a recursive usage, let's use 2/3 majority to approve these bylaws.
>
> A VOTE thread without having a discussion thread first is unlikely to
> not be hijacked by detailed discussion.
> I think it will be more successful to collect input first and then
> vote on something where all important input is featured in.
> As soon as discussions start to evolve on a VOTE thread, nobody knows
> what he is/has been voting for, really.
>
> =A0Bernd
>

From general-return-717-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Nov 07 14:50:36 2009
Return-Path: <general-return-717-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 18670 invoked from network); 7 Nov 2009 14:50:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Nov 2009 14:50:36 -0000
Received: (qmail 34899 invoked by uid 500); 7 Nov 2009 14:50:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34826 invoked by uid 500); 7 Nov 2009 14:50:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34816 invoked by uid 99); 7 Nov 2009 14:50:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 Nov 2009 14:50:34 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of fredwang222@gmail.com designates 209.85.216.204 as permitted sender)
Received: from [209.85.216.204] (HELO mail-px0-f204.google.com) (209.85.216.204)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 Nov 2009 14:50:31 +0000
Received: by pxi42 with SMTP id 42so1253070pxi.5
        for <general@hadoop.apache.org>; Sat, 07 Nov 2009 06:50:11 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=JlH21olt6GvdwAUUSjvACFm5hFnKWGDq/k10uaRDJAs=;
        b=KdMvjjmTSDX0GkiGe1gKvr0Eg/hjuMxr7btOUdqupYhc4XurHU8oSWkPMTtP/+wjBp
         oAyHetptcUi715V/DwAh13YqlNrVC1nTHxQeHtb+t+Aj0pBfmj4wDyu8RN4o+UK8dEgy
         gEHQG3WkjOKOWkn3fy8Ab27qLBusn3FqIttYk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=tBwVOKitsgi9idJNPNXB1+YrAujIV4rloyx1Y5iXhYMiZHHxXrgam5y00A6b0jdqdo
         r309sLq4GIEhAxi6/TcKH1Xd2pmiEId3bDA9zPz781mcBmr+CIjsDd1YwJIpLyiEFNuw
         T9fL6F/pgOHkYGbERF5dyLQkDd9A0oRbd9tfI=
MIME-Version: 1.0
Received: by 10.142.249.19 with SMTP id w19mr594140wfh.199.1257605411383; Sat, 
	07 Nov 2009 06:50:11 -0800 (PST)
Date: Sat, 7 Nov 2009 22:50:11 +0800
Message-ID: <702261350911070650p37e7e878jf4205c307ac296fb@mail.gmail.com>
Subject: PiEstimator example in my env.(pseudo distributed) run jar works; 
	.class works for local, but doesn't work for non-local(couldn't find class), 
	Need help!
From: fred wang <fredwang222@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502ce82d029510477c913ed

--00504502ce82d029510477c913ed
Content-Type: text/plain; charset=ISO-8859-1

(1)I copied PiEstimator in my env. and build a runnable jar, run the hadoop
jar , it works with correct result

(2) I generated .class in the direcotry ../bin
PiEstimator.class PiEstimator$HaltonSequence.class
PiEstimator$PiMapper.class PiEstimator$PiReducer.class
(3) when I add the ../bin path into the -classpath and try to run (I use
pseudo distributed config) with following error information
java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:354)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 5 more
Caused by: java.lang.RuntimeException: java.lang.RuntimeException:
java.lang.ClassNotFoundException: mypackage.examples.PiEstimator$PiMapper

(4) I run . class with additional options: -fs file:/// -jt local , the
result is correct .

I guess the -classpath doesn't be included in the map or reduce task, which
make .class couldn't be found. Not sure how I could make it found.

Thanks in advance

--00504502ce82d029510477c913ed--

From general-return-718-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 09 17:39:27 2009
Return-Path: <general-return-718-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 32639 invoked from network); 9 Nov 2009 17:39:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Nov 2009 17:39:27 -0000
Received: (qmail 56047 invoked by uid 500); 9 Nov 2009 17:39:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55981 invoked by uid 500); 9 Nov 2009 17:39:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55971 invoked by uid 99); 9 Nov 2009 17:39:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Nov 2009 17:39:25 +0000
X-ASF-Spam-Status: No, hits=-1.8 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of swatt@us.ibm.com designates 32.97.110.149 as permitted sender)
Received: from [32.97.110.149] (HELO e31.co.us.ibm.com) (32.97.110.149)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Nov 2009 17:39:14 +0000
Received: from d03relay05.boulder.ibm.com (d03relay05.boulder.ibm.com [9.17.195.107])
	by e31.co.us.ibm.com (8.14.3/8.13.1) with ESMTP id nA9HVoCp009230
	for <general@hadoop.apache.org>; Mon, 9 Nov 2009 10:31:51 -0700
Received: from d03av05.boulder.ibm.com (d03av05.boulder.ibm.com [9.17.195.85])
	by d03relay05.boulder.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id nA9HcNjI088194
	for <general@hadoop.apache.org>; Mon, 9 Nov 2009 10:38:27 -0700
Received: from d03av05.boulder.ibm.com (loopback [127.0.0.1])
	by d03av05.boulder.ibm.com (8.14.3/8.13.1/NCO v10.0 AVout) with ESMTP id nA9HcNXn030203
	for <general@hadoop.apache.org>; Mon, 9 Nov 2009 10:38:23 -0700
Received: from d03nm123.boulder.ibm.com (d03nm123.boulder.ibm.com [9.17.195.149])
	by d03av05.boulder.ibm.com (8.14.3/8.13.1/NCO v10.0 AVin) with ESMTP id nA9Hc96K029790
	for <general@hadoop.apache.org>; Mon, 9 Nov 2009 10:38:22 -0700
To: general@hadoop.apache.org
MIME-Version: 1.0
Subject: Hadoop Release Schedule
X-KeepSent: 027C9CF7:0E8714C6-86257669:005DFCE0;
 type=4; name=$KeepSent
X-Mailer: Lotus Notes Release 8.5 December 05, 2008
Message-ID: <OF027C9CF7.0E8714C6-ON86257669.005DFCE0-86257669.0060C04A@us.ibm.com>
From: Stephen Watt <swatt@us.ibm.com>
Date: Mon, 9 Nov 2009 11:36:49 -0600
X-MIMETrack: Serialize by Router on D03NM123/03/M/IBM(Release 8.0.1|February 07, 2008) at
 11/09/2009 10:38:22,
	Serialize complete at 11/09/2009 10:38:22
Content-Type: multipart/alternative; boundary="=_alternative 0060C04986257669_="
X-Virus-Checked: Checked by ClamAV on apache.org

--=_alternative 0060C04986257669_=
Content-Type: text/plain; charset="US-ASCII"

If one goes to the Hadoop-Common JIRA and clicks on the RoadMap Option and 
selects "All Versions" as the scope, one is able to see the different 
releases being worked on. In some cases in the past, I've seen what I 
construe to be release dates next the Release Heading. For example, Hadoop 
0.21.0 has 05/June/09 next to it. Are these dates effectively a stake in 
the ground for the community to work towards ? Is this report the 
definitive location to identify how close a particular tagged release is 
to being made generally available ? 

The way it appears to me is that once all the issues tagged for a 
particular release are closed and the release appears to work well in 
system testing (done independently by the community), a vote is taken and 
the release is made available. i.e. a release date that is driven more by 
the closure of open issues and features than a particular date the 
community is working towards. Is that correct ? It seems in the past that 
there has been a release about every 3 months.

Lastly, what about releases that appear to have been passed by, such as 
0.18.4 ?

Kind regards
Steve Watt
--=_alternative 0060C04986257669_=--

From general-return-719-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 10 05:27:29 2009
Return-Path: <general-return-719-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29104 invoked from network); 10 Nov 2009 05:27:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Nov 2009 05:27:29 -0000
Received: (qmail 42227 invoked by uid 500); 10 Nov 2009 05:27:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42148 invoked by uid 500); 10 Nov 2009 05:27:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42138 invoked by uid 99); 10 Nov 2009 05:27:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Nov 2009 05:27:27 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of asrabkin@gmail.com designates 209.85.160.50 as permitted sender)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Nov 2009 05:27:19 +0000
Received: by pwi6 with SMTP id 6so879247pwi.29
        for <general@hadoop.apache.org>; Mon, 09 Nov 2009 21:26:58 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=laJfCckYhbCGLNdzmAu6mr6Btzf1x5UylUY85LycPsE=;
        b=A5haiL2dCzyyEuXUcVoAGWxuClAbnLxAwxLG1Gbn8UJoxT/wg64Bp2hfM5ADoDS90J
         N2VNneB5G5w4cS4LfCvcvZuQpAIOL3wBPWs0N53f1kLOijH9oM+B64dEepeYQ8V9+asa
         tLFxvDHIKbLHoR84TxLYsko7kJK515PknZ3Cc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=SfcHdWU5gMH7t7fJ2x+siCU1MHvSrPQtjDYYbUHCUhU0od1ExwMGw+N6KI935JUU0H
         JFVNTnhFtQ8BweuUC9KbJVC0LL70rjkA1eKtOTpUieb6nWuYIpdiD3XAxPDkXMWRlJGv
         lJ6jxzxBm2fdLMqrFIUVNQWRTsfFxu4DU1vrk=
MIME-Version: 1.0
Received: by 10.143.20.42 with SMTP id x42mr891109wfi.225.1257830818554; Mon, 
	09 Nov 2009 21:26:58 -0800 (PST)
Date: Mon, 9 Nov 2009 21:26:58 -0800
Message-ID: <39b0afc00911092126k5a77ed1dw5a30adbf9cae3407@mail.gmail.com>
Subject: ANNOUNCE: Chukwa 0.3 released
From: Ariel Rabkin <asrabkin@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Howdy all.

The Chukwa development team is very pleased to announce our first public
release, Chukwa 0.3.  Chukwa is a log collection framework designed to
facilitate large-scale log storage and processing with Hadoop. Chukwa  has
been tested at scale and used in several production settings, and is reasonably
robust and well behaved.

See http://hadoop.apache.org/chukwa/docs/r0.3.0/ for an overview of
Chukwa and http://www.apache.org/dyn/closer.cgi/hadoop/chukwa/ for
download links. And feel free to email chukwa-user@hadoop.apache.org with
any questions or concerns.

--Ari

-- 
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department

From general-return-720-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 10 06:20:33 2009
Return-Path: <general-return-720-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 48834 invoked from network); 10 Nov 2009 06:20:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Nov 2009 06:20:32 -0000
Received: (qmail 70052 invoked by uid 500); 10 Nov 2009 06:20:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69887 invoked by uid 500); 10 Nov 2009 06:20:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69802 invoked by uid 99); 10 Nov 2009 06:20:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Nov 2009 06:20:27 +0000
X-ASF-Spam-Status: No, hits=-2.5 required=5.0
	tests=AWL,BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [203.199.18.82] (HELO mail1.impetus.co.in) (203.199.18.82)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Nov 2009 06:20:24 +0000
Received: from mail1.impetus.co.in ([192.168.100.28]) by mail1.impetus.co.in
 ([192.168.100.28]) with mapi; Tue, 10 Nov 2009 11:50:02 +0530
From: Sanjay Sharma <sanjay.sharma@impetus.co.in>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 10 Nov 2009 11:49:45 +0530
Subject: 1st Hadoop India User Group meet
Thread-Topic: 1st Hadoop India User Group meet
Thread-Index: Acphzcv/uxYtGsd9Rv2nKtqHmoWfSw==
Message-ID: <F0C7246DF63C124CA15A38CDC10CCD4A24FB220771@mail1.impetus.co.in>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

We are planning to hold first Hadoop India user group meet up on 28th Novem=
ber 2009 in Noida.

We would be talking about our experiences with Apache Hadoop/Hbase/Hive/PIG=
/Nutch/etc.

The agenda would be:
- Introductions
- Sharing experiences on Hadoop and related technologies
- Establishing agenda for the next few meetings
- Information exchange: tips, tricks, problems and open discussion
- Possible speaker TBD (invitations open!!)  {we do have something to share=
 on "Hadoop for newbie" & "Hadoop Advanced Tuning"}

My company (Impetus) would be providing the meeting room and we should be a=
ble to accommodate around 40-60 friendly people. Coffee, Tea, and some snac=
ks will be provided.

Please join the linked-in Hadoop India User Group (http://www.linkedin.com/=
groups?home=3D&gid=3D2258445&trk=3Danet_ug_hm) OR Yahoo group (http://tech.=
groups.yahoo.com/group/hadoopind/) and confirm your attendance.

Regards,
Sanjay Sharma

Follow our updates on www.twitter.com/impetuscalling.

* Impetus Technologies is exhibiting it capabilities in Mobile and Wireless=
 in the GSMA Mobile Asia Congress, Hong Kong from November 16-18, 2009. Vis=
it http://www.impetus.com/mlabs/GSMA_events.html for details.

NOTE: This message may contain information that is confidential, proprietar=
y, privileged or otherwise protected by law. The message is intended solely=
 for the named addressee. If received in error, please destroy and notify t=
he sender. Any use of this email is prohibited when received in error. Impe=
tus does not represent, warrant and/or guarantee, that the integrity of thi=
s communication has been maintained nor that the communication is free of e=
rrors, virus, interception or interference.

From general-return-721-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 10 09:59:25 2009
Return-Path: <general-return-721-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22144 invoked from network); 10 Nov 2009 09:59:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Nov 2009 09:59:25 -0000
Received: (qmail 13108 invoked by uid 500); 10 Nov 2009 09:59:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13028 invoked by uid 500); 10 Nov 2009 09:59:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13018 invoked by uid 99); 10 Nov 2009 09:59:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Nov 2009 09:59:24 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of nipen.mark@gmail.com designates 209.85.222.195 as permitted sender)
Received: from [209.85.222.195] (HELO mail-pz0-f195.google.com) (209.85.222.195)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Nov 2009 09:59:19 +0000
Received: by pzk33 with SMTP id 33so2865293pzk.2
        for <general@hadoop.apache.org>; Tue, 10 Nov 2009 01:58:59 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=TSkw1Wt8FzjqYjgPSj2vhmBqCx+ijHuHMJbz8WPQEwc=;
        b=AU8IUjQZrAg2BnnqrRytwjvBuXFmftx2Z/PsExGNfbpgXUZwNM2PIArOEqk5rTjD8p
         1LkIsQ4dkzjGVvNuCAI3QmsXiDlQob5KCcex0hYcPUs0m4TPbQMhT+a0sF7T7hHiIncc
         5DBt+TBxaPp2Yb78ZUTThCmM/DwNcCEhwEYFs=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=sqxHuA72wiQxkqptlxnQYG7LSVJf4rYIqcu1Ttj1zMpA2EFmjlOEi08gXJ81gPMtq7
         g0ghC1afGOIt/vKShETRu2Y0Qc26dLLCKdRNNBT+Rp7cm9bTvcGNXTHFian63s+L4jMf
         qHRFzsPtGaL7vH8z8+5dFoR6itOc+DgKN83bw=
MIME-Version: 1.0
Received: by 10.142.7.12 with SMTP id 12mr964613wfg.328.1257847139197; Tue, 10 
	Nov 2009 01:58:59 -0800 (PST)
Date: Tue, 10 Nov 2009 15:43:59 +0545
Message-ID: <b45b7eee0911100158t414633f9pcd3fffc66a221bae@mail.gmail.com>
Subject: Showing hadoop status
From: Mark N <nipen.mark@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502bc63e9e29c0478015b9a

--00504502bc63e9e29c0478015b9a
Content-Type: text/plain; charset=ISO-8859-1

Currently  I was trying to read the log files generated by hadoop so that I
could show the status of map/reduce jobs in user interfaces
(consider that UI is a separate application )

I was looking at JobTracker APIs and was trying use following APIs
 1. jobtracker.getAllJobs()  which returns array of jobstatus objects

 2. Then for each jobStatus we can call
    mapProgress() , isJobCompleted()  methods

The problem is how do I link existing jobTracker ( which is already running
)  with my  code ? i am trying to built something like  " hadoop UI ."

should I use jobClient  APIs ? Also is this a proper approach ?

thanks in advance . N mark.

--00504502bc63e9e29c0478015b9a--

From general-return-722-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 10 11:41:17 2009
Return-Path: <general-return-722-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54600 invoked from network); 10 Nov 2009 11:41:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Nov 2009 11:41:16 -0000
Received: (qmail 27862 invoked by uid 500); 10 Nov 2009 11:41:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27789 invoked by uid 500); 10 Nov 2009 11:41:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27779 invoked by uid 99); 10 Nov 2009 11:41:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Nov 2009 11:41:15 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of luis.junges@gmail.com designates 209.85.219.207 as permitted sender)
Received: from [209.85.219.207] (HELO mail-ew0-f207.google.com) (209.85.219.207)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Nov 2009 11:41:08 +0000
Received: by ewy3 with SMTP id 3so4110986ewy.37
        for <general@hadoop.apache.org>; Tue, 10 Nov 2009 03:40:47 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=fU7p5J0GGvgnmFoK/Ca51bdDmZWTlYhY+NSbGlhe0iU=;
        b=rE3xyzSgq0wXhaxZ/SsIqe7TCFja91ep7o2gq6PyRc7Qo3m597KYni81flGYDLNIvC
         WY5O9XyY3WVu1Ol1LGYX+resePV+ZYduoq7X4zwPXI2UjB6zHc2Rz3ID6Q4lX2DdHUjQ
         XGYO31a9TLqG9S8xkRfOWfE3rlnXGkNkkNwVc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=UHWlb88xG3G6uv+UepqHUriHD/wLyZy8Ljm5n0C+rqIGVeSr/PFndP4nk9BJnasYhv
         DbaCgFo2V2lOmnQmbWb+NxlcyZy5Ib26my18dlx+B8nKhwmelhq6WgHVON/ZDafmi/lG
         Fs4snrQuKXLnCYcGiMlwvDP1fnda8bZNuHFlo=
MIME-Version: 1.0
Received: by 10.213.23.203 with SMTP id s11mr4637311ebb.60.1257853247910; Tue, 
	10 Nov 2009 03:40:47 -0800 (PST)
Date: Tue, 10 Nov 2009 09:40:47 -0200
Message-ID: <752a037e0911100340u619ae830i807b718009e3efbc@mail.gmail.com>
Subject: version 0.20.2 release?
From: Luis Carlos Junges <luis.junges@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cdf75940572d0047802c851
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cdf75940572d0047802c851
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi,

i would like to know when would be the release of the version 0.20.2?

i found the bug HBASE-1327 on it and i can start my hbase cluster because o=
f
it.

http://issues.apache.org/jira/browse/HBASE-1327

is anyway to get a pre-release without this bug?

--=20
"A realidade de cada lugar e de cada =E9poca =E9 uma alucina=E7=E3o coletiv=
a."


Bloom, Howardh

--000e0cdf75940572d0047802c851--

From general-return-723-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 10 17:21:48 2009
Return-Path: <general-return-723-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 10873 invoked from network); 10 Nov 2009 17:21:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Nov 2009 17:21:48 -0000
Received: (qmail 28121 invoked by uid 500); 10 Nov 2009 17:21:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27583 invoked by uid 500); 10 Nov 2009 17:21:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27537 invoked by uid 99); 10 Nov 2009 17:21:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Nov 2009 17:21:45 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00,NO_RDNS_DOTCOM_HELO
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Nov 2009 17:21:41 +0000
Received: from sp1-ex07cas03.ds.corp.yahoo.com (sp1-ex07cas03.ds.corp.yahoo.com [216.252.116.151])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id nAAHKHr0082180;
	Tue, 10 Nov 2009 09:20:18 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:references:in-reply-to:accept-language:
	content-language:x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:
	content-type:content-transfer-encoding:mime-version;
	b=apNfqxijudhAXjZlWJ62XDsd6g7Ya28Ff4QDknkslbgAxyQONu9hNqB0LXkJ7HpI
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas03.ds.corp.yahoo.com ([216.252.116.151]) with mapi; Tue, 10 Nov
 2009 09:20:17 -0800
From: Dekel Tankel <dekel@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>,
        "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Date: Tue, 10 Nov 2009 09:20:15 -0800
Subject: Hadoop User Group (Bay Area) - next Wednesday (Nov 18th) at Yahoo!
Thread-Topic: Hadoop User Group (Bay Area) - next Wednesday (Nov 18th) at
 Yahoo!
Thread-Index: AcoSMRRU8X5auVLKR8ufbpFTQi1hLgQXaiyQCpfO4pAFTtKFYA==
Message-ID: <46E2672895E8744A97D7577A6110858B4FD60B2D55@SP1-EX07VS01.ds.corp.yahoo.com>
References: <46E2672895E8744A97D7577A6110858B4E1AC69A91@SP1-EX07VS01.ds.corp.yahoo.com>
  <46E2672895E8744A97D7577A6110858B4FD39F6EDF@SP1-EX07VS01.ds.corp.yahoo.com>
In-Reply-To: <46E2672895E8744A97D7577A6110858B4FD39F6EDF@SP1-EX07VS01.ds.corp.yahoo.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi all,

We are one week away from the next Bay Area Hadoop User Group  - Yahoo! Sun=
nyvale Campus, next Wednesday (Nov 18th) at 6PM

We have an exciting evening planed:

*Katta, Solr, Lucene and Hadoop - Searching at scale, Jason Rutherglen and =
Jason Venner

*Walking through the New File system API, Sanjay Radia, Yahoo!

*Keep your data in Jute but still use it in python, Paul Tarjan, Yahoo!


Please RSVP here:
http://www.meetup.com/hadoop/calendar/11724002/


Please note that this is the last HUG for 2009, as we will not have a meeti=
ng on December (due to the holidays).
We will open 2010 with a HUG on Jan 20th.

Looking forward to seeing you next week!

Dekel


From general-return-724-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 10 17:25:17 2009
Return-Path: <general-return-724-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12608 invoked from network); 10 Nov 2009 17:25:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Nov 2009 17:25:17 -0000
Received: (qmail 37957 invoked by uid 500); 10 Nov 2009 17:25:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37907 invoked by uid 500); 10 Nov 2009 17:25:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 4594 invoked by uid 99); 10 Nov 2009 11:09:46 -0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE,NO_RDNS_DOTCOM_HELO
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:in-reply-to:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=tJWUiKQGodT9XZK7s0lLBN8D20gSn/Z/XawsEoQsvpejcBDih+McPplHJpK2wMVk
From: Amogh Vasekar <amogh@yahoo-inc.com>
To: "mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>,
        "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 10 Nov 2009 16:38:44 +0530
Subject: Re: 1st Hadoop India User Group meet
Thread-Topic: 1st Hadoop India User Group meet
Thread-Index: Acphzcv/uxYtGsd9Rv2nKtqHmoWfSwAKF640
Message-ID: <C71F4794.3940%amogh@yahoo-inc.com>
In-Reply-To: <F0C7246DF63C124CA15A38CDC10CCD4A24FB220771@mail1.impetus.co.in>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C71F47943940amoghyahooinccom_"
MIME-Version: 1.0

--_000_C71F47943940amoghyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Is it Nov 28th or Dec 28th? Conflict in dates in mail and on site :)

Amogh


On 11/10/09 11:49 AM, "Sanjay Sharma" <sanjay.sharma@impetus.co.in> wrote:

We are planning to hold first Hadoop India user group meet up on 28th Novem=
ber 2009 in Noida.

We would be talking about our experiences with Apache Hadoop/Hbase/Hive/PIG=
/Nutch/etc.

The agenda would be:
- Introductions
- Sharing experiences on Hadoop and related technologies
- Establishing agenda for the next few meetings
- Information exchange: tips, tricks, problems and open discussion
- Possible speaker TBD (invitations open!!)  {we do have something to share=
 on "Hadoop for newbie" & "Hadoop Advanced Tuning"}

My company (Impetus) would be providing the meeting room and we should be a=
ble to accommodate around 40-60 friendly people. Coffee, Tea, and some snac=
ks will be provided.

Please join the linked-in Hadoop India User Group (http://www.linkedin.com/=
groups?home=3D&gid=3D2258445&trk=3Danet_ug_hm) OR Yahoo group (http://tech.=
groups.yahoo.com/group/hadoopind/) and confirm your attendance.

Regards,
Sanjay Sharma

Follow our updates on www.twitter.com/impetuscalling.

* Impetus Technologies is exhibiting it capabilities in Mobile and Wireless=
 in the GSMA Mobile Asia Congress, Hong Kong from November 16-18, 2009. Vis=
it http://www.impetus.com/mlabs/GSMA_events.html for details.

NOTE: This message may contain information that is confidential, proprietar=
y, privileged or otherwise protected by law. The message is intended solely=
 for the named addressee. If received in error, please destroy and notify t=
he sender. Any use of this email is prohibited when received in error. Impe=
tus does not represent, warrant and/or guarantee, that the integrity of thi=
s communication has been maintained nor that the communication is free of e=
rrors, virus, interception or interference.


--_000_C71F47943940amoghyahooinccom_--

From general-return-725-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 10 18:45:29 2009
Return-Path: <general-return-725-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 75055 invoked from network); 10 Nov 2009 18:45:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Nov 2009 18:45:29 -0000
Received: (qmail 1913 invoked by uid 500); 10 Nov 2009 18:45:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1851 invoked by uid 500); 10 Nov 2009 18:45:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1841 invoked by uid 99); 10 Nov 2009 18:45:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Nov 2009 18:45:28 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of tej2@umbc.edu designates 130.85.25.78 as permitted sender)
Received: from [130.85.25.78] (HELO mx3.umbc.edu) (130.85.25.78)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Nov 2009 18:45:17 +0000
Received: from smtp.umbc.edu (localhost [127.0.0.1])
	by umbc.edu (mx3.umbc.edu) with ESMTP id nAAIiurt027594;
	Tue, 10 Nov 2009 13:44:56 -0500 (EST)
Received: from [192.168.1.46] (pool-72-81-183-12.bltmmd.east.verizon.net [72.81.183.12])
	(authenticated bits=0)
	by smtp.umbc.edu (mx3-relay.umbc.edu) with ESMTP id nAAIitZB027577
	(version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO);
	Tue, 10 Nov 2009 13:44:55 -0500 (EST)
Subject: Re: Showing hadoop status
Mime-Version: 1.0 (Apple Message framework v1076)
Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes
From: Tejas Lagvankar <tej2@umbc.edu>
In-Reply-To: <b45b7eee0911100158t414633f9pcd3fffc66a221bae@mail.gmail.com>
Date: Tue, 10 Nov 2009 13:44:55 -0500
Cc: Mark N <nipen.mark@gmail.com>
Content-Transfer-Encoding: 7bit
Message-Id: <2951B209-6D73-42FD-B84F-CCFBFCF1A6AE@umbc.edu>
References: <b45b7eee0911100158t414633f9pcd3fffc66a221bae@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1076)
X-Milter-Key: 1258008296:933ceef8950cacee6c467bee0057a757
X-ClamAV: OK
X-Virus-Checked: Checked by ClamAV on apache.org

Why would you want to read the status and write some other application  
when Hadoop provides cluster monitoring webapps at ports 50030, 50060  
and 50070 ?

On Nov 10, 2009, at 4:58 AM, Mark N wrote:

> Currently  I was trying to read the log files generated by hadoop so  
> that I
> could show the status of map/reduce jobs in user interfaces
> (consider that UI is a separate application )
>
> I was looking at JobTracker APIs and was trying use following APIs
> 1. jobtracker.getAllJobs()  which returns array of jobstatus objects
>
> 2. Then for each jobStatus we can call
>    mapProgress() , isJobCompleted()  methods
>
> The problem is how do I link existing jobTracker ( which is already  
> running
> )  with my  code ? i am trying to built something like  " hadoop UI ."
>
> should I use jobClient  APIs ? Also is this a proper approach ?
>
> thanks in advance . N mark.

Tejas Lagvankar
meettejas@umbc.edu
www.umbc.edu/~tej2




From general-return-726-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 10 22:22:33 2009
Return-Path: <general-return-726-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68468 invoked from network); 10 Nov 2009 22:22:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Nov 2009 22:22:32 -0000
Received: (qmail 65632 invoked by uid 500); 10 Nov 2009 22:22:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65554 invoked by uid 500); 10 Nov 2009 22:22:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65544 invoked by uid 99); 10 Nov 2009 22:22:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Nov 2009 22:22:31 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of hkumar.arora@gmail.com designates 209.85.223.188 as permitted sender)
Received: from [209.85.223.188] (HELO mail-iw0-f188.google.com) (209.85.223.188)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Nov 2009 22:22:20 +0000
Received: by iwn26 with SMTP id 26so464421iwn.5
        for <general@hadoop.apache.org>; Tue, 10 Nov 2009 14:21:59 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=ssMADpe0XDbl2Si7SRDWw5aVDCkkyLCIm+rwWs1O+3c=;
        b=IuySMDjEZhSe0oRTLfwgzPSkTT1l4L9pMAhCI6pHclvoIqJ3/M3xCH9wYkWlh80NUM
         iD2BKwevpoaQPKwCuzcHzJVzLpXLHR1mFy/64rSheEYcaL249jTDhQXAvUg4tJezx/OI
         dOo5VYQBTW9bGBxkM6+VMCmv6cOA2rTfrY1cI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=AQp8mBqsYwT+SpubgD5gLLEXoKX/+mgFmyZAcLjX/VmSp9/TAIimyfg5BkN5Vk+oww
         tolIWBBBj1KoM9XWByCNOb3DwK4IJG0UEd9bnCWbhgFhwLnNm9VqNe3lU6Um084ai2oS
         gHsTuzjESxNv1pIfJWIsy9Mnj8s2kUu9CMRJc=
MIME-Version: 1.0
Received: by 10.231.170.201 with SMTP id e9mr1140982ibz.15.1257891719265; Tue, 
	10 Nov 2009 14:21:59 -0800 (PST)
In-Reply-To: <46E2672895E8744A97D7577A6110858B4FD60B2D55@SP1-EX07VS01.ds.corp.yahoo.com>
References: <46E2672895E8744A97D7577A6110858B4E1AC69A91@SP1-EX07VS01.ds.corp.yahoo.com>
	 <46E2672895E8744A97D7577A6110858B4FD39F6EDF@SP1-EX07VS01.ds.corp.yahoo.com>
	 <46E2672895E8744A97D7577A6110858B4FD60B2D55@SP1-EX07VS01.ds.corp.yahoo.com>
Date: Wed, 11 Nov 2009 07:21:59 +0900
Message-ID: <e497591c0911101421u4971c874ha7c33b6fb48d4ab5@mail.gmail.com>
Subject: Re: Hadoop User Group (Bay Area) - next Wednesday (Nov 18th) at 
	Yahoo!
From: Harshit Kumar <hkumar.arora@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504501810217bf2804780bbdef
X-Virus-Checked: Checked by ClamAV on apache.org

--00504501810217bf2804780bbdef
Content-Type: text/plain; charset=UTF-8

Hi

Any consideration of making this event available for global members of
hadoop community, say for instance, streaming it live.

Or at least record the event and upload on youtube.

Regards
H. Kumar
Phone(Mobile): +82-10-2892-9663
Phone(Office): +82-31-
skype: harshit900
Blog: http://harshitkumar.wordpress.com
Website: http:/kumarharmuscat.tripod.com


2009/11/11 Dekel Tankel <dekel@yahoo-inc.com>

> Hi all,
>
> We are one week away from the next Bay Area Hadoop User Group  - Yahoo!
> Sunnyvale Campus, next Wednesday (Nov 18th) at 6PM
>
> We have an exciting evening planed:
>
> *Katta, Solr, Lucene and Hadoop - Searching at scale, Jason Rutherglen and
> Jason Venner
>
> *Walking through the New File system API, Sanjay Radia, Yahoo!
>
> *Keep your data in Jute but still use it in python, Paul Tarjan, Yahoo!
>
>
> Please RSVP here:
> http://www.meetup.com/hadoop/calendar/11724002/
>
>
> Please note that this is the last HUG for 2009, as we will not have a
> meeting on December (due to the holidays).
> We will open 2010 with a HUG on Jan 20th.
>
> Looking forward to seeing you next week!
>
> Dekel
>
>

--00504501810217bf2804780bbdef--

From general-return-727-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 11 00:50:35 2009
Return-Path: <general-return-727-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34479 invoked from network); 11 Nov 2009 00:50:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Nov 2009 00:50:33 -0000
Received: (qmail 97499 invoked by uid 500); 11 Nov 2009 00:50:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97430 invoked by uid 500); 11 Nov 2009 00:50:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97387 invoked by uid 99); 11 Nov 2009 00:50:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Nov 2009 00:50:32 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of abhishek.vit@gmail.com designates 209.85.220.215 as permitted sender)
Received: from [209.85.220.215] (HELO mail-fx0-f215.google.com) (209.85.220.215)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Nov 2009 00:50:29 +0000
Received: by fxm7 with SMTP id 7so622258fxm.29
        for <general@hadoop.apache.org>; Tue, 10 Nov 2009 16:50:07 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=zVe8DRXxMwBKiz+Oald9vg7fGnoMqpZLvLzeOffK2i8=;
        b=PWcOewIAE8ItsM7jqHYQ2fjBQ98YmKZZXPNsC5GkMHg5tYNK8v+TMoyGvx9PBz5OR/
         B6Wul0Hj/lQO6nfYvnm9VMeC2tCFIbbXprCjsSoxzrsxTXGjI6XTys6G/KNlfj4j7l9x
         Kueb8wOEi/ef2eKk+J0Brc+lRvnnqIwaMz/kE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=tGNjqA3XSGl8VXOOROXJ7zUWPNzru4WGaxC4HmR8xMBbJWXdk2P1kqGefN9pxPfeey
         bM9dy7ayVlWqbQMf5ri31TdNvO6X1qozZhkXLAA1rpRhgVkA10zzCrUahVkdAJ7DvQUp
         W5cm0LF2zT15PWzyRx4lGFI0VeaJzvkWRwvpQ=
MIME-Version: 1.0
Received: by 10.204.9.11 with SMTP id j11mr801880bkj.115.1257900607482; Tue, 
	10 Nov 2009 16:50:07 -0800 (PST)
In-Reply-To: <e497591c0911101421u4971c874ha7c33b6fb48d4ab5@mail.gmail.com>
References: <46E2672895E8744A97D7577A6110858B4E1AC69A91@SP1-EX07VS01.ds.corp.yahoo.com>
	 <46E2672895E8744A97D7577A6110858B4FD39F6EDF@SP1-EX07VS01.ds.corp.yahoo.com>
	 <46E2672895E8744A97D7577A6110858B4FD60B2D55@SP1-EX07VS01.ds.corp.yahoo.com>
	 <e497591c0911101421u4971c874ha7c33b6fb48d4ab5@mail.gmail.com>
Date: Tue, 10 Nov 2009 19:50:07 -0500
Message-ID: <be9b52410911101650q1342b1bcoa63b575f77ff4e8@mail.gmail.com>
Subject: Re: Hadoop User Group (Bay Area) - next Wednesday (Nov 18th) at 
	Yahoo!
From: Abhishek Pratap <abhishek.vit@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00151758f422df2be904780dce31

--00151758f422df2be904780dce31
Content-Type: text/plain; charset=ISO-8859-1

Hi Harshit

I second your idea. Since it is development under rapid progress, streaming
live/recording will help reach out to lot of interested folks. I would be
one of them.

Cheers,
-Abhi

On Tue, Nov 10, 2009 at 5:21 PM, Harshit Kumar <hkumar.arora@gmail.com>wrote:

> Hi
>
> Any consideration of making this event available for global members of
> hadoop community, say for instance, streaming it live.
>
> Or at least record the event and upload on youtube.
>
> Regards
> H. Kumar
> Phone(Mobile): +82-10-2892-9663
> Phone(Office): +82-31-
> skype: harshit900
> Blog: http://harshitkumar.wordpress.com
> Website: http:/kumarharmuscat.tripod.com
>
>
> 2009/11/11 Dekel Tankel <dekel@yahoo-inc.com>
>
> > Hi all,
> >
> > We are one week away from the next Bay Area Hadoop User Group  - Yahoo!
> > Sunnyvale Campus, next Wednesday (Nov 18th) at 6PM
> >
> > We have an exciting evening planed:
> >
> > *Katta, Solr, Lucene and Hadoop - Searching at scale, Jason Rutherglen
> and
> > Jason Venner
> >
> > *Walking through the New File system API, Sanjay Radia, Yahoo!
> >
> > *Keep your data in Jute but still use it in python, Paul Tarjan, Yahoo!
> >
> >
> > Please RSVP here:
> > http://www.meetup.com/hadoop/calendar/11724002/
> >
> >
> > Please note that this is the last HUG for 2009, as we will not have a
> > meeting on December (due to the holidays).
> > We will open 2010 with a HUG on Jan 20th.
> >
> > Looking forward to seeing you next week!
> >
> > Dekel
> >
> >
>

--00151758f422df2be904780dce31--

From general-return-728-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 11 01:20:33 2009
Return-Path: <general-return-728-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47927 invoked from network); 11 Nov 2009 01:20:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Nov 2009 01:20:33 -0000
Received: (qmail 26147 invoked by uid 500); 11 Nov 2009 01:20:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26053 invoked by uid 500); 11 Nov 2009 01:20:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26043 invoked by uid 99); 11 Nov 2009 01:20:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Nov 2009 01:20:32 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 209.85.222.195 as permitted sender)
Received: from [209.85.222.195] (HELO mail-pz0-f195.google.com) (209.85.222.195)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Nov 2009 01:20:30 +0000
Received: by pzk33 with SMTP id 33so423331pzk.2
        for <general@hadoop.apache.org>; Tue, 10 Nov 2009 17:20:09 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:references:message-id:from:to
         :in-reply-to:content-type:content-transfer-encoding:x-mailer
         :mime-version:subject:date:cc;
        bh=TZwPQ+3NRvRpLNFd9Jp13RjF2CxaDeJDTsOwtpjgkvs=;
        b=rhUNSmxWSEZoE6ifJZIaw8ri93uo5obnsOgRwhz71EOF1TC6eA6jDFBY4IEud7VZL2
         r2KerhtTO8ZkdMrnsWS8j0ZOfhCjHGaXM7oLfKgADteDd4YWbjasXopYefSRyw3Mzmo7
         5ERVsfqyLWgFIDrXpdDKLbpK+igiopcuGCDKE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=references:message-id:from:to:in-reply-to:content-type
         :content-transfer-encoding:x-mailer:mime-version:subject:date:cc;
        b=cNn+tCf+G5OSnNMmQ/MtmFoPoe9mH8+FCIteMk0PnKQx3BYCPcs1jbD/ycizvpy7Xs
         gGoBdoQ28PjE/cvjsqutC/DDPc4zZot71E0SWIV3rjlaCdKC2kYCQA0GXvgyqOz3PqlR
         /zZINdb/cvYwu3XFeRlB6Un+ZAhTF3/J82BlQ=
Received: by 10.115.24.10 with SMTP id b10mr1667864waj.127.1257902409517;
        Tue, 10 Nov 2009 17:20:09 -0800 (PST)
Received: from ?10.252.20.158? ([166.205.131.44])
        by mx.google.com with ESMTPS id 21sm719482pxi.0.2009.11.10.17.20.07
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Tue, 10 Nov 2009 17:20:08 -0800 (PST)
References: <752a037e0911100340u619ae830i807b718009e3efbc@mail.gmail.com>
Message-Id: <225FE2E8-7D96-4788-BBF5-197AF5816A14@gmail.com>
From: Stack <saint.ack@gmail.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <752a037e0911100340u619ae830i807b718009e3efbc@mail.gmail.com>
Content-Type: text/plain;
	charset=utf-8;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: quoted-printable
X-Mailer: iPhone Mail (7D11)
Mime-Version: 1.0 (iPhone Mail 7D11)
Subject: Re: version 0.20.2 release?
Date: Tue, 10 Nov 2009 17:20:13 -0800
Cc: "general@hadoop.apache.org" <general@hadoop.apache.org>

An hbase release candidate for 0.20.2 was posted today.  FYI hbase =20
questions will get a faster reponse asked on hbase mailing list.

Yours
Stack



On Nov 10, 2009, at 3:40 AM, Luis Carlos Junges =20
<luis.junges@gmail.com> wrote:

> Hi,
>
> i would like to know when would be the release of the version 0.20.2?
>
> i found the bug HBASE-1327 on it and i can start my hbase cluster =20
> because of
> it.
>
> http://issues.apache.org/jira/browse/HBASE-1327
>
> is anyway to get a pre-release without this bug?
>
> --=20
> "A realidade de cada lugar e de cada =C3=A9poca =C3=A9 uma =
alucina=C3=A7=C3=A3o =20
> coletiva."
>
>
> Bloom, Howardh

From general-return-729-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 11 04:03:09 2009
Return-Path: <general-return-729-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4181 invoked from network); 11 Nov 2009 04:03:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Nov 2009 04:03:09 -0000
Received: (qmail 71357 invoked by uid 500); 11 Nov 2009 04:03:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71085 invoked by uid 500); 11 Nov 2009 04:03:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71075 invoked by uid 99); 11 Nov 2009 04:03:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Nov 2009 04:03:06 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of nipen.mark@gmail.com designates 209.85.160.50 as permitted sender)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Nov 2009 04:02:55 +0000
Received: by pwi6 with SMTP id 6so495289pwi.29
        for <general@hadoop.apache.org>; Tue, 10 Nov 2009 20:02:34 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:cc:content-type;
        bh=IpsQCX4MXxcGDCivWm94grp8e86HLeTo4S0qU85HNqM=;
        b=MMNvfR5juIhWInKlhpHHNZxJ5akBzpOMywBqgGCJDiAqnw4IuCaxR+q64wXoC31Rl0
         G8GBa/HjGP+QxbHC2Wcbs8xw1Oh1lAWzkhK6cywWcEPJUefHEv9nQGndiBwD5TEbWbNd
         SUyXovAOMkgUOjKa4MNOujs5NQ2lzewpHJ/mk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=EAgy4tE5nhaqyKzXl9HnLvB7JJ64CwI5L0AaL+LMy5ySEzmgDx828LJ4ap0ldQkFIg
         xBBOas5nhMCzxXFtjMmSEATLhTxM18LXWONWVnFtFpG86aXcZnW/LKNdGxmgVRUCX0i4
         eS/tP15Wy7DcOHzMPrMgyGyIiZD1NuEaWJSLw=
MIME-Version: 1.0
Received: by 10.142.195.4 with SMTP id s4mr107950wff.264.1257912154340; Tue, 
	10 Nov 2009 20:02:34 -0800 (PST)
In-Reply-To: <2951B209-6D73-42FD-B84F-CCFBFCF1A6AE@umbc.edu>
References: <b45b7eee0911100158t414633f9pcd3fffc66a221bae@mail.gmail.com>
	 <2951B209-6D73-42FD-B84F-CCFBFCF1A6AE@umbc.edu>
Date: Wed, 11 Nov 2009 09:47:34 +0545
Message-ID: <b45b7eee0911102002w2ef05567w2060dfc662e06192@mail.gmail.com>
Subject: Re: Showing hadoop status
From: Mark N <nipen.mark@gmail.com>
To: Tejas Lagvankar <tej2@umbc.edu>
Cc: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd1449c1e3d1d0478107f2b
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd1449c1e3d1d0478107f2b
Content-Type: text/plain; charset=ISO-8859-1

Am trying to invoke hadoop process from seperate user interface  where user
can see some kind of status.  has anyone tried on this approach?

thanks


On Wed, Nov 11, 2009 at 12:29 AM, Tejas Lagvankar <tej2@umbc.edu> wrote:

> Why would you want to read the status and write some other application when
> Hadoop provides cluster monitoring webapps at ports 50030, 50060 and 50070 ?
>
>
> On Nov 10, 2009, at 4:58 AM, Mark N wrote:
>
>  Currently  I was trying to read the log files generated by hadoop so that
>> I
>> could show the status of map/reduce jobs in user interfaces
>> (consider that UI is a separate application )
>>
>> I was looking at JobTracker APIs and was trying use following APIs
>> 1. jobtracker.getAllJobs()  which returns array of jobstatus objects
>>
>> 2. Then for each jobStatus we can call
>>   mapProgress() , isJobCompleted()  methods
>>
>> The problem is how do I link existing jobTracker ( which is already
>> running
>> )  with my  code ? i am trying to built something like  " hadoop UI ."
>>
>> should I use jobClient  APIs ? Also is this a proper approach ?
>>
>> thanks in advance . N mark.
>>
>
> Tejas Lagvankar
> meettejas@umbc.edu
> www.umbc.edu/~tej2 <http://www.umbc.edu/%7Etej2>
>
>
>
>

--000e0cd1449c1e3d1d0478107f2b--

From general-return-730-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 11 10:43:26 2009
Return-Path: <general-return-730-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78359 invoked from network); 11 Nov 2009 10:43:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Nov 2009 10:43:25 -0000
Received: (qmail 13869 invoked by uid 500); 11 Nov 2009 10:43:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13790 invoked by uid 500); 11 Nov 2009 10:43:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13778 invoked by uid 99); 11 Nov 2009 10:43:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Nov 2009 10:43:24 +0000
X-ASF-Spam-Status: No, hits=-1.5 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [145.100.8.32] (HELO planck.ka.sara.nl) (145.100.8.32)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Nov 2009 10:43:20 +0000
Received: from planck.ka.sara.nl ([145.100.8.32]) by planck.ka.sara.nl
 ([145.100.8.32]) with mapi; Wed, 11 Nov 2009 11:42:57 +0100
From: Evert Lammerts <Evert.Lammerts@sara.nl>
To: "'users@lists.opennebula.org'" <users@lists.opennebula.org>,
	"'general@hadoop.apache.org'" <general@hadoop.apache.org>
Date: Wed, 11 Nov 2009 11:42:56 +0100
Subject: Hadoop on Opennebula
Thread-Topic: Hadoop on Opennebula
Thread-Index: Acpiu7hZJPp+aKVJSbKkqn9GFSreZQ==
Message-ID: <ADF94D8555C7A246B86A633685E0178AB84B40DB72@planck.ka.sara.nl>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/signed; protocol="application/x-pkcs7-signature";
	micalg=SHA1; boundary="----=_NextPart_000_0000_01CA62C4.1A657810"
MIME-Version: 1.0

------=_NextPart_000_0000_01CA62C4.1A657810
Content-Type: multipart/alternative;
	boundary="----=_NextPart_001_0001_01CA62C4.1A657810"


------=_NextPart_001_0001_01CA62C4.1A657810
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit

Hi list,

 

We have a cluster with 44 cores on which we are evaluating OpenNebula. All
nodes run Ubuntu server 64bit. I'm trying to get Hadoop 0.20.1 to work, so
I'm following the cluster setup steps on hadoop.apache.org.

 

As a start I created three VM's running Ubuntu 9.10 32bit desktop edition.
After installing Sun's 1.6 JRE I put Hadoop into my homedir. I configured
the three installations of Hadoop as follows:

 

== conf/hadoop-env.sh ==

Set JAVA_HOME to the appropriate directory

 

== conf/core-site.xml ==

Set fs.default.name to the ip address of the designated namenode, on port
9000:

<property>

   <name>fs.default.name</name>

   <value>hdfs://XXX.XXX.X.XXX:9000/</value>

 </property>

 

== conf/hdfs-site.xml ==

Set dfs.name.dir to a directory in my homedir:

<property>

  <name>dfs.name.dir</name>

  <value>/home/cloud/var/log/hadoop/</value>

</property>

 

== conf/mapred-site.xml ==

Set mapred.job.tracker to the ip address of the designated jobtracker, on
port 9001:

<property>

  <name>mapred.job.tracker</name>

  <value>XXX.XXX.X.XXX:9001</value>

<property>

 

Apart from Hadoop configuration I manually set the hostname for the
namenode, jobtracker and slave (datanode & tasktracker) to respectively
hadoop-namenode, hadoop-jobtracker and hadoop-slave01.

 

I'm able to start the hdfs with bin/start-dfs.sh, and mapreduce with
bin/start-mapred.sh without any exceptions. However, when I now try to copy
files onto hdfs, I'm getting the following exception:

 

$ bin/hadoop fs -put conf input

09/11/11 05:24:47 WARN hdfs.DFSClient: DataStreamer Exception:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/user/cloud/input/capacity-scheduler.xml could only be replicated to 0
nodes, instead of 1

                at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNam
esystem.java:1267)

                at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)

                at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown
Source)

                at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

                at java.lang.reflect.Method.invoke(Method.java:597)

                at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)

                at
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)

                at
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)

                at java.security.AccessController.doPrivileged(Native
Method)

                at javax.security.auth.Subject.doAs(Subject.java:396)

                at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)

 

                at org.apache.hadoop.ipc.Client.call(Client.java:739)

                at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)

                at $Proxy0.addBlock(Unknown Source)

                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)

                at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)

                at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

                at java.lang.reflect.Method.invoke(Method.java:597)

                at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocati
onHandler.java:82)

                at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHand
ler.java:59)

                at $Proxy0.addBlock(Unknown Source)

                at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSCli
ent.java:2904)

                at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSCl
ient.java:2786)

                at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:
2076)

                at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.
java:2262)

 

09/11/11 05:24:47 WARN hdfs.DFSClient: Error Recovery for block null bad
datanode[0] nodes == null

09/11/11 05:24:47 WARN hdfs.DFSClient: Could not get block locations. Source
file "/user/cloud/input/capacity-scheduler.xml" - Aborting...

put: java.io.IOException: File /user/cloud/input/capacity-scheduler.xml
could only be replicated to 0 nodes, instead of 1

 

Can anybody shed light on this? I'm guessing it's a configuration issue, so
that's the direction I'm looking at.

 

Another question I have is more generally about getting Hadoop to work on a
cloud. The issue I foresee is about the ip addresses of the masters and
slaves. How do I dynamically configure the hadoop instances during start-up
of the images to end up with a namenode, a jobtracker and a number of
slaves? I'll need the ip addresses of all machines and all machines need a
unique hostname. Does anybody have any experience with this?

 

Thanks in advance!

 

Evert Lammerts

Adviseur

SARA Computing & Network Services 

High Performance Computing & Visualization

eScience Support Group

 

Phone: +31 20 888 4101

Email: evert.lammerts@sara.nl

 


------=_NextPart_001_0001_01CA62C4.1A657810
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" =
xmlns=3D"http://www.w3.org/TR/REC-html40">

<head>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<meta name=3DGenerator content=3D"Microsoft Word 12 (filtered medium)">
<style>
<!--
 /* Font Definitions */
 @font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
	{font-family:"Malgun Gothic";
	panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
	{font-family:"\@Malgun Gothic";}
 /* Style Definitions */
 p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0cm;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:purple;
	text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
	{mso-style-priority:34;
	margin-top:0cm;
	margin-right:0cm;
	margin-bottom:0cm;
	margin-left:36.0pt;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri","sans-serif";}
span.EmailStyle17
	{mso-style-type:personal-compose;
	font-family:"Calibri","sans-serif";
	color:windowtext;}
.MsoChpDefault
	{mso-style-type:export-only;}
@page Section1
	{size:612.0pt 792.0pt;
	margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.Section1
	{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
 <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
 <o:shapelayout v:ext=3D"edit">
  <o:idmap v:ext=3D"edit" data=3D"1" />
 </o:shapelayout></xml><![endif]-->
</head>

<body lang=3DEN-US link=3Dblue vlink=3Dpurple>

<div class=3DSection1>

<p class=3DMsoNormal>Hi list,<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>We have a cluster with 44 cores on which we are =
evaluating OpenNebula.
All nodes run Ubuntu server 64bit. I&#8217;m trying to get Hadoop 0.20.1 =
to
work, so I&#8217;m following the cluster setup steps on =
hadoop.apache.org.<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>As a start I created three VM&#8217;s running =
Ubuntu 9.10 32bit
desktop edition. After installing Sun&#8217;s 1.6 JRE I put Hadoop into =
my
homedir. I configured the three installations of Hadoop as =
follows:<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>=3D=3D conf/hadoop-env.sh =3D=3D<o:p></o:p></p>

<p class=3DMsoNormal>Set JAVA_HOME to the appropriate =
directory<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>=3D=3D conf/core-site.xml =3D=3D<o:p></o:p></p>

<p class=3DMsoNormal>Set fs.default.name to the ip address of the =
designated
namenode, on port 9000:<o:p></o:p></p>

<p class=3DMsoNormal>&lt;property&gt;<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp; =
&lt;name&gt;fs.default.name&lt;/name&gt;<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp; =
&lt;value&gt;hdfs://XXX.XXX.X.XXX:9000/&lt;/value&gt;<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp;&lt;/property&gt;<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>=3D=3D conf/hdfs-site.xml =3D=3D<o:p></o:p></p>

<p class=3DMsoNormal>Set dfs.name.dir to a directory in my =
homedir:<o:p></o:p></p>

<p class=3DMsoNormal>&lt;property&gt;<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp; =
&lt;name&gt;dfs.name.dir&lt;/name&gt;<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp; =
&lt;value&gt;/home/cloud/var/log/hadoop/&lt;/value&gt;<o:p></o:p></p>

<p class=3DMsoNormal>&lt;/property&gt;<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>=3D=3D conf/mapred-site.xml =3D=3D<o:p></o:p></p>

<p class=3DMsoNormal>Set mapred.job.tracker to the ip address of the =
designated
jobtracker, on port 9001:<o:p></o:p></p>

<p class=3DMsoNormal>&lt;property&gt;<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp; =
&lt;name&gt;mapred.job.tracker&lt;/name&gt;<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp; =
&lt;value&gt;XXX.XXX.X.XXX:9001&lt;/value&gt;<o:p></o:p></p>

<p class=3DMsoNormal>&lt;property&gt;<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>Apart from Hadoop configuration I manually set the =
hostname for
the namenode, jobtracker and slave (datanode &amp; tasktracker) to =
respectively
hadoop-namenode, hadoop-jobtracker and hadoop-slave01.<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>I&#8217;m able to start the hdfs with =
bin/start-dfs.sh, and
mapreduce with bin/start-mapred.sh without any exceptions. However, when =
I now
try to copy files onto hdfs, I&#8217;m getting the following =
exception:<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>$ bin/hadoop fs -put conf input<o:p></o:p></p>

<p class=3DMsoNormal>09/11/11 05:24:47 WARN hdfs.DFSClient: DataStreamer
Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: =
File
/user/cloud/input/capacity-scheduler.xml could only be replicated to 0 =
nodes,
instead of 1<o:p></o:p></p>

<p =
class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FS=
Namesystem.java:1267)<o:p></o:p></p>

<p =
class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:42=
2)<o:p></o:p></p>

<p =
class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
sun.reflect.GeneratedMethodAccessor20.invoke(Unknown =
Source)<o:p></o:p></p>

<p =
class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI=
mpl.java:25)<o:p></o:p></p>

<p =
class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at =
java.lang.reflect.Method.invoke(Method.java:597)<o:p></o:p></p>

<p =
class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)<o:p></o:p></p>

<p =
class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)<o:p></o:p></p=
>

<p =
class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)<o:p></o:p></p=
>

<p =
class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at =
java.security.AccessController.doPrivileged(Native
Method)<o:p></o:p></p>

<p =
class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
javax.security.auth.Subject.doAs(Subject.java:396)<o:p></o:p></p>

<p =
class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p =
class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
org.apache.hadoop.ipc.Client.call(Client.java:739)<o:p></o:p></p>

<p =
class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)<o:p></o:p></p>

<p =
class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at $Proxy0.addBlock(Unknown =
Source)<o:p></o:p></p>

<p =
class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native =
Method)<o:p></o:p></p>

<p =
class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java=
:39)<o:p></o:p></p>

<p =
class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI=
mpl.java:25)<o:p></o:p></p>

<p =
class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
java.lang.reflect.Method.invoke(Method.java:597)<o:p></o:p></p>

<p =
class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvoc=
ationHandler.java:82)<o:p></o:p></p>

<p =
class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationH=
andler.java:59)<o:p></o:p></p>

<p =
class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at $Proxy0.addBlock(Unknown =
Source)<o:p></o:p></p>

<p =
class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFS=
Client.java:2904)<o:p></o:p></p>

<p =
class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DF=
SClient.java:2786)<o:p></o:p></p>

<p =
class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at =
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.ja=
va:2076)<o:p></o:p></p>

<p =
class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClie=
nt.java:2262)<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>09/11/11 05:24:47 WARN hdfs.DFSClient: Error =
Recovery for
block null bad datanode[0] nodes =3D=3D null<o:p></o:p></p>

<p class=3DMsoNormal>09/11/11 05:24:47 WARN hdfs.DFSClient: Could not =
get block
locations. Source file =
&quot;/user/cloud/input/capacity-scheduler.xml&quot; -
Aborting...<o:p></o:p></p>

<p class=3DMsoNormal>put: java.io.IOException: File
/user/cloud/input/capacity-scheduler.xml could only be replicated to 0 =
nodes,
instead of 1<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>Can anybody shed light on this? I&#8217;m guessing =
it&#8217;s
a configuration issue, so that&#8217;s the direction I&#8217;m looking =
at.<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>Another question I have is more generally about =
getting Hadoop
to work on a cloud. The issue I foresee is about the ip addresses of the
masters and slaves. How do I dynamically configure the hadoop instances =
during
start-up of the images to end up with a namenode, a jobtracker and a =
number of
slaves? I&#8217;ll need the ip addresses of all machines and all =
machines need
a unique hostname&#8230; Does anybody have any experience with =
this?<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>Thanks in advance!<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal><span =
style=3D'font-size:10.0pt;font-family:"Courier New"'>Evert
Lammerts<o:p></o:p></span></p>

<p class=3DMsoNormal><span =
style=3D'font-size:10.0pt;font-family:"Courier =
New"'>Adviseur<o:p></o:p></span></p>

<p class=3DMsoNormal><span =
style=3D'font-size:10.0pt;font-family:"Courier New"'>SARA
Computing &amp; Network Services <o:p></o:p></span></p>

<p class=3DMsoNormal><span =
style=3D'font-size:10.0pt;font-family:"Courier New"'>High
Performance Computing &amp; Visualization<o:p></o:p></span></p>

<p class=3DMsoNormal><span =
style=3D'font-size:10.0pt;font-family:"Courier New"'>eScience
Support Group<o:p></o:p></span></p>

<p class=3DMsoNormal><span =
style=3D'font-size:10.0pt;font-family:"Courier =
New"'><o:p>&nbsp;</o:p></span></p>

<p class=3DMsoNormal><span =
style=3D'font-size:10.0pt;font-family:"Courier New"'>Phone:
+31 20 888 4101<o:p></o:p></span></p>

<p class=3DMsoNormal><span =
style=3D'font-size:10.0pt;font-family:"Courier New"'>Email:
evert.lammerts@sara.nl<o:p></o:p></span></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

</div>

</body>

</html>

------=_NextPart_001_0001_01CA62C4.1A657810--

------=_NextPart_000_0000_01CA62C4.1A657810
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIII9TCCAn0w
ggHmoAMCAQICEFiofjswHrxbxE7M/zOuXB8wDQYJKoZIhvcNAQEFBQAwYjELMAkGA1UEBhMCWkEx
JTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQ
ZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA5MDEyOTA5NTgyOFoXDTEwMDEyOTA5NTgy
OFowYzERMA8GA1UEBBMITGFtbWVydHMxDjAMBgNVBCoTBUV2ZXJ0MRcwFQYDVQQDEw5FdmVydCBM
YW1tZXJ0czElMCMGCSqGSIb3DQEJARYWZXZlcnQubGFtbWVydHNAc2FyYS5ubDCBnzANBgkqhkiG
9w0BAQEFAAOBjQAwgYkCgYEA6W7oHcE2VEPAb895157nhumm7Wnd8R1UVcsoT7BQpUSvWLPuyo70
KSI3JHAqprbWIQTDtHJ5eR+f4CktFclrf0vygCGtBIPBtsW9UO+NRxEYxtkBLEu2p/DXlnCQ04EB
auypiyGBL7an5HGPMU8B6F/UABjJbJ698G4ufdiuWOMCAwEAAaMzMDEwIQYDVR0RBBowGIEWZXZl
cnQubGFtbWVydHNAc2FyYS5ubDAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBBQUAA4GBAHzlG2fG
eJqSMAjBvpTN1y8tlbvl0vZ5Cxu0jYYc92aFVdhkina+JjD53dw204bdl+U9zbGHSLqlVoh92p+E
53yBRlEh9uJ/K6HcdBe/s2KEjZHfCGi756sP5M9z6txyxYOBtSv7EHKOL+HJMCCHbrVAICjOpxJY
QWwb5waKfaeUMIIDLTCCApagAwIBAgIBADANBgkqhkiG9w0BAQQFADCB0TELMAkGA1UEBhMCWkEx
FTATBgNVBAgTDFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJQ2FwZSBUb3duMRowGAYDVQQKExFUaGF3
dGUgQ29uc3VsdGluZzEoMCYGA1UECxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNlcyBEaXZpc2lvbjEk
MCIGA1UEAxMbVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIENBMSswKQYJKoZIhvcNAQkBFhxwZXJz
b25hbC1mcmVlbWFpbEB0aGF3dGUuY29tMB4XDTk2MDEwMTAwMDAwMFoXDTIwMTIzMTIzNTk1OVow
gdExCzAJBgNVBAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJuIENhcGUxEjAQBgNVBAcTCUNhcGUgVG93
bjEaMBgGA1UEChMRVGhhd3RlIENvbnN1bHRpbmcxKDAmBgNVBAsTH0NlcnRpZmljYXRpb24gU2Vy
dmljZXMgRGl2aXNpb24xJDAiBgNVBAMTG1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBDQTErMCkG
CSqGSIb3DQEJARYccGVyc29uYWwtZnJlZW1haWxAdGhhd3RlLmNvbTCBnzANBgkqhkiG9w0BAQEF
AAOBjQAwgYkCgYEA1GnX1LCUZFtx6UfYDFG26nKRsIRefS0Nj3sS34UldSh0OkIsYyeflXtL734Z
hx2G6qPduc6WZBrCFG5ErHzmj+hND3EfQDimAKOHePb5lIZererAXnbr2RSjXW56fAylS1V/Bhkp
f56aJtVquzgkCGqYx7Hao5iR/Xnb5VrEHLkCAwEAAaMTMBEwDwYDVR0TAQH/BAUwAwEB/zANBgkq
hkiG9w0BAQQFAAOBgQDH7JJ+Tvj1lqVnYiqk8E0RYNBvjWBYYawmu1I1XAjPMPuoSpaKH2JCI4wX
D/S6ZJwXrEcp352YXtJsYHFcoqzceePnbgBHH7UNKOgCneSa/RP0ptl8sfjcXyMmCZGAc9AUG95D
qYMl8uacLxXK/qarigd1iwzdUYRr5PjRzneigTCCAz8wggKooAMCAQICAQ0wDQYJKoZIhvcNAQEF
BQAwgdExCzAJBgNVBAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJuIENhcGUxEjAQBgNVBAcTCUNhcGUg
VG93bjEaMBgGA1UEChMRVGhhd3RlIENvbnN1bHRpbmcxKDAmBgNVBAsTH0NlcnRpZmljYXRpb24g
U2VydmljZXMgRGl2aXNpb24xJDAiBgNVBAMTG1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBDQTEr
MCkGCSqGSIb3DQEJARYccGVyc29uYWwtZnJlZW1haWxAdGhhd3RlLmNvbTAeFw0wMzA3MTcwMDAw
MDBaFw0xMzA3MTYyMzU5NTlaMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3Vs
dGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWlu
ZyBDQTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAxKY8VXNV+065yplaHmjAdQRwnd/p/6Me
7L3N9VvyGna9fww6YfK/Uc4B1OVQCjDXAmNaLIkVcI7dyfArhVqqP3FWy688Cwfn8R+RNiQqE88r
1fOCdz0Dviv+uxg+B79AgAJk16emu59l0cUqVIUPSAR/p7bRPGEEQB5kGXJgt/sCAwEAAaOBlDCB
kTASBgNVHRMBAf8ECDAGAQH/AgEAMEMGA1UdHwQ8MDowOKA2oDSGMmh0dHA6Ly9jcmwudGhhd3Rl
LmNvbS9UaGF3dGVQZXJzb25hbEZyZWVtYWlsQ0EuY3JsMAsGA1UdDwQEAwIBBjApBgNVHREEIjAg
pB4wHDEaMBgGA1UEAxMRUHJpdmF0ZUxhYmVsMi0xMzgwDQYJKoZIhvcNAQEFBQADgYEASIzRUIPq
Cy7MDaNmrGcPf6+svsIXoUOWlJ1/TCG4+DYfqi2fNi/A9BxQIJNwPP2t4WFiw9k6GX6EsZkbAMUa
C4J0niVQlGLH2ydxVyWN3amcOY6MIE9lX5Xa9/eH1sYITq726jTlEBpbNU1341YheILcIRk13iSx
0x1G/11fZU8xggL4MIIC9AIBATB2MGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29u
c3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNz
dWluZyBDQQIQWKh+OzAevFvETsz/M65cHzAJBgUrDgMCGgUAoIIB2DAYBgkqhkiG9w0BCQMxCwYJ
KoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0wOTExMTExMDQyNTJaMCMGCSqGSIb3DQEJBDEWBBQU
iu7Jm2MP3BZM1knitY3MCdyEgzBnBgkqhkiG9w0BCQ8xWjBYMAoGCCqGSIb3DQMHMA4GCCqGSIb3
DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG9w0DAgIBKDAHBgUrDgMCGjAK
BggqhkiG9w0CBTCBhQYJKwYBBAGCNxAEMXgwdjBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhh
d3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVt
YWlsIElzc3VpbmcgQ0ECEFiofjswHrxbxE7M/zOuXB8wgYcGCyqGSIb3DQEJEAILMXigdjBiMQsw
CQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UE
AxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0ECEFiofjswHrxbxE7M/zOuXB8w
DQYJKoZIhvcNAQEBBQAEgYBkhKlhriqcJTu/Jf0Hgka/9yyvUe5ZrB2vbYbzeyQGynB3RDNPpR6N
HG04HuW36N4ETHq/G1ibOdoNRNmUTX5SFIHJ+Rd/gAdUPAGe7VLPo3vR029LZQ5UZ4rv8JdsIzCg
7a541jR5i+r+RX4MhydnFd47loFpoWtaTyXpdveNdQAAAAAAAA==

------=_NextPart_000_0000_01CA62C4.1A657810--

From general-return-731-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 11 10:51:08 2009
Return-Path: <general-return-731-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81958 invoked from network); 11 Nov 2009 10:51:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Nov 2009 10:51:08 -0000
Received: (qmail 23840 invoked by uid 500); 11 Nov 2009 10:51:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23755 invoked by uid 500); 11 Nov 2009 10:51:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23745 invoked by uid 99); 11 Nov 2009 10:51:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Nov 2009 10:51:07 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Nov 2009 10:50:56 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 3C5C3B7F54
	for <general@hadoop.apache.org>; Wed, 11 Nov 2009 10:50:35 +0000 (GMT)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id Xnmq8NMyRVkl for <general@hadoop.apache.org>;
	Wed, 11 Nov 2009 10:50:28 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id B9773B7F3C
	for <general@hadoop.apache.org>; Wed, 11 Nov 2009 10:50:27 +0000 (GMT)
MailScanner-NULL-Check: 1258541417.30303@KuBFpZeCXHwOzsDVTOy2ZQ
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id nABAoGSM021146
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Wed, 11 Nov 2009 10:50:17 GMT
Message-ID: <4AFA96E8.2030303@apache.org>
Date: Wed, 11 Nov 2009 10:50:16 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Showing hadoop status
References: <b45b7eee0911100158t414633f9pcd3fffc66a221bae@mail.gmail.com>
In-Reply-To: <b45b7eee0911100158t414633f9pcd3fffc66a221bae@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: nABAoGSM021146
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

Mark N wrote:
> Currently  I was trying to read the log files generated by hadoop so that I
> could show the status of map/reduce jobs in user interfaces
> (consider that UI is a separate application )
> 
> I was looking at JobTracker APIs and was trying use following APIs
>  1. jobtracker.getAllJobs()  which returns array of jobstatus objects
> 
>  2. Then for each jobStatus we can call
>     mapProgress() , isJobCompleted()  methods
> 
> The problem is how do I link existing jobTracker ( which is already running
> )  with my  code ? i am trying to built something like  " hadoop UI ."
> 
> should I use jobClient  APIs ? Also is this a proper approach ?
> 
> thanks in advance . N mark.
> 

jobclient API is full featured, but brittle against versions. If you 
look at Hadoop Studio, they apparently have added a plug-in layer so 
they can work with different versions of hadoop more gracefully -you may 
want to use their code:  http://www.hadoopstudio.org/

Some of the JSP pages have status, there's a new one that pushes some 
XML content out, but nobody has yet sat down and written a stable, 
secure long-haul interface to the job tracker.

I've talked about it, got some ideas, but not started coding a proper 
RESTful front end
http://www.slideshare.net/steve_l/long-haul-hadoop

-steve

From general-return-732-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 11 12:00:14 2009
Return-Path: <general-return-732-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 809 invoked from network); 11 Nov 2009 12:00:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Nov 2009 12:00:12 -0000
Received: (qmail 94775 invoked by uid 500); 11 Nov 2009 12:00:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94687 invoked by uid 500); 11 Nov 2009 12:00:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94677 invoked by uid 99); 11 Nov 2009 12:00:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Nov 2009 12:00:11 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [156.148.72.33] (HELO raffaello.crs4.it) (156.148.72.33)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Nov 2009 12:00:08 +0000
Received: from [156.148.71.202] (neuron.crs4.it [156.148.71.202])
	by raffaello.crs4.it (Postfix) with ESMTP id 421276BB9F
	for <general@hadoop.apache.org>; Wed, 11 Nov 2009 12:59:34 +0100 (CET)
Message-ID: <4AFAA731.1000504@crs4.it>
Date: Wed, 11 Nov 2009 12:59:45 +0100
From: Simone Leo <simone.leo@crs4.it>
User-Agent: Thunderbird 2.0.0.23 (X11/20091007)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: SGE patch for HOD
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi everybody,

At CRS4, Distributed Computing Group (http://dc.crs4.it), we developed
and tested a simple patch that adds Grid Engine support to HOD. Since
porting HOD to SGE is listed in the Project Suggestions page at:

http://wiki.apache.org/hadoop/ProjectSuggestions

we decided to share the patch with the community.

The patch is available as an attachment to:

https://issues.apache.org/jira/browse/HADOOP-6369

After patching, HOD passes all unit tests at our site, and we tested the
SGE driver on a production cluster consisting of about 400 nodes. The
patch should preserve Torque functionality, although we were not able to
test this on production.

*NOTE*: the patch is for the official hadoop-0.20.1 HOD release. It
should be easy, however, to generate patches for other versions.

To patch, simply download the patch from the above JIRA link and run:

cd ${HADOOP_HOME}/src/contrib && patch -p0 <hod_ge.patch

-- 
Simone Leo
Distributed Computing group
Advanced Computing and Communications program
CRS4
POLARIS - Building #1
Piscina Manna
I-09010 Pula (CA) - Italy
e-mail: simleo@crs4.it
http://www.crs4.it

From general-return-733-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 11 17:33:24 2009
Return-Path: <general-return-733-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53004 invoked from network); 11 Nov 2009 17:33:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Nov 2009 17:33:24 -0000
Received: (qmail 3794 invoked by uid 500); 11 Nov 2009 17:33:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3705 invoked by uid 500); 11 Nov 2009 17:33:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3695 invoked by uid 99); 11 Nov 2009 17:33:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Nov 2009 17:33:23 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00,NO_RDNS_DOTCOM_HELO
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Nov 2009 17:33:19 +0000
Received: from SP1-EX07CAS02.ds.corp.yahoo.com (sp1-ex07cas02.ds.corp.yahoo.com [216.252.116.138])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id nABHWM6E096054
	for <general@hadoop.apache.org>; Wed, 11 Nov 2009 09:32:28 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:references:in-reply-to:accept-language:
	content-language:x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:
	content-type:content-transfer-encoding:mime-version;
	b=RpUusPonQZkEdB/i07TN0V2Q3aHvFoz05k/8kg4fK/v/IpYK8x60Cd66q5dYQ8bq
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 SP1-EX07CAS02.ds.corp.yahoo.com ([216.252.116.167]) with mapi; Wed, 11 Nov
 2009 09:32:26 -0800
From: Dekel Tankel <dekel@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 11 Nov 2009 09:32:25 -0800
Subject: RE: Hadoop User Group (Bay Area) - next Wednesday (Nov 18th) at
 	Yahoo!
Thread-Topic: Hadoop User Group (Bay Area) - next Wednesday (Nov 18th) at
 	Yahoo!
Thread-Index: AcpiaSnGH0jGsppyTpC/BN2d+8rrxAAi7NPQ
Message-ID: <46E2672895E8744A97D7577A6110858B4FD60B3072@SP1-EX07VS01.ds.corp.yahoo.com>
References: <46E2672895E8744A97D7577A6110858B4E1AC69A91@SP1-EX07VS01.ds.corp.yahoo.com>
	 <46E2672895E8744A97D7577A6110858B4FD39F6EDF@SP1-EX07VS01.ds.corp.yahoo.com>
	 <46E2672895E8744A97D7577A6110858B4FD60B2D55@SP1-EX07VS01.ds.corp.yahoo.com>
	 <e497591c0911101421u4971c874ha7c33b6fb48d4ab5@mail.gmail.com>
 <be9b52410911101650q1342b1bcoa63b575f77ff4e8@mail.gmail.com>
In-Reply-To: <be9b52410911101650q1342b1bcoa63b575f77ff4e8@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

I like this. Great idea.
Let me check our options...

	Dekel

-----Original Message-----
From: Abhishek Pratap [mailto:abhishek.vit@gmail.com]=20
Sent: Tuesday, November 10, 2009 4:50 PM
To: general@hadoop.apache.org
Subject: Re: Hadoop User Group (Bay Area) - next Wednesday (Nov 18th) at Ya=
hoo!

Hi Harshit

I second your idea. Since it is development under rapid progress, streaming
live/recording will help reach out to lot of interested folks. I would be
one of them.

Cheers,
-Abhi

On Tue, Nov 10, 2009 at 5:21 PM, Harshit Kumar <hkumar.arora@gmail.com>wrot=
e:

> Hi
>
> Any consideration of making this event available for global members of
> hadoop community, say for instance, streaming it live.
>
> Or at least record the event and upload on youtube.
>
> Regards
> H. Kumar
> Phone(Mobile): +82-10-2892-9663
> Phone(Office): +82-31-
> skype: harshit900
> Blog: http://harshitkumar.wordpress.com
> Website: http:/kumarharmuscat.tripod.com
>
>
> 2009/11/11 Dekel Tankel <dekel@yahoo-inc.com>
>
> > Hi all,
> >
> > We are one week away from the next Bay Area Hadoop User Group  - Yahoo!
> > Sunnyvale Campus, next Wednesday (Nov 18th) at 6PM
> >
> > We have an exciting evening planed:
> >
> > *Katta, Solr, Lucene and Hadoop - Searching at scale, Jason Rutherglen
> and
> > Jason Venner
> >
> > *Walking through the New File system API, Sanjay Radia, Yahoo!
> >
> > *Keep your data in Jute but still use it in python, Paul Tarjan, Yahoo!
> >
> >
> > Please RSVP here:
> > http://www.meetup.com/hadoop/calendar/11724002/
> >
> >
> > Please note that this is the last HUG for 2009, as we will not have a
> > meeting on December (due to the holidays).
> > We will open 2010 with a HUG on Jan 20th.
> >
> > Looking forward to seeing you next week!
> >
> > Dekel
> >
> >
>

From general-return-734-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 12 12:17:38 2009
Return-Path: <general-return-734-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 74441 invoked from network); 12 Nov 2009 12:17:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Nov 2009 12:17:38 -0000
Received: (qmail 50922 invoked by uid 500); 12 Nov 2009 12:17:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50827 invoked by uid 500); 12 Nov 2009 12:17:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50817 invoked by uid 99); 12 Nov 2009 12:17:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Nov 2009 12:17:36 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of nipen.mark@gmail.com designates 209.85.222.198 as permitted sender)
Received: from [209.85.222.198] (HELO mail-pz0-f198.google.com) (209.85.222.198)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Nov 2009 12:17:28 +0000
Received: by pzk36 with SMTP id 36so1428207pzk.5
        for <general@hadoop.apache.org>; Thu, 12 Nov 2009 04:17:07 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=9if/nwBXxlS8J9f/4rB1Z4Zlsbql4yjhGYB3LMb6Fr4=;
        b=cxLbMPpQVi/lbL98D13/malqU/N2GA62pGdpa6b1Uegpqy0+loDuQcMVuLZ1LbhyDW
         LtSMf8WaEFSaxYAaKKCEDYyHp+U/aifuC8f1Wm8E9sn/URcScmznLZyg6Xin3VmdHquS
         5RqTIWBwRbpIWoJ+XWST2ingHdZqmnQfpI3x8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=k28FcMI8/T8AGbGEV54wlLYCabQ/J7LDDiaZxfnnan2mpsmi6B+sl+sOj4T3yIwc5z
         QIKA6TwEfoubJ9jiOWuHKHBfRJD4mdMlFTHd1H8uJ13ua3ZeNu4kgXoYeqmqsmkWTwx1
         ctm+SdxLDa50PsB4QMwRoyrWvr7g8Wr2NvdxM=
MIME-Version: 1.0
Received: by 10.142.151.9 with SMTP id y9mr306398wfd.337.1258028226843; Thu, 
	12 Nov 2009 04:17:06 -0800 (PST)
Date: Thu, 12 Nov 2009 18:02:06 +0545
Message-ID: <b45b7eee0911120417p46368a80r460b97b55ac8bd49@mail.gmail.com>
Subject: configuring hadoop cluster
From: Mark N <nipen.mark@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd311ec9414fb04782b85fa
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd311ec9414fb04782b85fa
Content-Type: text/plain; charset=ISO-8859-1

is it possible to use local file system ( or NFS) with a distributed cluster
configuration ?
If its possible what are the configuration that I need to change

Currently we are trying to process documents from local file system instead
of HDFS.

thanks
Mark N

--000e0cd311ec9414fb04782b85fa--

From general-return-735-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 12 22:23:18 2009
Return-Path: <general-return-735-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 11978 invoked from network); 12 Nov 2009 22:23:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Nov 2009 22:23:18 -0000
Received: (qmail 17744 invoked by uid 500); 12 Nov 2009 22:23:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17660 invoked by uid 500); 12 Nov 2009 22:23:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17646 invoked by uid 99); 12 Nov 2009 22:23:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Nov 2009 22:23:16 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Nov 2009 22:23:07 +0000
Received: from [10.73.135.247] (wifi-e-135-247.corp.yahoo.com [10.73.135.247])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id nACMMhXt002333;
	Thu, 12 Nov 2009 14:22:43 -0800 (PST)
Message-ID: <4AFC8AB2.2010400@apache.org>
Date: Thu, 12 Nov 2009 14:22:42 -0800
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: HTTP transport?
References: <4AAAC403.80809@apache.org>
In-Reply-To: <4AAAC403.80809@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

One additional benefit of using HTTP is that people are always working 
to improve performance, and not only optimizing servers -- Google's SPDY:

http://www.readwriteweb.com/archives/spdy_google_wants_to_speed_up_the_web.php

Multiplexed requests, compressed headers, etc...

Patrick

Doug Cutting wrote:
> I'm considering an HTTP-based transport for Avro as the preferred, 
> high-performance option.
> 
> HTTP has lots of advantages.  In particular, it already has
>  - lots of authentication, authorization and encryption support;
>  - highly optimized servers;
>  - monitoring, logging, etc.
> 
> Tomcat and other servlet containers support async NIO, where a thread is 
> not required per connection.  A servlet can process bulk data with a 
> single copy to and from the socket (bypassing stream buffers).  Calls 
> can be multiplexed over a single HTTP connection using Comet events.
> 
> http://tomcat.apache.org/tomcat-6.0-doc/aio.html
> 
> Zero copy is not an option for servlets that generate arbitrary data, 
> but one can specify a file/start/length tuple and Tomcat will use 
> sendfile to write the response.  That means that while HDFS datanode 
> file reads could not be done via RPC, they could be done via HTTP with 
> zero-copy.  If authentication and authorization are already done in the 
> HTTP server, this may not be a big loss.  The HDFS client might make two 
> HTTP requests, one to read a files data, and another to read its 
> checksums.  The server would then stream the entire block to the client 
> using sendfile, using TCP flow control as today.
> 
> Thoughts?
> 
> Doug

From general-return-736-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 13 08:04:20 2009
Return-Path: <general-return-736-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29812 invoked from network); 13 Nov 2009 08:04:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Nov 2009 08:04:20 -0000
Received: (qmail 87729 invoked by uid 500); 13 Nov 2009 08:04:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87648 invoked by uid 500); 13 Nov 2009 08:04:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 48636 invoked by uid 99); 11 Nov 2009 18:02:59 -0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of tinova79@gmail.com designates 209.85.220.215 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:from:date:x-google-sender-auth:message-id:subject:to:cc
         :content-type;
        bh=5qJtPPDCo8oWa1CbIbSw1/9kBs+paZ+jJCQSpe09DvY=;
        b=J6dvYrPx7D0nzcdJb1GK4htETk00DDw10lazayjnhO6yFCO+NFbwjMIm26lF7joWoR
         FKS2Ts66X9bYutpwpvITRJRhCgZg5EEZtw2gCGo8c5i92t9ZqmlkwSUr5xXCYDXPG+mL
         ITPKqIF+xWKgEUzx1B7mlJCBzS8Dc2uhZTvRM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:from:date
         :x-google-sender-auth:message-id:subject:to:cc:content-type;
        b=KH4ZpYHek+WKYsPvjWtx18QDTPlMiQMekv++54By8HZqSkzJfPwzGszL3rfk50kw3Q
         ZeUfk+mhfv7wbK4wpBMeIKWjiyDcWSbdebg2MULwQkypn5QxKcqCn4QDvEyJE6GjUOLX
         c1xg4qmNAHQYcSNRRpIYKRh9lJVgEKHTbZy3I=
MIME-Version: 1.0
Sender: tinova79@gmail.com
In-Reply-To: <ADF94D8555C7A246B86A633685E0178AB84B40DB72@planck.ka.sara.nl>
References: <ADF94D8555C7A246B86A633685E0178AB84B40DB72@planck.ka.sara.nl>
From: Tino Vazquez <tinova@fdi.ucm.es>
Date: Wed, 11 Nov 2009 19:02:13 +0100
X-Google-Sender-Auth: bdbed9b66d03e4c0
Message-ID: <dc454f40911111002y2d290587r89879248ca92480@mail.gmail.com>
Subject: Re: [one-users] Hadoop on Opennebula
To: Evert Lammerts <Evert.Lammerts@sara.nl>
Cc: "users@lists.opennebula.org" <users@lists.opennebula.org>, 
	"general@hadoop.apache.org" <general@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=00151747868e2d9bcf04781c3bfb

--00151747868e2d9bcf04781c3bfb
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Hi Evert,

We are not Hadoop experts, so we will appreciate if you could redirect the
Hadoop file copying problem to the relevant mailing list.

About the execution of Hadoop in a cloud, OpenNebula can provide your Hadoo=
p
VMs with contextualization information. This means that the VM can have its
hostname and IP set, and this can be accessed by other VMs. So, for
instance, your master node can be given an IP address (or you can have this
IP to be fixed) and then the worker nodes will be able to access this
information (IP and/or hostname of the master), inside an attached ISO. Mor=
e
information on contextualization in OpenNebula can be found in [1]. I think
you have a very interesting use case, we are happy to provide support for
any contextualization related issue.

Hope it helps,

-Tino

[1] http://opennebula.org/doku.php?id=3Ddocumentation:rel1.4:cong


--
Constantino V=E1zquez, Grid Technology Engineer/Researcher:
http://www.dsa-research.org/tinova
DSA Research Group: http://dsa-research.org
Globus GridWay Metascheduler: http://www.GridWay.org
OpenNebula Virtual Infrastructure Engine: http://www.OpenNebula.org


On Wed, Nov 11, 2009 at 11:42 AM, Evert Lammerts <Evert.Lammerts@sara.nl>wr=
ote:

>  Hi list,
>
>
>
> We have a cluster with 44 cores on which we are evaluating OpenNebula. Al=
l
> nodes run Ubuntu server 64bit. I=92m trying to get Hadoop 0.20.1 to work,=
 so
> I=92m following the cluster setup steps on hadoop.apache.org.
>
>
>
> As a start I created three VM=92s running Ubuntu 9.10 32bit desktop editi=
on.
> After installing Sun=92s 1.6 JRE I put Hadoop into my homedir. I configur=
ed
> the three installations of Hadoop as follows:
>
>
>
> =3D=3D conf/hadoop-env.sh =3D=3D
>
> Set JAVA_HOME to the appropriate directory
>
>
>
> =3D=3D conf/core-site.xml =3D=3D
>
> Set fs.default.name to the ip address of the designated namenode, on port
> 9000:
>
> <property>
>
>    <name>fs.default.name</name>
>
>    <value>hdfs://XXX.XXX.X.XXX:9000/</value>
>
>  </property>
>
>
>
> =3D=3D conf/hdfs-site.xml =3D=3D
>
> Set dfs.name.dir to a directory in my homedir:
>
> <property>
>
>   <name>dfs.name.dir</name>
>
>   <value>/home/cloud/var/log/hadoop/</value>
>
> </property>
>
>
>
> =3D=3D conf/mapred-site.xml =3D=3D
>
> Set mapred.job.tracker to the ip address of the designated jobtracker, on
> port 9001:
>
> <property>
>
>   <name>mapred.job.tracker</name>
>
>   <value>XXX.XXX.X.XXX:9001</value>
>
> <property>
>
>
>
> Apart from Hadoop configuration I manually set the hostname for the
> namenode, jobtracker and slave (datanode & tasktracker) to respectively
> hadoop-namenode, hadoop-jobtracker and hadoop-slave01.
>
>
>
> I=92m able to start the hdfs with bin/start-dfs.sh, and mapreduce with
> bin/start-mapred.sh without any exceptions. However, when I now try to co=
py
> files onto hdfs, I=92m getting the following exception:
>
>
>
> $ bin/hadoop fs -put conf input
>
> 09/11/11 05:24:47 WARN hdfs.DFSClient: DataStreamer Exception:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/cloud/input/capacity-scheduler.xml could only be replicated to 0
> nodes, instead of 1
>
>                 at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FS=
Namesystem.java:1267)
>
>                 at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:42=
2)
>
>                 at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown
> Source)
>
>                 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI=
mpl.java:25)
>
>                 at java.lang.reflect.Method.invoke(Method.java:597)
>
>                 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>
>                 at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>
>                 at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>
>                 at java.security.AccessController.doPrivileged(Native
> Method)
>
>                 at javax.security.auth.Subject.doAs(Subject.java:396)
>
>                 at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
>
>
>                 at org.apache.hadoop.ipc.Client.call(Client.java:739)
>
>                 at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
>
>                 at $Proxy0.addBlock(Unknown Source)
>
>                 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>
>                 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java=
:39)
>
>                 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI=
mpl.java:25)
>
>                 at java.lang.reflect.Method.invoke(Method.java:597)
>
>                 at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvoc=
ationHandler.java:82)
>
>                 at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationH=
andler.java:59)
>
>                 at $Proxy0.addBlock(Unknown Source)
>
>                 at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFS=
Client.java:2904)
>
>                 at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DF=
SClient.java:2786)
>
>                 at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.ja=
va:2076)
>
>                 at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClie=
nt.java:2262)
>
>
>
> 09/11/11 05:24:47 WARN hdfs.DFSClient: Error Recovery for block null bad
> datanode[0] nodes =3D=3D null
>
> 09/11/11 05:24:47 WARN hdfs.DFSClient: Could not get block locations.
> Source file "/user/cloud/input/capacity-scheduler.xml" - Aborting...
>
> put: java.io.IOException: File /user/cloud/input/capacity-scheduler.xml
> could only be replicated to 0 nodes, instead of 1
>
>
>
> Can anybody shed light on this? I=92m guessing it=92s a configuration iss=
ue, so
> that=92s the direction I=92m looking at.
>
>
>
> Another question I have is more generally about getting Hadoop to work on=
 a
> cloud. The issue I foresee is about the ip addresses of the masters and
> slaves. How do I dynamically configure the hadoop instances during start-=
up
> of the images to end up with a namenode, a jobtracker and a number of
> slaves? I=92ll need the ip addresses of all machines and all machines nee=
d a
> unique hostname=85 Does anybody have any experience with this?
>
>
>
> Thanks in advance!
>
>
>
> Evert Lammerts
>
> Adviseur
>
> SARA Computing & Network Services
>
> High Performance Computing & Visualization
>
> eScience Support Group
>
>
>
> Phone: +31 20 888 4101
>
> Email: evert.lammerts@sara.nl
>
>
>
> _______________________________________________
> Users mailing list
> Users@lists.opennebula.org
> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>
>

--00151747868e2d9bcf04781c3bfb--

From general-return-737-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 16 07:10:49 2009
Return-Path: <general-return-737-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 79927 invoked from network); 16 Nov 2009 07:10:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Nov 2009 07:10:49 -0000
Received: (qmail 93449 invoked by uid 500); 16 Nov 2009 07:10:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93367 invoked by uid 500); 16 Nov 2009 07:10:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93357 invoked by uid 99); 16 Nov 2009 07:10:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Nov 2009 07:10:47 +0000
X-ASF-Spam-Status: No, hits=-1.9 required=5.0
	tests=AWL,BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of apurba@pramati.com designates 196.12.47.8 as permitted sender)
Received: from [196.12.47.8] (HELO mail.pramati.com) (196.12.47.8)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Nov 2009 07:10:43 +0000
Received: from localhost (unknown [127.0.0.1])
	by mail.pramati.com (Postfix) with ESMTP id A47FC7781EA
	for <general@hadoop.apache.org>; Mon, 16 Nov 2009 07:06:29 +0000 (UTC)
X-Virus-Scanned: amavisd-new at pramati.com
Received: from mail.pramati.com ([127.0.0.1])
	by localhost (mail.pramati.com [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id GGJXqUOueGvx for <general@hadoop.apache.org>;
	Mon, 16 Nov 2009 12:36:24 +0530 (IST)
Received: from [192.168.2.197] (unknown [192.168.2.197])
	by mail.pramati.com (Postfix) with ESMTP id 080717781CC
	for <general@hadoop.apache.org>; Mon, 16 Nov 2009 12:36:24 +0530 (IST)
Message-ID: <4B00FAD7.40202@pramati.com>
Date: Mon, 16 Nov 2009 12:40:15 +0530
From: Apurba Kumar Nath <apurba@pramati.com>
User-Agent: Thunderbird 1.5 (Windows/20051201)
MIME-Version: 1.0
To:  general@hadoop.apache.org
Subject: smart serialization for WordCount example
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

I was wondering if we could introduce a template to let the end user add 
strings and ints to the context and the template could convert them to 
the appropriate serializable type. I am suggesting that the current
context.write(word, one); where word is Text and one is IntWritable, use 
something like context.write(itr.nextToken, 1); now either the context 
is smart enough to convert itr.nextToken which is string into Text and 1 
which is int into IntWritable or the map method is worked on by some 
method in the template to do the same. The advantage is that for 
beginners they can still talk in ints and strings and other such classes 
and may not have to get exposed to IntWritable and other such more 
easily serializable wrappers. I can commit the patch if the idea seems ok.

-- 
Thanks
Apurba

Share interests with friends with just a click. Try it!
http://tellafriend.socialtwist.com



From general-return-738-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 16 17:13:41 2009
Return-Path: <general-return-738-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95880 invoked from network); 16 Nov 2009 17:13:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Nov 2009 17:13:40 -0000
Received: (qmail 80723 invoked by uid 500); 16 Nov 2009 17:13:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80647 invoked by uid 500); 16 Nov 2009 17:13:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80637 invoked by uid 99); 16 Nov 2009 17:13:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Nov 2009 17:13:39 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [68.142.207.139] (HELO web32208.mail.mud.yahoo.com) (68.142.207.139)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 16 Nov 2009 17:13:29 +0000
Received: (qmail 2999 invoked by uid 60001); 16 Nov 2009 17:13:06 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1258391586; bh=ZIST+Bdsj6WG75dJ1qK39q6Qk5sXLo5V64iBUgt4Cks=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=qeiH0Tc8WeurtUZ4F6lfBRgqy/GCsfZzBMpggQNfR8cnWCzIbLVWHJBbl0BCIAf0LtPXXtXYse/8rk1/cejfw4iC6WRWP4/2iT3mErV4dXZPuLH8xyhl2XEF/yXbh25WJqm6A1CfaAQ9O3qzkjx+sIklENQo+NOOTDxZ4ussIEQ=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=Yj/+44NCfok0DWRxwMMMQ+PaZaEmIJSwEA1UWwDuLz6ltZU82HBygcmMQL7IqZfn11aaJeko654CBsIWM+6F7YQ5DMA0E/vwiMNNYrJUiT2gj/8nIMkd1i8UBpguRNAVzFRHVORUyr9SVQoCyq0EJoBkfpdIYFYw4/xkdkKw43Y=;
Message-ID: <654001.2236.qm@web32208.mail.mud.yahoo.com>
X-YMail-OSG: l9c6254VM1lqnzgRLy95l9f15.zpzGji4lzbHH5AKSltuDPjKKBVD9VKksbM_c1jNmO.qFOGTa6v3ct16xMpdCPmYNKcwxqzhURROlJgJczp5EJcETSJZqdlGVsgDlF3giDWopOuwHgR1KFwGzGscv4Lg9MRJnmzBDxNfLOI3ENE_EZ2y8DIzpCXKLv0.74XQfvqM4EEndolvheiz.GRUxiZ.6wC3WGa9WjCMmXs6h.1FG.xf.X2Y3I.d0Mnx5zuMODd9kYsDG9wORddIAGhF4p.M4BFboroixXV4MyRm4hUxEcnpWvKrG5l5eeniAcxcgOEZKoUwwqbtb4Z_p2utA--
Received: from [69.143.207.68] by web32208.mail.mud.yahoo.com via HTTP; Mon, 16 Nov 2009 09:13:06 PST
X-Mailer: YahooMailClassic/8.1.6 YahooMailWebService/0.7.361.4
Date: Mon, 16 Nov 2009 09:13:06 -0800 (PST)
From: Rajiv Maheshwari <rajivm01@yahoo.com>
Subject: Map-reduce sorting on multiple keys
To: Hadoop Mail-list <general@hadoop.apache.org>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-219672166-1258391586=:2236"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-219672166-1258391586=:2236
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Hi everyone,

I have a need to sort the output of map on 2 keys (key1, key2) - first on k=
ey1, then on key2.

Example:
key1=A0 key2=A0=A0 values
-------------------------
0001=A0 0001=A0 values...
0001=A0 0002=A0 values...
0002=A0 0001=A0 values...
0002=A0 0005=A0 values...


I am thinking of the following solution approach:

Define KEY =3D key1, key2=A0=A0 /* concatenate keys */. Override default Ha=
shPartitioner and use only key1 in hashCode computation.


public class HashPartitioner<K2, V2> implements Partitioner<K2, V2> {

public void configure(JobConf job) {}

public int getPartition(K2 key, V2 value, int numPartitions) {

=A0=A0=A0 return (key.getKey1().hashCode() & Integer.MAX_VALUE) % numPartit=
ions;
=A0=A0=A0 }
}

Would this work?

Does anyone have any better ideas?

Thanks much,
Rajiv

=0A=0A=0A      
--0-219672166-1258391586=:2236--

From general-return-739-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 16 19:08:54 2009
Return-Path: <general-return-739-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44293 invoked from network); 16 Nov 2009 19:08:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Nov 2009 19:08:54 -0000
Received: (qmail 5921 invoked by uid 500); 16 Nov 2009 19:08:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5830 invoked by uid 500); 16 Nov 2009 19:08:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5820 invoked by uid 99); 16 Nov 2009 19:08:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Nov 2009 19:08:52 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of goutham.patnaik@gmail.com designates 209.85.210.194 as permitted sender)
Received: from [209.85.210.194] (HELO mail-yx0-f194.google.com) (209.85.210.194)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Nov 2009 19:08:50 +0000
Received: by yxe32 with SMTP id 32so3758795yxe.5
        for <general@hadoop.apache.org>; Mon, 16 Nov 2009 11:08:30 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :from:date:message-id:subject:to:content-type;
        bh=SpfdfREIA5MUrEcEJ2koeAtFbzB4A5QPU5BJfcSVC9A=;
        b=Pjlbj48b0prM+CGq55XiuS+smbMQqW5yMHJcuOBV7AS89UyhilOTMjmt8R8/mOv3Ab
         Qayar22hV9pAuNr11bJPu9rZV05HAkcd8vMGURTFWMZ3fR4C3aUIhq/dLUtZhyd7Pjnu
         vdEPeIXvsyeB+I7FGwMC8OVylbVffdE38zRIU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=K0nu5Kki3wjA9xh8jmvg6+JjiPx7jkIt5gsdOIb1qL9aD4jMv4oiALE4zrnfh2d1QF
         0ht2QaKc967FC7MQ12P9+6HqXNyXcwLWA13lhtH2Pr862IWIwydVKQbb1Q+jMYr3pnQT
         Czm/01pTzpnizxDYXyfICxtvIsqk5kEkWCwbc=
MIME-Version: 1.0
Received: by 10.150.43.17 with SMTP id q17mr14222048ybq.197.1258398509050; 
	Mon, 16 Nov 2009 11:08:29 -0800 (PST)
In-Reply-To: <654001.2236.qm@web32208.mail.mud.yahoo.com>
References: <654001.2236.qm@web32208.mail.mud.yahoo.com>
From: goutham patnaik <goutham.patnaik@gmail.com>
Date: Mon, 16 Nov 2009 11:08:09 -0800
Message-ID: <1b3669280911161108p4fbe9d20i12dd564a121aecac@mail.gmail.com>
Subject: Re: Map-reduce sorting on multiple keys
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd330581e2c5e047881bcfa

--000e0cd330581e2c5e047881bcfa
Content-Type: text/plain; charset=ISO-8859-1

Rajiv,

You could write your own class which implements the WritableComparable
interface and use this as your key class -  all u need to do is implement
the write, readFields and compareTo methods - the map will then sort your
keys using this method :

public class TupleKey implements WritableComparable {
 IntWritable k1;
 IntWritable k2;
.......
}

On Mon, Nov 16, 2009 at 9:13 AM, Rajiv Maheshwari <rajivm01@yahoo.com>wrote:

> Hi everyone,
>
> I have a need to sort the output of map on 2 keys (key1, key2) - first on
> key1, then on key2.
>
> Example:
> key1  key2   values
> -------------------------
> 0001  0001  values...
> 0001  0002  values...
> 0002  0001  values...
> 0002  0005  values...
>
>
> I am thinking of the following solution approach:
>
> Define KEY = key1, key2   /* concatenate keys */. Override default
> HashPartitioner and use only key1 in hashCode computation.
>
>
> public class HashPartitioner<K2, V2> implements Partitioner<K2, V2> {
>
> public void configure(JobConf job) {}
>
> public int getPartition(K2 key, V2 value, int numPartitions) {
>
>     return (key.getKey1().hashCode() & Integer.MAX_VALUE) % numPartitions;
>     }
> }
>
> Would this work?
>
> Does anyone have any better ideas?
>
> Thanks much,
> Rajiv
>
>
>
>
>

--000e0cd330581e2c5e047881bcfa--

From general-return-740-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 16 19:17:06 2009
Return-Path: <general-return-740-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46885 invoked from network); 16 Nov 2009 19:17:06 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Nov 2009 19:17:06 -0000
Received: (qmail 27229 invoked by uid 500); 16 Nov 2009 19:17:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27172 invoked by uid 500); 16 Nov 2009 19:17:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27162 invoked by uid 99); 16 Nov 2009 19:17:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Nov 2009 19:17:05 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [68.142.207.134] (HELO web32203.mail.mud.yahoo.com) (68.142.207.134)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 16 Nov 2009 19:17:03 +0000
Received: (qmail 52228 invoked by uid 60001); 16 Nov 2009 19:16:42 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1258399001; bh=eYAnehySQF66DBDcMe75Rsq9oiTKhvv3T/3teerWE18=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=Tm2Pv2Iw+CfYQEnYLne5dq6ju4gy1xIrWsT2jYBoDh0sZvJoVOaua1rXQp05J8ehCSrbE999p1jDWLlkxkIz1tz6pl9YrKiZlAW3DxG1uY+2vsHNs5hHAJFbQ34a7MosMeAsQaT4i3NTTCP8ZHkPbOy9p1SNMmdMjA7b9oRIpVI=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=yp0TFZ5aV7GWdjug91zEwdT5iGAiwlH7S9pJ3ZzPOwBQFVZJM6LoorUZWz0E6j/i9sYTMyKdlFyOx3qP3+oR47k7s0L2Em74B3TjRMHeRRu/Gw8fiNHAT2DHQ2kcGU3jKDD4Pum4lTFuhCNYO7Y6J3fUOSRd8DvyAs6lxV+fGL0=;
Message-ID: <895334.52215.qm@web32203.mail.mud.yahoo.com>
X-YMail-OSG: DyoHoQsVM1msu2fXeWZeEhEI6hERW1LLNjgKeOOMH5.sl5bO7q8x116KaZ_UF3IYMW7UCtmIoBxeU9_e6jPb2YfegMliHVU4Q2DtrhVIfScEW8RU5udOL9Wvb3StNMSXPzTA4yTyS1R5gXuUmtvXoOYFn3UVD_OabBZ0EXlVi5gc2mtw.VC7Q.PncKjqiijeE5qWur.x1VE5oLA8mekzkIr0Vg2FKXi0HXZZe.oosTkk4sNXp1GnIe0ZRUlucY4XyVgunR8Y7eHiUQOqRy7IZ0mlHs86D9HlB7N6em93jfLnPvutv_ye3JjOduOHM9sRN3jC1NMPRb8V7mCD.Q.n4PVa630-
Received: from [69.143.207.68] by web32203.mail.mud.yahoo.com via HTTP; Mon, 16 Nov 2009 11:16:41 PST
X-Mailer: YahooMailClassic/8.1.6 YahooMailWebService/0.7.361.4
Date: Mon, 16 Nov 2009 11:16:41 -0800 (PST)
From: Rajiv Maheshwari <rajivm01@yahoo.com>
Subject: Re: Map-reduce sorting on multiple keys
To: general@hadoop.apache.org
In-Reply-To: <1b3669280911161108p4fbe9d20i12dd564a121aecac@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1676722562-1258399001=:52215"

--0-1676722562-1258399001=:52215
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Thanks, appreciate it.

Rajiv

--- On Mon, 11/16/09, goutham patnaik <goutham.patnaik@gmail.com> wrote:

From: goutham patnaik <goutham.patnaik@gmail.com>
Subject: Re: Map-reduce sorting on multiple keys
To: general@hadoop.apache.org
Date: Monday, November 16, 2009, 11:08 AM

Rajiv,

You could write your own class which implements the WritableComparable
interface and use this as your key class -=A0 all u need to do is implement
the write, readFields and compareTo methods - the map will then sort your
keys using this method :

public class TupleKey implements WritableComparable {
 IntWritable k1;
 IntWritable k2;
.......
}

On Mon, Nov 16, 2009 at 9:13 AM, Rajiv Maheshwari <rajivm01@yahoo.com>wrote=
:

> Hi everyone,
>
> I have a need to sort the output of map on 2 keys (key1, key2) - first on
> key1, then on key2.
>
> Example:
> key1=A0 key2=A0=A0=A0values
> -------------------------
> 0001=A0 0001=A0 values...
> 0001=A0 0002=A0 values...
> 0002=A0 0001=A0 values...
> 0002=A0 0005=A0 values...
>
>
> I am thinking of the following solution approach:
>
> Define KEY =3D key1, key2=A0=A0=A0/* concatenate keys */. Override defaul=
t
> HashPartitioner and use only key1 in hashCode computation.
>
>
> public class HashPartitioner<K2, V2> implements Partitioner<K2, V2> {
>
> public void configure(JobConf job) {}
>
> public int getPartition(K2 key, V2 value, int numPartitions) {
>
>=A0 =A0=A0=A0return (key.getKey1().hashCode() & Integer.MAX_VALUE) % numPa=
rtitions;
>=A0 =A0=A0=A0}
> }
>
> Would this work?
>
> Does anyone have any better ideas?
>
> Thanks much,
> Rajiv
>
>
>
>
>
=0A=0A=0A      
--0-1676722562-1258399001=:52215--

From general-return-741-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 16 19:24:45 2009
Return-Path: <general-return-741-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 48878 invoked from network); 16 Nov 2009 19:24:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Nov 2009 19:24:45 -0000
Received: (qmail 47000 invoked by uid 500); 16 Nov 2009 19:24:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46933 invoked by uid 500); 16 Nov 2009 19:24:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46923 invoked by uid 99); 16 Nov 2009 19:24:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Nov 2009 19:24:44 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mailinglists19@gmail.com designates 209.85.220.215 as permitted sender)
Received: from [209.85.220.215] (HELO mail-fx0-f215.google.com) (209.85.220.215)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Nov 2009 19:24:42 +0000
Received: by fxm7 with SMTP id 7so6124531fxm.29
        for <general@hadoop.apache.org>; Mon, 16 Nov 2009 11:24:20 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=cppSxDfCGKyzolbaWLuilFXRWl6ApPql/bgMsQpxl8A=;
        b=U4eFBTY7VMKYUynieoH4qUcZLh3goErs9MNic0ChxDfVjM2QlUhc9rYTOwDt7omPyc
         ENSppmEoz9pae7gSojCpK3NI8CioI9t5EGLlr/VEC02RSkxZfo2tmP6rgpoabAPdsRjT
         QF0Zya25ulH6WNfT3X3ULTu8ZoAPVIQnZKJKk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=iCLC88jPyUFOGYcoq6OTCNHYYrVPpBRoWH7Gd2Lb8HXyXgwuNXtLK6VNiz5m2OVCdK
         803p6YIt652tNTAkCrD3UjE5wcX80XrXrWLuMsDN4bWuait9WyDTnkIyp5gjT5JivH4L
         X9yDMcB5FPDvHYzvt49w3VLBXnNiUfxk6iw5Y=
MIME-Version: 1.0
Received: by 10.216.85.5 with SMTP id t5mr1128490wee.142.1258399460612; Mon, 
	16 Nov 2009 11:24:20 -0800 (PST)
In-Reply-To: <1eabbac30911141443l4943be00q45111aec0b67bfcc@mail.gmail.com>
References: <1eabbac30911141443l4943be00q45111aec0b67bfcc@mail.gmail.com>
Date: Mon, 16 Nov 2009 11:24:20 -0800
Message-ID: <1eabbac30911161124p73a2a631u665ee28a765e332@mail.gmail.com>
Subject: Fwd: Custom Writable not working...
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d64861d5dc4d047881f448

--0016e6d64861d5dc4d047881f448
Content-Type: text/plain; charset=ISO-8859-1

---------- Forwarded message ----------
From: Something Something <mailinglists19@gmail.com>
Date: Sat, Nov 14, 2009 at 2:43 PM
Subject: Custom Writable not working...
To: hbase-user@hadoop.apache.org


Hello,

I created a custom writable class called, TextPair, that's identical to the
one in Tom White's O'Reilly book (page 97).  The only difference is that the
compare function works differently.  In my case I want (1, 6) to compare
equally to (6,1), so I have something like this...

  @Override
  public int compareTo(TextPair tp) {
    int cmpFirst = first.compareTo(tp.first);
    int cmpSecond = second.compareTo(tp.second);

    if (cmpFirst == 0 && cmpSecond == 0) {
      System.out.println("Equal: (" + this.toString() + ") and " + "(" +
tp.toString() + ")");
      return 0;
    } else {
      int cmpFirstSecond = first.compareTo(tp.second);
      int cmpSecondFirst = second.compareTo(tp.first);
      if (cmpFirstSecond == 0 && cmpSecondFirst == 0) {
        System.out.println("Equal: (" + this.toString() + ") and " + "(" +
tp.toString() + ")");
        return 0;
      }
    }
    System.out.println("!!NOT Equal: (" + this.toString() + ") and " + "(" +
tp.toString() + ")");
    return -1;
  }


In my Mapper class I am emitting...

context.write(new TextPair(first, second), new IntWritable(1));


So I am expecting that when my Reducer is called the keys:

(1,4) & (4,1) would get combined.  This is NOT happening.  In fact, even
(1,4) are not getting combined.  In other words, the reducer is getting
called multiple times with keys (1,4)  (Makes sense?)

What am I doing wrong?  Also, I noticed that even after Reducer gets called
the TextPair.compareTo method keeps getting called.  By the way, my Reducer
is very simple.  It just adds counts (like the Word count program in the
tutorial.)

Please help.  Thanks.

--0016e6d64861d5dc4d047881f448--

From general-return-742-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 16 19:26:16 2009
Return-Path: <general-return-742-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 51395 invoked from network); 16 Nov 2009 19:26:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Nov 2009 19:26:16 -0000
Received: (qmail 52778 invoked by uid 500); 16 Nov 2009 19:26:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52692 invoked by uid 500); 16 Nov 2009 19:26:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52682 invoked by uid 99); 16 Nov 2009 19:26:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Nov 2009 19:26:15 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mailinglists19@gmail.com designates 72.14.220.152 as permitted sender)
Received: from [72.14.220.152] (HELO fg-out-1718.google.com) (72.14.220.152)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Nov 2009 19:26:10 +0000
Received: by fg-out-1718.google.com with SMTP id d23so2328588fga.11
        for <general@hadoop.apache.org>; Mon, 16 Nov 2009 11:25:48 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=CvNPTg5YNMj6gFTv1o5/I15mewxqbdYibDZkBB+Rw4A=;
        b=FPIq1wCKFd5Io8PbXwz2Pdxu0yKtLYHystI09AeCBH8L8dxfzbX4CX9bmAd55QSsbL
         ddyUwIukEC/du8AEZdX7E5m46R9UgPn7rB61+YwxDW1nJ5XD3BLWDhM3xfZOs6kmPuYr
         PCXUWSGC48iYN3YzkbeW5JuWhNgipbvGTr3ik=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=H/2SyvcdHmzXGwBL/pYIWKw5G74cz4Sv36vrXeq3VNxPhdjbQsWVOYyIvrtZXCstCO
         OdpRpRJ+dTT4IdHrb50FQSEhAnuSeAEKpRDx3dH7WrSZ8YqFnqfAoupGASaSFBTO97s7
         0K944TJzDW0dwUAyFJh0HoMFLlk7u4vBYZaFQ=
MIME-Version: 1.0
Received: by 10.216.85.194 with SMTP id u44mr485638wee.65.1258399548605; Mon, 
	16 Nov 2009 11:25:48 -0800 (PST)
In-Reply-To: <895334.52215.qm@web32203.mail.mud.yahoo.com>
References: <1b3669280911161108p4fbe9d20i12dd564a121aecac@mail.gmail.com>
	 <895334.52215.qm@web32203.mail.mud.yahoo.com>
Date: Mon, 16 Nov 2009 11:25:48 -0800
Message-ID: <1eabbac30911161125y32715fceub0d52fe98e68dd7e@mail.gmail.com>
Subject: Re: Map-reduce sorting on multiple keys
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d7e744148457047881fae3

--0016e6d7e744148457047881fae3
Content-Type: text/plain; charset=ISO-8859-1

Goutham,

Can you please take a look at my email titled.. "Custom Writable not
working"?  (I just sent it a few minutes ago.)  It's similar to this one.
The only difference is, instead of IntWritable I am using Text.  But the
sort in Map is not working as expected.  Can you tell me why?  Thanks.


On Mon, Nov 16, 2009 at 11:16 AM, Rajiv Maheshwari <rajivm01@yahoo.com>wrote:

> Thanks, appreciate it.
>
> Rajiv
>
> --- On Mon, 11/16/09, goutham patnaik <goutham.patnaik@gmail.com> wrote:
>
> From: goutham patnaik <goutham.patnaik@gmail.com>
> Subject: Re: Map-reduce sorting on multiple keys
> To: general@hadoop.apache.org
> Date: Monday, November 16, 2009, 11:08 AM
>
> Rajiv,
>
> You could write your own class which implements the WritableComparable
> interface and use this as your key class -  all u need to do is implement
> the write, readFields and compareTo methods - the map will then sort your
> keys using this method :
>
> public class TupleKey implements WritableComparable {
>  IntWritable k1;
>  IntWritable k2;
> .......
> }
>
> On Mon, Nov 16, 2009 at 9:13 AM, Rajiv Maheshwari <rajivm01@yahoo.com
> >wrote:
>
> > Hi everyone,
> >
> > I have a need to sort the output of map on 2 keys (key1, key2) - first on
> > key1, then on key2.
> >
> > Example:
> > key1  key2   values
> > -------------------------
> > 0001  0001  values...
> > 0001  0002  values...
> > 0002  0001  values...
> > 0002  0005  values...
> >
> >
> > I am thinking of the following solution approach:
> >
> > Define KEY = key1, key2   /* concatenate keys */. Override default
> > HashPartitioner and use only key1 in hashCode computation.
> >
> >
> > public class HashPartitioner<K2, V2> implements Partitioner<K2, V2> {
> >
> > public void configure(JobConf job) {}
> >
> > public int getPartition(K2 key, V2 value, int numPartitions) {
> >
> >     return (key.getKey1().hashCode() & Integer.MAX_VALUE) %
> numPartitions;
> >     }
> > }
> >
> > Would this work?
> >
> > Does anyone have any better ideas?
> >
> > Thanks much,
> > Rajiv
> >
> >
> >
> >
> >
>
>
>
>
>

--0016e6d7e744148457047881fae3--

From general-return-743-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 16 19:44:15 2009
Return-Path: <general-return-743-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60440 invoked from network); 16 Nov 2009 19:44:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Nov 2009 19:44:14 -0000
Received: (qmail 2916 invoked by uid 500); 16 Nov 2009 19:44:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2832 invoked by uid 500); 16 Nov 2009 19:44:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2808 invoked by uid 99); 16 Nov 2009 19:44:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Nov 2009 19:44:10 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mailinglists19@gmail.com designates 209.85.218.223 as permitted sender)
Received: from [209.85.218.223] (HELO mail-bw0-f223.google.com) (209.85.218.223)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Nov 2009 19:44:02 +0000
Received: by bwz23 with SMTP id 23so6141616bwz.29
        for <general@hadoop.apache.org>; Mon, 16 Nov 2009 11:43:42 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=jholVNvDeXLnJKBAHNEDrHwlWAtqquZ5x6bUhHoxb7k=;
        b=GsLU9ZM32bWKqhEB+312xJX9L1Rb1Zm/on1dR63JjwQj9bxfrqrudU8MIGlXuLmDu+
         XMRYSQV5CYirCQnGmydPmkCmqyto9dFK73r3LPQS8fI5OSXv6bpLqnVBe/zQJ9k01JlB
         VQw/YTcEl5YLTUkU/yP3U7SRQijfDqf+Yt/MY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=Khvtif9k37Nha1QRUpyWDh/f1ZlKZE3YzMAV10C6KIsqb/vLsiUkAkQS5Y7uoS9LkY
         B9OZJ7RVQdTSfVtrupCWViZy9nv7lZm0OjKWlk41jGdX9ItUrYnad3d0a8SWkU5FX9OO
         3TltFONPz8Hyt1Zh9zG169w2V3MNY8LgQtqF4=
MIME-Version: 1.0
Received: by 10.216.85.5 with SMTP id t5mr1135783wee.142.1258400621958; Mon, 
	16 Nov 2009 11:43:41 -0800 (PST)
Date: Mon, 16 Nov 2009 11:43:41 -0800
Message-ID: <1eabbac30911161143t4102be0brd0555eed21dd8572@mail.gmail.com>
Subject: Values returned by Map to Reducer
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d648610e97d20478823a97
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d648610e97d20478823a97
Content-Type: text/plain; charset=ISO-8859-1

Does Hadoop Mapreduce guarantee that the *values* returned by Mapper to the
Reducer are sorted?  Can I safely assume that?  Would it always be true - at
least for 'Text' type?

public void reduce(Text key, *Iterable<Text> values*, Context context

--0016e6d648610e97d20478823a97--

From general-return-744-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 16 20:17:40 2009
Return-Path: <general-return-744-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 71033 invoked from network); 16 Nov 2009 20:17:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Nov 2009 20:17:40 -0000
Received: (qmail 83474 invoked by uid 500); 16 Nov 2009 20:17:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83387 invoked by uid 500); 16 Nov 2009 20:17:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83377 invoked by uid 99); 16 Nov 2009 20:17:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Nov 2009 20:17:39 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [68.142.207.136] (HELO web32205.mail.mud.yahoo.com) (68.142.207.136)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 16 Nov 2009 20:17:29 +0000
Received: (qmail 86190 invoked by uid 60001); 16 Nov 2009 20:17:08 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1258402627; bh=iYgbiD7No/Du9mMVKVmMR3aDOaF7zXcOUWN1/TGkEfs=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=FcB/H0BaBd/PtGybnG31NlSuYy8KBwUnZRoCGt0oDjj6I5LVg0px3AGxAU91HElnHjCUmNhR7lhnFqfd8Ky1K8an0ZGWHcMnjTnlkkGA61fDASrwWdMf5klr+MnRdKewI536fBvc8R+R847MgZgwtxV50/De7CjAF4WCSTQHZSk=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=HZ+7dI9eXehRQgXF5w/nL9vfYdtpTplL1H72zdlLKogwqqjA/aiuik5vHIYIU+L+6dgllZuQhELc17bib/kqXaF6NeL1SshHsCpSCpH589Ew2CGGqbdayZMbQlw6avDbVilhmcKCyEW330S1dNfFJ5k2XadvWQ2R6+4kaRh39hg=;
Message-ID: <937882.86142.qm@web32205.mail.mud.yahoo.com>
X-YMail-OSG: Us17y.AVM1m9.7ZNK38cHjc.Q1B0YEmgfXf0dU4I61UvlG5BFHluGGpFyxyswTs4LIY2uJx9vTSv9XNXDePS7Ysu63.7GCYObhujPagvYnt4f_5ZJ_p7uABVgltye0mAzr8vD3fV0ARCIpotEZws8boHAxVz8fNvxrqlNaWJQzjBNTXYc_Ja5KW4zvPcoq6IB7m44ilVSOw9E7OQMaAQrvfFay7jufmAmuJgGvN479t_.Tx_W0a_7ZKz_4D_9uAMoPNno22wBXlSQ.pTMlGaL3dciDi_K._9.Pt5R2M.hXXmoEEBmTO0wcCFcuAALoBsT1QgECvj8eYC8YP68STSTE5YoWo-
Received: from [69.143.207.68] by web32205.mail.mud.yahoo.com via HTTP; Mon, 16 Nov 2009 12:17:07 PST
X-Mailer: YahooMailClassic/8.1.6 YahooMailWebService/0.7.361.4
Date: Mon, 16 Nov 2009 12:17:07 -0800 (PST)
From: Rajiv Maheshwari <rajivm01@yahoo.com>
Subject: Re: Map-reduce sorting on multiple keys
To: general@hadoop.apache.org
In-Reply-To: <1b3669280911161108p4fbe9d20i12dd564a121aecac@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1584925627-1258402627=:86142"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-1584925627-1258402627=:86142
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

On a second thought implementing WritableComparable will not help in my cas=
e. I forgot to mention that I want to use multiple reducers. Records with s=
ame key1 splitting across multiple reducers will break the logic.

If only 1 reducer is being used, I guess sorting on multiple keys can be ac=
complished just by concatenating the keys, unless one has the need to chang=
e the compare method.

Thanks,
Rajiv

--- On Mon, 11/16/09, goutham patnaik <goutham.patnaik@gmail.com> wrote:

From: goutham patnaik <goutham.patnaik@gmail.com>
Subject: Re: Map-reduce sorting on multiple keys
To: general@hadoop.apache.org
Date: Monday, November 16, 2009, 11:08 AM

Rajiv,

You could write your own class which implements the WritableComparable
interface and use this as your key class -=A0 all u need to do is implement
the write, readFields and compareTo methods - the map will then sort your
keys using this method :

public class TupleKey implements WritableComparable {
 IntWritable k1;
 IntWritable k2;
.......
}

On Mon, Nov 16, 2009 at 9:13 AM, Rajiv Maheshwari <rajivm01@yahoo.com>wrote=
:

> Hi everyone,
>
> I have a need to sort the output of map on 2 keys (key1, key2) - first on
> key1, then on key2.
>
> Example:
> key1=A0 key2=A0=A0=A0values
> -------------------------
> 0001=A0 0001=A0 values...
> 0001=A0 0002=A0 values...
> 0002=A0 0001=A0 values...
> 0002=A0 0005=A0 values...
>
>
> I am thinking of the following solution approach:
>
> Define KEY =3D key1, key2=A0=A0=A0/* concatenate keys */. Override defaul=
t
> HashPartitioner and use only key1 in hashCode computation.
>
>
> public class HashPartitioner<K2, V2> implements Partitioner<K2, V2> {
>
> public void configure(JobConf job) {}
>
> public int getPartition(K2 key, V2 value, int numPartitions) {
>
>=A0 =A0=A0=A0return (key.getKey1().hashCode() & Integer.MAX_VALUE) % numPa=
rtitions;
>=A0 =A0=A0=A0}
> }
>
> Would this work?
>
> Does anyone have any better ideas?
>
> Thanks much,
> Rajiv
>
>
>
>
>
=0A=0A=0A      
--0-1584925627-1258402627=:86142--

From general-return-745-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 16 20:46:20 2009
Return-Path: <general-return-745-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82104 invoked from network); 16 Nov 2009 20:46:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Nov 2009 20:46:20 -0000
Received: (qmail 47995 invoked by uid 500); 16 Nov 2009 20:46:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47930 invoked by uid 500); 16 Nov 2009 20:46:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47920 invoked by uid 99); 16 Nov 2009 20:46:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Nov 2009 20:46:19 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [67.195.22.98] (HELO web112120.mail.gq1.yahoo.com) (67.195.22.98)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 16 Nov 2009 20:46:17 +0000
Received: (qmail 76856 invoked by uid 60001); 16 Nov 2009 20:45:56 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1258404356; bh=/gYgTc33rpUIXgZ27hp5LHxz1sHznJMVjJYIp9EOrwE=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:MIME-Version:Content-Type; b=Wcz9/RqHfrmC4q4LFecNDF52f3/AhLRP3EwNcEjvUh85rSynUZzjyhivHKTDtidTXXPepiusBlYszmYhVDc9S+XZ5cQ2vjB0ddbRMGSxgwCqC0xiYUGu5f5OpOr5bvbkAowG0dv/9/0y9QQW5KqRDZAdVC+gtrIqZJ39Yya+mhg=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:MIME-Version:Content-Type;
  b=nD6Jjuk/UNKKIPFmZsGVTVveIK+CkJ46LFArUHD6O8c7vRSPZ/av5JSW8XMpL0c/dB7JVx/0/zisGzF3FeBo/RhbK+gvR1bvJtiPfeAFSmMKDJMrVQrhmKt5qeIOWrvX97G4a/gvYEcKPJuYhDiUugup1V8dmd0XwB/1vskFQ0U=;
Message-ID: <240158.76395.qm@web112120.mail.gq1.yahoo.com>
X-YMail-OSG: hp8Siq8VM1kEpcvoYdcdQSMbfBspM5zUo2tWbh0C46TjELxtuZDmXrxuzWFW3azIxp3IlwhJJLO_rPMa0luBFDRAJOCf2ZBfkXhECTyt6C2Y04uwXG5JxWtELrAoCRb0Dd_LV36J9F_zNgIA0X9m9IE29lRCWYOojllPjkHsyfr37Hb8hF3FxbgBL2p5_nRlljsjpDvapO627EiVrl7ULppl0Mlkp6iOd_v0Qy8_y6Qoc0J5Aa4HG6X52kTHaqxkug--
Received: from [64.172.17.3] by web112120.mail.gq1.yahoo.com via HTTP; Mon, 16 Nov 2009 12:45:56 PST
X-Mailer: YahooMailClassic/8.1.6 YahooMailWebService/0.7.361.4
Date: Mon, 16 Nov 2009 12:45:56 -0800 (PST)
From: Steve Gao <steve.gao@yahoo.com>
Subject: What's different of 1)namenode metadata 2)image and 3)edit logs?
To: general@hadoop.apache.org
Cc: core-user@hadoop.apache.org, core-dev@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1555827857-1258404356=:76395"

--0-1555827857-1258404356=:76395
Content-Type: text/plain; charset=us-ascii

What are the differences between the 3 concepts: (1) namenode metadata, (2) namenode image, and (3) edit logs?

BTW,I know the secondary namenode backups namenode data. 

I was told that there is another daemon which also backups meta data . Is that true? What is the daemon? Thanks.





      
--0-1555827857-1258404356=:76395--

From general-return-746-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 16 22:40:43 2009
Return-Path: <general-return-746-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33494 invoked from network); 16 Nov 2009 22:40:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Nov 2009 22:40:43 -0000
Received: (qmail 85368 invoked by uid 500); 16 Nov 2009 22:40:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85280 invoked by uid 500); 16 Nov 2009 22:40:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 49966 invoked by uid 99); 16 Nov 2009 22:22:28 -0000
X-ASF-Spam-Status: No, hits=3.8 required=10.0
	tests=RCVD_NUMERIC_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:cc:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:x-originalarrivaltime;
	b=PqSkAbMKpPLzN4r52FHYA+VCUpNckBJpodeRmr11/YH1IwP6uQv5kmVhTM49SX4t
User-Agent: Microsoft-Entourage/12.20.0.090605
Date: Mon, 16 Nov 2009 14:20:49 -0800
Subject: Re: What's different of 1)namenode metadata 2)image and 3)edit logs?
From: Boris Shkolnik <borya@yahoo-inc.com>
To: <common-dev@hadoop.apache.org>, <general@hadoop.apache.org>
CC: <core-user@hadoop.apache.org>, <core-dev@hadoop.apache.org>
Message-ID: <C7271041.25A45%borya@yahoo-inc.com>
Thread-Topic: What's different of 1)namenode metadata 2)image and 3)edit logs?
Thread-Index: AcpnCwzEjk9fEIVvN0y9SnaYYe7Kag==
In-Reply-To: <240158.76395.qm@web112120.mail.gq1.yahoo.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-OriginalArrivalTime: 16 Nov 2009 22:21:24.0138 (UTC) FILETIME=[21B658A0:01CA670B]
X-Virus-Checked: Checked by ClamAV on apache.org

NameNode metadata - is in memory data structures describing file system
structure (directories, files, and file's blocks).

NameNode image is on-disk representation of the metadata. (fsimage file).

Edit logs - is a journal that keeps all the events that cause changes in the
metadata and allows restoration in case of NameNode failure.

Secondary Namenode - runs checkpoints from time to time. During a checkpoint
edit logs are merged into metadata and saved to fsimage, which is pushed to
the NameNode.

Boris.


On 11/16/09 12:45 PM, "Steve Gao" <steve.gao@yahoo.com> wrote:

> What are the differences between the 3 concepts: (1) namenode metadata, (2)
> namenode image, and (3) edit logs?
> 
> BTW,I know the secondary namenode backups namenode data.
> 
> I was told that there is another daemon which also backups meta data . Is that
> true? What is the daemon? Thanks.
> 
> 
> 
> 
> 
>       


From general-return-747-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 17 03:29:46 2009
Return-Path: <general-return-747-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43824 invoked from network); 17 Nov 2009 03:29:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Nov 2009 03:29:46 -0000
Received: (qmail 44439 invoked by uid 500); 17 Nov 2009 03:29:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44294 invoked by uid 500); 17 Nov 2009 03:29:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44284 invoked by uid 99); 17 Nov 2009 03:29:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Nov 2009 03:29:43 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of hkumar.arora@gmail.com designates 209.85.223.188 as permitted sender)
Received: from [209.85.223.188] (HELO mail-iw0-f188.google.com) (209.85.223.188)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Nov 2009 03:29:32 +0000
Received: by iwn26 with SMTP id 26so4904976iwn.5
        for <general@hadoop.apache.org>; Mon, 16 Nov 2009 19:29:11 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=SFi1yYg5EDjOebU96xzuCF4BUcUo+CJ2fW15IhvVMBE=;
        b=t2HwNRUIZr0yqMeYDwsiXIMxpFNfYzV9+RjBD/lM1akBqXtfsQxH1pi2j85dGVXx8h
         zbNmcGgrWZ5z89Udq3IRDvd/CxDm8VNTG9V2XoHMvLMCUitNtWZ5WWp9M8tPgyj9wgo0
         V7GjcGuclwlCcKcevMOtDoXtaFRHVwiplmc2c=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=QorCiLUmsxghQs2HhGf2/gR02w3kTzyXFbW8SxLv+nWVw9SSY+lbuN3rJ0QAVVSk8M
         oELi1uaUNa5ojEDohRemBZCgg5GztzoFeC8oe1Eu95swtPxTpaDJ1nLAl4OLyYACx5HT
         xsLxrfD5HjaiMzdUmgyn1KPkUzN/rNqVxtfO0=
MIME-Version: 1.0
Received: by 10.231.6.79 with SMTP id 15mr4858896iby.36.1258428551664; Mon, 16 
	Nov 2009 19:29:11 -0800 (PST)
In-Reply-To: <1eabbac30911161143t4102be0brd0555eed21dd8572@mail.gmail.com>
References: <1eabbac30911161143t4102be0brd0555eed21dd8572@mail.gmail.com>
Date: Tue, 17 Nov 2009 12:29:11 +0900
Message-ID: <e497591c0911161929j3f78b2b9g200115c7df52b8c8@mail.gmail.com>
Subject: Re: Values returned by Map to Reducer
From: Harshit Kumar <hkumar.arora@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00151774123ecc1770047888ba87
X-Virus-Checked: Checked by ClamAV on apache.org

--00151774123ecc1770047888ba87
Content-Type: text/plain; charset=UTF-8

Yes.

H. Kumar
Phone(Mobile): +82-10-2892-9663
Phone(Office): +82-31-
skype: harshit900
Blog: http://harshitkumar.wordpress.com
Website: http:/kumarharmuscat.tripod.com


2009/11/17 Something Something <mailinglists19@gmail.com>

> Does Hadoop Mapreduce guarantee that the *values* returned by Mapper to the
> Reducer are sorted?  Can I safely assume that?  Would it always be true -
> at
> least for 'Text' type?
>
> public void reduce(Text key, *Iterable<Text> values*, Context context
>

--00151774123ecc1770047888ba87--

From general-return-748-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 17 03:48:15 2009
Return-Path: <general-return-748-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 48628 invoked from network); 17 Nov 2009 03:48:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Nov 2009 03:48:15 -0000
Received: (qmail 51328 invoked by uid 500); 17 Nov 2009 03:48:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51198 invoked by uid 500); 17 Nov 2009 03:48:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51188 invoked by uid 99); 17 Nov 2009 03:48:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Nov 2009 03:48:13 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of owen.omalley@gmail.com designates 209.85.160.50 as permitted sender)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Nov 2009 03:48:05 +0000
Received: by pwi6 with SMTP id 6so3990753pwi.29
        for <general@hadoop.apache.org>; Mon, 16 Nov 2009 19:47:43 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=+pxlUJN08x3Nw84eBlPq3a9hyDJY1SMMplEdrhU3rX4=;
        b=ZKe4NXXCIUNO9B92j0SjEvRmJkISHwFIPZpEzD28k3IfH8Ju28fy+JYs1hL9sFb+5T
         e7aI0vL2PXNP5uGIkm6sHFqSdImbyNYF8ztiZbVsPMfwCsLdKDHOZJhE8ufjaWwBfSRQ
         U2dhA2CRGwLQItibjZxFLW+HqeAWojakBhoEE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=W5gZVFS+Ky4h9b4joLtwE+pl91xGVfcm9Anh7vYKG+JwIz4F/nBdsic/qlebqswaff
         C4q9xyu4mqwTzr0eSocnP8+bPsmMkv9bTTl+aydJv1e1EPhoJ9dHn2ZDvmhXAAQLpBt1
         TvHwHOVsnLL04hF8As4fXsgWlnuQ6ErwG9c28=
MIME-Version: 1.0
Received: by 10.140.199.14 with SMTP id w14mr487314rvf.63.1258429663877; Mon, 
	16 Nov 2009 19:47:43 -0800 (PST)
In-Reply-To: <1eabbac30911161143t4102be0brd0555eed21dd8572@mail.gmail.com>
References: <1eabbac30911161143t4102be0brd0555eed21dd8572@mail.gmail.com>
Date: Mon, 16 Nov 2009 19:47:43 -0800
Message-ID: <5f7f740b0911161947h5e001e15q9a15816f972aa08e@mail.gmail.com>
Subject: Re: Values returned by Map to Reducer
From: "Owen O'Malley" <owen.omalley@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd250c2171bb0047888fdcd
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd250c2171bb0047888fdcd
Content-Type: text/plain; charset=UTF-8

On Mon, Nov 16, 2009 at 11:43 AM, Something Something <
mailinglists19@gmail.com> wrote:

> Does Hadoop Mapreduce guarantee that the *values* returned by Mapper to the
> Reducer are sorted?  Can I safely assume that?  Would it always be true -
> at
> least for 'Text' type?
>
> public void reduce(Text key, *Iterable<Text> values*, Context context
>

No, the values will *not* be sorted. In fact, it will be non-deterministic
between multiple runs of the job with the same input. The keys will always
be sorted. If you want the values to be sorted, you need to take additional
steps. Please look at the SecondarySort example. It shows exactly how to get
the values sorted in the order you desire.

-- Owen

--000e0cd250c2171bb0047888fdcd--

From general-return-749-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 17 04:11:24 2009
Return-Path: <general-return-749-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57253 invoked from network); 17 Nov 2009 04:11:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Nov 2009 04:11:24 -0000
Received: (qmail 62437 invoked by uid 500); 17 Nov 2009 04:11:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62329 invoked by uid 500); 17 Nov 2009 04:11:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62319 invoked by uid 99); 17 Nov 2009 04:11:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Nov 2009 04:11:21 +0000
X-ASF-Spam-Status: No, hits=-2.4 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of hkumar.arora@gmail.com designates 209.85.223.188 as permitted sender)
Received: from [209.85.223.188] (HELO mail-iw0-f188.google.com) (209.85.223.188)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Nov 2009 04:11:19 +0000
Received: by iwn26 with SMTP id 26so4925518iwn.5
        for <general@hadoop.apache.org>; Mon, 16 Nov 2009 20:10:58 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=FUME7HjkGUZtAnSh/GSypixY2+01H1yyEXXrGAtbFFY=;
        b=Ik3lfkGokZ47c7uvA/CRftYdiLC85niarf2XtSbuwWtGUFNjzfbsEdNcp2AAnbvHz9
         xtyfVFTrLIdigKCkackOFKDMfRpoMxyIFY+o7Z/m7DztQx6sTFUs9qxsNCNDxxL9Oi8C
         tzW+bwVi/yV8H9nli04hB4b+jd5eUHCkA3IGc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=J0WQbkDG1SF4FvdySmEvzQNQ/4RViuSo5d7ItqOu2kDxjiFoH7nNJJb2jHBxWIsZ0J
         04I4aoQyjk+uUUwkgmHHJ88JwpTeiOQ3bSqFK8RUjWzAQeAvsol9W8xQOOVFvwe1beLO
         HKa76nLvz547hvjcQcgdf2Azt8pUyFKUw5tQQ=
MIME-Version: 1.0
Received: by 10.231.158.205 with SMTP id g13mr960199ibx.30.1258431058069; Mon, 
	16 Nov 2009 20:10:58 -0800 (PST)
In-Reply-To: <5f7f740b0911161947h5e001e15q9a15816f972aa08e@mail.gmail.com>
References: <1eabbac30911161143t4102be0brd0555eed21dd8572@mail.gmail.com>
	 <5f7f740b0911161947h5e001e15q9a15816f972aa08e@mail.gmail.com>
Date: Tue, 17 Nov 2009 13:10:58 +0900
Message-ID: <e497591c0911162010sbb6bd32g59aafcb8019bceda@mail.gmail.com>
Subject: Re: Values returned by Map to Reducer
From: Harshit Kumar <hkumar.arora@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504501416d30c7680478895009

--00504501416d30c7680478895009
Content-Type: text/plain; charset=UTF-8

Oh Yes, you are right, i replied to this mail, saying yes. However, that yes
was for the keys which are sorted.
sorry for the confusion.


H. Kumar
Phone(Mobile): +82-10-2892-9663
Phone(Office): +82-31-
skype: harshit900
Blog: http://harshitkumar.wordpress.com
Website: http:/kumarharmuscat.tripod.com


2009/11/17 Owen O'Malley <owen.omalley@gmail.com>

> On Mon, Nov 16, 2009 at 11:43 AM, Something Something <
> mailinglists19@gmail.com> wrote:
>
> > Does Hadoop Mapreduce guarantee that the *values* returned by Mapper to
> the
> > Reducer are sorted?  Can I safely assume that?  Would it always be true -
> > at
> > least for 'Text' type?
> >
> > public void reduce(Text key, *Iterable<Text> values*, Context context
> >
>
> No, the values will *not* be sorted. In fact, it will be non-deterministic
> between multiple runs of the job with the same input. The keys will always
> be sorted. If you want the values to be sorted, you need to take additional
> steps. Please look at the SecondarySort example. It shows exactly how to
> get
> the values sorted in the order you desire.
>
> -- Owen
>

--00504501416d30c7680478895009--

From general-return-750-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 17 07:38:44 2009
Return-Path: <general-return-750-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 28370 invoked from network); 17 Nov 2009 07:38:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Nov 2009 07:38:44 -0000
Received: (qmail 56694 invoked by uid 500); 17 Nov 2009 07:38:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56615 invoked by uid 500); 17 Nov 2009 07:38:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56604 invoked by uid 99); 17 Nov 2009 07:38:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Nov 2009 07:38:42 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of prashantsethia007@gmail.com designates 209.85.216.204 as permitted sender)
Received: from [209.85.216.204] (HELO mail-px0-f204.google.com) (209.85.216.204)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Nov 2009 07:38:34 +0000
Received: by pxi42 with SMTP id 42so1255281pxi.5
        for <general@hadoop.apache.org>; Mon, 16 Nov 2009 23:38:12 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=0RLSxxuJoipEiwmZvTCjMOf2n2PFadQLtieYgxL+pH0=;
        b=cfTUeEUmO3j1aNZJtx/0bos9cFCORpQcxEtyhyf2fnjtrEydE8eI1SoD6BWczpMbYM
         xRE4+zAhiJwiPBu1G8w7JIcSPvePzMbaMo8c6rqx+KqCgUVzGCnBAyPZ7nMu9EG145st
         xItd2lM8FnpRPYdmXzvjKYxIjSTeAx3ChXHvM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=l5nM9L3pDzlq1Ayqd0dyuaEtraa2Hhyrq9rwCEzOnwkyLOS/DovZDNsQnLedD07Sz1
         foCrUdWlKQW4Ey7L5CIE8ET8DLBQLOJvjPaOiHN+uq8QCD++5gBhXyotGirHqr0qU/cE
         2acyFy4KZljKm5W1P3zT6vAtM6I32HuTjSWbc=
MIME-Version: 1.0
Received: by 10.115.100.4 with SMTP id c4mr17536292wam.13.1258443492846; Mon, 
	16 Nov 2009 23:38:12 -0800 (PST)
Date: Tue, 17 Nov 2009 13:08:12 +0530
Message-ID: <1eb371370911162338i1875ae15wefd5711eeb70ed8d@mail.gmail.com>
Subject: Release of Hadoop 0.21
From: prashant sethia <prashantsethia007@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64dde645c6e0504788c3536
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e64dde645c6e0504788c3536
Content-Type: text/plain; charset=ISO-8859-1

Hi,

http://issues.apache.org/jira/browse/HDFS-385

Has anyone any idea about the new release of Hadoop (0.21) ?

Regards,
Prashant.

--0016e64dde645c6e0504788c3536--

From general-return-751-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 17 11:29:37 2009
Return-Path: <general-return-751-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 92175 invoked from network); 17 Nov 2009 11:29:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Nov 2009 11:29:37 -0000
Received: (qmail 77643 invoked by uid 500); 17 Nov 2009 11:29:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77558 invoked by uid 500); 17 Nov 2009 11:29:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77548 invoked by uid 99); 17 Nov 2009 11:29:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Nov 2009 11:29:36 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of goutham.patnaik@gmail.com designates 209.85.210.194 as permitted sender)
Received: from [209.85.210.194] (HELO mail-yx0-f194.google.com) (209.85.210.194)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Nov 2009 11:29:33 +0000
Received: by yxe32 with SMTP id 32so4376718yxe.5
        for <general@hadoop.apache.org>; Tue, 17 Nov 2009 03:29:12 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :from:date:message-id:subject:to:content-type;
        bh=MsdBcCohGZ5DNm7H6TbCcP177O0J4CTaPM9IOxE897A=;
        b=O9gJh9CmzRJBm5maA8rYeaURbZ7ZS4Y0h/2i9YkZPuX+WFnlMI6Z2iy3fjabjUKlxh
         Dc6BxXNzu26xgxRbT3lJuGw8fOf+6V7r3ielFWRvypusLi7TBfHQMug+wc/XEfESvLSx
         9b1T3i7YPLi5Kr1d1qp8Aed65+5wOI0FAlQS8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=J4D7+EEC2SHm6VUfMyR33lkwHnrFaoIKreOMQ/014XI0wryIpjDAS1U46g7FrJGlNA
         lITAs+MWxCUmmwr7Jqu8vW8zCOtOqRAjY8Me+sayqRYGot5Et1YbEFGYlPp1XNE3waFp
         q72tfFPlbPgfQLaMmMSSXKGQX6JtT30d3dEfQ=
MIME-Version: 1.0
Received: by 10.150.28.27 with SMTP id b27mr5108497ybb.59.1258457352355; Tue, 
	17 Nov 2009 03:29:12 -0800 (PST)
In-Reply-To: <937882.86142.qm@web32205.mail.mud.yahoo.com>
References: <1b3669280911161108p4fbe9d20i12dd564a121aecac@mail.gmail.com> 
	<937882.86142.qm@web32205.mail.mud.yahoo.com>
From: goutham patnaik <goutham.patnaik@gmail.com>
Date: Tue, 17 Nov 2009 03:28:52 -0800
Message-ID: <1b3669280911170328t23a2ac7ck300c8bf2b9df5fa1@mail.gmail.com>
Subject: Re: Map-reduce sorting on multiple keys
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd756cc73e51304788f6f84

--000e0cd756cc73e51304788f6f84
Content-Type: text/plain; charset=ISO-8859-1

you want tuple keys (key1, key2) that have the same values for key1 to go to
the same reducer ? You can do this by writing your own partition function as
you mentioned in your previous email - the partition function can simply use
key1 to determine how to partition the data

to make sure that the data that reaches a reducer is sorted the way you want
it (first by key1 and then by key2), you can write your own key class that
implements WritableComparable

does that answer your question ?

Goutham

On Mon, Nov 16, 2009 at 12:17 PM, Rajiv Maheshwari <rajivm01@yahoo.com>wrote:

> On a second thought implementing WritableComparable will not help in my
> case. I forgot to mention that I want to use multiple reducers. Records with
> same key1 splitting across multiple reducers will break the logic.
>
> If only 1 reducer is being used, I guess sorting on multiple keys can be
> accomplished just by concatenating the keys, unless one has the need to
> change the compare method.
>
> Thanks,
> Rajiv
>
> --- On Mon, 11/16/09, goutham patnaik <goutham.patnaik@gmail.com> wrote:
>
> From: goutham patnaik <goutham.patnaik@gmail.com>
> Subject: Re: Map-reduce sorting on multiple keys
> To: general@hadoop.apache.org
> Date: Monday, November 16, 2009, 11:08 AM
>
> Rajiv,
>
> You could write your own class which implements the WritableComparable
> interface and use this as your key class -  all u need to do is implement
> the write, readFields and compareTo methods - the map will then sort your
> keys using this method :
>
> public class TupleKey implements WritableComparable {
>  IntWritable k1;
>  IntWritable k2;
> .......
> }
>
> On Mon, Nov 16, 2009 at 9:13 AM, Rajiv Maheshwari <rajivm01@yahoo.com
> >wrote:
>
> > Hi everyone,
> >
> > I have a need to sort the output of map on 2 keys (key1, key2) - first on
> > key1, then on key2.
> >
> > Example:
> > key1  key2   values
> > -------------------------
> > 0001  0001  values...
> > 0001  0002  values...
> > 0002  0001  values...
> > 0002  0005  values...
> >
> >
> > I am thinking of the following solution approach:
> >
> > Define KEY = key1, key2   /* concatenate keys */. Override default
> > HashPartitioner and use only key1 in hashCode computation.
> >
> >
> > public class HashPartitioner<K2, V2> implements Partitioner<K2, V2> {
> >
> > public void configure(JobConf job) {}
> >
> > public int getPartition(K2 key, V2 value, int numPartitions) {
> >
> >     return (key.getKey1().hashCode() & Integer.MAX_VALUE) %
> numPartitions;
> >     }
> > }
> >
> > Would this work?
> >
> > Does anyone have any better ideas?
> >
> > Thanks much,
> > Rajiv
> >
> >
> >
> >
> >
>
>
>
>
>

--000e0cd756cc73e51304788f6f84--

From general-return-752-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 17 12:08:58 2009
Return-Path: <general-return-752-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1715 invoked from network); 17 Nov 2009 12:08:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Nov 2009 12:08:58 -0000
Received: (qmail 8111 invoked by uid 500); 17 Nov 2009 12:08:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8035 invoked by uid 500); 17 Nov 2009 12:08:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8023 invoked by uid 99); 17 Nov 2009 12:08:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Nov 2009 12:08:57 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.85.221.188] (HELO mail-qy0-f188.google.com) (209.85.221.188)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Nov 2009 12:08:54 +0000
Received: by qyk26 with SMTP id 26so2997646qyk.5
        for <general@hadoop.apache.org>; Tue, 17 Nov 2009 04:08:33 -0800 (PST)
MIME-Version: 1.0
Received: by 10.229.39.69 with SMTP id f5mr1201079qce.107.1258459713609; Tue, 
	17 Nov 2009 04:08:33 -0800 (PST)
In-Reply-To: <1eb371370911162338i1875ae15wefd5711eeb70ed8d@mail.gmail.com>
References: <1eb371370911162338i1875ae15wefd5711eeb70ed8d@mail.gmail.com>
Date: Tue, 17 Nov 2009 04:08:33 -0800
Message-ID: <c9118d4c0911170408k57845b9ax949d60c41f12567e@mail.gmail.com>
Subject: Re: Release of Hadoop 0.21
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364275f731a04404788ffc83

--0016364275f731a04404788ffc83
Content-Type: text/plain; charset=ISO-8859-1

Hey Prashant,

There are 24 outstanding issues before 0.21 can be released:

* 23 from 0.21 branch:
http://issues.apache.org/jira/browse/HDFS/fixforversion/12314046
* 1 from the Append branch:
http://issues.apache.org/jira/browse/HDFS/fixforversion/12314142

If you see an issue you think you can tackle, please submit a patch!

Regards,
Jeff

On Mon, Nov 16, 2009 at 11:38 PM, prashant sethia <
prashantsethia007@gmail.com> wrote:

> Hi,
>
> http://issues.apache.org/jira/browse/HDFS-385
>
> Has anyone any idea about the new release of Hadoop (0.21) ?
>
> Regards,
> Prashant.
>

--0016364275f731a04404788ffc83--

From general-return-753-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 18 23:25:50 2009
Return-Path: <general-return-753-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39266 invoked from network); 18 Nov 2009 23:25:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Nov 2009 23:25:50 -0000
Received: (qmail 96229 invoked by uid 500); 18 Nov 2009 23:25:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96000 invoked by uid 500); 18 Nov 2009 23:25:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95971 invoked by uid 99); 18 Nov 2009 23:25:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Nov 2009 23:25:46 +0000
X-ASF-Spam-Status: No, hits=0.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_BL_SPAMCOP_NET,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [207.241.231.239] (HELO mail.archive.org) (207.241.231.239)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Nov 2009 23:25:34 +0000
Received: from localhost (localhost [127.0.0.1])
	by mail.archive.org (Postfix) with ESMTP id 56F022ADDC4
	for <general@hadoop.apache.org>; Wed, 18 Nov 2009 15:38:02 -0800 (PST)
Received: from mail.archive.org ([127.0.0.1])
	by localhost (mail.archive.org [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id TqOYsdYW92q0 for <general@hadoop.apache.org>;
	Wed, 18 Nov 2009 15:38:01 -0800 (PST)
Received: from [10.30.0.149] (unknown [208.70.28.201])
	by mail.archive.org (Postfix) with ESMTPSA id 83EB52ADDC3
	for <general@hadoop.apache.org>; Wed, 18 Nov 2009 15:38:01 -0800 (PST)
From: Alexis Rossi <alexis@archive.org>
Content-Type: multipart/alternative; boundary=Apple-Mail-224--1040846191
Subject: Internet Archive hiring web crawl Tech Lead
Date: Wed, 18 Nov 2009 15:25:11 -0800
Message-Id: <9814EB55-7BF3-4E76-8833-EEA64C147D71@archive.org>
To: general@hadoop.apache.org
Mime-Version: 1.0 (Apple Message framework v1076)
X-Mailer: Apple Mail (2.1076)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-224--1040846191
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
	charset=us-ascii;
	format=flowed;
	delsp=yes

...and someone with hadoop experience would be very welcome!

We're hiring for a web crawling engineer to be the tech lead on a new  
project to crawl and archive the entire Internet. The non-profit  
Internet Archive is a really fun, collaborative place to work, and has  
just relocated to a beautiful old converted church in the inner  
Richmond district of San Francisco.

We're looking for people who are interested in crawling, analysis, and  
working on big open source projects. Here's the job description:
http://www.archive.org/about/webjobs.php#wwcengineer

Thanks,

Alexis Rossi
alexis@archive.org
--Apple-Mail-224--1040846191--

From general-return-754-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 19 03:42:13 2009
Return-Path: <general-return-754-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6107 invoked from network); 19 Nov 2009 03:42:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Nov 2009 03:42:13 -0000
Received: (qmail 80775 invoked by uid 500); 19 Nov 2009 03:42:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80627 invoked by uid 500); 19 Nov 2009 03:42:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80615 invoked by uid 99); 19 Nov 2009 03:42:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Nov 2009 03:42:11 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [67.23.23.233] (HELO mail.osdbzine.net) (67.23.23.233)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Nov 2009 03:42:00 +0000
Received: from www.osdbzine.net (localhost [127.0.0.1])
	by mail.osdbzine.net (Postfix) with ESMTP id DCED21EC15A
	for <general@hadoop.apache.org>; Wed, 18 Nov 2009 21:42:29 -0600 (CST)
Received: from 67.191.167.24
        (SquirrelMail authenticated user editor@osdbzine.net)
        by www.osdbzine.net with HTTP;
        Wed, 18 Nov 2009 21:42:29 -0600 (CST)
Message-ID: <d8bc0d4a810a4a148afa5f4e2db43f4f.squirrel@www.osdbzine.net>
Date: Wed, 18 Nov 2009 21:42:29 -0600 (CST)
Subject: Nov/Dec 2009 Open Source Database Magazine Released!
From: "Keith Murphy" <editor@osdbzine.net>
To: general@hadoop.apache.org
User-Agent: SquirrelMail/1.4.15
MIME-Version: 1.0
Content-Type: text/plain;charset=iso-8859-1
Content-Transfer-Encoding: 8bit
X-Priority: 3 (Normal)
Importance: Normal
X-Virus-Checked: Checked by ClamAV on apache.org

Exciting news!

The next issue of Open Source Database Magazine (http://www.osdbzine.net)
is now available. This information-packed issue has over 60 pages of
information including:

* Firebird’s Road Trip and What’s New with 2.5
* Coding Corner: Trees – Where’s the Performance?
* PostgreSQL’s tsvector: Secret Sauce for Search Engines
* The Lab: The XtraBackup Program for MySQL – Part Two
* Drizzle – A Lightweight Database for the Web
* Kontrollbase: Enterprise grade MySQL monitoring and analytics
* Creating a Twitter Mashup with MongoDB
* Introducing LucidDB

Plus the usual news and views. All of this for $4.95.  It is simply the
biggest and the best issue we have ever released. Ready to sign up? Head
over to http://www.osdbzine.net/signup.html to register and then you can
download the new issue. Curious as to what this is all about? I just
posted an online addendum to the Drizzle article in our free content
section (http://www.osdbzine.net/free.html) that will give you a taste of
what you can expect.

Thanks to the contributors. You all did a great job and I appreciate it!

Sorry the website isn’t currently as polished as the magazine. While it
doesn’t look pretty, it is functioning. And now that the magazine is out I
can turn my attention to it again.  If you experience any problems let me
know at editor AT osdbzine.net.

thanks,

keith


-- 
Editor
Open Source Database Magazine
http://www.osdbzine.net
editor@osdbzine.net


From general-return-755-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 19 06:53:16 2009
Return-Path: <general-return-755-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49213 invoked from network); 19 Nov 2009 06:53:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Nov 2009 06:53:16 -0000
Received: (qmail 602 invoked by uid 500); 19 Nov 2009 06:53:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 510 invoked by uid 500); 19 Nov 2009 06:53:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 51242 invoked by uid 99); 18 Nov 2009 09:47:18 -0000
X-ASF-Spam-Status: No, hits=3.7 required=10.0
	tests=HTML_MESSAGE,SPF_PASS,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Subject: FW: how to run the mapreduce job from localfilesystem but in hadoop cluster(fully distributed mode)
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01CA6834.7F385CCE"
Date: Wed, 18 Nov 2009 15:39:46 +0545
Content-class: urn:content-classes:message
Message-ID: <325521DDB16A1F4B90085DCCE3FFD12B5AD63C@mailsrv.nepasoft.com.np>
X-MimeOLE: Produced By Microsoft Exchange V6.5
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: how to run the mapreduce job from localfilesystem but in hadoop cluster(fully distributed mode)
Thread-Index: AcpoNFli17vyzowCQAySd/Spww0KqgAAHhoQ
From: "Roshan Karki" <roshan.karki@nepasoft.com.np>
To: <general@hadoop.apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

------_=_NextPart_001_01CA6834.7F385CCE
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

=20

=20

From: Roshan Karki=20
Sent: Wednesday, November 18, 2009 3:34 PM
To: 'common-user@hadoop.apache.org'
Subject: how to run the mapreduce job from localfilesystem but in hadoop
cluster(fully distributed mode)

=20

Hi ,

      I  have setup hadoop in a fully distributed mode(hadoop cluster).
The default file sytem is hdfs.I have two nodes cluster. The map reduce
job works fine when I give inputfile from hdfs location and output is
also generated in hdfs when running WordCount example. My
hadoop-site.xml looks like this:

<?xml version=3D"1.0"?>

<?xml-stylesheet type=3D"text/xsl" href=3D"configuration.xsl"?>

<!-- Put site-specific property overrrides in this file. -->

=20

<configuration>

  <property>

    <name>fs.default.name</name>

    <value>hdfs://server1.example.com</value>

  </property>

  <property>

    <name>mapred.job.tracker</name>

    <value>server1.example.com:9001</value>

  </property>

  <property>

    <name>dfs.data.dir</name>

    <value>/home/admin/hadoop/dfs/data</value>

  </property>

  <property>

    <name>dfs.name.dir</name>

    <value>/home/admin/hadoop/dfs/name</value>

  </property>

  <property>

    <name>hadoop.tmp.dir</name>

    <value>/tmp/hadoop</value>

  </property>

  <property>

    <name>mapred.system.dir</name>

    <value>/hadoop/mapred/system</value>

  </property>

  <property>

    <name>dfs.replication</name>

    <value>2</value>

  </property>

</configuration>

=20

The problem is that I don't want to give hdfs location for fileinput and
want to generated output in localfilesystem also giving the inputfile
from local but I want to do this in hadoop fully distributed mode
configuration(keeping the hadoop cluster alive).

Giving default filesystem other than hdfs doesn't run cluster.so

Is there any way to achieve this?

I also have nfs server running. All the nodes in hadoop cluster are also
the client of nfs,so they can share the exported /home/admin directory.

Does nfs help in this regard?

Do I have to change my hadoop-site.xml.If so what would it looks like?

Any help will be heartly appreciated.

I am looking forward for the kind response.

=20

=20


------_=_NextPart_001_01CA6834.7F385CCE--

From general-return-756-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 19 06:53:30 2009
Return-Path: <general-return-756-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49314 invoked from network); 19 Nov 2009 06:53:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Nov 2009 06:53:30 -0000
Received: (qmail 1923 invoked by uid 500); 19 Nov 2009 06:53:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1839 invoked by uid 500); 19 Nov 2009 06:53:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 34995 invoked by uid 99); 19 Nov 2009 00:50:15 -0000
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Message-Id: <8F1CCF11-3A9E-40FB-9399-A643C45CBCC6@schwartz-omalley.com>
From: Owen O'Malley <owen@schwartz-omalley.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Welcome to Zheng Shao to the Hadoop PMC
Date: Wed, 18 Nov 2009 16:49:28 -0800
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

The Hadoop PMC would like to welcome Zheng Shao as the newest member.  
Zheng started on MapReduce, but has spent a lot of time helping the  
Hive subproject. Zheng, please add yourself to the Hadoop TLP  
website's list of the PMC.

Thanks for your hard work on behalf of Hadoop,
    Owen

From general-return-757-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 19 11:19:07 2009
Return-Path: <general-return-757-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9540 invoked from network); 19 Nov 2009 11:19:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Nov 2009 11:19:07 -0000
Received: (qmail 75360 invoked by uid 500); 19 Nov 2009 11:19:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75294 invoked by uid 500); 19 Nov 2009 11:19:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75284 invoked by uid 99); 19 Nov 2009 11:19:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Nov 2009 11:19:06 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Nov 2009 11:18:56 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id AA2C51BA5F9
	for <general@hadoop.apache.org>; Thu, 19 Nov 2009 11:18:35 +0000 (GMT)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id sezNhB5ZE24p for <general@hadoop.apache.org>;
	Thu, 19 Nov 2009 11:18:35 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by colossus.hpl.hp.com (Postfix) with ESMTPS id 3E0531BA563
	for <general@hadoop.apache.org>; Thu, 19 Nov 2009 11:18:35 +0000 (GMT)
MailScanner-NULL-Check: 1259234304.95484@2WvvJ/O+eIy7Qsd8ZpmkQw
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id nAJBINwD001243
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Thu, 19 Nov 2009 11:18:24 GMT
Message-ID: <4B05297F.4070305@apache.org>
Date: Thu, 19 Nov 2009 11:18:23 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: FW: how to run the mapreduce job from localfilesystem but in
 hadoop cluster(fully distributed mode)
References: <325521DDB16A1F4B90085DCCE3FFD12B5AD63C@mailsrv.nepasoft.com.np>
In-Reply-To: <325521DDB16A1F4B90085DCCE3FFD12B5AD63C@mailsrv.nepasoft.com.np>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: nAJBINwD001243
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

Roshan Karki wrote:
>  

>  
> 
> The problem is that I don't want to give hdfs location for fileinput and
> want to generated output in localfilesystem also giving the inputfile
> from local but I want to do this in hadoop fully distributed mode
> configuration(keeping the hadoop cluster alive).
> 


Either upload the data files to the HDFS filestore before the run , or 
use file: URLs and make sure that your files are visible on every task 
tracker by mounting the same network drive on the same path, such as 
file:/mount/nfs/server1/work/data


From general-return-758-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 20 06:20:40 2009
Return-Path: <general-return-758-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12517 invoked from network); 20 Nov 2009 06:20:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Nov 2009 06:20:40 -0000
Received: (qmail 63292 invoked by uid 500); 20 Nov 2009 06:20:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63217 invoked by uid 500); 20 Nov 2009 06:20:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63207 invoked by uid 99); 20 Nov 2009 06:20:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Nov 2009 06:20:38 +0000
X-ASF-Spam-Status: No, hits=-2.0 required=5.0
	tests=AWL,BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yangxiao9901@gmail.com designates 209.85.222.198 as permitted sender)
Received: from [209.85.222.198] (HELO mail-pz0-f198.google.com) (209.85.222.198)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Nov 2009 06:20:35 +0000
Received: by pzk36 with SMTP id 36so2131925pzk.5
        for <general@hadoop.apache.org>; Thu, 19 Nov 2009 22:20:15 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=ySycgGayCOAT3hdz16dTejZsD3p9zKlHycjiQ67RN7M=;
        b=iCISKduTLoZUkFwmUaMk9GXSllitfKLe8qgaeqxPJRMmIcjJ+x2h75osPQxQnJluud
         W2XoenG0W69E7/4R/i84z52TtFBi0gb4lxG9p+BcKs/v5dLMQ7lrPJZ0klBZJi7qQLHq
         wWXo12dYvjCMojJF+7Nri0BbutSe9BCgw0PZM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=aPRklRqwW8cS66kk4wvGjXrb8hGh9RwdgQ7TaTl842Aixl6+dNFQxEQfc1m6U0rLUY
         aURR1cM3vGFdtIwy9y9qvWcjzwFrAiFav738DgLZoRjiLmvQD/NcwHJYFz10bPlajdSk
         drVw24/M3cQoYufGUjN6Zda1fIZXaWuHrg474=
MIME-Version: 1.0
Received: by 10.140.141.4 with SMTP id o4mr68644rvd.257.1258698014785; Thu, 19 
	Nov 2009 22:20:14 -0800 (PST)
Date: Fri, 20 Nov 2009 14:20:14 +0800
Message-ID: <5a921af20911192220q65188c86o3eda579da6943da2@mail.gmail.com>
Subject: Error when putting too large folder into Hadoop
From: xiao yang <yangxiao9901@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hi, all

I'm using hadoop-0.19.1, which is included in nutch 1.0. The HDFS is
deployed on 12 nodes, and they work fine.
Now I want to put a folder which is about 100GB large into hadoop.
Here is the command:
bin/hadoop fs -put /srcfolder /pub
When the source folder is less than 10GB, everything is OK. But if the
folder is too large (How large I haven't figured out yet). It will
fail in the process.
What's the problem? How can I fix it?

Following are the error messages on the screen:


2009-11-20 11:31:09,170 INFO  DFSClient - Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink 10.214.10.140:50010
2009-11-20 11:31:09,282 INFO  DFSClient - Abandoning block
blk_4524254776889347494_31880
2009-11-20 11:31:15,298 INFO  DFSClient - Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink 10.214.10.141:50010
2009-11-20 11:31:15,298 INFO  DFSClient - Abandoning block
blk_-6096773602253771497_31884
2009-11-20 11:31:22,105 INFO  DFSClient - Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink 10.214.10.141:50010
2009-11-20 11:31:22,106 INFO  DFSClient - Abandoning block
blk_7476142191552180978_31895
2009-11-20 11:31:28,109 INFO  DFSClient - Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink 10.214.10.145:50010
2009-11-20 11:31:28,109 INFO  DFSClient - Abandoning block
blk_1702775959905886525_31903
2009-11-20 11:31:34,114 INFO  DFSClient - Exception in
createBlockOutputStream java.io.IOException: Could not read from
stream
2009-11-20 11:31:34,114 INFO  DFSClient - Abandoning block
blk_-6529348247759206930_31906
2009-11-20 11:31:47,507 INFO  DFSClient - Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink 10.214.10.144:50010
2009-11-20 11:31:47,508 INFO  DFSClient - Abandoning block
blk_8773438940339692415_32051
2009-11-20 11:31:53,511 INFO  DFSClient - Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink 10.214.10.139:50010
2009-11-20 11:31:53,511 INFO  DFSClient - Abandoning block
blk_-2934984271889679562_32063
2009-11-20 11:31:59,515 INFO  DFSClient - Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink 10.214.10.145:50010
2009-11-20 11:31:59,515 INFO  DFSClient - Abandoning block
blk_-9114974684513095559_32065
2009-11-20 11:32:05,519 INFO  DFSClient - Exception in
createBlockOutputStream java.io.IOException: Bad connect ack with
firstBadLink 10.214.10.145:50010
2009-11-20 11:32:05,519 INFO  DFSClient - Abandoning block
blk_8604459663942116813_32071
2009-11-20 11:32:11,580 WARN  DFSClient - DataStreamer Exception:
java.io.IOException: Unable to create new block.
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2722)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1996)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)

2009-11-20 11:32:11,580 WARN  DFSClient - Error Recovery for block
blk_8604459663942116813_32071 bad datanode[1] nodes == null
2009-11-20 11:32:11,580 WARN  DFSClient - Could not get block
locations. Source file "/pub/comic/_Incoming_/1.wma" - Aborting...
put: Bad connect ack with firstBadLink 10.214.10.145:50010

From general-return-759-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Nov 21 07:06:07 2009
Return-Path: <general-return-759-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 99945 invoked from network); 21 Nov 2009 07:06:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Nov 2009 07:06:07 -0000
Received: (qmail 88258 invoked by uid 500); 21 Nov 2009 07:06:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88174 invoked by uid 500); 21 Nov 2009 07:06:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88162 invoked by uid 99); 21 Nov 2009 07:06:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Nov 2009 07:06:05 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [209.191.125.34] (HELO web38403.mail.mud.yahoo.com) (209.191.125.34)
    by apache.org (qpsmtpd/0.29) with SMTP; Sat, 21 Nov 2009 07:05:55 +0000
Received: (qmail 72558 invoked by uid 60001); 21 Nov 2009 07:05:34 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1258787134; bh=WXqKcB2ouaRNHIRTbEBArAHJcgMROYZtFmjoCneC8kA=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=yKKIahB1OTh+Ib1tTPpiaJ4k1g5jCCNvJmc9MIXiHfzIOjyPUY8ugu3LshTsFZJUzfJdfvDqjF1U7SFROWJZ0B7mNTAXGYnREpAw3fGQIKVgXjeTRO4o436GTnKaosduy4qrHUOc5VbswSU53r4zqAhoJG0Agigow+OmSxmS6u0=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=3l35LyPNMtXNuzF2fR06ZjI7/iBOv3wJp8orD4p2B1dBmxsX6wN7mvWpY1BJPTvazdiDVYb0fqIN7k/RFfPFEedrnsTG4V+99fSFk6teNe1H8Z5O+peTYur8z0JxiXTnzYVAy+76tYsYJDE0OluELVXsLGoSxvNfed/hlboYFG0=;
Message-ID: <235810.71840.qm@web38403.mail.mud.yahoo.com>
X-YMail-OSG: _XYKaSQVM1ltiyq7LD0Jmtr3w4r0t8zNmPo9arXG_SD1zEEFhgKgXJZRDsrObCbmENh1s1W17RElVYeKtfUwb9p06XaOWlePVzyGxvlhwpQc.KEiywsICo27GRhkcRZMoBeDIbBW4hwhGBjfFsr6eGQHM_46X_afb4yM9AxsJWRYKFXyrUt9D7ER3a6yDIUYHnfMzaOci6abyldaKmCjgTthiOSmxLJVFVi3X.x.q5_aYcFKzaA9MGm2.qfowXQe1qYnak.sgWQIfW_heHlRlWjEaK4vuRfq1wAFCwkV_0rl3_OjCBAdwfot.sg-
Received: from [75.69.131.160] by web38403.mail.mud.yahoo.com via HTTP; Fri, 20 Nov 2009 23:05:33 PST
X-Mailer: YahooMailRC/211.6 YahooMailWebService/0.8.100.260964
Date: Fri, 20 Nov 2009 23:05:33 -0800 (PST)
From: himanshu chandola <himanshu_coolguy@yahoo.com>
Subject: reduce copier failed
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,
I have a reduce copier that dies away everytime my 10 iterations of map and reduce pass through:
java.io.IOException: attempt_200910061342_0139_r_000001_0The reduce copier failed
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
    at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2210)


Finally the job dies due to this. I thought that it could happen due to low diskspace, but I've estimated the disk space required by the reduce output and the current free disk space is enough to accomodate this. 

If anyone could point out to anything it'll be great! I'm running cloudera's build :Version: 0.18.3-14.


Thanks



 Morpheus: Do you believe in fate, Neo?
Neo: No.
Morpheus: Why Not?
Neo: Because I don't like the idea that I'm not in control of my life.



      

From general-return-760-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Nov 22 00:58:00 2009
Return-Path: <general-return-760-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91799 invoked from network); 22 Nov 2009 00:58:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Nov 2009 00:58:00 -0000
Received: (qmail 54890 invoked by uid 500); 22 Nov 2009 00:57:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54797 invoked by uid 500); 22 Nov 2009 00:57:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54787 invoked by uid 99); 22 Nov 2009 00:57:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 Nov 2009 00:57:58 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mailinglists19@gmail.com designates 209.85.219.214 as permitted sender)
Received: from [209.85.219.214] (HELO mail-ew0-f214.google.com) (209.85.219.214)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 Nov 2009 00:57:56 +0000
Received: by ewy6 with SMTP id 6so790948ewy.29
        for <general@hadoop.apache.org>; Sat, 21 Nov 2009 16:57:35 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=LK/A2khQuQzgQ05JoQCu8gBXqcCVA/BWStzmlDFyEmE=;
        b=P2tIf01/jQYDTokAgUejwYWdKxAbzSCIiIPdQqv4Mz6Q820gGN/W5Y5LqDqGsSypy5
         rf9adXS8edQUIbQ0Lh4AQiLJlwjpUKR6dZeWn4/1LRfasnp9M/i/FrYJ6YY7GuyYoY41
         dvBgc5puGGoDpcqqX2azsRryhjTFhjGO5HYeM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=LBxv9Rjia0xXLciRf3v2FH7FSznSulzQ3zGztQtDkhaJ/Ud5tFntGBDBmHAAMzQJG7
         4XkV1S/FFkKJC+7/6bF2K4BA1ROmdNLy+eqqyDBRRGU4u63ledY8hFAyzHhXCql89TtI
         tlee1ngonajZI7f6ZHRtPkObOk/yQmyMTmErc=
MIME-Version: 1.0
Received: by 10.216.89.12 with SMTP id b12mr983975wef.93.1258851454993; Sat, 
	21 Nov 2009 16:57:34 -0800 (PST)
Date: Sat, 21 Nov 2009 16:57:34 -0800
Message-ID: <1eabbac30911211657i7812c67bs3a1565b0537b418f@mail.gmail.com>
Subject: Starting Hadoop cluster on EC2
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6da2e7eccc7150478eb3156

--0016e6da2e7eccc7150478eb3156
Content-Type: text/plain; charset=ISO-8859-1

Trying to start a Hadoop cluster on EC2.  (Yes, Cloudera's distribution
works well, but trying to do this myself so I can learn what's happening
behind the scene.)

I have a Master & a Slave.  When I start HDFS on the master, I get a message
saying "10.xxx.xxx.xxx (Permission denied)" - where 10.xxx is IP address of
the slave.

Basic problem (I think) is that I can't ssh from the master EC2 instance to
the Slave EC2 instance.  What's the best way to fix it?  I think I need the
"Key Pair" file on my master.  I have a key pair on my local machine, but
how do I transfer it to the EC2 machine?  (I know, I know, I agree.. I am
dumb :)  Should I FTP it?

Please help.  Thanks.

--0016e6da2e7eccc7150478eb3156--

From general-return-761-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Nov 22 01:30:55 2009
Return-Path: <general-return-761-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96194 invoked from network); 22 Nov 2009 01:30:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Nov 2009 01:30:55 -0000
Received: (qmail 60874 invoked by uid 500); 22 Nov 2009 01:30:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60777 invoked by uid 500); 22 Nov 2009 01:30:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60767 invoked by uid 99); 22 Nov 2009 01:30:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 Nov 2009 01:30:54 +0000
X-ASF-Spam-Status: No, hits=-1.5 required=5.0
	tests=AWL,BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.18.222.48] (HELO smtp2.4emm.com) (69.18.222.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 Nov 2009 01:30:51 +0000
Received: from EX2K7VS03.4emm.local ([192.168.160.203]) by HUB02.4emm.local
 ([192.168.161.133]) with mapi; Sat, 21 Nov 2009 20:29:55 -0500
From: Jason Lotz <jason.lotz@explorysmedical.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Sat, 21 Nov 2009 20:30:28 -0500
Subject: Re: reduce copier failed
Thread-Topic: reduce copier failed
Thread-Index: AcprE0sVcF4emhi4SYqdDdP4J3hCDQ==
Message-ID: <117E5534-0D2C-4B7B-880F-71BBCEB08C68@explorysmedical.com>
References: <235810.71840.qm@web38403.mail.mud.yahoo.com>
In-Reply-To: <235810.71840.qm@web38403.mail.mud.yahoo.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Are you writing to /tmp?  Depending on your OS and configuration, you =20
might be dealing with an issue where the folder does not have the =20
capability of growing to the full size of the available disk space.  =20
This is a common default config for various Linux distros for the /tmp =20
directory.

Jason

On Nov 21, 2009, at 2:05 AM, himanshu chandola wrote:

> Hi,
> I have a reduce copier that dies away everytime my 10 iterations of =20
> map and reduce pass through:
> java.io.IOException: attempt_200910061342_0139_r_000001_0The reduce =20
> copier failed
>    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
>    at org.apache.hadoop.mapred.TaskTracker$Child.main=20
> (TaskTracker.java:2210)
>
>
> Finally the job dies due to this. I thought that it could happen due =20
> to low diskspace, but I've estimated the disk space required by the =20
> reduce output and the current free disk space is enough to =20
> accomodate this.
>
> If anyone could point out to anything it'll be great! I'm running =20
> cloudera's build :Version: 0.18.3-14.
>
>
> Thanks
>
>
>
> Morpheus: Do you believe in fate, Neo?
> Neo: No.
> Morpheus: Why Not?
> Neo: Because I don't like the idea that I'm not in control of my life.
>
>
>
>


From general-return-762-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Nov 22 02:12:48 2009
Return-Path: <general-return-762-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 2145 invoked from network); 22 Nov 2009 02:12:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Nov 2009 02:12:48 -0000
Received: (qmail 74179 invoked by uid 500); 22 Nov 2009 02:12:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74097 invoked by uid 500); 22 Nov 2009 02:12:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74087 invoked by uid 99); 22 Nov 2009 02:12:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 Nov 2009 02:12:47 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of zjffdu@gmail.com designates 209.85.216.201 as permitted sender)
Received: from [209.85.216.201] (HELO mail-px0-f201.google.com) (209.85.216.201)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 Nov 2009 02:12:38 +0000
Received: by pxi39 with SMTP id 39so3082696pxi.2
        for <general@hadoop.apache.org>; Sat, 21 Nov 2009 18:12:17 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=B6deWrktnt7jbMPbJCovdCoewzDId1coR39o/Bo7YSs=;
        b=tUDgPMl27jg8Ncimi6NGHf0KWmc5LVmdQVUyqQ9JTraEGBp3oKGPrKYDS+F5HZOpZT
         VcWYzc7IvL5WS5Mb9u7uXBkjJnzdIkbZPowep2bvjcvP4AwyVahtLBLwXw0O1IUFchs6
         oA90xYLZGcay1qfZXnsiIH6jUeMLMZM93tKEc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=ITR2vmvzKmKZ7B5fAytHpT6MUMnBK/oZxTKpnqLKtef7iMCJ5aTVeKkwZ0gJZR/oVv
         lfaLkPEuvCYoRU2yA3Sq/Duc69CO/KXNrHn/RdKUMC4JnWkFzKBh/zbhGdtFVcqr95Cr
         K4m7b4YQX/BoVXXEWZZDenf2ySvSkskXiEZJs=
MIME-Version: 1.0
Received: by 10.142.60.3 with SMTP id i3mr336302wfa.147.1258855937551; Sat, 21 
	Nov 2009 18:12:17 -0800 (PST)
In-Reply-To: <1eabbac30911211657i7812c67bs3a1565b0537b418f@mail.gmail.com>
References: <1eabbac30911211657i7812c67bs3a1565b0537b418f@mail.gmail.com>
Date: Sat, 21 Nov 2009 18:12:17 -0800
Message-ID: <8211a1320911211812q64a7fb85jcdcda10be650d4ef@mail.gmail.com>
Subject: Re: Starting Hadoop cluster on EC2
From: Jeff Zhang <zjffdu@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502ad14fb2e510478ec3cdd
X-Virus-Checked: Checked by ClamAV on apache.org

--00504502ad14fb2e510478ec3cdd
Content-Type: text/plain; charset=UTF-8

You should scp the key-pair to EC2 machine

Jeff Zhang

On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
mailinglists19@gmail.com> wrote:

> Trying to start a Hadoop cluster on EC2.  (Yes, Cloudera's distribution
> works well, but trying to do this myself so I can learn what's happening
> behind the scene.)
>
> I have a Master & a Slave.  When I start HDFS on the master, I get a
> message
> saying "10.xxx.xxx.xxx (Permission denied)" - where 10.xxx is IP address of
> the slave.
>
> Basic problem (I think) is that I can't ssh from the master EC2 instance to
> the Slave EC2 instance.  What's the best way to fix it?  I think I need the
> "Key Pair" file on my master.  I have a key pair on my local machine, but
> how do I transfer it to the EC2 machine?  (I know, I know, I agree.. I am
> dumb :)  Should I FTP it?
>
> Please help.  Thanks.
>

--00504502ad14fb2e510478ec3cdd--

From general-return-763-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Nov 22 03:12:09 2009
Return-Path: <general-return-763-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 10892 invoked from network); 22 Nov 2009 03:12:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Nov 2009 03:12:09 -0000
Received: (qmail 94475 invoked by uid 500); 22 Nov 2009 03:12:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94339 invoked by uid 500); 22 Nov 2009 03:12:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94326 invoked by uid 99); 22 Nov 2009 03:12:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 Nov 2009 03:12:07 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [209.191.125.38] (HELO web38407.mail.mud.yahoo.com) (209.191.125.38)
    by apache.org (qpsmtpd/0.29) with SMTP; Sun, 22 Nov 2009 03:11:57 +0000
Received: (qmail 14088 invoked by uid 60001); 22 Nov 2009 03:11:36 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1258859496; bh=kOAZuAXw9at3K2MXhinE+PEaRe+umwLlCc8LlXnr3RU=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=SbS89G9Qonp9OE3EcD8ViNSkwr6BQ8R72XtOHVZYjoKjyqP+txnfIHOqUCVTcAOdMzwlk1upi7SREYc8pUKjE7nEq9Fmbs1ikVebvE8IcSsejd9uU48kwZnMzCFhk0p/vDjfqB/SoS9xIgNtJgQlv8X16qAeLEtChalECO46Q7U=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=LSFlaQAQmyO9cjMKsVzwPHzSigFzXZDoILlPhbzTRbg4gOyXcPQRkqIvXY0R/pCtUBSo9aEmnjCGtK1/H2TAJVFpMdbBQPdur2FxY7a5jpgULrMRaerEKbJSfPlYramQjAyYiHkSqAs5cxvTn3C7L6CmkywwdAJtZ+wM3ch1QQ4=;
Message-ID: <172592.13790.qm@web38407.mail.mud.yahoo.com>
X-YMail-OSG: AzNYOMQVM1mi721khBZrSh41OKMviyfSXqkE4YrWa1RU1KWHmrOb6aZ9AKcXzM0ZcZ_L9eh2X9V5.PXTC5d8QDby.qdkqv4M1dvD6fZ1SbpmK.KPY.75LcKqo8BP9GRRN1gJS7YT5TZWbJNygU055yQP_E2I_PgCVyG3LtS12G7Zx2O0xirof8rBnb4Q38Gw0hYR0YQH2uV3EApr_DLYWcUSc3WQAUS4pOOkG4m4fz0CU46tEET37Pws_pgExVzxhIpp89AjeIIDwtPrZCswqAao.RLujNy1YU0RxGMKantSUQwml2SWwhmyA3MYWqG6ztwu5g5w_2lH1IOwo78GVpLp6OLL7ZdLJa.Ovx6D2OSI1S9cChKOOudq
Received: from [75.69.131.160] by web38407.mail.mud.yahoo.com via HTTP; Sat, 21 Nov 2009 19:11:36 PST
X-Mailer: YahooMailRC/211.6 YahooMailWebService/0.8.100.260964
References: <235810.71840.qm@web38403.mail.mud.yahoo.com> <117E5534-0D2C-4B7B-880F-71BBCEB08C68@explorysmedical.com>
Date: Sat, 21 Nov 2009 19:11:36 -0800 (PST)
From: himanshu chandola <himanshu_coolguy@yahoo.com>
Subject: Re: reduce copier failed
To: general@hadoop.apache.org
In-Reply-To: <117E5534-0D2C-4B7B-880F-71BBCEB08C68@explorysmedical.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org

I am writing to a particular partition in the nodes of my cluster. 


Is that the only reason that this could happen ? Or does anyone know of any other cases ? 


Thanks.

 Morpheus: Do you believe in fate, Neo?
Neo: No.
Morpheus: Why Not?
Neo: Because I don't like the idea that I'm not in control of my life.



----- Original Message ----
From: Jason Lotz <jason.lotz@explorysmedical.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Sent: Sat, November 21, 2009 8:30:28 PM
Subject: Re: reduce copier failed

Are you writing to /tmp?  Depending on your OS and configuration, you  
might be dealing with an issue where the folder does not have the  
capability of growing to the full size of the available disk space.  
This is a common default config for various Linux distros for the /tmp  
directory.

Jason

On Nov 21, 2009, at 2:05 AM, himanshu chandola wrote:

> Hi,
> I have a reduce copier that dies away everytime my 10 iterations of  
> map and reduce pass through:
> java.io.IOException: attempt_200910061342_0139_r_000001_0The reduce  
> copier failed
>    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:255)
>    at org.apache.hadoop.mapred.TaskTracker$Child.main 
> (TaskTracker.java:2210)
>
>
> Finally the job dies due to this. I thought that it could happen due  
> to low diskspace, but I've estimated the disk space required by the  
> reduce output and the current free disk space is enough to  
> accomodate this.
>
> If anyone could point out to anything it'll be great! I'm running  
> cloudera's build :Version: 0.18.3-14.
>
>
> Thanks
>
>
>
> Morpheus: Do you believe in fate, Neo?
> Neo: No.
> Morpheus: Why Not?
> Neo: Because I don't like the idea that I'm not in control of my life.
>
>
>
>


      

From general-return-764-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Nov 22 05:39:01 2009
Return-Path: <general-return-764-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 26458 invoked from network); 22 Nov 2009 05:39:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Nov 2009 05:39:01 -0000
Received: (qmail 26445 invoked by uid 500); 22 Nov 2009 05:39:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26356 invoked by uid 500); 22 Nov 2009 05:38:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26346 invoked by uid 99); 22 Nov 2009 05:38:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 Nov 2009 05:38:59 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of zjffdu@gmail.com designates 209.85.222.198 as permitted sender)
Received: from [209.85.222.198] (HELO mail-pz0-f198.google.com) (209.85.222.198)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 Nov 2009 05:38:51 +0000
Received: by pzk36 with SMTP id 36so3198758pzk.5
        for <general@hadoop.apache.org>; Sat, 21 Nov 2009 21:38:30 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=Jcsk7OnYZzPQZLyK9hVBO/pFUVoSEBRtHc/sDHPHVYo=;
        b=SpdyjZr07iep0WBZ/QujZ9ZZDbdPg5p51eLUhVys6v63XSL5UQEy8xmIAqp53gVpDt
         sJJ0MJhTxRKIiLKk7GXIhhB9Xz7/iFQNtRsSmoodqn8tAocz0FZgLhxXSO3CuwvQiUkL
         6G9wdfdZXSK0brd6015BLbiLjK36mwMxwMxFg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=qV77TZW7LlwzWAQD0YVrDjAvEyRJ3bunVpdnaD/vL2Gcr2Ki/TpkE1rCsfC0phFEM8
         pdBr7wWxUfFcMPggxWlwmBcMLGjrCJ5KafTAijIBGD6IbEjuHWGoldwg5v1hH2tN2SES
         ovrtwiG1U15MmwWCUa18zwRIWcoYQBAO33uHY=
MIME-Version: 1.0
Received: by 10.142.66.21 with SMTP id o21mr359909wfa.47.1258868310029; Sat, 
	21 Nov 2009 21:38:30 -0800 (PST)
In-Reply-To: <1eabbac30911211657i7812c67bs3a1565b0537b418f@mail.gmail.com>
References: <1eabbac30911211657i7812c67bs3a1565b0537b418f@mail.gmail.com>
Date: Sat, 21 Nov 2009 21:38:30 -0800
Message-ID: <8211a1320911212138i3956dc19k5ce39ce399afec62@mail.gmail.com>
Subject: Re: Starting Hadoop cluster on EC2
From: Jeff Zhang <zjffdu@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636e908a17037750478ef1e6b
X-Virus-Checked: Checked by ClamAV on apache.org

--001636e908a17037750478ef1e6b
Content-Type: text/plain; charset=UTF-8

You can scp key-pair to ec2 machine


Jeff Zhang


On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
mailinglists19@gmail.com> wrote:

> Trying to start a Hadoop cluster on EC2.  (Yes, Cloudera's distribution
> works well, but trying to do this myself so I can learn what's happening
> behind the scene.)
>
> I have a Master & a Slave.  When I start HDFS on the master, I get a
> message
> saying "10.xxx.xxx.xxx (Permission denied)" - where 10.xxx is IP address of
> the slave.
>
> Basic problem (I think) is that I can't ssh from the master EC2 instance to
> the Slave EC2 instance.  What's the best way to fix it?  I think I need the
> "Key Pair" file on my master.  I have a key pair on my local machine, but
> how do I transfer it to the EC2 machine?  (I know, I know, I agree.. I am
> dumb :)  Should I FTP it?
>
> Please help.  Thanks.
>

--001636e908a17037750478ef1e6b--

From general-return-765-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Nov 22 10:07:15 2009
Return-Path: <general-return-765-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 61101 invoked from network); 22 Nov 2009 10:07:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Nov 2009 10:07:15 -0000
Received: (qmail 95494 invoked by uid 500); 22 Nov 2009 10:07:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95405 invoked by uid 500); 22 Nov 2009 10:07:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95395 invoked by uid 99); 22 Nov 2009 10:07:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 Nov 2009 10:07:13 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of zjffdu@gmail.com designates 209.85.160.50 as permitted sender)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 Nov 2009 10:07:05 +0000
Received: by pwi6 with SMTP id 6so3091985pwi.29
        for <general@hadoop.apache.org>; Sun, 22 Nov 2009 02:06:44 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=pfoCzhZPNJ2+AG1CXwK055WrO4VkSRlVVPwW3U3L5IA=;
        b=ivZIdqn3pzuvycZb809flFXkWY2AiT3KmdCPEEl8wJt80UAGKrbVdR7asy/jphMv4q
         LSpM9pA/afDxZXF0CcuEWI2aYHTA4OjrZ1ueY8dHcNwoqOQxtXzbvx9txDClkl9mguTt
         9PYTUWeMs5nEk6VvyU653CU7f1yjAeNTWGhUM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=hN2buprf3oF2gaPqp5cgw2vdZ4p4H2DRRnvuXZMxRA+oiIF8P3hJTcoqe9ApUJGdAu
         nUFNWWR3XdYRQx5yM6W9+lNqhiYBNKhdsAqxCmisQyuRjCvF5BUaDy/SAS1itS1z04Bq
         mAGcqO9y27vC70jrze4BwFNdxH5dB0TWuyco0=
MIME-Version: 1.0
Received: by 10.142.60.3 with SMTP id i3mr367643wfa.147.1258884404187; Sun, 22 
	Nov 2009 02:06:44 -0800 (PST)
In-Reply-To: <1eabbac30911211657i7812c67bs3a1565b0537b418f@mail.gmail.com>
References: <1eabbac30911211657i7812c67bs3a1565b0537b418f@mail.gmail.com>
Date: Sun, 22 Nov 2009 02:06:44 -0800
Message-ID: <8211a1320911220206k3f7c45d2k8b27c10e37e5b9c8@mail.gmail.com>
Subject: Re: Starting Hadoop cluster on EC2
From: Jeff Zhang <zjffdu@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502ad14b991f90478f2dd5a
X-Virus-Checked: Checked by ClamAV on apache.org

--00504502ad14b991f90478f2dd5a
Content-Type: text/plain; charset=UTF-8

You can scp key-pair to ec2 machine

Jeff Zhang

On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
mailinglists19@gmail.com> wrote:

> Trying to start a Hadoop cluster on EC2.  (Yes, Cloudera's distribution
> works well, but trying to do this myself so I can learn what's happening
> behind the scene.)
>
> I have a Master & a Slave.  When I start HDFS on the master, I get a
> message
> saying "10.xxx.xxx.xxx (Permission denied)" - where 10.xxx is IP address of
> the slave.
>
> Basic problem (I think) is that I can't ssh from the master EC2 instance to
> the Slave EC2 instance.  What's the best way to fix it?  I think I need the
> "Key Pair" file on my master.  I have a key pair on my local machine, but
> how do I transfer it to the EC2 machine?  (I know, I know, I agree.. I am
> dumb :)  Should I FTP it?
>
> Please help.  Thanks.
>

--00504502ad14b991f90478f2dd5a--

From general-return-766-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Nov 22 16:09:52 2009
Return-Path: <general-return-766-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 18184 invoked from network); 22 Nov 2009 16:09:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Nov 2009 16:09:52 -0000
Received: (qmail 89472 invoked by uid 500); 22 Nov 2009 16:09:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89386 invoked by uid 500); 22 Nov 2009 16:09:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89376 invoked by uid 99); 22 Nov 2009 16:09:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 Nov 2009 16:09:50 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mailinglists19@gmail.com designates 209.85.219.214 as permitted sender)
Received: from [209.85.219.214] (HELO mail-ew0-f214.google.com) (209.85.219.214)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 Nov 2009 16:09:40 +0000
Received: by ewy6 with SMTP id 6so1104860ewy.29
        for <general@hadoop.apache.org>; Sun, 22 Nov 2009 08:09:20 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=l3ccVqnhcboX/gW0n7IWuvUpdKuEohzVCxOkIVepJVY=;
        b=ZbOANPhyuowNfY6YnP8y91qxtXL+iAPZYpbGPXl/SS+dpJDQ/FACrhb6ol3rjdSLH5
         xS+7ZXy2iaWKCOB4IW/jmPfpFREqyzXiHro/Y3nmMzLQQU22/JGOP770v/NCU0x1QEsn
         lJAg76wz+bSygZ7hkzti5cXXTNwRVGJlpTPEM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=M+WKe/li7doc6VR85skg3+UbqRH6TzXokiXifNUkZYF1jGoBBRjX2J8ZntcZLXSXyD
         Erj8GArtSDI4X9w9a4QxvkBqUM7F/tGV565aOPGCD9hFIckJoZz/TL1jcX4qXG+Mn3fc
         HxlV4OuD9t2nXbMnDqVr/vyDKpaDUZFg27Ahg=
MIME-Version: 1.0
Received: by 10.216.91.73 with SMTP id g51mr1252807wef.68.1258906158366; Sun, 
	22 Nov 2009 08:09:18 -0800 (PST)
In-Reply-To: <8211a1320911211812q64a7fb85jcdcda10be650d4ef@mail.gmail.com>
References: <1eabbac30911211657i7812c67bs3a1565b0537b418f@mail.gmail.com>
	 <8211a1320911211812q64a7fb85jcdcda10be650d4ef@mail.gmail.com>
Date: Sun, 22 Nov 2009 08:09:18 -0800
Message-ID: <1eabbac30911220809l2b923866u4c79e4a37e2d9ad4@mail.gmail.com>
Subject: Re: Starting Hadoop cluster on EC2
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d58cb260616d0478f7ee0d
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d58cb260616d0478f7ee0d
Content-Type: text/plain; charset=ISO-8859-1

As per Wikipedia (http://en.wikipedia.org/wiki/Secure_copy)  "It does so by
connecting to the host using SSH and there executes an SCP server (scp)".

So if SSH isn't working SCP won't work, either.  In any case I tried to scp
but getting "Permission denied (public key)".

Any other ideas?  Thanks.


On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <zjffdu@gmail.com> wrote:

> You should scp the key-pair to EC2 machine
>
> Jeff Zhang
>
> On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
> mailinglists19@gmail.com> wrote:
>
> > Trying to start a Hadoop cluster on EC2.  (Yes, Cloudera's distribution
> > works well, but trying to do this myself so I can learn what's happening
> > behind the scene.)
> >
> > I have a Master & a Slave.  When I start HDFS on the master, I get a
> > message
> > saying "10.xxx.xxx.xxx (Permission denied)" - where 10.xxx is IP address
> of
> > the slave.
> >
> > Basic problem (I think) is that I can't ssh from the master EC2 instance
> to
> > the Slave EC2 instance.  What's the best way to fix it?  I think I need
> the
> > "Key Pair" file on my master.  I have a key pair on my local machine, but
> > how do I transfer it to the EC2 machine?  (I know, I know, I agree.. I am
> > dumb :)  Should I FTP it?
> >
> > Please help.  Thanks.
> >
>

--0016e6d58cb260616d0478f7ee0d--

From general-return-767-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Nov 22 16:19:47 2009
Return-Path: <general-return-767-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19506 invoked from network); 22 Nov 2009 16:19:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Nov 2009 16:19:47 -0000
Received: (qmail 97660 invoked by uid 500); 22 Nov 2009 16:19:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97575 invoked by uid 500); 22 Nov 2009 16:19:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97565 invoked by uid 99); 22 Nov 2009 16:19:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 Nov 2009 16:19:45 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of zjffdu@gmail.com designates 209.85.160.50 as permitted sender)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 Nov 2009 16:19:35 +0000
Received: by pwi6 with SMTP id 6so3181360pwi.29
        for <general@hadoop.apache.org>; Sun, 22 Nov 2009 08:19:14 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=fNU+xgrwGjK1TRLF5UyetrxoS12QZH8j5vrDZxandpQ=;
        b=XJ3n8Kkeb/Fa9Wq2pNdtZ8EKqUf6/94TbzGe+1xpuxmaGOdCYdDjSMgk83S30PF6W7
         bNHEcY0HbVs+Zq3NiyJQlavddhl8Rwi5YqntRe8u3wHwxjt2LIRcr49B8fm/LVtGRTsZ
         SHzLXvUlwbBZySh13dK5D0q1PcI+eUtTz+svA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=hm7RbLHlOtjRMl6CVEz/d8cWsiSxmcjbarbs5rYnlqQtWR4yg9Y2X6HpZIf4mn+IgG
         lPfX2YSBVo5k8pckd4bw9Y8vxzSExLyqaFnX8v3GQlSRI2ynTJGf60iDOA09cf+QwSt2
         uIN3/rGHC0Z5OG5JB4Y4/eCd8kzvVwiik/cAE=
MIME-Version: 1.0
Received: by 10.142.60.3 with SMTP id i3mr393057wfa.147.1258906754128; Sun, 22 
	Nov 2009 08:19:14 -0800 (PST)
In-Reply-To: <1eabbac30911220809l2b923866u4c79e4a37e2d9ad4@mail.gmail.com>
References: <1eabbac30911211657i7812c67bs3a1565b0537b418f@mail.gmail.com>
	 <8211a1320911211812q64a7fb85jcdcda10be650d4ef@mail.gmail.com>
	 <1eabbac30911220809l2b923866u4c79e4a37e2d9ad4@mail.gmail.com>
Date: Sun, 22 Nov 2009 08:19:14 -0800
Message-ID: <8211a1320911220819j8d33c10te498c18362e825e0@mail.gmail.com>
Subject: Re: Starting Hadoop cluster on EC2
From: Jeff Zhang <zjffdu@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502ad14e29a890478f811c2
X-Virus-Checked: Checked by ClamAV on apache.org

--00504502ad14e29a890478f811c2
Content-Type: text/plain; charset=UTF-8

The ssh is installed on ec2 by default, otherwise you have no way to login
to ec2


Jeff Zhang



On Sun, Nov 22, 2009 at 8:09 AM, Something Something <
mailinglists19@gmail.com> wrote:

> As per Wikipedia (http://en.wikipedia.org/wiki/Secure_copy)  "It does so
> by
> connecting to the host using SSH and there executes an SCP server (scp)".
>
> So if SSH isn't working SCP won't work, either.  In any case I tried to scp
> but getting "Permission denied (public key)".
>
> Any other ideas?  Thanks.
>
>
> On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <zjffdu@gmail.com> wrote:
>
> > You should scp the key-pair to EC2 machine
> >
> > Jeff Zhang
> >
> > On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
> > mailinglists19@gmail.com> wrote:
> >
> > > Trying to start a Hadoop cluster on EC2.  (Yes, Cloudera's distribution
> > > works well, but trying to do this myself so I can learn what's
> happening
> > > behind the scene.)
> > >
> > > I have a Master & a Slave.  When I start HDFS on the master, I get a
> > > message
> > > saying "10.xxx.xxx.xxx (Permission denied)" - where 10.xxx is IP
> address
> > of
> > > the slave.
> > >
> > > Basic problem (I think) is that I can't ssh from the master EC2
> instance
> > to
> > > the Slave EC2 instance.  What's the best way to fix it?  I think I need
> > the
> > > "Key Pair" file on my master.  I have a key pair on my local machine,
> but
> > > how do I transfer it to the EC2 machine?  (I know, I know, I agree.. I
> am
> > > dumb :)  Should I FTP it?
> > >
> > > Please help.  Thanks.
> > >
> >
>

--00504502ad14e29a890478f811c2--

From general-return-768-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Nov 22 16:37:13 2009
Return-Path: <general-return-768-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22235 invoked from network); 22 Nov 2009 16:37:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Nov 2009 16:37:13 -0000
Received: (qmail 13634 invoked by uid 500); 22 Nov 2009 16:37:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13560 invoked by uid 500); 22 Nov 2009 16:37:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13550 invoked by uid 99); 22 Nov 2009 16:37:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 Nov 2009 16:37:12 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mailinglists19@gmail.com designates 209.85.219.214 as permitted sender)
Received: from [209.85.219.214] (HELO mail-ew0-f214.google.com) (209.85.219.214)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 Nov 2009 16:37:09 +0000
Received: by ewy6 with SMTP id 6so1119973ewy.29
        for <general@hadoop.apache.org>; Sun, 22 Nov 2009 08:36:48 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=b24i5osrV0RO+zS5FfrLBNLi/pq3z0dpvQp0Pl7kHMo=;
        b=npb6KOsdokML7pCBvsrITmdMlrxYEuImF9AAKbdOxtbwn2qq8pr1mKC6AApZXWYcHT
         0x48dkwKKClzlVWHkHFWzFWfQEnSBIV6JdqK3UlymUq4SumUy/GStXwW4Y4YJpuMAcFP
         coMa7XKzbhBAYtSch6pyBV49RFQqxP4RO3AKQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=J8LmgdMlHwdJ0sH52XgjiIfEtt6GFWlAqRCbPhbLQw/RN1spBR7U0LsKKVL9NQHxc2
         mjQ1N7un54nh5OV3GjgP/+LlBnj7lhq92HTW1AnF891SQlqGLR17aIvpWA2ENK5TPYLQ
         c8RAip/Q/IrsdTPOE2AYhXWyACdwr2Cy9j+mo=
MIME-Version: 1.0
Received: by 10.216.87.9 with SMTP id x9mr1170865wee.0.1258907808102; Sun, 22 
	Nov 2009 08:36:48 -0800 (PST)
In-Reply-To: <8211a1320911220819j8d33c10te498c18362e825e0@mail.gmail.com>
References: <1eabbac30911211657i7812c67bs3a1565b0537b418f@mail.gmail.com>
	 <8211a1320911211812q64a7fb85jcdcda10be650d4ef@mail.gmail.com>
	 <1eabbac30911220809l2b923866u4c79e4a37e2d9ad4@mail.gmail.com>
	 <8211a1320911220819j8d33c10te498c18362e825e0@mail.gmail.com>
Date: Sun, 22 Nov 2009 08:36:48 -0800
Message-ID: <1eabbac30911220836t76895c04x6f7973d759c60c18@mail.gmail.com>
Subject: Re: Starting Hadoop cluster on EC2
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6db2996b4f9e20478f85074

--0016e6db2996b4f9e20478f85074
Content-Type: text/plain; charset=ISO-8859-1

Seems like I am not explaining my problem correctly.

My Key Pair file is on my machine at work which is behind a corp firewall.
 As such I can't 'scp' from the ec2 instance to my local machine at work to
'get' the file.  So I need a way to 'send' a file from my machine to the ec2
instance.  I tried using 'scp' (from my machine at work) to the ec2 machine
but it says "Permission denied".  Does this make sense?

May be I need to use the command line tools for EC2.  I am looking into
those right now, but if there's a better/easier way, please let me know.

Thanks once again for your help.


On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <zjffdu@gmail.com> wrote:

> The ssh is installed on ec2 by default, otherwise you have no way to login
> to ec2
>
>
> Jeff Zhang
>
>
>
> On Sun, Nov 22, 2009 at 8:09 AM, Something Something <
> mailinglists19@gmail.com> wrote:
>
> > As per Wikipedia (http://en.wikipedia.org/wiki/Secure_copy)  "It does so
> > by
> > connecting to the host using SSH and there executes an SCP server (scp)".
> >
> > So if SSH isn't working SCP won't work, either.  In any case I tried to
> scp
> > but getting "Permission denied (public key)".
> >
> > Any other ideas?  Thanks.
> >
> >
> > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <zjffdu@gmail.com> wrote:
> >
> > > You should scp the key-pair to EC2 machine
> > >
> > > Jeff Zhang
> > >
> > > On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
> > > mailinglists19@gmail.com> wrote:
> > >
> > > > Trying to start a Hadoop cluster on EC2.  (Yes, Cloudera's
> distribution
> > > > works well, but trying to do this myself so I can learn what's
> > happening
> > > > behind the scene.)
> > > >
> > > > I have a Master & a Slave.  When I start HDFS on the master, I get a
> > > > message
> > > > saying "10.xxx.xxx.xxx (Permission denied)" - where 10.xxx is IP
> > address
> > > of
> > > > the slave.
> > > >
> > > > Basic problem (I think) is that I can't ssh from the master EC2
> > instance
> > > to
> > > > the Slave EC2 instance.  What's the best way to fix it?  I think I
> need
> > > the
> > > > "Key Pair" file on my master.  I have a key pair on my local machine,
> > but
> > > > how do I transfer it to the EC2 machine?  (I know, I know, I agree..
> I
> > am
> > > > dumb :)  Should I FTP it?
> > > >
> > > > Please help.  Thanks.
> > > >
> > >
> >
>

--0016e6db2996b4f9e20478f85074--

From general-return-769-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Nov 22 16:42:03 2009
Return-Path: <general-return-769-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23597 invoked from network); 22 Nov 2009 16:42:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Nov 2009 16:42:03 -0000
Received: (qmail 27986 invoked by uid 500); 22 Nov 2009 16:42:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27634 invoked by uid 500); 22 Nov 2009 16:41:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27513 invoked by uid 99); 22 Nov 2009 16:41:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 Nov 2009 16:41:57 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [203.199.18.82] (HELO mail1.impetus.co.in) (203.199.18.82)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 Nov 2009 16:41:46 +0000
Received: from mail1.impetus.co.in ([192.168.100.28]) by mail1.impetus.co.in
 ([192.168.100.28]) with mapi; Sun, 22 Nov 2009 22:11:21 +0530
From: Sanjay Sharma <sanjay.sharma@impetus.co.in>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Sun, 22 Nov 2009 22:11:14 +0530
Subject: RE: 1st Hadoop India User Group meet
Thread-Topic: 1st Hadoop India User Group meet
Thread-Index: Acphzcv/uxYtGsd9Rv2nKtqHmoWfSwJwoEJQ
Message-ID: <F0C7246DF63C124CA15A38CDC10CCD4A259DB72EE4@mail1.impetus.co.in>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Here is the updated agenda for Hadoop India User Group meet up on 28th Nove=
mber 2009 in Noida-
- Introductions
- Sessions-
        - "Enterprise perspective of Hadoop" - by Rajeev Gupta, IBM Researc=
h Lab, IBM
        - "Apache Zookeeper" - by Aby Abraham, Yahoo! Bangalore
        - "Advanced Hadoop Tuning and Optimizations"- by Sanjay Sharma, Imp=
etus

- Establishing agenda for the next few meetings
- Information exchange: tips, tricks, problems and open discussion, sharing=
 experiences on Hadoop and related technologies

The registrations are still open at http://hug-india.eventbrite.com/



Regards,
Sanjay Sharma
Impetus
t: +91-120-4363300 Extn 2761
m: +91-9958790163

-----Original Message-----
From: Sanjay Sharma
Sent: Tuesday, November 10, 2009 11:50 AM
To: 'general@hadoop.apache.org'
Subject: 1st Hadoop India User Group meet

We are planning to hold first Hadoop India user group meet up on 28th Novem=
ber 2009 in Noida.

We would be talking about our experiences with Apache Hadoop/Hbase/Hive/PIG=
/Nutch/etc.

The agenda would be:
- Introductions
- Sharing experiences on Hadoop and related technologies
- Establishing agenda for the next few meetings
- Information exchange: tips, tricks, problems and open discussion
- Possible speaker TBD (invitations open!!)  {we do have something to share=
 on "Hadoop for newbie" & "Hadoop Advanced Tuning"}

My company (Impetus) would be providing the meeting room and we should be a=
ble to accommodate around 40-60 friendly people. Coffee, Tea, and some snac=
ks will be provided.

Please join the linked-in Hadoop India User Group (http://www.linkedin.com/=
groups?home=3D&gid=3D2258445&trk=3Danet_ug_hm) OR Yahoo group (http://tech.=
groups.yahoo.com/group/hadoopind/) and confirm your attendance.

Regards,
Sanjay Sharma

Follow our updates on www.twitter.com/impetuscalling.

* Impetus Technologies is exhibiting it capabilities in Mobile and Wireless=
 in the GSMA Mobile Asia Congress, Hong Kong from November 16-18, 2009. Vis=
it http://www.impetus.com/mlabs/GSMA_events.html for details.

NOTE: This message may contain information that is confidential, proprietar=
y, privileged or otherwise protected by law. The message is intended solely=
 for the named addressee. If received in error, please destroy and notify t=
he sender. Any use of this email is prohibited when received in error. Impe=
tus does not represent, warrant and/or guarantee, that the integrity of thi=
s communication has been maintained nor that the communication is free of e=
rrors, virus, interception or interference.

From general-return-770-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Nov 22 19:45:43 2009
Return-Path: <general-return-770-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70342 invoked from network); 22 Nov 2009 19:45:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Nov 2009 19:45:43 -0000
Received: (qmail 90100 invoked by uid 500); 22 Nov 2009 19:45:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90036 invoked by uid 500); 22 Nov 2009 19:45:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90026 invoked by uid 99); 22 Nov 2009 19:45:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 Nov 2009 19:45:41 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE,SUBJ_ALL_CAPS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cabiwan@gmail.com designates 209.85.220.224 as permitted sender)
Received: from [209.85.220.224] (HELO mail-fx0-f224.google.com) (209.85.220.224)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 Nov 2009 19:45:39 +0000
Received: by fxm24 with SMTP id 24so2086522fxm.11
        for <general@hadoop.apache.org>; Sun, 22 Nov 2009 11:45:17 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=nI4i1K03RRWFzU5cIwxki8LrohtWcjEFd1z4Tuem99A=;
        b=x87Kzr19eS+7Dl8qnEfVy6oVJXJNDGlfb9dYzwIxkzkZ9bn+MHUNL8Ggmd2U2qBrxJ
         WV1NYbFCvQtZaORJt+a5MfA7/9zhfbiCbHM5RPlTT9u8J4gJxTDEELt5NIoN9TsVG6ZG
         /w2VMaA3LnWpWkvGiBsMJIQq8DseG/gS4QA+c=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=sc7xFxLdLe/y95zj6EcS6v3DXL7lI9g3uuyaLjmbPy4JKwiTT5zcQTm/2qc7KZcoPF
         hCdw8cdde+/zsLmgLjbmo01ijinSF1j/AjZD/1XauEfGDdhlgFa48/5kLqC/QaAQBSbL
         xAoz3WwQKF8+EC552ErSE9kbtp4ig8IxZefbI=
MIME-Version: 1.0
Received: by 10.103.84.2 with SMTP id m2mr1837826mul.107.1258919117635; Sun, 
	22 Nov 2009 11:45:17 -0800 (PST)
Date: Sun, 22 Nov 2009 20:45:17 +0100
Message-ID: <309f76d00911221145g4b1ba10cx57888f06020c9ade@mail.gmail.com>
Subject: WRITING DEBUGGING MESSAGES TO STANDARD OUTPUT IN HADOOP 0.20.1
From: Alberto Luengo Cabanillas <cabiwan@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6550dcecec26c0478faf25c

--0016e6550dcecec26c0478faf25c
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi everyone! I=B4m developing a Hadoop project using Cywin, Eclipse and VMW=
are
(running a Hadoop 0.20.1 distribution, similar to Cloudera=B4s one). The
problem I=B4m facing now is that from inside Eclipse=B4s console I can see =
all
the messages launched by the jobtracker (i.e "09/11/22 20:28:02 INFO
mapred.JobClient: Spilled Records=3D200") and the System.out.println insert=
ed
in my code (like "***ENTERING PRIVATE FUNCTION****), but I can=B4t see any
message if I insert -by example- a System.out.println("INSIDE MAP
FUNCTION"). I also tried with context.setStatus("INSIDE MAP FUNCTION"), but
I didn=B4t obtain any results.
Is there any way to archive what I=B4m trying to do?
Thanks in advance.

--=20
Alberto

--0016e6550dcecec26c0478faf25c--

From general-return-771-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 23 02:31:13 2009
Return-Path: <general-return-771-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 90110 invoked from network); 23 Nov 2009 02:31:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Nov 2009 02:31:13 -0000
Received: (qmail 9754 invoked by uid 500); 23 Nov 2009 02:31:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 9600 invoked by uid 500); 23 Nov 2009 02:31:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 9589 invoked by uid 99); 23 Nov 2009 02:31:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Nov 2009 02:31:11 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jason.hadoop@gmail.com designates 209.85.222.198 as permitted sender)
Received: from [209.85.222.198] (HELO mail-pz0-f198.google.com) (209.85.222.198)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Nov 2009 02:31:01 +0000
Received: by pzk36 with SMTP id 36so3573746pzk.5
        for <general@hadoop.apache.org>; Sun, 22 Nov 2009 18:30:40 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=2prB6eP8UwSl2zWdebNvlCUTP6p4rdqepV4/5sehoag=;
        b=RQuxQ+d6QuU+L6H9ODFHTouQ62C+ErORuBo3uyDVEoIcYOeDkfxzaIHrYhGa1qzNyv
         Tbd7CmdXHugNP5PuS58Lzm5Mncg/RlHqE39KFBo0T9EfMPh4KAYUhNVkCW5cJUjqU0QB
         pSY2ZogBya6T76SkMXSOAEmMTesQ39v7cPe+M=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=smkIFWCo9Ijx/U0Mf1b8cOxEBGDxKCwQdS8hr2gCTcnxVedmG9pgriskkCKLuyDpDb
         SqlbeGtgbdT6bNyo4rLzLOZn9GNVOVMiTogi/SnmpQIZ3DGuVwMpiYLAUItWOHByY6Zh
         rgyH0MqIBg/gPe22zxuaDmDm2IRA9lJSEQIkw=
MIME-Version: 1.0
Received: by 10.140.134.14 with SMTP id h14mr263968rvd.297.1258943440558; Sun, 
	22 Nov 2009 18:30:40 -0800 (PST)
In-Reply-To: <309f76d00911221145g4b1ba10cx57888f06020c9ade@mail.gmail.com>
References: <309f76d00911221145g4b1ba10cx57888f06020c9ade@mail.gmail.com>
Date: Sun, 22 Nov 2009 18:30:40 -0800
Message-ID: <314098690911221830t459434dft5411dddb11e50725@mail.gmail.com>
Subject: Re: WRITING DEBUGGING MESSAGES TO STANDARD OUTPUT IN HADOOP 0.20.1
From: Jason Venner <jason.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd1728c911b860479009c1e
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd1728c911b860479009c1e
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

In general the stdout, stderr and syslog output for map and reduce tasks go
into the logs/userlogs directory on the task tracker node that ran the task=
.
The log file data is available vai the jobtracker web interface if you clic=
k
through the per task information enough times.

On Sun, Nov 22, 2009 at 11:45 AM, Alberto Luengo Cabanillas <
cabiwan@gmail.com> wrote:

> Hi everyone! I=B4m developing a Hadoop project using Cywin, Eclipse and
> VMWare
> (running a Hadoop 0.20.1 distribution, similar to Cloudera=B4s one). The
> problem I=B4m facing now is that from inside Eclipse=B4s console I can se=
e all
> the messages launched by the jobtracker (i.e "09/11/22 20:28:02 INFO
> mapred.JobClient: Spilled Records=3D200") and the System.out.println inse=
rted
> in my code (like "***ENTERING PRIVATE FUNCTION****), but I can=B4t see an=
y
> message if I insert -by example- a System.out.println("INSIDE MAP
> FUNCTION"). I also tried with context.setStatus("INSIDE MAP FUNCTION"), b=
ut
> I didn=B4t obtain any results.
> Is there any way to archive what I=B4m trying to do?
> Thanks in advance.
>
> --
> Alberto
>



--=20
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=3Djewlerymall
www.prohadoopbook.com a community for Hadoop Professionals

--000e0cd1728c911b860479009c1e--

From general-return-772-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 23 11:15:20 2009
Return-Path: <general-return-772-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5558 invoked from network); 23 Nov 2009 11:15:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Nov 2009 11:15:20 -0000
Received: (qmail 72018 invoked by uid 500); 23 Nov 2009 11:15:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71924 invoked by uid 500); 23 Nov 2009 11:15:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71913 invoked by uid 99); 23 Nov 2009 11:15:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Nov 2009 11:15:18 +0000
X-ASF-Spam-Status: No, hits=-5.6 required=5.0
	tests=AWL,BAYES_00,RCVD_IN_DNSWL_MED
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Nov 2009 11:15:16 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 9BEE0B7EA5
	for <general@hadoop.apache.org>; Mon, 23 Nov 2009 11:14:54 +0000 (GMT)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id aWHD6HM24Taq for <general@hadoop.apache.org>;
	Mon, 23 Nov 2009 11:14:47 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 3FA57B7E7F
	for <general@hadoop.apache.org>; Mon, 23 Nov 2009 11:14:47 +0000 (GMT)
MailScanner-NULL-Check: 1259579677.33733@87lmY9v4rj7eawtAGTzlZw
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id nANBEac2017511
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Mon, 23 Nov 2009 11:14:37 GMT
Message-ID: <4B0A6E9C.7060108@apache.org>
Date: Mon, 23 Nov 2009 11:14:36 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Starting Hadoop cluster on EC2
References: <1eabbac30911211657i7812c67bs3a1565b0537b418f@mail.gmail.com>	 <8211a1320911211812q64a7fb85jcdcda10be650d4ef@mail.gmail.com> <1eabbac30911220809l2b923866u4c79e4a37e2d9ad4@mail.gmail.com>
In-Reply-To: <1eabbac30911220809l2b923866u4c79e4a37e2d9ad4@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: nANBEac2017511
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

Something Something wrote:
> As per Wikipedia (http://en.wikipedia.org/wiki/Secure_copy)  "It does so by
> connecting to the host using SSH and there executes an SCP server (scp)".
> 
> So if SSH isn't working SCP won't work, either.  In any case I tried to scp
> but getting "Permission denied (public key)".
> 
when you create a VM on EC2 you give it the public key you want to 
SCP/SSH in on.

From general-return-773-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 23 16:11:39 2009
Return-Path: <general-return-773-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14935 invoked from network); 23 Nov 2009 16:11:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Nov 2009 16:11:39 -0000
Received: (qmail 45447 invoked by uid 500); 23 Nov 2009 16:11:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45382 invoked by uid 500); 23 Nov 2009 16:11:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45371 invoked by uid 99); 23 Nov 2009 16:11:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Nov 2009 16:11:37 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of vpuranik@gmail.com designates 209.85.216.201 as permitted sender)
Received: from [209.85.216.201] (HELO mail-px0-f201.google.com) (209.85.216.201)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Nov 2009 16:11:27 +0000
Received: by pxi39 with SMTP id 39so3918806pxi.2
        for <general@hadoop.apache.org>; Mon, 23 Nov 2009 08:11:05 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=SAdz9Z12rJIlXS/xznp9kNT0sNLEbpPl5qKxQqM9M2E=;
        b=e8tQtDoRgKnFY+UAcQCjQIl6BlpN1FXbR/T/7WkiB540qCTOInBJy2PNGsmZ2u8p3x
         Zr4J7VIInTUQA+0lDBb4T/8CMIkMqM9j3XZBEcHPLqH7p81dLJFO1Yj6hkEb2yqXbA2f
         aJ/+fCA96sO9Y07ZMO0Z86/h80P2+j/xJw2/Q=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=dJyCuKR7MFBxd8R2jlEe4enocvjhrXfD5t11hcRsEmzRqBORXKAzRCWrMozjXj73am
         ry9LFj6SY7A+OA1WFXaSfm1vk/ugx+lRSNplyFGrhJNdMD5Nei8Krf9/IoWgHK1HPGSl
         7P2gUImcpGPQvhh2Hw118gk8SpvLjDN2oOL9k=
MIME-Version: 1.0
Received: by 10.114.250.10 with SMTP id x10mr9236472wah.196.1258992659901; 
	Mon, 23 Nov 2009 08:10:59 -0800 (PST)
In-Reply-To: <1eabbac30911220836t76895c04x6f7973d759c60c18@mail.gmail.com>
References: <1eabbac30911211657i7812c67bs3a1565b0537b418f@mail.gmail.com>
	 <8211a1320911211812q64a7fb85jcdcda10be650d4ef@mail.gmail.com>
	 <1eabbac30911220809l2b923866u4c79e4a37e2d9ad4@mail.gmail.com>
	 <8211a1320911220819j8d33c10te498c18362e825e0@mail.gmail.com>
	 <1eabbac30911220836t76895c04x6f7973d759c60c18@mail.gmail.com>
Date: Mon, 23 Nov 2009 08:10:59 -0800
Message-ID: <2eef7bd70911230810l690624a1vd4fe8c208279293c@mail.gmail.com>
Subject: Re: Starting Hadoop cluster on EC2
From: Vaibhav Puranik <vpuranik@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016367f929844afab04790c1261
X-Virus-Checked: Checked by ClamAV on apache.org

--0016367f929844afab04790c1261
Content-Type: text/plain; charset=ISO-8859-1

In order to use scp from your machine to an ec2 instance you need to use scp
-i and give it the path of your keypair.

scp -i /path/my/ec2/keypair  <souce> <destination>

Secondly in order to configure an ec2 cluster for Hadoop, you need to make a
security group (let's call it hadoop) and give hadoop access to hadoop.

You can read more about security groups here -
http://somic.org/2009/09/21/security-groups-most-underappreciated-feature-of-amazon-ec2/

You also need to make sure that you can ssh from all the slaves to master
without specifying password.

Regards,
Vaibhav Puranik
GumGum

On Sun, Nov 22, 2009 at 8:36 AM, Something Something <
mailinglists19@gmail.com> wrote:

> Seems like I am not explaining my problem correctly.
>
> My Key Pair file is on my machine at work which is behind a corp firewall.
>  As such I can't 'scp' from the ec2 instance to my local machine at work to
> 'get' the file.  So I need a way to 'send' a file from my machine to the
> ec2
> instance.  I tried using 'scp' (from my machine at work) to the ec2 machine
> but it says "Permission denied".  Does this make sense?
>
> May be I need to use the command line tools for EC2.  I am looking into
> those right now, but if there's a better/easier way, please let me know.
>
> Thanks once again for your help.
>
>
> On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <zjffdu@gmail.com> wrote:
>
> > The ssh is installed on ec2 by default, otherwise you have no way to
> login
> > to ec2
> >
> >
> > Jeff Zhang
> >
> >
> >
> > On Sun, Nov 22, 2009 at 8:09 AM, Something Something <
> > mailinglists19@gmail.com> wrote:
> >
> > > As per Wikipedia (http://en.wikipedia.org/wiki/Secure_copy)  "It does
> so
> > > by
> > > connecting to the host using SSH and there executes an SCP server
> (scp)".
> > >
> > > So if SSH isn't working SCP won't work, either.  In any case I tried to
> > scp
> > > but getting "Permission denied (public key)".
> > >
> > > Any other ideas?  Thanks.
> > >
> > >
> > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <zjffdu@gmail.com> wrote:
> > >
> > > > You should scp the key-pair to EC2 machine
> > > >
> > > > Jeff Zhang
> > > >
> > > > On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
> > > > mailinglists19@gmail.com> wrote:
> > > >
> > > > > Trying to start a Hadoop cluster on EC2.  (Yes, Cloudera's
> > distribution
> > > > > works well, but trying to do this myself so I can learn what's
> > > happening
> > > > > behind the scene.)
> > > > >
> > > > > I have a Master & a Slave.  When I start HDFS on the master, I get
> a
> > > > > message
> > > > > saying "10.xxx.xxx.xxx (Permission denied)" - where 10.xxx is IP
> > > address
> > > > of
> > > > > the slave.
> > > > >
> > > > > Basic problem (I think) is that I can't ssh from the master EC2
> > > instance
> > > > to
> > > > > the Slave EC2 instance.  What's the best way to fix it?  I think I
> > need
> > > > the
> > > > > "Key Pair" file on my master.  I have a key pair on my local
> machine,
> > > but
> > > > > how do I transfer it to the EC2 machine?  (I know, I know, I
> agree..
> > I
> > > am
> > > > > dumb :)  Should I FTP it?
> > > > >
> > > > > Please help.  Thanks.
> > > > >
> > > >
> > >
> >
>

--0016367f929844afab04790c1261--

From general-return-774-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 23 22:10:44 2009
Return-Path: <general-return-774-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 64968 invoked from network); 23 Nov 2009 22:10:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Nov 2009 22:10:44 -0000
Received: (qmail 86850 invoked by uid 500); 23 Nov 2009 22:10:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86771 invoked by uid 500); 23 Nov 2009 22:10:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86761 invoked by uid 99); 23 Nov 2009 22:10:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Nov 2009 22:10:41 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of psdc1978@gmail.com designates 72.14.220.154 as permitted sender)
Received: from [72.14.220.154] (HELO fg-out-1718.google.com) (72.14.220.154)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Nov 2009 22:10:39 +0000
Received: by fg-out-1718.google.com with SMTP id d23so2335268fga.11
        for <general@hadoop.apache.org>; Mon, 23 Nov 2009 14:10:18 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=YlzLf5ppk/YmRpmSW5ieu4QWFTeAsVN4bMXu8LAR4ao=;
        b=XzwV1Vo40G380pVDdiFo6L1FcTUBDyzmsujzRISzHK807iHlxO0OYZLvIChT4DdNCr
         DfiDNPkXkUISX/yD04Yg+pUhO7/bSMx/S9b204rNmng/sfDCh5aQSJbMoy4BjJk0C9S1
         UpHTn5uIIDLVduPV9g6KWnnsqS+NqXsy8wIcM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=YSZOVWO7xOmy+wmO8KevOxxu1+AkFEFEAm0tJRjbPioBi4hFfj+NCZjSHFjOvJTLdK
         Ms524i+4EN7QxPr6Dk67iVwxGigAdqfzS3n9Zvhx93YejSZFfhSdrUgIEcI9Okie6OIs
         ztxTRel8KizoHiYL9V5wXT6t6vupgkHF/fES0=
MIME-Version: 1.0
Received: by 10.239.130.160 with SMTP id 32mr535895hbj.58.1259014216903; Mon, 
	23 Nov 2009 14:10:16 -0800 (PST)
Date: Mon, 23 Nov 2009 22:10:16 +0000
Message-ID: <61fec3fc0911231410y23a02102xf52f7bed2f2afa1e@mail.gmail.com>
Subject: How compile hadoop-0.20.1
From: psdc1978 <psdc1978@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f78c422a6e79047911178b

--001485f78c422a6e79047911178b
Content-Type: text/plain; charset=ISO-8859-1

Hi all,

1 - I'm trying to compile hadoop-0.20.1 but I'm facing several errors.
Hadoop includes several subprojects, but these subprojects are all compiled
in one file called hadoop-0.20.1-core.jar. If I just want to update the code
in the MapReduce component, do I have to compile all the project?

2 - Why Hadoop only have one jar for all the project? For example, MapReduce
subcomponent could be inside mapreduce.jar, Common subcomponent could be
inside common.jar, HDFS subcomponent could be inside hdfs.jar, etc.

3 - Is there any tutorial that helps how to compile Hadoop?

4 - Should I try to compile hadoop in windows with Cygwin, or should I
compile inside Ubuntu?

Thanks,

-- 
Pedro

--001485f78c422a6e79047911178b--

From general-return-775-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 23 23:46:33 2009
Return-Path: <general-return-775-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95535 invoked from network); 23 Nov 2009 23:46:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Nov 2009 23:46:32 -0000
Received: (qmail 10464 invoked by uid 500); 23 Nov 2009 23:46:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 10379 invoked by uid 500); 23 Nov 2009 23:46:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 10369 invoked by uid 99); 23 Nov 2009 23:46:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Nov 2009 23:46:31 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [209.191.125.39] (HELO web38408.mail.mud.yahoo.com) (209.191.125.39)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 23 Nov 2009 23:46:22 +0000
Received: (qmail 23378 invoked by uid 60001); 23 Nov 2009 23:46:00 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1259019960; bh=XgvPyWXksNrXV+zgxWv9taB+BfA6iHOq2lhigN8pYeU=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=X1eSxLy8ThkEYCNjwYtEgWMY0lhSQ53EOxP/SMJlDvKovFEpdaPO2WYqlhXuhgJgRn0Cxo+bbkcvAmZ2l/RiEbINSul3aY3isicU3rzMWdAXZ19ekQQT/qcUohGviKtTmSOVbXaHRmBVfm2uCDBQuLlf3Y7wT2z6piO73ruc1bQ=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=2LisUGCucoigqblj5T6SXZ4aPUj5AOAV2/OwGu2iMQbzauZdfmYfLWbNT3b7Wrk3+o7vhCh3ZqQFqCb5lu6BwuHZJlkrYrHmj7Ho3egGNPeanWEY5IcKmIB4AzolrOJexmBdtJZoeu080haBj18syGtw/V9as/WXfSasg0+BA+I=;
Message-ID: <38943.23289.qm@web38408.mail.mud.yahoo.com>
X-YMail-OSG: TVmsMr8VM1korGUJUipoJIO46_NvrnwF1hsZ11OYkO._kL0uZ9di96pKWhWAwxfYoid_HiudBH5dP6N1RoK._L0dE3bEjbuOk_vTvuFRWrfGtA9DOuzI0J.9QPYqp2lSx50kN0SjXbiS.ZNoppF23y_XoQH4NKCddnfaznKmo7TCFFR90baerODcWHRf9C6JPEWQC8ZCiLxIjggjcXw74el.9zIXv5RmBQAkYArTqzlW5TT8ICJmTAKOpFhBrmFxwCeW5PvUxX.XzLmZG.vdK_0O2Lmnv1zodl5VNtBPM8kYoK20fIqE_Npeb1gJcsmNDDBIVhfCkpuLi2ju0cw0.Q--
Received: from [129.170.212.188] by web38408.mail.mud.yahoo.com via HTTP; Mon, 23 Nov 2009 15:46:00 PST
X-Mailer: YahooMailRC/211.6 YahooMailWebService/0.8.100.260964
Date: Mon, 23 Nov 2009 15:46:00 -0800 (PST)
From: himanshu chandola <himanshu_coolguy@yahoo.com>
Subject: Size of BytesWritable
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Everyone,
I am writing binary files using SequenceFile.Writer and what I am seeing is that the resulting file is much greater than what it should be. 
Here's my calculation, if someone can point out what is wrong in this it would be great:

A Sequence File should have its own header : some constant bytes.
Every BytesWritable written to the sequence file should be : number of bytes in the byte array for the BytesWritable plus an object header (probably 4 bytes for the header). 

 

The total size of the file should be the sum of the above two. Is this fine ? 


Thanks 

Himanshu


Morpheus: Do you believe in fate, Neo?
Neo: No.
Morpheus: Why Not?
Neo: Because I don't like the idea that I'm not in control of my life.



      

From general-return-776-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 24 10:33:16 2009
Return-Path: <general-return-776-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 48516 invoked from network); 24 Nov 2009 10:33:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Nov 2009 10:33:16 -0000
Received: (qmail 50957 invoked by uid 500); 24 Nov 2009 10:33:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50870 invoked by uid 500); 24 Nov 2009 10:33:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50860 invoked by uid 99); 24 Nov 2009 10:33:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Nov 2009 10:33:14 +0000
X-ASF-Spam-Status: No, hits=-5.7 required=5.0
	tests=AWL,BAYES_00,RCVD_IN_DNSWL_MED
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Nov 2009 10:33:12 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 2A8CCB7CF0
	for <general@hadoop.apache.org>; Tue, 24 Nov 2009 10:32:49 +0000 (GMT)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id 0Ziv8fs0t9Fy for <general@hadoop.apache.org>;
	Tue, 24 Nov 2009 10:32:42 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 0FEFCB7F28
	for <general@hadoop.apache.org>; Tue, 24 Nov 2009 10:32:41 +0000 (GMT)
MailScanner-NULL-Check: 1259663551.8537@M1xOBSXSe6Nu3gvSI/cmXA
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id nAOAWUKi020510
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Tue, 24 Nov 2009 10:32:30 GMT
Message-ID: <4B0BB63E.4070306@apache.org>
Date: Tue, 24 Nov 2009 10:32:30 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: How compile hadoop-0.20.1
References: <61fec3fc0911231410y23a02102xf52f7bed2f2afa1e@mail.gmail.com>
In-Reply-To: <61fec3fc0911231410y23a02102xf52f7bed2f2afa1e@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: nAOAWUKi020510
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

psdc1978 wrote:
> Hi all,
> 
> 1 - I'm trying to compile hadoop-0.20.1 but I'm facing several errors.
> Hadoop includes several subprojects, but these subprojects are all compiled
> in one file called hadoop-0.20.1-core.jar. If I just want to update the code
> in the MapReduce component, do I have to compile all the project?

yes

> 
> 2 - Why Hadoop only have one jar for all the project? For example, MapReduce
> subcomponent could be inside mapreduce.jar, Common subcomponent could be
> inside common.jar, HDFS subcomponent could be inside hdfs.jar, etc.

in SVN_HEAD Hadoop it is split up. That will give you a version 
synchronisation problem that didn't exist before, of course


> 3 - Is there any tutorial that helps how to compile Hadoop?

Check out SVN, run "ant -p" to list the targets

> 
> 4 - Should I try to compile hadoop in windows with Cygwin, or should I
> compile inside Ubuntu?

If you have linux, I would go for that. It does build under cygwin, and 
on OS/X, but there's a lot to be said for working in Linux on your 
desktop so that the problems you encounter are the same as you'd see in 
the cluster

From general-return-777-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 24 14:24:27 2009
Return-Path: <general-return-777-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49850 invoked from network); 24 Nov 2009 14:24:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Nov 2009 14:24:27 -0000
Received: (qmail 60075 invoked by uid 500); 24 Nov 2009 14:24:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59529 invoked by uid 500); 24 Nov 2009 14:24:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59410 invoked by uid 99); 24 Nov 2009 14:24:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Nov 2009 14:24:24 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of abhilaash_cse@tce.edu designates 210.212.252.72 as permitted sender)
Received: from [210.212.252.72] (HELO tce.edu) (210.212.252.72)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Nov 2009 14:24:14 +0000
Received: from [127.0.0.1] (helo=mail.tce.edu)
	by tce.edu with esmtp (Exim 4.63)
	(envelope-from <abhilaash_cse@tce.edu>)
	id 1NCwJ3-0003gj-J0
	for general@hadoop.apache.org; Tue, 24 Nov 2009 19:53:52 +0530
Received: from 59.96.24.33
        (SquirrelMail authenticated user abhilaash_cse)
        by mail.tce.edu with HTTP;
        Tue, 24 Nov 2009 19:53:49 +0530
Message-ID: <6efd398094dc482ed37a662080836354.squirrel@mail.tce.edu>
Date: Tue, 24 Nov 2009 19:53:49 +0530
Subject: Doubts Regarding Capacity Scheduler's Function
From: "Abhilaash" <abhilaash_cse@tce.edu>
To: general@hadoop.apache.org
User-Agent: SquirrelMail/1.4.19
MIME-Version: 1.0
Content-Type: text/plain;charset=iso-8859-1
Content-Transfer-Encoding: 8bit
X-Priority: 3 (Normal)
Importance: Normal
X-XheaderVersion: 1.1
X-UserAgent: 
X-Spam-Score: -1.4 (-)
X-Virus-Checked: Checked by ClamAV on apache.org
X-Old-Spam-Status: No



Hi,

        I have installed hadoop and configured capacity scheduler with
three queues and run three jobs simultaneously,but i found that
the jobs are scheduled only to the default queue and the jobs were
not scheduled to the other queues configured by me.Is there any
more configuration to be done for scheduling of jobs to the user
defined queues?

        I have a basic doubt, whether Capacity scheduler schedules the job
as a whole(say wordcount) or schedules the tasks(maps and reduces
of the job) to the available nodes in the cluster.

        It would be very helpful if someone helps me to clearly understand
the working of the capacity scheduler.


Thanks in Advance.


Regards,
Abhilaash.


-- 
Nothing is impossible because impossible says i'm possible...


-----------------------------------------
This email was sent using TCEMail Service.
Thiagarajar College of Engineering
Madurai-625 015, India


From general-return-778-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 24 16:24:30 2009
Return-Path: <general-return-778-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98452 invoked from network); 24 Nov 2009 16:24:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Nov 2009 16:24:30 -0000
Received: (qmail 78035 invoked by uid 500); 24 Nov 2009 16:24:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77948 invoked by uid 500); 24 Nov 2009 16:24:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77936 invoked by uid 99); 24 Nov 2009 16:24:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Nov 2009 16:24:28 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Nov 2009 16:24:17 +0000
Received: from [192.168.1.71] (snvvpn2-10-73-152-c154.hq.corp.yahoo.com [10.73.152.154])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id nAOGNo6l011197
	for <general@hadoop.apache.org>; Tue, 24 Nov 2009 08:23:50 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:mime-version:
	subject:x-priority:date:references:x-mailer;
	b=UY1vlsZumJjdOhB+b8REtPoRsNVAWrz9brg2bV+U9dkRzpgDYjwzQqebx/uzKU61
Message-Id: <9FD0B40E-F7BA-4CFC-9BE9-3D52758E4EC8@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <6efd398094dc482ed37a662080836354.squirrel@mail.tce.edu>
Content-Type: multipart/alternative; boundary=Apple-Mail-156--547727165
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Doubts Regarding Capacity Scheduler's Function
X-Priority: 3 (Normal)
Date: Tue, 24 Nov 2009 08:23:50 -0800
References: <6efd398094dc482ed37a662080836354.squirrel@mail.tce.edu>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-156--547727165
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


On Nov 24, 2009, at 6:23 AM, Abhilaash wrote:
> I have installed hadoop and configured capacity scheduler with
> three queues and run three jobs simultaneously,but i found that
> the jobs are scheduled only to the default queue and the jobs were
> not scheduled to the other queues configured by me.Is there any
> more configuration to be done for scheduling of jobs to the user
> defined queues?

Each job needs to have the mapred.job.queue.name set to the right  
queue for the CS. It defaults to "default" queue.

>        I have a basic doubt, whether Capacity scheduler schedules  
> the job
> as a whole(say wordcount) or schedules the tasks(maps and reduces
> of the job) to the available nodes in the cluster.

Every Map-Reduce scheduler: DefaultScheduler, CapacityScheduler,  
FairScheduler, schedules one task at a time.

>        It would be very helpful if someone helps me to clearly  
> understand
> the working of the capacity scheduler.

http://hadoop.apache.org/common/docs/r0.20.0/capacity_scheduler.html -  
hope this helps.

Arun


--Apple-Mail-156--547727165--

From general-return-779-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 24 18:54:57 2009
Return-Path: <general-return-779-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69920 invoked from network); 24 Nov 2009 18:54:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Nov 2009 18:54:56 -0000
Received: (qmail 2557 invoked by uid 500); 24 Nov 2009 18:54:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2470 invoked by uid 500); 24 Nov 2009 18:54:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2460 invoked by uid 99); 24 Nov 2009 18:54:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Nov 2009 18:54:54 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mailinglists19@gmail.com designates 209.85.219.217 as permitted sender)
Received: from [209.85.219.217] (HELO mail-ew0-f217.google.com) (209.85.219.217)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Nov 2009 18:54:52 +0000
Received: by ewy9 with SMTP id 9so855554ewy.11
        for <general@hadoop.apache.org>; Tue, 24 Nov 2009 10:54:30 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=z/kk8HatPF7H34inWRZ995DzrKnn3mewHycieXr1bmI=;
        b=DVhEaEMfNs7E3sM8ZT3mSkRMQ9/vCJxePfsvCEKhZN7dMxuxbDv3uwqXcD9MiYTvcO
         j5XLkzF9qTTZ/fjQxDthaPswiOCCygdQJZ2CN47QfEHTNLt+2qFFEDIYzfYVZw7cs04W
         zfEIjJdkJ+c8G66ub2gRzMVx95ZO318RojFSc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=JVvxGXuoRWr8dFSVDQT9oqe/RaCRDBeXLe2RdjYmQWiA8oNu8roXC+6GKrVnguMsmI
         RtXJW/tVnSNUpaQ5EMttB0MypUuSXUXNyWcZSivIg/VeH6wJfwLI/hgqosnAKvWFNmtT
         v41VDKQkif/vXunCR+bcb+yJV8AfFA+ZfDLL8=
MIME-Version: 1.0
Received: by 10.216.87.133 with SMTP id y5mr2130418wee.139.1259088870311; Tue, 
	24 Nov 2009 10:54:30 -0800 (PST)
In-Reply-To: <2eef7bd70911230810l690624a1vd4fe8c208279293c@mail.gmail.com>
References: <1eabbac30911211657i7812c67bs3a1565b0537b418f@mail.gmail.com>
	 <8211a1320911211812q64a7fb85jcdcda10be650d4ef@mail.gmail.com>
	 <1eabbac30911220809l2b923866u4c79e4a37e2d9ad4@mail.gmail.com>
	 <8211a1320911220819j8d33c10te498c18362e825e0@mail.gmail.com>
	 <1eabbac30911220836t76895c04x6f7973d759c60c18@mail.gmail.com>
	 <2eef7bd70911230810l690624a1vd4fe8c208279293c@mail.gmail.com>
Date: Tue, 24 Nov 2009 10:54:30 -0800
Message-ID: <1eabbac30911241054w6ef4852n41ddf93198c27acd@mail.gmail.com>
Subject: Re: Starting Hadoop cluster on EC2
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d9a2c7db061904792278a0

--0016e6d9a2c7db061904792278a0
Content-Type: text/plain; charset=ISO-8859-1

Thanks, Vaibhav.  That was helpful.  I am able to scp from my local machine
after I opened port 22 under my security group.

But my cluster is still not starting as expected.  I mean... on my Master
when I run:

./start-all.sh

I get messages saying...

10.252.xxx.xxx: Permission denied (publickey).

where 10.252.xxx.xxx is the ip address of my Slave.


I can ssh from my Slave to master if I do this:

ssh -i <keypairname>.pem <Master's ip address>


Any suggestions?  Thanks for your help.




On Mon, Nov 23, 2009 at 8:10 AM, Vaibhav Puranik <vpuranik@gmail.com> wrote:

> In order to use scp from your machine to an ec2 instance you need to use
> scp
> -i and give it the path of your keypair.
>
> scp -i /path/my/ec2/keypair  <souce> <destination>
>
> Secondly in order to configure an ec2 cluster for Hadoop, you need to make
> a
> security group (let's call it hadoop) and give hadoop access to hadoop.
>
> You can read more about security groups here -
>
> http://somic.org/2009/09/21/security-groups-most-underappreciated-feature-of-amazon-ec2/
>
> You also need to make sure that you can ssh from all the slaves to master
> without specifying password.
>
> Regards,
> Vaibhav Puranik
> GumGum
>
> On Sun, Nov 22, 2009 at 8:36 AM, Something Something <
> mailinglists19@gmail.com> wrote:
>
> > Seems like I am not explaining my problem correctly.
> >
> > My Key Pair file is on my machine at work which is behind a corp
> firewall.
> >  As such I can't 'scp' from the ec2 instance to my local machine at work
> to
> > 'get' the file.  So I need a way to 'send' a file from my machine to the
> > ec2
> > instance.  I tried using 'scp' (from my machine at work) to the ec2
> machine
> > but it says "Permission denied".  Does this make sense?
> >
> > May be I need to use the command line tools for EC2.  I am looking into
> > those right now, but if there's a better/easier way, please let me know.
> >
> > Thanks once again for your help.
> >
> >
> > On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <zjffdu@gmail.com> wrote:
> >
> > > The ssh is installed on ec2 by default, otherwise you have no way to
> > login
> > > to ec2
> > >
> > >
> > > Jeff Zhang
> > >
> > >
> > >
> > > On Sun, Nov 22, 2009 at 8:09 AM, Something Something <
> > > mailinglists19@gmail.com> wrote:
> > >
> > > > As per Wikipedia (http://en.wikipedia.org/wiki/Secure_copy)  "It
> does
> > so
> > > > by
> > > > connecting to the host using SSH and there executes an SCP server
> > (scp)".
> > > >
> > > > So if SSH isn't working SCP won't work, either.  In any case I tried
> to
> > > scp
> > > > but getting "Permission denied (public key)".
> > > >
> > > > Any other ideas?  Thanks.
> > > >
> > > >
> > > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <zjffdu@gmail.com>
> wrote:
> > > >
> > > > > You should scp the key-pair to EC2 machine
> > > > >
> > > > > Jeff Zhang
> > > > >
> > > > > On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
> > > > > mailinglists19@gmail.com> wrote:
> > > > >
> > > > > > Trying to start a Hadoop cluster on EC2.  (Yes, Cloudera's
> > > distribution
> > > > > > works well, but trying to do this myself so I can learn what's
> > > > happening
> > > > > > behind the scene.)
> > > > > >
> > > > > > I have a Master & a Slave.  When I start HDFS on the master, I
> get
> > a
> > > > > > message
> > > > > > saying "10.xxx.xxx.xxx (Permission denied)" - where 10.xxx is IP
> > > > address
> > > > > of
> > > > > > the slave.
> > > > > >
> > > > > > Basic problem (I think) is that I can't ssh from the master EC2
> > > > instance
> > > > > to
> > > > > > the Slave EC2 instance.  What's the best way to fix it?  I think
> I
> > > need
> > > > > the
> > > > > > "Key Pair" file on my master.  I have a key pair on my local
> > machine,
> > > > but
> > > > > > how do I transfer it to the EC2 machine?  (I know, I know, I
> > agree..
> > > I
> > > > am
> > > > > > dumb :)  Should I FTP it?
> > > > > >
> > > > > > Please help.  Thanks.
> > > > > >
> > > > >
> > > >
> > >
> >
>

--0016e6d9a2c7db061904792278a0--

From general-return-780-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 24 19:00:48 2009
Return-Path: <general-return-780-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 71534 invoked from network); 24 Nov 2009 19:00:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Nov 2009 19:00:48 -0000
Received: (qmail 13676 invoked by uid 500); 24 Nov 2009 19:00:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13596 invoked by uid 500); 24 Nov 2009 19:00:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13586 invoked by uid 99); 24 Nov 2009 19:00:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Nov 2009 19:00:47 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Nov 2009 19:00:36 +0000
Received: from thickbeside-lm.corp.yahoo.com (thickbeside-lm.corp.yahoo.com [10.72.109.129])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id nAOIwq3P065292
	for <general@hadoop.apache.org>; Tue, 24 Nov 2009 10:58:53 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:content-type:
	content-transfer-encoding:mime-version:subject:date:x-mailer;
	b=qJrWdpmAO5dwnZEH47tkFE5wvxc+1vtsVI/AjOpzyUrd9TC1gtvCQORrUEYM+KyY
Message-Id: <ADFB9B4A-E2D4-4444-8171-21523CE6AE21@yahoo-inc.com>
From: Nigel Daley <ndaley@yahoo-inc.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Hadoop Team Openings at Yahoo!
Date: Tue, 24 Nov 2009 10:58:52 -0800
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

Folks, where' looking for a number of great software engineers,  
quality engineers, performance engineers, and managers to join the  
Sunnyvale, USA and Bangalore, India Hadoop Teams at Yahoo!

Please spread the word.  Job descriptions and details on how to apply  
are here:

US: http://tinyurl.com/hadoop-jobs
India: http://tinyurl.com/hadoop-india-jobs

Cheers,
Nigel

From general-return-781-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 24 19:17:22 2009
Return-Path: <general-return-781-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76587 invoked from network); 24 Nov 2009 19:17:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Nov 2009 19:17:22 -0000
Received: (qmail 42593 invoked by uid 500); 24 Nov 2009 19:17:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42514 invoked by uid 500); 24 Nov 2009 19:17:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42504 invoked by uid 99); 24 Nov 2009 19:17:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Nov 2009 19:17:20 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of vpuranik@gmail.com designates 209.85.160.50 as permitted sender)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Nov 2009 19:17:17 +0000
Received: by pwi6 with SMTP id 6so4711780pwi.29
        for <general@hadoop.apache.org>; Tue, 24 Nov 2009 11:16:57 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=KY6JkzQwk+gxV+P7D/utaOhTNKpUUS1j2qbNKryUo1g=;
        b=sQQfiM3pu3gHqhJwb7UOuETNnVydsPrJckVYeS4QnUx73/rRbjrSdEYYiCcj+6/RXY
         e4cbnH0sgbem/aWE8gdt7ULgfh5LPTqWQmlcSj2X0Zy+JxUDx6PCVuAzVQC/bBMflT1a
         bLB/vOXIuj4MOENV2uF3X2pjzFPs+IhX61CTw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=EeDMKm9k1YnfwmyNeSqac6L1lWwCjBCr5Wx/ElQ7RQftFZV2b2pf8PdY2B7TbxvELA
         WHuGqwpvNENKanyNycpzQI6PbHRaPFOT0g2ToKZwseLsOU0EkxkJac/Q3mRCg/2Gb7A3
         GQ11QSz8jsWZYFsl4z9l7f8Na7O4AG+lOGbc0=
MIME-Version: 1.0
Received: by 10.115.112.5 with SMTP id p5mr12805998wam.223.1259090217618; Tue, 
	24 Nov 2009 11:16:57 -0800 (PST)
In-Reply-To: <1eabbac30911241054w6ef4852n41ddf93198c27acd@mail.gmail.com>
References: <1eabbac30911211657i7812c67bs3a1565b0537b418f@mail.gmail.com>
	 <8211a1320911211812q64a7fb85jcdcda10be650d4ef@mail.gmail.com>
	 <1eabbac30911220809l2b923866u4c79e4a37e2d9ad4@mail.gmail.com>
	 <8211a1320911220819j8d33c10te498c18362e825e0@mail.gmail.com>
	 <1eabbac30911220836t76895c04x6f7973d759c60c18@mail.gmail.com>
	 <2eef7bd70911230810l690624a1vd4fe8c208279293c@mail.gmail.com>
	 <1eabbac30911241054w6ef4852n41ddf93198c27acd@mail.gmail.com>
Date: Tue, 24 Nov 2009 11:16:57 -0800
Message-ID: <2eef7bd70911241116n6f81550fpfeec6b88c382e0b4@mail.gmail.com>
Subject: Re: Starting Hadoop cluster on EC2
From: Vaibhav Puranik <vpuranik@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e649bc5e294f0b047922c900

--0016e649bc5e294f0b047922c900
Content-Type: text/plain; charset=ISO-8859-1

You need to be able to SSH without specifying the keypair from a slave to
master (I think reverse it true too, but I am not sure).

In order to achieve that you need to add the keypair  you are using to your
auhtorized_keys file in the home dir/.ssh folder.

Once you setup passphraseless ssh, it should start working. Furthrmore,
after you set this up, try to ssh manully (without key) and if it asks you
the following message - say yes:

The authenticity of host
'ec2-147-127-186-243.compute-1.amazonaws.com(147.127.186.243)' can't
be established.
RSA key fingerprint is 4b:63:e2:23:16:6f:b6:99:de:34:f6:9b:f5:55:73:8b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added
'ec2-147-127-186-243.compute-1.amazonaws.com,147.127.186.243'
(RSA) to the list of known hosts.

In other words, you have to make sure that the machine you are trying to ssh
is in your known_hosts file.

Regards,
Vaibhav

On Tue, Nov 24, 2009 at 10:54 AM, Something Something <
mailinglists19@gmail.com> wrote:

> Thanks, Vaibhav.  That was helpful.  I am able to scp from my local machine
> after I opened port 22 under my security group.
>
> But my cluster is still not starting as expected.  I mean... on my Master
> when I run:
>
> ./start-all.sh
>
> I get messages saying...
>
> 10.252.xxx.xxx: Permission denied (publickey).
>
> where 10.252.xxx.xxx is the ip address of my Slave.
>
>
> I can ssh from my Slave to master if I do this:
>
> ssh -i <keypairname>.pem <Master's ip address>
>
>
> Any suggestions?  Thanks for your help.
>
>
>
>
> On Mon, Nov 23, 2009 at 8:10 AM, Vaibhav Puranik <vpuranik@gmail.com>
> wrote:
>
> > In order to use scp from your machine to an ec2 instance you need to use
> > scp
> > -i and give it the path of your keypair.
> >
> > scp -i /path/my/ec2/keypair  <souce> <destination>
> >
> > Secondly in order to configure an ec2 cluster for Hadoop, you need to
> make
> > a
> > security group (let's call it hadoop) and give hadoop access to hadoop.
> >
> > You can read more about security groups here -
> >
> >
> http://somic.org/2009/09/21/security-groups-most-underappreciated-feature-of-amazon-ec2/
> >
> > You also need to make sure that you can ssh from all the slaves to master
> > without specifying password.
> >
> > Regards,
> > Vaibhav Puranik
> > GumGum
> >
> > On Sun, Nov 22, 2009 at 8:36 AM, Something Something <
> > mailinglists19@gmail.com> wrote:
> >
> > > Seems like I am not explaining my problem correctly.
> > >
> > > My Key Pair file is on my machine at work which is behind a corp
> > firewall.
> > >  As such I can't 'scp' from the ec2 instance to my local machine at
> work
> > to
> > > 'get' the file.  So I need a way to 'send' a file from my machine to
> the
> > > ec2
> > > instance.  I tried using 'scp' (from my machine at work) to the ec2
> > machine
> > > but it says "Permission denied".  Does this make sense?
> > >
> > > May be I need to use the command line tools for EC2.  I am looking into
> > > those right now, but if there's a better/easier way, please let me
> know.
> > >
> > > Thanks once again for your help.
> > >
> > >
> > > On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <zjffdu@gmail.com> wrote:
> > >
> > > > The ssh is installed on ec2 by default, otherwise you have no way to
> > > login
> > > > to ec2
> > > >
> > > >
> > > > Jeff Zhang
> > > >
> > > >
> > > >
> > > > On Sun, Nov 22, 2009 at 8:09 AM, Something Something <
> > > > mailinglists19@gmail.com> wrote:
> > > >
> > > > > As per Wikipedia (http://en.wikipedia.org/wiki/Secure_copy)  "It
> > does
> > > so
> > > > > by
> > > > > connecting to the host using SSH and there executes an SCP server
> > > (scp)".
> > > > >
> > > > > So if SSH isn't working SCP won't work, either.  In any case I
> tried
> > to
> > > > scp
> > > > > but getting "Permission denied (public key)".
> > > > >
> > > > > Any other ideas?  Thanks.
> > > > >
> > > > >
> > > > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <zjffdu@gmail.com>
> > wrote:
> > > > >
> > > > > > You should scp the key-pair to EC2 machine
> > > > > >
> > > > > > Jeff Zhang
> > > > > >
> > > > > > On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
> > > > > > mailinglists19@gmail.com> wrote:
> > > > > >
> > > > > > > Trying to start a Hadoop cluster on EC2.  (Yes, Cloudera's
> > > > distribution
> > > > > > > works well, but trying to do this myself so I can learn what's
> > > > > happening
> > > > > > > behind the scene.)
> > > > > > >
> > > > > > > I have a Master & a Slave.  When I start HDFS on the master, I
> > get
> > > a
> > > > > > > message
> > > > > > > saying "10.xxx.xxx.xxx (Permission denied)" - where 10.xxx is
> IP
> > > > > address
> > > > > > of
> > > > > > > the slave.
> > > > > > >
> > > > > > > Basic problem (I think) is that I can't ssh from the master EC2
> > > > > instance
> > > > > > to
> > > > > > > the Slave EC2 instance.  What's the best way to fix it?  I
> think
> > I
> > > > need
> > > > > > the
> > > > > > > "Key Pair" file on my master.  I have a key pair on my local
> > > machine,
> > > > > but
> > > > > > > how do I transfer it to the EC2 machine?  (I know, I know, I
> > > agree..
> > > > I
> > > > > am
> > > > > > > dumb :)  Should I FTP it?
> > > > > > >
> > > > > > > Please help.  Thanks.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

--0016e649bc5e294f0b047922c900--

From general-return-782-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 24 19:36:30 2009
Return-Path: <general-return-782-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82786 invoked from network); 24 Nov 2009 19:36:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Nov 2009 19:36:30 -0000
Received: (qmail 75305 invoked by uid 500); 24 Nov 2009 19:36:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75216 invoked by uid 500); 24 Nov 2009 19:36:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75206 invoked by uid 99); 24 Nov 2009 19:36:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Nov 2009 19:36:29 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of abhilaash_cse@tce.edu designates 210.212.252.72 as permitted sender)
Received: from [210.212.252.72] (HELO tce.edu) (210.212.252.72)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Nov 2009 19:36:19 +0000
Received: from [127.0.0.1] (helo=mail.tce.edu)
	by tce.edu with esmtp (Exim 4.63)
	(envelope-from <abhilaash_cse@tce.edu>)
	id 1ND1B3-0006To-Be
	for general@hadoop.apache.org; Wed, 25 Nov 2009 01:05:56 +0530
Received: from 59.96.24.33
        (SquirrelMail authenticated user abhilaash_cse)
        by mail.tce.edu with HTTP;
        Wed, 25 Nov 2009 01:05:53 +0530
Message-ID: <faeb26a0b719c863ee5e44bffb89ff79.squirrel@mail.tce.edu>
In-Reply-To: <9FD0B40E-F7BA-4CFC-9BE9-3D52758E4EC8@yahoo-inc.com>
References: <6efd398094dc482ed37a662080836354.squirrel@mail.tce.edu>
    <9FD0B40E-F7BA-4CFC-9BE9-3D52758E4EC8@yahoo-inc.com>
Date: Wed, 25 Nov 2009 01:05:53 +0530
Subject: Re: Doubts Regarding Capacity Scheduler's Function
From: "Abhilaash" <abhilaash_cse@tce.edu>
To: general@hadoop.apache.org
User-Agent: SquirrelMail/1.4.19
MIME-Version: 1.0
Content-Type: text/plain;charset=iso-8859-1
Content-Transfer-Encoding: 8bit
X-Priority: 3 (Normal)
Importance: Normal
X-XheaderVersion: 1.1
X-UserAgent: 
X-Spam-Score: -1.4 (-)
X-Virus-Checked: Checked by ClamAV on apache.org
X-Old-Spam-Status: No



Hi,

>> defined queues?
>
> Each job needs to have the mapred.job.queue.name set to the right
> queue for the CS. It defaults to "default" queue.
>

I have set that property to a user defined queue Queue1 and now my
submitted jobs are scheduled to Queue1,but I want to know the
configuration example to schedule job1 to default ,then job2 to
Queue2,etc.,

whether this can be done?

>
> http://hadoop.apache.org/common/docs/r0.20.0/capacity_scheduler.html -
> hope this helps.

Thanks,it helped me a lot.


Regards,
Abhilaash.

-- 
Nothing is impossible because impossible says i'm possible...


-----------------------------------------
This email was sent using TCEMail Service.
Thiagarajar College of Engineering
Madurai-625 015, India


From general-return-783-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 24 21:38:52 2009
Return-Path: <general-return-783-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22590 invoked from network); 24 Nov 2009 21:38:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Nov 2009 21:38:52 -0000
Received: (qmail 82654 invoked by uid 500); 24 Nov 2009 21:38:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82556 invoked by uid 500); 24 Nov 2009 21:38:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82546 invoked by uid 99); 24 Nov 2009 21:38:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Nov 2009 21:38:50 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mailinglists19@gmail.com designates 209.85.220.224 as permitted sender)
Received: from [209.85.220.224] (HELO mail-fx0-f224.google.com) (209.85.220.224)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Nov 2009 21:38:47 +0000
Received: by fxm24 with SMTP id 24so4141392fxm.11
        for <general@hadoop.apache.org>; Tue, 24 Nov 2009 13:38:25 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=pIkFXzbhwq3Adgw0YVYyenXakBSXU3DRlshjJZUNyGE=;
        b=dzVPJMG7VhG9tyVvq+YDWb9XddA+YuG1Dlj+yQpL1I4rfFWQf2iWR7CsAs6ZL61ZrA
         TECBkZXTc+pG6L2H+RktCSkrFQJ8CqEMdI19KRar1Xy8QOLJtqRO+rDLlMVFRdJjJxEJ
         BNIWBA8yfueS5tEDj7mxbAMVsWD1EWu7Hkh44=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=v7qTkQZkykA5PmE2xF4Vx0flfwtraINhyA4KuUaurWgApvx/WKz3hg+EvMN5UYD6Xk
         ShvESFCR8zo1Hbk7OkIP36k/7V9YJJmKOwBKcXYC//6EORy+fBjZMRNRWq4jZvW7/E1/
         zzJazg58an2yGZmyOCa523m9ZrE4UdPaXXNwU=
MIME-Version: 1.0
Received: by 10.216.91.82 with SMTP id g60mr2156872wef.98.1259098704224; Tue, 
	24 Nov 2009 13:38:24 -0800 (PST)
In-Reply-To: <2eef7bd70911241116n6f81550fpfeec6b88c382e0b4@mail.gmail.com>
References: <1eabbac30911211657i7812c67bs3a1565b0537b418f@mail.gmail.com>
	 <8211a1320911211812q64a7fb85jcdcda10be650d4ef@mail.gmail.com>
	 <1eabbac30911220809l2b923866u4c79e4a37e2d9ad4@mail.gmail.com>
	 <8211a1320911220819j8d33c10te498c18362e825e0@mail.gmail.com>
	 <1eabbac30911220836t76895c04x6f7973d759c60c18@mail.gmail.com>
	 <2eef7bd70911230810l690624a1vd4fe8c208279293c@mail.gmail.com>
	 <1eabbac30911241054w6ef4852n41ddf93198c27acd@mail.gmail.com>
	 <2eef7bd70911241116n6f81550fpfeec6b88c382e0b4@mail.gmail.com>
Date: Tue, 24 Nov 2009 13:38:24 -0800
Message-ID: <1eabbac30911241338g1067b99dra633f21f2d0cfd10@mail.gmail.com>
Subject: Re: Starting Hadoop cluster on EC2
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d7858e00a10b047924c37f

--0016e6d7858e00a10b047924c37f
Content-Type: text/plain; charset=ISO-8859-1

Sorry.. but not sure what you mean by... "In order to achieve that you need
to add the keypair  you are using to your
auhtorized_keys file in the home dir/.ssh folder."


I tried this....

cat KeyPair.pem >> /root/.ssh/authorized_keys

Is that what you meant by "add the keypair to authorized_keys"?

Still can't ssh from Slave to Master.  It says.....  Permission denied
(publickey).


Sorry, I am a newbie when it comes to such environment issues.  Thanks for
your patience.  It's greatly appreciated.


On Tue, Nov 24, 2009 at 11:16 AM, Vaibhav Puranik <vpuranik@gmail.com>wrote:

> You need to be able to SSH without specifying the keypair from a slave to
> master (I think reverse it true too, but I am not sure).
>
> In order to achieve that you need to add the keypair  you are using to your
> auhtorized_keys file in the home dir/.ssh folder.
>
> Once you setup passphraseless ssh, it should start working. Furthrmore,
> after you set this up, try to ssh manully (without key) and if it asks you
> the following message - say yes:
>
> The authenticity of host
> 'ec2-147-127-186-243.compute-1.amazonaws.com(147.127.186.243)' can't
> be established.
> RSA key fingerprint is 4b:63:e2:23:16:6f:b6:99:de:34:f6:9b:f5:55:73:8b.
> Are you sure you want to continue connecting (yes/no)? yes
> Warning: Permanently added
> 'ec2-147-127-186-243.compute-1.amazonaws.com,147.127.186.243'
> (RSA) to the list of known hosts.
>
> In other words, you have to make sure that the machine you are trying to
> ssh
> is in your known_hosts file.
>
> Regards,
> Vaibhav
>
> On Tue, Nov 24, 2009 at 10:54 AM, Something Something <
> mailinglists19@gmail.com> wrote:
>
> > Thanks, Vaibhav.  That was helpful.  I am able to scp from my local
> machine
> > after I opened port 22 under my security group.
> >
> > But my cluster is still not starting as expected.  I mean... on my Master
> > when I run:
> >
> > ./start-all.sh
> >
> > I get messages saying...
> >
> > 10.252.xxx.xxx: Permission denied (publickey).
> >
> > where 10.252.xxx.xxx is the ip address of my Slave.
> >
> >
> > I can ssh from my Slave to master if I do this:
> >
> > ssh -i <keypairname>.pem <Master's ip address>
> >
> >
> > Any suggestions?  Thanks for your help.
> >
> >
> >
> >
> > On Mon, Nov 23, 2009 at 8:10 AM, Vaibhav Puranik <vpuranik@gmail.com>
> > wrote:
> >
> > > In order to use scp from your machine to an ec2 instance you need to
> use
> > > scp
> > > -i and give it the path of your keypair.
> > >
> > > scp -i /path/my/ec2/keypair  <souce> <destination>
> > >
> > > Secondly in order to configure an ec2 cluster for Hadoop, you need to
> > make
> > > a
> > > security group (let's call it hadoop) and give hadoop access to hadoop.
> > >
> > > You can read more about security groups here -
> > >
> > >
> >
> http://somic.org/2009/09/21/security-groups-most-underappreciated-feature-of-amazon-ec2/
> > >
> > > You also need to make sure that you can ssh from all the slaves to
> master
> > > without specifying password.
> > >
> > > Regards,
> > > Vaibhav Puranik
> > > GumGum
> > >
> > > On Sun, Nov 22, 2009 at 8:36 AM, Something Something <
> > > mailinglists19@gmail.com> wrote:
> > >
> > > > Seems like I am not explaining my problem correctly.
> > > >
> > > > My Key Pair file is on my machine at work which is behind a corp
> > > firewall.
> > > >  As such I can't 'scp' from the ec2 instance to my local machine at
> > work
> > > to
> > > > 'get' the file.  So I need a way to 'send' a file from my machine to
> > the
> > > > ec2
> > > > instance.  I tried using 'scp' (from my machine at work) to the ec2
> > > machine
> > > > but it says "Permission denied".  Does this make sense?
> > > >
> > > > May be I need to use the command line tools for EC2.  I am looking
> into
> > > > those right now, but if there's a better/easier way, please let me
> > know.
> > > >
> > > > Thanks once again for your help.
> > > >
> > > >
> > > > On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <zjffdu@gmail.com>
> wrote:
> > > >
> > > > > The ssh is installed on ec2 by default, otherwise you have no way
> to
> > > > login
> > > > > to ec2
> > > > >
> > > > >
> > > > > Jeff Zhang
> > > > >
> > > > >
> > > > >
> > > > > On Sun, Nov 22, 2009 at 8:09 AM, Something Something <
> > > > > mailinglists19@gmail.com> wrote:
> > > > >
> > > > > > As per Wikipedia (http://en.wikipedia.org/wiki/Secure_copy)  "It
> > > does
> > > > so
> > > > > > by
> > > > > > connecting to the host using SSH and there executes an SCP server
> > > > (scp)".
> > > > > >
> > > > > > So if SSH isn't working SCP won't work, either.  In any case I
> > tried
> > > to
> > > > > scp
> > > > > > but getting "Permission denied (public key)".
> > > > > >
> > > > > > Any other ideas?  Thanks.
> > > > > >
> > > > > >
> > > > > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <zjffdu@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > You should scp the key-pair to EC2 machine
> > > > > > >
> > > > > > > Jeff Zhang
> > > > > > >
> > > > > > > On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
> > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > >
> > > > > > > > Trying to start a Hadoop cluster on EC2.  (Yes, Cloudera's
> > > > > distribution
> > > > > > > > works well, but trying to do this myself so I can learn
> what's
> > > > > > happening
> > > > > > > > behind the scene.)
> > > > > > > >
> > > > > > > > I have a Master & a Slave.  When I start HDFS on the master,
> I
> > > get
> > > > a
> > > > > > > > message
> > > > > > > > saying "10.xxx.xxx.xxx (Permission denied)" - where 10.xxx is
> > IP
> > > > > > address
> > > > > > > of
> > > > > > > > the slave.
> > > > > > > >
> > > > > > > > Basic problem (I think) is that I can't ssh from the master
> EC2
> > > > > > instance
> > > > > > > to
> > > > > > > > the Slave EC2 instance.  What's the best way to fix it?  I
> > think
> > > I
> > > > > need
> > > > > > > the
> > > > > > > > "Key Pair" file on my master.  I have a key pair on my local
> > > > machine,
> > > > > > but
> > > > > > > > how do I transfer it to the EC2 machine?  (I know, I know, I
> > > > agree..
> > > > > I
> > > > > > am
> > > > > > > > dumb :)  Should I FTP it?
> > > > > > > >
> > > > > > > > Please help.  Thanks.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

--0016e6d7858e00a10b047924c37f--

From general-return-784-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 24 21:44:36 2009
Return-Path: <general-return-784-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25065 invoked from network); 24 Nov 2009 21:44:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Nov 2009 21:44:36 -0000
Received: (qmail 96267 invoked by uid 500); 24 Nov 2009 21:44:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96173 invoked by uid 500); 24 Nov 2009 21:44:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96163 invoked by uid 99); 24 Nov 2009 21:44:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Nov 2009 21:44:35 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of vpuranik@gmail.com designates 209.85.160.50 as permitted sender)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Nov 2009 21:44:32 +0000
Received: by pwi6 with SMTP id 6so4806912pwi.29
        for <general@hadoop.apache.org>; Tue, 24 Nov 2009 13:44:11 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=JDTjYXJhl2gBqjLjhMXxipSNMs9WohilB5f3Px6QPQs=;
        b=dG6vy3g4yPeSoJPlt7Ng4cwsVIiJqq2RVNvCGrt2B4U7CiHhwc0eErVZqO8EJFPHAC
         APQWiS5OA03HvztRqmSKjbH9PGyhzAozAKbMAe2ZjiSU6CCPq7Q5hgKMW8ZmEZfA6n+M
         ZWQ2JcJLUctFARMvbdYjuKC+yrW35cwFL48wc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=UfgdRoLl/HiBGy7seeuxgpOJgJhzfIs1Dh3zzCd8f3m5TVTtFVube1T+/kI/8Bb1hH
         JtuDtz3/Fe+D53ndEIjxxMosJcfR6smRSDwSMprzMXnDABo/XyLuHD+OE5uRfE+3cxAt
         M/uCElhQ1TAKb+mZsDVJKDpQoWNmfZ5qp5mnQ=
MIME-Version: 1.0
Received: by 10.114.214.22 with SMTP id m22mr13276236wag.218.1259099050829; 
	Tue, 24 Nov 2009 13:44:10 -0800 (PST)
In-Reply-To: <1eabbac30911241338g1067b99dra633f21f2d0cfd10@mail.gmail.com>
References: <1eabbac30911211657i7812c67bs3a1565b0537b418f@mail.gmail.com>
	 <8211a1320911211812q64a7fb85jcdcda10be650d4ef@mail.gmail.com>
	 <1eabbac30911220809l2b923866u4c79e4a37e2d9ad4@mail.gmail.com>
	 <8211a1320911220819j8d33c10te498c18362e825e0@mail.gmail.com>
	 <1eabbac30911220836t76895c04x6f7973d759c60c18@mail.gmail.com>
	 <2eef7bd70911230810l690624a1vd4fe8c208279293c@mail.gmail.com>
	 <1eabbac30911241054w6ef4852n41ddf93198c27acd@mail.gmail.com>
	 <2eef7bd70911241116n6f81550fpfeec6b88c382e0b4@mail.gmail.com>
	 <1eabbac30911241338g1067b99dra633f21f2d0cfd10@mail.gmail.com>
Date: Tue, 24 Nov 2009 13:44:10 -0800
Message-ID: <2eef7bd70911241344p4225190fkaaafc160a5c9bb80@mail.gmail.com>
Subject: Re: Starting Hadoop cluster on EC2
From: Vaibhav Puranik <vpuranik@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64aee68a966d4047924d7b9

--0016e64aee68a966d4047924d7b9
Content-Type: text/plain; charset=ISO-8859-1

Yes, I pretty much meant that.

Add your master's public key to authorized_keys on slaves too.

Regards,
Vaibhav



On Tue, Nov 24, 2009 at 1:38 PM, Something Something <
mailinglists19@gmail.com> wrote:

> Sorry.. but not sure what you mean by... "In order to achieve that you need
> to add the keypair  you are using to your
> auhtorized_keys file in the home dir/.ssh folder."
>
>
> I tried this....
>
> cat KeyPair.pem >> /root/.ssh/authorized_keys
>
> Is that what you meant by "add the keypair to authorized_keys"?
>
> Still can't ssh from Slave to Master.  It says.....  Permission denied
> (publickey).
>
>
> Sorry, I am a newbie when it comes to such environment issues.  Thanks for
> your patience.  It's greatly appreciated.
>
>
> On Tue, Nov 24, 2009 at 11:16 AM, Vaibhav Puranik <vpuranik@gmail.com
> >wrote:
>
> > You need to be able to SSH without specifying the keypair from a slave to
> > master (I think reverse it true too, but I am not sure).
> >
> > In order to achieve that you need to add the keypair  you are using to
> your
> > auhtorized_keys file in the home dir/.ssh folder.
> >
> > Once you setup passphraseless ssh, it should start working. Furthrmore,
> > after you set this up, try to ssh manully (without key) and if it asks
> you
> > the following message - say yes:
> >
> > The authenticity of host
> > 'ec2-147-127-186-243.compute-1.amazonaws.com(147.127.186.243)' can't
> > be established.
> > RSA key fingerprint is 4b:63:e2:23:16:6f:b6:99:de:34:f6:9b:f5:55:73:8b.
> > Are you sure you want to continue connecting (yes/no)? yes
> > Warning: Permanently added
> > 'ec2-147-127-186-243.compute-1.amazonaws.com,147.127.186.243'
> > (RSA) to the list of known hosts.
> >
> > In other words, you have to make sure that the machine you are trying to
> > ssh
> > is in your known_hosts file.
> >
> > Regards,
> > Vaibhav
> >
> > On Tue, Nov 24, 2009 at 10:54 AM, Something Something <
> > mailinglists19@gmail.com> wrote:
> >
> > > Thanks, Vaibhav.  That was helpful.  I am able to scp from my local
> > machine
> > > after I opened port 22 under my security group.
> > >
> > > But my cluster is still not starting as expected.  I mean... on my
> Master
> > > when I run:
> > >
> > > ./start-all.sh
> > >
> > > I get messages saying...
> > >
> > > 10.252.xxx.xxx: Permission denied (publickey).
> > >
> > > where 10.252.xxx.xxx is the ip address of my Slave.
> > >
> > >
> > > I can ssh from my Slave to master if I do this:
> > >
> > > ssh -i <keypairname>.pem <Master's ip address>
> > >
> > >
> > > Any suggestions?  Thanks for your help.
> > >
> > >
> > >
> > >
> > > On Mon, Nov 23, 2009 at 8:10 AM, Vaibhav Puranik <vpuranik@gmail.com>
> > > wrote:
> > >
> > > > In order to use scp from your machine to an ec2 instance you need to
> > use
> > > > scp
> > > > -i and give it the path of your keypair.
> > > >
> > > > scp -i /path/my/ec2/keypair  <souce> <destination>
> > > >
> > > > Secondly in order to configure an ec2 cluster for Hadoop, you need to
> > > make
> > > > a
> > > > security group (let's call it hadoop) and give hadoop access to
> hadoop.
> > > >
> > > > You can read more about security groups here -
> > > >
> > > >
> > >
> >
> http://somic.org/2009/09/21/security-groups-most-underappreciated-feature-of-amazon-ec2/
> > > >
> > > > You also need to make sure that you can ssh from all the slaves to
> > master
> > > > without specifying password.
> > > >
> > > > Regards,
> > > > Vaibhav Puranik
> > > > GumGum
> > > >
> > > > On Sun, Nov 22, 2009 at 8:36 AM, Something Something <
> > > > mailinglists19@gmail.com> wrote:
> > > >
> > > > > Seems like I am not explaining my problem correctly.
> > > > >
> > > > > My Key Pair file is on my machine at work which is behind a corp
> > > > firewall.
> > > > >  As such I can't 'scp' from the ec2 instance to my local machine at
> > > work
> > > > to
> > > > > 'get' the file.  So I need a way to 'send' a file from my machine
> to
> > > the
> > > > > ec2
> > > > > instance.  I tried using 'scp' (from my machine at work) to the ec2
> > > > machine
> > > > > but it says "Permission denied".  Does this make sense?
> > > > >
> > > > > May be I need to use the command line tools for EC2.  I am looking
> > into
> > > > > those right now, but if there's a better/easier way, please let me
> > > know.
> > > > >
> > > > > Thanks once again for your help.
> > > > >
> > > > >
> > > > > On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <zjffdu@gmail.com>
> > wrote:
> > > > >
> > > > > > The ssh is installed on ec2 by default, otherwise you have no way
> > to
> > > > > login
> > > > > > to ec2
> > > > > >
> > > > > >
> > > > > > Jeff Zhang
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sun, Nov 22, 2009 at 8:09 AM, Something Something <
> > > > > > mailinglists19@gmail.com> wrote:
> > > > > >
> > > > > > > As per Wikipedia (http://en.wikipedia.org/wiki/Secure_copy)
>  "It
> > > > does
> > > > > so
> > > > > > > by
> > > > > > > connecting to the host using SSH and there executes an SCP
> server
> > > > > (scp)".
> > > > > > >
> > > > > > > So if SSH isn't working SCP won't work, either.  In any case I
> > > tried
> > > > to
> > > > > > scp
> > > > > > > but getting "Permission denied (public key)".
> > > > > > >
> > > > > > > Any other ideas?  Thanks.
> > > > > > >
> > > > > > >
> > > > > > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <zjffdu@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > You should scp the key-pair to EC2 machine
> > > > > > > >
> > > > > > > > Jeff Zhang
> > > > > > > >
> > > > > > > > On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
> > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > Trying to start a Hadoop cluster on EC2.  (Yes, Cloudera's
> > > > > > distribution
> > > > > > > > > works well, but trying to do this myself so I can learn
> > what's
> > > > > > > happening
> > > > > > > > > behind the scene.)
> > > > > > > > >
> > > > > > > > > I have a Master & a Slave.  When I start HDFS on the
> master,
> > I
> > > > get
> > > > > a
> > > > > > > > > message
> > > > > > > > > saying "10.xxx.xxx.xxx (Permission denied)" - where 10.xxx
> is
> > > IP
> > > > > > > address
> > > > > > > > of
> > > > > > > > > the slave.
> > > > > > > > >
> > > > > > > > > Basic problem (I think) is that I can't ssh from the master
> > EC2
> > > > > > > instance
> > > > > > > > to
> > > > > > > > > the Slave EC2 instance.  What's the best way to fix it?  I
> > > think
> > > > I
> > > > > > need
> > > > > > > > the
> > > > > > > > > "Key Pair" file on my master.  I have a key pair on my
> local
> > > > > machine,
> > > > > > > but
> > > > > > > > > how do I transfer it to the EC2 machine?  (I know, I know,
> I
> > > > > agree..
> > > > > > I
> > > > > > > am
> > > > > > > > > dumb :)  Should I FTP it?
> > > > > > > > >
> > > > > > > > > Please help.  Thanks.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

--0016e64aee68a966d4047924d7b9--

From general-return-785-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 24 22:07:25 2009
Return-Path: <general-return-785-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 32434 invoked from network); 24 Nov 2009 22:07:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Nov 2009 22:07:25 -0000
Received: (qmail 31784 invoked by uid 500); 24 Nov 2009 22:07:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31703 invoked by uid 500); 24 Nov 2009 22:07:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31693 invoked by uid 99); 24 Nov 2009 22:07:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Nov 2009 22:07:24 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mailinglists19@gmail.com designates 209.85.219.217 as permitted sender)
Received: from [209.85.219.217] (HELO mail-ew0-f217.google.com) (209.85.219.217)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Nov 2009 22:07:21 +0000
Received: by ewy9 with SMTP id 9so1072403ewy.11
        for <general@hadoop.apache.org>; Tue, 24 Nov 2009 14:06:59 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=HzlV3Cjj434bhw5hYB/+gDVXVbotu78M4g1iwPyMhqk=;
        b=vaf6cgSOjsEnARjRNi0/aD71JTH5f6Z4TR9SYvsZvLxDstAwZ1lUwTp/Xyf2I0MrSe
         3zeCBYhsa5ZxkhS5pLW7DCmGX62JIt3yDpEt6JGif/MH+hzedXt7Uc1EU0O6NZoZE/+n
         vhIEcJPaGhXEfXitVR5Yfe9tpMGb8xITIsBXw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=oyWoz+VEsrmaX77rV6lo4eTIH+FIKPVLvOpxHbteKewagbmd0GlX9udsstGcayW+KH
         XpcHk+nHx4D8wdGFovhpagdvLFrspLKpmchY/CK04UJOwAkl8xFaFZX0OPcRGPJgF5Ar
         BXbzZYE2cCP5dvGVqgC5pguZfSbCDsqzRcD0E=
MIME-Version: 1.0
Received: by 10.216.90.13 with SMTP id d13mr2225499wef.130.1259100418187; Tue, 
	24 Nov 2009 14:06:58 -0800 (PST)
In-Reply-To: <2eef7bd70911241344p4225190fkaaafc160a5c9bb80@mail.gmail.com>
References: <1eabbac30911211657i7812c67bs3a1565b0537b418f@mail.gmail.com>
	 <8211a1320911211812q64a7fb85jcdcda10be650d4ef@mail.gmail.com>
	 <1eabbac30911220809l2b923866u4c79e4a37e2d9ad4@mail.gmail.com>
	 <8211a1320911220819j8d33c10te498c18362e825e0@mail.gmail.com>
	 <1eabbac30911220836t76895c04x6f7973d759c60c18@mail.gmail.com>
	 <2eef7bd70911230810l690624a1vd4fe8c208279293c@mail.gmail.com>
	 <1eabbac30911241054w6ef4852n41ddf93198c27acd@mail.gmail.com>
	 <2eef7bd70911241116n6f81550fpfeec6b88c382e0b4@mail.gmail.com>
	 <1eabbac30911241338g1067b99dra633f21f2d0cfd10@mail.gmail.com>
	 <2eef7bd70911241344p4225190fkaaafc160a5c9bb80@mail.gmail.com>
Date: Tue, 24 Nov 2009 14:06:58 -0800
Message-ID: <1eabbac30911241406v1a684661va156ab6a856819c3@mail.gmail.com>
Subject: Re: Starting Hadoop cluster on EC2
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d7dff529a0c50479252953

--0016e6d7dff529a0c50479252953
Content-Type: text/plain; charset=ISO-8859-1

Awesome.  Thanks, Vaibhav.  Made progress.  No error messages from
./all-start.sh

I will test to see if the cluster is deployed properly.

Question:  On the slave, when I do:

ps -eaf | grep 'hadoop'

I don't see any processes.  Shouldn't I see 2 processes - one for Datanode
and the other for TaskTracker?

Please let me know.  Thanks again.


On Tue, Nov 24, 2009 at 1:44 PM, Vaibhav Puranik <vpuranik@gmail.com> wrote:

> Yes, I pretty much meant that.
>
> Add your master's public key to authorized_keys on slaves too.
>
> Regards,
> Vaibhav
>
>
>
> On Tue, Nov 24, 2009 at 1:38 PM, Something Something <
> mailinglists19@gmail.com> wrote:
>
> > Sorry.. but not sure what you mean by... "In order to achieve that you
> need
> > to add the keypair  you are using to your
> > auhtorized_keys file in the home dir/.ssh folder."
> >
> >
> > I tried this....
> >
> > cat KeyPair.pem >> /root/.ssh/authorized_keys
> >
> > Is that what you meant by "add the keypair to authorized_keys"?
> >
> > Still can't ssh from Slave to Master.  It says.....  Permission denied
> > (publickey).
> >
> >
> > Sorry, I am a newbie when it comes to such environment issues.  Thanks
> for
> > your patience.  It's greatly appreciated.
> >
> >
> > On Tue, Nov 24, 2009 at 11:16 AM, Vaibhav Puranik <vpuranik@gmail.com
> > >wrote:
> >
> > > You need to be able to SSH without specifying the keypair from a slave
> to
> > > master (I think reverse it true too, but I am not sure).
> > >
> > > In order to achieve that you need to add the keypair  you are using to
> > your
> > > auhtorized_keys file in the home dir/.ssh folder.
> > >
> > > Once you setup passphraseless ssh, it should start working. Furthrmore,
> > > after you set this up, try to ssh manully (without key) and if it asks
> > you
> > > the following message - say yes:
> > >
> > > The authenticity of host
> > > 'ec2-147-127-186-243.compute-1.amazonaws.com(147.127.186.243)' can't
> > > be established.
> > > RSA key fingerprint is 4b:63:e2:23:16:6f:b6:99:de:34:f6:9b:f5:55:73:8b.
> > > Are you sure you want to continue connecting (yes/no)? yes
> > > Warning: Permanently added
> > > 'ec2-147-127-186-243.compute-1.amazonaws.com,147.127.186.243'
> > > (RSA) to the list of known hosts.
> > >
> > > In other words, you have to make sure that the machine you are trying
> to
> > > ssh
> > > is in your known_hosts file.
> > >
> > > Regards,
> > > Vaibhav
> > >
> > > On Tue, Nov 24, 2009 at 10:54 AM, Something Something <
> > > mailinglists19@gmail.com> wrote:
> > >
> > > > Thanks, Vaibhav.  That was helpful.  I am able to scp from my local
> > > machine
> > > > after I opened port 22 under my security group.
> > > >
> > > > But my cluster is still not starting as expected.  I mean... on my
> > Master
> > > > when I run:
> > > >
> > > > ./start-all.sh
> > > >
> > > > I get messages saying...
> > > >
> > > > 10.252.xxx.xxx: Permission denied (publickey).
> > > >
> > > > where 10.252.xxx.xxx is the ip address of my Slave.
> > > >
> > > >
> > > > I can ssh from my Slave to master if I do this:
> > > >
> > > > ssh -i <keypairname>.pem <Master's ip address>
> > > >
> > > >
> > > > Any suggestions?  Thanks for your help.
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Nov 23, 2009 at 8:10 AM, Vaibhav Puranik <vpuranik@gmail.com
> >
> > > > wrote:
> > > >
> > > > > In order to use scp from your machine to an ec2 instance you need
> to
> > > use
> > > > > scp
> > > > > -i and give it the path of your keypair.
> > > > >
> > > > > scp -i /path/my/ec2/keypair  <souce> <destination>
> > > > >
> > > > > Secondly in order to configure an ec2 cluster for Hadoop, you need
> to
> > > > make
> > > > > a
> > > > > security group (let's call it hadoop) and give hadoop access to
> > hadoop.
> > > > >
> > > > > You can read more about security groups here -
> > > > >
> > > > >
> > > >
> > >
> >
> http://somic.org/2009/09/21/security-groups-most-underappreciated-feature-of-amazon-ec2/
> > > > >
> > > > > You also need to make sure that you can ssh from all the slaves to
> > > master
> > > > > without specifying password.
> > > > >
> > > > > Regards,
> > > > > Vaibhav Puranik
> > > > > GumGum
> > > > >
> > > > > On Sun, Nov 22, 2009 at 8:36 AM, Something Something <
> > > > > mailinglists19@gmail.com> wrote:
> > > > >
> > > > > > Seems like I am not explaining my problem correctly.
> > > > > >
> > > > > > My Key Pair file is on my machine at work which is behind a corp
> > > > > firewall.
> > > > > >  As such I can't 'scp' from the ec2 instance to my local machine
> at
> > > > work
> > > > > to
> > > > > > 'get' the file.  So I need a way to 'send' a file from my machine
> > to
> > > > the
> > > > > > ec2
> > > > > > instance.  I tried using 'scp' (from my machine at work) to the
> ec2
> > > > > machine
> > > > > > but it says "Permission denied".  Does this make sense?
> > > > > >
> > > > > > May be I need to use the command line tools for EC2.  I am
> looking
> > > into
> > > > > > those right now, but if there's a better/easier way, please let
> me
> > > > know.
> > > > > >
> > > > > > Thanks once again for your help.
> > > > > >
> > > > > >
> > > > > > On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <zjffdu@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > The ssh is installed on ec2 by default, otherwise you have no
> way
> > > to
> > > > > > login
> > > > > > > to ec2
> > > > > > >
> > > > > > >
> > > > > > > Jeff Zhang
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Sun, Nov 22, 2009 at 8:09 AM, Something Something <
> > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > >
> > > > > > > > As per Wikipedia (http://en.wikipedia.org/wiki/Secure_copy)
> >  "It
> > > > > does
> > > > > > so
> > > > > > > > by
> > > > > > > > connecting to the host using SSH and there executes an SCP
> > server
> > > > > > (scp)".
> > > > > > > >
> > > > > > > > So if SSH isn't working SCP won't work, either.  In any case
> I
> > > > tried
> > > > > to
> > > > > > > scp
> > > > > > > > but getting "Permission denied (public key)".
> > > > > > > >
> > > > > > > > Any other ideas?  Thanks.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <
> zjffdu@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > You should scp the key-pair to EC2 machine
> > > > > > > > >
> > > > > > > > > Jeff Zhang
> > > > > > > > >
> > > > > > > > > On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
> > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > > Trying to start a Hadoop cluster on EC2.  (Yes,
> Cloudera's
> > > > > > > distribution
> > > > > > > > > > works well, but trying to do this myself so I can learn
> > > what's
> > > > > > > > happening
> > > > > > > > > > behind the scene.)
> > > > > > > > > >
> > > > > > > > > > I have a Master & a Slave.  When I start HDFS on the
> > master,
> > > I
> > > > > get
> > > > > > a
> > > > > > > > > > message
> > > > > > > > > > saying "10.xxx.xxx.xxx (Permission denied)" - where
> 10.xxx
> > is
> > > > IP
> > > > > > > > address
> > > > > > > > > of
> > > > > > > > > > the slave.
> > > > > > > > > >
> > > > > > > > > > Basic problem (I think) is that I can't ssh from the
> master
> > > EC2
> > > > > > > > instance
> > > > > > > > > to
> > > > > > > > > > the Slave EC2 instance.  What's the best way to fix it?
>  I
> > > > think
> > > > > I
> > > > > > > need
> > > > > > > > > the
> > > > > > > > > > "Key Pair" file on my master.  I have a key pair on my
> > local
> > > > > > machine,
> > > > > > > > but
> > > > > > > > > > how do I transfer it to the EC2 machine?  (I know, I
> know,
> > I
> > > > > > agree..
> > > > > > > I
> > > > > > > > am
> > > > > > > > > > dumb :)  Should I FTP it?
> > > > > > > > > >
> > > > > > > > > > Please help.  Thanks.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

--0016e6d7dff529a0c50479252953--

From general-return-786-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 24 22:59:14 2009
Return-Path: <general-return-786-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49109 invoked from network); 24 Nov 2009 22:59:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Nov 2009 22:59:14 -0000
Received: (qmail 5094 invoked by uid 500); 24 Nov 2009 22:59:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5010 invoked by uid 500); 24 Nov 2009 22:59:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4996 invoked by uid 99); 24 Nov 2009 22:59:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Nov 2009 22:59:11 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of vpuranik@gmail.com designates 209.85.222.198 as permitted sender)
Received: from [209.85.222.198] (HELO mail-pz0-f198.google.com) (209.85.222.198)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Nov 2009 22:58:59 +0000
Received: by pzk36 with SMTP id 36so459096pzk.5
        for <general@hadoop.apache.org>; Tue, 24 Nov 2009 14:58:37 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=IC6rt/R4Ju/knc3aRbEhsaTzmbLkZVHigpnaS2mxzns=;
        b=qSptq+SNuvJVbWS9CS+bUew6BsQQL145p+sim1RVIBtn4eqswCtU437sIkTWPy1hKF
         UQns5HdTjQ11Ej46dTTw8rmgCaeNX6xqI+6DJLxPngxlVUhs/k+MKDCO2TDSfAgsVIVb
         Y273h6xZn7kj+UsrHAIENqcyvdHDwz7nyVjho=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=tg1/WKPVWYsY3JqeY9fzzZs2BMLLNu5oENLaKaRUr6w8/5vTx9l2B2rGDT2Oy15mjQ
         FJgeGJZafiGkeu9AZUwLfTXoyG/EvMsihYnUe+GAXqOpWGsKgLt9PYFHZ5ZhQTVBtOvZ
         ojOe9tjYF5EWn1SuMHOxOKGuo22b67e30PT6E=
MIME-Version: 1.0
Received: by 10.114.119.6 with SMTP id r6mr13669894wac.45.1259103517598; Tue, 
	24 Nov 2009 14:58:37 -0800 (PST)
In-Reply-To: <1eabbac30911241406v1a684661va156ab6a856819c3@mail.gmail.com>
References: <1eabbac30911211657i7812c67bs3a1565b0537b418f@mail.gmail.com>
	 <1eabbac30911220809l2b923866u4c79e4a37e2d9ad4@mail.gmail.com>
	 <8211a1320911220819j8d33c10te498c18362e825e0@mail.gmail.com>
	 <1eabbac30911220836t76895c04x6f7973d759c60c18@mail.gmail.com>
	 <2eef7bd70911230810l690624a1vd4fe8c208279293c@mail.gmail.com>
	 <1eabbac30911241054w6ef4852n41ddf93198c27acd@mail.gmail.com>
	 <2eef7bd70911241116n6f81550fpfeec6b88c382e0b4@mail.gmail.com>
	 <1eabbac30911241338g1067b99dra633f21f2d0cfd10@mail.gmail.com>
	 <2eef7bd70911241344p4225190fkaaafc160a5c9bb80@mail.gmail.com>
	 <1eabbac30911241406v1a684661va156ab6a856819c3@mail.gmail.com>
Date: Tue, 24 Nov 2009 14:58:37 -0800
Message-ID: <2eef7bd70911241458x6e2de4dfx60ef8e68b636f78e@mail.gmail.com>
Subject: Re: Starting Hadoop cluster on EC2
From: Vaibhav Puranik <vpuranik@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016369208c4e6e5f6047925e139
X-Virus-Checked: Checked by ClamAV on apache.org

--0016369208c4e6e5f6047925e139
Content-Type: text/plain; charset=ISO-8859-1

Install a program called Jps.

It's a ps for java processes. execute jps, you should see the processes (if
they are running).

Also make sure you check your logs.

Regards,
Vaibhav

On Tue, Nov 24, 2009 at 2:06 PM, Something Something <
mailinglists19@gmail.com> wrote:

> Awesome.  Thanks, Vaibhav.  Made progress.  No error messages from
> ./all-start.sh
>
> I will test to see if the cluster is deployed properly.
>
> Question:  On the slave, when I do:
>
> ps -eaf | grep 'hadoop'
>
> I don't see any processes.  Shouldn't I see 2 processes - one for Datanode
> and the other for TaskTracker?
>
> Please let me know.  Thanks again.
>
>
> On Tue, Nov 24, 2009 at 1:44 PM, Vaibhav Puranik <vpuranik@gmail.com>
> wrote:
>
> > Yes, I pretty much meant that.
> >
> > Add your master's public key to authorized_keys on slaves too.
> >
> > Regards,
> > Vaibhav
> >
> >
> >
> > On Tue, Nov 24, 2009 at 1:38 PM, Something Something <
> > mailinglists19@gmail.com> wrote:
> >
> > > Sorry.. but not sure what you mean by... "In order to achieve that you
> > need
> > > to add the keypair  you are using to your
> > > auhtorized_keys file in the home dir/.ssh folder."
> > >
> > >
> > > I tried this....
> > >
> > > cat KeyPair.pem >> /root/.ssh/authorized_keys
> > >
> > > Is that what you meant by "add the keypair to authorized_keys"?
> > >
> > > Still can't ssh from Slave to Master.  It says.....  Permission denied
> > > (publickey).
> > >
> > >
> > > Sorry, I am a newbie when it comes to such environment issues.  Thanks
> > for
> > > your patience.  It's greatly appreciated.
> > >
> > >
> > > On Tue, Nov 24, 2009 at 11:16 AM, Vaibhav Puranik <vpuranik@gmail.com
> > > >wrote:
> > >
> > > > You need to be able to SSH without specifying the keypair from a
> slave
> > to
> > > > master (I think reverse it true too, but I am not sure).
> > > >
> > > > In order to achieve that you need to add the keypair  you are using
> to
> > > your
> > > > auhtorized_keys file in the home dir/.ssh folder.
> > > >
> > > > Once you setup passphraseless ssh, it should start working.
> Furthrmore,
> > > > after you set this up, try to ssh manully (without key) and if it
> asks
> > > you
> > > > the following message - say yes:
> > > >
> > > > The authenticity of host
> > > > 'ec2-147-127-186-243.compute-1.amazonaws.com(147.127.186.243)' can't
> > > > be established.
> > > > RSA key fingerprint is
> 4b:63:e2:23:16:6f:b6:99:de:34:f6:9b:f5:55:73:8b.
> > > > Are you sure you want to continue connecting (yes/no)? yes
> > > > Warning: Permanently added
> > > > 'ec2-147-127-186-243.compute-1.amazonaws.com,147.127.186.243'
> > > > (RSA) to the list of known hosts.
> > > >
> > > > In other words, you have to make sure that the machine you are trying
> > to
> > > > ssh
> > > > is in your known_hosts file.
> > > >
> > > > Regards,
> > > > Vaibhav
> > > >
> > > > On Tue, Nov 24, 2009 at 10:54 AM, Something Something <
> > > > mailinglists19@gmail.com> wrote:
> > > >
> > > > > Thanks, Vaibhav.  That was helpful.  I am able to scp from my local
> > > > machine
> > > > > after I opened port 22 under my security group.
> > > > >
> > > > > But my cluster is still not starting as expected.  I mean... on my
> > > Master
> > > > > when I run:
> > > > >
> > > > > ./start-all.sh
> > > > >
> > > > > I get messages saying...
> > > > >
> > > > > 10.252.xxx.xxx: Permission denied (publickey).
> > > > >
> > > > > where 10.252.xxx.xxx is the ip address of my Slave.
> > > > >
> > > > >
> > > > > I can ssh from my Slave to master if I do this:
> > > > >
> > > > > ssh -i <keypairname>.pem <Master's ip address>
> > > > >
> > > > >
> > > > > Any suggestions?  Thanks for your help.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Nov 23, 2009 at 8:10 AM, Vaibhav Puranik <
> vpuranik@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > In order to use scp from your machine to an ec2 instance you need
> > to
> > > > use
> > > > > > scp
> > > > > > -i and give it the path of your keypair.
> > > > > >
> > > > > > scp -i /path/my/ec2/keypair  <souce> <destination>
> > > > > >
> > > > > > Secondly in order to configure an ec2 cluster for Hadoop, you
> need
> > to
> > > > > make
> > > > > > a
> > > > > > security group (let's call it hadoop) and give hadoop access to
> > > hadoop.
> > > > > >
> > > > > > You can read more about security groups here -
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://somic.org/2009/09/21/security-groups-most-underappreciated-feature-of-amazon-ec2/
> > > > > >
> > > > > > You also need to make sure that you can ssh from all the slaves
> to
> > > > master
> > > > > > without specifying password.
> > > > > >
> > > > > > Regards,
> > > > > > Vaibhav Puranik
> > > > > > GumGum
> > > > > >
> > > > > > On Sun, Nov 22, 2009 at 8:36 AM, Something Something <
> > > > > > mailinglists19@gmail.com> wrote:
> > > > > >
> > > > > > > Seems like I am not explaining my problem correctly.
> > > > > > >
> > > > > > > My Key Pair file is on my machine at work which is behind a
> corp
> > > > > > firewall.
> > > > > > >  As such I can't 'scp' from the ec2 instance to my local
> machine
> > at
> > > > > work
> > > > > > to
> > > > > > > 'get' the file.  So I need a way to 'send' a file from my
> machine
> > > to
> > > > > the
> > > > > > > ec2
> > > > > > > instance.  I tried using 'scp' (from my machine at work) to the
> > ec2
> > > > > > machine
> > > > > > > but it says "Permission denied".  Does this make sense?
> > > > > > >
> > > > > > > May be I need to use the command line tools for EC2.  I am
> > looking
> > > > into
> > > > > > > those right now, but if there's a better/easier way, please let
> > me
> > > > > know.
> > > > > > >
> > > > > > > Thanks once again for your help.
> > > > > > >
> > > > > > >
> > > > > > > On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <zjffdu@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > The ssh is installed on ec2 by default, otherwise you have no
> > way
> > > > to
> > > > > > > login
> > > > > > > > to ec2
> > > > > > > >
> > > > > > > >
> > > > > > > > Jeff Zhang
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sun, Nov 22, 2009 at 8:09 AM, Something Something <
> > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > As per Wikipedia (http://en.wikipedia.org/wiki/Secure_copy
> )
> > >  "It
> > > > > > does
> > > > > > > so
> > > > > > > > > by
> > > > > > > > > connecting to the host using SSH and there executes an SCP
> > > server
> > > > > > > (scp)".
> > > > > > > > >
> > > > > > > > > So if SSH isn't working SCP won't work, either.  In any
> case
> > I
> > > > > tried
> > > > > > to
> > > > > > > > scp
> > > > > > > > > but getting "Permission denied (public key)".
> > > > > > > > >
> > > > > > > > > Any other ideas?  Thanks.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <
> > zjffdu@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > You should scp the key-pair to EC2 machine
> > > > > > > > > >
> > > > > > > > > > Jeff Zhang
> > > > > > > > > >
> > > > > > > > > > On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
> > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > Trying to start a Hadoop cluster on EC2.  (Yes,
> > Cloudera's
> > > > > > > > distribution
> > > > > > > > > > > works well, but trying to do this myself so I can learn
> > > > what's
> > > > > > > > > happening
> > > > > > > > > > > behind the scene.)
> > > > > > > > > > >
> > > > > > > > > > > I have a Master & a Slave.  When I start HDFS on the
> > > master,
> > > > I
> > > > > > get
> > > > > > > a
> > > > > > > > > > > message
> > > > > > > > > > > saying "10.xxx.xxx.xxx (Permission denied)" - where
> > 10.xxx
> > > is
> > > > > IP
> > > > > > > > > address
> > > > > > > > > > of
> > > > > > > > > > > the slave.
> > > > > > > > > > >
> > > > > > > > > > > Basic problem (I think) is that I can't ssh from the
> > master
> > > > EC2
> > > > > > > > > instance
> > > > > > > > > > to
> > > > > > > > > > > the Slave EC2 instance.  What's the best way to fix it?
> >  I
> > > > > think
> > > > > > I
> > > > > > > > need
> > > > > > > > > > the
> > > > > > > > > > > "Key Pair" file on my master.  I have a key pair on my
> > > local
> > > > > > > machine,
> > > > > > > > > but
> > > > > > > > > > > how do I transfer it to the EC2 machine?  (I know, I
> > know,
> > > I
> > > > > > > agree..
> > > > > > > > I
> > > > > > > > > am
> > > > > > > > > > > dumb :)  Should I FTP it?
> > > > > > > > > > >
> > > > > > > > > > > Please help.  Thanks.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

--0016369208c4e6e5f6047925e139--

From general-return-787-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 25 00:11:50 2009
Return-Path: <general-return-787-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69351 invoked from network); 25 Nov 2009 00:11:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Nov 2009 00:11:50 -0000
Received: (qmail 85797 invoked by uid 500); 25 Nov 2009 00:11:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85769 invoked by uid 500); 25 Nov 2009 00:11:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85753 invoked by uid 99); 25 Nov 2009 00:11:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Nov 2009 00:11:48 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Nov 2009 00:11:45 +0000
Received: from wlanvpn-mc2e-246-217.corp.yahoo.com (wlanvpn-mc2e-246-217.corp.yahoo.com [172.21.148.217])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id nAP0ABiS056742;
	Tue, 24 Nov 2009 16:10:12 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=from:to:in-reply-to:subject:x-priority:references:
	message-id:content-type:content-transfer-encoding:mime-version:date:cc:x-mailer;
	b=AHyDs0/TxbYRG1U0DrwHJo73NmZgiqxAdRPwAt8Xw2TIrv0Y0AZindm/kdBYHIAc
From: Arun C Murthy <acm@yahoo-inc.com>
To: mapreduce-user@hadoop.apache.org
In-Reply-To: <faeb26a0b719c863ee5e44bffb89ff79.squirrel@mail.tce.edu>
Subject: Re: Doubts Regarding Capacity Scheduler's Function
X-Priority: 3 (Normal)
References: <6efd398094dc482ed37a662080836354.squirrel@mail.tce.edu> <9FD0B40E-F7BA-4CFC-9BE9-3D52758E4EC8@yahoo-inc.com> <faeb26a0b719c863ee5e44bffb89ff79.squirrel@mail.tce.edu>
Message-Id: <BA7F0D5A-B017-4CA0-B9FB-338262AF4629@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Date: Tue, 24 Nov 2009 16:10:11 -0800
Cc: general@hadoop.apache.org
X-Mailer: Apple Mail (2.936)

(moving to mapreduce-user@hadoop.apache.org)

> I have set that property to a user defined queue Queue1 and now my
> submitted jobs are scheduled to Queue1,but I want to know the
> configuration example to schedule job1 to default ,then job2 to
> Queue2,etc.,

Use one of the several ways to pass the queue name as a config variable:

JobConf j = new JobConf();

j.setQueueName("myqueue");

or cmd line:
hadoop jar myjob.jar -Dmapred.job.queue.name=myqueue <other_args>

Arun

From general-return-788-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 25 15:13:52 2009
Return-Path: <general-return-788-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 10738 invoked from network); 25 Nov 2009 15:13:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Nov 2009 15:13:52 -0000
Received: (qmail 61832 invoked by uid 500); 25 Nov 2009 15:13:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61224 invoked by uid 500); 25 Nov 2009 15:13:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61124 invoked by uid 99); 25 Nov 2009 15:13:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Nov 2009 15:13:49 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of abhilaash_cse@tce.edu designates 210.212.252.72 as permitted sender)
Received: from [210.212.252.72] (HELO tce.edu) (210.212.252.72)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Nov 2009 15:13:41 +0000
Received: from [127.0.0.1] (helo=mail.tce.edu)
	by tce.edu with esmtp (Exim 4.63)
	(envelope-from <abhilaash_cse@tce.edu>)
	id 1NDJYA-00074a-9L
	for general@hadoop.apache.org; Wed, 25 Nov 2009 20:42:58 +0530
Received: from 59.92.122.90
        (SquirrelMail authenticated user abhilaash_cse)
        by mail.tce.edu with HTTP;
        Wed, 25 Nov 2009 20:42:58 +0530
Message-ID: <a9c7053ca3e8b70bbc1e916613bbc43c.squirrel@mail.tce.edu>
In-Reply-To: <BA7F0D5A-B017-4CA0-B9FB-338262AF4629@yahoo-inc.com>
References: <6efd398094dc482ed37a662080836354.squirrel@mail.tce.edu>
    <9FD0B40E-F7BA-4CFC-9BE9-3D52758E4EC8@yahoo-inc.com>
    <faeb26a0b719c863ee5e44bffb89ff79.squirrel@mail.tce.edu>
    <BA7F0D5A-B017-4CA0-B9FB-338262AF4629@yahoo-inc.com>
Date: Wed, 25 Nov 2009 20:42:58 +0530
Subject: Re: Doubts Regarding Capacity Scheduler's Function
From: "Abhilaash" <abhilaash_cse@tce.edu>
To: general@hadoop.apache.org
User-Agent: SquirrelMail/1.4.19
MIME-Version: 1.0
Content-Type: text/plain;charset=iso-8859-1
Content-Transfer-Encoding: 8bit
X-Priority: 3 (Normal)
Importance: Normal
X-XheaderVersion: 1.1
X-UserAgent: 
X-Spam-Score: -1.4 (-)
X-Virus-Checked: Checked by ClamAV on apache.org
X-Old-Spam-Status: No



Hi,

> JobConf j = new JobConf();
>
> j.setQueueName("myqueue");
>
> or cmd line:
> hadoop jar myjob.jar -Dmapred.job.queue.name=myqueue <other_args>
>
> Arun
>

Thanks,It works now.


Regards,
Abhilaash.




-- 
Nothing is impossible because impossible says i'm possible...


-----------------------------------------
This email was sent using TCEMail Service.
Thiagarajar College of Engineering
Madurai-625 015, India


From general-return-789-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 25 16:38:48 2009
Return-Path: <general-return-789-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39454 invoked from network); 25 Nov 2009 16:38:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Nov 2009 16:38:47 -0000
Received: (qmail 40058 invoked by uid 500); 25 Nov 2009 16:38:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39962 invoked by uid 500); 25 Nov 2009 16:38:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39952 invoked by uid 99); 25 Nov 2009 16:38:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Nov 2009 16:38:45 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mailinglists19@gmail.com designates 72.14.220.153 as permitted sender)
Received: from [72.14.220.153] (HELO fg-out-1718.google.com) (72.14.220.153)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Nov 2009 16:38:33 +0000
Received: by fg-out-1718.google.com with SMTP id l26so66952fgb.11
        for <general@hadoop.apache.org>; Wed, 25 Nov 2009 08:38:13 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=OUGYz/Ly4ELjr2x6dREUUWPkl9q1WCCmV2b1L1qbctg=;
        b=tJYWLRpxIA1ps8Rma1tDkrJi/0+DrchvLnhS+I1IPgrQpOyBjkRKbUGP2n6SCFH4vj
         OD/9g8gpsDH7uknHvCdBuCj2frQnWMLzPbrCTHDR1wEsdvARRYkQs9AinpDiGAbcsOOW
         oeBHW+3Yn429fRG9N/wDtjWL3NyWITvlCX/KI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=J6PvSydfv7ri2GxLxIp92lx6liRRF9D31ZBnG3vKenpeL/LeK+eZXz5GJfb/uemIIh
         o55sgpzv9UIANzXyIjuBB5lgmD6lr6BuqoLVltgDABqDFB/APUBiuTYUX5Rd8upViq25
         a1Ww4FJXpqtehmlA7Z+stJPmxWU4x/ReatqII=
MIME-Version: 1.0
Received: by 10.216.86.129 with SMTP id w1mr2538383wee.145.1259167090999; Wed, 
	25 Nov 2009 08:38:10 -0800 (PST)
In-Reply-To: <2eef7bd70911241458x6e2de4dfx60ef8e68b636f78e@mail.gmail.com>
References: <1eabbac30911211657i7812c67bs3a1565b0537b418f@mail.gmail.com>
	 <8211a1320911220819j8d33c10te498c18362e825e0@mail.gmail.com>
	 <1eabbac30911220836t76895c04x6f7973d759c60c18@mail.gmail.com>
	 <2eef7bd70911230810l690624a1vd4fe8c208279293c@mail.gmail.com>
	 <1eabbac30911241054w6ef4852n41ddf93198c27acd@mail.gmail.com>
	 <2eef7bd70911241116n6f81550fpfeec6b88c382e0b4@mail.gmail.com>
	 <1eabbac30911241338g1067b99dra633f21f2d0cfd10@mail.gmail.com>
	 <2eef7bd70911241344p4225190fkaaafc160a5c9bb80@mail.gmail.com>
	 <1eabbac30911241406v1a684661va156ab6a856819c3@mail.gmail.com>
	 <2eef7bd70911241458x6e2de4dfx60ef8e68b636f78e@mail.gmail.com>
Date: Wed, 25 Nov 2009 08:38:10 -0800
Message-ID: <1eabbac30911250838w3d4b089ascb2e0b21245a60ae@mail.gmail.com>
Subject: Re: Starting Hadoop cluster on EC2
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d785452c0369047934af77
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d785452c0369047934af77
Content-Type: text/plain; charset=ISO-8859-1

Me again :(

I decided to start fresh so that I can document the steps, but now I can't
even go further than I did before :(

Please let me know what step I am missing:

1)  Launched 2 instances on AWS console - one for Master, one for Slave.

2)  In one command window, connected to instance 1:
ssh -i MyKeyPair.pem <instance 1>
(Got 'authenticity' warning.. typed 'yes')

This is my "Master" window.

3)  Opened another command window & connected to instance 2:
ssh -i MyKeyPair.pem <instance 2>
(Got 'authenticity' warning.. typed 'yes')

This is my "Slave" window.

4) Opened another command window and 'scp' the key pair (from my local
machine) to instance 1 & instance 2.
scp -i MyKeyPair.pem MyKeyPair.pem <master instance>:/tmp
scp -i MyKeyPair.pem MyKeyPair.pem <slave instance>:/tmp

5)  On BOTH (Master & Slave) added Key Pair to 'authorized_key'

cat /tmp/MyKeyPair.pem >> ~/.ssh/authorized_keys

6)  To setup passphraseless ssh ran these on BOTH - Master & Slave:
ssh localhost  (Got 'authenticiy' message.  Typed 'yes'.  Got Permission
denied (public key)
ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

7)  In order to ssh from Master to Slave (without entering KeyPair), copied
the key (that starts with 'sh-dss') from Slave's 'authorized_key' file to
Master's 'authoried_key' file.

Now, every time I try (from Master):

ssh <slave's ip address>

I get - "Permission denied (publickey).

But, when I type:

ssh -i /tmp/MyKeyPair.pem <slave's ip address>

It works.

I thought step #5 above should have resolved this.  What am I doing wrong?
I believe, in order for me to start the cluster, I should be able to 'ssh'
to slave without using key pair, correct?


Thanks for your help.


On Tue, Nov 24, 2009 at 2:58 PM, Vaibhav Puranik <vpuranik@gmail.com> wrote:

> Install a program called Jps.
>
> It's a ps for java processes. execute jps, you should see the processes (if
> they are running).
>
> Also make sure you check your logs.
>
> Regards,
> Vaibhav
>
> On Tue, Nov 24, 2009 at 2:06 PM, Something Something <
> mailinglists19@gmail.com> wrote:
>
> > Awesome.  Thanks, Vaibhav.  Made progress.  No error messages from
> > ./all-start.sh
> >
> > I will test to see if the cluster is deployed properly.
> >
> > Question:  On the slave, when I do:
> >
> > ps -eaf | grep 'hadoop'
> >
> > I don't see any processes.  Shouldn't I see 2 processes - one for
> Datanode
> > and the other for TaskTracker?
> >
> > Please let me know.  Thanks again.
> >
> >
> > On Tue, Nov 24, 2009 at 1:44 PM, Vaibhav Puranik <vpuranik@gmail.com>
> > wrote:
> >
> > > Yes, I pretty much meant that.
> > >
> > > Add your master's public key to authorized_keys on slaves too.
> > >
> > > Regards,
> > > Vaibhav
> > >
> > >
> > >
> > > On Tue, Nov 24, 2009 at 1:38 PM, Something Something <
> > > mailinglists19@gmail.com> wrote:
> > >
> > > > Sorry.. but not sure what you mean by... "In order to achieve that
> you
> > > need
> > > > to add the keypair  you are using to your
> > > > auhtorized_keys file in the home dir/.ssh folder."
> > > >
> > > >
> > > > I tried this....
> > > >
> > > > cat KeyPair.pem >> /root/.ssh/authorized_keys
> > > >
> > > > Is that what you meant by "add the keypair to authorized_keys"?
> > > >
> > > > Still can't ssh from Slave to Master.  It says.....  Permission
> denied
> > > > (publickey).
> > > >
> > > >
> > > > Sorry, I am a newbie when it comes to such environment issues.
>  Thanks
> > > for
> > > > your patience.  It's greatly appreciated.
> > > >
> > > >
> > > > On Tue, Nov 24, 2009 at 11:16 AM, Vaibhav Puranik <
> vpuranik@gmail.com
> > > > >wrote:
> > > >
> > > > > You need to be able to SSH without specifying the keypair from a
> > slave
> > > to
> > > > > master (I think reverse it true too, but I am not sure).
> > > > >
> > > > > In order to achieve that you need to add the keypair  you are using
> > to
> > > > your
> > > > > auhtorized_keys file in the home dir/.ssh folder.
> > > > >
> > > > > Once you setup passphraseless ssh, it should start working.
> > Furthrmore,
> > > > > after you set this up, try to ssh manully (without key) and if it
> > asks
> > > > you
> > > > > the following message - say yes:
> > > > >
> > > > > The authenticity of host
> > > > > 'ec2-147-127-186-243.compute-1.amazonaws.com(147.127.186.243)'
> can't
> > > > > be established.
> > > > > RSA key fingerprint is
> > 4b:63:e2:23:16:6f:b6:99:de:34:f6:9b:f5:55:73:8b.
> > > > > Are you sure you want to continue connecting (yes/no)? yes
> > > > > Warning: Permanently added
> > > > > 'ec2-147-127-186-243.compute-1.amazonaws.com,147.127.186.243'
> > > > > (RSA) to the list of known hosts.
> > > > >
> > > > > In other words, you have to make sure that the machine you are
> trying
> > > to
> > > > > ssh
> > > > > is in your known_hosts file.
> > > > >
> > > > > Regards,
> > > > > Vaibhav
> > > > >
> > > > > On Tue, Nov 24, 2009 at 10:54 AM, Something Something <
> > > > > mailinglists19@gmail.com> wrote:
> > > > >
> > > > > > Thanks, Vaibhav.  That was helpful.  I am able to scp from my
> local
> > > > > machine
> > > > > > after I opened port 22 under my security group.
> > > > > >
> > > > > > But my cluster is still not starting as expected.  I mean... on
> my
> > > > Master
> > > > > > when I run:
> > > > > >
> > > > > > ./start-all.sh
> > > > > >
> > > > > > I get messages saying...
> > > > > >
> > > > > > 10.252.xxx.xxx: Permission denied (publickey).
> > > > > >
> > > > > > where 10.252.xxx.xxx is the ip address of my Slave.
> > > > > >
> > > > > >
> > > > > > I can ssh from my Slave to master if I do this:
> > > > > >
> > > > > > ssh -i <keypairname>.pem <Master's ip address>
> > > > > >
> > > > > >
> > > > > > Any suggestions?  Thanks for your help.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Nov 23, 2009 at 8:10 AM, Vaibhav Puranik <
> > vpuranik@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > In order to use scp from your machine to an ec2 instance you
> need
> > > to
> > > > > use
> > > > > > > scp
> > > > > > > -i and give it the path of your keypair.
> > > > > > >
> > > > > > > scp -i /path/my/ec2/keypair  <souce> <destination>
> > > > > > >
> > > > > > > Secondly in order to configure an ec2 cluster for Hadoop, you
> > need
> > > to
> > > > > > make
> > > > > > > a
> > > > > > > security group (let's call it hadoop) and give hadoop access to
> > > > hadoop.
> > > > > > >
> > > > > > > You can read more about security groups here -
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://somic.org/2009/09/21/security-groups-most-underappreciated-feature-of-amazon-ec2/
> > > > > > >
> > > > > > > You also need to make sure that you can ssh from all the slaves
> > to
> > > > > master
> > > > > > > without specifying password.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Vaibhav Puranik
> > > > > > > GumGum
> > > > > > >
> > > > > > > On Sun, Nov 22, 2009 at 8:36 AM, Something Something <
> > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > >
> > > > > > > > Seems like I am not explaining my problem correctly.
> > > > > > > >
> > > > > > > > My Key Pair file is on my machine at work which is behind a
> > corp
> > > > > > > firewall.
> > > > > > > >  As such I can't 'scp' from the ec2 instance to my local
> > machine
> > > at
> > > > > > work
> > > > > > > to
> > > > > > > > 'get' the file.  So I need a way to 'send' a file from my
> > machine
> > > > to
> > > > > > the
> > > > > > > > ec2
> > > > > > > > instance.  I tried using 'scp' (from my machine at work) to
> the
> > > ec2
> > > > > > > machine
> > > > > > > > but it says "Permission denied".  Does this make sense?
> > > > > > > >
> > > > > > > > May be I need to use the command line tools for EC2.  I am
> > > looking
> > > > > into
> > > > > > > > those right now, but if there's a better/easier way, please
> let
> > > me
> > > > > > know.
> > > > > > > >
> > > > > > > > Thanks once again for your help.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <
> zjffdu@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > The ssh is installed on ec2 by default, otherwise you have
> no
> > > way
> > > > > to
> > > > > > > > login
> > > > > > > > > to ec2
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Jeff Zhang
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Sun, Nov 22, 2009 at 8:09 AM, Something Something <
> > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > > As per Wikipedia (
> http://en.wikipedia.org/wiki/Secure_copy
> > )
> > > >  "It
> > > > > > > does
> > > > > > > > so
> > > > > > > > > > by
> > > > > > > > > > connecting to the host using SSH and there executes an
> SCP
> > > > server
> > > > > > > > (scp)".
> > > > > > > > > >
> > > > > > > > > > So if SSH isn't working SCP won't work, either.  In any
> > case
> > > I
> > > > > > tried
> > > > > > > to
> > > > > > > > > scp
> > > > > > > > > > but getting "Permission denied (public key)".
> > > > > > > > > >
> > > > > > > > > > Any other ideas?  Thanks.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <
> > > zjffdu@gmail.com>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > You should scp the key-pair to EC2 machine
> > > > > > > > > > >
> > > > > > > > > > > Jeff Zhang
> > > > > > > > > > >
> > > > > > > > > > > On Sat, Nov 21, 2009 at 4:57 PM, Something Something <
> > > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Trying to start a Hadoop cluster on EC2.  (Yes,
> > > Cloudera's
> > > > > > > > > distribution
> > > > > > > > > > > > works well, but trying to do this myself so I can
> learn
> > > > > what's
> > > > > > > > > > happening
> > > > > > > > > > > > behind the scene.)
> > > > > > > > > > > >
> > > > > > > > > > > > I have a Master & a Slave.  When I start HDFS on the
> > > > master,
> > > > > I
> > > > > > > get
> > > > > > > > a
> > > > > > > > > > > > message
> > > > > > > > > > > > saying "10.xxx.xxx.xxx (Permission denied)" - where
> > > 10.xxx
> > > > is
> > > > > > IP
> > > > > > > > > > address
> > > > > > > > > > > of
> > > > > > > > > > > > the slave.
> > > > > > > > > > > >
> > > > > > > > > > > > Basic problem (I think) is that I can't ssh from the
> > > master
> > > > > EC2
> > > > > > > > > > instance
> > > > > > > > > > > to
> > > > > > > > > > > > the Slave EC2 instance.  What's the best way to fix
> it?
> > >  I
> > > > > > think
> > > > > > > I
> > > > > > > > > need
> > > > > > > > > > > the
> > > > > > > > > > > > "Key Pair" file on my master.  I have a key pair on
> my
> > > > local
> > > > > > > > machine,
> > > > > > > > > > but
> > > > > > > > > > > > how do I transfer it to the EC2 machine?  (I know, I
> > > know,
> > > > I
> > > > > > > > agree..
> > > > > > > > > I
> > > > > > > > > > am
> > > > > > > > > > > > dumb :)  Should I FTP it?
> > > > > > > > > > > >
> > > > > > > > > > > > Please help.  Thanks.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

--0016e6d785452c0369047934af77--

From general-return-790-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 25 17:22:12 2009
Return-Path: <general-return-790-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55709 invoked from network); 25 Nov 2009 17:22:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Nov 2009 17:22:11 -0000
Received: (qmail 34553 invoked by uid 500); 25 Nov 2009 17:22:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34455 invoked by uid 500); 25 Nov 2009 17:22:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34445 invoked by uid 99); 25 Nov 2009 17:22:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Nov 2009 17:22:09 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of vpuranik@gmail.com designates 209.85.216.204 as permitted sender)
Received: from [209.85.216.204] (HELO mail-px0-f204.google.com) (209.85.216.204)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Nov 2009 17:21:57 +0000
Received: by pxi42 with SMTP id 42so5061747pxi.5
        for <general@hadoop.apache.org>; Wed, 25 Nov 2009 09:21:36 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=ZlzbZ7jTtw9QaQMc7t5S0W1VKrLY0TSAaUtL6npYbGc=;
        b=ddFRyC6cowVVSKjQ2WHn5K/IEaWswioUVi2Vk3pKqweOC3aebqe5BmMe3Gge1m/QRm
         9E0tGr7GM9WTdDmp76NFWWvKAwfKZyv3OW0jWjOroI1oDd0DWwKBX1GUp2efIpOWI5H/
         xtGAHV3kb7EsRCWjvHvBH9alTCQ2wBWHW/uOw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=uF6M6kd3urxFMsw6fj102wDiWvF0nY/djuzCSu4UcuTi2TpwbZtshQhUsOazbW0cO1
         9kEXtvjqlWx6eT8/Vb0+JpjD428VkOw8qtqqHNpKrc5C5+bg1M15k16p7j6ujN60o8sC
         SbIAzLL2tfGWr2qUA9mQjkUFdZAn8SL38v6Y4=
MIME-Version: 1.0
Received: by 10.115.100.30 with SMTP id c30mr2741496wam.211.1259169693804; 
	Wed, 25 Nov 2009 09:21:33 -0800 (PST)
In-Reply-To: <1eabbac30911250838w3d4b089ascb2e0b21245a60ae@mail.gmail.com>
References: <1eabbac30911211657i7812c67bs3a1565b0537b418f@mail.gmail.com>
	 <1eabbac30911220836t76895c04x6f7973d759c60c18@mail.gmail.com>
	 <2eef7bd70911230810l690624a1vd4fe8c208279293c@mail.gmail.com>
	 <1eabbac30911241054w6ef4852n41ddf93198c27acd@mail.gmail.com>
	 <2eef7bd70911241116n6f81550fpfeec6b88c382e0b4@mail.gmail.com>
	 <1eabbac30911241338g1067b99dra633f21f2d0cfd10@mail.gmail.com>
	 <2eef7bd70911241344p4225190fkaaafc160a5c9bb80@mail.gmail.com>
	 <1eabbac30911241406v1a684661va156ab6a856819c3@mail.gmail.com>
	 <2eef7bd70911241458x6e2de4dfx60ef8e68b636f78e@mail.gmail.com>
	 <1eabbac30911250838w3d4b089ascb2e0b21245a60ae@mail.gmail.com>
Date: Wed, 25 Nov 2009 09:21:33 -0800
Message-ID: <2eef7bd70911250921p5d2558fq8766748c47dbfb3f@mail.gmail.com>
Subject: Re: Starting Hadoop cluster on EC2
From: Vaibhav Puranik <vpuranik@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64b90464fab830479354a5d
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e64b90464fab830479354a5d
Content-Type: text/plain; charset=ISO-8859-1

Actually I was wrong about adding keypair. It should be resolved by adding
master's public key to slave's authorized_keys. Amazon puts your keypair in
authroized_keys automatically. I missed that part.

Regards,
Vaibhav

On Wed, Nov 25, 2009 at 8:38 AM, Something Something <
mailinglists19@gmail.com> wrote:

> Me again :(
>
> I decided to start fresh so that I can document the steps, but now I can't
> even go further than I did before :(
>
> Please let me know what step I am missing:
>
> 1)  Launched 2 instances on AWS console - one for Master, one for Slave.
>
> 2)  In one command window, connected to instance 1:
> ssh -i MyKeyPair.pem <instance 1>
> (Got 'authenticity' warning.. typed 'yes')
>
> This is my "Master" window.
>
> 3)  Opened another command window & connected to instance 2:
> ssh -i MyKeyPair.pem <instance 2>
> (Got 'authenticity' warning.. typed 'yes')
>
> This is my "Slave" window.
>
> 4) Opened another command window and 'scp' the key pair (from my local
> machine) to instance 1 & instance 2.
> scp -i MyKeyPair.pem MyKeyPair.pem <master instance>:/tmp
> scp -i MyKeyPair.pem MyKeyPair.pem <slave instance>:/tmp
>
> 5)  On BOTH (Master & Slave) added Key Pair to 'authorized_key'
>
> cat /tmp/MyKeyPair.pem >> ~/.ssh/authorized_keys
>
> 6)  To setup passphraseless ssh ran these on BOTH - Master & Slave:
> ssh localhost  (Got 'authenticiy' message.  Typed 'yes'.  Got Permission
> denied (public key)
> ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
> cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
>
> 7)  In order to ssh from Master to Slave (without entering KeyPair), copied
> the key (that starts with 'sh-dss') from Slave's 'authorized_key' file to
> Master's 'authoried_key' file.
>
> Now, every time I try (from Master):
>
> ssh <slave's ip address>
>
> I get - "Permission denied (publickey).
>
> But, when I type:
>
> ssh -i /tmp/MyKeyPair.pem <slave's ip address>
>
> It works.
>
> I thought step #5 above should have resolved this.  What am I doing wrong?
> I believe, in order for me to start the cluster, I should be able to 'ssh'
> to slave without using key pair, correct?
>
>
> Thanks for your help.
>
>
> On Tue, Nov 24, 2009 at 2:58 PM, Vaibhav Puranik <vpuranik@gmail.com>
> wrote:
>
> > Install a program called Jps.
> >
> > It's a ps for java processes. execute jps, you should see the processes
> (if
> > they are running).
> >
> > Also make sure you check your logs.
> >
> > Regards,
> > Vaibhav
> >
> > On Tue, Nov 24, 2009 at 2:06 PM, Something Something <
> > mailinglists19@gmail.com> wrote:
> >
> > > Awesome.  Thanks, Vaibhav.  Made progress.  No error messages from
> > > ./all-start.sh
> > >
> > > I will test to see if the cluster is deployed properly.
> > >
> > > Question:  On the slave, when I do:
> > >
> > > ps -eaf | grep 'hadoop'
> > >
> > > I don't see any processes.  Shouldn't I see 2 processes - one for
> > Datanode
> > > and the other for TaskTracker?
> > >
> > > Please let me know.  Thanks again.
> > >
> > >
> > > On Tue, Nov 24, 2009 at 1:44 PM, Vaibhav Puranik <vpuranik@gmail.com>
> > > wrote:
> > >
> > > > Yes, I pretty much meant that.
> > > >
> > > > Add your master's public key to authorized_keys on slaves too.
> > > >
> > > > Regards,
> > > > Vaibhav
> > > >
> > > >
> > > >
> > > > On Tue, Nov 24, 2009 at 1:38 PM, Something Something <
> > > > mailinglists19@gmail.com> wrote:
> > > >
> > > > > Sorry.. but not sure what you mean by... "In order to achieve that
> > you
> > > > need
> > > > > to add the keypair  you are using to your
> > > > > auhtorized_keys file in the home dir/.ssh folder."
> > > > >
> > > > >
> > > > > I tried this....
> > > > >
> > > > > cat KeyPair.pem >> /root/.ssh/authorized_keys
> > > > >
> > > > > Is that what you meant by "add the keypair to authorized_keys"?
> > > > >
> > > > > Still can't ssh from Slave to Master.  It says.....  Permission
> > denied
> > > > > (publickey).
> > > > >
> > > > >
> > > > > Sorry, I am a newbie when it comes to such environment issues.
> >  Thanks
> > > > for
> > > > > your patience.  It's greatly appreciated.
> > > > >
> > > > >
> > > > > On Tue, Nov 24, 2009 at 11:16 AM, Vaibhav Puranik <
> > vpuranik@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > You need to be able to SSH without specifying the keypair from a
> > > slave
> > > > to
> > > > > > master (I think reverse it true too, but I am not sure).
> > > > > >
> > > > > > In order to achieve that you need to add the keypair  you are
> using
> > > to
> > > > > your
> > > > > > auhtorized_keys file in the home dir/.ssh folder.
> > > > > >
> > > > > > Once you setup passphraseless ssh, it should start working.
> > > Furthrmore,
> > > > > > after you set this up, try to ssh manully (without key) and if it
> > > asks
> > > > > you
> > > > > > the following message - say yes:
> > > > > >
> > > > > > The authenticity of host
> > > > > > 'ec2-147-127-186-243.compute-1.amazonaws.com(147.127.186.243)'
> > can't
> > > > > > be established.
> > > > > > RSA key fingerprint is
> > > 4b:63:e2:23:16:6f:b6:99:de:34:f6:9b:f5:55:73:8b.
> > > > > > Are you sure you want to continue connecting (yes/no)? yes
> > > > > > Warning: Permanently added
> > > > > > 'ec2-147-127-186-243.compute-1.amazonaws.com,147.127.186.243'
> > > > > > (RSA) to the list of known hosts.
> > > > > >
> > > > > > In other words, you have to make sure that the machine you are
> > trying
> > > > to
> > > > > > ssh
> > > > > > is in your known_hosts file.
> > > > > >
> > > > > > Regards,
> > > > > > Vaibhav
> > > > > >
> > > > > > On Tue, Nov 24, 2009 at 10:54 AM, Something Something <
> > > > > > mailinglists19@gmail.com> wrote:
> > > > > >
> > > > > > > Thanks, Vaibhav.  That was helpful.  I am able to scp from my
> > local
> > > > > > machine
> > > > > > > after I opened port 22 under my security group.
> > > > > > >
> > > > > > > But my cluster is still not starting as expected.  I mean... on
> > my
> > > > > Master
> > > > > > > when I run:
> > > > > > >
> > > > > > > ./start-all.sh
> > > > > > >
> > > > > > > I get messages saying...
> > > > > > >
> > > > > > > 10.252.xxx.xxx: Permission denied (publickey).
> > > > > > >
> > > > > > > where 10.252.xxx.xxx is the ip address of my Slave.
> > > > > > >
> > > > > > >
> > > > > > > I can ssh from my Slave to master if I do this:
> > > > > > >
> > > > > > > ssh -i <keypairname>.pem <Master's ip address>
> > > > > > >
> > > > > > >
> > > > > > > Any suggestions?  Thanks for your help.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Nov 23, 2009 at 8:10 AM, Vaibhav Puranik <
> > > vpuranik@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > In order to use scp from your machine to an ec2 instance you
> > need
> > > > to
> > > > > > use
> > > > > > > > scp
> > > > > > > > -i and give it the path of your keypair.
> > > > > > > >
> > > > > > > > scp -i /path/my/ec2/keypair  <souce> <destination>
> > > > > > > >
> > > > > > > > Secondly in order to configure an ec2 cluster for Hadoop, you
> > > need
> > > > to
> > > > > > > make
> > > > > > > > a
> > > > > > > > security group (let's call it hadoop) and give hadoop access
> to
> > > > > hadoop.
> > > > > > > >
> > > > > > > > You can read more about security groups here -
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://somic.org/2009/09/21/security-groups-most-underappreciated-feature-of-amazon-ec2/
> > > > > > > >
> > > > > > > > You also need to make sure that you can ssh from all the
> slaves
> > > to
> > > > > > master
> > > > > > > > without specifying password.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Vaibhav Puranik
> > > > > > > > GumGum
> > > > > > > >
> > > > > > > > On Sun, Nov 22, 2009 at 8:36 AM, Something Something <
> > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > Seems like I am not explaining my problem correctly.
> > > > > > > > >
> > > > > > > > > My Key Pair file is on my machine at work which is behind a
> > > corp
> > > > > > > > firewall.
> > > > > > > > >  As such I can't 'scp' from the ec2 instance to my local
> > > machine
> > > > at
> > > > > > > work
> > > > > > > > to
> > > > > > > > > 'get' the file.  So I need a way to 'send' a file from my
> > > machine
> > > > > to
> > > > > > > the
> > > > > > > > > ec2
> > > > > > > > > instance.  I tried using 'scp' (from my machine at work) to
> > the
> > > > ec2
> > > > > > > > machine
> > > > > > > > > but it says "Permission denied".  Does this make sense?
> > > > > > > > >
> > > > > > > > > May be I need to use the command line tools for EC2.  I am
> > > > looking
> > > > > > into
> > > > > > > > > those right now, but if there's a better/easier way, please
> > let
> > > > me
> > > > > > > know.
> > > > > > > > >
> > > > > > > > > Thanks once again for your help.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <
> > zjffdu@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > The ssh is installed on ec2 by default, otherwise you
> have
> > no
> > > > way
> > > > > > to
> > > > > > > > > login
> > > > > > > > > > to ec2
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Jeff Zhang
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Sun, Nov 22, 2009 at 8:09 AM, Something Something <
> > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > As per Wikipedia (
> > http://en.wikipedia.org/wiki/Secure_copy
> > > )
> > > > >  "It
> > > > > > > > does
> > > > > > > > > so
> > > > > > > > > > > by
> > > > > > > > > > > connecting to the host using SSH and there executes an
> > SCP
> > > > > server
> > > > > > > > > (scp)".
> > > > > > > > > > >
> > > > > > > > > > > So if SSH isn't working SCP won't work, either.  In any
> > > case
> > > > I
> > > > > > > tried
> > > > > > > > to
> > > > > > > > > > scp
> > > > > > > > > > > but getting "Permission denied (public key)".
> > > > > > > > > > >
> > > > > > > > > > > Any other ideas?  Thanks.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <
> > > > zjffdu@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > You should scp the key-pair to EC2 machine
> > > > > > > > > > > >
> > > > > > > > > > > > Jeff Zhang
> > > > > > > > > > > >
> > > > > > > > > > > > On Sat, Nov 21, 2009 at 4:57 PM, Something Something
> <
> > > > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Trying to start a Hadoop cluster on EC2.  (Yes,
> > > > Cloudera's
> > > > > > > > > > distribution
> > > > > > > > > > > > > works well, but trying to do this myself so I can
> > learn
> > > > > > what's
> > > > > > > > > > > happening
> > > > > > > > > > > > > behind the scene.)
> > > > > > > > > > > > >
> > > > > > > > > > > > > I have a Master & a Slave.  When I start HDFS on
> the
> > > > > master,
> > > > > > I
> > > > > > > > get
> > > > > > > > > a
> > > > > > > > > > > > > message
> > > > > > > > > > > > > saying "10.xxx.xxx.xxx (Permission denied)" - where
> > > > 10.xxx
> > > > > is
> > > > > > > IP
> > > > > > > > > > > address
> > > > > > > > > > > > of
> > > > > > > > > > > > > the slave.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Basic problem (I think) is that I can't ssh from
> the
> > > > master
> > > > > > EC2
> > > > > > > > > > > instance
> > > > > > > > > > > > to
> > > > > > > > > > > > > the Slave EC2 instance.  What's the best way to fix
> > it?
> > > >  I
> > > > > > > think
> > > > > > > > I
> > > > > > > > > > need
> > > > > > > > > > > > the
> > > > > > > > > > > > > "Key Pair" file on my master.  I have a key pair on
> > my
> > > > > local
> > > > > > > > > machine,
> > > > > > > > > > > but
> > > > > > > > > > > > > how do I transfer it to the EC2 machine?  (I know,
> I
> > > > know,
> > > > > I
> > > > > > > > > agree..
> > > > > > > > > > I
> > > > > > > > > > > am
> > > > > > > > > > > > > dumb :)  Should I FTP it?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Please help.  Thanks.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

--0016e64b90464fab830479354a5d--

From general-return-791-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 25 17:45:58 2009
Return-Path: <general-return-791-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68064 invoked from network); 25 Nov 2009 17:45:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Nov 2009 17:45:58 -0000
Received: (qmail 96792 invoked by uid 500); 25 Nov 2009 17:45:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96701 invoked by uid 500); 25 Nov 2009 17:45:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96691 invoked by uid 99); 25 Nov 2009 17:45:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Nov 2009 17:45:56 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mailinglists19@gmail.com designates 209.85.219.214 as permitted sender)
Received: from [209.85.219.214] (HELO mail-ew0-f214.google.com) (209.85.219.214)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Nov 2009 17:45:53 +0000
Received: by ewy6 with SMTP id 6so4660906ewy.29
        for <general@hadoop.apache.org>; Wed, 25 Nov 2009 09:45:31 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=hDGLG3B6Dhb59H3PWcRFVtyFupF3YeMa70DKs5QxDsQ=;
        b=LsJ3DZ6hF2wsFW8jvMSr+GN9/eCBiyYKhMZzwssH+Hvlp4XsASH74wVzwMFA28hYku
         H9apfOP0XKlVUVf9ttPIxTM1eGX4VNn1lEIB09OLpNt7S/L+MM6tb5GIusnguqdADwZ/
         SBR0yFB09PVy9b/HjfOklrsOlAbcRhBVYexl4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=NzhLakidQpjfZ32fKgGabhNFLbB3KOH5IoafjE+WGsHn2AuMpCJ4SE8doKrLVPnYUa
         kguzG0iUA43UWC+PwKWwKz1mmQj/5drsk1lMEF8dLKSxmKRzv9L9btDUiUNxBP+fPYlB
         8+bQn71XZebjs6kk3zFfHEE/tYUKf2NWaGHAY=
MIME-Version: 1.0
Received: by 10.216.89.85 with SMTP id b63mr2532129wef.175.1259171128560; Wed, 
	25 Nov 2009 09:45:28 -0800 (PST)
In-Reply-To: <2eef7bd70911250921p5d2558fq8766748c47dbfb3f@mail.gmail.com>
References: <1eabbac30911211657i7812c67bs3a1565b0537b418f@mail.gmail.com>
	 <2eef7bd70911230810l690624a1vd4fe8c208279293c@mail.gmail.com>
	 <1eabbac30911241054w6ef4852n41ddf93198c27acd@mail.gmail.com>
	 <2eef7bd70911241116n6f81550fpfeec6b88c382e0b4@mail.gmail.com>
	 <1eabbac30911241338g1067b99dra633f21f2d0cfd10@mail.gmail.com>
	 <2eef7bd70911241344p4225190fkaaafc160a5c9bb80@mail.gmail.com>
	 <1eabbac30911241406v1a684661va156ab6a856819c3@mail.gmail.com>
	 <2eef7bd70911241458x6e2de4dfx60ef8e68b636f78e@mail.gmail.com>
	 <1eabbac30911250838w3d4b089ascb2e0b21245a60ae@mail.gmail.com>
	 <2eef7bd70911250921p5d2558fq8766748c47dbfb3f@mail.gmail.com>
Date: Wed, 25 Nov 2009 09:45:28 -0800
Message-ID: <1eabbac30911250945i49684a5am6510cc6ffb7f86c8@mail.gmail.com>
Subject: Re: Starting Hadoop cluster on EC2
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d97076d44cd50479359f93

--0016e6d97076d44cd50479359f93
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Can you please explain what you mean by "add master's public key to slave's
authorized_keys"?  I have a feeling I am not doing this correctly.  What I
did is this:

1)  On Master, opened ~/.ssh/authorized_key and copied lines that look like
this...

ssh-dss
AAAAB3NzaC1kc3MAAACBAKJOife8p+6vjJ/jODSWaBgyawN7XWmasasasb0cVqhe1WOpnJQpXuJ=
eqsaBaZYHYxj8P74m1rOIdN9lSPAz6gEjHo4ft8UM7ZVExRC6HDM7JdfdferhwRgQSVI9ruTLls=
sO9i1D17mBnLotsgy92CyS7bsanM25nRqhmvqkX7tuKp66nlAAAAFQCnbwPn3v+ttorRZ4OijkA=
KoJaVFQAAAIAGVyZGwkm5DwYvige0VYhXZns8qSvW15dwSrS1a6k41q3C91AQZylvJYDnBjNGnW=
wUHfyWpyruzVCOskavEmsl6FcouU5Av1zLRT1sk4llUzZxXFtv6gzRBq2+aiCcoex3O+RRJbHa8=
9cMLy6jeZOCFahwVz6IyHcQ0b7gZjQgFQAAAIBA4vzSDy5oN27Ji3mF9ZklsSmlJJrrJxGJfOn6=
/BK5tyvBijLy6tEJHrH3dY8TEJUp2V9RINKwcsy5LdqyI5gJRWoM8WdNzmUoIOLAW0HIVl2MldK=
KqP/ctQwGI94EjykSUYVt/IAvUcjFuIrtiGrMDjsqfm7h1Gi2ZtDQ=3D=3D
root@domU-12-31-38-00-39-C8


to the end of ~/.ssh/authorized_key on Slave.

Something tells me there's a better way to do this.  Is there?


On Wed, Nov 25, 2009 at 9:21 AM, Vaibhav Puranik <vpuranik@gmail.com> wrote=
:

> Actually I was wrong about adding keypair. It should be resolved by addin=
g
> master's public key to slave's authorized_keys. Amazon puts your keypair =
in
> authroized_keys automatically. I missed that part.
>
> Regards,
> Vaibhav
>
> On Wed, Nov 25, 2009 at 8:38 AM, Something Something <
> mailinglists19@gmail.com> wrote:
>
> > Me again :(
> >
> > I decided to start fresh so that I can document the steps, but now I
> can't
> > even go further than I did before :(
> >
> > Please let me know what step I am missing:
> >
> > 1)  Launched 2 instances on AWS console - one for Master, one for Slave=
.
> >
> > 2)  In one command window, connected to instance 1:
> > ssh -i MyKeyPair.pem <instance 1>
> > (Got 'authenticity' warning.. typed 'yes')
> >
> > This is my "Master" window.
> >
> > 3)  Opened another command window & connected to instance 2:
> > ssh -i MyKeyPair.pem <instance 2>
> > (Got 'authenticity' warning.. typed 'yes')
> >
> > This is my "Slave" window.
> >
> > 4) Opened another command window and 'scp' the key pair (from my local
> > machine) to instance 1 & instance 2.
> > scp -i MyKeyPair.pem MyKeyPair.pem <master instance>:/tmp
> > scp -i MyKeyPair.pem MyKeyPair.pem <slave instance>:/tmp
> >
> > 5)  On BOTH (Master & Slave) added Key Pair to 'authorized_key'
> >
> > cat /tmp/MyKeyPair.pem >> ~/.ssh/authorized_keys
> >
> > 6)  To setup passphraseless ssh ran these on BOTH - Master & Slave:
> > ssh localhost  (Got 'authenticiy' message.  Typed 'yes'.  Got Permissio=
n
> > denied (public key)
> > ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
> > cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
> >
> > 7)  In order to ssh from Master to Slave (without entering KeyPair),
> copied
> > the key (that starts with 'sh-dss') from Slave's 'authorized_key' file =
to
> > Master's 'authoried_key' file.
> >
> > Now, every time I try (from Master):
> >
> > ssh <slave's ip address>
> >
> > I get - "Permission denied (publickey).
> >
> > But, when I type:
> >
> > ssh -i /tmp/MyKeyPair.pem <slave's ip address>
> >
> > It works.
> >
> > I thought step #5 above should have resolved this.  What am I doing
> wrong?
> > I believe, in order for me to start the cluster, I should be able to
> 'ssh'
> > to slave without using key pair, correct?
> >
> >
> > Thanks for your help.
> >
> >
> > On Tue, Nov 24, 2009 at 2:58 PM, Vaibhav Puranik <vpuranik@gmail.com>
> > wrote:
> >
> > > Install a program called Jps.
> > >
> > > It's a ps for java processes. execute jps, you should see the process=
es
> > (if
> > > they are running).
> > >
> > > Also make sure you check your logs.
> > >
> > > Regards,
> > > Vaibhav
> > >
> > > On Tue, Nov 24, 2009 at 2:06 PM, Something Something <
> > > mailinglists19@gmail.com> wrote:
> > >
> > > > Awesome.  Thanks, Vaibhav.  Made progress.  No error messages from
> > > > ./all-start.sh
> > > >
> > > > I will test to see if the cluster is deployed properly.
> > > >
> > > > Question:  On the slave, when I do:
> > > >
> > > > ps -eaf | grep 'hadoop'
> > > >
> > > > I don't see any processes.  Shouldn't I see 2 processes - one for
> > > Datanode
> > > > and the other for TaskTracker?
> > > >
> > > > Please let me know.  Thanks again.
> > > >
> > > >
> > > > On Tue, Nov 24, 2009 at 1:44 PM, Vaibhav Puranik <vpuranik@gmail.co=
m
> >
> > > > wrote:
> > > >
> > > > > Yes, I pretty much meant that.
> > > > >
> > > > > Add your master's public key to authorized_keys on slaves too.
> > > > >
> > > > > Regards,
> > > > > Vaibhav
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Nov 24, 2009 at 1:38 PM, Something Something <
> > > > > mailinglists19@gmail.com> wrote:
> > > > >
> > > > > > Sorry.. but not sure what you mean by... "In order to achieve
> that
> > > you
> > > > > need
> > > > > > to add the keypair  you are using to your
> > > > > > auhtorized_keys file in the home dir/.ssh folder."
> > > > > >
> > > > > >
> > > > > > I tried this....
> > > > > >
> > > > > > cat KeyPair.pem >> /root/.ssh/authorized_keys
> > > > > >
> > > > > > Is that what you meant by "add the keypair to authorized_keys"?
> > > > > >
> > > > > > Still can't ssh from Slave to Master.  It says.....  Permission
> > > denied
> > > > > > (publickey).
> > > > > >
> > > > > >
> > > > > > Sorry, I am a newbie when it comes to such environment issues.
> > >  Thanks
> > > > > for
> > > > > > your patience.  It's greatly appreciated.
> > > > > >
> > > > > >
> > > > > > On Tue, Nov 24, 2009 at 11:16 AM, Vaibhav Puranik <
> > > vpuranik@gmail.com
> > > > > > >wrote:
> > > > > >
> > > > > > > You need to be able to SSH without specifying the keypair fro=
m
> a
> > > > slave
> > > > > to
> > > > > > > master (I think reverse it true too, but I am not sure).
> > > > > > >
> > > > > > > In order to achieve that you need to add the keypair  you are
> > using
> > > > to
> > > > > > your
> > > > > > > auhtorized_keys file in the home dir/.ssh folder.
> > > > > > >
> > > > > > > Once you setup passphraseless ssh, it should start working.
> > > > Furthrmore,
> > > > > > > after you set this up, try to ssh manully (without key) and i=
f
> it
> > > > asks
> > > > > > you
> > > > > > > the following message - say yes:
> > > > > > >
> > > > > > > The authenticity of host
> > > > > > > 'ec2-147-127-186-243.compute-1.amazonaws.com(147.127.186.243)=
'
> > > can't
> > > > > > > be established.
> > > > > > > RSA key fingerprint is
> > > > 4b:63:e2:23:16:6f:b6:99:de:34:f6:9b:f5:55:73:8b.
> > > > > > > Are you sure you want to continue connecting (yes/no)? yes
> > > > > > > Warning: Permanently added
> > > > > > > 'ec2-147-127-186-243.compute-1.amazonaws.com,147.127.186.243'
> > > > > > > (RSA) to the list of known hosts.
> > > > > > >
> > > > > > > In other words, you have to make sure that the machine you ar=
e
> > > trying
> > > > > to
> > > > > > > ssh
> > > > > > > is in your known_hosts file.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Vaibhav
> > > > > > >
> > > > > > > On Tue, Nov 24, 2009 at 10:54 AM, Something Something <
> > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > >
> > > > > > > > Thanks, Vaibhav.  That was helpful.  I am able to scp from =
my
> > > local
> > > > > > > machine
> > > > > > > > after I opened port 22 under my security group.
> > > > > > > >
> > > > > > > > But my cluster is still not starting as expected.  I mean..=
.
> on
> > > my
> > > > > > Master
> > > > > > > > when I run:
> > > > > > > >
> > > > > > > > ./start-all.sh
> > > > > > > >
> > > > > > > > I get messages saying...
> > > > > > > >
> > > > > > > > 10.252.xxx.xxx: Permission denied (publickey).
> > > > > > > >
> > > > > > > > where 10.252.xxx.xxx is the ip address of my Slave.
> > > > > > > >
> > > > > > > >
> > > > > > > > I can ssh from my Slave to master if I do this:
> > > > > > > >
> > > > > > > > ssh -i <keypairname>.pem <Master's ip address>
> > > > > > > >
> > > > > > > >
> > > > > > > > Any suggestions?  Thanks for your help.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Nov 23, 2009 at 8:10 AM, Vaibhav Puranik <
> > > > vpuranik@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > In order to use scp from your machine to an ec2 instance
> you
> > > need
> > > > > to
> > > > > > > use
> > > > > > > > > scp
> > > > > > > > > -i and give it the path of your keypair.
> > > > > > > > >
> > > > > > > > > scp -i /path/my/ec2/keypair  <souce> <destination>
> > > > > > > > >
> > > > > > > > > Secondly in order to configure an ec2 cluster for Hadoop,
> you
> > > > need
> > > > > to
> > > > > > > > make
> > > > > > > > > a
> > > > > > > > > security group (let's call it hadoop) and give hadoop
> access
> > to
> > > > > > hadoop.
> > > > > > > > >
> > > > > > > > > You can read more about security groups here -
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://somic.org/2009/09/21/security-groups-most-underappreciated-feature=
-of-amazon-ec2/
> > > > > > > > >
> > > > > > > > > You also need to make sure that you can ssh from all the
> > slaves
> > > > to
> > > > > > > master
> > > > > > > > > without specifying password.
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Vaibhav Puranik
> > > > > > > > > GumGum
> > > > > > > > >
> > > > > > > > > On Sun, Nov 22, 2009 at 8:36 AM, Something Something <
> > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > > Seems like I am not explaining my problem correctly.
> > > > > > > > > >
> > > > > > > > > > My Key Pair file is on my machine at work which is behi=
nd
> a
> > > > corp
> > > > > > > > > firewall.
> > > > > > > > > >  As such I can't 'scp' from the ec2 instance to my loca=
l
> > > > machine
> > > > > at
> > > > > > > > work
> > > > > > > > > to
> > > > > > > > > > 'get' the file.  So I need a way to 'send' a file from =
my
> > > > machine
> > > > > > to
> > > > > > > > the
> > > > > > > > > > ec2
> > > > > > > > > > instance.  I tried using 'scp' (from my machine at work=
)
> to
> > > the
> > > > > ec2
> > > > > > > > > machine
> > > > > > > > > > but it says "Permission denied".  Does this make sense?
> > > > > > > > > >
> > > > > > > > > > May be I need to use the command line tools for EC2.  I
> am
> > > > > looking
> > > > > > > into
> > > > > > > > > > those right now, but if there's a better/easier way,
> please
> > > let
> > > > > me
> > > > > > > > know.
> > > > > > > > > >
> > > > > > > > > > Thanks once again for your help.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <
> > > zjffdu@gmail.com>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > The ssh is installed on ec2 by default, otherwise you
> > have
> > > no
> > > > > way
> > > > > > > to
> > > > > > > > > > login
> > > > > > > > > > > to ec2
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Jeff Zhang
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Nov 22, 2009 at 8:09 AM, Something Something =
<
> > > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > As per Wikipedia (
> > > http://en.wikipedia.org/wiki/Secure_copy
> > > > )
> > > > > >  "It
> > > > > > > > > does
> > > > > > > > > > so
> > > > > > > > > > > > by
> > > > > > > > > > > > connecting to the host using SSH and there executes
> an
> > > SCP
> > > > > > server
> > > > > > > > > > (scp)".
> > > > > > > > > > > >
> > > > > > > > > > > > So if SSH isn't working SCP won't work, either.  In
> any
> > > > case
> > > > > I
> > > > > > > > tried
> > > > > > > > > to
> > > > > > > > > > > scp
> > > > > > > > > > > > but getting "Permission denied (public key)".
> > > > > > > > > > > >
> > > > > > > > > > > > Any other ideas?  Thanks.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <
> > > > > zjffdu@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > You should scp the key-pair to EC2 machine
> > > > > > > > > > > > >
> > > > > > > > > > > > > Jeff Zhang
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sat, Nov 21, 2009 at 4:57 PM, Something
> Something
> > <
> > > > > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Trying to start a Hadoop cluster on EC2.  (Yes,
> > > > > Cloudera's
> > > > > > > > > > > distribution
> > > > > > > > > > > > > > works well, but trying to do this myself so I c=
an
> > > learn
> > > > > > > what's
> > > > > > > > > > > > happening
> > > > > > > > > > > > > > behind the scene.)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I have a Master & a Slave.  When I start HDFS o=
n
> > the
> > > > > > master,
> > > > > > > I
> > > > > > > > > get
> > > > > > > > > > a
> > > > > > > > > > > > > > message
> > > > > > > > > > > > > > saying "10.xxx.xxx.xxx (Permission denied)" -
> where
> > > > > 10.xxx
> > > > > > is
> > > > > > > > IP
> > > > > > > > > > > > address
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > the slave.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Basic problem (I think) is that I can't ssh fro=
m
> > the
> > > > > master
> > > > > > > EC2
> > > > > > > > > > > > instance
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > the Slave EC2 instance.  What's the best way to
> fix
> > > it?
> > > > >  I
> > > > > > > > think
> > > > > > > > > I
> > > > > > > > > > > need
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > "Key Pair" file on my master.  I have a key pai=
r
> on
> > > my
> > > > > > local
> > > > > > > > > > machine,
> > > > > > > > > > > > but
> > > > > > > > > > > > > > how do I transfer it to the EC2 machine?  (I
> know,
> > I
> > > > > know,
> > > > > > I
> > > > > > > > > > agree..
> > > > > > > > > > > I
> > > > > > > > > > > > am
> > > > > > > > > > > > > > dumb :)  Should I FTP it?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Please help.  Thanks.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

--0016e6d97076d44cd50479359f93--

From general-return-792-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 25 18:25:57 2009
Return-Path: <general-return-792-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82479 invoked from network); 25 Nov 2009 18:25:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Nov 2009 18:25:57 -0000
Received: (qmail 93930 invoked by uid 500); 25 Nov 2009 18:25:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93819 invoked by uid 500); 25 Nov 2009 18:25:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93809 invoked by uid 99); 25 Nov 2009 18:25:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Nov 2009 18:25:55 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of vpuranik@gmail.com designates 209.85.216.204 as permitted sender)
Received: from [209.85.216.204] (HELO mail-px0-f204.google.com) (209.85.216.204)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Nov 2009 18:25:42 +0000
Received: by pxi42 with SMTP id 42so5104006pxi.5
        for <general@hadoop.apache.org>; Wed, 25 Nov 2009 10:25:20 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=2BcNO2lEyDRDjrPRqJM8kLuHGhgjO3N8RSnLV+hbx+g=;
        b=qvrStNQmtyTFQl9BEukmJsxoE9V6z8fZSiWF7uaxJDWCTUkz03nI2Sl6lHpUjukRJV
         myGnXxc2yO6P0U2eWWgxadaxYOZqAekWDC1Q/5J3gLlNubORW0x9cMjcTOZcmgUR9eNe
         1sG3EurYewrGpg8DzORjzs0e7EoD36FL8KSGI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=dxI+31Pbb0jIB+D5XkvnpQ0wZZzEixPFnBxpvH6G7LQdjOEGrKFvwF1q9r4jQRQgRd
         riIf79mAJYma31wlsyUr2QbObGRT2SvRp5wZfe/ClWRD35jnsT3bqfaDpUzYJRlYTt5Q
         q0/JZscwLEU0J4aMfgrgpTrbdi6djAu28F05M=
MIME-Version: 1.0
Received: by 10.114.164.38 with SMTP id m38mr8275780wae.219.1259173518483; 
	Wed, 25 Nov 2009 10:25:18 -0800 (PST)
In-Reply-To: <1eabbac30911250945i49684a5am6510cc6ffb7f86c8@mail.gmail.com>
References: <1eabbac30911211657i7812c67bs3a1565b0537b418f@mail.gmail.com>
	 <1eabbac30911241054w6ef4852n41ddf93198c27acd@mail.gmail.com>
	 <2eef7bd70911241116n6f81550fpfeec6b88c382e0b4@mail.gmail.com>
	 <1eabbac30911241338g1067b99dra633f21f2d0cfd10@mail.gmail.com>
	 <2eef7bd70911241344p4225190fkaaafc160a5c9bb80@mail.gmail.com>
	 <1eabbac30911241406v1a684661va156ab6a856819c3@mail.gmail.com>
	 <2eef7bd70911241458x6e2de4dfx60ef8e68b636f78e@mail.gmail.com>
	 <1eabbac30911250838w3d4b089ascb2e0b21245a60ae@mail.gmail.com>
	 <2eef7bd70911250921p5d2558fq8766748c47dbfb3f@mail.gmail.com>
	 <1eabbac30911250945i49684a5am6510cc6ffb7f86c8@mail.gmail.com>
Date: Wed, 25 Nov 2009 10:25:18 -0800
Message-ID: <2eef7bd70911251025s73655a79ned5f6418caba81fb@mail.gmail.com>
Subject: Re: Starting Hadoop cluster on EC2
From: Vaibhav Puranik <vpuranik@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016367f934f47a03a0479362e2d
X-Virus-Checked: Checked by ClamAV on apache.org

--0016367f934f47a03a0479362e2d
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On master, in .ssh folder, there should be a file id_dsa.pub. This is
master's public key.

Contents of this file should be copied to slave's authorized_keys file.

Regards,
Vaibhav

On Wed, Nov 25, 2009 at 9:45 AM, Something Something <
mailinglists19@gmail.com> wrote:

> Can you please explain what you mean by "add master's public key to slave=
's
> authorized_keys"?  I have a feeling I am not doing this correctly.  What =
I
> did is this:
>
> 1)  On Master, opened ~/.ssh/authorized_key and copied lines that look li=
ke
> this...
>
> ssh-dss
>
> AAAAB3NzaC1kc3MAAACBAKJOife8p+6vjJ/jODSWaBgyawN7XWmasasasb0cVqhe1WOpnJQpX=
uJeqsaBaZYHYxj8P74m1rOIdN9lSPAz6gEjHo4ft8UM7ZVExRC6HDM7JdfdferhwRgQSVI9ruTL=
lssO9i1D17mBnLotsgy92CyS7bsanM25nRqhmvqkX7tuKp66nlAAAAFQCnbwPn3v+ttorRZ4Oij=
kAKoJaVFQAAAIAGVyZGwkm5DwYvige0VYhXZns8qSvW15dwSrS1a6k41q3C91AQZylvJYDnBjNG=
nWwUHfyWpyruzVCOskavEmsl6FcouU5Av1zLRT1sk4llUzZxXFtv6gzRBq2+aiCcoex3O+RRJbH=
a89cMLy6jeZOCFahwVz6IyHcQ0b7gZjQgFQAAAIBA4vzSDy5oN27Ji3mF9ZklsSmlJJrrJxGJfO=
n6/BK5tyvBijLy6tEJHrH3dY8TEJUp2V9RINKwcsy5LdqyI5gJRWoM8WdNzmUoIOLAW0HIVl2Ml=
dKKqP/ctQwGI94EjykSUYVt/IAvUcjFuIrtiGrMDjsqfm7h1Gi2ZtDQ=3D=3D
> root@domU-12-31-38-00-39-C8
>
>
> to the end of ~/.ssh/authorized_key on Slave.
>
> Something tells me there's a better way to do this.  Is there?
>
>
> On Wed, Nov 25, 2009 at 9:21 AM, Vaibhav Puranik <vpuranik@gmail.com>
> wrote:
>
> > Actually I was wrong about adding keypair. It should be resolved by
> adding
> > master's public key to slave's authorized_keys. Amazon puts your keypai=
r
> in
> > authroized_keys automatically. I missed that part.
> >
> > Regards,
> > Vaibhav
> >
> > On Wed, Nov 25, 2009 at 8:38 AM, Something Something <
> > mailinglists19@gmail.com> wrote:
> >
> > > Me again :(
> > >
> > > I decided to start fresh so that I can document the steps, but now I
> > can't
> > > even go further than I did before :(
> > >
> > > Please let me know what step I am missing:
> > >
> > > 1)  Launched 2 instances on AWS console - one for Master, one for
> Slave.
> > >
> > > 2)  In one command window, connected to instance 1:
> > > ssh -i MyKeyPair.pem <instance 1>
> > > (Got 'authenticity' warning.. typed 'yes')
> > >
> > > This is my "Master" window.
> > >
> > > 3)  Opened another command window & connected to instance 2:
> > > ssh -i MyKeyPair.pem <instance 2>
> > > (Got 'authenticity' warning.. typed 'yes')
> > >
> > > This is my "Slave" window.
> > >
> > > 4) Opened another command window and 'scp' the key pair (from my loca=
l
> > > machine) to instance 1 & instance 2.
> > > scp -i MyKeyPair.pem MyKeyPair.pem <master instance>:/tmp
> > > scp -i MyKeyPair.pem MyKeyPair.pem <slave instance>:/tmp
> > >
> > > 5)  On BOTH (Master & Slave) added Key Pair to 'authorized_key'
> > >
> > > cat /tmp/MyKeyPair.pem >> ~/.ssh/authorized_keys
> > >
> > > 6)  To setup passphraseless ssh ran these on BOTH - Master & Slave:
> > > ssh localhost  (Got 'authenticiy' message.  Typed 'yes'.  Got
> Permission
> > > denied (public key)
> > > ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
> > > cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
> > >
> > > 7)  In order to ssh from Master to Slave (without entering KeyPair),
> > copied
> > > the key (that starts with 'sh-dss') from Slave's 'authorized_key' fil=
e
> to
> > > Master's 'authoried_key' file.
> > >
> > > Now, every time I try (from Master):
> > >
> > > ssh <slave's ip address>
> > >
> > > I get - "Permission denied (publickey).
> > >
> > > But, when I type:
> > >
> > > ssh -i /tmp/MyKeyPair.pem <slave's ip address>
> > >
> > > It works.
> > >
> > > I thought step #5 above should have resolved this.  What am I doing
> > wrong?
> > > I believe, in order for me to start the cluster, I should be able to
> > 'ssh'
> > > to slave without using key pair, correct?
> > >
> > >
> > > Thanks for your help.
> > >
> > >
> > > On Tue, Nov 24, 2009 at 2:58 PM, Vaibhav Puranik <vpuranik@gmail.com>
> > > wrote:
> > >
> > > > Install a program called Jps.
> > > >
> > > > It's a ps for java processes. execute jps, you should see the
> processes
> > > (if
> > > > they are running).
> > > >
> > > > Also make sure you check your logs.
> > > >
> > > > Regards,
> > > > Vaibhav
> > > >
> > > > On Tue, Nov 24, 2009 at 2:06 PM, Something Something <
> > > > mailinglists19@gmail.com> wrote:
> > > >
> > > > > Awesome.  Thanks, Vaibhav.  Made progress.  No error messages fro=
m
> > > > > ./all-start.sh
> > > > >
> > > > > I will test to see if the cluster is deployed properly.
> > > > >
> > > > > Question:  On the slave, when I do:
> > > > >
> > > > > ps -eaf | grep 'hadoop'
> > > > >
> > > > > I don't see any processes.  Shouldn't I see 2 processes - one for
> > > > Datanode
> > > > > and the other for TaskTracker?
> > > > >
> > > > > Please let me know.  Thanks again.
> > > > >
> > > > >
> > > > > On Tue, Nov 24, 2009 at 1:44 PM, Vaibhav Puranik <
> vpuranik@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Yes, I pretty much meant that.
> > > > > >
> > > > > > Add your master's public key to authorized_keys on slaves too.
> > > > > >
> > > > > > Regards,
> > > > > > Vaibhav
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Nov 24, 2009 at 1:38 PM, Something Something <
> > > > > > mailinglists19@gmail.com> wrote:
> > > > > >
> > > > > > > Sorry.. but not sure what you mean by... "In order to achieve
> > that
> > > > you
> > > > > > need
> > > > > > > to add the keypair  you are using to your
> > > > > > > auhtorized_keys file in the home dir/.ssh folder."
> > > > > > >
> > > > > > >
> > > > > > > I tried this....
> > > > > > >
> > > > > > > cat KeyPair.pem >> /root/.ssh/authorized_keys
> > > > > > >
> > > > > > > Is that what you meant by "add the keypair to authorized_keys=
"?
> > > > > > >
> > > > > > > Still can't ssh from Slave to Master.  It says.....  Permissi=
on
> > > > denied
> > > > > > > (publickey).
> > > > > > >
> > > > > > >
> > > > > > > Sorry, I am a newbie when it comes to such environment issues=
.
> > > >  Thanks
> > > > > > for
> > > > > > > your patience.  It's greatly appreciated.
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Nov 24, 2009 at 11:16 AM, Vaibhav Puranik <
> > > > vpuranik@gmail.com
> > > > > > > >wrote:
> > > > > > >
> > > > > > > > You need to be able to SSH without specifying the keypair
> from
> > a
> > > > > slave
> > > > > > to
> > > > > > > > master (I think reverse it true too, but I am not sure).
> > > > > > > >
> > > > > > > > In order to achieve that you need to add the keypair  you a=
re
> > > using
> > > > > to
> > > > > > > your
> > > > > > > > auhtorized_keys file in the home dir/.ssh folder.
> > > > > > > >
> > > > > > > > Once you setup passphraseless ssh, it should start working.
> > > > > Furthrmore,
> > > > > > > > after you set this up, try to ssh manully (without key) and
> if
> > it
> > > > > asks
> > > > > > > you
> > > > > > > > the following message - say yes:
> > > > > > > >
> > > > > > > > The authenticity of host
> > > > > > > > 'ec2-147-127-186-243.compute-1.amazonaws.com
> (147.127.186.243)'
> > > > can't
> > > > > > > > be established.
> > > > > > > > RSA key fingerprint is
> > > > > 4b:63:e2:23:16:6f:b6:99:de:34:f6:9b:f5:55:73:8b.
> > > > > > > > Are you sure you want to continue connecting (yes/no)? yes
> > > > > > > > Warning: Permanently added
> > > > > > > > 'ec2-147-127-186-243.compute-1.amazonaws.com
> ,147.127.186.243'
> > > > > > > > (RSA) to the list of known hosts.
> > > > > > > >
> > > > > > > > In other words, you have to make sure that the machine you
> are
> > > > trying
> > > > > > to
> > > > > > > > ssh
> > > > > > > > is in your known_hosts file.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Vaibhav
> > > > > > > >
> > > > > > > > On Tue, Nov 24, 2009 at 10:54 AM, Something Something <
> > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > Thanks, Vaibhav.  That was helpful.  I am able to scp fro=
m
> my
> > > > local
> > > > > > > > machine
> > > > > > > > > after I opened port 22 under my security group.
> > > > > > > > >
> > > > > > > > > But my cluster is still not starting as expected.  I
> mean...
> > on
> > > > my
> > > > > > > Master
> > > > > > > > > when I run:
> > > > > > > > >
> > > > > > > > > ./start-all.sh
> > > > > > > > >
> > > > > > > > > I get messages saying...
> > > > > > > > >
> > > > > > > > > 10.252.xxx.xxx: Permission denied (publickey).
> > > > > > > > >
> > > > > > > > > where 10.252.xxx.xxx is the ip address of my Slave.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I can ssh from my Slave to master if I do this:
> > > > > > > > >
> > > > > > > > > ssh -i <keypairname>.pem <Master's ip address>
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Any suggestions?  Thanks for your help.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Nov 23, 2009 at 8:10 AM, Vaibhav Puranik <
> > > > > vpuranik@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > In order to use scp from your machine to an ec2 instanc=
e
> > you
> > > > need
> > > > > > to
> > > > > > > > use
> > > > > > > > > > scp
> > > > > > > > > > -i and give it the path of your keypair.
> > > > > > > > > >
> > > > > > > > > > scp -i /path/my/ec2/keypair  <souce> <destination>
> > > > > > > > > >
> > > > > > > > > > Secondly in order to configure an ec2 cluster for Hadoo=
p,
> > you
> > > > > need
> > > > > > to
> > > > > > > > > make
> > > > > > > > > > a
> > > > > > > > > > security group (let's call it hadoop) and give hadoop
> > access
> > > to
> > > > > > > hadoop.
> > > > > > > > > >
> > > > > > > > > > You can read more about security groups here -
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://somic.org/2009/09/21/security-groups-most-underappreciated-feature=
-of-amazon-ec2/
> > > > > > > > > >
> > > > > > > > > > You also need to make sure that you can ssh from all th=
e
> > > slaves
> > > > > to
> > > > > > > > master
> > > > > > > > > > without specifying password.
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > Vaibhav Puranik
> > > > > > > > > > GumGum
> > > > > > > > > >
> > > > > > > > > > On Sun, Nov 22, 2009 at 8:36 AM, Something Something <
> > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > Seems like I am not explaining my problem correctly.
> > > > > > > > > > >
> > > > > > > > > > > My Key Pair file is on my machine at work which is
> behind
> > a
> > > > > corp
> > > > > > > > > > firewall.
> > > > > > > > > > >  As such I can't 'scp' from the ec2 instance to my
> local
> > > > > machine
> > > > > > at
> > > > > > > > > work
> > > > > > > > > > to
> > > > > > > > > > > 'get' the file.  So I need a way to 'send' a file fro=
m
> my
> > > > > machine
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > ec2
> > > > > > > > > > > instance.  I tried using 'scp' (from my machine at
> work)
> > to
> > > > the
> > > > > > ec2
> > > > > > > > > > machine
> > > > > > > > > > > but it says "Permission denied".  Does this make sens=
e?
> > > > > > > > > > >
> > > > > > > > > > > May be I need to use the command line tools for EC2. =
 I
> > am
> > > > > > looking
> > > > > > > > into
> > > > > > > > > > > those right now, but if there's a better/easier way,
> > please
> > > > let
> > > > > > me
> > > > > > > > > know.
> > > > > > > > > > >
> > > > > > > > > > > Thanks once again for your help.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <
> > > > zjffdu@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > The ssh is installed on ec2 by default, otherwise y=
ou
> > > have
> > > > no
> > > > > > way
> > > > > > > > to
> > > > > > > > > > > login
> > > > > > > > > > > > to ec2
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Jeff Zhang
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Sun, Nov 22, 2009 at 8:09 AM, Something Somethin=
g
> <
> > > > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > As per Wikipedia (
> > > > http://en.wikipedia.org/wiki/Secure_copy
> > > > > )
> > > > > > >  "It
> > > > > > > > > > does
> > > > > > > > > > > so
> > > > > > > > > > > > > by
> > > > > > > > > > > > > connecting to the host using SSH and there execut=
es
> > an
> > > > SCP
> > > > > > > server
> > > > > > > > > > > (scp)".
> > > > > > > > > > > > >
> > > > > > > > > > > > > So if SSH isn't working SCP won't work, either.  =
In
> > any
> > > > > case
> > > > > > I
> > > > > > > > > tried
> > > > > > > > > > to
> > > > > > > > > > > > scp
> > > > > > > > > > > > > but getting "Permission denied (public key)".
> > > > > > > > > > > > >
> > > > > > > > > > > > > Any other ideas?  Thanks.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <
> > > > > > zjffdu@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > You should scp the key-pair to EC2 machine
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Jeff Zhang
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sat, Nov 21, 2009 at 4:57 PM, Something
> > Something
> > > <
> > > > > > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Trying to start a Hadoop cluster on EC2.  (Ye=
s,
> > > > > > Cloudera's
> > > > > > > > > > > > distribution
> > > > > > > > > > > > > > > works well, but trying to do this myself so I
> can
> > > > learn
> > > > > > > > what's
> > > > > > > > > > > > > happening
> > > > > > > > > > > > > > > behind the scene.)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I have a Master & a Slave.  When I start HDFS
> on
> > > the
> > > > > > > master,
> > > > > > > > I
> > > > > > > > > > get
> > > > > > > > > > > a
> > > > > > > > > > > > > > > message
> > > > > > > > > > > > > > > saying "10.xxx.xxx.xxx (Permission denied)" -
> > where
> > > > > > 10.xxx
> > > > > > > is
> > > > > > > > > IP
> > > > > > > > > > > > > address
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > the slave.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Basic problem (I think) is that I can't ssh
> from
> > > the
> > > > > > master
> > > > > > > > EC2
> > > > > > > > > > > > > instance
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > the Slave EC2 instance.  What's the best way =
to
> > fix
> > > > it?
> > > > > >  I
> > > > > > > > > think
> > > > > > > > > > I
> > > > > > > > > > > > need
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > "Key Pair" file on my master.  I have a key
> pair
> > on
> > > > my
> > > > > > > local
> > > > > > > > > > > machine,
> > > > > > > > > > > > > but
> > > > > > > > > > > > > > > how do I transfer it to the EC2 machine?  (I
> > know,
> > > I
> > > > > > know,
> > > > > > > I
> > > > > > > > > > > agree..
> > > > > > > > > > > > I
> > > > > > > > > > > > > am
> > > > > > > > > > > > > > > dumb :)  Should I FTP it?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Please help.  Thanks.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

--0016367f934f47a03a0479362e2d--

From general-return-793-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 27 17:42:08 2009
Return-Path: <general-return-793-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4095 invoked from network); 27 Nov 2009 17:42:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Nov 2009 17:42:08 -0000
Received: (qmail 77075 invoked by uid 500); 27 Nov 2009 17:42:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76987 invoked by uid 500); 27 Nov 2009 17:42:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76977 invoked by uid 99); 27 Nov 2009 17:42:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Nov 2009 17:42:05 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bruce.snyder@gmail.com designates 209.85.220.224 as permitted sender)
Received: from [209.85.220.224] (HELO mail-fx0-f224.google.com) (209.85.220.224)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Nov 2009 17:42:03 +0000
Received: by fxm24 with SMTP id 24so1510966fxm.11
        for <general@hadoop.apache.org>; Fri, 27 Nov 2009 09:41:42 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=2lRaDA3Mi/RGEXc9bbY1aMlzwYMYnfH0ZAAfLjhih8U=;
        b=Qu9oLtSoYatDdwdnDidhSo6bf6/8C40FsQu+OteFpN16islQ7ML9a3ZJjufgYBN3OK
         flxkk5OmkL56tMNHCrv4rK5fJZ90E9udJqrOylDMdygCG4p6c9X2ULtfIre4gKXrfcXQ
         rIfoaiUWy/tjNEi4xXlPBQOZggWkh0h19um+s=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=AkJjE9RR00hnGXN5VBKGlGm21+7stOzDFdy1GabU7uE+lEPGC5J7MQPFTQRyqhh7D3
         /xRPKJyASXCCiLff6J8Jim3laX/vddtOyD5vM8RuI1wOLkcAtGy8sbKYNs3GcAXiYpKJ
         6ltkFCd0NOB6HQ78qDzb+PcKtwM0ezl/WuazI=
MIME-Version: 1.0
Received: by 10.223.4.137 with SMTP id 9mr194332far.95.1259343701759; Fri, 27 
	Nov 2009 09:41:41 -0800 (PST)
Date: Fri, 27 Nov 2009 10:41:41 -0700
Message-ID: <7b3355cb0911270941s219c492w63d8ce6c83eeb96b@mail.gmail.com>
Subject: Introducing Cloud MapReduce
From: Bruce Snyder <bruce.snyder@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Given that Hadoop is the dominant project at the ASF for
cloud-oriented Java applications, I'd like to introduce a new project
in the open source world named Cloud MapReduce. Below are the
pertinent details for the project followed by some info describing why
I'm posting on this list.

What: Created by Huan Liu and Dan Orban at Accenture Labs, Cloud
MapReduce is a MapReduce framework built on top of the Amazon
Infrastructure services. It utilizes these services (Amazon EC2,
Amazon S3, Amazon SQS and Amazon SimpleDB) to create a unique
MapReduce framework with some distinct advantages. Cloud MapReduce
utilizes the Apache License 2.0.

Where: The code base currently lives in a GoogleCode project so it is
available for download by anyone:

http://code.google.com/p/cloudmapreduce/

There is also more information about Cloud MapReduce at the website,
including a detailed technical report.

Why: Cloud MapReduce offers three primary advantages over other cloud
frameworks:

1) It is faster than other implementations (e.g., 60 times faster than
Hadoop in one case. Speedup depends on the application and data.).
2) It is more scalable and more failure resistant because it has no
single point of bottleneck.
3) It is dramatically simpler with only 3,000 lines of code (e.g., two
orders of magnitude simpler than Hadoop).


Cloud MapReduce is a new framework that is seeking a good home.
Creator Huan Liu came to me because, in his words, he 'wants to build
a community around Cloud MapReduce.' What better place to do this than
the ASF! In speaking with Doug Cutting and others at ApacheCon US a
few weeks ago, we agreed that the best approach to begin would be via
this list in order to get the best breadth possible across the Apache
cloud-related community.

In finding a home at the ASF for Cloud MapReduce, we are aware that it
does not complement Hadoop really, so making it a subproject of Hadoop
is not necessarily the best approach. Another approach would be to
take it through the Incubator in order to eventually graduate as a top
level project.

Exposure to a wider audience is always the best way to grow an idea
and to explore corners that we may have overlooked. The intent of this
message is to start a conversation around Cloud MapReduce, ask
question, discuss its future and generally figure out how best to
bring it to the ASF. Hopefully the information here is enough for
folks to begin asking questions.

Bruce
-- 
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'

ActiveMQ in Action: http://bit.ly/2je6cQ
Blog: http://bruceblog.org/
Twitter: http://twitter.com/brucesnyder

From general-return-794-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 27 17:42:22 2009
Return-Path: <general-return-794-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4399 invoked from network); 27 Nov 2009 17:42:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Nov 2009 17:42:22 -0000
Received: (qmail 77904 invoked by uid 500); 27 Nov 2009 17:42:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77814 invoked by uid 500); 27 Nov 2009 17:42:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77804 invoked by uid 99); 27 Nov 2009 17:42:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Nov 2009 17:42:21 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE,NORMAL_HTTP_TO_IP,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of fredwang222@gmail.com designates 209.85.216.201 as permitted sender)
Received: from [209.85.216.201] (HELO mail-px0-f201.google.com) (209.85.216.201)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Nov 2009 17:42:18 +0000
Received: by pxi39 with SMTP id 39so1208600pxi.2
        for <general@hadoop.apache.org>; Fri, 27 Nov 2009 09:41:58 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=eE9zjKdxkC9NuM2+oK9rCAIllkEqlefPkm2iCNPWT34=;
        b=WKESHcKwWgEiTBTnBDQimbPI7fOJB9LQ+sm3HtwtIBdc5eya7/ASoifrQBnkTVv5zg
         6rAUS9GrezDdQ8SKFeL+BBM7ZNlIAgxDkmyYk6NIcCOap953hZaG/YG+Oo7tMm3q86v3
         SeKYMSB/4BOwevJdVF4TYv71U9W/kAJqUFEqI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=fNCIgPPzF53IK7Fvqxi/KHbG5ZSGzJxCEKkhrI1HwlXLX5voZw2TcJyqOECIi95uDU
         +z3e+MK5S2CdCQ2V9KieRIrp1Iz/QwSn7rCL0my8tjJtGZbHgrOin5s5FfybkzSIy/hi
         2U02PeNGlU+2oNEbCXSBLTDYj4v687Cwb5h2w=
MIME-Version: 1.0
Received: by 10.142.61.25 with SMTP id j25mr122798wfa.320.1259343718281; Fri, 
	27 Nov 2009 09:41:58 -0800 (PST)
Date: Sat, 28 Nov 2009 01:41:58 +0800
Message-ID: <702261350911270941h26165071i19415b71fe58b171@mail.gmail.com>
Subject: "org.apache.hadoop.mapred.TaskRunner: Communication exception: 
	java.io.IOException: Call to /127.0.0.1:" from time to time
From: fred wang <fredwang222@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636e0b604fa73c004795dce22

--001636e0b604fa73c004795dce22
Content-Type: text/plain; charset=ISO-8859-1

It occures in our hadoop cluster from time to time(the log is below). Anyone
could help? I would like to know what are in the process of(after)
committing? What kind of reasons could make communication error? I think the
tasknode is alive. Thanks for the help.


2009-11-25 01:21:30,476 WARN org.apache.hadoop.fs.FileSystem: "master1:9000"
is a deprecated filesystem name. Use "hdfs://master1:9000/" instead.
2009-11-25 01:21:30,541 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=MAP, sessionId=
2009-11-25 01:21:30,559 WARN org.apache.hadoop.fs.FileSystem: "master1:9000"
is a deprecated filesystem name. Use "hdfs://master1:9000/" instead.
2009-11-25 01:21:31,584 INFO com.ibm.jaql.io.AdapterStore: instantiating
empty AdapterStore
2009-11-25 01:21:31,616 INFO org.apache.hadoop.mapred.MapTask:
numReduceTasks: 27
2009-11-25 01:21:31,624 WARN org.apache.hadoop.fs.FileSystem: "master1:9000"
is a deprecated filesystem name. Use "hdfs://master1:9000/" instead.
2009-11-25 01:21:31,627 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb =
100
2009-11-25 01:21:31,799 INFO org.apache.hadoop.mapred.MapTask: data buffer =
79691776/99614720
2009-11-25 01:21:31,799 INFO org.apache.hadoop.mapred.MapTask: record buffer
= 262144/327680
2009-11-25 01:21:33,266 INFO org.apache.hadoop.mapred.MapTask: Starting
flush of map output
2009-11-25 01:21:33,277 WARN org.apache.hadoop.fs.FileSystem: "master1:9000"
is a deprecated filesystem name. Use "hdfs://master1:9000/" instead.
2009-11-25 01:21:37,187 INFO org.apache.hadoop.mapred.MapTask: Finished
spill 0
2009-11-25 01:21:37,236 WARN org.apache.hadoop.fs.FileSystem: "master1:9000"
is a deprecated filesystem name. Use "hdfs://master1:9000/" instead.
2009-11-25 01:21:37,271 INFO org.apache.hadoop.mapred.TaskRunner:
Task:attempt_200911171829_0181_m_000008_0 is done. And is in the process of
commiting
2009-11-25 01:22:37,323 INFO org.apache.hadoop.mapred.TaskRunner:
Communication exception: java.io.IOException: Call to
/127.0.0.1:53674failed on local exception:
java.nio.channels.ClosedByInterruptException
 at org.apache.hadoop.ipc.Client.wrapException(Client.java:774)
 at org.apache.hadoop.ipc.Client.call(Client.java:742)
 at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
 at org.apache.hadoop.mapred.$Proxy0.statusUpdate(Unknown Source)
 at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:543)
 at java.lang.Thread.run(Thread.java:619)
Caused by: java.nio.channels.ClosedByInterruptException
 at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
 at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:341)
 at
org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
 at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
 at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
 at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
 at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
 at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
 at java.io.DataOutputStream.flush(DataOutputStream.java:106)
 at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:480)
 at org.apache.hadoop.ipc.Client.call(Client.java:720)
 ... 4 more

2009-11-25 01:22:37,338 INFO org.apache.hadoop.mapred.TaskRunner: Task
'attempt_200911171829_0181_m_000008_0' done.


--------------------------------------------------------------------------------

--001636e0b604fa73c004795dce22--

From general-return-795-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 27 18:35:07 2009
Return-Path: <general-return-795-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16347 invoked from network); 27 Nov 2009 18:35:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Nov 2009 18:35:07 -0000
Received: (qmail 41422 invoked by uid 500); 27 Nov 2009 18:35:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 41349 invoked by uid 500); 27 Nov 2009 18:35:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 41339 invoked by uid 99); 27 Nov 2009 18:35:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Nov 2009 18:35:06 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of timrobertson100@gmail.com designates 74.125.78.26 as permitted sender)
Received: from [74.125.78.26] (HELO ey-out-2122.google.com) (74.125.78.26)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Nov 2009 18:34:55 +0000
Received: by ey-out-2122.google.com with SMTP id 4so431316eyf.23
        for <general@hadoop.apache.org>; Fri, 27 Nov 2009 10:34:35 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=ZrnP2JrysKozyUJrN585O8fdWlixKiXynrrr+MRH7fM=;
        b=fyTMK6TwuRlyOhtaCiaW1DIN9EX7ZW2U9NlU06SsSpZHBu3FKGQsoQ673H+KB/DLyr
         +aBi6019FpohCddju8pPsVrX3x/ZTTqMzZSiILtHkvaFEFwqWmfHV7aF7ZTNMX2yRokU
         ZEH/CkG+8Cywo1NNnx535uUguAOBYUx2T3cNU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=T21cgEeJgTERo7U4PXezKIos6ovlI6sy2F7EcizRTHpTgsL9kFcDRoSfxspMn2MYKh
         zmSfCaT3bmjKNlb07gCqzuwvo5HN/zcLLMV0gcrOMOTv2YdMUxmvwhybd6PL4Gyw/GsV
         ylscuzh5bQQDVunR7KkegU6grTsw0QmECoTbA=
MIME-Version: 1.0
Received: by 10.216.91.82 with SMTP id g60mr439135wef.98.1259346875348; Fri, 
	27 Nov 2009 10:34:35 -0800 (PST)
In-Reply-To: <7b3355cb0911270941s219c492w63d8ce6c83eeb96b@mail.gmail.com>
References: <7b3355cb0911270941s219c492w63d8ce6c83eeb96b@mail.gmail.com>
Date: Fri, 27 Nov 2009 19:34:35 +0100
Message-ID: <32120a6a0911271034i150078b6s8d0b23b882c6079@mail.gmail.com>
Subject: Re: Introducing Cloud MapReduce
From: Tim Robertson <timrobertson100@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Bruce,

Interesting stuff.  It looks like it only works with String Keys and
Values (possibly a reason you see large performance gains with simpler
serialization requirements) - have you any plans to support other
types in the roadmap? Perhaps with Protobufs or Avro serialization?

Have you considered using the same Mapper and Reducer class and method
signatures so that a MR job could be written and launched against
CloudMR or Hadoop?
[The reason I ask is I am hacking an implementation of a single JVM,
multithreaded MR framework that does this, so I can ship the same
analysis I run on large datasets to people to run on small datasets
also (much lighter than Hadoop but compatible).  I am putting it on
google code mapreduce4j.  It might be nice if there was a standard MR
API that people coded against and launched in various frameworks -
just a thought].

Cheers,
Tim



On Fri, Nov 27, 2009 at 6:41 PM, Bruce Snyder <bruce.snyder@gmail.com> wrote:
> Given that Hadoop is the dominant project at the ASF for
> cloud-oriented Java applications, I'd like to introduce a new project
> in the open source world named Cloud MapReduce. Below are the
> pertinent details for the project followed by some info describing why
> I'm posting on this list.
>
> What: Created by Huan Liu and Dan Orban at Accenture Labs, Cloud
> MapReduce is a MapReduce framework built on top of the Amazon
> Infrastructure services. It utilizes these services (Amazon EC2,
> Amazon S3, Amazon SQS and Amazon SimpleDB) to create a unique
> MapReduce framework with some distinct advantages. Cloud MapReduce
> utilizes the Apache License 2.0.
>
> Where: The code base currently lives in a GoogleCode project so it is
> available for download by anyone:
>
> http://code.google.com/p/cloudmapreduce/
>
> There is also more information about Cloud MapReduce at the website,
> including a detailed technical report.
>
> Why: Cloud MapReduce offers three primary advantages over other cloud
> frameworks:
>
> 1) It is faster than other implementations (e.g., 60 times faster than
> Hadoop in one case. Speedup depends on the application and data.).
> 2) It is more scalable and more failure resistant because it has no
> single point of bottleneck.
> 3) It is dramatically simpler with only 3,000 lines of code (e.g., two
> orders of magnitude simpler than Hadoop).
>
>
> Cloud MapReduce is a new framework that is seeking a good home.
> Creator Huan Liu came to me because, in his words, he 'wants to build
> a community around Cloud MapReduce.' What better place to do this than
> the ASF! In speaking with Doug Cutting and others at ApacheCon US a
> few weeks ago, we agreed that the best approach to begin would be via
> this list in order to get the best breadth possible across the Apache
> cloud-related community.
>
> In finding a home at the ASF for Cloud MapReduce, we are aware that it
> does not complement Hadoop really, so making it a subproject of Hadoop
> is not necessarily the best approach. Another approach would be to
> take it through the Incubator in order to eventually graduate as a top
> level project.
>
> Exposure to a wider audience is always the best way to grow an idea
> and to explore corners that we may have overlooked. The intent of this
> message is to start a conversation around Cloud MapReduce, ask
> question, discuss its future and generally figure out how best to
> bring it to the ASF. Hopefully the information here is enough for
> folks to begin asking questions.
>
> Bruce
> --
> perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> );'
>
> ActiveMQ in Action: http://bit.ly/2je6cQ
> Blog: http://bruceblog.org/
> Twitter: http://twitter.com/brucesnyder
>

From general-return-796-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 27 21:19:07 2009
Return-Path: <general-return-796-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54950 invoked from network); 27 Nov 2009 21:19:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Nov 2009 21:19:07 -0000
Received: (qmail 91059 invoked by uid 500); 27 Nov 2009 21:19:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90974 invoked by uid 500); 27 Nov 2009 21:19:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90964 invoked by uid 99); 27 Nov 2009 21:19:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Nov 2009 21:19:06 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of bruce.snyder@gmail.com designates 209.85.220.224 as permitted sender)
Received: from [209.85.220.224] (HELO mail-fx0-f224.google.com) (209.85.220.224)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Nov 2009 21:18:55 +0000
Received: by fxm24 with SMTP id 24so1630021fxm.11
        for <general@hadoop.apache.org>; Fri, 27 Nov 2009 13:18:35 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=4zVDQwe69tbByz07LbdIq4L/mXnm2Pf6PozBkuekyuY=;
        b=CecbGPctf+vY5+Q6emHjEbAx5PaicAUaZ7VDru/FudrmYDZk/czeHIDbPIAYLFboQy
         N830WpsDKBp/01wg18K//Y37L4tFXhHW0ky4wUZFjAkxBmFn1QlxlFwU9xJkHmYe3NGs
         IAMjH1QGrTHfU6qFPjMw/t5eP8GIrkQpkzLcs=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=CVKO6fHYwdyBmXDpenADn975gjbzBAmL9AUNnBbmKaDjLCLR9REF87j2la6ZX85O+r
         HvHpl8d2AROVBUbjYsY2qz1gZD+X+1h+PHuWdp3FLLWyo7FAWV7UlKqVT0KpP7+7667+
         HRVUzXswL0n4yeU7272YOLgk+kJH0+gz4C9EU=
MIME-Version: 1.0
Received: by 10.223.20.210 with SMTP id g18mr232280fab.9.1259356715548; Fri, 
	27 Nov 2009 13:18:35 -0800 (PST)
In-Reply-To: <32120a6a0911271034i150078b6s8d0b23b882c6079@mail.gmail.com>
References: <7b3355cb0911270941s219c492w63d8ce6c83eeb96b@mail.gmail.com>
	 <32120a6a0911271034i150078b6s8d0b23b882c6079@mail.gmail.com>
Date: Fri, 27 Nov 2009 14:18:35 -0700
Message-ID: <7b3355cb0911271318u4797fc2br51a241bd0fa95465@mail.gmail.com>
Subject: Re: Introducing Cloud MapReduce
From: Bruce Snyder <bruce.snyder@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Fri, Nov 27, 2009 at 11:34 AM, Tim Robertson
<timrobertson100@gmail.com> wrote:
> Hi Bruce,
>
> Interesting stuff. =A0It looks like it only works with String Keys and
> Values (possibly a reason you see large performance gains with simpler
> serialization requirements) - have you any plans to support other
> types in the roadmap? Perhaps with Protobufs or Avro serialization?

I can definitely see the need for supporting more than one type of
key/value. We'll need to add this to the roadmap, thanks.

> Have you considered using the same Mapper and Reducer class and method
> signatures so that a MR job could be written and launched against
> CloudMR or Hadoop?
> [The reason I ask is I am hacking an implementation of a single JVM,
> multithreaded MR framework that does this, so I can ship the same
> analysis I run on large datasets to people to run on small datasets
> also (much lighter than Hadoop but compatible). =A0I am putting it on
> google code mapreduce4j. =A0It might be nice if there was a standard MR
> API that people coded against and launched in various frameworks -
> just a thought].

Excellent points, Tim. A point that I raised was trying to determine a
standard sort of API that would work across implementations. So far
I'm not sure if this means that we match what is already out there or
if we provide interfaces for many languages or what. I like the idea,
but I'll let Huan chime in here.

Bruce
--=20
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=3D6-E+G-N>61E<D\!G;6%I=
;\"YC;VT*"
);'

ActiveMQ in Action: http://bit.ly/2je6cQ
Blog: http://bruceblog.org/
Twitter: http://twitter.com/brucesnyder

From general-return-797-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Nov 29 03:56:24 2009
Return-Path: <general-return-797-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96162 invoked from network); 29 Nov 2009 03:56:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Nov 2009 03:56:24 -0000
Received: (qmail 15970 invoked by uid 500); 29 Nov 2009 03:56:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15866 invoked by uid 500); 29 Nov 2009 03:56:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15856 invoked by uid 99); 29 Nov 2009 03:56:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 Nov 2009 03:56:20 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of zs68j2ee@gmail.com designates 209.85.222.198 as permitted sender)
Received: from [209.85.222.198] (HELO mail-pz0-f198.google.com) (209.85.222.198)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 Nov 2009 03:56:12 +0000
Received: by pzk36 with SMTP id 36so1794521pzk.5
        for <general@hadoop.apache.org>; Sat, 28 Nov 2009 19:55:51 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=zl1Lzc3o7wMJAXKjX5Ev9M0rZelg/K57WbCvulo5UpA=;
        b=IaVKYEJgYO5zfVxt0e1FBmRkHfhdoazApFM7ogPrzGI6KYeb6pC0CCzOQ90vfKQZut
         XOKR0ebX+IFPwj3pESCB8/pXlSL7oNzL2/5ylMqxyPTFYp0g95M94xOucS2smaJw1hLh
         TJO9G0po6nxitiztvvbLN5a9uK5ODO19C3r1M=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=QNs5saExfZQKM9sC+tB7CwGKsqleXUx+zdMQKysC22Dq8K07IoeEJpdafmLWcPnOAi
         MJWaZm7JKyCWHir9CS9fRJLLxWS8ZWZ7qVhduzsTR6zNPLc5RiCNxcc7iATSBySRoF2B
         GV+9db0Qeyp1cXwHFBLRf1x+I/6HY32oXSCRs=
MIME-Version: 1.0
Received: by 10.141.49.12 with SMTP id b12mr178113rvk.64.1259466951453; Sat, 
	28 Nov 2009 19:55:51 -0800 (PST)
In-Reply-To: <c28cbc140911281944j4925292chfc8f87ce1e4c3c28@mail.gmail.com>
References: <c28cbc140911281944j4925292chfc8f87ce1e4c3c28@mail.gmail.com>
Date: Sun, 29 Nov 2009 11:55:51 +0800
Message-ID: <c28cbc140911281955j162fcf10w3aade804544a0938@mail.gmail.com>
Subject: what is the major difference between Hadoop and cloudMapReduce?
From: tom <zs68j2ee@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

=C2=A0I know Hadoop is a suite of software service for the cluster
computing, including mapreduce implementation, distributed database,
distributed file system, etc.
=C2=A0the confusion to me is If cloudMapReduce just is focused on Cloud
Computing Infrastructure and must be depend on overall distributed
service provided and=C2=A0hidden by Cloud Computing Infrastructure?

 And, cloudMapReduce just can be a alternate implementation of MapReduce?
 Is it right?

=C2=A0Best regards
=C2=A0 =C2=A0Tom

From general-return-798-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Nov 29 04:37:32 2009
Return-Path: <general-return-798-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 2308 invoked from network); 29 Nov 2009 04:37:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Nov 2009 04:37:32 -0000
Received: (qmail 28445 invoked by uid 500); 29 Nov 2009 04:37:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28263 invoked by uid 500); 29 Nov 2009 04:37:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28253 invoked by uid 99); 29 Nov 2009 04:37:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 Nov 2009 04:37:29 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bruce.snyder@gmail.com designates 209.85.218.223 as permitted sender)
Received: from [209.85.218.223] (HELO mail-bw0-f223.google.com) (209.85.218.223)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 Nov 2009 04:37:24 +0000
Received: by bwz23 with SMTP id 23so1920654bwz.29
        for <general@hadoop.apache.org>; Sat, 28 Nov 2009 20:36:59 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=fOltpQOFzw/nnksecVuTK/hnWw+OIUDgdjpfsAQ9RQI=;
        b=cMyjkWS2XuXNlY8+4daRydySnVzb5Dt/IxvimwIZV0X+aPvE1RTg0hPO7dzPY01iyZ
         EdBKoxWWEdzRh+D/w/lcWKUm4YnCQDlsD8LJ2Lg/FCTn1dmxaDSaHNax1i1m66+kQKde
         TVqcLmVpGEVTe4xXVcDB8s+vJjinnPwyn7dAk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=wpFTcVwEnxc1Fvd9EZYjpymzMZCRG7KUYGpBTXDxsVHOeC21+hW8YbAaQ317Fo84UO
         UdQiuMutP/t9btWtY/usSLk/6ZkdHZ+CAp8yiANa6NN5Cn47bjPQZOTCC9r5PXZ9y2+f
         nWt4LwDByIq3Ewt9MOhZc1/IPR1kJbKA6rUV0=
MIME-Version: 1.0
Received: by 10.204.34.13 with SMTP id j13mr2972770bkd.32.1259469419765; Sat, 
	28 Nov 2009 20:36:59 -0800 (PST)
In-Reply-To: <c28cbc140911281955j162fcf10w3aade804544a0938@mail.gmail.com>
References: <c28cbc140911281944j4925292chfc8f87ce1e4c3c28@mail.gmail.com>
	 <c28cbc140911281955j162fcf10w3aade804544a0938@mail.gmail.com>
Date: Sat, 28 Nov 2009 21:36:59 -0700
Message-ID: <7b3355cb0911282036x394f0acft21da05803b16a01@mail.gmail.com>
Subject: Re: what is the major difference between Hadoop and cloudMapReduce?
From: Bruce Snyder <bruce.snyder@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Sat, Nov 28, 2009 at 8:55 PM, tom <zs68j2ee@gmail.com> wrote:
> Hi,
>
> =A0I know Hadoop is a suite of software service for the cluster
> computing, including mapreduce implementation, distributed database,
> distributed file system, etc.
> =A0the confusion to me is If cloudMapReduce just is focused on Cloud
> Computing Infrastructure and must be depend on overall distributed
> service provided and=A0hidden by Cloud Computing Infrastructure?
>
> =A0And, cloudMapReduce just can be a alternate implementation of MapReduc=
e?
> =A0Is it right?

Yes, Cloud MapReduce is an alternate implementation of MapReduce. The
difference between Cloud MapReduce and Hadoop is that Cloud MapReduce
depends upon the infrastructure services provided by Amazon (e.g.,
AMIs, S3, SQS and SimpleDB) instead of providing it's own
implementation of each.

On a side note, I find it interesting that Amazon is now offering a
product based on Hadoop named Amazon Elastic MapReduce:

http://aws.amazon.com/elasticmapreduce/

Bruce
--=20
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=3D6-E+G-N>61E<D\!G;6%I=
;\"YC;VT*"
);'

ActiveMQ in Action: http://bit.ly/2je6cQ
Blog: http://bruceblog.org/
Twitter: http://twitter.com/brucesnyder

From general-return-799-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Nov 29 10:13:29 2009
Return-Path: <general-return-799-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93217 invoked from network); 29 Nov 2009 10:13:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Nov 2009 10:13:29 -0000
Received: (qmail 7782 invoked by uid 500); 29 Nov 2009 10:13:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7686 invoked by uid 500); 29 Nov 2009 10:13:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7672 invoked by uid 99); 29 Nov 2009 10:13:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 Nov 2009 10:13:27 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of timrobertson100@gmail.com designates 209.85.219.217 as permitted sender)
Received: from [209.85.219.217] (HELO mail-ew0-f217.google.com) (209.85.219.217)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 Nov 2009 10:13:24 +0000
Received: by ewy9 with SMTP id 9so165834ewy.11
        for <general@hadoop.apache.org>; Sun, 29 Nov 2009 02:13:03 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=ygEU+JFBqWJMV4kDuorUGIf8VImZ9lrSy2ld/3SQyhE=;
        b=rZ4fVbe7OLOyLG2X3VaBc4dvNZtxNWAfyFb+zT2CMczONuxMeuvO4kZACSE4wFnhul
         FXW6WNdq1Y7ISEdKz/v9azWb4HEVRK2Qc4xH/zvQcA1E3OMTwYlo8n/HHEpXa8DdW0XN
         kGV2XX3YBV7XCHkMZnozDJdbgxziFiWhywIn8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=TP8H7ZxRKbJWPz1xiRkM7OhXsdc7lkeTY5Km5R0PvMtxDhOn/i66EvCYEBT+9MmHoH
         mv/vLLstSk5Jz0GhzOH3FWr5hDgh1G/2RMkz4bUDlKNaQTuCwvFUavbVJ6+HYzjBm5th
         ongurIXf6IeN/cfn9FC7+3KCp+NvtkqfJpQyI=
MIME-Version: 1.0
Received: by 10.216.87.7 with SMTP id x7mr946480wee.53.1259489583271; Sun, 29 
	Nov 2009 02:13:03 -0800 (PST)
In-Reply-To: <7b3355cb0911282036x394f0acft21da05803b16a01@mail.gmail.com>
References: <c28cbc140911281944j4925292chfc8f87ce1e4c3c28@mail.gmail.com>
	 <c28cbc140911281955j162fcf10w3aade804544a0938@mail.gmail.com>
	 <7b3355cb0911282036x394f0acft21da05803b16a01@mail.gmail.com>
Date: Sun, 29 Nov 2009 11:13:03 +0100
Message-ID: <32120a6a0911290213m5563e1fge8bcc4413909a505@mail.gmail.com>
Subject: Re: what is the major difference between Hadoop and cloudMapReduce?
From: Tim Robertson <timrobertson100@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Tom,

One major difference is that Hadoop allows working with any object
that you can serialize, rather than only Strings so it is applicable
for use with (e.g) image processing, or rendering to images or PDFs on
the output format.

Cheers
Tim



On Sun, Nov 29, 2009 at 5:36 AM, Bruce Snyder <bruce.snyder@gmail.com> wrot=
e:
> On Sat, Nov 28, 2009 at 8:55 PM, tom <zs68j2ee@gmail.com> wrote:
>> Hi,
>>
>> =A0I know Hadoop is a suite of software service for the cluster
>> computing, including mapreduce implementation, distributed database,
>> distributed file system, etc.
>> =A0the confusion to me is If cloudMapReduce just is focused on Cloud
>> Computing Infrastructure and must be depend on overall distributed
>> service provided and=A0hidden by Cloud Computing Infrastructure?
>>
>> =A0And, cloudMapReduce just can be a alternate implementation of MapRedu=
ce?
>> =A0Is it right?
>
> Yes, Cloud MapReduce is an alternate implementation of MapReduce. The
> difference between Cloud MapReduce and Hadoop is that Cloud MapReduce
> depends upon the infrastructure services provided by Amazon (e.g.,
> AMIs, S3, SQS and SimpleDB) instead of providing it's own
> implementation of each.
>
> On a side note, I find it interesting that Amazon is now offering a
> product based on Hadoop named Amazon Elastic MapReduce:
>
> http://aws.amazon.com/elasticmapreduce/
>
> Bruce
> --
> perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=3D6-E+G-N>61E<D\!G;6=
%I;\"YC;VT*"
> );'
>
> ActiveMQ in Action: http://bit.ly/2je6cQ
> Blog: http://bruceblog.org/
> Twitter: http://twitter.com/brucesnyder
>

From general-return-800-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Nov 29 16:16:28 2009
Return-Path: <general-return-800-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 65606 invoked from network); 29 Nov 2009 16:16:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Nov 2009 16:16:28 -0000
Received: (qmail 23419 invoked by uid 500); 29 Nov 2009 16:16:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23326 invoked by uid 500); 29 Nov 2009 16:16:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23316 invoked by uid 99); 29 Nov 2009 16:16:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 Nov 2009 16:16:26 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bruce.snyder@gmail.com designates 209.85.220.224 as permitted sender)
Received: from [209.85.220.224] (HELO mail-fx0-f224.google.com) (209.85.220.224)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 Nov 2009 16:16:22 +0000
Received: by fxm24 with SMTP id 24so2392924fxm.11
        for <general@hadoop.apache.org>; Sun, 29 Nov 2009 08:16:01 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=rmsVzmWHuTqPFj2vcWx1FRpfeTmnHIe1J0marxAcVuA=;
        b=RTEeS6doGry+6EmFnsUKJqU5gZQ7oXM9dX/573MjgT80ObzFpFfTM3O3n9jf/mssme
         i5uJQQC/ohokvbKCJWMJWxSzi3tmIj+V099RknXWkuI2pulmACrErFxa78x2p4Vzgw+9
         66ZE4KGakRPQx9aJXKEYmEIR+n2Y5ICTbbdLA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=rtT5MCj5Rl7sCpr+FJrDCE3hwGKgfmT52mSaj+yS9RrNUGaOzCp6icHlh0yhCH2L7X
         vB7Vgu+lNklZEA0Y1wcYL1QEL6M2wE7usdDdYC5Od/bnMebXEFUkFyScpeB8j7cecoPI
         ze4ABuHCmIj8FDBs4R0Q2Kuf7bTT5Ts1yZ1Xk=
MIME-Version: 1.0
Received: by 10.223.144.85 with SMTP id y21mr520153fau.1.1259511361242; Sun, 
	29 Nov 2009 08:16:01 -0800 (PST)
In-Reply-To: <32120a6a0911290213m5563e1fge8bcc4413909a505@mail.gmail.com>
References: <c28cbc140911281944j4925292chfc8f87ce1e4c3c28@mail.gmail.com>
	 <c28cbc140911281955j162fcf10w3aade804544a0938@mail.gmail.com>
	 <7b3355cb0911282036x394f0acft21da05803b16a01@mail.gmail.com>
	 <32120a6a0911290213m5563e1fge8bcc4413909a505@mail.gmail.com>
Date: Sun, 29 Nov 2009 09:16:01 -0700
Message-ID: <7b3355cb0911290816h1dc160e4ua30054c438702215@mail.gmail.com>
Subject: Re: what is the major difference between Hadoop and cloudMapReduce?
From: Bruce Snyder <bruce.snyder@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Sun, Nov 29, 2009 at 3:13 AM, Tim Robertson
<timrobertson100@gmail.com> wrote:
> Hi Tom,
>
> One major difference is that Hadoop allows working with any object
> that you can serialize, rather than only Strings so it is applicable
> for use with (e.g) image processing, or rendering to images or PDFs on
> the output format.

Yes, you noted this on the general@hadoop list. I think your
suggestion to expand this to support other types is a good one.

Bruce
-- 
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'

ActiveMQ in Action: http://bit.ly/2je6cQ
Blog: http://bruceblog.org/
Twitter: http://twitter.com/brucesnyder

From general-return-801-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Nov 29 18:15:46 2009
Return-Path: <general-return-801-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97999 invoked from network); 29 Nov 2009 18:15:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Nov 2009 18:15:46 -0000
Received: (qmail 76085 invoked by uid 500); 29 Nov 2009 18:15:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76001 invoked by uid 500); 29 Nov 2009 18:15:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75991 invoked by uid 99); 29 Nov 2009 18:15:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 Nov 2009 18:15:44 +0000
X-ASF-Spam-Status: No, hits=-2.5 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.85.222.198] (HELO mail-pz0-f198.google.com) (209.85.222.198)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 Nov 2009 18:15:41 +0000
Received: by pzk36 with SMTP id 36so1992875pzk.5
        for <general@hadoop.apache.org>; Sun, 29 Nov 2009 10:15:21 -0800 (PST)
MIME-Version: 1.0
Received: by 10.142.61.35 with SMTP id j35mr357386wfa.52.1259518521105; Sun, 
	29 Nov 2009 10:15:21 -0800 (PST)
In-Reply-To: <7b3355cb0911290816h1dc160e4ua30054c438702215@mail.gmail.com>
References: <c28cbc140911281944j4925292chfc8f87ce1e4c3c28@mail.gmail.com> 
	<c28cbc140911281955j162fcf10w3aade804544a0938@mail.gmail.com> 
	<7b3355cb0911282036x394f0acft21da05803b16a01@mail.gmail.com> 
	<32120a6a0911290213m5563e1fge8bcc4413909a505@mail.gmail.com> 
	<7b3355cb0911290816h1dc160e4ua30054c438702215@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Sun, 29 Nov 2009 10:15:01 -0800
Message-ID: <45f85f70911291015m5165421fsf75f28bc698f28ed@mail.gmail.com>
Subject: Re: what is the major difference between Hadoop and cloudMapReduce?
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636e0b62009dd5b0479868282

--001636e0b62009dd5b0479868282
Content-Type: text/plain; charset=ISO-8859-1

Another difference is that Cloud Mapreduce doesn't scale well when there are
a large number of values for the same key. They collect in a single SQS
queue per partition, and in the reducer reads that into a hashtable that
never spills to disk. So, if you have more values for a key than fit in RAM,
you can't run a job. Hadoop on the other hand spills reduce input to disk
when it's too large to sort in memory. The paper says to add more partitions
if you have a lot of data, but it doesn't help in the case of skewed reduce
input (very common in a lot of workloads)

-Todd

On Sun, Nov 29, 2009 at 8:16 AM, Bruce Snyder <bruce.snyder@gmail.com>wrote:

> On Sun, Nov 29, 2009 at 3:13 AM, Tim Robertson
> <timrobertson100@gmail.com> wrote:
> > Hi Tom,
> >
> > One major difference is that Hadoop allows working with any object
> > that you can serialize, rather than only Strings so it is applicable
> > for use with (e.g) image processing, or rendering to images or PDFs on
> > the output format.
>
> Yes, you noted this on the general@hadoop list. I think your
> suggestion to expand this to support other types is a good one.
>
> Bruce
> --
> perl -e 'print
> unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> );'
>
> ActiveMQ in Action: http://bit.ly/2je6cQ
> Blog: http://bruceblog.org/
> Twitter: http://twitter.com/brucesnyder
>

--001636e0b62009dd5b0479868282--

From general-return-802-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 30 07:52:35 2009
Return-Path: <general-return-802-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47183 invoked from network); 30 Nov 2009 07:52:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Nov 2009 07:52:35 -0000
Received: (qmail 91435 invoked by uid 500); 30 Nov 2009 07:52:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91341 invoked by uid 500); 30 Nov 2009 07:52:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91331 invoked by uid 99); 30 Nov 2009 07:52:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Nov 2009 07:52:33 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mailinglists19@gmail.com designates 209.85.219.217 as permitted sender)
Received: from [209.85.219.217] (HELO mail-ew0-f217.google.com) (209.85.219.217)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Nov 2009 07:52:29 +0000
Received: by ewy9 with SMTP id 9so856675ewy.11
        for <general@hadoop.apache.org>; Sun, 29 Nov 2009 23:52:07 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=GPgxUE3VdyUyR7gucPILdMTDoDD8aWQP/tqsMnq+np8=;
        b=Aa778vmzktaCfDT7lMkggmQqPG5Q+dJi7uXu/WlUTwNwIwhqkai7o56bRjkrRQuT8Q
         p6U0EyROUief6hBLzAeWY6hIV7fQRQRDWA0pbbZMpPvTUy1+RgqUy5oD47xJJlSMX8V/
         02migRdhEppmdUgxXnvgo0Z4W75GdxSNw7A+k=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=ZQsGruDjfqJYkzFhk7yvs8MAcgO9W/o1yy+YcpGhcllP0nFpk5ksXwowENjHcrS+Aq
         ypK6xl+rMgmm3aa+k4TjwS8+7iUQsQ4jbgg2ybgJNRzsOwjXIk+EGny/GPInNvGBQWbq
         jgD8oIid6G7YmwXVxdeH5PfXIjQnuo3JxTSvo=
MIME-Version: 1.0
Received: by 10.216.86.212 with SMTP id w62mr1367782wee.131.1259567525760; 
	Sun, 29 Nov 2009 23:52:05 -0800 (PST)
In-Reply-To: <2eef7bd70911251025s73655a79ned5f6418caba81fb@mail.gmail.com>
References: <1eabbac30911211657i7812c67bs3a1565b0537b418f@mail.gmail.com>
	 <2eef7bd70911241116n6f81550fpfeec6b88c382e0b4@mail.gmail.com>
	 <1eabbac30911241338g1067b99dra633f21f2d0cfd10@mail.gmail.com>
	 <2eef7bd70911241344p4225190fkaaafc160a5c9bb80@mail.gmail.com>
	 <1eabbac30911241406v1a684661va156ab6a856819c3@mail.gmail.com>
	 <2eef7bd70911241458x6e2de4dfx60ef8e68b636f78e@mail.gmail.com>
	 <1eabbac30911250838w3d4b089ascb2e0b21245a60ae@mail.gmail.com>
	 <2eef7bd70911250921p5d2558fq8766748c47dbfb3f@mail.gmail.com>
	 <1eabbac30911250945i49684a5am6510cc6ffb7f86c8@mail.gmail.com>
	 <2eef7bd70911251025s73655a79ned5f6418caba81fb@mail.gmail.com>
Date: Sun, 29 Nov 2009 23:52:05 -0800
Message-ID: <1eabbac30911292352k4f912452j52cab4b532460f05@mail.gmail.com>
Subject: Re: Starting Hadoop cluster on EC2
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d9a3a2f18ee6047991ea68

--0016e6d9a3a2f18ee6047991ea68
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Everything started working after I added the following line to hadoop-env.s=
h

export HADOOP_SSH_OPTS=3D"$HADOOP_SSH_OPTS -i /mnt/myVol/KeyPairName.pem "

Thanks for your help.


On Wed, Nov 25, 2009 at 10:25 AM, Vaibhav Puranik <vpuranik@gmail.com>wrote=
:

> On master, in .ssh folder, there should be a file id_dsa.pub. This is
> master's public key.
>
> Contents of this file should be copied to slave's authorized_keys file.
>
> Regards,
> Vaibhav
>
> On Wed, Nov 25, 2009 at 9:45 AM, Something Something <
> mailinglists19@gmail.com> wrote:
>
> > Can you please explain what you mean by "add master's public key to
> slave's
> > authorized_keys"?  I have a feeling I am not doing this correctly.  Wha=
t
> I
> > did is this:
> >
> > 1)  On Master, opened ~/.ssh/authorized_key and copied lines that look
> like
> > this...
> >
> > ssh-dss
> >
> >
> AAAAB3NzaC1kc3MAAACBAKJOife8p+6vjJ/jODSWaBgyawN7XWmasasasb0cVqhe1WOpnJQpX=
uJeqsaBaZYHYxj8P74m1rOIdN9lSPAz6gEjHo4ft8UM7ZVExRC6HDM7JdfdferhwRgQSVI9ruTL=
lssO9i1D17mBnLotsgy92CyS7bsanM25nRqhmvqkX7tuKp66nlAAAAFQCnbwPn3v+ttorRZ4Oij=
kAKoJaVFQAAAIAGVyZGwkm5DwYvige0VYhXZns8qSvW15dwSrS1a6k41q3C91AQZylvJYDnBjNG=
nWwUHfyWpyruzVCOskavEmsl6FcouU5Av1zLRT1sk4llUzZxXFtv6gzRBq2+aiCcoex3O+RRJbH=
a89cMLy6jeZOCFahwVz6IyHcQ0b7gZjQgFQAAAIBA4vzSDy5oN27Ji3mF9ZklsSmlJJrrJxGJfO=
n6/BK5tyvBijLy6tEJHrH3dY8TEJUp2V9RINKwcsy5LdqyI5gJRWoM8WdNzmUoIOLAW0HIVl2Ml=
dKKqP/ctQwGI94EjykSUYVt/IAvUcjFuIrtiGrMDjsqfm7h1Gi2ZtDQ=3D=3D
> > root@domU-12-31-38-00-39-C8
> >
> >
> > to the end of ~/.ssh/authorized_key on Slave.
> >
> > Something tells me there's a better way to do this.  Is there?
> >
> >
> > On Wed, Nov 25, 2009 at 9:21 AM, Vaibhav Puranik <vpuranik@gmail.com>
> > wrote:
> >
> > > Actually I was wrong about adding keypair. It should be resolved by
> > adding
> > > master's public key to slave's authorized_keys. Amazon puts your
> keypair
> > in
> > > authroized_keys automatically. I missed that part.
> > >
> > > Regards,
> > > Vaibhav
> > >
> > > On Wed, Nov 25, 2009 at 8:38 AM, Something Something <
> > > mailinglists19@gmail.com> wrote:
> > >
> > > > Me again :(
> > > >
> > > > I decided to start fresh so that I can document the steps, but now =
I
> > > can't
> > > > even go further than I did before :(
> > > >
> > > > Please let me know what step I am missing:
> > > >
> > > > 1)  Launched 2 instances on AWS console - one for Master, one for
> > Slave.
> > > >
> > > > 2)  In one command window, connected to instance 1:
> > > > ssh -i MyKeyPair.pem <instance 1>
> > > > (Got 'authenticity' warning.. typed 'yes')
> > > >
> > > > This is my "Master" window.
> > > >
> > > > 3)  Opened another command window & connected to instance 2:
> > > > ssh -i MyKeyPair.pem <instance 2>
> > > > (Got 'authenticity' warning.. typed 'yes')
> > > >
> > > > This is my "Slave" window.
> > > >
> > > > 4) Opened another command window and 'scp' the key pair (from my
> local
> > > > machine) to instance 1 & instance 2.
> > > > scp -i MyKeyPair.pem MyKeyPair.pem <master instance>:/tmp
> > > > scp -i MyKeyPair.pem MyKeyPair.pem <slave instance>:/tmp
> > > >
> > > > 5)  On BOTH (Master & Slave) added Key Pair to 'authorized_key'
> > > >
> > > > cat /tmp/MyKeyPair.pem >> ~/.ssh/authorized_keys
> > > >
> > > > 6)  To setup passphraseless ssh ran these on BOTH - Master & Slave:
> > > > ssh localhost  (Got 'authenticiy' message.  Typed 'yes'.  Got
> > Permission
> > > > denied (public key)
> > > > ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
> > > > cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
> > > >
> > > > 7)  In order to ssh from Master to Slave (without entering KeyPair)=
,
> > > copied
> > > > the key (that starts with 'sh-dss') from Slave's 'authorized_key'
> file
> > to
> > > > Master's 'authoried_key' file.
> > > >
> > > > Now, every time I try (from Master):
> > > >
> > > > ssh <slave's ip address>
> > > >
> > > > I get - "Permission denied (publickey).
> > > >
> > > > But, when I type:
> > > >
> > > > ssh -i /tmp/MyKeyPair.pem <slave's ip address>
> > > >
> > > > It works.
> > > >
> > > > I thought step #5 above should have resolved this.  What am I doing
> > > wrong?
> > > > I believe, in order for me to start the cluster, I should be able t=
o
> > > 'ssh'
> > > > to slave without using key pair, correct?
> > > >
> > > >
> > > > Thanks for your help.
> > > >
> > > >
> > > > On Tue, Nov 24, 2009 at 2:58 PM, Vaibhav Puranik <vpuranik@gmail.co=
m
> >
> > > > wrote:
> > > >
> > > > > Install a program called Jps.
> > > > >
> > > > > It's a ps for java processes. execute jps, you should see the
> > processes
> > > > (if
> > > > > they are running).
> > > > >
> > > > > Also make sure you check your logs.
> > > > >
> > > > > Regards,
> > > > > Vaibhav
> > > > >
> > > > > On Tue, Nov 24, 2009 at 2:06 PM, Something Something <
> > > > > mailinglists19@gmail.com> wrote:
> > > > >
> > > > > > Awesome.  Thanks, Vaibhav.  Made progress.  No error messages
> from
> > > > > > ./all-start.sh
> > > > > >
> > > > > > I will test to see if the cluster is deployed properly.
> > > > > >
> > > > > > Question:  On the slave, when I do:
> > > > > >
> > > > > > ps -eaf | grep 'hadoop'
> > > > > >
> > > > > > I don't see any processes.  Shouldn't I see 2 processes - one f=
or
> > > > > Datanode
> > > > > > and the other for TaskTracker?
> > > > > >
> > > > > > Please let me know.  Thanks again.
> > > > > >
> > > > > >
> > > > > > On Tue, Nov 24, 2009 at 1:44 PM, Vaibhav Puranik <
> > vpuranik@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Yes, I pretty much meant that.
> > > > > > >
> > > > > > > Add your master's public key to authorized_keys on slaves too=
.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Vaibhav
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Nov 24, 2009 at 1:38 PM, Something Something <
> > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > >
> > > > > > > > Sorry.. but not sure what you mean by... "In order to achie=
ve
> > > that
> > > > > you
> > > > > > > need
> > > > > > > > to add the keypair  you are using to your
> > > > > > > > auhtorized_keys file in the home dir/.ssh folder."
> > > > > > > >
> > > > > > > >
> > > > > > > > I tried this....
> > > > > > > >
> > > > > > > > cat KeyPair.pem >> /root/.ssh/authorized_keys
> > > > > > > >
> > > > > > > > Is that what you meant by "add the keypair to
> authorized_keys"?
> > > > > > > >
> > > > > > > > Still can't ssh from Slave to Master.  It says.....
>  Permission
> > > > > denied
> > > > > > > > (publickey).
> > > > > > > >
> > > > > > > >
> > > > > > > > Sorry, I am a newbie when it comes to such environment
> issues.
> > > > >  Thanks
> > > > > > > for
> > > > > > > > your patience.  It's greatly appreciated.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Nov 24, 2009 at 11:16 AM, Vaibhav Puranik <
> > > > > vpuranik@gmail.com
> > > > > > > > >wrote:
> > > > > > > >
> > > > > > > > > You need to be able to SSH without specifying the keypair
> > from
> > > a
> > > > > > slave
> > > > > > > to
> > > > > > > > > master (I think reverse it true too, but I am not sure).
> > > > > > > > >
> > > > > > > > > In order to achieve that you need to add the keypair  you
> are
> > > > using
> > > > > > to
> > > > > > > > your
> > > > > > > > > auhtorized_keys file in the home dir/.ssh folder.
> > > > > > > > >
> > > > > > > > > Once you setup passphraseless ssh, it should start workin=
g.
> > > > > > Furthrmore,
> > > > > > > > > after you set this up, try to ssh manully (without key) a=
nd
> > if
> > > it
> > > > > > asks
> > > > > > > > you
> > > > > > > > > the following message - say yes:
> > > > > > > > >
> > > > > > > > > The authenticity of host
> > > > > > > > > 'ec2-147-127-186-243.compute-1.amazonaws.com
> > (147.127.186.243)'
> > > > > can't
> > > > > > > > > be established.
> > > > > > > > > RSA key fingerprint is
> > > > > > 4b:63:e2:23:16:6f:b6:99:de:34:f6:9b:f5:55:73:8b.
> > > > > > > > > Are you sure you want to continue connecting (yes/no)? ye=
s
> > > > > > > > > Warning: Permanently added
> > > > > > > > > 'ec2-147-127-186-243.compute-1.amazonaws.com
> > ,147.127.186.243'
> > > > > > > > > (RSA) to the list of known hosts.
> > > > > > > > >
> > > > > > > > > In other words, you have to make sure that the machine yo=
u
> > are
> > > > > trying
> > > > > > > to
> > > > > > > > > ssh
> > > > > > > > > is in your known_hosts file.
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Vaibhav
> > > > > > > > >
> > > > > > > > > On Tue, Nov 24, 2009 at 10:54 AM, Something Something <
> > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > > Thanks, Vaibhav.  That was helpful.  I am able to scp
> from
> > my
> > > > > local
> > > > > > > > > machine
> > > > > > > > > > after I opened port 22 under my security group.
> > > > > > > > > >
> > > > > > > > > > But my cluster is still not starting as expected.  I
> > mean...
> > > on
> > > > > my
> > > > > > > > Master
> > > > > > > > > > when I run:
> > > > > > > > > >
> > > > > > > > > > ./start-all.sh
> > > > > > > > > >
> > > > > > > > > > I get messages saying...
> > > > > > > > > >
> > > > > > > > > > 10.252.xxx.xxx: Permission denied (publickey).
> > > > > > > > > >
> > > > > > > > > > where 10.252.xxx.xxx is the ip address of my Slave.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I can ssh from my Slave to master if I do this:
> > > > > > > > > >
> > > > > > > > > > ssh -i <keypairname>.pem <Master's ip address>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Any suggestions?  Thanks for your help.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, Nov 23, 2009 at 8:10 AM, Vaibhav Puranik <
> > > > > > vpuranik@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > In order to use scp from your machine to an ec2
> instance
> > > you
> > > > > need
> > > > > > > to
> > > > > > > > > use
> > > > > > > > > > > scp
> > > > > > > > > > > -i and give it the path of your keypair.
> > > > > > > > > > >
> > > > > > > > > > > scp -i /path/my/ec2/keypair  <souce> <destination>
> > > > > > > > > > >
> > > > > > > > > > > Secondly in order to configure an ec2 cluster for
> Hadoop,
> > > you
> > > > > > need
> > > > > > > to
> > > > > > > > > > make
> > > > > > > > > > > a
> > > > > > > > > > > security group (let's call it hadoop) and give hadoop
> > > access
> > > > to
> > > > > > > > hadoop.
> > > > > > > > > > >
> > > > > > > > > > > You can read more about security groups here -
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://somic.org/2009/09/21/security-groups-most-underappreciated-feature=
-of-amazon-ec2/
> > > > > > > > > > >
> > > > > > > > > > > You also need to make sure that you can ssh from all
> the
> > > > slaves
> > > > > > to
> > > > > > > > > master
> > > > > > > > > > > without specifying password.
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > Vaibhav Puranik
> > > > > > > > > > > GumGum
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Nov 22, 2009 at 8:36 AM, Something Something =
<
> > > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Seems like I am not explaining my problem correctly=
.
> > > > > > > > > > > >
> > > > > > > > > > > > My Key Pair file is on my machine at work which is
> > behind
> > > a
> > > > > > corp
> > > > > > > > > > > firewall.
> > > > > > > > > > > >  As such I can't 'scp' from the ec2 instance to my
> > local
> > > > > > machine
> > > > > > > at
> > > > > > > > > > work
> > > > > > > > > > > to
> > > > > > > > > > > > 'get' the file.  So I need a way to 'send' a file
> from
> > my
> > > > > > machine
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > ec2
> > > > > > > > > > > > instance.  I tried using 'scp' (from my machine at
> > work)
> > > to
> > > > > the
> > > > > > > ec2
> > > > > > > > > > > machine
> > > > > > > > > > > > but it says "Permission denied".  Does this make
> sense?
> > > > > > > > > > > >
> > > > > > > > > > > > May be I need to use the command line tools for EC2=
.
>  I
> > > am
> > > > > > > looking
> > > > > > > > > into
> > > > > > > > > > > > those right now, but if there's a better/easier way=
,
> > > please
> > > > > let
> > > > > > > me
> > > > > > > > > > know.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks once again for your help.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <
> > > > > zjffdu@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > The ssh is installed on ec2 by default, otherwise
> you
> > > > have
> > > > > no
> > > > > > > way
> > > > > > > > > to
> > > > > > > > > > > > login
> > > > > > > > > > > > > to ec2
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Jeff Zhang
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sun, Nov 22, 2009 at 8:09 AM, Something
> Something
> > <
> > > > > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > As per Wikipedia (
> > > > > http://en.wikipedia.org/wiki/Secure_copy
> > > > > > )
> > > > > > > >  "It
> > > > > > > > > > > does
> > > > > > > > > > > > so
> > > > > > > > > > > > > > by
> > > > > > > > > > > > > > connecting to the host using SSH and there
> executes
> > > an
> > > > > SCP
> > > > > > > > server
> > > > > > > > > > > > (scp)".
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > So if SSH isn't working SCP won't work, either.
>  In
> > > any
> > > > > > case
> > > > > > > I
> > > > > > > > > > tried
> > > > > > > > > > > to
> > > > > > > > > > > > > scp
> > > > > > > > > > > > > > but getting "Permission denied (public key)".
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Any other ideas?  Thanks.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <
> > > > > > > zjffdu@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > You should scp the key-pair to EC2 machine
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Jeff Zhang
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Sat, Nov 21, 2009 at 4:57 PM, Something
> > > Something
> > > > <
> > > > > > > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Trying to start a Hadoop cluster on EC2.
>  (Yes,
> > > > > > > Cloudera's
> > > > > > > > > > > > > distribution
> > > > > > > > > > > > > > > > works well, but trying to do this myself so=
 I
> > can
> > > > > learn
> > > > > > > > > what's
> > > > > > > > > > > > > > happening
> > > > > > > > > > > > > > > > behind the scene.)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I have a Master & a Slave.  When I start HD=
FS
> > on
> > > > the
> > > > > > > > master,
> > > > > > > > > I
> > > > > > > > > > > get
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > > message
> > > > > > > > > > > > > > > > saying "10.xxx.xxx.xxx (Permission denied)"=
 -
> > > where
> > > > > > > 10.xxx
> > > > > > > > is
> > > > > > > > > > IP
> > > > > > > > > > > > > > address
> > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > the slave.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Basic problem (I think) is that I can't ssh
> > from
> > > > the
> > > > > > > master
> > > > > > > > > EC2
> > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > the Slave EC2 instance.  What's the best wa=
y
> to
> > > fix
> > > > > it?
> > > > > > >  I
> > > > > > > > > > think
> > > > > > > > > > > I
> > > > > > > > > > > > > need
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > "Key Pair" file on my master.  I have a key
> > pair
> > > on
> > > > > my
> > > > > > > > local
> > > > > > > > > > > > machine,
> > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > how do I transfer it to the EC2 machine?  (=
I
> > > know,
> > > > I
> > > > > > > know,
> > > > > > > > I
> > > > > > > > > > > > agree..
> > > > > > > > > > > > > I
> > > > > > > > > > > > > > am
> > > > > > > > > > > > > > > > dumb :)  Should I FTP it?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Please help.  Thanks.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

--0016e6d9a3a2f18ee6047991ea68--

From general-return-803-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 30 09:36:22 2009
Return-Path: <general-return-803-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73397 invoked from network); 30 Nov 2009 09:36:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Nov 2009 09:36:22 -0000
Received: (qmail 87222 invoked by uid 500); 30 Nov 2009 09:36:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87118 invoked by uid 500); 30 Nov 2009 09:36:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87108 invoked by uid 99); 30 Nov 2009 09:36:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Nov 2009 09:36:20 +0000
X-ASF-Spam-Status: No, hits=-6.6 required=5.0
	tests=AWL,BAYES_00,RCVD_IN_DNSWL_MED
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of huan.liu@accenture.com designates 170.252.248.72 as permitted sender)
Received: from [170.252.248.72] (HELO amrmr1003.accenture.com) (170.252.248.72)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Nov 2009 09:36:17 +0000
Received: from AMRXV1003.dir.svc.accenture.com (amrxv1003.dir.svc.accenture.com [10.10.160.63])
	by amrmr1003.accenture.com (8.13.8/8.13.8) with ESMTP id nAU9ZkwY002063
	for <general@hadoop.apache.org>; Mon, 30 Nov 2009 03:35:55 -0600 (CST)
Received: from AMRXH3002.dir.svc.accenture.com ([10.63.34.24]) by AMRXV1003.dir.svc.accenture.com with Microsoft SMTPSVC(6.0.3790.1830); Mon, 30 Nov 2009 03:35:53 -0600
Received: from AMRXM3124.dir.svc.accenture.com ([10.63.34.14]) by AMRXH3002.dir.svc.accenture.com ([10.63.34.24]) with mapi; Mon, 30 Nov 2009 04:35:52 -0500
From: <huan.liu@accenture.com>
To: <general@hadoop.apache.org>
Content-Class: urn:content-classes:message
Importance: normal
Priority: normal
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.3790.3168
Date: Mon, 30 Nov 2009 04:35:20 -0500
Subject: RE: Introducing Cloud MapReduce
Thread-Topic: Introducing Cloud MapReduce
thread-index: Acpvp0TKSUz6VIBZSEyA06CquAPzPwB961SQ
Message-ID: <BD19120596A4D041940290E71BD0097E62E4411DD9@AMRXM3124.dir.svc.accenture.com>
References: <7b3355cb0911270941s219c492w63d8ce6c83eeb96b@mail.gmail.com> <32120a6a0911271034i150078b6s8d0b23b882c6079@mail.gmail.com> <7b3355cb0911271318u4797fc2br51a241bd0fa95465@mail.gmail.com>
In-Reply-To: <7b3355cb0911271318u4797fc2br51a241bd0fa95465@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
x-ems-proccessed: vrAiQuOOcsXVFhS7ec6D4A==
x-ems-stamp: etMPdlPBeLL1MrCNXtYgQA==
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginalArrivalTime: 30 Nov 2009 09:35:53.0451 (UTC) FILETIME=[82AC33B0:01CA71A0]

> From: Bruce Snyder [mailto:bruce.snyder@gmail.com]
> Sent: Friday, November 27, 2009 1:19 PM
> To: general@hadoop.apache.org
> Subject: Re: Introducing Cloud MapReduce
>=20
> On Fri, Nov 27, 2009 at 11:34 AM, Tim Robertson
> <timrobertson100@gmail.com> wrote:
> > Hi Bruce,
> >
> > Interesting stuff.  It looks like it only works with String Keys and
> > Values (possibly a reason you see large performance gains with
> simpler
> > serialization requirements) - have you any plans to support other
> > types in the roadmap? Perhaps with Protobufs or Avro serialization?
>=20
> I can definitely see the need for supporting more than one type of
> key/value. We'll need to add this to the roadmap, thanks.

Added to the roadmap. Thanks for the suggestion.
=20
> > Have you considered using the same Mapper and Reducer class and
> method
> > signatures so that a MR job could be written and launched against
> > CloudMR or Hadoop?
> > [The reason I ask is I am hacking an implementation of a single JVM,
> > multithreaded MR framework that does this, so I can ship the same
> > analysis I run on large datasets to people to run on small datasets
> > also (much lighter than Hadoop but compatible).  I am putting it on
> > google code mapreduce4j.  It might be nice if there was a standard =
MR
> > API that people coded against and launched in various frameworks -
> > just a thought].
>=20
> Excellent points, Tim. A point that I raised was trying to determine a
> standard sort of API that would work across implementations. So far
> I'm not sure if this means that we match what is already out there or
> if we provide interfaces for many languages or what. I like the idea,
> but I'll let Huan chime in here.

The eventual goal is to be able to run a Hadoop job as it is in Cloud =
MapReduce, so that one can easily port his/her application to a =
different platform, but we are some distance away from that. To be fully =
compatible, it requires us to implement/fake a lot of interfaces in =
Hadoop, such as the different input readers and the configuration =
mechanism (many parameters are not needed in CMR). We will look into the =
mapreduce4j implementation to see what we can leverage. Thanks for the =
excellent suggestion.

-Huan



This message is for the designated recipient only and may contain =
privileged, proprietary, or otherwise private information.  If you have =
received it in error, please notify the sender immediately and delete =
the original.  Any other use of the email by you is prohibited.

From general-return-804-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 30 09:49:28 2009
Return-Path: <general-return-804-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81150 invoked from network); 30 Nov 2009 09:49:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Nov 2009 09:49:27 -0000
Received: (qmail 18378 invoked by uid 500); 30 Nov 2009 09:49:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18278 invoked by uid 500); 30 Nov 2009 09:49:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18158 invoked by uid 99); 30 Nov 2009 09:49:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Nov 2009 09:49:25 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of huan.liu@accenture.com designates 170.252.248.72 as permitted sender)
Received: from [170.252.248.72] (HELO amrmr1003.accenture.com) (170.252.248.72)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Nov 2009 09:49:16 +0000
Received: from AMRXV1004.dir.svc.accenture.com (amrxv1004.dir.svc.accenture.com [10.10.160.64])
	by amrmr1003.accenture.com (8.13.8/8.13.8) with ESMTP id nAU9mZYP013289
	for <general@hadoop.apache.org>; Mon, 30 Nov 2009 03:48:54 -0600 (CST)
Received: from AMRXH3003.dir.svc.accenture.com ([10.63.34.25]) by AMRXV1004.dir.svc.accenture.com with Microsoft SMTPSVC(6.0.3790.1830); Mon, 30 Nov 2009 03:48:51 -0600
Received: from AMRXM3124.dir.svc.accenture.com ([10.63.34.14]) by AMRXH3003.dir.svc.accenture.com ([10.63.34.25]) with mapi; Mon, 30 Nov 2009 04:48:51 -0500
From: <huan.liu@accenture.com>
Content-Class: urn:content-classes:message
Importance: normal
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.3790.3168
Priority: normal
To: <general@hadoop.apache.org>
Date: Mon, 30 Nov 2009 04:48:18 -0500
Subject: RE: what is the major difference between Hadoop and cloudMapReduce?
Thread-Topic: what is the major difference between Hadoop and cloudMapReduce?
thread-index: AcpxH/2sedDYMy4sQW68WFU69/WGTgAgH1gA
Message-ID: <BD19120596A4D041940290E71BD0097E62E4411DDC@AMRXM3124.dir.svc.accenture.com>
References: <c28cbc140911281944j4925292chfc8f87ce1e4c3c28@mail.gmail.com> 	<c28cbc140911281955j162fcf10w3aade804544a0938@mail.gmail.com> 	<7b3355cb0911282036x394f0acft21da05803b16a01@mail.gmail.com> 	<32120a6a0911290213m5563e1fge8bcc4413909a505@mail.gmail.com> 	<7b3355cb0911290816h1dc160e4ua30054c438702215@mail.gmail.com> <45f85f70911291015m5165421fsf75f28bc698f28ed@mail.gmail.com>
In-Reply-To: <45f85f70911291015m5165421fsf75f28bc698f28ed@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-cr-puzzleid: {F495B747-5C01-4E04-B23E-3471B59521E1}
x-cr-hashedpuzzle: Wjo= AnVH BDJm BkiM B/BN ERCE Egep EsIr FOv1 GYTO HQwU Hffy Hv7b JJzA JNvh KQdv;1;ZwBlAG4AZQByAGEAbABAAGgAYQBkAG8AbwBwAC4AYQBwAGEAYwBoAGUALgBvAHIAZwA=;Sosha1_v1;7;{F495B747-5C01-4E04-B23E-3471B59521E1};aAB1AGEAbgAuAGwAaQB1AEAAYQBjAGMAZQBuAHQAdQByAGUALgBjAG8AbQA=;Mon, 30 Nov 2009 09:48:18 GMT;UgBFADoAIAB3AGgAYQB0ACAAaQBzACAAdABoAGUAIABtAGEAagBvAHIAIABkAGkAZgBmAGUAcgBlAG4AYwBlACAAYgBlAHQAdwBlAGUAbgAgAEgAYQBkAG8AbwBwACAAYQBuAGQAIABjAGwAbwB1AGQATQBhAHAAUgBlAGQAdQBjAGUAPwA=
acceptlanguage: en-US
x-ems-proccessed: vrAiQuOOcsXVFhS7ec6D4A==
x-ems-stamp: 5AoCVe+HDnU0sosatIXf7g==
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginalArrivalTime: 30 Nov 2009 09:48:51.0962 (UTC) FILETIME=[52B38DA0:01CA71A2]
X-Virus-Checked: Checked by ClamAV on apache.org

Todd,=20

We do not keep all values for a key in memory. Instead, we only keep the =
partial reduce result in memory, but throw away the value as soon as it =
is used. The point you raised is still very valid if the reduce state =
maintained per key is large, which I hope is a very rare case. If you =
have some concrete workload examples, it will help us prioritize the =
development effort. I can definitely see the benefits of introducing a =
paging mechanism to spill partial reduce results to the output SQS queue =
in the future. Thanks.

-Huan


> -----Original Message-----
> From: Todd Lipcon [mailto:todd@cloudera.com]
> Sent: Sunday, November 29, 2009 10:15 AM
> To: general@hadoop.apache.org
> Subject: Re: what is the major difference between Hadoop and
> cloudMapReduce?
>=20
> Another difference is that Cloud Mapreduce doesn't scale well when
> there are
> a large number of values for the same key. They collect in a single =
SQS
> queue per partition, and in the reducer reads that into a hashtable
> that
> never spills to disk. So, if you have more values for a key than fit =
in
> RAM,
> you can't run a job. Hadoop on the other hand spills reduce input to
> disk
> when it's too large to sort in memory. The paper says to add more
> partitions
> if you have a lot of data, but it doesn't help in the case of skewed
> reduce
> input (very common in a lot of workloads)
>=20
> -Todd
>=20
> On Sun, Nov 29, 2009 at 8:16 AM, Bruce Snyder
> <bruce.snyder@gmail.com>wrote:
>=20
> > On Sun, Nov 29, 2009 at 3:13 AM, Tim Robertson
> > <timrobertson100@gmail.com> wrote:
> > > Hi Tom,
> > >
> > > One major difference is that Hadoop allows working with any object
> > > that you can serialize, rather than only Strings so it is
> applicable
> > > for use with (e.g) image processing, or rendering to images or =
PDFs
> on
> > > the output format.
> >
> > Yes, you noted this on the general@hadoop list. I think your
> > suggestion to expand this to support other types is a good one.
> >
> > Bruce
> > --
> > perl -e 'print
> > =
unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=3D6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> > );'
> >
> > ActiveMQ in Action: http://bit.ly/2je6cQ
> > Blog: http://bruceblog.org/
> > Twitter: http://twitter.com/brucesnyder
> >


This message is for the designated recipient only and may contain =
privileged, proprietary, or otherwise private information.  If you have =
received it in error, please notify the sender immediately and delete =
the original.  Any other use of the email by you is prohibited.

From general-return-805-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 30 16:07:22 2009
Return-Path: <general-return-805-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27843 invoked from network); 30 Nov 2009 16:07:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Nov 2009 16:07:22 -0000
Received: (qmail 7096 invoked by uid 500); 30 Nov 2009 16:07:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7031 invoked by uid 500); 30 Nov 2009 16:07:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7020 invoked by uid 99); 30 Nov 2009 16:07:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Nov 2009 16:07:20 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of vpuranik@gmail.com designates 209.85.216.187 as permitted sender)
Received: from [209.85.216.187] (HELO mail-px0-f187.google.com) (209.85.216.187)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Nov 2009 16:07:07 +0000
Received: by pxi17 with SMTP id 17so4462962pxi.30
        for <general@hadoop.apache.org>; Mon, 30 Nov 2009 08:06:45 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=rsGRkhOb6RBeznZFerGBjUHiLkJ3RSutvS7v1vMIZBo=;
        b=lvy2jw7iOv/jNovjQ01iuypGYcLu9327Z3N/riyOf3gZ44OH7CwAttBLO1zc78jTbn
         8Pyi8j69e31fMaGJ5fh6GTEoGgTM8fOwZrQ8+TsYU+cHZobt8nSHMmb5jDkbOA5oz/V5
         Mcrw6RBv0wo01l0fJtwCk0Mi0Nxxb3Nc5g3Qo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=ox0aBZfmOx2/5Upu9iPxVK7dnY8mgUpVwTapW792AYll8COOM1DyViHHsj2s0l8WL9
         Plf4rB+hYgQ3On59PZwvMUf86AMchqFhjjCcILLW/XHQ/wfSGV03f9UAsdPpFiDkGM8U
         pefcY01C5Hp29NPOkzdm2pW2uqAutlpqBtRNc=
MIME-Version: 1.0
Received: by 10.114.55.34 with SMTP id d34mr7939920waa.225.1259597205307; Mon, 
	30 Nov 2009 08:06:45 -0800 (PST)
In-Reply-To: <1eabbac30911292352k4f912452j52cab4b532460f05@mail.gmail.com>
References: <1eabbac30911211657i7812c67bs3a1565b0537b418f@mail.gmail.com>
	 <1eabbac30911241338g1067b99dra633f21f2d0cfd10@mail.gmail.com>
	 <2eef7bd70911241344p4225190fkaaafc160a5c9bb80@mail.gmail.com>
	 <1eabbac30911241406v1a684661va156ab6a856819c3@mail.gmail.com>
	 <2eef7bd70911241458x6e2de4dfx60ef8e68b636f78e@mail.gmail.com>
	 <1eabbac30911250838w3d4b089ascb2e0b21245a60ae@mail.gmail.com>
	 <2eef7bd70911250921p5d2558fq8766748c47dbfb3f@mail.gmail.com>
	 <1eabbac30911250945i49684a5am6510cc6ffb7f86c8@mail.gmail.com>
	 <2eef7bd70911251025s73655a79ned5f6418caba81fb@mail.gmail.com>
	 <1eabbac30911292352k4f912452j52cab4b532460f05@mail.gmail.com>
Date: Mon, 30 Nov 2009 08:06:45 -0800
Message-ID: <2eef7bd70911300806n274a3819oe06cd2b3d488fc0e@mail.gmail.com>
Subject: Re: Starting Hadoop cluster on EC2
From: Vaibhav Puranik <vpuranik@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016369204c7fb8356047998d3da
X-Virus-Checked: Checked by ClamAV on apache.org

--0016369204c7fb8356047998d3da
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Thanks. I didn't know that you could do that.

Regards,
Vaibhav

On Sun, Nov 29, 2009 at 11:52 PM, Something Something <
mailinglists19@gmail.com> wrote:

> Everything started working after I added the following line to
> hadoop-env.sh
>
> export HADOOP_SSH_OPTS=3D"$HADOOP_SSH_OPTS -i /mnt/myVol/KeyPairName.pem =
"
>
> Thanks for your help.
>
>
> On Wed, Nov 25, 2009 at 10:25 AM, Vaibhav Puranik <vpuranik@gmail.com
> >wrote:
>
> > On master, in .ssh folder, there should be a file id_dsa.pub. This is
> > master's public key.
> >
> > Contents of this file should be copied to slave's authorized_keys file.
> >
> > Regards,
> > Vaibhav
> >
> > On Wed, Nov 25, 2009 at 9:45 AM, Something Something <
> > mailinglists19@gmail.com> wrote:
> >
> > > Can you please explain what you mean by "add master's public key to
> > slave's
> > > authorized_keys"?  I have a feeling I am not doing this correctly.
>  What
> > I
> > > did is this:
> > >
> > > 1)  On Master, opened ~/.ssh/authorized_key and copied lines that loo=
k
> > like
> > > this...
> > >
> > > ssh-dss
> > >
> > >
> >
> AAAAB3NzaC1kc3MAAACBAKJOife8p+6vjJ/jODSWaBgyawN7XWmasasasb0cVqhe1WOpnJQpX=
uJeqsaBaZYHYxj8P74m1rOIdN9lSPAz6gEjHo4ft8UM7ZVExRC6HDM7JdfdferhwRgQSVI9ruTL=
lssO9i1D17mBnLotsgy92CyS7bsanM25nRqhmvqkX7tuKp66nlAAAAFQCnbwPn3v+ttorRZ4Oij=
kAKoJaVFQAAAIAGVyZGwkm5DwYvige0VYhXZns8qSvW15dwSrS1a6k41q3C91AQZylvJYDnBjNG=
nWwUHfyWpyruzVCOskavEmsl6FcouU5Av1zLRT1sk4llUzZxXFtv6gzRBq2+aiCcoex3O+RRJbH=
a89cMLy6jeZOCFahwVz6IyHcQ0b7gZjQgFQAAAIBA4vzSDy5oN27Ji3mF9ZklsSmlJJrrJxGJfO=
n6/BK5tyvBijLy6tEJHrH3dY8TEJUp2V9RINKwcsy5LdqyI5gJRWoM8WdNzmUoIOLAW0HIVl2Ml=
dKKqP/ctQwGI94EjykSUYVt/IAvUcjFuIrtiGrMDjsqfm7h1Gi2ZtDQ=3D=3D
> > > root@domU-12-31-38-00-39-C8
> > >
> > >
> > > to the end of ~/.ssh/authorized_key on Slave.
> > >
> > > Something tells me there's a better way to do this.  Is there?
> > >
> > >
> > > On Wed, Nov 25, 2009 at 9:21 AM, Vaibhav Puranik <vpuranik@gmail.com>
> > > wrote:
> > >
> > > > Actually I was wrong about adding keypair. It should be resolved by
> > > adding
> > > > master's public key to slave's authorized_keys. Amazon puts your
> > keypair
> > > in
> > > > authroized_keys automatically. I missed that part.
> > > >
> > > > Regards,
> > > > Vaibhav
> > > >
> > > > On Wed, Nov 25, 2009 at 8:38 AM, Something Something <
> > > > mailinglists19@gmail.com> wrote:
> > > >
> > > > > Me again :(
> > > > >
> > > > > I decided to start fresh so that I can document the steps, but no=
w
> I
> > > > can't
> > > > > even go further than I did before :(
> > > > >
> > > > > Please let me know what step I am missing:
> > > > >
> > > > > 1)  Launched 2 instances on AWS console - one for Master, one for
> > > Slave.
> > > > >
> > > > > 2)  In one command window, connected to instance 1:
> > > > > ssh -i MyKeyPair.pem <instance 1>
> > > > > (Got 'authenticity' warning.. typed 'yes')
> > > > >
> > > > > This is my "Master" window.
> > > > >
> > > > > 3)  Opened another command window & connected to instance 2:
> > > > > ssh -i MyKeyPair.pem <instance 2>
> > > > > (Got 'authenticity' warning.. typed 'yes')
> > > > >
> > > > > This is my "Slave" window.
> > > > >
> > > > > 4) Opened another command window and 'scp' the key pair (from my
> > local
> > > > > machine) to instance 1 & instance 2.
> > > > > scp -i MyKeyPair.pem MyKeyPair.pem <master instance>:/tmp
> > > > > scp -i MyKeyPair.pem MyKeyPair.pem <slave instance>:/tmp
> > > > >
> > > > > 5)  On BOTH (Master & Slave) added Key Pair to 'authorized_key'
> > > > >
> > > > > cat /tmp/MyKeyPair.pem >> ~/.ssh/authorized_keys
> > > > >
> > > > > 6)  To setup passphraseless ssh ran these on BOTH - Master & Slav=
e:
> > > > > ssh localhost  (Got 'authenticiy' message.  Typed 'yes'.  Got
> > > Permission
> > > > > denied (public key)
> > > > > ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
> > > > > cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
> > > > >
> > > > > 7)  In order to ssh from Master to Slave (without entering
> KeyPair),
> > > > copied
> > > > > the key (that starts with 'sh-dss') from Slave's 'authorized_key'
> > file
> > > to
> > > > > Master's 'authoried_key' file.
> > > > >
> > > > > Now, every time I try (from Master):
> > > > >
> > > > > ssh <slave's ip address>
> > > > >
> > > > > I get - "Permission denied (publickey).
> > > > >
> > > > > But, when I type:
> > > > >
> > > > > ssh -i /tmp/MyKeyPair.pem <slave's ip address>
> > > > >
> > > > > It works.
> > > > >
> > > > > I thought step #5 above should have resolved this.  What am I doi=
ng
> > > > wrong?
> > > > > I believe, in order for me to start the cluster, I should be able
> to
> > > > 'ssh'
> > > > > to slave without using key pair, correct?
> > > > >
> > > > >
> > > > > Thanks for your help.
> > > > >
> > > > >
> > > > > On Tue, Nov 24, 2009 at 2:58 PM, Vaibhav Puranik <
> vpuranik@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Install a program called Jps.
> > > > > >
> > > > > > It's a ps for java processes. execute jps, you should see the
> > > processes
> > > > > (if
> > > > > > they are running).
> > > > > >
> > > > > > Also make sure you check your logs.
> > > > > >
> > > > > > Regards,
> > > > > > Vaibhav
> > > > > >
> > > > > > On Tue, Nov 24, 2009 at 2:06 PM, Something Something <
> > > > > > mailinglists19@gmail.com> wrote:
> > > > > >
> > > > > > > Awesome.  Thanks, Vaibhav.  Made progress.  No error messages
> > from
> > > > > > > ./all-start.sh
> > > > > > >
> > > > > > > I will test to see if the cluster is deployed properly.
> > > > > > >
> > > > > > > Question:  On the slave, when I do:
> > > > > > >
> > > > > > > ps -eaf | grep 'hadoop'
> > > > > > >
> > > > > > > I don't see any processes.  Shouldn't I see 2 processes - one
> for
> > > > > > Datanode
> > > > > > > and the other for TaskTracker?
> > > > > > >
> > > > > > > Please let me know.  Thanks again.
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Nov 24, 2009 at 1:44 PM, Vaibhav Puranik <
> > > vpuranik@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Yes, I pretty much meant that.
> > > > > > > >
> > > > > > > > Add your master's public key to authorized_keys on slaves
> too.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Vaibhav
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Nov 24, 2009 at 1:38 PM, Something Something <
> > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > Sorry.. but not sure what you mean by... "In order to
> achieve
> > > > that
> > > > > > you
> > > > > > > > need
> > > > > > > > > to add the keypair  you are using to your
> > > > > > > > > auhtorized_keys file in the home dir/.ssh folder."
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I tried this....
> > > > > > > > >
> > > > > > > > > cat KeyPair.pem >> /root/.ssh/authorized_keys
> > > > > > > > >
> > > > > > > > > Is that what you meant by "add the keypair to
> > authorized_keys"?
> > > > > > > > >
> > > > > > > > > Still can't ssh from Slave to Master.  It says.....
> >  Permission
> > > > > > denied
> > > > > > > > > (publickey).
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Sorry, I am a newbie when it comes to such environment
> > issues.
> > > > > >  Thanks
> > > > > > > > for
> > > > > > > > > your patience.  It's greatly appreciated.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Nov 24, 2009 at 11:16 AM, Vaibhav Puranik <
> > > > > > vpuranik@gmail.com
> > > > > > > > > >wrote:
> > > > > > > > >
> > > > > > > > > > You need to be able to SSH without specifying the keypa=
ir
> > > from
> > > > a
> > > > > > > slave
> > > > > > > > to
> > > > > > > > > > master (I think reverse it true too, but I am not sure)=
.
> > > > > > > > > >
> > > > > > > > > > In order to achieve that you need to add the keypair  y=
ou
> > are
> > > > > using
> > > > > > > to
> > > > > > > > > your
> > > > > > > > > > auhtorized_keys file in the home dir/.ssh folder.
> > > > > > > > > >
> > > > > > > > > > Once you setup passphraseless ssh, it should start
> working.
> > > > > > > Furthrmore,
> > > > > > > > > > after you set this up, try to ssh manully (without key)
> and
> > > if
> > > > it
> > > > > > > asks
> > > > > > > > > you
> > > > > > > > > > the following message - say yes:
> > > > > > > > > >
> > > > > > > > > > The authenticity of host
> > > > > > > > > > 'ec2-147-127-186-243.compute-1.amazonaws.com
> > > (147.127.186.243)'
> > > > > > can't
> > > > > > > > > > be established.
> > > > > > > > > > RSA key fingerprint is
> > > > > > > 4b:63:e2:23:16:6f:b6:99:de:34:f6:9b:f5:55:73:8b.
> > > > > > > > > > Are you sure you want to continue connecting (yes/no)?
> yes
> > > > > > > > > > Warning: Permanently added
> > > > > > > > > > 'ec2-147-127-186-243.compute-1.amazonaws.com
> > > ,147.127.186.243'
> > > > > > > > > > (RSA) to the list of known hosts.
> > > > > > > > > >
> > > > > > > > > > In other words, you have to make sure that the machine
> you
> > > are
> > > > > > trying
> > > > > > > > to
> > > > > > > > > > ssh
> > > > > > > > > > is in your known_hosts file.
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > Vaibhav
> > > > > > > > > >
> > > > > > > > > > On Tue, Nov 24, 2009 at 10:54 AM, Something Something <
> > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks, Vaibhav.  That was helpful.  I am able to scp
> > from
> > > my
> > > > > > local
> > > > > > > > > > machine
> > > > > > > > > > > after I opened port 22 under my security group.
> > > > > > > > > > >
> > > > > > > > > > > But my cluster is still not starting as expected.  I
> > > mean...
> > > > on
> > > > > > my
> > > > > > > > > Master
> > > > > > > > > > > when I run:
> > > > > > > > > > >
> > > > > > > > > > > ./start-all.sh
> > > > > > > > > > >
> > > > > > > > > > > I get messages saying...
> > > > > > > > > > >
> > > > > > > > > > > 10.252.xxx.xxx: Permission denied (publickey).
> > > > > > > > > > >
> > > > > > > > > > > where 10.252.xxx.xxx is the ip address of my Slave.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I can ssh from my Slave to master if I do this:
> > > > > > > > > > >
> > > > > > > > > > > ssh -i <keypairname>.pem <Master's ip address>
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Any suggestions?  Thanks for your help.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Nov 23, 2009 at 8:10 AM, Vaibhav Puranik <
> > > > > > > vpuranik@gmail.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > In order to use scp from your machine to an ec2
> > instance
> > > > you
> > > > > > need
> > > > > > > > to
> > > > > > > > > > use
> > > > > > > > > > > > scp
> > > > > > > > > > > > -i and give it the path of your keypair.
> > > > > > > > > > > >
> > > > > > > > > > > > scp -i /path/my/ec2/keypair  <souce> <destination>
> > > > > > > > > > > >
> > > > > > > > > > > > Secondly in order to configure an ec2 cluster for
> > Hadoop,
> > > > you
> > > > > > > need
> > > > > > > > to
> > > > > > > > > > > make
> > > > > > > > > > > > a
> > > > > > > > > > > > security group (let's call it hadoop) and give hado=
op
> > > > access
> > > > > to
> > > > > > > > > hadoop.
> > > > > > > > > > > >
> > > > > > > > > > > > You can read more about security groups here -
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://somic.org/2009/09/21/security-groups-most-underappreciated-feature=
-of-amazon-ec2/
> > > > > > > > > > > >
> > > > > > > > > > > > You also need to make sure that you can ssh from al=
l
> > the
> > > > > slaves
> > > > > > > to
> > > > > > > > > > master
> > > > > > > > > > > > without specifying password.
> > > > > > > > > > > >
> > > > > > > > > > > > Regards,
> > > > > > > > > > > > Vaibhav Puranik
> > > > > > > > > > > > GumGum
> > > > > > > > > > > >
> > > > > > > > > > > > On Sun, Nov 22, 2009 at 8:36 AM, Something Somethin=
g
> <
> > > > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Seems like I am not explaining my problem
> correctly.
> > > > > > > > > > > > >
> > > > > > > > > > > > > My Key Pair file is on my machine at work which i=
s
> > > behind
> > > > a
> > > > > > > corp
> > > > > > > > > > > > firewall.
> > > > > > > > > > > > >  As such I can't 'scp' from the ec2 instance to m=
y
> > > local
> > > > > > > machine
> > > > > > > > at
> > > > > > > > > > > work
> > > > > > > > > > > > to
> > > > > > > > > > > > > 'get' the file.  So I need a way to 'send' a file
> > from
> > > my
> > > > > > > machine
> > > > > > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > > > ec2
> > > > > > > > > > > > > instance.  I tried using 'scp' (from my machine a=
t
> > > work)
> > > > to
> > > > > > the
> > > > > > > > ec2
> > > > > > > > > > > > machine
> > > > > > > > > > > > > but it says "Permission denied".  Does this make
> > sense?
> > > > > > > > > > > > >
> > > > > > > > > > > > > May be I need to use the command line tools for
> EC2.
> >  I
> > > > am
> > > > > > > > looking
> > > > > > > > > > into
> > > > > > > > > > > > > those right now, but if there's a better/easier
> way,
> > > > please
> > > > > > let
> > > > > > > > me
> > > > > > > > > > > know.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks once again for your help.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang <
> > > > > > zjffdu@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > The ssh is installed on ec2 by default, otherwi=
se
> > you
> > > > > have
> > > > > > no
> > > > > > > > way
> > > > > > > > > > to
> > > > > > > > > > > > > login
> > > > > > > > > > > > > > to ec2
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Jeff Zhang
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sun, Nov 22, 2009 at 8:09 AM, Something
> > Something
> > > <
> > > > > > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > As per Wikipedia (
> > > > > > http://en.wikipedia.org/wiki/Secure_copy
> > > > > > > )
> > > > > > > > >  "It
> > > > > > > > > > > > does
> > > > > > > > > > > > > so
> > > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > connecting to the host using SSH and there
> > executes
> > > > an
> > > > > > SCP
> > > > > > > > > server
> > > > > > > > > > > > > (scp)".
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > So if SSH isn't working SCP won't work, eithe=
r.
> >  In
> > > > any
> > > > > > > case
> > > > > > > > I
> > > > > > > > > > > tried
> > > > > > > > > > > > to
> > > > > > > > > > > > > > scp
> > > > > > > > > > > > > > > but getting "Permission denied (public key)".
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Any other ideas?  Thanks.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang <
> > > > > > > > zjffdu@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > You should scp the key-pair to EC2 machine
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Jeff Zhang
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Sat, Nov 21, 2009 at 4:57 PM, Something
> > > > Something
> > > > > <
> > > > > > > > > > > > > > > > mailinglists19@gmail.com> wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Trying to start a Hadoop cluster on EC2.
> >  (Yes,
> > > > > > > > Cloudera's
> > > > > > > > > > > > > > distribution
> > > > > > > > > > > > > > > > > works well, but trying to do this myself =
so
> I
> > > can
> > > > > > learn
> > > > > > > > > > what's
> > > > > > > > > > > > > > > happening
> > > > > > > > > > > > > > > > > behind the scene.)
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I have a Master & a Slave.  When I start
> HDFS
> > > on
> > > > > the
> > > > > > > > > master,
> > > > > > > > > > I
> > > > > > > > > > > > get
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > message
> > > > > > > > > > > > > > > > > saying "10.xxx.xxx.xxx (Permission denied=
)"
> -
> > > > where
> > > > > > > > 10.xxx
> > > > > > > > > is
> > > > > > > > > > > IP
> > > > > > > > > > > > > > > address
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > the slave.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Basic problem (I think) is that I can't s=
sh
> > > from
> > > > > the
> > > > > > > > master
> > > > > > > > > > EC2
> > > > > > > > > > > > > > > instance
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > the Slave EC2 instance.  What's the best
> way
> > to
> > > > fix
> > > > > > it?
> > > > > > > >  I
> > > > > > > > > > > think
> > > > > > > > > > > > I
> > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > "Key Pair" file on my master.  I have a k=
ey
> > > pair
> > > > on
> > > > > > my
> > > > > > > > > local
> > > > > > > > > > > > > machine,
> > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > how do I transfer it to the EC2 machine?
>  (I
> > > > know,
> > > > > I
> > > > > > > > know,
> > > > > > > > > I
> > > > > > > > > > > > > agree..
> > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > am
> > > > > > > > > > > > > > > > > dumb :)  Should I FTP it?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Please help.  Thanks.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

--0016369204c7fb8356047998d3da--

From general-return-806-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 30 16:16:12 2009
Return-Path: <general-return-806-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37029 invoked from network); 30 Nov 2009 16:16:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Nov 2009 16:16:12 -0000
Received: (qmail 25241 invoked by uid 500); 30 Nov 2009 16:16:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25160 invoked by uid 500); 30 Nov 2009 16:16:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25150 invoked by uid 99); 30 Nov 2009 16:16:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Nov 2009 16:16:10 +0000
X-ASF-Spam-Status: No, hits=-2.5 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.85.216.187] (HELO mail-px0-f187.google.com) (209.85.216.187)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Nov 2009 16:16:08 +0000
Received: by pxi17 with SMTP id 17so4474330pxi.30
        for <general@hadoop.apache.org>; Mon, 30 Nov 2009 08:15:47 -0800 (PST)
MIME-Version: 1.0
Received: by 10.142.60.5 with SMTP id i5mr466924wfa.102.1259597747410; Mon, 30 
	Nov 2009 08:15:47 -0800 (PST)
In-Reply-To: <BD19120596A4D041940290E71BD0097E62E4411DDC@AMRXM3124.dir.svc.accenture.com>
References: <c28cbc140911281944j4925292chfc8f87ce1e4c3c28@mail.gmail.com> 
	<c28cbc140911281955j162fcf10w3aade804544a0938@mail.gmail.com> 
	<7b3355cb0911282036x394f0acft21da05803b16a01@mail.gmail.com> 
	<32120a6a0911290213m5563e1fge8bcc4413909a505@mail.gmail.com> 
	<7b3355cb0911290816h1dc160e4ua30054c438702215@mail.gmail.com> 
	<45f85f70911291015m5165421fsf75f28bc698f28ed@mail.gmail.com> 
	<BD19120596A4D041940290E71BD0097E62E4411DDC@AMRXM3124.dir.svc.accenture.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 30 Nov 2009 08:15:27 -0800
Message-ID: <45f85f70911300815q2e86c64r92e28654af38840d@mail.gmail.com>
Subject: Re: what is the major difference between Hadoop and cloudMapReduce?
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502ad3b4b57f8047998f466

--00504502ad3b4b57f8047998f466
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Nov 30, 2009 at 1:48 AM, <huan.liu@accenture.com> wrote:

> Todd,
>
> We do not keep all values for a key in memory. Instead, we only keep the
> partial reduce result in memory, but throw away the value as soon as it is
> used. The point you raised is still very valid if the reduce state
> maintained per key is large, which I hope is a very rare case. If you have
> some concrete workload examples, it will help us prioritize the development
> effort. I can definitely see the benefits of introducing a paging mechanism
> to spill partial reduce results to the output SQS queue in the future.
> Thanks.
>

Hi Huan,

I guess I misremembered or misread the paper.

Given this technique, doesn't it mean that reducers can only work when
commutative and associative?

-Todd


>
> > -----Original Message-----
> > From: Todd Lipcon [mailto:todd@cloudera.com]
> > Sent: Sunday, November 29, 2009 10:15 AM
> > To: general@hadoop.apache.org
> > Subject: Re: what is the major difference between Hadoop and
> > cloudMapReduce?
> >
> > Another difference is that Cloud Mapreduce doesn't scale well when
> > there are
> > a large number of values for the same key. They collect in a single SQS
> > queue per partition, and in the reducer reads that into a hashtable
> > that
> > never spills to disk. So, if you have more values for a key than fit in
> > RAM,
> > you can't run a job. Hadoop on the other hand spills reduce input to
> > disk
> > when it's too large to sort in memory. The paper says to add more
> > partitions
> > if you have a lot of data, but it doesn't help in the case of skewed
> > reduce
> > input (very common in a lot of workloads)
> >
> > -Todd
> >
> > On Sun, Nov 29, 2009 at 8:16 AM, Bruce Snyder
> > <bruce.snyder@gmail.com>wrote:
> >
> > > On Sun, Nov 29, 2009 at 3:13 AM, Tim Robertson
> > > <timrobertson100@gmail.com> wrote:
> > > > Hi Tom,
> > > >
> > > > One major difference is that Hadoop allows working with any object
> > > > that you can serialize, rather than only Strings so it is
> > applicable
> > > > for use with (e.g) image processing, or rendering to images or PDFs
> > on
> > > > the output format.
> > >
> > > Yes, you noted this on the general@hadoop list. I think your
> > > suggestion to expand this to support other types is a good one.
> > >
> > > Bruce
> > > --
> > > perl -e 'print
> > > unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> > > );'
> > >
> > > ActiveMQ in Action: http://bit.ly/2je6cQ
> > > Blog: http://bruceblog.org/
> > > Twitter: http://twitter.com/brucesnyder
> > >
>
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise private information.  If you have
> received it in error, please notify the sender immediately and delete the
> original.  Any other use of the email by you is prohibited.
>

--00504502ad3b4b57f8047998f466--

From general-return-807-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 30 19:55:05 2009
Return-Path: <general-return-807-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55846 invoked from network); 30 Nov 2009 19:55:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Nov 2009 19:55:05 -0000
Received: (qmail 48134 invoked by uid 500); 30 Nov 2009 19:55:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48079 invoked by uid 500); 30 Nov 2009 19:55:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48069 invoked by uid 99); 30 Nov 2009 19:55:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Nov 2009 19:55:03 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=HTML_MESSAGE,MIME_QP_LONG_LINE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of vogt7@llnl.gov designates 128.115.41.83 as permitted sender)
Received: from [128.115.41.83] (HELO smtp.llnl.gov) (128.115.41.83)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Nov 2009 19:54:54 +0000
X-Attachments: None
Received: from poisonivy.llnl.gov (HELO [128.115.210.109]) ([128.115.210.109])
  by smtp.llnl.gov with ESMTP; 30 Nov 2009 11:54:32 -0800
User-Agent: Microsoft-Entourage/12.23.0.091001
Date: Mon, 30 Nov 2009 11:54:31 -0800
Subject: Scribe vs. Chukwa
From: Kim Vogt <vogt7@llnl.gov>
To: <general@hadoop.apache.org>
Message-ID: <C73962F7.448B%vogt7@llnl.gov>
Thread-Topic: Scribe vs. Chukwa
Thread-Index: Acpx9u50+HtKxaEtHUuAzUfAGnnRsg==
Mime-version: 1.0
Content-type: multipart/alternative;
	boundary="B_3342426872_847723"
X-Virus-Checked: Checked by ClamAV on apache.org

--B_3342426872_847723
Content-type: text/plain;
	charset="ISO-8859-1"
Content-transfer-encoding: quoted-printable

Hi,

My team is looking into using Scribe or Chukwa for hadoop log collection.  =
I
was wondering if anyone had any opinions about one vs. the other?  I
apologize if this topic was covered before, but I don=B9t see a link to the
archives for this mailing list.

Thanks,

Kim

--B_3342426872_847723--



From general-return-808-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 30 21:57:22 2009
Return-Path: <general-return-808-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24137 invoked from network); 30 Nov 2009 21:57:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Nov 2009 21:57:22 -0000
Received: (qmail 61757 invoked by uid 500); 30 Nov 2009 21:57:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61678 invoked by uid 500); 30 Nov 2009 21:57:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61668 invoked by uid 99); 30 Nov 2009 21:57:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Nov 2009 21:57:20 +0000
X-ASF-Spam-Status: No, hits=3.0 required=10.0
	tests=MIME_QP_LONG_LINE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Nov 2009 21:57:09 +0000
Received: from SNV-EXPF01.ds.corp.yahoo.com (snv-expf01.ds.corp.yahoo.com [207.126.227.250])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id nAULuJuG088202
	for <general@hadoop.apache.org>; Mon, 30 Nov 2009 13:56:19 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:x-originalarrivaltime;
	b=uULhc4EAQ4jZu2BnyIftLZGaWcqgVDODOWmzVMfmJrDaV/mJGzczfRxx5iiDkT4N
Received: from SNV-EXVS06.ds.corp.yahoo.com ([207.126.227.234]) by SNV-EXPF01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Mon, 30 Nov 2009 13:56:19 -0800
Received: from 10.72.111.153 ([10.72.111.153]) by SNV-EXVS06.ds.corp.yahoo.com ([207.126.227.82]) via Exchange Front-End Server snv-webmail.corp.yahoo.com ([207.126.227.59]) with Microsoft Exchange Server HTTP-DAV ;
 Mon, 30 Nov 2009 21:55:58 +0000
User-Agent: Microsoft-Entourage/12.23.0.091001
Date: Mon, 30 Nov 2009 13:55:58 -0800
Subject: Re: Scribe vs. Chukwa
From: Eric Yang <eyang@yahoo-inc.com>
To: <general@hadoop.apache.org>
Message-ID: <C7397F6E.53EA%eyang@yahoo-inc.com>
Thread-Topic: Scribe vs. Chukwa
Thread-Index: Acpx9u50+HtKxaEtHUuAzUfAGnnRsgAEPdkX
In-Reply-To: <C73962F7.448B%vogt7@llnl.gov>
Mime-version: 1.0
Content-type: text/plain;
	charset="ISO-8859-1"
Content-transfer-encoding: quoted-printable
X-OriginalArrivalTime: 30 Nov 2009 21:56:19.0658 (UTC) FILETIME=[F2C16AA0:01CA7207]
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Kim,

Scribe works well for simple deployment.  The complexity increases when
"central scribe server" is multi-machines deployment.  Basically, it
requires a reverse proxy to load balance the data collection.  (
http://www.cloudera.com/blog/2008/11/02/configuring-and-using-scribe-for-ha=
d
oop-log-collection/ )  I have not used scribe personally, therefore someone
else could fill in the experience.

Chukwa was designed to be fault tolerant log collection/analytics platform.
Each chukwa agent automatically creates it's own routing table to chukwa
collectors.  Therefore, Chukwa does not require a reverse proxy.  However,
Chukwa Agent requires knowledge of all collector addresses, hence the
initial deployment complexity may be a little higher than scribe.  The
largest test for Chukwa deployment was 50 Chukwa collectors running on top
of 100 dedicated hadoop nodes to process log files from a data center.
(Which was decommissioned due to lack of log files)  Base on my experience,
a single collector with 8GB allocated RAM could handle all log files from
2000 hadoop nodes + System Metrics (top, df, sar, iostat, netstat, vmstat
output). =20

Chukwa does not have a direct log file viewer, instead, it has an analytics
engine which computes various facts and provide reports.  There are frequen=
t
requests about log file viewer but it hasn't been implemented.  We only hav=
e
command line utility to dump the log files because it is difficult to view
terabytes of log file.  At some point in the future, when a full body index
engine is implemented, then we will provide log file search.

In essence, it depends on what you are looking for.  If you are looking for
simple log collection and viewer, Scribe is probably a good tool.  If you
are looking for log collection and reporting platform, Chukwa is a good
solution.

Regards,
Eric

On 11/30/09 11:54 AM, "Kim Vogt" <vogt7@llnl.gov> wrote:

> Hi,
>=20
> My team is looking into using Scribe or Chukwa for hadoop log collection.=
  I
> was wondering if anyone had any opinions about one vs. the other?  I
> apologize if this topic was covered before, but I don=B9t see a link to the
> archives for this mailing list.
>=20
> Thanks,
>=20
> Kim


From general-return-2465-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 01 01:58:19 2010
Return-Path: <general-return-2465-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91363 invoked from network); 1 Dec 2010 01:58:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Dec 2010 01:58:18 -0000
Received: (qmail 46168 invoked by uid 500); 1 Dec 2010 01:58:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46088 invoked by uid 500); 1 Dec 2010 01:58:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46070 invoked by uid 99); 1 Dec 2010 01:58:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Dec 2010 01:58:16 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of shv.hadoop@gmail.com designates 209.85.213.48 as permitted sender)
Received: from [209.85.213.48] (HELO mail-yw0-f48.google.com) (209.85.213.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Dec 2010 01:58:12 +0000
Received: by ywo32 with SMTP id 32so121569ywo.35
        for <general@hadoop.apache.org>; Tue, 30 Nov 2010 17:57:51 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=iSPS4pbfI/YMu26oZPgekqKJJyLAJPlttSSQ0r5WRBs=;
        b=wr5AMMGQ7lN54sMx22sA0oJqgVh85/u/KugvQOsur5y7CjG6JRq/xDmIM/E+Putp8i
         0oXHDtQvH5d+whFunQJTsbyasNWGnyFB11MYKxReICbgzJ1PbbQ79jptfnELJAfB5Wfd
         mAmiZtwFky5AqkEe/MM3wlkrtEGc09HOtAjE8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=QNIIKUj0YGg+KAT8Q7q6skKdLymR8ugJMwoW/0tK2HrzhdLhT5N36DB8lA7C8ReLy6
         OmZaWWA1lshwIrf6gaUPj98aVILOxdzf0CMqRwecOyyoKHU+f7j4+hJm1yPbXhYLZxKy
         spMI63QPF4+Ti8gGKtc4iRnj+pOb8XDQnYLc8=
MIME-Version: 1.0
Received: by 10.150.158.2 with SMTP id g2mr13839477ybe.187.1291168670744; Tue,
 30 Nov 2010 17:57:50 -0800 (PST)
Received: by 10.236.108.180 with HTTP; Tue, 30 Nov 2010 17:57:50 -0800 (PST)
In-Reply-To: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>
Date: Tue, 30 Nov 2010 17:57:50 -0800
Message-ID: <AANLkTik1F-DCc0GNrk0dEqzRp5f-MvOQSjsg5+Z9neXf@mail.gmail.com>
Subject: Re: [VOTE] Direction for Hadoop development
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd59e76f7033b04964fa1c3

--000e0cd59e76f7033b04964fa1c3
Content-Type: text/plain; charset=ISO-8859-1

This sounds like an important issue. But I personally don't understand what
exactly the controversy is, and therefore what is this vote about, and what
are the choices, if any.
What I understand is that the issue spans over at least two (long) issues
and different discussion threads. Could somebody knowledgeable make an
independent digest of what is going on and how it stands. I am probably not
alone who struggles with this.
Is there a simple answer, like
"you vetoed yesterday - now it's my turn" or
"Avro should/not hold the monopoly for Hadoop serialization"?
These were just humorous examples.

Thanks,
--Konstantin

On Mon, Nov 29, 2010 at 2:30 PM, Owen O'Malley <oom@yahoo-inc.com> wrote:

> All,
>   Based on the discussion on HADOOP-6685, there is a pretty fundamental
> difference of opinion about how Hadoop should evolve. We need to figure out
> how the majority of the PMC wants the project to evolve to understand which
> patches move us forward. Please vote whether you approve of the following
> direction. Clearly as the author, I'm +1.
>
> -- Owen
>
> Hadoop has always included library code so that users had a strong
> foundation to build their applications on without needing to continually
> reinvent the wheel. This combination of framework and powerful library code
> is a common pattern for successful projects, such as Java, Lucene, etc.
> Toward that end, we need to continue to extend the Hadoop library code and
> actively maintain it as the framework evolves. Continuing support for
> SequenceFile and TFile, which are both widely used is mandatory. The
> opposite pattern of implementing the framework and letting each distribution
> add the required libraries will lead to increased community fragmentation
> and vendor lock in.
>
> Hadoop's generic serialization framework had a lot of promise when it was
> introduced, but has been hampered by a lack of plugins other than Writables
> and Java serialization. Supporting a wide range of serializations natively
> in Hadoop will give the users new capabilities. Currently, to support Avro
> or ProtoBuf objects mutually incompatible third party solutions are
> required. It benefits Hadoop to support them with a common framework that
> will support all of them. In particular, having easy, out of the box support
> for Thrift, ProtoBufs, Avro, and our legacy serializations is a desired
> state.
>
> As a distributed system, there are many instances where Hadoop needs to
> serialize data. Many of those applications need a lightweight, versioned
> serialization framework like ProtocolBuffers or Thrift and using them is
> appropriate. Adding dependences on Thrift and ProtocolBuffers to the
> previous dependence on Avro is acceptable.

--000e0cd59e76f7033b04964fa1c3--

From general-return-2466-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 01 12:26:49 2010
Return-Path: <general-return-2466-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 75276 invoked from network); 1 Dec 2010 12:26:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Dec 2010 12:26:49 -0000
Received: (qmail 31939 invoked by uid 500); 1 Dec 2010 12:26:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31702 invoked by uid 500); 1 Dec 2010 12:26:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31694 invoked by uid 99); 1 Dec 2010 12:26:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Dec 2010 12:26:46 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Dec 2010 12:26:37 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id BCF72B7E0A
	for <general@hadoop.apache.org>; Wed,  1 Dec 2010 12:26:14 +0000 (GMT)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id rIJjh+Io2+lA for <general@hadoop.apache.org>;
	Wed,  1 Dec 2010 12:26:13 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 8032CB7E09
	for <general@hadoop.apache.org>; Wed,  1 Dec 2010 12:26:11 +0000 (GMT)
MailScanner-NULL-Check: 1291811157.3476@CsvElZUJOaIralffvs9nOQ
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id oB1CPu1A006711
	for <general@hadoop.apache.org>; Wed, 1 Dec 2010 12:25:56 GMT
Message-ID: <4CF63ED4.2000802@apache.org>
Date: Wed, 01 Dec 2010 12:25:56 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.15) Gecko/20101027 Thunderbird/3.0.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Direction for Hadoop development
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>
In-Reply-To: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: oB1CPu1A006711
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 29/11/10 22:30, Owen O'Malley wrote:
> All,
> Based on the discussion on HADOOP-6685, there is a pretty fundamental
> difference of opinion about how Hadoop should evolve. We need to figure
> out how the majority of the PMC wants the project to evolve to
> understand which patches move us forward. Please vote whether you
> approve of the following direction. Clearly as the author, I'm +1.
>
> -- Owen
>
> Hadoop has always included library code so that users had a strong
> foundation to build their applications on without needing to continually
> reinvent the wheel. This combination of framework and powerful library
> code is a common pattern for successful projects, such as Java, Lucene,
> etc. Toward that end, we need to continue to extend the Hadoop library
> code and actively maintain it as the framework evolves. Continuing
> support for SequenceFile and TFile, which are both widely used is
> mandatory. The opposite pattern of implementing the framework and
> letting each distribution add the required libraries will lead to
> increased community fragmentation and vendor lock in.
>
> Hadoop's generic serialization framework had a lot of promise when it
> was introduced, but has been hampered by a lack of plugins other than
> Writables and Java serialization. Supporting a wide range of
> serializations natively in Hadoop will give the users new capabilities.
> Currently, to support Avro or ProtoBuf objects mutually incompatible
> third party solutions are required. It benefits Hadoop to support them
> with a common framework that will support all of them. In particular,
> having easy, out of the box support for Thrift, ProtoBufs, Avro, and our
> legacy serializations is a desired state.
>
> As a distributed system, there are many instances where Hadoop needs to
> serialize data. Many of those applications need a lightweight, versioned
> serialization framework like ProtocolBuffers or Thrift and using them is
> appropriate. Adding dependences on Thrift and ProtocolBuffers to the
> previous dependence on Avro is acceptable.


I'm happy with new build-time dependencies on these libraries, with one 
big warning. Until an official, non-incubation release of Thrift comes 
out (and thrift moves from incubation), the Apache Management will veto 
any redistribution of the thrift JARs; they aren't signed off as for 
public use.

I'm not so sure about more runtime depencencies that go all the way into 
the classpath of the things working with HDFS, or files created in it, 
because that leads to version problems in private code. [Inevitably 
Hadoop will end up adopting for some OSGi-like classpath setup, but I'm 
not pushing for that as it has its own interesting issues].

At the same time -you can't add features without adding dependencies 
except by playing rebasing tricks, and I have mixed feelings about those 
tricks:
   good: lets the hadoop team push things out on their schedule
   bad: impossible to push out security bug fixes to dependent libraries 
without rebuilding and re-releasing things. Your ops team will hate you.

For the bad reason, and because it's extra work, I avoid playing 
rebasing games, just try and do classpaths right in the first place 
-which is easier said than done.

One part of the HADOOP-6685 discussion raised was JSON as a format for 
things. Adopting JSON -and deciding which JSON parser to use- is trouble.


Ignoring the ongoing discussion of serialization formats, the question 
"should we use JSON?" really leads back to "which external JSON parser 
do we want to use?", which is a separate -and significant problem.

I say this as someone who has three separate json parsers on the runtime 
classpath of something whose functional tests are failing in a hudson 
window blinking at me alongside this email application.

gson: http://code.google.com/p/google-gson/
  http://mvnrepository.com/artifact/com.google.code.gson/gson/1.4
  com.google.code.gson/gson-1.5.1; no runtime dependencies
-some people like the seamless binding to java objects, which I view as 
repeating the same mistakes as WS-*.

json-lib: http://json-lib.sourceforge.net/
http://mvnrepository.com/artifact/net.sf.json-lib/json-lib/2.3
at runtime tends to need the usual commons-logging back end and
  net.sf.json-lib/json-lib-2.3
  net.sf.ezmorph/ezmorph-1.06
  commons-lang-2.4
  commons-collections-3.2.1
-low level, DOM-ish, could be improved to be more Java-5-intuitive

Jackson: http://jackson.codehaus.org
org.codehaus.jackson/jackson-core-asl-1.6.2
org.codehaus.jackson/jackson-asl/0.9.5

Now, before someone points out that three JSON parsers is too many, this 
same code has log4J, SLF4J (with a back end to JSCL), a patched back end 
logger for Jetty to avoid SLF4J where possible, and a custom JCL 
back-end. XML side there's xerces and xalan instead of the JVM versions, 
and hibernate pulling in dom4j alongside. Test runs add htlmunit to the 
classpath, which pulls in the older httpclient libs, along with the 
http-core stuff I've switched to.

Java library versions -while more manageable than native library 
versions- are a pain. Regardless of the ugliness of XML or the 
mediocrity of DOM, running over to JSON just because DOM is unwieldy is 
replacing one source of trouble for another.

If Hadoop is going to use JSON in places, then the discussion/decision 
about which JSON parser to stick on the classpath is worthy of a JIRA 
issue all of its own.

-steve

(returning to his failing tests)




From general-return-2467-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 01 15:22:26 2010
Return-Path: <general-return-2467-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 48679 invoked from network); 1 Dec 2010 15:22:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Dec 2010 15:22:26 -0000
Received: (qmail 37981 invoked by uid 500); 1 Dec 2010 15:22:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37624 invoked by uid 500); 1 Dec 2010 15:22:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37614 invoked by uid 99); 1 Dec 2010 15:22:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Dec 2010 15:22:22 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 01 Dec 2010 15:22:20 +0000
Received: (qmail 48569 invoked by uid 99); 1 Dec 2010 15:21:58 -0000
Received: from localhost.apache.org (HELO [192.168.43.54]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Dec 2010 15:21:58 +0000
Message-ID: <4CF66812.4080602@apache.org>
Date: Wed, 01 Dec 2010 07:21:54 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Direction for Hadoop development
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>	<4CF433D4.6090407@apache.org> <AANLkTineTnsShLoa5vO-X9UAp_DUMrHRjc2ry=i4qkXs@mail.gmail.com>
In-Reply-To: <AANLkTineTnsShLoa5vO-X9UAp_DUMrHRjc2ry=i4qkXs@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 11/29/2010 04:22 PM, Owen O'Malley wrote:
>Using a vote to
> decide on project direction is far better than vetoing patches based on
> disjoint visions of how to move forward. If the project votes for a
> direction, a veto that is based on an opposing direction is clearly invalid.

This is unfortunately not the way Apache projects operate.  Vetos are 
not overridden by a majority votes.

> For example, in your last MapReduce (MAPREDUCE-980) patch you
> added avro and paranamer as dependences.

If I'm not mistaken, that only adds a dependency to the JobTracker.  We 
don't create specific classpaths for daemons than for user code, but we 
probably should, so that things that only the daemon uses are not also 
placed on the users classpath.

Doug

From general-return-2468-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 01 15:40:37 2010
Return-Path: <general-return-2468-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54114 invoked from network); 1 Dec 2010 15:40:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Dec 2010 15:40:37 -0000
Received: (qmail 74592 invoked by uid 500); 1 Dec 2010 15:40:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74503 invoked by uid 500); 1 Dec 2010 15:40:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74492 invoked by uid 99); 1 Dec 2010 15:40:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Dec 2010 15:40:34 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.62.64] (HELO qmta07.westchester.pa.mail.comcast.net) (76.96.62.64)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Dec 2010 15:40:25 +0000
Received: from omta15.westchester.pa.mail.comcast.net ([76.96.62.87])
	by qmta07.westchester.pa.mail.comcast.net with comcast
	id do0m1f0031swQuc57rg66N; Wed, 01 Dec 2010 15:40:06 +0000
Received: from [10.0.0.13] ([98.234.216.58])
	by omta15.westchester.pa.mail.comcast.net with comcast
	id drg41f00P1GAoR63brg5hw; Wed, 01 Dec 2010 15:40:05 +0000
Message-Id: <D414D20A-A909-46A9-AAA2-52742E6395CF@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <4CF66812.4080602@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Direction for Hadoop development
Date: Wed, 1 Dec 2010 07:40:03 -0800
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>	<4CF433D4.6090407@apache.org> <AANLkTineTnsShLoa5vO-X9UAp_DUMrHRjc2ry=i4qkXs@mail.gmail.com> <4CF66812.4080602@apache.org>
X-Mailer: Apple Mail (2.936)


On Dec 1, 2010, at 7:21 AM, Doug Cutting wrote:

> On 11/29/2010 04:22 PM, Owen O'Malley wrote:
>> Using a vote to
>> decide on project direction is far better than vetoing patches  
>> based on
>> disjoint visions of how to move forward. If the project votes for a
>> direction, a veto that is based on an opposing direction is clearly  
>> invalid.
>
> This is unfortunately not the way Apache projects operate.  Vetos  
> are not overridden by a majority votes.

This isn't overriding the veto. You've based your veto on changes in  
project plan that have never been discussed or agreed to. At Ian's  
suggestion, I wrote my statement to clarify what the project wanted to  
do. It is completely appropriate for the PMC to vote about what the  
project should do going forward. I would hope that if the PMC votes  
for my proposal that you'd withdraw your veto since the basis for your  
veto would have been rejected.

-- Owen

From general-return-2469-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 01 16:29:42 2010
Return-Path: <general-return-2469-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78992 invoked from network); 1 Dec 2010 16:29:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Dec 2010 16:29:41 -0000
Received: (qmail 82856 invoked by uid 500); 1 Dec 2010 16:29:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82585 invoked by uid 500); 1 Dec 2010 16:29:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82572 invoked by uid 99); 1 Dec 2010 16:29:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Dec 2010 16:29:36 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jaybooth@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Dec 2010 16:29:30 +0000
Received: by qwj9 with SMTP id 9so1743529qwj.35
        for <general@hadoop.apache.org>; Wed, 01 Dec 2010 08:29:09 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=5CSCbJKJUXA8hvAIl7t/DP61jX//njBTOE1a5gyW7dM=;
        b=m7fpcS0BP1gIc9RgM3lZAS+3AjHRdq4KQx8kGsOKYwCJkj0UqncVWYMOzsGSCibsLZ
         fUq1aRIB9e9cpJuWzKCFOclZwCU1/iWfjWKqmNq+i4Zx8hDwTgF//xwX+2QTzlnDvZEQ
         XzztVehFU4hF07uVHlSVHrcvJaCQ3tLies60w=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=CaDlanL1C94ClWf4e3Wpn/s3ZN28alyddB+JiWYwGHnvm13SjYgzNvETUlnxP6prkY
         wBC3N6Pj9uoUQEoLnCdI1UNXhHs5qbRaRWy0LSGdYZm/aaTy8UZFSfH5Q1PaKFdMlvlG
         DPCSXt1ZzBi2oGy5IGYloz1hNFlhOmS2E3Nys=
MIME-Version: 1.0
Received: by 10.224.2.70 with SMTP id 6mr1749532qai.83.1291220949582; Wed, 01
 Dec 2010 08:29:09 -0800 (PST)
Received: by 10.220.159.136 with HTTP; Wed, 1 Dec 2010 08:29:09 -0800 (PST)
In-Reply-To: <4CF66812.4080602@apache.org>
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>
	<4CF433D4.6090407@apache.org>
	<AANLkTineTnsShLoa5vO-X9UAp_DUMrHRjc2ry=i4qkXs@mail.gmail.com>
	<4CF66812.4080602@apache.org>
Date: Wed, 1 Dec 2010 11:29:09 -0500
Message-ID: <AANLkTimvm+YDzSYan9fGpVsADAdSkfiVu=C9HA3KhDL0@mail.gmail.com>
Subject: Re: [VOTE] Direction for Hadoop development
From: Jay Booth <jaybooth@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175caaae06c50904965bcea0

--0015175caaae06c50904965bcea0
Content-Type: text/plain; charset=ISO-8859-1

>
>
>  For example, in your last MapReduce (MAPREDUCE-980) patch you
>> added avro and paranamer as dependences.
>>
>
> If I'm not mistaken, that only adds a dependency to the JobTracker.  We
> don't create specific classpaths for daemons than for user code, but we
> probably should, so that things that only the daemon uses are not also
> placed on the users classpath.
>
>
+1 to separate classpaths for daemons as an eventual goal.  As a user, I've
definitely lost an afternoon to commons-lang version mismatches.   If we can
add fewer things to the Task classpath, that's fewer potential future lost
afternoons.

I'm not a PMC member, but I suspect I spend more time doing user-level grunt
work than many PMC members, so from that perspective:

On internal-to-hadoop serialization:
I'm going to spend 99% of my time not caring about these formats and the
other 1% of the time needing to know what's going on with them
*immediately*.  Right now I know nothing about protobuf.. learning new
things is always great but "while my production job is broken and I'm trying
to debug it" isn't really going to be the best time and place for it.  JSON
on the other hand is human readable and never going to change.  I feel a lot
safer with JSON than with any binary format, especially considering that we
could all be using NewHawtUnforeseenLibrary or
IncompatibleWithPreviousReleaseLibrary for our binary serialization in a
couple years.

On packaging serialization lib dependencies:
Again, additional versioned dependencies on the Task classpath scare me, and
that goes double for serialization.  I could see a couple ways around it
that fall prey to the inner-framework antipattern, and for what it's worth,
I'd be willing to accept that additional kludginess if it meant that I
wasn't strictly dependent on avro x.x or thrift y.y.  What if I'm reading a
file that was encoded with an incompatible version?  This gets way out of
scope from the immediate issue but if I could ship my own serialization
library in an assembly jar, and maybe override an additional method or
supply a MapOutputEncoder or something, I'd take that tradeoff over being
bound to a particular version until the next version of Hadoop comes out.
 If there were sensible defaults in place, it might not even mean more
complexity for the average job.

--0015175caaae06c50904965bcea0--

From general-return-2470-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 01 17:35:58 2010
Return-Path: <general-return-2470-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16471 invoked from network); 1 Dec 2010 17:35:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Dec 2010 17:35:58 -0000
Received: (qmail 45400 invoked by uid 500); 1 Dec 2010 17:35:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45343 invoked by uid 500); 1 Dec 2010 17:35:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45335 invoked by uid 99); 1 Dec 2010 17:35:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Dec 2010 17:35:56 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jonathan.seidman@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Dec 2010 17:35:51 +0000
Received: by wwb34 with SMTP id 34so293058wwb.29
        for <general@hadoop.apache.org>; Wed, 01 Dec 2010 09:35:29 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=wLGqONK9Cufc6eS7IgWKWqcqTvqG4UwPcJzEyyJli2Y=;
        b=xL3ZckRKdxtCMdRllDg5lW4GLEVEAzBt6iUPCW4LrZIFr9owO9+iHuPxRz7UTTiyGM
         ObH97aSp//BqL4RdlgQYEHhTqyRXYQ33CXmjyX4ogls6Ok6lsUpxkZMzeJbx2ZsBCq0B
         TfZEMnGB4b8Nbzyj5wiF58H3vST5CPFZAhcVg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=j9pfOW3cTJvg+mlFVn52INqrLbfJWkfzdrJgo1Nt9fhidQJ2qqYoCd8rt8Uz71sSGd
         6IKZQ3RqeB1zRtEZuun7Ysz7vH5HpsktQ4eOMEvHGwIkMxmLXg9tKSEp7EVjJdyShcLe
         xpSzrj09DOswncPkC63uV/JKSesF/Zo/EOywc=
MIME-Version: 1.0
Received: by 10.227.144.65 with SMTP id y1mr9746419wbu.199.1291224929297; Wed,
 01 Dec 2010 09:35:29 -0800 (PST)
Received: by 10.227.196.135 with HTTP; Wed, 1 Dec 2010 09:35:29 -0800 (PST)
Date: Wed, 1 Dec 2010 11:35:29 -0600
Message-ID: <AANLkTinGf7bhFth9GZ-aN1sY12yv2T3nWm7LYVYkhHiS@mail.gmail.com>
Subject: [ANNOUNCE] Chicago Hadoop User Group December Meetup - Intro to HBase
From: Jonathan Seidman <jonathan.seidman@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636831b923cb0e504965cbbd2

--001636831b923cb0e504965cbbd2
Content-Type: text/plain; charset=ISO-8859-1

The next meeting of the Chicago area Hadoop User Group will be help next
Wednesday, December 8th from 5:30-7:30PM, and will feature an introduction
to HBase by Lars George of Cloudera. The meeting will be hosted by Navteq,
located in the Boeing building at 425 W. Randolph Street in downtown Chicago.
Slots are limited, so RSVP now.

More details and RSVP here:
http://www.meetup.com/Chicago-area-Hadoop-User-Group-CHUG/calendar/15500107/

Hope to see you there!

Jonathan

--001636831b923cb0e504965cbbd2--

From general-return-2471-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 01 19:12:28 2010
Return-Path: <general-return-2471-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73237 invoked from network); 1 Dec 2010 19:12:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Dec 2010 19:12:27 -0000
Received: (qmail 41028 invoked by uid 500); 1 Dec 2010 19:12:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40800 invoked by uid 500); 1 Dec 2010 19:12:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40792 invoked by uid 99); 1 Dec 2010 19:12:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Dec 2010 19:12:25 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.62.64] (HELO qmta07.westchester.pa.mail.comcast.net) (76.96.62.64)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Dec 2010 19:12:15 +0000
Received: from omta08.westchester.pa.mail.comcast.net ([76.96.62.12])
	by qmta07.westchester.pa.mail.comcast.net with comcast
	id dmuB1f0010Fqzac57vBvWD; Wed, 01 Dec 2010 19:11:55 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta08.westchester.pa.mail.comcast.net with comcast
	id dvBj1f0052SbwD53UvBnwM; Wed, 01 Dec 2010 19:11:52 +0000
Message-Id: <A32CDC9F-D6C6-4BA6-A44B-5FF206C1DF49@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <AANLkTik1F-DCc0GNrk0dEqzRp5f-MvOQSjsg5+Z9neXf@mail.gmail.com>
Content-Type: multipart/alternative; boundary=Apple-Mail-48--609110743
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Direction for Hadoop development
Date: Wed, 1 Dec 2010 11:11:41 -0800
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com> <AANLkTik1F-DCc0GNrk0dEqzRp5f-MvOQSjsg5+Z9neXf@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-48--609110743
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


On Nov 30, 2010, at 5:57 PM, Konstantin Shvachko wrote:

> This sounds like an important issue. But I personally don't  
> understand what
> exactly the controversy is, and therefore what is this vote about,  
> and what
> are the choices, if any.

The question is how the Hadoop project wants to move forward.

It was motivated by Doug's veto of HADOOP-6685, which was based on his  
personal decisions about how the project should go forward and not on  
anything that had been decided by the PMC.

These decisions are much more important to MapReduce, which is a  
framework, than HDFS which is a client/server model.

1. Should Hadoop include a user-facing library of useful code?

There has been a suggestion that user-facing library code, such as  
SequenceFile, TFile, DistCp, etc. should be deprecated and that Hadoop  
should allow third party projects like Avro to supply the user-facing  
library code that makes Hadoop usable. I think it is critical that we  
keep those components as part of Hadoop and extend them as the  
framework evolves. Users depend heavily on SequenceFile for storing  
their data in Hadoop and they should not  be deprecated as Doug has  
suggested.

2. Should MapReduce support non-Writables through the pipeline out of  
the box?

There has also been a discussion about whether we should support non- 
Writables natively. There is already library code in Avro that lets  
users use Avro types in a custom MapReduce API. A general MapReduce  
API that encompasses all of the serialization frameworks and does not  
lock users into a particular one is much more powerful.

Furthermore, making it convenient for the users, by including the  
plugins in the default configuration and class path, will enable the  
use of Avro, Thrift and ProtoBuf objects by people who would rather  
not focus on serialization. Avro and Writables should not be the only  
first class serializations that Hadoop supports by default.

3. Should a framework dependency on ProtoBuf be allowed?

Doug has added several framework dependences on Avro. The question is  
whether it is acceptable to use the ProtoBuf library in the framework.  
Avro is good for uses where there are a lot of objects of the same  
type. ProtoBuf is better for small number of objects. The question is  
whether Avro, JSON, and XML should be the only serialization libraries  
that are acceptable to use in the framework.

-- Owen
--Apple-Mail-48--609110743--

From general-return-2472-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 01 22:38:24 2010
Return-Path: <general-return-2472-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 61758 invoked from network); 1 Dec 2010 22:38:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Dec 2010 22:38:23 -0000
Received: (qmail 31142 invoked by uid 500); 1 Dec 2010 22:38:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31075 invoked by uid 500); 1 Dec 2010 22:38:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31067 invoked by uid 99); 1 Dec 2010 22:38:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Dec 2010 22:38:22 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of akimball83@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Dec 2010 22:38:15 +0000
Received: by bwz9 with SMTP id 9so7312696bwz.35
        for <general@hadoop.apache.org>; Wed, 01 Dec 2010 14:37:55 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:from:date
         :message-id:subject:to:content-type:content-transfer-encoding;
        bh=DrSsH9RiROEHc/qEVL8ZjdgRvMt2FosCpqBtP1mDhu4=;
        b=Eq0lblg8Jyt+RwfqLxEPk5YWLmcFiIYuxIsC+QQz8vm7eM9U/4Uomkc/mNR2q9V+Df
         CLnRa58QdHJHNGJcD/t5MATQnArb5w2eUS7EB/iBoPyJrzBdDkRuXADYhWCxUV9tNPFE
         8/v8OUIjqIuaNXiT0GYACGOW+KKLZoWWkwdKo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        b=xiv6JxYF9SZEzNhk/jQj5Kx8fzbJ0OGBPapD3qL1+frQj3GOlAKn1vn8EDjjDlQ6lE
         iL261wQAl0emlbE9f4RWMSA8bKZpnw9alcMRuXiygSLOr9z3fa7OtkufKpfEb5fn0Zwf
         lyOTeLKBWZkhazMxquj8T4xqKHF+KjzQxiZi8=
Received: by 10.204.118.7 with SMTP id t7mr1872690bkq.97.1291243074764; Wed,
 01 Dec 2010 14:37:54 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.99.69 with HTTP; Wed, 1 Dec 2010 14:37:34 -0800 (PST)
From: Aaron Kimball <akimball83@gmail.com>
Date: Wed, 1 Dec 2010 14:37:34 -0800
Message-ID: <AANLkTiks11L57qh9VKQavxuSCTwN6vzjjmHe-scadHgP@mail.gmail.com>
Subject: Announcing the first San Francisco Hadoop meetup -- Jan 12, 2011
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Hadoop fans,

Earlier in the month I proposed a San Francisco-based Hadoop meetup.
We are pleased to announce that we will be holding this event on
Wednesday, January 12, 2011 from 6pm to 8pm.

Rapleaf has graciously offered to host the event in their SF office at
3rd and Mission (667 Mission St., 4th Floor). We are=A0also thankful to
Cloudera, who will be helping to sponsor the event.

As mentioned in the initial proposal, the format of this event will be
different than the presentation-based HUG; there will not be prepared
lectures. Instead, all participants are asked to discuss their usage
of, or questions about Hadoop. =A0Please come prepared to introduce
yourself and provide a one--two minute summary of=A0what you're working
on with=A0Hadoop, areas you're interested in learning more about, and/or
challenges that you're facing that might be addressed by a larger
Hadoop community.=A0All are welcome (developers, users, or=A0managers;
experienced or newbies).

Based on the common themes that arise, we will break into smaller
groups to discuss these in greater depth. Several experienced
Hadoopers will be on hand to help moderate small group discussions.

Agenda:
6pm - Socializing
6:30pm - Introductions, overview
7pm - Breakout sessions begin
8pm - Conclusion

I hope to see many of you there! The space available has a limit of 40
participants; please RSVP at
http://www.meetup.com/hadoopsf/calendar/15601471/. The meeting
invitation is hosted at=A0http://www.meetup.com/hadoopsf=A0-- please sign
up for this meetup group to get information about future meetups.
Based on the success of the initial event, we hope to hold more of
these in the future.

Regards,
- Aaron Kimball

From general-return-2473-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 02 05:24:38 2010
Return-Path: <general-return-2473-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33226 invoked from network); 2 Dec 2010 05:24:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Dec 2010 05:24:38 -0000
Received: (qmail 64545 invoked by uid 500); 2 Dec 2010 05:24:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 64346 invoked by uid 500); 2 Dec 2010 05:24:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 64338 invoked by uid 99); 2 Dec 2010 05:24:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Dec 2010 05:24:36 +0000
X-ASF-Spam-Status: No, hits=3.7 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of prashantsethia007@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Dec 2010 05:24:28 +0000
Received: by qwj9 with SMTP id 9so2427402qwj.35
        for <general@hadoop.apache.org>; Wed, 01 Dec 2010 21:24:07 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=Ime1372bSJDY3esQRaOpss4slFTqZPiPP95CeWlg8A0=;
        b=U66q9K8iNlZYjpMlJ5D+q6HJmU4l+TxGee87rkxuEWV6QhEQOFMj1oZXaSwOALRzsH
         dYZiMYVvqz37OBfwYWUTAYI88MD8WdRo7tarRCp8mB3PSrxtCwBBeWGLrNjHKQ0dgsWB
         Vr3G/csRmB/ZMM12TpVrnmNIMQzuYs5RXk+7Q=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=aK5LQ/i3Seo/l7FTGIt7QSDztHlfwwb+azrwxwZY3t+m2A1CFDkZ0Y31N6mdfGxDlZ
         YbpnBfWmxb6sz9uU57NF7SAXef0BvgXuWBPb8I977oAMj9dNkAcllIlVa4xNQVVaM35z
         OgJxzQV6MnKNm1a+BIpjQC/qGr+wqAVM9etaQ=
MIME-Version: 1.0
Received: by 10.229.250.130 with SMTP id mo2mr8151712qcb.126.1291267447180;
 Wed, 01 Dec 2010 21:24:07 -0800 (PST)
Received: by 10.229.44.74 with HTTP; Wed, 1 Dec 2010 21:24:07 -0800 (PST)
Date: Thu, 2 Dec 2010 00:24:07 -0500
Message-ID: <AANLkTimwX2wm7GaHf1ESTTsNL+U7DMma0Sy6R6k58Kn1@mail.gmail.com>
Subject: Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
From: prashant sethia <prashantsethia007@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016363b804c7fcd85049666a18d
X-Virus-Checked: Checked by ClamAV on apache.org

--0016363b804c7fcd85049666a18d
Content-Type: text/plain; charset=ISO-8859-1

Hi all,

I have installed Hadoop 0.20.2 on a cluster of 6 computers. Everything runs
fine when I run an application on Hadoop with one system in the cluster. But
as soon as I include more than one system and  then run a job, I get the
following error after completion of the map task:

Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.

The value of "ulimit -n" is set to 35000. I am not using any DNS names, so
no DNS resolution problems.

What can be the problem then?

Same happens with Hadoop 0.21.0 version.


Thanks,
Prashant.

--0016363b804c7fcd85049666a18d--

From general-return-2474-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 02 11:44:35 2010
Return-Path: <general-return-2474-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 79540 invoked from network); 2 Dec 2010 11:44:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Dec 2010 11:44:35 -0000
Received: (qmail 32956 invoked by uid 500); 2 Dec 2010 11:44:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32537 invoked by uid 500); 2 Dec 2010 11:44:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32529 invoked by uid 99); 2 Dec 2010 11:44:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Dec 2010 11:44:33 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Dec 2010 11:44:24 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 50A001BA47A
	for <general@hadoop.apache.org>; Thu,  2 Dec 2010 11:44:03 +0000 (GMT)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id F+kX8ZFnfn-T for <general@hadoop.apache.org>;
	Thu,  2 Dec 2010 11:44:02 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by colossus.hpl.hp.com (Postfix) with ESMTPS id 1F8C21BA688
	for <general@hadoop.apache.org>; Thu,  2 Dec 2010 11:44:01 +0000 (GMT)
MailScanner-NULL-Check: 1291895027.3617@awpVYkG7iL7tBMwjMAjQGw
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id oB2BhkTp002824
	for <general@hadoop.apache.org>; Thu, 2 Dec 2010 11:43:46 GMT
Message-ID: <4CF78672.30800@apache.org>
Date: Thu, 02 Dec 2010 11:43:46 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.15) Gecko/20101027 Thunderbird/3.0.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
References: <AANLkTimwX2wm7GaHf1ESTTsNL+U7DMma0Sy6R6k58Kn1@mail.gmail.com>
In-Reply-To: <AANLkTimwX2wm7GaHf1ESTTsNL+U7DMma0Sy6R6k58Kn1@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: oB2BhkTp002824
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 02/12/10 05:24, prashant sethia wrote:
> Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES

I don't know about other people's causes of this message, but searching 
for it throws up the only time I've seen it

http://jira.smartfrog.org/jira/browse/SFOS-121

the mapred.task.tracker.http.address address was wrong.

If your route cause is the same as mine, look at networking and hostname 
settings, both /etc/hosts and the config files of the services

From general-return-2475-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Dec 06 17:16:45 2010
Return-Path: <general-return-2475-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 20770 invoked from network); 6 Dec 2010 17:16:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Dec 2010 17:16:45 -0000
Received: (qmail 93522 invoked by uid 500); 6 Dec 2010 17:16:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93411 invoked by uid 500); 6 Dec 2010 17:16:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93403 invoked by uid 99); 6 Dec 2010 17:16:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Dec 2010 17:16:42 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Dec 2010 17:16:37 +0000
Received: from [10.72.120.91] (tryarticle-lm.corp.yahoo.com [10.72.120.91])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id oB6HG08l066451
	for <general@hadoop.apache.org>; Mon, 6 Dec 2010 09:16:00 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1291655760;
	bh=pjSOqvfUNNkdgWOcoPhfV5afu25vNGjaw98LrDoGDoE=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:Mime-Version:Subject:
	 Date:References;
	b=sLUdOB/TKwAarTkU/l7yd+i/BHx+/tjTbtGNyTrr56QDuYuSQw3saEVx7C3ArB9f9
	 MAZKp6qNmal/zWENTuK+FmszjxNXm6ah4UftH6mfu9YTZ0swphEYu+uFedTwM880My
	 cdP1ZlRmMQ1WtrWNFyQsa6HOiMnZIQlZOCJwkIf8=
Message-Id: <0E66662F-E044-4738-8C46-EA0EBCE674E0@yahoo-inc.com>
From: "Owen O'Malley" <oom@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <A32CDC9F-D6C6-4BA6-A44B-5FF206C1DF49@apache.org>
Content-Type: multipart/alternative; boundary=Apple-Mail-56--184051914
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Direction for Hadoop development
Date: Mon, 6 Dec 2010 09:16:00 -0800
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com> <AANLkTik1F-DCc0GNrk0dEqzRp5f-MvOQSjsg5+Z9neXf@mail.gmail.com> <A32CDC9F-D6C6-4BA6-A44B-5FF206C1DF49@apache.org>
X-Mailer: Apple Mail (2.936)

--Apple-Mail-56--184051914
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


On Dec 1, 2010, at 11:11 AM, Owen O'Malley wrote:

All,
    We really need some guidance on the general direction for the  
project. Please comment and/or vote. If no one cares, then I'll  
probably commit it to Yahoo's internal branch.

-- Owen

> The question is how the Hadoop project wants to move forward.
>
> It was motivated by Doug's veto of HADOOP-6685, which was based on  
> his personal decisions about how the project should go forward and  
> not on anything that had been decided by the PMC.
>
> These decisions are much more important to MapReduce, which is a  
> framework, than HDFS which is a client/server model.
>
> 1. Should Hadoop include a user-facing library of useful code?
>
> There has been a suggestion that user-facing library code, such as  
> SequenceFile, TFile, DistCp, etc. should be deprecated and that  
> Hadoop should allow third party projects like Avro to supply the  
> user-facing library code that makes Hadoop usable. I think it is  
> critical that we keep those components as part of Hadoop and extend  
> them as the framework evolves. Users depend heavily on SequenceFile  
> for storing their data in Hadoop and they should not  be deprecated  
> as Doug has suggested.
>
> 2. Should MapReduce support non-Writables through the pipeline out  
> of the box?
>
> There has also been a discussion about whether we should support non- 
> Writables natively. There is already library code in Avro that lets  
> users use Avro types in a custom MapReduce API. A general MapReduce  
> API that encompasses all of the serialization frameworks and does  
> not lock users into a particular one is much more powerful.
>
> Furthermore, making it convenient for the users, by including the  
> plugins in the default configuration and class path, will enable the  
> use of Avro, Thrift and ProtoBuf objects by people who would rather  
> not focus on serialization. Avro and Writables should not be the  
> only first class serializations that Hadoop supports by default.
>
> 3. Should a framework dependency on ProtoBuf be allowed?
>
> Doug has added several framework dependences on Avro. The question  
> is whether it is acceptable to use the ProtoBuf library in the  
> framework. Avro is good for uses where there are a lot of objects of  
> the same type. ProtoBuf is better for small number of objects. The  
> question is whether Avro, JSON, and XML should be the only  
> serialization libraries that are acceptable to use in the framework.


--Apple-Mail-56--184051914--

From general-return-2476-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Dec 06 17:51:04 2010
Return-Path: <general-return-2476-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31266 invoked from network); 6 Dec 2010 17:51:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Dec 2010 17:51:04 -0000
Received: (qmail 42801 invoked by uid 500); 6 Dec 2010 17:51:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42433 invoked by uid 500); 6 Dec 2010 17:51:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42418 invoked by uid 99); 6 Dec 2010 17:51:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Dec 2010 17:51:01 +0000
X-ASF-Spam-Status: No, hits=-0.1 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of Deepika.Khera@avg.com designates 193.85.188.248 as permitted sender)
Received: from [193.85.188.248] (HELO ms.grisoft.cz) (193.85.188.248)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Dec 2010 17:50:55 +0000
Received: from phobos.cz.avg.com (phobos.cz.avg.com [192.168.200.160])
	(using TLSv1 with cipher AES128-SHA (128/128 bits))
	(No client certificate requested)
	by ms.grisoft.cz (Postfix) with ESMTP id 5EFFF38969;
	Mon,  6 Dec 2010 18:50:33 +0100 (CET)
Received: from miranda.us.avg.com (10.32.34.13) by phobos.cz.avg.com
 (192.168.200.160) with Microsoft SMTP Server (TLS) id 8.3.106.1; Mon, 6 Dec
 2010 18:50:33 +0100
Received: from miranda.us.avg.com ([10.32.34.13]) by miranda ([10.32.34.13])
 with mapi; Mon, 6 Dec 2010 12:50:31 -0500
From: Deepika Khera <Deepika.Khera@avg.com>
To: "mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>,
	"general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 6 Dec 2010 12:50:30 -0500
Subject: distcp fails with ConnectException
Thread-Topic: distcp fails with ConnectException
Thread-Index: AcuVbhL0oXWtRiapTAGklUVhrb6WIw==
Message-ID: <DA852C02397DBC4EBFAA1B7192FBD1506207CABCB0@miranda>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_DA852C02397DBC4EBFAA1B7192FBD1506207CABCB0miranda_"
MIME-Version: 1.0

--_000_DA852C02397DBC4EBFAA1B7192FBD1506207CABCB0miranda_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

I am trying to run distcp between 2  hdfs clusters and I am getting a Conne=
ctException .

The Command I am trying to run is of the form (considering clusters hadoop-=
A & hadoop-B):

./hadoop distcp -update hdfs://hadoop-A:8020/dira/part-r-00000 hdfs://hadoo=
p-B:8020/dirb/

and I am running it on the destination cluster (hadoop-B).

The stacktrace for the exception is :

Copy failed: java.net.ConnectException: Call to hadoop-A/10.0.173.11:8020 f=
ailed on connection exception: java.net.ConnectException: Connection refuse=
d
        at org.apache.hadoop.ipc.Client.wrapException(Client.java:767)
        at org.apache.hadoop.ipc.Client.call(Client.java:743)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
        at $Proxy0.getProtocolVersion(Unknown Source)
        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
        at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.jav=
a:113)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:214)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:177)
        at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(Distribu=
tedFileSystem.java:82)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java=
:1378)


I have tried ping and ssh in between these clusters and it is working fine.=
 On both clusters namenode is running as the same user.

The strange part is if I try the command on a single cluster (both source &=
 destination are on same DFS-  so its a simple copy), it still fails with t=
he same exception. So I mean I run the command below on cluster A.

./hadoop distcp -update hdfs://hadoop-A:8020/dir1/part-r-00000 hdfs://hadoo=
p-A:8020/dir2/

Is there anything else I need to have to get the distcp working?  Any hints=
 on what I could check will be helpful.

Thanks,
Deepika


--_000_DA852C02397DBC4EBFAA1B7192FBD1506207CABCB0miranda_--

From general-return-2477-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Dec 06 18:41:05 2010
Return-Path: <general-return-2477-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66805 invoked from network); 6 Dec 2010 18:41:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Dec 2010 18:41:04 -0000
Received: (qmail 27208 invoked by uid 500); 6 Dec 2010 18:41:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27158 invoked by uid 500); 6 Dec 2010 18:41:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27150 invoked by uid 99); 6 Dec 2010 18:41:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Dec 2010 18:41:03 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 06 Dec 2010 18:41:00 +0000
Received: (qmail 66714 invoked by uid 99); 6 Dec 2010 18:40:39 -0000
Received: from localhost.apache.org (HELO mail-gy0-f176.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Dec 2010 18:40:39 +0000
Received: by gyf1 with SMTP id 1so7220333gyf.35
        for <general@hadoop.apache.org>; Mon, 06 Dec 2010 10:40:37 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.34.139 with SMTP id l11mr6276932ibd.141.1291660837452;
 Mon, 06 Dec 2010 10:40:37 -0800 (PST)
Received: by 10.231.167.199 with HTTP; Mon, 6 Dec 2010 10:40:37 -0800 (PST)
In-Reply-To: <0E66662F-E044-4738-8C46-EA0EBCE674E0@yahoo-inc.com>
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>
	<AANLkTik1F-DCc0GNrk0dEqzRp5f-MvOQSjsg5+Z9neXf@mail.gmail.com>
	<A32CDC9F-D6C6-4BA6-A44B-5FF206C1DF49@apache.org>
	<0E66662F-E044-4738-8C46-EA0EBCE674E0@yahoo-inc.com>
Date: Mon, 6 Dec 2010 10:40:37 -0800
Message-ID: <AANLkTinxLVDjvDwtfxkiFdi5cbKgSM5DOCeG56aZjFM9@mail.gmail.com>
Subject: Re: [VOTE] Direction for Hadoop development
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

SequenceFile is an experimental file format; as long as it continues
to support existing data, I see no reason to block its continued
evolution. TFile is also part of the common project. The project could
support still more, provided the tools and documentation were
available to help users select the one that fits their use case.

This question is backwards. If the assertion is that a part of the
framework's development should be arrested, that claim requires a
discussion and vote. The PMC should not have to weigh in on allowing
code to change. -C

On Mon, Dec 6, 2010 at 9:16 AM, Owen O'Malley <oom@yahoo-inc.com> wrote:
>
> On Dec 1, 2010, at 11:11 AM, Owen O'Malley wrote:
>
> All,
> =A0 We really need some guidance on the general direction for the project=
.
> Please comment and/or vote. If no one cares, then I'll probably commit it=
 to
> Yahoo's internal branch.
>
> -- Owen
>
>> The question is how the Hadoop project wants to move forward.
>>
>> It was motivated by Doug's veto of HADOOP-6685, which was based on his
>> personal decisions about how the project should go forward and not on
>> anything that had been decided by the PMC.
>>
>> These decisions are much more important to MapReduce, which is a
>> framework, than HDFS which is a client/server model.
>>
>> 1. Should Hadoop include a user-facing library of useful code?
>>
>> There has been a suggestion that user-facing library code, such as
>> SequenceFile, TFile, DistCp, etc. should be deprecated and that Hadoop
>> should allow third party projects like Avro to supply the user-facing
>> library code that makes Hadoop usable. I think it is critical that we ke=
ep
>> those components as part of Hadoop and extend them as the framework evol=
ves.
>> Users depend heavily on SequenceFile for storing their data in Hadoop an=
d
>> they should not =A0be deprecated as Doug has suggested.
>>
>> 2. Should MapReduce support non-Writables through the pipeline out of th=
e
>> box?
>>
>> There has also been a discussion about whether we should support
>> non-Writables natively. There is already library code in Avro that lets
>> users use Avro types in a custom MapReduce API. A general MapReduce API =
that
>> encompasses all of the serialization frameworks and does not lock users =
into
>> a particular one is much more powerful.
>>
>> Furthermore, making it convenient for the users, by including the plugin=
s
>> in the default configuration and class path, will enable the use of Avro=
,
>> Thrift and ProtoBuf objects by people who would rather not focus on
>> serialization. Avro and Writables should not be the only first class
>> serializations that Hadoop supports by default.
>>
>> 3. Should a framework dependency on ProtoBuf be allowed?
>>
>> Doug has added several framework dependences on Avro. The question is
>> whether it is acceptable to use the ProtoBuf library in the framework. A=
vro
>> is good for uses where there are a lot of objects of the same type. Prot=
oBuf
>> is better for small number of objects. The question is whether Avro, JSO=
N,
>> and XML should be the only serialization libraries that are acceptable t=
o
>> use in the framework.
>
>

From general-return-2478-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Dec 06 18:47:58 2010
Return-Path: <general-return-2478-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69569 invoked from network); 6 Dec 2010 18:47:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Dec 2010 18:47:58 -0000
Received: (qmail 39523 invoked by uid 500); 6 Dec 2010 18:47:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39472 invoked by uid 500); 6 Dec 2010 18:47:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39464 invoked by uid 99); 6 Dec 2010 18:47:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Dec 2010 18:47:56 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Dec 2010 18:47:52 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id oB6IkXaR046827
	for <general@hadoop.apache.org>; Mon, 6 Dec 2010 10:46:33 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1291661193;
	bh=0L7t0PEJ25c4CLNB2+yBGgQ10FKOQww0w55e7XNdAuk=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:Mime-Version:Subject:
	 Date:References;
	b=XHRdz3Usmi7YesNkh2gq7vRUQaN6EpPK2l8/iSi4o0bKWHp4jg5bRzPFsbAJY9TaK
	 but5KzVShttsvNUeWrqri3pVi1odr0STjttxq4soef/HqziOpim1GqcXeFcC3E109J
	 nB+TvuTRlA5Po3TO3MwyPAqHarHVTMXD4/IqHfec=
Message-Id: <797F5D37-68A4-43DE-89CA-70C1A38B258B@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <AANLkTinxLVDjvDwtfxkiFdi5cbKgSM5DOCeG56aZjFM9@mail.gmail.com>
Content-Type: multipart/alternative; boundary=Apple-Mail-308--178619512
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Direction for Hadoop development
Date: Mon, 6 Dec 2010 10:46:32 -0800
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com> <AANLkTik1F-DCc0GNrk0dEqzRp5f-MvOQSjsg5+Z9neXf@mail.gmail.com> <A32CDC9F-D6C6-4BA6-A44B-5FF206C1DF49@apache.org> <0E66662F-E044-4738-8C46-EA0EBCE674E0@yahoo-inc.com> <AANLkTinxLVDjvDwtfxkiFdi5cbKgSM5DOCeG56aZjFM9@mail.gmail.com>
X-Mailer: Apple Mail (2.936)

--Apple-Mail-308--178619512
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


On Dec 6, 2010, at 10:40 AM, Chris Douglas wrote:
>
> This question is backwards. If the assertion is that a part of the
> framework's development should be arrested, that claim requires a
> discussion and vote. The PMC should not have to weigh in on allowing
> code to change. -C
>

Agreed.

Arresting development on SequenceFile is preposterous. There are  
several petabytes of data sitting on it all over for several reasons,  
including legacy. Stopping development on it is unreasonable. Apache  
Hadoop is volunteer driven, volunteers should be allowed to contribute  
as they see fit.

+1

Arun

> On Mon, Dec 6, 2010 at 9:16 AM, Owen O'Malley <oom@yahoo-inc.com>  
> wrote:
>>
>> On Dec 1, 2010, at 11:11 AM, Owen O'Malley wrote:
>>
>> All,
>>   We really need some guidance on the general direction for the  
>> project.
>> Please comment and/or vote. If no one cares, then I'll probably  
>> commit it to
>> Yahoo's internal branch.
>>
>> -- Owen
>>
>>> The question is how the Hadoop project wants to move forward.
>>>
>>> It was motivated by Doug's veto of HADOOP-6685, which was based on  
>>> his
>>> personal decisions about how the project should go forward and not  
>>> on
>>> anything that had been decided by the PMC.
>>>
>>> These decisions are much more important to MapReduce, which is a
>>> framework, than HDFS which is a client/server model.
>>>
>>> 1. Should Hadoop include a user-facing library of useful code?
>>>
>>> There has been a suggestion that user-facing library code, such as
>>> SequenceFile, TFile, DistCp, etc. should be deprecated and that  
>>> Hadoop
>>> should allow third party projects like Avro to supply the user- 
>>> facing
>>> library code that makes Hadoop usable. I think it is critical that  
>>> we keep
>>> those components as part of Hadoop and extend them as the  
>>> framework evolves.
>>> Users depend heavily on SequenceFile for storing their data in  
>>> Hadoop and
>>> they should not  be deprecated as Doug has suggested.
>>>
>>> 2. Should MapReduce support non-Writables through the pipeline out  
>>> of the
>>> box?
>>>
>>> There has also been a discussion about whether we should support
>>> non-Writables natively. There is already library code in Avro that  
>>> lets
>>> users use Avro types in a custom MapReduce API. A general  
>>> MapReduce API that
>>> encompasses all of the serialization frameworks and does not lock  
>>> users into
>>> a particular one is much more powerful.
>>>
>>> Furthermore, making it convenient for the users, by including the  
>>> plugins
>>> in the default configuration and class path, will enable the use  
>>> of Avro,
>>> Thrift and ProtoBuf objects by people who would rather not focus on
>>> serialization. Avro and Writables should not be the only first class
>>> serializations that Hadoop supports by default.
>>>
>>> 3. Should a framework dependency on ProtoBuf be allowed?
>>>
>>> Doug has added several framework dependences on Avro. The question  
>>> is
>>> whether it is acceptable to use the ProtoBuf library in the  
>>> framework. Avro
>>> is good for uses where there are a lot of objects of the same  
>>> type. ProtoBuf
>>> is better for small number of objects. The question is whether  
>>> Avro, JSON,
>>> and XML should be the only serialization libraries that are  
>>> acceptable to
>>> use in the framework.
>>
>>


--Apple-Mail-308--178619512--

From general-return-2479-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Dec 06 19:30:33 2010
Return-Path: <general-return-2479-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93101 invoked from network); 6 Dec 2010 19:30:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Dec 2010 19:30:32 -0000
Received: (qmail 7269 invoked by uid 500); 6 Dec 2010 19:30:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7204 invoked by uid 500); 6 Dec 2010 19:30:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7158 invoked by uid 99); 6 Dec 2010 19:30:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Dec 2010 19:30:29 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 06 Dec 2010 19:30:27 +0000
Received: (qmail 92766 invoked by uid 99); 6 Dec 2010 19:30:05 -0000
Received: from localhost.apache.org (HELO [192.168.168.16]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Dec 2010 19:30:05 +0000
Message-ID: <4CFD39BC.1080005@apache.org>
Date: Mon, 06 Dec 2010 11:30:04 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Direction for Hadoop development
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>	<4CF433D4.6090407@apache.org> <AANLkTineTnsShLoa5vO-X9UAp_DUMrHRjc2ry=i4qkXs@mail.gmail.com> <4CF66812.4080602@apache.org> <D414D20A-A909-46A9-AAA2-52742E6395CF@apache.org>
In-Reply-To: <D414D20A-A909-46A9-AAA2-52742E6395CF@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 12/01/2010 07:40 AM, Owen O'Malley wrote:
>> This is unfortunately not the way Apache projects operate. Vetos are
>> not overridden by a majority votes.
>
> This isn't overriding the veto. You've based your veto on changes in
> project plan that have never been discussed or agreed to.

Apache projects don't have project plans that are created by majority votes.

Have a look at, e.g.:

http://httpd.apache.org/dev/guidelines.html

Majority votes are only used for releases.  Plans are announced to get 
feedback, to aid in consensus building.

Doug

From general-return-2480-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Dec 06 21:15:16 2010
Return-Path: <general-return-2480-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 41600 invoked from network); 6 Dec 2010 21:15:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Dec 2010 21:15:15 -0000
Received: (qmail 91191 invoked by uid 500); 6 Dec 2010 21:15:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91123 invoked by uid 500); 6 Dec 2010 21:15:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91115 invoked by uid 99); 6 Dec 2010 21:15:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Dec 2010 21:15:14 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of tom.e.white@gmail.com designates 209.85.215.43 as permitted sender)
Received: from [209.85.215.43] (HELO mail-ew0-f43.google.com) (209.85.215.43)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Dec 2010 21:15:08 +0000
Received: by ewy22 with SMTP id 22so8362704ewy.2
        for <general@hadoop.apache.org>; Mon, 06 Dec 2010 13:14:47 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=FjjavxEiEKtbCRdIK1c5nqAvYQkxMSJhTLX5PoxNJQg=;
        b=gbBKvsQU0f5dCIDyEg9g91OWMg300uZtEpJkryigwTJ42NTdndzjXCcGWi05glUeUw
         5ttxDEOHN2O06Kl8HNYV3OOV0B/p7z1iLMr4PleicD1Pv/ngbFXyL1kJyUMV44ZPjMjk
         Zmyqr6ISuV3X1sKcMbjc0vqxjFDQd2A2gEChE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=ZBDPzRcxitpqhuxU2Le0bZ5QS7mGzMGiGzxTrZRPcane+MWJwTBSbviC2cXL6c3moi
         CDCZYzkb9+qINPm5/gb3LBM3RAXP6pcvonBsRvlnDO1pvDLTklbQVK4l9McWeD8MzA6q
         tX//cMkRA4kpkZbn+QKIOaEDc2meaP4lRv8Ks=
Received: by 10.216.208.230 with SMTP id q80mr518964weo.103.1291670087617;
 Mon, 06 Dec 2010 13:14:47 -0800 (PST)
MIME-Version: 1.0
Received: by 10.216.19.193 with HTTP; Mon, 6 Dec 2010 13:14:27 -0800 (PST)
In-Reply-To: <0E66662F-E044-4738-8C46-EA0EBCE674E0@yahoo-inc.com>
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>
 <AANLkTik1F-DCc0GNrk0dEqzRp5f-MvOQSjsg5+Z9neXf@mail.gmail.com>
 <A32CDC9F-D6C6-4BA6-A44B-5FF206C1DF49@apache.org> <0E66662F-E044-4738-8C46-EA0EBCE674E0@yahoo-inc.com>
From: Tom White <tom.e.white@gmail.com>
Date: Mon, 6 Dec 2010 13:14:27 -0800
Message-ID: <AANLkTi=1+aA81QRCcAWTjpxoK5-iUhzSSY8fe1FqKRdY@mail.gmail.com>
Subject: Re: [VOTE] Direction for Hadoop development
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Mon, Dec 6, 2010 at 9:16 AM, Owen O'Malley <oom@yahoo-inc.com> wrote:
>
> All,
> =A0 We really need some guidance on the general direction for the project=
.
> Please comment and/or vote. If no one cares, then I'll probably commit it=
 to
> Yahoo's internal branch.
>

An alternative to this would be to create a project for
serializations. Other projects have successfully provided external
libraries for Hadoop, e.g. Pig, Hive, Elephant Bird, Plume, Cascading,
Mahout etc. This would make it possible for users to choose which
library they wanted to use, and allows updates to the library to be
driven by the library's release schedule, not Hadoop's.

Tom

From general-return-2481-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Dec 06 22:41:18 2010
Return-Path: <general-return-2481-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 80452 invoked from network); 6 Dec 2010 22:41:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Dec 2010 22:41:17 -0000
Received: (qmail 34743 invoked by uid 500); 6 Dec 2010 22:41:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34612 invoked by uid 500); 6 Dec 2010 22:41:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34604 invoked by uid 99); 6 Dec 2010 22:41:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Dec 2010 22:41:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 06 Dec 2010 22:41:16 +0000
Received: (qmail 80335 invoked by uid 99); 6 Dec 2010 22:40:55 -0000
Received: from localhost.apache.org (HELO mail-yw0-f48.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Dec 2010 22:40:55 +0000
Received: by ywo7 with SMTP id 7so176206ywo.35
        for <general@hadoop.apache.org>; Mon, 06 Dec 2010 14:40:54 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.206.143 with SMTP id fu15mr541048ibb.191.1291675254454;
 Mon, 06 Dec 2010 14:40:54 -0800 (PST)
Received: by 10.231.167.199 with HTTP; Mon, 6 Dec 2010 14:40:54 -0800 (PST)
In-Reply-To: <4CFD39BC.1080005@apache.org>
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>
	<4CF433D4.6090407@apache.org>
	<AANLkTineTnsShLoa5vO-X9UAp_DUMrHRjc2ry=i4qkXs@mail.gmail.com>
	<4CF66812.4080602@apache.org>
	<D414D20A-A909-46A9-AAA2-52742E6395CF@apache.org>
	<4CFD39BC.1080005@apache.org>
Date: Mon, 6 Dec 2010 14:40:54 -0800
Message-ID: <AANLkTinO_WpU3WSx6AZDN2tM7C=+Ja_hQ0n7qb_jLw2h@mail.gmail.com>
Subject: Re: [VOTE] Direction for Hadoop development
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Dec 6, 2010 at 11:30 AM, Doug Cutting <cutting@apache.org> wrote:
> On 12/01/2010 07:40 AM, Owen O'Malley wrote:
>>>
>>> This is unfortunately not the way Apache projects operate. Vetos are
>>> not overridden by a majority votes.
>>
>> This isn't overriding the veto. You've based your veto on changes in
>> project plan that have never been discussed or agreed to.
>
> Apache projects don't have project plans that are created by majority votes.

This reasoning is exactly opposite the facts at issue.

The veto of HADOOP-6685 was based, in part, on a component that the
patch proposed to modify. Due to an individual's project plan- one
that "does not support" extending that component- that change was not
allowed to go forward.

Asserting that the PMC does not create project plans by majority vote
is correct in some ways; the PMC does not plan out every feature and
pre-approve all work contributed in a release. This does not require
that its members remain collectively mute on the project's direction.
It is nonsense to assert that every PMC member has the right to block
work because it conflicts with their personal vision for Hadoop. That
doesn't scale. Instead, it makes authority so ambiguous and diffuse
that project direction defaults to the agenda of bullies.

This has been turned on its head. The procedural question is whether
individual votes establish project plans, not majority votes.

To avoid a pedantic, lawyerly debate on whether this aspect of the
veto was valid, its premise was raised as an issue for the community,
so it could reach consensus and define the scope of its project. This
approach was supposed to incite more constructive debate on an obvious
difference in how participants see Hadoop development over the long
term. So do you want to have that discussion, or do you want to bicker
about whether you'll be forced to accept the result? -C

On the vote: I'm +1 on supporting library/platform code in the Hadoop
project, particularly in MapReduce. Reducing MR to a distributed sort
implementation is not a direction I'm interested in.

From general-return-2482-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Dec 06 23:44:49 2010
Return-Path: <general-return-2482-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 13622 invoked from network); 6 Dec 2010 23:44:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Dec 2010 23:44:49 -0000
Received: (qmail 15656 invoked by uid 500); 6 Dec 2010 23:44:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15608 invoked by uid 500); 6 Dec 2010 23:44:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15600 invoked by uid 99); 6 Dec 2010 23:44:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Dec 2010 23:44:47 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [209.191.69.21] (HELO web30004.mail.mud.yahoo.com) (209.191.69.21)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 06 Dec 2010 23:44:40 +0000
Received: (qmail 14576 invoked by uid 60001); 6 Dec 2010 23:44:18 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1291679057; bh=cVId71CVy3v0qRubjPXTyl8HU3OKNhfIWnSDqegbQVo=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=ChQ2sXvIfqnjqLgs++2AoSi1yQHS5nRacmR9v9hVZt04Jff74OkqnPE2rJip5THpvVVOGqTgEf0J+h7O0Fe9+TML/XSrz+vXGPf+zF1OAEGzoyS+2oW6Il7DflNNpLjZSppG2AtarK8fKDoF7VzVib2wGVUkBvFGgVgJQFDn3YQ=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding;
  b=fGREJZVVqLXhpKNkiYiPDoso5ktZOHru4YdOWnut5pqAco0rjWsASGMySm9vgu1j15cDB1nEMXCzSeD5NvhEzHyv1W/Sqzfrwc2JUMHVy12gbOFR++rrI4P1VUOWGCEMc0vQKivSsmzsykNXPQ/Wfo+TLESS8/f1lKTyy8tijx0=;
Message-ID: <940252.13484.qm@web30004.mail.mud.yahoo.com>
X-YMail-OSG: JdOkZpAVM1k7SU6P4gJFQEmaQNraaEp0HdOVG8TxFs2H4dG
 hN9jw0POY_5o5D_QOpBB6xA602JEJ3OUUGxzL6XDoZZS5hhptVnU3oj3DQqu
 RTnlLteFY35LQCPcPvDCiIzrgDtyNE7tb82v2.grFS56XpbY0jHQO158a9aS
 GQK6zI2aXS0vmHSAXqMybuM6LgdwR6HV8JipxkUVuWWrOZKB68DdPrh.CicV
 3e3yXfPKlZBwAeZIHe8yUd5xn2qYyltdVJ78B6WX5QOS_LG1F2Ax857z5GKg
 hl6ZC3UgH2YvK9j6xzdPztb6f35E51KCbUYWPL.YkHL7kJn7Q4SsgvR9IGjd
 2ZEUl8f3c8yWM_LUFrX_AjmYMu34GJvIosFjMWRz0QrYc.BnRegEngVYIBSW
 vG00-
Received: from [209.131.62.113] by web30004.mail.mud.yahoo.com via HTTP; Mon, 06 Dec 2010 15:44:17 PST
X-Mailer: YahooMailClassic/11.4.20 YahooMailWebService/0.8.107.285259
Date: Mon, 6 Dec 2010 15:44:17 -0800 (PST)
From: Debabrata Biswas <ds_biswas@yahoo.com>
Subject: Re: distcp fails with ConnectException
To: general@hadoop.apache.org
In-Reply-To: <DA852C02397DBC4EBFAA1B7192FBD1506207CABCB0@miranda>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

[I use 'gridops' : you could use any valid queue name]. =0APlease try with =
this.=0Ahadoop distcp -Dmapred.job.queue.name=3Dgridops  -m 3 -i -ppgu hdfs=
://hadoop-A:8020/dir1/part-r-00000 hdfs://hadoop-A:8020/dir2/=0A=0AThanks,=
=0A-Deb=0A=0A--- On Mon, 12/6/10, Deepika Khera <Deepika.Khera@avg.com> wro=
te:=0A=0A> From: Deepika Khera <Deepika.Khera@avg.com>=0A> Subject: distcp =
fails with ConnectException=0A> To: "mapreduce-user@hadoop.apache.org" <map=
reduce-user@hadoop.apache.org>, "general@hadoop.apache.org" <general@hadoop=
.apache.org>=0A> Date: Monday, December 6, 2010, 9:50 AM=0A> I am trying to=
 run distcp between=0A> 2=A0 hdfs clusters and I am getting a ConnectExcept=
ion .=0A> =0A> The Command I am trying to run is of the form (considering=
=0A> clusters hadoop-A & hadoop-B):=0A> =0A> ./hadoop distcp -update=0A> hd=
fs://hadoop-A:8020/dira/part-r-00000=0A> hdfs://hadoop-B:8020/dirb/=0A> =0A=
> and I am running it on the destination cluster (hadoop-B).=0A> =0A> The s=
tacktrace for the exception is :=0A> =0A> Copy failed: java.net.ConnectExce=
ption: Call to=0A> hadoop-A/10.0.173.11:8020 failed on connection exception=
:=0A> java.net.ConnectException: Connection refused=0A> =A0 =A0 =A0 =A0 at=
=0A> org.apache.hadoop.ipc.Client.wrapException(Client.java:767)=0A> =A0 =
=A0 =A0 =A0 at=0A> org.apache.hadoop.ipc.Client.call(Client.java:743)=0A> =
=A0 =A0 =A0 =A0 at=0A> org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:22=
0)=0A> =A0 =A0 =A0 =A0 at=0A> $Proxy0.getProtocolVersion(Unknown Source)=0A=
> =A0 =A0 =A0 =A0 at=0A> org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)=
=0A> =A0 =A0 =A0 =A0 at=0A> org.apache.hadoop.hdfs.DFSClient.createRPCNamen=
ode(DFSClient.java:113)=0A> =A0 =A0 =A0 =A0 at=0A> org.apache.hadoop.hdfs.D=
FSClient.<init>(DFSClient.java:214)=0A> =A0 =A0 =A0 =A0 at=0A> org.apache.h=
adoop.hdfs.DFSClient.<init>(DFSClient.java:177)=0A> =A0 =A0 =A0 =A0 at=0A> =
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSyst=
em.java:82)=0A> =A0 =A0 =A0 =A0 at=0A> org.apache.hadoop.fs.FileSystem.crea=
teFileSystem(FileSystem.java:1378)=0A> =0A> =0A> I have tried ping and ssh =
in between these clusters and it=0A> is working fine. On both clusters name=
node is running as the=0A> same user.=0A> =0A> The strange part is if I try=
 the command on a single=0A> cluster (both source & destination are on same=
=0A> DFS-=A0 so its a simple copy), it still fails with the=0A> same except=
ion. So I mean I run the command below on cluster=0A> A.=0A> =0A> ./hadoop =
distcp -update=0A> hdfs://hadoop-A:8020/dir1/part-r-00000=0A> hdfs://hadoop=
-A:8020/dir2/=0A> =0A> Is there anything else I need to have to get the dis=
tcp=0A> working?=A0 Any hints on what I could check will be=0A> helpful.=0A=
> =0A> Thanks,=0A> Deepika=0A> =0A> =0A=0A=0A      

From general-return-2483-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Dec 06 23:45:36 2010
Return-Path: <general-return-2483-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14140 invoked from network); 6 Dec 2010 23:45:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Dec 2010 23:45:36 -0000
Received: (qmail 18559 invoked by uid 500); 6 Dec 2010 23:45:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18483 invoked by uid 500); 6 Dec 2010 23:45:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18444 invoked by uid 99); 6 Dec 2010 23:45:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Dec 2010 23:45:34 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 06 Dec 2010 23:45:32 +0000
Received: (qmail 13821 invoked by uid 99); 6 Dec 2010 23:45:10 -0000
Received: from localhost.apache.org (HELO [192.168.168.16]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Dec 2010 23:45:10 +0000
Message-ID: <4CFD7585.70803@apache.org>
Date: Mon, 06 Dec 2010 15:45:09 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Direction for Hadoop development
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>	<4CF433D4.6090407@apache.org>	<AANLkTineTnsShLoa5vO-X9UAp_DUMrHRjc2ry=i4qkXs@mail.gmail.com>	<4CF66812.4080602@apache.org>	<D414D20A-A909-46A9-AAA2-52742E6395CF@apache.org>	<4CFD39BC.1080005@apache.org> <AANLkTinO_WpU3WSx6AZDN2tM7C=+Ja_hQ0n7qb_jLw2h@mail.gmail.com>
In-Reply-To: <AANLkTinO_WpU3WSx6AZDN2tM7C=+Ja_hQ0n7qb_jLw2h@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 12/06/2010 02:40 PM, Chris Douglas wrote:
> It is nonsense to assert that every PMC member has the right to block
> work because it conflicts with their personal vision  [ ... ]

This is the way Apache projects operate.  It requires that folks listen 
to criticism and potentially accept compromises if they wish to make 
progress.  If folks cannot reach consensus in an area then that area 
will not make progress.

> On the vote: I'm +1 on supporting library/platform code in the Hadoop
> project, particularly in MapReduce. Reducing MR to a distributed sort
> implementation is not a direction I'm interested in.

I am interested in having this project primarily deliver a reliable, 
efficient MapReduce kernel implementation.  That's the core 
functionality that folks seek to not recreate.  The project should focus 
on a minimal, low-level MapReduce API for this kernel and permit other 
projects to build higher-level abstractions.

Doug

From general-return-2484-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 07 01:10:10 2010
Return-Path: <general-return-2484-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67226 invoked from network); 7 Dec 2010 01:10:10 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Dec 2010 01:10:10 -0000
Received: (qmail 99076 invoked by uid 500); 7 Dec 2010 01:10:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98908 invoked by uid 500); 7 Dec 2010 01:10:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98900 invoked by uid 99); 7 Dec 2010 01:10:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 01:10:08 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [208.97.132.119] (HELO homiemail-a67.g.dreamhost.com) (208.97.132.119)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 01:10:02 +0000
Received: from homiemail-a67.g.dreamhost.com (localhost [127.0.0.1])
	by homiemail-a67.g.dreamhost.com (Postfix) with ESMTP id C14B96D406D
	for <general@hadoop.apache.org>; Mon,  6 Dec 2010 17:09:41 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; q=dns; s=
	gbiv.com; b=Zjuq4Gh5SlqxDtsUt4zpjV/hLwT93WmNKxiAFqjzr21+BrIB/F/M
	+2brXXB8LZAJjIYUPxd/BIiPs1R7pJw/yXOiJdP+0oC3R3T0BV+M+SNJE50a+Oqj
	yyrj10DoJmha1HeG09IMNV30d0mvq5L3NxMoWutMg27O2aR8IrERuoo=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; s=gbiv.com;
	 bh=LgIkuB11d0FzrFSdRgOt1jimh9o=; b=CWuY41Zv84c3IfE97eSQel8IKTYG
	qW+wcEoaxdVWd0sNC3UXaUm8cwklcRr3ZKkMRHm9CfBCM+I9SaNt0MGen08NIsdb
	y2PnJ6wtyVv3zro5f0MkqadPBPZadtHh/GxMc6X00VczvCSLK6Fygcu7WiwnVPNw
	JW2WKbAlqbJqJXQ=
Received: from di-524.corp.day.com (wsip-98-189-13-228.oc.oc.cox.net [98.189.13.228])
	(Authenticated sender: fielding@gbiv.com)
	by homiemail-a67.g.dreamhost.com (Postfix) with ESMTPA id 7BCD16D405E
	for <general@hadoop.apache.org>; Mon,  6 Dec 2010 17:09:41 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [VOTE] Direction for Hadoop development
From: "Roy T. Fielding" <fielding@gbiv.com>
In-Reply-To: <4CFD7585.70803@apache.org>
Date: Mon, 6 Dec 2010 17:09:41 -0800
Content-Transfer-Encoding: quoted-printable
Message-Id: <5A920FAB-F76B-4EF7-8740-5698F99A1EC2@gbiv.com>
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>	<4CF433D4.6090407@apache.org>	<AANLkTineTnsShLoa5vO-X9UAp_DUMrHRjc2ry=i4qkXs@mail.gmail.com>	<4CF66812.4080602@apache.org>	<D414D20A-A909-46A9-AAA2-52742E6395CF@apache.org>	<4CFD39BC.1080005@apache.org> <AANLkTinO_WpU3WSx6AZDN2tM7C=+Ja_hQ0n7qb_jLw2h@mail.gmail.com> <4CFD7585.70803@apache.org>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

On Dec 6, 2010, at 3:45 PM, Doug Cutting wrote:
> On 12/06/2010 02:40 PM, Chris Douglas wrote:
>> It is nonsense to assert that every PMC member has the right to block
>> work because it conflicts with their personal vision  [ ... ]
>=20
> This is the way Apache projects operate.  It requires that folks =
listen to criticism and potentially accept compromises if they wish to =
make progress.  If folks cannot reach consensus in an area then that =
area will not make progress.

Generally speaking, vetoing extension interfaces without a compelling
technical reason is not the way Apache operates.  We make extensions
modular so that diverse collaborators can specialize according to
their own needs, not just your needs.

The compelling reason would be a measured performance impact or some
other objective degradation of the existing product that can be
evaluated by others as a cost/benefit tradeoff and perhaps compensated
by modifying the implementation.

>> On the vote: I'm +1 on supporting library/platform code in the Hadoop
>> project, particularly in MapReduce. Reducing MR to a distributed sort
>> implementation is not a direction I'm interested in.
>=20
> I am interested in having this project primarily deliver a reliable, =
efficient MapReduce kernel implementation.  That's the core =
functionality that folks seek to not recreate.  The project should focus =
on a minimal, low-level MapReduce API for this kernel and permit other =
projects to build higher-level abstractions.

That is something people can vote on.  Changes to the existing products,
including plans like the one Owen described, are subject to vote if =
anyone
disagrees with them.  They are also subject to veto if and only if they
are to be applied to the current release branch (or a released branch).

If a PMC member insists on making design opinion the sole basis of their
vetoes, then they are not collaborating with the rest of the PMC.  The
board will recommend that such a person be removed from the PMC so that
the majority can continue to develop the product in peace.  If there is
enough interest in a parallel line of development, then the board will
recommend splitting the PMC into two or more projects that can compete
on the merits of their own designs, with the existing product name
remaining with the majority.  Both recommendations are based on our
experience with Tomcat (which quickly solved the disagreement on its
own, once the choices were laid out, by allowing divergent designs on
separate major versions of the same product).

....Roy


From general-return-2485-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 07 02:11:29 2010
Return-Path: <general-return-2485-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 86130 invoked from network); 7 Dec 2010 02:11:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Dec 2010 02:11:28 -0000
Received: (qmail 34800 invoked by uid 500); 7 Dec 2010 02:11:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34749 invoked by uid 500); 7 Dec 2010 02:11:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34741 invoked by uid 99); 7 Dec 2010 02:11:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 02:11:27 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of amp@opendns.com designates 67.215.68.163 as permitted sender)
Received: from [67.215.68.163] (HELO mail.opendns.com) (67.215.68.163)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 02:11:19 +0000
Received: from Adams-Desktop.local ([67.215.69.42])
	(authenticated bits=0)
	by mail.opendns.com (8.14.3/8.14.3/Debian-5) with ESMTP id oB72Av6L009749
	(version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Tue, 7 Dec 2010 02:10:57 GMT
Message-ID: <4CFD97B1.5030607@opendns.com>
Date: Mon, 06 Dec 2010 18:10:57 -0800
From: Adam Phelps <amp@opendns.com>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Several problems after CDB3b2 to CDB3b3 upgrade
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

I'm testing a hadoop version upgrade on a prototype EC2 cluster, but 
while I've now gotten most of it up and running (well, HDFS and HBase at 
least) I'm hitting some odd problems getting our M/R jobs to run.

(I followed all the instructions at 
https://wiki.cloudera.com/display/DOC/Hadoop+Upgrade+from+CDH2+or+CDH3b2+to+CDH3b3 
as well as fixing a number of problems that came up in that process.)

They current problem I'm stuck on appears to be a classpath issue, but 
one I can't figure out.  When running a job I hit this error:

10/12/07 02:01:05 INFO mapred.JobClient: Task Id : 
attempt_201012062243_0009_m_000182_0, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: 
org.apache.hadoop.hbase.mapreduce.HFileOutputFormat
         at 
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:973)
         at 
org.apache.hadoop.mapreduce.JobContext.getOutputFormatClass(JobContext.java:236)
         at org.apache.hadoop.mapred.Task.initialize(Task.java:484)
         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:298)
         at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
         at java.security.AccessController.doPrivileged(Native Method)
         at javax.security.auth.Subject.doAs(Subject.java:396)
         at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
         at org.apache.hadoop.mapred.Child.main(Child.java:211)

We do use HFileOutputFormat in our M/R job, however as far as I can tell 
that should be handled by out existing classpath:

10/12/07 02:07:25 INFO zookeeper.ZooKeeper: Client 
environment:java.class.path=/usr/lib/hadoop-0.20/conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/lib/hadoop-0.20:/usr/lib 
hadoop-0.20/hadoop-
...
/jsp-2.1.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar::/usr/lib/hbase/hbase.jar:/usr/lib/hbase/conf:/usr/lib/zookeeper/zookeeper.jar 
/usr/lib/hbase/hbase.jar:/usr/lib/hbase/conf:/usr/lib/zookeeper/zookeeper.jar

It looks to me like HFileOutputFormat should be covered by that class path:

# jar tf /usr/lib/hbase/hbase.jar | grep HFileOutputFormat
org/apache/hadoop/hbase/mapreduce/HFileOutputFormat$WriterLength.class
org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.class
org/apache/hadoop/hbase/mapreduce/HFileOutputFormat$1.class

Any ideas here?

I have another similar issue, although with this one I have to assume 
that some package that was previously included with the base cloudera 
packages is no longer included:

Exception in thread "main" java.lang.NoClassDefFoundError: 
com/google/common/base/Function
         at 
org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.addDependencyJars(TableMapReduceUtil.java:247)
         at 
org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob(TableMapReduceUtil.java:81)

Thanks
- Adam

From general-return-2486-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 07 02:19:44 2010
Return-Path: <general-return-2486-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87783 invoked from network); 7 Dec 2010 02:19:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Dec 2010 02:19:43 -0000
Received: (qmail 39508 invoked by uid 500); 7 Dec 2010 02:19:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39455 invoked by uid 500); 7 Dec 2010 02:19:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39447 invoked by uid 99); 7 Dec 2010 02:19:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 02:19:42 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [208.97.132.145] (HELO homiemail-a66.g.dreamhost.com) (208.97.132.145)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 02:19:34 +0000
Received: from homiemail-a66.g.dreamhost.com (localhost [127.0.0.1])
	by homiemail-a66.g.dreamhost.com (Postfix) with ESMTP id A81B1350075
	for <general@hadoop.apache.org>; Mon,  6 Dec 2010 18:19:13 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; q=dns; s=
	gbiv.com; b=q56rJCuzhoYCdUzPIKg0peMGYdLU+ls0SkatKUY2kYEbJSfz9wPL
	NTcF+uy83zRvybpVvnX01JWZqsDNnEQ6cx42aAnfeBGnIzHAu/hiAeoaIfWAJVxs
	285S/f9sj+kfDbAJgyi9oKq38jDamf4ql/m/U/q+EnnFsv6EtigBT3c=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; s=gbiv.com;
	 bh=pbnmyRweemZ1iGoRvBZHtZODhMQ=; b=5BzqxXD0n0ufY6BX9EqVIrshfwAl
	qViUioEliu80nVdFbvzlmO7GEmf7n+0NrAtKXvSc5Vp4jhNAPeSyPZMw/fPT8T1N
	msCUDDfqfz/FQX1oTtL70X+gbiRhoft2dPhN6iRbnCRmHbR1X4pykgY40XbPQavP
	3YkUJzUwlCL9caI=
Received: from di-524.corp.day.com (wsip-98-189-13-228.oc.oc.cox.net [98.189.13.228])
	(Authenticated sender: fielding@gbiv.com)
	by homiemail-a66.g.dreamhost.com (Postfix) with ESMTPA id 89D89350063
	for <general@hadoop.apache.org>; Mon,  6 Dec 2010 18:19:13 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Several problems after CDB3b2 to CDB3b3 upgrade
From: "Roy T. Fielding" <fielding@gbiv.com>
In-Reply-To: <4CFD97B1.5030607@opendns.com>
Date: Mon, 6 Dec 2010 18:19:12 -0800
Content-Transfer-Encoding: 7bit
Message-Id: <CBB01202-7F79-4ABD-B6A3-22574FCE6A3D@gbiv.com>
References: <4CFD97B1.5030607@opendns.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

Questions about CDH are not appropriate for this mailing list.

....Roy

From general-return-2487-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 07 02:21:26 2010
Return-Path: <general-return-2487-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87979 invoked from network); 7 Dec 2010 02:21:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Dec 2010 02:21:26 -0000
Received: (qmail 41303 invoked by uid 500); 7 Dec 2010 02:21:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 41039 invoked by uid 500); 7 Dec 2010 02:21:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 41031 invoked by uid 99); 7 Dec 2010 02:21:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 02:21:24 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 02:21:19 +0000
Received: by iwn7 with SMTP id 7so506812iwn.35
        for <general@hadoop.apache.org>; Mon, 06 Dec 2010 18:20:58 -0800 (PST)
Received: by 10.231.35.11 with SMTP id n11mr6786586ibd.168.1291688458802; Mon,
 06 Dec 2010 18:20:58 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.154.18 with HTTP; Mon, 6 Dec 2010 18:20:38 -0800 (PST)
In-Reply-To: <4CFD97B1.5030607@opendns.com>
References: <4CFD97B1.5030607@opendns.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 6 Dec 2010 18:20:38 -0800
Message-ID: <AANLkTi=EV+yAbPjUcThBB+bKCM1Vpmr2wQfrC70MJ0JG@mail.gmail.com>
Subject: Re: Several problems after CDB3b2 to CDB3b3 upgrade
To: CDH Users <cdh-user@cloudera.org>
Content-Type: multipart/alternative; boundary=000325576e32bf57480496c8a759

--000325576e32bf57480496c8a759
Content-Type: text/plain; charset=ISO-8859-1

Hi Adam,

Please use the cdh-user list for Cloudera specific questions. (general now
BCC)

It seems to me that you aren't getting the hbase or guava jars on the
classpath of the Hadoop task. That's independent of whether they're on the
classpath of the program that submits the job.

You can likely use TableMapReduceUtil.addDependencyJars(job) to fix this.

Thanks
-Todd

On Mon, Dec 6, 2010 at 6:10 PM, Adam Phelps <amp@opendns.com> wrote:

> I'm testing a hadoop version upgrade on a prototype EC2 cluster, but while
> I've now gotten most of it up and running (well, HDFS and HBase at least)
> I'm hitting some odd problems getting our M/R jobs to run.
>
> (I followed all the instructions at
> https://wiki.cloudera.com/display/DOC/Hadoop+Upgrade+from+CDH2+or+CDH3b2+to+CDH3b3as well as fixing a number of problems that came up in that process.)
>
> They current problem I'm stuck on appears to be a classpath issue, but one
> I can't figure out.  When running a job I hit this error:
>
> 10/12/07 02:01:05 INFO mapred.JobClient: Task Id :
> attempt_201012062243_0009_m_000182_0, Status : FAILED
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat
>        at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:973)
>        at
> org.apache.hadoop.mapreduce.JobContext.getOutputFormatClass(JobContext.java:236)
>        at org.apache.hadoop.mapred.Task.initialize(Task.java:484)
>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:298)
>        at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at javax.security.auth.Subject.doAs(Subject.java:396)
>        at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063)
>        at org.apache.hadoop.mapred.Child.main(Child.java:211)
>
> We do use HFileOutputFormat in our M/R job, however as far as I can tell
> that should be handled by out existing classpath:
>
> 10/12/07 02:07:25 INFO zookeeper.ZooKeeper: Client
> environment:java.class.path=/usr/lib/hadoop-0.20/conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/lib/hadoop-0.20:/usr/lib
> hadoop-0.20/hadoop-
> ...
> /jsp-2.1.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar::/usr/lib/hbase/hbase.jar:/usr/lib/hbase/conf:/usr/lib/zookeeper/zookeeper.jar
> /usr/lib/hbase/hbase.jar:/usr/lib/hbase/conf:/usr/lib/zookeeper/zookeeper.jar
>
> It looks to me like HFileOutputFormat should be covered by that class path:
>
> # jar tf /usr/lib/hbase/hbase.jar | grep HFileOutputFormat
> org/apache/hadoop/hbase/mapreduce/HFileOutputFormat$WriterLength.class
> org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.class
> org/apache/hadoop/hbase/mapreduce/HFileOutputFormat$1.class
>
> Any ideas here?
>
> I have another similar issue, although with this one I have to assume that
> some package that was previously included with the base cloudera packages is
> no longer included:
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/google/common/base/Function
>        at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.addDependencyJars(TableMapReduceUtil.java:247)
>        at
> org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initTableMapperJob(TableMapReduceUtil.java:81)
>
> Thanks
> - Adam
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--000325576e32bf57480496c8a759--

From general-return-2488-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 07 02:38:23 2010
Return-Path: <general-return-2488-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 92540 invoked from network); 7 Dec 2010 02:38:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Dec 2010 02:38:22 -0000
Received: (qmail 58827 invoked by uid 500); 7 Dec 2010 02:38:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58510 invoked by uid 500); 7 Dec 2010 02:38:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58502 invoked by uid 99); 7 Dec 2010 02:38:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 02:38:20 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of amp@opendns.com designates 67.215.68.163 as permitted sender)
Received: from [67.215.68.163] (HELO mail.opendns.com) (67.215.68.163)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 02:38:11 +0000
Received: from Adams-Desktop.local ([67.215.69.42])
	(authenticated bits=0)
	by mail.opendns.com (8.14.3/8.14.3/Debian-5) with ESMTP id oB72bnvI014169
	(version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NO);
	Tue, 7 Dec 2010 02:37:49 GMT
Message-ID: <4CFD9DFD.8070107@opendns.com>
Date: Mon, 06 Dec 2010 18:37:49 -0800
From: Adam Phelps <amp@opendns.com>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6
MIME-Version: 1.0
To: general@hadoop.apache.org
CC: Todd Lipcon <todd@cloudera.com>, CDH Users <cdh-user@cloudera.org>
Subject: Re: Several problems after CDB3b2 to CDB3b3 upgrade
References: <4CFD97B1.5030607@opendns.com> <AANLkTi=EV+yAbPjUcThBB+bKCM1Vpmr2wQfrC70MJ0JG@mail.gmail.com>
In-Reply-To: <AANLkTi=EV+yAbPjUcThBB+bKCM1Vpmr2wQfrC70MJ0JG@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 12/6/10 6:20 PM, Todd Lipcon wrote:
> Hi Adam,
>
> Please use the cdh-user list for Cloudera specific questions. (general now
> BCC)

Ah, I'll get on that list now.  Sorry for posting to the wrong list.

> It seems to me that you aren't getting the hbase or guava jars on the
> classpath of the Hadoop task. That's independent of whether they're on the
> classpath of the program that submits the job.

Thanks, I'll take a look to see where those are specified.

- Adam

From general-return-2489-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 07 03:37:27 2010
Return-Path: <general-return-2489-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 13148 invoked from network); 7 Dec 2010 03:37:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Dec 2010 03:37:27 -0000
Received: (qmail 4351 invoked by uid 500); 7 Dec 2010 03:37:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4058 invoked by uid 500); 7 Dec 2010 03:37:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4050 invoked by uid 99); 7 Dec 2010 03:37:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 03:37:24 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.161.179] (HELO mail-gx0-f179.google.com) (209.85.161.179)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 03:37:20 +0000
Received: by gxk21 with SMTP id 21so7874915gxk.38
        for <general@hadoop.apache.org>; Mon, 06 Dec 2010 19:36:58 -0800 (PST)
MIME-Version: 1.0
Received: by 10.100.201.8 with SMTP id y8mr4530291anf.169.1291693018697; Mon,
 06 Dec 2010 19:36:58 -0800 (PST)
Received: by 10.236.109.129 with HTTP; Mon, 6 Dec 2010 19:36:58 -0800 (PST)
In-Reply-To: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>
Date: Mon, 6 Dec 2010 22:36:58 -0500
Message-ID: <AANLkTi=CDRWzCcz3DcP9aBBbzQA3ADwi33YL3HLu0FrQ@mail.gmail.com>
Subject: Re: [VOTE] Direction for Hadoop development
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I'm going to rather purposefully ignore larger questions like how the
ASF works or doesn't, veto usage, etc. I'm not well versed enough in
the Apache way to weigh in.

As someone who sees a lot of Hadoop clusters at many different
companies, I would like to see Hadoop's serialization system(s)
change. I think Hadoop should support interfaces to control
serialization plugin lifecycle and element serialization to / from an
abstract notion of a datum and bytes only. I would like to not mention
a serialization implementation by name in Hadoop proper, at all. A
single implementation to serve as a reference implementation makes
sense. To preserve backward compatibility and existing investment, it
makes sense for that to be Writable (whether we like it or not).
Additional implementations should be either "contrib" status (if
that's still an option) or externally managed (probably preferred due
to release cycle synchronization / update issues). The default
classpath should remain as free of mandatory external dependencies as
possible; library conflicts are still an extremely sore spot in Java
development at many sites I visit and forcing a large commercial
entity to use version X o something like Avro, Thrift, PB is almost a
non-starter for many.

If a PB / Thrift / Avro serialization implementation is part of
contrib or externally managed, it requires the user to understand this
dependency exists and manage the classpath. The precedent in my mind
is the scheduler situation; most folks run with either the cap or fair
schedulers but FIFO provides a default. If you opt to use one and it
comes with dependencies, that's your business. I think we can simplify
serialization plugin configuration via a classpath include system by
using something like run-parts or similar and the current
configuration system, but that's another issue.

In absence of an "opt in" serialization configuration pattern, we must
at least provide an "opt out." If a user uses thrift for their own MR
jobs internally, we shouldn't throw a monkey wrench into their life by
demanding it for core Hadoop. Provide them a means to de-configure
built in serialization impls and remove thrift from the classpath.

I'm a bit confused as to how this equates with sequence files being
deprecated or arrested. I tried to read HADOOP-6685 but there's a lot
of internal references and context I feel like I'm missing. Suffice it
to say, sequence files can *not* be broken for existing data for the
reasons everyone has stated. If we choose to focus development
elsewhere ("soft deprecate") or actively encourage users elsewhere
("@Deprecated") is an issue I think we can sever from this discussion.

tl;dr version:

- Don't break existing SequenceFiles.
- Serialization should be a richer interface to support plugin
lifecycle, serialize / deserialize only and be retrofitted using PB,
Avro, and Thrift as immediate consumer use cases. Serialization APIs
should be promoted to a(n officially) public, documented, API suited
to deal with modern serialization lib requirements.
- Common, HDFS, MR should contain as few mandatory external deps as
humanly possible because Java classloader semantics and a lack of
internal dep isolation is just kookoo for cocoa puffs. (Simplify it
and bring on our OSGI overlords.)
- We (non-committers / users / casual contributors) want only for
Hadoop to mature in features and stability, be an inviting community
to new potential contributors and users, and to be around for a long
time.

Regards, respect, and thanks to all.

On Mon, Nov 29, 2010 at 5:30 PM, Owen O'Malley <oom@yahoo-inc.com> wrote:
> All,
> =A0 Based on the discussion on HADOOP-6685, there is a pretty fundamental
> difference of opinion about how Hadoop should evolve. We need to figure o=
ut
> how the majority of the PMC wants the project to evolve to understand whi=
ch
> patches move us forward. Please vote whether you approve of the following
> direction. Clearly as the author, I'm +1.
>
> -- Owen
>
> Hadoop has always included library code so that users had a strong
> foundation to build their applications on without needing to continually
> reinvent the wheel. This combination of framework and powerful library co=
de
> is a common pattern for successful projects, such as Java, Lucene, etc.
> Toward that end, we need to continue to extend the Hadoop library code an=
d
> actively maintain it as the framework evolves. Continuing support for
> SequenceFile and TFile, which are both widely used is mandatory. The
> opposite pattern of implementing the framework and letting each distribut=
ion
> add the required libraries will lead to increased community fragmentation
> and vendor lock in.
>
> Hadoop's generic serialization framework had a lot of promise when it was
> introduced, but has been hampered by a lack of plugins other than Writabl=
es
> and Java serialization. Supporting a wide range of serializations nativel=
y
> in Hadoop will give the users new capabilities. Currently, to support Avr=
o
> or ProtoBuf objects mutually incompatible third party solutions are
> required. It benefits Hadoop to support them with a common framework that
> will support all of them. In particular, having easy, out of the box supp=
ort
> for Thrift, ProtoBufs, Avro, and our legacy serializations is a desired
> state.
>
> As a distributed system, there are many instances where Hadoop needs to
> serialize data. Many of those applications need a lightweight, versioned
> serialization framework like ProtocolBuffers or Thrift and using them is
> appropriate. Adding dependences on Thrift and ProtocolBuffers to the
> previous dependence on Avro is acceptable.



--=20
Eric Sammer
twitter: esammer
data: www.cloudera.com

From general-return-2490-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 07 08:14:38 2010
Return-Path: <general-return-2490-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42784 invoked from network); 7 Dec 2010 08:14:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Dec 2010 08:14:38 -0000
Received: (qmail 78600 invoked by uid 500); 7 Dec 2010 08:14:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78198 invoked by uid 500); 7 Dec 2010 08:14:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78190 invoked by uid 99); 7 Dec 2010 08:14:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 08:14:36 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.30.40] (HELO qmta04.emeryville.ca.mail.comcast.net) (76.96.30.40)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 08:14:27 +0000
Received: from omta02.emeryville.ca.mail.comcast.net ([76.96.30.19])
	by qmta04.emeryville.ca.mail.comcast.net with comcast
	id g8CG1f0010QkzPwA48E7bY; Tue, 07 Dec 2010 08:14:07 +0000
Received: from [10.0.0.13] ([209.131.62.115])
	by omta02.emeryville.ca.mail.comcast.net with comcast
	id g8Dy1f0052VBGtd8N8E1y5; Tue, 07 Dec 2010 08:14:05 +0000
Message-Id: <DA67105B-24BA-4EB1-8C23-D62B2E081164@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <AANLkTi=CDRWzCcz3DcP9aBBbzQA3ADwi33YL3HLu0FrQ@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Direction for Hadoop development
Date: Tue, 7 Dec 2010 00:13:58 -0800
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com> <AANLkTi=CDRWzCcz3DcP9aBBbzQA3ADwi33YL3HLu0FrQ@mail.gmail.com>
X-Mailer: Apple Mail (2.936)


On Dec 6, 2010, at 7:36 PM, Eric Sammer wrote:

Eric,
    Since this is mostly technical, it probably should be on the  
h-6685 jira instead of general@hadoop.

> I think Hadoop should support interfaces to control
> serialization plugin lifecycle and element serialization to / from an
> abstract notion of a datum and bytes only.

The core of my h-6685 patch updates the API to replace the typename  
with a serialization name and serialization specific metadata. That  
metadata is a set of bytes that are defined by the serialization. The  
typename alone is insufficient for Avro and having additional metadata  
will be useful for the other serializations as well.

Doug suggested that I add a user-friendly pair of methods and I did.  
While they are redundant, the set of serializations isn't expected to  
be large and therefore the extra code isn't much.

> I would like to not mention
> a serialization implementation by name in Hadoop proper, at all.

My patch removes some of the lingering references to Writables in  
SequenceFile, MapFile, etc. and moves them over the generic  
serialization API. The framework will likely continue depend on  
whichever serialization is used for RPC. Currently that is Writables,  
but will likely transition to either Avro or ProtoBuf in the coming  
year.

> A
> single implementation to serve as a reference implementation makes
> sense.

A critical part of Hadoop's usability comes from its framework  
combined with library code that allows users to get the desired  
functionality without writing it themselves. Sure, it is easy to write  
a hash table yourself, but it is far easier to use the one bundled  
with Java.

> The default
> classpath should remain as free of mandatory external dependencies as
> possible; library conflicts are still an extremely sore spot in Java
> development at many sites I visit and forcing a large commercial
> entity to use version X o something like Avro, Thrift, PB is almost a
> non-starter for many.

I discussed this problem in the jira, but either the MapReduce user is  
using the X library or doesn't care the version of X. If they are  
using it, it is far more convenient to have the serialization on the  
classpath. There is a missing feature that we need to address to put  
the user's files ahead of the system ones. I'll file a jira for that.

It might also make sense for us to shade some of our dependencies, but  
that is a much bigger issue and is far from clear cut.

> If a PB / Thrift / Avro serialization implementation is part of
> contrib or externally managed, it requires the user to understand this
> dependency exists and manage the classpath.

The goal is to make Hadoop useful out of the box. If we make it so  
that Hadoop is only useful once it is bundled with 15 other projects,  
that is good for people who sell distributions that include Hadoop,  
but not for the project.

> I think we can simplify
> serialization plugin configuration via a classpath include system by
> using something like run-parts or similar and the current
> configuration system, but that's another issue.

The current patch loads the serialization plugins based on the  
configuration. If you don't want to support thrift, don't configure  
it. The same holds true of the other serializations, even writable.

> I'm a bit confused as to how this equates with sequence files being
> deprecated or arrested.

Doug vetoed my patch partially based on his assertion that  
SequenceFiles should be deprecated and that Hadoop should just be the  
framework with no library code.

>  If we choose to focus development
> elsewhere ("soft deprecate") or actively encourage users elsewhere
> ("@Deprecated") is an issue I think we can sever from this discussion.

At this point the PMC has supported continuing to invest in developing  
SequenceFiles.

> - Don't break existing SequenceFiles.

That goes without saying, everyone has petabytes of data in them.

> - Common, HDFS, MR should contain as few mandatory external deps as
> humanly possible because Java classloader semantics and a lack of
> internal dep isolation is just kookoo for cocoa puffs. (Simplify it
> and bring on our OSGI overlords.)

That is a much bigger discussion that we should probably have. There  
are costs on both sides in terms of debugging and understandability.  
In particular, in most cases we are much better off using a library  
that has the right functionality that re-implementing it ourselves.

> - We (non-committers / users / casual contributors) want only for
> Hadoop to mature in features and stability, be an inviting community
> to new potential contributors and users, and to be around for a long
> time.

I want that too.

-- Owen



From general-return-2491-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 07 10:24:09 2010
Return-Path: <general-return-2491-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94672 invoked from network); 7 Dec 2010 10:24:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Dec 2010 10:24:08 -0000
Received: (qmail 15137 invoked by uid 500); 7 Dec 2010 10:24:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14870 invoked by uid 500); 7 Dec 2010 10:24:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14862 invoked by uid 99); 7 Dec 2010 10:24:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 10:24:06 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 10:24:01 +0000
Received: by yxm8 with SMTP id 8so7697363yxm.35
        for <general@hadoop.apache.org>; Tue, 07 Dec 2010 02:23:40 -0800 (PST)
MIME-Version: 1.0
Received: by 10.42.73.136 with SMTP id s8mr1213822icj.449.1291717419631; Tue,
 07 Dec 2010 02:23:39 -0800 (PST)
Received: by 10.42.169.10 with HTTP; Tue, 7 Dec 2010 02:23:39 -0800 (PST)
In-Reply-To: <DA67105B-24BA-4EB1-8C23-D62B2E081164@apache.org>
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>
	<AANLkTi=CDRWzCcz3DcP9aBBbzQA3ADwi33YL3HLu0FrQ@mail.gmail.com>
	<DA67105B-24BA-4EB1-8C23-D62B2E081164@apache.org>
Date: Tue, 7 Dec 2010 02:23:39 -0800
Message-ID: <AANLkTi=Xo=XRK3X+_Go7xjoSAQU-vxyGOoTk=ftUxi+_@mail.gmail.com>
Subject: Re: [VOTE] Direction for Hadoop development
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba6e867ef283010496cf65de

--90e6ba6e867ef283010496cf65de
Content-Type: text/plain; charset=ISO-8859-1

>
> A critical part of Hadoop's usability comes from its framework combined
> with library code that allows users to get the desired functionality without
> writing it themselves.
>


> The goal is to make Hadoop useful out of the box.
>

To the best of my knowledge, Owen, your organization requires users to
petition a committee before writing MapReduce jobs. At Facebook, the vast
majority of jobs are submitted via Hive. Our customers at Cloudera primarily
consume MapReduce through Pig, Hive, and other high-level tools.

Users of Hadoop have moved beyond MapReduce. The community would be far
better served by a compact, reliable, and efficient kernel. That's the
project direction Doug has suggested for MapReduce, and it's one that Eric
and Tom have supported. I also support this direction for the project.

We're clearly having a hard time, as a community, agreeing on standards for
library code. We've also shipped updates to the framework without updating
the library code, seriously damaging the usability of the project. In this
discussion, we're prioritizing the rapidly shrinking proportion of users of
MapReduce library code in favor of the far larger community of consumers of
the framework.

Arun recently asked on Quora about issues that users face with Hadoop
MapReduce: http://qr.ae/pPNK. There are currently five issues brought up
there, with 19 votes for those issues; none of them are addressed directly
by this extended debate.

I'd be ecstatic to see this discussion result in moving the file formats,
input and output formats, and other library code out to a separate Apache
project or Github where they can evolve rapidly based on user needs, so that
the MapReduce project can begin to address some of the outstanding issues
with the framework itself.

HDFS, HBase, Hive, Pig, Oozie, and other Hadoop-related projects continue to
make forward progress at a remarkable rate; I'd like to see MapReduce return
to health as well. Clearing away these major sources of conflict seems like
one promising path forward.

So, I'm not on the PMC, but I'm -1 on the proposed vote.

--90e6ba6e867ef283010496cf65de--

From general-return-2492-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 07 11:28:10 2010
Return-Path: <general-return-2492-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23180 invoked from network); 7 Dec 2010 11:28:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Dec 2010 11:28:09 -0000
Received: (qmail 26409 invoked by uid 500); 7 Dec 2010 11:28:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26105 invoked by uid 500); 7 Dec 2010 11:28:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26097 invoked by uid 99); 7 Dec 2010 11:28:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 11:28:07 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of shv.hadoop@gmail.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 11:28:02 +0000
Received: by gyf1 with SMTP id 1so7753971gyf.35
        for <general@hadoop.apache.org>; Tue, 07 Dec 2010 03:27:41 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=XMWalalq89tb7GeePxLAVN/aCdc2klD6jAtpnhf/6jg=;
        b=WS8ciy1aVSx31qn9nZEmDq8iRSFnJHPfW3F/g5OKaJvHFQubOto7ZdZ8qXKBpdgtk/
         JWW6oFaEpkZ51GYiAZgXRk8ORPonZi0H5Za/jHsjEGRRpiuvz6evjQ/HZcPgDxPqJfUK
         6OosMmCR3PyC+xOwlV4Lq/R+BQvg4OvzuLmyo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=VmysPttS27jxQh4iqDIvwzf1E0NKP0pxPnuW7DTFiDm/aOGZiKkDLjvEV2qPaFywup
         Iwplyhfp52GkWLF2zKWMONapd8kX+PGPbsHwx7hY2SP3PC0e7cm6ibUAwby6aYl83meX
         IEy6hq+2s+/xUWBBwRG8S2mc//m8kIKW874NE=
MIME-Version: 1.0
Received: by 10.150.203.15 with SMTP id a15mr1634959ybg.136.1291721261626;
 Tue, 07 Dec 2010 03:27:41 -0800 (PST)
Received: by 10.236.109.15 with HTTP; Tue, 7 Dec 2010 03:27:41 -0800 (PST)
In-Reply-To: <0E66662F-E044-4738-8C46-EA0EBCE674E0@yahoo-inc.com>
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>
	<AANLkTik1F-DCc0GNrk0dEqzRp5f-MvOQSjsg5+Z9neXf@mail.gmail.com>
	<A32CDC9F-D6C6-4BA6-A44B-5FF206C1DF49@apache.org>
	<0E66662F-E044-4738-8C46-EA0EBCE674E0@yahoo-inc.com>
Date: Tue, 7 Dec 2010 03:27:41 -0800
Message-ID: <AANLkTin43+WOUSRKVGMMb-iFZ5q64=wFvSw=DweszWis@mail.gmail.com>
Subject: Re: [VOTE] Direction for Hadoop development
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd3755af2b48e0496d04a85

--000e0cd3755af2b48e0496d04a85
Content-Type: text/plain; charset=ISO-8859-1

It really takes time to understand the issue. I will spend more time reading
through it.

So far I feel that we need to distinguish between
a) issues that define the general direction for the project, and
b) the specifics of the implementation proposed by Owen, including decisions
induced by that implementation.

The main contradictory issue on which Owen and Doug disagree (other people
as well) is whether
Hadoop should support multiple serializations or be based on one designated
serialization.
This is a defining general direction a-issue. I believe this is vote-able.

The question of introducing dependency on ProtoBuf is a b-issue, as it can
be implemented differently.
Say with "pluggable" APIs as Tom proposed. This is probably a consensus-type
issue.

Looks to me if we decide on multiple vs designated serializations, some
b-issues may
be automatically ruled out or in.

Thanks,
--Konstantin

P.S. We used to have a tradition of presenting design documents before
introducing such big changes.
I believe a discussion of a design doc would have reduced tensions we face
now.



On Mon, Dec 6, 2010 at 9:16 AM, Owen O'Malley <oom@yahoo-inc.com> wrote:

>
> On Dec 1, 2010, at 11:11 AM, Owen O'Malley wrote:
>
> All,
>   We really need some guidance on the general direction for the project.
> Please comment and/or vote. If no one cares, then I'll probably commit it to
> Yahoo's internal branch.
>
> -- Owen
>
>  The question is how the Hadoop project wants to move forward.
>>
>> It was motivated by Doug's veto of HADOOP-6685, which was based on his
>> personal decisions about how the project should go forward and not on
>> anything that had been decided by the PMC.
>>
>> These decisions are much more important to MapReduce, which is a
>> framework, than HDFS which is a client/server model.
>>
>> 1. Should Hadoop include a user-facing library of useful code?
>>
>> There has been a suggestion that user-facing library code, such as
>> SequenceFile, TFile, DistCp, etc. should be deprecated and that Hadoop
>> should allow third party projects like Avro to supply the user-facing
>> library code that makes Hadoop usable. I think it is critical that we keep
>> those components as part of Hadoop and extend them as the framework evolves.
>> Users depend heavily on SequenceFile for storing their data in Hadoop and
>> they should not  be deprecated as Doug has suggested.
>>
>> 2. Should MapReduce support non-Writables through the pipeline out of the
>> box?
>>
>> There has also been a discussion about whether we should support
>> non-Writables natively. There is already library code in Avro that lets
>> users use Avro types in a custom MapReduce API. A general MapReduce API that
>> encompasses all of the serialization frameworks and does not lock users into
>> a particular one is much more powerful.
>>
>> Furthermore, making it convenient for the users, by including the plugins
>> in the default configuration and class path, will enable the use of Avro,
>> Thrift and ProtoBuf objects by people who would rather not focus on
>> serialization. Avro and Writables should not be the only first class
>> serializations that Hadoop supports by default.
>>
>> 3. Should a framework dependency on ProtoBuf be allowed?
>>
>> Doug has added several framework dependences on Avro. The question is
>> whether it is acceptable to use the ProtoBuf library in the framework. Avro
>> is good for uses where there are a lot of objects of the same type. ProtoBuf
>> is better for small number of objects. The question is whether Avro, JSON,
>> and XML should be the only serialization libraries that are acceptable to
>> use in the framework.
>>
>
>

--000e0cd3755af2b48e0496d04a85--

From general-return-2493-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 07 15:56:37 2010
Return-Path: <general-return-2493-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29620 invoked from network); 7 Dec 2010 15:56:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Dec 2010 15:56:28 -0000
Received: (qmail 90546 invoked by uid 500); 7 Dec 2010 15:56:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90291 invoked by uid 500); 7 Dec 2010 15:56:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90282 invoked by uid 99); 7 Dec 2010 15:56:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 15:56:25 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 15:56:19 +0000
Received: from [192.168.1.71] (snvvpn2-10-73-152-c36.hq.corp.yahoo.com [10.73.152.36])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id oB7FtXn0048762
	for <general@hadoop.apache.org>; Tue, 7 Dec 2010 07:55:33 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1291737333;
	bh=uZk5t0oHFwQR59LYpm+uF4vz2p/xLDnq6p87Rm1fcLo=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:Mime-Version:Subject:
	 Date:References;
	b=Rd8XYnYWwTGT5PXlWBTxXy+o0U7FSmVU+7Blyh6xHPZM8IyRGFf/8p35fqd5Ixyth
	 Jy7OB/cqpLIHtDLlnApfesKrHIMvfUDgzmeL0aTJZSW7QQ4vmXhNLvZytlNE6giJ7m
	 SLknrQ/dHrSVIu/zaI2Z+A3Q+79eG5OsVu77swSM=
Message-Id: <3DADA469-AC41-4D4C-97F1-FA13077A3B24@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <AANLkTi=CDRWzCcz3DcP9aBBbzQA3ADwi33YL3HLu0FrQ@mail.gmail.com>
Content-Type: multipart/alternative; boundary=Apple-Mail-323--102479568
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Direction for Hadoop development
Date: Tue, 7 Dec 2010 07:55:32 -0800
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com> <AANLkTi=CDRWzCcz3DcP9aBBbzQA3ADwi33YL3HLu0FrQ@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-323--102479568
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


On Dec 6, 2010, at 7:36 PM, Eric Sammer wrote:

> I'm a bit confused as to how this equates with sequence files being
> deprecated or arrested. I tried to read HADOOP-6685 but there's a lot
> of internal references and context I feel like I'm missing. Suffice it
> to say, sequence files can *not* be broken for existing data for the
> reasons everyone has stated. If we choose to focus development
> elsewhere ("soft deprecate") or actively encourage users elsewhere
> ("@Deprecated") is an issue I think we can sever from this discussion.

I'm surprised that your are confused.

http://s.apache.org/h6685-veto

Doug is very clear that he is vetoing the patch based on 2 reasons:
a) dependency on PB
b) extension to SequenceFile

a) is technical, and we can debate about it.

b) isn't. It's his 'vision' for the project, a vision which hasn't  
been ratified by the PMC.

Arun
--Apple-Mail-323--102479568--

From general-return-2494-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 07 16:07:06 2010
Return-Path: <general-return-2494-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43498 invoked from network); 7 Dec 2010 16:07:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Dec 2010 16:07:05 -0000
Received: (qmail 10133 invoked by uid 500); 7 Dec 2010 16:07:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 9960 invoked by uid 500); 7 Dec 2010 16:07:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 9952 invoked by uid 99); 7 Dec 2010 16:07:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 16:07:03 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jaybooth@gmail.com designates 209.85.216.176 as permitted sender)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 16:06:57 +0000
Received: by qyk10 with SMTP id 10so68438qyk.14
        for <general@hadoop.apache.org>; Tue, 07 Dec 2010 08:06:37 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=U6KuneBJIdd1WSiwgRI7Trk4K9KpxJAaRfdINclZyBk=;
        b=XKpmwkSOXYF4emc4PKP+9wEXHh4GCkqgqQh7ZxkvN1XcceMeo5FMnRC4YkHveZwM23
         CCvtXgpd7Cy33qXuXRG8YYsG7UdZABv2BOiBAaPBI380ICmsPaKtImQVnp5jj7Cxf/I0
         pk9Srj8rMjohzpRi7adH9e+XlyiZdhMOIjEqI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=rRGXcXHg03vWK6gWBdoa4UY9tzyk3N5qn02xYN05ZtreGahgqQL+vj0TcpbrECRCSj
         zYdEqRJfnPUxZhj39Zt0lM4ubbS5l+PhZFoGEabBRme1O3ocso8gf6iKmFSpDci5l5N6
         rg2M+vxu4Nx6hg0d7rbtnJfEoQNYVFzAMfseA=
MIME-Version: 1.0
Received: by 10.229.183.66 with SMTP id cf2mr5935705qcb.57.1291737996725; Tue,
 07 Dec 2010 08:06:36 -0800 (PST)
Received: by 10.220.159.136 with HTTP; Tue, 7 Dec 2010 08:06:36 -0800 (PST)
In-Reply-To: <3DADA469-AC41-4D4C-97F1-FA13077A3B24@yahoo-inc.com>
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>
	<AANLkTi=CDRWzCcz3DcP9aBBbzQA3ADwi33YL3HLu0FrQ@mail.gmail.com>
	<3DADA469-AC41-4D4C-97F1-FA13077A3B24@yahoo-inc.com>
Date: Tue, 7 Dec 2010 11:06:36 -0500
Message-ID: <AANLkTimWRnKN6tjG+=bUFXNJXrTZ+KySUF=0V0CsdweC@mail.gmail.com>
Subject: Re: [VOTE] Direction for Hadoop development
From: Jay Booth <jaybooth@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163646dace700ef70496d430de

--00163646dace700ef70496d430de
Content-Type: text/plain; charset=ISO-8859-1

a)  On the PB dependency..  can't we just use JSON and call it a day?  I
mean, we're gonna have a new dependency so that we can encode a single
tuple?  That doesn't even make engineering sense, let alone that the choice
of PB looks like a deliberate decision to try and tweak Doug's nose, whether
that was the intention or not.  Even if you could make a case for some very
minor benefit of using PB instead of one of the 3 serialization methods
already on the classpath, it's hard to see why it's worth going to the mat
over it.  And again, as a user, every additional classpath element in Hadoop
is a potential future conflict that I'll have to sort out for some
non-exciting business process I'm writing.

b)  Agreeing with Eric..  backwards compatibility is essential for sequence
file.  It seems to me that past a certain point, it's easier to just make a
new file format rather than cramming further functionality and
backwards-compatibility layers into the SequenceFile class, but as long as
it's backward compatible then I'm sure people will be fine.


On Tue, Dec 7, 2010 at 10:55 AM, Arun C Murthy <acm@yahoo-inc.com> wrote:

>
> On Dec 6, 2010, at 7:36 PM, Eric Sammer wrote:
>
>  I'm a bit confused as to how this equates with sequence files being
>> deprecated or arrested. I tried to read HADOOP-6685 but there's a lot
>> of internal references and context I feel like I'm missing. Suffice it
>> to say, sequence files can *not* be broken for existing data for the
>> reasons everyone has stated. If we choose to focus development
>> elsewhere ("soft deprecate") or actively encourage users elsewhere
>> ("@Deprecated") is an issue I think we can sever from this discussion.
>>
>
> I'm surprised that your are confused.
>
> http://s.apache.org/h6685-veto
>
> Doug is very clear that he is vetoing the patch based on 2 reasons:
> a) dependency on PB
> b) extension to SequenceFile
>
> a) is technical, and we can debate about it.
>
> b) isn't. It's his 'vision' for the project, a vision which hasn't been
> ratified by the PMC.
>
> Arun

--00163646dace700ef70496d430de--

From general-return-2495-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 07 16:13:33 2010
Return-Path: <general-return-2495-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46049 invoked from network); 7 Dec 2010 16:13:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Dec 2010 16:13:33 -0000
Received: (qmail 23078 invoked by uid 500); 7 Dec 2010 16:13:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22242 invoked by uid 500); 7 Dec 2010 16:13:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21929 invoked by uid 99); 7 Dec 2010 16:13:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 16:13:29 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 16:13:24 +0000
Received: from [192.168.1.71] (snvvpn2-10-73-152-c36.hq.corp.yahoo.com [10.73.152.36])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id oB7GCYWX053966
	for <general@hadoop.apache.org>; Tue, 7 Dec 2010 08:12:34 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1291738354;
	bh=rmzsgHpVNcX2fRJ/6OpmPNIC3E88FMtFftM8Xmk3XHk=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=skmNsckJf9xT2vNBEUEOC6qUUOJwgWE3hdYBim5Y24s0nxX6cpLesP3eZdShKtDcM
	 2d4orBA5UmZNmwdotsdlrvr8cBTsKN46tvuP2RLe+UMRUdEhV2FTsWvua8/OKKaGU3
	 BNQxciX17bFBOLicmnLNoN7G7hp3uZ2mh1pqb/rg=
Message-Id: <2E0EE1F3-F322-4EDA-95D3-47FE97B34383@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <AANLkTi=Xo=XRK3X+_Go7xjoSAQU-vxyGOoTk=ftUxi+_@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Direction for Hadoop development
Date: Tue, 7 Dec 2010 08:12:33 -0800
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com> <AANLkTi=CDRWzCcz3DcP9aBBbzQA3ADwi33YL3HLu0FrQ@mail.gmail.com> <DA67105B-24BA-4EB1-8C23-D62B2E081164@apache.org> <AANLkTi=Xo=XRK3X+_Go7xjoSAQU-vxyGOoTk=ftUxi+_@mail.gmail.com>
X-Mailer: Apple Mail (2.936)


On Dec 7, 2010, at 2:23 AM, Jeff Hammerbacher wrote:

> To the best of my knowledge, Owen, your organization requires users to
> petition a committee before writing MapReduce jobs.

I'd appreciate if we could keep the discussion technical and did not  
resort to snide comments. Thank you.

> Users of Hadoop have moved beyond MapReduce. The community would be  
> far
> better served by a compact, reliable, and efficient kernel. That's the
> project direction Doug has suggested for MapReduce, and it's one  
> that Eric
> and Tom have supported. I also support this direction for the project.
>

This is a great discussion to have, if Doug could start it, rather  
than put forward his word as the law.

However, this is not germane to the discussion at hand.

The discussion at hand is simple: Doug has vetoed this patch for 2  
reasons:
a) dependency on PB
b) extension to SequenceFile

a) is technical, b) isn't. This discussion is about b).

> I'd be ecstatic to see this discussion result in moving the file  
> formats,
> input and output formats, and other library code out to a separate  
> Apache
> project or Github where they can evolve rapidly based on user needs,  
> so that
> the MapReduce project can begin to address some of the outstanding  
> issues
> with the framework itself.

Again, no one is proposing new file formats here. SequenceFile is an  
important file format for several reasons:
>

- It's been bundled with Hadoop for nearly 5 years now
- Several users store petabytes of data on it

Blocking extensions to SequenceFile is unreasonable as has been noted  
by several folks, there is no *technical* reason to do that.

People are welcome to start any number of file-formats and input/ 
output libraries either in Apache or outside, no one is proposing  
otherwise.

Arun




From general-return-2496-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 07 16:47:01 2010
Return-Path: <general-return-2496-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60683 invoked from network); 7 Dec 2010 16:47:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Dec 2010 16:47:00 -0000
Received: (qmail 79219 invoked by uid 500); 7 Dec 2010 16:46:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78860 invoked by uid 500); 7 Dec 2010 16:46:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78845 invoked by uid 99); 7 Dec 2010 16:46:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 16:46:55 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 16:46:50 +0000
Received: from [192.168.1.71] (snvvpn2-10-73-152-c36.hq.corp.yahoo.com [10.73.152.36])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id oB7GjX2j057786
	for <general@hadoop.apache.org>; Tue, 7 Dec 2010 08:45:33 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1291740333;
	bh=rGeZfZ60myqjHn+5D/9wc8L11wR8XTeSKM6Im2FPaEo=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=YAoKJ93cdnjlAzjpiWVWfZv6j8OFygDiOU2CyVMrehzpOL96cf3eD+FUDKtPIzzU4
	 Leb2YU7PeYe3w0nmPcYpT5FU6TYwJaD6Cy+nH9bJl7J6eUBpbpSDF9IrlLU+irUZwc
	 sd5+x9fzMUINhWW4xFzyyXq/6FmjkVIV8iosUHds=
Message-Id: <CEB3A0DE-A110-4DD7-BF6F-F739016DA7F4@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <5A920FAB-F76B-4EF7-8740-5698F99A1EC2@gbiv.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Direction for Hadoop development
Date: Tue, 7 Dec 2010 08:45:32 -0800
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com> <4CF433D4.6090407@apache.org> <AANLkTineTnsShLoa5vO-X9UAp_DUMrHRjc2ry=i4qkXs@mail.gmail.com> <4CF66812.4080602@apache.org> <D414D20A-A909-46A9-AAA2-52742E6395CF@apache.org> <4CFD39BC.1080005@apache.org> <AANLkTinO_WpU3WSx6AZDN2tM7C=+Ja_hQ0n7qb_jLw2h@mail.gmail.com> <4CFD7585.70803@apache.org> <5A920FAB-F76B-4EF7-8740-5698F99A1EC2@gbiv.com>
X-Mailer: Apple Mail (2.936)


On Dec 6, 2010, at 5:09 PM, Roy T. Fielding wrote:

> On Dec 6, 2010, at 3:45 PM, Doug Cutting wrote:
>> On 12/06/2010 02:40 PM, Chris Douglas wrote:
>>> It is nonsense to assert that every PMC member has the right to  
>>> block
>>> work because it conflicts with their personal vision  [ ... ]
>>
>> This is the way Apache projects operate.  It requires that folks  
>> listen to criticism and potentially accept compromises if they wish  
>> to make progress.  If folks cannot reach consensus in an area then  
>> that area will not make progress.
>
> Generally speaking, vetoing extension interfaces without a compelling
> technical reason is not the way Apache operates.  We make extensions
> modular so that diverse collaborators can specialize according to
> their own needs, not just your needs.


Thanks for the clarifications Roy.

Doug, do you wish to retract your veto on extensions to SequenceFile?

We can continue discussions on the rest of the technical merits of the  
jira.

thanks,
Arun

From general-return-2497-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 07 17:18:56 2010
Return-Path: <general-return-2497-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 80383 invoked from network); 7 Dec 2010 17:18:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Dec 2010 17:18:56 -0000
Received: (qmail 27977 invoked by uid 500); 7 Dec 2010 17:18:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27892 invoked by uid 500); 7 Dec 2010 17:18:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27884 invoked by uid 99); 7 Dec 2010 17:18:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 17:18:54 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 07 Dec 2010 17:18:53 +0000
Received: (qmail 80115 invoked by uid 99); 7 Dec 2010 17:18:33 -0000
Received: from localhost.apache.org (HELO [192.168.168.16]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 17:18:33 +0000
Message-ID: <4CFE6C68.7000503@apache.org>
Date: Tue, 07 Dec 2010 09:18:32 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Direction for Hadoop development
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>	<4CF433D4.6090407@apache.org>	<AANLkTineTnsShLoa5vO-X9UAp_DUMrHRjc2ry=i4qkXs@mail.gmail.com>	<4CF66812.4080602@apache.org>	<D414D20A-A909-46A9-AAA2-52742E6395CF@apache.org>	<4CFD39BC.1080005@apache.org> <AANLkTinO_WpU3WSx6AZDN2tM7C=+Ja_hQ0n7qb_jLw2h@mail.gmail.com> <4CFD7585.70803@apache.org> <5A920FAB-F76B-4EF7-8740-5698F99A1EC2@gbiv.com>
In-Reply-To: <5A920FAB-F76B-4EF7-8740-5698F99A1EC2@gbiv.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 12/06/2010 05:09 PM, Roy T. Fielding wrote:
> Generally speaking, vetoing extension interfaces without a compelling
> technical reason is not the way Apache operates.  We make extensions
> modular so that diverse collaborators can specialize according to
> their own needs, not just your needs.

Roy, thanks for your thoughts here.

I have not intended to veto an extension mechanism.  In this case, we 
already have an extension mechanism.  The proposal is to change the 
extension mechanism incompatibly with unclear benefits, add 
implementations of several extensions to the kernel, and incompatibly 
change a widely-used file format.  In general I support improving 
extension mechanisms, but oppose gratuitous changes to file formats and 
the inclusion of new user-level functionality in the kernel.  I'd like 
the issue to focus solely on the extension mechanism to clarify the 
discussion, not on adding extensions to the kernel or file formats.

Tom long ago provided patches showing how the existing configuration 
system can provide equivalent extension implementations outside of the 
kernel with no incompatible changes.  (MAPREDUCE-376 and MAPREDUCE-377)

> Changes to the existing products,
> including plans like the one Owen described, are subject to vote if anyone
> disagrees with them.

Is this described somewhere?  The HTTPD page says, "Long term plans are 
simply announcements that group members are working on particular issues 
related to the Apache software. These are not voted on [...]."

> They are also subject to veto if and only if they
> are to be applied to the current release branch (or a released branch).

Owen intends to merge this patch to a release branch.

> The compelling reason would be a measured performance impact or some
> other objective degradation of the existing product that can be
> evaluated by others as a cost/benefit tradeoff and perhaps compensated
> by modifying the implementation.

Files written by the proposed new version would not be readable by older 
versions of Hadoop.  An unaltered application that upgrades to the newer 
version would begin creating files that could not be interchanged with 
folks running the older version.

> If a PMC member insists on making design opinion the sole basis of their
> vetoes, then they are not collaborating with the rest of the PMC.  The
> board will recommend that such a person be removed from the PMC so that
> the majority can continue to develop the product in peace.

I am not the sole PMC member to express these opinions.

Doug

From general-return-2498-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 07 17:22:36 2010
Return-Path: <general-return-2498-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81811 invoked from network); 7 Dec 2010 17:22:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Dec 2010 17:22:35 -0000
Received: (qmail 35017 invoked by uid 500); 7 Dec 2010 17:22:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34971 invoked by uid 500); 7 Dec 2010 17:22:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34963 invoked by uid 99); 7 Dec 2010 17:22:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 17:22:34 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 07 Dec 2010 17:22:33 +0000
Received: (qmail 81424 invoked by uid 99); 7 Dec 2010 17:22:13 -0000
Received: from localhost.apache.org (HELO [192.168.168.16]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 17:22:13 +0000
Message-ID: <4CFE6D41.3050809@apache.org>
Date: Tue, 07 Dec 2010 09:22:09 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Direction for Hadoop development
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>	<AANLkTik1F-DCc0GNrk0dEqzRp5f-MvOQSjsg5+Z9neXf@mail.gmail.com>	<A32CDC9F-D6C6-4BA6-A44B-5FF206C1DF49@apache.org>	<0E66662F-E044-4738-8C46-EA0EBCE674E0@yahoo-inc.com> <AANLkTin43+WOUSRKVGMMb-iFZ5q64=wFvSw=DweszWis@mail.gmail.com>
In-Reply-To: <AANLkTin43+WOUSRKVGMMb-iFZ5q64=wFvSw=DweszWis@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 12/07/2010 03:27 AM, Konstantin Shvachko wrote:
> The main contradictory issue on which Owen and Doug disagree (other people
> as well) is whether
> Hadoop should support multiple serializations or be based on one designated
> serialization.
> This is a defining general direction a-issue. I believe this is vote-able.

I no longer think we should add any new serialization implementations to 
the kernel.  We might provide implementations as separate libraries that 
folks choose to use, but we should work to make sure that user code is 
well distinguished from the kernel and also try not to pollute the users 
classpath with particular versions of popular libraries.

Doug

From general-return-2499-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 07 17:26:46 2010
Return-Path: <general-return-2499-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83648 invoked from network); 7 Dec 2010 17:26:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Dec 2010 17:26:46 -0000
Received: (qmail 44778 invoked by uid 500); 7 Dec 2010 17:26:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44641 invoked by uid 500); 7 Dec 2010 17:26:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44633 invoked by uid 99); 7 Dec 2010 17:26:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 17:26:44 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 07 Dec 2010 17:26:42 +0000
Received: (qmail 83586 invoked by uid 99); 7 Dec 2010 17:26:21 -0000
Received: from localhost.apache.org (HELO [192.168.168.16]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 17:26:21 +0000
Message-ID: <4CFE6E3C.7010401@apache.org>
Date: Tue, 07 Dec 2010 09:26:20 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Direction for Hadoop development
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com> <AANLkTi=CDRWzCcz3DcP9aBBbzQA3ADwi33YL3HLu0FrQ@mail.gmail.com> <DA67105B-24BA-4EB1-8C23-D62B2E081164@apache.org> <AANLkTi=Xo=XRK3X+_Go7xjoSAQU-vxyGOoTk=ftUxi+_@mail.gmail.com> <2E0EE1F3-F322-4EDA-95D3-47FE97B34383@yahoo-inc.com>
In-Reply-To: <2E0EE1F3-F322-4EDA-95D3-47FE97B34383@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 12/07/2010 08:12 AM, Arun C Murthy wrote:
> Blocking extensions to SequenceFile is unreasonable as has been noted by
> several folks, there is no *technical* reason to do that.

The change to SequenceFile is incompatible with older versions of 
Hadoop.  It changes the file's version number so that older versions 
will not be able to read data written by newer versions.  This is a 
technical issue.

Doug

From general-return-2500-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 07 18:08:58 2010
Return-Path: <general-return-2500-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19687 invoked from network); 7 Dec 2010 18:08:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Dec 2010 18:08:57 -0000
Received: (qmail 94617 invoked by uid 500); 7 Dec 2010 18:08:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94533 invoked by uid 500); 7 Dec 2010 18:08:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94525 invoked by uid 99); 7 Dec 2010 18:08:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 18:08:56 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.83.46] (HELO mail-gw0-f46.google.com) (74.125.83.46)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 18:08:50 +0000
Received: by gwj20 with SMTP id 20so190254gwj.33
        for <general@hadoop.apache.org>; Tue, 07 Dec 2010 10:08:30 -0800 (PST)
MIME-Version: 1.0
Received: by 10.150.49.1 with SMTP id w1mr2333491ybw.210.1291745309897; Tue,
 07 Dec 2010 10:08:29 -0800 (PST)
Received: by 10.236.109.129 with HTTP; Tue, 7 Dec 2010 10:08:29 -0800 (PST)
In-Reply-To: <DA67105B-24BA-4EB1-8C23-D62B2E081164@apache.org>
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>
	<AANLkTi=CDRWzCcz3DcP9aBBbzQA3ADwi33YL3HLu0FrQ@mail.gmail.com>
	<DA67105B-24BA-4EB1-8C23-D62B2E081164@apache.org>
Date: Tue, 7 Dec 2010 13:08:29 -0500
Message-ID: <AANLkTimn6OEAoQsRR13rU1LoCZu8q6-CUFG=PSsk-NKq@mail.gmail.com>
Subject: Re: [VOTE] Direction for Hadoop development
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Thanks for your response Owen. I'll common on the JIRA with my
opinion. I didn't want to muddy the existing conversation, but if it
helps to have user level input, I'm happy to throw my hat in the ring.

Just the summary version this time:

Non-technical:

- I believe we need to temper our goals of stability with the need for
growth and improvement. The project should be free to innovate. We all
agree on this. How we do that is the question to me. We should take a
(brief) step back to make a decision on that.
- We should reevaluate how most people view and are using Hadoop to
help us make these decisions. For instance, do people see Hadoop as a
turn key system that includes everything required or do they view it
as a framework for building custom data systems? What I've seen and
believe is that it's more the latter and having some "after market"
customization is normal. The community / ASF spinning off projects
like Pig, Hive, ZK, Chukwa, and others reinforce this in my mind;
these are not bits of Hadoop proper, but natural extensions with their
own development path and release schedules.
- No one benefits from Hadoop being difficult for people to use
including those of us at Cloudera[1]. I don't want anyone to see us as
wanting to create complexity. We all benefit from a healthy Hadoop
community.

Technical:

- Any modification to SequenceFile (and friends) worry me as so much
is tightly bound to them. This is something I think is an artifact of
people coding to the implementation rather than the interface, so to
speak.
- Generally, I agree with a lot of Owen's motivation (e.g. codifying
the serialization system, using multiple libs to prove it's proper)
but some of the implementation can be more forgiving of some usage
patterns in the wild (e.g. the conflicting dep version issues, whether
future dev on some these file formats should be extracted from the
Hadoop proper).

Proposal:

- Codify (by vote) whether design plans are required or if an informal
email indicating intent is sufficient, and under what circumstances.
Provide examples to clarify circumstances. Solves the long term but
not HADOOP-6685.
- Focus the discussion on evaluation of proposals for remedying the
process for conflict resolution. I know some exist, but they're
drastic (removal of PMC members, for instance).
- After consensus on above, focus the conversation (in another thread
or on JIRA, whatever is most appropriate) on HADOOP-6685 so no one is
blocked.
- Put the community of users first in all areas of development and interact=
ion.

[1] I am officially speaking out of school. I am not an official
spokesperson for Cloudera. This is my opinion and I happen to work at
Cloudera.

On Tue, Dec 7, 2010 at 3:13 AM, Owen O'Malley <omalley@apache.org> wrote:
>
> On Dec 6, 2010, at 7:36 PM, Eric Sammer wrote:
>
> Eric,
> =A0 Since this is mostly technical, it probably should be on the h-6685 j=
ira
> instead of general@hadoop.
>
>> I think Hadoop should support interfaces to control
>> serialization plugin lifecycle and element serialization to / from an
>> abstract notion of a datum and bytes only.
>
> The core of my h-6685 patch updates the API to replace the typename with =
a
> serialization name and serialization specific metadata. That metadata is =
a
> set of bytes that are defined by the serialization. The typename alone is
> insufficient for Avro and having additional metadata will be useful for t=
he
> other serializations as well.
>
> Doug suggested that I add a user-friendly pair of methods and I did. Whil=
e
> they are redundant, the set of serializations isn't expected to be large =
and
> therefore the extra code isn't much.
>
>> I would like to not mention
>> a serialization implementation by name in Hadoop proper, at all.
>
> My patch removes some of the lingering references to Writables in
> SequenceFile, MapFile, etc. and moves them over the generic serialization
> API. The framework will likely continue depend on whichever serialization=
 is
> used for RPC. Currently that is Writables, but will likely transition to
> either Avro or ProtoBuf in the coming year.
>
>> A
>> single implementation to serve as a reference implementation makes
>> sense.
>
> A critical part of Hadoop's usability comes from its framework combined w=
ith
> library code that allows users to get the desired functionality without
> writing it themselves. Sure, it is easy to write a hash table yourself, b=
ut
> it is far easier to use the one bundled with Java.
>
>> The default
>> classpath should remain as free of mandatory external dependencies as
>> possible; library conflicts are still an extremely sore spot in Java
>> development at many sites I visit and forcing a large commercial
>> entity to use version X o something like Avro, Thrift, PB is almost a
>> non-starter for many.
>
> I discussed this problem in the jira, but either the MapReduce user is us=
ing
> the X library or doesn't care the version of X. If they are using it, it =
is
> far more convenient to have the serialization on the classpath. There is =
a
> missing feature that we need to address to put the user's files ahead of =
the
> system ones. I'll file a jira for that.
>
> It might also make sense for us to shade some of our dependencies, but th=
at
> is a much bigger issue and is far from clear cut.
>
>> If a PB / Thrift / Avro serialization implementation is part of
>> contrib or externally managed, it requires the user to understand this
>> dependency exists and manage the classpath.
>
> The goal is to make Hadoop useful out of the box. If we make it so that
> Hadoop is only useful once it is bundled with 15 other projects, that is
> good for people who sell distributions that include Hadoop, but not for t=
he
> project.
>
>> I think we can simplify
>> serialization plugin configuration via a classpath include system by
>> using something like run-parts or similar and the current
>> configuration system, but that's another issue.
>
> The current patch loads the serialization plugins based on the
> configuration. If you don't want to support thrift, don't configure it. T=
he
> same holds true of the other serializations, even writable.
>
>> I'm a bit confused as to how this equates with sequence files being
>> deprecated or arrested.
>
> Doug vetoed my patch partially based on his assertion that SequenceFiles
> should be deprecated and that Hadoop should just be the framework with no
> library code.
>
>> =A0If we choose to focus development
>> elsewhere ("soft deprecate") or actively encourage users elsewhere
>> ("@Deprecated") is an issue I think we can sever from this discussion.
>
> At this point the PMC has supported continuing to invest in developing
> SequenceFiles.
>
>> - Don't break existing SequenceFiles.
>
> That goes without saying, everyone has petabytes of data in them.
>
>> - Common, HDFS, MR should contain as few mandatory external deps as
>> humanly possible because Java classloader semantics and a lack of
>> internal dep isolation is just kookoo for cocoa puffs. (Simplify it
>> and bring on our OSGI overlords.)
>
> That is a much bigger discussion that we should probably have. There are
> costs on both sides in terms of debugging and understandability. In
> particular, in most cases we are much better off using a library that has
> the right functionality that re-implementing it ourselves.
>
>> - We (non-committers / users / casual contributors) want only for
>> Hadoop to mature in features and stability, be an inviting community
>> to new potential contributors and users, and to be around for a long
>> time.
>
> I want that too.
>
> -- Owen
>
>
>



--=20
Eric Sammer
twitter: esammer
data: www.cloudera.com

From general-return-2501-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 07 18:25:47 2010
Return-Path: <general-return-2501-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 26582 invoked from network); 7 Dec 2010 18:25:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Dec 2010 18:25:46 -0000
Received: (qmail 26226 invoked by uid 500); 7 Dec 2010 18:25:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26154 invoked by uid 500); 7 Dec 2010 18:25:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26146 invoked by uid 99); 7 Dec 2010 18:25:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 18:25:45 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.27.228] (HELO qmta15.emeryville.ca.mail.comcast.net) (76.96.27.228)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 18:25:36 +0000
Received: from omta04.emeryville.ca.mail.comcast.net ([76.96.30.35])
	by qmta15.emeryville.ca.mail.comcast.net with comcast
	id gJ9L1f0080lTkoCAFJRGLt; Tue, 07 Dec 2010 18:25:16 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta04.emeryville.ca.mail.comcast.net with comcast
	id gJR71f00A2SbwD58QJR9hJ; Tue, 07 Dec 2010 18:25:14 +0000
Message-Id: <DE84623C-739D-4C02-9EB4-29AD42739C89@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <4CFE6E3C.7010401@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Direction for Hadoop development
Date: Tue, 7 Dec 2010 10:25:05 -0800
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com> <AANLkTi=CDRWzCcz3DcP9aBBbzQA3ADwi33YL3HLu0FrQ@mail.gmail.com> <DA67105B-24BA-4EB1-8C23-D62B2E081164@apache.org> <AANLkTi=Xo=XRK3X+_Go7xjoSAQU-vxyGOoTk=ftUxi+_@mail.gmail.com> <2E0EE1F3-F322-4EDA-95D3-47FE97B34383@yahoo-inc.com> <4CFE6E3C.7010401@apache.org>
X-Mailer: Apple Mail (2.936)


On Dec 7, 2010, at 9:26 AM, Doug Cutting wrote:

> On 12/07/2010 08:12 AM, Arun C Murthy wrote:
>> Blocking extensions to SequenceFile is unreasonable as has been  
>> noted by
>> several folks, there is no *technical* reason to do that.
>
> The change to SequenceFile is incompatible with older versions of  
> Hadoop.  It changes the file's version number so that older versions  
> will not be able to read data written by newer versions.  This is a  
> technical issue.

The new code reads the new or old versions of SequenceFile seamlessly  
using auto-detection of the version.  The old code fails with an  
explicit message saying that it can't read this version. This is the  
only mechanism available when upgrading a file format with a single  
version number and is the mechanism that we've used 6 times in the past.

If we'd used ProtocolBuffers for the SequenceFile header, we'd have  
more options for backwards compatibility, but we didn't.

-- Owen

From general-return-2502-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 07 18:27:16 2010
Return-Path: <general-return-2502-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 28595 invoked from network); 7 Dec 2010 18:27:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Dec 2010 18:27:15 -0000
Received: (qmail 35087 invoked by uid 500); 7 Dec 2010 18:27:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34945 invoked by uid 500); 7 Dec 2010 18:27:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34937 invoked by uid 99); 7 Dec 2010 18:27:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 18:27:14 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of shv.hadoop@gmail.com designates 74.125.83.46 as permitted sender)
Received: from [74.125.83.46] (HELO mail-gw0-f46.google.com) (74.125.83.46)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 18:27:06 +0000
Received: by gwj20 with SMTP id 20so204607gwj.33
        for <general@hadoop.apache.org>; Tue, 07 Dec 2010 10:26:45 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=lAMQtnNtXp/qb03RX4onCTGyRM0gMnw0akkbJsTna94=;
        b=CEsVwA0kdzgPomeWUAMtXqFeJyPSG2NAAaQUKYHtXYDrssShDXMysQuwCeMgllE6qe
         7dr56y7gcZl9SNI3h+OBV397G8kvEWzfhNp8hKl5KYqUKHAGvbGQZg7MXCyaxL82lBxj
         JvMWnhIdpCpx5nzL3po2ZDn3W3BcmiEr2CUwk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=RVGAJgLcQ2IdELzqGF/e3jaZeLX6OOaWQwnWFGZcaXVA9l6JE89mMM5HHkkM16PgZX
         7CmvHs41mOc0sQ7tsSapoYDW2I6qhGxi1Wck43/Ta6BNePXxexBOjqWMtvDFsprCz5mQ
         UoAzeZs+QSQd9SLdwMd1xjzyU/QoM7XabZ4MM=
MIME-Version: 1.0
Received: by 10.150.203.15 with SMTP id a15mr2415730ybg.136.1291746405187;
 Tue, 07 Dec 2010 10:26:45 -0800 (PST)
Received: by 10.236.109.15 with HTTP; Tue, 7 Dec 2010 10:26:44 -0800 (PST)
In-Reply-To: <4CFE6D41.3050809@apache.org>
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>
	<AANLkTik1F-DCc0GNrk0dEqzRp5f-MvOQSjsg5+Z9neXf@mail.gmail.com>
	<A32CDC9F-D6C6-4BA6-A44B-5FF206C1DF49@apache.org>
	<0E66662F-E044-4738-8C46-EA0EBCE674E0@yahoo-inc.com>
	<AANLkTin43+WOUSRKVGMMb-iFZ5q64=wFvSw=DweszWis@mail.gmail.com>
	<4CFE6D41.3050809@apache.org>
Date: Tue, 7 Dec 2010 10:26:44 -0800
Message-ID: <AANLkTin9qS0YDbMUPA29TZ0UWN+=f9XVL7rZ88rKk7y_@mail.gmail.com>
Subject: Re: [VOTE] Direction for Hadoop development
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd3755a9f00b80496d625f0
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd3755a9f00b80496d625f0
Content-Type: text/plain; charset=ISO-8859-1

> I no longer think we should add any new serialization implementations to
the kernel.

Not clear. Do you propose to keep current serialization(s) and not add new
ones?
Or do you propose to replace current serialization by abstract interfaces
and move implementations to libraries?

--Konstantin

On Tue, Dec 7, 2010 at 9:22 AM, Doug Cutting <cutting@apache.org> wrote:

> On 12/07/2010 03:27 AM, Konstantin Shvachko wrote:
>
>> The main contradictory issue on which Owen and Doug disagree (other people
>> as well) is whether
>> Hadoop should support multiple serializations or be based on one
>> designated
>> serialization.
>> This is a defining general direction a-issue. I believe this is vote-able.
>>
>
> I no longer think we should add any new serialization implementations to
> the kernel.  We might provide implementations as separate libraries that
> folks choose to use, but we should work to make sure that user code is well
> distinguished from the kernel and also try not to pollute the users
> classpath with particular versions of popular libraries.
>
> Doug
>

--000e0cd3755a9f00b80496d625f0--

From general-return-2503-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 07 22:38:12 2010
Return-Path: <general-return-2503-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44219 invoked from network); 7 Dec 2010 22:38:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Dec 2010 22:38:11 -0000
Received: (qmail 11042 invoked by uid 500); 7 Dec 2010 22:38:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 10951 invoked by uid 500); 7 Dec 2010 22:38:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 10943 invoked by uid 99); 7 Dec 2010 22:38:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 22:38:10 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [208.97.132.202] (HELO homiemail-a27.g.dreamhost.com) (208.97.132.202)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Dec 2010 22:38:03 +0000
Received: from homiemail-a27.g.dreamhost.com (localhost [127.0.0.1])
	by homiemail-a27.g.dreamhost.com (Postfix) with ESMTP id 3C767598088
	for <general@hadoop.apache.org>; Tue,  7 Dec 2010 14:37:43 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; q=dns; s=
	gbiv.com; b=Q4dEr7A0rfqoI1OlKA3nPEWbAN+PCDqUjveV3otCf5KhlDfGO5iu
	kuu/CZ12OoF3vwpUtBNkTpeUqDlYTxra30NDOokFQe2wSw/6imgtUCDiEcXZhOrJ
	nI18uzx1wFyE4W6gLkzD8aJitLRQi0TZNWWVCazYNNz5e840wR9rrdI=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; s=gbiv.com;
	 bh=Qs6/KSTZ2RZp+Z8oUZsqZUS9Qx4=; b=PFI4V3yytO3EXzjgmBHp056SJO1P
	uzetmcXOgnYhGLkvL+1Rzy8GkXID4Uqf6ZXj/nilSpsBlQFzy68Bk2v7/If3oqec
	uTjbeFi9vsZZ5YjC4ovFjhW8xwSjN2Y2lQTo2yaW660BgKsVUYLyQhy+tIAD6fTS
	54zgdOXNc/vX2EE=
Received: from [192.168.1.66] (99-21-208-82.lightspeed.irvnca.sbcglobal.net [99.21.208.82])
	(Authenticated sender: fielding@gbiv.com)
	by homiemail-a27.g.dreamhost.com (Postfix) with ESMTPA id E83E2598087
	for <general@hadoop.apache.org>; Tue,  7 Dec 2010 14:37:42 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [VOTE] Direction for Hadoop development
From: "Roy T. Fielding" <fielding@gbiv.com>
In-Reply-To: <4CFE6C68.7000503@apache.org>
Date: Tue, 7 Dec 2010 14:37:42 -0800
Content-Transfer-Encoding: quoted-printable
Message-Id: <DE2F0813-8D15-4D50-9C6C-5A6EBB3974C8@gbiv.com>
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>	<4CF433D4.6090407@apache.org>	<AANLkTineTnsShLoa5vO-X9UAp_DUMrHRjc2ry=i4qkXs@mail.gmail.com>	<4CF66812.4080602@apache.org>	<D414D20A-A909-46A9-AAA2-52742E6395CF@apache.org>	<4CFD39BC.1080005@apache.org> <AANLkTinO_WpU3WSx6AZDN2tM7C=+Ja_hQ0n7qb_jLw2h@mail.gmail.com> <4CFD7585.70803@apache.org> <5A920FAB-F76B-4EF7-8740-5698F99A1EC2@gbiv.com> <4CFE6C68.7000503@apache.org>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

On Dec 7, 2010, at 9:18 AM, Doug Cutting wrote:

> On 12/06/2010 05:09 PM, Roy T. Fielding wrote:
>> Generally speaking, vetoing extension interfaces without a compelling
>> technical reason is not the way Apache operates.  We make extensions
>> modular so that diverse collaborators can specialize according to
>> their own needs, not just your needs.
>=20
> Roy, thanks for your thoughts here.
>=20
> I have not intended to veto an extension mechanism.  In this case, we =
already have an extension mechanism.

It is my understanding that the existing extension mechanism did not
support the desired extensions, so improving it makes sense.

>  The proposal is to change the extension mechanism incompatibly with =
unclear benefits,

Good, these are technical reasons.  The benefits can be cleared by docs.
By incompatible, I assume you mean forward-compatibility of old versions
of Hadoop reading newer files.  Can we fix that by having the new
implementation use the old file format by default until it is configured
to use one of the new interfaces for writing?

> add implementations of several extensions to the kernel, and =
incompatibly change a widely-used file format.

You keep referring to the kernel as if it were a product.  I don't see
a kernel product in the list of things released by Apache Hadoop.
If there were such a product, then it would make sense for Apache Hadoop
to also release ancillary products for common libraries, test =
frameworks,
and modular storage interfaces.  Rearchitecting the Hadoop product suite
into such a logical arrangement would make sense, and after such an
architecture is put into place then "keeping the kernel simple" would
be a reason to veto a change to the kernel.

> In general I support improving extension mechanisms, but oppose =
gratuitous changes to file formats and the inclusion of new user-level =
functionality in the kernel.

Persistence is not usually considered user-level functionality, nor do
the proposed changes seem gratuitous.  Owen said the reason was to
support type-safety, which may well be a desirable feature for some
users.  I think it makes sense to find a way to modularize the feature
such that this functionality is only brought in when configured
by the user.

>  I'd like the issue to focus solely on the extension mechanism to =
clarify the discussion, not on adding extensions to the kernel or file =
formats.

That is irrelevant.  Doing development via jira discussion is inherently
dysfunctional because it promotes such bureaucratic nonsense instead of
working towards a common solution via iterative development.  My goal
here is to fix this goofy behavior, not reinforce it.

> Tom long ago provided patches showing how the existing configuration =
system can provide equivalent extension implementations outside of the =
kernel with no incompatible changes.  (MAPREDUCE-376 and MAPREDUCE-377)

They both seem to be active and unfinished.  If they are equivalent =
fixes
to the same problem, then I suggest applying them to a branch, =
documenting
how they work, and then agreeing to have a bake-off.  A bake-off is a
decision made by performance and feature-completeness as an objective
way to resolve an impasse due to mutually exclusive vetoes.  All sides =
agree
to drop the veto and accept whichever performs best, by majority =
decision.

>> Changes to the existing products,
>> including plans like the one Owen described, are subject to vote if =
anyone
>> disagrees with them.
>=20
> Is this described somewhere?  The HTTPD page says, "Long term plans =
are simply announcements that group members are working on particular =
issues related to the Apache software. These are not voted on [...]."

All action items can be voted on.  What we are talking about here is a
short term plan, and it is listed as a type of action item under
changes to products.

>> They are also subject to veto if and only if they
>> are to be applied to the current release branch (or a released =
branch).
>=20
> Owen intends to merge this patch to a release branch.

Right.

>> The compelling reason would be a measured performance impact or some
>> other objective degradation of the existing product that can be
>> evaluated by others as a cost/benefit tradeoff and perhaps =
compensated
>> by modifying the implementation.
>=20
> Files written by the proposed new version would not be readable by =
older versions of Hadoop.  An unaltered application that upgrades to the =
newer version would begin creating files that could not be interchanged =
with folks running the older version.

Good reason, so let's fix that.  Note, however, that one valid solution =
is
to simply release it as a new major version of the product.

>> If a PMC member insists on making design opinion the sole basis of =
their
>> vetoes, then they are not collaborating with the rest of the PMC.  =
The
>> board will recommend that such a person be removed from the PMC so =
that
>> the majority can continue to develop the product in peace.
>=20
> I am not the sole PMC member to express these opinions.

No, but the other objections seem to be suffering from a lack of =
independent
thought and are predicated on the theory that outside organizations will
satisfy the needs of our users instead of an Apache project solving them
directly.  That is extremely annoying to me, since the only reason I am =
here
is to deal with some folks' failure to think independently of their =
employer.

....Roy


From general-return-2504-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 08 03:32:27 2010
Return-Path: <general-return-2504-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68898 invoked from network); 8 Dec 2010 03:32:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Dec 2010 03:32:26 -0000
Received: (qmail 82166 invoked by uid 500); 8 Dec 2010 03:32:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81887 invoked by uid 500); 8 Dec 2010 03:32:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81879 invoked by uid 99); 8 Dec 2010 03:32:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Dec 2010 03:32:24 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [64.78.17.16] (HELO EXHUB018-1.exch018.msoutlookonline.net) (64.78.17.16)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Dec 2010 03:32:15 +0000
Received: from EXVMBX018-1.exch018.msoutlookonline.net ([64.78.17.47]) by
 EXHUB018-1.exch018.msoutlookonline.net ([64.78.17.16]) with mapi; Tue, 7 Dec
 2010 19:31:54 -0800
From: Scott Carey <scott@richrelevance.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 7 Dec 2010 19:33:35 -0800
Subject: Re: [VOTE] Direction for Hadoop development
Thread-Topic: [VOTE] Direction for Hadoop development
Thread-Index: AcuWiHTQceIcGmOYTT6DRRdx+DlGNQ==
Message-ID: <EEEB7D0A-3122-4D89-BBB0-E1C47965A01E@richrelevance.com>
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>
 <4CF433D4.6090407@apache.org>
 <AANLkTineTnsShLoa5vO-X9UAp_DUMrHRjc2ry=i4qkXs@mail.gmail.com>
In-Reply-To: <AANLkTineTnsShLoa5vO-X9UAp_DUMrHRjc2ry=i4qkXs@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


On Nov 29, 2010, at 4:22 PM, Owen O'Malley wrote:

> I do not support adding new dependencies to the classpath of MapReduce us=
er
>> tasks.
>=20
>=20
> That isn't reasonable. As Hadoop evolves, we have and will continue to ad=
d
> dependences. For example, in your last MapReduce (MAPREDUCE-980) patch yo=
u
> added avro and paranamer as dependences.
>=20

As a non PMC member:

Hadoop has already put enough stuff on the classpath to force me to make a =
custom build to use (in 0.19 was the start, and now no distribution can wor=
k without modification).  This is because of it stuffing more and more thin=
gs on the classpath.

It is completely reaonable to ask that the environment that user code runs =
in not be polluted with libraries that are not exposed in the Hadoop API, a=
nd debate the merits of a patch based on the inclusion of an additional jar=
 on that classpath.

Webapp containers, OSGi, other classloader systems, or dependency rebasing =
(jarjar links, maven shade, etc) help solve this sort of mess.Even more cru=
dely, the user's lib directory doesn't have to be Hadoop's full lib directo=
ry, and the order of inclusion of jars can help.

Either way, if Hadoop wants to be an application execution framework, it ca=
n't just throw whatever it wants on the classpath forever.
If one wants to provide lots of tools as part of a rich environment for use=
rs, the user has to either be able to easily _opt in_ to having those tools=
 available on their class path or _opt out_ of having them there.


Now, this is really a tangent to other issues at hand.=20
I'd like to suggest that rather than point fingers at who added what to wha=
t classpath and when, it is just noted that classpath management is a probl=
em that Hadoop needs to solve and not ignore.  I'm pretty sure there's a JI=
RA on it somewhere already.
Until it is solved to some degree (since on a scale of 1 to 10 dealing with=
 classpath collisions, Hadoop is currently somewhere between 0 and 1), its =
going to limit what can be built without causing user applications to break=
 on an upgrade.   Whether those new features are good or bad on its own mer=
its is being conflated with classpath problems that it introduces for users=
.

> -- Owen


From general-return-2505-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 08 05:27:00 2010
Return-Path: <general-return-2505-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 8816 invoked from network); 8 Dec 2010 05:26:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Dec 2010 05:26:59 -0000
Received: (qmail 31974 invoked by uid 500); 8 Dec 2010 05:26:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31616 invoked by uid 500); 8 Dec 2010 05:26:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31606 invoked by uid 99); 8 Dec 2010 05:26:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Dec 2010 05:26:57 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.104 as permitted sender)
Received: from [17.148.16.104] (HELO asmtpout029.mac.com) (17.148.16.104)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Dec 2010 05:26:49 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; charset=us-ascii
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp029.mac.com (Sun Java(tm) System Messaging Server 6.3-7.04 (built Sep
 26 2008; 64bit)) with ESMTPSA id <0LD300AE4GFP1P40@asmtp029.mac.com> for
 general@hadoop.apache.org; Tue, 07 Dec 2010 21:26:15 -0800 (PST)
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1012070253
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2010-12-08_03:2010-12-08,2010-12-08,1970-01-01 signatures=0
Subject: Re: [DISCUSS] clarifying the bylaws
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <159E99C4-B71C-437E-9640-AA24C50D636E@apache.org>
Date: Tue, 07 Dec 2010 21:26:13 -0800
Message-id: <77BB1207-9DB3-4B5F-A67B-6222D34D52B2@mac.com>
References: <159E99C4-B71C-437E-9640-AA24C50D636E@apache.org>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

+1, this fixes the issues I raised.

N.

On Nov 29, 2010, at 11:31 AM, Owen O'Malley wrote:

> As Nigel pointed out, the bylaws didn't match the discussion.
> 
> Here is a proposed patch.
> 
> *** bylaws.txt  2010-11-29 11:26:53.000000000 -0800
> --- bylaws2.txt 2010-11-29 11:20:05.000000000 -0800
> ***************
> *** 207,214 ****
>              by a committer. This includes source code, documentation, website
>              content, etc.
> 
> !             Lazy consensus of active committers, but with a minimum of
> !             one +1. The code can be committed after the first +1.
> 
>           * Release Plan
> 
> --- 207,213 ----
>              by a committer. This includes source code, documentation, website
>              content, etc.
> 
> !             Lazy consensus of active committers.
> 
>           * Release Plan
> 
> ***************
> *** 269,275 ****
> 
>              Modifying this document.
> 
> !             Lazy majority of active PMC members
> 
>     * Voting Timeframes
> 
> --- 268,274 ----
> 
>              Modifying this document.
> 
> !             Majority of active PMC members
> 
>     * Voting Timeframes


From general-return-2506-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 08 18:12:37 2010
Return-Path: <general-return-2506-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21134 invoked from network); 8 Dec 2010 18:12:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Dec 2010 18:12:36 -0000
Received: (qmail 91145 invoked by uid 500); 8 Dec 2010 18:12:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90997 invoked by uid 500); 8 Dec 2010 18:12:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90989 invoked by uid 99); 8 Dec 2010 18:12:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Dec 2010 18:12:34 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 08 Dec 2010 18:12:31 +0000
Received: (qmail 21051 invoked by uid 99); 8 Dec 2010 18:12:10 -0000
Received: from localhost.apache.org (HELO [192.168.168.16]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Dec 2010 18:12:10 +0000
Message-ID: <4CFFCA79.3090901@apache.org>
Date: Wed, 08 Dec 2010 10:12:09 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Direction for Hadoop development
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>	<4CF433D4.6090407@apache.org>	<AANLkTineTnsShLoa5vO-X9UAp_DUMrHRjc2ry=i4qkXs@mail.gmail.com>	<4CF66812.4080602@apache.org>	<D414D20A-A909-46A9-AAA2-52742E6395CF@apache.org>	<4CFD39BC.1080005@apache.org> <AANLkTinO_WpU3WSx6AZDN2tM7C=+Ja_hQ0n7qb_jLw2h@mail.gmail.com> <4CFD7585.70803@apache.org> <5A920FAB-F76B-4EF7-8740-5698F99A1EC2@gbiv.com> <4CFE6C68.7000503@apache.org> <DE2F0813-8D15-4D50-9C6C-5A6EBB3974C8@gbiv.com>
In-Reply-To: <DE2F0813-8D15-4D50-9C6C-5A6EBB3974C8@gbiv.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 12/07/2010 02:37 PM, Roy T. Fielding wrote:
> Good, these are technical reasons.  The benefits can be cleared by docs.
> By incompatible, I assume you mean forward-compatibility of old versions
> of Hadoop reading newer files.  Can we fix that by having the new
> implementation use the old file format by default until it is configured
> to use one of the new interfaces for writing?

+1

> You keep referring to the kernel as if it were a product.  I don't see
> a kernel product in the list of things released by Apache Hadoop.

The line is fairly clear.  The kernel is the daemons plus the framework 
code that invokes user code.  The set of pluggable user implementations 
is fairly small: InputFormat, OutputFormat, Mapper, Reducer, RawComparator.

SequenceFile was originally part of the kernel but is now only used by 
user-level InputFormats and OutputFormats.

> If there were such a product, then it would make sense for Apache Hadoop
> to also release ancillary products for common libraries, test frameworks,
> and modular storage interfaces.  Rearchitecting the Hadoop product suite
> into such a logical arrangement would make sense, and after such an
> architecture is put into place then "keeping the kernel simple" would
> be a reason to veto a change to the kernel.

Such a re-arrangement has been proposed but not completed.  Relevant 
issues are MAPREDUCE-1638, MAPREDUCE-1453, and MAPREDUCE-1700.  It 
mostly involves build issues; the architecture already largely supports 
the distinction.

>> Tom long ago provided patches showing how the existing
>> configuration system can provide equivalent extension
>> implementations outside of the kernel with no incompatible changes.
>> (MAPREDUCE-376 and MAPREDUCE-377)
> They both seem to be active and unfinished.  If they are equivalent fixes
> to the same problem, then I suggest applying them to a branch, documenting
> how they work, and then agreeing to have a bake-off.  A bake-off is a
> decision made by performance and feature-completeness as an objective
> way to resolve an impasse due to mutually exclusive vetoes.  All sides agree
> to drop the veto and accept whichever performs best, by majority decision.

A bake-off could be a good way to resolve this.  Performance differences 
would not likely be measurable, but folks might examine user programs 
and consider compatibility and support implications and vote accordingly.

> All action items can be voted on.  What we are talking about here is a
> short term plan, and it is listed as a type of action item under
> changes to products.

Then voting on specific short-term actions might be a good way to 
resolve this.

Some specific short-term questions we might vote on:

1. Should we add specific versions of Protocol Buffers and Thrift to the 
classpath of every MapReduce program?

2. Should SequenceFile be forward-compatible, i.e., if an existing 
program that stores Writables in a SequenceFile is run against the new 
version, should the old version still be able to read the output of the 
new version?

3. Should we continue support a specified interchange format and/or data 
model for configuration data, or should configurations rather be opaque 
binary data?  An interchange format might be JSON.  An interchange data 
model might  Map<String,Value> where values can be strings, booleans, 
numbers, bytes or nested configuration data, defined by a standard API 
that all configurable items would support.  A specified format or model 
would permit things like using -D to set configuration options and 
permit generic interaction with external configuration systems.  With 
opaque binary configurations, each configurable item would provide its 
own API and would require specific new code that calls this API for each 
parameter that could be set with -D or from an external configuration 
system.

>>> They are also subject to veto if and only if they
>>> are to be applied to the current release branch (or a released branch).
>>
>> Owen intends to merge this patch to a release branch.
>
> Right.

So votes on action items would be simple majority if they're not 
intended to be merged to a release branch, and vetoable if they are?  Is 
that right?

Doug

From general-return-2507-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 08 18:56:25 2010
Return-Path: <general-return-2507-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49831 invoked from network); 8 Dec 2010 18:56:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Dec 2010 18:56:25 -0000
Received: (qmail 69227 invoked by uid 500); 8 Dec 2010 18:56:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69146 invoked by uid 500); 8 Dec 2010 18:56:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69137 invoked by uid 99); 8 Dec 2010 18:56:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Dec 2010 18:56:23 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 08 Dec 2010 18:56:21 +0000
Received: (qmail 49756 invoked by uid 99); 8 Dec 2010 18:55:59 -0000
Received: from localhost.apache.org (HELO [192.168.168.16]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Dec 2010 18:55:59 +0000
Message-ID: <4CFFD4BE.9050704@apache.org>
Date: Wed, 08 Dec 2010 10:55:58 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Direction for Hadoop development
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>	<AANLkTik1F-DCc0GNrk0dEqzRp5f-MvOQSjsg5+Z9neXf@mail.gmail.com>	<A32CDC9F-D6C6-4BA6-A44B-5FF206C1DF49@apache.org>	<0E66662F-E044-4738-8C46-EA0EBCE674E0@yahoo-inc.com>	<AANLkTin43+WOUSRKVGMMb-iFZ5q64=wFvSw=DweszWis@mail.gmail.com>	<4CFE6D41.3050809@apache.org> <AANLkTin9qS0YDbMUPA29TZ0UWN+=f9XVL7rZ88rKk7y_@mail.gmail.com>
In-Reply-To: <AANLkTin9qS0YDbMUPA29TZ0UWN+=f9XVL7rZ88rKk7y_@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 12/07/2010 10:26 AM, Konstantin Shvachko wrote:
>> I no longer think we should add any new serialization implementations to
> the kernel.
>
> Not clear. Do you propose to keep current serialization(s) and not add new
> ones?
> Or do you propose to replace current serialization by abstract interfaces
> and move implementations to libraries?

We can't move existing serialization implementations to an optional 
library without breaking compatibility.  Long-term that might be nice, 
but I am not proposing that short-term.  Short-term I propose we avoid 
adding new serialization implementations to the default classpath, 
especially those that add new dependencies to every task.  Long-term we 
might split library code into perhaps a few categories:
  - mandatory: this might include, e.g., IdentityMapper and 
IdentityReducer, the default implementations.
  - back-compatible: the collection of library components that were 
provided on the default classpath and can be enabled for back-compatible 
behavior.
  - optional: components that jobs can optionally depend on.  This is 
where new components that are not mandatory would be added.

Doug



From general-return-2508-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 08 19:21:12 2010
Return-Path: <general-return-2508-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66066 invoked from network); 8 Dec 2010 19:21:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Dec 2010 19:21:11 -0000
Received: (qmail 3192 invoked by uid 500); 8 Dec 2010 19:21:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3120 invoked by uid 500); 8 Dec 2010 19:21:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3112 invoked by uid 99); 8 Dec 2010 19:21:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Dec 2010 19:21:10 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 08 Dec 2010 19:21:07 +0000
Received: (qmail 66002 invoked by uid 99); 8 Dec 2010 19:20:45 -0000
Received: from localhost.apache.org (HELO [192.168.168.16]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Dec 2010 19:20:45 +0000
Message-ID: <4CFFDA8C.2000306@apache.org>
Date: Wed, 08 Dec 2010 11:20:44 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Direction for Hadoop development
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com> <AANLkTi=CDRWzCcz3DcP9aBBbzQA3ADwi33YL3HLu0FrQ@mail.gmail.com> <DA67105B-24BA-4EB1-8C23-D62B2E081164@apache.org> <AANLkTi=Xo=XRK3X+_Go7xjoSAQU-vxyGOoTk=ftUxi+_@mail.gmail.com> <2E0EE1F3-F322-4EDA-95D3-47FE97B34383@yahoo-inc.com> <4CFE6E3C.7010401@apache.org> <DE84623C-739D-4C02-9EB4-29AD42739C89@apache.org>
In-Reply-To: <DE84623C-739D-4C02-9EB4-29AD42739C89@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 12/07/2010 10:25 AM, Owen O'Malley wrote:
> The new code reads the new or old versions of SequenceFile seamlessly
> using auto-detection of the version. The old code fails with an explicit
> message saying that it can't read this version. This is the only
> mechanism available when upgrading a file format with a single version
> number and is the mechanism that we've used 6 times in the past.

The last such change was nearly four years ago, in:

https://issues.apache.org/jira/browse/HADOOP-732

The quantity of data stored in SequenceFiles has greatly increased over 
the past four years.  The project's concern for compatibility has also 
correspondingly increased over that time.

The new format version might not be written when folks are using 
Writable or some other serialization currently supported by 
SequenceFile.  The only situation in your patch where the new version is 
required is for Avro.  You might simply drop support for Avro and leave 
the file version number alone since Avro already includes a container 
file format.  Or you might only use the new format version for 
non-class-determined serializations like Avro.  Or you might use 
SequenceFile's existing metadata for non-class-determined serializations 
like Avro and leave the file version number alone.

Doug

From general-return-2509-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 08 20:41:25 2010
Return-Path: <general-return-2509-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 2076 invoked from network); 8 Dec 2010 20:41:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Dec 2010 20:41:25 -0000
Received: (qmail 51931 invoked by uid 500); 8 Dec 2010 20:41:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51872 invoked by uid 500); 8 Dec 2010 20:41:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51863 invoked by uid 99); 8 Dec 2010 20:41:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Dec 2010 20:41:23 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of anthony.urso@gmail.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Dec 2010 20:41:18 +0000
Received: by gyf1 with SMTP id 1so1073150gyf.35
        for <general@hadoop.apache.org>; Wed, 08 Dec 2010 12:40:57 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=ApcLufbkF0fH4AYULe26zVuFJrrend8SrFHXtxuED2g=;
        b=bsv3o3DkA45Y7wO05w0NTpt+OfCFNHKHrT1aVcOng2iqhdLugHdUEFlgFQHQ7pu6BV
         w+Wc4TC45VzmW2t0aBOEaGtc92ELpcVxEd4PlGicuR5cGeTmJJKY7a28Zww21ubsqx8f
         6gOyXG0LVi3CIxhbRloEKv2rsnWEpVQVKMZ6U=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=bTNniZSxTiGBBpMKb/5bPeq5tvLzlPLC9iy45m5+eT8qaycHk7ke1BOczv5y/H8tqZ
         JPfLH2fFgWlkiNHnXshlP0bwK9iB6uEaxLO9miW2kjlRrZMyMPSTFB1cJr4vktcB7M3R
         4Q4QCQMN99Ff/M74MNAJJnyUf0ytmFwjZ7q30=
MIME-Version: 1.0
Received: by 10.90.75.18 with SMTP id x18mr12270795aga.5.1291840857807; Wed,
 08 Dec 2010 12:40:57 -0800 (PST)
Received: by 10.90.78.20 with HTTP; Wed, 8 Dec 2010 12:40:57 -0800 (PST)
Date: Wed, 8 Dec 2010 12:40:57 -0800
Message-ID: <AANLkTikNNrH2ySg2_JcuDCRATiP33LqXoZoKg8tAmMsn@mail.gmail.com>
Subject: Announcing KeptCollections, distributed Java Collections for ZooKeeper
From: Anthony Urso <anthony.urso@gmail.com>
To: user@zookeeper.apache.org, dev@zookeeper.apache.org, 
	general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

I am pleased to announce the initial release of KeptCollections, a
library of drop-in replacements for standard Java Collections that use
Apache ZooKeeper as a backing store.

KeptCollections are designed to make it easy for anyone to write
distributed applications without having to learn the intricacies of
ZooKeeper, or distributed programming in general.

The collections use the well-known JDK APIs, yet any changes made to
any of these collections by one node are seen by all other nodes
within milliseconds, allowing for easy communication between processes in a
computing cluster.

More information here:

https://github.com/anthonyu/KeptCollections/wiki

and all code is available from:

https://github.com/anthonyu/KeptCollections

Please try it out, and let me know any problems you experience via
github issues or this email address.

Cheers,
Anthony

From general-return-2510-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 08 23:16:19 2010
Return-Path: <general-return-2510-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78582 invoked from network); 8 Dec 2010 23:16:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Dec 2010 23:16:18 -0000
Received: (qmail 19341 invoked by uid 500); 8 Dec 2010 23:16:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19129 invoked by uid 500); 8 Dec 2010 23:16:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19121 invoked by uid 99); 8 Dec 2010 23:16:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Dec 2010 23:16:16 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ted.dunning@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Dec 2010 23:16:12 +0000
Received: by wye20 with SMTP id 20so1742907wye.35
        for <general@hadoop.apache.org>; Wed, 08 Dec 2010 15:15:50 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:cc:content-type;
        bh=DjJapUrjBx0PMpz5Ce5I02Sec1CZdIGJJ6yJ2ghBN8o=;
        b=ifboN63UpXQNBKhHqhTgEUHQIxOk1RfKH9rmVvRbpBKSkhFadR5DRYmSm4Net/qCTH
         KCZ14hrf9u6MBp+5xiWT5BYsmo36K8V4g5faIT0Gv5g0NvyR0LA0Gh0lnz5+HHWiKFOD
         gUZljaBC5eTKF0212j3VLUpWp4v6+u+W3dGz4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :cc:content-type;
        b=DXaiYDN2URM7KYxW1chonDltsVCqoqgHsLHvANCb8DMZoxNkIhBuiafP3jmfr14DhV
         EoSqCL7hXHBsZExBvNs0rk7U+kPRFoqH12es9tzAntva7wf9tqtkA6IFUgi8xa4lCl2L
         S9VsfD5EvBv3VFHlYLtlg5spfse023Zpi6tGg=
Received: by 10.216.169.6 with SMTP id m6mr1211794wel.49.1291850150712; Wed,
 08 Dec 2010 15:15:50 -0800 (PST)
MIME-Version: 1.0
Received: by 10.216.158.68 with HTTP; Wed, 8 Dec 2010 15:15:30 -0800 (PST)
In-Reply-To: <AANLkTikNNrH2ySg2_JcuDCRATiP33LqXoZoKg8tAmMsn@mail.gmail.com>
References: <AANLkTikNNrH2ySg2_JcuDCRATiP33LqXoZoKg8tAmMsn@mail.gmail.com>
From: Ted Dunning <ted.dunning@gmail.com>
Date: Wed, 8 Dec 2010 15:15:30 -0800
Message-ID: <AANLkTi=gAobRq2Vi23ZjvC1KFcQKUs3V+rg88GKpaOop@mail.gmail.com>
Subject: Re: Announcing KeptCollections, distributed Java Collections for ZooKeeper
To: user@zookeeper.apache.org
Cc: dev@zookeeper.apache.org, general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016367b67c65614980496ee4d84

--0016367b67c65614980496ee4d84
Content-Type: text/plain; charset=UTF-8

This looks very useful and looks like nice work.

I note that the methods used are prone to race conditions, but if you are
just thinking about shared maps, this probably isn't important.

On Wed, Dec 8, 2010 at 12:40 PM, Anthony Urso <anthony.urso@gmail.com>wrote:

> I am pleased to announce the initial release of KeptCollections, a
> library of drop-in replacements for standard Java Collections that use
> Apache ZooKeeper as a backing store.
>
> KeptCollections are designed to make it easy for anyone to write
> distributed applications without having to learn the intricacies of
> ZooKeeper, or distributed programming in general.
>
> The collections use the well-known JDK APIs, yet any changes made to
> any of these collections by one node are seen by all other nodes
> within milliseconds, allowing for easy communication between processes in a
> computing cluster.
>
> More information here:
>
> https://github.com/anthonyu/KeptCollections/wiki
>
> and all code is available from:
>
> https://github.com/anthonyu/KeptCollections
>
> Please try it out, and let me know any problems you experience via
> github issues or this email address.
>
> Cheers,
> Anthony
>

--0016367b67c65614980496ee4d84--

From general-return-2511-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 08 23:41:07 2010
Return-Path: <general-return-2511-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 89512 invoked from network); 8 Dec 2010 23:41:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Dec 2010 23:41:06 -0000
Received: (qmail 66775 invoked by uid 500); 8 Dec 2010 23:41:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66650 invoked by uid 500); 8 Dec 2010 23:41:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 63875 invoked by uid 99); 8 Dec 2010 23:37:38 -0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ewhauser@gmail.com designates 209.85.161.47 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:cc:content-type;
        bh=KTSKBCB/ag+CLhIKbgaG+wyE2nfjGyu2FG6tfwCY8fI=;
        b=w03iVAoDVu1dE2dy94UJFzkov1onePX6w7Tl6TNLOos8sbSQAx7PZwS2s2+W0KmT4F
         ID8nuAbTa2V6MUIaWMGVkrLfcVMRoPAWfOVcbSOGDFw6NsY/b7fkW7PzzidpPrm/soVP
         UAmU8hiFmTQ6ybTy028SGcZXGU6EIDuTxBF0c=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :cc:content-type;
        b=BZ6BxQowu3m9fKAv0c/ZPkuzMuLWar7y7UcoFJjtv7ziIgegQOAia6I3ETS+pMHA4c
         hfx7W8dG+gi08dX8Ouq0oICJfYamDjYojh4D/n8ATJtLnmnCAZmnjKDyvf3OKUNXgmXz
         fhdVjfYJVCWlK5Du4XR08xINYIYQE3AOf/6LQ=
MIME-Version: 1.0
In-Reply-To: <AANLkTi=gAobRq2Vi23ZjvC1KFcQKUs3V+rg88GKpaOop@mail.gmail.com>
References: <AANLkTikNNrH2ySg2_JcuDCRATiP33LqXoZoKg8tAmMsn@mail.gmail.com> <AANLkTi=gAobRq2Vi23ZjvC1KFcQKUs3V+rg88GKpaOop@mail.gmail.com>
From: Eric Hauser <ewhauser@gmail.com>
Date: Wed, 8 Dec 2010 18:33:28 -0500
Message-ID: <AANLkTi=tekqH557XC8tqRLynY8mh_6teGogOaYyVasfK@mail.gmail.com>
Subject: Re: Announcing KeptCollections, distributed Java Collections for ZooKeeper
To: dev@zookeeper.apache.org
Cc: user@zookeeper.apache.org, general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Out of curiosity, why not just use Redis for this?

On Wed, Dec 8, 2010 at 6:15 PM, Ted Dunning <ted.dunning@gmail.com> wrote:
> This looks very useful and looks like nice work.
>
> I note that the methods used are prone to race conditions, but if you are
> just thinking about shared maps, this probably isn't important.
>
> On Wed, Dec 8, 2010 at 12:40 PM, Anthony Urso <anthony.urso@gmail.com>wrote:
>
>> I am pleased to announce the initial release of KeptCollections, a
>> library of drop-in replacements for standard Java Collections that use
>> Apache ZooKeeper as a backing store.
>>
>> KeptCollections are designed to make it easy for anyone to write
>> distributed applications without having to learn the intricacies of
>> ZooKeeper, or distributed programming in general.
>>
>> The collections use the well-known JDK APIs, yet any changes made to
>> any of these collections by one node are seen by all other nodes
>> within milliseconds, allowing for easy communication between processes in a
>> computing cluster.
>>
>> More information here:
>>
>> https://github.com/anthonyu/KeptCollections/wiki
>>
>> and all code is available from:
>>
>> https://github.com/anthonyu/KeptCollections
>>
>> Please try it out, and let me know any problems you experience via
>> github issues or this email address.
>>
>> Cheers,
>> Anthony
>>
>

From general-return-2512-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 08 23:44:52 2010
Return-Path: <general-return-2512-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 90390 invoked from network); 8 Dec 2010 23:44:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Dec 2010 23:44:52 -0000
Received: (qmail 69859 invoked by uid 500); 8 Dec 2010 23:44:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69804 invoked by uid 500); 8 Dec 2010 23:44:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69796 invoked by uid 99); 8 Dec 2010 23:44:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Dec 2010 23:44:50 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of anthony.urso@gmail.com designates 209.85.213.176 as permitted sender)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Dec 2010 23:44:45 +0000
Received: by yxm8 with SMTP id 8so1184184yxm.35
        for <general@hadoop.apache.org>; Wed, 08 Dec 2010 15:44:25 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:cc:content-type;
        bh=gqWYB4zuDV6/Fw9WqngMbQmIi8qeC1wvib3Oj9hP3NA=;
        b=LpuSM/ow3J9HRDM0/5Fm7MZVvW1wteFteR58Bl90sDudR/ezdNc+D8ScxUTZC8J1i2
         Fo0fjgkXYtWWAFaPMdi59Kw0Ntq8EKxwB9R8lJC/4G0zhRzEC6wkjmJXXKvhYmRaZ+3I
         PO3zwNe0ka9BMtL5wn3UOo9gSBOeNOW57GbdU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:cc:content-type;
        b=U/lJBM2f3+9RqzXVY2LVrRPkNcngzY9C+Y1uaS+GLENnehBVFpQBuhKlJj/nxKZGP+
         I3FahXpVlQMmuVBuIdXG/DlLaySJ7/UlcEZeLyETGaynhD4F6mAB5xqnV+fI1MhMf8cc
         XcF78EmwtIPhqf8GQFN+wxY4DopNv0oHmOjjs=
MIME-Version: 1.0
Received: by 10.90.4.34 with SMTP id 34mr12309616agd.140.1291851864742; Wed,
 08 Dec 2010 15:44:24 -0800 (PST)
Sender: anthony.urso@gmail.com
Received: by 10.90.78.20 with HTTP; Wed, 8 Dec 2010 15:44:24 -0800 (PST)
In-Reply-To: <AANLkTi=gAobRq2Vi23ZjvC1KFcQKUs3V+rg88GKpaOop@mail.gmail.com>
References: <AANLkTikNNrH2ySg2_JcuDCRATiP33LqXoZoKg8tAmMsn@mail.gmail.com>
	<AANLkTi=gAobRq2Vi23ZjvC1KFcQKUs3V+rg88GKpaOop@mail.gmail.com>
Date: Wed, 8 Dec 2010 15:44:24 -0800
X-Google-Sender-Auth: vPMA-CzqhH2CIW5f56sFgTi7mSY
Message-ID: <AANLkTi=ALY2YZVg_eqQ_DAZNntVNb+t7xoWEGQO=BNs8@mail.gmail.com>
Subject: Re: Announcing KeptCollections, distributed Java Collections for ZooKeeper
From: Anthony Urso <anthonyu@cs.ucla.edu>
To: general@hadoop.apache.org
Cc: user@zookeeper.apache.org, dev@zookeeper.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Oh, thanks for the heads up on the race conditions.  I'd like to
eliminate them all, so let me know specifics and I will work on them,
or just send a patch if you see an obvious solution.

Thanks for the feedback!
Anthony

On Wed, Dec 8, 2010 at 3:15 PM, Ted Dunning <ted.dunning@gmail.com> wrote:
> This looks very useful and looks like nice work.
>
> I note that the methods used are prone to race conditions, but if you are
> just thinking about shared maps, this probably isn't important.
>
> On Wed, Dec 8, 2010 at 12:40 PM, Anthony Urso <anthony.urso@gmail.com>wrote:
>
>> I am pleased to announce the initial release of KeptCollections, a
>> library of drop-in replacements for standard Java Collections that use
>> Apache ZooKeeper as a backing store.
>>
>> KeptCollections are designed to make it easy for anyone to write
>> distributed applications without having to learn the intricacies of
>> ZooKeeper, or distributed programming in general.
>>
>> The collections use the well-known JDK APIs, yet any changes made to
>> any of these collections by one node are seen by all other nodes
>> within milliseconds, allowing for easy communication between processes in a
>> computing cluster.
>>
>> More information here:
>>
>> https://github.com/anthonyu/KeptCollections/wiki
>>
>> and all code is available from:
>>
>> https://github.com/anthonyu/KeptCollections
>>
>> Please try it out, and let me know any problems you experience via
>> github issues or this email address.
>>
>> Cheers,
>> Anthony
>>
>

From general-return-2513-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 08 23:48:53 2010
Return-Path: <general-return-2513-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91734 invoked from network); 8 Dec 2010 23:48:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Dec 2010 23:48:53 -0000
Received: (qmail 76191 invoked by uid 500); 8 Dec 2010 23:48:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76113 invoked by uid 500); 8 Dec 2010 23:48:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76080 invoked by uid 99); 8 Dec 2010 23:48:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Dec 2010 23:48:51 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of anthony.urso@gmail.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Dec 2010 23:48:45 +0000
Received: by gyf1 with SMTP id 1so1191643gyf.35
        for <general@hadoop.apache.org>; Wed, 08 Dec 2010 15:48:24 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:cc:content-type;
        bh=C2oPoXSpY7deqMAeC5iW0S8ul4peUFfktkKLuSZozjE=;
        b=LXdytWI26oqZAHVtRXSgipYV7WbjRrJP5tgfln/A/XsPKksS1ER2wMKG97roJogUsG
         IA0Emeex7/ZQanr2GOYEzJJzxA/didx79uVrcKYh1nd2GLt8rzf4o/XNvLZWggs1vVln
         RqBLKJcbrjU2zpYcrOe4RTlqNmnGXPmvLIf4w=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:cc:content-type;
        b=rbH6EXcZCUAlhNMka5ZJdqRh2uyNfynKSs6szerTJZvMnO7npukEd5hE2MroOBDIY3
         Q6p4NDvd0clxYBYqf60RpIyknn5it8DnBjVEq117EAt0hpbi7Z2Bnvud4fn+OrkYgENL
         JQEXNnDAGn5qnTtBRjjabrwjvY1c1QLuaoxj4=
MIME-Version: 1.0
Received: by 10.90.75.18 with SMTP id x18mr12510390aga.5.1291852104030; Wed,
 08 Dec 2010 15:48:24 -0800 (PST)
Sender: anthony.urso@gmail.com
Received: by 10.90.78.20 with HTTP; Wed, 8 Dec 2010 15:48:23 -0800 (PST)
In-Reply-To: <AANLkTi=tekqH557XC8tqRLynY8mh_6teGogOaYyVasfK@mail.gmail.com>
References: <AANLkTikNNrH2ySg2_JcuDCRATiP33LqXoZoKg8tAmMsn@mail.gmail.com>
	<AANLkTi=gAobRq2Vi23ZjvC1KFcQKUs3V+rg88GKpaOop@mail.gmail.com>
	<AANLkTi=tekqH557XC8tqRLynY8mh_6teGogOaYyVasfK@mail.gmail.com>
Date: Wed, 8 Dec 2010 15:48:23 -0800
X-Google-Sender-Auth: QRohcojWjjc0MrwyRM9Ns5RTBYE
Message-ID: <AANLkTinLOFffn8iiZJtUnKH6w10qEW+v-zJ8aHFZkj47@mail.gmail.com>
Subject: Re: Announcing KeptCollections, distributed Java Collections for ZooKeeper
From: Anthony Urso <anthonyu@cs.ucla.edu>
To: dev@zookeeper.apache.org
Cc: user@zookeeper.apache.org, general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Eric:

This is pretty different from redis, but a Java Collections interface
to redis would be awesome, too.

Cheers,
Anthony

On Wed, Dec 8, 2010 at 3:33 PM, Eric Hauser <ewhauser@gmail.com> wrote:
> Out of curiosity, why not just use Redis for this?
>
> On Wed, Dec 8, 2010 at 6:15 PM, Ted Dunning <ted.dunning@gmail.com> wrote:
>> This looks very useful and looks like nice work.
>>
>> I note that the methods used are prone to race conditions, but if you are
>> just thinking about shared maps, this probably isn't important.
>>
>> On Wed, Dec 8, 2010 at 12:40 PM, Anthony Urso <anthony.urso@gmail.com>wrote:
>>
>>> I am pleased to announce the initial release of KeptCollections, a
>>> library of drop-in replacements for standard Java Collections that use
>>> Apache ZooKeeper as a backing store.
>>>
>>> KeptCollections are designed to make it easy for anyone to write
>>> distributed applications without having to learn the intricacies of
>>> ZooKeeper, or distributed programming in general.
>>>
>>> The collections use the well-known JDK APIs, yet any changes made to
>>> any of these collections by one node are seen by all other nodes
>>> within milliseconds, allowing for easy communication between processes in a
>>> computing cluster.
>>>
>>> More information here:
>>>
>>> https://github.com/anthonyu/KeptCollections/wiki
>>>
>>> and all code is available from:
>>>
>>> https://github.com/anthonyu/KeptCollections
>>>
>>> Please try it out, and let me know any problems you experience via
>>> github issues or this email address.
>>>
>>> Cheers,
>>> Anthony
>>>
>>
>

From general-return-2514-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 08 23:56:40 2010
Return-Path: <general-return-2514-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95545 invoked from network); 8 Dec 2010 23:56:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Dec 2010 23:56:40 -0000
Received: (qmail 90016 invoked by uid 500); 8 Dec 2010 23:56:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89814 invoked by uid 500); 8 Dec 2010 23:56:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89806 invoked by uid 99); 8 Dec 2010 23:56:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Dec 2010 23:56:38 +0000
X-ASF-Spam-Status: No, hits=4.0 required=10.0
	tests=FREEMAIL_FROM,FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ted.dunning@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Dec 2010 23:56:32 +0000
Received: by wwd20 with SMTP id 20so1557019wwd.29
        for <general@hadoop.apache.org>; Wed, 08 Dec 2010 15:56:12 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:cc:content-type;
        bh=dghyyqBdacr+oHV7ROzA3x/eRM4kS586dawMzsAtIKc=;
        b=kA5Qu2QMqD6rb8NwY1g+G2D7R3J24zJjwZWK+ep8LjC+0+v8Cr5fUCqYLcjimdsa6F
         7Dynus+/sz/XLSE7jKJmhWJoxGsFRUDUth8DDsMn+qm7UEq6qrR6oDXc5bMUJjJFuX7y
         lElrgwOl6M3hjUJ1vHC+kcdg8SubzY0tfXZxI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :cc:content-type;
        b=aeZ3kkmkbVMT1YGN7b9S033aC0HjeDRuUTEwWK+uQwgxoGt6yFiWUTq56hjlfG5w+5
         5PsEH1l9Zurb47ZVc0UzP/KBx9qqq+AXv7ls11nOgBtxC6YAHEA77SAihS83O5K/UJEQ
         gJnnWGYp7s86Dg59zAYkNNDmZAD4zqxnEfSTw=
Received: by 10.216.162.70 with SMTP id x48mr2649647wek.4.1291852571698; Wed,
 08 Dec 2010 15:56:11 -0800 (PST)
MIME-Version: 1.0
Received: by 10.216.158.68 with HTTP; Wed, 8 Dec 2010 15:55:51 -0800 (PST)
In-Reply-To: <AANLkTi=ALY2YZVg_eqQ_DAZNntVNb+t7xoWEGQO=BNs8@mail.gmail.com>
References: <AANLkTikNNrH2ySg2_JcuDCRATiP33LqXoZoKg8tAmMsn@mail.gmail.com>
 <AANLkTi=gAobRq2Vi23ZjvC1KFcQKUs3V+rg88GKpaOop@mail.gmail.com> <AANLkTi=ALY2YZVg_eqQ_DAZNntVNb+t7xoWEGQO=BNs8@mail.gmail.com>
From: Ted Dunning <ted.dunning@gmail.com>
Date: Wed, 8 Dec 2010 15:55:51 -0800
Message-ID: <AANLkTinj1GteucbCqe227v1wpqfyQ=R2_ufJYnjTo=wq@mail.gmail.com>
Subject: Re: Announcing KeptCollections, distributed Java Collections for ZooKeeper
To: user@zookeeper.apache.org
Cc: general@hadoop.apache.org, dev@zookeeper.apache.org
Content-Type: multipart/alternative; boundary=001485f44f32a367770496eedd62
X-Virus-Checked: Checked by ClamAV on apache.org

--001485f44f32a367770496eedd62
Content-Type: text/plain; charset=UTF-8

I filed an issue with a description and a suggested alternative.

On Wed, Dec 8, 2010 at 3:44 PM, Anthony Urso <anthonyu@cs.ucla.edu> wrote:

> Oh, thanks for the heads up on the race conditions.  I'd like to
> eliminate them all, so let me know specifics and I will work on them,
> or just send a patch if you see an obvious solution.
>
> Thanks for the feedback!
> Anthony
>
> On Wed, Dec 8, 2010 at 3:15 PM, Ted Dunning <ted.dunning@gmail.com> wrote:
> > This looks very useful and looks like nice work.
> >
> > I note that the methods used are prone to race conditions, but if you are
> > just thinking about shared maps, this probably isn't important.
> >
> > On Wed, Dec 8, 2010 at 12:40 PM, Anthony Urso <anthony.urso@gmail.com
> >wrote:
> >
> >> I am pleased to announce the initial release of KeptCollections, a
> >> library of drop-in replacements for standard Java Collections that use
> >> Apache ZooKeeper as a backing store.
> >>
> >> KeptCollections are designed to make it easy for anyone to write
> >> distributed applications without having to learn the intricacies of
> >> ZooKeeper, or distributed programming in general.
> >>
> >> The collections use the well-known JDK APIs, yet any changes made to
> >> any of these collections by one node are seen by all other nodes
> >> within milliseconds, allowing for easy communication between processes
> in a
> >> computing cluster.
> >>
> >> More information here:
> >>
> >> https://github.com/anthonyu/KeptCollections/wiki
> >>
> >> and all code is available from:
> >>
> >> https://github.com/anthonyu/KeptCollections
> >>
> >> Please try it out, and let me know any problems you experience via
> >> github issues or this email address.
> >>
> >> Cheers,
> >> Anthony
> >>
> >
>

--001485f44f32a367770496eedd62--

From general-return-2515-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 09 00:00:39 2010
Return-Path: <general-return-2515-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97590 invoked from network); 9 Dec 2010 00:00:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Dec 2010 00:00:39 -0000
Received: (qmail 1628 invoked by uid 500); 9 Dec 2010 00:00:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1482 invoked by uid 500); 9 Dec 2010 00:00:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 98409 invoked by uid 99); 8 Dec 2010 23:58:57 -0000
X-ASF-Spam-Status: No, hits=4.0 required=10.0
	tests=FREEMAIL_FROM,FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of gaurav.gs.sharma@gmail.com designates 74.125.82.176 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:cc:content-type;
        bh=gJ1gyLpI8FQyvvDb5AfYZ8IV4LD70kReLwrGT15TKQM=;
        b=scvuqPGIqdS11QMLPKgQXJv2z4zGKn4LQnMHqMGBRHowTjgWMldMH15Xm910ctG+M7
         dnp2v0l1d2FYW1znawFkfeeTnUOFg+oWdm+4qOrmySsRDDwFv1uwNdoKjQwkOzYaRCuk
         AVT+cW4rzqIZBoKv54TYpdG8FwLDm5vYjNt2s=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :cc:content-type;
        b=i4YJA08Mbhuu8sfleQRYqKT3FTRTNQWl+uv53GmPZbdbvHRjCg7stI5stwYM7FYfXi
         50eHcnCw6y4yvcGkD50lg2MdBtQx0jv4eLwh70C+/rsGObRoaBVPotX7hVRVn8cFtD4h
         e8/BTeRdA/wmfnvndZVyos0VLGXHu9GWsbukY=
MIME-Version: 1.0
In-Reply-To: <AANLkTinLOFffn8iiZJtUnKH6w10qEW+v-zJ8aHFZkj47@mail.gmail.com>
References: <AANLkTikNNrH2ySg2_JcuDCRATiP33LqXoZoKg8tAmMsn@mail.gmail.com>
 <AANLkTi=gAobRq2Vi23ZjvC1KFcQKUs3V+rg88GKpaOop@mail.gmail.com>
 <AANLkTi=tekqH557XC8tqRLynY8mh_6teGogOaYyVasfK@mail.gmail.com> <AANLkTinLOFffn8iiZJtUnKH6w10qEW+v-zJ8aHFZkj47@mail.gmail.com>
From: Gaurav Sharma <gaurav.gs.sharma@gmail.com>
Date: Wed, 8 Dec 2010 18:58:10 -0500
Message-ID: <AANLkTimh50Fbww2dX57uM9xdvv_-Wvfc3MHj_JXAXQjz@mail.gmail.com>
Subject: Re: Announcing KeptCollections, distributed Java Collections for ZooKeeper
To: user@zookeeper.apache.org
Cc: dev@zookeeper.apache.org, general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364efa26f1fc140496eee514
X-Virus-Checked: Checked by ClamAV on apache.org

--0016364efa26f1fc140496eee514
Content-Type: text/plain; charset=ISO-8859-1

For those interested in a Redis Collections implementation, please take a
look here:
https://github.com/gsharma/johm/tree/master/src/main/java/redis/clients/johm

specifically the CollectionMap, CollectionSet, CollectionSortedSet,
CollectionList classes.

On Wed, Dec 8, 2010 at 6:48 PM, Anthony Urso <anthonyu@cs.ucla.edu> wrote:

> Eric:
>
> This is pretty different from redis, but a Java Collections interface
> to redis would be awesome, too.
>
> Cheers,
> Anthony
>
> On Wed, Dec 8, 2010 at 3:33 PM, Eric Hauser <ewhauser@gmail.com> wrote:
> > Out of curiosity, why not just use Redis for this?
> >
> > On Wed, Dec 8, 2010 at 6:15 PM, Ted Dunning <ted.dunning@gmail.com>
> wrote:
> >> This looks very useful and looks like nice work.
> >>
> >> I note that the methods used are prone to race conditions, but if you
> are
> >> just thinking about shared maps, this probably isn't important.
> >>
> >> On Wed, Dec 8, 2010 at 12:40 PM, Anthony Urso <anthony.urso@gmail.com
> >wrote:
> >>
> >>> I am pleased to announce the initial release of KeptCollections, a
> >>> library of drop-in replacements for standard Java Collections that use
> >>> Apache ZooKeeper as a backing store.
> >>>
> >>> KeptCollections are designed to make it easy for anyone to write
> >>> distributed applications without having to learn the intricacies of
> >>> ZooKeeper, or distributed programming in general.
> >>>
> >>> The collections use the well-known JDK APIs, yet any changes made to
> >>> any of these collections by one node are seen by all other nodes
> >>> within milliseconds, allowing for easy communication between processes
> in a
> >>> computing cluster.
> >>>
> >>> More information here:
> >>>
> >>> https://github.com/anthonyu/KeptCollections/wiki
> >>>
> >>> and all code is available from:
> >>>
> >>> https://github.com/anthonyu/KeptCollections
> >>>
> >>> Please try it out, and let me know any problems you experience via
> >>> github issues or this email address.
> >>>
> >>> Cheers,
> >>> Anthony
> >>>
> >>
> >
>

--0016364efa26f1fc140496eee514--

From general-return-2516-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 09 00:01:12 2010
Return-Path: <general-return-2516-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97828 invoked from network); 9 Dec 2010 00:01:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Dec 2010 00:01:11 -0000
Received: (qmail 3314 invoked by uid 500); 9 Dec 2010 00:01:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2995 invoked by uid 500); 9 Dec 2010 00:01:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2972 invoked by uid 99); 9 Dec 2010 00:01:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Dec 2010 00:01:06 +0000
X-ASF-Spam-Status: No, hits=4.0 required=10.0
	tests=FREEMAIL_FROM,FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of gaurav.gs.sharma@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Dec 2010 00:00:59 +0000
Received: by wwd20 with SMTP id 20so1560489wwd.29
        for <general@hadoop.apache.org>; Wed, 08 Dec 2010 16:00:39 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:cc:content-type;
        bh=VCEtBDnCv/bl6AVleaFJRD2Q39Rk5ez/q6KR1gYAlOc=;
        b=Iiu4PAojoVjBj+rBNXrCfGLCbFB3HmDHH59RKuDjOYoSbC21U/K5EDCwWSms5uUFs8
         mDdsBfuXn8Y00403aRvNoUOsNd/NQFFq9K9KF47WzdK7lVQ+08MBg+N7qWl8stnd5OWE
         E6D3LwIRnlw3JyH2owqB1dqE2PsPIQisNse88=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :cc:content-type;
        b=AQYVPSjClrjnah5eKJYDyK/AcXcdIW1xIX8Z1hWR/hpgSqsPN1ZbRrlITUrouoOXV0
         8N59nkfQakkpUfrvUXiY5EZ6lw32SCLD3R3rCxBgG2ifIiX4nMbl90T9XALpT5oxaufG
         PFpmXob/OsE9YpmDhjWNemsXoLrKcTWaT22hs=
Received: by 10.216.162.84 with SMTP id x62mr120029wek.106.1291852839525; Wed,
 08 Dec 2010 16:00:39 -0800 (PST)
MIME-Version: 1.0
Received: by 10.216.62.74 with HTTP; Wed, 8 Dec 2010 16:00:19 -0800 (PST)
In-Reply-To: <AANLkTimh50Fbww2dX57uM9xdvv_-Wvfc3MHj_JXAXQjz@mail.gmail.com>
References: <AANLkTikNNrH2ySg2_JcuDCRATiP33LqXoZoKg8tAmMsn@mail.gmail.com>
 <AANLkTi=gAobRq2Vi23ZjvC1KFcQKUs3V+rg88GKpaOop@mail.gmail.com>
 <AANLkTi=tekqH557XC8tqRLynY8mh_6teGogOaYyVasfK@mail.gmail.com>
 <AANLkTinLOFffn8iiZJtUnKH6w10qEW+v-zJ8aHFZkj47@mail.gmail.com> <AANLkTimh50Fbww2dX57uM9xdvv_-Wvfc3MHj_JXAXQjz@mail.gmail.com>
From: Gaurav Sharma <gaurav.gs.sharma@gmail.com>
Date: Wed, 8 Dec 2010 19:00:19 -0500
Message-ID: <AANLkTi=igwKyA=r63KOEY3f6qMRF8_5xxGzm=rtYandz@mail.gmail.com>
Subject: Re: Announcing KeptCollections, distributed Java Collections for ZooKeeper
To: user@zookeeper.apache.org
Cc: dev@zookeeper.apache.org, general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364580109a1f370496eeed08
X-Virus-Checked: Checked by ClamAV on apache.org

--0016364580109a1f370496eeed08
Content-Type: text/plain; charset=ISO-8859-1

Forgot to mention that the backing implementations are here:
https://github.com/gsharma/johm/tree/master/src/main/java/redis/clients/johm/collections

and you can try our JOhm from the latest jar here:
https://github.com/downloads/xetorthio/johm/johm-0.4.0.jar

On Wed, Dec 8, 2010 at 6:58 PM, Gaurav Sharma <gaurav.gs.sharma@gmail.com>wrote:

> For those interested in a Redis Collections implementation, please take a
> look here:
>
> https://github.com/gsharma/johm/tree/master/src/main/java/redis/clients/johm
>
> specifically the CollectionMap, CollectionSet, CollectionSortedSet,
> CollectionList classes.
>
>
> On Wed, Dec 8, 2010 at 6:48 PM, Anthony Urso <anthonyu@cs.ucla.edu> wrote:
>
>> Eric:
>>
>> This is pretty different from redis, but a Java Collections interface
>> to redis would be awesome, too.
>>
>> Cheers,
>> Anthony
>>
>> On Wed, Dec 8, 2010 at 3:33 PM, Eric Hauser <ewhauser@gmail.com> wrote:
>> > Out of curiosity, why not just use Redis for this?
>> >
>> > On Wed, Dec 8, 2010 at 6:15 PM, Ted Dunning <ted.dunning@gmail.com>
>> wrote:
>> >> This looks very useful and looks like nice work.
>> >>
>> >> I note that the methods used are prone to race conditions, but if you
>> are
>> >> just thinking about shared maps, this probably isn't important.
>> >>
>> >> On Wed, Dec 8, 2010 at 12:40 PM, Anthony Urso <anthony.urso@gmail.com
>> >wrote:
>> >>
>> >>> I am pleased to announce the initial release of KeptCollections, a
>> >>> library of drop-in replacements for standard Java Collections that use
>> >>> Apache ZooKeeper as a backing store.
>> >>>
>> >>> KeptCollections are designed to make it easy for anyone to write
>> >>> distributed applications without having to learn the intricacies of
>> >>> ZooKeeper, or distributed programming in general.
>> >>>
>> >>> The collections use the well-known JDK APIs, yet any changes made to
>> >>> any of these collections by one node are seen by all other nodes
>> >>> within milliseconds, allowing for easy communication between processes
>> in a
>> >>> computing cluster.
>> >>>
>> >>> More information here:
>> >>>
>> >>> https://github.com/anthonyu/KeptCollections/wiki
>> >>>
>> >>> and all code is available from:
>> >>>
>> >>> https://github.com/anthonyu/KeptCollections
>> >>>
>> >>> Please try it out, and let me know any problems you experience via
>> >>> github issues or this email address.
>> >>>
>> >>> Cheers,
>> >>> Anthony
>> >>>
>> >>
>> >
>>
>
>

--0016364580109a1f370496eeed08--

From general-return-2517-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 09 21:42:26 2010
Return-Path: <general-return-2517-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 90043 invoked from network); 9 Dec 2010 21:42:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Dec 2010 21:42:25 -0000
Received: (qmail 56001 invoked by uid 500); 9 Dec 2010 21:42:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55750 invoked by uid 500); 9 Dec 2010 21:42:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 34678 invoked by uid 99); 9 Dec 2010 21:27:44 -0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of adam.rosien@gmail.com designates 74.125.82.48 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:cc:content-type;
        bh=/cSIWXYBkfRF7NPIZyxDkCB+iuWMVqnPEikjYlkBNG0=;
        b=mYHbHn9R7TqY95GTHsIjLIbZR07emSWh6TddvhkTi3j8JMiWYFHmpcj5Fq7mjqAcm4
         H2QEdZj8qRzeeIRUN20siuu1KdCIfu3bZ/Gde+dtRpjeiVUpOXk8LYlQUusT25a7CjKH
         fSVb4n/V/HP8LxiumAraBb1AH4sioNQs7pRVY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:cc:content-type;
        b=AU135+ZVZhC1AySwcT7/BAQt3sDJb9B4qJO/Qy8FYEjuxx5KC56CoW3TLrc0V4QL9E
         2XSOud6eyYXgjMUcSzFldXAFeXJ8SGy2t/6ZVDFpg9PV8bNZbzIe0ZyHXLQDe0YyUvIk
         IY6oASstGn/uS1OoV4bpqsLPg0TSzKcaGbkEQ=
MIME-Version: 1.0
Sender: adam.rosien@gmail.com
In-Reply-To: <AANLkTikNNrH2ySg2_JcuDCRATiP33LqXoZoKg8tAmMsn@mail.gmail.com>
References: <AANLkTikNNrH2ySg2_JcuDCRATiP33LqXoZoKg8tAmMsn@mail.gmail.com>
Date: Thu, 9 Dec 2010 13:27:17 -0800
X-Google-Sender-Auth: AaCCglpm6J0tt0RVw3r0qDQjWQM
Message-ID: <AANLkTim9Tq64z-tPsq38q3EGPo71zG-bfFe9cZwmtpJU@mail.gmail.com>
Subject: Re: Announcing KeptCollections, distributed Java Collections for ZooKeeper
From: Adam Rosien <adam@rosien.net>
To: user@zookeeper.apache.org
Cc: dev@zookeeper.apache.org, general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Making a distributed system look like a collection is a very dubious
proposition in my opinion. They are fundamentally different
abstractions and reminds me of
http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Computing.

.. Adam

On Wed, Dec 8, 2010 at 12:40 PM, Anthony Urso <anthony.urso@gmail.com> wrote:
> I am pleased to announce the initial release of KeptCollections, a
> library of drop-in replacements for standard Java Collections that use
> Apache ZooKeeper as a backing store.
>
> KeptCollections are designed to make it easy for anyone to write
> distributed applications without having to learn the intricacies of
> ZooKeeper, or distributed programming in general.
>
> The collections use the well-known JDK APIs, yet any changes made to
> any of these collections by one node are seen by all other nodes
> within milliseconds, allowing for easy communication between processes in a
> computing cluster.
>
> More information here:
>
> https://github.com/anthonyu/KeptCollections/wiki
>
> and all code is available from:
>
> https://github.com/anthonyu/KeptCollections
>
> Please try it out, and let me know any problems you experience via
> github issues or this email address.
>
> Cheers,
> Anthony
>

From general-return-2518-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 09 22:20:47 2010
Return-Path: <general-return-2518-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 10521 invoked from network); 9 Dec 2010 22:20:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Dec 2010 22:20:46 -0000
Received: (qmail 18509 invoked by uid 500); 9 Dec 2010 22:20:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18453 invoked by uid 500); 9 Dec 2010 22:20:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18445 invoked by uid 99); 9 Dec 2010 22:20:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Dec 2010 22:20:44 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [98.138.90.71] (HELO nm8.bullet.mail.ne1.yahoo.com) (98.138.90.71)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 09 Dec 2010 22:20:36 +0000
Received: from [98.138.90.48] by nm8.bullet.mail.ne1.yahoo.com with NNFMP; 09 Dec 2010 22:20:15 -0000
Received: from [98.138.89.160] by tm1.bullet.mail.ne1.yahoo.com with NNFMP; 09 Dec 2010 22:20:15 -0000
Received: from [127.0.0.1] by omp1016.mail.ne1.yahoo.com with NNFMP; 09 Dec 2010 22:20:15 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 653981.69420.bm@omp1016.mail.ne1.yahoo.com
Received: (qmail 96553 invoked by uid 60001); 9 Dec 2010 22:20:15 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1291933215; bh=T5nHlJM8oH0uqQPTHKU22Z+ImgqkbgjLnoUj8PYZkx0=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=OF4zJrq10vcqyxxcOEjvkm3bjBMAcarELr1MVSkW30RPX2xEo1kUYLGdv+hu7LwrYBMCCxRCJr0VSiFuMfjkA6Pf5X216u7F5fAmRnYXc9vddXqiQh2G2e4N4icruYwagMXyUSRLopjEO/a/V8kEIlaILWy6G1PuFQlbKZytwVY=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=BryY8ZNFYoX3S2SpmaEYvLIyMK07dckpy/E8Lt285fy4nSQlU5qJL+QLjVIP4AxbNkWgKPczuNKsz/0IFRT8gz4P8WNXSZELcXKF2rKBoNStCL9GBOr7zMqwUIJdlj1imHRFaQTkcbiaOfiq/vjBnnZv2+u5xA/+9Yj9BorqGkU=;
Message-ID: <288424.95491.qm@web114514.mail.gq1.yahoo.com>
X-YMail-OSG: xRnNJ6EVM1kJXhShYC8P9dT5QDcUOS3cF85WWaj3fs4sQfe
 kMZKBjSV_LkcqRqErMudt0sjfWHNN_4kPk1Uyg_vZiureAOv9lyOqbnsNuzv
 50yXbu0ESVzM0qfKBwka_MEysigKyjgCvBYXoljoNwC7Bp7E_Ul8ro7o898l
 iBdZ1P80xhGt0jTTcBGsO19SYSuUbC_b.0Px3HX_P.sJAHIeKIdTkT7T9C28
 44GGNlTPXCOXMyi3X8UQbR5P.jG.NQYVYHCcflLDPF6Y.le52uYap52G193t
 i1ByxNvrwpv4nIQMaKOfm5RGMxTEs0Su9PIlnY.76ezdGXocUwQ--
Received: from [67.180.84.177] by web114514.mail.gq1.yahoo.com via HTTP; Thu, 09 Dec 2010 14:20:14 PST
X-Mailer: YahooMailRC/553 YahooMailWebService/0.8.107.285259
Date: Thu, 9 Dec 2010 14:20:14 -0800 (PST)
From: C J <c.joshi26@yahoo.com>
Subject: Multiple hadoop configurations
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1172938479-1291933214=:95491"

--0-1172938479-1291933214=:95491
Content-Type: text/plain; charset=us-ascii

Hi,

I have 2 hadoop clusters . Both these clusters have their own set of jobs 
running. I also have some distcp jobs which copy over data from one cluster to 
another.

I want to be able to control the jobs on both the clusters through one scheduler 
(so I can coordinate the jobs).

I am wondering when I need to trigger a job through a scheduler, how can I send 
to one of the 2 clusters. 


2 clusters means 2 sets of configuration files.Is there any way to get a handle 
to one of these clusters, by specifying the configuration file name or 
something?

Will appreciate any help or clues.

Thanks.



      
--0-1172938479-1291933214=:95491--

From general-return-2519-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 09 22:25:25 2010
Return-Path: <general-return-2519-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 11405 invoked from network); 9 Dec 2010 22:25:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Dec 2010 22:25:25 -0000
Received: (qmail 27087 invoked by uid 500); 9 Dec 2010 22:25:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27044 invoked by uid 500); 9 Dec 2010 22:25:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27036 invoked by uid 99); 9 Dec 2010 22:25:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Dec 2010 22:25:23 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Dec 2010 22:25:18 +0000
Received: by wye20 with SMTP id 20so2963321wye.35
        for <general@hadoop.apache.org>; Thu, 09 Dec 2010 14:24:54 -0800 (PST)
Received: by 10.216.163.203 with SMTP id a53mr2312242wel.104.1291933494879;
 Thu, 09 Dec 2010 14:24:54 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.21.83 with HTTP; Thu, 9 Dec 2010 14:24:34 -0800 (PST)
In-Reply-To: <288424.95491.qm@web114514.mail.gq1.yahoo.com>
References: <288424.95491.qm@web114514.mail.gq1.yahoo.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Thu, 9 Dec 2010 14:24:34 -0800
X-Google-Sender-Auth: CzotEMnsB4SBYBaHbIPfGpd8Fr4
Message-ID: <AANLkTinxisAkvE-atsuRMf6hbEMM6otiMr0H6R+Bz+_4@mail.gmail.com>
Subject: Re: Multiple hadoop configurations
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

I believe the answer you are looking for is Oozie coordinator, but I
am not which version of it supports multiple clusters.

On Thu, Dec 9, 2010 at 14:20, C J <c.joshi26@yahoo.com> wrote:
> Hi,
>
> I have 2 hadoop clusters . Both these clusters have their own set of jobs
> running. I also have some distcp jobs which copy over data from one cluster to
> another.
>
> I want to be able to control the jobs on both the clusters through one scheduler
> (so I can coordinate the jobs).
>
> I am wondering when I need to trigger a job through a scheduler, how can I send
> to one of the 2 clusters.
>
>
> 2 clusters means 2 sets of configuration files.Is there any way to get a handle
> to one of these clusters, by specifying the configuration file name or
> something?
>
> Will appreciate any help or clues.
>
> Thanks.
>
>
>
>

From general-return-2520-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 09 23:02:06 2010
Return-Path: <general-return-2520-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54870 invoked from network); 9 Dec 2010 23:02:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Dec 2010 23:02:06 -0000
Received: (qmail 78791 invoked by uid 500); 9 Dec 2010 23:02:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78747 invoked by uid 500); 9 Dec 2010 23:02:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78737 invoked by uid 99); 9 Dec 2010 23:02:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Dec 2010 23:02:03 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [98.139.91.209] (HELO nm15-vm1.bullet.mail.sp2.yahoo.com) (98.139.91.209)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 09 Dec 2010 23:01:55 +0000
Received: from [98.139.91.65] by nm15.bullet.mail.sp2.yahoo.com with NNFMP; 09 Dec 2010 23:01:34 -0000
Received: from [98.139.91.13] by tm5.bullet.mail.sp2.yahoo.com with NNFMP; 09 Dec 2010 23:01:33 -0000
Received: from [127.0.0.1] by omp1013.mail.sp2.yahoo.com with NNFMP; 09 Dec 2010 23:01:33 -0000
X-Yahoo-Newman-Property: ymail-5
X-Yahoo-Newman-Id: 952989.40101.bm@omp1013.mail.sp2.yahoo.com
Received: (qmail 12566 invoked by uid 60001); 9 Dec 2010 23:01:33 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1291935693; bh=nz6x81XnKXJK/7+DF+j8NnLYeeiRYodIUcqlo9rfMU8=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=0hlWBDg/m9OaBb1b0GBLbjpUXTSh72SKV8EW+mIEGPtMEhH0OITCFOSXpPAlmdxrFe6HII4IOy0pvdbdStBJSVH4qz1KB9vI9B//kzKX8r0edzNZhElED8kq3e9OaoTqbY9wizIEmpaw7DRE8T7kQ767aZfu4gqhbZhR6bgEqJ8=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=N6GUYbeYwjLaNNg2zvRSkHXgdp+MITJjPAIyXklS8mbRoZl3g+/u4BhhzWQxSTX6CTS/JFLavtlCPNCSp0q2eZYPz+SvnWVIPtf460+FzAFASA8W/BIszIhzd9s3PV1j7i79meR5dEhBaJSQKi8go7QJc6OmE5R+OmCosIVp7eg=;
Message-ID: <422363.10927.qm@web114517.mail.gq1.yahoo.com>
X-YMail-OSG: 3uGcZL0VM1mAfgiki69fPYVrRXELfs.9mQhOEjGmDq5PWQ_
 liE_7WyelZrmsG51Rub5yHdk5_gY.NWxzDMLjbl9jt.OqVZTP5rxRaYkOqYq
 y2OKCJ5C57K.4XIrxAgdYGdPcYarWBb4gB0jqMLthATp6Zg7dge9FmSflf8S
 EHH0O.ojsIlpU9Nh4pJK6WBRggGtwOWLN94kH6jfZpJnSCvkAVRGM4xr.Jpu
 _ddp.9rTf97FxX3_4ws22oRHQ5fuORAIBh4vSLqYobGJNPl0YmQAWnNxvut1
 cQps2tiAx4pR6bIewaPL9eEJv0Pq3QL_nbFOke.J.YD1JEsAoHg--
Received: from [67.180.84.177] by web114517.mail.gq1.yahoo.com via HTTP; Thu, 09 Dec 2010 15:01:32 PST
X-Mailer: YahooMailRC/553 YahooMailWebService/0.8.107.285259
References: <288424.95491.qm@web114514.mail.gq1.yahoo.com> <AANLkTinxisAkvE-atsuRMf6hbEMM6otiMr0H6R+Bz+_4@mail.gmail.com>
Date: Thu, 9 Dec 2010 15:01:32 -0800 (PST)
From: C J <c.joshi26@yahoo.com>
Subject: Re: Multiple hadoop configurations
To: general@hadoop.apache.org
In-Reply-To: <AANLkTinxisAkvE-atsuRMf6hbEMM6otiMr0H6R+Bz+_4@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1403983603-1291935692=:10927"

--0-1403983603-1291935692=:10927
Content-Type: text/plain; charset=us-ascii

Thanks Konstantin. My issue is more with supporting the multiple clusters . I am 
using quartz for scheduling the jobs (currently it is 2 separate schedulers for 
the 2 clusters). 


If while submitting a job I am able to specify which cluster it should trigger 
the job on (by giving the handle to the appropriate cluster), I think I will be 
able to manage it.

Thanks,
Deepika




________________________________
From: Konstantin Boudnik <cos@apache.org>
To: general@hadoop.apache.org
Sent: Thu, December 9, 2010 2:24:34 PM
Subject: Re: Multiple hadoop configurations

I believe the answer you are looking for is Oozie coordinator, but I
am not which version of it supports multiple clusters.

On Thu, Dec 9, 2010 at 14:20, C J <c.joshi26@yahoo.com> wrote:
> Hi,
>
> I have 2 hadoop clusters . Both these clusters have their own set of jobs
> running. I also have some distcp jobs which copy over data from one cluster to
> another.
>
> I want to be able to control the jobs on both the clusters through one 
>scheduler
> (so I can coordinate the jobs).
>
> I am wondering when I need to trigger a job through a scheduler, how can I 
send
> to one of the 2 clusters.
>
>
> 2 clusters means 2 sets of configuration files.Is there any way to get a 
handle
> to one of these clusters, by specifying the configuration file name or
> something?
>
> Will appreciate any help or clues.
>
> Thanks.
>
>
>
>



      
--0-1403983603-1291935692=:10927--

From general-return-2521-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 09 23:23:04 2010
Return-Path: <general-return-2521-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68595 invoked from network); 9 Dec 2010 23:23:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Dec 2010 23:23:03 -0000
Received: (qmail 99090 invoked by uid 500); 9 Dec 2010 23:23:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99014 invoked by uid 500); 9 Dec 2010 23:23:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98878 invoked by uid 99); 9 Dec 2010 23:23:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Dec 2010 23:23:01 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ted.dunning@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Dec 2010 23:22:55 +0000
Received: by wwd20 with SMTP id 20so2781626wwd.29
        for <general@hadoop.apache.org>; Thu, 09 Dec 2010 15:22:35 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:cc:content-type;
        bh=2ROuq9G0xn1CaeK2Uo30djgc8U9MHT5JSLGECcaX/Kc=;
        b=Bz+YuC9duXFBUKjdXxVLPVGYgwF3pTc2Zi3ng6ZRsvJWf0XiBHCzyQ+HktyhjQr4fh
         AvkWYCMjzrrnMsoe3MH8LoMg4RJn0WytUo+d/qUWJJVVUBENtguT4CJdWmgC3RCpM0b3
         HuYcWchqM6RGaRSy4BR/Vj1svzC/xjtcQx0Is=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :cc:content-type;
        b=AwDhVWDqK1h0ya5ldiowFw+BHQTXoscvbh8GmjiWIAFm6tYblCccK9XrKabDgNd3xa
         H63r/BKQ7x2npJpJfsgGRTR0V0XzEQxi6R7INnQnTSB8I2Ywlf53VQywIOsGn71mAaei
         +RxE9l2xvZRNGYjvp48uXkS+4hBpnkSvWhd6w=
Received: by 10.216.162.70 with SMTP id x48mr1273070wek.4.1291936955467; Thu,
 09 Dec 2010 15:22:35 -0800 (PST)
MIME-Version: 1.0
Received: by 10.216.158.68 with HTTP; Thu, 9 Dec 2010 15:22:15 -0800 (PST)
In-Reply-To: <AANLkTim9Tq64z-tPsq38q3EGPo71zG-bfFe9cZwmtpJU@mail.gmail.com>
References: <AANLkTikNNrH2ySg2_JcuDCRATiP33LqXoZoKg8tAmMsn@mail.gmail.com> <AANLkTim9Tq64z-tPsq38q3EGPo71zG-bfFe9cZwmtpJU@mail.gmail.com>
From: Ted Dunning <ted.dunning@gmail.com>
Date: Thu, 9 Dec 2010 15:22:15 -0800
Message-ID: <AANLkTikU05yqj7Zmbdy-hBF3zk1HqsuRBzjjw-7vzinr@mail.gmail.com>
Subject: Re: Announcing KeptCollections, distributed Java Collections for ZooKeeper
To: user@zookeeper.apache.org
Cc: dev@zookeeper.apache.org, general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f44f324d87f7049702830a
X-Virus-Checked: Checked by ClamAV on apache.org

--001485f44f324d87f7049702830a
Content-Type: text/plain; charset=UTF-8

It definitely can go that way.

But some things are pretty nicely viewed that way.  For instance, wouldn't
it be nice to have a "live processes" set in different programs?  Of course,
you have to want watch for race conditions and you can't really believe it
is the same from moment to moment.  But with those caveats, it would be
*really* nice.

On Thu, Dec 9, 2010 at 1:27 PM, Adam Rosien <adam@rosien.net> wrote:

> Making a distributed system look like a collection is a very dubious
> proposition in my opinion. They are fundamentally different
> abstractions and reminds me of
> http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Computing.
>
> .. Adam
>
> On Wed, Dec 8, 2010 at 12:40 PM, Anthony Urso <anthony.urso@gmail.com>
> wrote:
> > I am pleased to announce the initial release of KeptCollections, a
> > library of drop-in replacements for standard Java Collections that use
> > Apache ZooKeeper as a backing store.
> >
> > KeptCollections are designed to make it easy for anyone to write
> > distributed applications without having to learn the intricacies of
> > ZooKeeper, or distributed programming in general.
> >
> > The collections use the well-known JDK APIs, yet any changes made to
> > any of these collections by one node are seen by all other nodes
> > within milliseconds, allowing for easy communication between processes in
> a
> > computing cluster.
> >
> > More information here:
> >
> > https://github.com/anthonyu/KeptCollections/wiki
> >
> > and all code is available from:
> >
> > https://github.com/anthonyu/KeptCollections
> >
> > Please try it out, and let me know any problems you experience via
> > github issues or this email address.
> >
> > Cheers,
> > Anthony
> >
>

--001485f44f324d87f7049702830a--

From general-return-2522-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Dec 10 01:10:48 2010
Return-Path: <general-return-2522-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25806 invoked from network); 10 Dec 2010 01:10:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Dec 2010 01:10:47 -0000
Received: (qmail 55542 invoked by uid 500); 10 Dec 2010 01:10:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55387 invoked by uid 500); 10 Dec 2010 01:10:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 25719 invoked by uid 99); 10 Dec 2010 00:46:48 -0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of adam.rosien@gmail.com designates 74.125.82.48 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:cc:content-type:content-transfer-encoding;
        bh=avAZwt0uH05asaEft3RyOseWxefCRgK/3AGnQom6OY0=;
        b=bpUzezYw4BdnZIeXrkeSIZBsfvySTo2Voqp+6KmI9Y/Y/xCmcgDf3L0eA8gHBjxKiv
         aTtU/n1nQSY7gX/62T5a9JM1RKrI3pVAb8fF1Uvyz4O3+H7qVUcluLBDkilKUv1Bv4J6
         YMXqDdPbQWZKluj4aDSEYYDK+GYOGj9s0thpc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:cc:content-type
         :content-transfer-encoding;
        b=ThhBI+5Dp2cGb4Y12fKgSxELveSK/1C9Zkpi8NchA/feeG+OL+tCfirc3WgoSBkfpG
         syqKRRI8H9Tmtmc95zmqNJu6tbbY5/bkx4h03nZgT8rR6lN0tKW0boiwipQlWuyWJgHf
         PEymLiJ9DQyAwEz0tDeTQCPsC19IsbwvExmus=
MIME-Version: 1.0
Sender: adam.rosien@gmail.com
In-Reply-To: <AANLkTikU05yqj7Zmbdy-hBF3zk1HqsuRBzjjw-7vzinr@mail.gmail.com>
References: <AANLkTikNNrH2ySg2_JcuDCRATiP33LqXoZoKg8tAmMsn@mail.gmail.com>
	<AANLkTim9Tq64z-tPsq38q3EGPo71zG-bfFe9cZwmtpJU@mail.gmail.com>
	<AANLkTikU05yqj7Zmbdy-hBF3zk1HqsuRBzjjw-7vzinr@mail.gmail.com>
Date: Thu, 9 Dec 2010 16:46:23 -0800
X-Google-Sender-Auth: uyVFESgqo3kd6X-b8v7duHbTTXc
Message-ID: <AANLkTinPH9WeK_0mSo7FVUFQdFos0M60H80KaF6VU=Ny@mail.gmail.com>
Subject: Re: Announcing KeptCollections, distributed Java Collections for ZooKeeper
From: Adam Rosien <adam@rosien.net>
To: user@zookeeper.apache.org
Cc: dev@zookeeper.apache.org, general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Yeah, perhaps I was a bit harsh. In my own systems I specifically
create new types that hold zk-managed data structures, e.g.
ZkSynced<T>, to inform clients that the semantics are different. You
can get copies of the current value which lets clients know the value
may change at any point.

.. Adam

On Thu, Dec 9, 2010 at 3:22 PM, Ted Dunning <ted.dunning@gmail.com> wrote:
> It definitely can go that way.
>
> But some things are pretty nicely viewed that way. =A0For instance, would=
n't
> it be nice to have a "live processes" set in different programs? =A0Of co=
urse,
> you have to want watch for race conditions and you can't really believe i=
t
> is the same from moment to moment. =A0But with those caveats, it would be
> *really* nice.
>
> On Thu, Dec 9, 2010 at 1:27 PM, Adam Rosien <adam@rosien.net> wrote:
>
>> Making a distributed system look like a collection is a very dubious
>> proposition in my opinion. They are fundamentally different
>> abstractions and reminds me of
>> http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Computing.
>>
>> .. Adam
>>
>> On Wed, Dec 8, 2010 at 12:40 PM, Anthony Urso <anthony.urso@gmail.com>
>> wrote:
>> > I am pleased to announce the initial release of KeptCollections, a
>> > library of drop-in replacements for standard Java Collections that use
>> > Apache ZooKeeper as a backing store.
>> >
>> > KeptCollections are designed to make it easy for anyone to write
>> > distributed applications without having to learn the intricacies of
>> > ZooKeeper, or distributed programming in general.
>> >
>> > The collections use the well-known JDK APIs, yet any changes made to
>> > any of these collections by one node are seen by all other nodes
>> > within milliseconds, allowing for easy communication between processes=
 in
>> a
>> > computing cluster.
>> >
>> > More information here:
>> >
>> > https://github.com/anthonyu/KeptCollections/wiki
>> >
>> > and all code is available from:
>> >
>> > https://github.com/anthonyu/KeptCollections
>> >
>> > Please try it out, and let me know any problems you experience via
>> > github issues or this email address.
>> >
>> > Cheers,
>> > Anthony
>> >
>>
>

From general-return-2523-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Dec 10 01:17:32 2010
Return-Path: <general-return-2523-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27181 invoked from network); 10 Dec 2010 01:17:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Dec 2010 01:17:32 -0000
Received: (qmail 64105 invoked by uid 500); 10 Dec 2010 01:17:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63978 invoked by uid 500); 10 Dec 2010 01:17:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63970 invoked by uid 99); 10 Dec 2010 01:17:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Dec 2010 01:17:31 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.215.176] (HELO mail-ey0-f176.google.com) (209.85.215.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Dec 2010 01:17:23 +0000
Received: by eyz10 with SMTP id 10so2065674eyz.35
        for <general@hadoop.apache.org>; Thu, 09 Dec 2010 17:17:03 -0800 (PST)
Received: by 10.213.7.2 with SMTP id b2mr44660ebb.38.1291943823148; Thu, 09
 Dec 2010 17:17:03 -0800 (PST)
MIME-Version: 1.0
Received: by 10.213.31.145 with HTTP; Thu, 9 Dec 2010 17:16:31 -0800 (PST)
In-Reply-To: <422363.10927.qm@web114517.mail.gq1.yahoo.com>
References: <288424.95491.qm@web114514.mail.gq1.yahoo.com> <AANLkTinxisAkvE-atsuRMf6hbEMM6otiMr0H6R+Bz+_4@mail.gmail.com>
 <422363.10927.qm@web114517.mail.gq1.yahoo.com>
From: Alejandro Abdelnur <tucu@cloudera.com>
Date: Fri, 10 Dec 2010 09:16:31 +0800
Message-ID: <AANLkTim0M9+Y-Yc9GR=9AtTZEGVTSuPrFMHTvNp6Kpn5@mail.gmail.com>
Subject: Re: Multiple hadoop configurations
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Deepika,

If both of your clusters run the same version of Hadoop, then -as
Konstantin suggested- you could use Oozie. Oozie does not rely on
local Hadoop configuration files to determine the JT/NN to use, you
specify them in the Oozie workflow application XML for each job.

Cheers.

Alejandro

On Fri, Dec 10, 2010 at 7:01 AM, C J <c.joshi26@yahoo.com> wrote:
> Thanks Konstantin. My issue is more with supporting the multiple clusters . I am
> using quartz for scheduling the jobs (currently it is 2 separate schedulers for
> the 2 clusters).
>
>
> If while submitting a job I am able to specify which cluster it should trigger
> the job on (by giving the handle to the appropriate cluster), I think I will be
> able to manage it.
>
> Thanks,
> Deepika
>
>
>
>
> ________________________________
> From: Konstantin Boudnik <cos@apache.org>
> To: general@hadoop.apache.org
> Sent: Thu, December 9, 2010 2:24:34 PM
> Subject: Re: Multiple hadoop configurations
>
> I believe the answer you are looking for is Oozie coordinator, but I
> am not which version of it supports multiple clusters.
>
> On Thu, Dec 9, 2010 at 14:20, C J <c.joshi26@yahoo.com> wrote:
>> Hi,
>>
>> I have 2 hadoop clusters . Both these clusters have their own set of jobs
>> running. I also have some distcp jobs which copy over data from one cluster to
>> another.
>>
>> I want to be able to control the jobs on both the clusters through one
>>scheduler
>> (so I can coordinate the jobs).
>>
>> I am wondering when I need to trigger a job through a scheduler, how can I
> send
>> to one of the 2 clusters.
>>
>>
>> 2 clusters means 2 sets of configuration files.Is there any way to get a
> handle
>> to one of these clusters, by specifying the configuration file name or
>> something?
>>
>> Will appreciate any help or clues.
>>
>> Thanks.
>>
>>
>>
>>
>
>
>
>

From general-return-2524-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Dec 10 04:37:32 2010
Return-Path: <general-return-2524-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84656 invoked from network); 10 Dec 2010 04:37:32 -0000
Received: from unknown (HELO mail.apache.org) (::)
  by ::ffff:140.211.11.9 with SMTP; 10 Dec 2010 04:37:32 -0000
Received: (qmail 94469 invoked by uid 500); 10 Dec 2010 04:37:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94239 invoked by uid 500); 10 Dec 2010 04:37:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94229 invoked by uid 99); 10 Dec 2010 04:37:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Dec 2010 04:37:30 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Dec 2010 04:37:22 +0000
Received: from EGL-EX07CAS03.ds.corp.yahoo.com (egl-ex07cas03.eglbp.corp.yahoo.com [203.83.248.219])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id oBA4adBH046536
	for <general@hadoop.apache.org>; Thu, 9 Dec 2010 20:36:39 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1291955800;
	bh=7t+cDH0I6Q4NzChw2nRzMPKwkYqSg+IpYixNwdAGJkc=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 MIME-Version;
	b=jhgz64+5k+rBysmq1mfpwtWLvEmenzD4j1zlPBhqp5eI83kgfebbju3c089ncJrmb
	 2v9Pi0wYrOoAwVnuEr6c3zNwtKgSpAddvZRNoLIoTjr8PWlYM/0LLFFcrTGojOSFAx
	 iWSLQ0sc3EMOvhNQK3KBVCT5jV2mhoOFuDoNtGx0=
Received: from EGL-EX07VS02.ds.corp.yahoo.com ([203.83.248.206]) by
 EGL-EX07CAS03.ds.corp.yahoo.com ([203.83.248.219]) with mapi; Fri, 10 Dec
 2010 10:06:39 +0530
From: Venkatesh S <svenkat@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 10 Dec 2010 10:06:29 +0530
Subject: Re: Multiple hadoop configurations
Thread-Topic: Multiple hadoop configurations
Thread-Index: AcuYCBpsaHt3cv0IQwOPsFTDgRClxAAG7WDM
Message-ID: <C927AC25.4939C%svenkat@yahoo-inc.com>
In-Reply-To: <AANLkTim0M9+Y-Yc9GR=9AtTZEGVTSuPrFMHTvNp6Kpn5@mail.gmail.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C927AC254939Csvenkatyahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C927AC254939Csvenkatyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

If you are not running the same version, you could use Reverse Class Loader=
 trick to be able to launch jobs to multiple clusters.

-Venkatesh


On 12/10/10 6:46 AM, "Alejandro Abdelnur" <tucu@cloudera.com> wrote:

Deepika,

If both of your clusters run the same version of Hadoop, then -as
Konstantin suggested- you could use Oozie. Oozie does not rely on
local Hadoop configuration files to determine the JT/NN to use, you
specify them in the Oozie workflow application XML for each job.

Cheers.

Alejandro

On Fri, Dec 10, 2010 at 7:01 AM, C J <c.joshi26@yahoo.com> wrote:
> Thanks Konstantin. My issue is more with supporting the multiple clusters=
 . I am
> using quartz for scheduling the jobs (currently it is 2 separate schedule=
rs for
> the 2 clusters).
>
>
> If while submitting a job I am able to specify which cluster it should tr=
igger
> the job on (by giving the handle to the appropriate cluster), I think I w=
ill be
> able to manage it.
>
> Thanks,
> Deepika
>
>
>
>
> ________________________________
> From: Konstantin Boudnik <cos@apache.org>
> To: general@hadoop.apache.org
> Sent: Thu, December 9, 2010 2:24:34 PM
> Subject: Re: Multiple hadoop configurations
>
> I believe the answer you are looking for is Oozie coordinator, but I
> am not which version of it supports multiple clusters.
>
> On Thu, Dec 9, 2010 at 14:20, C J <c.joshi26@yahoo.com> wrote:
>> Hi,
>>
>> I have 2 hadoop clusters . Both these clusters have their own set of job=
s
>> running. I also have some distcp jobs which copy over data from one clus=
ter to
>> another.
>>
>> I want to be able to control the jobs on both the clusters through one
>>scheduler
>> (so I can coordinate the jobs).
>>
>> I am wondering when I need to trigger a job through a scheduler, how can=
 I
> send
>> to one of the 2 clusters.
>>
>>
>> 2 clusters means 2 sets of configuration files.Is there any way to get a
> handle
>> to one of these clusters, by specifying the configuration file name or
>> something?
>>
>> Will appreciate any help or clues.
>>
>> Thanks.
>>
>>
>>
>>
>
>
>
>


--_000_C927AC254939Csvenkatyahooinccom_--

From general-return-2525-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Dec 10 08:05:10 2010
Return-Path: <general-return-2525-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72052 invoked from network); 10 Dec 2010 08:05:10 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Dec 2010 08:05:10 -0000
Received: (qmail 31404 invoked by uid 500); 10 Dec 2010 08:05:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31095 invoked by uid 500); 10 Dec 2010 08:05:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31077 invoked by uid 99); 10 Dec 2010 08:05:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Dec 2010 08:05:07 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [98.139.91.86] (HELO nm16.bullet.mail.sp2.yahoo.com) (98.139.91.86)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 10 Dec 2010 08:05:01 +0000
Received: from [98.139.91.68] by nm16.bullet.mail.sp2.yahoo.com with NNFMP; 10 Dec 2010 08:04:39 -0000
Received: from [98.139.91.54] by tm8.bullet.mail.sp2.yahoo.com with NNFMP; 10 Dec 2010 08:04:39 -0000
Received: from [127.0.0.1] by omp1054.mail.sp2.yahoo.com with NNFMP; 10 Dec 2010 08:04:39 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 443446.55936.bm@omp1054.mail.sp2.yahoo.com
Received: (qmail 57095 invoked by uid 60001); 10 Dec 2010 08:04:39 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1291968279; bh=F3CSm+4C0g0IVQsUmiIqzvaAkrPlqoZQhvBZYvZSszs=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=yiwSahq9nBgErwh7bCabzSqkprZ5JhiMNaFQ09uEvhrtaSnY2e2QXB5x0vOD/9KYTuNEtQlh312AeuPe4LuN3gR5D5bO+jyNYoJlx9CCoQyR4AryX+YpvzNlfgaEdRbYFEQs9dV8tBoLZwClBDXgv+Il6uz5GOcnASWZ7LVCEns=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=yDdOK17z9ejDfZ9omhfdM30JyXERp9PfGMIJZOQgFoDrYELi7QwWR3MIiEp1vNdZrH+9t3e2H5M2adacUSgiUHr3SvPA2mL/WK+pWiZHUUaCb/wOVtGqckw8mvhfID5yaqoCE24Lz6p7oPXIAD3Iuf364ZtO12PojlUpvQiEyg4=;
Message-ID: <29834.55843.qm@web114506.mail.gq1.yahoo.com>
X-YMail-OSG: 8NLKPxQVM1lxhkpWhKIJSS._pVORbMJpRJewhl07RtteMVO
 BFwMSTk8voJ5TStTj4yx1bS44o4AaOd.7vFwO4BO_EW9ecbGSYC23irIAg96
 tc9lSm1DbVQ7QD9SsUYFd5D5mOqBVpa_6MWJGruxiTSQH2TufL_Vq2iW79ON
 SGpP0NvlXYbUbyQaEcYFZs.zj98VZTWwiPVXF95nYVNQEz6W9lh4Cb1rEbFd
 afkJm0LQna6RH13ZHnnp1SKvDSRNY2uiAYD5_2zliKHMo6fLWb97MQhIfADN
 HLTdwz_Uw8YoAwEv8wUQ7OQ303MipbYK4TIqe_Uw1LwJPb2RDVw--
Received: from [67.180.84.177] by web114506.mail.gq1.yahoo.com via HTTP; Fri, 10 Dec 2010 00:04:38 PST
X-Mailer: YahooMailRC/553 YahooMailWebService/0.8.107.285259
References: <C927AC25.4939C%svenkat@yahoo-inc.com>
Date: Fri, 10 Dec 2010 00:04:38 -0800 (PST)
From: C J <c.joshi26@yahoo.com>
Subject: Re: Multiple hadoop configurations
To: general@hadoop.apache.org
In-Reply-To: <C927AC25.4939C%svenkat@yahoo-inc.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-163395178-1291968278=:55843"

--0-163395178-1291968278=:55843
Content-Type: text/plain; charset=us-ascii

Thanks everyone. I do have the same version of hadoop on both clusters. And 
oozie does seem to be the best option to get what I need.

- C




________________________________
From: Venkatesh S <svenkat@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Sent: Thu, December 9, 2010 8:36:29 PM
Subject: Re: Multiple hadoop configurations

If you are not running the same version, you could use Reverse Class Loader 
trick to be able to launch jobs to multiple clusters.

-Venkatesh


On 12/10/10 6:46 AM, "Alejandro Abdelnur" <tucu@cloudera.com> wrote:

Deepika,

If both of your clusters run the same version of Hadoop, then -as
Konstantin suggested- you could use Oozie. Oozie does not rely on
local Hadoop configuration files to determine the JT/NN to use, you
specify them in the Oozie workflow application XML for each job.

Cheers.

Alejandro

On Fri, Dec 10, 2010 at 7:01 AM, C J <c.joshi26@yahoo.com> wrote:
> Thanks Konstantin. My issue is more with supporting the multiple clusters . I 
>am
> using quartz for scheduling the jobs (currently it is 2 separate schedulers 
for
> the 2 clusters).
>
>
> If while submitting a job I am able to specify which cluster it should trigger
> the job on (by giving the handle to the appropriate cluster), I think I will 
be
> able to manage it.
>
> Thanks,
> Deepika
>
>
>
>
> ________________________________
> From: Konstantin Boudnik <cos@apache.org>
> To: general@hadoop.apache.org
> Sent: Thu, December 9, 2010 2:24:34 PM
> Subject: Re: Multiple hadoop configurations
>
> I believe the answer you are looking for is Oozie coordinator, but I
> am not which version of it supports multiple clusters.
>
> On Thu, Dec 9, 2010 at 14:20, C J <c.joshi26@yahoo.com> wrote:
>> Hi,
>>
>> I have 2 hadoop clusters . Both these clusters have their own set of jobs
>> running. I also have some distcp jobs which copy over data from one cluster 
to
>> another.
>>
>> I want to be able to control the jobs on both the clusters through one
>>scheduler
>> (so I can coordinate the jobs).
>>
>> I am wondering when I need to trigger a job through a scheduler, how can I
> send
>> to one of the 2 clusters.
>>
>>
>> 2 clusters means 2 sets of configuration files.Is there any way to get a
> handle
>> to one of these clusters, by specifying the configuration file name or
>> something?
>>
>> Will appreciate any help or clues.
>>
>> Thanks.
>>
>>
>>
>>
>
>
>
>


      
--0-163395178-1291968278=:55843--

From general-return-2526-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Dec 12 12:00:06 2010
Return-Path: <general-return-2526-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16690 invoked from network); 12 Dec 2010 11:59:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Dec 2010 11:59:59 -0000
Received: (qmail 17379 invoked by uid 500); 12 Dec 2010 11:59:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17064 invoked by uid 500); 12 Dec 2010 11:59:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17056 invoked by uid 99); 12 Dec 2010 11:59:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Dec 2010 11:59:55 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of tu219717@gmail.com designates 209.85.212.178 as permitted sender)
Received: from [209.85.212.178] (HELO mail-px0-f178.google.com) (209.85.212.178)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Dec 2010 11:59:47 +0000
Received: by pxi9 with SMTP id 9so1779044pxi.37
        for <general@hadoop.apache.org>; Sun, 12 Dec 2010 03:59:26 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=rBWbv7SmlQFjldsffHsuRfijjPPNPHVtL+iyKWb7JsE=;
        b=NO3gDV5/kj7jQpeHCuT2PpgFHK2Mhog+R5Yn1zDEATzMEeWhUcM9xH8+5cYNWyaiXG
         I6t5PoCYGaF9Myw+eaNGDjcwqSDmCz/f/T9ESsXesZihT2uy0RfYAw/M9w0sC7wI+aGG
         tdN756TdFYo3WaKamW5ApQ1WzizAnIaQzww9g=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=Gf27cpgXgS8QeG8nP3Or1DlS0BImpDepvCOke8vyPcmk7o2tzJN1O1IIpbEnSceWw2
         sdYr+I+k862oCgV1n12GgmkucnXGrDJkOoIJqPM2E88nBcHVMLYgBoqBxB39jD7Px7i5
         K1v49g3vP5HCDOxzoJpLguimxIdIZczZckIM4=
MIME-Version: 1.0
Received: by 10.142.50.3 with SMTP id x3mr1738909wfx.120.1292155164549; Sun,
 12 Dec 2010 03:59:24 -0800 (PST)
Received: by 10.143.3.18 with HTTP; Sun, 12 Dec 2010 03:59:24 -0800 (PST)
Date: Sun, 12 Dec 2010 12:59:24 +0100
Message-ID: <AANLkTi=TFo+zGMdzFLTQ+c7wZN+GN0HxiG_Fjd-AxirG@mail.gmail.com>
Subject: Problem with links to "Read-only default configuration"
From: =?ISO-8859-2?Q?Tomasz_Uli=F1ski?= <tu219717@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

I am browsing default configurations of past releases and there is a
problem with links to "read-only default configuration" files.
E.g.
On the page:
http://hadoop.apache.org/common/docs/r0.20.2/cluster_setup.html#Configuration+Files
the following links:
src/core/core-default.xml
src/hdfs/hdfs-default.xml
src/mapred/mapred-default.xml
point to the "current" version, not r0.20.2

It is not only problem with r0.20.2 release, but also with others.
Could you fix it?

Best Regards,
Tomek

From general-return-2527-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Dec 13 17:29:47 2010
Return-Path: <general-return-2527-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 10562 invoked from network); 13 Dec 2010 17:29:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Dec 2010 17:29:46 -0000
Received: (qmail 29638 invoked by uid 500); 13 Dec 2010 17:29:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29544 invoked by uid 500); 13 Dec 2010 17:29:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29536 invoked by uid 99); 13 Dec 2010 17:29:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Dec 2010 17:29:44 +0000
X-ASF-Spam-Status: No, hits=2.1 required=10.0
	tests=FREEMAIL_FROM,HK_RANDOM_ENVFROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of billmcn@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Dec 2010 17:29:38 +0000
Received: by qwh6 with SMTP id 6so6793768qwh.35
        for <general@hadoop.apache.org>; Mon, 13 Dec 2010 09:29:17 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=75XCDyxrEKcSUQVH5hJHMbiYGJ8Z+2ZTDYNc0tYrRHs=;
        b=GPUEH75kPEU1z+iR2qWQKS1mPVxMIK62ChA3YMhxfWgXVX4Wv9AJvq+f6a4z4Bq70q
         ekzjWdsKlUITUrTo1Fk5ZQmhKXxdzpP1ljps/p4vz0DPCirS6doF9evXXymwAzsuFDSP
         L0P1VRjYjF2oOjosC8D2kvNkQ/zmHF7rDyjTU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=INFjrWPvz8x8qfOEclm+sAlJEeY7bLcr8cFWAKD5xxI9dhMhgP3VBf2cA286zAZ5ZK
         2N+M+vmk9gj4k9yixIHIRZJMWKpmZI011DNinVgkOD5SAwcFs5SkJwYs9Ddk0ZBRVGiG
         LkeKmZ9djv2WRaoVl0dbYV9cWc3DAXnGz6+ls=
MIME-Version: 1.0
Received: by 10.224.80.135 with SMTP id t7mr4318972qak.11.1292261357307; Mon,
 13 Dec 2010 09:29:17 -0800 (PST)
Received: by 10.220.102.193 with HTTP; Mon, 13 Dec 2010 09:29:17 -0800 (PST)
Date: Mon, 13 Dec 2010 09:29:17 -0800
Message-ID: <AANLkTi=4q=QtWz2ZFgr0fG6_7MEMbLj+JdR2MFYFD7-k@mail.gmail.com>
Subject: How do I log from my map/reduce application?
From: "W.P. McNeill" <billmcn@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175cb05228c9a504974e0b74
X-Virus-Checked: Checked by ClamAV on apache.org

--0015175cb05228c9a504974e0b74
Content-Type: text/plain; charset=UTF-8

I would like to use Hadoop's Log4j infrastructure to do logging from my
map/reduce application.  I think I've got everything set up correctly, but I
am still unable to specify the logging level I want.

By default Hadoop is set up to log at level INFO.  The first line of its
log4j.properties file looks like this:

hadoop.root.logger=INFO,console


I have an application whose reducer looks like this:

package com.me;

public class MyReducer<...> extends Reducer<...> {
   private static Logger logger =
Logger.getLogger(MyReducer.class.getName());

   ...
   protected void reduce(...) {
       logger.debug("My message");
       ...
   }
}


I've added the following line to the Hadoop log4j.properties file:

log4j.logger.com.me.MyReducer=DEBUG


I expect the Hadoop system to log at level INFO, but my application to log
at level DEBUG, so that I see "My message" in the logs for the reducer task.
 However, my application does not produce any log4j output.  If I change the
line in my reducer to read logger.info("My message") the message does get
logged, so somehow I'm failing to specify that log level for this class.

I've also tried changing the log4j line for my app to
read log4j.logger.com.me.MyReducer=DEBUG,console and get the same result.

I've been through the Hadoop and log4j documentation and I can't figure out
what I'm doing wrong.  Any suggestions?

Thanks.

--0015175cb05228c9a504974e0b74--

From general-return-2528-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 14 03:09:32 2010
Return-Path: <general-return-2528-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14828 invoked from network); 14 Dec 2010 03:09:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Dec 2010 03:09:32 -0000
Received: (qmail 57395 invoked by uid 500); 14 Dec 2010 03:09:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56915 invoked by uid 500); 14 Dec 2010 03:09:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56899 invoked by uid 99); 14 Dec 2010 03:09:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Dec 2010 03:09:29 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.30.40] (HELO qmta04.emeryville.ca.mail.comcast.net) (76.96.30.40)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Dec 2010 03:09:19 +0000
Received: from omta24.emeryville.ca.mail.comcast.net ([76.96.30.92])
	by qmta04.emeryville.ca.mail.comcast.net with comcast
	id ipye1f0151zF43QA4r8yoM; Tue, 14 Dec 2010 03:08:58 +0000
Received: from [10.0.0.13] ([98.234.216.58])
	by omta24.emeryville.ca.mail.comcast.net with comcast
	id ir8w1f00g1GAoR68kr8xPG; Tue, 14 Dec 2010 03:08:57 +0000
Message-Id: <62FFAEB9-C6E7-4A82-BE1F-5DBB6C0A2CFF@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <DE2F0813-8D15-4D50-9C6C-5A6EBB3974C8@gbiv.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Direction for Hadoop development
Date: Mon, 13 Dec 2010 19:08:56 -0800
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>	<4CF433D4.6090407@apache.org>	<AANLkTineTnsShLoa5vO-X9UAp_DUMrHRjc2ry=i4qkXs@mail.gmail.com>	<4CF66812.4080602@apache.org>	<D414D20A-A909-46A9-AAA2-52742E6395CF@apache.org>	<4CFD39BC.1080005@apache.org> <AANLkTinO_WpU3WSx6AZDN2tM7C=+Ja_hQ0n7qb_jLw2h@mail.gmail.com> <4CFD7585.70803@apache.org> <5A920FAB-F76B-4EF7-8740-5698F99A1EC2@gbiv.com> <4CFE6C68.7000503@apache.org> <DE2F0813-8D15-4D50-9C6C-5A6EBB3974C8@gbiv.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


On Dec 7, 2010, at 2:37 PM, Roy T. Fielding wrote:

>> The proposal is to change the extension mechanism incompatibly with  
>> unclear benefits,
>
> Good, these are technical reasons.  The benefits can be cleared by  
> docs.
> By incompatible, I assume you mean forward-compatibility of old  
> versions
> of Hadoop reading newer files.  Can we fix that by having the new
> implementation use the old file format by default until it is  
> configured
> to use one of the new interfaces for writing?


There are two goals here. The first is to extend the serialization  
plugin interface. The current patch does things completely compatibly  
including a shim that will use the previous plugins to satisfy the new  
API. The benefits are also clear. Avro serialization is possible when  
it wasn't previously. It also provides a wide range of opportunities  
that weren't previously possible.

The file format was changed as a demonstration that the serialization  
interface was useful and complete. The file change is also backwards  
compatible and will automatically read old versions of the file. Old  
versions of the code will complain with an error message if they are  
given a new version. This is exactly the pattern we have used in the  
past.

So, no there are no technical issues with the patch as it stands.

> You keep referring to the kernel as if it were a product.  I don't see
> a kernel product in the list of things released by Apache Hadoop.

The kernel is a very loosely defined concept. Utilities that are  
currently used by the framework are "kernel" others are just used by  
the users. Some classes are clearly kernel and some are clearly  
library, but there are some such as BooleanWritable that aren't  
obvious. It would take a fair amount of work and likely some  
duplication to segregate out the library code. I also worry that  
creating such a project would make Hadoop less useful out of the box  
and decrease the value of the Apache release of Hadoop.

But back to the original point. Doug's (and Tom's) veto was based on:
1. Modification to SequenceFile.
2. It introduces a dependence on Protocol Buffers.

There was strong consensus that SequenceFile was required and should  
be updated as the framework evolves. The second is not a technical  
reason. I believe that the entire veto should be considered invalid.

-- Owen

From general-return-2529-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 14 04:49:48 2010
Return-Path: <general-return-2529-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39064 invoked from network); 14 Dec 2010 04:49:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Dec 2010 04:49:47 -0000
Received: (qmail 19509 invoked by uid 500); 14 Dec 2010 04:49:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19128 invoked by uid 500); 14 Dec 2010 04:49:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19116 invoked by uid 99); 14 Dec 2010 04:49:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Dec 2010 04:49:44 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.213.48] (HELO mail-yw0-f48.google.com) (209.85.213.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Dec 2010 04:49:38 +0000
Received: by ywo7 with SMTP id 7so135024ywo.35
        for <general@hadoop.apache.org>; Mon, 13 Dec 2010 20:49:17 -0800 (PST)
MIME-Version: 1.0
Received: by 10.236.110.162 with SMTP id u22mr10140796yhg.51.1292302157359;
 Mon, 13 Dec 2010 20:49:17 -0800 (PST)
Received: by 10.236.111.45 with HTTP; Mon, 13 Dec 2010 20:49:17 -0800 (PST)
In-Reply-To: <62FFAEB9-C6E7-4A82-BE1F-5DBB6C0A2CFF@apache.org>
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>
	<4CF433D4.6090407@apache.org>
	<AANLkTineTnsShLoa5vO-X9UAp_DUMrHRjc2ry=i4qkXs@mail.gmail.com>
	<4CF66812.4080602@apache.org>
	<D414D20A-A909-46A9-AAA2-52742E6395CF@apache.org>
	<4CFD39BC.1080005@apache.org>
	<AANLkTinO_WpU3WSx6AZDN2tM7C=+Ja_hQ0n7qb_jLw2h@mail.gmail.com>
	<4CFD7585.70803@apache.org>
	<5A920FAB-F76B-4EF7-8740-5698F99A1EC2@gbiv.com>
	<4CFE6C68.7000503@apache.org>
	<DE2F0813-8D15-4D50-9C6C-5A6EBB3974C8@gbiv.com>
	<62FFAEB9-C6E7-4A82-BE1F-5DBB6C0A2CFF@apache.org>
Date: Mon, 13 Dec 2010 23:49:17 -0500
Message-ID: <AANLkTink2p6ePi2wjZ21uuBnmUqX2ZYjcOqszXZ1x-p5@mail.gmail.com>
Subject: Re: [VOTE] Direction for Hadoop development
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0023547c8c9b082a0a0497578bc1

--0023547c8c9b082a0a0497578bc1
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Dec 13, 2010 at 10:08 PM, Owen O'Malley <omalley@apache.org> wrote:

>
> On Dec 7, 2010, at 2:37 PM, Roy T. Fielding wrote:
>
>  The proposal is to change the extension mechanism incompatibly with
>>> unclear benefits,
>>>
>>
>> Good, these are technical reasons.  The benefits can be cleared by docs.
>> By incompatible, I assume you mean forward-compatibility of old versions
>> of Hadoop reading newer files.  Can we fix that by having the new
>> implementation use the old file format by default until it is configured
>> to use one of the new interfaces for writing?
>>
>
>
> There are two goals here. The first is to extend the serialization plugin
> interface. The current patch does things completely compatibly including a
> shim that will use the previous plugins to satisfy the new API. The benefits
> are also clear. Avro serialization is possible when it wasn't previously. It
> also provides a wide range of opportunities that weren't previously
> possible.
>
> The file format was changed as a demonstration that the serialization
> interface was useful and complete. The file change is also backwards
> compatible and will automatically read old versions of the file. Old
> versions of the code will complain with an error message if they are given a
> new version. This is exactly the pattern we have used in the past.
>
> So, no there are no technical issues with the patch as it stands.


One of the technical issues is the fact that this precludes users from using
PB (or thrift or avro) in their jobs if the version required conflicts with
what Hadoop proper has on the classpath. We've already seen these kinds of
conflicts with other libraries in the wild and I would like to minimize this
possibility in the future. Was there something in the patch that addressed
this (I may have missed it; only did a cursory scan through)?

Jumping back to the "non-technical" issue, I really think it would help to
develop a course of action for resolution similar to what I suggested
earlier. It doesn't need to be specifically what I suggested, but I do think
that consensus building and conflict resolution are in the best interest of
the community. I feel like we could debate what people said, did, meant, or
the specifics of this issue for a long time.

Thanks and regards.
-- 
Eric Sammer
twitter: esammer
data: www.cloudera.com

--0023547c8c9b082a0a0497578bc1--

From general-return-2530-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 14 05:25:31 2010
Return-Path: <general-return-2530-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53291 invoked from network); 14 Dec 2010 05:25:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Dec 2010 05:25:30 -0000
Received: (qmail 37265 invoked by uid 500); 14 Dec 2010 05:25:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36793 invoked by uid 500); 14 Dec 2010 05:25:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 36223 invoked by uid 99); 14 Dec 2010 05:23:31 -0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <DA51D8AE-52A8-4A54-A8AF-AE07CD840C6E@apache.org>
References: <DA51D8AE-52A8-4A54-A8AF-AE07CD840C6E@apache.org>
Date: Tue, 14 Dec 2010 16:23:01 +1100
Message-ID: <AANLkTi=xcZVivY8ha2gmr3=-LXz-CQ39SJhSBiKFbZSv@mail.gmail.com>
Subject: Fwd: [PROPOSAL] Mesos Project
From: Ian Holsman <lists@holsman.net>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba6e89aab1eb9804975803e5
X-Virus-Checked: Checked by ClamAV on apache.org

--90e6ba6e89aab1eb9804975803e5
Content-Type: text/plain; charset=ISO-8859-1

Thought this may be of interest to people on this list.

---------- Forwarded message ----------
From: Matei Zaharia <matei@apache.org>
Date: Tue, Dec 14, 2010 at 9:08 AM
Subject: [PROPOSAL] Mesos Project
To: general@incubator.apache.org
Cc: Benjamin Hindman <benh@eecs.berkeley.edu>, Andy Konwinski <
andyk@berkeley.edu>, Ali Ghodsi <alig@eecs.berkeley.edu>


We would like to propose Mesos as an incubator proposal.

Mesos is a resource manager for clusters that provides resource sharing and
isolation across distributed applications like Apache Hadoop, MPI, or web
applications. It started as a research project at UC Berkeley, but is now
being used both by other Berkeley groups and at Twitter. We open sourced
Mesos in August and would like to grow a broader community around it.

Our proposal is included below and available on the wiki at
http://wiki.apache.org/incubator/MesosProposal. We look forward to hearing
feedback and questions on the proposal. Also, let us know if you're
interested in being a mentor.

Thanks,

Matei Zaharia, Benjamin Hindman, Andy Konwinski, and Ali Ghodsi



= Abstract =

Mesos is a cluster manager that provides resource sharing and
isolation across cluster applications.



= Proposal =

Mesos is system for sharing resources between cluster applications such
as Hadoop MapReduce, HBase, MPI, and web applications.
It is motivated by three use cases. First, organizations that use
several of these applications can use Mesos to share nodes between them,
increasing utilization and simplifying management. Second, inspired by
MapReduce, a wide array of new cluster programming frameworks are being
proposed, such as Apache Hama, Microsoft Dryad, and Google's Pregel and
Caffeine. Mesos provides a common interface for such frameworks to share
resources, allowing organizations to use multiple frameworks in the same
cluster. Third, Mesos allows users of a framework such as Hadoop to have
multiple instances of the framework on the same cluster, facilitating
workload isolation and incremental deployment of upgrades.



= Background =

Mesos was inspired by operational issues experienced in large Apache Hadoop
deployments as well as a desire to provide a management system for a
wider range of cluster applications. The Apache Hadoop community has long
realized that the current model of having one instance of MapReduce
control a whole cluster leads to problems with isolation (one job may
cause the master to crash, killing all the other jobs), scalability,
and software upgrades (an upgrade must be deployed on the whole cluster).
Statically partitioning resources into multiple fixed-size MapReduce
clusters
is unattractive because it lowers both utilization and data locality.
The community has discussed a two-level scheduling model where a simple,
robust low-level layer enables multiple applications to launch tasks
(https://issues.apache.org/jira/browse/MAPREDUCE-279). Mesos is such a
layer,
with the additional goal of supporting non-Hadoop applications as well.

Mesos started as a research project at UC Berkeley, but is now being
tested at several companies (including Twitter and Facebook), and has
attracted
interest from other industry users and researchers as well. We are
therefore proposing to place Mesos in the Apache incubator and build an
open source community around it.



= Rationale =

Although a variety of cluster schedulers (e.g. Torque, Sun Grid Engine)
already exist in the scientific computing community, they are not well
suited for today's data center environment.
These schedulers generally give jobs coarse-grained static allocations of
the cluster (e.g. X nodes for the full duration of the job).
This is problematic because many cluster applications are elastic
(can scale up and down), so utilization is not optimal under static
partitioning, and because data-intensive applications such as MapReduce
need to run a few tasks on every node of the cluster to read data locally.
To address these challenges, Mesos is designed around two principles:

 * Fine-grained sharing: Mesos allocates resources at the level of "tasks"
  within a job, allowing applications to scale up and down over time and
  to take turns accessing data on cluster nodes.
 * Application-controlled scheduling: Applications control which nodes
  their tasks run on, allowing them to achieve placement goals such as
  data locality.

In addition to these principles, Mesos is designed to be simple, scalable
and robust, becuase a cluster manager must be highly available to support
applications and should not become a bottleneck. Application-controlled
scheduling already simplifies our design by pushing much of the complex
logic of tracking job state to applications. In addition, Mesos employs an
optimized C++ message-passing library to achieve scalability and supports
master failover using Apache ZooKeeper.

Mesos already supports running Hadoop and MPI. We plan to add support
for other systems as requested (and contributed) by the community.



= Current Status =

== Meritocracy ==

Our intent with this incubator proposal is to start building a diverse
developer community around Mesos following the Apache meritocracy model.
We have wanted to make the project open source and encourage contributors
from multiple organizations from the start. We plan to provide plenty
of support to new developers and to quickly recruit those who make solid
contributions to committer status.

== Community ==

Mesos is currently being used by developers at Twitter and researchers in
computer science and civil engineering at Berkeley. We hope to extend the
user
and developer base further in the future. The current developers and users
are all interested in building a solid open source community around Mesos.

To work towards an open source community, we have been using the GitHub
issue
tracker and mailing lists at Berkeley for development discussions within our
group for several months now.

== Core Developers ==

Mesos was started by three graduate students at UC Berkeley (Benjamin
Hindman,
Andy Konwinski and Matei Zaharia), who were soon joined by a postdoc from
the Swedish Institute of Computer Science (Ali Ghodsi). Although started as
a research project, Mesos was always intended to solve operational issues
with large clusters and to become an open-source project, building on our
successful experience doing research that has been incorporated into Apache
Hadoop
(several scheduling algorithms).

== Alignment ==

The ASF is a natural host for Mesos given that it is already the home of
Hadoop, HBase, Cassandra, and other emerging cloud software projects.
Mesos was designed to support Hadoop from the beginning in order to solve
operational challenges in Hadoop clusters, and it aims to support a wide
range
of applications beyond Hadoop as well. Mesos complements the existing Apache
cloud computing projects by providing a unified way to manage these systems
and to share resources and data between them.



= Known Risks =

== Orphaned Products ==

With the current core developers of Mesos being graduate students, there
is a risk that these developers will eventually move on to other projects.
However, because of the broad scope of Mesos, we all plan to continue
working
on projects related to it in the next several years. We are also actively
working with developers at other organizations, such as Twitter, who are
good candidates to become contributors.

== Inexperience with Open Source ==

All of the core developers are active users and followers of open source.
Matei Zaharia is a Hadoop committer and has experience with the Apache
infrastructure and development process. Andy Konwinski has contributed
patches to Hadoop through the Apache infrastructure as well. Ali Ghodsi
has released open source software as part of his PhD work that was adopted
by a Swedish company.

== Homogeneous Developers ==

The current core developers are all researchers (graduate students and a
young professor). However, we hope to establish a developer community
that includes contributors from several corporations, and we are already
working towards this with Twitter and Facebook.

== Reliance on Salaried Developers ==

Given that the project started in an academic research environment, the
core developers are all interested in it primarily for its own sake rather
than for the sake of employment. We all intend to continue working on Mesos
as volunteers.

== Relationships with Other Apache Products ==

Mesos needs to work well with Hadoop, HBase, and other cloud software
projects. Being hosted on the same infrastructure will facilitate this
and ultimately help out both Mesos and the projects that can now be
managed using it. There is, however, a risk that new projects will be built
to run solely on Mesos, introducing a dependency.

== An Excessive Fascination with the Apache Brand ==

While we respect the reputation of the Apache brand and have no doubts that
it will attract contributors and users, our interest is primarily to give
Mesos a solid home as an open source project following an established
development model. Locating the project in Apache will also facilitate
collaboration with Hadoop, HBase, and other Apache cluster computing
projects,
as discussed in the Alignment section.



= Documentation =

Information about Mesos can be found at http://mesos.berkeley.edu.
The following sources may be useful to start with:

 * Documentation for GitHub release: http://github.com/mesos/mesos/wiki
 * Presentation at Hadoop User Group:
http://www.cs.berkeley.edu/~matei/talks/2010/hug_mesos.pdf
 * Tech report on system design and current features:
http://mesos.berkeley.edu/mesos_tech_report.pdf (paper to appear at NSDI
2011 conference)



= Initial Source =

Mesos has been under development since spring 2009 by a team of graduate
students and researchers. It is currently hosted on GitHub under a BSD
license at http://github.com/mesos/mesos.



= External Dependencies =

The dependencies all have Apache compatible licenses, including BSD, MIT,
Boost, and Apache 2.0.



= Cryptography =

Not applicable.



= Required Resources =

== Mailing Lists ==

 * mesos-private for private PMC discussions (with moderated subscriptions)
 * mesos-dev
 * mesos-commits
 * mesos-user



== Subversion Directory ==

https://svn.apache.org/repos/asf/incubator/mesos



== Issue Tracking ==

JIRA Mesos (MESOS)



== Other Resources ==

The existing code already has unit tests, so we would like a Hudson instance
to run them whenever a new patch is submitted. This can be added after
project
creation.



= Initial Committers =

 * Ali Ghodsi (ali at sics dot se)
 * Benjamin Hindman (benh at eecs dot berkeley dot edu)
 * Andy Konwinski (andyk at eecs dot berkeley dot edu)
 * Matei Zaharia (matei at apache dot org)

A CLA is already on file for Matei Zaharia.


= Affiliations =

 * Ali Ghodsi (UC Berkeley / Swedish Institute of Computer Science)
 * Benjamin Hindman (UC Berkeley)
 * Andy Konwinski (UC Berkeley)
 * Matei Zaharia (UC Berkeley)



= Sponsors =

== Champion ==

Tom White

== Nominated Mentors ==

 * Dhruba Borthakur
 * Brian McCallister
 * Tom White

== Sponsoring Entity ==

Incubator PMC



---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

--90e6ba6e89aab1eb9804975803e5--

From general-return-2531-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 14 05:27:50 2010
Return-Path: <general-return-2531-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53918 invoked from network); 14 Dec 2010 05:27:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Dec 2010 05:27:49 -0000
Received: (qmail 41532 invoked by uid 500); 14 Dec 2010 05:27:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 41434 invoked by uid 500); 14 Dec 2010 05:27:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 36281 invoked by uid 99); 14 Dec 2010 05:24:21 -0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of sangee.sha@gmail.com designates 209.85.161.47 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=k7FH+TfQWezheg0LGFdzmxfLzneBzBMI4okrUUZCQLs=;
        b=IxYCasIjDrCYO2J7/Xh3TywjWx8vfdn/YTHSfpTSpKw6LcgXwmLWsVMbkfqhcIDKTA
         zQLF4Rva+EUqjQPXAMNehMqlXLcomURXCwCU/EUIj0lXnIl8SITezC5N7AcRsrukhEWU
         ujNcZ6wdqwlcXu9FiJcR4pAa02aqznKBBx2vc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=wh1wP35JndLLTVcCBSVIfOLTaSAJOwgtxNVhWlsYP1grQxtLWXRx/ucKe+u2kgY1P6
         Ca4yxuxMR8QcqmBr26pQmPLf4J4hP3hvX0f2UMtp4juichwDJCbLkz3ETQtDL7TxY429
         wLtxHyzTsTxB/bHHKSLZzfPKvOOUAG7rfwTjE=
MIME-Version: 1.0
Date: Tue, 14 Dec 2010 10:53:53 +0530
Message-ID: <AANLkTikHyvR=ifVibKpdnHxXm7NgU=cdZ9+Yz3idwTg6@mail.gmail.com>
Subject: Using Sqoop with Hive
From: sangeetha s <sangee.sha@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c5baf9cd7905049758066b

--001636c5baf9cd7905049758066b
Content-Type: text/plain; charset=ISO-8859-1

Hi all,

Currently I am working with hadoop+ hive from apache distribution. Since I
am in need to transfer data from myql to hive , I wish to use sqoop with
hadoop. While doing so I was instructed to use cloud era's distribution of
Hadoop.But I don't want to switch from Apache distribution to CDH.

Is there any other way to download sqoop,I wish to stay with the existing
setup (Hadoop+Hive).?


Kindly tell me your suggestions.

Thank you,


-- 
Regards,
Sangita S.

--001636c5baf9cd7905049758066b--

From general-return-2532-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 14 05:44:22 2010
Return-Path: <general-return-2532-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70047 invoked from network); 14 Dec 2010 05:44:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Dec 2010 05:44:22 -0000
Received: (qmail 53763 invoked by uid 500); 14 Dec 2010 05:44:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53481 invoked by uid 500); 14 Dec 2010 05:44:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53460 invoked by uid 99); 14 Dec 2010 05:44:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Dec 2010 05:44:18 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.59.211] (HELO QMTA11.westchester.pa.mail.comcast.net) (76.96.59.211)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Dec 2010 05:44:10 +0000
Received: from omta19.westchester.pa.mail.comcast.net ([76.96.62.98])
	by QMTA11.westchester.pa.mail.comcast.net with comcast
	id itdX1f00227AodY5Btjqsb; Tue, 14 Dec 2010 05:43:50 +0000
Received: from [10.0.0.13] ([98.234.216.58])
	by omta19.westchester.pa.mail.comcast.net with comcast
	id itjp1f0021GAoR63ftjqVw; Tue, 14 Dec 2010 05:43:50 +0000
Message-Id: <6974FEB1-0E72-4FBF-8196-BF4B4E4D77BC@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <AANLkTink2p6ePi2wjZ21uuBnmUqX2ZYjcOqszXZ1x-p5@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Direction for Hadoop development
Date: Mon, 13 Dec 2010 21:43:47 -0800
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com> <4CF433D4.6090407@apache.org> <AANLkTineTnsShLoa5vO-X9UAp_DUMrHRjc2ry=i4qkXs@mail.gmail.com> <4CF66812.4080602@apache.org> <D414D20A-A909-46A9-AAA2-52742E6395CF@apache.org> <4CFD39BC.1080005@apache.org> <AANLkTinO_WpU3WSx6AZDN2tM7C=+Ja_hQ0n7qb_jLw2h@mail.gmail.com> <4CFD7585.70803@apache.org> <5A920FAB-F76B-4EF7-8740-5698F99A1EC2@gbiv.com> <4CFE6C68.7000503@apache.org> <DE2F0813-8D15-4D50-9C6C-5A6EBB3974C8@gbiv.com> <62FFAEB9-C6E7-4A82-BE1F-5DBB6C0A2CFF@apache.org> <AANLkTink2p6ePi2wjZ21uuBnmUqX2ZYjcOqszXZ1x-p5@mail.gmail.com>
X-Mailer: Apple Mail (2.936)


On Dec 13, 2010, at 8:49 PM, Eric Sammer wrote:

> One of the technical issues is the fact that this precludes users  
> from using
> PB (or thrift or avro) in their jobs if the version required  
> conflicts with
> what Hadoop proper has on the classpath.

This is currently true of all of our libraries and is addressed by  
MAPREDUCE-1938. After that is committed, users who want to override to  
a newer version just need to configure their job to do so. Using a new  
library, just because it is a serialization library other than Avro,  
is not an acceptable reason to veto a patch.

-- Owen

From general-return-2533-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 14 05:50:06 2010
Return-Path: <general-return-2533-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70777 invoked from network); 14 Dec 2010 05:50:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Dec 2010 05:50:06 -0000
Received: (qmail 58256 invoked by uid 500); 14 Dec 2010 05:50:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57893 invoked by uid 500); 14 Dec 2010 05:50:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57884 invoked by uid 99); 14 Dec 2010 05:50:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Dec 2010 05:50:03 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Dec 2010 05:49:57 +0000
Received: by qyk10 with SMTP id 10so312306qyk.14
        for <general@hadoop.apache.org>; Mon, 13 Dec 2010 21:49:36 -0800 (PST)
MIME-Version: 1.0
Received: by 10.229.238.16 with SMTP id kq16mr4460318qcb.134.1292305775943;
 Mon, 13 Dec 2010 21:49:35 -0800 (PST)
Received: by 10.229.225.137 with HTTP; Mon, 13 Dec 2010 21:49:35 -0800 (PST)
In-Reply-To: <AANLkTikHyvR=ifVibKpdnHxXm7NgU=cdZ9+Yz3idwTg6@mail.gmail.com>
References: <AANLkTikHyvR=ifVibKpdnHxXm7NgU=cdZ9+Yz3idwTg6@mail.gmail.com>
Date: Mon, 13 Dec 2010 21:49:35 -0800
Message-ID: <AANLkTinJzTXFSEfVaAeVQC+6wArnW08j_7DCuEQqYg+Q@mail.gmail.com>
Subject: Re: Using Sqoop with Hive
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Cc: sqoop-user@cloudera.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hey Sangita,

You can download a release tarball from the Sqoop github page:

https://github.com/cloudera/sqoop/downloads

If you hit any snags send mail to sqoop-user@cloudera.org

https://groups.google.com/a/cloudera.org/group/sqoop-user/topics

Thanks,
Eli

On Mon, Dec 13, 2010 at 9:23 PM, sangeetha s <sangee.sha@gmail.com> wrote:
> Hi all,
>
> Currently I am working with hadoop+ hive from apache distribution. Since I
> am in need to transfer data from myql to hive , I wish to use sqoop with
> hadoop. While doing so I was instructed to use cloud era's distribution of
> Hadoop.But I don't want to switch from Apache distribution to CDH.
>
> Is there any other way to download sqoop,I wish to stay with the existing
> setup (Hadoop+Hive).?
>
>
> Kindly tell me your suggestions.
>
> Thank you,
>
>
> --
> Regards,
> Sangita S.
>

From general-return-2534-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 14 07:15:23 2010
Return-Path: <general-return-2534-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 7056 invoked from network); 14 Dec 2010 07:15:23 -0000
Received: from unknown (HELO mail.apache.org) (::)
  by ::ffff:140.211.11.9 with SMTP; 14 Dec 2010 07:15:23 -0000
Received: (qmail 17706 invoked by uid 500); 14 Dec 2010 07:15:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17367 invoked by uid 500); 14 Dec 2010 07:15:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17359 invoked by uid 99); 14 Dec 2010 07:15:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Dec 2010 07:15:20 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Dec 2010 07:15:13 +0000
Received: by gyf1 with SMTP id 1so185389gyf.35
        for <general@hadoop.apache.org>; Mon, 13 Dec 2010 23:14:51 -0800 (PST)
MIME-Version: 1.0
Received: by 10.236.103.145 with SMTP id f17mr10403827yhg.12.1292310891008;
 Mon, 13 Dec 2010 23:14:51 -0800 (PST)
Received: by 10.236.111.45 with HTTP; Mon, 13 Dec 2010 23:14:50 -0800 (PST)
In-Reply-To: <6974FEB1-0E72-4FBF-8196-BF4B4E4D77BC@apache.org>
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>
	<4CF433D4.6090407@apache.org>
	<AANLkTineTnsShLoa5vO-X9UAp_DUMrHRjc2ry=i4qkXs@mail.gmail.com>
	<4CF66812.4080602@apache.org>
	<D414D20A-A909-46A9-AAA2-52742E6395CF@apache.org>
	<4CFD39BC.1080005@apache.org>
	<AANLkTinO_WpU3WSx6AZDN2tM7C=+Ja_hQ0n7qb_jLw2h@mail.gmail.com>
	<4CFD7585.70803@apache.org>
	<5A920FAB-F76B-4EF7-8740-5698F99A1EC2@gbiv.com>
	<4CFE6C68.7000503@apache.org>
	<DE2F0813-8D15-4D50-9C6C-5A6EBB3974C8@gbiv.com>
	<62FFAEB9-C6E7-4A82-BE1F-5DBB6C0A2CFF@apache.org>
	<AANLkTink2p6ePi2wjZ21uuBnmUqX2ZYjcOqszXZ1x-p5@mail.gmail.com>
	<6974FEB1-0E72-4FBF-8196-BF4B4E4D77BC@apache.org>
Date: Tue, 14 Dec 2010 02:14:50 -0500
Message-ID: <AANLkTimEHDvu93by4uuHuUhdJ-ghXLJuQW3bYtB5iKGT@mail.gmail.com>
Subject: Re: [VOTE] Direction for Hadoop development
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0023547c902399175a049759933c
X-Virus-Checked: Checked by ClamAV on apache.org

--0023547c902399175a049759933c
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Dec 14, 2010 at 12:43 AM, Owen O'Malley <omalley@apache.org> wrote:

>
> On Dec 13, 2010, at 8:49 PM, Eric Sammer wrote:
>
>  One of the technical issues is the fact that this precludes users from
>> using
>> PB (or thrift or avro) in their jobs if the version required conflicts
>> with
>> what Hadoop proper has on the classpath.
>>
>
> This is currently true of all of our libraries and is addressed by
> MAPREDUCE-1938. After that is committed, users who want to override to a
> newer version just need to configure their job to do so.


That's definitely nice. I hadn't seen that one.

Using a new library, just because it is a serialization library other than
> Avro, is not an acceptable reason to veto a patch.
>

That I'm not really qualified to say. I don't really know the rules on
vetoes. But again, I'm more interested in the larger issue you raised (the
subject of the thread). Part of direction for Hadoop, to me, is to get to a
point where we're spending time working together. Again, I propose:

- Codify (by vote) whether design plans are required or if an informal email
indicating intent is sufficient, and under what circumstances. Provide
examples to clarify circumstances. Solves the long term but not HADOOP-6685.
- Focus the discussion on evaluation of proposals for remedying the process
for conflict resolution. I know some exist, but they're drastic (removal of
PMC members, for instance).
- After consensus on above, focus the conversation (in another thread or on
JIRA, whatever is most appropriate) on HADOOP-6685 so no one is
blocked.
- Put the community of users first in all areas of development and
interaction.

To the last point:

I understand there's contention from past issues. I genuinely believe
everyone has the users' interests at heart. I'm saying this as a user: this
kind of contention is not in anyone's interest. We need true resolution to
past issues, consensus on what the goals are and generally how to get there
including how to resolve further disagreement, and only then can we jump
back into the immediate issue where there is disagreement. I no longer care
how corny I sound about this (and it's about to get corny). I implore all
parties involved to take a long look at how we interact and to approach this
with renewed respect for each other, the project, and the users. Decide to
let previous cruft go and start anew. Do that by building consensus on
getting out of a veto stalemate and coming up with a long term plan that
makes sense to everyone.

To the specific issue:

Owen, would you be amenable to working to find a way to remove the PB dep in
support of HADOOP-6685 and handling bootstrapping with either one of the
existing deps or simple hard coded length, type, value serialization /
deserialization similar to Writables? I understand your points about PB
being solid, but Hadoop is already thick with deps (some of which do handle
this, even if not in the preferred / most optimal format) and MR-1938 is
still a ways off.

Doug, is there any way to get past the objection to the SequenceFile update?
It is a widely used format and is currently in Hadoop core. While I agree
Hadoop should be a "kernel" as one artifact and libs as another, I think it
would be less friction and cleaner to come up with a plan on how to get to
that state independent of pending issues right now. It seems like
maintaining backwards compat is critical to Owen et al as well and I'm sure
we can come up with modifications to the patch to make it forward compat as
well (if it's not already; I'm unclear on / don't remember this point).

This, to me, looks like an achievable goal that doesn't compromise the
functionality of HADOOP-6685 and leaves the door open to discussion of a
stronger kernel / lib separation.

Regards, respect, and no longer afraid of being corny on general@,
-- 
Eric Sammer
twitter: esammer
data: www.cloudera.com

--0023547c902399175a049759933c--

From general-return-2535-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 14 19:09:33 2010
Return-Path: <general-return-2535-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27903 invoked from network); 14 Dec 2010 19:09:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Dec 2010 19:09:33 -0000
Received: (qmail 82768 invoked by uid 500); 14 Dec 2010 19:09:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82650 invoked by uid 500); 14 Dec 2010 19:09:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82642 invoked by uid 99); 14 Dec 2010 19:09:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Dec 2010 19:09:30 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.62.40] (HELO qmta04.westchester.pa.mail.comcast.net) (76.96.62.40)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Dec 2010 19:09:22 +0000
Received: from omta22.westchester.pa.mail.comcast.net ([76.96.62.73])
	by qmta04.westchester.pa.mail.comcast.net with comcast
	id izLS1f00B1ap0As54792Yf; Tue, 14 Dec 2010 19:09:02 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta22.westchester.pa.mail.comcast.net with comcast
	id j78s1f00P2SbwD53i78vUj; Tue, 14 Dec 2010 19:09:00 +0000
Message-Id: <2C407569-EA1B-4CE9-B584-F81B2DE1D88E@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <AANLkTimEHDvu93by4uuHuUhdJ-ghXLJuQW3bYtB5iKGT@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Direction for Hadoop development
Date: Tue, 14 Dec 2010 11:08:51 -0800
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com> <4CF433D4.6090407@apache.org> <AANLkTineTnsShLoa5vO-X9UAp_DUMrHRjc2ry=i4qkXs@mail.gmail.com> <4CF66812.4080602@apache.org> <D414D20A-A909-46A9-AAA2-52742E6395CF@apache.org> <4CFD39BC.1080005@apache.org> <AANLkTinO_WpU3WSx6AZDN2tM7C=+Ja_hQ0n7qb_jLw2h@mail.gmail.com> <4CFD7585.70803@apache.org> <5A920FAB-F76B-4EF7-8740-5698F99A1EC2@gbiv.com> <4CFE6C68.7000503@apache.org> <DE2F0813-8D15-4D50-9C6C-5A6EBB3974C8@gbiv.com> <62FFAEB9-C6E7-4A82-BE1F-5DBB6C0A2CFF@apache.org> <AANLkTink2p6ePi2wjZ21uuBnmUqX2ZYjcOqszXZ1x-p5@mail.gmail.com> <6974FEB1-0E72-4FBF-8196-BF4B4E4D77BC@apache.org> <AANLkTimEHDvu93by4uuHuUhdJ-ghXLJuQW3bYtB5iKGT@mail.gmail.com>
X-Mailer: Apple Mail (2.936)


On Dec 13, 2010, at 11:14 PM, Eric Sammer wrote:

> - Codify (by vote) whether design plans are required or if an  
> informal email
> indicating intent is sufficient, and under what circumstances. Provide
> examples to clarify circumstances. Solves the long term but not  
> HADOOP-6685.

We had a presentation about my plans for this jira in June and both  
Tom and Doug attended and asked questions. It wasn't a lack of  
communication. At that time, they didn't like the proposal but didn't  
plan to block it.

In general code changes shouldn't require a vote. The goal is to work  
together as a community to produce code, not make everything a  
lawyerly argument.

> Owen, would you be amenable to working to find a way to remove the  
> PB dep in
> support of HADOOP-6685 and handling bootstrapping with either one of  
> the
> existing deps or simple hard coded length, type, value serialization /
> deserialization similar to Writables?

Of course it is possible, but it is a far worse engineering solution.  
ProtocolBuffers do exactly what I need, it is foolish to implement an  
hand-crafted replacement. My point is that a dependence on Avro was  
accepted without issue. No one had an issue when I added snakeyaml.  
All of the objections are fundamentally based in a dislike of  
serializations other than Yarn.

-- Owen

From general-return-2536-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 14 20:21:20 2010
Return-Path: <general-return-2536-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56675 invoked from network); 14 Dec 2010 20:21:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Dec 2010 20:21:19 -0000
Received: (qmail 80743 invoked by uid 500); 14 Dec 2010 20:21:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80691 invoked by uid 500); 14 Dec 2010 20:21:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80678 invoked by uid 99); 14 Dec 2010 20:21:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Dec 2010 20:21:17 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [85.214.115.60] (HELO h1349259.stratoserver.net) (85.214.115.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Dec 2010 20:21:17 +0000
Received: from isabel-drost.de ([85.214.115.60] helo=motokokusanagi.localnet)
	by h1349259.stratoserver.net with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
	(Exim 4.69)
	(envelope-from <isabel@apache.org>)
	id 1PSbMj-0005fP-TJ; Tue, 14 Dec 2010 20:20:53 +0000
From: Isabel Drost <isabel@apache.org>
Reply-To: isabel@apache.org
To: hadoopberlin@isabel-drost.de
Subject: Apache Mahout Hackathon - Berlin - Feb 2011
Date: Tue, 14 Dec 2010 21:20:14 +0100
User-Agent: KMail/1.13.5 (Linux/2.6.32-5-amd64; KDE/4.4.5; x86_64; ; )
Cc: user@mahout.apache.org,
 general@lucene.apache.org,
 general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/signed;
  boundary="nextPart1421423.Zk8J7MEp3j";
  protocol="application/pgp-signature";
  micalg=pgp-sha1
Content-Transfer-Encoding: 7bit
Message-Id: <201012142120.21130.isabel@apache.org>

--nextPart1421423.Zk8J7MEp3j
Content-Type: Text/Plain;
  charset="us-ascii"
Content-Transfer-Encoding: quoted-printable


Hello,

early 2011 - on February 19th/20th to be more precise - the first Apache Ma=
hout=20
Hackathon is scheduled to take place at c-base in Berlin. The Hackathon wil=
l=20
take one weekend. There will be plenty of time to hack on your favourite Ma=
hout=20
issue, to get in touch with local Mahout committers, get your machine learn=
ing=20
project off the ground. The venue features a bar that sells drinks (includi=
ng=20
Club Mate) so no need to bring those.

Please register at https://www.xing.com/events/apache-mahout-hackathon-6476=
03 if=20
you are planning to attend this event so we can plan for enough space for=20
everyone. If you have not registered for the event there is no guarantee yo=
u=20
will be admitted.

If you'd like to support the event: We'd love to provide pizza and drinks f=
or=20
free. If you are interested in sponsoring, please contact me at=20
isabel@apache.org

A special Thank You to c-base for providing the location free of charge.

=46eel free to forward this information to anyone who might be interested, =
tweet=20
the event, include information on your blog if you are attending. Check the=
=20
above link to learn of potential changes.

Looking forward to a fun and productive weekend,
Isabel

--nextPart1421423.Zk8J7MEp3j
Content-Type: application/pgp-signature; name=signature.asc 
Content-Description: This is a digitally signed message part.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEABECAAYFAk0H0X8ACgkQPJpCBAwIhbT3qQCfQ1ZzcXwIxMhh/jeQBgT+wibt
edYAn2dvD37VVLoNF+QtfCwFJOpCCWkj
=3/8D
-----END PGP SIGNATURE-----

--nextPart1421423.Zk8J7MEp3j--

From general-return-2537-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 15 12:21:49 2010
Return-Path: <general-return-2537-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29574 invoked from network); 15 Dec 2010 12:21:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Dec 2010 12:21:48 -0000
Received: (qmail 43723 invoked by uid 500); 15 Dec 2010 12:21:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 43261 invoked by uid 500); 15 Dec 2010 12:21:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 43253 invoked by uid 99); 15 Dec 2010 12:21:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Dec 2010 12:21:45 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Dec 2010 12:21:37 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id B1607B7DD4
	for <general@hadoop.apache.org>; Wed, 15 Dec 2010 12:21:14 +0000 (GMT)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id T+Opg2sSOopE for <general@hadoop.apache.org>;
	Wed, 15 Dec 2010 12:21:13 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 9160FB7DD3
	for <general@hadoop.apache.org>; Wed, 15 Dec 2010 12:21:13 +0000 (GMT)
MailScanner-NULL-Check: 1293020459.66333@GXdB3OxgiaxCNDGjcM7/UA
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id oBFCKvvV010028
	for <general@hadoop.apache.org>; Wed, 15 Dec 2010 12:20:57 GMT
Message-ID: <4D08B2A9.5010304@apache.org>
Date: Wed, 15 Dec 2010 12:20:57 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.15) Gecko/20101027 Thunderbird/3.0.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: How do I log from my map/reduce application?
References: <AANLkTi=4q=QtWz2ZFgr0fG6_7MEMbLj+JdR2MFYFD7-k@mail.gmail.com>
In-Reply-To: <AANLkTi=4q=QtWz2ZFgr0fG6_7MEMbLj+JdR2MFYFD7-k@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: oBFCKvvV010028
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 13/12/10 17:29, W.P. McNeill wrote:
> I would like to use Hadoop's Log4j infrastructure to do logging from my
> map/reduce application.  I think I've got everything set up correctly, but I
> am still unable to specify the logging level I want.
>
> By default Hadoop is set up to log at level INFO.  The first line of its
> log4j.properties file looks like this:
>
> hadoop.root.logger=INFO,console
>
>
> I have an application whose reducer looks like this:
>
> package com.me;
>
> public class MyReducer<...>  extends Reducer<...>  {
>     private static Logger logger =
> Logger.getLogger(MyReducer.class.getName());
>
>     ...
>     protected void reduce(...) {
>         logger.debug("My message");
>         ...
>     }
> }
>
>
> I've added the following line to the Hadoop log4j.properties file:
>
> log4j.logger.com.me.MyReducer=DEBUG
>
>
> I expect the Hadoop system to log at level INFO, but my application to log
> at level DEBUG, so that I see "My message" in the logs for the reducer task.
>   However, my application does not produce any log4j output.  If I change the
> line in my reducer to read logger.info("My message") the message does get
> logged, so somehow I'm failing to specify that log level for this class.
>
> I've also tried changing the log4j line for my app to
> read log4j.logger.com.me.MyReducer=DEBUG,console and get the same result.
>
> I've been through the Hadoop and log4j documentation and I can't figure out
> what I'm doing wrong.  Any suggestions?

I'd start with your logger logging @info level which log4j.properties 
file it's finding on the classpath. The code I use to find a resource 
looks like this:


    public URL findResource(String resource) {
         return getClass().getClassLoader().getResource(resource);
     }

Once you know which log4j file is being used to drive your code, then 
you can start tweaking it...

From general-return-2538-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 16 18:55:28 2010
Return-Path: <general-return-2538-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29861 invoked from network); 16 Dec 2010 18:55:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Dec 2010 18:55:28 -0000
Received: (qmail 39360 invoked by uid 500); 16 Dec 2010 18:55:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39247 invoked by uid 500); 16 Dec 2010 18:55:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39239 invoked by uid 99); 16 Dec 2010 18:55:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Dec 2010 18:55:25 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Dec 2010 18:55:19 +0000
Received: by vws18 with SMTP id 18so1503854vws.35
        for <general@hadoop.apache.org>; Thu, 16 Dec 2010 10:54:58 -0800 (PST)
Received: by 10.220.192.193 with SMTP id dr1mr2180449vcb.100.1292525698767;
        Thu, 16 Dec 2010 10:54:58 -0800 (PST)
Received: from [10.172.2.226] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id u4sm118816vch.36.2010.12.16.10.54.56
        (version=SSLv3 cipher=RC4-MD5);
        Thu, 16 Dec 2010 10:54:57 -0800 (PST)
From: Ian Holsman <ian@holsman.net>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: Plans for the 0.22 Release
Date: Fri, 17 Dec 2010 05:54:55 +1100
Message-Id: <27F4E872-C226-4386-9148-077F0044B6E9@Holsman.net>
To: general@hadoop.apache.org
Mime-Version: 1.0 (Apple Message framework v1082)
X-Mailer: Apple Mail (2.1082)

Hi Guys.

how are we going with the 0.22 release.

--
Ian Holsman
Ian@Holsman.net
PH: +1-703 879-3128 AOLIM: ianholsman Skype:iholsman

We are not afraid of the truth, in fact we plan on taking the truth out =
for a nice meal while we persuade it to adopt our views


From general-return-2539-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 16 19:16:27 2010
Return-Path: <general-return-2539-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47530 invoked from network); 16 Dec 2010 19:16:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Dec 2010 19:16:27 -0000
Received: (qmail 76305 invoked by uid 500); 16 Dec 2010 19:16:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76264 invoked by uid 500); 16 Dec 2010 19:16:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76256 invoked by uid 99); 16 Dec 2010 19:16:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Dec 2010 19:16:26 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.62.16] (HELO qmta01.westchester.pa.mail.comcast.net) (76.96.62.16)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Dec 2010 19:16:17 +0000
Received: from omta15.westchester.pa.mail.comcast.net ([76.96.62.87])
	by qmta01.westchester.pa.mail.comcast.net with comcast
	id jugq1f0081swQuc51vFxod; Thu, 16 Dec 2010 19:15:57 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta15.westchester.pa.mail.comcast.net with comcast
	id jvFn1f00A2SbwD53bvFrDL; Thu, 16 Dec 2010 19:15:55 +0000
Message-Id: <05ED6FF7-6236-47F4-B5DD-8E927971C72D@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <27F4E872-C226-4386-9148-077F0044B6E9@Holsman.net>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Plans for the 0.22 Release
Date: Thu, 16 Dec 2010 11:15:38 -0800
References: <27F4E872-C226-4386-9148-077F0044B6E9@Holsman.net>
X-Mailer: Apple Mail (2.936)


On Dec 16, 2010, at 10:54 AM, Ian Holsman wrote:

> how are we going with the 0.22 release.

Progress has been blocked because HADOOP-6685 is still blocked. I feel  
that HADOOP-6685 is critical for Hadoop and don't see much point in  
working on a 0.22 release without it. In particular, HADOOP-6685  
removes the need for third party serialization libraries that work  
around the current limitations of the MapReduce framework, but  
splinter the user community.

If someone else wants to take over as release manager for 0.22 while  
we find a solution to make forward progress on HADOOP-6685, that would  
be fine.

-- Owen

From general-return-2540-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 16 20:27:03 2010
Return-Path: <general-return-2540-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 89716 invoked from network); 16 Dec 2010 20:27:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Dec 2010 20:27:03 -0000
Received: (qmail 19446 invoked by uid 500); 16 Dec 2010 20:27:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19362 invoked by uid 500); 16 Dec 2010 20:27:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19354 invoked by uid 99); 16 Dec 2010 20:27:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Dec 2010 20:27:02 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Dec 2010 20:26:52 +0000
Received: by vws18 with SMTP id 18so1557027vws.35
        for <general@hadoop.apache.org>; Thu, 16 Dec 2010 12:26:31 -0800 (PST)
Received: by 10.220.185.204 with SMTP id cp12mr2134948vcb.235.1292531191201;
        Thu, 16 Dec 2010 12:26:31 -0800 (PST)
Received: from [10.172.2.226] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id r11sm195732vbx.11.2010.12.16.12.26.27
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Thu, 16 Dec 2010 12:26:30 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Plans for the 0.22 Release
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <05ED6FF7-6236-47F4-B5DD-8E927971C72D@apache.org>
Date: Fri, 17 Dec 2010 07:26:23 +1100
Content-Transfer-Encoding: quoted-printable
Message-Id: <39694716-7EF6-41B8-8862-D2DE03A4560C@Holsman.NET>
References: <27F4E872-C226-4386-9148-077F0044B6E9@Holsman.net> <05ED6FF7-6236-47F4-B5DD-8E927971C72D@apache.org>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org


On Dec 17, 2010, at 6:15 AM, Owen O'Malley wrote:

>=20
> On Dec 16, 2010, at 10:54 AM, Ian Holsman wrote:
>=20
>> how are we going with the 0.22 release.
>=20
> Progress has been blocked because HADOOP-6685 is still blocked. I feel =
that HADOOP-6685 is critical for Hadoop and don't see much point in =
working on a 0.22 release without it.

While it may be critical, I still don't understand why we couldn't =
release without it, and carry on the discussion.=20
Putting so much emphasis and pressure on this issue is unwarranted.=20

Everyone who has discussed this patch has said it isn't critical to =
hadoop, and It's holding up everything else 0.22 is going to bring.=20
There are now 4 separate companies who have released their own 0.20 =
version because of the delay on our releasing...
Delaying this release another 2-3 months for a single patch seems a bit =
extreme to me.

=20

> In particular, HADOOP-6685 removes the need for third party =
serialization libraries that work around the current limitations of the =
MapReduce framework, but splinter the user community.
>=20
> If someone else wants to take over as release manager for 0.22 while =
we find a solution to make forward progress on HADOOP-6685, that would =
be fine.
>=20
> -- Owen


From general-return-2541-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 16 22:01:09 2010
Return-Path: <general-return-2541-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 40624 invoked from network); 16 Dec 2010 22:01:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Dec 2010 22:01:09 -0000
Received: (qmail 77345 invoked by uid 500); 16 Dec 2010 22:01:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77231 invoked by uid 500); 16 Dec 2010 22:01:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77222 invoked by uid 99); 16 Dec 2010 22:01:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Dec 2010 22:01:03 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of owen.omalley@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Dec 2010 22:00:54 +0000
Received: by vws18 with SMTP id 18so10068vws.35
        for <general@hadoop.apache.org>; Thu, 16 Dec 2010 14:00:33 -0800 (PST)
Received: by 10.220.181.133 with SMTP id by5mr2318375vcb.182.1292536833801;
        Thu, 16 Dec 2010 14:00:33 -0800 (PST)
Received: from [10.72.120.91] (nat-dip4.cfw-a-gci.corp.yahoo.com [209.131.62.113])
        by mx.google.com with ESMTPS id bq5sm190621vcb.32.2010.12.16.14.00.32
        (version=SSLv3 cipher=RC4-MD5);
        Thu, 16 Dec 2010 14:00:33 -0800 (PST)
Message-Id: <9FC0DEC0-524A-4A35-88A6-17473DD74929@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <39694716-7EF6-41B8-8862-D2DE03A4560C@Holsman.NET>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Plans for the 0.22 Release
Date: Thu, 16 Dec 2010 14:00:29 -0800
References: <27F4E872-C226-4386-9148-077F0044B6E9@Holsman.net> <05ED6FF7-6236-47F4-B5DD-8E927971C72D@apache.org> <39694716-7EF6-41B8-8862-D2DE03A4560C@Holsman.NET>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


On Dec 16, 2010, at 12:26 PM, Ian Holsman wrote:

> While it may be critical, I still don't understand why we couldn't  
> release without it, and carry on the discussion.
> Putting so much emphasis and pressure on this issue is unwarranted.

It took months to stabilize 0.21 and it took a lot of Tom's time.  
H-6685 is important to me and needs to be addressed to move the  
project forward.

> Everyone who has discussed this patch has said it isn't critical to  
> hadoop, and It's holding up everything else 0.22 is going to bring.

I disagree that it isn't critical to Hadoop, but I'm not holding up  
0.22. I'm just not volunteering to spend my time working on it, if it  
doesn't have the features that I think it needs.

-- Owen


From general-return-2542-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 16 22:38:00 2010
Return-Path: <general-return-2542-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63531 invoked from network); 16 Dec 2010 22:38:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Dec 2010 22:38:00 -0000
Received: (qmail 21644 invoked by uid 500); 16 Dec 2010 22:37:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21599 invoked by uid 500); 16 Dec 2010 22:37:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21591 invoked by uid 99); 16 Dec 2010 22:37:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Dec 2010 22:37:58 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of kengoou@gmail.com designates 74.125.83.176 as permitted sender)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Dec 2010 22:37:53 +0000
Received: by pvb32 with SMTP id 32so8968pvb.35
        for <general@hadoop.apache.org>; Thu, 16 Dec 2010 14:37:33 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:from:date
         :message-id:subject:to:content-type;
        bh=gRVyPVgWgtG2oDcKnxZebRgXcJbEYY96xf8ukeWrhIw=;
        b=Tcnqarwmjca00Lt/wKaG8Z/8+n50nE/Y1APxYTBzLClSGNY/Q44HAT0GxmLHczJjsx
         lKK+YgTI6OpIClGmsD2B71snHjk2C9Zc6PoHyNnFb3IALA7sq14VgaqZoTsGS7WtJN8o
         teSTSceWvQhXGBsYtUUs2NoYH36Mon/El2KsA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type;
        b=dhB9qSK3ejRzGmLXF05KoSMkedQ1iVUU83S1/X9FU5fOYe01WoYFBtsVp6cvMj8qL6
         66vukAZN+zKrXxIcRskHpscglBzxZYzs9aYnywIAgKBbUbsh7y2x+vMOCCqJnK8TvAjU
         7fnp9MsWo74FjaxWEuXIBg8XduffWSW9yW4cU=
Received: by 10.142.166.20 with SMTP id o20mr96739wfe.382.1292539053006; Thu,
 16 Dec 2010 14:37:33 -0800 (PST)
MIME-Version: 1.0
Received: by 10.142.43.15 with HTTP; Thu, 16 Dec 2010 14:37:12 -0800 (PST)
From: kengoou <kengoou@gmail.com>
Date: Fri, 17 Dec 2010 07:37:12 +0900
Message-ID: <AANLkTimo=tE9KfBBwbtL_zhhrAMJqW_iGfsobDmxrMQq@mail.gmail.com>
Subject: I may translate and publish the content of ML in blog in Japanese?
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

This is kengoou that researches hadoop in Japan.
It is thought that the content of this ML is translated and will be
published in blog in Japanese by it ,
because Japanese hadoop information is a little.

Are there any problems after related to the right for this matter?

Thank you and best regards.

http://d.hatena.ne.jp/kengoou/searchdiary?word=*[hadoop]
kengoou@twitter

From general-return-2543-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 16 23:11:11 2010
Return-Path: <general-return-2543-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 86086 invoked from network); 16 Dec 2010 23:11:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Dec 2010 23:11:11 -0000
Received: (qmail 70424 invoked by uid 500); 16 Dec 2010 23:11:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70358 invoked by uid 500); 16 Dec 2010 23:11:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70349 invoked by uid 99); 16 Dec 2010 23:11:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Dec 2010 23:11:09 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Dec 2010 23:11:00 +0000
Received: by vws18 with SMTP id 18so29746vws.35
        for <general@hadoop.apache.org>; Thu, 16 Dec 2010 15:10:39 -0800 (PST)
Received: by 10.220.198.196 with SMTP id ep4mr10917vcb.110.1292541039063;
        Thu, 16 Dec 2010 15:10:39 -0800 (PST)
Received: from [10.172.2.226] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id q5sm218126vcr.15.2010.12.16.15.10.37
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Thu, 16 Dec 2010 15:10:38 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Plans for the 0.22 Release
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <9FC0DEC0-524A-4A35-88A6-17473DD74929@apache.org>
Date: Fri, 17 Dec 2010 10:10:33 +1100
Content-Transfer-Encoding: quoted-printable
Message-Id: <C9F82B37-7397-46D8-9788-679380749C1B@Holsman.NET>
References: <27F4E872-C226-4386-9148-077F0044B6E9@Holsman.net> <05ED6FF7-6236-47F4-B5DD-8E927971C72D@apache.org> <39694716-7EF6-41B8-8862-D2DE03A4560C@Holsman.NET> <9FC0DEC0-524A-4A35-88A6-17473DD74929@apache.org>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org


On Dec 17, 2010, at 9:00 AM, Owen O'Malley wrote:

>> Everyone who has discussed this patch has said it isn't critical to =
hadoop, and It's holding up everything else 0.22 is going to bring.
>=20
> I disagree that it isn't critical to Hadoop, but I'm not holding up =
0.22. I'm just not volunteering to spend my time working on it, if it =
doesn't have the features that I think it needs.
>=20
> -- Owen
>=20

That's a fair point.. and this is a volunteer effort.
Do we have anybody else who is willing to be the release manager for 22 =
with 6685?


From general-return-2544-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Dec 17 05:16:39 2010
Return-Path: <general-return-2544-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25705 invoked from network); 17 Dec 2010 05:16:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Dec 2010 05:16:38 -0000
Received: (qmail 98890 invoked by uid 500); 17 Dec 2010 05:16:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98576 invoked by uid 500); 17 Dec 2010 05:16:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98563 invoked by uid 99); 17 Dec 2010 05:16:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Dec 2010 05:16:36 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.83.46] (HELO mail-gw0-f46.google.com) (74.125.83.46)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Dec 2010 05:16:36 +0000
Received: by gwj20 with SMTP id 20so164674gwj.33
        for <general@hadoop.apache.org>; Thu, 16 Dec 2010 21:16:15 -0800 (PST)
Received: by 10.151.27.10 with SMTP id e10mr2097234ybj.225.1292562975126;
        Thu, 16 Dec 2010 21:16:15 -0800 (PST)
Received: from tp (c-24-4-185-157.hsd1.ca.comcast.net [24.4.185.157])
        by mx.google.com with ESMTPS id 72sm616720yhl.38.2010.12.16.21.16.13
        (version=SSLv3 cipher=RC4-MD5);
        Thu, 16 Dec 2010 21:16:14 -0800 (PST)
Sender: Konstantin Boudnik <cos@boudnik.org>
Date: Thu, 16 Dec 2010 21:16:10 -0800
From: Konstantin Boudnik <cos@apache.org>
To: general@hadoop.apache.org
Subject: Re: Plans for the 0.22 Release
Message-ID: <20101217051610.GC19072@tp>
References: <27F4E872-C226-4386-9148-077F0044B6E9@Holsman.net>
 <05ED6FF7-6236-47F4-B5DD-8E927971C72D@apache.org>
 <39694716-7EF6-41B8-8862-D2DE03A4560C@Holsman.NET>
 <9FC0DEC0-524A-4A35-88A6-17473DD74929@apache.org>
 <C9F82B37-7397-46D8-9788-679380749C1B@Holsman.NET>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature"; boundary="n+lFg1Zro7sl44OB"
Content-Disposition: inline
In-Reply-To: <C9F82B37-7397-46D8-9788-679380749C1B@Holsman.NET>
X-Organization: It's something of 'Cos
X-PGP-Key: http://www.boudnik.org/~cos/pubkey.asc
User-Agent: Mutt/1.5.20 (2009-06-14)

--n+lFg1Zro7sl44OB
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Fri, Dec 17, 2010 at 10:10AM, Ian Holsman wrote:
>=20
> On Dec 17, 2010, at 9:00 AM, Owen O'Malley wrote:
>=20
> >> Everyone who has discussed this patch has said it isn't critical to ha=
doop, and It's holding up everything else 0.22 is going to bring.
> >=20
> > I disagree that it isn't critical to Hadoop, but I'm not holding up 0.2=
2. I'm just not volunteering to spend my time working on it, if it doesn't =
have the features that I think it needs.
> >=20
> > -- Owen
> >=20
>=20
> That's a fair point.. and this is a volunteer effort.
> Do we have anybody else who is willing to be the release manager for 22 w=
ith 6685?

You meant to say "...to be the release manager for 22 without 6685?", didn'=
t you?

Cos


--n+lFg1Zro7sl44OB
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iF4EAREIAAYFAk0K8hkACgkQenyFlstYjhK6xwEA4yMihNnyX24JAvx+h1clYYV2
oaf3FKrX6SCWtOOUTh4BAOHH0ulnfc5oJR3XuFzHZk5GztNvQPxAjZVKJaEkapb2
=5J7O
-----END PGP SIGNATURE-----

--n+lFg1Zro7sl44OB--

From general-return-2545-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Dec 17 05:37:41 2010
Return-Path: <general-return-2545-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 41397 invoked from network); 17 Dec 2010 05:37:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Dec 2010 05:37:40 -0000
Received: (qmail 9497 invoked by uid 500); 17 Dec 2010 05:37:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 9295 invoked by uid 500); 17 Dec 2010 05:37:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 9285 invoked by uid 99); 17 Dec 2010 05:37:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Dec 2010 05:37:38 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Dec 2010 05:37:30 +0000
Received: by vws18 with SMTP id 18so113038vws.35
        for <general@hadoop.apache.org>; Thu, 16 Dec 2010 21:37:09 -0800 (PST)
Received: by 10.220.198.196 with SMTP id ep4mr103020vcb.110.1292564229756;
        Thu, 16 Dec 2010 21:37:09 -0800 (PST)
Received: from [10.172.2.226] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id e18sm388005vbm.15.2010.12.16.21.37.07
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Thu, 16 Dec 2010 21:37:09 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Plans for the 0.22 Release
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <20101217051610.GC19072@tp>
Date: Fri, 17 Dec 2010 16:37:03 +1100
Content-Transfer-Encoding: quoted-printable
Message-Id: <D5755408-E83E-4526-A8C0-B289D875F632@Holsman.NET>
References: <27F4E872-C226-4386-9148-077F0044B6E9@Holsman.net> <05ED6FF7-6236-47F4-B5DD-8E927971C72D@apache.org> <39694716-7EF6-41B8-8862-D2DE03A4560C@Holsman.NET> <9FC0DEC0-524A-4A35-88A6-17473DD74929@apache.org> <C9F82B37-7397-46D8-9788-679380749C1B@Holsman.NET> <20101217051610.GC19072@tp>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)


On Dec 17, 2010, at 4:16 PM, Konstantin Boudnik wrote:
>=20
>=20
> You meant to say "...to be the release manager for 22 without 6685?", =
didn't you?

Yes..=20
It's been a long day ;-(

>=20
> Cos
>=20


From general-return-2546-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Dec 17 22:34:50 2010
Return-Path: <general-return-2546-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91284 invoked from network); 17 Dec 2010 22:34:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Dec 2010 22:34:50 -0000
Received: (qmail 17820 invoked by uid 500); 17 Dec 2010 22:34:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17763 invoked by uid 500); 17 Dec 2010 22:34:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17754 invoked by uid 99); 17 Dec 2010 22:34:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Dec 2010 22:34:47 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of shv.hadoop@gmail.com designates 209.85.213.176 as permitted sender)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Dec 2010 22:34:42 +0000
Received: by yxm8 with SMTP id 8so618157yxm.35
        for <general@hadoop.apache.org>; Fri, 17 Dec 2010 14:34:22 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=ykieK1Rs8dhjByt1ga6fULTqqL9D2u3zjZXQUTJ8uQQ=;
        b=UefKviDVRYTqrG45fr6HfYmyFLUF+M38DPn5mkkFnNB/H+/JeeizP2MtjQQ4/lndsN
         AmtdZcncm5qaz9aHxYrxBt210+rIGcMYO3Xgn6In2ae7ORvjkdauCPmpIoRd+4FTrd1W
         hD+aPIwzWbUH9uIPcbz+EQBytmS9s1Xw/PZVQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=gubR1KiyjIsHgU4psqDmKSkgvvTuM2H0DJW/vRqM8pxV5aAuQPn3Etw1Z74mNYVXq1
         d+Gupa98U5slWcZzNMY89Iyor31uSlzvNY7aLyeSNfRbTe9zcYxV0dTwS6PSdXiSXKeu
         Bmr74MSIQb+DscotnFTk7SYFvo9V4KgH1meU4=
MIME-Version: 1.0
Received: by 10.236.108.177 with SMTP id q37mr2637914yhg.50.1292625261939;
 Fri, 17 Dec 2010 14:34:21 -0800 (PST)
Received: by 10.236.109.15 with HTTP; Fri, 17 Dec 2010 14:34:21 -0800 (PST)
In-Reply-To: <C9F82B37-7397-46D8-9788-679380749C1B@Holsman.NET>
References: <27F4E872-C226-4386-9148-077F0044B6E9@Holsman.net>
	<05ED6FF7-6236-47F4-B5DD-8E927971C72D@apache.org>
	<39694716-7EF6-41B8-8862-D2DE03A4560C@Holsman.NET>
	<9FC0DEC0-524A-4A35-88A6-17473DD74929@apache.org>
	<C9F82B37-7397-46D8-9788-679380749C1B@Holsman.NET>
Date: Fri, 17 Dec 2010 14:34:21 -0800
Message-ID: <AANLkTin9ZdZAVZSvE6yeSWHJKtfON0YiM5Gr55oLD62P@mail.gmail.com>
Subject: Re: Plans for the 0.22 Release
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba5bca2d90cdd30497a2c50e

--90e6ba5bca2d90cdd30497a2c50e
Content-Type: text/plain; charset=ISO-8859-1

Owen, Doug, Tom

Could you please formulate and reply to this email separately
     what would be an *ACCEPTABLE *resolution of
     HADOOP-6685 for *YOU *to move *0.22* forward.
Just trying to get something to work with to get us beyond the stagnation
point.

It could be "I want this patch in/out as is, final answer". Then we are
stuck.
But at least we will know there is no resolution to hope for anymore.
And we have to find other ways based on that fact.

It could be a zero-option plan - remove dependencies both for Avro and
ProtocolBuffers out into libraries, similar to schedulers.
Or something else.
Let's see if there is any common ground. If there is
we can further talk about implementation and in the mean time declare
the 0.22 freeze contingent on the completion of H-6685.

Thanks,
--Konstantin


On Thu, Dec 16, 2010 at 3:10 PM, Ian Holsman <hadoop@holsman.net> wrote:

>
> On Dec 17, 2010, at 9:00 AM, Owen O'Malley wrote:
>
> >> Everyone who has discussed this patch has said it isn't critical to
> hadoop, and It's holding up everything else 0.22 is going to bring.
> >
> > I disagree that it isn't critical to Hadoop, but I'm not holding up 0.22.
> I'm just not volunteering to spend my time working on it, if it doesn't have
> the features that I think it needs.
> >
> > -- Owen
> >
>
> That's a fair point.. and this is a volunteer effort.
> Do we have anybody else who is willing to be the release manager for 22
> with 6685?
>
>

--90e6ba5bca2d90cdd30497a2c50e--

From general-return-2547-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Dec 17 22:52:30 2010
Return-Path: <general-return-2547-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95055 invoked from network); 17 Dec 2010 22:52:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Dec 2010 22:52:30 -0000
Received: (qmail 29264 invoked by uid 500); 17 Dec 2010 22:52:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29217 invoked by uid 500); 17 Dec 2010 22:52:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29209 invoked by uid 99); 17 Dec 2010 22:52:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Dec 2010 22:52:29 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 17 Dec 2010 22:52:27 +0000
Received: (qmail 94762 invoked by uid 99); 17 Dec 2010 22:52:05 -0000
Received: from localhost.apache.org (HELO [192.168.168.16]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Dec 2010 22:52:05 +0000
Message-ID: <4D0BE994.4020207@apache.org>
Date: Fri, 17 Dec 2010 14:52:04 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Plans for the 0.22 Release
References: <27F4E872-C226-4386-9148-077F0044B6E9@Holsman.net>	<05ED6FF7-6236-47F4-B5DD-8E927971C72D@apache.org>	<39694716-7EF6-41B8-8862-D2DE03A4560C@Holsman.NET>	<9FC0DEC0-524A-4A35-88A6-17473DD74929@apache.org>	<C9F82B37-7397-46D8-9788-679380749C1B@Holsman.NET> <AANLkTin9ZdZAVZSvE6yeSWHJKtfON0YiM5Gr55oLD62P@mail.gmail.com>
In-Reply-To: <AANLkTin9ZdZAVZSvE6yeSWHJKtfON0YiM5Gr55oLD62P@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 12/17/2010 02:34 PM, Konstantin Shvachko wrote:
> It could be a zero-option plan - remove dependencies both for Avro and
> ProtocolBuffers out into libraries, similar to schedulers.

I'd be fine with removing Avro from the mapreduce user's classpath. 
It's currently an unused option for RPC, and is used server-side on the 
jobtracker.  It could be removed from the user classpath today without 
loss of critical functionality.

Doug

From general-return-2548-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Dec 17 23:20:10 2010
Return-Path: <general-return-2548-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12716 invoked from network); 17 Dec 2010 23:20:10 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Dec 2010 23:20:10 -0000
Received: (qmail 53321 invoked by uid 500); 17 Dec 2010 23:20:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53261 invoked by uid 500); 17 Dec 2010 23:20:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53244 invoked by uid 99); 17 Dec 2010 23:20:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Dec 2010 23:20:08 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jghoman@gmail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Dec 2010 23:20:01 +0000
Received: by iwn2 with SMTP id 2so1353886iwn.35
        for <general@hadoop.apache.org>; Fri, 17 Dec 2010 15:19:40 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=Lm1jxK7U94xDzJKHLPqTOY+pM8fIbj427H6Cbi+1YzU=;
        b=EWDFXV/iIBlK9crIard3i9FAZzbQdebauECaQrDiTmVEGVD7osEYQ5M8NnvpUL885H
         wABzdtrLtlV1o+1IMf609zVGZb9IEwnnepceGOo+OWQZk+FzGFIpqMYrAJ/n2y5yxtGf
         1nD9lhPppRqtELvK7/Z+L5cJ7OqOi22chbnPI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=wmdxl+oIh1LX7WDJYwL2celm6lGaSBJ6eihRZ+n2xb2dmNeemTugOn8bJa+QZj/4Am
         0na5Kvm6qxX9jgog07Xq6nPlQp4wT70DAy5hsHvkYFxENxQKRkdxEns3fx8j+ywZuEZX
         UloZMDyP1cfrkd9qXiYpP5gAuV5MyHEvytqAs=
Received: by 10.231.15.202 with SMTP id l10mr1387685iba.19.1292627980615; Fri,
 17 Dec 2010 15:19:40 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.25.163 with HTTP; Fri, 17 Dec 2010 15:19:20 -0800 (PST)
In-Reply-To: <4CE47C88.5050203@yahoo-inc.com>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com> <20101020195420.GG2075@tp>
 <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com> <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
 <7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com> <AANLkTimLXA+CvUs-09LMN3ob6GZqmhxiyrKO7pzWEfse@mail.gmail.com>
 <4CE46119.2030509@yahoo-inc.com> <8617BECB-78B4-42B6-B592-D7FC1F8DA923@mac.com>
 <4CE47C88.5050203@yahoo-inc.com>
From: Jakob Homan <jghoman@gmail.com>
Date: Fri, 17 Dec 2010 15:19:20 -0800
Message-ID: <AANLkTinTM5h-3NupRSsFh-DnyvFYw+_OsXpASxcQUSDn@mail.gmail.com>
Subject: Re: Patch testing
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

So, with test-patch updated to show the failing tests, saving the
developers the need to go and verify that the failed tests are all
known, how do people feel about turning on test-patch again for HDFS
and mapred?  I think it'll help prevent any more tests from entering
the "yeah, we know" category.

Thanks,
jg


On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <jhoman@yahoo-inc.com> wrote:
> True, each patch would get a -1 and the failing tests would need to be
> verified as those known bad (BTW, it would be great if Hudson could list
> which tests failed in the message it posts to JIRA). =A0But that's still =
quite
> a bit less error-prone work than if the developer runs the tests and
> test-patch themselves. =A0Also, with 22 being cut, there are a lot of pat=
ches
> up in the air and several developers are juggling multiple patches. =A0Th=
e
> more automation we can have, even if it's not perfect, will decrease erro=
rs
> we may make.
> -jg
>
> Nigel Daley wrote:
>>
>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>
>>>> It's also ready to run on MapReduce and HDFS but we won't turn it on
>>>> until these projects build and test cleanly. =A0Looks like both these =
projects
>>>> currently have test failures.
>>>
>>> Assuming the projects are compiling and building, is there a reason to
>>> not turn it on despite the test failures? Hudson is invaluable to devel=
opers
>>> who then don't have to run the tests and test-patch themselves. =A0We d=
idn't
>>> turn Hudson off when it was working previously and there were known
>>> failures. =A0I think one of the reasons we have more failing tests now =
is the
>>> higher cost of doing Hudson's work (not a great excuse I know). =A0This=
 is
>>> particularly true now because several of the failing tests involve test=
s
>>> timing out, making the whole testing regime even longer.
>>
>> Every single patch would get a -1 and need investigation. =A0Currently, =
that
>> would be about 83 investigations between MR and HDFS issues that are in
>> patch available state. =A0Shouldn't we focus on getting these tests fixe=
d or
>> removed/? =A0Also, I need to get MAPREDUCE-2172 fixed (applies to HDFS a=
s
>> well) before I turn this on.
>>
>> Cheers,
>> Nige
>
>

From general-return-2549-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Dec 17 23:24:08 2010
Return-Path: <general-return-2549-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14591 invoked from network); 17 Dec 2010 23:24:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Dec 2010 23:24:08 -0000
Received: (qmail 58133 invoked by uid 500); 17 Dec 2010 23:24:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57857 invoked by uid 500); 17 Dec 2010 23:24:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57849 invoked by uid 99); 17 Dec 2010 23:24:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Dec 2010 23:24:06 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Dec 2010 23:24:00 +0000
Received: by qwh6 with SMTP id 6so1273423qwh.35
        for <general@hadoop.apache.org>; Fri, 17 Dec 2010 15:23:39 -0800 (PST)
MIME-Version: 1.0
Received: by 10.229.91.211 with SMTP id o19mr1269358qcm.286.1292628219024;
 Fri, 17 Dec 2010 15:23:39 -0800 (PST)
Received: by 10.229.225.137 with HTTP; Fri, 17 Dec 2010 15:23:38 -0800 (PST)
In-Reply-To: <AANLkTinTM5h-3NupRSsFh-DnyvFYw+_OsXpASxcQUSDn@mail.gmail.com>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com>
	<20101020195420.GG2075@tp>
	<53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com>
	<4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
	<7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com>
	<AANLkTimLXA+CvUs-09LMN3ob6GZqmhxiyrKO7pzWEfse@mail.gmail.com>
	<4CE46119.2030509@yahoo-inc.com>
	<8617BECB-78B4-42B6-B592-D7FC1F8DA923@mac.com>
	<4CE47C88.5050203@yahoo-inc.com>
	<AANLkTinTM5h-3NupRSsFh-DnyvFYw+_OsXpASxcQUSDn@mail.gmail.com>
Date: Fri, 17 Dec 2010 15:23:38 -0800
Message-ID: <AANLkTin3=k6BbZq9SX1Aw1Jy_u2W1FTRByCexs6zixLr@mail.gmail.com>
Subject: Re: Patch testing
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

+1

I think all the known failing tests should block the release as well.

Thanks,
Eli

On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jghoman@gmail.com> wrote:
> So, with test-patch updated to show the failing tests, saving the
> developers the need to go and verify that the failed tests are all
> known, how do people feel about turning on test-patch again for HDFS
> and mapred? =A0I think it'll help prevent any more tests from entering
> the "yeah, we know" category.
>
> Thanks,
> jg
>
>
> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <jhoman@yahoo-inc.com> wrote=
:
>> True, each patch would get a -1 and the failing tests would need to be
>> verified as those known bad (BTW, it would be great if Hudson could list
>> which tests failed in the message it posts to JIRA). =A0But that's still=
 quite
>> a bit less error-prone work than if the developer runs the tests and
>> test-patch themselves. =A0Also, with 22 being cut, there are a lot of pa=
tches
>> up in the air and several developers are juggling multiple patches. =A0T=
he
>> more automation we can have, even if it's not perfect, will decrease err=
ors
>> we may make.
>> -jg
>>
>> Nigel Daley wrote:
>>>
>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>>
>>>>> It's also ready to run on MapReduce and HDFS but we won't turn it on
>>>>> until these projects build and test cleanly. =A0Looks like both these=
 projects
>>>>> currently have test failures.
>>>>
>>>> Assuming the projects are compiling and building, is there a reason to
>>>> not turn it on despite the test failures? Hudson is invaluable to deve=
lopers
>>>> who then don't have to run the tests and test-patch themselves. =A0We =
didn't
>>>> turn Hudson off when it was working previously and there were known
>>>> failures. =A0I think one of the reasons we have more failing tests now=
 is the
>>>> higher cost of doing Hudson's work (not a great excuse I know). =A0Thi=
s is
>>>> particularly true now because several of the failing tests involve tes=
ts
>>>> timing out, making the whole testing regime even longer.
>>>
>>> Every single patch would get a -1 and need investigation. =A0Currently,=
 that
>>> would be about 83 investigations between MR and HDFS issues that are in
>>> patch available state. =A0Shouldn't we focus on getting these tests fix=
ed or
>>> removed/? =A0Also, I need to get MAPREDUCE-2172 fixed (applies to HDFS =
as
>>> well) before I turn this on.
>>>
>>> Cheers,
>>> Nige
>>
>>
>

From general-return-2550-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Dec 17 23:26:31 2010
Return-Path: <general-return-2550-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14907 invoked from network); 17 Dec 2010 23:26:31 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Dec 2010 23:26:31 -0000
Received: (qmail 59754 invoked by uid 500); 17 Dec 2010 23:26:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59566 invoked by uid 500); 17 Dec 2010 23:26:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59558 invoked by uid 99); 17 Dec 2010 23:26:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Dec 2010 23:26:30 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jghoman@gmail.com designates 209.85.210.176 as permitted sender)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Dec 2010 23:26:25 +0000
Received: by iyb26 with SMTP id 26so871820iyb.35
        for <general@hadoop.apache.org>; Fri, 17 Dec 2010 15:26:04 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=r/sIADtBxWtNXazDNIHo1V7cNCLc6cO6aLUgdRu8ieE=;
        b=wc8dPz542L5RUuyFomp+PzRthiQr9ycCevul6WeeDbRdz8RDuuyknejA+yt1ZMYwBj
         BFi6Kzzu7X/6I5IkHul4G5O19sQTU3ROpGjJ+Snq2UH6OaKKuZP8jmkgzUMDmhINnYOV
         psM+xQPPZBY5KApJPLkr1Xe3eHYQmA0N6IrRk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=ZYNmAua021hCz2r9FyVzx9oOj2yayy0KPquCXCTxFzDBaYYSzJM2BBbTVVwK148INV
         Vf5dSI5bYMKFeN/c14lULw5/3ZKmfEPSczI21/BX6FVydJMPHqjGJRINoUiP/EUqtARp
         2HPqfYEN2rf66pTbuXwRL0eOOQSawXchlMLpE=
Received: by 10.231.182.85 with SMTP id cb21mr1377250ibb.44.1292628364792;
 Fri, 17 Dec 2010 15:26:04 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.25.163 with HTTP; Fri, 17 Dec 2010 15:25:44 -0800 (PST)
In-Reply-To: <AANLkTin3=k6BbZq9SX1Aw1Jy_u2W1FTRByCexs6zixLr@mail.gmail.com>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com> <20101020195420.GG2075@tp>
 <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com> <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
 <7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com> <AANLkTimLXA+CvUs-09LMN3ob6GZqmhxiyrKO7pzWEfse@mail.gmail.com>
 <4CE46119.2030509@yahoo-inc.com> <8617BECB-78B4-42B6-B592-D7FC1F8DA923@mac.com>
 <4CE47C88.5050203@yahoo-inc.com> <AANLkTinTM5h-3NupRSsFh-DnyvFYw+_OsXpASxcQUSDn@mail.gmail.com>
 <AANLkTin3=k6BbZq9SX1Aw1Jy_u2W1FTRByCexs6zixLr@mail.gmail.com>
From: Jakob Homan <jghoman@gmail.com>
Date: Fri, 17 Dec 2010 15:25:44 -0800
Message-ID: <AANLkTimaE2MJ4X4EFTDiKEWVZcODSxKkHWHMyW34x2fM@mail.gmail.com>
Subject: Re: Patch testing
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Doh.  Forgot to include a shout-out to Nigel for adding the
failing-tests-list to test-patch. Thanks!

On Fri, Dec 17, 2010 at 3:23 PM, Eli Collins <eli@cloudera.com> wrote:
> +1
>
> I think all the known failing tests should block the release as well.
>
> Thanks,
> Eli
>
> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jghoman@gmail.com> wrote:
>> So, with test-patch updated to show the failing tests, saving the
>> developers the need to go and verify that the failed tests are all
>> known, how do people feel about turning on test-patch again for HDFS
>> and mapred? =A0I think it'll help prevent any more tests from entering
>> the "yeah, we know" category.
>>
>> Thanks,
>> jg
>>
>>
>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <jhoman@yahoo-inc.com> wrot=
e:
>>> True, each patch would get a -1 and the failing tests would need to be
>>> verified as those known bad (BTW, it would be great if Hudson could lis=
t
>>> which tests failed in the message it posts to JIRA). =A0But that's stil=
l quite
>>> a bit less error-prone work than if the developer runs the tests and
>>> test-patch themselves. =A0Also, with 22 being cut, there are a lot of p=
atches
>>> up in the air and several developers are juggling multiple patches. =A0=
The
>>> more automation we can have, even if it's not perfect, will decrease er=
rors
>>> we may make.
>>> -jg
>>>
>>> Nigel Daley wrote:
>>>>
>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>>>
>>>>>> It's also ready to run on MapReduce and HDFS but we won't turn it on
>>>>>> until these projects build and test cleanly. =A0Looks like both thes=
e projects
>>>>>> currently have test failures.
>>>>>
>>>>> Assuming the projects are compiling and building, is there a reason t=
o
>>>>> not turn it on despite the test failures? Hudson is invaluable to dev=
elopers
>>>>> who then don't have to run the tests and test-patch themselves. =A0We=
 didn't
>>>>> turn Hudson off when it was working previously and there were known
>>>>> failures. =A0I think one of the reasons we have more failing tests no=
w is the
>>>>> higher cost of doing Hudson's work (not a great excuse I know). =A0Th=
is is
>>>>> particularly true now because several of the failing tests involve te=
sts
>>>>> timing out, making the whole testing regime even longer.
>>>>
>>>> Every single patch would get a -1 and need investigation. =A0Currently=
, that
>>>> would be about 83 investigations between MR and HDFS issues that are i=
n
>>>> patch available state. =A0Shouldn't we focus on getting these tests fi=
xed or
>>>> removed/? =A0Also, I need to get MAPREDUCE-2172 fixed (applies to HDFS=
 as
>>>> well) before I turn this on.
>>>>
>>>> Cheers,
>>>> Nige
>>>
>>>
>>
>

From general-return-2551-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Dec 17 23:46:59 2010
Return-Path: <general-return-2551-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19704 invoked from network); 17 Dec 2010 23:46:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Dec 2010 23:46:59 -0000
Received: (qmail 77498 invoked by uid 500); 17 Dec 2010 23:46:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77361 invoked by uid 500); 17 Dec 2010 23:46:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77353 invoked by uid 99); 17 Dec 2010 23:46:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Dec 2010 23:46:57 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Dec 2010 23:46:51 +0000
Received: by wwd20 with SMTP id 20so1273792wwd.29
        for <general@hadoop.apache.org>; Fri, 17 Dec 2010 15:46:30 -0800 (PST)
Received: by 10.216.63.15 with SMTP id z15mr1745142wec.74.1292629590352; Fri,
 17 Dec 2010 15:46:30 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.59.13 with HTTP; Fri, 17 Dec 2010 15:46:10 -0800 (PST)
In-Reply-To: <AANLkTinTM5h-3NupRSsFh-DnyvFYw+_OsXpASxcQUSDn@mail.gmail.com>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com> <20101020195420.GG2075@tp>
 <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com> <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
 <7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com> <AANLkTimLXA+CvUs-09LMN3ob6GZqmhxiyrKO7pzWEfse@mail.gmail.com>
 <4CE46119.2030509@yahoo-inc.com> <8617BECB-78B4-42B6-B592-D7FC1F8DA923@mac.com>
 <4CE47C88.5050203@yahoo-inc.com> <AANLkTinTM5h-3NupRSsFh-DnyvFYw+_OsXpASxcQUSDn@mail.gmail.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Fri, 17 Dec 2010 15:46:10 -0800
X-Google-Sender-Auth: NnlyVHrpatNusZAgFe64D2XBGI8
Message-ID: <AANLkTimBUeuYDBzvKsAZuXmhjgMMOAHDKvnAkSot8G7A@mail.gmail.com>
Subject: Re: Patch testing
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

I do believe that it makes sense to wait a bit longer before doing
this. If HDFS is added to the test-patch queue right now we get
nothing but dozens of -1'ed patches. Number of failing tests has been
reduced from 16 down to 4. Of those for 1 is a real bug (reviled by
HDFS-903) and other three (all of the same nature) are needed to be
investigated more.

I'm for fixing the trunk first.

On Fri, Dec 17, 2010 at 15:19, Jakob Homan <jghoman@gmail.com> wrote:
> So, with test-patch updated to show the failing tests, saving the
> developers the need to go and verify that the failed tests are all
> known, how do people feel about turning on test-patch again for HDFS
> and mapred? =A0I think it'll help prevent any more tests from entering
> the "yeah, we know" category.
>
> Thanks,
> jg
>
>
> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <jhoman@yahoo-inc.com> wrote=
:
>> True, each patch would get a -1 and the failing tests would need to be
>> verified as those known bad (BTW, it would be great if Hudson could list
>> which tests failed in the message it posts to JIRA). =A0But that's still=
 quite
>> a bit less error-prone work than if the developer runs the tests and
>> test-patch themselves. =A0Also, with 22 being cut, there are a lot of pa=
tches
>> up in the air and several developers are juggling multiple patches. =A0T=
he
>> more automation we can have, even if it's not perfect, will decrease err=
ors
>> we may make.
>> -jg
>>
>> Nigel Daley wrote:
>>>
>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>>
>>>>> It's also ready to run on MapReduce and HDFS but we won't turn it on
>>>>> until these projects build and test cleanly. =A0Looks like both these=
 projects
>>>>> currently have test failures.
>>>>
>>>> Assuming the projects are compiling and building, is there a reason to
>>>> not turn it on despite the test failures? Hudson is invaluable to deve=
lopers
>>>> who then don't have to run the tests and test-patch themselves. =A0We =
didn't
>>>> turn Hudson off when it was working previously and there were known
>>>> failures. =A0I think one of the reasons we have more failing tests now=
 is the
>>>> higher cost of doing Hudson's work (not a great excuse I know). =A0Thi=
s is
>>>> particularly true now because several of the failing tests involve tes=
ts
>>>> timing out, making the whole testing regime even longer.
>>>
>>> Every single patch would get a -1 and need investigation. =A0Currently,=
 that
>>> would be about 83 investigations between MR and HDFS issues that are in
>>> patch available state. =A0Shouldn't we focus on getting these tests fix=
ed or
>>> removed/? =A0Also, I need to get MAPREDUCE-2172 fixed (applies to HDFS =
as
>>> well) before I turn this on.
>>>
>>> Cheers,
>>> Nige
>>
>>
>

From general-return-2552-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Dec 17 23:48:36 2010
Return-Path: <general-return-2552-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19899 invoked from network); 17 Dec 2010 23:48:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Dec 2010 23:48:36 -0000
Received: (qmail 78910 invoked by uid 500); 17 Dec 2010 23:48:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78841 invoked by uid 500); 17 Dec 2010 23:48:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78833 invoked by uid 99); 17 Dec 2010 23:48:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Dec 2010 23:48:34 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Dec 2010 23:48:30 +0000
Received: by wwd20 with SMTP id 20so1274659wwd.29
        for <general@hadoop.apache.org>; Fri, 17 Dec 2010 15:48:08 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=AQENCSYq86SH550hVPpOhuNGPQJl15/5ks0pkzbRbxI=;
        b=aUl4NGQhfiwlWTYjzaEJDA+vLpXc/LSwLVxmY1qqWcg4EhToYmO0F3uRAvkf2n/pmA
         GIrNULQafPbS1rdg+FdXy1tsel+Qp4vi+dnEsasYaFtX2ZYVMwJAbJWeD5m1VxBj7UFa
         y9ilpU7xGT4t/9T/WqYqXRcHr65GUaImbNPl4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=QhT52kRYHZafGMeftsC+R9GolgcL8pugQw3bZmf+jFjuqRT/4HuWPDseeNl0jcjVMD
         nQutUVqiTbgt1e8NUHXJzKN7Ec9zQIz5gmtqoSrXNGoevLF7odGHqBhtTqb43PJ1T+p9
         rxEl/zQQSfsS12C7c2P9dD2gd5XrkxNQXHJWk=
MIME-Version: 1.0
Received: by 10.216.179.144 with SMTP id h16mr1745760wem.64.1292629688674;
 Fri, 17 Dec 2010 15:48:08 -0800 (PST)
Received: by 10.216.242.205 with HTTP; Fri, 17 Dec 2010 15:48:08 -0800 (PST)
In-Reply-To: <AANLkTinTM5h-3NupRSsFh-DnyvFYw+_OsXpASxcQUSDn@mail.gmail.com>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com>
	<20101020195420.GG2075@tp>
	<53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com>
	<4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
	<7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com>
	<AANLkTimLXA+CvUs-09LMN3ob6GZqmhxiyrKO7pzWEfse@mail.gmail.com>
	<4CE46119.2030509@yahoo-inc.com>
	<8617BECB-78B4-42B6-B592-D7FC1F8DA923@mac.com>
	<4CE47C88.5050203@yahoo-inc.com>
	<AANLkTinTM5h-3NupRSsFh-DnyvFYw+_OsXpASxcQUSDn@mail.gmail.com>
Date: Fri, 17 Dec 2010 15:48:08 -0800
Message-ID: <AANLkTimSOxaXZi-VLjwuTB5Ch5tRt0bKozc5L+Nx=ZrQ@mail.gmail.com>
Subject: Re: Patch testing
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64c1e766b6ab60497a3cd88

--0016e64c1e766b6ab60497a3cd88
Content-Type: text/plain; charset=ISO-8859-1

+1, thanks for doing this.

On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jghoman@gmail.com> wrote:

> So, with test-patch updated to show the failing tests, saving the
> developers the need to go and verify that the failed tests are all
> known, how do people feel about turning on test-patch again for HDFS
> and mapred?  I think it'll help prevent any more tests from entering
> the "yeah, we know" category.
>
> Thanks,
> jg
>
>
> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <jhoman@yahoo-inc.com> wrote:
> > True, each patch would get a -1 and the failing tests would need to be
> > verified as those known bad (BTW, it would be great if Hudson could list
> > which tests failed in the message it posts to JIRA).  But that's still
> quite
> > a bit less error-prone work than if the developer runs the tests and
> > test-patch themselves.  Also, with 22 being cut, there are a lot of
> patches
> > up in the air and several developers are juggling multiple patches.  The
> > more automation we can have, even if it's not perfect, will decrease
> errors
> > we may make.
> > -jg
> >
> > Nigel Daley wrote:
> >>
> >> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
> >>
> >>>> It's also ready to run on MapReduce and HDFS but we won't turn it on
> >>>> until these projects build and test cleanly.  Looks like both these
> projects
> >>>> currently have test failures.
> >>>
> >>> Assuming the projects are compiling and building, is there a reason to
> >>> not turn it on despite the test failures? Hudson is invaluable to
> developers
> >>> who then don't have to run the tests and test-patch themselves.  We
> didn't
> >>> turn Hudson off when it was working previously and there were known
> >>> failures.  I think one of the reasons we have more failing tests now is
> the
> >>> higher cost of doing Hudson's work (not a great excuse I know).  This
> is
> >>> particularly true now because several of the failing tests involve
> tests
> >>> timing out, making the whole testing regime even longer.
> >>
> >> Every single patch would get a -1 and need investigation.  Currently,
> that
> >> would be about 83 investigations between MR and HDFS issues that are in
> >> patch available state.  Shouldn't we focus on getting these tests fixed
> or
> >> removed/?  Also, I need to get MAPREDUCE-2172 fixed (applies to HDFS as
> >> well) before I turn this on.
> >>
> >> Cheers,
> >> Nige
> >
> >
>



-- 
Connect to me at http://www.facebook.com/dhruba

--0016e64c1e766b6ab60497a3cd88--

From general-return-2553-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Dec 17 23:56:22 2010
Return-Path: <general-return-2553-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22058 invoked from network); 17 Dec 2010 23:56:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Dec 2010 23:56:22 -0000
Received: (qmail 84556 invoked by uid 500); 17 Dec 2010 23:56:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84505 invoked by uid 500); 17 Dec 2010 23:56:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84497 invoked by uid 99); 17 Dec 2010 23:56:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Dec 2010 23:56:20 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jghoman@gmail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Dec 2010 23:56:16 +0000
Received: by iwn2 with SMTP id 2so1381383iwn.35
        for <general@hadoop.apache.org>; Fri, 17 Dec 2010 15:55:56 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=x+nNr6xqtG9loIpGt5nqYzIRoG2bJieHoyPM9QXMj4U=;
        b=SDyaVUWGzukdGVxK3lLXAwC2mMUhJvqiKZ8hkMZxiBRFcVBNP+757e7n6E/tHvC+Re
         Gq/5zQ7gQe0PCCpCXy7QeO6myJY2pQN5k5xI0aK7dh1TGWmHpJOpj6lgQOGanSFZD67/
         8pmnh2OzMEtO3KYUQI6wcIseu/f28l0fkDDVE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=YplcG9x4xpxsjWI3FQtG7bMUS9RvMAtfXACAOWJKchEw2iKzBKroYVe51VQQYA4HY7
         PSQb5mAWU/4jLwkXlEmOBYctG84Eqq9dL0U8w3TkmnYoQC1f1zQU0AjUoV+dUblvrsGV
         4opWPaQBiKCZQ4KtoNB+ZwitxPAzlVvsdW2zo=
Received: by 10.42.176.129 with SMTP id be1mr1491731icb.48.1292630156090; Fri,
 17 Dec 2010 15:55:56 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.25.163 with HTTP; Fri, 17 Dec 2010 15:55:35 -0800 (PST)
In-Reply-To: <AANLkTimSOxaXZi-VLjwuTB5Ch5tRt0bKozc5L+Nx=ZrQ@mail.gmail.com>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com> <20101020195420.GG2075@tp>
 <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com> <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
 <7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com> <AANLkTimLXA+CvUs-09LMN3ob6GZqmhxiyrKO7pzWEfse@mail.gmail.com>
 <4CE46119.2030509@yahoo-inc.com> <8617BECB-78B4-42B6-B592-D7FC1F8DA923@mac.com>
 <4CE47C88.5050203@yahoo-inc.com> <AANLkTinTM5h-3NupRSsFh-DnyvFYw+_OsXpASxcQUSDn@mail.gmail.com>
 <AANLkTimSOxaXZi-VLjwuTB5Ch5tRt0bKozc5L+Nx=ZrQ@mail.gmail.com>
From: Jakob Homan <jghoman@gmail.com>
Date: Fri, 17 Dec 2010 15:55:35 -0800
Message-ID: <AANLkTineU2v=Sx-=oC=6ROt6Os8TEbnS3L8xz1ZkoxCc@mail.gmail.com>
Subject: Re: Patch testing
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

> If HDFS is added to the test-patch queue right now we get
> nothing but dozens of -1'ed patches.
There aren't dozens of patches being submitted currently.  The -1
isn't the important thing, it's the grunt work of actually running
(and waiting) for the tests, test-patch, etc. that Hudson does so that
the developer doesn't have to.

On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <dhruba@gmail.com> wrote:
> +1, thanks for doing this.
>
> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jghoman@gmail.com> wrote:
>
>> So, with test-patch updated to show the failing tests, saving the
>> developers the need to go and verify that the failed tests are all
>> known, how do people feel about turning on test-patch again for HDFS
>> and mapred? =A0I think it'll help prevent any more tests from entering
>> the "yeah, we know" category.
>>
>> Thanks,
>> jg
>>
>>
>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <jhoman@yahoo-inc.com> wrot=
e:
>> > True, each patch would get a -1 and the failing tests would need to be
>> > verified as those known bad (BTW, it would be great if Hudson could li=
st
>> > which tests failed in the message it posts to JIRA). =A0But that's sti=
ll
>> quite
>> > a bit less error-prone work than if the developer runs the tests and
>> > test-patch themselves. =A0Also, with 22 being cut, there are a lot of
>> patches
>> > up in the air and several developers are juggling multiple patches. =
=A0The
>> > more automation we can have, even if it's not perfect, will decrease
>> errors
>> > we may make.
>> > -jg
>> >
>> > Nigel Daley wrote:
>> >>
>> >> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>> >>
>> >>>> It's also ready to run on MapReduce and HDFS but we won't turn it o=
n
>> >>>> until these projects build and test cleanly. =A0Looks like both the=
se
>> projects
>> >>>> currently have test failures.
>> >>>
>> >>> Assuming the projects are compiling and building, is there a reason =
to
>> >>> not turn it on despite the test failures? Hudson is invaluable to
>> developers
>> >>> who then don't have to run the tests and test-patch themselves. =A0W=
e
>> didn't
>> >>> turn Hudson off when it was working previously and there were known
>> >>> failures. =A0I think one of the reasons we have more failing tests n=
ow is
>> the
>> >>> higher cost of doing Hudson's work (not a great excuse I know). =A0T=
his
>> is
>> >>> particularly true now because several of the failing tests involve
>> tests
>> >>> timing out, making the whole testing regime even longer.
>> >>
>> >> Every single patch would get a -1 and need investigation. =A0Currentl=
y,
>> that
>> >> would be about 83 investigations between MR and HDFS issues that are =
in
>> >> patch available state. =A0Shouldn't we focus on getting these tests f=
ixed
>> or
>> >> removed/? =A0Also, I need to get MAPREDUCE-2172 fixed (applies to HDF=
S as
>> >> well) before I turn this on.
>> >>
>> >> Cheers,
>> >> Nige
>> >
>> >
>>
>
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
>

From general-return-2554-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Dec 18 00:17:51 2010
Return-Path: <general-return-2554-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34963 invoked from network); 18 Dec 2010 00:17:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Dec 2010 00:17:51 -0000
Received: (qmail 96499 invoked by uid 500); 18 Dec 2010 00:17:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96444 invoked by uid 500); 18 Dec 2010 00:17:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96436 invoked by uid 99); 18 Dec 2010 00:17:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Dec 2010 00:17:50 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Dec 2010 00:17:45 +0000
Received: by wye20 with SMTP id 20so1285232wye.35
        for <general@hadoop.apache.org>; Fri, 17 Dec 2010 16:17:24 -0800 (PST)
Received: by 10.216.163.203 with SMTP id a53mr1747962wel.104.1292631443841;
 Fri, 17 Dec 2010 16:17:23 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.59.13 with HTTP; Fri, 17 Dec 2010 16:17:03 -0800 (PST)
In-Reply-To: <AANLkTineU2v=Sx-=oC=6ROt6Os8TEbnS3L8xz1ZkoxCc@mail.gmail.com>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com> <20101020195420.GG2075@tp>
 <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com> <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
 <7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com> <AANLkTimLXA+CvUs-09LMN3ob6GZqmhxiyrKO7pzWEfse@mail.gmail.com>
 <4CE46119.2030509@yahoo-inc.com> <8617BECB-78B4-42B6-B592-D7FC1F8DA923@mac.com>
 <4CE47C88.5050203@yahoo-inc.com> <AANLkTinTM5h-3NupRSsFh-DnyvFYw+_OsXpASxcQUSDn@mail.gmail.com>
 <AANLkTimSOxaXZi-VLjwuTB5Ch5tRt0bKozc5L+Nx=ZrQ@mail.gmail.com> <AANLkTineU2v=Sx-=oC=6ROt6Os8TEbnS3L8xz1ZkoxCc@mail.gmail.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Fri, 17 Dec 2010 16:17:03 -0800
X-Google-Sender-Auth: 6_KCzhU1-4D5v-tB8j253kms5mM
Message-ID: <AANLkTin+oucXdCdOo7LQTtqndTf1axiT0gQntiOZ_mQJ@mail.gmail.com>
Subject: Re: Patch testing
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Considering that because of these 4 faulty cases every patch will be
-1'ed a patch author will still have to look at it and make a comment
why this particular -1 isn't valid. Lesser work, perhaps, but messier
IMO. I'm not blocking it - I just feel like there's a better way.

--
=A0 Take care,
Konstantin (Cos) Boudnik



On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jghoman@gmail.com> wrote:
>> If HDFS is added to the test-patch queue right now we get
>> nothing but dozens of -1'ed patches.
> There aren't dozens of patches being submitted currently. =A0The -1
> isn't the important thing, it's the grunt work of actually running
> (and waiting) for the tests, test-patch, etc. that Hudson does so that
> the developer doesn't have to.
>
> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <dhruba@gmail.com> wrot=
e:
>> +1, thanks for doing this.
>>
>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jghoman@gmail.com> wrote:
>>
>>> So, with test-patch updated to show the failing tests, saving the
>>> developers the need to go and verify that the failed tests are all
>>> known, how do people feel about turning on test-patch again for HDFS
>>> and mapred? =A0I think it'll help prevent any more tests from entering
>>> the "yeah, we know" category.
>>>
>>> Thanks,
>>> jg
>>>
>>>
>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <jhoman@yahoo-inc.com> wro=
te:
>>> > True, each patch would get a -1 and the failing tests would need to b=
e
>>> > verified as those known bad (BTW, it would be great if Hudson could l=
ist
>>> > which tests failed in the message it posts to JIRA). =A0But that's st=
ill
>>> quite
>>> > a bit less error-prone work than if the developer runs the tests and
>>> > test-patch themselves. =A0Also, with 22 being cut, there are a lot of
>>> patches
>>> > up in the air and several developers are juggling multiple patches. =
=A0The
>>> > more automation we can have, even if it's not perfect, will decrease
>>> errors
>>> > we may make.
>>> > -jg
>>> >
>>> > Nigel Daley wrote:
>>> >>
>>> >> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>> >>
>>> >>>> It's also ready to run on MapReduce and HDFS but we won't turn it =
on
>>> >>>> until these projects build and test cleanly. =A0Looks like both th=
ese
>>> projects
>>> >>>> currently have test failures.
>>> >>>
>>> >>> Assuming the projects are compiling and building, is there a reason=
 to
>>> >>> not turn it on despite the test failures? Hudson is invaluable to
>>> developers
>>> >>> who then don't have to run the tests and test-patch themselves. =A0=
We
>>> didn't
>>> >>> turn Hudson off when it was working previously and there were known
>>> >>> failures. =A0I think one of the reasons we have more failing tests =
now is
>>> the
>>> >>> higher cost of doing Hudson's work (not a great excuse I know). =A0=
This
>>> is
>>> >>> particularly true now because several of the failing tests involve
>>> tests
>>> >>> timing out, making the whole testing regime even longer.
>>> >>
>>> >> Every single patch would get a -1 and need investigation. =A0Current=
ly,
>>> that
>>> >> would be about 83 investigations between MR and HDFS issues that are=
 in
>>> >> patch available state. =A0Shouldn't we focus on getting these tests =
fixed
>>> or
>>> >> removed/? =A0Also, I need to get MAPREDUCE-2172 fixed (applies to HD=
FS as
>>> >> well) before I turn this on.
>>> >>
>>> >> Cheers,
>>> >> Nige
>>> >
>>> >
>>>
>>
>>
>>
>> --
>> Connect to me at http://www.facebook.com/dhruba
>>
>

From general-return-2555-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Dec 18 03:02:07 2010
Return-Path: <general-return-2555-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91534 invoked from network); 18 Dec 2010 03:02:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Dec 2010 03:02:07 -0000
Received: (qmail 82568 invoked by uid 500); 18 Dec 2010 03:02:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82513 invoked by uid 500); 18 Dec 2010 03:02:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82504 invoked by uid 99); 18 Dec 2010 03:02:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Dec 2010 03:02:04 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Dec 2010 03:01:46 +0000
Received: by wwd20 with SMTP id 20so1358895wwd.29
        for <general@hadoop.apache.org>; Fri, 17 Dec 2010 19:01:25 -0800 (PST)
Received: by 10.216.186.76 with SMTP id v54mr4592185wem.74.1292641285456; Fri,
 17 Dec 2010 19:01:25 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.59.13 with HTTP; Fri, 17 Dec 2010 19:01:05 -0800 (PST)
In-Reply-To: <AANLkTin+oucXdCdOo7LQTtqndTf1axiT0gQntiOZ_mQJ@mail.gmail.com>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com> <20101020195420.GG2075@tp>
 <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com> <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
 <7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com> <AANLkTimLXA+CvUs-09LMN3ob6GZqmhxiyrKO7pzWEfse@mail.gmail.com>
 <4CE46119.2030509@yahoo-inc.com> <8617BECB-78B4-42B6-B592-D7FC1F8DA923@mac.com>
 <4CE47C88.5050203@yahoo-inc.com> <AANLkTinTM5h-3NupRSsFh-DnyvFYw+_OsXpASxcQUSDn@mail.gmail.com>
 <AANLkTimSOxaXZi-VLjwuTB5Ch5tRt0bKozc5L+Nx=ZrQ@mail.gmail.com>
 <AANLkTineU2v=Sx-=oC=6ROt6Os8TEbnS3L8xz1ZkoxCc@mail.gmail.com> <AANLkTin+oucXdCdOo7LQTtqndTf1axiT0gQntiOZ_mQJ@mail.gmail.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Fri, 17 Dec 2010 19:01:05 -0800
X-Google-Sender-Auth: _NgXICKG3sWtdhV9XvoAuS8jtDg
Message-ID: <AANLkTinGn4Pb6_LkrY4-kK56PdHX4C1Zfu7A2PVfUoub@mail.gmail.com>
Subject: Re: Patch testing
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

One more issue needs to be addressed before test-patch is turned on HDFS is
  https://issues.apache.org/jira/browse/HDFS-1511
--
=A0 Take care,
Konstantin (Cos) Boudnik



On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <cos@apache.org> wrote:
> Considering that because of these 4 faulty cases every patch will be
> -1'ed a patch author will still have to look at it and make a comment
> why this particular -1 isn't valid. Lesser work, perhaps, but messier
> IMO. I'm not blocking it - I just feel like there's a better way.
>
> --
> =A0 Take care,
> Konstantin (Cos) Boudnik
>
>
>
> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jghoman@gmail.com> wrote:
>>> If HDFS is added to the test-patch queue right now we get
>>> nothing but dozens of -1'ed patches.
>> There aren't dozens of patches being submitted currently. =A0The -1
>> isn't the important thing, it's the grunt work of actually running
>> (and waiting) for the tests, test-patch, etc. that Hudson does so that
>> the developer doesn't have to.
>>
>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <dhruba@gmail.com> wro=
te:
>>> +1, thanks for doing this.
>>>
>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jghoman@gmail.com> wrote:
>>>
>>>> So, with test-patch updated to show the failing tests, saving the
>>>> developers the need to go and verify that the failed tests are all
>>>> known, how do people feel about turning on test-patch again for HDFS
>>>> and mapred? =A0I think it'll help prevent any more tests from entering
>>>> the "yeah, we know" category.
>>>>
>>>> Thanks,
>>>> jg
>>>>
>>>>
>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <jhoman@yahoo-inc.com> wr=
ote:
>>>> > True, each patch would get a -1 and the failing tests would need to =
be
>>>> > verified as those known bad (BTW, it would be great if Hudson could =
list
>>>> > which tests failed in the message it posts to JIRA). =A0But that's s=
till
>>>> quite
>>>> > a bit less error-prone work than if the developer runs the tests and
>>>> > test-patch themselves. =A0Also, with 22 being cut, there are a lot o=
f
>>>> patches
>>>> > up in the air and several developers are juggling multiple patches. =
=A0The
>>>> > more automation we can have, even if it's not perfect, will decrease
>>>> errors
>>>> > we may make.
>>>> > -jg
>>>> >
>>>> > Nigel Daley wrote:
>>>> >>
>>>> >> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>>> >>
>>>> >>>> It's also ready to run on MapReduce and HDFS but we won't turn it=
 on
>>>> >>>> until these projects build and test cleanly. =A0Looks like both t=
hese
>>>> projects
>>>> >>>> currently have test failures.
>>>> >>>
>>>> >>> Assuming the projects are compiling and building, is there a reaso=
n to
>>>> >>> not turn it on despite the test failures? Hudson is invaluable to
>>>> developers
>>>> >>> who then don't have to run the tests and test-patch themselves. =
=A0We
>>>> didn't
>>>> >>> turn Hudson off when it was working previously and there were know=
n
>>>> >>> failures. =A0I think one of the reasons we have more failing tests=
 now is
>>>> the
>>>> >>> higher cost of doing Hudson's work (not a great excuse I know). =
=A0This
>>>> is
>>>> >>> particularly true now because several of the failing tests involve
>>>> tests
>>>> >>> timing out, making the whole testing regime even longer.
>>>> >>
>>>> >> Every single patch would get a -1 and need investigation. =A0Curren=
tly,
>>>> that
>>>> >> would be about 83 investigations between MR and HDFS issues that ar=
e in
>>>> >> patch available state. =A0Shouldn't we focus on getting these tests=
 fixed
>>>> or
>>>> >> removed/? =A0Also, I need to get MAPREDUCE-2172 fixed (applies to H=
DFS as
>>>> >> well) before I turn this on.
>>>> >>
>>>> >> Cheers,
>>>> >> Nige
>>>> >
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> Connect to me at http://www.facebook.com/dhruba
>>>
>>
>

From general-return-2556-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Dec 18 03:17:30 2010
Return-Path: <general-return-2556-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1821 invoked from network); 18 Dec 2010 03:17:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Dec 2010 03:17:30 -0000
Received: (qmail 91284 invoked by uid 500); 18 Dec 2010 03:17:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91132 invoked by uid 500); 18 Dec 2010 03:17:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91124 invoked by uid 99); 18 Dec 2010 03:17:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Dec 2010 03:17:28 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Dec 2010 03:17:23 +0000
Received: by iwn2 with SMTP id 2so1505244iwn.35
        for <general@hadoop.apache.org>; Fri, 17 Dec 2010 19:17:03 -0800 (PST)
Received: by 10.231.16.131 with SMTP id o3mr1539073iba.5.1292642223113; Fri,
 17 Dec 2010 19:17:03 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.188.102 with HTTP; Fri, 17 Dec 2010 19:16:43 -0800 (PST)
In-Reply-To: <AANLkTineU2v=Sx-=oC=6ROt6Os8TEbnS3L8xz1ZkoxCc@mail.gmail.com>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com> <20101020195420.GG2075@tp>
 <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com> <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
 <7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com> <AANLkTimLXA+CvUs-09LMN3ob6GZqmhxiyrKO7pzWEfse@mail.gmail.com>
 <4CE46119.2030509@yahoo-inc.com> <8617BECB-78B4-42B6-B592-D7FC1F8DA923@mac.com>
 <4CE47C88.5050203@yahoo-inc.com> <AANLkTinTM5h-3NupRSsFh-DnyvFYw+_OsXpASxcQUSDn@mail.gmail.com>
 <AANLkTimSOxaXZi-VLjwuTB5Ch5tRt0bKozc5L+Nx=ZrQ@mail.gmail.com> <AANLkTineU2v=Sx-=oC=6ROt6Os8TEbnS3L8xz1ZkoxCc@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Fri, 17 Dec 2010 19:16:43 -0800
Message-ID: <AANLkTinZCV0jvHOVr-NfsCBhide9R8YFfAABS_0-=AkB@mail.gmail.com>
Subject: Re: Patch testing
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Fri, Dec 17, 2010 at 3:55 PM, Jakob Homan <jghoman@gmail.com> wrote:
>> If HDFS is added to the test-patch queue right now we get
>> nothing but dozens of -1'ed patches.
> There aren't dozens of patches being submitted currently. =A0The -1
> isn't the important thing, it's the grunt work of actually running
> (and waiting) for the tests, test-patch, etc. that Hudson does so that
> the developer doesn't have to.
>

I agree with Jakob. I've had to run and re-run the test-patch and unit
tests probably 30 times over the last two weeks, and it takes a lot of
effort, since my own infrastructure for doing this is a bit messy. I'd
much rather just reply to the Hudson comments saying "these are known
issues" than have to run the tests, check the results, copy and paste
them and *then* say "these are known issues" anyway!

> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <dhruba@gmail.com> wrot=
e:
>> +1, thanks for doing this.
>>
>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jghoman@gmail.com> wrote:
>>
>>> So, with test-patch updated to show the failing tests, saving the
>>> developers the need to go and verify that the failed tests are all
>>> known, how do people feel about turning on test-patch again for HDFS
>>> and mapred? =A0I think it'll help prevent any more tests from entering
>>> the "yeah, we know" category.
>>>
>>> Thanks,
>>> jg
>>>
>>>
>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <jhoman@yahoo-inc.com> wro=
te:
>>> > True, each patch would get a -1 and the failing tests would need to b=
e
>>> > verified as those known bad (BTW, it would be great if Hudson could l=
ist
>>> > which tests failed in the message it posts to JIRA). =A0But that's st=
ill
>>> quite
>>> > a bit less error-prone work than if the developer runs the tests and
>>> > test-patch themselves. =A0Also, with 22 being cut, there are a lot of
>>> patches
>>> > up in the air and several developers are juggling multiple patches. =
=A0The
>>> > more automation we can have, even if it's not perfect, will decrease
>>> errors
>>> > we may make.
>>> > -jg
>>> >
>>> > Nigel Daley wrote:
>>> >>
>>> >> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>> >>
>>> >>>> It's also ready to run on MapReduce and HDFS but we won't turn it =
on
>>> >>>> until these projects build and test cleanly. =A0Looks like both th=
ese
>>> projects
>>> >>>> currently have test failures.
>>> >>>
>>> >>> Assuming the projects are compiling and building, is there a reason=
 to
>>> >>> not turn it on despite the test failures? Hudson is invaluable to
>>> developers
>>> >>> who then don't have to run the tests and test-patch themselves. =A0=
We
>>> didn't
>>> >>> turn Hudson off when it was working previously and there were known
>>> >>> failures. =A0I think one of the reasons we have more failing tests =
now is
>>> the
>>> >>> higher cost of doing Hudson's work (not a great excuse I know). =A0=
This
>>> is
>>> >>> particularly true now because several of the failing tests involve
>>> tests
>>> >>> timing out, making the whole testing regime even longer.
>>> >>
>>> >> Every single patch would get a -1 and need investigation. =A0Current=
ly,
>>> that
>>> >> would be about 83 investigations between MR and HDFS issues that are=
 in
>>> >> patch available state. =A0Shouldn't we focus on getting these tests =
fixed
>>> or
>>> >> removed/? =A0Also, I need to get MAPREDUCE-2172 fixed (applies to HD=
FS as
>>> >> well) before I turn this on.
>>> >>
>>> >> Cheers,
>>> >> Nige
>>> >
>>> >
>>>
>>
>>
>>
>> --
>> Connect to me at http://www.facebook.com/dhruba
>>
>



--=20
Todd Lipcon
Software Engineer, Cloudera

From general-return-2557-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Dec 18 03:22:58 2010
Return-Path: <general-return-2557-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 2322 invoked from network); 18 Dec 2010 03:22:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Dec 2010 03:22:58 -0000
Received: (qmail 94273 invoked by uid 500); 18 Dec 2010 03:22:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94070 invoked by uid 500); 18 Dec 2010 03:22:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94062 invoked by uid 99); 18 Dec 2010 03:22:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Dec 2010 03:22:56 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.103 as permitted sender)
Received: from [17.148.16.103] (HELO asmtpout028.mac.com) (17.148.16.103)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Dec 2010 03:22:51 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.101.150.89] ([166.205.136.43])
 by asmtp028.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LDL00LK3TDFQF80@asmtp028.mac.com>
 for general@hadoop.apache.org; Fri, 17 Dec 2010 19:22:30 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2010-12-17_08:2010-12-17,2010-12-17,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1012170219
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com>
 <20101020195420.GG2075@tp>
 <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com>
 <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
 <7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com>
 <AANLkTimLXA+CvUs-09LMN3ob6GZqmhxiyrKO7pzWEfse@mail.gmail.com>
 <4CE46119.2030509@yahoo-inc.com>
 <8617BECB-78B4-42B6-B592-D7FC1F8DA923@mac.com>
 <4CE47C88.5050203@yahoo-inc.com>
 <AANLkTinTM5h-3NupRSsFh-DnyvFYw+_OsXpASxcQUSDn@mail.gmail.com>
 <AANLkTimSOxaXZi-VLjwuTB5Ch5tRt0bKozc5L+Nx=ZrQ@mail.gmail.com>
 <AANLkTineU2v=Sx-=oC=6ROt6Os8TEbnS3L8xz1ZkoxCc@mail.gmail.com>
 <AANLkTin+oucXdCdOo7LQTtqndTf1axiT0gQntiOZ_mQJ@mail.gmail.com>
 <AANLkTinGn4Pb6_LkrY4-kK56PdHX4C1Zfu7A2PVfUoub@mail.gmail.com>
In-reply-to: <AANLkTinGn4Pb6_LkrY4-kK56PdHX4C1Zfu7A2PVfUoub@mail.gmail.com>
Message-id: <41FB0800-3703-49C1-8069-DEB74FFE6FAC@mac.com>
Cc: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: iPhone Mail (8C148)
From: Nigel Daley <ndaley@mac.com>
Subject: Re: Patch testing
Date: Fri, 17 Dec 2010 19:22:22 -0800
To: "general@hadoop.apache.org" <general@hadoop.apache.org>

I agree with Cos on fixing HDFS-1511 first. Once that is done I'll enable hdfs patch testing. 

Cheers,
Nige

Sent from my iPhone4

On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <cos@apache.org> wrote:

> One more issue needs to be addressed before test-patch is turned on HDFS is
>  https://issues.apache.org/jira/browse/HDFS-1511
> --
>   Take care,
> Konstantin (Cos) Boudnik
> 
> 
> 
> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <cos@apache.org> wrote:
>> Considering that because of these 4 faulty cases every patch will be
>> -1'ed a patch author will still have to look at it and make a comment
>> why this particular -1 isn't valid. Lesser work, perhaps, but messier
>> IMO. I'm not blocking it - I just feel like there's a better way.
>> 
>> --
>>   Take care,
>> Konstantin (Cos) Boudnik
>> 
>> 
>> 
>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jghoman@gmail.com> wrote:
>>>> If HDFS is added to the test-patch queue right now we get
>>>> nothing but dozens of -1'ed patches.
>>> There aren't dozens of patches being submitted currently.  The -1
>>> isn't the important thing, it's the grunt work of actually running
>>> (and waiting) for the tests, test-patch, etc. that Hudson does so that
>>> the developer doesn't have to.
>>> 
>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <dhruba@gmail.com> wrote:
>>>> +1, thanks for doing this.
>>>> 
>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jghoman@gmail.com> wrote:
>>>> 
>>>>> So, with test-patch updated to show the failing tests, saving the
>>>>> developers the need to go and verify that the failed tests are all
>>>>> known, how do people feel about turning on test-patch again for HDFS
>>>>> and mapred?  I think it'll help prevent any more tests from entering
>>>>> the "yeah, we know" category.
>>>>> 
>>>>> Thanks,
>>>>> jg
>>>>> 
>>>>> 
>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <jhoman@yahoo-inc.com> wrote:
>>>>>> True, each patch would get a -1 and the failing tests would need to be
>>>>>> verified as those known bad (BTW, it would be great if Hudson could list
>>>>>> which tests failed in the message it posts to JIRA).  But that's still
>>>>> quite
>>>>>> a bit less error-prone work than if the developer runs the tests and
>>>>>> test-patch themselves.  Also, with 22 being cut, there are a lot of
>>>>> patches
>>>>>> up in the air and several developers are juggling multiple patches.  The
>>>>>> more automation we can have, even if it's not perfect, will decrease
>>>>> errors
>>>>>> we may make.
>>>>>> -jg
>>>>>> 
>>>>>> Nigel Daley wrote:
>>>>>>> 
>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>>>>>> 
>>>>>>>>> It's also ready to run on MapReduce and HDFS but we won't turn it on
>>>>>>>>> until these projects build and test cleanly.  Looks like both these
>>>>> projects
>>>>>>>>> currently have test failures.
>>>>>>>> 
>>>>>>>> Assuming the projects are compiling and building, is there a reason to
>>>>>>>> not turn it on despite the test failures? Hudson is invaluable to
>>>>> developers
>>>>>>>> who then don't have to run the tests and test-patch themselves.  We
>>>>> didn't
>>>>>>>> turn Hudson off when it was working previously and there were known
>>>>>>>> failures.  I think one of the reasons we have more failing tests now is
>>>>> the
>>>>>>>> higher cost of doing Hudson's work (not a great excuse I know).  This
>>>>> is
>>>>>>>> particularly true now because several of the failing tests involve
>>>>> tests
>>>>>>>> timing out, making the whole testing regime even longer.
>>>>>>> 
>>>>>>> Every single patch would get a -1 and need investigation.  Currently,
>>>>> that
>>>>>>> would be about 83 investigations between MR and HDFS issues that are in
>>>>>>> patch available state.  Shouldn't we focus on getting these tests fixed
>>>>> or
>>>>>>> removed/?  Also, I need to get MAPREDUCE-2172 fixed (applies to HDFS as
>>>>>>> well) before I turn this on.
>>>>>>> 
>>>>>>> Cheers,
>>>>>>> Nige
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Connect to me at http://www.facebook.com/dhruba
>>>> 
>>> 
>> 

From general-return-2558-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Dec 18 03:42:31 2010
Return-Path: <general-return-2558-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6662 invoked from network); 18 Dec 2010 03:42:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Dec 2010 03:42:30 -0000
Received: (qmail 10206 invoked by uid 500); 18 Dec 2010 03:42:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 9911 invoked by uid 500); 18 Dec 2010 03:42:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 9903 invoked by uid 99); 18 Dec 2010 03:42:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Dec 2010 03:42:28 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jghoman@gmail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Dec 2010 03:42:21 +0000
Received: by iwn2 with SMTP id 2so1518980iwn.35
        for <general@hadoop.apache.org>; Fri, 17 Dec 2010 19:42:01 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=Mq0cf3TYVtYEu1+xF8Gruo/B9rXmNV/J5q1U8nEyBcI=;
        b=bqrmLhnjYyCYtVpqbU31knoQoJgrlvQFudHkvcqbkYLawyXs7z0AoWbRNgwKk6B3x1
         PKS1c9YZP0YCSZ9l3mf9h8kAzCpn0Tgi3NTOgogNelxbucHjHDJtJdT9OQIqZ8iqfZ/C
         QfvYoN2mPNlN+CW2ZC9M8JmmOuCjwlK0foKCo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=ZZU7i9QRUSWDpQkYTOiuLCaW7LYkgKQ7RCyM+PnWBq0VoFUJe1eoDMKDT3mC++vmPS
         9SjS3hSs4lLK68Bu9/xd1DAXHbClF0X3xQGOD/MnL2HHP3LpUyt3OD+9WyY7nsszcevB
         +KHJLo/Uwfbb/oEs3hreTC+mxuaS7Cr3sbxmQ=
Received: by 10.231.16.68 with SMTP id n4mr1549175iba.94.1292643720997; Fri,
 17 Dec 2010 19:42:00 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.25.163 with HTTP; Fri, 17 Dec 2010 19:41:40 -0800 (PST)
In-Reply-To: <41FB0800-3703-49C1-8069-DEB74FFE6FAC@mac.com>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com> <20101020195420.GG2075@tp>
 <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com> <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
 <7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com> <AANLkTimLXA+CvUs-09LMN3ob6GZqmhxiyrKO7pzWEfse@mail.gmail.com>
 <4CE46119.2030509@yahoo-inc.com> <8617BECB-78B4-42B6-B592-D7FC1F8DA923@mac.com>
 <4CE47C88.5050203@yahoo-inc.com> <AANLkTinTM5h-3NupRSsFh-DnyvFYw+_OsXpASxcQUSDn@mail.gmail.com>
 <AANLkTimSOxaXZi-VLjwuTB5Ch5tRt0bKozc5L+Nx=ZrQ@mail.gmail.com>
 <AANLkTineU2v=Sx-=oC=6ROt6Os8TEbnS3L8xz1ZkoxCc@mail.gmail.com>
 <AANLkTin+oucXdCdOo7LQTtqndTf1axiT0gQntiOZ_mQJ@mail.gmail.com>
 <AANLkTinGn4Pb6_LkrY4-kK56PdHX4C1Zfu7A2PVfUoub@mail.gmail.com> <41FB0800-3703-49C1-8069-DEB74FFE6FAC@mac.com>
From: Jakob Homan <jghoman@gmail.com>
Date: Fri, 17 Dec 2010 19:41:40 -0800
Message-ID: <AANLkTik_rNLMHj3JT6Ekdop-NU_qzcB=VdsX+k4uFBtf@mail.gmail.com>
Subject: Re: Patch testing
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Ok.  I'll get a patch out for 1511 tomorrow, unless someone wants to
whip one up tonight.


On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley <ndaley@mac.com> wrote:
> I agree with Cos on fixing HDFS-1511 first. Once that is done I'll enable=
 hdfs patch testing.
>
> Cheers,
> Nige
>
> Sent from my iPhone4
>
> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <cos@apache.org> wrote:
>
>> One more issue needs to be addressed before test-patch is turned on HDFS=
 is
>> =A0https://issues.apache.org/jira/browse/HDFS-1511
>> --
>> =A0 Take care,
>> Konstantin (Cos) Boudnik
>>
>>
>>
>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <cos@apache.org> wrote=
:
>>> Considering that because of these 4 faulty cases every patch will be
>>> -1'ed a patch author will still have to look at it and make a comment
>>> why this particular -1 isn't valid. Lesser work, perhaps, but messier
>>> IMO. I'm not blocking it - I just feel like there's a better way.
>>>
>>> --
>>> =A0 Take care,
>>> Konstantin (Cos) Boudnik
>>>
>>>
>>>
>>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jghoman@gmail.com> wrote:
>>>>> If HDFS is added to the test-patch queue right now we get
>>>>> nothing but dozens of -1'ed patches.
>>>> There aren't dozens of patches being submitted currently. =A0The -1
>>>> isn't the important thing, it's the grunt work of actually running
>>>> (and waiting) for the tests, test-patch, etc. that Hudson does so that
>>>> the developer doesn't have to.
>>>>
>>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <dhruba@gmail.com> w=
rote:
>>>>> +1, thanks for doing this.
>>>>>
>>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jghoman@gmail.com> wrot=
e:
>>>>>
>>>>>> So, with test-patch updated to show the failing tests, saving the
>>>>>> developers the need to go and verify that the failed tests are all
>>>>>> known, how do people feel about turning on test-patch again for HDFS
>>>>>> and mapred? =A0I think it'll help prevent any more tests from enteri=
ng
>>>>>> the "yeah, we know" category.
>>>>>>
>>>>>> Thanks,
>>>>>> jg
>>>>>>
>>>>>>
>>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <jhoman@yahoo-inc.com> =
wrote:
>>>>>>> True, each patch would get a -1 and the failing tests would need to=
 be
>>>>>>> verified as those known bad (BTW, it would be great if Hudson could=
 list
>>>>>>> which tests failed in the message it posts to JIRA). =A0But that's =
still
>>>>>> quite
>>>>>>> a bit less error-prone work than if the developer runs the tests an=
d
>>>>>>> test-patch themselves. =A0Also, with 22 being cut, there are a lot =
of
>>>>>> patches
>>>>>>> up in the air and several developers are juggling multiple patches.=
 =A0The
>>>>>>> more automation we can have, even if it's not perfect, will decreas=
e
>>>>>> errors
>>>>>>> we may make.
>>>>>>> -jg
>>>>>>>
>>>>>>> Nigel Daley wrote:
>>>>>>>>
>>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>>>>>>>
>>>>>>>>>> It's also ready to run on MapReduce and HDFS but we won't turn i=
t on
>>>>>>>>>> until these projects build and test cleanly. =A0Looks like both =
these
>>>>>> projects
>>>>>>>>>> currently have test failures.
>>>>>>>>>
>>>>>>>>> Assuming the projects are compiling and building, is there a reas=
on to
>>>>>>>>> not turn it on despite the test failures? Hudson is invaluable to
>>>>>> developers
>>>>>>>>> who then don't have to run the tests and test-patch themselves. =
=A0We
>>>>>> didn't
>>>>>>>>> turn Hudson off when it was working previously and there were kno=
wn
>>>>>>>>> failures. =A0I think one of the reasons we have more failing test=
s now is
>>>>>> the
>>>>>>>>> higher cost of doing Hudson's work (not a great excuse I know). =
=A0This
>>>>>> is
>>>>>>>>> particularly true now because several of the failing tests involv=
e
>>>>>> tests
>>>>>>>>> timing out, making the whole testing regime even longer.
>>>>>>>>
>>>>>>>> Every single patch would get a -1 and need investigation. =A0Curre=
ntly,
>>>>>> that
>>>>>>>> would be about 83 investigations between MR and HDFS issues that a=
re in
>>>>>>>> patch available state. =A0Shouldn't we focus on getting these test=
s fixed
>>>>>> or
>>>>>>>> removed/? =A0Also, I need to get MAPREDUCE-2172 fixed (applies to =
HDFS as
>>>>>>>> well) before I turn this on.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Nige
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Connect to me at http://www.facebook.com/dhruba
>>>>>
>>>>
>>>
>

From general-return-2559-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Dec 18 04:32:23 2010
Return-Path: <general-return-2559-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21850 invoked from network); 18 Dec 2010 04:32:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Dec 2010 04:32:22 -0000
Received: (qmail 34656 invoked by uid 500); 18 Dec 2010 04:32:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34500 invoked by uid 500); 18 Dec 2010 04:32:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34492 invoked by uid 99); 18 Dec 2010 04:32:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Dec 2010 04:32:20 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Dec 2010 04:32:15 +0000
Received: by wye20 with SMTP id 20so1395222wye.35
        for <general@hadoop.apache.org>; Fri, 17 Dec 2010 20:31:54 -0800 (PST)
Received: by 10.216.46.19 with SMTP id q19mr1707835web.0.1292646713422; Fri,
 17 Dec 2010 20:31:53 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.59.13 with HTTP; Fri, 17 Dec 2010 20:31:33 -0800 (PST)
In-Reply-To: <AANLkTik_rNLMHj3JT6Ekdop-NU_qzcB=VdsX+k4uFBtf@mail.gmail.com>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com> <20101020195420.GG2075@tp>
 <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com> <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
 <7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com> <AANLkTimLXA+CvUs-09LMN3ob6GZqmhxiyrKO7pzWEfse@mail.gmail.com>
 <4CE46119.2030509@yahoo-inc.com> <8617BECB-78B4-42B6-B592-D7FC1F8DA923@mac.com>
 <4CE47C88.5050203@yahoo-inc.com> <AANLkTinTM5h-3NupRSsFh-DnyvFYw+_OsXpASxcQUSDn@mail.gmail.com>
 <AANLkTimSOxaXZi-VLjwuTB5Ch5tRt0bKozc5L+Nx=ZrQ@mail.gmail.com>
 <AANLkTineU2v=Sx-=oC=6ROt6Os8TEbnS3L8xz1ZkoxCc@mail.gmail.com>
 <AANLkTin+oucXdCdOo7LQTtqndTf1axiT0gQntiOZ_mQJ@mail.gmail.com>
 <AANLkTinGn4Pb6_LkrY4-kK56PdHX4C1Zfu7A2PVfUoub@mail.gmail.com>
 <41FB0800-3703-49C1-8069-DEB74FFE6FAC@mac.com> <AANLkTik_rNLMHj3JT6Ekdop-NU_qzcB=VdsX+k4uFBtf@mail.gmail.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Fri, 17 Dec 2010 20:31:33 -0800
X-Google-Sender-Auth: JO3p-aEIzkBcdA3g4w6PwqxPyb4
Message-ID: <AANLkTin9=FPxv+0EydK4mWp_1zJbgGTddevvF9MrHW80@mail.gmail.com>
Subject: Re: Patch testing
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Thanks Jacob. I am wasted already but I can do it on Sun, I think,
unless it is done earlier.
--
=A0 Take care,
Konstantin (Cos) Boudnik



On Fri, Dec 17, 2010 at 19:41, Jakob Homan <jghoman@gmail.com> wrote:
> Ok. =A0I'll get a patch out for 1511 tomorrow, unless someone wants to
> whip one up tonight.
>
>
> On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley <ndaley@mac.com> wrote:
>> I agree with Cos on fixing HDFS-1511 first. Once that is done I'll enabl=
e hdfs patch testing.
>>
>> Cheers,
>> Nige
>>
>> Sent from my iPhone4
>>
>> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <cos@apache.org> wrote:
>>
>>> One more issue needs to be addressed before test-patch is turned on HDF=
S is
>>> =A0https://issues.apache.org/jira/browse/HDFS-1511
>>> --
>>> =A0 Take care,
>>> Konstantin (Cos) Boudnik
>>>
>>>
>>>
>>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <cos@apache.org> wrot=
e:
>>>> Considering that because of these 4 faulty cases every patch will be
>>>> -1'ed a patch author will still have to look at it and make a comment
>>>> why this particular -1 isn't valid. Lesser work, perhaps, but messier
>>>> IMO. I'm not blocking it - I just feel like there's a better way.
>>>>
>>>> --
>>>> =A0 Take care,
>>>> Konstantin (Cos) Boudnik
>>>>
>>>>
>>>>
>>>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jghoman@gmail.com> wrote:
>>>>>> If HDFS is added to the test-patch queue right now we get
>>>>>> nothing but dozens of -1'ed patches.
>>>>> There aren't dozens of patches being submitted currently. =A0The -1
>>>>> isn't the important thing, it's the grunt work of actually running
>>>>> (and waiting) for the tests, test-patch, etc. that Hudson does so tha=
t
>>>>> the developer doesn't have to.
>>>>>
>>>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <dhruba@gmail.com> =
wrote:
>>>>>> +1, thanks for doing this.
>>>>>>
>>>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jghoman@gmail.com> wro=
te:
>>>>>>
>>>>>>> So, with test-patch updated to show the failing tests, saving the
>>>>>>> developers the need to go and verify that the failed tests are all
>>>>>>> known, how do people feel about turning on test-patch again for HDF=
S
>>>>>>> and mapred? =A0I think it'll help prevent any more tests from enter=
ing
>>>>>>> the "yeah, we know" category.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> jg
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <jhoman@yahoo-inc.com>=
 wrote:
>>>>>>>> True, each patch would get a -1 and the failing tests would need t=
o be
>>>>>>>> verified as those known bad (BTW, it would be great if Hudson coul=
d list
>>>>>>>> which tests failed in the message it posts to JIRA). =A0But that's=
 still
>>>>>>> quite
>>>>>>>> a bit less error-prone work than if the developer runs the tests a=
nd
>>>>>>>> test-patch themselves. =A0Also, with 22 being cut, there are a lot=
 of
>>>>>>> patches
>>>>>>>> up in the air and several developers are juggling multiple patches=
. =A0The
>>>>>>>> more automation we can have, even if it's not perfect, will decrea=
se
>>>>>>> errors
>>>>>>>> we may make.
>>>>>>>> -jg
>>>>>>>>
>>>>>>>> Nigel Daley wrote:
>>>>>>>>>
>>>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>>>>>>>>
>>>>>>>>>>> It's also ready to run on MapReduce and HDFS but we won't turn =
it on
>>>>>>>>>>> until these projects build and test cleanly. =A0Looks like both=
 these
>>>>>>> projects
>>>>>>>>>>> currently have test failures.
>>>>>>>>>>
>>>>>>>>>> Assuming the projects are compiling and building, is there a rea=
son to
>>>>>>>>>> not turn it on despite the test failures? Hudson is invaluable t=
o
>>>>>>> developers
>>>>>>>>>> who then don't have to run the tests and test-patch themselves. =
=A0We
>>>>>>> didn't
>>>>>>>>>> turn Hudson off when it was working previously and there were kn=
own
>>>>>>>>>> failures. =A0I think one of the reasons we have more failing tes=
ts now is
>>>>>>> the
>>>>>>>>>> higher cost of doing Hudson's work (not a great excuse I know). =
=A0This
>>>>>>> is
>>>>>>>>>> particularly true now because several of the failing tests invol=
ve
>>>>>>> tests
>>>>>>>>>> timing out, making the whole testing regime even longer.
>>>>>>>>>
>>>>>>>>> Every single patch would get a -1 and need investigation. =A0Curr=
ently,
>>>>>>> that
>>>>>>>>> would be about 83 investigations between MR and HDFS issues that =
are in
>>>>>>>>> patch available state. =A0Shouldn't we focus on getting these tes=
ts fixed
>>>>>>> or
>>>>>>>>> removed/? =A0Also, I need to get MAPREDUCE-2172 fixed (applies to=
 HDFS as
>>>>>>>>> well) before I turn this on.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Nige
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Connect to me at http://www.facebook.com/dhruba
>>>>>>
>>>>>
>>>>
>>
>

From general-return-2560-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Dec 20 07:28:04 2010
Return-Path: <general-return-2560-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23958 invoked from network); 20 Dec 2010 07:28:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Dec 2010 07:28:04 -0000
Received: (qmail 28257 invoked by uid 500); 20 Dec 2010 07:28:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27919 invoked by uid 500); 20 Dec 2010 07:28:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27910 invoked by uid 99); 20 Dec 2010 07:28:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Dec 2010 07:28:00 +0000
X-ASF-Spam-Status: No, hits=-1997.8 required=10.0
	tests=ALL_TRUSTED,HTML_MESSAGE,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 20 Dec 2010 07:27:57 +0000
Received: (qmail 23915 invoked by uid 99); 20 Dec 2010 07:27:36 -0000
Received: from localhost.apache.org (HELO mail-bw0-f43.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Dec 2010 07:27:36 +0000
Received: by bwz14 with SMTP id 14so3656655bwz.2
        for <general@hadoop.apache.org>; Sun, 19 Dec 2010 23:27:33 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.122.17 with SMTP id j17mr677635bkr.4.1292830053842; Sun,
 19 Dec 2010 23:27:33 -0800 (PST)
Received: by 10.204.76.14 with HTTP; Sun, 19 Dec 2010 23:27:32 -0800 (PST)
In-Reply-To: <4D0BE994.4020207@apache.org>
References: <27F4E872-C226-4386-9148-077F0044B6E9@Holsman.net>
	<05ED6FF7-6236-47F4-B5DD-8E927971C72D@apache.org>
	<39694716-7EF6-41B8-8862-D2DE03A4560C@Holsman.NET>
	<9FC0DEC0-524A-4A35-88A6-17473DD74929@apache.org>
	<C9F82B37-7397-46D8-9788-679380749C1B@Holsman.NET>
	<AANLkTin9ZdZAVZSvE6yeSWHJKtfON0YiM5Gr55oLD62P@mail.gmail.com>
	<4D0BE994.4020207@apache.org>
Date: Sun, 19 Dec 2010 23:27:32 -0800
Message-ID: <AANLkTin1783GhAchiT4reLbcC20ZHvC+zNts1JiBsuA_@mail.gmail.com>
Subject: Re: Plans for the 0.22 Release
From: "Owen O'Malley" <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6de04bf1d3d990497d27411
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6de04bf1d3d990497d27411
Content-Type: text/plain; charset=UTF-8

On Fri, Dec 17, 2010 at 2:52 PM, Doug Cutting <cutting@apache.org> wrote:

> On 12/17/2010 02:34 PM, Konstantin Shvachko wrote:
>
>> It could be a zero-option plan - remove dependencies both for Avro and
>> ProtocolBuffers out into libraries, similar to schedulers.
>>
>
> I'd be fine with removing Avro from the mapreduce user's classpath. It's
> currently an unused option for RPC, and is used server-side on the
> jobtracker.  It could be removed from the user classpath today without loss
> of critical functionality.


That would be non-trivial and no one has requested that. You haven't
answered Konstantin's question.

What do you consider the technical reasons for rejecting the patch as is?

-- Owen

--0016e6de04bf1d3d990497d27411--

From general-return-2561-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Dec 20 19:39:56 2010
Return-Path: <general-return-2561-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73891 invoked from network); 20 Dec 2010 19:39:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Dec 2010 19:39:56 -0000
Received: (qmail 90824 invoked by uid 500); 20 Dec 2010 19:39:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90768 invoked by uid 500); 20 Dec 2010 19:39:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90760 invoked by uid 99); 20 Dec 2010 19:39:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Dec 2010 19:39:53 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of shv.hadoop@gmail.com designates 209.85.213.48 as permitted sender)
Received: from [209.85.213.48] (HELO mail-yw0-f48.google.com) (209.85.213.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Dec 2010 19:39:45 +0000
Received: by ywo7 with SMTP id 7so1736189ywo.35
        for <general@hadoop.apache.org>; Mon, 20 Dec 2010 11:39:25 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=5Q2U1tLrrwvqUC7+RZpRl1cY0F0DidZUeHuRXth+Ors=;
        b=HYfcYpA/c9IrTsm61kGFbFOEqPOk+A36wG3KlzITbl2SXbymnpepvpF4oqXcGdLoRj
         tVP8AZlZqtQwkhICH2yT6yeXF4t179oFZw53LF12UlXJ+HgrFbwNlpo8RTfK6qJLViy1
         4anRvJv0dRvTHLqp6xJO1BpjVandNSIdNWofg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=GJxha1ig7JdMUy/ig7skAhJmopv+JeAmy4WrKUsEnRx1tho5lsv+A0wDGOBWFJ6Ohj
         U0nAa3arhGCIc9LSzcoASAFULyT4J8fv8ErVPa4mGPRxZucR495neqmsdI8IHXlPck+W
         gGwZV63b8PnS8sBnbtNoEtYsxG2hlhYpWbXzs=
MIME-Version: 1.0
Received: by 10.236.95.144 with SMTP id p16mr8444846yhf.11.1292873964975; Mon,
 20 Dec 2010 11:39:24 -0800 (PST)
Received: by 10.236.109.15 with HTTP; Mon, 20 Dec 2010 11:39:24 -0800 (PST)
In-Reply-To: <AANLkTin1783GhAchiT4reLbcC20ZHvC+zNts1JiBsuA_@mail.gmail.com>
References: <27F4E872-C226-4386-9148-077F0044B6E9@Holsman.net>
	<05ED6FF7-6236-47F4-B5DD-8E927971C72D@apache.org>
	<39694716-7EF6-41B8-8862-D2DE03A4560C@Holsman.NET>
	<9FC0DEC0-524A-4A35-88A6-17473DD74929@apache.org>
	<C9F82B37-7397-46D8-9788-679380749C1B@Holsman.NET>
	<AANLkTin9ZdZAVZSvE6yeSWHJKtfON0YiM5Gr55oLD62P@mail.gmail.com>
	<4D0BE994.4020207@apache.org>
	<AANLkTin1783GhAchiT4reLbcC20ZHvC+zNts1JiBsuA_@mail.gmail.com>
Date: Mon, 20 Dec 2010 11:39:24 -0800
Message-ID: <AANLkTinj8XNy_xQVHc6SYwc8ZUOJ0hEbNUPrHb-7oVHT@mail.gmail.com>
Subject: Re: Plans for the 0.22 Release
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=002354435d106bf7200497dcad80
X-Virus-Checked: Checked by ClamAV on apache.org

--002354435d106bf7200497dcad80
Content-Type: text/plain; charset=ISO-8859-1

My question is
    What would be an *ACCEPTABLE *resolution of HADOOP-6685 for *YOU *to
move *0.22* forward?
Asking Owen as the author of the patch, Doug and Tom as they vetoed it.

If I am asking the wrong question please let me know.
I am not proposing a particular solution, just trying to understand
1. if there is any
2. What is it if there is.

If, say removing dependency for Avro and updating the patch symmetrically
removing PB dependency is a solution, if there is a solution to SequenceFile
and byte array issues, then lets do it.
If not then lets face it.

Thanks,
--Konstantin

On Sun, Dec 19, 2010 at 11:27 PM, Owen O'Malley <omalley@apache.org> wrote:

> On Fri, Dec 17, 2010 at 2:52 PM, Doug Cutting <cutting@apache.org> wrote:
>
> > On 12/17/2010 02:34 PM, Konstantin Shvachko wrote:
> >
> >> It could be a zero-option plan - remove dependencies both for Avro and
> >> ProtocolBuffers out into libraries, similar to schedulers.
> >>
> >
> > I'd be fine with removing Avro from the mapreduce user's classpath. It's
> > currently an unused option for RPC, and is used server-side on the
> > jobtracker.  It could be removed from the user classpath today without
> loss
> > of critical functionality.
>
>
> That would be non-trivial and no one has requested that. You haven't
> answered Konstantin's question.
>
> What do you consider the technical reasons for rejecting the patch as is?
>
> -- Owen
>

--002354435d106bf7200497dcad80--

From general-return-2562-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Dec 20 20:13:03 2010
Return-Path: <general-return-2562-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 90037 invoked from network); 20 Dec 2010 20:13:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Dec 2010 20:13:03 -0000
Received: (qmail 32343 invoked by uid 500); 20 Dec 2010 20:13:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32292 invoked by uid 500); 20 Dec 2010 20:13:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32284 invoked by uid 99); 20 Dec 2010 20:13:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Dec 2010 20:13:01 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of tom.e.white@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Dec 2010 20:12:55 +0000
Received: by wye20 with SMTP id 20so3462533wye.35
        for <general@hadoop.apache.org>; Mon, 20 Dec 2010 12:12:34 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=S4T+PcX1wz8RYTN7rKTZFRvVBA8hUasbD1WBt0B6SJE=;
        b=YTkSXlW+mXDIuppAtW5ndymAKml474aB9KXSQ4st+h5eUtrxyHSFr+lH+fRuPknvsC
         wweIEbExZaD4Ns3KiDW/KQbotA1+sxnP/C8+X4KpZPJ73c9Vk4XbuKtr/1tyO5RSfO5s
         JLZf5mGjv1hgzNUD2hSKvxX08MO7vInZQ2XJw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=lLMXRVBV+q8fs2uM4HzLBSg3Qvj4bg2chf8HOk58pGk13CGxfAyY8ZnZeGLMlqQP1o
         5JiycRdyKImfMX2lqTFGDYJJx0DVOjex3yE0NMeqj/QKC6Yo4J/gYhdweb9iayYQj5+D
         8ZdYn0HU2kn0eyR+uFxTzd5nmexDBsDxmRBes=
Received: by 10.216.160.129 with SMTP id u1mr7778133wek.88.1292875954039; Mon,
 20 Dec 2010 12:12:34 -0800 (PST)
MIME-Version: 1.0
Received: by 10.216.19.193 with HTTP; Mon, 20 Dec 2010 12:12:13 -0800 (PST)
In-Reply-To: <AANLkTinj8XNy_xQVHc6SYwc8ZUOJ0hEbNUPrHb-7oVHT@mail.gmail.com>
References: <27F4E872-C226-4386-9148-077F0044B6E9@Holsman.net>
 <05ED6FF7-6236-47F4-B5DD-8E927971C72D@apache.org> <39694716-7EF6-41B8-8862-D2DE03A4560C@Holsman.NET>
 <9FC0DEC0-524A-4A35-88A6-17473DD74929@apache.org> <C9F82B37-7397-46D8-9788-679380749C1B@Holsman.NET>
 <AANLkTin9ZdZAVZSvE6yeSWHJKtfON0YiM5Gr55oLD62P@mail.gmail.com>
 <4D0BE994.4020207@apache.org> <AANLkTin1783GhAchiT4reLbcC20ZHvC+zNts1JiBsuA_@mail.gmail.com>
 <AANLkTinj8XNy_xQVHc6SYwc8ZUOJ0hEbNUPrHb-7oVHT@mail.gmail.com>
From: Tom White <tom.e.white@gmail.com>
Date: Mon, 20 Dec 2010 12:12:13 -0800
Message-ID: <AANLkTi=hArW-UNO4dwN0pqtSfkYpr-yTBNj_Zn4pJrFT@mail.gmail.com>
Subject: Re: Plans for the 0.22 Release
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

[Sending this again as my original post didn't make it to the list for
some reason. Can't see it in the moderation queue either.]

I don't personally see HADOOP-6685 as a blocker for a 0.22 release,
since there is a lot of value in there already that has not been
released yet, such as security. However, to get HADOOP-6685 resolved,
from my point of view the main thing to sort out are the modularity
concerns that I and others have raised, so that serializations are
pluggable and don't add potentially incompatible libraries onto the
user's classpath.

Thanks,
Tom

On Mon, Dec 20, 2010 at 11:39 AM, Konstantin Shvachko
<shv.hadoop@gmail.com> wrote:
> My question is
> =A0 =A0What would be an *ACCEPTABLE *resolution of HADOOP-6685 for *YOU *=
to
> move *0.22* forward?
> Asking Owen as the author of the patch, Doug and Tom as they vetoed it.
>
> If I am asking the wrong question please let me know.
> I am not proposing a particular solution, just trying to understand
> 1. if there is any
> 2. What is it if there is.
>
> If, say removing dependency for Avro and updating the patch symmetrically
> removing PB dependency is a solution, if there is a solution to SequenceF=
ile
> and byte array issues, then lets do it.
> If not then lets face it.
>
> Thanks,
> --Konstantin
>
> On Sun, Dec 19, 2010 at 11:27 PM, Owen O'Malley <omalley@apache.org> wrot=
e:
>
>> On Fri, Dec 17, 2010 at 2:52 PM, Doug Cutting <cutting@apache.org> wrote=
:
>>
>> > On 12/17/2010 02:34 PM, Konstantin Shvachko wrote:
>> >
>> >> It could be a zero-option plan - remove dependencies both for Avro an=
d
>> >> ProtocolBuffers out into libraries, similar to schedulers.
>> >>
>> >
>> > I'd be fine with removing Avro from the mapreduce user's classpath. It=
's
>> > currently an unused option for RPC, and is used server-side on the
>> > jobtracker. =A0It could be removed from the user classpath today witho=
ut
>> loss
>> > of critical functionality.
>>
>>
>> That would be non-trivial and no one has requested that. You haven't
>> answered Konstantin's question.
>>
>> What do you consider the technical reasons for rejecting the patch as is=
?
>>
>> -- Owen
>>
>

From general-return-2563-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Dec 20 20:43:53 2010
Return-Path: <general-return-2563-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97611 invoked from network); 20 Dec 2010 20:43:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Dec 2010 20:43:52 -0000
Received: (qmail 70283 invoked by uid 500); 20 Dec 2010 20:43:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70201 invoked by uid 500); 20 Dec 2010 20:43:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70193 invoked by uid 99); 20 Dec 2010 20:43:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Dec 2010 20:43:50 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jghoman@gmail.com designates 209.85.210.176 as permitted sender)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Dec 2010 20:43:46 +0000
Received: by iyb26 with SMTP id 26so3008705iyb.35
        for <general@hadoop.apache.org>; Mon, 20 Dec 2010 12:43:25 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=6yd4d8yjDAgk9q8YuTAo61yBFjjJRrSpAZ83W9hquNA=;
        b=Mx5Ew0PFoSshdaYbeKdSO10eFX1vyi4SmLwEeekQEgQg8QyJFeqEppJe8p7WNAPiFP
         zkQhdhCpj8BYhWvqdQrYxSOVjOEyuHMPGZwyNSYyADHbt29gVwCnRrZ58bCVWHQ9MIsL
         3IsAhHQRtNixBLry+Z44rlKqJgUpOVyFK/i/0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=lZhfvJsxhTbFd07dVDCk1sNiWRjXo6oc9VG64j2Y1GzSTUdIM8PLRT4TkB3C6qqDge
         wM7ZXSrmF7Z7SLs6Jxfcy/ewztqOiVNs5FkpyCftflsn8d12mT11swbbD+tYOPLBtVsI
         LQL4hVZQqKZOAKcJ1Q2411THKs8O5LZT2/DKo=
Received: by 10.231.30.73 with SMTP id t9mr4544027ibc.144.1292877805801; Mon,
 20 Dec 2010 12:43:25 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.25.163 with HTTP; Mon, 20 Dec 2010 12:42:41 -0800 (PST)
In-Reply-To: <AANLkTin9=FPxv+0EydK4mWp_1zJbgGTddevvF9MrHW80@mail.gmail.com>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com> <20101020195420.GG2075@tp>
 <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com> <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
 <7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com> <AANLkTimLXA+CvUs-09LMN3ob6GZqmhxiyrKO7pzWEfse@mail.gmail.com>
 <4CE46119.2030509@yahoo-inc.com> <8617BECB-78B4-42B6-B592-D7FC1F8DA923@mac.com>
 <4CE47C88.5050203@yahoo-inc.com> <AANLkTinTM5h-3NupRSsFh-DnyvFYw+_OsXpASxcQUSDn@mail.gmail.com>
 <AANLkTimSOxaXZi-VLjwuTB5Ch5tRt0bKozc5L+Nx=ZrQ@mail.gmail.com>
 <AANLkTineU2v=Sx-=oC=6ROt6Os8TEbnS3L8xz1ZkoxCc@mail.gmail.com>
 <AANLkTin+oucXdCdOo7LQTtqndTf1axiT0gQntiOZ_mQJ@mail.gmail.com>
 <AANLkTinGn4Pb6_LkrY4-kK56PdHX4C1Zfu7A2PVfUoub@mail.gmail.com>
 <41FB0800-3703-49C1-8069-DEB74FFE6FAC@mac.com> <AANLkTik_rNLMHj3JT6Ekdop-NU_qzcB=VdsX+k4uFBtf@mail.gmail.com>
 <AANLkTin9=FPxv+0EydK4mWp_1zJbgGTddevvF9MrHW80@mail.gmail.com>
From: Jakob Homan <jghoman@gmail.com>
Date: Mon, 20 Dec 2010 12:42:41 -0800
Message-ID: <AANLkTiku_OvZuuOMWuXMcVf4UxB+W-SfW1WWV6rU0cXB@mail.gmail.com>
Subject: Re: Patch testing
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I committed HDFS-1511 this morning.  We should be good to go.  I can
haz snooty robot butler?

On Fri, Dec 17, 2010 at 8:31 PM, Konstantin Boudnik <cos@apache.org> wrote:
> Thanks Jacob. I am wasted already but I can do it on Sun, I think,
> unless it is done earlier.
> --
> =A0 Take care,
> Konstantin (Cos) Boudnik
>
>
>
> On Fri, Dec 17, 2010 at 19:41, Jakob Homan <jghoman@gmail.com> wrote:
>> Ok. =A0I'll get a patch out for 1511 tomorrow, unless someone wants to
>> whip one up tonight.
>>
>>
>> On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley <ndaley@mac.com> wrote:
>>> I agree with Cos on fixing HDFS-1511 first. Once that is done I'll enab=
le hdfs patch testing.
>>>
>>> Cheers,
>>> Nige
>>>
>>> Sent from my iPhone4
>>>
>>> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <cos@apache.org> wrote:
>>>
>>>> One more issue needs to be addressed before test-patch is turned on HD=
FS is
>>>> =A0https://issues.apache.org/jira/browse/HDFS-1511
>>>> --
>>>> =A0 Take care,
>>>> Konstantin (Cos) Boudnik
>>>>
>>>>
>>>>
>>>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <cos@apache.org> wro=
te:
>>>>> Considering that because of these 4 faulty cases every patch will be
>>>>> -1'ed a patch author will still have to look at it and make a comment
>>>>> why this particular -1 isn't valid. Lesser work, perhaps, but messier
>>>>> IMO. I'm not blocking it - I just feel like there's a better way.
>>>>>
>>>>> --
>>>>> =A0 Take care,
>>>>> Konstantin (Cos) Boudnik
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jghoman@gmail.com> wrote:
>>>>>>> If HDFS is added to the test-patch queue right now we get
>>>>>>> nothing but dozens of -1'ed patches.
>>>>>> There aren't dozens of patches being submitted currently. =A0The -1
>>>>>> isn't the important thing, it's the grunt work of actually running
>>>>>> (and waiting) for the tests, test-patch, etc. that Hudson does so th=
at
>>>>>> the developer doesn't have to.
>>>>>>
>>>>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <dhruba@gmail.com>=
 wrote:
>>>>>>> +1, thanks for doing this.
>>>>>>>
>>>>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jghoman@gmail.com> wr=
ote:
>>>>>>>
>>>>>>>> So, with test-patch updated to show the failing tests, saving the
>>>>>>>> developers the need to go and verify that the failed tests are all
>>>>>>>> known, how do people feel about turning on test-patch again for HD=
FS
>>>>>>>> and mapred? =A0I think it'll help prevent any more tests from ente=
ring
>>>>>>>> the "yeah, we know" category.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> jg
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <jhoman@yahoo-inc.com=
> wrote:
>>>>>>>>> True, each patch would get a -1 and the failing tests would need =
to be
>>>>>>>>> verified as those known bad (BTW, it would be great if Hudson cou=
ld list
>>>>>>>>> which tests failed in the message it posts to JIRA). =A0But that'=
s still
>>>>>>>> quite
>>>>>>>>> a bit less error-prone work than if the developer runs the tests =
and
>>>>>>>>> test-patch themselves. =A0Also, with 22 being cut, there are a lo=
t of
>>>>>>>> patches
>>>>>>>>> up in the air and several developers are juggling multiple patche=
s. =A0The
>>>>>>>>> more automation we can have, even if it's not perfect, will decre=
ase
>>>>>>>> errors
>>>>>>>>> we may make.
>>>>>>>>> -jg
>>>>>>>>>
>>>>>>>>> Nigel Daley wrote:
>>>>>>>>>>
>>>>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>>>>>>>>>
>>>>>>>>>>>> It's also ready to run on MapReduce and HDFS but we won't turn=
 it on
>>>>>>>>>>>> until these projects build and test cleanly. =A0Looks like bot=
h these
>>>>>>>> projects
>>>>>>>>>>>> currently have test failures.
>>>>>>>>>>>
>>>>>>>>>>> Assuming the projects are compiling and building, is there a re=
ason to
>>>>>>>>>>> not turn it on despite the test failures? Hudson is invaluable =
to
>>>>>>>> developers
>>>>>>>>>>> who then don't have to run the tests and test-patch themselves.=
 =A0We
>>>>>>>> didn't
>>>>>>>>>>> turn Hudson off when it was working previously and there were k=
nown
>>>>>>>>>>> failures. =A0I think one of the reasons we have more failing te=
sts now is
>>>>>>>> the
>>>>>>>>>>> higher cost of doing Hudson's work (not a great excuse I know).=
 =A0This
>>>>>>>> is
>>>>>>>>>>> particularly true now because several of the failing tests invo=
lve
>>>>>>>> tests
>>>>>>>>>>> timing out, making the whole testing regime even longer.
>>>>>>>>>>
>>>>>>>>>> Every single patch would get a -1 and need investigation. =A0Cur=
rently,
>>>>>>>> that
>>>>>>>>>> would be about 83 investigations between MR and HDFS issues that=
 are in
>>>>>>>>>> patch available state. =A0Shouldn't we focus on getting these te=
sts fixed
>>>>>>>> or
>>>>>>>>>> removed/? =A0Also, I need to get MAPREDUCE-2172 fixed (applies t=
o HDFS as
>>>>>>>>>> well) before I turn this on.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Nige
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Connect to me at http://www.facebook.com/dhruba
>>>>>>>
>>>>>>
>>>>>
>>>
>>
>

From general-return-2564-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 21 19:25:50 2010
Return-Path: <general-return-2564-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42092 invoked from network); 21 Dec 2010 19:25:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Dec 2010 19:25:50 -0000
Received: (qmail 66242 invoked by uid 500); 21 Dec 2010 19:25:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66177 invoked by uid 500); 21 Dec 2010 19:25:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66169 invoked by uid 99); 21 Dec 2010 19:25:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Dec 2010 19:25:47 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.99 as permitted sender)
Received: from [17.148.16.99] (HELO asmtpout024.mac.com) (17.148.16.99)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Dec 2010 19:25:38 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp024.mac.com (Oracle Communications Messaging Exchange Server 7u4-18.01
 64bit (built Jul 15 2010)) with ESMTPSA id <0LDS00C8YLXL13B0@asmtp024.mac.com>
 for general@hadoop.apache.org; Tue, 21 Dec 2010 11:24:58 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2010-12-21_07:2010-12-21,2010-12-21,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1012210096
Subject: Re: Plans for the 0.22 Release
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <C9F82B37-7397-46D8-9788-679380749C1B@Holsman.NET>
Date: Tue, 21 Dec 2010 11:24:57 -0800
Message-id: <2DE2238D-DDFF-4CE3-B50C-39D617AA13E8@mac.com>
References: <27F4E872-C226-4386-9148-077F0044B6E9@Holsman.net>
 <05ED6FF7-6236-47F4-B5DD-8E927971C72D@apache.org>
 <39694716-7EF6-41B8-8862-D2DE03A4560C@Holsman.NET>
 <9FC0DEC0-524A-4A35-88A6-17473DD74929@apache.org>
 <C9F82B37-7397-46D8-9788-679380749C1B@Holsman.NET>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

I think it's important we get more frequent releases back on track, so I'll volunteer to do it.  Its been a while since I was the release manager so I'll need help.

Looks like in Jira we currently have:
  26 MR blockers: http://tinyurl.com/23s5vzg
  17 HDFS blockers: http://tinyurl.com/27w82vl
  6 HADOOP blockers: http://tinyurl.com/2bmycl2
Now that we are past feature freeze (i.e. branch creation), not all of these are truly blockers.  I'll start cleaning up the list and ping folks after the holidays.

Cheers,
Nige

On Dec 16, 2010, at 3:10 PM, Ian Holsman wrote:

> 
> On Dec 17, 2010, at 9:00 AM, Owen O'Malley wrote:
> 
>>> Everyone who has discussed this patch has said it isn't critical to hadoop, and It's holding up everything else 0.22 is going to bring.
>> 
>> I disagree that it isn't critical to Hadoop, but I'm not holding up 0.22. I'm just not volunteering to spend my time working on it, if it doesn't have the features that I think it needs.
>> 
>> -- Owen
>> 
> 
> That's a fair point.. and this is a volunteer effort.
> Do we have anybody else who is willing to be the release manager for 22 with 6685?
> 


From general-return-2565-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 21 21:08:50 2010
Return-Path: <general-return-2565-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 2565 invoked from network); 21 Dec 2010 21:08:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Dec 2010 21:08:50 -0000
Received: (qmail 20889 invoked by uid 500); 21 Dec 2010 21:08:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20832 invoked by uid 500); 21 Dec 2010 21:08:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 20824 invoked by uid 99); 21 Dec 2010 21:08:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Dec 2010 21:08:48 +0000
X-ASF-Spam-Status: No, hits=-1997.8 required=10.0
	tests=ALL_TRUSTED,HTML_MESSAGE,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 21 Dec 2010 21:08:46 +0000
Received: (qmail 2521 invoked by uid 99); 21 Dec 2010 21:08:24 -0000
Received: from localhost.apache.org (HELO mail-bw0-f43.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Dec 2010 21:08:24 +0000
Received: by bwz14 with SMTP id 14so6041360bwz.2
        for <general@hadoop.apache.org>; Tue, 21 Dec 2010 13:08:21 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.71.20 with SMTP id f20mr5169954bkj.139.1292965701623; Tue,
 21 Dec 2010 13:08:21 -0800 (PST)
Received: by 10.204.76.14 with HTTP; Tue, 21 Dec 2010 13:08:20 -0800 (PST)
In-Reply-To: <AANLkTi=hArW-UNO4dwN0pqtSfkYpr-yTBNj_Zn4pJrFT@mail.gmail.com>
References: <27F4E872-C226-4386-9148-077F0044B6E9@Holsman.net>
	<05ED6FF7-6236-47F4-B5DD-8E927971C72D@apache.org>
	<39694716-7EF6-41B8-8862-D2DE03A4560C@Holsman.NET>
	<9FC0DEC0-524A-4A35-88A6-17473DD74929@apache.org>
	<C9F82B37-7397-46D8-9788-679380749C1B@Holsman.NET>
	<AANLkTin9ZdZAVZSvE6yeSWHJKtfON0YiM5Gr55oLD62P@mail.gmail.com>
	<4D0BE994.4020207@apache.org>
	<AANLkTin1783GhAchiT4reLbcC20ZHvC+zNts1JiBsuA_@mail.gmail.com>
	<AANLkTinj8XNy_xQVHc6SYwc8ZUOJ0hEbNUPrHb-7oVHT@mail.gmail.com>
	<AANLkTi=hArW-UNO4dwN0pqtSfkYpr-yTBNj_Zn4pJrFT@mail.gmail.com>
Date: Tue, 21 Dec 2010 13:08:20 -0800
Message-ID: <AANLkTik-6M_6Q4AC3ZtfnjU2V55S8Yq9bcHuCuQJ37JW@mail.gmail.com>
Subject: Re: Plans for the 0.22 Release
From: "Owen O'Malley" <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6de04215a1ea20497f20936
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6de04215a1ea20497f20936
Content-Type: text/plain; charset=UTF-8

On Mon, Dec 20, 2010 at 12:12 PM, Tom White <tom.e.white@gmail.com> wrote:

> However, to get HADOOP-6685 resolved,
> from my point of view the main thing to sort out are the modularity
> concerns that I and others have raised, so that serializations are
> pluggable and don't add potentially incompatible libraries onto the
> user's classpath.
>

The serializers are already completely pluggable. If you want to propose a
plan for reducing the number of jars on the user's task classpath that might
be interesting. Clearly, it would be a separate jira, since it has nothing
to do with the serialization interface.

-- Owen

--0016e6de04215a1ea20497f20936--

From general-return-2566-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 21 21:18:56 2010
Return-Path: <general-return-2566-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 7150 invoked from network); 21 Dec 2010 21:18:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Dec 2010 21:18:56 -0000
Received: (qmail 33548 invoked by uid 500); 21 Dec 2010 21:18:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33479 invoked by uid 500); 21 Dec 2010 21:18:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33471 invoked by uid 99); 21 Dec 2010 21:18:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Dec 2010 21:18:54 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Dec 2010 21:18:47 +0000
Received: by vws18 with SMTP id 18so1843009vws.35
        for <general@hadoop.apache.org>; Tue, 21 Dec 2010 13:18:26 -0800 (PST)
Received: by 10.220.171.9 with SMTP id f9mr1542653vcz.275.1292966305976;
        Tue, 21 Dec 2010 13:18:25 -0800 (PST)
Received: from [10.172.0.49] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id y4sm1171678vch.35.2010.12.21.13.18.23
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Tue, 21 Dec 2010 13:18:25 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Plans for the 0.22 Release
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <2DE2238D-DDFF-4CE3-B50C-39D617AA13E8@mac.com>
Date: Wed, 22 Dec 2010 08:18:19 +1100
Content-Transfer-Encoding: quoted-printable
Message-Id: <2262BB4D-4D8B-4122-A498-493306860AAF@Holsman.NET>
References: <27F4E872-C226-4386-9148-077F0044B6E9@Holsman.net> <05ED6FF7-6236-47F4-B5DD-8E927971C72D@apache.org> <39694716-7EF6-41B8-8862-D2DE03A4560C@Holsman.NET> <9FC0DEC0-524A-4A35-88A6-17473DD74929@apache.org> <C9F82B37-7397-46D8-9788-679380749C1B@Holsman.NET> <2DE2238D-DDFF-4CE3-B50C-39D617AA13E8@mac.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

Thanks Nige.

On Dec 22, 2010, at 6:24 AM, Nigel Daley wrote:

> I think it's important we get more frequent releases back on track, so =
I'll volunteer to do it.  Its been a while since I was the release =
manager so I'll need help.
>=20
> Looks like in Jira we currently have:
>  26 MR blockers: http://tinyurl.com/23s5vzg
>  17 HDFS blockers: http://tinyurl.com/27w82vl
>  6 HADOOP blockers: http://tinyurl.com/2bmycl2
> Now that we are past feature freeze (i.e. branch creation), not all of =
these are truly blockers.  I'll start cleaning up the list and ping =
folks after the holidays.
>=20
> Cheers,
> Nige
>=20
> On Dec 16, 2010, at 3:10 PM, Ian Holsman wrote:
>=20
>>=20
>> On Dec 17, 2010, at 9:00 AM, Owen O'Malley wrote:
>>=20
>>>> Everyone who has discussed this patch has said it isn't critical to =
hadoop, and It's holding up everything else 0.22 is going to bring.
>>>=20
>>> I disagree that it isn't critical to Hadoop, but I'm not holding up =
0.22. I'm just not volunteering to spend my time working on it, if it =
doesn't have the features that I think it needs.
>>>=20
>>> -- Owen
>>>=20
>>=20
>> That's a fair point.. and this is a volunteer effort.
>> Do we have anybody else who is willing to be the release manager for =
22 with 6685?
>>=20
>=20


From general-return-2567-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 21 21:34:33 2010
Return-Path: <general-return-2567-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 13046 invoked from network); 21 Dec 2010 21:34:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Dec 2010 21:34:33 -0000
Received: (qmail 68200 invoked by uid 500); 21 Dec 2010 21:34:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68091 invoked by uid 500); 21 Dec 2010 21:34:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68083 invoked by uid 99); 21 Dec 2010 21:34:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Dec 2010 21:34:31 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.105 as permitted sender)
Received: from [17.148.16.105] (HELO asmtpout030.mac.com) (17.148.16.105)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Dec 2010 21:34:24 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; charset=us-ascii
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp030.mac.com (Sun Java(tm) System Messaging Server 6.3-8.01 (built Dec
 16 2008; 32bit)) with ESMTPSA id <0LDS00ECLRW2VF70@asmtp030.mac.com> for
 general@hadoop.apache.org; Tue, 21 Dec 2010 13:33:39 -0800 (PST)
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1012210124
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2010-12-21_07:2010-12-21,2010-12-21,1970-01-01 signatures=0
Subject: Re: Patch testing
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <AANLkTiku_OvZuuOMWuXMcVf4UxB+W-SfW1WWV6rU0cXB@mail.gmail.com>
Date: Tue, 21 Dec 2010 13:33:37 -0800
Message-id: <7B140F50-27E0-47C7-8EF8-B897D26CEE49@mac.com>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com>
 <20101020195420.GG2075@tp>
 <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com>
 <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
 <7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com>
 <AANLkTimLXA+CvUs-09LMN3ob6GZqmhxiyrKO7pzWEfse@mail.gmail.com>
 <4CE46119.2030509@yahoo-inc.com>
 <8617BECB-78B4-42B6-B592-D7FC1F8DA923@mac.com>
 <4CE47C88.5050203@yahoo-inc.com>
 <AANLkTinTM5h-3NupRSsFh-DnyvFYw+_OsXpASxcQUSDn@mail.gmail.com>
 <AANLkTimSOxaXZi-VLjwuTB5Ch5tRt0bKozc5L+Nx=ZrQ@mail.gmail.com>
 <AANLkTineU2v=Sx-=oC=6ROt6Os8TEbnS3L8xz1ZkoxCc@mail.gmail.com>
 <AANLkTin+oucXdCdOo7LQTtqndTf1axiT0gQntiOZ_mQJ@mail.gmail.com>
 <AANLkTinGn4Pb6_LkrY4-kK56PdHX4C1Zfu7A2PVfUoub@mail.gmail.com>
 <41FB0800-3703-49C1-8069-DEB74FFE6FAC@mac.com>
 <AANLkTik_rNLMHj3JT6Ekdop-NU_qzcB=VdsX+k4uFBtf@mail.gmail.com>
 <AANLkTin9=FPxv+0EydK4mWp_1zJbgGTddevvF9MrHW80@mail.gmail.com>
 <AANLkTiku_OvZuuOMWuXMcVf4UxB+W-SfW1WWV6rU0cXB@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

Ok, HDFS is now enabled.  You'll see a stream of updates shortly on the ~30 Patch Available HDFS issues.

Nige

On Dec 20, 2010, at 12:42 PM, Jakob Homan wrote:

> I committed HDFS-1511 this morning.  We should be good to go.  I can
> haz snooty robot butler?
> 
> On Fri, Dec 17, 2010 at 8:31 PM, Konstantin Boudnik <cos@apache.org> wrote:
>> Thanks Jacob. I am wasted already but I can do it on Sun, I think,
>> unless it is done earlier.
>> --
>>   Take care,
>> Konstantin (Cos) Boudnik
>> 
>> 
>> 
>> On Fri, Dec 17, 2010 at 19:41, Jakob Homan <jghoman@gmail.com> wrote:
>>> Ok.  I'll get a patch out for 1511 tomorrow, unless someone wants to
>>> whip one up tonight.
>>> 
>>> 
>>> On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley <ndaley@mac.com> wrote:
>>>> I agree with Cos on fixing HDFS-1511 first. Once that is done I'll enable hdfs patch testing.
>>>> 
>>>> Cheers,
>>>> Nige
>>>> 
>>>> Sent from my iPhone4
>>>> 
>>>> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <cos@apache.org> wrote:
>>>> 
>>>>> One more issue needs to be addressed before test-patch is turned on HDFS is
>>>>>  https://issues.apache.org/jira/browse/HDFS-1511
>>>>> --
>>>>>   Take care,
>>>>> Konstantin (Cos) Boudnik
>>>>> 
>>>>> 
>>>>> 
>>>>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <cos@apache.org> wrote:
>>>>>> Considering that because of these 4 faulty cases every patch will be
>>>>>> -1'ed a patch author will still have to look at it and make a comment
>>>>>> why this particular -1 isn't valid. Lesser work, perhaps, but messier
>>>>>> IMO. I'm not blocking it - I just feel like there's a better way.
>>>>>> 
>>>>>> --
>>>>>>   Take care,
>>>>>> Konstantin (Cos) Boudnik
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jghoman@gmail.com> wrote:
>>>>>>>> If HDFS is added to the test-patch queue right now we get
>>>>>>>> nothing but dozens of -1'ed patches.
>>>>>>> There aren't dozens of patches being submitted currently.  The -1
>>>>>>> isn't the important thing, it's the grunt work of actually running
>>>>>>> (and waiting) for the tests, test-patch, etc. that Hudson does so that
>>>>>>> the developer doesn't have to.
>>>>>>> 
>>>>>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <dhruba@gmail.com> wrote:
>>>>>>>> +1, thanks for doing this.
>>>>>>>> 
>>>>>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jghoman@gmail.com> wrote:
>>>>>>>> 
>>>>>>>>> So, with test-patch updated to show the failing tests, saving the
>>>>>>>>> developers the need to go and verify that the failed tests are all
>>>>>>>>> known, how do people feel about turning on test-patch again for HDFS
>>>>>>>>> and mapred?  I think it'll help prevent any more tests from entering
>>>>>>>>> the "yeah, we know" category.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> jg
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <jhoman@yahoo-inc.com> wrote:
>>>>>>>>>> True, each patch would get a -1 and the failing tests would need to be
>>>>>>>>>> verified as those known bad (BTW, it would be great if Hudson could list
>>>>>>>>>> which tests failed in the message it posts to JIRA).  But that's still
>>>>>>>>> quite
>>>>>>>>>> a bit less error-prone work than if the developer runs the tests and
>>>>>>>>>> test-patch themselves.  Also, with 22 being cut, there are a lot of
>>>>>>>>> patches
>>>>>>>>>> up in the air and several developers are juggling multiple patches.  The
>>>>>>>>>> more automation we can have, even if it's not perfect, will decrease
>>>>>>>>> errors
>>>>>>>>>> we may make.
>>>>>>>>>> -jg
>>>>>>>>>> 
>>>>>>>>>> Nigel Daley wrote:
>>>>>>>>>>> 
>>>>>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>>>>>>>>>> 
>>>>>>>>>>>>> It's also ready to run on MapReduce and HDFS but we won't turn it on
>>>>>>>>>>>>> until these projects build and test cleanly.  Looks like both these
>>>>>>>>> projects
>>>>>>>>>>>>> currently have test failures.
>>>>>>>>>>>> 
>>>>>>>>>>>> Assuming the projects are compiling and building, is there a reason to
>>>>>>>>>>>> not turn it on despite the test failures? Hudson is invaluable to
>>>>>>>>> developers
>>>>>>>>>>>> who then don't have to run the tests and test-patch themselves.  We
>>>>>>>>> didn't
>>>>>>>>>>>> turn Hudson off when it was working previously and there were known
>>>>>>>>>>>> failures.  I think one of the reasons we have more failing tests now is
>>>>>>>>> the
>>>>>>>>>>>> higher cost of doing Hudson's work (not a great excuse I know).  This
>>>>>>>>> is
>>>>>>>>>>>> particularly true now because several of the failing tests involve
>>>>>>>>> tests
>>>>>>>>>>>> timing out, making the whole testing regime even longer.
>>>>>>>>>>> 
>>>>>>>>>>> Every single patch would get a -1 and need investigation.  Currently,
>>>>>>>>> that
>>>>>>>>>>> would be about 83 investigations between MR and HDFS issues that are in
>>>>>>>>>>> patch available state.  Shouldn't we focus on getting these tests fixed
>>>>>>>>> or
>>>>>>>>>>> removed/?  Also, I need to get MAPREDUCE-2172 fixed (applies to HDFS as
>>>>>>>>>>> well) before I turn this on.
>>>>>>>>>>> 
>>>>>>>>>>> Cheers,
>>>>>>>>>>> Nige
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> Connect to me at http://www.facebook.com/dhruba
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>> 
>> 


From general-return-2568-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 21 23:00:12 2010
Return-Path: <general-return-2568-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57149 invoked from network); 21 Dec 2010 23:00:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Dec 2010 23:00:12 -0000
Received: (qmail 82448 invoked by uid 500); 21 Dec 2010 23:00:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82402 invoked by uid 500); 21 Dec 2010 23:00:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82394 invoked by uid 99); 21 Dec 2010 23:00:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Dec 2010 23:00:08 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Dec 2010 23:00:00 +0000
Received: by wwd20 with SMTP id 20so4689179wwd.29
        for <general@hadoop.apache.org>; Tue, 21 Dec 2010 14:59:39 -0800 (PST)
Received: by 10.216.170.133 with SMTP id p5mr6646683wel.89.1292972379839; Tue,
 21 Dec 2010 14:59:39 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.59.13 with HTTP; Tue, 21 Dec 2010 14:59:19 -0800 (PST)
In-Reply-To: <2DE2238D-DDFF-4CE3-B50C-39D617AA13E8@mac.com>
References: <27F4E872-C226-4386-9148-077F0044B6E9@Holsman.net>
 <05ED6FF7-6236-47F4-B5DD-8E927971C72D@apache.org> <39694716-7EF6-41B8-8862-D2DE03A4560C@Holsman.NET>
 <9FC0DEC0-524A-4A35-88A6-17473DD74929@apache.org> <C9F82B37-7397-46D8-9788-679380749C1B@Holsman.NET>
 <2DE2238D-DDFF-4CE3-B50C-39D617AA13E8@mac.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Tue, 21 Dec 2010 14:59:19 -0800
X-Google-Sender-Auth: ux8gwzaXvL4IUem1UBp1v9m1cxY
Message-ID: <AANLkTikhaipJmsC0aCAMekV-tpRKzqaWaH118mEmmZ9R@mail.gmail.com>
Subject: Re: Plans for the 0.22 Release
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

+1

On Tue, Dec 21, 2010 at 11:24, Nigel Daley <ndaley@mac.com> wrote:
> I think it's important we get more frequent releases back on track, so I'=
ll volunteer to do it. =A0Its been a while since I was the release manager =
so I'll need help.
>
> Looks like in Jira we currently have:
> =A026 MR blockers: http://tinyurl.com/23s5vzg
> =A017 HDFS blockers: http://tinyurl.com/27w82vl
> =A06 HADOOP blockers: http://tinyurl.com/2bmycl2
> Now that we are past feature freeze (i.e. branch creation), not all of th=
ese are truly blockers. =A0I'll start cleaning up the list and ping folks a=
fter the holidays.
>
> Cheers,
> Nige
>
> On Dec 16, 2010, at 3:10 PM, Ian Holsman wrote:
>
>>
>> On Dec 17, 2010, at 9:00 AM, Owen O'Malley wrote:
>>
>>>> Everyone who has discussed this patch has said it isn't critical to ha=
doop, and It's holding up everything else 0.22 is going to bring.
>>>
>>> I disagree that it isn't critical to Hadoop, but I'm not holding up 0.2=
2. I'm just not volunteering to spend my time working on it, if it doesn't =
have the features that I think it needs.
>>>
>>> -- Owen
>>>
>>
>> That's a fair point.. and this is a volunteer effort.
>> Do we have anybody else who is willing to be the release manager for 22 =
with 6685?
>>
>
>

From general-return-2569-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 22 00:33:33 2010
Return-Path: <general-return-2569-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 10944 invoked from network); 22 Dec 2010 00:33:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Dec 2010 00:33:33 -0000
Received: (qmail 80785 invoked by uid 500); 22 Dec 2010 00:33:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80709 invoked by uid 500); 22 Dec 2010 00:33:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80701 invoked by uid 99); 22 Dec 2010 00:33:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Dec 2010 00:33:31 +0000
X-ASF-Spam-Status: No, hits=-0.2 required=10.0
	tests=DATE_IN_PAST_24_48,FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of tom.e.white@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Dec 2010 00:33:26 +0000
Received: by wwd20 with SMTP id 20so4750835wwd.29
        for <general@hadoop.apache.org>; Tue, 21 Dec 2010 16:33:05 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=7c9t+RzFDW6Lpz2wadTMQ7SjqG4tAFtvvTVopKDs18c=;
        b=RlPZkp6SNT9dYXYCSM0zioL4Xc9+yo1k/M/Ncf4cSeLWAV9JqyCgds7tFXhGwdK6QC
         5fh6Bq9AKVYM02lmt3n8yrlFK4wvpxZVSpyEwR8Pcdqn6WleiSAxASRGiuGQZ5XBNJY3
         e4aA9KOWomOAMl5PaE7+NArjq+SF6GuJ4P6nA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=eG0lb4tVoUI3dKz2c+rThs524vPfoEqmKdMkLe+Ge6OkH2x5bWccYKua9E2ss+KO9X
         7BwaSuk3SqVzobncrth5APth427zUcRU0iv0UXa1xj2MbywQCnvt+pL/0N5wtd3YF6iQ
         MH+xQ0vqj/as+Z9zlsyFOi2X12TVQcUVujsJs=
Received: by 10.216.28.19 with SMTP id f19mr6719324wea.88.1292977985217; Tue,
 21 Dec 2010 16:33:05 -0800 (PST)
MIME-Version: 1.0
Received: by 10.216.19.193 with HTTP; Mon, 20 Dec 2010 09:17:30 -0800 (PST)
In-Reply-To: <AANLkTin9ZdZAVZSvE6yeSWHJKtfON0YiM5Gr55oLD62P@mail.gmail.com>
References: <27F4E872-C226-4386-9148-077F0044B6E9@Holsman.net>
 <05ED6FF7-6236-47F4-B5DD-8E927971C72D@apache.org> <39694716-7EF6-41B8-8862-D2DE03A4560C@Holsman.NET>
 <9FC0DEC0-524A-4A35-88A6-17473DD74929@apache.org> <C9F82B37-7397-46D8-9788-679380749C1B@Holsman.NET>
 <AANLkTin9ZdZAVZSvE6yeSWHJKtfON0YiM5Gr55oLD62P@mail.gmail.com>
From: Tom White <tom.e.white@gmail.com>
Date: Mon, 20 Dec 2010 09:17:30 -0800
Message-ID: <AANLkTi=Ne1oOQ8ioqJsXxX=bU43YA4DZTbbNmNjvY1hM@mail.gmail.com>
Subject: Re: Plans for the 0.22 Release
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Fri, Dec 17, 2010 at 2:34 PM, Konstantin Shvachko
<shv.hadoop@gmail.com> wrote:
> Owen, Doug, Tom
>
> Could you please formulate and reply to this email separately
> =A0 =A0 what would be an *ACCEPTABLE *resolution of
> =A0 =A0 HADOOP-6685 for *YOU *to move *0.22* forward.
> Just trying to get something to work with to get us beyond the stagnation
> point.
>
> It could be "I want this patch in/out as is, final answer". Then we are
> stuck.
> But at least we will know there is no resolution to hope for anymore.
> And we have to find other ways based on that fact.
>
> It could be a zero-option plan - remove dependencies both for Avro and
> ProtocolBuffers out into libraries, similar to schedulers.
> Or something else.
> Let's see if there is any common ground. If there is
> we can further talk about implementation and in the mean time declare
> the 0.22 freeze contingent on the completion of H-6685.

I don't personally see HADOOP-6685 as a blocker for a 0.22 release,
since there is a lot of value in there already that has not been
released yet, such as security. However, to get HADOOP-6685 resolved,
from my point of view the main thing to sort out are the modularity
concerns that I and others have raised, so that serializations are
pluggable and don't add potentially incompatible libraries onto the
user's classpath.

Thanks,
Tom

>
> Thanks,
> --Konstantin
>
>
> On Thu, Dec 16, 2010 at 3:10 PM, Ian Holsman <hadoop@holsman.net> wrote:
>
>>
>> On Dec 17, 2010, at 9:00 AM, Owen O'Malley wrote:
>>
>> >> Everyone who has discussed this patch has said it isn't critical to
>> hadoop, and It's holding up everything else 0.22 is going to bring.
>> >
>> > I disagree that it isn't critical to Hadoop, but I'm not holding up 0.=
22.
>> I'm just not volunteering to spend my time working on it, if it doesn't =
have
>> the features that I think it needs.
>> >
>> > -- Owen
>> >
>>
>> That's a fair point.. and this is a volunteer effort.
>> Do we have anybody else who is willing to be the release manager for 22
>> with 6685?
>>
>>
>

From general-return-2570-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 22 07:57:04 2010
Return-Path: <general-return-2570-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 90740 invoked from network); 22 Dec 2010 07:57:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Dec 2010 07:57:02 -0000
Received: (qmail 52636 invoked by uid 500); 22 Dec 2010 07:57:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51830 invoked by uid 500); 22 Dec 2010 07:56:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51773 invoked by uid 99); 22 Dec 2010 07:56:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Dec 2010 07:56:58 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [88.198.2.104] (HELO koch.ro) (88.198.2.104)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Dec 2010 07:56:53 +0000
Received: from 46-127-147-3.dclient.hispeed.ch ([46.127.147.3] helo=jona.localnet)
	by koch.ro with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
	(Exim 4.69)
	(envelope-from <thomas@koch.ro>)
	id 1PVJYl-0005is-Ix; Wed, 22 Dec 2010 08:56:31 +0100
From: Thomas Koch <thomas@koch.ro>
Reply-To: thomas@koch.ro
To: general@hadoop.apache.org,
 infrastructure-dev@apache.org,
 Jukka Zitting <jukka.zitting@gmail.com>,
 dev@lucene.apache.org
Subject: Test instance of Gerrit?
Date: Wed, 22 Dec 2010 08:56:27 +0100
User-Agent: KMail/1.13.5 (Linux/2.6.32-4-amd64; KDE/4.4.5; x86_64; ; )
MIME-Version: 1.0
Content-Type: text/plain;
  charset="us-ascii"
Content-Transfer-Encoding: 7bit
Message-Id: <201012220856.27550.thomas@koch.ro>

Hi,

the topic of official GIT repos for apache projects has been raised again a 
few weeks ago on infrastructure-dev@a.o.[1] However I can't see who is doing 
what in which direction over there, if anything is moving at all.

[1] http://comments.gmane.org/gmane.comp.apache.infrastructure.devel/1151

I proposed a draft for a requirements document for the GIT setup and suggested 
to give Gerrit a try. There hasn't been a reply however since more than two 
weeks. Gerrit is a repository manager developed for the android project and 
also used by the repository foundation. It includes patch review, workflow and 
CLA-tracking.

[2] http://code.google.com/p/gerrit/

Now my question is, whether there's some interest from developers to give 
Gerrit a try? In this case, I'd setup a private toy instance and import your 
project.
I don't want to shoot anybody from infrastructure-dev@a.o in the back. I just 
think it would help to have some hands-on experience so that people can make 
better judgements.

Best regards,

Thomas Koch, http://www.koch.ro

From general-return-2571-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 22 15:04:55 2010
Return-Path: <general-return-2571-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25338 invoked from network); 22 Dec 2010 15:04:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Dec 2010 15:04:55 -0000
Received: (qmail 45336 invoked by uid 500); 22 Dec 2010 15:04:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45165 invoked by uid 500); 22 Dec 2010 15:04:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45157 invoked by uid 99); 22 Dec 2010 15:04:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Dec 2010 15:04:51 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bjoern@schiessle.org designates 78.46.69.5 as permitted sender)
Received: from [78.46.69.5] (HELO zucker.schokokeks.org) (78.46.69.5)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Dec 2010 15:04:43 +0000
Received: from laptop (infedyn152.informatik.uni-stuttgart.de [::ffff:129.69.220.152])
  (AUTH: LOGIN schiesbn@schokokeks.org, TLS: TLSv1/SSLv3,128bits,AES128-SHA)
  by zucker.schokokeks.org with ESMTPSA; Wed, 22 Dec 2010 16:04:20 +0100
  id 000000000001C006.000000004D121375.00004D28
Date: Wed, 22 Dec 2010 16:03:11 +0100
From: Bjoern Schiessle <bjoern@schiessle.org>
To: general@hadoop.apache.org
Subject: namenode doesn't start after reboot
Message-ID: <20101222160311.7ffb9b6d@laptop>
X-Mailer: Claws Mail 3.7.6 (GTK+ 2.22.0; i486-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi,

After a Kernel update and a reboot the namenode doesn't start. I run the
Cloudera cdh3 Hadoop distribution. I have already searched for a solution.
It looks like I'm not the only one with such a problem. Sadly I could only
find descriptions of similar problems, but no solutions...

This is the error message from the namenode log file:


2010-12-22 16:13:04,830 INFO org.apache.hadoop.hdfs.server.namenode.NameNod=
e: STARTUP_MSG:=20
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host =3D pcube/129.69.216.24
STARTUP_MSG:   args =3D []
STARTUP_MSG:   version =3D 0.20.2+737
STARTUP_MSG:   build =3D  -r 98c55c28258aa6f42250569bd7fa431ac657bdbd; comp=
iled by 'root' on Mon Oct 11 17:21:30 UTC 2010
************************************************************/
2010-12-22 16:13:05,001 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Init=
ializing JVM Metrics with processName=3DNameNode, sessionId=3Dnull
2010-12-22 16:13:05,007 INFO org.apache.hadoop.hdfs.server.namenode.metrics=
.NameNodeMetrics: Initializing NameNodeMeterics using context object:org.ap=
ache.hadoop.metrics.spi.NullContext
2010-12-22 16:13:05,036 INFO org.apache.hadoop.hdfs.server.namenode.FSNames=
ystem: fsOwner=3Dhdfs
2010-12-22 16:13:05,036 INFO org.apache.hadoop.hdfs.server.namenode.FSNames=
ystem: supergroup=3Dsupergroup
2010-12-22 16:13:05,036 INFO org.apache.hadoop.hdfs.server.namenode.FSNames=
ystem: isPermissionEnabled=3Dfalse
2010-12-22 16:13:05,040 INFO org.apache.hadoop.hdfs.server.namenode.FSNames=
ystem: isAccessTokenEnabled=3Dfalse accessKeyUpdateInterval=3D0 min(s), acc=
essTokenLifetime=3D0 min(s)
2010-12-22 16:13:05,335 INFO org.apache.hadoop.hdfs.server.namenode.metrics=
.FSNamesystemMetrics: Initializing FSNamesystemMetrics using context object=
:org.apache.hadoop.metrics.spi.NullContext
2010-12-22 16:13:05,336 INFO org.apache.hadoop.hdfs.server.namenode.FSNames=
ystem: Registered FSNamesystemStatusMBean
2010-12-22 16:13:05,361 INFO org.apache.hadoop.hdfs.server.common.Storage: =
Number of files =3D 72
2010-12-22 16:13:05,374 INFO org.apache.hadoop.hdfs.server.common.Storage: =
Number of files under construction =3D 3
2010-12-22 16:13:05,375 INFO org.apache.hadoop.hdfs.server.common.Storage: =
Image file of size 8822 loaded in 0 seconds.
2010-12-22 16:13:05,377 ERROR org.apache.hadoop.hdfs.server.namenode.NameNo=
de: java.lang.NullPointerException
	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addChild(FSDirectory=
.java:1088)
	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addChild(FSDirectory=
.java:1100)
	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addNode(FSDirectory.=
java:1003)
	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedAddFile(F=
SDirectory.java:206)
	at org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.=
java:637)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java=
:1034)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java=
:845)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FS=
Image.java:379)
	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirect=
ory.java:99)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesy=
stem.java:343)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem=
.java:317)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.jav=
a:214)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:39=
4)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode=
.java:1148)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1157)

2010-12-22 16:13:05,377 INFO org.apache.hadoop.hdfs.server.namenode.NameNod=
e: SHUTDOWN_MSG:=20
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at pcube/129.69.216.24
************************************************************/

Any idea what could be wrong and how I can get my namenode up running again?

Thanks a lot!
Bj=F6rn

From general-return-2572-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 22 17:44:54 2010
Return-Path: <general-return-2572-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9757 invoked from network); 22 Dec 2010 17:44:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Dec 2010 17:44:53 -0000
Received: (qmail 52530 invoked by uid 500); 22 Dec 2010 17:44:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52203 invoked by uid 500); 22 Dec 2010 17:44:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52077 invoked by uid 99); 22 Dec 2010 17:44:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Dec 2010 17:44:15 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of Ken_Walker@ca.ibm.com designates 32.97.182.144 as permitted sender)
Received: from [32.97.182.144] (HELO e4.ny.us.ibm.com) (32.97.182.144)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Dec 2010 17:44:15 +0000
Received: from d01dlp01.pok.ibm.com (d01dlp01.pok.ibm.com [9.56.224.56])
	by e4.ny.us.ibm.com (8.14.4/8.13.1) with ESMTP id oBMHQE28017721
	for <general@hadoop.apache.org>; Wed, 22 Dec 2010 12:26:26 -0500
Received: from d01relay01.pok.ibm.com (d01relay01.pok.ibm.com [9.56.227.233])
	by d01dlp01.pok.ibm.com (Postfix) with ESMTP id 1D4B4728051
	for <general@hadoop.apache.org>; Wed, 22 Dec 2010 12:43:50 -0500 (EST)
Received: from d01av04.pok.ibm.com (d01av04.pok.ibm.com [9.56.224.64])
	by d01relay01.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id oBMHhnXB373898
	for <general@hadoop.apache.org>; Wed, 22 Dec 2010 12:43:49 -0500
Received: from d01av04.pok.ibm.com (loopback [127.0.0.1])
	by d01av04.pok.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id oBMHhnAu010023
	for <general@hadoop.apache.org>; Wed, 22 Dec 2010 12:43:49 -0500
Received: from d25ml04.torolab.ibm.com (d25ml04.torolab.ibm.com [9.26.6.105])
	by d01av04.pok.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id oBMHhnOc009959
	for <general@hadoop.apache.org>; Wed, 22 Dec 2010 12:43:49 -0500
In-Reply-To: <2DE2238D-DDFF-4CE3-B50C-39D617AA13E8@mac.com>
References: <27F4E872-C226-4386-9148-077F0044B6E9@Holsman.net> <05ED6FF7-6236-47F4-B5DD-8E927971C72D@apache.org> <39694716-7EF6-41B8-8862-D2DE03A4560C@Holsman.NET> <9FC0DEC0-524A-4A35-88A6-17473DD74929@apache.org> <C9F82B37-7397-46D8-9788-679380749C1B@Holsman.NET> <2DE2238D-DDFF-4CE3-B50C-39D617AA13E8@mac.com>
Subject: Re: Plans for the 0.22 Release
X-KeepSent: 7374242E:2BE4BD82-85257801:00613180;
 type=4; name=$KeepSent
To: general@hadoop.apache.org
X-Mailer: Lotus Notes Release 8.5.1FP1 January 05, 2010
Message-ID: <OF7374242E.2BE4BD82-ON85257801.00613180-85257801.006158C2@ca.ibm.com>
From: Ken Walker <Ken_Walker@ca.ibm.com>
Date: Wed, 22 Dec 2010 12:43:47 -0500
X-MIMETrack: Serialize by Router on D25ML04/25/M/IBM(Release 8.0.2FP5|April 13, 2010) at
 12/22/2010 12:43:48
MIME-Version: 1.0
Content-type: multipart/related; 
	Boundary="0__=0ABBF292DFF2B7108f9e8a93df938690918c0ABBF292DFF2B710"
X-Content-Scanned: Fidelis XPS MAILER

--0__=0ABBF292DFF2B7108f9e8a93df938690918c0ABBF292DFF2B710
Content-type: multipart/alternative; 
	Boundary="1__=0ABBF292DFF2B7108f9e8a93df938690918c0ABBF292DFF2B710"

--1__=0ABBF292DFF2B7108f9e8a93df938690918c0ABBF292DFF2B710
Content-type: text/plain; charset=US-ASCII
Content-transfer-encoding: quoted-printable


Not sure what state this issue is in, would be nice for 0.22

https://issues.apache.org/jira/browse/HADOOP-6941

I don't see it in the lists below.  It's a bit of a blocker running on
non-Sun (Oracle) JREs.
                                                                       =
          
                                                                       =
          
                                                                       =
          
                                                                       =
          
                                                                       =
          
 Ken (K.N.)                                  2670 Queensview           =
          
 Walker                                     Drive                      =
          
                                                                       =
          
 J9 Java                                     Ottawa, Ontario K2B       =
          
 Class                                      8K1                        =
          
 Libraries                                                             =
          
 Lead                                                                  =
          
                                                                       =
          
 Java                                        Canada                    =
          
 Technology                                                            =
          
 Centre                                                                =
          
                                                                       =
          
 IBM                                                                   =
          
 Software                                                              =
          
 Group                                                                 =
          
                                                                       =
          
 Phone:     +1-613-356-5021                                            =
          
                                                                       =
          
 e-mail:    ken_walker@ca.ibm.com                                      =
          
                                                                       =
          
                                                                       =
          
                                                                       =
          
                                                                       =
          
                                                                       =
          






|------------>
| From:      |
|------------>
  >--------------------------------------------------------------------=
-----------------------------------------------------------------------=
-------|
  |Nigel Daley <ndaley@mac.com>                                        =
                                                                       =
       |
  >--------------------------------------------------------------------=
-----------------------------------------------------------------------=
-------|
|------------>
| To:        |
|------------>
  >--------------------------------------------------------------------=
-----------------------------------------------------------------------=
-------|
  |general@hadoop.apache.org                                           =
                                                                       =
       |
  >--------------------------------------------------------------------=
-----------------------------------------------------------------------=
-------|
|------------>
| Date:      |
|------------>
  >--------------------------------------------------------------------=
-----------------------------------------------------------------------=
-------|
  |2010/12/21 02:26 PM                                                 =
                                                                       =
       |
  >--------------------------------------------------------------------=
-----------------------------------------------------------------------=
-------|
|------------>
| Subject:   |
|------------>
  >--------------------------------------------------------------------=
-----------------------------------------------------------------------=
-------|
  |Re: Plans for the 0.22 Release                                      =
                                                                       =
       |
  >--------------------------------------------------------------------=
-----------------------------------------------------------------------=
-------|





I think it's important we get more frequent releases back on track, so =
I'll
volunteer to do it.  Its been a while since I was the release manager s=
o
I'll need help.

Looks like in Jira we currently have:
  26 MR blockers: http://tinyurl.com/23s5vzg
  17 HDFS blockers: http://tinyurl.com/27w82vl
  6 HADOOP blockers: http://tinyurl.com/2bmycl2
Now that we are past feature freeze (i.e. branch creation), not all of
these are truly blockers.  I'll start cleaning up the list and ping fol=
ks
after the holidays.

Cheers,
Nige

On Dec 16, 2010, at 3:10 PM, Ian Holsman wrote:

>
> On Dec 17, 2010, at 9:00 AM, Owen O'Malley wrote:
>
>>> Everyone who has discussed this patch has said it isn't critical to=

hadoop, and It's holding up everything else 0.22 is going to bring.
>>
>> I disagree that it isn't critical to Hadoop, but I'm not holding up
0.22. I'm just not volunteering to spend my time working on it, if it
doesn't have the features that I think it needs.
>>
>> -- Owen
>>
>
> That's a fair point.. and this is a volunteer effort.
> Do we have anybody else who is willing to be the release manager for =
22
with 6685?
>


=

--1__=0ABBF292DFF2B7108f9e8a93df938690918c0ABBF292DFF2B710
Content-type: text/html; charset=US-ASCII
Content-Disposition: inline
Content-transfer-encoding: quoted-printable

<html><body>
<p>Not sure what state this issue is in, would be nice for 0.22<br>
<br>
<a href=3D"https://issues.apache.org/jira/browse/HADOOP-6941">https://i=
ssues.apache.org/jira/browse/HADOOP-6941</a><br>
<br>
I don't see it in the lists below.  It's a bit of a blocker running on =
non-Sun (Oracle) JREs.<br>
<br>

<table border=3D"0" cellspacing=3D"0" cellpadding=3D"0">
<tr valign=3D"top"><td width=3D"600" colspan=3D"4" valign=3D"middle"><h=
r width=3D"100%" size=3D"2" align=3D"left" style=3D"color:#696969; "></=
td></tr>

<tr valign=3D"top"><td width=3D"600" colspan=3D"4" valign=3D"middle"><i=
mg width=3D"1" height=3D"1" src=3D"cid:1__=3D0ABBF292DFF2B7108f9e8a93df=
938@ca.ibm.com" border=3D"0" alt=3D""></td></tr>

<tr valign=3D"top"><td width=3D"320" colspan=3D"2" valign=3D"middle"><b=
>Ken (K.N.) Walker</b></td><td width=3D"171" valign=3D"middle"><font si=
ze=3D"2" color=3D"#696969"> 2670 Queensview Drive</font></td><td width=3D=
"109" rowspan=3D"3" valign=3D"middle"><div align=3D"right"><img src=3D"=
cid:2__=3D0ABBF292DFF2B7108f9e8a93df938@ca.ibm.com" width=3D"83" height=
=3D"30"></div></td></tr>

<tr valign=3D"top"><td width=3D"320" colspan=3D"2" valign=3D"middle"><f=
ont size=3D"2" color=3D"#696969">J9 Java Class Libraries Lead</font></t=
d><td width=3D"171" valign=3D"middle"><font size=3D"2" color=3D"#696969=
"> Ottawa, Ontario K2B 8K1</font></td></tr>

<tr valign=3D"top"><td width=3D"320" colspan=3D"2" valign=3D"middle"><f=
ont size=3D"2" color=3D"#696969">Java Technology Centre</font></td><td =
width=3D"171" valign=3D"middle"><font size=3D"2" color=3D"#696969"> Can=
ada</font></td></tr>

<tr valign=3D"top"><td width=3D"320" colspan=3D"2" valign=3D"middle"><f=
ont size=3D"2" color=3D"#696969">IBM Software Group</font></td><td widt=
h=3D"171" valign=3D"middle"><font size=3D"2" color=3D"#696969"> </font>=
</td><td width=3D"109" valign=3D"middle"><img width=3D"1" height=3D"1" =
src=3D"cid:1__=3D0ABBF292DFF2B7108f9e8a93df938@ca.ibm.com" border=3D"0"=
 alt=3D""></td></tr>

<tr valign=3D"top"><td width=3D"69" valign=3D"middle"><font size=3D"2" =
color=3D"#696969">Phone:</font></td><td width=3D"251" valign=3D"middle"=
><font size=3D"2" color=3D"#696969">+1-613-356-5021</font></td><td widt=
h=3D"171" valign=3D"middle"><font size=3D"2" color=3D"#696969">  </font=
></td><td width=3D"109" valign=3D"middle"><img width=3D"1" height=3D"1"=
 src=3D"cid:1__=3D0ABBF292DFF2B7108f9e8a93df938@ca.ibm.com" border=3D"0=
" alt=3D""></td></tr>

<tr valign=3D"top"><td width=3D"69" valign=3D"middle"><font size=3D"2" =
color=3D"#696969">e-mail:</font></td><td width=3D"251" valign=3D"middle=
"><font size=3D"2" color=3D"#696969">ken_walker@ca.ibm.com</font></td><=
td width=3D"171" valign=3D"middle"><font size=3D"2" color=3D"#696969"> =
 </font></td><td width=3D"109" valign=3D"middle"><img width=3D"1" heigh=
t=3D"1" src=3D"cid:1__=3D0ABBF292DFF2B7108f9e8a93df938@ca.ibm.com" bord=
er=3D"0" alt=3D""></td></tr>

<tr valign=3D"top"><td width=3D"600" colspan=3D"4" valign=3D"middle"><f=
ont size=3D"2" color=3D"#696969"> </font></td></tr>

<tr valign=3D"top"><td width=3D"600" colspan=3D"4" valign=3D"middle"><f=
ont size=3D"2" color=3D"#696969"> </font></td></tr>
</table>
<br>
<br>
<br>
<img width=3D"16" height=3D"16" src=3D"cid:3__=3D0ABBF292DFF2B7108f9e8a=
93df938@ca.ibm.com" border=3D"0" alt=3D"Inactive hide details for Nigel=
 Daley ---2010/12/21 02:26:32 PM---I think it's important we get more f=
requent releases back on "><font color=3D"#424282">Nigel Daley ---2010/=
12/21 02:26:32 PM---I think it's important we get more frequent release=
s back on track, so I'll volunteer to do it.  Its</font><br>
<br>

<table width=3D"100%" border=3D"0" cellspacing=3D"0" cellpadding=3D"0">=

<tr valign=3D"top"><td width=3D"1%"><img width=3D"96" height=3D"1" src=3D=
"cid:1__=3D0ABBF292DFF2B7108f9e8a93df938@ca.ibm.com" border=3D"0" alt=3D=
""><br>
<font size=3D"2" color=3D"#5F5F5F">From:</font></td><td width=3D"100%">=
<img width=3D"1" height=3D"1" src=3D"cid:1__=3D0ABBF292DFF2B7108f9e8a93=
df938@ca.ibm.com" border=3D"0" alt=3D""><br>
<font size=3D"2">Nigel Daley &lt;ndaley@mac.com&gt;</font></td></tr>

<tr valign=3D"top"><td width=3D"1%"><img width=3D"96" height=3D"1" src=3D=
"cid:1__=3D0ABBF292DFF2B7108f9e8a93df938@ca.ibm.com" border=3D"0" alt=3D=
""><br>
<font size=3D"2" color=3D"#5F5F5F">To:</font></td><td width=3D"100%"><i=
mg width=3D"1" height=3D"1" src=3D"cid:1__=3D0ABBF292DFF2B7108f9e8a93df=
938@ca.ibm.com" border=3D"0" alt=3D""><br>
<font size=3D"2">general@hadoop.apache.org</font></td></tr>

<tr valign=3D"top"><td width=3D"1%"><img width=3D"96" height=3D"1" src=3D=
"cid:1__=3D0ABBF292DFF2B7108f9e8a93df938@ca.ibm.com" border=3D"0" alt=3D=
""><br>
<font size=3D"2" color=3D"#5F5F5F">Date:</font></td><td width=3D"100%">=
<img width=3D"1" height=3D"1" src=3D"cid:1__=3D0ABBF292DFF2B7108f9e8a93=
df938@ca.ibm.com" border=3D"0" alt=3D""><br>
<font size=3D"2">2010/12/21 02:26 PM</font></td></tr>

<tr valign=3D"top"><td width=3D"1%"><img width=3D"96" height=3D"1" src=3D=
"cid:1__=3D0ABBF292DFF2B7108f9e8a93df938@ca.ibm.com" border=3D"0" alt=3D=
""><br>
<font size=3D"2" color=3D"#5F5F5F">Subject:</font></td><td width=3D"100=
%"><img width=3D"1" height=3D"1" src=3D"cid:1__=3D0ABBF292DFF2B7108f9e8=
a93df938@ca.ibm.com" border=3D"0" alt=3D""><br>
<font size=3D"2">Re: Plans for the 0.22 Release</font></td></tr>
</table>
<hr width=3D"100%" size=3D"2" align=3D"left" noshade style=3D"color:#80=
91A5; "><br>
<br>
<br>
<tt>I think it's important we get more frequent releases back on track,=
 so I'll volunteer to do it. &nbsp;Its been a while since I was the rel=
ease manager so I'll need help.<br>
<br>
Looks like in Jira we currently have:<br>
 &nbsp;26 MR blockers: </tt><tt><a href=3D"http://tinyurl.com/23s5vzg">=
http://tinyurl.com/23s5vzg</a></tt><tt><br>
 &nbsp;17 HDFS blockers: </tt><tt><a href=3D"http://tinyurl.com/27w82vl=
">http://tinyurl.com/27w82vl</a></tt><tt><br>
 &nbsp;6 HADOOP blockers: </tt><tt><a href=3D"http://tinyurl.com/2bmycl=
2">http://tinyurl.com/2bmycl2</a></tt><tt><br>
Now that we are past feature freeze (i.e. branch creation), not all of =
these are truly blockers. &nbsp;I'll start cleaning up the list and pin=
g folks after the holidays.<br>
<br>
Cheers,<br>
Nige<br>
<br>
On Dec 16, 2010, at 3:10 PM, Ian Holsman wrote:<br>
<br>
&gt; <br>
&gt; On Dec 17, 2010, at 9:00 AM, Owen O'Malley wrote:<br>
&gt; <br>
&gt;&gt;&gt; Everyone who has discussed this patch has said it isn't cr=
itical to hadoop, and It's holding up everything else 0.22 is going to =
bring.<br>
&gt;&gt; <br>
&gt;&gt; I disagree that it isn't critical to Hadoop, but I'm not holdi=
ng up 0.22. I'm just not volunteering to spend my time working on it, i=
f it doesn't have the features that I think it needs.<br>
&gt;&gt; <br>
&gt;&gt; -- Owen<br>
&gt;&gt; <br>
&gt; <br>
&gt; That's a fair point.. and this is a volunteer effort.<br>
&gt; Do we have anybody else who is willing to be the release manager f=
or 22 with 6685?<br>
&gt; <br>
<br>
</tt><br>
<br>
</body></html>=


--1__=0ABBF292DFF2B7108f9e8a93df938690918c0ABBF292DFF2B710--


--0__=0ABBF292DFF2B7108f9e8a93df938690918c0ABBF292DFF2B710--


From general-return-2573-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 22 18:02:12 2010
Return-Path: <general-return-2573-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 15467 invoked from network); 22 Dec 2010 18:02:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Dec 2010 18:02:11 -0000
Received: (qmail 95446 invoked by uid 500); 22 Dec 2010 18:02:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95362 invoked by uid 500); 22 Dec 2010 18:02:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95343 invoked by uid 99); 22 Dec 2010 18:02:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Dec 2010 18:02:09 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dsikar@gmail.com designates 209.85.213.48 as permitted sender)
Received: from [209.85.213.48] (HELO mail-yw0-f48.google.com) (209.85.213.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Dec 2010 18:02:03 +0000
Received: by ywo7 with SMTP id 7so2771035ywo.35
        for <general@hadoop.apache.org>; Wed, 22 Dec 2010 10:01:42 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=SFoWc6OZIL0gZ67myRS20YS/7PU80v8mao+mjxKZKEU=;
        b=VjzpHPs9RL6YExfEWIeH9d33MKlYRXLfYV95I4LMROMqUWWq2IO613WuGdbt2tjR46
         TkLRwVhwNlhxe7KUWi2qi/bSNsIm1jsnfnT6b/klQRe4Ab+y4+EBHSvcW+thBmkWy15k
         Wwa3cbnofJV046VfpIgQnA0suliLRc8DnZ1C0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=EsWZchD49SZAnRDhzEaj1F6GaYDGQ4rUPZNrDduHNPmCIOq1Dr7pudI/LMPaxNmGBg
         bb1h752QDB/5Cpk3hfC/63jPzJTEnysA8uQV+9z5sVa2SDFKIvkIHUe2a9uhxDtciZHT
         3UEogXA8VOEUwNBfQakSADnUWIRzwxSWn9llo=
MIME-Version: 1.0
Received: by 10.90.49.7 with SMTP id w7mr4950839agw.55.1293040902832; Wed, 22
 Dec 2010 10:01:42 -0800 (PST)
Received: by 10.90.62.15 with HTTP; Wed, 22 Dec 2010 10:01:42 -0800 (PST)
In-Reply-To: <20101222160311.7ffb9b6d@laptop>
References: <20101222160311.7ffb9b6d@laptop>
Date: Wed, 22 Dec 2010 18:01:42 +0000
Message-ID: <AANLkTike7k5nqViAsdyC_mR6LOBVrrFFqsyGJo=-W6GQ@mail.gmail.com>
Subject: Re: namenode doesn't start after reboot
From: daniel sikar <dsikar@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I can't help but with hindsight - it's advisable to snapshot your
namenodes as HDFS dies with them.

On 22 December 2010 15:03, Bjoern Schiessle <bjoern@schiessle.org> wrote:
> Hi,
>
> After a Kernel update and a reboot the namenode doesn't start. I run the
> Cloudera cdh3 Hadoop distribution. I have already searched for a solution=
.
> It looks like I'm not the only one with such a problem. Sadly I could onl=
y
> find descriptions of similar problems, but no solutions...
>
> This is the error message from the namenode log file:
>
>
> 2010-12-22 16:13:04,830 INFO org.apache.hadoop.hdfs.server.namenode.NameN=
ode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG: =A0 host =3D pcube/129.69.216.24
> STARTUP_MSG: =A0 args =3D []
> STARTUP_MSG: =A0 version =3D 0.20.2+737
> STARTUP_MSG: =A0 build =3D =A0-r 98c55c28258aa6f42250569bd7fa431ac657bdbd=
; compiled by 'root' on Mon Oct 11 17:21:30 UTC 2010
> ************************************************************/
> 2010-12-22 16:13:05,001 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: In=
itializing JVM Metrics with processName=3DNameNode, sessionId=3Dnull
> 2010-12-22 16:13:05,007 INFO org.apache.hadoop.hdfs.server.namenode.metri=
cs.NameNodeMetrics: Initializing NameNodeMeterics using context object:org.=
apache.hadoop.metrics.spi.NullContext
> 2010-12-22 16:13:05,036 INFO org.apache.hadoop.hdfs.server.namenode.FSNam=
esystem: fsOwner=3Dhdfs
> 2010-12-22 16:13:05,036 INFO org.apache.hadoop.hdfs.server.namenode.FSNam=
esystem: supergroup=3Dsupergroup
> 2010-12-22 16:13:05,036 INFO org.apache.hadoop.hdfs.server.namenode.FSNam=
esystem: isPermissionEnabled=3Dfalse
> 2010-12-22 16:13:05,040 INFO org.apache.hadoop.hdfs.server.namenode.FSNam=
esystem: isAccessTokenEnabled=3Dfalse accessKeyUpdateInterval=3D0 min(s), a=
ccessTokenLifetime=3D0 min(s)
> 2010-12-22 16:13:05,335 INFO org.apache.hadoop.hdfs.server.namenode.metri=
cs.FSNamesystemMetrics: Initializing FSNamesystemMetrics using context obje=
ct:org.apache.hadoop.metrics.spi.NullContext
> 2010-12-22 16:13:05,336 INFO org.apache.hadoop.hdfs.server.namenode.FSNam=
esystem: Registered FSNamesystemStatusMBean
> 2010-12-22 16:13:05,361 INFO org.apache.hadoop.hdfs.server.common.Storage=
: Number of files =3D 72
> 2010-12-22 16:13:05,374 INFO org.apache.hadoop.hdfs.server.common.Storage=
: Number of files under construction =3D 3
> 2010-12-22 16:13:05,375 INFO org.apache.hadoop.hdfs.server.common.Storage=
: Image file of size 8822 loaded in 0 seconds.
> 2010-12-22 16:13:05,377 ERROR org.apache.hadoop.hdfs.server.namenode.Name=
Node: java.lang.NullPointerException
> =A0 =A0 =A0 =A0at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addC=
hild(FSDirectory.java:1088)
> =A0 =A0 =A0 =A0at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addC=
hild(FSDirectory.java:1100)
> =A0 =A0 =A0 =A0at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addN=
ode(FSDirectory.java:1003)
> =A0 =A0 =A0 =A0at org.apache.hadoop.hdfs.server.namenode.FSDirectory.unpr=
otectedAddFile(FSDirectory.java:206)
> =A0 =A0 =A0 =A0at org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFS=
Edits(FSEditLog.java:637)
> =A0 =A0 =A0 =A0at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEd=
its(FSImage.java:1034)
> =A0 =A0 =A0 =A0at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSIm=
age(FSImage.java:845)
> =A0 =A0 =A0 =A0at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverT=
ransitionRead(FSImage.java:379)
> =A0 =A0 =A0 =A0at org.apache.hadoop.hdfs.server.namenode.FSDirectory.load=
FSImage(FSDirectory.java:99)
> =A0 =A0 =A0 =A0at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.ini=
tialize(FSNamesystem.java:343)
> =A0 =A0 =A0 =A0at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<in=
it>(FSNamesystem.java:317)
> =A0 =A0 =A0 =A0at org.apache.hadoop.hdfs.server.namenode.NameNode.initial=
ize(NameNode.java:214)
> =A0 =A0 =A0 =A0at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(=
NameNode.java:394)
> =A0 =A0 =A0 =A0at org.apache.hadoop.hdfs.server.namenode.NameNode.createN=
ameNode(NameNode.java:1148)
> =A0 =A0 =A0 =A0at org.apache.hadoop.hdfs.server.namenode.NameNode.main(Na=
meNode.java:1157)
>
> 2010-12-22 16:13:05,377 INFO org.apache.hadoop.hdfs.server.namenode.NameN=
ode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at pcube/129.69.216.24
> ************************************************************/
>
> Any idea what could be wrong and how I can get my namenode up running aga=
in?
>
> Thanks a lot!
> Bj=F6rn
>

From general-return-2574-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 22 19:50:13 2010
Return-Path: <general-return-2574-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 65880 invoked from network); 22 Dec 2010 19:50:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Dec 2010 19:50:12 -0000
Received: (qmail 41843 invoked by uid 500); 22 Dec 2010 19:50:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 41622 invoked by uid 500); 22 Dec 2010 19:50:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 41614 invoked by uid 99); 22 Dec 2010 19:50:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Dec 2010 19:50:11 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of shv.hadoop@gmail.com designates 209.85.213.176 as permitted sender)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Dec 2010 19:50:05 +0000
Received: by yxm8 with SMTP id 8so2806997yxm.35
        for <general@hadoop.apache.org>; Wed, 22 Dec 2010 11:49:44 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=JCAAtuDU1ib2S1Gm36HnVjIUHyJwWOzYb/SyTyvSr+0=;
        b=McQ904JRZvLdsOG1Fg460Fg6kHo11Xf6wtXyjKmAh9p0abTJdAEsO3Q/0MAM4dcYgT
         I8OdI63IVS3RrJmeFA88XLcvy1xhdCwsg+2nokWc/EuOFEVSJpzzu3L1zBB7Edt91cq2
         sOnA7xQCHOO8u2OYpizH3Pa7vTEX+9eUulF+U=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=j6SgOYA/APflULnYdZVQ83pKAqnBlMyaAisXIgfDqex/r3mYnWOaguvcXlSnNvfOaq
         W8ya+9tSrupKCpAuA634ak2wjJ1VmcRnFs+KmhQ+yREC0P/YBKqFdVYuZjOzWRdo2Aap
         aiOe0DEl4MQtuReF/wuEg/ACkKpgR8qEbFLk0=
MIME-Version: 1.0
Received: by 10.236.103.44 with SMTP id e32mr3821030yhg.37.1293047384288; Wed,
 22 Dec 2010 11:49:44 -0800 (PST)
Received: by 10.236.109.15 with HTTP; Wed, 22 Dec 2010 11:49:44 -0800 (PST)
In-Reply-To: <2DE2238D-DDFF-4CE3-B50C-39D617AA13E8@mac.com>
References: <27F4E872-C226-4386-9148-077F0044B6E9@Holsman.net>
	<05ED6FF7-6236-47F4-B5DD-8E927971C72D@apache.org>
	<39694716-7EF6-41B8-8862-D2DE03A4560C@Holsman.NET>
	<9FC0DEC0-524A-4A35-88A6-17473DD74929@apache.org>
	<C9F82B37-7397-46D8-9788-679380749C1B@Holsman.NET>
	<2DE2238D-DDFF-4CE3-B50C-39D617AA13E8@mac.com>
Date: Wed, 22 Dec 2010 11:49:44 -0800
Message-ID: <AANLkTinCeHEvy7+sWMmwf3aFsC_3AiP=P_HGvaBOaCgS@mail.gmail.com>
Subject: Re: Plans for the 0.22 Release
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0023547c8c0104ade90498050e3e

--0023547c8c0104ade90498050e3e
Content-Type: text/plain; charset=ISO-8859-1

This is great! Thanks, Nigel.

I haven't seen the feature freeze for 0.22 formally announced.
May be it worth confirming it now by setting the start day, as of yesterday?

--Konstantin

On Tue, Dec 21, 2010 at 11:24 AM, Nigel Daley <ndaley@mac.com> wrote:

> I think it's important we get more frequent releases back on track, so I'll
> volunteer to do it.  Its been a while since I was the release manager so
> I'll need help.
>
> Looks like in Jira we currently have:
>  26 MR blockers: http://tinyurl.com/23s5vzg
>  17 HDFS blockers: http://tinyurl.com/27w82vl
>  6 HADOOP blockers: http://tinyurl.com/2bmycl2
> Now that we are past feature freeze (i.e. branch creation), not all of
> these are truly blockers.  I'll start cleaning up the list and ping folks
> after the holidays.
>
> Cheers,
> Nige
>
> On Dec 16, 2010, at 3:10 PM, Ian Holsman wrote:
>
> >
> > On Dec 17, 2010, at 9:00 AM, Owen O'Malley wrote:
> >
> >>> Everyone who has discussed this patch has said it isn't critical to
> hadoop, and It's holding up everything else 0.22 is going to bring.
> >>
> >> I disagree that it isn't critical to Hadoop, but I'm not holding up
> 0.22. I'm just not volunteering to spend my time working on it, if it
> doesn't have the features that I think it needs.
> >>
> >> -- Owen
> >>
> >
> > That's a fair point.. and this is a volunteer effort.
> > Do we have anybody else who is willing to be the release manager for 22
> with 6685?
> >
>
>

--0023547c8c0104ade90498050e3e--

From general-return-2575-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 22 23:31:20 2010
Return-Path: <general-return-2575-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76790 invoked from network); 22 Dec 2010 23:31:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Dec 2010 23:31:20 -0000
Received: (qmail 18470 invoked by uid 500); 22 Dec 2010 23:31:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18423 invoked by uid 500); 22 Dec 2010 23:31:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18415 invoked by uid 99); 22 Dec 2010 23:31:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Dec 2010 23:31:18 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Dec 2010 23:31:12 +0000
Received: by fxm2 with SMTP id 2so5634991fxm.35
        for <general@hadoop.apache.org>; Wed, 22 Dec 2010 15:30:51 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        bh=imA9QrMy5kUTBeq2u5lK4S6Dw5hC+M/zi+6sAteDIsQ=;
        b=BBcPUgTH36sNRbG8Pz1btAvwurFNkLwOulANVLclKLF4XeQ7F9tRF0wObz0s7oG/+t
         RJkJ1lbNTCTXvTLWpwxWHsS2xHoBxPd+gPtZ70l0+NRiEYBjzMlYd7HEIPeb/SDWzEwq
         8d4/qYSb/f7ImGpPBP70oK0d3w0k8M7x5Wsj4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        b=JoTCgyglMxhUNyOqBmz0VDGi7eWEtyJf4GCJZcVcjgXDamEgBIrGWpmL5dLgj+kcqu
         DvilQWnzT6HlcCZX6X4ndglUJ90UgTheO2NzWjM0SxbLSorHcxZDblp1ExMlvV8GbQK3
         BHRTCFQrpD53HVx4ZQcdYAvqecz+Renvdyq7M=
MIME-Version: 1.0
Received: by 10.223.96.198 with SMTP id i6mr238233fan.10.1293060650966; Wed,
 22 Dec 2010 15:30:50 -0800 (PST)
Sender: saint.ack@gmail.com
Received: by 10.223.83.9 with HTTP; Wed, 22 Dec 2010 15:30:50 -0800 (PST)
Date: Wed, 22 Dec 2010 15:30:50 -0800
X-Google-Sender-Auth: gvYS9bVclohJtuG6LlZ7-1IkCQA
Message-ID: <AANLkTinMrd-acs=91xxw4MxHis8-RPSPtfhy7b4gu41O@mail.gmail.com>
Subject: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of
 branch-0.20-append branch?
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

I propose cutting a release from the tip of the branch-0.20-append
branch [1].  I suggest the release be called hadoop-0.20.0-append.  I
volunteer to run the release process. Are folks OK with this?

Here's some background.

The branch-0.20-append was forked from branch-0.20 a few months ago by
Dhruba to add an append/sync to 0.20.x era HDFS.  The added append
facility is made of the patches attached to HDFS-200 and then a bunch
of fixup patches done by Dhruba, Hairong, Nicolas, Todd, and others.
For a complete list of differences from the tip of the Hadoop
branch-0.20, see the CHANGE.txt file in branch-0.20-append [2].  The
HDFS-200 append/sync is not the same as the append/sync implementation
that is in hadoop 0.21.x and hadoop TRUNK.

The branch-0.20-append is a relatively small deviation from hadoop
0.20.x for those who want an append/sync in an (Apache) hadoop 0.20.x
[3].  Its for those unwilling to upgrade their clusters to hadoop
0.21.0 and for those who can't wait on the coming hadoop 0.22.0.  For
applications like HBase [4], an application that runs on HDFS and
"loses data" if no working append/sync, its critical that there is an
Apache release with a working append/sync.

A few of us have been playing with this branch with a while and it
seems to do the right thing.  Its fairly close to what FB runs
internally (correct me if I'm wrong in this last statement Dhruba).

Thanks,
St.Ack

1. http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20-append/
2. http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20-append/CHANGES.txt?view=markup
3. Cloudera's CDH3Beta2/3 already include an append/sync based off the
HDFS-200++ work.   There is no 'official' Apache hadoop 0.20.x with a
working append/sync.
4. http://hbase.apache.org

From general-return-2576-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 00:06:39 2010
Return-Path: <general-return-2576-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57431 invoked from network); 23 Dec 2010 00:06:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 00:06:38 -0000
Received: (qmail 53807 invoked by uid 500); 23 Dec 2010 00:06:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53769 invoked by uid 500); 23 Dec 2010 00:06:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53761 invoked by uid 99); 23 Dec 2010 00:06:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 00:06:37 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 00:06:28 +0000
Received: by vws18 with SMTP id 18so2222978vws.35
        for <general@hadoop.apache.org>; Wed, 22 Dec 2010 16:06:07 -0800 (PST)
Received: by 10.220.188.141 with SMTP id da13mr2057014vcb.76.1293062765490;
        Wed, 22 Dec 2010 16:06:05 -0800 (PST)
Received: from [10.172.0.49] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id w7sm1537330vch.44.2010.12.22.16.06.01
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Wed, 22 Dec 2010 16:06:04 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of branch-0.20-append branch?
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <AANLkTinMrd-acs=91xxw4MxHis8-RPSPtfhy7b4gu41O@mail.gmail.com>
Date: Thu, 23 Dec 2010 11:05:57 +1100
Content-Transfer-Encoding: quoted-printable
Message-Id: <7277DB0B-9B56-4B57-A574-F2764C11FC10@Holsman.NET>
References: <AANLkTinMrd-acs=91xxw4MxHis8-RPSPtfhy7b4gu41O@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

Hi St.Ack.

In general I'm opposed to such a thing.

There are already 5 Hadoop 20.x releases out there, I don't think there =
is a need for another. (personal opinion, not a veto or speaking as the =
chair)

Is there a reason why we couldn't create a hadoop 0.20.3 release that =
has this patch inside of it, as well as other fixes that have been =
applied since 0.20.2 (~26 patches)? Would this be too much effort for =
you to RM?..=20

I understand there is a large QA effort you would be taking on if you =
do.

does the Append/Sync semantics break/deviate too much from 0.20.2 ?

I really don't want to come to a^h^h^h^hget out of the situation where =
we have multiple releases of 0.20 each with a unique feature.


On Dec 23, 2010, at 10:30 AM, Stack wrote:

> I propose cutting a release from the tip of the branch-0.20-append
> branch [1].  I suggest the release be called hadoop-0.20.0-append.  I
> volunteer to run the release process. Are folks OK with this?
>=20
> Here's some background.
>=20
> The branch-0.20-append was forked from branch-0.20 a few months ago by
> Dhruba to add an append/sync to 0.20.x era HDFS.  The added append
> facility is made of the patches attached to HDFS-200 and then a bunch
> of fixup patches done by Dhruba, Hairong, Nicolas, Todd, and others.
> For a complete list of differences from the tip of the Hadoop
> branch-0.20, see the CHANGE.txt file in branch-0.20-append [2].  The
> HDFS-200 append/sync is not the same as the append/sync implementation
> that is in hadoop 0.21.x and hadoop TRUNK.
>=20
> The branch-0.20-append is a relatively small deviation from hadoop
> 0.20.x for those who want an append/sync in an (Apache) hadoop 0.20.x
> [3].  Its for those unwilling to upgrade their clusters to hadoop
> 0.21.0 and for those who can't wait on the coming hadoop 0.22.0.  For
> applications like HBase [4], an application that runs on HDFS and
> "loses data" if no working append/sync, its critical that there is an
> Apache release with a working append/sync.
>=20
> A few of us have been playing with this branch with a while and it
> seems to do the right thing.  Its fairly close to what FB runs
> internally (correct me if I'm wrong in this last statement Dhruba).
>=20
> Thanks,
> St.Ack
>=20
> 1. =
http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20-append/
> 2. =
http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20-append/CHA=
NGES.txt?view=3Dmarkup
> 3. Cloudera's CDH3Beta2/3 already include an append/sync based off the
> HDFS-200++ work.   There is no 'official' Apache hadoop 0.20.x with a
> working append/sync.
> 4. http://hbase.apache.org


From general-return-2577-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 00:33:49 2010
Return-Path: <general-return-2577-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70488 invoked from network); 23 Dec 2010 00:33:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 00:33:49 -0000
Received: (qmail 72453 invoked by uid 500); 23 Dec 2010 00:33:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72328 invoked by uid 500); 23 Dec 2010 00:33:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72320 invoked by uid 99); 23 Dec 2010 00:33:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 00:33:47 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 00:33:43 +0000
Received: by fxm2 with SMTP id 2so5669196fxm.35
        for <general@hadoop.apache.org>; Wed, 22 Dec 2010 16:33:22 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        bh=0jAw9WC0GYgza4DcxR4FttGwrZSFQ5VlY83PjD/nWnQ=;
        b=fw8IOAyjOQfeGwumItExNrNv364p3ktY5d0x/8fH+Dv2UglOiBK/ry/5lqpMlaUsws
         MpdR5Gpl5CzaLflpgHUkHuYs/QoQYWelpvKxchOYOFWfCfFR/LWQ2/n14W8INTEe4+Ev
         roaNGH63sMm5hAw8J+Gny/DuLzr1msD3f9Wj4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=ir4bNJVfSQImb+lI+iJGKTRlabBZcAIE3G/hYc4ZtWAiCQBrMWn9r5DrXkxtgCQtLb
         DhPXMiQdMWUFwpt3OGHoM0MOPAPY/mQski1n8QNdDamQV/iOpm5whS41RBySJXuoeVyj
         ZyODA6I4HbtkefwZ9GCMhxbvBijiEc0qlEYyI=
MIME-Version: 1.0
Received: by 10.223.122.146 with SMTP id l18mr3308711far.67.1293064402188;
 Wed, 22 Dec 2010 16:33:22 -0800 (PST)
Sender: saint.ack@gmail.com
Received: by 10.223.83.9 with HTTP; Wed, 22 Dec 2010 16:33:22 -0800 (PST)
In-Reply-To: <7277DB0B-9B56-4B57-A574-F2764C11FC10@Holsman.NET>
References: <AANLkTinMrd-acs=91xxw4MxHis8-RPSPtfhy7b4gu41O@mail.gmail.com>
	<7277DB0B-9B56-4B57-A574-F2764C11FC10@Holsman.NET>
Date: Wed, 22 Dec 2010 16:33:22 -0800
X-Google-Sender-Auth: 5bY_mIY4tdPi9UYF3RmdTPjIqMc
Message-ID: <AANLkTinMW7nvmapbkJwOGGNvaLNSrXSBr9c08JnBSHp4@mail.gmail.com>
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of
 branch-0.20-append branch?
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Dec 22, 2010 at 4:05 PM, Ian Holsman <hadoop@holsman.net> wrote:
> There are already 5 Hadoop 20.x releases out there, I don't think there is a need for another. (personal opinion, not a veto or speaking as the chair)
>

Are you counting other than Apache releases?  (I see only 4 here, two
of which probably should be removed:
http://www.gtlib.gatech.edu/pub/apache//hadoop/core/.)

> Is there a reason why we couldn't create a hadoop 0.20.3 release that has this patch inside of it, as well as other fixes that have been applied since 0.20.2 (~26 patches)? Would this be too much effort for you to RM?..
>

I'd like that but my sense is the general populace of hadoopers would
think the append/sync suite of patches destabilizing -- append/sync
has a long 'history' in hadoop -- and a violation of the general
principal that bug fixes only are added on a branch.


> I really don't want to come to a^h^h^h^hget out of the situation where we have multiple releases of 0.20 each with a unique feature.
>

Sure.  The notion has been broached before up on these lists -- e.g.
there was talk of a 0.20 Apache release that had security in it -- and
at the time folks seemed amenable.

Thanks for getting the discussion off the ground,
St.Ack

From general-return-2578-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 01:04:38 2010
Return-Path: <general-return-2578-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84912 invoked from network); 23 Dec 2010 01:04:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 01:04:38 -0000
Received: (qmail 94699 invoked by uid 500); 23 Dec 2010 01:04:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94553 invoked by uid 500); 23 Dec 2010 01:04:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94544 invoked by uid 99); 23 Dec 2010 01:04:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 01:04:34 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 01:04:29 +0000
Received: by vws18 with SMTP id 18so2238624vws.35
        for <general@hadoop.apache.org>; Wed, 22 Dec 2010 17:04:08 -0800 (PST)
Received: by 10.220.186.200 with SMTP id ct8mr1786540vcb.256.1293066247400;
        Wed, 22 Dec 2010 17:04:07 -0800 (PST)
Received: from [10.172.0.49] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id bq5sm1540948vcb.32.2010.12.22.17.04.03
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Wed, 22 Dec 2010 17:04:06 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of branch-0.20-append branch?
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <AANLkTinMW7nvmapbkJwOGGNvaLNSrXSBr9c08JnBSHp4@mail.gmail.com>
Date: Thu, 23 Dec 2010 12:03:59 +1100
Content-Transfer-Encoding: quoted-printable
Message-Id: <C475DC5D-7CAC-47E4-838B-AED9C8274E5B@Holsman.NET>
References: <AANLkTinMrd-acs=91xxw4MxHis8-RPSPtfhy7b4gu41O@mail.gmail.com> <7277DB0B-9B56-4B57-A574-F2764C11FC10@Holsman.NET> <AANLkTinMW7nvmapbkJwOGGNvaLNSrXSBr9c08JnBSHp4@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)


On Dec 23, 2010, at 11:33 AM, Stack wrote:

> On Wed, Dec 22, 2010 at 4:05 PM, Ian Holsman <hadoop@holsman.net> =
wrote:
>> There are already 5 Hadoop 20.x releases out there, I don't think =
there is a need for another. (personal opinion, not a veto or speaking =
as the chair)
>>=20
>=20
> Are you counting other than Apache releases?  (I see only 4 here, two
> of which probably should be removed:
> http://www.gtlib.gatech.edu/pub/apache//hadoop/core/.)


yes.. I was referring to the external companies who have decided to =
release their own version, for their own business purposes. (please =
don't take that as a negative).

>=20
>> Is there a reason why we couldn't create a hadoop 0.20.3 release that =
has this patch inside of it, as well as other fixes that have been =
applied since 0.20.2 (~26 patches)? Would this be too much effort for =
you to RM?..
>>=20
>=20
> I'd like that but my sense is the general populace of hadoopers would
> think the append/sync suite of patches destabilizing -- append/sync
> has a long 'history' in hadoop -- and a violation of the general
> principal that bug fixes only are added on a branch.

I'm open with adding it, as lack of append/sync could be seen as a bug =
to some. (yes i'm playing with words)
>=20
>=20
>> I really don't want to come to a^h^h^h^hget out of the situation =
where we have multiple releases of 0.20 each with a unique feature.
>>=20
>=20
> Sure.  The notion has been broached before up on these lists -- e.g.
> there was talk of a 0.20 Apache release that had security in it -- and
> at the time folks seemed amenable.

I think that approach encourages groups of individuals/companies to =
huddle up together to build large features without taking the larger =
group into account and then 'drop' the feature off and wait for others =
to thank them & port it to their releases. We then become multiple =
communities instead of a single one.=20

We will end up with Apache+Security release vs Apache+Append release vs =
Apache+Avatar release, with various bug-fixes sprinkled into each.
And I'm not sure which release Pig or Hbase would target to develop =
against.

Thats why I think we should go to 0.22 ASAP and get companies to build =
their new features on trunk against that.

>=20
> Thanks for getting the discussion off the ground,
> St.Ack


From general-return-2579-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 01:30:46 2010
Return-Path: <general-return-2579-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 3400 invoked from network); 23 Dec 2010 01:30:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 01:30:46 -0000
Received: (qmail 14616 invoked by uid 500); 23 Dec 2010 01:30:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14565 invoked by uid 500); 23 Dec 2010 01:30:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14557 invoked by uid 99); 23 Dec 2010 01:30:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 01:30:44 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of li.j2ee@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 01:30:39 +0000
Received: by bwz8 with SMTP id 8so6042872bwz.35
        for <general@hadoop.apache.org>; Wed, 22 Dec 2010 17:30:18 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=FHXZPPydYZmlzo3TfpVOy8m/ycgBhxHiIASOqyVo0dg=;
        b=pKMZVBoyouTNmMMBitF69Y3TnEeRw7lYg+lKmmEUBQS5dBF6pe68ceTCwnCzxrgg3q
         G3Q8tv4ljz24vGW9/35HHscUzszueytOb2+rlp4et9Q2hhW182mVVeQAq4bGzgyqf79j
         Ldrxxlh0iUPyEoLMcSVHg6izfMuk7lt3Ynk2w=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Z7WzM8Wl0Ll8qfceOchdafDLlryRtAd61muoLb9a8Fc+sop0qBfgkDX7XS04/pdQYl
         OumCOTZ3tcePqhia9qK317EW73C/OpI9m39sAzLJpRNQP54jA2ILcU1qsQsLtQo7q3ug
         ptN+P65VUsN8nu+DXsTFlyRzfPW5vtDaETD/A=
MIME-Version: 1.0
Received: by 10.204.102.146 with SMTP id g18mr2605390bko.163.1293067817964;
 Wed, 22 Dec 2010 17:30:17 -0800 (PST)
Received: by 10.204.138.67 with HTTP; Wed, 22 Dec 2010 17:30:17 -0800 (PST)
In-Reply-To: <AANLkTike7k5nqViAsdyC_mR6LOBVrrFFqsyGJo=-W6GQ@mail.gmail.com>
References: <20101222160311.7ffb9b6d@laptop>
	<AANLkTike7k5nqViAsdyC_mR6LOBVrrFFqsyGJo=-W6GQ@mail.gmail.com>
Date: Thu, 23 Dec 2010 09:30:17 +0800
Message-ID: <AANLkTi=9_-CPn+GON+YqLra1sjDX3-i-tqZ4rv39rTfT@mail.gmail.com>
Subject: Re: namenode doesn't start after reboot
From: li ping <li.j2ee@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d6456ff5d387049809cfe1

--0016e6d6456ff5d387049809cfe1
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

It seems the exception occurs during NameNode loads the editlog.
make sure the editlog file exists. or you can debug the application to see
what's wrong.

On Thu, Dec 23, 2010 at 2:01 AM, daniel sikar <dsikar@gmail.com> wrote:

> I can't help but with hindsight - it's advisable to snapshot your
> namenodes as HDFS dies with them.
>
> On 22 December 2010 15:03, Bjoern Schiessle <bjoern@schiessle.org> wrote:
> > Hi,
> >
> > After a Kernel update and a reboot the namenode doesn't start. I run th=
e
> > Cloudera cdh3 Hadoop distribution. I have already searched for a
> solution.
> > It looks like I'm not the only one with such a problem. Sadly I could
> only
> > find descriptions of similar problems, but no solutions...
> >
> > This is the error message from the namenode log file:
> >
> >
> > 2010-12-22 16:13:04,830 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
> > /************************************************************
> > STARTUP_MSG: Starting NameNode
> > STARTUP_MSG:   host =3D pcube/129.69.216.24
> > STARTUP_MSG:   args =3D []
> > STARTUP_MSG:   version =3D 0.20.2+737
> > STARTUP_MSG:   build =3D  -r 98c55c28258aa6f42250569bd7fa431ac657bdbd;
> compiled by 'root' on Mon Oct 11 17:21:30 UTC 2010
> > ************************************************************/
> > 2010-12-22 16:13:05,001 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=3DNameNode, sessionId=3Dnull
> > 2010-12-22 16:13:05,007 INFO
> org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics: Initializ=
ing
> NameNodeMeterics using context
> object:org.apache.hadoop.metrics.spi.NullContext
> > 2010-12-22 16:13:05,036 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=3Dhdfs
> > 2010-12-22 16:13:05,036 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=3Dsupergr=
oup
> > 2010-12-22 16:13:05,036 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> isPermissionEnabled=3Dfalse
> > 2010-12-22 16:13:05,040 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> isAccessTokenEnabled=3Dfalse accessKeyUpdateInterval=3D0 min(s),
> accessTokenLifetime=3D0 min(s)
> > 2010-12-22 16:13:05,335 INFO
> org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics:
> Initializing FSNamesystemMetrics using context
> object:org.apache.hadoop.metrics.spi.NullContext
> > 2010-12-22 16:13:05,336 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
> FSNamesystemStatusMBean
> > 2010-12-22 16:13:05,361 INFO
> org.apache.hadoop.hdfs.server.common.Storage: Number of files =3D 72
> > 2010-12-22 16:13:05,374 INFO
> org.apache.hadoop.hdfs.server.common.Storage: Number of files under
> construction =3D 3
> > 2010-12-22 16:13:05,375 INFO
> org.apache.hadoop.hdfs.server.common.Storage: Image file of size 8822 loa=
ded
> in 0 seconds.
> > 2010-12-22 16:13:05,377 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode:
> java.lang.NullPointerException
> >        at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.addChild(FSDirectory.j=
ava:1088)
> >        at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.addChild(FSDirectory.j=
ava:1100)
> >        at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.addNode(FSDirectory.ja=
va:1003)
> >        at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedAddFile(FSD=
irectory.java:206)
> >        at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.ja=
va:637)
> >        at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:1=
034)
> >        at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:8=
45)
> >        at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSIm=
age.java:379)
> >        at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirector=
y.java:99)
> >        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesyst=
em.java:343)
> >        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.j=
ava:317)
> >        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:=
214)
> >        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:394)
> >        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.j=
ava:1148)
> >        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1157)
> >
> > 2010-12-22 16:13:05,377 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> > /************************************************************
> > SHUTDOWN_MSG: Shutting down NameNode at pcube/129.69.216.24
> > ************************************************************/
> >
> > Any idea what could be wrong and how I can get my namenode up running
> again?
> >
> > Thanks a lot!
> > Bj=C3=B6rn
> >
>



--=20
-----=E6=9D=8E=E5=B9=B3

--0016e6d6456ff5d387049809cfe1--

From general-return-2580-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 01:47:13 2010
Return-Path: <general-return-2580-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9176 invoked from network); 23 Dec 2010 01:47:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 01:47:13 -0000
Received: (qmail 30241 invoked by uid 500); 23 Dec 2010 01:47:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30128 invoked by uid 500); 23 Dec 2010 01:47:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30120 invoked by uid 99); 23 Dec 2010 01:47:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 01:47:11 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 01:47:05 +0000
Received: by fxm2 with SMTP id 2so5707604fxm.35
        for <general@hadoop.apache.org>; Wed, 22 Dec 2010 17:46:44 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=3iqluXAPJ7Tq6CCcNkHaoS7R6Sg+ckZOMOa3q9vM7Dw=;
        b=ncKBpmOPUMryyyTyIig5vODGq9IlCy0BK2oNkRFx5lwrChx0XTFvpOHiOTLOImapS5
         4l6/d40mc4w/ELkFQmHIZydpEMg1Ox/mnInsLY+YMNa8yCG002Z2dOFteqdCDZJDexVG
         D8QV8SZ5yRacf+cCBiEwMI5r1d09CfnMctp4I=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=rpsfgi4tXmIvjmLxM/hcV5o/PL+S/odKxuduAOROUE1F91PnGh9Ihwez1iU9FW4lrr
         ArDQ2VfvfNXTt9QiQUyfQgWL5Oq0je8NumM1laTfW4G1PUHPJ5tOHC1cnppyIo94LRNu
         siZVN9qdx4J0LEZyLpmnU0F+/bibQ6EYh0heg=
MIME-Version: 1.0
Received: by 10.223.72.14 with SMTP id k14mr802414faj.45.1293068803364; Wed,
 22 Dec 2010 17:46:43 -0800 (PST)
Received: by 10.223.83.200 with HTTP; Wed, 22 Dec 2010 17:46:43 -0800 (PST)
In-Reply-To: <C475DC5D-7CAC-47E4-838B-AED9C8274E5B@Holsman.NET>
References: <AANLkTinMrd-acs=91xxw4MxHis8-RPSPtfhy7b4gu41O@mail.gmail.com>
	<7277DB0B-9B56-4B57-A574-F2764C11FC10@Holsman.NET>
	<AANLkTinMW7nvmapbkJwOGGNvaLNSrXSBr9c08JnBSHp4@mail.gmail.com>
	<C475DC5D-7CAC-47E4-838B-AED9C8274E5B@Holsman.NET>
Date: Wed, 22 Dec 2010 17:46:43 -0800
Message-ID: <AANLkTi=FgorNPmkjoP_46vknF9Mh9gykHqKYQm1v2aot@mail.gmail.com>
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of
 branch-0.20-append branch?
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf3054a203b1d70004980a0a93

--20cf3054a203b1d70004980a0a93
Content-Type: text/plain; charset=ISO-8859-1

> Thats why I think we should go to 0.22 ASAP and get companies to build
their new features on trunk against that.

There was a thread in Nov - 'Caution using Hadoop 0.21'
It would be helpful to see response to 0.22


> >
> > Thanks for getting the discussion off the ground,
> > St.Ack
>
>

--20cf3054a203b1d70004980a0a93--

From general-return-2581-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 01:47:38 2010
Return-Path: <general-return-2581-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9422 invoked from network); 23 Dec 2010 01:47:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 01:47:37 -0000
Received: (qmail 31924 invoked by uid 500); 23 Dec 2010 01:47:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31880 invoked by uid 500); 23 Dec 2010 01:47:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31870 invoked by uid 99); 23 Dec 2010 01:47:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 01:47:36 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [98.139.52.219] (HELO nm22.bullet.mail.ac4.yahoo.com) (98.139.52.219)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 23 Dec 2010 01:47:28 +0000
Received: from [98.139.52.195] by nm22.bullet.mail.ac4.yahoo.com with NNFMP; 23 Dec 2010 01:47:07 -0000
Received: from [98.139.52.158] by tm8.bullet.mail.ac4.yahoo.com with NNFMP; 23 Dec 2010 01:47:07 -0000
Received: from [127.0.0.1] by omp1041.mail.ac4.yahoo.com with NNFMP; 23 Dec 2010 01:47:07 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 236839.75870.bm@omp1041.mail.ac4.yahoo.com
Received: (qmail 99646 invoked by uid 60001); 23 Dec 2010 01:47:07 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1293068827; bh=b3KHOrGnZ+wYUYHZ3FxBRzUz/RDQ+CiHx0oPTNbbTes=; h=Message-ID:X-YMail-OSG:Received:X-RocketYMMF:X-Mailer:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=QD7ZlGNAbLjZC4H6ohEyoaVXdVhOL3fgERc0qfZx7hILRmQf1xhKUVa7CFzxIc2Y4oOwnAYQTc7eQYHeN1nm3sFHEdw335bL0Dltg/pk2z2DizyStdFqNm7MQLV8ii/mduyOikNIvPeCwzMno/jfea5MURGtmQVfErmW+Y0Vx0I=
Message-ID: <70915.86754.qm@web65513.mail.ac4.yahoo.com>
X-YMail-OSG: 3W9NZLkVM1kwPm8ayxSj845GAohnkPcUtuQb6WYQaMUF7Ih
 agUaLS4s3ee6Q2_lz15bnFrxIULeKV6NKp1fYfyxDYeRh86HAlzOaPSRRZoD
 Bo4yJ3YX37_yX3CyA9ueSsTSCuKAnUTw.FTD9u5idfumVDcNuL7WKJuFr.QU
 PPbIuNXXlAjrvuAPPVRVTmwW4WcuFd8KZAjrlA3BLLB.7R_sReTFPQrt4_lX
 h88j_sKCK8LGf2_GJK9pd2zEqe4IVyP8EhHgCYwoxf9YlupbI8.HOq7mKPo_
 IKTZnY4l8LAoZ9_f5lpvlxcJzkneS7l25eRxwav1sLhQIPpxg5un1UqYkASk
 rzTiEuBKF2.hWti4NX6rmFBHQTORfX6I9G7AnuQGpBKG850xAfBjTBJl3RdE
 jgv7cpJFzw6Is
Received: from [69.111.79.82] by web65513.mail.ac4.yahoo.com via HTTP; Wed, 22 Dec 2010 17:47:06 PST
X-RocketYMMF: apurtell
X-Mailer: YahooMailClassic/11.4.20 YahooMailWebService/0.8.107.285259
Date: Wed, 22 Dec 2010 17:47:06 -0800 (PST)
From: Andrew Purtell <apurtell@apache.org>
Reply-To: apurtell@apache.org
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of branch-0.20-append branch?
To: general@hadoop.apache.org
In-Reply-To: <C475DC5D-7CAC-47E4-838B-AED9C8274E5B@Holsman.NET>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

I'm on the HBase PMC.=0A=0A> We will end up with Apache+Security release vs=
=0A> Apache+Append release vs Apache+Avatar release,=0A=0AThe current situa=
tion is pretty close to this. =0A=0AHBase has no suitable binary ASF Hadoop=
 release to work against, currently. Vanilla version 0.20 does not have syn=
c/append support. We recommend users adopt Cloudera's CDH3 beta 2, or compi=
le the 0.20-append branch from source. Version 0.21 is marked as unstable, =
was not tested at scale by Yahoo (unlike 0.20), and has been panned by many=
 would be adopters, if the various tweets and blog posts I have seen in tha=
t regard are any indication.=0A=0A> Thats why I think we should go to 0.22 =
ASAP and get=0A> companies to build their new features on trunk against=0A>=
 that.=0A=0AIf Hadoop 0.22 is not vetted at high scale as was 0.20 -- this =
is the current situation with 0.21 -- then I fear the current situation wil=
l not change and HBase will still to refer would be users to a non-ASF rele=
ase or a source-only branch. =0A=0ABest regards,=0A=0A    - Andy=0A=0AProbl=
ems worthy of attack prove their worth by hitting back.=0A  - Piet Hein (vi=
a Tom White)=0A=0A=0A--- On Wed, 12/22/10, Ian Holsman <hadoop@holsman.net>=
 wrote:=0A=0A> From: Ian Holsman <hadoop@holsman.net>=0A> Subject: Re: DISC=
USSION: Cut a hadoop-0.20.0-append release from the tip of branch-0.20-appe=
nd branch?=0A> To: general@hadoop.apache.org=0A> Date: Wednesday, December =
22, 2010, 5:03 PM=0A> =0A> On Dec 23, 2010, at 11:33 AM, Stack wrote:=0A> =
=0A> > On Wed, Dec 22, 2010 at 4:05 PM, Ian Holsman <hadoop@holsman.net>=0A=
> wrote:=0A> >> There are already 5 Hadoop 20.x releases out=0A> there, I d=
on't think there is a need for another. (personal=0A> opinion, not a veto o=
r speaking as the chair)=0A> >> =0A> > =0A> > Are you counting other than A=
pache releases?=A0 (I=0A> see only 4 here, two=0A> > of which probably shou=
ld be removed:=0A> > http://www.gtlib.gatech.edu/pub/apache//hadoop/core/.)=
=0A> =0A> =0A> yes.. I was referring to the external companies who have=0A>=
 decided to release their own version, for their own business=0A> purposes.=
 (please don't take that as a negative).=0A> =0A> > =0A> >> Is there a reas=
on why we couldn't create a hadoop=0A> 0.20.3 release that has this patch i=
nside of it, as well as=0A> other fixes that have been applied since 0.20.2=
 (~26=0A> patches)? Would this be too much effort for you to RM?..=0A> >> =
=0A> > =0A> > I'd like that but my sense is the general populace of=0A> had=
oopers would=0A> > think the append/sync suite of patches destabilizing=0A>=
 -- append/sync=0A> > has a long 'history' in hadoop -- and a violation of=
=0A> the general=0A> > principal that bug fixes only are added on a branch.=
=0A> =0A> I'm open with adding it, as lack of append/sync could be=0A> seen=
 as a bug to some. (yes i'm playing with words)=0A> > =0A> > =0A> >> I real=
ly don't want to come to a^h^h^h^hget out of=0A> the situation where we hav=
e multiple releases of 0.20 each=0A> with a unique feature.=0A> >> =0A> > =
=0A> > Sure.=A0 The notion has been broached before up on=0A> these lists -=
- e.g.=0A> > there was talk of a 0.20 Apache release that had=0A> security =
in it -- and=0A> > at the time folks seemed amenable.=0A> =0A> I think that=
 approach encourages groups of=0A> individuals/companies to huddle up toget=
her to build large=0A> features without taking the larger group into accoun=
t and=0A> then 'drop' the feature off and wait for others to thank=0A> them=
 & port it to their releases. We then become=0A> multiple communities inste=
ad of a single one. =0A> =0A> We will end up with Apache+Security release v=
s=0A> Apache+Append release vs Apache+Avatar release, with various=0A> bug-=
fixes sprinkled into each.=0A> And I'm not sure which release Pig or Hbase =
would target to=0A> develop against.=0A> =0A> Thats why I think we should g=
o to 0.22 ASAP and get=0A> companies to build their new features on trunk a=
gainst=0A> that.=0A> =0A> > =0A> > Thanks for getting the discussion off th=
e ground,=0A> > St.Ack=0A> =0A> =0A=0A=0A      

From general-return-2582-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 03:32:05 2010
Return-Path: <general-return-2582-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 48734 invoked from network); 23 Dec 2010 03:32:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 03:32:05 -0000
Received: (qmail 86406 invoked by uid 500); 23 Dec 2010 03:32:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86209 invoked by uid 500); 23 Dec 2010 03:32:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86196 invoked by uid 99); 23 Dec 2010 03:32:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 03:32:02 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 03:31:55 +0000
Received: by vws18 with SMTP id 18so2274169vws.35
        for <general@hadoop.apache.org>; Wed, 22 Dec 2010 19:31:34 -0800 (PST)
Received: by 10.220.98.208 with SMTP id r16mr2013865vcn.135.1293075092599;
        Wed, 22 Dec 2010 19:31:32 -0800 (PST)
Received: from [10.172.0.49] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id s9sm1567945vcr.46.2010.12.22.19.31.26
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Wed, 22 Dec 2010 19:31:32 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of branch-0.20-append branch?
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <70915.86754.qm@web65513.mail.ac4.yahoo.com>
Date: Thu, 23 Dec 2010 14:31:22 +1100
Content-Transfer-Encoding: quoted-printable
Message-Id: <772F419F-1B2F-4BDA-870A-EB66FC4218DB@Holsman.NET>
References: <70915.86754.qm@web65513.mail.ac4.yahoo.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org


On Dec 23, 2010, at 12:47 PM, Andrew Purtell wrote:

> I'm on the HBase PMC.
>=20
>> We will end up with Apache+Security release vs
>> Apache+Append release vs Apache+Avatar release,
>=20
> The current situation is pretty close to this.=20

agreed. and I would like to make it better
>=20
> HBase has no suitable binary ASF Hadoop release to work against, =
currently. Vanilla version 0.20 does not have sync/append support. We =
recommend users adopt Cloudera's CDH3 beta 2, or compile the 0.20-append =
branch from source. Version 0.21 is marked as unstable, was not tested =
at scale by Yahoo (unlike 0.20), and has been panned by many would be =
adopters, if the various tweets and blog posts I have seen in that =
regard are any indication.
>=20

I'd like to make two points here:

1. There is no substitute for your own QA team.

You can never rely on a single company to do your testing for you. While =
it was great Yahoo tested the initial releases, you can see by their own =
distribution that what they were/are running is different to what other =
people are running.=20

It is better for people to not blindly trust that just because company X =
is claiming that the are running something that it will work for them, =
and we cannot just rely on a individual or single company to provide =
that service going forward. Communities don't work that way. And =
reliance on a single company to provide your core infrastructure for =
gratis isn't really going to end up well either.
=20
Saying that, we are very lucky that Yahoo has chosen to openly =
contribute as much as they have, and I look forward to them and other =
large installation's contributions and participation going forward.

2. Hadoop is only one piece of the puzzle for most installations.

One of the other issues with 0.21 (and with future releases going =
forward) is that 3rd parties did not port/upgrade their software to run =
with our new APIs. Without major software like Hbase, Pig, Hive being =
able to run on the platform, major installations won't even bother =
looking at it.

I don't expect people to immediately upgrade to 0.22 when we release it. =
I expect it will take a good 3-6 months until people have the software =
they run available on it, and possibly a point release with some of =
problems people have found in their own testing fixed in our and other =
software.=20

Like I said, I don't mind getting 0.20.3 released with the append/sync =
patch applied to it (with the other 20 or so patches), but I don't think =
the Hadoop team is large enough to support all the different releases =
as-is, let alone another one.


--Ian


>> Thats why I think we should go to 0.22 ASAP and get
>> companies to build their new features on trunk against
>> that.
>=20
> If Hadoop 0.22 is not vetted at high scale as was 0.20 -- this is the =
current situation with 0.21 -- then I fear the current situation will =
not change and HBase will still to refer would be users to a non-ASF =
release or a source-only branch.=20
>=20
> Best regards,
>=20
>    - Andy
>=20
> Problems worthy of attack prove their worth by hitting back.
>  - Piet Hein (via Tom White)
>=20
>=20
> --- On Wed, 12/22/10, Ian Holsman <hadoop@holsman.net> wrote:
>=20
>> From: Ian Holsman <hadoop@holsman.net>
>> Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the =
tip of branch-0.20-append branch?
>> To: general@hadoop.apache.org
>> Date: Wednesday, December 22, 2010, 5:03 PM
>>=20
>> On Dec 23, 2010, at 11:33 AM, Stack wrote:
>>=20
>>> On Wed, Dec 22, 2010 at 4:05 PM, Ian Holsman <hadoop@holsman.net>
>> wrote:
>>>> There are already 5 Hadoop 20.x releases out
>> there, I don't think there is a need for another. (personal
>> opinion, not a veto or speaking as the chair)
>>>>=20
>>>=20
>>> Are you counting other than Apache releases?  (I
>> see only 4 here, two
>>> of which probably should be removed:
>>> http://www.gtlib.gatech.edu/pub/apache//hadoop/core/.)
>>=20
>>=20
>> yes.. I was referring to the external companies who have
>> decided to release their own version, for their own business
>> purposes. (please don't take that as a negative).
>>=20
>>>=20
>>>> Is there a reason why we couldn't create a hadoop
>> 0.20.3 release that has this patch inside of it, as well as
>> other fixes that have been applied since 0.20.2 (~26
>> patches)? Would this be too much effort for you to RM?..
>>>>=20
>>>=20
>>> I'd like that but my sense is the general populace of
>> hadoopers would
>>> think the append/sync suite of patches destabilizing
>> -- append/sync
>>> has a long 'history' in hadoop -- and a violation of
>> the general
>>> principal that bug fixes only are added on a branch.
>>=20
>> I'm open with adding it, as lack of append/sync could be
>> seen as a bug to some. (yes i'm playing with words)
>>>=20
>>>=20
>>>> I really don't want to come to a^h^h^h^hget out of
>> the situation where we have multiple releases of 0.20 each
>> with a unique feature.
>>>>=20
>>>=20
>>> Sure.  The notion has been broached before up on
>> these lists -- e.g.
>>> there was talk of a 0.20 Apache release that had
>> security in it -- and
>>> at the time folks seemed amenable.
>>=20
>> I think that approach encourages groups of
>> individuals/companies to huddle up together to build large
>> features without taking the larger group into account and
>> then 'drop' the feature off and wait for others to thank
>> them & port it to their releases. We then become
>> multiple communities instead of a single one.=20
>>=20
>> We will end up with Apache+Security release vs
>> Apache+Append release vs Apache+Avatar release, with various
>> bug-fixes sprinkled into each.
>> And I'm not sure which release Pig or Hbase would target to
>> develop against.
>>=20
>> Thats why I think we should go to 0.22 ASAP and get
>> companies to build their new features on trunk against
>> that.
>>=20
>>>=20
>>> Thanks for getting the discussion off the ground,
>>> St.Ack
>>=20
>>=20
>=20
>=20
>=20


From general-return-2583-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 05:32:21 2010
Return-Path: <general-return-2583-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87985 invoked from network); 23 Dec 2010 05:32:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 05:32:21 -0000
Received: (qmail 49683 invoked by uid 500); 23 Dec 2010 05:32:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49347 invoked by uid 500); 23 Dec 2010 05:32:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49333 invoked by uid 99); 23 Dec 2010 05:32:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 05:32:18 +0000
X-ASF-Spam-Status: No, hits=-1997.8 required=10.0
	tests=ALL_TRUSTED,HTML_MESSAGE,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 23 Dec 2010 05:32:16 +0000
Received: (qmail 87934 invoked by uid 99); 23 Dec 2010 05:31:54 -0000
Received: from localhost.apache.org (HELO mail-bw0-f48.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 05:31:54 +0000
Received: by bwz8 with SMTP id 8so6157892bwz.35
        for <general@hadoop.apache.org>; Wed, 22 Dec 2010 21:31:52 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.122.17 with SMTP id j17mr4203940bkr.4.1293082312538; Wed,
 22 Dec 2010 21:31:52 -0800 (PST)
Received: by 10.204.76.14 with HTTP; Wed, 22 Dec 2010 21:31:52 -0800 (PST)
In-Reply-To: <7277DB0B-9B56-4B57-A574-F2764C11FC10@Holsman.NET>
References: <AANLkTinMrd-acs=91xxw4MxHis8-RPSPtfhy7b4gu41O@mail.gmail.com>
	<7277DB0B-9B56-4B57-A574-F2764C11FC10@Holsman.NET>
Date: Wed, 22 Dec 2010 21:31:52 -0800
Message-ID: <AANLkTinV0Q70ZZwrmZgdTgbANA4E3gfG3twu=GVakgef@mail.gmail.com>
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of
 branch-0.20-append branch?
From: "Owen O'Malley" <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6de04bfe77afc04980d2fec
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6de04bfe77afc04980d2fec
Content-Type: text/plain; charset=UTF-8

On Wed, Dec 22, 2010 at 4:05 PM, Ian Holsman <hadoop@holsman.net> wrote:

> Hi St.Ack.
>
> In general I'm opposed to such a thing.
>
> There are already 5 Hadoop 20.x releases out there, I don't think there is
> a need for another.
>

Stack is trying to get an Apache version of Hadoop that solves his problem.
He's been asking for it for a year now. If the 20-append branch is stable,
we should release it.


>
> Is there a reason why we couldn't create a hadoop 0.20.3 release that has
> this patch inside of it, as well as other fixes that have been applied since
> 0.20.2 (~26 patches)?


It is of course possible, but no one has done the work to build the version
and test it out. As to what to call it, I'm a little hesitant to call it
0.20.3 at this point, since the last time I asked it was considered a fairly
risky change. If it goes badly, it would mean an incompatible revert onto a
branch that has been stable for 1.5 years. I'd much rather call it
0.20-append.0 and use 0.20.3 for straight bugfixes.

-- Owen

--0016e6de04bfe77afc04980d2fec--

From general-return-2584-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 06:16:29 2010
Return-Path: <general-return-2584-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 3842 invoked from network); 23 Dec 2010 06:16:29 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 06:16:29 -0000
Received: (qmail 76579 invoked by uid 500); 23 Dec 2010 06:16:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76293 invoked by uid 500); 23 Dec 2010 06:16:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76284 invoked by uid 99); 23 Dec 2010 06:16:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 06:16:26 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 06:16:20 +0000
Received: by fxm2 with SMTP id 2so5831390fxm.35
        for <general@hadoop.apache.org>; Wed, 22 Dec 2010 22:16:00 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:content-type:content-transfer-encoding;
        bh=RLl0cY8olPKnJudpcgBTWxu/naJakzMk2sPtIYHsPVw=;
        b=o0ey5fSTVaKVHBqfDbrtAj9UKv+B3v5fzHBA+NB0gzPG/CQd3W1IQoimms9fTtwY/9
         Riw3Xu6p6oc6jIm2d8Ub3OeXeWJ0qFrhyUh6zvh2JdnFWr3sloT+KPtauFVrlAPO/jJu
         Huj06ocF7zUtsPdh7kJ+OVRGkMjNXSjZ/HhBg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=nbT+zzzMHQ5+oUYq1oBEifwaMRgPN8gQTlqUwCxLNoFpyZJBlw+eD/5kqtibSc7/Mt
         BU7254uVC3Gaa+zIo4OmaR/gFmhQiz8/O+dDaHTccz4T8CkrovRLmZjdVaqTnotzggVn
         lUTl1FB2C/5Od8yZHt3s50zWHyIHLi4k4h0HY=
MIME-Version: 1.0
Received: by 10.223.125.196 with SMTP id z4mr3814747far.124.1293084960499;
 Wed, 22 Dec 2010 22:16:00 -0800 (PST)
Sender: saint.ack@gmail.com
Received: by 10.223.83.9 with HTTP; Wed, 22 Dec 2010 22:16:00 -0800 (PST)
In-Reply-To: <C475DC5D-7CAC-47E4-838B-AED9C8274E5B@Holsman.NET>
References: <AANLkTinMrd-acs=91xxw4MxHis8-RPSPtfhy7b4gu41O@mail.gmail.com>
	<7277DB0B-9B56-4B57-A574-F2764C11FC10@Holsman.NET>
	<AANLkTinMW7nvmapbkJwOGGNvaLNSrXSBr9c08JnBSHp4@mail.gmail.com>
	<C475DC5D-7CAC-47E4-838B-AED9C8274E5B@Holsman.NET>
Date: Wed, 22 Dec 2010 22:16:00 -0800
X-Google-Sender-Auth: vsVNaYLZGUfR28ZI8DlUiZbjmMs
Message-ID: <AANLkTinnP2JB_Dqpxu3+3izOfG8AEB1t7zw00Ow9xdC8@mail.gmail.com>
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of
 branch-0.20-append branch?
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Wed, Dec 22, 2010 at 5:03 PM, Ian Holsman <hadoop@holsman.net> wrote:>
>> Are you counting other than Apache releases? =A0(I see only 4 here, two
>> of which probably should be removed:
>> http://www.gtlib.gatech.edu/pub/apache//hadoop/core/.)
>
>
> yes.. I was referring to the external companies who have decided to relea=
se their own version, for their own business purposes. (please don't take t=
hat as a negative).
>

Oh.  I was not counting those at all.

Currently, over in HBase we tell users build and 'trust' your own
Hadoop binary from the tip of what to them probably looks like some
random Hadoop branch OR go get the slick Cloudera pre-builts since
Cloudera's CDH3s have append/sync.  By offering to release a
hadoop-0.20.0-append, I was just trying to make some remiss for a
gaping hole in the Apache Hadoop offering.


>>> Is there a reason why we couldn't create a hadoop 0.20.3 release that h=
as this patch inside of it, as well as other fixes that have been applied s=
ince 0.20.2 (~26 patches)? Would this be too much effort for you to RM?..
>>>
>> ...
>
> I'm open with adding it, as lack of append/sync could be seen as a bug to=
 some. (yes i'm playing with words)

My guess is that few would see it the way you do.  Append/sync has had
a long torturous history.  HADOOP-1700, the original append issue, was
originally opened in August 2007.  There have been two
implementations.  The one in branch-0.20-append is the 'deprecated'
implementation; i.e. its not the append that is in Hadoop TRUNK
(though IIUC the 'deprecated' append runs on the largest 'known' HDFS
cluster).  At least once, append was part of a release and then pulled
because it was 'destabilizing'.  It might be hard getting such a
storied, scarred feature in as a 'bug fix'.  If it did go in, the
append/sync is of such a reputation that it might sully the current
good standing hadoop 0.20 branch releases hold.

That said, I'm cavalier and if others are game, I'd be up for running
a 0.20.3 release that included it.

> Thats why I think we should go to 0.22 ASAP and get companies to build th=
eir new features on trunk against that.
>

Waiting on 0.22 and its adoption is not going to work for HBase.  The
HBase project would be long dead if waiting on 0.22 were the only
option available to us.  In fact we'd be dead already if it wasn't for
the lifeline thrown us by the folks who hooked us up with
branch-0.20-append.

St.Ack

From general-return-2585-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 06:25:20 2010
Return-Path: <general-return-2585-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4463 invoked from network); 23 Dec 2010 06:25:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 06:25:19 -0000
Received: (qmail 80860 invoked by uid 500); 23 Dec 2010 06:25:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80722 invoked by uid 500); 23 Dec 2010 06:25:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80714 invoked by uid 99); 23 Dec 2010 06:25:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 06:25:16 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=MIME_QP_LONG_LINE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.161.176] (HELO mail-gx0-f176.google.com) (209.85.161.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 06:25:10 +0000
Received: by gxk4 with SMTP id 4so2726756gxk.35
        for <general@hadoop.apache.org>; Wed, 22 Dec 2010 22:24:49 -0800 (PST)
Received: by 10.150.52.9 with SMTP id z9mr11657350ybz.335.1293085488046;
        Wed, 22 Dec 2010 22:24:48 -0800 (PST)
Received: from [49.194.41.124] ([49.194.41.124])
        by mx.google.com with ESMTPS id r24sm9519004yba.18.2010.12.22.22.24.45
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Wed, 22 Dec 2010 22:24:47 -0800 (PST)
References: <AANLkTinMrd-acs=91xxw4MxHis8-RPSPtfhy7b4gu41O@mail.gmail.com> <7277DB0B-9B56-4B57-A574-F2764C11FC10@Holsman.NET> <AANLkTinMW7nvmapbkJwOGGNvaLNSrXSBr9c08JnBSHp4@mail.gmail.com> <C475DC5D-7CAC-47E4-838B-AED9C8274E5B@Holsman.NET> <AANLkTinnP2JB_Dqpxu3+3izOfG8AEB1t7zw00Ow9xdC8@mail.gmail.com>
In-Reply-To: <AANLkTinnP2JB_Dqpxu3+3izOfG8AEB1t7zw00Ow9xdC8@mail.gmail.com>
Mime-Version: 1.0 (iPhone Mail 8C148)
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii
Message-Id: <EF3B67E6-26D2-42B3-AC99-01426AADC931@holsman.net>
Cc: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: iPhone Mail (8C148)
From: Ian Holsman <hadoop@holsman.net>
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of branch-0.20-append branch?
Date: Thu, 23 Dec 2010 17:24:38 +1100
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

In that case, I'm +1 on releasing a 20+append branch, but am nervous on how m=
uch effort will be put into testing it. But this option is better than the c=
urrent apache alternative out there as you and Owen mentioned.=20

---
Ian Holsman - 703 879-3128

I saw the angel in the marble and carved until I set him free -- Michelangel=
o

On 23/12/2010, at 5:16 PM, Stack <stack@duboce.net> wrote:

> On Wed, Dec 22, 2010 at 5:03 PM, Ian Holsman <hadoop@holsman.net> wrote:>
>>> Are you counting other than Apache releases?  (I see only 4 here, two
>>> of which probably should be removed:
>>> http://www.gtlib.gatech.edu/pub/apache//hadoop/core/.)
>>=20
>>=20
>> yes.. I was referring to the external companies who have decided to relea=
se their own version, for their own business purposes. (please don't take th=
at as a negative).
>>=20
>=20
> Oh.  I was not counting those at all.
>=20
> Currently, over in HBase we tell users build and 'trust' your own
> Hadoop binary from the tip of what to them probably looks like some
> random Hadoop branch OR go get the slick Cloudera pre-builts since
> Cloudera's CDH3s have append/sync.  By offering to release a
> hadoop-0.20.0-append, I was just trying to make some remiss for a
> gaping hole in the Apache Hadoop offering.
>=20
>=20
>>>> Is there a reason why we couldn't create a hadoop 0.20.3 release that h=
as this patch inside of it, as well as other fixes that have been applied si=
nce 0.20.2 (~26 patches)? Would this be too much effort for you to RM?..
>>>>=20
>>> ...
>>=20
>> I'm open with adding it, as lack of append/sync could be seen as a bug to=
 some. (yes i'm playing with words)
>=20
> My guess is that few would see it the way you do.  Append/sync has had
> a long torturous history.  HADOOP-1700, the original append issue, was
> originally opened in August 2007.  There have been two
> implementations.  The one in branch-0.20-append is the 'deprecated'
> implementation; i.e. its not the append that is in Hadoop TRUNK
> (though IIUC the 'deprecated' append runs on the largest 'known' HDFS
> cluster).  At least once, append was part of a release and then pulled
> because it was 'destabilizing'.  It might be hard getting such a
> storied, scarred feature in as a 'bug fix'.  If it did go in, the
> append/sync is of such a reputation that it might sully the current
> good standing hadoop 0.20 branch releases hold.
>=20
> That said, I'm cavalier and if others are game, I'd be up for running
> a 0.20.3 release that included it.
>=20
>> Thats why I think we should go to 0.22 ASAP and get companies to build th=
eir new features on trunk against that.
>>=20
>=20
> Waiting on 0.22 and its adoption is not going to work for HBase.  The
> HBase project would be long dead if waiting on 0.22 were the only
> option available to us.  In fact we'd be dead already if it wasn't for
> the lifeline thrown us by the folks who hooked us up with
> branch-0.20-append.
>=20
> St.Ack

From general-return-2586-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 07:08:28 2010
Return-Path: <general-return-2586-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29438 invoked from network); 23 Dec 2010 07:08:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 07:08:28 -0000
Received: (qmail 15098 invoked by uid 500); 23 Dec 2010 07:08:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15024 invoked by uid 500); 23 Dec 2010 07:08:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15015 invoked by uid 99); 23 Dec 2010 07:08:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 07:08:25 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [208.97.132.126] (HELO homiemail-a65.g.dreamhost.com) (208.97.132.126)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 07:08:19 +0000
Received: from homiemail-a65.g.dreamhost.com (localhost [127.0.0.1])
	by homiemail-a65.g.dreamhost.com (Postfix) with ESMTP id 301B07E406F
	for <general@hadoop.apache.org>; Wed, 22 Dec 2010 23:07:59 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; q=dns; s=
	gbiv.com; b=vS3z1kMwY4RlRC0JERFvEuI9eQyCU3Lp4nqMWPlz3j8WtvjEe2fk
	7IQXteDO2/2NauW5/hOggjc7AlYXIEfH8ot02R4us4T+Ggf4EbTUPf3r72Ye4F98
	E4F/BGHWW06Dfu0dV9GsJbL8aiLxEKqIsUATXABS0nxyYxw4snGzijo=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; s=gbiv.com;
	 bh=3YoFaZ65ZHMtd9n2gIrzdCzd2Lg=; b=O9D9xJG84mXcaactzHw29xdt0rWb
	y3txUqJvzRiPNO/vVfQTY7yy/HPXlYyPTIzhRhtxpegRlEYZ/KBlLZTpL/iYCCtP
	hp+eq7oUHg1MmujbOvFBtRootKLR8mIGQf7WQDXp0adDDaW18F1drY9YkMiYP3ae
	vCNBnPuUSStz6O0=
Received: from [192.168.1.66] (99-21-208-82.lightspeed.irvnca.sbcglobal.net [99.21.208.82])
	(Authenticated sender: fielding@gbiv.com)
	by homiemail-a65.g.dreamhost.com (Postfix) with ESMTPA id 0DE3D7E4065
	for <general@hadoop.apache.org>; Wed, 22 Dec 2010 23:07:59 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of branch-0.20-append branch?
From: "Roy T. Fielding" <fielding@gbiv.com>
In-Reply-To: <EF3B67E6-26D2-42B3-AC99-01426AADC931@holsman.net>
Date: Wed, 22 Dec 2010 23:07:58 -0800
Content-Transfer-Encoding: quoted-printable
Message-Id: <21FEBF88-2D56-48E0-B86C-334E94F04188@gbiv.com>
References: <AANLkTinMrd-acs=91xxw4MxHis8-RPSPtfhy7b4gu41O@mail.gmail.com> <7277DB0B-9B56-4B57-A574-F2764C11FC10@Holsman.NET> <AANLkTinMW7nvmapbkJwOGGNvaLNSrXSBr9c08JnBSHp4@mail.gmail.com> <C475DC5D-7CAC-47E4-838B-AED9C8274E5B@Holsman.NET> <AANLkTinnP2JB_Dqpxu3+3izOfG8AEB1t7zw00Ow9xdC8@mail.gmail.com> <EF3B67E6-26D2-42B3-AC99-01426AADC931@holsman.net>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

On Dec 22, 2010, at 10:24 PM, Ian Holsman wrote:

> In that case, I'm +1 on releasing a 20+append branch, but am nervous =
on how much effort will be put into testing it. But this option is =
better than the current apache alternative out there as you and Owen =
mentioned.=20

Features are not release version tags.  If there is a security bug
found then we would have to release a new version of the append
version, and a round of severe trout slapping would result.

If someone builds the source package and there are enough votes to
release it, then the version number is one of 0.22.0 (assuming trunk
hasn't been released yet), 0.23.0 (if 0.22.x is already published),
or 1.0.0.

That is, unless the PMC decides to make it a separate product, in
which case it would be hadoopend-0.20.0 (or something like that)
and will either die a slow and painful death or someone else will
pick it up and fork the project.

....Roy


From general-return-2587-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 08:01:01 2010
Return-Path: <general-return-2587-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 40384 invoked from network); 23 Dec 2010 08:01:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 08:01:00 -0000
Received: (qmail 58556 invoked by uid 500); 23 Dec 2010 08:00:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58239 invoked by uid 500); 23 Dec 2010 08:00:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58226 invoked by uid 99); 23 Dec 2010 08:00:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 08:00:57 +0000
X-ASF-Spam-Status: No, hits=-1997.8 required=10.0
	tests=ALL_TRUSTED,HTML_MESSAGE,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 23 Dec 2010 08:00:55 +0000
Received: (qmail 40279 invoked by uid 99); 23 Dec 2010 08:00:33 -0000
Received: from localhost.apache.org (HELO mail-bw0-f48.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 08:00:33 +0000
Received: by bwz8 with SMTP id 8so6235305bwz.35
        for <general@hadoop.apache.org>; Thu, 23 Dec 2010 00:00:31 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.65.135 with SMTP id j7mr2967687bki.85.1293091230961; Thu,
 23 Dec 2010 00:00:30 -0800 (PST)
Received: by 10.204.76.14 with HTTP; Thu, 23 Dec 2010 00:00:30 -0800 (PST)
In-Reply-To: <21FEBF88-2D56-48E0-B86C-334E94F04188@gbiv.com>
References: <AANLkTinMrd-acs=91xxw4MxHis8-RPSPtfhy7b4gu41O@mail.gmail.com>
	<7277DB0B-9B56-4B57-A574-F2764C11FC10@Holsman.NET>
	<AANLkTinMW7nvmapbkJwOGGNvaLNSrXSBr9c08JnBSHp4@mail.gmail.com>
	<C475DC5D-7CAC-47E4-838B-AED9C8274E5B@Holsman.NET>
	<AANLkTinnP2JB_Dqpxu3+3izOfG8AEB1t7zw00Ow9xdC8@mail.gmail.com>
	<EF3B67E6-26D2-42B3-AC99-01426AADC931@holsman.net>
	<21FEBF88-2D56-48E0-B86C-334E94F04188@gbiv.com>
Date: Thu, 23 Dec 2010 00:00:30 -0800
Message-ID: <AANLkTi=kVi=k6GV4tC8LNrpX4HaMxqRK72JJwOjN78tL@mail.gmail.com>
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of
 branch-0.20-append branch?
From: "Owen O'Malley" <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c5b5ae7bd0ac04980f4392
X-Virus-Checked: Checked by ClamAV on apache.org

--001636c5b5ae7bd0ac04980f4392
Content-Type: text/plain; charset=UTF-8

On Wed, Dec 22, 2010 at 11:07 PM, Roy T. Fielding <fielding@gbiv.com> wrote:

> Features are not release version tags.  If there is a security bug
> found then we would have to release a new version of the append
> version, and a round of severe trout slapping would result.
>

Yeah, it isn't a perfect solution and it doesn't scale to a second tag, but
the problem is that this is effectively a release branch between 0.20 and
0.21. Of course I agree that any critical bugs would need to be fixed in the
append branch as well as the 0.20 and 0.21 branches.

If you want to stick to pure numbers and we want to leave ourselves a way to
bugfix the 0.20 branch without append, we'd could use a version string like
0.20.100, etc. Not pretty, but it does preserve the numeric ordering and
suggest a version jump.

If I remember right, there were also protocol changes in the append branch,
which was another reason we didn't want to put it directly into the 0.20
branch.

-- Owen

--001636c5b5ae7bd0ac04980f4392--

From general-return-2588-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 10:53:43 2010
Return-Path: <general-return-2588-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 877 invoked from network); 23 Dec 2010 10:53:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 10:53:42 -0000
Received: (qmail 18284 invoked by uid 500); 23 Dec 2010 10:53:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18092 invoked by uid 500); 23 Dec 2010 10:53:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18080 invoked by uid 99); 23 Dec 2010 10:53:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 10:53:40 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bjoern@schiessle.org designates 78.46.69.5 as permitted sender)
Received: from [78.46.69.5] (HELO zucker.schokokeks.org) (78.46.69.5)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 10:53:34 +0000
Received: from pcmoholynagy.informatik.uni-stuttgart.de (pcmoholynagy.informatik.uni-stuttgart.de [::ffff:129.69.216.55])
  (AUTH: LOGIN schiesbn@schokokeks.org, TLS: TLSv1/SSLv3,128bits,AES128-SHA)
  by zucker.schokokeks.org with ESMTPSA; Thu, 23 Dec 2010 11:53:11 +0100
  id 0000000000018010.000000004D132A17.00007FD1
Date: Thu, 23 Dec 2010 11:50:04 +0100
From: Bjoern Schiessle <bjoern@schiessle.org>
To: general@hadoop.apache.org
Subject: Re: namenode doesn't start after reboot
Message-ID: <20101223115004.7f8dc48c@pcmoholynagy.informatik.uni-stuttgart.de>
In-Reply-To: <AANLkTi=9_-CPn+GON+YqLra1sjDX3-i-tqZ4rv39rTfT@mail.gmail.com>
References: <20101222160311.7ffb9b6d@laptop>
	<AANLkTike7k5nqViAsdyC_mR6LOBVrrFFqsyGJo=-W6GQ@mail.gmail.com>
	<AANLkTi=9_-CPn+GON+YqLra1sjDX3-i-tqZ4rv39rTfT@mail.gmail.com>
X-Mailer: Claws Mail 3.7.4 (GTK+ 2.20.1; i486-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi,

On Thu, 23 Dec 2010 09:30:17 +0800 li ping wrote:
> It seems the exception occurs during NameNode loads the editlog.
> make sure the editlog file exists. or you can debug the application to
> see what's wrong.

last night I tried to fix the problem and did a big mistake. Instead of
copying /var/lib/hadoop-0.20/cache/hadoop/dfs/name/current/edits and
edits.new to a backup I moved them and later delete the only version
hence I thought I have a copy.

The good thing: The namenode starts again.
The bad thing: My file system is now in an inconsistent state.

Probably the only solution is to reformat the hdfs and start from
scratch. Thankfully there wasn't that much data stored at the hdfs until
now but I definitely have to make sure that this doesn't happens again:

1. I have set up a second dfs.name.dir which is stored at another
computer (mounted by sshfs)
2. I will install a backup script similar to:
http://blog.milford.io/2010/10/simple-hadoop-namenode-backup-script

Do you think this should be enough to overcome such situations in the
future? Any additional ideas how to make it more safe?

I'm still a little bit afraid if I think about the next time I will have
to reboot the server. Shouldn't a reboot safely stop and restart all
Hadoop services? Is there any thing I can do to make sure that the next
reboot will not cause the same problems?

Thanks a lot!
Bj=F6rn



From general-return-2589-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 12:46:17 2010
Return-Path: <general-return-2589-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47745 invoked from network); 23 Dec 2010 12:46:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 12:46:17 -0000
Received: (qmail 97759 invoked by uid 500); 23 Dec 2010 12:46:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97618 invoked by uid 500); 23 Dec 2010 12:46:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97610 invoked by uid 99); 23 Dec 2010 12:46:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 12:46:14 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of li.j2ee@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 12:46:09 +0000
Received: by bwz8 with SMTP id 8so6433695bwz.35
        for <general@hadoop.apache.org>; Thu, 23 Dec 2010 04:45:48 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=RV7+mxD4rTwposdnJWoNBoIhaXbdJLkdh+m7TMHp96M=;
        b=HbrzkcY6pICvWjPSW2qDluR8KjvcKNZ4xhm9KzaIzH4V+3zkgGAEsnnKQyt3YShEuF
         l4rrdnphu4mDtg57xo2iJL831kTb6tUbiVk+YQ3aLq9Zk3LWzTSrvP4aXvTGUKBba2PT
         A8qcgSDE518iLpBZfQBctivTadUW8G9E6ED28=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=dLE06yiXNUtNyHgczq0EFu4jTMZkYl0GWVN/TxZN4kSjkaxZqD2cXLV/awT4o/VW9c
         WxRAsEWMcPcg5SFJr3aKO3ZPVpUCYSjiETEKOorwd3+kwmQ9auyY/bilU37ZfLSBU/o1
         N8XdYZZz1Me8vBvLM7tulDupQ6rFuIODsGZAs=
MIME-Version: 1.0
Received: by 10.204.99.145 with SMTP id u17mr7078505bkn.1.1293108347647; Thu,
 23 Dec 2010 04:45:47 -0800 (PST)
Received: by 10.204.138.67 with HTTP; Thu, 23 Dec 2010 04:45:47 -0800 (PST)
In-Reply-To: <20101223115004.7f8dc48c@pcmoholynagy.informatik.uni-stuttgart.de>
References: <20101222160311.7ffb9b6d@laptop>
	<AANLkTike7k5nqViAsdyC_mR6LOBVrrFFqsyGJo=-W6GQ@mail.gmail.com>
	<AANLkTi=9_-CPn+GON+YqLra1sjDX3-i-tqZ4rv39rTfT@mail.gmail.com>
	<20101223115004.7f8dc48c@pcmoholynagy.informatik.uni-stuttgart.de>
Date: Thu, 23 Dec 2010 20:45:47 +0800
Message-ID: <AANLkTimjm8vn54VKJGDvTMx07YNWUa0ofZJO2QiwkUs+@mail.gmail.com>
Subject: Re: namenode doesn't start after reboot
From: li ping <li.j2ee@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636457ce4b7b9100498133f2e

--001636457ce4b7b9100498133f2e
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

As far as I know, setup a backup namenode dir is enough.

I haven't use the hadoop in a production environment. So, I can't tell you
what would be right way to reboot the server.

On Thu, Dec 23, 2010 at 6:50 PM, Bjoern Schiessle <bjoern@schiessle.org>wro=
te:

> Hi,
>
> On Thu, 23 Dec 2010 09:30:17 +0800 li ping wrote:
> > It seems the exception occurs during NameNode loads the editlog.
> > make sure the editlog file exists. or you can debug the application to
> > see what's wrong.
>
> last night I tried to fix the problem and did a big mistake. Instead of
> copying /var/lib/hadoop-0.20/cache/hadoop/dfs/name/current/edits and
> edits.new to a backup I moved them and later delete the only version
> hence I thought I have a copy.
>
> The good thing: The namenode starts again.
> The bad thing: My file system is now in an inconsistent state.
>
> Probably the only solution is to reformat the hdfs and start from
> scratch. Thankfully there wasn't that much data stored at the hdfs until
> now but I definitely have to make sure that this doesn't happens again:
>
> 1. I have set up a second dfs.name.dir which is stored at another
> computer (mounted by sshfs)
> 2. I will install a backup script similar to:
> http://blog.milford.io/2010/10/simple-hadoop-namenode-backup-script
>
> Do you think this should be enough to overcome such situations in the
> future? Any additional ideas how to make it more safe?
>
> I'm still a little bit afraid if I think about the next time I will have
> to reboot the server. Shouldn't a reboot safely stop and restart all
> Hadoop services? Is there any thing I can do to make sure that the next
> reboot will not cause the same problems?
>
> Thanks a lot!
> Bj=C3=B6rn
>
>
>


--=20
-----=E6=9D=8E=E5=B9=B3

--001636457ce4b7b9100498133f2e--

From general-return-2590-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 15:29:48 2010
Return-Path: <general-return-2590-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43018 invoked from network); 23 Dec 2010 15:29:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 15:29:48 -0000
Received: (qmail 57390 invoked by uid 500); 23 Dec 2010 15:29:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57138 invoked by uid 500); 23 Dec 2010 15:29:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57130 invoked by uid 99); 23 Dec 2010 15:29:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 15:29:45 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of patodirahul@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 15:29:41 +0000
Received: by wye20 with SMTP id 20so6547561wye.35
        for <general@hadoop.apache.org>; Thu, 23 Dec 2010 07:29:20 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=pKBotC9L9+dtzrXItTYgcyTDEyMgv0V7kratpnOlKvk=;
        b=d2XDqWnLvOEpVY96A974l04IomFWjQj+jLeio6T9u+z7FcOvj/u8SE9+5vGRFXDaVF
         ui6J4uvrCDr86a1gF2b4lTscQ5LrFuNdBmUVWhgwMsV7e6JSouLpoLiLGTJl1Upfgl0V
         bbwl89SL4Jz4imxNXGd/W0KpTDB1A0FeaPGD4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=BerdVfXysBQgFexK5SR7vgJZluDmdUPpKrZsfvwtIM21HF4VBjfHWWy0qZhiydBZlU
         iKKpudypvjzK2ITe4C2boMBtOhU7QIMvWViunjwZg5gLYHZVKUuc9eMfV3ZOrtuGtGca
         +Ugp9jArAgMMKFR+/Pc78u24p94j/cJ/6lue8=
MIME-Version: 1.0
Received: by 10.216.68.4 with SMTP id k4mr8953240wed.63.1293118159926; Thu, 23
 Dec 2010 07:29:19 -0800 (PST)
Received: by 10.216.16.131 with HTTP; Thu, 23 Dec 2010 07:29:19 -0800 (PST)
In-Reply-To: <AANLkTimjm8vn54VKJGDvTMx07YNWUa0ofZJO2QiwkUs+@mail.gmail.com>
References: <20101222160311.7ffb9b6d@laptop>
	<AANLkTike7k5nqViAsdyC_mR6LOBVrrFFqsyGJo=-W6GQ@mail.gmail.com>
	<AANLkTi=9_-CPn+GON+YqLra1sjDX3-i-tqZ4rv39rTfT@mail.gmail.com>
	<20101223115004.7f8dc48c@pcmoholynagy.informatik.uni-stuttgart.de>
	<AANLkTimjm8vn54VKJGDvTMx07YNWUa0ofZJO2QiwkUs+@mail.gmail.com>
Date: Thu, 23 Dec 2010 20:59:19 +0530
Message-ID: <AANLkTinhHSKq18aQvz-ALLz+d-4Rjt23SQ8bm_CH2s4c@mail.gmail.com>
Subject: Re: namenode doesn't start after reboot
From: rahul patodi <patodirahul@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0ce0b8fa9339fb04981588e7

--000e0ce0b8fa9339fb04981588e7
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi,
If you want to reboot the server:
1. stop mapred
2. stop dfs
the reboot
when you again want to restart hadoop
firstly start dfs then mepred.

--=20
*Regards*,
Rahul Patodi
Software Engineer,
Impetus Infotech (India) Pvt Ltd,
www.impetus.com
Mob:09907074413


On Thu, Dec 23, 2010 at 6:15 PM, li ping <li.j2ee@gmail.com> wrote:

> As far as I know, setup a backup namenode dir is enough.
>
> I haven't use the hadoop in a production environment. So, I can't tell yo=
u
> what would be right way to reboot the server.
>
> On Thu, Dec 23, 2010 at 6:50 PM, Bjoern Schiessle <bjoern@schiessle.org
> >wrote:
>
> > Hi,
> >
> > On Thu, 23 Dec 2010 09:30:17 +0800 li ping wrote:
> > > It seems the exception occurs during NameNode loads the editlog.
> > > make sure the editlog file exists. or you can debug the application t=
o
> > > see what's wrong.
> >
> > last night I tried to fix the problem and did a big mistake. Instead of
> > copying /var/lib/hadoop-0.20/cache/hadoop/dfs/name/current/edits and
> > edits.new to a backup I moved them and later delete the only version
> > hence I thought I have a copy.
> >
> > The good thing: The namenode starts again.
> > The bad thing: My file system is now in an inconsistent state.
> >
> > Probably the only solution is to reformat the hdfs and start from
> > scratch. Thankfully there wasn't that much data stored at the hdfs unti=
l
> > now but I definitely have to make sure that this doesn't happens again:
> >
> > 1. I have set up a second dfs.name.dir which is stored at another
> > computer (mounted by sshfs)
> > 2. I will install a backup script similar to:
> > http://blog.milford.io/2010/10/simple-hadoop-namenode-backup-script
> >
> > Do you think this should be enough to overcome such situations in the
> > future? Any additional ideas how to make it more safe?
> >
> > I'm still a little bit afraid if I think about the next time I will hav=
e
> > to reboot the server. Shouldn't a reboot safely stop and restart all
> > Hadoop services? Is there any thing I can do to make sure that the next
> > reboot will not cause the same problems?
> >
> > Thanks a lot!
> > Bj=C3=B6rn
> >
> >
> >
>
>
> --
> -----=E6=9D=8E=E5=B9=B3
>

--000e0ce0b8fa9339fb04981588e7--

From general-return-2591-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 17:16:32 2010
Return-Path: <general-return-2591-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19205 invoked from network); 23 Dec 2010 17:16:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 17:16:32 -0000
Received: (qmail 86615 invoked by uid 500); 23 Dec 2010 17:16:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86509 invoked by uid 500); 23 Dec 2010 17:16:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86501 invoked by uid 99); 23 Dec 2010 17:16:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 17:16:30 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 17:16:24 +0000
Received: by qyk10 with SMTP id 10so6501281qyk.14
        for <general@hadoop.apache.org>; Thu, 23 Dec 2010 09:16:02 -0800 (PST)
Received: by 10.229.225.133 with SMTP id is5mr7285164qcb.280.1293124561856;
 Thu, 23 Dec 2010 09:16:01 -0800 (PST)
MIME-Version: 1.0
Received: by 10.229.82.17 with HTTP; Thu, 23 Dec 2010 09:15:41 -0800 (PST)
In-Reply-To: <AANLkTinhHSKq18aQvz-ALLz+d-4Rjt23SQ8bm_CH2s4c@mail.gmail.com>
References: <20101222160311.7ffb9b6d@laptop> <AANLkTike7k5nqViAsdyC_mR6LOBVrrFFqsyGJo=-W6GQ@mail.gmail.com>
 <AANLkTi=9_-CPn+GON+YqLra1sjDX3-i-tqZ4rv39rTfT@mail.gmail.com>
 <20101223115004.7f8dc48c@pcmoholynagy.informatik.uni-stuttgart.de>
 <AANLkTimjm8vn54VKJGDvTMx07YNWUa0ofZJO2QiwkUs+@mail.gmail.com> <AANLkTinhHSKq18aQvz-ALLz+d-4Rjt23SQ8bm_CH2s4c@mail.gmail.com>
From: "Aaron T. Myers" <atm@cloudera.com>
Date: Thu, 23 Dec 2010 09:15:41 -0800
Message-ID: <AANLkTin+n_GL5CtLsMrCgdz0URKkWO0vtQCsB-jca51y@mail.gmail.com>
Subject: Re: namenode doesn't start after reboot
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163630f20928e97d0498170632
X-Virus-Checked: Checked by ClamAV on apache.org

--00163630f20928e97d0498170632
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

All this aside, you really shouldn't have to "safely" stop all the Hadoop
services when you reboot any of your servers. Hadoop should be able to
survive a crash of any of the daemons. Any circumstance in which Hadoop
currently corrupts the edits log or fsimage is a serious bug, and should be
reported via JIRA.

--
Aaron T. Myers
Software Engineer, Cloudera



On Thu, Dec 23, 2010 at 7:29 AM, rahul patodi <patodirahul@gmail.com> wrote=
:

> Hi,
> If you want to reboot the server:
> 1. stop mapred
> 2. stop dfs
> the reboot
> when you again want to restart hadoop
> firstly start dfs then mepred.
>
> --
> *Regards*,
> Rahul Patodi
> Software Engineer,
> Impetus Infotech (India) Pvt Ltd,
> www.impetus.com
> Mob:09907074413
>
>
> On Thu, Dec 23, 2010 at 6:15 PM, li ping <li.j2ee@gmail.com> wrote:
>
> > As far as I know, setup a backup namenode dir is enough.
> >
> > I haven't use the hadoop in a production environment. So, I can't tell
> you
> > what would be right way to reboot the server.
> >
> > On Thu, Dec 23, 2010 at 6:50 PM, Bjoern Schiessle <bjoern@schiessle.org
> > >wrote:
> >
> > > Hi,
> > >
> > > On Thu, 23 Dec 2010 09:30:17 +0800 li ping wrote:
> > > > It seems the exception occurs during NameNode loads the editlog.
> > > > make sure the editlog file exists. or you can debug the application
> to
> > > > see what's wrong.
> > >
> > > last night I tried to fix the problem and did a big mistake. Instead =
of
> > > copying /var/lib/hadoop-0.20/cache/hadoop/dfs/name/current/edits and
> > > edits.new to a backup I moved them and later delete the only version
> > > hence I thought I have a copy.
> > >
> > > The good thing: The namenode starts again.
> > > The bad thing: My file system is now in an inconsistent state.
> > >
> > > Probably the only solution is to reformat the hdfs and start from
> > > scratch. Thankfully there wasn't that much data stored at the hdfs
> until
> > > now but I definitely have to make sure that this doesn't happens agai=
n:
> > >
> > > 1. I have set up a second dfs.name.dir which is stored at another
> > > computer (mounted by sshfs)
> > > 2. I will install a backup script similar to:
> > > http://blog.milford.io/2010/10/simple-hadoop-namenode-backup-script
> > >
> > > Do you think this should be enough to overcome such situations in the
> > > future? Any additional ideas how to make it more safe?
> > >
> > > I'm still a little bit afraid if I think about the next time I will
> have
> > > to reboot the server. Shouldn't a reboot safely stop and restart all
> > > Hadoop services? Is there any thing I can do to make sure that the ne=
xt
> > > reboot will not cause the same problems?
> > >
> > > Thanks a lot!
> > > Bj=C3=B6rn
> > >
> > >
> > >
> >
> >
> > --
> > -----=E6=9D=8E=E5=B9=B3
> >
>

--00163630f20928e97d0498170632--

From general-return-2592-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 17:27:44 2010
Return-Path: <general-return-2592-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22416 invoked from network); 23 Dec 2010 17:27:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 17:27:44 -0000
Received: (qmail 97631 invoked by uid 500); 23 Dec 2010 17:27:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97591 invoked by uid 500); 23 Dec 2010 17:27:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97583 invoked by uid 99); 23 Dec 2010 17:27:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 17:27:42 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 17:27:36 +0000
Received: by fxm2 with SMTP id 2so6335036fxm.35
        for <general@hadoop.apache.org>; Thu, 23 Dec 2010 09:27:15 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        bh=O6J9VtLS7yzra/OilAAEloxAY6KI8fvlr0vGY/T1yJA=;
        b=hiAa6mYqMYaMFFUbNQFqOJmRpnw3kNujieodHi/MU5BIR18uIB97fq1opSIOwvh7q+
         NOXL9s2UMbu+c68ZP+mXEi2LkMdFkX/OWGc3ifTyvbV7kpEhedgvIlq5BYUBQKi+/zJl
         gxpsvsL8KsWrbKgOu9pP3Dsnx7gMmKgDB2hrY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=LfmkXer5lVdaOe3L4OUVMevWPN6VqAJ5qfbQCtxGL/ltZsVdx60XhAC+rQj+X3uISF
         O039bDxwlZBAKUx9RATEaRVlguptytIg3TIT1yKlbEh4VJngYnA4B3agJi9FMBwjoI3s
         YoLojP+mBSnK01UbiSaX1SoDHCnu+glj6yMcI=
MIME-Version: 1.0
Received: by 10.223.96.198 with SMTP id i6mr1285645fan.10.1293125234907; Thu,
 23 Dec 2010 09:27:14 -0800 (PST)
Sender: saint.ack@gmail.com
Received: by 10.223.83.9 with HTTP; Thu, 23 Dec 2010 09:27:14 -0800 (PST)
In-Reply-To: <AANLkTi=kVi=k6GV4tC8LNrpX4HaMxqRK72JJwOjN78tL@mail.gmail.com>
References: <AANLkTinMrd-acs=91xxw4MxHis8-RPSPtfhy7b4gu41O@mail.gmail.com>
	<7277DB0B-9B56-4B57-A574-F2764C11FC10@Holsman.NET>
	<AANLkTinMW7nvmapbkJwOGGNvaLNSrXSBr9c08JnBSHp4@mail.gmail.com>
	<C475DC5D-7CAC-47E4-838B-AED9C8274E5B@Holsman.NET>
	<AANLkTinnP2JB_Dqpxu3+3izOfG8AEB1t7zw00Ow9xdC8@mail.gmail.com>
	<EF3B67E6-26D2-42B3-AC99-01426AADC931@holsman.net>
	<21FEBF88-2D56-48E0-B86C-334E94F04188@gbiv.com>
	<AANLkTi=kVi=k6GV4tC8LNrpX4HaMxqRK72JJwOjN78tL@mail.gmail.com>
Date: Thu, 23 Dec 2010 09:27:14 -0800
X-Google-Sender-Auth: 9VpLeav-Md8HRdFsYnezAuOeFh0
Message-ID: <AANLkTi=BsauOvTMP1W6=r=P8mKB+eKQSaWvcUCgMQbwe@mail.gmail.com>
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of
 branch-0.20-append branch?
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Thu, Dec 23, 2010 at 12:00 AM, Owen O'Malley <omalley@apache.org> wrote:
> If I remember right, there were also protocol changes in the append branch,
> which was another reason we didn't want to put it directly into the 0.20
> branch.
>

That is indeed the case Owen.
St.Ack

From general-return-2593-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 17:33:38 2010
Return-Path: <general-return-2593-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 26878 invoked from network); 23 Dec 2010 17:33:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 17:33:38 -0000
Received: (qmail 8868 invoked by uid 500); 23 Dec 2010 17:33:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8728 invoked by uid 500); 23 Dec 2010 17:33:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8720 invoked by uid 99); 23 Dec 2010 17:33:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 17:33:36 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mcsrivas@gmail.com designates 209.85.216.169 as permitted sender)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 17:33:29 +0000
Received: by qyk7 with SMTP id 7so7238800qyk.14
        for <general@hadoop.apache.org>; Thu, 23 Dec 2010 09:33:09 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=qzMFRUosJeEoHGphnbIab9KVKPP5P0/OT5okoOceZIM=;
        b=BkgT0Ty3oRAm2NPbT8AM7a61EuCJkYbBrMGc0cF6RhHN0cFAJQjBNPebz2fQW1yhrC
         InwDYTtMT5aPSG1d478asO0WWwAFtieLxyRVkg4f1ZcbG+fsaR2F4rNnjC9UITTGtvZu
         +c0BFR9kOzMB/4TrAPAuyXzRM9oJXkbXpPCX4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=diDWkDanokl1b33NVo9XY56kiQ8avxMkvGqpRSTN8YlP4cbDhNcbWALZMT/sbf1U/m
         GC3bZhebaSZyR8WIRpAzcH3BTM0FhalMoOxk7E7Lf7Q+SX0JLYcUrLi9TkmhYKRUgHvf
         QdjlKP9FrETw0bhue44gbFOKw4Y/h5jkHCCjU=
MIME-Version: 1.0
Received: by 10.224.37.141 with SMTP id x13mr8099233qad.76.1293125589074; Thu,
 23 Dec 2010 09:33:09 -0800 (PST)
Received: by 10.220.87.20 with HTTP; Thu, 23 Dec 2010 09:33:08 -0800 (PST)
In-Reply-To: <AANLkTi=kVi=k6GV4tC8LNrpX4HaMxqRK72JJwOjN78tL@mail.gmail.com>
References: <AANLkTinMrd-acs=91xxw4MxHis8-RPSPtfhy7b4gu41O@mail.gmail.com>
	<7277DB0B-9B56-4B57-A574-F2764C11FC10@Holsman.NET>
	<AANLkTinMW7nvmapbkJwOGGNvaLNSrXSBr9c08JnBSHp4@mail.gmail.com>
	<C475DC5D-7CAC-47E4-838B-AED9C8274E5B@Holsman.NET>
	<AANLkTinnP2JB_Dqpxu3+3izOfG8AEB1t7zw00Ow9xdC8@mail.gmail.com>
	<EF3B67E6-26D2-42B3-AC99-01426AADC931@holsman.net>
	<21FEBF88-2D56-48E0-B86C-334E94F04188@gbiv.com>
	<AANLkTi=kVi=k6GV4tC8LNrpX4HaMxqRK72JJwOjN78tL@mail.gmail.com>
Date: Thu, 23 Dec 2010 09:33:08 -0800
Message-ID: <AANLkTikHfMxBk7SH+4oq94r894W4Qr2MOJWJH15nWeHi@mail.gmail.com>
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of
 branch-0.20-append branch?
From: "M. C. Srivas" <mcsrivas@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175cddbe6303c40498174363
X-Virus-Checked: Checked by ClamAV on apache.org

--0015175cddbe6303c40498174363
Content-Type: text/plain; charset=ISO-8859-1

[ Sorry if this is be-laboring the obvious ]

There are two append solutions floating around, and they are incompatible
with each other. Thus, the two "branches" will forever remain incompatible
with each other, regardless of how they are numbered (0.22,  0.23,  0.20.3,
e.t.c.)

Unless both are merged into one branch, and a switch provided to  "use
HDFS-200 append" or "use 0.22 append", we have effectively split Hadoop into
two.


On Thu, Dec 23, 2010 at 12:00 AM, Owen O'Malley <omalley@apache.org> wrote:

> On Wed, Dec 22, 2010 at 11:07 PM, Roy T. Fielding <fielding@gbiv.com>
> wrote:
>
> > Features are not release version tags.  If there is a security bug
> > found then we would have to release a new version of the append
> > version, and a round of severe trout slapping would result.
> >
>
> Yeah, it isn't a perfect solution and it doesn't scale to a second tag, but
> the problem is that this is effectively a release branch between 0.20 and
> 0.21. Of course I agree that any critical bugs would need to be fixed in
> the
> append branch as well as the 0.20 and 0.21 branches.
>
> If you want to stick to pure numbers and we want to leave ourselves a way
> to
> bugfix the 0.20 branch without append, we'd could use a version string like
> 0.20.100, etc. Not pretty, but it does preserve the numeric ordering and
> suggest a version jump.
>
> If I remember right, there were also protocol changes in the append branch,
> which was another reason we didn't want to put it directly into the 0.20
> branch.
>
> -- Owen
>

--0015175cddbe6303c40498174363--

From general-return-2594-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 17:39:06 2010
Return-Path: <general-return-2594-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 28458 invoked from network); 23 Dec 2010 17:39:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 17:39:06 -0000
Received: (qmail 19499 invoked by uid 500); 23 Dec 2010 17:39:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19272 invoked by uid 500); 23 Dec 2010 17:39:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19263 invoked by uid 99); 23 Dec 2010 17:39:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 17:39:04 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jghoman@gmail.com designates 209.85.210.176 as permitted sender)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 17:38:57 +0000
Received: by iyb26 with SMTP id 26so6145943iyb.35
        for <general@hadoop.apache.org>; Thu, 23 Dec 2010 09:38:36 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=JJJaIreut9JqTr5AlCX9cay7izu+9zAKfFHp7/3e36s=;
        b=tiQVRDtGiwX9fvQBHf5qQMDHGKqGHw6TBnY8lL7ma3XWylw4spMZ3Bnb5nVFlMj+PJ
         aw6Qh1MQ19wUjHq4OEesTlkfqltJCltxRS+A2zmKWP9XzKucIoodHSvzx6oGiq2Tx0u2
         H83+OX0HQ3EIQNGBRhGtardoqelIYgFRn6yKY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=sceWFbnarh68+OhzSGUJLd+8DI4zsnXikULUlq/RcyX5TaBJDJgLhkunZrO6qjLnDS
         Dp65PDkDop7jZ+2fyf9Q9ttqQUVQUWJCrp/i6mJYPtoppiq2P7jB60jbzaF9tDbArufK
         boGZBdT97FRlLmLRPJsFF34E+dm5uxyoOliP4=
Received: by 10.231.14.74 with SMTP id f10mr8437120iba.119.1293125916178; Thu,
 23 Dec 2010 09:38:36 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.25.163 with HTTP; Thu, 23 Dec 2010 09:38:15 -0800 (PST)
In-Reply-To: <AANLkTikHfMxBk7SH+4oq94r894W4Qr2MOJWJH15nWeHi@mail.gmail.com>
References: <AANLkTinMrd-acs=91xxw4MxHis8-RPSPtfhy7b4gu41O@mail.gmail.com>
 <7277DB0B-9B56-4B57-A574-F2764C11FC10@Holsman.NET> <AANLkTinMW7nvmapbkJwOGGNvaLNSrXSBr9c08JnBSHp4@mail.gmail.com>
 <C475DC5D-7CAC-47E4-838B-AED9C8274E5B@Holsman.NET> <AANLkTinnP2JB_Dqpxu3+3izOfG8AEB1t7zw00Ow9xdC8@mail.gmail.com>
 <EF3B67E6-26D2-42B3-AC99-01426AADC931@holsman.net> <21FEBF88-2D56-48E0-B86C-334E94F04188@gbiv.com>
 <AANLkTi=kVi=k6GV4tC8LNrpX4HaMxqRK72JJwOjN78tL@mail.gmail.com> <AANLkTikHfMxBk7SH+4oq94r894W4Qr2MOJWJH15nWeHi@mail.gmail.com>
From: Jakob Homan <jghoman@gmail.com>
Date: Thu, 23 Dec 2010 09:38:15 -0800
Message-ID: <AANLkTikiUzhMkh9y4Eaud4swibOmVcYL=0PFE28rms8n@mail.gmail.com>
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of
 branch-0.20-append branch?
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

It's difficult to support this proposal knowing how much time would be
spent preparing an official release, continuing to support it and
continuing to two support two separate implementations of append.  I
believe that effort would be better spent getting out a kick-ass 22
(or, barring that, a *really* kick-ass 23).

The Promised Land that we say we're all trying to get to is regular,
timely, feature-complete, tested, innovative but stable releases of
new versions of Apache Hadoop.  Missing out any one of those criteria
discovered will continue (and has continued) the current situation
where quasi-official branches and outside distributions fill the void
such a release should.  The effort to maintain this offical branch and
fix the bugs that will be discovered could be better spent moving us
closer to that goal.

I'm certainly sympathetic to the difficult position our quagmire has
placed HBase into.  However, the current proposal would hurt HDFS to
help HBase. The best solution for that project, as well as for HDFS,
is to get HDFS back to a healthy release cycle; not prolong or codify
the current ad-hoc state of affairs.  Let's stop digging this hole.
-jakob

On Thu, Dec 23, 2010 at 9:33 AM, M. C. Srivas <mcsrivas@gmail.com> wrote:
> [ Sorry if this is be-laboring the obvious ]
>
> There are two append solutions floating around, and they are incompatible
> with each other. Thus, the two "branches" will forever remain incompatibl=
e
> with each other, regardless of how they are numbered (0.22, =A00.23, =A00=
.20.3,
> e.t.c.)
>
> Unless both are merged into one branch, and a switch provided to =A0"use
> HDFS-200 append" or "use 0.22 append", we have effectively split Hadoop i=
nto
> two.
>
>
> On Thu, Dec 23, 2010 at 12:00 AM, Owen O'Malley <omalley@apache.org> wrot=
e:
>
>> On Wed, Dec 22, 2010 at 11:07 PM, Roy T. Fielding <fielding@gbiv.com>
>> wrote:
>>
>> > Features are not release version tags. =A0If there is a security bug
>> > found then we would have to release a new version of the append
>> > version, and a round of severe trout slapping would result.
>> >
>>
>> Yeah, it isn't a perfect solution and it doesn't scale to a second tag, =
but
>> the problem is that this is effectively a release branch between 0.20 an=
d
>> 0.21. Of course I agree that any critical bugs would need to be fixed in
>> the
>> append branch as well as the 0.20 and 0.21 branches.
>>
>> If you want to stick to pure numbers and we want to leave ourselves a wa=
y
>> to
>> bugfix the 0.20 branch without append, we'd could use a version string l=
ike
>> 0.20.100, etc. Not pretty, but it does preserve the numeric ordering and
>> suggest a version jump.
>>
>> If I remember right, there were also protocol changes in the append bran=
ch,
>> which was another reason we didn't want to put it directly into the 0.20
>> branch.
>>
>> -- Owen
>>
>

From general-return-2595-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 18:16:06 2010
Return-Path: <general-return-2595-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46357 invoked from network); 23 Dec 2010 18:16:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 18:16:05 -0000
Received: (qmail 64442 invoked by uid 500); 23 Dec 2010 18:16:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 64269 invoked by uid 500); 23 Dec 2010 18:16:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 64261 invoked by uid 99); 23 Dec 2010 18:16:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 18:16:04 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mcsrivas@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 18:15:59 +0000
Received: by qwh6 with SMTP id 6so6704677qwh.35
        for <general@hadoop.apache.org>; Thu, 23 Dec 2010 10:15:38 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=yvC4UyM80c2HltvjtGXWGIruqXWJYH3jNZz8VG3IS+o=;
        b=kLQo5gr3k8UGOT8muGqDMDvadykgsSFvj7qa5a+nICx4F3ostjrYv48ejeJalAG2pq
         TvHn9rabRhnSBid+fPFWKRAQZVbpt5BXLXwhriwWRsdk72pnbmjVLAwS8aCUXt7KJ6JO
         lJKPol6zemNc2nQmqC48/FGjFWhqH64eioZ8k=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=kuYCWcsh3rmH2MjtwcGqRxpG0u7lhOesEMvyp60FDhC946IPPFIYBHGMhj1tFrUNsV
         WjE171WuyF8hYXf/x93RO6+jgZp2KEzdTKU7Td64r0X0lDDL/IRcIuMNwVYULOvves2p
         NtcMHIvvqvvuPQ0KS0VMuPfnhLl+6FqnleJcg=
MIME-Version: 1.0
Received: by 10.224.6.141 with SMTP id 13mr8136548qaz.26.1293128138269; Thu,
 23 Dec 2010 10:15:38 -0800 (PST)
Received: by 10.220.87.20 with HTTP; Thu, 23 Dec 2010 10:15:38 -0800 (PST)
In-Reply-To: <AANLkTikiUzhMkh9y4Eaud4swibOmVcYL=0PFE28rms8n@mail.gmail.com>
References: <AANLkTinMrd-acs=91xxw4MxHis8-RPSPtfhy7b4gu41O@mail.gmail.com>
	<7277DB0B-9B56-4B57-A574-F2764C11FC10@Holsman.NET>
	<AANLkTinMW7nvmapbkJwOGGNvaLNSrXSBr9c08JnBSHp4@mail.gmail.com>
	<C475DC5D-7CAC-47E4-838B-AED9C8274E5B@Holsman.NET>
	<AANLkTinnP2JB_Dqpxu3+3izOfG8AEB1t7zw00Ow9xdC8@mail.gmail.com>
	<EF3B67E6-26D2-42B3-AC99-01426AADC931@holsman.net>
	<21FEBF88-2D56-48E0-B86C-334E94F04188@gbiv.com>
	<AANLkTi=kVi=k6GV4tC8LNrpX4HaMxqRK72JJwOjN78tL@mail.gmail.com>
	<AANLkTikHfMxBk7SH+4oq94r894W4Qr2MOJWJH15nWeHi@mail.gmail.com>
	<AANLkTikiUzhMkh9y4Eaud4swibOmVcYL=0PFE28rms8n@mail.gmail.com>
Date: Thu, 23 Dec 2010 10:15:38 -0800
Message-ID: <AANLkTi=nhy0a46x3chPbB2S91ujtqTzd=DHPsJCjYECp@mail.gmail.com>
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of
 branch-0.20-append branch?
From: "M. C. Srivas" <mcsrivas@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175cb71a54a5a5049817dbb6

--0015175cb71a54a5a5049817dbb6
Content-Type: text/plain; charset=ISO-8859-1

On Thu, Dec 23, 2010 at 9:38 AM, Jakob Homan <jghoman@gmail.com> wrote:

> It's difficult to support this proposal knowing how much time would be
> spent preparing an official release, continuing to support it and
> continuing to two support two separate implementations of append.  I
> believe that effort would be better spent getting out a kick-ass 22
> (or, barring that, a *really* kick-ass 23).
>

Regardless, there will still be 2 incompatible "branches". And that is only
the beginning.

Some future features will be done only on branch 1 (since company 1 uses
that), and other features on branch 2 (by company 2, since they prefer
branch 2),  thereby further separating the two branches.

If the goal is to avoid the split, then there are only 2 choices:
  (a) merge both
  (b) abandon one or the other.

Which one is one willing to stomach?




>
> The Promised Land that we say we're all trying to get to is regular,
> timely, feature-complete, tested, innovative but stable releases of
> new versions of Apache Hadoop.  Missing out any one of those criteria
> discovered will continue (and has continued) the current situation
> where quasi-official branches and outside distributions fill the void
> such a release should.  The effort to maintain this offical branch and
> fix the bugs that will be discovered could be better spent moving us
> closer to that goal.
>
> I'm certainly sympathetic to the difficult position our quagmire has
> placed HBase into.  However, the current proposal would hurt HDFS to
> help HBase. The best solution for that project, as well as for HDFS,
> is to get HDFS back to a healthy release cycle; not prolong or codify
> the current ad-hoc state of affairs.  Let's stop digging this hole.

-jakob
>
> On Thu, Dec 23, 2010 at 9:33 AM, M. C. Srivas <mcsrivas@gmail.com> wrote:
> > [ Sorry if this is be-laboring the obvious ]
> >
> > There are two append solutions floating around, and they are incompatible
> > with each other. Thus, the two "branches" will forever remain
> incompatible
> > with each other, regardless of how they are numbered (0.22,  0.23,
>  0.20.3,
> > e.t.c.)
> >
> > Unless both are merged into one branch, and a switch provided to  "use
> > HDFS-200 append" or "use 0.22 append", we have effectively split Hadoop
> into
> > two.
> >
> >
> > On Thu, Dec 23, 2010 at 12:00 AM, Owen O'Malley <omalley@apache.org>
> wrote:
> >
> >> On Wed, Dec 22, 2010 at 11:07 PM, Roy T. Fielding <fielding@gbiv.com>
> >> wrote:
> >>
> >> > Features are not release version tags.  If there is a security bug
> >> > found then we would have to release a new version of the append
> >> > version, and a round of severe trout slapping would result.
> >> >
> >>
> >> Yeah, it isn't a perfect solution and it doesn't scale to a second tag,
> but
> >> the problem is that this is effectively a release branch between 0.20
> and
> >> 0.21. Of course I agree that any critical bugs would need to be fixed in
> >> the
> >> append branch as well as the 0.20 and 0.21 branches.
> >>
> >> If you want to stick to pure numbers and we want to leave ourselves a
> way
> >> to
> >> bugfix the 0.20 branch without append, we'd could use a version string
> like
> >> 0.20.100, etc. Not pretty, but it does preserve the numeric ordering and
> >> suggest a version jump.
> >>
> >> If I remember right, there were also protocol changes in the append
> branch,
> >> which was another reason we didn't want to put it directly into the 0.20
> >> branch.
> >>
> >> -- Owen
> >>
> >
>

--0015175cb71a54a5a5049817dbb6--

From general-return-2596-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 19:40:42 2010
Return-Path: <general-return-2596-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77900 invoked from network); 23 Dec 2010 19:40:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 19:40:41 -0000
Received: (qmail 57330 invoked by uid 500); 23 Dec 2010 19:40:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57290 invoked by uid 500); 23 Dec 2010 19:40:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57279 invoked by uid 99); 23 Dec 2010 19:40:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 19:40:40 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ryanobjc@gmail.com designates 209.85.210.176 as permitted sender)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 19:40:35 +0000
Received: by iyb26 with SMTP id 26so6248420iyb.35
        for <general@hadoop.apache.org>; Thu, 23 Dec 2010 11:40:14 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=zOheWhwMJSMzOwUbmfS63Mbvr2yid4OtKiaodsAiXzQ=;
        b=hXx/dP22DbdTf3vW/u+wbesliCHcXpO2qzrJIxZCMhP0EWoFK2QIv44zUtcEnpES9j
         Z+rK80sQHUzIBXJj27DWMn4lh4cMMAbWNRrsoEPGyq4OLhlO1yRACuGQdHqW+NW2jPIs
         70Dbs7xC3Uq0rVAhGZ//lAFfRj1KUXXWZeZzg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=lDo/u54rota1v4YiXsdAygn68bLHXZPWz2RrKzflApzRdeY5tEF3E2IlwFZO2tqjK/
         t+NE2c4SOMPMLTpqJFmu71jtrhGcSqAWTHUlw9NTmyUR+RSo9Opxi0qPM7+FbpCKqhjw
         6PgHeg52Vg9VVe5Mc33YuvPFRHdV6eRiCsxEo=
MIME-Version: 1.0
Received: by 10.231.207.66 with SMTP id fx2mr8460685ibb.108.1293133214465;
 Thu, 23 Dec 2010 11:40:14 -0800 (PST)
Received: by 10.231.38.3 with HTTP; Thu, 23 Dec 2010 11:40:14 -0800 (PST)
In-Reply-To: <AANLkTikiUzhMkh9y4Eaud4swibOmVcYL=0PFE28rms8n@mail.gmail.com>
References: <AANLkTinMrd-acs=91xxw4MxHis8-RPSPtfhy7b4gu41O@mail.gmail.com>
	<7277DB0B-9B56-4B57-A574-F2764C11FC10@Holsman.NET>
	<AANLkTinMW7nvmapbkJwOGGNvaLNSrXSBr9c08JnBSHp4@mail.gmail.com>
	<C475DC5D-7CAC-47E4-838B-AED9C8274E5B@Holsman.NET>
	<AANLkTinnP2JB_Dqpxu3+3izOfG8AEB1t7zw00Ow9xdC8@mail.gmail.com>
	<EF3B67E6-26D2-42B3-AC99-01426AADC931@holsman.net>
	<21FEBF88-2D56-48E0-B86C-334E94F04188@gbiv.com>
	<AANLkTi=kVi=k6GV4tC8LNrpX4HaMxqRK72JJwOjN78tL@mail.gmail.com>
	<AANLkTikHfMxBk7SH+4oq94r894W4Qr2MOJWJH15nWeHi@mail.gmail.com>
	<AANLkTikiUzhMkh9y4Eaud4swibOmVcYL=0PFE28rms8n@mail.gmail.com>
Date: Thu, 23 Dec 2010 11:40:14 -0800
Message-ID: <AANLkTimBj9WMdEf3yQBbGTYw0JsiQQ=rH_mFTSYMZvBq@mail.gmail.com>
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of
 branch-0.20-append branch?
From: Ryan Rawson <ryanobjc@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

The append solution in 0.22 that you are referring to was supposed to
be out 13-15 months ago.  Pardon if I look for solutions that deploy 4
months ago (as the 0.20 append branch did).

Another 12-15 months of delay is not exactly helping HDFS either.

-ryan

On Thu, Dec 23, 2010 at 9:38 AM, Jakob Homan <jghoman@gmail.com> wrote:
> It's difficult to support this proposal knowing how much time would be
> spent preparing an official release, continuing to support it and
> continuing to two support two separate implementations of append. =A0I
> believe that effort would be better spent getting out a kick-ass 22
> (or, barring that, a *really* kick-ass 23).
>
> The Promised Land that we say we're all trying to get to is regular,
> timely, feature-complete, tested, innovative but stable releases of
> new versions of Apache Hadoop. =A0Missing out any one of those criteria
> discovered will continue (and has continued) the current situation
> where quasi-official branches and outside distributions fill the void
> such a release should. =A0The effort to maintain this offical branch and
> fix the bugs that will be discovered could be better spent moving us
> closer to that goal.
>
> I'm certainly sympathetic to the difficult position our quagmire has
> placed HBase into. =A0However, the current proposal would hurt HDFS to
> help HBase. The best solution for that project, as well as for HDFS,
> is to get HDFS back to a healthy release cycle; not prolong or codify
> the current ad-hoc state of affairs. =A0Let's stop digging this hole.
> -jakob
>
> On Thu, Dec 23, 2010 at 9:33 AM, M. C. Srivas <mcsrivas@gmail.com> wrote:
>> [ Sorry if this is be-laboring the obvious ]
>>
>> There are two append solutions floating around, and they are incompatibl=
e
>> with each other. Thus, the two "branches" will forever remain incompatib=
le
>> with each other, regardless of how they are numbered (0.22, =A00.23, =A0=
0.20.3,
>> e.t.c.)
>>
>> Unless both are merged into one branch, and a switch provided to =A0"use
>> HDFS-200 append" or "use 0.22 append", we have effectively split Hadoop =
into
>> two.
>>
>>
>> On Thu, Dec 23, 2010 at 12:00 AM, Owen O'Malley <omalley@apache.org> wro=
te:
>>
>>> On Wed, Dec 22, 2010 at 11:07 PM, Roy T. Fielding <fielding@gbiv.com>
>>> wrote:
>>>
>>> > Features are not release version tags. =A0If there is a security bug
>>> > found then we would have to release a new version of the append
>>> > version, and a round of severe trout slapping would result.
>>> >
>>>
>>> Yeah, it isn't a perfect solution and it doesn't scale to a second tag,=
 but
>>> the problem is that this is effectively a release branch between 0.20 a=
nd
>>> 0.21. Of course I agree that any critical bugs would need to be fixed i=
n
>>> the
>>> append branch as well as the 0.20 and 0.21 branches.
>>>
>>> If you want to stick to pure numbers and we want to leave ourselves a w=
ay
>>> to
>>> bugfix the 0.20 branch without append, we'd could use a version string =
like
>>> 0.20.100, etc. Not pretty, but it does preserve the numeric ordering an=
d
>>> suggest a version jump.
>>>
>>> If I remember right, there were also protocol changes in the append bra=
nch,
>>> which was another reason we didn't want to put it directly into the 0.2=
0
>>> branch.
>>>
>>> -- Owen
>>>
>>
>

From general-return-2597-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 20:01:38 2010
Return-Path: <general-return-2597-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81425 invoked from network); 23 Dec 2010 20:01:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 20:01:38 -0000
Received: (qmail 75072 invoked by uid 500); 23 Dec 2010 20:01:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75028 invoked by uid 500); 23 Dec 2010 20:01:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75020 invoked by uid 99); 23 Dec 2010 20:01:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 20:01:36 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 20:01:28 +0000
Received: by iwn2 with SMTP id 2so7408596iwn.35
        for <general@hadoop.apache.org>; Thu, 23 Dec 2010 12:01:05 -0800 (PST)
Received: by 10.231.144.21 with SMTP id x21mr8478181ibu.30.1293134465546; Thu,
 23 Dec 2010 12:01:05 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.188.102 with HTTP; Thu, 23 Dec 2010 12:00:45 -0800 (PST)
In-Reply-To: <AANLkTi=nhy0a46x3chPbB2S91ujtqTzd=DHPsJCjYECp@mail.gmail.com>
References: <AANLkTinMrd-acs=91xxw4MxHis8-RPSPtfhy7b4gu41O@mail.gmail.com>
 <7277DB0B-9B56-4B57-A574-F2764C11FC10@Holsman.NET> <AANLkTinMW7nvmapbkJwOGGNvaLNSrXSBr9c08JnBSHp4@mail.gmail.com>
 <C475DC5D-7CAC-47E4-838B-AED9C8274E5B@Holsman.NET> <AANLkTinnP2JB_Dqpxu3+3izOfG8AEB1t7zw00Ow9xdC8@mail.gmail.com>
 <EF3B67E6-26D2-42B3-AC99-01426AADC931@holsman.net> <21FEBF88-2D56-48E0-B86C-334E94F04188@gbiv.com>
 <AANLkTi=kVi=k6GV4tC8LNrpX4HaMxqRK72JJwOjN78tL@mail.gmail.com>
 <AANLkTikHfMxBk7SH+4oq94r894W4Qr2MOJWJH15nWeHi@mail.gmail.com>
 <AANLkTikiUzhMkh9y4Eaud4swibOmVcYL=0PFE28rms8n@mail.gmail.com> <AANLkTi=nhy0a46x3chPbB2S91ujtqTzd=DHPsJCjYECp@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Thu, 23 Dec 2010 12:00:45 -0800
Message-ID: <AANLkTi=q3bMy6xAj6c6kKD=kgrABVD7Rh2JS=GqcTSm5@mail.gmail.com>
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of
 branch-0.20-append branch?
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64761da7738690498195418
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e64761da7738690498195418
Content-Type: text/plain; charset=ISO-8859-1

On Thu, Dec 23, 2010 at 10:15 AM, M. C. Srivas <mcsrivas@gmail.com> wrote:

> Regardless, there will still be 2 incompatible "branches". And that is only
> the beginning.
>
> Some future features will be done only on branch 1 (since company 1 uses
> that), and other features on branch 2 (by company 2, since they prefer
> branch 2),  thereby further separating the two branches.
>
> If the goal is to avoid the split, then there are only 2 choices:
>  (a) merge both
>  (b) abandon one or the other.
>
>
The 0.20 append solution has never been seen as a fork. It's a stop-gap
fixup of the 0.20 append feature, but we don't intend to forward-port that
append implementation into trunk. From an API perspective it's very close to
the 0.22 version, and I think everyone fully intends to abandon the
0.20-append work once 0.22 append has been heavily tested for HBase
workloads.


>
> >
> > The Promised Land that we say we're all trying to get to is regular,
> > timely, feature-complete, tested, innovative but stable releases of
> > new versions of Apache Hadoop.  Missing out any one of those criteria
> > discovered will continue (and has continued) the current situation
> > where quasi-official branches and outside distributions fill the void
> > such a release should.  The effort to maintain this offical branch and
> > fix the bugs that will be discovered could be better spent moving us
> > closer to that goal.
> >
>

+1. Interestingly, the work on 0.20-append uncovered a number of bugs that
also will apply to 0.22's implementation. So it wasn't all a wasted effort
;-)

-- 
Todd Lipcon
Software Engineer, Cloudera

--0016e64761da7738690498195418--

From general-return-2598-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 20:03:41 2010
Return-Path: <general-return-2598-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81807 invoked from network); 23 Dec 2010 20:03:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 20:03:40 -0000
Received: (qmail 78203 invoked by uid 500); 23 Dec 2010 20:03:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78108 invoked by uid 500); 23 Dec 2010 20:03:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78100 invoked by uid 99); 23 Dec 2010 20:03:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 20:03:38 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 20:03:33 +0000
Received: by iyb26 with SMTP id 26so6265879iyb.35
        for <general@hadoop.apache.org>; Thu, 23 Dec 2010 12:03:12 -0800 (PST)
Received: by 10.231.33.202 with SMTP id i10mr8430354ibd.55.1293134592481; Thu,
 23 Dec 2010 12:03:12 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.188.102 with HTTP; Thu, 23 Dec 2010 12:02:51 -0800 (PST)
In-Reply-To: <20101223115004.7f8dc48c@pcmoholynagy.informatik.uni-stuttgart.de>
References: <20101222160311.7ffb9b6d@laptop> <AANLkTike7k5nqViAsdyC_mR6LOBVrrFFqsyGJo=-W6GQ@mail.gmail.com>
 <AANLkTi=9_-CPn+GON+YqLra1sjDX3-i-tqZ4rv39rTfT@mail.gmail.com> <20101223115004.7f8dc48c@pcmoholynagy.informatik.uni-stuttgart.de>
From: Todd Lipcon <todd@cloudera.com>
Date: Thu, 23 Dec 2010 12:02:51 -0800
Message-ID: <AANLkTimvsa__1gC5iL4PNGNt_sgTgTGB7UR4BxSbVyNM@mail.gmail.com>
Subject: Re: namenode doesn't start after reboot
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=002215048dc7081c9a0498195cb3

--002215048dc7081c9a0498195cb3
Content-Type: text/plain; charset=ISO-8859-1

On Thu, Dec 23, 2010 at 2:50 AM, Bjoern Schiessle <bjoern@schiessle.org>wrote:

>
> 1. I have set up a second dfs.name.dir which is stored at another
> computer (mounted by sshfs)
>

I would strongly discourage the use of sshfs for the name dir. For one, it's
slow, and for two, I've sen it have some really weird semantics where it's
doing write-back caching.

Just take a look at its manpage and you should get scared about using it for
a critical mount point like this.

A soft interruptable NFS mount is a much safer bet.

-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

--002215048dc7081c9a0498195cb3--

From general-return-2599-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 20:48:26 2010
Return-Path: <general-return-2599-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96775 invoked from network); 23 Dec 2010 20:48:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 20:48:25 -0000
Received: (qmail 21061 invoked by uid 500); 23 Dec 2010 20:48:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20982 invoked by uid 500); 23 Dec 2010 20:48:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 20974 invoked by uid 99); 23 Dec 2010 20:48:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 20:48:24 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jghoman@gmail.com designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 20:48:17 +0000
Received: by ewy20 with SMTP id 20so322769ewy.35
        for <general@hadoop.apache.org>; Thu, 23 Dec 2010 12:47:57 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type;
        bh=7HSVmee+ZY0YkaXp8No5Qstmr7HWaawz6TYDVjIfu5o=;
        b=DaXu7wnO3+yQaeSlsPHtqUd5JrqViZGPOMC0WPjSQqJAn6YfOWG9FT0x2m+2NojBZm
         p2tN570DPbqIc5K4BUP+65C9mCvn7qWzS0oIRBmS6Dwab9pGAxC7O57Lf8qdW9ZgMAnd
         3fuqHmWwEr27xrLvCgoyKgYzbyZdP4Bxomoxo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=ku0QwiHFKm25XH1HchWLOKMTUyO/D7WY17TaFHlDUa1Z3axF1Qf+AVLWLQNTNHtC2g
         tn3Z/DWrvX0qDTOuXcJieGTZ5MSFjow5BFKUbHn5WXd6Xcprt/o9vt3KxgjJYISppPVH
         RrcS9WZpEH8QnMMye3wMAqoZYtG0TS5G6pqkY=
Received: by 10.213.15.11 with SMTP id i11mr6973816eba.98.1293137277648; Thu,
 23 Dec 2010 12:47:57 -0800 (PST)
MIME-Version: 1.0
Received: by 10.213.26.15 with HTTP; Thu, 23 Dec 2010 12:47:37 -0800 (PST)
In-Reply-To: <AANLkTimvsa__1gC5iL4PNGNt_sgTgTGB7UR4BxSbVyNM@mail.gmail.com>
References: <20101222160311.7ffb9b6d@laptop> <AANLkTike7k5nqViAsdyC_mR6LOBVrrFFqsyGJo=-W6GQ@mail.gmail.com>
 <AANLkTi=9_-CPn+GON+YqLra1sjDX3-i-tqZ4rv39rTfT@mail.gmail.com>
 <20101223115004.7f8dc48c@pcmoholynagy.informatik.uni-stuttgart.de> <AANLkTimvsa__1gC5iL4PNGNt_sgTgTGB7UR4BxSbVyNM@mail.gmail.com>
From: Jakob Homan <jghoman@gmail.com>
Date: Thu, 23 Dec 2010 12:47:37 -0800
Message-ID: <AANLkTi=acodAzbcafYW0n5v6eQ-m8S-rPaN-efdEcxV0@mail.gmail.com>
Subject: Re: namenode doesn't start after reboot
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Please move discussions of CDH issues to Cloudera's lists.  Thanks.

On Thu, Dec 23, 2010 at 12:02 PM, Todd Lipcon <todd@cloudera.com> wrote:
> On Thu, Dec 23, 2010 at 2:50 AM, Bjoern Schiessle <bjoern@schiessle.org>wrote:
>
>>
>> 1. I have set up a second dfs.name.dir which is stored at another
>> computer (mounted by sshfs)
>>
>
> I would strongly discourage the use of sshfs for the name dir. For one, it's
> slow, and for two, I've sen it have some really weird semantics where it's
> doing write-back caching.
>
> Just take a look at its manpage and you should get scared about using it for
> a critical mount point like this.
>
> A soft interruptable NFS mount is a much safer bet.
>
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

From general-return-2600-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 21:08:53 2010
Return-Path: <general-return-2600-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9314 invoked from network); 23 Dec 2010 21:08:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 21:08:53 -0000
Received: (qmail 39569 invoked by uid 500); 23 Dec 2010 21:08:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39489 invoked by uid 500); 23 Dec 2010 21:08:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39481 invoked by uid 99); 23 Dec 2010 21:08:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 21:08:52 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 21:08:47 +0000
Received: by iyb26 with SMTP id 26so6313911iyb.35
        for <general@hadoop.apache.org>; Thu, 23 Dec 2010 13:08:26 -0800 (PST)
Received: by 10.231.199.77 with SMTP id er13mr4083278ibb.105.1293138506868;
 Thu, 23 Dec 2010 13:08:26 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.188.102 with HTTP; Thu, 23 Dec 2010 13:08:06 -0800 (PST)
In-Reply-To: <AANLkTi=acodAzbcafYW0n5v6eQ-m8S-rPaN-efdEcxV0@mail.gmail.com>
References: <20101222160311.7ffb9b6d@laptop> <AANLkTike7k5nqViAsdyC_mR6LOBVrrFFqsyGJo=-W6GQ@mail.gmail.com>
 <AANLkTi=9_-CPn+GON+YqLra1sjDX3-i-tqZ4rv39rTfT@mail.gmail.com>
 <20101223115004.7f8dc48c@pcmoholynagy.informatik.uni-stuttgart.de>
 <AANLkTimvsa__1gC5iL4PNGNt_sgTgTGB7UR4BxSbVyNM@mail.gmail.com> <AANLkTi=acodAzbcafYW0n5v6eQ-m8S-rPaN-efdEcxV0@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Thu, 23 Dec 2010 13:08:06 -0800
Message-ID: <AANLkTik9ZbzpBHfEx-N3RCBqxecFBSvuz+8Nh7GqxPu3@mail.gmail.com>
Subject: Re: namenode doesn't start after reboot
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba47682b58e8bc04981a4502

--90e6ba47682b58e8bc04981a4502
Content-Type: text/plain; charset=ISO-8859-1

On Thu, Dec 23, 2010 at 12:47 PM, Jakob Homan <jghoman@gmail.com> wrote:

> Please move discussions of CDH issues to Cloudera's lists.  Thanks.
>

Hi Jakob,

These bugs are clearly not CDH-specific. NameNode corruption bugs, and best
practices with regard to the storage of NN metadata, are clearly applicable
to any version of Hadoop that users may run, be it Apache, Yahoo, Facebook,
0.20, 0.21, or trunk. If you have reason to believe my suggestion you quoted
below is somehow not relevant to the larger community I would love to hear
it.

My understanding of the ASF goals is that we should encourage a cohesive
community. Asking users of CDH to move general Hadoop questions off of ASF
mailing lists just because of their choice in distros encourages a fractured
community rather than a cohesive one.

Clearly. if a user has a question specifically about Cloudera packaging they
should be directed to the CDH lists so as not to clutter non-CDH users'
inboxes with irrelevant questions. I think if you browse the archives you'll
find that Cloudera employees have been consistent about doing this since we
started the cdh-user list several months ago. But if an issue is a bug that
is likely to occur in trunk, it makes sense to me to leave it on the list
associated with the core project.

Personally I do my best to answer questions on the ASF lists regardless of
which distro the person is using - though our distros have some divergence
in backported patch sets, it's rare that a bug in one distro doesn't allow
us to fix a bug in trunk. I can readily pull up several recent examples of
this, and I'm surprised that there isn't more concern in the general
community about bugs that may result in NN metadata corruption.

Thanks,
-Todd


>
> On Thu, Dec 23, 2010 at 12:02 PM, Todd Lipcon <todd@cloudera.com> wrote:
> > On Thu, Dec 23, 2010 at 2:50 AM, Bjoern Schiessle <bjoern@schiessle.org
> >wrote:
> >
> >>
> >> 1. I have set up a second dfs.name.dir which is stored at another
> >> computer (mounted by sshfs)
> >>
> >
> > I would strongly discourage the use of sshfs for the name dir. For one,
> it's
> > slow, and for two, I've sen it have some really weird semantics where
> it's
> > doing write-back caching.
> >
> > Just take a look at its manpage and you should get scared about using it
> for
> > a critical mount point like this.
> >
> > A soft interruptable NFS mount is a much safer bet.
> >
> > -Todd
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--90e6ba47682b58e8bc04981a4502--

From general-return-2601-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 22:05:20 2010
Return-Path: <general-return-2601-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29646 invoked from network); 23 Dec 2010 22:05:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 22:05:20 -0000
Received: (qmail 87505 invoked by uid 500); 23 Dec 2010 22:05:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87297 invoked by uid 500); 23 Dec 2010 22:05:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87289 invoked by uid 99); 23 Dec 2010 22:05:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 22:05:13 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bjoern@schiessle.org designates 78.46.69.5 as permitted sender)
Received: from [78.46.69.5] (HELO zucker.schokokeks.org) (78.46.69.5)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 22:05:05 +0000
Received: from woody.localdomain (HSI-KBW-095-208-083-064.hsi5.kabel-badenwuerttemberg.de [::ffff:95.208.83.64])
  (AUTH: PLAIN schiesbn@schokokeks.org, TLS: TLSv1/SSLv3,256bits,AES256-SHA)
  by zucker.schokokeks.org with ESMTPSA; Thu, 23 Dec 2010 23:04:43 +0100
  id 0000000000018019.000000004D13C77B.00001EE5
Received: from woody (localhost [127.0.0.1])
	by woody.localdomain (Postfix) with ESMTP id 562E487DDD;
	Thu, 23 Dec 2010 23:05:06 +0100 (CET)
Date: Thu, 23 Dec 2010 23:05:05 +0100
From: Bjoern Schiessle <bjoern@schiessle.org>
To: general@hadoop.apache.org
Subject: Re: namenode doesn't start after reboot
Message-ID: <20101223230505.1f4105f3@woody>
In-Reply-To: <AANLkTin+n_GL5CtLsMrCgdz0URKkWO0vtQCsB-jca51y@mail.gmail.com>
References: <20101222160311.7ffb9b6d@laptop>
	<AANLkTike7k5nqViAsdyC_mR6LOBVrrFFqsyGJo=-W6GQ@mail.gmail.com>
	<AANLkTi=9_-CPn+GON+YqLra1sjDX3-i-tqZ4rv39rTfT@mail.gmail.com>
	<20101223115004.7f8dc48c@pcmoholynagy.informatik.uni-stuttgart.de>
	<AANLkTimjm8vn54VKJGDvTMx07YNWUa0ofZJO2QiwkUs+@mail.gmail.com>
	<AANLkTinhHSKq18aQvz-ALLz+d-4Rjt23SQ8bm_CH2s4c@mail.gmail.com>
	<AANLkTin+n_GL5CtLsMrCgdz0URKkWO0vtQCsB-jca51y@mail.gmail.com>
X-Mailer: Claws Mail 3.5.0 (GTK+ 2.12.12; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi,

On Thu, 23 Dec 2010 09:15:41 -0800 Aaron T. Myers wrote:
> All this aside, you really shouldn't have to "safely" stop all the
> Hadoop services when you reboot any of your servers. Hadoop should be
> able to survive a crash of any of the daemons. Any circumstance in
> which Hadoop currently corrupts the edits log or fsimage is a serious
> bug, and should be reported via JIRA.

this is also what I would expect. Nevertheless the last reboot caused the
problem described at the beginning of the thread. During the tests today
I always stopped the Datanode and Namenode by my own which works
flawlessly. To be a little bit more safe I wrote my own stop-script which
stops the datanode and the namenode before shutdown.

best wishes,
Bj=C3=B6rn

From general-return-2602-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 22:06:27 2010
Return-Path: <general-return-2602-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31413 invoked from network); 23 Dec 2010 22:06:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 22:06:26 -0000
Received: (qmail 89444 invoked by uid 500); 23 Dec 2010 22:06:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89401 invoked by uid 500); 23 Dec 2010 22:06:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89393 invoked by uid 99); 23 Dec 2010 22:06:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 22:06:25 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bjoern@schiessle.org designates 78.46.69.5 as permitted sender)
Received: from [78.46.69.5] (HELO zucker.schokokeks.org) (78.46.69.5)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 22:06:18 +0000
Received: from woody.localdomain (HSI-KBW-095-208-083-064.hsi5.kabel-badenwuerttemberg.de [::ffff:95.208.83.64])
  (AUTH: PLAIN schiesbn@schokokeks.org, TLS: TLSv1/SSLv3,256bits,AES256-SHA)
  by zucker.schokokeks.org with ESMTPSA; Thu, 23 Dec 2010 23:05:57 +0100
  id 000000000001802A.000000004D13C7C5.00002047
Received: from woody (localhost [127.0.0.1])
	by woody.localdomain (Postfix) with ESMTP id 3FDA687DDD;
	Thu, 23 Dec 2010 23:06:21 +0100 (CET)
Date: Thu, 23 Dec 2010 23:06:20 +0100
From: Bjoern Schiessle <bjoern@schiessle.org>
To: general@hadoop.apache.org
Subject: Re: namenode doesn't start after reboot
Message-ID: <20101223230620.6aa117d3@woody>
In-Reply-To: <AANLkTimvsa__1gC5iL4PNGNt_sgTgTGB7UR4BxSbVyNM@mail.gmail.com>
References: <20101222160311.7ffb9b6d@laptop>
	<AANLkTike7k5nqViAsdyC_mR6LOBVrrFFqsyGJo=-W6GQ@mail.gmail.com>
	<AANLkTi=9_-CPn+GON+YqLra1sjDX3-i-tqZ4rv39rTfT@mail.gmail.com>
	<20101223115004.7f8dc48c@pcmoholynagy.informatik.uni-stuttgart.de>
	<AANLkTimvsa__1gC5iL4PNGNt_sgTgTGB7UR4BxSbVyNM@mail.gmail.com>
X-Mailer: Claws Mail 3.5.0 (GTK+ 2.12.12; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On Thu, 23 Dec 2010 12:02:51 -0800 Todd Lipcon wrote:
> On Thu, Dec 23, 2010 at 2:50 AM, Bjoern Schiessle
> <bjoern@schiessle.org>wrote:
>=20
> >
> > 1. I have set up a second dfs.name.dir which is stored at another
> > computer (mounted by sshfs)
> >
>=20
> I would strongly discourage the use of sshfs for the name dir. For one,
> it's slow, and for two, I've sen it have some really weird semantics
> where it's doing write-back caching.

Thanks for this insights. I now switched to NFS.

Thanks,
Bj=C3=B6rn

From general-return-2603-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 22:19:01 2010
Return-Path: <general-return-2603-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33984 invoked from network); 23 Dec 2010 22:19:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 22:19:00 -0000
Received: (qmail 1028 invoked by uid 500); 23 Dec 2010 22:18:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 758 invoked by uid 500); 23 Dec 2010 22:18:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 749 invoked by uid 99); 23 Dec 2010 22:18:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 22:18:59 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of shv.hadoop@gmail.com designates 209.85.213.176 as permitted sender)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 22:18:54 +0000
Received: by yxm8 with SMTP id 8so3369386yxm.35
        for <general@hadoop.apache.org>; Thu, 23 Dec 2010 14:18:33 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=I7F4rgIFKt5pfDmfqDzlGHhNG57ftYXH85/61BXmczE=;
        b=dBjBp/+AHefy4SD9+mIkjAvbAw+RtsuCmuXnbkKhWSQRydwxl/FKdgDLl6bSBe2+P+
         9cc9QHmHtf25WSFH8q1IGYmMVszJKU+dCRiCyioTakAFTu4hr1B9MYJKBkdMc4MfGhda
         Phv+plSACWZKIkyKFMLrP50W3QmYWrM0CZHQQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=pKeI50rTpvo4CLcg1TdV5ktSaZQTeBgZdwpd7cUAvaLT1vzo4686gSg0dQhNbivVkP
         HqkotDmQ2nTGeWY8OVH7zZUx/P+oHF82lw6xgQ6rlD6rSV6AlZJ+p6ZwspWar/pY73CP
         BcJSFIH/kmVRagiYmFWTu0UjpEjQHdlQXUh4U=
MIME-Version: 1.0
Received: by 10.236.103.44 with SMTP id e32mr6761101yhg.37.1293142713244; Thu,
 23 Dec 2010 14:18:33 -0800 (PST)
Received: by 10.236.109.15 with HTTP; Thu, 23 Dec 2010 14:18:33 -0800 (PST)
In-Reply-To: <AANLkTimBj9WMdEf3yQBbGTYw0JsiQQ=rH_mFTSYMZvBq@mail.gmail.com>
References: <AANLkTinMrd-acs=91xxw4MxHis8-RPSPtfhy7b4gu41O@mail.gmail.com>
	<7277DB0B-9B56-4B57-A574-F2764C11FC10@Holsman.NET>
	<AANLkTinMW7nvmapbkJwOGGNvaLNSrXSBr9c08JnBSHp4@mail.gmail.com>
	<C475DC5D-7CAC-47E4-838B-AED9C8274E5B@Holsman.NET>
	<AANLkTinnP2JB_Dqpxu3+3izOfG8AEB1t7zw00Ow9xdC8@mail.gmail.com>
	<EF3B67E6-26D2-42B3-AC99-01426AADC931@holsman.net>
	<21FEBF88-2D56-48E0-B86C-334E94F04188@gbiv.com>
	<AANLkTi=kVi=k6GV4tC8LNrpX4HaMxqRK72JJwOjN78tL@mail.gmail.com>
	<AANLkTikHfMxBk7SH+4oq94r894W4Qr2MOJWJH15nWeHi@mail.gmail.com>
	<AANLkTikiUzhMkh9y4Eaud4swibOmVcYL=0PFE28rms8n@mail.gmail.com>
	<AANLkTimBj9WMdEf3yQBbGTYw0JsiQQ=rH_mFTSYMZvBq@mail.gmail.com>
Date: Thu, 23 Dec 2010 14:18:33 -0800
Message-ID: <AANLkTi=zTJ8oTT-aXCBYvCyjARYKmOHHt52YFmgggy8X@mail.gmail.com>
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of
 branch-0.20-append branch?
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0023547c8c01111adc04981b4006

--0023547c8c01111adc04981b4006
Content-Type: text/plain; charset=ISO-8859-1

I also think building 0.20-append will be a major distraction from moving
0.22 forward with all the great new features, including the new append
implementation, sitting on the bench because we are delaying the release.
It seems to be beneficial for the entire community to focus on 0.22 rather
than chasing both birds.

I hear a concern that 0.22 will lack large scale testing as was the case
with 0.21.
I'd like to volunteer to put as many large scale resources, as I can grasp,
into stabilizing of 0.22. Under Nigel's management of course.
This should get us to production quality in 3-6 months rather than
"another 12-15". I also hope it can go even faster/better if others
could join the effort. I see > 100 companies claiming they are powered by
Apache Hadoop.

I also hope with this effort HBase will be able to start moving to the new
append implementation in the next 2-3 months, which in turn will help 0.22
HDFS
rather than divert resources from it as it would have be with 0.20-append.

Stack, will this plan will work for HBase survival?

One other thought. Apache Hadoop community is not in control of external
releases and distributions, but we should not fork our own releases by
introducing
competing apis. If we can keep the dev line relatively straight the external
releases
will follow.

Thanks,
--Konstantin


On Thu, Dec 23, 2010 at 11:40 AM, Ryan Rawson <ryanobjc@gmail.com> wrote:

> The append solution in 0.22 that you are referring to was supposed to
> be out 13-15 months ago.  Pardon if I look for solutions that deploy 4
> months ago (as the 0.20 append branch did).
>
> Another 12-15 months of delay is not exactly helping HDFS either.
>
> -ryan
>
> On Thu, Dec 23, 2010 at 9:38 AM, Jakob Homan <jghoman@gmail.com> wrote:
> > It's difficult to support this proposal knowing how much time would be
> > spent preparing an official release, continuing to support it and
> > continuing to two support two separate implementations of append.  I
> > believe that effort would be better spent getting out a kick-ass 22
> > (or, barring that, a *really* kick-ass 23).
> >
> > The Promised Land that we say we're all trying to get to is regular,
> > timely, feature-complete, tested, innovative but stable releases of
> > new versions of Apache Hadoop.  Missing out any one of those criteria
> > discovered will continue (and has continued) the current situation
> > where quasi-official branches and outside distributions fill the void
> > such a release should.  The effort to maintain this offical branch and
> > fix the bugs that will be discovered could be better spent moving us
> > closer to that goal.
> >
> > I'm certainly sympathetic to the difficult position our quagmire has
> > placed HBase into.  However, the current proposal would hurt HDFS to
> > help HBase. The best solution for that project, as well as for HDFS,
> > is to get HDFS back to a healthy release cycle; not prolong or codify
> > the current ad-hoc state of affairs.  Let's stop digging this hole.
> > -jakob
> >
> > On Thu, Dec 23, 2010 at 9:33 AM, M. C. Srivas <mcsrivas@gmail.com>
> wrote:
> >> [ Sorry if this is be-laboring the obvious ]
> >>
> >> There are two append solutions floating around, and they are
> incompatible
> >> with each other. Thus, the two "branches" will forever remain
> incompatible
> >> with each other, regardless of how they are numbered (0.22,  0.23,
>  0.20.3,
> >> e.t.c.)
> >>
> >> Unless both are merged into one branch, and a switch provided to  "use
> >> HDFS-200 append" or "use 0.22 append", we have effectively split Hadoop
> into
> >> two.
> >>
> >>
> >> On Thu, Dec 23, 2010 at 12:00 AM, Owen O'Malley <omalley@apache.org>
> wrote:
> >>
> >>> On Wed, Dec 22, 2010 at 11:07 PM, Roy T. Fielding <fielding@gbiv.com>
> >>> wrote:
> >>>
> >>> > Features are not release version tags.  If there is a security bug
> >>> > found then we would have to release a new version of the append
> >>> > version, and a round of severe trout slapping would result.
> >>> >
> >>>
> >>> Yeah, it isn't a perfect solution and it doesn't scale to a second tag,
> but
> >>> the problem is that this is effectively a release branch between 0.20
> and
> >>> 0.21. Of course I agree that any critical bugs would need to be fixed
> in
> >>> the
> >>> append branch as well as the 0.20 and 0.21 branches.
> >>>
> >>> If you want to stick to pure numbers and we want to leave ourselves a
> way
> >>> to
> >>> bugfix the 0.20 branch without append, we'd could use a version string
> like
> >>> 0.20.100, etc. Not pretty, but it does preserve the numeric ordering
> and
> >>> suggest a version jump.
> >>>
> >>> If I remember right, there were also protocol changes in the append
> branch,
> >>> which was another reason we didn't want to put it directly into the
> 0.20
> >>> branch.
> >>>
> >>> -- Owen
> >>>
> >>
> >
>

--0023547c8c01111adc04981b4006--

From general-return-2604-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 22:39:37 2010
Return-Path: <general-return-2604-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38353 invoked from network); 23 Dec 2010 22:39:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 22:39:37 -0000
Received: (qmail 17376 invoked by uid 500); 23 Dec 2010 22:39:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17329 invoked by uid 500); 23 Dec 2010 22:39:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17321 invoked by uid 99); 23 Dec 2010 22:39:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 22:39:35 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ryanobjc@gmail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 22:39:28 +0000
Received: by iwn2 with SMTP id 2so7524948iwn.35
        for <general@hadoop.apache.org>; Thu, 23 Dec 2010 14:39:07 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=4hou5W2I6qjMWb2nLtNqeTUaggvgFNRboKoxRHWzWi0=;
        b=dwKQVW2cAsKsMVkXigzvcYjfFbl6AcX0Z5ko7ApCGU2CDUMzOSLzKjYYwqvVJWelZD
         QpAn8uKq+19FJ4iVmN6cLSD933V4AswA+VF+d30VGbenAs8F9dylMDr1yx6pfnGEjzfi
         TKEjVJppnJmJbR8oS2ZxWNsXLi5MjrFagBhfk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=R6racP4QUSAaISYHMJ0k/bXq47cdl68WbR+cS7dXmEP3F7cwgCY3U87XMEGRQ/tgF7
         5GU6xT5ekt+JrrNPIY9U27mQwLnIITaaj0QyrjQ4dRJsljwpnNB5mIjhObvRhz/eAZoA
         iv8qgBGmsYmMWmjkCnbry9kGVMsmxe98EpI1k=
MIME-Version: 1.0
Received: by 10.231.38.6 with SMTP id z6mr8739406ibd.8.1293143947338; Thu, 23
 Dec 2010 14:39:07 -0800 (PST)
Received: by 10.231.38.3 with HTTP; Thu, 23 Dec 2010 14:39:07 -0800 (PST)
In-Reply-To: <AANLkTi=zTJ8oTT-aXCBYvCyjARYKmOHHt52YFmgggy8X@mail.gmail.com>
References: <AANLkTinMrd-acs=91xxw4MxHis8-RPSPtfhy7b4gu41O@mail.gmail.com>
	<7277DB0B-9B56-4B57-A574-F2764C11FC10@Holsman.NET>
	<AANLkTinMW7nvmapbkJwOGGNvaLNSrXSBr9c08JnBSHp4@mail.gmail.com>
	<C475DC5D-7CAC-47E4-838B-AED9C8274E5B@Holsman.NET>
	<AANLkTinnP2JB_Dqpxu3+3izOfG8AEB1t7zw00Ow9xdC8@mail.gmail.com>
	<EF3B67E6-26D2-42B3-AC99-01426AADC931@holsman.net>
	<21FEBF88-2D56-48E0-B86C-334E94F04188@gbiv.com>
	<AANLkTi=kVi=k6GV4tC8LNrpX4HaMxqRK72JJwOjN78tL@mail.gmail.com>
	<AANLkTikHfMxBk7SH+4oq94r894W4Qr2MOJWJH15nWeHi@mail.gmail.com>
	<AANLkTikiUzhMkh9y4Eaud4swibOmVcYL=0PFE28rms8n@mail.gmail.com>
	<AANLkTimBj9WMdEf3yQBbGTYw0JsiQQ=rH_mFTSYMZvBq@mail.gmail.com>
	<AANLkTi=zTJ8oTT-aXCBYvCyjARYKmOHHt52YFmgggy8X@mail.gmail.com>
Date: Thu, 23 Dec 2010 14:39:07 -0800
Message-ID: <AANLkTinHt928uox=NQju5Vey91YfTN28NzTA6wZEK8X8@mail.gmail.com>
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of
 branch-0.20-append branch?
From: Ryan Rawson <ryanobjc@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

How does stack volunteering his time to release an existing branch
divert resources?

Without an ASF release of 0.20-append I will keep having to recommend
an external vendor's release of Hadoop.


On Thu, Dec 23, 2010 at 2:18 PM, Konstantin Shvachko
<shv.hadoop@gmail.com> wrote:
> I also think building 0.20-append will be a major distraction from moving
> 0.22 forward with all the great new features, including the new append
> implementation, sitting on the bench because we are delaying the release.
> It seems to be beneficial for the entire community to focus on 0.22 rathe=
r
> than chasing both birds.
>
> I hear a concern that 0.22 will lack large scale testing as was the case
> with 0.21.
> I'd like to volunteer to put as many large scale resources, as I can gras=
p,
> into stabilizing of 0.22. Under Nigel's management of course.
> This should get us to production quality in 3-6 months rather than
> "another 12-15". I also hope it can go even faster/better if others
> could join the effort. I see > 100 companies claiming they are powered by
> Apache Hadoop.
>
> I also hope with this effort HBase will be able to start moving to the ne=
w
> append implementation in the next 2-3 months, which in turn will help 0.2=
2
> HDFS
> rather than divert resources from it as it would have be with 0.20-append=
.
>
> Stack, will this plan will work for HBase survival?
>
> One other thought. Apache Hadoop community is not in control of external
> releases and distributions, but we should not fork our own releases by
> introducing
> competing apis. If we can keep the dev line relatively straight the exter=
nal
> releases
> will follow.
>
> Thanks,
> --Konstantin
>
>
> On Thu, Dec 23, 2010 at 11:40 AM, Ryan Rawson <ryanobjc@gmail.com> wrote:
>
>> The append solution in 0.22 that you are referring to was supposed to
>> be out 13-15 months ago. =A0Pardon if I look for solutions that deploy 4
>> months ago (as the 0.20 append branch did).
>>
>> Another 12-15 months of delay is not exactly helping HDFS either.
>>
>> -ryan
>>
>> On Thu, Dec 23, 2010 at 9:38 AM, Jakob Homan <jghoman@gmail.com> wrote:
>> > It's difficult to support this proposal knowing how much time would be
>> > spent preparing an official release, continuing to support it and
>> > continuing to two support two separate implementations of append. =A0I
>> > believe that effort would be better spent getting out a kick-ass 22
>> > (or, barring that, a *really* kick-ass 23).
>> >
>> > The Promised Land that we say we're all trying to get to is regular,
>> > timely, feature-complete, tested, innovative but stable releases of
>> > new versions of Apache Hadoop. =A0Missing out any one of those criteri=
a
>> > discovered will continue (and has continued) the current situation
>> > where quasi-official branches and outside distributions fill the void
>> > such a release should. =A0The effort to maintain this offical branch a=
nd
>> > fix the bugs that will be discovered could be better spent moving us
>> > closer to that goal.
>> >
>> > I'm certainly sympathetic to the difficult position our quagmire has
>> > placed HBase into. =A0However, the current proposal would hurt HDFS to
>> > help HBase. The best solution for that project, as well as for HDFS,
>> > is to get HDFS back to a healthy release cycle; not prolong or codify
>> > the current ad-hoc state of affairs. =A0Let's stop digging this hole.
>> > -jakob
>> >
>> > On Thu, Dec 23, 2010 at 9:33 AM, M. C. Srivas <mcsrivas@gmail.com>
>> wrote:
>> >> [ Sorry if this is be-laboring the obvious ]
>> >>
>> >> There are two append solutions floating around, and they are
>> incompatible
>> >> with each other. Thus, the two "branches" will forever remain
>> incompatible
>> >> with each other, regardless of how they are numbered (0.22, =A00.23,
>> =A00.20.3,
>> >> e.t.c.)
>> >>
>> >> Unless both are merged into one branch, and a switch provided to =A0"=
use
>> >> HDFS-200 append" or "use 0.22 append", we have effectively split Hado=
op
>> into
>> >> two.
>> >>
>> >>
>> >> On Thu, Dec 23, 2010 at 12:00 AM, Owen O'Malley <omalley@apache.org>
>> wrote:
>> >>
>> >>> On Wed, Dec 22, 2010 at 11:07 PM, Roy T. Fielding <fielding@gbiv.com=
>
>> >>> wrote:
>> >>>
>> >>> > Features are not release version tags. =A0If there is a security b=
ug
>> >>> > found then we would have to release a new version of the append
>> >>> > version, and a round of severe trout slapping would result.
>> >>> >
>> >>>
>> >>> Yeah, it isn't a perfect solution and it doesn't scale to a second t=
ag,
>> but
>> >>> the problem is that this is effectively a release branch between 0.2=
0
>> and
>> >>> 0.21. Of course I agree that any critical bugs would need to be fixe=
d
>> in
>> >>> the
>> >>> append branch as well as the 0.20 and 0.21 branches.
>> >>>
>> >>> If you want to stick to pure numbers and we want to leave ourselves =
a
>> way
>> >>> to
>> >>> bugfix the 0.20 branch without append, we'd could use a version stri=
ng
>> like
>> >>> 0.20.100, etc. Not pretty, but it does preserve the numeric ordering
>> and
>> >>> suggest a version jump.
>> >>>
>> >>> If I remember right, there were also protocol changes in the append
>> branch,
>> >>> which was another reason we didn't want to put it directly into the
>> 0.20
>> >>> branch.
>> >>>
>> >>> -- Owen
>> >>>
>> >>
>> >
>>
>

From general-return-2605-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 22:46:59 2010
Return-Path: <general-return-2605-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39618 invoked from network); 23 Dec 2010 22:46:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 22:46:59 -0000
Received: (qmail 25899 invoked by uid 500); 23 Dec 2010 22:46:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25802 invoked by uid 500); 23 Dec 2010 22:46:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25794 invoked by uid 99); 23 Dec 2010 22:46:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 22:46:57 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ryanobjc@gmail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 22:46:52 +0000
Received: by iwn2 with SMTP id 2so7529692iwn.35
        for <general@hadoop.apache.org>; Thu, 23 Dec 2010 14:46:32 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=nfgsvZVEwCnxYs/nOS0+wUg/HY4sbxCb8M7apb/99Ac=;
        b=Ci/sMICy7iHrNHUxrm9nIBc++5X/BQnb66KnLHM5l0TjSIJ4rEtsQYKS+CY5B47QU0
         a09PAYHv9QKBT6XtH313KOdiw/MIH/LrCo+9rOrb0lNpgrPwSLnuov6UCaXgBaz6TTw2
         8Hgy59WnFY7TwTjf+4VTkm3pwv4nCQpeq97Uk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=qEJQJqIJc6RquELUa1/SiXLZIZ4Sl4J2m8U6h/HulRrMGZ57y2rFZEe+FnOyJIb1VL
         sc9IxE37HsZkBdSM1scEZLNrW0zSv9IAQj2I7kBDYhG+GZByhZQy+dwdms1Kl5+LpEo1
         H+7WNEWFVYKItU3LkE0Vh0BnQ7WKT2B9ViZFc=
MIME-Version: 1.0
Received: by 10.231.166.205 with SMTP id n13mr484494iby.58.1293144392115; Thu,
 23 Dec 2010 14:46:32 -0800 (PST)
Received: by 10.231.38.3 with HTTP; Thu, 23 Dec 2010 14:46:32 -0800 (PST)
In-Reply-To: <AANLkTi=acodAzbcafYW0n5v6eQ-m8S-rPaN-efdEcxV0@mail.gmail.com>
References: <20101222160311.7ffb9b6d@laptop>
	<AANLkTike7k5nqViAsdyC_mR6LOBVrrFFqsyGJo=-W6GQ@mail.gmail.com>
	<AANLkTi=9_-CPn+GON+YqLra1sjDX3-i-tqZ4rv39rTfT@mail.gmail.com>
	<20101223115004.7f8dc48c@pcmoholynagy.informatik.uni-stuttgart.de>
	<AANLkTimvsa__1gC5iL4PNGNt_sgTgTGB7UR4BxSbVyNM@mail.gmail.com>
	<AANLkTi=acodAzbcafYW0n5v6eQ-m8S-rPaN-efdEcxV0@mail.gmail.com>
Date: Thu, 23 Dec 2010 14:46:32 -0800
Message-ID: <AANLkTi=cRYTLO0WuaRzv2QvgovkjsjGpkfvrtF5-HxsF@mail.gmail.com>
Subject: Re: namenode doesn't start after reboot
From: Ryan Rawson <ryanobjc@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I think the bug might be related to this:

https://issues.apache.org/jira/browse/HDFS-686

and

https://issues.apache.org/jira/browse/HDFS-1002



On Thu, Dec 23, 2010 at 12:47 PM, Jakob Homan <jghoman@gmail.com> wrote:
> Please move discussions of CDH issues to Cloudera's lists. =A0Thanks.
>
> On Thu, Dec 23, 2010 at 12:02 PM, Todd Lipcon <todd@cloudera.com> wrote:
>> On Thu, Dec 23, 2010 at 2:50 AM, Bjoern Schiessle <bjoern@schiessle.org>=
wrote:
>>
>>>
>>> 1. I have set up a second dfs.name.dir which is stored at another
>>> computer (mounted by sshfs)
>>>
>>
>> I would strongly discourage the use of sshfs for the name dir. For one, =
it's
>> slow, and for two, I've sen it have some really weird semantics where it=
's
>> doing write-back caching.
>>
>> Just take a look at its manpage and you should get scared about using it=
 for
>> a critical mount point like this.
>>
>> A soft interruptable NFS mount is a much safer bet.
>>
>> -Todd
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>

From general-return-2606-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 23:34:19 2010
Return-Path: <general-return-2606-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56052 invoked from network); 23 Dec 2010 23:34:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 23:34:18 -0000
Received: (qmail 54043 invoked by uid 500); 23 Dec 2010 23:34:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53999 invoked by uid 500); 23 Dec 2010 23:34:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53991 invoked by uid 99); 23 Dec 2010 23:34:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 23:34:17 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=MIME_QP_LONG_LINE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.176] (HELO mail-px0-f176.google.com) (209.85.212.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 23:34:11 +0000
Received: by pxi11 with SMTP id 11so1414576pxi.35
        for <general@hadoop.apache.org>; Thu, 23 Dec 2010 15:33:50 -0800 (PST)
Received: by 10.142.50.3 with SMTP id x3mr6944022wfx.120.1293147230466;
        Thu, 23 Dec 2010 15:33:50 -0800 (PST)
Received: from [49.194.41.124] ([49.194.41.124])
        by mx.google.com with ESMTPS id p8sm11387953wff.16.2010.12.23.15.33.45
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Thu, 23 Dec 2010 15:33:49 -0800 (PST)
References: <AANLkTinMrd-acs=91xxw4MxHis8-RPSPtfhy7b4gu41O@mail.gmail.com> <7277DB0B-9B56-4B57-A574-F2764C11FC10@Holsman.NET> <AANLkTinMW7nvmapbkJwOGGNvaLNSrXSBr9c08JnBSHp4@mail.gmail.com> <C475DC5D-7CAC-47E4-838B-AED9C8274E5B@Holsman.NET> <AANLkTinnP2JB_Dqpxu3+3izOfG8AEB1t7zw00Ow9xdC8@mail.gmail.com> <EF3B67E6-26D2-42B3-AC99-01426AADC931@holsman.net> <21FEBF88-2D56-48E0-B86C-334E94F04188@gbiv.com> <AANLkTi=kVi=k6GV4tC8LNrpX4HaMxqRK72JJwOjN78tL@mail.gmail.com> <AANLkTikHfMxBk7SH+4oq94r894W4Qr2MOJWJH15nWeHi@mail.gmail.com> <AANLkTikiUzhMkh9y4Eaud4swibOmVcYL=0PFE28rms8n@mail.gmail.com> <AANLkTimBj9WMdEf3yQBbGTYw0JsiQQ=rH_mFTSYMZvBq@mail.gmail.com> <AANLkTi=zTJ8oTT-aXCBYvCyjARYKmOHHt52YFmgggy8X@mail.gmail.com> <AANLkTinHt928uox=NQju5Vey91YfTN28NzTA6wZEK8X8@mail.gmail.com>
In-Reply-To: <AANLkTinHt928uox=NQju5Vey91YfTN28NzTA6wZEK8X8@mail.gmail.com>
Mime-Version: 1.0 (iPhone Mail 8C148)
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii
Message-Id: <ED3EE45A-6A69-43C1-BFA5-C1CD6C337204@holsman.net>
Cc: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: iPhone Mail (8C148)
From: Ian Holsman <hadoop@holsman.net>
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of branch-0.20-append branch?
Date: Fri, 24 Dec 2010 10:33:33 +1100
To: "general@hadoop.apache.org" <general@hadoop.apache.org>

The release is one issue, but ongoing maintenance of it is another, which is=
 the point roy raised.=20

It's a concern if we have a security issue, and who will patch it (and test i=
t) going forward.=20

---
Ian Holsman - 703 879-3128

I saw the angel in the marble and carved until I set him free -- Michelangel=
o

On 24/12/2010, at 9:39 AM, Ryan Rawson <ryanobjc@gmail.com> wrote:

> How does stack volunteering his time to release an existing branch
> divert resources?
>=20
> Without an ASF release of 0.20-append I will keep having to recommend
> an external vendor's release of Hadoop.
>=20
>=20
> On Thu, Dec 23, 2010 at 2:18 PM, Konstantin Shvachko
> <shv.hadoop@gmail.com> wrote:
>> I also think building 0.20-append will be a major distraction from moving=

>> 0.22 forward with all the great new features, including the new append
>> implementation, sitting on the bench because we are delaying the release.=

>> It seems to be beneficial for the entire community to focus on 0.22 rathe=
r
>> than chasing both birds.
>>=20
>> I hear a concern that 0.22 will lack large scale testing as was the case
>> with 0.21.
>> I'd like to volunteer to put as many large scale resources, as I can gras=
p,
>> into stabilizing of 0.22. Under Nigel's management of course.
>> This should get us to production quality in 3-6 months rather than
>> "another 12-15". I also hope it can go even faster/better if others
>> could join the effort. I see > 100 companies claiming they are powered by=

>> Apache Hadoop.
>>=20
>> I also hope with this effort HBase will be able to start moving to the ne=
w
>> append implementation in the next 2-3 months, which in turn will help 0.2=
2
>> HDFS
>> rather than divert resources from it as it would have be with 0.20-append=
.
>>=20
>> Stack, will this plan will work for HBase survival?
>>=20
>> One other thought. Apache Hadoop community is not in control of external
>> releases and distributions, but we should not fork our own releases by
>> introducing
>> competing apis. If we can keep the dev line relatively straight the exter=
nal
>> releases
>> will follow.
>>=20
>> Thanks,
>> --Konstantin
>>=20
>>=20
>> On Thu, Dec 23, 2010 at 11:40 AM, Ryan Rawson <ryanobjc@gmail.com> wrote:=

>>=20
>>> The append solution in 0.22 that you are referring to was supposed to
>>> be out 13-15 months ago.  Pardon if I look for solutions that deploy 4
>>> months ago (as the 0.20 append branch did).
>>>=20
>>> Another 12-15 months of delay is not exactly helping HDFS either.
>>>=20
>>> -ryan
>>>=20
>>> On Thu, Dec 23, 2010 at 9:38 AM, Jakob Homan <jghoman@gmail.com> wrote:
>>>> It's difficult to support this proposal knowing how much time would be
>>>> spent preparing an official release, continuing to support it and
>>>> continuing to two support two separate implementations of append.  I
>>>> believe that effort would be better spent getting out a kick-ass 22
>>>> (or, barring that, a *really* kick-ass 23).
>>>>=20
>>>> The Promised Land that we say we're all trying to get to is regular,
>>>> timely, feature-complete, tested, innovative but stable releases of
>>>> new versions of Apache Hadoop.  Missing out any one of those criteria
>>>> discovered will continue (and has continued) the current situation
>>>> where quasi-official branches and outside distributions fill the void
>>>> such a release should.  The effort to maintain this offical branch and
>>>> fix the bugs that will be discovered could be better spent moving us
>>>> closer to that goal.
>>>>=20
>>>> I'm certainly sympathetic to the difficult position our quagmire has
>>>> placed HBase into.  However, the current proposal would hurt HDFS to
>>>> help HBase. The best solution for that project, as well as for HDFS,
>>>> is to get HDFS back to a healthy release cycle; not prolong or codify
>>>> the current ad-hoc state of affairs.  Let's stop digging this hole.
>>>> -jakob
>>>>=20
>>>> On Thu, Dec 23, 2010 at 9:33 AM, M. C. Srivas <mcsrivas@gmail.com>
>>> wrote:
>>>>> [ Sorry if this is be-laboring the obvious ]
>>>>>=20
>>>>> There are two append solutions floating around, and they are
>>> incompatible
>>>>> with each other. Thus, the two "branches" will forever remain
>>> incompatible
>>>>> with each other, regardless of how they are numbered (0.22,  0.23,
>>>  0.20.3,
>>>>> e.t.c.)
>>>>>=20
>>>>> Unless both are merged into one branch, and a switch provided to  "use=

>>>>> HDFS-200 append" or "use 0.22 append", we have effectively split Hadoo=
p
>>> into
>>>>> two.
>>>>>=20
>>>>>=20
>>>>> On Thu, Dec 23, 2010 at 12:00 AM, Owen O'Malley <omalley@apache.org>
>>> wrote:
>>>>>=20
>>>>>> On Wed, Dec 22, 2010 at 11:07 PM, Roy T. Fielding <fielding@gbiv.com>=

>>>>>> wrote:
>>>>>>=20
>>>>>>> Features are not release version tags.  If there is a security bug
>>>>>>> found then we would have to release a new version of the append
>>>>>>> version, and a round of severe trout slapping would result.
>>>>>>>=20
>>>>>>=20
>>>>>> Yeah, it isn't a perfect solution and it doesn't scale to a second ta=
g,
>>> but
>>>>>> the problem is that this is effectively a release branch between 0.20=

>>> and
>>>>>> 0.21. Of course I agree that any critical bugs would need to be fixed=

>>> in
>>>>>> the
>>>>>> append branch as well as the 0.20 and 0.21 branches.
>>>>>>=20
>>>>>> If you want to stick to pure numbers and we want to leave ourselves a=

>>> way
>>>>>> to
>>>>>> bugfix the 0.20 branch without append, we'd could use a version strin=
g
>>> like
>>>>>> 0.20.100, etc. Not pretty, but it does preserve the numeric ordering
>>> and
>>>>>> suggest a version jump.
>>>>>>=20
>>>>>> If I remember right, there were also protocol changes in the append
>>> branch,
>>>>>> which was another reason we didn't want to put it directly into the
>>> 0.20
>>>>>> branch.
>>>>>>=20
>>>>>> -- Owen
>>>>>>=20
>>>>>=20
>>>>=20
>>>=20
>>=20

From general-return-2607-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Dec 23 23:53:26 2010
Return-Path: <general-return-2607-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59270 invoked from network); 23 Dec 2010 23:53:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Dec 2010 23:53:26 -0000
Received: (qmail 67741 invoked by uid 500); 23 Dec 2010 23:53:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67673 invoked by uid 500); 23 Dec 2010 23:53:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67665 invoked by uid 99); 23 Dec 2010 23:53:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 23:53:24 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Dec 2010 23:53:18 +0000
Received: by iwn2 with SMTP id 2so7569914iwn.35
        for <general@hadoop.apache.org>; Thu, 23 Dec 2010 15:52:57 -0800 (PST)
Received: by 10.231.17.205 with SMTP id t13mr8554348iba.80.1293148376916; Thu,
 23 Dec 2010 15:52:56 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.188.102 with HTTP; Thu, 23 Dec 2010 15:52:36 -0800 (PST)
In-Reply-To: <ED3EE45A-6A69-43C1-BFA5-C1CD6C337204@holsman.net>
References: <AANLkTinMrd-acs=91xxw4MxHis8-RPSPtfhy7b4gu41O@mail.gmail.com>
 <7277DB0B-9B56-4B57-A574-F2764C11FC10@Holsman.NET> <AANLkTinMW7nvmapbkJwOGGNvaLNSrXSBr9c08JnBSHp4@mail.gmail.com>
 <C475DC5D-7CAC-47E4-838B-AED9C8274E5B@Holsman.NET> <AANLkTinnP2JB_Dqpxu3+3izOfG8AEB1t7zw00Ow9xdC8@mail.gmail.com>
 <EF3B67E6-26D2-42B3-AC99-01426AADC931@holsman.net> <21FEBF88-2D56-48E0-B86C-334E94F04188@gbiv.com>
 <AANLkTi=kVi=k6GV4tC8LNrpX4HaMxqRK72JJwOjN78tL@mail.gmail.com>
 <AANLkTikHfMxBk7SH+4oq94r894W4Qr2MOJWJH15nWeHi@mail.gmail.com>
 <AANLkTikiUzhMkh9y4Eaud4swibOmVcYL=0PFE28rms8n@mail.gmail.com>
 <AANLkTimBj9WMdEf3yQBbGTYw0JsiQQ=rH_mFTSYMZvBq@mail.gmail.com>
 <AANLkTi=zTJ8oTT-aXCBYvCyjARYKmOHHt52YFmgggy8X@mail.gmail.com>
 <AANLkTinHt928uox=NQju5Vey91YfTN28NzTA6wZEK8X8@mail.gmail.com> <ED3EE45A-6A69-43C1-BFA5-C1CD6C337204@holsman.net>
From: Todd Lipcon <todd@cloudera.com>
Date: Thu, 23 Dec 2010 15:52:36 -0800
Message-ID: <AANLkTi=3wmCcA6TvdS6r_NOgPETwn_Hgd7gxw=_ho353@mail.gmail.com>
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of
 branch-0.20-append branch?
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000325575a1aa5e53b04981c91e8
X-Virus-Checked: Checked by ClamAV on apache.org

--000325575a1aa5e53b04981c91e8
Content-Type: text/plain; charset=ISO-8859-1

On Thu, Dec 23, 2010 at 3:33 PM, Ian Holsman <hadoop@holsman.net> wrote:

> The release is one issue, but ongoing maintenance of it is another, which
> is the point roy raised.
>
> It's a concern if we have a security issue, and who will patch it (and test
> it) going forward.
>

The nice thing is that Hadoop 0.20.x (without security patches) has no
guarantees as to security. So, I can't imagine any security issue that could
possibly exist that would be worth addressing. It doesn't run as root so
root escalation is only possible with a JVM bug, and it's trivially possible
to read anyone's data since 0.20 has no strong authentication.

Thanks
-Todd



>
> ---
> Ian Holsman - 703 879-3128
>
> I saw the angel in the marble and carved until I set him free --
> Michelangelo
>
> On 24/12/2010, at 9:39 AM, Ryan Rawson <ryanobjc@gmail.com> wrote:
>
> > How does stack volunteering his time to release an existing branch
> > divert resources?
> >
> > Without an ASF release of 0.20-append I will keep having to recommend
> > an external vendor's release of Hadoop.
> >
> >
> > On Thu, Dec 23, 2010 at 2:18 PM, Konstantin Shvachko
> > <shv.hadoop@gmail.com> wrote:
> >> I also think building 0.20-append will be a major distraction from
> moving
> >> 0.22 forward with all the great new features, including the new append
> >> implementation, sitting on the bench because we are delaying the
> release.
> >> It seems to be beneficial for the entire community to focus on 0.22
> rather
> >> than chasing both birds.
> >>
> >> I hear a concern that 0.22 will lack large scale testing as was the case
> >> with 0.21.
> >> I'd like to volunteer to put as many large scale resources, as I can
> grasp,
> >> into stabilizing of 0.22. Under Nigel's management of course.
> >> This should get us to production quality in 3-6 months rather than
> >> "another 12-15". I also hope it can go even faster/better if others
> >> could join the effort. I see > 100 companies claiming they are powered
> by
> >> Apache Hadoop.
> >>
> >> I also hope with this effort HBase will be able to start moving to the
> new
> >> append implementation in the next 2-3 months, which in turn will help
> 0.22
> >> HDFS
> >> rather than divert resources from it as it would have be with
> 0.20-append.
> >>
> >> Stack, will this plan will work for HBase survival?
> >>
> >> One other thought. Apache Hadoop community is not in control of external
> >> releases and distributions, but we should not fork our own releases by
> >> introducing
> >> competing apis. If we can keep the dev line relatively straight the
> external
> >> releases
> >> will follow.
> >>
> >> Thanks,
> >> --Konstantin
> >>
> >>
> >> On Thu, Dec 23, 2010 at 11:40 AM, Ryan Rawson <ryanobjc@gmail.com>
> wrote:
> >>
> >>> The append solution in 0.22 that you are referring to was supposed to
> >>> be out 13-15 months ago.  Pardon if I look for solutions that deploy 4
> >>> months ago (as the 0.20 append branch did).
> >>>
> >>> Another 12-15 months of delay is not exactly helping HDFS either.
> >>>
> >>> -ryan
> >>>
> >>> On Thu, Dec 23, 2010 at 9:38 AM, Jakob Homan <jghoman@gmail.com>
> wrote:
> >>>> It's difficult to support this proposal knowing how much time would be
> >>>> spent preparing an official release, continuing to support it and
> >>>> continuing to two support two separate implementations of append.  I
> >>>> believe that effort would be better spent getting out a kick-ass 22
> >>>> (or, barring that, a *really* kick-ass 23).
> >>>>
> >>>> The Promised Land that we say we're all trying to get to is regular,
> >>>> timely, feature-complete, tested, innovative but stable releases of
> >>>> new versions of Apache Hadoop.  Missing out any one of those criteria
> >>>> discovered will continue (and has continued) the current situation
> >>>> where quasi-official branches and outside distributions fill the void
> >>>> such a release should.  The effort to maintain this offical branch and
> >>>> fix the bugs that will be discovered could be better spent moving us
> >>>> closer to that goal.
> >>>>
> >>>> I'm certainly sympathetic to the difficult position our quagmire has
> >>>> placed HBase into.  However, the current proposal would hurt HDFS to
> >>>> help HBase. The best solution for that project, as well as for HDFS,
> >>>> is to get HDFS back to a healthy release cycle; not prolong or codify
> >>>> the current ad-hoc state of affairs.  Let's stop digging this hole.
> >>>> -jakob
> >>>>
> >>>> On Thu, Dec 23, 2010 at 9:33 AM, M. C. Srivas <mcsrivas@gmail.com>
> >>> wrote:
> >>>>> [ Sorry if this is be-laboring the obvious ]
> >>>>>
> >>>>> There are two append solutions floating around, and they are
> >>> incompatible
> >>>>> with each other. Thus, the two "branches" will forever remain
> >>> incompatible
> >>>>> with each other, regardless of how they are numbered (0.22,  0.23,
> >>>  0.20.3,
> >>>>> e.t.c.)
> >>>>>
> >>>>> Unless both are merged into one branch, and a switch provided to
>  "use
> >>>>> HDFS-200 append" or "use 0.22 append", we have effectively split
> Hadoop
> >>> into
> >>>>> two.
> >>>>>
> >>>>>
> >>>>> On Thu, Dec 23, 2010 at 12:00 AM, Owen O'Malley <omalley@apache.org>
> >>> wrote:
> >>>>>
> >>>>>> On Wed, Dec 22, 2010 at 11:07 PM, Roy T. Fielding <
> fielding@gbiv.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Features are not release version tags.  If there is a security bug
> >>>>>>> found then we would have to release a new version of the append
> >>>>>>> version, and a round of severe trout slapping would result.
> >>>>>>>
> >>>>>>
> >>>>>> Yeah, it isn't a perfect solution and it doesn't scale to a second
> tag,
> >>> but
> >>>>>> the problem is that this is effectively a release branch between
> 0.20
> >>> and
> >>>>>> 0.21. Of course I agree that any critical bugs would need to be
> fixed
> >>> in
> >>>>>> the
> >>>>>> append branch as well as the 0.20 and 0.21 branches.
> >>>>>>
> >>>>>> If you want to stick to pure numbers and we want to leave ourselves
> a
> >>> way
> >>>>>> to
> >>>>>> bugfix the 0.20 branch without append, we'd could use a version
> string
> >>> like
> >>>>>> 0.20.100, etc. Not pretty, but it does preserve the numeric ordering
> >>> and
> >>>>>> suggest a version jump.
> >>>>>>
> >>>>>> If I remember right, there were also protocol changes in the append
> >>> branch,
> >>>>>> which was another reason we didn't want to put it directly into the
> >>> 0.20
> >>>>>> branch.
> >>>>>>
> >>>>>> -- Owen
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--000325575a1aa5e53b04981c91e8--

From general-return-2608-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Dec 24 00:37:27 2010
Return-Path: <general-return-2608-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82650 invoked from network); 24 Dec 2010 00:37:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Dec 2010 00:37:27 -0000
Received: (qmail 11760 invoked by uid 500); 24 Dec 2010 00:37:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11678 invoked by uid 500); 24 Dec 2010 00:37:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11663 invoked by uid 99); 24 Dec 2010 00:37:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Dec 2010 00:37:25 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [98.139.52.228] (HELO nm7-vm0.bullet.mail.ac4.yahoo.com) (98.139.52.228)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 24 Dec 2010 00:37:16 +0000
Received: from [98.139.52.197] by nm7.bullet.mail.ac4.yahoo.com with NNFMP; 24 Dec 2010 00:36:55 -0000
Received: from [98.139.52.170] by tm10.bullet.mail.ac4.yahoo.com with NNFMP; 24 Dec 2010 00:36:55 -0000
Received: from [127.0.0.1] by omp1053.mail.ac4.yahoo.com with NNFMP; 24 Dec 2010 00:36:55 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 319196.28018.bm@omp1053.mail.ac4.yahoo.com
Received: (qmail 91934 invoked by uid 60001); 24 Dec 2010 00:36:55 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1293151015; bh=ZZY5UBDt/mqudcMKpFvVOt0T2bsKPJgAIrY6HQfdA4A=; h=Message-ID:X-YMail-OSG:Received:X-RocketYMMF:X-Mailer:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=49J+Wgd5ZqX3TlpOCOkxph/2JSPvef30oFxWuFitM4q0ihv8PK9n7X9Rb0hH/dLl7ciZtI2bFSuYJEl2XQ3B74gVqZA8ljHt6Gi8Qye1u6WHdSwiBjl1T4wg0Bv5MbHRbE4NqAyit1R6QnUI0EjjFKSQdjsAHd1hx1TDg3VuDrI=
Message-ID: <61840.91458.qm@web65503.mail.ac4.yahoo.com>
X-YMail-OSG: b.IrugcVM1nGo09EAG.tlZrjOk7vguTdA2NWKosoA_UtoFD
 SLSid7kdMNJIELGDccNWBpaKFiNfb4NZ.Wh1eU6vkbOkTyDmF68GP8hg_gfG
 fjm51.hDkZMqEZyPAgSgbcRxUPgd._43pSlk5LkSrNaYh4sKeVPc0WZn8moF
 WWkXqhngfq.kZAbudb0W7qEhnupQIvh6SGXu3lHRT9XG2PvFZOSJRxILv8h4
 QyoMdJFi53CQz9382NBp5w2W8KS6yCthCuRFvuXIC1DmGqQbZDgJv9bD8mT8
 7IMhRXi7pN7PHE6QE91vSptvJnHwjwq1R2gblZ7umR50heH0pq.QlZDpiTxm
 bE6EJng.8Us2N72QPLmimnCCeRe0zRK_3XwZtc4R_HBw.HJH68NPcOJSkr63
 iyJAvAz3luIs-
Received: from [166.190.158.37] by web65503.mail.ac4.yahoo.com via HTTP; Thu, 23 Dec 2010 16:36:54 PST
X-RocketYMMF: apurtell
X-Mailer: YahooMailClassic/11.4.20 YahooMailWebService/0.8.107.285259
Date: Thu, 23 Dec 2010 16:36:54 -0800 (PST)
From: Andrew Purtell <apurtell@apache.org>
Reply-To: apurtell@apache.org
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of branch-0.20-append branch?
To: general@hadoop.apache.org
In-Reply-To: <AANLkTinHt928uox=NQju5Vey91YfTN28NzTA6wZEK8X8@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

I hope that 22 will be an answer. I think I would be more comfortable with =
that answer if Hadoop Core were not so obviously internally conflicted and =
sclerotic. Potential HBase/Hadoop adopters have confidence in 20 seeing the=
 production deployments of it. 21 was to all indications I have seen a dud.=
 There is no reasonable basis as of yet to presume 22 will be "kick ass". =
=0A=0AI, at least, was hoping that promoting 0.20-append from its de-facto =
status to something official could be a fig leaf for HBase while Hadoop Cor=
e gets its house in order.=0A=0ABest regards,=0A=0A    - Andy=0A=0AProblems=
 worthy of attack prove their worth by hitting back.=0A  - Piet Hein (via T=
om White)=0A=0A=0A--- On Thu, 12/23/10, Ryan Rawson <ryanobjc@gmail.com> wr=
ote:=0A=0A> From: Ryan Rawson <ryanobjc@gmail.com>=0A> Subject: Re: DISCUSS=
ION: Cut a hadoop-0.20.0-append release from the tip of branch-0.20-append =
branch?=0A> To: general@hadoop.apache.org=0A> Date: Thursday, December 23, =
2010, 2:39 PM=0A> How does stack volunteering his time=0A> to release an ex=
isting branch=0A> divert resources?=0A> =0A> Without an ASF release of 0.20=
-append I will keep having to=0A> recommend an external vendor's release of=
 Hadoop.=0A> =0A> =0A> On Thu, Dec 23, 2010 at 2:18 PM, Konstantin Shvachko=
=0A> <shv.hadoop@gmail.com>=0A> wrote:=0A> > I also think building 0.20-app=
end will be a major=0A> distraction from moving=0A> > 0.22 forward with all=
 the great new features,=0A> including the new append=0A> > implementation,=
 sitting on the bench because we are=0A> delaying the release.=0A> > It see=
ms to be beneficial for the entire community to=0A> focus on 0.22 rather=0A=
> > than chasing both birds.=0A> >=0A> > I hear a concern that 0.22 will la=
ck large scale=0A> testing as was the case=0A> > with 0.21.=0A> > I'd like =
to volunteer to put as many large scale=0A> resources, as I can grasp,=0A> =
> into stabilizing of 0.22. Under Nigel's management of=0A> course.=0A> > T=
his should get us to production quality in 3-6 months=0A> rather than=0A> >=
 "another 12-15". I also hope it can go even=0A> faster/better if others=0A=
> > could join the effort. I see > 100 companies=0A> claiming they are powe=
red by=0A> > Apache Hadoop.=0A> >=0A> > I also hope with this effort HBase =
will be able to=0A> start moving to the new=0A> > append implementation in =
the next 2-3 months, which in=0A> turn will help 0.22=0A> > HDFS=0A> > rath=
er than divert resources from it as it would have=0A> be with 0.20-append.=
=0A> >=0A> > Stack, will this plan will work for HBase survival?=0A> >=0A> =
> One other thought. Apache Hadoop community is not in=0A> control of exter=
nal=0A> > releases and distributions, but we should not fork our=0A> own re=
leases by=0A> > introducing=0A> > competing apis. If we can keep the dev li=
ne relatively=0A> straight the external=0A> > releases=0A> > will follow.=
=0A> >=0A> > Thanks,=0A> > --Konstantin=0A> >=0A> >=0A> > On Thu, Dec 23, 2=
010 at 11:40 AM, Ryan Rawson <ryanobjc@gmail.com>=0A> wrote:=0A> >=0A> >> T=
he append solution in 0.22 that you are referring=0A> to was supposed to=0A=
> >> be out 13-15 months ago. =A0Pardon if I look for=0A> solutions that de=
ploy 4=0A> >> months ago (as the 0.20 append branch did).=0A> >>=0A> >> Ano=
ther 12-15 months of delay is not exactly=0A> helping HDFS either.=0A> >>=
=0A> >> -ryan=0A> >>=0A> >> On Thu, Dec 23, 2010 at 9:38 AM, Jakob Homan=0A=
> <jghoman@gmail.com>=0A> wrote:=0A> >> > It's difficult to support this pr=
oposal=0A> knowing how much time would be=0A> >> > spent preparing an offic=
ial release,=0A> continuing to support it and=0A> >> > continuing to two su=
pport two separate=0A> implementations of append. =A0I=0A> >> > believe tha=
t effort would be better spent=0A> getting out a kick-ass 22=0A> >> > (or, =
barring that, a *really* kick-ass 23).=0A> >> >=0A> >> > The Promised Land =
that we say we're all=0A> trying to get to is regular,=0A> >> > timely, fea=
ture-complete, tested, innovative=0A> but stable releases of=0A> >> > new v=
ersions of Apache Hadoop. =A0Missing out=0A> any one of those criteria=0A> =
>> > discovered will continue (and has continued)=0A> the current situation=
=0A> >> > where quasi-official branches and outside=0A> distributions fill =
the void=0A> >> > such a release should. =A0The effort to=0A> maintain this=
 offical branch and=0A> >> > fix the bugs that will be discovered could be=
=0A> better spent moving us=0A> >> > closer to that goal.=0A> >> >=0A> >> >=
 I'm certainly sympathetic to the difficult=0A> position our quagmire has=
=0A> >> > placed HBase into. =A0However, the current=0A> proposal would hur=
t HDFS to=0A> >> > help HBase. The best solution for that=0A> project, as w=
ell as for HDFS,=0A> >> > is to get HDFS back to a healthy release=0A> cycl=
e; not prolong or codify=0A> >> > the current ad-hoc state of affairs. =A0L=
et's=0A> stop digging this hole.=0A> >> > -jakob=0A> >> >=0A> >> > On Thu, =
Dec 23, 2010 at 9:33 AM, M. C. Srivas=0A> <mcsrivas@gmail.com>=0A> >> wrote=
:=0A> >> >> [ Sorry if this is be-laboring the=0A> obvious ]=0A> >> >>=0A> =
>> >> There are two append solutions floating=0A> around, and they are=0A> =
>> incompatible=0A> >> >> with each other. Thus, the two "branches"=0A> wil=
l forever remain=0A> >> incompatible=0A> >> >> with each other, regardless =
of how they=0A> are numbered (0.22, =A00.23,=0A> >> =A00.20.3,=0A> >> >> e.=
t.c.)=0A> >> >>=0A> >> >> Unless both are merged into one branch,=0A> and a=
 switch provided to =A0"use=0A> >> >> HDFS-200 append" or "use 0.22 append"=
, we=0A> have effectively split Hadoop=0A> >> into=0A> >> >> two.=0A> >> >>=
=0A> >> >>=0A> >> >> On Thu, Dec 23, 2010 at 12:00 AM, Owen=0A> O'Malley <o=
malley@apache.org>=0A> >> wrote:=0A> >> >>=0A> >> >>> On Wed, Dec 22, 2010 =
at 11:07 PM, Roy=0A> T. Fielding <fielding@gbiv.com>=0A> >> >>> wrote:=0A> =
>> >>>=0A> >> >>> > Features are not release version=0A> tags. =A0If there =
is a security bug=0A> >> >>> > found then we would have to=0A> release a ne=
w version of the append=0A> >> >>> > version, and a round of severe=0A> tro=
ut slapping would result.=0A> >> >>> >=0A> >> >>>=0A> >> >>> Yeah, it isn't=
 a perfect solution and=0A> it doesn't scale to a second tag,=0A> >> but=0A=
> >> >>> the problem is that this is=0A> effectively a release branch betwe=
en 0.20=0A> >> and=0A> >> >>> 0.21. Of course I agree that any=0A> critical=
 bugs would need to be fixed=0A> >> in=0A> >> >>> the=0A> >> >>> append bra=
nch as well as the 0.20 and=0A> 0.21 branches.=0A> >> >>>=0A> >> >>> If you=
 want to stick to pure numbers=0A> and we want to leave ourselves a=0A> >> =
way=0A> >> >>> to=0A> >> >>> bugfix the 0.20 branch without=0A> append, we'=
d could use a version string=0A> >> like=0A> >> >>> 0.20.100, etc. Not pret=
ty, but it=0A> does preserve the numeric ordering=0A> >> and=0A> >> >>> sug=
gest a version jump.=0A> >> >>>=0A> >> >>> If I remember right, there were =
also=0A> protocol changes in the append=0A> >> branch,=0A> >> >>> which was=
 another reason we didn't=0A> want to put it directly into the=0A> >> 0.20=
=0A> >> >>> branch.=0A> >> >>>=0A> >> >>> -- Owen=0A> >> >>>=0A> >> >>=0A> =
>> >=0A> >>=0A> >=0A> =0A=0A=0A      

From general-return-2609-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Dec 24 01:32:32 2010
Return-Path: <general-return-2609-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4077 invoked from network); 24 Dec 2010 01:32:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Dec 2010 01:32:32 -0000
Received: (qmail 57584 invoked by uid 500); 24 Dec 2010 01:32:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57524 invoked by uid 500); 24 Dec 2010 01:32:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57511 invoked by uid 99); 24 Dec 2010 01:32:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Dec 2010 01:32:30 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Dec 2010 01:32:25 +0000
Received: by iwn2 with SMTP id 2so7627857iwn.35
        for <general@hadoop.apache.org>; Thu, 23 Dec 2010 17:32:04 -0800 (PST)
MIME-Version: 1.0
Received: by 10.42.225.202 with SMTP id it10mr8987993icb.496.1293154324822;
 Thu, 23 Dec 2010 17:32:04 -0800 (PST)
Received: by 10.42.178.193 with HTTP; Thu, 23 Dec 2010 17:32:04 -0800 (PST)
In-Reply-To: <61840.91458.qm@web65503.mail.ac4.yahoo.com>
References: <AANLkTinHt928uox=NQju5Vey91YfTN28NzTA6wZEK8X8@mail.gmail.com>
	<61840.91458.qm@web65503.mail.ac4.yahoo.com>
Date: Thu, 23 Dec 2010 17:32:04 -0800
Message-ID: <AANLkTi=zf5r_8b4kPhZPKhXbdnjdkkD5K57-Pso0AXL-@mail.gmail.com>
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of
 branch-0.20-append branch?
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org, apurtell@apache.org
Content-Type: multipart/alternative; boundary=20cf30549e612bba3604981df444

--20cf30549e612bba3604981df444
Content-Type: text/plain; charset=ISO-8859-1

After reading through the reasoning on both sides of this issue, I agree
with Ian, Konstantin, and Jakob. Nigel has already volunteered to run the
0.22 release process; let's put our energy there. Stack, the energy you
would have put into the 0.20-append release could help ensure the 0.22
release makes it out in short order. That way HBase will be able to take
advantage of both append (don't lose data) and security (don't give it
away), and we won't derail the Hadoop Core release process, which has
actually been regaining some momentum over the past several months: we got
0.21 out the door! we have a release manager for 0.22!

As Roy points out, the Apache Hadoop release train has already passed 0.20;
for those that require a 0.20-based HDFS with append, there are multiple
places in the open source world to retrieve such bits, including the
0.20-append branch of the HDFS project. If the HBase community requires an
ASF project to release such an artifact, as Roy points out, it can certainly
done as a new project separate from HDFS.

On Thu, Dec 23, 2010 at 4:36 PM, Andrew Purtell <apurtell@apache.org> wrote:

> I hope that 22 will be an answer. I think I would be more comfortable with
> that answer if Hadoop Core were not so obviously internally conflicted and
> sclerotic. Potential HBase/Hadoop adopters have confidence in 20 seeing the
> production deployments of it. 21 was to all indications I have seen a dud.
> There is no reasonable basis as of yet to presume 22 will be "kick ass".
>
> I, at least, was hoping that promoting 0.20-append from its de-facto status
> to something official could be a fig leaf for HBase while Hadoop Core gets
> its house in order.
>
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back.
>  - Piet Hein (via Tom White)
>
>
> --- On Thu, 12/23/10, Ryan Rawson <ryanobjc@gmail.com> wrote:
>
> > From: Ryan Rawson <ryanobjc@gmail.com>
> > Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip
> of branch-0.20-append branch?
> > To: general@hadoop.apache.org
> > Date: Thursday, December 23, 2010, 2:39 PM
> > How does stack volunteering his time
> > to release an existing branch
> > divert resources?
> >
> > Without an ASF release of 0.20-append I will keep having to
> > recommend an external vendor's release of Hadoop.
> >
> >
> > On Thu, Dec 23, 2010 at 2:18 PM, Konstantin Shvachko
> > <shv.hadoop@gmail.com>
> > wrote:
> > > I also think building 0.20-append will be a major
> > distraction from moving
> > > 0.22 forward with all the great new features,
> > including the new append
> > > implementation, sitting on the bench because we are
> > delaying the release.
> > > It seems to be beneficial for the entire community to
> > focus on 0.22 rather
> > > than chasing both birds.
> > >
> > > I hear a concern that 0.22 will lack large scale
> > testing as was the case
> > > with 0.21.
> > > I'd like to volunteer to put as many large scale
> > resources, as I can grasp,
> > > into stabilizing of 0.22. Under Nigel's management of
> > course.
> > > This should get us to production quality in 3-6 months
> > rather than
> > > "another 12-15". I also hope it can go even
> > faster/better if others
> > > could join the effort. I see > 100 companies
> > claiming they are powered by
> > > Apache Hadoop.
> > >
> > > I also hope with this effort HBase will be able to
> > start moving to the new
> > > append implementation in the next 2-3 months, which in
> > turn will help 0.22
> > > HDFS
> > > rather than divert resources from it as it would have
> > be with 0.20-append.
> > >
> > > Stack, will this plan will work for HBase survival?
> > >
> > > One other thought. Apache Hadoop community is not in
> > control of external
> > > releases and distributions, but we should not fork our
> > own releases by
> > > introducing
> > > competing apis. If we can keep the dev line relatively
> > straight the external
> > > releases
> > > will follow.
> > >
> > > Thanks,
> > > --Konstantin
> > >
> > >
> > > On Thu, Dec 23, 2010 at 11:40 AM, Ryan Rawson <ryanobjc@gmail.com>
> > wrote:
> > >
> > >> The append solution in 0.22 that you are referring
> > to was supposed to
> > >> be out 13-15 months ago.  Pardon if I look for
> > solutions that deploy 4
> > >> months ago (as the 0.20 append branch did).
> > >>
> > >> Another 12-15 months of delay is not exactly
> > helping HDFS either.
> > >>
> > >> -ryan
> > >>
> > >> On Thu, Dec 23, 2010 at 9:38 AM, Jakob Homan
> > <jghoman@gmail.com>
> > wrote:
> > >> > It's difficult to support this proposal
> > knowing how much time would be
> > >> > spent preparing an official release,
> > continuing to support it and
> > >> > continuing to two support two separate
> > implementations of append.  I
> > >> > believe that effort would be better spent
> > getting out a kick-ass 22
> > >> > (or, barring that, a *really* kick-ass 23).
> > >> >
> > >> > The Promised Land that we say we're all
> > trying to get to is regular,
> > >> > timely, feature-complete, tested, innovative
> > but stable releases of
> > >> > new versions of Apache Hadoop.  Missing out
> > any one of those criteria
> > >> > discovered will continue (and has continued)
> > the current situation
> > >> > where quasi-official branches and outside
> > distributions fill the void
> > >> > such a release should.  The effort to
> > maintain this offical branch and
> > >> > fix the bugs that will be discovered could be
> > better spent moving us
> > >> > closer to that goal.
> > >> >
> > >> > I'm certainly sympathetic to the difficult
> > position our quagmire has
> > >> > placed HBase into.  However, the current
> > proposal would hurt HDFS to
> > >> > help HBase. The best solution for that
> > project, as well as for HDFS,
> > >> > is to get HDFS back to a healthy release
> > cycle; not prolong or codify
> > >> > the current ad-hoc state of affairs.  Let's
> > stop digging this hole.
> > >> > -jakob
> > >> >
> > >> > On Thu, Dec 23, 2010 at 9:33 AM, M. C. Srivas
> > <mcsrivas@gmail.com>
> > >> wrote:
> > >> >> [ Sorry if this is be-laboring the
> > obvious ]
> > >> >>
> > >> >> There are two append solutions floating
> > around, and they are
> > >> incompatible
> > >> >> with each other. Thus, the two "branches"
> > will forever remain
> > >> incompatible
> > >> >> with each other, regardless of how they
> > are numbered (0.22,  0.23,
> > >>  0.20.3,
> > >> >> e.t.c.)
> > >> >>
> > >> >> Unless both are merged into one branch,
> > and a switch provided to  "use
> > >> >> HDFS-200 append" or "use 0.22 append", we
> > have effectively split Hadoop
> > >> into
> > >> >> two.
> > >> >>
> > >> >>
> > >> >> On Thu, Dec 23, 2010 at 12:00 AM, Owen
> > O'Malley <omalley@apache.org>
> > >> wrote:
> > >> >>
> > >> >>> On Wed, Dec 22, 2010 at 11:07 PM, Roy
> > T. Fielding <fielding@gbiv.com>
> > >> >>> wrote:
> > >> >>>
> > >> >>> > Features are not release version
> > tags.  If there is a security bug
> > >> >>> > found then we would have to
> > release a new version of the append
> > >> >>> > version, and a round of severe
> > trout slapping would result.
> > >> >>> >
> > >> >>>
> > >> >>> Yeah, it isn't a perfect solution and
> > it doesn't scale to a second tag,
> > >> but
> > >> >>> the problem is that this is
> > effectively a release branch between 0.20
> > >> and
> > >> >>> 0.21. Of course I agree that any
> > critical bugs would need to be fixed
> > >> in
> > >> >>> the
> > >> >>> append branch as well as the 0.20 and
> > 0.21 branches.
> > >> >>>
> > >> >>> If you want to stick to pure numbers
> > and we want to leave ourselves a
> > >> way
> > >> >>> to
> > >> >>> bugfix the 0.20 branch without
> > append, we'd could use a version string
> > >> like
> > >> >>> 0.20.100, etc. Not pretty, but it
> > does preserve the numeric ordering
> > >> and
> > >> >>> suggest a version jump.
> > >> >>>
> > >> >>> If I remember right, there were also
> > protocol changes in the append
> > >> branch,
> > >> >>> which was another reason we didn't
> > want to put it directly into the
> > >> 0.20
> > >> >>> branch.
> > >> >>>
> > >> >>> -- Owen
> > >> >>>
> > >> >>
> > >> >
> > >>
> > >
> >
>
>
>
>

--20cf30549e612bba3604981df444--

From general-return-2610-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Dec 24 05:18:24 2010
Return-Path: <general-return-2610-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82404 invoked from network); 24 Dec 2010 05:18:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Dec 2010 05:18:23 -0000
Received: (qmail 66093 invoked by uid 500); 24 Dec 2010 05:18:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65970 invoked by uid 500); 24 Dec 2010 05:18:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65962 invoked by uid 99); 24 Dec 2010 05:18:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Dec 2010 05:18:21 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Dec 2010 05:18:15 +0000
Received: by wye20 with SMTP id 20so7097561wye.35
        for <general@hadoop.apache.org>; Thu, 23 Dec 2010 21:17:53 -0800 (PST)
Received: by 10.216.63.15 with SMTP id z15mr9737392wec.74.1293167873526; Thu,
 23 Dec 2010 21:17:53 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.59.13 with HTTP; Thu, 23 Dec 2010 21:17:33 -0800 (PST)
In-Reply-To: <AANLkTi=zTJ8oTT-aXCBYvCyjARYKmOHHt52YFmgggy8X@mail.gmail.com>
References: <AANLkTinMrd-acs=91xxw4MxHis8-RPSPtfhy7b4gu41O@mail.gmail.com>
 <7277DB0B-9B56-4B57-A574-F2764C11FC10@Holsman.NET> <AANLkTinMW7nvmapbkJwOGGNvaLNSrXSBr9c08JnBSHp4@mail.gmail.com>
 <C475DC5D-7CAC-47E4-838B-AED9C8274E5B@Holsman.NET> <AANLkTinnP2JB_Dqpxu3+3izOfG8AEB1t7zw00Ow9xdC8@mail.gmail.com>
 <EF3B67E6-26D2-42B3-AC99-01426AADC931@holsman.net> <21FEBF88-2D56-48E0-B86C-334E94F04188@gbiv.com>
 <AANLkTi=kVi=k6GV4tC8LNrpX4HaMxqRK72JJwOjN78tL@mail.gmail.com>
 <AANLkTikHfMxBk7SH+4oq94r894W4Qr2MOJWJH15nWeHi@mail.gmail.com>
 <AANLkTikiUzhMkh9y4Eaud4swibOmVcYL=0PFE28rms8n@mail.gmail.com>
 <AANLkTimBj9WMdEf3yQBbGTYw0JsiQQ=rH_mFTSYMZvBq@mail.gmail.com> <AANLkTi=zTJ8oTT-aXCBYvCyjARYKmOHHt52YFmgggy8X@mail.gmail.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Thu, 23 Dec 2010 21:17:33 -0800
X-Google-Sender-Auth: hA5yv3PqOqEMdLSqBOmyh-xQeXo
Message-ID: <AANLkTimkEQfCiAOyvVkjJzouAw0wNg7x9d7nA7ofy8Kr@mail.gmail.com>
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of
 branch-0.20-append branch?
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Thu, Dec 23, 2010 at 14:18, Konstantin Shvachko <shv.hadoop@gmail.com> wrote:
> I also think building 0.20-append will be a major distraction from moving
> 0.22 forward with all the great new features, including the new append
> implementation, sitting on the bench because we are delaying the release.
> It seems to be beneficial for the entire community to focus on 0.22 rather
> than chasing both birds.
>
> I hear a concern that 0.22 will lack large scale testing as was the case
> with 0.21.
> I'd like to volunteer to put as many large scale resources, as I can grasp,
> into stabilizing of 0.22. Under Nigel's management of course.
> This should get us to production quality in 3-6 months rather than
> "another 12-15". I also hope it can go even faster/better if others
> could join the effort. I see > 100 companies claiming they are powered by
> Apache Hadoop.

On the similar note I's like to emphasize that a significant part of
my time is going to be devoted to building system & scale testing
infrastructure which would usable out of the box by any of those 100+
companies if they are willing to put any effort into testing of 0.22.

Cos

From general-return-2611-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Dec 24 07:53:20 2010
Return-Path: <general-return-2611-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50568 invoked from network); 24 Dec 2010 07:53:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Dec 2010 07:53:20 -0000
Received: (qmail 29599 invoked by uid 500); 24 Dec 2010 07:53:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29489 invoked by uid 500); 24 Dec 2010 07:53:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29481 invoked by uid 99); 24 Dec 2010 07:53:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Dec 2010 07:53:17 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Dec 2010 07:53:10 +0000
Received: by fxm2 with SMTP id 2so6793448fxm.35
        for <general@hadoop.apache.org>; Thu, 23 Dec 2010 23:52:50 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:content-type:content-transfer-encoding;
        bh=KF0Zhy31hX971ynuZfGyZkIOmkUiFiT0a7EBMbfvkZQ=;
        b=xzLHD3xu1Qd2qTvty/zXo9eBNp86aQd3CR2vh6wTqRb220Wfcky2zuvvQL3QB/MIIn
         ls/ygblbBG2ATqQ+vphzDzIJwaCBg6sxPOHpODhE9Wby3lbQ6jK+Vl7mSPygS+xORgqP
         e7iaYpwEkqcyolwSFUrqdUhQLKUpAPjjawcOM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=o7k+YuAwU4Cl4gSYKBF4/xVFcOZmG3ngUj1drEcvQXKO0ysUOG6AiA95MOdqI+bOtW
         6FFAO8QFx11Q5JUal64MfGD3/CldKIjMrTX+D4ma8qY+Ivf3Tq04FTQ4UnehrN2cME9I
         FcNbNNaurz3pzd5x8mH5cQLGVJCXf3sl17g7I=
MIME-Version: 1.0
Received: by 10.223.125.196 with SMTP id z4mr5250335far.124.1293177170034;
 Thu, 23 Dec 2010 23:52:50 -0800 (PST)
Sender: saint.ack@gmail.com
Received: by 10.223.83.9 with HTTP; Thu, 23 Dec 2010 23:52:49 -0800 (PST)
In-Reply-To: <AANLkTi=zf5r_8b4kPhZPKhXbdnjdkkD5K57-Pso0AXL-@mail.gmail.com>
References: <AANLkTinHt928uox=NQju5Vey91YfTN28NzTA6wZEK8X8@mail.gmail.com>
	<61840.91458.qm@web65503.mail.ac4.yahoo.com>
	<AANLkTi=zf5r_8b4kPhZPKhXbdnjdkkD5K57-Pso0AXL-@mail.gmail.com>
Date: Thu, 23 Dec 2010 23:52:49 -0800
X-Google-Sender-Auth: elmGw7W-THiDmJorBtbKYpFNU9c
Message-ID: <AANLkTikO3ZkgnSQDzZ6XjN1rgCp4LfpEQEnMAUi1EUfH@mail.gmail.com>
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of
 branch-0.20-append branch?
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

The intent of the proposed release off the branch-0.20-append was
never to derail, =93hurt=94, or distract from the Hadoop 0.22 effort.  The
HBase crew are up for helping out testing and debugging and the intent
is to run atop the 0.22 version of append as well as 0.20=92s append.  A
release off the branch-0.20-append branch was more about a =91stop-gap=92,
see Todd=92s explication above, or a =91fig-leaf=92 as Andrew describes it
while 0.22 is stabilizing.

Suggestions that projects like HBase hibernate until 0.22 don=92t help
(See Ryan=92s comments for a sense of why). We can just keep on with
what we=92ve been doing up to this if the feeling is that an append
release could somehow jeopardize the 0.22 effort.  Its kinda hokey
having to point users at some random looking branch [1] telling them
build their own but thankfully this is not their only option.

I=92ve enjoyed the healthy back and forth,

St.Ack
1. To be clear, 'random looking branch' is fruit of a bunch of
hardwork by Facebookers and Clouderians.

From general-return-2612-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Dec 24 18:57:36 2010
Return-Path: <general-return-2612-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21467 invoked from network); 24 Dec 2010 18:57:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Dec 2010 18:57:36 -0000
Received: (qmail 84466 invoked by uid 500); 24 Dec 2010 18:57:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84415 invoked by uid 500); 24 Dec 2010 18:57:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84407 invoked by uid 99); 24 Dec 2010 18:57:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Dec 2010 18:57:33 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 24 Dec 2010 18:57:33 +0000
Received: (qmail 21340 invoked by uid 99); 24 Dec 2010 18:57:13 -0000
Received: from localhost.apache.org (HELO mail-iy0-f176.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Dec 2010 18:57:13 +0000
Received: by iyb26 with SMTP id 26so7132536iyb.35
        for <general@hadoop.apache.org>; Fri, 24 Dec 2010 10:57:12 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.17.203 with SMTP id t11mr9446803iba.141.1293217032350;
 Fri, 24 Dec 2010 10:57:12 -0800 (PST)
Received: by 10.231.167.199 with HTTP; Fri, 24 Dec 2010 10:57:12 -0800 (PST)
In-Reply-To: <AANLkTikO3ZkgnSQDzZ6XjN1rgCp4LfpEQEnMAUi1EUfH@mail.gmail.com>
References: <AANLkTinHt928uox=NQju5Vey91YfTN28NzTA6wZEK8X8@mail.gmail.com>
	<61840.91458.qm@web65503.mail.ac4.yahoo.com>
	<AANLkTi=zf5r_8b4kPhZPKhXbdnjdkkD5K57-Pso0AXL-@mail.gmail.com>
	<AANLkTikO3ZkgnSQDzZ6XjN1rgCp4LfpEQEnMAUi1EUfH@mail.gmail.com>
Date: Fri, 24 Dec 2010 10:57:12 -0800
Message-ID: <AANLkTikMnE+vKdsu1R2LZ23V4v8jmiSCgYY9sMXZEZPF@mail.gmail.com>
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of
 branch-0.20-append branch?
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Does anything go wrong if HBase were to release the 0.20-append branch
as its own product?

Even if it were short-lived, it sounds like that would give HBase
users a working append, the HBase project could decide when to retire
that work (and support it concurrently with post-0.20 append), and it
sidesteps the versioning issue. -C

On Thu, Dec 23, 2010 at 11:52 PM, Stack <stack@duboce.net> wrote:
> The intent of the proposed release off the branch-0.20-append was
> never to derail, =93hurt=94, or distract from the Hadoop 0.22 effort. =A0=
The
> HBase crew are up for helping out testing and debugging and the intent
> is to run atop the 0.22 version of append as well as 0.20=92s append. =A0=
A
> release off the branch-0.20-append branch was more about a =91stop-gap=92=
,
> see Todd=92s explication above, or a =91fig-leaf=92 as Andrew describes i=
t
> while 0.22 is stabilizing.
>
> Suggestions that projects like HBase hibernate until 0.22 don=92t help
> (See Ryan=92s comments for a sense of why). We can just keep on with
> what we=92ve been doing up to this if the feeling is that an append
> release could somehow jeopardize the 0.22 effort. =A0Its kinda hokey
> having to point users at some random looking branch [1] telling them
> build their own but thankfully this is not their only option.
>
> I=92ve enjoyed the healthy back and forth,
>
> St.Ack
> 1. To be clear, 'random looking branch' is fruit of a bunch of
> hardwork by Facebookers and Clouderians.
>

From general-return-2613-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Dec 24 19:07:36 2010
Return-Path: <general-return-2613-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31063 invoked from network); 24 Dec 2010 19:07:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Dec 2010 19:07:35 -0000
Received: (qmail 88460 invoked by uid 500); 24 Dec 2010 19:07:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88363 invoked by uid 500); 24 Dec 2010 19:07:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88355 invoked by uid 99); 24 Dec 2010 19:07:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Dec 2010 19:07:34 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Dec 2010 19:07:26 +0000
Received: by fxm2 with SMTP id 2so7172634fxm.35
        for <general@hadoop.apache.org>; Fri, 24 Dec 2010 11:07:06 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        bh=zTturMsdH2ELtWhfVnwXgKQeqhvX1wK+IYhiMaV4QS8=;
        b=KT7wmtGLjxRsMFqFsv/O1Z++7ZVgk5KbmaOBbIbHUePjpqupJUn2y59EOLqNW1UggB
         4w1BR5sZ0OoTvQ8w+tvANluDVanr1xSH47BugqTF1PRbGWfyn1Fi+k9jnnd1HPxTzuoh
         s6LnY3KrPuKHP1g0XAXh27BKRbGYstI73obnU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=qMNbyWzzUYYrXELEsXWGLF+REGyV3jYFzDzuL2EGexkz7Lf9IO5fSapYDtBrX1I6Eb
         Egd1xgMYOe4YHpjxPmZpxb1VHR962ZUoSfMDkhHfYG7ByXPmiulT2PMiLR4F4RX+JeLJ
         p5uUO2Ivy6p1xwmQOpumM1ywgHGsqwijp0LgM=
MIME-Version: 1.0
Received: by 10.223.125.196 with SMTP id z4mr10934far.124.1293217604011; Fri,
 24 Dec 2010 11:06:44 -0800 (PST)
Sender: saint.ack@gmail.com
Received: by 10.223.83.9 with HTTP; Fri, 24 Dec 2010 11:06:43 -0800 (PST)
In-Reply-To: <AANLkTikMnE+vKdsu1R2LZ23V4v8jmiSCgYY9sMXZEZPF@mail.gmail.com>
References: <AANLkTinHt928uox=NQju5Vey91YfTN28NzTA6wZEK8X8@mail.gmail.com>
	<61840.91458.qm@web65503.mail.ac4.yahoo.com>
	<AANLkTi=zf5r_8b4kPhZPKhXbdnjdkkD5K57-Pso0AXL-@mail.gmail.com>
	<AANLkTikO3ZkgnSQDzZ6XjN1rgCp4LfpEQEnMAUi1EUfH@mail.gmail.com>
	<AANLkTikMnE+vKdsu1R2LZ23V4v8jmiSCgYY9sMXZEZPF@mail.gmail.com>
Date: Fri, 24 Dec 2010 11:06:43 -0800
X-Google-Sender-Auth: qr8xhjbSNrOlFOyOOx6b5yzRsrM
Message-ID: <AANLkTimv8mGCL3+q6KS+nc3XCECCxQLSr+QJkD7aDt_V@mail.gmail.com>
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of
 branch-0.20-append branch?
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Fri, Dec 24, 2010 at 10:57 AM, Chris Douglas <cdouglas@apache.org> wrote:
> Does anything go wrong if HBase were to release the 0.20-append branch
> as its own product?
>

This is an interesting notion.  We'd host it at hbase.apache.org
alongside our download?  Would that be OK with others?
St.Ack

From general-return-2614-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Dec 24 19:29:28 2010
Return-Path: <general-return-2614-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34650 invoked from network); 24 Dec 2010 19:29:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Dec 2010 19:29:28 -0000
Received: (qmail 99789 invoked by uid 500); 24 Dec 2010 19:29:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99720 invoked by uid 500); 24 Dec 2010 19:29:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 99710 invoked by uid 99); 24 Dec 2010 19:29:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Dec 2010 19:29:27 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Dec 2010 19:29:20 +0000
Received: by iyb26 with SMTP id 26so7150315iyb.35
        for <general@hadoop.apache.org>; Fri, 24 Dec 2010 11:28:58 -0800 (PST)
MIME-Version: 1.0
Received: by 10.42.213.200 with SMTP id gx8mr9800792icb.362.1293218938001;
 Fri, 24 Dec 2010 11:28:58 -0800 (PST)
Received: by 10.42.178.193 with HTTP; Fri, 24 Dec 2010 11:28:57 -0800 (PST)
In-Reply-To: <AANLkTimv8mGCL3+q6KS+nc3XCECCxQLSr+QJkD7aDt_V@mail.gmail.com>
References: <AANLkTinHt928uox=NQju5Vey91YfTN28NzTA6wZEK8X8@mail.gmail.com>
	<61840.91458.qm@web65503.mail.ac4.yahoo.com>
	<AANLkTi=zf5r_8b4kPhZPKhXbdnjdkkD5K57-Pso0AXL-@mail.gmail.com>
	<AANLkTikO3ZkgnSQDzZ6XjN1rgCp4LfpEQEnMAUi1EUfH@mail.gmail.com>
	<AANLkTikMnE+vKdsu1R2LZ23V4v8jmiSCgYY9sMXZEZPF@mail.gmail.com>
	<AANLkTimv8mGCL3+q6KS+nc3XCECCxQLSr+QJkD7aDt_V@mail.gmail.com>
Date: Fri, 24 Dec 2010 11:28:57 -0800
Message-ID: <AANLkTimfCtsj0--j9f-NwocWvueHdcDwbU+bSGhbnhhW@mail.gmail.com>
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of
 branch-0.20-append branch?
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=002354477c6e6a998004982cff1b
X-Virus-Checked: Checked by ClamAV on apache.org

--002354477c6e6a998004982cff1b
Content-Type: text/plain; charset=ISO-8859-1

That sounds like a reasonable solution to me: the HBase team bears the
burden of cutting an maintaining the release, while Hadoop Core can proceed
with 0.22. HBase had its own version of ZooKeeper in there for a while, if I
recall correctly, so it's not without precedent. No funky version numbers
have to be floating around Hadoop-land, and hopefully HBase can move back to
HDFS when 0.22 is released. It's not ideal, but potentially the best
solution given the current constraints.

On Fri, Dec 24, 2010 at 11:06 AM, Stack <stack@duboce.net> wrote:

> On Fri, Dec 24, 2010 at 10:57 AM, Chris Douglas <cdouglas@apache.org>
> wrote:
> > Does anything go wrong if HBase were to release the 0.20-append branch
> > as its own product?
> >
>
> This is an interesting notion.  We'd host it at hbase.apache.org
> alongside our download?  Would that be OK with others?
> St.Ack
>

--002354477c6e6a998004982cff1b--

From general-return-2615-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Dec 24 19:32:04 2010
Return-Path: <general-return-2615-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35484 invoked from network); 24 Dec 2010 19:32:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Dec 2010 19:32:04 -0000
Received: (qmail 2618 invoked by uid 500); 24 Dec 2010 19:32:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2566 invoked by uid 500); 24 Dec 2010 19:32:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2558 invoked by uid 99); 24 Dec 2010 19:32:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Dec 2010 19:32:02 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 24 Dec 2010 19:32:00 +0000
Received: (qmail 35284 invoked by uid 99); 24 Dec 2010 19:31:39 -0000
Received: from localhost.apache.org (HELO mail-iy0-f176.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Dec 2010 19:31:39 +0000
Received: by iyb26 with SMTP id 26so7151839iyb.35
        for <general@hadoop.apache.org>; Fri, 24 Dec 2010 11:31:38 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.19.77 with SMTP id z13mr9419855iba.166.1293219098269; Fri,
 24 Dec 2010 11:31:38 -0800 (PST)
Received: by 10.231.167.199 with HTTP; Fri, 24 Dec 2010 11:31:38 -0800 (PST)
In-Reply-To: <AANLkTimv8mGCL3+q6KS+nc3XCECCxQLSr+QJkD7aDt_V@mail.gmail.com>
References: <AANLkTinHt928uox=NQju5Vey91YfTN28NzTA6wZEK8X8@mail.gmail.com>
	<61840.91458.qm@web65503.mail.ac4.yahoo.com>
	<AANLkTi=zf5r_8b4kPhZPKhXbdnjdkkD5K57-Pso0AXL-@mail.gmail.com>
	<AANLkTikO3ZkgnSQDzZ6XjN1rgCp4LfpEQEnMAUi1EUfH@mail.gmail.com>
	<AANLkTikMnE+vKdsu1R2LZ23V4v8jmiSCgYY9sMXZEZPF@mail.gmail.com>
	<AANLkTimv8mGCL3+q6KS+nc3XCECCxQLSr+QJkD7aDt_V@mail.gmail.com>
Date: Fri, 24 Dec 2010 11:31:38 -0800
Message-ID: <AANLkTikkK_YWUNaVsH-9Mr7BM_L02HmntAOLtk_XrVyY@mail.gmail.com>
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of
 branch-0.20-append branch?
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Calling it something other than Hadoop would avoid confusing users
(and HBase could then release bug fixes, etc. on its own schedule),
but from how it's been described: this is acknowledging the reality of
the situation, not proposing something radical.

HBase can be backed by the HBase FooFS and HDFS. If the former can be
retired as a legacy platform that'd be ideal, but Hadoop will have to
earn it. -C

On Fri, Dec 24, 2010 at 11:06 AM, Stack <stack@duboce.net> wrote:
> On Fri, Dec 24, 2010 at 10:57 AM, Chris Douglas <cdouglas@apache.org> wro=
te:
>> Does anything go wrong if HBase were to release the 0.20-append branch
>> as its own product?
>>
>
> This is an interesting notion. =A0We'd host it at hbase.apache.org
> alongside our download? =A0Would that be OK with others?
> St.Ack
>

From general-return-2616-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Dec 25 21:21:45 2010
Return-Path: <general-return-2616-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12648 invoked from network); 25 Dec 2010 21:21:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Dec 2010 21:21:45 -0000
Received: (qmail 31714 invoked by uid 500); 25 Dec 2010 21:21:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31547 invoked by uid 500); 25 Dec 2010 21:21:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 73077 invoked by uid 99); 25 Dec 2010 06:43:44 -0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
From: Arun C Murthy <arunc@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 24 Dec 2010 22:41:13 -0800
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of
 branch-0.20-append branch?
Thread-Topic: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of
 branch-0.20-append branch?
Thread-Index: Acuj/tK5zWUIRp8ZT8636FkVxtLG1Q==
Message-ID: <BD302D1E-F424-4C5F-8443-6584037DA5AD@yahoo-inc.com>
References: <AANLkTinHt928uox=NQju5Vey91YfTN28NzTA6wZEK8X8@mail.gmail.com>
 <61840.91458.qm@web65503.mail.ac4.yahoo.com>
 <AANLkTi=zf5r_8b4kPhZPKhXbdnjdkkD5K57-Pso0AXL-@mail.gmail.com>
 <AANLkTikO3ZkgnSQDzZ6XjN1rgCp4LfpEQEnMAUi1EUfH@mail.gmail.com>
 <AANLkTikMnE+vKdsu1R2LZ23V4v8jmiSCgYY9sMXZEZPF@mail.gmail.com>
 <AANLkTimv8mGCL3+q6KS+nc3XCECCxQLSr+QJkD7aDt_V@mail.gmail.com>
 <AANLkTikkK_YWUNaVsH-9Mr7BM_L02HmntAOLtk_XrVyY@mail.gmail.com>
In-Reply-To: <AANLkTikkK_YWUNaVsH-9Mr7BM_L02HmntAOLtk_XrVyY@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

I have a feeling we are thinking too much here.=20

The reality is that the Hadoop community was in favor of porting append (fi=
xes?) back to a branch based off hadoop-0.20 a while ago (Dhruba proposed t=
he branch).

I see no reason we can't release it now, under a reasonable release name.

As Stack and Ryan have pointed out, we have severely hampered HBase so far.=
 It behooves us to facilitate users of the entire stack to easily access Ap=
ache releases. Plus, Stack and co. are volunteering their own time (thanks!=
).

Arun

Sent from my iPhone

On Dec 24, 2010, at 11:33 AM, "Chris Douglas" <cdouglas@apache.org> wrote:

> Calling it something other than Hadoop would avoid confusing users
> (and HBase could then release bug fixes, etc. on its own schedule),
> but from how it's been described: this is acknowledging the reality of
> the situation, not proposing something radical.
>=20
> HBase can be backed by the HBase FooFS and HDFS. If the former can be
> retired as a legacy platform that'd be ideal, but Hadoop will have to
> earn it. -C
>=20
> On Fri, Dec 24, 2010 at 11:06 AM, Stack <stack@duboce.net> wrote:
>> On Fri, Dec 24, 2010 at 10:57 AM, Chris Douglas <cdouglas@apache.org> wr=
ote:
>>> Does anything go wrong if HBase were to release the 0.20-append branch
>>> as its own product?
>>>=20
>>=20
>> This is an interesting notion.  We'd host it at hbase.apache.org
>> alongside our download?  Would that be OK with others?
>> St.Ack
>>=20

From general-return-2617-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Dec 26 06:03:36 2010
Return-Path: <general-return-2617-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57931 invoked from network); 26 Dec 2010 06:03:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Dec 2010 06:03:35 -0000
Received: (qmail 16570 invoked by uid 500); 26 Dec 2010 06:03:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16435 invoked by uid 500); 26 Dec 2010 06:03:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16427 invoked by uid 99); 26 Dec 2010 06:03:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Dec 2010 06:03:33 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of liulei412@gmail.com designates 209.85.218.48 as permitted sender)
Received: from [209.85.218.48] (HELO mail-yi0-f48.google.com) (209.85.218.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Dec 2010 06:03:26 +0000
Received: by yib17 with SMTP id 17so1679936yib.35
        for <general@hadoop.apache.org>; Sat, 25 Dec 2010 22:03:05 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=gER+zBeC+fow005Hoe1FSACsuKtuXWRWJeYSNkW+OfI=;
        b=Wpnf0PPaq8AXfoP5ZlFMWl4M0xUXmb/GyjKGGBkSXbgLUIFNfYcTzpYXv0RVF2CHNY
         DJwJPJgAaEGALSBJIeSg+C1t1HqfyNXVd+Idj6D5jlJV8BpkhmkgJMNuWvOD868tZhVT
         ml8nfPFrTSVBF83+ZQxHxiiFBi5cuwfd9Q6ic=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=xtkp2Y3+uDoFZuNYxqa5/1TI4xKMSS2n8ZvwnrZCtUK+jL6u/mlNH+GHwtINhpMqEH
         6zPehTPu/EPp/hVuhamISX9F8Xr0FmWLH9ovEhHbli8BatIDHWA7OunKha/Boiq+Y72e
         NWX6bqRbzbtUFYgK8iHdj9inEwq2I99tU6Nfw=
MIME-Version: 1.0
Received: by 10.151.147.8 with SMTP id z8mr15584831ybn.101.1293343385089; Sat,
 25 Dec 2010 22:03:05 -0800 (PST)
Received: by 10.151.38.17 with HTTP; Sat, 25 Dec 2010 22:03:05 -0800 (PST)
Date: Sun, 26 Dec 2010 14:03:05 +0800
Message-ID: <AANLkTikAXnBtEKTvy8Rn-zE4vTk0f3vNSDTqaJur8sdh@mail.gmail.com>
Subject: how does hadoop handle the counter of the failed task and speculative task
From: lei liu <liulei412@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001517511b040a75e2049849f9ff

--001517511b040a75e2049849f9ff
Content-Type: text/plain; charset=ISO-8859-1

I define the counter to count the bad records, there is below code in map
task;
 reporter.incrCounter("bad',"records', 1),

When the job is completed, the pritnt the result to use below code:
long total = counters.findCounter("bad","records").getCounter();


But I have two questions about the counter:
1. If the map task is retried 4 times, and the last map task is successful,
I think the counter of the other three map tasks should be not included in
ultima result, is that right?
2. If there are speculative tasks,  I think the counter of speculative tasks
should be not included in ultima result, is that right?


Thanks,

LiuLei

--001517511b040a75e2049849f9ff--

From general-return-2618-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Dec 26 09:59:09 2010
Return-Path: <general-return-2618-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23827 invoked from network); 26 Dec 2010 09:59:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Dec 2010 09:59:09 -0000
Received: (qmail 81357 invoked by uid 500); 26 Dec 2010 09:59:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81146 invoked by uid 500); 26 Dec 2010 09:59:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81138 invoked by uid 99); 26 Dec 2010 09:59:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Dec 2010 09:59:06 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Dec 2010 09:58:59 +0000
Received: by fxm2 with SMTP id 2so7807832fxm.35
        for <general@hadoop.apache.org>; Sun, 26 Dec 2010 01:58:39 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=m6S2XFBjCNMeyleowQwICW2HIMXBKeNCPgalCGfoNeo=;
        b=VUdD0t7efJo/AyFlcmm4HyAtS1ZHkToQEvawcRoAHZxwiorekqJUcXEJJGOBAl1Zw7
         CCocvOuRV8lVpowx5U+U+4RAoc8BO5gRtfF5+3gpq95t08yxUbDHRnqQZOpTAZ3LfULr
         Rs33LB1LoAuer0x93MqwFEjbprWUK/m4LX7xY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=J9d2uqdGyT5V5Ljw0NBJ4oPIU9Au++dz3jjGiGpz5mVZ6P9288CSPu8lDaw7RSTFTG
         f4sk8WclBDb8piytSltIQraAiheC+5VHIw8L81N57lJDLenTquPODW/PCzQGn2+RqxbP
         OVgSSw8+uz0ZY5VFBNAk+jy+NVemX6VfZuYDU=
Received: by 10.223.114.65 with SMTP id d1mr1817216faq.36.1293357518524; Sun,
 26 Dec 2010 01:58:38 -0800 (PST)
MIME-Version: 1.0
Received: by 10.223.120.14 with HTTP; Sun, 26 Dec 2010 01:58:18 -0800 (PST)
In-Reply-To: <AANLkTikAXnBtEKTvy8Rn-zE4vTk0f3vNSDTqaJur8sdh@mail.gmail.com>
References: <AANLkTikAXnBtEKTvy8Rn-zE4vTk0f3vNSDTqaJur8sdh@mail.gmail.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Sun, 26 Dec 2010 15:28:18 +0530
Message-ID: <AANLkTikQj2zRuC5PdffBseYJum+AsM1AtCEGzNLZDTTR@mail.gmail.com>
Subject: Re: how does hadoop handle the counter of the failed task and
 speculative task
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

On Sun, Dec 26, 2010 at 11:33 AM, lei liu <liulei412@gmail.com> wrote:
> But I have two questions about the counter:
> 1. If the map task is retried 4 times, and the last map task is successfu=
l,
> I think the counter of the other three map tasks should be not included i=
n
> ultima result, is that right?

Yes.

> 2. If there are speculative tasks, =A0I think the counter of speculative =
tasks
> should be not included in ultima result, is that right?

Yes.

A Job's counters are calculated via the sum of all TaskInProgress-es
(TIPs) under it ("It" being the JIP - JobInProgress). So the end
result would be only the successful ones as the counters are
recomputed for all changes in the TIP's status. During execution, it
would project the best available counter (as in, from the task attempt
that has the best overall progress compared to others).

--=20
Harsh J
www.harshj.com

From general-return-2619-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Dec 26 10:52:32 2010
Return-Path: <general-return-2619-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38712 invoked from network); 26 Dec 2010 10:52:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Dec 2010 10:52:32 -0000
Received: (qmail 382 invoked by uid 500); 26 Dec 2010 10:52:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 177 invoked by uid 500); 26 Dec 2010 10:52:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 169 invoked by uid 99); 26 Dec 2010 10:52:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Dec 2010 10:52:29 +0000
X-ASF-Spam-Status: No, hits=3.7 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of liulei412@gmail.com designates 209.85.213.176 as permitted sender)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Dec 2010 10:52:22 +0000
Received: by yxm8 with SMTP id 8so4089532yxm.35
        for <general@hadoop.apache.org>; Sun, 26 Dec 2010 02:52:01 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=CvScoQZLHIVtGcnHSor21Ef5095lh7Oz3hamAZkZaOg=;
        b=LOvdmFvh1fssRwTVtHDUkB4cHQPzhM93LnWTeRoeIGzeJ6dcJxkTKxNyhtuSOxvF2i
         Xhr9LXDwuKQfcdetX2HPiWSyrvDIyHhIkUGGTzUDT7EaXUHsARiUxZUEUEOIQUSkPyvX
         DJ+Yt3cQrMSQahY8HzQOsNmMT8qngidYYt/4E=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=fWMJmDPWdT/Oy35VL2kLWYTrjqBY+zGZIf61Sab7Yt7k9UxE9WLn+c3uuLVgK6+0Ja
         i4WTUUgYKlvDeCwcn4yS8cVvX9HrjLVbWGCGxP3ih0rjY+pcb+HQT2HebxM+2LBanJab
         nECFIfoxAfqqqOerX6g9rZCRM/ZA6jxxUgXNg=
MIME-Version: 1.0
Received: by 10.150.200.16 with SMTP id x16mr15569166ybf.215.1293360721812;
 Sun, 26 Dec 2010 02:52:01 -0800 (PST)
Received: by 10.151.38.17 with HTTP; Sun, 26 Dec 2010 02:52:01 -0800 (PST)
Date: Sun, 26 Dec 2010 18:52:01 +0800
Message-ID: <AANLkTinyL05=-UQ+4Vs45R_rgxH-47jfddxBzq-7BXC=@mail.gmail.com>
Subject: use hadoop to create solr index
From: lei liu <liulei412@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd348ac63dba504984e02a4

--000e0cd348ac63dba504984e02a4
Content-Type: text/plain; charset=ISO-8859-1

I want to use hadoop to create solr index, I need to add solr.jar,lucene.jar
and other many jars to job classpath, I know the Jobconf.setJar(String jar)
method can add one jar to job classpath, I use the method to add the jar
which include map/reduce classes to job classpath, how can I add other jars
to job classpath?


Thanks,


LiuLei

--000e0cd348ac63dba504984e02a4--

From general-return-2620-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Dec 26 11:29:36 2010
Return-Path: <general-return-2620-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50904 invoked from network); 26 Dec 2010 11:29:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Dec 2010 11:29:35 -0000
Received: (qmail 10479 invoked by uid 500); 26 Dec 2010 11:29:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 10135 invoked by uid 500); 26 Dec 2010 11:29:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 10127 invoked by uid 99); 26 Dec 2010 11:29:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Dec 2010 11:29:32 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Dec 2010 11:29:26 +0000
Received: by fxm2 with SMTP id 2so7831072fxm.35
        for <general@hadoop.apache.org>; Sun, 26 Dec 2010 03:29:06 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type;
        bh=CpvamFyzTXzn++pFTyO9PiS7wQeGvnc5TY4XY01nMl4=;
        b=f7S9YJKqbKNKky7IKsvuTkejR27bmgMh8PWN8/rZiBFwl10SvgSNR20Kb/TPeEcDyb
         fMRgZYfTzcjK1z/EdyhDaao6dNwV/ybaGsLg2MwQGOWf0RT4zehQN7SOoWUBtrcfheZq
         HmItLSokn5WnZCmEkK7rimZwWmpwkK+MSbxaY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=Wo/Nr4qUBAYQlGcR1NYNFfXl4H8YoAJ27M4sVZy8kL/3ppD0Wldm/IBEF5Jv0zsQPU
         wINxjUK0sNURL9nZWG7fNJsG2L8v3tl0bkLRELQ3J2HApfBdopR1/dTUGb1UGxfmjR25
         /VNHpOY2ALh6ZI7yXZarh8vTTmDdrUZg0iezk=
Received: by 10.223.87.3 with SMTP id u3mr1810923fal.131.1293362944870; Sun,
 26 Dec 2010 03:29:04 -0800 (PST)
MIME-Version: 1.0
Received: by 10.223.120.14 with HTTP; Sun, 26 Dec 2010 03:28:44 -0800 (PST)
In-Reply-To: <AANLkTinyL05=-UQ+4Vs45R_rgxH-47jfddxBzq-7BXC=@mail.gmail.com>
References: <AANLkTinyL05=-UQ+4Vs45R_rgxH-47jfddxBzq-7BXC=@mail.gmail.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Sun, 26 Dec 2010 16:58:44 +0530
Message-ID: <AANLkTin=E2YkSmEcVEuZj+VLExce-3u1ZKqEgeuczBTa@mail.gmail.com>
Subject: Re: use hadoop to create solr index
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Use the DistributedCache. Load your jars to HDFS and then use DC to
add it to your job driver program.

http://hadoop.apache.org/common/docs/r0.20.2/api/org/apache/hadoop/filecache/DistributedCache.html
[Look at addFileToClassPath(...)]

Or use the GenericOption "-libjars" from command-line while triggering
the job, as documented here:
http://hadoop.apache.org/common/docs/r0.20.2/commands_manual.html#Generic+Options.

-- 
Harsh J
www.harshj.com

From general-return-2621-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Dec 27 15:05:02 2010
Return-Path: <general-return-2621-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31199 invoked from network); 27 Dec 2010 15:05:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Dec 2010 15:05:02 -0000
Received: (qmail 66455 invoked by uid 500); 27 Dec 2010 15:05:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66302 invoked by uid 500); 27 Dec 2010 15:04:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66294 invoked by uid 99); 27 Dec 2010 15:04:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Dec 2010 15:04:58 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of magawake@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Dec 2010 15:04:51 +0000
Received: by qwh6 with SMTP id 6so8916756qwh.35
        for <general@hadoop.apache.org>; Mon, 27 Dec 2010 07:04:30 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=wjNdJqxCkLyNmFY9m6lgH66e52QMDl9rvpe8ZBpddxM=;
        b=M17i9HXsxmgC5RceiFKwZFVWzNtAM0ar48WT1Hff+I+g8LCHd+A0FppTDj+dbfLesl
         DtGlyvK3X1O+X4NZHZv8gZ02P4gD80zELUU1AYuJbrm8C6sEzYkVZEoZigk0HlbOgF+8
         b4gdSsnnZOpPo6wFRM+30zlixvTSnQ6kpd5N0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=o7ypPFNSsFfnABZ9qHQkv33O/s+Xuj7B+VcAbuNjpOdtGqjgkRRta3nApeDayI5uhl
         2SnCpZ9YMjXGJ+9bj/lm7/pdShrW1eCk6HEjStlhFGicPZGxfZPywTxdFXFslr41QO1V
         JtJ19ojJ5q0pKNCLJcreUtDd45qE5DbIHb150=
MIME-Version: 1.0
Received: by 10.224.199.6 with SMTP id eq6mr11671805qab.272.1293462270046;
 Mon, 27 Dec 2010 07:04:30 -0800 (PST)
Received: by 10.220.118.20 with HTTP; Mon, 27 Dec 2010 07:04:30 -0800 (PST)
Date: Mon, 27 Dec 2010 10:04:30 -0500
Message-ID: <AANLkTimo1MdB5oZXy6n9gPOv+ppZ=OXFa=0_PKvZFHUF@mail.gmail.com>
Subject: Namenode failover
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
X-Virus-Checked: Checked by ClamAV on apache.org

I was wondering if anyone has the procedure for Namenode failover for
0.21? I have seen the instructions for 0.20
(http://wiki.apache.org/hadoop/NameNodeFailover) but it seems 0.21 has
new features such as a backup node. Would a backup node make the
failover transparent?

TIA

From general-return-2622-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Dec 27 15:33:38 2010
Return-Path: <general-return-2622-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 48548 invoked from network); 27 Dec 2010 15:33:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Dec 2010 15:33:38 -0000
Received: (qmail 94099 invoked by uid 500); 27 Dec 2010 15:33:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93920 invoked by uid 500); 27 Dec 2010 15:33:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93912 invoked by uid 99); 27 Dec 2010 15:33:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Dec 2010 15:33:35 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Dec 2010 15:33:30 +0000
Received: by fxm2 with SMTP id 2so8415065fxm.35
        for <general@hadoop.apache.org>; Mon, 27 Dec 2010 07:33:09 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type;
        bh=ZyJyWZkR5/AaBMMwmzcj9kX/alp4YjLFyyzn4pJTQzI=;
        b=pluuFdkqHh9/LempQu8fnKGgsUgpKFcDiRXp1VyIPZ9ozxAFn4gK87HV8JZDnlDu//
         ynkfQ9UWf6m8TYIB6Y94/9VznC3KWmGSk5MNFeHkpiPxkseVYC3gQ6QWpmQaSYisoZ7q
         EE8S79a3xDIGlPbLif0U/QFLvXMdQMl5aQbMU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=tFC1m/za9wjMh33Uc3bO7P4JiWDvzRtY1P6Oarc4GuJztUez6a0ZnirCbqVFMxtMc9
         lbJ/goPgdXioQsxl8uHkeFUniaMepLSXMCMUq/7fFcepO2F55675iD01NvA30auII2v6
         DtF+LD7B+/c8LsbEi7y9MJ7IRnN4Fm9NSP8+Y=
Received: by 10.223.114.65 with SMTP id d1mr3059590faq.36.1293463988602; Mon,
 27 Dec 2010 07:33:08 -0800 (PST)
MIME-Version: 1.0
Received: by 10.223.120.14 with HTTP; Mon, 27 Dec 2010 07:32:48 -0800 (PST)
In-Reply-To: <AANLkTimo1MdB5oZXy6n9gPOv+ppZ=OXFa=0_PKvZFHUF@mail.gmail.com>
References: <AANLkTimo1MdB5oZXy6n9gPOv+ppZ=OXFa=0_PKvZFHUF@mail.gmail.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Mon, 27 Dec 2010 21:02:48 +0530
Message-ID: <AANLkTi=L=0-K7WSNfmDMh0xrym1qz6hxpvLVpLb4i2MG@mail.gmail.com>
Subject: Re: Namenode failover
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

I think AvatarNode is what you're looking for. The BN will not take over AFAIK.

On Mon, Dec 27, 2010 at 8:34 PM, Mag Gam <magawake@gmail.com> wrote:
> I was wondering if anyone has the procedure for Namenode failover for
> 0.21? I have seen the instructions for 0.20
> (http://wiki.apache.org/hadoop/NameNodeFailover) but it seems 0.21 has
> new features such as a backup node. Would a backup node make the
> failover transparent?
>
> TIA
>



-- 
Harsh J
www.harshj.com

From general-return-2623-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Dec 27 17:12:52 2010
Return-Path: <general-return-2623-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88179 invoked from network); 27 Dec 2010 17:12:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Dec 2010 17:12:51 -0000
Received: (qmail 86307 invoked by uid 500); 27 Dec 2010 17:12:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86074 invoked by uid 500); 27 Dec 2010 17:12:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86066 invoked by uid 99); 27 Dec 2010 17:12:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Dec 2010 17:12:48 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Dec 2010 17:12:41 +0000
Received: by fxm2 with SMTP id 2so8482291fxm.35
        for <general@hadoop.apache.org>; Mon, 27 Dec 2010 09:12:21 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        bh=A+w/fCBDNojqj5N8Efyb0nATBOwHedVP9zaKEYlHM5c=;
        b=fe+3o0d+KRbfeGbrd++WbMNec41FrS5ehO4aB5vNlVlGg7ods6UgkLknGxXzqkJO0S
         mdsEjF90prr4WRpmnnhUGJfkRsZ9fL9eQkbqfar6cphNLnWwjJVmopdZw2Qmbcem9zRk
         yeMs+5LNx/D/VzaJpexx5yWBotCDuZoqfV5Rk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=wGx/3hPPbHmQ/SAjc+2tOKRt4/O6Myq2dpycGwuLmBExusiFA6M4WxP4Ibt6Vz84Qb
         A6/mmqDjasxLKbIKjpWTP3ekV+aOKzE86gtgFKOy1wx5YWfpOpIAs712DeWzEgBQvRoF
         4rFTqsiQNqxt12XscUgMGDMksWBr+QT7RXdP0=
MIME-Version: 1.0
Received: by 10.223.86.199 with SMTP id t7mr6226797fal.29.1293469940848; Mon,
 27 Dec 2010 09:12:20 -0800 (PST)
Sender: saint.ack@gmail.com
Received: by 10.223.83.9 with HTTP; Mon, 27 Dec 2010 09:12:20 -0800 (PST)
In-Reply-To: <AANLkTikkK_YWUNaVsH-9Mr7BM_L02HmntAOLtk_XrVyY@mail.gmail.com>
References: <AANLkTinHt928uox=NQju5Vey91YfTN28NzTA6wZEK8X8@mail.gmail.com>
	<61840.91458.qm@web65503.mail.ac4.yahoo.com>
	<AANLkTi=zf5r_8b4kPhZPKhXbdnjdkkD5K57-Pso0AXL-@mail.gmail.com>
	<AANLkTikO3ZkgnSQDzZ6XjN1rgCp4LfpEQEnMAUi1EUfH@mail.gmail.com>
	<AANLkTikMnE+vKdsu1R2LZ23V4v8jmiSCgYY9sMXZEZPF@mail.gmail.com>
	<AANLkTimv8mGCL3+q6KS+nc3XCECCxQLSr+QJkD7aDt_V@mail.gmail.com>
	<AANLkTikkK_YWUNaVsH-9Mr7BM_L02HmntAOLtk_XrVyY@mail.gmail.com>
Date: Mon, 27 Dec 2010 09:12:20 -0800
X-Google-Sender-Auth: Tg2wYt7DCEQNOHuj4_Yz_HVbho4
Message-ID: <AANLkTimcGFMN1qdEFZEcDShrTMCpA8uh2t=m3i6hZZHQ@mail.gmail.com>
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of
 branch-0.20-append branch?
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Fri, Dec 24, 2010 at 11:31 AM, Chris Douglas <cdouglas@apache.org> wrote:
> Calling it something other than Hadoop would avoid confusing users
> (and HBase could then release bug fixes, etc. on its own schedule),
> but from how it's been described: this is acknowledging the reality of
> the situation, not proposing something radical.
>

Calling it other than Hadoop would only confuse the situation even
more; "Trust all your data to fooFS!".  It'd also reeks of HDFS 'fork'
(HBase is not yet up for taking on such a burden).

I liked my original reading of your suggestion Chris -- even if it was
perhaps not what you intended -- where HBase would host
hadoop-0.20-append.  Thats not on?

St.Ack
P.S. Arun, I agree.

From general-return-2624-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Dec 27 19:20:34 2010
Return-Path: <general-return-2624-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37737 invoked from network); 27 Dec 2010 19:20:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Dec 2010 19:20:33 -0000
Received: (qmail 3191 invoked by uid 500); 27 Dec 2010 19:20:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3117 invoked by uid 500); 27 Dec 2010 19:20:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3109 invoked by uid 99); 27 Dec 2010 19:20:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Dec 2010 19:20:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 27 Dec 2010 19:20:29 +0000
Received: (qmail 37705 invoked by uid 99); 27 Dec 2010 19:20:07 -0000
Received: from localhost.apache.org (HELO mail-iy0-f176.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Dec 2010 19:20:07 +0000
Received: by iyb26 with SMTP id 26so9162185iyb.35
        for <general@hadoop.apache.org>; Mon, 27 Dec 2010 11:20:06 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.32.7 with SMTP id a7mr12659372ibd.16.1293477606582; Mon,
 27 Dec 2010 11:20:06 -0800 (PST)
Received: by 10.231.167.199 with HTTP; Mon, 27 Dec 2010 11:20:06 -0800 (PST)
In-Reply-To: <AANLkTimcGFMN1qdEFZEcDShrTMCpA8uh2t=m3i6hZZHQ@mail.gmail.com>
References: <AANLkTinHt928uox=NQju5Vey91YfTN28NzTA6wZEK8X8@mail.gmail.com>
	<61840.91458.qm@web65503.mail.ac4.yahoo.com>
	<AANLkTi=zf5r_8b4kPhZPKhXbdnjdkkD5K57-Pso0AXL-@mail.gmail.com>
	<AANLkTikO3ZkgnSQDzZ6XjN1rgCp4LfpEQEnMAUi1EUfH@mail.gmail.com>
	<AANLkTikMnE+vKdsu1R2LZ23V4v8jmiSCgYY9sMXZEZPF@mail.gmail.com>
	<AANLkTimv8mGCL3+q6KS+nc3XCECCxQLSr+QJkD7aDt_V@mail.gmail.com>
	<AANLkTikkK_YWUNaVsH-9Mr7BM_L02HmntAOLtk_XrVyY@mail.gmail.com>
	<AANLkTimcGFMN1qdEFZEcDShrTMCpA8uh2t=m3i6hZZHQ@mail.gmail.com>
Date: Mon, 27 Dec 2010 11:20:06 -0800
Message-ID: <AANLkTikPzcsMy8PZ3GPB4VyLcSARPoCEnn+XFhV0i6sW@mail.gmail.com>
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of
 branch-0.20-append branch?
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

> On Fri, Dec 24, 2010 at 11:31 AM, Chris Douglas <cdouglas@apache.org> wro=
te:
> Calling it other than Hadoop would only confuse the situation even
> more; "Trust all your data to fooFS!". =A0It'd also reeks of HDFS 'fork'
> (HBase is not yet up for taking on such a burden).

Unless I'm missing something, it is a fork. It's a temporary, friendly
fork, but it's what the HBase project has been using and supporting
for months. It hasn't had a label assigned to it, but it's a product
(a feature with a mostly-shared implementation across other forks, at
any rate).

> I liked my original reading of your suggestion Chris -- even if it was
> perhaps not what you intended -- where HBase would host
> hadoop-0.20-append. =A0Thats not on?

Your original reading was what I intended. The obstacle to releasing a
variant of Hadoop from the HBase project is the name. I'd be surprised
if TLPs were permitted to release under another project's name, even
if the other endorsed it. If that assumption is not a real constraint,
then I agree that there's no point in calling it something else. -C

From general-return-2625-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 28 17:09:59 2010
Return-Path: <general-return-2625-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 18535 invoked from network); 28 Dec 2010 17:09:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Dec 2010 17:09:58 -0000
Received: (qmail 79581 invoked by uid 500); 28 Dec 2010 17:09:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79216 invoked by uid 500); 28 Dec 2010 17:09:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 79206 invoked by uid 99); 28 Dec 2010 17:09:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Dec 2010 17:09:55 +0000
X-ASF-Spam-Status: No, hits=-1997.8 required=10.0
	tests=ALL_TRUSTED,HTML_MESSAGE,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 28 Dec 2010 17:09:53 +0000
Received: (qmail 18262 invoked by uid 99); 28 Dec 2010 17:09:31 -0000
Received: from localhost.apache.org (HELO mail-bw0-f48.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Dec 2010 17:09:31 +0000
Received: by bwz8 with SMTP id 8so9819927bwz.35
        for <general@hadoop.apache.org>; Tue, 28 Dec 2010 09:09:29 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.113.65 with SMTP id z1mr11390451bkp.58.1293556169159; Tue,
 28 Dec 2010 09:09:29 -0800 (PST)
Received: by 10.204.76.14 with HTTP; Tue, 28 Dec 2010 09:09:29 -0800 (PST)
In-Reply-To: <201012220856.27550.thomas@koch.ro>
References: <201012220856.27550.thomas@koch.ro>
Date: Tue, 28 Dec 2010 09:09:29 -0800
Message-ID: <AANLkTimRtoHJQ26111PayuAtQD6mSbXuSuar=C13d++G@mail.gmail.com>
Subject: Re: Test instance of Gerrit?
From: "Owen O'Malley" <omalley@apache.org>
To: infrastructure-dev@apache.org, thomas@koch.ro, general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502cd8ff5b50b04987b83e5
X-Virus-Checked: Checked by ClamAV on apache.org

--00504502cd8ff5b50b04987b83e5
Content-Type: text/plain; charset=UTF-8

On Tue, Dec 21, 2010 at 11:56 PM, Thomas Koch <thomas@koch.ro> wrote:

Now my question is, whether there's some interest from developers to give
> Gerrit a try? In this case, I'd setup a private toy instance and import
> your
> project.
>

I'll give it a try. Could you import hadoop common, hdfs, and mapreduce?

-- Owen

--00504502cd8ff5b50b04987b83e5--

From general-return-2626-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 28 22:05:19 2010
Return-Path: <general-return-2626-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 26369 invoked from network); 28 Dec 2010 22:05:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Dec 2010 22:05:18 -0000
Received: (qmail 85636 invoked by uid 500); 28 Dec 2010 22:05:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85555 invoked by uid 500); 28 Dec 2010 22:05:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85547 invoked by uid 99); 28 Dec 2010 22:05:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Dec 2010 22:05:17 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [192.101.109.31] (HELO emailgw03.pnl.gov) (192.101.109.31)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Dec 2010 22:05:10 +0000
X-IronPort-AV: E=Sophos;i="4.60,241,1291622400"; 
   d="scan'208";a="35761421"
Received: from emailhub02.pnl.gov ([130.20.251.62])
  by emailgw03.pnl.gov with ESMTP/TLS/AES128-SHA; 28 Dec 2010 14:04:49 -0800
Received: from Email05.pnl.gov ([130.20.251.69]) by emailhub02.pnl.gov
 ([130.20.251.62]) with mapi; Tue, 28 Dec 2010 14:04:48 -0800
From: "Taylor, Ronald C" <ronald.taylor@pnl.gov>
To: "'user@hbase.apache.org'" <user@hbase.apache.org>,
	"'general@hadoop.apache.org'" <general@hadoop.apache.org>
CC: "Taylor, Ronald C" <ronald.taylor@pnl.gov>, "Fox, Kevin M"
	<kevin.fox@pnl.gov>, "Brown, David M JR" <david.brown@pnl.gov>
Date: Tue, 28 Dec 2010 14:04:48 -0800
Subject: What is the fastest way to get a large amount of data into the
  Hadoop HDFS file system  (or Hbase)?
Thread-Topic: What is the fastest way to get a large amount of data into the
  Hadoop HDFS file system  (or Hbase)?
Thread-Index: Acul2gQhGBzjuFTiQL2qQO67SCiwKwA/gmuw
Message-ID: <A632B3F93D861843A3C966F3962C3F84010822D50026@EMAIL05.pnl.gov>
References: <590321.36008.qm@web31813.mail.mud.yahoo.com>
 <536061.77419.qm@web130101.mail.mud.yahoo.com>
 <A632B3F93D861843A3C966F3962C3F84010822D5000C@EMAIL05.pnl.gov>
 <4D18AF52.9080800@mozilla.com>
In-Reply-To: <4D18AF52.9080800@mozilla.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


Folks,

We plan on uploading large amounts of data on a regular basis onto a Hadoop=
 cluster, with Hbase operating on top of Hadoop. Figure eventually on the o=
rder of multiple terabytes per week. So - we are concerned about doing the =
uploads themselves as fast as possible from our native Linux file system in=
to HDFS. Figure files will be in, roughly, the 1 to 300 GB range.=20

Off the top of my head, I'm thinking that doing this in parallel using a Ja=
va MapReduce program would work fastest. So my idea would be to have a file=
 listing all the data files (full paths) to be uploaded, one per line, and =
then use that listing file as input to a MapReduce program.=20

Each Mapper would then upload one of the data files (using "hadoop fs -copy=
FromLocal <source> <dest>") in parallel with all the other Mappers, with th=
e Mappers operating on all the nodes of the cluster, spreading out the file=
 upload across the nodes.

Does that sound like a wise way to approach this? Are there better methods?=
 Anything else out there for doing automated upload in parallel? We would v=
ery much appreciate advice in this area, since we believe upload speed migh=
t become a bottleneck.

  - Ron Taylor

___________________________________________
Ronald Taylor, Ph.D.
Computational Biology & Bioinformatics Group

Pacific Northwest National Laboratory
902 Battelle Boulevard
P.O. Box 999, Mail Stop J4-33
Richland, WA  99352 USA
Office:  509-372-6568
Email: ronald.taylor@pnl.gov



From general-return-2627-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 28 22:27:24 2010
Return-Path: <general-return-2627-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33821 invoked from network); 28 Dec 2010 22:27:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Dec 2010 22:27:23 -0000
Received: (qmail 8955 invoked by uid 500); 28 Dec 2010 22:27:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8879 invoked by uid 500); 28 Dec 2010 22:27:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8824 invoked by uid 99); 28 Dec 2010 22:27:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Dec 2010 22:27:21 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of patrickangeles@gmail.com designates 209.85.215.176 as permitted sender)
Received: from [209.85.215.176] (HELO mail-ey0-f176.google.com) (209.85.215.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Dec 2010 22:27:14 +0000
Received: by eyz10 with SMTP id 10so4285752eyz.35
        for <general@hadoop.apache.org>; Tue, 28 Dec 2010 14:26:54 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:cc:content-type;
        bh=ot9wrs+lB/C7R3jZNp6+9iHipDUp6lVxgiet6/ySjmY=;
        b=QPBRiCNDOx400S9RKlGSzd/wH/yTI7ptyutHswz7GWECi1vMBRBISQ6ksXQknbXJTl
         SjGWsHAiIHuaAEOrPJgvHMgboDZsUT7dTrDx8tIxW7qeuSaaiblA2xK+1Vx25Foip/Lk
         rITOgkPhCMj2/oly7r6Z9s4ZqyBsskGpf3phc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:cc:content-type;
        b=GONJUhNSi4WlLbZ45QZX00lTYstknkakHoyVaGSPbJ6WZyCgIFCG/bmRy3TOqDnIzA
         yW1EPS4L10HISjxoLi0GhrS8VDRqRDp2Be5+yZ2ddJvslrzomfc14GcGkj8D9ZVdFqKa
         W6EECj3teinxY0meu4oXDoyX6ih++yHutMUvY=
MIME-Version: 1.0
Received: by 10.213.26.74 with SMTP id d10mr12381052ebc.61.1293575214266; Tue,
 28 Dec 2010 14:26:54 -0800 (PST)
Sender: patrickangeles@gmail.com
Received: by 10.213.108.147 with HTTP; Tue, 28 Dec 2010 14:26:54 -0800 (PST)
In-Reply-To: <A632B3F93D861843A3C966F3962C3F84010822D50026@EMAIL05.pnl.gov>
References: <590321.36008.qm@web31813.mail.mud.yahoo.com>
	<536061.77419.qm@web130101.mail.mud.yahoo.com>
	<A632B3F93D861843A3C966F3962C3F84010822D5000C@EMAIL05.pnl.gov>
	<4D18AF52.9080800@mozilla.com>
	<A632B3F93D861843A3C966F3962C3F84010822D50026@EMAIL05.pnl.gov>
Date: Tue, 28 Dec 2010 17:26:54 -0500
X-Google-Sender-Auth: IbSbpCBlG5eBbtBpr8OTHzRo-Ag
Message-ID: <AANLkTimxVqwnJDbq9pi7weR2KTAijWpUX1+_xqvAemTw@mail.gmail.com>
Subject: Re: What is the fastest way to get a large amount of data into the
 Hadoop HDFS file system (or Hbase)?
From: Patrick Angeles <patrick@cloudera.com>
To: general@hadoop.apache.org
Cc: "user@hbase.apache.org" <user@hbase.apache.org>, "Taylor, Ronald C" <ronald.taylor@pnl.gov>, 
	"Fox, Kevin M" <kevin.fox@pnl.gov>, "Brown, David M JR" <david.brown@pnl.gov>
Content-Type: multipart/alternative; boundary=0015174bdc9c22f88304987ff38c
X-Virus-Checked: Checked by ClamAV on apache.org

--0015174bdc9c22f88304987ff38c
Content-Type: text/plain; charset=ISO-8859-1

Ron,

While MapReduce can help to parallelize the load effort, your likely
bottleneck is the source system (where the files come from). If the files
are coming from a single server, then parallelizing the load won't gain you
much past a certain point. You have to figure in how fast you can read the
file(s) off disk(s) and push the bits through your network and finally onto
HDFS.

The best scenario is if you can parallelize the reads and have a fat network
pipe (10GbE or more) going into your Hadoop cluster.

Regards,

- Patrick

On Tue, Dec 28, 2010 at 5:04 PM, Taylor, Ronald C <ronald.taylor@pnl.gov>wrote:

>
> Folks,
>
> We plan on uploading large amounts of data on a regular basis onto a Hadoop
> cluster, with Hbase operating on top of Hadoop. Figure eventually on the
> order of multiple terabytes per week. So - we are concerned about doing the
> uploads themselves as fast as possible from our native Linux file system
> into HDFS. Figure files will be in, roughly, the 1 to 300 GB range.
>
> Off the top of my head, I'm thinking that doing this in parallel using a
> Java MapReduce program would work fastest. So my idea would be to have a
> file listing all the data files (full paths) to be uploaded, one per line,
> and then use that listing file as input to a MapReduce program.
>
> Each Mapper would then upload one of the data files (using "hadoop fs
> -copyFromLocal <source> <dest>") in parallel with all the other Mappers,
> with the Mappers operating on all the nodes of the cluster, spreading out
> the file upload across the nodes.
>
> Does that sound like a wise way to approach this? Are there better methods?
> Anything else out there for doing automated upload in parallel? We would
> very much appreciate advice in this area, since we believe upload speed
> might become a bottleneck.
>
>  - Ron Taylor
>
> ___________________________________________
> Ronald Taylor, Ph.D.
> Computational Biology & Bioinformatics Group
>
> Pacific Northwest National Laboratory
> 902 Battelle Boulevard
> P.O. Box 999, Mail Stop J4-33
> Richland, WA  99352 USA
> Office:  509-372-6568
> Email: ronald.taylor@pnl.gov
>
>
>

--0015174bdc9c22f88304987ff38c--

From general-return-2628-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Dec 28 22:48:24 2010
Return-Path: <general-return-2628-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38946 invoked from network); 28 Dec 2010 22:48:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Dec 2010 22:48:23 -0000
Received: (qmail 33475 invoked by uid 500); 28 Dec 2010 22:48:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33392 invoked by uid 500); 28 Dec 2010 22:48:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33376 invoked by uid 99); 28 Dec 2010 22:48:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Dec 2010 22:48:20 +0000
X-ASF-Spam-Status: No, hits=-0.1 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [192.101.109.31] (HELO emailgw03.pnl.gov) (192.101.109.31)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Dec 2010 22:48:13 +0000
X-IronPort-AV: E=Sophos;i="4.60,241,1291622400"; 
   d="scan'208,217";a="35763781"
Received: from emailhub02.pnl.gov ([130.20.251.62])
  by emailgw03.pnl.gov with ESMTP/TLS/AES128-SHA; 28 Dec 2010 14:47:53 -0800
Received: from Email05.pnl.gov ([130.20.251.69]) by emailhub02.pnl.gov
 ([130.20.251.62]) with mapi; Tue, 28 Dec 2010 14:47:52 -0800
From: "Taylor, Ronald C" <ronald.taylor@pnl.gov>
To: 'Patrick Angeles' <patrick@cloudera.com>, "'user@hbase.apache.org'"
	<user@hbase.apache.org>, "'general@hadoop.apache.org'"
	<general@hadoop.apache.org>
CC: "Fox, Kevin M" <kevin.fox@pnl.gov>, "Brown, David M JR"
	<david.brown@pnl.gov>, "Taylor, Ronald C" <ronald.taylor@pnl.gov>
Date: Tue, 28 Dec 2010 14:47:51 -0800
Subject: RE: What is the fastest way to get a large amount of data into the
 Hadoop HDFS file system (or Hbase)?
Thread-Topic: What is the fastest way to get a large amount of data into the
 Hadoop HDFS file system (or Hbase)?
Thread-Index: Acum3lXvlgdunCAPQK+rrVcPxbnXKgAAF6jw
Message-ID: <A632B3F93D861843A3C966F3962C3F84010822D50028@EMAIL05.pnl.gov>
References: <590321.36008.qm@web31813.mail.mud.yahoo.com>
	<536061.77419.qm@web130101.mail.mud.yahoo.com>
	<A632B3F93D861843A3C966F3962C3F84010822D5000C@EMAIL05.pnl.gov>
	<4D18AF52.9080800@mozilla.com>
	<A632B3F93D861843A3C966F3962C3F84010822D50026@EMAIL05.pnl.gov>
 <AANLkTimxVqwnJDbq9pi7weR2KTAijWpUX1+_xqvAemTw@mail.gmail.com>
In-Reply-To: <AANLkTimxVqwnJDbq9pi7weR2KTAijWpUX1+_xqvAemTw@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_A632B3F93D861843A3C966F3962C3F84010822D50028EMAIL05pnlg_"
MIME-Version: 1.0

--_000_A632B3F93D861843A3C966F3962C3F84010822D50028EMAIL05pnlg_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Patrick,

Thanks for the info (and quick reply). I want to make sure I understand: Pr=
esuming that the data files are coming off a set of disk drives attached to=
 a single Linux file server, you say I need two things to optimize the tran=
sfer:

1)  a fat network pipe

2) some way of parallelizing the reads

So - I will check into network hardware, in regard to (1). But for (2), is =
the MapReduce method that I was think of, a way that uses "hadoop fs -copyF=
romLocal" in each Mapper, a good way to go at the destination end? I believ=
e that you were saying that it is indeed OK, but I want to double-check, si=
nce this will be a critical piece of our work flow.

 Ron


________________________________
From: patrickangeles@gmail.com [mailto:patrickangeles@gmail.com] On Behalf =
Of Patrick Angeles
Sent: Tuesday, December 28, 2010 2:27 PM
To: general@hadoop.apache.org
Cc: user@hbase.apache.org; Taylor, Ronald C; Fox, Kevin M; Brown, David M J=
R
Subject: Re: What is the fastest way to get a large amount of data into the=
 Hadoop HDFS file system (or Hbase)?

Ron,

While MapReduce can help to parallelize the load effort, your likely bottle=
neck is the source system (where the files come from). If the files are com=
ing from a single server, then parallelizing the load won't gain you much p=
ast a certain point. You have to figure in how fast you can read the file(s=
) off disk(s) and push the bits through your network and finally onto HDFS.

The best scenario is if you can parallelize the reads and have a fat networ=
k pipe (10GbE or more) going into your Hadoop cluster.

Regards,

- Patrick

On Tue, Dec 28, 2010 at 5:04 PM, Taylor, Ronald C <ronald.taylor@pnl.gov<ma=
ilto:ronald.taylor@pnl.gov>> wrote:

Folks,

We plan on uploading large amounts of data on a regular basis onto a Hadoop=
 cluster, with Hbase operating on top of Hadoop. Figure eventually on the o=
rder of multiple terabytes per week. So - we are concerned about doing the =
uploads themselves as fast as possible from our native Linux file system in=
to HDFS. Figure files will be in, roughly, the 1 to 300 GB range.

Off the top of my head, I'm thinking that doing this in parallel using a Ja=
va MapReduce program would work fastest. So my idea would be to have a file=
 listing all the data files (full paths) to be uploaded, one per line, and =
then use that listing file as input to a MapReduce program.

Each Mapper would then upload one of the data files (using "hadoop fs -copy=
FromLocal <source> <dest>") in parallel with all the other Mappers, with th=
e Mappers operating on all the nodes of the cluster, spreading out the file=
 upload across the nodes.

Does that sound like a wise way to approach this? Are there better methods?=
 Anything else out there for doing automated upload in parallel? We would v=
ery much appreciate advice in this area, since we believe upload speed migh=
t become a bottleneck.

 - Ron Taylor

___________________________________________
Ronald Taylor, Ph.D.
Computational Biology & Bioinformatics Group

Pacific Northwest National Laboratory
902 Battelle Boulevard
P.O. Box 999, Mail Stop J4-33
Richland, WA  99352 USA
Office:  509-372-6568
Email: ronald.taylor@pnl.gov<mailto:ronald.taylor@pnl.gov>




--_000_A632B3F93D861843A3C966F3962C3F84010822D50028EMAIL05pnlg_--

From general-return-2629-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 29 00:05:12 2010
Return-Path: <general-return-2629-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 65448 invoked from network); 29 Dec 2010 00:05:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Dec 2010 00:05:11 -0000
Received: (qmail 16879 invoked by uid 500); 29 Dec 2010 00:05:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16761 invoked by uid 500); 29 Dec 2010 00:05:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16753 invoked by uid 99); 29 Dec 2010 00:05:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Dec 2010 00:05:10 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [192.101.109.31] (HELO emailgw03.pnl.gov) (192.101.109.31)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Dec 2010 00:05:02 +0000
X-IronPort-AV: E=Sophos;i="4.60,242,1291622400"; 
   d="scan'208";a="35767280"
Received: from emailhub01.pnl.gov ([130.20.251.61])
  by emailgw03.pnl.gov with ESMTP/TLS/AES128-SHA; 28 Dec 2010 16:04:42 -0800
Received: from Email05.pnl.gov ([130.20.251.69]) by emailhub01.pnl.gov
 ([130.20.251.61]) with mapi; Tue, 28 Dec 2010 16:04:41 -0800
From: "Taylor, Ronald C" <ronald.taylor@pnl.gov>
To: "Fox, Kevin M" <kevin.fox@pnl.gov>, Patrick Angeles <patrick@cloudera.com>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>,
	"user@hbase.apache.org" <user@hbase.apache.org>, "Brown, David M JR"
	<david.brown@pnl.gov>, "Taylor, Ronald C" <ronald.taylor@pnl.gov>
Date: Tue, 28 Dec 2010 16:04:41 -0800
Subject: RE: What is the fastest way to get a large amount of data into the
 Hadoop HDFS file system (or Hbase)?
Thread-Topic: What is the fastest way to get a large amount of data into the
 Hadoop HDFS file system (or Hbase)?
Thread-Index: Acum6GiB7w6BPDa9Qom2VCQHANgdsgAApZCw
Message-ID: <A632B3F93D861843A3C966F3962C3F84010822D50029@EMAIL05.pnl.gov>
References: <590321.36008.qm@web31813.mail.mud.yahoo.com>
	 <536061.77419.qm@web130101.mail.mud.yahoo.com>
	 <A632B3F93D861843A3C966F3962C3F84010822D5000C@EMAIL05.pnl.gov>
	 <4D18AF52.9080800@mozilla.com>
	 <A632B3F93D861843A3C966F3962C3F84010822D50026@EMAIL05.pnl.gov>
	 <AANLkTimxVqwnJDbq9pi7weR2KTAijWpUX1+_xqvAemTw@mail.gmail.com>
 <1293579543.5455.150.camel@sledge.emsl.pnl.gov>
In-Reply-To: <1293579543.5455.150.camel@sledge.emsl.pnl.gov>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

=20
Hi Kevin,

So - from what Patrick and Ted are saying it sounds like we want the best w=
ay to parallelize a source-based push, rather than doing a parallelized pul=
l through a MapReduce program. And I see that what you ask about below is o=
n parallelizing a push, so we are on the same page.
 Ron

-----Original Message-----
From: Fox, Kevin M=20
Sent: Tuesday, December 28, 2010 3:39 PM
To: Patrick Angeles
Cc: general@hadoop.apache.org; user@hbase.apache.org; Taylor, Ronald C; Bro=
wn, David M JR
Subject: Re: What is the fastest way to get a large amount of data into the=
 Hadoop HDFS file system (or Hbase)?

On Tue, 2010-12-28 at 14:26 -0800, Patrick Angeles wrote:
> Ron,
>=20
>=20
> While MapReduce can help to parallelize the load effort, your likely=20
> bottleneck is the source system (where the files come from). If the=20
> files are coming from a single server, then parallelizing the load=20
> won't gain you much past a certain point. You have to figure in how=20
> fast you can read the file(s) off disk(s) and push the bits through=20
> your network and finally onto HDFS.
>=20
>=20
> The best scenario is if you can parallelize the reads and have a fat=20
> network pipe (10GbE or more) going into your Hadoop cluster.


We have a way to parallelize a push from the archive storage cluster to the=
 hadoop storage cluster.

Is there a way to target a particular storage node with a push into the had=
oop file system? The hadoop cluster nodes are 1gig attached to its core swi=
tch and we have a 10 gig uplink to the core from the storage archive. Say, =
we have 4 nodes in each storage cluster (we have more, just a simplified ex=
ample):

a0 --\                                /-- h0
a1 --+                                +-- h1
a2 --+ (A switch) -10gige- (h switch) +-- h2
a3 --/                                \-- h3

I want to be able to have a0 talk to h0 and not have h0 decide the data bel=
ongs on h3, slowing down a3's ability to write data into h3, greatly reduci=
ng bandwidth.

Thanks,
Kevin

>=20
>=20
> Regards,
>=20
>=20
> - Patrick
>=20
> On Tue, Dec 28, 2010 at 5:04 PM, Taylor, Ronald C=20
> <ronald.taylor@pnl.gov> wrote:
>        =20
>         Folks,
>        =20
>         We plan on uploading large amounts of data on a regular basis
>         onto a Hadoop cluster, with Hbase operating on top of Hadoop.
>         Figure eventually on the order of multiple terabytes per week.
>         So - we are concerned about doing the uploads themselves as
>         fast as possible from our native Linux file system into HDFS.
>         Figure files will be in, roughly, the 1 to 300 GB range.
>        =20
>         Off the top of my head, I'm thinking that doing this in
>         parallel using a Java MapReduce program would work fastest. So
>         my idea would be to have a file listing all the data files
>         (full paths) to be uploaded, one per line, and then use that
>         listing file as input to a MapReduce program.
>        =20
>         Each Mapper would then upload one of the data files (using
>         "hadoop fs -copyFromLocal <source> <dest>") in parallel with
>         all the other Mappers, with the Mappers operating on all the
>         nodes of the cluster, spreading out the file upload across the
>         nodes.
>        =20
>         Does that sound like a wise way to approach this? Are there
>         better methods? Anything else out there for doing automated
>         upload in parallel? We would very much appreciate advice in
>         this area, since we believe upload speed might become a
>         bottleneck.
>        =20
>          - Ron Taylor
>        =20
>         ___________________________________________
>         Ronald Taylor, Ph.D.
>         Computational Biology & Bioinformatics Group
>        =20
>         Pacific Northwest National Laboratory
>         902 Battelle Boulevard
>         P.O. Box 999, Mail Stop J4-33
>         Richland, WA  99352 USA
>         Office:  509-372-6568
>         Email: ronald.taylor@pnl.gov
>        =20
>        =20
>=20
>=20



From general-return-2630-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 29 06:32:37 2010
Return-Path: <general-return-2630-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 8515 invoked from network); 29 Dec 2010 06:32:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Dec 2010 06:32:37 -0000
Received: (qmail 62348 invoked by uid 500); 29 Dec 2010 06:32:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62159 invoked by uid 500); 29 Dec 2010 06:32:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62148 invoked by uid 99); 29 Dec 2010 06:32:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Dec 2010 06:32:34 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Dec 2010 06:32:26 +0000
Received: by fxm2 with SMTP id 2so9550723fxm.35
        for <general@hadoop.apache.org>; Tue, 28 Dec 2010 22:32:06 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:content-type:content-transfer-encoding;
        bh=TXrXepWiglW3ToPJfXdWOYh93FwudsEgfpzQxzdoTDg=;
        b=UVktc0PRtTnO0PdegcpTA1wPyMBGh31Cb2q013iCJiKRryWEqxyhWgQRAnR5Lkda9j
         0IBpzoLEKvYqRB1tM5aJYSnyK+e107NrpGIFSqlZq+SsgMQh9NI5KHfoi7jcOnfX1rzW
         9FG/mOJweaQoyajPA/aR+y13DDl+5UHEFPomI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=uaokLBDy1TFe0G6U8FYhAB7dpNoZUpg8UW6CK4KGbrgyYSHtcfFZcjiA4R9UvrpjY9
         51LoZIH4LvbCFMi/HKGKM7kzxA9F8N/kItqoDu3NgVcVN2Y29DHUCC8j6h4BENm9wOV/
         dM8Kmn/PIahkGiaIbVTTAQGy494vJyaazi0HI=
MIME-Version: 1.0
Received: by 10.223.110.77 with SMTP id m13mr558691fap.86.1293604326129; Tue,
 28 Dec 2010 22:32:06 -0800 (PST)
Sender: saint.ack@gmail.com
Received: by 10.223.83.9 with HTTP; Tue, 28 Dec 2010 22:32:06 -0800 (PST)
In-Reply-To: <AANLkTikPzcsMy8PZ3GPB4VyLcSARPoCEnn+XFhV0i6sW@mail.gmail.com>
References: <AANLkTinHt928uox=NQju5Vey91YfTN28NzTA6wZEK8X8@mail.gmail.com>
	<61840.91458.qm@web65503.mail.ac4.yahoo.com>
	<AANLkTi=zf5r_8b4kPhZPKhXbdnjdkkD5K57-Pso0AXL-@mail.gmail.com>
	<AANLkTikO3ZkgnSQDzZ6XjN1rgCp4LfpEQEnMAUi1EUfH@mail.gmail.com>
	<AANLkTikMnE+vKdsu1R2LZ23V4v8jmiSCgYY9sMXZEZPF@mail.gmail.com>
	<AANLkTimv8mGCL3+q6KS+nc3XCECCxQLSr+QJkD7aDt_V@mail.gmail.com>
	<AANLkTikkK_YWUNaVsH-9Mr7BM_L02HmntAOLtk_XrVyY@mail.gmail.com>
	<AANLkTimcGFMN1qdEFZEcDShrTMCpA8uh2t=m3i6hZZHQ@mail.gmail.com>
	<AANLkTikPzcsMy8PZ3GPB4VyLcSARPoCEnn+XFhV0i6sW@mail.gmail.com>
Date: Tue, 28 Dec 2010 22:32:06 -0800
X-Google-Sender-Auth: a4XvLoqfwEJyiRVsLcpsCMR7fRg
Message-ID: <AANLkTi=TbkJuS=PrMy6HFTfDqcmrowuJ97rDknntrk7-@mail.gmail.com>
Subject: Re: DISCUSSION: Cut a hadoop-0.20.0-append release from the tip of
 branch-0.20-append branch?
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Chris:

What you say seems sensible enough but its not what I want (smile).
My sense is that for HBase, unless the package we host at
hbase.apache.org is called hadoop-0.20.0-append -- so its clear its an
untainted bundle made from the tip of the append branch -- then we're
only going to confuse; we'll be spending our time quelling queries
about the "hbase version" of hadoop.

I'm going to pass on trying to offer an append release bundle off the
branch-0.20-append branch.  For the next HBase (imminent) release,
we'll just keep on with telling folks build their own hadoop from the
append branch or go get CDH3 (The HBase release after that will be
about getting us up on hadoop 0.22).

Thanks all,
St.Ack


On Mon, Dec 27, 2010 at 11:20 AM, Chris Douglas <cdouglas@apache.org> wrote=
:
>> On Fri, Dec 24, 2010 at 11:31 AM, Chris Douglas <cdouglas@apache.org> wr=
ote:
>> Calling it other than Hadoop would only confuse the situation even
>> more; "Trust all your data to fooFS!". =A0It'd also reeks of HDFS 'fork'
>> (HBase is not yet up for taking on such a burden).
>
> Unless I'm missing something, it is a fork. It's a temporary, friendly
> fork, but it's what the HBase project has been using and supporting
> for months. It hasn't had a label assigned to it, but it's a product
> (a feature with a mostly-shared implementation across other forks, at
> any rate).
>
>> I liked my original reading of your suggestion Chris -- even if it was
>> perhaps not what you intended -- where HBase would host
>> hadoop-0.20-append. =A0Thats not on?
>
> Your original reading was what I intended. The obstacle to releasing a
> variant of Hadoop from the HBase project is the name. I'd be surprised
> if TLPs were permitted to release under another project's name, even
> if the other endorsed it. If that assumption is not a real constraint,
> then I agree that there's no point in calling it something else. -C
>

From general-return-2631-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 29 21:17:04 2010
Return-Path: <general-return-2631-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53197 invoked from network); 29 Dec 2010 21:17:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Dec 2010 21:17:03 -0000
Received: (qmail 52145 invoked by uid 500); 29 Dec 2010 21:17:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52024 invoked by uid 500); 29 Dec 2010 21:17:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52016 invoked by uid 99); 29 Dec 2010 21:17:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Dec 2010 21:17:01 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dean.hiller@broadridge.com designates 64.18.2.157 as permitted sender)
Received: from [64.18.2.157] (HELO exprod7og102.obsmtp.com) (64.18.2.157)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Dec 2010 21:16:53 +0000
Received: from source ([167.212.2.180]) (using TLSv1) by exprod7ob102.postini.com ([64.18.6.12]) with SMTP
	ID DSNKTRulKw74Bi5yQmVdEorNYCAdA+YjfArT@postini.com; Wed, 29 Dec 2010 13:16:33 PST
Received: from JOSQHUBA01.jsq.bsg.ad.adp.com (josqhuba01 [149.83.60.74])
	by mail.broadridge.com (8.13.8/8.13.8) with ESMTP id oBTLGQte5414978;
	Wed, 29 Dec 2010 16:16:26 -0500
Received: from DNVREMSA01.jsq.bsg.ad.adp.com ([206.88.42.250]) by JOSQHUBA01.jsq.bsg.ad.adp.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Wed, 29 Dec 2010 16:16:26 -0500
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: RE: What is the fastest way to get a large amount of data into the Hadoop HDFS file system (or Hbase)?
X-MimeOLE: Produced By Microsoft Exchange V6.5
Date: Wed, 29 Dec 2010 14:16:22 -0700
Message-ID: <7CA0D5FE7FA83048893E9230C1E9C0280B8C0AF0@DNVREMSA01.jsq.bsg.ad.adp.com>
In-Reply-To: <A632B3F93D861843A3C966F3962C3F84010822D50029@EMAIL05.pnl.gov>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: What is the fastest way to get a large amount of data into the Hadoop HDFS file system (or Hbase)?
Thread-Index: Acum6GiB7w6BPDa9Qom2VCQHANgdsgAApZCwACycgrA=
References: <590321.36008.qm@web31813.mail.mud.yahoo.com> <536061.77419.qm@web130101.mail.mud.yahoo.com> <A632B3F93D861843A3C966F3962C3F84010822D5000C@EMAIL05.pnl.gov> <4D18AF52.9080800@mozilla.com> <A632B3F93D861843A3C966F3962C3F84010822D50026@EMAIL05.pnl.gov> <AANLkTimxVqwnJDbq9pi7weR2KTAijWpUX1+_xqvAemTw@mail.gmail.com> <1293579543.5455.150.camel@sledge.emsl.pnl.gov> <A632B3F93D861843A3C966F3962C3F84010822D50029@EMAIL05.pnl.gov>
From: "Hiller, Dean  (Contractor)" <dean.hiller@broadridge.com>
To: <general@hadoop.apache.org>, "Fox, Kevin M" <kevin.fox@pnl.gov>,
        "Patrick Angeles" <patrick@cloudera.com>
Cc: <user@hbase.apache.org>, "Brown, David M JR" <david.brown@pnl.gov>
X-OriginalArrivalTime: 29 Dec 2010 21:16:26.0962 (UTC) FILETIME=[A75A7B20:01CBA79D]

I wonder if having linux mount hdfs would help here so as people put the
file on your linux /hdfs directory, it was actually writing to hdfs and
not linux ;) (yeah, you still have that one machine bottle neck as the
files come in unless that can be clustered too somehow).  Just google
mounting hdfs from linux....something that sounds pretty cool that we
may be using later. =20

Later,
Dean

-----Original Message-----
From: Taylor, Ronald C [mailto:ronald.taylor@pnl.gov]=20
Sent: Tuesday, December 28, 2010 5:05 PM
To: Fox, Kevin M; Patrick Angeles
Cc: general@hadoop.apache.org; user@hbase.apache.org; Brown, David M JR;
Taylor, Ronald C
Subject: RE: What is the fastest way to get a large amount of data into
the Hadoop HDFS file system (or Hbase)?

=20
Hi Kevin,

So - from what Patrick and Ted are saying it sounds like we want the
best way to parallelize a source-based push, rather than doing a
parallelized pull through a MapReduce program. And I see that what you
ask about below is on parallelizing a push, so we are on the same page.
 Ron

-----Original Message-----
From: Fox, Kevin M=20
Sent: Tuesday, December 28, 2010 3:39 PM
To: Patrick Angeles
Cc: general@hadoop.apache.org; user@hbase.apache.org; Taylor, Ronald C;
Brown, David M JR
Subject: Re: What is the fastest way to get a large amount of data into
the Hadoop HDFS file system (or Hbase)?

On Tue, 2010-12-28 at 14:26 -0800, Patrick Angeles wrote:
> Ron,
>=20
>=20
> While MapReduce can help to parallelize the load effort, your likely=20
> bottleneck is the source system (where the files come from). If the=20
> files are coming from a single server, then parallelizing the load=20
> won't gain you much past a certain point. You have to figure in how=20
> fast you can read the file(s) off disk(s) and push the bits through=20
> your network and finally onto HDFS.
>=20
>=20
> The best scenario is if you can parallelize the reads and have a fat=20
> network pipe (10GbE or more) going into your Hadoop cluster.


We have a way to parallelize a push from the archive storage cluster to
the hadoop storage cluster.

Is there a way to target a particular storage node with a push into the
hadoop file system? The hadoop cluster nodes are 1gig attached to its
core switch and we have a 10 gig uplink to the core from the storage
archive. Say, we have 4 nodes in each storage cluster (we have more,
just a simplified example):

a0 --\                                /-- h0
a1 --+                                +-- h1
a2 --+ (A switch) -10gige- (h switch) +-- h2
a3 --/                                \-- h3

I want to be able to have a0 talk to h0 and not have h0 decide the data
belongs on h3, slowing down a3's ability to write data into h3, greatly
reducing bandwidth.

Thanks,
Kevin

>=20
>=20
> Regards,
>=20
>=20
> - Patrick
>=20
> On Tue, Dec 28, 2010 at 5:04 PM, Taylor, Ronald C=20
> <ronald.taylor@pnl.gov> wrote:
>        =20
>         Folks,
>        =20
>         We plan on uploading large amounts of data on a regular basis
>         onto a Hadoop cluster, with Hbase operating on top of Hadoop.
>         Figure eventually on the order of multiple terabytes per week.
>         So - we are concerned about doing the uploads themselves as
>         fast as possible from our native Linux file system into HDFS.
>         Figure files will be in, roughly, the 1 to 300 GB range.
>        =20
>         Off the top of my head, I'm thinking that doing this in
>         parallel using a Java MapReduce program would work fastest. So
>         my idea would be to have a file listing all the data files
>         (full paths) to be uploaded, one per line, and then use that
>         listing file as input to a MapReduce program.
>        =20
>         Each Mapper would then upload one of the data files (using
>         "hadoop fs -copyFromLocal <source> <dest>") in parallel with
>         all the other Mappers, with the Mappers operating on all the
>         nodes of the cluster, spreading out the file upload across the
>         nodes.
>        =20
>         Does that sound like a wise way to approach this? Are there
>         better methods? Anything else out there for doing automated
>         upload in parallel? We would very much appreciate advice in
>         this area, since we believe upload speed might become a
>         bottleneck.
>        =20
>          - Ron Taylor
>        =20
>         ___________________________________________
>         Ronald Taylor, Ph.D.
>         Computational Biology & Bioinformatics Group
>        =20
>         Pacific Northwest National Laboratory
>         902 Battelle Boulevard
>         P.O. Box 999, Mail Stop J4-33
>         Richland, WA  99352 USA
>         Office:  509-372-6568
>         Email: ronald.taylor@pnl.gov
>        =20
>        =20
>=20
>=20


This message and any attachments are intended only for the use of the add=
ressee and
may contain information that is privileged and confidential. If the reade=
r of the =

message is not the intended recipient or an authorized representative of =
the
intended recipient, you are hereby notified that any dissemination of thi=
s
communication is strictly prohibited. If you have received this communica=
tion in
error, please notify us immediately by e-mail and delete the message and =
any
attachments from your system.
=0D

From general-return-2632-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 29 21:21:39 2010
Return-Path: <general-return-2632-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54011 invoked from network); 29 Dec 2010 21:21:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Dec 2010 21:21:38 -0000
Received: (qmail 58804 invoked by uid 500); 29 Dec 2010 21:21:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58646 invoked by uid 500); 29 Dec 2010 21:21:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58638 invoked by uid 99); 29 Dec 2010 21:21:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Dec 2010 21:21:37 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [192.101.109.31] (HELO emailgw03.pnl.gov) (192.101.109.31)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Dec 2010 21:21:28 +0000
X-IronPort-AV: E=Sophos;i="4.60,246,1291622400"; 
   d="scan'208";a="35813809"
Received: from emailhub02.pnl.gov ([130.20.251.62])
  by emailgw03.pnl.gov with ESMTP/TLS/AES128-SHA; 29 Dec 2010 13:21:06 -0800
Received: from Email05.pnl.gov ([130.20.251.69]) by emailhub02.pnl.gov
 ([130.20.251.62]) with mapi; Wed, 29 Dec 2010 13:21:05 -0800
From: "Fox, Kevin M" <kevin.fox@pnl.gov>
To: "'Hiller, Dean  (Contractor)'" <dean.hiller@broadridge.com>,
	"general@hadoop.apache.org" <general@hadoop.apache.org>, Patrick Angeles
	<patrick@cloudera.com>
CC: "user@hbase.apache.org" <user@hbase.apache.org>, "Brown, David M JR"
	<david.brown@pnl.gov>
Date: Wed, 29 Dec 2010 13:21:04 -0800
Subject: RE: What is the fastest way to get a large amount of data into the
 Hadoop HDFS file system (or Hbase)?
Thread-Topic: What is the fastest way to get a large amount of data into the
 Hadoop HDFS file system (or Hbase)?
Thread-Index: Acum6GiB7w6BPDa9Qom2VCQHANgdsgAApZCwACycgrAAACopMA==
Message-ID: <81A22C12505EF7419E792D791BC28B3CFC78FB2C0D@EMAIL05.pnl.gov>
References: <590321.36008.qm@web31813.mail.mud.yahoo.com>
 <536061.77419.qm@web130101.mail.mud.yahoo.com>
 <A632B3F93D861843A3C966F3962C3F84010822D5000C@EMAIL05.pnl.gov>
 <4D18AF52.9080800@mozilla.com>
 <A632B3F93D861843A3C966F3962C3F84010822D50026@EMAIL05.pnl.gov>
 <AANLkTimxVqwnJDbq9pi7weR2KTAijWpUX1+_xqvAemTw@mail.gmail.com>
 <1293579543.5455.150.camel@sledge.emsl.pnl.gov>
 <A632B3F93D861843A3C966F3962C3F84010822D50029@EMAIL05.pnl.gov>
 <7CA0D5FE7FA83048893E9230C1E9C0280B8C0AF0@DNVREMSA01.jsq.bsg.ad.adp.com>
In-Reply-To: <7CA0D5FE7FA83048893E9230C1E9C0280B8C0AF0@DNVREMSA01.jsq.bsg.ad.adp.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

http://wiki.apache.org/hadoop/MountableHDFS

Under Known Issues:
2. Writes are approximately 33% slower than the DFSClient. TBD how to optim=
ize this. see: HADOOP-3805  - try using -obig_writes if on a >2.6.26 kernel=
, should perform much better since bigger writes implies less context switc=
hing.

3. Reads are ~20-30% slower even with the read buffering.

Sounds like just pushing it in would be better.

Thanks,
Kevin
-----Original Message-----
From: Hiller, Dean (Contractor) [mailto:dean.hiller@broadridge.com]=20
Sent: Wednesday, December 29, 2010 1:16 PM
To: general@hadoop.apache.org; Fox, Kevin M; Patrick Angeles
Cc: user@hbase.apache.org; Brown, David M JR
Subject: RE: What is the fastest way to get a large amount of data into the=
 Hadoop HDFS file system (or Hbase)?

I wonder if having linux mount hdfs would help here so as people put the
file on your linux /hdfs directory, it was actually writing to hdfs and
not linux ;) (yeah, you still have that one machine bottle neck as the
files come in unless that can be clustered too somehow).  Just google
mounting hdfs from linux....something that sounds pretty cool that we
may be using later. =20

Later,
Dean

-----Original Message-----
From: Taylor, Ronald C [mailto:ronald.taylor@pnl.gov]=20
Sent: Tuesday, December 28, 2010 5:05 PM
To: Fox, Kevin M; Patrick Angeles
Cc: general@hadoop.apache.org; user@hbase.apache.org; Brown, David M JR;
Taylor, Ronald C
Subject: RE: What is the fastest way to get a large amount of data into
the Hadoop HDFS file system (or Hbase)?

=20
Hi Kevin,

So - from what Patrick and Ted are saying it sounds like we want the
best way to parallelize a source-based push, rather than doing a
parallelized pull through a MapReduce program. And I see that what you
ask about below is on parallelizing a push, so we are on the same page.
 Ron

-----Original Message-----
From: Fox, Kevin M=20
Sent: Tuesday, December 28, 2010 3:39 PM
To: Patrick Angeles
Cc: general@hadoop.apache.org; user@hbase.apache.org; Taylor, Ronald C;
Brown, David M JR
Subject: Re: What is the fastest way to get a large amount of data into
the Hadoop HDFS file system (or Hbase)?

On Tue, 2010-12-28 at 14:26 -0800, Patrick Angeles wrote:
> Ron,
>=20
>=20
> While MapReduce can help to parallelize the load effort, your likely=20
> bottleneck is the source system (where the files come from). If the=20
> files are coming from a single server, then parallelizing the load=20
> won't gain you much past a certain point. You have to figure in how=20
> fast you can read the file(s) off disk(s) and push the bits through=20
> your network and finally onto HDFS.
>=20
>=20
> The best scenario is if you can parallelize the reads and have a fat=20
> network pipe (10GbE or more) going into your Hadoop cluster.


We have a way to parallelize a push from the archive storage cluster to
the hadoop storage cluster.

Is there a way to target a particular storage node with a push into the
hadoop file system? The hadoop cluster nodes are 1gig attached to its
core switch and we have a 10 gig uplink to the core from the storage
archive. Say, we have 4 nodes in each storage cluster (we have more,
just a simplified example):

a0 --\                                /-- h0
a1 --+                                +-- h1
a2 --+ (A switch) -10gige- (h switch) +-- h2
a3 --/                                \-- h3

I want to be able to have a0 talk to h0 and not have h0 decide the data
belongs on h3, slowing down a3's ability to write data into h3, greatly
reducing bandwidth.

Thanks,
Kevin

>=20
>=20
> Regards,
>=20
>=20
> - Patrick
>=20
> On Tue, Dec 28, 2010 at 5:04 PM, Taylor, Ronald C=20
> <ronald.taylor@pnl.gov> wrote:
>        =20
>         Folks,
>        =20
>         We plan on uploading large amounts of data on a regular basis
>         onto a Hadoop cluster, with Hbase operating on top of Hadoop.
>         Figure eventually on the order of multiple terabytes per week.
>         So - we are concerned about doing the uploads themselves as
>         fast as possible from our native Linux file system into HDFS.
>         Figure files will be in, roughly, the 1 to 300 GB range.
>        =20
>         Off the top of my head, I'm thinking that doing this in
>         parallel using a Java MapReduce program would work fastest. So
>         my idea would be to have a file listing all the data files
>         (full paths) to be uploaded, one per line, and then use that
>         listing file as input to a MapReduce program.
>        =20
>         Each Mapper would then upload one of the data files (using
>         "hadoop fs -copyFromLocal <source> <dest>") in parallel with
>         all the other Mappers, with the Mappers operating on all the
>         nodes of the cluster, spreading out the file upload across the
>         nodes.
>        =20
>         Does that sound like a wise way to approach this? Are there
>         better methods? Anything else out there for doing automated
>         upload in parallel? We would very much appreciate advice in
>         this area, since we believe upload speed might become a
>         bottleneck.
>        =20
>          - Ron Taylor
>        =20
>         ___________________________________________
>         Ronald Taylor, Ph.D.
>         Computational Biology & Bioinformatics Group
>        =20
>         Pacific Northwest National Laboratory
>         902 Battelle Boulevard
>         P.O. Box 999, Mail Stop J4-33
>         Richland, WA  99352 USA
>         Office:  509-372-6568
>         Email: ronald.taylor@pnl.gov
>        =20
>        =20
>=20
>=20


This message and any attachments are intended only for the use of the addre=
ssee and
may contain information that is privileged and confidential. If the reader =
of the=20
message is not the intended recipient or an authorized representative of th=
e
intended recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communicati=
on in
error, please notify us immediately by e-mail and delete the message and an=
y
attachments from your system.



From general-return-2633-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Dec 29 21:29:43 2010
Return-Path: <general-return-2633-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55385 invoked from network); 29 Dec 2010 21:29:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Dec 2010 21:29:42 -0000
Received: (qmail 65396 invoked by uid 500); 29 Dec 2010 21:29:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65279 invoked by uid 500); 29 Dec 2010 21:29:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65030 invoked by uid 99); 29 Dec 2010 21:29:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Dec 2010 21:29:39 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of dean.hiller@broadridge.com designates 64.18.2.215 as permitted sender)
Received: from [64.18.2.215] (HELO exprod7og114.obsmtp.com) (64.18.2.215)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Dec 2010 21:29:29 +0000
Received: from source ([167.212.2.180]) (using TLSv1) by exprod7ob114.postini.com ([64.18.6.12]) with SMTP
	ID DSNKTRuoDY+Jd64U6no7r74PwrSBNcqZyBey@postini.com; Wed, 29 Dec 2010 13:29:09 PST
Received: from JOSQHUBA01.jsq.bsg.ad.adp.com (josqhuba01 [149.83.60.74])
	by mail.broadridge.com (8.13.8/8.13.8) with ESMTP id oBTLSgJL7721120;
	Wed, 29 Dec 2010 16:28:43 -0500
Received: from DNVREMSA01.jsq.bsg.ad.adp.com ([206.88.42.250]) by JOSQHUBA01.jsq.bsg.ad.adp.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Wed, 29 Dec 2010 16:28:43 -0500
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: RE: What is the fastest way to get a large amount of data into the Hadoop HDFS file system (or Hbase)?
X-MimeOLE: Produced By Microsoft Exchange V6.5
Date: Wed, 29 Dec 2010 14:28:40 -0700
Message-ID: <7CA0D5FE7FA83048893E9230C1E9C0280B8C0B18@DNVREMSA01.jsq.bsg.ad.adp.com>
In-Reply-To: <81A22C12505EF7419E792D791BC28B3CFC78FB2C0D@EMAIL05.pnl.gov>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: What is the fastest way to get a large amount of data into the Hadoop HDFS file system (or Hbase)?
Thread-Index: Acum6GiB7w6BPDa9Qom2VCQHANgdsgAApZCwACycgrAAACopMAAATrjA
References: <590321.36008.qm@web31813.mail.mud.yahoo.com> <536061.77419.qm@web130101.mail.mud.yahoo.com> <A632B3F93D861843A3C966F3962C3F84010822D5000C@EMAIL05.pnl.gov> <4D18AF52.9080800@mozilla.com> <A632B3F93D861843A3C966F3962C3F84010822D50026@EMAIL05.pnl.gov> <AANLkTimxVqwnJDbq9pi7weR2KTAijWpUX1+_xqvAemTw@mail.gmail.com> <1293579543.5455.150.camel@sledge.emsl.pnl.gov> <A632B3F93D861843A3C966F3962C3F84010822D50029@EMAIL05.pnl.gov> <7CA0D5FE7FA83048893E9230C1E9C0280B8C0AF0@DNVREMSA01.jsq.bsg.ad.adp.com> <81A22C12505EF7419E792D791BC28B3CFC78FB2C0D@EMAIL05.pnl.gov>
From: "Hiller, Dean  (Contractor)" <dean.hiller@broadridge.com>
To: "Fox, Kevin M" <kevin.fox@pnl.gov>, <general@hadoop.apache.org>,
        "Patrick Angeles" <patrick@cloudera.com>
Cc: <user@hbase.apache.org>, "Brown, David M JR" <david.brown@pnl.gov>
X-OriginalArrivalTime: 29 Dec 2010 21:28:43.0374 (UTC) FILETIME=[5E4A08E0:01CBA79F]
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks of the info, missed that at the bottom of that page.
Dean

-----Original Message-----
From: Fox, Kevin M [mailto:kevin.fox@pnl.gov]=20
Sent: Wednesday, December 29, 2010 2:21 PM
To: Hiller, Dean (Contractor); general@hadoop.apache.org; Patrick
Angeles
Cc: user@hbase.apache.org; Brown, David M JR
Subject: RE: What is the fastest way to get a large amount of data into
the Hadoop HDFS file system (or Hbase)?

http://wiki.apache.org/hadoop/MountableHDFS

Under Known Issues:
2. Writes are approximately 33% slower than the DFSClient. TBD how to
optimize this. see: HADOOP-3805  - try using -obig_writes if on a
>2.6.26 kernel, should perform much better since bigger writes implies
less context switching.

3. Reads are ~20-30% slower even with the read buffering.

Sounds like just pushing it in would be better.

Thanks,
Kevin
-----Original Message-----
From: Hiller, Dean (Contractor) [mailto:dean.hiller@broadridge.com]=20
Sent: Wednesday, December 29, 2010 1:16 PM
To: general@hadoop.apache.org; Fox, Kevin M; Patrick Angeles
Cc: user@hbase.apache.org; Brown, David M JR
Subject: RE: What is the fastest way to get a large amount of data into
the Hadoop HDFS file system (or Hbase)?

I wonder if having linux mount hdfs would help here so as people put the
file on your linux /hdfs directory, it was actually writing to hdfs and
not linux ;) (yeah, you still have that one machine bottle neck as the
files come in unless that can be clustered too somehow).  Just google
mounting hdfs from linux....something that sounds pretty cool that we
may be using later. =20

Later,
Dean

-----Original Message-----
From: Taylor, Ronald C [mailto:ronald.taylor@pnl.gov]=20
Sent: Tuesday, December 28, 2010 5:05 PM
To: Fox, Kevin M; Patrick Angeles
Cc: general@hadoop.apache.org; user@hbase.apache.org; Brown, David M JR;
Taylor, Ronald C
Subject: RE: What is the fastest way to get a large amount of data into
the Hadoop HDFS file system (or Hbase)?

=20
Hi Kevin,

So - from what Patrick and Ted are saying it sounds like we want the
best way to parallelize a source-based push, rather than doing a
parallelized pull through a MapReduce program. And I see that what you
ask about below is on parallelizing a push, so we are on the same page.
 Ron

-----Original Message-----
From: Fox, Kevin M=20
Sent: Tuesday, December 28, 2010 3:39 PM
To: Patrick Angeles
Cc: general@hadoop.apache.org; user@hbase.apache.org; Taylor, Ronald C;
Brown, David M JR
Subject: Re: What is the fastest way to get a large amount of data into
the Hadoop HDFS file system (or Hbase)?

On Tue, 2010-12-28 at 14:26 -0800, Patrick Angeles wrote:
> Ron,
>=20
>=20
> While MapReduce can help to parallelize the load effort, your likely=20
> bottleneck is the source system (where the files come from). If the=20
> files are coming from a single server, then parallelizing the load=20
> won't gain you much past a certain point. You have to figure in how=20
> fast you can read the file(s) off disk(s) and push the bits through=20
> your network and finally onto HDFS.
>=20
>=20
> The best scenario is if you can parallelize the reads and have a fat=20
> network pipe (10GbE or more) going into your Hadoop cluster.


We have a way to parallelize a push from the archive storage cluster to
the hadoop storage cluster.

Is there a way to target a particular storage node with a push into the
hadoop file system? The hadoop cluster nodes are 1gig attached to its
core switch and we have a 10 gig uplink to the core from the storage
archive. Say, we have 4 nodes in each storage cluster (we have more,
just a simplified example):

a0 --\                                /-- h0
a1 --+                                +-- h1
a2 --+ (A switch) -10gige- (h switch) +-- h2
a3 --/                                \-- h3

I want to be able to have a0 talk to h0 and not have h0 decide the data
belongs on h3, slowing down a3's ability to write data into h3, greatly
reducing bandwidth.

Thanks,
Kevin

>=20
>=20
> Regards,
>=20
>=20
> - Patrick
>=20
> On Tue, Dec 28, 2010 at 5:04 PM, Taylor, Ronald C=20
> <ronald.taylor@pnl.gov> wrote:
>        =20
>         Folks,
>        =20
>         We plan on uploading large amounts of data on a regular basis
>         onto a Hadoop cluster, with Hbase operating on top of Hadoop.
>         Figure eventually on the order of multiple terabytes per week.
>         So - we are concerned about doing the uploads themselves as
>         fast as possible from our native Linux file system into HDFS.
>         Figure files will be in, roughly, the 1 to 300 GB range.
>        =20
>         Off the top of my head, I'm thinking that doing this in
>         parallel using a Java MapReduce program would work fastest. So
>         my idea would be to have a file listing all the data files
>         (full paths) to be uploaded, one per line, and then use that
>         listing file as input to a MapReduce program.
>        =20
>         Each Mapper would then upload one of the data files (using
>         "hadoop fs -copyFromLocal <source> <dest>") in parallel with
>         all the other Mappers, with the Mappers operating on all the
>         nodes of the cluster, spreading out the file upload across the
>         nodes.
>        =20
>         Does that sound like a wise way to approach this? Are there
>         better methods? Anything else out there for doing automated
>         upload in parallel? We would very much appreciate advice in
>         this area, since we believe upload speed might become a
>         bottleneck.
>        =20
>          - Ron Taylor
>        =20
>         ___________________________________________
>         Ronald Taylor, Ph.D.
>         Computational Biology & Bioinformatics Group
>        =20
>         Pacific Northwest National Laboratory
>         902 Battelle Boulevard
>         P.O. Box 999, Mail Stop J4-33
>         Richland, WA  99352 USA
>         Office:  509-372-6568
>         Email: ronald.taylor@pnl.gov
>        =20
>        =20
>=20
>=20


This message and any attachments are intended only for the use of the
addressee and
may contain information that is privileged and confidential. If the
reader of the=20
message is not the intended recipient or an authorized representative of
the
intended recipient, you are hereby notified that any dissemination of
this
communication is strictly prohibited. If you have received this
communication in
error, please notify us immediately by e-mail and delete the message and
any
attachments from your system.


This message and any attachments are intended only for the use of the add=
ressee and
may contain information that is privileged and confidential. If the reade=
r of the =

message is not the intended recipient or an authorized representative of =
the
intended recipient, you are hereby notified that any dissemination of thi=
s
communication is strictly prohibited. If you have received this communica=
tion in
error, please notify us immediately by e-mail and delete the message and =
any
attachments from your system.
=0D

From general-return-2643-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jan 01 17:12:33 2011
Return-Path: <general-return-2643-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17725 invoked from network); 1 Jan 2011 17:12:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jan 2011 17:12:33 -0000
Received: (qmail 70594 invoked by uid 500); 31 Dec 2010 16:02:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70383 invoked by uid 500); 31 Dec 2010 16:02:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70374 invoked by uid 99); 31 Dec 2010 16:02:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Dec 2010 16:02:31 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of magawake@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Dec 2010 16:02:24 +0000
Received: by qwh6 with SMTP id 6so12533372qwh.35
        for <general@hadoop.apache.org>; Fri, 31 Dec 2010 08:02:03 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=u+SpjDaHlCf7iaP7RCk1QqaBDHI8bkpWDDZm0MyvrAo=;
        b=lOvxgEF/mgyK0yrlFl2oz5OSZbhmwHvuOXqBnQyGrtqgCfk/4+DUX6MJv4lVsMUeMW
         prNUh1myscVjI2/gJh1ODnDgJoiZ0L6+EhbGDQ1sVtBEX8Qm3uYwgVPmogAVVL/0ESyP
         JZ99Lgq32kNROlpKggjXggzaiUi28TiINTVSk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=WQPRt5rVGHBPFUE67nwJ4BxlTqPVMPKVNO6Mhv0EUtUBGhsyS6SqXaVRzJ0c2pHyx2
         Aj3Z+5UG+PeEAWW1ZbBOi8bi9Y8L/acs53Ji9yKpvbf5sO7x+gEZ3TdGnjO4Eh0EAH4O
         Yr2UphkQ35YSbrgXK1Z13zLxeymn1ncLQo8ZM=
MIME-Version: 1.0
Received: by 10.224.67.72 with SMTP id q8mr3987630qai.222.1293811323718; Fri,
 31 Dec 2010 08:02:03 -0800 (PST)
Received: by 10.220.118.20 with HTTP; Fri, 31 Dec 2010 08:02:03 -0800 (PST)
Date: Fri, 31 Dec 2010 11:02:03 -0500
Message-ID: <AANLkTi=cyTOYxSbvn_YqRTuHPW0b74=EVWhky+oqxSGM@mail.gmail.com>
Subject: dfs.datanode.du.pct question
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8

On some of my data nodes I have 2 filesystems. /fs1 and /fs2. Is it
possible to set the du pct so, for fs1 its 80% usage and /fs2 its 20%
usage?


TIA

From general-return-2644-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jan 01 17:52:36 2011
Return-Path: <general-return-2644-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 62241 invoked from network); 1 Jan 2011 17:52:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jan 2011 17:52:36 -0000
Received: (qmail 682 invoked by uid 500); 31 Dec 2010 22:57:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 606 invoked by uid 500); 31 Dec 2010 22:57:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 598 invoked by uid 99); 31 Dec 2010 22:57:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Dec 2010 22:57:14 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Dec 2010 22:57:09 +0000
Received: by fxm2 with SMTP id 2so11484191fxm.35
        for <general@hadoop.apache.org>; Fri, 31 Dec 2010 14:56:48 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type;
        bh=8BUbopKnV40GXAIwaI3WB5FfjKlPYp6G4el82bwi8mI=;
        b=F9O6HQBDjsYUnZELe+X5va/0Jz7VjvF3vLC7vMEtlubhGdZcTzCuFRFcCi6TxyxmdG
         BojYOqu4fkZtBMUkwpOenw5n25gFCDW561L+y+p5K0lqoZB3InWf3o3Vlw2HbFZzh57W
         +Z8NZ82FA7emEmcUKNgCOsPExbrP9xTE8sYZ8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=gchK1Dr5jkf6ZHwYhW/xKqlbUH1xFx2ZW+DtU02j/kYVhwZ3/QyT9u0fvS8ggkiKNY
         /o5YxaMrbfalo4Cql1oe5ctZEaKCEkUH3QaVRG5w+o3vBuiecchaKhwwo/HcBY/UiPLb
         ov6/vxh5uR9t+PMsv8DXiK1nAxKaxLxaAQvII=
Received: by 10.223.79.68 with SMTP id o4mr646741fak.0.1293836207576; Fri, 31
 Dec 2010 14:56:47 -0800 (PST)
MIME-Version: 1.0
Received: by 10.223.120.14 with HTTP; Fri, 31 Dec 2010 14:56:27 -0800 (PST)
In-Reply-To: <AANLkTi=cyTOYxSbvn_YqRTuHPW0b74=EVWhky+oqxSGM@mail.gmail.com>
References: <AANLkTi=cyTOYxSbvn_YqRTuHPW0b74=EVWhky+oqxSGM@mail.gmail.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Sat, 1 Jan 2011 04:26:27 +0530
Message-ID: <AANLkTikJMzXhYA1wJwxYm=9Zc=PdAtz6-5mOoUQEF5Dh@mail.gmail.com>
Subject: Re: dfs.datanode.du.pct question
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hi,

Isn't this property a legacy one? 0.18 is the last release I see with
this property. I've always used dfs.datanode.du.reserved instead,
reserving constant space across all dfs.data.dir volumes.

On Fri, Dec 31, 2010 at 9:32 PM, Mag Gam <magawake@gmail.com> wrote:
> On some of my data nodes I have 2 filesystems. /fs1 and /fs2. Is it
> possible to set the du pct so, for fs1 its 80% usage and /fs2 its 20%
> usage?

According to the release's code: No.

(Unless two DataNodes are run off the same machine, which is not recommended.)

-- 
Harsh J
www.harshj.com

From general-return-2645-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jan 01 19:15:28 2011
Return-Path: <general-return-2645-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 52838 invoked from network); 1 Jan 2011 19:15:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jan 2011 19:15:28 -0000
Received: (qmail 67099 invoked by uid 500); 1 Jan 2011 05:41:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66784 invoked by uid 500); 1 Jan 2011 05:41:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66772 invoked by uid 99); 1 Jan 2011 05:41:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 01 Jan 2011 05:41:25 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 01 Jan 2011 05:41:18 +0000
Received: by wye20 with SMTP id 20so12785108wye.35
        for <general@hadoop.apache.org>; Fri, 31 Dec 2010 21:40:57 -0800 (PST)
Received: by 10.216.63.15 with SMTP id z15mr18274173wec.74.1293860457139; Fri,
 31 Dec 2010 21:40:57 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.59.13 with HTTP; Fri, 31 Dec 2010 21:40:37 -0800 (PST)
From: Konstantin Boudnik <cos@apache.org>
Date: Fri, 31 Dec 2010 21:40:37 -0800
X-Google-Sender-Auth: DscZFy4B220wNeHv3_I3mBwCucU
Message-ID: <AANLkTi=7R8OnF=vOQ7O5+-280LtPyp4tHSOXpGG2SXWj@mail.gmail.com>
Subject: Happy New Year, Hadoop!
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Best wishes and warmest regards to everyone who was part of
fast-growing Hadoop ecosystem the last year!
2011 promises to be an amazing time for ever increasing Hadoop's
popularity, robustness and success!

Happy New Year everyone!

--
=A0 Take care,
Konstantin (Cos) Boudnik

From general-return-2646-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jan 01 19:22:39 2011
Return-Path: <general-return-2646-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57737 invoked from network); 1 Jan 2011 19:22:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jan 2011 19:22:39 -0000
Received: (qmail 70945 invoked by uid 500); 1 Jan 2011 05:55:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70594 invoked by uid 500); 1 Jan 2011 05:55:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70586 invoked by uid 99); 1 Jan 2011 05:55:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 01 Jan 2011 05:55:56 +0000
X-ASF-Spam-Status: No, hits=1.9 required=10.0
	tests=SPF_HELO_SOFTFAIL,SPF_SOFTFAIL
X-Spam-Check-By: apache.org
Received-SPF: softfail (athena.apache.org: transitioning domain of maheswaran@ikyaglobal.com does not designate 121.243.47.5 as permitted sender)
Received: from [121.243.47.5] (HELO ikyaglobal.com) (121.243.47.5)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 01 Jan 2011 05:55:50 +0000
Received: (qmail 25476 invoked by uid 109); 1 Jan 2011 05:52:30 -0000
Received: from 14.99.3.27 by mail (envelope-from <maheswaran@ikyaglobal.com>, uid 112) with qmail-scanner-1.25st 
 (perlscan: 1.25st. dmf: ???.  
 Clear:RC:1(14.99.3.27):. 
 Processed in 0.236487 secs); 01 Jan 2011 05:52:30 -0000
Received: from static-27.3.99.14.tataidc.co.in (HELO ikyahp530) (maheswaran@[14.99.3.27])
          (envelope-sender <maheswaran@ikyaglobal.com>)
          by ikyaglobal.com (qmail-ldap-1.03) with SMTP
          for <general@hadoop.apache.org>; 1 Jan 2011 05:52:29 -0000
From: "maheswaran" <maheswaran@ikyaglobal.com>
To: <general@hadoop.apache.org>
References: <AANLkTi=7R8OnF=vOQ7O5+-280LtPyp4tHSOXpGG2SXWj@mail.gmail.com>
In-Reply-To: <AANLkTi=7R8OnF=vOQ7O5+-280LtPyp4tHSOXpGG2SXWj@mail.gmail.com>
Subject: RE: Happy New Year, Hadoop!
Date: Sat, 1 Jan 2011 11:25:21 +0530
Message-ID: <!&!AAAAAAAAAAAYAAAAAAAAAI1vVE9i5vZHgXDQJN1+35/CgAAAEAAAAAtxGzR9jXdLg7xlmcLja5cBAAAAAA==@ikyaglobal.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
X-Mailer: Microsoft Office Outlook 12.0
Thread-Index: AcupdiV3c8N/e7EnRTCAQ8/+j2JmagAAjGvQ
Content-Language: en-us

Thanks for the wishes and wish you the same for the entire HADOOP =
Family.


Regards,

Maheswaran A
+91 98403 48330

-----Original Message-----
From: cos@boudnik.org [mailto:cos@boudnik.org] On Behalf Of Konstantin
Boudnik
Sent: Saturday, January 01, 2011 11:11 AM
To: general@hadoop.apache.org
Subject: Happy New Year, Hadoop!

Best wishes and warmest regards to everyone who was part of
fast-growing Hadoop ecosystem the last year!
2011 promises to be an amazing time for ever increasing Hadoop's
popularity, robustness and success!

Happy New Year everyone!

--
=A0 Take care,
Konstantin (Cos) Boudnik


From general-return-2642-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jan 01 20:27:48 2011
Return-Path: <general-return-2642-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96661 invoked from network); 1 Jan 2011 20:27:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jan 2011 20:27:48 -0000
Received: (qmail 32654 invoked by uid 500); 31 Dec 2010 12:09:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32463 invoked by uid 500); 31 Dec 2010 12:09:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32443 invoked by uid 99); 31 Dec 2010 12:09:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Dec 2010 12:09:45 +0000
X-ASF-Spam-Status: No, hits=-0.1 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of chenqibj@cn.ibm.com designates 202.81.31.148 as permitted sender)
Received: from [202.81.31.148] (HELO e23smtp06.au.ibm.com) (202.81.31.148)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Dec 2010 12:09:36 +0000
Received: from d23relay05.au.ibm.com (d23relay05.au.ibm.com [202.81.31.247])
	by e23smtp06.au.ibm.com (8.14.4/8.13.1) with ESMTP id oBVC8wnT005664
	for <general@hadoop.apache.org>; Fri, 31 Dec 2010 23:08:58 +1100
Received: from d23av03.au.ibm.com (d23av03.au.ibm.com [9.190.234.97])
	by d23relay05.au.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id oBVC9EEE1802330
	for <general@hadoop.apache.org>; Fri, 31 Dec 2010 23:09:15 +1100
Received: from d23av03.au.ibm.com (loopback [127.0.0.1])
	by d23av03.au.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id oBVC9ERb025599
	for <general@hadoop.apache.org>; Fri, 31 Dec 2010 23:09:14 +1100
Received: from d23m0037.cn.ibm.com (d23m0037.cn.ibm.com [9.181.2.105])
	by d23av03.au.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id oBVC9DF2025591
	for <general@hadoop.apache.org>; Fri, 31 Dec 2010 23:09:14 +1100
Subject: Keith is  out of office 
Auto-Submitted: auto-generated
From: Qi BJ Chen <chenqibj@cn.ibm.com>
To: general@hadoop.apache.org
Message-ID: <OF97AEABC3.5FC67390-ON4825780A.0042C31C-4825780A.0042C31C@cn.ibm.com>
Date: Fri, 31 Dec 2010 20:09:13 +0800
X-MIMETrack: Serialize by Router on D23M0037/23/M/IBM(Release 8.5.1FP4|July 25, 2010) at
 12/31/2010 20:09:13
MIME-Version: 1.0
Content-type: multipart/alternative; 
	Boundary="0__=C7BBF299DFD1458C8f9e8a93df938690918cC7BBF299DFD1458C"
Content-Disposition: inline

--0__=C7BBF299DFD1458C8f9e8a93df938690918cC7BBF299DFD1458C
Content-type: text/plain; charset=US-ASCII


I will be out of the office starting  2010-12-31 and will not return until
2011-01-06.

--0__=C7BBF299DFD1458C8f9e8a93df938690918cC7BBF299DFD1458C--


From general-return-2634-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jan 01 20:42:25 2011
Return-Path: <general-return-2634-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 492 invoked from network); 1 Jan 2011 20:42:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jan 2011 20:42:25 -0000
Received: (qmail 28565 invoked by uid 500); 31 Dec 2010 08:42:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28213 invoked by uid 500); 31 Dec 2010 08:42:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 63214 invoked by uid 99); 28 Dec 2010 23:08:19 -0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
X-Originating-IP: [67.160.196.149]
In-Reply-To: <A632B3F93D861843A3C966F3962C3F84010822D50028@EMAIL05.pnl.gov>
References: <590321.36008.qm@web31813.mail.mud.yahoo.com> <536061.77419.qm@web130101.mail.mud.yahoo.com>
 <A632B3F93D861843A3C966F3962C3F84010822D5000C@EMAIL05.pnl.gov>
 <4D18AF52.9080800@mozilla.com> <A632B3F93D861843A3C966F3962C3F84010822D50026@EMAIL05.pnl.gov>
 <AANLkTimxVqwnJDbq9pi7weR2KTAijWpUX1+_xqvAemTw@mail.gmail.com> <A632B3F93D861843A3C966F3962C3F84010822D50028@EMAIL05.pnl.gov>
From: Ted Dunning <tdunning@maprtech.com>
Date: Tue, 28 Dec 2010 15:07:31 -0800
Message-ID: <AANLkTimpOP+jormE+yV7ZZAm3hNDDh-o_Cfum1jXSWPZ@mail.gmail.com>
Subject: Re: What is the fastest way to get a large amount of data into the
 Hadoop HDFS file system (or Hbase)?
To: user@hbase.apache.org
Cc: Patrick Angeles <patrick@cloudera.com>, 
	"general@hadoop.apache.org" <general@hadoop.apache.org>, "Fox, Kevin M" <kevin.fox@pnl.gov>, 
	"Brown, David M JR" <david.brown@pnl.gov>, "Taylor, Ronald C" <ronald.taylor@pnl.gov>
Content-Type: multipart/alternative; boundary=90e6ba4fbd9c98ea48049880857d

--90e6ba4fbd9c98ea48049880857d
Content-Type: text/plain; charset=ISO-8859-1

if the data is coming off of a single machine then simply running multiple
threads on that machine spraying the data into the cluster is likely to be
faster than a map-reduce program.  The reason is that you can run the
spraying process continuously and can tune it to carefully saturate your
outbound link toward the cluster.  With a map-reduce program it will be very
easy to flatten the link.

Another issue is that it is easy to push data to the cluster from a local
disk rather than to pull it from nodes in the cluster because most network
file protocols aren't as efficient as you might like.

On Tue, Dec 28, 2010 at 2:47 PM, Taylor, Ronald C <ronald.taylor@pnl.gov>wrote:

> 2) some way of parallelizing the reads
>
> So - I will check into network hardware, in regard to (1). But for (2), is
> the MapReduce method that I was think of, a way that uses "hadoop fs
> -copyFromLocal" in each Mapper, a good way to go at the destination end? I
> believe that you were saying that it is indeed OK, but I want to
> double-check, since this will be a critical piece of our work flow.
>

--90e6ba4fbd9c98ea48049880857d--

From general-return-2635-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jan 01 20:54:30 2011
Return-Path: <general-return-2635-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 2499 invoked from network); 1 Jan 2011 20:54:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jan 2011 20:54:29 -0000
Received: (qmail 29674 invoked by uid 500); 31 Dec 2010 08:42:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29593 invoked by uid 500); 31 Dec 2010 08:42:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 75472 invoked by uid 99); 28 Dec 2010 23:16:03 -0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <AANLkTimpOP+jormE+yV7ZZAm3hNDDh-o_Cfum1jXSWPZ@mail.gmail.com>
References: <590321.36008.qm@web31813.mail.mud.yahoo.com>
	<536061.77419.qm@web130101.mail.mud.yahoo.com>
	<A632B3F93D861843A3C966F3962C3F84010822D5000C@EMAIL05.pnl.gov>
	<4D18AF52.9080800@mozilla.com>
	<A632B3F93D861843A3C966F3962C3F84010822D50026@EMAIL05.pnl.gov>
	<AANLkTimxVqwnJDbq9pi7weR2KTAijWpUX1+_xqvAemTw@mail.gmail.com>
	<A632B3F93D861843A3C966F3962C3F84010822D50028@EMAIL05.pnl.gov>
	<AANLkTimpOP+jormE+yV7ZZAm3hNDDh-o_Cfum1jXSWPZ@mail.gmail.com>
Date: Tue, 28 Dec 2010 18:15:37 -0500
Message-ID: <AANLkTinKw6wLmT4VY7OWnUf=r99B8Qa=tHCZ8juiai2w@mail.gmail.com>
Subject: Re: What is the fastest way to get a large amount of data into the
 Hadoop HDFS file system (or Hbase)?
From: Patrick Angeles <patrick@cloudera.com>
To: Ted Dunning <tdunning@maprtech.com>
Cc: user@hbase.apache.org, 
	"general@hadoop.apache.org" <general@hadoop.apache.org>, "Fox, Kevin M" <kevin.fox@pnl.gov>, 
	"Brown, David M JR" <david.brown@pnl.gov>, "Taylor, Ronald C" <ronald.taylor@pnl.gov>
Content-Type: multipart/alternative; boundary=001636832e8a5d3c57049880a193
X-Virus-Checked: Checked by ClamAV on apache.org

--001636832e8a5d3c57049880a193
Content-Type: text/plain; charset=ISO-8859-1

To add to what Ted says,

"hadoop fs -copyFromLocal" assumes that the file is present locally on the
datanode. That means the file would have to have been transfered to the
node, copied to local disk, and only after that is it written to HDFS, so
there's an extra trip to disk that you could have avoided.

You might have avoided that using a direct pull from the MapReduce task that
writes directly to HDFS. But, as Ted mentions, that is not as efficient as a
source-based push and is also more complex.

On Tue, Dec 28, 2010 at 6:07 PM, Ted Dunning <tdunning@maprtech.com> wrote:

> if the data is coming off of a single machine then simply running multiple
> threads on that machine spraying the data into the cluster is likely to be
> faster than a map-reduce program.  The reason is that you can run the
> spraying process continuously and can tune it to carefully saturate your
> outbound link toward the cluster.  With a map-reduce program it will be very
> easy to flatten the link.
>
> Another issue is that it is easy to push data to the cluster from a local
> disk rather than to pull it from nodes in the cluster because most network
> file protocols aren't as efficient as you might like.
>
>
> On Tue, Dec 28, 2010 at 2:47 PM, Taylor, Ronald C <ronald.taylor@pnl.gov>wrote:
>
>> 2) some way of parallelizing the reads
>>
>> So - I will check into network hardware, in regard to (1). But for (2), is
>> the MapReduce method that I was think of, a way that uses "hadoop fs
>> -copyFromLocal" in each Mapper, a good way to go at the destination end? I
>> believe that you were saying that it is indeed OK, but I want to
>> double-check, since this will be a critical piece of our work flow.
>>
>
>

--001636832e8a5d3c57049880a193--

From general-return-2636-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jan 01 20:54:34 2011
Return-Path: <general-return-2636-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 2579 invoked from network); 1 Jan 2011 20:54:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jan 2011 20:54:34 -0000
Received: (qmail 31022 invoked by uid 500); 31 Dec 2010 08:42:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30978 invoked by uid 500); 31 Dec 2010 08:42:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 86456 invoked by uid 99); 28 Dec 2010 23:29:47 -0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
X-IronPort-AV: E=Sophos;i="4.60,241,1291622400"; 
   d="scan'208";a="35765936"
Subject: Re: What is the fastest way to get a large amount of data into the
 Hadoop HDFS file system  (or Hbase)?
From: Kevin Fox <Kevin.Fox@pnl.gov>
To: "Taylor, Ronald C" <ronald.taylor@pnl.gov>
Cc: "'user@hbase.apache.org'" <user@hbase.apache.org>,  "'general@hadoop.apache.org'" <general@hadoop.apache.org>, "Brown, David M JR" <david.brown@pnl.gov>,  kevin.glass@pnl.gov
In-Reply-To: <A632B3F93D861843A3C966F3962C3F84010822D50026@EMAIL05.pnl.gov>
References: <590321.36008.qm@web31813.mail.mud.yahoo.com>
	 <536061.77419.qm@web130101.mail.mud.yahoo.com>
	 <A632B3F93D861843A3C966F3962C3F84010822D5000C@EMAIL05.pnl.gov>
	 <4D18AF52.9080800@mozilla.com>
	 <A632B3F93D861843A3C966F3962C3F84010822D50026@EMAIL05.pnl.gov>
Content-Type: text/plain; charset="UTF-8"
Date: Tue, 28 Dec 2010 15:29:20 -0800
Message-ID: <1293578960.5455.141.camel@sledge.emsl.pnl.gov>
Mime-Version: 1.0
X-Mailer: Evolution 2.28.3 
Content-Transfer-Encoding: 7bit

As I understand it, and please correct me if I'm wrong, a map/reduce job
has an instance of a FileSystem object on either side. One that the data
is read out of on the map side, and one the data is fed into on the
reduce side.

Can't you run the map reduce job on the storage cluster that stores the
archival data, feeding the map side with
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/fs/RawLocalFileSystem.html
from the mounted, posix parallel file system,
and feed it into the hadoop cluster on the reduce side. That would mean,
the network in the middle would only see the reduced data set cross the
wire and you could parallelize the data reduction as close to the
archive as possible.

Thanks,
Kevin

On Tue, 2010-12-28 at 14:04 -0800, Taylor, Ronald C wrote:
> Folks,
> 
> We plan on uploading large amounts of data on a regular basis onto a Hadoop cluster, with Hbase operating on top of Hadoop. Figure eventually on the order of multiple terabytes per week. So - we are concerned about doing the uploads themselves as fast as possible from our native Linux file system into HDFS. Figure files will be in, roughly, the 1 to 300 GB range. 
> 
> Off the top of my head, I'm thinking that doing this in parallel using a Java MapReduce program would work fastest. So my idea would be to have a file listing all the data files (full paths) to be uploaded, one per line, and then use that listing file as input to a MapReduce program. 
> 
> Each Mapper would then upload one of the data files (using "hadoop fs -copyFromLocal <source> <dest>") in parallel with all the other Mappers, with the Mappers operating on all the nodes of the cluster, spreading out the file upload across the nodes.
> 
> Does that sound like a wise way to approach this? Are there better methods? Anything else out there for doing automated upload in parallel? We would very much appreciate advice in this area, since we believe upload speed might become a bottleneck.
> 
>   - Ron Taylor
> 
> ___________________________________________
> Ronald Taylor, Ph.D.
> Computational Biology & Bioinformatics Group
> 
> Pacific Northwest National Laboratory
> 902 Battelle Boulevard
> P.O. Box 999, Mail Stop J4-33
> Richland, WA  99352 USA
> Office:  509-372-6568
> Email: ronald.taylor@pnl.gov
> 
> 



From general-return-2637-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jan 01 20:54:41 2011
Return-Path: <general-return-2637-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 2692 invoked from network); 1 Jan 2011 20:54:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jan 2011 20:54:41 -0000
Received: (qmail 32409 invoked by uid 500); 31 Dec 2010 08:42:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32355 invoked by uid 500); 31 Dec 2010 08:42:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 3081 invoked by uid 99); 28 Dec 2010 23:39:32 -0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
X-IronPort-AV: E=Sophos;i="4.60,241,1291622400"; 
   d="scan'208";a="35766339"
Subject: Re: What is the fastest way to get a large amount of data into the
 Hadoop HDFS file system (or Hbase)?
From: Kevin Fox <Kevin.Fox@pnl.gov>
To: Patrick Angeles <patrick@cloudera.com>
Cc: "general@hadoop.apache.org" <general@hadoop.apache.org>,  "user@hbase.apache.org" <user@hbase.apache.org>, "Taylor, Ronald C" <ronald.taylor@pnl.gov>, "Brown, David M JR" <david.brown@pnl.gov>
In-Reply-To: <AANLkTimxVqwnJDbq9pi7weR2KTAijWpUX1+_xqvAemTw@mail.gmail.com>
References: <590321.36008.qm@web31813.mail.mud.yahoo.com>
	 <536061.77419.qm@web130101.mail.mud.yahoo.com>
	 <A632B3F93D861843A3C966F3962C3F84010822D5000C@EMAIL05.pnl.gov>
	 <4D18AF52.9080800@mozilla.com>
	 <A632B3F93D861843A3C966F3962C3F84010822D50026@EMAIL05.pnl.gov>
	 <AANLkTimxVqwnJDbq9pi7weR2KTAijWpUX1+_xqvAemTw@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Date: Tue, 28 Dec 2010 15:39:03 -0800
Message-ID: <1293579543.5455.150.camel@sledge.emsl.pnl.gov>
Mime-Version: 1.0
X-Mailer: Evolution 2.28.3 
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On Tue, 2010-12-28 at 14:26 -0800, Patrick Angeles wrote:
> Ron,
> 
> 
> While MapReduce can help to parallelize the load effort, your likely
> bottleneck is the source system (where the files come from). If the
> files are coming from a single server, then parallelizing the load
> won't gain you much past a certain point. You have to figure in how
> fast you can read the file(s) off disk(s) and push the bits through
> your network and finally onto HDFS.
> 
> 
> The best scenario is if you can parallelize the reads and have a fat
> network pipe (10GbE or more) going into your Hadoop cluster.


We have a way to parallelize a push from the archive storage cluster to
the hadoop storage cluster.

Is there a way to target a particular storage node with a push into the
hadoop file system? The hadoop cluster nodes are 1gig attached to its
core switch and we have a 10 gig uplink to the core from the storage
archive. Say, we have 4 nodes in each storage cluster (we have more,
just a simplified example):

a0 --\                                /-- h0
a1 --+                                +-- h1
a2 --+ (A switch) -10gige- (h switch) +-- h2
a3 --/                                \-- h3

I want to be able to have a0 talk to h0 and not have h0 decide the data
belongs on h3, slowing down a3's ability to write data into h3, greatly
reducing bandwidth.

Thanks,
Kevin

> 
> 
> Regards,
> 
> 
> - Patrick
> 
> On Tue, Dec 28, 2010 at 5:04 PM, Taylor, Ronald C
> <ronald.taylor@pnl.gov> wrote:
>         
>         Folks,
>         
>         We plan on uploading large amounts of data on a regular basis
>         onto a Hadoop cluster, with Hbase operating on top of Hadoop.
>         Figure eventually on the order of multiple terabytes per week.
>         So - we are concerned about doing the uploads themselves as
>         fast as possible from our native Linux file system into HDFS.
>         Figure files will be in, roughly, the 1 to 300 GB range.
>         
>         Off the top of my head, I'm thinking that doing this in
>         parallel using a Java MapReduce program would work fastest. So
>         my idea would be to have a file listing all the data files
>         (full paths) to be uploaded, one per line, and then use that
>         listing file as input to a MapReduce program.
>         
>         Each Mapper would then upload one of the data files (using
>         "hadoop fs -copyFromLocal <source> <dest>") in parallel with
>         all the other Mappers, with the Mappers operating on all the
>         nodes of the cluster, spreading out the file upload across the
>         nodes.
>         
>         Does that sound like a wise way to approach this? Are there
>         better methods? Anything else out there for doing automated
>         upload in parallel? We would very much appreciate advice in
>         this area, since we believe upload speed might become a
>         bottleneck.
>         
>          - Ron Taylor
>         
>         ___________________________________________
>         Ronald Taylor, Ph.D.
>         Computational Biology & Bioinformatics Group
>         
>         Pacific Northwest National Laboratory
>         902 Battelle Boulevard
>         P.O. Box 999, Mail Stop J4-33
>         Richland, WA  99352 USA
>         Office:  509-372-6568
>         Email: ronald.taylor@pnl.gov
>         
>         
> 
> 



From general-return-2638-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jan 01 20:54:51 2011
Return-Path: <general-return-2638-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 2829 invoked from network); 1 Jan 2011 20:54:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jan 2011 20:54:51 -0000
Received: (qmail 33808 invoked by uid 500); 31 Dec 2010 08:42:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33764 invoked by uid 500); 31 Dec 2010 08:42:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 19526 invoked by uid 99); 29 Dec 2010 00:06:07 -0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
X-IronPort-AV: E=Sophos;i="4.60,242,1291622400"; 
   d="scan'208";a="35767312"
Subject: Re: What is the fastest way to get a large amount of data into the
 Hadoop HDFS file system  (or Hbase)?
From: Kevin Fox <Kevin.Fox@pnl.gov>
To: "'general@hadoop.apache.org'" <general@hadoop.apache.org>
In-Reply-To: <A632B3F93D861843A3C966F3962C3F84010822D50026@EMAIL05.pnl.gov>
References: <590321.36008.qm@web31813.mail.mud.yahoo.com>
	 <536061.77419.qm@web130101.mail.mud.yahoo.com>
	 <A632B3F93D861843A3C966F3962C3F84010822D5000C@EMAIL05.pnl.gov>
	 <4D18AF52.9080800@mozilla.com>
	 <A632B3F93D861843A3C966F3962C3F84010822D50026@EMAIL05.pnl.gov>
Content-Type: text/plain; charset="UTF-8"
Date: Tue, 28 Dec 2010 16:05:36 -0800
Message-ID: <1293581136.5455.156.camel@sledge.emsl.pnl.gov>
Mime-Version: 1.0
X-Mailer: Evolution 2.28.3 
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

As I understand it, and please correct me if I'm wrong, a map/reduce job
has an instance of a FileSystem object on either side. One that the data
is read out of on the map side, and one the data is fed into on the
reduce side.

Can't you run the map reduce job on the storage cluster that stores the
archival data, feeding the map side with
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/fs/RawLocalFileSystem.html
from the mounted, posix parallel file system,
and feed it into the hadoop cluster on the reduce side. That would mean,
the network in the middle would only see the reduced data set cross the
wire and you could parallelize the data reduction as close to the
archive as possible.

Thanks,
Kevin

On Tue, 2010-12-28 at 14:04 -0800, Taylor, Ronald C wrote:
> Folks,
> 
> We plan on uploading large amounts of data on a regular basis onto a Hadoop cluster, with Hbase operating on top of Hadoop. Figure eventually on the order of multiple terabytes per week. So - we are concerned about doing the uploads themselves as fast as possible from our native Linux file system into HDFS. Figure files will be in, roughly, the 1 to 300 GB range. 
> 
> Off the top of my head, I'm thinking that doing this in parallel using a Java MapReduce program would work fastest. So my idea would be to have a file listing all the data files (full paths) to be uploaded, one per line, and then use that listing file as input to a MapReduce program. 
> 
> Each Mapper would then upload one of the data files (using "hadoop fs -copyFromLocal <source> <dest>") in parallel with all the other Mappers, with the Mappers operating on all the nodes of the cluster, spreading out the file upload across the nodes.
> 
> Does that sound like a wise way to approach this? Are there better methods? Anything else out there for doing automated upload in parallel? We would very much appreciate advice in this area, since we believe upload speed might become a bottleneck.
> 
>   - Ron Taylor
> 
> ___________________________________________
> Ronald Taylor, Ph.D.
> Computational Biology & Bioinformatics Group
> 
> Pacific Northwest National Laboratory
> 902 Battelle Boulevard
> P.O. Box 999, Mail Stop J4-33
> Richland, WA  99352 USA
> Office:  509-372-6568
> Email: ronald.taylor@pnl.gov
> 
> 



From general-return-2639-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jan 01 20:55:10 2011
Return-Path: <general-return-2639-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 2963 invoked from network); 1 Jan 2011 20:55:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jan 2011 20:55:09 -0000
Received: (qmail 35662 invoked by uid 500); 31 Dec 2010 08:43:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 35604 invoked by uid 500); 31 Dec 2010 08:43:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 19672 invoked by uid 99); 29 Dec 2010 00:06:13 -0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
X-IronPort-AV: E=Sophos;i="4.60,242,1291622400"; 
   d="scan'208";a="54803483"
Subject: Re: What is the fastest way to get a large amount of data into the
 Hadoop HDFS file system (or Hbase)?
From: Kevin Fox <Kevin.Fox@pnl.gov>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <AANLkTimxVqwnJDbq9pi7weR2KTAijWpUX1+_xqvAemTw@mail.gmail.com>
References: <590321.36008.qm@web31813.mail.mud.yahoo.com>
	 <536061.77419.qm@web130101.mail.mud.yahoo.com>
	 <A632B3F93D861843A3C966F3962C3F84010822D5000C@EMAIL05.pnl.gov>
	 <4D18AF52.9080800@mozilla.com>
	 <A632B3F93D861843A3C966F3962C3F84010822D50026@EMAIL05.pnl.gov>
	 <AANLkTimxVqwnJDbq9pi7weR2KTAijWpUX1+_xqvAemTw@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Date: Tue, 28 Dec 2010 16:05:46 -0800
Message-ID: <1293581146.5455.157.camel@sledge.emsl.pnl.gov>
Mime-Version: 1.0
X-Mailer: Evolution 2.28.3 
Content-Transfer-Encoding: 7bit

On Tue, 2010-12-28 at 14:26 -0800, Patrick Angeles wrote:
> Ron,
> 
> 
> While MapReduce can help to parallelize the load effort, your likely
> bottleneck is the source system (where the files come from). If the
> files are coming from a single server, then parallelizing the load
> won't gain you much past a certain point. You have to figure in how
> fast you can read the file(s) off disk(s) and push the bits through
> your network and finally onto HDFS.
> 
> 
> The best scenario is if you can parallelize the reads and have a fat
> network pipe (10GbE or more) going into your Hadoop cluster.
> 

We have a way to parallelize a push from the archive storage cluster to
the hadoop storage cluster.

Is there a way to target a particular storage node with a push into the
hadoop file system? The hadoop cluster nodes are 1gig attached to its
core switch and we have a 10 gig uplink to the core from the storage
archive. Say, we have 4 nodes in each storage cluster (we have more,
just a simplified example):

a0 --\                                /-- h0
a1 --+                                +-- h1
a2 --+ (A switch) -10gige- (h switch) +-- h2
a3 --/                                \-- h3

I want to be able to have a0 talk to h0 and not have h0 decide the data
belongs on h3, slowing down a3's ability to write data into h3, greatly
reducing bandwidth.

Thanks,
Kevin

> 
> Regards,
> 
> 
> - Patrick
> 
> On Tue, Dec 28, 2010 at 5:04 PM, Taylor, Ronald C
> <ronald.taylor@pnl.gov> wrote:
>         
>         Folks,
>         
>         We plan on uploading large amounts of data on a regular basis
>         onto a Hadoop cluster, with Hbase operating on top of Hadoop.
>         Figure eventually on the order of multiple terabytes per week.
>         So - we are concerned about doing the uploads themselves as
>         fast as possible from our native Linux file system into HDFS.
>         Figure files will be in, roughly, the 1 to 300 GB range.
>         
>         Off the top of my head, I'm thinking that doing this in
>         parallel using a Java MapReduce program would work fastest. So
>         my idea would be to have a file listing all the data files
>         (full paths) to be uploaded, one per line, and then use that
>         listing file as input to a MapReduce program.
>         
>         Each Mapper would then upload one of the data files (using
>         "hadoop fs -copyFromLocal <source> <dest>") in parallel with
>         all the other Mappers, with the Mappers operating on all the
>         nodes of the cluster, spreading out the file upload across the
>         nodes.
>         
>         Does that sound like a wise way to approach this? Are there
>         better methods? Anything else out there for doing automated
>         upload in parallel? We would very much appreciate advice in
>         this area, since we believe upload speed might become a
>         bottleneck.
>         
>          - Ron Taylor
>         
>         ___________________________________________
>         Ronald Taylor, Ph.D.
>         Computational Biology & Bioinformatics Group
>         
>         Pacific Northwest National Laboratory
>         902 Battelle Boulevard
>         P.O. Box 999, Mail Stop J4-33
>         Richland, WA  99352 USA
>         Office:  509-372-6568
>         Email: ronald.taylor@pnl.gov
>         
>         
> 
> 



From general-return-2640-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jan 01 20:55:11 2011
Return-Path: <general-return-2640-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 3054 invoked from network); 1 Jan 2011 20:55:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jan 2011 20:55:11 -0000
Received: (qmail 37055 invoked by uid 500); 31 Dec 2010 08:43:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36986 invoked by uid 500); 31 Dec 2010 08:43:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 22451 invoked by uid 99); 29 Dec 2010 00:11:58 -0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
X-IronPort-AV: E=Sophos;i="4.60,242,1291622400"; 
   d="scan'208";a="35767460"
Subject: RE: What is the fastest way to get a large amount of data into the
 Hadoop HDFS file system (or Hbase)?
From: Kevin Fox <Kevin.Fox@pnl.gov>
To: "Taylor, Ronald C" <ronald.taylor@pnl.gov>
Cc: Patrick Angeles <patrick@cloudera.com>, "general@hadoop.apache.org" <general@hadoop.apache.org>, "user@hbase.apache.org" <user@hbase.apache.org>,  "Brown, David M JR" <david.brown@pnl.gov>, kevin.glass@pnl.gov
In-Reply-To: <A632B3F93D861843A3C966F3962C3F84010822D50029@EMAIL05.pnl.gov>
References: <590321.36008.qm@web31813.mail.mud.yahoo.com>
	 <536061.77419.qm@web130101.mail.mud.yahoo.com>
	 <A632B3F93D861843A3C966F3962C3F84010822D5000C@EMAIL05.pnl.gov>
	 <4D18AF52.9080800@mozilla.com>
	 <A632B3F93D861843A3C966F3962C3F84010822D50026@EMAIL05.pnl.gov>
	 <AANLkTimxVqwnJDbq9pi7weR2KTAijWpUX1+_xqvAemTw@mail.gmail.com>
	 <1293579543.5455.150.camel@sledge.emsl.pnl.gov>
	 <A632B3F93D861843A3C966F3962C3F84010822D50029@EMAIL05.pnl.gov>
Content-Type: text/plain; charset="UTF-8"
Date: Tue, 28 Dec 2010 16:11:31 -0800
Message-ID: <1293581491.5455.163.camel@sledge.emsl.pnl.gov>
Mime-Version: 1.0
X-Mailer: Evolution 2.28.3 
Content-Transfer-Encoding: 7bit

On Tue, 2010-12-28 at 16:04 -0800, Taylor, Ronald C wrote:
> Hi Kevin,
> 
> So - from what Patrick and Ted are saying it sounds like we want the best way to parallelize a source-based push, rather than doing a parallelized pull through a MapReduce program. And I see that what you ask about below is on parallelizing a push, so we are on the same page.
>  Ron

Hi Ron,

I think there are merits for both approaches, a depending on the type of
data it is. If you only ever need a subset of the data for reprocessing,
map/reducing then transfer might be better. If you want all the raw data
of a particular set, push transfer may make more sense.

So we'd need solutions to both:
*parallel map reduce on non hdfs storage cluster, push to hdfs
reprocessing cluster.
*parallel push data from non hdfs storage cluster, into hdfs
reprocessing cluster.

Thanks,
Kevin

> 
> -----Original Message-----
> From: Fox, Kevin M 
> Sent: Tuesday, December 28, 2010 3:39 PM
> To: Patrick Angeles
> Cc: general@hadoop.apache.org; user@hbase.apache.org; Taylor, Ronald C; Brown, David M JR
> Subject: Re: What is the fastest way to get a large amount of data into the Hadoop HDFS file system (or Hbase)?
> 
> On Tue, 2010-12-28 at 14:26 -0800, Patrick Angeles wrote:
> > Ron,
> > 
> > 
> > While MapReduce can help to parallelize the load effort, your likely 
> > bottleneck is the source system (where the files come from). If the 
> > files are coming from a single server, then parallelizing the load 
> > won't gain you much past a certain point. You have to figure in how 
> > fast you can read the file(s) off disk(s) and push the bits through 
> > your network and finally onto HDFS.
> > 
> > 
> > The best scenario is if you can parallelize the reads and have a fat 
> > network pipe (10GbE or more) going into your Hadoop cluster.
> 
> 
> We have a way to parallelize a push from the archive storage cluster to the hadoop storage cluster.
> 
> Is there a way to target a particular storage node with a push into the hadoop file system? The hadoop cluster nodes are 1gig attached to its core switch and we have a 10 gig uplink to the core from the storage archive. Say, we have 4 nodes in each storage cluster (we have more, just a simplified example):
> 
> a0 --\                                /-- h0
> a1 --+                                +-- h1
> a2 --+ (A switch) -10gige- (h switch) +-- h2
> a3 --/                                \-- h3
> 
> I want to be able to have a0 talk to h0 and not have h0 decide the data belongs on h3, slowing down a3's ability to write data into h3, greatly reducing bandwidth.
> 
> Thanks,
> Kevin
> 
> > 
> > 
> > Regards,
> > 
> > 
> > - Patrick
> > 
> > On Tue, Dec 28, 2010 at 5:04 PM, Taylor, Ronald C 
> > <ronald.taylor@pnl.gov> wrote:
> >         
> >         Folks,
> >         
> >         We plan on uploading large amounts of data on a regular basis
> >         onto a Hadoop cluster, with Hbase operating on top of Hadoop.
> >         Figure eventually on the order of multiple terabytes per week.
> >         So - we are concerned about doing the uploads themselves as
> >         fast as possible from our native Linux file system into HDFS.
> >         Figure files will be in, roughly, the 1 to 300 GB range.
> >         
> >         Off the top of my head, I'm thinking that doing this in
> >         parallel using a Java MapReduce program would work fastest. So
> >         my idea would be to have a file listing all the data files
> >         (full paths) to be uploaded, one per line, and then use that
> >         listing file as input to a MapReduce program.
> >         
> >         Each Mapper would then upload one of the data files (using
> >         "hadoop fs -copyFromLocal <source> <dest>") in parallel with
> >         all the other Mappers, with the Mappers operating on all the
> >         nodes of the cluster, spreading out the file upload across the
> >         nodes.
> >         
> >         Does that sound like a wise way to approach this? Are there
> >         better methods? Anything else out there for doing automated
> >         upload in parallel? We would very much appreciate advice in
> >         this area, since we believe upload speed might become a
> >         bottleneck.
> >         
> >          - Ron Taylor
> >         
> >         ___________________________________________
> >         Ronald Taylor, Ph.D.
> >         Computational Biology & Bioinformatics Group
> >         
> >         Pacific Northwest National Laboratory
> >         902 Battelle Boulevard
> >         P.O. Box 999, Mail Stop J4-33
> >         Richland, WA  99352 USA
> >         Office:  509-372-6568
> >         Email: ronald.taylor@pnl.gov
> >         
> >         
> > 
> > 
> 
> 



From general-return-2641-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jan 01 20:55:29 2011
Return-Path: <general-return-2641-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 3178 invoked from network); 1 Jan 2011 20:55:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jan 2011 20:55:28 -0000
Received: (qmail 38611 invoked by uid 500); 31 Dec 2010 08:43:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38406 invoked by uid 500); 31 Dec 2010 08:43:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 57229 invoked by uid 99); 29 Dec 2010 21:19:46 -0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
X-Originating-IP: [67.160.196.149]
In-Reply-To: <7CA0D5FE7FA83048893E9230C1E9C0280B8C0AF0@DNVREMSA01.jsq.bsg.ad.adp.com>
References: <590321.36008.qm@web31813.mail.mud.yahoo.com> <536061.77419.qm@web130101.mail.mud.yahoo.com>
 <A632B3F93D861843A3C966F3962C3F84010822D5000C@EMAIL05.pnl.gov>
 <4D18AF52.9080800@mozilla.com> <A632B3F93D861843A3C966F3962C3F84010822D50026@EMAIL05.pnl.gov>
 <AANLkTimxVqwnJDbq9pi7weR2KTAijWpUX1+_xqvAemTw@mail.gmail.com>
 <1293579543.5455.150.camel@sledge.emsl.pnl.gov> <A632B3F93D861843A3C966F3962C3F84010822D50029@EMAIL05.pnl.gov>
 <7CA0D5FE7FA83048893E9230C1E9C0280B8C0AF0@DNVREMSA01.jsq.bsg.ad.adp.com>
From: Ted Dunning <tdunning@maprtech.com>
Date: Wed, 29 Dec 2010 13:19:00 -0800
Message-ID: <AANLkTinkSt8Kf1QD97oSoLLe=A0cFTzC6Oi4FKzOcr1Z@mail.gmail.com>
Subject: Re: What is the fastest way to get a large amount of data into the
 Hadoop HDFS file system (or Hbase)?
To: user@hbase.apache.org
Cc: general@hadoop.apache.org, "Fox, Kevin M" <kevin.fox@pnl.gov>, 
	Patrick Angeles <patrick@cloudera.com>, "Brown, David M JR" <david.brown@pnl.gov>
Content-Type: multipart/alternative; boundary=20cf30050ec05687570498931fcd

--20cf30050ec05687570498931fcd
Content-Type: text/plain; charset=ISO-8859-1

The problem there is that HDFS isn't a first class file system.  That means
that the nice and easy ways of mounting will lead to problems (notably NFS
which maintains no state will require random write capabilities).

On Wed, Dec 29, 2010 at 1:16 PM, Hiller, Dean (Contractor) <
dean.hiller@broadridge.com> wrote:

> I wonder if having linux mount hdfs would help here so as people put the
> file on your linux /hdfs directory, it was actually writing to hdfs and
> not linux ;) (yeah, you still have that one machine bottle neck as the
> files come in unless that can be clustered too somehow).  Just google
> mounting hdfs from linux....something that sounds pretty cool that we
> may be using later.
>
> Later,
> Dean
>
> -----Original Message-----
> From: Taylor, Ronald C [mailto:ronald.taylor@pnl.gov]
> Sent: Tuesday, December 28, 2010 5:05 PM
> To: Fox, Kevin M; Patrick Angeles
> Cc: general@hadoop.apache.org; user@hbase.apache.org; Brown, David M JR;
> Taylor, Ronald C
> Subject: RE: What is the fastest way to get a large amount of data into
> the Hadoop HDFS file system (or Hbase)?
>
>
> Hi Kevin,
>
> So - from what Patrick and Ted are saying it sounds like we want the
> best way to parallelize a source-based push, rather than doing a
> parallelized pull through a MapReduce program. And I see that what you
> ask about below is on parallelizing a push, so we are on the same page.
>  Ron
>
> -----Original Message-----
> From: Fox, Kevin M
> Sent: Tuesday, December 28, 2010 3:39 PM
> To: Patrick Angeles
> Cc: general@hadoop.apache.org; user@hbase.apache.org; Taylor, Ronald C;
> Brown, David M JR
> Subject: Re: What is the fastest way to get a large amount of data into
> the Hadoop HDFS file system (or Hbase)?
>
> On Tue, 2010-12-28 at 14:26 -0800, Patrick Angeles wrote:
> > Ron,
> >
> >
> > While MapReduce can help to parallelize the load effort, your likely
> > bottleneck is the source system (where the files come from). If the
> > files are coming from a single server, then parallelizing the load
> > won't gain you much past a certain point. You have to figure in how
> > fast you can read the file(s) off disk(s) and push the bits through
> > your network and finally onto HDFS.
> >
> >
> > The best scenario is if you can parallelize the reads and have a fat
> > network pipe (10GbE or more) going into your Hadoop cluster.
>
>
> We have a way to parallelize a push from the archive storage cluster to
> the hadoop storage cluster.
>
> Is there a way to target a particular storage node with a push into the
> hadoop file system? The hadoop cluster nodes are 1gig attached to its
> core switch and we have a 10 gig uplink to the core from the storage
> archive. Say, we have 4 nodes in each storage cluster (we have more,
> just a simplified example):
>
> a0 --\                                /-- h0
> a1 --+                                +-- h1
> a2 --+ (A switch) -10gige- (h switch) +-- h2
> a3 --/                                \-- h3
>
> I want to be able to have a0 talk to h0 and not have h0 decide the data
> belongs on h3, slowing down a3's ability to write data into h3, greatly
> reducing bandwidth.
>
> Thanks,
> Kevin
>
> >
> >
> > Regards,
> >
> >
> > - Patrick
> >
> > On Tue, Dec 28, 2010 at 5:04 PM, Taylor, Ronald C
> > <ronald.taylor@pnl.gov> wrote:
> >
> >         Folks,
> >
> >         We plan on uploading large amounts of data on a regular basis
> >         onto a Hadoop cluster, with Hbase operating on top of Hadoop.
> >         Figure eventually on the order of multiple terabytes per week.
> >         So - we are concerned about doing the uploads themselves as
> >         fast as possible from our native Linux file system into HDFS.
> >         Figure files will be in, roughly, the 1 to 300 GB range.
> >
> >         Off the top of my head, I'm thinking that doing this in
> >         parallel using a Java MapReduce program would work fastest. So
> >         my idea would be to have a file listing all the data files
> >         (full paths) to be uploaded, one per line, and then use that
> >         listing file as input to a MapReduce program.
> >
> >         Each Mapper would then upload one of the data files (using
> >         "hadoop fs -copyFromLocal <source> <dest>") in parallel with
> >         all the other Mappers, with the Mappers operating on all the
> >         nodes of the cluster, spreading out the file upload across the
> >         nodes.
> >
> >         Does that sound like a wise way to approach this? Are there
> >         better methods? Anything else out there for doing automated
> >         upload in parallel? We would very much appreciate advice in
> >         this area, since we believe upload speed might become a
> >         bottleneck.
> >
> >          - Ron Taylor
> >
> >         ___________________________________________
> >         Ronald Taylor, Ph.D.
> >         Computational Biology & Bioinformatics Group
> >
> >         Pacific Northwest National Laboratory
> >         902 Battelle Boulevard
> >         P.O. Box 999, Mail Stop J4-33
> >         Richland, WA  99352 USA
> >         Office:  509-372-6568
> >         Email: ronald.taylor@pnl.gov
> >
> >
> >
> >
>
>
> This message and any attachments are intended only for the use of the
> addressee and
> may contain information that is privileged and confidential. If the reader
> of the
> message is not the intended recipient or an authorized representative of
> the
> intended recipient, you are hereby notified that any dissemination of this
> communication is strictly prohibited. If you have received this
> communication in
> error, please notify us immediately by e-mail and delete the message and
> any
> attachments from your system.
>
>

--20cf30050ec05687570498931fcd--

From general-return-3213-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 01 08:20:10 2011
Return-Path: <general-return-3213-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 79553 invoked from network); 1 Apr 2011 08:20:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Apr 2011 08:20:09 -0000
Received: (qmail 74928 invoked by uid 500); 1 Apr 2011 08:20:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74160 invoked by uid 500); 1 Apr 2011 08:20:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74152 invoked by uid 99); 1 Apr 2011 08:20:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 08:20:04 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of atm@cloudera.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 08:19:56 +0000
Received: by qwj9 with SMTP id 9so2740412qwj.35
        for <general@hadoop.apache.org>; Fri, 01 Apr 2011 01:19:35 -0700 (PDT)
Received: by 10.229.200.105 with SMTP id ev41mr3139348qcb.0.1301645975157;
 Fri, 01 Apr 2011 01:19:35 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.142.15 with HTTP; Fri, 1 Apr 2011 01:19:05 -0700 (PDT)
From: "Aaron T. Myers" <atm@cloudera.com>
Date: Fri, 1 Apr 2011 01:19:05 -0700
Message-ID: <AANLkTimDH1zNAFNnwP__f0o+BAzVKZ3aagC8YaURTsF+@mail.gmail.com>
Subject: Proposal: Further Project Split(s)
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636ef0437f8fb9f049fd711f2
X-Virus-Checked: Checked by ClamAV on apache.org

--001636ef0437f8fb9f049fd711f2
Content-Type: text/plain; charset=ISO-8859-1

Hello Hadoop Community,

Given the tremendous positive feedback we've all had regarding the HDFS,
MapReduce, and Common project split, I'd like to propose we take the next
step and further separate the existing projects.

I propose we begin by splitting the MapReduce project into separate "Map"
and "Reduce" sub-projects. This will provide us the opportunity to tease out
the complex interdependencies between "map" and "reduce" that exist today,
to encourage us to write more modular and isolated code, which should speed
releases. This will also aid our users who exclusively run map-only or
reduce-only jobs. These are important use-cases, and so should be given high
priority.

Given that these two portions of the existing MapReduce project share a
great deal of code, we will likely need to release these two new projects
concurrently at first, but the eventual goal should certainly be to be able
to release "Map" and "Reduce" independently. This seems intuitive to me,
given the remarkable recent advancements in the academic community regarding
"reduce," while the research coming out of the "map" academics has largely
stagnated of late.

If this proposal is accepted, and it has the success I think it will, then
we should strongly consider splitting the other two projects as well. My gut
instinct is that we should split "HDFS" into "HD" and "FS" sub-projects, and
simply rename the "Common" project to "C'Mon." We can think about the
details of what exactly these project splits mean later.

Please let me know what you think.

Best,
Aaron

--001636ef0437f8fb9f049fd711f2--

From general-return-3214-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 01 08:26:14 2011
Return-Path: <general-return-3214-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1212 invoked from network); 1 Apr 2011 08:26:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Apr 2011 08:26:13 -0000
Received: (qmail 86099 invoked by uid 500); 1 Apr 2011 08:26:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85964 invoked by uid 500); 1 Apr 2011 08:26:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85956 invoked by uid 99); 1 Apr 2011 08:26:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 08:26:10 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 08:26:03 +0000
Received: by iwr19 with SMTP id 19so4736547iwr.35
        for <general@hadoop.apache.org>; Fri, 01 Apr 2011 01:25:42 -0700 (PDT)
Received: by 10.42.75.66 with SMTP id z2mr4679982icj.235.1301646342537; Fri,
 01 Apr 2011 01:25:42 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.42.220.194 with HTTP; Fri, 1 Apr 2011 01:25:22 -0700 (PDT)
In-Reply-To: <AANLkTimDH1zNAFNnwP__f0o+BAzVKZ3aagC8YaURTsF+@mail.gmail.com>
References: <AANLkTimDH1zNAFNnwP__f0o+BAzVKZ3aagC8YaURTsF+@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Fri, 1 Apr 2011 01:25:22 -0700
Message-ID: <AANLkTimz3jcqMHOKURyA8F5siY7gfKbAVS4mNzGV5DG8@mail.gmail.com>
Subject: Re: Proposal: Further Project Split(s)
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba6e84fedec331049fd7279c
X-Virus-Checked: Checked by ClamAV on apache.org

--90e6ba6e84fedec331049fd7279c
Content-Type: text/plain; charset=ISO-8859-1

+4.01. This is a terrific idea.

On Fri, Apr 1, 2011 at 1:19 AM, Aaron T. Myers <atm@cloudera.com> wrote:

> Hello Hadoop Community,
>
> Given the tremendous positive feedback we've all had regarding the HDFS,
> MapReduce, and Common project split, I'd like to propose we take the next
> step and further separate the existing projects.
>
> I propose we begin by splitting the MapReduce project into separate "Map"
> and "Reduce" sub-projects. This will provide us the opportunity to tease
> out
> the complex interdependencies between "map" and "reduce" that exist today,
> to encourage us to write more modular and isolated code, which should speed
> releases. This will also aid our users who exclusively run map-only or
> reduce-only jobs. These are important use-cases, and so should be given
> high
> priority.
>
> Given that these two portions of the existing MapReduce project share a
> great deal of code, we will likely need to release these two new projects
> concurrently at first, but the eventual goal should certainly be to be able
> to release "Map" and "Reduce" independently. This seems intuitive to me,
> given the remarkable recent advancements in the academic community
> regarding
> "reduce," while the research coming out of the "map" academics has largely
> stagnated of late.
>
> If this proposal is accepted, and it has the success I think it will, then
> we should strongly consider splitting the other two projects as well. My
> gut
> instinct is that we should split "HDFS" into "HD" and "FS" sub-projects,
> and
> simply rename the "Common" project to "C'Mon." We can think about the
> details of what exactly these project splits mean later.
>
> Please let me know what you think.
>
> Best,
> Aaron
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--90e6ba6e84fedec331049fd7279c--

From general-return-3215-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 01 08:57:29 2011
Return-Path: <general-return-3215-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93231 invoked from network); 1 Apr 2011 08:57:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Apr 2011 08:57:29 -0000
Received: (qmail 22077 invoked by uid 500); 1 Apr 2011 08:57:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21999 invoked by uid 500); 1 Apr 2011 08:57:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21991 invoked by uid 99); 1 Apr 2011 08:57:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 08:57:25 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 01 Apr 2011 08:57:23 +0000
Received: (qmail 93170 invoked by uid 99); 1 Apr 2011 08:57:01 -0000
Received: from localhost.apache.org (HELO mail-iy0-f176.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 08:57:01 +0000
Received: by iym1 with SMTP id 1so4755372iym.35
        for <general@hadoop.apache.org>; Fri, 01 Apr 2011 01:57:00 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.150.205 with SMTP id z13mr3614888ibv.177.1301648220710;
 Fri, 01 Apr 2011 01:57:00 -0700 (PDT)
Received: by 10.231.59.135 with HTTP; Fri, 1 Apr 2011 01:57:00 -0700 (PDT)
In-Reply-To: <AANLkTimDH1zNAFNnwP__f0o+BAzVKZ3aagC8YaURTsF+@mail.gmail.com>
References: <AANLkTimDH1zNAFNnwP__f0o+BAzVKZ3aagC8YaURTsF+@mail.gmail.com>
Date: Fri, 1 Apr 2011 01:57:00 -0700
Message-ID: <AANLkTinBRgowHV2HNdkM1CHG+jF39B88pcY=3wQpFC=j@mail.gmail.com>
Subject: Re: Proposal: Further Project Split(s)
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Experience developing Hadoop has shown that we not only need to
partition our projects for more active releases, but we also should
explore speculative project splits. For this, a Hadoop.next() project
should track the development of a project scheduler that can partition
the Hadoop subprojects, possibly running a second version of a
subproject in parallel. Downstream subprojects and TLPs automatically
accept whichever releases first as a dependency. Implementation should
combine ant, ivy, maven, and at least one legacy Hadoop build tool (to
be written).

Of course, not all of these subprojects will succeed. When one fails
(or is too slow with its project reports), the project scheduler will
be responsible for respawning it in the Incubator.

The project scheduler will, of course, be pluggable. -C

On Fri, Apr 1, 2011 at 1:19 AM, Aaron T. Myers <atm@cloudera.com> wrote:
> Hello Hadoop Community,
>
> Given the tremendous positive feedback we've all had regarding the HDFS,
> MapReduce, and Common project split, I'd like to propose we take the next
> step and further separate the existing projects.
>
> I propose we begin by splitting the MapReduce project into separate "Map"
> and "Reduce" sub-projects. This will provide us the opportunity to tease out
> the complex interdependencies between "map" and "reduce" that exist today,
> to encourage us to write more modular and isolated code, which should speed
> releases. This will also aid our users who exclusively run map-only or
> reduce-only jobs. These are important use-cases, and so should be given high
> priority.
>
> Given that these two portions of the existing MapReduce project share a
> great deal of code, we will likely need to release these two new projects
> concurrently at first, but the eventual goal should certainly be to be able
> to release "Map" and "Reduce" independently. This seems intuitive to me,
> given the remarkable recent advancements in the academic community regarding
> "reduce," while the research coming out of the "map" academics has largely
> stagnated of late.
>
> If this proposal is accepted, and it has the success I think it will, then
> we should strongly consider splitting the other two projects as well. My gut
> instinct is that we should split "HDFS" into "HD" and "FS" sub-projects, and
> simply rename the "Common" project to "C'Mon." We can think about the
> details of what exactly these project splits mean later.
>
> Please let me know what you think.
>
> Best,
> Aaron
>

From general-return-3216-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 01 15:27:56 2011
Return-Path: <general-return-3216-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91534 invoked from network); 1 Apr 2011 15:27:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Apr 2011 15:27:56 -0000
Received: (qmail 27219 invoked by uid 500); 1 Apr 2011 15:27:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27146 invoked by uid 500); 1 Apr 2011 15:27:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27136 invoked by uid 99); 1 Apr 2011 15:27:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 15:27:54 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.94 as permitted sender)
Received: from [17.148.16.94] (HELO asmtpout019.mac.com) (17.148.16.94)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 15:27:46 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.31]
 (c-71-198-192-174.hsd1.ca.comcast.net [71.198.192.174])
 by asmtp019.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LIZ002ZLC8W3N10@asmtp019.mac.com>
 for general@hadoop.apache.org; Fri, 01 Apr 2011 08:26:59 -0700 (PDT)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-04-01_07:2011-04-01,2011-04-01,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=2 spamscore=2
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1104010057
Subject: Re: Proposal: Further Project Split(s)
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <AANLkTinBRgowHV2HNdkM1CHG+jF39B88pcY=3wQpFC=j@mail.gmail.com>
Date: Fri, 01 Apr 2011 08:26:57 -0700
Message-id: <57B22EAA-4A53-46CF-AD73-51D251F72E6E@mac.com>
References: <AANLkTimDH1zNAFNnwP__f0o+BAzVKZ3aagC8YaURTsF+@mail.gmail.com>
 <AANLkTinBRgowHV2HNdkM1CHG+jF39B88pcY=3wQpFC=j@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

-1+2.  This could potentially allow us to replace Jenkins with Hadoop for our build and test infrastructure.  That would be awesome!

n.

On Apr 1, 2011, at 1:57 AM, Chris Douglas wrote:

> Experience developing Hadoop has shown that we not only need to
> partition our projects for more active releases, but we also should
> explore speculative project splits. For this, a Hadoop.next() project
> should track the development of a project scheduler that can partition
> the Hadoop subprojects, possibly running a second version of a
> subproject in parallel. Downstream subprojects and TLPs automatically
> accept whichever releases first as a dependency. Implementation should
> combine ant, ivy, maven, and at least one legacy Hadoop build tool (to
> be written).
> 
> Of course, not all of these subprojects will succeed. When one fails
> (or is too slow with its project reports), the project scheduler will
> be responsible for respawning it in the Incubator.
> 
> The project scheduler will, of course, be pluggable. -C
> 
> On Fri, Apr 1, 2011 at 1:19 AM, Aaron T. Myers <atm@cloudera.com> wrote:
>> Hello Hadoop Community,
>> 
>> Given the tremendous positive feedback we've all had regarding the HDFS,
>> MapReduce, and Common project split, I'd like to propose we take the next
>> step and further separate the existing projects.
>> 
>> I propose we begin by splitting the MapReduce project into separate "Map"
>> and "Reduce" sub-projects. This will provide us the opportunity to tease out
>> the complex interdependencies between "map" and "reduce" that exist today,
>> to encourage us to write more modular and isolated code, which should speed
>> releases. This will also aid our users who exclusively run map-only or
>> reduce-only jobs. These are important use-cases, and so should be given high
>> priority.
>> 
>> Given that these two portions of the existing MapReduce project share a
>> great deal of code, we will likely need to release these two new projects
>> concurrently at first, but the eventual goal should certainly be to be able
>> to release "Map" and "Reduce" independently. This seems intuitive to me,
>> given the remarkable recent advancements in the academic community regarding
>> "reduce," while the research coming out of the "map" academics has largely
>> stagnated of late.
>> 
>> If this proposal is accepted, and it has the success I think it will, then
>> we should strongly consider splitting the other two projects as well. My gut
>> instinct is that we should split "HDFS" into "HD" and "FS" sub-projects, and
>> simply rename the "Common" project to "C'Mon." We can think about the
>> details of what exactly these project splits mean later.
>> 
>> Please let me know what you think.
>> 
>> Best,
>> Aaron
>> 


From general-return-3217-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 01 16:13:43 2011
Return-Path: <general-return-3217-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 48470 invoked from network); 1 Apr 2011 16:13:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Apr 2011 16:13:43 -0000
Received: (qmail 17901 invoked by uid 500); 1 Apr 2011 16:13:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17850 invoked by uid 500); 1 Apr 2011 16:13:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17842 invoked by uid 99); 1 Apr 2011 16:13:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 16:13:41 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of patrickangeles@gmail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 16:13:34 +0000
Received: by iwr19 with SMTP id 19so5318406iwr.35
        for <general@hadoop.apache.org>; Fri, 01 Apr 2011 09:13:13 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:cc:content-type;
        bh=GhX+5bF0tP4w1nfrl2QU1tfQ58oC2INWaU1LkDkMWuw=;
        b=bq2T3UMqu28AoWd4IbhokyvK3qStmVdFOA6axPdF5d/2swIOdNIEAZXwCyJ0Yddvsl
         ODXVprRdrKz/WbRiQ+JvKEs2v9A1OpFmff2TpdMsMLOUNdXZy3AXS6qLolmgEwg9Txyf
         duW2W7LWEfdyu7auSqjAsNPry3XsMU51/wVNc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:cc:content-type;
        b=MOt80yWE9sDg04mTNlALxvgTVhcaK9bI0J0UodyRbPcymIgmgoEXKmhacV28Z2b+Hg
         LcMWVWZSlJ1L2TpTbs1y+k86IJQQhu0TlWWDWTTmUGnoBBX4TBHkWgmxbyEA/+pDS8Fz
         IXY+kcPql05XcBgYrSthro+DJAjz6VMx8ZX4s=
MIME-Version: 1.0
Received: by 10.42.136.138 with SMTP id u10mr1639466ict.104.1301674393208;
 Fri, 01 Apr 2011 09:13:13 -0700 (PDT)
Sender: patrickangeles@gmail.com
Received: by 10.42.2.82 with HTTP; Fri, 1 Apr 2011 09:13:13 -0700 (PDT)
In-Reply-To: <57B22EAA-4A53-46CF-AD73-51D251F72E6E@mac.com>
References: <AANLkTimDH1zNAFNnwP__f0o+BAzVKZ3aagC8YaURTsF+@mail.gmail.com>
	<AANLkTinBRgowHV2HNdkM1CHG+jF39B88pcY=3wQpFC=j@mail.gmail.com>
	<57B22EAA-4A53-46CF-AD73-51D251F72E6E@mac.com>
Date: Fri, 1 Apr 2011 12:13:13 -0400
X-Google-Sender-Auth: fd6Vjs7NK3a1UfViK-nPsJB-fTk
Message-ID: <AANLkTinbzYrMguy-yDfZDCNufbFEjKVj4VdSjb9=_fJZ@mail.gmail.com>
Subject: Re: Proposal: Further Project Split(s)
From: Patrick Angeles <patrick@cloudera.com>
To: general@hadoop.apache.org
Cc: Nigel Daley <ndaley@mac.com>
Content-Type: multipart/alternative; boundary=90e6ba1eff48d20901049fddaf26
X-Virus-Checked: Checked by ClamAV on apache.org

--90e6ba1eff48d20901049fddaf26
Content-Type: text/plain; charset=ISO-8859-1

+1

This will allow Hadoop to better compete with GoDaddy's "Hadoop Killer"
skunkworks project.

On Fri, Apr 1, 2011 at 11:26 AM, Nigel Daley <ndaley@mac.com> wrote:

> -1+2.  This could potentially allow us to replace Jenkins with Hadoop for
> our build and test infrastructure.  That would be awesome!
>
> n.
>
> On Apr 1, 2011, at 1:57 AM, Chris Douglas wrote:
>
> > Experience developing Hadoop has shown that we not only need to
> > partition our projects for more active releases, but we also should
> > explore speculative project splits. For this, a Hadoop.next() project
> > should track the development of a project scheduler that can partition
> > the Hadoop subprojects, possibly running a second version of a
> > subproject in parallel. Downstream subprojects and TLPs automatically
> > accept whichever releases first as a dependency. Implementation should
> > combine ant, ivy, maven, and at least one legacy Hadoop build tool (to
> > be written).
> >
> > Of course, not all of these subprojects will succeed. When one fails
> > (or is too slow with its project reports), the project scheduler will
> > be responsible for respawning it in the Incubator.
> >
> > The project scheduler will, of course, be pluggable. -C
> >
> > On Fri, Apr 1, 2011 at 1:19 AM, Aaron T. Myers <atm@cloudera.com> wrote:
> >> Hello Hadoop Community,
> >>
> >> Given the tremendous positive feedback we've all had regarding the HDFS,
> >> MapReduce, and Common project split, I'd like to propose we take the
> next
> >> step and further separate the existing projects.
> >>
> >> I propose we begin by splitting the MapReduce project into separate
> "Map"
> >> and "Reduce" sub-projects. This will provide us the opportunity to tease
> out
> >> the complex interdependencies between "map" and "reduce" that exist
> today,
> >> to encourage us to write more modular and isolated code, which should
> speed
> >> releases. This will also aid our users who exclusively run map-only or
> >> reduce-only jobs. These are important use-cases, and so should be given
> high
> >> priority.
> >>
> >> Given that these two portions of the existing MapReduce project share a
> >> great deal of code, we will likely need to release these two new
> projects
> >> concurrently at first, but the eventual goal should certainly be to be
> able
> >> to release "Map" and "Reduce" independently. This seems intuitive to me,
> >> given the remarkable recent advancements in the academic community
> regarding
> >> "reduce," while the research coming out of the "map" academics has
> largely
> >> stagnated of late.
> >>
> >> If this proposal is accepted, and it has the success I think it will,
> then
> >> we should strongly consider splitting the other two projects as well. My
> gut
> >> instinct is that we should split "HDFS" into "HD" and "FS" sub-projects,
> and
> >> simply rename the "Common" project to "C'Mon." We can think about the
> >> details of what exactly these project splits mean later.
> >>
> >> Please let me know what you think.
> >>
> >> Best,
> >> Aaron
> >>
>
>

--90e6ba1eff48d20901049fddaf26--

From general-return-3218-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 01 17:07:34 2011
Return-Path: <general-return-3218-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49220 invoked from network); 1 Apr 2011 17:07:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Apr 2011 17:07:33 -0000
Received: (qmail 34890 invoked by uid 500); 1 Apr 2011 17:07:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34843 invoked by uid 500); 1 Apr 2011 17:07:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34835 invoked by uid 99); 1 Apr 2011 17:07:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 17:07:31 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [128.149.139.109] (HELO mail.jpl.nasa.gov) (128.149.139.109)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 17:07:24 +0000
Received: from mail.jpl.nasa.gov (altvirehtstap02.jpl.nasa.gov [128.149.137.73])
	by smtp.jpl.nasa.gov (Switch-3.4.3/Switch-3.4.3) with ESMTP id p31H6xfr001531
	(using TLSv1/SSLv3 with cipher RC4-MD5 (128 bits) verified NO)
	for <general@hadoop.apache.org>; Fri, 1 Apr 2011 10:07:02 -0700
Received: from ALTPHYEMBEVSP20.RES.AD.JPL ([128.149.137.82]) by
 ALTVIREHTSTAP02.RES.AD.JPL ([128.149.137.73]) with mapi; Fri, 1 Apr 2011
 10:07:00 -0700
From: "Mattmann, Chris A (388J)" <chris.a.mattmann@jpl.nasa.gov>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 1 Apr 2011 10:06:51 -0700
Subject: Re: Proposal: Further Project Split(s)
Thread-Topic: Proposal: Further Project Split(s)
Thread-Index: AcvwjzYIw9lpNvzvTmCVhsoVnU62ow==
Message-ID: <2305C49F-5920-4837-8A38-31EB1023955A@jpl.nasa.gov>
References: <AANLkTimDH1zNAFNnwP__f0o+BAzVKZ3aagC8YaURTsF+@mail.gmail.com>
 <AANLkTinBRgowHV2HNdkM1CHG+jF39B88pcY=3wQpFC=j@mail.gmail.com>
In-Reply-To: <AANLkTinBRgowHV2HNdkM1CHG+jF39B88pcY=3wQpFC=j@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Source-IP: altvirehtstap02.jpl.nasa.gov [128.149.137.73]
X-Source-Sender: chris.a.mattmann@jpl.nasa.gov
X-AUTH: Authorized
X-Virus-Checked: Checked by ClamAV on apache.org

LOL@Chris!!!

On Apr 1, 2011, at 1:57 AM, Chris Douglas wrote:

> Experience developing Hadoop has shown that we not only need to
> partition our projects for more active releases, but we also should
> explore speculative project splits. For this, a Hadoop.next() project
> should track the development of a project scheduler that can partition
> the Hadoop subprojects, possibly running a second version of a
> subproject in parallel. Downstream subprojects and TLPs automatically
> accept whichever releases first as a dependency. Implementation should
> combine ant, ivy, maven, and at least one legacy Hadoop build tool (to
> be written).
>=20
> Of course, not all of these subprojects will succeed. When one fails
> (or is too slow with its project reports), the project scheduler will
> be responsible for respawning it in the Incubator.
>=20
> The project scheduler will, of course, be pluggable. -C
>=20
> On Fri, Apr 1, 2011 at 1:19 AM, Aaron T. Myers <atm@cloudera.com> wrote:
>> Hello Hadoop Community,
>>=20
>> Given the tremendous positive feedback we've all had regarding the HDFS,
>> MapReduce, and Common project split, I'd like to propose we take the nex=
t
>> step and further separate the existing projects.
>>=20
>> I propose we begin by splitting the MapReduce project into separate "Map=
"
>> and "Reduce" sub-projects. This will provide us the opportunity to tease=
 out
>> the complex interdependencies between "map" and "reduce" that exist toda=
y,
>> to encourage us to write more modular and isolated code, which should sp=
eed
>> releases. This will also aid our users who exclusively run map-only or
>> reduce-only jobs. These are important use-cases, and so should be given =
high
>> priority.
>>=20
>> Given that these two portions of the existing MapReduce project share a
>> great deal of code, we will likely need to release these two new project=
s
>> concurrently at first, but the eventual goal should certainly be to be a=
ble
>> to release "Map" and "Reduce" independently. This seems intuitive to me,
>> given the remarkable recent advancements in the academic community regar=
ding
>> "reduce," while the research coming out of the "map" academics has large=
ly
>> stagnated of late.
>>=20
>> If this proposal is accepted, and it has the success I think it will, th=
en
>> we should strongly consider splitting the other two projects as well. My=
 gut
>> instinct is that we should split "HDFS" into "HD" and "FS" sub-projects,=
 and
>> simply rename the "Common" project to "C'Mon." We can think about the
>> details of what exactly these project splits mean later.
>>=20
>> Please let me know what you think.
>>=20
>> Best,
>> Aaron
>>=20


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


From general-return-3219-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 01 18:03:53 2011
Return-Path: <general-return-3219-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 32755 invoked from network); 1 Apr 2011 18:03:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Apr 2011 18:03:53 -0000
Received: (qmail 21908 invoked by uid 500); 1 Apr 2011 18:03:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21833 invoked by uid 500); 1 Apr 2011 18:03:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 81987 invoked by uid 99); 1 Apr 2011 17:38:24 -0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=O3Ws5pG5DvFzOUDqWRD3WN0Fa1DfQN1OPMwTON/AbHLFPMVA6KpAKlx0
   RY38Te8EmdwZxly0AvhFBO++DBbTPApxpAGlwUWHkWxxOG6bKptMsslMV
   iplabzH+v1MMLbz;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1301679497; x=1333215497;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20Proposal:=20Further=20Project=20Split(s
   )|Date:=20Fri,=201=20Apr=202011=2017:40:32=20+0000
   |Message-ID:=20<1A2A2DF1-E442-418D-A6A4-AC214C36F213@link
   edin.com>|To:=20"<general@hadoop.apache.org>"=20<general@
   hadoop.apache.org>|MIME-Version:=201.0
   |Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<5B2C46534FDF8F4293A617DAD471BC49@linkedin
   .com>|In-Reply-To:=20<AANLkTinBRgowHV2HNdkM1CHG+jF39B88pc
   Y=3D3wQpFC=3Dj@mail.gmail.com>|References:=20<AANLkTimDH1
   zNAFNnwP__f0o+BAzVKZ3aagC8YaURTsF+@mail.gmail.com>=0D=0A
   =20<AANLkTinBRgowHV2HNdkM1CHG+jF39B88pcY=3D3wQpFC=3Dj@mai
   l.gmail.com>;
  bh=i3blXvL/Qyz/vWu5m2pp5ukYb+RV9EMnD3UObLsVtwk=;
  b=bB2RsqyFqpnLwJTm69hwv1bL/YaxZbepIzdU1UJEBY+zxKHmniIra1oG
   OvAiLRLYx007bGeXopAyenW0FJZcqEFcP6f+dsmYv6NDlIy4aVrZVx+yf
   TN5Uy5TP9+c1T/Y;
X-IronPort-AV: E=Sophos;i="4.63,283,1299484800"; 
   d="scan'208";a="21753120"
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: Proposal: Further Project Split(s)
Thread-Topic: Proposal: Further Project Split(s)
Thread-Index: AQHL8EX7jHElUjWn+kiKhYleqOdIq5RJKlIAgACRjoA=
Date: Fri, 1 Apr 2011 17:40:32 +0000
Message-ID: <1A2A2DF1-E442-418D-A6A4-AC214C36F213@linkedin.com>
References: <AANLkTimDH1zNAFNnwP__f0o+BAzVKZ3aagC8YaURTsF+@mail.gmail.com>
 <AANLkTinBRgowHV2HNdkM1CHG+jF39B88pcY=3wQpFC=j@mail.gmail.com>
In-Reply-To: <AANLkTinBRgowHV2HNdkM1CHG+jF39B88pcY=3wQpFC=j@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <5B2C46534FDF8F4293A617DAD471BC49@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


On Apr 1, 2011, at 1:57 AM, Chris Douglas wrote:

> Experience developing Hadoop has shown that we not only need to
> partition our projects for more active releases, but we also should
> explore speculative project splits. For this, a Hadoop.next() project
> should track the development of a project scheduler that can partition
> the Hadoop subprojects, possibly running a second version of a
> subproject in parallel. Downstream subprojects and TLPs automatically
> accept whichever releases first as a dependency. Implementation should
> combine ant, ivy, maven, and at least one legacy Hadoop build tool (to
> be written).


-1, until it supports eclipse.



From general-return-3220-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 01 18:30:21 2011
Return-Path: <general-return-3220-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 3555 invoked from network); 1 Apr 2011 18:30:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Apr 2011 18:30:21 -0000
Received: (qmail 59301 invoked by uid 500); 1 Apr 2011 18:30:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59181 invoked by uid 500); 1 Apr 2011 18:30:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59173 invoked by uid 99); 1 Apr 2011 18:30:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 18:30:19 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [129.93.165.11] (HELO cse-mail.unl.edu) (129.93.165.11)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 18:30:10 +0000
Received: from cse-barracuda.cse.unl.edu (cse-barracuda.unl.edu [129.93.164.185])
	(authenticated bits=0)
	by cse-mail.unl.edu (8.14.3/8.14.3) with ESMTP id p31ITjYf024646
	for <general@hadoop.apache.org>; Fri, 1 Apr 2011 13:29:50 -0500 (CDT)
X-ASG-Debug-ID: 1301682584-4b5c35b48ff1-IjwG86
Received: from cse.unl.edu (cse.unl.edu [129.93.165.2]) by cse-barracuda.cse.unl.edu with ESMTP id QNRrSxc9n9Ij6eGg for <general@hadoop.apache.org>; Fri, 01 Apr 2011 13:29:44 -0500 (CDT)
X-Barracuda-Envelope-From: bbockelm@cse.unl.edu
X-Barracuda-RBL-Trusted-Forwarder: 129.93.165.2
Received: from pcp088890pcs.unl.edu (pcp088890pcs.unl.edu [129.93.158.5])
	(authenticated bits=0)
	by cse.unl.edu (8.14.4/8.14.3) with ESMTP id p31ITiO4010352
	(version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO)
	for <general@hadoop.apache.org>; Fri, 1 Apr 2011 13:29:44 -0500
From: Brian Bockelman <bbockelm@cse.unl.edu>
X-Barracuda-Apparent-Source-IP: 129.93.158.5
Mime-Version: 1.0 (Apple Message framework v1082)
Content-Type: multipart/signed; boundary=Apple-Mail-447--894646415; protocol="application/pkcs7-signature"; micalg=sha1
Subject: Re: Proposal: Further Project Split(s)
Date: Fri, 1 Apr 2011 13:29:44 -0500
X-ASG-Orig-Subj: Re: Proposal: Further Project Split(s)
In-Reply-To: <1A2A2DF1-E442-418D-A6A4-AC214C36F213@linkedin.com>
To: general@hadoop.apache.org
References: <AANLkTimDH1zNAFNnwP__f0o+BAzVKZ3aagC8YaURTsF+@mail.gmail.com> <AANLkTinBRgowHV2HNdkM1CHG+jF39B88pcY=3wQpFC=j@mail.gmail.com> <1A2A2DF1-E442-418D-A6A4-AC214C36F213@linkedin.com>
Message-Id: <ECE4DDA5-8D88-47BF-A9E5-316C3243A7E7@cse.unl.edu>
X-Mailer: Apple Mail (2.1082)
X-Barracuda-Connect: cse.unl.edu[129.93.165.2]
X-Barracuda-Start-Time: 1301682584
X-Barracuda-URL: http://cse-barracuda.unl.edu:8000/cgi-mod/mark.cgi
X-Virus-Scanned: clamav-milter 0.97 at cse-mail
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using global scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=9.0 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.2.59622
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0.1 (cse-mail.unl.edu [129.93.165.11]); Fri, 01 Apr 2011 13:29:50 -0500 (CDT)
X-Virus-Status: Clean

--Apple-Mail-447--894646415
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
	charset=us-ascii


On Apr 1, 2011, at 12:40 PM, Allen Wittenauer wrote:

> 
> On Apr 1, 2011, at 1:57 AM, Chris Douglas wrote:
> 
>> Experience developing Hadoop has shown that we not only need to
>> partition our projects for more active releases, but we also should
>> explore speculative project splits. For this, a Hadoop.next() project
>> should track the development of a project scheduler that can partition
>> the Hadoop subprojects, possibly running a second version of a
>> subproject in parallel. Downstream subprojects and TLPs automatically
>> accept whichever releases first as a dependency. Implementation should
>> combine ant, ivy, maven, and at least one legacy Hadoop build tool (to
>> be written).
> 
> 
> -1, until it supports eclipse.
> 

-1, until it supports emacs
--Apple-Mail-447--894646415
Content-Disposition: attachment;
	filename=smime.p7s
Content-Type: application/pkcs7-signature;
	name=smime.p7s
Content-Transfer-Encoding: base64

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIIKDCCA/gw
ggLgoAMCAQICASkwDQYJKoZIhvcNAQEFBQAwdTETMBEGCgmSJomT8ixkARkWA25ldDESMBAGCgmS
JomT8ixkARkWAkVTMQ4wDAYDVQQKEwVFU25ldDEgMB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9y
aXRpZXMxGDAWBgNVBAMTD0VTbmV0IFJvb3QgQ0EgMTAeFw0wMjEyMDUwODAwMDBaFw0xMzAxMjUw
ODAwMDBaMGkxEzARBgoJkiaJk/IsZAEZFgNvcmcxGDAWBgoJkiaJk/IsZAEZFghET0VHcmlkczEg
MB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9yaXRpZXMxFjAUBgNVBAMTDURPRUdyaWRzIENBIDEw
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC09dYjYaPbCD5mtbiQb7Ka3y1qAm0ZcqKC
FciWcfe8Kwcuy9tjHuIsLf9ZItdkDW4xy8sua9nJlx3KlwjtumTMtOtg35KZCknUd8KM4VGTSFdL
VG9AbNayef76caVCGM1+jyF0Lq03kauGOPTcNfZe1TZa3e1c9rc8ljV5OSWa/mfsCACyS5zFIWu0
yIDNyJdf+n0hwaPN53wllpJ30taD+JBjQ7h2k4xRWzeaznLOb9OztZVRA/1sVze+iczFh2xwa4Vd
Gy0eIIPw1pfvYwxO36rm0S109qvbsNlaroPRbxerPKakQLpKe034Xcx7gBPqUk/FxoRRWin5EWN3
rz9LAgMBAAGjgZ4wgZswDgYDVR0PAQH/BAQDAgGGMBEGCWCGSAGG+EIBAQQEAwIAhzAdBgNVHQ4E
FgQUyhkdEo5upDhdQtQxDgjb2Y0XDV0wHwYDVR0jBBgwFoAUvF1NSC/4NZRZq1yJSz7RsjoUAeow
DwYDVR0TAQH/BAUwAwEB/zAlBgNVHREEHjAcgRpET0VHcmlkcy1DQS0xQGRvZWdyaWRzLm9yZzAN
BgkqhkiG9w0BAQUFAAOCAQEAZNVrIDLqe39CEOiJt7Q7EpBPhAihMvDTSf/42u0SMbUmChww4mLm
ph5DBghZUVF8Yn59kRZMn1QLOtO1HzLqvAvPITacZVPlJgG2IXzlR636YghZFAycbIUEOJDBHR4v
tQO1KDxgZwvAbtmKIoxvhUCq2xsfFt9kCBBn+JYtQ6O5LsBJq3PmuubeMcc7mbQAfJZ7h/3Qghgk
FIhmE1+LBXPJbkuP8vgfg6h2BKoAf5TFfZECgGZKimfN110tBvfedGZwYYd3/GsJc83B0JN1gny0
gqNVPm392UchXGeBRrHnm2gkhIkr48Oq6EmNGV9/a6XfbplQW/JWbtPVPWkaizCCBCgwggMQoAMC
AQICAwCvNzANBgkqhkiG9w0BAQUFADBpMRMwEQYKCZImiZPyLGQBGRYDb3JnMRgwFgYKCZImiZPy
LGQBGRYIRE9FR3JpZHMxIDAeBgNVBAsTF0NlcnRpZmljYXRlIEF1dGhvcml0aWVzMRYwFAYDVQQD
Ew1ET0VHcmlkcyBDQSAxMB4XDTEwMDUyNDE4MTg0OVoXDTExMDUyNDE4MTg0OVowYTETMBEGCgmS
JomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCGRvZWdyaWRzMQ8wDQYDVQQLEwZQZW9wbGUx
HzAdBgNVBAMTFkJyaWFuIEJvY2tlbG1hbiA1MDQzMDcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw
ggEKAoIBAQCwiNKRMFRMWG+AJyo/G5bYisPndrH+44JqRHdxDzMaQP59gxrBO2koRwg/13gINe3e
J8QL6bX4ANz/y1uglsytmoJwK5J9fxNYgJgbja83kO0j0HNL6oIyBhGStluaNTbiHrk8GA95M9VR
XRqpYiwKvT1F0KS2r7sZ+PWevbAek787eTqg51yuvUUlIBPgTm1kV3vZs21oeIZUuw7wPGXBKN49
XqIDsamUIqiFARwPgqKR9eo6itlYy2NrHo0hHLXew37rEOcKv/0g4pI4J/y4+1qB7fN3nMkIMack
FWfAQTngcnH/JpKmh8fmXdkeVv8EKYUXIgkUI5pb+Ak105olAgMBAAGjgeAwgd0wEQYJYIZIAYb4
QgEBBAQDAgWgMA4GA1UdDwEB/wQEAwIF4DA2BgNVHSAELzAtMA0GCyqGSIb3TAMHAQMBMAwGCiqG
SIb3TAUCAgEwDgYMKoZIhvdMBQIDAgEBMD4GA1UdHwQ3MDUwM6AxoC+GLWh0dHA6Ly9jcmwuZG9l
Z3JpZHMub3JnLzFjM2YyY2E4LzFjM2YyY2E4LmNybDAfBgNVHREEGDAWgRRiYm9ja2VsbUBjc2Uu
dW5sLmVkdTAfBgNVHSMEGDAWgBTKGR0Sjm6kOF1C1DEOCNvZjRcNXTANBgkqhkiG9w0BAQUFAAOC
AQEAlQSD+8Cvb0GxWqD4xhXd8Sl5MJRr1uJxMeGoMA4RZAJuyvVlBUx8v5moqY0XHMfNI+FulyMx
wgOoNfvF3dluz3J4C/u5NvzfNqikLj++sL4XDaZoxSHLo9cJxVTcM15Gogct+kvIF1+msEsqLlNR
/lqUVE/o8ANdD6PVx/044f/Dzi6s+6jZmBz/vWPI77ymT1EHaAkaHDqoNIlItPQrAHdkJWY67v1z
s6mDrKaspF/2ThDdYax208o1oLFd8wY8kQUdTBlMmUAbchQnjOC9vH17w6meDc8VxD+pEL3vAiG2
JN2vzQ3IJCYTCTmagUyiLWHFEudH8Brn43NY0/HwKTGCAv0wggL5AgEBMHAwaTETMBEGCgmSJomT
8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0
ZSBBdXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3JpZHMgQ0EgMQIDAK83MAkGBSsOAwIaBQCgggFi
MBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTExMDQwMTE4Mjk0NFow
IwYJKoZIhvcNAQkEMRYEFDjel99x2uZrfuDiTrTtPtO2dkunMH8GCSsGAQQBgjcQBDFyMHAwaTET
MBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERPRUdyaWRzMSAwHgYDVQQLExdD
ZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3JpZHMgQ0EgMQIDAK83MIGBBgsq
hkiG9w0BCRACCzFyoHAwaTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERP
RUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3Jp
ZHMgQ0EgMQIDAK83MA0GCSqGSIb3DQEBAQUABIIBAEUYdgWEAyJQURu7+ZqFXzTAka2Zo8r/k5me
5BRekF7WmfR8Qf+24hqH+lm50CVTwcLAHZSo8otr8Q39DW/VsqAj6VKJbuQZTzOFzZ7jGpvgYAQF
CmwSRPYRLhApY3yk/0iu8209nZ6EiANMZ1rojSNccILn20gbGuos85jq4yS6h4J5Rcj/+rhL1PQ6
7YkYAlCSNwj6eAWC3c/YHOCAnRlbSNwEMkol73pNoVNcmalVROZBtCfWnzqP5Sy/goixZLhskKI+
LZ2mTt7swDa6PuJLi7MdN1/HMGo3NhVU8Gpm91l6KuI/1uKUBYWW+4NRrNot3KpX5syQ2bTE7CIp
fQEAAAAAAAA=

--Apple-Mail-447--894646415--

From general-return-3221-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 01 18:36:11 2011
Return-Path: <general-return-3221-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53207 invoked from network); 1 Apr 2011 18:36:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Apr 2011 18:36:11 -0000
Received: (qmail 71231 invoked by uid 500); 1 Apr 2011 18:36:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71165 invoked by uid 500); 1 Apr 2011 18:36:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71157 invoked by uid 99); 1 Apr 2011 18:36:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 18:36:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 01 Apr 2011 18:36:07 +0000
Received: (qmail 49861 invoked by uid 99); 1 Apr 2011 18:35:45 -0000
Received: from localhost.apache.org (HELO mail-wy0-f176.google.com) (127.0.0.1)
  (smtp-auth username mahadev, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 18:35:45 +0000
Received: by wyb40 with SMTP id 40so4913490wyb.35
        for <general@hadoop.apache.org>; Fri, 01 Apr 2011 11:35:43 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.220.194 with SMTP id o44mr1131196wep.105.1301682943386;
 Fri, 01 Apr 2011 11:35:43 -0700 (PDT)
Received: by 10.216.153.196 with HTTP; Fri, 1 Apr 2011 11:35:43 -0700 (PDT)
In-Reply-To: <ECE4DDA5-8D88-47BF-A9E5-316C3243A7E7@cse.unl.edu>
References: <AANLkTimDH1zNAFNnwP__f0o+BAzVKZ3aagC8YaURTsF+@mail.gmail.com>
	<AANLkTinBRgowHV2HNdkM1CHG+jF39B88pcY=3wQpFC=j@mail.gmail.com>
	<1A2A2DF1-E442-418D-A6A4-AC214C36F213@linkedin.com>
	<ECE4DDA5-8D88-47BF-A9E5-316C3243A7E7@cse.unl.edu>
Date: Fri, 1 Apr 2011 11:35:43 -0700
Message-ID: <AANLkTi=5GgacRzhL72RtK4Xoq+OFhYCLoSGvrhO3hRJM@mail.gmail.com>
Subject: Re: Proposal: Further Project Split(s)
From: Mahadev Konar <mahadev@apache.org>
To: general@hadoop.apache.org
Cc: Brian Bockelman <bbockelm@cse.unl.edu>
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

+1. Brilliant idea!

>> On Apr 1, 2011, at 1:57 AM, Chris Douglas wrote:
>>
>>> Experience developing Hadoop has shown that we not only need to
>>> partition our projects for more active releases, but we also should
>>> explore speculative project splits. For this, a Hadoop.next() project
>>> should track the development of a project scheduler that can partition
>>> the Hadoop subprojects, possibly running a second version of a
>>> subproject in parallel. Downstream subprojects and TLPs automatically
>>> accept whichever releases first as a dependency. Implementation should
>>> combine ant, ivy, maven, and at least one legacy Hadoop build tool (to
>>> be written).
>>
>>
>> -1, until it supports eclipse.
>>
>
> -1, until it supports emacs



-- 
thanks
mahadev
@mahadevkonar

From general-return-3222-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 01 18:37:49 2011
Return-Path: <general-return-3222-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53984 invoked from network); 1 Apr 2011 18:37:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Apr 2011 18:37:49 -0000
Received: (qmail 74436 invoked by uid 500); 1 Apr 2011 18:37:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74375 invoked by uid 500); 1 Apr 2011 18:37:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74367 invoked by uid 99); 1 Apr 2011 18:37:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 18:37:47 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of aaa@cloudera.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 18:37:41 +0000
Received: by vws7 with SMTP id 7so4544615vws.35
        for <general@hadoop.apache.org>; Fri, 01 Apr 2011 11:37:20 -0700 (PDT)
Received: by 10.52.0.205 with SMTP id 13mr5717016vdg.58.1301683040599; Fri, 01
 Apr 2011 11:37:20 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.52.160.68 with HTTP; Fri, 1 Apr 2011 11:30:17 -0700 (PDT)
In-Reply-To: <1A2A2DF1-E442-418D-A6A4-AC214C36F213@linkedin.com>
References: <AANLkTimDH1zNAFNnwP__f0o+BAzVKZ3aagC8YaURTsF+@mail.gmail.com>
 <AANLkTinBRgowHV2HNdkM1CHG+jF39B88pcY=3wQpFC=j@mail.gmail.com> <1A2A2DF1-E442-418D-A6A4-AC214C36F213@linkedin.com>
From: Amr Awadallah <aaa@cloudera.com>
Date: Fri, 1 Apr 2011 11:30:17 -0700
Message-ID: <AANLkTi=TnZ8KiYikyqrs+KO54PC=ixawcGbL-EW4dYBz@mail.gmail.com>
Subject: Re: Proposal: Further Project Split(s)
To: general@hadoop.apache.org
Cc: Allen Wittenauer <awittenauer@linkedin.com>
Content-Type: multipart/alternative; boundary=20cf304346e43ec077049fdfb395
X-Virus-Checked: Checked by ClamAV on apache.org

--20cf304346e43ec077049fdfb395
Content-Type: text/plain; charset=ISO-8859-1

Strong -1 from me, this *idiotic* since we first need to split the NN and DN
into separate projects.

-- amr

On Fri, Apr 1, 2011 at 10:40 AM, Allen Wittenauer
<awittenauer@linkedin.com>wrote:

>
> On Apr 1, 2011, at 1:57 AM, Chris Douglas wrote:
>
> > Experience developing Hadoop has shown that we not only need to
> > partition our projects for more active releases, but we also should
> > explore speculative project splits. For this, a Hadoop.next() project
> > should track the development of a project scheduler that can partition
> > the Hadoop subprojects, possibly running a second version of a
> > subproject in parallel. Downstream subprojects and TLPs automatically
> > accept whichever releases first as a dependency. Implementation should
> > combine ant, ivy, maven, and at least one legacy Hadoop build tool (to
> > be written).
>
>
> -1, until it supports eclipse.
>
>
>

--20cf304346e43ec077049fdfb395--

From general-return-3223-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 01 18:42:41 2011
Return-Path: <general-return-3223-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55355 invoked from network); 1 Apr 2011 18:42:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Apr 2011 18:42:41 -0000
Received: (qmail 84666 invoked by uid 500); 1 Apr 2011 18:42:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84608 invoked by uid 500); 1 Apr 2011 18:42:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84583 invoked by uid 99); 1 Apr 2011 18:42:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 18:42:40 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.178] (HELO mail-px0-f178.google.com) (209.85.212.178)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 18:42:34 +0000
Received: by pxi1 with SMTP id 1so1348650pxi.37
        for <general@hadoop.apache.org>; Fri, 01 Apr 2011 11:42:14 -0700 (PDT)
Received: by 10.142.142.4 with SMTP id p4mr3560267wfd.43.1301683334055; Fri,
 01 Apr 2011 11:42:14 -0700 (PDT)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.68.44.35 with HTTP; Fri, 1 Apr 2011 11:41:54 -0700 (PDT)
In-Reply-To: <57B22EAA-4A53-46CF-AD73-51D251F72E6E@mac.com>
References: <AANLkTimDH1zNAFNnwP__f0o+BAzVKZ3aagC8YaURTsF+@mail.gmail.com>
 <AANLkTinBRgowHV2HNdkM1CHG+jF39B88pcY=3wQpFC=j@mail.gmail.com> <57B22EAA-4A53-46CF-AD73-51D251F72E6E@mac.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Fri, 1 Apr 2011 11:41:54 -0700
X-Google-Sender-Auth: d6P3_8JD1TLevqRK-GLnTGDY_2U
Message-ID: <AANLkTi=RKGFt6iiKtyLxfHG-505je+GzcokgNR+RakB=@mail.gmail.com>
Subject: Re: Proposal: Further Project Split(s)
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Fri, Apr 1, 2011 at 08:26, Nigel Daley <ndaley@mac.com> wrote:
> -1+2. =A0This could potentially allow us to replace Jenkins with Hadoop f=
or our build and test infrastructure. =A0That would be awesome!

Has anyone checked a calendar lately?

> On Apr 1, 2011, at 1:57 AM, Chris Douglas wrote:
>
>> Experience developing Hadoop has shown that we not only need to
>> partition our projects for more active releases, but we also should
>> explore speculative project splits. For this, a Hadoop.next() project
>> should track the development of a project scheduler that can partition
>> the Hadoop subprojects, possibly running a second version of a
>> subproject in parallel. Downstream subprojects and TLPs automatically
>> accept whichever releases first as a dependency. Implementation should
>> combine ant, ivy, maven, and at least one legacy Hadoop build tool (to
>> be written).
>>
>> Of course, not all of these subprojects will succeed. When one fails
>> (or is too slow with its project reports), the project scheduler will
>> be responsible for respawning it in the Incubator.
>>
>> The project scheduler will, of course, be pluggable. -C
>>
>> On Fri, Apr 1, 2011 at 1:19 AM, Aaron T. Myers <atm@cloudera.com> wrote:
>>> Hello Hadoop Community,
>>>
>>> Given the tremendous positive feedback we've all had regarding the HDFS=
,
>>> MapReduce, and Common project split, I'd like to propose we take the ne=
xt
>>> step and further separate the existing projects.
>>>
>>> I propose we begin by splitting the MapReduce project into separate "Ma=
p"
>>> and "Reduce" sub-projects. This will provide us the opportunity to teas=
e out
>>> the complex interdependencies between "map" and "reduce" that exist tod=
ay,
>>> to encourage us to write more modular and isolated code, which should s=
peed
>>> releases. This will also aid our users who exclusively run map-only or
>>> reduce-only jobs. These are important use-cases, and so should be given=
 high
>>> priority.
>>>
>>> Given that these two portions of the existing MapReduce project share a
>>> great deal of code, we will likely need to release these two new projec=
ts
>>> concurrently at first, but the eventual goal should certainly be to be =
able
>>> to release "Map" and "Reduce" independently. This seems intuitive to me=
,
>>> given the remarkable recent advancements in the academic community rega=
rding
>>> "reduce," while the research coming out of the "map" academics has larg=
ely
>>> stagnated of late.
>>>
>>> If this proposal is accepted, and it has the success I think it will, t=
hen
>>> we should strongly consider splitting the other two projects as well. M=
y gut
>>> instinct is that we should split "HDFS" into "HD" and "FS" sub-projects=
, and
>>> simply rename the "Common" project to "C'Mon." We can think about the
>>> details of what exactly these project splits mean later.
>>>
>>> Please let me know what you think.
>>>
>>> Best,
>>> Aaron
>>>
>
>

From general-return-3224-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 01 19:35:30 2011
Return-Path: <general-return-3224-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 48702 invoked from network); 1 Apr 2011 19:35:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Apr 2011 19:35:30 -0000
Received: (qmail 53544 invoked by uid 500); 1 Apr 2011 19:35:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53500 invoked by uid 500); 1 Apr 2011 19:35:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53492 invoked by uid 99); 1 Apr 2011 19:35:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 19:35:29 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 01 Apr 2011 19:35:26 +0000
Received: (qmail 44897 invoked by uid 99); 1 Apr 2011 19:35:04 -0000
Received: from localhost.apache.org (HELO [192.168.100.72]) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 19:35:04 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Proposal: Further Project Split(s)
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <AANLkTi=RKGFt6iiKtyLxfHG-505je+GzcokgNR+RakB=@mail.gmail.com>
Date: Fri, 1 Apr 2011 12:35:00 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <717BAFAB-5D9E-4462-9414-9A7F553748BE@apache.org>
References: <AANLkTimDH1zNAFNnwP__f0o+BAzVKZ3aagC8YaURTsF+@mail.gmail.com> <AANLkTinBRgowHV2HNdkM1CHG+jF39B88pcY=3wQpFC=j@mail.gmail.com> <57B22EAA-4A53-46CF-AD73-51D251F72E6E@mac.com> <AANLkTi=RKGFt6iiKtyLxfHG-505je+GzcokgNR+RakB=@mail.gmail.com>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org


On Apr 1, 2011, at 11:41 AM, Konstantin Boudnik wrote:

> On Fri, Apr 1, 2011 at 08:26, Nigel Daley <ndaley@mac.com> wrote:
>> -1+2.  This could potentially allow us to replace Jenkins with Hadoop =
for our build and test infrastructure.  That would be awesome!
>=20
> Has anyone checked a calendar lately?


	No.  My calendar application's map tasks are stuck behind our =
PYMK workflow.


From general-return-3225-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 01 19:46:53 2011
Return-Path: <general-return-3225-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68764 invoked from network); 1 Apr 2011 19:46:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Apr 2011 19:46:53 -0000
Received: (qmail 68388 invoked by uid 500); 1 Apr 2011 19:46:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68338 invoked by uid 500); 1 Apr 2011 19:46:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68330 invoked by uid 99); 1 Apr 2011 19:46:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 19:46:51 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Apr 2011 19:46:45 +0000
Received: by pzk10 with SMTP id 10so1059603pzk.35
        for <general@hadoop.apache.org>; Fri, 01 Apr 2011 12:46:25 -0700 (PDT)
Received: by 10.142.44.13 with SMTP id r13mr3476224wfr.328.1301687185060; Fri,
 01 Apr 2011 12:46:25 -0700 (PDT)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.68.44.35 with HTTP; Fri, 1 Apr 2011 12:46:05 -0700 (PDT)
In-Reply-To: <717BAFAB-5D9E-4462-9414-9A7F553748BE@apache.org>
References: <AANLkTimDH1zNAFNnwP__f0o+BAzVKZ3aagC8YaURTsF+@mail.gmail.com>
 <AANLkTinBRgowHV2HNdkM1CHG+jF39B88pcY=3wQpFC=j@mail.gmail.com>
 <57B22EAA-4A53-46CF-AD73-51D251F72E6E@mac.com> <AANLkTi=RKGFt6iiKtyLxfHG-505je+GzcokgNR+RakB=@mail.gmail.com>
 <717BAFAB-5D9E-4462-9414-9A7F553748BE@apache.org>
From: Konstantin Boudnik <cos@apache.org>
Date: Fri, 1 Apr 2011 12:46:05 -0700
X-Google-Sender-Auth: hCOuQjo-oI1DPtiskceCUGCAkVM
Message-ID: <AANLkTintOVxyn1kOWD0w0w4MhbWGAKFdicy5LUU6vriQ@mail.gmail.com>
Subject: Re: Proposal: Further Project Split(s)
To: general@hadoop.apache.org
Cc: Allen Wittenauer <aw@apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

And I tend to believe to all sort of stuff on this particular day
because this happens to be my birthday ;(

On Fri, Apr 1, 2011 at 12:35, Allen Wittenauer <aw@apache.org> wrote:
>
> On Apr 1, 2011, at 11:41 AM, Konstantin Boudnik wrote:
>
>> On Fri, Apr 1, 2011 at 08:26, Nigel Daley <ndaley@mac.com> wrote:
>>> -1+2. =A0This could potentially allow us to replace Jenkins with Hadoop=
 for our build and test infrastructure. =A0That would be awesome!
>>
>> Has anyone checked a calendar lately?
>
>
> =A0 =A0 =A0 =A0No. =A0My calendar application's map tasks are stuck behin=
d our PYMK workflow.
>
>

From general-return-3226-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 04 19:20:17 2011
Return-Path: <general-return-3226-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 26867 invoked from network); 4 Apr 2011 19:20:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Apr 2011 19:20:17 -0000
Received: (qmail 13117 invoked by uid 500); 4 Apr 2011 19:20:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13030 invoked by uid 500); 4 Apr 2011 19:20:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13022 invoked by uid 99); 4 Apr 2011 19:20:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Apr 2011 19:20:15 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 74.125.82.42 as permitted sender)
Received: from [74.125.82.42] (HELO mail-ww0-f42.google.com) (74.125.82.42)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Apr 2011 19:20:08 +0000
Received: by wwk4 with SMTP id 4so2143333wwk.5
        for <general@hadoop.apache.org>; Mon, 04 Apr 2011 12:19:48 -0700 (PDT)
Received: by 10.227.1.151 with SMTP id 23mr4409783wbf.175.1301944788156; Mon,
 04 Apr 2011 12:19:48 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.227.129.141 with HTTP; Mon, 4 Apr 2011 12:19:27 -0700 (PDT)
In-Reply-To: <D3254862-5A94-4121-9B0B-9AE276D0955C@mac.com>
References: <D97CF74A-147E-4573-B188-2F9C48CFC482@mac.com> <AANLkTi=F2eBsACBfjYSVava+VwFcsRpqZdRRN+OHXNRJ@mail.gmail.com>
 <823262F0-EE37-4A0A-8F8C-D9422541D67B@mac.com> <0CBAFDBB-AC2A-495D-8936-4DF4D6731B69@Holsman.NET>
 <AANLkTimUhDbTnwn6Hk2oYXFJMts+A=FnL-q-veXqX1m0@mail.gmail.com>
 <F2EC1B1B-A2BB-4FC8-9842-3904A25338EE@gbiv.com> <AANLkTikVcGcKHnh9EgUDQr2JGLPvkBvBRgn4my3uJfOc@mail.gmail.com>
 <864B55E7-8A46-458F-8B16-8252A2011835@Holsman.NET> <AANLkTikhe6YjUBGY8LMc3OG=b7VXWz7WhyNKGkJRu_7K@mail.gmail.com>
 <FB7E6664-F292-480F-8C4F-2B3CCAB9FFB7@yahoo-inc.com> <D3254862-5A94-4121-9B0B-9AE276D0955C@mac.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 4 Apr 2011 12:19:27 -0700
Message-ID: <BANLkTinND1e0rDZrPeke1WqCqDG_BPUhZg@mail.gmail.com>
Subject: Re: [VOTE] Abandon hdfsproxy HDFS contrib
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0022159751aa9d82fe04a01ca40f
X-Virus-Checked: Checked by ClamAV on apache.org

--0022159751aa9d82fe04a01ca40f
Content-Type: text/plain; charset=ISO-8859-1

Could those of you who -1ed the removal of HDFS Proxy please look into the
test that has been failing our Hudson build for the last several months:
https://issues.apache.org/jira/browse/HDFS-1666

<https://issues.apache.org/jira/browse/HDFS-1666>It is one thing to say that
we "should" maintain a piece of code, but it's another to actually maintain
it. In my mind, part of maintaining a project involves addressing consistent
test failures as high priority items.

-Todd

On Tue, Feb 22, 2011 at 9:27 PM, Nigel Daley <ndaley@mac.com> wrote:

> For closure, this vote fails due to a couple binding -1 votes.
>
> Nige
>
> On Feb 18, 2011, at 4:46 AM, Eric Baldeschwieler wrote:
>
> > Hi Bernd,
> >
> > Apache Hadoop is about scale. Most clusters will always be small, but
> Hadoop is going mainstream precisely because it scales to huge data and
> cluster sizes.
> >
> > There are lots of systems that work well on 10 node clusters. People
> select   Hadoop because they are confident that as their business / problem
> grows, Hadoop can grow with it.
> >
> > ---
> > E14 - via iPhone
> >
> > On Feb 17, 2011, at 7:25 AM, "Bernd Fondermann" <
> bernd.fondermann@googlemail.com> wrote:
> >
> >> On Thu, Feb 17, 2011 at 14:58, Ian Holsman <hadoop@holsman.net> wrote:
> >>> Hi Bernd.
> >>>
> >>> On Feb 17, 2011, at 7:43 AM, Bernd Fondermann wrote:
> >>>>
> >>>> We have the very unfortunate situation here at Hadoop where Apache
> >>>> Hadoop is not the primary and foremost place of Hadoop development.
> >>>> Instead, code is developed internally at Yahoo and then contributed in
> >>>> (smaller or larger) chunks to Hadoop.
> >>>
> >>> This has been the situation in the past,
> >>> but as you can see in the last month, this has changed.
> >>>
> >>> Yahoo! has publicly committed to move their development into the main
> code base, and you can see they have started doing this with the 20.100
> branch,
> >>> and their recent commits to trunk.
> >>> Combine this with Nige taking on the 0.22 release branch, (and
> sheperding it into a stable release) and I think we have are addressing your
> concerns.
> >>>
> >>> They have also started bringing the discussions back on the list, see
> the recent discussion about Jobtracker-nextgen Arun has re-started in
> MAPREDUCE-279.
> >>>
> >>> I'm not saying it's perfect, but I think the major players understand
> there is an issue, and they are *ALL* moving in the right direction.
> >>
> >> I enthusiastically would like to see your optimism be verified.
> >> Maybe I'm misreading the statements issued publicly, but I don't think
> >> that this is fully understood. I agree though that it's a move into
> >> the right direction.
> >>
> >>>> This is open source development upside down.
> >>>> It is not ok for people to diff ASF svn against their internal code
> >>>> and provide the diff as a patch without reviewing IP first for every
> >>>> line of code changed.
> >>>> For larger chunks I'd suggest to even go via the Incubator IP
> clearance process.
> >>>> Only then will we force committers to primarily work here in the open
> >>>> and return to what I'd consider a healthy project.
> >>>>
> >>>> To be honest: Hadoop is in the process of falling apart.
> >>>> Contrib Code gets moved out of Apache instead of being maintained
> here.
> >>>> Discussions are seldom consense-driven.
> >>>> Release branches stagnate.
> >>>
> >>> True. releases do take a long time. This is mainly due to it being
> extremely hard to test and verify that a release is stable.
> >>> It's not enough to just run the thing on 4 machines, you need at least
> 50 to test some of the major problems. This requires some serious $ for
> someone to verify.
> >>
> >> It has been proposed on the list before, IIRC. Don't know how to get
> >> there, but the project seriously needs access to a cluster of this
> >> size.
> >>
> >>>> Downstream projects like HBase don't get proper support.
> >>>> Production setups are made from 3rd party distributions.
> >>>> Development is not happening here, but elsewhere behind corporate
> doors.
> >>>> Discussion about future developments are started on corporate blogs (
> >>>>
> http://developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextgen/
> >>>> ) instead of on the proper mailing list.
> >>>> Hurdles for committing are way too high.
> >>>> On the bright side, new committers and PMC members are added, this is
> >>>> an improvement.
> >>>>
> >>>> I'd suggest to move away from relying on large code dumps from
> >>>> corporations, and move back to the ASF-proven "individual committer
> >>>> commits on trunk"-model where more committers can get involved.
> >>>> If that means not to support high end cluster sizes for some months,
> >>>> well, so be it.
> >>>
> >>>> Average committers cannot run - e.g. test - on high
> >>>> end cluster sizes. If that would mean they cannot participate, then
> >>>> the open source project better concentrate on small and medium sized
> >>>> cluster instead.
> >>>
> >>>
> >>> Well.. that's one approach.. but there are several companies out there
> who rely on apache's hadoop to power their large clusters, so I'd hate to
> see hadoop become something that only runs well on
> >>> 10-nodes.. as I don't think that will help anyone either.
> >>
> >> But only looking at high-end scale doesn't help either.
> >>
> >> Lets face the fact that Hadoop is now moving from early adaptors phase
> >> into a much broader market. I predict that small to medium sized
> >> clusters will be the majority of Hadoop deployments in a few month
> >> time. 4000, or even 500 machines is the high-end range. If the open
> >> source project Hadoop cannot support those users adequately (without
> >> becoming defunct), the committership might be better off to focus on
> >> the low-end and medium sized users.
> >>
> >> I'm not suggesting to turn away from the handfull (?) of high-end
> >> users. They certainly have most valuable input. But also, *they*
> >> obviously have the resources in terms of larger clusters and
> >> developers to deal with their specific setups. Obviously, they don't
> >> need to rely on the open source project to make releases. In fact,
> >> they *do* work on their own Hadoop derivatives.
> >> All the other users, the hundreds of boring small cluster users, don't
> >> have that choice. They *depend* on the open source releases.
> >>
> >> Hadoop is an Apache project, to provide HDFS and MR free of charge to
> >> the general public. Not only to me - nor to only one or two big
> >> companies either.
> >> Focus on all the users.
> >>
> >> Bernd
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

--0022159751aa9d82fe04a01ca40f--

From general-return-3227-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 05 20:01:22 2011
Return-Path: <general-return-3227-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 90738 invoked from network); 5 Apr 2011 20:01:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Apr 2011 20:01:21 -0000
Received: (qmail 30569 invoked by uid 500); 5 Apr 2011 20:01:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30485 invoked by uid 500); 5 Apr 2011 20:01:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30477 invoked by uid 99); 5 Apr 2011 20:01:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Apr 2011 20:01:18 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of tom.e.white@gmail.com designates 74.125.83.176 as permitted sender)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Apr 2011 20:01:13 +0000
Received: by pve37 with SMTP id 37so434290pve.35
        for <general@hadoop.apache.org>; Tue, 05 Apr 2011 13:00:53 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type;
        bh=A3vf6ccBRudTYB1aX8L/xhsNNTKfLVHhg+5XTu9YYTo=;
        b=HR4YlQvj0QyBwwGRpyXwkqYj8OcQ1VthRHGuEe8vvQH/gsQ6XYhRP9X3uVurcX7jCO
         YXCUeA0FBJ8vjXQgxb5GyT8voupznt78jpo5tElaWtoyYq6HKzcqpwRvBGv/7RkDPKMI
         FRqAtCasdLxBi8bwJaMQ/nz4zPSV9oQB37j0Q=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=pSDUJoaZ1sz9KaReOWlCjN0e0HHIgOxUgMSgTkow6kJMZubB2c+B2cGztAnZEl4L47
         spU023Wl2X31MKR5edEHM4Gg105KTJfd0320hHBvuMxQbeqQME1eE0x3im9WPDbCpEY+
         LGbstpskMJt6AVrQTRxRZQnYz1GFNw8/VZgiM=
Received: by 10.142.233.15 with SMTP id f15mr57157wfh.394.1302033653148; Tue,
 05 Apr 2011 13:00:53 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.68.47.68 with HTTP; Tue, 5 Apr 2011 13:00:33 -0700 (PDT)
In-Reply-To: <AANLkTi=owD6w2UA+-eCdDP67F=2NMChHLJxi1HYY+RG6@mail.gmail.com>
References: <AANLkTi=owD6w2UA+-eCdDP67F=2NMChHLJxi1HYY+RG6@mail.gmail.com>
From: Tom White <tom.e.white@gmail.com>
Date: Tue, 5 Apr 2011 13:00:33 -0700
Message-ID: <BANLkTim7sijRYVRmXx3LymEfskzBJL978w@mail.gmail.com>
Subject: Re: Maintenance of Hadoop 0.21 branch?
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

I'm not currently planning on doing a 0.21.1 release. I agree with
Todd that it would be better to spend time getting 0.22 ready for a
release.

Cheers,
Tom

On Wed, Mar 30, 2011 at 2:41 PM, Todd Lipcon <todd@cloudera.com> wrote:
> Hi all,
>
> Some recent discussion on HDFS-1786 has raised an interesting question: does
> anyone plan on maintaining the 0.21 branch and eventually releasing an
> 0.21.1? Should we bother to commit bug fixes to this branch?
>
> It seems to me that our time would be better spent getting 0.22 and trunk
> back to a green state so we can talk about releasing them, rather than
> applying patches to a branch with no releases planned.
>
> Of course a decision now doesn't preclude anyone from stepping up later,
> backporting patches to 0.21, and releasing an 0.21.1.
>
> Thanks
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

From general-return-3228-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 06 20:19:02 2011
Return-Path: <general-return-3228-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56182 invoked from network); 6 Apr 2011 20:19:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Apr 2011 20:19:00 -0000
Received: (qmail 18279 invoked by uid 500); 6 Apr 2011 20:18:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17877 invoked by uid 500); 6 Apr 2011 20:18:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17793 invoked by uid 99); 6 Apr 2011 20:18:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Apr 2011 20:18:43 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Apr 2011 20:18:39 +0000
Received: by pve37 with SMTP id 37so1043375pve.35
        for <general@hadoop.apache.org>; Wed, 06 Apr 2011 13:18:18 -0700 (PDT)
Received: by 10.142.76.16 with SMTP id y16mr47277wfa.157.1302121098149; Wed,
 06 Apr 2011 13:18:18 -0700 (PDT)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.68.52.97 with HTTP; Wed, 6 Apr 2011 13:17:58 -0700 (PDT)
In-Reply-To: <AANLkTika3WVVhJKXNQ2O4Kfd16jamiT5k2RJSy=Yi8ao@mail.gmail.com>
References: <FF258184-01BE-419F-B5A0-69051472BBDB@Holsman.net>
 <AANLkTimsma0e9cGAN0o_zz4dBDNRnkDXEgKcVx00zbEV@mail.gmail.com>
 <597116E4-1991-43A6-B35B-BAA9F577897E@Holsman.NET> <AANLkTi=cPCdv1ipMSN41Cpabhg_jbSZvDRh8_BzyUFR_@mail.gmail.com>
 <AANLkTika3WVVhJKXNQ2O4Kfd16jamiT5k2RJSy=Yi8ao@mail.gmail.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Wed, 6 Apr 2011 13:17:58 -0700
X-Google-Sender-Auth: LHUTyWKSFTgO78JKXPBBgWRMfGg
Message-ID: <BANLkTim5wm0oupnJ5VNnKXj7z__64H6WQw@mail.gmail.com>
Subject: Re: Apache Sonar - http://analysis.apache.org/ does anyone want
 hadoop in there?
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I have submitted for enabling Sonar analysis on HDFS code. Would
appreciate if someone takes a look so we can move forward with it
(HDFS-1741).
--
=A0 Take care,
Konstantin (Cos) Boudnik

On Wed, Mar 9, 2011 at 08:30, Konstantin Boudnik <cos@apache.org> wrote:
> I have looked around Apache's Sonar server and it seems to be
> sufficient to just have a correct pom.xml in place. I have opened
> =A0https://issues.apache.org/jira/browse/HDFS-1741
> Will attend to it shortly.
> --
> =A0 Take care,
> Konstantin (Cos) Boudnik
>
> On Wed, Mar 9, 2011 at 08:06, Konstantin Boudnik <cos@apache.org> wrote:
>> Hi Ian.
>>
>> yes, Sonar needs to have a pom'ed project. In case of an ant build
>> such a pom can be created to 'wrap' around build.xml and is usually
>> quite simple to manufacture. I have done it in the past for some HDFS
>> experiments without much of a hassle.
>>
>> Here's the example:
>> =A0http://docs.codehaus.org/display/SONAR/Analyse+with+Maven
>> in "Analyze projects which are not built with Maven" section.
>>
>> There's one more caveat though: handling of unit tests reports. It can
>> be done with some new features Sonar has. See here:
>> =A0http://www.sonarsource.org/tag/ant/
>>
>> I can quickly provide a working fixture to add say HDFS into Sonar as
>> a starter. Pl. let me know where it needs to be put (perhaps a JIRA
>> with a patch would do to?).
>>
>> Thanks for looking into this,
>> =A0Cos
>>
>> On Wed, Mar 9, 2011 at 06:10, Ian Holsman <hadoop@holsman.net> wrote:
>>> Hi Cos.
>>> apparently they can't build from ant in the current version.
>>> I'll need to have a look at it.. they sent me some info.
>>>
>>> On Mar 3, 2011, at 12:34 PM, Konstantin Boudnik wrote:
>>>
>>>> Ian,
>>>>
>>>> Sonar was around for a while and Apache has this setup since late
>>>> 2009, I believe. =A0I had asked =A0INFRA folks on a few occasions abou=
t
>>>> adding Hadoop repos to their installation, but never heard anything
>>>> back.
>>>>
>>>> I'd be awesome if you can talk to the right people so it'd happen even=
tually.
>>>> --
>>>> =A0 Thanks in advance,
>>>> Konstantin (Cos) Boudnik
>>>>
>>>>
>>>> On Thu, Mar 3, 2011 at 05:59, Ian Holsman <ian@holsman.net> wrote:
>>>>> I just discovered this Apache site,
>>>>> Apache Sonar performs source code analysis on the java code, and it l=
ooks pretty, and I'm sure some people would find it useful.
>>>>>
>>>>> If people are interested I can start putting finding out how to put t=
he hadoop code in there, so we can get stats on our own code.
>>>>>
>>>>> Regards
>>>>> Ian
>>>>> --
>>>>> Ian Holsman
>>>>> Ian@Holsman.net
>>>>> PH: +1-703 879-3128 AOLIM: ianholsman Skype:iholsman
>>>>>
>>>>> To know recursion, you must first know recursion.
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>>
>>
>

From general-return-3229-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 07 20:21:58 2011
Return-Path: <general-return-3229-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88470 invoked from network); 7 Apr 2011 20:21:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Apr 2011 20:21:58 -0000
Received: (qmail 1941 invoked by uid 500); 7 Apr 2011 20:21:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1813 invoked by uid 500); 7 Apr 2011 20:21:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 25916 invoked by uid 99); 7 Apr 2011 17:44:27 -0000
X-ASF-Spam-Status: No, hits=3.3 required=5.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1302198205;
	bh=w+eKsQcGSouHJG55RVY/3KOqP2Ekp5cfa8NJjh6fBaA=;
	h=From:To:Date:Subject:Message-ID:Content-Type:MIME-Version;
	b=TDxi0wEAw1Jc1FrSOK3WfFSqAz7nMGZy+W6XJwvS7QBRkg3RSZomoqLAhzKBHA721
	 8SoCkLbcap2HSg3rl//h1I92ydP5h89uKgmfeqljGGw96z/awsvFVbRI1pbCY94HmV
	 FZRcOUYpLhPpxf6W1thppHNovvzsKvxi/21TWS5M=
From: Avik Dey <avik@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 7 Apr 2011 10:43:23 -0700
Subject: [hadoop] Hadoop Summit 2011 by Yahoo!: June 29th, Santa Clara
 Convention Center. Register and submit abstract for presentation:
 www.hadoopsummit.org
Thread-Topic: [hadoop] Hadoop Summit 2011 by Yahoo!: June 29th, Santa Clara
 Convention Center. Register and submit abstract for presentation:
 www.hadoopsummit.org
Thread-Index: Acv00YctyjaYKtraRSmrGT2TejdLtw==
Message-ID: <2931D8EDF4B12743B93584669F76353330982C0E5D@SP1-EX07VS02.ds.corp.yahoo.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_2931D8EDF4B12743B93584669F76353330982C0E5DSP1EX07VS02ds_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_2931D8EDF4B12743B93584669F76353330982C0E5DSP1EX07VS02ds_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi All,

Yahoo! is proud to host the 4th annual Hadoop Summit 2011: June 29th, Santa=
 Clara Convention Center, California.

Registration and Call for Presentations are now open at: http://www.hadoops=
ummit.org<http://www.hadoopsummit.org/>

This year, we have less keynotes and more tech talks, even in the morning s=
ession. The afternoon session is tech talks only, followed by a panel discu=
ssion, and wrapping up with an evening reception. Details to be posted soon=
.

For the afternoon session we still have three tracks. However, based on int=
erest from the community we have tuned their focus a bit.

The three tracks now are:


*         Community: Presentations and discussions about Hadoop roadmap, co=
ntribution and best practices.


*         Operations and Management: Presentations and discussions about op=
erations and management of Hadoop clusters and best practices.


*         Applications and Research: Case studies about innovative applicat=
ions and Hadoop technologies based research studies.

This is a terrific opportunity for you to share your experience with thousa=
nds of Hadoop users by submitting a session abstract  for one of the tracks=
 above.

Submission deadline is May 6th, here is the URL to submit your abstract:

http://www.easychair.org/conferences/?conf=3Dhs-2011

Track Co-Chairs this year who will be reviewing your abstracts are:

*         Andreas Neumann - Yahoo!

*         Benjamin Reed - Yahoo!

*         Dhruba Borthakur - Facebook

*         Michael Stack - StumbleUpon

*         Milind Bhandarkar - Linkedin

*         Owen O'Malley - Yahoo!

With a great, diverse team of track co-chairs, we are looking forward to ev=
en more interesting presentations and discussions this year!

Register now<http://www.hadoopsummit.org/>, space is limited!

Looking forward to meeting you at the Summit.

Avik


--_000_2931D8EDF4B12743B93584669F76353330982C0E5DSP1EX07VS02ds_--

From general-return-3230-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 07 22:54:16 2011
Return-Path: <general-return-3230-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6027 invoked from network); 7 Apr 2011 22:54:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Apr 2011 22:54:15 -0000
Received: (qmail 42736 invoked by uid 500); 7 Apr 2011 22:54:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42696 invoked by uid 500); 7 Apr 2011 22:54:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42687 invoked by uid 99); 7 Apr 2011 22:54:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Apr 2011 22:54:14 +0000
X-ASF-Spam-Status: No, hits=2.9 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Apr 2011 22:54:07 +0000
Received: from l2tp-8-240.corp.yahoo.com (l2tp-8-240.corp.yahoo.com [10.73.8.240])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p37MqixH040056
	for <general@hadoop.apache.org>; Thu, 7 Apr 2011 15:52:45 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1302216765;
	bh=rBfUPF8k3rzc+pMxDZJIxwQG5PkW/RfytJExQymYC2c=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:Mime-Version:Subject:
	 Date:References;
	b=auJThgfRl847XfeRjYH8mUDV3rmFVuE4kcdj48jRiRpZnS8HZKRjECIl19bn5mmiJ
	 xTpRrJh8KSFbJSBIRClme/uXaPY/s4PJeA1eMFcjo2h6M1Ijg5sziuYEvAiKj7YGBe
	 /UqOTPDaTjfY9hC1WNgsYqJ+OkRWjQjp4GMg4onk=
Message-Id: <41978952-FE14-4CD9-8C39-BB3B0E458D55@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <75192454-0158-4C84-9F79-337A16AA913E@yahoo-inc.com>
Content-Type: multipart/alternative; boundary=Apple-Mail-25--360465881
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [ANNOUNCEMENT] Yahoo focusing on Apache Hadoop, discontinuing "The Yahoo Distribution of Hadoop"
Date: Thu, 7 Apr 2011 15:52:44 -0700
References: <55BA39F5-5017-42B1-BE25-A643CCEADC75@yahoo-inc.com> <354259E0-9D22-47B5-B247-4187182B60DF@apache.org> <75192454-0158-4C84-9F79-337A16AA913E@yahoo-inc.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-25--360465881
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


On Feb 14, 2011, at 1:34 PM, Arun C Murthy wrote:
>
> As the final installment in this process, I've started a discussion on
> us contributing a re-factor of Map-Reduce in https://issues.apache.org/jira/browse/MAPREDUCE-279
> .



Hi Folks,

We wanted to share our thoughts around the co-development of the  
NextGen MapReduce branch (Jira MR-279), maintaining the branch-0.20- 
security and merging the work on the security branch with trunk.   
We've concluded that it does not make sense for us to port a very  
small subset of the work from the branch-0.20-security to the Hadoop  
mainline.  The JIRAs we don't plan to port all effect areas of the  
mainline that are going to be replaced by work in the NextGen  
MapReduce branch (http://svn.apache.org/viewvc/hadoop/mapreduce/branches/MR-279/ 
).

We've been working on the NextGen MapReduce branch (MAPREDUCE-279)  
within Apache for a while now and are excited about it's progress.  We  
think that this branch will be a huge improvement in scalability,  
performance and functionality.  We are now confident that we can get  
it ready for release in in the next few months.  We believe that the  
next major release of Apache Hadoop we will test at Yahoo will include  
the work in this branch and we are committed to merging the NextGen  
branch into the mainline after the PMC approves the merge.

Meanwhile, we have continued to find and fix bugs on branch-0.20- 
security and have been working to port that work into the Hadoop  
mainline.  Most of this work is done and we've also brought all the  
patches in from our github branch into apache subversion, so that it  
is easy for everyone to see the work remaining.  What we've found is  
that some of the work in branch-0.20-security is in code sections that  
have been completely replaced / refactored in the NextGen MapReduce  
branch.  Since we are committed to the NextGen branch, we don't think  
there is any upside in porting this code into portions of mainline we  
expect to discard. All of these JIRAs will be fixed in the NextGen  
MapReduce branch and through there ultimately in trunk (assuming the  
PMC approves the merge).

So at this point it is our intent to not port the JIRAs listed above  
to trunk, but to wait until we merge NextGen into trunk to resolve  
these issues there.  If you are interested in seeing these issues  
ported to mainline, let us know.  We are happy to help review your  
patches and explain context to anyone who is interested in doing this  
work.

Arun and Eric

--Apple-Mail-25--360465881--

From general-return-3231-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 07 23:23:41 2011
Return-Path: <general-return-3231-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 86835 invoked from network); 7 Apr 2011 23:23:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Apr 2011 23:23:40 -0000
Received: (qmail 61816 invoked by uid 500); 7 Apr 2011 23:23:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61745 invoked by uid 500); 7 Apr 2011 23:23:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61736 invoked by uid 99); 7 Apr 2011 23:23:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Apr 2011 23:23:39 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Apr 2011 23:23:32 +0000
Received: by ewy22 with SMTP id 22so1260357ewy.35
        for <general@hadoop.apache.org>; Thu, 07 Apr 2011 16:23:12 -0700 (PDT)
Received: by 10.14.45.157 with SMTP id p29mr672176eeb.245.1302218592191; Thu,
 07 Apr 2011 16:23:12 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.14.119.199 with HTTP; Thu, 7 Apr 2011 16:22:52 -0700 (PDT)
In-Reply-To: <41978952-FE14-4CD9-8C39-BB3B0E458D55@yahoo-inc.com>
References: <55BA39F5-5017-42B1-BE25-A643CCEADC75@yahoo-inc.com>
 <354259E0-9D22-47B5-B247-4187182B60DF@apache.org> <75192454-0158-4C84-9F79-337A16AA913E@yahoo-inc.com>
 <41978952-FE14-4CD9-8C39-BB3B0E458D55@yahoo-inc.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Thu, 7 Apr 2011 16:22:52 -0700
Message-ID: <BANLkTikFDcqrMyFjGUBoF-r73HkmVVCgQw@mail.gmail.com>
Subject: Re: [ANNOUNCEMENT] Yahoo focusing on Apache Hadoop, discontinuing
 "The Yahoo Distribution of Hadoop"
To: general@hadoop.apache.org
Cc: Arun C Murthy <acm@yahoo-inc.com>
Content-Type: multipart/alternative; boundary=0016e656850e9b84d804a05c64f4
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e656850e9b84d804a05c64f4
Content-Type: text/plain; charset=ISO-8859-1

Is there a list available of which patches you've made this decision about?
I'm curious, for example, about MAPREDUCE-2178 -- as of today, the MR
security in trunk has a serious vulnerability. Do we plan on fixing it, or
will the answer be that, if anyone needs security, they must update to "MR
Next Gen"?

-Todd

On Thu, Apr 7, 2011 at 3:52 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:

>
> On Feb 14, 2011, at 1:34 PM, Arun C Murthy wrote:
>
>>
>> As the final installment in this process, I've started a discussion on
>> us contributing a re-factor of Map-Reduce in
>> https://issues.apache.org/jira/browse/MAPREDUCE-279
>> .
>>
>
>
>
> Hi Folks,
>
> We wanted to share our thoughts around the co-development of the NextGen
> MapReduce branch (Jira MR-279), maintaining the branch-0.20-security and
> merging the work on the security branch with trunk.  We've concluded that it
> does not make sense for us to port a very small subset of the work from the
> branch-0.20-security to the Hadoop mainline.  The JIRAs we don't plan to
> port all effect areas of the mainline that are going to be replaced by work
> in the NextGen MapReduce branch (
> http://svn.apache.org/viewvc/hadoop/mapreduce/branches/MR-279/).
>
> We've been working on the NextGen MapReduce branch (MAPREDUCE-279) within
> Apache for a while now and are excited about it's progress.  We think that
> this branch will be a huge improvement in scalability, performance and
> functionality.  We are now confident that we can get it ready for release in
> in the next few months.  We believe that the next major release of Apache
> Hadoop we will test at Yahoo will include the work in this branch and we are
> committed to merging the NextGen branch into the mainline after the PMC
> approves the merge.
>
> Meanwhile, we have continued to find and fix bugs on branch-0.20-security
> and have been working to port that work into the Hadoop mainline.  Most of
> this work is done and we've also brought all the patches in from our github
> branch into apache subversion, so that it is easy for everyone to see the
> work remaining.  What we've found is that some of the work in
> branch-0.20-security is in code sections that have been completely replaced
> / refactored in the NextGen MapReduce branch.  Since we are committed to the
> NextGen branch, we don't think there is any upside in porting this code into
> portions of mainline we expect to discard. All of these JIRAs will be fixed
> in the NextGen MapReduce branch and through there ultimately in trunk
> (assuming the PMC approves the merge).
>
> So at this point it is our intent to not port the JIRAs listed above to
> trunk, but to wait until we merge NextGen into trunk to resolve these issues
> there.  If you are interested in seeing these issues ported to mainline, let
> us know.  We are happy to help review your patches and explain context to
> anyone who is interested in doing this work.
>
> Arun and Eric
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--0016e656850e9b84d804a05c64f4--

From general-return-3232-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 08 14:45:01 2011
Return-Path: <general-return-3232-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 86402 invoked from network); 8 Apr 2011 14:45:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Apr 2011 14:45:00 -0000
Received: (qmail 85848 invoked by uid 500); 8 Apr 2011 14:44:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85782 invoked by uid 500); 8 Apr 2011 14:44:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85774 invoked by uid 99); 8 Apr 2011 14:44:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Apr 2011 14:44:58 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of krjeschke@omniti.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Apr 2011 14:44:52 +0000
Received: by wwi18 with SMTP id 18so4748586wwi.29
        for <general@hadoop.apache.org>; Fri, 08 Apr 2011 07:44:32 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.145.134 with SMTP id p6mr1956466wej.112.1302273872250;
 Fri, 08 Apr 2011 07:44:32 -0700 (PDT)
Received: by 10.216.242.141 with HTTP; Fri, 8 Apr 2011 07:44:32 -0700 (PDT)
Date: Fri, 8 Apr 2011 10:44:32 -0400
Message-ID: <BANLkTinZBpAnVYQjsUkGyv8+7gg=GF4jHw@mail.gmail.com>
Subject: Surge 2011 CFP Deadline Extended
From: Katherine Jeschke <krjeschke@omniti.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d647f08e486604a0694319
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d647f08e486604a0694319
Content-Type: text/plain; charset=ISO-8859-1

OmniTI is pleased to announce that the CFP deadline for Surge 2011, the
Scalability and Performance Conference, (Baltimore: Sept 28-30, 2011) has
been extended to 23:59:59 EDT, April 17, 2011. The event focuses upon case
studies that demonstrate successes (and failures) in Web applications and
Internet architectures. New this year: Hack Day and Unconference on
September 28th.

For information about topics: http://omniti.com/surge/2011. Get inspired by
the 2010 sessions, now online at (http://omniti.com/surge/2010).

2010 attendees compared Surge to the early days of Velocity, and our
speakers received 3.5-4 out of 4 stars for quality of presentation and
quality of content! Nearly 90% of first-year attendees are planning to come
again in 2011.

For more information about the CFP or sponsorship of the event, please
contact us: surge (AT) omniti (DOT) com.


-- 
Katherine Jeschke
Marketing Director
OmniTI Computer Consulting, Inc.
7070 Samuel Morse Drive, Ste.150
Columbia, MD 21046
O: 410/872-4910, 222
C: 443/643-6140
omniti.com
circonus.com

--0016e6d647f08e486604a0694319--

From general-return-3233-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 08 17:35:21 2011
Return-Path: <general-return-3233-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76901 invoked from network); 8 Apr 2011 17:35:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Apr 2011 17:35:20 -0000
Received: (qmail 69161 invoked by uid 500); 8 Apr 2011 17:35:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69096 invoked by uid 500); 8 Apr 2011 17:35:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69088 invoked by uid 99); 8 Apr 2011 17:35:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Apr 2011 17:35:18 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Apr 2011 17:35:13 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p38HYKPJ029943
	for <general@hadoop.apache.org>; Fri, 8 Apr 2011 10:34:20 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1302284060;
	bh=SXmjn18mHO2/NY5iDO3SZiGCgdNdWsMMqaQUykRrJC0=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=Pz22z4OvPFgBW5nYRI8UbfLlTSWlzIPDZxma31cEC3jMPTIuwu0qTPXO0L0mSPov7
	 RO7hTMbvsybUKalrl51b+u+iPvEQ9KT1WaGrwV5f4zOLQdK6/Btenlg8Wzxq/QJDnx
	 dTYtI+nfa7monGqT47NQyWDtGcF1Z97LVhdpCPYE=
Message-Id: <7CCF44E2-EE8D-4A5E-BE5B-5163B37D2342@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <BANLkTikFDcqrMyFjGUBoF-r73HkmVVCgQw@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [ANNOUNCEMENT] Yahoo focusing on Apache Hadoop, discontinuing "The Yahoo Distribution of Hadoop"
Date: Fri, 8 Apr 2011 10:34:22 -0700
References: <55BA39F5-5017-42B1-BE25-A643CCEADC75@yahoo-inc.com> <354259E0-9D22-47B5-B247-4187182B60DF@apache.org> <75192454-0158-4C84-9F79-337A16AA913E@yahoo-inc.com> <41978952-FE14-4CD9-8C39-BB3B0E458D55@yahoo-inc.com> <BANLkTikFDcqrMyFjGUBoF-r73HkmVVCgQw@mail.gmail.com>
X-Mailer: Apple Mail (2.936)

Todd,

On Apr 7, 2011, at 4:22 PM, Todd Lipcon wrote:

> Is there a list available of which patches you've made this decision  
> about? I'm curious, for example, about MAPREDUCE-2178 -- as of  
> today, the MR security in trunk has a serious vulnerability. Do we  
> plan on fixing it, or will the answer be that, if anyone needs  
> security, they must update to "MR Next Gen"?

Apologies if my original message was abstruse - I want to ensure that  
there is no confusion between 'forward-port' and 'merge from yahoo- 
merge branch'.

Let me try to explain again: there are several forward ports from the  
hadoop-0.20-2xx (branch-0.20-security) which are complete, including  
MAPREDUCE-2178. They are currently part of the 'yahoo-merge' branch in  
MapReduce. These are awaiting a merge into trunk. Trunk (with a few  
merges from yahoo-merge) will have a complete security implementation.

My message was intended to highlight some small number of features/ 
bugs which are/will-be in hadoop-0.20.2xx. Here is a nearly complete  
list of such jiras: MAPREDUCE-517, MAPREDUCE-1872, MAPREDUCE-291,  
MAPREDUCE-2418, MAPREDUCE-2409, MAPREDUCE-2411. I'll check to ensure  
there aren't others.	

Hope that makes sense. Again, apologies for any confusion I've caused.

thanks,
Arun


From general-return-3234-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 08 17:40:12 2011
Return-Path: <general-return-3234-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83938 invoked from network); 8 Apr 2011 17:40:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Apr 2011 17:40:12 -0000
Received: (qmail 76004 invoked by uid 500); 8 Apr 2011 17:40:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75867 invoked by uid 500); 8 Apr 2011 17:40:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75850 invoked by uid 99); 8 Apr 2011 17:40:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Apr 2011 17:40:08 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Apr 2011 17:40:04 +0000
Received: by fxm7 with SMTP id 7so4136995fxm.35
        for <general@hadoop.apache.org>; Fri, 08 Apr 2011 10:39:42 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:date:x-google-sender-auth
         :message-id:subject:from:to:content-type;
        bh=j+RmhaJHdjjNqxQIfv1qkIAsEkhYaiUl1ForBbA//qM=;
        b=h0WXcDDQ0J6OJwmRY6C7ZFWqrINvbwZI/7azUIVQZEMrXj3p89DwTCdDbSCjfY/bFG
         psy6rsk3cJLZqyi0AwLHDlLkLxbc240YlrSDa77H9bve2LStgMc8rev6wKS7XgkTXtL0
         +sezgRNxUMhI5JmWrBXvA3Nv10XZM+s+jmP5c=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        b=IXEdAc0EMihyYMqn2RT1aHDw1BYQjhb3d7ScbfjdSfUTd/V27REUDJKjMz2Cozhhzi
         CKJZUuvDD8HGhIN90MxHGRas/R9eazEhSMQe88X3LyzW9OXs1cSaQJkkJTRVQ8BH2ntc
         Q4py0g2PyR5aSNFjFmLBDXL7Z/8u5EgDgMS58=
MIME-Version: 1.0
Received: by 10.223.25.197 with SMTP id a5mr2604004fac.29.1302284382757; Fri,
 08 Apr 2011 10:39:42 -0700 (PDT)
Sender: saint.ack@gmail.com
Received: by 10.223.96.135 with HTTP; Fri, 8 Apr 2011 10:39:42 -0700 (PDT)
Date: Fri, 8 Apr 2011 10:39:42 -0700
X-Google-Sender-Auth: yobYcmI-1EYGL1tEjvPawRQnJVI
Message-ID: <BANLkTincvMXiCdEn=vjMCu5xg7d_1Syk8g@mail.gmail.com>
Subject: ANN: HBase 0.90.2 is available for download
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org, Hbase-User <user@hbase.apache.org>
Content-Type: text/plain; charset=ISO-8859-1

The Apache HBase team is happy to announce the general availability of HBase
0.90.2, available from your Apache mirror of choice:

 http://www.apache.org/dyn/closer.cgi/hbase/

HBase 0.90.2 is a maintenance release that fixes several important bugs
since version 0.90.1, while retaining API and data compatibility. The
release notes may be found on the Apache JIRA:

  http://su.pr/1F6omq

Users upgrading from HBase 0.90.0 or 0.90.1 may upgrade clients and servers
separately, though it is recommended that both be upgraded. If upgrading
from a version of HBase prior to 0.90.0, please read the notes accompanying
that release:

  http://osdir.com/ml/general-hadoop-apache/2011-01/msg00208.html

As always, many thanks to those who contributed to this release!

-The HBase Team

From general-return-3235-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 08 18:09:25 2011
Return-Path: <general-return-3235-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87790 invoked from network); 8 Apr 2011 18:09:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Apr 2011 18:09:25 -0000
Received: (qmail 19561 invoked by uid 500); 8 Apr 2011 18:09:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19515 invoked by uid 500); 8 Apr 2011 18:09:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19506 invoked by uid 99); 8 Apr 2011 18:09:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Apr 2011 18:09:23 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates 209.85.215.176 as permitted sender)
Received: from [209.85.215.176] (HELO mail-ey0-f176.google.com) (209.85.215.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Apr 2011 18:09:18 +0000
Received: by eya28 with SMTP id 28so1557375eya.35
        for <general@hadoop.apache.org>; Fri, 08 Apr 2011 11:08:57 -0700 (PDT)
Received: by 10.14.32.13 with SMTP id n13mr1164601eea.21.1302286137188; Fri,
 08 Apr 2011 11:08:57 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.14.119.199 with HTTP; Fri, 8 Apr 2011 11:08:37 -0700 (PDT)
In-Reply-To: <7CCF44E2-EE8D-4A5E-BE5B-5163B37D2342@yahoo-inc.com>
References: <55BA39F5-5017-42B1-BE25-A643CCEADC75@yahoo-inc.com>
 <354259E0-9D22-47B5-B247-4187182B60DF@apache.org> <75192454-0158-4C84-9F79-337A16AA913E@yahoo-inc.com>
 <41978952-FE14-4CD9-8C39-BB3B0E458D55@yahoo-inc.com> <BANLkTikFDcqrMyFjGUBoF-r73HkmVVCgQw@mail.gmail.com>
 <7CCF44E2-EE8D-4A5E-BE5B-5163B37D2342@yahoo-inc.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Fri, 8 Apr 2011 11:08:37 -0700
Message-ID: <BANLkTinqaNunCOaUgO1BaMp_0TUjtFP9vw@mail.gmail.com>
Subject: Re: [ANNOUNCEMENT] Yahoo focusing on Apache Hadoop, discontinuing
 "The Yahoo Distribution of Hadoop"
To: general@hadoop.apache.org
Cc: Arun C Murthy <acm@yahoo-inc.com>
Content-Type: multipart/alternative; boundary=00235445b9509a603b04a06c1e1b

--00235445b9509a603b04a06c1e1b
Content-Type: text/plain; charset=ISO-8859-1

On Fri, Apr 8, 2011 at 10:34 AM, Arun C Murthy <acm@yahoo-inc.com> wrote:

>
> On Apr 7, 2011, at 4:22 PM, Todd Lipcon wrote:
>
>  Is there a list available of which patches you've made this decision
>> about? I'm curious, for example, about MAPREDUCE-2178 -- as of today, the MR
>> security in trunk has a serious vulnerability. Do we plan on fixing it, or
>> will the answer be that, if anyone needs security, they must update to "MR
>> Next Gen"?
>>
>
> Apologies if my original message was abstruse - I want to ensure that there
> is no confusion between 'forward-port' and 'merge from yahoo-merge branch'.
>
> Let me try to explain again: there are several forward ports from the
> hadoop-0.20-2xx (branch-0.20-security) which are complete, including
> MAPREDUCE-2178. They are currently part of the 'yahoo-merge' branch in
> MapReduce. These are awaiting a merge into trunk. Trunk (with a few merges
> from yahoo-merge) will have a complete security implementation.
>

Ah, OK, I see. That makes sense.


>
> My message was intended to highlight some small number of features/bugs
> which are/will-be in hadoop-0.20.2xx. Here is a nearly complete list of such
> jiras: MAPREDUCE-517, MAPREDUCE-1872, MAPREDUCE-291, MAPREDUCE-2418,
> MAPREDUCE-2409, MAPREDUCE-2411. I'll check to ensure there aren't others.
>
>
>
Looking briefly at those, it seems that the ones that are clear bugs (with
small fixes) should be put in the current MR implementation:
MAPREDUCE-2411
MAPREDUCE-2409
MAPREDUCE-2418 (maybe)

These all have patches that are pretty small, and I'd imagine would apply
pretty easily to trunk. Let me know if you'd like any help forward-porting.

The other ones, as new features/improvements, I'd agree it makes sense not
to waste effort re-implementing them for trunk MR, but rather to make sure
they're incorporated in next-gen.

-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

--00235445b9509a603b04a06c1e1b--

From general-return-3236-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 08 21:21:19 2011
Return-Path: <general-return-3236-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5194 invoked from network); 8 Apr 2011 21:21:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Apr 2011 21:21:18 -0000
Received: (qmail 21491 invoked by uid 500); 8 Apr 2011 21:21:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21418 invoked by uid 500); 8 Apr 2011 21:21:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21410 invoked by uid 99); 8 Apr 2011 21:21:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Apr 2011 21:21:17 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Apr 2011 21:21:09 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p38LKPbI002073
	for <general@hadoop.apache.org>; Fri, 8 Apr 2011 14:20:26 -0700 (PDT)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by
 SP2-EX07CAS01.ds.corp.yahoo.com ([98.137.59.37]) with mapi; Fri, 8 Apr 2011
 14:20:25 -0700
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 8 Apr 2011 14:20:15 -0700
Subject: Re: [ANNOUNCEMENT] Yahoo focusing on Apache Hadoop, discontinuing
 "The Yahoo Distribution of Hadoop"
Thread-Topic: [ANNOUNCEMENT] Yahoo focusing on Apache Hadoop, discontinuing
 "The Yahoo Distribution of Hadoop"
Thread-Index: Acv2MsYpxOy5hhjfRnav21625sQrFQ==
Message-ID: <64807CFA-0C26-4B8E-A81F-5922DF6D6E39@yahoo-inc.com>
References: <55BA39F5-5017-42B1-BE25-A643CCEADC75@yahoo-inc.com>
 <354259E0-9D22-47B5-B247-4187182B60DF@apache.org>
 <75192454-0158-4C84-9F79-337A16AA913E@yahoo-inc.com>
 <41978952-FE14-4CD9-8C39-BB3B0E458D55@yahoo-inc.com>
 <BANLkTikFDcqrMyFjGUBoF-r73HkmVVCgQw@mail.gmail.com>
 <7CCF44E2-EE8D-4A5E-BE5B-5163B37D2342@yahoo-inc.com>
 <BANLkTinqaNunCOaUgO1BaMp_0TUjtFP9vw@mail.gmail.com>
In-Reply-To: <BANLkTinqaNunCOaUgO1BaMp_0TUjtFP9vw@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks Todd, your help with the jiras you IDed would be welcome!

---
E14 - typing on glass

On Apr 8, 2011, at 11:09 AM, "Todd Lipcon" <todd@cloudera.com> wrote:

> On Fri, Apr 8, 2011 at 10:34 AM, Arun C Murthy <acm@yahoo-inc.com> wrote:
>=20
>>=20
>> On Apr 7, 2011, at 4:22 PM, Todd Lipcon wrote:
>>=20
>> Is there a list available of which patches you've made this decision
>>> about? I'm curious, for example, about MAPREDUCE-2178 -- as of today, t=
he MR
>>> security in trunk has a serious vulnerability. Do we plan on fixing it,=
 or
>>> will the answer be that, if anyone needs security, they must update to =
"MR
>>> Next Gen"?
>>>=20
>>=20
>> Apologies if my original message was abstruse - I want to ensure that th=
ere
>> is no confusion between 'forward-port' and 'merge from yahoo-merge branc=
h'.
>>=20
>> Let me try to explain again: there are several forward ports from the
>> hadoop-0.20-2xx (branch-0.20-security) which are complete, including
>> MAPREDUCE-2178. They are currently part of the 'yahoo-merge' branch in
>> MapReduce. These are awaiting a merge into trunk. Trunk (with a few merg=
es
>> from yahoo-merge) will have a complete security implementation.
>>=20
>=20
> Ah, OK, I see. That makes sense.
>=20
>=20
>>=20
>> My message was intended to highlight some small number of features/bugs
>> which are/will-be in hadoop-0.20.2xx. Here is a nearly complete list of =
such
>> jiras: MAPREDUCE-517, MAPREDUCE-1872, MAPREDUCE-291, MAPREDUCE-2418,
>> MAPREDUCE-2409, MAPREDUCE-2411. I'll check to ensure there aren't others=
.
>>=20
>>=20
>>=20
> Looking briefly at those, it seems that the ones that are clear bugs (wit=
h
> small fixes) should be put in the current MR implementation:
> MAPREDUCE-2411
> MAPREDUCE-2409
> MAPREDUCE-2418 (maybe)
>=20
> These all have patches that are pretty small, and I'd imagine would apply
> pretty easily to trunk. Let me know if you'd like any help forward-portin=
g.
>=20
> The other ones, as new features/improvements, I'd agree it makes sense no=
t
> to waste effort re-implementing them for trunk MR, but rather to make sur=
e
> they're incorporated in next-gen.
>=20
> -Todd
> --=20
> Todd Lipcon
> Software Engineer, Cloudera

From general-return-3237-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 08 21:41:02 2011
Return-Path: <general-return-3237-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82571 invoked from network); 8 Apr 2011 21:41:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Apr 2011 21:41:02 -0000
Received: (qmail 57005 invoked by uid 500); 8 Apr 2011 21:41:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56947 invoked by uid 500); 8 Apr 2011 21:41:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56939 invoked by uid 99); 8 Apr 2011 21:41:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Apr 2011 21:41:00 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Apr 2011 21:40:56 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p38LeJpJ084563
	for <general@hadoop.apache.org>; Fri, 8 Apr 2011 14:40:19 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1302298819;
	bh=gMauk1+TuP8VJ/T8JWQ3lFGEXBDK+vlitGND0dx5WW8=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=ogXX03WAflAL0jVP+Y/Wlt7fZyOEPBcGUN0kl91DmF37IIZNm7lXJiNx8QoaOXE//
	 eaLc/m2iXOGjkKcAZXcqK53PLgepCelsIJDnv9t+rARl3Ql26jGmAR+wwOI6sOxB/I
	 1X2om24BIFYNA8sM3syqPh6pm8StU0Aoh4cqfFgA=
Message-Id: <F7D8AC9F-21FA-417A-B3B5-3442D38BF0AE@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <BANLkTinqaNunCOaUgO1BaMp_0TUjtFP9vw@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [ANNOUNCEMENT] Yahoo focusing on Apache Hadoop, discontinuing "The Yahoo Distribution of Hadoop"
Date: Fri, 8 Apr 2011 14:40:19 -0700
References: <55BA39F5-5017-42B1-BE25-A643CCEADC75@yahoo-inc.com> <354259E0-9D22-47B5-B247-4187182B60DF@apache.org> <75192454-0158-4C84-9F79-337A16AA913E@yahoo-inc.com> <41978952-FE14-4CD9-8C39-BB3B0E458D55@yahoo-inc.com> <BANLkTikFDcqrMyFjGUBoF-r73HkmVVCgQw@mail.gmail.com> <7CCF44E2-EE8D-4A5E-BE5B-5163B37D2342@yahoo-inc.com> <BANLkTinqaNunCOaUgO1BaMp_0TUjtFP9vw@mail.gmail.com>
X-Mailer: Apple Mail (2.936)


On Apr 8, 2011, at 11:08 AM, Todd Lipcon wrote:

> These all have patches that are pretty small, and I'd imagine would  
> apply pretty easily to trunk. Let me know if you'd like any help  
> forward-porting.
>

Thanks Todd, I'm happy to help review etc.

> The other ones, as new features/improvements, I'd agree it makes  
> sense not to waste effort re-implementing them for trunk MR, but  
> rather to make sure they're incorporated in next-gen.

Yep, exactly. Glad to know it makes sense.

thanks,
Arun

From general-return-3238-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 10 04:53:46 2011
Return-Path: <general-return-3238-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73191 invoked from network); 10 Apr 2011 04:53:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Apr 2011 04:53:46 -0000
Received: (qmail 9070 invoked by uid 500); 10 Apr 2011 04:53:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8344 invoked by uid 500); 10 Apr 2011 04:53:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8336 invoked by uid 99); 10 Apr 2011 04:53:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Apr 2011 04:53:30 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.95 as permitted sender)
Received: from [17.148.16.95] (HELO asmtpout020.mac.com) (17.148.16.95)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Apr 2011 04:53:24 +0000
MIME-version: 1.0
Content-type: multipart/alternative;
 boundary="Boundary_(ID_jHg2v1w4fa7q8uNB1GNaEw)"
Received: from [10.0.1.31] ([71.198.192.174])
 by asmtp020.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LJF001WC6WEKR20@asmtp020.mac.com>
 for general@hadoop.apache.org; Sat, 09 Apr 2011 21:53:04 -0700 (PDT)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-04-10_02:2011-04-09,2011-04-10,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1104090149
From: Nigel Daley <ndaley@mac.com>
Subject: Re: Maintenance of Hadoop 0.21 branch?
Date: Sat, 09 Apr 2011 21:53:02 -0700
In-reply-to: <BANLkTim7sijRYVRmXx3LymEfskzBJL978w@mail.gmail.com>
To: general@hadoop.apache.org
References: <AANLkTi=owD6w2UA+-eCdDP67F=2NMChHLJxi1HYY+RG6@mail.gmail.com>
 <BANLkTim7sijRYVRmXx3LymEfskzBJL978w@mail.gmail.com>
Message-id: <AA3E0D0D-2227-4ED1-85B8-01006D2B2D18@mac.com>
X-Mailer: Apple Mail (2.1084)

--Boundary_(ID_jHg2v1w4fa7q8uNB1GNaEw)
Content-type: text/plain; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT

I'm fine w/ abandoning but that needs to be clearly communicated with changes to the website.  This is complicated by the fact that it looks like 0.21 docs are the only ones on the website for hdfs and mapreduce (which seems like a bug):
http://hadoop.apache.org/hdfs/docs/
http://hadoop.apache.org/mapreduce/docs/
Common has a bunch of back revs:
http://hadoop.apache.org/common/docs/

Nige

On Apr 5, 2011, at 1:00 PM, Tom White wrote:

> I'm not currently planning on doing a 0.21.1 release. I agree with
> Todd that it would be better to spend time getting 0.22 ready for a
> release.
> 
> Cheers,
> Tom
> 
> On Wed, Mar 30, 2011 at 2:41 PM, Todd Lipcon <todd@cloudera.com> wrote:
>> Hi all,
>> 
>> Some recent discussion on HDFS-1786 has raised an interesting question: does
>> anyone plan on maintaining the 0.21 branch and eventually releasing an
>> 0.21.1? Should we bother to commit bug fixes to this branch?
>> 
>> It seems to me that our time would be better spent getting 0.22 and trunk
>> back to a green state so we can talk about releasing them, rather than
>> applying patches to a branch with no releases planned.
>> 
>> Of course a decision now doesn't preclude anyone from stepping up later,
>> backporting patches to 0.21, and releasing an 0.21.1.
>> 
>> Thanks
>> -Todd
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>> 


--Boundary_(ID_jHg2v1w4fa7q8uNB1GNaEw)--

From general-return-3239-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 10 05:01:44 2011
Return-Path: <general-return-3239-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 86280 invoked from network); 10 Apr 2011 05:01:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Apr 2011 05:01:43 -0000
Received: (qmail 14535 invoked by uid 500); 10 Apr 2011 05:01:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14318 invoked by uid 500); 10 Apr 2011 05:01:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14308 invoked by uid 99); 10 Apr 2011 05:01:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Apr 2011 05:01:41 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.103 as permitted sender)
Received: from [17.148.16.103] (HELO asmtpout028.mac.com) (17.148.16.103)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Apr 2011 05:01:34 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.31] ([71.198.192.174])
 by asmtp028.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LJF00AMY79ZLU10@asmtp028.mac.com>
 for general@hadoop.apache.org; Sat, 09 Apr 2011 22:01:13 -0700 (PDT)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-04-10_02:2011-04-09,2011-04-10,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1104090150
Subject: Re: [VOTE] Abandon hdfsproxy HDFS contrib
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <BANLkTinND1e0rDZrPeke1WqCqDG_BPUhZg@mail.gmail.com>
Date: Sat, 09 Apr 2011 22:01:11 -0700
Message-id: <31722744-6C23-4AB0-BB5C-86CE3D2A3522@mac.com>
References: <D97CF74A-147E-4573-B188-2F9C48CFC482@mac.com>
 <AANLkTi=F2eBsACBfjYSVava+VwFcsRpqZdRRN+OHXNRJ@mail.gmail.com>
 <823262F0-EE37-4A0A-8F8C-D9422541D67B@mac.com>
 <0CBAFDBB-AC2A-495D-8936-4DF4D6731B69@Holsman.NET>
 <AANLkTimUhDbTnwn6Hk2oYXFJMts+A=FnL-q-veXqX1m0@mail.gmail.com>
 <F2EC1B1B-A2BB-4FC8-9842-3904A25338EE@gbiv.com>
 <AANLkTikVcGcKHnh9EgUDQr2JGLPvkBvBRgn4my3uJfOc@mail.gmail.com>
 <864B55E7-8A46-458F-8B16-8252A2011835@Holsman.NET>
 <AANLkTikhe6YjUBGY8LMc3OG=b7VXWz7WhyNKGkJRu_7K@mail.gmail.com>
 <FB7E6664-F292-480F-8C4F-2B3CCAB9FFB7@yahoo-inc.com>
 <D3254862-5A94-4121-9B0B-9AE276D0955C@mac.com>
 <BANLkTinND1e0rDZrPeke1WqCqDG_BPUhZg@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

AFAICT, Owen was the one to -1 removal of HDFS Proxy.  Owen, are you guys maintaining this?

Cheers,
Nige

On Apr 4, 2011, at 12:19 PM, Todd Lipcon wrote:

> Could those of you who -1ed the removal of HDFS Proxy please look into the
> test that has been failing our Hudson build for the last several months:
> https://issues.apache.org/jira/browse/HDFS-1666
> 
> <https://issues.apache.org/jira/browse/HDFS-1666>It is one thing to say that
> we "should" maintain a piece of code, but it's another to actually maintain
> it. In my mind, part of maintaining a project involves addressing consistent
> test failures as high priority items.
> 
> -Todd
> 
> On Tue, Feb 22, 2011 at 9:27 PM, Nigel Daley <ndaley@mac.com> wrote:
> 
>> For closure, this vote fails due to a couple binding -1 votes.
>> 
>> Nige
>> 
>> On Feb 18, 2011, at 4:46 AM, Eric Baldeschwieler wrote:
>> 
>>> Hi Bernd,
>>> 
>>> Apache Hadoop is about scale. Most clusters will always be small, but
>> Hadoop is going mainstream precisely because it scales to huge data and
>> cluster sizes.
>>> 
>>> There are lots of systems that work well on 10 node clusters. People
>> select   Hadoop because they are confident that as their business / problem
>> grows, Hadoop can grow with it.
>>> 
>>> ---
>>> E14 - via iPhone
>>> 
>>> On Feb 17, 2011, at 7:25 AM, "Bernd Fondermann" <
>> bernd.fondermann@googlemail.com> wrote:
>>> 
>>>> On Thu, Feb 17, 2011 at 14:58, Ian Holsman <hadoop@holsman.net> wrote:
>>>>> Hi Bernd.
>>>>> 
>>>>> On Feb 17, 2011, at 7:43 AM, Bernd Fondermann wrote:
>>>>>> 
>>>>>> We have the very unfortunate situation here at Hadoop where Apache
>>>>>> Hadoop is not the primary and foremost place of Hadoop development.
>>>>>> Instead, code is developed internally at Yahoo and then contributed in
>>>>>> (smaller or larger) chunks to Hadoop.
>>>>> 
>>>>> This has been the situation in the past,
>>>>> but as you can see in the last month, this has changed.
>>>>> 
>>>>> Yahoo! has publicly committed to move their development into the main
>> code base, and you can see they have started doing this with the 20.100
>> branch,
>>>>> and their recent commits to trunk.
>>>>> Combine this with Nige taking on the 0.22 release branch, (and
>> sheperding it into a stable release) and I think we have are addressing your
>> concerns.
>>>>> 
>>>>> They have also started bringing the discussions back on the list, see
>> the recent discussion about Jobtracker-nextgen Arun has re-started in
>> MAPREDUCE-279.
>>>>> 
>>>>> I'm not saying it's perfect, but I think the major players understand
>> there is an issue, and they are *ALL* moving in the right direction.
>>>> 
>>>> I enthusiastically would like to see your optimism be verified.
>>>> Maybe I'm misreading the statements issued publicly, but I don't think
>>>> that this is fully understood. I agree though that it's a move into
>>>> the right direction.
>>>> 
>>>>>> This is open source development upside down.
>>>>>> It is not ok for people to diff ASF svn against their internal code
>>>>>> and provide the diff as a patch without reviewing IP first for every
>>>>>> line of code changed.
>>>>>> For larger chunks I'd suggest to even go via the Incubator IP
>> clearance process.
>>>>>> Only then will we force committers to primarily work here in the open
>>>>>> and return to what I'd consider a healthy project.
>>>>>> 
>>>>>> To be honest: Hadoop is in the process of falling apart.
>>>>>> Contrib Code gets moved out of Apache instead of being maintained
>> here.
>>>>>> Discussions are seldom consense-driven.
>>>>>> Release branches stagnate.
>>>>> 
>>>>> True. releases do take a long time. This is mainly due to it being
>> extremely hard to test and verify that a release is stable.
>>>>> It's not enough to just run the thing on 4 machines, you need at least
>> 50 to test some of the major problems. This requires some serious $ for
>> someone to verify.
>>>> 
>>>> It has been proposed on the list before, IIRC. Don't know how to get
>>>> there, but the project seriously needs access to a cluster of this
>>>> size.
>>>> 
>>>>>> Downstream projects like HBase don't get proper support.
>>>>>> Production setups are made from 3rd party distributions.
>>>>>> Development is not happening here, but elsewhere behind corporate
>> doors.
>>>>>> Discussion about future developments are started on corporate blogs (
>>>>>> 
>> http://developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextgen/
>>>>>> ) instead of on the proper mailing list.
>>>>>> Hurdles for committing are way too high.
>>>>>> On the bright side, new committers and PMC members are added, this is
>>>>>> an improvement.
>>>>>> 
>>>>>> I'd suggest to move away from relying on large code dumps from
>>>>>> corporations, and move back to the ASF-proven "individual committer
>>>>>> commits on trunk"-model where more committers can get involved.
>>>>>> If that means not to support high end cluster sizes for some months,
>>>>>> well, so be it.
>>>>> 
>>>>>> Average committers cannot run - e.g. test - on high
>>>>>> end cluster sizes. If that would mean they cannot participate, then
>>>>>> the open source project better concentrate on small and medium sized
>>>>>> cluster instead.
>>>>> 
>>>>> 
>>>>> Well.. that's one approach.. but there are several companies out there
>> who rely on apache's hadoop to power their large clusters, so I'd hate to
>> see hadoop become something that only runs well on
>>>>> 10-nodes.. as I don't think that will help anyone either.
>>>> 
>>>> But only looking at high-end scale doesn't help either.
>>>> 
>>>> Lets face the fact that Hadoop is now moving from early adaptors phase
>>>> into a much broader market. I predict that small to medium sized
>>>> clusters will be the majority of Hadoop deployments in a few month
>>>> time. 4000, or even 500 machines is the high-end range. If the open
>>>> source project Hadoop cannot support those users adequately (without
>>>> becoming defunct), the committership might be better off to focus on
>>>> the low-end and medium sized users.
>>>> 
>>>> I'm not suggesting to turn away from the handfull (?) of high-end
>>>> users. They certainly have most valuable input. But also, *they*
>>>> obviously have the resources in terms of larger clusters and
>>>> developers to deal with their specific setups. Obviously, they don't
>>>> need to rely on the open source project to make releases. In fact,
>>>> they *do* work on their own Hadoop derivatives.
>>>> All the other users, the hundreds of boring small cluster users, don't
>>>> have that choice. They *depend* on the open source releases.
>>>> 
>>>> Hadoop is an Apache project, to provide HDFS and MR free of charge to
>>>> the general public. Not only to me - nor to only one or two big
>>>> companies either.
>>>> Focus on all the users.
>>>> 
>>>> Bernd
>> 
>> 
> 
> 
> -- 
> Todd Lipcon
> Software Engineer, Cloudera


From general-return-3240-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 10 05:14:40 2011
Return-Path: <general-return-3240-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 28530 invoked from network); 10 Apr 2011 05:14:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Apr 2011 05:14:40 -0000
Received: (qmail 23227 invoked by uid 500); 10 Apr 2011 05:14:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23182 invoked by uid 500); 10 Apr 2011 05:14:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23174 invoked by uid 99); 10 Apr 2011 05:14:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Apr 2011 05:14:35 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.104 as permitted sender)
Received: from [17.148.16.104] (HELO asmtpout029.mac.com) (17.148.16.104)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Apr 2011 05:14:28 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.31] ([71.198.192.174])
 by asmtp029.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LJF004027UQL570@asmtp029.mac.com>
 for general@hadoop.apache.org; Sat, 09 Apr 2011 22:13:40 -0700 (PDT)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-04-10_02:2011-04-09,2011-04-10,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1104090152
Subject: Re: [DISCUSS] Move common, hdfs,
 mapreduce contrib components to apache-extras.org or elsewhere
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <7F080B69-E0AE-4A38-9928-048377E73E72@yahoo-inc.com>
Date: Sat, 09 Apr 2011 22:13:38 -0700
Message-id: <E057847D-7F7A-4CFE-8893-B77C6B05AC05@mac.com>
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com>
 <AANLkTi=75e6oyBuGVivXP3is+76gOODSc1NcLKfg=VxF@mail.gmail.com>
 <AANLkTi=HQ1cn++JO6g+vp-cJAtAjB00Ok-FW2VrfTiSO@mail.gmail.com>
 <5E398D0F-ABCC-497B-939A-9213A5B3C7C4@mac.com>
 <7F080B69-E0AE-4A38-9928-048377E73E72@yahoo-inc.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org


On Mar 5, 2011, at 1:29 PM, Eric Baldeschwieler wrote:

> Hi Nigel,
> 
> I liked your previous plan.  Could you summarize your current plan?

I basically gave up.  Most everyone I talked to at various meetups was enthusiastically supportive of removing contrib components.  When put to a vote on list, however, most were vetoed by 1 or 2 people.
 
> Are we now planning to leave inactive projects in place?

I guess so.  HDFS Proxy seems like an example of this.

>  Ship them as source?

I would support that as a fallback.

> Seems like this still sends a confusing message.  I'm all for removing them from trunk in SVN, leaving a wiki link to their most recent state and letting their contributors make a case to revive them in whatever place they choose.
> 
> Keeping projects with small constituencies in our tree seems counter productive.

Agreed, but we couldn't reach consensus.

Cheers,
Nige


> On Feb 11, 2011, at 10:01 PM, Nigel Daley wrote:
> 
>> +1.  And only include them as source in releases.
>> 
>> Nige 
>> 
>> On Feb 11, 2011, at 10:12 AM, Tom White wrote:
>> 
>>> For contrib components that we elect not to remove, I suggest that we
>>> remove them from the main build, so that failures in a contrib
>>> component don't hinder the main build and release. This is what
>>> ZooKeeper does, for example.
>>> 
>>> Tom
>>> 
>>> On Fri, Feb 11, 2011 at 2:03 AM, Bernd Fondermann
>>> <bernd.fondermann@googlemail.com> wrote:
>>>> -1.
>>>> 
>>>> Move it away from TRUNK so it doesn't affect builds is a much better
>>>> (and simpler) solution. If someone wants to revive it, he can within
>>>> the bounds of Apache Hadoop and will become a part of the Hadoop
>>>> community, which would be good.
>>>> If you'd move it off-site, if the code ever gets new contributors,
>>>> it's hard to integrate them (code and contributors) into Hadoop again.
>>>> 
>>>> AFAIUI, apache-extras is for placing non-Apache code closer to the
>>>> related Apache projects, not for moving our code away from our own
>>>> premises.
>>>> 
>>>> Bernd
>>>> 
>>>> On Mon, Jan 31, 2011 at 04:42, Nigel Daley <ndaley@mac.com> wrote:
>>>>> Folks,
>>>>> 
>>>>> Now that http://apache-extras.org is launched (https://blogs.apache.org/foundation/entry/the_apache_software_foundation_launches) I'd like to start a discussion on moving contrib components out of common, mapreduce, and hdfs.
>>>>> 
>>>>> These contrib components complicate the builds, cause test failures that nobody seems to care about, have releases that are tied to Hadoop's long release cycles, etc.  Most folks I've talked with agree that these contrib components would be better served by being pulled out of Hadoop and hosted elsewhere. The new apache-extras code hosting site seems like a natural *default* location for migrating these contrib projects.  Perhaps some should graduate from contrib to src (ie from contrib to core of the project they're included in).  If folks agree, we'll need to come up with a mapping of contrib component to it's final destination and file a jira.
>>>>> 
>>>>> Here are the contrib components by project (hopefully I didn't miss any).
>>>>> 
>>>>> Common Contrib:
>>>>> failmon
>>>>> hod
>>>>> test
>>>>> 
>>>>> 
>>>>> MapReduce Contrib:
>>>>> capacity-scheduler -- move to MR core?
>>>>> data_join
>>>>> dynamic-scheduler
>>>>> eclipse-plugin
>>>>> fairscheduler -- move to MR core?
>>>>> gridmix
>>>>> index
>>>>> mrunit
>>>>> mumak
>>>>> raid
>>>>> sqoop
>>>>> streaming -- move to MR core?
>>>>> vaidya
>>>>> vertica
>>>>> 
>>>>> 
>>>>> HDFS Contrib:
>>>>> fuse-dfs
>>>>> hdfsproxy
>>>>> thriftfs
>>>>> 
>>>>> 
>>>>> Cheers,
>>>>> Nige
>>>>> 
>>>> 
>> 
> 


From general-return-3241-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 10 05:52:14 2011
Return-Path: <general-return-3241-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 79875 invoked from network); 10 Apr 2011 05:52:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Apr 2011 05:52:13 -0000
Received: (qmail 34055 invoked by uid 500); 10 Apr 2011 05:52:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33464 invoked by uid 500); 10 Apr 2011 05:52:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33453 invoked by uid 99); 10 Apr 2011 05:52:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Apr 2011 05:52:08 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.96 as permitted sender)
Received: from [17.148.16.96] (HELO asmtpout021.mac.com) (17.148.16.96)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Apr 2011 05:52:00 +0000
MIME-version: 1.0
Content-type: multipart/alternative;
 boundary="Boundary_(ID_83DnkAv8KvVdNItvfTDWdQ)"
Received: from [10.0.1.31] ([71.198.192.174])
 by asmtp021.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LJF00FWG9KLF270@asmtp021.mac.com>
 for general@hadoop.apache.org; Sat, 09 Apr 2011 22:50:58 -0700 (PDT)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-04-10_02:2011-04-09,2011-04-10,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1104090157
From: Nigel Daley <ndaley@mac.com>
Subject: Re: [VOTE] Abandon mrunit MapReduce contrib
Date: Sat, 09 Apr 2011 22:50:45 -0700
In-reply-to: <33A12FDB-820A-4CCC-9AB7-0521B41952B1@mac.com>
To: general@hadoop.apache.org
References: <77405974-6771-4604-926B-976240743F3C@mac.com>
 <AANLkTim80KJ3f14iM6Ltzy7UG613myhN-AiwQ3pCmVdi@mail.gmail.com>
 <AANLkTiknkrWK=GzGMVj4_Z6p5Jn8TQu1P5TkaFSgORJk@mail.gmail.com>
 <AANLkTin0TguLYHi9X0uL3ZF7tCz5xTHBYa7056oKF2Ae@mail.gmail.com>
 <93644D3B-4830-4813-92BE-809738969CF9@apache.org>
 <AANLkTi=dbGSLhtFA+Rf9LgaX05v45HDLoQDE_2C5ULAf@mail.gmail.com>
 <65311075-A30E-4971-A2BA-C75409C6486E@jpl.nasa.gov>
 <AANLkTi=XbpVUmg_TpH7eqEn8w7Gcu9KiwtJmKMLa5w5n@mail.gmail.com>
 <FAE19BAA-47A7-4747-BDD8-5AA42C259EB6@jpl.nasa.gov>
 <AANLkTin2JvOGdZT5g5tmTGeCpD3Qz2soZWcTQtHMsSza@mail.gmail.com>
 <AANLkTim+FQuoS+mR7YuTf0kw1KVxCdy14wDjf2u_cZdB@mail.gmail.com>
 <33A12FDB-820A-4CCC-9AB7-0521B41952B1@mac.com>
Message-id: <C2160E91-8133-4945-B2DD-497DE058E310@mac.com>
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

--Boundary_(ID_83DnkAv8KvVdNItvfTDWdQ)
Content-type: text/plain; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT

mrunit has been moved to the incubator:
https://svn.apache.org/repos/asf/incubator/mrunit/

I'll file an issue to remove the code from Hadoop SVN.

Nige

On Feb 11, 2011, at 9:57 PM, Nigel Daley wrote:

> This is great!  So we'll leave mrunit in contrib until it can be moved to incubator.
> 
> Nige
> 
> On Feb 11, 2011, at 2:26 PM, Eric Sammer wrote:
> 
>> Just to add to the option of going to incubator, I'm fine with that as well.
>> Github was an easy thing to get started and I was under the impression we
>> needed some greater degree of committer diversity and, frankly, a bigger
>> project. If mrunit is a candidate, keeping this under the ASF umbrella is
>> more than fine with me.
>> 
>> On Fri, Feb 11, 2011 at 5:10 PM, Aaron Kimball <akimball83@gmail.com> wrote:
>> 
>>> The main reason I am interested in removing MRUnit from Hadoop is that I
>>> believe that MRUnit deserves its own release cycle. I think this is in the
>>> best interest of its users.
>>> 
>>> MRUnit is valuable to users of several different versions of Hadoop. But
>>> MRUnit has only ever been committed to version 0.21 and above -- even
>>> though
>>> in practice, the majority (dare I say--all) of its users are running on
>>> 0.20. The only place today to get a version of MRUnit compatible with 0.20
>>> has been through a Cloudera release, which backported the entire MRUnit
>>> patchset.
>>> 
>>> My thoughts on MRUnit in 0.20.100 resonate with Eric's. There will be
>>> further fixes to MRUnit and its lightweight codebase can be released far
>>> more rapidly than whenever the next 0.20.1xx release of Hadoop would occur.
>>> Given that MRUnit has already been in the repository since April 2009 (see
>>> https://issues.apache.org/jira/browse/HADOOP-5518) and has yet to see an
>>> Apache 0.20-based release, I do not think it is in the best interest of the
>>> library's userbase to couple MRUnit's release cycle to that of Hadoop
>>> itself.
>>> 
>>> Perhaps more importantly, access to new features in MRUnit should not
>>> require upgrading one's entire Hadoop deployment; this is a client library
>>> that depends only on Hadoop's public APIs.
>>> 
>>> My primary concern is to move MRUnit to a place where the community can
>>> derive the most benefit from it. The Apache Incubator could fulfill this
>>> role; given the presence of individuals willing to mentor this project, I
>>> believe this would be a successful way to release MRUnit more quickly and
>>> continue to work to grow the MRUnit community.
>>> 
>>> Regards,
>>> - Aaron
>>> 
>>> 
>>> On Fri, Feb 11, 2011 at 11:57 AM, Mattmann, Chris A (388J) <
>>> chris.a.mattmann@jpl.nasa.gov> wrote:
>>> 
>>>> Awesome Patrick, we'd probably need one more active mentor. Any takers?
>>>> 
>>>> After we get that, then we cook up a proposal on the Incubator wiki here
>>>> [1], and follow the process here [2] to get started...
>>>> 
>>>> Cheers,
>>>> Chris
>>>> 
>>>> [1] http://wiki.apache.org/incubator/MRUnitProposal
>>>> [2] http://incubator.apache.org/guides/proposal.html
>>>> 
>>>> On Feb 11, 2011, at 11:52 AM, Patrick Hunt wrote:
>>>> 
>>>>> On Fri, Feb 11, 2011 at 9:44 AM, Mattmann, Chris A (388J)
>>>>> <chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>>> Guys, BTW, if you need help or a mentor in Apache Incubator-ville for
>>>> MRUnit, I would be happy to help.
>>>>> 
>>>>> I was going to suggest the same thing (mrunit to incubator). I would
>>>>> also be happy to be a mentor.
>>>>> 
>>>>> Patrick
>>>>> 
>>>>>> 
>>>>>> On Feb 11, 2011, at 9:04 AM, Eric Sammer wrote:
>>>>>> 
>>>>>>> On Fri, Feb 11, 2011 at 11:48 AM, Owen O'Malley <omalley@apache.org>
>>>> wrote:
>>>>>>> 
>>>>>>>> On Feb 11, 2011, at 8:02 AM, Eric Sammer wrote:
>>>>>>>> 
>>>>>>>> - allow mrunit to have its own release cycle. This is, I think, the
>>>> most
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> important.
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> If you submit your work to Apache we can evaluate it for inclusion
>>> in
>>>> the
>>>>>>>> 0.20.100 branch to get your changes released in a timely manner.
>>>>>>> 
>>>>>>> 
>>>>>>> I'm thinking in general (beyond the next immediate release).
>>>> Independent of
>>>>>>> where mrunit goes, I think it should leave the contrib tree to
>>>> facilitate
>>>>>>> light weight releases (the dependency on Hadoop proper is a public
>>>> facing
>>>>>>> API - a pure client). I think most projects could benefit from this
>>>> with the
>>>>>>> exception of things that are tightly coupled to Hadoop releases or
>>>> touch
>>>>>>> non-public APIs.
>>>>>>> 
>>>>>>> 
>>>>>>>> I would actually prefer to move it to Extras or Incubator and leave
>>>> this
>>>>>>>>> within the ASF.
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> Extras is **NOT** inside of the ASF. Extras is a source hosting
>>> system
>>>> for
>>>>>>>> non-Apache projects that are related to Apache projects.
>>>>>>> 
>>>>>>> 
>>>>>>> Got it. Thanks for correcting me. I only mentioned it because someone
>>>>>>> suggested it to me initially.
>>>>>>> 
>>>>>>> 
>>>>>>>> Right now, I picked github because of the ability to easily
>>>>>>>> collaborate with others (and to use git).
>>>>>>>> 
>>>>>>> 
>>>>>>> I agree that it is unfortunate that Apache doesn't yet support
>>>> read-write
>>>>>>>> git access. However, you'll find that building a community is easier
>>>> at
>>>>>>>> Apache than at github.
>>>>>>>> 
>>>>>>> 
>>>>>>>> -- Owen
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> Eric Sammer
>>>>>>> twitter: esammer
>>>>>>> data: www.cloudera.com
>>>>>> 
>>>>>> 
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>> Chris Mattmann, Ph.D.
>>>>>> Senior Computer Scientist
>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>> Email: chris.a.mattmann@nasa.gov
>>>>>> WWW:   http://sunset.usc.edu/~mattmann/
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> Chris Mattmann, Ph.D.
>>>> Senior Computer Scientist
>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> Office: 171-266B, Mailstop: 171-246
>>>> Email: chris.a.mattmann@nasa.gov
>>>> WWW:   http://sunset.usc.edu/~mattmann/
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> Adjunct Assistant Professor, Computer Science Department
>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> 
>>>> 
>>> 
>> 
>> 
>> 
>> -- 
>> Eric Sammer
>> twitter: esammer
>> data: www.cloudera.com
> 


--Boundary_(ID_83DnkAv8KvVdNItvfTDWdQ)--

From general-return-3242-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 10 06:10:03 2011
Return-Path: <general-return-3242-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34192 invoked from network); 10 Apr 2011 06:10:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Apr 2011 06:10:02 -0000
Received: (qmail 44241 invoked by uid 500); 10 Apr 2011 06:10:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 43734 invoked by uid 500); 10 Apr 2011 06:10:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 43565 invoked by uid 99); 10 Apr 2011 06:09:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Apr 2011 06:09:58 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.95 as permitted sender)
Received: from [17.148.16.95] (HELO asmtpout020.mac.com) (17.148.16.95)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Apr 2011 06:09:49 +0000
MIME-version: 1.0
Content-type: multipart/alternative;
 boundary="Boundary_(ID_kQNMshv4ySB0L5RxZHdg5w)"
Received: from [10.0.1.31] ([71.198.192.174])
 by asmtp020.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LJF003OEAFRQB20@asmtp020.mac.com>
 for general@hadoop.apache.org; Sat, 09 Apr 2011 23:09:28 -0700 (PDT)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-04-10_02:2011-04-09,2011-04-10,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1104090160
From: Nigel Daley <ndaley@mac.com>
Subject: HADOOP-7106: Re-organize hadoop subversion layout
Date: Sat, 09 Apr 2011 23:09:27 -0700
In-reply-to: <B98835A6-F4B7-426E-B22F-B2BEBE273C47@mac.com>
To: general@hadoop.apache.org
References: <C955EB7A.2E753%sureshms@yahoo-inc.com>
 <E2932D45-FF26-4579-A975-A98D2578622A@mac.com>
 <B98835A6-F4B7-426E-B22F-B2BEBE273C47@mac.com>
Message-id: <4112B27D-7DCF-4C8E-AD61-6884BA7E738C@mac.com>
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

--Boundary_(ID_kQNMshv4ySB0L5RxZHdg5w)
Content-type: text/plain; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT

All, 

As discussed in Jan/Feb, I'd like to coordinate a date for committing the re-organization of our svn layout: https://issues.apache.org/jira/browse/HADOOP-7106.  I propose Thursday April 21 at 11am PDT.

- I will send out reminders leading up to that date.
- I will announce on IRC when I'm about to start the changes.
- I will run the script to make the changes.
- Ian, can you update the asf-authorization-template file and the asf-mailer.conf files at the same time?
- Owen/Todd/Jukka, can you make sure that actions needed by git users are taken care of at the same time? (what are these)

More info on this change is at http://wiki.apache.org/hadoop/ProjectSplit

Cheers,
Nige

--Boundary_(ID_kQNMshv4ySB0L5RxZHdg5w)--

From general-return-3243-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 10 10:13:38 2011
Return-Path: <general-return-3243-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 90494 invoked from network); 10 Apr 2011 10:13:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Apr 2011 10:13:38 -0000
Received: (qmail 89198 invoked by uid 500); 10 Apr 2011 10:06:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88843 invoked by uid 500); 10 Apr 2011 10:06:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 54586 invoked by uid 99); 10 Apr 2011 06:57:46 -0000
X-ASF-Spam-Status: No, hits=2.9 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <BANLkTincvMXiCdEn=vjMCu5xg7d_1Syk8g@mail.gmail.com>
References: <BANLkTincvMXiCdEn=vjMCu5xg7d_1Syk8g@mail.gmail.com>
Date: Sun, 10 Apr 2011 09:57:18 +0300
Message-ID: <BANLkTikLb1s-L5QLs+NhJGNECbbVnYTXuw@mail.gmail.com>
Subject: Re: ANN: HBase 0.90.2 is available for download
From: Lior Schachter <liors@infolinks.com>
To: Stack <stack@duboce.net>, general@hadoop.apache.org
Cc: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=20cf307f32e04db1e204a08af8ed

--20cf307f32e04db1e204a08af8ed
Content-Type: text/plain; charset=ISO-8859-1

Hi Stack,

We already have version 0.90.1 installed on our production cluster and we
want to upgrade to 0.90.2.
Obviously we should upgrade the hbase-0.90.1.jar with the new jar.
Should we upgrade other libraries/configuration files ?
Is there a maven repository with the new version ?

Thanks,
Lior



On Fri, Apr 8, 2011 at 8:39 PM, Stack <stack@duboce.net> wrote:

> The Apache HBase team is happy to announce the general availability of
> HBase
> 0.90.2, available from your Apache mirror of choice:
>
>  http://www.apache.org/dyn/closer.cgi/hbase/
>
> HBase 0.90.2 is a maintenance release that fixes several important bugs
> since version 0.90.1, while retaining API and data compatibility. The
> release notes may be found on the Apache JIRA:
>
>  http://su.pr/1F6omq
>
> Users upgrading from HBase 0.90.0 or 0.90.1 may upgrade clients and servers
> separately, though it is recommended that both be upgraded. If upgrading
> from a version of HBase prior to 0.90.0, please read the notes accompanying
> that release:
>
>  http://osdir.com/ml/general-hadoop-apache/2011-01/msg00208.html
>
> As always, many thanks to those who contributed to this release!
>
> -The HBase Team
>

--20cf307f32e04db1e204a08af8ed--

From general-return-3244-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 10 12:37:07 2011
Return-Path: <general-return-3244-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57305 invoked from network); 10 Apr 2011 12:37:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Apr 2011 12:37:07 -0000
Received: (qmail 59504 invoked by uid 500); 10 Apr 2011 12:37:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59427 invoked by uid 500); 10 Apr 2011 12:37:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59418 invoked by uid 99); 10 Apr 2011 12:37:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Apr 2011 12:37:05 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of magawake@gmail.com designates 209.85.218.48 as permitted sender)
Received: from [209.85.218.48] (HELO mail-yi0-f48.google.com) (209.85.218.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Apr 2011 12:36:58 +0000
Received: by yia28 with SMTP id 28so2470962yia.35
        for <general@hadoop.apache.org>; Sun, 10 Apr 2011 05:36:37 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=6qPZqQtzRh/5+SiFNJR8YOvj4avSEbGClQgmWP5WVhc=;
        b=XBiMqsG1lRknFd7NiLdIeZlcYd/eUe7YCcML0AL7dYRwZfXOD0yqpUMMG9sT63vp9X
         TkVAbwpAEEkEzlC7E8I5j3J6fr5eg4zixyF6IJyjxO/ckQOMUl0kkcTCYhu5RoC03Fwo
         3bLKMuidyFKRV0in1IBWr+GdtjWFfRhW1cHTk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=OYv+ARCV4hkRK4q9Ofa4dfuEmtwTsslu+MI8z+59vUnK1K8uU6wMKCxioc+dyM4ZF/
         dyB/nOIKi1DrxsZWI3LO8ZPsQg0udOtso0/cSK3Xy3iPXujxJSSSQYvzYnZU7Co2nSEr
         52tqstxujr93a45aR7EbvKBXnZxKVTMGBmBik=
MIME-Version: 1.0
Received: by 10.236.161.163 with SMTP id w23mr5451323yhk.245.1302438997181;
 Sun, 10 Apr 2011 05:36:37 -0700 (PDT)
Received: by 10.236.111.34 with HTTP; Sun, 10 Apr 2011 05:36:37 -0700 (PDT)
Date: Sun, 10 Apr 2011 08:36:37 -0400
Message-ID: <BANLkTiny4wC1q8oLb5hdogdL94JdN0MXQg@mail.gmail.com>
Subject: hbase
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
X-Virus-Checked: Checked by ClamAV on apache.org

Just curious, does hbase require mapreduce? Basically, I have several
terabytes of data and I would like to query it similar to sql fashion.
Was wondering if mapreduce was required.

From general-return-3245-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 10 13:57:04 2011
Return-Path: <general-return-3245-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1668 invoked from network); 10 Apr 2011 13:57:04 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Apr 2011 13:57:04 -0000
Received: (qmail 81781 invoked by uid 500); 10 Apr 2011 13:57:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81712 invoked by uid 500); 10 Apr 2011 13:57:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81704 invoked by uid 99); 10 Apr 2011 13:57:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Apr 2011 13:57:02 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates 209.85.212.178 as permitted sender)
Received: from [209.85.212.178] (HELO mail-px0-f178.google.com) (209.85.212.178)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Apr 2011 13:56:55 +0000
Received: by pxi1 with SMTP id 1so3193055pxi.37
        for <general@hadoop.apache.org>; Sun, 10 Apr 2011 06:56:34 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=Bitt5HtiUa7Ms3ccW7/1K+zyN99Ko7Sv/WIeVOCTIug=;
        b=NsB2NO80InIlmpj1cAdCBMSdEcpq2hkYsaTLUTtwD8g4AHPJqVpLTIVBtEMAaY/iZP
         DhwK5N0sYa8dS1UiOZWWgkT9qHe8iuS7MbJm00JvfHDCh0jIUsGojR/2qemEMhSreVyn
         X53BYbw1KmiomuDccWj6eJKjZxTxJpePjhFM4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=ssEPG7y1hOnd+B81QllRIcSW8p9krRpXk9dwlSg2MdPUUiP9VREL23PB4oz9lR6xQC
         clPHtIoEoP0p+TqSV0eeopwEk8g2UWm5aXZs3ZFR+E1BkiR/RgdeLlo1+LEkFCYOM53/
         DX4qulfmYSQUkL476oaj0eIgYMd8R05r3Xlgs=
MIME-Version: 1.0
Received: by 10.143.47.6 with SMTP id z6mr4042494wfj.227.1302443794033; Sun,
 10 Apr 2011 06:56:34 -0700 (PDT)
Received: by 10.68.62.166 with HTTP; Sun, 10 Apr 2011 06:56:34 -0700 (PDT)
In-Reply-To: <BANLkTiny4wC1q8oLb5hdogdL94JdN0MXQg@mail.gmail.com>
References: <BANLkTiny4wC1q8oLb5hdogdL94JdN0MXQg@mail.gmail.com>
Date: Sun, 10 Apr 2011 06:56:34 -0700
Message-ID: <BANLkTikXx9XLmv9Gi6Zt-bg2Yarn5Y578A@mail.gmail.com>
Subject: Re: hbase
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd480ecaeee7204a090d3f2
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd480ecaeee7204a090d3f2
Content-Type: text/plain; charset=ISO-8859-1

Mapreduce isn't required.
For your query, Hive would be a better fit.

On Sun, Apr 10, 2011 at 5:36 AM, Mag Gam <magawake@gmail.com> wrote:

> Just curious, does hbase require mapreduce? Basically, I have several
> terabytes of data and I would like to query it similar to sql fashion.
> Was wondering if mapreduce was required.
>

--000e0cd480ecaeee7204a090d3f2--

From general-return-3246-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 10 15:37:10 2011
Return-Path: <general-return-3246-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 40155 invoked from network); 10 Apr 2011 15:37:10 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Apr 2011 15:37:10 -0000
Received: (qmail 40191 invoked by uid 500); 10 Apr 2011 15:37:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40121 invoked by uid 500); 10 Apr 2011 15:37:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40113 invoked by uid 99); 10 Apr 2011 15:37:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Apr 2011 15:37:08 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of magawake@gmail.com designates 209.85.218.48 as permitted sender)
Received: from [209.85.218.48] (HELO mail-yi0-f48.google.com) (209.85.218.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Apr 2011 15:37:02 +0000
Received: by yia28 with SMTP id 28so2503159yia.35
        for <general@hadoop.apache.org>; Sun, 10 Apr 2011 08:36:41 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:cc:content-type;
        bh=FPt3cKYqf80RrE2AqsESxb/bMmgHruekmdkjZJG5vz8=;
        b=NnGihRG11z0NVgQohfDNloS3GHmrQtvnAz+5U40xRzESm3E+UIvpQlGmZyWm+N2cnL
         ganeydcI3gLCK6JM8WFd3fRxIu4zYqwfeanMsbmK1luj46JYbikMNx8XXEG0cIR5kjIV
         AgMhLaTMCg90GGWM1En4OT67CkAd1pMhZn22k=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=eGQ8u4lcDIg4YE0uNmzKEvLayfEmaRtM7ZVAvT9tmwYfcp3i+rjtQqkdc08MoIdwe2
         EqMy3dyBH+ewvDl92rgW/s4I1I/G9SO+pCVpNTd1htrGQ3lEy3vyeqaPBQikWeBnb9Cw
         4a81Hdu/aqo166gEDv2KMnvIPzFkzfVEJ2sd0=
MIME-Version: 1.0
Received: by 10.236.176.227 with SMTP id b63mr5172422yhm.494.1302449801858;
 Sun, 10 Apr 2011 08:36:41 -0700 (PDT)
Received: by 10.236.111.34 with HTTP; Sun, 10 Apr 2011 08:36:41 -0700 (PDT)
In-Reply-To: <BANLkTikXx9XLmv9Gi6Zt-bg2Yarn5Y578A@mail.gmail.com>
References: <BANLkTiny4wC1q8oLb5hdogdL94JdN0MXQg@mail.gmail.com>
	<BANLkTikXx9XLmv9Gi6Zt-bg2Yarn5Y578A@mail.gmail.com>
Date: Sun, 10 Apr 2011 11:36:41 -0400
Message-ID: <BANLkTikb5=4yJrXapTrF_y0EcvVVCWFFDw@mail.gmail.com>
Subject: Re: hbase
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Cc: Ted Yu <yuzhihong@gmail.com>
Content-Type: text/plain; charset=UTF-8

It seems with HIVE is a large latency. Is there anything else out
there which has less latencies?



On Sun, Apr 10, 2011 at 9:56 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> Mapreduce isn't required.
> For your query, Hive would be a better fit.
>
> On Sun, Apr 10, 2011 at 5:36 AM, Mag Gam <magawake@gmail.com> wrote:
>
>> Just curious, does hbase require mapreduce? Basically, I have several
>> terabytes of data and I would like to query it similar to sql fashion.
>> Was wondering if mapreduce was required.
>>
>

From general-return-3247-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 11 12:19:21 2011
Return-Path: <general-return-3247-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6659 invoked from network); 11 Apr 2011 12:19:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Apr 2011 12:19:20 -0000
Received: (qmail 52504 invoked by uid 500); 11 Apr 2011 12:19:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52136 invoked by uid 500); 11 Apr 2011 12:19:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52092 invoked by uid 99); 11 Apr 2011 12:19:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Apr 2011 12:19:17 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [85.214.115.60] (HELO h1349259.stratoserver.net) (85.214.115.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Apr 2011 12:19:12 +0000
Received: from isabel-drost.de ([85.214.115.60] helo=motokokusanaghi.localnet)
	by h1349259.stratoserver.net with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
	(Exim 4.69)
	(envelope-from <isabel@apache.org>)
	id 1Q9G4u-0006f9-Ea; Mon, 11 Apr 2011 12:18:48 +0000
From: Isabel Drost <isabel@apache.org>
Reply-To: isabel@apache.org
To: isabel@apache.org
Subject: Berlin Buzzwords - Schedule published/ Call for Hackathons
Date: Mon, 11 Apr 2011 14:18:44 +0200
User-Agent: KMail/1.13.5 (Linux/2.6.32-5-amd64; KDE/4.4.5; x86_64; ; )
MIME-Version: 1.0
Content-Type: multipart/signed;
  boundary="nextPart2864944.n8WrOiHzbG";
  protocol="application/pgp-signature";
  micalg=pgp-sha1
Content-Transfer-Encoding: 7bit
Message-Id: <201104111418.46016.isabel@apache.org>

--nextPart2864944.n8WrOiHzbG
Content-Type: Text/Plain;
  charset="us-ascii"
Content-Transfer-Encoding: quoted-printable


Hello,

as of today the schedule of Berlin Buzzwords is available online at:

http://berlinbuzzwords.de/content/program

We are amazed by the amount and quality of the submissions we got. As a res=
ult=20
this year we have added one more track to the main conference. If you haven=
't=20
done so already, make sure to book your ticket now - early bird tickets are=
=20
already sold out.

As we would like to give visitors of our main conference a reason to stay i=
n=20
town for the whole week, we have been talking to local co-working spaces an=
d=20
companies asking them for free space and WiFi to host Hackathons right afte=
r the=20
main conference - that is on June 8th through 10th.

If you would like to gather with fellow developers and users of your projec=
t,=20
fix bugs together, hack on new features or give users a hands-on introducti=
on to=20
your tools, please submit your workshop proposal to our wiki:

http://berlinbuzzwords.de/node/428

Please note that slots are assigned on a first come first serve basis. We a=
re=20
doing our best to get you connected, however space is limited.

The deal is simple: We get you in touch with a conference room provider. Yo=
ur=20
event gets promoted in our schedule. Co-Ordination however is completely up=
 to=20
you: Make sure to provide an interesting abstract, provide a Hackathon=20
registration area - see the Barcamp page for a good example:

http://berlinbuzzwords.de/wiki/barcamp

Attending Hackathons requires a Berlin Buzzwords ticket and (then free)=20
registration at the Hackathon in question.


Looking forward to meeting you in Berlin,

The Berlin Buzzwords team.

--nextPart2864944.n8WrOiHzbG
Content-Type: application/pgp-signature; name=signature.asc 
Content-Description: This is a digitally signed message part.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEABECAAYFAk2i8aUACgkQPJpCBAwIhbT59ACgkWxk/rb74ijF4tCTPvnpg3YB
LckAoKjdSDg++xmEUIvvfTLGESsEM09Y
=r5Fb
-----END PGP SIGNATURE-----

--nextPart2864944.n8WrOiHzbG--

From general-return-3248-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 11 15:25:49 2011
Return-Path: <general-return-3248-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25560 invoked from network); 11 Apr 2011 15:25:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Apr 2011 15:25:48 -0000
Received: (qmail 37554 invoked by uid 500); 11 Apr 2011 15:25:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37510 invoked by uid 500); 11 Apr 2011 15:25:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37502 invoked by uid 99); 11 Apr 2011 15:25:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Apr 2011 15:25:46 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Apr 2011 15:25:40 +0000
Received: by qwj9 with SMTP id 9so4483931qwj.35
        for <general@hadoop.apache.org>; Mon, 11 Apr 2011 08:25:19 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:cc:content-type
         :content-transfer-encoding;
        bh=BaGwmCiLaKCNZrv2gqNJq7lJVFh3XSoB714khRACLa0=;
        b=EkdJcDzW3/DEQ8FtXiyMhAXZwU0G7/lQIHfvOsaKg3e/jwGIyE34YzOCY1kLpSEFyY
         Eyv08RPQB2u89lPK8E+71vfD5OMqkJfMzN6/X/4O8yjuNRVtoe/if25qKCtxuK20DQw4
         yoauGHaxvwI9yFwxOZnDwMQoSqIR9W/rOp1UM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:cc:content-type
         :content-transfer-encoding;
        b=mkSWGUh+Rok0ncibp8SOZJCths3QoVXsq9XNsaPsALFPfQ20WlDuThfQ4FAj5DhdSw
         W3K6deorPADq3Gh/6zaLKh81K2zUE0LF0FJqcE5jpL8knwD61t+5u7l0c+ywrsU2ej4U
         +Q9Gntgjw5+utE+5cf6OECOZpG3JGSe5DW0fw=
MIME-Version: 1.0
Received: by 10.52.186.166 with SMTP id fl6mr3676736vdc.179.1302535519398;
 Mon, 11 Apr 2011 08:25:19 -0700 (PDT)
Sender: saint.ack@gmail.com
Received: by 10.52.166.198 with HTTP; Mon, 11 Apr 2011 08:25:19 -0700 (PDT)
In-Reply-To: <BANLkTikLb1s-L5QLs+NhJGNECbbVnYTXuw@mail.gmail.com>
References: <BANLkTincvMXiCdEn=vjMCu5xg7d_1Syk8g@mail.gmail.com>
	<BANLkTikLb1s-L5QLs+NhJGNECbbVnYTXuw@mail.gmail.com>
Date: Mon, 11 Apr 2011 08:25:19 -0700
X-Google-Sender-Auth: 7gSNQXoukW1a-guEZr4YMJrf7N4
Message-ID: <BANLkTimGzxOWa83ay9kqALs-4VKVUoGSvw@mail.gmail.com>
Subject: Re: ANN: HBase 0.90.2 is available for download
From: Stack <stack@duboce.net>
To: Lior Schachter <liors@infolinks.com>
Cc: general@hadoop.apache.org, user@hbase.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Lior and Joe:

Sorry for the mvn lag.  The mvn deploy system is smarter than me.  The
deploy requires four full builds of HBase running all tests.  I
inevitably get distracted and forget the process or else I am demented
and answer one of the questions just off and then I have to start
over.  I'm not sure how it all works either and should take the time
to figure it (but again, distraction, dementia take over...).

It should be up soon.  I'm on it.

St.Ack


On Sat, Apr 9, 2011 at 11:57 PM, Lior Schachter <liors@infolinks.com> wrote=
:
> Hi Stack,
>
> We already have version 0.90.1 installed on our production cluster and we
> want to upgrade to 0.90.2.
> Obviously we should upgrade the hbase-0.90.1.jar with the new jar.
> Should we upgrade other libraries/configuration files ?
> Is there a maven repository with the new version ?
>
> Thanks,
> Lior
>
>
>
> On Fri, Apr 8, 2011 at 8:39 PM, Stack <stack@duboce.net> wrote:
>>
>> The Apache HBase team is happy to announce the general availability of
>> HBase
>> 0.90.2, available from your Apache mirror of choice:
>>
>> =A0http://www.apache.org/dyn/closer.cgi/hbase/
>>
>> HBase 0.90.2 is a maintenance release that fixes several important bugs
>> since version 0.90.1, while retaining API and data compatibility. The
>> release notes may be found on the Apache JIRA:
>>
>> =A0http://su.pr/1F6omq
>>
>> Users upgrading from HBase 0.90.0 or 0.90.1 may upgrade clients and
>> servers
>> separately, though it is recommended that both be upgraded. If upgrading
>> from a version of HBase prior to 0.90.0, please read the notes
>> accompanying
>> that release:
>>
>> =A0http://osdir.com/ml/general-hadoop-apache/2011-01/msg00208.html
>>
>> As always, many thanks to those who contributed to this release!
>>
>> -The HBase Team
>
>

From general-return-3249-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 11 15:54:30 2011
Return-Path: <general-return-3249-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33601 invoked from network); 11 Apr 2011 15:54:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Apr 2011 15:54:29 -0000
Received: (qmail 82675 invoked by uid 500); 11 Apr 2011 15:54:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82614 invoked by uid 500); 11 Apr 2011 15:54:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82604 invoked by uid 99); 11 Apr 2011 15:54:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Apr 2011 15:54:28 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mlortiz@uci.cu designates 200.55.140.180 as permitted sender)
Received: from [200.55.140.180] (HELO mx3.uci.cu) (200.55.140.180)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 11 Apr 2011 15:54:20 +0000
Received: (qmail 21442 invoked by uid 507); 11 Apr 2011 15:53:37 -0000
Received: from 10.0.0.185 by ns3.uci.cu (envelope-from <mlortiz@uci.cu>, uid 501) with qmail-scanner-2.01st 
 (avp: 5.0.2.0. spamassassin: 3.0.6. perlscan: 2.01st.  
 Clear:RC:1(10.0.0.185):. 
 Processed in 0.022846 secs); 11 Apr 2011 15:53:37 -0000
Received: from unknown (HELO ucimail4.uci.cu) (10.0.0.185)
  by 0 with SMTP; 11 Apr 2011 15:53:37 -0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by ucimail4.uci.cu (Postfix) with ESMTP id E6DC61324682
	for <general@hadoop.apache.org>; Mon, 11 Apr 2011 11:53:37 -0400 (CDT)
X-Virus-Scanned: amavisd-new at uci.cu
Received: from ucimail4.uci.cu ([127.0.0.1])
	by localhost (ucimail4.uci.cu [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id LGudfuTEVDmj; Mon, 11 Apr 2011 11:53:37 -0400 (CDT)
Received: from [10.36.18.29] (skynet2010.uci.cu [10.36.18.29])
	by ucimail4.uci.cu (Postfix) with ESMTP id 15F5613246B6
	for <general@hadoop.apache.org>; Mon, 11 Apr 2011 11:53:36 -0400 (CDT)
Message-ID: <4DA32452.90804@uci.cu>
Date: Mon, 11 Apr 2011 11:54:58 -0400
From: Marcos Ortiz <mlortiz@uci.cu>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; es-ES; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Where I can find all info about the next ApacheCon
References: <BANLkTincvMXiCdEn=vjMCu5xg7d_1Syk8g@mail.gmail.com>	<BANLkTikLb1s-L5QLs+NhJGNECbbVnYTXuw@mail.gmail.com> <BANLkTimGzxOWa83ay9kqALs-4VKVUoGSvw@mail.gmail.com>
In-Reply-To: <BANLkTimGzxOWa83ay9kqALs-4VKVUoGSvw@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit
X-Virus-Checked: Checked by ClamAV on apache.org


Regards to all the Apache colleagues.
I had the completed information about the next ApacheCon on a message, 
but I lost it.
Where I can find it?
If the completed information is on one email, please send to me.

Thanks a  lot for your time

-- 
Marcos Luís Ortíz Valmaseda
  Software Engineer (Large-Scaled Distributed Systems)
  University of Information Sciences,
  La Habana, Cuba
  Linux User # 418229



From general-return-3250-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 11 16:00:30 2011
Return-Path: <general-return-3250-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37568 invoked from network); 11 Apr 2011 16:00:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Apr 2011 16:00:29 -0000
Received: (qmail 91838 invoked by uid 500); 11 Apr 2011 16:00:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91790 invoked by uid 500); 11 Apr 2011 16:00:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91782 invoked by uid 99); 11 Apr 2011 16:00:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Apr 2011 16:00:28 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jdcryans@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Apr 2011 16:00:22 +0000
Received: by pwi14 with SMTP id 14so3073508pwi.35
        for <general@hadoop.apache.org>; Mon, 11 Apr 2011 09:00:01 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=19a5BMLSvVZq9fp5gEcm2Ad+TddEgUvYoA9rCmXJbUw=;
        b=kOT9LydxgN4Wx2/ZX84DdeuWXcQLxrsarW7A0I0hDvbBtHsyuGAxvk6fPySzTRIHfi
         VgHG2xg4iUDJI85hyvIyRrKCiRJjQRA6zKRKXPlKhtv/CW0DwPSklEMOZzWrVe84SBcW
         cx/RS1twq8ZlTabxaSYcrs1GQl48tIEWk/HSE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=eUd0q0RqhjRpONy0t11G6Hdfa7QFjgVvzzV8vRDWYsep+5+UMxyy9WCN7wUb2m3Mqr
         NCbDgiYNrFmCv7zmi1TT0dGyPNN3MPvQEuiAe6z4ZusOHDpqbDB7ytET0bY0emb0K5Xr
         9ZcOUkWE4JSXelP2kcY97V2J8gq86jjc0bDmA=
MIME-Version: 1.0
Received: by 10.142.67.30 with SMTP id p30mr5490645wfa.112.1302537601188; Mon,
 11 Apr 2011 09:00:01 -0700 (PDT)
Sender: jdcryans@gmail.com
Received: by 10.68.48.168 with HTTP; Mon, 11 Apr 2011 09:00:01 -0700 (PDT)
In-Reply-To: <4DA32452.90804@uci.cu>
References: <BANLkTincvMXiCdEn=vjMCu5xg7d_1Syk8g@mail.gmail.com>
	<BANLkTikLb1s-L5QLs+NhJGNECbbVnYTXuw@mail.gmail.com>
	<BANLkTimGzxOWa83ay9kqALs-4VKVUoGSvw@mail.gmail.com>
	<4DA32452.90804@uci.cu>
Date: Mon, 11 Apr 2011 09:00:01 -0700
X-Google-Sender-Auth: GOsSmsWx4mhfGQFBsr0T4OfaTMo
Message-ID: <BANLkTikPhp9kiyRbBFawGCcMtLXMJYvXeQ@mail.gmail.com>
Subject: Re: Where I can find all info about the next ApacheCon
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

http://www.apachecon.com

You're welcome.

J-D

On Mon, Apr 11, 2011 at 8:54 AM, Marcos Ortiz <mlortiz@uci.cu> wrote:
>
> Regards to all the Apache colleagues.
> I had the completed information about the next ApacheCon on a message, bu=
t I
> lost it.
> Where I can find it?
> If the completed information is on one email, please send to me.
>
> Thanks a =A0lot for your time
>
> --
> Marcos Lu=EDs Ort=EDz Valmaseda
> =A0Software Engineer (Large-Scaled Distributed Systems)
> =A0University of Information Sciences,
> =A0La Habana, Cuba
> =A0Linux User # 418229
>
>
>

From general-return-3251-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 11 16:12:57 2011
Return-Path: <general-return-3251-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 90321 invoked from network); 11 Apr 2011 16:12:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Apr 2011 16:12:57 -0000
Received: (qmail 9230 invoked by uid 500); 11 Apr 2011 16:12:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 9178 invoked by uid 500); 11 Apr 2011 16:12:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 9170 invoked by uid 99); 11 Apr 2011 16:12:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Apr 2011 16:12:55 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of tom.e.white@gmail.com designates 209.85.210.48 as permitted sender)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Apr 2011 16:12:50 +0000
Received: by pzk10 with SMTP id 10so3053992pzk.35
        for <general@hadoop.apache.org>; Mon, 11 Apr 2011 09:12:30 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:cc:content-type:content-transfer-encoding;
        bh=+mbfjcTWnVOlCPk5hJhyDybBemRwKCje/a3NjBo6tt4=;
        b=Bc9tykpq7zNVxov1Kt3kpBh2AUxDpQJMPQ7XJmdbxvlS/2K2jC+SxFeRQ2BmztWeY/
         oGMi7ENuAvFclGGSMeA5sqp8fm+4ivhofUPk2YfecIbVE7W6RjEhSrjktrCtA3t20YZr
         YzGVnn1ZAYyM+Xh4zJAMhjXdCi5l7owju0Hsw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :cc:content-type:content-transfer-encoding;
        b=Ctu9sB1b/7NOYkspYrl5DwSWMhtZx20tSG7yGjMhY4ppWGelq9wPqqq9bh0YYoCPL3
         950xeeNarOVZe5Z336jrvpyzgTsECaJoN3n+Rd23/UqwI6Q3pawrb6DKhHRKUoccR6ia
         HvZajwxrqMMdo7J5ZgV9lJaLtnSNhZ3Tu18f4=
Received: by 10.142.144.20 with SMTP id r20mr786833wfd.76.1302538350052; Mon,
 11 Apr 2011 09:12:30 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.68.56.233 with HTTP; Mon, 11 Apr 2011 09:12:10 -0700 (PDT)
In-Reply-To: <AA3E0D0D-2227-4ED1-85B8-01006D2B2D18@mac.com>
References: <AANLkTi=owD6w2UA+-eCdDP67F=2NMChHLJxi1HYY+RG6@mail.gmail.com>
 <BANLkTim7sijRYVRmXx3LymEfskzBJL978w@mail.gmail.com> <AA3E0D0D-2227-4ED1-85B8-01006D2B2D18@mac.com>
From: Tom White <tom.e.white@gmail.com>
Date: Mon, 11 Apr 2011 09:12:10 -0700
Message-ID: <BANLkTim=uzJYGmk1dSXjnq=wsfMjS5w0XA@mail.gmail.com>
Subject: Re: Maintenance of Hadoop 0.21 branch?
To: general@hadoop.apache.org
Cc: Nigel Daley <ndaley@mac.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Sat, Apr 9, 2011 at 9:53 PM, Nigel Daley <ndaley@mac.com> wrote:
> I'm fine w/ abandoning but that needs to be clearly communicated with cha=
nges to the website. =A0This is complicated by the fact that it looks like =
0.21 docs are the only ones on the website for hdfs and mapreduce (which se=
ems like a bug):
> http://hadoop.apache.org/hdfs/docs/
> http://hadoop.apache.org/mapreduce/docs/
> Common has a bunch of back revs:
> http://hadoop.apache.org/common/docs/

This is an artifact of the project split, since there were no separate
HDFS and MapReduce projects before release 0.21.0. The documentation
for HDFS and MapReduce was all in Common/Core for 0.20.2 and lower.

The websites (http://hadoop.apache.org/hdfs/,
http://hadoop.apache.org/mapreduce/) link to previous versions, but do
we need to add links under http://hadoop.apache.org/hdfs/docs and
http://hadoop.apache.org/mapreduce/docs/ too?

Tom

>
> Nige
>
> On Apr 5, 2011, at 1:00 PM, Tom White wrote:
>
>> I'm not currently planning on doing a 0.21.1 release. I agree with
>> Todd that it would be better to spend time getting 0.22 ready for a
>> release.
>>
>> Cheers,
>> Tom
>>
>> On Wed, Mar 30, 2011 at 2:41 PM, Todd Lipcon <todd@cloudera.com> wrote:
>>> Hi all,
>>>
>>> Some recent discussion on HDFS-1786 has raised an interesting question:=
 does
>>> anyone plan on maintaining the 0.21 branch and eventually releasing an
>>> 0.21.1? Should we bother to commit bug fixes to this branch?
>>>
>>> It seems to me that our time would be better spent getting 0.22 and tru=
nk
>>> back to a green state so we can talk about releasing them, rather than
>>> applying patches to a branch with no releases planned.
>>>
>>> Of course a decision now doesn't preclude anyone from stepping up later=
,
>>> backporting patches to 0.21, and releasing an 0.21.1.
>>>
>>> Thanks
>>> -Todd
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>>
>
>

From general-return-3252-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 11 19:28:14 2011
Return-Path: <general-return-3252-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44340 invoked from network); 11 Apr 2011 19:28:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Apr 2011 19:28:14 -0000
Received: (qmail 27364 invoked by uid 500); 11 Apr 2011 19:28:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27299 invoked by uid 500); 11 Apr 2011 19:28:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27291 invoked by uid 99); 11 Apr 2011 19:28:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Apr 2011 19:28:12 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mlortiz@uci.cu designates 200.55.140.180 as permitted sender)
Received: from [200.55.140.180] (HELO mx3.uci.cu) (200.55.140.180)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 11 Apr 2011 19:28:06 +0000
Received: (qmail 11864 invoked by uid 507); 11 Apr 2011 19:27:39 -0000
Received: from 10.0.0.185 by ns3.uci.cu (envelope-from <mlortiz@uci.cu>, uid 501) with qmail-scanner-2.01st 
 (avp: 5.0.2.0. spamassassin: 3.0.6. perlscan: 2.01st.  
 Clear:RC:1(10.0.0.185):. 
 Processed in 0.024092 secs); 11 Apr 2011 19:27:39 -0000
Received: from unknown (HELO ucimail4.uci.cu) (10.0.0.185)
  by 0 with SMTP; 11 Apr 2011 19:27:39 -0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by ucimail4.uci.cu (Postfix) with ESMTP id 60DC313246F7
	for <general@hadoop.apache.org>; Mon, 11 Apr 2011 15:27:38 -0400 (CDT)
X-Virus-Scanned: amavisd-new at uci.cu
Received: from ucimail4.uci.cu ([127.0.0.1])
	by localhost (ucimail4.uci.cu [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id 55kKK3ZTumo7; Mon, 11 Apr 2011 15:27:35 -0400 (CDT)
Received: from [10.36.18.29] (skynet2010.uci.cu [10.36.18.29])
	by ucimail4.uci.cu (Postfix) with ESMTP id C8E3B1324687
	for <general@hadoop.apache.org>; Mon, 11 Apr 2011 15:27:34 -0400 (CDT)
Message-ID: <4DA3567A.8080608@uci.cu>
Date: Mon, 11 Apr 2011 15:28:58 -0400
From: Marcos Ortiz <mlortiz@uci.cu>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; es-ES; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Mailing list of the Apache Mahout Project
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit

Regards to all
I´m trying to subscribe to the mailing list of the Apache Mahout project 
but it fails.
Which is the right list?
I tried with with mahout-user-subscribe@mahout.apache.org and it failed.

Thanks a lot

-- 
Marcos Luís Ortíz Valmaseda
  Software Engineer (Large-Scaled Distributed Systems)
  University of Information Sciences,
  La Habana, Cuba
  Linux User # 418229


From general-return-3253-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 12 17:49:25 2011
Return-Path: <general-return-3253-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66061 invoked from network); 12 Apr 2011 17:49:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Apr 2011 17:49:25 -0000
Received: (qmail 79023 invoked by uid 500); 12 Apr 2011 17:49:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78872 invoked by uid 500); 12 Apr 2011 17:49:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78753 invoked by uid 99); 12 Apr 2011 17:49:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Apr 2011 17:49:21 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Apr 2011 17:49:16 +0000
Received: by vws7 with SMTP id 7so8182378vws.35
        for <general@hadoop.apache.org>; Tue, 12 Apr 2011 10:48:55 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:cc:content-type
         :content-transfer-encoding;
        bh=9ZcFcDbwZbWMX+6++q16Pvqa+trHvLMScIb8iktvVU0=;
        b=B0brrWsqmf/PEozUXBd1dJnK+fAFshZHSD56Ch8p9RPAQUXtjmso2+/mMeCdH6motD
         T/noAou1ozQojOfsf2VAwURfrTlUB2Hh3/3GLJvirkZpWxXoTBlMCE9Gzr7G7sBDKpsx
         ayQfQ40v7WIXmJqCGJJyKOltsbJ4ctJVEVFVw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:cc:content-type
         :content-transfer-encoding;
        b=p76/FNXzXe8gkXcty1AxyuHcIwuyuJyy1ZzAI0k/Tw1tHI9eG+i/2obVEcFx/lCJlO
         0xZgEYoWbeRIxeTsCG6pgy70sy9b1Qwe200Czb8XEX6GS8fPFqbaFFaj3fdcFpW6Gfyd
         H39zwW7s5iepPMzGg9Lie7z6ESzxlzcZwN7Gk=
MIME-Version: 1.0
Received: by 10.52.89.82 with SMTP id bm18mr4351185vdb.99.1302630535306; Tue,
 12 Apr 2011 10:48:55 -0700 (PDT)
Sender: saint.ack@gmail.com
Received: by 10.52.166.198 with HTTP; Tue, 12 Apr 2011 10:48:55 -0700 (PDT)
In-Reply-To: <BANLkTimGzxOWa83ay9kqALs-4VKVUoGSvw@mail.gmail.com>
References: <BANLkTincvMXiCdEn=vjMCu5xg7d_1Syk8g@mail.gmail.com>
	<BANLkTikLb1s-L5QLs+NhJGNECbbVnYTXuw@mail.gmail.com>
	<BANLkTimGzxOWa83ay9kqALs-4VKVUoGSvw@mail.gmail.com>
Date: Tue, 12 Apr 2011 10:48:55 -0700
X-Google-Sender-Auth: tCedyd0Fo5aPHTOgIUN2EI2Lp6s
Message-ID: <BANLkTi=GvEA17Ba3k-z94PL73xHDnFMrYw@mail.gmail.com>
Subject: Re: ANN: HBase 0.90.2 is available for download
From: Stack <stack@duboce.net>
To: Lior Schachter <liors@infolinks.com>
Cc: general@hadoop.apache.org, user@hbase.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Its up in the release repository now.  Let me know if any issue with
it (I'm still not sure how I made it happen -- I need to do this
process again soon).
Thanks for your patience,
St.Ack

On Mon, Apr 11, 2011 at 8:25 AM, Stack <stack@duboce.net> wrote:
> Lior and Joe:
>
> Sorry for the mvn lag. =A0The mvn deploy system is smarter than me. =A0Th=
e
> deploy requires four full builds of HBase running all tests. =A0I
> inevitably get distracted and forget the process or else I am demented
> and answer one of the questions just off and then I have to start
> over. =A0I'm not sure how it all works either and should take the time
> to figure it (but again, distraction, dementia take over...).
>
> It should be up soon. =A0I'm on it.
>
> St.Ack
>
>
> On Sat, Apr 9, 2011 at 11:57 PM, Lior Schachter <liors@infolinks.com> wro=
te:
>> Hi Stack,
>>
>> We already have version 0.90.1 installed on our production cluster and w=
e
>> want to upgrade to 0.90.2.
>> Obviously we should upgrade the hbase-0.90.1.jar with the new jar.
>> Should we upgrade other libraries/configuration files ?
>> Is there a maven repository with the new version ?
>>
>> Thanks,
>> Lior
>>
>>
>>
>> On Fri, Apr 8, 2011 at 8:39 PM, Stack <stack@duboce.net> wrote:
>>>
>>> The Apache HBase team is happy to announce the general availability of
>>> HBase
>>> 0.90.2, available from your Apache mirror of choice:
>>>
>>> =A0http://www.apache.org/dyn/closer.cgi/hbase/
>>>
>>> HBase 0.90.2 is a maintenance release that fixes several important bugs
>>> since version 0.90.1, while retaining API and data compatibility. The
>>> release notes may be found on the Apache JIRA:
>>>
>>> =A0http://su.pr/1F6omq
>>>
>>> Users upgrading from HBase 0.90.0 or 0.90.1 may upgrade clients and
>>> servers
>>> separately, though it is recommended that both be upgraded. If upgradin=
g
>>> from a version of HBase prior to 0.90.0, please read the notes
>>> accompanying
>>> that release:
>>>
>>> =A0http://osdir.com/ml/general-hadoop-apache/2011-01/msg00208.html
>>>
>>> As always, many thanks to those who contributed to this release!
>>>
>>> -The HBase Team
>>
>>
>

From general-return-3254-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 13 23:17:30 2011
Return-Path: <general-return-3254-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55145 invoked from network); 13 Apr 2011 23:17:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Apr 2011 23:17:30 -0000
Received: (qmail 81926 invoked by uid 500); 13 Apr 2011 23:17:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81856 invoked by uid 500); 13 Apr 2011 23:17:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81848 invoked by uid 99); 13 Apr 2011 23:17:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Apr 2011 23:17:28 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [98.136.86.194] (HELO web65903.mail.ac4.yahoo.com) (98.136.86.194)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 13 Apr 2011 23:17:20 +0000
Received: (qmail 76800 invoked by uid 60001); 13 Apr 2011 23:16:59 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1302736619; bh=nZ8onj60b78UBtoksa9KtVF2eRM+Nafsvqd2TQ1n4lQ=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=sIvYDlUEzt9WM7XV8K0JzcGSolYspzLf3kPstV0Sq8hKDF3pvK1vQRNs/Kdoaf4cJO4XntyoRZZoiRmevcx82VBT5nfC17+4ijVXVdwdhygot3WRJzL8BUX0kYV4DloEfYfrsa38od1tUHjM12o99U26PdEyY3hB4OvXRPKEibU=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=NNZUSgLY0+BAQ2zlDrU5bHHO9dlbBpvigYP0yWEj/I+DLQK/HiY2HiSS7mH/FzScsFjEhzUWr8DdVLTGDbxsjK3ccUPQaB0fKiVA9dv3FMBlF9ckmVeTDRsuCNk3xG8P4tCHZfrUdSx/6Xuf07nQ0bg65/DDdYUFSwk+2PHBf+k=;
Message-ID: <834460.72891.qm@web65903.mail.ac4.yahoo.com>
X-YMail-OSG: 6xJKUiwVM1mLyvrgdOW08xj59YmxozjaObqBTK2hBpYFauM
 dxNv04wBKTOzwP9wMWGKJ8CJK7SVrHjAQEeq_oBTEpeDcdxCd6GSsmRJ1TsV
 RvR1Ngqyf4hS2diFF2rw_MFHrS87Ui4dRj_2K5fnstblY85OeVG0Nh8rPr0f
 wIfsu67Xb87gNyUxSQpPNewpvuT4iT.YWSqSoNxUVI0oSRtVSV5UcSZ_Fxw6
 l45UpUavKsBsZwt2UL_xsS24XbVHSTqzWr6wYYAn8GNSn55TWBqukX30OxzV
 m3y3cu7PEq37pYso.8ui_Bwy36dfkV_UR_T1sSwGpTbfh0Y.KzzbI__BgTwG
 z78UB5nUFVJM3BBxaJ4pN36LoCvFXImBjc0riov9WyPBnjSnDW7B_CgskDBS
 Bf7qRCILfFbG_nyaL8ZSkbsTR6tojk4Ry
Received: from [209.131.62.113] by web65903.mail.ac4.yahoo.com via HTTP; Wed, 13 Apr 2011 16:16:59 PDT
X-Mailer: YahooMailRC/559 YahooMailWebService/0.8.109.295617
Date: Wed, 13 Apr 2011 16:16:59 -0700 (PDT)
From: "Tsz Wo \(Nicholas\), Sze" <s29752-hadoopgeneral@yahoo.com>
Subject: [ANNOUNCE] Hadoop Common/HDFS/MapReduce Committer: Koji Noguchi
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1884554487-1302736619=:72891"

--0-1884554487-1302736619=:72891
Content-Type: text/plain; charset=us-ascii

Hi all,

The Hadoop PMC has elected Koji Noguchi as a committer of Hadoop 
Common/HDFS/MapReduce and he has accepted.  Welcome aboard Koji!

Thanks.
Tsz-Wo (Nicholas) Sze

--0-1884554487-1302736619=:72891--

From general-return-3255-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 13 23:38:04 2011
Return-Path: <general-return-3255-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39251 invoked from network); 13 Apr 2011 23:38:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Apr 2011 23:38:03 -0000
Received: (qmail 97839 invoked by uid 500); 13 Apr 2011 23:38:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97788 invoked by uid 500); 13 Apr 2011 23:38:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97780 invoked by uid 99); 13 Apr 2011 23:38:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Apr 2011 23:38:02 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 13 Apr 2011 23:38:00 +0000
Received: (qmail 36576 invoked by uid 99); 13 Apr 2011 23:37:38 -0000
Received: from localhost.apache.org (HELO awittena-mn.linkedin.biz) (127.0.0.1)
  (smtp-auth username aw, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Apr 2011 23:37:38 +0000
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [ANNOUNCE] Hadoop Common/HDFS/MapReduce Committer: Koji Noguchi
From: Allen Wittenauer <aw@apache.org>
In-Reply-To: <834460.72891.qm@web65903.mail.ac4.yahoo.com>
Date: Wed, 13 Apr 2011 16:37:37 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <66E52B80-612A-431B-A556-F78693070040@apache.org>
References: <834460.72891.qm@web65903.mail.ac4.yahoo.com>
To: <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org


On Apr 13, 2011, at 4:16 PM, Tsz Wo (Nicholas), Sze wrote:

> Hi all,
>=20
> The Hadoop PMC has elected Koji Noguchi as a committer of Hadoop=20
> Common/HDFS/MapReduce and he has accepted.  Welcome aboard Koji!


	Yay! Congrats!

	Now there are even more people who Nicolas can attack about how =
they aren't as awesome as he is.



From general-return-3256-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 13 23:52:53 2011
Return-Path: <general-return-3256-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85173 invoked from network); 13 Apr 2011 23:52:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Apr 2011 23:52:53 -0000
Received: (qmail 14451 invoked by uid 500); 13 Apr 2011 23:52:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14401 invoked by uid 500); 13 Apr 2011 23:52:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14393 invoked by uid 99); 13 Apr 2011 23:52:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Apr 2011 23:52:51 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mlortiz@uci.cu designates 200.55.140.180 as permitted sender)
Received: from [200.55.140.180] (HELO mx3.uci.cu) (200.55.140.180)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 13 Apr 2011 23:52:45 +0000
Received: (qmail 6740 invoked by uid 507); 13 Apr 2011 23:52:19 -0000
Received: from 10.0.0.185 by ns3.uci.cu (envelope-from <mlortiz@uci.cu>, uid 501) with qmail-scanner-2.01st 
 (avp: 5.0.2.0. spamassassin: 3.0.6. perlscan: 2.01st.  
 Clear:RC:1(10.0.0.185):. 
 Processed in 0.025387 secs); 13 Apr 2011 23:52:19 -0000
Received: from unknown (HELO ucimail4.uci.cu) (10.0.0.185)
  by 0 with SMTP; 13 Apr 2011 23:52:19 -0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by ucimail4.uci.cu (Postfix) with ESMTP id 71DF81324957;
	Wed, 13 Apr 2011 19:52:19 -0400 (CDT)
X-Virus-Scanned: amavisd-new at uci.cu
Received: from ucimail4.uci.cu ([127.0.0.1])
	by localhost (ucimail4.uci.cu [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id mmPgYTw57Nk7; Wed, 13 Apr 2011 19:52:19 -0400 (CDT)
Received: from [10.8.46.154] (unknown [10.8.46.154])
	by ucimail4.uci.cu (Postfix) with ESMTP id 3EB7D1324954;
	Wed, 13 Apr 2011 19:52:19 -0400 (CDT)
Message-ID: <4DA63E8F.4030600@uci.cu>
Date: Wed, 13 Apr 2011 19:53:43 -0430
From: Marcos Ortiz <mlortiz@uci.cu>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.14) Gecko/20110223 Lightning/1.0b2 Thunderbird/3.1.8
MIME-Version: 1.0
To: general@hadoop.apache.org
CC: "Tsz Wo \(Nicholas\), Sze" <s29752-hadoopgeneral@yahoo.com>
Subject: Re: [ANNOUNCE] Hadoop Common/HDFS/MapReduce Committer: Koji Noguchi
References: <834460.72891.qm@web65903.mail.ac4.yahoo.com>
In-Reply-To: <834460.72891.qm@web65903.mail.ac4.yahoo.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit

On 04/13/2011 06:46 PM, Tsz Wo (Nicholas), Sze wrote:
> Hi all,
>
> The Hadoop PMC has elected Koji Noguchi as a committer of Hadoop
> Common/HDFS/MapReduce and he has accepted.  Welcome aboard Koji!
>
> Thanks.
> Tsz-Wo (Nicholas) Sze
>
Well, congratulations, Koji.
That's one of my dreams.
Regards

-- 
Marcos Luís Ortíz Valmaseda
  Software Engineer (Large-Scaled Distributed Systems)
  University of Information Sciences,
  La Habana, Cuba
  Linux User # 418229


From general-return-3257-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 13 23:55:26 2011
Return-Path: <general-return-3257-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85818 invoked from network); 13 Apr 2011 23:55:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Apr 2011 23:55:26 -0000
Received: (qmail 16314 invoked by uid 500); 13 Apr 2011 23:55:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16258 invoked by uid 500); 13 Apr 2011 23:55:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16250 invoked by uid 99); 13 Apr 2011 23:55:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Apr 2011 23:55:24 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Apr 2011 23:55:17 +0000
Received: from sp1-ex07cas01.ds.corp.yahoo.com (sp1-ex07cas01.ds.corp.yahoo.com [216.252.116.137])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p3DNscI4043726
	for <general@hadoop.apache.org>; Wed, 13 Apr 2011 16:54:38 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1302738878;
	bh=3HtETb3z61zrqpnfXMnWvR/EYX1cLhnPwk7dSbkws1k=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:MIME-Version;
	b=wOSaQjQ+q41al+LN9PqfbgABTi0crme45J/w1mzh3hYuDSo9BpxfUKJYY6s/fWbab
	 XCXpAFA1n4qVYcGwmkOVSNuJp5ULHqi2fgv1UtsTkRcuBWpmcOyLqBPEe5fYEYvQSm
	 uuqhqa4VDtQDSSfXblT2bK8PAm1pylwzk0np/F3w=
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas01.ds.corp.yahoo.com ([216.252.116.137]) with mapi; Wed, 13 Apr
 2011 16:54:37 -0700
From: Suresh Srinivas <sureshms@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 13 Apr 2011 16:54:36 -0700
Subject: Re: [ANNOUNCE] Hadoop Common/HDFS/MapReduce Committer: Koji Noguchi
Thread-Topic: [ANNOUNCE] Hadoop Common/HDFS/MapReduce Committer: Koji Noguchi
Thread-Index: Acv6M93DEZaDcUSeQ8uCB8lmxGfFyAAAkbe8
Message-ID: <C9CB85CC.19676%sureshms@yahoo-inc.com>
In-Reply-To: <66E52B80-612A-431B-A556-F78693070040@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-Entourage/13.8.0.101117
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

I am sure Koji is reasonable and has a way about him that will not drive
Nicolas to that :-)

Welcome Koji!=20


On 4/13/11 4:37 PM, "Allen Wittenauer" <aw@apache.org> wrote:

>=20
> On Apr 13, 2011, at 4:16 PM, Tsz Wo (Nicholas), Sze wrote:
>=20
>> Hi all,
>>=20
>> The Hadoop PMC has elected Koji Noguchi as a committer of Hadoop
>> Common/HDFS/MapReduce and he has accepted.  Welcome aboard Koji!
>=20
>=20
> Yay! Congrats!
>=20
> Now there are even more people who Nicolas can attack about how they aren=
't as
> awesome as he is.
>=20
>=20


From general-return-3258-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 14 02:51:28 2011
Return-Path: <general-return-3258-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 65107 invoked from network); 14 Apr 2011 02:51:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Apr 2011 02:51:27 -0000
Received: (qmail 24728 invoked by uid 500); 14 Apr 2011 02:51:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24465 invoked by uid 500); 14 Apr 2011 02:51:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24457 invoked by uid 99); 14 Apr 2011 02:51:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Apr 2011 02:51:24 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [209.191.69.25] (HELO web30008.mail.mud.yahoo.com) (209.191.69.25)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 14 Apr 2011 02:51:16 +0000
Received: (qmail 60681 invoked by uid 60001); 14 Apr 2011 02:50:52 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1302749452; bh=DZ4Gi3LoESJanXs5Y8kwTQz3QuWLZwJEZYX3/j4aHGc=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=u+RmQnqquq+t3lfzZFF0uFV7rAOyZDaWGtA2cCMNPHjc2Cm2FFoUZWQiWiOsGwluJZHiFSGgcHUFepw2JdiBNAH7+R9QBn/FWirHor2AOqVKlzV5VsfXBhzHXX/QdQF/O9jI0v0A1/A2hp7csOVVVJqX3XuI/Ss5/Tsgv9SF/sw=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding;
  b=e3xGAV8rViBoyDK7z1Lh88gqXF8R/JcwIFgPaLhL52BJk3GpM8gDXlkYt1Tr5Cdv1r38VzYVu7bj/r/l9zm0shc6GnbimDW3edO60eVeGc5s2WGxE1HfMayjvZ/ukBKfkN5M85mYy2L6Dt3IRb0O+oSAYvDNHyOjj5SsE/FIwJ4=;
Message-ID: <861360.49994.qm@web30008.mail.mud.yahoo.com>
X-YMail-OSG: 2DbY.54VM1lU3msrPlte53_tqTKbSXZfz88V9j512p15iWU
 42w_7ruPdag4ZOmiweA68CrlD2Kkhs1vHcxfy1Crt4Nnm4AC.FCYRVd93oVU
 mjLqb6GVjjGNwywuQSn_e2R.nHXh6TrDccenxwzPZB2ZYP.zu_yfoYP153Fq
 t92jD772YIMJJkLJRS8R_NL.8Mlz4eR5wuth1Tn_A1k45C64F28bmTDVpMzM
 G5Ob3zRVacVm0xIFpHI2YgqZCgDm6oMZqd5Y2Dzhht7moUo6pRyYrKpgagZX
 HB.C8ml_.UvBilBV__uzkzZYijZWR8cgC04AaeaPPeT2Sb0NBf_eRgyYLpWw
 Xh.x7WZ7yRmpyKg5XlPx9F7jrRjZYrmJxGlBf_rS_DZDea3gQpx.ItQXGQ85
 MpxsptHQrRV1d
Received: from [24.6.189.183] by web30008.mail.mud.yahoo.com via HTTP; Wed, 13 Apr 2011 19:50:52 PDT
X-Mailer: YahooMailWebService/0.8.109.295617
References: <834460.72891.qm@web65903.mail.ac4.yahoo.com> <4DA63E8F.4030600@uci.cu>
Date: Wed, 13 Apr 2011 19:50:52 -0700 (PDT)
From: Debabrata Biswas <ds_biswas@yahoo.com>
Reply-To: Debabrata Biswas <ds_biswas@yahoo.com>
Subject: Re: [ANNOUNCE] Hadoop Common/HDFS/MapReduce Committer: Koji Noguchi
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <4DA63E8F.4030600@uci.cu>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Congratulations=A0Koji Noguchi. It's a great accolade.=0ABest Regards,=0A-D=
eb=0A=0A=0A=0A----- Original Message -----=0AFrom: Marcos Ortiz <mlortiz@uc=
i.cu>=0ATo: general@hadoop.apache.org=0ACc: "Tsz Wo (Nicholas), Sze" <s2975=
2-hadoopgeneral@yahoo.com>=0ASent: Wednesday, April 13, 2011 5:23 PM=0ASubj=
ect: Re: [ANNOUNCE] Hadoop Common/HDFS/MapReduce Committer: Koji Noguchi=0A=
=0AOn 04/13/2011 06:46 PM, Tsz Wo (Nicholas), Sze wrote:=0A> Hi all,=0A> =
=0A> The Hadoop PMC has elected Koji Noguchi as a committer of Hadoop=0A> C=
ommon/HDFS/MapReduce and he has accepted.=A0 Welcome aboard Koji!=0A> =0A> =
Thanks.=0A> Tsz-Wo (Nicholas) Sze=0A> =0AWell, congratulations, Koji.=0ATha=
t's one of my dreams.=0ARegards=0A=0A-- Marcos Lu=EDs Ort=EDz Valmaseda=0AS=
oftware Engineer (Large-Scaled Distributed Systems)=0AUniversity of Informa=
tion Sciences,=0ALa Habana, Cuba=0ALinux User # 418229

From general-return-3259-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 14 07:04:42 2011
Return-Path: <general-return-3259-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85119 invoked from network); 14 Apr 2011 07:04:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Apr 2011 07:04:42 -0000
Received: (qmail 63925 invoked by uid 500); 14 Apr 2011 07:04:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63871 invoked by uid 500); 14 Apr 2011 07:04:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63848 invoked by uid 99); 14 Apr 2011 07:04:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Apr 2011 07:04:38 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Apr 2011 07:04:30 +0000
Received: from SP2-EX07CAS03.ds.corp.yahoo.com (sp2-ex07cas03.corp.sp2.yahoo.com [98.137.59.35])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p3E73cqY040842
	for <general@hadoop.apache.org>; Thu, 14 Apr 2011 00:03:38 -0700 (PDT)
Received: from SP2-EX07VS04.ds.corp.yahoo.com ([98.137.59.33]) by
 SP2-EX07CAS03.ds.corp.yahoo.com ([98.137.59.35]) with mapi; Thu, 14 Apr 2011
 00:03:37 -0700
From: Koji Noguchi <knoguchi@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 14 Apr 2011 00:03:34 -0700
Subject: Re: [ANNOUNCE] Hadoop Common/HDFS/MapReduce Committer: Koji Noguchi
Thread-Topic: [ANNOUNCE] Hadoop Common/HDFS/MapReduce Committer: Koji Noguchi
Thread-Index: Acv6M93DEZaDcUSeQ8uCB8lmxGfFyAAAkbe8AA77Q0M=
Message-ID: <C9CBEA56.145EB%knoguchi@yahoo-inc.com>
In-Reply-To: <C9CB85CC.19676%sureshms@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
user-agent: Microsoft-Entourage/13.9.0.110114
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks Everyone! =20

Koji

On 4/13/11 4:54 PM, "Suresh Srinivas" <sureshms@yahoo-inc.com> wrote:

> I am sure Koji is reasonable and has a way about him that will not drive
> Nicolas to that :-)
>=20
> Welcome Koji!=20
>=20
>=20
> On 4/13/11 4:37 PM, "Allen Wittenauer" <aw@apache.org> wrote:
>=20
>>=20
>> On Apr 13, 2011, at 4:16 PM, Tsz Wo (Nicholas), Sze wrote:
>>=20
>>> Hi all,
>>>=20
>>> The Hadoop PMC has elected Koji Noguchi as a committer of Hadoop
>>> Common/HDFS/MapReduce and he has accepted.  Welcome aboard Koji!
>>=20
>>=20
>> Yay! Congrats!
>>=20
>> Now there are even more people who Nicolas can attack about how they are=
n't
>> as
>> awesome as he is.
>>=20
>>=20
>=20


From general-return-3260-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 14 15:04:58 2011
Return-Path: <general-return-3260-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 71436 invoked from network); 14 Apr 2011 15:04:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Apr 2011 15:04:57 -0000
Received: (qmail 23713 invoked by uid 500); 14 Apr 2011 15:04:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23612 invoked by uid 500); 14 Apr 2011 15:04:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23604 invoked by uid 99); 14 Apr 2011 15:04:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Apr 2011 15:04:55 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Apr 2011 15:04:49 +0000
Received: from [192.168.1.71] (snvvpn4-10-72-168-c84.hq.corp.yahoo.com [10.72.168.84])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p3EF3x1W083638
	for <general@hadoop.apache.org>; Thu, 14 Apr 2011 08:03:59 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1302793440;
	bh=IG/tHaxk/nC2eTr3OgQn01Nyf34dgYfhHVFc17dU0Tg=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=fqWwubtsxrBpVdrDAJe7u/n9Ik9+oKb4DaYnxh7uQw+NJrUR5O0x92e0JTfOMIlHi
	 i6vazrCKtfpR9Y29BjopDJqFmUrH3N7KSnayYFYymRL8nSCZmY1f4kTBxRSMgfpL7W
	 ef/Sc+ggz7Gm8VKjKikfaNQGCwwh1AqIO1xwfCA4=
Message-Id: <A366CA81-70D7-4D0A-8274-8D369C3CBC2A@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <834460.72891.qm@web65903.mail.ac4.yahoo.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [ANNOUNCE] Hadoop Common/HDFS/MapReduce Committer: Koji Noguchi
Date: Thu, 14 Apr 2011 08:03:59 -0700
References: <834460.72891.qm@web65903.mail.ac4.yahoo.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


On Apr 13, 2011, at 4:16 PM, Tsz Wo (Nicholas), Sze wrote:

> Hi all,
>
> The Hadoop PMC has elected Koji Noguchi as a committer of Hadoop
> Common/HDFS/MapReduce and he has accepted.  Welcome aboard Koji!
>

Congratulations Koji! Thanks for all your efforts, we look forward to  
your continued involvement, especially as the voice of the user.

Arun

From general-return-3261-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 14 19:54:27 2011
Return-Path: <general-return-3261-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 62705 invoked from network); 14 Apr 2011 19:54:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Apr 2011 19:54:26 -0000
Received: (qmail 97659 invoked by uid 500); 14 Apr 2011 19:54:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97598 invoked by uid 500); 14 Apr 2011 19:54:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97590 invoked by uid 99); 14 Apr 2011 19:54:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Apr 2011 19:54:25 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mbhandarkar@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Apr 2011 19:54:19 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:CC:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:In-Reply-To:Accept-Language:
   Content-Language:X-MS-Has-Attach:X-MS-TNEF-Correlator:
   x-originating-ip:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version;
  b=hL2n4UQEOynRaCrNK4op/fICVVSuXTcbcXHn7z6fZ+5QX1FsHyUm8nh2
   PnjL9lF4oYufLN26ALeMFCZzMy9j7SY/RCCKqO8ct4fu3UyinOOOciW5g
   oSDi5VdvsZg+y8x;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=mbhandarkar@linkedin.com; q=dns/txt;
  s=proddkim; t=1302810859; x=1334346859;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Milind=20Bhandarkar=20<mbhandarkar@linkedin.com>
   |Subject:=20Re:=20[ANNOUNCE]=20Hadoop=20Common/HDFS/MapRe
   duce=20Committer:=20Koji=20Noguchi|Date:=20Thu,=2014=20Ap
   r=202011=2019:56:53=20+0000|Message-ID:=20<C9CC9ECA.21B6%
   mbhandarkar@linkedin.com>|To:=20"general@hadoop.apache.or
   g"=20<general@hadoop.apache.org>|CC:=20"Tsz=20Wo=20(Nicho
   las),=20Sze"=20<s29752-hadoopgeneral@yahoo.com>
   |MIME-Version:=201.0|Content-Transfer-Encoding:=20quoted-
   printable|Content-ID:=20<78A03EA91EBD444F80574FB11CC3E954
   @linkedin.com>|In-Reply-To:=20<4DA63E8F.4030600@uci.cu>;
  bh=j0M42xzT61oqvrNoNHILBYOE0s7A7Yb3i6fr4/ZZMqE=;
  b=Ez0CK016U0Q84frFCY/7zGxkCC4Hm44HepUqvYM6fDoSe0cMAxTH/Vef
   DAqBBRjWOAtytjnnajREfA0GcN0vgzJHrgI0Qm9kAWCg/9eF9EpLUIK2m
   GCjaXRaGguRd/RZ;
X-IronPort-AV: E=Sophos;i="4.64,213,1301900400"; 
   d="scan'208";a="22176927"
Received: from ESV4-EXC02.linkedin.biz ([fe80::4d74:48bd:e0bd:13ee]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi id 14.01.0251.000; Thu,
 14 Apr 2011 12:56:53 -0700
From: Milind Bhandarkar <mbhandarkar@linkedin.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "Tsz Wo (Nicholas), Sze" <s29752-hadoopgeneral@yahoo.com>
Subject: Re: [ANNOUNCE] Hadoop Common/HDFS/MapReduce Committer: Koji Noguchi
Thread-Topic: [ANNOUNCE] Hadoop Common/HDFS/MapReduce Committer: Koji Noguchi
Thread-Index: AQHL+jFdPG6mfGGvm0iJtKGz5+/GfZRc9WGAgADRm4A=
Date: Thu, 14 Apr 2011 19:56:53 +0000
Message-ID: <C9CC9ECA.21B6%mbhandarkar@linkedin.com>
In-Reply-To: <4DA63E8F.4030600@uci.cu>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="iso-8859-1"
Content-ID: <78A03EA91EBD444F80574FB11CC3E954@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Congratulations Koji !

- milind
--=20
Milind Bhandarkar
mbhandarkar@linkedin.com
+1-650-776-3167






On 4/13/11 5:23 PM, "Marcos Ortiz" <mlortiz@uci.cu> wrote:

>On 04/13/2011 06:46 PM, Tsz Wo (Nicholas), Sze wrote:
>> Hi all,
>>
>> The Hadoop PMC has elected Koji Noguchi as a committer of Hadoop
>> Common/HDFS/MapReduce and he has accepted.  Welcome aboard Koji!
>>
>> Thanks.
>> Tsz-Wo (Nicholas) Sze
>>
>Well, congratulations, Koji.
>That's one of my dreams.
>Regards
>
>--=20
>Marcos Lu=EDs Ort=EDz Valmaseda
>  Software Engineer (Large-Scaled Distributed Systems)
>  University of Information Sciences,
>  La Habana, Cuba
>  Linux User # 418229
>


From general-return-3262-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 15 04:54:41 2011
Return-Path: <general-return-3262-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17312 invoked from network); 15 Apr 2011 04:54:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Apr 2011 04:54:41 -0000
Received: (qmail 90076 invoked by uid 500); 15 Apr 2011 04:54:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90029 invoked by uid 500); 15 Apr 2011 04:54:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90021 invoked by uid 99); 15 Apr 2011 04:54:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Apr 2011 04:54:35 +0000
X-ASF-Spam-Status: No, hits=3.3 required=5.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Apr 2011 04:54:28 +0000
Received: from EGL-EX07CAS03.ds.corp.yahoo.com (egl-ex07cas03.eglbp.corp.yahoo.com [203.83.248.219])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p3F4rhGB052711
	for <general@hadoop.apache.org>; Thu, 14 Apr 2011 21:53:44 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1302843224;
	bh=vAGOUsNor+ji7Kf6T4cpuD6iqrlO5VhQ0cYpP7WmoxE=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 MIME-Version;
	b=tsCAxiw+5FpE4HlA4vprx5GC1PUrPZgNdgjVZAcSLKUH/S56djxPtFb1h2LEtIi9t
	 Mxx4Dn+sdL2NhRaXWDviH0krrLzD6x8fe9sa73+RKJfWbmzUJzoXiaZi1h9xCYMaQ3
	 v54L19bBR/26tQ16lX68n/eV7Tq1iMg3EaUqtVIE=
Received: from EGL-EX07VS01.ds.corp.yahoo.com ([203.83.248.205]) by
 EGL-EX07CAS03.ds.corp.yahoo.com ([203.83.248.219]) with mapi; Fri, 15 Apr
 2011 10:23:42 +0530
From: "Ankur C. Goel" <gankur@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 15 Apr 2011 10:23:41 +0530
Subject: Re: [ANNOUNCE] Hadoop Common/HDFS/MapReduce Committer: Koji Noguchi
Thread-Topic: [ANNOUNCE] Hadoop Common/HDFS/MapReduce Committer: Koji Noguchi
Thread-Index: AQHL+jFdPG6mfGGvm0iJtKGz5+/GfZRc9WGAgADRm4CAAJbPOQ==
Message-ID: <C9CDCD2D.102AD%gankur@yahoo-inc.com>
In-Reply-To: <C9CC9ECA.21B6%mbhandarkar@linkedin.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C9CDCD2D102ADgankuryahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C9CDCD2D102ADgankuryahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


Congratulations Koji !

Regards
-@nkur

On 4/15/11 1:26 AM, "Milind Bhandarkar" <mbhandarkar@linkedin.com> wrote:

Congratulations Koji !

- milind
--
Milind Bhandarkar
mbhandarkar@linkedin.com
+1-650-776-3167






On 4/13/11 5:23 PM, "Marcos Ortiz" <mlortiz@uci.cu> wrote:

>On 04/13/2011 06:46 PM, Tsz Wo (Nicholas), Sze wrote:
>> Hi all,
>>
>> The Hadoop PMC has elected Koji Noguchi as a committer of Hadoop
>> Common/HDFS/MapReduce and he has accepted.  Welcome aboard Koji!
>>
>> Thanks.
>> Tsz-Wo (Nicholas) Sze
>>
>Well, congratulations, Koji.
>That's one of my dreams.
>Regards
>
>--
>Marcos Lu=EDs Ort=EDz Valmaseda
>  Software Engineer (Large-Scaled Distributed Systems)
>  University of Information Sciences,
>  La Habana, Cuba
>  Linux User # 418229
>



--_000_C9CDCD2D102ADgankuryahooinccom_--

From general-return-3263-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 15 12:11:40 2011
Return-Path: <general-return-3263-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60510 invoked from network); 15 Apr 2011 12:11:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Apr 2011 12:11:40 -0000
Received: (qmail 70710 invoked by uid 500); 15 Apr 2011 12:11:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70626 invoked by uid 500); 15 Apr 2011 12:11:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70617 invoked by uid 99); 15 Apr 2011 12:11:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Apr 2011 12:11:37 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Apr 2011 12:11:30 +0000
Received: by qwj9 with SMTP id 9so2045050qwj.35
        for <general@hadoop.apache.org>; Fri, 15 Apr 2011 05:11:09 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:from:content-type:content-transfer-encoding
         :subject:date:message-id:to:mime-version:x-mailer;
        bh=5j5Hrlko9sCHp8SOlAtgaJGruDl2Ipg/SwRID79LCr0=;
        b=WOhC7/lwbIXCzP16m689hYDplEIU0LvYLCelh/x+URXFZ5EaSP2dBpouC5xn/oyO5H
         ChfhriAarNQs8tITSaMLb1JFbBAYX66sW6efj7brpgNlRaKGjKcTt3WNv0NgKwLZK4DE
         jgSuhXNgW9ttDZ5mdTkv9k3x43qk7oXFwck8w=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=from:content-type:content-transfer-encoding:subject:date:message-id
         :to:mime-version:x-mailer;
        b=TpTIMp9iBjJYj0ZMjQbxDNiUh6vZe+vpGhlq0wCulrWDJZIeud8mYgEZyn9lgez6wA
         gS8HA0jwpHwhZJlLWWGyROPiVTQcFHvNRyQr+nn2kynvpSM9+8JkiwE16jgrXC8a854l
         QVBWrED/V+8YmMEd7pLargjNMMSzyKNCGGSeo=
Received: by 10.229.118.195 with SMTP id w3mr1366562qcq.203.1302869469674;
        Fri, 15 Apr 2011 05:11:09 -0700 (PDT)
Received: from [10.180.150.19] (h-64-236-128-43.nat.aol.com [64.236.128.43])
        by mx.google.com with ESMTPS id s9sm1901054qco.36.2011.04.15.05.11.08
        (version=TLSv1/SSLv3 cipher=OTHER);
        Fri, 15 Apr 2011 05:11:08 -0700 (PDT)
From: Ian Holsman <hadoop@holsman.net>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: Hadoop 0.20.100
Date: Fri, 15 Apr 2011 08:11:06 -0400
Message-Id: <13A19700-AAB7-4F87-AA51-FEB5A4E2781E@Holsman.NET>
To: general@hadoop.apache.org
Mime-Version: 1.0 (Apple Message framework v1084)
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

hey guys.
are we still planning to build a release for this version? or have we =
skipped it and moved to 0.20.20x ?

TIA
Ian=

From general-return-3264-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 15 15:54:51 2011
Return-Path: <general-return-3264-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98722 invoked from network); 15 Apr 2011 15:54:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Apr 2011 15:54:50 -0000
Received: (qmail 67328 invoked by uid 500); 15 Apr 2011 15:54:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67232 invoked by uid 500); 15 Apr 2011 15:54:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67220 invoked by uid 99); 15 Apr 2011 15:54:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Apr 2011 15:54:49 +0000
X-ASF-Spam-Status: No, hits=-1997.8 required=5.0
	tests=ALL_TRUSTED,HTML_MESSAGE,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 15 Apr 2011 15:54:48 +0000
Received: (qmail 98347 invoked by uid 99); 15 Apr 2011 15:54:27 -0000
Received: from localhost.apache.org (HELO mail-qy0-f169.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Apr 2011 15:54:27 +0000
Received: by qyk2 with SMTP id 2so4915344qyk.14
        for <general@hadoop.apache.org>; Fri, 15 Apr 2011 08:54:26 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.220.149 with SMTP id hy21mr1633252qcb.51.1302882866669;
 Fri, 15 Apr 2011 08:54:26 -0700 (PDT)
Received: by 10.229.250.147 with HTTP; Fri, 15 Apr 2011 08:54:26 -0700 (PDT)
Date: Fri, 15 Apr 2011 08:54:26 -0700
Message-ID: <BANLkTi=ucRDbFh+x+SJ0NagawT1b0GZLgg@mail.gmail.com>
Subject: Re: Hadoop security branch releases (was Hadoop 0.20.100)
From: "Owen O'Malley" <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163630fac373a87c04a0f70ea1

--00163630fac373a87c04a0f70ea1
Content-Type: text/plain; charset=UTF-8

Hi all,
   I've been doing a bit of travelling, but I'm back now. (Just buried in
email, even more than normal.) The general strategy is that the branch
(branch-0.20-security) has the current work on the branch. The releases off
of the security branch are labelled with numbers in the 200 range. We
originally planned on using the 100 range, but I'd like to bring it into
sync with the Yahoo numbering. (The intent was to reduce confusion. The
github push I did last summer, which was the basis for CDH3, was labelled
104. I really don't want inconsistent versions with the same number.) So the
megapatch that Arun pushed was 202. I'm in the last stages of fixing the
contrib unit tests for 203 and hope to be able to call a vote shortly. I
would expect a 204 release next month branching off of branch-0.20-security.

-- Owen

--00163630fac373a87c04a0f70ea1--

From general-return-3265-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 15 17:19:43 2011
Return-Path: <general-return-3265-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73128 invoked from network); 15 Apr 2011 17:19:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Apr 2011 17:19:42 -0000
Received: (qmail 98560 invoked by uid 500); 15 Apr 2011 17:19:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98507 invoked by uid 500); 15 Apr 2011 17:19:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98499 invoked by uid 99); 15 Apr 2011 17:19:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Apr 2011 17:19:41 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 15 Apr 2011 17:19:40 +0000
Received: (qmail 71252 invoked by uid 99); 15 Apr 2011 17:19:19 -0000
Received: from localhost.apache.org (HELO [192.168.42.176]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Apr 2011 17:19:19 +0000
Message-ID: <4DA87E14.8000600@apache.org>
Date: Fri, 15 Apr 2011 10:19:16 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.14) Gecko/20110223 Thunderbird/3.1.8
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Hadoop security branch releases (was Hadoop 0.20.100)
References: <BANLkTi=ucRDbFh+x+SJ0NagawT1b0GZLgg@mail.gmail.com>
In-Reply-To: <BANLkTi=ucRDbFh+x+SJ0NagawT1b0GZLgg@mail.gmail.com>
X-Enigmail-Version: 1.1.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Is the intent for these branches to do anything but mirror Y!'s internal
branches?

Doug

On 04/15/2011 08:54 AM, Owen O'Malley wrote:
> Hi all,
>    I've been doing a bit of travelling, but I'm back now. (Just buried in
> email, even more than normal.) The general strategy is that the branch
> (branch-0.20-security) has the current work on the branch. The releases off
> of the security branch are labelled with numbers in the 200 range. We
> originally planned on using the 100 range, but I'd like to bring it into
> sync with the Yahoo numbering. (The intent was to reduce confusion. The
> github push I did last summer, which was the basis for CDH3, was labelled
> 104. I really don't want inconsistent versions with the same number.) So the
> megapatch that Arun pushed was 202. I'm in the last stages of fixing the
> contrib unit tests for 203 and hope to be able to call a vote shortly. I
> would expect a 204 release next month branching off of branch-0.20-security.
> 
> -- Owen
> 

From general-return-3266-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 15 18:19:38 2011
Return-Path: <general-return-3266-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70534 invoked from network); 15 Apr 2011 18:19:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Apr 2011 18:19:37 -0000
Received: (qmail 35759 invoked by uid 500); 15 Apr 2011 18:19:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 35696 invoked by uid 500); 15 Apr 2011 18:19:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 35688 invoked by uid 99); 15 Apr 2011 18:19:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Apr 2011 18:19:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 15 Apr 2011 18:19:35 +0000
Received: (qmail 68769 invoked by uid 99); 15 Apr 2011 18:19:14 -0000
Received: from localhost.apache.org (HELO mail-iw0-f176.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Apr 2011 18:19:14 +0000
Received: by iwr19 with SMTP id 19so3667950iwr.35
        for <general@hadoop.apache.org>; Fri, 15 Apr 2011 11:19:13 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.140.34 with SMTP id g34mr1782914ibu.195.1302891553762;
 Fri, 15 Apr 2011 11:19:13 -0700 (PDT)
Received: by 10.231.166.76 with HTTP; Fri, 15 Apr 2011 11:19:13 -0700 (PDT)
In-Reply-To: <4DA87E14.8000600@apache.org>
References: <BANLkTi=ucRDbFh+x+SJ0NagawT1b0GZLgg@mail.gmail.com>
	<4DA87E14.8000600@apache.org>
Date: Fri, 15 Apr 2011 11:19:13 -0700
Message-ID: <BANLkTikUBL9x8mqppntUtPA4eNw38UOfcQ@mail.gmail.com>
Subject: Re: Hadoop security branch releases (was Hadoop 0.20.100)
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

The branches are Apache branches. They are intended to be developed
here, as was discussed on general@. For all effects and purposes, Y!
retired its internal branches. If you missed the discussion, it's in
the archives.

This message is about version numbers, and version numbers only. The
branch version numbering follows what Y! has been using internally. It
also does not conflict with the version numbers pushed to github and
absorbed by other consumers of that branch, e.g. CDH and Facebook.

The intent is to reduce confusion among users and release engineers in
organizations that were pulling from the retired repositories. The
intent is to ease their transition (back) to Apache. -C

On Fri, Apr 15, 2011 at 10:19 AM, Doug Cutting <cutting@apache.org> wrote:
> Is the intent for these branches to do anything but mirror Y!'s internal
> branches?
>
> Doug
>
> On 04/15/2011 08:54 AM, Owen O'Malley wrote:
>> Hi all,
>> =A0 =A0I've been doing a bit of travelling, but I'm back now. (Just buri=
ed in
>> email, even more than normal.) The general strategy is that the branch
>> (branch-0.20-security) has the current work on the branch. The releases =
off
>> of the security branch are labelled with numbers in the 200 range. We
>> originally planned on using the 100 range, but I'd like to bring it into
>> sync with the Yahoo numbering. (The intent was to reduce confusion. The
>> github push I did last summer, which was the basis for CDH3, was labelle=
d
>> 104. I really don't want inconsistent versions with the same number.) So=
 the
>> megapatch that Arun pushed was 202. I'm in the last stages of fixing the
>> contrib unit tests for 203 and hope to be able to call a vote shortly. I
>> would expect a 204 release next month branching off of branch-0.20-secur=
ity.
>>
>> -- Owen
>>
>

From general-return-3267-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Apr 19 13:37:58 2011
Return-Path: <general-return-3267-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 132901DC6
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 19 Apr 2011 13:37:58 +0000 (UTC)
Received: (qmail 92772 invoked by uid 500); 19 Apr 2011 02:24:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91935 invoked by uid 500); 19 Apr 2011 02:24:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91797 invoked by uid 99); 19 Apr 2011 02:24:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Apr 2011 02:24:36 +0000
X-ASF-Spam-Status: No, hits=3.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of akimball83@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Apr 2011 02:24:31 +0000
Received: by pwi14 with SMTP id 14so3875475pwi.35
        for <multiple recipients>; Mon, 18 Apr 2011 19:24:11 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:from:date:message-id:subject:to
         :content-type;
        bh=ca76R1+hGhfP/8q4gClmmuuiILLM8hWuD2Dw9/Lz3As=;
        b=gGDm310GxSSqHGKkH/0A3bQX+jyp43RddFpul9KveR28IBHgmdI6WGlqrMuREkkz6n
         7YaES0PRyKnnmzz16HCAKqCiquDAXPFpQzLPeTSRRSVSRli5uI3EgYSzrMMBbFPyYyPX
         aoE/klNn5F3gtZJFAaQllmDAvDm9pIVTUSCRo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type;
        b=JEDdWihgEQSroyGKigwDFR85nTwbIP0xevPtRkedRP5B0vdSDExFZwOW3RA/5G/5hn
         GMsw3zGZBFPHedJY6QgFtExeUVCOoZedTAdX9cZDo+vVCaR6grVRdE289bDBlU6XsS8b
         aeJI1bNciqQoe6yWSpqVCvkkLA8weEN3R3K7c=
Received: by 10.68.49.103 with SMTP id t7mr7626293pbn.330.1303179851067; Mon,
 18 Apr 2011 19:24:11 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.68.62.6 with HTTP; Mon, 18 Apr 2011 19:23:51 -0700 (PDT)
From: Aaron Kimball <akimball83@gmail.com>
Date: Mon, 18 Apr 2011 19:23:51 -0700
Message-ID: <BANLkTini5JashVxBi7xajoi8n1C5yfCMfg@mail.gmail.com>
Subject: April SFHUG recap, May SFHUG meetup announcement
To: general@hadoop.apache.org, core-user@hadoop.apache.org, 
	user@pig.apache.org, user@hive.apache.org
Content-Type: multipart/alternative; boundary=bcaec544eee219efe604a13c3460

--bcaec544eee219efe604a13c3460
Content-Type: text/plain; charset=ISO-8859-1

Hello Hadoop fans,

This last week we had a very successful meetup of the SF Hadoop User Group,
hosted by Twitter.
Breakout topics included:
* Log analysis
* Cluster resource management
* FlumeBase
* Reusable MapReduce scripts
* Languages for MapReduce programming
* Hadoop Roadmap
* Oozie Best Practices
* HBase how-to
* NameNode SPOF best practices
* Dirty Data

Notes for some of these breakout sessions are posted on the meetup group's
message board at: http://bit.ly/hIpU7G

I am also pleased to announce that the May Hadoop meetup will be held on
Wednesday, May 11, from 6pm to 8pm.

This meetup will be hosted by our friends at Cloudera, in their new SF
office.

As usual, we will use the discussion-based "unconference" format. At the
beginning of the meetup we will collaboratively construct an agenda
consisting of several discussion breakout groups. All participants may
propose a topic and volunteer to facilitate a discussion. All Hadoop-related
topics are encouraged, and all members of the Hadoop community are welcome.

Event schedule:

6pm - Welcome
6:30pm - Introductions; start creating agenda
Breakout sessions begin as soon as we're ready
8pm - Conclusion

Food and refreshments will be provided, courtesy of Cloudera. Please RSVP
at http://bit.ly/hwMCI2

Looking forward to seeing you there!
Regards,
- Aaron Kimball

--bcaec544eee219efe604a13c3460--

From general-return-3268-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Apr 19 15:28:00 2011
Return-Path: <general-return-3268-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AAFE61A3E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 19 Apr 2011 15:28:00 +0000 (UTC)
Received: (qmail 8882 invoked by uid 500); 19 Apr 2011 15:27:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8790 invoked by uid 500); 19 Apr 2011 15:27:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8780 invoked by uid 99); 19 Apr 2011 15:27:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Apr 2011 15:27:58 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 74.125.83.176 is neither permitted nor denied by domain of james@tynt.com)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Apr 2011 15:27:51 +0000
Received: by pve37 with SMTP id 37so4042766pve.35
        for <general@hadoop.apache.org>; Tue, 19 Apr 2011 08:27:29 -0700 (PDT)
Received: by 10.142.171.4 with SMTP id t4mr3660487wfe.434.1303226849752;
        Tue, 19 Apr 2011 08:27:29 -0700 (PDT)
Received: from [192.168.15.238] ([68.144.33.41])
        by mx.google.com with ESMTPS id k6sm5824wfa.5.2011.04.19.08.27.28
        (version=TLSv1/SSLv3 cipher=OTHER);
        Tue, 19 Apr 2011 08:27:28 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: Hadoop in Canada
From: James Seigel <james@tynt.com>
In-Reply-To: <20110331094236.5335a912@essen.neofonie.priv>
Date: Tue, 19 Apr 2011 09:27:27 -0600
Content-Transfer-Encoding: 7bit
Message-Id: <84410BE7-0385-45C5-A915-8518860C8762@tynt.com>
References: <46ED7CAE-5DE6-4453-8E04-802055EBE379@tynt.com> <AANLkTikg03bDbxqXrbzbN3LTzBJycBRG1yQO8F1Wrx=T@mail.gmail.com> <20110331094236.5335a912@essen.neofonie.priv>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

Hey thanks for the heads up..I am going to try and get there!

Are you in Vancouver?

J


On 2011-03-31, at 1:42 AM, Isabel Drost wrote:

> On Tue, 29 Mar 11 Jean-Daniel Cryans wrote:
>> I would actually love to see something getting organized and I'd be on
>> the first plane to Y**
> 
> Not exactly a meetup but Apache Con (<http://na11.apachecon.com/>) is
> to take place in Vancouver, includes a "cloud computing" track focused
> on Hadoop and friends, call for participation is open.
> 
> Isabel


From general-return-3269-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Tue Apr 19 16:08:28 2011
Return-Path: <general-return-3269-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0A253184B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Tue, 19 Apr 2011 16:08:28 +0000 (UTC)
Received: (qmail 74048 invoked by uid 500); 19 Apr 2011 16:08:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73989 invoked by uid 500); 19 Apr 2011 16:08:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73981 invoked by uid 99); 19 Apr 2011 16:08:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Apr 2011 16:08:26 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of michael_segel@hotmail.com designates 65.55.34.12 as permitted sender)
Received: from [65.55.34.12] (HELO col0-omc1-s2.col0.hotmail.com) (65.55.34.12)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Apr 2011 16:08:17 +0000
Received: from COL117-W28 ([65.55.34.8]) by col0-omc1-s2.col0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Tue, 19 Apr 2011 09:07:56 -0700
Message-ID: <COL117-W28DA57A2636C59C0B125398F900@phx.gbl>
X-Originating-IP: [65.167.11.254]
From: Michael Segel <michael_segel@hotmail.com>
To: <general@hadoop.apache.org>
Subject: Would being able to track the time a task was active?
Date: Tue, 19 Apr 2011 11:07:55 -0500
Importance: Normal
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginalArrivalTime: 19 Apr 2011 16:07:56.0123 (UTC) FILETIME=[F1E2DAB0:01CBFEAB]
X-Virus-Checked: Checked by ClamAV on apache.org


Ok=2C=20

Maybe I'm missing something=2C but when I look at my jobs running=2C I see =
the job start time and then the end time of the specific task running.=20

Wouldn't it make sense to know the time it took for the task to run=2C rath=
er than count from the launch of the job?
So if I have a task that has to wait for other tasks ahead of it in the que=
ue to run=2C I'd like to see if the task took 20 mins to run like the other=
s=2C rather than 2 hours and some minutes because the job had so many tasks=
 and fair scheduler put some tasks on hold?

Just a thought.

-Mike


 		 	   		  =

From general-return-3270-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Apr 20 05:03:28 2011
Return-Path: <general-return-3270-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D52F280D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 20 Apr 2011 05:03:28 +0000 (UTC)
Received: (qmail 18434 invoked by uid 500); 20 Apr 2011 05:03:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18370 invoked by uid 500); 20 Apr 2011 05:03:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18362 invoked by uid 99); 20 Apr 2011 05:03:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Apr 2011 05:03:23 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.98 as permitted sender)
Received: from [17.148.16.98] (HELO asmtpout023.mac.com) (17.148.16.98)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Apr 2011 05:03:16 +0000
MIME-version: 1.0
Content-type: multipart/alternative;
 boundary="Boundary_(ID_ft1ZFsw2HIiAU3NwCzPjAw)"
Received: from [10.0.1.31] ([71.198.192.174])
 by asmtp023.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LJX00I61Q0TK020@asmtp023.mac.com>
 for general@hadoop.apache.org; Tue, 19 Apr 2011 22:02:55 -0700 (PDT)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-04-20_01:2011-04-20,2011-04-19,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1104190130
From: Nigel Daley <ndaley@mac.com>
Subject: Re: HADOOP-7106: Re-organize hadoop subversion layout
Date: Tue, 19 Apr 2011 22:02:53 -0700
In-reply-to: <4112B27D-7DCF-4C8E-AD61-6884BA7E738C@mac.com>
To: general@hadoop.apache.org, Ian Holsman <ian@holsman.net>,
 Todd Lipcon <todd@cloudera.com>, Owen O'Malley <oom@yahoo-inc.com>
References: <C955EB7A.2E753%sureshms@yahoo-inc.com>
 <E2932D45-FF26-4579-A975-A98D2578622A@mac.com>
 <B98835A6-F4B7-426E-B22F-B2BEBE273C47@mac.com>
 <4112B27D-7DCF-4C8E-AD61-6884BA7E738C@mac.com>
Message-id: <844F39DC-0A5D-4AAE-AF8D-EFD4CC83C18A@mac.com>
X-Mailer: Apple Mail (2.1084)

--Boundary_(ID_ft1ZFsw2HIiAU3NwCzPjAw)
Content-type: text/plain; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT

I'm still planning to make this SVN change on Thursday this week.  

Ian, Owen, Todd, note the questions I ask you below.  Can you help with these on Thursday?

Thanks,
Nige

On Apr 9, 2011, at 11:09 PM, Nigel Daley wrote:

> All, 
> 
> As discussed in Jan/Feb, I'd like to coordinate a date for committing the re-organization of our svn layout: https://issues.apache.org/jira/browse/HADOOP-7106.  I propose Thursday April 21 at 11am PDT.
> 
> - I will send out reminders leading up to that date.
> - I will announce on IRC when I'm about to start the changes.
> - I will run the script to make the changes.
> - Ian, can you update the asf-authorization-template file and the asf-mailer.conf files at the same time?
> - Owen/Todd/Jukka, can you make sure that actions needed by git users are taken care of at the same time? (what are these?)
> 
> More info on this change is at http://wiki.apache.org/hadoop/ProjectSplit
> 
> Cheers,
> Nige


--Boundary_(ID_ft1ZFsw2HIiAU3NwCzPjAw)--

From general-return-3271-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Apr 20 05:21:13 2011
Return-Path: <general-return-3271-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 798008A4
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 20 Apr 2011 05:21:13 +0000 (UTC)
Received: (qmail 28786 invoked by uid 500); 20 Apr 2011 05:21:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28745 invoked by uid 500); 20 Apr 2011 05:21:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28737 invoked by uid 99); 20 Apr 2011 05:21:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Apr 2011 05:21:10 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Apr 2011 05:21:02 +0000
Received: by bwz8 with SMTP id 8so597949bwz.35
        for <general@hadoop.apache.org>; Tue, 19 Apr 2011 22:20:42 -0700 (PDT)
Received: by 10.204.39.68 with SMTP id f4mr5959775bke.17.1303276842155; Tue,
 19 Apr 2011 22:20:42 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.38.14 with HTTP; Tue, 19 Apr 2011 22:20:22 -0700 (PDT)
In-Reply-To: <844F39DC-0A5D-4AAE-AF8D-EFD4CC83C18A@mac.com>
References: <C955EB7A.2E753%sureshms@yahoo-inc.com> <E2932D45-FF26-4579-A975-A98D2578622A@mac.com>
 <B98835A6-F4B7-426E-B22F-B2BEBE273C47@mac.com> <4112B27D-7DCF-4C8E-AD61-6884BA7E738C@mac.com>
 <844F39DC-0A5D-4AAE-AF8D-EFD4CC83C18A@mac.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Tue, 19 Apr 2011 22:20:22 -0700
Message-ID: <BANLkTin+SyXE0tnfwhhH9P=wR0b7wCOHaw@mail.gmail.com>
Subject: Re: HADOOP-7106: Re-organize hadoop subversion layout
To: Nigel Daley <ndaley@mac.com>
Cc: general@hadoop.apache.org, Ian Holsman <ian@holsman.net>, 
	"Owen O'Malley" <oom@yahoo-inc.com>
Content-Type: multipart/alternative; boundary=bcaec555571238816e04a152c93d
X-Virus-Checked: Checked by ClamAV on apache.org

--bcaec555571238816e04a152c93d
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Apr 19, 2011 at 10:02 PM, Nigel Daley <ndaley@mac.com> wrote:

> I'm still planning to make this SVN change on Thursday this week.
>
> Ian, Owen, Todd, note the questions I ask you below.  Can you help with
> these on Thursday?
>

Unfortunately I'm out of the office most of the day on Thursday with a
customer. I'll be available Thursday evening, though, to help with any
cleanup/etc.

I'm currently looking into how the git mirrors are setup in Apache-land.

My guess is that there will be some disturbance to developers on Thurs
afternoon / Friday as this gets sorted out, even if we try to plan as much
as possible. Would it be better to do this on Friday so that we have the
weekend to fix up broken pieces before people get to work on Monday?

-Todd


> On Apr 9, 2011, at 11:09 PM, Nigel Daley wrote:
>
> All,
>
> As discussed in Jan/Feb, I'd like to coordinate a date for committing the
> re-organization of our svn layout:
> https://issues.apache.org/jira/browse/HADOOP-7106.  I propose Thursday
> April 21 at 11am PDT.
>
> - I will send out reminders leading up to that date.
> - I will announce on IRC when I'm about to start the changes.
> - I will run the script to make the changes.
> - Ian, can you update the asf-authorization-template file and the
> asf-mailer.conf files at the same time?
> - Owen/Todd/Jukka, can you make sure that actions needed by git users are
> taken care of at the same time? (what are these?)
>
> More info on this change is at http://wiki.apache.org/hadoop/ProjectSplit
>
> Cheers,
> Nige
>
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

--bcaec555571238816e04a152c93d--

From general-return-3272-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Apr 20 05:30:52 2011
Return-Path: <general-return-3272-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D15E0EF2
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 20 Apr 2011 05:30:52 +0000 (UTC)
Received: (qmail 33150 invoked by uid 500); 20 Apr 2011 05:30:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33105 invoked by uid 500); 20 Apr 2011 05:30:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33097 invoked by uid 99); 20 Apr 2011 05:30:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Apr 2011 05:30:49 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.94 as permitted sender)
Received: from [17.148.16.94] (HELO asmtpout019.mac.com) (17.148.16.94)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Apr 2011 05:30:39 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.31] ([71.198.192.174])
 by asmtp019.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LJX0085URAGIR90@asmtp019.mac.com>
 for general@hadoop.apache.org; Tue, 19 Apr 2011 22:30:18 -0700 (PDT)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-04-20_01:2011-04-20,2011-04-19,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1104190133
Subject: Re: HADOOP-7106: Re-organize hadoop subversion layout
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <BANLkTin+SyXE0tnfwhhH9P=wR0b7wCOHaw@mail.gmail.com>
Date: Tue, 19 Apr 2011 22:30:16 -0700
Cc: Ian Holsman <ian@holsman.net>, Owen O'Malley <oom@yahoo-inc.com>
Message-id: <2BA11C0A-DA91-4672-A558-B08F8275D100@mac.com>
References: <C955EB7A.2E753%sureshms@yahoo-inc.com>
 <E2932D45-FF26-4579-A975-A98D2578622A@mac.com>
 <B98835A6-F4B7-426E-B22F-B2BEBE273C47@mac.com>
 <4112B27D-7DCF-4C8E-AD61-6884BA7E738C@mac.com>
 <844F39DC-0A5D-4AAE-AF8D-EFD4CC83C18A@mac.com>
 <BANLkTin+SyXE0tnfwhhH9P=wR0b7wCOHaw@mail.gmail.com>
To: general@hadoop.apache.org, Todd Lipcon <todd@cloudera.com>
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

I could start this at 2pm on Friday if that suited folks better.

Nige

On Apr 19, 2011, at 10:20 PM, Todd Lipcon wrote:

> On Tue, Apr 19, 2011 at 10:02 PM, Nigel Daley <ndaley@mac.com> wrote:
> 
>> I'm still planning to make this SVN change on Thursday this week.
>> 
>> Ian, Owen, Todd, note the questions I ask you below.  Can you help with
>> these on Thursday?
>> 
> 
> Unfortunately I'm out of the office most of the day on Thursday with a
> customer. I'll be available Thursday evening, though, to help with any
> cleanup/etc.
> 
> I'm currently looking into how the git mirrors are setup in Apache-land.
> 
> My guess is that there will be some disturbance to developers on Thurs
> afternoon / Friday as this gets sorted out, even if we try to plan as much
> as possible. Would it be better to do this on Friday so that we have the
> weekend to fix up broken pieces before people get to work on Monday?
> 
> -Todd
> 
> 
>> On Apr 9, 2011, at 11:09 PM, Nigel Daley wrote:
>> 
>> All,
>> 
>> As discussed in Jan/Feb, I'd like to coordinate a date for committing the
>> re-organization of our svn layout:
>> https://issues.apache.org/jira/browse/HADOOP-7106.  I propose Thursday
>> April 21 at 11am PDT.
>> 
>> - I will send out reminders leading up to that date.
>> - I will announce on IRC when I'm about to start the changes.
>> - I will run the script to make the changes.
>> - Ian, can you update the asf-authorization-template file and the
>> asf-mailer.conf files at the same time?
>> - Owen/Todd/Jukka, can you make sure that actions needed by git users are
>> taken care of at the same time? (what are these?)
>> 
>> More info on this change is at http://wiki.apache.org/hadoop/ProjectSplit
>> 
>> Cheers,
>> Nige
>> 
>> 
>> 
> 
> 
> -- 
> Todd Lipcon
> Software Engineer, Cloudera


From general-return-3273-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Apr 20 05:59:33 2011
Return-Path: <general-return-3273-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5A20429D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 20 Apr 2011 05:59:33 +0000 (UTC)
Received: (qmail 63983 invoked by uid 500); 20 Apr 2011 05:59:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63934 invoked by uid 500); 20 Apr 2011 05:59:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63926 invoked by uid 99); 20 Apr 2011 05:59:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Apr 2011 05:59:31 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Apr 2011 05:59:25 +0000
Received: by bwz8 with SMTP id 8so620086bwz.35
        for <general@hadoop.apache.org>; Tue, 19 Apr 2011 22:59:04 -0700 (PDT)
Received: by 10.204.16.140 with SMTP id o12mr1448821bka.125.1303279144270;
 Tue, 19 Apr 2011 22:59:04 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.38.14 with HTTP; Tue, 19 Apr 2011 22:58:44 -0700 (PDT)
In-Reply-To: <BANLkTin+SyXE0tnfwhhH9P=wR0b7wCOHaw@mail.gmail.com>
References: <C955EB7A.2E753%sureshms@yahoo-inc.com> <E2932D45-FF26-4579-A975-A98D2578622A@mac.com>
 <B98835A6-F4B7-426E-B22F-B2BEBE273C47@mac.com> <4112B27D-7DCF-4C8E-AD61-6884BA7E738C@mac.com>
 <844F39DC-0A5D-4AAE-AF8D-EFD4CC83C18A@mac.com> <BANLkTin+SyXE0tnfwhhH9P=wR0b7wCOHaw@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Tue, 19 Apr 2011 22:58:44 -0700
Message-ID: <BANLkTi=+cQBhsNjwdzB+fhcDkePJ=gSOqQ@mail.gmail.com>
Subject: Re: HADOOP-7106: Re-organize hadoop subversion layout
To: Nigel Daley <ndaley@mac.com>
Cc: general@hadoop.apache.org, Ian Holsman <ian@holsman.net>, 
	"Owen O'Malley" <oom@yahoo-inc.com>
Content-Type: multipart/alternative; boundary=00032555a16e6ffd7a04a15352e0

--00032555a16e6ffd7a04a15352e0
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Apr 19, 2011 at 10:20 PM, Todd Lipcon <todd@cloudera.com> wrote:

>
> I'm currently looking into how the git mirrors are setup in Apache-land.
>

Git-wise, I think we have two options:

Option 1)
- Create a new git mirror for the new hadoop/ tree. This will have no
history.
- On the Apache side, fetch the split-project git mirrors into the combined
git mirror as branches - eg hadoop-hdfs.git:trunk becomes a branch named
something like pre-HADOOP-7106/hdfs/trunk. Thus, when any user fetches,
he'll get all the git objects from "prehistory" as well without having to
add separate remotes.
- Add a script or README file explaining how to set up git grafts on the
combined hadoop.git so that the new combination branch "foo" looks like a
merge of pre-HADOOP-7106/{hdfs,common,mapred}/foo. Since git grafts are
local constructs, each git user would have to run this script once after
checking out the git tree, after which the history would be "healed"

Pros:
 - all existing sha1s stay the same.
 - Any local branches people might have for works in progress should
continue to refer to proper SHA1s and should rebase relatively easily onto
the combined trunk
 - Should be reasonably simple to implement

Cons:
 - users have to run a script upon checkout in order to graft back together
history

Option 2)
- Use git-filter-branch on the split repos to rewrite them as if they always
took place in their new subdirectories.
- Fetch these repos into the merged repo
- Set up grafts in the merged repo
- Run git-filter-branch --all in the merged repo, which will make the grafts
permanent
- May have to run git-filter-branch to rewrite some of the git-svn-info:
commit messages to trick git-svn.

This option basically rewrites history so that it looks like the original
project split did what we're planning to do now.

Pros:
 - we have a single cohesive git repo with no need to have users set up
grafts

Cons:
 - all of our SHA1s between the original split and now would change (making
it harder to rebase local branches for example)
 - way more opportunity for error, I think.

I'm leaning towards option 1 above, and happy to write the script which
installs the grafts into the user's local repo.

-Todd


>
>> On Apr 9, 2011, at 11:09 PM, Nigel Daley wrote:
>>
>> All,
>>
>> As discussed in Jan/Feb, I'd like to coordinate a date for committing the
>> re-organization of our svn layout:
>> https://issues.apache.org/jira/browse/HADOOP-7106.  I propose Thursday
>> April 21 at 11am PDT.
>>
>> - I will send out reminders leading up to that date.
>> - I will announce on IRC when I'm about to start the changes.
>> - I will run the script to make the changes.
>> - Ian, can you update the asf-authorization-template file and the
>> asf-mailer.conf files at the same time?
>> - Owen/Todd/Jukka, can you make sure that actions needed by git users are
>> taken care of at the same time? (what are these?)
>>
>> More info on this change is at http://wiki.apache.org/hadoop/ProjectSplit
>>
>> Cheers,
>> Nige
>>
>>
>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--00032555a16e6ffd7a04a15352e0--

From general-return-3274-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Apr 20 08:34:25 2011
Return-Path: <general-return-3274-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 61CEC11FC
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 20 Apr 2011 08:34:25 +0000 (UTC)
Received: (qmail 89096 invoked by uid 500); 20 Apr 2011 08:07:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88490 invoked by uid 500); 20 Apr 2011 08:07:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88469 invoked by uid 99); 20 Apr 2011 08:07:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Apr 2011 08:07:42 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of owen.omalley@gmail.com designates 209.85.210.48 as permitted sender)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Apr 2011 08:07:35 +0000
Received: by pzk10 with SMTP id 10so383846pzk.35
        for <general@hadoop.apache.org>; Wed, 20 Apr 2011 01:07:15 -0700 (PDT)
Received: by 10.68.50.161 with SMTP id d1mr10440677pbo.439.1303286835022;
        Wed, 20 Apr 2011 01:07:15 -0700 (PDT)
Received: from [10.0.0.59] (c-98-234-216-58.hsd1.ca.comcast.net [98.234.216.58])
        by mx.google.com with ESMTPS id t9sm483677pbq.31.2011.04.20.01.07.13
        (version=SSLv3 cipher=OTHER);
        Wed, 20 Apr 2011 01:07:13 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: HADOOP-7106: Re-organize hadoop subversion layout
From: Owen O'Malley <omalley@apache.org>
In-Reply-To: <4112B27D-7DCF-4C8E-AD61-6884BA7E738C@mac.com>
Date: Wed, 20 Apr 2011 01:07:11 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <F4C6326C-4C03-4012-96B3-318774511B41@apache.org>
References: <C955EB7A.2E753%sureshms@yahoo-inc.com> <E2932D45-FF26-4579-A975-A98D2578622A@mac.com> <B98835A6-F4B7-426E-B22F-B2BEBE273C47@mac.com> <4112B27D-7DCF-4C8E-AD61-6884BA7E738C@mac.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)


On Apr 9, 2011, at 11:09 PM, Nigel Daley wrote:

> As discussed in Jan/Feb, I'd like to coordinate a date for committing =
the re-organization of our svn layout: =
https://issues.apache.org/jira/browse/HADOOP-7106.  I propose Thursday =
April 21 at 11am PDT.

This is still premature.

Your patch is out of date with respect to the current branches. Please =
update it and upload a new version.

What changes do we want to make to the asf-authorization file? Clearly =
that needs to be decided before anything is done.

What changes are we making to the notifications? That also hasn't been =
discussed at all.

Again, has anyone talked to Jukka and the rest of the infrastructure-dev =
to see if there is anything that we need to do to minimize transition =
pain? Clearly, the notifications for git.apache.org need to be =
maintained, but I don't know if there are other actions that would help =
minimize the data loss.

Clearly any change where all of the git hashes change is massively =
disruptive. I suspect there is some magic you can do to keep more of the =
history, but I don't know the subversion-git bridge very well.

-- Owen


From general-return-3275-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Apr 20 13:07:41 2011
Return-Path: <general-return-3275-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E1A33E02
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 20 Apr 2011 13:07:41 +0000 (UTC)
Received: (qmail 93545 invoked by uid 500); 20 Apr 2011 13:07:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93480 invoked by uid 500); 20 Apr 2011 13:07:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93468 invoked by uid 99); 20 Apr 2011 13:07:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Apr 2011 13:07:39 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Apr 2011 13:07:31 +0000
Received: by vws7 with SMTP id 7so836798vws.35
        for <general@hadoop.apache.org>; Wed, 20 Apr 2011 06:07:10 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:subject:mime-version:content-type:from
         :in-reply-to:date:cc:content-transfer-encoding:message-id:references
         :to:x-mailer;
        bh=vIenxH9hYRPcicsYJxDnMS09yCkQahkMdkvcX3t8kvo=;
        b=Ufbzp0hHwCbjIVuD+fQIhw85Dm4yKtYKwTVAcK484NmmHNs8gwzu4eoTLdg3LSRYGg
         6mIM9qUBP1RbRULOmGJeG0JoNOA897wigkuG5bPQ7AtOny6hSf36OWPrtcpGttXZrDVS
         kuiYj2jhXRUvrVJDVM5aeTf72IChfk16/tpAQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=subject:mime-version:content-type:from:in-reply-to:date:cc
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=F/aGzXGd/fy/OUTqle2f8IsqVQQt75ug+MDWZbMvOhCH8CFEE19oVr15Ro9UPUwlsO
         cAYliCkTpe4YQDV0s9MSLdiZrPM7m/WuL6K8lAxY76l+TRaLPSnQmDqyXrSrcgh7pbnW
         EuUyqbSF7scrtc+1KHC6eZQaJlTVlQlpUjxHU=
Received: by 10.52.0.66 with SMTP id 2mr1415520vdc.308.1303304828687;
        Wed, 20 Apr 2011 06:07:08 -0700 (PDT)
Received: from [10.172.0.111] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id e20sm503876vbz.8.2011.04.20.06.07.06
        (version=SSLv3 cipher=OTHER);
        Wed, 20 Apr 2011 06:07:07 -0700 (PDT)
Subject: Re: HADOOP-7106: Re-organize hadoop subversion layout
Mime-Version: 1.0 (Apple Message framework v1084)
Content-Type: text/plain; charset=us-ascii
From: Ian Holsman <ian@holsman.net>
In-Reply-To: <BANLkTi=+cQBhsNjwdzB+fhcDkePJ=gSOqQ@mail.gmail.com>
Date: Wed, 20 Apr 2011 14:07:04 +0100
Cc: Nigel Daley <ndaley@mac.com>,
 general@hadoop.apache.org,
 "Owen O'Malley" <oom@yahoo-inc.com>
Content-Transfer-Encoding: quoted-printable
Message-Id: <B093A57F-B55B-445B-9C11-7362DC443EE4@holsman.net>
References: <C955EB7A.2E753%sureshms@yahoo-inc.com> <E2932D45-FF26-4579-A975-A98D2578622A@mac.com> <B98835A6-F4B7-426E-B22F-B2BEBE273C47@mac.com> <4112B27D-7DCF-4C8E-AD61-6884BA7E738C@mac.com> <844F39DC-0A5D-4AAE-AF8D-EFD4CC83C18A@mac.com> <BANLkTin+SyXE0tnfwhhH9P=wR0b7wCOHaw@mail.gmail.com> <BANLkTi=+cQBhsNjwdzB+fhcDkePJ=gSOqQ@mail.gmail.com>
To: Todd Lipcon <todd@cloudera.com>
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

I'm traveling at the moment (when am I not), and probably won't have =
access until monday.
I'll look at the files, and see if I can do some stuff today without =
impacting anything.
On Apr 20, 2011, at 6:58 AM, Todd Lipcon wrote:

> On Tue, Apr 19, 2011 at 10:20 PM, Todd Lipcon <todd@cloudera.com> =
wrote:
>=20
> I'm currently looking into how the git mirrors are setup in =
Apache-land.
>=20
> Git-wise, I think we have two options:
>=20
> Option 1)
> - Create a new git mirror for the new hadoop/ tree. This will have no =
history.
> - On the Apache side, fetch the split-project git mirrors into the =
combined git mirror as branches - eg hadoop-hdfs.git:trunk becomes a =
branch named something like pre-HADOOP-7106/hdfs/trunk. Thus, when any =
user fetches, he'll get all the git objects from "prehistory" as well =
without having to add separate remotes.
> - Add a script or README file explaining how to set up git grafts on =
the combined hadoop.git so that the new combination branch "foo" looks =
like a merge of pre-HADOOP-7106/{hdfs,common,mapred}/foo. Since git =
grafts are local constructs, each git user would have to run this script =
once after checking out the git tree, after which the history would be =
"healed"
>=20
> Pros:
>  - all existing sha1s stay the same.
>  - Any local branches people might have for works in progress should =
continue to refer to proper SHA1s and should rebase relatively easily =
onto the combined trunk
>  - Should be reasonably simple to implement
>=20
> Cons:
>  - users have to run a script upon checkout in order to graft back =
together history
>=20
> Option 2)
> - Use git-filter-branch on the split repos to rewrite them as if they =
always took place in their new subdirectories.
> - Fetch these repos into the merged repo
> - Set up grafts in the merged repo
> - Run git-filter-branch --all in the merged repo, which will make the =
grafts permanent
> - May have to run git-filter-branch to rewrite some of the =
git-svn-info: commit messages to trick git-svn.
>=20
> This option basically rewrites history so that it looks like the =
original project split did what we're planning to do now.
>=20
> Pros:
>  - we have a single cohesive git repo with no need to have users set =
up grafts
>=20
> Cons:
>  - all of our SHA1s between the original split and now would change =
(making it harder to rebase local branches for example)
>  - way more opportunity for error, I think.
>=20
> I'm leaning towards option 1 above, and happy to write the script =
which installs the grafts into the user's local repo.
>=20
> -Todd
>=20
> =20
> On Apr 9, 2011, at 11:09 PM, Nigel Daley wrote:
>=20
>> All,=20
>>=20
>> As discussed in Jan/Feb, I'd like to coordinate a date for committing =
the re-organization of our svn layout: =
https://issues.apache.org/jira/browse/HADOOP-7106.  I propose Thursday =
April 21 at 11am PDT.
>>=20
>> - I will send out reminders leading up to that date.
>> - I will announce on IRC when I'm about to start the changes.
>> - I will run the script to make the changes.
>> - Ian, can you update the asf-authorization-template file and the =
asf-mailer.conf files at the same time?
>> - Owen/Todd/Jukka, can you make sure that actions needed by git users =
are taken care of at the same time? (what are these?)
>>=20
>> More info on this change is at =
http://wiki.apache.org/hadoop/ProjectSplit
>>=20
>> Cheers,
>> Nige
>=20
>=20
>=20
>=20
> --=20
> Todd Lipcon
> Software Engineer, Cloudera
>=20
>=20
>=20
> --=20
> Todd Lipcon
> Software Engineer, Cloudera

--
Ian Holsman
Ian@Holsman.net
PH: +1-703 879-3128 AOLIM: ianholsman Skype:iholsman

To know recursion, you must first know recursion.




From general-return-3276-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Apr 20 15:00:51 2011
Return-Path: <general-return-3276-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8070E179A
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 20 Apr 2011 15:00:51 +0000 (UTC)
Received: (qmail 51574 invoked by uid 500); 20 Apr 2011 15:00:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51495 invoked by uid 500); 20 Apr 2011 15:00:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51485 invoked by uid 99); 20 Apr 2011 15:00:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Apr 2011 15:00:49 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of owen.omalley@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Apr 2011 15:00:41 +0000
Received: by pwi14 with SMTP id 14so615734pwi.35
        for <general@hadoop.apache.org>; Wed, 20 Apr 2011 08:00:20 -0700 (PDT)
Received: by 10.68.41.67 with SMTP id d3mr9973231pbl.287.1303311620580;
        Wed, 20 Apr 2011 08:00:20 -0700 (PDT)
Received: from [10.0.0.59] (c-98-234-216-58.hsd1.ca.comcast.net [98.234.216.58])
        by mx.google.com with ESMTPS id y7sm678039pbc.36.2011.04.20.08.00.18
        (version=SSLv3 cipher=OTHER);
        Wed, 20 Apr 2011 08:00:19 -0700 (PDT)
Subject: Re: HADOOP-7106: Re-organize hadoop subversion layout
Mime-Version: 1.0 (Apple Message framework v1084)
Content-Type: text/plain; charset=us-ascii
From: Owen O'Malley <omalley@apache.org>
In-Reply-To: <BANLkTi=+cQBhsNjwdzB+fhcDkePJ=gSOqQ@mail.gmail.com>
Date: Wed, 20 Apr 2011 08:00:16 -0700
Cc: Nigel Daley <ndaley@mac.com>,
 Ian Holsman <ian@holsman.net>,
 infrastructure-dev@apache.org
Content-Transfer-Encoding: quoted-printable
Message-Id: <9B529359-B589-4229-BA96-188C762A8E31@apache.org>
References: <C955EB7A.2E753%sureshms@yahoo-inc.com> <E2932D45-FF26-4579-A975-A98D2578622A@mac.com> <B98835A6-F4B7-426E-B22F-B2BEBE273C47@mac.com> <4112B27D-7DCF-4C8E-AD61-6884BA7E738C@mac.com> <844F39DC-0A5D-4AAE-AF8D-EFD4CC83C18A@mac.com> <BANLkTin+SyXE0tnfwhhH9P=wR0b7wCOHaw@mail.gmail.com> <BANLkTi=+cQBhsNjwdzB+fhcDkePJ=gSOqQ@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org


On Apr 19, 2011, at 10:58 PM, Todd Lipcon wrote:

> On Tue, Apr 19, 2011 at 10:20 PM, Todd Lipcon <todd@cloudera.com> =
wrote:
>=20
>>=20
>> I'm currently looking into how the git mirrors are setup in =
Apache-land.

Uh, why isn't infra-dev on this thread?

For those on infra-dev, the context is that Nigel is trying to merge =
together the source trees of the Hadoop sub-projects that were split =
apart 2 years ago. So he is taking:

prefix =3D http://svn.apache.org/repos/asf/hadoop/

$prefix/common/trunk -> $prefix/trunk/common
$prefix/hdfs/trunk -> $prefix/trunk/hdfs
$prefix/mapreduce/trunk -> $prefix/trunk/mapreduce

and play similar games with the rest of the branches and tags. For more =
details look at HADOOP-7106.

=46rom the project split, subversion was able to track the history =
across the subversion moves between projects, but not git.

Four questions:
 1. Is there anything we can do to minimize the history loss in git?
 2. Are we going to be able to preserve our sha's or are they going to =
change again?
 3. What changes do we need to make to the subversion notification file?
 4. Are there any other changes that need to be coordinated?

After considering it this morning, I believe that the least disruptive =
move is to leave common at the same url and merge hdfs and mapreduce =
back in:

$prefix/common/trunk/* -> $prefix/common/trunk/common/*
$prefix/hdfs/trunk -> $prefix/common/trunk/hdfs
$prefix/mapreduce/trunk -> $prefix/common/trunk/mapreduce

This will preserve the hashes and history for common (and the 20 =
branches). We'll still need to play git voodoo to get git history for =
hdfs and mapreduce, but it is far better than starting a brand new git =
clone.

-- Owen



From general-return-3277-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Apr 20 15:52:31 2011
Return-Path: <general-return-3277-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D510F177E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 20 Apr 2011 15:52:31 +0000 (UTC)
Received: (qmail 49370 invoked by uid 500); 20 Apr 2011 15:52:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49298 invoked by uid 500); 20 Apr 2011 15:52:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 17697 invoked by uid 99); 20 Apr 2011 15:33:12 -0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of paul.joseph.davis@gmail.com designates 209.85.220.176 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:cc:content-type:content-transfer-encoding;
        bh=I8s326SR6mXUNVlt21aRlXViK2id+e7xuzy924DMA4s=;
        b=F+iYXMtk7vNw2/mvVg7pA0V0yV2V1VTZNoaZE8Vyqild+4UYpISw2hPBwneEOv5UrF
         AoHVGLzoKzxdORRGok8dZPVY1tjV9dT/PxX7YWweFcvUmlywdHdX+DVYEhYlYbC5hH6Z
         5+o5u+F6EXZb5XGdHTxz+ZDYjAOt5UOPBZw+c=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :cc:content-type:content-transfer-encoding;
        b=lvhYDszeo9CAvwD3L3OajJmfuS8ozB3muwsc/cXFFh7+dAxnJkRcf+8H9CCn28YAOH
         FET75zg+uZLAe9olRUHL2HmSAavZ0Bt+DEZtzdHPJU4/IBFpZ0MINHhPok5CsrlNT9f3
         OCHnHqzyV8EfywwmdA24ABUnhWMAYaekhEo1Q=
MIME-Version: 1.0
In-Reply-To: <9B529359-B589-4229-BA96-188C762A8E31@apache.org>
References: <C955EB7A.2E753%sureshms@yahoo-inc.com> <E2932D45-FF26-4579-A975-A98D2578622A@mac.com>
 <B98835A6-F4B7-426E-B22F-B2BEBE273C47@mac.com> <4112B27D-7DCF-4C8E-AD61-6884BA7E738C@mac.com>
 <844F39DC-0A5D-4AAE-AF8D-EFD4CC83C18A@mac.com> <BANLkTin+SyXE0tnfwhhH9P=wR0b7wCOHaw@mail.gmail.com>
 <BANLkTi=+cQBhsNjwdzB+fhcDkePJ=gSOqQ@mail.gmail.com> <9B529359-B589-4229-BA96-188C762A8E31@apache.org>
From: Paul Davis <paul.joseph.davis@gmail.com>
Date: Wed, 20 Apr 2011 11:32:04 -0400
Message-ID: <BANLkTi=cfVFviBLRJ0McgKk_RutWED-6yQ@mail.gmail.com>
Subject: Re: HADOOP-7106: Re-organize hadoop subversion layout
To: infrastructure-dev@apache.org
Cc: general@hadoop.apache.org, Nigel Daley <ndaley@mac.com>, 
	Ian Holsman <ian@holsman.net>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

> From the project split, subversion was able to track the history across t=
he subversion moves between projects, but not git.
>
> Four questions:
> =A01. Is there anything we can do to minimize the history loss in git?

Glancing at the Git history on GitHub for Hadoop HDFS my guess is that
the mirror script is just pulling from $prefix/hdfs. I'm not entirely
certain how promiscuous it'll look for movements from elsewhere in the
svn repo for history outside the root URL.

If you're really wanting to make sure to keep the history in Git
intact my suggestion would be to setup a temporary svn server locally
and test our mirroring scripts against the commands you intend to run.

Paul Davis

From general-return-3278-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Apr 20 19:27:10 2011
Return-Path: <general-return-3278-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B012E108A
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 20 Apr 2011 19:27:10 +0000 (UTC)
Received: (qmail 81319 invoked by uid 500); 20 Apr 2011 19:27:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81266 invoked by uid 500); 20 Apr 2011 19:27:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81258 invoked by uid 99); 20 Apr 2011 19:27:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Apr 2011 19:27:08 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Apr 2011 19:27:02 +0000
Received: by pwi14 with SMTP id 14so794027pwi.35
        for <general@hadoop.apache.org>; Wed, 20 Apr 2011 12:26:41 -0700 (PDT)
Received: by 10.68.38.226 with SMTP id j2mr4657455pbk.471.1303327601144; Wed,
 20 Apr 2011 12:26:41 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.68.51.136 with HTTP; Wed, 20 Apr 2011 12:26:21 -0700 (PDT)
In-Reply-To: <9B529359-B589-4229-BA96-188C762A8E31@apache.org>
References: <C955EB7A.2E753%sureshms@yahoo-inc.com> <E2932D45-FF26-4579-A975-A98D2578622A@mac.com>
 <B98835A6-F4B7-426E-B22F-B2BEBE273C47@mac.com> <4112B27D-7DCF-4C8E-AD61-6884BA7E738C@mac.com>
 <844F39DC-0A5D-4AAE-AF8D-EFD4CC83C18A@mac.com> <BANLkTin+SyXE0tnfwhhH9P=wR0b7wCOHaw@mail.gmail.com>
 <BANLkTi=+cQBhsNjwdzB+fhcDkePJ=gSOqQ@mail.gmail.com> <9B529359-B589-4229-BA96-188C762A8E31@apache.org>
From: Konstantin Boudnik <cos@boudnik.org>
Date: Wed, 20 Apr 2011 12:26:21 -0700
Message-ID: <BANLkTinGniXEfE_dtUez+FjpTvaGS7FaQw@mail.gmail.com>
Subject: Re: HADOOP-7106: Re-organize hadoop subversion layout
To: general@hadoop.apache.org
Cc: "Owen O'Malley" <omalley@apache.org>, Nigel Daley <ndaley@mac.com>, Ian Holsman <ian@holsman.net>, 
	infrastructure-dev@apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Apr 20, 2011 at 08:00, Owen O'Malley <omalley@apache.org> wrote:
> After considering it this morning, I believe that the least disruptive move is to leave common at the same url and merge hdfs and mapreduce back in:
>
> $prefix/common/trunk/* -> $prefix/common/trunk/common/*
> $prefix/hdfs/trunk -> $prefix/common/trunk/hdfs
> $prefix/mapreduce/trunk -> $prefix/common/trunk/mapreduce

This seems like adding an insult to injury by creating an artificial
SVN layout and moving what deemed to be an unrelated components into
common's namespace.

The original proposal seems much more straight forward.

Cos

> This will preserve the hashes and history for common (and the 20 branches). We'll still need to play git voodoo to get git history for hdfs and mapreduce, but it is far better than starting a brand new git clone.
>
> -- Owen

>

From general-return-3279-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Apr 21 04:43:32 2011
Return-Path: <general-return-3279-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 406972D8E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 21 Apr 2011 04:43:32 +0000 (UTC)
Received: (qmail 35856 invoked by uid 500); 21 Apr 2011 04:43:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34982 invoked by uid 500); 21 Apr 2011 04:43:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34965 invoked by uid 99); 21 Apr 2011 04:43:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Apr 2011 04:43:19 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.102 as permitted sender)
Received: from [17.148.16.102] (HELO asmtpout027.mac.com) (17.148.16.102)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Apr 2011 04:43:12 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.8] ([71.198.193.36])
 by asmtp027.mac.com (Oracle Communications Messaging Exchange Server 7u4-18.01
 64bit (built Jul 15 2010)) with ESMTPSA id <0LJZ00C3CJQQK930@asmtp027.mac.com>
 for general@hadoop.apache.org; Wed, 20 Apr 2011 21:42:28 -0700 (PDT)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-04-20_09:2011-04-21,2011-04-20,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1104200162
Subject: Re: HADOOP-7106: Re-organize hadoop subversion layout
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <9B529359-B589-4229-BA96-188C762A8E31@apache.org>
Date: Wed, 20 Apr 2011 21:42:26 -0700
Cc: Ian Holsman <ian@holsman.net>, infrastructure-dev@apache.org
Message-id: <522E7C8B-C159-4520-A758-6E994195D0DC@mac.com>
References: <C955EB7A.2E753%sureshms@yahoo-inc.com>
 <E2932D45-FF26-4579-A975-A98D2578622A@mac.com>
 <B98835A6-F4B7-426E-B22F-B2BEBE273C47@mac.com>
 <4112B27D-7DCF-4C8E-AD61-6884BA7E738C@mac.com>
 <844F39DC-0A5D-4AAE-AF8D-EFD4CC83C18A@mac.com>
 <BANLkTin+SyXE0tnfwhhH9P=wR0b7wCOHaw@mail.gmail.com>
 <BANLkTi=+cQBhsNjwdzB+fhcDkePJ=gSOqQ@mail.gmail.com>
 <9B529359-B589-4229-BA96-188C762A8E31@apache.org>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)

Owen, I'll admit I'm not familiar with all the git details/issues in your proposal, but I think the layout change you propose is fine and seems to solve the git issues with very minimal impact on the layout.

Let's shoot for doing this next Friday, April 29 at 2pm PDT.  I'll update the patch and send out a reminder about this later next week.

Thanks,
Nige

On Apr 20, 2011, at 8:00 AM, Owen O'Malley wrote:

> 
> On Apr 19, 2011, at 10:58 PM, Todd Lipcon wrote:
> 
>> On Tue, Apr 19, 2011 at 10:20 PM, Todd Lipcon <todd@cloudera.com> wrote:
>> 
>>> 
>>> I'm currently looking into how the git mirrors are setup in Apache-land.
> 
> Uh, why isn't infra-dev on this thread?
> 
> For those on infra-dev, the context is that Nigel is trying to merge together the source trees of the Hadoop sub-projects that were split apart 2 years ago. So he is taking:
> 
> prefix = http://svn.apache.org/repos/asf/hadoop/
> 
> $prefix/common/trunk -> $prefix/trunk/common
> $prefix/hdfs/trunk -> $prefix/trunk/hdfs
> $prefix/mapreduce/trunk -> $prefix/trunk/mapreduce
> 
> and play similar games with the rest of the branches and tags. For more details look at HADOOP-7106.
> 
> From the project split, subversion was able to track the history across the subversion moves between projects, but not git.
> 
> Four questions:
> 1. Is there anything we can do to minimize the history loss in git?
> 2. Are we going to be able to preserve our sha's or are they going to change again?
> 3. What changes do we need to make to the subversion notification file?
> 4. Are there any other changes that need to be coordinated?
> 
> After considering it this morning, I believe that the least disruptive move is to leave common at the same url and merge hdfs and mapreduce back in:
> 
> $prefix/common/trunk/* -> $prefix/common/trunk/common/*
> $prefix/hdfs/trunk -> $prefix/common/trunk/hdfs
> $prefix/mapreduce/trunk -> $prefix/common/trunk/mapreduce
> 
> This will preserve the hashes and history for common (and the 20 branches). We'll still need to play git voodoo to get git history for hdfs and mapreduce, but it is far better than starting a brand new git clone.
> 
> -- Owen
> 
> 


From general-return-3280-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Apr 21 10:34:01 2011
Return-Path: <general-return-3280-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D071A2F3E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 21 Apr 2011 10:34:01 +0000 (UTC)
Received: (qmail 81091 invoked by uid 500); 21 Apr 2011 10:34:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81024 invoked by uid 500); 21 Apr 2011 10:33:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81015 invoked by uid 99); 21 Apr 2011 10:33:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Apr 2011 10:33:59 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Apr 2011 10:33:53 +0000
Received: by qwj9 with SMTP id 9so1178960qwj.35
        for <general@hadoop.apache.org>; Thu, 21 Apr 2011 03:33:32 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:subject:mime-version:content-type:from
         :in-reply-to:date:cc:content-transfer-encoding:message-id:references
         :to:x-mailer;
        bh=F83oq1QZq+AYowDgmu9CGckdqvBZ1D+p+R3rOy5YZck=;
        b=XREqemGO7FE+PfDe8VEGYD/0HWEzI4uOsv/aHyq45y8esbmZQk4q0ODEs/a6BEq5SL
         pt97V1dfyzN0gHWpdtwtbvuR0w5SW4qzOxFxCeytz7AfDTYO2yRG8xZzA46a7onPzkTQ
         vMKnBcsPZI9zy18OWsUXI5WTESrT3tVzIOTYQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=subject:mime-version:content-type:from:in-reply-to:date:cc
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=ZeBrDwQbMqvm6ii+YWoZ3mNKnWZ3FL9pKlnt5lOdfwi9NysJ9ubI08id93ZNVy825P
         FASO8n5/dnkP4mel+eWjl/YhXH8ySLTxEEFGUdfo7d3eZhf5uAweda3wO9pJkQqF4I2r
         bcLBedNeOhWR7T/P2JJdly2KfozhdjTFRit4s=
Received: by 10.229.74.206 with SMTP id v14mr6181434qcj.69.1303382012050;
        Thu, 21 Apr 2011 03:33:32 -0700 (PDT)
Received: from [10.172.0.111] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id s16sm1285978qco.25.2011.04.21.03.33.29
        (version=SSLv3 cipher=OTHER);
        Thu, 21 Apr 2011 03:33:30 -0700 (PDT)
Subject: Re: HADOOP-7106: Re-organize hadoop subversion layout
Mime-Version: 1.0 (Apple Message framework v1084)
Content-Type: text/plain; charset=windows-1252
From: Ian Holsman <ian@holsman.net>
In-Reply-To: <BANLkTinGniXEfE_dtUez+FjpTvaGS7FaQw@mail.gmail.com>
Date: Thu, 21 Apr 2011 11:33:27 +0100
Cc: general@hadoop.apache.org,
 "Owen O'Malley" <omalley@apache.org>,
 Nigel Daley <ndaley@mac.com>,
 infrastructure-dev@apache.org
Content-Transfer-Encoding: quoted-printable
Message-Id: <2F4D07A8-7934-4B89-BB04-E81BE588E657@holsman.net>
References: <C955EB7A.2E753%sureshms@yahoo-inc.com> <E2932D45-FF26-4579-A975-A98D2578622A@mac.com> <B98835A6-F4B7-426E-B22F-B2BEBE273C47@mac.com> <4112B27D-7DCF-4C8E-AD61-6884BA7E738C@mac.com> <844F39DC-0A5D-4AAE-AF8D-EFD4CC83C18A@mac.com> <BANLkTin+SyXE0tnfwhhH9P=wR0b7wCOHaw@mail.gmail.com> <BANLkTi=+cQBhsNjwdzB+fhcDkePJ=gSOqQ@mail.gmail.com> <9B529359-B589-4229-BA96-188C762A8E31@apache.org> <BANLkTinGniXEfE_dtUez+FjpTvaGS7FaQw@mail.gmail.com>
To: Konstantin Boudnik <cos@boudnik.org>
X-Mailer: Apple Mail (2.1084)

the other issue is when you create a branch, as you would need to update =
the SVN/mail configs to add each branch independently.. unless you can =
do wildcard matching in the config files.. infra??? can you do this?

On Apr 20, 2011, at 8:26 PM, Konstantin Boudnik wrote:

> On Wed, Apr 20, 2011 at 08:00, Owen O'Malley <omalley@apache.org> =
wrote:
>> After considering it this morning, I believe that the least =
disruptive move is to leave common at the same url and merge hdfs and =
mapreduce back in:
>>=20
>> $prefix/common/trunk/* -> $prefix/common/trunk/common/*
>> $prefix/hdfs/trunk -> $prefix/common/trunk/hdfs
>> $prefix/mapreduce/trunk -> $prefix/common/trunk/mapreduce
>=20
> This seems like adding an insult to injury by creating an artificial
> SVN layout and moving what deemed to be an unrelated components into
> common's namespace.
>=20
> The original proposal seems much more straight forward.
>=20
> Cos
>=20
>> This will preserve the hashes and history for common (and the 20 =
branches). We'll still need to play git voodoo to get git history for =
hdfs and mapreduce, but it is far better than starting a brand new git =
clone.
>>=20
>> -- Owen
>=20
>>=20

--
Ian Holsman
Ian@Holsman.net
PH: +1-703 879-3128 AOLIM: ianholsman Skype:iholsman

=93If we knew what it was we were doing, it would not be called =
research, would it?=94
 =96 Albert Einstein


From general-return-3281-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Apr 21 16:53:01 2011
Return-Path: <general-return-3281-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ABEA922BE
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 21 Apr 2011 16:53:01 +0000 (UTC)
Received: (qmail 57574 invoked by uid 500); 21 Apr 2011 16:53:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57528 invoked by uid 500); 21 Apr 2011 16:53:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57520 invoked by uid 99); 21 Apr 2011 16:53:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Apr 2011 16:53:00 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Apr 2011 16:52:54 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=Tt4YJAWE8XG9YpsnbcUAt2p6J73gIRqFgmNTcWyip8Gg3gODje9LtW+c
   t4h6cBMGp647kRJNkexC8VNW+GxpbDbVH72Ebkz4pwPEXbaHV15ZgIsd7
   7WlRJV+Dh9K8pkl;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1303404774; x=1334940774;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20[VOTE]=20Abandon=20hod=20Common=20contr
   ib|Date:=20Thu,=2021=20Apr=202011=2016:55:37=20+0000
   |Message-ID:=20<0DCCF5A1-48A5-42B5-B609-BCB81E3D8948@link
   edin.com>|To:=20"<general@hadoop.apache.org>"=20<general@
   hadoop.apache.org>|MIME-Version:=201.0
   |Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<3683620D169183489F4FA6EB4984EC35@linkedin
   .com>|In-Reply-To:=20<84B97117-5DA4-4A32-912A-8DBFCE3D063
   E@linkedin.com>|References:=20<AC35A7EF-1D68-4055-8D47-ED
   A2FCF8C2F6@mac.com>=0D=0A=20<84B97117-5DA4-4A32-912A-8DBF
   CE3D063E@linkedin.com>;
  bh=MJyYxoy7rDizGslq3mw1Y23ec0SaFCw0T8CXMoijj24=;
  b=yn40LySsgh8+8LQ3BAhjZ3bGMLs8tjj/xcmgxg+KH8iRbE3AqjfSAMNm
   0witzcSTovKMq2Ah7/RUQ+twrCsotJbZh9p+H/g8Tti07U66hTyeJHHlS
   9fmlH3IGN8Z/ST6;
X-IronPort-AV: E=Sophos;i="4.64,252,1301900400"; 
   d="scan'208";a="22420310"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi id 14.01.0251.000; Thu,
 21 Apr 2011 09:55:38 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: [VOTE] Abandon hod Common contrib
Thread-Topic: [VOTE] Abandon hod Common contrib
Thread-Index: AQHLyZbt98XqvxdoQkW+Mjf3xnP4gZP9GkiAgGxQ6YA=
Date: Thu, 21 Apr 2011 16:55:37 +0000
Message-ID: <0DCCF5A1-48A5-42B5-B609-BCB81E3D8948@linkedin.com>
References: <AC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com>
 <84B97117-5DA4-4A32-912A-8DBFCE3D063E@linkedin.com>
In-Reply-To: <84B97117-5DA4-4A32-912A-8DBFCE3D063E@linkedin.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <3683620D169183489F4FA6EB4984EC35@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


On Feb 11, 2011, at 11:48 AM, Allen Wittenauer wrote:

>=20
> On Feb 10, 2011, at 6:51 PM, Nigel Daley wrote:
>=20
>> I think the PMC should abandon the hod common contrib component.  It's l=
ast meaningful contribution was February 2009:
>>=20
>> HADOOP-2898. Provide an option to specify a port range for Hadoop servic=
es provisioned by HOD. Contributed by Peeyush Bishnoi.
>>=20
>> There are 44 unresolved contrib/hod issues in Jira, 1 of them Patch Avai=
lable since November 2009 (HADOOP-6369).
>=20
> -1


	I'm removing my -1 to make this a +/-0.=20

	Do what you want.


From general-return-3282-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Apr 27 12:20:27 2011
Return-Path: <general-return-3282-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 356332669
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 27 Apr 2011 12:20:27 +0000 (UTC)
Received: (qmail 75497 invoked by uid 500); 27 Apr 2011 12:20:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75417 invoked by uid 500); 27 Apr 2011 12:20:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75408 invoked by uid 99); 27 Apr 2011 12:20:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Apr 2011 12:20:24 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Apr 2011 12:20:17 +0000
Received: by qwj9 with SMTP id 9so1008306qwj.35
        for <general@hadoop.apache.org>; Wed, 27 Apr 2011 05:19:56 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:from:content-type:content-transfer-encoding
         :subject:date:references:to:message-id:mime-version:x-mailer;
        bh=oPpsU/UJv+GLB7/bQr4mqUYISHTCX+caNc+8G5oc4UY=;
        b=d2IVoX3JYVvnEOOZd7qt8R6bbpbr0xe+AlVVhKW6qOHFkJfNeYKPaM43Fput5Ww2cY
         kLCsbH7WblkjowVt9vT5StG+kkmTjN+C8g2j4zRzVF+sxnAp6BnFwtZWqNS9sBncpu0T
         aGvZ7mlcQ6Fz6+On13yqiikIL3ACws8/Xa+/M=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=from:content-type:content-transfer-encoding:subject:date:references
         :to:message-id:mime-version:x-mailer;
        b=J1vulXWObnMkx22j3KPnMR0mMDWXxP8x7qAPf+gJtUynJnhYibZfxmWE0vbKGkUwzC
         3ugQ7FE+TQ7TBA6jDKJ+Q9rN6W3izfBmCadFsBVBYqAQSkIRiNyWuur86vfBM/AXstfq
         Ci3mqTv7p/Cm6IWIdwoUTl2O/OSFwbKcuO+1g=
Received: by 10.229.56.148 with SMTP id y20mr1630610qcg.283.1303906796515;
        Wed, 27 Apr 2011 05:19:56 -0700 (PDT)
Received: from malinux.office.aol.com (h-64-236-128-42.nat.aol.com [64.236.128.42])
        by mx.google.com with ESMTPS id s16sm536193qco.37.2011.04.27.05.19.55
        (version=SSLv3 cipher=OTHER);
        Wed, 27 Apr 2011 05:19:55 -0700 (PDT)
From: Ian Holsman <ian@holsman.net>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: removing old releases
Date: Wed, 27 Apr 2011 08:19:54 -0400
References: <4DB80719.10903@apache.org>
To: general@hadoop.apache.org
Message-Id: <7CD77ED8-72DC-4A09-BA34-B08B74B9D592@Holsman.net>
Mime-Version: 1.0 (Apple Message framework v1084)
X-Mailer: Apple Mail (2.1084)

I've had a look at hadoop's dist area =
(http://www.apache.org/dist/hadoop/ )=20
and was wondering which of these people think we should do here.

If no one objects within a week, I will remove everything except the =
hadoop-0.20.2,=20
hadoop-21.0, and stable directories there.

Thanks
Ian

Begin forwarded message:

> From: Mark Thomas <m>
> Date: April 27, 2011 8:07:53 AM EDT
> Subject: Re: [ACTION REQUIRED] www.apache.org/dist/<tlp> housekeeping
>=20

> PMC members,
>=20
> Six weeks ago, the infrastructure team sent you the e-mail below. To =
the
> projects that were already following the release guidelines and to =
those
> projects that have since cleaned up their /dist area - thank you. The
> infrastructure team really does appreciate you doing this.
>=20
> Regrettably, a large number of PMCs have chosen to ignore the message
> below. To those PMCs, you have one month (until 31 May 2011) to clean =
up
> your dist area or the infrastructure team will simply remove all files
> that are more than twelve months old.
>=20
> Mark
> on behalf of the ASF Infrastructure Team
>=20
> On 10/03/2011 08:16, Mark Thomas wrote:
>> PMC members,
>>=20
>> As the ASF grows in size, so does the total size of the distribution
>> artefacts we ask our mirror community to support for us. The larger =
this
>> total size, the greater the strain on both ASF infrastructure and on =
the
>> mirroring system.
>>=20
>> As per the release guidelines [1], only current releases should be
>> available at http://www.apache.org/dist/. Monitoring of
>> http://www.apache.org/dist/ [2] shows that some projects are not
>> removing old releases. This is placing an unnecessary strain on both =
ASF
>> infrastructure and on our mirror volunteers.
>>=20
>> Thanks to those PMCs that have been removing old releases from their
>> distribution directory. The infrastructure appreciates you keeping on
>> top of this.
>>=20
>> PMCs that have not been removing old releases are required to review
>> their current distribution directory and remove any old releases.
>> - PMCs using svnpubsub should remove old releases via svn.
>> - PMCs not using svnpubsub should remove old releases directly from
>> /www/www.apache.org/dist/<tlp> on people.apache.org at. Note that any
>> deletions may take up to 24 hours to replicate to =
http://www.apache.org/dist
>> In both cases it may take longer for changes to replicate to mirrors.
>>=20
>> Old releases removed from http://www.apache.org/dist/ are not lost.
>> Release are automatically copied to http://archive.apache.org/dist/ =
and
>> are never deleted.
>>=20
>> This inevitably raises the question what is a current release and =
what
>> is an old release. To some extent, this varies from project to =
project
>> but typically it amounts to the following:
>> a) latest release of the current branch
>> b) latest stable release of the current branch
>> c) latest stable release of previous branches
>>=20
>> It is hard to give concrete examples that apply to all projects since
>> each project is free to use its own release numbering scheme. =
However, a
>> project that includes versions 2.1.0, 2.1.1 and 2.1.2 in its release
>> directory almost certainly has some cleaning up to do. A project that
>> includes 1.0.6, 1.1.5 and 2.0.7 probably doesn't.
>>=20
>> If you have any questions about how to manage your distribution
>> directory please contact the infrastructure team.
>>=20
>> Thanks in advance,
>>=20
>> Mark
>> on behalf of the ASF Infrastructure Team
>>=20
>>=20
>> [1] http://www.apache.org/dev/release.html
>> [2] http://people.apache.org/~henkp/tlps/
>=20
>=20
>=20

--
Ian Holsman
Ian@Holsman.net
PH: +1-703 879-3128 AOLIM: ianholsman Skype:iholsman

Never explain - your friends do not need it and your enemies will not =
believe you anyway. Elbert Hubbard





From general-return-3283-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Apr 27 20:37:50 2011
Return-Path: <general-return-3283-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8F63A66A
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 27 Apr 2011 20:37:50 +0000 (UTC)
Received: (qmail 31877 invoked by uid 500); 27 Apr 2011 20:37:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31811 invoked by uid 500); 27 Apr 2011 20:37:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31803 invoked by uid 99); 27 Apr 2011 20:37:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Apr 2011 20:37:48 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Apr 2011 20:37:41 +0000
Received: by qyk30 with SMTP id 30so1383444qyk.14
        for <general@hadoop.apache.org>; Wed, 27 Apr 2011 13:37:19 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=yl1MWs1cNBec+cPIML2gAA6bhS+JVk/CpoHL91ezeWQ=;
        b=XLnAKRE3sFubWBhgHXdlst+FuysHXOiM3wsbMCmMHqqTnAzr1xZmp8ztONKaR1yctq
         6bkh+bJuv3h3/WFLV7BR58Tt+ukfyUbl9WzBui07Bi+LbokvQsClDm1pG8NQX1QdcVrp
         HTflQHk8z8XV1cwf6ZgZvIEBVn7x321cPSvSc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=RelWsTBAZCBFNic2765X9avXUIscBZ4FCw2JM13o23chj2TYtqRSHSq8kUXJWUsf3I
         3BhzMYOQmfIozTzEJYs97FU2U9tHgNmTYcMYvlTrVrZVc/yWCunEWG+pgV8F3JwFWvgI
         eH/5Z8xwJf6OTN/AFWDseLf0pSO1hu6cU4Mt0=
Received: by 10.229.20.77 with SMTP id e13mr2143512qcb.193.1303936638047;
        Wed, 27 Apr 2011 13:37:18 -0700 (PDT)
Received: from [10.180.150.116] (h-64-236-128-40.nat.aol.com [64.236.128.40])
        by mx.google.com with ESMTPS id f5sm817114qck.32.2011.04.27.13.37.16
        (version=TLSv1/SSLv3 cipher=OTHER);
        Wed, 27 Apr 2011 13:37:16 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: removing old releases
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <7CD77ED8-72DC-4A09-BA34-B08B74B9D592@Holsman.net>
Date: Wed, 27 Apr 2011 16:37:15 -0400
Content-Transfer-Encoding: quoted-printable
Message-Id: <1BF00F04-E81D-4A10-93CE-2F2E29973DD1@Holsman.NET>
References: <4DB80719.10903@apache.org> <7CD77ED8-72DC-4A09-BA34-B08B74B9D592@Holsman.net>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

just to clarify.. there is a archive area - =
http://archive.apache.org/dist/hadoop/core/ where old releases are kept, =
so we are not deleting anything forever, just not having it available =
via the mirrors and taking up valuable diskspace etc etc.
On Apr 27, 2011, at 8:19 AM, Ian Holsman wrote:

> I've had a look at hadoop's dist area =
(http://www.apache.org/dist/hadoop/ )=20
> and was wondering which of these people think we should do here.
>=20
> If no one objects within a week, I will remove everything except the =
hadoop-0.20.2,=20
> hadoop-21.0, and stable directories there.
>=20
> Thanks
> Ian
>=20
> Begin forwarded message:
>=20
>> From: Mark Thomas <m>
>> Date: April 27, 2011 8:07:53 AM EDT
>> Subject: Re: [ACTION REQUIRED] www.apache.org/dist/<tlp> housekeeping
>>=20
>=20
>> PMC members,
>>=20
>> Six weeks ago, the infrastructure team sent you the e-mail below. To =
the
>> projects that were already following the release guidelines and to =
those
>> projects that have since cleaned up their /dist area - thank you. The
>> infrastructure team really does appreciate you doing this.
>>=20
>> Regrettably, a large number of PMCs have chosen to ignore the message
>> below. To those PMCs, you have one month (until 31 May 2011) to clean =
up
>> your dist area or the infrastructure team will simply remove all =
files
>> that are more than twelve months old.
>>=20
>> Mark
>> on behalf of the ASF Infrastructure Team
>>=20
>> On 10/03/2011 08:16, Mark Thomas wrote:
>>> PMC members,
>>>=20
>>> As the ASF grows in size, so does the total size of the distribution
>>> artefacts we ask our mirror community to support for us. The larger =
this
>>> total size, the greater the strain on both ASF infrastructure and on =
the
>>> mirroring system.
>>>=20
>>> As per the release guidelines [1], only current releases should be
>>> available at http://www.apache.org/dist/. Monitoring of
>>> http://www.apache.org/dist/ [2] shows that some projects are not
>>> removing old releases. This is placing an unnecessary strain on both =
ASF
>>> infrastructure and on our mirror volunteers.
>>>=20
>>> Thanks to those PMCs that have been removing old releases from their
>>> distribution directory. The infrastructure appreciates you keeping =
on
>>> top of this.
>>>=20
>>> PMCs that have not been removing old releases are required to review
>>> their current distribution directory and remove any old releases.
>>> - PMCs using svnpubsub should remove old releases via svn.
>>> - PMCs not using svnpubsub should remove old releases directly from
>>> /www/www.apache.org/dist/<tlp> on people.apache.org at. Note that =
any
>>> deletions may take up to 24 hours to replicate to =
http://www.apache.org/dist
>>> In both cases it may take longer for changes to replicate to =
mirrors.
>>>=20
>>> Old releases removed from http://www.apache.org/dist/ are not lost.
>>> Release are automatically copied to http://archive.apache.org/dist/ =
and
>>> are never deleted.
>>>=20
>>> This inevitably raises the question what is a current release and =
what
>>> is an old release. To some extent, this varies from project to =
project
>>> but typically it amounts to the following:
>>> a) latest release of the current branch
>>> b) latest stable release of the current branch
>>> c) latest stable release of previous branches
>>>=20
>>> It is hard to give concrete examples that apply to all projects =
since
>>> each project is free to use its own release numbering scheme. =
However, a
>>> project that includes versions 2.1.0, 2.1.1 and 2.1.2 in its release
>>> directory almost certainly has some cleaning up to do. A project =
that
>>> includes 1.0.6, 1.1.5 and 2.0.7 probably doesn't.
>>>=20
>>> If you have any questions about how to manage your distribution
>>> directory please contact the infrastructure team.
>>>=20
>>> Thanks in advance,
>>>=20
>>> Mark
>>> on behalf of the ASF Infrastructure Team
>>>=20
>>>=20
>>> [1] http://www.apache.org/dev/release.html
>>> [2] http://people.apache.org/~henkp/tlps/
>>=20
>>=20
>>=20
>=20
> --
> Ian Holsman
> Ian@Holsman.net
> PH: +1-703 879-3128 AOLIM: ianholsman Skype:iholsman
>=20
> Never explain - your friends do not need it and your enemies will not =
believe you anyway. Elbert Hubbard
>=20
>=20
>=20
>=20


From general-return-3284-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Apr 27 21:01:05 2011
Return-Path: <general-return-3284-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 59CF5EE2
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 27 Apr 2011 21:01:05 +0000 (UTC)
Received: (qmail 77534 invoked by uid 500); 27 Apr 2011 21:01:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77470 invoked by uid 500); 27 Apr 2011 21:01:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77462 invoked by uid 99); 27 Apr 2011 21:01:02 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Apr 2011 21:01:02 +0000
Received: from localhost (HELO mail-iw0-f176.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Apr 2011 21:01:02 +0000
Received: by iwr19 with SMTP id 19so2678907iwr.35
        for <general@hadoop.apache.org>; Wed, 27 Apr 2011 14:01:01 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.61.65 with SMTP id s1mr2025083ibh.120.1303938061837; Wed,
 27 Apr 2011 14:01:01 -0700 (PDT)
Received: by 10.231.166.76 with HTTP; Wed, 27 Apr 2011 14:01:01 -0700 (PDT)
In-Reply-To: <7CD77ED8-72DC-4A09-BA34-B08B74B9D592@Holsman.net>
References: <4DB80719.10903@apache.org>
	<7CD77ED8-72DC-4A09-BA34-B08B74B9D592@Holsman.net>
Date: Wed, 27 Apr 2011 14:01:01 -0700
Message-ID: <BANLkTimTTaULkxUej85f3TZFyB4=rkBtxQ@mail.gmail.com>
Subject: Re: removing old releases
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

No need to wait a week, if you have time to clear it out. Thanks for
taking care of this, Ian. -C

On Wed, Apr 27, 2011 at 5:19 AM, Ian Holsman <ian@holsman.net> wrote:
> I've had a look at hadoop's dist area (http://www.apache.org/dist/hadoop/ )
> and was wondering which of these people think we should do here.
>
> If no one objects within a week, I will remove everything except the hadoop-0.20.2,
> hadoop-21.0, and stable directories there.
>
> Thanks
> Ian
>
> Begin forwarded message:
>
>> From: Mark Thomas <m>
>> Date: April 27, 2011 8:07:53 AM EDT
>> Subject: Re: [ACTION REQUIRED] www.apache.org/dist/<tlp> housekeeping
>>
>
>> PMC members,
>>
>> Six weeks ago, the infrastructure team sent you the e-mail below. To the
>> projects that were already following the release guidelines and to those
>> projects that have since cleaned up their /dist area - thank you. The
>> infrastructure team really does appreciate you doing this.
>>
>> Regrettably, a large number of PMCs have chosen to ignore the message
>> below. To those PMCs, you have one month (until 31 May 2011) to clean up
>> your dist area or the infrastructure team will simply remove all files
>> that are more than twelve months old.
>>
>> Mark
>> on behalf of the ASF Infrastructure Team
>>
>> On 10/03/2011 08:16, Mark Thomas wrote:
>>> PMC members,
>>>
>>> As the ASF grows in size, so does the total size of the distribution
>>> artefacts we ask our mirror community to support for us. The larger this
>>> total size, the greater the strain on both ASF infrastructure and on the
>>> mirroring system.
>>>
>>> As per the release guidelines [1], only current releases should be
>>> available at http://www.apache.org/dist/. Monitoring of
>>> http://www.apache.org/dist/ [2] shows that some projects are not
>>> removing old releases. This is placing an unnecessary strain on both ASF
>>> infrastructure and on our mirror volunteers.
>>>
>>> Thanks to those PMCs that have been removing old releases from their
>>> distribution directory. The infrastructure appreciates you keeping on
>>> top of this.
>>>
>>> PMCs that have not been removing old releases are required to review
>>> their current distribution directory and remove any old releases.
>>> - PMCs using svnpubsub should remove old releases via svn.
>>> - PMCs not using svnpubsub should remove old releases directly from
>>> /www/www.apache.org/dist/<tlp> on people.apache.org at. Note that any
>>> deletions may take up to 24 hours to replicate to http://www.apache.org/dist
>>> In both cases it may take longer for changes to replicate to mirrors.
>>>
>>> Old releases removed from http://www.apache.org/dist/ are not lost.
>>> Release are automatically copied to http://archive.apache.org/dist/ and
>>> are never deleted.
>>>
>>> This inevitably raises the question what is a current release and what
>>> is an old release. To some extent, this varies from project to project
>>> but typically it amounts to the following:
>>> a) latest release of the current branch
>>> b) latest stable release of the current branch
>>> c) latest stable release of previous branches
>>>
>>> It is hard to give concrete examples that apply to all projects since
>>> each project is free to use its own release numbering scheme. However, a
>>> project that includes versions 2.1.0, 2.1.1 and 2.1.2 in its release
>>> directory almost certainly has some cleaning up to do. A project that
>>> includes 1.0.6, 1.1.5 and 2.0.7 probably doesn't.
>>>
>>> If you have any questions about how to manage your distribution
>>> directory please contact the infrastructure team.
>>>
>>> Thanks in advance,
>>>
>>> Mark
>>> on behalf of the ASF Infrastructure Team
>>>
>>>
>>> [1] http://www.apache.org/dev/release.html
>>> [2] http://people.apache.org/~henkp/tlps/
>>
>>
>>
>
> --
> Ian Holsman
> Ian@Holsman.net
> PH: +1-703 879-3128 AOLIM: ianholsman Skype:iholsman
>
> Never explain - your friends do not need it and your enemies will not believe you anyway. Elbert Hubbard
>
>
>
>
>

From general-return-3285-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Wed Apr 27 22:06:27 2011
Return-Path: <general-return-3285-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0DED52335
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Wed, 27 Apr 2011 22:06:27 +0000 (UTC)
Received: (qmail 74218 invoked by uid 500); 27 Apr 2011 22:06:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74167 invoked by uid 500); 27 Apr 2011 22:06:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74159 invoked by uid 99); 27 Apr 2011 22:06:25 -0000
Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Apr 2011 22:06:25 +0000
Received: from localhost (HELO mail-qy0-f176.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Apr 2011 22:06:25 +0000
Received: by qyk30 with SMTP id 30so1433058qyk.14
        for <general@hadoop.apache.org>; Wed, 27 Apr 2011 15:06:24 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.106.214 with SMTP id y22mr2136340qco.105.1303941984235;
 Wed, 27 Apr 2011 15:06:24 -0700 (PDT)
Received: by 10.229.235.131 with HTTP; Wed, 27 Apr 2011 15:06:24 -0700 (PDT)
In-Reply-To: <BANLkTimTTaULkxUej85f3TZFyB4=rkBtxQ@mail.gmail.com>
References: <4DB80719.10903@apache.org>
	<7CD77ED8-72DC-4A09-BA34-B08B74B9D592@Holsman.net>
	<BANLkTimTTaULkxUej85f3TZFyB4=rkBtxQ@mail.gmail.com>
Date: Wed, 27 Apr 2011 15:06:24 -0700
Message-ID: <BANLkTikRTz9ZvYtzHc+ZC6VnyWU6Xbim3Q@mail.gmail.com>
Subject: Re: removing old releases
From: "Owen O'Malley" <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=002354470558c732ff04a1eda6ae

--002354470558c732ff04a1eda6ae
Content-Type: text/plain; charset=UTF-8

On Wed, Apr 27, 2011 at 2:01 PM, Chris Douglas <cdouglas@apache.org> wrote:

> No need to wait a week, if you have time to clear it out. Thanks for
> taking care of this, Ian.


+1 on cleaning it up now. I doubt that there would be any debate that the
"current" releases are 0.20.2 and 0.21.0. Also removing the ex-subprojects
can proceed immediately as none of those releases are current either.

--002354470558c732ff04a1eda6ae--

From general-return-3286-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Apr 28 06:02:49 2011
Return-Path: <general-return-3286-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B19F9976
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 28 Apr 2011 06:02:49 +0000 (UTC)
Received: (qmail 3522 invoked by uid 500); 28 Apr 2011 06:02:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3467 invoked by uid 500); 28 Apr 2011 06:02:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3437 invoked by uid 99); 28 Apr 2011 06:02:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Apr 2011 06:02:45 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.97 as permitted sender)
Received: from [17.148.16.97] (HELO asmtpout022.mac.com) (17.148.16.97)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Apr 2011 06:02:35 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.8] ([71.198.193.36])
 by asmtp022.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LKC00IZ9M3PCV10@asmtp022.mac.com>
 for general@hadoop.apache.org; Wed, 27 Apr 2011 23:02:14 -0700 (PDT)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-04-28_03:2011-04-27,2011-04-28,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1104270211
Subject: Re: removing old releases
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <BANLkTimTTaULkxUej85f3TZFyB4=rkBtxQ@mail.gmail.com>
Date: Wed, 27 Apr 2011 23:02:13 -0700
Message-id: <678B57E6-0F3E-4081-B92E-CCF4ADC10B7F@mac.com>
References: <4DB80719.10903@apache.org>
 <7CD77ED8-72DC-4A09-BA34-B08B74B9D592@Holsman.net>
 <BANLkTimTTaULkxUej85f3TZFyB4=rkBtxQ@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

+1. Thanks Ian.

n.

On Apr 27, 2011, at 2:01 PM, Chris Douglas wrote:

> No need to wait a week, if you have time to clear it out. Thanks for
> taking care of this, Ian. -C
> 
> On Wed, Apr 27, 2011 at 5:19 AM, Ian Holsman <ian@holsman.net> wrote:
>> I've had a look at hadoop's dist area (http://www.apache.org/dist/hadoop/ )
>> and was wondering which of these people think we should do here.
>> 
>> If no one objects within a week, I will remove everything except the hadoop-0.20.2,
>> hadoop-21.0, and stable directories there.
>> 
>> Thanks
>> Ian
>> 
>> Begin forwarded message:
>> 
>>> From: Mark Thomas <m>
>>> Date: April 27, 2011 8:07:53 AM EDT
>>> Subject: Re: [ACTION REQUIRED] www.apache.org/dist/<tlp> housekeeping
>>> 
>> 
>>> PMC members,
>>> 
>>> Six weeks ago, the infrastructure team sent you the e-mail below. To the
>>> projects that were already following the release guidelines and to those
>>> projects that have since cleaned up their /dist area - thank you. The
>>> infrastructure team really does appreciate you doing this.
>>> 
>>> Regrettably, a large number of PMCs have chosen to ignore the message
>>> below. To those PMCs, you have one month (until 31 May 2011) to clean up
>>> your dist area or the infrastructure team will simply remove all files
>>> that are more than twelve months old.
>>> 
>>> Mark
>>> on behalf of the ASF Infrastructure Team
>>> 
>>> On 10/03/2011 08:16, Mark Thomas wrote:
>>>> PMC members,
>>>> 
>>>> As the ASF grows in size, so does the total size of the distribution
>>>> artefacts we ask our mirror community to support for us. The larger this
>>>> total size, the greater the strain on both ASF infrastructure and on the
>>>> mirroring system.
>>>> 
>>>> As per the release guidelines [1], only current releases should be
>>>> available at http://www.apache.org/dist/. Monitoring of
>>>> http://www.apache.org/dist/ [2] shows that some projects are not
>>>> removing old releases. This is placing an unnecessary strain on both ASF
>>>> infrastructure and on our mirror volunteers.
>>>> 
>>>> Thanks to those PMCs that have been removing old releases from their
>>>> distribution directory. The infrastructure appreciates you keeping on
>>>> top of this.
>>>> 
>>>> PMCs that have not been removing old releases are required to review
>>>> their current distribution directory and remove any old releases.
>>>> - PMCs using svnpubsub should remove old releases via svn.
>>>> - PMCs not using svnpubsub should remove old releases directly from
>>>> /www/www.apache.org/dist/<tlp> on people.apache.org at. Note that any
>>>> deletions may take up to 24 hours to replicate to http://www.apache.org/dist
>>>> In both cases it may take longer for changes to replicate to mirrors.
>>>> 
>>>> Old releases removed from http://www.apache.org/dist/ are not lost.
>>>> Release are automatically copied to http://archive.apache.org/dist/ and
>>>> are never deleted.
>>>> 
>>>> This inevitably raises the question what is a current release and what
>>>> is an old release. To some extent, this varies from project to project
>>>> but typically it amounts to the following:
>>>> a) latest release of the current branch
>>>> b) latest stable release of the current branch
>>>> c) latest stable release of previous branches
>>>> 
>>>> It is hard to give concrete examples that apply to all projects since
>>>> each project is free to use its own release numbering scheme. However, a
>>>> project that includes versions 2.1.0, 2.1.1 and 2.1.2 in its release
>>>> directory almost certainly has some cleaning up to do. A project that
>>>> includes 1.0.6, 1.1.5 and 2.0.7 probably doesn't.
>>>> 
>>>> If you have any questions about how to manage your distribution
>>>> directory please contact the infrastructure team.
>>>> 
>>>> Thanks in advance,
>>>> 
>>>> Mark
>>>> on behalf of the ASF Infrastructure Team
>>>> 
>>>> 
>>>> [1] http://www.apache.org/dev/release.html
>>>> [2] http://people.apache.org/~henkp/tlps/
>>> 
>>> 
>>> 
>> 
>> --
>> Ian Holsman
>> Ian@Holsman.net
>> PH: +1-703 879-3128 AOLIM: ianholsman Skype:iholsman
>> 
>> Never explain - your friends do not need it and your enemies will not believe you anyway. Elbert Hubbard
>> 
>> 
>> 
>> 
>> 


From general-return-3287-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Apr 28 13:46:11 2011
Return-Path: <general-return-3287-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F405E3E9D
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 28 Apr 2011 13:46:10 +0000 (UTC)
Received: (qmail 32398 invoked by uid 500); 28 Apr 2011 13:46:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32316 invoked by uid 500); 28 Apr 2011 13:46:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32308 invoked by uid 99); 28 Apr 2011 13:46:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Apr 2011 13:46:08 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Apr 2011 13:46:02 +0000
Received: by vws7 with SMTP id 7so3179189vws.35
        for <general@hadoop.apache.org>; Thu, 28 Apr 2011 06:45:40 -0700 (PDT)
Received: by 10.52.70.171 with SMTP id n11mr4882490vdu.53.1303998340750; Thu,
 28 Apr 2011 06:45:40 -0700 (PDT)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.52.156.163 with HTTP; Thu, 28 Apr 2011 06:45:16 -0700 (PDT)
In-Reply-To: <BANLkTim5wm0oupnJ5VNnKXj7z__64H6WQw@mail.gmail.com>
References: <FF258184-01BE-419F-B5A0-69051472BBDB@Holsman.net>
 <AANLkTimsma0e9cGAN0o_zz4dBDNRnkDXEgKcVx00zbEV@mail.gmail.com>
 <597116E4-1991-43A6-B35B-BAA9F577897E@Holsman.NET> <AANLkTi=cPCdv1ipMSN41Cpabhg_jbSZvDRh8_BzyUFR_@mail.gmail.com>
 <AANLkTika3WVVhJKXNQ2O4Kfd16jamiT5k2RJSy=Yi8ao@mail.gmail.com> <BANLkTim5wm0oupnJ5VNnKXj7z__64H6WQw@mail.gmail.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Thu, 28 Apr 2011 06:45:16 -0700
X-Google-Sender-Auth: HfItWbIDet90hGiTcHNMnTRsK_g
Message-ID: <BANLkTi=CdazmWt5=3FvuYy223Zu8yiD0NA@mail.gmail.com>
Subject: Re: Apache Sonar - http://analysis.apache.org/ does anyone want
 hadoop in there?
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Any one is willing to take a look at the patch? Pretty straightforward...
--
=A0 Take care,
Konstantin (Cos) Boudnik

On Wed, Apr 6, 2011 at 13:17, Konstantin Boudnik <cos@apache.org> wrote:
> I have submitted for enabling Sonar analysis on HDFS code. Would
> appreciate if someone takes a look so we can move forward with it
> (HDFS-1741).
> --
> =A0 Take care,
> Konstantin (Cos) Boudnik
>
> On Wed, Mar 9, 2011 at 08:30, Konstantin Boudnik <cos@apache.org> wrote:
>> I have looked around Apache's Sonar server and it seems to be
>> sufficient to just have a correct pom.xml in place. I have opened
>> =A0https://issues.apache.org/jira/browse/HDFS-1741
>> Will attend to it shortly.
>> --
>> =A0 Take care,
>> Konstantin (Cos) Boudnik
>>
>> On Wed, Mar 9, 2011 at 08:06, Konstantin Boudnik <cos@apache.org> wrote:
>>> Hi Ian.
>>>
>>> yes, Sonar needs to have a pom'ed project. In case of an ant build
>>> such a pom can be created to 'wrap' around build.xml and is usually
>>> quite simple to manufacture. I have done it in the past for some HDFS
>>> experiments without much of a hassle.
>>>
>>> Here's the example:
>>> =A0http://docs.codehaus.org/display/SONAR/Analyse+with+Maven
>>> in "Analyze projects which are not built with Maven" section.
>>>
>>> There's one more caveat though: handling of unit tests reports. It can
>>> be done with some new features Sonar has. See here:
>>> =A0http://www.sonarsource.org/tag/ant/
>>>
>>> I can quickly provide a working fixture to add say HDFS into Sonar as
>>> a starter. Pl. let me know where it needs to be put (perhaps a JIRA
>>> with a patch would do to?).
>>>
>>> Thanks for looking into this,
>>> =A0Cos
>>>
>>> On Wed, Mar 9, 2011 at 06:10, Ian Holsman <hadoop@holsman.net> wrote:
>>>> Hi Cos.
>>>> apparently they can't build from ant in the current version.
>>>> I'll need to have a look at it.. they sent me some info.
>>>>
>>>> On Mar 3, 2011, at 12:34 PM, Konstantin Boudnik wrote:
>>>>
>>>>> Ian,
>>>>>
>>>>> Sonar was around for a while and Apache has this setup since late
>>>>> 2009, I believe. =A0I had asked =A0INFRA folks on a few occasions abo=
ut
>>>>> adding Hadoop repos to their installation, but never heard anything
>>>>> back.
>>>>>
>>>>> I'd be awesome if you can talk to the right people so it'd happen eve=
ntually.
>>>>> --
>>>>> =A0 Thanks in advance,
>>>>> Konstantin (Cos) Boudnik
>>>>>
>>>>>
>>>>> On Thu, Mar 3, 2011 at 05:59, Ian Holsman <ian@holsman.net> wrote:
>>>>>> I just discovered this Apache site,
>>>>>> Apache Sonar performs source code analysis on the java code, and it =
looks pretty, and I'm sure some people would find it useful.
>>>>>>
>>>>>> If people are interested I can start putting finding out how to put =
the hadoop code in there, so we can get stats on our own code.
>>>>>>
>>>>>> Regards
>>>>>> Ian
>>>>>> --
>>>>>> Ian Holsman
>>>>>> Ian@Holsman.net
>>>>>> PH: +1-703 879-3128 AOLIM: ianholsman Skype:iholsman
>>>>>>
>>>>>> To know recursion, you must first know recursion.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>
>>
>

From general-return-3288-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Apr 28 13:49:32 2011
Return-Path: <general-return-3288-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A185D3C1F
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 28 Apr 2011 13:49:32 +0000 (UTC)
Received: (qmail 34882 invoked by uid 500); 28 Apr 2011 13:49:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34819 invoked by uid 500); 28 Apr 2011 13:49:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34811 invoked by uid 99); 28 Apr 2011 13:49:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Apr 2011 13:49:31 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Apr 2011 13:49:23 +0000
Received: by qwj9 with SMTP id 9so1799225qwj.35
        for <general@hadoop.apache.org>; Thu, 28 Apr 2011 06:49:02 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=BE+kHZqBYtKF0dd4XbppdN02XXwtBfRVMdbfLYPWaVA=;
        b=vLoVlki3l0PR8K4L2ErpJblwiU6oaqEPu36R+G1JBIBKOzvkupLvKDt5rujdRdtZZS
         HfTwMbvW2PaXbsYdtgzkkUUNzAihsgt8p2uTOl+6JMcTdyOrewD9ZVtZ+upsTCS4WhxB
         9FplEmPVCeHHYDBeqBo/XDvT2ejaNqhYqqGZw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=MfPxTYjVBHCVrKibsI1/l5vOlEOs9Ws71kBY3FT3A00e0trL2sX5to/l2Gt5EnYQTB
         xLNqcO2xDZne7DI+WB+xduCgEb/2r+yLRnfNuD18hK2yoab0zJYa/9oNqdf+PmsbtbJS
         jRVcweKjSSazl5hqcBQxhfEAXuXOaoiWxhD7A=
Received: by 10.224.126.221 with SMTP id d29mr2763378qas.201.1303998542495;
        Thu, 28 Apr 2011 06:49:02 -0700 (PDT)
Received: from [10.180.150.116] (h-64-236-128-40.nat.aol.com [64.236.128.40])
        by mx.google.com with ESMTPS id f5sm1310637qck.32.2011.04.28.06.49.01
        (version=TLSv1/SSLv3 cipher=OTHER);
        Thu, 28 Apr 2011 06:49:01 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: removing old releases
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <678B57E6-0F3E-4081-B92E-CCF4ADC10B7F@mac.com>
Date: Thu, 28 Apr 2011 09:49:00 -0400
Content-Transfer-Encoding: quoted-printable
Message-Id: <BBF524DD-EE12-4E9C-BC21-A3A5B62EBF40@Holsman.NET>
References: <4DB80719.10903@apache.org> <7CD77ED8-72DC-4A09-BA34-B08B74B9D592@Holsman.net> <BANLkTimTTaULkxUej85f3TZFyB4=rkBtxQ@mail.gmail.com> <678B57E6-0F3E-4081-B92E-CCF4ADC10B7F@mac.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

and done.
I left chukwa 0.4.0 there as they are currently pointing to it on their =
release page.

On Apr 28, 2011, at 2:02 AM, Nigel Daley wrote:

> +1. Thanks Ian.
>=20
> n.
>=20
> On Apr 27, 2011, at 2:01 PM, Chris Douglas wrote:
>=20
>> No need to wait a week, if you have time to clear it out. Thanks for
>> taking care of this, Ian. -C
>>=20
>> On Wed, Apr 27, 2011 at 5:19 AM, Ian Holsman <ian@holsman.net> wrote:
>>> I've had a look at hadoop's dist area =
(http://www.apache.org/dist/hadoop/ )
>>> and was wondering which of these people think we should do here.
>>>=20
>>> If no one objects within a week, I will remove everything except the =
hadoop-0.20.2,
>>> hadoop-21.0, and stable directories there.
>>>=20
>>> Thanks
>>> Ian
>>>=20
>>> Begin forwarded message:
>>>=20
>>>> From: Mark Thomas <m>
>>>> Date: April 27, 2011 8:07:53 AM EDT
>>>> Subject: Re: [ACTION REQUIRED] www.apache.org/dist/<tlp> =
housekeeping
>>>>=20
>>>=20
>>>> PMC members,
>>>>=20
>>>> Six weeks ago, the infrastructure team sent you the e-mail below. =
To the
>>>> projects that were already following the release guidelines and to =
those
>>>> projects that have since cleaned up their /dist area - thank you. =
The
>>>> infrastructure team really does appreciate you doing this.
>>>>=20
>>>> Regrettably, a large number of PMCs have chosen to ignore the =
message
>>>> below. To those PMCs, you have one month (until 31 May 2011) to =
clean up
>>>> your dist area or the infrastructure team will simply remove all =
files
>>>> that are more than twelve months old.
>>>>=20
>>>> Mark
>>>> on behalf of the ASF Infrastructure Team
>>>>=20
>>>> On 10/03/2011 08:16, Mark Thomas wrote:
>>>>> PMC members,
>>>>>=20
>>>>> As the ASF grows in size, so does the total size of the =
distribution
>>>>> artefacts we ask our mirror community to support for us. The =
larger this
>>>>> total size, the greater the strain on both ASF infrastructure and =
on the
>>>>> mirroring system.
>>>>>=20
>>>>> As per the release guidelines [1], only current releases should be
>>>>> available at http://www.apache.org/dist/. Monitoring of
>>>>> http://www.apache.org/dist/ [2] shows that some projects are not
>>>>> removing old releases. This is placing an unnecessary strain on =
both ASF
>>>>> infrastructure and on our mirror volunteers.
>>>>>=20
>>>>> Thanks to those PMCs that have been removing old releases from =
their
>>>>> distribution directory. The infrastructure appreciates you keeping =
on
>>>>> top of this.
>>>>>=20
>>>>> PMCs that have not been removing old releases are required to =
review
>>>>> their current distribution directory and remove any old releases.
>>>>> - PMCs using svnpubsub should remove old releases via svn.
>>>>> - PMCs not using svnpubsub should remove old releases directly =
from
>>>>> /www/www.apache.org/dist/<tlp> on people.apache.org at. Note that =
any
>>>>> deletions may take up to 24 hours to replicate to =
http://www.apache.org/dist
>>>>> In both cases it may take longer for changes to replicate to =
mirrors.
>>>>>=20
>>>>> Old releases removed from http://www.apache.org/dist/ are not =
lost.
>>>>> Release are automatically copied to =
http://archive.apache.org/dist/ and
>>>>> are never deleted.
>>>>>=20
>>>>> This inevitably raises the question what is a current release and =
what
>>>>> is an old release. To some extent, this varies from project to =
project
>>>>> but typically it amounts to the following:
>>>>> a) latest release of the current branch
>>>>> b) latest stable release of the current branch
>>>>> c) latest stable release of previous branches
>>>>>=20
>>>>> It is hard to give concrete examples that apply to all projects =
since
>>>>> each project is free to use its own release numbering scheme. =
However, a
>>>>> project that includes versions 2.1.0, 2.1.1 and 2.1.2 in its =
release
>>>>> directory almost certainly has some cleaning up to do. A project =
that
>>>>> includes 1.0.6, 1.1.5 and 2.0.7 probably doesn't.
>>>>>=20
>>>>> If you have any questions about how to manage your distribution
>>>>> directory please contact the infrastructure team.
>>>>>=20
>>>>> Thanks in advance,
>>>>>=20
>>>>> Mark
>>>>> on behalf of the ASF Infrastructure Team
>>>>>=20
>>>>>=20
>>>>> [1] http://www.apache.org/dev/release.html
>>>>> [2] http://people.apache.org/~henkp/tlps/
>>>>=20
>>>>=20
>>>>=20
>>>=20
>>> --
>>> Ian Holsman
>>> Ian@Holsman.net
>>> PH: +1-703 879-3128 AOLIM: ianholsman Skype:iholsman
>>>=20
>>> Never explain - your friends do not need it and your enemies will =
not believe you anyway. Elbert Hubbard
>>>=20
>>>=20
>>>=20
>>>=20
>>>=20
>=20


From general-return-3289-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Thu Apr 28 22:50:23 2011
Return-Path: <general-return-3289-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E444F3B90
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Thu, 28 Apr 2011 22:50:22 +0000 (UTC)
Received: (qmail 16171 invoked by uid 500); 28 Apr 2011 22:50:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16113 invoked by uid 500); 28 Apr 2011 22:50:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16105 invoked by uid 99); 28 Apr 2011 22:50:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Apr 2011 22:50:21 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of eli@cloudera.com designates 209.85.220.176 as permitted sender)
Received: from [209.85.220.176] (HELO mail-vx0-f176.google.com) (209.85.220.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Apr 2011 22:50:16 +0000
Received: by vxa37 with SMTP id 37so3808198vxa.35
        for <general@hadoop.apache.org>; Thu, 28 Apr 2011 15:49:55 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.52.184.98 with SMTP id et2mr446559vdc.285.1304030995397; Thu,
 28 Apr 2011 15:49:55 -0700 (PDT)
Received: by 10.52.111.131 with HTTP; Thu, 28 Apr 2011 15:49:55 -0700 (PDT)
In-Reply-To: <BANLkTi=CdazmWt5=3FvuYy223Zu8yiD0NA@mail.gmail.com>
References: <FF258184-01BE-419F-B5A0-69051472BBDB@Holsman.net>
	<AANLkTimsma0e9cGAN0o_zz4dBDNRnkDXEgKcVx00zbEV@mail.gmail.com>
	<597116E4-1991-43A6-B35B-BAA9F577897E@Holsman.NET>
	<AANLkTi=cPCdv1ipMSN41Cpabhg_jbSZvDRh8_BzyUFR_@mail.gmail.com>
	<AANLkTika3WVVhJKXNQ2O4Kfd16jamiT5k2RJSy=Yi8ao@mail.gmail.com>
	<BANLkTim5wm0oupnJ5VNnKXj7z__64H6WQw@mail.gmail.com>
	<BANLkTi=CdazmWt5=3FvuYy223Zu8yiD0NA@mail.gmail.com>
Date: Thu, 28 Apr 2011 15:49:55 -0700
Message-ID: <BANLkTinSp4Oo6BDBNTiWiA=0cLpTtvwHhw@mail.gmail.com>
Subject: Re: Apache Sonar - http://analysis.apache.org/ does anyone want
 hadoop in there?
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I replied on jira.

On Thu, Apr 28, 2011 at 6:45 AM, Konstantin Boudnik <cos@apache.org> wrote:
> Any one is willing to take a look at the patch? Pretty straightforward...
> --
> =A0 Take care,
> Konstantin (Cos) Boudnik
>
> On Wed, Apr 6, 2011 at 13:17, Konstantin Boudnik <cos@apache.org> wrote:
>> I have submitted for enabling Sonar analysis on HDFS code. Would
>> appreciate if someone takes a look so we can move forward with it
>> (HDFS-1741).
>> --
>> =A0 Take care,
>> Konstantin (Cos) Boudnik
>>
>> On Wed, Mar 9, 2011 at 08:30, Konstantin Boudnik <cos@apache.org> wrote:
>>> I have looked around Apache's Sonar server and it seems to be
>>> sufficient to just have a correct pom.xml in place. I have opened
>>> =A0https://issues.apache.org/jira/browse/HDFS-1741
>>> Will attend to it shortly.
>>> --
>>> =A0 Take care,
>>> Konstantin (Cos) Boudnik
>>>
>>> On Wed, Mar 9, 2011 at 08:06, Konstantin Boudnik <cos@apache.org> wrote=
:
>>>> Hi Ian.
>>>>
>>>> yes, Sonar needs to have a pom'ed project. In case of an ant build
>>>> such a pom can be created to 'wrap' around build.xml and is usually
>>>> quite simple to manufacture. I have done it in the past for some HDFS
>>>> experiments without much of a hassle.
>>>>
>>>> Here's the example:
>>>> =A0http://docs.codehaus.org/display/SONAR/Analyse+with+Maven
>>>> in "Analyze projects which are not built with Maven" section.
>>>>
>>>> There's one more caveat though: handling of unit tests reports. It can
>>>> be done with some new features Sonar has. See here:
>>>> =A0http://www.sonarsource.org/tag/ant/
>>>>
>>>> I can quickly provide a working fixture to add say HDFS into Sonar as
>>>> a starter. Pl. let me know where it needs to be put (perhaps a JIRA
>>>> with a patch would do to?).
>>>>
>>>> Thanks for looking into this,
>>>> =A0Cos
>>>>
>>>> On Wed, Mar 9, 2011 at 06:10, Ian Holsman <hadoop@holsman.net> wrote:
>>>>> Hi Cos.
>>>>> apparently they can't build from ant in the current version.
>>>>> I'll need to have a look at it.. they sent me some info.
>>>>>
>>>>> On Mar 3, 2011, at 12:34 PM, Konstantin Boudnik wrote:
>>>>>
>>>>>> Ian,
>>>>>>
>>>>>> Sonar was around for a while and Apache has this setup since late
>>>>>> 2009, I believe. =A0I had asked =A0INFRA folks on a few occasions ab=
out
>>>>>> adding Hadoop repos to their installation, but never heard anything
>>>>>> back.
>>>>>>
>>>>>> I'd be awesome if you can talk to the right people so it'd happen ev=
entually.
>>>>>> --
>>>>>> =A0 Thanks in advance,
>>>>>> Konstantin (Cos) Boudnik
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 3, 2011 at 05:59, Ian Holsman <ian@holsman.net> wrote:
>>>>>>> I just discovered this Apache site,
>>>>>>> Apache Sonar performs source code analysis on the java code, and it=
 looks pretty, and I'm sure some people would find it useful.
>>>>>>>
>>>>>>> If people are interested I can start putting finding out how to put=
 the hadoop code in there, so we can get stats on our own code.
>>>>>>>
>>>>>>> Regards
>>>>>>> Ian
>>>>>>> --
>>>>>>> Ian Holsman
>>>>>>> Ian@Holsman.net
>>>>>>> PH: +1-703 879-3128 AOLIM: ianholsman Skype:iholsman
>>>>>>>
>>>>>>> To know recursion, you must first know recursion.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

From general-return-3290-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Apr 29 05:08:05 2011
Return-Path: <general-return-3290-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 89DB628A3
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 29 Apr 2011 05:08:05 +0000 (UTC)
Received: (qmail 19880 invoked by uid 500); 29 Apr 2011 05:08:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19697 invoked by uid 500); 29 Apr 2011 05:08:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19684 invoked by uid 99); 29 Apr 2011 05:08:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Apr 2011 05:08:01 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.98 as permitted sender)
Received: from [17.148.16.98] (HELO asmtpout023.mac.com) (17.148.16.98)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Apr 2011 05:07:54 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.8] ([71.198.193.36])
 by asmtp023.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LKE001H6E7J0340@asmtp023.mac.com>
 for general@hadoop.apache.org; Thu, 28 Apr 2011 22:06:58 -0700 (PDT)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-04-29_03:2011-04-28,2011-04-29,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1104280218
Subject: Re: HADOOP-7106: Re-organize hadoop subversion layout
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <522E7C8B-C159-4520-A758-6E994195D0DC@mac.com>
Date: Thu, 28 Apr 2011 22:06:55 -0700
Cc: Ian Holsman <ian@holsman.net>, infrastructure-dev@apache.org
Message-id: <83FF361E-EA96-4E82-9084-19D91BCEB608@mac.com>
References: <C955EB7A.2E753%sureshms@yahoo-inc.com>
 <E2932D45-FF26-4579-A975-A98D2578622A@mac.com>
 <B98835A6-F4B7-426E-B22F-B2BEBE273C47@mac.com>
 <4112B27D-7DCF-4C8E-AD61-6884BA7E738C@mac.com>
 <844F39DC-0A5D-4AAE-AF8D-EFD4CC83C18A@mac.com>
 <BANLkTin+SyXE0tnfwhhH9P=wR0b7wCOHaw@mail.gmail.com>
 <BANLkTi=+cQBhsNjwdzB+fhcDkePJ=gSOqQ@mail.gmail.com>
 <9B529359-B589-4229-BA96-188C762A8E31@apache.org>
 <522E7C8B-C159-4520-A758-6E994195D0DC@mac.com>
To: general@hadoop.apache.org, Owen O'Malley <oom@yahoo-inc.com>,
 Todd Lipcon <todd@cloudera.com>, Suresh Srinivas <sureshms@yahoo-inc.com>
X-Mailer: Apple Mail (2.1084)

As announced last week, I'm planning to do this at 2pm PDT tomorrow (Friday) April 29.  

Suresh, when do you plan to commit HFS-1052?  That should be done first.

Owen or Todd, did you want to follow Paul's advice:
> If you're really wanting to make sure to keep the history in Git
> intact my suggestion would be to setup a temporary svn server locally
> and test our mirroring scripts against the commands you intend to run.
If so, how much more time do you need?

Cheers,
Nige

On Apr 20, 2011, at 9:42 PM, Nigel Daley wrote:

> Owen, I'll admit I'm not familiar with all the git details/issues in your proposal, but I think the layout change you propose is fine and seems to solve the git issues with very minimal impact on the layout.
> 
> Let's shoot for doing this next Friday, April 29 at 2pm PDT.  I'll update the patch and send out a reminder about this later next week.
> 
> Thanks,
> Nige
> 
> On Apr 20, 2011, at 8:00 AM, Owen O'Malley wrote:
> 
>> 
>> On Apr 19, 2011, at 10:58 PM, Todd Lipcon wrote:
>> 
>>> On Tue, Apr 19, 2011 at 10:20 PM, Todd Lipcon <todd@cloudera.com> wrote:
>>> 
>>>> 
>>>> I'm currently looking into how the git mirrors are setup in Apache-land.
>> 
>> Uh, why isn't infra-dev on this thread?
>> 
>> For those on infra-dev, the context is that Nigel is trying to merge together the source trees of the Hadoop sub-projects that were split apart 2 years ago. So he is taking:
>> 
>> prefix = http://svn.apache.org/repos/asf/hadoop/
>> 
>> $prefix/common/trunk -> $prefix/trunk/common
>> $prefix/hdfs/trunk -> $prefix/trunk/hdfs
>> $prefix/mapreduce/trunk -> $prefix/trunk/mapreduce
>> 
>> and play similar games with the rest of the branches and tags. For more details look at HADOOP-7106.
>> 
>> From the project split, subversion was able to track the history across the subversion moves between projects, but not git.
>> 
>> Four questions:
>> 1. Is there anything we can do to minimize the history loss in git?
>> 2. Are we going to be able to preserve our sha's or are they going to change again?
>> 3. What changes do we need to make to the subversion notification file?
>> 4. Are there any other changes that need to be coordinated?
>> 
>> After considering it this morning, I believe that the least disruptive move is to leave common at the same url and merge hdfs and mapreduce back in:
>> 
>> $prefix/common/trunk/* -> $prefix/common/trunk/common/*
>> $prefix/hdfs/trunk -> $prefix/common/trunk/hdfs
>> $prefix/mapreduce/trunk -> $prefix/common/trunk/mapreduce
>> 
>> This will preserve the hashes and history for common (and the 20 branches). We'll still need to play git voodoo to get git history for hdfs and mapreduce, but it is far better than starting a brand new git clone.
>> 
>> -- Owen
>> 
>> 
> 


From general-return-3291-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Apr 29 06:25:33 2011
Return-Path: <general-return-3291-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D7E2A215B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 29 Apr 2011 06:25:33 +0000 (UTC)
Received: (qmail 82797 invoked by uid 500); 29 Apr 2011 06:25:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82584 invoked by uid 500); 29 Apr 2011 06:25:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82569 invoked by uid 99); 29 Apr 2011 06:25:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Apr 2011 06:25:31 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Apr 2011 06:25:23 +0000
Received: by bwz8 with SMTP id 8so4897517bwz.35
        for <general@hadoop.apache.org>; Thu, 28 Apr 2011 23:25:03 -0700 (PDT)
Received: by 10.204.39.68 with SMTP id f4mr416716bke.17.1304058303184; Thu, 28
 Apr 2011 23:25:03 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.205.82.70 with HTTP; Thu, 28 Apr 2011 23:24:43 -0700 (PDT)
In-Reply-To: <83FF361E-EA96-4E82-9084-19D91BCEB608@mac.com>
References: <C955EB7A.2E753%sureshms@yahoo-inc.com> <E2932D45-FF26-4579-A975-A98D2578622A@mac.com>
 <B98835A6-F4B7-426E-B22F-B2BEBE273C47@mac.com> <4112B27D-7DCF-4C8E-AD61-6884BA7E738C@mac.com>
 <844F39DC-0A5D-4AAE-AF8D-EFD4CC83C18A@mac.com> <BANLkTin+SyXE0tnfwhhH9P=wR0b7wCOHaw@mail.gmail.com>
 <BANLkTi=+cQBhsNjwdzB+fhcDkePJ=gSOqQ@mail.gmail.com> <9B529359-B589-4229-BA96-188C762A8E31@apache.org>
 <522E7C8B-C159-4520-A758-6E994195D0DC@mac.com> <83FF361E-EA96-4E82-9084-19D91BCEB608@mac.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Thu, 28 Apr 2011 23:24:43 -0700
Message-ID: <BANLkTimLoMzdA2BCZtxSNbsvXqq39XiVtw@mail.gmail.com>
Subject: Re: HADOOP-7106: Re-organize hadoop subversion layout
To: Nigel Daley <ndaley@mac.com>
Cc: general@hadoop.apache.org, "Owen O'Malley" <oom@yahoo-inc.com>, 
	Suresh Srinivas <sureshms@yahoo-inc.com>, Ian Holsman <ian@holsman.net>, infrastructure-dev@apache.org
Content-Type: multipart/alternative; boundary=bcaec5555712ed806104a208bb3a
X-Virus-Checked: Checked by ClamAV on apache.org

--bcaec5555712ed806104a208bb3a
Content-Type: text/plain; charset=ISO-8859-1

On Thu, Apr 28, 2011 at 10:06 PM, Nigel Daley <ndaley@mac.com> wrote:

> As announced last week, I'm planning to do this at 2pm PDT tomorrow
> (Friday) April 29.
>
> Suresh, when do you plan to commit HFS-1052?  That should be done first.
>
> Owen or Todd, did you want to follow Paul's advice:
> > If you're really wanting to make sure to keep the history in Git
> > intact my suggestion would be to setup a temporary svn server locally
> > and test our mirroring scripts against the commands you intend to run.
> If so, how much more time do you need?
>

Wasn't sure how to go about doing that. I guess we need to talk to infra
about it? Do you know how we might clone the SVN repos themselves to test
with?

-Todd

On Apr 20, 2011, at 9:42 PM, Nigel Daley wrote:
>
> > Owen, I'll admit I'm not familiar with all the git details/issues in your
> proposal, but I think the layout change you propose is fine and seems to
> solve the git issues with very minimal impact on the layout.
> >
> > Let's shoot for doing this next Friday, April 29 at 2pm PDT.  I'll update
> the patch and send out a reminder about this later next week.
> >
> > Thanks,
> > Nige
> >
> > On Apr 20, 2011, at 8:00 AM, Owen O'Malley wrote:
> >
> >>
> >> On Apr 19, 2011, at 10:58 PM, Todd Lipcon wrote:
> >>
> >>> On Tue, Apr 19, 2011 at 10:20 PM, Todd Lipcon <todd@cloudera.com>
> wrote:
> >>>
> >>>>
> >>>> I'm currently looking into how the git mirrors are setup in
> Apache-land.
> >>
> >> Uh, why isn't infra-dev on this thread?
> >>
> >> For those on infra-dev, the context is that Nigel is trying to merge
> together the source trees of the Hadoop sub-projects that were split apart 2
> years ago. So he is taking:
> >>
> >> prefix = http://svn.apache.org/repos/asf/hadoop/
> >>
> >> $prefix/common/trunk -> $prefix/trunk/common
> >> $prefix/hdfs/trunk -> $prefix/trunk/hdfs
> >> $prefix/mapreduce/trunk -> $prefix/trunk/mapreduce
> >>
> >> and play similar games with the rest of the branches and tags. For more
> details look at HADOOP-7106.
> >>
> >> From the project split, subversion was able to track the history across
> the subversion moves between projects, but not git.
> >>
> >> Four questions:
> >> 1. Is there anything we can do to minimize the history loss in git?
> >> 2. Are we going to be able to preserve our sha's or are they going to
> change again?
> >> 3. What changes do we need to make to the subversion notification file?
> >> 4. Are there any other changes that need to be coordinated?
> >>
> >> After considering it this morning, I believe that the least disruptive
> move is to leave common at the same url and merge hdfs and mapreduce back
> in:
> >>
> >> $prefix/common/trunk/* -> $prefix/common/trunk/common/*
> >> $prefix/hdfs/trunk -> $prefix/common/trunk/hdfs
> >> $prefix/mapreduce/trunk -> $prefix/common/trunk/mapreduce
> >>
> >> This will preserve the hashes and history for common (and the 20
> branches). We'll still need to play git voodoo to get git history for hdfs
> and mapreduce, but it is far better than starting a brand new git clone.
> >>
> >> -- Owen
> >>
> >>
> >
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

--bcaec5555712ed806104a208bb3a--

From general-return-3292-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Apr 29 12:03:14 2011
Return-Path: <general-return-3292-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 400312F2C
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 29 Apr 2011 12:03:14 +0000 (UTC)
Received: (qmail 2947 invoked by uid 500); 29 Apr 2011 12:03:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2839 invoked by uid 500); 29 Apr 2011 12:03:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2824 invoked by uid 99); 29 Apr 2011 12:03:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Apr 2011 12:03:05 +0000
X-ASF-Spam-Status: No, hits=3.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eltonsky9404@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Apr 2011 12:02:57 +0000
Received: by wyb40 with SMTP id 40so4449234wyb.35
        for <multiple recipients>; Fri, 29 Apr 2011 05:02:37 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=h7/uyrCEKhd6TLMmphLKlg5GwDAU+hhwcZV9oVLKhx0=;
        b=t85Nc+xevW48ef6J/ZzHQ3I4+GoRLHXi58BPrL6iWF8tonlt3/dKLwiwKGiY5eW/Dh
         +OR0mURMCIKIFvE14am8M7D86rsmp3jpEi7T3+Q0Me6gkcXZRICYOjmnQV4LL6x+weHv
         5A+4og0Q9Twy9DfJT4sHsc6eOLkuR89hnpdMw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=ccTBoTIhyPREuQNNWMdpn5ASDmCNaGVlKWo/5s6BqOYH01VB7Sr34uE8kcnTdEiowM
         uqCIbrYe83hyUZy0l27sNZJbzwLkYud0gf0+9c8cUaAHbKMeiW9hDLMQqeqhinYgeiJv
         oj1G2QUKbUgdLJxJVx7pQ+XwXKkDYkKW/IQZQ=
MIME-Version: 1.0
Received: by 10.227.100.219 with SMTP id z27mr782271wbn.45.1304078557698; Fri,
 29 Apr 2011 05:02:37 -0700 (PDT)
Received: by 10.227.42.75 with HTTP; Fri, 29 Apr 2011 05:02:37 -0700 (PDT)
Date: Fri, 29 Apr 2011 22:02:37 +1000
Message-ID: <BANLkTik5e-NbZeOi8+WmorYcb1SP-YACEQ@mail.gmail.com>
Subject: Applications creates bigger output than input?
From: elton sky <eltonsky9404@gmail.com>
To: common-user <common-user@hadoop.apache.org>, general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd308ac30d96c04a20d73a5
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd308ac30d96c04a20d73a5
Content-Type: text/plain; charset=ISO-8859-1

One of assumptions map reduce made, I think, is that size of map's output is
smaller than input. Although we can see many applications have the same size
of output with input, like, sort, merge,etc.
For my benchmark purpose, I am looking for some non-trivial, real life
applications which creates *bigger* output than its input. Trivial example I
can think about is cross join...

I really appreciate if you share your knowledge with me.

--000e0cd308ac30d96c04a20d73a5--

From general-return-3293-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Apr 29 13:13:07 2011
Return-Path: <general-return-3293-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 164ADA23
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 29 Apr 2011 13:13:07 +0000 (UTC)
Received: (qmail 10215 invoked by uid 500); 29 Apr 2011 13:13:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 9980 invoked by uid 500); 29 Apr 2011 13:13:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 9965 invoked by uid 99); 29 Apr 2011 13:13:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Apr 2011 13:13:02 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of john.meagher@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Apr 2011 13:12:54 +0000
Received: by fxm7 with SMTP id 7so4327672fxm.35
        for <multiple recipients>; Fri, 29 Apr 2011 06:12:34 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:cc:content-type;
        bh=Yi8KgAjJt9uCMrhw6rcacPHzYg1Kcs8wJJG0ROUqc7A=;
        b=UfUSNgJD3tPY7ag9jdj0wVz2fYGFEfXWethfCTewxb2tYTcNyajEciJvEUJzD2c526
         wke2RjVpTa5ugaYgfq5nZSP5j6F+Uz8tDXN7rDhT3Ut0EmqSgX6joHDxaBVeriGP5kpx
         Msnil7vY0+oS0iFuLupylwqXFeqcKNWCIpLKQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=jXDxMLFoWIZ4sPn1QFW1lVqW80ITJs3sIUVYWd17lvESp+rw+60X/EVaRpxrrT6jDQ
         bmjYdTyIJpdhUgpQWnR3D/iry1KI4e7f6EU9fON4HWY/jStxzL6kGLQeAUBaaqmYBJkb
         AAdT6uQjQnKN0Tu3eALJte9/E1/xR6K5nGaBg=
MIME-Version: 1.0
Received: by 10.223.6.11 with SMTP id 11mr1573416fax.101.1304082754273; Fri,
 29 Apr 2011 06:12:34 -0700 (PDT)
Received: by 10.223.122.11 with HTTP; Fri, 29 Apr 2011 06:12:33 -0700 (PDT)
In-Reply-To: <BANLkTik5e-NbZeOi8+WmorYcb1SP-YACEQ@mail.gmail.com>
References: <BANLkTik5e-NbZeOi8+WmorYcb1SP-YACEQ@mail.gmail.com>
Date: Fri, 29 Apr 2011 09:12:33 -0400
Message-ID: <BANLkTikSHSLAdn+34dGE3BTnuscc9g9EPA@mail.gmail.com>
Subject: Re: Applications creates bigger output than input?
From: John Meagher <john.meagher@gmail.com>
To: common-user@hadoop.apache.org
Cc: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Another case is augmenting data.  This is sometimes done outside of MR
in an ETL flow, but can be done as an MR job.  Doing something like
this is using Hadoop to handle the scaling issues, but really isn't
what MR is intended for.

A real example of this is:

* Input: standard apache weblog
* Data added...
  - Geolocation of IP
  - Decoding URL
  - Adding information based on visited URL / Ref URL ...
  - Adding information based on the user
* Output complex binary object to a sequence file


On Fri, Apr 29, 2011 at 08:02, elton sky <eltonsky9404@gmail.com> wrote:
> One of assumptions map reduce made, I think, is that size of map's output is
> smaller than input. Although we can see many applications have the same size
> of output with input, like, sort, merge,etc.
> For my benchmark purpose, I am looking for some non-trivial, real life
> applications which creates *bigger* output than its input. Trivial example I
> can think about is cross join...
>
> I really appreciate if you share your knowledge with me.
>

From general-return-3294-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Apr 29 14:52:01 2011
Return-Path: <general-return-3294-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 30DF7205B
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 29 Apr 2011 14:52:01 +0000 (UTC)
Received: (qmail 56112 invoked by uid 500); 29 Apr 2011 14:51:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56031 invoked by uid 500); 29 Apr 2011 14:51:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56015 invoked by uid 99); 29 Apr 2011 14:51:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Apr 2011 14:51:56 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [167.206.4.197] (HELO mta2.srv.hcvlny.cv.net) (167.206.4.197)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Apr 2011 14:51:46 +0000
Received: from wm2.srv.hcvlny.cv.net (wm2.srv.hcvlny.cv.net [167.206.10.8])
 by mta2.srv.hcvlny.cv.net
 (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007))
 with ESMTPA id <0LKF009HC59N7E10@mta2.srv.hcvlny.cv.net>; Fri,
 29 Apr 2011 10:51:24 -0400 (EDT)
Received: from [38.122.12.254]	by webtop.webmail.optimum.net with HTTP; Fri,
 29 Apr 2011 10:51:23 -0400
Date: Fri, 29 Apr 2011 10:51:23 -0400 (EDT)
From: MATTHEW DOERING <doering@optonline.net>
Subject: RE: Applications creates bigger output than input?
X-Originating-IP: [38.122.12.254]
To: common-user@hadoop.apache.org
Cc: common-user@hadoop.apache.org, general@hadoop.apache.org
Message-id:
 <26107912.1798433.1304088684090.JavaMail.doering@mail.srv.smfrct.cv.net>
MIME-version: 1.0
Content-type: text/plain; charset=UTF-8; format=flowed; delsp=no
Content-transfer-encoding: 7BIT
Content-disposition: inline
X-SID: 8
X-Authuserid: doering@optonline.net
User-Agent: Laszlo Mail 3
X-Virus-Checked: Checked by ClamAV on apache.org

If you are looking for exponential output volumes then run a  market 
basket test.  You will need only a few thousand baskets with 100 items 
or so.  Set min support very low (5%) and confidence at 80% and you 
should get a flood of data.  Any FIS algorythm should do the job.


On Fri, Apr 29, 2011 at 8:02 AM, elton sky wrote:

> One of assumptions map reduce made, I think, is that size of map's 
> output is
> smaller than input. Although we can see many applications have the 
> same size
> of output with input, like, sort, merge,etc.
> For my benchmark purpose, I am looking for some non-trivial, real life
> applications which creates *bigger* output than its input. Trivial 
> example I
> can think about is cross join...
>
> I really appreciate if you share your knowledge with me.

From general-return-3295-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Apr 29 18:19:19 2011
Return-Path: <general-return-3295-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7758E27AC
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 29 Apr 2011 18:19:19 +0000 (UTC)
Received: (qmail 23910 invoked by uid 500); 29 Apr 2011 18:19:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23862 invoked by uid 500); 29 Apr 2011 18:19:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23854 invoked by uid 99); 29 Apr 2011 18:19:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Apr 2011 18:19:17 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 69.147.107.20 is neither permitted nor denied by domain of sureshms@yahoo-inc.com)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Apr 2011 18:19:11 +0000
Received: from sp1-ex07cas03.ds.corp.yahoo.com (sp1-ex07cas03.ds.corp.yahoo.com [216.252.116.151])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p3TIHtYT040814;
	Fri, 29 Apr 2011 11:17:55 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1304101075;
	bh=0cfjSL1g/171SczPzUFh51AYPTOmfgWejQdr+Pjqwys=;
	h=From:To:CC:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:MIME-Version;
	b=mK0YqQFtu9DOVULh7J2BxAFjNihoNkg4+7zq/XpHBOJ5qSQuGHSxnn7ZOOtpW2Wz4
	 an4Ah60xlyvgQDGWybFv8SBDsmFF8jZBs2d2PeFEtiugeUvckT5g/ZIxf9KeEN2fCV
	 x7+Y1lOclufzZgUxnSe/nUXQgREGE0qO1Ai+x3/s=
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas03.ds.corp.yahoo.com ([216.252.116.151]) with mapi; Fri, 29 Apr
 2011 11:17:54 -0700
From: Suresh Srinivas <sureshms@yahoo-inc.com>
To: Nigel Daley <ndaley@mac.com>,
        "general@hadoop.apache.org"
	<general@hadoop.apache.org>,
        "Owen O'Malley" <oom@yahoo-inc.com>, Todd Lipcon
	<todd@cloudera.com>
CC: Ian Holsman <ian@holsman.net>,
        "infrastructure-dev@apache.org"
	<infrastructure-dev@apache.org>
Date: Fri, 29 Apr 2011 11:17:53 -0700
Subject: Re: HADOOP-7106: Re-organize hadoop subversion layout
Thread-Topic: HADOOP-7106: Re-organize hadoop subversion layout
Thread-Index: AcwGK2XMVLqB9tLiRfSZFSVfV0bQcgAbluGe
Message-ID: <C9E04EE1.19E2B%sureshms@yahoo-inc.com>
In-Reply-To: <83FF361E-EA96-4E82-9084-19D91BCEB608@mac.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-Entourage/13.8.0.101117
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Nigel,

I have committed federation merge into trunk. Thank you for waiting for it
to be done.

Regards,
Suresh


On 4/28/11 10:06 PM, "Nigel Daley" <ndaley@mac.com> wrote:

> As announced last week, I'm planning to do this at 2pm PDT tomorrow (Frid=
ay)
> April 29. =20
>=20
> Suresh, when do you plan to commit HFS-1052?  That should be done first.
>=20
> Owen or Todd, did you want to follow Paul's advice:
>> If you're really wanting to make sure to keep the history in Git
>> intact my suggestion would be to setup a temporary svn server locally
>> and test our mirroring scripts against the commands you intend to run.
> If so, how much more time do you need?
>=20
> Cheers,
> Nige
>=20
> On Apr 20, 2011, at 9:42 PM, Nigel Daley wrote:
>=20
>> Owen, I'll admit I'm not familiar with all the git details/issues in you=
r
>> proposal, but I think the layout change you propose is fine and seems to
>> solve the git issues with very minimal impact on the layout.
>>=20
>> Let's shoot for doing this next Friday, April 29 at 2pm PDT.  I'll updat=
e the
>> patch and send out a reminder about this later next week.
>>=20
>> Thanks,
>> Nige
>>=20
>> On Apr 20, 2011, at 8:00 AM, Owen O'Malley wrote:
>>=20
>>>=20
>>> On Apr 19, 2011, at 10:58 PM, Todd Lipcon wrote:
>>>=20
>>>> On Tue, Apr 19, 2011 at 10:20 PM, Todd Lipcon <todd@cloudera.com> wrot=
e:
>>>>=20
>>>>>=20
>>>>> I'm currently looking into how the git mirrors are setup in Apache-la=
nd.
>>>=20
>>> Uh, why isn't infra-dev on this thread?
>>>=20
>>> For those on infra-dev, the context is that Nigel is trying to merge
>>> together the source trees of the Hadoop sub-projects that were split ap=
art 2
>>> years ago. So he is taking:
>>>=20
>>> prefix =3D http://svn.apache.org/repos/asf/hadoop/
>>>=20
>>> $prefix/common/trunk -> $prefix/trunk/common
>>> $prefix/hdfs/trunk -> $prefix/trunk/hdfs
>>> $prefix/mapreduce/trunk -> $prefix/trunk/mapreduce
>>>=20
>>> and play similar games with the rest of the branches and tags. For more
>>> details look at HADOOP-7106.
>>>=20
>>> From the project split, subversion was able to track the history across=
 the
>>> subversion moves between projects, but not git.
>>>=20
>>> Four questions:
>>> 1. Is there anything we can do to minimize the history loss in git?
>>> 2. Are we going to be able to preserve our sha's or are they going to c=
hange
>>> again?
>>> 3. What changes do we need to make to the subversion notification file?
>>> 4. Are there any other changes that need to be coordinated?
>>>=20
>>> After considering it this morning, I believe that the least disruptive =
move
>>> is to leave common at the same url and merge hdfs and mapreduce back in=
:
>>>=20
>>> $prefix/common/trunk/* -> $prefix/common/trunk/common/*
>>> $prefix/hdfs/trunk -> $prefix/common/trunk/hdfs
>>> $prefix/mapreduce/trunk -> $prefix/common/trunk/mapreduce
>>>=20
>>> This will preserve the hashes and history for common (and the 20 branch=
es).
>>> We'll still need to play git voodoo to get git history for hdfs and
>>> mapreduce, but it is far better than starting a brand new git clone.
>>>=20
>>> -- Owen
>>>=20
>>>=20
>>=20
>=20


From general-return-3296-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Apr 29 18:37:31 2011
Return-Path: <general-return-3296-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DE2E72519
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 29 Apr 2011 18:37:30 +0000 (UTC)
Received: (qmail 41279 invoked by uid 500); 29 Apr 2011 18:37:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 41230 invoked by uid 500); 29 Apr 2011 18:37:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 41221 invoked by uid 99); 29 Apr 2011 18:37:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Apr 2011 18:37:29 +0000
X-ASF-Spam-Status: No, hits=2.9 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Apr 2011 18:37:24 +0000
Received: from battlerock-lm.corp.yahoo.com (battlerock-lm.corp.yahoo.com [10.72.123.81])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p3TIaPKd076807;
	Fri, 29 Apr 2011 11:36:25 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1304102185;
	bh=t/Kfd5j7vEoOl8GINQ4jeHepURS0EnO9Wjv8SBG8D8g=;
	h=Subject:Mime-Version:Content-Type:From:In-Reply-To:Date:Cc:
	 Message-Id:References:To;
	b=IpruIabfQ/FQrLihnfEmlzdXLM74bQipaah4UNRVuCXELr4x7CuIF6jAdhxcy+xnb
	 p1hqxd8SKky5il6R+ro2WNPM1cinDSQfc6PdTKh59/8pM+zYZu1FdwpDF7uU29IFf/
	 H4S5tgXVbJ7Hf9MN5kzcfClAZHLyYC/LZumuzzxE=
Subject: Re: HADOOP-7106: Re-organize hadoop subversion layout
Mime-Version: 1.0 (Apple Message framework v1084)
Content-Type: multipart/alternative; boundary=Apple-Mail-1--622529222
From: "Owen O'Malley" <oom@yahoo-inc.com>
In-Reply-To: <BANLkTimLoMzdA2BCZtxSNbsvXqq39XiVtw@mail.gmail.com>
Date: Fri, 29 Apr 2011 11:36:25 -0700
Cc: Nigel Daley <ndaley@mac.com>,
        "general@hadoop.apache.org" <general@hadoop.apache.org>,
        Suresh Srinivas <sureshms@yahoo-inc.com>,
        Ian Holsman <ian@holsman.net>,
        "infrastructure-dev@apache.org" <infrastructure-dev@apache.org>
Message-Id: <AF47F47F-4712-43BE-ACCA-B6B1F00FC697@yahoo-inc.com>
References: <C955EB7A.2E753%sureshms@yahoo-inc.com> <E2932D45-FF26-4579-A975-A98D2578622A@mac.com> <B98835A6-F4B7-426E-B22F-B2BEBE273C47@mac.com> <4112B27D-7DCF-4C8E-AD61-6884BA7E738C@mac.com> <844F39DC-0A5D-4AAE-AF8D-EFD4CC83C18A@mac.com> <BANLkTin+SyXE0tnfwhhH9P=wR0b7wCOHaw@mail.gmail.com> <BANLkTi=+cQBhsNjwdzB+fhcDkePJ=gSOqQ@mail.gmail.com> <9B529359-B589-4229-BA96-188C762A8E31@apache.org> <522E7C8B-C159-4520-A758-6E994195D0DC@mac.com> <83FF361E-EA96-4E82-9084-19D91BCEB608@mac.com> <BANLkTimLoMzdA2BCZtxSNbsvXqq39XiVtw@mail.gmail.com>
To: Todd Lipcon <todd@cloudera.com>
X-Mailer: Apple Mail (2.1084)

--Apple-Mail-1--622529222
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii


On Apr 28, 2011, at 11:24 PM, Todd Lipcon wrote:

> Wasn't sure how to go about doing that. I guess we need to talk to =
infra about it? Do you know how we might clone the SVN repos themselves =
to test with?

It looks like there are svn dumps at http://svn-master.apache.org/dump/ =
from 2 april 2011. You should be able to use those to setup a local =
subversion.

-- Owen


--Apple-Mail-1--622529222--

From general-return-3297-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Fri Apr 29 20:18:16 2011
Return-Path: <general-return-3297-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C002B2EFE
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Fri, 29 Apr 2011 20:18:16 +0000 (UTC)
Received: (qmail 49751 invoked by uid 500); 29 Apr 2011 20:18:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49697 invoked by uid 500); 29 Apr 2011 20:18:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49689 invoked by uid 99); 29 Apr 2011 20:18:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Apr 2011 20:18:15 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.102 as permitted sender)
Received: from [17.148.16.102] (HELO asmtpout027.mac.com) (17.148.16.102)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Apr 2011 20:18:06 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from lt-a3-110073.jiveland.com ([67.136.90.250])
 by asmtp027.mac.com (Oracle Communications Messaging Exchange Server 7u4-18.01
 64bit (built Jul 15 2010)) with ESMTPSA id <0LKF00M1DKC1SX30@asmtp027.mac.com>
 for general@hadoop.apache.org; Fri, 29 Apr 2011 13:16:51 -0700 (PDT)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-04-29_07:2011-04-29,2011-04-29,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1104290139
Subject: Re: HADOOP-7106: Re-organize hadoop subversion layout
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <AF47F47F-4712-43BE-ACCA-B6B1F00FC697@yahoo-inc.com>
Date: Fri, 29 Apr 2011 13:16:49 -0700
Cc: Ian Holsman <ian@holsman.net>, infrastructure-dev@apache.org
Message-id: <4EB044AB-A4F8-4530-AB68-0A675D96BCF8@mac.com>
References: <C955EB7A.2E753%sureshms@yahoo-inc.com>
 <E2932D45-FF26-4579-A975-A98D2578622A@mac.com>
 <B98835A6-F4B7-426E-B22F-B2BEBE273C47@mac.com>
 <4112B27D-7DCF-4C8E-AD61-6884BA7E738C@mac.com>
 <844F39DC-0A5D-4AAE-AF8D-EFD4CC83C18A@mac.com>
 <BANLkTin+SyXE0tnfwhhH9P=wR0b7wCOHaw@mail.gmail.com>
 <BANLkTi=+cQBhsNjwdzB+fhcDkePJ=gSOqQ@mail.gmail.com>
 <9B529359-B589-4229-BA96-188C762A8E31@apache.org>
 <522E7C8B-C159-4520-A758-6E994195D0DC@mac.com>
 <83FF361E-EA96-4E82-9084-19D91BCEB608@mac.com>
 <BANLkTimLoMzdA2BCZtxSNbsvXqq39XiVtw@mail.gmail.com>
 <AF47F47F-4712-43BE-ACCA-B6B1F00FC697@yahoo-inc.com>
To: Todd Lipcon <todd@cloudera.com>, general@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

I can't do this at 2pm now.  Todd, I suspect you want more time to try out the svn/git test anyways. 

Let's shoot for next Wednesday at 2pm.  Ian should be back by then too.  Any objections?

Cheers,
Nige

On Apr 29, 2011, at 11:36 AM, Owen O'Malley wrote:

> 
> On Apr 28, 2011, at 11:24 PM, Todd Lipcon wrote:
> 
>> Wasn't sure how to go about doing that. I guess we need to talk to infra about it? Do you know how we might clone the SVN repos themselves to test with?
> 
> It looks like there are svn dumps at http://svn-master.apache.org/dump/ from 2 april 2011. You should be able to use those to setup a local subversion.
> 
> -- Owen
> 


From general-return-3298-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat Apr 30 07:19:18 2011
Return-Path: <general-return-3298-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 45E6737EA
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat, 30 Apr 2011 07:19:18 +0000 (UTC)
Received: (qmail 88433 invoked by uid 500); 30 Apr 2011 07:19:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87930 invoked by uid 500); 30 Apr 2011 07:19:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87913 invoked by uid 99); 30 Apr 2011 07:19:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 30 Apr 2011 07:19:02 +0000
X-ASF-Spam-Status: No, hits=3.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of eltonsky9404@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 30 Apr 2011 07:18:55 +0000
Received: by wyb40 with SMTP id 40so5238346wyb.35
        for <multiple recipients>; Sat, 30 Apr 2011 00:18:34 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=gtaDLoCD5IGUFLtKLhXAM/H1FJH2zLHQaLMtKr/iQuI=;
        b=X74LpE9EYjK4TjQtS/IrOVPvIQB/MtrRYjVtq1gcL96gZN2xc2ojQvZDwPe7kvXX28
         yZNjCEawlST5EtDnRi5N93GDu7xsJ9T8Rfwh8Q42bqGcYEhpwDuBoMWpQuTtDWF7jutB
         GnsYWCe1ajVVO6JDoMXpBe64ueIBThOWZgF9Y=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=ZWd2vr5ezHFlHe7NLq9gUZNwBqpP3yUNMQ2bwYYKZDZHz7yox67JYqtTL18g54Zt/Q
         6OkZ94rxJ0TgeD84rCshqpXpIw8G22iRiIXoIpZdxAzT9Seg5lRuF3qx/Xuhuy5GYJjm
         09BZUdZbPJ9sbUgLeAivbjWNH5tsFvBIImSZk=
MIME-Version: 1.0
Received: by 10.227.100.219 with SMTP id z27mr1764308wbn.45.1304147914066;
 Sat, 30 Apr 2011 00:18:34 -0700 (PDT)
Received: by 10.227.42.75 with HTTP; Sat, 30 Apr 2011 00:18:34 -0700 (PDT)
Date: Sat, 30 Apr 2011 17:18:34 +1000
Message-ID: <BANLkTinbKUikZGx_c9h6oN=_yveF8j2K6Q@mail.gmail.com>
Subject: questions about hadoop map reduce and compute intensive related applications
From: elton sky <eltonsky9404@gmail.com>
To: common-user <common-user@hadoop.apache.org>, general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd308ac270b9104a21d9917

--000e0cd308ac270b9104a21d9917
Content-Type: text/plain; charset=ISO-8859-1

I got 2 questions:

1. I am wondering how hadoop MR performs when it runs compute intensive
applications, e.g. Monte carlo method compute PI. There's a example in 0.21,
QuasiMonteCarlo, but that example doesn't use random number and it generates
psudo input upfront. If we use distributed random number generation, then I
guess the performance of hadoop should be similar with some message passing
framework, like MPI. So my guess is by using proper method hadoop would be
good in compute intensive applications compared with MPI.

2. I am looking for some applications, which has large data sets and
requires intensive computation. An application can be divided into a
workflow, including either map reduce operations, and message passing like
operations. For example, in step 1 I use hadoop MR processes 10TB of data
and generates small output, say, 10GB. This 10GB can be fit into memory and
they are better be processed with some interprocess communication, which
will boost the performance. So in step 2 I will use MPI, etc.

Is there any application has this property, perhaps in some scientific
research area? Or it's just alright to use map reduce itself?

Regards,
Elton

--000e0cd308ac270b9104a21d9917--

From general-return-3299-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org  Sat Apr 30 20:31:48 2011
Return-Path: <general-return-3299-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5A5C5312E
	for <apmail-hadoop-general-archive@minotaur.apache.org>; Sat, 30 Apr 2011 20:31:48 +0000 (UTC)
Received: (qmail 9903 invoked by uid 500); 30 Apr 2011 20:31:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 9377 invoked by uid 500); 30 Apr 2011 20:31:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8907 invoked by uid 99); 30 Apr 2011 20:31:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 30 Apr 2011 20:31:43 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.220.176] (HELO mail-vx0-f176.google.com) (209.85.220.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 30 Apr 2011 20:31:33 +0000
Received: by vxa37 with SMTP id 37so5546249vxa.35
        for <multiple recipients>; Sat, 30 Apr 2011 13:31:12 -0700 (PDT)
Received: by 10.52.174.243 with SMTP id bv19mr2303953vdc.93.1304195472146;
 Sat, 30 Apr 2011 13:31:12 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.52.160.194 with HTTP; Sat, 30 Apr 2011 13:30:52 -0700 (PDT)
X-Originating-IP: [67.160.196.149]
In-Reply-To: <BANLkTinbKUikZGx_c9h6oN=_yveF8j2K6Q@mail.gmail.com>
References: <BANLkTinbKUikZGx_c9h6oN=_yveF8j2K6Q@mail.gmail.com>
From: Ted Dunning <tdunning@maprtech.com>
Date: Sat, 30 Apr 2011 13:30:52 -0700
Message-ID: <BANLkTi=2EaAmxaibRPiX=AEkVeL8XKmdvA@mail.gmail.com>
Subject: Re: questions about hadoop map reduce and compute intensive related applications
To: common-user@hadoop.apache.org
Cc: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=bcaec51a8e0cd5c0a504a228ab30
X-Virus-Checked: Checked by ClamAV on apache.org

--bcaec51a8e0cd5c0a504a228ab30
Content-Type: text/plain; charset=ISO-8859-1

On Sat, Apr 30, 2011 at 12:18 AM, elton sky <eltonsky9404@gmail.com> wrote:

> I got 2 questions:
>
> 1. I am wondering how hadoop MR performs when it runs compute intensive
> applications, e.g. Monte carlo method compute PI. There's a example in
> 0.21,
> QuasiMonteCarlo, but that example doesn't use random number and it
> generates
> psudo input upfront. If we use distributed random number generation, then I
> guess the performance of hadoop should be similar with some message passing
> framework, like MPI. So my guess is by using proper method hadoop would be
> good in compute intensive applications compared with MPI.
>

Not quite sure what algorithms you mean here, but for trivial parallelism,
map-reduce is a fine way to go.

MPI supports node-to-node communications in ways that map-reduce does not,
however, which requires that you iterate map-reduce steps for many
algorithms.   With Hadoop's current implementation, this is horrendously
slow (minimum 20-30 seconds per iteration).

Sometimes you can avoid this by clever tricks.  For instance, random
projection can compute the key step in an SVD decomposition with one
map-reduce while the comparable Lanczos algorithm requires more than one
step per eigenvector (and we often want 100 of them!).

Sometimes, however, there are no known algorithms that avoid the need for
repeated communication.  For these problems, Hadoop as it stands may be a
poor fit.  Help is on the way, however, with the MapReduce 2.0 work because
that will allow much more flexible models of computation.


> 2. I am looking for some applications, which has large data sets and
> requires intensive computation. An application can be divided into a
> workflow, including either map reduce operations, and message passing like
> operations. For example, in step 1 I use hadoop MR processes 10TB of data
> and generates small output, say, 10GB. This 10GB can be fit into memory and
> they are better be processed with some interprocess communication, which
> will boost the performance. So in step 2 I will use MPI, etc.
>

Some machine learning algorithms require features that are much smaller than
the original input.  This leads to exactly the pattern you describe.
 Integrating MPI with map-reduce is currently difficult and/or very ugly,
however.  Not impossible and there are hackish ways to do the job, but they
are hacks.

--bcaec51a8e0cd5c0a504a228ab30--

From general-return-2311-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 01 17:54:12 2010
Return-Path: <general-return-2311-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 86950 invoked from network); 1 Nov 2010 17:54:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Nov 2010 17:54:12 -0000
Received: (qmail 9481 invoked by uid 500); 1 Nov 2010 17:54:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 9419 invoked by uid 500); 1 Nov 2010 17:54:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 9411 invoked by uid 99); 1 Nov 2010 17:54:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Nov 2010 17:54:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 01 Nov 2010 17:54:38 +0000
Received: (qmail 86916 invoked by uid 99); 1 Nov 2010 17:53:45 -0000
Received: from localhost.apache.org (HELO mail-ww0-f42.google.com) (127.0.0.1)
  (smtp-auth username phunt, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Nov 2010 17:53:45 +0000
Received: by wwb17 with SMTP id 17so172556wwb.5
        for <general@hadoop.apache.org>; Mon, 01 Nov 2010 10:54:14 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.227.155.66 with SMTP id r2mr15655379wbw.116.1288634054818;
 Mon, 01 Nov 2010 10:54:14 -0700 (PDT)
Received: by 10.227.152.139 with HTTP; Mon, 1 Nov 2010 10:54:14 -0700 (PDT)
In-Reply-To: <AANLkTikeJ9TkXHbD7w1a1CmsQ2jdVQrXnX5sX2JaUsUc@mail.gmail.com>
References: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
	<AANLkTikeJ9TkXHbD7w1a1CmsQ2jdVQrXnX5sX2JaUsUc@mail.gmail.com>
Date: Mon, 1 Nov 2010 10:54:14 -0700
Message-ID: <AANLkTim14_vCUTJmn4MwZpCDrUOX_-C-UM3QFALOugEj@mail.gmail.com>
Subject: Re: [VOTE] ZooKeeper as TLP?
From: Patrick Hunt <phunt@apache.org>
To: general@hadoop.apache.org, hadoop@holsman.net
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

The vote passes, with 23 +1 votes and no -1 votes.

Ian, can you please add this resolution to the agenda for the week
board meeting?

Thanks,

Patrick

On Sat, Oct 30, 2010 at 12:40 AM, Hemanth Yamijala <yhemanth@gmail.com> wro=
te:
> +1
>
> On Wed, Oct 27, 2010 at 11:34 PM, Patrick Hunt <phunt@apache.org> wrote:
>> Please vote as to whether you think ZooKeeper should become a
>> top-level Apache project.
>>
>> The ZooKeeper development community has already voted on and approved
>> ( http://bit.ly/9VCoTi ) the following draft board resolution. It
>> lists all current active ZooKeeper committers (sans one committer who
>> has not been active on the project since it's inception
>> http://bit.ly/9wLkVV) as initial members of the project management
>> committee (PMC) and myself, Patrick Hunt, as the initial chair.
>>
>>
>> Do the good people of Hadoop support sending this request on to the
>> Hadoop PMC?
>>
>> Regards,
>>
>> Patrick
>>
>> ------------
>>
>> =A0 =A0X. Establish the Apache ZooKeeper Project
>>
>> =A0 =A0 =A0 WHEREAS, the Board of Directors deems it to be in the best
>> =A0 =A0 =A0 interests of the Foundation and consistent with the
>> =A0 =A0 =A0 Foundation's purpose to establish a Project Management
>> =A0 =A0 =A0 Committee charged with the creation and maintenance of
>> =A0 =A0 =A0 open-source software related to distributed system coordinat=
ion
>> =A0 =A0 =A0 for distribution at no charge to the public.
>>
>> =A0 =A0 =A0 NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>> =A0 =A0 =A0 Committee (PMC), to be known as the "Apache ZooKeeper Projec=
t",
>> =A0 =A0 =A0 be and hereby is established pursuant to Bylaws of the
>> =A0 =A0 =A0 Foundation; and be it further
>>
>> =A0 =A0 =A0 RESOLVED, that the Apache ZooKeeper Project be and hereby is
>> =A0 =A0 =A0 responsible for the creation and maintenance of software
>> =A0 =A0 =A0 related to distributed system coordination; and be it furthe=
r
>>
>> =A0 =A0 =A0 RESOLVED, that the office of "Vice President, Apache ZooKeep=
er" be
>> =A0 =A0 =A0 and hereby is created, the person holding such office to
>> =A0 =A0 =A0 serve at the direction of the Board of Directors as the chai=
r
>> =A0 =A0 =A0 of the Apache ZooKeeper Project, and to have primary respons=
ibility
>> =A0 =A0 =A0 for management of the projects within the scope of
>> =A0 =A0 =A0 responsibility of the Apache ZooKeeper Project; and be it fu=
rther
>>
>> =A0 =A0 =A0 RESOLVED, that the persons listed immediately below be and
>> =A0 =A0 =A0 hereby are appointed to serve as the initial members of the
>> =A0 =A0 =A0 Apache ZooKeeper Project:
>>
>> =A0 =A0 =A0 =A0 * Patrick Hunt =A0 =A0 =A0 =A0 <phunt@apache.org>
>> =A0 =A0 =A0 =A0 * Flavio Junqueira =A0 =A0 <fpj@apache.org>
>> =A0 =A0 =A0 =A0 * Mahadev Konar =A0 =A0 =A0 =A0<mahadev@apache.org>
>> =A0 =A0 =A0 =A0 * Benjamin Reed =A0 =A0 =A0 =A0<breed@apache.org>
>> =A0 =A0 =A0 =A0 * Henry Robinson =A0 =A0 =A0 <henry@apache.org>
>>
>> =A0 =A0 =A0 NOW, THEREFORE, BE IT FURTHER RESOLVED, that Patrick Hunt
>> =A0 =A0 =A0 be appointed to the office of Vice President, Apache ZooKeep=
er, to
>> =A0 =A0 =A0 serve in accordance with and subject to the direction of the
>> =A0 =A0 =A0 Board of Directors and the Bylaws of the Foundation until
>> =A0 =A0 =A0 death, resignation, retirement, removal or disqualification,
>> =A0 =A0 =A0 or until a successor is appointed; and be it further
>>
>> =A0 =A0 =A0 RESOLVED, that the initial Apache ZooKeeper PMC be and hereb=
y is
>> =A0 =A0 =A0 tasked with the creation of a set of bylaws intended to
>> =A0 =A0 =A0 encourage open development and increased participation in th=
e
>> =A0 =A0 =A0 Apache ZooKeeper Project; and be it further
>>
>> =A0 =A0 =A0 RESOLVED, that the Apache ZooKeeper Project be and hereby
>> =A0 =A0 =A0 is tasked with the migration and rationalization of the Apac=
he
>> =A0 =A0 =A0 Hadoop ZooKeeper sub-project; and be it further
>>
>> =A0 =A0 =A0 RESOLVED, that all responsibilities pertaining to the Apache
>> =A0 =A0 =A0 Hadoop ZooKeeper sub-project encumbered upon the
>> =A0 =A0 =A0 Apache Hadoop Project are hereafter discharged.
>>
>

From general-return-2312-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 02 00:40:21 2010
Return-Path: <general-return-2312-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 74953 invoked from network); 2 Nov 2010 00:40:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Nov 2010 00:40:21 -0000
Received: (qmail 3551 invoked by uid 500); 2 Nov 2010 00:40:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3488 invoked by uid 500); 2 Nov 2010 00:40:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3480 invoked by uid 99); 2 Nov 2010 00:40:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Nov 2010 00:40:51 +0000
X-ASF-Spam-Status: No, hits=4.1 required=10.0
	tests=HTML_MESSAGE,MISSING_HEADERS,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Nov 2010 00:40:44 +0000
Received: by wyg36 with SMTP id 36so6515416wyg.35
        for <general@hadoop.apache.org>; Mon, 01 Nov 2010 17:40:23 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.53.4 with SMTP id f4mr377170wec.4.1288658422559; Mon, 01
 Nov 2010 17:40:22 -0700 (PDT)
Received: by 10.216.87.78 with HTTP; Mon, 1 Nov 2010 17:40:22 -0700 (PDT)
In-Reply-To: <AANLkTim14_vCUTJmn4MwZpCDrUOX_-C-UM3QFALOugEj@mail.gmail.com>
References: <AANLkTikTVJ53i8dzLQZ5Cj6x_PfUtzfi+3BLA0DVi8_a@mail.gmail.com>
	<AANLkTikeJ9TkXHbD7w1a1CmsQ2jdVQrXnX5sX2JaUsUc@mail.gmail.com>
	<AANLkTim14_vCUTJmn4MwZpCDrUOX_-C-UM3QFALOugEj@mail.gmail.com>
Date: Tue, 2 Nov 2010 11:40:22 +1100
Message-ID: <AANLkTi=w6QzkmU44KXhKSULCnNr+ydvJ2SUvtX4bem=Z@mail.gmail.com>
Subject: Re: [VOTE] ZooKeeper as TLP?
From: Ian Holsman <hadoop@holsman.net>
Cc: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6da86628374ff0494072bd5
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6da86628374ff0494072bd5
Content-Type: text/plain; charset=ISO-8859-1

sure.. I'll be glad too.

On Tue, Nov 2, 2010 at 4:54 AM, Patrick Hunt <phunt@apache.org> wrote:

> The vote passes, with 23 +1 votes and no -1 votes.
>
> Ian, can you please add this resolution to the agenda for the week
> board meeting?
>
> Thanks,
>
> Patrick
>
> On Sat, Oct 30, 2010 at 12:40 AM, Hemanth Yamijala <yhemanth@gmail.com>
> wrote:
> > +1
> >
> > On Wed, Oct 27, 2010 at 11:34 PM, Patrick Hunt <phunt@apache.org> wrote:
> >> Please vote as to whether you think ZooKeeper should become a
> >> top-level Apache project.
> >>
> >> The ZooKeeper development community has already voted on and approved
> >> ( http://bit.ly/9VCoTi ) the following draft board resolution. It
> >> lists all current active ZooKeeper committers (sans one committer who
> >> has not been active on the project since it's inception
> >> http://bit.ly/9wLkVV) as initial members of the project management
> >> committee (PMC) and myself, Patrick Hunt, as the initial chair.
> >>
> >>
> >> Do the good people of Hadoop support sending this request on to the
> >> Hadoop PMC?
> >>
> >> Regards,
> >>
> >> Patrick
> >>
> >> ------------
> >>
> >>    X. Establish the Apache ZooKeeper Project
> >>
> >>       WHEREAS, the Board of Directors deems it to be in the best
> >>       interests of the Foundation and consistent with the
> >>       Foundation's purpose to establish a Project Management
> >>       Committee charged with the creation and maintenance of
> >>       open-source software related to distributed system coordination
> >>       for distribution at no charge to the public.
> >>
> >>       NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> >>       Committee (PMC), to be known as the "Apache ZooKeeper Project",
> >>       be and hereby is established pursuant to Bylaws of the
> >>       Foundation; and be it further
> >>
> >>       RESOLVED, that the Apache ZooKeeper Project be and hereby is
> >>       responsible for the creation and maintenance of software
> >>       related to distributed system coordination; and be it further
> >>
> >>       RESOLVED, that the office of "Vice President, Apache ZooKeeper" be
> >>       and hereby is created, the person holding such office to
> >>       serve at the direction of the Board of Directors as the chair
> >>       of the Apache ZooKeeper Project, and to have primary
> responsibility
> >>       for management of the projects within the scope of
> >>       responsibility of the Apache ZooKeeper Project; and be it further
> >>
> >>       RESOLVED, that the persons listed immediately below be and
> >>       hereby are appointed to serve as the initial members of the
> >>       Apache ZooKeeper Project:
> >>
> >>         * Patrick Hunt         <phunt@apache.org>
> >>         * Flavio Junqueira     <fpj@apache.org>
> >>         * Mahadev Konar        <mahadev@apache.org>
> >>         * Benjamin Reed        <breed@apache.org>
> >>         * Henry Robinson       <henry@apache.org>
> >>
> >>       NOW, THEREFORE, BE IT FURTHER RESOLVED, that Patrick Hunt
> >>       be appointed to the office of Vice President, Apache ZooKeeper, to
> >>       serve in accordance with and subject to the direction of the
> >>       Board of Directors and the Bylaws of the Foundation until
> >>       death, resignation, retirement, removal or disqualification,
> >>       or until a successor is appointed; and be it further
> >>
> >>       RESOLVED, that the initial Apache ZooKeeper PMC be and hereby is
> >>       tasked with the creation of a set of bylaws intended to
> >>       encourage open development and increased participation in the
> >>       Apache ZooKeeper Project; and be it further
> >>
> >>       RESOLVED, that the Apache ZooKeeper Project be and hereby
> >>       is tasked with the migration and rationalization of the Apache
> >>       Hadoop ZooKeeper sub-project; and be it further
> >>
> >>       RESOLVED, that all responsibilities pertaining to the Apache
> >>       Hadoop ZooKeeper sub-project encumbered upon the
> >>       Apache Hadoop Project are hereafter discharged.
> >>
> >
>

--0016e6da86628374ff0494072bd5--

From general-return-2313-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 02 18:25:51 2010
Return-Path: <general-return-2313-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76207 invoked from network); 2 Nov 2010 18:25:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Nov 2010 18:25:51 -0000
Received: (qmail 65890 invoked by uid 500); 2 Nov 2010 18:26:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65828 invoked by uid 500); 2 Nov 2010 18:26:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65820 invoked by uid 99); 2 Nov 2010 18:26:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Nov 2010 18:26:20 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [64.186.241.227] (HELO sri01.semanticresearch.local) (64.186.241.227)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 02 Nov 2010 18:26:14 +0000
Received: from localhost (sri01.semanticresearch.local [127.0.0.1])
	by sri01.semanticresearch.local (Postfix) with ESMTP id A6DE93060002
	for <general@hadoop.apache.org>; Tue,  2 Nov 2010 11:25:47 -0700 (PDT)
X-Virus-Scanned: amavisd-new at sri01.semanticresearch.local
Received: from sri01.semanticresearch.local ([127.0.0.1])
	by localhost (sri01.semanticresearch.local [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id MMGnd6WrcC8H for <general@hadoop.apache.org>;
	Tue,  2 Nov 2010 11:25:47 -0700 (PDT)
Received: from sri01.semanticresearch.local (sri01.semanticresearch.local [127.0.0.1])
	by sri01.semanticresearch.local (Postfix) with ESMTP id 5B0503060001
	for <general@hadoop.apache.org>; Tue,  2 Nov 2010 11:25:47 -0700 (PDT)
From: "Mark Laffoon" <mlaffoon@semanticresearch.com>
To: <general@hadoop.apache.org>
Subject: web-based file transfer
Date: Tue, 2 Nov 2010 11:25:47 -0700 (PDT)
Message-ID: <033e01cb7abb$623b7d50$26b277f0$@com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_NextPart_000_033F_01CB7A80.B5DCA550"
X-Mailer: Microsoft Office Outlook 12.0
X-Mailer: Zimbra 5.0.13_GA_2791.RHEL5 (ZimbraConnectorForOutlook/5.0.2893.13)
Thread-Index: Act6u2HNx5BSRl6vQtil6xKpsp646g==
Content-Language: en-us
X-Originating-IP: [172.27.16.186]

------=_NextPart_000_033F_01CB7A80.B5DCA550
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit

We want to enable our web-based client (i.e. browser client, java applet,
whatever?) to transfer files into a system backed by hdfs. The obvious
simple solution is to do http file uploads, then copy the file to hdfs. I
was wondering if there is a way to do it with an hdfs-enabled applet where
the server gives the client the necessary hadoop configuration
information, and the client applet pushes the data directly into hdfs. 

 

Has anybody done this or something similar? Can you give me a starting
point (I'm about to go wander through the hadoop CLI code to get ideas).

 

Thanks,

Mark


------=_NextPart_000_033F_01CB7A80.B5DCA550--

From general-return-2314-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 02 18:43:59 2010
Return-Path: <general-return-2314-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 89392 invoked from network); 2 Nov 2010 18:43:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Nov 2010 18:43:58 -0000
Received: (qmail 13838 invoked by uid 500); 2 Nov 2010 18:44:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13681 invoked by uid 500); 2 Nov 2010 18:44:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13664 invoked by uid 99); 2 Nov 2010 18:44:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Nov 2010 18:44:28 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Nov 2010 18:44:22 +0000
Received: by wyg36 with SMTP id 36so7397793wyg.35
        for <general@hadoop.apache.org>; Tue, 02 Nov 2010 11:44:02 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.176.20 with SMTP id a20mr10660038wem.14.1288723441164;
 Tue, 02 Nov 2010 11:44:01 -0700 (PDT)
Received: by 10.216.161.201 with HTTP; Tue, 2 Nov 2010 11:44:01 -0700 (PDT)
In-Reply-To: <033e01cb7abb$623b7d50$26b277f0$@com>
References: <033e01cb7abb$623b7d50$26b277f0$@com>
Date: Tue, 2 Nov 2010 14:44:01 -0400
Message-ID: <AANLkTi=BPo3g3zVD2P3qjW_65sh-O6js8mrGup-E4oPf@mail.gmail.com>
Subject: Re: web-based file transfer
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

I would recommend against clients pushing data directly to hdfs like
this for a few reasons.

1. The HDFS cluster would need to be directly exposed to a public
network; you don't want to do this.
2. You'd be applying (presumably) a high concurrent load to HDFS which
isn't its strong point.

>From an architecture point of view, it's much nicer to have a queuing
system between the upload and ingestion into HDFS that you can
throttle and control, if necessary. This also allows you to isolate
the cluster from the outside world. As to not bottleneck on a single
writer, you can have uploaded files land in a queue and have multiple
competing consumers popping files (or file names upon which to
operate) out of the queue and handling the writing in parallel while
being able to control the number of workers. If the initial upload is
to a shared device like NFS, you can have writers live on multiple
boxes and distribute the work.

Another option is to consider Flume, but only if you can deal with the
fact that it effectively throws away the notion of files and treats
their contents as individual events, etc.
http://github.com/cloudera/flume.

Hope that helps.

On Tue, Nov 2, 2010 at 2:25 PM, Mark Laffoon
<mlaffoon@semanticresearch.com> wrote:
> We want to enable our web-based client (i.e. browser client, java applet,
> whatever?) to transfer files into a system backed by hdfs. The obvious
> simple solution is to do http file uploads, then copy the file to hdfs. I
> was wondering if there is a way to do it with an hdfs-enabled applet where
> the server gives the client the necessary hadoop configuration
> information, and the client applet pushes the data directly into hdfs.
>
>
>
> Has anybody done this or something similar? Can you give me a starting
> point (I'm about to go wander through the hadoop CLI code to get ideas).
>
>
>
> Thanks,
>
> Mark
>
>



-- 
Eric Sammer
twitter: esammer
data: www.cloudera.com

From general-return-2315-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 03 05:23:17 2010
Return-Path: <general-return-2315-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77952 invoked from network); 3 Nov 2010 05:23:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Nov 2010 05:23:16 -0000
Received: (qmail 30597 invoked by uid 500); 3 Nov 2010 05:23:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30164 invoked by uid 500); 3 Nov 2010 05:23:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30155 invoked by uid 99); 3 Nov 2010 05:23:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Nov 2010 05:23:42 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Nov 2010 05:23:37 +0000
Received: by iwn7 with SMTP id 7so288820iwn.35
        for <general@hadoop.apache.org>; Tue, 02 Nov 2010 22:23:16 -0700 (PDT)
Received: by 10.42.22.140 with SMTP id o12mr5152041icb.466.1288761795297;
        Tue, 02 Nov 2010 22:23:15 -0700 (PDT)
Received: from [110.22.134.89] ([110.22.134.89])
        by mx.google.com with ESMTPS id fw4sm355270ibb.13.2010.11.02.22.23.10
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Tue, 02 Nov 2010 22:23:14 -0700 (PDT)
References: <033e01cb7abb$623b7d50$26b277f0$@com> <AANLkTi=BPo3g3zVD2P3qjW_65sh-O6js8mrGup-E4oPf@mail.gmail.com>
In-Reply-To: <AANLkTi=BPo3g3zVD2P3qjW_65sh-O6js8mrGup-E4oPf@mail.gmail.com>
Mime-Version: 1.0 (iPhone Mail 8B117)
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
	charset=us-ascii
Message-Id: <E3F1EA91-FEB2-4C33-B00E-122D6F1EF1CF@holsman.net>
Cc: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: iPhone Mail (8B117)
From: Ian Holsman <hadoop@holsman.net>
Subject: Re: web-based file transfer
Date: Wed, 3 Nov 2010 16:22:02 +1100
To: "general@hadoop.apache.org" <general@hadoop.apache.org>

Doesn't chukwa do something like this?

---
Ian Holsman - 703 879-3128

I saw the angel in the marble and carved until I set him free -- Michelangelo

On 03/11/2010, at 5:44 AM, Eric Sammer <esammer@cloudera.com> wrote:

> I would recommend against clients pushing data directly to hdfs like
> this for a few reasons.
> 
> 1. The HDFS cluster would need to be directly exposed to a public
> network; you don't want to do this.
> 2. You'd be applying (presumably) a high concurrent load to HDFS which
> isn't its strong point.
> 
> From an architecture point of view, it's much nicer to have a queuing
> system between the upload and ingestion into HDFS that you can
> throttle and control, if necessary. This also allows you to isolate
> the cluster from the outside world. As to not bottleneck on a single
> writer, you can have uploaded files land in a queue and have multiple
> competing consumers popping files (or file names upon which to
> operate) out of the queue and handling the writing in parallel while
> being able to control the number of workers. If the initial upload is
> to a shared device like NFS, you can have writers live on multiple
> boxes and distribute the work.
> 
> Another option is to consider Flume, but only if you can deal with the
> fact that it effectively throws away the notion of files and treats
> their contents as individual events, etc.
> http://github.com/cloudera/flume.
> 
> Hope that helps.
> 
> On Tue, Nov 2, 2010 at 2:25 PM, Mark Laffoon
> <mlaffoon@semanticresearch.com> wrote:
>> We want to enable our web-based client (i.e. browser client, java applet,
>> whatever?) to transfer files into a system backed by hdfs. The obvious
>> simple solution is to do http file uploads, then copy the file to hdfs. I
>> was wondering if there is a way to do it with an hdfs-enabled applet where
>> the server gives the client the necessary hadoop configuration
>> information, and the client applet pushes the data directly into hdfs.
>> 
>> 
>> 
>> Has anybody done this or something similar? Can you give me a starting
>> point (I'm about to go wander through the hadoop CLI code to get ideas).
>> 
>> 
>> 
>> Thanks,
>> 
>> Mark
>> 
>> 
> 
> 
> 
> -- 
> Eric Sammer
> twitter: esammer
> data: www.cloudera.com

From general-return-2316-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 03 05:43:28 2010
Return-Path: <general-return-2316-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81825 invoked from network); 3 Nov 2010 05:43:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Nov 2010 05:43:28 -0000
Received: (qmail 45001 invoked by uid 500); 3 Nov 2010 05:43:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44721 invoked by uid 500); 3 Nov 2010 05:43:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44711 invoked by uid 99); 3 Nov 2010 05:43:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Nov 2010 05:43:55 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of shv.hadoop@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Nov 2010 05:43:50 +0000
Received: by pwj9 with SMTP id 9so121585pwj.35
        for <general@hadoop.apache.org>; Tue, 02 Nov 2010 22:43:30 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=wSUTujQblbKCGJaIQavHB8eusF9QkX9rQqcXzokgi80=;
        b=c0YlbedyJzPUEE4Ra7dRkvFq7KQd6SOudgdC1zR6SRiMKTt/jP+SRgeviZZ1vGzQHO
         K7wxTHus6jlb1jjGfC2ADBDQQ2aKvQpChtv4IyuZ+lvBWrvLjwMoiyAV9mRowJiguvwP
         CSwOuO3YgsrRQFS0gqH9ku+5ChL36zbGjg0dE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=gNoRXfZs8QJH5bclyN6uWla0qCBrY4qy7ERBuVnrq2vkeyEia2TEA5MD8xjlxqnucw
         yTYZskJeUSxPQeym1KW/JIBlIyTlwa5z60cWktSzQHfpAyYdEFTMKOopvY0w3WghvMQY
         u+QeO9n2mA7OgX+kTOKGRfwrF3PdI2IlB30YM=
MIME-Version: 1.0
Received: by 10.142.161.12 with SMTP id j12mr64435wfe.201.1288763010071; Tue,
 02 Nov 2010 22:43:30 -0700 (PDT)
Received: by 10.142.114.20 with HTTP; Tue, 2 Nov 2010 22:43:30 -0700 (PDT)
In-Reply-To: <33C92F80-1C08-48D0-AF52-162FF6BB0379@yahoo-inc.com>
References: <AANLkTikkU7w4BMMD9qz8z4xYYMe832-aJRV-zR89M-_6@mail.gmail.com>
	<33C92F80-1C08-48D0-AF52-162FF6BB0379@yahoo-inc.com>
Date: Tue, 2 Nov 2010 22:43:30 -0700
Message-ID: <AANLkTikW0jm0mS_KKbPBfBiC7oTfk2z_1XQ1968fx4cD@mail.gmail.com>
Subject: Re: Hadoop code reviews on ReviewBoard?
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd14fae6a3e5a04941f850d

--000e0cd14fae6a3e5a04941f850d
Content-Type: text/plain; charset=ISO-8859-1

It looks very hard to follow.
I have really hard time matching jira and review board comments.
Especially when they interleave.
Why again do we need a second system for tracking issues?
--Konstantin

On Tue, Oct 26, 2010 at 3:49 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:

> +1
>
> On Oct 26, 2010, at 2:50 PM, Dhruba Borthakur wrote:
>
> I saw an ASF announcement that there's now a review board instance
>> available, it will be nice if we can try it out.
>>
>> https://blogs.apache.org/infra/entry/reviewboard_instance_running_at_the
>>
>> I filed the following JIRA to see if we can use review board to review
>> Hadoop JIRAS.
>>
>> https://issues.apache.org/jira/browse/INFRA-3108
>>
>> thanks,
>> dhruba
>>
>
>

--000e0cd14fae6a3e5a04941f850d--

From general-return-2317-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 03 11:35:48 2010
Return-Path: <general-return-2317-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98452 invoked from network); 3 Nov 2010 11:35:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Nov 2010 11:35:47 -0000
Received: (qmail 55792 invoked by uid 500); 3 Nov 2010 11:36:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55418 invoked by uid 500); 3 Nov 2010 11:36:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55410 invoked by uid 99); 3 Nov 2010 11:36:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Nov 2010 11:36:11 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL,URI_OBFU_WWW
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Nov 2010 11:36:05 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id EB29C1BA5FA
	for <general@hadoop.apache.org>; Wed,  3 Nov 2010 11:35:43 +0000 (GMT)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id SQAnj0m2Cp1V for <general@hadoop.apache.org>;
	Wed,  3 Nov 2010 11:35:43 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by colossus.hpl.hp.com (Postfix) with ESMTPS id 1641D1BA59D
	for <general@hadoop.apache.org>; Wed,  3 Nov 2010 11:35:42 +0000 (GMT)
MailScanner-NULL-Check: 1289388922.05139@hixvR3Ol+0/CudDeblngsA
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id oA3BZLw9010972
	for <general@hadoop.apache.org>; Wed, 3 Nov 2010 11:35:21 GMT
Message-ID: <4CD148F9.8080600@apache.org>
Date: Wed, 03 Nov 2010 11:35:21 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.14) Gecko/20101006 Thunderbird/3.0.9
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: web-based file transfer
References: <033e01cb7abb$623b7d50$26b277f0$@com>
In-Reply-To: <033e01cb7abb$623b7d50$26b277f0$@com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: oA3BZLw9010972
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 02/11/10 18:25, Mark Laffoon wrote:
> We want to enable our web-based client (i.e. browser client, java applet,
> whatever?) to transfer files into a system backed by hdfs. The obvious
> simple solution is to do http file uploads, then copy the file to hdfs. I
> was wondering if there is a way to do it with an hdfs-enabled applet where
> the server gives the client the necessary hadoop configuration
> information, and the client applet pushes the data directly into hdfs.


I recall some work done with webdav
   https://issues.apache.org/jira/browse/HDFS-225
-but I don't think it's progressed

I've done things like this in the past with servlets and forms; the 
webapp you deploy has the hadoop configuration (and the network rights 
to talk to HDFS in the datacentre), the clients PUT/POST up content

http://www.slideshare.net/steve_l/long-haul-hadoop

However, you are limited to 2GB worth of upload/download in most web 
clients, some (chrome) go up to 4GB but you are pushing the limit there. 
Even all the Java servlet APIs assume that the content-length header 
fits into a signed 32 bit integer and gets unhappy once you go over 2GB 
(something I worry about in 
http://jira.smartfrog.org/jira/browse/SFOS-1476 )

Because Hadoop really likes large files -tens to hundreds of GB in a big 
cluster- I don't think the current web infrastructure is up to working 
with it.


that said, looking at hudson for the nightly runs of my bulk IO tests , 
jetty will serve up 4GB in 5 minutes (loopback if), and I can POST  or 
PUT up 4GB, but I have to get/set content length headers myself rather 
than rely on the java.net client and servlet implementations to handle it:

http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/components/www/src/org/smartfrog/services/www/bulkio/client/SunJavaBulkIOClient.java?revision=8430&view=markup

If you can control the client, then maybe you would be able to do >4GB 
uploads, but otherwise you are stuck with data <2GB in size, which is, 
-what- 4-8 blocks in a production cluster?

-steve

From general-return-2318-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 03 16:05:20 2010
Return-Path: <general-return-2318-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 26258 invoked from network); 3 Nov 2010 16:05:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Nov 2010 16:05:20 -0000
Received: (qmail 31704 invoked by uid 500); 3 Nov 2010 16:05:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31420 invoked by uid 500); 3 Nov 2010 16:05:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31412 invoked by uid 99); 3 Nov 2010 16:05:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Nov 2010 16:05:47 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.215.176] (HELO mail-ey0-f176.google.com) (209.85.215.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Nov 2010 16:05:43 +0000
Received: by eyz10 with SMTP id 10so354406eyz.35
        for <general@hadoop.apache.org>; Wed, 03 Nov 2010 09:05:21 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.71.206 with SMTP id r56mr793182wed.29.1288800320135; Wed,
 03 Nov 2010 09:05:20 -0700 (PDT)
Received: by 10.216.161.201 with HTTP; Wed, 3 Nov 2010 09:05:19 -0700 (PDT)
In-Reply-To: <E3F1EA91-FEB2-4C33-B00E-122D6F1EF1CF@holsman.net>
References: <033e01cb7abb$623b7d50$26b277f0$@com>
	<AANLkTi=BPo3g3zVD2P3qjW_65sh-O6js8mrGup-E4oPf@mail.gmail.com>
	<E3F1EA91-FEB2-4C33-B00E-122D6F1EF1CF@holsman.net>
Date: Wed, 3 Nov 2010 12:05:19 -0400
Message-ID: <AANLkTin-PkJvFy5R-wXwMzcQ50xZVkgbGKRU_6UA0GXg@mail.gmail.com>
Subject: Re: web-based file transfer
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Something like it, but Chukwa is more similar to Flume. For *files*
one may want something slightly different. For a stream of (data)
events, Chukwa, Flume, or Scribe are appropriate.

On Wed, Nov 3, 2010 at 1:22 AM, Ian Holsman <hadoop@holsman.net> wrote:
> Doesn't chukwa do something like this?
>
> ---
> Ian Holsman - 703 879-3128
>
> I saw the angel in the marble and carved until I set him free -- Michelangelo
>
> On 03/11/2010, at 5:44 AM, Eric Sammer <esammer@cloudera.com> wrote:
>
>> I would recommend against clients pushing data directly to hdfs like
>> this for a few reasons.
>>
>> 1. The HDFS cluster would need to be directly exposed to a public
>> network; you don't want to do this.
>> 2. You'd be applying (presumably) a high concurrent load to HDFS which
>> isn't its strong point.
>>
>> From an architecture point of view, it's much nicer to have a queuing
>> system between the upload and ingestion into HDFS that you can
>> throttle and control, if necessary. This also allows you to isolate
>> the cluster from the outside world. As to not bottleneck on a single
>> writer, you can have uploaded files land in a queue and have multiple
>> competing consumers popping files (or file names upon which to
>> operate) out of the queue and handling the writing in parallel while
>> being able to control the number of workers. If the initial upload is
>> to a shared device like NFS, you can have writers live on multiple
>> boxes and distribute the work.
>>
>> Another option is to consider Flume, but only if you can deal with the
>> fact that it effectively throws away the notion of files and treats
>> their contents as individual events, etc.
>> http://github.com/cloudera/flume.
>>
>> Hope that helps.
>>
>> On Tue, Nov 2, 2010 at 2:25 PM, Mark Laffoon
>> <mlaffoon@semanticresearch.com> wrote:
>>> We want to enable our web-based client (i.e. browser client, java applet,
>>> whatever?) to transfer files into a system backed by hdfs. The obvious
>>> simple solution is to do http file uploads, then copy the file to hdfs. I
>>> was wondering if there is a way to do it with an hdfs-enabled applet where
>>> the server gives the client the necessary hadoop configuration
>>> information, and the client applet pushes the data directly into hdfs.
>>>
>>>
>>>
>>> Has anybody done this or something similar? Can you give me a starting
>>> point (I'm about to go wander through the hadoop CLI code to get ideas).
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Mark
>>>
>>>
>>
>>
>>
>> --
>> Eric Sammer
>> twitter: esammer
>> data: www.cloudera.com
>



-- 
Eric Sammer
twitter: esammer
data: www.cloudera.com

From general-return-2319-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 03 16:13:30 2010
Return-Path: <general-return-2319-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38565 invoked from network); 3 Nov 2010 16:13:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Nov 2010 16:13:30 -0000
Received: (qmail 44854 invoked by uid 500); 3 Nov 2010 16:14:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44676 invoked by uid 500); 3 Nov 2010 16:13:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44664 invoked by uid 99); 3 Nov 2010 16:13:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Nov 2010 16:13:58 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [145.100.8.49] (HELO smtp.sara.nl) (145.100.8.49)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Nov 2010 16:13:53 +0000
Received: from planck.ka.sara.nl (145.100.8.32) by sara-exch-fe1.ka.sara.nl
 (145.100.8.49) with Microsoft SMTP Server (TLS) id 14.0.702.0; Wed, 3 Nov
 2010 17:13:29 +0100
Received: from planck.ka.sara.nl ([145.100.8.32]) by planck.ka.sara.nl
 ([145.100.8.32]) with mapi; Wed, 3 Nov 2010 17:13:08 +0100
From: Evert Lammerts <Evert.Lammerts@sara.nl>
To: Evert Lammerts <Evert.Lammerts@sara.nl>
Date: Wed, 3 Nov 2010 17:13:08 +0100
Subject: Hadoop workshop at SARA, December 7
Thread-Topic: Hadoop workshop at SARA, December 7
Thread-Index: Act7cgFN0DF7es6wRtSfJfyOQQ/Gtg==
Message-ID: <ADF94D8555C7A246B86A633685E0178AB90A474A7D@planck.ka.sara.nl>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/signed; protocol="application/x-pkcs7-signature";
	micalg=SHA1; boundary="----=_NextPart_000_0139_01CB7B7A.631D6B10"
MIME-Version: 1.0

------=_NextPart_000_0139_01CB7B7A.631D6B10
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit

What: SARA Apache Hadoop computing workshop
When: Tuesday, December 7th
Where: SARA, Science Park 121 Amsterdam, room TBA


Dear readers,

You are all invited to participate in the first Apache Hadoop "hackathon" in 
The Netherlands on Tuesday, December 7th. This event is also the kick-off of 
the SARA Proof-of-Concept Hadoop service. More information on the event and 
the technology can be found below.

If you are interested in participating, please register as soon as possible, 
and not later than December 1st by sending an email to evert.lammerts@sara.nl.

Best regards,
Evert Lammerts

===================================================

SARA Apache Hadoop "Hackathon"

Back in 2004, driven by the need to process extreme amounts of data, Google 
developed a system that exists of the Google File System and MapReduce. Based 
on two papers that Google published on the topic[1,2], Doug Cutting[3] created 
what is now the Apache Hadoop software stack. Apache Hadoop is used in several 
capacities by a number of internet giants (e.g. Yahoo!, IBM, eBay, Facebook, 
Last.fm, LinkedIn, Twitter and many more[4]). It's ability to store and 
process extremely large datasets makes it particularly beneficial for sciences 
as Natural Language Processing, BioInformatics, Social Sciences and 
Humanities. SARA provides a Proof-of-Concept Apache Hadoop storage and 
computing service for scientific use[5].

The "hackathon" will be a hands-on introduction to Hadoop. You are invited to 
come spend a day at SARA to play with the system, with the support of two 
experienced users, Edgar Meij (UvA) and Djoerd Hiemstra (UT). You can work on 
your own or in groups and analyze your own dataset or play with for example 
the Wikipedia, ENRON or White House access records datasets. Programming 
skills will be assumed, and you need to take your own laptops.

Coffee and soft drinks will be provided throughout the day, and lunch will be 
served at 12.00 AM.

Program:

09:30 - 09:35   Introduction to the day
09:35 - 10:15   Examples of data analysis with Hadoop
10:30 - 17:00   Hackathon
17:00 - end     Presentation of results


[1] http://labs.google.com/papers/gfs.html
[2] http://labs.google.com/papers/mapreduce.html
[3] http://en.wikipedia.org/wiki/Doug_Cutting
[4] http://wiki.apache.org/hadoop/PoweredBy
[5] http://www.sara.nl/news/recent/20101103/Hadoop_proof-of-concept.html


Evert Lammerts
Consultant eScience Support
SARA Computing & Network Services
High Performance Computing & Visualization

Phone: +31 20 888 4101
Email: evert.lammerts@sara.nl
http://www.sara.nl



------=_NextPart_000_0139_01CB7B7A.631D6B10
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIPUDCCBDIw
ggMaoAMCAQICAQEwDQYJKoZIhvcNAQEFBQAwezELMAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0
ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0
ZWQxITAfBgNVBAMMGEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczAeFw0wNDAxMDEwMDAwMDBaFw0y
ODEyMzEyMzU5NTlaMHsxCzAJBgNVBAYTAkdCMRswGQYDVQQIDBJHcmVhdGVyIE1hbmNoZXN0ZXIx
EDAOBgNVBAcMB1NhbGZvcmQxGjAYBgNVBAoMEUNvbW9kbyBDQSBMaW1pdGVkMSEwHwYDVQQDDBhB
QUEgQ2VydGlmaWNhdGUgU2VydmljZXMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC+
QJ30buHqdoccTUVEjr5GyIMGncEq/hgfjuQC+vOrXVCKFjELmgbQxXAizUktVGPMtm5oRgtT6stM
JMC8ck7q8RWu9FSaEgrDerIzYOLaiVXzIljz3tzP74OGooyUT59o8piQRoQnx3a/48w1LIteB2Rl
gsBIsKiR+WGfdiBQqJHHZrXreGIDVvCKGhPqMaMeoJn9OPb2JzJYbwf1a7j7FCuvt6rM1mNfc4za
BZmoOKjLF3g2UazpnvR4Oo3PD9lC4pgMqy+fDgHe75+ZSfEt36x0TRuYtUfF5SnR+ZAYx2KcvoPH
Jns+iiXHwN2d5jVoECCdj9je0sOEnA1e6C/JAgMBAAGjgcAwgb0wHQYDVR0OBBYEFKARCiM+lvEH
7OKvKe+CpX/QMKS0MA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MHsGA1UdHwR0MHIw
OKA2oDSGMmh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0FBQUNlcnRpZmljYXRlU2VydmljZXMuY3Js
MDagNKAyhjBodHRwOi8vY3JsLmNvbW9kby5uZXQvQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmww
DQYJKoZIhvcNAQEFBQADggEBAAhW/ALwm+j/pPrWe8ZEgM5PxMX2AFjMpra8FEloBHbo5u5d7AIP
YNaNUBhPJk4B4+awpe6/vHRUQb/9/BK4x09a9IlgBX9gtwVK8/bxwr/EuXSGti19a8zS80bdL8bg
asPDNAMsfZbdWsIOpwqZwQWLqwwv81w6z2w3VQmH3lNAbFjv/LarZW4E9hvcPOBaFcae2fFZSDAh
ZQNs7Okhc+ybA6HgN62gFRiP+roCzqcsqRATLNTlCCarIpdg+JBedNSimlO98qlo4KJuwtdssaMP
nr/raOdW8q7y4ys4OgmBtWuF174t7T8at7Jj4vViLILUagBBUPE5g5+V6TaWmG4wggTdMIIDxaAD
AgECAhBxkvvmGV+sTRKFdHE0ohinMA0GCSqGSIb3DQEBBQUAMHsxCzAJBgNVBAYTAkdCMRswGQYD
VQQIDBJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcMB1NhbGZvcmQxGjAYBgNVBAoMEUNvbW9k
byBDQSBMaW1pdGVkMSEwHwYDVQQDDBhBQUEgQ2VydGlmaWNhdGUgU2VydmljZXMwHhcNMDQwMTAx
MDAwMDAwWhcNMjgxMjMxMjM1OTU5WjCBrjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYD
VQQHEw5TYWx0IExha2UgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYD
VQQLExhodHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1VU0VSRmlyc3QtQ2xp
ZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBFbWFpbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBALI5haTyfatBO2JGN67NwWB1vDll+UoaR6K5zEjMapjVTTUZuaRC5c5J4oovHnzSMQfHTrSD
ZJ0uKdWiZMSFvYVRNXmkTmiQexx6pJKoF/KYFfKTzMmkMpW7DE8wvZigC4vlbhuiRvp4vKJvq1le
pS/Pytptqi/rrKGzaqq3Lmc1i3nhHmmI4uZGzaCl6r4LznY6eg6b6vzaJ1s9cx8i5khhxkzzabGo
Lhu21DEgLLyCio6kDqXXiUP8FlqvHXHXEVnauocNr/rz4cLwpMVnjNbWVDreCqS6A3ezZcj9HtN0
YqoYymiTHqGFfvVHZcv4TVcodNI0/zC27vZiMBSMLOsCAwEAAaOCAScwggEjMB8GA1UdIwQYMBaA
FKARCiM+lvEH7OKvKe+CpX/QMKS0MB0GA1UdDgQWBBSJgmd9xJ0mcABLtFBIfN49rgRufTAOBgNV
HQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUH
AwQwEQYDVR0gBAowCDAGBgRVHSAAMHsGA1UdHwR0MHIwOKA2oDSGMmh0dHA6Ly9jcmwuY29tb2Rv
Y2EuY29tL0FBQUNlcnRpZmljYXRlU2VydmljZXMuY3JsMDagNKAyhjBodHRwOi8vY3JsLmNvbW9k
by5uZXQvQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwEQYJYIZIAYb4QgEBBAQDAgEGMA0GCSqG
SIb3DQEBBQUAA4IBAQCdlcs8uH6lCcQevwvCx3aOOTyUxhCqTwzJ4KuEXYlU4GU7820cfDcsJVRf
liH8N4SRnRXcFE+Bz1Qda2xFYMct+ZdRTPlmyjyggoymyPDi6dRK+ew/VsnddozDggFPbADzHhph
dARHA6nGQFeRvGUixSdnT1fbZFrZjR+6hi/0Bq6cae3p9M8pF9jgSp8aIC+XTFG7RgfEijdOIOMJ
MWjHnsSLneh+EbwyaBCWEZhE2CpRYE2I63Q630MGMsg5Vow6EVLTQaRDA/Tt7zMn2zngFE4mydj1
OeKJuJNdtykmQeqzm66D/Hd1yujKtf7iZUpjPkTE0MNeh3OpmByvfxV/MIIGNTCCBR2gAwIBAgIR
APUHLR99vIaqc6YYFoh++L8wDQYJKoZIhvcNAQEFBQAwga4xCzAJBgNVBAYTAlVTMQswCQYDVQQI
EwJVVDEXMBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNVBAoTFVRoZSBVU0VSVFJVU1QgTmV0
d29yazEhMB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3QuY29tMTYwNAYDVQQDEy1VVE4tVVNF
UkZpcnN0LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1haWwwHhcNMTAwMjA4MDAwMDAwWhcN
MTEwMjA4MjM1OTU5WjCB4DE1MDMGA1UECxMsQ29tb2RvIFRydXN0IE5ldHdvcmsgLSBQRVJTT05B
IE5PVCBWQUxJREFURUQxRjBEBgNVBAsTPVRlcm1zIGFuZCBDb25kaXRpb25zIG9mIHVzZTogaHR0
cDovL3d3dy5jb21vZG8ubmV0L3JlcG9zaXRvcnkxHzAdBgNVBAsTFihjKTIwMDMgQ29tb2RvIExp
bWl0ZWQxFzAVBgNVBAMTDkV2ZXJ0IExhbW1lcnRzMSUwIwYJKoZIhvcNAQkBFhZldmVydC5sYW1t
ZXJ0c0BzYXJhLm5sMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtYAj4sXuBqEwPxYY
BMkYLPGrnqT35USIERoWs26xtSY+dLVc7HqaHDZ1lU5m9wFx/Jqph1urlRuAcCj03kGSS23Ta6kt
HTRMYRCHI3nOLfEEbH4POT+wDytORUHNeEKp/PrBn1kmKk+dk+bJI7MNdy2XkiEeO+OqKmLiBGkT
glIxngOeqgqR2IvmVv2hLAyzq8Ax20inwveZsDMa7QdQLSVUaX1kSRSihwz4Jw9X/K+6oA3ZLGp9
pZYnFHBNTHM2DR/C7dNwQgbZS7TY/Jb1kObYNhwD2JzzJlkoe0blR17MeJkF7/+Y414j0AdPFB7I
ggZ4Tm7Maheer9cIQSsvIQIDAQABo4ICGDCCAhQwHwYDVR0jBBgwFoAUiYJnfcSdJnAAS7RQSHze
Pa4Ebn0wHQYDVR0OBBYEFNTyUAqBNg9JXuiPHa9jfLvRM3IyMA4GA1UdDwEB/wQEAwIFoDAMBgNV
HRMBAf8EAjAAMCAGA1UdJQQZMBcGCCsGAQUFBwMEBgsrBgEEAbIxAQMFAjARBglghkgBhvhCAQEE
BAMCBSAwRgYDVR0gBD8wPTA7BgwrBgEEAbIxAQIBAQEwKzApBggrBgEFBQcCARYdaHR0cHM6Ly9z
ZWN1cmUuY29tb2RvLm5ldC9DUFMwgaUGA1UdHwSBnTCBmjBMoEqgSIZGaHR0cDovL2NybC5jb21v
ZG9jYS5jb20vVVROLVVTRVJGaXJzdC1DbGllbnRBdXRoZW50aWNhdGlvbmFuZEVtYWlsLmNybDBK
oEigRoZEaHR0cDovL2NybC5jb21vZG8ubmV0L1VUTi1VU0VSRmlyc3QtQ2xpZW50QXV0aGVudGlj
YXRpb25hbmRFbWFpbC5jcmwwbAYIKwYBBQUHAQEEYDBeMDYGCCsGAQUFBzAChipodHRwOi8vY3J0
LmNvbW9kb2NhLmNvbS9VVE5BQUFDbGllbnRDQS5jcnQwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3Nw
LmNvbW9kb2NhLmNvbTAhBgNVHREEGjAYgRZldmVydC5sYW1tZXJ0c0BzYXJhLm5sMA0GCSqGSIb3
DQEBBQUAA4IBAQBaJvuiBPgqdq9MEYH9dj2vW8hprAIBEnOOvfXdoP604kBhSbj3s/l0ZQyQY35W
YVAltuJQ365uJKigPQHxS+slpUOAJPEVNmwPUIW0GAjxCPxgLdJgAc4jOV8f7oF3pnfI4ukCmjPN
LpaylZzMhFt+t6zDWiMtUqyxK/4on62LoLp2D88lxwxI3q3i00bPlV0wZ+mjzS7vrQGcUgDPWEBL
FD0I3pR+Z1EH1qjelBmBJV9paVzfzmoU93LIDJA7/MEUXd3JiMZGEoVd2qE5qIZO4fNJ+Cyo6tu1
ECSQxoL6qN+sBwekiFA9Q/zlUp0HfU6Jt8WmtZ44SOh67LcmCL4ZMYIEaDCCBGQCAQEwgcQwga4x
CzAJBgNVBAYTAlVTMQswCQYDVQQIEwJVVDEXMBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNV
BAoTFVRoZSBVU0VSVFJVU1QgTmV0d29yazEhMB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3Qu
Y29tMTYwNAYDVQQDEy1VVE4tVVNFUkZpcnN0LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1h
aWwCEQD1By0ffbyGqnOmGBaIfvi/MAkGBSsOAwIaBQCgggJ4MBgGCSqGSIb3DQEJAzELBgkqhkiG
9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTEwMTEwMzE2MTMwOFowIwYJKoZIhvcNAQkEMRYEFKyK7Ikj
RKZ1uh9vzwl/DzPa32hxMGcGCSqGSIb3DQEJDzFaMFgwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwIC
AgCAMA0GCCqGSIb3DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMAcGBSsOAwIaMAoGCCqG
SIb3DQIFMIHVBgkrBgEEAYI3EAQxgccwgcQwga4xCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJVVDEX
MBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNVBAoTFVRoZSBVU0VSVFJVU1QgTmV0d29yazEh
MB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3QuY29tMTYwNAYDVQQDEy1VVE4tVVNFUkZpcnN0
LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1haWwCEQD1By0ffbyGqnOmGBaIfvi/MIHXBgsq
hkiG9w0BCRACCzGBx6CBxDCBrjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5T
YWx0IExha2UgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYDVQQLExho
dHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1VU0VSRmlyc3QtQ2xpZW50IEF1
dGhlbnRpY2F0aW9uIGFuZCBFbWFpbAIRAPUHLR99vIaqc6YYFoh++L8wDQYJKoZIhvcNAQEBBQAE
ggEAPjom3mRPlZJT4yngjjuquKsdV89HNOBF6Ee/w+sv3917I15eAAW8u0qw7OqJz9A+h9vNvYwk
IcnOtvViTHInL1Yip/Oa4taYiEO/zA1iPdpA3q6hrE54mfVglzH0OO7PRJvq68qZCtWhDiPS+ZeI
ayBKX+jqOweiX9I5zb0aKBS3zYk5ERW5gqEF0PBQvEd7Tha6zRwSverHPSfd7qDSKJMqMW2rSZff
83b/WvSX5t+UpdrqlmWR21whYTmVt2XWPQs1XboOD4jQqtiIq2r63gzPD0JDbKbMBKb88hbJswOh
IYQbSqQI3VAOp5G93n/XlXCynbomSvY938ebK1EaggAAAAAAAA==

------=_NextPart_000_0139_01CB7B7A.631D6B10--

From general-return-2320-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 03 16:32:06 2010
Return-Path: <general-return-2320-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49333 invoked from network); 3 Nov 2010 16:32:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Nov 2010 16:32:06 -0000
Received: (qmail 84174 invoked by uid 500); 3 Nov 2010 16:32:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83645 invoked by uid 500); 3 Nov 2010 16:32:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83299 invoked by uid 99); 3 Nov 2010 16:32:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Nov 2010 16:32:35 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [98.139.91.81] (HELO nm11.bullet.mail.sp2.yahoo.com) (98.139.91.81)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 03 Nov 2010 16:32:25 +0000
Received: from [98.139.91.61] by nm11.bullet.mail.sp2.yahoo.com with NNFMP; 03 Nov 2010 16:32:03 -0000
Received: from [98.139.91.60] by tm1.bullet.mail.sp2.yahoo.com with NNFMP; 03 Nov 2010 16:32:03 -0000
Received: from [127.0.0.1] by omp1060.mail.sp2.yahoo.com with NNFMP; 03 Nov 2010 16:32:03 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 428270.13035.bm@omp1060.mail.sp2.yahoo.com
Received: (qmail 71679 invoked by uid 60001); 3 Nov 2010 16:32:03 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1288801922; bh=2QHeBIOLiOpUfa0IELYExLPQkHy4ewd2GePqHPEKAqo=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=lGY6HNKI13Ze5bXg3eN4uSUmGrseV29VyVO+4gSw4RWcWeuKCchjtuntviw12jKZBhbqs3oCQw34lSx1mPLo4QgtqAgf0O8S0+T8sTfk13TUGbYgpRjtS2epKv2HGesZV9Gph3d9kuuaHx2B0ghvKDsw+eiPovz71MFUO/EEa1M=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=CFBzdNoir46FWApBXYiE1GdQo1PhHteQUTrfqp06/cQhgkkv4dfgVRhhklFypQqMztdXExuDtxXNMO12Hs3gwQlqiRPYXf1xXTcnHRitgFh/Gujzm5bS/KnJfkVe5xnGBzamLPQXf3tbOnsGHlHUJA7ycqpP2iVX3z0ZpyC9u5s=;
Message-ID: <975551.64007.qm@web112101.mail.gq1.yahoo.com>
X-YMail-OSG: h.m.I8UVM1kCuRnQnHD.f_FalhZXUTzgc6_HjcYoh3enww8
 pHqfatBdfhc0JS22s.xth9VZ_rFy36i5lHWU3AnOxSEOnKZWcRiTLVbFO0X2
 HWKzu8xkh24Rd0xBgaKPcwNKt2ZRaYmPDMY9fbHKzjdMDMaLnpgWSvpUnnmx
 73mailjG4akxnEMnLrD0cefDiFvzp8jRiIBNe67UQnuJoC7mJqz93PZpiCxq
 LL6jxvSuU1_.8E24lk2zjwFCWdl.sE.VHvXOU20zKGmFTK_Z5vx11.wFKmsh
 w9IUphH84EBKbFWwFyeIA_W6c0V.utx0eo26QJoleS4mv_NM.60z.vA--
Received: from [82.130.74.71] by web112101.mail.gq1.yahoo.com via HTTP; Wed, 03 Nov 2010 09:32:02 PDT
X-Mailer: YahooMailClassic/11.4.9 YahooMailWebService/0.8.107.284920
Date: Wed, 3 Nov 2010 09:32:02 -0700 (PDT)
From: Grandl Robert <rgrandl@yahoo.com>
Subject: Hadoop on PlanetLab nodes
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-68915803-1288801922=:64007"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-68915803-1288801922=:64007
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Hi all,

I am trying to set up Hadoop=A0 on PlanetLab nodes.

However, as far as I seen I need password less SSH login between master and=
 slave nodes in Hadoop.=20
My=0Amain problem is for planet lab nodes I have a privatekey I always use=
=0Awhen connect to planet lab nodes. I am using=A0 ssh -i ~/.ssh/privatekey=
=0Auser@planet_lab_node

Also, the username on planet lab nodes is different from the user name I ha=
ve on my local cluster.=20

Is there any possibility to use hadoop on planet lab nodes ?

Any hints, suggestions or links will be valuable for me.

Thanks,
Robert=0A=0A=0A      
--0-68915803-1288801922=:64007--

From general-return-2321-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 03 16:54:04 2010
Return-Path: <general-return-2321-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63689 invoked from network); 3 Nov 2010 16:54:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Nov 2010 16:54:03 -0000
Received: (qmail 37960 invoked by uid 500); 3 Nov 2010 16:54:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37752 invoked by uid 500); 3 Nov 2010 16:54:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37738 invoked by uid 99); 3 Nov 2010 16:54:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Nov 2010 16:54:31 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Nov 2010 16:54:26 +0000
Received: by fxm7 with SMTP id 7so699448fxm.35
        for <general@hadoop.apache.org>; Wed, 03 Nov 2010 09:54:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=S3f9OHIgmaqVVi4/ke6ihvNA1/6b6R+hMcotLrViFj0=;
        b=atXGNDDZSzxst5tb16XxSNVHykV+7oJeQud2mi70HLg0SwGyunedg9vq9ZZ4QNaOvy
         OuVKtllj6F/a3zoqUDQ9OkfF2nIOzZnbZHhig/dfLArQfwAvS4b4WzEEqLS52o4tiK0W
         uvsAwIZxbhn4On73jVTCcdJMiiL6kW0STxg3g=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=fACEIGFHXMcRPaDETC49YCPWTqgLDYZRKcXAHV83miLwNqjulf/H10AIAcrgwIYnyQ
         ZXxPMmAOR9lXUyviAi2yjsTWo2zmvg1331cxi+7hKOWZ+Tq/C7oFcw99+QYxrw9FC5y9
         0EuxO4I0V3KuL45lpS8j4Qr6TEiwIXei5Ti94=
Received: by 10.223.100.9 with SMTP id w9mr5278856fan.12.1288803245076; Wed,
 03 Nov 2010 09:54:05 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.223.120.80 with HTTP; Wed, 3 Nov 2010 09:53:43 -0700 (PDT)
In-Reply-To: <975551.64007.qm@web112101.mail.gq1.yahoo.com>
References: <975551.64007.qm@web112101.mail.gq1.yahoo.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Wed, 3 Nov 2010 22:23:43 +0530
Message-ID: <AANLkTimzTjm11o+rNsgbSzabObAY_R++5Uqe+dKWM1fa@mail.gmail.com>
Subject: Re: Hadoop on PlanetLab nodes
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi,

On Wed, Nov 3, 2010 at 10:02 PM, Grandl Robert <rgrandl@yahoo.com> wrote:
> Hi all,
>
> I am trying to set up Hadoop=A0 on PlanetLab nodes.
>
> However, as far as I seen I need password less SSH login between master a=
nd slave nodes in Hadoop.

This is "needed" for ease of setup, but you can as well start all
nodes manually yourselves without requiring SSH as a dependency. Just
for your information (won't be the right way out, perhaps).

For instance, you can ssh manually and issue "hadoop tasktracker" and
"hadoop datanode" commands in the nodes to start those daemons.

> My
> main problem is for planet lab nodes I have a privatekey I always use
> when connect to planet lab nodes. I am using=A0 ssh -i ~/.ssh/privatekey
> user@planet_lab_node
>
> Also, the username on planet lab nodes is different from the user name I =
have on my local cluster.
>
> Is there any possibility to use hadoop on planet lab nodes ?
>
> Any hints, suggestions or links will be valuable for me.
>
> Thanks,
> Robert
>
>
>



--=20
Harsh J
www.harshj.com

From general-return-2322-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 03 17:03:26 2010
Return-Path: <general-return-2322-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68556 invoked from network); 3 Nov 2010 17:03:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Nov 2010 17:03:25 -0000
Received: (qmail 56786 invoked by uid 500); 3 Nov 2010 17:03:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56508 invoked by uid 500); 3 Nov 2010 17:03:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56495 invoked by uid 99); 3 Nov 2010 17:03:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Nov 2010 17:03:54 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Nov 2010 17:03:49 +0000
Received: by wyg36 with SMTP id 36so854247wyg.35
        for <general@hadoop.apache.org>; Wed, 03 Nov 2010 10:03:27 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.176.20 with SMTP id a20mr11863327wem.14.1288803807505;
 Wed, 03 Nov 2010 10:03:27 -0700 (PDT)
Received: by 10.216.161.201 with HTTP; Wed, 3 Nov 2010 10:03:27 -0700 (PDT)
In-Reply-To: <AANLkTimzTjm11o+rNsgbSzabObAY_R++5Uqe+dKWM1fa@mail.gmail.com>
References: <975551.64007.qm@web112101.mail.gq1.yahoo.com>
	<AANLkTimzTjm11o+rNsgbSzabObAY_R++5Uqe+dKWM1fa@mail.gmail.com>
Date: Wed, 3 Nov 2010 13:03:27 -0400
Message-ID: <AANLkTimAyBdNDxLpA5pRw7fJjNKLvacnndfao-EXNNgA@mail.gmail.com>
Subject: Re: Hadoop on PlanetLab nodes
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Better than passwordless ssh is to use ssh-agent and load your key
(providing the passphrase once). See 'man ssh-agent' for details,
although it's pretty straight forward.

Short version:

# See if ssh-agent is already running for your user by trying to list
# keys it knows about. That's a "dash-ell."
ssh-add -l

# Load an ssh key.
ssh-add ~/.ssh/id_rsa
<enter passphrase>

If your public key is already on all the nodes, watch as magically
start-all/stop-all work without requiring a password.

WARNING: When working with authentication, remote login, and anything
security related, *never* blindly take advice. Always take the time to
understand the ramifications. See man pages for ssh, ssh-agent,
ssh-add, and ssh_config.

Hope that helps.

On Wed, Nov 3, 2010 at 12:53 PM, Harsh J <qwertymaniac@gmail.com> wrote:
> Hi,
>
> On Wed, Nov 3, 2010 at 10:02 PM, Grandl Robert <rgrandl@yahoo.com> wrote:
>> Hi all,
>>
>> I am trying to set up Hadoop=A0 on PlanetLab nodes.
>>
>> However, as far as I seen I need password less SSH login between master =
and slave nodes in Hadoop.
>
> This is "needed" for ease of setup, but you can as well start all
> nodes manually yourselves without requiring SSH as a dependency. Just
> for your information (won't be the right way out, perhaps).
>
> For instance, you can ssh manually and issue "hadoop tasktracker" and
> "hadoop datanode" commands in the nodes to start those daemons.
>
>> My
>> main problem is for planet lab nodes I have a privatekey I always use
>> when connect to planet lab nodes. I am using=A0 ssh -i ~/.ssh/privatekey
>> user@planet_lab_node
>>
>> Also, the username on planet lab nodes is different from the user name I=
 have on my local cluster.
>>
>> Is there any possibility to use hadoop on planet lab nodes ?
>>
>> Any hints, suggestions or links will be valuable for me.
>>
>> Thanks,
>> Robert
>>
>>
>>
>
>
>
> --
> Harsh J
> www.harshj.com
>



--=20
Eric Sammer
twitter: esammer
data: www.cloudera.com

From general-return-2323-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 03 17:16:16 2010
Return-Path: <general-return-2323-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84090 invoked from network); 3 Nov 2010 17:16:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Nov 2010 17:16:16 -0000
Received: (qmail 76935 invoked by uid 500); 3 Nov 2010 17:16:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76709 invoked by uid 500); 3 Nov 2010 17:16:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76701 invoked by uid 99); 3 Nov 2010 17:16:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Nov 2010 17:16:45 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Nov 2010 17:16:35 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id CCEE3B7E0D
	for <general@hadoop.apache.org>; Wed,  3 Nov 2010 17:16:14 +0000 (GMT)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id PdSUQ3npOVaL for <general@hadoop.apache.org>;
	Wed,  3 Nov 2010 17:16:14 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 17F9BB7E08
	for <general@hadoop.apache.org>; Wed,  3 Nov 2010 17:16:13 +0000 (GMT)
MailScanner-NULL-Check: 1289409341.49953@lu5lrlSFtNqY/RZEb+E3rg
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id oA3HFeCB000533
	for <general@hadoop.apache.org>; Wed, 3 Nov 2010 17:15:40 GMT
Message-ID: <4CD198BC.1060304@apache.org>
Date: Wed, 03 Nov 2010 17:15:40 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.14) Gecko/20101006 Thunderbird/3.0.9
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Hadoop on PlanetLab nodes
References: <975551.64007.qm@web112101.mail.gq1.yahoo.com>
In-Reply-To: <975551.64007.qm@web112101.mail.gq1.yahoo.com>
Content-Type: text/plain; charset=ISO-8859-2; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: oA3HFeCB000533
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 03/11/10 16:32, Grandl Robert wrote:
> Hi all,
>
> I am trying to set up Hadoop  on PlanetLab nodes.
>
> However, as far as I seen I need password less SSH login between master and slave nodes in Hadoop.
> My
> main problem is for planet lab nodes I have a privatekey I always use
> when connect to planet lab nodes. I am using  ssh -i ~/.ssh/privatekey
> user@planet_lab_node
>
> Also, the username on planet lab nodes is different from the user name I have on my local cluster.
>
> Is there any possibility to use hadoop on planet lab nodes ?
>

I wouldn't rush to do it as their nodes are distributed with odd 
networking -you get a wide cluster, not a fast one.

Better to try for cluster time on OpenCirrus, which does run Hadoop.

From general-return-2324-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 03 17:56:19 2010
Return-Path: <general-return-2324-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 507 invoked from network); 3 Nov 2010 17:56:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Nov 2010 17:56:18 -0000
Received: (qmail 47983 invoked by uid 500); 3 Nov 2010 17:56:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47865 invoked by uid 500); 3 Nov 2010 17:56:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47857 invoked by uid 99); 3 Nov 2010 17:56:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Nov 2010 17:56:46 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Nov 2010 17:56:39 +0000
Received: by qyk5 with SMTP id 5so219679qyk.14
        for <general@hadoop.apache.org>; Wed, 03 Nov 2010 10:56:18 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.238.16 with SMTP id kq16mr2303574qcb.134.1288806978599;
 Wed, 03 Nov 2010 10:56:18 -0700 (PDT)
Received: by 10.229.225.137 with HTTP; Wed, 3 Nov 2010 10:56:18 -0700 (PDT)
In-Reply-To: <AANLkTikW0jm0mS_KKbPBfBiC7oTfk2z_1XQ1968fx4cD@mail.gmail.com>
References: <AANLkTikkU7w4BMMD9qz8z4xYYMe832-aJRV-zR89M-_6@mail.gmail.com>
	<33C92F80-1C08-48D0-AF52-162FF6BB0379@yahoo-inc.com>
	<AANLkTikW0jm0mS_KKbPBfBiC7oTfk2z_1XQ1968fx4cD@mail.gmail.com>
Date: Wed, 3 Nov 2010 10:56:18 -0700
Message-ID: <AANLkTi=JjC=+L3mDPvfkiTM5Ag6dG5eeSk4PL5kNsaSu@mail.gmail.com>
Subject: Re: Hadoop code reviews on ReviewBoard?
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hey Cos,

I think the setup used for Hive and HBase posts the RB comments to the
jira which makes it easier to follow.

RB isn't a second system for tracking issues, jira isn't a code review
system, and RB isn't an issue tracker. I think the idea is to have
issue/design-level discussion on jira and code-level discussion on RB.

Thanks,
Eli

On Tue, Nov 2, 2010 at 10:43 PM, Konstantin Shvachko
<shv.hadoop@gmail.com> wrote:
> It looks very hard to follow.
> I have really hard time matching jira and review board comments.
> Especially when they interleave.
> Why again do we need a second system for tracking issues?
> --Konstantin
>
> On Tue, Oct 26, 2010 at 3:49 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:
>
>> +1
>>
>> On Oct 26, 2010, at 2:50 PM, Dhruba Borthakur wrote:
>>
>> I saw an ASF announcement that there's now a review board instance
>>> available, it will be nice if we can try it out.
>>>
>>> https://blogs.apache.org/infra/entry/reviewboard_instance_running_at_the
>>>
>>> I filed the following JIRA to see if we can use review board to review
>>> Hadoop JIRAS.
>>>
>>> https://issues.apache.org/jira/browse/INFRA-3108
>>>
>>> thanks,
>>> dhruba
>>>
>>
>>
>

From general-return-2325-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 03 18:28:52 2010
Return-Path: <general-return-2325-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31650 invoked from network); 3 Nov 2010 18:28:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Nov 2010 18:28:51 -0000
Received: (qmail 91970 invoked by uid 500); 3 Nov 2010 18:29:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91795 invoked by uid 500); 3 Nov 2010 18:29:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91787 invoked by uid 99); 3 Nov 2010 18:29:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Nov 2010 18:29:20 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.213.48] (HELO mail-yw0-f48.google.com) (209.85.213.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Nov 2010 18:29:14 +0000
Received: by ywp4 with SMTP id 4so796307ywp.35
        for <general@hadoop.apache.org>; Wed, 03 Nov 2010 11:28:53 -0700 (PDT)
Received: by 10.223.114.82 with SMTP id d18mr809175faq.66.1288808932215;
        Wed, 03 Nov 2010 11:28:52 -0700 (PDT)
Received: from tp (c-24-4-185-157.hsd1.ca.comcast.net [24.4.185.157])
        by mx.google.com with ESMTPS id 10sm3852680fax.42.2010.11.03.11.28.50
        (version=SSLv3 cipher=RC4-MD5);
        Wed, 03 Nov 2010 11:28:51 -0700 (PDT)
Sender: Konstantin Boudnik <cos@boudnik.org>
Date: Wed, 3 Nov 2010 11:28:47 -0700
From: Konstantin Boudnik <cos@apache.org>
To: general@hadoop.apache.org
Subject: Re: Hadoop code reviews on ReviewBoard?
Message-ID: <20101103182846.GD7137@tp>
References: <AANLkTikkU7w4BMMD9qz8z4xYYMe832-aJRV-zR89M-_6@mail.gmail.com>
 <33C92F80-1C08-48D0-AF52-162FF6BB0379@yahoo-inc.com>
 <AANLkTikW0jm0mS_KKbPBfBiC7oTfk2z_1XQ1968fx4cD@mail.gmail.com>
 <AANLkTi=JjC=+L3mDPvfkiTM5Ag6dG5eeSk4PL5kNsaSu@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature"; boundary="7DO5AaGCk89r4vaK"
Content-Disposition: inline
In-Reply-To: <AANLkTi=JjC=+L3mDPvfkiTM5Ag6dG5eeSk4PL5kNsaSu@mail.gmail.com>
X-Organization: It's something of 'Cos
X-PGP-Key: http://www.boudnik.org/~cos/pubkey.asc
User-Agent: Mutt/1.5.20 (2009-06-14)

--7DO5AaGCk89r4vaK
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Hey Eli.

the explanation is right on the money except that other Konstantin was aski=
ng ;)

Cos

On Wed, Nov 03, 2010 at 10:56AM, Eli Collins wrote:
> Hey Cos,
>=20
> I think the setup used for Hive and HBase posts the RB comments to the
> jira which makes it easier to follow.
>=20
> RB isn't a second system for tracking issues, jira isn't a code review
> system, and RB isn't an issue tracker. I think the idea is to have
> issue/design-level discussion on jira and code-level discussion on RB.
>=20
> Thanks,
> Eli
>=20
> On Tue, Nov 2, 2010 at 10:43 PM, Konstantin Shvachko
> <shv.hadoop@gmail.com> wrote:
> > It looks very hard to follow.
> > I have really hard time matching jira and review board comments.
> > Especially when they interleave.
> > Why again do we need a second system for tracking issues?
> > --Konstantin
> >
> > On Tue, Oct 26, 2010 at 3:49 PM, Arun C Murthy <acm@yahoo-inc.com> wrot=
e:
> >
> >> +1
> >>
> >> On Oct 26, 2010, at 2:50 PM, Dhruba Borthakur wrote:
> >>
> >> I saw an ASF announcement that there's now a review board instance
> >>> available, it will be nice if we can try it out.
> >>>
> >>> https://blogs.apache.org/infra/entry/reviewboard_instance_running_at_=
the
> >>>
> >>> I filed the following JIRA to see if we can use review board to review
> >>> Hadoop JIRAS.
> >>>
> >>> https://issues.apache.org/jira/browse/INFRA-3108
> >>>
> >>> thanks,
> >>> dhruba
> >>>
> >>
> >>
> >

--7DO5AaGCk89r4vaK
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iF4EAREIAAYFAkzRqd4ACgkQenyFlstYjhIEigEAu8B02SVMMrxEPfSBS+GO6Eep
yovFJ9NabIiN8fIL+N0A/0+oVIYEvrZj/tJ8ACjhZwp515cOwIGAueq1zgkqtGny
=meaK
-----END PGP SIGNATURE-----

--7DO5AaGCk89r4vaK--

From general-return-2326-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 03 23:19:31 2010
Return-Path: <general-return-2326-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63829 invoked from network); 3 Nov 2010 23:19:31 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Nov 2010 23:19:31 -0000
Received: (qmail 51798 invoked by uid 500); 3 Nov 2010 23:20:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51736 invoked by uid 500); 3 Nov 2010 23:20:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51728 invoked by uid 99); 3 Nov 2010 23:20:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Nov 2010 23:20:00 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [195.232.224.71] (HELO mailout02.vodafone.com) (195.232.224.71)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Nov 2010 23:19:53 +0000
Received: from mailint02 (localhost [127.0.0.1])
	by mailout02 (Postfix) with ESMTP id 84C8F214480
	for <general@hadoop.apache.org>; Thu,  4 Nov 2010 00:19:32 +0100 (CET)
Received: from avoexs02.internal.vodafone.com (unknown [145.230.4.135])
	(using TLSv1 with cipher RC4-MD5 (128/128 bits))
	(No client certificate requested)
	by mailint02 (Postfix) with ESMTPS id 791AE2144AB
	for <general@hadoop.apache.org>; Thu,  4 Nov 2010 00:19:32 +0100 (CET)
Received: from VF-MBX13.internal.vodafone.com ([145.230.5.24]) by avoexs02.internal.vodafone.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Thu, 4 Nov 2010 00:19:33 +0100
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Subject: RE: web-based file transfer
Date: Thu, 4 Nov 2010 00:19:32 +0100
Message-ID: <219D8244D980254ABF28AB469AD4E98F0346A152@VF-MBX13.internal.vodafone.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: web-based file transfer
Thread-Index: Act7cP6h/AoLuUpGTa+NPpITyj0u4wAOt7oD
References: <033e01cb7abb$623b7d50$26b277f0$@com><AANLkTi=BPo3g3zVD2P3qjW_65sh-O6js8mrGup-E4oPf@mail.gmail.com><E3F1EA91-FEB2-4C33-B00E-122D6F1EF1CF@holsman.net> <AANLkTin-PkJvFy5R-wXwMzcQ50xZVkgbGKRU_6UA0GXg@mail.gmail.com>
From: "Gibbon, Robert, VF-Group" <Robert.Gibbon@vodafone.com>
To: <general@hadoop.apache.org>,
	<general@hadoop.apache.org>
X-OriginalArrivalTime: 03 Nov 2010 23:19:33.0790 (UTC) FILETIME=[931CFBE0:01CB7BAD]
X-Virus-Checked: Checked by ClamAV on apache.org

Check out HDFS over WebDAV

- http://www.hadoop.iponweb.net/Home/hdfs-over-webdav

WebDAV is an HTTP based protocol for accessing remote filesystems.

I'm running an adapted version of this. It runs under Jetty which is =
pretty industry standard and is built on Apache JackRabbit which is =
pretty production stable too. I lashed together a custom JAAS =
authentication module to authenticate it against our user database.

You can mount WebDAV on Linux using FUSE and WDFS, or script sessions =
with cadaver on Solaris/Unix without mounting WebDav. It works pretty =
sweet on Windows and Apple, too.=20

Recent versions of Jetty have built in traffic shaping and QoS features, =
although you might get more mileage from HAProxy or a hardware =
loadbalancer.

It works pretty sweet as it enforces HDFS permissions (if you have them =
enabled). To get Hadoop permission integrity enforced on MapReduce jobs =
check out Oozie - it's a job submission proxy which runs under Tomcat =
(might work with Jetty too - haven't tried) and can use a custom =
ServletFilter for authentication which you can also patch onto your own =
user database/directory.=20

Then you just need to seal the perimeter of your cluster with Firewall =
rules and you're good to go

No more Kerberos!
R

-----Original Message-----
From: Eric Sammer [mailto:esammer@cloudera.com]
Sent: Wed 11/3/2010 5:05 PM
To: general@hadoop.apache.org
Subject: Re: web-based file transfer
=20
Something like it, but Chukwa is more similar to Flume. For *files*
one may want something slightly different. For a stream of (data)
events, Chukwa, Flume, or Scribe are appropriate.

On Wed, Nov 3, 2010 at 1:22 AM, Ian Holsman <hadoop@holsman.net> wrote:
> Doesn't chukwa do something like this?
>
> ---
> Ian Holsman - 703 879-3128
>
> I saw the angel in the marble and carved until I set him free -- =
Michelangelo
>
> On 03/11/2010, at 5:44 AM, Eric Sammer <esammer@cloudera.com> wrote:
>
>> I would recommend against clients pushing data directly to hdfs like
>> this for a few reasons.
>>
>> 1. The HDFS cluster would need to be directly exposed to a public
>> network; you don't want to do this.
>> 2. You'd be applying (presumably) a high concurrent load to HDFS =
which
>> isn't its strong point.
>>
>> From an architecture point of view, it's much nicer to have a queuing
>> system between the upload and ingestion into HDFS that you can
>> throttle and control, if necessary. This also allows you to isolate
>> the cluster from the outside world. As to not bottleneck on a single
>> writer, you can have uploaded files land in a queue and have multiple
>> competing consumers popping files (or file names upon which to
>> operate) out of the queue and handling the writing in parallel while
>> being able to control the number of workers. If the initial upload is
>> to a shared device like NFS, you can have writers live on multiple
>> boxes and distribute the work.
>>
>> Another option is to consider Flume, but only if you can deal with =
the
>> fact that it effectively throws away the notion of files and treats
>> their contents as individual events, etc.
>> http://github.com/cloudera/flume.
>>
>> Hope that helps.
>>
>> On Tue, Nov 2, 2010 at 2:25 PM, Mark Laffoon
>> <mlaffoon@semanticresearch.com> wrote:
>>> We want to enable our web-based client (i.e. browser client, java =
applet,
>>> whatever?) to transfer files into a system backed by hdfs. The =
obvious
>>> simple solution is to do http file uploads, then copy the file to =
hdfs. I
>>> was wondering if there is a way to do it with an hdfs-enabled applet =
where
>>> the server gives the client the necessary hadoop configuration
>>> information, and the client applet pushes the data directly into =
hdfs.
>>>
>>>
>>>
>>> Has anybody done this or something similar? Can you give me a =
starting
>>> point (I'm about to go wander through the hadoop CLI code to get =
ideas).
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Mark
>>>
>>>
>>
>>
>>
>> --
>> Eric Sammer
>> twitter: esammer
>> data: www.cloudera.com
>



--=20
Eric Sammer
twitter: esammer
data: www.cloudera.com


From general-return-2327-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 04 03:43:37 2010
Return-Path: <general-return-2327-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88754 invoked from network); 4 Nov 2010 03:43:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Nov 2010 03:43:36 -0000
Received: (qmail 60162 invoked by uid 500); 4 Nov 2010 03:44:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59752 invoked by uid 500); 4 Nov 2010 03:44:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59744 invoked by uid 99); 4 Nov 2010 03:44:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Nov 2010 03:44:03 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Nov 2010 03:43:58 +0000
Received: by wyg36 with SMTP id 36so1457711wyg.35
        for <general@hadoop.apache.org>; Wed, 03 Nov 2010 20:43:36 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.227.132.137 with SMTP id b9mr149215wbt.48.1288842215921; Wed,
 03 Nov 2010 20:43:35 -0700 (PDT)
Received: by 10.216.239.200 with HTTP; Wed, 3 Nov 2010 20:43:35 -0700 (PDT)
In-Reply-To: <AANLkTimAyBdNDxLpA5pRw7fJjNKLvacnndfao-EXNNgA@mail.gmail.com>
References: <975551.64007.qm@web112101.mail.gq1.yahoo.com>
	<AANLkTimzTjm11o+rNsgbSzabObAY_R++5Uqe+dKWM1fa@mail.gmail.com>
	<AANLkTimAyBdNDxLpA5pRw7fJjNKLvacnndfao-EXNNgA@mail.gmail.com>
Date: Thu, 4 Nov 2010 14:43:35 +1100
Message-ID: <AANLkTimDWZsTemXbS-ph7unD_ubf6XU42O5RMwzgwHGg@mail.gmail.com>
Subject: Re: Hadoop on PlanetLab nodes
From: Ian Holsman <hadoop@holsman.net>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163646bc4873976e049431f689

--00163646bc4873976e049431f689
Content-Type: text/plain; charset=ISO-8859-1

to answer the 2nd part of your question.. how do I log on with a different
userID.
you can create a file called
.ssh/config

and in that you can specify the userID to use as a default, as well as other
parameters
for example:

I have the following set up for all hosts on example.com

 33 Host *.example.com
 34         Port 22
 35         Protocol 2
 36         Compression no
 38         User xyz

man ssh_config has more details.

Regards
Ian


On Thu, Nov 4, 2010 at 4:03 AM, Eric Sammer <esammer@cloudera.com> wrote:

> Better than passwordless ssh is to use ssh-agent and load your key
> (providing the passphrase once). See 'man ssh-agent' for details,
> although it's pretty straight forward.
>
> Short version:
>
> # See if ssh-agent is already running for your user by trying to list
> # keys it knows about. That's a "dash-ell."
> ssh-add -l
>
> # Load an ssh key.
> ssh-add ~/.ssh/id_rsa
> <enter passphrase>
>
> If your public key is already on all the nodes, watch as magically
> start-all/stop-all work without requiring a password.
>
> WARNING: When working with authentication, remote login, and anything
> security related, *never* blindly take advice. Always take the time to
> understand the ramifications. See man pages for ssh, ssh-agent,
> ssh-add, and ssh_config.
>
> Hope that helps.
>
> On Wed, Nov 3, 2010 at 12:53 PM, Harsh J <qwertymaniac@gmail.com> wrote:
> > Hi,
> >
> > On Wed, Nov 3, 2010 at 10:02 PM, Grandl Robert <rgrandl@yahoo.com>
> wrote:
> >> Hi all,
> >>
> >> I am trying to set up Hadoop  on PlanetLab nodes.
> >>
> >> However, as far as I seen I need password less SSH login between master
> and slave nodes in Hadoop.
> >
> > This is "needed" for ease of setup, but you can as well start all
> > nodes manually yourselves without requiring SSH as a dependency. Just
> > for your information (won't be the right way out, perhaps).
> >
> > For instance, you can ssh manually and issue "hadoop tasktracker" and
> > "hadoop datanode" commands in the nodes to start those daemons.
> >
> >> My
> >> main problem is for planet lab nodes I have a privatekey I always use
> >> when connect to planet lab nodes. I am using  ssh -i ~/.ssh/privatekey
> >> user@planet_lab_node
> >>
> >> Also, the username on planet lab nodes is different from the user name I
> have on my local cluster.
> >>
> >> Is there any possibility to use hadoop on planet lab nodes ?
> >>
> >> Any hints, suggestions or links will be valuable for me.
> >>
> >> Thanks,
> >> Robert
> >>
> >>
> >>
> >
> >
> >
> > --
> > Harsh J
> > www.harshj.com
> >
>
>
>
> --
> Eric Sammer
> twitter: esammer
> data: www.cloudera.com
>

--00163646bc4873976e049431f689--

From general-return-2328-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 04 06:15:24 2010
Return-Path: <general-return-2328-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44804 invoked from network); 4 Nov 2010 06:15:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Nov 2010 06:15:24 -0000
Received: (qmail 60366 invoked by uid 500); 4 Nov 2010 06:15:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60273 invoked by uid 500); 4 Nov 2010 06:15:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60265 invoked by uid 99); 4 Nov 2010 06:15:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Nov 2010 06:15:50 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of shv.hadoop@gmail.com designates 74.125.83.176 as permitted sender)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Nov 2010 06:15:44 +0000
Received: by pvg16 with SMTP id 16so568872pvg.35
        for <general@hadoop.apache.org>; Wed, 03 Nov 2010 23:15:22 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=wOdyzM8AJENA36kfE4TRjEztMcioWUBsL5JpLIbR2+8=;
        b=iJ2hfA6oRmNlMycZZOdfgrHnYcN8HgvKo1IwgIB7jXhMxhnMgLAB7h9qUGtTl57i9J
         jMVf7iI92JAr+hxWy1QXiR1Bz9vWg/r3se1rKFR1sNSszbqZyIK2y9sd6CdRsiLf/4zK
         9kqOidta5gofjQBvIKNaNlUsQMbC39yXdBm6Y=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=RRNZGxUEg2MzjdjloDhN30v2S2gZCKF/CN8/Aekph+TCiKCInaEGVpdqnTnqJP0m/9
         AQ8V5kHFJIxL0PBOEWaXwIlprE7ajm9aEf4TKMp/yBrRBUmvL/YsFhKXX/4TU2Mf5NqO
         e30NlE6g8Vy8KlxjcId3DmPYvPIv/li8Qh1pk=
MIME-Version: 1.0
Received: by 10.142.213.15 with SMTP id l15mr250888wfg.87.1288851322477; Wed,
 03 Nov 2010 23:15:22 -0700 (PDT)
Received: by 10.142.114.20 with HTTP; Wed, 3 Nov 2010 23:15:22 -0700 (PDT)
In-Reply-To: <AANLkTi=JjC=+L3mDPvfkiTM5Ag6dG5eeSk4PL5kNsaSu@mail.gmail.com>
References: <AANLkTikkU7w4BMMD9qz8z4xYYMe832-aJRV-zR89M-_6@mail.gmail.com>
	<33C92F80-1C08-48D0-AF52-162FF6BB0379@yahoo-inc.com>
	<AANLkTikW0jm0mS_KKbPBfBiC7oTfk2z_1XQ1968fx4cD@mail.gmail.com>
	<AANLkTi=JjC=+L3mDPvfkiTM5Ag6dG5eeSk4PL5kNsaSu@mail.gmail.com>
Date: Wed, 3 Nov 2010 23:15:22 -0700
Message-ID: <AANLkTi=idCK_Q8a75rL46O=yqKDLiamrO+aP136icW1V@mail.gmail.com>
Subject: Re: Hadoop code reviews on ReviewBoard?
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd2296e3e9e150494341566
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd2296e3e9e150494341566
Content-Type: text/plain; charset=ISO-8859-1

 > I think the idea is to have issue/design-level discussion on jira and
code-level discussion on RB.

This is exactly my question. Why do we need two places to have the same
discussion?
Even if the comments are automatically replicated in jira and/or vice versa.
Why not choose one and stick with it?

Thanks,
--Konst

On Wed, Nov 3, 2010 at 10:56 AM, Eli Collins <eli@cloudera.com> wrote:

> Hey Cos,
>
> I think the setup used for Hive and HBase posts the RB comments to the
> jira which makes it easier to follow.
>
> RB isn't a second system for tracking issues, jira isn't a code review
> system, and RB isn't an issue tracker. I think the idea is to have
> issue/design-level discussion on jira and code-level discussion on RB.
>
> Thanks,
> Eli
>
> On Tue, Nov 2, 2010 at 10:43 PM, Konstantin Shvachko
> <shv.hadoop@gmail.com> wrote:
> > It looks very hard to follow.
> > I have really hard time matching jira and review board comments.
> > Especially when they interleave.
> > Why again do we need a second system for tracking issues?
> > --Konstantin
> >
> > On Tue, Oct 26, 2010 at 3:49 PM, Arun C Murthy <acm@yahoo-inc.com>
> wrote:
> >
> >> +1
> >>
> >> On Oct 26, 2010, at 2:50 PM, Dhruba Borthakur wrote:
> >>
> >> I saw an ASF announcement that there's now a review board instance
> >>> available, it will be nice if we can try it out.
> >>>
> >>>
> https://blogs.apache.org/infra/entry/reviewboard_instance_running_at_the
> >>>
> >>> I filed the following JIRA to see if we can use review board to review
> >>> Hadoop JIRAS.
> >>>
> >>> https://issues.apache.org/jira/browse/INFRA-3108
> >>>
> >>> thanks,
> >>> dhruba
> >>>
> >>
> >>
> >
>

--000e0cd2296e3e9e150494341566--

From general-return-2329-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 04 06:20:24 2010
Return-Path: <general-return-2329-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45568 invoked from network); 4 Nov 2010 06:20:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Nov 2010 06:20:24 -0000
Received: (qmail 64744 invoked by uid 500); 4 Nov 2010 06:20:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 64597 invoked by uid 500); 4 Nov 2010 06:20:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 64589 invoked by uid 99); 4 Nov 2010 06:20:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Nov 2010 06:20:52 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Nov 2010 06:20:46 +0000
Received: by ewy3 with SMTP id 3so801995ewy.35
        for <general@hadoop.apache.org>; Wed, 03 Nov 2010 23:20:25 -0700 (PDT)
Received: by 10.213.16.72 with SMTP id n8mr315455eba.38.1288851625055; Wed, 03
 Nov 2010 23:20:25 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.213.14.78 with HTTP; Wed, 3 Nov 2010 23:19:54 -0700 (PDT)
In-Reply-To: <AANLkTi=idCK_Q8a75rL46O=yqKDLiamrO+aP136icW1V@mail.gmail.com>
References: <AANLkTikkU7w4BMMD9qz8z4xYYMe832-aJRV-zR89M-_6@mail.gmail.com>
 <33C92F80-1C08-48D0-AF52-162FF6BB0379@yahoo-inc.com> <AANLkTikW0jm0mS_KKbPBfBiC7oTfk2z_1XQ1968fx4cD@mail.gmail.com>
 <AANLkTi=JjC=+L3mDPvfkiTM5Ag6dG5eeSk4PL5kNsaSu@mail.gmail.com> <AANLkTi=idCK_Q8a75rL46O=yqKDLiamrO+aP136icW1V@mail.gmail.com>
From: Alejandro Abdelnur <tucu@cloudera.com>
Date: Thu, 4 Nov 2010 14:19:54 +0800
Message-ID: <AANLkTimR0yj3QR2-qY1=iqQg6_XbRm8+BTdKUwBtCDCf@mail.gmail.com>
Subject: Re: Hadoop code reviews on ReviewBoard?
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015174c3fce47c79d0494342737
X-Virus-Checked: Checked by ClamAV on apache.org

--0015174c3fce47c79d0494342737
Content-Type: text/plain; charset=ISO-8859-1

Hey Konstantin,

have you seen how you can comment on patches and see side by side the
changes in RB?

It is quite handy.

Alejandro

On Thu, Nov 4, 2010 at 2:15 PM, Konstantin Shvachko <shv.hadoop@gmail.com>wrote:

>  > I think the idea is to have issue/design-level discussion on jira and
> code-level discussion on RB.
>
> This is exactly my question. Why do we need two places to have the same
> discussion?
> Even if the comments are automatically replicated in jira and/or vice
> versa.
> Why not choose one and stick with it?
>
> Thanks,
> --Konst
>
> On Wed, Nov 3, 2010 at 10:56 AM, Eli Collins <eli@cloudera.com> wrote:
>
> > Hey Cos,
> >
> > I think the setup used for Hive and HBase posts the RB comments to the
> > jira which makes it easier to follow.
> >
> > RB isn't a second system for tracking issues, jira isn't a code review
> > system, and RB isn't an issue tracker. I think the idea is to have
> > issue/design-level discussion on jira and code-level discussion on RB.
> >
> > Thanks,
> > Eli
> >
> > On Tue, Nov 2, 2010 at 10:43 PM, Konstantin Shvachko
> > <shv.hadoop@gmail.com> wrote:
> > > It looks very hard to follow.
> > > I have really hard time matching jira and review board comments.
> > > Especially when they interleave.
> > > Why again do we need a second system for tracking issues?
> > > --Konstantin
> > >
> > > On Tue, Oct 26, 2010 at 3:49 PM, Arun C Murthy <acm@yahoo-inc.com>
> > wrote:
> > >
> > >> +1
> > >>
> > >> On Oct 26, 2010, at 2:50 PM, Dhruba Borthakur wrote:
> > >>
> > >> I saw an ASF announcement that there's now a review board instance
> > >>> available, it will be nice if we can try it out.
> > >>>
> > >>>
> > https://blogs.apache.org/infra/entry/reviewboard_instance_running_at_the
> > >>>
> > >>> I filed the following JIRA to see if we can use review board to
> review
> > >>> Hadoop JIRAS.
> > >>>
> > >>> https://issues.apache.org/jira/browse/INFRA-3108
> > >>>
> > >>> thanks,
> > >>> dhruba
> > >>>
> > >>
> > >>
> > >
> >
>

--0015174c3fce47c79d0494342737--

From general-return-2330-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 04 06:49:57 2010
Return-Path: <general-return-2330-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56833 invoked from network); 4 Nov 2010 06:49:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Nov 2010 06:49:56 -0000
Received: (qmail 79481 invoked by uid 500); 4 Nov 2010 06:50:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79145 invoked by uid 500); 4 Nov 2010 06:50:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 79137 invoked by uid 99); 4 Nov 2010 06:50:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Nov 2010 06:50:25 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Nov 2010 06:50:18 +0000
Received: by qyk4 with SMTP id 4so998890qyk.14
        for <general@hadoop.apache.org>; Wed, 03 Nov 2010 23:49:57 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.224.214.5 with SMTP id gy5mr268795qab.215.1288853397866; Wed,
 03 Nov 2010 23:49:57 -0700 (PDT)
Received: by 10.229.225.137 with HTTP; Wed, 3 Nov 2010 23:49:57 -0700 (PDT)
In-Reply-To: <AANLkTi=idCK_Q8a75rL46O=yqKDLiamrO+aP136icW1V@mail.gmail.com>
References: <AANLkTikkU7w4BMMD9qz8z4xYYMe832-aJRV-zR89M-_6@mail.gmail.com>
	<33C92F80-1C08-48D0-AF52-162FF6BB0379@yahoo-inc.com>
	<AANLkTikW0jm0mS_KKbPBfBiC7oTfk2z_1XQ1968fx4cD@mail.gmail.com>
	<AANLkTi=JjC=+L3mDPvfkiTM5Ag6dG5eeSk4PL5kNsaSu@mail.gmail.com>
	<AANLkTi=idCK_Q8a75rL46O=yqKDLiamrO+aP136icW1V@mail.gmail.com>
Date: Wed, 3 Nov 2010 23:49:57 -0700
Message-ID: <AANLkTi=d9M48Mj+Pgn4mKPUOOEkXiCvHtK04rTN+DY3R@mail.gmail.com>
Subject: Re: Hadoop code reviews on ReviewBoard?
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Wed, Nov 3, 2010 at 11:15 PM, Konstantin Shvachko
<shv.hadoop@gmail.com> wrote:
> =A0> I think the idea is to have issue/design-level discussion on jira an=
d
> code-level discussion on RB.
>
> This is exactly my question. Why do we need two places to have the same
> discussion?
> Even if the comments are automatically replicated in jira and/or vice ver=
sa.
> Why not choose one and stick with it?

For a lot of jiras issue level discussion (what's the problem? how
should we solve it?) is very different from code level discussion
(does this code do what we claim it should? Is it sufficient to be
checked in?).

Thanks,
Eli

>
> Thanks,
> --Konst
>
> On Wed, Nov 3, 2010 at 10:56 AM, Eli Collins <eli@cloudera.com> wrote:
>
>> Hey Cos,
>>
>> I think the setup used for Hive and HBase posts the RB comments to the
>> jira which makes it easier to follow.
>>
>> RB isn't a second system for tracking issues, jira isn't a code review
>> system, and RB isn't an issue tracker. I think the idea is to have
>> issue/design-level discussion on jira and code-level discussion on RB.
>>
>> Thanks,
>> Eli
>>
>> On Tue, Nov 2, 2010 at 10:43 PM, Konstantin Shvachko
>> <shv.hadoop@gmail.com> wrote:
>> > It looks very hard to follow.
>> > I have really hard time matching jira and review board comments.
>> > Especially when they interleave.
>> > Why again do we need a second system for tracking issues?
>> > --Konstantin
>> >
>> > On Tue, Oct 26, 2010 at 3:49 PM, Arun C Murthy <acm@yahoo-inc.com>
>> wrote:
>> >
>> >> +1
>> >>
>> >> On Oct 26, 2010, at 2:50 PM, Dhruba Borthakur wrote:
>> >>
>> >> I saw an ASF announcement that there's now a review board instance
>> >>> available, it will be nice if we can try it out.
>> >>>
>> >>>
>> https://blogs.apache.org/infra/entry/reviewboard_instance_running_at_the
>> >>>
>> >>> I filed the following JIRA to see if we can use review board to revi=
ew
>> >>> Hadoop JIRAS.
>> >>>
>> >>> https://issues.apache.org/jira/browse/INFRA-3108
>> >>>
>> >>> thanks,
>> >>> dhruba
>> >>>
>> >>
>> >>
>> >
>>
>

From general-return-2331-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 04 06:58:52 2010
Return-Path: <general-return-2331-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57549 invoked from network); 4 Nov 2010 06:58:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Nov 2010 06:58:51 -0000
Received: (qmail 82219 invoked by uid 500); 4 Nov 2010 06:59:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82160 invoked by uid 500); 4 Nov 2010 06:59:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82149 invoked by uid 99); 4 Nov 2010 06:59:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Nov 2010 06:59:20 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of shv.hadoop@gmail.com designates 209.85.210.48 as permitted sender)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Nov 2010 06:59:13 +0000
Received: by pzk3 with SMTP id 3so136884pzk.35
        for <general@hadoop.apache.org>; Wed, 03 Nov 2010 23:58:52 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=GXoST9EyMwE+wUWTvq6uSCIH4ng2krRnTxycs/HWNrI=;
        b=kC01LmYmLZrk8/me+hUNoKJTYx/M5GDbYalcBQqYAbuCps5tO/vODD+HODA2cdhIY5
         t3Zdrm7mSukWgSjVArPs8uL1qs6H0VEIPKjTbyfzeBYl1G4zH8dkMTCopbMg4J0YD86Q
         lwKv19PT5z3pGrPnrqS4FL+XiDQK5GZri6wD4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=omJwDYli5Dp1bWps+jFPauClGzKOAnrWBb6rsEO4Ll+h9krnKvOAHNs47Pd4I88cg9
         SELsC59bzC5Bpl68yIX8xXrp8BKOxamksKD1Di3S2J0UlRKbe8x0SULAmReYKssKCLsC
         +wGX524yOgtvOinldxHtKgoixBkxKGwTwHOic=
MIME-Version: 1.0
Received: by 10.142.144.7 with SMTP id r7mr256682wfd.258.1288853931149; Wed,
 03 Nov 2010 23:58:51 -0700 (PDT)
Received: by 10.142.114.20 with HTTP; Wed, 3 Nov 2010 23:58:51 -0700 (PDT)
In-Reply-To: <AANLkTimR0yj3QR2-qY1=iqQg6_XbRm8+BTdKUwBtCDCf@mail.gmail.com>
References: <AANLkTikkU7w4BMMD9qz8z4xYYMe832-aJRV-zR89M-_6@mail.gmail.com>
	<33C92F80-1C08-48D0-AF52-162FF6BB0379@yahoo-inc.com>
	<AANLkTikW0jm0mS_KKbPBfBiC7oTfk2z_1XQ1968fx4cD@mail.gmail.com>
	<AANLkTi=JjC=+L3mDPvfkiTM5Ag6dG5eeSk4PL5kNsaSu@mail.gmail.com>
	<AANLkTi=idCK_Q8a75rL46O=yqKDLiamrO+aP136icW1V@mail.gmail.com>
	<AANLkTimR0yj3QR2-qY1=iqQg6_XbRm8+BTdKUwBtCDCf@mail.gmail.com>
Date: Wed, 3 Nov 2010 23:58:51 -0700
Message-ID: <AANLkTinr2dTRSgY0+NNmf+vELB=SoNUJhV9Eyq7EUSQ5@mail.gmail.com>
Subject: Re: Hadoop code reviews on ReviewBoard?
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd328e8bbc7c6049434b0a8
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd328e8bbc7c6049434b0a8
Content-Type: text/plain; charset=ISO-8859-1

Yes, it is pretty handy. It would be really cool to have it integrated with
Jira.
But having comments in two different places is extremely distracting.

Is it only me who finds it hard to match jira messages with RB ones,
and duplicating the descriptions, affected versions, etc. in both systems?

--Konstantin


On Wed, Nov 3, 2010 at 11:19 PM, Alejandro Abdelnur <tucu@cloudera.com>wrote:

> Hey Konstantin,
>
> have you seen how you can comment on patches and see side by side the
> changes in RB?
>
> It is quite handy.
>
> Alejandro
>
> On Thu, Nov 4, 2010 at 2:15 PM, Konstantin Shvachko <shv.hadoop@gmail.com
> >wrote:
>
> >  > I think the idea is to have issue/design-level discussion on jira and
> > code-level discussion on RB.
> >
> > This is exactly my question. Why do we need two places to have the same
> > discussion?
> > Even if the comments are automatically replicated in jira and/or vice
> > versa.
> > Why not choose one and stick with it?
> >
> > Thanks,
> > --Konst
> >
> > On Wed, Nov 3, 2010 at 10:56 AM, Eli Collins <eli@cloudera.com> wrote:
> >
> > > Hey Cos,
> > >
> > > I think the setup used for Hive and HBase posts the RB comments to the
> > > jira which makes it easier to follow.
> > >
> > > RB isn't a second system for tracking issues, jira isn't a code review
> > > system, and RB isn't an issue tracker. I think the idea is to have
> > > issue/design-level discussion on jira and code-level discussion on RB.
> > >
> > > Thanks,
> > > Eli
> > >
> > > On Tue, Nov 2, 2010 at 10:43 PM, Konstantin Shvachko
> > > <shv.hadoop@gmail.com> wrote:
> > > > It looks very hard to follow.
> > > > I have really hard time matching jira and review board comments.
> > > > Especially when they interleave.
> > > > Why again do we need a second system for tracking issues?
> > > > --Konstantin
> > > >
> > > > On Tue, Oct 26, 2010 at 3:49 PM, Arun C Murthy <acm@yahoo-inc.com>
> > > wrote:
> > > >
> > > >> +1
> > > >>
> > > >> On Oct 26, 2010, at 2:50 PM, Dhruba Borthakur wrote:
> > > >>
> > > >> I saw an ASF announcement that there's now a review board instance
> > > >>> available, it will be nice if we can try it out.
> > > >>>
> > > >>>
> > >
> https://blogs.apache.org/infra/entry/reviewboard_instance_running_at_the
> > > >>>
> > > >>> I filed the following JIRA to see if we can use review board to
> > review
> > > >>> Hadoop JIRAS.
> > > >>>
> > > >>> https://issues.apache.org/jira/browse/INFRA-3108
> > > >>>
> > > >>> thanks,
> > > >>> dhruba
> > > >>>
> > > >>
> > > >>
> > > >
> > >
> >
>

--000e0cd328e8bbc7c6049434b0a8--

From general-return-2332-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 04 07:02:24 2010
Return-Path: <general-return-2332-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58450 invoked from network); 4 Nov 2010 07:02:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Nov 2010 07:02:24 -0000
Received: (qmail 84494 invoked by uid 500); 4 Nov 2010 07:02:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84371 invoked by uid 500); 4 Nov 2010 07:02:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84363 invoked by uid 99); 4 Nov 2010 07:02:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Nov 2010 07:02:53 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Nov 2010 07:02:48 +0000
Received: from [192.168.1.71] (snvvpn3-10-72-76-c52.hq.corp.yahoo.com [10.72.76.52])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id oA470w4J041268
	for <general@hadoop.apache.org>; Thu, 4 Nov 2010 00:00:59 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1288854059;
	bh=pHN/z6eVpsDkxIIa+pGt/QENkQPO84VXana3AJQcXFE=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=hcEPgdqSCgqqY32oZVRCOUlbGIhtpeYrHK2GbcMQLzqeOFfO5q1Jkx9wBz8a2xi0r
	 IFuacgMz3h+VLbVj89Mq3WHvAxnllVOjGfyAlxibk2WaL/kEXZqCw4gWt8pUJfUyui
	 X4ofed8NutdvJxQsOo2RHyhI9voOL+u0wC32r6jg=
Message-Id: <7767ED1D-124C-4976-99A3-99530CA9C605@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <AANLkTinr2dTRSgY0+NNmf+vELB=SoNUJhV9Eyq7EUSQ5@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Hadoop code reviews on ReviewBoard?
Date: Thu, 4 Nov 2010 00:00:58 -0700
References: <AANLkTikkU7w4BMMD9qz8z4xYYMe832-aJRV-zR89M-_6@mail.gmail.com> <33C92F80-1C08-48D0-AF52-162FF6BB0379@yahoo-inc.com> <AANLkTikW0jm0mS_KKbPBfBiC7oTfk2z_1XQ1968fx4cD@mail.gmail.com> <AANLkTi=JjC=+L3mDPvfkiTM5Ag6dG5eeSk4PL5kNsaSu@mail.gmail.com> <AANLkTi=idCK_Q8a75rL46O=yqKDLiamrO+aP136icW1V@mail.gmail.com> <AANLkTimR0yj3QR2-qY1=iqQg6_XbRm8+BTdKUwBtCDCf@mail.gmail.com> <AANLkTinr2dTRSgY0+NNmf+vELB=SoNUJhV9Eyq7EUSQ5@mail.gmail.com>
X-Mailer: Apple Mail (2.936)

Konst,

  Todd has volunteered to get RB to send msgs directly to jira, that  
should help...

Arun

On Nov 3, 2010, at 11:58 PM, Konstantin Shvachko wrote:

> Yes, it is pretty handy. It would be really cool to have it  
> integrated with
> Jira.
> But having comments in two different places is extremely distracting.
>
> Is it only me who finds it hard to match jira messages with RB ones,
> and duplicating the descriptions, affected versions, etc. in both  
> systems?
>
> --Konstantin
>
>
> On Wed, Nov 3, 2010 at 11:19 PM, Alejandro Abdelnur  
> <tucu@cloudera.com>wrote:
>
>> Hey Konstantin,
>>
>> have you seen how you can comment on patches and see side by side the
>> changes in RB?
>>
>> It is quite handy.
>>
>> Alejandro
>>
>> On Thu, Nov 4, 2010 at 2:15 PM, Konstantin Shvachko <shv.hadoop@gmail.com
>>> wrote:
>>
>>>> I think the idea is to have issue/design-level discussion on jira  
>>>> and
>>> code-level discussion on RB.
>>>
>>> This is exactly my question. Why do we need two places to have the  
>>> same
>>> discussion?
>>> Even if the comments are automatically replicated in jira and/or  
>>> vice
>>> versa.
>>> Why not choose one and stick with it?
>>>
>>> Thanks,
>>> --Konst
>>>
>>> On Wed, Nov 3, 2010 at 10:56 AM, Eli Collins <eli@cloudera.com>  
>>> wrote:
>>>
>>>> Hey Cos,
>>>>
>>>> I think the setup used for Hive and HBase posts the RB comments  
>>>> to the
>>>> jira which makes it easier to follow.
>>>>
>>>> RB isn't a second system for tracking issues, jira isn't a code  
>>>> review
>>>> system, and RB isn't an issue tracker. I think the idea is to have
>>>> issue/design-level discussion on jira and code-level discussion  
>>>> on RB.
>>>>
>>>> Thanks,
>>>> Eli
>>>>
>>>> On Tue, Nov 2, 2010 at 10:43 PM, Konstantin Shvachko
>>>> <shv.hadoop@gmail.com> wrote:
>>>>> It looks very hard to follow.
>>>>> I have really hard time matching jira and review board comments.
>>>>> Especially when they interleave.
>>>>> Why again do we need a second system for tracking issues?
>>>>> --Konstantin
>>>>>
>>>>> On Tue, Oct 26, 2010 at 3:49 PM, Arun C Murthy <acm@yahoo-inc.com>
>>>> wrote:
>>>>>
>>>>>> +1
>>>>>>
>>>>>> On Oct 26, 2010, at 2:50 PM, Dhruba Borthakur wrote:
>>>>>>
>>>>>> I saw an ASF announcement that there's now a review board  
>>>>>> instance
>>>>>>> available, it will be nice if we can try it out.
>>>>>>>
>>>>>>>
>>>>
>> https://blogs.apache.org/infra/entry/reviewboard_instance_running_at_the
>>>>>>>
>>>>>>> I filed the following JIRA to see if we can use review board to
>>> review
>>>>>>> Hadoop JIRAS.
>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/INFRA-3108
>>>>>>>
>>>>>>> thanks,
>>>>>>> dhruba
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>


From general-return-2333-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 04 11:58:57 2010
Return-Path: <general-return-2333-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 65334 invoked from network); 4 Nov 2010 11:58:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Nov 2010 11:58:56 -0000
Received: (qmail 31691 invoked by uid 500); 4 Nov 2010 11:59:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31461 invoked by uid 500); 4 Nov 2010 11:59:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31450 invoked by uid 99); 4 Nov 2010 11:59:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Nov 2010 11:59:20 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Nov 2010 11:59:14 +0000
Received: from EGL-EX07CAS03.ds.corp.yahoo.com (egl-ex07cas03.eglbp.corp.yahoo.com [203.83.248.219])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id oA4BwQlf025502
	for <general@hadoop.apache.org>; Thu, 4 Nov 2010 04:58:27 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1288871907;
	bh=OBbtINSufch1Y10Pt8oDxRKG6mI528JMC97GV8gbRWg=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 MIME-Version;
	b=A8bAjudRqGdqZc2LySuezpGU44xNgbcU7S9QJ742CXc8/sd6K4FXS09vIAVDIOYrP
	 AyARdD6rzDdP+Gn5l33g6wO/3Lrxjyg0/aOCNcMScvFs7iKSKgRM09W2b3/5Y1LzgG
	 3IsKKqoAFizpl1vFwZsC1Lm4bffFxPOX2FXWyxto=
Received: from EGL-EX07VS02.ds.corp.yahoo.com ([203.83.248.206]) by
 EGL-EX07CAS03.ds.corp.yahoo.com ([203.83.248.219]) with mapi; Thu, 4 Nov 2010
 17:28:26 +0530
From: Venkatesh S <svenkat@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 4 Nov 2010 17:28:11 +0530
Subject: Re: web-based file transfer
Thread-Topic: web-based file transfer
Thread-Index: Act6u2HNx5BSRl6vQtil6xKpsp646gBXCutS
Message-ID: <C8F89DAB.477AA%svenkat@yahoo-inc.com>
In-Reply-To: <033e01cb7abb$623b7d50$26b277f0$@com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C8F89DAB477AAsvenkatyahooinccom_"
MIME-Version: 1.0

--_000_C8F89DAB477AAsvenkatyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

There is also HDFS Proxy in the contrib that only does Listing & Stream fil=
e now  over HTTP. But we are very close to getting the next version of Prox=
y with R+W (full FS access).

Venkatesh


On 11/2/10 11:55 PM, "Mark Laffoon" <mlaffoon@semanticresearch.com> wrote:

We want to enable our web-based client (i.e. browser client, java applet,
whatever?) to transfer files into a system backed by hdfs. The obvious
simple solution is to do http file uploads, then copy the file to hdfs. I
was wondering if there is a way to do it with an hdfs-enabled applet where
the server gives the client the necessary hadoop configuration
information, and the client applet pushes the data directly into hdfs.



Has anybody done this or something similar? Can you give me a starting
point (I'm about to go wander through the hadoop CLI code to get ideas).



Thanks,

Mark



--_000_C8F89DAB477AAsvenkatyahooinccom_--

From general-return-2334-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 04 19:39:40 2010
Return-Path: <general-return-2334-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34279 invoked from network); 4 Nov 2010 19:39:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Nov 2010 19:39:39 -0000
Received: (qmail 45421 invoked by uid 500); 4 Nov 2010 19:40:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44227 invoked by uid 500); 4 Nov 2010 19:40:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44208 invoked by uid 99); 4 Nov 2010 19:40:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Nov 2010 19:40:03 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of akimball83@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Nov 2010 19:39:56 +0000
Received: by bwz19 with SMTP id 19so2097257bwz.35
        for <multiple recipients>; Thu, 04 Nov 2010 12:39:36 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:from:date
         :message-id:subject:to:content-type;
        bh=VlD7+xAGT5ar4WTjV/KBElxfXiO8Z1KuYhKtvUCplnI=;
        b=fpOeZI01JnV2Tp1gaDF7ECMDkPniZQ/8LQrXTM+TM+coAITlkEYagTmz3rJCPPXQ+P
         Q0zva0KcLKkjLDoYxwfPkoN4UZFVXDE/wtRtjM8aNsChBGz8cdrppSxGQyhl2T0fccOG
         ATkwcIIchyje4Tg9sk41oc9iBQgazzWhPAZoI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type;
        b=hF99Kt08wlDZC5h9IJspMty0EPctKOhN6kkGZXHsOeoa18hCCpKXrHGD57mqCIqdAy
         esH9jT/aIas/UJAP5sl2bzk1Gm85t5z7SJhBRQSp54LMmkZGb3YE+DdlL1z/Ek13VBEB
         Rq2uu3GJR/uNQncYmMrC5xFbuzAmFh8O3grtU=
Received: by 10.204.104.5 with SMTP id m5mr1034385bko.47.1288899576449; Thu,
 04 Nov 2010 12:39:36 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.76.193 with HTTP; Thu, 4 Nov 2010 12:39:15 -0700 (PDT)
From: Aaron Kimball <akimball83@gmail.com>
Date: Thu, 4 Nov 2010 12:39:15 -0700
Message-ID: <AANLkTikHtc03HN+FkWOckOh87PPxq1NxePnV2LtYT7pa@mail.gmail.com>
Subject: San Francisco Hadoop meetup
To: general@hadoop.apache.org, core-user@hadoop.apache.org, 
	user@pig.apache.org, user@hive.apache.org
Content-Type: multipart/alternative; boundary=0016e6d7e13d67cce004943f5111
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d7e13d67cce004943f5111
Content-Type: text/plain; charset=ISO-8859-1

Hello Hadoop fans,

The Bay Area Hadoop User Group meetups are a long hike for those of us who
come from San Francisco. I'd like to gauge interest in SF-centric Hadoop
gatherings.

In contrast to the presentation-based format of the usual HUG meetings, I'm
interested in holding events that are more discussion-based: an environment
facilitating people who are working on similar problems to share tips for
handling common scenarios and identify common goals for development. Of
course, if you have a relevant presentation that would benefit the group as
a whole, such offers are gladly accepted.

All industries welcome. If you'd like to come, I'd encourage you to think of
a two minute summary about what you're working on with Hadoop, areas you're
interested in learning more about, and/or challenges that you're facing that
might be addressed by a larger Hadoop community. We can build the discussion
from there, and may break into smaller groups. All are welcome (developers,
users, or managers; experienced or newbies). Though I'd like to hold this in
San Francisco, folks from outside the city are of course welcome too.

If you're interested in joining us, please fill out the following:
* I've created a short survey to help understand days / times that would
work for the most people:  http://bit.ly/ajK26U
* Please also join the meetup group at http://meetup.com/hadoopsf -- We'll
use this to plan the event, RSVP information, etc.

I'm looking forward to meeting more of you!
- Aaron Kimball

--0016e6d7e13d67cce004943f5111--

From general-return-2335-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 05 01:47:54 2010
Return-Path: <general-return-2335-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 41992 invoked from network); 5 Nov 2010 01:47:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Nov 2010 01:47:54 -0000
Received: (qmail 10679 invoked by uid 500); 5 Nov 2010 01:48:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 10602 invoked by uid 500); 5 Nov 2010 01:48:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 10594 invoked by uid 99); 5 Nov 2010 01:48:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Nov 2010 01:48:23 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jeba.ride@gmail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Nov 2010 01:48:17 +0000
Received: by iwn7 with SMTP id 7so2193687iwn.35
        for <general@hadoop.apache.org>; Thu, 04 Nov 2010 18:47:57 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=k+RlUtzy4Jly086BCX0iJZ4UKT783vuN0PYtSSqGtZQ=;
        b=VmBLTnm93GjzCfU0OMkppK9y+cABKmnjj9+9rZKl1Ow0pdCzUehjl6/hqlv91LBs7v
         /P1OFKmaOxacIJoaJlr9qGcYGHkIUbQwsVK5hi8Gb7i7yuRMMrU1DsT8317qsHTfGwkY
         O2Hja4pnuy/U0152yLpdtP9Xg4ToV/MpByTSk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=PQb7SXOukCvpH8LGutAr2mNNjv60dQ1M+PS38uVf8c2rBlD/GMtbTrU+jdBqYrk6ck
         1fC6QbiO9Tdc3bd1UWrY8WoiV3537UhHrH24wNdkJj88eEfa8ejM1wGoygQDaOu5icDc
         v9ViR772qn4po4rjvr4RdapetKc4wemh5OHa4=
MIME-Version: 1.0
Received: by 10.42.204.67 with SMTP id fl3mr943522icb.406.1288921677464; Thu,
 04 Nov 2010 18:47:57 -0700 (PDT)
Received: by 10.231.30.70 with HTTP; Thu, 4 Nov 2010 18:47:57 -0700 (PDT)
Date: Fri, 5 Nov 2010 07:17:57 +0530
Message-ID: <AANLkTim-XhCQLLgzT6w3m=Q4VHvjDV=j+mQ9331TPeOP@mail.gmail.com>
Subject: Keeping Different folders for Different Hadoop Datanodes
From: Clement Jebakumar <jeba.ride@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf303ea25eba82e004944476ca

--20cf303ea25eba82e004944476ca
Content-Type: text/plain; charset=ISO-8859-1

Hello,

Is there a way to specify different folder paths to different datanode
instances in Hadoop?
I need to attach a another harddisk to a PC and want to use it for hadoop.
Is it possible?

*Clement Jebakumar,*
111/27 Keelamutharamman Kovil Street,
Tenkasi, 627 811
http://www.declum.com/clement.html

--20cf303ea25eba82e004944476ca--

From general-return-2336-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 05 04:39:10 2010
Return-Path: <general-return-2336-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5037 invoked from network); 5 Nov 2010 04:39:10 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Nov 2010 04:39:10 -0000
Received: (qmail 4349 invoked by uid 500); 5 Nov 2010 04:39:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4183 invoked by uid 500); 5 Nov 2010 04:39:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4175 invoked by uid 99); 5 Nov 2010 04:39:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Nov 2010 04:39:36 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Nov 2010 04:39:32 +0000
Received: by fxm7 with SMTP id 7so2159303fxm.35
        for <general@hadoop.apache.org>; Thu, 04 Nov 2010 21:39:11 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type;
        bh=RyvQqmm1s3QhMvEt/37tbTGaWOSYnFjiPDgUcGJ5Z3k=;
        b=JcO4SvYh8jlni4CgtRRMIXbC8MmjlIuMee52SozyFn5MX63JhbyHs99ZqfYy5KDD0w
         KVopHCvhNTRWzPCaLRn0GdCKfHIdTm1yARrdCb5HQDV0+LNun/VHVHfeB5yg7ETHupwW
         rcQi9USRTjjnxL5gDnHR0a5HT2ZcTcrkBWRz0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=DGkmlPliKGBoj4amr+G81Ld5P2yKZ4CT4o6dwiOo7Wo3Q2NexFY2BN2XulM9Se0uTs
         ldrVOYiqQfRhrWE9Pirl4VfbKBrSK02aRNfuzyNql90uD9zu5m9a27nXH5xdsmcktWFT
         HAeisPhXmlzH2nYWdBMkdXFA6U98W5Bzg+H6A=
Received: by 10.223.119.132 with SMTP id z4mr521976faq.31.1288931951072; Thu,
 04 Nov 2010 21:39:11 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.223.120.80 with HTTP; Thu, 4 Nov 2010 21:38:50 -0700 (PDT)
In-Reply-To: <AANLkTim-XhCQLLgzT6w3m=Q4VHvjDV=j+mQ9331TPeOP@mail.gmail.com>
References: <AANLkTim-XhCQLLgzT6w3m=Q4VHvjDV=j+mQ9331TPeOP@mail.gmail.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Fri, 5 Nov 2010 10:08:50 +0530
Message-ID: <AANLkTi=ksteH1wqu_eKo3ccOBRZqBvcC767tfQwh30y-@mail.gmail.com>
Subject: Re: Keeping Different folders for Different Hadoop Datanodes
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hi,

On Fri, Nov 5, 2010 at 7:17 AM, Clement Jebakumar <jeba.ride@gmail.com> wrote:
> Hello,
>
> Is there a way to specify different folder paths to different datanode
> instances in Hadoop?

The property "dfs.data.dir" can be a comma separated list of paths,
enabling a DataNode to use more than one mount point for itself.

To run multiple DataNode instances itself on a single machine (which
isn't a good idea in production), see this discussion:
http://search-hadoop.com/m/sApJY1zWgQV

> I need to attach a another harddisk to a PC and want to use it for hadoop.
> Is it possible?

Yes, just modify the "dfs.data.dir" property to reflect the new mount
point location and restart your DataNode. Copy a selection of data
already residing on your HDFS, to balance things between the disks
(you can delete the older copy of those files after the operation).

>
> *Clement Jebakumar,*
> 111/27 Keelamutharamman Kovil Street,
> Tenkasi, 627 811
> http://www.declum.com/clement.html
>



-- 
Harsh J
www.harshj.com

From general-return-2337-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 05 14:29:44 2010
Return-Path: <general-return-2337-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43111 invoked from network); 5 Nov 2010 14:29:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Nov 2010 14:29:44 -0000
Received: (qmail 16834 invoked by uid 500); 5 Nov 2010 14:30:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15251 invoked by uid 500); 5 Nov 2010 14:30:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15223 invoked by uid 99); 5 Nov 2010 14:30:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Nov 2010 14:30:06 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.62.24] (HELO qmta02.westchester.pa.mail.comcast.net) (76.96.62.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Nov 2010 14:29:57 +0000
Received: from omta11.westchester.pa.mail.comcast.net ([76.96.62.36])
	by qmta02.westchester.pa.mail.comcast.net with comcast
	id TNxT1f0090mv7h052SVdUP; Fri, 05 Nov 2010 14:29:37 +0000
Received: from [10.20.90.238] ([209.131.62.115])
	by omta11.westchester.pa.mail.comcast.net with comcast
	id TSVM1f0012VBGtd3XSVRx8; Fri, 05 Nov 2010 14:29:35 +0000
Message-Id: <4E8C27CF-9715-453C-A17F-EA7ED2C19DF2@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=Apple-Mail-4--724969474
Subject: A discussion of a new Powered By Hadoop logo
Mime-Version: 1.0 (Apple Message framework v936)
Date: Fri, 5 Nov 2010 07:29:19 -0700
X-Mailer: Apple Mail (2.936)

--Apple-Mail-4--724969474
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit

All,
    There is a discussion about creating a new "Powered by Hadoop"  
logo that is happening on Jira at https://issues.apache.org/jira/browse/HADOOP-7020 
  if anyone has any graphical talent. I prefer a version based on  
Apache's logo. Here is an example from the Tomcat project.

https://svn.apache.org/repos/asf/tomcat/sandbox/comet/webapps/ROOT/tomcat-power.gif
http://tomcat.apache.org/images/tomcat.gif

-- Owen
--Apple-Mail-4--724969474--

From general-return-2338-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 05 18:34:49 2010
Return-Path: <general-return-2338-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14272 invoked from network); 5 Nov 2010 18:34:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Nov 2010 18:34:48 -0000
Received: (qmail 33762 invoked by uid 500); 5 Nov 2010 18:35:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33264 invoked by uid 500); 5 Nov 2010 18:35:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32957 invoked by uid 99); 5 Nov 2010 18:35:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Nov 2010 18:35:18 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of anthony.urso@gmail.com designates 209.85.161.176 as permitted sender)
Received: from [209.85.161.176] (HELO mail-gx0-f176.google.com) (209.85.161.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Nov 2010 18:35:12 +0000
Received: by gxk4 with SMTP id 4so1092011gxk.35
        for <multiple recipients>; Fri, 05 Nov 2010 11:34:51 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=Co4SpVNYeKbAxswhrkN2tEqFggVraoBzG2NmcBbYmTk=;
        b=BbTV8R7mYXszV5XzXfjVrdlKIBMy/vqka6MV8ojOQbjnzQH0DOluWsQfpBrB9PnsyX
         PMVaTSOqPB3jjMh1Qjm5bfoBzLzRRZHTZpl22H8Ta2wHv1L7iH+q6bD3OhG69fQrj8Kd
         Y7w8sKZJhSvZgdA/B0Ct0qXVvVz5eewZgIuck=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=PCobxXsxTwJ7CmVdLHgO70QqVoutxfkK1bIi9O3pr2qz8geKui1NyPnFVgR1uQkGT1
         LqOFxOfqm2MxUqdpkCZ4r8mNIxMEJPhuQH2uvCAFm7GMIlRqqOhapqGFSKK3wVEcgJOz
         YZxlsM2qVgsSGfdKvHkux1QNNzmetwtDGvqwc=
MIME-Version: 1.0
Received: by 10.91.27.26 with SMTP id e26mr1727427agj.19.1288982091098; Fri,
 05 Nov 2010 11:34:51 -0700 (PDT)
Received: by 10.90.89.7 with HTTP; Fri, 5 Nov 2010 11:34:51 -0700 (PDT)
Date: Fri, 5 Nov 2010 11:34:51 -0700
Message-ID: <AANLkTin1fTqsD7tcsXiLWNvjamgfDGvwaX+wW=A_m2oN@mail.gmail.com>
Subject: Announcing Sizzle, a compiler and runtime for the Sawzall language
From: Anthony Urso <anthony.urso@gmail.com>
To: general@hadoop.apache.org, mapreduce-user@hadoop.apache.org, 
	mapreduce-dev@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

I am pleased to announce the v0.0 release of Sizzle, a compiler and
runtime for the Sawzall language. Sizzle targets Hadoop directly, by
compiling Sawzall programs into Hadoop job jars that can be run
anywhere Hadoop is installed, without requiring a Sawzall interpreter
to also be present.

Although the current release should be considered developer preview
release, it is currently possible to run non-trivial Sawzall programs
on a 0.21 Hadoop cluster or your local desktop.

More information here:

https://github.com/anthonyu/Sizzle/wiki

and all code is available from:

https://github.com/anthonyu/Sizzle

Please try it out, and let me know any problems you experience via
github issues, @SizzleLanguage, or this email address.

Cheers,
Anthony

From general-return-2339-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 05 20:08:57 2010
Return-Path: <general-return-2339-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85965 invoked from network); 5 Nov 2010 20:08:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Nov 2010 20:08:56 -0000
Received: (qmail 42563 invoked by uid 500); 5 Nov 2010 20:09:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42498 invoked by uid 500); 5 Nov 2010 20:09:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42490 invoked by uid 99); 5 Nov 2010 20:09:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Nov 2010 20:09:25 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Nov 2010 20:09:17 +0000
Received: by wwb17 with SMTP id 17so1605055wwb.29
        for <general@hadoop.apache.org>; Fri, 05 Nov 2010 13:08:57 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.161.17 with SMTP id v17mr1605437wek.1.1288987736752; Fri,
 05 Nov 2010 13:08:56 -0700 (PDT)
Received: by 10.216.239.200 with HTTP; Fri, 5 Nov 2010 13:08:56 -0700 (PDT)
In-Reply-To: <4E8C27CF-9715-453C-A17F-EA7ED2C19DF2@apache.org>
References: <4E8C27CF-9715-453C-A17F-EA7ED2C19DF2@apache.org>
Date: Sat, 6 Nov 2010 07:08:56 +1100
Message-ID: <AANLkTimus_h0606qs1jespCGUVKxxL4yy2LBxqrwkbeQ@mail.gmail.com>
Subject: Re: A discussion of a new Powered By Hadoop logo
From: Ian Holsman <hadoop@holsman.net>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016367fb44f2b8492049453d8b9
X-Virus-Checked: Checked by ClamAV on apache.org

--0016367fb44f2b8492049453d8b9
Content-Type: text/plain; charset=ISO-8859-1

What other projects usually do is announce a contest, and let the user base
decide.
shall we start one up?


On Sat, Nov 6, 2010 at 1:29 AM, Owen O'Malley <omalley@apache.org> wrote:

> All,
>   There is a discussion about creating a new "Powered by Hadoop" logo that
> is happening on Jira at https://issues.apache.org/jira/browse/HADOOP-7020if anyone has any graphical talent. I prefer a version based on Apache's
> logo. Here is an example from the Tomcat project.
>
>
> https://svn.apache.org/repos/asf/tomcat/sandbox/comet/webapps/ROOT/tomcat-power.gif
> http://tomcat.apache.org/images/tomcat.gif
>
> -- Owen

--0016367fb44f2b8492049453d8b9--

From general-return-2340-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 05 20:15:29 2010
Return-Path: <general-return-2340-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 89373 invoked from network); 5 Nov 2010 20:15:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Nov 2010 20:15:28 -0000
Received: (qmail 53362 invoked by uid 500); 5 Nov 2010 20:15:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53295 invoked by uid 500); 5 Nov 2010 20:15:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53287 invoked by uid 99); 5 Nov 2010 20:15:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Nov 2010 20:15:58 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of wbsrvc@gmail.com designates 209.85.216.169 as permitted sender)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Nov 2010 20:15:53 +0000
Received: by qyk5 with SMTP id 5so2885716qyk.14
        for <general@hadoop.apache.org>; Fri, 05 Nov 2010 13:15:33 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=IxWy9mjFGnR4XCqrB+WOx7JOiL7KG0QedcBupB1EIlI=;
        b=ZjaiP6cQHZx6+Dy6b+JKRMZTztQlqsdmwPFFFtC+Mm8dt8cHh8zDUafOPQv0buM7Dh
         rBZRM3ScRc8b8psS0L7WzZM8tduxur97AFogOPzlPnaN1yIjzO6fQtcLO/Adgl+LqQ+p
         Ryp75jgE75yh1B0ntV2j5H/NpPR8gDru6cMjo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=jyGVvUdsNsIoJw/psEypcggUvWEU8XO6IQ8PN4hKJoL0QnOCD1YhtetMQtdY8D00Sl
         m6Q/9nFFUEFxrQnRh8u1yEE0fGbRap7foJDaOaL9BxzT5dChmj/ZQQsgpfbWbRFYAa4z
         XlrRVjOW02okTHAVlA8r7AZu2hXki0K3IF5XM=
MIME-Version: 1.0
Received: by 10.224.173.26 with SMTP id n26mr1321181qaz.360.1288988132852;
 Fri, 05 Nov 2010 13:15:32 -0700 (PDT)
Received: by 10.229.33.83 with HTTP; Fri, 5 Nov 2010 13:15:32 -0700 (PDT)
In-Reply-To: <AANLkTimus_h0606qs1jespCGUVKxxL4yy2LBxqrwkbeQ@mail.gmail.com>
References: <4E8C27CF-9715-453C-A17F-EA7ED2C19DF2@apache.org>
	<AANLkTimus_h0606qs1jespCGUVKxxL4yy2LBxqrwkbeQ@mail.gmail.com>
Date: Fri, 5 Nov 2010 13:15:32 -0700
Message-ID: <AANLkTim925vsYgrDJdur_72=u2q7RAbU80MWY7PT6xDV@mail.gmail.com>
Subject: Re: A discussion of a new Powered By Hadoop logo
From: web service <wbsrvc@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000325573576c7456e049453efcc

--000325573576c7456e049453efcc
Content-Type: text/plain; charset=ISO-8859-1

I think Logo that has been suggested/posted somehow doesn't mean or seem
more like the hadoop elephant, the helmet is overpowerering. Again, that is
my perception.

So a contest would be a good idea, with so much discussion already
happening.


On Fri, Nov 5, 2010 at 1:08 PM, Ian Holsman <hadoop@holsman.net> wrote:

> What other projects usually do is announce a contest, and let the user base
> decide.
> shall we start one up?
>
>
> On Sat, Nov 6, 2010 at 1:29 AM, Owen O'Malley <omalley@apache.org> wrote:
>
> > All,
> >   There is a discussion about creating a new "Powered by Hadoop" logo
> that
> > is happening on Jira at
> https://issues.apache.org/jira/browse/HADOOP-7020if anyone has any
> graphical talent. I prefer a version based on Apache's
> > logo. Here is an example from the Tomcat project.
> >
> >
> >
> https://svn.apache.org/repos/asf/tomcat/sandbox/comet/webapps/ROOT/tomcat-power.gif
> > http://tomcat.apache.org/images/tomcat.gif
> >
> > -- Owen
>

--000325573576c7456e049453efcc--

From general-return-2341-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 05 21:34:43 2010
Return-Path: <general-return-2341-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34832 invoked from network); 5 Nov 2010 21:34:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Nov 2010 21:34:42 -0000
Received: (qmail 93427 invoked by uid 500); 5 Nov 2010 21:35:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93345 invoked by uid 500); 5 Nov 2010 21:35:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93337 invoked by uid 99); 5 Nov 2010 21:35:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Nov 2010 21:35:12 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [64.186.241.227] (HELO sri01.semanticresearch.local) (64.186.241.227)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 05 Nov 2010 21:35:08 +0000
Received: from localhost (sri01.semanticresearch.local [127.0.0.1])
	by sri01.semanticresearch.local (Postfix) with ESMTP id 26CB13060002
	for <general@hadoop.apache.org>; Fri,  5 Nov 2010 14:34:34 -0700 (PDT)
X-Virus-Scanned: amavisd-new at sri01.semanticresearch.local
Received: from sri01.semanticresearch.local ([127.0.0.1])
	by localhost (sri01.semanticresearch.local [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id mG8r+jge9ILL for <general@hadoop.apache.org>;
	Fri,  5 Nov 2010 14:34:30 -0700 (PDT)
Received: from sri01.semanticresearch.local (sri01.semanticresearch.local [127.0.0.1])
	by sri01.semanticresearch.local (Postfix) with ESMTP id 5DE2E3060001
	for <general@hadoop.apache.org>; Fri,  5 Nov 2010 14:34:30 -0700 (PDT)
From: "Mark Laffoon" <mlaffoon@semanticresearch.com>
To: <general@hadoop.apache.org>
References: <033e01cb7abb$623b7d50$26b277f0$@com><AANLkTi=BPo3g3zVD2P3qjW_65sh-O6js8mrGup-E4oPf@mail.gmail.com><E3F1EA91-FEB2-4C33-B00E-122D6F1EF1CF@holsman.net> <AANLkTin-PkJvFy5R-wXwMzcQ50xZVkgbGKRU_6UA0GXg@mail.gmail.com> <219D8244D980254ABF28AB469AD4E98F0346A152@VF-MBX13.internal.vodafone.com>
In-Reply-To: <219D8244D980254ABF28AB469AD4E98F0346A152@VF-MBX13.internal.vodafone.com>
Subject: RE: web-based file transfer
Date: Fri, 5 Nov 2010 14:34:30 -0700 (PDT)
Message-ID: <056e01cb7d31$3f54e970$bdfebc50$@com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Mailer: Microsoft Office Outlook 12.0
X-Mailer: Zimbra 5.0.13_GA_2791.RHEL5 (ZimbraConnectorForOutlook/5.0.2893.13)
Thread-Index: Act7cP6h/AoLuUpGTa+NPpITyj0u4wAOt7oDAGE/eeA=
Content-Language: en-us
X-Originating-IP: [172.27.16.186]

Robert,

What are the performance characteristics like for the webdav solution? I
ask for two reasons: since it is implemented over tcp it probably isn't
much faster than http fileupload; I've had previous experience with webdav
(on top of object stores) and we found the protocol to be very "chatty".

Since our use-case is fairly simple (just need to transfer lots of files
from lots of clients; navigating the results isn't necessary), will the
webdav solution be too much?

Comments?

Thanks!

-----Original Message-----
From: Gibbon, Robert, VF-Group [mailto:Robert.Gibbon@vodafone.com] 
Sent: Wednesday, November 03, 2010 4:20 PM
To: general@hadoop.apache.org; general@hadoop.apache.org
Subject: RE: web-based file transfer

Check out HDFS over WebDAV

- http://www.hadoop.iponweb.net/Home/hdfs-over-webdav

WebDAV is an HTTP based protocol for accessing remote filesystems.

I'm running an adapted version of this. It runs under Jetty which is
pretty industry standard and is built on Apache JackRabbit which is pretty
production stable too. I lashed together a custom JAAS authentication
module to authenticate it against our user database.

You can mount WebDAV on Linux using FUSE and WDFS, or script sessions with
cadaver on Solaris/Unix without mounting WebDav. It works pretty sweet on
Windows and Apple, too. 

Recent versions of Jetty have built in traffic shaping and QoS features,
although you might get more mileage from HAProxy or a hardware
loadbalancer.

It works pretty sweet as it enforces HDFS permissions (if you have them
enabled). To get Hadoop permission integrity enforced on MapReduce jobs
check out Oozie - it's a job submission proxy which runs under Tomcat
(might work with Jetty too - haven't tried) and can use a custom
ServletFilter for authentication which you can also patch onto your own
user database/directory. 

Then you just need to seal the perimeter of your cluster with Firewall
rules and you're good to go

No more Kerberos!
R

-----Original Message-----
From: Eric Sammer [mailto:esammer@cloudera.com]
Sent: Wed 11/3/2010 5:05 PM
To: general@hadoop.apache.org
Subject: Re: web-based file transfer
 
Something like it, but Chukwa is more similar to Flume. For *files*
one may want something slightly different. For a stream of (data)
events, Chukwa, Flume, or Scribe are appropriate.

On Wed, Nov 3, 2010 at 1:22 AM, Ian Holsman <hadoop@holsman.net> wrote:
> Doesn't chukwa do something like this?
>
> ---
> Ian Holsman - 703 879-3128
>
> I saw the angel in the marble and carved until I set him free --
Michelangelo
>
> On 03/11/2010, at 5:44 AM, Eric Sammer <esammer@cloudera.com> wrote:
>
>> I would recommend against clients pushing data directly to hdfs like
>> this for a few reasons.
>>
>> 1. The HDFS cluster would need to be directly exposed to a public
>> network; you don't want to do this.
>> 2. You'd be applying (presumably) a high concurrent load to HDFS which
>> isn't its strong point.
>>
>> From an architecture point of view, it's much nicer to have a queuing
>> system between the upload and ingestion into HDFS that you can
>> throttle and control, if necessary. This also allows you to isolate
>> the cluster from the outside world. As to not bottleneck on a single
>> writer, you can have uploaded files land in a queue and have multiple
>> competing consumers popping files (or file names upon which to
>> operate) out of the queue and handling the writing in parallel while
>> being able to control the number of workers. If the initial upload is
>> to a shared device like NFS, you can have writers live on multiple
>> boxes and distribute the work.
>>
>> Another option is to consider Flume, but only if you can deal with the
>> fact that it effectively throws away the notion of files and treats
>> their contents as individual events, etc.
>> http://github.com/cloudera/flume.
>>
>> Hope that helps.
>>
>> On Tue, Nov 2, 2010 at 2:25 PM, Mark Laffoon
>> <mlaffoon@semanticresearch.com> wrote:
>>> We want to enable our web-based client (i.e. browser client, java
applet,
>>> whatever?) to transfer files into a system backed by hdfs. The obvious
>>> simple solution is to do http file uploads, then copy the file to
hdfs. I
>>> was wondering if there is a way to do it with an hdfs-enabled applet
where
>>> the server gives the client the necessary hadoop configuration
>>> information, and the client applet pushes the data directly into hdfs.
>>>
>>>
>>>
>>> Has anybody done this or something similar? Can you give me a starting
>>> point (I'm about to go wander through the hadoop CLI code to get
ideas).
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Mark
>>>
>>>
>>
>>
>>
>> --
>> Eric Sammer
>> twitter: esammer
>> data: www.cloudera.com
>



-- 
Eric Sammer
twitter: esammer
data: www.cloudera.com


From general-return-2342-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 05 21:44:41 2010
Return-Path: <general-return-2342-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38767 invoked from network); 5 Nov 2010 21:44:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Nov 2010 21:44:40 -0000
Received: (qmail 6405 invoked by uid 500); 5 Nov 2010 21:45:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6338 invoked by uid 500); 5 Nov 2010 21:45:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6328 invoked by uid 99); 5 Nov 2010 21:45:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Nov 2010 21:45:10 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of lars.george@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Nov 2010 21:45:05 +0000
Received: by fxm7 with SMTP id 7so2833508fxm.35
        for <multiple recipients>; Fri, 05 Nov 2010 14:44:45 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type:content-transfer-encoding;
        bh=IvRnTBlRHrq8cx6rut6gn/HDcMDvfmgaLjihxZVP8hs=;
        b=uq2PK85akoarU6xVKpvWtDBfeWa9sAbJJFm9qytBOoFYpI2Mor41my2tEd9Z6kY1vB
         6gEWUVJteDa/dYjJbS8UbeirWY/s6x60QWFzKAsYrtmvF1PgjqlWNdpPG/XSSGWR18Tp
         u+gLWH4sHsErOnAMpZnxTvV47JR5FocmfW7ow=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=NBQHq0UjphPHIuVN9GruGTxa1UoxOdqtsUf4BlaQr+6KChB2AB86iMuOyM8BOQKGHR
         ty6BDFOxfGxs2+KdX7zevQovvAQTPIaUzpecb9g5lx3/O3eCDTX0xIiOAOvmGj16uL2S
         x3NHmyhKeD9aB/nvvuRROvHNwp89xqDx+Fqwg=
MIME-Version: 1.0
Received: by 10.223.122.133 with SMTP id l5mr1419888far.52.1288993484096; Fri,
 05 Nov 2010 14:44:44 -0700 (PDT)
Received: by 10.223.114.74 with HTTP; Fri, 5 Nov 2010 14:44:44 -0700 (PDT)
Date: Fri, 5 Nov 2010 22:44:44 +0100
Message-ID: <AANLkTi=bvWKNdrTj_qS9x_hP7Lg9uFT-U1OiOiC_Dck=@mail.gmail.com>
Subject: [ANN] Next Munich OpenHUG Meeting
From: Lars George <lars.george@gmail.com>
To: nosql-discussion@googlegroups.com, general@hadoop.apache.org, 
	hbase-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Next Munich OpenHUG Meeting

I am pleased to invite you to our next Munich Open Hadoop User Group Meetin=
g!

Like always we are looking forward to see everyone again and are
welcoming new attendees to join our group. We are enthusiast about all
things related to scalable, distributed storage system. We are not
limiting us to a particular system but appreciate anyone who would
like to share about their experiences.

Please note the change in location for this meeting.

When: Thursday November 25th, 2010 at 4:30pm (open end)
Where: SKYTEC AG, Keltenring 11, D-82041 Oberhaching, M=FCnchen (map can
be found here: http://www.skytecag.com/unternehmen/ueber-uns/locations/anfa=
hrtsplan-muenchen/)

Thanks to Paul Rogalinski and Skytec AG (http://www.skytecag.com/) for
helping organize the event and for providing the infrastructure.

We have quite a few very interesting talks scheduled:

- "Map/Reduce Einf=FChrung mit HelloWorld Beispielen in Java und
stdin/out (Shell-Skripte)" by Paul Rogalinski
- "CouchDB und MongoDB - Pitfalls und Gr=FCnde f=FCr Einsatz im Enterprise
/ J2EE Umfeld" by Marc Weinberger
- "SOLR: Prototypen mit Velocity, dynamischer Index." by Chantal Ackermann
- "Datenmodellierung und Architektur beim Einsatz von
Semantik-Web-Technologien in der Praxis" by Sacha Berger
- "Hadoop and HBase - Ein =DCberblick" by Lars George

As usual this is followed by an open discussion at a nearby
pub/restaurant so that we can also enjoy some food and beers. We do
have a sponsor for that, thanks to Sistrix (http://www.sistrix.de/)!

Please RSVP here:

Xing: http://www.xing.com/events/munich-openhug-meeting-593962

Yahoo Upcoming:
http://upcoming.yahoo.com/event/7279395/BY/Oberhaching/Munich-OpenHUG-Meeti=
ng/Skytec-AG

LinkedIn: http://events.linkedin.com/Munich-OpenHUG-Meeting/pub/477768

Looking forward to seeing you there!

Best regards,
Lars

From general-return-2343-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 05 23:20:03 2010
Return-Path: <general-return-2343-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81220 invoked from network); 5 Nov 2010 23:20:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Nov 2010 23:20:02 -0000
Received: (qmail 8449 invoked by uid 500); 5 Nov 2010 23:20:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8398 invoked by uid 500); 5 Nov 2010 23:20:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8390 invoked by uid 99); 5 Nov 2010 23:20:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Nov 2010 23:20:32 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [195.232.224.74] (HELO mailout05.vodafone.com) (195.232.224.74)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Nov 2010 23:20:25 +0000
Received: from mailint05 (localhost [127.0.0.1])
	by mailout05 (Postfix) with ESMTP id 853E41469E5
	for <general@hadoop.apache.org>; Sat,  6 Nov 2010 00:20:04 +0100 (CET)
Received: from avoexs02.internal.vodafone.com (unknown [145.230.4.135])
	(using TLSv1 with cipher RC4-MD5 (128/128 bits))
	(No client certificate requested)
	by mailint05 (Postfix) with ESMTPS id 6A8411463F7
	for <general@hadoop.apache.org>; Sat,  6 Nov 2010 00:20:04 +0100 (CET)
Received: from VF-MBX13.internal.vodafone.com ([145.230.5.24]) by avoexs02.internal.vodafone.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Sat, 6 Nov 2010 00:20:03 +0100
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/mixed;
	boundary="----_=_NextPart_001_01CB7D3F.F934D29A"
Subject: RE: web-based file transfer
Date: Sat, 6 Nov 2010 00:19:09 +0100
Message-ID: <219D8244D980254ABF28AB469AD4E98F0346A159@VF-MBX13.internal.vodafone.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: <219D8244D980254ABF28AB469AD4E98F0346A159@VF-MBX13.internal.vodafone.com>
Thread-Topic: web-based file transfer
Thread-Index: Act7cP6h/AoLuUpGTa+NPpITyj0u4wAOt7oDAGE/eeAAA79+3A==
References: <033e01cb7abb$623b7d50$26b277f0$@com><AANLkTi=BPo3g3zVD2P3qjW_65sh-O6js8mrGup-E4oPf@mail.gmail.com><E3F1EA91-FEB2-4C33-B00E-122D6F1EF1CF@holsman.net> <AANLkTin-PkJvFy5R-wXwMzcQ50xZVkgbGKRU_6UA0GXg@mail.gmail.com> <219D8244D980254ABF28AB469AD4E98F0346A152@VF-MBX13.internal.vodafone.com> <056e01cb7d31$3f54e970$bdfebc50$@com>
From: "Gibbon, Robert, VF-Group" <Robert.Gibbon@vodafone.com>
To: <general@hadoop.apache.org>
X-OriginalArrivalTime: 05 Nov 2010 23:20:03.0903 (UTC) FILETIME=[F9E35CF0:01CB7D3F]
X-Virus-Checked: Checked by ClamAV on apache.org

------_=_NextPart_001_01CB7D3F.F934D29A
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable



> What are the performance characteristics like for the webdav solution?

The HDFS over WebDAV setup is horizontally scalable, just keep adding =
Jettys and put a round-robin VIP on the front. It is stateless so =
there's no need for sticky session.

It is not especially chatty unless doing complex directory traversals - =
HTTP/Put & HTTP/Get - much the same as most REST implementations in =
fact.

For us, it's more than good enough.



-----Original Message-----
From: Mark Laffoon [mailto:mlaffoon@semanticresearch.com]
Sent: Fri 11/5/2010 10:34 PM
To: general@hadoop.apache.org
Subject: RE: web-based file transfer
=20
Robert,

What are the performance characteristics like for the webdav solution? I
ask for two reasons: since it is implemented over tcp it probably isn't
much faster than http fileupload; I've had previous experience with =
webdav
(on top of object stores) and we found the protocol to be very "chatty".

Since our use-case is fairly simple (just need to transfer lots of files
from lots of clients; navigating the results isn't necessary), will the
webdav solution be too much?

Comments?

Thanks!

-----Original Message-----
From: Gibbon, Robert, VF-Group [mailto:Robert.Gibbon@vodafone.com]=20
Sent: Wednesday, November 03, 2010 4:20 PM
To: general@hadoop.apache.org; general@hadoop.apache.org
Subject: RE: web-based file transfer

Check out HDFS over WebDAV

- http://www.hadoop.iponweb.net/Home/hdfs-over-webdav

WebDAV is an HTTP based protocol for accessing remote filesystems.

I'm running an adapted version of this. It runs under Jetty which is
pretty industry standard and is built on Apache JackRabbit which is =
pretty
production stable too. I lashed together a custom JAAS authentication
module to authenticate it against our user database.

You can mount WebDAV on Linux using FUSE and WDFS, or script sessions =
with
cadaver on Solaris/Unix without mounting WebDav. It works pretty sweet =
on
Windows and Apple, too.=20

Recent versions of Jetty have built in traffic shaping and QoS features,
although you might get more mileage from HAProxy or a hardware
loadbalancer.

It works pretty sweet as it enforces HDFS permissions (if you have them
enabled). To get Hadoop permission integrity enforced on MapReduce jobs
check out Oozie - it's a job submission proxy which runs under Tomcat
(might work with Jetty too - haven't tried) and can use a custom
ServletFilter for authentication which you can also patch onto your own
user database/directory.=20

Then you just need to seal the perimeter of your cluster with Firewall
rules and you're good to go

No more Kerberos!
R

-----Original Message-----
From: Eric Sammer [mailto:esammer@cloudera.com]
Sent: Wed 11/3/2010 5:05 PM
To: general@hadoop.apache.org
Subject: Re: web-based file transfer
=20
Something like it, but Chukwa is more similar to Flume. For *files*
one may want something slightly different. For a stream of (data)
events, Chukwa, Flume, or Scribe are appropriate.

On Wed, Nov 3, 2010 at 1:22 AM, Ian Holsman <hadoop@holsman.net> wrote:
> Doesn't chukwa do something like this?
>
> ---
> Ian Holsman - 703 879-3128
>
> I saw the angel in the marble and carved until I set him free --
Michelangelo
>
> On 03/11/2010, at 5:44 AM, Eric Sammer <esammer@cloudera.com> wrote:
>
>> I would recommend against clients pushing data directly to hdfs like
>> this for a few reasons.
>>
>> 1. The HDFS cluster would need to be directly exposed to a public
>> network; you don't want to do this.
>> 2. You'd be applying (presumably) a high concurrent load to HDFS =
which
>> isn't its strong point.
>>
>> From an architecture point of view, it's much nicer to have a queuing
>> system between the upload and ingestion into HDFS that you can
>> throttle and control, if necessary. This also allows you to isolate
>> the cluster from the outside world. As to not bottleneck on a single
>> writer, you can have uploaded files land in a queue and have multiple
>> competing consumers popping files (or file names upon which to
>> operate) out of the queue and handling the writing in parallel while
>> being able to control the number of workers. If the initial upload is
>> to a shared device like NFS, you can have writers live on multiple
>> boxes and distribute the work.
>>
>> Another option is to consider Flume, but only if you can deal with =
the
>> fact that it effectively throws away the notion of files and treats
>> their contents as individual events, etc.
>> http://github.com/cloudera/flume.
>>
>> Hope that helps.
>>
>> On Tue, Nov 2, 2010 at 2:25 PM, Mark Laffoon
>> <mlaffoon@semanticresearch.com> wrote:
>>> We want to enable our web-based client (i.e. browser client, java
applet,
>>> whatever?) to transfer files into a system backed by hdfs. The =
obvious
>>> simple solution is to do http file uploads, then copy the file to
hdfs. I
>>> was wondering if there is a way to do it with an hdfs-enabled applet
where
>>> the server gives the client the necessary hadoop configuration
>>> information, and the client applet pushes the data directly into =
hdfs.
>>>
>>>
>>>
>>> Has anybody done this or something similar? Can you give me a =
starting
>>> point (I'm about to go wander through the hadoop CLI code to get
ideas).
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Mark
>>>
>>>
>>
>>
>>
>> --
>> Eric Sammer
>> twitter: esammer
>> data: www.cloudera.com
>



--=20
Eric Sammer
twitter: esammer
data: www.cloudera.com



------_=_NextPart_001_01CB7D3F.F934D29A--

From general-return-2344-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 08 14:34:30 2010
Return-Path: <general-return-2344-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38522 invoked from network); 8 Nov 2010 14:34:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Nov 2010 14:34:30 -0000
Received: (qmail 50936 invoked by uid 500); 8 Nov 2010 14:34:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48993 invoked by uid 500); 8 Nov 2010 14:34:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48964 invoked by uid 99); 8 Nov 2010 14:34:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Nov 2010 14:34:52 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of shujamughal@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Nov 2010 14:34:46 +0000
Received: by wyf19 with SMTP id 19so2442367wyf.35
        for <multiple recipients>; Mon, 08 Nov 2010 06:34:26 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=LBQ7BlwLMdf/CVK9k+9XM7smV287ZNw65bRFG/e+BeE=;
        b=NemdiTGkCiBYGPgu6kw1YJWMXZHfSFxVvGUqH0wt7Y5yuXAnxUmKNhOtB7nWo67hD9
         TI6Lq1/QDKZgm5Ry47aLDxxWTIPm92tYQTR5K3VsleeGcK+CwjJJA+apoywdgHyqqL/s
         Jht7KzcJkRRCznk0drAO3Y7CwyoUdtZFTXTR8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=YOQOVE9kWTm+9aUY1aLIfuiYsFlgdwNbeqOHoAnHUd+agiOnpJ8O+UHNLDqxXkugrj
         Oz3FVLVeWi32HGKdo1UtksBsZFrt85u6giXS3OwfNm8k/JaJDHiEGhsTOnqtzxMQpKyo
         /uk2rLkrFRehwuwMUNykYZk4Eo+uW3X70Q/cE=
MIME-Version: 1.0
Received: by 10.216.68.21 with SMTP id k21mr75718wed.107.1289226865374; Mon,
 08 Nov 2010 06:34:25 -0800 (PST)
Received: by 10.216.10.69 with HTTP; Mon, 8 Nov 2010 06:34:25 -0800 (PST)
Date: Mon, 8 Nov 2010 19:34:25 +0500
Message-ID: <AANLkTi=zFtT1_W7Lbqbi84rL99Cd9NQfdEL=wEz4ckp2@mail.gmail.com>
Subject: Configure Ganglia with Hadoop
From: Shuja Rehman <shujamughal@gmail.com>
To: common-user@hadoop.apache.org, user@hbase.apache.org, 
	mapreduce-user@hadoop.apache.org, general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0ce0f7a858779004948b851c
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0ce0f7a858779004948b851c
Content-Type: text/plain; charset=ISO-8859-1

Hi
I have cluster of 4 machines and want to configure ganglia for monitoring
purpose. I have read the wiki and add the following lines to
hadoop-metrics.properties on each machine.

dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
dfs.period=10
dfs.servers=10.10.10.2:8649

mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
mapred.period=10
mapred.servers=10.10.10.2:8649

jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
jvm.period=10
jvm.servers=10.10.10.2:8649

rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext
rpc.period=10
rpc.servers=10.10.10.2:8649


where 10.10.10.2 is the machine where i am running gmeated and web front
end. Will  I need to same ip in all machine as i do here or need to give
machine own ip in each file? and is there anything more to do to setup it
with hadoop?



-- 
Regards
Shuja-ur-Rehman Baig
<http://pk.linkedin.com/in/shujamughal>

--000e0ce0f7a858779004948b851c--

From general-return-2345-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 08 14:42:09 2010
Return-Path: <general-return-2345-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42030 invoked from network); 8 Nov 2010 14:42:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Nov 2010 14:42:09 -0000
Received: (qmail 65432 invoked by uid 500); 8 Nov 2010 14:42:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 64790 invoked by uid 500); 8 Nov 2010 14:42:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 64753 invoked by uid 99); 8 Nov 2010 14:42:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Nov 2010 14:42:33 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=MIME_BASE64_BLANKS,MIME_BASE64_TEXT,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 65.55.88.14 is neither permitted nor denied by domain of jon.creasy@announcemedia.com)
Received: from [65.55.88.14] (HELO TX2EHSOBE009.bigfish.com) (65.55.88.14)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Nov 2010 14:42:27 +0000
Received: from mail17-tx2-R.bigfish.com (10.9.14.250) by
 TX2EHSOBE009.bigfish.com (10.9.40.29) with Microsoft SMTP Server id
 14.1.225.8; Mon, 8 Nov 2010 14:42:06 +0000
Received: from mail17-tx2 (localhost.localdomain [127.0.0.1])	by
 mail17-tx2-R.bigfish.com (Postfix) with ESMTP id A0E1014D84B6;	Mon,  8 Nov
 2010 14:42:06 +0000 (UTC)
X-SpamScore: -19
X-BigFish: VPS-19(zz1432N98dN9371Pzz1202hzz8275bhz2ei2a8h)
X-Forefront-Antispam-Report: KIP:(null);UIP:(null);IPVD:NLI;H:VA3DIAHUB045.RED001.local;RD:smtp801.microsoftonline.com;EFVD:NLI
Received: from mail17-tx2 (localhost.localdomain [127.0.0.1]) by mail17-tx2
 (MessageSwitch) id 1289227326300734_18469; Mon,  8 Nov 2010 14:42:06 +0000
 (UTC)
Received: from TX2EHSMHS018.bigfish.com (unknown [10.9.14.251])	by
 mail17-tx2.bigfish.com (Postfix) with ESMTP id 3C3B9920051;	Mon,  8 Nov 2010
 14:42:06 +0000 (UTC)
Received: from VA3DIAHUB045.RED001.local (65.55.171.153) by
 TX2EHSMHS018.bigfish.com (10.9.99.118) with Microsoft SMTP Server (TLS) id
 14.1.225.8; Mon, 8 Nov 2010 14:42:05 +0000
Received: from VA3DIAXVS401.RED001.local ([10.32.57.17]) by
 VA3DIAHUB045.RED001.local ([10.32.21.119]) with mapi; Mon, 8 Nov 2010
 06:41:53 -0800
From: Jonathan Creasy <jon.creasy@Announcemedia.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>,
	"user@hbase.apache.org" <user@hbase.apache.org>,
	"mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>,
	"general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 8 Nov 2010 06:41:43 -0800
Subject: Re: Configure Ganglia with Hadoop
Thread-Topic: Configure Ganglia with Hadoop
Thread-Index: Act/UxVeIDr0ooh7R0uJqxJI0ECXiQ==
Message-ID: <F19813F8-C245-4E83-890A-FED04C0093D5@Announcemedia.com>
References: <AANLkTi=zFtT1_W7Lbqbi84rL99Cd9NQfdEL=wEz4ckp2@mail.gmail.com>
In-Reply-To: <AANLkTi=zFtT1_W7Lbqbi84rL99Cd9NQfdEL=wEz4ckp2@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-OriginatorOrg: announcemedia.com

SWYgSSByZW1lbWJlciBjb3JyZWN0bHkgdGhlIGRlZmF1bHQgY29uZmlnIHVzZXMgbXVsdGljYXN0
IHNvIHlvdSBzaG9pbGQgY29uZmlndXJlIHRoZW0gYWxsIHdpdGggdGhlIHNhbWUNCm11bHRpY2Fz
dCBpcC4gU29tZSBndWlkZXMgaGF2ZSB5b3Ugc3dhcCB0aGF0IG91dCB3aXRoIFVEUCBzbyBpZiB5
b3UgZGlkIHRoYXQgdGhlbiB5b3Ugc2hvdWxkIGhhdmUgdGhlIGlwIHlvdSBoYXZlIGluIHRoZXJl
LiBUaGlzIGlzIGluIHRoZSAnTGlzdGVuZXInIGFuZCAnU2VuZGVyJyBwYXJ0IG9mIHRoZSBHYW5n
bGlhIGNvbmZpZy4NCg0KVGhlcmUgYXJlIGFkZGl0aW9uYWwgcGFyYW1ldGVycyB3aGljaCBnbyBp
biB0aGUgaGRmcyBjb25maWcgd2hpY2ggdGVsbCBpdCB0byBzZW5kIGRhdGEgdG8gR2FuZ2xpYSB3
aGljaCB3aGlsZSBub3QgcmVxdWlyZWQgd2lsbCBnaXZlIHlvdSBIYWRvb3Agc3BlY2lmaWMgbWV0
cmljcy4NCg0KSSB3b3VsZCBiZSBoYXBweSB0byBzZW5kIG1vcmUgZGV0YWlsIHdoZW4gSSBhcnJp
dmUgYXQgbXkgb2ZmaWNlIGlmIG5vIG9uZSBlbHNlIGhhcyBhbnN3ZXJlZC4NCg0KDQoNCk9uIE5v
diA4LCAyMDEwLCBhdCA4OjM2IEFNLCAiU2h1amEgUmVobWFuIiA8c2h1amFtdWdoYWxAZ21haWwu
Y29tPiB3cm90ZToNCg0KPiBIaQ0KPiBJIGhhdmUgY2x1c3RlciBvZiA0IG1hY2hpbmVzIGFuZCB3
YW50IHRvIGNvbmZpZ3VyZSBnYW5nbGlhIGZvciBtb25pdG9yaW5nDQo+IHB1cnBvc2UuIEkgaGF2
ZSByZWFkIHRoZSB3aWtpIGFuZCBhZGQgdGhlIGZvbGxvd2luZyBsaW5lcyB0bw0KPiBoYWRvb3At
bWV0cmljcy5wcm9wZXJ0aWVzIG9uIGVhY2ggbWFjaGluZS4NCj4gDQo+IGRmcy5jbGFzcz1vcmcu
YXBhY2hlLmhhZG9vcC5tZXRyaWNzLmdhbmdsaWEuR2FuZ2xpYUNvbnRleHQNCj4gZGZzLnBlcmlv
ZD0xMA0KPiBkZnMuc2VydmVycz0xMC4xMC4xMC4yOjg2NDkNCj4gDQo+IG1hcHJlZC5jbGFzcz1v
cmcuYXBhY2hlLmhhZG9vcC5tZXRyaWNzLmdhbmdsaWEuR2FuZ2xpYUNvbnRleHQNCj4gbWFwcmVk
LnBlcmlvZD0xMA0KPiBtYXByZWQuc2VydmVycz0xMC4xMC4xMC4yOjg2NDkNCj4gDQo+IGp2bS5j
bGFzcz1vcmcuYXBhY2hlLmhhZG9vcC5tZXRyaWNzLmdhbmdsaWEuR2FuZ2xpYUNvbnRleHQNCj4g
anZtLnBlcmlvZD0xMA0KPiBqdm0uc2VydmVycz0xMC4xMC4xMC4yOjg2NDkNCj4gDQo+IHJwYy5j
bGFzcz1vcmcuYXBhY2hlLmhhZG9vcC5tZXRyaWNzLmdhbmdsaWEuR2FuZ2xpYUNvbnRleHQNCj4g
cnBjLnBlcmlvZD0xMA0KPiBycGMuc2VydmVycz0xMC4xMC4xMC4yOjg2NDkNCj4gDQo+IA0KPiB3
aGVyZSAxMC4xMC4xMC4yIGlzIHRoZSBtYWNoaW5lIHdoZXJlIGkgYW0gcnVubmluZyBnbWVhdGVk
IGFuZCB3ZWIgZnJvbnQNCj4gZW5kLiBXaWxsICBJIG5lZWQgdG8gc2FtZSBpcCBpbiBhbGwgbWFj
aGluZSBhcyBpIGRvIGhlcmUgb3IgbmVlZCB0byBnaXZlDQo+IG1hY2hpbmUgb3duIGlwIGluIGVh
Y2ggZmlsZT8gYW5kIGlzIHRoZXJlIGFueXRoaW5nIG1vcmUgdG8gZG8gdG8gc2V0dXAgaXQNCj4g
d2l0aCBoYWRvb3A/DQo+IA0KPiANCj4gDQo+IC0tIA0KPiBSZWdhcmRzDQo+IFNodWphLXVyLVJl
aG1hbiBCYWlnDQo+IDxodHRwOi8vcGsubGlua2VkaW4uY29tL2luL3NodWphbXVnaGFsPg0K


From general-return-2346-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 08 14:43:15 2010
Return-Path: <general-return-2346-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42561 invoked from network); 8 Nov 2010 14:43:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Nov 2010 14:43:15 -0000
Received: (qmail 69874 invoked by uid 500); 8 Nov 2010 14:43:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69494 invoked by uid 500); 8 Nov 2010 14:43:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69470 invoked by uid 99); 8 Nov 2010 14:43:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Nov 2010 14:43:42 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=MIME_BASE64_BLANKS,MIME_BASE64_TEXT,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 216.32.180.11 is neither permitted nor denied by domain of jon.creasy@announcemedia.com)
Received: from [216.32.180.11] (HELO VA3EHSOBE001.bigfish.com) (216.32.180.11)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Nov 2010 14:43:35 +0000
Received: from mail94-va3-R.bigfish.com (10.7.14.249) by
 VA3EHSOBE001.bigfish.com (10.7.40.21) with Microsoft SMTP Server id
 14.1.225.8; Mon, 8 Nov 2010 14:43:13 +0000
Received: from mail94-va3 (localhost.localdomain [127.0.0.1])	by
 mail94-va3-R.bigfish.com (Postfix) with ESMTP id 747D0968484;	Mon,  8 Nov
 2010 14:43:13 +0000 (UTC)
X-SpamScore: -19
X-BigFish: VPS-19(zz1432N98dN9371Pzz1202hzz8275bhz2ei2a8h61h)
X-Spam-TCS-SCL: 0:0
X-Forefront-Antispam-Report: KIP:(null);UIP:(null);IPVD:NLI;H:VA3DIAHUB008.RED001.local;RD:smtp801.microsoftonline.com;EFVD:NLI
Received: from mail94-va3 (localhost.localdomain [127.0.0.1]) by mail94-va3
 (MessageSwitch) id 128922739372695_5581; Mon,  8 Nov 2010 14:43:13 +0000
 (UTC)
Received: from VA3EHSMHS030.bigfish.com (unknown [10.7.14.235])	by
 mail94-va3.bigfish.com (Postfix) with ESMTP id E53547E004F;	Mon,  8 Nov 2010
 14:43:12 +0000 (UTC)
Received: from VA3DIAHUB008.RED001.local (65.55.171.153) by
 VA3EHSMHS030.bigfish.com (10.7.99.40) with Microsoft SMTP Server (TLS) id
 14.1.225.8; Mon, 8 Nov 2010 14:43:11 +0000
Received: from VA3DIAXVS401.RED001.local ([10.32.57.17]) by
 VA3DIAHUB008.RED001.local ([10.32.16.179]) with mapi; Mon, 8 Nov 2010
 06:42:54 -0800
From: Jonathan Creasy <jon.creasy@Announcemedia.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>,
	"user@hbase.apache.org" <user@hbase.apache.org>,
	"mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>,
	"general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 8 Nov 2010 06:42:44 -0800
Subject: Re: Configure Ganglia with Hadoop
Thread-Topic: Configure Ganglia with Hadoop
Thread-Index: Act/Uzlr1Wly6DUrRFm63+OE0ZTFlA==
Message-ID: <2025C303-04A3-4041-BCCD-ADE71ED6B178@Announcemedia.com>
References: <AANLkTi=zFtT1_W7Lbqbi84rL99Cd9NQfdEL=wEz4ckp2@mail.gmail.com>
In-Reply-To: <AANLkTi=zFtT1_W7Lbqbi84rL99Cd9NQfdEL=wEz4ckp2@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-OriginatorOrg: announcemedia.com
X-Virus-Checked: Checked by ClamAV on apache.org

QW5kIGluIG15IHplYWwgdG8gYmUgaGVscGZ1bCBJIGRpZG4ndCBwcm9wZXJseSByZWFkIHlvdXIg
cXVlc3Rpb24gc28gbXkgcmVzcG9uc2UgaXMgbW9zdGx5IHVzZWxlc3MuIDopDQoNCg0KDQpPbiBO
b3YgOCwgMjAxMCwgYXQgODozNiBBTSwgIlNodWphIFJlaG1hbiIgPHNodWphbXVnaGFsQGdtYWls
LmNvbT4gd3JvdGU6DQoNCj4gSGkNCj4gSSBoYXZlIGNsdXN0ZXIgb2YgNCBtYWNoaW5lcyBhbmQg
d2FudCB0byBjb25maWd1cmUgZ2FuZ2xpYSBmb3IgbW9uaXRvcmluZw0KPiBwdXJwb3NlLiBJIGhh
dmUgcmVhZCB0aGUgd2lraSBhbmQgYWRkIHRoZSBmb2xsb3dpbmcgbGluZXMgdG8NCj4gaGFkb29w
LW1ldHJpY3MucHJvcGVydGllcyBvbiBlYWNoIG1hY2hpbmUuDQo+IA0KPiBkZnMuY2xhc3M9b3Jn
LmFwYWNoZS5oYWRvb3AubWV0cmljcy5nYW5nbGlhLkdhbmdsaWFDb250ZXh0DQo+IGRmcy5wZXJp
b2Q9MTANCj4gZGZzLnNlcnZlcnM9MTAuMTAuMTAuMjo4NjQ5DQo+IA0KPiBtYXByZWQuY2xhc3M9
b3JnLmFwYWNoZS5oYWRvb3AubWV0cmljcy5nYW5nbGlhLkdhbmdsaWFDb250ZXh0DQo+IG1hcHJl
ZC5wZXJpb2Q9MTANCj4gbWFwcmVkLnNlcnZlcnM9MTAuMTAuMTAuMjo4NjQ5DQo+IA0KPiBqdm0u
Y2xhc3M9b3JnLmFwYWNoZS5oYWRvb3AubWV0cmljcy5nYW5nbGlhLkdhbmdsaWFDb250ZXh0DQo+
IGp2bS5wZXJpb2Q9MTANCj4ganZtLnNlcnZlcnM9MTAuMTAuMTAuMjo4NjQ5DQo+IA0KPiBycGMu
Y2xhc3M9b3JnLmFwYWNoZS5oYWRvb3AubWV0cmljcy5nYW5nbGlhLkdhbmdsaWFDb250ZXh0DQo+
IHJwYy5wZXJpb2Q9MTANCj4gcnBjLnNlcnZlcnM9MTAuMTAuMTAuMjo4NjQ5DQo+IA0KPiANCj4g
d2hlcmUgMTAuMTAuMTAuMiBpcyB0aGUgbWFjaGluZSB3aGVyZSBpIGFtIHJ1bm5pbmcgZ21lYXRl
ZCBhbmQgd2ViIGZyb250DQo+IGVuZC4gV2lsbCAgSSBuZWVkIHRvIHNhbWUgaXAgaW4gYWxsIG1h
Y2hpbmUgYXMgaSBkbyBoZXJlIG9yIG5lZWQgdG8gZ2l2ZQ0KPiBtYWNoaW5lIG93biBpcCBpbiBl
YWNoIGZpbGU/IGFuZCBpcyB0aGVyZSBhbnl0aGluZyBtb3JlIHRvIGRvIHRvIHNldHVwIGl0DQo+
IHdpdGggaGFkb29wPw0KPiANCj4gDQo+IA0KPiAtLSANCj4gUmVnYXJkcw0KPiBTaHVqYS11ci1S
ZWhtYW4gQmFpZw0KPiA8aHR0cDovL3BrLmxpbmtlZGluLmNvbS9pbi9zaHVqYW11Z2hhbD4NCg==


From general-return-2347-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 08 15:31:32 2010
Return-Path: <general-return-2347-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76350 invoked from network); 8 Nov 2010 15:31:31 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Nov 2010 15:31:31 -0000
Received: (qmail 90119 invoked by uid 500); 8 Nov 2010 15:32:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89800 invoked by uid 500); 8 Nov 2010 15:31:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89778 invoked by uid 99); 8 Nov 2010 15:31:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Nov 2010 15:31:58 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=MIME_BASE64_BLANKS,MIME_BASE64_TEXT,RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 216.32.180.16 is neither permitted nor denied by domain of jon.creasy@announcemedia.com)
Received: from [216.32.180.16] (HELO VA3EHSOBE009.bigfish.com) (216.32.180.16)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Nov 2010 15:31:52 +0000
Received: from mail24-va3-R.bigfish.com (10.7.14.240) by
 VA3EHSOBE009.bigfish.com (10.7.40.29) with Microsoft SMTP Server id
 14.1.225.8; Mon, 8 Nov 2010 15:31:31 +0000
Received: from mail24-va3 (localhost.localdomain [127.0.0.1])	by
 mail24-va3-R.bigfish.com (Postfix) with ESMTP id 1A5651128489;	Mon,  8 Nov
 2010 15:30:04 +0000 (UTC)
X-SpamScore: -19
X-BigFish: VPS-19(zz1432N98dN9371Pzz1202hzz8275bhz2ei2a8h61h)
X-Spam-TCS-SCL: 0:0
X-Forefront-Antispam-Report: KIP:(null);UIP:(null);IPVD:NLI;H:VA3DIAHUB044.RED001.local;RD:smtp801.microsoftonline.com;EFVD:NLI
Received: from mail24-va3 (localhost.localdomain [127.0.0.1]) by mail24-va3
 (MessageSwitch) id 1289230203392322_18397; Mon,  8 Nov 2010 15:30:03 +0000
 (UTC)
Received: from VA3EHSMHS016.bigfish.com (unknown [10.7.14.248])	by
 mail24-va3.bigfish.com (Postfix) with ESMTP id 37C071C8804F;	Mon,  8 Nov 2010
 15:30:03 +0000 (UTC)
Received: from VA3DIAHUB044.RED001.local (65.55.171.153) by
 VA3EHSMHS016.bigfish.com (10.7.99.26) with Microsoft SMTP Server (TLS) id
 14.1.225.8; Mon, 8 Nov 2010 15:31:27 +0000
Received: from VA3DIAXVS401.RED001.local ([10.32.57.17]) by
 VA3DIAHUB044.RED001.local ([10.32.21.118]) with mapi; Mon, 8 Nov 2010
 07:31:27 -0800
From: Jonathan Creasy <jon.creasy@Announcemedia.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>,
	"user@hbase.apache.org" <user@hbase.apache.org>,
	"mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>
Date: Mon, 8 Nov 2010 07:31:06 -0800
Subject: Re: Configure Ganglia with Hadoop
Thread-Topic: Configure Ganglia with Hadoop
Thread-Index: Act/WgGnIlB9CTglQhqoWQnr57y9vw==
Message-ID: <8280F561-1AE3-459E-8A94-C20B4CDD739A@announcemedia.microsoftonline.com>
References: <AANLkTi=zFtT1_W7Lbqbi84rL99Cd9NQfdEL=wEz4ckp2@mail.gmail.com>
In-Reply-To: <AANLkTi=zFtT1_W7Lbqbi84rL99Cd9NQfdEL=wEz4ckp2@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-OriginatorOrg: announcemedia.com

VGhpcyBpcyB0aGUgY29ycmVjdCBjb25maWd1cmF0aW9uLCBhbmQgdGhlcmUgc2hvdWxkIGJlIG5v
dGhpbmcgbW9yZSBuZWVkZWQuIEkgZG9uJ3QgdGhpbmsgdGhhdCB0aGVzZSBjb25maWd1cmF0aW9u
IGNoYW5nZXMgd2lsbCB0YWtlIGFmZmVjdCBvbiB0aGUgZmx5IHNvIHlvdSB3b3VsZCBuZWVkIHRv
IHJlc3RhcnQgdGhlIGRhdGFub2RlIGFuZCBuYW1lbm9kZSBwcm9jZXNzZXMgaWYgSSB1bmRlcnN0
YW5kIGNvcnJlY3RseS4gDQoNCldoZW4geW91IGJyb3dzZSB5b3VyIHlvdSB3aWxsIHNlZSBzb21l
IG1vcmUgbWV0cmljczoNCg0KZGZzLkZTRGlyZWN0b3J5LmZpbGVzX2RlbGV0ZWQNCmRmcy5GU05h
bWVzeXN0ZW0uQmxvY2tDYXBhY2l0eQ0KZGZzLkZTTmFtZXN5c3RlbS5CbG9ja3NUb3RhbA0KZGZz
LkZTTmFtZXN5c3RlbS5DYXBhY2l0eVJlbWFpbmluZ0dCDQpkZnMuRlNOYW1lc3lzdGVtLkNhcGFj
aXR5VG90YWxHQg0KZGZzLkZTTmFtZXN5c3RlbS5DYXBhY2l0eVVzZWRHQg0KZGZzLkZTTmFtZXN5
c3RlbS5Db3JydXB0QmxvY2tzDQpkZnMuRlNOYW1lc3lzdGVtLkV4Y2Vzc0Jsb2Nrcw0KZGZzLkZT
TmFtZXN5c3RlbS5GaWxlc1RvdGFsDQpkZnMuRlNOYW1lc3lzdGVtLk1pc3NpbmdCbG9ja3MNCmRm
cy5GU05hbWVzeXN0ZW0uUGVuZGluZ0RlbGV0aW9uQmxvY2tzDQpkZnMuRlNOYW1lc3lzdGVtLlBl
bmRpbmdSZXBsaWNhdGlvbkJsb2Nrcw0KZGZzLkZTTmFtZXN5c3RlbS5TY2hlZHVsZWRSZXBsaWNh
dGlvbkJsb2Nrcw0KZGZzLkZTTmFtZXN5c3RlbS5Ub3RhbExvYWQNCmRmcy5GU05hbWVzeXN0ZW0u
VW5kZXJSZXBsaWNhdGVkQmxvY2tzDQpkZnMuZGF0YW5vZGUuYmxvY2tDaGVja3N1bU9wX2F2Z190
aW1lDQpkZnMuZGF0YW5vZGUuYmxvY2tDaGVja3N1bU9wX251bV9vcHMNCmRmcy5kYXRhbm9kZS5i
bG9ja1JlcG9ydHNfYXZnX3RpbWUNCmRmcy5kYXRhbm9kZS5ibG9ja1JlcG9ydHNfbnVtX29wcw0K
ZGZzLmRhdGFub2RlLmJsb2NrX3ZlcmlmaWNhdGlvbl9mYWlsdXJlcw0KZGZzLmRhdGFub2RlLmJs
b2Nrc19yZWFkDQpkZnMuZGF0YW5vZGUuYmxvY2tzX3JlbW92ZWQNCmRmcy5kYXRhbm9kZS5ibG9j
a3NfcmVwbGljYXRlZA0KZGZzLmRhdGFub2RlLmJsb2Nrc192ZXJpZmllZA0KZGZzLmRhdGFub2Rl
LmJsb2Nrc193cml0dGVuDQpkZnMuZGF0YW5vZGUuYnl0ZXNfcmVhZA0KZGZzLmRhdGFub2RlLmJ5
dGVzX3dyaXR0ZW4NCmRmcy5kYXRhbm9kZS5jb3B5QmxvY2tPcF9hdmdfdGltZQ0KZGZzLmRhdGFu
b2RlLmNvcHlCbG9ja09wX251bV9vcHMNCmRmcy5kYXRhbm9kZS5oZWFydEJlYXRzX2F2Z190aW1l
DQpkZnMuZGF0YW5vZGUuaGVhcnRCZWF0c19udW1fb3BzDQpkZnMuZGF0YW5vZGUucmVhZEJsb2Nr
T3BfYXZnX3RpbWUNCmRmcy5kYXRhbm9kZS5yZWFkQmxvY2tPcF9udW1fb3BzDQpkZnMuZGF0YW5v
ZGUucmVhZE1ldGFkYXRhT3BfYXZnX3RpbWUNCmRmcy5kYXRhbm9kZS5yZWFkTWV0YWRhdGFPcF9u
dW1fb3BzDQpkZnMuZGF0YW5vZGUucmVhZHNfZnJvbV9sb2NhbF9jbGllbnQNCmRmcy5kYXRhbm9k
ZS5yZWFkc19mcm9tX3JlbW90ZV9jbGllbnQNCmRmcy5kYXRhbm9kZS5yZXBsYWNlQmxvY2tPcF9h
dmdfdGltZQ0KZGZzLmRhdGFub2RlLnJlcGxhY2VCbG9ja09wX251bV9vcHMNCmRmcy5kYXRhbm9k
ZS53cml0ZUJsb2NrT3BfYXZnX3RpbWUNCmRmcy5kYXRhbm9kZS53cml0ZUJsb2NrT3BfbnVtX29w
cw0KZGZzLmRhdGFub2RlLndyaXRlc19mcm9tX2xvY2FsX2NsaWVudA0KZGZzLmRhdGFub2RlLndy
aXRlc19mcm9tX3JlbW90ZV9jbGllbnQNCmRmcy5uYW1lbm9kZS5BZGRCbG9ja09wcw0KZGZzLm5h
bWVub2RlLkNyZWF0ZUZpbGVPcHMNCmRmcy5uYW1lbm9kZS5EZWxldGVGaWxlT3BzDQpkZnMubmFt
ZW5vZGUuRmlsZUluZm9PcHMNCmRmcy5uYW1lbm9kZS5GaWxlc0FwcGVuZGVkDQpkZnMubmFtZW5v
ZGUuRmlsZXNDcmVhdGVkDQpkZnMubmFtZW5vZGUuRmlsZXNSZW5hbWVkDQpkZnMubmFtZW5vZGUu
R2V0QmxvY2tMb2NhdGlvbnMNCmRmcy5uYW1lbm9kZS5HZXRMaXN0aW5nT3BzDQpkZnMubmFtZW5v
ZGUuSm91cm5hbFRyYW5zYWN0aW9uc0JhdGNoZWRJblN5bmMNCmRmcy5uYW1lbm9kZS5TYWZlbW9k
ZVRpbWUNCmRmcy5uYW1lbm9kZS5TeW5jc19hdmdfdGltZQ0KZGZzLm5hbWVub2RlLlN5bmNzX251
bV9vcHMNCmRmcy5uYW1lbm9kZS5UcmFuc2FjdGlvbnNfYXZnX3RpbWUNCmRmcy5uYW1lbm9kZS5U
cmFuc2FjdGlvbnNfbnVtX29wcw0KZGZzLm5hbWVub2RlLmJsb2NrUmVwb3J0X2F2Z190aW1lDQpk
ZnMubmFtZW5vZGUuYmxvY2tSZXBvcnRfbnVtX29wcw0KZGZzLm5hbWVub2RlLmZzSW1hZ2VMb2Fk
VGltZQ0KanZtLm1ldHJpY3MuZ2NDb3VudA0KanZtLm1ldHJpY3MuZ2NUaW1lTWlsbGlzDQpqdm0u
bWV0cmljcy5sb2dFcnJvcg0KanZtLm1ldHJpY3MubG9nRmF0YWwNCmp2bS5tZXRyaWNzLmxvZ0lu
Zm8NCmp2bS5tZXRyaWNzLmxvZ1dhcm4NCmp2bS5tZXRyaWNzLm1heE1lbW9yeU0NCmp2bS5tZXRy
aWNzLm1lbUhlYXBDb21taXR0ZWRNDQpqdm0ubWV0cmljcy5tZW1IZWFwVXNlZE0NCmp2bS5tZXRy
aWNzLm1lbU5vbkhlYXBDb21taXR0ZWRNDQpqdm0ubWV0cmljcy5tZW1Ob25IZWFwVXNlZE0NCmp2
bS5tZXRyaWNzLnRocmVhZHNCbG9ja2VkDQpqdm0ubWV0cmljcy50aHJlYWRzTmV3DQpqdm0ubWV0
cmljcy50aHJlYWRzUnVubmFibGUNCmp2bS5tZXRyaWNzLnRocmVhZHNUZXJtaW5hdGVkDQpqdm0u
bWV0cmljcy50aHJlYWRzVGltZWRXYWl0aW5nDQpqdm0ubWV0cmljcy50aHJlYWRzV2FpdGluZw0K
cnBjLm1ldHJpY3MuTnVtT3BlbkNvbm5lY3Rpb25zDQpycGMubWV0cmljcy5ScGNQcm9jZXNzaW5n
VGltZV9hdmdfdGltZQ0KcnBjLm1ldHJpY3MuUnBjUHJvY2Vzc2luZ1RpbWVfbnVtX29wcw0KcnBj
Lm1ldHJpY3MuUnBjUXVldWVUaW1lX2F2Z190aW1lDQpycGMubWV0cmljcy5ScGNRdWV1ZVRpbWVf
bnVtX29wcw0KcnBjLm1ldHJpY3MuYWJhbmRvbkJsb2NrX2F2Z190aW1lDQpycGMubWV0cmljcy5h
YmFuZG9uQmxvY2tfbnVtX29wcw0KcnBjLm1ldHJpY3MuYWRkQmxvY2tfYXZnX3RpbWUNCnJwYy5t
ZXRyaWNzLmFkZEJsb2NrX251bV9vcHMNCnJwYy5tZXRyaWNzLmJsb2NrUmVjZWl2ZWRfYXZnX3Rp
bWUNCnJwYy5tZXRyaWNzLmJsb2NrUmVjZWl2ZWRfbnVtX29wcw0KcnBjLm1ldHJpY3MuYmxvY2tS
ZXBvcnRfYXZnX3RpbWUNCnJwYy5tZXRyaWNzLmJsb2NrUmVwb3J0X251bV9vcHMNCnJwYy5tZXRy
aWNzLmNhbGxRdWV1ZUxlbg0KcnBjLm1ldHJpY3MuY29tcGxldGVfYXZnX3RpbWUNCnJwYy5tZXRy
aWNzLmNvbXBsZXRlX251bV9vcHMNCnJwYy5tZXRyaWNzLmNyZWF0ZV9hdmdfdGltZQ0KcnBjLm1l
dHJpY3MuY3JlYXRlX251bV9vcHMNCnJwYy5tZXRyaWNzLmdldEVkaXRMb2dTaXplX2F2Z190aW1l
DQpycGMubWV0cmljcy5nZXRFZGl0TG9nU2l6ZV9udW1fb3BzDQpycGMubWV0cmljcy5nZXRQcm90
b2NvbFZlcnNpb25fYXZnX3RpbWUNCnJwYy5tZXRyaWNzLmdldFByb3RvY29sVmVyc2lvbl9udW1f
b3BzDQpycGMubWV0cmljcy5yZWdpc3Rlcl9hdmdfdGltZQ0KcnBjLm1ldHJpY3MucmVnaXN0ZXJf
bnVtX29wcw0KcnBjLm1ldHJpY3MucmVuYW1lX2F2Z190aW1lDQpycGMubWV0cmljcy5yZW5hbWVf
bnVtX29wcw0KcnBjLm1ldHJpY3MucmVuZXdMZWFzZV9hdmdfdGltZQ0KcnBjLm1ldHJpY3MucmVu
ZXdMZWFzZV9udW1fb3BzDQpycGMubWV0cmljcy5yb2xsRWRpdExvZ19hdmdfdGltZQ0KcnBjLm1l
dHJpY3Mucm9sbEVkaXRMb2dfbnVtX29wcw0KcnBjLm1ldHJpY3Mucm9sbEZzSW1hZ2VfYXZnX3Rp
bWUNCnJwYy5tZXRyaWNzLnJvbGxGc0ltYWdlX251bV9vcHMNCnJwYy5tZXRyaWNzLnNlbmRIZWFy
dGJlYXRfYXZnX3RpbWUNCnJwYy5tZXRyaWNzLnNlbmRIZWFydGJlYXRfbnVtX29wcw0KcnBjLm1l
dHJpY3MudmVyc2lvblJlcXVlc3RfYXZnX3RpbWUNCnJwYy5tZXRyaWNzLnZlcnNpb25SZXF1ZXN0
X251bV9vcHMNCg0KLUpvbmF0aGFuDQoNCk9uIE5vdiA4LCAyMDEwLCBhdCA4OjM0IEFNLCBTaHVq
YSBSZWhtYW4gd3JvdGU6DQoNCj4gSGkNCj4gSSBoYXZlIGNsdXN0ZXIgb2YgNCBtYWNoaW5lcyBh
bmQgd2FudCB0byBjb25maWd1cmUgZ2FuZ2xpYSBmb3IgbW9uaXRvcmluZw0KPiBwdXJwb3NlLiBJ
IGhhdmUgcmVhZCB0aGUgd2lraSBhbmQgYWRkIHRoZSBmb2xsb3dpbmcgbGluZXMgdG8NCj4gaGFk
b29wLW1ldHJpY3MucHJvcGVydGllcyBvbiBlYWNoIG1hY2hpbmUuDQo+IA0KPiBkZnMuY2xhc3M9
b3JnLmFwYWNoZS5oYWRvb3AubWV0cmljcy5nYW5nbGlhLkdhbmdsaWFDb250ZXh0DQo+IGRmcy5w
ZXJpb2Q9MTANCj4gZGZzLnNlcnZlcnM9MTAuMTAuMTAuMjo4NjQ5DQo+IA0KPiBtYXByZWQuY2xh
c3M9b3JnLmFwYWNoZS5oYWRvb3AubWV0cmljcy5nYW5nbGlhLkdhbmdsaWFDb250ZXh0DQo+IG1h
cHJlZC5wZXJpb2Q9MTANCj4gbWFwcmVkLnNlcnZlcnM9MTAuMTAuMTAuMjo4NjQ5DQo+IA0KPiBq
dm0uY2xhc3M9b3JnLmFwYWNoZS5oYWRvb3AubWV0cmljcy5nYW5nbGlhLkdhbmdsaWFDb250ZXh0
DQo+IGp2bS5wZXJpb2Q9MTANCj4ganZtLnNlcnZlcnM9MTAuMTAuMTAuMjo4NjQ5DQo+IA0KPiBy
cGMuY2xhc3M9b3JnLmFwYWNoZS5oYWRvb3AubWV0cmljcy5nYW5nbGlhLkdhbmdsaWFDb250ZXh0
DQo+IHJwYy5wZXJpb2Q9MTANCj4gcnBjLnNlcnZlcnM9MTAuMTAuMTAuMjo4NjQ5DQo+IA0KPiAN
Cj4gd2hlcmUgMTAuMTAuMTAuMiBpcyB0aGUgbWFjaGluZSB3aGVyZSBpIGFtIHJ1bm5pbmcgZ21l
YXRlZCBhbmQgd2ViIGZyb250DQo+IGVuZC4gV2lsbCAgSSBuZWVkIHRvIHNhbWUgaXAgaW4gYWxs
IG1hY2hpbmUgYXMgaSBkbyBoZXJlIG9yIG5lZWQgdG8gZ2l2ZQ0KPiBtYWNoaW5lIG93biBpcCBp
biBlYWNoIGZpbGU/IGFuZCBpcyB0aGVyZSBhbnl0aGluZyBtb3JlIHRvIGRvIHRvIHNldHVwIGl0
DQo+IHdpdGggaGFkb29wPw0KPiANCj4gDQo+IA0KPiAtLSANCj4gUmVnYXJkcw0KPiBTaHVqYS11
ci1SZWhtYW4gQmFpZw0KPiA8aHR0cDovL3BrLmxpbmtlZGluLmNvbS9pbi9zaHVqYW11Z2hhbD4N
Cg0K


From general-return-2348-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 08 16:39:03 2010
Return-Path: <general-return-2348-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22917 invoked from network); 8 Nov 2010 16:39:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Nov 2010 16:39:03 -0000
Received: (qmail 41124 invoked by uid 500); 8 Nov 2010 16:39:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 41072 invoked by uid 500); 8 Nov 2010 16:39:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 41055 invoked by uid 99); 8 Nov 2010 16:39:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Nov 2010 16:39:32 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,HTML_OBFUSCATE_05_10,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jonathan.seidman@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Nov 2010 16:39:27 +0000
Received: by wyf19 with SMTP id 19so2572221wyf.35
        for <general@hadoop.apache.org>; Mon, 08 Nov 2010 08:39:05 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=+qnQlVqqr2//8dPAxQztQdlkvOcUIcXmOGDXAPVxAl0=;
        b=VerCSdWq74Y/wGK3VvbE6JfX+GmdY+sFXpDi3AhjLYAWO1FAc8HzgbfQgadrLgmCh8
         CJ+oYuCrJxlhzcMHg1HzmCPbSMNOotsu7Px1R2YA+gYK41nxzzRD50mzmFqow6fS+uAW
         G9tIxO6Em8mXRz6CcMKlVvfM2xE3KEIlnGfCw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=GwbAP64fTv0rd91U5/CYL//YMp98Ft86JUwxORRICetYgZ5vF+m2vGhSsmIm9yTHkA
         8Ir3hWKoAzIyzTkANBfOaRjlsm6YPQ7qbil/XObRY225ZbSBsIkCIDPA5hGx4TuNMbyk
         BPn5KzLnqVxQ4TvOBQOBa+mTBU2T55OAPOlC0=
MIME-Version: 1.0
Received: by 10.227.134.206 with SMTP id k14mr5498033wbt.160.1289234344281;
 Mon, 08 Nov 2010 08:39:04 -0800 (PST)
Received: by 10.216.86.9 with HTTP; Mon, 8 Nov 2010 08:39:04 -0800 (PST)
Date: Mon, 8 Nov 2010 10:39:04 -0600
Message-ID: <AANLkTimJLuJX9YukiAid=ueN610hRuDWXraJV8BgKRRW@mail.gmail.com>
Subject: [ANNOUNCE] Two Talks by Jeff Hammerbacher on Nov. 11th at Orbitz
 Worldwide HQ in Chicago
From: Jonathan Seidman <jonathan.seidman@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636831b201f959d04948d4302

--001636831b201f959d04948d4302
Content-Type: text/plain; charset=ISO-8859-1

Orbitz is pleased to welcome Jeff Hammerbacher, Vice President, Products at
Cloudera Inc. for two talks at Orbitz Worldwide headquarters in Chicago on
Thursday, November 11th. Both talks are open to the public:

At 11:30 Jeff will speak on "Experiences Evolving a New Analytical Platform"
as part of the Orbitz IDEAS series. RSVP here: http:/...
/www.meetup.com/orbitzideas/calendar/15258988<http://www.meetup.com/orbitzideas/calendar/15258988/>

In the evening Jeff will be speaking to the Chicago Hadoop User Group. RSVP
for the meeting here: http://www.meetup.com/Chicago-area-Hadoo
p-User-Group-CHUG/calendar/15203244/

Thanks! Hope t see you there.

Jonathan

--001636831b201f959d04948d4302--

From general-return-2349-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 08 18:40:13 2010
Return-Path: <general-return-2349-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95682 invoked from network); 8 Nov 2010 18:40:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Nov 2010 18:40:13 -0000
Received: (qmail 67313 invoked by uid 500); 8 Nov 2010 18:40:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67225 invoked by uid 500); 8 Nov 2010 18:40:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67211 invoked by uid 99); 8 Nov 2010 18:40:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Nov 2010 18:40:42 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of shujamughal@gmail.com designates 209.85.213.176 as permitted sender)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Nov 2010 18:40:38 +0000
Received: by yxn22 with SMTP id 22so4189267yxn.35
        for <multiple recipients>; Mon, 08 Nov 2010 10:40:17 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:cc:content-type;
        bh=fgpaQZV6HVjOMDNiR+huQPaAzCaeYKzQjGMV8BcQdbY=;
        b=kx57pzVN8zB615+5RUAy0op/Jzdy06q50iN6Rr4zgRvQGX2ZNcnmHusceFDd9d2/nM
         1enI1gCqyUSroWoDPS7HahdvKSq8hsLwXENwTQCIb0qbEutxaU7PBG4Ro/xYbjAIHjtY
         q3zdFORsuZ2stsYwBXzweDbx3XoRPenCQdgPc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=QliA1BQszD0PizE7RWEyxzO8URaEALZW9s+Z1jDJ7sJqmMOx2yt249ZpfWBPBGNqlK
         RcLecdzLzgj63e1p4z26BKNGAtSLzc70PMYOlRfDL/EYSUgUwv4GZTTn+MHHphwD+IWT
         mF9MiYZ6fDfPuSEYupGoK98U0PP0pmrW/ZExo=
MIME-Version: 1.0
Received: by 10.216.11.131 with SMTP id 3mr371827wex.92.1289241615528; Mon, 08
 Nov 2010 10:40:15 -0800 (PST)
Received: by 10.216.10.69 with HTTP; Mon, 8 Nov 2010 10:40:14 -0800 (PST)
In-Reply-To: <AANLkTimihLb2sgZWFXpRmutWQR3Py+ZoW64wSW4MH3Un@mail.gmail.com>
References: <AANLkTi=zFtT1_W7Lbqbi84rL99Cd9NQfdEL=wEz4ckp2@mail.gmail.com>
	<8280F561-1AE3-459E-8A94-C20B4CDD739A@announcemedia.microsoftonline.com>
	<AANLkTimihLb2sgZWFXpRmutWQR3Py+ZoW64wSW4MH3Un@mail.gmail.com>
Date: Mon, 8 Nov 2010 23:40:14 +0500
Message-ID: <AANLkTik3y8DEXF7+zHDyMJ5WrEqRN=wQsS=b-_gc7QzR@mail.gmail.com>
Subject: Re: Configure Ganglia with Hadoop
From: Shuja Rehman <shujamughal@gmail.com>
To: common-user@hadoop.apache.org
Cc: "general@hadoop.apache.org" <general@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=0016364c7d9985f67e04948ef40f

--0016364c7d9985f67e04948ef40f
Content-Type: text/plain; charset=ISO-8859-1

Hi
I have follow the article, i have one confusion do i need to change
gmond.config file on each node??

host {
  location = "unspecified"
}

/* Feel free to specify as many udp_send_channels as you like.  Gmond
   used to only support having a single channel */
udp_send_channel {
  mcast_join = 239.2.11.71
  port = 8649
  ttl = 1
}

/* You can specify as many udp_recv_channels as you like as well. */
udp_recv_channel {
  mcast_join = 239.2.11.71
  port = 8649
  bind = 239.2.11.71
}

/* You can specify as many tcp_accept_channels as you like to share
   an xml description of the state of the cluster */
tcp_accept_channel {
  port = 8649
}


and i need to replace the 239.2.11.71 with that specific machine ip e.g
10.10.10.2 in 1st machine case and 10.10.10.3 for 2nd machine and so on?

On Mon, Nov 8, 2010 at 10:07 PM, Abhinay Mehta <abhinay.mehta@gmail.com>wrote:

> Me and a colleague of mine (Ryan Greenhall) setup Ganglia on our hadoop
> cluster, he has written a summary of what we did to get it to work, you
> might find it useful:
>
> http://forwardtechnology.co.uk/blog/4cc841609f4e6a021100004f
>
> Regards,
> Abhinay Mehta
>
>
> On 8 November 2010 15:31, Jonathan Creasy <jon.creasy@announcemedia.com
> >wrote:
>
> > This is the correct configuration, and there should be nothing more
> needed.
> > I don't think that these configuration changes will take affect on the
> fly
> > so you would need to restart the datanode and namenode processes if I
> > understand correctly.
> >
> > When you browse your you will see some more metrics:
> >
> > dfs.FSDirectory.files_deleted
> > dfs.FSNamesystem.BlockCapacity
> > dfs.FSNamesystem.BlocksTotal
> > dfs.FSNamesystem.CapacityRemainingGB
> > dfs.FSNamesystem.CapacityTotalGB
> > dfs.FSNamesystem.CapacityUsedGB
> > dfs.FSNamesystem.CorruptBlocks
> > dfs.FSNamesystem.ExcessBlocks
> > dfs.FSNamesystem.FilesTotal
> > dfs.FSNamesystem.MissingBlocks
> > dfs.FSNamesystem.PendingDeletionBlocks
> > dfs.FSNamesystem.PendingReplicationBlocks
> > dfs.FSNamesystem.ScheduledReplicationBlocks
> > dfs.FSNamesystem.TotalLoad
> > dfs.FSNamesystem.UnderReplicatedBlocks
> > dfs.datanode.blockChecksumOp_avg_time
> > dfs.datanode.blockChecksumOp_num_ops
> > dfs.datanode.blockReports_avg_time
> > dfs.datanode.blockReports_num_ops
> > dfs.datanode.block_verification_failures
> > dfs.datanode.blocks_read
> > dfs.datanode.blocks_removed
> > dfs.datanode.blocks_replicated
> > dfs.datanode.blocks_verified
> > dfs.datanode.blocks_written
> > dfs.datanode.bytes_read
> > dfs.datanode.bytes_written
> > dfs.datanode.copyBlockOp_avg_time
> > dfs.datanode.copyBlockOp_num_ops
> > dfs.datanode.heartBeats_avg_time
> > dfs.datanode.heartBeats_num_ops
> > dfs.datanode.readBlockOp_avg_time
> > dfs.datanode.readBlockOp_num_ops
> > dfs.datanode.readMetadataOp_avg_time
> > dfs.datanode.readMetadataOp_num_ops
> > dfs.datanode.reads_from_local_client
> > dfs.datanode.reads_from_remote_client
> > dfs.datanode.replaceBlockOp_avg_time
> > dfs.datanode.replaceBlockOp_num_ops
> > dfs.datanode.writeBlockOp_avg_time
> > dfs.datanode.writeBlockOp_num_ops
> > dfs.datanode.writes_from_local_client
> > dfs.datanode.writes_from_remote_client
> > dfs.namenode.AddBlockOps
> > dfs.namenode.CreateFileOps
> > dfs.namenode.DeleteFileOps
> > dfs.namenode.FileInfoOps
> > dfs.namenode.FilesAppended
> > dfs.namenode.FilesCreated
> > dfs.namenode.FilesRenamed
> > dfs.namenode.GetBlockLocations
> > dfs.namenode.GetListingOps
> > dfs.namenode.JournalTransactionsBatchedInSync
> > dfs.namenode.SafemodeTime
> > dfs.namenode.Syncs_avg_time
> > dfs.namenode.Syncs_num_ops
> > dfs.namenode.Transactions_avg_time
> > dfs.namenode.Transactions_num_ops
> > dfs.namenode.blockReport_avg_time
> > dfs.namenode.blockReport_num_ops
> > dfs.namenode.fsImageLoadTime
> > jvm.metrics.gcCount
> > jvm.metrics.gcTimeMillis
> > jvm.metrics.logError
> > jvm.metrics.logFatal
> > jvm.metrics.logInfo
> > jvm.metrics.logWarn
> > jvm.metrics.maxMemoryM
> > jvm.metrics.memHeapCommittedM
> > jvm.metrics.memHeapUsedM
> > jvm.metrics.memNonHeapCommittedM
> > jvm.metrics.memNonHeapUsedM
> > jvm.metrics.threadsBlocked
> > jvm.metrics.threadsNew
> > jvm.metrics.threadsRunnable
> > jvm.metrics.threadsTerminated
> > jvm.metrics.threadsTimedWaiting
> > jvm.metrics.threadsWaiting
> > rpc.metrics.NumOpenConnections
> > rpc.metrics.RpcProcessingTime_avg_time
> > rpc.metrics.RpcProcessingTime_num_ops
> > rpc.metrics.RpcQueueTime_avg_time
> > rpc.metrics.RpcQueueTime_num_ops
> > rpc.metrics.abandonBlock_avg_time
> > rpc.metrics.abandonBlock_num_ops
> > rpc.metrics.addBlock_avg_time
> > rpc.metrics.addBlock_num_ops
> > rpc.metrics.blockReceived_avg_time
> > rpc.metrics.blockReceived_num_ops
> > rpc.metrics.blockReport_avg_time
> > rpc.metrics.blockReport_num_ops
> > rpc.metrics.callQueueLen
> > rpc.metrics.complete_avg_time
> > rpc.metrics.complete_num_ops
> > rpc.metrics.create_avg_time
> > rpc.metrics.create_num_ops
> > rpc.metrics.getEditLogSize_avg_time
> > rpc.metrics.getEditLogSize_num_ops
> > rpc.metrics.getProtocolVersion_avg_time
> > rpc.metrics.getProtocolVersion_num_ops
> > rpc.metrics.register_avg_time
> > rpc.metrics.register_num_ops
> > rpc.metrics.rename_avg_time
> > rpc.metrics.rename_num_ops
> > rpc.metrics.renewLease_avg_time
> > rpc.metrics.renewLease_num_ops
> > rpc.metrics.rollEditLog_avg_time
> > rpc.metrics.rollEditLog_num_ops
> > rpc.metrics.rollFsImage_avg_time
> > rpc.metrics.rollFsImage_num_ops
> > rpc.metrics.sendHeartbeat_avg_time
> > rpc.metrics.sendHeartbeat_num_ops
> > rpc.metrics.versionRequest_avg_time
> > rpc.metrics.versionRequest_num_ops
> >
> > -Jonathan
> >
> > On Nov 8, 2010, at 8:34 AM, Shuja Rehman wrote:
> >
> > > Hi
> > > I have cluster of 4 machines and want to configure ganglia for
> monitoring
> > > purpose. I have read the wiki and add the following lines to
> > > hadoop-metrics.properties on each machine.
> > >
> > > dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext
> > > dfs.period=10
> > > dfs.servers=10.10.10.2:8649
> > >
> > > mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext
> > > mapred.period=10
> > > mapred.servers=10.10.10.2:8649
> > >
> > > jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext
> > > jvm.period=10
> > > jvm.servers=10.10.10.2:8649
> > >
> > > rpc.class=org.apache.hadoop.metrics.ganglia.GangliaContext
> > > rpc.period=10
> > > rpc.servers=10.10.10.2:8649
> > >
> > >
> > > where 10.10.10.2 is the machine where i am running gmeated and web
> front
> > > end. Will  I need to same ip in all machine as i do here or need to
> give
> > > machine own ip in each file? and is there anything more to do to setup
> it
> > > with hadoop?
> > >
> > >
> > >
> > > --
> > > Regards
> > > Shuja-ur-Rehman Baig
> > > <http://pk.linkedin.com/in/shujamughal>
> >
> >
>



-- 
Regards
Shuja-ur-Rehman Baig
<http://pk.linkedin.com/in/shujamughal>

--0016364c7d9985f67e04948ef40f--

From general-return-2350-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 08 18:51:46 2010
Return-Path: <general-return-2350-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 99465 invoked from network); 8 Nov 2010 18:51:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Nov 2010 18:51:45 -0000
Received: (qmail 82620 invoked by uid 500); 8 Nov 2010 18:52:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82546 invoked by uid 500); 8 Nov 2010 18:52:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82532 invoked by uid 99); 8 Nov 2010 18:52:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Nov 2010 18:52:15 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,MIME_BASE64_TEXT,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 65.55.88.14 is neither permitted nor denied by domain of jon.creasy@announcemedia.com)
Received: from [65.55.88.14] (HELO TX2EHSOBE008.bigfish.com) (65.55.88.14)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Nov 2010 18:52:08 +0000
Received: from mail173-tx2-R.bigfish.com (10.9.14.244) by
 TX2EHSOBE008.bigfish.com (10.9.40.28) with Microsoft SMTP Server id
 14.1.225.8; Mon, 8 Nov 2010 18:51:46 +0000
Received: from mail173-tx2 (localhost.localdomain [127.0.0.1])	by
 mail173-tx2-R.bigfish.com (Postfix) with ESMTP id F11F818B82F7;	Mon,  8 Nov
 2010 18:51:45 +0000 (UTC)
X-SpamScore: -20
X-BigFish: VPS-20(zz1454K9370J98dN168aJ9371Pzz1202hzz8275bh8275dhz2ei2a8h61h)
X-Spam-TCS-SCL: 0:0
X-Forefront-Antispam-Report: KIP:(null);UIP:(null);IPVD:NLI;H:VA3DIAHUB010.RED001.local;RD:smtp801.microsoftonline.com;EFVD:NLI
Received: from mail173-tx2 (localhost.localdomain [127.0.0.1]) by mail173-tx2
 (MessageSwitch) id 128924230476465_10326; Mon,  8 Nov 2010 18:51:44 +0000
 (UTC)
Received: from TX2EHSMHS001.bigfish.com (unknown [10.9.14.241])	by
 mail173-tx2.bigfish.com (Postfix) with ESMTP id 0D8D920804F;	Mon,  8 Nov 2010
 18:51:44 +0000 (UTC)
Received: from VA3DIAHUB010.RED001.local (65.55.171.153) by
 TX2EHSMHS001.bigfish.com (10.9.99.101) with Microsoft SMTP Server (TLS) id
 14.1.225.8; Mon, 8 Nov 2010 18:51:42 +0000
Received: from VA3DIAXVS401.RED001.local ([10.32.57.17]) by
 VA3DIAHUB010.RED001.local ([10.32.16.181]) with mapi; Mon, 8 Nov 2010
 10:51:34 -0800
From: Jonathan Creasy <jon.creasy@Announcemedia.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Date: Mon, 8 Nov 2010 10:51:28 -0800
Subject: Re: Configure Ganglia with Hadoop
Thread-Topic: Configure Ganglia with Hadoop
Thread-Index: Act/dfY/NgyQBBsyQDuBHaVYFjXn5A==
Message-ID: <FCFFA9CA-EE3C-4FA4-BFEA-95532F0BA3A2@announcemedia.microsoftonline.com>
References: <AANLkTi=zFtT1_W7Lbqbi84rL99Cd9NQfdEL=wEz4ckp2@mail.gmail.com>
 <8280F561-1AE3-459E-8A94-C20B4CDD739A@announcemedia.microsoftonline.com>
 <AANLkTimihLb2sgZWFXpRmutWQR3Py+ZoW64wSW4MH3Un@mail.gmail.com>
 <AANLkTik3y8DEXF7+zHDyMJ5WrEqRN=wQsS=b-_gc7QzR@mail.gmail.com>
In-Reply-To: <AANLkTik3y8DEXF7+zHDyMJ5WrEqRN=wQsS=b-_gc7QzR@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_FCFFA9CAEE3C4FA4BFEA95532F0BA3A2announcemediamicrosofto_"
MIME-Version: 1.0
X-OriginatorOrg: announcemedia.com
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_FCFFA9CAEE3C4FA4BFEA95532F0BA3A2announcemediamicrosofto_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: base64

DQpJZiB5b3VyIG5ldHdvcmsgc3VwcG9ydHMgbXVsdGljYXN0IHRoZW4gdGhhdCBzaG91bGQgd29y
ayBmaW5lLg0KDQpPdXJzPGh0dHA6Ly9ibG9nLnN0bGhhZG9vcC5vcmc+IGRvZXNuJ3Qgc28gd2Ug
aGFkIHRvIHVzZSBVRFAsIHRoaXMgbWVhbnMgcHV0dGluZyB0aGUgZm9sbG93aW5nIGluIGFsbCBv
ZiB0aGUgZ21vbmQuY29uZiBmaWxlcy4NCg0KdWRwX3NlbmRfY2hhbm5lbCB7DQpob3N0ID0gZ2Fu
Z2xpYW5vZGUNCnBvcnQgPSA4NjAxDQp9DQoNCnVkcF9yZWN2X2NoYW5uZWwgew0KcG9ydCA9IDg2
MDENCmZhbWlseSA9IGluZXQ0DQp9DQoNCg0KT24gTm92IDgsIDIwMTAsIGF0IDEyOjQwIFBNLCBT
aHVqYSBSZWhtYW4gd3JvdGU6DQoNCkhpDQpJIGhhdmUgZm9sbG93IHRoZSBhcnRpY2xlLCBpIGhh
dmUgb25lIGNvbmZ1c2lvbiBkbyBpIG5lZWQgdG8gY2hhbmdlDQpnbW9uZC5jb25maWcgZmlsZSBv
biBlYWNoIG5vZGU/Pw0KDQpob3N0IHsNCiBsb2NhdGlvbiA9ICJ1bnNwZWNpZmllZCINCn0NCg0K
LyogRmVlbCBmcmVlIHRvIHNwZWNpZnkgYXMgbWFueSB1ZHBfc2VuZF9jaGFubmVscyBhcyB5b3Ug
bGlrZS4gIEdtb25kDQogIHVzZWQgdG8gb25seSBzdXBwb3J0IGhhdmluZyBhIHNpbmdsZSBjaGFu
bmVsICovDQp1ZHBfc2VuZF9jaGFubmVsIHsNCiBtY2FzdF9qb2luID0gMjM5LjIuMTEuNzENCiBw
b3J0ID0gODY0OQ0KIHR0bCA9IDENCn0NCg0KLyogWW91IGNhbiBzcGVjaWZ5IGFzIG1hbnkgdWRw
X3JlY3ZfY2hhbm5lbHMgYXMgeW91IGxpa2UgYXMgd2VsbC4gKi8NCnVkcF9yZWN2X2NoYW5uZWwg
ew0KIG1jYXN0X2pvaW4gPSAyMzkuMi4xMS43MQ0KIHBvcnQgPSA4NjQ5DQogYmluZCA9IDIzOS4y
LjExLjcxDQp9DQoNCi8qIFlvdSBjYW4gc3BlY2lmeSBhcyBtYW55IHRjcF9hY2NlcHRfY2hhbm5l
bHMgYXMgeW91IGxpa2UgdG8gc2hhcmUNCiAgYW4geG1sIGRlc2NyaXB0aW9uIG9mIHRoZSBzdGF0
ZSBvZiB0aGUgY2x1c3RlciAqLw0KdGNwX2FjY2VwdF9jaGFubmVsIHsNCiBwb3J0ID0gODY0OQ0K
fQ0KDQoNCmFuZCBpIG5lZWQgdG8gcmVwbGFjZSB0aGUgMjM5LjIuMTEuNzEgd2l0aCB0aGF0IHNw
ZWNpZmljIG1hY2hpbmUgaXAgZS5nDQoxMC4xMC4xMC4yIGluIDFzdCBtYWNoaW5lIGNhc2UgYW5k
IDEwLjEwLjEwLjMgZm9yIDJuZCBtYWNoaW5lIGFuZCBzbyBvbj8NCg0KT24gTW9uLCBOb3YgOCwg
MjAxMCBhdCAxMDowNyBQTSwgQWJoaW5heSBNZWh0YSA8YWJoaW5heS5tZWh0YUBnbWFpbC5jb208
bWFpbHRvOmFiaGluYXkubWVodGFAZ21haWwuY29tPj53cm90ZToNCg0KTWUgYW5kIGEgY29sbGVh
Z3VlIG9mIG1pbmUgKFJ5YW4gR3JlZW5oYWxsKSBzZXR1cCBHYW5nbGlhIG9uIG91ciBoYWRvb3AN
CmNsdXN0ZXIsIGhlIGhhcyB3cml0dGVuIGEgc3VtbWFyeSBvZiB3aGF0IHdlIGRpZCB0byBnZXQg
aXQgdG8gd29yaywgeW91DQptaWdodCBmaW5kIGl0IHVzZWZ1bDoNCg0KaHR0cDovL2ZvcndhcmR0
ZWNobm9sb2d5LmNvLnVrL2Jsb2cvNGNjODQxNjA5ZjRlNmEwMjExMDAwMDRmDQoNClJlZ2FyZHMs
DQpBYmhpbmF5IE1laHRhDQoNCg0KT24gOCBOb3ZlbWJlciAyMDEwIDE1OjMxLCBKb25hdGhhbiBD
cmVhc3kgPGpvbi5jcmVhc3lAYW5ub3VuY2VtZWRpYS5jb208bWFpbHRvOmpvbi5jcmVhc3lAYW5u
b3VuY2VtZWRpYS5jb20+DQp3cm90ZToNCg0KVGhpcyBpcyB0aGUgY29ycmVjdCBjb25maWd1cmF0
aW9uLCBhbmQgdGhlcmUgc2hvdWxkIGJlIG5vdGhpbmcgbW9yZQ0KbmVlZGVkLg0KSSBkb24ndCB0
aGluayB0aGF0IHRoZXNlIGNvbmZpZ3VyYXRpb24gY2hhbmdlcyB3aWxsIHRha2UgYWZmZWN0IG9u
IHRoZQ0KZmx5DQpzbyB5b3Ugd291bGQgbmVlZCB0byByZXN0YXJ0IHRoZSBkYXRhbm9kZSBhbmQg
bmFtZW5vZGUgcHJvY2Vzc2VzIGlmIEkNCnVuZGVyc3RhbmQgY29ycmVjdGx5Lg0KDQpXaGVuIHlv
dSBicm93c2UgeW91ciB5b3Ugd2lsbCBzZWUgc29tZSBtb3JlIG1ldHJpY3M6DQoNCmRmcy5GU0Rp
cmVjdG9yeS5maWxlc19kZWxldGVkDQpkZnMuRlNOYW1lc3lzdGVtLkJsb2NrQ2FwYWNpdHkNCmRm
cy5GU05hbWVzeXN0ZW0uQmxvY2tzVG90YWwNCmRmcy5GU05hbWVzeXN0ZW0uQ2FwYWNpdHlSZW1h
aW5pbmdHQg0KZGZzLkZTTmFtZXN5c3RlbS5DYXBhY2l0eVRvdGFsR0INCmRmcy5GU05hbWVzeXN0
ZW0uQ2FwYWNpdHlVc2VkR0INCmRmcy5GU05hbWVzeXN0ZW0uQ29ycnVwdEJsb2Nrcw0KZGZzLkZT
TmFtZXN5c3RlbS5FeGNlc3NCbG9ja3MNCmRmcy5GU05hbWVzeXN0ZW0uRmlsZXNUb3RhbA0KZGZz
LkZTTmFtZXN5c3RlbS5NaXNzaW5nQmxvY2tzDQpkZnMuRlNOYW1lc3lzdGVtLlBlbmRpbmdEZWxl
dGlvbkJsb2Nrcw0KZGZzLkZTTmFtZXN5c3RlbS5QZW5kaW5nUmVwbGljYXRpb25CbG9ja3MNCmRm
cy5GU05hbWVzeXN0ZW0uU2NoZWR1bGVkUmVwbGljYXRpb25CbG9ja3MNCmRmcy5GU05hbWVzeXN0
ZW0uVG90YWxMb2FkDQpkZnMuRlNOYW1lc3lzdGVtLlVuZGVyUmVwbGljYXRlZEJsb2Nrcw0KZGZz
LmRhdGFub2RlLmJsb2NrQ2hlY2tzdW1PcF9hdmdfdGltZQ0KZGZzLmRhdGFub2RlLmJsb2NrQ2hl
Y2tzdW1PcF9udW1fb3BzDQpkZnMuZGF0YW5vZGUuYmxvY2tSZXBvcnRzX2F2Z190aW1lDQpkZnMu
ZGF0YW5vZGUuYmxvY2tSZXBvcnRzX251bV9vcHMNCmRmcy5kYXRhbm9kZS5ibG9ja192ZXJpZmlj
YXRpb25fZmFpbHVyZXMNCmRmcy5kYXRhbm9kZS5ibG9ja3NfcmVhZA0KZGZzLmRhdGFub2RlLmJs
b2Nrc19yZW1vdmVkDQpkZnMuZGF0YW5vZGUuYmxvY2tzX3JlcGxpY2F0ZWQNCmRmcy5kYXRhbm9k
ZS5ibG9ja3NfdmVyaWZpZWQNCmRmcy5kYXRhbm9kZS5ibG9ja3Nfd3JpdHRlbg0KZGZzLmRhdGFu
b2RlLmJ5dGVzX3JlYWQNCmRmcy5kYXRhbm9kZS5ieXRlc193cml0dGVuDQpkZnMuZGF0YW5vZGUu
Y29weUJsb2NrT3BfYXZnX3RpbWUNCmRmcy5kYXRhbm9kZS5jb3B5QmxvY2tPcF9udW1fb3BzDQpk
ZnMuZGF0YW5vZGUuaGVhcnRCZWF0c19hdmdfdGltZQ0KZGZzLmRhdGFub2RlLmhlYXJ0QmVhdHNf
bnVtX29wcw0KZGZzLmRhdGFub2RlLnJlYWRCbG9ja09wX2F2Z190aW1lDQpkZnMuZGF0YW5vZGUu
cmVhZEJsb2NrT3BfbnVtX29wcw0KZGZzLmRhdGFub2RlLnJlYWRNZXRhZGF0YU9wX2F2Z190aW1l
DQpkZnMuZGF0YW5vZGUucmVhZE1ldGFkYXRhT3BfbnVtX29wcw0KZGZzLmRhdGFub2RlLnJlYWRz
X2Zyb21fbG9jYWxfY2xpZW50DQpkZnMuZGF0YW5vZGUucmVhZHNfZnJvbV9yZW1vdGVfY2xpZW50
DQpkZnMuZGF0YW5vZGUucmVwbGFjZUJsb2NrT3BfYXZnX3RpbWUNCmRmcy5kYXRhbm9kZS5yZXBs
YWNlQmxvY2tPcF9udW1fb3BzDQpkZnMuZGF0YW5vZGUud3JpdGVCbG9ja09wX2F2Z190aW1lDQpk
ZnMuZGF0YW5vZGUud3JpdGVCbG9ja09wX251bV9vcHMNCmRmcy5kYXRhbm9kZS53cml0ZXNfZnJv
bV9sb2NhbF9jbGllbnQNCmRmcy5kYXRhbm9kZS53cml0ZXNfZnJvbV9yZW1vdGVfY2xpZW50DQpk
ZnMubmFtZW5vZGUuQWRkQmxvY2tPcHMNCmRmcy5uYW1lbm9kZS5DcmVhdGVGaWxlT3BzDQpkZnMu
bmFtZW5vZGUuRGVsZXRlRmlsZU9wcw0KZGZzLm5hbWVub2RlLkZpbGVJbmZvT3BzDQpkZnMubmFt
ZW5vZGUuRmlsZXNBcHBlbmRlZA0KZGZzLm5hbWVub2RlLkZpbGVzQ3JlYXRlZA0KZGZzLm5hbWVu
b2RlLkZpbGVzUmVuYW1lZA0KZGZzLm5hbWVub2RlLkdldEJsb2NrTG9jYXRpb25zDQpkZnMubmFt
ZW5vZGUuR2V0TGlzdGluZ09wcw0KZGZzLm5hbWVub2RlLkpvdXJuYWxUcmFuc2FjdGlvbnNCYXRj
aGVkSW5TeW5jDQpkZnMubmFtZW5vZGUuU2FmZW1vZGVUaW1lDQpkZnMubmFtZW5vZGUuU3luY3Nf
YXZnX3RpbWUNCmRmcy5uYW1lbm9kZS5TeW5jc19udW1fb3BzDQpkZnMubmFtZW5vZGUuVHJhbnNh
Y3Rpb25zX2F2Z190aW1lDQpkZnMubmFtZW5vZGUuVHJhbnNhY3Rpb25zX251bV9vcHMNCmRmcy5u
YW1lbm9kZS5ibG9ja1JlcG9ydF9hdmdfdGltZQ0KZGZzLm5hbWVub2RlLmJsb2NrUmVwb3J0X251
bV9vcHMNCmRmcy5uYW1lbm9kZS5mc0ltYWdlTG9hZFRpbWUNCmp2bS5tZXRyaWNzLmdjQ291bnQN
Cmp2bS5tZXRyaWNzLmdjVGltZU1pbGxpcw0KanZtLm1ldHJpY3MubG9nRXJyb3INCmp2bS5tZXRy
aWNzLmxvZ0ZhdGFsDQpqdm0ubWV0cmljcy5sb2dJbmZvDQpqdm0ubWV0cmljcy5sb2dXYXJuDQpq
dm0ubWV0cmljcy5tYXhNZW1vcnlNDQpqdm0ubWV0cmljcy5tZW1IZWFwQ29tbWl0dGVkTQ0KanZt
Lm1ldHJpY3MubWVtSGVhcFVzZWRNDQpqdm0ubWV0cmljcy5tZW1Ob25IZWFwQ29tbWl0dGVkTQ0K
anZtLm1ldHJpY3MubWVtTm9uSGVhcFVzZWRNDQpqdm0ubWV0cmljcy50aHJlYWRzQmxvY2tlZA0K
anZtLm1ldHJpY3MudGhyZWFkc05ldw0KanZtLm1ldHJpY3MudGhyZWFkc1J1bm5hYmxlDQpqdm0u
bWV0cmljcy50aHJlYWRzVGVybWluYXRlZA0KanZtLm1ldHJpY3MudGhyZWFkc1RpbWVkV2FpdGlu
Zw0KanZtLm1ldHJpY3MudGhyZWFkc1dhaXRpbmcNCnJwYy5tZXRyaWNzLk51bU9wZW5Db25uZWN0
aW9ucw0KcnBjLm1ldHJpY3MuUnBjUHJvY2Vzc2luZ1RpbWVfYXZnX3RpbWUNCnJwYy5tZXRyaWNz
LlJwY1Byb2Nlc3NpbmdUaW1lX251bV9vcHMNCnJwYy5tZXRyaWNzLlJwY1F1ZXVlVGltZV9hdmdf
dGltZQ0KcnBjLm1ldHJpY3MuUnBjUXVldWVUaW1lX251bV9vcHMNCnJwYy5tZXRyaWNzLmFiYW5k
b25CbG9ja19hdmdfdGltZQ0KcnBjLm1ldHJpY3MuYWJhbmRvbkJsb2NrX251bV9vcHMNCnJwYy5t
ZXRyaWNzLmFkZEJsb2NrX2F2Z190aW1lDQpycGMubWV0cmljcy5hZGRCbG9ja19udW1fb3BzDQpy
cGMubWV0cmljcy5ibG9ja1JlY2VpdmVkX2F2Z190aW1lDQpycGMubWV0cmljcy5ibG9ja1JlY2Vp
dmVkX251bV9vcHMNCnJwYy5tZXRyaWNzLmJsb2NrUmVwb3J0X2F2Z190aW1lDQpycGMubWV0cmlj
cy5ibG9ja1JlcG9ydF9udW1fb3BzDQpycGMubWV0cmljcy5jYWxsUXVldWVMZW4NCnJwYy5tZXRy
aWNzLmNvbXBsZXRlX2F2Z190aW1lDQpycGMubWV0cmljcy5jb21wbGV0ZV9udW1fb3BzDQpycGMu
bWV0cmljcy5jcmVhdGVfYXZnX3RpbWUNCnJwYy5tZXRyaWNzLmNyZWF0ZV9udW1fb3BzDQpycGMu
bWV0cmljcy5nZXRFZGl0TG9nU2l6ZV9hdmdfdGltZQ0KcnBjLm1ldHJpY3MuZ2V0RWRpdExvZ1Np
emVfbnVtX29wcw0KcnBjLm1ldHJpY3MuZ2V0UHJvdG9jb2xWZXJzaW9uX2F2Z190aW1lDQpycGMu
bWV0cmljcy5nZXRQcm90b2NvbFZlcnNpb25fbnVtX29wcw0KcnBjLm1ldHJpY3MucmVnaXN0ZXJf
YXZnX3RpbWUNCnJwYy5tZXRyaWNzLnJlZ2lzdGVyX251bV9vcHMNCnJwYy5tZXRyaWNzLnJlbmFt
ZV9hdmdfdGltZQ0KcnBjLm1ldHJpY3MucmVuYW1lX251bV9vcHMNCnJwYy5tZXRyaWNzLnJlbmV3
TGVhc2VfYXZnX3RpbWUNCnJwYy5tZXRyaWNzLnJlbmV3TGVhc2VfbnVtX29wcw0KcnBjLm1ldHJp
Y3Mucm9sbEVkaXRMb2dfYXZnX3RpbWUNCnJwYy5tZXRyaWNzLnJvbGxFZGl0TG9nX251bV9vcHMN
CnJwYy5tZXRyaWNzLnJvbGxGc0ltYWdlX2F2Z190aW1lDQpycGMubWV0cmljcy5yb2xsRnNJbWFn
ZV9udW1fb3BzDQpycGMubWV0cmljcy5zZW5kSGVhcnRiZWF0X2F2Z190aW1lDQpycGMubWV0cmlj
cy5zZW5kSGVhcnRiZWF0X251bV9vcHMNCnJwYy5tZXRyaWNzLnZlcnNpb25SZXF1ZXN0X2F2Z190
aW1lDQpycGMubWV0cmljcy52ZXJzaW9uUmVxdWVzdF9udW1fb3BzDQoNCi1Kb25hdGhhbg0KDQpP
biBOb3YgOCwgMjAxMCwgYXQgODozNCBBTSwgU2h1amEgUmVobWFuIHdyb3RlOg0KDQpIaQ0KSSBo
YXZlIGNsdXN0ZXIgb2YgNCBtYWNoaW5lcyBhbmQgd2FudCB0byBjb25maWd1cmUgZ2FuZ2xpYSBm
b3INCm1vbml0b3JpbmcNCnB1cnBvc2UuIEkgaGF2ZSByZWFkIHRoZSB3aWtpIGFuZCBhZGQgdGhl
IGZvbGxvd2luZyBsaW5lcyB0bw0KaGFkb29wLW1ldHJpY3MucHJvcGVydGllcyBvbiBlYWNoIG1h
Y2hpbmUuDQoNCmRmcy5jbGFzcz1vcmcuYXBhY2hlLmhhZG9vcC5tZXRyaWNzLmdhbmdsaWEuR2Fu
Z2xpYUNvbnRleHQNCmRmcy5wZXJpb2Q9MTANCmRmcy5zZXJ2ZXJzPTEwLjEwLjEwLjI6ODY0OQ0K
DQptYXByZWQuY2xhc3M9b3JnLmFwYWNoZS5oYWRvb3AubWV0cmljcy5nYW5nbGlhLkdhbmdsaWFD
b250ZXh0DQptYXByZWQucGVyaW9kPTEwDQptYXByZWQuc2VydmVycz0xMC4xMC4xMC4yOjg2NDkN
Cg0KanZtLmNsYXNzPW9yZy5hcGFjaGUuaGFkb29wLm1ldHJpY3MuZ2FuZ2xpYS5HYW5nbGlhQ29u
dGV4dA0KanZtLnBlcmlvZD0xMA0KanZtLnNlcnZlcnM9MTAuMTAuMTAuMjo4NjQ5DQoNCnJwYy5j
bGFzcz1vcmcuYXBhY2hlLmhhZG9vcC5tZXRyaWNzLmdhbmdsaWEuR2FuZ2xpYUNvbnRleHQNCnJw
Yy5wZXJpb2Q9MTANCnJwYy5zZXJ2ZXJzPTEwLjEwLjEwLjI6ODY0OQ0KDQoNCndoZXJlIDEwLjEw
LjEwLjIgaXMgdGhlIG1hY2hpbmUgd2hlcmUgaSBhbSBydW5uaW5nIGdtZWF0ZWQgYW5kIHdlYg0K
ZnJvbnQNCmVuZC4gV2lsbCAgSSBuZWVkIHRvIHNhbWUgaXAgaW4gYWxsIG1hY2hpbmUgYXMgaSBk
byBoZXJlIG9yIG5lZWQgdG8NCmdpdmUNCm1hY2hpbmUgb3duIGlwIGluIGVhY2ggZmlsZT8gYW5k
IGlzIHRoZXJlIGFueXRoaW5nIG1vcmUgdG8gZG8gdG8gc2V0dXANCml0DQp3aXRoIGhhZG9vcD8N
Cg0KDQoNCi0tDQpSZWdhcmRzDQpTaHVqYS11ci1SZWhtYW4gQmFpZw0KPGh0dHA6Ly9way5saW5r
ZWRpbi5jb20vaW4vc2h1amFtdWdoYWw+DQoNCg0KDQoNCg0KDQotLQ0KUmVnYXJkcw0KU2h1amEt
dXItUmVobWFuIEJhaWcNCjxodHRwOi8vcGsubGlua2VkaW4uY29tL2luL3NodWphbXVnaGFsPg0K
DQo=

--_000_FCFFA9CAEE3C4FA4BFEA95532F0BA3A2announcemediamicrosofto_--

From general-return-2351-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 08 19:34:19 2010
Return-Path: <general-return-2351-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21401 invoked from network); 8 Nov 2010 19:34:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Nov 2010 19:34:19 -0000
Received: (qmail 39457 invoked by uid 500); 8 Nov 2010 19:34:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39401 invoked by uid 500); 8 Nov 2010 19:34:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39393 invoked by uid 99); 8 Nov 2010 19:34:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Nov 2010 19:34:48 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jonathan.seidman@gmail.com designates 209.85.213.48 as permitted sender)
Received: from [209.85.213.48] (HELO mail-yw0-f48.google.com) (209.85.213.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Nov 2010 19:34:43 +0000
Received: by ywp4 with SMTP id 4so4236052ywp.35
        for <general@hadoop.apache.org>; Mon, 08 Nov 2010 11:34:22 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=MK8B+/V6c3Agv4qFZYedjUXLG0/8c/bA/pUDrkxiYqE=;
        b=peWnjCvedvWXvPoo+PsDGkdT2l1WpBtxTsrMohvz1e3D3fZU2oS4JJFrpPtm4ctc11
         iD1JU5xtPcRte936iQ3/IVv2K/EJA+oKYnN2X+vT/JX5l3A5NTeJ76xafSIvPS8xlAaD
         GVpTMMHKnoPwqFH6lUoFDxVPFsMM9tCNUxAzI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=ql/Xw2IcC0dXpSXIBUvp6pYYy8FhKXye4jbpgMbi/qdviAEKyyLSYwGjO6UwU44C1P
         Bz5s5umrL2jtZ1Ji7zC9fkUj/DDTwoLkwvQw9Pl68BARmKP3S4ILvrPR/Va+ZRCIumlR
         315Q0jVAEBG5wg4UhEj/4tgz4UhP8kUv4E/l4=
MIME-Version: 1.0
Received: by 10.216.30.2 with SMTP id j2mr7585577wea.33.1289244861916; Mon, 08
 Nov 2010 11:34:21 -0800 (PST)
Received: by 10.216.86.9 with HTTP; Mon, 8 Nov 2010 11:34:21 -0800 (PST)
In-Reply-To: <AANLkTimJLuJX9YukiAid=ueN610hRuDWXraJV8BgKRRW@mail.gmail.com>
References: <AANLkTimJLuJX9YukiAid=ueN610hRuDWXraJV8BgKRRW@mail.gmail.com>
Date: Mon, 8 Nov 2010 13:34:21 -0600
Message-ID: <AANLkTin1vyS0hmGEPfVtMAO3+dvW_RaHjJz-6XMabgA2@mail.gmail.com>
Subject: Re: [ANNOUNCE] Two Talks by Jeff Hammerbacher on Nov. 11th at Orbitz
 Worldwide HQ in Chicago
From: Jonathan Seidman <jonathan.seidman@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6dd8e7605e47c04948fb6d6

--0016e6dd8e7605e47c04948fb6d6
Content-Type: text/plain; charset=ISO-8859-1

Here's a correct URL for the IDEAS talk:
http://www.meetup.com/orbitzideas/calendar/15258988/

On Mon, Nov 8, 2010 at 10:39 AM, Jonathan Seidman <
jonathan.seidman@gmail.com> wrote:

> Orbitz is pleased to welcome Jeff Hammerbacher, Vice President, Products at
> Cloudera Inc. for two talks at Orbitz Worldwide headquarters in Chicago on
> Thursday, November 11th. Both talks are open to the public:
>
> At 11:30 Jeff will speak on "Experiences Evolving a New Analytical
> Platform" as part of the Orbitz IDEAS series. RSVP here: http:/...
> /www.meetup.com/orbitzideas/calendar/15258988<http://www.meetup.com/orbitzideas/calendar/15258988/>
>
> In the evening Jeff will be speaking to the Chicago Hadoop User Group. RSVP
> for the meeting here: http://www.meetup.com/Chicago-area-Hadoo
> p-User-Group-CHUG/calendar/15203244/
>
> Thanks! Hope t see you there.
>
> Jonathan
>

--0016e6dd8e7605e47c04948fb6d6--

From general-return-2352-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 09 03:12:33 2010
Return-Path: <general-return-2352-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 51506 invoked from network); 9 Nov 2010 03:12:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Nov 2010 03:12:32 -0000
Received: (qmail 18555 invoked by uid 500); 9 Nov 2010 03:13:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18400 invoked by uid 500); 9 Nov 2010 03:12:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18391 invoked by uid 99); 9 Nov 2010 03:12:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Nov 2010 03:12:58 +0000
X-ASF-Spam-Status: No, hits=-1997.8 required=10.0
	tests=ALL_TRUSTED,HTML_MESSAGE,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 09 Nov 2010 03:12:58 +0000
Received: (qmail 51084 invoked by uid 99); 9 Nov 2010 03:12:02 -0000
Received: from localhost.apache.org (HELO mail-bw0-f48.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Nov 2010 03:12:02 +0000
Received: by bwz9 with SMTP id 9so830355bwz.35
        for <general@hadoop.apache.org>; Mon, 08 Nov 2010 19:12:32 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.53.9 with SMTP id k9mr5862231bkg.102.1289272352086; Mon,
 08 Nov 2010 19:12:32 -0800 (PST)
Received: by 10.204.29.18 with HTTP; Mon, 8 Nov 2010 19:12:32 -0800 (PST)
Date: Mon, 8 Nov 2010 19:12:32 -0800
Message-ID: <AANLkTiksbYv6R+eS-L-Tmb=Dn+xb98PZjSxRAPAaUtLJ@mail.gmail.com>
Subject: Making release branch for 0.22
From: "Owen O'Malley" <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502d9ff9099250494961ca5

--00504502d9ff9099250494961ca5
Content-Type: text/plain; charset=UTF-8

All,
   I plan on making the release branch for 0.22 next week (11/15). After
that point, only critical bug fixes or jiras that I've approved as release
manager can be applied to it.

-- Owen

--00504502d9ff9099250494961ca5--

From general-return-2353-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 09 12:39:30 2010
Return-Path: <general-return-2353-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6785 invoked from network); 9 Nov 2010 12:39:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Nov 2010 12:39:30 -0000
Received: (qmail 39707 invoked by uid 500); 9 Nov 2010 12:40:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39367 invoked by uid 500); 9 Nov 2010 12:39:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39359 invoked by uid 99); 9 Nov 2010 12:39:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Nov 2010 12:39:56 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of wlangiewicz@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Nov 2010 12:39:48 +0000
Received: by fxm7 with SMTP id 7so4952022fxm.35
        for <general@hadoop.apache.org>; Tue, 09 Nov 2010 04:39:28 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from
         :user-agent:mime-version:to:subject:content-type
         :content-transfer-encoding;
        bh=RZGN16Of/xTROtLZ3DzniXcXRV5oidUFBoFznfs3XFo=;
        b=v0KaMCojwZSI2+tyVJmjeJfRs/FVTVZ9v5AMFzDLnrAjL+czYFOe2lpUVkH4poATl2
         U4GRGhMCvum4MQAHFTViJ9L9GTI0vV1vL1TGRa1iClVsaJWll+XmozdO7EYibDQNQ8os
         eFiP1qHtiCpi36vQs5aJD6pUyiNu4Fsf3qUDo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject
         :content-type:content-transfer-encoding;
        b=cK5WhvU9GJZHjS4qsshIOrlzRrg/Z/VAzpMF2JG66WP7XUTt6kdGdQv/ZHApu7vVuR
         3ie67E5u4Ig3PNFABcFKgoYXCvoJxTXxTAsGKy9v1b8L3wJCjcJNn4SkdNog/l+fVlAj
         D5tvNz7ildP7JYqn314c3k7pl1wGXmYXJTr44=
Received: by 10.223.97.75 with SMTP id k11mr4997909fan.85.1289306367823;
        Tue, 09 Nov 2010 04:39:27 -0800 (PST)
Received: from [192.168.0.69] ([195.88.186.242])
        by mx.google.com with ESMTPS id z2sm621042fam.15.2010.11.09.04.39.25
        (version=SSLv3 cipher=RC4-MD5);
        Tue, 09 Nov 2010 04:39:26 -0800 (PST)
Message-ID: <4CD940FC.5000306@gmail.com>
Date: Tue, 09 Nov 2010 13:39:24 +0100
From: Wojciech Langiewicz <wlangiewicz@gmail.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: cleaning up 'hadoop.tmp.dir' ?
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hello,
I have a problem with my tmp dir. Right now it exceeded 4TB on every of 
my 4 machines, so it has 16TB in total.

I'm running out of space on servers, so I would like to know if it is 
safe to clean this directory completly, or if there is configuration 
option that would do this for me.

Any help will be appreciated.

--
Wojciech Langiewicz

From general-return-2354-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 09 13:46:59 2010
Return-Path: <general-return-2354-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 40178 invoked from network); 9 Nov 2010 13:46:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Nov 2010 13:46:58 -0000
Received: (qmail 43061 invoked by uid 500); 9 Nov 2010 13:47:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42862 invoked by uid 500); 9 Nov 2010 13:47:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42854 invoked by uid 99); 9 Nov 2010 13:47:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Nov 2010 13:47:25 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of msegel@navteq.com designates 65.167.11.240 as permitted sender)
Received: from [65.167.11.240] (HELO xmailchi.navteq.com) (65.167.11.240)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Nov 2010 13:47:17 +0000
Received: from chicago-spamfw.navteq.com (imailchi [10.8.5.161])
	by xmailchi.navteq.com (8.13.8/8.13.6) with ESMTP id oA9DJX8h007429
	for <general@hadoop.apache.org>; Tue, 9 Nov 2010 07:19:33 -0600
Received: from imailchi.hq.navteq.com (localhost [127.0.0.1])
	by chicago-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id 75EC842E8960
	for <general@hadoop.apache.org>; Tue,  9 Nov 2010 07:46:56 -0600 (CST)
Received: from imailchi.hq.navteq.com (imailchi.hq.navteq.com [10.8.120.104]) by chicago-spamfw.navteq.com with ESMTP id tLEoUbd6F3F8LG2l for <general@hadoop.apache.org>; Tue, 09 Nov 2010 07:46:56 -0600 (CST)
Received: from hq-ex-ht02.ad.navteq.com (hq-ex-ht02.ad.navteq.com [10.8.222.172])
	by imailchi.hq.navteq.com (8.12.9/8.12.9) with ESMTP id oA9Dkutf003380
	for <general@hadoop.apache.org>; Tue, 9 Nov 2010 07:46:56 -0600
Received: from hq-ex-mb02.ad.navteq.com ([fe80::880d:318e:e860:a8db]) by
 hq-ex-ht02.ad.navteq.com ([fe80::24f6:ca50:734d:df6e%14]) with mapi; Tue, 9
 Nov 2010 07:45:30 -0600
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 9 Nov 2010 07:46:55 -0600
Subject: RE: cleaning up 'hadoop.tmp.dir' ?
Thread-Topic: cleaning up 'hadoop.tmp.dir' ?
Thread-Index: AcuACzs3ZPjZt1pHSqWS0qHcCQR4wQACSzuA
Message-ID: <ABC24175AFD3BE4DA15F4CD375ED413D0639E23F24@hq-ex-mb02.ad.navteq.com>
References: <4CD940FC.5000306@gmail.com>
In-Reply-To: <4CD940FC.5000306@gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1
X-Virus-Checked: Checked by ClamAV on apache.org

Assuming you didn't change anything with the job history and you don't have jobs that run for more than 24 hours...
You should be able to remove anything that hasn't been touched for the past 24 hours.

So you could create a cron job to remove the files.

HTH

-Mike


-----Original Message-----
From: Wojciech Langiewicz [mailto:wlangiewicz@gmail.com] 
Sent: Tuesday, November 09, 2010 6:39 AM
To: general@hadoop.apache.org
Subject: cleaning up 'hadoop.tmp.dir' ?

Hello,
I have a problem with my tmp dir. Right now it exceeded 4TB on every of 
my 4 machines, so it has 16TB in total.

I'm running out of space on servers, so I would like to know if it is 
safe to clean this directory completly, or if there is configuration 
option that would do this for me.

Any help will be appreciated.

--
Wojciech Langiewicz


The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above.  If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited.  If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.

From general-return-2355-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 09 14:04:07 2010
Return-Path: <general-return-2355-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46292 invoked from network); 9 Nov 2010 14:04:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Nov 2010 14:04:07 -0000
Received: (qmail 75239 invoked by uid 500); 9 Nov 2010 14:04:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74760 invoked by uid 500); 9 Nov 2010 14:04:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74752 invoked by uid 99); 9 Nov 2010 14:04:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Nov 2010 14:04:33 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of wlangiewicz@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Nov 2010 14:04:24 +0000
Received: by fxm7 with SMTP id 7so5026285fxm.35
        for <general@hadoop.apache.org>; Tue, 09 Nov 2010 06:04:04 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=Q9C3xUcnGynskZ9HODC4W/mtUVO3qs0rwQyx8GlxEik=;
        b=jly4ymq5eWOIwLAh6ALmuaCyMrUl99nx/giA/sRuH+mcxX9NB1O417aZVIxkXzGg47
         VAx28Z90vnDxPkte84xUyvN4PVJ0srdnkHPUrFDjU74xqgpQQnR8cb3gpn4500mReDDc
         DbgxBY9JuY6OcaaBG55ROC0dhTGqQw4L+vtOI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:content-type:content-transfer-encoding;
        b=NtIFeITOsRFEe4yMvJtwmnUU5orxnm8S8DtRf/RPYwz/Q11W+iGpbTkveSrr6vlbha
         Kaj/8bBT80rL99yFTqcqR2HRJEsp1zBDi1+iik6pD0sFgYkmsj+3tx3QLWmrGHl5XTEB
         rIy0CrHzm3Ok67w5ixk1EQEuxUdAV0FEoZk/c=
Received: by 10.223.116.9 with SMTP id k9mr5076986faq.124.1289311444273;
        Tue, 09 Nov 2010 06:04:04 -0800 (PST)
Received: from [172.19.30.231] (static.nk-net.pl [195.88.186.3])
        by mx.google.com with ESMTPS id 16sm677641fal.0.2010.11.09.06.04.02
        (version=SSLv3 cipher=RC4-MD5);
        Tue, 09 Nov 2010 06:04:03 -0800 (PST)
Message-ID: <4CD954D1.1080403@gmail.com>
Date: Tue, 09 Nov 2010 15:04:01 +0100
From: Wojciech Langiewicz <wlangiewicz@gmail.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: cleaning up 'hadoop.tmp.dir' ?
References: <4CD940FC.5000306@gmail.com> <ABC24175AFD3BE4DA15F4CD375ED413D0639E23F24@hq-ex-mb02.ad.navteq.com>
In-Reply-To: <ABC24175AFD3BE4DA15F4CD375ED413D0639E23F24@hq-ex-mb02.ad.navteq.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks for the answer, but what about HDFS data?

In tmp dir I have another 2: dfs and mapred
Directory 'dfs' seems to contain block from the HDFS, is it safe to 
delete them?

W dniu 09.11.2010 14:46, Segel, Mike wrote:
> Assuming you didn't change anything with the job history and you don't have jobs that run for more than 24 hours...
> You should be able to remove anything that hasn't been touched for the past 24 hours.
>
> So you could create a cron job to remove the files.
>
> HTH
>
> -Mike
>
>
> -----Original Message-----
> From: Wojciech Langiewicz [mailto:wlangiewicz@gmail.com]
> Sent: Tuesday, November 09, 2010 6:39 AM
> To: general@hadoop.apache.org
> Subject: cleaning up 'hadoop.tmp.dir' ?
>
> Hello,
> I have a problem with my tmp dir. Right now it exceeded 4TB on every of
> my 4 machines, so it has 16TB in total.
>
> I'm running out of space on servers, so I would like to know if it is
> safe to clean this directory completly, or if there is configuration
> option that would do this for me.
>
> Any help will be appreciated.
>
> --
> Wojciech Langiewicz
>
>
> The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above.  If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited.  If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.


From general-return-2356-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 09 14:19:05 2010
Return-Path: <general-return-2356-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60045 invoked from network); 9 Nov 2010 14:19:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Nov 2010 14:19:05 -0000
Received: (qmail 10477 invoked by uid 500); 9 Nov 2010 14:19:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 10195 invoked by uid 500); 9 Nov 2010 14:19:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 10186 invoked by uid 99); 9 Nov 2010 14:19:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Nov 2010 14:19:31 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Nov 2010 14:19:24 +0000
Received: by fxm7 with SMTP id 7so5041277fxm.35
        for <general@hadoop.apache.org>; Tue, 09 Nov 2010 06:19:04 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type;
        bh=dpyoWLHP6d9V2gM1etzoyBSOtmRrrAPdI5P3qiaP2xU=;
        b=gd6kphVoW7/x5ftujUClTPQLBlMJA5xIXld1ema+U3xS4FL6o2KF3Q/XkumyXkozZT
         pdsXf8JmL5hcGPt2CtyIHRDsi6+w9Z2DZWCqAtLogvuPMZAACyoOc5UCZ2Y/TBr/pvle
         oMS6Y1NTJ+vEcn91kxhiwO82/+8iiOkfAtGf4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=rFX6Pc+QNLGK1lSQHK1FBHdWifUyRvf2LYtR+TsKSE3BWjMEkmPuQzj8lCy4Aqke8b
         8vWwzGZNIA2Z9drLHlg7dg+GffOcKj7xPl0EVNQ/Eqa6lxm2/4cNzsD6tfg7Wc7IXlLw
         UZLqNFuT7780zJUEOKrIVU3gYUIdVwLiYxJDw=
Received: by 10.223.89.142 with SMTP id e14mr5141362fam.143.1289312344010;
 Tue, 09 Nov 2010 06:19:04 -0800 (PST)
MIME-Version: 1.0
Received: by 10.223.113.145 with HTTP; Tue, 9 Nov 2010 06:18:43 -0800 (PST)
In-Reply-To: <4CD954D1.1080403@gmail.com>
References: <4CD940FC.5000306@gmail.com> <ABC24175AFD3BE4DA15F4CD375ED413D0639E23F24@hq-ex-mb02.ad.navteq.com>
 <4CD954D1.1080403@gmail.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Tue, 9 Nov 2010 19:48:43 +0530
Message-ID: <AANLkTi=ydQ+pxweDNiU-nhO8GqqihuWfb6ZywkXO7p0P@mail.gmail.com>
Subject: Re: cleaning up 'hadoop.tmp.dir' ?
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hey,

On Tue, Nov 9, 2010 at 7:34 PM, Wojciech Langiewicz
<wlangiewicz@gmail.com> wrote:
> Thanks for the answer, but what about HDFS data?
>
> In tmp dir I have another 2: dfs and mapred
> Directory 'dfs' seems to contain block from the HDFS, is it safe to delete
> them?
>

NO! You'll lose your blocks if you delete that dfs/data directory (If
it is in use and I suppose that's what is taking up much space).

In production, the dfs.data.dir property should not point any of its
directories to a /tmp subdir which may be wiped upon boot.

-- 
Harsh J
www.harshj.com

From general-return-2357-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 09 14:19:46 2010
Return-Path: <general-return-2357-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60273 invoked from network); 9 Nov 2010 14:19:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Nov 2010 14:19:45 -0000
Received: (qmail 11946 invoked by uid 500); 9 Nov 2010 14:20:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11862 invoked by uid 500); 9 Nov 2010 14:20:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11854 invoked by uid 99); 9 Nov 2010 14:20:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Nov 2010 14:20:14 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of msegel@navteq.com designates 65.167.11.240 as permitted sender)
Received: from [65.167.11.240] (HELO xmailchi.navteq.com) (65.167.11.240)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Nov 2010 14:20:08 +0000
Received: from chicago-spamfw.navteq.com (imailchi [10.8.5.161])
	by xmailchi.navteq.com (8.13.8/8.13.6) with ESMTP id oA9DqOHn008471
	for <general@hadoop.apache.org>; Tue, 9 Nov 2010 07:52:24 -0600
Received: from imailchi.hq.navteq.com (localhost [127.0.0.1])
	by chicago-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id C252A42E93B8
	for <general@hadoop.apache.org>; Tue,  9 Nov 2010 08:19:46 -0600 (CST)
Received: from imailchi.hq.navteq.com (imailchi.hq.navteq.com [10.8.120.104]) by chicago-spamfw.navteq.com with ESMTP id a6NFXDDLrlO1S3Sv for <general@hadoop.apache.org>; Tue, 09 Nov 2010 08:19:46 -0600 (CST)
Received: from hq-ex-ht01.ad.navteq.com (hq-ex-ht01.ad.navteq.com [10.8.222.51])
	by imailchi.hq.navteq.com (8.12.9/8.12.9) with ESMTP id oA9EJktf009504
	for <general@hadoop.apache.org>; Tue, 9 Nov 2010 08:19:46 -0600
Received: from hq-ex-mb02.ad.navteq.com ([fe80::880d:318e:e860:a8db]) by
 hq-ex-ht01.ad.navteq.com ([fe80::55a6:5b94:5c00:fe39%12]) with mapi; Tue, 9
 Nov 2010 08:19:46 -0600
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 9 Nov 2010 08:19:45 -0600
Subject: RE: cleaning up 'hadoop.tmp.dir' ?
Thread-Topic: cleaning up 'hadoop.tmp.dir' ?
Thread-Index: AcuAFw1LJTOHPxDVRKaiceCvpsk7VwAAJ7IA
Message-ID: <ABC24175AFD3BE4DA15F4CD375ED413D0639E23F45@hq-ex-mb02.ad.navteq.com>
References: <4CD940FC.5000306@gmail.com>
 <ABC24175AFD3BE4DA15F4CD375ED413D0639E23F24@hq-ex-mb02.ad.navteq.com>
 <4CD954D1.1080403@gmail.com>
In-Reply-To: <4CD954D1.1080403@gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1

SSdtIG5vdCBzdXJlIGlmIHlvdSBtZWFudCBzb21ldGhpbmcgb24gdGhlIGhhZG9vcCBIREZTLCBv
ciBzb21ldGhpbmcgaW4gdGhlIHVuaXggL3RtcCBkaXJlY3RvcnkgdGhhdOKAmXMgaW4gYSBzdWJk
aXJlY3RvcnkuDQoNCkFnYWluLCB0aGUgZ2VuZXJhbCBydWxlIG9mIHRodW1iIGlzIHRoYXQgaWYg
YSBmaWxlIGhhc24ndCBiZWVuIHRvdWNoZWQgb3IgYWNjZXNzZWQgZm9yIHRoZSBwYXN0IDI0IGhv
dXJzLCBpdCBzaG91bGQgYmUgb2sgdG8gZGVsZXRlLg0KTm90ZTogSWYgeW91J3JlIHBhcmFub2lk
IGFuZCB5b3UgaGF2ZSBhIGxvdCBvZiBmaWxlcyB0aGF0IGFyZSBtdWNoIG9sZGVyLCB5b3UgbWF5
IHdhbnQgdG8gY29uc2lkZXIgZGVsZXRpbmcgYW55dGhpbmcgdGhhdCBoYXNuJ3QgYmVlbiB0b3Vj
aGVkIGluIHRoZSBwYXN0IHdlZWsuIEFuZCBieSB0b3VjaGVkLCBJIG1lYW4gbGFzdCBhY2Nlc3Mg
dGltZSwgbm90IGxhc3QgbW9kaWZpZWQgdGltZS4NCg0KSFRIDQoNCi1NaWtlDQoNCg0KLS0tLS1P
cmlnaW5hbCBNZXNzYWdlLS0tLS0NCkZyb206IFdvamNpZWNoIExhbmdpZXdpY3ogW21haWx0bzp3
bGFuZ2lld2ljekBnbWFpbC5jb21dIA0KU2VudDogVHVlc2RheSwgTm92ZW1iZXIgMDksIDIwMTAg
ODowNCBBTQ0KVG86IGdlbmVyYWxAaGFkb29wLmFwYWNoZS5vcmcNClN1YmplY3Q6IFJlOiBjbGVh
bmluZyB1cCAnaGFkb29wLnRtcC5kaXInID8NCg0KVGhhbmtzIGZvciB0aGUgYW5zd2VyLCBidXQg
d2hhdCBhYm91dCBIREZTIGRhdGE/DQoNCkluIHRtcCBkaXIgSSBoYXZlIGFub3RoZXIgMjogZGZz
IGFuZCBtYXByZWQNCkRpcmVjdG9yeSAnZGZzJyBzZWVtcyB0byBjb250YWluIGJsb2NrIGZyb20g
dGhlIEhERlMsIGlzIGl0IHNhZmUgdG8gDQpkZWxldGUgdGhlbT8NCg0KVyBkbml1IDA5LjExLjIw
MTAgMTQ6NDYsIFNlZ2VsLCBNaWtlIHdyb3RlOg0KPiBBc3N1bWluZyB5b3UgZGlkbid0IGNoYW5n
ZSBhbnl0aGluZyB3aXRoIHRoZSBqb2IgaGlzdG9yeSBhbmQgeW91IGRvbid0IGhhdmUgam9icyB0
aGF0IHJ1biBmb3IgbW9yZSB0aGFuIDI0IGhvdXJzLi4uDQo+IFlvdSBzaG91bGQgYmUgYWJsZSB0
byByZW1vdmUgYW55dGhpbmcgdGhhdCBoYXNuJ3QgYmVlbiB0b3VjaGVkIGZvciB0aGUgcGFzdCAy
NCBob3Vycy4NCj4NCj4gU28geW91IGNvdWxkIGNyZWF0ZSBhIGNyb24gam9iIHRvIHJlbW92ZSB0
aGUgZmlsZXMuDQo+DQo+IEhUSA0KPg0KPiAtTWlrZQ0KPg0KPg0KPiAtLS0tLU9yaWdpbmFsIE1l
c3NhZ2UtLS0tLQ0KPiBGcm9tOiBXb2pjaWVjaCBMYW5naWV3aWN6IFttYWlsdG86d2xhbmdpZXdp
Y3pAZ21haWwuY29tXQ0KPiBTZW50OiBUdWVzZGF5LCBOb3ZlbWJlciAwOSwgMjAxMCA2OjM5IEFN
DQo+IFRvOiBnZW5lcmFsQGhhZG9vcC5hcGFjaGUub3JnDQo+IFN1YmplY3Q6IGNsZWFuaW5nIHVw
ICdoYWRvb3AudG1wLmRpcicgPw0KPg0KPiBIZWxsbywNCj4gSSBoYXZlIGEgcHJvYmxlbSB3aXRo
IG15IHRtcCBkaXIuIFJpZ2h0IG5vdyBpdCBleGNlZWRlZCA0VEIgb24gZXZlcnkgb2YNCj4gbXkg
NCBtYWNoaW5lcywgc28gaXQgaGFzIDE2VEIgaW4gdG90YWwuDQo+DQo+IEknbSBydW5uaW5nIG91
dCBvZiBzcGFjZSBvbiBzZXJ2ZXJzLCBzbyBJIHdvdWxkIGxpa2UgdG8ga25vdyBpZiBpdCBpcw0K
PiBzYWZlIHRvIGNsZWFuIHRoaXMgZGlyZWN0b3J5IGNvbXBsZXRseSwgb3IgaWYgdGhlcmUgaXMg
Y29uZmlndXJhdGlvbg0KPiBvcHRpb24gdGhhdCB3b3VsZCBkbyB0aGlzIGZvciBtZS4NCj4NCj4g
QW55IGhlbHAgd2lsbCBiZSBhcHByZWNpYXRlZC4NCj4NCj4gLS0NCj4gV29qY2llY2ggTGFuZ2ll
d2ljeg0KPg0KPg0KPiBUaGUgaW5mb3JtYXRpb24gY29udGFpbmVkIGluIHRoaXMgY29tbXVuaWNh
dGlvbiBtYXkgYmUgQ09ORklERU5USUFMIGFuZCBpcyBpbnRlbmRlZCBvbmx5IGZvciB0aGUgdXNl
IG9mIHRoZSByZWNpcGllbnQocykgbmFtZWQgYWJvdmUuICBJZiB5b3UgYXJlIG5vdCB0aGUgaW50
ZW5kZWQgcmVjaXBpZW50LCB5b3UgYXJlIGhlcmVieSBub3RpZmllZCB0aGF0IGFueSBkaXNzZW1p
bmF0aW9uLCBkaXN0cmlidXRpb24sIG9yIGNvcHlpbmcgb2YgdGhpcyBjb21tdW5pY2F0aW9uLCBv
ciBhbnkgb2YgaXRzIGNvbnRlbnRzLCBpcyBzdHJpY3RseSBwcm9oaWJpdGVkLiAgSWYgeW91IGhh
dmUgcmVjZWl2ZWQgdGhpcyBjb21tdW5pY2F0aW9uIGluIGVycm9yLCBwbGVhc2Ugbm90aWZ5IHRo
ZSBzZW5kZXIgYW5kIGRlbGV0ZS9kZXN0cm95IHRoZSBvcmlnaW5hbCBtZXNzYWdlIGFuZCBhbnkg
Y29weSBvZiBpdCBmcm9tIHlvdXIgY29tcHV0ZXIgb3IgcGFwZXIgZmlsZXMuDQoNCg0KDQpUaGUg
aW5mb3JtYXRpb24gY29udGFpbmVkIGluIHRoaXMgY29tbXVuaWNhdGlvbiBtYXkgYmUgQ09ORklE
RU5USUFMIGFuZCBpcyBpbnRlbmRlZCBvbmx5IGZvciB0aGUgdXNlIG9mIHRoZSByZWNpcGllbnQo
cykgbmFtZWQgYWJvdmUuICBJZiB5b3UgYXJlIG5vdCB0aGUgaW50ZW5kZWQgcmVjaXBpZW50LCB5
b3UgYXJlIGhlcmVieSBub3RpZmllZCB0aGF0IGFueSBkaXNzZW1pbmF0aW9uLCBkaXN0cmlidXRp
b24sIG9yIGNvcHlpbmcgb2YgdGhpcyBjb21tdW5pY2F0aW9uLCBvciBhbnkgb2YgaXRzIGNvbnRl
bnRzLCBpcyBzdHJpY3RseSBwcm9oaWJpdGVkLiAgSWYgeW91IGhhdmUgcmVjZWl2ZWQgdGhpcyBj
b21tdW5pY2F0aW9uIGluIGVycm9yLCBwbGVhc2Ugbm90aWZ5IHRoZSBzZW5kZXIgYW5kIGRlbGV0
ZS9kZXN0cm95IHRoZSBvcmlnaW5hbCBtZXNzYWdlIGFuZCBhbnkgY29weSBvZiBpdCBmcm9tIHlv
dXIgY29tcHV0ZXIgb3IgcGFwZXIgZmlsZXMuDQo=

From general-return-2358-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 09 14:21:43 2010
Return-Path: <general-return-2358-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60807 invoked from network); 9 Nov 2010 14:21:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Nov 2010 14:21:43 -0000
Received: (qmail 17623 invoked by uid 500); 9 Nov 2010 14:22:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17523 invoked by uid 500); 9 Nov 2010 14:22:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17515 invoked by uid 99); 9 Nov 2010 14:22:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Nov 2010 14:22:11 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of msegel@navteq.com designates 65.167.11.240 as permitted sender)
Received: from [65.167.11.240] (HELO xmailchi.navteq.com) (65.167.11.240)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Nov 2010 14:22:04 +0000
Received: from chicago-spamfw.navteq.com (imailchi [10.8.5.161])
	by xmailchi.navteq.com (8.13.8/8.13.6) with ESMTP id oA9DsK2L008521
	for <general@hadoop.apache.org>; Tue, 9 Nov 2010 07:54:20 -0600
Received: from imailchi.hq.navteq.com (localhost [127.0.0.1])
	by chicago-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id AEC2942E946C
	for <general@hadoop.apache.org>; Tue,  9 Nov 2010 08:21:42 -0600 (CST)
Received: from imailchi.hq.navteq.com (imailchi.hq.navteq.com [10.8.120.104]) by chicago-spamfw.navteq.com with ESMTP id Zlih37dckpcsEXCg for <general@hadoop.apache.org>; Tue, 09 Nov 2010 08:21:42 -0600 (CST)
Received: from hq-ex-ht01.ad.navteq.com (hq-ex-ht01.ad.navteq.com [10.8.222.51])
	by imailchi.hq.navteq.com (8.12.9/8.12.9) with ESMTP id oA9ELgtf009779
	for <general@hadoop.apache.org>; Tue, 9 Nov 2010 08:21:42 -0600
Received: from hq-ex-mb02.ad.navteq.com ([fe80::880d:318e:e860:a8db]) by
 hq-ex-ht01.ad.navteq.com ([fe80::55a6:5b94:5c00:fe39%12]) with mapi; Tue, 9
 Nov 2010 08:21:42 -0600
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 9 Nov 2010 08:21:41 -0600
Subject: RE: cleaning up 'hadoop.tmp.dir' ?
Thread-Topic: cleaning up 'hadoop.tmp.dir' ?
Thread-Index: AcuAGSUSA/aN7Y3MQDWfsU813PA66wAAA7EQ
Message-ID: <ABC24175AFD3BE4DA15F4CD375ED413D0639E23F4A@hq-ex-mb02.ad.navteq.com>
References: <4CD940FC.5000306@gmail.com>
 <ABC24175AFD3BE4DA15F4CD375ED413D0639E23F24@hq-ex-mb02.ad.navteq.com>
 <4CD954D1.1080403@gmail.com>
 <AANLkTi=ydQ+pxweDNiU-nhO8GqqihuWfb6ZywkXO7p0P@mail.gmail.com>
In-Reply-To: <AANLkTi=ydQ+pxweDNiU-nhO8GqqihuWfb6ZywkXO7p0P@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1
X-Virus-Checked: Checked by ClamAV on apache.org

That's actually a good point.

I'm looking at it from the Unix admin point of view. Anything in /tmp is =
just that... a temp file that you don't care about.
(Although I think in Cloudera's CDH3 they store the PIDs in /tmp??? (Goin=
g from memory and I'm still on my first cup of coffee.))

-Mike

-----Original Message-----
From: Harsh J [mailto:qwertymaniac@gmail.com]=20
Sent: Tuesday, November 09, 2010 8:19 AM
To: general@hadoop.apache.org
Subject: Re: cleaning up 'hadoop.tmp.dir' ?

Hey,

On Tue, Nov 9, 2010 at 7:34 PM, Wojciech Langiewicz
<wlangiewicz@gmail.com> wrote:
> Thanks for the answer, but what about HDFS data?
>
> In tmp dir I have another 2: dfs and mapred
> Directory 'dfs' seems to contain block from the HDFS, is it safe to del=
ete
> them?
>

NO! You'll lose your blocks if you delete that dfs/data directory (If
it is in use and I suppose that's what is taking up much space).

In production, the dfs.data.dir property should not point any of its
directories to a /tmp subdir which may be wiped upon boot.

--=20
Harsh J
www.harshj.com


The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.

From general-return-2359-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 09 14:29:30 2010
Return-Path: <general-return-2359-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 62668 invoked from network); 9 Nov 2010 14:29:29 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Nov 2010 14:29:29 -0000
Received: (qmail 29411 invoked by uid 500); 9 Nov 2010 14:29:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29031 invoked by uid 500); 9 Nov 2010 14:29:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29022 invoked by uid 99); 9 Nov 2010 14:29:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Nov 2010 14:29:57 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of wlangiewicz@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Nov 2010 14:29:49 +0000
Received: by fxm7 with SMTP id 7so5052050fxm.35
        for <general@hadoop.apache.org>; Tue, 09 Nov 2010 06:29:28 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=s9mxAqQychn6yA7JXWzEggLZefyXSrP2c68thEJItIg=;
        b=NGxFJ3kGur/1P1juU6ZYdS00ixxVsm7ojw+YTsGBc1K0sslaKbGjTvddeZnCVoQ8R2
         GHyTLh+IDaRnOND/BgxxUOKb7AChXdTLIUhKcyLDD+Z695se40qme9aN4NFbzve+PaK/
         1FHJRgetTr9OqeR0cPxAh0DMoPAHLna2KwMww=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:content-type:content-transfer-encoding;
        b=rnXhrY7HavSxOZ1YPV1Y1VixE5sBKpjJjL34XbuZAf9axPn6HMNU9SrnXy5rF2hTef
         JO9guUK4yba9F8mh052t6xmjJQ6U2uXaNf+mr2EQruEG0ulFqp7yQ3P45VSXdCyTiHtp
         GXyq30TJseyP7XXVSzTN/X7aHUO/hbE64gUQ8=
Received: by 10.223.86.16 with SMTP id q16mr5184793fal.58.1289312968322;
        Tue, 09 Nov 2010 06:29:28 -0800 (PST)
Received: from [172.19.30.231] (static.nk-net.pl [195.88.186.3])
        by mx.google.com with ESMTPS id y19sm690510fau.41.2010.11.09.06.29.26
        (version=SSLv3 cipher=RC4-MD5);
        Tue, 09 Nov 2010 06:29:27 -0800 (PST)
Message-ID: <4CD95AC5.2010609@gmail.com>
Date: Tue, 09 Nov 2010 15:29:25 +0100
From: Wojciech Langiewicz <wlangiewicz@gmail.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: cleaning up 'hadoop.tmp.dir' ?
References: <4CD940FC.5000306@gmail.com> <ABC24175AFD3BE4DA15F4CD375ED413D0639E23F24@hq-ex-mb02.ad.navteq.com> <4CD954D1.1080403@gmail.com> <AANLkTi=ydQ+pxweDNiU-nhO8GqqihuWfb6ZywkXO7p0P@mail.gmail.com>
In-Reply-To: <AANLkTi=ydQ+pxweDNiU-nhO8GqqihuWfb6ZywkXO7p0P@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

W dniu 09.11.2010 15:18, Harsh J wrote:
> On Tue, Nov 9, 2010 at 7:34 PM, Wojciech Langiewicz
> <wlangiewicz@gmail.com>  wrote:
>> In tmp dir I have another 2: dfs and mapred
>> Directory 'dfs' seems to contain block from the HDFS, is it safe to delete
>> them?
>>
>
> NO! You'll lose your blocks if you delete that dfs/data directory (If
> it is in use and I suppose that's what is taking up much space).
>
> In production, the dfs.data.dir property should not point any of its
> directories to a /tmp subdir which may be wiped upon boot.
>

Thank you very much for answer, this was as I suspected.

So in fact those files are real data inside HDFS ?

What might be causing situation where I have about 5TB in HDFS and 
hadoop tmp dirs have about 16TB in total?

--
Wojciech Langiewicz

From general-return-2360-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 09 15:06:16 2010
Return-Path: <general-return-2360-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93564 invoked from network); 9 Nov 2010 15:06:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Nov 2010 15:06:15 -0000
Received: (qmail 29608 invoked by uid 500); 9 Nov 2010 15:06:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29368 invoked by uid 500); 9 Nov 2010 15:06:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29359 invoked by uid 99); 9 Nov 2010 15:06:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Nov 2010 15:06:41 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of qwertymaniac@gmail.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Nov 2010 15:06:36 +0000
Received: by gyf2 with SMTP id 2so4945719gyf.35
        for <general@hadoop.apache.org>; Tue, 09 Nov 2010 07:06:16 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type;
        bh=+H4JXcWqJcgezoJi/TTKJVPBsU/byGNdx1R4FP05cAY=;
        b=AKoWFJcoWn5KiIopb6IW6++euE1lEw3SnMmPE37VXW33zh+afqzBkcbLMjV4fnvBVf
         5vtfLH9SvYzFbZUGRdO3BGMnPg8GdJA3SlFtD0fStWFhKHzeW3KIvjihj72lZnSlXzNs
         5fies2vboB0dkGWSkVEAGruDUMkS2Mbb/zPZ8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=a9dUkHdqYJtHrQLyjhYal38Fj2VqeH2/28B+lJrAuYRQ6hSytxSaQQQ7K2ZWIUCbvt
         6Bvfhno/QhcMGMATtVnYz0oXjseuVoqbfTjzNVmAljW0lztjazDPhq0rLl3I/zMgiq99
         X5Mx7FR4qIsWljkVi0zNFsFWqFvYsvRIw+cHk=
Received: by 10.204.102.76 with SMTP id f12mr6673650bko.57.1289315175607; Tue,
 09 Nov 2010 07:06:15 -0800 (PST)
MIME-Version: 1.0
Received: by 10.223.113.145 with HTTP; Tue, 9 Nov 2010 07:05:55 -0800 (PST)
In-Reply-To: <ABC24175AFD3BE4DA15F4CD375ED413D0639E23F4A@hq-ex-mb02.ad.navteq.com>
References: <4CD940FC.5000306@gmail.com> <ABC24175AFD3BE4DA15F4CD375ED413D0639E23F24@hq-ex-mb02.ad.navteq.com>
 <4CD954D1.1080403@gmail.com> <AANLkTi=ydQ+pxweDNiU-nhO8GqqihuWfb6ZywkXO7p0P@mail.gmail.com>
 <ABC24175AFD3BE4DA15F4CD375ED413D0639E23F4A@hq-ex-mb02.ad.navteq.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Tue, 9 Nov 2010 20:35:55 +0530
Message-ID: <AANLkTin8hWsa1_NCWPJ-CpzbZ-CQJzTEi808A=GEs-Sp@mail.gmail.com>
Subject: Re: cleaning up 'hadoop.tmp.dir' ?
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Nov 9, 2010 at 7:51 PM, Segel, Mike <msegel@navteq.com> wrote:
> That's actually a good point.
>
> I'm looking at it from the Unix admin point of view. Anything in /tmp is just that... a temp file that you don't care about.
> (Although I think in Cloudera's CDH3 they store the PIDs in /tmp??? (Going from memory and I'm still on my first cup of coffee.))
>

If am right, they set hadoop.tmp.dir to
/var/lib/hadoop/cache-something in their bundled configuration. This
is sufficient in moving all files that otherwise would go to /tmp, as
all defaults (like name dir, data dir, mapred/local, mapred/io, etc.)
use ${hadoop.tmp.dir}/ as their prefix.

-- 
Harsh J
www.harshj.com

From general-return-2361-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 09 16:24:14 2010
Return-Path: <general-return-2361-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54956 invoked from network); 9 Nov 2010 16:24:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Nov 2010 16:24:14 -0000
Received: (qmail 29317 invoked by uid 500); 9 Nov 2010 16:24:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29233 invoked by uid 500); 9 Nov 2010 16:24:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29219 invoked by uid 99); 9 Nov 2010 16:24:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Nov 2010 16:24:41 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Nov 2010 16:24:35 +0000
Received: by qwk3 with SMTP id 3so4911856qwk.35
        for <general@hadoop.apache.org>; Tue, 09 Nov 2010 08:24:14 -0800 (PST)
Received: by 10.224.205.200 with SMTP id fr8mr5445827qab.198.1289319854616;
 Tue, 09 Nov 2010 08:24:14 -0800 (PST)
MIME-Version: 1.0
Received: by 10.229.114.209 with HTTP; Tue, 9 Nov 2010 08:23:54 -0800 (PST)
In-Reply-To: <4CD95AC5.2010609@gmail.com>
References: <4CD940FC.5000306@gmail.com> <ABC24175AFD3BE4DA15F4CD375ED413D0639E23F24@hq-ex-mb02.ad.navteq.com>
 <4CD954D1.1080403@gmail.com> <AANLkTi=ydQ+pxweDNiU-nhO8GqqihuWfb6ZywkXO7p0P@mail.gmail.com>
 <4CD95AC5.2010609@gmail.com>
From: Aaron Myers <atm@cloudera.com>
Date: Tue, 9 Nov 2010 08:23:54 -0800
Message-ID: <AANLkTikBr6pTwFbnRFdSXMOkmQfZh+X1XG84GXBV_OPf@mail.gmail.com>
Subject: Re: cleaning up 'hadoop.tmp.dir' ?
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf300faae5efb3b10494a12b47

--20cf300faae5efb3b10494a12b47
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Nov 9, 2010 at 6:29 AM, Wojciech Langiewicz
<wlangiewicz@gmail.com>wrote:

>
> What might be causing situation where I have about 5TB in HDFS and hadoop
> tmp dirs have about 16TB in total?
>

If indeed this is the block data of your HDFS files, then this makes perfect
sense. HDFS by default replicates every block 3 times, so ~5TB used in HDFS
is ~15TB raw on disk.

Aaron

--20cf300faae5efb3b10494a12b47--

From general-return-2362-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 09 16:28:31 2010
Return-Path: <general-return-2362-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57069 invoked from network); 9 Nov 2010 16:28:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Nov 2010 16:28:30 -0000
Received: (qmail 46317 invoked by uid 500); 9 Nov 2010 16:29:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46158 invoked by uid 500); 9 Nov 2010 16:28:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46150 invoked by uid 99); 9 Nov 2010 16:28:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Nov 2010 16:28:59 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of wlangiewicz@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Nov 2010 16:28:52 +0000
Received: by fxm7 with SMTP id 7so5196399fxm.35
        for <general@hadoop.apache.org>; Tue, 09 Nov 2010 08:28:30 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=3wvCa421ZThAd++csvNm4XLN+Bnc7ynPJsW4DaYloOI=;
        b=fB4zVGftLDFLim481Z0SQoNg+cnKDmSz8PYg14eU566LhPylOmPUgOoWFy8MVd7nCY
         65Pq9Lgr8nrVOlDL5k9pBCBtJynDYQCPA6+XDhi1MTTZCw4Sz+I0BSjv+TKi0jk9Wvuz
         DtDr99eN+bXrI3KOlzOl7RYiX6g1BSprRjX9w=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:content-type:content-transfer-encoding;
        b=FlKMXwbNIX2S7kgCL8zIfbR3npOqBRJNJGkDnHnPm6H1yuypXXORVjrLfndvo3JCZF
         azEe3oTc5vJeZI3LWw062baTJqVc6JiJcq31TkZj903v4Ffp8PIfhN2d7JndoA6nwiFV
         Xj2qi9/rYJ4h/1JSORlowU8v2aHBsBX2eSdu4=
Received: by 10.223.83.133 with SMTP id f5mr5336156fal.29.1289320110659;
        Tue, 09 Nov 2010 08:28:30 -0800 (PST)
Received: from [172.19.30.231] (static.nk-net.pl [195.88.186.3])
        by mx.google.com with ESMTPS id 15sm776230fal.22.2010.11.09.08.28.29
        (version=SSLv3 cipher=RC4-MD5);
        Tue, 09 Nov 2010 08:28:29 -0800 (PST)
Message-ID: <4CD976AC.6030203@gmail.com>
Date: Tue, 09 Nov 2010 17:28:28 +0100
From: Wojciech Langiewicz <wlangiewicz@gmail.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: cleaning up 'hadoop.tmp.dir' ?
References: <4CD940FC.5000306@gmail.com> <ABC24175AFD3BE4DA15F4CD375ED413D0639E23F24@hq-ex-mb02.ad.navteq.com> <4CD954D1.1080403@gmail.com> <AANLkTi=ydQ+pxweDNiU-nhO8GqqihuWfb6ZywkXO7p0P@mail.gmail.com> <4CD95AC5.2010609@gmail.com> <AANLkTikBr6pTwFbnRFdSXMOkmQfZh+X1XG84GXBV_OPf@mail.gmail.com>
In-Reply-To: <AANLkTikBr6pTwFbnRFdSXMOkmQfZh+X1XG84GXBV_OPf@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

W dniu 09.11.2010 17:23, Aaron Myers pisze:
> On Tue, Nov 9, 2010 at 6:29 AM, Wojciech Langiewicz
> <wlangiewicz@gmail.com>wrote:
>
>>
>> What might be causing situation where I have about 5TB in HDFS and hadoop
>> tmp dirs have about 16TB in total?
>>
>
> If indeed this is the block data of your HDFS files, then this makes perfect
> sense. HDFS by default replicates every block 3 times, so ~5TB used in HDFS
> is ~15TB raw on disk.

You are right, I wonder why didn't I though about it before.
Thanks for all the answers:)

But name of this option 'hadoop.tmp.dir' is at least a little confusing.

--
Wojciech Langiewicz

From general-return-2363-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 09 16:48:07 2010
Return-Path: <general-return-2363-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 64639 invoked from network); 9 Nov 2010 16:48:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Nov 2010 16:48:07 -0000
Received: (qmail 2244 invoked by uid 500); 9 Nov 2010 16:48:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2181 invoked by uid 500); 9 Nov 2010 16:48:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2173 invoked by uid 99); 9 Nov 2010 16:48:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Nov 2010 16:48:36 +0000
X-ASF-Spam-Status: No, hits=0.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS,URI_OBFU_WWW
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [195.232.224.74] (HELO mailout05.vodafone.com) (195.232.224.74)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Nov 2010 16:48:28 +0000
Received: from mailint05 (localhost [127.0.0.1])
	by mailout05 (Postfix) with ESMTP id C4685146834
	for <general@hadoop.apache.org>; Tue,  9 Nov 2010 17:48:07 +0100 (CET)
Received: from avoexs04.internal.vodafone.com (avoexs04.dc-ratingen.de [145.230.4.198])
	(using TLSv1 with cipher RC4-MD5 (128/128 bits))
	(No client certificate requested)
	by mailint05 (Postfix) with ESMTPS id AA5D714650C
	for <general@hadoop.apache.org>; Tue,  9 Nov 2010 17:48:07 +0100 (CET)
Received: from VF-MBX13.internal.vodafone.com ([145.230.5.24]) by avoexs04.internal.vodafone.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Tue, 9 Nov 2010 17:48:06 +0100
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/mixed;
	boundary="----_=_NextPart_001_01CB802D.E296B520"
Subject: RE: web-based file transfer
Date: Tue, 9 Nov 2010 17:48:07 +0100
Message-ID: <219D8244D980254ABF28AB469AD4E98F0346A15F@VF-MBX13.internal.vodafone.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: <219D8244D980254ABF28AB469AD4E98F0346A15F@VF-MBX13.internal.vodafone.com>
Thread-Topic: web-based file transfer
Thread-Index: Act7S1n8eBK7QP69Rtiun5Wkz5WaWQE35FUV
References: <033e01cb7abb$623b7d50$26b277f0$@com> <4CD148F9.8080600@apache.org>
From: "Gibbon, Robert, VF-Group" <Robert.Gibbon@vodafone.com>
To: <general@hadoop.apache.org>
X-OriginalArrivalTime: 09 Nov 2010 16:48:06.0788 (UTC) FILETIME=[E23F6440:01CB802D]
X-Virus-Checked: Checked by ClamAV on apache.org

------_=_NextPart_001_01CB802D.E296B520
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

>Even all the Java servlet APIs assume that the content-length header=20
>fits into a signed 32 bit integer and gets unhappy once you go over 2GB =

>(something I worry about in
>http://jira.smartfrog.org/jira/browse/SFOS-1476 )

I built my HDFS webDAV implementation to reference the JackRabbit 1.6.4 =
library - AFAIK it uses a long for the content-length header since =
release 1.5.5, not a 32bit int:

https://issues.apache.org/jira/browse/JCR-2009

That means any large file limitations are going to be on the client =
side, especially for 32bit OSs, so yes, might be worth thinking about =
leveraging HAR archives to keep the file count down if you do choose to =
go down the same route.

R

On 02/11/10 18:25, Mark Laffoon wrote:
> We want to enable our web-based client (i.e. browser client, java =
applet,
> whatever?) to transfer files into a system backed by hdfs. The obvious
> simple solution is to do http file uploads, then copy the file to =
hdfs. I
> was wondering if there is a way to do it with an hdfs-enabled applet =
where
> the server gives the client the necessary hadoop configuration
> information, and the client applet pushes the data directly into hdfs.


I recall some work done with webdav
   https://issues.apache.org/jira/browse/HDFS-225
-but I don't think it's progressed

I've done things like this in the past with servlets and forms; the=20
webapp you deploy has the hadoop configuration (and the network rights=20
to talk to HDFS in the datacentre), the clients PUT/POST up content

http://www.slideshare.net/steve_l/long-haul-hadoop

However, you are limited to 2GB worth of upload/download in most web=20
clients, some (chrome) go up to 4GB but you are pushing the limit there. =

Even all the Java servlet APIs assume that the content-length header=20
fits into a signed 32 bit integer and gets unhappy once you go over 2GB=20
(something I worry about in=20
http://jira.smartfrog.org/jira/browse/SFOS-1476 )

Because Hadoop really likes large files -tens to hundreds of GB in a big =

cluster- I don't think the current web infrastructure is up to working=20
with it.


that said, looking at hudson for the nightly runs of my bulk IO tests ,=20
jetty will serve up 4GB in 5 minutes (loopback if), and I can POST  or=20
PUT up 4GB, but I have to get/set content length headers myself rather=20
than rely on the java.net client and servlet implementations to handle =
it:

http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/componen=
ts/www/src/org/smartfrog/services/www/bulkio/client/SunJavaBulkIOClient.j=
ava?revision=3D8430&view=3Dmarkup

If you can control the client, then maybe you would be able to do >4GB=20
uploads, but otherwise you are stuck with data <2GB in size, which is,=20
-what- 4-8 blocks in a production cluster?

-steve


------_=_NextPart_001_01CB802D.E296B520--

From general-return-2364-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 09 18:16:26 2010
Return-Path: <general-return-2364-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33222 invoked from network); 9 Nov 2010 18:16:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Nov 2010 18:16:26 -0000
Received: (qmail 36111 invoked by uid 500); 9 Nov 2010 18:16:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 35970 invoked by uid 500); 9 Nov 2010 18:16:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 35962 invoked by uid 99); 9 Nov 2010 18:16:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Nov 2010 18:16:52 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Nov 2010 18:16:46 +0000
Received: by vws16 with SMTP id 16so1736498vws.35
        for <general@hadoop.apache.org>; Tue, 09 Nov 2010 10:16:25 -0800 (PST)
Received: by 10.229.249.203 with SMTP id ml11mr3966944qcb.266.1289326584514;
 Tue, 09 Nov 2010 10:16:24 -0800 (PST)
MIME-Version: 1.0
Received: by 10.229.114.209 with HTTP; Tue, 9 Nov 2010 10:16:04 -0800 (PST)
In-Reply-To: <4CD976AC.6030203@gmail.com>
References: <4CD940FC.5000306@gmail.com> <ABC24175AFD3BE4DA15F4CD375ED413D0639E23F24@hq-ex-mb02.ad.navteq.com>
 <4CD954D1.1080403@gmail.com> <AANLkTi=ydQ+pxweDNiU-nhO8GqqihuWfb6ZywkXO7p0P@mail.gmail.com>
 <4CD95AC5.2010609@gmail.com> <AANLkTikBr6pTwFbnRFdSXMOkmQfZh+X1XG84GXBV_OPf@mail.gmail.com>
 <4CD976AC.6030203@gmail.com>
From: Aaron Myers <atm@cloudera.com>
Date: Tue, 9 Nov 2010 10:16:04 -0800
Message-ID: <AANLkTim0tGnAxvfFYL9+=dUhGj7OCpqm70+mP1NGSgth@mail.gmail.com>
Subject: Re: cleaning up 'hadoop.tmp.dir' ?
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64caa7e11c9550494a2bda6

--0016e64caa7e11c9550494a2bda6
Content-Type: text/plain; charset=ISO-8859-1

Hi Wojciech Langiewicz,

On Tue, Nov 9, 2010 at 8:28 AM, Wojciech Langiewicz
<wlangiewicz@gmail.com>wrote:

> But name of this option 'hadoop.tmp.dir' is at least a little confusing.
>

You can set the dfs.data.dir config value to a separate directory from the
hadoop.tmp.dir. By default, the value of this variable is set to
"file://${hadoop.tmp.dir}/dfs/data", which is why the HDFS data blocks were
put there as a subdirectory of the hadoop.tmp.dir.

See these for more details:

http://hadoop.apache.org/common/docs/r0.20.1/core-default.html
http://hadoop.apache.org/common/docs/r0.20.1/hdfs-default.html

Hope that helps,
Aaron

--0016e64caa7e11c9550494a2bda6--

From general-return-2365-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 09 18:42:00 2010
Return-Path: <general-return-2365-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50199 invoked from network); 9 Nov 2010 18:42:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Nov 2010 18:42:00 -0000
Received: (qmail 80588 invoked by uid 500); 9 Nov 2010 18:42:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80352 invoked by uid 500); 9 Nov 2010 18:42:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 44848 invoked by uid 99); 9 Nov 2010 14:35:42 -0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Message-Id: <4CD95C0E.107@tum.de>
Date: Tue, 09 Nov 2010 15:34:54 +0100
From: Benjamin Gufler <benjamin.gufler@tum.de>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.15) Gecko/20101030 Lightning/1.0b1 Icedove/3.0.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: cleaning up 'hadoop.tmp.dir' ?
References: <4CD940FC.5000306@gmail.com> <ABC24175AFD3BE4DA15F4CD375ED413D0639E23F24@hq-ex-mb02.ad.navteq.com> <4CD954D1.1080403@gmail.com> <AANLkTi=ydQ+pxweDNiU-nhO8GqqihuWfb6ZywkXO7p0P@mail.gmail.com> <4CD95AC5.2010609@gmail.com>
In-Reply-To: <4CD95AC5.2010609@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 2010-11-09 15:29, Wojciech Langiewicz wrote:
> What might be causing situation where I have about 5TB in HDFS and
> hadoop tmp dirs have about 16TB in total?

Have you turned on replication? A replication factor of 3 would explain 
the size difference.

Ben

From general-return-2366-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 11 09:45:11 2010
Return-Path: <general-return-2366-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 40273 invoked from network); 11 Nov 2010 09:45:10 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Nov 2010 09:45:10 -0000
Received: (qmail 76195 invoked by uid 500); 11 Nov 2010 09:45:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76044 invoked by uid 500); 11 Nov 2010 09:45:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76036 invoked by uid 99); 11 Nov 2010 09:45:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Nov 2010 09:45:38 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [145.100.8.49] (HELO smtp.sara.nl) (145.100.8.49)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Nov 2010 09:45:32 +0000
Received: from planck.ka.sara.nl (145.100.8.32) by sara-exch-fe1.ka.sara.nl
 (145.100.8.49) with Microsoft SMTP Server (TLS) id 14.0.702.0; Thu, 11 Nov
 2010 10:45:08 +0100
Received: from planck.ka.sara.nl ([145.100.8.32]) by planck.ka.sara.nl
 ([145.100.8.32]) with mapi; Thu, 11 Nov 2010 10:44:48 +0100
From: Evert Lammerts <Evert.Lammerts@sara.nl>
To: "'general@hadoop.apache.org'" <general@hadoop.apache.org>
Date: Thu, 11 Nov 2010 10:42:46 +0100
Subject: CDH3 / 0.20 config
Thread-Topic: CDH3 / 0.20 config
Thread-Index: AcuBhMuYvBqHcNNAQYa/y4cfC3nxSw==
Message-ID: <ADF94D8555C7A246B86A633685E0178AB90A474AB2@planck.ka.sara.nl>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/signed; protocol="application/x-pkcs7-signature";
	micalg=SHA1; boundary="----=_NextPart_000_005E_01CB818D.2D9432D0"
MIME-Version: 1.0

------=_NextPart_000_005E_01CB818D.2D9432D0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit

Hi all,

Can I use these pages as a reference for configuration? Are they not
outdated for / ahead of version 0.20? 

core-site.xml:
http://hadoop.apache.org/common/docs/current/core-default.html
hdfs-site.xml: http://hadoop.apache.org/hdfs/docs/current/hdfs-default.html
mapred-site.xml:
http://hadoop.apache.org/mapreduce/docs/current/mapred-default.html

Are there any such pages for other config files? (fair- /
capacity-scheduler, hadoop-policy, mapred-queue-acls)

A good configuration overview is lacking at the moment, as far as I can see.
If time permits, I might provide one for 0.20.

Cheers,

Evert Lammerts
Consultant eScience Support
SARA Computing & Network Services 
High Performance Computing & Visualization

Phone: +31 20 888 4101
Email: evert.lammerts@sara.nl
http://www.sara.nl



------=_NextPart_000_005E_01CB818D.2D9432D0
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIPUDCCBDIw
ggMaoAMCAQICAQEwDQYJKoZIhvcNAQEFBQAwezELMAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0
ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0
ZWQxITAfBgNVBAMMGEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczAeFw0wNDAxMDEwMDAwMDBaFw0y
ODEyMzEyMzU5NTlaMHsxCzAJBgNVBAYTAkdCMRswGQYDVQQIDBJHcmVhdGVyIE1hbmNoZXN0ZXIx
EDAOBgNVBAcMB1NhbGZvcmQxGjAYBgNVBAoMEUNvbW9kbyBDQSBMaW1pdGVkMSEwHwYDVQQDDBhB
QUEgQ2VydGlmaWNhdGUgU2VydmljZXMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC+
QJ30buHqdoccTUVEjr5GyIMGncEq/hgfjuQC+vOrXVCKFjELmgbQxXAizUktVGPMtm5oRgtT6stM
JMC8ck7q8RWu9FSaEgrDerIzYOLaiVXzIljz3tzP74OGooyUT59o8piQRoQnx3a/48w1LIteB2Rl
gsBIsKiR+WGfdiBQqJHHZrXreGIDVvCKGhPqMaMeoJn9OPb2JzJYbwf1a7j7FCuvt6rM1mNfc4za
BZmoOKjLF3g2UazpnvR4Oo3PD9lC4pgMqy+fDgHe75+ZSfEt36x0TRuYtUfF5SnR+ZAYx2KcvoPH
Jns+iiXHwN2d5jVoECCdj9je0sOEnA1e6C/JAgMBAAGjgcAwgb0wHQYDVR0OBBYEFKARCiM+lvEH
7OKvKe+CpX/QMKS0MA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MHsGA1UdHwR0MHIw
OKA2oDSGMmh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0FBQUNlcnRpZmljYXRlU2VydmljZXMuY3Js
MDagNKAyhjBodHRwOi8vY3JsLmNvbW9kby5uZXQvQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmww
DQYJKoZIhvcNAQEFBQADggEBAAhW/ALwm+j/pPrWe8ZEgM5PxMX2AFjMpra8FEloBHbo5u5d7AIP
YNaNUBhPJk4B4+awpe6/vHRUQb/9/BK4x09a9IlgBX9gtwVK8/bxwr/EuXSGti19a8zS80bdL8bg
asPDNAMsfZbdWsIOpwqZwQWLqwwv81w6z2w3VQmH3lNAbFjv/LarZW4E9hvcPOBaFcae2fFZSDAh
ZQNs7Okhc+ybA6HgN62gFRiP+roCzqcsqRATLNTlCCarIpdg+JBedNSimlO98qlo4KJuwtdssaMP
nr/raOdW8q7y4ys4OgmBtWuF174t7T8at7Jj4vViLILUagBBUPE5g5+V6TaWmG4wggTdMIIDxaAD
AgECAhBxkvvmGV+sTRKFdHE0ohinMA0GCSqGSIb3DQEBBQUAMHsxCzAJBgNVBAYTAkdCMRswGQYD
VQQIDBJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcMB1NhbGZvcmQxGjAYBgNVBAoMEUNvbW9k
byBDQSBMaW1pdGVkMSEwHwYDVQQDDBhBQUEgQ2VydGlmaWNhdGUgU2VydmljZXMwHhcNMDQwMTAx
MDAwMDAwWhcNMjgxMjMxMjM1OTU5WjCBrjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYD
VQQHEw5TYWx0IExha2UgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYD
VQQLExhodHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1VU0VSRmlyc3QtQ2xp
ZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBFbWFpbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBALI5haTyfatBO2JGN67NwWB1vDll+UoaR6K5zEjMapjVTTUZuaRC5c5J4oovHnzSMQfHTrSD
ZJ0uKdWiZMSFvYVRNXmkTmiQexx6pJKoF/KYFfKTzMmkMpW7DE8wvZigC4vlbhuiRvp4vKJvq1le
pS/Pytptqi/rrKGzaqq3Lmc1i3nhHmmI4uZGzaCl6r4LznY6eg6b6vzaJ1s9cx8i5khhxkzzabGo
Lhu21DEgLLyCio6kDqXXiUP8FlqvHXHXEVnauocNr/rz4cLwpMVnjNbWVDreCqS6A3ezZcj9HtN0
YqoYymiTHqGFfvVHZcv4TVcodNI0/zC27vZiMBSMLOsCAwEAAaOCAScwggEjMB8GA1UdIwQYMBaA
FKARCiM+lvEH7OKvKe+CpX/QMKS0MB0GA1UdDgQWBBSJgmd9xJ0mcABLtFBIfN49rgRufTAOBgNV
HQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUH
AwQwEQYDVR0gBAowCDAGBgRVHSAAMHsGA1UdHwR0MHIwOKA2oDSGMmh0dHA6Ly9jcmwuY29tb2Rv
Y2EuY29tL0FBQUNlcnRpZmljYXRlU2VydmljZXMuY3JsMDagNKAyhjBodHRwOi8vY3JsLmNvbW9k
by5uZXQvQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwEQYJYIZIAYb4QgEBBAQDAgEGMA0GCSqG
SIb3DQEBBQUAA4IBAQCdlcs8uH6lCcQevwvCx3aOOTyUxhCqTwzJ4KuEXYlU4GU7820cfDcsJVRf
liH8N4SRnRXcFE+Bz1Qda2xFYMct+ZdRTPlmyjyggoymyPDi6dRK+ew/VsnddozDggFPbADzHhph
dARHA6nGQFeRvGUixSdnT1fbZFrZjR+6hi/0Bq6cae3p9M8pF9jgSp8aIC+XTFG7RgfEijdOIOMJ
MWjHnsSLneh+EbwyaBCWEZhE2CpRYE2I63Q630MGMsg5Vow6EVLTQaRDA/Tt7zMn2zngFE4mydj1
OeKJuJNdtykmQeqzm66D/Hd1yujKtf7iZUpjPkTE0MNeh3OpmByvfxV/MIIGNTCCBR2gAwIBAgIR
APUHLR99vIaqc6YYFoh++L8wDQYJKoZIhvcNAQEFBQAwga4xCzAJBgNVBAYTAlVTMQswCQYDVQQI
EwJVVDEXMBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNVBAoTFVRoZSBVU0VSVFJVU1QgTmV0
d29yazEhMB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3QuY29tMTYwNAYDVQQDEy1VVE4tVVNF
UkZpcnN0LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1haWwwHhcNMTAwMjA4MDAwMDAwWhcN
MTEwMjA4MjM1OTU5WjCB4DE1MDMGA1UECxMsQ29tb2RvIFRydXN0IE5ldHdvcmsgLSBQRVJTT05B
IE5PVCBWQUxJREFURUQxRjBEBgNVBAsTPVRlcm1zIGFuZCBDb25kaXRpb25zIG9mIHVzZTogaHR0
cDovL3d3dy5jb21vZG8ubmV0L3JlcG9zaXRvcnkxHzAdBgNVBAsTFihjKTIwMDMgQ29tb2RvIExp
bWl0ZWQxFzAVBgNVBAMTDkV2ZXJ0IExhbW1lcnRzMSUwIwYJKoZIhvcNAQkBFhZldmVydC5sYW1t
ZXJ0c0BzYXJhLm5sMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtYAj4sXuBqEwPxYY
BMkYLPGrnqT35USIERoWs26xtSY+dLVc7HqaHDZ1lU5m9wFx/Jqph1urlRuAcCj03kGSS23Ta6kt
HTRMYRCHI3nOLfEEbH4POT+wDytORUHNeEKp/PrBn1kmKk+dk+bJI7MNdy2XkiEeO+OqKmLiBGkT
glIxngOeqgqR2IvmVv2hLAyzq8Ax20inwveZsDMa7QdQLSVUaX1kSRSihwz4Jw9X/K+6oA3ZLGp9
pZYnFHBNTHM2DR/C7dNwQgbZS7TY/Jb1kObYNhwD2JzzJlkoe0blR17MeJkF7/+Y414j0AdPFB7I
ggZ4Tm7Maheer9cIQSsvIQIDAQABo4ICGDCCAhQwHwYDVR0jBBgwFoAUiYJnfcSdJnAAS7RQSHze
Pa4Ebn0wHQYDVR0OBBYEFNTyUAqBNg9JXuiPHa9jfLvRM3IyMA4GA1UdDwEB/wQEAwIFoDAMBgNV
HRMBAf8EAjAAMCAGA1UdJQQZMBcGCCsGAQUFBwMEBgsrBgEEAbIxAQMFAjARBglghkgBhvhCAQEE
BAMCBSAwRgYDVR0gBD8wPTA7BgwrBgEEAbIxAQIBAQEwKzApBggrBgEFBQcCARYdaHR0cHM6Ly9z
ZWN1cmUuY29tb2RvLm5ldC9DUFMwgaUGA1UdHwSBnTCBmjBMoEqgSIZGaHR0cDovL2NybC5jb21v
ZG9jYS5jb20vVVROLVVTRVJGaXJzdC1DbGllbnRBdXRoZW50aWNhdGlvbmFuZEVtYWlsLmNybDBK
oEigRoZEaHR0cDovL2NybC5jb21vZG8ubmV0L1VUTi1VU0VSRmlyc3QtQ2xpZW50QXV0aGVudGlj
YXRpb25hbmRFbWFpbC5jcmwwbAYIKwYBBQUHAQEEYDBeMDYGCCsGAQUFBzAChipodHRwOi8vY3J0
LmNvbW9kb2NhLmNvbS9VVE5BQUFDbGllbnRDQS5jcnQwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3Nw
LmNvbW9kb2NhLmNvbTAhBgNVHREEGjAYgRZldmVydC5sYW1tZXJ0c0BzYXJhLm5sMA0GCSqGSIb3
DQEBBQUAA4IBAQBaJvuiBPgqdq9MEYH9dj2vW8hprAIBEnOOvfXdoP604kBhSbj3s/l0ZQyQY35W
YVAltuJQ365uJKigPQHxS+slpUOAJPEVNmwPUIW0GAjxCPxgLdJgAc4jOV8f7oF3pnfI4ukCmjPN
LpaylZzMhFt+t6zDWiMtUqyxK/4on62LoLp2D88lxwxI3q3i00bPlV0wZ+mjzS7vrQGcUgDPWEBL
FD0I3pR+Z1EH1qjelBmBJV9paVzfzmoU93LIDJA7/MEUXd3JiMZGEoVd2qE5qIZO4fNJ+Cyo6tu1
ECSQxoL6qN+sBwekiFA9Q/zlUp0HfU6Jt8WmtZ44SOh67LcmCL4ZMYIEaDCCBGQCAQEwgcQwga4x
CzAJBgNVBAYTAlVTMQswCQYDVQQIEwJVVDEXMBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNV
BAoTFVRoZSBVU0VSVFJVU1QgTmV0d29yazEhMB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3Qu
Y29tMTYwNAYDVQQDEy1VVE4tVVNFUkZpcnN0LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1h
aWwCEQD1By0ffbyGqnOmGBaIfvi/MAkGBSsOAwIaBQCgggJ4MBgGCSqGSIb3DQEJAzELBgkqhkiG
9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTEwMTExMTA5NDI0NlowIwYJKoZIhvcNAQkEMRYEFBrbr6W/
gARVijjOYgei2/QB8mCxMGcGCSqGSIb3DQEJDzFaMFgwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwIC
AgCAMA0GCCqGSIb3DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMAcGBSsOAwIaMAoGCCqG
SIb3DQIFMIHVBgkrBgEEAYI3EAQxgccwgcQwga4xCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJVVDEX
MBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNVBAoTFVRoZSBVU0VSVFJVU1QgTmV0d29yazEh
MB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3QuY29tMTYwNAYDVQQDEy1VVE4tVVNFUkZpcnN0
LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1haWwCEQD1By0ffbyGqnOmGBaIfvi/MIHXBgsq
hkiG9w0BCRACCzGBx6CBxDCBrjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5T
YWx0IExha2UgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYDVQQLExho
dHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1VU0VSRmlyc3QtQ2xpZW50IEF1
dGhlbnRpY2F0aW9uIGFuZCBFbWFpbAIRAPUHLR99vIaqc6YYFoh++L8wDQYJKoZIhvcNAQEBBQAE
ggEAV0MRMdGx1k4G4InyB+SsgkbNBEQI95qX98GucAwmdS6rbw9XQEd7cBh/gqaftMGgPoTv6Unk
mI/SWCow1DSFsNCOnijm+Ej4TWI35cm9qMEFj3EiupGfJq/IOaBrB5oJHt5nG5yggZxPgjcK0bT7
VsxwOd/rSMvqx12Y946Mb+HUJHCDWTVFCkzROmnwYht25Sw0wXKwKZHY68udbwu0Om49lzdlYjBz
d/pLSEYHpJQBX3yGCHE9A+Iqw8TLLj+aeRfPMpvcHu8Ix0oByv7D114bU6Use7hcrNdkly55ZPeQ
Xl4xGsAKKghG09zBKmYnaQQ1SZczDx7AYvbyGZRpEwAAAAAAAA==

------=_NextPart_000_005E_01CB818D.2D9432D0--

From general-return-2367-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 11 10:00:49 2010
Return-Path: <general-return-2367-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45392 invoked from network); 11 Nov 2010 10:00:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Nov 2010 10:00:48 -0000
Received: (qmail 89868 invoked by uid 500); 11 Nov 2010 10:01:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89572 invoked by uid 500); 11 Nov 2010 10:01:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89558 invoked by uid 99); 11 Nov 2010 10:01:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Nov 2010 10:01:15 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Nov 2010 10:01:09 +0000
Received: by fxm10 with SMTP id 10so795027fxm.35
        for <general@hadoop.apache.org>; Thu, 11 Nov 2010 02:00:49 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type;
        bh=RMZkmaZxl7scEe0rbsoaQeCF0KskpFBV8KWLaOxzDGM=;
        b=Kj32OAg1wNFpvwl5uUcAGHQ1D9J54aXDDRW4b6KeuATb+1nIBFh34rlLcZrPqLmHbs
         Tgsr36mSUAv+Clu67mWFgVB5ez0pPaedPclUhXxL+2VYpe20NLDb4B9bi+JN3jz4Xegl
         5qQCUyrB8/TqnSOKQgSyapqz/22A+lcwnNB9I=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=HoliqwT9gpVdTbbmvUpqVDN0+dUfcfQf8Pt364+ShliDRF+Kj0JgLXHNBxFfucxozg
         SzC+yFKHPIHmM9hMEPWKC24MFbwQb/9jTYdSNnpNBvf0Zp8LpycZ8iTHidvSOtmVYhkU
         3e4SbdAppW26V0Yjegk030/l/uw7ZF/ivd+8o=
Received: by 10.223.89.142 with SMTP id e14mr104658fam.143.1289469649390; Thu,
 11 Nov 2010 02:00:49 -0800 (PST)
MIME-Version: 1.0
Received: by 10.223.113.145 with HTTP; Thu, 11 Nov 2010 02:00:29 -0800 (PST)
In-Reply-To: <ADF94D8555C7A246B86A633685E0178AB90A474AB2@planck.ka.sara.nl>
References: <ADF94D8555C7A246B86A633685E0178AB90A474AB2@planck.ka.sara.nl>
From: Harsh J <qwertymaniac@gmail.com>
Date: Thu, 11 Nov 2010 15:30:29 +0530
Message-ID: <AANLkTinnUyzYeknsWpB1XH1jHDahxGPKTPZqZ5s0Z9a_@mail.gmail.com>
Subject: Re: CDH3 / 0.20 config
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hello,

On Thu, Nov 11, 2010 at 3:12 PM, Evert Lammerts <Evert.Lammerts@sara.nl> wrote:
> Hi all,
>
> Can I use these pages as a reference for configuration? Are they not
> outdated for / ahead of version 0.20?
>
> core-site.xml:
> http://hadoop.apache.org/common/docs/current/core-default.html
> hdfs-site.xml: http://hadoop.apache.org/hdfs/docs/current/hdfs-default.html
> mapred-site.xml:
> http://hadoop.apache.org/mapreduce/docs/current/mapred-default.html

For 0.20.2, see these specific urls:
core-site.xml:
http://hadoop.apache.org/common/docs/r0.20.2/core-default.html
hdfs-site.xml:
http://hadoop.apache.org/hdfs/docs/r0.20.2/hdfs-default.html
mapred-site.xml:
http://hadoop.apache.org/mapreduce/docs/r0.20.0/mapred-default.html

In trunk, many of these properties have been renamed, and their former
counterparts deprecated.

-- 
Harsh J
www.harshj.com

From general-return-2368-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 11 10:16:56 2010
Return-Path: <general-return-2368-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58075 invoked from network); 11 Nov 2010 10:16:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Nov 2010 10:16:56 -0000
Received: (qmail 12714 invoked by uid 500); 11 Nov 2010 10:17:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12415 invoked by uid 500); 11 Nov 2010 10:17:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12407 invoked by uid 99); 11 Nov 2010 10:17:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Nov 2010 10:17:24 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [145.100.8.49] (HELO smtp.sara.nl) (145.100.8.49)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Nov 2010 10:17:16 +0000
Received: from planck.ka.sara.nl (145.100.8.32) by sara-exch-fe1.ka.sara.nl
 (145.100.8.49) with Microsoft SMTP Server (TLS) id 14.0.702.0; Thu, 11 Nov
 2010 11:16:53 +0100
Received: from planck.ka.sara.nl ([145.100.8.32]) by planck.ka.sara.nl
 ([145.100.8.32]) with mapi; Thu, 11 Nov 2010 11:16:53 +0100
From: Evert Lammerts <Evert.Lammerts@sara.nl>
To: "'general@hadoop.apache.org'" <general@hadoop.apache.org>
Date: Thu, 11 Nov 2010 11:14:52 +0100
Subject: RE: CDH3 / 0.20 config
Thread-Topic: CDH3 / 0.20 config
Thread-Index: AcuBh2pfifx3QHosTS+luxq8X56VrAAAaX/A
Message-ID: <ADF94D8555C7A246B86A633685E0178AB90A474ABB@planck.ka.sara.nl>
References: <ADF94D8555C7A246B86A633685E0178AB90A474AB2@planck.ka.sara.nl>
 <AANLkTinnUyzYeknsWpB1XH1jHDahxGPKTPZqZ5s0Z9a_@mail.gmail.com>
In-Reply-To: <AANLkTinnUyzYeknsWpB1XH1jHDahxGPKTPZqZ5s0Z9a_@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/signed; protocol="application/x-pkcs7-signature";
	micalg=SHA1; boundary="----=_NextPart_000_00CC_01CB8191.A9C3BED0"
MIME-Version: 1.0

------=_NextPart_000_00CC_01CB8191.A9C3BED0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit

> hdfs-site.xml:
> http://hadoop.apache.org/hdfs/docs/r0.20.2/hdfs-default.html
> mapred-site.xml:
> http://hadoop.apache.org/mapreduce/docs/r0.20.0/mapred-default.html

These two don't exist. With r0.21 the URLs do work, but is that info valid
for 0.20?

Cheers, Evert

------=_NextPart_000_00CC_01CB8191.A9C3BED0
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIPUDCCBDIw
ggMaoAMCAQICAQEwDQYJKoZIhvcNAQEFBQAwezELMAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0
ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0
ZWQxITAfBgNVBAMMGEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczAeFw0wNDAxMDEwMDAwMDBaFw0y
ODEyMzEyMzU5NTlaMHsxCzAJBgNVBAYTAkdCMRswGQYDVQQIDBJHcmVhdGVyIE1hbmNoZXN0ZXIx
EDAOBgNVBAcMB1NhbGZvcmQxGjAYBgNVBAoMEUNvbW9kbyBDQSBMaW1pdGVkMSEwHwYDVQQDDBhB
QUEgQ2VydGlmaWNhdGUgU2VydmljZXMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC+
QJ30buHqdoccTUVEjr5GyIMGncEq/hgfjuQC+vOrXVCKFjELmgbQxXAizUktVGPMtm5oRgtT6stM
JMC8ck7q8RWu9FSaEgrDerIzYOLaiVXzIljz3tzP74OGooyUT59o8piQRoQnx3a/48w1LIteB2Rl
gsBIsKiR+WGfdiBQqJHHZrXreGIDVvCKGhPqMaMeoJn9OPb2JzJYbwf1a7j7FCuvt6rM1mNfc4za
BZmoOKjLF3g2UazpnvR4Oo3PD9lC4pgMqy+fDgHe75+ZSfEt36x0TRuYtUfF5SnR+ZAYx2KcvoPH
Jns+iiXHwN2d5jVoECCdj9je0sOEnA1e6C/JAgMBAAGjgcAwgb0wHQYDVR0OBBYEFKARCiM+lvEH
7OKvKe+CpX/QMKS0MA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MHsGA1UdHwR0MHIw
OKA2oDSGMmh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0FBQUNlcnRpZmljYXRlU2VydmljZXMuY3Js
MDagNKAyhjBodHRwOi8vY3JsLmNvbW9kby5uZXQvQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmww
DQYJKoZIhvcNAQEFBQADggEBAAhW/ALwm+j/pPrWe8ZEgM5PxMX2AFjMpra8FEloBHbo5u5d7AIP
YNaNUBhPJk4B4+awpe6/vHRUQb/9/BK4x09a9IlgBX9gtwVK8/bxwr/EuXSGti19a8zS80bdL8bg
asPDNAMsfZbdWsIOpwqZwQWLqwwv81w6z2w3VQmH3lNAbFjv/LarZW4E9hvcPOBaFcae2fFZSDAh
ZQNs7Okhc+ybA6HgN62gFRiP+roCzqcsqRATLNTlCCarIpdg+JBedNSimlO98qlo4KJuwtdssaMP
nr/raOdW8q7y4ys4OgmBtWuF174t7T8at7Jj4vViLILUagBBUPE5g5+V6TaWmG4wggTdMIIDxaAD
AgECAhBxkvvmGV+sTRKFdHE0ohinMA0GCSqGSIb3DQEBBQUAMHsxCzAJBgNVBAYTAkdCMRswGQYD
VQQIDBJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcMB1NhbGZvcmQxGjAYBgNVBAoMEUNvbW9k
byBDQSBMaW1pdGVkMSEwHwYDVQQDDBhBQUEgQ2VydGlmaWNhdGUgU2VydmljZXMwHhcNMDQwMTAx
MDAwMDAwWhcNMjgxMjMxMjM1OTU5WjCBrjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYD
VQQHEw5TYWx0IExha2UgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYD
VQQLExhodHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1VU0VSRmlyc3QtQ2xp
ZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBFbWFpbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBALI5haTyfatBO2JGN67NwWB1vDll+UoaR6K5zEjMapjVTTUZuaRC5c5J4oovHnzSMQfHTrSD
ZJ0uKdWiZMSFvYVRNXmkTmiQexx6pJKoF/KYFfKTzMmkMpW7DE8wvZigC4vlbhuiRvp4vKJvq1le
pS/Pytptqi/rrKGzaqq3Lmc1i3nhHmmI4uZGzaCl6r4LznY6eg6b6vzaJ1s9cx8i5khhxkzzabGo
Lhu21DEgLLyCio6kDqXXiUP8FlqvHXHXEVnauocNr/rz4cLwpMVnjNbWVDreCqS6A3ezZcj9HtN0
YqoYymiTHqGFfvVHZcv4TVcodNI0/zC27vZiMBSMLOsCAwEAAaOCAScwggEjMB8GA1UdIwQYMBaA
FKARCiM+lvEH7OKvKe+CpX/QMKS0MB0GA1UdDgQWBBSJgmd9xJ0mcABLtFBIfN49rgRufTAOBgNV
HQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUH
AwQwEQYDVR0gBAowCDAGBgRVHSAAMHsGA1UdHwR0MHIwOKA2oDSGMmh0dHA6Ly9jcmwuY29tb2Rv
Y2EuY29tL0FBQUNlcnRpZmljYXRlU2VydmljZXMuY3JsMDagNKAyhjBodHRwOi8vY3JsLmNvbW9k
by5uZXQvQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwEQYJYIZIAYb4QgEBBAQDAgEGMA0GCSqG
SIb3DQEBBQUAA4IBAQCdlcs8uH6lCcQevwvCx3aOOTyUxhCqTwzJ4KuEXYlU4GU7820cfDcsJVRf
liH8N4SRnRXcFE+Bz1Qda2xFYMct+ZdRTPlmyjyggoymyPDi6dRK+ew/VsnddozDggFPbADzHhph
dARHA6nGQFeRvGUixSdnT1fbZFrZjR+6hi/0Bq6cae3p9M8pF9jgSp8aIC+XTFG7RgfEijdOIOMJ
MWjHnsSLneh+EbwyaBCWEZhE2CpRYE2I63Q630MGMsg5Vow6EVLTQaRDA/Tt7zMn2zngFE4mydj1
OeKJuJNdtykmQeqzm66D/Hd1yujKtf7iZUpjPkTE0MNeh3OpmByvfxV/MIIGNTCCBR2gAwIBAgIR
APUHLR99vIaqc6YYFoh++L8wDQYJKoZIhvcNAQEFBQAwga4xCzAJBgNVBAYTAlVTMQswCQYDVQQI
EwJVVDEXMBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNVBAoTFVRoZSBVU0VSVFJVU1QgTmV0
d29yazEhMB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3QuY29tMTYwNAYDVQQDEy1VVE4tVVNF
UkZpcnN0LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1haWwwHhcNMTAwMjA4MDAwMDAwWhcN
MTEwMjA4MjM1OTU5WjCB4DE1MDMGA1UECxMsQ29tb2RvIFRydXN0IE5ldHdvcmsgLSBQRVJTT05B
IE5PVCBWQUxJREFURUQxRjBEBgNVBAsTPVRlcm1zIGFuZCBDb25kaXRpb25zIG9mIHVzZTogaHR0
cDovL3d3dy5jb21vZG8ubmV0L3JlcG9zaXRvcnkxHzAdBgNVBAsTFihjKTIwMDMgQ29tb2RvIExp
bWl0ZWQxFzAVBgNVBAMTDkV2ZXJ0IExhbW1lcnRzMSUwIwYJKoZIhvcNAQkBFhZldmVydC5sYW1t
ZXJ0c0BzYXJhLm5sMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtYAj4sXuBqEwPxYY
BMkYLPGrnqT35USIERoWs26xtSY+dLVc7HqaHDZ1lU5m9wFx/Jqph1urlRuAcCj03kGSS23Ta6kt
HTRMYRCHI3nOLfEEbH4POT+wDytORUHNeEKp/PrBn1kmKk+dk+bJI7MNdy2XkiEeO+OqKmLiBGkT
glIxngOeqgqR2IvmVv2hLAyzq8Ax20inwveZsDMa7QdQLSVUaX1kSRSihwz4Jw9X/K+6oA3ZLGp9
pZYnFHBNTHM2DR/C7dNwQgbZS7TY/Jb1kObYNhwD2JzzJlkoe0blR17MeJkF7/+Y414j0AdPFB7I
ggZ4Tm7Maheer9cIQSsvIQIDAQABo4ICGDCCAhQwHwYDVR0jBBgwFoAUiYJnfcSdJnAAS7RQSHze
Pa4Ebn0wHQYDVR0OBBYEFNTyUAqBNg9JXuiPHa9jfLvRM3IyMA4GA1UdDwEB/wQEAwIFoDAMBgNV
HRMBAf8EAjAAMCAGA1UdJQQZMBcGCCsGAQUFBwMEBgsrBgEEAbIxAQMFAjARBglghkgBhvhCAQEE
BAMCBSAwRgYDVR0gBD8wPTA7BgwrBgEEAbIxAQIBAQEwKzApBggrBgEFBQcCARYdaHR0cHM6Ly9z
ZWN1cmUuY29tb2RvLm5ldC9DUFMwgaUGA1UdHwSBnTCBmjBMoEqgSIZGaHR0cDovL2NybC5jb21v
ZG9jYS5jb20vVVROLVVTRVJGaXJzdC1DbGllbnRBdXRoZW50aWNhdGlvbmFuZEVtYWlsLmNybDBK
oEigRoZEaHR0cDovL2NybC5jb21vZG8ubmV0L1VUTi1VU0VSRmlyc3QtQ2xpZW50QXV0aGVudGlj
YXRpb25hbmRFbWFpbC5jcmwwbAYIKwYBBQUHAQEEYDBeMDYGCCsGAQUFBzAChipodHRwOi8vY3J0
LmNvbW9kb2NhLmNvbS9VVE5BQUFDbGllbnRDQS5jcnQwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3Nw
LmNvbW9kb2NhLmNvbTAhBgNVHREEGjAYgRZldmVydC5sYW1tZXJ0c0BzYXJhLm5sMA0GCSqGSIb3
DQEBBQUAA4IBAQBaJvuiBPgqdq9MEYH9dj2vW8hprAIBEnOOvfXdoP604kBhSbj3s/l0ZQyQY35W
YVAltuJQ365uJKigPQHxS+slpUOAJPEVNmwPUIW0GAjxCPxgLdJgAc4jOV8f7oF3pnfI4ukCmjPN
LpaylZzMhFt+t6zDWiMtUqyxK/4on62LoLp2D88lxwxI3q3i00bPlV0wZ+mjzS7vrQGcUgDPWEBL
FD0I3pR+Z1EH1qjelBmBJV9paVzfzmoU93LIDJA7/MEUXd3JiMZGEoVd2qE5qIZO4fNJ+Cyo6tu1
ECSQxoL6qN+sBwekiFA9Q/zlUp0HfU6Jt8WmtZ44SOh67LcmCL4ZMYIEaDCCBGQCAQEwgcQwga4x
CzAJBgNVBAYTAlVTMQswCQYDVQQIEwJVVDEXMBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNV
BAoTFVRoZSBVU0VSVFJVU1QgTmV0d29yazEhMB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3Qu
Y29tMTYwNAYDVQQDEy1VVE4tVVNFUkZpcnN0LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1h
aWwCEQD1By0ffbyGqnOmGBaIfvi/MAkGBSsOAwIaBQCgggJ4MBgGCSqGSIb3DQEJAzELBgkqhkiG
9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTEwMTExMTEwMTQ1MlowIwYJKoZIhvcNAQkEMRYEFP5aWskt
YA8915Wc7Q8l15YJj5OCMGcGCSqGSIb3DQEJDzFaMFgwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwIC
AgCAMA0GCCqGSIb3DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMAcGBSsOAwIaMAoGCCqG
SIb3DQIFMIHVBgkrBgEEAYI3EAQxgccwgcQwga4xCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJVVDEX
MBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNVBAoTFVRoZSBVU0VSVFJVU1QgTmV0d29yazEh
MB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3QuY29tMTYwNAYDVQQDEy1VVE4tVVNFUkZpcnN0
LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1haWwCEQD1By0ffbyGqnOmGBaIfvi/MIHXBgsq
hkiG9w0BCRACCzGBx6CBxDCBrjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5T
YWx0IExha2UgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYDVQQLExho
dHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1VU0VSRmlyc3QtQ2xpZW50IEF1
dGhlbnRpY2F0aW9uIGFuZCBFbWFpbAIRAPUHLR99vIaqc6YYFoh++L8wDQYJKoZIhvcNAQEBBQAE
ggEAPS4AUfTon2nU3kENbNOEejBKbL+ziYi0HvkClm2Osgb1FQDrXFVjHtABqeuqn9ylGxGitPuv
5CPzb+TpO5qhkIqx3GJEO62nIlPGt632rYTxcPR395moYK+iUKRlMvlHA1zK6dQWRvU2BI/ujJKD
TaAbjxeL1Du7A3l/WD3EiOWCdjaIm7MkV8KxXq5ZvLpsj+0FID63wmT/zG9QrUuPDQ0J9+4F3Ena
+YPhV/nd/4JzRzC6VMnx+vJZdO+G01MS+Gs0ns5oea1WkFpd2i0znYNzkAB/fbMGitxbfLC+EIKH
Kk49h0RTRqF71bnA7FSMD4ycT8m6RWkn0Row0ts86gAAAAAAAA==

------=_NextPart_000_00CC_01CB8191.A9C3BED0--

From general-return-2369-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 11 10:41:53 2010
Return-Path: <general-return-2369-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69650 invoked from network); 11 Nov 2010 10:41:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Nov 2010 10:41:53 -0000
Received: (qmail 45695 invoked by uid 500); 11 Nov 2010 10:42:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45511 invoked by uid 500); 11 Nov 2010 10:42:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45499 invoked by uid 99); 11 Nov 2010 10:42:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Nov 2010 10:42:21 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [98.139.91.98] (HELO nm28.bullet.mail.sp2.yahoo.com) (98.139.91.98)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 11 Nov 2010 10:42:14 +0000
Received: from [98.139.91.61] by nm28.bullet.mail.sp2.yahoo.com with NNFMP; 11 Nov 2010 10:41:53 -0000
Received: from [98.139.91.33] by tm1.bullet.mail.sp2.yahoo.com with NNFMP; 11 Nov 2010 10:41:53 -0000
Received: from [127.0.0.1] by omp1033.mail.sp2.yahoo.com with NNFMP; 11 Nov 2010 10:41:53 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 772645.31588.bm@omp1033.mail.sp2.yahoo.com
Received: (qmail 91219 invoked by uid 60001); 11 Nov 2010 10:41:53 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1289472113; bh=pYhbeCIWNO/Z3aCZEMC1ycY2aQCZrOwS/4lF+Y3vprk=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=NeczgIVjwZMrjb/cHldcFHn4lvs9WaN1FUEVOJ1tacg4m6UmmhKhb12RMPTNUjiUkFAzHokQ9tnmrx0iQ4UZ2OOXW39MQLbSkYgmCnW7E3MwkHWzwgsmT+dKbVMChpT0QGPDR1xEAYwfGWauoAj0HEgnuTIPETfkhlbInb7P1dk=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=KThyA7Yg5YA9d4KuSIUnHgV0l05phHStn2LrvCVRQP25z71profwwLM1UC6tCvkobrAfEL9gwBF1DtykGJ+POfs+XHl+X3j+6A/Lakp91Su0ck7vyi91QfofUsLvfVjbEWwPOZMlkmroeIHdKANi3Z/Nv46fm8apzdaDRZiqNCE=;
Message-ID: <365982.91127.qm@web112117.mail.gq1.yahoo.com>
X-YMail-OSG: deVw.mAVM1nsRYjzQhdygWg9lx8OcP2QxXg5j0pbaKvZaHV
 T3CIs0OGR
Received: from [129.132.85.169] by web112117.mail.gq1.yahoo.com via HTTP; Thu, 11 Nov 2010 02:41:52 PST
X-Mailer: YahooMailClassic/11.4.9 YahooMailWebService/0.8.107.285259
Date: Thu, 11 Nov 2010 02:41:52 -0800 (PST)
From: Grandl Robert <rgrandl@yahoo.com>
Subject: Hadoop - stop-dfs.sh, stop-mapred.sh does not stop all instances
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1855586541-1289472112=:91127"

--0-1855586541-1289472112=:91127
Content-Type: text/plain; charset=us-ascii

Hi all,

Probably is a stupid question but I don't figure out what happens.

For example I start on master, start-dfs.sh, start-mapred.sh

However, when I want to stop: stop-mapred, stop-dfs, only one slave instance is killed. 

For example in on the master I am doing something like that:

rgrandl@ikq01:~/hadoop/hadoop-0.21.0$ bin/stop-mapred.sh 
stopping jobtracker
ikq02.rrr: no tasktracker to stop
ikq03.rrr: stopping tasktracker

But I have a tasktracker on ikq02 as well which is not stopped anyway.
rgrandl@ikq02:~$ jps
2462 Jps
2211 TaskTracker
2109 DataNode

Same thing for stop-dfs.sh


Always the tasktracker, datanode instances are killed only on one slave node, even if I have maybe 20 slaves or more. 

Thanks for any clue,
Robert





      
--0-1855586541-1289472112=:91127--

From general-return-2370-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 11 10:45:26 2010
Return-Path: <general-return-2370-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72437 invoked from network); 11 Nov 2010 10:45:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Nov 2010 10:45:26 -0000
Received: (qmail 50351 invoked by uid 500); 11 Nov 2010 10:45:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50149 invoked by uid 500); 11 Nov 2010 10:45:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50141 invoked by uid 99); 11 Nov 2010 10:45:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Nov 2010 10:45:55 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Nov 2010 10:45:48 +0000
Received: by fxm10 with SMTP id 10so817974fxm.35
        for <general@hadoop.apache.org>; Thu, 11 Nov 2010 02:45:28 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type;
        bh=lchiolcccRzt1813Qy5vhNwd4Er3q7PCgjQyc9l/3/g=;
        b=ejX+3N0kfnHZh76nt1J8EsqIaAFdK7+WZegd4SmTFznBOYBlKVnH14291BYdfhOoym
         /n1m/ESZQsu2PZopDsB7OCalasRTn86B3DTJvoK3jfEl0bJ8qrCiIXBZ/8wUalGf2Ks+
         Yo9cb1wWHtFuFAtRhHa2EttMdni3jrdm5v7TU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=W0Kbv+mLWu0cmJyC1TaOejwvjOPH5lvh8cNDNABs+B/Y5cWbKyqcho4QUXTNo5db0u
         hz6c/0VcQw5Df94n6sjN+oQF6GpDl2X3qQ9kwh+sU+am27reYOC+BqGSviEBq9y1Kgvk
         QhOfwNIaNqwJmKJVLyUIR007Hyt2cneXZX7tU=
Received: by 10.223.86.76 with SMTP id r12mr115610fal.86.1289472328554; Thu,
 11 Nov 2010 02:45:28 -0800 (PST)
MIME-Version: 1.0
Received: by 10.223.113.145 with HTTP; Thu, 11 Nov 2010 02:45:08 -0800 (PST)
In-Reply-To: <ADF94D8555C7A246B86A633685E0178AB90A474ABB@planck.ka.sara.nl>
References: <ADF94D8555C7A246B86A633685E0178AB90A474AB2@planck.ka.sara.nl>
 <AANLkTinnUyzYeknsWpB1XH1jHDahxGPKTPZqZ5s0Z9a_@mail.gmail.com> <ADF94D8555C7A246B86A633685E0178AB90A474ABB@planck.ka.sara.nl>
From: Harsh J <qwertymaniac@gmail.com>
Date: Thu, 11 Nov 2010 16:15:08 +0530
Message-ID: <AANLkTinMLFJ15Jhvak6ekB67TvaC==g8w8y2Z28tppAb@mail.gmail.com>
Subject: Re: CDH3 / 0.20 config
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Thu, Nov 11, 2010 at 3:44 PM, Evert Lammerts <Evert.Lammerts@sara.nl> wrote:
>> hdfs-site.xml:
>> http://hadoop.apache.org/hdfs/docs/r0.20.2/hdfs-default.html
>> mapred-site.xml:
>> http://hadoop.apache.org/mapreduce/docs/r0.20.0/mapred-default.html
>
> These two don't exist. With r0.21 the URLs do work, but is that info valid
> for 0.20?

Oops, forgot that the split happened later. These should work fine,
and apply properly to 0.20.2:
http://hadoop.apache.org/common/docs/r0.20.2/hdfs-default.html
http://hadoop.apache.org/common/docs/r0.20.2/mapred-default.html

r0.21 is a minor release, so using their config page should be okay
too. I do not know how many new props have been added though.

>
> Cheers, Evert
>



-- 
Harsh J
www.harshj.com

From general-return-2371-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 11 17:20:36 2010
Return-Path: <general-return-2371-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5942 invoked from network); 11 Nov 2010 17:20:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Nov 2010 17:20:36 -0000
Received: (qmail 91668 invoked by uid 500); 11 Nov 2010 17:21:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91566 invoked by uid 500); 11 Nov 2010 17:21:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91558 invoked by uid 99); 11 Nov 2010 17:21:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Nov 2010 17:21:05 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Nov 2010 17:21:00 +0000
Received: by vws16 with SMTP id 16so455123vws.35
        for <general@hadoop.apache.org>; Thu, 11 Nov 2010 09:20:39 -0800 (PST)
MIME-Version: 1.0
Received: by 10.229.91.9 with SMTP id k9mr931112qcm.248.1289496038746; Thu, 11
 Nov 2010 09:20:38 -0800 (PST)
Received: by 10.229.225.137 with HTTP; Thu, 11 Nov 2010 09:20:38 -0800 (PST)
In-Reply-To: <ADF94D8555C7A246B86A633685E0178AB90A474AB2@planck.ka.sara.nl>
References: <ADF94D8555C7A246B86A633685E0178AB90A474AB2@planck.ka.sara.nl>
Date: Thu, 11 Nov 2010 09:20:38 -0800
Message-ID: <AANLkTi=t3ytpuYs4-ZiJZchn40pLVOWd-avna1C9gRdi@mail.gmail.com>
Subject: Re: CDH3 / 0.20 config
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636164a2b542c8d0494ca312a

--001636164a2b542c8d0494ca312a
Content-Type: text/plain; charset=ISO-8859-1

On Thu, Nov 11, 2010 at 1:42 AM, Evert Lammerts <Evert.Lammerts@sara.nl>wrote:

> Hi all,
>
> Can I use these pages as a reference for configuration? Are they not
> outdated for / ahead of version 0.20?
>
> core-site.xml:
> http://hadoop.apache.org/common/docs/current/core-default.html
> hdfs-site.xml:
> http://hadoop.apache.org/hdfs/docs/current/hdfs-default.html
> mapred-site.xml:
> http://hadoop.apache.org/mapreduce/docs/current/mapred-default.html
>
> Are there any such pages for other config files? (fair- /
> capacity-scheduler, hadoop-policy, mapred-queue-acls)
>
>
These are in the docs, eg
http://hadoop.apache.org/common/docs/r0.20.2/service_level_auth.html
http://hadoop.apache.org/common/docs/r0.20.2/fair_scheduler.html
http://hadoop.apache.org/common/docs/r0.20.2/capacity_scheduler.html

A good configuration overview is lacking at the moment, as far as I can see.
> If time permits, I might provide one for 0.20.
>

That would be great.

Thanks,
Eli


>
> Cheers,
>
> Evert Lammerts
> Consultant eScience Support
> SARA Computing & Network Services
> High Performance Computing & Visualization
>
> Phone: +31 20 888 4101
> Email: evert.lammerts@sara.nl
> http://www.sara.nl
>
>
>

--001636164a2b542c8d0494ca312a--

From general-return-2372-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 11 20:28:40 2010
Return-Path: <general-return-2372-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 7268 invoked from network); 11 Nov 2010 20:28:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Nov 2010 20:28:40 -0000
Received: (qmail 22486 invoked by uid 500); 11 Nov 2010 20:29:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22371 invoked by uid 500); 11 Nov 2010 20:29:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22362 invoked by uid 99); 11 Nov 2010 20:29:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Nov 2010 20:29:10 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.161.176] (HELO mail-gx0-f176.google.com) (209.85.161.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Nov 2010 20:29:04 +0000
Received: by gxk4 with SMTP id 4so1667946gxk.35
        for <general@hadoop.apache.org>; Thu, 11 Nov 2010 12:28:43 -0800 (PST)
Received: by 10.42.136.8 with SMTP id r8mr1422434ict.84.1289507322010;
        Thu, 11 Nov 2010 12:28:42 -0800 (PST)
Received: from localhost (c-24-4-185-157.hsd1.ca.comcast.net [24.4.185.157])
        by mx.google.com with ESMTPS id 8sm2807080iba.16.2010.11.11.12.28.40
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Thu, 11 Nov 2010 12:28:40 -0800 (PST)
Date: Thu, 11 Nov 2010 12:28:38 -0800
From: Konstantin Boudnik <cos@boudnik.org>
To: general@hadoop.apache.org
Subject: Re: Hadoop - stop-dfs.sh, stop-mapred.sh does not stop all
	instances
Message-ID: <20101111202838.GA18696@linspire.com>
References: <365982.91127.qm@web112117.mail.gq1.yahoo.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
In-Reply-To: <365982.91127.qm@web112117.mail.gq1.yahoo.com>
X-Organization: It's something of 'Cos
User-Agent: Mutt/1.5.18 (2008-05-17)

Without much details I only can offer to check if your master hosts (NN, JT)
have passwordless access to all slaves

Cos

On Thu, Nov 11, 2010 at 02:41AM, Grandl Robert wrote:
> Hi all,
>=20
> Probably is a stupid question but I don't figure out what happens.
>=20
> For example I start on master, start-dfs.sh, start-mapred.sh
>=20
> However, when I want to stop: stop-mapred, stop-dfs, only one slave insta=
nce is killed.=20
>=20
> For example in on the master I am doing something like that:
>=20
> rgrandl@ikq01:~/hadoop/hadoop-0.21.0$ bin/stop-mapred.sh=20
> stopping jobtracker
> ikq02.rrr: no tasktracker to stop
> ikq03.rrr: stopping tasktracker
>=20
> But I have a tasktracker on ikq02 as well which is not stopped anyway.
> rgrandl@ikq02:~$ jps
> 2462 Jps
> 2211 TaskTracker
> 2109 DataNode
>=20
> Same thing for stop-dfs.sh
>=20
>=20
> Always the tasktracker, datanode instances are killed only on one slave n=
ode, even if I have maybe 20 slaves or more.=20
>=20
> Thanks for any clue,
> Robert
>=20
>=20
>=20
>=20
>=20
>      =20

From general-return-2373-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 12 02:17:29 2010
Return-Path: <general-return-2373-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70814 invoked from network); 12 Nov 2010 02:17:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Nov 2010 02:17:28 -0000
Received: (qmail 37315 invoked by uid 500); 12 Nov 2010 02:17:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37231 invoked by uid 500); 12 Nov 2010 02:17:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37223 invoked by uid 99); 12 Nov 2010 02:17:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Nov 2010 02:17:57 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of wbsrvc@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Nov 2010 02:17:50 +0000
Received: by qwf7 with SMTP id 7so617034qwf.35
        for <general@hadoop.apache.org>; Thu, 11 Nov 2010 18:17:30 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=pV+prLxQck9H59aSl7hJNNgv1vbKPgCXhrCRtE++ACU=;
        b=b8ZzzsP6Xp90z0I7aJtPskzqO60hORSsG00sxXFXirxI0ZJ7BonZ8NPh7kwen6Dj8k
         yvn3/PT8rHXKS+YtsL3eXUDNExDSykHOF4Rw4W1Dv6yKLbOGzrLnO3LCOdnGAOCyOZ8e
         +5t3dpBe6RfZKHB9n4QCftBAKDHZJw7VCQSAg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=WUfH51aRgpLZGJBr54MboWAByzvrMyQBr0jCJ9m/FnmGCxsJoPGcxpzr6cnk05iBME
         vAbUAUoug8IPuS15Ol6USEzeyr2iG8c2diryLGyiMvmzcY2BS8B4VzCkvQADW99sdgBO
         iNAQ5gRRSFN3ys0hoSG0G9iToSd7G4U9LHUmU=
MIME-Version: 1.0
Received: by 10.229.90.196 with SMTP id j4mr1341740qcm.270.1289528250079; Thu,
 11 Nov 2010 18:17:30 -0800 (PST)
Received: by 10.229.33.83 with HTTP; Thu, 11 Nov 2010 18:17:30 -0800 (PST)
Date: Thu, 11 Nov 2010 19:17:30 -0700
Message-ID: <AANLkTinQVUC0U9CEONpW13+MLXHOVQWAqfvyhG=1btjw@mail.gmail.com>
Subject: running hadoop jobs from within a program
From: web service <wbsrvc@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636310649461a5d0494d1b1ef

--001636310649461a5d0494d1b1ef
Content-Type: text/plain; charset=ISO-8859-1

Hi,
  Currently I run my sample hadoop job from a bash script using the
following command ...

[code]
tmp="$HADOOP_BIN jar $JAR_LOC  $MAIN_CLASS /user/joe/input/input-$i/
/user/vadmin/output/output-$i/
$tmp
[/code]

However, I would want to write a timer that would do some cleanup after the
jobs are  complete and restart the jobs after x hours. What I am looking for
is
the ability to invoke job from within a program and not the jar command
thing.

-Mac

--001636310649461a5d0494d1b1ef--

From general-return-2374-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 12 02:18:48 2010
Return-Path: <general-return-2374-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 71270 invoked from network); 12 Nov 2010 02:18:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Nov 2010 02:18:48 -0000
Received: (qmail 39488 invoked by uid 500); 12 Nov 2010 02:19:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39410 invoked by uid 500); 12 Nov 2010 02:19:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39359 invoked by uid 99); 12 Nov 2010 02:19:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Nov 2010 02:19:12 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of wbsrvc@gmail.com designates 209.85.216.169 as permitted sender)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Nov 2010 02:19:06 +0000
Received: by qyk8 with SMTP id 8so103364qyk.14
        for <general@hadoop.apache.org>; Thu, 11 Nov 2010 18:18:45 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=QIJA6xzCLDDJAw31peRMOcyqezI6M3vC0ZdjhXktOXg=;
        b=NN9hLKli716fh2LNqPfR/8ri39JZiBdbKnV5Ay0ZVzkeQVFk8rNZAEQs+MIFZzVEDw
         6G6NndeM7GivedO6CY4KF7GMn4SVOOX7Bi/+zlEsnog74DTUqG9/A2HzWcbEFfdqm3ad
         qETX1pPYHfgSKlou4fC+P4REVf395wregWMw8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=LJ/aIYnwD1Y4+ZDq25GX8QCB4d1Wu0mQ41e96kzfsbelLCq0VN9QzH99GvJraeq68x
         KZhPXie31j6GDMO59qzRdR112InDqGCXxc1dIGhKE9g/7Ip28XzR15lGL9U0rRn55UJZ
         4Zj9GYumhCa/hIRZsjRSl+Acloqx+V4kmdbUw=
MIME-Version: 1.0
Received: by 10.229.85.6 with SMTP id m6mr1365479qcl.80.1289528325593; Thu, 11
 Nov 2010 18:18:45 -0800 (PST)
Received: by 10.229.33.83 with HTTP; Thu, 11 Nov 2010 18:18:45 -0800 (PST)
Date: Thu, 11 Nov 2010 19:18:45 -0700
Message-ID: <AANLkTimKwDRCPi95y1URNQ2nMm1_1CUk=7HsEAdrbkjU@mail.gmail.com>
Subject: browsing mailing list
From: web service <wbsrvc@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163642752dc65af90494d1b5c1

--00163642752dc65af90494d1b5c1
Content-Type: text/plain; charset=ISO-8859-1

This might be silly question, but how is the mailing list supposed to be
searched/browsed ? Is there any client to download all the messages ? What
do you guys do ?

-Mac

--00163642752dc65af90494d1b5c1--

From general-return-2375-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 12 02:26:33 2010
Return-Path: <general-return-2375-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72809 invoked from network); 12 Nov 2010 02:26:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Nov 2010 02:26:33 -0000
Received: (qmail 50218 invoked by uid 500); 12 Nov 2010 02:27:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50162 invoked by uid 500); 12 Nov 2010 02:27:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50154 invoked by uid 99); 12 Nov 2010 02:27:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Nov 2010 02:27:03 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of shrijeet@rocketfuelinc.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Nov 2010 02:26:58 +0000
Received: by fxm10 with SMTP id 10so1580466fxm.35
        for <general@hadoop.apache.org>; Thu, 11 Nov 2010 18:26:36 -0800 (PST)
MIME-Version: 1.0
Received: by 10.223.114.202 with SMTP id f10mr893117faq.67.1289528795952; Thu,
 11 Nov 2010 18:26:35 -0800 (PST)
Received: by 10.223.98.204 with HTTP; Thu, 11 Nov 2010 18:26:35 -0800 (PST)
In-Reply-To: <AANLkTimKwDRCPi95y1URNQ2nMm1_1CUk=7HsEAdrbkjU@mail.gmail.com>
References: <AANLkTimKwDRCPi95y1URNQ2nMm1_1CUk=7HsEAdrbkjU@mail.gmail.com>
Date: Thu, 11 Nov 2010 18:26:35 -0800
Message-ID: <AANLkTikdAgZ9zCpk+JYJK=zxXGxNn2c60-P_8iw28nPJ@mail.gmail.com>
Subject: Re: browsing mailing list
From: Shrijeet Paliwal <shrijeet@rocketfuel.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

...because you asked, what do you guys do...

I do http://www.search-hadoop.com/

-Shrijeet

On Thu, Nov 11, 2010 at 6:18 PM, web service <wbsrvc@gmail.com> wrote:
> This might be silly question, but how is the mailing list supposed to be
> searched/browsed ? Is there any client to download all the messages ? What
> do you guys do ?
>
> -Mac
>

From general-return-2376-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 12 02:28:44 2010
Return-Path: <general-return-2376-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73285 invoked from network); 12 Nov 2010 02:28:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Nov 2010 02:28:44 -0000
Received: (qmail 53174 invoked by uid 500); 12 Nov 2010 02:29:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53086 invoked by uid 500); 12 Nov 2010 02:29:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53075 invoked by uid 99); 12 Nov 2010 02:29:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Nov 2010 02:29:14 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of wbsrvc@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Nov 2010 02:29:09 +0000
Received: by qwf7 with SMTP id 7so625920qwf.35
        for <general@hadoop.apache.org>; Thu, 11 Nov 2010 18:28:48 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=hlNoEvPFQFSFQQMIHoAtH6AYPMabhw+VLokBMg+3nGs=;
        b=m4lOEPeboeEWH7mAyfDwKDv/Otp9pDvM7be0VaznkQzT0CwEODvuhM1HTGLrDp9bUb
         tJHNDKy/U/RnCZ9LrtuWuCSfrhDvAirXl3kIHC0RaTvpdR8ImfGstzxSNx8d+Jo9fCUO
         OlYSDWJxVTvy1Ba8QPQhgennjn3bm27/LvL/o=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=l/6RAnCHsOXm7GPCnCVdhZoU2gFv5f/2J8fe0J2V4EzzfiG6S/8NWFxuzFsEmRUXOl
         q1rw2JvorFqXZIy6/kSUWzG27Kym8SGnjnhmn9sziof6RPCpMzJ/GbXtIa/HEVT9b15Y
         Jgt60d8+WcsuHDU+a2H9LkP2W09phTky2YbuM=
MIME-Version: 1.0
Received: by 10.229.85.6 with SMTP id m6mr1373813qcl.80.1289528925969; Thu, 11
 Nov 2010 18:28:45 -0800 (PST)
Received: by 10.229.33.83 with HTTP; Thu, 11 Nov 2010 18:28:45 -0800 (PST)
In-Reply-To: <AANLkTikdAgZ9zCpk+JYJK=zxXGxNn2c60-P_8iw28nPJ@mail.gmail.com>
References: <AANLkTimKwDRCPi95y1URNQ2nMm1_1CUk=7HsEAdrbkjU@mail.gmail.com>
	<AANLkTikdAgZ9zCpk+JYJK=zxXGxNn2c60-P_8iw28nPJ@mail.gmail.com>
Date: Thu, 11 Nov 2010 19:28:45 -0700
Message-ID: <AANLkTim39MjLCX=7swNJ+5i8FZMbiPgJ9TQn+c3mY_0L@mail.gmail.com>
Subject: Re: browsing mailing list
From: web service <wbsrvc@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163642752d8fb7c50494d1d96e

--00163642752d8fb7c50494d1d96e
Content-Type: text/plain; charset=ISO-8859-1

thanks.

On Thu, Nov 11, 2010 at 7:26 PM, Shrijeet Paliwal
<shrijeet@rocketfuel.com>wrote:

> ...because you asked, what do you guys do...
>
> I do http://www.search-hadoop.com/
>
> -Shrijeet
>
> On Thu, Nov 11, 2010 at 6:18 PM, web service <wbsrvc@gmail.com> wrote:
> > This might be silly question, but how is the mailing list supposed to be
> > searched/browsed ? Is there any client to download all the messages ?
> What
> > do you guys do ?
> >
> > -Mac
> >
>

--00163642752d8fb7c50494d1d96e--

From general-return-2377-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 12 03:16:45 2010
Return-Path: <general-return-2377-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94220 invoked from network); 12 Nov 2010 03:16:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Nov 2010 03:16:45 -0000
Received: (qmail 94954 invoked by uid 500); 12 Nov 2010 03:17:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94844 invoked by uid 500); 12 Nov 2010 03:17:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94836 invoked by uid 99); 12 Nov 2010 03:17:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Nov 2010 03:17:13 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of wbsrvc@gmail.com designates 209.85.216.169 as permitted sender)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Nov 2010 03:17:04 +0000
Received: by qyk8 with SMTP id 8so148783qyk.14
        for <general@hadoop.apache.org>; Thu, 11 Nov 2010 19:16:43 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=mO/7S9JreUWjl+rGsNgthmTDoSJa7VPVIXaHuTIUp9I=;
        b=DI02iXXTKZd92r8YiE1oZXw2/AtKnmP1kVYmDCfrYFkXqYYJnHgBMf7ozkOku0kSD+
         cuFYXS0SJbTXS/gjOUzk+7AfvlkYYoLBMsX25g4QRFpDwG1NuL8tzDHt5YMJfPeAqqba
         E0p/srNxWra7UgznI28hqHfCrqHRlFcZsrhlg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=wlz9Pt8o7oIYLyM1yMi1mzmvu2r5koHwXspLkPkKT0zGn9EcLOxFu5vzx45C/OQA6b
         IMYTvDPvHvUOGvF1KTxBhaH4YCqzywEgUa7f8dJb5X0rGYbrjOtL9EYtidXnAy6a00nT
         LCtgImP+CZO5wcGwxjhVp3vEwIsFAzfvNxBa4=
MIME-Version: 1.0
Received: by 10.229.90.196 with SMTP id j4mr1393166qcm.270.1289531803616; Thu,
 11 Nov 2010 19:16:43 -0800 (PST)
Received: by 10.229.33.83 with HTTP; Thu, 11 Nov 2010 19:16:43 -0800 (PST)
In-Reply-To: <AANLkTinQVUC0U9CEONpW13+MLXHOVQWAqfvyhG=1btjw@mail.gmail.com>
References: <AANLkTinQVUC0U9CEONpW13+MLXHOVQWAqfvyhG=1btjw@mail.gmail.com>
Date: Thu, 11 Nov 2010 20:16:43 -0700
Message-ID: <AANLkTi=-h=6ii-iArmoRwfbZqN5UvSC6wRdk3R+NvdEw@mail.gmail.com>
Subject: Re: running hadoop jobs from within a program
From: web service <wbsrvc@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163631064914c5d80494d2851b
X-Virus-Checked: Checked by ClamAV on apache.org

--00163631064914c5d80494d2851b
Content-Type: text/plain; charset=ISO-8859-1

would submitting, say for example 3 jobs from a jobclient be different than
invoking the below command 3 times ?

On Thu, Nov 11, 2010 at 7:17 PM, web service <wbsrvc@gmail.com> wrote:

> Hi,
>   Currently I run my sample hadoop job from a bash script using the
> following command ...
>
> [code]
> tmp="$HADOOP_BIN jar $JAR_LOC  $MAIN_CLASS /user/joe/input/input-$i/
> /user/vadmin/output/output-$i/
> $tmp
> [/code]
>
> However, I would want to write a timer that would do some cleanup after the
> jobs are  complete and restart the jobs after x hours. What I am looking for
> is
> the ability to invoke job from within a program and not the jar command
> thing.
>
> -Mac
>

--00163631064914c5d80494d2851b--

From general-return-2378-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 12 08:55:16 2010
Return-Path: <general-return-2378-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30337 invoked from network); 12 Nov 2010 08:55:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Nov 2010 08:55:16 -0000
Received: (qmail 19796 invoked by uid 500); 12 Nov 2010 08:55:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19649 invoked by uid 500); 12 Nov 2010 08:55:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19630 invoked by uid 99); 12 Nov 2010 08:55:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Nov 2010 08:55:45 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dsikar@gmail.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Nov 2010 08:55:38 +0000
Received: by gyf2 with SMTP id 2so1961178gyf.35
        for <general@hadoop.apache.org>; Fri, 12 Nov 2010 00:55:18 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=v5/QimbCOTOPuMVj45f91RzxtD/d15o+KNkvwzsvi+k=;
        b=i4oKCbt2LNQ1zgcD7BuSCzKR7Yn6JvAX0MW7z/xOq7O+K1q/wJfXGBLRfgjgU0bLSL
         4vURQSsc4SEZuJadeBKpR/MKMiB4m6kpDtRM3xamrZYkNjwKMxP8O/natmCV2pVN1EcS
         PaVFdeWabt/0EcnFSlLNI0ex0fLdBCKhmTKqA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=huFXRxusytLdnfyRhnA1ZlieL6aeyTbPCCWpIdbgOyAAY8W7oXGIFXdcqAjLM4CrgH
         ekKJKHBpytcILIzqjPRxFrifWy99/tyTHH0nO/iTtqLbM8DgImRl3U1IxHyphSdTY3Db
         t+rBKAyS1mKa7Xi1tnNj5B8OmXshQ+ojtfVY8=
MIME-Version: 1.0
Received: by 10.91.11.24 with SMTP id o24mr2830058agi.55.1289552117664; Fri,
 12 Nov 2010 00:55:17 -0800 (PST)
Received: by 10.90.78.5 with HTTP; Fri, 12 Nov 2010 00:55:17 -0800 (PST)
In-Reply-To: <AANLkTinQVUC0U9CEONpW13+MLXHOVQWAqfvyhG=1btjw@mail.gmail.com>
References: <AANLkTinQVUC0U9CEONpW13+MLXHOVQWAqfvyhG=1btjw@mail.gmail.com>
Date: Fri, 12 Nov 2010 08:55:17 +0000
Message-ID: <AANLkTinr0mO0N4c8+oqk1T5=9NbmdVxVpbbGJnxC4jKV@mail.gmail.com>
Subject: Re: running hadoop jobs from within a program
From: daniel sikar <dsikar@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I suggest you write a loop in your bash script, grepping for finished,
then take it from there.
Also, you can submit the same job as many times as you like.

On 12 November 2010 02:17, web service <wbsrvc@gmail.com> wrote:
> Hi,
> =A0Currently I run my sample hadoop job from a bash script using the
> following command ...
>
> [code]
> tmp=3D"$HADOOP_BIN jar $JAR_LOC =A0$MAIN_CLASS /user/joe/input/input-$i/
> /user/vadmin/output/output-$i/
> $tmp
> [/code]
>
> However, I would want to write a timer that would do some cleanup after t=
he
> jobs are =A0complete and restart the jobs after x hours. What I am lookin=
g for
> is
> the ability to invoke job from within a program and not the jar command
> thing.
>
> -Mac
>

From general-return-2379-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 12 09:31:12 2010
Return-Path: <general-return-2379-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 48667 invoked from network); 12 Nov 2010 09:31:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Nov 2010 09:31:12 -0000
Received: (qmail 51391 invoked by uid 500); 12 Nov 2010 09:31:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51266 invoked by uid 500); 12 Nov 2010 09:31:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51258 invoked by uid 99); 12 Nov 2010 09:31:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Nov 2010 09:31:41 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.215.176] (HELO mail-ey0-f176.google.com) (209.85.215.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Nov 2010 09:31:34 +0000
Received: by eyz10 with SMTP id 10so1692493eyz.35
        for <general@hadoop.apache.org>; Fri, 12 Nov 2010 01:31:13 -0800 (PST)
Received: by 10.213.8.67 with SMTP id g3mr1614167ebg.38.1289554272928; Fri, 12
 Nov 2010 01:31:12 -0800 (PST)
MIME-Version: 1.0
Received: by 10.213.14.78 with HTTP; Fri, 12 Nov 2010 01:30:42 -0800 (PST)
In-Reply-To: <AANLkTinr0mO0N4c8+oqk1T5=9NbmdVxVpbbGJnxC4jKV@mail.gmail.com>
References: <AANLkTinQVUC0U9CEONpW13+MLXHOVQWAqfvyhG=1btjw@mail.gmail.com> <AANLkTinr0mO0N4c8+oqk1T5=9NbmdVxVpbbGJnxC4jKV@mail.gmail.com>
From: Alejandro Abdelnur <tucu@cloudera.com>
Date: Fri, 12 Nov 2010 01:30:42 -0800
Message-ID: <AANLkTinYmQX7LKy3XP7wx0DudwTiYkBTpWJd7DUw8QwS@mail.gmail.com>
Subject: Re: running hadoop jobs from within a program
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Mac,

You should a look at Oozie, it will allow you to do what you describe.

You can either build Oozie from https://github.com/yahoo/oozie or
download CDH3b3 distribution from http://www.cloudera.com/downloads/
(Oozie is preconfigured to work with CHD3b3 Hadoop).

Hope this helps.

Alejandro

On Fri, Nov 12, 2010 at 12:55 AM, daniel sikar <dsikar@gmail.com> wrote:
> I suggest you write a loop in your bash script, grepping for finished,
> then take it from there.
> Also, you can submit the same job as many times as you like.
>
> On 12 November 2010 02:17, web service <wbsrvc@gmail.com> wrote:
>> Hi,
>> =A0Currently I run my sample hadoop job from a bash script using the
>> following command ...
>>
>> [code]
>> tmp=3D"$HADOOP_BIN jar $JAR_LOC =A0$MAIN_CLASS /user/joe/input/input-$i/
>> /user/vadmin/output/output-$i/
>> $tmp
>> [/code]
>>
>> However, I would want to write a timer that would do some cleanup after =
the
>> jobs are =A0complete and restart the jobs after x hours. What I am looki=
ng for
>> is
>> the ability to invoke job from within a program and not the jar command
>> thing.
>>
>> -Mac
>>
>

From general-return-2380-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 12 16:55:10 2010
Return-Path: <general-return-2380-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69788 invoked from network); 12 Nov 2010 16:55:10 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Nov 2010 16:55:10 -0000
Received: (qmail 80144 invoked by uid 500); 12 Nov 2010 16:55:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79954 invoked by uid 500); 12 Nov 2010 16:55:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 79946 invoked by uid 99); 12 Nov 2010 16:55:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Nov 2010 16:55:39 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of wbsrvc@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Nov 2010 16:55:33 +0000
Received: by vws19 with SMTP id 19so169800vws.35
        for <general@hadoop.apache.org>; Fri, 12 Nov 2010 08:55:12 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=Mxd+HM2VnBSlvOQOrmOm1PK0nML5kKgFakroI3TWJNM=;
        b=rRVR8h6ePlV1OyjsH9T1/3kb/S/kshL4UlrQzJuzsk2EzDsCVETUF/pQiOA1llau1E
         ly/B00ksSGwPV65NUyMrKsCLAZfoDP+chprcEUBQpnxNdlBwApbf96SfxxsdGv4ostO1
         nv7l5a02ols9h3waHsUbGK+pyb/Rt3/t1Tc4k=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=MfiqczTv8NFdQ6e8U370xwYA87l9nQlFu+GYwVKEa5J+TpOYvNO7Nq/wfWSVcwiQSz
         5V4iQIXGs2h9fOg0qJQEHcepUE0Oih4HKROA5wcIF9K7uty7rjTtP2xeuS+y/pqRCe4N
         r8/7ow1gB9FS/+RJNgfPqpV7Ul9RqCEXTWUUU=
MIME-Version: 1.0
Received: by 10.229.242.66 with SMTP id lh2mr2043464qcb.118.1289580912164;
 Fri, 12 Nov 2010 08:55:12 -0800 (PST)
Received: by 10.229.33.83 with HTTP; Fri, 12 Nov 2010 08:55:11 -0800 (PST)
In-Reply-To: <AANLkTinr0mO0N4c8+oqk1T5=9NbmdVxVpbbGJnxC4jKV@mail.gmail.com>
References: <AANLkTinQVUC0U9CEONpW13+MLXHOVQWAqfvyhG=1btjw@mail.gmail.com>
	<AANLkTinr0mO0N4c8+oqk1T5=9NbmdVxVpbbGJnxC4jKV@mail.gmail.com>
Date: Fri, 12 Nov 2010 08:55:11 -0800
Message-ID: <AANLkTim_ZJLjmVUBLDAtzPYqcSxp=4jSDi7tG0COT-hx@mail.gmail.com>
Subject: Re: running hadoop jobs from within a program
From: web service <wbsrvc@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016362843d62dc0850494ddf41f

--0016362843d62dc0850494ddf41f
Content-Type: text/plain; charset=ISO-8859-1

Thanks, but submitting three different jobs say using

JobClient.submitjob(jobconf1);
JobClient.submitjob(jobconf2);
JobClient.submitjob(jobconf3)

different from running -
tmp="$HADOOP_BIN jar $JAR_LOC  $MAIN_CLASS /user/joe/input/input-1/
/user/vadmin/output/output-1/
tmp="$HADOOP_BIN jar $JAR_LOC  $MAIN_CLASS /user/joe/input/input-2/
/user/vadmin/output/output-2/
tmp="$HADOOP_BIN jar $JAR_LOC  $MAIN_CLASS /user/joe/input/input-3/
/user/vadmin/output/output-3/

I guess every job can have specific jvm options. and I hope that every
submitted job runs in a separate jvm, No ?

On Fri, Nov 12, 2010 at 12:55 AM, daniel sikar <dsikar@gmail.com> wrote:

> I suggest you write a loop in your bash script, grepping for finished,
> then take it from there.
> Also, you can submit the same job as many times as you like.
>
> On 12 November 2010 02:17, web service <wbsrvc@gmail.com> wrote:
> > Hi,
> >  Currently I run my sample hadoop job from a bash script using the
> > following command ...
> >
> > [code]
> > tmp="$HADOOP_BIN jar $JAR_LOC  $MAIN_CLASS /user/joe/input/input-$i/
> > /user/vadmin/output/output-$i/
> > $tmp
> > [/code]
> >
> > However, I would want to write a timer that would do some cleanup after
> the
> > jobs are  complete and restart the jobs after x hours. What I am looking
> for
> > is
> > the ability to invoke job from within a program and not the jar command
> > thing.
> >
> > -Mac
> >
>

--0016362843d62dc0850494ddf41f--

From general-return-2381-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 12 17:52:47 2010
Return-Path: <general-return-2381-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 10133 invoked from network); 12 Nov 2010 17:52:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Nov 2010 17:52:46 -0000
Received: (qmail 70638 invoked by uid 500); 12 Nov 2010 17:53:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70571 invoked by uid 500); 12 Nov 2010 17:53:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70525 invoked by uid 99); 12 Nov 2010 17:53:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Nov 2010 17:53:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 12 Nov 2010 17:53:16 +0000
Received: (qmail 10018 invoked by uid 99); 12 Nov 2010 17:52:24 -0000
Received: from localhost.apache.org (HELO mail-ww0-f42.google.com) (127.0.0.1)
  (smtp-auth username phunt, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Nov 2010 17:52:24 +0000
Received: by wwd20 with SMTP id 20so302130wwd.5
        for <multiple recipients>; Fri, 12 Nov 2010 09:52:53 -0800 (PST)
MIME-Version: 1.0
Received: by 10.227.147.196 with SMTP id m4mr2744878wbv.57.1289584373191; Fri,
 12 Nov 2010 09:52:53 -0800 (PST)
Received: by 10.227.152.139 with HTTP; Fri, 12 Nov 2010 09:52:53 -0800 (PST)
Date: Fri, 12 Nov 2010 09:52:53 -0800
Message-ID: <AANLkTinpnu37bykcV2gaYh8_X4QjHJNe=Ew0BGfcejL_@mail.gmail.com>
Subject: [ANNOUNCE] Apache ZooKeeper 3.3.2
From: Patrick Hunt <phunt@apache.org>
To: announce@apache.org, 
	"zookeeper-user@hadoop.apache.org" <zookeeper-user@hadoop.apache.org>, 
	"zookeeper-dev@hadoop.apache.org" <zookeeper-dev@hadoop.apache.org>, general@hadoop.apache.org, 
	private@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

The Apache ZooKeeper team is proud to announce Apache ZooKeeper version 3.3.2

ZooKeeper is a high-performance coordination service for distributed
applications. It exposes common services - such as naming,
configuration management, synchronization, and group services - in a
simple interface so you don't have to write them from scratch. You can
use it off-the-shelf to implement consensus, group management, leader
election, and presence protocols. And you can build on it for your
own, specific needs.

For ZooKeeper release details and downloads, visit:
http://hadoop.apache.org/zookeeper/releases.html

ZooKeeper 3.3.2 Release Notes are at:
http://hadoop.apache.org/zookeeper/docs/r3.3.2/releasenotes.html

Regards,

The ZooKeeper Team

From general-return-2382-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 12 19:06:17 2010
Return-Path: <general-return-2382-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63105 invoked from network); 12 Nov 2010 19:06:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Nov 2010 19:06:16 -0000
Received: (qmail 99275 invoked by uid 500); 12 Nov 2010 19:06:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99236 invoked by uid 500); 12 Nov 2010 19:06:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 99223 invoked by uid 99); 12 Nov 2010 19:06:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Nov 2010 19:06:42 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of Peter.Minearo@reardencommerce.com designates 64.41.141.19 as permitted sender)
Received: from [64.41.141.19] (HELO barracuda.mygazoo.com) (64.41.141.19)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Nov 2010 19:06:35 +0000
X-ASG-Debug-ID: 1289588769-3095aa900001-IjwG86
Received: from sccorpmail01.mygazoo.com (sccorpmail01.mygazoo.com [10.38.20.8]) by barracuda.mygazoo.com with ESMTP id uaF0cZhsZbQ3wGWq for <general@hadoop.apache.org>; Fri, 12 Nov 2010 11:06:09 -0800 (PST)
X-Barracuda-Envelope-From: Peter.Minearo@Reardencommerce.com
Received: from sccorpmail02.mygazoo.com ([10.38.20.9]) by sccorpmail01.mygazoo.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Fri, 12 Nov 2010 11:06:09 -0800
Received: from sccorpmail03.mygazoo.com (10.38.20.15) by
 sccorpmail02.mygazoo.com (10.38.20.9) with Microsoft SMTP Server (TLS) id
 8.2.254.0; Fri, 12 Nov 2010 11:06:08 -0800
X-Barracuda-BBL-IP: 10.38.20.9
X-Barracuda-RBL-IP: 10.38.20.9
Received: from fccorpmail03.mygazoo.com ([fe80::6116:1dc:5867:46dc]) by
 sccorpmail03.mygazoo.com ([fe80::5cfd:f627:680f:3e2a%18]) with mapi id
 14.01.0255.000; Fri, 12 Nov 2010 11:06:08 -0800
From: Peter Minearo <Peter.Minearo@Reardencommerce.com>
X-Barracuda-Apparent-Source-IP: fe80::6116:1dc:5867:46dc
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Cloudera CDH3 install on Ubuntu 10.04 (lucid)
Thread-Topic: Cloudera CDH3 install on Ubuntu 10.04 (lucid)
X-ASG-Orig-Subj: Cloudera CDH3 install on Ubuntu 10.04 (lucid)
Thread-Index: AcuCnA2hU9lcsVVcQBSDXhBCkmHiqg==
Date: Fri, 12 Nov 2010 19:06:08 +0000
Message-ID: <B75E035660BF70488B180EFA991B7368083A31@fccorpmail03.mygazoo.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [63.81.0.20]
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginalArrivalTime: 12 Nov 2010 19:06:09.0182 (UTC) FILETIME=[AA2DABE0:01CB829C]
X-Barracuda-Connect: sccorpmail01.mygazoo.com[10.38.20.8]
X-Barracuda-Start-Time: 1289588769
X-Barracuda-URL: http://10.38.16.13:8000/cgi-mod/mark.cgi
X-Virus-Scanned: by bsmtpd at mygazoo.com
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=9.0 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.2.46444
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------

Not sure who or where to send this to, but this is for the Cloudera distrib=
ution of Hadoop:

I just tried to install CDH3 on my Ubuntu machine

> more /etc/lsb-release
DISTRIB_ID=3DUbuntu
DISTRIB_RELEASE=3D10.04
DISTRIB_CODENAME=3Dlucid
DISTRIB_DESCRIPTION=3D"Ubuntu 10.04.1 LTS"

I got the following error:
> sudo apt-get install hadoop-0.20
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
  hadoop-0.20: Depends: sun-java6-jre but it is not installable
               Depends: sun-java6-bin but it is not installable
E: Broken packages

I have Jave 1.6 installed:
> java -version
java version "1.6.0_22"
Java(TM) SE Runtime Environment (build 1.6.0_22-b04)
Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03, mixed mode)

What I found was, in order to get the install to work properly; you have to=
 do an extra step.   In Step #4 on https://docs.cloudera.com/display/DOC/Ha=
doop+Installation+%28CDH3%29  there is:

> apt-cache search hadoop
> sudo apt-get install hadoop-0.20

Between these 2 commands there are 2 other commands that are needed:
> echo "deb http://archive.canonical.com/ lucid partner" | sudo tee /etc/ap=
t/sources.list.d/partner.list
> apt-get update

So, until Cloudera fixes the install (or documentation) in order to install=
 the CDH3 on Ubuntu 10.04 (lucid) the steps should look like:

> apt-cache search hadoop
> echo "deb http://archive.canonical.com/ lucid partner" | sudo tee /etc/ap=
t/sources.list.d/partner.list
> apt-get update
> sudo apt-get install hadoop-0.20

Does anyone know, if this is being looked at?

Thanks,

Peter



From general-return-2383-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 12 19:48:36 2010
Return-Path: <general-return-2383-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83556 invoked from network); 12 Nov 2010 19:48:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Nov 2010 19:48:36 -0000
Received: (qmail 70572 invoked by uid 500); 12 Nov 2010 19:49:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70524 invoked by uid 500); 12 Nov 2010 19:49:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70516 invoked by uid 99); 12 Nov 2010 19:49:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Nov 2010 19:49:06 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Nov 2010 19:48:57 +0000
Received: from SP2-EX07CAS02.ds.corp.yahoo.com (sp2-ex07cas02.corp.sp2.yahoo.com [98.137.59.38])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id oACJmFZu058973
	for <general@hadoop.apache.org>; Fri, 12 Nov 2010 11:48:15 -0800 (PST)
Received: from SP2-EX07VS07.ds.corp.yahoo.com ([98.137.59.26]) by
 SP2-EX07CAS02.ds.corp.yahoo.com ([98.137.59.38]) with mapi; Fri, 12 Nov 2010
 11:48:15 -0800
From: Alfred Thompson <atx@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 12 Nov 2010 11:48:14 -0800
Subject: Re: [ANNOUNCE] Apache ZooKeeper 3.3.2
Thread-Topic: [ANNOUNCE] Apache ZooKeeper 3.3.2
Thread-Index: AcuCkpMB4jnzGb7BRCeRD1Atgp8ONAAD/d8X
Message-ID: <C902D9FD.1280D%atx@yahoo-inc.com>
In-Reply-To: <AANLkTinpnu37bykcV2gaYh8_X4QjHJNe=Ew0BGfcejL_@mail.gmail.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C902D9FD1280Datxyahooinccom_"
MIME-Version: 1.0

--_000_C902D9FD1280Datxyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Well done.


On 11/12/10 10:52 AM, "Patrick Hunt" <phunt@apache.org> wrote:

The Apache ZooKeeper team is proud to announce Apache ZooKeeper version 3.3=
.2

ZooKeeper is a high-performance coordination service for distributed
applications. It exposes common services - such as naming,
configuration management, synchronization, and group services - in a
simple interface so you don't have to write them from scratch. You can
use it off-the-shelf to implement consensus, group management, leader
election, and presence protocols. And you can build on it for your
own, specific needs.

For ZooKeeper release details and downloads, visit:
http://hadoop.apache.org/zookeeper/releases.html

ZooKeeper 3.3.2 Release Notes are at:
http://hadoop.apache.org/zookeeper/docs/r3.3.2/releasenotes.html

Regards,

The ZooKeeper Team


--_000_C902D9FD1280Datxyahooinccom_--

From general-return-2384-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 12 22:04:29 2010
Return-Path: <general-return-2384-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50186 invoked from network); 12 Nov 2010 22:04:29 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Nov 2010 22:04:28 -0000
Received: (qmail 38680 invoked by uid 500); 12 Nov 2010 22:04:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38606 invoked by uid 500); 12 Nov 2010 22:04:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 38598 invoked by uid 99); 12 Nov 2010 22:04:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Nov 2010 22:04:56 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Nov 2010 22:04:50 +0000
Received: by qyk1 with SMTP id 1so414813qyk.14
        for <general@hadoop.apache.org>; Fri, 12 Nov 2010 14:04:29 -0800 (PST)
MIME-Version: 1.0
Received: by 10.229.233.74 with SMTP id jx10mr2325012qcb.96.1289599469102;
 Fri, 12 Nov 2010 14:04:29 -0800 (PST)
Received: by 10.229.225.137 with HTTP; Fri, 12 Nov 2010 14:04:29 -0800 (PST)
In-Reply-To: <B75E035660BF70488B180EFA991B7368083A31@fccorpmail03.mygazoo.com>
References: <B75E035660BF70488B180EFA991B7368083A31@fccorpmail03.mygazoo.com>
Date: Fri, 12 Nov 2010 14:04:29 -0800
Message-ID: <AANLkTimab+J+_MdQ9QAAhHjN34hC=MSVvHbjRdytKSGx@mail.gmail.com>
Subject: Re: Cloudera CDH3 install on Ubuntu 10.04 (lucid)
From: Eli Collins <eli@cloudera.com>
To: Peter.Minearo@reardencommerce.com, CDH Users <cdh-user@cloudera.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hey Peter,

Moving this thread to the CDH user mailing list (BCC general@).

This step is covered in the JDK installation page which is linked to
in the red "Important" section at the top of the installation page you
were looking at.

https://docs.cloudera.com/display/DOC/Java+Development+Kit+Installation

Thanks,
Eli


On Fri, Nov 12, 2010 at 11:06 AM, Peter Minearo
<Peter.Minearo@reardencommerce.com> wrote:
> Not sure who or where to send this to, but this is for the Cloudera distr=
ibution of Hadoop:
>
> I just tried to install CDH3 on my Ubuntu machine
>
>> more /etc/lsb-release
> DISTRIB_ID=3DUbuntu
> DISTRIB_RELEASE=3D10.04
> DISTRIB_CODENAME=3Dlucid
> DISTRIB_DESCRIPTION=3D"Ubuntu 10.04.1 LTS"
>
> I got the following error:
>> sudo apt-get install hadoop-0.20
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> Some packages could not be installed. This may mean that you have
> requested an impossible situation or if you are using the unstable
> distribution that some required packages have not yet been created
> or been moved out of Incoming.
> The following information may help to resolve the situation:
>
> The following packages have unmet dependencies:
> =A0hadoop-0.20: Depends: sun-java6-jre but it is not installable
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 Depends: sun-java6-bin but it is not installa=
ble
> E: Broken packages
>
> I have Jave 1.6 installed:
>> java -version
> java version "1.6.0_22"
> Java(TM) SE Runtime Environment (build 1.6.0_22-b04)
> Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03, mixed mode)
>
> What I found was, in order to get the install to work properly; you have =
to do an extra step. =A0 In Step #4 on https://docs.cloudera.com/display/DO=
C/Hadoop+Installation+%28CDH3%29 =A0there is:
>
>> apt-cache search hadoop
>> sudo apt-get install hadoop-0.20
>
> Between these 2 commands there are 2 other commands that are needed:
>> echo "deb http://archive.canonical.com/ lucid partner" | sudo tee /etc/a=
pt/sources.list.d/partner.list
>> apt-get update
>
> So, until Cloudera fixes the install (or documentation) in order to insta=
ll the CDH3 on Ubuntu 10.04 (lucid) the steps should look like:
>
>> apt-cache search hadoop
>> echo "deb http://archive.canonical.com/ lucid partner" | sudo tee /etc/a=
pt/sources.list.d/partner.list
>> apt-get update
>> sudo apt-get install hadoop-0.20
>
> Does anyone know, if this is being looked at?
>
> Thanks,
>
> Peter
>
>
>

From general-return-2385-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 12 23:49:08 2010
Return-Path: <general-return-2385-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95922 invoked from network); 12 Nov 2010 23:49:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Nov 2010 23:49:08 -0000
Received: (qmail 6657 invoked by uid 500); 12 Nov 2010 23:49:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6587 invoked by uid 500); 12 Nov 2010 23:49:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6579 invoked by uid 99); 12 Nov 2010 23:49:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Nov 2010 23:49:38 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of amp@opendns.com designates 67.215.68.163 as permitted sender)
Received: from [67.215.68.163] (HELO mail.opendns.com) (67.215.68.163)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Nov 2010 23:49:30 +0000
Received: from Adams-Desktop.local ([67.215.69.42])
	(authenticated bits=0)
	by mail.opendns.com (8.14.3/8.14.3/Debian-5) with ESMTP id oACNn9s8001920
	(version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Fri, 12 Nov 2010 23:49:09 GMT
Message-ID: <4CDDD275.9020901@opendns.com>
Date: Fri, 12 Nov 2010 15:49:09 -0800
From: Adam Phelps <amp@opendns.com>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Starting hadoop services at boot
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

I'm sure there is some detail I'm missing, but I've been testing node 
failures and noticed that the various services (datanode, tasktracker, 
regionserver in my case) don't automatically restart on boot (I'm 
running on Ubuntu on EC2).  Its currently hiding from me, however I 
assume there is a configuration setting somewhere that enables this?

I noticed that there various start scripts in /etc/init.d:

root@ip-10-123-1-151:~/control# ls /etc/init.d/hadoop*
/etc/init.d/hadoop-0.20-datanode    /etc/init.d/hadoop-0.20-namenode 
        /etc/init.d/hadoop-0.20-tasktracker 
/etc/init.d/hadoop-hbase-regionserver
/etc/init.d/hadoop-0.20-jobtracker 
/etc/init.d/hadoop-0.20-secondarynamenode 
/etc/init.d/hadoop-hbase-master      /etc/init.d/hadoop-hbase-thrift

However these don't actually appear to work in this setup:

root@ip-10-123-1-151:~/control# /etc/init.d/hadoop-0.20-namenode stop
Stopping Hadoop namenode daemon: no namenode to stop
ERROR
root@ip-10-123-1-151:~/control# /etc/init.d/hadoop-0.20-namenode start
Starting Hadoop namenode daemon: starting namenode, logging to 
/usr/lib/hadoop-0.20/bin/../logs/hadoop-root-namenode-ip-10-123-1-151.out
/usr/lib/hadoop-0.20/bin/hadoop-daemon.sh: line 114: 
/var/run/hadoop/hadoop-root-namenode.pid: Permission denied
nice: cannot set niceness: Permission denied
ERROR.

I'm currently starting services using "hadoop-daemon.sh start XXX".

Thanks
- Adam

From general-return-2386-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Nov 14 02:54:02 2010
Return-Path: <general-return-2386-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60075 invoked from network); 14 Nov 2010 02:54:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Nov 2010 02:54:01 -0000
Received: (qmail 39550 invoked by uid 500); 14 Nov 2010 02:54:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39307 invoked by uid 500); 14 Nov 2010 02:54:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39299 invoked by uid 99); 14 Nov 2010 02:54:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 14 Nov 2010 02:54:29 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of wbsrvc@gmail.com designates 209.85.216.176 as permitted sender)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 14 Nov 2010 02:54:23 +0000
Received: by qyk29 with SMTP id 29so1518216qyk.14
        for <general@hadoop.apache.org>; Sat, 13 Nov 2010 18:54:02 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=dBVsMpk3BSCCEaV7yD8tRs7Tp9tq6LOiU9Hb+KspOW8=;
        b=WKWFAEnToH2m8LgJju/VeECKRBwhXPar6pwnqYLAZqOCsUo9tmi+jHLySVl+U1Yzi5
         Ok1VJcndfQDuBr7tsPHArHtytDWgeW5wTXpg6AJOI9muvyn8A1X0wQBE6DfuUr8LE23l
         Kpar7xpkx+UPJjVpf8pZGEo36+UHzWV5zeFsc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=mDzHbl7MmqgjG6fcDw2L2FMTc2fzKta9ETZYdE8IAJIsbcG2lx2RM7A40kYqmuXc6h
         wwyOWhJQ8pTlXpXvwDhcXgbyabGD2Ha08DPZ/xRZkXgRNYloucVhoziFP5dH21ppBl86
         H8DpYmieLY+BiaMCjPjGRWES07LvQSD5wxwgM=
MIME-Version: 1.0
Received: by 10.229.242.66 with SMTP id lh2mr3493314qcb.118.1289703242633;
 Sat, 13 Nov 2010 18:54:02 -0800 (PST)
Received: by 10.229.33.83 with HTTP; Sat, 13 Nov 2010 18:54:02 -0800 (PST)
Date: Sat, 13 Nov 2010 19:54:02 -0700
Message-ID: <AANLkTineMQyYr3mSHLXxE+2W5+Tt2_mZkrNma4n+Q160@mail.gmail.com>
Subject: Accessing hdfs from a program not run with bin/hadoop
From: web service <wbsrvc@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016362843d6a4907c0494fa6f39

--0016362843d6a4907c0494fa6f39
Content-Type: text/plain; charset=ISO-8859-1

I am running a program with the following code.

[code]
............
            conf.addResource(new
Path("file:///home/mac/software/hadoop/hadoop-0.20.2/conf/core-site.xml"));
            fs = FileSystem.get(conf);
            fs.delete(new
Path("hdfs://localhost:9000/user/mac/output/output-1"), true);
............
[/code]

but I get the following exception, not sure what else configuration I would
want to add.


[code]
Exception --
.......
Wrong FS: hdfs://localhost:9000/user/mac/output/output-1, expected: file:///
.......

[/code]

--0016362843d6a4907c0494fa6f39--

From general-return-2387-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Nov 14 05:18:30 2010
Return-Path: <general-return-2387-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 11604 invoked from network); 14 Nov 2010 05:18:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Nov 2010 05:18:30 -0000
Received: (qmail 72538 invoked by uid 500); 14 Nov 2010 05:19:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72334 invoked by uid 500); 14 Nov 2010 05:18:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72326 invoked by uid 99); 14 Nov 2010 05:18:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 14 Nov 2010 05:18:58 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of wbsrvc@gmail.com designates 209.85.216.169 as permitted sender)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 14 Nov 2010 05:18:52 +0000
Received: by qyk1 with SMTP id 1so1320620qyk.14
        for <general@hadoop.apache.org>; Sat, 13 Nov 2010 21:18:31 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=c9pxumRkLomYw5TMyed2rcKCVm5vEeA+ALRtdIbrB2g=;
        b=vjz2I1t85ZgvEDs5gru+Yv28iXtCi1NpImhWMvK7fYEXc7xik5gaZoTsRQU0Nd0OVA
         syz3/Crw3AsNSRPFdVriAsOpA3xuqv/0aDMlvtrd7pr+CekVE2l4yFA70TqAKyHKyDJn
         XGgApqI8WyXUbvS7eHSuP9GSYsaLs0163gUsw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=aVzXpd7PAOF/LpBwmc2D2kr3zLMM/8jDdNW/FHTSQ4n78poGjruz/jqpsIJnrizkRE
         AzN8L7R51Q3+UeNgPyb4G/Uw6nRC69lSvcV5NtLLcBSjktgJrnxeAAnLlQ5QBe/MHBVF
         07adhjw+kzY8Cy/Lhkq556rQkYMPa3ek0xAAc=
MIME-Version: 1.0
Received: by 10.229.90.196 with SMTP id j4mr3595330qcm.270.1289711911282; Sat,
 13 Nov 2010 21:18:31 -0800 (PST)
Received: by 10.229.33.83 with HTTP; Sat, 13 Nov 2010 21:18:31 -0800 (PST)
In-Reply-To: <AANLkTineMQyYr3mSHLXxE+2W5+Tt2_mZkrNma4n+Q160@mail.gmail.com>
References: <AANLkTineMQyYr3mSHLXxE+2W5+Tt2_mZkrNma4n+Q160@mail.gmail.com>
Date: Sat, 13 Nov 2010 22:18:31 -0700
Message-ID: <AANLkTintCKVn+bFZzCuX+sik76oOFE5A=6w1318AZNPF@mail.gmail.com>
Subject: Re: Accessing hdfs from a program not run with bin/hadoop
From: web service <wbsrvc@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163631064955a8970494fc74e9

--00163631064955a8970494fc74e9
Content-Type: text/plain; charset=ISO-8859-1

Ok, I need to access file system appropriately.
This works.

hdfs = FileSystem.get(new URI(HDFS_PATH),conf);



On Sat, Nov 13, 2010 at 7:54 PM, web service <wbsrvc@gmail.com> wrote:

> I am running a program with the following code.
>
> [code]
> ............
>             conf.addResource(new
> Path("file:///home/mac/software/hadoop/hadoop-0.20.2/conf/core-site.xml"));
>             fs = FileSystem.get(conf);
>             fs.delete(new
> Path("hdfs://localhost:9000/user/mac/output/output-1"), true);
> ............
> [/code]
>
> but I get the following exception, not sure what else configuration I would
> want to add.
>
>
> [code]
> Exception --
> .......
> Wrong FS: hdfs://localhost:9000/user/mac/output/output-1, expected:
> file:///
> .......
>
> [/code]
>

--00163631064955a8970494fc74e9--

From general-return-2388-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Nov 14 12:23:09 2010
Return-Path: <general-return-2388-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76671 invoked from network); 14 Nov 2010 12:23:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Nov 2010 12:23:08 -0000
Received: (qmail 50563 invoked by uid 500); 14 Nov 2010 12:23:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50317 invoked by uid 500); 14 Nov 2010 12:23:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50309 invoked by uid 99); 14 Nov 2010 12:23:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 14 Nov 2010 12:23:36 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 14 Nov 2010 12:23:31 +0000
Received: by fxm10 with SMTP id 10so3045433fxm.35
        for <general@hadoop.apache.org>; Sun, 14 Nov 2010 04:23:09 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=PIHqZIRH1RBWI4QpUUEICtQSqvYFIVJKdeilWdRdMEc=;
        b=AOggxWE8JiSSp9wslJoB2PXMD6N8euNv8evtwioU3datLfp/27PEjsgjKJdTJCi1/W
         lRPBMM+va/Y6b0zBF7Sh/L6DQmh4m/DZO+RtrSofjUcet3XoW342hsAjlPftpfg+Jr1q
         yCqDTTfM6eWfLHLcLo25QOAAIwwgDUet1ytPg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=THY+/xjNKNmhy7wnQHOPFlidWdKSH+J9RNVtzvzN7lkd9btQol1IVLzyiZbouCCO8c
         gFuYSypQYpvs+7tuo1TyngkWGXosl0yns1nlMYyAQkeHR0aj56Eitm68FpgpRDtPUszs
         oeO1Y0fp+iirAtPAVcfXU4WPWkl/QYAzLwMkA=
Received: by 10.223.110.148 with SMTP id n20mr1866472fap.48.1289737389497;
 Sun, 14 Nov 2010 04:23:09 -0800 (PST)
MIME-Version: 1.0
Received: by 10.223.113.145 with HTTP; Sun, 14 Nov 2010 04:22:49 -0800 (PST)
In-Reply-To: <AANLkTim_ZJLjmVUBLDAtzPYqcSxp=4jSDi7tG0COT-hx@mail.gmail.com>
References: <AANLkTinQVUC0U9CEONpW13+MLXHOVQWAqfvyhG=1btjw@mail.gmail.com>
 <AANLkTinr0mO0N4c8+oqk1T5=9NbmdVxVpbbGJnxC4jKV@mail.gmail.com> <AANLkTim_ZJLjmVUBLDAtzPYqcSxp=4jSDi7tG0COT-hx@mail.gmail.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Sun, 14 Nov 2010 17:52:49 +0530
Message-ID: <AANLkTi=bifFfn=OZ5eOaXPdYaMb1XrwB-noqeAqDOF=8@mail.gmail.com>
Subject: Re: running hadoop jobs from within a program
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hello,

On Fri, Nov 12, 2010 at 10:25 PM, web service <wbsrvc@gmail.com> wrote:
> Thanks, but submitting three different jobs say using
>
> JobClient.submitjob(jobconf1);
> JobClient.submitjob(jobconf2);
> JobClient.submitjob(jobconf3)
>
> different from running -
> tmp=3D"$HADOOP_BIN jar $JAR_LOC =A0$MAIN_CLASS /user/joe/input/input-1/
> /user/vadmin/output/output-1/
> tmp=3D"$HADOOP_BIN jar $JAR_LOC =A0$MAIN_CLASS /user/joe/input/input-2/
> /user/vadmin/output/output-2/
> tmp=3D"$HADOOP_BIN jar $JAR_LOC =A0$MAIN_CLASS /user/joe/input/input-3/
> /user/vadmin/output/output-3/

It isn't different. In both cases a new JobID is assigned for each job
created and its specific configuration is associated to it upon
submission.

>
> I guess every job can have specific jvm options. and I hope that every
> submitted job runs in a separate jvm, No ?

Yes, each Task (Map or Reduce, under the Job) runs in a separate JVM
(although JVMs can be reused using a tweak).

--=20
Harsh J
www.harshj.com

From general-return-2389-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Nov 14 22:40:38 2010
Return-Path: <general-return-2389-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 15904 invoked from network); 14 Nov 2010 22:40:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Nov 2010 22:40:37 -0000
Received: (qmail 14643 invoked by uid 500); 14 Nov 2010 22:41:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14600 invoked by uid 500); 14 Nov 2010 22:41:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14592 invoked by uid 99); 14 Nov 2010 22:41:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 14 Nov 2010 22:41:07 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of wbsrvc@gmail.com designates 209.85.216.176 as permitted sender)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 14 Nov 2010 22:40:59 +0000
Received: by qyk29 with SMTP id 29so2059413qyk.14
        for <general@hadoop.apache.org>; Sun, 14 Nov 2010 14:40:38 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=ifFgTI3TXXBpZAVWA/ZnscC8SDHuf8V4xqoWm2XwY0A=;
        b=dYGcmHvle5AC2YjQF79HKs1HpkER+WrmOXD289lcxegs880nuiTCXFj4BS0fmu/BMj
         mZU3rYdD2w5zPCBDjTZi7OL0GL95hWpYZvncD1leL8Gzr+sDIwPtERw0fuHPelKpmyL7
         tF6cjNM8YwdeNd1luFXLDDbWpLZ/dcyNO3gsY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=rKCnT/W+3N6yP61tT7Z1tyyo55GICiR4E7noMhtwD6i5fG9OrSFNlMjMcW4JGX6Iu3
         i31/QIBsAprRTmG43rRVOF9hiw95wtimPIjFEC3fJXq3C9fqL1IiO415pnXbQ8YgGAsB
         snQKGz9SOHuvlmg+PREXQ5y3i5D/g0kfCcMK4=
MIME-Version: 1.0
Received: by 10.229.85.208 with SMTP id p16mr4263509qcl.4.1289774437920; Sun,
 14 Nov 2010 14:40:37 -0800 (PST)
Received: by 10.229.33.83 with HTTP; Sun, 14 Nov 2010 14:40:37 -0800 (PST)
In-Reply-To: <AANLkTi=bifFfn=OZ5eOaXPdYaMb1XrwB-noqeAqDOF=8@mail.gmail.com>
References: <AANLkTinQVUC0U9CEONpW13+MLXHOVQWAqfvyhG=1btjw@mail.gmail.com>
	<AANLkTinr0mO0N4c8+oqk1T5=9NbmdVxVpbbGJnxC4jKV@mail.gmail.com>
	<AANLkTim_ZJLjmVUBLDAtzPYqcSxp=4jSDi7tG0COT-hx@mail.gmail.com>
	<AANLkTi=bifFfn=OZ5eOaXPdYaMb1XrwB-noqeAqDOF=8@mail.gmail.com>
Date: Sun, 14 Nov 2010 14:40:37 -0800
Message-ID: <AANLkTinjX0BXZZw2EeUyGGx+aSy-CoyjhXuUDP4x102e@mail.gmail.com>
Subject: Re: running hadoop jobs from within a program
From: web service <wbsrvc@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016368322d436712804950b0334
X-Virus-Checked: Checked by ClamAV on apache.org

--0016368322d436712804950b0334
Content-Type: text/plain; charset=ISO-8859-1

Thanks, had figured it out. It is fun to figure out how things work :)

On Sun, Nov 14, 2010 at 4:22 AM, Harsh J <qwertymaniac@gmail.com> wrote:

> Hello,
>
> On Fri, Nov 12, 2010 at 10:25 PM, web service <wbsrvc@gmail.com> wrote:
> > Thanks, but submitting three different jobs say using
> >
> > JobClient.submitjob(jobconf1);
> > JobClient.submitjob(jobconf2);
> > JobClient.submitjob(jobconf3)
> >
> > different from running -
> > tmp="$HADOOP_BIN jar $JAR_LOC  $MAIN_CLASS /user/joe/input/input-1/
> > /user/vadmin/output/output-1/
> > tmp="$HADOOP_BIN jar $JAR_LOC  $MAIN_CLASS /user/joe/input/input-2/
> > /user/vadmin/output/output-2/
> > tmp="$HADOOP_BIN jar $JAR_LOC  $MAIN_CLASS /user/joe/input/input-3/
> > /user/vadmin/output/output-3/
>
> It isn't different. In both cases a new JobID is assigned for each job
> created and its specific configuration is associated to it upon
> submission.
>
> >
> > I guess every job can have specific jvm options. and I hope that every
> > submitted job runs in a separate jvm, No ?
>
> Yes, each Task (Map or Reduce, under the Job) runs in a separate JVM
> (although JVMs can be reused using a tweak).
>
> --
> Harsh J
> www.harshj.com
>

--0016368322d436712804950b0334--

From general-return-2390-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 16 11:00:31 2010
Return-Path: <general-return-2390-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 11832 invoked from network); 16 Nov 2010 11:00:31 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Nov 2010 11:00:31 -0000
Received: (qmail 97967 invoked by uid 500); 16 Nov 2010 11:01:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97765 invoked by uid 500); 16 Nov 2010 11:00:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97757 invoked by uid 99); 16 Nov 2010 11:00:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Nov 2010 11:00:59 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [156.148.72.33] (HELO raffaello.crs4.it) (156.148.72.33)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Nov 2010 11:00:52 +0000
Received: from [156.148.71.202] (neuron.crs4.it [156.148.71.202])
	by raffaello.crs4.it (Postfix) with ESMTP id 78FF36BBA1
	for <general@hadoop.apache.org>; Tue, 16 Nov 2010 11:57:18 +0100 (CET)
Message-ID: <4CE2644D.6010606@crs4.it>
Date: Tue, 16 Nov 2010 12:00:29 +0100
From: Simone Leo <simone.leo@crs4.it>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.12) Gecko/20101102 Lightning/1.0b3pre Thunderbird/3.1.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Announcing Pydoop 0.3.7_rc1
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit

Hello everybody,

The new Pydoop release with support for Hadoop 0.21 is out:

Support for 0.20.2 is still there: you decide which version you want to
use at compile time, by defining the HADOOP_HOME environment variable.

As usual, we appreciate everyone's feedback.

Links:
 * download page: http://sourceforge.net/projects/pydoop/files
 * release notes:
http://sourceforge.net/apps/mediawiki/pydoop/index.php?title=Release_Notes

-- 
Simone Leo
Data Fusion - Distributed Computing
CRS4
POLARIS - Building #1
Piscina Manna
I-09010 Pula (CA) - Italy
e-mail: simone.leo@crs4.it
http://www.crs4.it

From general-return-2391-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 16 15:25:17 2010
Return-Path: <general-return-2391-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42209 invoked from network); 16 Nov 2010 15:25:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Nov 2010 15:25:16 -0000
Received: (qmail 17934 invoked by uid 500); 16 Nov 2010 15:25:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17834 invoked by uid 500); 16 Nov 2010 15:25:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 4491 invoked by uid 99); 15 Nov 2010 20:52:51 -0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cornelio.inigof@gmail.com designates 74.125.82.48 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=C3zpUQpG3i6mxcj1ZQsy5LewjTYNpEUdvhwnjAjqbtQ=;
        b=RNalcgsSCfOfMEplmcEOWWtFuVAvfAtfphP3ntQhX+eWoy3ldul+W7tMR3QXh6JJde
         zC/Xpm4udSwIwyQ50m+0LhH0TVMP6vaHrSaNa6wcjma9ZB8128Qhr3Lhktr9NLsoJ/rk
         NsWF26Csi3XNnm7SsxskkfXC9NcKty0LhsH2U=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=E3XNmwboGnSzMKGb7xg/BsoO3inpbPqo7kg9q8sxOtlI3fhypYPNOoLfLH9GYQu8z7
         NdRvCtZdTIvpfVbgWeUzrsngKpNp7xvuWnzGcbU1BflOeMU8KLNsKIXlkW1KhNJwrCqE
         sUamzE2/+kzbRYNZsW4Vd5SsCkeKKFthrL6N8=
MIME-Version: 1.0
Date: Mon, 15 Nov 2010 12:52:22 -0800
Message-ID: <AANLkTi=pCpBqJyAJgdSCCkY=G9zqji8GEa2Uc2uT4nmd@mail.gmail.com>
Subject: hadoop application to pig?
From: =?ISO-8859-1?Q?Cornelio_I=F1igo?= <cornelio.inigof@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364d1ca5ef5fbb04951d9d9b

--0016364d1ca5ef5fbb04951d9d9b
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi

My name is Cornelio I=F1igo and I=B4m a developer just beginning with this =
of
hadoop and pig.
I have a doubt about developing an application on pig, I already have my
program on hadoop, this program gets just a column from a dataset (csv file=
)
and process this data with some functions (like language analisis, analysis
of the content)
 note that in the process of the file I dont use FILTERS COUNTS or any buil=
t
in function of Pig, I think that all the fucntions have to be User Defined
Functions

 so Is a good idea (has sense ) to develop this program in Pig?

Thanks in advice

--=20
*Cornelio*

--0016364d1ca5ef5fbb04951d9d9b--

From general-return-2392-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 16 15:46:37 2010
Return-Path: <general-return-2392-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49550 invoked from network); 16 Nov 2010 15:46:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Nov 2010 15:46:36 -0000
Received: (qmail 53614 invoked by uid 500); 16 Nov 2010 15:47:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53500 invoked by uid 500); 16 Nov 2010 15:47:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53491 invoked by uid 99); 16 Nov 2010 15:47:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Nov 2010 15:47:05 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of dgisrael@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Nov 2010 15:46:57 +0000
Received: by pwj9 with SMTP id 9so174246pwj.35
        for <general@hadoop.apache.org>; Tue, 16 Nov 2010 07:46:36 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=YrhwB/U3g0VQNPM/Ipr61kjGLeDw4hoO8CO5K03TLco=;
        b=ElbBoVV9ixOw6nk41Plt19dyJ1tZtgbgydr1QNjQxgGK6fJxjoSDJHQiExjl50OuSx
         l6jEy/la9LBi3Rub7r/wzIBtVQ8PFRpSWK+lG5BfUCR2OgN/XvIiLlpFu1hJROkLCyzn
         faahWjJRS47KEyba59BAGq2j76Owqriq+GA3U=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=MoI1P8iswpUM2QS1RDZKSnZTHCKlJYm5KktYrRrbfYRoRWu2Zr4eOecSTC+OArI+Um
         7kuAtLcdp0J0MpRh5xUd/oO8pbWo7Y+aK6Layi9bxEt2zXq0VQViuxyAoUxRLFezruEh
         N+7XrqRHLxyq5XRuMRUHzBjW0WQG7zoOTPG+0=
MIME-Version: 1.0
Received: by 10.223.87.67 with SMTP id v3mr4746124fal.130.1289922395138; Tue,
 16 Nov 2010 07:46:35 -0800 (PST)
Received: by 10.223.13.68 with HTTP; Tue, 16 Nov 2010 07:46:35 -0800 (PST)
In-Reply-To: <AANLkTi=pCpBqJyAJgdSCCkY=G9zqji8GEa2Uc2uT4nmd@mail.gmail.com>
References: <AANLkTi=pCpBqJyAJgdSCCkY=G9zqji8GEa2Uc2uT4nmd@mail.gmail.com>
Date: Tue, 16 Nov 2010 17:46:35 +0200
Message-ID: <AANLkTinFaSTCngDXC-7qZzswHvwBAnSF9hSCSTMjHexX@mail.gmail.com>
Subject: Re: hadoop application to pig?
From: David Gruzman <dgisrael@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf30434548266d7404952d76c8
X-Virus-Checked: Checked by ClamAV on apache.org

--20cf30434548266d7404952d76c8
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I am developing the similar application using vanilla map-reduce jobs. I
think that any specific processing
can be developed using MapReduce and probabbly will be more efficient.
Higher level services like Pig or Hive should be used if you need to issue
"ad hoc" queries to the data.
With best regards,
David

On Mon, Nov 15, 2010 at 10:52 PM, Cornelio I=F1igo
<cornelio.inigof@gmail.com>wrote:

> Hi
>
> My name is Cornelio I=F1igo and I=B4m a developer just beginning with thi=
s of
> hadoop and pig.
> I have a doubt about developing an application on pig, I already have my
> program on hadoop, this program gets just a column from a dataset (csv
> file)
> and process this data with some functions (like language analisis, analys=
is
> of the content)
>  note that in the process of the file I dont use FILTERS COUNTS or any
> built
> in function of Pig, I think that all the fucntions have to be User Define=
d
> Functions
>
>  so Is a good idea (has sense ) to develop this program in Pig?
>
> Thanks in advice
>
> --
> *Cornelio*
>

--20cf30434548266d7404952d76c8--

From general-return-2393-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 16 18:07:19 2010
Return-Path: <general-return-2393-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 28680 invoked from network); 16 Nov 2010 18:07:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Nov 2010 18:07:18 -0000
Received: (qmail 5559 invoked by uid 500); 16 Nov 2010 18:07:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5254 invoked by uid 500); 16 Nov 2010 18:07:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5194 invoked by uid 99); 16 Nov 2010 18:07:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Nov 2010 18:07:45 +0000
X-ASF-Spam-Status: No, hits=-0.1 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of gcabrer@us.ibm.com designates 32.97.110.149 as permitted sender)
Received: from [32.97.110.149] (HELO e31.co.us.ibm.com) (32.97.110.149)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Nov 2010 18:07:35 +0000
Received: from d03relay02.boulder.ibm.com (d03relay02.boulder.ibm.com [9.17.195.227])
	by e31.co.us.ibm.com (8.14.4/8.13.1) with ESMTP id oAGHrusD007176
	for <general@hadoop.apache.org>; Tue, 16 Nov 2010 10:53:56 -0700
Received: from d03av01.boulder.ibm.com (d03av01.boulder.ibm.com [9.17.195.167])
	by d03relay02.boulder.ibm.com (8.13.8/8.13.8/NCO v9.1) with ESMTP id oAGI72Uf178642
	for <general@hadoop.apache.org>; Tue, 16 Nov 2010 11:07:04 -0700
Received: from d03av01.boulder.ibm.com (loopback [127.0.0.1])
	by d03av01.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id oAGI70kR028471
	for <general@hadoop.apache.org>; Tue, 16 Nov 2010 11:07:00 -0700
Received: from d03nm119.boulder.ibm.com (d03nm119.boulder.ibm.com [9.17.195.145])
	by d03av01.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id oAGI6wRf028220
	for <general@hadoop.apache.org>; Tue, 16 Nov 2010 11:06:59 -0700
Subject: Resources on building Hadoop with Apache Harmony Select
X-KeepSent: 2847073D:085BCB36-862577DD:0063032E;
 type=4; name=$KeepSent
To: general@hadoop.apache.org
X-Mailer: Lotus Notes Build V852_06012010 June 01, 2010
Message-ID: <OF2847073D.085BCB36-ON862577DD.0063032E-862577DD.00638346@us.ibm.com>
From: Guillermo Cabrera <gcabrer@us.ibm.com>
Date: Tue, 16 Nov 2010 12:06:56 -0600
X-MIMETrack: Serialize by Router on D03NM119/03/M/IBM(Release 8.5.1FP2|March 17, 2010) at
 11/16/2010 11:06:58
MIME-Version: 1.0
Content-type: multipart/alternative; 
	Boundary="0__=09BBFD4EDFF085BE8f9e8a93df938690918c09BBFD4EDFF085BE"
Content-Disposition: inline
X-Virus-Checked: Checked by ClamAV on apache.org

--0__=09BBFD4EDFF085BE8f9e8a93df938690918c09BBFD4EDFF085BE
Content-type: text/plain; charset=US-ASCII
Content-transfer-encoding: quoted-printable



Hello:

For the past few months, we have been working on getting Hadoop (common=
,
hdfs and mapreduce) working on Apache Harmony Select. We have documente=
d
the steps we followed, issues we encountered and scripts we used in thi=
s
process. Please refer to the following link for further information.

http://wiki.apache.org/hadoop/HadoopOnHarmony

Regards,
Guillermo
----------------------------------------------
IBM Emerging Internet Technology Group=

--0__=09BBFD4EDFF085BE8f9e8a93df938690918c09BBFD4EDFF085BE--


From general-return-2394-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 16 22:33:00 2010
Return-Path: <general-return-2394-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77761 invoked from network); 16 Nov 2010 22:32:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Nov 2010 22:32:59 -0000
Received: (qmail 22558 invoked by uid 500); 16 Nov 2010 22:33:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22466 invoked by uid 500); 16 Nov 2010 22:33:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22457 invoked by uid 99); 16 Nov 2010 22:33:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Nov 2010 22:33:29 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=MIME_QP_LONG_LINE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Nov 2010 22:33:20 +0000
Received: by yxm8 with SMTP id 8so873103yxm.35
        for <general@hadoop.apache.org>; Tue, 16 Nov 2010 14:32:59 -0800 (PST)
Received: by 10.151.13.20 with SMTP id q20mr12786305ybi.434.1289946778981;
        Tue, 16 Nov 2010 14:32:58 -0800 (PST)
Received: from [10.139.42.54] ([166.137.8.36])
        by mx.google.com with ESMTPS id i64sm1095508yha.10.2010.11.16.14.32.57
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Tue, 16 Nov 2010 14:32:58 -0800 (PST)
References: <AANLkTi=pCpBqJyAJgdSCCkY=G9zqji8GEa2Uc2uT4nmd@mail.gmail.com> <AANLkTinFaSTCngDXC-7qZzswHvwBAnSF9hSCSTMjHexX@mail.gmail.com>
In-Reply-To: <AANLkTinFaSTCngDXC-7qZzswHvwBAnSF9hSCSTMjHexX@mail.gmail.com>
Mime-Version: 1.0 (iPhone Mail 8B117)
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8
Message-Id: <FCF57ACB-4D7A-44BA-B89F-D01A9EF0CD39@holsman.net>
Cc: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: iPhone Mail (8B117)
From: Ian Holsman <hadoop@holsman.net>
Subject: Re: hadoop application to pig?
Date: Tue, 16 Nov 2010 17:32:47 -0500
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

I think it comes down to if you can translate your existing jobs into udfs a=
nd custom steps which can run inside of a pig script, and how much effort th=
at will take.=20

Writing pig scripts is much easier than writing m-r ones in my experience.=20=


---
Ian Holsman - 703 879-3128

I saw the angel in the marble and carved until I set him free -- Michelangel=
o

On 16/11/2010, at 10:46 AM, David Gruzman <dgisrael@gmail.com> wrote:

> I am developing the similar application using vanilla map-reduce jobs. I
> think that any specific processing
> can be developed using MapReduce and probabbly will be more efficient.
> Higher level services like Pig or Hive should be used if you need to issue=

> "ad hoc" queries to the data.
> With best regards,
> David
>=20
> On Mon, Nov 15, 2010 at 10:52 PM, Cornelio I=C3=B1igo
> <cornelio.inigof@gmail.com>wrote:
>=20
>> Hi
>>=20
>> My name is Cornelio I=C3=B1igo and I=C2=B4m a developer just beginning wi=
th this of
>> hadoop and pig.
>> I have a doubt about developing an application on pig, I already have my
>> program on hadoop, this program gets just a column from a dataset (csv
>> file)
>> and process this data with some functions (like language analisis, analys=
is
>> of the content)
>> note that in the process of the file I dont use FILTERS COUNTS or any
>> built
>> in function of Pig, I think that all the fucntions have to be User Define=
d
>> Functions
>>=20
>> so Is a good idea (has sense ) to develop this program in Pig?
>>=20
>> Thanks in advice
>>=20
>> --
>> *Cornelio*
>>=20

From general-return-2395-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 17 06:46:57 2010
Return-Path: <general-return-2395-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19209 invoked from network); 17 Nov 2010 06:46:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Nov 2010 06:46:57 -0000
Received: (qmail 16747 invoked by uid 500); 17 Nov 2010 06:47:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16526 invoked by uid 500); 17 Nov 2010 06:47:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16518 invoked by uid 99); 17 Nov 2010 06:47:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Nov 2010 06:47:26 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.98 as permitted sender)
Received: from [17.148.16.98] (HELO asmtpout023.mac.com) (17.148.16.98)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Nov 2010 06:47:18 +0000
MIME-version: 1.0
Content-type: multipart/alternative;
 boundary="Boundary_(ID_yJJ6TMWsPcbnXv8ticquWQ)"
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp023.mac.com (Sun Java(tm) System Messaging Server 6.3-8.01 (built Dec
 16 2008; 32bit)) with ESMTPSA id <0LC0006P7O54MA60@asmtp023.mac.com> for
 general@hadoop.apache.org; Tue, 16 Nov 2010 22:46:17 -0800 (PST)
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1011160287
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2010-11-16_12:2010-11-16,2010-11-16,1970-01-01 signatures=0
From: Nigel Daley <ndaley@mac.com>
Subject: Re: Patch testing
Date: Tue, 16 Nov 2010 22:46:16 -0800
In-reply-to: <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
To: general@hadoop.apache.org
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com>
 <20101020195420.GG2075@tp>
 <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com>
 <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
Message-id: <7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com>
X-Mailer: Apple Mail (2.1081)
X-Virus-Checked: Checked by ClamAV on apache.org

--Boundary_(ID_yJJ6TMWsPcbnXv8ticquWQ)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT

Just to follow up, the pre-commit patch testing is functioning again on Zookeeper and Hadoop Common.  It's also ready to run on MapReduce and HDFS but we won't turn it on until these projects build and test cleanly.  Looks like both these projects currently have test failures.  Are there Jira's to fix these tests?

New instructions for re-trigger testing of a patch are at https://hudson.apache.org/hudson/job/PreCommit-Admin/

Cheers,
Nige

On Oct 20, 2010, at 1:30 PM, Nigel Daley wrote:

> Thanks Giri.
> 
>> Longterm solution: 
>> email patch process and queue creation is un-reliable. As a long term solution we need to use jira-cli command line tool to read patch directly from jira. 
> 
> Ya, that's what I've got mostly working on my machine.  Any open Jira's for this work?  If not, I'll open one.
> 
>> And doing this would require introducing a new workflow in jira. Apart from the current status called "patch-available"
> 
> Not sure that's necessary.  I think if we change the patch testing model a little, we can keep the current workflow.  Also, given some recent advances in Hudson, I think we can reduce all the patch jobs down to 1 admin job covering *all* projects and 1 job per project -- and we should be able to drastically improve utilization of the slaves we have.  Let's move this discussion to a ticket.  LMK if I need to open one.
> 
> Cheers,
> Nige
> 
> On Oct 20, 2010, at 1:17 PM, Giridharan Kesavan wrote:
> 
>> 
>> Old times:
>> The hudson patch-testing admin job used to run on the master machine hudson.zones.apache.org as this machine used to parse the emails from jira and create the patch queue for testing.
>> 
>> Current situation: 
>> hudson.zones.apache.org machine is no more a master and we have a new machine called aegis.apache.org as the master machine. 
>> 
>> This machine is configured to process the email and create the patch queue.  <done>
>> Though we are able to get the email and create the patch queue on the new aegis.apache.org machine, still we wont be able to trigger patch admin job on the hudson master as this is not allowed in the new setup.  
>> 
>> Infra team doesn't want us to run any builds on the hudson master. Instead they want all the builds to run on the slave machine.  
>> 
>> I got the password-less access for hudson user from hudson slave h1.grid.sp2.yahoo.net to aegis.apache.org like 2 weeks back. <done>   
>> 
>> The plan here is to read the patch queue on aegis from h1 and schedule builds.   <I'm working on getting this to work>
>> 
>> Longterm solution: 
>> email patch process and queue creation is un-reliable. As a long term solution we need to use jira-cli command line tool to read patch directly from jira. 
>> And doing this would require introducing a new workflow in jira. Apart from the current status called "patch-available"
>> 
>> Thanks,
>> Giri
>> 
>> On Oct 20, 2010, at 12:54 PM, Konstantin Boudnik wrote:
>> 
>>> Thanks for getting to it, Nigel. This is absolutely useful and allows to keep
>>> weeds out of the trunk up to a certain degree.
>>> 
>>> There were at least two slightly different sources of information about what's
>>> going on with test-patch process. Would you mind to post a brief about the
>>> situation so comments can be more substantial?
>>> 
>>> Thanks,
>>> Cos
>>> 
>>> On Wed, Oct 20, 2010 at 12:47PM, Nigel Daley wrote:
>>>> Folks, 
>>>> 
>>>> I'm working to get the pre-commit patch testing running again for HDFS,
>>>> HADOOP, and MAPREDUCE patches.  Since I've been on a bit of a hiatus from
>>>> day-to-day involvement w/ the project over the last 6 months (new job), I
>>>> want to check in and make sure folks would still find this pre-commit
>>>> testing useful.  Also, happy to hear any suggested improvement.  Let me
>>>> know.  
>>>> 
>>>> Cheers,
>>>> Nige
>> 
> 


--Boundary_(ID_yJJ6TMWsPcbnXv8ticquWQ)--

From general-return-2396-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 17 07:24:45 2010
Return-Path: <general-return-2396-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 36548 invoked from network); 17 Nov 2010 07:24:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Nov 2010 07:24:45 -0000
Received: (qmail 48651 invoked by uid 500); 17 Nov 2010 07:25:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48446 invoked by uid 500); 17 Nov 2010 07:25:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48433 invoked by uid 99); 17 Nov 2010 07:25:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Nov 2010 07:25:14 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Nov 2010 07:25:09 +0000
Received: by yxm8 with SMTP id 8so1070157yxm.35
        for <general@hadoop.apache.org>; Tue, 16 Nov 2010 23:24:46 -0800 (PST)
Received: by 10.100.208.12 with SMTP id f12mr5973384ang.38.1289978686364;
        Tue, 16 Nov 2010 23:24:46 -0800 (PST)
Received: from tp (c-24-4-185-157.hsd1.ca.comcast.net [24.4.185.157])
        by mx.google.com with ESMTPS id g29sm6069658anh.16.2010.11.16.23.24.43
        (version=SSLv3 cipher=RC4-MD5);
        Tue, 16 Nov 2010 23:24:45 -0800 (PST)
Sender: Konstantin Boudnik <cos@boudnik.org>
Date: Tue, 16 Nov 2010 23:24:38 -0800
From: Konstantin Boudnik <cos@apache.org>
To: general@hadoop.apache.org
Subject: Re: Patch testing
Message-ID: <20101117072437.GF6649@tp>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com>
 <20101020195420.GG2075@tp>
 <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com>
 <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
 <7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature"; boundary="WK3l2KTTmXPVedZ6"
Content-Disposition: inline
In-Reply-To: <7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com>
X-Organization: It's something of 'Cos
X-PGP-Key: http://www.boudnik.org/~cos/pubkey.asc
User-Agent: Mutt/1.5.20 (2009-06-14)

--WK3l2KTTmXPVedZ6
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Tue, Nov 16, 2010 at 10:46PM, Nigel Daley wrote:
> Just to follow up, the pre-commit patch testing is functioning again on
> Zookeeper and Hadoop Common.  It's also ready to run on MapReduce and HDFS
> but we won't turn it on until these projects build and test cleanly.  Loo=
ks
> like both these projects currently have test failures.  Are there Jira's =
to
> fix these tests?

In HDFS some of the tests are covered by JIRAs for sure. I haven't checked =
MR
lately but it should be the same there.

> New instructions for re-trigger testing of a patch are at https://hudson.=
apache.org/hudson/job/PreCommit-Admin/
>=20
> Cheers,
> Nige
>=20
> On Oct 20, 2010, at 1:30 PM, Nigel Daley wrote:
>=20
> > Thanks Giri.
> >=20
> >> Longterm solution:=20
> >> email patch process and queue creation is un-reliable. As a long term =
solution we need to use jira-cli command line tool to read patch directly f=
rom jira.=20
> >=20
> > Ya, that's what I've got mostly working on my machine.  Any open Jira's=
 for this work?  If not, I'll open one.
> >=20
> >> And doing this would require introducing a new workflow in jira. Apart=
 from the current status called "patch-available"
> >=20
> > Not sure that's necessary.  I think if we change the patch testing mode=
l a little, we can keep the current workflow.  Also, given some recent adva=
nces in Hudson, I think we can reduce all the patch jobs down to 1 admin jo=
b covering *all* projects and 1 job per project -- and we should be able to=
 drastically improve utilization of the slaves we have.  Let's move this di=
scussion to a ticket.  LMK if I need to open one.
> >=20
> > Cheers,
> > Nige
> >=20
> > On Oct 20, 2010, at 1:17 PM, Giridharan Kesavan wrote:
> >=20
> >>=20
> >> Old times:
> >> The hudson patch-testing admin job used to run on the master machine h=
udson.zones.apache.org as this machine used to parse the emails from jira a=
nd create the patch queue for testing.
> >>=20
> >> Current situation:=20
> >> hudson.zones.apache.org machine is no more a master and we have a new =
machine called aegis.apache.org as the master machine.=20
> >>=20
> >> This machine is configured to process the email and create the patch q=
ueue.  <done>
> >> Though we are able to get the email and create the patch queue on the =
new aegis.apache.org machine, still we wont be able to trigger patch admin =
job on the hudson master as this is not allowed in the new setup. =20
> >>=20
> >> Infra team doesn't want us to run any builds on the hudson master. Ins=
tead they want all the builds to run on the slave machine. =20
> >>=20
> >> I got the password-less access for hudson user from hudson slave h1.gr=
id.sp2.yahoo.net to aegis.apache.org like 2 weeks back. <done>  =20
> >>=20
> >> The plan here is to read the patch queue on aegis from h1 and schedule=
 builds.   <I'm working on getting this to work>
> >>=20
> >> Longterm solution:=20
> >> email patch process and queue creation is un-reliable. As a long term =
solution we need to use jira-cli command line tool to read patch directly f=
rom jira.=20
> >> And doing this would require introducing a new workflow in jira. Apart=
 from the current status called "patch-available"
> >>=20
> >> Thanks,
> >> Giri
> >>=20
> >> On Oct 20, 2010, at 12:54 PM, Konstantin Boudnik wrote:
> >>=20
> >>> Thanks for getting to it, Nigel. This is absolutely useful and allows=
 to keep
> >>> weeds out of the trunk up to a certain degree.
> >>>=20
> >>> There were at least two slightly different sources of information abo=
ut what's
> >>> going on with test-patch process. Would you mind to post a brief abou=
t the
> >>> situation so comments can be more substantial?
> >>>=20
> >>> Thanks,
> >>> Cos
> >>>=20
> >>> On Wed, Oct 20, 2010 at 12:47PM, Nigel Daley wrote:
> >>>> Folks,=20
> >>>>=20
> >>>> I'm working to get the pre-commit patch testing running again for HD=
FS,
> >>>> HADOOP, and MAPREDUCE patches.  Since I've been on a bit of a hiatus=
 from
> >>>> day-to-day involvement w/ the project over the last 6 months (new jo=
b), I
> >>>> want to check in and make sure folks would still find this pre-commit
> >>>> testing useful.  Also, happy to hear any suggested improvement.  Let=
 me
> >>>> know. =20
> >>>>=20
> >>>> Cheers,
> >>>> Nige
> >>=20
> >=20
>=20

--WK3l2KTTmXPVedZ6
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iF4EAREIAAYFAkzjgzUACgkQenyFlstYjhJhuQEAwwpXfmmrtFTIZ5YjBdiCO3iU
rK25hNjJScI8Y45uXcMA/A7a+nkshsvf7OJ/XRhpAKh5wvwdzYzDK6UNlgmVBwoA
=8PGM
-----END PGP SIGNATURE-----

--WK3l2KTTmXPVedZ6--

From general-return-2397-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 17 13:54:57 2010
Return-Path: <general-return-2397-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 20486 invoked from network); 17 Nov 2010 13:54:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Nov 2010 13:54:56 -0000
Received: (qmail 80759 invoked by uid 500); 17 Nov 2010 13:55:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80410 invoked by uid 500); 17 Nov 2010 13:55:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 17437 invoked by uid 99); 16 Nov 2010 23:58:27 -0000
X-ASF-Spam-Status: No, hits=-8.1 required=10.0
	tests=ENV_AND_HDR_SPF_MATCH,HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS,USER_IN_DEF_SPF_WL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of adamgray@amazon.com designates 72.21.196.25 as permitted sender)
X-IronPort-AV: E=Sophos;i="4.59,208,1288569600"; 
   d="scan'208,217";a="662725872"
From: "Gray, Adam" <adamgray@amazon.com>
To: "nosql-discussion@googlegroups.com" <nosql-discussion@googlegroups.com>,
        "general@hadoop.apache.org" <general@hadoop.apache.org>,
        "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>,
        "mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>,
        "hdfs-user@hadoop.apache.org" <hdfs-user@hadoop.apache.org>
Date: Tue, 16 Nov 2010 15:57:57 -0800
Subject: Announcing HPC Instances on Amazon Elastic MapReduce
Thread-Topic: Announcing HPC Instances on Amazon Elastic MapReduce
Thread-Index: AcuETZfUv5NRD+ycS8GZuMD9RncnagBnB/fw
Message-ID: <2ACA5D24A0D78040A739A36B5EE9F4B3091ADBB9@EX-SEA19-A.ant.amazon.com>
References: <11956009.3016591289774702373.JavaMail.em-build@massmail-sender-na-5001.iad5.amazon.com>
In-Reply-To: <11956009.3016591289774702373.JavaMail.em-build@massmail-sender-na-5001.iad5.amazon.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_2ACA5D24A0D78040A739A36B5EE9F4B3091ADBB9EXSEA19Aantamaz_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_2ACA5D24A0D78040A739A36B5EE9F4B3091ADBB9EXSEA19Aantamaz_
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64

V2UgYXJlIGV4Y2l0ZWQgdG8gYW5ub3VuY2UgdGhhdCBBbWF6b24gRWxhc3RpYyBNYXBSZWR1Y2U8
aHR0cDovL3d3dy5hbWF6b24uY29tL2dwL3IuaHRtbD9SPTNRODlTOVdQWVFLRTEmQz0zNFM0WEtU
QVhHTUpWJkg9TlhLWTJVOFBMTUFVR004UUFBQTlXTUoxRTdXQSZUPVRDJlU9aHR0cCUzQSUyRiUy
RmF3cy5hbWF6b24uY29tJTJGZWxhc3RpY21hcHJlZHVjZSUyRiUzRnJlZl8lM0RwZV8yMTcwXzE3
NTg3NzEwPiBjYW4gbm93IHRha2UgYWR2YW50YWdlIG9mIENsdXN0ZXIgQ29tcHV0ZSAoY2MxKSBh
bmQgQ2x1c3RlciBHUFUgKGNnMSkgaW5zdGFuY2VzLCBnaXZpbmcgeW91IHRoZSBhYmlsaXR5IHRv
IGNvbWJpbmUgSGFkb29wJ3MgbWFzc2l2ZWx5IHBhcmFsbGVsaXplZCBhcmNoaXRlY3R1cmUgd2l0
aCBoaWdoIHBlcmZvcm1hbmNlIGNvbXB1dGluZy4gWW91IGNhbiBmb2N1cyBvbiBkZXZlbG9waW5n
IEhQQyBhcHBsaWNhdGlvbnMgYW5kIEVsYXN0aWMgTWFwUmVkdWNlIHdpbGwgaGFuZGxlIHdvcmts
b2FkIHBhcmFsbGVsaXphdGlvbiwgbm9kZSBjb25maWd1cmF0aW9uIGFuZCBzY2FsaW5nLCBhbmQg
Y2x1c3RlciBtYW5hZ2VtZW50LiBGdXJ0aGVyLCBFbGFzdGljIE1hcFJlZHVjZSBhcHBsaWNhdGlv
bnMgdGhhdCBhcmUgSS9PIGludGVuc2l2ZSBoYXZlIHRoZSBvcHBvcnR1bml0eSB0byByZWFsaXpl
IHBlcmZvcm1hbmNlIGltcHJvdmVtZW50IGJ5IGxldmVyYWdpbmcgdGhlIGxvdyBsYXRlbmN5LCBm
dWxsIGJpc2VjdGlvbiBiYW5kd2lkdGggMTAgR2JwcyBFdGhlcm5ldCBuZXR3b3JrIGJldHdlZW4g
dGhlIGluc3RhbmNlcyBpbiB0aGUgY2x1c3Rlci4NCg0KQ2x1c3RlciBDb21wdXRlIFF1YWRydXBs
ZSBFeHRyYSBMYXJnZSAoY2MxLjR4bGFyZ2UpIGluc3RhbmNlcyBoYXZlIHRoZSBmb2xsb3dpbmcg
Y29uZmlndXJhdGlvbjoNCg0KICogICBBIHBhaXIgb2YgcXVhZC1jb3JlIEludGVsICJOZWhhbGVt
IjxodHRwOi8vd3d3LmFtYXpvbi5jb20vZ3Avci5odG1sP1I9M1E4OVM5V1BZUUtFMSZDPTM0UzRY
S1RBWEdNSlYmSD1CVVYwRkFBQ1FORTJPT0g2RzNTRDFXMEhFV0lBJlQ9VEMmVT1odHRwJTNBJTJG
JTJGZW4ud2lraXBlZGlhLm9yZyUyRndpa2klMkZOZWhhbGVtXyUyNTI4bWljcm9hcmNoaXRlY3R1
cmUlMjUyOT4gWDU1NzA8aHR0cDovL3d3dy5hbWF6b24uY29tL2dwL3IuaHRtbD9SPTNRODlTOVdQ
WVFLRTEmQz0zNFM0WEtUQVhHTUpWJkg9VFdFVFdaTkhWRFhXTlNQSkJQS05aSkw0Q1lTQSZUPVRD
JlU9aHR0cCUzQSUyRiUyRmFyay5pbnRlbC5jb20lMkZQcm9kdWN0LmFzcHglM0ZpZCUzRDM3MTEx
PiBwcm9jZXNzb3JzIHdpdGggYSB0b3RhbCBvZiAzMy41IEVDVSAoRUMyIENvbXB1dGUgVW5pdHMp
DQogKiAgIDIzIEdCIG9mIFJBTQ0KICogICAxNjkwIEdCIG9mIGxvY2FsIGluc3RhbmNlIHN0b3Jh
Z2UNCiAqICAgMTAgR2JwcyBuZXR3b3JraW5nIHdpdGggbmV0d29yayBsb2NhbGl0eSBiZXR3ZWVu
IG5vZGVzDQoNCkNsdXN0ZXIgR1BVIFF1YWRydXBsZSBFeHRyYSBMYXJnZSAoY2cxLjR4bGFyZ2Up
IGluc3RhbmNlcyBhZGQgYSBwYWlyIG9mIE5WSURJQSBUZXNsYcKuIE0yMDUwIEdQVXM8aHR0cDov
L3d3dy5hbWF6b24uY29tL2dwL3IuaHRtbD9SPTNRODlTOVdQWVFLRTEmQz0zNFM0WEtUQVhHTUpW
Jkg9MUlYWjRWWFdLT1cyTUlVWlZFRUZHWFdHR1RHQSZUPVRDJlU9aHR0cCUzQSUyRiUyRnd3dy5u
dmlkaWEuY29tJTJGb2JqZWN0JTJGcHJvZHVjdF90ZXNsYV9NMjA1MF9NMjA3MF91cy5odG1sPiBh
bmQgd2hlbiBsYXVuY2hlZCB3aXRoIEVsYXN0aWMgTWFwUmVkdWNlIGhhdmUgdGhlIE5WSURJQSBH
UFUgZHJpdmVyIGFuZCBDVURBPGh0dHA6Ly93d3cuYW1hem9uLmNvbS9ncC9yLmh0bWw/Uj0zUTg5
UzlXUFlRS0UxJkM9MzRTNFhLVEFYR01KViZIPUpFTEdSVFNUR0NWQU9HVEhaRVBXVkpPWU1TVUEm
VD1UQyZVPWh0dHAlM0ElMkYlMkZ3d3cubnZpZGlhLmNvbSUyRm9iamVjdCUyRndoYXRfaXNfY3Vk
YV9uZXcuaHRtbD4gcnVudGltZSBpbnN0YWxsZWQuDQoNClBsZWFzZSByZWZlciB0byB0aGUgSGln
aCBQZXJmb3JtYW5jZSBDb21wdXRpbmc8aHR0cDovL3d3dy5hbWF6b24uY29tL2dwL3IuaHRtbD9S
PTNRODlTOVdQWVFLRTEmQz0zNFM0WEtUQVhHTUpWJkg9M1E4SU85QU1XMkdUWThCVFpCUFRBWEhW
QVhTQSZUPVRDJlU9aHR0cCUzQSUyRiUyRmF3cy5hbWF6b24uY29tJTJGaHBjLWFwcGxpY2F0aW9u
cyUyRiUzRnJlZl8lM0RwZV8yMTcwXzE3NTg3NzEwPiBBV1MgU29sdXRpb25zIHBhZ2UgZm9yIG1v
cmUgZGV0YWlscyBvbiBob3cgSFBDIGluc3RhbmNlcyBjYW4gYmUgbGV2ZXJhZ2VkLg0KDQpTaW5j
ZXJlbHksDQoNClRoZSBBbWF6b24gRWxhc3RpYyBNYXBSZWR1Y2UgVGVhbQ0KDQpBbWF6b24gV2Vi
IFNlcnZpY2VzIExMQyBpcyBhIHN1YnNpZGlhcnkgb2YgQW1hem9uLmNvbSwgSW5jLiBBbWF6b24u
Y29tIGlzIGEgcmVnaXN0ZXJlZCB0cmFkZW1hcmsgb2YgQW1hem9uLmNvbSwgSW5jLiBUaGlzIG1l
c3NhZ2UgcHJvZHVjZWQgYW5kIGRpc3RyaWJ1dGVkIGJ5IEFtYXpvbiBXZWIgU2VydmljZXMsIExM
QywgMTIwMCAxMnRoIEF2ZSBTb3V0aCwgU2VhdHRsZSwgV0EgOTgxNDQuDQpbaHR0cDovL3d3dy5h
bWF6b24uY29tL2dwL3IuaHRtbD9SPTNRODlTOVdQWVFLRTEmQz0zNFM0WEtUQVhHTUpWJkg9R1BW
QkREUUJZWkFCWlhYV1MyTEJNQkhBMkZTQSZUPVRFJlU9aHR0cCUzQSUyRiUyRmltYWdlcy5hbWF6
b24uY29tJTJGaW1hZ2VzJTJGRyUyRjAxJTJGbmF2JTJGdHJhbnNwLmdpZl0NCg==

--_000_2ACA5D24A0D78040A739A36B5EE9F4B3091ADBB9EXSEA19Aantamaz_--

From general-return-2398-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 17 19:51:22 2010
Return-Path: <general-return-2398-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21437 invoked from network); 17 Nov 2010 19:51:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Nov 2010 19:51:22 -0000
Received: (qmail 58276 invoked by uid 500); 17 Nov 2010 19:51:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58171 invoked by uid 500); 17 Nov 2010 19:51:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58163 invoked by uid 99); 17 Nov 2010 19:51:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Nov 2010 19:51:51 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Nov 2010 19:51:44 +0000
Received: by qyk31 with SMTP id 31so389554qyk.14
        for <general@hadoop.apache.org>; Wed, 17 Nov 2010 11:51:23 -0800 (PST)
MIME-Version: 1.0
Received: by 10.229.98.135 with SMTP id q7mr4617453qcn.286.1290023482816; Wed,
 17 Nov 2010 11:51:22 -0800 (PST)
Received: by 10.229.225.137 with HTTP; Wed, 17 Nov 2010 11:51:22 -0800 (PST)
In-Reply-To: <7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com>
	<20101020195420.GG2075@tp>
	<53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com>
	<4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
	<7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com>
Date: Wed, 17 Nov 2010 11:51:22 -0800
Message-ID: <AANLkTimLXA+CvUs-09LMN3ob6GZqmhxiyrKO7pzWEfse@mail.gmail.com>
Subject: Re: Patch testing
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Here are the jiras for hdfs test failures on trunk:

HDFS-1306 TestFileAppend4 fails
HDFS-1503 TestSaveNamespace fails when FI is enabled
HDFS-1206 TestFiHFlush fails intermittently
HDFS-1496 TestStorageRestore is failing after HDFS-903 fix
HDFS-1502 TestBlockRecovery triggers NPE in assert
HDFS-613 TestBalancer and TestBlockTokenWithDFS fail Balancer assert


On Tue, Nov 16, 2010 at 10:46 PM, Nigel Daley <ndaley@mac.com> wrote:
> Just to follow up, the pre-commit patch testing is functioning again on Z=
ookeeper and Hadoop Common. =A0It's also ready to run on MapReduce and HDFS=
 but we won't turn it on until these projects build and test cleanly. =A0Lo=
oks like both these projects currently have test failures. =A0Are there Jir=
a's to fix these tests?
>
> New instructions for re-trigger testing of a patch are at https://hudson.=
apache.org/hudson/job/PreCommit-Admin/
>
> Cheers,
> Nige
>
> On Oct 20, 2010, at 1:30 PM, Nigel Daley wrote:
>
>> Thanks Giri.
>>
>>> Longterm solution:
>>> email patch process and queue creation is un-reliable. As a long term s=
olution we need to use jira-cli command line tool to read patch directly fr=
om jira.
>>
>> Ya, that's what I've got mostly working on my machine. =A0Any open Jira'=
s for this work? =A0If not, I'll open one.
>>
>>> And doing this would require introducing a new workflow in jira. Apart =
from the current status called "patch-available"
>>
>> Not sure that's necessary. =A0I think if we change the patch testing mod=
el a little, we can keep the current workflow. =A0Also, given some recent a=
dvances in Hudson, I think we can reduce all the patch jobs down to 1 admin=
 job covering *all* projects and 1 job per project -- and we should be able=
 to drastically improve utilization of the slaves we have. =A0Let's move th=
is discussion to a ticket. =A0LMK if I need to open one.
>>
>> Cheers,
>> Nige
>>
>> On Oct 20, 2010, at 1:17 PM, Giridharan Kesavan wrote:
>>
>>>
>>> Old times:
>>> The hudson patch-testing admin job used to run on the master machine hu=
dson.zones.apache.org as this machine used to parse the emails from jira an=
d create the patch queue for testing.
>>>
>>> Current situation:
>>> hudson.zones.apache.org machine is no more a master and we have a new m=
achine called aegis.apache.org as the master machine.
>>>
>>> This machine is configured to process the email and create the patch qu=
eue. =A0<done>
>>> Though we are able to get the email and create the patch queue on the n=
ew aegis.apache.org machine, still we wont be able to trigger patch admin j=
ob on the hudson master as this is not allowed in the new setup.
>>>
>>> Infra team doesn't want us to run any builds on the hudson master. Inst=
ead they want all the builds to run on the slave machine.
>>>
>>> I got the password-less access for hudson user from hudson slave h1.gri=
d.sp2.yahoo.net to aegis.apache.org like 2 weeks back. <done>
>>>
>>> The plan here is to read the patch queue on aegis from h1 and schedule =
builds. =A0 <I'm working on getting this to work>
>>>
>>> Longterm solution:
>>> email patch process and queue creation is un-reliable. As a long term s=
olution we need to use jira-cli command line tool to read patch directly fr=
om jira.
>>> And doing this would require introducing a new workflow in jira. Apart =
from the current status called "patch-available"
>>>
>>> Thanks,
>>> Giri
>>>
>>> On Oct 20, 2010, at 12:54 PM, Konstantin Boudnik wrote:
>>>
>>>> Thanks for getting to it, Nigel. This is absolutely useful and allows =
to keep
>>>> weeds out of the trunk up to a certain degree.
>>>>
>>>> There were at least two slightly different sources of information abou=
t what's
>>>> going on with test-patch process. Would you mind to post a brief about=
 the
>>>> situation so comments can be more substantial?
>>>>
>>>> Thanks,
>>>> Cos
>>>>
>>>> On Wed, Oct 20, 2010 at 12:47PM, Nigel Daley wrote:
>>>>> Folks,
>>>>>
>>>>> I'm working to get the pre-commit patch testing running again for HDF=
S,
>>>>> HADOOP, and MAPREDUCE patches. =A0Since I've been on a bit of a hiatu=
s from
>>>>> day-to-day involvement w/ the project over the last 6 months (new job=
), I
>>>>> want to check in and make sure folks would still find this pre-commit
>>>>> testing useful. =A0Also, happy to hear any suggested improvement. =A0=
Let me
>>>>> know.
>>>>>
>>>>> Cheers,
>>>>> Nige
>>>
>>
>
>

From general-return-2399-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 17 23:11:42 2010
Return-Path: <general-return-2399-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55185 invoked from network); 17 Nov 2010 23:11:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Nov 2010 23:11:41 -0000
Received: (qmail 25269 invoked by uid 500); 17 Nov 2010 23:12:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25202 invoked by uid 500); 17 Nov 2010 23:12:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25194 invoked by uid 99); 17 Nov 2010 23:12:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Nov 2010 23:12:09 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Nov 2010 23:12:02 +0000
Received: from [172.21.148.196] (wlanvpn-snve-246-196.corp.yahoo.com [172.21.148.196])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id oAHNBLZ4015724
	for <general@hadoop.apache.org>; Wed, 17 Nov 2010 15:11:22 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1290035482;
	bh=3gnglGsl67vB9QcWNNOPfm7F3h/WKsakIToD2yFzz/4=;
	h=Message-ID:Date:From:MIME-Version:To:Subject:References:
	 In-Reply-To:Content-Type:Content-Transfer-Encoding;
	b=cP6wIIZ3pAoh72e/RyhjUQU3l1u1lZ/pkzQ+S3ZZ6PFRKB+7VRwlcTb96bEKVIook
	 Lb/Cz9l4EZLfxn/StrSaoUCNtqEypo0W9XnKpLhHyNzGK0Ju6LZ6lWjd/ejlf1gAFv
	 Qf6f2FDmWJtoARsA2oFe3wFL3Un2oHnFMcaXIZzE=
Message-ID: <4CE46119.2030509@yahoo-inc.com>
Date: Wed, 17 Nov 2010 15:11:21 -0800
From: Jakob Homan <jhoman@yahoo-inc.com>
User-Agent: Thunderbird 2.0.0.12 (Macintosh/20080213)
MIME-Version: 1.0
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: Patch testing
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com>	<20101020195420.GG2075@tp>	<53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com>	<4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>	<7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com> <AANLkTimLXA+CvUs-09LMN3ob6GZqmhxiyrKO7pzWEfse@mail.gmail.com>
In-Reply-To: <AANLkTimLXA+CvUs-09LMN3ob6GZqmhxiyrKO7pzWEfse@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

 > It's also ready to run on MapReduce and HDFS but we won't turn it on 
until these projects build and test cleanly.  Looks like both these 
projects currently have test failures.

Assuming the projects are compiling and building, is there a reason to 
not turn it on despite the test failures? Hudson is invaluable to 
developers who then don't have to run the tests and test-patch 
themselves.  We didn't turn Hudson off when it was working previously 
and there were known failures.  I think one of the reasons we have more 
failing tests now is the higher cost of doing Hudson's work (not a great 
excuse I know).  This is particularly true now because several of the 
failing tests involve tests timing out, making the whole testing regime 
even longer.

-Jakob

From general-return-2400-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 18 00:31:56 2010
Return-Path: <general-return-2400-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1583 invoked from network); 18 Nov 2010 00:31:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Nov 2010 00:31:55 -0000
Received: (qmail 20804 invoked by uid 500); 18 Nov 2010 00:32:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20656 invoked by uid 500); 18 Nov 2010 00:32:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 20648 invoked by uid 99); 18 Nov 2010 00:32:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Nov 2010 00:32:25 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.98 as permitted sender)
Received: from [17.148.16.98] (HELO asmtpout023.mac.com) (17.148.16.98)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Nov 2010 00:32:18 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; charset=us-ascii
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp023.mac.com (Sun Java(tm) System Messaging Server 6.3-8.01 (built Dec
 16 2008; 32bit)) with ESMTPSA id <0LC200DH91GRA680@asmtp023.mac.com> for
 general@hadoop.apache.org; Wed, 17 Nov 2010 16:31:39 -0800 (PST)
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1011170242
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2010-11-17_08:2010-11-17,2010-11-17,1970-01-01 signatures=0
Subject: Re: Patch testing
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <4CE46119.2030509@yahoo-inc.com>
Date: Wed, 17 Nov 2010 16:31:38 -0800
Message-id: <8617BECB-78B4-42B6-B592-D7FC1F8DA923@mac.com>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com>
 <20101020195420.GG2075@tp>
 <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com>
 <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
 <7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com>
 <AANLkTimLXA+CvUs-09LMN3ob6GZqmhxiyrKO7pzWEfse@mail.gmail.com>
 <4CE46119.2030509@yahoo-inc.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1081)


On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:

> > It's also ready to run on MapReduce and HDFS but we won't turn it on until these projects build and test cleanly.  Looks like both these projects currently have test failures.
> 
> Assuming the projects are compiling and building, is there a reason to not turn it on despite the test failures? Hudson is invaluable to developers who then don't have to run the tests and test-patch themselves.  We didn't turn Hudson off when it was working previously and there were known failures.  I think one of the reasons we have more failing tests now is the higher cost of doing Hudson's work (not a great excuse I know).  This is particularly true now because several of the failing tests involve tests timing out, making the whole testing regime even longer.

Every single patch would get a -1 and need investigation.  Currently, that would be about 83 investigations between MR and HDFS issues that are in patch available state.  Shouldn't we focus on getting these tests fixed or removed/?  Also, I need to get MAPREDUCE-2172 fixed (applies to HDFS as well) before I turn this on.

Cheers,
Nige

From general-return-2401-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 18 01:08:41 2010
Return-Path: <general-return-2401-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 18793 invoked from network); 18 Nov 2010 01:08:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Nov 2010 01:08:41 -0000
Received: (qmail 48922 invoked by uid 500); 18 Nov 2010 01:09:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48841 invoked by uid 500); 18 Nov 2010 01:09:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48833 invoked by uid 99); 18 Nov 2010 01:09:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Nov 2010 01:09:06 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Nov 2010 01:09:01 +0000
Received: from [172.21.148.196] (wlanvpn-snve-246-196.corp.yahoo.com [172.21.148.196])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id oAI18OxR057830
	for <general@hadoop.apache.org>; Wed, 17 Nov 2010 17:08:25 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1290042505;
	bh=WD/zQX4BCybVIV8W5JOzst0eHQ2J5Puhmn/0utuVNrg=;
	h=Message-ID:Date:From:MIME-Version:To:Subject:References:
	 In-Reply-To:Content-Type:Content-Transfer-Encoding;
	b=Y+zpk9lmMe9bL45O4ojxSjk+tyuxqp4SkQ7XFYXVu41QR6yFc+uoEqBcu6iGG6063
	 eU4pSEm1J/w5IlxpvnHP/jeb2+D3/ayj5xVpmcZUigZzUHzK0aFdSq5y8qP8Hf9THQ
	 hFj2CzMRQoxCxD8zwRlYzl6cYBRwJeWEgrrDclp4=
Message-ID: <4CE47C88.5050203@yahoo-inc.com>
Date: Wed, 17 Nov 2010 17:08:24 -0800
From: Jakob Homan <jhoman@yahoo-inc.com>
User-Agent: Thunderbird 2.0.0.12 (Macintosh/20080213)
MIME-Version: 1.0
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: Patch testing
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com> <20101020195420.GG2075@tp> <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com> <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com> <7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com> <AANLkTimLXA+CvUs-09LMN3ob6GZqmhxiyrKO7pzWEfse@mail.gmail.com> <4CE46119.2030509@yahoo-inc.com> <8617BECB-78B4-42B6-B592-D7FC1F8DA923@mac.com>
In-Reply-To: <8617BECB-78B4-42B6-B592-D7FC1F8DA923@mac.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

True, each patch would get a -1 and the failing tests would need to be 
verified as those known bad (BTW, it would be great if Hudson could list 
which tests failed in the message it posts to JIRA).  But that's still 
quite a bit less error-prone work than if the developer runs the tests 
and test-patch themselves.  Also, with 22 being cut, there are a lot of 
patches up in the air and several developers are juggling multiple 
patches.  The more automation we can have, even if it's not perfect, 
will decrease errors we may make.
-jg

Nigel Daley wrote:
> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
> 
>>> It's also ready to run on MapReduce and HDFS but we won't turn it on until these projects build and test cleanly.  Looks like both these projects currently have test failures.
>> Assuming the projects are compiling and building, is there a reason to not turn it on despite the test failures? Hudson is invaluable to developers who then don't have to run the tests and test-patch themselves.  We didn't turn Hudson off when it was working previously and there were known failures.  I think one of the reasons we have more failing tests now is the higher cost of doing Hudson's work (not a great excuse I know).  This is particularly true now because several of the failing tests involve tests timing out, making the whole testing regime even longer.
> 
> Every single patch would get a -1 and need investigation.  Currently, that would be about 83 investigations between MR and HDFS issues that are in patch available state.  Shouldn't we focus on getting these tests fixed or removed/?  Also, I need to get MAPREDUCE-2172 fixed (applies to HDFS as well) before I turn this on.
> 
> Cheers,
> Nige


From general-return-2402-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 18 14:54:52 2010
Return-Path: <general-return-2402-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 79775 invoked from network); 18 Nov 2010 14:54:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Nov 2010 14:54:51 -0000
Received: (qmail 56492 invoked by uid 500); 18 Nov 2010 14:55:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56355 invoked by uid 500); 18 Nov 2010 14:55:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 15273 invoked by uid 99); 18 Nov 2010 00:22:24 -0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of Sridhar.Yalamanchili@prosum.com designates 207.105.115.217 as permitted sender)
From: Sridhar Yalamanchili <Sridhar.Yalamanchili@prosum.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 17 Nov 2010 16:21:53 -0800
Subject: Hadoop Developers Needed; Los Angeles, CA
Thread-Topic: Hadoop Developers Needed; Los Angeles, CA
Thread-Index: AcuGtpnVcmzBGyXzRLeM2lJNO5Heaw==
Message-ID: <E7ABF95661A337488ECCB8A1CF717278A3A5EF67E6@Pomegranate.prosum.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_E7ABF95661A337488ECCB8A1CF717278A3A5EF67E6Pomegranatepr_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_E7ABF95661A337488ECCB8A1CF717278A3A5EF67E6Pomegranatepr_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

To All Hadoop Developers:

I'm a Managing Partner in a Technology Staffing and Consulting firm in Los =
Angeles, CA. We've been engaged by a well funded start-up in Los Angeles, C=
A to find Hadoop Developers for permanent roles within this company. If you=
 are interested, I would love to have a conversation with you about the rol=
es. If you'd like to forward a resume to be reviewed, please feel free to d=
o so.

Thanks! I look forward to hearing from you.

Best regards,

Sridhar Yalamanchili
Managing Partner
Prosum Staffing

2321 Rosecrans Avenue, Suite 4225
El Segundo, CA 90245

Ph.: (310) 426-0688
FAX: (310) 426-0690

Email: sridhar.yalamanchili@prosum.com

Web: www.prosum.com


--_000_E7ABF95661A337488ECCB8A1CF717278A3A5EF67E6Pomegranatepr_--

From general-return-2403-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 18 15:59:25 2010
Return-Path: <general-return-2403-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25388 invoked from network); 18 Nov 2010 15:59:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Nov 2010 15:59:25 -0000
Received: (qmail 84634 invoked by uid 500); 18 Nov 2010 15:59:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84427 invoked by uid 500); 18 Nov 2010 15:59:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84419 invoked by uid 99); 18 Nov 2010 15:59:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Nov 2010 15:59:52 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of bernd.fondermann@googlemail.com designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Nov 2010 15:59:46 +0000
Received: by ewy5 with SMTP id 5so2066527ewy.35
        for <general@hadoop.apache.org>; Thu, 18 Nov 2010 07:59:26 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=Nqybb6u2NOY/cEg0W/POPJ2Kpc6UZox6R4b2fOJMe5I=;
        b=BHqOIU4C5xsZeynmAv1dNMGJfQwXOLBl7HUWvyvghFd2AWOdKKupEcGiC+sBeIGeB1
         YbaxnM918zCjGL0lyv6Vu2C17BxB1GbcRG2bWW/einqznvFDuvRrpvxFRrVj5DMrC8aS
         /J6CMrSgK4lTWMTTVrATLoVYeZciTh2hnMHNM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=atvVGMOuOcIl6ywDlbz/IJwCxEQcUFhILoJdFuKkRAkM16WsqCgk9Ck1yjZPNjqBf6
         BAMZpY108EqADHWqHskHlJ8h8X/lp9GpBtTfhHRhTqGMLZSKRJzAGUhwtPtPb8zsq755
         8oRNf4agt9c8l7/FXsvxY1uK0X4XaC8LfGgOw=
MIME-Version: 1.0
Received: by 10.213.35.7 with SMTP id n7mr628951ebd.40.1290095965966; Thu, 18
 Nov 2010 07:59:25 -0800 (PST)
Received: by 10.213.32.139 with HTTP; Thu, 18 Nov 2010 07:59:25 -0800 (PST)
In-Reply-To: <AANLkTim39MjLCX=7swNJ+5i8FZMbiPgJ9TQn+c3mY_0L@mail.gmail.com>
References: <AANLkTimKwDRCPi95y1URNQ2nMm1_1CUk=7HsEAdrbkjU@mail.gmail.com>
	<AANLkTikdAgZ9zCpk+JYJK=zxXGxNn2c60-P_8iw28nPJ@mail.gmail.com>
	<AANLkTim39MjLCX=7swNJ+5i8FZMbiPgJ9TQn+c3mY_0L@mail.gmail.com>
Date: Thu, 18 Nov 2010 16:59:25 +0100
Message-ID: <AANLkTi=ZbQ_QNg7g-1jgUoj82KjUjwcy3d0xzTr9QSjS@mail.gmail.com>
Subject: Re: browsing mailing list
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Fri, Nov 12, 2010 at 03:28, web service <wbsrvc@gmail.com> wrote:
> thanks.
>
> On Thu, Nov 11, 2010 at 7:26 PM, Shrijeet Paliwal
> <shrijeet@rocketfuel.com>wrote:
>
>> ...because you asked, what do you guys do...
>>
>> I do http://www.search-hadoop.com/

Alternatively:
  http://markmail.org/search/?q=list%3Ahadoop
The "official" and only authoritative site for browsing the list would be
  http://mail-archives.apache.org/mod_mbox/hadoop-general/

  Bernd

From general-return-2404-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 19 00:36:59 2010
Return-Path: <general-return-2404-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16723 invoked from network); 19 Nov 2010 00:36:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Nov 2010 00:36:59 -0000
Received: (qmail 74582 invoked by uid 500); 19 Nov 2010 00:37:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74498 invoked by uid 500); 19 Nov 2010 00:37:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74490 invoked by uid 99); 19 Nov 2010 00:37:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Nov 2010 00:37:28 +0000
X-ASF-Spam-Status: No, hits=2.1 required=10.0
	tests=TO_NO_BRKTS_DIRECT
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 19 Nov 2010 00:37:26 +0000
Received: (qmail 16631 invoked by uid 99); 19 Nov 2010 00:36:33 -0000
Received: from localhost.apache.org (HELO [192.168.168.134]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Nov 2010 00:36:33 +0000
Message-ID: <4CE5C6B0.8050301@apache.org>
Date: Thu, 18 Nov 2010 16:37:04 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Zookeeper is now a TLP
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

At yesterday's board meeting, the resolution passed to create a 
Top-Level project for Zookeper.  Congratulations!

Doug

From general-return-2405-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 19 00:46:45 2010
Return-Path: <general-return-2405-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 20677 invoked from network); 19 Nov 2010 00:46:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Nov 2010 00:46:44 -0000
Received: (qmail 86338 invoked by uid 500); 19 Nov 2010 00:47:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86291 invoked by uid 500); 19 Nov 2010 00:47:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86283 invoked by uid 99); 19 Nov 2010 00:47:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Nov 2010 00:47:15 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Nov 2010 00:47:07 +0000
Received: by wwb39 with SMTP id 39so4049435wwb.29
        for <general@hadoop.apache.org>; Thu, 18 Nov 2010 16:46:47 -0800 (PST)
MIME-Version: 1.0
Received: by 10.216.140.37 with SMTP id d37mr223001wej.31.1290127604994; Thu,
 18 Nov 2010 16:46:44 -0800 (PST)
Received: by 10.216.239.200 with HTTP; Thu, 18 Nov 2010 16:46:44 -0800 (PST)
Date: Thu, 18 Nov 2010 19:46:44 -0500
Message-ID: <AANLkTim-6FLttddjyD-5d91JaeqGjJktVhozgm5prPmk@mail.gmail.com>
Subject: hadoop-6685
From: Ian Holsman <hadoop@holsman.net>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d9a2669c557104955d3df7
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d9a2669c557104955d3df7
Content-Type: text/plain; charset=ISO-8859-1

Hi Guys,

It seems like there is a lot of discussion on this patch by a couple of
people, and I'd be interested to hear what other people think of it


https://issues.apache.org/jira/browse/HADOOP-6685

--0016e6d9a2669c557104955d3df7--

From general-return-2406-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Nov 20 06:45:17 2010
Return-Path: <general-return-2406-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 41786 invoked from network); 20 Nov 2010 06:45:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Nov 2010 06:45:16 -0000
Received: (qmail 70787 invoked by uid 500); 20 Nov 2010 06:45:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70548 invoked by uid 500); 20 Nov 2010 06:45:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70540 invoked by uid 99); 20 Nov 2010 06:45:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 20 Nov 2010 06:45:43 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.30.32] (HELO qmta03.emeryville.ca.mail.comcast.net) (76.96.30.32)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 20 Nov 2010 06:45:34 +0000
Received: from omta23.emeryville.ca.mail.comcast.net ([76.96.30.90])
	by qmta03.emeryville.ca.mail.comcast.net with comcast
	id ZJju1f0021wfjNsA3JlEax; Sat, 20 Nov 2010 06:45:14 +0000
Received: from [10.0.0.13] ([209.131.62.115])
	by omta23.emeryville.ca.mail.comcast.net with comcast
	id ZJl41f0012VBGtd8jJl7p2; Sat, 20 Nov 2010 06:45:12 +0000
Message-Id: <270521A0-A54F-4482-8A36-ABD6FB5ECD96@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: [VOTE] Hadoop Bylaws
Date: Fri, 19 Nov 2010 22:45:04 -0800
X-Mailer: Apple Mail (2.936)

All,
    I've updated the proposed bylaws that I sent out a few weeks ago  
as suggested by making the vote for removal be a lazy 2/3, so now all  
of the votes are lazy. As before, by a recursive application of the  
bylaws let's let the vote last 7 days (11/26) and require a lazy  
majority of PMC members to pass.

-- Owen

Apache Hadoop Project Bylaws

This document defines the bylaws under which the Apache Hadoop project
operates. It defines the roles and responsibilities of the project,
who may vote, how voting works, how conflicts are resolved, etc.

Hadoop is a project of the <a
href="http://www.apache.org/foundation/">Apache Software
Foundation</a>.  The foundation holds the trademark on the name
"Hadoop" and copyright on Apache code
including the code in the Hadoop codebase. The <a
href="http://www.apache.org/foundation/faq.html">foundation FAQ</a>
explains the operation and background of the foundation.

Hadoop is typical of Apache projects in that it operates under a set
of principles, known collectively as the &quot;Apache Way&quot;. If
you are new to Apache development, please refer to the <a
href="http://incubator.apache.org">Incubator project</a> for more
information on how Apache projects operate.

Roles and Responsibilities

Apache projects define a set of roles with associated rights and
responsibilities. These roles govern what tasks an individual may
perform within the project. The roles are defined in the following
sections

    * Users

         The most important participants in the project are people who
         use our software. The majority of our developers start out as
         users and guide their development efforts from the user's
         perspective.

        Users contribute to the Apache projects by providing feedback
         to developers in the form of bug reports and feature
         suggestions. As well, users participate in the Apache
         community by helping other users on mailing lists and user
         support forums.

    * Contributors

         All of the volunteers who are contributing time, code,
         documentation, or resources to the Hadoop Project. A  
contributor
         that makes sustained, welcome contributions to the project may
         be invited to become a Committer, though the exact timing of
         such invitations depends on many factors.

     * Committers

        The project's Committers are responsible for the project's
         technical management. Committers have access to a specified
         set of subprojects' subversion repositories. Committers on
         subprojects may cast binding votes on any technical discussion
         regarding that subproject.

        Committer access is by invitation only and must be approved by
         lazy consensus of the active PMC members. A Committer is
         considered emeritus by their own declaration or by not
         contributing in any form to the project for over six
         months. An emeritus committer may request reinstatement of
         commit access from the PMC. Such reinstatement is subject to
         lazy consensus of active PMC members.

       All Apache committers are required to have a signed Contributor
         License Agreement (CLA) on file with the Apache Software
         Foundation. There is a <a
         href="http://www.apache.org/dev/committers.html">Committer
         FAQ</a> which provides more details on the requirements for
         Committers

        A committer who makes a sustained contribution to the project
         may be invited to become a member of the PMC. The form of
         contribution is not limited to code. It can also include code
         review, helping out users on the mailing lists, documentation,
         testing, etc.

    * Project Management Committee

        The Project Management Committee (PMC) for Apache Hadoop was
         created by the Apache Board in January 2008 when Hadoop moved
         out of Lucene and became a top level project at Apache. The
         PMC is responsible to the board and the ASF for the management
         and oversight of the Apache Hadoop codebase. The
         responsibilities of the PMC include

        * Deciding what is distributed as products of the Apache Hadoop
         project.  In particular all releases must be approved by the
         PMC

         * Maintaining the project's shared resources, including the  
codebase
             repository, mailing lists, websites.

         * Speaking on behalf of the project.

        * Resolving license disputes regarding products of the project

         * Nominating new PMC members and committers

         * Maintaining these bylaws and other guidelines of the project

        Membership of the PMC is by invitation only and must be
         approved by a lazy consensus of active PMC members. A PMC
         member is considered &quot;emeritus&quot; by their own
         declaration or by not contributing in any form to the project
         for over six months. An emeritus member may request
         reinstatement to the PMC. Such reinstatement is subject to
         lazy consensus of the active PMC members.

        The chair of the PMC is appointed by the ASF board. The chair
         is an office holder of the Apache Software Foundation (Vice
         President, Apache Hadoop) and has primary responsibility to
         the board for the management of the projects within the scope
         of the Hadoop PMC. The chair reports to the board quarterly on
         developments within the Hadoop project.

	When the current chair of the PMC resigns, the PMC votes to
	recommend a new chair using lazy consensus, but the decision
	must be ratified by the Apache board.

Decision Making

       Within the Hadoop project, different types of decisions require
       different forms of approval. For example, the <a href="#Roles
       and Responsibilities">previous section</a> describes several
       decisions which require &quot;lazy consensus&quot;
       approval. This section defines how voting is performed, the
       types of approvals, and which types of decision require which
       type of approval.

     * Voting

         Decisions regarding the project are made by votes on the
         primary project development mailing list
         (general@hadoop.apache.org).  Where necessary, PMC voting may
         take place on the private Hadoop PMC mailing list.  Votes are
         clearly indicated by subject line starting with [VOTE].  Votes
         may contain multiple items for approval and these should be
         clearly separated. Voting is carried out by replying to the
         vote mail. Voting may take four flavors

         * +1 &quot;Yes,&quot; &quot;Agree,&quot; or &quot;the action  
should be
             performed.&quot; In general, this vote also indicates a  
willingness
             on the behalf of the voter in &quot;making it happen&quot;

          * +0 This vote indicates a willingness for the action under
             consideration to go ahead. The voter, however will not be  
able
             to help.

          * -0 This vote indicates that the voter does not, in  
general, agree with
             the proposed action but is not concerned enough to  
prevent the
             action going ahead.

           * -1 This is a negative vote. On issues where consensus is  
required,
             this vote counts as a <strong>veto</strong>. All vetoes  
must
             contain an explanation of why the veto is appropriate.  
Vetoes with
             no explanation are void. It may also be appropriate for a  
-1 vote
             to include an alternative course of action.

         All participants in the Hadoop project are encouraged to show  
their
         agreement with or against a particular action by voting. For  
technical
         decisions, only the votes of active committers are binding.  
Non binding
         votes are still useful for those with binding votes to  
understand the
         perception of an action in the wider Hadoop community. For  
PMC decisions,
         only the votes of PMC members are binding.

        Voting can also be applied to changes made to the Hadoop  
codebase. These
         typically take the form of a veto (-1) in reply to the commit  
message
         sent when the commit is made.

     * Approvals

         These are the types of approvals that can be sought.  
Different actions
         require different types of approvals

         * Lazy Consensus - Lazy consensus requires 3 binding +1 votes
            and no binding vetoes.

          * Lazy Majority - A lazy majority vote requires 3 binding +1
             votes and more binding +1 votes than -1 votes.

         * Lazy 2/3 Majority - Lazy 2/3 majority votes requires at
            least 3 votes and twice as many +1 votes as -1 votes.

     * Vetoes

         A valid, binding veto cannot be overruled. If a veto is cast,
         it must be accompanied by a valid reason explaining the
         reasons for the veto. The validity of a veto, if challenged,
         can be confirmed by anyone who has a binding vote. This does
         not necessarily signify agreement with the veto - merely that
         the veto is valid.

        If you disagree with a valid veto, you must lobby the person  
casting
         the veto to withdraw their veto. If a veto is not withdrawn,  
any action
         that has been vetoed must be reversed in a timely manner.

     * Actions

         This section describes the various actions which are  
undertaken within
         the project, the corresponding approval required for that  
action and
         those who have binding votes over the action.

          * Code Change

             A change made to a codebase of the project and committed
             by a committer. This includes source code, documentation,  
website
             content, etc.

             Lazy consensus of active committers.

          * Release Plan

             Defines the timetable and actions for a release. The plan  
also
             nominates a Release Manager.

             Lazy majority of active committers

          * Product Release

             When a release of one of the project's products is ready,  
a vote is
             required to accept the release as an official release of  
the
             project.

            Lazy Majority of active PMC members

           * Adoption of New Codebase

             When the codebase for an existing, released product is to  
be
             replaced with an alternative codebase. If such a vote  
fails to
             gain approval, the existing code base will continue.

            This also covers the creation of new sub-projects
             within the project

            Lazy 2/3 majority of PMC members

           * New Committer

             When a new committer is proposed for the project

             Lazy consensus of active PMC members

           * New PMC Member

             When a committer is proposed for the PMC

            Lazy consensus of active PMC members

           * Committer Removal

            When removal of commit privileges is sought.  Note: Such
            actions will also be referred to the ASF board by the PMC
            chair

             Lazy 2/3 majority of active PMC members (excluding the
             committer in question if a member of the PMC).

           * PMC Member Removal

             When removal of a PMC member is sought.  Note: Such
             actions will also be referred to the ASF board by the PMC
             chair.

             Lazy 2/3 majority of active PMC members (excluding the  
member in question)

           * Modifying Bylaws

             Modifying this document.

             Majority of active PMC members

    * Voting Timeframes

         Votes are open for a period of 7 days to allow all active
         voters time to consider the vote. Votes relating to code
         changes are not subject to a strict timetable but should be
         made as timely as possible.


From general-return-2407-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Nov 20 15:29:19 2010
Return-Path: <general-return-2407-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 32834 invoked from network); 20 Nov 2010 15:29:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Nov 2010 15:29:19 -0000
Received: (qmail 4009 invoked by uid 500); 20 Nov 2010 15:29:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3847 invoked by uid 500); 20 Nov 2010 15:29:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 81510 invoked by uid 99); 20 Nov 2010 14:42:01 -0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awakenrz@gmail.com designates 209.85.160.176 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:from:content-type
         :content-transfer-encoding:subject:date:message-id:to:mime-version
         :x-mailer;
        bh=AOW2HxphHhb204iMb8gaznjGf+4k099wdkdMuksGDWs=;
        b=nHq+ISkDkauNK6Ks1J2gqWDPYAFDX7z19Uw81OjR8WPGclZjasQSmZ9eQMCjxx2E1k
         mQ5CVnjyvNo6YNPcWSGAJnPDNQjQy0wvyNt36PZ0oucU6OHEalun3aWCmz8tZlJZcAWb
         LEfzjr5A1lTp3PTsUyjGLKR6+W/M6/jnmPZJQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=from:content-type:content-transfer-encoding:subject:date:message-id
         :to:mime-version:x-mailer;
        b=li658r9UDKbj4ef0H2egdJrBUZPqxhIMPdR8OUlVPGcEjyXe15ZKqh5Syfgm7ClvD/
         gPIKMmzpBYlV9C6sxS7nihejmHh9FHUueWoN4Ydhh3QvyXIvwSJTVCVTzy8bHAVkgkbG
         kL69CDxpOEv7knA3Eyf9acigItXgc5JwqGDzI=
From: Ren Zuocheng <awakenrz@gmail.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: The documentation for startup is out of date.
Date: Sat, 20 Nov 2010 22:41:21 +0800
Message-Id: <7866CBAB-0B74-4250-A890-D634BB0D1D14@gmail.com>
To: general@hadoop.apache.org
Mime-Version: 1.0 (Apple Message framework v1081)
X-Mailer: Apple Mail (2.1081)
X-Virus-Checked: Checked by ClamAV on apache.org

Dear all,
I'm new to hadoop and following the documentation for startup users to =
set up my first WordCounter in hadoop. However, I find out that there =
are some flaws in the documentation
An example is the hadoop-core-${HADOOP_VERSION}.jar has been renamed to =
hadoop-common-${HADOOP_VERSION}.jar. There are much more minor errors =
like this, which may cause a novice to feel confused or misled.
And I really want to write an update for that. Can I? If I can, how can =
I do that?=

From general-return-2408-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Nov 20 18:43:50 2010
Return-Path: <general-return-2408-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17274 invoked from network); 20 Nov 2010 18:43:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Nov 2010 18:43:50 -0000
Received: (qmail 53877 invoked by uid 500); 20 Nov 2010 18:44:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53821 invoked by uid 500); 20 Nov 2010 18:44:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53813 invoked by uid 99); 20 Nov 2010 18:44:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 20 Nov 2010 18:44:19 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 20 Nov 2010 18:44:13 +0000
Received: by qwh5 with SMTP id 5so409339qwh.35
        for <general@hadoop.apache.org>; Sat, 20 Nov 2010 10:43:52 -0800 (PST)
MIME-Version: 1.0
Received: by 10.229.233.74 with SMTP id jx10mr3211753qcb.96.1290278632128;
 Sat, 20 Nov 2010 10:43:52 -0800 (PST)
Received: by 10.229.225.137 with HTTP; Sat, 20 Nov 2010 10:43:52 -0800 (PST)
In-Reply-To: <7866CBAB-0B74-4250-A890-D634BB0D1D14@gmail.com>
References: <7866CBAB-0B74-4250-A890-D634BB0D1D14@gmail.com>
Date: Sat, 20 Nov 2010 10:43:52 -0800
Message-ID: <AANLkTinUmMt0xCHFMeoTboyBLgF3x5NOhU+ho=bLfHdg@mail.gmail.com>
Subject: Re: The documentation for startup is out of date.
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hey Ren,

Assume you're referring to pages like
http://hadoop.apache.org/mapreduce/docs/current/mapred_tutorial.html

The current release is 21, so to update those docs checkout the
branch-0.21 branch from the common, mapreduce, and hdfs repository.
The docs live in src/docs (eg this page is
src/docs/src/documentation/content/xdocs/mapred_tutorial.xml in the
mapreduce repo).

First check if the change has already been made on trunk, if not make
it there and we can backport it to the branch-0.21 branches.  Please
file a jira (one for each project) for the docs bugs you find. No need
to put every individual bug in it's own jira.

Thanks for contributing!

Eli

On Sat, Nov 20, 2010 at 6:41 AM, Ren Zuocheng <awakenrz@gmail.com> wrote:
> Dear all,
> I'm new to hadoop and following the documentation for startup users to set up my first WordCounter in hadoop. However, I find out that there are some flaws in the documentation
> An example is the hadoop-core-${HADOOP_VERSION}.jar has been renamed to hadoop-common-${HADOOP_VERSION}.jar. There are much more minor errors like this, which may cause a novice to feel confused or misled.
> And I really want to write an update for that. Can I? If I can, how can I do that?

From general-return-2409-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Nov 21 02:47:01 2010
Return-Path: <general-return-2409-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 90017 invoked from network); 21 Nov 2010 02:47:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Nov 2010 02:47:00 -0000
Received: (qmail 82527 invoked by uid 500); 21 Nov 2010 02:47:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82404 invoked by uid 500); 21 Nov 2010 02:47:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82396 invoked by uid 99); 21 Nov 2010 02:47:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 21 Nov 2010 02:47:30 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awakenrz@gmail.com designates 209.85.161.194 as permitted sender)
Received: from [209.85.161.194] (HELO mail-gx0-f194.google.com) (209.85.161.194)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 21 Nov 2010 02:47:21 +0000
Received: by gxk1 with SMTP id 1so1919895gxk.5
        for <general@hadoop.apache.org>; Sat, 20 Nov 2010 18:47:00 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:subject:mime-version
         :content-type:from:resent-from:date:content-transfer-encoding
         :resent-date:resent-to:message-id:to:x-mailer;
        bh=CuVCM5VTMfsWinnxL+NvmJd9BYsl3RM7rnfbG4WhExc=;
        b=km0Zk8nr/8ilMmkxGmCo71dayhR5QQ04ypWEaTtIWgMtJvbNb0LKT4i87ngaMeuRhV
         J7KqYqb/fNP5tEcAB2U82ASg73iiivMyL8Mb27T7FbyVkjpa783HSFpLfbwDTALoXY8b
         QQDhj4ylx6a88W9iujpz0XzhCeaKQ23ipmgDw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=subject:mime-version:content-type:from:resent-from:date
         :content-transfer-encoding:resent-date:resent-to:message-id:to
         :x-mailer;
        b=lyZoeGJ02xLK2RiEw5RNP2WYb0n9Tg2tMCAV9awukJMvJKn3e6wOY1/S8qmO+oSJb+
         5NscHom8W40469qT5hFHblSTnZsDVm0V6S01Ghc1cZo5zemWHfYPQ3Bjdw1CqrjIpElV
         qw6uZZW0KYYGfXmSTGGA+s5ggmArotlVUE2fQ=
Received: by 10.150.181.20 with SMTP id d20mr6453742ybf.126.1290307620431;
        Sat, 20 Nov 2010 18:47:00 -0800 (PST)
Received: from [192.168.1.134] ([219.224.168.157])
        by mx.google.com with ESMTPS id q5sm983405ybe.6.2010.11.20.18.46.58
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Sat, 20 Nov 2010 18:46:59 -0800 (PST)
Subject: The documentation for startup is out of date.
Mime-Version: 1.0 (Apple Message framework v1081)
Content-Type: text/plain; charset=us-ascii
From: Ren Zuocheng <awakenrz@gmail.com>
Resent-From: Ren Zuocheng <awakenrz@gmail.com>
Date: Sat, 20 Nov 2010 22:41:21 +0800
Content-Transfer-Encoding: quoted-printable
Resent-Date: Sun, 21 Nov 2010 10:46:52 +0800
Resent-To: general@hadoop.apache.org
Message-Id: <7866CBAB-0B74-4250-A890-D634BB0D1D14@gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1081)
X-Virus-Checked: Checked by ClamAV on apache.org
Resent-Message-Id: <20101121024728.CDE73816043@nike.apache.org>

Dear all,
(I've sent this email, but I forgot to subscribe to the maillist and did =
not receive any reply... So I resent the message. Sorry about this.)
I'm new to hadoop and following the documentation for startup users to =
set up my first WordCounter in hadoop. However, I find out that there =
are some flaws in the documentation
An example is the hadoop-core-${HADOOP_VERSION}.jar has been renamed to =
hadoop-common-${HADOOP_VERSION}.jar. There are much more minor errors =
like this, which may cause a novice to feel confused or misled.
And I really want to write an update for that. Can I? If I can, how can =
I do that?=

From general-return-2410-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Nov 21 09:21:24 2010
Return-Path: <general-return-2410-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 32883 invoked from network); 21 Nov 2010 09:21:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Nov 2010 09:21:24 -0000
Received: (qmail 77880 invoked by uid 500); 21 Nov 2010 09:21:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77677 invoked by uid 500); 21 Nov 2010 09:21:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77668 invoked by uid 99); 21 Nov 2010 09:21:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 21 Nov 2010 09:21:52 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [81.169.146.162] (HELO mo-p00-ob.rzone.de) (81.169.146.162)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 21 Nov 2010 09:21:46 +0000
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; t=1290331284; l=332;
	s=domk; d=brainlounge.de;
	h=Content-Transfer-Encoding:Content-Type:In-Reply-To:References:
	Subject:To:MIME-Version:From:Date:X-RZG-CLASS-ID:X-RZG-AUTH;
	bh=1Qwz39x1jRJLo60yBcq66tDtGmQ=;
	b=wsDEnJBE6J4R4cNIBNS5eJAJYGsLPeSL1djQ9xvfJ/zOR+QRr34nyRIzKq2BBE3ZrRh
	SxvD1FsKNsviChMfePsNDGHups+1AW0IDunr/B6wPybncclxY9t08BJTUA1BByFRXX+/J
	XGKU7wuJQC1INj0zc29QBJ56J/Kj+mclQkQ=
X-RZG-AUTH: :LmkWe0TmffCene0uRkJY6AqDWBjdcSs8EVOKK4t9JJpSMjf2AKg+TcRtFaNcoY+nN7sOdsXpAGV4qjhwQA==
X-RZG-CLASS-ID: mo00
Received: from vitamin-2.local (e180150126.adsl.alicedsl.de [85.180.150.126])
	by post.strato.de (jimi mo63) (RZmta 24.6) with ESMTP id 20579amAL8LOAs
	for <general@hadoop.apache.org>; Sun, 21 Nov 2010 10:21:23 +0100 (MET)
Message-ID: <4CE8E493.7000307@brainlounge.de>
Date: Sun, 21 Nov 2010 10:21:23 +0100
From: Bernd Fondermann <bf_jak@brainlounge.de>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.15) Gecko/20101027 Thunderbird/3.0.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Cloudera CDH3 install on Ubuntu 10.04 (lucid)
References: <B75E035660BF70488B180EFA991B7368083A31@fccorpmail03.mygazoo.com>
In-Reply-To: <B75E035660BF70488B180EFA991B7368083A31@fccorpmail03.mygazoo.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 12.11.10 20:06, Peter Minearo wrote:
> Not sure who or where to send this to, but this is for the Cloudera distribution of Hadoop:

Unfortunately, as CDH is not supported by Apache Hadoop, you'd have to 
contact the vendor directly. The vendor has a support forum reachable 
from his website if you dig little bit.

   Bernd

From general-return-2411-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Nov 21 14:21:21 2010
Return-Path: <general-return-2411-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 61451 invoked from network); 21 Nov 2010 14:21:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Nov 2010 14:21:21 -0000
Received: (qmail 25333 invoked by uid 500); 21 Nov 2010 14:21:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25146 invoked by uid 500); 21 Nov 2010 14:21:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25138 invoked by uid 99); 21 Nov 2010 14:21:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 21 Nov 2010 14:21:49 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of guosxu@163.com designates 220.181.13.102 as permitted sender)
Received: from [220.181.13.102] (HELO m13-102.163.com) (220.181.13.102)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 21 Nov 2010 14:21:41 +0000
Received: from guosxu ( [116.237.46.177] ) by ajax-webmail-wmsvr102
 (Coremail) ; Sun, 21 Nov 2010 22:21:09 +0800 (CST)
Date: Sun, 21 Nov 2010 22:21:09 +0800 (CST)
From: =?GBK?B?ufnLs9Dx?= <guosxu@163.com>
To: general@hadoop.apache.org
Message-ID: <173adea.55a5.12c6ecf51da.Coremail.guosxu@163.com>
In-Reply-To: <4CE5C6B0.8050301@apache.org>
References: <4CE5C6B0.8050301@apache.org>
Subject: Re:Zookeeper is now a TLP
MIME-Version: 1.0
Content-Type: text/plain; charset=GBK
Content-Transfer-Encoding: quoted-printable
X-Originating-IP: [116.237.46.177]
X-Priority: 3
X-Mailer: Coremail Webmail Server Version SP_ntes V3.5 build
 101020(12106.3506.3507) Copyright (c) 2002-2010 www.mailtech.cn 163com
X-CM-TRANSID:ZsGowLD7T_PVKulMxk0KAA--.12064W
X-CM-SenderInfo: pjxr25rx6rljoofrz/1tbiFBqQd0zBuStGggAAsy
X-Coremail-Antispam: 1U5529EdanIXcx71UUUUU7vcSsGvfC2KfnxnUU==

oh,yee! Congratulations!!!
At 2010-11-19 08:37:04=A3=AC"Doug Cutting" <cutting@apache.org> wrote:
>At yesterday's board meeting, the resolution passed to create a=20
>Top-Level project for Zookeper.  Congratulations!
>
>Doug


From general-return-2412-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Nov 21 17:12:11 2010
Return-Path: <general-return-2412-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34417 invoked from network); 21 Nov 2010 17:12:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Nov 2010 17:12:11 -0000
Received: (qmail 25406 invoked by uid 500); 21 Nov 2010 17:12:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25305 invoked by uid 500); 21 Nov 2010 17:12:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25297 invoked by uid 99); 21 Nov 2010 17:12:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 21 Nov 2010 17:12:39 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of patodirahul@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 21 Nov 2010 17:12:35 +0000
Received: by wwb39 with SMTP id 39so6548181wwb.29
        for <general@hadoop.apache.org>; Sun, 21 Nov 2010 09:12:14 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=+Leq37+dt0K6fO6nPjCQexkZpy0iowZYclhj4gXRUZE=;
        b=aX5ORWt4IQV1/8SSS6mkg7gGvh5Tbe5ThNcy4OICj0Njl35KNVD3NBF+G/sPy6Ur6z
         ccYZMmZW9LLLcOJM9CPSe12esdZLzR1cyTM5fDXF2Qyo21rQq/cbFBids1JWpcSjsASM
         YCyl8mcOgNAci03w5RtyU8mKzRSTtXYOEzbrM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=n5hGpsZCIKUIm1bZL+YSXQlaxuoRiBlEqvEo0A3QxCT0Mecig6MuPvmuQBlUjJAg/b
         rUqfX0CqsKwjgmZTE11LJQFOkJDHBgdgX87qZPRx4lTsY+8TWKZAMuFyHxXnNKUd0PfX
         9l+x7haFytin+Jcl42bYGIHtQlZJ+cfDOmp0E=
MIME-Version: 1.0
Received: by 10.216.186.141 with SMTP id w13mr790040wem.91.1290359533250; Sun,
 21 Nov 2010 09:12:13 -0800 (PST)
Received: by 10.216.131.197 with HTTP; Sun, 21 Nov 2010 09:12:13 -0800 (PST)
In-Reply-To: <4CE8E493.7000307@brainlounge.de>
References: <B75E035660BF70488B180EFA991B7368083A31@fccorpmail03.mygazoo.com>
	<4CE8E493.7000307@brainlounge.de>
Date: Sun, 21 Nov 2010 22:42:13 +0530
Message-ID: <AANLkTi=JTKUa3Om394E5tWd5wvQWscPznsCBdDQxeAvC@mail.gmail.com>
Subject: Re: Cloudera CDH3 install on Ubuntu 10.04 (lucid)
From: rahul patodi <patodirahul@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163649a6bf9cacea0495933da7

--00163649a6bf9cacea0495933da7
Content-Type: text/plain; charset=ISO-8859-1

I have installed cloudera distribution for hadoop CDH3 on ubuntu
successfully in following 3 modes:
1. standalone mode   (
http://cloudera-tutorial.blogspot.com/2010/11/running-cloudera-in-standalone-mode.html
)
2. pseudo distributed mode   (
http://cloudera-tutorial.blogspot.com/2010/11/running-cloudera-in-pseudo-distributed.html
)
3. distributed mode   (
http://cloudera-tutorial.blogspot.com/2010/11/running-cloudera-in-distributed-mode.html
)

please follow respective links


regards
Rahul Patodi
http://cloudera-tutorial.blogspot.com
Impetus Infotech (India) Pvt. Ltd.

On Sun, Nov 21, 2010 at 2:51 PM, Bernd Fondermann <bf_jak@brainlounge.de>wrote:

> On 12.11.10 20:06, Peter Minearo wrote:
>
>> Not sure who or where to send this to, but this is for the Cloudera
>> distribution of Hadoop:
>>
>
> Unfortunately, as CDH is not supported by Apache Hadoop, you'd have to
> contact the vendor directly. The vendor has a support forum reachable from
> his website if you dig little bit.
>
>  Bernd
>



-- 
-Thanks and Regards,
Rahul Patodi
Associate Software Engineer,
Impetus Infotech (India) Private Limited,
www.impetus.com
Mob:09907074413

--00163649a6bf9cacea0495933da7--

From general-return-2413-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 22 16:32:16 2010
Return-Path: <general-return-2413-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56744 invoked from network); 22 Nov 2010 16:32:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Nov 2010 16:32:15 -0000
Received: (qmail 14203 invoked by uid 500); 22 Nov 2010 16:32:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14091 invoked by uid 500); 22 Nov 2010 16:32:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14083 invoked by uid 99); 22 Nov 2010 16:32:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Nov 2010 16:32:44 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [98.139.91.74] (HELO nm4.bullet.mail.sp2.yahoo.com) (98.139.91.74)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 22 Nov 2010 16:32:34 +0000
Received: from [98.139.91.68] by nm4.bullet.mail.sp2.yahoo.com with NNFMP; 22 Nov 2010 16:32:12 -0000
Received: from [98.139.91.40] by tm8.bullet.mail.sp2.yahoo.com with NNFMP; 22 Nov 2010 16:32:12 -0000
Received: from [127.0.0.1] by omp1040.mail.sp2.yahoo.com with NNFMP; 22 Nov 2010 16:32:12 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 606122.70480.bm@omp1040.mail.sp2.yahoo.com
Received: (qmail 9516 invoked by uid 60001); 22 Nov 2010 16:32:12 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1290443532; bh=CAfgP8rvkh73AFTo6fEp919UyaGyWQ2RpIswaNJURxg=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=3jqbGIg0RsW4av89LHHZsw88MLBFi8AG+u81Dbi2trZVT5zAhL8d1an0S/b/bBODDSlSvLjHZdC3aSauUtILb9WZGbpNe0ZQRUVn1a4wCdCFsO/HLTMPEi4XFhM4NvUznnJ7rFUPL0MGn5tyjC6gI+nf7ZS/o27J7y/d1UbO1GQ=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=ADgT8IGBlnB2BcwNX1l2IYh5mqJ9Js8aQKI+mfXK3SPZ4CDRRYHhWGu2b6uEexO5P/7zRLQEU0HqXmw0177oj9MU+pidfKmtd00YQO2ZeXVjF9Jh4zyNI5DBWQ2rUBBRoi/AbQMV+PpQHwvKYxgQWaiudHVjzG1wkVH8OljwvAU=;
Message-ID: <126335.8536.qm@web112111.mail.gq1.yahoo.com>
X-YMail-OSG: 0KLBxmoVM1nyi8.bQpT7kLkaLlgPF2VHwvwq.rBJvmyCRVJ
 AptW_G2S12XLa7Um3jd5129o117hedkzpptoWTCUBXsZM6QTrwqlP89chuzg
 AVW8Xxj4tB4TLGgVh0FZIqGTHZ3BomG7RvhrG7vJ374NzArG8ttiCg.oHDqx
 lndfhdXHbo.b82uCBk2BATM35VXn2Vs8e6VU7oBFMYSXXtBAFQjbA8ksPZ54
 YgNIQWA1vuZTXEST4nz7LEW2QNPJwuSJlTY.0N0o3C0nOs90g2leLtQti979
 NqlcubHFpZVtGHf0dqsL5kQRcaXQlU7LFP43BvKdVi4uIZoZO.5KuU.xwG1p
 r_LsN
Received: from [129.132.85.169] by web112111.mail.gq1.yahoo.com via HTTP; Mon, 22 Nov 2010 08:32:12 PST
X-Mailer: YahooMailClassic/11.4.9 YahooMailWebService/0.8.107.285259
Date: Mon, 22 Nov 2010 08:32:12 -0800 (PST)
From: Grandl Robert <rgrandl@yahoo.com>
Subject: Hadoop - how exactly is a slot defined
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1891580573-1290443532=:8536"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-1891580573-1290443532=:8536
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Hi all,

I have troubles in understanding what exactly a slot is. Always we are talk=
ing about tasks assigned to slots, but I did not found anywhere what exactl=
y a slot is. I assume it represent some allocation of RAM memory as well as=
 with some computation power.=20

However, can somebody explain me what exactly a slot means (in terms of res=
ources allocated for a slot) and how this mapping(between slot and physical=
 resources) is done in Hadoop ? Or give me some hints about the files in th=
e Hadoop=A0 where it may should be ?

Thanks a lot,
Robert
=0A=0A
--0-1891580573-1290443532=:8536--

From general-return-2414-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 22 16:42:28 2010
Return-Path: <general-return-2414-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 61335 invoked from network); 22 Nov 2010 16:42:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Nov 2010 16:42:28 -0000
Received: (qmail 24443 invoked by uid 500); 22 Nov 2010 16:42:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24382 invoked by uid 500); 22 Nov 2010 16:42:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24374 invoked by uid 99); 22 Nov 2010 16:42:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Nov 2010 16:42:57 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [63.245.208.176] (HELO dm-mail02.mozilla.org) (63.245.208.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Nov 2010 16:42:51 +0000
X-Virus-Scanned: amavisd-new at mozilla.org
Received: from joubert.local (v74-nslb.mozilla.org [10.2.74.4])
	(Authenticated sender: xstevens@mozilla.com)
	by dm-mail02.mozilla.org (Postfix) with ESMTP id AAD6B824126
	for <general@hadoop.apache.org>; Mon, 22 Nov 2010 08:41:40 -0800 (PST)
Message-ID: <4CEA9D44.2090600@mozilla.com>
Date: Mon, 22 Nov 2010 08:41:40 -0800
From: Xavier Stevens <xstevens@mozilla.com>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Hadoop - how exactly is a slot defined
References: <126335.8536.qm@web112111.mail.gq1.yahoo.com>
In-Reply-To: <126335.8536.qm@web112111.mail.gq1.yahoo.com>
X-Enigmail-Version: 1.1.1
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi Robert,

Task slots are a way of constraining the amount of resources that will
be used on a given physical machine.  I believe the defaults set map
task slots to 2 and reduce task slots to 1.  Many of us use numbers
higher than this depending on our applications.  When picking a number
you want to keep in mind how many CPU cores and memory you have on a
physical machine.  If you set the child java opts to use 512MB of
memory; that number is per task.  So lets say you have 4 map task slots
and 2 reduce task slots.  Conceivably at some point all 6 slots would be
running tasks at some point which means you need at least 6 x 512MB of
memory plus a little extra for other processes (TaskTracker, DataNode,
etc.) and the OS.

A good starting point is to work backwards from how much physical memory
you have a decide how much you would like for tasks to have available. 
Or if you aren't worried about being memory heavy then you pick the
number of task slots to be close to the number of cores.

Hope this helps!

Cheers,


-Xavier

On 11/22/10 8:32 AM, Grandl Robert wrote:
> Hi all,
>
> I have troubles in understanding what exactly a slot is. Always we are talking about tasks assigned to slots, but I did not found anywhere what exactly a slot is. I assume it represent some allocation of RAM memory as well as with some computation power. 
>
> However, can somebody explain me what exactly a slot means (in terms of resources allocated for a slot) and how this mapping(between slot and physical resources) is done in Hadoop ? Or give me some hints about the files in the Hadoop  where it may should be ?
>
> Thanks a lot,
> Robert
>
>
>

From general-return-2415-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 22 16:52:25 2010
Return-Path: <general-return-2415-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 8418 invoked from network); 22 Nov 2010 16:52:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Nov 2010 16:52:25 -0000
Received: (qmail 41080 invoked by uid 500); 22 Nov 2010 16:52:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 41039 invoked by uid 500); 22 Nov 2010 16:52:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 41031 invoked by uid 99); 22 Nov 2010 16:52:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Nov 2010 16:52:55 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of qwertymaniac@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Nov 2010 16:52:49 +0000
Received: by pwj9 with SMTP id 9so1986408pwj.35
        for <general@hadoop.apache.org>; Mon, 22 Nov 2010 08:52:28 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=QrMKIUMO0PYIkUWKABUiURvEvB9qHXrYgN92RdxRmyU=;
        b=qF68Js2BS0VP88WRJ0QJesnAkPaByQWfLNejN7R23OSW5wMMSTXeffpgaJSDbFR/Cy
         85lLBV9HwFHFeaKhrLWrad4YcZ2lbuY2xybpFXr+T5nLS/OkXKyQkkaAzpcwC8gD4jph
         k0XNqVJP89dR5F7d7rQ7j6G7flNE2SHdGAKeo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=DjgFDP9OLz9REkS5LbJEsVNhCCmvzuAyDNaGIq84/lXTuyswJM7fL+V/B5lXetAxbo
         574D1prwrNzP60N8ddz2koPsPUU8iTobZlEavf07OsHr4+0k5RorCqZq3Sgcw/FHb7kK
         QDt6QwqfSAnLsi9GOVmAx5XGS07Sy3bnOkEsY=
Received: by 10.223.110.148 with SMTP id n20mr4362989fap.48.1290444746328;
 Mon, 22 Nov 2010 08:52:26 -0800 (PST)
MIME-Version: 1.0
Received: by 10.223.113.145 with HTTP; Mon, 22 Nov 2010 08:52:06 -0800 (PST)
In-Reply-To: <126335.8536.qm@web112111.mail.gq1.yahoo.com>
References: <126335.8536.qm@web112111.mail.gq1.yahoo.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Mon, 22 Nov 2010 22:22:06 +0530
Message-ID: <AANLkTimad3DwTUfozirAzFFX_WEj5eXyKwDwsjB0j1Ju@mail.gmail.com>
Subject: Re: Hadoop - how exactly is a slot defined
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

On Mon, Nov 22, 2010 at 10:02 PM, Grandl Robert <rgrandl@yahoo.com> wrote:
> Hi all,
>
> I have troubles in understanding what exactly a slot is. Always we are ta=
lking about tasks assigned to slots, but I did not found anywhere what exac=
tly a slot is. I assume it represent some allocation of RAM memory as well =
as with some computation power.
>
> However, can somebody explain me what exactly a slot means (in terms of r=
esources allocated for a slot) and how this mapping(between slot and physic=
al resources) is done in Hadoop ? Or give me some hints about the files in =
the Hadoop=A0 where it may should be ?

A slot is of two types -- Map slot and Reduce slot. A slot represents
an ability to run one of these "Tasks" (map/reduce tasks) individually
at a point of time. Therefore, multiple slots on a TaskTracker means
multiple "Tasks" may execute in parallel.

Right now total slots in a TaskTracker is =3D=3D
mapred.tasktracker.map.tasks.maximum for Maps and
mapred.tasktracker.reduce.tasks.maximum for Reduces.

Hadoop is indeed trying to go towards the dynamic slot concept, which
could rely on the current resources available on a system, but work
for this is still in conceptual phases. TaskTrackers emit system
status (like CPU load, utilization, memory available/user, load
averages) in their heartbeats today (and is utilized by certain
schedulers, I think Capacity Scheduler uses it to determine stuff),
but the concept of slots is still fixed as a maximum to the above two
configurations on each TaskTracker.

For code on how slots are checked/utilized, see any Scheduler plugin's
code -- LimitTasksPerJobTaskScheduler, CapacityTaskScheduler for
example.

>
> Thanks a lot,
> Robert
>
>
>



--=20
Harsh J
www.harshj.com

From general-return-2416-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 22 17:38:21 2010
Return-Path: <general-return-2416-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44370 invoked from network); 22 Nov 2010 17:38:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Nov 2010 17:38:21 -0000
Received: (qmail 23586 invoked by uid 500); 22 Nov 2010 17:38:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23518 invoked by uid 500); 22 Nov 2010 17:38:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23507 invoked by uid 99); 22 Nov 2010 17:38:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Nov 2010 17:38:51 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [98.139.91.97] (HELO nm27.bullet.mail.sp2.yahoo.com) (98.139.91.97)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 22 Nov 2010 17:38:44 +0000
Received: from [98.139.91.68] by nm27.bullet.mail.sp2.yahoo.com with NNFMP; 22 Nov 2010 17:38:23 -0000
Received: from [98.139.91.26] by tm8.bullet.mail.sp2.yahoo.com with NNFMP; 22 Nov 2010 17:38:23 -0000
Received: from [127.0.0.1] by omp1026.mail.sp2.yahoo.com with NNFMP; 22 Nov 2010 17:38:23 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 812477.55270.bm@omp1026.mail.sp2.yahoo.com
Received: (qmail 93485 invoked by uid 60001); 22 Nov 2010 17:38:23 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1290447503; bh=AJbzeQHd5SGYEJJbEWaz/HvIV+5/xG9ZpVb4WtS2Y0o=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=gaQ2a20BAdqrc2DpRK8rZpF4AfVx6kM2OPKDL1wc+IFl474XB+/mF/hORKDntkRzctzLsDUJzytMHtOVAQu2coQry1kZ6QVTMvKTvGPiLDNRzJmNPUOZB+R1s04isKLrJ9/7R/XGuvyYj5HZHPNpcZ4LxFtZg3d+Pnpxk4pgJq0=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=VLhC6F6HPY+AN3i512BwUShnsUxXyK7MMx/oP7jcQhOpaEcZ414qOMZOp2F7kC+y481uAFU2Gu/OWunn0pazUYmz/AjoAIJRoazGHteUxLwHLmcy0LwivQU7eh1hJ9iY/eBOs1zHxU9h4qVAK+CmgEo4b6mC/4Qy9PZBRZbKcT4=;
Message-ID: <405329.93459.qm@web112118.mail.gq1.yahoo.com>
X-YMail-OSG: iN9r6MoVM1mhnog6wyg3STJyx3zmZO6lGKuVhJQqwFnosDf
 IbtJbftOQ2i0LYM.9CzcSSwLb9Sj5231egEg3r3BWTIMwIwpxInkUXTDVape
 MU3vI.akhdCIoB19XpKPx5HrDt90S5sTuuOk69YxeFtRZH4GW0JZCiM1t0r1
 .amAVg0u2ChoXEWjble8lwNmzjo6eiyG4iGgkUOOzsOq.eK7QSbMUCOVd.Ip
 9a5xLz1mT8P5LPJY2I_fD4yEv87bOsatma9IjEFQh7wPO2hbK84bh1lZkoec
 CX0hUiylg_ZG8pTnXWtIObRVJkBXNLJ_zeTktBD_apl38U6F.YKa6Cex3LxM
 x0WKphsWwyo3Q1LaDXQSBal686SkUWvEOm8DSB6o5xpU1
Received: from [129.132.85.169] by web112118.mail.gq1.yahoo.com via HTTP; Mon, 22 Nov 2010 09:38:22 PST
X-Mailer: YahooMailClassic/11.4.9 YahooMailWebService/0.8.107.285259
Date: Mon, 22 Nov 2010 09:38:22 -0800 (PST)
From: Grandl Robert <rgrandl@yahoo.com>
Subject: Re: Hadoop - how exactly is a slot defined
To: general@hadoop.apache.org
In-Reply-To: <AANLkTimad3DwTUfozirAzFFX_WEj5eXyKwDwsjB0j1Ju@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-288647005-1290447502=:93459"

--0-288647005-1290447502=:93459
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Thanks all for your comments.

However, I still have some doubts.=20

Basically I can control the number of map/reduce slots with
mapred.tasktracker.map.tasks.maximum
mapred.tasktracker.reduce.tasks.maximum=20

but, it is possible to set different number of map/reduce slots for differe=
nt slaves ?

For example If I am running in a heterogeneous environment, where each slav=
e have different configuration, it is possible to set different number of s=
lots based on the specific machine configurations ?=20
For the moment I observed that I can modify only on the master this paramet=
ers, therefore all the nodes will run with same number of map/reduce slots =
careless of whatever resources(CPU,MEMORY) offer each other.=20

Thanks for any clue.

Robert



--- On Mon, 11/22/10, Harsh J <qwertymaniac@gmail.com> wrote:

From: Harsh J <qwertymaniac@gmail.com>
Subject: Re: Hadoop - how exactly is a slot defined
To: general@hadoop.apache.org
Date: Monday, November 22, 2010, 6:52 PM

Hi,

On Mon, Nov 22, 2010 at 10:02 PM, Grandl Robert <rgrandl@yahoo.com> wrote:
> Hi all,
>
> I have troubles in understanding what exactly a slot is. Always we are ta=
lking about tasks assigned to slots, but I did not found anywhere what exac=
tly a slot is. I assume it represent some allocation of RAM memory as well =
as with some computation power.
>
> However, can somebody explain me what exactly a slot means (in terms of r=
esources allocated for a slot) and how this mapping(between slot and physic=
al resources) is done in Hadoop ? Or give me some hints about the files in =
the Hadoop=A0 where it may should be ?

A slot is of two types -- Map slot and Reduce slot. A slot represents
an ability to run one of these "Tasks" (map/reduce tasks) individually
at a point of time. Therefore, multiple slots on a TaskTracker means
multiple "Tasks" may execute in parallel.

Right now total slots in a TaskTracker is =3D=3D
mapred.tasktracker.map.tasks.maximum for Maps and
mapred.tasktracker.reduce.tasks.maximum for Reduces.

Hadoop is indeed trying to go towards the dynamic slot concept, which
could rely on the current resources available on a system, but work
for this is still in conceptual phases. TaskTrackers emit system
status (like CPU load, utilization, memory available/user, load
averages) in their heartbeats today (and is utilized by certain
schedulers, I think Capacity Scheduler uses it to determine stuff),
but the concept of slots is still fixed as a maximum to the above two
configurations on each TaskTracker.

For code on how slots are checked/utilized, see any Scheduler plugin's
code -- LimitTasksPerJobTaskScheduler, CapacityTaskScheduler for
example.

>
> Thanks a lot,
> Robert
>
>
>



--=20
Harsh J
www.harshj.com
=0A=0A=0A      
--0-288647005-1290447502=:93459--

From general-return-2417-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 22 18:11:40 2010
Return-Path: <general-return-2417-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72382 invoked from network); 22 Nov 2010 18:11:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Nov 2010 18:11:39 -0000
Received: (qmail 81291 invoked by uid 500); 22 Nov 2010 18:12:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81056 invoked by uid 500); 22 Nov 2010 18:12:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81048 invoked by uid 99); 22 Nov 2010 18:12:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Nov 2010 18:12:09 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Nov 2010 18:12:02 +0000
Received: by fxm2 with SMTP id 2so2998363fxm.35
        for <general@hadoop.apache.org>; Mon, 22 Nov 2010 10:11:42 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type;
        bh=YDytVvJenH5avP0FXCqOPWKZICh4YYw6Ub0wUHQngpk=;
        b=EEVOlO4y7ohp0L3TIRO4JdHsrZTFy8YjjHZsgdSit8mVYAtJxODXxRm5P+QbvaISof
         atM3E7py6P1FPm9OJOBpOjrczvpLenkx9XQGw9RE0RpVcKefcKm8b5FmAnpOBvyA/lBZ
         iizEog3mm1kruen0jSJmJ7oerq3LggE2K3Ic0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=E3BIuh205hpjeZSgeyNdkAElMNqO6FEqSBR3nzLmnWD9O4y7iT8QDN11aC+fqcSVez
         uZ0wy93Oc9U0scn1MHFTcPNYiRVHkvYdQ5uIcHAYp9Y0NFPXfVWLgFw01YapuAITxFfT
         cU8Sl0zISOx+bJ6mW9vPzZau3ZJPnb9y0tKt8=
Received: by 10.223.83.133 with SMTP id f5mr3616962fal.29.1290449502262; Mon,
 22 Nov 2010 10:11:42 -0800 (PST)
MIME-Version: 1.0
Received: by 10.223.113.145 with HTTP; Mon, 22 Nov 2010 10:11:22 -0800 (PST)
In-Reply-To: <405329.93459.qm@web112118.mail.gq1.yahoo.com>
References: <AANLkTimad3DwTUfozirAzFFX_WEj5eXyKwDwsjB0j1Ju@mail.gmail.com> <405329.93459.qm@web112118.mail.gq1.yahoo.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Mon, 22 Nov 2010 23:41:22 +0530
Message-ID: <AANLkTinMEdxmyVzatCG-=NyuBmQz9tnEE3xzm9Ldgi2y@mail.gmail.com>
Subject: Re: Hadoop - how exactly is a slot defined
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

Answers inline.

On Mon, Nov 22, 2010 at 11:08 PM, Grandl Robert <rgrandl@yahoo.com> wrote:
> Thanks all for your comments.
>
> However, I still have some doubts.
>
> Basically I can control the number of map/reduce slots with
> mapred.tasktracker.map.tasks.maximum
> mapred.tasktracker.reduce.tasks.maximum
>
> but, it is possible to set different number of map/reduce slots for different slaves ?

Yes, this setting is 'tasktracker' specific, as the property name
goes. Each TaskTracker can have a different config to load from.

>
> For example If I am running in a heterogeneous environment, where each slave have different configuration, it is possible to set different number of slots based on the specific machine configurations ?

Yes, give each machine a unique value via its local copy of mapred-site.xml

> For the moment I observed that I can modify only on the master this parameters, therefore all the nodes will run with same number of map/reduce slots careless of whatever resources(CPU,MEMORY) offer each other.

Not really, your slave machine's config file (conf/mapred-site.xml)
needs to reflect the proper settings you need it to use for its
TaskTracker (DataNodes have specific configuration as well).

-- 
Harsh J
www.harshj.com

From general-return-2418-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 22 18:27:25 2010
Return-Path: <general-return-2418-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 92391 invoked from network); 22 Nov 2010 18:27:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Nov 2010 18:27:25 -0000
Received: (qmail 10382 invoked by uid 500); 22 Nov 2010 18:27:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 10303 invoked by uid 500); 22 Nov 2010 18:27:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 10295 invoked by uid 99); 22 Nov 2010 18:27:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Nov 2010 18:27:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 22 Nov 2010 18:27:54 +0000
Received: (qmail 92328 invoked by uid 99); 22 Nov 2010 18:27:02 -0000
Received: from localhost.apache.org (HELO mail-pw0-f48.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Nov 2010 18:27:02 +0000
Received: by pwj9 with SMTP id 9so2069720pwj.35
        for <general@hadoop.apache.org>; Mon, 22 Nov 2010 10:27:33 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.182.13 with SMTP id ca13mr7216759ibb.16.1290450453186;
 Mon, 22 Nov 2010 10:27:33 -0800 (PST)
Received: by 10.231.167.199 with HTTP; Mon, 22 Nov 2010 10:27:33 -0800 (PST)
In-Reply-To: <270521A0-A54F-4482-8A36-ABD6FB5ECD96@apache.org>
References: <270521A0-A54F-4482-8A36-ABD6FB5ECD96@apache.org>
Date: Mon, 22 Nov 2010 10:27:33 -0800
Message-ID: <AANLkTinm5ojG_ctoa+fy_EN99ojvoYNvTjhZVUwN+dq3@mail.gmail.com>
Subject: Re: [VOTE] Hadoop Bylaws
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

+1 -C

---

Summary of protocol for major votes:
  Adding code: Lazy consensus (committers)
  Release plan: Lazy majority (committers)
  Releasing code: Lazy majority (PMC)
  New codebase/subproject: Lazy 2/3 majority (PMC)
  Adding contributor: Lazy consensus (PMC)
  Removing contributor: Lazy 2/3 majority (PMC)
  Change bylaws: Majority (PMC)
  7 day voting period

On Fri, Nov 19, 2010 at 10:45 PM, Owen O'Malley <omalley@apache.org> wrote:
> All,
> =A0 I've updated the proposed bylaws that I sent out a few weeks ago as
> suggested by making the vote for removal be a lazy 2/3, so now all of the
> votes are lazy. As before, by a recursive application of the bylaws let's
> let the vote last 7 days (11/26) and require a lazy majority of PMC membe=
rs
> to pass.
>
> -- Owen
>
> Apache Hadoop Project Bylaws
>
> This document defines the bylaws under which the Apache Hadoop project
> operates. It defines the roles and responsibilities of the project,
> who may vote, how voting works, how conflicts are resolved, etc.
>
> Hadoop is a project of the <a
> href=3D"http://www.apache.org/foundation/">Apache Software
> Foundation</a>. =A0The foundation holds the trademark on the name
> "Hadoop" and copyright on Apache code
> including the code in the Hadoop codebase. The <a
> href=3D"http://www.apache.org/foundation/faq.html">foundation FAQ</a>
> explains the operation and background of the foundation.
>
> Hadoop is typical of Apache projects in that it operates under a set
> of principles, known collectively as the &quot;Apache Way&quot;. If
> you are new to Apache development, please refer to the <a
> href=3D"http://incubator.apache.org">Incubator project</a> for more
> information on how Apache projects operate.
>
> Roles and Responsibilities
>
> Apache projects define a set of roles with associated rights and
> responsibilities. These roles govern what tasks an individual may
> perform within the project. The roles are defined in the following
> sections
>
> =A0 * Users
>
> =A0 =A0 =A0 =A0The most important participants in the project are people =
who
> =A0 =A0 =A0 =A0use our software. The majority of our developers start out=
 as
> =A0 =A0 =A0 =A0users and guide their development efforts from the user's
> =A0 =A0 =A0 =A0perspective.
>
> =A0 =A0 =A0 Users contribute to the Apache projects by providing feedback
> =A0 =A0 =A0 =A0to developers in the form of bug reports and feature
> =A0 =A0 =A0 =A0suggestions. As well, users participate in the Apache
> =A0 =A0 =A0 =A0community by helping other users on mailing lists and user
> =A0 =A0 =A0 =A0support forums.
>
> =A0 * Contributors
>
> =A0 =A0 =A0 =A0All of the volunteers who are contributing time, code,
> =A0 =A0 =A0 =A0documentation, or resources to the Hadoop Project. A contr=
ibutor
> =A0 =A0 =A0 =A0that makes sustained, welcome contributions to the project=
 may
> =A0 =A0 =A0 =A0be invited to become a Committer, though the exact timing =
of
> =A0 =A0 =A0 =A0such invitations depends on many factors.
>
> =A0 =A0* Committers
>
> =A0 =A0 =A0 The project's Committers are responsible for the project's
> =A0 =A0 =A0 =A0technical management. Committers have access to a specifie=
d
> =A0 =A0 =A0 =A0set of subprojects' subversion repositories. Committers on
> =A0 =A0 =A0 =A0subprojects may cast binding votes on any technical discus=
sion
> =A0 =A0 =A0 =A0regarding that subproject.
>
> =A0 =A0 =A0 Committer access is by invitation only and must be approved b=
y
> =A0 =A0 =A0 =A0lazy consensus of the active PMC members. A Committer is
> =A0 =A0 =A0 =A0considered emeritus by their own declaration or by not
> =A0 =A0 =A0 =A0contributing in any form to the project for over six
> =A0 =A0 =A0 =A0months. An emeritus committer may request reinstatement of
> =A0 =A0 =A0 =A0commit access from the PMC. Such reinstatement is subject =
to
> =A0 =A0 =A0 =A0lazy consensus of active PMC members.
>
> =A0 =A0 =A0All Apache committers are required to have a signed Contributo=
r
> =A0 =A0 =A0 =A0License Agreement (CLA) on file with the Apache Software
> =A0 =A0 =A0 =A0Foundation. There is a <a
> =A0 =A0 =A0 =A0href=3D"http://www.apache.org/dev/committers.html">Committ=
er
> =A0 =A0 =A0 =A0FAQ</a> which provides more details on the requirements fo=
r
> =A0 =A0 =A0 =A0Committers
>
> =A0 =A0 =A0 A committer who makes a sustained contribution to the project
> =A0 =A0 =A0 =A0may be invited to become a member of the PMC. The form of
> =A0 =A0 =A0 =A0contribution is not limited to code. It can also include c=
ode
> =A0 =A0 =A0 =A0review, helping out users on the mailing lists, documentat=
ion,
> =A0 =A0 =A0 =A0testing, etc.
>
> =A0 * Project Management Committee
>
> =A0 =A0 =A0 The Project Management Committee (PMC) for Apache Hadoop was
> =A0 =A0 =A0 =A0created by the Apache Board in January 2008 when Hadoop mo=
ved
> =A0 =A0 =A0 =A0out of Lucene and became a top level project at Apache. Th=
e
> =A0 =A0 =A0 =A0PMC is responsible to the board and the ASF for the manage=
ment
> =A0 =A0 =A0 =A0and oversight of the Apache Hadoop codebase. The
> =A0 =A0 =A0 =A0responsibilities of the PMC include
>
> =A0 =A0 =A0 * Deciding what is distributed as products of the Apache Hado=
op
> =A0 =A0 =A0 =A0project. =A0In particular all releases must be approved by=
 the
> =A0 =A0 =A0 =A0PMC
>
> =A0 =A0 =A0 =A0* Maintaining the project's shared resources, including th=
e codebase
> =A0 =A0 =A0 =A0 =A0 =A0repository, mailing lists, websites.
>
> =A0 =A0 =A0 =A0* Speaking on behalf of the project.
>
> =A0 =A0 =A0 * Resolving license disputes regarding products of the projec=
t
>
> =A0 =A0 =A0 =A0* Nominating new PMC members and committers
>
> =A0 =A0 =A0 =A0* Maintaining these bylaws and other guidelines of the pro=
ject
>
> =A0 =A0 =A0 Membership of the PMC is by invitation only and must be
> =A0 =A0 =A0 =A0approved by a lazy consensus of active PMC members. A PMC
> =A0 =A0 =A0 =A0member is considered &quot;emeritus&quot; by their own
> =A0 =A0 =A0 =A0declaration or by not contributing in any form to the proj=
ect
> =A0 =A0 =A0 =A0for over six months. An emeritus member may request
> =A0 =A0 =A0 =A0reinstatement to the PMC. Such reinstatement is subject to
> =A0 =A0 =A0 =A0lazy consensus of the active PMC members.
>
> =A0 =A0 =A0 The chair of the PMC is appointed by the ASF board. The chair
> =A0 =A0 =A0 =A0is an office holder of the Apache Software Foundation (Vic=
e
> =A0 =A0 =A0 =A0President, Apache Hadoop) and has primary responsibility t=
o
> =A0 =A0 =A0 =A0the board for the management of the projects within the sc=
ope
> =A0 =A0 =A0 =A0of the Hadoop PMC. The chair reports to the board quarterl=
y on
> =A0 =A0 =A0 =A0developments within the Hadoop project.
>
> =A0 =A0 =A0 =A0When the current chair of the PMC resigns, the PMC votes t=
o
> =A0 =A0 =A0 =A0recommend a new chair using lazy consensus, but the decisi=
on
> =A0 =A0 =A0 =A0must be ratified by the Apache board.
>
> Decision Making
>
> =A0 =A0 =A0Within the Hadoop project, different types of decisions requir=
e
> =A0 =A0 =A0different forms of approval. For example, the <a href=3D"#Role=
s
> =A0 =A0 =A0and Responsibilities">previous section</a> describes several
> =A0 =A0 =A0decisions which require &quot;lazy consensus&quot;
> =A0 =A0 =A0approval. This section defines how voting is performed, the
> =A0 =A0 =A0types of approvals, and which types of decision require which
> =A0 =A0 =A0type of approval.
>
> =A0 =A0* Voting
>
> =A0 =A0 =A0 =A0Decisions regarding the project are made by votes on the
> =A0 =A0 =A0 =A0primary project development mailing list
> =A0 =A0 =A0 =A0(general@hadoop.apache.org). =A0Where necessary, PMC votin=
g may
> =A0 =A0 =A0 =A0take place on the private Hadoop PMC mailing list. =A0Vote=
s are
> =A0 =A0 =A0 =A0clearly indicated by subject line starting with [VOTE]. =
=A0Votes
> =A0 =A0 =A0 =A0may contain multiple items for approval and these should b=
e
> =A0 =A0 =A0 =A0clearly separated. Voting is carried out by replying to th=
e
> =A0 =A0 =A0 =A0vote mail. Voting may take four flavors
>
> =A0 =A0 =A0 =A0* +1 &quot;Yes,&quot; &quot;Agree,&quot; or &quot;the acti=
on should
> be
> =A0 =A0 =A0 =A0 =A0 =A0performed.&quot; In general, this vote also indica=
tes a
> willingness
> =A0 =A0 =A0 =A0 =A0 =A0on the behalf of the voter in &quot;making it happ=
en&quot;
>
> =A0 =A0 =A0 =A0 * +0 This vote indicates a willingness for the action und=
er
> =A0 =A0 =A0 =A0 =A0 =A0consideration to go ahead. The voter, however will=
 not be able
> =A0 =A0 =A0 =A0 =A0 =A0to help.
>
> =A0 =A0 =A0 =A0 * -0 This vote indicates that the voter does not, in gene=
ral, agree
> with
> =A0 =A0 =A0 =A0 =A0 =A0the proposed action but is not concerned enough to=
 prevent the
> =A0 =A0 =A0 =A0 =A0 =A0action going ahead.
>
> =A0 =A0 =A0 =A0 =A0* -1 This is a negative vote. On issues where consensu=
s is
> required,
> =A0 =A0 =A0 =A0 =A0 =A0this vote counts as a <strong>veto</strong>. All v=
etoes must
> =A0 =A0 =A0 =A0 =A0 =A0contain an explanation of why the veto is appropri=
ate. Vetoes
> with
> =A0 =A0 =A0 =A0 =A0 =A0no explanation are void. It may also be appropriat=
e for a -1 vote
> =A0 =A0 =A0 =A0 =A0 =A0to include an alternative course of action.
>
> =A0 =A0 =A0 =A0All participants in the Hadoop project are encouraged to s=
how their
> =A0 =A0 =A0 =A0agreement with or against a particular action by voting. F=
or
> technical
> =A0 =A0 =A0 =A0decisions, only the votes of active committers are binding=
. Non
> binding
> =A0 =A0 =A0 =A0votes are still useful for those with binding votes to und=
erstand the
> =A0 =A0 =A0 =A0perception of an action in the wider Hadoop community. For=
 PMC
> decisions,
> =A0 =A0 =A0 =A0only the votes of PMC members are binding.
>
> =A0 =A0 =A0 Voting can also be applied to changes made to the Hadoop code=
base.
> These
> =A0 =A0 =A0 =A0typically take the form of a veto (-1) in reply to the com=
mit message
> =A0 =A0 =A0 =A0sent when the commit is made.
>
> =A0 =A0* Approvals
>
> =A0 =A0 =A0 =A0These are the types of approvals that can be sought. Diffe=
rent
> actions
> =A0 =A0 =A0 =A0require different types of approvals
>
> =A0 =A0 =A0 =A0* Lazy Consensus - Lazy consensus requires 3 binding +1 vo=
tes
> =A0 =A0 =A0 =A0 =A0 and no binding vetoes.
>
> =A0 =A0 =A0 =A0 * Lazy Majority - A lazy majority vote requires 3 binding=
 +1
> =A0 =A0 =A0 =A0 =A0 =A0votes and more binding +1 votes than -1 votes.
>
> =A0 =A0 =A0 =A0* Lazy 2/3 Majority - Lazy 2/3 majority votes requires at
> =A0 =A0 =A0 =A0 =A0 least 3 votes and twice as many +1 votes as -1 votes.
>
> =A0 =A0* Vetoes
>
> =A0 =A0 =A0 =A0A valid, binding veto cannot be overruled. If a veto is ca=
st,
> =A0 =A0 =A0 =A0it must be accompanied by a valid reason explaining the
> =A0 =A0 =A0 =A0reasons for the veto. The validity of a veto, if challenge=
d,
> =A0 =A0 =A0 =A0can be confirmed by anyone who has a binding vote. This do=
es
> =A0 =A0 =A0 =A0not necessarily signify agreement with the veto - merely t=
hat
> =A0 =A0 =A0 =A0the veto is valid.
>
> =A0 =A0 =A0 If you disagree with a valid veto, you must lobby the person =
casting
> =A0 =A0 =A0 =A0the veto to withdraw their veto. If a veto is not withdraw=
n, any
> action
> =A0 =A0 =A0 =A0that has been vetoed must be reversed in a timely manner.
>
> =A0 =A0* Actions
>
> =A0 =A0 =A0 =A0This section describes the various actions which are under=
taken
> within
> =A0 =A0 =A0 =A0the project, the corresponding approval required for that =
action and
> =A0 =A0 =A0 =A0those who have binding votes over the action.
>
> =A0 =A0 =A0 =A0 * Code Change
>
> =A0 =A0 =A0 =A0 =A0 =A0A change made to a codebase of the project and com=
mitted
> =A0 =A0 =A0 =A0 =A0 =A0by a committer. This includes source code, documen=
tation, website
> =A0 =A0 =A0 =A0 =A0 =A0content, etc.
>
> =A0 =A0 =A0 =A0 =A0 =A0Lazy consensus of active committers.
>
> =A0 =A0 =A0 =A0 * Release Plan
>
> =A0 =A0 =A0 =A0 =A0 =A0Defines the timetable and actions for a release. T=
he plan also
> =A0 =A0 =A0 =A0 =A0 =A0nominates a Release Manager.
>
> =A0 =A0 =A0 =A0 =A0 =A0Lazy majority of active committers
>
> =A0 =A0 =A0 =A0 * Product Release
>
> =A0 =A0 =A0 =A0 =A0 =A0When a release of one of the project's products is=
 ready, a vote
> is
> =A0 =A0 =A0 =A0 =A0 =A0required to accept the release as an official rele=
ase of the
> =A0 =A0 =A0 =A0 =A0 =A0project.
>
> =A0 =A0 =A0 =A0 =A0 Lazy Majority of active PMC members
>
> =A0 =A0 =A0 =A0 =A0* Adoption of New Codebase
>
> =A0 =A0 =A0 =A0 =A0 =A0When the codebase for an existing, released produc=
t is to be
> =A0 =A0 =A0 =A0 =A0 =A0replaced with an alternative codebase. If such a v=
ote fails to
> =A0 =A0 =A0 =A0 =A0 =A0gain approval, the existing code base will continu=
e.
>
> =A0 =A0 =A0 =A0 =A0 This also covers the creation of new sub-projects
> =A0 =A0 =A0 =A0 =A0 =A0within the project
>
> =A0 =A0 =A0 =A0 =A0 Lazy 2/3 majority of PMC members
>
> =A0 =A0 =A0 =A0 =A0* New Committer
>
> =A0 =A0 =A0 =A0 =A0 =A0When a new committer is proposed for the project
>
> =A0 =A0 =A0 =A0 =A0 =A0Lazy consensus of active PMC members
>
> =A0 =A0 =A0 =A0 =A0* New PMC Member
>
> =A0 =A0 =A0 =A0 =A0 =A0When a committer is proposed for the PMC
>
> =A0 =A0 =A0 =A0 =A0 Lazy consensus of active PMC members
>
> =A0 =A0 =A0 =A0 =A0* Committer Removal
>
> =A0 =A0 =A0 =A0 =A0 When removal of commit privileges is sought. =A0Note:=
 Such
> =A0 =A0 =A0 =A0 =A0 actions will also be referred to the ASF board by the=
 PMC
> =A0 =A0 =A0 =A0 =A0 chair
>
> =A0 =A0 =A0 =A0 =A0 =A0Lazy 2/3 majority of active PMC members (excluding=
 the
> =A0 =A0 =A0 =A0 =A0 =A0committer in question if a member of the PMC).
>
> =A0 =A0 =A0 =A0 =A0* PMC Member Removal
>
> =A0 =A0 =A0 =A0 =A0 =A0When removal of a PMC member is sought. =A0Note: S=
uch
> =A0 =A0 =A0 =A0 =A0 =A0actions will also be referred to the ASF board by =
the PMC
> =A0 =A0 =A0 =A0 =A0 =A0chair.
>
> =A0 =A0 =A0 =A0 =A0 =A0Lazy 2/3 majority of active PMC members (excluding=
 the member in
> question)
>
> =A0 =A0 =A0 =A0 =A0* Modifying Bylaws
>
> =A0 =A0 =A0 =A0 =A0 =A0Modifying this document.
>
> =A0 =A0 =A0 =A0 =A0 =A0Majority of active PMC members
>
> =A0 * Voting Timeframes
>
> =A0 =A0 =A0 =A0Votes are open for a period of 7 days to allow all active
> =A0 =A0 =A0 =A0voters time to consider the vote. Votes relating to code
> =A0 =A0 =A0 =A0changes are not subject to a strict timetable but should b=
e
> =A0 =A0 =A0 =A0made as timely as possible.
>
>

From general-return-2419-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 22 18:33:19 2010
Return-Path: <general-return-2419-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96278 invoked from network); 22 Nov 2010 18:33:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Nov 2010 18:33:19 -0000
Received: (qmail 21855 invoked by uid 500); 22 Nov 2010 18:33:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21704 invoked by uid 500); 22 Nov 2010 18:33:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21695 invoked by uid 99); 22 Nov 2010 18:33:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Nov 2010 18:33:49 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Nov 2010 18:33:42 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id oAMIWpFb026971
	for <general@hadoop.apache.org>; Mon, 22 Nov 2010 10:32:51 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1290450772;
	bh=vwQoGe/eT54UkhIzBMX+K0ZgGwnf+UdFacATwep9FSg=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=ONmQUKAf0wo/HQq47P1bBAITcMgcATm/eFlXgokeJPr/BVaS7bM8q0BtH3n1yvn9j
	 dg9f1x73XAKnzpQLSRW1tMqdgBCk98Yuq+oKwECaW6u5dgZgX9xOGAWpxzMspc/uDw
	 niqFwOzwRECWPg9zdOk0Z2NCDdN6rMlJ0ewpiuOU=
Message-Id: <E5F77221-9320-4251-9F15-2119F7435670@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <270521A0-A54F-4482-8A36-ABD6FB5ECD96@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Hadoop Bylaws
Date: Mon, 22 Nov 2010 10:32:51 -0800
References: <270521A0-A54F-4482-8A36-ABD6FB5ECD96@apache.org>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

+1, especially given the modifications to incorporate 2/3 Lazy  
Consensus for removals.

Arun

On Nov 19, 2010, at 10:45 PM, Owen O'Malley wrote:

> All,
>    I've updated the proposed bylaws that I sent out a few weeks ago
> as suggested by making the vote for removal be a lazy 2/3, so now all
> of the votes are lazy. As before, by a recursive application of the
> bylaws let's let the vote last 7 days (11/26) and require a lazy
> majority of PMC members to pass.
>
> -- Owen
>
> Apache Hadoop Project Bylaws
>
> This document defines the bylaws under which the Apache Hadoop project
> operates. It defines the roles and responsibilities of the project,
> who may vote, how voting works, how conflicts are resolved, etc.
>
> Hadoop is a project of the <a
> href="http://www.apache.org/foundation/">Apache Software
> Foundation</a>.  The foundation holds the trademark on the name
> "Hadoop" and copyright on Apache code
> including the code in the Hadoop codebase. The <a
> href="http://www.apache.org/foundation/faq.html">foundation FAQ</a>
> explains the operation and background of the foundation.
>
> Hadoop is typical of Apache projects in that it operates under a set
> of principles, known collectively as the &quot;Apache Way&quot;. If
> you are new to Apache development, please refer to the <a
> href="http://incubator.apache.org">Incubator project</a> for more
> information on how Apache projects operate.
>
> Roles and Responsibilities
>
> Apache projects define a set of roles with associated rights and
> responsibilities. These roles govern what tasks an individual may
> perform within the project. The roles are defined in the following
> sections
>
>    * Users
>
>         The most important participants in the project are people who
>         use our software. The majority of our developers start out as
>         users and guide their development efforts from the user's
>         perspective.
>
>        Users contribute to the Apache projects by providing feedback
>         to developers in the form of bug reports and feature
>         suggestions. As well, users participate in the Apache
>         community by helping other users on mailing lists and user
>         support forums.
>
>    * Contributors
>
>         All of the volunteers who are contributing time, code,
>         documentation, or resources to the Hadoop Project. A
> contributor
>         that makes sustained, welcome contributions to the project may
>         be invited to become a Committer, though the exact timing of
>         such invitations depends on many factors.
>
>     * Committers
>
>        The project's Committers are responsible for the project's
>         technical management. Committers have access to a specified
>         set of subprojects' subversion repositories. Committers on
>         subprojects may cast binding votes on any technical discussion
>         regarding that subproject.
>
>        Committer access is by invitation only and must be approved by
>         lazy consensus of the active PMC members. A Committer is
>         considered emeritus by their own declaration or by not
>         contributing in any form to the project for over six
>         months. An emeritus committer may request reinstatement of
>         commit access from the PMC. Such reinstatement is subject to
>         lazy consensus of active PMC members.
>
>       All Apache committers are required to have a signed Contributor
>         License Agreement (CLA) on file with the Apache Software
>         Foundation. There is a <a
>         href="http://www.apache.org/dev/committers.html">Committer
>         FAQ</a> which provides more details on the requirements for
>         Committers
>
>        A committer who makes a sustained contribution to the project
>         may be invited to become a member of the PMC. The form of
>         contribution is not limited to code. It can also include code
>         review, helping out users on the mailing lists, documentation,
>         testing, etc.
>
>    * Project Management Committee
>
>        The Project Management Committee (PMC) for Apache Hadoop was
>         created by the Apache Board in January 2008 when Hadoop moved
>         out of Lucene and became a top level project at Apache. The
>         PMC is responsible to the board and the ASF for the management
>         and oversight of the Apache Hadoop codebase. The
>         responsibilities of the PMC include
>
>        * Deciding what is distributed as products of the Apache Hadoop
>         project.  In particular all releases must be approved by the
>         PMC
>
>         * Maintaining the project's shared resources, including the
> codebase
>             repository, mailing lists, websites.
>
>         * Speaking on behalf of the project.
>
>        * Resolving license disputes regarding products of the project
>
>         * Nominating new PMC members and committers
>
>         * Maintaining these bylaws and other guidelines of the project
>
>        Membership of the PMC is by invitation only and must be
>         approved by a lazy consensus of active PMC members. A PMC
>         member is considered &quot;emeritus&quot; by their own
>         declaration or by not contributing in any form to the project
>         for over six months. An emeritus member may request
>         reinstatement to the PMC. Such reinstatement is subject to
>         lazy consensus of the active PMC members.
>
>        The chair of the PMC is appointed by the ASF board. The chair
>         is an office holder of the Apache Software Foundation (Vice
>         President, Apache Hadoop) and has primary responsibility to
>         the board for the management of the projects within the scope
>         of the Hadoop PMC. The chair reports to the board quarterly on
>         developments within the Hadoop project.
>
>        When the current chair of the PMC resigns, the PMC votes to
>        recommend a new chair using lazy consensus, but the decision
>        must be ratified by the Apache board.
>
> Decision Making
>
>       Within the Hadoop project, different types of decisions require
>       different forms of approval. For example, the <a href="#Roles
>       and Responsibilities">previous section</a> describes several
>       decisions which require &quot;lazy consensus&quot;
>       approval. This section defines how voting is performed, the
>       types of approvals, and which types of decision require which
>       type of approval.
>
>     * Voting
>
>         Decisions regarding the project are made by votes on the
>         primary project development mailing list
>         (general@hadoop.apache.org).  Where necessary, PMC voting may
>         take place on the private Hadoop PMC mailing list.  Votes are
>         clearly indicated by subject line starting with [VOTE].  Votes
>         may contain multiple items for approval and these should be
>         clearly separated. Voting is carried out by replying to the
>         vote mail. Voting may take four flavors
>
>         * +1 &quot;Yes,&quot; &quot;Agree,&quot; or &quot;the action
> should be
>             performed.&quot; In general, this vote also indicates a
> willingness
>             on the behalf of the voter in &quot;making it happen&quot;
>
>          * +0 This vote indicates a willingness for the action under
>             consideration to go ahead. The voter, however will not be
> able
>             to help.
>
>          * -0 This vote indicates that the voter does not, in
> general, agree with
>             the proposed action but is not concerned enough to
> prevent the
>             action going ahead.
>
>           * -1 This is a negative vote. On issues where consensus is
> required,
>             this vote counts as a <strong>veto</strong>. All vetoes
> must
>             contain an explanation of why the veto is appropriate.
> Vetoes with
>             no explanation are void. It may also be appropriate for a
> -1 vote
>             to include an alternative course of action.
>
>         All participants in the Hadoop project are encouraged to show
> their
>         agreement with or against a particular action by voting. For
> technical
>         decisions, only the votes of active committers are binding.
> Non binding
>         votes are still useful for those with binding votes to
> understand the
>         perception of an action in the wider Hadoop community. For
> PMC decisions,
>         only the votes of PMC members are binding.
>
>        Voting can also be applied to changes made to the Hadoop
> codebase. These
>         typically take the form of a veto (-1) in reply to the commit
> message
>         sent when the commit is made.
>
>     * Approvals
>
>         These are the types of approvals that can be sought.
> Different actions
>         require different types of approvals
>
>         * Lazy Consensus - Lazy consensus requires 3 binding +1 votes
>            and no binding vetoes.
>
>          * Lazy Majority - A lazy majority vote requires 3 binding +1
>             votes and more binding +1 votes than -1 votes.
>
>         * Lazy 2/3 Majority - Lazy 2/3 majority votes requires at
>            least 3 votes and twice as many +1 votes as -1 votes.
>
>     * Vetoes
>
>         A valid, binding veto cannot be overruled. If a veto is cast,
>         it must be accompanied by a valid reason explaining the
>         reasons for the veto. The validity of a veto, if challenged,
>         can be confirmed by anyone who has a binding vote. This does
>         not necessarily signify agreement with the veto - merely that
>         the veto is valid.
>
>        If you disagree with a valid veto, you must lobby the person
> casting
>         the veto to withdraw their veto. If a veto is not withdrawn,
> any action
>         that has been vetoed must be reversed in a timely manner.
>
>     * Actions
>
>         This section describes the various actions which are
> undertaken within
>         the project, the corresponding approval required for that
> action and
>         those who have binding votes over the action.
>
>          * Code Change
>
>             A change made to a codebase of the project and committed
>             by a committer. This includes source code, documentation,
> website
>             content, etc.
>
>             Lazy consensus of active committers.
>
>          * Release Plan
>
>             Defines the timetable and actions for a release. The plan
> also
>             nominates a Release Manager.
>
>             Lazy majority of active committers
>
>          * Product Release
>
>             When a release of one of the project's products is ready,
> a vote is
>             required to accept the release as an official release of
> the
>             project.
>
>            Lazy Majority of active PMC members
>
>           * Adoption of New Codebase
>
>             When the codebase for an existing, released product is to
> be
>             replaced with an alternative codebase. If such a vote
> fails to
>             gain approval, the existing code base will continue.
>
>            This also covers the creation of new sub-projects
>             within the project
>
>            Lazy 2/3 majority of PMC members
>
>           * New Committer
>
>             When a new committer is proposed for the project
>
>             Lazy consensus of active PMC members
>
>           * New PMC Member
>
>             When a committer is proposed for the PMC
>
>            Lazy consensus of active PMC members
>
>           * Committer Removal
>
>            When removal of commit privileges is sought.  Note: Such
>            actions will also be referred to the ASF board by the PMC
>            chair
>
>             Lazy 2/3 majority of active PMC members (excluding the
>             committer in question if a member of the PMC).
>
>           * PMC Member Removal
>
>             When removal of a PMC member is sought.  Note: Such
>             actions will also be referred to the ASF board by the PMC
>             chair.
>
>             Lazy 2/3 majority of active PMC members (excluding the
> member in question)
>
>           * Modifying Bylaws
>
>             Modifying this document.
>
>             Majority of active PMC members
>
>    * Voting Timeframes
>
>         Votes are open for a period of 7 days to allow all active
>         voters time to consider the vote. Votes relating to code
>         changes are not subject to a strict timetable but should be
>         made as timely as possible.
>


From general-return-2420-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 23 03:32:27 2010
Return-Path: <general-return-2420-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95944 invoked from network); 23 Nov 2010 03:32:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Nov 2010 03:32:27 -0000
Received: (qmail 26788 invoked by uid 500); 23 Nov 2010 03:32:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26521 invoked by uid 500); 23 Nov 2010 03:32:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26513 invoked by uid 99); 23 Nov 2010 03:32:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Nov 2010 03:32:53 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,MIME_QP_LONG_LINE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awakenrz@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Nov 2010 03:32:45 +0000
Received: by pwj9 with SMTP id 9so2230959pwj.35
        for <general@hadoop.apache.org>; Mon, 22 Nov 2010 19:32:25 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:subject:from:content-type
         :x-mailer:message-id:date:to:content-transfer-encoding:mime-version;
        bh=n5biZ05/6ECVWbk3SM4SP+7oYjc9hxWMABN3pK+VZSc=;
        b=CFwAtEp9pPQCdIteAKoBTyMweYds40j655tevbk/o0D9JCi4tR49Imc/IK++jg5XkK
         6Gyd4AxbEHKP8Q5mEbSEbKeE1abjj7Uba4m+888rEPJMv1NsdUGkXvcmoYMgReDlRcjb
         u9aqNlW19SwFxRYaJv79dGsACG2gWHICyIsWU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=subject:from:content-type:x-mailer:message-id:date:to
         :content-transfer-encoding:mime-version;
        b=bQOhiR6KUx6D0zD3ZFbKQTbKOO7YJEeyTczRF7+jXVqxYZmzn6HXv5CresWPFwlW2R
         lnTA5bgL+YX+hmJuO2p335BASlJqKRzwxkDvaGnldBDVY9m7lib0a7s384EnoVCxhtM0
         0u8hGUbrH4F7r8DZvZ7QlL60jyQ2C1URo1JPQ=
Received: by 10.142.166.20 with SMTP id o20mr5863959wfe.382.1290483145216;
        Mon, 22 Nov 2010 19:32:25 -0800 (PST)
Received: from [192.168.1.138] ([219.224.168.157])
        by mx.google.com with ESMTPS id w14sm7286628wfd.6.2010.11.22.19.32.22
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Mon, 22 Nov 2010 19:32:23 -0800 (PST)
Subject: How does Hadoop do sort(shuffle) after map exactly?
From: Ren Zuocheng <awakenrz@gmail.com>
Content-Type: text/plain;
	charset=us-ascii
X-Mailer: iPhone Mail (8B117)
Message-Id: <5E602823-3C67-425D-AE84-F141C5EBDBD8@gmail.com>
Date: Tue, 23 Nov 2010 11:29:59 +0800
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (iPhone Mail 8B117)

Hi,=20
I'm new to Hadoop and I want to know its implementation better. I was always=
 wondering after mapping, how each reduce task get its input. It is said in g=
oogle's paper and hadoop's documentation that a sort is done to aggregate th=
e same key of the map output. But there is no detailed explanation of how it=
 is implemented and my intuition is that perhaps a global hashing will work b=
etter than sorting. So I really want to know the details and see whether my i=
ntuition is right. If I can find out that in the source code, where should I=
 start with?

Sent from my iPhone=

From general-return-2421-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 23 06:28:16 2010
Return-Path: <general-return-2421-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81907 invoked from network); 23 Nov 2010 06:28:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Nov 2010 06:28:16 -0000
Received: (qmail 26973 invoked by uid 500); 23 Nov 2010 06:28:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26645 invoked by uid 500); 23 Nov 2010 06:28:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26629 invoked by uid 99); 23 Nov 2010 06:28:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Nov 2010 06:28:42 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Nov 2010 06:28:35 +0000
Received: by fxm2 with SMTP id 2so3668788fxm.35
        for <general@hadoop.apache.org>; Mon, 22 Nov 2010 22:28:15 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=Y54CurQJ82iqVwMCvkYysISKlQqBNNZK18Fl0GUWshk=;
        b=gJfSw5OQsIcubenm+nbH1aF/jN0+5juXNpD/TkgXjoUzjAMIMTdI+ltWu99YD7GPcI
         mlfbFKorfQ0OzoCazLOYQSr3WtTjaZgq/LBA4XNp7qtpMBny9AlJ/MWhGedofEmeZQnZ
         Sr/Xuts3VNFK1YLRuOq+dYqvggU+EFM0Z4hf4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=SoqH83wdeZJUhWbBks1/OZHmTv+mLAUvVTKn27vxm+RY7oME88eVRRDA4rxEjmQauX
         evl1T3+i9e7CK51mDXo72UuKgMmBHvrLO+F3xu3/N6/a5At2iXvC31AjcHrFZNpNLVya
         P4SWfUu8Jdb3jL4ggsbDqtOnGhj9+eCdsgbkk=
Received: by 10.223.83.133 with SMTP id f5mr4319639fal.29.1290493695191; Mon,
 22 Nov 2010 22:28:15 -0800 (PST)
MIME-Version: 1.0
Received: by 10.223.113.145 with HTTP; Mon, 22 Nov 2010 22:27:55 -0800 (PST)
In-Reply-To: <5E602823-3C67-425D-AE84-F141C5EBDBD8@gmail.com>
References: <5E602823-3C67-425D-AE84-F141C5EBDBD8@gmail.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Tue, 23 Nov 2010 11:57:55 +0530
Message-ID: <AANLkTinsAOyMqvODvsLyyhuM2kCsFa7Qn9KD9+9QMkxD@mail.gmail.com>
Subject: Re: How does Hadoop do sort(shuffle) after map exactly?
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

On Tue, Nov 23, 2010 at 8:59 AM, Ren Zuocheng <awakenrz@gmail.com> wrote:
> Hi,
> I'm new to Hadoop and I want to know its implementation better. I was alw=
ays wondering after mapping, how each reduce task get its input. It is said=
 in google's paper and hadoop's documentation that a sort is done to aggreg=
ate the same key of the map output. But there is no detailed explanation of=
 how it is implemented and my intuition is that perhaps a global hashing wi=
ll work better than sorting. So I really want to know the details and see w=
hether my intuition is right. If I can find out that in the source code, wh=
ere should I start with?
>

Read the MapTask and ReduceTask classes, and you'll find how the
Shuffle is implemented within.

The whole process is like this if am right:
MapTask                                                         || ReduceTa=
sk
Map Emit -> Partition -> (Combine) Sort -> Spill || Shuffle (Combine)
-> Merge -> Reduce

--=20
Harsh J
www.harshj.com

From general-return-2422-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 23 12:18:36 2010
Return-Path: <general-return-2422-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54128 invoked from network); 23 Nov 2010 12:18:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Nov 2010 12:18:35 -0000
Received: (qmail 83343 invoked by uid 500); 23 Nov 2010 12:19:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82537 invoked by uid 500); 23 Nov 2010 12:19:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82525 invoked by uid 99); 23 Nov 2010 12:19:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Nov 2010 12:19:02 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of agrawalprachi88@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Nov 2010 12:18:56 +0000
Received: by wyi11 with SMTP id 11so16655wyi.35
        for <general@hadoop.apache.org>; Tue, 23 Nov 2010 04:18:35 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=9cQieWjYvyZWZpcmU463c94Zz+pmBh4kCVNNp3sOdfA=;
        b=w/QEZRscD3zVCmYVXHVPcrhHWRQtsi0ng8wejuoofXi6QP8KN6i/5KfirBrRaD2qH2
         /wFypU3N4aae+r7XP85szyUvSwp7KSNUrAKYl8MlymDdMhXfTlyo1ZJhHnFfsx2MBIkc
         F2wrEYjAiwuEabyiGrSu/4USDL4YrsDQbdNyI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=AMrCMJ55t4irkqfU9mXV4MCSYTEPNHZtuRNUGGZFck0FxCzIMj/M+DoCWLHt8p2IVQ
         lnuH6oVqmdbmZMoD4J4eOdCsaAtLTye+NxaXb3xhRMZ5oN/pAecbW0LlDwOM/h3hNWz2
         KduoZTiVsAhMx39Db037KGARSByQYjxJpxONs=
MIME-Version: 1.0
Received: by 10.227.157.79 with SMTP id a15mr7553430wbx.208.1290514714489;
 Tue, 23 Nov 2010 04:18:34 -0800 (PST)
Received: by 10.227.145.207 with HTTP; Tue, 23 Nov 2010 04:18:34 -0800 (PST)
In-Reply-To: <AANLkTikfqsS9wk0gmRtbxPE5to3eJD+KvGPF3U2WKg7_@mail.gmail.com>
References: <AANLkTikfqsS9wk0gmRtbxPE5to3eJD+KvGPF3U2WKg7_@mail.gmail.com>
Date: Tue, 23 Nov 2010 17:48:34 +0530
Message-ID: <AANLkTi=4uTm8U9+aPnd2hyzPycoJX-TTDXYqwOqzBseL@mail.gmail.com>
Subject: how to upgrade hadoop
From: prachi agrawal <agrawalprachi88@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636458f4e2273e10495b75fe8

--001636458f4e2273e10495b75fe8
Content-Type: text/plain; charset=ISO-8859-1

how can we upgrade hadoop without any exception also describe upgrade
procedure concept
i tried to upgrade but got lots of exceptions
like:
inconsistance state exception,
org.apache.hadoop.security.AccessControlException
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
initialization failed.
org.apache.hadoop.hdfs.server.namenode.NameNode:
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
initialization failed

--001636458f4e2273e10495b75fe8--

From general-return-2423-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 23 12:31:02 2010
Return-Path: <general-return-2423-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63308 invoked from network); 23 Nov 2010 12:31:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Nov 2010 12:31:02 -0000
Received: (qmail 93945 invoked by uid 500); 23 Nov 2010 12:31:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93786 invoked by uid 500); 23 Nov 2010 12:31:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93778 invoked by uid 99); 23 Nov 2010 12:31:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Nov 2010 12:31:31 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of patodirahul@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Nov 2010 12:31:24 +0000
Received: by wyi11 with SMTP id 11so28244wyi.35
        for <general@hadoop.apache.org>; Tue, 23 Nov 2010 04:31:04 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=jSE08K8RVM6EeAUAaAyoW+dXCfv0yoTnntKWfsIx8qg=;
        b=O4mzsnk3hKLpAg4EQpltRNKJ7SuX7/o3FFyd0xP5rGycOuZ5tCxQLwOP1hRLWzM+pF
         pxU20tfIZz89N9J1twwRgqQkROrnK6OctdoSTXvk4/2rIhEpX9367pjNbDwjs1Au8VcJ
         YHt3AabYK+O6XmeAMk+ZLRCla9LGX7hzxyhvI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=gqtZ2IHxavLHWQiZPB+tEPHRCHeuiD3zxEM0LAFvRSyARHWGoWEQGB3L2eZAAY/H83
         vzmZXKrEqBtTvF0rnQvtId7FAc5A1ilc3UvLtux8PZ0rID8RjJZ40VkrQbxpMV+2ELCz
         UtzIYumMliCkTl3Wlyw1g7HXi2cG7pIiHsfFo=
MIME-Version: 1.0
Received: by 10.216.186.141 with SMTP id w13mr811320wem.91.1290515463388; Tue,
 23 Nov 2010 04:31:03 -0800 (PST)
Received: by 10.216.131.197 with HTTP; Tue, 23 Nov 2010 04:31:03 -0800 (PST)
In-Reply-To: <AANLkTi=4uTm8U9+aPnd2hyzPycoJX-TTDXYqwOqzBseL@mail.gmail.com>
References: <AANLkTikfqsS9wk0gmRtbxPE5to3eJD+KvGPF3U2WKg7_@mail.gmail.com>
	<AANLkTi=4uTm8U9+aPnd2hyzPycoJX-TTDXYqwOqzBseL@mail.gmail.com>
Date: Tue, 23 Nov 2010 18:01:03 +0530
Message-ID: <AANLkTimN50fX8Lg5KVXSZGqmi05yKvd8SfOXVj3DZZfK@mail.gmail.com>
Subject: Re: how to upgrade hadoop
From: rahul patodi <patodirahul@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163649a6bfc5bde10495b78b31
X-Virus-Checked: Checked by ClamAV on apache.org

--00163649a6bfc5bde10495b78b31
Content-Type: text/plain; charset=ISO-8859-1

I have upgraded hadoop from 0.18.X to 0.19.X then to 0.20.X successfully
please follow this link:

http://hadoop-tutorial.blogspot.com/2010/11/upgrade-hadoop.html

If you get any error please leave a comment


-Thanks and Regards,
Rahul Patodi
Associate Software Engineer,
Impetus Infotech (India) Private Limited,
www.impetus.com
Mob:09907074413

On Tue, Nov 23, 2010 at 5:48 PM, prachi agrawal
<agrawalprachi88@gmail.com>wrote:

> how can we upgrade hadoop without any exception also describe upgrade
> procedure concept
> i tried to upgrade but got lots of exceptions
> like:
> inconsistance state exception,
> org.apache.hadoop.security.AccessControlException
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
> initialization failed.
> org.apache.hadoop.hdfs.server.namenode.NameNode:
> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
> initialization failed
>



--

--00163649a6bfc5bde10495b78b31--

From general-return-2424-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 23 12:54:00 2010
Return-Path: <general-return-2424-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72719 invoked from network); 23 Nov 2010 12:53:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Nov 2010 12:53:59 -0000
Received: (qmail 14038 invoked by uid 500); 23 Nov 2010 12:54:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13743 invoked by uid 500); 23 Nov 2010 12:54:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 46669 invoked by uid 99); 23 Nov 2010 10:31:37 -0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of agrawalprachi88@gmail.com designates 74.125.82.48 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=lMYQXVbtK30dE4gPIb5wntweVZwd1/DpGDP+zcJuYVw=;
        b=lIRlrqL/cVDOc8nuVqaCyfguMXwvZBBQgyXizEp769IOVYxDUdMefgT2d3EtNV2dow
         N3tnWes91ROrryn5Oqbb/vmaMvRC6SqV62iPoo2Vp5LxLs41EfaI75iOJP68vA67/1C0
         GaIcFbXLNv6e+hcsMB+L5mw168ExacjTonYoQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=T7GKDZlr70cg9Vri0mqSV8JVyUiyFjJ67KmjlNbCe4/sLkEahCUKR0lZx01Qg5J1e+
         YsYwogNafV8ESyeqkkwE+hUDXPcftufl7HVTsfEriItNVk1yOah/KtYsXNANfx0O5bi0
         woxlK/oDlzQ1PWokrBoAWvv/+o7ISJwlD9kIY=
MIME-Version: 1.0
Date: Tue, 23 Nov 2010 16:01:08 +0530
Message-ID: <AANLkTikfqsS9wk0gmRtbxPE5to3eJD+KvGPF3U2WKg7_@mail.gmail.com>
Subject: how to upgrade hadoop
From: prachi agrawal <agrawalprachi88@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636458f4ef122b40495b5de19

--001636458f4ef122b40495b5de19
Content-Type: text/plain; charset=ISO-8859-1

how can we upgrade hadoop without any exception also describe upgrade
procedure concept
i tried to upgrade but got lots of exceptions
like:
inconsistance state exception,
org.apache.hadoop.security.AccessControlException
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
initialization failed.
org.apache.hadoop.hdfs.server.namenode.NameNode:
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
initialization failed

--001636458f4ef122b40495b5de19--

From general-return-2425-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 23 16:31:30 2010
Return-Path: <general-return-2425-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 92258 invoked from network); 23 Nov 2010 16:31:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Nov 2010 16:31:30 -0000
Received: (qmail 64440 invoked by uid 500); 23 Nov 2010 16:32:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 64263 invoked by uid 500); 23 Nov 2010 16:31:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 64255 invoked by uid 99); 23 Nov 2010 16:31:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Nov 2010 16:31:56 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of phaidinyak@local.com designates 70.183.28.5 as permitted sender)
Received: from [70.183.28.5] (HELO mail11.local.com) (70.183.28.5)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Nov 2010 16:31:50 +0000
Received: from IRV1HUBCAS01.eLiberation.com (10.1.190.40) by mail11.local.com
 (10.1.230.41) with Microsoft SMTP Server (TLS) id 8.3.83.0; Tue, 23 Nov 2010
 08:32:08 -0800
Received: from IRV1EXMB01.eLiberation.com ([fe80::895:3aed:8948:77ab]) by
 IRV1HUBCAS01.eLiberation.com ([fe80::7853:3c40:9449:94df%13]) with mapi; Tue,
 23 Nov 2010 08:31:28 -0800
From: Peter Haidinyak <phaidinyak@local.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 23 Nov 2010 08:31:28 -0800
Subject: ClassNotFoundException
Thread-Topic: ClassNotFoundException
Thread-Index: AcuI4vJ0EOl4rWscSaCG1uOPJHhAIwCRpcxg
Message-ID: <321C2E54215EEB41A581FDD9DAECBC5DFCB2AE2C@IRV1EXMB01.eLiberation.com>
References: <7866CBAB-0B74-4250-A890-D634BB0D1D14@gmail.com>
 <AANLkTinUmMt0xCHFMeoTboyBLgF3x5NOhU+ho=bLfHdg@mail.gmail.com>
In-Reply-To: <AANLkTinUmMt0xCHFMeoTboyBLgF3x5NOhU+ho=bLfHdg@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi,
  I just installed hadoop .21 (linux) and am trying to use the new API so I=
 copied the example code from the Job.java source...


    final Job job =3D new Job(new Configuration(true));
    job.setJarByClass(SearchLogSorter.class);

    job.setJobName("Search Log Sort");

when I compile and jar this and run it I get a=20

[hadoop@cala3stgiss01 hadoop]$ hadoop -jar jars/SearchLogSorter.jar a s
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoo=
p/mapreduce/Job
        at com.local.ctr.importer.SearchLogSorter.main(SearchLogSorter.java=
:96)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.Jo=
b
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
        ... 1 more

I executed 'hadoop classpath' and the 'hadoop-mapred-0.21.0.jar' is in the =
path.

Any ideas?

Also, will the jar be renamed to 'hadoop-mapreduce.*.jar in the future?

Thanks

-Pete

From general-return-2426-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 24 16:53:16 2010
Return-Path: <general-return-2426-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81992 invoked from network); 24 Nov 2010 16:53:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Nov 2010 16:53:15 -0000
Received: (qmail 78393 invoked by uid 500); 24 Nov 2010 16:53:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78197 invoked by uid 500); 24 Nov 2010 16:53:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78182 invoked by uid 99); 24 Nov 2010 16:53:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Nov 2010 16:53:44 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [98.139.91.220] (HELO nm21-vm0.bullet.mail.sp2.yahoo.com) (98.139.91.220)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 24 Nov 2010 16:53:35 +0000
Received: from [98.139.91.69] by nm21.bullet.mail.sp2.yahoo.com with NNFMP; 24 Nov 2010 16:53:14 -0000
Received: from [98.139.91.33] by tm9.bullet.mail.sp2.yahoo.com with NNFMP; 24 Nov 2010 16:53:14 -0000
Received: from [127.0.0.1] by omp1033.mail.sp2.yahoo.com with NNFMP; 24 Nov 2010 16:53:14 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 32177.46383.bm@omp1033.mail.sp2.yahoo.com
Received: (qmail 78860 invoked by uid 60001); 24 Nov 2010 16:53:02 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1290617582; bh=h+Q3rK/nTGKbEk8kNOuYWI8A4vuFjzRzw3ov56LmaS4=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=C8XexSHgPvEBZkxeLdcDothzDDGVUFXuqCt6nSjr+jacn9NSY6BzxpxsNZOHMufm3FaA/4A3msv08z4t3ry2x6DkilWgC8n8eRGEFtj1LR3dR+OnTmDW7ZLkNXcG7fOp2HrETbFgIS/MdG2SK/nbOnXXhorKI9+7E9DDgfWwZek=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=EVVHXjfOnTx5TVR0niI9EDaD18eRufMkhdnK3NN3lQuzykvzhBV648RBMcqtV7Qx5SZ1HfFSgxC8KnrdkVnz3rlmVOYNx3W/pt8vi+q/7+JYk3sV5GzdawaPEoj+UvMtLEyfGH5maxbutX2oTxr+GqsCD7DXmcXWOf+E/oFgNT0=;
Message-ID: <636142.67622.qm@web112113.mail.gq1.yahoo.com>
X-YMail-OSG: 3dS6KukVM1mbwqOSMTS37QnVACAtj5BAK8LKVBdZbzf.VGr
 fP78vCP4TMkofH_Vb8x_ZYzlPwSNw43ZaazJZRKsB0Uc2FoFPRmkFI5zX5yV
 QdEqkXf6knzCcZx5ggzPRklGRRmGbVUm8ym_3wJ5ZRla6RkPqmVMib.LYwKp
 kIFuFWcHeoLnR47zY.CxzF1AJWn_ij0xpPAMb1ulD.hzBslg.1uMgVmoaIJ5
 rmJmQ.RBvqlHJu.EmJhSBP6K3i3KBAEMFw4.azaieGgf23zsHdptqV_vQbCn
 x91fMb_2iPwPLsrt62TSRijrOLWWTV4QJ3HNWZMro4A8FH.IL0KkCKX4Svo_
 z.Z2lEAigVyM.ufMsyiDrOsO73KFqNC7vBDEszQFJc1eg
Received: from [85.1.185.210] by web112113.mail.gq1.yahoo.com via HTTP; Wed, 24 Nov 2010 08:53:01 PST
X-Mailer: YahooMailClassic/11.4.9 YahooMailWebService/0.8.107.285259
Date: Wed, 24 Nov 2010 08:53:01 -0800 (PST)
From: Grandl Robert <rgrandl@yahoo.com>
Subject: Re: Hadoop - how exactly is a slot defined
To: general@hadoop.apache.org
In-Reply-To: <AANLkTimad3DwTUfozirAzFFX_WEj5eXyKwDwsjB0j1Ju@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1656217391-1290617581=:67622"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-1656217391-1290617581=:67622
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Hi,
I am sorry bothering again about this subject, but still I am not very conv=
inced what Hadoop assumes a slot is. I understood it represent smth in term=
s of CPU/Memory, so you have to allocate corresponding numbers of map/reduc=
e slots based on your configurations.
BUT, I cannot understand yet, if Hadoop make any mapping between the concep=
t of slot and physical resources itself, or are just some numbers and you c=
an go over only with this numbers.=A0
I looked on the code, but I am not able to figure out if Hadoop really did =
some checking between number of slots and physical resources, or just is li=
mited by the 2 numbers(for maximum number of map slots and reduce slots) an=
d play with this numbers only. That means, the user should give more interp=
retation of what a slot really may be: (Only one slot per core, one slot pe=
r 512 MB, etc) when configure the number of map/reduce slots on his machine=
s.
Thanks in advance for any clue.
Cheers,Robert

--- On Mon, 11/22/10, Harsh J <qwertymaniac@gmail.com> wrote:

From: Harsh J <qwertymaniac@gmail.com>
Subject: Re: Hadoop - how exactly is a slot defined
To: general@hadoop.apache.org
Date: Monday, November 22, 2010, 6:52 PM

Hi,

On Mon, Nov 22, 2010 at 10:02 PM, Grandl Robert <rgrandl@yahoo.com> wrote:
> Hi all,
>
> I have troubles in understanding what exactly a slot is. Always we are ta=
lking about tasks assigned to slots, but I did not found anywhere what exac=
tly a slot is. I assume it represent some allocation of RAM memory as well =
as with some computation power.
>
> However, can somebody explain me what exactly a slot means (in terms of r=
esources allocated for a slot) and how this mapping(between slot and physic=
al resources) is done in Hadoop ? Or give me some hints about the files in =
the Hadoop=A0 where it may should be ?

A slot is of two types -- Map slot and Reduce slot. A slot represents
an ability to run one of these "Tasks" (map/reduce tasks) individually
at a point of time. Therefore, multiple slots on a TaskTracker means
multiple "Tasks" may execute in parallel.

Right now total slots in a TaskTracker is =3D=3D
mapred.tasktracker.map.tasks.maximum for Maps and
mapred.tasktracker.reduce.tasks.maximum for Reduces.

Hadoop is indeed trying to go towards the dynamic slot concept, which
could rely on the current resources available on a system, but work
for this is still in conceptual phases. TaskTrackers emit system
status (like CPU load, utilization, memory available/user, load
averages) in their heartbeats today (and is utilized by certain
schedulers, I think Capacity Scheduler uses it to determine stuff),
but the concept of slots is still fixed as a maximum to the above two
configurations on each TaskTracker.

For code on how slots are checked/utilized, see any Scheduler plugin's
code -- LimitTasksPerJobTaskScheduler, CapacityTaskScheduler for
example.

>
> Thanks a lot,
> Robert
>
>
>



--=20
Harsh J
www.harshj.com
=0A=0A=0A      
--0-1656217391-1290617581=:67622--

From general-return-2427-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 24 17:04:31 2010
Return-Path: <general-return-2427-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 90080 invoked from network); 24 Nov 2010 17:04:31 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Nov 2010 17:04:31 -0000
Received: (qmail 1217 invoked by uid 500); 24 Nov 2010 17:04:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1135 invoked by uid 500); 24 Nov 2010 17:04:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1127 invoked by uid 99); 24 Nov 2010 17:04:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Nov 2010 17:04:58 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=MIME_BASE64_BLANKS,MIME_BASE64_TEXT,RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 65.55.88.14 is neither permitted nor denied by domain of jon.creasy@announcemedia.com)
Received: from [65.55.88.14] (HELO TX2EHSOBE009.bigfish.com) (65.55.88.14)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Nov 2010 17:04:50 +0000
Received: from mail181-tx2-R.bigfish.com (10.9.14.251) by
 TX2EHSOBE009.bigfish.com (10.9.40.29) with Microsoft SMTP Server id
 14.1.225.8; Wed, 24 Nov 2010 17:04:29 +0000
Received: from mail181-tx2 (localhost.localdomain [127.0.0.1])	by
 mail181-tx2-R.bigfish.com (Postfix) with ESMTP id 48BED1C1849A	for
 <general@hadoop.apache.org>; Wed, 24 Nov 2010 17:04:29 +0000 (UTC)
X-SpamScore: -36
X-BigFish: VPS-36(zzbb2dK1411I1432N98dN14ffO1443N9371Pzz1202hzz8275bh8275dhz2ei2a8h691h668h67dh685h61h)
X-Spam-TCS-SCL: 0:0
X-Forefront-Antispam-Report: KIP:(null);UIP:(null);IPVD:NLI;H:VA3DIAHUB009.RED001.local;RD:smtp801.microsoftonline.com;EFVD:NLI
Received: from mail181-tx2 (localhost.localdomain [127.0.0.1]) by mail181-tx2
 (MessageSwitch) id 1290618268760859_24393; Wed, 24 Nov 2010 17:04:28 +0000
 (UTC)
Received: from TX2EHSMHS029.bigfish.com (unknown [10.9.14.251])	by
 mail181-tx2.bigfish.com (Postfix) with ESMTP id B6CC2144004F	for
 <general@hadoop.apache.org>; Wed, 24 Nov 2010 17:04:28 +0000 (UTC)
Received: from VA3DIAHUB009.RED001.local (65.55.171.153) by
 TX2EHSMHS029.bigfish.com (10.9.99.129) with Microsoft SMTP Server (TLS) id
 14.1.225.8; Wed, 24 Nov 2010 17:04:25 +0000
Received: from VA3DIAXVS401.RED001.local ([10.32.57.16]) by
 VA3DIAHUB009.RED001.local ([10.32.16.180]) with mapi; Wed, 24 Nov 2010
 09:04:22 -0800
From: Jonathan Creasy <jon.creasy@Announcemedia.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 24 Nov 2010 09:04:20 -0800
Subject: Re: Hadoop - how exactly is a slot defined
Thread-Topic: Hadoop - how exactly is a slot defined
Thread-Index: AcuL+aOoZAv9esjaQomPBeLjodDRCA==
Message-ID: <9A758C50-80BB-4AA9-819A-EB4756E8BF45@announcemedia.microsoftonline.com>
References: <636142.67622.qm@web112113.mail.gq1.yahoo.com>
In-Reply-To: <636142.67622.qm@web112113.mail.gq1.yahoo.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-OriginatorOrg: announcemedia.com

Um9iZXJ0LCANCg0KSGFkb29wIGlzIG5vdCBjdXJyZW50bHkgZG9pbmcgYW55IGR5bmFtaWMgZGV0
ZWN0aW9uIG9mIHJlc291cmNlcyB0byBkZXRlcm1pbmUgdGhlIG51bWJlciBvZiBzbG90cy4gSWYg
SSB0b2xkIEhhZG9vcCBpdCBjb3VsZCBydW4gMyw1ODcgbWFwIHRhc2tzLCBpdCBtaWdodCB3ZWxs
IHRyeSB0byBkbyBpdC4gDQoNCldlIHVzZSBzdGFuZGFyZHMgdG8gZGV0ZXJtaW5lIGhvdyBtYW55
IG1hcCBhbmQgcmVkdWNlIHRhc2tzIGEgbm9kZSBpcyBhbGxvd2VkOg0KDQpFYWNoIE1hcC9SZWR1
Y2UgVGFzayBpcyBnaXZlbjoNCjJHQiBvZiBSYW0NCjEgQ29yZQ0KNTBHQiBvZiB0bXAgZGlzayBz
cGFjZQ0KDQpUaGUgZm9ybXVsYSBmb3IgbWFwL3JlZHVjZSBzbG90cyBsb29rcyBzb21ldGhpbmcg
bGlrZSB0aGlzIGluIG91ciBlbnZpcm9ubWVudDoNCg0KRyA9IEdCIG9mIFJhbQ0KRCA9IERpc2sg
c3BhY2UgaW4gL3RtcA0KQyA9IGNvdW50IG9mIENQVSBjb3Jlcw0KDQpUaGUgbWluaW11bSBvZjog
DQooRy0yKS8yDQpELzUwDQpDLTENCg0KVGhlc2UgbnVtYmVycyBhcmVuJ3QgcHVibGlzaGVkIGFu
eXdoZXJlIGFuZCBtYXkgY29tcGxldGVseSBmbHkgaW4gdGhlIGZhY2Ugb2YgY29udmVudGlvbmFs
IHdpc2RvbSBidXQgaXQncyB3aGF0IHdlIGFyZSB1c2luZyBhbmQgc28gZmFyLCBzZWVtcyB0byB3
b3JrIGZvciB1cy4gDQoNCi1Kb25hdGhhbg0KDQoNCk9uIE5vdiAyNCwgMjAxMCwgYXQgMTA6NTMg
QU0sIEdyYW5kbCBSb2JlcnQgd3JvdGU6DQoNCj4gSGksDQo+IEkgYW0gc29ycnkgYm90aGVyaW5n
IGFnYWluIGFib3V0IHRoaXMgc3ViamVjdCwgYnV0IHN0aWxsIEkgYW0gbm90IHZlcnkgY29udmlu
Y2VkIHdoYXQgSGFkb29wIGFzc3VtZXMgYSBzbG90IGlzLiBJIHVuZGVyc3Rvb2QgaXQgcmVwcmVz
ZW50IHNtdGggaW4gdGVybXMgb2YgQ1BVL01lbW9yeSwgc28geW91IGhhdmUgdG8gYWxsb2NhdGUg
Y29ycmVzcG9uZGluZyBudW1iZXJzIG9mIG1hcC9yZWR1Y2Ugc2xvdHMgYmFzZWQgb24geW91ciBj
b25maWd1cmF0aW9ucy4NCj4gQlVULCBJIGNhbm5vdCB1bmRlcnN0YW5kIHlldCwgaWYgSGFkb29w
IG1ha2UgYW55IG1hcHBpbmcgYmV0d2VlbiB0aGUgY29uY2VwdCBvZiBzbG90IGFuZCBwaHlzaWNh
bCByZXNvdXJjZXMgaXRzZWxmLCBvciBhcmUganVzdCBzb21lIG51bWJlcnMgYW5kIHlvdSBjYW4g
Z28gb3ZlciBvbmx5IHdpdGggdGhpcyBudW1iZXJzLiANCj4gSSBsb29rZWQgb24gdGhlIGNvZGUs
IGJ1dCBJIGFtIG5vdCBhYmxlIHRvIGZpZ3VyZSBvdXQgaWYgSGFkb29wIHJlYWxseSBkaWQgc29t
ZSBjaGVja2luZyBiZXR3ZWVuIG51bWJlciBvZiBzbG90cyBhbmQgcGh5c2ljYWwgcmVzb3VyY2Vz
LCBvciBqdXN0IGlzIGxpbWl0ZWQgYnkgdGhlIDIgbnVtYmVycyhmb3IgbWF4aW11bSBudW1iZXIg
b2YgbWFwIHNsb3RzIGFuZCByZWR1Y2Ugc2xvdHMpIGFuZCBwbGF5IHdpdGggdGhpcyBudW1iZXJz
IG9ubHkuIFRoYXQgbWVhbnMsIHRoZSB1c2VyIHNob3VsZCBnaXZlIG1vcmUgaW50ZXJwcmV0YXRp
b24gb2Ygd2hhdCBhIHNsb3QgcmVhbGx5IG1heSBiZTogKE9ubHkgb25lIHNsb3QgcGVyIGNvcmUs
IG9uZSBzbG90IHBlciA1MTIgTUIsIGV0Yykgd2hlbiBjb25maWd1cmUgdGhlIG51bWJlciBvZiBt
YXAvcmVkdWNlIHNsb3RzIG9uIGhpcyBtYWNoaW5lcy4NCj4gVGhhbmtzIGluIGFkdmFuY2UgZm9y
IGFueSBjbHVlLg0KPiBDaGVlcnMsUm9iZXJ0DQo+IA0KPiAtLS0gT24gTW9uLCAxMS8yMi8xMCwg
SGFyc2ggSiA8cXdlcnR5bWFuaWFjQGdtYWlsLmNvbT4gd3JvdGU6DQo+IA0KPiBGcm9tOiBIYXJz
aCBKIDxxd2VydHltYW5pYWNAZ21haWwuY29tPg0KPiBTdWJqZWN0OiBSZTogSGFkb29wIC0gaG93
IGV4YWN0bHkgaXMgYSBzbG90IGRlZmluZWQNCj4gVG86IGdlbmVyYWxAaGFkb29wLmFwYWNoZS5v
cmcNCj4gRGF0ZTogTW9uZGF5LCBOb3ZlbWJlciAyMiwgMjAxMCwgNjo1MiBQTQ0KPiANCj4gSGks
DQo+IA0KPiBPbiBNb24sIE5vdiAyMiwgMjAxMCBhdCAxMDowMiBQTSwgR3JhbmRsIFJvYmVydCA8
cmdyYW5kbEB5YWhvby5jb20+IHdyb3RlOg0KPj4gSGkgYWxsLA0KPj4gDQo+PiBJIGhhdmUgdHJv
dWJsZXMgaW4gdW5kZXJzdGFuZGluZyB3aGF0IGV4YWN0bHkgYSBzbG90IGlzLiBBbHdheXMgd2Ug
YXJlIHRhbGtpbmcgYWJvdXQgdGFza3MgYXNzaWduZWQgdG8gc2xvdHMsIGJ1dCBJIGRpZCBub3Qg
Zm91bmQgYW55d2hlcmUgd2hhdCBleGFjdGx5IGEgc2xvdCBpcy4gSSBhc3N1bWUgaXQgcmVwcmVz
ZW50IHNvbWUgYWxsb2NhdGlvbiBvZiBSQU0gbWVtb3J5IGFzIHdlbGwgYXMgd2l0aCBzb21lIGNv
bXB1dGF0aW9uIHBvd2VyLg0KPj4gDQo+PiBIb3dldmVyLCBjYW4gc29tZWJvZHkgZXhwbGFpbiBt
ZSB3aGF0IGV4YWN0bHkgYSBzbG90IG1lYW5zIChpbiB0ZXJtcyBvZiByZXNvdXJjZXMgYWxsb2Nh
dGVkIGZvciBhIHNsb3QpIGFuZCBob3cgdGhpcyBtYXBwaW5nKGJldHdlZW4gc2xvdCBhbmQgcGh5
c2ljYWwgcmVzb3VyY2VzKSBpcyBkb25lIGluIEhhZG9vcCA/IE9yIGdpdmUgbWUgc29tZSBoaW50
cyBhYm91dCB0aGUgZmlsZXMgaW4gdGhlIEhhZG9vcCAgd2hlcmUgaXQgbWF5IHNob3VsZCBiZSA/
DQo+IA0KPiBBIHNsb3QgaXMgb2YgdHdvIHR5cGVzIC0tIE1hcCBzbG90IGFuZCBSZWR1Y2Ugc2xv
dC4gQSBzbG90IHJlcHJlc2VudHMNCj4gYW4gYWJpbGl0eSB0byBydW4gb25lIG9mIHRoZXNlICJU
YXNrcyIgKG1hcC9yZWR1Y2UgdGFza3MpIGluZGl2aWR1YWxseQ0KPiBhdCBhIHBvaW50IG9mIHRp
bWUuIFRoZXJlZm9yZSwgbXVsdGlwbGUgc2xvdHMgb24gYSBUYXNrVHJhY2tlciBtZWFucw0KPiBt
dWx0aXBsZSAiVGFza3MiIG1heSBleGVjdXRlIGluIHBhcmFsbGVsLg0KPiANCj4gUmlnaHQgbm93
IHRvdGFsIHNsb3RzIGluIGEgVGFza1RyYWNrZXIgaXMgPT0NCj4gbWFwcmVkLnRhc2t0cmFja2Vy
Lm1hcC50YXNrcy5tYXhpbXVtIGZvciBNYXBzIGFuZA0KPiBtYXByZWQudGFza3RyYWNrZXIucmVk
dWNlLnRhc2tzLm1heGltdW0gZm9yIFJlZHVjZXMuDQo+IA0KPiBIYWRvb3AgaXMgaW5kZWVkIHRy
eWluZyB0byBnbyB0b3dhcmRzIHRoZSBkeW5hbWljIHNsb3QgY29uY2VwdCwgd2hpY2gNCj4gY291
bGQgcmVseSBvbiB0aGUgY3VycmVudCByZXNvdXJjZXMgYXZhaWxhYmxlIG9uIGEgc3lzdGVtLCBi
dXQgd29yaw0KPiBmb3IgdGhpcyBpcyBzdGlsbCBpbiBjb25jZXB0dWFsIHBoYXNlcy4gVGFza1Ry
YWNrZXJzIGVtaXQgc3lzdGVtDQo+IHN0YXR1cyAobGlrZSBDUFUgbG9hZCwgdXRpbGl6YXRpb24s
IG1lbW9yeSBhdmFpbGFibGUvdXNlciwgbG9hZA0KPiBhdmVyYWdlcykgaW4gdGhlaXIgaGVhcnRi
ZWF0cyB0b2RheSAoYW5kIGlzIHV0aWxpemVkIGJ5IGNlcnRhaW4NCj4gc2NoZWR1bGVycywgSSB0
aGluayBDYXBhY2l0eSBTY2hlZHVsZXIgdXNlcyBpdCB0byBkZXRlcm1pbmUgc3R1ZmYpLA0KPiBi
dXQgdGhlIGNvbmNlcHQgb2Ygc2xvdHMgaXMgc3RpbGwgZml4ZWQgYXMgYSBtYXhpbXVtIHRvIHRo
ZSBhYm92ZSB0d28NCj4gY29uZmlndXJhdGlvbnMgb24gZWFjaCBUYXNrVHJhY2tlci4NCj4gDQo+
IEZvciBjb2RlIG9uIGhvdyBzbG90cyBhcmUgY2hlY2tlZC91dGlsaXplZCwgc2VlIGFueSBTY2hl
ZHVsZXIgcGx1Z2luJ3MNCj4gY29kZSAtLSBMaW1pdFRhc2tzUGVySm9iVGFza1NjaGVkdWxlciwg
Q2FwYWNpdHlUYXNrU2NoZWR1bGVyIGZvcg0KPiBleGFtcGxlLg0KPiANCj4+IA0KPj4gVGhhbmtz
IGEgbG90LA0KPj4gUm9iZXJ0DQo+PiANCj4+IA0KPj4gDQo+IA0KPiANCj4gDQo+IC0tIA0KPiBI
YXJzaCBKDQo+IHd3dy5oYXJzaGouY29tDQo+IA0KPiANCj4gDQoNCg==


From general-return-2428-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 24 17:22:14 2010
Return-Path: <general-return-2428-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 10151 invoked from network); 24 Nov 2010 17:22:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Nov 2010 17:22:14 -0000
Received: (qmail 41814 invoked by uid 500); 24 Nov 2010 17:22:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 41747 invoked by uid 500); 24 Nov 2010 17:22:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 41739 invoked by uid 99); 24 Nov 2010 17:22:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Nov 2010 17:22:44 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of phaidinyak@local.com designates 70.183.28.5 as permitted sender)
Received: from [70.183.28.5] (HELO mail11.local.com) (70.183.28.5)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Nov 2010 17:22:36 +0000
Received: from IRV1HUBCAS01.eLiberation.com (10.1.190.40) by mail11.local.com
 (10.1.230.41) with Microsoft SMTP Server (TLS) id 8.3.83.0; Wed, 24 Nov 2010
 09:22:52 -0800
Received: from IRV1EXMB01.eLiberation.com ([fe80::895:3aed:8948:77ab]) by
 IRV1HUBCAS01.eLiberation.com ([fe80::7853:3c40:9449:94df%13]) with mapi; Wed,
 24 Nov 2010 09:22:14 -0800
From: Peter Haidinyak <phaidinyak@local.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 24 Nov 2010 09:22:13 -0800
Subject: Setting up a three node cluster
Thread-Topic: Setting up a three node cluster
Thread-Index: AcuL/CJZ4kepYGaOTb6cR2v2br1bwg==
Message-ID: <321C2E54215EEB41A581FDD9DAECBC5DFCB2AED9@IRV1EXMB01.eLiberation.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_321C2E54215EEB41A581FDD9DAECBC5DFCB2AED9IRV1EXMB01eLibe_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_321C2E54215EEB41A581FDD9DAECBC5DFCB2AED9IRV1EXMB01eLibe_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi,
   I just inherited three Linux boxes and I need to setup a three node clus=
ter. I've  been Googling around looking for a 'Best Practice' on setting up=
 a three node cluster without success. Could one of y'all have suggestions =
(with configuration file setting) on how to do this?

Thanks

-Pete

--_000_321C2E54215EEB41A581FDD9DAECBC5DFCB2AED9IRV1EXMB01eLibe_--

From general-return-2429-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 24 17:33:03 2010
Return-Path: <general-return-2429-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17199 invoked from network); 24 Nov 2010 17:33:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Nov 2010 17:33:03 -0000
Received: (qmail 62783 invoked by uid 500); 24 Nov 2010 17:33:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62680 invoked by uid 500); 24 Nov 2010 17:33:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62672 invoked by uid 99); 24 Nov 2010 17:33:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Nov 2010 17:33:34 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Nov 2010 17:33:29 +0000
Received: by fxm2 with SMTP id 2so5371792fxm.35
        for <general@hadoop.apache.org>; Wed, 24 Nov 2010 09:33:07 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=3QAvTEQakhFMKAgqNj327Y5GAcgjuCEyl5bgmy4da7k=;
        b=V1tt88yh8bTm7rnHhCsUQ4S6KKbTtJFQ1BWmNSHCB/XVQCtqSc3D2rOALUdgjzlyED
         E36otG7w1IG94/eN1/+ipt4Ts5y1NNhHjoLiJYemsquqTLb402Ya6H+Fsz/d6XLSIf+3
         bHOWgQCiVL9aGSmDrHo/zCsm9RQ6JbYkENJy0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=XgrsJ8lXDxuS5hwHudVx+Oc6ZqYioRZujXpK2Ohunz3SKPHlOJEe+EItj9V+33Lcu+
         TgkYbQS2vu+TpdOGiHIlxias82tiSTOQfOIac0OKNkfEQ63GqHedOGb/HKhg1+VQIT3X
         J0Rr1YuAy7isa1veSBsOzXKdG16wDsUTtdhdc=
Received: by 10.223.89.142 with SMTP id e14mr7499948fam.143.1290619987645;
 Wed, 24 Nov 2010 09:33:07 -0800 (PST)
MIME-Version: 1.0
Received: by 10.223.113.145 with HTTP; Wed, 24 Nov 2010 09:32:47 -0800 (PST)
In-Reply-To: <636142.67622.qm@web112113.mail.gq1.yahoo.com>
References: <AANLkTimad3DwTUfozirAzFFX_WEj5eXyKwDwsjB0j1Ju@mail.gmail.com> <636142.67622.qm@web112113.mail.gq1.yahoo.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Wed, 24 Nov 2010 23:02:47 +0530
Message-ID: <AANLkTikUJkuoA5xhWiN9jTthiBgJYTutgqHTTDiQBVRz@mail.gmail.com>
Subject: Re: Hadoop - how exactly is a slot defined
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi,

On Wed, Nov 24, 2010 at 10:23 PM, Grandl Robert <rgrandl@yahoo.com> wrote:
> Hi,
> I am sorry bothering again about this subject, but still I am not very co=
nvinced what Hadoop assumes a slot is. I understood it represent smth in te=
rms of CPU/Memory, so you have to allocate corresponding numbers of map/red=
uce slots based on your configurations.
> BUT, I cannot understand yet, if Hadoop make any mapping between the conc=
ept of slot and physical resources itself, or are just some numbers and you=
 can go over only with this numbers.

The slot amount is the user's homework for now.

> I looked on the code, but I am not able to figure out if Hadoop really di=
d some checking between number of slots and physical resources, or just is =
limited by the 2 numbers(for maximum number of map slots and reduce slots) =
and play with this numbers only. That means, the user should give more inte=
rpretation of what a slot really may be: (Only one slot per core, one slot =
per 512 MB, etc) when configure the number of map/reduce slots on his machi=
nes.

Yes, Hadoop does not dynamically detect any such thing yet. The setup
is ignorant to a machine's hardware and blindly relies on the
configurations passed at start up.

I usually set M =3D No. of CPUs + 1, and R =3D Prime nearest No. of CPUs.
But needs may vary depending on the nature of jobs it is going to
perform; sometimes you may need lesser CPU but more Memory/Task, so
configure based on your application knowledge.

--=20
Harsh J
www.harshj.com

From general-return-2430-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 24 17:33:13 2010
Return-Path: <general-return-2430-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17328 invoked from network); 24 Nov 2010 17:33:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Nov 2010 17:33:13 -0000
Received: (qmail 64055 invoked by uid 500); 24 Nov 2010 17:33:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 64007 invoked by uid 500); 24 Nov 2010 17:33:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63999 invoked by uid 99); 24 Nov 2010 17:33:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Nov 2010 17:33:43 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=MIME_BASE64_BLANKS,MIME_BASE64_TEXT,RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 65.55.88.14 is neither permitted nor denied by domain of jon.creasy@announcemedia.com)
Received: from [65.55.88.14] (HELO TX2EHSOBE007.bigfish.com) (65.55.88.14)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Nov 2010 17:33:37 +0000
Received: from mail152-tx2-R.bigfish.com (10.9.14.243) by
 TX2EHSOBE007.bigfish.com (10.9.40.27) with Microsoft SMTP Server id
 14.1.225.8; Wed, 24 Nov 2010 17:33:01 +0000
Received: from mail152-tx2 (localhost.localdomain [127.0.0.1])	by
 mail152-tx2-R.bigfish.com (Postfix) with ESMTP id 71A8C30824C	for
 <general@hadoop.apache.org>; Wed, 24 Nov 2010 17:32:45 +0000 (UTC)
X-SpamScore: -19
X-BigFish: VPS-19(zz1432N98dN9371Pzz1202hzz8275bhz2ei2a8h691h668h67dh685h61h)
X-Spam-TCS-SCL: 0:0
X-Forefront-Antispam-Report: KIP:(null);UIP:(null);IPVD:NLI;H:VA3DIAHUB031.RED001.local;RD:smtp801.microsoftonline.com;EFVD:NLI
Received: from mail152-tx2 (localhost.localdomain [127.0.0.1]) by mail152-tx2
 (MessageSwitch) id 1290619965164516_26052; Wed, 24 Nov 2010 17:32:45 +0000
 (UTC)
Received: from TX2EHSMHS027.bigfish.com (unknown [10.9.14.252])	by
 mail152-tx2.bigfish.com (Postfix) with ESMTP id 2655117D8054	for
 <general@hadoop.apache.org>; Wed, 24 Nov 2010 17:32:45 +0000 (UTC)
Received: from VA3DIAHUB031.RED001.local (65.55.171.153) by
 TX2EHSMHS027.bigfish.com (10.9.99.127) with Microsoft SMTP Server (TLS) id
 14.1.225.8; Wed, 24 Nov 2010 17:32:32 +0000
Received: from VA3DIAXVS401.RED001.local ([10.32.57.16]) by
 VA3DIAHUB031.RED001.local ([10.32.21.31]) with mapi; Wed, 24 Nov 2010
 09:32:07 -0800
From: Jonathan Creasy <jon.creasy@Announcemedia.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 24 Nov 2010 09:31:22 -0800
Subject: Re: Setting up a three node cluster
Thread-Topic: Setting up a three node cluster
Thread-Index: AcuL/YQHVVdS9GAhShiBIX7DtK8Y5A==
Message-ID: <11ED1E76-C613-4C3C-A0AF-80CAFFE5D1B7@announcemedia.microsoftonline.com>
References: <321C2E54215EEB41A581FDD9DAECBC5DFCB2AED9@IRV1EXMB01.eLiberation.com>
In-Reply-To: <321C2E54215EEB41A581FDD9DAECBC5DFCB2AED9@IRV1EXMB01.eLiberation.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-OriginatorOrg: announcemedia.com

T24gTm92IDI0LCAyMDEwLCBhdCAxMToyMiBBTSwgUGV0ZXIgSGFpZGlueWFrIHdyb3RlOg0KDQo+
IEhpLA0KPiAgIEkganVzdCBpbmhlcml0ZWQgdGhyZWUgTGludXggYm94ZXMgYW5kIEkgbmVlZCB0
byBzZXR1cCBhIHRocmVlIG5vZGUgY2x1c3Rlci4gSSd2ZSAgYmVlbiBHb29nbGluZyBhcm91bmQg
bG9va2luZyBmb3IgYSAnQmVzdCBQcmFjdGljZScgb24gc2V0dGluZyB1cCBhIHRocmVlIG5vZGUg
Y2x1c3RlciB3aXRob3V0IHN1Y2Nlc3MuIENvdWxkIG9uZSBvZiB5J2FsbCBoYXZlIHN1Z2dlc3Rp
b25zICh3aXRoIGNvbmZpZ3VyYXRpb24gZmlsZSBzZXR0aW5nKSBvbiBob3cgdG8gZG8gdGhpcz8N
Cj4gDQo+IFRoYW5rcw0KPiANCj4gLVBldGUNCg0KDQpZb3Ugc2hvdWxkIGJlIGFibGUgdG8gZm9s
bG93IHRoZSBkaXJlY3Rpb25zIGhlcmU6DQoNCmh0dHBzOi8vd2lraS5jbG91ZGVyYS5jb20vZGlz
cGxheS9ET0MvSGFkb29wK0RlcGxveW1lbnQrJTI4Q0RIMyUyOStvbithK0NsdXN0ZXINCg0KSSB3
b3VsZCBzZXR1cCBvbmUgbm9kZSBhcyBhIG5hbWVub2RlL2pvYnRyYWNrZXIvc2Vjb25kYXJ5bmFt
ZW5vZGUgYW5kIHRoZSBvdGhlciB0d28gYXMgdGFza3RyYWNrZXIvZGF0YW5vZGVzLg0KDQpPYnZp
b3VzbHkgYXMgdGhlIG51bWJlciBvZiBub2RlcyBhcHByb2FjaGVkIGFib3V0IGZpdmUsIEkgd291
bGQgc3BsaXQgdGhlIGZpcnN0IGJveCB0byBuYW1lbm9kZSBhbmQgam9idHJhY2tlcitzbm4uIA0K
DQotSm9uYXRoYW4=


From general-return-2431-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 24 17:36:00 2010
Return-Path: <general-return-2431-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 18662 invoked from network); 24 Nov 2010 17:36:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Nov 2010 17:36:00 -0000
Received: (qmail 68720 invoked by uid 500); 24 Nov 2010 17:36:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68641 invoked by uid 500); 24 Nov 2010 17:36:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68633 invoked by uid 99); 24 Nov 2010 17:36:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Nov 2010 17:36:30 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of patodirahul@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Nov 2010 17:36:26 +0000
Received: by wyf28 with SMTP id 28so445459wyf.35
        for <general@hadoop.apache.org>; Wed, 24 Nov 2010 09:36:05 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=bu3i25ToIExklkvwF4PD1jME+DsqueLHK6gAEp6RziA=;
        b=PGWnFAKG3R9tKKC8v6iUuyU3ZD2MHA2D+cpXAikRcLIMcSNcYkhR06iyoe1JUP+zfE
         1haaGdqQBDSq1yDOm3dRo3JNxlf+23yRh+YEUKnQ381BdqC0fJC6MUKvcuNESwXrznw8
         PS/lyYE+y/wyphbFhtdRhRX1/fz7awsKFMUXg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=qdBBkl36CrkMOmed9NDmraBX02givW7mPcLgcBPc5stlY0dTVQeu8wBC85wt9aqXF8
         6iXiLlecn1NtE33jxptNmFJqQgMKnuOB2SX/P1KD7pkqwoP+3ParpY3Y7SSKzlkoWQ4N
         vrE3IwCE/wjyqUYQsNbPvv81QcpmtM3q3ksfI=
MIME-Version: 1.0
Received: by 10.216.47.71 with SMTP id s49mr2649670web.106.1290620164222; Wed,
 24 Nov 2010 09:36:04 -0800 (PST)
Received: by 10.216.131.197 with HTTP; Wed, 24 Nov 2010 09:36:04 -0800 (PST)
In-Reply-To: <321C2E54215EEB41A581FDD9DAECBC5DFCB2AED9@IRV1EXMB01.eLiberation.com>
References: <321C2E54215EEB41A581FDD9DAECBC5DFCB2AED9@IRV1EXMB01.eLiberation.com>
Date: Wed, 24 Nov 2010 23:06:04 +0530
Message-ID: <AANLkTinMkBHbhf5rjV0+u8c4S+a1c+JHaRQqXYC8TB+b@mail.gmail.com>
Subject: Re: Setting up a three node cluster
From: rahul patodi <patodirahul@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f19a0e6dba790495cfec79

--001485f19a0e6dba790495cfec79
Content-Type: text/plain; charset=ISO-8859-1

Hi peter,
I have created a blog for this, you can refer it:
http://hadoop-tutorial.blogspot.com/2010/11/running-hadoop-in-distributed-mode.html
in this blog all the steps are well tested,
in case if you get any problem you can post comment, i would love to solve
all the problems.

-- 
-Thanks and Regards,
Rahul Patodi
Associate Software Engineer,
Impetus Infotech (India) Private Limited,
www.impetus.com
Mob:09907074413


On Wed, Nov 24, 2010 at 10:52 PM, Peter Haidinyak <phaidinyak@local.com>wrote:

> Hi,
>   I just inherited three Linux boxes and I need to setup a three node
> cluster. I've  been Googling around looking for a 'Best Practice' on setting
> up a three node cluster without success. Could one of y'all have suggestions
> (with configuration file setting) on how to do this?
>
> Thanks
>
> -Pete
>

--001485f19a0e6dba790495cfec79--

From general-return-2432-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 24 17:38:30 2010
Return-Path: <general-return-2432-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19567 invoked from network); 24 Nov 2010 17:38:29 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Nov 2010 17:38:29 -0000
Received: (qmail 71795 invoked by uid 500); 24 Nov 2010 17:39:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71611 invoked by uid 500); 24 Nov 2010 17:39:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71603 invoked by uid 99); 24 Nov 2010 17:39:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Nov 2010 17:39:00 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of patodirahul@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Nov 2010 17:38:54 +0000
Received: by wyf28 with SMTP id 28so447595wyf.35
        for <general@hadoop.apache.org>; Wed, 24 Nov 2010 09:38:34 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=LH/gtpE8ZWSaB17knKTOCI7rj/4Vyv/6hMOFSCR40So=;
        b=o+5QGbjF+IHvhzFDiHb5JVuKes3sb+J/KfDNlCznhd55KgxiHPGN3OUCBFYHKmmrYh
         RL5xXJobKYY2ltfCrA+lOx9w70grBapoaBog11pqfxboWDhxOuaNVo02dCqXmaBfQbjg
         pCdjxA3gzG7O904RLkewMqWNy8hYgguZDMtwU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=lQxDC0/6b8vcUE5NL9yj3mwV0m2XksUUalu2ROG0Ze1OhFxevWRljvsOoTjobzmR19
         yPvlGxQbvFUUMheIQO+CKZ+WpfL1gMzBN9Yyogv2RwdYrxW0OkkCK7QEic0yn0QL/J/5
         +SF5mSnzG7iLgg/LvHBGwy2i710SF3i2GUHlE=
MIME-Version: 1.0
Received: by 10.216.186.141 with SMTP id w13mr2642831wem.91.1290620313413;
 Wed, 24 Nov 2010 09:38:33 -0800 (PST)
Received: by 10.216.131.197 with HTTP; Wed, 24 Nov 2010 09:38:33 -0800 (PST)
In-Reply-To: <AANLkTinMkBHbhf5rjV0+u8c4S+a1c+JHaRQqXYC8TB+b@mail.gmail.com>
References: <321C2E54215EEB41A581FDD9DAECBC5DFCB2AED9@IRV1EXMB01.eLiberation.com>
	<AANLkTinMkBHbhf5rjV0+u8c4S+a1c+JHaRQqXYC8TB+b@mail.gmail.com>
Date: Wed, 24 Nov 2010 23:08:33 +0530
Message-ID: <AANLkTinLATXn1tODjBuSBpxLxZuCC-ovW9FLP+H0W3oi@mail.gmail.com>
Subject: Re: Setting up a three node cluster
From: rahul patodi <patodirahul@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163649a6bf52298e0495cff55f
X-Virus-Checked: Checked by ClamAV on apache.org

--00163649a6bf52298e0495cff55f
Content-Type: text/plain; charset=ISO-8859-1

if you want to setup cloudera hadoop bundle, you can refer
http://cloudera-tutorial.blogspot.com/2010/11/running-cloudera-in-distributed-mode.html


On Wed, Nov 24, 2010 at 11:06 PM, rahul patodi <patodirahul@gmail.com>wrote:

> Hi peter,
> I have created a blog for this, you can refer it:
>
> http://hadoop-tutorial.blogspot.com/2010/11/running-hadoop-in-distributed-mode.html
> in this blog all the steps are well tested,
> in case if you get any problem you can post comment, i would love to solve
> all the problems.
>
> --
> -Thanks and Regards,
> Rahul Patodi
> Associate Software Engineer,
> Impetus Infotech (India) Private Limited,
> www.impetus.com
> Mob:09907074413
>
>
>
> On Wed, Nov 24, 2010 at 10:52 PM, Peter Haidinyak <phaidinyak@local.com>wrote:
>
>> Hi,
>>   I just inherited three Linux boxes and I need to setup a three node
>> cluster. I've  been Googling around looking for a 'Best Practice' on setting
>> up a three node cluster without success. Could one of y'all have suggestions
>> (with configuration file setting) on how to do this?
>>
>> Thanks
>>
>> -Pete
>>
>
>
>
>
>


-- 
-Thanks and Regards,
Rahul Patodi
Associate Software Engineer,
Impetus Infotech (India) Private Limited,
www.impetus.com
Mob:09907074413

--00163649a6bf52298e0495cff55f--

From general-return-2433-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Nov 24 19:17:30 2010
Return-Path: <general-return-2433-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95379 invoked from network); 24 Nov 2010 19:17:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Nov 2010 19:17:30 -0000
Received: (qmail 49705 invoked by uid 500); 24 Nov 2010 19:18:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49492 invoked by uid 500); 24 Nov 2010 19:18:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49484 invoked by uid 99); 24 Nov 2010 19:18:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Nov 2010 19:18:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 24 Nov 2010 19:18:00 +0000
Received: (qmail 95324 invoked by uid 99); 24 Nov 2010 19:17:07 -0000
Received: from localhost.apache.org (HELO mail-bw0-f48.google.com) (127.0.0.1)
  (smtp-auth username phunt, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Nov 2010 19:17:07 +0000
Received: by bwz9 with SMTP id 9so157992bwz.35
        for <multiple recipients>; Wed, 24 Nov 2010 11:17:37 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.122.8 with SMTP id j8mr9313159bkr.135.1290626256770; Wed,
 24 Nov 2010 11:17:36 -0800 (PST)
Received: by 10.204.48.25 with HTTP; Wed, 24 Nov 2010 11:17:35 -0800 (PST)
Date: Wed, 24 Nov 2010 11:17:35 -0800
Message-ID: <AANLkTinmunPYTPuC7LvVHi2h67phvakUsnjcv6Pt49UA@mail.gmail.com>
Subject: FYI: ZooKeeper has changed mailing list names
From: Patrick Hunt <phunt@apache.org>
To: dev@zookeeper.apache.org, user@zookeeper.apache.org, 
	general@hadoop.apache.org, zookeeper-dev@hadoop.apache.org, 
	zookeeper-user <zookeeper-user@hadoop.apache.org>
Content-Type: text/plain; charset=ISO-8859-1

Please note, as part of the move to TLP ZooKeeper's mailing lists have
changed. If you were a subscriber to one of the old lists you have
been migrated automatically to the new (no changes needed on your
part, please start using the new names when sending emails).

The developer and user lists are now respectively:

dev@zookeeper.apache.org
user@zookeeper.apache.org

Let me know if there are any questions,

Patrick

From general-return-2434-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 25 00:00:02 2010
Return-Path: <general-return-2434-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30520 invoked from network); 25 Nov 2010 00:00:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Nov 2010 00:00:00 -0000
Received: (qmail 97491 invoked by uid 500); 25 Nov 2010 00:00:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97415 invoked by uid 500); 25 Nov 2010 00:00:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97407 invoked by uid 99); 25 Nov 2010 00:00:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Nov 2010 00:00:28 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of phaidinyak@local.com designates 70.183.28.5 as permitted sender)
Received: from [70.183.28.5] (HELO mail11.local.com) (70.183.28.5)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Nov 2010 00:00:23 +0000
Received: from IRV1HUBCAS01.eLiberation.com (10.1.190.40) by mail11.local.com
 (10.1.230.41) with Microsoft SMTP Server (TLS) id 8.3.83.0; Wed, 24 Nov 2010
 16:00:34 -0800
Received: from IRV1EXMB01.eLiberation.com ([fe80::895:3aed:8948:77ab]) by
 IRV1HUBCAS01.eLiberation.com ([fe80::7853:3c40:9449:94df%13]) with mapi; Wed,
 24 Nov 2010 15:59:57 -0800
From: Peter Haidinyak <phaidinyak@local.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 24 Nov 2010 15:59:55 -0800
Subject: java.util.zip.ZipException: error in opening zip file
Thread-Topic: java.util.zip.ZipException: error in opening zip file
Thread-Index: AcuMM7F9YvAys4GFSjuC7+MitDKC9g==
Message-ID: <321C2E54215EEB41A581FDD9DAECBC5DFCB2AF57@IRV1EXMB01.eLiberation.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_321C2E54215EEB41A581FDD9DAECBC5DFCB2AF57IRV1EXMB01eLibe_"
MIME-Version: 1.0

--_000_321C2E54215EEB41A581FDD9DAECBC5DFCB2AF57IRV1EXMB01eLibe_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi,
  Well, I got my three servers up running hadoop version 0.20.2 with one of=
 the servers being the NameNode/JobTracker and the other two being data nod=
es. The problems is when I try to run the wordcount example I get the follo=
wing exception. I hit <ctrl>c after a second since it takes awhile to timeo=
ut.
Does anyone have an idea on this?

Thanks

-Pete

[hadoop@caiss01a hadoop]$ hadoop jar hadoop-0.20.2-examples.jar wordcount /=
input_logs/output.txt /tmp/result.out
10/11/24 15:58:09 INFO input.FileInputFormat: Total input paths to process =
: 1
10/11/24 15:58:09 INFO mapred.JobClient: Running job: job_201011241548_0004
10/11/24 15:58:10 INFO mapred.JobClient:  map 0% reduce 0%
10/11/24 15:58:13 INFO mapred.JobClient: Task Id : attempt_201011241548_000=
4_m_000009_0, Status : FAILED
Error initializing attempt_201011241548_0004_m_000009_0:
java.util.zip.ZipException: error in opening zip file
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.<init>(ZipFile.java:114)
        at java.util.jar.JarFile.<init>(JarFile.java:133)
        at java.util.jar.JarFile.<init>(JarFile.java:97)
        at org.apache.hadoop.util.RunJar.unJar(RunJar.java:36)
        at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.jav=
a:815)
        at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.ja=
va:1664)
        at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.jav=
a:97)
        at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracke=
r.java:1629)

10/11/24 15:58:13 WARN mapred.JobClient: Error reading task outputhttp://ca=
iss01b.epilotcolo.eliberation.com:50060/tasklog?plaintext=3Dtrue&taskid=3Da=
ttempt_201011241548_0004_m_000009_0&filter=3Dstdout
10/11/24 15:58:13 WARN mapred.JobClient: Error reading task outputhttp://ca=
iss01b.epilotcolo.eliberation.com:50060/tasklog?plaintext=3Dtrue&taskid=3Da=
ttempt_201011241548_0004_m_000009_0&filter=3Dstderr
10/11/24 15:58:16 INFO mapred.JobClient: Task Id : attempt_201011241548_000=
4_m_000009_1, Status : FAILED
Error initializing attempt_201011241548_0004_m_000009_1:
java.util.zip.ZipException: error in opening zip file
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.<init>(ZipFile.java:114)
        at java.util.jar.JarFile.<init>(JarFile.java:133)
        at java.util.jar.JarFile.<init>(JarFile.java:97)
        at org.apache.hadoop.util.RunJar.unJar(RunJar.java:36)
        at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.jav=
a:815)
        at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.ja=
va:1664)
        at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.jav=
a:97)
        at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracke=
r.java:1629)

10/11/24 15:58:16 WARN mapred.JobClient: Error reading task outputhttp://ca=
iss01b.epilotcolo.eliberation.com:50060/tasklog?plaintext=3Dtrue&taskid=3Da=
ttempt_201011241548_0004_m_000009_1&filter=3Dstdout
10/11/24 15:58:16 WARN mapred.JobClient: Error reading task outputhttp://ca=
iss01b.epilotcolo.eliberation.com:50060/tasklog?plaintext=3Dtrue&taskid=3Da=
ttempt_201011241548_0004_m_000009_1&filter=3Dstderr
[hadoop@caiss01a hadoop]$

--_000_321C2E54215EEB41A581FDD9DAECBC5DFCB2AF57IRV1EXMB01eLibe_--

From general-return-2435-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 25 12:50:02 2010
Return-Path: <general-return-2435-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39362 invoked from network); 25 Nov 2010 12:50:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Nov 2010 12:50:02 -0000
Received: (qmail 6229 invoked by uid 500); 25 Nov 2010 12:50:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5974 invoked by uid 500); 25 Nov 2010 12:50:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5960 invoked by uid 99); 25 Nov 2010 12:50:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Nov 2010 12:50:00 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [98.139.91.232] (HELO nm27-vm0.bullet.mail.sp2.yahoo.com) (98.139.91.232)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 25 Nov 2010 12:49:53 +0000
Received: from [98.139.91.63] by nm27.bullet.mail.sp2.yahoo.com with NNFMP; 25 Nov 2010 12:49:32 -0000
Received: from [98.139.91.40] by tm3.bullet.mail.sp2.yahoo.com with NNFMP; 25 Nov 2010 12:49:32 -0000
Received: from [127.0.0.1] by omp1040.mail.sp2.yahoo.com with NNFMP; 25 Nov 2010 12:49:32 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 298863.58551.bm@omp1040.mail.sp2.yahoo.com
Received: (qmail 76672 invoked by uid 60001); 25 Nov 2010 12:42:51 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1290688971; bh=htP/cJLNHXIonInB8WulHo3L/ZV5jE+1OJqGxuo74A8=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=r/1dFwiVqFRa4c58Whnr/Ftis6sb44L897K2F3j7VETsv2j1qii1qifSrsIWbdLVDqMAegfWVO7GhWNb+nEEaFYCfUiP08vp/63sSv48DCjFtT3Y3RmoRYO/oOJueMQnUixEwp+zX95Ch5Ep6f1mCuTMqCqdviQJMe1NVGPg4qk=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=gntrQriX83asXH9pJ0aQBp/rAnEkXtZEHRgdPxFU3koSnS3qDLRco8YqZsL1S1EH2wCfq+dAx+CBE9zaMOCY6d+lnu9NKeD6Xv9AnUM0keTZxxH3omkXYl4H0VsE9tESyjmectLzMMgzJNV9lVDmYDJJsRRRyzquYyqM52AXZXk=;
Message-ID: <349178.76184.qm@web112116.mail.gq1.yahoo.com>
X-YMail-OSG: wc1T8PUVM1mixOPHBiUZ8CY16ibVLA7qyfOEbId2OW5cZmT
 9kJdOwHpfZDLONnUKr93Vb1jzCDe51oxJjoOn3L.IeU8ta02j2x7DXemJakr
 yHZJk4VxRHXKrgBTjVOew80SOAaBZbQ1THYI1Ea1wa0TnDnKlKT.7m4B3qZs
 yeICjxjz8DzUgPtbg4SAG5U2P2jHJziQlTb6GY1oaIZy2S2AU6nFZ0w0g5bZ
 sVgcgFiNrGrtqNJxSHkUJiZM2nCwI2rC3AQDaKAlJWGKI7RMC3QKRupu2kiS
 .lMr6hEEOUCUf8lrefbnCryDcXjF6nUm7M7NCu9J4f_fWt.JEZIrW0HoXmnd
 tlVGAzJ2NFiAtgoEvognfRFOIzivoG6xMKgdkJLpdVWcl
Received: from [83.79.69.169] by web112116.mail.gq1.yahoo.com via HTTP; Thu, 25 Nov 2010 04:42:50 PST
X-Mailer: YahooMailClassic/11.4.9 YahooMailWebService/0.8.107.285259
Date: Thu, 25 Nov 2010 04:42:50 -0800 (PST)
From: Grandl Robert <rgrandl@yahoo.com>
Subject: Re: Hadoop - how exactly is a slot defined
To: general@hadoop.apache.org
In-Reply-To: <9A758C50-80BB-4AA9-819A-EB4756E8BF45@announcemedia.microsoftonline.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1315008215-1290688970=:76184"

--0-1315008215-1290688970=:76184
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Thanks to you all for the explanations.
So, as far as I understand, if I configure 4 map slots per node(let's say -=
 512 MB RAM per slot as my node has 2 GB in total) the hadoop will always t=
ry to allocate 4 slots ? =A0Does the node report on the hearbteat that it h=
as 4 free slots ?=A0
But then, my question comes: what if another workload contend with hadoop w=
orkload at a moment, that means few resources available now for hadoop. Did=
 hadoop still report he has 4 slots free and implicitly try to allocate tas=
ks for these 4 slots ?
Thank you again for your promptly answers.
Cheers,Robert
--- On Wed, 11/24/10, Jonathan Creasy <jon.creasy@Announcemedia.com> wrote:

From: Jonathan Creasy <jon.creasy@Announcemedia.com>
Subject: Re: Hadoop - how exactly is a slot defined
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wednesday, November 24, 2010, 7:04 PM

Robert,=20

Hadoop is not currently doing any dynamic detection of resources to determi=
ne the number of slots. If I told Hadoop it could run 3,587 map tasks, it m=
ight well try to do it.=20

We use standards to determine how many map and reduce tasks a node is allow=
ed:

Each Map/Reduce Task is given:
2GB of Ram
1 Core
50GB of tmp disk space

The formula for map/reduce slots looks something like this in our environme=
nt:

G =3D GB of Ram
D =3D Disk space in /tmp
C =3D count of CPU cores

The minimum of:=20
(G-2)/2
D/50
C-1

These numbers aren't published anywhere and may completely fly in the face =
of conventional wisdom but it's what we are using and so far, seems to work=
 for us.=20

-Jonathan


On Nov 24, 2010, at 10:53 AM, Grandl Robert wrote:

> Hi,
> I am sorry bothering again about this subject, but still I am not very co=
nvinced what Hadoop assumes a slot is. I understood it represent smth in te=
rms of CPU/Memory, so you have to allocate corresponding numbers of map/red=
uce slots based on your configurations.
> BUT, I cannot understand yet, if Hadoop make any mapping between the conc=
ept of slot and physical resources itself, or are just some numbers and you=
 can go over only with this numbers.=20
> I looked on the code, but I am not able to figure out if Hadoop really di=
d some checking between number of slots and physical resources, or just is =
limited by the 2 numbers(for maximum number of map slots and reduce slots) =
and play with this numbers only. That means, the user should give more inte=
rpretation of what a slot really may be: (Only one slot per core, one slot =
per 512 MB, etc) when configure the number of map/reduce slots on his machi=
nes.
> Thanks in advance for any clue.
> Cheers,Robert
>=20
> --- On Mon, 11/22/10, Harsh J <qwertymaniac@gmail.com> wrote:
>=20
> From: Harsh J <qwertymaniac@gmail.com>
> Subject: Re: Hadoop - how exactly is a slot defined
> To: general@hadoop.apache.org
> Date: Monday, November 22, 2010, 6:52 PM
>=20
> Hi,
>=20
> On Mon, Nov 22, 2010 at 10:02 PM, Grandl Robert <rgrandl@yahoo.com> wrote=
:
>> Hi all,
>>=20
>> I have troubles in understanding what exactly a slot is. Always we are t=
alking about tasks assigned to slots, but I did not found anywhere what exa=
ctly a slot is. I assume it represent some allocation of RAM memory as well=
 as with some computation power.
>>=20
>> However, can somebody explain me what exactly a slot means (in terms of =
resources allocated for a slot) and how this mapping(between slot and physi=
cal resources) is done in Hadoop ? Or give me some hints about the files in=
 the Hadoop=A0 where it may should be ?
>=20
> A slot is of two types -- Map slot and Reduce slot. A slot represents
> an ability to run one of these "Tasks" (map/reduce tasks) individually
> at a point of time. Therefore, multiple slots on a TaskTracker means
> multiple "Tasks" may execute in parallel.
>=20
> Right now total slots in a TaskTracker is =3D=3D
> mapred.tasktracker.map.tasks.maximum for Maps and
> mapred.tasktracker.reduce.tasks.maximum for Reduces.
>=20
> Hadoop is indeed trying to go towards the dynamic slot concept, which
> could rely on the current resources available on a system, but work
> for this is still in conceptual phases. TaskTrackers emit system
> status (like CPU load, utilization, memory available/user, load
> averages) in their heartbeats today (and is utilized by certain
> schedulers, I think Capacity Scheduler uses it to determine stuff),
> but the concept of slots is still fixed as a maximum to the above two
> configurations on each TaskTracker.
>=20
> For code on how slots are checked/utilized, see any Scheduler plugin's
> code -- LimitTasksPerJobTaskScheduler, CapacityTaskScheduler for
> example.
>=20
>>=20
>> Thanks a lot,
>> Robert
>>=20
>>=20
>>=20
>=20
>=20
>=20
> --=20
> Harsh J
> www.harshj.com
>=20
>=20
>=20

=0A=0A=0A      
--0-1315008215-1290688970=:76184--

From general-return-2436-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 25 13:00:40 2010
Return-Path: <general-return-2436-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 41650 invoked from network); 25 Nov 2010 13:00:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Nov 2010 13:00:39 -0000
Received: (qmail 16038 invoked by uid 500); 25 Nov 2010 13:00:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15793 invoked by uid 500); 25 Nov 2010 13:00:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15785 invoked by uid 99); 25 Nov 2010 13:00:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Nov 2010 13:00:36 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Nov 2010 13:00:28 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 812B9B7DFB
	for <general@hadoop.apache.org>; Thu, 25 Nov 2010 13:00:06 +0000 (GMT)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id r9gGaF+0qO3Y for <general@hadoop.apache.org>;
	Thu, 25 Nov 2010 13:00:05 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 79533B7DFA
	for <general@hadoop.apache.org>; Thu, 25 Nov 2010 13:00:05 +0000 (GMT)
MailScanner-NULL-Check: 1291294791.38448@i61CsaN1lv/5BGDE3gPDJw
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id oAPCxoND005905
	for <general@hadoop.apache.org>; Thu, 25 Nov 2010 12:59:50 GMT
Message-ID: <4CEE5DC6.3080505@apache.org>
Date: Thu, 25 Nov 2010 12:59:50 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.15) Gecko/20101027 Thunderbird/3.0.10
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Hadoop - how exactly is a slot defined
References: <349178.76184.qm@web112116.mail.gq1.yahoo.com>
In-Reply-To: <349178.76184.qm@web112116.mail.gq1.yahoo.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: oAPCxoND005905
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 25/11/10 12:42, Grandl Robert wrote:
> Thanks to you all for the explanations.
> So, as far as I understand, if I configure 4 map slots per node(let's say - 512 MB RAM per slot as my node has 2 GB in total) the hadoop will always try to allocate 4 slots ?  Does the node report on the hearbteat that it has 4 free slots ?

> But then, my question comes: what if another workload contend with hadoop workload at a moment, that means few resources available now for hadoop. Did hadoop still report he has 4 slots free and implicitly try to allocate tasks for these 4 slots ?

Do you mean other system workload? Are your machines accepting work from 
other places?

The JobTracker will push out work to the nodes, and remember which 
machines it has given work to -so won't overcommit them. If you are 
doing other work on the same machine, it won't know about that, and 
still push out jobs that will now take longer.

-steve




From general-return-2437-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Nov 25 19:18:45 2010
Return-Path: <general-return-2437-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25075 invoked from network); 25 Nov 2010 19:18:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Nov 2010 19:18:44 -0000
Received: (qmail 24976 invoked by uid 500); 25 Nov 2010 19:18:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24883 invoked by uid 500); 25 Nov 2010 19:18:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24868 invoked by uid 99); 25 Nov 2010 19:18:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Nov 2010 19:18:41 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [145.100.8.49] (HELO smtp.sara.nl) (145.100.8.49)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Nov 2010 19:18:35 +0000
Received: from planck.ka.sara.nl (145.100.8.32) by sara-exch-fe1.ka.sara.nl
 (145.100.8.49) with Microsoft SMTP Server (TLS) id 14.0.702.0; Thu, 25 Nov
 2010 20:18:11 +0100
Received: from planck.ka.sara.nl ([145.100.8.32]) by planck.ka.sara.nl
 ([145.100.8.32]) with mapi; Thu, 25 Nov 2010 20:18:11 +0100
From: Evert Lammerts <Evert.Lammerts@sara.nl>
To: "'general@hadoop.apache.org'" <general@hadoop.apache.org>,
	"'hdfs-user@hadoop.apache.org'" <hdfs-user@hadoop.apache.org>
Date: Thu, 25 Nov 2010 20:16:10 +0100
Subject: interfaces to HDFS icw Kerberos
Thread-Topic: interfaces to HDFS icw Kerberos
Thread-Index: AcuM1Tff1f0m01iZSCSfczKa4w4r/Q==
Message-ID: <ADF94D8555C7A246B86A633685E0178AB90A474B1A@planck.ka.sara.nl>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/signed; protocol="application/x-pkcs7-signature";
	micalg=SHA1; boundary="----=_NextPart_000_018F_01CB8CDD.99FC9740"
MIME-Version: 1.0

------=_NextPart_000_018F_01CB8CDD.99FC9740
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit

Hi list,

We're considering to provide our users with FTP and WebDAV interfaces (with
software provided here: http://www.hadoop.iponweb.net/). These both support
user accounts, so we'll be able to deal with authentication. We're
evaluating Cloudera's Hue, which we have coupled to our LDAP service for
authentication, as an interface to MapReduce.

These solutions are not the most beautiful in terms of authentication. We'd
much prefer to use Kerberos as provided by Y!. But if we do so, how will we
enable users to get data from the outside world onto HDFS? How do others
provide secure but easy interfaces to HDFS?

Kind regards,
Evert Lammerts


------=_NextPart_000_018F_01CB8CDD.99FC9740
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIPUDCCBDIw
ggMaoAMCAQICAQEwDQYJKoZIhvcNAQEFBQAwezELMAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0
ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0
ZWQxITAfBgNVBAMMGEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczAeFw0wNDAxMDEwMDAwMDBaFw0y
ODEyMzEyMzU5NTlaMHsxCzAJBgNVBAYTAkdCMRswGQYDVQQIDBJHcmVhdGVyIE1hbmNoZXN0ZXIx
EDAOBgNVBAcMB1NhbGZvcmQxGjAYBgNVBAoMEUNvbW9kbyBDQSBMaW1pdGVkMSEwHwYDVQQDDBhB
QUEgQ2VydGlmaWNhdGUgU2VydmljZXMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC+
QJ30buHqdoccTUVEjr5GyIMGncEq/hgfjuQC+vOrXVCKFjELmgbQxXAizUktVGPMtm5oRgtT6stM
JMC8ck7q8RWu9FSaEgrDerIzYOLaiVXzIljz3tzP74OGooyUT59o8piQRoQnx3a/48w1LIteB2Rl
gsBIsKiR+WGfdiBQqJHHZrXreGIDVvCKGhPqMaMeoJn9OPb2JzJYbwf1a7j7FCuvt6rM1mNfc4za
BZmoOKjLF3g2UazpnvR4Oo3PD9lC4pgMqy+fDgHe75+ZSfEt36x0TRuYtUfF5SnR+ZAYx2KcvoPH
Jns+iiXHwN2d5jVoECCdj9je0sOEnA1e6C/JAgMBAAGjgcAwgb0wHQYDVR0OBBYEFKARCiM+lvEH
7OKvKe+CpX/QMKS0MA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MHsGA1UdHwR0MHIw
OKA2oDSGMmh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0FBQUNlcnRpZmljYXRlU2VydmljZXMuY3Js
MDagNKAyhjBodHRwOi8vY3JsLmNvbW9kby5uZXQvQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmww
DQYJKoZIhvcNAQEFBQADggEBAAhW/ALwm+j/pPrWe8ZEgM5PxMX2AFjMpra8FEloBHbo5u5d7AIP
YNaNUBhPJk4B4+awpe6/vHRUQb/9/BK4x09a9IlgBX9gtwVK8/bxwr/EuXSGti19a8zS80bdL8bg
asPDNAMsfZbdWsIOpwqZwQWLqwwv81w6z2w3VQmH3lNAbFjv/LarZW4E9hvcPOBaFcae2fFZSDAh
ZQNs7Okhc+ybA6HgN62gFRiP+roCzqcsqRATLNTlCCarIpdg+JBedNSimlO98qlo4KJuwtdssaMP
nr/raOdW8q7y4ys4OgmBtWuF174t7T8at7Jj4vViLILUagBBUPE5g5+V6TaWmG4wggTdMIIDxaAD
AgECAhBxkvvmGV+sTRKFdHE0ohinMA0GCSqGSIb3DQEBBQUAMHsxCzAJBgNVBAYTAkdCMRswGQYD
VQQIDBJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcMB1NhbGZvcmQxGjAYBgNVBAoMEUNvbW9k
byBDQSBMaW1pdGVkMSEwHwYDVQQDDBhBQUEgQ2VydGlmaWNhdGUgU2VydmljZXMwHhcNMDQwMTAx
MDAwMDAwWhcNMjgxMjMxMjM1OTU5WjCBrjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYD
VQQHEw5TYWx0IExha2UgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYD
VQQLExhodHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1VU0VSRmlyc3QtQ2xp
ZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBFbWFpbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBALI5haTyfatBO2JGN67NwWB1vDll+UoaR6K5zEjMapjVTTUZuaRC5c5J4oovHnzSMQfHTrSD
ZJ0uKdWiZMSFvYVRNXmkTmiQexx6pJKoF/KYFfKTzMmkMpW7DE8wvZigC4vlbhuiRvp4vKJvq1le
pS/Pytptqi/rrKGzaqq3Lmc1i3nhHmmI4uZGzaCl6r4LznY6eg6b6vzaJ1s9cx8i5khhxkzzabGo
Lhu21DEgLLyCio6kDqXXiUP8FlqvHXHXEVnauocNr/rz4cLwpMVnjNbWVDreCqS6A3ezZcj9HtN0
YqoYymiTHqGFfvVHZcv4TVcodNI0/zC27vZiMBSMLOsCAwEAAaOCAScwggEjMB8GA1UdIwQYMBaA
FKARCiM+lvEH7OKvKe+CpX/QMKS0MB0GA1UdDgQWBBSJgmd9xJ0mcABLtFBIfN49rgRufTAOBgNV
HQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUH
AwQwEQYDVR0gBAowCDAGBgRVHSAAMHsGA1UdHwR0MHIwOKA2oDSGMmh0dHA6Ly9jcmwuY29tb2Rv
Y2EuY29tL0FBQUNlcnRpZmljYXRlU2VydmljZXMuY3JsMDagNKAyhjBodHRwOi8vY3JsLmNvbW9k
by5uZXQvQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwEQYJYIZIAYb4QgEBBAQDAgEGMA0GCSqG
SIb3DQEBBQUAA4IBAQCdlcs8uH6lCcQevwvCx3aOOTyUxhCqTwzJ4KuEXYlU4GU7820cfDcsJVRf
liH8N4SRnRXcFE+Bz1Qda2xFYMct+ZdRTPlmyjyggoymyPDi6dRK+ew/VsnddozDggFPbADzHhph
dARHA6nGQFeRvGUixSdnT1fbZFrZjR+6hi/0Bq6cae3p9M8pF9jgSp8aIC+XTFG7RgfEijdOIOMJ
MWjHnsSLneh+EbwyaBCWEZhE2CpRYE2I63Q630MGMsg5Vow6EVLTQaRDA/Tt7zMn2zngFE4mydj1
OeKJuJNdtykmQeqzm66D/Hd1yujKtf7iZUpjPkTE0MNeh3OpmByvfxV/MIIGNTCCBR2gAwIBAgIR
APUHLR99vIaqc6YYFoh++L8wDQYJKoZIhvcNAQEFBQAwga4xCzAJBgNVBAYTAlVTMQswCQYDVQQI
EwJVVDEXMBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNVBAoTFVRoZSBVU0VSVFJVU1QgTmV0
d29yazEhMB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3QuY29tMTYwNAYDVQQDEy1VVE4tVVNF
UkZpcnN0LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1haWwwHhcNMTAwMjA4MDAwMDAwWhcN
MTEwMjA4MjM1OTU5WjCB4DE1MDMGA1UECxMsQ29tb2RvIFRydXN0IE5ldHdvcmsgLSBQRVJTT05B
IE5PVCBWQUxJREFURUQxRjBEBgNVBAsTPVRlcm1zIGFuZCBDb25kaXRpb25zIG9mIHVzZTogaHR0
cDovL3d3dy5jb21vZG8ubmV0L3JlcG9zaXRvcnkxHzAdBgNVBAsTFihjKTIwMDMgQ29tb2RvIExp
bWl0ZWQxFzAVBgNVBAMTDkV2ZXJ0IExhbW1lcnRzMSUwIwYJKoZIhvcNAQkBFhZldmVydC5sYW1t
ZXJ0c0BzYXJhLm5sMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtYAj4sXuBqEwPxYY
BMkYLPGrnqT35USIERoWs26xtSY+dLVc7HqaHDZ1lU5m9wFx/Jqph1urlRuAcCj03kGSS23Ta6kt
HTRMYRCHI3nOLfEEbH4POT+wDytORUHNeEKp/PrBn1kmKk+dk+bJI7MNdy2XkiEeO+OqKmLiBGkT
glIxngOeqgqR2IvmVv2hLAyzq8Ax20inwveZsDMa7QdQLSVUaX1kSRSihwz4Jw9X/K+6oA3ZLGp9
pZYnFHBNTHM2DR/C7dNwQgbZS7TY/Jb1kObYNhwD2JzzJlkoe0blR17MeJkF7/+Y414j0AdPFB7I
ggZ4Tm7Maheer9cIQSsvIQIDAQABo4ICGDCCAhQwHwYDVR0jBBgwFoAUiYJnfcSdJnAAS7RQSHze
Pa4Ebn0wHQYDVR0OBBYEFNTyUAqBNg9JXuiPHa9jfLvRM3IyMA4GA1UdDwEB/wQEAwIFoDAMBgNV
HRMBAf8EAjAAMCAGA1UdJQQZMBcGCCsGAQUFBwMEBgsrBgEEAbIxAQMFAjARBglghkgBhvhCAQEE
BAMCBSAwRgYDVR0gBD8wPTA7BgwrBgEEAbIxAQIBAQEwKzApBggrBgEFBQcCARYdaHR0cHM6Ly9z
ZWN1cmUuY29tb2RvLm5ldC9DUFMwgaUGA1UdHwSBnTCBmjBMoEqgSIZGaHR0cDovL2NybC5jb21v
ZG9jYS5jb20vVVROLVVTRVJGaXJzdC1DbGllbnRBdXRoZW50aWNhdGlvbmFuZEVtYWlsLmNybDBK
oEigRoZEaHR0cDovL2NybC5jb21vZG8ubmV0L1VUTi1VU0VSRmlyc3QtQ2xpZW50QXV0aGVudGlj
YXRpb25hbmRFbWFpbC5jcmwwbAYIKwYBBQUHAQEEYDBeMDYGCCsGAQUFBzAChipodHRwOi8vY3J0
LmNvbW9kb2NhLmNvbS9VVE5BQUFDbGllbnRDQS5jcnQwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3Nw
LmNvbW9kb2NhLmNvbTAhBgNVHREEGjAYgRZldmVydC5sYW1tZXJ0c0BzYXJhLm5sMA0GCSqGSIb3
DQEBBQUAA4IBAQBaJvuiBPgqdq9MEYH9dj2vW8hprAIBEnOOvfXdoP604kBhSbj3s/l0ZQyQY35W
YVAltuJQ365uJKigPQHxS+slpUOAJPEVNmwPUIW0GAjxCPxgLdJgAc4jOV8f7oF3pnfI4ukCmjPN
LpaylZzMhFt+t6zDWiMtUqyxK/4on62LoLp2D88lxwxI3q3i00bPlV0wZ+mjzS7vrQGcUgDPWEBL
FD0I3pR+Z1EH1qjelBmBJV9paVzfzmoU93LIDJA7/MEUXd3JiMZGEoVd2qE5qIZO4fNJ+Cyo6tu1
ECSQxoL6qN+sBwekiFA9Q/zlUp0HfU6Jt8WmtZ44SOh67LcmCL4ZMYIEaDCCBGQCAQEwgcQwga4x
CzAJBgNVBAYTAlVTMQswCQYDVQQIEwJVVDEXMBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNV
BAoTFVRoZSBVU0VSVFJVU1QgTmV0d29yazEhMB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3Qu
Y29tMTYwNAYDVQQDEy1VVE4tVVNFUkZpcnN0LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1h
aWwCEQD1By0ffbyGqnOmGBaIfvi/MAkGBSsOAwIaBQCgggJ4MBgGCSqGSIb3DQEJAzELBgkqhkiG
9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTEwMTEyNTE5MTYxMFowIwYJKoZIhvcNAQkEMRYEFI+tI3hl
uUblaAZpEQaMey3TLc5YMGcGCSqGSIb3DQEJDzFaMFgwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwIC
AgCAMA0GCCqGSIb3DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMAcGBSsOAwIaMAoGCCqG
SIb3DQIFMIHVBgkrBgEEAYI3EAQxgccwgcQwga4xCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJVVDEX
MBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNVBAoTFVRoZSBVU0VSVFJVU1QgTmV0d29yazEh
MB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3QuY29tMTYwNAYDVQQDEy1VVE4tVVNFUkZpcnN0
LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1haWwCEQD1By0ffbyGqnOmGBaIfvi/MIHXBgsq
hkiG9w0BCRACCzGBx6CBxDCBrjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5T
YWx0IExha2UgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYDVQQLExho
dHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1VU0VSRmlyc3QtQ2xpZW50IEF1
dGhlbnRpY2F0aW9uIGFuZCBFbWFpbAIRAPUHLR99vIaqc6YYFoh++L8wDQYJKoZIhvcNAQEBBQAE
ggEAoO1lO2+SGUgAoI//eyUw/oEJoOGbM18AnABBP2dSDpKkc+qE5MI37S1BOcUIQYD53Nh5+Qxg
kR+xy1++bAkXtroGiQoPGMfeasApYmLV/sGAFCOMfV8DNwQQM2PR8RMC+qakeqx1S5Qq75rHrZYi
tVgkmv7aEYXVzNSKff7FNt2y+n+2GQlZFvYCTkj+VMGuZPK1aXqdGGM8nWo/64wuUE+/V2cbfioj
c+d/DPCNZm3LseUBFJWntqk4n37Dli7iTdlSuf4ylEfkxPhZrInKHuAbSG+Io38R1mlT734cqMpH
bA8rRnUt5hIesCLWeyaAC4ld3qkm4mPelVqaGiQW1wAAAAAAAA==

------=_NextPart_000_018F_01CB8CDD.99FC9740--

From general-return-2438-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 26 05:52:20 2010
Return-Path: <general-return-2438-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84903 invoked from network); 26 Nov 2010 05:52:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Nov 2010 05:52:19 -0000
Received: (qmail 37276 invoked by uid 500); 26 Nov 2010 05:52:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37061 invoked by uid 500); 26 Nov 2010 05:52:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37053 invoked by uid 99); 26 Nov 2010 05:52:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Nov 2010 05:52:17 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of kiddlee1989@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Nov 2010 05:52:09 +0000
Received: by qwd7 with SMTP id 7so415488qwd.35
        for <general@hadoop.apache.org>; Thu, 25 Nov 2010 21:51:48 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:from:date
         :message-id:subject:to:content-type;
        bh=Yig7OkVP0Z4qKKo3GowmURx+2cxO2uvCFQjJsoG4CmI=;
        b=eyW0P7NbRkeGOx9aYTxoHCqC6rXSH73+SAP2tRneVmv4kRDqMW5l8OTOb2zOHeihgZ
         /CzWvFzcSXpYe/Mo69149jjvOblAOkSiVvKU9w5P/mFjcKVZcgvSrrWCzcWe7UTwZ803
         Vta/XUVdra6S0vrw3XOKcN50Wb3wQi8/O6UGs=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type;
        b=f0Yt6Fk3irP7zcZ5NLspz8CC2EuSwMYpyv9R80RW02uYAqN4YwOMk8s64cU9gGqlOh
         k2h984wJrkEZXNOyD71+vEGgstoOpucEOaCtepVucOgLP+9XU8j/7ruzCUNvwu3BfWq3
         /97LdoUg28kcxTV8aLy1TLLx3FKKkdI7faZZI=
Received: by 10.229.246.145 with SMTP id ly17mr1511460qcb.11.1290750708544;
 Thu, 25 Nov 2010 21:51:48 -0800 (PST)
MIME-Version: 1.0
Received: by 10.220.128.136 with HTTP; Thu, 25 Nov 2010 21:51:28 -0800 (PST)
From: kiddlee <kiddlee1989@gmail.com>
Date: Fri, 26 Nov 2010 13:51:28 +0800
Message-ID: <AANLkTikZVWWK+tPUCkQ8QS+9P0U=VvtNuB4nja00XwkU@mail.gmail.com>
Subject: test
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f80ea479f74c0495ee5122
X-Virus-Checked: Checked by ClamAV on apache.org

--001485f80ea479f74c0495ee5122
Content-Type: text/plain; charset=UTF-8

test

-- 

yours sincerely, kiddlee

--001485f80ea479f74c0495ee5122--

From general-return-2439-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Nov 26 21:13:56 2010
Return-Path: <general-return-2439-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44939 invoked from network); 26 Nov 2010 21:13:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Nov 2010 21:13:55 -0000
Received: (qmail 29201 invoked by uid 500); 26 Nov 2010 21:13:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28715 invoked by uid 500); 26 Nov 2010 21:13:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28701 invoked by uid 99); 26 Nov 2010 21:13:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Nov 2010 21:13:52 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Nov 2010 21:13:48 +0000
Received: by iwn35 with SMTP id 35so190689iwn.35
        for <multiple recipients>; Fri, 26 Nov 2010 13:13:27 -0800 (PST)
Received: by 10.231.144.21 with SMTP id x21mr2041407ibu.126.1290806005359;
 Fri, 26 Nov 2010 13:13:25 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.58.205 with HTTP; Fri, 26 Nov 2010 13:12:55 -0800 (PST)
In-Reply-To: <ADF94D8555C7A246B86A633685E0178AB90A474B1A@planck.ka.sara.nl>
References: <ADF94D8555C7A246B86A633685E0178AB90A474B1A@planck.ka.sara.nl>
From: Vinithra Varadharajan <vinithra@cloudera.com>
Date: Fri, 26 Nov 2010 13:12:55 -0800
Message-ID: <AANLkTimipwfMOjg2iyUUAYGvmm22AZbBc=dtAeStEBdo@mail.gmail.com>
Subject: Re: interfaces to HDFS icw Kerberos
To: general@hadoop.apache.org, Evert.Lammerts@sara.nl
Cc: "hdfs-user@hadoop.apache.org" <hdfs-user@hadoop.apache.org>, Hue-Users <hue-user@cloudera.org>
Content-Type: multipart/alternative; boundary=0016e64761da6c648f0495fb3188

--0016e64761da6c648f0495fb3188
Content-Type: text/plain; charset=ISO-8859-1

+ Hue-user mailing list

Hi Evert,

Which version of Hue and CDH are you using? CDH3b3 includes Yahoo's security
patches, which provide Kerberos authentication. In CDH3b3, we have made
changes to Hue's filebrowser application, which provides an interface to
upload data into HDFS, so that it works with Hadoop's authentication. Is
this similar to what you're looking for?

Thanks,
Vinithra

On Thu, Nov 25, 2010 at 11:16 AM, Evert Lammerts <Evert.Lammerts@sara.nl>wrote:

> Hi list,
>
> We're considering to provide our users with FTP and WebDAV interfaces (with
> software provided here: http://www.hadoop.iponweb.net/). These both
> support
> user accounts, so we'll be able to deal with authentication. We're
> evaluating Cloudera's Hue, which we have coupled to our LDAP service for
> authentication, as an interface to MapReduce.
>
> These solutions are not the most beautiful in terms of authentication. We'd
> much prefer to use Kerberos as provided by Y!. But if we do so, how will we
> enable users to get data from the outside world onto HDFS? How do others
> provide secure but easy interfaces to HDFS?
>
> Kind regards,
> Evert Lammerts
>
>

--0016e64761da6c648f0495fb3188--

From general-return-2440-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Nov 27 05:58:22 2010
Return-Path: <general-return-2440-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 15306 invoked from network); 27 Nov 2010 05:58:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Nov 2010 05:58:21 -0000
Received: (qmail 56754 invoked by uid 500); 27 Nov 2010 05:58:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56484 invoked by uid 500); 27 Nov 2010 05:58:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56475 invoked by uid 99); 27 Nov 2010 05:58:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 27 Nov 2010 05:58:19 +0000
X-ASF-Spam-Status: No, hits=-1997.8 required=10.0
	tests=ALL_TRUSTED,HTML_MESSAGE,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Sat, 27 Nov 2010 05:58:17 +0000
Received: (qmail 15279 invoked by uid 99); 27 Nov 2010 05:57:55 -0000
Received: from localhost.apache.org (HELO mail-bw0-f48.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Sat, 27 Nov 2010 05:57:55 +0000
Received: by bwz9 with SMTP id 9so2673661bwz.35
        for <general@hadoop.apache.org>; Fri, 26 Nov 2010 21:57:53 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.113.131 with SMTP id a3mr2647363bkq.123.1290837473203;
 Fri, 26 Nov 2010 21:57:53 -0800 (PST)
Received: by 10.204.162.196 with HTTP; Fri, 26 Nov 2010 21:57:53 -0800 (PST)
In-Reply-To: <270521A0-A54F-4482-8A36-ABD6FB5ECD96@apache.org>
References: <270521A0-A54F-4482-8A36-ABD6FB5ECD96@apache.org>
Date: Fri, 26 Nov 2010 21:57:53 -0800
Message-ID: <AANLkTikWUHBedykosKNqPweuOvu2qCKfcf_+HwwivsQa@mail.gmail.com>
Subject: Re: [VOTE] Hadoop Bylaws
From: "Owen O'Malley" <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d785980d985c049602852e
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d785980d985c049602852e
Content-Type: text/plain; charset=UTF-8

The bylaws pass with 3 +1 votes and no -1 votes.

-- Owen

--0016e6d785980d985c049602852e--

From general-return-2441-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Nov 27 06:24:45 2010
Return-Path: <general-return-2441-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31704 invoked from network); 27 Nov 2010 06:24:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Nov 2010 06:24:45 -0000
Received: (qmail 64089 invoked by uid 500); 27 Nov 2010 06:24:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63815 invoked by uid 500); 27 Nov 2010 06:24:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63807 invoked by uid 99); 27 Nov 2010 06:24:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 27 Nov 2010 06:24:41 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.105 as permitted sender)
Received: from [17.148.16.105] (HELO asmtpout030.mac.com) (17.148.16.105)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 27 Nov 2010 06:24:32 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; charset=us-ascii
Received: from [172.20.10.2] ([166.205.138.154])
 by asmtp030.mac.com (Sun Java(tm) System Messaging Server 6.3-8.01 (built Dec
 16 2008; 32bit)) with ESMTPSA id <0LCJ006MW5RRI500@asmtp030.mac.com> for
 general@hadoop.apache.org; Fri, 26 Nov 2010 22:23:54 -0800 (PST)
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1011260191
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2010-11-26_11:2010-11-26,2010-11-26,1970-01-01 signatures=0
Subject: Re: [VOTE] Hadoop Bylaws
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <270521A0-A54F-4482-8A36-ABD6FB5ECD96@apache.org>
Date: Fri, 26 Nov 2010 22:23:51 -0800
Message-id: <B30A4FEB-A595-4905-B3B8-B869CB559F73@mac.com>
References: <270521A0-A54F-4482-8A36-ABD6FB5ECD96@apache.org>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

Finally got some time to review this in detail.  I don't understand these 2:

1)
[snip]
>    * Approvals
> 
>        These are the types of approvals that can be sought. Different actions
>        require different types of approvals
> 
>        * Lazy Consensus - Lazy consensus requires 3 binding +1 votes
>           and no binding vetoes.
[snip]
>    * Actions
> 
>        This section describes the various actions which are undertaken within
>        the project, the corresponding approval required for that action and
>        those who have binding votes over the action.
> 
>         * Code Change
> 
>            A change made to a codebase of the project and committed
>            by a committer. This includes source code, documentation, website
>            content, etc.
> 
>            Lazy consensus of active committers.

Is the intention then that all code changes need an explicit +1 from 3 committers?  That's what this bylaw says.


2)
>          * Modifying Bylaws
> 
>            Modifying this document.
> 
>            Majority of active PMC members

Was this intended to be lazy majority?  You stated that 'all votes are lazy' but this one is not.

Cheers,
Nige

From general-return-2442-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Nov 27 11:30:05 2010
Return-Path: <general-return-2442-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 13601 invoked from network); 27 Nov 2010 11:30:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Nov 2010 11:30:04 -0000
Received: (qmail 58407 invoked by uid 500); 27 Nov 2010 11:30:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58205 invoked by uid 500); 27 Nov 2010 11:30:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58190 invoked by uid 99); 27 Nov 2010 11:30:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 27 Nov 2010 11:30:01 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [145.100.8.49] (HELO smtp.sara.nl) (145.100.8.49)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 27 Nov 2010 11:29:54 +0000
Received: from planck.ka.sara.nl (145.100.8.32) by sara-exch-fe1.ka.sara.nl
 (145.100.8.49) with Microsoft SMTP Server (TLS) id 14.0.702.0; Sat, 27 Nov
 2010 12:29:33 +0100
Received: from planck.ka.sara.nl ([145.100.8.32]) by planck.ka.sara.nl
 ([145.100.8.32]) with mapi; Sat, 27 Nov 2010 12:29:12 +0100
From: Evert Lammerts <Evert.Lammerts@sara.nl>
To: Vinithra Varadharajan <vinithra@cloudera.com>, "general@hadoop.apache.org"
	<general@hadoop.apache.org>
CC: "hdfs-user@hadoop.apache.org" <hdfs-user@hadoop.apache.org>, Hue-Users
	<hue-user@cloudera.org>
Date: Sat, 27 Nov 2010 12:29:11 +0100
Subject: RE: interfaces to HDFS icw Kerberos
Thread-Topic: interfaces to HDFS icw Kerberos
Thread-Index: AcuNrsrXyhcorlqwRB2lpoaN711T7gAdELnb
Message-ID: <ADF94D8555C7A246B86A633685E0178AB90A35C850@planck.ka.sara.nl>
References: <ADF94D8555C7A246B86A633685E0178AB90A474B1A@planck.ka.sara.nl>,<AANLkTimipwfMOjg2iyUUAYGvmm22AZbBc=dtAeStEBdo@mail.gmail.com>
In-Reply-To: <AANLkTimipwfMOjg2iyUUAYGvmm22AZbBc=dtAeStEBdo@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Vinithra, others,

We are using CDH3b3 (which works amazingly well!!). And it's nice see Y!'s =
Kerberos solution coupled to HDFS. But I wouldn't use Hue to upload a set o=
f files accumulating to 100's of GBs or a number of TB's. Browser-based app=
lications are not suitable for that, in my experience. Do you have differen=
t experiences with Hue? (To be fair, we haven't tested its performance yet.=
)

We are setting up a cluster that will be shared by people from a number of =
different institutes, all working on different cases with different data. T=
heir work and data should be protected, also from each other. At the same t=
ime they need to be able to transfer their data onto HDFS (with a high enou=
gh throughput) from their local clusters / machines. Is there a standard th=
at others are using and that works for shared clusters? How are Y! people g=
etting their data onto HDFS?

Right now we are using SFTP. We handle authentication a bit 'hacky', but it=
 works: we've coupled our LDAP server to Hue through an Auth*Handler, which=
 also allows for executing a script that updates authentication tokens for =
our FTP. So far the throughput is far from high enough though - 1.5 MB/s - =
with data going over the line unencrypted. Unless we can get that up signif=
icantly, while providing the option to encrypt the data on the wire, that w=
ill probably not be a long term solution.

If anybody can share experiences on transparently and securely getting data=
 onto HDFS from external locations, that would be much appreciated!

Cheers,
Evert

________________________________________
From: Vinithra Varadharajan [vinithra@cloudera.com]
Sent: Friday, November 26, 2010 10:12 PM
To: general@hadoop.apache.org; Evert Lammerts
Cc: hdfs-user@hadoop.apache.org; Hue-Users
Subject: Re: interfaces to HDFS icw Kerberos

+ Hue-user mailing list

Hi Evert,

Which version of Hue and CDH are you using? CDH3b3 includes Yahoo's securit=
y patches, which provide Kerberos authentication. In CDH3b3, we have made c=
hanges to Hue's filebrowser application, which provides an interface to upl=
oad data into HDFS, so that it works with Hadoop's authentication. Is this =
similar to what you're looking for?

Thanks,
Vinithra

On Thu, Nov 25, 2010 at 11:16 AM, Evert Lammerts <Evert.Lammerts@sara.nl<ma=
ilto:Evert.Lammerts@sara.nl>> wrote:
Hi list,

We're considering to provide our users with FTP and WebDAV interfaces (with
software provided here: http://www.hadoop.iponweb.net/). These both support
user accounts, so we'll be able to deal with authentication. We're
evaluating Cloudera's Hue, which we have coupled to our LDAP service for
authentication, as an interface to MapReduce.

These solutions are not the most beautiful in terms of authentication. We'd
much prefer to use Kerberos as provided by Y!. But if we do so, how will we
enable users to get data from the outside world onto HDFS? How do others
provide secure but easy interfaces to HDFS?

Kind regards,
Evert Lammerts



From general-return-2443-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 29 05:04:02 2010
Return-Path: <general-return-2443-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4824 invoked from network); 29 Nov 2010 05:04:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Nov 2010 05:04:01 -0000
Received: (qmail 80696 invoked by uid 500); 29 Nov 2010 05:03:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80393 invoked by uid 500); 29 Nov 2010 05:03:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80367 invoked by uid 99); 29 Nov 2010 05:03:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Nov 2010 05:03:55 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Nov 2010 05:03:48 +0000
Received: from EGL-EX07CAS02.ds.corp.yahoo.com (egl-ex07cas02.eglbp.corp.yahoo.com [203.83.248.209])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id oAT52sdF045731;
	Sun, 28 Nov 2010 21:02:55 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1291006975;
	bh=mhfwvQQd7pUnlMenK+J5lttDIx05eOJhW3LGXhBEyl0=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:MIME-Version;
	b=acCSP0XLlQamRrwYCVYKT+5UWi3QJCcnXVilsxaGvqyps+50Ugjm3vrd0f4aSx9TJ
	 9s4c0OmRA+KoeKE8mQm/YEV80+3tVhFMQKn6ZpOnP7Zlu9wSB1WAy7xsIj4nHgFyi+
	 n8TWrPFPBE/pgeDc3vl2yvIyoaRknQcsoBev16JM=
Received: from EGL-EX07VS02.ds.corp.yahoo.com ([203.83.248.206]) by
 EGL-EX07CAS02.ds.corp.yahoo.com ([203.83.248.216]) with mapi; Mon, 29 Nov
 2010 10:32:54 +0530
From: Venkatesh S <svenkat@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>,
        "hdfs-user@hadoop.apache.org" <hdfs-user@hadoop.apache.org>
Date: Mon, 29 Nov 2010 10:32:46 +0530
Subject: Re: interfaces to HDFS icw Kerberos
Thread-Topic: interfaces to HDFS icw Kerberos
Thread-Index: AcuM1Tff1f0m01iZSCSfczKa4w4r/QCrXF5L
Message-ID: <C91931CE.488CA%svenkat@yahoo-inc.com>
In-Reply-To: <ADF94D8555C7A246B86A633685E0178AB90A474B1A@planck.ka.sara.nl>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

We use HDFS Proxy which is part of contrib. The new version yet to be
released has both R+W and supports SPNEGO (Kerberos over HTTP). Its been
working well for us.

Thanks,
Venkatesh


On 11/26/10 12:46 AM, "Evert Lammerts" <Evert.Lammerts@sara.nl> wrote:

> Hi list,
>=20
> We're considering to provide our users with FTP and WebDAV interfaces (wi=
th
> software provided here: http://www.hadoop.iponweb.net/). These both suppo=
rt
> user accounts, so we'll be able to deal with authentication. We're
> evaluating Cloudera's Hue, which we have coupled to our LDAP service for
> authentication, as an interface to MapReduce.
>=20
> These solutions are not the most beautiful in terms of authentication. We=
'd
> much prefer to use Kerberos as provided by Y!. But if we do so, how will =
we
> enable users to get data from the outside world onto HDFS? How do others
> provide secure but easy interfaces to HDFS?
>=20
> Kind regards,
> Evert Lammerts
>=20


From general-return-2444-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 29 16:16:09 2010
Return-Path: <general-return-2444-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 26172 invoked from network); 29 Nov 2010 16:16:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Nov 2010 16:16:09 -0000
Received: (qmail 17779 invoked by uid 500); 29 Nov 2010 16:16:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17513 invoked by uid 500); 29 Nov 2010 16:16:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17501 invoked by uid 99); 29 Nov 2010 16:16:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Nov 2010 16:16:04 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of phaidinyak@local.com designates 70.183.28.5 as permitted sender)
Received: from [70.183.28.5] (HELO mail11.local.com) (70.183.28.5)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Nov 2010 16:15:56 +0000
Received: from IRV1HUBCAS01.eLiberation.com (10.1.190.40) by mail11.local.com
 (10.1.230.41) with Microsoft SMTP Server (TLS) id 8.3.83.0; Mon, 29 Nov 2010
 08:16:06 -0800
Received: from IRV1EXMB01.eLiberation.com ([fe80::895:3aed:8948:77ab]) by
 IRV1HUBCAS01.eLiberation.com ([fe80::7853:3c40:9449:94df%13]) with mapi; Mon,
 29 Nov 2010 08:15:34 -0800
From: Peter Haidinyak <phaidinyak@local.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 29 Nov 2010 08:15:34 -0800
Subject: HELP PLEASE - java.util.zip.ZipException: error in opening zip file
Thread-Topic: HELP PLEASE - java.util.zip.ZipException: error in opening zip
 file
Thread-Index: AcuP4KbC8OPZtJ1PR0WyZFENNIYUGA==
Message-ID: <321C2E54215EEB41A581FDD9DAECBC5DFCB2AF86@IRV1EXMB01.eLiberation.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_321C2E54215EEB41A581FDD9DAECBC5DFCB2AF86IRV1EXMB01eLibe_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_321C2E54215EEB41A581FDD9DAECBC5DFCB2AF86IRV1EXMB01eLibe_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi,

  Well, I got my three servers up running hadoop version 0.20.2 with one of=
 the servers being the NameNode/JobTracker and the other two being data nod=
es. The problems is when I try to run the wordcount example I get the follo=
wing exception. I hit <ctrl>c after a second since it takes awhile to timeo=
ut.

Does anyone have an idea on this?



Thanks



-Pete



[hadoop@caiss01a hadoop]$ hadoop jar hadoop-0.20.2-examples.jar wordcount /=
input_logs/output.txt /tmp/result.out

10/11/24 15:58:09 INFO input.FileInputFormat: Total input paths to process =
: 1

10/11/24 15:58:09 INFO mapred.JobClient: Running job: job_201011241548_0004

10/11/24 15:58:10 INFO mapred.JobClient:  map 0% reduce 0%

10/11/24 15:58:13 INFO mapred.JobClient: Task Id : attempt_201011241548_000=
4_m_000009_0, Status : FAILED Error initializing attempt_201011241548_0004_=
m_000009_0:

java.util.zip.ZipException: error in opening zip file

        at java.util.zip.ZipFile.open(Native Method)

        at java.util.zip.ZipFile.<init>(ZipFile.java:114)

        at java.util.jar.JarFile.<init>(JarFile.java:133)

        at java.util.jar.JarFile.<init>(JarFile.java:97)

        at org.apache.hadoop.util.RunJar.unJar(RunJar.java:36)

        at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.jav=
a:815)

        at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.ja=
va:1664)

        at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.jav=
a:97)

        at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracke=
r.java:1629)



10/11/24 15:58:13 WARN mapred.JobClient: Error reading task outputhttp://ca=
iss01b.epilotcolo.eliberation.com:50060/tasklog?plaintext=3Dtrue&taskid=3Da=
ttempt_201011241548_0004_m_000009_0&filter=3Dstdout

10/11/24 15:58:13 WARN mapred.JobClient: Error reading task outputhttp://ca=
iss01b.epilotcolo.eliberation.com:50060/tasklog?plaintext=3Dtrue&taskid=3Da=
ttempt_201011241548_0004_m_000009_0&filter=3Dstderr

10/11/24 15:58:16 INFO mapred.JobClient: Task Id : attempt_201011241548_000=
4_m_000009_1, Status : FAILED Error initializing attempt_201011241548_0004_=
m_000009_1:

java.util.zip.ZipException: error in opening zip file

        at java.util.zip.ZipFile.open(Native Method)

        at java.util.zip.ZipFile.<init>(ZipFile.java:114)

        at java.util.jar.JarFile.<init>(JarFile.java:133)

        at java.util.jar.JarFile.<init>(JarFile.java:97)

        at org.apache.hadoop.util.RunJar.unJar(RunJar.java:36)

        at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.jav=
a:815)

        at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.ja=
va:1664)

        at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.jav=
a:97)

        at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracke=
r.java:1629)



10/11/24 15:58:16 WARN mapred.JobClient: Error reading task outputhttp://ca=
iss01b.epilotcolo.eliberation.com:50060/tasklog?plaintext=3Dtrue&taskid=3Da=
ttempt_201011241548_0004_m_000009_1&filter=3Dstdout

10/11/24 15:58:16 WARN mapred.JobClient: Error reading task outputhttp://ca=
iss01b.epilotcolo.eliberation.com:50060/tasklog?plaintext=3Dtrue&taskid=3Da=
ttempt_201011241548_0004_m_000009_1&filter=3Dstderr

[hadoop@caiss01a hadoop]$


--_000_321C2E54215EEB41A581FDD9DAECBC5DFCB2AF86IRV1EXMB01eLibe_--

From general-return-2445-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 29 17:58:03 2010
Return-Path: <general-return-2445-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12696 invoked from network); 29 Nov 2010 17:58:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Nov 2010 17:58:03 -0000
Received: (qmail 73741 invoked by uid 500); 29 Nov 2010 17:58:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73545 invoked by uid 500); 29 Nov 2010 17:58:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73537 invoked by uid 99); 29 Nov 2010 17:58:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Nov 2010 17:58:00 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dsikar@gmail.com designates 209.85.213.176 as permitted sender)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Nov 2010 17:57:56 +0000
Received: by yxm8 with SMTP id 8so2624736yxm.35
        for <general@hadoop.apache.org>; Mon, 29 Nov 2010 09:57:35 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=KYR4ja8Mm4Lv/sLyPzTN4z10zUKFe1M5fQGDVLEh++Y=;
        b=RfmRuUM17Y8VVg1RSSSqdgri7kZ5PK0MnsrzHWPQrKdDEvGY/5zvSRX+H7D2neq/H2
         0T7s1sA8SEbPEfSEUlHypHGorj26wykzSvpoqLF/H1w6eaqUsMD+J6xQyJOJeUdis0n4
         kQShnhUyyj/9C0LWM0FZXWxPjYil7h0xYwzGE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=Odrulrgczw8fwMnRZKSNzFKdMt5YztuGSL4n5APMLNitOZ1S2mUCsaTPZo5nOltRtM
         N1sY+6sO5mSlPJylmnPafiyEDPhUlrDRBZnmsaev1e/ka7ELrIWlA9Wt6skiTM6zC3XL
         S/NcplzbIZ66nsuruu+AvUmTN+CsuTVfHFO8U=
MIME-Version: 1.0
Received: by 10.90.98.12 with SMTP id v12mr6088860agb.109.1291053455625; Mon,
 29 Nov 2010 09:57:35 -0800 (PST)
Received: by 10.90.62.15 with HTTP; Mon, 29 Nov 2010 09:57:35 -0800 (PST)
In-Reply-To: <321C2E54215EEB41A581FDD9DAECBC5DFCB2AF86@IRV1EXMB01.eLiberation.com>
References: <321C2E54215EEB41A581FDD9DAECBC5DFCB2AF86@IRV1EXMB01.eLiberation.com>
Date: Mon, 29 Nov 2010 17:57:35 +0000
Message-ID: <AANLkTinZ=O1m5CC8ZN3hWwBuQ4ga68eEOTgb5hMYGg9Q@mail.gmail.com>
Subject: Re: HELP PLEASE - java.util.zip.ZipException: error in opening zip file
From: daniel sikar <dsikar@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Have you checked if the output folder /tmp was created in HDFS? If
yes, that could be at least one issue. Make sure the output folder
does not exist in HDFS before running the job.

On 29 November 2010 16:15, Peter Haidinyak <phaidinyak@local.com> wrote:
> Hi,
>
> =A0Well, I got my three servers up running hadoop version 0.20.2 with one=
 of the servers being the NameNode/JobTracker and the other two being data =
nodes. The problems is when I try to run the wordcount example I get the fo=
llowing exception. I hit <ctrl>c after a second since it takes awhile to ti=
meout.
>
> Does anyone have an idea on this?
>
>
>
> Thanks
>
>
>
> -Pete
>
>
>
> [hadoop@caiss01a hadoop]$ hadoop jar hadoop-0.20.2-examples.jar wordcount=
 /input_logs/output.txt /tmp/result.out
>
> 10/11/24 15:58:09 INFO input.FileInputFormat: Total input paths to proces=
s : 1
>
> 10/11/24 15:58:09 INFO mapred.JobClient: Running job: job_201011241548_00=
04
>
> 10/11/24 15:58:10 INFO mapred.JobClient: =A0map 0% reduce 0%
>
> 10/11/24 15:58:13 INFO mapred.JobClient: Task Id : attempt_201011241548_0=
004_m_000009_0, Status : FAILED Error initializing attempt_201011241548_000=
4_m_000009_0:
>
> java.util.zip.ZipException: error in opening zip file
>
> =A0 =A0 =A0 =A0at java.util.zip.ZipFile.open(Native Method)
>
> =A0 =A0 =A0 =A0at java.util.zip.ZipFile.<init>(ZipFile.java:114)
>
> =A0 =A0 =A0 =A0at java.util.jar.JarFile.<init>(JarFile.java:133)
>
> =A0 =A0 =A0 =A0at java.util.jar.JarFile.<init>(JarFile.java:97)
>
> =A0 =A0 =A0 =A0at org.apache.hadoop.util.RunJar.unJar(RunJar.java:36)
>
> =A0 =A0 =A0 =A0at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTr=
acker.java:815)
>
> =A0 =A0 =A0 =A0at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskT=
racker.java:1664)
>
> =A0 =A0 =A0 =A0at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTr=
acker.java:97)
>
> =A0 =A0 =A0 =A0at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(T=
askTracker.java:1629)
>
>
>
> 10/11/24 15:58:13 WARN mapred.JobClient: Error reading task outputhttp://=
caiss01b.epilotcolo.eliberation.com:50060/tasklog?plaintext=3Dtrue&taskid=
=3Dattempt_201011241548_0004_m_000009_0&filter=3Dstdout
>
> 10/11/24 15:58:13 WARN mapred.JobClient: Error reading task outputhttp://=
caiss01b.epilotcolo.eliberation.com:50060/tasklog?plaintext=3Dtrue&taskid=
=3Dattempt_201011241548_0004_m_000009_0&filter=3Dstderr
>
> 10/11/24 15:58:16 INFO mapred.JobClient: Task Id : attempt_201011241548_0=
004_m_000009_1, Status : FAILED Error initializing attempt_201011241548_000=
4_m_000009_1:
>
> java.util.zip.ZipException: error in opening zip file
>
> =A0 =A0 =A0 =A0at java.util.zip.ZipFile.open(Native Method)
>
> =A0 =A0 =A0 =A0at java.util.zip.ZipFile.<init>(ZipFile.java:114)
>
> =A0 =A0 =A0 =A0at java.util.jar.JarFile.<init>(JarFile.java:133)
>
> =A0 =A0 =A0 =A0at java.util.jar.JarFile.<init>(JarFile.java:97)
>
> =A0 =A0 =A0 =A0at org.apache.hadoop.util.RunJar.unJar(RunJar.java:36)
>
> =A0 =A0 =A0 =A0at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTr=
acker.java:815)
>
> =A0 =A0 =A0 =A0at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskT=
racker.java:1664)
>
> =A0 =A0 =A0 =A0at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTr=
acker.java:97)
>
> =A0 =A0 =A0 =A0at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(T=
askTracker.java:1629)
>
>
>
> 10/11/24 15:58:16 WARN mapred.JobClient: Error reading task outputhttp://=
caiss01b.epilotcolo.eliberation.com:50060/tasklog?plaintext=3Dtrue&taskid=
=3Dattempt_201011241548_0004_m_000009_1&filter=3Dstdout
>
> 10/11/24 15:58:16 WARN mapred.JobClient: Error reading task outputhttp://=
caiss01b.epilotcolo.eliberation.com:50060/tasklog?plaintext=3Dtrue&taskid=
=3Dattempt_201011241548_0004_m_000009_1&filter=3Dstderr
>
> [hadoop@caiss01a hadoop]$
>
>

From general-return-2446-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 29 19:20:25 2010
Return-Path: <general-return-2446-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 80307 invoked from network); 29 Nov 2010 19:20:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Nov 2010 19:20:25 -0000
Received: (qmail 14858 invoked by uid 500); 29 Nov 2010 19:20:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14785 invoked by uid 500); 29 Nov 2010 19:20:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14777 invoked by uid 99); 29 Nov 2010 19:20:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Nov 2010 19:20:23 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.62.56] (HELO qmta06.westchester.pa.mail.comcast.net) (76.96.62.56)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Nov 2010 19:20:15 +0000
Received: from omta15.westchester.pa.mail.comcast.net ([76.96.62.87])
	by qmta06.westchester.pa.mail.comcast.net with comcast
	id d0FA1f0021swQuc567KuZx; Mon, 29 Nov 2010 19:19:54 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta15.westchester.pa.mail.comcast.net with comcast
	id d7Kl1f00K2SbwD53b7KoPZ; Mon, 29 Nov 2010 19:19:52 +0000
Message-Id: <C5DF467A-0D69-4396-B8D6-52156704AD8E@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <B30A4FEB-A595-4905-B3B8-B869CB559F73@mac.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Hadoop Bylaws
Date: Mon, 29 Nov 2010 11:19:44 -0800
References: <270521A0-A54F-4482-8A36-ABD6FB5ECD96@apache.org> <B30A4FEB-A595-4905-B3B8-B869CB559F73@mac.com>
X-Mailer: Apple Mail (2.936)

On Nov 26, 2010, at 10:23 PM, Nigel Daley wrote:

> Is the intention then that all code changes need an explicit +1 from  
> 3 committers?  That's what this bylaw says.

You're right. That slipped through. I'll start a vote to amend it.

> Was this intended to be lazy majority?  You stated that 'all votes  
> are lazy' but this one is not.


You're right again. I'll include that too.

-- Owen

From general-return-2447-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 29 19:32:36 2010
Return-Path: <general-return-2447-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84426 invoked from network); 29 Nov 2010 19:32:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Nov 2010 19:32:36 -0000
Received: (qmail 33441 invoked by uid 500); 29 Nov 2010 19:32:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33338 invoked by uid 500); 29 Nov 2010 19:32:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33330 invoked by uid 99); 29 Nov 2010 19:32:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Nov 2010 19:32:34 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.62.32] (HELO qmta03.westchester.pa.mail.comcast.net) (76.96.62.32)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Nov 2010 19:32:26 +0000
Received: from omta11.westchester.pa.mail.comcast.net ([76.96.62.36])
	by qmta03.westchester.pa.mail.comcast.net with comcast
	id d58x1f0020mv7h0537Y6cU; Mon, 29 Nov 2010 19:32:06 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta11.westchester.pa.mail.comcast.net with comcast
	id d7Xu1f00B2SbwD53X7XxRp; Mon, 29 Nov 2010 19:32:01 +0000
Message-Id: <159E99C4-B71C-437E-9640-AA24C50D636E@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: [DISCUSS] clarifying the bylaws
Date: Mon, 29 Nov 2010 11:31:52 -0800
X-Mailer: Apple Mail (2.936)

As Nigel pointed out, the bylaws didn't match the discussion.

Here is a proposed patch.

*** bylaws.txt  2010-11-29 11:26:53.000000000 -0800
--- bylaws2.txt 2010-11-29 11:20:05.000000000 -0800
***************
*** 207,214 ****
               by a committer. This includes source code,  
documentation, website
               content, etc.

!             Lazy consensus of active committers, but with a minimum of
!             one +1. The code can be committed after the first +1.

            * Release Plan

--- 207,213 ----
               by a committer. This includes source code,  
documentation, website
               content, etc.

!             Lazy consensus of active committers.

            * Release Plan

***************
*** 269,275 ****

               Modifying this document.

!             Lazy majority of active PMC members

      * Voting Timeframes

--- 268,274 ----

               Modifying this document.

!             Majority of active PMC members

      * Voting Timeframes
  
  

From general-return-2448-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 29 19:57:05 2010
Return-Path: <general-return-2448-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98799 invoked from network); 29 Nov 2010 19:57:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Nov 2010 19:57:05 -0000
Received: (qmail 74538 invoked by uid 500); 29 Nov 2010 19:57:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74484 invoked by uid 500); 29 Nov 2010 19:57:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74476 invoked by uid 99); 29 Nov 2010 19:57:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Nov 2010 19:57:03 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of b.aravinth@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Nov 2010 19:56:56 +0000
Received: by wyf28 with SMTP id 28so5188916wyf.35
        for <general@hadoop.apache.org>; Mon, 29 Nov 2010 11:56:35 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=9aGaNugrxXnUiLLGYYiRhJU5JO7cbV+fluD97iyYDEw=;
        b=YAu0vDBYo0SX1XrH1UruMAy9oGLOd6m3jtC3nPUnQiKWLvoMNM1tnSSaI1okp8h+dV
         5WIIn/+2cBQNZnUdwEjYqGDfWjtL0Pb+02BVtIwDAHgtWw7V403VZM+Iul/ArBei9H6e
         iihcUY7NNdvkhIreiZgES3wfKAfmfVbsIK2qI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=JndS3eEszbKtLqhGnFyaZoyrLNHKlxzYG3imawk91WhM252knzD/DFB68qrHMHV9nC
         ix+jUd4ZJdnacdzx5yOHw1yFNevQk/thF+NBurn/FoNP1hdfcGeGjBpfykl9UQiSrVfD
         TVH6e4c7FTaP6MLdEWQqsKDhnkadaLw7r/3Wc=
MIME-Version: 1.0
Received: by 10.216.164.194 with SMTP id c44mr5808443wel.107.1291060595574;
 Mon, 29 Nov 2010 11:56:35 -0800 (PST)
Received: by 10.216.86.203 with HTTP; Mon, 29 Nov 2010 11:56:35 -0800 (PST)
Date: Mon, 29 Nov 2010 14:56:35 -0500
Message-ID: <AANLkTimMLSJfw=jjXcE7j-y=L4isJBBiS03SBjy0dxNc@mail.gmail.com>
Subject: Image as input to M-R in Hadoop
From: Aravinth Bheemaraj <b.aravinth@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364267352ecf810496367818
X-Virus-Checked: Checked by ClamAV on apache.org

--0016364267352ecf810496367818
Content-Type: text/plain; charset=ISO-8859-1

Hi,

I am a beginner to Hadoop and I am looking for some help in implementing the
Mapper with an image as input. Is there any predefined Writable class for
processing image? If so, how do I use it?

Also I have read somewhere that compressed formats cannot be processed in
Hadoop. If this is true, am I making any sense in saying that the JPEG
images (which are also compressed format) cannot be processed by Hadoop?
Please correct me if I have misunderstood this concept.

Thanks,
-- 
Aravinth

--0016364267352ecf810496367818--

From general-return-2449-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 29 21:09:09 2010
Return-Path: <general-return-2449-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 51611 invoked from network); 29 Nov 2010 21:09:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Nov 2010 21:09:09 -0000
Received: (qmail 74022 invoked by uid 500); 29 Nov 2010 21:09:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73979 invoked by uid 500); 29 Nov 2010 21:09:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73971 invoked by uid 99); 29 Nov 2010 21:09:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Nov 2010 21:09:06 +0000
X-ASF-Spam-Status: No, hits=4.7 required=10.0
	tests=FREEMAIL_FROM,FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of michael_segel@hotmail.com designates 65.55.34.15 as permitted sender)
Received: from [65.55.34.15] (HELO col0-omc1-s5.col0.hotmail.com) (65.55.34.15)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Nov 2010 21:08:57 +0000
Received: from COL117-W39 ([65.55.34.7]) by col0-omc1-s5.col0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Mon, 29 Nov 2010 13:08:37 -0800
Message-ID: <COL117-W3943971B1C5C6EE54285578F240@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_2daace45-fdbb-4bec-a74d-ab6511a6837d_"
X-Originating-IP: [173.15.87.33]
From: Michael Segel <michael_segel@hotmail.com>
To: <general@hadoop.apache.org>
Subject: RE: Image as input to M-R in Hadoop
Date: Mon, 29 Nov 2010 15:08:37 -0600
Importance: Normal
In-Reply-To: <AANLkTimMLSJfw=jjXcE7j-y=L4isJBBiS03SBjy0dxNc@mail.gmail.com>
References: <AANLkTimMLSJfw=jjXcE7j-y=L4isJBBiS03SBjy0dxNc@mail.gmail.com>
MIME-Version: 1.0
X-OriginalArrivalTime: 29 Nov 2010 21:08:37.0532 (UTC) FILETIME=[9728A5C0:01CB9009]

--_2daace45-fdbb-4bec-a74d-ab6511a6837d_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


Hi=2C
The short answer is yes you can process images in Hadoop.
Think of the image as a multi-line byte stream.

As to an existing class=2C I don't believe that it exists=2C but shouldn't =
be too difficult to cobble.=20
(If you can read in XML records for processing you should be able to read i=
n a file containing a series of images.)

Note: I'm assuming that you're either reading in a directory w multiple ima=
ge files=2C or an image file w multiple images. Otherwise you probably don'=
t want to use Hadoop.


> Date: Mon=2C 29 Nov 2010 14:56:35 -0500
> Subject: Image as input to M-R in Hadoop
> From: b.aravinth@gmail.com
> To: general@hadoop.apache.org
>=20
> Hi=2C
>=20
> I am a beginner to Hadoop and I am looking for some help in implementing =
the
> Mapper with an image as input. Is there any predefined Writable class for
> processing image? If so=2C how do I use it?
>=20
> Also I have read somewhere that compressed formats cannot be processed in
> Hadoop. If this is true=2C am I making any sense in saying that the JPEG
> images (which are also compressed format) cannot be processed by Hadoop?
> Please correct me if I have misunderstood this concept.
>=20
> Thanks=2C
> --=20
> Aravinth
 		 	   		  =

--_2daace45-fdbb-4bec-a74d-ab6511a6837d_--

From general-return-2450-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 29 22:14:10 2010
Return-Path: <general-return-2450-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83902 invoked from network); 29 Nov 2010 22:14:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Nov 2010 22:14:09 -0000
Received: (qmail 66511 invoked by uid 500); 29 Nov 2010 22:14:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66468 invoked by uid 500); 29 Nov 2010 22:14:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66460 invoked by uid 99); 29 Nov 2010 22:14:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Nov 2010 22:14:08 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of b.aravinth@gmail.com designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Nov 2010 22:14:00 +0000
Received: by ewy9 with SMTP id 9so2481988ewy.35
        for <general@hadoop.apache.org>; Mon, 29 Nov 2010 14:13:39 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=f4UYZkAg3iDdMai5EUJMxAM0vsDuiiwQI6Aj2NCZrGo=;
        b=b4k3pJ0IQoqCkIph9LcoTxJu1zuQau3w8tX3GJvwBkMLghHnsg9UI0JonoA1tr3o/O
         Kyo60ZMEHytgHWrhhUT6uXD5dyFiLy8e3goucWjcna4VpD9qDCIkQY8qNslLssks5ZDe
         AWzj7MHzd+vJDxgtXudtOF+6fore9frDXnM9Q=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=v7H36Sazu+zY8QKJfAnA5mDwvHT8FdN9hRBECWmqXr99dtSuCXMbf48kY63ebT4qIJ
         x5G98lsD6t1zdjupyqD3dFiMNGmu/468ViGCl7Ut2UzOQVtWaeNN6S6efUIZiDSc+Stc
         GfuI2PwLp/16yrf+7dw/SNevOh+FdFwFv4P94=
MIME-Version: 1.0
Received: by 10.216.155.68 with SMTP id i46mr6012264wek.92.1291068818973; Mon,
 29 Nov 2010 14:13:38 -0800 (PST)
Received: by 10.216.86.203 with HTTP; Mon, 29 Nov 2010 14:13:38 -0800 (PST)
In-Reply-To: <COL117-W3943971B1C5C6EE54285578F240@phx.gbl>
References: <AANLkTimMLSJfw=jjXcE7j-y=L4isJBBiS03SBjy0dxNc@mail.gmail.com>
	<COL117-W3943971B1C5C6EE54285578F240@phx.gbl>
Date: Mon, 29 Nov 2010 17:13:38 -0500
Message-ID: <AANLkTi=bqmkDV=u=3N2YnXAdhLACSPMznuzx9wp6P65J@mail.gmail.com>
Subject: Re: Image as input to M-R in Hadoop
From: Aravinth Bheemaraj <b.aravinth@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e65b48fa55f2e304963862d2
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e65b48fa55f2e304963862d2
Content-Type: text/plain; charset=ISO-8859-1

Michael, thanks a lot for your reply.

I got to compare the images based on pixels. So is it possible to process
the image based on Pixel values rather than XML records?

I have read somewhere that the class "InputFormat" can be customized to
handle images by extending "InputSplit" and "RecordReader". But I am unsure
of the methods which are to be overridden so that I can access pixels of the
image. Is there anyway you can help me with this?

Regarding the note, I am reading in a directory with multiple image files.

On Mon, Nov 29, 2010 at 4:08 PM, Michael Segel <michael_segel@hotmail.com>wrote:

>
> Hi,
> The short answer is yes you can process images in Hadoop.
> Think of the image as a multi-line byte stream.
>
> As to an existing class, I don't believe that it exists, but shouldn't be
> too difficult to cobble.
> (If you can read in XML records for processing you should be able to read
> in a file containing a series of images.)
>
> Note: I'm assuming that you're either reading in a directory w multiple
> image files, or an image file w multiple images. Otherwise you probably
> don't want to use Hadoop.
>
>
> > Date: Mon, 29 Nov 2010 14:56:35 -0500
> > Subject: Image as input to M-R in Hadoop
> > From: b.aravinth@gmail.com
> > To: general@hadoop.apache.org
> >
> > Hi,
> >
> > I am a beginner to Hadoop and I am looking for some help in implementing
> the
> > Mapper with an image as input. Is there any predefined Writable class for
> > processing image? If so, how do I use it?
> >
> > Also I have read somewhere that compressed formats cannot be processed in
> > Hadoop. If this is true, am I making any sense in saying that the JPEG
> > images (which are also compressed format) cannot be processed by Hadoop?
> > Please correct me if I have misunderstood this concept.
> >
> > Thanks,
> > --
> > Aravinth
>
>



-- 
Aravinth Bheemaraj
University of Florida

--0016e65b48fa55f2e304963862d2--

From general-return-2451-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 29 22:31:22 2010
Return-Path: <general-return-2451-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 89321 invoked from network); 29 Nov 2010 22:31:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Nov 2010 22:31:22 -0000
Received: (qmail 97257 invoked by uid 500); 29 Nov 2010 22:31:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97214 invoked by uid 500); 29 Nov 2010 22:31:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97206 invoked by uid 99); 29 Nov 2010 22:31:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Nov 2010 22:31:20 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Nov 2010 22:31:15 +0000
Received: from [10.72.120.91] (tryarticle-lm.corp.yahoo.com [10.72.120.91])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id oATMUfFT076001
	for <general@hadoop.apache.org>; Mon, 29 Nov 2010 14:30:41 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1291069841;
	bh=GlcBIiq45lImNwqzVGBbQ0W1HUqYy16vv6GpmVSV3QM=;
	h=Message-Id:From:To:Content-Type:Mime-Version:Subject:Date;
	b=Nir8a2iKJ6n1AO0fFv1oozrbLomkX4Nr+Y/er3jhcsv2lEHacm2YsP6wfjAcGCErm
	 gzM6Yq0+EHLJmFw629JtVZwXJDBQ75PfmMRUtg94uR0OKy3CVgWRD6g7ab2B40n7oU
	 Ut80jWHRWTmBah6DNuL+iejLwOndZqi0dEAWLC7Q=
Message-Id: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>
From: "Owen O'Malley" <oom@yahoo-inc.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=Apple-Mail-39--769971144
Mime-Version: 1.0 (Apple Message framework v936)
Subject: [VOTE] Direction for Hadoop development
Date: Mon, 29 Nov 2010 14:30:41 -0800
X-Mailer: Apple Mail (2.936)

--Apple-Mail-39--769971144
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit

All,
    Based on the discussion on HADOOP-6685, there is a pretty  
fundamental difference of opinion about how Hadoop should evolve. We  
need to figure out how the majority of the PMC wants the project to  
evolve to understand which patches move us forward. Please vote  
whether you approve of the following direction. Clearly as the author,  
I'm +1.

-- Owen

Hadoop has always included library code so that users had a strong  
foundation to build their applications on without needing to  
continually reinvent the wheel. This combination of framework and  
powerful library code is a common pattern for successful projects,  
such as Java, Lucene, etc. Toward that end, we need to continue to  
extend the Hadoop library code and actively maintain it as the  
framework evolves. Continuing support for SequenceFile and TFile,  
which are both widely used is mandatory. The opposite pattern of  
implementing the framework and letting each distribution add the  
required libraries will lead to increased community fragmentation and  
vendor lock in.

Hadoop's generic serialization framework had a lot of promise when it  
was introduced, but has been hampered by a lack of plugins other than  
Writables and Java serialization. Supporting a wide range of  
serializations natively in Hadoop will give the users new  
capabilities. Currently, to support Avro or ProtoBuf objects mutually  
incompatible third party solutions are required. It benefits Hadoop to  
support them with a common framework that will support all of them. In  
particular, having easy, out of the box support for Thrift, ProtoBufs,  
Avro, and our legacy serializations is a desired state.

As a distributed system, there are many instances where Hadoop needs  
to serialize data. Many of those applications need a lightweight,  
versioned serialization framework like ProtocolBuffers or Thrift and  
using them is appropriate. Adding dependences on Thrift and  
ProtocolBuffers to the previous dependence on Avro is acceptable.
--Apple-Mail-39--769971144--

From general-return-2452-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 29 22:50:06 2010
Return-Path: <general-return-2452-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94101 invoked from network); 29 Nov 2010 22:50:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Nov 2010 22:50:06 -0000
Received: (qmail 25337 invoked by uid 500); 29 Nov 2010 22:50:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25114 invoked by uid 500); 29 Nov 2010 22:50:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25102 invoked by uid 99); 29 Nov 2010 22:50:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Nov 2010 22:50:04 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of shrijeet@rocketfuelinc.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Nov 2010 22:49:58 +0000
Received: by fxm2 with SMTP id 2so3978538fxm.35
        for <general@hadoop.apache.org>; Mon, 29 Nov 2010 14:49:37 -0800 (PST)
MIME-Version: 1.0
Received: by 10.223.83.133 with SMTP id f5mr6019465fal.29.1291070976980; Mon,
 29 Nov 2010 14:49:36 -0800 (PST)
Received: by 10.223.96.131 with HTTP; Mon, 29 Nov 2010 14:49:36 -0800 (PST)
In-Reply-To: <AANLkTi=bqmkDV=u=3N2YnXAdhLACSPMznuzx9wp6P65J@mail.gmail.com>
References: <AANLkTimMLSJfw=jjXcE7j-y=L4isJBBiS03SBjy0dxNc@mail.gmail.com>
	<COL117-W3943971B1C5C6EE54285578F240@phx.gbl>
	<AANLkTi=bqmkDV=u=3N2YnXAdhLACSPMznuzx9wp6P65J@mail.gmail.com>
Date: Mon, 29 Nov 2010 14:49:36 -0800
Message-ID: <AANLkTin7imD4_WRJ_zRsEveXm1-wjT6Z8gtRMyJP0qX+@mail.gmail.com>
Subject: Re: Image as input to M-R in Hadoop
From: Shrijeet Paliwal <shrijeet@rocketfuel.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

This gentleman here (see below) is doing a hadoop streaming magic and
seems to be playing with the image features in map reducy way. Its not
using hadoop's java api though, so no help there.
Still you can check and see if the articles gives you some clues,
http://techportal.ibuildings.com/2009/11/02/precision-color-searching-with-gmagick-and-amazon-elastic-mapreduce/

PS: Pardon if the motivation in the article is orthogonal to yours.

-Shrijeet

On Mon, Nov 29, 2010 at 2:13 PM, Aravinth Bheemaraj
<b.aravinth@gmail.com> wrote:
> Michael, thanks a lot for your reply.
>
> I got to compare the images based on pixels. So is it possible to process
> the image based on Pixel values rather than XML records?
>
> I have read somewhere that the class "InputFormat" can be customized to
> handle images by extending "InputSplit" and "RecordReader". But I am unsure
> of the methods which are to be overridden so that I can access pixels of the
> image. Is there anyway you can help me with this?
>
> Regarding the note, I am reading in a directory with multiple image files.
>
> On Mon, Nov 29, 2010 at 4:08 PM, Michael Segel <michael_segel@hotmail.com>wrote:
>
>>
>> Hi,
>> The short answer is yes you can process images in Hadoop.
>> Think of the image as a multi-line byte stream.
>>
>> As to an existing class, I don't believe that it exists, but shouldn't be
>> too difficult to cobble.
>> (If you can read in XML records for processing you should be able to read
>> in a file containing a series of images.)
>>
>> Note: I'm assuming that you're either reading in a directory w multiple
>> image files, or an image file w multiple images. Otherwise you probably
>> don't want to use Hadoop.
>>
>>
>> > Date: Mon, 29 Nov 2010 14:56:35 -0500
>> > Subject: Image as input to M-R in Hadoop
>> > From: b.aravinth@gmail.com
>> > To: general@hadoop.apache.org
>> >
>> > Hi,
>> >
>> > I am a beginner to Hadoop and I am looking for some help in implementing
>> the
>> > Mapper with an image as input. Is there any predefined Writable class for
>> > processing image? If so, how do I use it?
>> >
>> > Also I have read somewhere that compressed formats cannot be processed in
>> > Hadoop. If this is true, am I making any sense in saying that the JPEG
>> > images (which are also compressed format) cannot be processed by Hadoop?
>> > Please correct me if I have misunderstood this concept.
>> >
>> > Thanks,
>> > --
>> > Aravinth
>>
>>
>
>
>
> --
> Aravinth Bheemaraj
> University of Florida
>

From general-return-2453-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 29 23:06:18 2010
Return-Path: <general-return-2453-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6681 invoked from network); 29 Nov 2010 23:06:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Nov 2010 23:06:18 -0000
Received: (qmail 36826 invoked by uid 500); 29 Nov 2010 23:06:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36737 invoked by uid 500); 29 Nov 2010 23:06:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36553 invoked by uid 99); 29 Nov 2010 23:06:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Nov 2010 23:06:12 +0000
X-ASF-Spam-Status: No, hits=0.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_SOFTFAIL
X-Spam-Check-By: apache.org
Received-SPF: softfail (athena.apache.org: transitioning domain of Greg.Troyan@zecco.net does not designate 64.18.0.184 as permitted sender)
Received: from [64.18.0.184] (HELO exprod5og107.obsmtp.com) (64.18.0.184)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 29 Nov 2010 23:06:07 +0000
Received: from source ([38.102.132.228]) (using TLSv1) by exprod5ob107.postini.com ([64.18.4.12]) with SMTP
	ID DSNKTPQxyikE3FC6CXEgKTQQFFJcxPpp3tZm@postini.com; Mon, 29 Nov 2010 15:05:47 PST
Received: from mx1bg.zecco.local ([10.3.30.16]) by mx1bg.zecco.local
 ([10.3.30.16]) with mapi; Mon, 29 Nov 2010 15:04:55 -0800
From: Greg Troyan <Greg.Troyan@zecco.net>
To: "'general@hadoop.apache.org'" <general@hadoop.apache.org>
Date: Mon, 29 Nov 2010 15:04:55 -0800
Subject: Help with "Hadoop common not found" and bug HADOOP-6953 
Thread-Topic: Help with "Hadoop common not found" and bug HADOOP-6953 
Thread-Index: AcuQGdUr5H3+zBcFSxSt0s1x5DEUTw==
Message-ID: <1766D7881BAA7C4386198E998D962F7F3713336537@mx1bg.zecco.local>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
x-tm-as-product-ver: SMEX-8.6.0.1386-6.500.1024-17796.005
x-tm-as-result: No--53.694800-5.000000-31
x-tm-as-user-approved-sender: No
x-tm-as-user-blocked-sender: No
Content-Type: multipart/alternative;
	boundary="_000_1766D7881BAA7C4386198E998D962F7F3713336537mx1bgzeccoloc_"
MIME-Version: 1.0

--_000_1766D7881BAA7C4386198E998D962F7F3713336537mx1bgzeccoloc_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Like others, I am experiencing the Hadoop common not found" error when star=
ting hadoop using start-dfs.sh under Hadoop v0.21.

I have found this link that shows how to edit hdfs-config.sh and mapred-con=
fig.sh:

https://issues.apache.org/jira/browse/HADOOP-6953

But it's still not working for me.

I've added the code from that link to both files to no effect.

To start with the most obvious: Could it be that I've pasted the code in th=
e wrong place in the files? (I basically plopped the code at the beginning =
of the file.)

Here's what my hdsf-config.sh looks like:

this=3D"${BASH_SOURCE-$0}"
while [ -h "$this" ]; do
ls=3D`ls -ld "$this"`
link=3D`expr "$ls" : '.-> (.)$'`
if expr "$link" : './.' > /dev/null; then
this=3D"$link"
else
this=3D`dirname "$this"`/"$link"
fi
done

# convert relative path to absolute path
common_bin=3D`dirname "$this"`
script=3D`basename "$this"`
common_bin=3D`cd "$common_bin"; pwd`
this=3D"$common_bin/$script"

# the root of the Hadoop installation
#TODO: change the env variable when dir structure is changed
export HADOOP_HOME=3D`dirname "$this"`/..
export HADOOP_COMMON_HOME=3D"${HADOOP_HOME}"

bin=3D`dirname "$0"`
bin=3D`cd "$bin"; pwd`

export HADOOP_HDFS_HOME=3D"${HADOOP_HDFS_HOME:-$bin/..}"

if [ -d "${HADOOP_COMMON_HOME}" ]; then
  . "$HADOOP_COMMON_HOME"/bin/hadoop-config.sh
elif [ -d "${HADOOP_HOME}" ]; then
  . "$HADOOP_HOME"/bin/hadoop-config.sh
else
  echo "Hadoop common not found."
  exit
fi





________________________________
Zecco.com is a financial portal of Zecco Holdings, Inc. Zecco Holdings, Inc=
. is not a securities broker/dealer. Securities offered through Zecco Tradi=
ng, Inc. The content of this message and its attachments is intended only f=
or the use of the intended recipient and may contain confidential and privi=
leged information. If you are not the intended recipient, any dissemination=
, distribution, or copying of this message or its attachments is prohibited=
. If you received this message in error, please notify the sender by replyi=
ng to this email immediately and delete this message and its attachments fr=
om your computer.

--_000_1766D7881BAA7C4386198E998D962F7F3713336537mx1bgzeccoloc_--

From general-return-2454-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Nov 29 23:14:55 2010
Return-Path: <general-return-2454-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 10121 invoked from network); 29 Nov 2010 23:14:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Nov 2010 23:14:55 -0000
Received: (qmail 49479 invoked by uid 500); 29 Nov 2010 23:14:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49397 invoked by uid 500); 29 Nov 2010 23:14:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49386 invoked by uid 99); 29 Nov 2010 23:14:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Nov 2010 23:14:54 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 29 Nov 2010 23:14:51 +0000
Received: (qmail 9877 invoked by uid 99); 29 Nov 2010 23:14:30 -0000
Received: from localhost.apache.org (HELO [192.168.168.144]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Nov 2010 23:14:30 +0000
Message-ID: <4CF433D4.6090407@apache.org>
Date: Mon, 29 Nov 2010 15:14:28 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Direction for Hadoop development
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>
In-Reply-To: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Owen,

First, I don't see the yes/no issue that you'd like us to vote on here. 
  We vote on patches.  We vote on releases.  We vote on committers.  We 
don't vote on a project direction statement.  Rather folks present 
plans, others may present their conflicting concerns, and we need to get 
these to meet in order to make progress on a particular issue.

I too support continuing support for SequenceFile.

I too support adding flexible serialization APIs to MapReduce.

I do not support extending SequenceFile's format in substantial ways.  A 
proliferation of expressively equivalent yet incompatible file formats 
hinders the interoperable evolution of the Hadoop ecosystem.

I do not support adding new dependencies to the classpath of MapReduce 
user tasks.  We want to provide as much flexibility to user code as 
possible.  The more libraries the system includes the greater the 
potential for version conflicts.  As the Hadoop ecosystem expands, 
MapReduce should seek primarily to be an efficient, reliable kernel, not 
an extensive library of tools.

So I agree with some of your points, but not with others.

Cheers,

Doug

On 11/29/2010 02:30 PM, Owen O'Malley wrote:
> All,
> Based on the discussion on HADOOP-6685, there is a pretty fundamental
> difference of opinion about how Hadoop should evolve. We need to figure
> out how the majority of the PMC wants the project to evolve to
> understand which patches move us forward. Please vote whether you
> approve of the following direction. Clearly as the author, I'm +1.
>
> -- Owen
>
> Hadoop has always included library code so that users had a strong
> foundation to build their applications on without needing to continually
> reinvent the wheel. This combination of framework and powerful library
> code is a common pattern for successful projects, such as Java, Lucene,
> etc. Toward that end, we need to continue to extend the Hadoop library
> code and actively maintain it as the framework evolves. Continuing
> support for SequenceFile and TFile, which are both widely used is
> mandatory. The opposite pattern of implementing the framework and
> letting each distribution add the required libraries will lead to
> increased community fragmentation and vendor lock in.
>
> Hadoop's generic serialization framework had a lot of promise when it
> was introduced, but has been hampered by a lack of plugins other than
> Writables and Java serialization. Supporting a wide range of
> serializations natively in Hadoop will give the users new capabilities.
> Currently, to support Avro or ProtoBuf objects mutually incompatible
> third party solutions are required. It benefits Hadoop to support them
> with a common framework that will support all of them. In particular,
> having easy, out of the box support for Thrift, ProtoBufs, Avro, and our
> legacy serializations is a desired state.
>
> As a distributed system, there are many instances where Hadoop needs to
> serialize data. Many of those applications need a lightweight, versioned
> serialization framework like ProtocolBuffers or Thrift and using them is
> appropriate. Adding dependences on Thrift and ProtocolBuffers to the
> previous dependence on Avro is acceptable.

From general-return-2455-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 30 00:22:37 2010
Return-Path: <general-return-2455-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37412 invoked from network); 30 Nov 2010 00:22:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Nov 2010 00:22:37 -0000
Received: (qmail 12241 invoked by uid 500); 30 Nov 2010 00:22:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12197 invoked by uid 500); 30 Nov 2010 00:22:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12189 invoked by uid 99); 30 Nov 2010 00:22:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Nov 2010 00:22:35 +0000
X-ASF-Spam-Status: No, hits=-1997.8 required=10.0
	tests=ALL_TRUSTED,HTML_MESSAGE,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 30 Nov 2010 00:22:33 +0000
Received: (qmail 37352 invoked by uid 99); 30 Nov 2010 00:22:11 -0000
Received: from localhost.apache.org (HELO mail-bw0-f48.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Nov 2010 00:22:11 +0000
Received: by bwz9 with SMTP id 9so4999737bwz.35
        for <general@hadoop.apache.org>; Mon, 29 Nov 2010 16:22:09 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.54.147 with SMTP id q19mr5908896bkg.69.1291076529306; Mon,
 29 Nov 2010 16:22:09 -0800 (PST)
Received: by 10.204.57.75 with HTTP; Mon, 29 Nov 2010 16:22:09 -0800 (PST)
In-Reply-To: <4CF433D4.6090407@apache.org>
References: <BB46C07C-5881-4C9F-A121-B35F458DA88D@yahoo-inc.com>
	<4CF433D4.6090407@apache.org>
Date: Mon, 29 Nov 2010 16:22:09 -0800
Message-ID: <AANLkTineTnsShLoa5vO-X9UAp_DUMrHRjc2ry=i4qkXs@mail.gmail.com>
Subject: Re: [VOTE] Direction for Hadoop development
From: "Owen O'Malley" <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c5a938e83d8004963a2d24
X-Virus-Checked: Checked by ClamAV on apache.org

--001636c5a938e83d8004963a2d24
Content-Type: text/plain; charset=UTF-8

On Mon, Nov 29, 2010 at 3:14 PM, Doug Cutting <cutting@apache.org> wrote:

We don't vote on a project direction statement.  Rather folks present plans,
> others may present their conflicting concerns, and we need to get these to
> meet in order to make progress on a particular issue.
>

We haven't in the past, but clearly there is currently disagreement on the
direction for Hadoop that is blocking forward progress. Using a vote to
decide on project direction is far better than vetoing patches based on
disjoint visions of how to move forward. If the project votes for a
direction, a veto that is based on an opposing direction is clearly invalid.

I too support continuing support for SequenceFile.
>

I said far more than that. I said that it should be actively maintained as
the framework evolves. Clearly the generic serialization changes are far
more useful if they include a file format that uses them. Extending
SequenceFile and TFile is a good thing.

I do not support extending SequenceFile's format in substantial ways.  A
> proliferation of expressively equivalent yet incompatible file formats
> hinders the interoperable evolution of the Hadoop ecosystem.
>

There is no equivalent functionality and even if there was, it would still
be worthwhile extending SequenceFile since they are so heavily used. Making
SequenceFile support the new serialization API increases their value with
minimal disruption to users.

I do not support adding new dependencies to the classpath of MapReduce user
> tasks.


That isn't reasonable. As Hadoop evolves, we have and will continue to add
dependences. For example, in your last MapReduce (MAPREDUCE-980) patch you
added avro and paranamer as dependences.

-- Owen

--001636c5a938e83d8004963a2d24--

From general-return-2456-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 30 00:53:07 2010
Return-Path: <general-return-2456-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 52548 invoked from network); 30 Nov 2010 00:53:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Nov 2010 00:53:07 -0000
Received: (qmail 37475 invoked by uid 500); 30 Nov 2010 00:53:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37292 invoked by uid 500); 30 Nov 2010 00:53:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37283 invoked by uid 99); 30 Nov 2010 00:53:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Nov 2010 00:53:06 +0000
X-ASF-Spam-Status: No, hits=2.5 required=10.0
	tests=FREEMAIL_FROM,FREEMAIL_REPLY,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of dsikar@gmail.com designates 209.85.161.176 as permitted sender)
Received: from [209.85.161.176] (HELO mail-gx0-f176.google.com) (209.85.161.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Nov 2010 00:52:59 +0000
Received: by gxk4 with SMTP id 4so2871390gxk.35
        for <general@hadoop.apache.org>; Mon, 29 Nov 2010 16:52:38 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=rUPZiQr/YQs4j7MJk7wWnR+CU6KL+2Z6F7VlRHvVjkg=;
        b=S/AweoyN6naELuSjozShtC514HUytdaTJjBnJPn3H137LsTAh6LLJaL/8hZ1Qb0puX
         TqFXqnYkb/DtVzw+IyFsE/Idul3fc8oY+WVW6dHkTNJG0God9zAdnqZqqF/3Qs3ulTtM
         41pM+NlT4Er0fE+tnEPJvOrcKY7aexuYKBjG8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=YrDtfICOAFr97ZCvXR8Pd23Gcs1Qm2gkBUutm8RYZ3VLQRX6fGXFr5lI8sYLLd/Uqj
         vgxOKJuWO2IVWx4nkuey4wWLvaMCMmT/RvBDyD5TaYvEc6FjVuu0S25zfRS/Pc60G8mQ
         vpoca6uDHwbrJqzg3xjexMKPjf9sGZ+ZdmYwo=
MIME-Version: 1.0
Received: by 10.91.121.20 with SMTP id y20mr10069188agm.28.1291078358105; Mon,
 29 Nov 2010 16:52:38 -0800 (PST)
Received: by 10.90.62.15 with HTTP; Mon, 29 Nov 2010 16:52:38 -0800 (PST)
In-Reply-To: <AANLkTin7imD4_WRJ_zRsEveXm1-wjT6Z8gtRMyJP0qX+@mail.gmail.com>
References: <AANLkTimMLSJfw=jjXcE7j-y=L4isJBBiS03SBjy0dxNc@mail.gmail.com>
	<COL117-W3943971B1C5C6EE54285578F240@phx.gbl>
	<AANLkTi=bqmkDV=u=3N2YnXAdhLACSPMznuzx9wp6P65J@mail.gmail.com>
	<AANLkTin7imD4_WRJ_zRsEveXm1-wjT6Z8gtRMyJP0qX+@mail.gmail.com>
Date: Tue, 30 Nov 2010 00:52:38 +0000
Message-ID: <AANLkTinx1E9AVwMk=4r3+tUWLoxu+Ncyk5xGr1+YaOEy@mail.gmail.com>
Subject: Re: Image as input to M-R in Hadoop
From: daniel sikar <dsikar@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Aravinth

This is probably do-able with Hadoop Streaming.

Imagine you have copied a bunch of image files to HDFS and now you
want to point them to say, an executable. Odds are that executable
already exists with some command line options that would take amongst
other things, the file path of the image you would like to process.

Hadoop Streaming makes a number of environment variables available at
runtime, for instance "map_input_file" which gives you the file name
of the file being processed, and so forth. My guess is that there is
also an environment variable that will give you the filepath in the
local filesystem.

You need to code that in plus add a -file parameter to specify your
executable. If you are using Amazon's EMR, you will need to put your
code and executable into an S3 bucket, then specify the bucket name to
Hadoop Streaming.

Good luck

Daniel

On 29 November 2010 22:49, Shrijeet Paliwal <shrijeet@rocketfuel.com> wrote:
> This gentleman here (see below) is doing a hadoop streaming magic and
> seems to be playing with the image features in map reducy way. Its not
> using hadoop's java api though, so no help there.
> Still you can check and see if the articles gives you some clues,
> http://techportal.ibuildings.com/2009/11/02/precision-color-searching-with-gmagick-and-amazon-elastic-mapreduce/
>
> PS: Pardon if the motivation in the article is orthogonal to yours.
>
> -Shrijeet
>
> On Mon, Nov 29, 2010 at 2:13 PM, Aravinth Bheemaraj
> <b.aravinth@gmail.com> wrote:
>> Michael, thanks a lot for your reply.
>>
>> I got to compare the images based on pixels. So is it possible to process
>> the image based on Pixel values rather than XML records?
>>
>> I have read somewhere that the class "InputFormat" can be customized to
>> handle images by extending "InputSplit" and "RecordReader". But I am unsure
>> of the methods which are to be overridden so that I can access pixels of the
>> image. Is there anyway you can help me with this?
>>
>> Regarding the note, I am reading in a directory with multiple image files.
>>
>> On Mon, Nov 29, 2010 at 4:08 PM, Michael Segel <michael_segel@hotmail.com>wrote:
>>
>>>
>>> Hi,
>>> The short answer is yes you can process images in Hadoop.
>>> Think of the image as a multi-line byte stream.
>>>
>>> As to an existing class, I don't believe that it exists, but shouldn't be
>>> too difficult to cobble.
>>> (If you can read in XML records for processing you should be able to read
>>> in a file containing a series of images.)
>>>
>>> Note: I'm assuming that you're either reading in a directory w multiple
>>> image files, or an image file w multiple images. Otherwise you probably
>>> don't want to use Hadoop.
>>>
>>>
>>> > Date: Mon, 29 Nov 2010 14:56:35 -0500
>>> > Subject: Image as input to M-R in Hadoop
>>> > From: b.aravinth@gmail.com
>>> > To: general@hadoop.apache.org
>>> >
>>> > Hi,
>>> >
>>> > I am a beginner to Hadoop and I am looking for some help in implementing
>>> the
>>> > Mapper with an image as input. Is there any predefined Writable class for
>>> > processing image? If so, how do I use it?
>>> >
>>> > Also I have read somewhere that compressed formats cannot be processed in
>>> > Hadoop. If this is true, am I making any sense in saying that the JPEG
>>> > images (which are also compressed format) cannot be processed by Hadoop?
>>> > Please correct me if I have misunderstood this concept.
>>> >
>>> > Thanks,
>>> > --
>>> > Aravinth
>>>
>>>
>>
>>
>>
>> --
>> Aravinth Bheemaraj
>> University of Florida
>>
>

From general-return-2457-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 30 12:33:35 2010
Return-Path: <general-return-2457-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59239 invoked from network); 30 Nov 2010 12:33:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Nov 2010 12:33:35 -0000
Received: (qmail 4938 invoked by uid 500); 30 Nov 2010 12:33:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4688 invoked by uid 500); 30 Nov 2010 12:33:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4671 invoked by uid 99); 30 Nov 2010 12:33:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Nov 2010 12:33:32 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [195.232.224.70] (HELO mailout01.vodafone.com) (195.232.224.70)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Nov 2010 12:33:27 +0000
Received: from mailint01 (localhost [127.0.0.1])
	by mailout01 (Postfix) with ESMTP id BF48C4DE5
	for <general@hadoop.apache.org>; Tue, 30 Nov 2010 13:33:04 +0100 (CET)
Received: from avoexs04.internal.vodafone.com (avoexs04.dc-ratingen.de [145.230.4.198])
	(using TLSv1 with cipher RC4-MD5 (128/128 bits))
	(No client certificate requested)
	by mailint01 (Postfix) with ESMTPS id AFF7B4DD8;
	Tue, 30 Nov 2010 13:33:04 +0100 (CET)
Received: from VF-MBX13.internal.vodafone.com ([145.230.5.24]) by avoexs04.internal.vodafone.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Tue, 30 Nov 2010 13:33:08 +0100
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable
Subject: RE: interfaces to HDFS icw Kerberos
Date: Tue, 30 Nov 2010 13:33:04 +0100
Message-ID: <219D8244D980254ABF28AB469AD4E98F04013AEE@VF-MBX13.internal.vodafone.com>
In-Reply-To: <ADF94D8555C7A246B86A633685E0178AB90A35C850@planck.ka.sara.nl>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: interfaces to HDFS icw Kerberos
Thread-Index: AcuNrsrXyhcorlqwRB2lpoaN711T7gAdELnbAJmzDhA=
References: <ADF94D8555C7A246B86A633685E0178AB90A474B1A@planck.ka.sara.nl>,<AANLkTimipwfMOjg2iyUUAYGvmm22AZbBc=dtAeStEBdo@mail.gmail.com> <ADF94D8555C7A246B86A633685E0178AB90A35C850@planck.ka.sara.nl>
From: "Gibbon, Robert, VF-Group" <Robert.Gibbon@vodafone.com>
To: <general@hadoop.apache.org>,
	"Vinithra Varadharajan" <vinithra@cloudera.com>
Cc: <hdfs-user@hadoop.apache.org>,
	"Hue-Users" <hue-user@cloudera.org>
X-OriginalArrivalTime: 30 Nov 2010 12:33:08.0065 (UTC) FILETIME=[BE2C4510:01CB908A]

Hi Evert,

We use the WebDAV tool integrated with HUE for authenticated ad-hoc
read/write access but for full throttle inbound data, our implementation
uses Flume.

With that said, the WebDAV solution is horizontally scalable  - it is a
stateless web app - so a software or hardware loadbalancer could be your
friend here to get the throughput up.

FTP is a session based protocol - and I have not looked in detail at the
HDFS-FTP implementation so it might not be so easy to scale it sideways.

HTH
Robert

-----Original Message-----
From: Evert Lammerts [mailto:Evert.Lammerts@sara.nl]=20
Sent: Samstag, 27. November 2010 12:29
To: Vinithra Varadharajan; general@hadoop.apache.org
Cc: hdfs-user@hadoop.apache.org; Hue-Users
Subject: RE: interfaces to HDFS icw Kerberos

Hi Vinithra, others,

We are using CDH3b3 (which works amazingly well!!). And it's nice see
Y!'s Kerberos solution coupled to HDFS. But I wouldn't use Hue to upload
a set of files accumulating to 100's of GBs or a number of TB's.
Browser-based applications are not suitable for that, in my experience.
Do you have different experiences with Hue? (To be fair, we haven't
tested its performance yet.)

We are setting up a cluster that will be shared by people from a number
of different institutes, all working on different cases with different
data. Their work and data should be protected, also from each other. At
the same time they need to be able to transfer their data onto HDFS
(with a high enough throughput) from their local clusters / machines. Is
there a standard that others are using and that works for shared
clusters? How are Y! people getting their data onto HDFS?

Right now we are using SFTP. We handle authentication a bit 'hacky', but
it works: we've coupled our LDAP server to Hue through an Auth*Handler,
which also allows for executing a script that updates authentication
tokens for our FTP. So far the throughput is far from high enough though
- 1.5 MB/s - with data going over the line unencrypted. Unless we can
get that up significantly, while providing the option to encrypt the
data on the wire, that will probably not be a long term solution.

If anybody can share experiences on transparently and securely getting
data onto HDFS from external locations, that would be much appreciated!

Cheers,
Evert

________________________________________
From: Vinithra Varadharajan [vinithra@cloudera.com]
Sent: Friday, November 26, 2010 10:12 PM
To: general@hadoop.apache.org; Evert Lammerts
Cc: hdfs-user@hadoop.apache.org; Hue-Users
Subject: Re: interfaces to HDFS icw Kerberos

+ Hue-user mailing list

Hi Evert,

Which version of Hue and CDH are you using? CDH3b3 includes Yahoo's
security patches, which provide Kerberos authentication. In CDH3b3, we
have made changes to Hue's filebrowser application, which provides an
interface to upload data into HDFS, so that it works with Hadoop's
authentication. Is this similar to what you're looking for?

Thanks,
Vinithra

On Thu, Nov 25, 2010 at 11:16 AM, Evert Lammerts
<Evert.Lammerts@sara.nl<mailto:Evert.Lammerts@sara.nl>> wrote:
Hi list,

We're considering to provide our users with FTP and WebDAV interfaces
(with software provided here: http://www.hadoop.iponweb.net/). These
both support user accounts, so we'll be able to deal with
authentication. We're evaluating Cloudera's Hue, which we have coupled
to our LDAP service for authentication, as an interface to MapReduce.

These solutions are not the most beautiful in terms of authentication.
We'd much prefer to use Kerberos as provided by Y!. But if we do so, how
will we enable users to get data from the outside world onto HDFS? How
do others provide secure but easy interfaces to HDFS?

Kind regards,
Evert Lammerts



From general-return-2458-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 30 15:27:08 2010
Return-Path: <general-return-2458-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 61201 invoked from network); 30 Nov 2010 15:27:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Nov 2010 15:27:07 -0000
Received: (qmail 94649 invoked by uid 500); 30 Nov 2010 15:27:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94371 invoked by uid 500); 30 Nov 2010 15:27:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94340 invoked by uid 99); 30 Nov 2010 15:27:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Nov 2010 15:27:04 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of agrawalprachi88@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Nov 2010 15:26:58 +0000
Received: by wwb17 with SMTP id 17so2954100wwb.29
        for <general@hadoop.apache.org>; Tue, 30 Nov 2010 07:26:37 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=CcoS/j4JC9z53i/eXfyh6NRwt53kBeTG3tCBPbzZFRs=;
        b=Z8iRRwV/LwC84BGx+Dp1UvCDbmc4fEg7quQ1rn9RvJITlQ6p+C+iytxdgHmebWy99P
         FxqhWm+DmB5XdYxSzCaJahfTrlEITwxIQpW0STt0B3Iumn8NdPXk2fHExGp+IKgChjgN
         SczKOCA0LzzlSLFOntjxMkFq666lfAFvw17G4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=kL/PommenkNH86j5fehg0zY66mq7zV51l5lk0e1/jmjyFGB7HGefl/K8HRYXJ/Z3tB
         Xwi9viJHiqvxt/Lo0/iJe067upsrH0FjQbjBOO3/RXYLZtbeh2Zrqds3wkdYucoZVM5W
         cZs32vICy/8C3wKYKGMucNt0PXjIoo3z+RzdU=
MIME-Version: 1.0
Received: by 10.227.142.85 with SMTP id p21mr7921092wbu.150.1291130795025;
 Tue, 30 Nov 2010 07:26:35 -0800 (PST)
Received: by 10.227.145.207 with HTTP; Tue, 30 Nov 2010 07:26:34 -0800 (PST)
Date: Tue, 30 Nov 2010 20:56:34 +0530
Message-ID: <AANLkTingBDYxGiMd5OiYOZnfnM4RxpsZX8NYPPCA_mAP@mail.gmail.com>
Subject: how to uninstall hue
From: prachi agrawal <agrawalprachi88@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e649c71a658714049646d026

--0016e649c71a658714049646d026
Content-Type: text/plain; charset=ISO-8859-1

i have installed hue using

apt-get install hue

it is installed on cloudera running in pseudo distributed mode

thanx in advance

--0016e649c71a658714049646d026--

From general-return-2459-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 30 15:42:12 2010
Return-Path: <general-return-2459-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67161 invoked from network); 30 Nov 2010 15:42:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Nov 2010 15:42:12 -0000
Received: (qmail 23989 invoked by uid 500); 30 Nov 2010 15:42:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23787 invoked by uid 500); 30 Nov 2010 15:42:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23779 invoked by uid 99); 30 Nov 2010 15:42:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Nov 2010 15:42:10 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bernd.fondermann@googlemail.com designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Nov 2010 15:42:05 +0000
Received: by ewy9 with SMTP id 9so2900238ewy.35
        for <general@hadoop.apache.org>; Tue, 30 Nov 2010 07:41:43 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=Hu2oRVAFMHtTe+tNHk1/H8r5wN3sr2CEHNu4TorVzTk=;
        b=QqIgw+//ad+tiw7V6Jj3INB3sdKZeU00Oerm+q8F5gr0FAZ/tCK8ByKnU5Or6tL6Ag
         fL36J9SQeuy+mpROeU700VvZb2CZ2X/idIqgW5MENp16MxJK0qbOx9Wu8PG9VtZFKy2C
         1aIdmVKPk5GlXLMnOJms8/9fbAEa09CsKv3EY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=bdk2sh57TRnqL9X2rxrA03a2WcO+vbe+dqfhM6tRvhGCKvZAq0bGfOS+wm1wPvYIyZ
         c0AQrxtD0JHNnAlUt6dWxUt/3+dTUYU+d0jMz3uHKCuQBVaOOdezzpz+5wPVfXaxDfMC
         Ovy6+S8qv04Aq6lADxYDHHaCC/tWEjZBOuu5w=
MIME-Version: 1.0
Received: by 10.213.31.78 with SMTP id x14mr2635417ebc.1.1291131703131; Tue,
 30 Nov 2010 07:41:43 -0800 (PST)
Received: by 10.213.105.83 with HTTP; Tue, 30 Nov 2010 07:41:43 -0800 (PST)
In-Reply-To: <AANLkTingBDYxGiMd5OiYOZnfnM4RxpsZX8NYPPCA_mAP@mail.gmail.com>
References: <AANLkTingBDYxGiMd5OiYOZnfnM4RxpsZX8NYPPCA_mAP@mail.gmail.com>
Date: Tue, 30 Nov 2010 16:41:43 +0100
Message-ID: <AANLkTikKOG9qu=7rUq0HPmkCLM3LA=nNy_Sn_n_asEuL@mail.gmail.com>
Subject: Re: how to uninstall hue
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hi,

Please contact the vendor of this product for any support.

  Bernd

On Tue, Nov 30, 2010 at 16:26, prachi agrawal <agrawalprachi88@gmail.com> wrote:
> i have installed hue using
>
> apt-get install hue
>
> it is installed on cloudera running in pseudo distributed mode
>
> thanx in advance
>

From general-return-2460-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 30 15:45:33 2010
Return-Path: <general-return-2460-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67905 invoked from network); 30 Nov 2010 15:45:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Nov 2010 15:45:32 -0000
Received: (qmail 29331 invoked by uid 500); 30 Nov 2010 15:45:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29282 invoked by uid 500); 30 Nov 2010 15:45:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29274 invoked by uid 99); 30 Nov 2010 15:45:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Nov 2010 15:45:30 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of msegel@navteq.com designates 65.167.11.240 as permitted sender)
Received: from [65.167.11.240] (HELO xmailchi.navteq.com) (65.167.11.240)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Nov 2010 15:45:24 +0000
Received: from chicago-spamfw.navteq.com (imailchi [10.8.5.161])
	by xmailchi.navteq.com (8.13.8/8.13.6) with ESMTP id oAUFGhZ5012250
	for <general@hadoop.apache.org>; Tue, 30 Nov 2010 09:16:43 -0600
Received: from imailchi.hq.navteq.com (localhost [127.0.0.1])
	by chicago-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id 180C243B5136
	for <general@hadoop.apache.org>; Tue, 30 Nov 2010 09:45:00 -0600 (CST)
Received: from imailchi.hq.navteq.com (imailchi.hq.navteq.com [10.8.120.104]) by chicago-spamfw.navteq.com with ESMTP id BLXHHPIRtPs5KEii for <general@hadoop.apache.org>; Tue, 30 Nov 2010 09:45:00 -0600 (CST)
Received: from hq-ex-ht02.ad.navteq.com (hq-ex-ht02.ad.navteq.com [10.8.222.172])
	by imailchi.hq.navteq.com (8.12.9/8.12.9) with ESMTP id oAUFixtf024044
	for <general@hadoop.apache.org>; Tue, 30 Nov 2010 09:44:59 -0600
Received: from hq-ex-mb02.ad.navteq.com ([fe80::880d:318e:e860:a8db]) by
 hq-ex-ht02.ad.navteq.com ([fe80::24f6:ca50:734d:df6e%14]) with mapi; Tue, 30
 Nov 2010 09:43:20 -0600
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 30 Nov 2010 09:44:58 -0600
Subject: RE: how to uninstall hue
Thread-Topic: how to uninstall hue
Thread-Index: AcuQpSiwh3p6Vi17S76E7Vp8OUqsOQAABu/A
Message-ID: <ABC24175AFD3BE4DA15F4CD375ED413D063AC0E689@hq-ex-mb02.ad.navteq.com>
References: <AANLkTingBDYxGiMd5OiYOZnfnM4RxpsZX8NYPPCA_mAP@mail.gmail.com>
 <AANLkTikKOG9qu=7rUq0HPmkCLM3LA=nNy_Sn_n_asEuL@mail.gmail.com>
In-Reply-To: <AANLkTikKOG9qu=7rUq0HPmkCLM3LA=nNy_Sn_n_asEuL@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1
X-Virus-Checked: Checked by ClamAV on apache.org

The first thing you want to do is to edit the configuration files and rem=
ove references to the hue plug in. Then after you restart, Hue won't work=
 and you don't have to worry about the classes going away when you remove=
 the product.

Just did a quick google and here's the top result from the search:
http://www.linuxquestions.org/questions/debian-26/how-do-i-get-apt-get-to=
-completely-uninstall-a-package-237772/

That should help you get started....


-----Original Message-----
From: Bernd Fondermann [mailto:bernd.fondermann@googlemail.com]=20
Sent: Tuesday, November 30, 2010 9:42 AM
To: general@hadoop.apache.org
Subject: Re: how to uninstall hue

Hi,

Please contact the vendor of this product for any support.

  Bernd

On Tue, Nov 30, 2010 at 16:26, prachi agrawal <agrawalprachi88@gmail.com>=
 wrote:
> i have installed hue using
>
> apt-get install hue
>
> it is installed on cloudera running in pseudo distributed mode
>
> thanx in advance
>


The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.

From general-return-2461-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 30 16:06:22 2010
Return-Path: <general-return-2461-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81800 invoked from network); 30 Nov 2010 16:06:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Nov 2010 16:06:22 -0000
Received: (qmail 72673 invoked by uid 500); 30 Nov 2010 16:06:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72450 invoked by uid 500); 30 Nov 2010 16:06:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72201 invoked by uid 99); 30 Nov 2010 16:06:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Nov 2010 16:06:15 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jcoveney@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Nov 2010 16:06:07 +0000
Received: by bwz9 with SMTP id 9so5717318bwz.35
        for <general@hadoop.apache.org>; Tue, 30 Nov 2010 08:05:47 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=SycUCF+tUxWij62uVW/wJyJ/83gStdTYpz99VJ+/prw=;
        b=bY5dVQvRoAOlJd6t+6j5MbkcnosfEMx/hSm5fvNhGC/xS3gQIqKCD/fJfnNXh5hqQQ
         60c35rGBxpGHZqgG8bwIGBrT8Y0IKr2IuidKA81jTcLuT8x7XH6Jgw28Z1EcdXu9rTHU
         oHsZLgb7M1RS1M23E0IKB1sxm3/hwK/OzrIRs=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=oLJC9z07NqBswvPsnNykfyQojONhmzWvtubb8TpxO6dVq8qq614pBg5U6xZZbvp0Z2
         Mk4ck3srjCgkDk734Lhbyft6NyH2dE86J59kCFdSbeW1CGUkrE9qaSTrTch0a1n/2iYW
         5o8oQNXbl/hWNIoXtoGTZjE3Sh9QWgfxAeeEc=
MIME-Version: 1.0
Received: by 10.204.101.11 with SMTP id a11mr6762342bko.159.1291133146860;
 Tue, 30 Nov 2010 08:05:46 -0800 (PST)
Received: by 10.204.64.199 with HTTP; Tue, 30 Nov 2010 08:05:46 -0800 (PST)
Date: Tue, 30 Nov 2010 11:05:46 -0500
Message-ID: <AANLkTins=pKV3Zb_PUE9Stch0wCqS4DQ0dcjmjM4dMnb@mail.gmail.com>
Subject: Help: frustration with types and whatnot while trying to do a
 conditional sum
From: Jonathan Coveney <jcoveney@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6dd98b89391a60496475ca3
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6dd98b89391a60496475ca3
Content-Type: text/plain; charset=ISO-8859-1

I appreciate any help you can give. I've searched around and haven't found
anything directly related... I've gone through documentation but can't find
a real reason why this doesn't work.

Here is the jist of my code (n1 is arbitrary, just to group by, n2 is either
null or a large integer):

table = LOAD stuff AS (n1:chararray, n2:chararray, other irrelevant stuff);
pared = foreach table generate n1, n2;
grouped = group pared by n1;
counted  = foreach grouped generate group, (double)SUM((IsEmpty(pared.n2) ?
0:1))/(double)COUNT(pared.n2) as ratio:double;
ordered = order counted by ratio desc;
limited = limit ordered 200;
dump limited;

This gets this error:

ERROR 1045: Could not infer the matching function for
org.apache.pig.builtin.SUM as multiple or none of them fit. Please use an
explicit cast.

If I take out the double parenthesis in the counted sum

ERROR 1000: Error during parsing. Invalid alias: SUM in {group:
chararray,pared: {n1: chararray,n2: chararray}}

I THINK the error is that sum wants the column of a bag as an input, not
actual integers...so I thought I'd try and make that happen by making the
input take the form I want.

So in order to try and get around this, I thought this might work (changing
only these lines)

pared = foreach beacon_fact generate n1, (IsEmpty(n2) ? 0 : 1) as ooz:int;
grouped = group pared by n1;
counted  = foreach grouped generate group,
(double)SUM(pared.n1)/(double)COUNT(pared.n2) as ratio:double;

But this gives this error:
ERROR 1000: Error during parsing. Invalid alias: n2 in {n1: chararray,ooz:
int}

I have no real clue why this fails... I tried breaking it up into two steps
and it doesn't matter.

I'd ideally like to do this without making a UDF, as I feel the base
functionality should support it. Not sure.

Either way, I'd appreciate any help or pointers, as well as any rationale as
to why it does or doesn't work within the pig framework. The whole bag
system is still somewhat counterintuitive.

Thank you for your time

--0016e6dd98b89391a60496475ca3--

From general-return-2462-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 30 16:14:42 2010
Return-Path: <general-return-2462-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85273 invoked from network); 30 Nov 2010 16:14:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Nov 2010 16:14:42 -0000
Received: (qmail 92692 invoked by uid 500); 30 Nov 2010 16:14:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92530 invoked by uid 500); 30 Nov 2010 16:14:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 92509 invoked by uid 99); 30 Nov 2010 16:14:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Nov 2010 16:14:39 +0000
X-ASF-Spam-Status: No, hits=-1997.8 required=10.0
	tests=ALL_TRUSTED,HTML_MESSAGE,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 30 Nov 2010 16:14:39 +0000
Received: (qmail 85162 invoked by uid 99); 30 Nov 2010 16:14:18 -0000
Received: from localhost.apache.org (HELO mail-bw0-f48.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Nov 2010 16:14:18 +0000
Received: by bwz9 with SMTP id 9so5725066bwz.35
        for <general@hadoop.apache.org>; Tue, 30 Nov 2010 08:14:16 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.126.5 with SMTP id a5mr6946529bks.177.1291133654953; Tue,
 30 Nov 2010 08:14:14 -0800 (PST)
Received: by 10.204.57.75 with HTTP; Tue, 30 Nov 2010 08:14:14 -0800 (PST)
In-Reply-To: <AANLkTins=pKV3Zb_PUE9Stch0wCqS4DQ0dcjmjM4dMnb@mail.gmail.com>
References: <AANLkTins=pKV3Zb_PUE9Stch0wCqS4DQ0dcjmjM4dMnb@mail.gmail.com>
Date: Tue, 30 Nov 2010 08:14:14 -0800
Message-ID: <AANLkTi=dBPnBWTbt158NgAote8bShGwqjWbsQXuqvF4n@mail.gmail.com>
Subject: Re: Help: frustration with types and whatnot while trying to do a
 conditional sum
From: "Owen O'Malley" <omalley@apache.org>
To: user@pig.apache.org
Content-Type: multipart/alternative; boundary=0016e6d625f8dd941a0496477a77

--0016e6d625f8dd941a0496477a77
Content-Type: text/plain; charset=UTF-8

Pig has moved to its own mailing lists. Please follow up over there.
-- Owen

On Tue, Nov 30, 2010 at 8:05 AM, Jonathan Coveney <jcoveney@gmail.com>wrote:

> I appreciate any help you can give. I've searched around and haven't found
> anything directly related... I've gone through documentation but can't find
> a real reason why this doesn't work.
>
> Here is the jist of my code (n1 is arbitrary, just to group by, n2 is
> either
> null or a large integer):
>
> table = LOAD stuff AS (n1:chararray, n2:chararray, other irrelevant stuff);
> pared = foreach table generate n1, n2;
> grouped = group pared by n1;
> counted  = foreach grouped generate group, (double)SUM((IsEmpty(pared.n2) ?
> 0:1))/(double)COUNT(pared.n2) as ratio:double;
> ordered = order counted by ratio desc;
> limited = limit ordered 200;
> dump limited;
>
> This gets this error:
>
> ERROR 1045: Could not infer the matching function for
> org.apache.pig.builtin.SUM as multiple or none of them fit. Please use an
> explicit cast.
>
> If I take out the double parenthesis in the counted sum
>
> ERROR 1000: Error during parsing. Invalid alias: SUM in {group:
> chararray,pared: {n1: chararray,n2: chararray}}
>
> I THINK the error is that sum wants the column of a bag as an input, not
> actual integers...so I thought I'd try and make that happen by making the
> input take the form I want.
>
> So in order to try and get around this, I thought this might work (changing
> only these lines)
>
> pared = foreach beacon_fact generate n1, (IsEmpty(n2) ? 0 : 1) as ooz:int;
> grouped = group pared by n1;
> counted  = foreach grouped generate group,
> (double)SUM(pared.n1)/(double)COUNT(pared.n2) as ratio:double;
>
> But this gives this error:
> ERROR 1000: Error during parsing. Invalid alias: n2 in {n1: chararray,ooz:
> int}
>
> I have no real clue why this fails... I tried breaking it up into two steps
> and it doesn't matter.
>
> I'd ideally like to do this without making a UDF, as I feel the base
> functionality should support it. Not sure.
>
> Either way, I'd appreciate any help or pointers, as well as any rationale
> as
> to why it does or doesn't work within the pig framework. The whole bag
> system is still somewhat counterintuitive.
>
> Thank you for your time
>

--0016e6d625f8dd941a0496477a77--

From general-return-2463-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 30 16:16:48 2010
Return-Path: <general-return-2463-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 86140 invoked from network); 30 Nov 2010 16:16:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Nov 2010 16:16:48 -0000
Received: (qmail 99593 invoked by uid 500); 30 Nov 2010 16:16:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99450 invoked by uid 500); 30 Nov 2010 16:16:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 99441 invoked by uid 99); 30 Nov 2010 16:16:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Nov 2010 16:16:46 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jcoveney@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Nov 2010 16:16:39 +0000
Received: by bwz9 with SMTP id 9so5727023bwz.35
        for <general@hadoop.apache.org>; Tue, 30 Nov 2010 08:16:15 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=iJQ1AoYW3/0ZODeSwO/sJkuku4tsd1hAzQDJJqFaquk=;
        b=jb6pCs8xLCTsTDZqsf7M5bysd4b33OIGHv3iY67B9ivJf/ZfCAoc4f9Ar4j5nbFNLZ
         E2pd8DxtRnRY5vsaPK4ftKQodoM3ftiA89x/ovx0nREy4RLlPJUOuiTG21QpfQhXzWH4
         6kln7D4c80zdyySon0EbzrR658MGkRsQ223/Y=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=pDlRPd10rHyaqRlEdrNwNoHyM516tyckr9bFnUemChJlDfRb1fi9sX0EpN8zuzJVIO
         eKTBuV0ZH68aD1Zz9dksRUlnj4dA24AyPL1hb6G+ePtezDNld3gcHgSSdVD39TopW8Qw
         b+CuqBXN5S5lbPtHDHEM2yQWTKkKYtz+gg8K4=
MIME-Version: 1.0
Received: by 10.204.75.194 with SMTP id z2mr6916777bkj.132.1291133774991; Tue,
 30 Nov 2010 08:16:14 -0800 (PST)
Received: by 10.204.64.199 with HTTP; Tue, 30 Nov 2010 08:16:14 -0800 (PST)
In-Reply-To: <AANLkTi=dBPnBWTbt158NgAote8bShGwqjWbsQXuqvF4n@mail.gmail.com>
References: <AANLkTins=pKV3Zb_PUE9Stch0wCqS4DQ0dcjmjM4dMnb@mail.gmail.com>
	<AANLkTi=dBPnBWTbt158NgAote8bShGwqjWbsQXuqvF4n@mail.gmail.com>
Date: Tue, 30 Nov 2010 11:16:14 -0500
Message-ID: <AANLkTimHH4wwXgxv+Ox30Mk4cKtyOHsO-WV1n6OXNMaH@mail.gmail.com>
Subject: Re: Help: frustration with types and whatnot while trying to do a
 conditional sum
From: Jonathan Coveney <jcoveney@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f7d7d20414fc04964782e3

--001485f7d7d20414fc04964782e3
Content-Type: text/plain; charset=ISO-8859-1

I am sorry, I got confused. I will do that.

2010/11/30 Owen O'Malley <omalley@apache.org>

> Pig has moved to its own mailing lists. Please follow up over there.
> -- Owen
>
> On Tue, Nov 30, 2010 at 8:05 AM, Jonathan Coveney <jcoveney@gmail.com
> >wrote:
>
> > I appreciate any help you can give. I've searched around and haven't
> found
> > anything directly related... I've gone through documentation but can't
> find
> > a real reason why this doesn't work.
> >
> > Here is the jist of my code (n1 is arbitrary, just to group by, n2 is
> > either
> > null or a large integer):
> >
> > table = LOAD stuff AS (n1:chararray, n2:chararray, other irrelevant
> stuff);
> > pared = foreach table generate n1, n2;
> > grouped = group pared by n1;
> > counted  = foreach grouped generate group, (double)SUM((IsEmpty(pared.n2)
> ?
> > 0:1))/(double)COUNT(pared.n2) as ratio:double;
> > ordered = order counted by ratio desc;
> > limited = limit ordered 200;
> > dump limited;
> >
> > This gets this error:
> >
> > ERROR 1045: Could not infer the matching function for
> > org.apache.pig.builtin.SUM as multiple or none of them fit. Please use an
> > explicit cast.
> >
> > If I take out the double parenthesis in the counted sum
> >
> > ERROR 1000: Error during parsing. Invalid alias: SUM in {group:
> > chararray,pared: {n1: chararray,n2: chararray}}
> >
> > I THINK the error is that sum wants the column of a bag as an input, not
> > actual integers...so I thought I'd try and make that happen by making the
> > input take the form I want.
> >
> > So in order to try and get around this, I thought this might work
> (changing
> > only these lines)
> >
> > pared = foreach beacon_fact generate n1, (IsEmpty(n2) ? 0 : 1) as
> ooz:int;
> > grouped = group pared by n1;
> > counted  = foreach grouped generate group,
> > (double)SUM(pared.n1)/(double)COUNT(pared.n2) as ratio:double;
> >
> > But this gives this error:
> > ERROR 1000: Error during parsing. Invalid alias: n2 in {n1:
> chararray,ooz:
> > int}
> >
> > I have no real clue why this fails... I tried breaking it up into two
> steps
> > and it doesn't matter.
> >
> > I'd ideally like to do this without making a UDF, as I feel the base
> > functionality should support it. Not sure.
> >
> > Either way, I'd appreciate any help or pointers, as well as any rationale
> > as
> > to why it does or doesn't work within the pig framework. The whole bag
> > system is still somewhat counterintuitive.
> >
> > Thank you for your time
> >
>

--001485f7d7d20414fc04964782e3--

From general-return-2464-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 30 16:32:35 2010
Return-Path: <general-return-2464-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 92179 invoked from network); 30 Nov 2010 16:32:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Nov 2010 16:32:34 -0000
Received: (qmail 24979 invoked by uid 500); 30 Nov 2010 16:32:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23674 invoked by uid 500); 30 Nov 2010 16:32:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23312 invoked by uid 99); 30 Nov 2010 16:32:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Nov 2010 16:32:30 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.161.176] (HELO mail-gx0-f176.google.com) (209.85.161.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Nov 2010 16:32:24 +0000
Received: by gxk4 with SMTP id 4so3318828gxk.35
        for <general@hadoop.apache.org>; Tue, 30 Nov 2010 08:32:03 -0800 (PST)
Received: by 10.229.91.194 with SMTP id o2mr6231944qcm.250.1291134722418; Tue,
 30 Nov 2010 08:32:02 -0800 (PST)
MIME-Version: 1.0
Received: by 10.229.251.205 with HTTP; Tue, 30 Nov 2010 08:31:42 -0800 (PST)
In-Reply-To: <AANLkTikKOG9qu=7rUq0HPmkCLM3LA=nNy_Sn_n_asEuL@mail.gmail.com>
References: <AANLkTingBDYxGiMd5OiYOZnfnM4RxpsZX8NYPPCA_mAP@mail.gmail.com> <AANLkTikKOG9qu=7rUq0HPmkCLM3LA=nNy_Sn_n_asEuL@mail.gmail.com>
From: Philip Zeyliger <philip@cloudera.com>
Date: Tue, 30 Nov 2010 08:31:42 -0800
Message-ID: <AANLkTi=G3NmvVbhs+nPr-7b-a5+5OKaiuTv_Q0kc-XcK@mail.gmail.com>
Subject: Re: how to uninstall hue
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Bernd is referring to the mailing list at
https://groups.google.com/a/cloudera.org/group/hue-user/topics, which
is the right place for discussions of Hue.

-- Philip

On Tue, Nov 30, 2010 at 7:41 AM, Bernd Fondermann
<bernd.fondermann@googlemail.com> wrote:
> Hi,
>
> Please contact the vendor of this product for any support.
>
> =A0Bernd
>
> On Tue, Nov 30, 2010 at 16:26, prachi agrawal <agrawalprachi88@gmail.com>=
 wrote:
>> i have installed hue using
>>
>> apt-get install hue
>>
>> it is installed on cloudera running in pseudo distributed mode
>>
>> thanx in advance
>>
>

From general-return-148-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 02 22:05:44 2009
Return-Path: <general-return-148-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85684 invoked from network); 2 Apr 2009 22:05:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Apr 2009 22:05:43 -0000
Received: (qmail 25537 invoked by uid 500); 2 Apr 2009 22:05:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25478 invoked by uid 500); 2 Apr 2009 22:05:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25468 invoked by uid 99); 2 Apr 2009 22:05:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Apr 2009 22:05:42 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cutting@gmail.com designates 209.85.200.175 as permitted sender)
Received: from [209.85.200.175] (HELO wf-out-1314.google.com) (209.85.200.175)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Apr 2009 22:05:33 +0000
Received: by wf-out-1314.google.com with SMTP id 23so786625wfg.2
        for <general@hadoop.apache.org>; Thu, 02 Apr 2009 15:05:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:sender:message-id:date:from
         :user-agent:mime-version:to:subject:content-type
         :content-transfer-encoding;
        bh=15CxOtfQuEu3JISIDNL6t7h7J0fZqffuVDdvbDeZHHY=;
        b=IO0Co4YcuxDe4V5rmYWhrzqQgB0kAdMOE1jejCgGvFMAENhTKVgTiIrdjVwlnGW6Sr
         OxSXaXjtsjaujgxMHvUKaElZia12Tqwsgjpc5ZMZMXVGDQyMbIOVkOZOFHwz9zyeFgKy
         S3jQAaHGnFP/PkV+pwykOyNa6C2tp3HQrutSQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:date:from:user-agent:mime-version:to:subject
         :content-type:content-transfer-encoding;
        b=eCTmycRqveuFrYTkIaVfW51kZSCjkk1Nfz0GW6aYAv0DfIq0lDiJdgW9z5CSQ73/IP
         cd24ddjtOvDukP03jLWHbrorLMvhu6zaNDW9jaGob4ewI6a21ryV3qcucDFwNnmq4ipE
         JRzWzKZGXkmx4gr28rTZNUjtyL/+wNjT+1H4c=
Received: by 10.142.156.19 with SMTP id d19mr115506wfe.6.1238709912371;
        Thu, 02 Apr 2009 15:05:12 -0700 (PDT)
Received: from ?192.168.168.16? (c-76-103-155-128.hsd1.ca.comcast.net [76.103.155.128])
        by mx.google.com with ESMTPS id 24sm2202336wfc.57.2009.04.02.15.05.11
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Thu, 02 Apr 2009 15:05:11 -0700 (PDT)
Sender: Doug Cutting <cutting@gmail.com>
Message-ID: <49D53694.1050906@apache.org>
Date: Thu, 02 Apr 2009 15:05:08 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.21 (X11/20090318)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: [PROPOSAL] new subproject: Avro
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

I propose we add a new Hadoop subproject for Avro, a serialization 
system.  My ambition is for Avro to replace both Hadoop's RPC and to be 
used for most Hadoop data files, e.g., by Pig, Hive, etc.

Initial committers would be Sharad Agarwal and me, both existing Hadoop 
committers.  We are the sole authors of this software to date.

The code is currently at:

http://people.apache.org/~cutting/avro.git/

To learn more:

git clone http://people.apache.org/~cutting/avro.git/ avro
cat avro/README.txt

Comments?  Questions?

Doug

From general-return-149-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 02 22:20:47 2009
Return-Path: <general-return-149-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95062 invoked from network); 2 Apr 2009 22:20:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Apr 2009 22:20:43 -0000
Received: (qmail 42241 invoked by uid 500); 2 Apr 2009 22:20:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42164 invoked by uid 500); 2 Apr 2009 22:20:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42154 invoked by uid 99); 2 Apr 2009 22:20:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Apr 2009 22:20:43 +0000
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.62.80] (HELO QMTA08.westchester.pa.mail.comcast.net) (76.96.62.80)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Apr 2009 22:20:33 +0000
Received: from OMTA02.westchester.pa.mail.comcast.net ([76.96.62.19])
	by QMTA08.westchester.pa.mail.comcast.net with comcast
	id aaKW1b0030QuhwU58mL1Jz; Thu, 02 Apr 2009 22:20:01 +0000
Received: from battlerock-lm.corp.yahoo.com ([209.131.62.113])
	by OMTA02.westchester.pa.mail.comcast.net with comcast
	id amL31b00N2SbwD53NmL6xa; Thu, 02 Apr 2009 22:20:10 +0000
Message-Id: <4CB9034E-05FB-4200-AF55-FFD78B2EEFCE@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <49D53694.1050906@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: Re: [PROPOSAL] new subproject: Avro
Date: Thu, 2 Apr 2009 15:20:01 -0700
References: <49D53694.1050906@apache.org>
X-Mailer: Apple Mail (2.930.3)
X-Virus-Checked: Checked by ClamAV on apache.org


On Apr 2, 2009, at 3:05 PM, Doug Cutting wrote:

> I propose we add a new Hadoop subproject for Avro, a serialization  
> system.

+1. Even if Justin comes after me for allowing another Jakarta. *smile*

-- Owen

From general-return-150-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 03 00:12:07 2009
Return-Path: <general-return-150-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22381 invoked from network); 3 Apr 2009 00:12:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Apr 2009 00:12:07 -0000
Received: (qmail 44559 invoked by uid 500); 3 Apr 2009 00:12:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44484 invoked by uid 500); 3 Apr 2009 00:12:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44474 invoked by uid 99); 3 Apr 2009 00:12:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 00:12:06 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of vermaabhishekp@gmail.com designates 209.85.217.164 as permitted sender)
Received: from [209.85.217.164] (HELO mail-gx0-f164.google.com) (209.85.217.164)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 00:11:59 +0000
Received: by gxk8 with SMTP id 8so1871316gxk.5
        for <general@hadoop.apache.org>; Thu, 02 Apr 2009 17:11:38 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=PgcczehY2Rt6G/XEOxX4gG0AOUnDBJ+bouPt2FOZSJI=;
        b=So51eUsB0IhBQn5YPNHJfH00GsysqPgISnF2PthRMewY4xoeoPrNkKouYex2USECSh
         CNw9qhi6saSTYZSzIs/dQhlk9R/P2VKEOPOFYuRpOFYUHeKTYx5PGcY71yVP95LnsE2m
         0vLErtswVYcmJVRSAZpdXrOfPVxJmIeQEJSS0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Yat3+nZXYg4sMhH70f/g1YmOtxrvkOppi1HSshQBRWmUxOn4QG7rHZdXD0LJBCMab7
         fuQVe4QY/Z9Hf2flv7b3QzmpH2Sduiz71ngUFLO1iWWbFsvZObpio9MQWrnnqn5YVejK
         oVvaky5CAat0AiymwEDrUatO7RMQ6LTo4sXQc=
MIME-Version: 1.0
Received: by 10.151.9.17 with SMTP id m17mr1067760ybi.173.1238717498192; Thu, 
	02 Apr 2009 17:11:38 -0700 (PDT)
In-Reply-To: <4CB9034E-05FB-4200-AF55-FFD78B2EEFCE@apache.org>
References: <49D53694.1050906@apache.org>
	 <4CB9034E-05FB-4200-AF55-FFD78B2EEFCE@apache.org>
Date: Thu, 2 Apr 2009 19:11:38 -0500
Message-ID: <3c682ecd0904021711x41fe4dd2j291f2077284d5558@mail.gmail.com>
Subject: Re: [PROPOSAL] new subproject: Avro
From: Abhishek Verma <vermaabhishekp@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd47b067505ac04669b647c
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd47b067505ac04669b647c
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

I am a newbie here. Why not use something existing like protocol buffers :
http://code.google.com/p/protobuf/ which is open source and works amazingly
well.

On Thu, Apr 2, 2009 at 5:20 PM, Owen O'Malley <omalley@apache.org> wrote:

>
> On Apr 2, 2009, at 3:05 PM, Doug Cutting wrote:
>
>  I propose we add a new Hadoop subproject for Avro, a serialization system.
>>
>
> +1. Even if Justin comes after me for allowing another Jakarta. *smile*
>
> -- Owen
>

--000e0cd47b067505ac04669b647c--

From general-return-151-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 03 04:29:36 2009
Return-Path: <general-return-151-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96586 invoked from network); 3 Apr 2009 04:29:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Apr 2009 04:29:36 -0000
Received: (qmail 29016 invoked by uid 500); 3 Apr 2009 04:29:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28925 invoked by uid 500); 3 Apr 2009 04:29:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28912 invoked by uid 99); 3 Apr 2009 04:29:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 04:29:35 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.62.32] (HELO QMTA03.westchester.pa.mail.comcast.net) (76.96.62.32)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 04:29:25 +0000
Received: from OMTA09.westchester.pa.mail.comcast.net ([76.96.62.20])
	by QMTA03.westchester.pa.mail.comcast.net with comcast
	id arrU1b0070SCNGk53sUyiw; Fri, 03 Apr 2009 04:28:58 +0000
Received: from [10.0.0.13] ([209.131.62.115])
	by OMTA09.westchester.pa.mail.comcast.net with comcast
	id asUv1b0072VBGtd3VsUy6x; Fri, 03 Apr 2009 04:29:03 +0000
Message-Id: <8BBAB2C9-FCF9-4261-9E4B-282CD4196FA2@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <3c682ecd0904021711x41fe4dd2j291f2077284d5558@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: Re: [PROPOSAL] new subproject: Avro
Date: Thu, 2 Apr 2009 21:28:53 -0700
References: <49D53694.1050906@apache.org> <4CB9034E-05FB-4200-AF55-FFD78B2EEFCE@apache.org> <3c682ecd0904021711x41fe4dd2j291f2077284d5558@mail.gmail.com>
X-Mailer: Apple Mail (2.930.3)
X-Virus-Checked: Checked by ClamAV on apache.org


On Apr 2, 2009, at 5:11 PM, Abhishek Verma wrote:

> I am a newbie here. Why not use something existing like protocol  
> buffers :
> http://code.google.com/p/protobuf/ which is open source and works  
> amazingly
> well.

There are two blockers for protocol buffers that make them suboptimal  
for Hadoop. They are:

1. Protocol buffers are open source, but the community isn't open.  
Google doesn't seem interested in getting patches from outside of  
itself. If we needed something changed in protocol buffers, we'd end  
up needing to fork the project to make any progress.

2. Protocol buffers (and thrift) encode the field names as id numbers.  
That means that if you read them into dynamic language like Python  
that it has to use the field numbers instead of the field names. In  
Avro, the field names are saved and there are no field ids.

A final point is that since the schema isn't inlined in Avro, the  
binary representation is much tighter than protocol buffers.

-- Owen

From general-return-152-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 03 16:07:21 2009
Return-Path: <general-return-152-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68108 invoked from network); 3 Apr 2009 16:07:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Apr 2009 16:07:21 -0000
Received: (qmail 66183 invoked by uid 500); 3 Apr 2009 16:07:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66120 invoked by uid 500); 3 Apr 2009 16:07:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66110 invoked by uid 99); 3 Apr 2009 16:07:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 16:07:21 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cutting@gmail.com designates 209.85.198.236 as permitted sender)
Received: from [209.85.198.236] (HELO rv-out-0506.google.com) (209.85.198.236)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 16:07:11 +0000
Received: by rv-out-0506.google.com with SMTP id k40so1031527rvb.29
        for <general@hadoop.apache.org>; Fri, 03 Apr 2009 09:06:50 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:sender:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=z99Eb/dwtmLhbuyCDLGb0vEm/xEfREuGws/ngueGBuk=;
        b=T0Wqb6/9YekUYzSrtUUFNtGrEhW6jblA77WwcrAQB1Izkk0850vRoMQgM7L2K+bDAu
         C4SPCgOcaeKuh1q+v7jUFRH0Z7v5rkOP20E/+gt0fqMKs3Ni/gvKZ2amwkcNGUOuiL7t
         vqA8RI4Y4jgITN/tWvh6FHn1oezPdyyJSX1KA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:date:from:user-agent:mime-version:to:subject
         :references:in-reply-to:content-type:content-transfer-encoding;
        b=oM1y737FvMRKTw08+BfDLikiWQ+DWi5q6SzjCIWJDGKn05RMJVGH02gMF8QP0xmUYD
         is8W/mDvjuj/wNP0tRh24RftqE0qhYfE6nVTTHd5f+Ag9HUDDx3Vl/a+YvbNn1ZL26DB
         B+mQr4Ct1X1VYZPhno4mTvkxAssZdQAcv7UrA=
Received: by 10.142.179.2 with SMTP id b2mr365670wff.308.1238774810498;
        Fri, 03 Apr 2009 09:06:50 -0700 (PDT)
Received: from ?192.168.168.16? (c-76-103-155-128.hsd1.ca.comcast.net [76.103.155.128])
        by mx.google.com with ESMTPS id 24sm3195874wfc.37.2009.04.03.09.06.48
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Fri, 03 Apr 2009 09:06:49 -0700 (PDT)
Sender: Doug Cutting <cutting@gmail.com>
Message-ID: <49D63415.8060004@apache.org>
Date: Fri, 03 Apr 2009 09:06:45 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.21 (X11/20090318)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [PROPOSAL] new subproject: Avro
References: <49D53694.1050906@apache.org> <4CB9034E-05FB-4200-AF55-FFD78B2EEFCE@apache.org> <3c682ecd0904021711x41fe4dd2j291f2077284d5558@mail.gmail.com> <8BBAB2C9-FCF9-4261-9E4B-282CD4196FA2@apache.org>
In-Reply-To: <8BBAB2C9-FCF9-4261-9E4B-282CD4196FA2@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Owen O'Malley wrote:
> 2. Protocol buffers (and thrift) encode the field names as id numbers. 
> That means that if you read them into dynamic language like Python that 
> it has to use the field numbers instead of the field names. In Avro, the 
> field names are saved and there are no field ids.

This hints at a related problem with Thrift and Protocol Buffers, which 
is that they require one to generate code for each datatype one 
processes.  This is awkward in dynamic environments, where one would 
like to write a script (Pig, Python, Perl, Hive, whatever) to process 
input data and generate output data, without having to locate the IDL 
for each input file, run an IDL compiler, load the generated code, 
generate an IDL file for the output, run the compiler again, load the 
output code and finally write your output.  Avro rather lets you simply 
open your inputs, examine their datatypes, specify output types and 
write them.

Avro's Java implementation currently includes three different data 
representations:

  - a "generic" representation uses a standard set of datastructures for 
all datatypes: records are represented as Map<String,Object>, arrays as 
List<Object>, longs as Long, etc.

  - a "reflect" representation uses Java reflection to permit one to 
read and write existing Java classes with Avro.

  - a "specific" representation generates Java classes that are compiled 
and loaded, much like Thrift and Protocol Buffers.

We don't expect most scripting languages to use more than a single 
representation.  Implementing Avro is quite simple, by design.  We have 
a Python implementation, and hope to add more soon.

Doug

From general-return-153-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 03 16:24:59 2009
Return-Path: <general-return-153-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76470 invoked from network); 3 Apr 2009 16:24:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Apr 2009 16:24:59 -0000
Received: (qmail 3726 invoked by uid 500); 3 Apr 2009 16:24:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3641 invoked by uid 500); 3 Apr 2009 16:24:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3590 invoked by uid 99); 3 Apr 2009 16:24:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 16:24:56 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bryan@rapleaf.com designates 216.74.32.93 as permitted sender)
Received: from [216.74.32.93] (HELO mail.rapleaf.com) (216.74.32.93)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 16:24:48 +0000
Received: from mail.rapleaf.com (localhost.localdomain [127.0.0.1])
	by mail.rapleaf.com (Postfix) with ESMTP id B4E9712502F9;
	Fri,  3 Apr 2009 09:24:27 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=rapleaf.com; q=dns; s=m1; b=D8xvc
	47iI34hHj2SOZqehAYdXYfMuRImdbUMWXuFkRomWeR/EURGu+AoEG6b3q90uChQy
	RzPpOK0N7Q9NOB6qAqh6p0Qyv5FcdIKnIi18CQJPBPHxWanBydu+qLYGnHCvLnuD
	FqSWx2diCKiYK9oHcwxBy8i726dmSkjKWzTTk0=
Received: from [10.100.18.119] (unknown [10.100.18.119])
	by mail.rapleaf.com (Postfix) with ESMTP id 98201125016F;
	Fri,  3 Apr 2009 09:24:27 -0700 (PDT)
In-Reply-To: <49D63415.8060004@apache.org>
References: <49D53694.1050906@apache.org> <4CB9034E-05FB-4200-AF55-FFD78B2EEFCE@apache.org> <3c682ecd0904021711x41fe4dd2j291f2077284d5558@mail.gmail.com> <8BBAB2C9-FCF9-4261-9E4B-282CD4196FA2@apache.org> <49D63415.8060004@apache.org>
Mime-Version: 1.0 (Apple Message framework v753.1)
Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed
Message-Id: <718F2DEF-B305-49C9-B62F-155D2F4CE12F@rapleaf.com>
Cc: thrift-user@incubator.apache.org,
 thrift-dev@incubator.apache.org
Content-Transfer-Encoding: 7bit
From: Bryan Duxbury <bryan@rapleaf.com>
Subject: Re: [PROPOSAL] new subproject: Avro
Date: Fri, 3 Apr 2009 09:24:26 -0700
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.753.1)
X-Virus-Checked: Checked by ClamAV on apache.org

It sounds like what you want is the option avoid pre-generated  
classes. If that's the only thing you need, it seems like we could  
bolt that on to Thrift with almost no work. I assume you'd have the  
schema stored in metadata or file header or something, right? (You  
wouldn't want to store the field names in the binary encoding as  
strings, since that would probably very quickly dwarf the size of the  
actual data in a lot of cases.)

If my assumptions are correct, it seems like it'd be a lot smarter to  
leverage existing Thrift infrastructure and encoding work rather than  
duplicating it for this lone feature.

-Bryan

On Apr 3, 2009, at 9:06 AM, Doug Cutting wrote:

> Owen O'Malley wrote:
>> 2. Protocol buffers (and thrift) encode the field names as id  
>> numbers. That means that if you read them into dynamic language  
>> like Python that it has to use the field numbers instead of the  
>> field names. In Avro, the field names are saved and there are no  
>> field ids.
>
> This hints at a related problem with Thrift and Protocol Buffers,  
> which is that they require one to generate code for each datatype  
> one processes.  This is awkward in dynamic environments, where one  
> would like to write a script (Pig, Python, Perl, Hive, whatever) to  
> process input data and generate output data, without having to  
> locate the IDL for each input file, run an IDL compiler, load the  
> generated code, generate an IDL file for the output, run the  
> compiler again, load the output code and finally write your  
> output.  Avro rather lets you simply open your inputs, examine  
> their datatypes, specify output types and write them.
>
> Avro's Java implementation currently includes three different data  
> representations:
>
>  - a "generic" representation uses a standard set of datastructures  
> for all datatypes: records are represented as Map<String,Object>,  
> arrays as List<Object>, longs as Long, etc.
>
>  - a "reflect" representation uses Java reflection to permit one to  
> read and write existing Java classes with Avro.
>
>  - a "specific" representation generates Java classes that are  
> compiled and loaded, much like Thrift and Protocol Buffers.
>
> We don't expect most scripting languages to use more than a single  
> representation.  Implementing Avro is quite simple, by design.  We  
> have a Python implementation, and hope to add more soon.
>
> Doug


From general-return-154-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 03 17:28:40 2009
Return-Path: <general-return-154-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4307 invoked from network); 3 Apr 2009 17:28:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Apr 2009 17:28:40 -0000
Received: (qmail 7492 invoked by uid 500); 3 Apr 2009 17:28:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7428 invoked by uid 500); 3 Apr 2009 17:28:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7418 invoked by uid 99); 3 Apr 2009 17:28:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 17:28:39 +0000
X-ASF-Spam-Status: No, hits=1.6 required=10.0
	tests=MISSING_HEADERS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cutting@gmail.com designates 209.85.198.228 as permitted sender)
Received: from [209.85.198.228] (HELO rv-out-0506.google.com) (209.85.198.228)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 17:28:30 +0000
Received: by rv-out-0506.google.com with SMTP id g37so355220rvb.5
        for <general@hadoop.apache.org>; Fri, 03 Apr 2009 10:28:08 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:sender:message-id:date:from
         :user-agent:mime-version:cc:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=QW4DS5cIAXcGDwZbGEamNeba4kNDHYONTcZ8kRoUt7A=;
        b=EOlOA1+WmS8pFjrLflHUvp5mvDGDxQD3X/d4HtMFayrBeB9BcsXtVbgd/dFYIMMi8C
         CxKF/aELd0UFM/3aNLEOJ6sowQjkUBD4jGa1TZHJikv9f4003RbuWVCvfn8C0BWnw8Ml
         Sdjl1ij6Y89T6PAxxAFtigDfM3D6/PsQis4Ks=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:date:from:user-agent:mime-version:cc:subject
         :references:in-reply-to:content-type:content-transfer-encoding;
        b=G4Alt/bWe1XOEYmFHHB9ZOGR1eygzqPkuvJyxrkr88Z70y1ZdEXtHhlIen+MEYQlLU
         MEiWBI8F6K53JuPCdS+tVf7kqvrSNwNHOch7p+Tvnw+sa2sj5jB/f7XZyVZi/VnzI0Dl
         92nnwIlgH3ufQEKvnu2x2P2W0ou2EZYTX2J+c=
Received: by 10.142.88.4 with SMTP id l4mr389155wfb.117.1238779688900;
        Fri, 03 Apr 2009 10:28:08 -0700 (PDT)
Received: from ?192.168.168.16? (c-76-103-155-128.hsd1.ca.comcast.net [76.103.155.128])
        by mx.google.com with ESMTPS id 24sm3288987wff.2.2009.04.03.10.28.07
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Fri, 03 Apr 2009 10:28:07 -0700 (PDT)
Sender: Doug Cutting <cutting@gmail.com>
Message-ID: <49D64723.7020706@apache.org>
Date: Fri, 03 Apr 2009 10:28:03 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.21 (X11/20090318)
MIME-Version: 1.0
CC: general@hadoop.apache.org
Subject: Re: [PROPOSAL] new subproject: Avro
References: <49D53694.1050906@apache.org> <4CB9034E-05FB-4200-AF55-FFD78B2EEFCE@apache.org> <3c682ecd0904021711x41fe4dd2j291f2077284d5558@mail.gmail.com> <8BBAB2C9-FCF9-4261-9E4B-282CD4196FA2@apache.org> <49D63415.8060004@apache.org> <718F2DEF-B305-49C9-B62F-155D2F4CE12F@rapleaf.com>
In-Reply-To: <718F2DEF-B305-49C9-B62F-155D2F4CE12F@rapleaf.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Bryan Duxbury wrote:
> It sounds like what you want is the option avoid pre-generated classes.

That's part of it.  But, once you have the schema, you might as well 
take advantage of it.

With the schema in hand, you don't need to tag data with field numbers 
or types, since that's all there in the schema.  So, having the schema, 
you can use a simpler data format.

Also, with the schema, resolving version differences is simplified. 
Developers don't need to assign field numbers, but can just use names. 
For performance, one can internally use field numbers while reading, to 
avoid string comparisons, but developers need no longer specify these, 
but can use names, as in most software.  Here having the schema means we 
can simplify the IDL and its versioning semantics.

> If that's the only thing you need, it seems like we could bolt that on 
> to Thrift with almost no work.

Would you write parsers for Thrift's IDL in every language?  Or would 
you use JSON, as Avro does, to avoid that?

Once you're using a different IDL and a different data format, what's 
shared with Thrift?  Fundamentally, those two things define a 
serialization system, no?

> I assume you'd have the schema stored in 
> metadata or file header or something, right? (You wouldn't want to store 
> the field names in the binary encoding as strings, since that would 
> probably very quickly dwarf the size of the actual data in a lot of cases.)

Yes, in data files the schema is typically stored in the metadata.

> If my assumptions are correct, it seems like it'd be a lot smarter to 
> leverage existing Thrift infrastructure and encoding work rather than 
> duplicating it for this lone feature.

What specific shared infrastructure would be leveraged?  For Hadoop's 
RPC, I hope to adapt Hadoop's client and server implementations as a 
transport, as these have been highly tuned for Hadoop's performance 
requirements.

Doug

From general-return-155-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 03 17:40:15 2009
Return-Path: <general-return-155-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12353 invoked from network); 3 Apr 2009 17:40:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Apr 2009 17:40:15 -0000
Received: (qmail 25051 invoked by uid 500); 3 Apr 2009 17:40:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25004 invoked by uid 500); 3 Apr 2009 17:40:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24994 invoked by uid 99); 3 Apr 2009 17:40:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 17:40:15 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cutting@gmail.com designates 209.85.200.174 as permitted sender)
Received: from [209.85.200.174] (HELO wf-out-1314.google.com) (209.85.200.174)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 17:40:07 +0000
Received: by wf-out-1314.google.com with SMTP id 23so1185626wfg.2
        for <general@hadoop.apache.org>; Fri, 03 Apr 2009 10:39:46 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:sender:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=mS9Sf2Kind2oOvEjaxytM4RdNV/Dm6zIg9QZ9wh1PFw=;
        b=Sh/PxfKuICwdRQW78cKB7gi4tszb8cIZmq87pgiL5E4bA6O0sZ/pUJQMptMGS2tNji
         hcrGletGRvB6DhhNuuz2zl8DjxYHHX1MevUFDd2K7Qgy2EcUhs9eDAMMH/xY2AiuWrJT
         lzeLZjA8ZdGzXw3hAG3e7KkHA+YJxvQCByrqY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:date:from:user-agent:mime-version:to:subject
         :references:in-reply-to:content-type:content-transfer-encoding;
        b=wDVAeTUwTgfQThiu0mjO00uL1FnEtdGF1XNmHFsu6dph46kYJqX0i5ZnnGxICXiwbK
         ppces00Zj2X4j/Tw0AgqPaG8laPODgLmQPtf0HRPk9TcIsOWG0GjFPBb2scxn7s0TXtX
         8W0i02IaFghxnxD9LlKIiHzkPs3fNsPq63ans=
Received: by 10.143.31.4 with SMTP id i4mr385094wfj.247.1238780386805;
        Fri, 03 Apr 2009 10:39:46 -0700 (PDT)
Received: from ?192.168.168.16? (c-76-103-155-128.hsd1.ca.comcast.net [76.103.155.128])
        by mx.google.com with ESMTPS id 28sm3308068wfg.11.2009.04.03.10.39.45
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Fri, 03 Apr 2009 10:39:45 -0700 (PDT)
Sender: Doug Cutting <cutting@gmail.com>
Message-ID: <49D649DE.9070709@apache.org>
Date: Fri, 03 Apr 2009 10:39:42 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.21 (X11/20090318)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [PROPOSAL] new subproject: Avro
References: <49D53694.1050906@apache.org>
In-Reply-To: <49D53694.1050906@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Doug Cutting wrote:
> git clone http://people.apache.org/~cutting/avro.git/ avro
> cat avro/README.txt

I've posted the generated documentation.  The specification is at:

http://people.apache.org/~cutting/avro/spec.html

and the javadoc is at:

http://people.apache.org/~cutting/avro/api/index.html

Doug

From general-return-156-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 03 17:44:07 2009
Return-Path: <general-return-156-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14538 invoked from network); 3 Apr 2009 17:44:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Apr 2009 17:44:07 -0000
Received: (qmail 27237 invoked by uid 500); 3 Apr 2009 17:44:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27173 invoked by uid 500); 3 Apr 2009 17:44:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27163 invoked by uid 99); 3 Apr 2009 17:44:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 17:44:06 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 17:43:57 +0000
Received: from [192.168.0.199] (snvvpn1-10-72-245-c238.hq.corp.yahoo.com [10.72.245.238])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n33HhABT077943
	for <general@hadoop.apache.org>; Fri, 3 Apr 2009 10:43:10 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=Em1RErvmX2CPKuOF3CEcRmLNgSOQ1MUg7mAHOTUPiMxnxb3vM4/DS2sI/k4ixV58
Message-Id: <711F0681-940A-4470-9816-931B5B146620@yahoo-inc.com>
From: Alan Gates <gates@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <49D53694.1050906@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: Re: [PROPOSAL] new subproject: Avro
Date: Fri, 3 Apr 2009 10:43:10 -0700
References: <49D53694.1050906@apache.org>
X-Mailer: Apple Mail (2.930.3)
X-Virus-Checked: Checked by ClamAV on apache.org

+1.  Pig would be happy to use a cross language serialization package  
that did not require pre-compilation to read and write.

Alan.

On Apr 2, 2009, at 3:05 PM, Doug Cutting wrote:

> I propose we add a new Hadoop subproject for Avro, a serialization  
> system.  My ambition is for Avro to replace both Hadoop's RPC and to  
> be used for most Hadoop data files, e.g., by Pig, Hive, etc.
>
> Initial committers would be Sharad Agarwal and me, both existing  
> Hadoop committers.  We are the sole authors of this software to date.
>
> The code is currently at:
>
> http://people.apache.org/~cutting/avro.git/
>
> To learn more:
>
> git clone http://people.apache.org/~cutting/avro.git/ avro
> cat avro/README.txt
>
> Comments?  Questions?
>
> Doug


From general-return-157-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 03 18:05:53 2009
Return-Path: <general-return-157-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 20560 invoked from network); 3 Apr 2009 18:05:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Apr 2009 18:05:53 -0000
Received: (qmail 31989 invoked by uid 500); 3 Apr 2009 17:50:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31967 invoked by uid 500); 3 Apr 2009 17:50:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31956 invoked by uid 99); 3 Apr 2009 17:50:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 17:50:45 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bryan@rapleaf.com designates 216.74.32.93 as permitted sender)
Received: from [216.74.32.93] (HELO mail.rapleaf.com) (216.74.32.93)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 17:50:37 +0000
Received: from mail.rapleaf.com (localhost.localdomain [127.0.0.1])
	by mail.rapleaf.com (Postfix) with ESMTP id 04CF012502EE
	for <general@hadoop.apache.org>; Fri,  3 Apr 2009 10:50:16 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=rapleaf.com; q=dns; s=m1; b=DIn5X
	cjLT5KTGDMYlP0HO3o7/V0eLC2777KowoPCPpCK1VPbPKL/ExoW4N61AR6fO20yF
	r8ufFp+BJBQPahYko5KQ6vcXITtxxMnpFiL6EwC5VaFIGsmN+bWpzSfEud68XZZ1
	yzNBdfk0WhL3IheOnrfoX+xhTP63FZop+p5dKE=
Received: from [10.100.18.119] (unknown [10.100.18.119])
	by mail.rapleaf.com (Postfix) with ESMTP id E3AA3125016F
	for <general@hadoop.apache.org>; Fri,  3 Apr 2009 10:50:15 -0700 (PDT)
Mime-Version: 1.0 (Apple Message framework v753.1)
In-Reply-To: <49D64723.7020706@apache.org>
References: <49D53694.1050906@apache.org> <4CB9034E-05FB-4200-AF55-FFD78B2EEFCE@apache.org> <3c682ecd0904021711x41fe4dd2j291f2077284d5558@mail.gmail.com> <8BBAB2C9-FCF9-4261-9E4B-282CD4196FA2@apache.org> <49D63415.8060004@apache.org> <718F2DEF-B305-49C9-B62F-155D2F4CE12F@rapleaf.com> <49D64723.7020706@apache.org>
Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed
Message-Id: <D30942BD-7B3D-4B0C-BB24-D7A560C71D07@rapleaf.com>
Content-Transfer-Encoding: 7bit
From: Bryan Duxbury <bryan@rapleaf.com>
Subject: Re: [PROPOSAL] new subproject: Avro
Date: Fri, 3 Apr 2009 10:50:14 -0700
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.753.1)
X-Virus-Checked: Checked by ClamAV on apache.org

> With the schema in hand, you don't need to tag data with field  
> numbers or types, since that's all there in the schema.  So, having  
> the schema, you can use a simpler data format.

To a degree, we already have that in Thrift - we call it the  
DenseProtocol.

> Would you write parsers for Thrift's IDL in every language?  Or  
> would you use JSON, as Avro does, to avoid that?

When it comes to having a code-usable IDL for the schema, I'm totally  
pro-JSON.

> Once you're using a different IDL and a different data format,  
> what's shared with Thrift?  Fundamentally, those two things define  
> a serialization system, no?

It's not actually a different data format, is it? You're saying that  
the user wouldn't specify the field IDs, but you'd fundamentally  
still use field ids for compactness and the like. You may not use  
actual Thrift generated objects, but you could certainly use Binary  
or Compact protocol from Thrift to do all the writing to the wire.

You might also be able to use (or contribute to) Thrift's RPC-level  
stuff like server implementations. We have some respectable Java  
servers written, and if those aren't enough for your uses, I'd  
actually be really interested in seeing if we could generalize some  
of the Hadoop stuff to be useful within Thrift.

The bottom line is that I would love to see greater cooperation  
between Hadoop and Thrift. Unless it's impossible or impractical for  
Thrift to be useful here, I think we'd be willing to work towards  
Hadoop's needs.

-Bryan


From general-return-158-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 03 18:52:33 2009
Return-Path: <general-return-158-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45707 invoked from network); 3 Apr 2009 18:52:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Apr 2009 18:52:32 -0000
Received: (qmail 84265 invoked by uid 500); 3 Apr 2009 18:38:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84243 invoked by uid 500); 3 Apr 2009 18:38:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84232 invoked by uid 99); 3 Apr 2009 18:38:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 18:38:26 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cutting@gmail.com designates 209.85.198.226 as permitted sender)
Received: from [209.85.198.226] (HELO rv-out-0506.google.com) (209.85.198.226)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 18:38:17 +0000
Received: by rv-out-0506.google.com with SMTP id k40so1081284rvb.29
        for <general@hadoop.apache.org>; Fri, 03 Apr 2009 11:37:55 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:sender:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=J+BmWUtmICOfcWQebHNOFKM43BV2Rx8aK36Kt2egw7A=;
        b=wXAArA87okR7tuPhGiRlpLehsPBIDwnmEc0u6Zbvp6oV5bOgQR/8FVVYHJsJIGwgv/
         yyantRsKepCTYeZxVkb+bflLiDUBo30tN3VrR2WQfb1iG2YSCUnBYo2V0MfzMjKkmVbM
         xLi3JWdi5jfwzACWtZvtEJiwHmolTTaMpGzMA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:date:from:user-agent:mime-version:to:subject
         :references:in-reply-to:content-type:content-transfer-encoding;
        b=i9UI7jIkUeAvCgO11gMie6I5w1eqrzg4FLOZ8CyRi4a08P2fZDNJ1l0ZbvkqKmGguH
         vqRhkbpGdTyyvjArdUM27a63ODUyxdWrjcSeywjUkJudRvQC90OSl2azMWBgXknrIjUP
         e/PFKU5wXDomkXPl8WrwetJCkBei6ottrTpbw=
Received: by 10.142.88.3 with SMTP id l3mr395071wfb.339.1238783875897;
        Fri, 03 Apr 2009 11:37:55 -0700 (PDT)
Received: from ?192.168.168.16? (c-76-103-155-128.hsd1.ca.comcast.net [76.103.155.128])
        by mx.google.com with ESMTPS id 22sm425355wfd.26.2009.04.03.11.37.54
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Fri, 03 Apr 2009 11:37:55 -0700 (PDT)
Sender: Doug Cutting <cutting@gmail.com>
Message-ID: <49D6577F.4080307@apache.org>
Date: Fri, 03 Apr 2009 11:37:51 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.21 (X11/20090318)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [PROPOSAL] new subproject: Avro
References: <49D53694.1050906@apache.org> <4CB9034E-05FB-4200-AF55-FFD78B2EEFCE@apache.org> <3c682ecd0904021711x41fe4dd2j291f2077284d5558@mail.gmail.com> <8BBAB2C9-FCF9-4261-9E4B-282CD4196FA2@apache.org> <49D63415.8060004@apache.org> <718F2DEF-B305-49C9-B62F-155D2F4CE12F@rapleaf.com> <49D64723.7020706@apache.org> <D30942BD-7B3D-4B0C-BB24-D7A560C71D07@rapleaf.com>
In-Reply-To: <D30942BD-7B3D-4B0C-BB24-D7A560C71D07@rapleaf.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Bryan Duxbury wrote:
> It's not actually a different data format, is it? You're saying that the 
> user wouldn't specify the field IDs, but you'd fundamentally still use 
> field ids for compactness and the like.

Field ids are not present in Avro data except in the schema.  A record's 
fields are serialized in the order that the fields occur in the records 
schema, with no per-field annotations whatsoever.  For example, a record 
that contains a string and an int is serialized simply as a string 
followed by an int, nothing before, nothing between and nothing after. 
So, yes, it is a different data format.

> The bottom line is that I would love to see greater cooperation between 
> Hadoop and Thrift. Unless it's impossible or impractical for Thrift to 
> be useful here, I think we'd be willing to work towards Hadoop's needs.

Perhaps Thrift could be augmented to support Avro's JSON schemas and 
serialization.  Then it could interoperate with other Avro-based 
systems.  But then Thrift would have yet another serialization format, 
that every language would need to implement for it to be useful...

Avro will only ever have one serialization format.  Thrift fundamentally 
standardizes an API, not a data format.  Avro fundamentally is a data 
format specification, like XML.  Thrift could implement this 
specification.  The Avro project includes reference implementations, but 
the format is intended to be simple enough and the specification stable 
enough that others might reasonably develop alternate, independent 
implementations.

Doug

From general-return-159-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 03 19:04:18 2009
Return-Path: <general-return-159-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55959 invoked from network); 3 Apr 2009 19:04:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Apr 2009 19:04:18 -0000
Received: (qmail 29341 invoked by uid 500); 3 Apr 2009 19:04:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29290 invoked by uid 500); 3 Apr 2009 19:04:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29279 invoked by uid 99); 3 Apr 2009 19:04:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 19:04:17 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [192.18.43.133] (HELO sca-es-mail-2.sun.com) (192.18.43.133)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 19:04:08 +0000
Received: from fe-sfbay-09.sun.com ([192.18.43.129])
	by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n33J3aLo024950
	for <general@hadoop.apache.org>; Fri, 3 Apr 2009 12:03:48 -0700 (PDT)
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; delsp=yes; format=flowed; charset=US-ASCII
Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com
 (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009))
 id <0KHJ00500GIC1E00@fe-sfbay-09.sun.com> for general@hadoop.apache.org; Fri,
 03 Apr 2009 12:03:36 -0700 (PDT)
Received: from [42.132.180.78] (wireless-am10.ucsd.edu [128.54.48.14])
 by fe-sfbay-09.sun.com
 (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009))
 with ESMTPSA id <0KHJ00HPEGXZA8B0@fe-sfbay-09.sun.com> for
 general@hadoop.apache.org; Fri, 03 Apr 2009 12:03:36 -0700 (PDT)
Date: Fri, 03 Apr 2009 12:03:35 -0700
From: George Porter <George.Porter@Sun.COM>
Subject: Re: [PROPOSAL] new subproject: Avro
In-reply-to: <49D6577F.4080307@apache.org>
Sender: George.Porter@Sun.COM
To: general@hadoop.apache.org
Message-id: <F3482D1C-F72C-4DD7-B06C-73D1109C42E2@Sun.com>
X-Mailer: Apple Mail (2.930.3)
References: <49D53694.1050906@apache.org>
 <4CB9034E-05FB-4200-AF55-FFD78B2EEFCE@apache.org>
 <3c682ecd0904021711x41fe4dd2j291f2077284d5558@mail.gmail.com>
 <8BBAB2C9-FCF9-4261-9E4B-282CD4196FA2@apache.org>
 <49D63415.8060004@apache.org>
 <718F2DEF-B305-49C9-B62F-155D2F4CE12F@rapleaf.com>
 <49D64723.7020706@apache.org>
 <D30942BD-7B3D-4B0C-BB24-D7A560C71D07@rapleaf.com>
 <49D6577F.4080307@apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org


On Apr 3, 2009, at 11:37 AM, Doug Cutting wrote:
>>
>
> Field ids are not present in Avro data except in the schema.  A  
> record's fields are serialized in the order that the fields occur in  
> the records schema, with no per-field annotations whatsoever.  For  
> example, a record that contains a string and an int is serialized  
> simply as a string followed by an int, nothing before, nothing  
> between and nothing after. So, yes, it is a different data format.

While this representation would certainly be as compact as possible,  
wouldn't it prevent evolving the data structure over time?  One of the  
nice features of Google Protocol Buffers and Thrift is that you can  
evolve the set of fields over time, and older/newer clients can talk  
to older/newer services.  If the proposed Avro is evolvable, then  
perhaps I'm misunderstanding your statement about the lack of IDs in  
the serialized data.

I also agree with Bryan, in that it would be unfortunate to have two  
different Apache projects with overlapping goals.  Regardless of  
features, both protocol buffers and thrift have the advantage of being  
debugged in mission-critical production environments.

-George

From general-return-160-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 03 19:50:04 2009
Return-Path: <general-return-160-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 71615 invoked from network); 3 Apr 2009 19:50:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Apr 2009 19:50:02 -0000
Received: (qmail 84447 invoked by uid 500); 3 Apr 2009 19:50:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84391 invoked by uid 500); 3 Apr 2009 19:50:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84381 invoked by uid 99); 3 Apr 2009 19:50:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 19:50:02 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [64.78.17.18] (HELO EXHUB018-3.exch018.msoutlookonline.net) (64.78.17.18)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 19:49:52 +0000
Received: from EXVMBX018-1.exch018.msoutlookonline.net ([64.78.17.47]) by
 EXHUB018-3.exch018.msoutlookonline.net ([64.78.17.18]) with mapi; Fri, 3 Apr
 2009 12:49:31 -0700
From: Scott Carey <scott@richrelevance.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 3 Apr 2009 12:49:29 -0700
Subject: Re: [PROPOSAL] new subproject: Avro
Thread-Topic: [PROPOSAL] new subproject: Avro
Thread-Index: Acm0jv948NH3v2V3QM+326WbEtDlVQABk1vG
Message-ID: <C5FBB659.42D7%scott@richrelevance.com>
In-Reply-To: <F3482D1C-F72C-4DD7-B06C-73D1109C42E2@Sun.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

On 4/3/09 12:03 PM, "George Porter" <George.Porter@Sun.COM> wrote:

>=20
>=20
> On Apr 3, 2009, at 11:37 AM, Doug Cutting wrote:
>>>=20
>>=20
>> Field ids are not present in Avro data except in the schema.  A
>> record's fields are serialized in the order that the fields occur in
>> the records schema, with no per-field annotations whatsoever.  For
>> example, a record that contains a string and an int is serialized
>> simply as a string followed by an int, nothing before, nothing
>> between and nothing after. So, yes, it is a different data format.
>=20
> While this representation would certainly be as compact as possible,
> wouldn't it prevent evolving the data structure over time?  One of the
> nice features of Google Protocol Buffers and Thrift is that you can
> evolve the set of fields over time, and older/newer clients can talk
> to older/newer services.  If the proposed Avro is evolvable, then
> perhaps I'm misunderstanding your statement about the lack of IDs in
> the serialized data.

>From a quick perusal of the serialization format -- it contains headers wit=
h
type/schema information, and other metadata blocks.  The types can be
inferred from this, and if this is done right then older/newer clients will
be able to read things just fine.  What can't be done is mixing two
different formats in the same stream if headers define the format of the
whole stream.

I have not looked much deeper than that, but it looks like schema evolution
is feasible.

>=20
> I also agree with Bryan, in that it would be unfortunate to have two
> different Apache projects with overlapping goals.  Regardless of
> features, both protocol buffers and thrift have the advantage of being
> debugged in mission-critical production environments.
>=20
> -George
>=20


From general-return-161-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 03 19:59:48 2009
Return-Path: <general-return-161-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 74446 invoked from network); 3 Apr 2009 19:59:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Apr 2009 19:59:48 -0000
Received: (qmail 95300 invoked by uid 500); 3 Apr 2009 19:59:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95216 invoked by uid 500); 3 Apr 2009 19:59:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95204 invoked by uid 99); 3 Apr 2009 19:59:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 19:59:47 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of bryan@rapleaf.com designates 216.74.32.93 as permitted sender)
Received: from [216.74.32.93] (HELO mail.rapleaf.com) (216.74.32.93)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 19:59:38 +0000
Received: from mail.rapleaf.com (localhost.localdomain [127.0.0.1])
	by mail.rapleaf.com (Postfix) with ESMTP id 7031612502F9
	for <general@hadoop.apache.org>; Fri,  3 Apr 2009 12:59:16 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=rapleaf.com; q=dns; s=m1; b=DLaXP
	JGPJYVjXM8Dj38/hVXyHWyfW0sTjlafqf2KqcbF899tIc9GBpegor3qotTOp/qQl
	1MUizEF/zVTTNs9m/0cGfwcV3wWYGURaKrIiE2yRKRXe+D4mA+E1xCfFbZOucnd4
	CMbp03xjGwVQjcS3dkdM0EAMja0Bc/hquDqZrU=
Received: from [10.100.18.119] (unknown [10.100.18.119])
	by mail.rapleaf.com (Postfix) with ESMTP id 5DB8712502EE
	for <general@hadoop.apache.org>; Fri,  3 Apr 2009 12:59:16 -0700 (PDT)
Mime-Version: 1.0 (Apple Message framework v753.1)
In-Reply-To: <49D6577F.4080307@apache.org>
References: <49D53694.1050906@apache.org> <4CB9034E-05FB-4200-AF55-FFD78B2EEFCE@apache.org> <3c682ecd0904021711x41fe4dd2j291f2077284d5558@mail.gmail.com> <8BBAB2C9-FCF9-4261-9E4B-282CD4196FA2@apache.org> <49D63415.8060004@apache.org> <718F2DEF-B305-49C9-B62F-155D2F4CE12F@rapleaf.com> <49D64723.7020706@apache.org> <D30942BD-7B3D-4B0C-BB24-D7A560C71D07@rapleaf.com> <49D6577F.4080307@apache.org>
Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed
Message-Id: <E44CB57B-6ECB-42B1-9AFE-EC82C20EBA9F@rapleaf.com>
Content-Transfer-Encoding: 7bit
From: Bryan Duxbury <bryan@rapleaf.com>
Subject: Re: [PROPOSAL] new subproject: Avro
Date: Fri, 3 Apr 2009 12:59:15 -0700
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.753.1)
X-Virus-Checked: Checked by ClamAV on apache.org

> Field ids are not present in Avro data except in the schema.  A  
> record's fields are serialized in the order that the fields occur  
> in the records schema, with no per-field annotations whatsoever.   
> For example, a record that contains a string and an int is  
> serialized simply as a string followed by an int, nothing before,  
> nothing between and nothing after. So, yes, it is a different data  
> format.

So you can't serialize nulls? It also seems like this would make  
forward/backward compatibility a little more complex. Thrift solves  
this problem by using tags to indicate what kind of field you're  
working with.

-Bryan

From general-return-162-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 03 20:03:11 2009
Return-Path: <general-return-162-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76294 invoked from network); 3 Apr 2009 20:03:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Apr 2009 20:03:11 -0000
Received: (qmail 98887 invoked by uid 500); 3 Apr 2009 20:03:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98803 invoked by uid 500); 3 Apr 2009 20:03:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98793 invoked by uid 99); 3 Apr 2009 20:03:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 20:03:10 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cutting@gmail.com designates 209.85.200.175 as permitted sender)
Received: from [209.85.200.175] (HELO wf-out-1314.google.com) (209.85.200.175)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 20:03:02 +0000
Received: by wf-out-1314.google.com with SMTP id 23so1239066wfg.2
        for <general@hadoop.apache.org>; Fri, 03 Apr 2009 13:02:42 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:sender:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=0wrte25rKZ80+aEA93U7ChcBHYp4e8aklH+uJCUhxv8=;
        b=r1nQe84ao/72FItl/kY3EEKg0Pko5xpi9gC6TJgbONUzX5f2cJKDsoYsFbcdhkzSsB
         5usapylOh14DWaN5B1dQJlmgoNPoFU99TkBvjt1LYvayABXePUgYSveVloWuIISXCQPY
         hIOBP7mj24pZdQcyJIBFCCgohGGIL67TLfAw8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:date:from:user-agent:mime-version:to:subject
         :references:in-reply-to:content-type:content-transfer-encoding;
        b=aLi06On5Rzdyeb+0G1QPigbOX5O2oJLs71WzeiC1rINrCwMuXHFYAL3LtuHK3QtYzc
         WvYes+NIbNifHajdBrJah2bkg/7XqVXmAQSAyrbq05Trlw1pViI9hKjfWJBFBtlVsX1f
         LpPJTkOUOtcnlBnCS+GvCorxiy06wBEVqnvN8=
Received: by 10.142.52.7 with SMTP id z7mr414122wfz.267.1238788962322;
        Fri, 03 Apr 2009 13:02:42 -0700 (PDT)
Received: from ?192.168.168.16? (c-76-103-155-128.hsd1.ca.comcast.net [76.103.155.128])
        by mx.google.com with ESMTPS id 22sm3454385wfg.3.2009.04.03.13.02.40
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Fri, 03 Apr 2009 13:02:41 -0700 (PDT)
Sender: Doug Cutting <cutting@gmail.com>
Message-ID: <49D66B40.5070509@apache.org>
Date: Fri, 03 Apr 2009 13:02:08 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.21 (X11/20090318)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [PROPOSAL] new subproject: Avro
References: <49D53694.1050906@apache.org> <4CB9034E-05FB-4200-AF55-FFD78B2EEFCE@apache.org> <3c682ecd0904021711x41fe4dd2j291f2077284d5558@mail.gmail.com> <8BBAB2C9-FCF9-4261-9E4B-282CD4196FA2@apache.org> <49D63415.8060004@apache.org> <718F2DEF-B305-49C9-B62F-155D2F4CE12F@rapleaf.com> <49D64723.7020706@apache.org> <D30942BD-7B3D-4B0C-BB24-D7A560C71D07@rapleaf.com> <49D6577F.4080307@apache.org> <F3482D1C-F72C-4DD7-B06C-73D1109C42E2@Sun.com>
In-Reply-To: <F3482D1C-F72C-4DD7-B06C-73D1109C42E2@Sun.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

George Porter wrote:
> While this representation would certainly be as compact as possible, 
> wouldn't it prevent evolving the data structure over time?  One of the 
> nice features of Google Protocol Buffers and Thrift is that you can 
> evolve the set of fields over time, and older/newer clients can talk to 
> older/newer services.  If the proposed Avro is evolvable, then perhaps 
> I'm misunderstanding your statement about the lack of IDs in the 
> serialized data.

Avro supports schema evolution.  In Avro, the schema used to write the 
data must be available when the data is read.  (In files, it is 
typically stored in the file metadata.)

If you have the schema that was used to write the data, and you're 
expecting a slightly different schema, then you simply keep those fields 
that are in both schemas and skip those not.  This is equivalent to 
Thrift and Protocol Buffer's support for schema evolution, but does not 
require manually assigning numeric field ids.

This feature can also be used to support projection.  If you have 
records with many large fields, but only need a single field in a 
particular computation, then you can specify an expected schema with 
only that field, and the runtime will efficiently skip all of the other 
fields, returning a record with just the single, expected field.

> I also agree with Bryan, in that it would be unfortunate to have two 
> different Apache projects with overlapping goals.

We already have both Thrift and Etch in the incubator, which have 
similar goals.  Apache does not attempt to mandate that projects have 
disjoint goals.  There are many ways to slice things, and Apache prefers 
to rely on survival of the fittest rather than forcing things together.

> Regardless of 
> features, both protocol buffers and thrift have the advantage of being 
> debugged in mission-critical production environments.

Yes, but, as I've argued in other messages in this thread, they do not 
support the dynamic features we need.  Adding those features would add 
new code that would share little with existing code in those projects. 
So, while the projects are conceptually similar, the implementations are 
necessarily different, and, without significant code overlap, separate 
projects seem more natural.

Doug

From general-return-163-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 03 20:36:54 2009
Return-Path: <general-return-163-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85447 invoked from network); 3 Apr 2009 20:36:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Apr 2009 20:36:51 -0000
Received: (qmail 26198 invoked by uid 500); 3 Apr 2009 20:24:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26175 invoked by uid 500); 3 Apr 2009 20:24:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26165 invoked by uid 99); 3 Apr 2009 20:24:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 20:24:34 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [192.18.43.132] (HELO sca-es-mail-1.sun.com) (192.18.43.132)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 20:24:26 +0000
Received: from fe-sfbay-10.sun.com ([192.18.43.129])
	by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n33KO5gA005504
	for <general@hadoop.apache.org>; Fri, 3 Apr 2009 13:24:05 -0700 (PDT)
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; delsp=yes; format=flowed; charset=US-ASCII
Received: from conversion-daemon.fe-sfbay-10.sun.com by fe-sfbay-10.sun.com
 (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009))
 id <0KHJ00J00KMNHQ00@fe-sfbay-10.sun.com> for general@hadoop.apache.org; Fri,
 03 Apr 2009 13:24:05 -0700 (PDT)
Received: from [42.132.180.78] (wireless-am10.ucsd.edu [128.54.48.14])
 by fe-sfbay-10.sun.com
 (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009))
 with ESMTPSA id <0KHJ00GRAKO5XH90@fe-sfbay-10.sun.com> for
 general@hadoop.apache.org; Fri, 03 Apr 2009 13:24:05 -0700 (PDT)
Date: Fri, 03 Apr 2009 13:24:05 -0700
From: George Porter <George.Porter@Sun.COM>
Subject: Re: [PROPOSAL] new subproject: Avro
In-reply-to: <49D66B40.5070509@apache.org>
Sender: George.Porter@Sun.COM
To: general@hadoop.apache.org
Message-id: <68F2489F-05D7-4D8F-924B-EBF740579355@Sun.com>
X-Mailer: Apple Mail (2.930.3)
References: <49D53694.1050906@apache.org>
 <4CB9034E-05FB-4200-AF55-FFD78B2EEFCE@apache.org>
 <3c682ecd0904021711x41fe4dd2j291f2077284d5558@mail.gmail.com>
 <8BBAB2C9-FCF9-4261-9E4B-282CD4196FA2@apache.org>
 <49D63415.8060004@apache.org>
 <718F2DEF-B305-49C9-B62F-155D2F4CE12F@rapleaf.com>
 <49D64723.7020706@apache.org>
 <D30942BD-7B3D-4B0C-BB24-D7A560C71D07@rapleaf.com>
 <49D6577F.4080307@apache.org> <F3482D1C-F72C-4DD7-B06C-73D1109C42E2@Sun.com>
 <49D66B40.5070509@apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org


On Apr 3, 2009, at 1:02 PM, Doug Cutting wrote:

> George Porter wrote:
>> While this representation would certainly be as compact as  
>> possible, wouldn't it prevent evolving the data structure over  
>> time?  One of the nice features of Google Protocol Buffers and  
>> Thrift is that you can evolve the set of fields over time, and  
>> older/newer clients can talk to older/newer services.  If the  
>> proposed Avro is evolvable, then perhaps I'm misunderstanding your  
>> statement about the lack of IDs in the serialized data.
>
> Avro supports schema evolution.  In Avro, the schema used to write  
> the data must be available when the data is read.  (In files, it is  
> typically stored in the file metadata.)
>
> If you have the schema that was used to write the data, and you're  
> expecting a slightly different schema, then you simply keep those  
> fields that are in both schemas and skip those not.  This is  
> equivalent to Thrift and Protocol Buffer's support for schema  
> evolution, but does not require manually assigning numeric field ids.
>
> This feature can also be used to support projection.  If you have  
> records with many large fields, but only need a single field in a  
> particular computation, then you can specify an expected schema with  
> only that field, and the runtime will efficiently skip all of the  
> other fields, returning a record with just the single, expected field.

Thanks for the clarification--I better understand the schema  
relationship.  The projection feature is a nice feature, especially  
since it seems like it would be able to support "sparse files" where  
you want to just peek at large structs without invoking a lot of disk- 
io (for data serialized on-disk).

>
>
>> I also agree with Bryan, in that it would be unfortunate to have  
>> two different Apache projects with overlapping goals.
>
> We already have both Thrift and Etch in the incubator, which have  
> similar goals.  Apache does not attempt to mandate that projects  
> have disjoint goals.  There are many ways to slice things, and  
> Apache prefers to rely on survival of the fittest rather than  
> forcing things together.
>
>> Regardless of features, both protocol buffers and thrift have the  
>> advantage of being debugged in mission-critical production  
>> environments.
>
> Yes, but, as I've argued in other messages in this thread, they do  
> not support the dynamic features we need.  Adding those features  
> would add new code that would share little with existing code in  
> those projects. So, while the projects are conceptually similar, the  
> implementations are necessarily different, and, without significant  
> code overlap, separate projects seem more natural.
>
> Doug

Makes sense.  Thanks,
George

From general-return-164-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 03 20:37:01 2009
Return-Path: <general-return-164-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 86129 invoked from network); 3 Apr 2009 20:36:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Apr 2009 20:36:54 -0000
Received: (qmail 27474 invoked by uid 500); 3 Apr 2009 20:27:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27453 invoked by uid 500); 3 Apr 2009 20:27:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27443 invoked by uid 99); 3 Apr 2009 20:27:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 20:27:41 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [206.190.38.122] (HELO web50902.mail.re2.yahoo.com) (206.190.38.122)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 03 Apr 2009 20:27:32 +0000
Received: (qmail 59825 invoked by uid 60001); 3 Apr 2009 20:27:10 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1238790430; bh=6RkKtHoVGczPss7hcsS6Bkp5TtUnqxv4WkrEUXnjzTY=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=cQVjlqVlUPFAQKirT/+StiQxU2PtETbp5KSJ+nc/OCT59fVQo9Hm8HV2ataIYczOPdCCbus5XHtHy/cibXmJpeWCycK4IOGzTO2lFkf7LrBvur/3v+ROUsCkvqaCrLlIFJmyfJXPD8jkOGH7T9vp53HX+VjXrzyNbJAmAcrqRKM=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=FeZIgb9QjYIKdqKnRTQ+JA+R7kEbHY+xHXjQeKJ4m47Zb8cgfxrfb//NUG388kHAqPf20ZT/CEQsYQMviaFPkCHvhW8UhvxAcO8CEO8rvgLLfaR5Rgy3dBOrz+C9w72CSuM9lOpbNsN2NQOT8wpTdXTkGZE+5VvVUBATscQP/V0=;
Message-ID: <758794.58891.qm@web50902.mail.re2.yahoo.com>
X-YMail-OSG: IBoK20YVM1kU2cS64He7SnIo.b4AHA4nWYa9h.NRdYL6C0mrTCd8mEJgABMMtyinw5YGnH1u73MFpsNemBK21J7E_Su.zyV610s1H.rePQxICnGeF8IvPfyDCZvqFuEnGBYMZAXNo0vS1EOgbDeaHLeFMHVcUwZ4U_tcNUX68CwNJtnnxy7Gh4vJvPJhW6B7b6dSgBDDW3oKh6OsH3cj_Xc1RCprEZAG_njp8pQidmaCjtCmNNDaxJX5yoX8EDyiQ0BeghK8eM9kQ48eO_SK9Y0IOJOxofjo1vpTGy15dw23Nn.u.0T9BaHshn0UfQVJMYI3feaTHSJhWNWyyoGA
Received: from [98.234.219.2] by web50902.mail.re2.yahoo.com via HTTP; Fri, 03 Apr 2009 13:27:10 PDT
X-Mailer: YahooMailRC/1277.35 YahooMailWebService/0.7.289.1
References: <49D53694.1050906@apache.org>
Date: Fri, 3 Apr 2009 13:27:10 -0700 (PDT)
From: Sameer Paranjpye <sparanjpye@yahoo.com>
Subject: Re: [PROPOSAL] new subproject: Avro
To: general@hadoop.apache.org
In-Reply-To: <49D53694.1050906@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org


+1

While protocol buffers and thrift have similar goals. Avro takes a different approach to schema evolution and reconciliation. I feel that Avros tighter layout of data and schema management is better suited for many of Hadoop and Pigs use cases for large data sets/tables on HDFS. Field ids start to matter when milions of objects have hundreds of fields each. There is, of course, the storage overhead. Schema management becomes hard especially if there are cases where field ids need to be assigned manually.


----- Original Message ----
From: Doug Cutting <cutting@apache.org>
To: general@hadoop.apache.org
Sent: Thursday, April 2, 2009 3:05:08 PM
Subject: [PROPOSAL] new subproject: Avro

I propose we add a new Hadoop subproject for Avro, a serialization system.  My ambition is for Avro to replace both Hadoop's RPC and to be used for most Hadoop data files, e.g., by Pig, Hive, etc.

Initial committers would be Sharad Agarwal and me, both existing Hadoop committers.  We are the sole authors of this software to date.

The code is currently at:

http://people.apache.org/~cutting/avro.git/

To learn more:

git clone http://people.apache.org/~cutting/avro.git/ avro
cat avro/README.txt

Comments?  Questions?

Doug



      

From general-return-165-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 03 20:49:24 2009
Return-Path: <general-return-165-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95461 invoked from network); 3 Apr 2009 20:49:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Apr 2009 20:49:24 -0000
Received: (qmail 58746 invoked by uid 500); 3 Apr 2009 20:49:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58671 invoked by uid 500); 3 Apr 2009 20:49:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58661 invoked by uid 99); 3 Apr 2009 20:49:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 20:49:23 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 20:49:14 +0000
Received: from [10.0.1.196] (snvvpn2-10-72-76-c99.hq.corp.yahoo.com [10.72.76.99])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n33Kl5Gk088088
	for <general@hadoop.apache.org>; Fri, 3 Apr 2009 13:47:05 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=Ur4Gr3HPLbvFwe7nHqELRHAVszIh8Oh/WgLPzowd90MnUlTiphFbwD5IHLHU0SUn
Message-Id: <3DBB0E23-7B6D-4541-A0D6-3687DF02C397@yahoo-inc.com>
From: Nigel Daley <ndaley@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <49D53694.1050906@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: Re: [PROPOSAL] new subproject: Avro
Date: Fri, 3 Apr 2009 13:47:04 -0700
References: <49D53694.1050906@apache.org>
X-Mailer: Apple Mail (2.930.3)
X-Virus-Checked: Checked by ClamAV on apache.org

+1.

On Apr 2, 2009, at 3:05 PM, Doug Cutting wrote:

> I propose we add a new Hadoop subproject for Avro, a serialization  
> system.  My ambition is for Avro to replace both Hadoop's RPC and to  
> be used for most Hadoop data files, e.g., by Pig, Hive, etc.
>
> Initial committers would be Sharad Agarwal and me, both existing  
> Hadoop committers.  We are the sole authors of this software to date.
>
> The code is currently at:
>
> http://people.apache.org/~cutting/avro.git/
>
> To learn more:
>
> git clone http://people.apache.org/~cutting/avro.git/ avro
> cat avro/README.txt
>
> Comments?  Questions?
>
> Doug


From general-return-166-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 03 21:00:43 2009
Return-Path: <general-return-166-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98540 invoked from network); 3 Apr 2009 21:00:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Apr 2009 21:00:43 -0000
Received: (qmail 69106 invoked by uid 500); 3 Apr 2009 21:00:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69035 invoked by uid 500); 3 Apr 2009 21:00:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69025 invoked by uid 99); 3 Apr 2009 21:00:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 21:00:42 +0000
X-ASF-Spam-Status: No, hits=4.2 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 21:00:31 +0000
Received: from sp1-ex07cas01.ds.corp.yahoo.com (sp1-ex07cas01.ds.corp.yahoo.com [216.252.116.137])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n33KxuxA053324
	for <general@hadoop.apache.org>; Fri, 3 Apr 2009 13:59:56 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:in-reply-to:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=anVY80lL0Ls8JrEZgIAZVFs10cs0XrA9R9B0n+42zIMISXd45nkLPo9cGN59nTh0
Received: from SP1-EX07VS02.ds.corp.yahoo.com ([216.252.116.135]) by
 sp1-ex07cas01.ds.corp.yahoo.com ([216.252.116.137]) with mapi; Fri, 3 Apr
 2009 13:59:56 -0700
From: Jerome Boulon <jboulon@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 3 Apr 2009 13:59:54 -0700
Subject: Re: [PROPOSAL] new subproject: Avro
Thread-Topic: [PROPOSAL] new subproject: Avro
Thread-Index: Acmz32qJXRwwUeKUTougg4PRElvdIAAv7irY
Message-ID: <C5FBC6DA.198B7%jboulon@yahoo-inc.com>
In-Reply-To: <49D53694.1050906@apache.org>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C5FBC6DA198B7jboulonyahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C5FBC6DA198B7jboulonyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Chukwa needs a cross language serialization/RPC framework.
During demux, being able to create dynamically the schema based on the reco=
rdType will be a big plus instead of using an hardcoded one,
The projection feature will also be useful.

+1

/Jerome.


On 4/2/09 3:05 PM, "Doug Cutting" <cutting@apache.org> wrote:

I propose we add a new Hadoop subproject for Avro, a serialization
system.  My ambition is for Avro to replace both Hadoop's RPC and to be
used for most Hadoop data files, e.g., by Pig, Hive, etc.

Initial committers would be Sharad Agarwal and me, both existing Hadoop
committers.  We are the sole authors of this software to date.

The code is currently at:

http://people.apache.org/~cutting/avro.git/

To learn more:

git clone http://people.apache.org/~cutting/avro.git/ avro
cat avro/README.txt

Comments?  Questions?

Doug


--_000_C5FBC6DA198B7jboulonyahooinccom_--

From general-return-167-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 03 22:10:49 2009
Return-Path: <general-return-167-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24148 invoked from network); 3 Apr 2009 22:10:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Apr 2009 22:10:49 -0000
Received: (qmail 62632 invoked by uid 500); 3 Apr 2009 22:10:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62551 invoked by uid 500); 3 Apr 2009 22:10:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62518 invoked by uid 99); 3 Apr 2009 22:10:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 22:10:48 +0000
X-ASF-Spam-Status: No, hits=-8.0 required=10.0
	tests=RCVD_IN_DNSWL_HI,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of Jim.Kellerman@microsoft.com designates 131.107.115.212 as permitted sender)
Received: from [131.107.115.212] (HELO smtp.microsoft.com) (131.107.115.212)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 22:10:40 +0000
Received: from TK5-EXHUB-C102.redmond.corp.microsoft.com (157.54.18.53) by
 TK5-EXGWY-E801.partners.extranet.microsoft.com (10.251.56.50) with Microsoft
 SMTP Server (TLS) id 8.2.99.4; Fri, 3 Apr 2009 15:10:19 -0700
Received: from NA-EXMSG-C103.redmond.corp.microsoft.com ([157.54.110.52]) by
 TK5-EXHUB-C102.redmond.corp.microsoft.com ([157.54.18.53]) with mapi; Fri, 3
 Apr 2009 15:10:12 -0700
From: "Jim Kellerman (POWERSET)" <Jim.Kellerman@microsoft.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 3 Apr 2009 15:10:17 -0700
Subject: RE: [PROPOSAL] new subproject: Avro
Thread-Topic: [PROPOSAL] new subproject: Avro
Thread-Index: Acm0nbcWMxtRL47oQNeZ3l2aOazQIgACX2CQ
Message-ID: <EAFDEC03CDA5D644878904F4A8F0158A6191AC88EA@NA-EXMSG-C103.redmond.corp.microsoft.com>
References: <49D53694.1050906@apache.org>
 <3DBB0E23-7B6D-4541-A0D6-3687DF02C397@yahoo-inc.com>
In-Reply-To: <3DBB0E23-7B6D-4541-A0D6-3687DF02C397@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

> -----Original Message-----
> On Apr 2, 2009, at 3:05 PM, Doug Cutting wrote:
>=20
> I propose we add a new Hadoop subproject for Avro, a serialization
> system.  My ambition is for Avro to replace both Hadoop's RPC and to
> be used for most Hadoop data files, e.g., by Pig, Hive, etc.
>
> Initial committers would be Sharad Agarwal and me, both existing
> Hadoop committers.  We are the sole authors of this software to date.
>
> The code is currently at:
>
> http://people.apache.org/~cutting/avro.git/
>
> To learn more:
>
> git clone http://people.apache.org/~cutting/avro.git/ avro
> cat avro/README.txt
>
> Comments?  Questions?
>
> Doug

After reading all the messages about Avro, I'm still not sure I understand
why we should invent "yet another wheel". There are a number of people in
the community who have significant investments in Thrift, and I have yet
to see a compelling argument for Avro over Thrift.

My understanding is that Thrift already supports multi-language bindings,
something the HBase community has been asking for, for some time.

It is also my understanding (based on the email thread) that Avro only
supports Java and python. That is a step backwards from Thrift.

It appears that Avro uses introspection heavily, which is expensive in
applications that require a high message rate.

So I guess my question is why Avro? I may be thick, but it seems to me
as if it is just another wheel of a different color. If I could see a
point by point comparison between Avro and Thrift I could be convinced
that Avro is the way to go. So far, I have not seen any compelling reason
to re-invent the wheel.

+-0

---
Jim Kellerman, Powerset (Live Search, Microsoft Corporation)




From general-return-168-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 03 22:45:31 2009
Return-Path: <general-return-168-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39641 invoked from network); 3 Apr 2009 22:45:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Apr 2009 22:45:31 -0000
Received: (qmail 96134 invoked by uid 500); 3 Apr 2009 22:45:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96069 invoked by uid 500); 3 Apr 2009 22:45:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96059 invoked by uid 99); 3 Apr 2009 22:45:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 22:45:30 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cutting@gmail.com designates 209.85.200.169 as permitted sender)
Received: from [209.85.200.169] (HELO wf-out-1314.google.com) (209.85.200.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Apr 2009 22:45:21 +0000
Received: by wf-out-1314.google.com with SMTP id 23so1290931wfg.2
        for <general@hadoop.apache.org>; Fri, 03 Apr 2009 15:45:00 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:sender:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=ib2/8KzQBziNAPkQCTR/6OxJEJ2tuAfAtBHqIOUa/9U=;
        b=stmy7YR/6v7Rv9yHK0eT5OWPhdoaMsHCv5oSXH9YVlAseFBUU64D09A3vhH39y/w4P
         n4RS8zmKpyAOcMrww4u0dGLm7SxqgxF8h4+ONpNh2FZH5EUxfcr2JXANfghzViHi052X
         bvx2QrZwE3PAfCH7IjpQ2q3p7FTDIL1x532M0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:date:from:user-agent:mime-version:to:subject
         :references:in-reply-to:content-type:content-transfer-encoding;
        b=xesD/Z+zpov+46IHu6ghit5Fn2em7JeuFYVGlhTrT89us/xrtufs7b1BNiz//MVRXW
         0Jwnr1ccZNXRM2ZaUMYB2t+QoooAolOE9rfg11ui1LyBSbp+2W/XKdhJ9zDniSlHu0em
         KjCYxIKVedkwQsYP5U0BBnpyHQ/x6OKYVZDT0=
Received: by 10.142.186.9 with SMTP id j9mr457102wff.5.1238798700423;
        Fri, 03 Apr 2009 15:45:00 -0700 (PDT)
Received: from ?192.168.168.16? (c-76-103-155-128.hsd1.ca.comcast.net [76.103.155.128])
        by mx.google.com with ESMTPS id 24sm3569952wff.42.2009.04.03.15.44.58
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Fri, 03 Apr 2009 15:44:59 -0700 (PDT)
Sender: Doug Cutting <cutting@gmail.com>
Message-ID: <49D69169.2070208@apache.org>
Date: Fri, 03 Apr 2009 15:44:57 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.21 (X11/20090318)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [PROPOSAL] new subproject: Avro
References: <49D53694.1050906@apache.org> <3DBB0E23-7B6D-4541-A0D6-3687DF02C397@yahoo-inc.com> <EAFDEC03CDA5D644878904F4A8F0158A6191AC88EA@NA-EXMSG-C103.redmond.corp.microsoft.com>
In-Reply-To: <EAFDEC03CDA5D644878904F4A8F0158A6191AC88EA@NA-EXMSG-C103.redmond.corp.microsoft.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Jim Kellerman (POWERSET) wrote:
> It is also my understanding (based on the email thread) that Avro only
> supports Java and python. That is a step backwards from Thrift.

We intend to add support for more languages.  Avro is not complete.

> It appears that Avro uses introspection heavily, which is expensive in
> applications that require a high message rate.

It only uses introspection if you wish to use your existing Java classes 
to represent Avro data.  There are three representations in Java: 
generic (uses Map<String,Object> for records, List<Object> for arrays), 
specific (generates a java class for each Avro record, like Thrift) and 
reflect (uses reflection to access existing classes).  So introspection 
is optional.  And, while introspection is indeed slow for processing 
file-based data, it would probably not a bottleneck for most RPC 
protocols and might be a useful tool to migrate existing code to Avro.

> So I guess my question is why Avro?

The compelling case is dynamic data types.  Pig, Hive, Python, Perl etc. 
scripts should not have to generate a Thrift IDL file each time they 
wish to write a data file with a new schema, nor should they need to run 
the Thrift compiler for each data file they wish to read.  For 
production applications, code-generation is not an imposition and may 
offer increased opportunities for optimization and error checking, but 
for exploration and experimentation, a very common use case for Hadoop, 
one would like to be able to browse datasets and build mapreduce 
programs more interactively.

Doug

From general-return-169-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 06 05:34:37 2009
Return-Path: <general-return-169-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58094 invoked from network); 6 Apr 2009 05:34:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Apr 2009 05:34:36 -0000
Received: (qmail 66419 invoked by uid 500); 6 Apr 2009 05:34:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66328 invoked by uid 500); 6 Apr 2009 05:34:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66318 invoked by uid 99); 6 Apr 2009 05:34:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Apr 2009 05:34:35 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Apr 2009 05:34:26 +0000
Received: from [192.168.1.71] (snvvpn2-10-72-76-c109.hq.corp.yahoo.com [10.72.76.109])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n365XrDd098512
	for <general@hadoop.apache.org>; Sun, 5 Apr 2009 22:33:53 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:mime-version:
	subject:date:references:x-mailer;
	b=Z4Q6iq5gMTH5ez4rdiLhswILv32YQpYcMozFAIigeESdr56Mj5C5VeLAN2MZisI2
Message-Id: <9645918B-C037-4219-8629-05686401793B@yahoo-inc.com>
From: Sanjay Radia <sradia@yahoo-inc.com>
To: <general@hadoop.apache.org>
In-Reply-To: <49D53694.1050906@apache.org>
Content-Type: multipart/alternative; boundary=Apple-Mail-2-843315381
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: Re: [PROPOSAL] new subproject: Avro
Date: Sun, 5 Apr 2009 22:33:56 -0700
References: <49D53694.1050906@apache.org>
X-Mailer: Apple Mail (2.930.3)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-2-843315381
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


On Apr 2, 2009, at 3:05 PM, Doug Cutting wrote:

> I propose we add a new Hadoop subproject for Avro, a serialization
> system.  My ambition is for Avro to replace both Hadoop's RPC and to  
> be
> used for most Hadoop data files, e.g., by Pig, Hive, etc.
>
> Initial committers would be Sharad Agarwal and me, both existing  
> Hadoop
> committers.  We are the sole authors of this software to date.
>
> The code is currently at:
>
> http://people.apache.org/~cutting/avro.git/
>
> To learn more:
>
> git clone http://people.apache.org/~cutting/avro.git/ avro
> cat avro/README.txt
>
> Comments?  Questions?
>
> Doug
>
+1

sanjay
--Apple-Mail-2-843315381--

From general-return-170-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 06 05:43:24 2009
Return-Path: <general-return-170-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63571 invoked from network); 6 Apr 2009 05:43:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Apr 2009 05:43:24 -0000
Received: (qmail 70000 invoked by uid 500); 6 Apr 2009 05:43:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69922 invoked by uid 500); 6 Apr 2009 05:43:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69912 invoked by uid 99); 6 Apr 2009 05:43:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Apr 2009 05:43:22 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 74.125.46.29 as permitted sender)
Received: from [74.125.46.29] (HELO yw-out-2324.google.com) (74.125.46.29)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Apr 2009 05:43:16 +0000
Received: by yw-out-2324.google.com with SMTP id 2so1243743ywt.29
        for <general@hadoop.apache.org>; Sun, 05 Apr 2009 22:42:55 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=iTyphvajyxON0ASIjdk5Mq5uTGByNbAumwfhoWx0BLs=;
        b=Z4R67wuByruh0FuMgWhPjbPkRrxve2p13pCFkzliA9LgQ208AldlQ6YJU6ON2Q3BKa
         i+yD9yX5UQiFjasI0aD6Hy9b6xlUwpv/FD9iDzknumwcWgDNSwvX/2JrvcogFhlwk3vr
         MXB8sI9g7Qw1gJXFic/HSi6hKomDZdpsXZpG0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=QV8H2vuq0BnoBnyjHFBinUmFQl+KCfjFMTpZ7fEG/wrXFqx49wtA+NnVMpsjBDlDou
         d8T087meEfHgUaAheiqTKH52wmPNQb+Kcoew6UbCMtSryQTj8AqwKtQbJKCNvDioLfW4
         N1BlE16+Jx18lB/OBLp9pXfcclvLjfshXk29U=
MIME-Version: 1.0
Received: by 10.151.103.11 with SMTP id f11mr7743340ybm.80.1238996575139; Sun, 
	05 Apr 2009 22:42:55 -0700 (PDT)
In-Reply-To: <9645918B-C037-4219-8629-05686401793B@yahoo-inc.com>
References: <49D53694.1050906@apache.org>
	 <9645918B-C037-4219-8629-05686401793B@yahoo-inc.com>
Date: Sun, 5 Apr 2009 22:42:55 -0700
Message-ID: <4aa34eb70904052242s2fe92931n1d3963a671b9cbce@mail.gmail.com>
Subject: Re: [PROPOSAL] new subproject: Avro
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00151750e7f0bd3a690466dc5e2d
X-Virus-Checked: Checked by ClamAV on apache.org

--00151750e7f0bd3a690466dc5e2d
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

+1. Awesome!

-dhruba

On Sun, Apr 5, 2009 at 10:33 PM, Sanjay Radia <sradia@yahoo-inc.com> wrote:

>
> On Apr 2, 2009, at 3:05 PM, Doug Cutting wrote:
>
>  I propose we add a new Hadoop subproject for Avro, a serialization
>> system.  My ambition is for Avro to replace both Hadoop's RPC and to be
>> used for most Hadoop data files, e.g., by Pig, Hive, etc.
>>
>> Initial committers would be Sharad Agarwal and me, both existing Hadoop
>> committers.  We are the sole authors of this software to date.
>>
>> The code is currently at:
>>
>> http://people.apache.org/~cutting/avro.git/<http://people.apache.org/%7Ecutting/avro.git/>
>>
>> To learn more:
>>
>> git clone http://people.apache.org/~cutting/avro.git/<http://people.apache.org/%7Ecutting/avro.git/>avro
>> cat avro/README.txt
>>
>> Comments?  Questions?
>>
>> Doug
>>
>>  +1
>
> sanjay

--00151750e7f0bd3a690466dc5e2d--

From general-return-171-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 06 07:23:50 2009
Return-Path: <general-return-171-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14039 invoked from network); 6 Apr 2009 07:23:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Apr 2009 07:23:49 -0000
Received: (qmail 40253 invoked by uid 500); 6 Apr 2009 07:23:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40200 invoked by uid 500); 6 Apr 2009 07:23:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40190 invoked by uid 99); 6 Apr 2009 07:23:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Apr 2009 07:23:49 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [76.13.9.53] (HELO web65509.mail.ac4.yahoo.com) (76.13.9.53)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 06 Apr 2009 07:23:39 +0000
Received: (qmail 23889 invoked by uid 60001); 6 Apr 2009 07:23:18 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1239002598; bh=UGhBDEehlvVQay2BUkN0BmTxhG+AMwJ5Ox/HOXZAQs0=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:MIME-Version:Content-Type; b=XpVKLzjReDdpa0mQbenUumKspV1kQzUJJG0YfgbjIW8IjIBUiyahtEPWkudPg/2TPfTh3DOelDp7VLAveyo8DArAOyRapxl5ADypMJd5nj2Ae7iYBvE4mXXiYePYkygW4nxyAnZ4j+Oyjlaf+eWedMuPI6JyLTmWDIDWQcipl3g=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:MIME-Version:Content-Type;
  b=YSbahDhUCzdclcd7rSFgdRBL2Si1B3js8GN983B8+B5zr0i8PuzADjD5DLCW4xWjFqjvXG7hStTmuxYZ5+3eMsxVIu+25ItxbiilItGUt69t7lizf9hQdm7yhs6xVBCVNEDNS0c3h45VF9oNHxFRpqGXLNZnrgMqYplOcxqMHgk=;
Message-ID: <417838.23294.qm@web65509.mail.ac4.yahoo.com>
X-YMail-OSG: nrCQCcAVM1msVeYNEdnj5WkU3AKnH8UgVyR7zLkK7fBnvVeUWrPhhx1_2oeqa5PQs8fpuMKUwhrhUaOuDkeaqvtqAiwCu2fG4mJf.jgGxebHcFJfBDeRbUl27YJhJHl8OpXnH8YaxKeBNOotVdPfkq17gfJlh4jizdKClYMqQTz.FBvX6wdRdZQoFszhzLtEirf3wqimRQKf9G_yCCR7V4fOFVtWwozaelrOvSaA89ax5qiPOOEbXUWtdLVvkC9r1hYiQ7cT6kYBwgLefb.m.GN8r767jFnNEcAkSySy6mDew7SB2UEBHm.p8efczoWBpkDvaOkuEUslnc4XL0Y-
Received: from [75.0.178.207] by web65509.mail.ac4.yahoo.com via HTTP; Mon, 06 Apr 2009 00:23:18 PDT
X-Mailer: YahooMailRC/1277.35 YahooMailWebService/0.7.289.1
Date: Mon, 6 Apr 2009 00:23:18 -0700 (PDT)
From: Chad Walters <chad_walters@yahoo.com>
Subject: RE: [PROPOSAL] new subproject: Avro
To: general@hadoop.apache.org
Cc: thrift-user@incubator.apache.org, thrift-dev@incubator.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org


Cross-posting to the Thrift dev and user lists since folks there may be interested in this. It appears that my attempts to subscribe to general@hadoop.apache.org from my work email were silently failing somewhere along the line -- I'll try not to take it personally. ;) Some others have experienced this too -- so if you didn't get a subscription confirmation message, then it failed. Try from a different address, I guess. You can view the thread here without being subscribed: http://mail-archives.apache.org/mod_mbox/hadoop-general/200904.mbox/browser

Doug,

First, let me say that I think Avro has a lot of useful features -- features that I would like to see fully supported in Thrift. At a minimum, I would like for us to be able to hash out the details to guarantee that there can really be full interoperability between Avro and Thrift. I am really interested in working cooperatively and collaboratively on this and I am willing to put in significant time on design and communication to help make full interoperability possible (I am unfortunately not able to contribute code directly at this time).

Second, I think all of this decision about where Avro should live requires more thought and more discussion. I'd love to hear from more folks outside of Yahoo on this topic: so far all of the +1 votes have come from Yahoo employees. I'd also love to hear from other folks who have significant investments in both Thrift and Hadoop.

Some points to think about:

-- You suggest that there is not a lot in Thrift that Avro can leverage. I think you may be overlooking the fact that Thrift has a user base and a community of developers who are very interested in issues of cross-language data serialization and interoperability. Thrift has committers with expertise in a pretty big set of languages and leveraging this could get Avro's functionality onto more languages faster than the current path. Also, there is in fact significant overlap between Hadoop users and Thrift users at this point, as well as significant use of Thrift in more than one Hadoop sub-project.

At the code level, Thrift contains a transport abstraction and multiple different transport and server implementations in many different target languages. If there were closer collaboration, Avro could certainly benefit from leveraging the existing ones and any additional contributions in this area would benefit both projects.

-- You also suggest that the two are largely disjoint from a technical perspective:
"Thrift fundamentally standardizes an API, not a data format.
Avro fundamentally is a data format specification, like XML."

I agree with the fundamental part but I think that doesn't bring to light enough of what is in common and what is different for purposes of this discussion.

Thrift specifies a type system, an API for data formats and transport mechanisms, a schema resolution algorithm, and provides implementations of several distinct data formats and transports.

Avro specifies a single data format but it also brings along several other things as well, including a type system, specific RPC mechanism and a schema resolution algorithm.

The most significant issue is that both of them specify a type system. At a very minimum I would like to see Avro and Thrift make agreements on that type system. The fact that there is significant existing investment in the Thrift type system by the Thrift community should weigh somewhere in this discussion. Obviously, the technical needs of Avro will also have weight there, but where there is room for choice, the Thrift choices should be respected. Arbitrary changes here will make it unnecessarily painful, perhaps impossible, for Thrift to directly adopt Avro and instead Thrift will be forced to make an "Avro-like" data specification, hampering interoperability for everyone.

There may be pitfalls in the other areas of overlap as well that would prevent real interoperability -- let's elucidate them in further discussions.

-- Avro appears to have 3 primary features that Thrift does not currently support sufficiently:
1. Schema serialization allowing for compact representation of files containing large numbers of records of identical types
2. Dynamic interpretation of schemas, which improves ease-of-use in dynamic languages (like the Python Hadoop Streaming use case)
3. Lazy partial deserialization to support "projection"

Note that features 1 and 3 are independent of whether schemas are dynamicly interpreted or compiled into static bindings.

WRT #1: Thrift's DenseProtocol goes some distance towards this although it doesn't go the whole way. Thrift can easily be extended to further compact the DenseProtocol's wire format for special cases where all fields are required. We have had significant discussions on the Thrift list about doing more in this area previously but we couldn't get folks from Hadoop who cared most about this use case to participate with us on capturing a complete set of requirements and so there was no strong driver for it.

WRT #2: I totally understand the case you make for dynamic interpretation in ad hoc data processing. I would love to see Thrift enhanced to do this kind of thing.

WRT #3: Partial deserialization seems like a really useful feature for several use cases, not just for "projection". I think Thrift could and should be extended to support this functionality, and it should be available for both static bindings and dynamic schema interpretation via field names and field IDs where possible.

-- You state:
"Perhaps Thrift could be augmented to support Avro's JSON schemas and 
serialization.  Then it could interoperate with other Avro-based 
systems.  But then Thrift would have yet another serialization format, 
that every language would need to implement for it to be useful..."

First, that "Perhaps" hides a lot of complexity and unless that is hashed out ahead of time I am pretty sure the real answer will be "Thrift cannot be augmented to support Avro directly but instead could be augmented to support something that looks quite a bit like Avro but differs in mostly unimportant ways." To me that seems like a shame.

Furthermore, you say that last part ("Thrift would have yet another serialization format...") like it is a bad thing... Note that it is an explicit design goal of Thrift to allow for multiple different serialization formats so that lots of different use cases can be supported by the same fundamental framework. This is a clear recognition that there is no one-size-fits-all answer for data serialization (fast RPC vs compact archival record data vs human readability, to name a few salient use cases). For a compelling enough use case, there is no reason not to port new protocols across multiple languages (generally done on an as-needed basis by someone who wants that functionality in that language). Another great feature of the protocol abstraction is that it allows data to be seamlessly moved from one serialization format to another as, say, it is read out of archival storage and sent on as RPC.

Also, doesn't Avro essentially contain "another serialization format that every language would need to implement for it to be useful"? Seems like the same basic set of work to me, whether it is in Avro or Thrift.

-- You state:
"Avro fundamentally is a data format specification, like XML.  Thrift could implement this specification.  The Avro project includes reference implementations, but the format is intended to be simple enough and the specification stable enough that others might reasonably develop alternate, independent implementations."

I think this is a bit inaccurate. First there is the issue of type system compatibility that I raised above and the plausibility of satisfying that "could" without refinement and collaboration on Avro's specification. Furthermore, stated goal of the subproject is "for Avro to replace both Hadoop's RPC and to be used for most Hadoop data files". This will bring in quite a bit beyond a reference implementation of a data format specification, especially depending on how many languages you intend to build RPC support for (Java, Python, C++ all mentioned at some point -- others?). I don't think it is unreasonable that the significant proportion of folks in the Hadoop community who are also using Thrift are puzzled about why there isn't more consideration being given to convergence between Avro and Thrift.

-- You state:
"Also, with the schema, resolving version differences is simplified. 
Developers don't need to assign field numbers, but can just use names. 
For performance, one can internally use field numbers while reading, to 
avoid string comparisons, but developers need no longer specify these, 
but can use names, as in most software.  Here having the schema means we 
can simplify the IDL and its versioning semantics."

The simplification comes simply not having the field IDs in the IDL? I am not sure why having sequential id numbers after each field is considered to be so onerous. I honestly have never heard a single Thrift user complain about this. Anyone doing more than just that is doing something advanced that wouldn't be possible without the field IDs (like renaming a field). I think having to deal with JSON syntax in the Avro IDL is actually more annoying for humans than the application of field IDs, both with the added syntactic punctuation and the increased verbosity. If the field IDs are really so objectionable, Thrift could allow them to be optional for purely dynamic usages.

I also don't see why matching names is considered easier than matching numbers, which is essentially what the versioning semantics come down to in the end. Am I missing something here?

-- You state:
"Would you write parsers for Thrift's IDL in every language?  Or would 
you use JSON, as Avro does, to avoid that?"

Here I totally agree with you: a JSON IDL is better for machine parsing than Thrift's current IDL, which is targeted more at human parsing. And given that I agree that some form of dynamic interpretation is a useful feature, I don't see any reason why a JSON version of the IDL couldn't become part of the picture. Furthermore, the Thrift IDL compiler could easily be extended to take this JSON format as both an input (in addition to the current Thrift IDL) and output.

An alternative is would just be to have the other languages bind to the Thrift IDL parser directly -- most languages bind to C (granted for some it is easier than others) -- and get back the parsed data structure to interpret off of.

-- By making Avro a sub-project of Hadoop, I believe you will succeed in producing an improved version of Hadoop Record IO and a better RPC mechanism than the current Hadoop RPC. However, I don't think that this will result in better general RPC than Thrift and it will certainly be much less performant for RPC in a wide range of applications.

Consider an alternative: making Avro more like a sub-project of Thrift or just implementing it directly in Thrift. In that case, I think the end result will be a powerful and flexible "one-stop shop" for data serialization for RPC and archival purposes with the ability to bring both static and dynamic capabilities as needed for particular application purposes. To me this seems like a bigger win for both Hadoop and for Thrift.

Thanks for reading through to this point. I look forward to further discussion.

Chad

From general-return-172-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 06 17:37:02 2009
Return-Path: <general-return-172-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 40012 invoked from network); 6 Apr 2009 17:37:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Apr 2009 17:37:02 -0000
Received: (qmail 76829 invoked by uid 500); 6 Apr 2009 17:37:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76760 invoked by uid 500); 6 Apr 2009 17:37:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 23852 invoked by uid 99); 6 Apr 2009 06:51:58 -0000
X-ASF-Spam-Status: No, hits=-8.0 required=10.0
	tests=RCVD_IN_DNSWL_HI,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of Chad.Walters@microsoft.com designates 131.107.115.214 as permitted sender)
From: Chad Walters <Chad.Walters@microsoft.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Sun, 5 Apr 2009 23:51:27 -0700
Subject: RE: [PROPOSAL] new subproject: Avro
Thread-Topic: [PROPOSAL] new subproject: Avro
Thread-Index: Acm2eIiMha+SR8YIcEuSMApUfi2fXgAC5Lx/
Message-ID: <C5FEF47F.90BAC%cwalter@microsoft.com>
In-Reply-To: <C5FEE114.90BA5%cwalter@microsoft.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Doug,

First, let me say that I think Avro has a lot of useful features -- feature=
s
that I would like to see fully supported in Thrift. At a minimum, I would
like for us to be able to hash out the details to guarantee that there can
really be full interoperability between Avro and Thrift. I am really
interested in working cooperatively and collaboratively on this and I am
willing to put in significant time on design and communication to help make
full interoperability possible (I am unfortunately not able to contribute
code directly at this time).

Second, I think all of this decision about where Avro should live requires
more thought and more discussion. I'd love to hear from more folks outside
of Yahoo on this topic: so far all of the +1 votes have come from Yahoo
employees. I'd also love to hear from other folks who have significant
investments in both Thrift and Hadoop.

Some points to think about:

-- You suggest that there is not a lot in Thrift that Avro can leverage. I
think you may be overlooking the fact that Thrift has a user base and a
community of developers who are very interested in issues of cross-language
data serialization and interoperability. Thrift has committers with
expertise in a pretty big set of languages and leveraging this could get
Avro's functionality onto more languages faster than the current path. Also=
,
there is in fact significant overlap between Hadoop users and Thrift users
at this point, as well as significant use of Thrift in more than one Hadoop
sub-project.

At the code level, Thrift contains a transport abstraction and multiple
different transport and server implementations in many different target
languages. If there were closer collaboration, Avro could certainly benefit
from leveraging the existing ones and any additional contributions in this
area would benefit both projects.

-- You also suggest that the two are largely disjoint from a technical
perspective:
"Thrift fundamentally standardizes an API, not a data format.
Avro fundamentally is a data format specification, like XML."

I agree with the fundamental part but I think that doesn't bring to light
enough of what is in common and what is different for purposes of this
discussion.

Thrift specifies a type system, an API for data formats and transport
mechanisms, a schema resolution algorithm, and provides implementations of
several distinct data formats and transports.

Avro specifies a single data format but it also brings along several other
things as well, including a type system, specific RPC mechanism and a schem=
a
resolution algorithm.

The most significant issue is that both of them specify a type system. At a
very minimum I would like to see Avro and Thrift make agreements on that
type system. The fact that there is significant existing investment in the
Thrift type system by the Thrift community should weigh somewhere in this
discussion. Obviously, the technical needs of Avro will also have weight
there, but where there is room for choice, the Thrift choices should be
respected. Arbitrary changes here will make it unnecessarily painful,
perhaps impossible, for Thrift to directly adopt Avro and instead Thrift
will be forced to make an "Avro-like" data specification, hampering
interoperability for everyone.

There may be pitfalls in the other areas of overlap as well that would
prevent real interoperability -- let's elucidate them in further
discussions.

-- Avro appears to have 3 primary features that Thrift does not currently
support sufficiently:
1. Schema serialization allowing for compact representation of files
containing large numbers of records of identical types
2. Dynamic interpretation of schemas, which improves ease-of-use in dynamic
languages (like the Python Hadoop Streaming use case)
3. Lazy partial deserialization to support "projection"

Note that features 1 and 3 are independent of whether schemas are dynamicly
interpreted or compiled into static bindings.

WRT #1: Thrift's DenseProtocol goes some distance towards this although it
doesn't go the whole way. Thrift can easily be extended to further compact
the DenseProtocol's wire format for special cases where all fields are
required. We have had significant discussions on the Thrift list about doin=
g
more in this area previously but we couldn't get folks from Hadoop who care=
d
most about this use case to participate with us on capturing a complete set
of requirements and so there was no strong driver for it.

WRT #2: I totally understand the case you make for dynamic interpretation i=
n
ad hoc data processing. I would love to see Thrift enhanced to do this kind
of thing.

WRT #3: Partial deserialization seems like a really useful feature for
several use cases, not just for "projection". I think Thrift could and
should be extended to support this functionality, and it should be availabl=
e
for both static bindings and dynamic schema interpretation via field names
and field IDs where possible.

-- You state:
"Perhaps Thrift could be augmented to support Avro's JSON schemas and
serialization.  Then it could interoperate with other Avro-based
systems.  But then Thrift would have yet another serialization format,
that every language would need to implement for it to be useful..."

First, that "Perhaps" hides a lot of complexity and unless that is hashed
out ahead of time I am pretty sure the real answer will be "Thrift cannot b=
e
augmented to support Avro directly but instead could be augmented to suppor=
t
something that looks quite a bit like Avro but differs in mostly unimportan=
t
ways." To me that seems like a shame.

Furthermore, you say that last part ("Thrift would have yet another
serialization format...") like it is a bad thing... Note that it is an
explicit design goal of Thrift to allow for multiple different serializatio=
n
formats so that lots of different use cases can be supported by the same
fundamental framework. This is a clear recognition that there is no
one-size-fits-all answer for data serialization (fast RPC vs compact
archival record data vs human readability, to name a few salient use cases)=
.
For a compelling enough use case, there is no reason not to port new
protocols across multiple languages (generally done on an as-needed basis b=
y
someone who wants that functionality in that language). Another great
feature of the protocol abstraction is that it allows data to be seamlessly
moved from one serialization format to another as, say, it is read out of
archival storage and sent on as RPC.

Also, doesn't Avro essentially contain "another serialization format that
every language would need to implement for it to be useful"? Seems like the
same basic set of work to me, whether it is in Avro or Thrift.

-- You state:
"Avro fundamentally is a data format specification, like XML.  Thrift could
implement this specification.  The Avro project includes reference
implementations, but the format is intended to be simple enough and the
specification stable enough that others might reasonably develop alternate,
independent implementations."

I think this is a bit inaccurate. First there is the issue of type system
compatibility that I raised above and the plausibility of satisfying that
"could" without refinement and collaboration on Avro's specification.
Furthermore, stated goal of the subproject is "for Avro to replace both
Hadoop's RPC and to be used for most Hadoop data files". This will bring in
quite a bit beyond a reference implementation of a data format
specification, especially depending on how many languages you intend to
build RPC support for (Java, Python, C++ all mentioned at some point --
others?). I don't think it is unreasonable that the significant proportion
of folks in the Hadoop community who are also using Thrift are puzzled abou=
t
why there isn't more consideration being given to convergence between Avro
and Thrift.

-- You state:
"Also, with the schema, resolving version differences is simplified.
Developers don't need to assign field numbers, but can just use names.
For performance, one can internally use field numbers while reading, to
avoid string comparisons, but developers need no longer specify these,
but can use names, as in most software.  Here having the schema means we
can simplify the IDL and its versioning semantics."

The simplification comes simply not having the field IDs in the IDL? I am
not sure why having sequential id numbers after each field is considered to
be so onerous. I honestly have never heard a single Thrift user complain
about this. Anyone doing more than just that is doing something advanced
that wouldn't be possible without the field IDs (like renaming a field). I
think having to deal with JSON syntax in the Avro IDL is actually more
annoying for humans than the application of field IDs, both with the added
syntactic punctuation and the increased verbosity. If the field IDs are
really so objectionable, Thrift could allow them to be optional for purely
dynamic usages.

I also don't see why matching names is considered easier than matching
numbers, which is essentially what the versioning semantics come down to in
the end. Am I missing something here?

-- You state:
"Would you write parsers for Thrift's IDL in every language?  Or would
you use JSON, as Avro does, to avoid that?"

Here I totally agree with you: a JSON IDL is better for machine parsing tha=
n
Thrift's current IDL, which is targeted more at human parsing. And given
that I agree that some form of dynamic interpretation is a useful feature, =
I
don't see any reason why a JSON version of the IDL couldn't become part of
the picture. Furthermore, the Thrift IDL compiler could easily be extended
to take this JSON format as both an input (in addition to the current Thrif=
t
IDL) and output.

An alternative is would just be to have the other languages bind to the
Thrift IDL parser directly -- most languages bind to C (granted for some it
is easier than others) -- and get back the parsed data structure to
interpret off of.

-- By making Avro a sub-project of Hadoop, I believe you will succeed in
producing an improved version of Hadoop Record IO and a better RPC mechanis=
m
than the current Hadoop RPC. However, I don't think that this will result i=
n
better general RPC than Thrift and it will certainly be much less performan=
t
for RPC in a wide range of applications.

Consider an alternative: making Avro more like a sub-project of Thrift or
just implementing it directly in Thrift. In that case, I think the end
result will be a powerful and flexible "one-stop shop" for data
serialization for RPC and archival purposes with the ability to bring both
static and dynamic capabilities as needed for particular application
purposes. To me this seems like a bigger win for both Hadoop and for Thrift=
.

Thanks for reading through to this point. I look forward to further
discussion.

Chad


From general-return-173-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 06 19:13:05 2009
Return-Path: <general-return-173-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 89172 invoked from network); 6 Apr 2009 19:13:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Apr 2009 19:13:05 -0000
Received: (qmail 23016 invoked by uid 500); 6 Apr 2009 19:13:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22937 invoked by uid 500); 6 Apr 2009 19:13:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22927 invoked by uid 99); 6 Apr 2009 19:13:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Apr 2009 19:13:04 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cutting@gmail.com designates 209.85.200.170 as permitted sender)
Received: from [209.85.200.170] (HELO wf-out-1314.google.com) (209.85.200.170)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Apr 2009 19:12:56 +0000
Received: by wf-out-1314.google.com with SMTP id 23so2217131wfg.2
        for <general@hadoop.apache.org>; Mon, 06 Apr 2009 12:12:35 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:sender:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=jpyo13yTdEPxVyrj7win3rutTiQyQanhNXbSHrbUmnw=;
        b=KN6x0SAZuJABIca3BrIFmeGy9Q4aazr2khJGgE+EJZnX/4uKT3t3NTw5OTr7Fcje7F
         ylTMfkYCIh1BxyzZ0qzoSf0pD7vjYMbplIQsS5bbrZNDfsDPxUoal2bp/z9/7pjb6OwP
         45fqmhrm0lCFCzKvIiuYnhcncTpmZhiT/tpKQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:date:from:user-agent:mime-version:to:subject
         :references:in-reply-to:content-type:content-transfer-encoding;
        b=vXYgfn9f+5YXhIhFIqMmqMCaRJuaveAP02xG3LrZR4ZuFzdoUlFyaWyMjSxNpd2tX/
         xS8H0dJi2oxuj6RM2kQrRYOJsiacVMsj3zUeZJgrnLBPrUqslYKu2+jkce2HDCF9PX86
         ZhlBjda8Wqbm+LeCrSIWnILeItP+dtGO9edHE=
Received: by 10.142.174.13 with SMTP id w13mr1372145wfe.156.1239045155856;
        Mon, 06 Apr 2009 12:12:35 -0700 (PDT)
Received: from ?192.168.168.16? (c-76-103-155-128.hsd1.ca.comcast.net [76.103.155.128])
        by mx.google.com with ESMTPS id 9sm954963wfc.0.2009.04.06.12.12.34
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Mon, 06 Apr 2009 12:12:34 -0700 (PDT)
Sender: Doug Cutting <cutting@gmail.com>
Message-ID: <49DA5420.4030208@apache.org>
Date: Mon, 06 Apr 2009 12:12:32 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.21 (X11/20090318)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [PROPOSAL] new subproject: Avro
References: <417838.23294.qm@web65509.mail.ac4.yahoo.com>
In-Reply-To: <417838.23294.qm@web65509.mail.ac4.yahoo.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Chad Walters wrote:
> -- You suggest that there is not a lot in Thrift that Avro can
> leverage. I think you may be overlooking the fact that Thrift has a
> user base and a community of developers who are very interested in
> issues of cross-language data serialization and interoperability.

I meant that in terms of common code, not coders.  Coders can belong to 
more than one community but code should generally not.  Hadoop Core has 
become a sprawling community that we're trying to split.  It's more 
productive to have have more, small communities than few large ones.  A 
project needs a handful of active developers, but too many and it 
becomes ungainly.  So, if it's technically possible for a codebase to be 
distinct, and it can attract enough active developers to sustain itself, 
that is a preferable structure.

> At the code level, Thrift contains a transport abstraction and
> multiple different transport and server implementations in many
> different target languages. If there were closer collaboration, Avro
> could certainly benefit from leveraging the existing ones and any
> additional contributions in this area would benefit both projects.

The transport and server implementations are indeed an area where code 
could potentially be shared between Avro and Thrift.  Perhaps someone 
could start a separate project with reusable transport and server 
implementations to support RPC?  In any case, Avro primarily specifies a 
binary message format, not a full transport.  We hope to piggyback off 
other transport implementations, like HTTP servers, etc.  Full 
transports involve authentication, authorization, encryption, etc., 
which are outside of the scope of Avro.

> The most significant issue is that both of them specify a type
> system. At a very minimum I would like to see Avro and Thrift make
> agreements on that type system.

This makes good sense.  It would be good if these were interoperable.

Thrift has byte and i16, which Avro does not currently.  I'd like to add 
a fixed<n> primitive type to Avro, where n is the number of bytes and is 
specified in the schema, so that one could, e.g., define a byte as 
fixed<1>, i16 as a fixed<2> and md5 as a fixed<16>.

Thrift has both lists and sets, Avro has just arrays, which are 
equivalent to lists (they're ordered).  Perhaps Avro could add sets. 
Are they leveraged heavily in Thrift?  I've not heard much call for them 
in Avro yet.

Avro has single-float, Thrift does not.  Avro could perhaps lose this.

Avro distinguishes UTF-8 text strings from byte strings, while Thrift 
does not.  I am reluctant to lose this distinction.

Avro has unions and a null type, while Thrift does not.  Does Thrift 
support recursive data structures?

> Furthermore, you say that last part ("Thrift would have yet another
> serialization format...") like it is a bad thing... 

When faced with multiple programming and scripting languages, multiple 
serialization formats should be discouraged, or one ends up with 
multiplicative compatibility problems.  A single, primary data format 
would vastly simplify the Hadoop ecosystem.  Yes, folks need to be able 
to easily import and export data, but expecting scripts in arbitrary 
languages to be able to process data in arbitrary formats seems unwise.

> Note that it is
> an explicit design goal of Thrift to allow for multiple different
> serialization formats so that lots of different use cases can be
> supported by the same fundamental framework.

That's not a design goal of Avro, which intends to provide a single, 
well-specified, easy to implement serialization format.  This is not in 
conflict with Thrift, it's just a different goal.

> Also, doesn't Avro essentially contain "another serialization format
> that every language would need to implement for it to be useful"?
> Seems like the same basic set of work to me, whether it is in Avro or
> Thrift.

None of Thrift's existing formats solve the problems Avro seeks to. 
Thrift may be able to incorporate Avro's format, if it has good format 
generalizations, ideally using Avro's code.  So there should be little 
duplication of effort in such an approach.

> The simplification comes simply not having the field IDs in the IDL?
> I am not sure why having sequential id numbers after each field is
> considered to be so onerous.

I didn't say it was onerous, I said that, like in most data structure 
languages (e.g., programming languages), Avro permits folks to name 
fields with symbolic names alone.  In human-authored software, symbolic 
naming is generally preferable to numeric naming.  Is that really a 
matter of dispute?

> If the field IDs are really so
> objectionable, Thrift could allow them to be optional for purely
> dynamic usages.

Optional features increase compatibility complexity and are harder to 
maintain and test.  A Thrift IDL without numbers would not provide 
versioning features to non-dynamic languages.

> I also don't see why matching names is considered easier than
> matching numbers, which is essentially what the versioning semantics
> come down to in the end. Am I missing something here?

They are formally equivalent.  For machines, matching numbers is easier, 
but people usually prefer to operate on names, and names can be 
automatically mapped to numbers.

> Consider an alternative: making Avro more like a sub-project of
> Thrift or just implementing it directly in Thrift.

I looked into changing Thrift to support Avro's features, and it was 
very messy.  Perhaps someone else could do this more easily.

Building Avro as a part of Thrift would take considerably more effort 
for me and I think offer little more than it does separately.  If you 
feel differently, you are free to fork Avro, start a competitor, provide 
patches that integrate it into Thrift, or whatever.

> In that case, I
> think the end result will be a powerful and flexible "one-stop shop"
> for data serialization for RPC and archival purposes with the ability
> to bring both static and dynamic capabilities as needed for
> particular application purposes. To me this seems like a bigger win
> for both Hadoop and for Thrift.

It could be a floor wax and a dessert topping!

Doug


From general-return-174-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 06 20:09:24 2009
Return-Path: <general-return-174-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4697 invoked from network); 6 Apr 2009 20:09:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Apr 2009 20:09:24 -0000
Received: (qmail 15853 invoked by uid 500); 6 Apr 2009 20:09:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15776 invoked by uid 500); 6 Apr 2009 20:09:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15766 invoked by uid 99); 6 Apr 2009 20:09:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Apr 2009 20:09:23 +0000
X-ASF-Spam-Status: No, hits=1.6 required=10.0
	tests=MISSING_HEADERS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cutting@gmail.com designates 209.85.198.235 as permitted sender)
Received: from [209.85.198.235] (HELO rv-out-0506.google.com) (209.85.198.235)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Apr 2009 20:09:15 +0000
Received: by rv-out-0506.google.com with SMTP id k40so1932092rvb.29
        for <general@hadoop.apache.org>; Mon, 06 Apr 2009 13:08:55 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:sender:message-id:date:from
         :user-agent:mime-version:cc:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=4WOiax8QCDai2E03Wx+qh8VOP12Rr1U371dXILalhJo=;
        b=W4YxsiTUx6fw5joLSJsRNRnMrl/WeNv2B/hTdlB8aPb+RS92qgIYiVoculL1JCgPTZ
         s07LpkX94fdKlfZ9R+DgbRtoRuAS3dHv6qr5syBmIrVzuVk52UWte62YOGbrOhqKGltZ
         TksPIyVRDJ10FGYLoiIlLmK+7KaWkbjdCfRYM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:date:from:user-agent:mime-version:cc:subject
         :references:in-reply-to:content-type:content-transfer-encoding;
        b=wvatToVTtd3GvLogk3kuUwhr5tqO4cvS/Ug0BVsuGW9aJE3XS/2KTfltpA7irgBoDj
         xlDZLkuKeeIOfoai15DUZcNk4NRvqEd9ckqZQZMceH1AZJyKFwT29edu1rlDBMi3sFKR
         CaRRbL3UFjiUtH89jrEzOVyYr18Mfr4gdvPgM=
Received: by 10.142.178.9 with SMTP id a9mr1384373wff.73.1239048535171;
        Mon, 06 Apr 2009 13:08:55 -0700 (PDT)
Received: from ?192.168.168.16? (c-76-103-155-128.hsd1.ca.comcast.net [76.103.155.128])
        by mx.google.com with ESMTPS id 28sm7991961wfg.11.2009.04.06.13.08.53
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Mon, 06 Apr 2009 13:08:54 -0700 (PDT)
Sender: Doug Cutting <cutting@gmail.com>
Message-ID: <49DA6154.3040006@apache.org>
Date: Mon, 06 Apr 2009 13:08:52 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.21 (X11/20090318)
MIME-Version: 1.0
CC: general@hadoop.apache.org
Subject: Re: [PROPOSAL] new subproject: Avro
References: <417838.23294.qm@web65509.mail.ac4.yahoo.com>
In-Reply-To: <417838.23294.qm@web65509.mail.ac4.yahoo.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Chad Walters wrote:
> so far all of the +1 votes have come from Yahoo employees.

Not quite.  Jeff Hammerbacher is with Cloudera.  Sameer Paranjpye is 
with Mechanical Zoo.  Dhruba Borthakur is with Facebook.

Doug


From general-return-175-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 06 20:26:39 2009
Return-Path: <general-return-175-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9309 invoked from network); 6 Apr 2009 20:26:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Apr 2009 20:26:39 -0000
Received: (qmail 38854 invoked by uid 500); 6 Apr 2009 20:26:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38770 invoked by uid 500); 6 Apr 2009 20:26:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 38760 invoked by uid 99); 6 Apr 2009 20:26:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Apr 2009 20:26:39 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [192.18.43.133] (HELO sca-es-mail-2.sun.com) (192.18.43.133)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Apr 2009 20:26:30 +0000
Received: from fe-sfbay-09.sun.com ([192.18.43.129])
	by sca-es-mail-2.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n36KPvtJ009124
	for <general@hadoop.apache.org>; Mon, 6 Apr 2009 13:26:09 -0700 (PDT)
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; delsp=yes; format=flowed; charset=US-ASCII
Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com
 (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009))
 id <0KHP00F004L0AK00@fe-sfbay-09.sun.com> for general@hadoop.apache.org; Mon,
 06 Apr 2009 13:25:57 -0700 (PDT)
Received: from [192.168.1.40] ([unknown] [129.150.20.106])
 by fe-sfbay-09.sun.com
 (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009))
 with ESMTPSA id <0KHP005F04R7W880@fe-sfbay-09.sun.com> for
 general@hadoop.apache.org; Mon, 06 Apr 2009 13:25:55 -0700 (PDT)
Date: Mon, 06 Apr 2009 13:25:54 -0700
From: George Porter <George.Porter@Sun.COM>
Subject: Re: [PROPOSAL] new subproject: Avro
In-reply-to: <49DA6154.3040006@apache.org>
Sender: George.Porter@Sun.COM
To: general@hadoop.apache.org
Message-id: <1D314A78-E219-41DC-9900-CABEEE26AC55@Sun.com>
X-Mailer: Apple Mail (2.930.3)
References: <417838.23294.qm@web65509.mail.ac4.yahoo.com>
 <49DA6154.3040006@apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

One thing that you might consider is adding the ability to add path  
state to the serialization to better support tracing and  
instrumentation of the RPC layer with Avro.  I did something like this  
with Thrift, where I simply added a "hidden" parameter with a well  
known, but unused parameter.  If the receiver understood the tracing  
format, it could pull metadata from that parameter, and if not, than  
it would just ignore it.  Based on your description of the  
"projection" feature, it seems like that would apply here as well.

Thanks,
-George

On Apr 6, 2009, at 1:08 PM, Doug Cutting wrote:

> Chad Walters wrote:
>> so far all of the +1 votes have come from Yahoo employees.
>
> Not quite.  Jeff Hammerbacher is with Cloudera.  Sameer Paranjpye is  
> with Mechanical Zoo.  Dhruba Borthakur is with Facebook.
>
> Doug
>


From general-return-176-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 06 20:39:33 2009
Return-Path: <general-return-176-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12464 invoked from network); 6 Apr 2009 20:39:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Apr 2009 20:39:33 -0000
Received: (qmail 55374 invoked by uid 500); 6 Apr 2009 20:39:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55300 invoked by uid 500); 6 Apr 2009 20:39:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55290 invoked by uid 99); 6 Apr 2009 20:39:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Apr 2009 20:39:32 +0000
X-ASF-Spam-Status: No, hits=3.8 required=10.0
	tests=RCVD_NUMERIC_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.245.171.18] (HELO EXCHANGE1.agiliti.net) (216.245.171.18)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Apr 2009 20:39:23 +0000
Received: from 209.98.197.77 ([209.98.197.77]) by EXCHANGE1.agiliti.net ([216.245.171.17]) via Exchange Front-End Server exchange.digitalnorth.net ([216.245.171.30]) with Microsoft Exchange Server HTTP-DAV ;
 Mon,  6 Apr 2009 20:38:54 +0000
User-Agent: Microsoft-Entourage/11.4.0.080122
Date: Mon, 06 Apr 2009 15:38:53 -0500
Subject: Re: [PROPOSAL] new subproject: Avro
From: Brian Forney <bforney@integral7.com>
To: <general@hadoop.apache.org>
Message-ID: <C5FFD28D.1C32%bforney@integral7.com>
Thread-Topic: [PROPOSAL] new subproject: Avro
Thread-Index: Acm297LR8XKgmiLqEd6TrwAbY724hg==
In-Reply-To: <49D53694.1050906@apache.org>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 4/2/09 5:05 PM, "Doug Cutting" <cutting@apache.org> wrote:

> I propose we add a new Hadoop subproject for Avro, a serialization
> system.  My ambition is for Avro to replace both Hadoop's RPC and to be
> used for most Hadoop data files, e.g., by Pig, Hive, etc.
> 
> Initial committers would be Sharad Agarwal and me, both existing Hadoop
> committers.  We are the sole authors of this software to date.
> 
> The code is currently at:
> 
> http://people.apache.org/~cutting/avro.git/
> 
> To learn more:
> 
> git clone http://people.apache.org/~cutting/avro.git/ avro
> cat avro/README.txt
> 
> Comments?  Questions?
> 
> Doug
> 

+1


From general-return-177-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 06 20:49:13 2009
Return-Path: <general-return-177-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 15151 invoked from network); 6 Apr 2009 20:49:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Apr 2009 20:49:12 -0000
Received: (qmail 67402 invoked by uid 500); 6 Apr 2009 20:49:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67317 invoked by uid 500); 6 Apr 2009 20:49:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67276 invoked by uid 99); 6 Apr 2009 20:49:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Apr 2009 20:49:12 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cutting@gmail.com designates 209.85.200.173 as permitted sender)
Received: from [209.85.200.173] (HELO wf-out-1314.google.com) (209.85.200.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Apr 2009 20:49:02 +0000
Received: by wf-out-1314.google.com with SMTP id 23so2253244wfg.2
        for <general@hadoop.apache.org>; Mon, 06 Apr 2009 13:48:41 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:sender:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=qjciQXmwnBcEXKk+wH2x1Mz3WaY1BW+YumOy6rwVKzc=;
        b=m9hbx5tyZB+PwGVq+7iR4/ByESKhMPcNojN9N1UPEv79nCoLGlAZ6ykKT8OaPIElfI
         Y9fhnFH2p5z8vLgF5X6QJJXNkoS8pqU2QSNyhYtIoxgzvtEtivR9+SV3bacRyIdMLjJ6
         e/6q9ztHMhSjVwP5x3fbPDn0PY5WlwHkb3ibI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:date:from:user-agent:mime-version:to:subject
         :references:in-reply-to:content-type:content-transfer-encoding;
        b=JGtvHR9YcV+8rILB/XtvJzZaqOESazk63zPoTmeCHsRVxl5QmhQHrNxZaATfoC/bWn
         VmP+VtAoyn1oYYGi4cZGJ5yDhM4tfU+muDvtuO9mqsYEEolKy9enXI/p4nCYADh1hArW
         0ej39wIsD2HZVOFxxz4qCWIqJSzBQ1VuTy1Ks=
Received: by 10.142.100.1 with SMTP id x1mr1386068wfb.256.1239050921642;
        Mon, 06 Apr 2009 13:48:41 -0700 (PDT)
Received: from ?192.168.168.16? (c-76-103-155-128.hsd1.ca.comcast.net [76.103.155.128])
        by mx.google.com with ESMTPS id 24sm4865057wff.22.2009.04.06.13.48.40
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Mon, 06 Apr 2009 13:48:40 -0700 (PDT)
Sender: Doug Cutting <cutting@gmail.com>
Message-ID: <49DA6AA6.8010307@apache.org>
Date: Mon, 06 Apr 2009 13:48:38 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.21 (X11/20090318)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [PROPOSAL] new subproject: Avro
References: <417838.23294.qm@web65509.mail.ac4.yahoo.com> <49DA6154.3040006@apache.org> <1D314A78-E219-41DC-9900-CABEEE26AC55@Sun.com>
In-Reply-To: <1D314A78-E219-41DC-9900-CABEEE26AC55@Sun.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

George Porter wrote:
> One thing that you might consider is adding the ability to add path 
> state to the serialization to better support tracing and instrumentation 
> of the RPC layer with Avro.

Such cross-system tracing would be great, but I think this might best be 
done as part of request and response metadata, rather than made part of 
the user's request and response data.

You and I worked this nearly through for Hadoop's existing RPC:

http://issues.apache.org/jira/browse/HADOOP-4049

Unfortunately, that patch stalled awaiting benchmarks, and now is stale. 
  Avro currently specifies a request and response payload format, but 
does not say much about metadata.  In an HTTP-based transport, tracing 
might be done through headers.  As we adapt Hadoop's optimized RPC 
transport to support Avro messages we should probably insert a generic 
metadata layer, analogous to HTTP's headers, to support features such as 
this.

Doug

From general-return-178-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 06 21:03:00 2009
Return-Path: <general-return-178-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19954 invoked from network); 6 Apr 2009 21:03:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Apr 2009 21:03:00 -0000
Received: (qmail 89413 invoked by uid 500); 6 Apr 2009 21:02:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89325 invoked by uid 500); 6 Apr 2009 21:02:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89315 invoked by uid 99); 6 Apr 2009 21:02:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Apr 2009 21:02:59 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [192.18.43.132] (HELO sca-es-mail-1.sun.com) (192.18.43.132)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Apr 2009 21:02:50 +0000
Received: from fe-sfbay-09.sun.com ([192.18.43.129])
	by sca-es-mail-1.sun.com (8.13.7+Sun/8.12.9) with ESMTP id n36L2Uek015980
	for <general@hadoop.apache.org>; Mon, 6 Apr 2009 14:02:30 -0700 (PDT)
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; delsp=yes; format=flowed; charset=US-ASCII
Received: from conversion-daemon.fe-sfbay-09.sun.com by fe-sfbay-09.sun.com
 (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009))
 id <0KHP008005U8ON00@fe-sfbay-09.sun.com> for general@hadoop.apache.org; Mon,
 06 Apr 2009 14:02:30 -0700 (PDT)
Received: from [192.168.1.40] ([unknown] [129.150.20.106])
 by fe-sfbay-09.sun.com
 (Sun Java(tm) System Messaging Server 7.0-5.01 64bit (built Feb 19 2009))
 with ESMTPSA id <0KHP005T36G4W8A0@fe-sfbay-09.sun.com> for
 general@hadoop.apache.org; Mon, 06 Apr 2009 14:02:29 -0700 (PDT)
Date: Mon, 06 Apr 2009 14:02:28 -0700
From: George Porter <George.Porter@Sun.COM>
Subject: Re: [PROPOSAL] new subproject: Avro
In-reply-to: <49DA6AA6.8010307@apache.org>
Sender: George.Porter@Sun.COM
To: general@hadoop.apache.org
Message-id: <51A0963D-4522-4D16-95AF-125AFEC150E2@Sun.com>
X-Mailer: Apple Mail (2.930.3)
References: <417838.23294.qm@web65509.mail.ac4.yahoo.com>
 <49DA6154.3040006@apache.org> <1D314A78-E219-41DC-9900-CABEEE26AC55@Sun.com>
 <49DA6AA6.8010307@apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

>
> Unfortunately, that patch stalled awaiting benchmarks, and now is  
> stale.  Avro currently specifies a request and response payload  
> format, but does not say much about metadata.  In an HTTP-based  
> transport, tracing might be done through headers.  As we adapt  
> Hadoop's optimized RPC transport to support Avro messages we should  
> probably insert a generic metadata layer, analogous to HTTP's  
> headers, to support features such as this.
>
> Doug

Doug,

This would be great.  While it is always possible to add extension  
data to the payload itself, having such support in the transport  
itself can be quite useful, in my opinion.  Especially if there are a  
variety of different payloads, you don't have to extend each one.

-George

From general-return-179-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 06 21:06:23 2009
Return-Path: <general-return-179-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 20702 invoked from network); 6 Apr 2009 21:06:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Apr 2009 21:06:23 -0000
Received: (qmail 91453 invoked by uid 500); 6 Apr 2009 21:06:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91383 invoked by uid 500); 6 Apr 2009 21:06:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91373 invoked by uid 99); 6 Apr 2009 21:06:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Apr 2009 21:06:22 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [76.13.9.45] (HELO web65501.mail.ac4.yahoo.com) (76.13.9.45)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 06 Apr 2009 21:06:13 +0000
Received: (qmail 47829 invoked by uid 60001); 6 Apr 2009 21:05:52 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1239051951; bh=x7euSOU1KR6Ls6jEWGoPETDkU/EqCTd7x7QuvN7W+2Y=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=vNC8LkgunDAqIRco9GAY+xDWcSn5ngZo9icwYa4DSZVkdYjn+IbbOFq6fGrx4hjLfbkEBNnnRZ3yBPP3ePi7TL6Ube+2J+cBKqWQdLgfz7ByoIw84a6XNda7o54ASat7h9FmCjkr0AgPRd2LkdXHswwx4htAycvwJuF82+9XVic=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=30yaLiuoNCWrLuPB/ASG5rYml/wRseC7UPgdo9Nc+1xaySAwTtYSCxS2CB4CZ7Gom5Q49JD3vFSUNPe+6PkdoMwVRItCDR9j/Q6CUFUvIoiU59rRi4CCYRQvaTEzx8xqmxGLVgO31dNf5cHWP8fJbYacICuY4lpBQRGs+EIv0D0=;
Message-ID: <951605.46755.qm@web65501.mail.ac4.yahoo.com>
X-YMail-OSG: Hzs6qXsVM1k9PaeQIPlKin1DtASR8WJhmFSbqzTKIhYtVDmG5LBbAqFMRWXs6p38_2oxO0x02PLTE7YsedvjtQfzdKolyDuVhd2pMEDtNWR5PYEv8lAkMUWJItvDv33JbUh1fG3X07bXI36M7Uj4hzdgD0jcIi85.ua4LLtDZ9bPYgpxM0gH0FDDHm7ySNTtY.JXi.ACazsP1Q.nRKaHXXmugG87D6RdyMIyUVtyrYvwiV78t6wUCcLHCq1HbAnChPwwD9holwDCNbIft_p3oMYsjBs0WZz0tWAJpwZz6A2IsQ6mC9x3OGw-
Received: from [208.84.6.194] by web65501.mail.ac4.yahoo.com via HTTP; Mon, 06 Apr 2009 14:05:51 PDT
X-Mailer: YahooMailRC/1277.35 YahooMailWebService/0.7.289.1
References: <417838.23294.qm@web65509.mail.ac4.yahoo.com> <49DA6154.3040006@apache.org>
Date: Mon, 6 Apr 2009 14:05:51 -0700 (PDT)
From: Chad Walters <chad_walters@yahoo.com>
Subject: Re: [PROPOSAL] new subproject: Avro
To: general@hadoop.apache.org
In-Reply-To: <49DA6154.3040006@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org


Not to get too technical on you, but:
-- I don't see any message from Jeff on this topic

-- Sameer has either just left or is still in the process of leaving Yahoo
-- Dhruba's message was posted after my message was composed and while I was still having trouble getting my email to be recognized by the subscription system

Chad


----- Original Message ----
From: Doug Cutting <cutting@apache.org>
Cc: general@hadoop.apache.org
Sent: Monday, April 6, 2009 1:08:52 PM
Subject: Re: [PROPOSAL] new subproject: Avro

Chad Walters wrote:
> so far all of the +1 votes have come from Yahoo employees.

Not quite.  Jeff Hammerbacher is with Cloudera.  Sameer Paranjpye is with Mechanical Zoo.  Dhruba Borthakur is with Facebook.

Doug

From general-return-180-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 06 22:18:01 2009
Return-Path: <general-return-180-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59189 invoked from network); 6 Apr 2009 22:18:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Apr 2009 22:18:01 -0000
Received: (qmail 17159 invoked by uid 500); 6 Apr 2009 22:18:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17088 invoked by uid 500); 6 Apr 2009 22:18:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17078 invoked by uid 99); 6 Apr 2009 22:18:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Apr 2009 22:18:00 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cutting@gmail.com designates 209.85.198.226 as permitted sender)
Received: from [209.85.198.226] (HELO rv-out-0506.google.com) (209.85.198.226)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Apr 2009 22:17:51 +0000
Received: by rv-out-0506.google.com with SMTP id g37so1453855rvb.5
        for <general@hadoop.apache.org>; Mon, 06 Apr 2009 15:17:30 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:sender:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=kNuneQmyFMqpE4Ep+60gaXxEuPYZoFpbFI5yg/QK1mo=;
        b=AniYTR7ZS+Z0t3xf4z2Sgv1/nh95VjKIfcJIRDBsnooJhTiwaovMJe2FrXGs+8UgjS
         Pu5QULynqagSNTfPA9+bDAPnIBB0Hovpu1E7vbCw3YxruEQQtJUSTRyzIjsaVXJyfcLc
         92f6EL1ySxSRKSZf7SgVaT/unInKjnfJj0Cx0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:date:from:user-agent:mime-version:to:subject
         :references:in-reply-to:content-type:content-transfer-encoding;
        b=FPG6N4vqpDg5yDo7Y1LTEKHVEVHECPVnL2KWFHP0KAA10bvgHeThGEmSzA/vCFVSUh
         Yz867SXDL9nJyZ52Cg4sMxEm1x+jEjOj8Kkg1c7d326t9cigePun+bucmi0VKywfPJI4
         Texa9PCJdBZJg1EgAMHSkteAvSBteFDI4RDSs=
Received: by 10.142.12.14 with SMTP id 14mr1403968wfl.119.1239056250132;
        Mon, 06 Apr 2009 15:17:30 -0700 (PDT)
Received: from ?192.168.168.16? (c-76-103-155-128.hsd1.ca.comcast.net [76.103.155.128])
        by mx.google.com with ESMTPS id 24sm8158278wff.2.2009.04.06.15.17.28
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Mon, 06 Apr 2009 15:17:29 -0700 (PDT)
Sender: Doug Cutting <cutting@gmail.com>
Message-ID: <49DA7F77.4030107@apache.org>
Date: Mon, 06 Apr 2009 15:17:27 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.21 (X11/20090318)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [PROPOSAL] new subproject: Avro
References: <417838.23294.qm@web65509.mail.ac4.yahoo.com> <49DA6154.3040006@apache.org> <951605.46755.qm@web65501.mail.ac4.yahoo.com>
In-Reply-To: <951605.46755.qm@web65501.mail.ac4.yahoo.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Chad Walters wrote:
> Not to get too technical on you, but: -- I don't see any message from
> Jeff on this topic

Sorry, you're right.  I inadvertently counted a private response.  I'll 
let Jeff speak in public for himself from now on!

Doug

From general-return-181-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 07 00:18:39 2009
Return-Path: <general-return-181-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96281 invoked from network); 7 Apr 2009 00:18:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Apr 2009 00:18:39 -0000
Received: (qmail 55040 invoked by uid 500); 7 Apr 2009 00:18:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54964 invoked by uid 500); 7 Apr 2009 00:18:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54954 invoked by uid 99); 7 Apr 2009 00:18:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Apr 2009 00:18:38 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of kevin.clark@gmail.com designates 74.125.44.29 as permitted sender)
Received: from [74.125.44.29] (HELO yx-out-2324.google.com) (74.125.44.29)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Apr 2009 00:18:29 +0000
Received: by yx-out-2324.google.com with SMTP id 3so1495808yxj.29
        for <general@hadoop.apache.org>; Mon, 06 Apr 2009 17:17:58 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=2tKFzS/GnyDEhhNCO5K6GdZ4lKZppze5rnDfte0xidQ=;
        b=e9aFDGUyNgH63NlibqiN228ThW6t1jxwRg0FhQS1bSX3KKcnUwf5pH7afR+R1iqJjd
         OyPUSVSc4XDR1ikpdvcpyVRE8513xccH9TV3duQaO3gD25ZrDABIjURbhuJr00rPbc7K
         67N/X3hXv2vd00ov/yiixzYJjQrQeJ5WOmoeU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=SXXkBS519RfG7Ea82MF6N5JEn1NLXzP2R/c5T/FyR1avQI+hljetwqOOZCUHrvHBov
         QBoHIxRxlv7537oaovNOWT3fo5s6I3a4I2QawLh39I1eDocEf1Q1mToyfEj16T432wzQ
         0mzEUKPXYc0noiCYdUD+jZe1FR6sLETfox2Gs=
MIME-Version: 1.0
Received: by 10.151.78.15 with SMTP id f15mr9441564ybl.62.1239063477677; Mon, 
	06 Apr 2009 17:17:57 -0700 (PDT)
In-Reply-To: <49DA5420.4030208@apache.org>
References: <417838.23294.qm@web65509.mail.ac4.yahoo.com>
	 <49DA5420.4030208@apache.org>
Date: Mon, 6 Apr 2009 17:17:57 -0700
Message-ID: <f53e6950904061717u7671b297he1f6bb875a038277@mail.gmail.com>
Subject: Re: [PROPOSAL] new subproject: Avro
From: Kevin Clark <kevin.clark@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Doug,

On Mon, Apr 6, 2009 at 12:12 PM, Doug Cutting <cutting@apache.org> wrote:
> Chad Walters wrote:
>>
>> -- You suggest that there is not a lot in Thrift that Avro can
>> leverage. I think you may be overlooking the fact that Thrift has a
>> user base and a community of developers who are very interested in
>> issues of cross-language data serialization and interoperability.
>
> I meant that in terms of common code, not coders. =A0Coders can belong to=
 more
> than one community but code should generally not. =A0Hadoop Core has beco=
me a
> sprawling community that we're trying to split. =A0It's more productive t=
o
> have have more, small communities than few large ones. =A0A project needs=
 a
> handful of active developers, but too many and it becomes ungainly. =A0So=
, if
> it's technically possible for a codebase to be distinct, and it can attra=
ct
> enough active developers to sustain itself, that is a preferable structur=
e.

I agree with you in general, but cross language libraries require
larger communities than other projects. It's non-trivial to gather
groups of coders to support each language the project chooses to
include. Right now Thrift has some level of support for a dozen
languages. We've been really very active in the last several months,
and devs have come out of the woodwork to extend their favorite
language(s) binding(s). The overhead for those people (or some
equivalent group) to pay attention to another mailing list, another
bug tracker, another irc channel, and another community isn't trivial.
I understand that developing the code itself may be more convenient
for some, but I think that the community that supports the code is
what really counts. If we can share that, and still achieve our goals,
I think we'll be better off.

Of course, this assumes that one of the primary goals of Avro is to be
cross language. Is that the case, or have I misunderstood?


> Avro has unions and a null type, while Thrift does not. =A0Does Thrift su=
pport
> recursive data structures?

We don't support recursive data structures. We do, however, have a
ticket open where we're discussing union support (THRIFT-409).


In your post you talk about the problems associated with supporting
multiple serialization formats. One of the things I like about Thrift
is that even though Thrift supports many different things, application
developers aren't at all obligated to. In fact, I don't expect anyone
does. It would be perfectly reasonable for Hadoop to specify that they
use the Avro data format for transmissions, and the cross language
library to provide the API could be Thrift. I think you said something
similar in your post, but if not please do clarify.



On the "names vs field ids" issue:

I know that the Ruby and Java Thrift libraries provide name-based
access to this information, and know of no restriction that would keep
the others from doing the same. It's just a matter of a little code.


>> Consider an alternative: making Avro more like a sub-project of
>> Thrift or just implementing it directly in Thrift.
>
> I looked into changing Thrift to support Avro's features, and it was very
> messy. =A0Perhaps someone else could do this more easily.
>
> Building Avro as a part of Thrift would take considerably more effort for=
 me
> and I think offer little more than it does separately. =A0If you feel
> differently, you are free to fork Avro, start a competitor, provide patch=
es
> that integrate it into Thrift, or whatever.

I'd again like to appeal to you that it's the community that's harder
to develop than the code, and we've got one already. I also don't see
the implementation being especially difficult, but maybe we're looking
at different information. I'd be happy to talk with you about it if
you're open to the idea.

The goals of Avro seem to be consistent with the goals of each of
Thrift's contributors who have developed a new protocol. We can
already offer the things you've stated you don't want to develop, and
I think we've got a lot more to gain working together than we do
working separately.

That being said, I'm fairly confident we'll be providing an Avro
protocol on our own at some point if you're not interested in working
together. But I think if we go down that path we're doing a disservice
to users of both Thrift and Avro.

--=20
Kevin Clark
http://glu.ttono.us

From general-return-182-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 07 04:15:36 2009
Return-Path: <general-return-182-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68740 invoked from network); 7 Apr 2009 04:15:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Apr 2009 04:15:36 -0000
Received: (qmail 93125 invoked by uid 500); 7 Apr 2009 04:15:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93037 invoked by uid 500); 7 Apr 2009 04:15:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93027 invoked by uid 99); 7 Apr 2009 04:15:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Apr 2009 04:15:34 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cutting@gmail.com designates 209.85.198.238 as permitted sender)
Received: from [209.85.198.238] (HELO rv-out-0506.google.com) (209.85.198.238)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Apr 2009 04:15:25 +0000
Received: by rv-out-0506.google.com with SMTP id k40so2067914rvb.29
        for <general@hadoop.apache.org>; Mon, 06 Apr 2009 21:15:03 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:sender:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=5P8kAufLI82N0hLu1QYfEvNw0yflNVX0F0xyDIzfClE=;
        b=ZERc0HU32H6LBEloYk/buicSIIIiU0C9NYd6Y90eMXcGKCu4VNAp0tfwnlURidkVWv
         RdjKtVIPRUvacQn3pe8AWW55FaBotXvFg9qYXzzWbwLunOELgYmkAy5E3MjRggL7f9/0
         Wgm/2jzVz4KRYhwYkoYL1NQlS1Ko8EVW6TG/k=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:date:from:user-agent:mime-version:to:subject
         :references:in-reply-to:content-type:content-transfer-encoding;
        b=LjH9s4wdhHMYrhRMI+7MiSV/0UUkLO2g9ESahqG13KwlsTW1krF++iZF0F6+Iv4S92
         sPBibpulmQUR6tXroPFi2ySUYo3YGqAXZHwgnhtEwUhLenHMd8fwyfr43C6OQ1y4sLXv
         X9jK0NwMIvjhv3rXwZCsBIE7taHYEFm8JUvuI=
Received: by 10.143.2.19 with SMTP id e19mr1488764wfi.96.1239077703806;
        Mon, 06 Apr 2009 21:15:03 -0700 (PDT)
Received: from ?192.168.168.106? (c-76-103-155-128.hsd1.ca.comcast.net [76.103.155.128])
        by mx.google.com with ESMTPS id 30sm5828488wfa.38.2009.04.06.21.15.02
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Mon, 06 Apr 2009 21:15:03 -0700 (PDT)
Sender: Doug Cutting <cutting@gmail.com>
Message-ID: <49DAD345.9070801@apache.org>
Date: Mon, 06 Apr 2009 21:15:01 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.21 (X11/20090318)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [PROPOSAL] new subproject: Avro
References: <417838.23294.qm@web65509.mail.ac4.yahoo.com>	 <49DA5420.4030208@apache.org> <f53e6950904061717u7671b297he1f6bb875a038277@mail.gmail.com>
In-Reply-To: <f53e6950904061717u7671b297he1f6bb875a038277@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Kevin Clark wrote:
> The overhead for those people (or some
> equivalent group) to pay attention to another mailing list, another
> bug tracker, another irc channel, and another community isn't trivial.

Communities form around code, and, if Avro's code is largely disjoint 
from Thrift's, we should not assume that everyone in the Thrift 
community cares about Avro or vice versa.

> Of course, this assumes that one of the primary goals of Avro is to be
> cross language. Is that the case, or have I misunderstood?

Yes, that is a goal.

> It would be perfectly reasonable for Hadoop to specify that they
> use the Avro data format for transmissions, and the cross language
> library to provide the API could be Thrift. I think you said something
> similar in your post, but if not please do clarify.

Yes, perhaps this could be done.  I am not convinced that TProtocol is 
an ideal API for reading and writing Avro data, but it could perhaps be 
made to work reasonably well.

> That being said, I'm fairly confident we'll be providing an Avro
> protocol on our own at some point if you're not interested in working
> together. But I think if we go down that path we're doing a disservice
> to users of both Thrift and Avro.

I have never said I was not interested in working together.  I've said 
that I think Avro is fundamentally different from Thrift.  Avro is a 
specific format, Thrift is a generic API for various formats, none like 
Avro.  They might be made to work together.  But at this point I see no 
point in forcing them together.  If TProtocol's API is a good match for 
Avro's format and features, then it should be easy for folks to 
implement TProtocol using Avro's code and include Avro in Thrift.  If 
the match is not good then perhaps we can adjust Thrift and/or Avro to 
improve it.

Doug

From general-return-183-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 07 08:57:30 2009
Return-Path: <general-return-183-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88445 invoked from network); 7 Apr 2009 08:57:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Apr 2009 08:57:30 -0000
Received: (qmail 12824 invoked by uid 500); 7 Apr 2009 08:57:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12741 invoked by uid 500); 7 Apr 2009 08:57:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12731 invoked by uid 99); 7 Apr 2009 08:57:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Apr 2009 08:57:29 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [76.13.9.54] (HELO web65510.mail.ac4.yahoo.com) (76.13.9.54)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 07 Apr 2009 08:57:21 +0000
Received: (qmail 90468 invoked by uid 60001); 7 Apr 2009 08:56:59 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1239094619; bh=gYqu5Mivh89JtXzVs0G6i8TLiLe7hdceQfTJ1PVCLNs=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=mAftWTIhdmW7rBdGHGPmVYYBRI/0AA1eMKXCl5Heos4N/skMO35o9hvr0U7/FHMse7VUTD7jQjE1t6n82RpTVato3ZGliEVeXSl7P7+lrHmNZS+TPr1qZ84yYnj7ocTC9K/OtJN6OE5UmygbpUXKmY/kFrMh6nQJvyHrRIyQEF4=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=C711e1b5W3VoJYC3+rHGDugcSeBiqy3iPyzs0cB5h9M9g88mvWnTGVPEOXnF93te4kkCPxe2MXj25QMHS6oKr+QowaR1sm9p3Vqrjty5wB28KOOXophideeYzlRfPmYejJSoTwT/LMRaSqyja3rSmz7qBW7hiHkjfYyKptJVfGM=;
Message-ID: <771249.90372.qm@web65510.mail.ac4.yahoo.com>
X-YMail-OSG: 660pxz0VM1nXqVL_vWCTlN9V7K7gsFDGOIxEAuQJp3FUMq.DOW5LNxPYPyqabNN5bluoMo.rLNZsmwzJEZdZJaYINcPNisyM4m7BHfKebUixhlP8Wr4FQvhJRxxxDwihkuG4S7CWjNu81uSQ7FK2dABw.lUMsYez3a_2sW2R9fMdr2QBdOgihl.HAiJjq7Uz446WIX3Q0.XnDeaG2oZq_q9DfXkUuHoW99M1KTWgfoDP_r7JhE2qZgYH5GLcSVYVYwTWNfBGpGlVOgsY6.A_at0o9aGEkcBsr2lK5sxsc0nENon3DlwMNyM-
Received: from [75.0.178.207] by web65510.mail.ac4.yahoo.com via HTTP; Tue, 07 Apr 2009 01:56:59 PDT
X-Mailer: YahooMailRC/1277.35 YahooMailWebService/0.7.289.1
References: <417838.23294.qm@web65509.mail.ac4.yahoo.com>  <49DA5420.4030208@apache.org> <f53e6950904061717u7671b297he1f6bb875a038277@mail.gmail.com> <49DAD345.9070801@apache.org>
Date: Tue, 7 Apr 2009 01:56:59 -0700 (PDT)
From: Chad Walters <chad_walters@yahoo.com>
Subject: Re: [PROPOSAL] new subproject: Avro
To: general@hadoop.apache.org
In-Reply-To: <49DAD345.9070801@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org


Doug,

> I have never said I was not interested in working together.

That's great -- glad to hear that you are open to collaboration. My concern is that by making a separate (sub)project, however, it may be difficult for us to work together in practice, and in particular it may be difficult for Thrift to leverage Avro's source code.

> I've
said that I think Avro is fundamentally different from Thrift.
> Avro is
a specific format, Thrift is a generic API for various formats, none
like Avro
> They might be made to work together.  But at this point I
see no point in forcing
> them together.

I don't think that they are as far apart as you are making it sound with this statement. I do think, however, that it will be very difficult for them to work together properly if the goal of code reuse by Thrift is not an explicit goal of Avro. The easiest way I can come up with to guarantee this is simply to incorporate Avro's feature set into Thrift. If you have other mechanisms for doing this, I'd love to hear them.

> If TProtocol's API is a good
match for Avro's format and features,
> then it should be easy for folks
to implement TProtocol using Avro's code and
> include Avro in Thrift. 
If the match is not good then perhaps we can adjust Thrift
> and/or Avro
to improve it.

Absolutely. And right now, there are sufficient differences in the type system and other areas that do require some adjustments, likely some on both sides (although, as I said in my previous email, we need to account for the fact that Thrift has current users to support so backwards-compatibility will need to be a consideration).

> Communities form around code, and, if Avro's code is largely disjoint
> from Thrift's, we should not assume that everyone in the Thrift
> community cares about Avro or vice versa.

IMO communities form around shared goals and purposes. Code and designs are created to achieve those purposes; they are also malleable and can be bent to achieve new goals and purposes. If we can find common cause, then we form a common community.

You have some features that you want to satisfy for Hadoop's purposes: compact serialization of large files containing many records of identical structure; partial deserialization in support of projection; dynamic interpretation of object schemas; better/more efficient RPC -- all delivered across multiple languages. The first three are also use cases that are of interest to some portion of the Thrift community and the fourth is something that Thrift already provides.

Avro at this point is fairly nascent -- you have a design, some code, a couple of developers, and a target group of future users who seem very receptive to what you are working on. You do not have current users, however, and that should mean that you have some degree of flexibility to your design where it doesn't make a material difference to the use cases you are trying to solve.

If you are willing to make some modifications to that design and
code, the work on Avro could also work directly towards extending
Thrift's functionality. I am pretty certain that the Thrift community
would be willing to make some reasonable modifications and extensions
to Thrift to smooth the way for this as well.

I think that by working closely with the Thrift community directly in the Thrift code base, you will get several significant benefits. You will be able to directly leverage the transport and server implementations in Thrift today and any future work in this area is also beneficial. You will have a built-in set of developers and committers across many languages who are already familiar with issues in cross-language serialization (and I agree with Kevin that this is not as portable as you seem to think it is). You will be able to avoid writing lots of parts of an RPC framework in multiple languages that you would need to write to make Avro a stand-alone solution for Hadoop. You would have a significant role in shaping the direction of Thrift to make sure that it remains a strong solution for Hadoop.

> I've said that I think Avro is fundamentally different from Thrift.  Avro
> is a specific format, Thrift is a generic API for various formats, none like Avro.

It is clear to me that a slightly modified version of Avro's data format should fit just fine as a Thrift TProtocol implementation. Out of the box this would, of course, only provide for statically generated bindings, but this is enough to satisfy the first of the desired features I described above.

The second feature, partial deserialization, is a feature that I would like to see in Thrift for a variety of use cases, not just your projection use case -- for example, message routing where only a message header is deserialized to determine where to pass along an otherwise uninterpreted block of data. This feature is not tightly coupled to the Avro data format in any way. As you have stated, this is possible to do when you have the schema in hand. Note that he static bindings in Thrift are another way that the schema can be transmitted -- in fact, the whole schema could just be retrievable from the bindings directly and fed into whatever mechanism is availabe for dynamic interpretation. But we wouldn't have to go so far as that for field look up by name -- as Kevin pointed out, the Java and Ruby Thrift libraries already have mechanisms for sufficient introspection to accomplish the right kind of lookups, I believe, and the other libraries could be
 extended to do the same quite easily. So partial deserialization can be supported via either dynamic interpretation and/or via introspection features of the static bindings.

To support the second use case, dynamic schema interpretation, there is definitely significant new code to be written. Note that this code is essentially the same code wherever you are writing it. Whatever work you are doing in Avro to be able to dynamically interpret JSON IDL could just be directly implemented in Thrift -- we would just define a JSON version of the Thrift IDL which would look a lot like Avro's IDL. To help further with interoperability we could make the Thrift compiler generate the JSON IDL from the Thrift IDL as another output target.

The basic upshot of the above is that it is not that hard to see how Avro could be directly integrated into Thrift if you were willing to entertain that option and I believe that you would see significant benefits that would more than offset the impact to your own ease of development about which you expressed concerns.

To touch on a couple specific responses from your previous email to me:

>> If the field IDs are really so
>> objectionable, Thrift could allow them to be optional for purely
>> dynamic usages.
>
> Optional features increase compatibility complexity and are harder
> to maintain and test. A Thrift IDL without numbers would not provide
> versioning features to non-dynamic languages.

Let me rephrase my suggestion because I think I may not have put it across as clearly as I could have. I am proposing that the IDL would only allow for field IDs to be omitted in the case where the schema was being interpreted dynamically -- no static bindings could be generated from IDL without fully specified field IDs. So if you are only interested in dynamic interpretation, you never have to look at or even think about field IDs. Does that in any way alter your stance here?

> It could be a floor wax and a dessert topping!

Love the SNL reference, but I don't think it is really appropos. My vision for Thrft with Avro's features folded in as a unified framework for cross-language serialization, covering a variety of use cases, is not jamming two completely heterogeneous things together. I can easily see wanting to take structures represented in one serialization format from disk and send them out over RPC. Thrift provides the means to do this kind of thing seemlessly, with formats appropriate to both use cases, rather than selecting a format that is good for one use case and so-so for the other.

Chad



----- Original Message ----
From: Doug Cutting <cutting@apache.org>
To: general@hadoop.apache.org
Sent: Monday, April 6, 2009 9:15:01 PM
Subject: Re: [PROPOSAL] new subproject: Avro

Kevin Clark wrote:
> The overhead for those people (or some
> equivalent group) to pay attention to another mailing list, another
> bug tracker, another irc channel, and another community isn't trivial.

Communities form around code, and, if Avro's code is largely disjoint from Thrift's, we should not assume that everyone in the Thrift community cares about Avro or vice versa.

> Of course, this assumes that one of the primary goals of Avro is to be
> cross language. Is that the case, or have I misunderstood?

Yes, that is a goal.

> It would be perfectly reasonable for Hadoop to specify that they
> use the Avro data format for transmissions, and the cross language
> library to provide the API could be Thrift. I think you said something
> similar in your post, but if not please do clarify.

Yes, perhaps this could be done.  I am not convinced that TProtocol is an ideal API for reading and writing Avro data, but it could perhaps be made to work reasonably well.

> That being said, I'm fairly confident we'll be providing an Avro
> protocol on our own at some point if you're not interested in working
> together. But I think if we go down that path we're doing a disservice
> to users of both Thrift and Avro.

I have never said I was not interested in working together.  I've said that I think Avro is fundamentally different from Thrift.  Avro is a specific format, Thrift is a generic API for various formats, none like Avro.  They might be made to work together.  But at this point I see no point in forcing them together.  If TProtocol's API is a good match for Avro's format and features, then it should be easy for folks to implement TProtocol using Avro's code and include Avro in Thrift.  If the match is not good then perhaps we can adjust Thrift and/or Avro to improve it.

Doug


From general-return-184-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 07 16:17:13 2009
Return-Path: <general-return-184-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57578 invoked from network); 7 Apr 2009 16:17:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Apr 2009 16:17:13 -0000
Received: (qmail 60604 invoked by uid 500); 7 Apr 2009 16:17:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60546 invoked by uid 500); 7 Apr 2009 16:17:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60536 invoked by uid 99); 7 Apr 2009 16:17:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Apr 2009 16:17:12 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cutting@gmail.com designates 209.85.200.169 as permitted sender)
Received: from [209.85.200.169] (HELO wf-out-1314.google.com) (209.85.200.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Apr 2009 16:17:03 +0000
Received: by wf-out-1314.google.com with SMTP id 23so2633851wfg.2
        for <general@hadoop.apache.org>; Tue, 07 Apr 2009 09:16:42 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:sender:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=Zu0x/iB9s8j2nhAdOQvJ3Brd8/4jB1iFVLA6cKBDVGk=;
        b=AYDE/Us2hemkPQICNe6WYg5RWtLA8Mj9w2pDoiat+s+4SoTlcRezIjeEXPbVIIa1Y1
         Ld3cfarvGZPHIHXiA6Z8EH2hOtNK1N8lMSky8oIjtdrX7GXJIXazh0/eD/912e0pk+tj
         LuuaEuEgJ4LjHpaKmBVM/daEq+Jq0VW6KSNkM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:date:from:user-agent:mime-version:to:subject
         :references:in-reply-to:content-type:content-transfer-encoding;
        b=JGMT8Q5cYaahjaiN16UhHx+7dCLmk/WVRlUqW22FY/KEOJPu1QU2bijDwbrW4N8DNq
         6Tyctmg3kICwYOOAmuUVQjEnuwgw+96NGrjXkVWEZ/O8vtDWvMaL6IfYsaayWazs7ZDc
         SHtKnXPy/PqZYtG5docbPIkOA2vHQ6gT8KIRY=
Received: by 10.142.77.11 with SMTP id z11mr72299wfa.277.1239121002031;
        Tue, 07 Apr 2009 09:16:42 -0700 (PDT)
Received: from ?192.168.168.16? (c-76-103-155-128.hsd1.ca.comcast.net [76.103.155.128])
        by mx.google.com with ESMTPS id 22sm539299wfg.23.2009.04.07.09.16.40
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Tue, 07 Apr 2009 09:16:41 -0700 (PDT)
Sender: Doug Cutting <cutting@gmail.com>
Message-ID: <49DB7C67.3070405@apache.org>
Date: Tue, 07 Apr 2009 09:16:39 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.21 (X11/20090318)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [PROPOSAL] new subproject: Avro
References: <417838.23294.qm@web65509.mail.ac4.yahoo.com>  <49DA5420.4030208@apache.org> <f53e6950904061717u7671b297he1f6bb875a038277@mail.gmail.com> <49DAD345.9070801@apache.org> <771249.90372.qm@web65510.mail.ac4.yahoo.com>
In-Reply-To: <771249.90372.qm@web65510.mail.ac4.yahoo.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Chad Walters wrote:
> I do think, however, that it will be very
> difficult for them to work together properly if the goal of code
> reuse by Thrift is not an explicit goal of Avro.

Code reuse is an explicit goal of Avro.  It's an open source project 
with public APIs intended to expose all of its functionality.

> I think that by working closely with the Thrift community directly in
> the Thrift code base, you will get several significant benefits.

It's not like I did not consider this approach, evolving Thrift to 
better support my needs.  In fact, I considered it for months before 
abandoning it.  I am very familiar with these arguments.

I am starting a new serialization project fully aware of the hazards.  I 
feel that, on balance, it is considerably simpler for Avro to be 
developed separately and that this will not adversely affect its users 
or its developer community.  You may disagree.  As volunteers here, we 
are both free to do as we choose.

> To support the second use case, dynamic schema interpretation, there
> is definitely significant new code to be written. Note that this code
> is essentially the same code wherever you are writing it.

This is a primary case for Avro.  Without it, Avro's a non-starter. 
And, as you note, this is new code that must be written for each 
platform.  That's primarily what Avro is.  Fitting this code into Thrift 
would only make it more complicated.

> Whatever
> work you are doing in Avro to be able to dynamically interpret JSON
> IDL could just be directly implemented in Thrift -- we would just
> define a JSON version of the Thrift IDL which would look a lot like
> Avro's IDL. To help further with interoperability we could make the
> Thrift compiler generate the JSON IDL from the Thrift IDL as another
> output target.

Sure, we could bolt Avro's features onto the side of Thrift, but that 
doesn't make it easier for me to deliver Avro's features nor any easier 
for folks to use them.  And Thrift doesn't need a second IDL format.  It 
already suffers from too many formats.  I seek a single format, not a 
multitude.

> The basic upshot of the above is that it is not that hard to see how
> Avro could be directly integrated into Thrift if you were willing to
> entertain that option and I believe that you would see significant
> benefits that would more than offset the impact to your own ease of
> development about which you expressed concerns.

I am unlikely to implement it myself, as it does not address my needs.

> I am proposing that the IDL would
> only allow for field IDs to be omitted in the case where the schema
> was being interpreted dynamically -- no static bindings could be
> generated from IDL without fully specified field IDs. So if you are
> only interested in dynamic interpretation, you never have to look at
> or even think about field IDs. Does that in any way alter your stance
> here?

Not really.  It adds an "except on Tuesday" clause in the specification, 
which is not ideal.  In Avro we can generate static bindings without 
using field ids.

>> It could be a floor wax and a dessert topping!
> 
> Love the SNL reference, but I don't think it is really appropos. My
> vision for Thrft with Avro's features folded in as a unified
> framework for cross-language serialization, covering a variety of use
> cases, is not jamming two completely heterogeneous things together. I
> can easily see wanting to take structures represented in one
> serialization format from disk and send them out over RPC. Thrift
> provides the means to do this kind of thing seemlessly, with formats
> appropriate to both use cases, rather than selecting a format that is
> good for one use case and so-so for the other.

I believe that the cost of supporting multiple formats is too high.  We 
differ on that point.  I don't think one-stop-shopping is appropriate 
here, but prefer to provide an ala-carte format.

Doug

From general-return-185-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 07 17:27:04 2009
Return-Path: <general-return-185-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 90087 invoked from network); 7 Apr 2009 17:27:04 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Apr 2009 17:27:04 -0000
Received: (qmail 96757 invoked by uid 500); 7 Apr 2009 17:27:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96699 invoked by uid 500); 7 Apr 2009 17:27:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96689 invoked by uid 99); 7 Apr 2009 17:27:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Apr 2009 17:27:03 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of tom.e.white@gmail.com designates 209.85.221.134 as permitted sender)
Received: from [209.85.221.134] (HELO mail-qy0-f134.google.com) (209.85.221.134)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Apr 2009 17:26:55 +0000
Received: by qyk40 with SMTP id 40so6886215qyk.5
        for <general@hadoop.apache.org>; Tue, 07 Apr 2009 10:26:34 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=ND+9QuOAKNQMYfnqxV5Twa47nSlEQ8lvGcmtpVliAo4=;
        b=MA7hETHwnEk6z+jXieMqjwZGW8qsozMvLmE82v22zeCIwq9sHCh5kDU1tzUgL8kenX
         WGoslWhJ91/2H+XAS++9F5QOY6oanRE5oj2jP1pv6iBc1tgS2JjAapRXsn3IEBtKTmuE
         wXaMMeO77VT7s603ePKs+YKXg3yGmLYY2+EiI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=vOBCEMQmPy+zbFU8X7veXDuxzycjNsUHzpPSMwv/3rm+4qqIacOivpF6Mn3jHzAyK0
         wS4BAB+ZUGIsgBjGqxK0p3IAmJxUuBsq8KQOBpgJzekhongDjLYLGw35FGS/MTmfy9uJ
         8lyBAVwiXKhJaF59Pch1/6gKLjdVLItM5em6I=
MIME-Version: 1.0
Received: by 10.231.15.202 with SMTP id l10mr129193iba.35.1239125194279; Tue, 
	07 Apr 2009 10:26:34 -0700 (PDT)
In-Reply-To: <49D53694.1050906@apache.org>
References: <49D53694.1050906@apache.org>
Date: Tue, 7 Apr 2009 18:26:34 +0100
Message-ID: <b23e01b50904071026m49fe1672i3fd883de81bd1cbb@mail.gmail.com>
Subject: Re: [PROPOSAL] new subproject: Avro
From: Tom White <tom.e.white@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Thu, Apr 2, 2009 at 11:05 PM, Doug Cutting <cutting@apache.org> wrote:
> I propose we add a new Hadoop subproject for Avro, a serialization system=
.

+1

Tom

> =A0My ambition is for Avro to replace both Hadoop's RPC and to be used fo=
r
> most Hadoop data files, e.g., by Pig, Hive, etc.
>
> Initial committers would be Sharad Agarwal and me, both existing Hadoop
> committers. =A0We are the sole authors of this software to date.
>
> The code is currently at:
>
> http://people.apache.org/~cutting/avro.git/
>
> To learn more:
>
> git clone http://people.apache.org/~cutting/avro.git/ avro
> cat avro/README.txt
>
> Comments? =A0Questions?
>
> Doug
>

From general-return-186-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 07 20:08:40 2009
Return-Path: <general-return-186-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 62334 invoked from network); 7 Apr 2009 20:08:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Apr 2009 20:08:40 -0000
Received: (qmail 83023 invoked by uid 500); 7 Apr 2009 20:08:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82882 invoked by uid 500); 7 Apr 2009 20:08:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82858 invoked by uid 99); 7 Apr 2009 20:08:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Apr 2009 20:08:39 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [208.97.132.5] (HELO spunkymail-a14.g.dreamhost.com) (208.97.132.5)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Apr 2009 20:08:28 +0000
Received: from [10.0.0.3] (adsl-065-013-152-164.sip.rdu.bellsouth.net [65.13.152.164])
	by spunkymail-a14.g.dreamhost.com (Postfix) with ESMTP id DE257190E2C;
	Tue,  7 Apr 2009 13:08:05 -0700 (PDT)
Message-Id: <7EDF8CB8-388C-4A44-974E-6977E7AEB396@apache.org>
From: Grant Ingersoll <gsingers@apache.org>
To: mahout-user@lucene.apache.org, announce@apache.org,
	general@lucene.apache.org, general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=Apple-Mail-9-982163751
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: [ANNOUNCE] Apache Mahout 0.1 Released
Date: Tue, 7 Apr 2009 16:08:05 -0400
X-Mailer: Apple Mail (2.930.3)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-9-982163751
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit

The Apache Lucene project is pleased to announce the release of Apache  
Mahout 0.1.
Apache Mahout is a subproject of Apache Lucene with the goal of  
delivering scalable
machine learning algorithm implementations under the Apache license.   
The first public
release includes implementations for clustering, classification,
collaborative filtering and evolutionary programming.

Highlights include:
1. Taste Collaborative Filtering
2. Several distributed clustering implementations: k-Means, Fuzzy k- 
Means, Dirchlet, Mean-Shift and Canopy
3. Distributed Naive Bayes and Complementary Naive Bayes  
classification implementations
4. Distributed fitness function implementation for the Watchmaker  
evolutionary programming library
5.  Most implementations are built on top of Apache Hadoop (http://hadoop.apache.org 
) for scalability

The release contents have been pushed out to the main Apache release
site and the m2 ibiblio sync repository.

Apache Mahout 0.1 is the project's first release and is focused on  
establishing a baseline release while
attracting more contributors. Details can
be found in JIRA:

https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310751&styleName=Html&version=12312976

Apache Mahout is available in source form from the following download  
page:
http://www.apache.org/dyn/closer.cgi/lucene/mahout/0.1/mahout-0.1-project.tar.gz

Apache Mahout is also available for Maven 2 users via
the Central Maven Repositories:
http://repo1.maven.org/maven2/org/apache/mahout/
http://mirrors.ibiblio.org/pub/mirrors/maven2/org/apache/mahout/

When downloading from a mirror site, please remember to verify the  
downloads
using signatures found on the Apache site:
http://www.apache.org/dist/lucene/mahout/KEYS

For more information on Apache Mahout, visit the project home page:
http://lucene.apache.org/mahout
--Apple-Mail-9-982163751--

From general-return-187-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 08 03:34:05 2009
Return-Path: <general-return-187-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33583 invoked from network); 8 Apr 2009 03:34:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Apr 2009 03:34:05 -0000
Received: (qmail 70656 invoked by uid 500); 8 Apr 2009 03:34:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70566 invoked by uid 500); 8 Apr 2009 03:34:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70556 invoked by uid 99); 8 Apr 2009 03:34:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Apr 2009 03:34:03 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cutting@gmail.com designates 209.85.200.172 as permitted sender)
Received: from [209.85.200.172] (HELO wf-out-1314.google.com) (209.85.200.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Apr 2009 03:33:55 +0000
Received: by wf-out-1314.google.com with SMTP id 23so2875924wfg.2
        for <general@hadoop.apache.org>; Tue, 07 Apr 2009 20:33:35 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:sender:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=NggndcqBnrs7PKxDD35xchQ0ucCZWURMWs1F/ZjT7ho=;
        b=cN3rhe/qRHZLp6KRArIk7RQI4W3oNF5jSDimDT4pxIzZXFP7tHdlcd/8ZhvtEd0o5x
         Kye3uQ+x2SAjERGB4XNsrE2rpe3CPCKJWrGuerTLj1bkgO2lpGXooLVsvH3u4uKZhOPX
         pRmlfhCz5qv6f+phHmTpwg32OZGHr1f7+xKes=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:date:from:user-agent:mime-version:to:subject
         :references:in-reply-to:content-type:content-transfer-encoding;
        b=MYMc9XCsZnq7665Lhzzdg4AwGcNjGvAy9Tplf/DICYPWW6Gp42ywZuFhEiKHz1qmFX
         qyKEL5hfnloV4KxLnbt61l3QlfRBYQ1Odez3sH91d6cxtgejLDlpgFsVRmbtp1imVzTz
         yqsXsSvleb2sGU4yZ3mrmHekIm9iIoMz4g/qY=
Received: by 10.143.6.19 with SMTP id j19mr267875wfi.128.1239161615189;
        Tue, 07 Apr 2009 20:33:35 -0700 (PDT)
Received: from ?192.168.168.106? (c-76-103-155-128.hsd1.ca.comcast.net [76.103.155.128])
        by mx.google.com with ESMTPS id 30sm1737305wff.27.2009.04.07.20.33.33
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Tue, 07 Apr 2009 20:33:34 -0700 (PDT)
Sender: Doug Cutting <cutting@gmail.com>
Message-ID: <49DC1B0C.4000805@apache.org>
Date: Tue, 07 Apr 2009 20:33:32 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.21 (X11/20090318)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [PROPOSAL] new subproject: Avro
References: <49D53694.1050906@apache.org>
In-Reply-To: <49D53694.1050906@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

To be clear, since a few folks have missed this point: Avro is not 
complete.  At some point in the future, before people start using it as 
a format for persistent data, we'll need to stop altering its 
specification, or at least do so much more cautiously.  But before then, 
my immediate goal to move development from private to open so that we 
have a chance to incorporate feedback before we lock down the specification.

For example, several folks have raised the issue of compatibility with 
Thrift.  We certainly want to avoid gratuitous incompatibilities.  There 
are also features clearly missing from Avro that we expect to add before 
we make a release, like default values, a more efficient RPC handshake, 
etc.  And some features that we might consider removing, if they're not 
broadly useful and inhibit interoperability, like single-float, which 
isn't in Thrift, Python, etc.  And I expect there will be more such 
issues raised in the coming weeks and months.

But before we can discuss and resolve such issues we need a forum in 
which to do so.  That's all I am after at this point: mailing lists, a 
bug database, a public source code repository, etc., so that we can 
start accepting patches, adding committers, etc.

Three days have now passed since I initially proposed this, the nominal 
time for an Apache vote.  Is there anyone who strongly opposes taking 
the development of Avro public as a Hadoop subproject?  Only PMC votes 
are binding, but I would vastly prefer that the broader community also 
supports this step in the process.

Thanks,

Doug

Doug Cutting wrote:
> I propose we add a new Hadoop subproject for Avro, a serialization 
> system.  My ambition is for Avro to replace both Hadoop's RPC and to be 
> used for most Hadoop data files, e.g., by Pig, Hive, etc.
> 
> Initial committers would be Sharad Agarwal and me, both existing Hadoop 
> committers.  We are the sole authors of this software to date.
> 
> The code is currently at:
> 
> http://people.apache.org/~cutting/avro.git/
> 
> To learn more:
> 
> git clone http://people.apache.org/~cutting/avro.git/ avro
> cat avro/README.txt
> 
> Comments?  Questions?
> 
> Doug

From general-return-188-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 08 07:04:05 2009
Return-Path: <general-return-188-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27016 invoked from network); 8 Apr 2009 07:04:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Apr 2009 07:04:05 -0000
Received: (qmail 66434 invoked by uid 500); 8 Apr 2009 07:04:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66346 invoked by uid 500); 8 Apr 2009 07:04:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66336 invoked by uid 99); 8 Apr 2009 07:04:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Apr 2009 07:04:04 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [76.13.9.54] (HELO web65510.mail.ac4.yahoo.com) (76.13.9.54)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 08 Apr 2009 07:03:56 +0000
Received: (qmail 87956 invoked by uid 60001); 8 Apr 2009 07:03:34 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1239174214; bh=w35QaqTH0nG8qYndnWWgk38ujKnXmB6REZL3xm1ZX4I=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=duW0IJR/u8m9oeQ9cnNdc9K0vka6uxUf8KoWW4ssHmFm52S4W9IjsHVfRh52dlyUss3H9DSSbVNpr8WgjgBTkjs5m38ha6mJuP6xsvalOSID6g5afTJXrkWJ2FuG0RbNkFeUtqWqwT75DotyQLWc/rPiH5zAyLh6JaCRZbYZ4KE=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=JeT7JgQo1PQKl9JIM69XmAhUjJT4ubiBM79c0NN1P4jiBjsDjVbFPFVANBgdHOJTtyZ7lW88uJEihB1QxQa6qH8NTvAyshJvb37MGL0NpLrgP3uA4ONMBpXXRvqfQ9v42Nxq2rdCGJc0dcPe7OaGVEt1STNazMTaQ5NWKLa9Akg=;
Message-ID: <727470.75806.qm@web65510.mail.ac4.yahoo.com>
X-YMail-OSG: P9YkSAgVM1mvUtpVZxLoylItzO99imGhNd0ux0tq6lB1dI_C.p0.LCk3Y5wHeoBzZQKusnt1gzqMIEH5BRYAiFCX5gAT4LcxSSoZGBlVFdNWR8c._.mshB2tGJksTXVGC1wRkJcJJ1o5RH4xmCjruYOT5iqFFyyKYNUSIyIGms2li7h51gnc3OCYrsBMOA1.wPfr4OvQu2N9g11VKoaQLGH.sg_jbANnSYOOvhkPFezbJAOT1lZCR3v7jMNUFQeIO.yS1_gzAB_MXNEf_h99exQzLMZWu6QGmY5O0z3Ct49XM2IIMBsZugqTWqIQmj99okRPCj1TBBOwtmBZ9c8-
Received: from [75.0.178.207] by web65510.mail.ac4.yahoo.com via HTTP; Wed, 08 Apr 2009 00:03:33 PDT
X-Mailer: YahooMailRC/1277.35 YahooMailWebService/0.7.289.1
References: <49D53694.1050906@apache.org> <49DC1B0C.4000805@apache.org>
Date: Wed, 8 Apr 2009 00:03:33 -0700 (PDT)
From: Chad Walters <chad_walters@yahoo.com>
Subject: Re: [PROPOSAL] new subproject: Avro
To: general@hadoop.apache.org
In-Reply-To: <49DC1B0C.4000805@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org


Doug,

After our off-list chat, and given that you have indicated that the design is still in flux and that you are open to discussing changes that would permit interoperability, I am not as concerned as I was.

My urgency came from concern that once the design was put in place as part of an Apache subproject, rather than open sourced in some other less prominent forum, it would increase the barrier to interoperability; in particular, I was concerned that people would assume the design of the data format was fully-baked and start persisting large amounts of data in some early version of the format, potentially prematurely ossifying the design in a state unsuited for compatibility with Thrift. Given your clarifications around this, my fears clearly were not well-founded.

Please accept my apology if I came across as obstructionist. I was honestly advocating on behalf of what I believe is in the best interest our shared user base.

Clearly we have some disagreements about the value of some of Thrift's design choices and what those mean for various use cases. I think we also have some differences of opinion about the relative difficulty of implementation versus the value of interoperability. Hopefully, the next few months will afford an opportunity to examine the sources of those disagreements and see if they can be resolved.

Sincerely,

Chad



----- Original Message ----
From: Doug Cutting <cutting@apache.org>
To: general@hadoop.apache.org
Sent: Tuesday, April 7, 2009 8:33:32 PM
Subject: Re: [PROPOSAL] new subproject: Avro

To be clear, since a few folks have missed this point: Avro is not complete.  At some point in the future, before people start using it as a format for persistent data, we'll need to stop altering its specification, or at least do so much more cautiously.  But before then, my immediate goal to move development from private to open so that we have a chance to incorporate feedback before we lock down the specification.

For example, several folks have raised the issue of compatibility with Thrift.  We certainly want to avoid gratuitous incompatibilities.  There are also features clearly missing from Avro that we expect to add before we make a release, like default values, a more efficient RPC handshake, etc.  And some features that we might consider removing, if they're not broadly useful and inhibit interoperability, like single-float, which isn't in Thrift, Python, etc.  And I expect there will be more such issues raised in the coming weeks and months.

But before we can discuss and resolve such issues we need a forum in which to do so.  That's all I am after at this point: mailing lists, a bug database, a public source code repository, etc., so that we can start accepting patches, adding committers, etc.

Three days have now passed since I initially proposed this, the nominal time for an Apache vote.  Is there anyone who strongly opposes taking the development of Avro public as a Hadoop subproject?  Only PMC votes are binding, but I would vastly prefer that the broader community also supports this step in the process.

Thanks,

Doug

Doug Cutting wrote:
> I propose we add a new Hadoop subproject for Avro, a serialization system.  My ambition is for Avro to replace both Hadoop's RPC and to be used for most Hadoop data files, e.g., by Pig, Hive, etc.
> 
> Initial committers would be Sharad Agarwal and me, both existing Hadoop committers.  We are the sole authors of this software to date.
> 
> The code is currently at:
> 
> http://people.apache.org/~cutting/avro.git/
> 
> To learn more:
> 
> git clone http://people.apache.org/~cutting/avro.git/ avro
> cat avro/README.txt
> 
> Comments?  Questions?
> 
> Doug


From general-return-189-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 08 18:04:09 2009
Return-Path: <general-return-189-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 92436 invoked from network); 8 Apr 2009 18:04:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Apr 2009 18:04:09 -0000
Received: (qmail 79800 invoked by uid 500); 8 Apr 2009 18:04:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79731 invoked by uid 500); 8 Apr 2009 18:04:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 79721 invoked by uid 99); 8 Apr 2009 18:04:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Apr 2009 18:04:08 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cutting@gmail.com designates 209.85.198.238 as permitted sender)
Received: from [209.85.198.238] (HELO rv-out-0506.google.com) (209.85.198.238)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Apr 2009 18:03:59 +0000
Received: by rv-out-0506.google.com with SMTP id k40so202112rvb.29
        for <general@hadoop.apache.org>; Wed, 08 Apr 2009 11:03:38 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:sender:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=nnHohUKjYmz+ZH46AYPhGrWjU7R6JU7nd5QDjMkvXEU=;
        b=l8viWOBwrw9jAa5S0FFBkPSDazEbAQREkOE1A5PqoHTbHjcLd4+4RNzr/HV/PD5F7B
         NCIj2XfdxJNRd+tkCULYUcIlV3eseL/Dah3SYZEz6tfR0IaCr3J+/711sGzFlVRUvmnv
         stQ49mqnOi20qGMU0GSoCUUPdM7NuLLINnojQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:date:from:user-agent:mime-version:to:subject
         :references:in-reply-to:content-type:content-transfer-encoding;
        b=NGvtmzHvBK/+qnfHDzHUDt3v+xD/8xgW5w8px9l9vqq9BJzUYaELfESPAfeK2dr2//
         veJbDZJpW65iunZS5e8mXJ3nsRkPxTl6nN1ybLbE96fkgPacDDS/5lGmgVBDlZ1nVJGF
         3FMiLArg8exTwT4v0Qaq+y1yVl5t6yQ1NVCow=
Received: by 10.141.4.20 with SMTP id g20mr626692rvi.173.1239213818123;
        Wed, 08 Apr 2009 11:03:38 -0700 (PDT)
Received: from ?172.21.202.185? (nat-dip5.cfw-a-gci.corp.yahoo.com [209.131.62.114])
        by mx.google.com with ESMTPS id f21sm25929643rvb.5.2009.04.08.11.03.36
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Wed, 08 Apr 2009 11:03:36 -0700 (PDT)
Sender: Doug Cutting <cutting@gmail.com>
Message-ID: <49DCE6F7.9050803@apache.org>
Date: Wed, 08 Apr 2009 11:03:35 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.21 (X11/20090318)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: [RESULT] new subproject: Avro
References: <49D53694.1050906@apache.org> <49DC1B0C.4000805@apache.org>
In-Reply-To: <49DC1B0C.4000805@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Doug Cutting wrote:
> Three days have now passed since I initially proposed this, the nominal 
> time for an Apache vote.  Is there anyone who strongly opposes taking 
> the development of Avro public as a Hadoop subproject?

With 6 +1 votes from the PMC (Alan, Dhruba, Nigel, Owen, Tom & me) and 
no -1 votes, this has passed.  I will now ask infrastructure to create 
the mailing lists, and create the Jira instance, svn repo, website & 
wiki myself.

I'll send a message to this list once this is complete and the project 
is open for business.

Thanks,

Doug

From general-return-190-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 09 22:33:47 2009
Return-Path: <general-return-190-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 2967 invoked from network); 9 Apr 2009 22:33:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Apr 2009 22:33:47 -0000
Received: (qmail 26993 invoked by uid 500); 9 Apr 2009 22:33:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26909 invoked by uid 500); 9 Apr 2009 22:33:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26899 invoked by uid 99); 9 Apr 2009 22:33:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Apr 2009 22:33:46 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cutting@gmail.com designates 209.85.198.230 as permitted sender)
Received: from [209.85.198.230] (HELO rv-out-0506.google.com) (209.85.198.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Apr 2009 22:33:37 +0000
Received: by rv-out-0506.google.com with SMTP id k40so724306rvb.29
        for <general@hadoop.apache.org>; Thu, 09 Apr 2009 15:33:16 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:sender:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=9J824bGMEJey47JRRTKvbADsL+qZACZK6VC5k8AM99k=;
        b=LlW8y3vJqpI/Chv3oqlfekikf3xH4fSSDPbTK+b1sTKKMVPaZWOmM+nywdzlhUdFMr
         Lm2NPIw56sbvT2ZHApzz1rg9P9dhQiGtMasbUCU9pkY2eUn+HuweRW1iW3+mkzui00+A
         qjVrQvUK08RbK88SEsDc/U9F0zSJiZf3JQkWM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:date:from:user-agent:mime-version:to:subject
         :references:in-reply-to:content-type:content-transfer-encoding;
        b=pSAxjK3out5+P+SQtZJsq+x4AJBNaKpGdRIP1AA2HVoeFUniUuP4oNqLv4b3Xb1MsV
         ypKsSTCVm0mgOT0AdLCaO+rZ3XOU8ZOV5zWDp4aDGS/30PkFxZE27BNo2p1GW78DebZD
         lKTjpaYtyU+/vf9NC/7ENIiabgEr/bKa7l9hk=
Received: by 10.142.157.9 with SMTP id f9mr1029476wfe.341.1239316396061;
        Thu, 09 Apr 2009 15:33:16 -0700 (PDT)
Received: from ?192.168.168.16? (c-76-103-155-128.hsd1.ca.comcast.net [76.103.155.128])
        by mx.google.com with ESMTPS id 31sm1194825wff.15.2009.04.09.15.33.14
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Thu, 09 Apr 2009 15:33:15 -0700 (PDT)
Sender: Doug Cutting <cutting@gmail.com>
Message-ID: <49DE77A9.6030804@apache.org>
Date: Thu, 09 Apr 2009 15:33:13 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.21 (X11/20090318)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [RESULT] new subproject: Avro
References: <49D53694.1050906@apache.org> <49DC1B0C.4000805@apache.org> <49DCE6F7.9050803@apache.org>
In-Reply-To: <49DCE6F7.9050803@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

The mailing lists are now created.

If you are interested you may subscribe by sending messages to:
   avro-dev-subscribe at hadoop.apache.org
   avro-user-subscribe at hadoop.apache.org
   avro-commits-subscribe at hadoop.apache.org

Please join one or more of these lists if you are interested in Avro.  I 
do not intend to send more messages to general@ about Avro in the near 
future.

I've also uploaded the code to subversion, at:

   http://svn.apache.org/repos/asf/hadoop/avro/trunk/

A Jira bug database is at:

   https://issues.apache.org/jira/browse/AVRO

I will create a website and Wiki ASAP.

Doug

Doug Cutting wrote:
> Doug Cutting wrote:
>> Three days have now passed since I initially proposed this, the 
>> nominal time for an Apache vote.  Is there anyone who strongly opposes 
>> taking the development of Avro public as a Hadoop subproject?
> 
> With 6 +1 votes from the PMC (Alan, Dhruba, Nigel, Owen, Tom & me) and 
> no -1 votes, this has passed.  I will now ask infrastructure to create 
> the mailing lists, and create the Jira instance, svn repo, website & 
> wiki myself.
> 
> I'll send a message to this list once this is complete and the project 
> is open for business.
> 
> Thanks,
> 
> Doug

From general-return-191-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 11 00:29:14 2009
Return-Path: <general-return-191-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 26833 invoked from network); 11 Apr 2009 00:29:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Apr 2009 00:29:14 -0000
Received: (qmail 65431 invoked by uid 500); 11 Apr 2009 00:29:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65347 invoked by uid 500); 11 Apr 2009 00:29:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65337 invoked by uid 99); 11 Apr 2009 00:29:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Apr 2009 00:29:13 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of sethladd@gmail.com designates 209.85.198.237 as permitted sender)
Received: from [209.85.198.237] (HELO rv-out-0506.google.com) (209.85.198.237)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Apr 2009 00:29:04 +0000
Received: by rv-out-0506.google.com with SMTP id k40so1149182rvb.29
        for <general@hadoop.apache.org>; Fri, 10 Apr 2009 17:28:44 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type:content-transfer-encoding;
        bh=iBOJic3sSzLnw3dzY/CdW1GWn15lPhycPXZPss5tU2E=;
        b=d92n3njc8EeJsppkhhLVGzMTy6wfeY4MefeIyJtto7ITJkjvgPdrbxSvG7cMMgW6pG
         hVdCdV4Yxg/vmM++IAS0KClveDrVXWcCFtxG/bXtQe79bZKBPllACrGR4y1e88gPWf6s
         dE7NT3Tlr9jwVGH4XGg0+31VY5yx8pAoezUwU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=ZIyfaBTyToIfK2Mz1KgVxh8IxAn/TZ6vVhoXV4og+BMQdO8uDlt+lJYPrpASBx9eS9
         V7VbVCUaM9P977oi8jbCLLJ8TRmo0lyl38CDVYzNzfmVZSZLAxJn9fLDKBiPnBf7aANH
         H+rYjXrtty20ukNYgDTI+bNpSIsP7Wbh9if8w=
MIME-Version: 1.0
Received: by 10.142.58.20 with SMTP id g20mr1552667wfa.191.1239409724117; Fri, 
	10 Apr 2009 17:28:44 -0700 (PDT)
Date: Fri, 10 Apr 2009 14:28:44 -1000
Message-ID: <64f07e2c0904101728g7dfcfeeesc8fd7253b1f409c3@mail.gmail.com>
Subject: Error initializing attempt
From: Seth Ladd <sethladd@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hello,

I am using Hadoop 0.18.3 and Pig 0.2.0 across a small cluster of 4
machines.  I am using start-all.sh to boot up the cluster.  The HDFS
system appears to be working really well.  I am able to copy files in
and out of the filesystem.

However, when I try to submit a Pig job, I am greeted by this exception:

2009-04-10 14:17:27,145 INFO org.apache.hadoop.mapred.TaskTracker:
Starting tracker tracker_sdi-kenglish:localhost/127.0.0.1:37109
2009-04-10 14:17:27,249 INFO org.apache.hadoop.mapred.TaskTracker:
Starting thread: Map-events fetcher for all reduce tasks on
tracker_sdi-kenglish:localhost/127.0.0.1:37109
2009-04-10 14:18:07,361 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction: attempt_200904101417_0001_m_000002_0
2009-04-10 14:18:10,353 WARN org.apache.hadoop.mapred.TaskTracker:
Error initializing attempt_200904101417_0001_m_000002_0:
java.io.IOException: Call to sladd/208.67.216.132:8020 failed on local
exception: java.io.EOFException
	at org.apache.hadoop.ipc.Client.wrapException(Client.java:751)
	at org.apache.hadoop.ipc.Client.call(Client.java:719)
	at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
	at org.apache.hadoop.dfs.$Proxy5.getProtocolVersion(Unknown Source)
	at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:348)
	at org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:103)
	at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:172)
	at org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:67)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1339)
	at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1351)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:213)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:175)
	at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:638)
	at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1297)
	at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:937)
	at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:1334)
	at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:2343)
Caused by: java.io.EOFException
	at java.io.DataInputStream.readFully(DataInputStream.java:180)
	at java.io.DataInputStream.readFully(DataInputStream.java:152)
	at org.apache.hadoop.io.WritableUtils.readString(WritableUtils.java:115)
	at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:509)
	at org.apache.hadoop.ipc.Client$Connection.run(Client.java:442)

What's VERY odd about this, is I don't have a 208.67.216.132 node in
my system at all.  Where is hadoop getting this IP from?

My hadoop-site.xml is below:

<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://10.0.6.110/</value>
    <final>true</final>
  </property>
  <property>
    <name>mapred.job.tracker</name>
    <value>10.0.6.110:8012</value>
    <final>true</final>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/opt/cluster/hadoop-tmp</value>
    <final>true</final>
  </property>
</configuration>

I've confirmed SSH works just fine, ping works, etc.  HDFS works as well.

Does the above exception have any clues as to why I can't run a Pig
MapReduce job?

Your help or tips are much appreciated,
Seth

From general-return-192-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 11 00:52:15 2009
Return-Path: <general-return-192-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31685 invoked from network); 11 Apr 2009 00:52:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Apr 2009 00:52:15 -0000
Received: (qmail 75096 invoked by uid 500); 11 Apr 2009 00:52:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75029 invoked by uid 500); 11 Apr 2009 00:52:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75019 invoked by uid 99); 11 Apr 2009 00:52:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Apr 2009 00:52:14 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of sethladd@gmail.com designates 209.85.200.168 as permitted sender)
Received: from [209.85.200.168] (HELO wf-out-1314.google.com) (209.85.200.168)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Apr 2009 00:52:06 +0000
Received: by wf-out-1314.google.com with SMTP id 23so1126573wfg.2
        for <general@hadoop.apache.org>; Fri, 10 Apr 2009 17:51:45 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=f8FXU3q+BGVRzSzBlkHc7/64F1zicAve2IqPFxXPEhM=;
        b=j32gOrOElq27UIb3QdIxEKSrgpI9jpRozolxwufl1RZIUvOwI0q7Q3i9Hcs4vVmOCE
         o1IkRYepkH/H5RxeS/LQKEDo20IYyoeuGQU8B0CwYH/xPTQHvXNr+66Y0EiXDs0Ms0VA
         s+jxQYjSOiXQmoJXVHxqhEF7cASG8c2CJaQBc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=KlB7jd74j5ml6dpdGR9Ag4EO0BAfXnz5LJCXb5/Wf6Xn64Gg9EDj3VPYL7q8qMCCkU
         NXHnZaG5pdGzfWPn8eXmlmNd+eDWmGTP6FrfzDFi8HzQFAYnqgP0c1Ky51hCMv2e5tX0
         No8icpwe6TCi9xKEBCmoZGOLsCDcIBswp4Jso=
MIME-Version: 1.0
Received: by 10.143.29.17 with SMTP id g17mr1565061wfj.109.1239411105458; Fri, 
	10 Apr 2009 17:51:45 -0700 (PDT)
In-Reply-To: <64f07e2c0904101728g7dfcfeeesc8fd7253b1f409c3@mail.gmail.com>
References: <64f07e2c0904101728g7dfcfeeesc8fd7253b1f409c3@mail.gmail.com>
Date: Fri, 10 Apr 2009 14:51:45 -1000
Message-ID: <64f07e2c0904101751x658e3abdke40274eda037015e@mail.gmail.com>
Subject: Re: Error initializing attempt
From: Seth Ladd <sethladd@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

To answer my own question, I added explicit entries for my hosts in
the cluster to each /etc/hosts file.  Apparently, our DNS was mucking
up the name resolution (even though I had specified the IP address in
the config files)

Lession learned: ensure there's a solid entry for your master node in
each slave node's /etc/hosts file.

I hope this saves someone else next time it happens.

On Fri, Apr 10, 2009 at 2:28 PM, Seth Ladd <sethladd@gmail.com> wrote:
> Hello,
>
> I am using Hadoop 0.18.3 and Pig 0.2.0 across a small cluster of 4
> machines. =C2=A0I am using start-all.sh to boot up the cluster. =C2=A0The=
 HDFS
> system appears to be working really well. =C2=A0I am able to copy files i=
n
> and out of the filesystem.
>
> However, when I try to submit a Pig job, I am greeted by this exception:
>
> 2009-04-10 14:17:27,145 INFO org.apache.hadoop.mapred.TaskTracker:
> Starting tracker tracker_sdi-kenglish:localhost/127.0.0.1:37109
> 2009-04-10 14:17:27,249 INFO org.apache.hadoop.mapred.TaskTracker:
> Starting thread: Map-events fetcher for all reduce tasks on
> tracker_sdi-kenglish:localhost/127.0.0.1:37109
> 2009-04-10 14:18:07,361 INFO org.apache.hadoop.mapred.TaskTracker:
> LaunchTaskAction: attempt_200904101417_0001_m_000002_0
> 2009-04-10 14:18:10,353 WARN org.apache.hadoop.mapred.TaskTracker:
> Error initializing attempt_200904101417_0001_m_000002_0:
> java.io.IOException: Call to sladd/208.67.216.132:8020 failed on local
> exception: java.io.EOFException
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.ipc.Client.wrapException(=
Client.java:751)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.ipc.Client.call(Client.ja=
va:719)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.ipc.RPC$Invoker.invoke(RP=
C.java:216)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.dfs.$Proxy5.getProtocolVe=
rsion(Unknown Source)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.ipc.RPC.getProxy(RPC.java=
:348)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.dfs.DFSClient.createRPCNa=
menode(DFSClient.java:103)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.dfs.DFSClient.<init>(DFSC=
lient.java:172)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.dfs.DistributedFileSystem=
.initialize(DistributedFileSystem.java:67)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.fs.FileSystem.createFileS=
ystem(FileSystem.java:1339)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.fs.FileSystem.access$300(=
FileSystem.java:56)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.fs.FileSystem$Cache.get(F=
ileSystem.java:1351)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.fs.FileSystem.get(FileSys=
tem.java:213)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.fs.Path.getFileSystem(Pat=
h.java:175)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.mapred.TaskTracker.locali=
zeJob(TaskTracker.java:638)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.mapred.TaskTracker.startN=
ewTask(TaskTracker.java:1297)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.mapred.TaskTracker.offerS=
ervice(TaskTracker.java:937)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.mapred.TaskTracker.run(Ta=
skTracker.java:1334)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.mapred.TaskTracker.main(T=
askTracker.java:2343)
> Caused by: java.io.EOFException
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.io.DataInputStream.readFully(DataInput=
Stream.java:180)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.io.DataInputStream.readFully(DataInput=
Stream.java:152)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.io.WritableUtils.readStri=
ng(WritableUtils.java:115)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.ipc.Client$Connection.rec=
eiveResponse(Client.java:509)
> =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.hadoop.ipc.Client$Connection.run=
(Client.java:442)
>
> What's VERY odd about this, is I don't have a 208.67.216.132 node in
> my system at all. =C2=A0Where is hadoop getting this IP from?
>
> My hadoop-site.xml is below:
>
> <configuration>
> =C2=A0<property>
> =C2=A0 =C2=A0<name>fs.default.name</name>
> =C2=A0 =C2=A0<value>hdfs://10.0.6.110/</value>
> =C2=A0 =C2=A0<final>true</final>
> =C2=A0</property>
> =C2=A0<property>
> =C2=A0 =C2=A0<name>mapred.job.tracker</name>
> =C2=A0 =C2=A0<value>10.0.6.110:8012</value>
> =C2=A0 =C2=A0<final>true</final>
> =C2=A0</property>
> =C2=A0<property>
> =C2=A0 =C2=A0<name>hadoop.tmp.dir</name>
> =C2=A0 =C2=A0<value>/opt/cluster/hadoop-tmp</value>
> =C2=A0 =C2=A0<final>true</final>
> =C2=A0</property>
> </configuration>
>
> I've confirmed SSH works just fine, ping works, etc. =C2=A0HDFS works as =
well.
>
> Does the above exception have any clues as to why I can't run a Pig
> MapReduce job?
>
> Your help or tips are much appreciated,
> Seth
>

From general-return-193-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 13 17:02:19 2009
Return-Path: <general-return-193-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 51517 invoked from network); 13 Apr 2009 17:02:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Apr 2009 17:02:19 -0000
Received: (qmail 1133 invoked by uid 500); 13 Apr 2009 17:02:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98917 invoked by uid 500); 13 Apr 2009 17:01:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 87899 invoked by uid 99); 13 Apr 2009 11:28:05 -0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ankur.goel@corp.aol.com designates 64.12.143.146 as permitted sender)
Date: Mon, 13 Apr 2009 16:57:26 +0530 (IST)
From: Ankur Goel <ankur.goel@corp.aol.com>
To: general@hadoop.apache.org
Message-ID: <19580353.281239622042153.JavaMail.ankur@localhost.localdomain>
In-Reply-To: <10963932.261239621910111.JavaMail.ankur@localhost.localdomain>
Subject: Re: [PROPOSAL] new subproject: Avro
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-OriginalArrivalTime: 13 Apr 2009 11:27:31.0156 (UTC) FILETIME=[D564A140:01C9BC2A]
X-AOL-IP: 10.178.121.20
X-Virus-Checked: Checked by ClamAV on apache.org

How fast do we expect the new serialization system to be when it replaces existing serialization mechanism in Hadoop RPC?

A clear description of the existing bottlenecks and the performance goals for this system would help developers interested in contributing.

-Ankur


-------- Original Message --------
Subject: 	[PROPOSAL] new subproject: Avro
Date: 	Thu, 02 Apr 2009 15:05:08 -0700
From: 	Doug Cutting <cutting@apache.org>
Reply-To: 	general@hadoop.apache.org
To: 	general@hadoop.apache.org



I propose we add a new Hadoop subproject for Avro, a serialization 
system.  My ambition is for Avro to replace both Hadoop's RPC and to be 
used for most Hadoop data files, e.g., by Pig, Hive, etc.

Initial committers would be Sharad Agarwal and me, both existing Hadoop 
committers.  We are the sole authors of this software to date.

The code is currently at:

http://people.apache.org/~cutting/avro.git/

To learn more:

git clone http://people.apache.org/~cutting/avro.git/ avro
cat avro/README.txt

Comments?  Questions?

Doug



From general-return-194-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 13 17:54:54 2009
Return-Path: <general-return-194-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 86923 invoked from network); 13 Apr 2009 17:54:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Apr 2009 17:54:53 -0000
Received: (qmail 83337 invoked by uid 500); 13 Apr 2009 17:54:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83310 invoked by uid 500); 13 Apr 2009 17:54:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83300 invoked by uid 99); 13 Apr 2009 17:54:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Apr 2009 17:54:53 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cutting@gmail.com designates 209.85.217.164 as permitted sender)
Received: from [209.85.217.164] (HELO mail-gx0-f164.google.com) (209.85.217.164)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Apr 2009 17:54:44 +0000
Received: by gxk8 with SMTP id 8so4481759gxk.5
        for <general@hadoop.apache.org>; Mon, 13 Apr 2009 10:54:23 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:sender:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=4mayAX8Hz98Jau2OhXnLT4YJd0ilBN98HWcJxkR7Cg0=;
        b=C8++NyTvq5HHIaXaJlOxYHRRGW6By5+z+iiv6GMRsXBbbUh2VeNUlm9js52rTvsC4/
         2iEPkAu2NCXqNzejMHDLqkMTOTC54j/+/20A3lSWkKjM982eVcsV3nGXmYnnQJwq7Liu
         6Ir+RNFvnKXVjFNhIe22XKN7WxpTJvTFyranQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:date:from:user-agent:mime-version:to:subject
         :references:in-reply-to:content-type:content-transfer-encoding;
        b=vNQg6LRrLJ9sDHwoloAAlpYyJba73Sa9IGLV7hpJYbrXax2RHRdm4BSD9usYsTHcuk
         181qVKAfLeVq7/l5ti6Pv6toJaKd5n+e1H1o5k+8AXmwKMmiXiMYBkJBynASh+xtsMtY
         K/Mx9uLIwNBwmoxlmtLC4kPcQdGfdp0CQQ+w0=
Received: by 10.143.31.4 with SMTP id i4mr2712061wfj.102.1239645262405;
        Mon, 13 Apr 2009 10:54:22 -0700 (PDT)
Received: from ?192.168.168.16? (c-76-103-155-128.hsd1.ca.comcast.net [76.103.155.128])
        by mx.google.com with ESMTPS id 29sm18448746wfg.33.2009.04.13.10.54.20
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Mon, 13 Apr 2009 10:54:21 -0700 (PDT)
Sender: Doug Cutting <cutting@gmail.com>
Message-ID: <49E37C4A.4030904@apache.org>
Date: Mon, 13 Apr 2009 10:54:18 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.21 (X11/20090318)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [PROPOSAL] new subproject: Avro
References: <19580353.281239622042153.JavaMail.ankur@localhost.localdomain>
In-Reply-To: <19580353.281239622042153.JavaMail.ankur@localhost.localdomain>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Ankur Goel wrote:
> How fast do we expect the new serialization system to be when it
> replaces existing serialization mechanism in Hadoop RPC?

I hope that Avro will make its first release this summer.  Sometime soon 
after, I hope that we can start moving Hadoop Core's trunk RPC onto 
Avro.  We may start developing an experimental version of Hadoop Core 
that uses Avro in a branch before Avro is released.  This is all 
speculative, of course.  Any detailed discussion of Hadoop Core's future 
belongs on the core-dev@ and of Avro's future on avro-dev@.

> A clear description of the existing bottlenecks and the performance
> goals for this system would help developers interested in
> contributing.

Adding Avro to Hadoop Core is not primarily about performance but rather 
about compatibility and security.

Hadoop's existing RPC is not a performance bottleneck, nor is HDFS's 
data transfer protocol.  However, currently, Hadoop requires that 
clients and servers must run the exact same version of code, since the 
existing RPC is not tolerant of protocol changes.  We'd like to change 
that, so that one can run older clients against newer servers and vice 
versa.  Longer term, we'd also like to permit clients in languages other 
than Java.  We intend Avro to provide a change-tolerant, cross-platform 
RPC solution.

We'd also like Hadoop to become more secure.  Currently Hadoop uses 
three different communications mechanisms: RPC, HTTP (for shuffle) and a 
raw socket-based protocol for HDFS data transfers.  It would be best not 
to have to re-implement security features for each of these.  So we hope 
that we can make Avro perform well enough to replace not only Hadoop's 
RPC, but also HTTP in the shuffle and the HDFS data transfer protocol.

If you're interested in discussing Avro further, I encourage you to join 
the Avro mailing lists.

http://hadoop.apache.org/avro/mailing_lists.html

Doug

From general-return-195-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 22 21:53:59 2009
Return-Path: <general-return-195-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23123 invoked from network); 22 Apr 2009 21:53:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Apr 2009 21:53:59 -0000
Received: (qmail 89559 invoked by uid 500); 22 Apr 2009 21:53:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89126 invoked by uid 500); 22 Apr 2009 21:53:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88761 invoked by uid 99); 22 Apr 2009 21:53:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Apr 2009 21:53:56 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Apr 2009 21:53:46 +0000
Received: from wlan-snve-153-226.corp.yahoo.com (snvvpn1-10-72-72-c103.hq.corp.yahoo.com [10.72.72.103])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n3MLr3DL038643;
	Wed, 22 Apr 2009 14:53:03 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:content-type:
	content-transfer-encoding:mime-version:subject:date:x-mailer;
	b=FNEIB6lE71Sj4jtPY07gYcpw0jAD02Oc/YxzwSvfuaTHKvEYbzA3FjowZsexfdjD
Message-Id: <0D44727A-01E3-4305-9199-809A68361592@yahoo-inc.com>
From: Nigel Daley <ndaley@yahoo-inc.com>
To: general@hadoop.apache.org, core-user@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: [ANNOUNCE] Hadoop release 0.20.0 available
Date: Wed, 22 Apr 2009 14:53:03 -0700
X-Mailer: Apple Mail (2.930.3)
X-Virus-Checked: Checked by ClamAV on apache.org

Release 0.20.0 contains many improvements, new features, bug fixes and  
optimizations.

For Hadoop release details and downloads, visit:
http://hadoop.apache.org/core/releases.html

Hadoop 0.20.0 Release Notes are at
http://hadoop.apache.org/core/docs/r0.20.0/releasenotes.html

Thanks to all who contributed to this release!

Nigel


From general-return-196-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 25 00:28:16 2009
Return-Path: <general-return-196-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 26016 invoked from network); 25 Apr 2009 00:28:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Apr 2009 00:28:14 -0000
Received: (qmail 15377 invoked by uid 500); 25 Apr 2009 00:28:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15306 invoked by uid 500); 25 Apr 2009 00:28:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15296 invoked by uid 99); 25 Apr 2009 00:28:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Apr 2009 00:28:12 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ddbrodsky@gmail.com designates 209.85.219.171 as permitted sender)
Received: from [209.85.219.171] (HELO mail-ew0-f171.google.com) (209.85.219.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Apr 2009 00:28:03 +0000
Received: by ewy19 with SMTP id 19so1363047ewy.29
        for <general@hadoop.apache.org>; Fri, 24 Apr 2009 17:27:43 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:from:to
         :content-type:content-transfer-encoding:mime-version:subject:date
         :x-mailer;
        bh=/LqKsCUBq5U/lExdH2fWve9LaZhDMgUK8ZJ6L+tXzrk=;
        b=i2u0oMO9h0e+VFCmzD1ladk2c+oNlBEp30zkkUl9Jyq1pLlQ83mHPBGLix6d/NVddj
         dZ5aiUxDtWjXhUHVbb+Y1WXLWZE6EN8QuesDEDQgPosQL0iNtD7TwpZsGqLJYbE3iaaQ
         GsdpbDEzwSaYcuCTaUb35j8UOZxrm6hvlvYCs=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:from:to:content-type:content-transfer-encoding
         :mime-version:subject:date:x-mailer;
        b=LB/laiCK3VHmN2UqWgk339KokQiT+PiclTKLQlT7wAFZuntZrq+7I2uEaTi919aw7+
         vp6X8Q+lrLBPyxpYdzZjRoEhz57Iu/P4TZBBfNG/r/rjFsC1yuLDr+ntk7PX1NMXr2KL
         nUMlIWUIXTzDCW17GvqP/H1RZGUhsJ/XBW9jw=
Received: by 10.210.56.7 with SMTP id e7mr3014790eba.40.1240619263207;
        Fri, 24 Apr 2009 17:27:43 -0700 (PDT)
Received: from ?192.168.1.141? ([96.49.149.60])
        by mx.google.com with ESMTPS id 7sm2561761eyb.15.2009.04.24.17.27.41
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Fri, 24 Apr 2009 17:27:42 -0700 (PDT)
Message-Id: <B9EED8A9-C156-41ED-B1CD-381F1328715D@gmail.com>
From: Dmitry Brodsky <ddbrodsky@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: problems compiling libhdfs
Date: Fri, 24 Apr 2009 17:27:38 -0700
X-Mailer: Apple Mail (2.930.3)
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

I am trying to compile libhdfs on hadoop 0.20 and it's in a number of  
ways.  This is on Fedora Core 10 (64bit).  First the configure scripts  
are saying that fcntl.h is present but are not compileable.  Also, all  
the -m switches are generated without either a 32 or 64 bit designation.

Has anybody else has had trouble compiling libhdfs?

Thanks!
ttyl
Dima


From general-return-197-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 26 16:31:22 2009
Return-Path: <general-return-197-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66132 invoked from network); 26 Apr 2009 16:31:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Apr 2009 16:31:22 -0000
Received: (qmail 51759 invoked by uid 500); 26 Apr 2009 16:31:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51660 invoked by uid 500); 26 Apr 2009 16:31:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 36484 invoked by uid 99); 26 Apr 2009 16:01:42 -0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Message-ID: <376978.92484.qm@web306.biz.mail.mud.yahoo.com>
X-YMail-OSG: QXryDk0VM1lYxDJXidWKleQd_dOVFgRmP7VDQ8EwtHoOhgWNOOCX._9a4mmevbFux1ol1BbVq6iyXhDUHXi5XdddpYoRP8BKGy8XcxH21nk8bQgm9qORprNEHwhhcrn4_8OrUP4if0Jqm0V2O0viKXGI7TsLXgpwggE6eVEq..lnkaXkGHHQV.AIaE14WDV.JfqXf4KVd81fNS0_IuJFkygZ90ix6uuYMVthJAzLmy4GM1oCB4DXzeG8fV2iipOo8moYbsOEAHgUhM.SqqjLNSCvPjWY0UwIT4HKRu1pJDoJrEdcF.THUqQZirAWzmGst6ra5HdWtbtlfLmR6XWfug--
X-Mailer: YahooMailClassic/5.2.20 YahooMailWebService/0.7.289.1
Date: Sun, 26 Apr 2009 09:01:10 -0700 (PDT)
From: Bonesata <bonesata@bonesata.com>
Subject: (event) 5/19 How Hadoop Enables Big Data for Every Enterprise >> Christopher Bisciglia
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org


For registration and more information:
http://www.meetup.com/CIO-IT-Executives/calendar/10266376/

Do you have too much data? Do you find yourself discarding data due to performance and cost considerations? Have you heard about Hadoop, but find yourself wondering how it fits with the rest of your systems?

Come here the answers to these questions and more at Cloudera's office
on Tuesday May 19th.

This talk will provide a high level, mildly technical overview of Hadoop and existing architectures for working with large volumes of data. We'll pay special attention to how to augment your existing systems with Hadoop so each can focus on what they do best. Working together, they can deliver a more powerful, scalable, and robust solution.

Our speaker: Christophe Bisciglia, Founder, Cloudera

From general-return-198-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 28 10:47:37 2009
Return-Path: <general-return-198-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5181 invoked from network); 28 Apr 2009 10:47:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Apr 2009 10:47:37 -0000
Received: (qmail 47674 invoked by uid 500); 28 Apr 2009 10:47:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47203 invoked by uid 500); 28 Apr 2009 10:47:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46987 invoked by uid 99); 28 Apr 2009 10:47:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Apr 2009 10:47:33 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: unknown p4:64.236.200.131 (athena.apache.org: encountered unrecognized mechanism during SPF processing of domain of ankur.goel@corp.aol.com)
Received: from [205.188.249.133] (HELO omr-d35.mx.aol.com) (205.188.249.133)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Apr 2009 10:47:26 +0000
Received: from AOLDTCMEH01.ad.office.aol.com (aoldtcmeh01.office.aol.com [10.180.121.20]) by omr-d35.mx.aol.com (v117.7) with ESMTP id MAILOMRD353-7ecf49f6dea571; Tue, 28 Apr 2009 06:47:01 -0400
Received: from AOLMTCMEI01.ad.office.aol.com ([10.178.3.18]) by AOLDTCMEH01.ad.office.aol.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Tue, 28 Apr 2009 06:47:01 -0400
Received: from localhost ([10.178.3.10]) by AOLMTCMEI01.ad.office.aol.com over TLS secured channel with Microsoft SMTPSVC(6.0.3790.3959);
	 Tue, 28 Apr 2009 06:46:59 -0400
Date: Tue, 28 Apr 2009 16:16:51 +0530 (IST)
From: Ankur Goel <ankur.goel@corp.aol.com>
To: core-user@hadoop.apache.org, general@hadoop.apache.org
Message-ID: <14627742.451240915608268.JavaMail.ankur@localhost.localdomain>
In-Reply-To: <6416040.431240915475918.JavaMail.ankur@localhost.localdomain>
Subject: Hadoop / MySQL
MIME-Version: 1.0
Content-Type: multipart/alternative; 
	boundary="----=_Part_14_13944117.1240915608267"
X-OriginalArrivalTime: 28 Apr 2009 10:47:00.0409 (UTC) FILETIME=[A8C05290:01C9C7EE]
X-AOL-IP: 10.180.121.20
X-Virus-Checked: Checked by ClamAV on apache.org

------=_Part_14_13944117.1240915608267
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


hello hadoop users, 
Recently I had a chance to lead a team building a log-processing system that uses Hadoop and MySQL. The system's goal was to process the incoming information as quickly as possible (real time or near real time), and make it available for querying in MySQL. I thought it would be good to share the experience and the challenges with the community. Couldn't think of a better place than these mailing lists as I am not much of a blogger :-) 

The information flow in the system looks something like 

[Apache-Servers] -> [Hadoop] -> [MySQL-shards] -> [Query-Tools] 

Transferring from Apache-Servers to Hadoop was quite easy as we just had to organize the data in timely buckets (directories). Once that was running smooth we had to make sure that map-reduce jobs are fired at regular intervals and they pick up the right data. The jobs would then process/aggregate the date and dump the info into MySQL shards from the reducers [we have our own DB partioning set up]. This is where we hit major bottlenecks [any surprises? :-)] 

The table engine used was InnoDB as there was a need for fast replication and writes but only moderate reads (should eventually support high read rates). The data would take up quite a while to load completely far away from being near-real time. And so our optimization journey begin. 

1. We tried to optimize/tune InnoDB parameters like increasing the buffer pool size to 75 % of available RAM. This helped but only till the time DBs were lightly loaded i.e. innoDB had sufficient buffer pool to host the data and indexes. 

2. We also realized that InnoDB has considerable locking overhead because of which write concurrency is really bad when you have a large number of concurrent threads doing writes. The default thread concurrency for us was set to no_of_cpu * 2 = 8 which is what the official documentation advises as the optimal limit. So we limited the number of reduce tasks and consequently the number of concurrent writes and boy the performance improved 4x. We were almost there :-) 

3. Next thing we tried is the standard DB optimzation techniques like de-normalizing the schema and dropping constraints. This gave only a minor performance improvement, nothing earth shattering. Note that we were already caching connections in reducers to each MySQL shard and partionining logic was embedded into reducers. 

4. Falling still short of our performance objectives, we finally we decided to get rid of JDBC writes from reducers and work on an alternative that uses MySQLs LOAD utility. 
- The processing would partition the data into MySQL shard specific files resident in HDFS. 
- A script would then spawn processes via ssh on different physical machines to download this data. 
- Each spawned process just downloads the data for the shard it should upload to. 
- All the processes then start uploading data in parallel into their respective MySQL shards using LOAD DATA infile. 

This proved to be the fastest approach, even in the wake of increasing data loads. The enitre processing/loading would complete in less than 6 min. The system has been holding up quite well so far, even though we've had to limit the number of days for which we keep the data or else the MySQLs get overwhelmed. 

Hope this is helpful to people. 

Regards 
-Ankur 

------=_Part_14_13944117.1240915608267--

From general-return-199-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 28 14:21:32 2009
Return-Path: <general-return-199-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 3227 invoked from network); 28 Apr 2009 14:21:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Apr 2009 14:21:32 -0000
Received: (qmail 61882 invoked by uid 500); 28 Apr 2009 14:21:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61793 invoked by uid 500); 28 Apr 2009 14:21:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61772 invoked by uid 99); 28 Apr 2009 14:21:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Apr 2009 14:21:28 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of peter.skomoroch@gmail.com designates 209.85.217.215 as permitted sender)
Received: from [209.85.217.215] (HELO mail-gx0-f215.google.com) (209.85.217.215)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Apr 2009 14:21:17 +0000
Received: by gxk11 with SMTP id 11so1059170gxk.5
        for <multiple recipients>; Tue, 28 Apr 2009 07:20:56 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:cc:content-type;
        bh=R+EV03M2ZfQbEO5h8TwYQD280NkAtzKNHcj7Pll7lgk=;
        b=b/4u2EIMuYJiMTi54hvRt+1jdYZCPcjUuztba8eIprcWFIt9xiaN14AP0WIK5BD0KK
         wlg6v/nTsXBXTwzIscIOT36epGX5H02Ivv0wQCDWK3RsE6hVLybfT8+JB7hPWvNpUqrb
         YEb43JtylbFwiDOrKwINwE3reehgrTU1UVXWc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=QZiNEmDYod9wn8qJkJbxzz8ExkR5EfHLJACz/xaXWHFRuVS5BXW92tUIz4sOWr64wU
         Z8T6cXIAa5m6evBH+qkTLh7YMUPvhtFPG4haBfAt5bFGPGxVzrTN1S2/6pu90+vHGguS
         i335brfMFR2ZaKZgaSxmmnjUZhT+O/XsgfAIg=
MIME-Version: 1.0
Received: by 10.150.146.14 with SMTP id t14mr12354134ybd.85.1240928456036; 
	Tue, 28 Apr 2009 07:20:56 -0700 (PDT)
In-Reply-To: <14627742.451240915608268.JavaMail.ankur@localhost.localdomain>
References: <6416040.431240915475918.JavaMail.ankur@localhost.localdomain>
	 <14627742.451240915608268.JavaMail.ankur@localhost.localdomain>
Date: Tue, 28 Apr 2009 10:20:55 -0400
Message-ID: <e4fc0d2a0904280720j71985d36v338f944a7df7788@mail.gmail.com>
Subject: Re: Hadoop / MySQL
From: Peter Skomoroch <peter.skomoroch@gmail.com>
To: core-user@hadoop.apache.org
Cc: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd3beb2d0508204689e2b98
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd3beb2d0508204689e2b98
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Thanks for sharing sounds like a nice system - I always advise people to
avoid direct SQL inserts for batch jobs / large amounts of data and use
MySQL's optimized LOAD utility like you did.  Same goes for Oracle...
Nothing brings a DB server to its knees like a ton of individual inserts on
indexed tables..

On Tue, Apr 28, 2009 at 6:46 AM, Ankur Goel <ankur.goel@corp.aol.com> wrote:

>
> hello hadoop users,
> Recently I had a chance to lead a team building a log-processing system
> that uses Hadoop and MySQL. The system's goal was to process the incoming
> information as quickly as possible (real time or near real time), and make
> it available for querying in MySQL. I thought it would be good to share the
> experience and the challenges with the community. Couldn't think of a better
> place than these mailing lists as I am not much of a blogger :-)
>
> The information flow in the system looks something like
>
> [Apache-Servers] -> [Hadoop] -> [MySQL-shards] -> [Query-Tools]
>
> Transferring from Apache-Servers to Hadoop was quite easy as we just had to
> organize the data in timely buckets (directories). Once that was running
> smooth we had to make sure that map-reduce jobs are fired at regular
> intervals and they pick up the right data. The jobs would then
> process/aggregate the date and dump the info into MySQL shards from the
> reducers [we have our own DB partioning set up]. This is where we hit major
> bottlenecks [any surprises? :-)]
>
> The table engine used was InnoDB as there was a need for fast replication
> and writes but only moderate reads (should eventually support high read
> rates). The data would take up quite a while to load completely far away
> from being near-real time. And so our optimization journey begin.
>
> 1. We tried to optimize/tune InnoDB parameters like increasing the buffer
> pool size to 75 % of available RAM. This helped but only till the time DBs
> were lightly loaded i.e. innoDB had sufficient buffer pool to host the data
> and indexes.
>
> 2. We also realized that InnoDB has considerable locking overhead because
> of which write concurrency is really bad when you have a large number of
> concurrent threads doing writes. The default thread concurrency for us was
> set to no_of_cpu * 2 = 8 which is what the official documentation advises as
> the optimal limit. So we limited the number of reduce tasks and consequently
> the number of concurrent writes and boy the performance improved 4x. We were
> almost there :-)
>
> 3. Next thing we tried is the standard DB optimzation techniques like
> de-normalizing the schema and dropping constraints. This gave only a minor
> performance improvement, nothing earth shattering. Note that we were already
> caching connections in reducers to each MySQL shard and partionining logic
> was embedded into reducers.
>
> 4. Falling still short of our performance objectives, we finally we decided
> to get rid of JDBC writes from reducers and work on an alternative that uses
> MySQLs LOAD utility.
> - The processing would partition the data into MySQL shard specific files
> resident in HDFS.
> - A script would then spawn processes via ssh on different physical
> machines to download this data.
> - Each spawned process just downloads the data for the shard it should
> upload to.
> - All the processes then start uploading data in parallel into their
> respective MySQL shards using LOAD DATA infile.
>
> This proved to be the fastest approach, even in the wake of increasing data
> loads. The enitre processing/loading would complete in less than 6 min. The
> system has been holding up quite well so far, even though we've had to limit
> the number of days for which we keep the data or else the MySQLs get
> overwhelmed.
>
> Hope this is helpful to people.
>
> Regards
> -Ankur
>



-- 
Peter N. Skomoroch
617.285.8348
http://www.datawrangling.com
http://delicious.com/pskomoroch
http://twitter.com/peteskomoroch

--000e0cd3beb2d0508204689e2b98--

From general-return-200-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 28 16:29:52 2009
Return-Path: <general-return-200-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68072 invoked from network); 28 Apr 2009 16:29:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Apr 2009 16:29:52 -0000
Received: (qmail 10167 invoked by uid 500); 28 Apr 2009 16:29:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 10128 invoked by uid 500); 28 Apr 2009 16:29:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 39800 invoked by uid 99); 28 Apr 2009 11:47:15 -0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:cc:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=ai/2DDu1tCXIJaNuNmK/87TCHQ/OFf1gO7f6Zrt7lVtj3bdAnP2Xoz00h3QhmJkG
Message-ID: <49F6EC77.8040209@yahoo-inc.com>
Date: Tue, 28 Apr 2009 19:45:59 +0800
From: Yi-Kai Tsai <yikai@yahoo-inc.com>
User-Agent: Thunderbird 2.0.0.21 (Windows/20090302)
MIME-Version: 1.0
To: "core-user@hadoop.apache.org" <core-user@hadoop.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: Hadoop / MySQL
References: <14627742.451240915608268.JavaMail.ankur@localhost.localdomain>
In-Reply-To: <14627742.451240915608268.JavaMail.ankur@localhost.localdomain>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Ankur

Nice share , btw whats your query behavior ?
I'm asking because if the query is simple or could be 
transform/normalized , you could try output to HBase directly?

Yi-Kai

> hello hadoop users, 
> Recently I had a chance to lead a team building a log-processing system that uses Hadoop and MySQL. The system's goal was to process the incoming information as quickly as possible (real time or near real time), and make it available for querying in MySQL. I thought it would be good to share the experience and the challenges with the community. Couldn't think of a better place than these mailing lists as I am not much of a blogger :-) 
>
> The information flow in the system looks something like 
>
> [Apache-Servers] -> [Hadoop] -> [MySQL-shards] -> [Query-Tools] 
>
> Transferring from Apache-Servers to Hadoop was quite easy as we just had to organize the data in timely buckets (directories). Once that was running smooth we had to make sure that map-reduce jobs are fired at regular intervals and they pick up the right data. The jobs would then process/aggregate the date and dump the info into MySQL shards from the reducers [we have our own DB partioning set up]. This is where we hit major bottlenecks [any surprises? :-)] 
>
> The table engine used was InnoDB as there was a need for fast replication and writes but only moderate reads (should eventually support high read rates). The data would take up quite a while to load completely far away from being near-real time. And so our optimization journey begin. 
>
> 1. We tried to optimize/tune InnoDB parameters like increasing the buffer pool size to 75 % of available RAM. This helped but only till the time DBs were lightly loaded i.e. innoDB had sufficient buffer pool to host the data and indexes. 
>
> 2. We also realized that InnoDB has considerable locking overhead because of which write concurrency is really bad when you have a large number of concurrent threads doing writes. The default thread concurrency for us was set to no_of_cpu * 2 = 8 which is what the official documentation advises as the optimal limit. So we limited the number of reduce tasks and consequently the number of concurrent writes and boy the performance improved 4x. We were almost there :-) 
>
> 3. Next thing we tried is the standard DB optimzation techniques like de-normalizing the schema and dropping constraints. This gave only a minor performance improvement, nothing earth shattering. Note that we were already caching connections in reducers to each MySQL shard and partionining logic was embedded into reducers. 
>
> 4. Falling still short of our performance objectives, we finally we decided to get rid of JDBC writes from reducers and work on an alternative that uses MySQLs LOAD utility. 
> - The processing would partition the data into MySQL shard specific files resident in HDFS. 
> - A script would then spawn processes via ssh on different physical machines to download this data. 
> - Each spawned process just downloads the data for the shard it should upload to. 
> - All the processes then start uploading data in parallel into their respective MySQL shards using LOAD DATA infile. 
>
> This proved to be the fastest approach, even in the wake of increasing data loads. The enitre processing/loading would complete in less than 6 min. The system has been holding up quite well so far, even though we've had to limit the number of days for which we keep the data or else the MySQLs get overwhelmed. 
>
> Hope this is helpful to people. 
>
> Regards 
> -Ankur 
>   


-- 
Yi-Kai Tsai (cuma) <yikai@yahoo-inc.com>, Asia Search Engineering.


From general-return-201-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 28 18:45:15 2009
Return-Path: <general-return-201-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54400 invoked from network); 28 Apr 2009 18:45:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Apr 2009 18:45:14 -0000
Received: (qmail 22418 invoked by uid 500); 28 Apr 2009 18:45:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22348 invoked by uid 500); 28 Apr 2009 18:45:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22330 invoked by uid 99); 28 Apr 2009 18:45:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Apr 2009 18:45:13 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of p0941p@gmail.com designates 209.85.217.167 as permitted sender)
Received: from [209.85.217.167] (HELO mail-gx0-f167.google.com) (209.85.217.167)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Apr 2009 18:45:07 +0000
Received: by gxk11 with SMTP id 11so1363922gxk.5
        for <multiple recipients>; Tue, 28 Apr 2009 11:44:46 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=hwrkGfIYj89FUz2DtXwcW0Zi1+Xn2m77kGknkBaata8=;
        b=K74eXcIlYtr+/WffmulQnbpsnAIiOq9Uet9gh61m+eIkEcP5H+lyGPgSNQpszJBVSi
         U6Y73fhXGZiirmlFlzyj8Aqo21vSwEmxQAMzO8oaKROZDv8wh1j6L3lOkiNzeB46BvOh
         jgm1oRsuY0NjK9IPrxBHMO4pLT4kXnJ2gt9EE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=lSbybRieHjFTrdvKWfmV7GE+HoD3zlkPQ7NzcycZVBJ8mV6XnlJildFp5rDokB9rdY
         MwkEr0bXdQxealiVlDBQq/aH3KO6ERr3USr+IKilefgA83zgt7K/BmQMfifALiica4ob
         qK9yvlZv1EmTBoeYph5A7Kk4p2ZtVh/+Uxz18=
MIME-Version: 1.0
Received: by 10.151.73.3 with SMTP id a3mr11472903ybl.126.1240944286121; Tue, 
	28 Apr 2009 11:44:46 -0700 (PDT)
Date: Tue, 28 Apr 2009 11:44:46 -0700
Message-ID: <9c39bdeb0904281144n7267a25ap6526e537fe52b1@mail.gmail.com>
Subject: programming java ee and hadoop at the same time
From: George Pang <p0941p@gmail.com>
To: core-user@hadoop.apache.org, general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001e680f1a4c5c3f0d0468a1db40
X-Virus-Checked: Checked by ClamAV on apache.org

--001e680f1a4c5c3f0d0468a1db40
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hello users,
I am trying to program a hadoop powered web application with Eclipse as IDE.
 Now I have both hadoop perspective and java ee perspective.  I wonder if I
can have these two together, that I can use the mapper and producers in my
servlet.  Anyone has experience about this?  I will appreciate it.

George

--001e680f1a4c5c3f0d0468a1db40--

From general-return-202-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 30 21:59:16 2009
Return-Path: <general-return-202-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21206 invoked from network); 30 Apr 2009 21:59:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Apr 2009 21:59:16 -0000
Received: (qmail 38194 invoked by uid 500); 30 Apr 2009 21:32:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38140 invoked by uid 500); 30 Apr 2009 21:32:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 38130 invoked by uid 99); 30 Apr 2009 21:32:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Apr 2009 21:32:01 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of msa@schor.com designates 70.85.130.16 as permitted sender)
Received: from [70.85.130.16] (HELO gateway14.websitewelcome.com) (70.85.130.16)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 30 Apr 2009 21:31:50 +0000
Received: (qmail 24578 invoked from network); 30 Apr 2009 21:35:51 -0000
Received: from gator74.hostgator.com (67.18.27.130)
  by gateway14.websitewelcome.com with SMTP; 30 Apr 2009 21:35:51 -0000
Received: from [129.34.20.19] (port=40096 helo=[9.2.34.89])
	by gator74.hostgator.com with esmtpsa (TLSv1:AES256-SHA:256)
	(Exim 4.69)
	(envelope-from <msa@schor.com>)
	id 1Lzdqn-0003qu-RQ
	for general@hadoop.apache.org; Thu, 30 Apr 2009 16:31:26 -0500
Message-ID: <49FA18B1.8090701@schor.com>
Date: Thu, 30 Apr 2009 17:31:29 -0400
From: Marshall Schor <msa@schor.com>
User-Agent: Thunderbird 2.0.0.21 (Windows/20090302)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Implementing compareTo in user-written keys where one extends the
 other is error prone
X-Enigmail-Version: 0.95.7
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - gator74.hostgator.com
X-AntiAbuse: Original Domain - hadoop.apache.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - schor.com
X-Virus-Checked: Checked by ClamAV on apache.org

Hi.  It took me a good part of a day to figure out what was going wrong,
so I'm sharing this in hopes of learning something from the community or
getting hadoop improved to avoid this kind of error for future users.

I have 2 key classes, one holds a String, the other one extends that,
and adds a boolean.

I implemented the first key class (let's call it Super)

public class Super implements WritableComparable<Super> {
 . . .
  public int compareTo(Super o) {
    // sort on string value
    . . .
  }

I implemented the 2nd key class (let's call it Sub)

public class Sub extends Super {
 . . .
  public int compareTo(Sub o) {
    // sort on boolean value
    . . .
    // if equal, use the super:
    ... else
     return super.compareTo(o);
  }


With this setup, I used the "Sub" class as a mapper output key, and
expected the sort on the boolean value to happen first, then for equal
values there, the sort on the string values.

What actually happened, was that the sort on the boolean value was
skipped completely, and only the sort on the string was done.

The reason for this is that (in 0.19.1 release) the WritableCompator
instance that is created (using the defaults - no custom Comparator)
knows the class is "Sub", and calls from the key value it created, and
calls the compareTo method, passing it the other key.  Both of these
keys are of type Sub.  However, they are passed via this code in
WritableComparator:

 public int compare(WritableComparable a, WritableComparable b) {
    return a.compareTo(b);
  }

Java uses the interface spec for WritableComparable that was declared,
in this case WritableComparable<Super>, and infers that the arg type for
the compareTo is Super.  So it "skips" calling the compareTo in Sub, and
just calls the one in Super.

The workaround is to change the signature of Sub's compareTo method to
match the spec in the interface, namely it has to take the Super as an
argument, and then cast it to Sub.

This seems like a very error prone design.  Am I doing something wrong,
or can this be improved so that this kind of error is avoided?

-Marshall Schor

From general-return-203-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 30 22:22:56 2009
Return-Path: <general-return-203-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 26112 invoked from network); 30 Apr 2009 22:22:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Apr 2009 22:22:55 -0000
Received: (qmail 93554 invoked by uid 500); 30 Apr 2009 22:22:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93468 invoked by uid 500); 30 Apr 2009 22:22:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93458 invoked by uid 99); 30 Apr 2009 22:22:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Apr 2009 22:22:55 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of msa@schor.com designates 69.56.142.19 as permitted sender)
Received: from [69.56.142.19] (HELO gateway01.websitewelcome.com) (69.56.142.19)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 30 Apr 2009 22:22:45 +0000
Received: (qmail 31211 invoked from network); 30 Apr 2009 22:24:03 -0000
Received: from gator74.hostgator.com (67.18.27.130)
  by gateway01.websitewelcome.com with SMTP; 30 Apr 2009 22:24:03 -0000
Received: from [129.34.20.19] (port=16946 helo=[9.2.34.89])
	by gator74.hostgator.com with esmtpsa (TLSv1:AES256-SHA:256)
	(Exim 4.69)
	(envelope-from <msa@schor.com>)
	id 1Lzee5-0002N1-Jq
	for general@hadoop.apache.org; Thu, 30 Apr 2009 17:22:21 -0500
Message-ID: <49FA249E.3030106@schor.com>
Date: Thu, 30 Apr 2009 18:22:22 -0400
From: Marshall Schor <msa@schor.com>
User-Agent: Thunderbird 2.0.0.21 (Windows/20090302)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: classpath for finding Key classes
X-Enigmail-Version: 0.95.7
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - gator74.hostgator.com
X-AntiAbuse: Original Domain - hadoop.apache.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - schor.com
X-Virus-Checked: Checked by ClamAV on apache.org

In hadoop, one can define the classes to be used for Keys and Values.  I
do this.  When I make my giant Jar file holding everything needed for
running my application, I include these classes.

However, I've discovered that that is not enough it seems (in 0.19.1
version - in case that matters :-) ).  The job start up processes is
reading the configuration and finding the names of my Key classes, and
tries to load them.  But it is not using the giant Jar for my job,
(yet), so it doesn't find them.

A work-around that I've found is to include my giant Jar as the argument
to -libjars - that seems to get the class path set up so the startup /
validation code can find my classes.  This seems wasteful - having the
giant jar in two places...

Is there a best practices way to do this that's better than this?

Thanks. -Marshall Schor

From general-return-32-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Sep 05 21:38:16 2008
Return-Path: <general-return-32-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 62619 invoked from network); 5 Sep 2008 21:38:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 5 Sep 2008 21:38:15 -0000
Received: (qmail 74971 invoked by uid 500); 5 Sep 2008 21:38:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74948 invoked by uid 500); 5 Sep 2008 21:38:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 69968 invoked by uid 99); 5 Sep 2008 21:34:41 -0000
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=RCVD_IN_BL_SPAMCOP_NET,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of owen.omalley@gmail.com designates 209.85.198.227 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:from:to
         :content-type:content-transfer-encoding:mime-version:subject:date
         :x-mailer:sender;
        bh=FYA58+Wd1LqYUjreKqfSqjdu/veVhy4O7x06s7DSC9Q=;
        b=mLRch1lQwf11J+90Yb2GPuPLw2LdZS9euEus2VN3BFWbyM1scCU98Ipb/RcKhdqPkP
         +duJXVpT7LbKcgs2SnHaLFtvf2bni+Nn7OD/W+wRC0fcVddqQO2x+FZ5ZVFw7DySj8ha
         aKJrNKsWgYDmawRBFov2OdlKbQI7pjFdvdkvA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:from:to:content-type:content-transfer-encoding
         :mime-version:subject:date:x-mailer:sender;
        b=plEU8+f4M8APcbaY3wtXt72mO4ero51HrxkD1QGYSDaixqXU7On2aVr10GN24us68c
         NLrB5Z/Ept65N96IHay1CfNfteFFIt75NiXO3rOOJ99W3feXsKy+8f/1M0bJHFkAVkau
         Wh4zGuf3VyFNZyB9IvMHqHPwwhWn0Qkt9aoHw=
Message-Id: <B40A8D12-6B19-415F-8FFE-50FF2BD362F2@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v926)
Subject: [VOTE] Changing email notifications
Date: Fri, 5 Sep 2008 14:34:01 -0700
X-Mailer: Apple Mail (2.926)
Sender: Owen O'Malley <owen.omalley@gmail.com>
X-Virus-Checked: Checked by ClamAV on apache.org

All, core-dev is pretty overwhelming. I'd like to change the jira  
notifications so that:

jira created -> core-dev
other jira traffic -> a new list core-jira@hadoop.apache.org.

Most of us have filters like that anyways, and it would make it a lot  
less intimidating for developers getting started. I'm obviously +1.  
Thoughts?

-- Owen

From general-return-33-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Sep 05 21:48:20 2008
Return-Path: <general-return-33-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 66015 invoked from network); 5 Sep 2008 21:48:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 5 Sep 2008 21:48:20 -0000
Received: (qmail 88103 invoked by uid 500); 5 Sep 2008 21:48:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88085 invoked by uid 500); 5 Sep 2008 21:48:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88070 invoked by uid 99); 5 Sep 2008 21:48:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Sep 2008 14:48:13 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 72.14.220.155 as permitted sender)
Received: from [72.14.220.155] (HELO fg-out-1718.google.com) (72.14.220.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Sep 2008 21:47:15 +0000
Received: by fg-out-1718.google.com with SMTP id l26so819530fgb.35
        for <general@hadoop.apache.org>; Fri, 05 Sep 2008 14:47:43 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:to
         :subject:in-reply-to:mime-version:content-type
         :content-transfer-encoding:content-disposition:references;
        bh=q86MPN001ag/mxak0oYdGDLi7G8VdbYml6OvwbfrQUw=;
        b=vGZU9MIhaL7VODevnJ7W4wmtaOySUP4LceRFFUbtqtk2q2BeUgndzkO9fD3ic/pX6s
         Dry5M0FR4ay23+svA6045Etdt1nei0FU0oBXPGfkMhlKIQ33bDZF7/+EqGrkMrEC7EPa
         p4tdHbSwR4g6a0awJxPTJHlZdaYYkfQC+YR3U=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version
         :content-type:content-transfer-encoding:content-disposition
         :references;
        b=EMY+0EdSMA25LKh8Kzlv9TlS+y7wYE4AIHVkah4QEbdJUbQZ+X8dcwIlnsPk8I/nN9
         jydkpdrcGoBVxNComhaFXqodmpM6xOhO2FLRyAdJxOI6x+Xi6jR44MyAfFjlzcq2AzwF
         Mnrvh3E0RFMcJiL3smTnJ2lBausfzfwTKFbnE=
Received: by 10.187.175.6 with SMTP id c6mr2607834fap.89.1220651263508;
        Fri, 05 Sep 2008 14:47:43 -0700 (PDT)
Received: by 10.187.235.3 with HTTP; Fri, 5 Sep 2008 14:47:43 -0700 (PDT)
Message-ID: <4aa34eb70809051447x1cd04446pbee68b55764edd0c@mail.gmail.com>
Date: Fri, 5 Sep 2008 14:47:43 -0700
From: "Dhruba Borthakur" <dhruba@gmail.com>
To: general@hadoop.apache.org
Subject: Re: [VOTE] Changing email notifications
In-Reply-To: <B40A8D12-6B19-415F-8FFE-50FF2BD362F2@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <B40A8D12-6B19-415F-8FFE-50FF2BD362F2@apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

I like this idea. As long as I listen to core-dev, I can watch JIRAs
that I am interested in. All other jira traffic that goes to
core-jira@hadoop.apache.org I can safely ignore.

+1.

dhruba

On Fri, Sep 5, 2008 at 2:34 PM, Owen O'Malley <omalley@apache.org> wrote:
> All, core-dev is pretty overwhelming. I'd like to change the jira
> notifications so that:
>
> jira created -> core-dev
> other jira traffic -> a new list core-jira@hadoop.apache.org.
>
> Most of us have filters like that anyways, and it would make it a lot less
> intimidating for developers getting started. I'm obviously +1. Thoughts?
>
> -- Owen
>

From general-return-34-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Sep 05 21:51:02 2008
Return-Path: <general-return-34-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 67436 invoked from network); 5 Sep 2008 21:51:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 5 Sep 2008 21:51:02 -0000
Received: (qmail 93876 invoked by uid 500); 5 Sep 2008 21:51:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93854 invoked by uid 500); 5 Sep 2008 21:51:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93843 invoked by uid 99); 5 Sep 2008 21:51:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Sep 2008 14:51:00 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cutting@gmail.com designates 209.85.200.170 as permitted sender)
Received: from [209.85.200.170] (HELO wf-out-1314.google.com) (209.85.200.170)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Sep 2008 21:50:01 +0000
Received: by wf-out-1314.google.com with SMTP id 24so635256wfg.2
        for <general@hadoop.apache.org>; Fri, 05 Sep 2008 14:50:22 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding:sender;
        bh=1L88Gup4bNXNZGuMS4sU8sBLDmmKBNyvVwATC3Pd5hg=;
        b=G6qMLOmKLRt5kUtFmGP7r2vmOL9Olb5TpKnsJp/v7iuhm4+MmSZ+BguzOrPcUZN9x/
         Y6dYENKTCKEnLN8NXyy/DcQ02J22h+N53SPzkiTDzJZppZuucQ12emzmPcSSAwNLQVeQ
         KprH9V3Q/PVRJWrWTrKo2gzX8osBb8zdE8z5U=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:content-type:content-transfer-encoding:sender;
        b=PSPbMrNvkY0/8+uxgDktmSEh2NojnbucUp6GrRNeqI2cZRT6Tj6BzfjxxEjB6WCyjs
         fc8H51+Q+y0waZDixUkMlF+PMAnqQKJyAJUViXLApudfsTth+dF7JrVQrlui1ltWxkVx
         l9K/VRfG/MBJDcM1VWCTtRllcA1Y0PGuA1vP4=
Received: by 10.142.88.4 with SMTP id l4mr4240748wfb.238.1220651422530;
        Fri, 05 Sep 2008 14:50:22 -0700 (PDT)
Received: from ?192.168.168.16? ( [76.103.191.253])
        by mx.google.com with ESMTPS id 22sm1143404wfi.14.2008.09.05.14.50.21
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Fri, 05 Sep 2008 14:50:22 -0700 (PDT)
Message-ID: <48C1A99D.2010505@apache.org>
Date: Fri, 05 Sep 2008 14:50:21 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.16 (X11/20080724)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Changing email notifications
References: <B40A8D12-6B19-415F-8FFE-50FF2BD362F2@apache.org>
In-Reply-To: <B40A8D12-6B19-415F-8FFE-50FF2BD362F2@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Sender: Doug Cutting <cutting@gmail.com>
X-Virus-Checked: Checked by ClamAV on apache.org

+0

We really need to split hadoop-core into two or more subprojects 
Keeping HDFS, MapReduce, Hive, HOD, etc. in a single project is not 
scalable.  Nigel vetoed this because he thinks it will complicate QA, 
but I think (a) it will simplify so many other people's lives that it is 
well worth any such cost and (b) the net change in QA effort will be 
quite moderate.  Yes, there will be more projects to QA, but they'll be 
simpler projects.

Modifying the mailing lists is a band aid.  It will help those who're 
not already doing similar things with filters already, but it will cause 
everyone who does have filters to have to re-jigger their filters. 
There will be some confusion and time wasted explaining it to folks.

So I'm fine with this change being made, but it doesn't really address 
the glaring problem that we have too much happening under a single roof.

Doug

Owen O'Malley wrote:
> All, core-dev is pretty overwhelming. I'd like to change the jira 
> notifications so that:
> 
> jira created -> core-dev
> other jira traffic -> a new list core-jira@hadoop.apache.org.
> 
> Most of us have filters like that anyways, and it would make it a lot 
> less intimidating for developers getting started. I'm obviously +1. 
> Thoughts?
> 
> -- Owen

From general-return-35-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Sep 06 00:23:21 2008
Return-Path: <general-return-35-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 52783 invoked from network); 6 Sep 2008 00:23:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 6 Sep 2008 00:23:21 -0000
Received: (qmail 14559 invoked by uid 500); 6 Sep 2008 00:21:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14534 invoked by uid 500); 6 Sep 2008 00:21:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14493 invoked by uid 99); 6 Sep 2008 00:21:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Sep 2008 17:21:53 -0700
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 06 Sep 2008 00:20:45 +0000
Received: from thickbeside-lm.corp.yahoo.com (thickbeside-lm.corp.yahoo.com [10.72.109.129])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id m860Iv6d099748
	for <general@hadoop.apache.org>; Fri, 5 Sep 2008 17:18:57 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=HT09FtzqN3VN2rrK+s9Qv3rX7c7xvVr7pMxVEVJ4ihdkXnmVxXXyWBOlpRY3v3KC
Message-Id: <09791C7C-1B85-4E89-AFA6-CDD3470EEFDD@yahoo-inc.com>
From: Nigel Daley <ndaley@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <48C1A99D.2010505@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v928.1)
Subject: Re: [VOTE] Changing email notifications
Date: Fri, 5 Sep 2008 17:18:56 -0700
References: <B40A8D12-6B19-415F-8FFE-50FF2BD362F2@apache.org> <48C1A99D.2010505@apache.org>
X-Mailer: Apple Mail (2.928.1)
X-Virus-Checked: Checked by ClamAV on apache.org

I'm coming around on splitting up HDFS and Mapred etc.  I'll reply to  
that thread with a new shinny +1.

So that leaves me -1 on this proposal of splitting up the core/jira  
traffic.

Nige


On Sep 5, 2008, at 2:50 PM, Doug Cutting wrote:

> +0
>
> We really need to split hadoop-core into two or more subprojects  
> Keeping HDFS, MapReduce, Hive, HOD, etc. in a single project is not  
> scalable.  Nigel vetoed this because he thinks it will complicate  
> QA, but I think (a) it will simplify so many other people's lives  
> that it is well worth any such cost and (b) the net change in QA  
> effort will be quite moderate.  Yes, there will be more projects to  
> QA, but they'll be simpler projects.
>
> Modifying the mailing lists is a band aid.  It will help those  
> who're not already doing similar things with filters already, but it  
> will cause everyone who does have filters to have to re-jigger their  
> filters. There will be some confusion and time wasted explaining it  
> to folks.
>
> So I'm fine with this change being made, but it doesn't really  
> address the glaring problem that we have too much happening under a  
> single roof.
>
> Doug
>
> Owen O'Malley wrote:
>> All, core-dev is pretty overwhelming. I'd like to change the jira  
>> notifications so that:
>> jira created -> core-dev
>> other jira traffic -> a new list core-jira@hadoop.apache.org.
>> Most of us have filters like that anyways, and it would make it a  
>> lot less intimidating for developers getting started. I'm obviously  
>> +1. Thoughts?
>> -- Owen


From general-return-36-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Sep 06 00:23:59 2008
Return-Path: <general-return-36-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 54370 invoked from network); 6 Sep 2008 00:23:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 6 Sep 2008 00:23:58 -0000
Received: (qmail 17519 invoked by uid 500); 6 Sep 2008 00:23:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17501 invoked by uid 500); 6 Sep 2008 00:23:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17490 invoked by uid 99); 6 Sep 2008 00:23:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Sep 2008 17:23:56 -0700
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 06 Sep 2008 00:22:55 +0000
Received: from thickbeside-lm.corp.yahoo.com (thickbeside-lm.corp.yahoo.com [10.72.109.129])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id m860LtV7000928
	for <general@hadoop.apache.org>; Fri, 5 Sep 2008 17:21:55 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=XMyvUrXTVYlRYEvvNymYVdmGKe0184iy8SNTk2Tq1VjdQcfPGPHr04ltFmMM6YAa
Message-Id: <AA730EEB-8F4D-45DC-A785-A5B5512F8347@yahoo-inc.com>
From: Nigel Daley <ndaley@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <175CD9D8-34A3-4A66-87B0-359EA5E22169@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v928.1)
Subject: Re: [VOTE] Should we create sub-projects for HDFS and Map/Reduce?
Date: Fri, 5 Sep 2008 17:21:55 -0700
References: <688CEB2A-13EF-4C38-855A-3D6D4C3485B3@yahoo-inc.com> <2DA5A7B1-0CC0-4C4F-BE80-AE1E3C7B47A3@yahoo-inc.com> <489CA74F.8000901@apache.org> <175CD9D8-34A3-4A66-87B0-359EA5E22169@yahoo-inc.com>
X-Mailer: Apple Mail (2.928.1)
X-Virus-Checked: Checked by ClamAV on apache.org


On Aug 15, 2008, at 10:03 PM, Nigel Daley wrote:

>> Another benefit is that it would increase the separation of these  
>> technologies, so that, e.g., folks could more easily run different  
>> versions of mapreduce on top of different versions of HDFS.   
>> Currently we make no such guarantees.  Folks would be able to  
>> upgrade to, e.g., the next release of mapreduce on a subset of  
>> their cluster without upgrading their HDFS.  That's not currently  
>> supported.  As we move towards splitting mapreduce into a scheduler  
>> and runtime, where folks can specify a different runtime per job,  
>> this will be even more critical.
>
> Sounds like we simply need to create separate jar files for these  
> different components.  This can be done in the current project.
>
> Wouldn't the amount of effort to make this split and get it right be  
> better spent on getting all components of Hadoop to 1.0 (API  
> stability)?  The proposal feels like a distraction to me at this  
> point in the project.
>
> Nige

I'd like to retract the -1 vote that I gave this proposal earlier.   
One compelling reason (for me) to split HDFS and Map/Reduce into  
separate sub-projects is that (hopefully) the *configs* for each layer  
will be clearer and simpler.

So I'm now +1 on this proposal.

Nige

From general-return-37-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Sep 14 04:42:24 2008
Return-Path: <general-return-37-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 81217 invoked from network); 14 Sep 2008 04:42:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 14 Sep 2008 04:42:23 -0000
Received: (qmail 65290 invoked by uid 500); 14 Sep 2008 04:42:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65262 invoked by uid 500); 14 Sep 2008 04:42:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65251 invoked by uid 99); 14 Sep 2008 04:42:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 13 Sep 2008 21:42:20 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of owen.omalley@gmail.com designates 209.85.198.227 as permitted sender)
Received: from [209.85.198.227] (HELO rv-out-0506.google.com) (209.85.198.227)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 14 Sep 2008 04:41:19 +0000
Received: by rv-out-0506.google.com with SMTP id k40so1422102rvb.29
        for <general@hadoop.apache.org>; Sat, 13 Sep 2008 21:41:51 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:from:to
         :content-type:content-transfer-encoding:mime-version:subject:date
         :x-mailer:sender;
        bh=H0TAVAi8qcOh5Jv71Dkm5hejyCO1k7H+jCmEb6AAe2w=;
        b=u26+rAYgWdfiKYyEfxGbz+VBGnmxzfXIbpit51Hm8XNnPAuH71cQSvra1BYHLAMvXy
         szEpNOSgSJACHw9XWaY8BWIxj1GGOfSuoQ13NuWp1wxI8EeqCUIxonliXg0uxjdvn7Af
         VPSIpv2maMcaYVoSGIAvmhdW7s3/HVn2pBqho=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:from:to:content-type:content-transfer-encoding
         :mime-version:subject:date:x-mailer:sender;
        b=bushHh4win8GWF1x5J6w3gjN4z0PUyGnkhE8Wli3EcanTi/lSP82nc+xLZInr2oMJ4
         nCBqb+TTqmoJqyTE8xjU2UORJL19N9aYxX8O3Hmh+nPrMLQUNdSHND55XTh1HeWci49/
         iFs0IFewIrLXN3Twqw42yZmFUGJK5wQBPhLPs=
Received: by 10.140.125.1 with SMTP id x1mr3787296rvc.217.1221367310519;
        Sat, 13 Sep 2008 21:41:50 -0700 (PDT)
Received: from ?10.65.111.22? ( [209.131.62.115])
        by mx.google.com with ESMTPS id c20sm19633575rvf.3.2008.09.13.21.41.49
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Sat, 13 Sep 2008 21:41:49 -0700 (PDT)
Message-Id: <7A76F46E-F2B3-406F-8B20-4E0109A003EE@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v926)
Subject: Hadoop Git repositories available for cloning
Date: Sat, 13 Sep 2008 21:41:48 -0700
X-Mailer: Apple Mail (2.926)
Sender: Owen O'Malley <owen.omalley@gmail.com>
X-Virus-Checked: Checked by ClamAV on apache.org

Hi all,
   Jukka Zitting has made git repositories that track Apache  
subversion of all 3 Hadoop subprojects, Core, HBase, and Zookeeper.

http://jukka.zitting.name/git/

-- Owen

From general-return-38-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 15 22:03:36 2008
Return-Path: <general-return-38-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 73706 invoked from network); 15 Sep 2008 22:03:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 15 Sep 2008 22:03:36 -0000
Received: (qmail 67745 invoked by uid 500); 15 Sep 2008 22:03:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67723 invoked by uid 500); 15 Sep 2008 22:03:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67712 invoked by uid 99); 15 Sep 2008 22:03:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Sep 2008 15:03:33 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 209.85.128.191 as permitted sender)
Received: from [209.85.128.191] (HELO fk-out-0910.google.com) (209.85.128.191)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Sep 2008 22:02:34 +0000
Received: by fk-out-0910.google.com with SMTP id 26so2139326fkx.13
        for <general@hadoop.apache.org>; Mon, 15 Sep 2008 15:03:06 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:to
         :subject:in-reply-to:mime-version:content-type
         :content-transfer-encoding:content-disposition:references;
        bh=qjw2js3FcbRWWkpT0QRWD6+Jmd9HSsuvkkpjbEd0O3I=;
        b=TNw+OWqHbMFV6z/aIz+Y9pZVerkte3NvxmYFGyp0/2VLfUadJl47QJ4vV6rNoiS6qx
         fwshJr2scnTKjXjesZgQ4NhEmqLhJZjptatYghcS+Ked50MPadgh0QX3W6QR288Qkv8b
         knmferHq8KWvYmQgCgHMldNSzMXC+J3H/PuWY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version
         :content-type:content-transfer-encoding:content-disposition
         :references;
        b=t3BeJEqACtGnSocScfILVZLQaA9mBAnMPjTrFhhELoV92TFFUj6qsHO5nCLHNiTFYs
         U583MKOBTV6GIBpXMFeEOv7M/PF84URGcSV6T/Di2sHJOwlGAh/rNrF4N9HU9pQsr73V
         0LY++4IW4THQh0jAL3YywO3LsnbFWWaDcseYY=
Received: by 10.187.161.10 with SMTP id n10mr19055fao.43.1221516186413;
        Mon, 15 Sep 2008 15:03:06 -0700 (PDT)
Received: by 10.187.235.3 with HTTP; Mon, 15 Sep 2008 15:03:06 -0700 (PDT)
Message-ID: <4aa34eb70809151503k2cf48770qd6d85d4007eb3196@mail.gmail.com>
Date: Mon, 15 Sep 2008 15:03:06 -0700
From: "Dhruba Borthakur" <dhruba@gmail.com>
To: general@hadoop.apache.org
Subject: Re: [VOTE] Should we create sub-projects for HDFS and Map/Reduce?
In-Reply-To: <AA730EEB-8F4D-45DC-A785-A5B5512F8347@yahoo-inc.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <688CEB2A-13EF-4C38-855A-3D6D4C3485B3@yahoo-inc.com>
	 <2DA5A7B1-0CC0-4C4F-BE80-AE1E3C7B47A3@yahoo-inc.com>
	 <489CA74F.8000901@apache.org>
	 <175CD9D8-34A3-4A66-87B0-359EA5E22169@yahoo-inc.com>
	 <AA730EEB-8F4D-45DC-A785-A5B5512F8347@yahoo-inc.com>
X-Virus-Checked: Checked by ClamAV on apache.org

+1

I would prefer to keep hdfs and mapreduce together because I believe
that this arrangement catches incompatability sooner that later. But
with the coming of age of both these modules separately, I guess it is
time to grow them on their own individual turf!

thanks,
dhruba

On Fri, Sep 5, 2008 at 5:21 PM, Nigel Daley <ndaley@yahoo-inc.com> wrote:
>
> On Aug 15, 2008, at 10:03 PM, Nigel Daley wrote:
>
>>> Another benefit is that it would increase the separation of these
>>> technologies, so that, e.g., folks could more easily run different versions
>>> of mapreduce on top of different versions of HDFS.  Currently we make no
>>> such guarantees.  Folks would be able to upgrade to, e.g., the next release
>>> of mapreduce on a subset of their cluster without upgrading their HDFS.
>>>  That's not currently supported.  As we move towards splitting mapreduce
>>> into a scheduler and runtime, where folks can specify a different runtime
>>> per job, this will be even more critical.
>>
>> Sounds like we simply need to create separate jar files for these
>> different components.  This can be done in the current project.
>>
>> Wouldn't the amount of effort to make this split and get it right be
>> better spent on getting all components of Hadoop to 1.0 (API stability)?
>>  The proposal feels like a distraction to me at this point in the project.
>>
>> Nige
>
> I'd like to retract the -1 vote that I gave this proposal earlier.  One
> compelling reason (for me) to split HDFS and Map/Reduce into separate
> sub-projects is that (hopefully) the *configs* for each layer will be
> clearer and simpler.
>
> So I'm now +1 on this proposal.
>
> Nige
>

From general-return-39-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 15 22:10:22 2008
Return-Path: <general-return-39-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 77716 invoked from network); 15 Sep 2008 22:10:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 15 Sep 2008 22:10:22 -0000
Received: (qmail 77428 invoked by uid 500); 15 Sep 2008 22:10:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77407 invoked by uid 500); 15 Sep 2008 22:10:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77396 invoked by uid 99); 15 Sep 2008 22:10:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Sep 2008 15:10:19 -0700
X-ASF-Spam-Status: No, hits=4.0 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_BL_SPAMCOP_NET,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.30.64] (HELO QMTA07.emeryville.ca.mail.comcast.net) (76.96.30.64)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Sep 2008 22:09:19 +0000
Received: from OMTA03.emeryville.ca.mail.comcast.net ([76.96.30.27])
	by QMTA07.emeryville.ca.mail.comcast.net with comcast
	id F5xS1a00K0b6N64A7A9jEP; Mon, 15 Sep 2008 22:09:43 +0000
Received: from battlerock-lm.corp.yahoo.com ([209.131.62.113])
	by OMTA03.emeryville.ca.mail.comcast.net with comcast
	id FA9N1a00Z2SbwD58PA9T2U; Mon, 15 Sep 2008 22:09:38 +0000
X-Authority-Analysis: v=1.0 c=1 a=YC1Ora_X4Q4A:10 a=SLclp5Jn_4gA:10
 a=rwkqtkcKwTFGcIoYXY8A:9 a=fwJqHX_d6-LChCBWTrEVw8a1M1QA:4 a=0HbQqGD8AL8A:10
Message-Id: <E371E4CF-9BBD-4881-B88B-811750A2EAD2@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <AA730EEB-8F4D-45DC-A785-A5B5512F8347@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v926)
Subject: Re: [VOTE] Should we create sub-projects for HDFS and Map/Reduce?
Date: Mon, 15 Sep 2008 15:09:22 -0700
References: <688CEB2A-13EF-4C38-855A-3D6D4C3485B3@yahoo-inc.com> <2DA5A7B1-0CC0-4C4F-BE80-AE1E3C7B47A3@yahoo-inc.com> <489CA74F.8000901@apache.org> <175CD9D8-34A3-4A66-87B0-359EA5E22169@yahoo-inc.com> <AA730EEB-8F4D-45DC-A785-A5B5512F8347@yahoo-inc.com>
X-Mailer: Apple Mail (2.926)
X-Virus-Checked: Checked by ClamAV on apache.org

Ok, so in very protracted voting, the results are:

PMC +1's: Arun, Dhruba, Doug, Nigel, Owen, Tom

So the vote passes.

-- Owen

From general-return-40-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 15 22:21:29 2008
Return-Path: <general-return-40-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 86099 invoked from network); 15 Sep 2008 22:21:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 15 Sep 2008 22:21:29 -0000
Received: (qmail 88238 invoked by uid 500); 15 Sep 2008 22:21:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88220 invoked by uid 500); 15 Sep 2008 22:21:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88209 invoked by uid 99); 15 Sep 2008 22:21:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Sep 2008 15:21:26 -0700
X-ASF-Spam-Status: No, hits=4.0 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_BL_SPAMCOP_NET,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.62.96] (HELO QMTA09.westchester.pa.mail.comcast.net) (76.96.62.96)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Sep 2008 22:20:26 +0000
Received: from OMTA04.westchester.pa.mail.comcast.net ([76.96.62.35])
	by QMTA09.westchester.pa.mail.comcast.net with comcast
	id F8rn1a00l0ldTLk59ALy9D; Mon, 15 Sep 2008 22:20:58 +0000
Received: from battlerock-lm.corp.yahoo.com ([209.131.62.113])
	by OMTA04.westchester.pa.mail.comcast.net with comcast
	id FALd1a0042SbwD53QALi2A; Mon, 15 Sep 2008 22:20:53 +0000
X-Authority-Analysis: v=1.0 c=1 a=gz-5--JcPi5Fb0raVxIA:9
 a=PKxEtFZ-xcYhgslnY8691zLjSuMA:4 a=MInBT2ynV54A:10
Message-Id: <FE98652A-B068-48DD-9619-5B7F7EE96C2B@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v926)
Subject: [VOTE] Split Hive into a sub-project
Date: Mon, 15 Sep 2008 15:20:36 -0700
X-Mailer: Apple Mail (2.926)
X-Virus-Checked: Checked by ClamAV on apache.org

With Map/Reduce and HDFS being split up, I think we should split out  
Hive into a sub-project also. It is already a large code-base and  
seems to be picking up external users. Clearly, I'm +1.

-- Owen

From general-return-41-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 15 22:39:15 2008
Return-Path: <general-return-41-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 159 invoked from network); 15 Sep 2008 22:39:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 15 Sep 2008 22:39:15 -0000
Received: (qmail 4112 invoked by uid 500); 15 Sep 2008 22:39:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4091 invoked by uid 500); 15 Sep 2008 22:39:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4080 invoked by uid 99); 15 Sep 2008 22:39:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Sep 2008 15:39:11 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cutting@gmail.com designates 64.233.166.182 as permitted sender)
Received: from [64.233.166.182] (HELO py-out-1112.google.com) (64.233.166.182)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Sep 2008 22:38:12 +0000
Received: by py-out-1112.google.com with SMTP id f47so1900044pye.8
        for <general@hadoop.apache.org>; Mon, 15 Sep 2008 15:38:27 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding:sender;
        bh=/nvUGJhsziXAq8j2hYKH74OwksNShAbH7ozxtqyl0TI=;
        b=UVAIl/6KF43cXEqzA5rZeUtmmu+7wMokpwzWy3WXpLlN/A+9EiSKXu3ZRDdAWAyAqT
         5xpVBekNw0bxkXLVKyZ8cIa0/xaEWhkGnOOMqabJ8qrs5Bj/PNNxN2gfHVgt36NG0Z25
         8bbXezFyFHWVYow+02AooOECyJg6jG8jA/qmY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:content-type:content-transfer-encoding:sender;
        b=fKduHgKPNxVm4afiadPWMh6DbjTKgmKiGqHSXLNDszTGcZNQk/4Q9W02OwVkj4X3ot
         jUJONo2KDfG6SZhC1tl/SjXdA6GlQd40qS9we8F3uS0WGHWxAjqvm4M7KhhCxC5EuG94
         kfd+yWelx/hKMJ2y4sFlJN3upJ1ptILK4yXCY=
Received: by 10.142.211.1 with SMTP id j1mr59774wfg.313.1221518307176;
        Mon, 15 Sep 2008 15:38:27 -0700 (PDT)
Received: from ?192.168.168.16? ( [76.103.191.253])
        by mx.google.com with ESMTPS id 32sm22711127wfc.12.2008.09.15.15.38.26
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Mon, 15 Sep 2008 15:38:26 -0700 (PDT)
Message-ID: <48CEE3E2.8030802@apache.org>
Date: Mon, 15 Sep 2008 15:38:26 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.16 (X11/20080724)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Split Hive into a sub-project
References: <FE98652A-B068-48DD-9619-5B7F7EE96C2B@apache.org>
In-Reply-To: <FE98652A-B068-48DD-9619-5B7F7EE96C2B@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Sender: Doug Cutting <cutting@gmail.com>
X-Virus-Checked: Checked by ClamAV on apache.org

+1, but is anyone who works on Hive on the PMC?

Once we agree to all these splits, scheduling and planning them will 
sure be fun!

Doug

Owen O'Malley wrote:
> With Map/Reduce and HDFS being split up, I think we should split out 
> Hive into a sub-project also. It is already a large code-base and seems 
> to be picking up external users. Clearly, I'm +1.
> 
> -- Owen

From general-return-42-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 15 22:47:46 2008
Return-Path: <general-return-42-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 7995 invoked from network); 15 Sep 2008 22:47:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 15 Sep 2008 22:47:46 -0000
Received: (qmail 11673 invoked by uid 500); 15 Sep 2008 22:47:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11647 invoked by uid 500); 15 Sep 2008 22:47:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11636 invoked by uid 99); 15 Sep 2008 22:47:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Sep 2008 15:47:43 -0700
X-ASF-Spam-Status: No, hits=4.0 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_BL_SPAMCOP_NET,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.30.56] (HELO QMTA06.emeryville.ca.mail.comcast.net) (76.96.30.56)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Sep 2008 22:46:43 +0000
Received: from OMTA10.emeryville.ca.mail.comcast.net ([76.96.30.28])
	by QMTA06.emeryville.ca.mail.comcast.net with comcast
	id F7bi1a0010cQ2SLA6AnFcC; Mon, 15 Sep 2008 22:47:15 +0000
Received: from battlerock-lm.corp.yahoo.com ([209.131.62.113])
	by OMTA10.emeryville.ca.mail.comcast.net with comcast
	id FAmu1a0082SbwD58WAmzGx; Mon, 15 Sep 2008 22:47:10 +0000
X-Authority-Analysis: v=1.0 c=1 a=E8KE17LsI4cA:10 a=1YF0TBQqt31x0LhLTo8A:9
 a=dFMkrW1ZqXQDF1H8UbUA21hy8QsA:4 a=WuK_CZDBSqoA:10
Message-Id: <E4C95AE2-8F9A-4705-8BC2-EDAED7B7A7F1@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <48CEE3E2.8030802@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v926)
Subject: Re: [VOTE] Split Hive into a sub-project
Date: Mon, 15 Sep 2008 15:46:54 -0700
References: <FE98652A-B068-48DD-9619-5B7F7EE96C2B@apache.org> <48CEE3E2.8030802@apache.org>
X-Mailer: Apple Mail (2.926)
X-Virus-Checked: Checked by ClamAV on apache.org


On Sep 15, 2008, at 3:38 PM, Doug Cutting wrote:

> +1, but is anyone who works on Hive on the PMC?

Dhruba is on the PMC and has been doing most of their commits, so he  
would be a reasonable proxy.

-- Owen

From general-return-43-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 15 22:49:03 2008
Return-Path: <general-return-43-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 9644 invoked from network); 15 Sep 2008 22:49:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 15 Sep 2008 22:49:03 -0000
Received: (qmail 16267 invoked by uid 500); 15 Sep 2008 22:48:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16249 invoked by uid 500); 15 Sep 2008 22:48:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16238 invoked by uid 99); 15 Sep 2008 22:48:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Sep 2008 15:48:59 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 72.14.220.154 as permitted sender)
Received: from [72.14.220.154] (HELO fg-out-1718.google.com) (72.14.220.154)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Sep 2008 22:48:01 +0000
Received: by fg-out-1718.google.com with SMTP id l26so1486664fgb.35
        for <general@hadoop.apache.org>; Mon, 15 Sep 2008 15:48:15 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:to
         :subject:in-reply-to:mime-version:content-type
         :content-transfer-encoding:content-disposition:references;
        bh=Cqvbwi5HqyI2ZWPX93Y2SDKjsqOPfPXsh1UxzdouU0c=;
        b=icOCryUM8/nHGWHo0T+//hUE3om9vkHdHlkdqM1gL6P4YkZLvipirFedZ9t82SYKmJ
         y+cTUk9UQrT/gpC7OSW8fqQOjVqhjNFzMN5RjZ1nchHIhjDa8F2cQQUBLM0C7Al649L1
         v6Ii8WEswVg+YUkayTvi8KHyB2ya8eCiQ4XkY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version
         :content-type:content-transfer-encoding:content-disposition
         :references;
        b=EMYbO6yFB8Ri7rPwzRkpt4m2IP7SaiU/9nR2JJe59OfjajnPXxkR8w8+Op5NehIxC+
         6rxTh/LMArAsFeCVhVKBSUchcWVYkFZChaFkAagkD12e3s0Nj74npg7D7BKE/ah0illN
         v6/SDA8WwtCRhBvSS77zfkjGFeK3u3oPLu2Z0=
Received: by 10.187.190.6 with SMTP id s6mr23260fap.51.1221518895713;
        Mon, 15 Sep 2008 15:48:15 -0700 (PDT)
Received: by 10.187.235.3 with HTTP; Mon, 15 Sep 2008 15:48:15 -0700 (PDT)
Message-ID: <4aa34eb70809151548q19161967mf8c4c215eade2c1d@mail.gmail.com>
Date: Mon, 15 Sep 2008 15:48:15 -0700
From: "Dhruba Borthakur" <dhruba@gmail.com>
To: general@hadoop.apache.org
Subject: Re: [VOTE] Split Hive into a sub-project
In-Reply-To: <E4C95AE2-8F9A-4705-8BC2-EDAED7B7A7F1@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <FE98652A-B068-48DD-9619-5B7F7EE96C2B@apache.org>
	 <48CEE3E2.8030802@apache.org>
	 <E4C95AE2-8F9A-4705-8BC2-EDAED7B7A7F1@apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

+1. This will allow us to have an independent release schedule for Hive.

-dhruba

On Mon, Sep 15, 2008 at 3:46 PM, Owen O'Malley <omalley@apache.org> wrote:
>
> On Sep 15, 2008, at 3:38 PM, Doug Cutting wrote:
>
>> +1, but is anyone who works on Hive on the PMC?
>
> Dhruba is on the PMC and has been doing most of their commits, so he would
> be a reasonable proxy.
>
> -- Owen
>

From general-return-44-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 15 23:05:17 2008
Return-Path: <general-return-44-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 25173 invoked from network); 15 Sep 2008 23:05:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 15 Sep 2008 23:05:17 -0000
Received: (qmail 38170 invoked by uid 500); 15 Sep 2008 23:05:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38067 invoked by uid 500); 15 Sep 2008 23:05:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 38056 invoked by uid 99); 15 Sep 2008 23:05:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Sep 2008 16:05:14 -0700
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Sep 2008 23:04:13 +0000
Received: from wlan-mc2-145-249.corp.yahoo.com (snvvpn1-10-72-73-c69.corp.yahoo.com [10.72.73.69])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id m8FN2Cvp011804
	for <general@hadoop.apache.org>; Mon, 15 Sep 2008 16:02:12 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=t1Ue0w70asphly79JOy5VZ6kvKeN2qVORrg97/OO/YxP46jGZw6doVgk5gVTB6Iz
Message-Id: <5D47BCE4-3701-4E4C-BEDA-6871ECDA7A60@yahoo-inc.com>
From: Nigel Daley <ndaley@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <FE98652A-B068-48DD-9619-5B7F7EE96C2B@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v928.1)
Subject: Re: [VOTE] Split Hive into a sub-project
Date: Mon, 15 Sep 2008 16:02:12 -0700
References: <FE98652A-B068-48DD-9619-5B7F7EE96C2B@apache.org>
X-Mailer: Apple Mail (2.928.1)
X-Virus-Checked: Checked by ClamAV on apache.org

+1

On Sep 15, 2008, at 3:20 PM, Owen O'Malley wrote:

> With Map/Reduce and HDFS being split up, I think we should split out  
> Hive into a sub-project also. It is already a large code-base and  
> seems to be picking up external users. Clearly, I'm +1.
>
> -- Owen


From general-return-45-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 15 23:18:06 2008
Return-Path: <general-return-45-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 32428 invoked from network); 15 Sep 2008 23:18:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 15 Sep 2008 23:18:05 -0000
Received: (qmail 46982 invoked by uid 500); 15 Sep 2008 23:18:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46962 invoked by uid 500); 15 Sep 2008 23:18:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46951 invoked by uid 99); 15 Sep 2008 23:18:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Sep 2008 16:18:02 -0700
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Sep 2008 23:17:01 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id m8FNGBGA049644
	for <general@hadoop.apache.org>; Mon, 15 Sep 2008 16:16:12 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=qg5iisEUg/9ml+6NeSHpIXu8XOtnFYrDaDHEKidzdC6f/5pZleE7Cx1rgz93hJyj
Message-Id: <457F4921-353D-479D-B383-378DF7BCA3B5@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <FE98652A-B068-48DD-9619-5B7F7EE96C2B@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v928.1)
Subject: Re: [VOTE] Split Hive into a sub-project
Date: Mon, 15 Sep 2008 16:16:11 -0700
References: <FE98652A-B068-48DD-9619-5B7F7EE96C2B@apache.org>
X-Mailer: Apple Mail (2.928.1)
X-Virus-Checked: Checked by ClamAV on apache.org


On Sep 15, 2008, at 3:20 PM, Owen O'Malley wrote:

> With Map/Reduce and HDFS being split up, I think we should split out  
> Hive into a sub-project also. It is already a large code-base and  
> seems to be picking up external users. Clearly, I'm +1.
>

+1
It might be a tad early, but I am reasonably comfortable that Hive  
will have a healthy community for itself.

Arun


From general-return-46-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 16 07:54:10 2008
Return-Path: <general-return-46-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 19560 invoked from network); 16 Sep 2008 07:54:10 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 16 Sep 2008 07:54:10 -0000
Received: (qmail 77561 invoked by uid 500); 16 Sep 2008 07:54:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77532 invoked by uid 500); 16 Sep 2008 07:54:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77521 invoked by uid 99); 16 Sep 2008 07:54:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Sep 2008 00:54:07 -0700
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: error (athena.apache.org: encountered temporary error during SPF processing of domain of enis.soz.nutch@gmail.com)
Received: from [66.206.198.176] (HELO Generic33.powerdnn.com) (66.206.198.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Sep 2008 07:53:08 +0000
Received: from [192.168.2.15] ([85.105.135.220]) by Generic33.powerdnn.com with MailEnable ESMTP; Tue, 16 Sep 2008 00:55:47 -0700
Message-ID: <48CF65F1.4060404@gmail.com>
Date: Tue, 16 Sep 2008 10:53:21 +0300
From: Enis Soztutar <enis.soz.nutch@gmail.com>
User-Agent: Thunderbird 2.0.0.16 (X11/20080724)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Split Hive into a sub-project
References: <FE98652A-B068-48DD-9619-5B7F7EE96C2B@apache.org>
In-Reply-To: <FE98652A-B068-48DD-9619-5B7F7EE96C2B@apache.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-ME-Bayesian: 0.000000
X-Virus-Checked: Checked by ClamAV on apache.org

+1, Hive already generates lots of traffic, that we were initially 
concerned about.

Owen O'Malley wrote:
> With Map/Reduce and HDFS being split up, I think we should split out 
> Hive into a sub-project also. It is already a large code-base and 
> seems to be picking up external users. Clearly, I'm +1.
>
> -- Owen
>


From general-return-47-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Sep 17 07:05:15 2008
Return-Path: <general-return-47-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 29193 invoked from network); 17 Sep 2008 07:05:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 17 Sep 2008 07:05:15 -0000
Received: (qmail 52628 invoked by uid 500); 17 Sep 2008 07:05:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52606 invoked by uid 500); 17 Sep 2008 07:05:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52590 invoked by uid 99); 17 Sep 2008 07:05:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Sep 2008 00:05:09 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of rasmussen.bryan@gmail.com designates 209.85.217.13 as permitted sender)
Received: from [209.85.217.13] (HELO mail-gx0-f13.google.com) (209.85.217.13)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Sep 2008 07:04:06 +0000
Received: by gxk6 with SMTP id 6so19918380gxk.5
        for <general@hadoop.apache.org>; Wed, 17 Sep 2008 00:03:39 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:to
         :subject:mime-version:content-type:content-transfer-encoding
         :content-disposition;
        bh=iLyDQ/u+4uxeCv8WmHrQrhge/uOy2dytzc+tjXnAFuc=;
        b=mBI6zovAk7FSyHn5MKBbyasLmhAaD6GeelMHkQXQoU3t2ykCWSCN/VcNfA/NdX97Ir
         1ebn5n3NUbYQiQv0g+lJ67XFsjBi89BcWw4O1P+kfPNBuHXOT3xAhKfplZM1HYv6KNdq
         n0huC92Z2DIY6WDhg7QLHEx6dn2yvjp1kdOS8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:mime-version:content-type
         :content-transfer-encoding:content-disposition;
        b=m8x2OGYSivRZbmvRH1pwxs3N4d0EtN1ibgyE0syUeLQofdSmmr1X/DJKdJaQHENnVe
         UxtmqxunCOJXyxdLyTdMg73aJ4oecc9icLgCwTiH4L7/wDw6yRSelngguD1LXj0G88OY
         5ZWFsM9XQ2Jz2XVPwJxM8TP6vUoJBG4a24CTA=
Received: by 10.64.88.5 with SMTP id l5mr305303qbb.61.1221635018414;
        Wed, 17 Sep 2008 00:03:38 -0700 (PDT)
Received: by 10.65.81.20 with HTTP; Wed, 17 Sep 2008 00:03:38 -0700 (PDT)
Message-ID: <3bb44c6e0809170003yfe0b1cdv7ba7b13b286f6759@mail.gmail.com>
Date: Wed, 17 Sep 2008 09:03:38 +0200
From: "bryan rasmussen" <rasmussen.bryan@gmail.com>
To: general@hadoop.apache.org
Subject: suggestions for using Hadoop/HBase to make a delicious type bookmarking service
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

I'm starting the initial design phase of a service that is similar to
the various bookmarking services out there and am thinking about maybe
using Hadoop/Hbase. Does anyone here have some suggestions, pointers
(aside from the normal faq, documentation) about using Hadoop and/or
HBase for building such a solution? Since I am just in the design
phase I have to decide what technologies I should use, and their
suitability for the tasks at hand.

Best Regards,
Bryan Rasmussen

From general-return-48-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Sep 17 09:43:55 2008
Return-Path: <general-return-48-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 17030 invoked from network); 17 Sep 2008 09:43:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 17 Sep 2008 09:43:55 -0000
Received: (qmail 38373 invoked by uid 500); 17 Sep 2008 09:43:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38262 invoked by uid 500); 17 Sep 2008 09:43:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 38251 invoked by uid 99); 17 Sep 2008 09:43:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Sep 2008 02:43:51 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of tom.e.white@gmail.com designates 209.85.198.232 as permitted sender)
Received: from [209.85.198.232] (HELO rv-out-0506.google.com) (209.85.198.232)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Sep 2008 09:42:53 +0000
Received: by rv-out-0506.google.com with SMTP id k40so2837000rvb.29
        for <general@hadoop.apache.org>; Wed, 17 Sep 2008 02:43:25 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:to
         :subject:in-reply-to:mime-version:content-type
         :content-transfer-encoding:content-disposition:references;
        bh=aRIlesE5VaDe/pEgCFDXl2pbIR7Ac/1lucla6w5CcwE=;
        b=VHF5ZJ6KnKdGppG6uWR3M6zsKF8TC903joDdIlEwCKpB4dZHbqJLF83MCyTsDrb87T
         n12/MeWWn/uw8o71QPhkxkVmSpOwRSMIFpP6f0GJoHJCPerX/oGFN4XmiQ0JA77omNms
         4ZkCD9g5mmzMUSwj+Afw8i31jqwoxFw9JsQn0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version
         :content-type:content-transfer-encoding:content-disposition
         :references;
        b=SE4GHdnntEzGwysaaxeimwxC4oJbwFiG2HyvjKu1vcprqwUR24s7BA3fvabxbopxR/
         /HJ/Lzf0cBu0GsK3/dS3R5IJb+jfyqeDV9ScTVae0/Id6uuppTbOlOF8C44Nq76H/j44
         u/p+POZ8EEfLu+S9HdkTbCb3FlWg7nO5fVz6Y=
Received: by 10.140.170.12 with SMTP id s12mr6399191rve.101.1221644605900;
        Wed, 17 Sep 2008 02:43:25 -0700 (PDT)
Received: by 10.140.188.16 with HTTP; Wed, 17 Sep 2008 02:43:25 -0700 (PDT)
Message-ID: <b23e01b50809170243l658c492bt948f658bb1eca831@mail.gmail.com>
Date: Wed, 17 Sep 2008 10:43:25 +0100
From: "Tom White" <tom.e.white@gmail.com>
To: general@hadoop.apache.org
Subject: Re: [VOTE] Should we create sub-projects for HDFS and Map/Reduce?
In-Reply-To: <E371E4CF-9BBD-4881-B88B-811750A2EAD2@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <688CEB2A-13EF-4C38-855A-3D6D4C3485B3@yahoo-inc.com>
	 <2DA5A7B1-0CC0-4C4F-BE80-AE1E3C7B47A3@yahoo-inc.com>
	 <489CA74F.8000901@apache.org>
	 <175CD9D8-34A3-4A66-87B0-359EA5E22169@yahoo-inc.com>
	 <AA730EEB-8F4D-45DC-A785-A5B5512F8347@yahoo-inc.com>
	 <E371E4CF-9BBD-4881-B88B-811750A2EAD2@apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

BTW The initial work to sort out the module dependencies (but not
actually split the projects) is being carried out in
https://issues.apache.org/jira/browse/HADOOP-3750.

Tom

On Mon, Sep 15, 2008 at 11:09 PM, Owen O'Malley <omalley@apache.org> wrote:
> Ok, so in very protracted voting, the results are:
>
> PMC +1's: Arun, Dhruba, Doug, Nigel, Owen, Tom
>
> So the vote passes.
>
> -- Owen
>

From general-return-49-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Sep 17 09:45:04 2008
Return-Path: <general-return-49-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 18241 invoked from network); 17 Sep 2008 09:45:04 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 17 Sep 2008 09:45:04 -0000
Received: (qmail 39809 invoked by uid 500); 17 Sep 2008 09:45:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39783 invoked by uid 500); 17 Sep 2008 09:45:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39772 invoked by uid 99); 17 Sep 2008 09:45:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Sep 2008 02:45:00 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of tom.e.white@gmail.com designates 209.85.198.238 as permitted sender)
Received: from [209.85.198.238] (HELO rv-out-0506.google.com) (209.85.198.238)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Sep 2008 09:44:02 +0000
Received: by rv-out-0506.google.com with SMTP id k40so2837410rvb.29
        for <general@hadoop.apache.org>; Wed, 17 Sep 2008 02:44:35 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:to
         :subject:in-reply-to:mime-version:content-type
         :content-transfer-encoding:content-disposition:references;
        bh=UBJBmg1hf8GsuJ9GVO/+NDuKvQVAWAISZxW7f6Iey3U=;
        b=eO2Fzbqm2RszyZQPpGspNs4y3NxPrviXxbnoH2CAy0usqoe0o46WslyHN2WsYI24cl
         aaxossB05ISVLQ8cehQ4bFy5SzTHdGDIfztaC7IoUmtIWjnStU87jMnSmlefs2OqAB99
         LR1U/ilJFbuW7PyEXq4/RBmjGPeM9DiZlGwhk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version
         :content-type:content-transfer-encoding:content-disposition
         :references;
        b=g7gsU/jQRwIt1lNOudhyqgDxhJ2rCvQAD2WTHWpAyrB6BInGxXRfSkr4I1wxrqDDk9
         g897rfV5389kvXQedfW+XwEMJyIKCtOlovL1VCoGFxVGk0t9GH2WcjIFzTf4oqCttAaF
         Y+U1T6Vr5OhOBhG0bwBsTp0flp/I4zAQDvOgY=
Received: by 10.141.53.20 with SMTP id f20mr6396147rvk.128.1221644674978;
        Wed, 17 Sep 2008 02:44:34 -0700 (PDT)
Received: by 10.140.188.16 with HTTP; Wed, 17 Sep 2008 02:44:34 -0700 (PDT)
Message-ID: <b23e01b50809170244k2a57f018x343c7b56eec065b6@mail.gmail.com>
Date: Wed, 17 Sep 2008 10:44:34 +0100
From: "Tom White" <tom.e.white@gmail.com>
To: general@hadoop.apache.org
Subject: Re: [VOTE] Split Hive into a sub-project
In-Reply-To: <FE98652A-B068-48DD-9619-5B7F7EE96C2B@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <FE98652A-B068-48DD-9619-5B7F7EE96C2B@apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

+1

On Mon, Sep 15, 2008 at 11:20 PM, Owen O'Malley <omalley@apache.org> wrote:
> With Map/Reduce and HDFS being split up, I think we should split out Hive
> into a sub-project also. It is already a large code-base and seems to be
> picking up external users. Clearly, I'm +1.
>
> -- Owen
>

From general-return-50-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Sep 17 12:32:09 2008
Return-Path: <general-return-50-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 14874 invoked from network); 17 Sep 2008 12:32:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 17 Sep 2008 12:32:09 -0000
Received: (qmail 60052 invoked by uid 500); 17 Sep 2008 12:32:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59994 invoked by uid 500); 17 Sep 2008 12:32:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59973 invoked by uid 99); 17 Sep 2008 12:32:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Sep 2008 05:32:04 -0700
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of tang9527@gmail.com designates 209.85.142.191 as permitted sender)
Received: from [209.85.142.191] (HELO ti-out-0910.google.com) (209.85.142.191)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Sep 2008 12:31:06 +0000
Received: by ti-out-0910.google.com with SMTP id d27so1980391tid.9
        for <general@hadoop.apache.org>; Wed, 17 Sep 2008 05:31:37 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:to
         :subject:in-reply-to:mime-version:content-type:references;
        bh=eIr9ox16pG60mXGA63VFeJmZZt42DQaPF7IlJ0ZCUOE=;
        b=foz+Y5pLIC9HrOhlmgLpvXwIJifg1DUiE+e5cBAQ0ZIEG9Ja/0vxaeGm5dTqIZ5zc6
         4RBJnPYxeqz3aF9R3b8anCGzi4WntJNNeWYnVBNr76Q659TWIqE2csa2JhZXUtcm3Xy3
         HzwENF1fWLqjJetYnarPbk3khgcY5O73TDG08=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version
         :content-type:references;
        b=q/j5/L60MATk7OK5yLYd3+OISn0QWfbfGzg96fKX8RVgXstl6nUQViTAQjB6UrvKaX
         AkHDq83d9MUB0E2ZqxOEzML5y1e6bws/2L3V0IcPBQN+/ln4dAMdavvvPldluCuxIaVd
         YbcdtjrAdSo8byv0/uXloSDxFQRnz26IkMJIs=
Received: by 10.110.109.12 with SMTP id h12mr3166104tic.51.1221654697722;
        Wed, 17 Sep 2008 05:31:37 -0700 (PDT)
Received: by 10.110.57.13 with HTTP; Wed, 17 Sep 2008 05:31:37 -0700 (PDT)
Message-ID: <98fa49780809170531s2257c86cj783d28f873ed9d48@mail.gmail.com>
Date: Wed, 17 Sep 2008 20:31:37 +0800
From: tang9527@gmail.com
To: general@hadoop.apache.org
Subject: Re: [VOTE] Split Hive into a sub-project
In-Reply-To: <b23e01b50809170244k2a57f018x343c7b56eec065b6@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; 
	boundary="----=_Part_37665_27244116.1221654697709"
References: <FE98652A-B068-48DD-9619-5B7F7EE96C2B@apache.org>
	 <b23e01b50809170244k2a57f018x343c7b56eec065b6@mail.gmail.com>
X-Virus-Checked: Checked by ClamAV on apache.org

------=_Part_37665_27244116.1221654697709
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

+1

2008/9/17 Tom White <tom.e.white@gmail.com>

> +1
>
> On Mon, Sep 15, 2008 at 11:20 PM, Owen O'Malley <omalley@apache.org>
> wrote:
> > With Map/Reduce and HDFS being split up, I think we should split out Hive
> > into a sub-project also. It is already a large code-base and seems to be
> > picking up external users. Clearly, I'm +1.
> >
> > -- Owen
> >
>



-- 
jim

------=_Part_37665_27244116.1221654697709--

From general-return-51-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Sep 18 01:43:44 2008
Return-Path: <general-return-51-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 51947 invoked from network); 18 Sep 2008 01:43:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 18 Sep 2008 01:43:44 -0000
Received: (qmail 75530 invoked by uid 500); 18 Sep 2008 01:43:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75511 invoked by uid 500); 18 Sep 2008 01:43:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75500 invoked by uid 99); 18 Sep 2008 01:43:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Sep 2008 18:43:41 -0700
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS,URIBL_RHS_DOB,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of conorharty@gmail.com designates 209.85.128.190 as permitted sender)
Received: from [209.85.128.190] (HELO fk-out-0910.google.com) (209.85.128.190)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Sep 2008 01:42:43 +0000
Received: by fk-out-0910.google.com with SMTP id 26so3187161fkx.13
        for <general@hadoop.apache.org>; Wed, 17 Sep 2008 18:43:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:to
         :subject:mime-version:content-type;
        bh=uetWFyRotXxhYMkVuOC90wJ9G8l7Fd2Cd8sa7Q6keBk=;
        b=Q7yYPmPTnEjAKA/e0dn74RANoO0r4nfDchQmGeY2s9RHzWw4Tur9wjAcl6DsLB3HqK
         mEvbmRqwNRFiGvfSTl13U/MY8/j6o3ggZ2ilnK+x05+Bql3rvYh8s1bPPPI6rTWKmpkp
         vV4nwJ/bRI7rdO5JTldJtsOsKKbGPAygel4+w=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:mime-version:content-type;
        b=CRBBy8gWyy2BUzCA3TaMm7lrgx/4WpJgMr0DMxHoMZe8pOpWntNrW0FAZJ3dsJImhk
         qHp6giAlBEFofJs4rUAjJo3b4o+oVtIoJit90Q1qFaqXv4FlDweY6cK+rqW1hEVeXRFI
         M4HKWjMEAbtCEqeU19UlnCiMtS3MGhnEj/+CE=
Received: by 10.180.214.13 with SMTP id m13mr2408448bkg.75.1221702185345;
        Wed, 17 Sep 2008 18:43:05 -0700 (PDT)
Received: by 10.181.32.6 with HTTP; Wed, 17 Sep 2008 18:43:05 -0700 (PDT)
Message-ID: <861511ca0809171843y39afc45fie5781e0777c09d44@mail.gmail.com>
Date: Thu, 18 Sep 2008 12:43:05 +1100
From: "Conor Harty" <conorharty@gmail.com>
To: general@hadoop.apache.org
Subject: HBase master does not start
MIME-Version: 1.0
Content-Type: multipart/alternative; 
	boundary="----=_Part_842_15776858.1221702185354"
X-Virus-Checked: Checked by ClamAV on apache.org

------=_Part_842_15776858.1221702185354
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

The problem we have is, if we stop hbase and hadoop and then start hadoop
again and try to start hbase, hbase master doesnt start up. Here is the
error we get:


ERROR master.HMaster: Can not start master
java.net.BindException: Problem binding to /127.0.0.1:60000
    at org.apache.hadoop.ipc.Server.bind(Server.java:175)
    at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:234)
    at org.apache.hadoop.ipc.Server.<init>(Server.java:960)
    at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.<init>(HbaseRPC.java:457)
    at org.apache.hadoop.hbase.ipc.HbaseRPC.getServer(HbaseRPC.java:419)
    at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:220)
    at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:148)
    at
org.apache.hadoop.hbase.LocalHBaseCluster.<init>(LocalHBaseCluster.java:94)
    at
org.apache.hadoop.hbase.LocalHBaseCluster.<init>(LocalHBaseCluster.java:77)
    at org.apache.hadoop.hbase.master.HMaster.doMain(HMaster.java:794)
    at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:832)

------=_Part_842_15776858.1221702185354--

From general-return-52-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Sep 18 17:56:50 2008
Return-Path: <general-return-52-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 80874 invoked from network); 18 Sep 2008 17:56:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 18 Sep 2008 17:56:50 -0000
Received: (qmail 9265 invoked by uid 500); 18 Sep 2008 17:56:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 9119 invoked by uid 500); 18 Sep 2008 17:56:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 9097 invoked by uid 99); 18 Sep 2008 17:56:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Sep 2008 10:56:39 -0700
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Sep 2008 17:55:38 +0000
Received: from [10.0.1.3] (snvvpn1-10-72-73-c38.corp.yahoo.com [10.72.73.38])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id m8IHsKjP032114;
	Thu, 18 Sep 2008 10:54:20 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:content-type:
	content-transfer-encoding:mime-version:subject:date:x-mailer;
	b=nmnXdC8bkz2NI9perKDfYgGpyzOzypHTfh8s1CJAERWR7zkUqTEsrYPRmHtU5OhR
Message-Id: <531E543B-B733-4405-B95C-B0C5D3E83974@yahoo-inc.com>
From: Nigel Daley <ndaley@yahoo-inc.com>
To: core-user@hadoop.apache.org, general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v928.1)
Subject: [ANNOUNCE] Hadoop release 0.18.1 available
Date: Thu, 18 Sep 2008 10:54:20 -0700
X-Mailer: Apple Mail (2.928.1)
X-Virus-Checked: Checked by ClamAV on apache.org

Release 0.18.1 fixes 9 critical bugs in 0.18.0.

For Hadoop release details and downloads, visit:
http://hadoop.apache.org/core/releases.html

Hadoop 0.18.1 Release Notes are at
http://hadoop.apache.org/core/docs/r0.18.1/releasenotes.html

Thanks to all who contributed to this release!

Nigel

From general-return-53-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Sep 20 23:29:44 2008
Return-Path: <general-return-53-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 69189 invoked from network); 20 Sep 2008 23:29:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 20 Sep 2008 23:29:43 -0000
Received: (qmail 72186 invoked by uid 500); 20 Sep 2008 23:29:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72156 invoked by uid 500); 20 Sep 2008 23:29:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72145 invoked by uid 99); 20 Sep 2008 23:29:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 20 Sep 2008 16:29:40 -0700
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of owen.omalley@gmail.com designates 209.85.217.13 as permitted sender)
Received: from [209.85.217.13] (HELO mail-gx0-f13.google.com) (209.85.217.13)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 20 Sep 2008 23:28:41 +0000
Received: by gxk6 with SMTP id 6so1828540gxk.5
        for <general@hadoop.apache.org>; Sat, 20 Sep 2008 16:28:13 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:sender
         :to:subject:in-reply-to:mime-version:content-type:references
         :x-google-sender-auth;
        bh=soz0G3k1DnjIUVC/y21o0s2ceh6mTjm5Si4NtOKRa3s=;
        b=xa8aG3tc7acqz3XRFhcuWwRHqDSi/hfoVZvBmKcv6sN1tU5wZeL4UdMjLTJq2gPuzI
         3v1dSOcn3jVCHb7Yo1gMMsgn8wIwuasCYdJXksI8o3XoYu1JYK43xtTLt8/4XvOHFbNt
         czjZRwHN+YhYRbN3Gn3X4QHfUNCmjcJP1bqBg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:sender:to:subject:in-reply-to:mime-version
         :content-type:references:x-google-sender-auth;
        b=N7fETNX4B3U8lLT18OcrlR+XOOX5CLd7TR2GbMvp0Qm5f5nTXQGblwQzEKf0xrVVja
         +2ZAfJFdDqM5frNPhmZUkJXbbxoYXXEuwvFLKA03tX69k+T61BQINZUpqsyE/rffe0Ih
         DgW1kaQpDzTYqKMDVzeAk6c+KWiREZZk47f5Q=
Received: by 10.151.9.1 with SMTP id m1mr2218974ybi.223.1221953293570;
        Sat, 20 Sep 2008 16:28:13 -0700 (PDT)
Received: by 10.151.84.6 with HTTP; Sat, 20 Sep 2008 16:28:13 -0700 (PDT)
Message-ID: <5f7f740b0809201628r5c5f8d22re9dcfd101557d1ed@mail.gmail.com>
Date: Sat, 20 Sep 2008 16:28:13 -0700
From: "Owen O'Malley" <omalley@apache.org>
Sender: owen.omalley@gmail.com
To: general@hadoop.apache.org
Subject: Re: [VOTE] Split Hive into a sub-project
In-Reply-To: <FE98652A-B068-48DD-9619-5B7F7EE96C2B@apache.org>
MIME-Version: 1.0
Content-Type: multipart/alternative; 
	boundary="----=_Part_62953_29054220.1221953293567"
References: <FE98652A-B068-48DD-9619-5B7F7EE96C2B@apache.org>
X-Google-Sender-Auth: c57489c3ab44534b
X-Virus-Checked: Checked by ClamAV on apache.org

------=_Part_62953_29054220.1221953293567
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

On Mon, Sep 15, 2008 at 3:20 PM, Owen O'Malley <omalley@apache.org> wrote:

> With Map/Reduce and HDFS being split up, I think we should split out Hive
> into a sub-project also. It is already a large code-base and seems to be
> picking up external users. Clearly, I'm +1.


With 7 +1 votes from PMC members and no -1 votes, the vote passes.

------=_Part_62953_29054220.1221953293567--

From general-return-54-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 22 17:54:11 2008
Return-Path: <general-return-54-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 58152 invoked from network); 22 Sep 2008 17:54:10 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 22 Sep 2008 17:54:10 -0000
Received: (qmail 30280 invoked by uid 500); 22 Sep 2008 17:54:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30264 invoked by uid 500); 22 Sep 2008 17:54:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30253 invoked by uid 99); 22 Sep 2008 17:54:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Sep 2008 10:54:07 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [63.203.238.117] (HELO dns.duboce.net) (63.203.238.117)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Sep 2008 17:53:07 +0000
Received: by dns.duboce.net (Postfix, from userid 1008)
	id F2D4DC51B; Mon, 22 Sep 2008 09:19:26 -0700 (PDT)
X-Spam-Checker-Version: SpamAssassin 3.1.4 (2006-07-26) on dns.duboce.net
X-Spam-Level: 
Received: from durruti.local (durruti.desk.hq.powerset.com [208.84.6.91])
	by dns.duboce.net (Postfix) with ESMTP id CCD64C256
	for <general@hadoop.apache.org>; Mon, 22 Sep 2008 09:19:21 -0700 (PDT)
Message-ID: <48D7DB79.20509@duboce.net>
Date: Mon, 22 Sep 2008 10:52:57 -0700
From: stack <stack@duboce.net>
User-Agent: Thunderbird 2.0.0.16 (Macintosh/20080707)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: HBase master does not start
References: <861511ca0809171843y39afc45fie5781e0777c09d44@mail.gmail.com>
In-Reply-To: <861511ca0809171843y39afc45fie5781e0777c09d44@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org
X-Old-Spam-Status: No, score=-3.3 required=5.0 tests=AWL,BAYES_00,
	FORGED_RCVD_HELO autolearn=ham version=3.1.4

Conor:

Something must already be running on port 60000 on localhost -- probably 
an old instance of hbase.  It probably wasn't shut down properly.  Kill 
it if you have to.

St.Ack

P.S.  A better place for these questions is the hbase mailing lists.

Conor Harty wrote:
> The problem we have is, if we stop hbase and hadoop and then start hadoop
> again and try to start hbase, hbase master doesnt start up. Here is the
> error we get:
>
>
> ERROR master.HMaster: Can not start master
> java.net.BindException: Problem binding to /127.0.0.1:60000
>     at org.apache.hadoop.ipc.Server.bind(Server.java:175)
>     at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:234)
>     at org.apache.hadoop.ipc.Server.<init>(Server.java:960)
>     at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.<init>(HbaseRPC.java:457)
>     at org.apache.hadoop.hbase.ipc.HbaseRPC.getServer(HbaseRPC.java:419)
>     at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:220)
>     at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:148)
>     at
> org.apache.hadoop.hbase.LocalHBaseCluster.<init>(LocalHBaseCluster.java:94)
>     at
> org.apache.hadoop.hbase.LocalHBaseCluster.<init>(LocalHBaseCluster.java:77)
>     at org.apache.hadoop.hbase.master.HMaster.doMain(HMaster.java:794)
>     at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:832)
>
>   


From general-return-472-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 01 16:47:54 2009
Return-Path: <general-return-472-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 41142 invoked from network); 1 Sep 2009 16:47:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Sep 2009 16:47:54 -0000
Received: (qmail 51466 invoked by uid 500); 1 Sep 2009 16:47:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51312 invoked by uid 500); 1 Sep 2009 16:47:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51288 invoked by uid 99); 1 Sep 2009 16:47:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 16:47:53 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [207.126.228.150] (HELO rsmtp2.corp.yahoo.com) (207.126.228.150)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 16:47:42 +0000
Received: from [172.21.148.25] (nat-dip10.cfw-b-gci.corp.yahoo.com [209.131.62.145])
	(authenticated bits=0)
	by rsmtp2.corp.yahoo.com (8.13.8/8.13.8/y.rout) with ESMTP id n81GlD46022051
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Tue, 1 Sep 2009 09:47:14 -0700 (PDT)
Message-ID: <4A9D5011.3050206@apache.org>
Date: Tue, 01 Sep 2009 09:47:13 -0700
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org, private@hadoop.apache.org
Subject: [Fwd: [VOTE] Release ZooKeeper 3.2.1 (candidate 0)]
Content-Type: multipart/mixed;
 boundary="------------070002090205000206020800"
X-Virus-Checked: Checked by ClamAV on apache.org

--------------070002090205000206020800
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hadoop PMC,

Please test and vote on this release in zookeeper-dev list.

Thanks,

Patrick

--------------070002090205000206020800
Content-Type: message/rfc822;
 name="[VOTE] Release ZooKeeper 3.2.1 (candidate 0).eml"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename="[VOTE] Release ZooKeeper 3.2.1 (candidate 0).eml"

Message-ID: <4A982047.9070206@apache.org>
Date: Fri, 28 Aug 2009 11:21:59 -0700
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: "zookeeper-dev@hadoop.apache.org" <zookeeper-dev@hadoop.apache.org>
Subject: [VOTE] Release ZooKeeper 3.2.1 (candidate 0)
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

I've created a candidate build for ZooKeeper 3.2.1. This is a bug fix 
release addressing a number of significant issues -- see the release 
notes for details.

*** Please download, test and VOTE before the
*** vote closes EOD on Wednesday, September 2.***

http://people.apache.org/~phunt/zookeeper-3.2.1-candidate-0/

Should we release this?

Patrick



--------------070002090205000206020800--

From general-return-473-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Sep 02 22:04:47 2009
Return-Path: <general-return-473-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 40990 invoked from network); 2 Sep 2009 22:04:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Sep 2009 22:04:47 -0000
Received: (qmail 34291 invoked by uid 500); 2 Sep 2009 22:04:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34228 invoked by uid 500); 2 Sep 2009 22:04:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34218 invoked by uid 99); 2 Sep 2009 22:04:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Sep 2009 22:04:46 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of hkumar.arora@gmail.com designates 209.85.223.174 as permitted sender)
Received: from [209.85.223.174] (HELO mail-iw0-f174.google.com) (209.85.223.174)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Sep 2009 22:04:38 +0000
Received: by iwn4 with SMTP id 4so517224iwn.3
        for <general@hadoop.apache.org>; Wed, 02 Sep 2009 15:04:18 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=TxlX7oqsVHVueWhLsuj3uQh1YN+mkcYZRGWyI9ar/R8=;
        b=kWrsReAXysoGFJS5MtW2ANi82u1mqX8AXcka+K+IzgmzVpe1IQobup+5BX1ZbZjQEs
         Zz4/KUdFqXRjbhq94ooT39pKyTDp/ZKw+2NaOykORfIjk/LGSmFA1uvpKVGc9Brjddi8
         0qhHmYlcTkn4Hotiv4tquKoF1g6Wday/CeQ4M=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=nJvg24HXnyhOLI8tg+snh/EMP8soV7nn+4TtG/x7ohVgSAtFVBWhkvUptCSS+kI/7J
         K1jgU6tEf6EmiEN/65rqhmA0f/2rwXpxh51tlnA3Ym5ZBQT+w3N4aQXbXFEFdPdb4Tkr
         cpmZvhVGm2Oz1uM498RGNBbMNRkUjZLSwd1YY=
MIME-Version: 1.0
Received: by 10.231.40.227 with SMTP id l35mr8736536ibe.28.1251929058037; Wed, 
	02 Sep 2009 15:04:18 -0700 (PDT)
Date: Thu, 3 Sep 2009 07:04:15 +0900
Message-ID: <e497591c0909021504m77e5e914md2df2c9b1dc3fb7a@mail.gmail.com>
Subject: Hadoop executing a custom WRITABLE type
From: Harshit Kumar <hkumar.arora@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175cdf50c9e9eb04729f727d
X-Virus-Checked: Checked by ClamAV on apache.org

--0015175cdf50c9e9eb04729f727d
Content-Type: text/plain; charset=UTF-8

Hi

I am executing a custom writable type called as Duo. It is a class that
implement the Writable Interface.

input is a text file, in which each record consist of 3 words. for ex:
hello come here
Where are you
How are you

In the Driver class, Mapper makes the first word as key, and create an
object of type Duo from the rest of 2 words.

Here is the code below


import java.io.IOException;
import java.util.*;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.mapred.lib.NullOutputFormat;


   public class WordCount {

    public static class Map extends MapReduceBase implements
Mapper<LongWritable, Text, Text, Duo> {
      private Text word = new Text();

       public void map(LongWritable key, Text value, OutputCollector<Text,
Duo> output, Reporter reporter) throws IOException {
           String line = value.toString();
           StringTokenizer tokenizer = new StringTokenizer(line);
           while (tokenizer.hasMoreTokens()) {
               word.set(tokenizer.nextToken());
               output.collect(word, new Duo("kumar","kumar"));
           }
       }
    }

    public static class Reduce extends MapReduceBase implements
Reducer<Text, Duo, Text, IntWritable> {

       public void reduce(Text key, Iterator<Duo> values,
OutputCollector<Text, IntWritable> output, Reporter reporter) throws
IOException {
       Configuration conf = new Configuration();
       FileSystem fs = FileSystem.get(conf);
       Path subjectFile = new Path(key.toString());
       FSDataOutputStream out = fs.create(subjectFile);
           while (values.hasNext()) {
           out.writeChars(key.toString());
           Duo d = values.next();
           out.writeChars(d.getProperty());
           out.writeChars(d.getObject());
           }
       }
    }

    public static void main(String[] args) throws Exception {
       System.out.println("1");
     JobConf conf = new JobConf(WordCount.class);
       conf.setJobName("wordcount");

       conf.setOutputKeyClass(Text.class);
       conf.setOutputValueClass(Duo.class);

       conf.setMapperClass(Map.class);
       conf.setCombinerClass(Reduce.class);
       conf.setReducerClass(Reduce.class);

       conf.setInputFormat(TextInputFormat.class);
       conf.setOutputFormat(NullOutputFormat.class);

       FileInputFormat.setInputPaths(conf, new Path(args[0]));
       FileOutputFormat.setOutputPath(conf, new Path(args[1]));
       System.out.println("2");
       JobClient.runJob(conf);
    }

   }


-----------------------------------
code for Duo class is as follows

import org.apache.hadoop.io.*;
import java.io.*;

//public class Duo implements WritableComparable<Duo>{
public class Duo implements Writable{
public String property;
public String object;
 public Duo(){
set(new String(),new String());
}
public Duo(String p, String o){
set(p,o);
}
 public void set(String p, String o){
property=p;
object=o;
}
 public String getProperty(){
return property;
}
 public String getObject(){
return object;
}
 public void write(DataOutput out) throws IOException {
out.writeChars(property);
out.writeChars(object);
}
 public void readFields(DataInput in) throws IOException {
property = in.readLine();
object = in.readLine();
}
 public int hashCode(){
return property.hashCode()* 163 +object.hashCode();
}
 public boolean equals(Object other){
if (other instanceof Duo){
Duo od = (Duo) other;
return property.equals(od.property) && object.equals(od);
}
return false;
}
 public String toString(){
return property + "\t" + object;
}
   public int compareTo(Duo other){
if (property.compareTo(other.property)==0)
return object.compareTo(other.object);
else
return property.compareTo(other.property);
}
 }

When I execute the driver program, the following error is displayed. Please
help me explain this problem and if possible suggest some solution also.

1
2
09/09/03 06:40:41 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
09/09/03 06:40:42 INFO mapred.FileInputFormat: Total input paths to process
: 1
09/09/03 06:40:42 INFO mapred.FileInputFormat: Total input paths to process
: 1
09/09/03 06:40:43 INFO mapred.JobClient: Running job: job_200909021822_0046
09/09/03 06:40:44 INFO mapred.JobClient:  map 0% reduce 0%
09/09/03 06:40:50 INFO mapred.JobClient: Task Id :
attempt_200909021822_0046_m_000000_0, Status : FAILED
java.lang.RuntimeException: java.lang.RuntimeException:
java.lang.ClassNotFoundException: Duo
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:680)
at org.apache.hadoop.mapred.JobConf.getOutputValueClass(JobConf.java:681)
at org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:567)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:383)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:193)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2198)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Duo
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:648)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:672)
... 5 more
Caused by: java.lang.ClassNotFoundException: Duo
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:628)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:646)
... 6 more


I appreciate your efforts to read this message

Thanks
H. Kumar

--0015175cdf50c9e9eb04729f727d--

From general-return-474-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Sep 03 02:53:40 2009
Return-Path: <general-return-474-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 8914 invoked from network); 3 Sep 2009 02:53:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Sep 2009 02:53:40 -0000
Received: (qmail 37962 invoked by uid 500); 3 Sep 2009 02:53:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37870 invoked by uid 500); 3 Sep 2009 02:53:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37860 invoked by uid 99); 3 Sep 2009 02:53:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Sep 2009 02:53:38 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of deepak.halale@gmail.com designates 209.85.219.208 as permitted sender)
Received: from [209.85.219.208] (HELO mail-ew0-f208.google.com) (209.85.219.208)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Sep 2009 02:53:30 +0000
Received: by ewy4 with SMTP id 4so1443833ewy.36
        for <general@hadoop.apache.org>; Wed, 02 Sep 2009 19:53:09 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=pXuahYOcbfIynxQQTqY9leit5A2fvZ9JWrVorrtnowg=;
        b=j+O2IfDnpBvc69Y3GRgcjpPxOjf7Qm85G/Wvks8Dg0pXgRA3iR0NXODpl+2XWTUDhI
         h+p7GnZr79Y+ic+v5nU6OLdy12saxxIzVpVVIF1c3qlUUD+z8EMAcRAnHdGRlMepb5YW
         PAUH26+T5QAGwRLsDWfWAGm0AYI60X0zJbFW0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=blr72X6cGPXqvRO+WaHpLJOKu2nJzub/TEDNrHewXpRMcXUQp4Ks7ttLd/x38WJNY4
         lK+2yUpHkJlmqsC/LhYLYgrCWbBWZ2sEoBo/qMhGnwW4MviCSbBdHm8TmjwpcVVwwOzw
         Oq0xnFec5thZpCzXjBMk1JCjFw1el7GWfVERA=
MIME-Version: 1.0
Received: by 10.216.88.3 with SMTP id z3mr1618288wee.94.1251946389312; Wed, 02 
	Sep 2009 19:53:09 -0700 (PDT)
Date: Wed, 2 Sep 2009 22:53:09 -0400
Message-ID: <15678300909021953j37a386d8v1e1eec3dfc06f80e@mail.gmail.com>
Subject: hadoop configuration help
From: Deepak Halale <deepak.halale@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6db2c95d017170472a37b60
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6db2c95d017170472a37b60
Content-Type: text/plain; charset=ISO-8859-1

Hi,
I am trying to configure Hadoop distributed environment .
I downloaded hadoop RPM from cloudera, installed it on MACHINE1 (Ubuntu
Linux) and MACHINE2 (Fedora Linux)

Installed Hadoop RPM on MACHINE2 and Debian Packages on MACHINE1

edited etc/hadoop/conf.pseudo/masters file on MACHINE1 with MACHINE2 name in
masters file

added MACHINE name in salves file of MACHINE2  under
etc/hadoop/conf.my_cluster/slaves  (conf.my_cluster is the alternatives on
MACHINE2)

was able to SSN from MACHINE2 to MACHINE1

I am missing something?

getting the following error while running mapreduce job from MACHINE2

09/09/02 22:42:15 INFO dfs.DFSClient: org.apache.hadoop.ipc.RemoteException:
java.io.IOException: File
/var/lib/hadoop/cache/hadoop/mapred/system/job_200909021816_0003/job.jar
could only be replicated to 0 nodes, instead of 1
        at
org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1123)
        at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:330)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:890)
        at org.apache.hadoop.ipc.Client.call(Client.java:716)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
        at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
        at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
        at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
        at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2450)
        at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2333)
        at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1800(DFSClient.java:1745)
        at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1922)



Thanks
Deepak

--0016e6db2c95d017170472a37b60--

From general-return-475-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Sep 03 04:02:38 2009
Return-Path: <general-return-475-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27107 invoked from network); 3 Sep 2009 04:02:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Sep 2009 04:02:38 -0000
Received: (qmail 91958 invoked by uid 500); 3 Sep 2009 04:02:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91820 invoked by uid 500); 3 Sep 2009 04:02:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91796 invoked by uid 99); 3 Sep 2009 04:02:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Sep 2009 04:02:36 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Sep 2009 04:02:27 +0000
Received: from [10.72.168.254] (snvvpn4-10-72-168-c254.hq.corp.yahoo.com [10.72.168.254])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n83420aw019767;
	Wed, 2 Sep 2009 21:02:00 -0700 (PDT)
Message-ID: <4A9F3FB8.8070807@apache.org>
Date: Wed, 02 Sep 2009 21:02:00 -0700
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org, private@hadoop.apache.org
Subject: [Fwd: [Fwd: [VOTE] Release ZooKeeper 3.2.1 (candidate 0)]]
Content-Type: multipart/mixed;
 boundary="------------040207050502080009000106"
X-Virus-Checked: Checked by ClamAV on apache.org

--------------040207050502080009000106
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hadoop PMC,

Please test and vote on this release in zookeeper-dev list.

Patrick

--------------040207050502080009000106
Content-Type: message/rfc822;
 name="[Fwd: [VOTE] Release ZooKeeper 3.2.1 (candidate 0)].eml"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename*0="[Fwd: [VOTE] Release ZooKeeper 3.2.1 (candidate 0)].eml"

Delivered-To: phunt1@gmail.com
Received: by 10.100.196.16 with SMTP id t16cs156791anf;
        Tue, 1 Sep 2009 09:47:54 -0700 (PDT)
Received: by 10.115.101.10 with SMTP id d10mr4779768wam.61.1251823673533;
        Tue, 01 Sep 2009 09:47:53 -0700 (PDT)
Return-Path: <phunt@apache.org>
Received: from minotaur.apache.org (minotaur.apache.org [140.211.11.9])
        by mx.google.com with SMTP id 41si17521378pzk.4.2009.09.01.09.47.53;
        Tue, 01 Sep 2009 09:47:53 -0700 (PDT)
Received-SPF: pass (google.com: best guess record for domain of phunt@apache.org designates 140.211.11.9 as permitted sender) client-ip=140.211.11.9;
Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of phunt@apache.org designates 140.211.11.9 as permitted sender) smtp.mail=phunt@apache.org
Received: (qmail 41049 invoked by uid 2923); 1 Sep 2009 16:47:53 -0000
Delivered-To: phunt@minotaur.apache.org
Received: (qmail 41044 invoked from network); 1 Sep 2009 16:47:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Sep 2009 16:47:53 -0000
Received: (qmail 51295 invoked by uid 500); 1 Sep 2009 16:47:53 -0000
Delivered-To: apmail-phunt@apache.org
Received: (qmail 51288 invoked by uid 99); 1 Sep 2009 16:47:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 16:47:53 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [207.126.228.150] (HELO rsmtp2.corp.yahoo.com) (207.126.228.150)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 16:47:42 +0000
Received: from [172.21.148.25] (nat-dip10.cfw-b-gci.corp.yahoo.com [209.131.62.145])
	(authenticated bits=0)
	by rsmtp2.corp.yahoo.com (8.13.8/8.13.8/y.rout) with ESMTP id n81GlD46022051
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Tue, 1 Sep 2009 09:47:14 -0700 (PDT)
Message-ID: <4A9D5011.3050206@apache.org>
Date: Tue, 01 Sep 2009 09:47:13 -0700
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org, private@hadoop.apache.org
Subject: [Fwd: [VOTE] Release ZooKeeper 3.2.1 (candidate 0)]
Content-Type: multipart/mixed;
 boundary="------------070002090205000206020800"
X-Virus-Checked: Checked by ClamAV on apache.org

This is a multi-part message in MIME format.
--------------070002090205000206020800
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hadoop PMC,

Please test and vote on this release in zookeeper-dev list.

Thanks,

Patrick

--------------070002090205000206020800
Content-Type: message/rfc822;
 name="[VOTE] Release ZooKeeper 3.2.1 (candidate 0).eml"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename="[VOTE] Release ZooKeeper 3.2.1 (candidate 0).eml"

Message-ID: <4A982047.9070206@apache.org>
Date: Fri, 28 Aug 2009 11:21:59 -0700
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: "zookeeper-dev@hadoop.apache.org" <zookeeper-dev@hadoop.apache.org>
Subject: [VOTE] Release ZooKeeper 3.2.1 (candidate 0)
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

I've created a candidate build for ZooKeeper 3.2.1. This is a bug fix 
release addressing a number of significant issues -- see the release 
notes for details.

*** Please download, test and VOTE before the
*** vote closes EOD on Wednesday, September 2.***

http://people.apache.org/~phunt/zookeeper-3.2.1-candidate-0/

Should we release this?

Patrick



--------------070002090205000206020800--

--------------040207050502080009000106--

From general-return-476-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Sep 03 20:34:56 2009
Return-Path: <general-return-476-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 28959 invoked from network); 3 Sep 2009 20:34:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Sep 2009 20:34:56 -0000
Received: (qmail 69252 invoked by uid 500); 3 Sep 2009 20:34:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69119 invoked by uid 500); 3 Sep 2009 20:34:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69097 invoked by uid 99); 3 Sep 2009 20:34:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Sep 2009 20:34:54 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 209.85.221.204 as permitted sender)
Received: from [209.85.221.204] (HELO mail-qy0-f204.google.com) (209.85.221.204)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Sep 2009 20:34:47 +0000
Received: by qyk42 with SMTP id 42so327886qyk.10
        for <multiple recipients>; Thu, 03 Sep 2009 13:34:25 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:message-id:subject:from:to:cc
         :content-type;
        bh=zLsdp+3OXluzOFPROHBb246fx/suNDVcSxRAx6/aqJw=;
        b=WyTWDeSNR1i77vTDHF4HZxXaGpy6ahQtuP9SucPLKfS2XKuHEodu5udMogWWSWXqN3
         il0sybeOUG/cFxKKUhb2p42IFv5NWX0qCS64I3XjPo79QpEH9NWzUzTdShwTYpbW6efU
         nMsGae9FqczLHfSUVn0WkTMpJXwhzyTHk96zI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:cc:content-type;
        b=J3mOFKpD0MUOEw6howYLtD8HZrGzEezm83vL/c+/nJZ20TQOrmPAzJLiU6LjPAti8U
         zUSCKilryfY6if6rv81pRcDzUGbxKENHYGiZVslxNHf6BRyXYPjW3J5RQKdINy5yuAfK
         RoQjRj4h/wRhmc/tTV/07ingqqASIK+0nPzuY=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
Received: by 10.229.116.140 with SMTP id m12mr3136179qcq.54.1252010064428; 
	Thu, 03 Sep 2009 13:34:24 -0700 (PDT)
In-Reply-To: <4A9F3FB8.8070807@apache.org>
References: <4A9F3FB8.8070807@apache.org>
Date: Thu, 3 Sep 2009 13:34:24 -0700
X-Google-Sender-Auth: aec4bdbffc7e20e6
Message-ID: <7c962aed0909031334x1796077cq97ce52c0202d5523@mail.gmail.com>
Subject: Re: [Fwd: [Fwd: [VOTE] Release ZooKeeper 3.2.1 (candidate 0)]]
From: stack <stack@duboce.net>
To: general@hadoop.apache.org
Cc: private@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00c09fa216fe253f940472b24fd2
X-Virus-Checked: Checked by ClamAV on apache.org

--00c09fa216fe253f940472b24fd2
Content-Type: text/plain; charset=ISO-8859-1

+1

I downloaded it, reviewed documentation and then put it in place of the
patched ZK in hbase (This candidate includes the patch that made us patch
our ZK in the first place).   I put up a load on a small cluster with a
quorum of three servers.  This candidate runs as good/bad as the patched ZK
we ship with.

St.Ack


On Wed, Sep 2, 2009 at 9:02 PM, Patrick Hunt <phunt@apache.org> wrote:

> Hadoop PMC,
>
> Please test and vote on this release in zookeeper-dev list.
>
> Patrick
>
> Hadoop PMC,
>
> Please test and vote on this release in zookeeper-dev list.
>
> Thanks,
>
> Patrick
>
> I've created a candidate build for ZooKeeper 3.2.1. This is a bug fix
> release addressing a number of significant issues -- see the release notes
> for details.
>
> *** Please download, test and VOTE before the
> *** vote closes EOD on Wednesday, September 2.***
>
> http://people.apache.org/~phunt/zookeeper-3.2.1-candidate-0/<http://people.apache.org/%7Ephunt/zookeeper-3.2.1-candidate-0/>
>
> Should we release this?
>
> Patrick
>
>
>
>

--00c09fa216fe253f940472b24fd2--

From general-return-477-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Sep 04 16:17:31 2009
Return-Path: <general-return-477-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16454 invoked from network); 4 Sep 2009 16:17:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Sep 2009 16:17:30 -0000
Received: (qmail 79717 invoked by uid 500); 4 Sep 2009 16:17:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79646 invoked by uid 500); 4 Sep 2009 16:17:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 79636 invoked by uid 99); 4 Sep 2009 16:17:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Sep 2009 16:17:29 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_SOFTFAIL
X-Spam-Check-By: apache.org
Received-SPF: softfail (athena.apache.org: transitioning domain of benjamin.cotton@lehman.com does not designate 68.142.237.108 as permitted sender)
Received: from [68.142.237.108] (HELO n1.bullet.mail.re3.yahoo.com) (68.142.237.108)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 04 Sep 2009 16:17:19 +0000
Received: from [68.142.237.89] by n1.bullet.mail.re3.yahoo.com with NNFMP; 04 Sep 2009 16:16:57 -0000
Received: from [66.196.97.145] by t5.bullet.re3.yahoo.com with NNFMP; 04 Sep 2009 16:16:57 -0000
Received: from [127.0.0.1] by omp203.mail.re3.yahoo.com with NNFMP; 04 Sep 2009 16:16:57 -0000
X-Yahoo-Newman-Id: 538781.30740.bm@omp203.mail.re3.yahoo.com
Received: (qmail 32061 invoked from network); 4 Sep 2009 16:16:57 -0000
Received: from unknown (HELO ?192.168.1.116?) (Benjamin.Cotton@216.27.139.111 with plain)
  by smtp104.plus.mail.re1.yahoo.com with SMTP; 4 Sep 2009 16:16:57 -0000
X-Yahoo-SMTP: Hfpa_R6swBDM9C7.Z6BBpwK7KXuKR3g-
X-YMail-OSG: 2FkPHiYVM1lyL2QuIcm_tjMyZGiCl6P3Y9gRreY91GNj6sM5j0AxUFnN7gfKimfqS.dYE.ckZ_hnxapC5xk_9C7S13BoOvnBlRx.1k2Hgm3szKng6aIMwLzZngKOplSTTpkpjdm8GjHExUsQicEfPcJSe1NYjzUR.S4uiUe4TXSv6G3lSs9ICvEX2i7PAS_Mc_qe6o4YKyOs8vnB3gYqPAu7TYvFqL23_1EIgxBl5_XXHFzEwuePg3vFPI_CDh8QzmVkrrneuviUEb62lTsPU808mMXq4vGRc9ZanaxPi1xNjfAaRfWnJ6bd_efC9IiuHOSGng--
X-Yahoo-Newman-Property: ymail-3
Message-ID: <4AA13D78.30304@lehman.com>
Date: Fri, 04 Sep 2009 12:16:56 -0400
From: "benjamin.cotton@lehman.com" <Benjamin.Cotton@lehman.com>
User-Agent: Thunderbird 2.0.0.23 (Windows/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: drivers to bridge familiar SQL queries to Hadoop MapReduce internals?
Content-Type: multipart/alternative;
 boundary="------------050208040208000706070501"
X-Virus-Checked: Checked by ClamAV on apache.org

--------------050208040208000706070501
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit


I am brand new to Hadoop and have a very newbie question:  Is it a 
Hadoop community priority to  build drivers (or layers of drivers) that 
will help bridge simple, familiar SQL queries to Hadoop MapReduce 
internals  - liberating the application query developer from having to 
necessarily learn Hadoop-specific technologies, APIs, and tactics?

E.g. in   the "Hadoop - The Definitive Guide" initial example, I would 
like to STILL just be able to write

Select avg(weatherStationTable.airTemp), max(weatherStationTable.airTemp)
from   weatherStationTable
group by  weatherStationTable.year

and depend on some Driver (or layer of Drivers) to bridge that familiar 
SQL relational query to a Hadoop MapReduce job that is deployed across 
the HDFS (or other  Hadoop-specific data hostng layer) to  execute in 
Hadoop and return my result.

 is the notion of this potential capability off-the mark re: current 
Hadoop community development priorities?

--------------050208040208000706070501--


From general-return-478-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Sep 04 16:35:43 2009
Return-Path: <general-return-478-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23150 invoked from network); 4 Sep 2009 16:35:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Sep 2009 16:35:42 -0000
Received: (qmail 22067 invoked by uid 500); 4 Sep 2009 16:35:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22014 invoked by uid 500); 4 Sep 2009 16:35:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22004 invoked by uid 99); 4 Sep 2009 16:35:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Sep 2009 16:35:42 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.220.225] (HELO mail-fx0-f225.google.com) (209.85.220.225)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Sep 2009 16:35:31 +0000
Received: by fxm25 with SMTP id 25so407819fxm.29
        for <general@hadoop.apache.org>; Fri, 04 Sep 2009 09:35:10 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.86.249.22 with SMTP id w22mr5173625fgh.1.1252082109736; Fri, 
	04 Sep 2009 09:35:09 -0700 (PDT)
In-Reply-To: <4AA13D78.30304@lehman.com>
References: <4AA13D78.30304@lehman.com>
From: Philip Zeyliger <philip@cloudera.com>
Date: Fri, 4 Sep 2009 09:34:49 -0700
Message-ID: <15da8a100909040934l6fd1ad66uee2ce3f12cadf92@mail.gmail.com>
Subject: Re: drivers to bridge familiar SQL queries to Hadoop MapReduce 
	internals?
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Benjamin,

This is actually very much on the mark.

Take a look at the Hive project -- http://hadoop.apache.org/hive/ ,
also video at http://www.cloudera.com/hadoop-training-hive-introduction.
 Hive is a SQL-like interface developed initially at Facebook for
exactly that.  Pig is also working on something similar -- see
http://issues.apache.org/jira/browse/PIG-824.

Cheers,

-- Philip



On Fri, Sep 4, 2009 at 9:16 AM,
benjamin.cotton@lehman.com<Benjamin.Cotton@lehman.com> wrote:
>
> I am brand new to Hadoop and have a very newbie question: =A0Is it a Hado=
op
> community priority to =A0build drivers (or layers of drivers) that will h=
elp
> bridge simple, familiar SQL queries to Hadoop MapReduce internals =A0-
> liberating the application query developer from having to necessarily lea=
rn
> Hadoop-specific technologies, APIs, and tactics?
>
> E.g. in =A0 the "Hadoop - The Definitive Guide" initial example, I would =
like
> to STILL just be able to write
>
> Select avg(weatherStationTable.airTemp), max(weatherStationTable.airTemp)
> from =A0 weatherStationTable
> group by =A0weatherStationTable.year
>
> and depend on some Driver (or layer of Drivers) to bridge that familiar S=
QL
> relational query to a Hadoop MapReduce job that is deployed across the HD=
FS
> (or other =A0Hadoop-specific data hostng layer) to =A0execute in Hadoop a=
nd
> return my result.
>
> is the notion of this potential capability off-the mark re: current Hadoo=
p
> community development priorities?
>

From general-return-479-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Sep 04 17:29:16 2009
Return-Path: <general-return-479-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54658 invoked from network); 4 Sep 2009 17:29:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Sep 2009 17:29:16 -0000
Received: (qmail 78693 invoked by uid 500); 4 Sep 2009 17:29:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78607 invoked by uid 500); 4 Sep 2009 17:29:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78578 invoked by uid 99); 4 Sep 2009 17:29:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Sep 2009 17:29:13 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 04 Sep 2009 17:29:11 +0000
Received: (qmail 54510 invoked by uid 99); 4 Sep 2009 17:28:50 -0000
Received: from localhost.apache.org (HELO [192.168.168.101]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Sep 2009 17:28:50 +0000
Message-ID: <4AA14E51.2020409@apache.org>
Date: Fri, 04 Sep 2009 10:28:49 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: [Fwd: [VOTE] Avro release 1.1.0 (candidate 0)]
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

FYI, I just started a release vote on avro-dev.

If you have time, please evaluate the candidate and vote there.

Thanks,

Doug

-------- Original Message --------
Subject: [VOTE] Avro release 1.1.0 (candidate 0)
Date: Fri, 04 Sep 2009 10:18:48 -0700
From: Doug Cutting <cutting@apache.org>
To: avro-dev@hadoop.apache.org

I have created a candidate build for Avro release 1.1.0.  This fixes
bugs in release 1.0.0 and adds a few new features, including comparators
and a json encoding.

Please download, test and vote by 9 September.

  http://people.apache.org/~cutting/avro-1.1.0-candidate-0/

Thanks!

Doug


From general-return-480-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Sep 05 22:51:51 2009
Return-Path: <general-return-480-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72181 invoked from network); 5 Sep 2009 22:51:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Sep 2009 22:51:51 -0000
Received: (qmail 59506 invoked by uid 500); 5 Sep 2009 22:51:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59329 invoked by uid 500); 5 Sep 2009 22:51:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59176 invoked by uid 99); 5 Sep 2009 22:51:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Sep 2009 22:51:49 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Sep 2009 22:51:41 +0000
Received: from [10.73.152.20] (snvvpn2-10-73-152-c20.hq.corp.yahoo.com [10.73.152.20])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n85MnDth008049;
	Sat, 5 Sep 2009 15:49:13 -0700 (PDT)
Message-ID: <4AA2EAE9.5090809@apache.org>
Date: Sat, 05 Sep 2009 15:49:13 -0700
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: announce@apache.org,
        "zookeeper-user@hadoop.apache.org" <zookeeper-user@hadoop.apache.org>,
        "zookeeper-dev@hadoop.apache.org" <zookeeper-dev@hadoop.apache.org>,
        general@hadoop.apache.org, private@hadoop.apache.org
Subject: [ANNOUNCE] Apache ZooKeeper 3.2.1
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

The Apache ZooKeeper team is proud to announce Apache ZooKeeper version 
3.2.1.

ZooKeeper is a high-performance coordination service for distributed 
applications. It exposes common services - such as naming, configuration 
management, synchronization, and group services - in a simple interface 
so you don't have to write them from scratch. You can use it 
off-the-shelf to implement consensus, group management, leader election, 
and presence protocols. And you can build on it for your own, specific 
needs.

If you are upgrading from version 2.2.1 on SourceForge be sure to review 
the 3.0.1 release notes for migration instructions.

For ZooKeeper release details and downloads, visit:
http://hadoop.apache.org/zookeeper/releases.html

ZooKeeper 3.2.1 Release Notes are at:
http://hadoop.apache.org/zookeeper/docs/r3.2.1/releasenotes.html

Detailed JIRA changlog can be viewed here:
http://tinyurl.com/lxltfh

Regards,

The ZooKeeper Team


From general-return-481-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Sep 05 22:53:51 2009
Return-Path: <general-return-481-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72421 invoked from network); 5 Sep 2009 22:53:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Sep 2009 22:53:51 -0000
Received: (qmail 60485 invoked by uid 500); 5 Sep 2009 22:53:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60389 invoked by uid 500); 5 Sep 2009 22:53:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60379 invoked by uid 99); 5 Sep 2009 22:53:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Sep 2009 22:53:50 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [209.191.125.34] (HELO web38403.mail.mud.yahoo.com) (209.191.125.34)
    by apache.org (qpsmtpd/0.29) with SMTP; Sat, 05 Sep 2009 22:53:39 +0000
Received: (qmail 29596 invoked by uid 60001); 5 Sep 2009 22:53:17 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1252191197; bh=JNo0D3td3t9oScqMQI+SiYRuhocQlYodi+ptu7UvmXM=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=ocUpsSn+UmPdHugJJmwxjv7pX8//FkzmmJhqG4Segtj+GRy0JOKXSkidTEGzP60lWEryn+M2Yyaoiy7psBVe54u4Ny8wI2VRHfJAuAn05hAPE65EupnJRUr7SpoLnIIQSaTQEiZsWVOXNgbP0/8tNa18BbGNOeiBy7Yen481x9M=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=3FtHIIsEn8DAz9r0prZmSiLHmkeEO8nW/eKz9u0izPbpIjOAAY7oQmtx0NbAXxPAb3q/OXLwKXFjdLTJsbHMmpff51QF+jOi4xoOpZHq13caj7YJW1W7nw1XEX8f0Rgwiqp5DkTk5s+p/E/tJ23zE39PwTh/So0W6HhO4riHVb4=;
Message-ID: <372864.29057.qm@web38403.mail.mud.yahoo.com>
X-YMail-OSG: ClY7MUYVM1njPl1LBYlkrSollJWPHWAUHzSqiCsgnZoO5l4U.u.ptprY_rT2FMMXv7cj7nFovz6d46g5hm6GPH_RvSH6PpEFYiv377JLBpQmLHfy0N.C.i1hqYsV6JZwYTaQb_WWBS5ezStFkumDijS7eI1hg6DzyYDLuejtCPC_S.QgcqGPpJNGYXlk4ibyMfYePfwzP.5TfxkhZ124KAx4hTzWep9q5BDo359KhDotFDNJunCXK3euUSpyvkF17n2bZSa.JdEl5X2kdeuGyWECL1aKjZFf12NaKyfKD0cL_9k-
Received: from [75.69.131.160] by web38403.mail.mud.yahoo.com via HTTP; Sat, 05 Sep 2009 15:53:17 PDT
X-Mailer: YahooMailRC/1358.27 YahooMailWebService/0.7.338.2
Date: Sat, 5 Sep 2009 15:53:17 -0700 (PDT)
From: himanshu chandola <himanshu_coolguy@yahoo.com>
Subject: printing only values and not keys in hadoop output files
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Everyone,
I'm a new bie to hadoop and had this question.
Normally hadoop spits output data in files in the form : 
key value

If i would like it to only write the value and not the key, what do I have to do ?



thanks a lot in advance,

Himanshu
Morpheus: Do you believe in fate, Neo?
Neo: No.
Morpheus: Why Not?
Neo: Because I don't like the idea that I'm not in control of my life.



      

From general-return-482-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Sep 05 23:08:29 2009
Return-Path: <general-return-482-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 74680 invoked from network); 5 Sep 2009 23:08:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Sep 2009 23:08:29 -0000
Received: (qmail 77713 invoked by uid 500); 5 Sep 2009 23:08:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77639 invoked by uid 500); 5 Sep 2009 23:08:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77629 invoked by uid 99); 5 Sep 2009 23:08:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Sep 2009 23:08:28 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of stuart.white1@gmail.com designates 209.85.216.185 as permitted sender)
Received: from [209.85.216.185] (HELO mail-px0-f185.google.com) (209.85.216.185)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Sep 2009 23:08:19 +0000
Received: by pxi15 with SMTP id 15so1535116pxi.23
        for <general@hadoop.apache.org>; Sat, 05 Sep 2009 16:07:57 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=4i6zioyNGYBUfuSNAmt5is58WjcwLrFwnlm3rLIkvaI=;
        b=ZVPLXe6scvurc0Ji7XjmoJVlDfEDCd/+v0kgSmvLvZlRR33FPC6h8626rp8tZ4EuZF
         LTnau+3bX3J7O2T3OH6TNdVv/4bjb3RCe+MtLnXJaXH52w7WOyxR+wSRG2OWjAXf6eJe
         965MMg7hkB5hkb9uh97PL6Ukb2hpBxm+Y8KIs=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=ajkI4FU0iWnZajEmrOk3wU8nsvMmZB2rjWgf34pYfUxEjD+fRtCNmUKpnSssw89eMM
         fPInLzQF1S4D4lv9iF3hm9Xniz52/QxirgW+xHd8bDXP0qKm+VNXct2YxrSh1CIHZ40x
         7baoZtcws4jK3Sj+VRtp8GP95jXPrNqs71WjA=
MIME-Version: 1.0
Received: by 10.143.26.42 with SMTP id d42mr421434wfj.219.1252192077915; Sat, 
	05 Sep 2009 16:07:57 -0700 (PDT)
In-Reply-To: <372864.29057.qm@web38403.mail.mud.yahoo.com>
References: <372864.29057.qm@web38403.mail.mud.yahoo.com>
Date: Sat, 5 Sep 2009 18:07:57 -0500
Message-ID: <4af5cd780909051607j2da2c9dci899039ba1ceb56fe@mail.gmail.com>
Subject: Re: printing only values and not keys in hadoop output files
From: Stuart White <stuart.white1@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636e0b180fea8a40472dcaf2b
X-Virus-Checked: Checked by ClamAV on apache.org

--001636e0b180fea8a40472dcaf2b
Content-Type: text/plain; charset=ISO-8859-1

Assuming you're writing plain-text output files using TextOutputFormat,
which normally outputs key/value pairs delimited with a Tab, you can emit a
null for your key, and neither a key nor the Tab delimiter will get written
to the output file.  (At least, I'm pretty sure that's right... I'm not
where I can run a quick test to verify...)


On Sat, Sep 5, 2009 at 5:53 PM, himanshu chandola <
himanshu_coolguy@yahoo.com> wrote:

> Hi Everyone,
> I'm a new bie to hadoop and had this question.
> Normally hadoop spits output data in files in the form :
> key value
>
> If i would like it to only write the value and not the key, what do I have
> to do ?
>
>
>
> thanks a lot in advance,
>
> Himanshu
> Morpheus: Do you believe in fate, Neo?
> Neo: No.
> Morpheus: Why Not?
> Neo: Because I don't like the idea that I'm not in control of my life.
>
>
>
>
>

--001636e0b180fea8a40472dcaf2b--

From general-return-483-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Sep 05 23:48:32 2009
Return-Path: <general-return-483-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78491 invoked from network); 5 Sep 2009 23:48:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Sep 2009 23:48:31 -0000
Received: (qmail 89416 invoked by uid 500); 5 Sep 2009 23:48:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89351 invoked by uid 500); 5 Sep 2009 23:48:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89341 invoked by uid 99); 5 Sep 2009 23:48:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Sep 2009 23:48:30 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.191.125.35] (HELO web38404.mail.mud.yahoo.com) (209.191.125.35)
    by apache.org (qpsmtpd/0.29) with SMTP; Sat, 05 Sep 2009 23:48:21 +0000
Received: (qmail 13812 invoked by uid 60001); 5 Sep 2009 23:47:59 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1252194479; bh=AKQH4BRi6M6I82PUvtoWt83uBwKByY7crsT2xMVf8vI=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=R3HXt8TaGoTzZMc2B1taUPFgBjTBrZgyaUpVe1xtqGfRHeDrrD2mKDq9lc860OCh6iT/qzNWXfFE5wuHCn6iKc/WeBtOgBPbmjTW8EC3xG3aTWo6M46zHktItBiPawsaguDtAiW0pWFTCUJmsz8j6/hPE0ipuh0bmsbFeU65sEw=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=Iq4BeqPhWXQFXZ8FccAa3ndQ837NZsBfvdX7wiurM5Oe4bpZiZi9G95S33Oz4sakX/8ny0qAdVigZQyZlixDf88uIsDo9lNOrtQkaC4vG+QNlIpQjQT6Cni/LdcQi3L4x0E8zC2ubr43Dpiz0YmvK303UpfDF6kcefB0nzjMWOk=;
Message-ID: <220039.13746.qm@web38404.mail.mud.yahoo.com>
X-YMail-OSG: Q6r7QlkVM1nKdDv7NSVpwcmmRTXPVDHUf5Y4z1cg0KV0isurpRPhPgjyMpWbDZrjcbnhqX9XVtN12DfRSOuEMDfVCduyO6QwoMjrD_ongSw6KuKNmkzRG.0BDUgaAhUFn6hzaxWouDAyAa_3DDEbIQTMcwIvWeJjPj7rWFgdAIMk7_AGnBeZcYyRQBRDu1K9GPSwV0pVXbpUsDfWKNThyNuJ9qpFyX0KASVJuoYzEQk8dVZRsUJ7bSxPFsbbKvxqONcxq4sDFNd1jbV..vue3z0B6vD07MdAhw3XLjVOwcMghntEWj0UCP4BEI1ebiaPJholZ8Zhoz5cizDAkUIQV2nb9AEzruLLVZKJUhZflvSTVPzBnJHt_JoFFMFMStUlwyXZ
Received: from [75.69.131.160] by web38404.mail.mud.yahoo.com via HTTP; Sat, 05 Sep 2009 16:47:58 PDT
X-Mailer: YahooMailRC/1358.27 YahooMailWebService/0.7.338.2
References: <372864.29057.qm@web38403.mail.mud.yahoo.com> <4af5cd780909051607j2da2c9dci899039ba1ceb56fe@mail.gmail.com>
Date: Sat, 5 Sep 2009 16:47:58 -0700 (PDT)
From: himanshu chandola <himanshu_coolguy@yahoo.com>
Subject: Re: printing only values and not keys in hadoop output files
To: general@hadoop.apache.org
In-Reply-To: <4af5cd780909051607j2da2c9dci899039ba1ceb56fe@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks for the suggestion.

I am using TextOutputFormat and I think that should work. But I use MultipleOutputFormat to output key,value pairs with certain key values to a file with a given name while others to a file with a different name. So if I output null for my keys after the reducer, I guess it would create problems.


Any other way you could think of ?
Morpheus: Do you believe in fate, Neo?
Neo: No.
Morpheus: Why Not?
Neo: Because I don't like the idea that I'm not in control of my life.



----- Original Message ----
From: Stuart White <stuart.white1@gmail.com>
To: general@hadoop.apache.org
Sent: Saturday, September 5, 2009 7:07:57 PM
Subject: Re: printing only values and not keys in hadoop output files

Assuming you're writing plain-text output files using TextOutputFormat,
which normally outputs key/value pairs delimited with a Tab, you can emit a
null for your key, and neither a key nor the Tab delimiter will get written
to the output file.  (At least, I'm pretty sure that's right... I'm not
where I can run a quick test to verify...)


On Sat, Sep 5, 2009 at 5:53 PM, himanshu chandola <
himanshu_coolguy@yahoo.com> wrote:

> Hi Everyone,
> I'm a new bie to hadoop and had this question.
> Normally hadoop spits output data in files in the form :
> key value
>
> If i would like it to only write the value and not the key, what do I have
> to do ?
>
>
>
> thanks a lot in advance,
>
> Himanshu
> Morpheus: Do you believe in fate, Neo?
> Neo: No.
> Morpheus: Why Not?
> Neo: Because I don't like the idea that I'm not in control of my life.
>
>
>
>
>



      

From general-return-484-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Sep 06 19:49:14 2009
Return-Path: <general-return-484-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85580 invoked from network); 6 Sep 2009 19:49:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Sep 2009 19:49:13 -0000
Received: (qmail 83605 invoked by uid 500); 6 Sep 2009 19:49:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83515 invoked by uid 500); 6 Sep 2009 19:49:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83505 invoked by uid 99); 6 Sep 2009 19:49:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Sep 2009 19:49:13 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [209.191.125.39] (HELO web38408.mail.mud.yahoo.com) (209.191.125.39)
    by apache.org (qpsmtpd/0.29) with SMTP; Sun, 06 Sep 2009 19:49:02 +0000
Received: (qmail 25785 invoked by uid 60001); 6 Sep 2009 19:48:40 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1252266519; bh=umNQSJBRKjg82SoTNHTjHnP9yIi5t3ihj/VtehJn5TY=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=ufZJSvj4zIBUsomoVfKYhp9pD66LVAk+9X2xluOqOqZsoGFiCpwFTna+NYRgbJdlqLOGiol/joNNII9iNXR895+j43Q9WGPwudilzeUpCM73Kg+D9TJKVqvp83XFqlB6jOrdyshSSXgo8+bMtqo/KCTbNzxoHbN293p9+neyj9I=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=aJjRLipEFjOGje48hh3Add0vKCF2/ipkqEkOejYG9p3ax4EugcH9A4vVIY2xleT5/FsQuNq4YrmbLDg16YQ9kvFqAO0H7z08IHLX+cVt9Xcc1do2yvfa7R1JJn2tYlvL9rOBRiYYPN+PjyH3zZTcZtMYBkraMzVeGXM9lAgf+ww=;
Message-ID: <950772.25530.qm@web38408.mail.mud.yahoo.com>
X-YMail-OSG: TD35CG8VM1lCxDVC7gffxX3D9qBCdbU1mtRD1LoyhdNRGX1ipi6ezNQCIBH2QJdtdjTYPi0CXwxK1wTw3HNlJ_bed8E2gmCFyoFrPt8E0kuJT4BxTJ01CVmR6nEXcsQeslKBmj11QVQL2f0XeMPCa7HlM0e6TCkiD_HspeR9etR5PspnzbDo7NxDuiZAjla3hYMqzSGIVsSjJmaDhy0JOqKWEt0J218yglAg9zrQDtKRhFBslBJ4tFnrk0eU6lUVh6t1BvL8vL_h8LIGMrpKo8nkdC0QHp_hFhnwYbB54yahJ3ml8FlJpbciOLA.RMnJ0h69jqO2wZCb1jFm9xDvn6GQ5rNW3JS5YtIXWKxCizqLQNvI4cpPJdHi6adxtddRXEBH
Received: from [75.69.131.160] by web38408.mail.mud.yahoo.com via HTTP; Sun, 06 Sep 2009 12:48:39 PDT
X-Mailer: YahooMailRC/1358.27 YahooMailWebService/0.7.338.2
References: <372864.29057.qm@web38403.mail.mud.yahoo.com> <4af5cd780909051607j2da2c9dci899039ba1ceb56fe@mail.gmail.com> <220039.13746.qm@web38404.mail.mud.yahoo.com>
Date: Sun, 6 Sep 2009 12:48:39 -0700 (PDT)
From: himanshu chandola <himanshu_coolguy@yahoo.com>
Subject: Re: printing only values and not keys in hadoop output files
To: general@hadoop.apache.org
In-Reply-To: <220039.13746.qm@web38404.mail.mud.yahoo.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org

I actually fixed that by overriding generateActualKey in MultipleOutputFormat which I use to return null.

thanks 

 Morpheus: Do you believe in fate, Neo?
Neo: No.
Morpheus: Why Not?
Neo: Because I don't like the idea that I'm not in control of my life.



----- Original Message ----
From: himanshu chandola <himanshu_coolguy@yahoo.com>
To: general@hadoop.apache.org
Sent: Saturday, September 5, 2009 7:47:58 PM
Subject: Re: printing only values and not keys in hadoop output files

Thanks for the suggestion.

I am using TextOutputFormat and I think that should work. But I use MultipleOutputFormat to output key,value pairs with certain key values to a file with a given name while others to a file with a different name. So if I output null for my keys after the reducer, I guess it would create problems.


Any other way you could think of ?
Morpheus: Do you believe in fate, Neo?
Neo: No.
Morpheus: Why Not?
Neo: Because I don't like the idea that I'm not in control of my life.



----- Original Message ----
From: Stuart White <stuart.white1@gmail.com>
To: general@hadoop.apache.org
Sent: Saturday, September 5, 2009 7:07:57 PM
Subject: Re: printing only values and not keys in hadoop output files

Assuming you're writing plain-text output files using TextOutputFormat,
which normally outputs key/value pairs delimited with a Tab, you can emit a
null for your key, and neither a key nor the Tab delimiter will get written
to the output file.  (At least, I'm pretty sure that's right... I'm not
where I can run a quick test to verify...)


On Sat, Sep 5, 2009 at 5:53 PM, himanshu chandola <
himanshu_coolguy@yahoo.com> wrote:

> Hi Everyone,
> I'm a new bie to hadoop and had this question.
> Normally hadoop spits output data in files in the form :
> key value
>
> If i would like it to only write the value and not the key, what do I have
> to do ?
>
>
>
> thanks a lot in advance,
>
> Himanshu
> Morpheus: Do you believe in fate, Neo?
> Neo: No.
> Morpheus: Why Not?
> Neo: Because I don't like the idea that I'm not in control of my life.
>
>
>
>
>


      

From general-return-485-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Sep 06 19:53:22 2009
Return-Path: <general-return-485-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85962 invoked from network); 6 Sep 2009 19:53:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Sep 2009 19:53:22 -0000
Received: (qmail 85945 invoked by uid 500); 6 Sep 2009 19:53:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85864 invoked by uid 500); 6 Sep 2009 19:53:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85854 invoked by uid 99); 6 Sep 2009 19:53:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Sep 2009 19:53:21 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.191.125.34] (HELO web38403.mail.mud.yahoo.com) (209.191.125.34)
    by apache.org (qpsmtpd/0.29) with SMTP; Sun, 06 Sep 2009 19:53:12 +0000
Received: (qmail 48787 invoked by uid 60001); 6 Sep 2009 19:52:49 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1252266769; bh=LaMooYY7fk0rBqqCbL16QrQ+ExaMSzYXoTiHzdXgei4=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=re4ewXmSZABcqlFkVPYrP1IFm6Fmbe4eILg1hHjMFMoBbkBHAZSBmUwOmescB2MqzflTmcjYsP5BGzZyJjpiMjqXmsXbub3R1sZvA1oxEm03fBV+mx9vNEvwwXVoHm9LCAEbIAAzv1Xh2vK/URrryfZesv6EeysvgudbmK6WzFI=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=LnXnjbibP54MvrTSWYL5eFr+zTNDmPdtjzskZiroaZxLA6awAKSe1dt2i177D6PFA/79yI/2RutTZq+XTnYJYdIBx6jMhSEjTcixJ1wrmHEeO7GbCKBnB8jJTFVky4vjSU8ZHJo/7mgj7gJGK0ruERvWmjam6HJLPCBqou6TwNo=;
Message-ID: <805475.48507.qm@web38403.mail.mud.yahoo.com>
X-YMail-OSG: c9k1u6YVM1k3rwIJpYMMjZyRl.3nfQi9yBLtRN9QZ0sdLPPwj9P2296fscbhKOBtNVup.0hkQ8yyiaKGa320JB3WLuFmbR1pX1NtxG28BOx8h3ZkuyI6Amc7ApXeblbW4tL2BAk0vR3coLH96mPhrELTduFXHXNDX2XBRN_UQAKFoHHufyvCRQGprOVvWPkTeZvjG2ysNckNrpNt2dOrrEwWfnlUW7cBzZYmICXF96yPafMvnY6lo33OAU3hsDvtFFLyN.FwiYqP0lV4vA0jaC0rc46gH2Ugp5twRIpQft1vkiFadS6gsM4bocImmYhGJXA29UFd8rH7KA6Uvpea4Wo9FhpUx4BhJ0LV
Received: from [75.69.131.160] by web38403.mail.mud.yahoo.com via HTTP; Sun, 06 Sep 2009 12:52:49 PDT
X-Mailer: YahooMailRC/1358.27 YahooMailWebService/0.7.338.2
Date: Sun, 6 Sep 2009 12:52:49 -0700 (PDT)
From: himanshu chandola <himanshu_coolguy@yahoo.com>
Subject: questions on map and reduce
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Everyone,
What would be the best way to get map to output values  in different formats (reduce needs to add the integer ones) . I realize that writing an ObjectWritable (not of the primitive type that comes with hadoop) might be the best. I basically am worried that if I output integers to Text and convert them back to IntWritable, it would be a performance overhead. Would it be significant enough to worry about ? And can you think of anything other than ObjectWritable or having everything as Text to do this ?

And another question was is there a way to get reduce to set values in JobConf or do I just flush that into disk and let the sequential job to read that. Basically , am I reading too much into the performance overhead thing ?



Thanks

 Morpheus: Do you believe in fate, Neo?
Neo: No.
Morpheus: Why Not?
Neo: Because I don't like the idea that I'm not in control of my life.



      

From general-return-486-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Sep 06 19:53:26 2009
Return-Path: <general-return-486-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 86021 invoked from network); 6 Sep 2009 19:53:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Sep 2009 19:53:26 -0000
Received: (qmail 86530 invoked by uid 500); 6 Sep 2009 19:53:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86454 invoked by uid 500); 6 Sep 2009 19:53:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86444 invoked by uid 99); 6 Sep 2009 19:53:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Sep 2009 19:53:25 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of monika.m.moser@googlemail.com designates 72.14.220.159 as permitted sender)
Received: from [72.14.220.159] (HELO fg-out-1718.google.com) (72.14.220.159)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Sep 2009 19:53:15 +0000
Received: by fg-out-1718.google.com with SMTP id e12so505270fga.11
        for <general@hadoop.apache.org>; Sun, 06 Sep 2009 12:52:53 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=8PZ5ZvoRxL7YOFzmuDF9u/erJuoA+9+L8Liyl1EPhPw=;
        b=G8Y30ThMRGNY/J3+py41y9dyfKuGUYwW0w0OKFvT0jiQbcsGWlCq1oo5IMP1HVvN6P
         MlyY2LPPmDAhKBpI5yWEFnXPNuDdXiVYBbHDnNQDW7PDAKU8QCI2jUnrNuwSV9Wf8IKe
         InCH5+mhkZyw1XlX7XjTptpDHYYsyXQAFDJ/4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=ohNjWQQHo56h8EkuM19HLNLvBjYzPtGYZbLAPSDUOUk+8zxIP97DxxBQ8/S/d1Yf3l
         OqBpX3SyOgwrjFiBG/KgYIpIreUa345pGU26FN0WnLT+cG4rz8L3HE+S4Gk3tjaQGZOq
         eGkS0lWBj0NCSSwgbdaJfXnRtZnY4HiCUbHoY=
MIME-Version: 1.0
Received: by 10.86.210.32 with SMTP id i32mr7161845fgg.60.1252266773738; Sun, 
	06 Sep 2009 12:52:53 -0700 (PDT)
In-Reply-To: <950772.25530.qm@web38408.mail.mud.yahoo.com>
References: <372864.29057.qm@web38403.mail.mud.yahoo.com>
	 <4af5cd780909051607j2da2c9dci899039ba1ceb56fe@mail.gmail.com>
	 <220039.13746.qm@web38404.mail.mud.yahoo.com>
	 <950772.25530.qm@web38408.mail.mud.yahoo.com>
Date: Sun, 6 Sep 2009 21:52:53 +0200
Message-ID: <ee4539410909061252t39e665b4m669af275f325f5d3@mail.gmail.com>
Subject: Re: printing only values and not keys in hadoop output files
From: Monika Moser <monika.m.moser@googlemail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00c09ffb54783677b00472ee1490
X-Virus-Checked: Checked by ClamAV on apache.org

--00c09ffb54783677b00472ee1490
Content-Type: text/plain; charset=ISO-8859-1

You can use the NullWritable Class in the output format in order to suppress
output either for the key or for the value.
Also in this case there will be no separator written.

Monika


2009/9/6 himanshu chandola <himanshu_coolguy@yahoo.com>

> I actually fixed that by overriding generateActualKey in
> MultipleOutputFormat which I use to return null.
>
> thanks
>
>  Morpheus: Do you believe in fate, Neo?
> Neo: No.
> Morpheus: Why Not?
> Neo: Because I don't like the idea that I'm not in control of my life.
>
>
>
> ----- Original Message ----
> From: himanshu chandola <himanshu_coolguy@yahoo.com>
> To: general@hadoop.apache.org
> Sent: Saturday, September 5, 2009 7:47:58 PM
> Subject: Re: printing only values and not keys in hadoop output files
>
> Thanks for the suggestion.
>
> I am using TextOutputFormat and I think that should work. But I use
> MultipleOutputFormat to output key,value pairs with certain key values to a
> file with a given name while others to a file with a different name. So if I
> output null for my keys after the reducer, I guess it would create problems.
>
>
> Any other way you could think of ?
> Morpheus: Do you believe in fate, Neo?
> Neo: No.
> Morpheus: Why Not?
> Neo: Because I don't like the idea that I'm not in control of my life.
>
>
>
> ----- Original Message ----
> From: Stuart White <stuart.white1@gmail.com>
> To: general@hadoop.apache.org
> Sent: Saturday, September 5, 2009 7:07:57 PM
> Subject: Re: printing only values and not keys in hadoop output files
>
> Assuming you're writing plain-text output files using TextOutputFormat,
> which normally outputs key/value pairs delimited with a Tab, you can emit a
> null for your key, and neither a key nor the Tab delimiter will get written
> to the output file.  (At least, I'm pretty sure that's right... I'm not
> where I can run a quick test to verify...)
>
>
> On Sat, Sep 5, 2009 at 5:53 PM, himanshu chandola <
> himanshu_coolguy@yahoo.com> wrote:
>
> > Hi Everyone,
> > I'm a new bie to hadoop and had this question.
> > Normally hadoop spits output data in files in the form :
> > key value
> >
> > If i would like it to only write the value and not the key, what do I
> have
> > to do ?
> >
> >
> >
> > thanks a lot in advance,
> >
> > Himanshu
> > Morpheus: Do you believe in fate, Neo?
> > Neo: No.
> > Morpheus: Why Not?
> > Neo: Because I don't like the idea that I'm not in control of my life.
> >
> >
> >
> >
> >
>
>
>
>

--00c09ffb54783677b00472ee1490--

From general-return-487-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Sep 06 23:05:49 2009
Return-Path: <general-return-487-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19776 invoked from network); 6 Sep 2009 23:05:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Sep 2009 23:05:49 -0000
Received: (qmail 90555 invoked by uid 500); 6 Sep 2009 23:05:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90469 invoked by uid 500); 6 Sep 2009 23:05:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90437 invoked by uid 99); 6 Sep 2009 23:05:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Sep 2009 23:05:48 +0000
X-ASF-Spam-Status: No, hits=3.7 required=10.0
	tests=ASF_LIST_OPS,HEAD_LONG,HTML_FONT_FACE_BAD,HTML_FONT_SIZE_HUGE,HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of digvijaysinghshaktawat@gmail.com designates 209.85.222.202 as permitted sender)
Received: from [209.85.222.202] (HELO mail-pz0-f202.google.com) (209.85.222.202)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Sep 2009 23:05:35 +0000
Received: by pzk40 with SMTP id 40so604782pzk.30
        for <multiple recipients>; Sun, 06 Sep 2009 16:05:13 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=kmh4qIw+gv9BRLTmDebrM5f82TRIbbPxNLsRlkQlhHA=;
        b=EioHjwYJBasFShe00FdHrW7f3yVSE70Ziw99fVqlkNMLVOpgR02BWI5vjMT04lc6Ms
         VY8aAwBmsmP+tRKB0ZQ1xJknKIhqRua4Wdc3eBvvulbI9wsLfsb2aYJgxGkfk7KZimDg
         ckYuhAnn04zYKY/8LrWqdJaR6RuyXtsQrVMIg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=N6V78ui1TlNrKsqbVmpdx7JsUO7KuEPx2maKnGIUTHv7P4OzEwltOKjnUCsV8e+b3z
         5oNIwek6QTuusqeg47g3ffaDlWDpmJAjcb8eflIMGMS62RCn7yzIcavds+MzZaXuNx8o
         WLvxjdzfG/5+nAKK185hQvr66YdLDUC+3c1+s=
MIME-Version: 1.0
Received: by 10.142.8.12 with SMTP id 12mr457826wfh.70.1252276316327; Sun, 06 
	Sep 2009 15:31:56 -0700 (PDT)
In-Reply-To: <8e0e69ea0909061502x6df82175y91f642dd8882e895@mail.gmail.com>
References: <8e0e69ea0909061502x6df82175y91f642dd8882e895@mail.gmail.com>
Date: Mon, 7 Sep 2009 04:01:56 +0530
Message-ID: <bf8befe30909061531j5832c0faud7d74c941cc8a4b7@mail.gmail.com>
Subject: Fwd: MUST READ - we can all use a little luck in our lives...
From: digvijay singh shaktawat <digvijaysinghshaktawat@gmail.com>
To: =?ISO-8859-1?B?rUFycGl0IEpvc2hpIH4uLi4uLml0cyBhIG5ldyBzdGFydA==?= <arpitdjosh@gmail.com>, 
	=?ISO-8859-1?Q?=ADRishitosh_=AD_Bhatia?= <rsbhatia.eden@gmail.com>, 
	=?UTF-8?B?J+KAoCBBZlRlUmdsT3cg4oCgJyAn?= <tastemaker19@gmail.com>, 
	=?ISO-8859-1?Q?=BBM=DFR=AB_=D8PTIMIST_kunal_sanghvi?= <kunalsanghvi@gmail.com>, 
	"@bH!$HEk Vy@$" <abhishekvyas777@yahoo.co.in>, "//eht@ ." <am9993011880@gmail.com>, 
	"/0te fr t/j... www.new7wonders.com" <namitjain111@gmail.com>, 
	"<<>> Can't Get U Out Of My Head.." <ankitnichani@gmail.com>, 
	=?UTF-8?B?4pa6IMOFz4DOmM6Yz4Hil4QtIC3CpC0tXl0gQCBIb21lIFteLS3CpC0gLQ==?= <anoopdh@yahoo.com>, 
	=?UTF-8?B?4piF4peP4pWg4pWjxJA54oSi4pS84oSi4peP4pSM4oip4pSQKOKXo+KXoiDilojilZHkuYLQs86YY2s=?=
	=?UTF-8?B?xaHPhM6x4pWR4paIKOKXoyDil6IpR+C4hOCusEfguKPRgsSD?= <harshhd9@gmail.com>, 
	=?UTF-8?B?4pmlIOC4hOC5ktGS4LmA4Lij0ZLRlNC6IOKZpQ==?= <abhishekkasliwa@gmail.com>, 
	=?UTF-8?B?4pmlUmFodWzimaUgamFpbuKZpeKZpQ==?= <rahul.hi@gmail.com>, 
	"Ab Pappu saala nachega..." <puneet2757@gmail.com>, Abhijay Sisodia <abhijaysisodia@gmail.com>, 
	abhijay sisodia <abhijaysisodia@yahoo.co.in>, Abhishek Gupta <abhishekguptaiitb@gmail.com>, 
	"ABHISHEK... I PLAY TO WIN!!!" <vyas1717@gmail.com>, ADITYA PAREEK <adytommyusa@yahoo.co.in>, 
	"ADITYA...... PAREEK DIU ROXXXXXXX!!!!" <pareek.aditya@gmail.com>, afhaam qureshi <ben_afflek19@yahoo.com>, 
	"Akash...I M LOST IN ORKUT" <sumit.khera007@gmail.com>, akshat oswal <akshatoswal@gmail.com>, 
	akshat oswal <akshatoswal_11@yahoo.co.in>, algorythmus2008@gmail.com, 
	"analprakhar\"best of THE BEAST\"........." <analprakhar@gmail.com>, 
	"anchit (bhanu).......back from home" <lukhha@gmail.com>, 
	"anchit(bhanu) ...........tech time" <anchit_iitb@yahoo.co.in>, Anil Kishore <anil.daiict@gmail.com>, 
	"ANKIT ##K^^OW ME```## K^^OW =@LIFE" <vit.ankit@gmail.com>, "Ankit anubhav $$$$$$$$$$$$$$$" <koolritchie@gmail.com>, 
	"Ankit....... @VIT" <ankit_hishyguy@yahoo.co.in>, 
	"ankur .......missing home" <agrawalgood@gmail.com>, ankur mittal <ankurmittal123@gmail.com>, 
	"ankur mittal bvp I.T ki jai ho!!" <ankurmittal123@yahoo.co.in>, =?windows-1252?B?QW5rdXIgmSAuLi4uLi4=?= <masterankur@gmail.com>, 
	"aNshuL ....\" in crow'th dimension\"" <yhpans@gmail.com>, 
	=?UTF-8?B?QW5zaHVsIMKrz4ZlYWNlZnVswqvPiWFycmnimLxywqs=?= <anshul.go87@gmail.com>, 
	anshulika singh <raunak2727@gmail.com>, Anurag Garg <boilergarg@gmail.com>, 
	Anurag Garg <garg@purdue.edu>, 
	"arjun Singhal...vote for taj" <singhal_arjun19@yahoo.co.in>, Arpit maru <arpit.maru@gmail.com>, 
	ARPIT-A DAY DREAMMER <iictarpit@gmail.com>, Arti Krishnawat <artikrishnawat@gmail.com>, 
	Ashish Aeran <aeran.ashish@gmail.com>, Atishay <atishay.kumar@gmail.com>, 
	"avi.... New Hair Style..." <manit.avi@gmail.com>, "avi....new no. 9993794257" <aviavi007@yahoo.com>, 
	"AVINASH.. 9723895020!!!" <wiedler@gmail.com>, Ayush Modi <ayushmod@gmail.com>, 
	"ayush modi..." <ayush_modi@hotmail.com>, 
	"Being AbHimaNyu ...ThinKi|| DifferentlY..." <abhi.baidya@gmail.com>, bharath@vmukti.com, bhumika.planet@gmail.com, 
	"check my album.. web-site updated" <shriyasingh@gmail.com>, chitbhav bhatjiwale <kool_hotguy007@yahoo.co.in>, 
	"chitbhav<>" <chitbhav@gmail.com>, common-user@hadoop.apache.org, 
	Dhaval Sanghvi <sanghvi.dhaval86@gmail.com>, 
	digvijay singh shaktawat <digvijaysinghshaktawat@gmail.com>, digvijay.shaktawat@gmail.com, 
	Divyendra Chauhan <divyendrachauhan@gmail.com>, dorav kumar <doravkumar.goyal@gmail.com>, 
	Ehtesham Hazarika <ehteshamhazarika@gmail.com>, 
	Ehtesham Hazarika <ehtesham_hazarika@yahoo.com>, 
	"Every thin U can imagine is REAL!!!...." <afhaam12345@gmail.com>, 
	fren list full join on s rajora2 <rajoras@gmail.com>, Gaurav Singh <gaurav1712@gmail.com>, 
	general-sc.1250981800.ndbmfpfinodhoijpmloj-digvijaysinghshaktawat=gmail.com@hadoop.apache.org, 
	general-subscribe@hadoop.apache.org, general@hadoop.apache.org, 
	Himanshu gupta <hemantoocool@gmail.com>, himanshu gupta <heman_too_cool@yahoo.co.in>, 
	"hnkuprai.. m so messd up rite nw.!" <pankhy@gmail.com>, "I hate HOLIDAYS.........." <pracheerag87@gmail.com>, 
	"Ishan - Rare Orkutting 4m now on ...." <ishanjain16@yahoo.com>, 
	"Jackson Life is ful of fun!!!!!!!!!!" <jackson.garg@gmail.com>, "jigyasa.. njoin at home...." <jigz_107@yahoo.com>, 
	"kamakhya ..lifes so cul wid CrIcKeT.." <kamakhya30@yahoo.com>, kamakhya pratap <kamakhya22@gmail.com>, 
	kartik kudada <kartik.kudada@gmail.com>, lebinitiz <lebinitiz@gmail.com>, 
	"M@dDy~~If U W@Nt SoMe~~ CoMe GeT SoMe" <maddyonlyluvs@gmail.com>, 
	"Marlo Group .... Ankit Agarwal" <ankitatdaiict@gmail.com>, 
	"miles to go..tnt the king of agraba......." <tanaythetnt@gmail.com>, "mohit @ 9608339390" <mohit.vit@gmail.com>, 
	mohit agarwal <heart_mohit87@yahoo.com>, mohit mittal <mohitnadbai@gmail.com>, 
	"moving towards the correct side ....." <manishguptaiitb@gmail.com>, 
	"Nakul raavviiing visit www.iitr.ac.in/rave" <nakulbende@gmail.com>, Naman Patni <namanpatni@gmail.com>, 
	navneet singh <nav.attitude@gmail.com>, Neha Garg <nehagarg07@gmail.com>, 
	"No orkuting till 3 dec......................." <pracheer_ag87@yahoo.co.in>, "pankhuri.. ." <pankhuri_patni@yahoo.co.in>, 
	"parth patel (satyameva jayate)" <parth.gmc@gmail.com>, Pawan garg <p4pawan@gmail.com>, 
	pawan garg <p4pawan_g@yahoo.co.in>, pk girpade <priyanka.svnit@gmail.com>, 
	PRABHAT PRATAP RAO <prabhat_tony_rao@yahoo.co.in>, PRABHAT RAO <prabsiiit@gmail.com>, 
	Prateek Agarwal <agarwal.prateek@gmail.com>, 
	"praveen it$ begining of new er@" <praveensvnit87@gmail.com>, 
	"Praveen..... GOOD ol' days are over..." <pravn.iitb@gmail.com>, PrAvIr JhA <jhapravir@gmail.com>, 
	"puneet... ALIGATOR BREAKS FREE" <puneet2787@yahoo.co.in>, Rahul Ladhania <heyrahulhere@gmail.com>, 
	Rahul Sharma <rashanality@gmail.com>, rahulsharma.bitsg@gmail.com, 
	Rajeev Gupta <grajeev@in.ibm.com>, rakesh kumar <rakesh.iict@gmail.com>, 
	"Rakesh More........." <ra9826110074@gmail.com>, ravi agrawal <src.ravi@gmail.com>, 
	"Rishika ~ in Bbay now...!!! missin Pune.." <kothari.rishika@gmail.com>, 
	"Rohan - \"BUTRU\" -..........In GOA..........." <bindasbandahaii@gmail.com>, 
	rohan --- hoping fr brighter sunshine in life <rohanricky@gmail.com>, "Rohit09376910288 ..!!" <glancerohit@gmail.com>, 
	romit pandey <romit_pandey2003@yahoo.co.in>, 
	romit PROUD TO BE AN INDIAN <romitpandey@gmail.com>, saleema omar <saleema2omar@gmail.com>, 
	sameer khan <spectrojin@gmail.com>, 
	"sameer khan ** sacred 4 all times++" <spectrojin_005@yahoo.co.in>, Sarah Lal <connect2sarah@gmail.com>, 
	Saurabh Arora <ishere.arora@gmail.com>, SEEMANT GUPTA <arpi.seem0507@gmail.com>, 
	shivraj singh rathore <shivraj.rathore@conizant.com>, 
	"Shrenik:-TrUsT A PoIsEn snake BuT NeVr GIrL" <citmills@gmail.com>, shweta.svnit@gmail.com, shweta@vmukti.com, 
	"SIDDHARTH:- HAPPY AND NJOYING" <siddharth09111987@gmail.com>, Siddhartha Lal <siddhartha797@gmail.com>, 
	Sky Dots Systems <skyinfonet@gmail.com>, somya garg <somya.garg@gmail.com>, 
	"Sonali ..." <sonali.karia@gmail.com>, sopan.shewale@gmail.com, 
	"Sourabh...raped in this sem" <iitr.sourabh@gmail.com>, suman_dongre <suman_dongre@yahoo.com>, sumanmdongre@gmail.com, 
	"SUMIT.......... ." <styleagain@gmail.com>, support@salesforce.com, 
	suresh remember moments not days <sankrank@gmail.com>, "swasti ." <dr.swasti19@rediffmail.com>, sweta@vmukti.com, 
	TANMAY D Party Starter <terrifictanmay@gmail.com>, Taru Sardana <tarusardana@gmail.com>, 
	Tejas der Bonbon <tej2687@gmail.com>, The Darker side of LIGHT <jatinder.chhabra@gmail.com>, 
	Trusha Dave <hospitality@nmumang.org>, urmi patel <urmipatel86@gmail.com>, 
	urmi patel <urmi_patel86@yahoo.co.in>, "vaibhav jindal ." <rushto.vj@gmail.com>, 
	Vanshvardhan Singh Hada <staytunedwithvansh@hotmail.com>, 
	"varun : mob. no. changed" <varungupta.the.great@gmail.com>, varun gupta <meetvarun1987@yahoo.co.in>, 
	Vasistha Chowdhary <vchowdhary01@gmail.com>, 
	"Vibhor Tulsyan on -**---TOUR---**-" <vibhortulsyan@gmail.com>, 
	"vijit....HurRaY :D ItS TimE 2 ----->HOme" <akkubest@gmail.com>, Vikas Gupta <vikas2003hi@yahoo.co.in>, 
	"Vikas..... HP 7 too gud" <thegr8vikas@gmail.com>, "www.moodi.org -mauly" <smartyetsweet@gmail.com>, 
	=?UTF-8?B?0YLQvdGUINC90ZTOsdGP0YIg0L3OsdGVIM6xINC8zrnQuOKIgiDPg2YgzrnRgtGVIM+Dz4nQuCDOsQ==?=
	=?UTF-8?B?4bmJw7jDuM+B?= <anoopdh2005@gmail.com>
Content-Type: multipart/alternative; boundary=00504502be98fed6ad0472f04c4b
X-Virus-Checked: Checked by ClamAV on apache.org

--00504502be98fed6ad0472f04c4b
Content-Type: text/plain; charset=ISO-8859-1

---------- Forwarded message ----------
From: Sameer Khan <spectrojin@gmail.com>
Date: Mon, Sep 7, 2009 at 3:32 AM
Subject: MUST READ - we can all use a little luck in our lives...
To: "desparate 2 make .....undeterred progress...." <
lovin_johnny@hotmail.com>



 Good advice! Believe!

 *This is without a doubt one of the nicest good luck forwards I have
received. Hope it works for you -- and me! *



*Lotus Touts: You have 6 minutes*



*There's some mighty fine advice in these words, even if you're not
superstitious. This Lotus Touts has been sent to you for good luck from the
Anthony Robbins organization. It has been sent around the world ten times so
far. *



*Do not keep this message..*



*The Lotus Touts must leave your hands in 6 MINUTES. Otherwise you will get
a very unpleasant surprise. This is true, even if you are not superstitious,
agnostic, or otherwise faith impaired. *



*ONE. Give people more than they expect and do it cheerfully.*



*TWO. Marry a man/woman you love to talk to. As you get older, their
conversational skills will be as important as any other. *



*THREE. Don't believe all you hear, spend all you have or sleep all you
want. *



*FOUR.. When you say, 'I love you,' mean it.** *



*FIVE. When you say, 'I'm sorry,' look the person in the eye. *



*SIX. Be engaged at least six months before you get married. *



*SEVEN. **Believe in love at first sight.** *



*EIGHT. Never laugh at anyone's dreams. People who don't have dreams don't
have much. *



*NINE. Love deeply and passionately. You might get hurt but it's the only
way to live life completely. *



*TEN.. In disagreements, fight fairly. No name calling. *



*ELEVEN.** **Don't judge people by their relatives.** *



*TWELVE. Talk slowly but think quickly. *



*THIRTEEN! .. When someone asks you a question you don't want to answer,
smile and ask, 'Why do you want to know?' *



*FOURTEEN. Remember that great love and great achievements involve great
risk. *



*FIFTEEN. Say 'bless you' when you hear someone sneeze. *



*SIXTEEN. When you lose, don't lose the lesson.*



*SEVENTEEN. **Remember the three R's: Respect for self; Respect for others;
and Responsibility for all your actions. *



*EIGHTEEN. Don't let a little dispute injure a great friendship. *



*NINETEEN. When you realize you've made a mistake, take immediate steps to
correct it.** *



*TWENTY. Smile when picking up the phone. The caller will hear it in your
voice *



*TWENTY- ONE.. Spend some time alone. *



*Now, here's the FUN part!  *

*Send this to at least 5 people and your life will improve. *


*  *
*1-4 people: Your life will improve slightly. *
*  *
*5-9 people: Your life will improve to your liking.*
*  *
*9-14 people: You will have at least 5 surprises in the next 3 weeks *

*15 and above: Your life will improve drastically and everything you ever
dreamed of will begin to take shape. *


*A true friend is someone who reaches for your hand and touches your heart.*
* Do not keep this message.*


 ------------------------------




-- 
Sameer khan's BLOG - http://sameerkhansblog.blogspot.com/

Follow @ Twitter - http://twitter.com/Sameer_Khan

Facebook Page - http://www.facebook.com/pages/Sameer-Khans-BLOG/73173892290

Friend Feed - http://friendfeed.com/spectrojin

--00504502be98fed6ad0472f04c4b--

From general-return-488-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 07 09:48:46 2009
Return-Path: <general-return-488-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63357 invoked from network); 7 Sep 2009 09:48:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Sep 2009 09:48:46 -0000
Received: (qmail 56578 invoked by uid 500); 7 Sep 2009 09:48:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56497 invoked by uid 500); 7 Sep 2009 09:48:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56487 invoked by uid 99); 7 Sep 2009 09:48:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Sep 2009 09:48:45 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of hkumar.arora@gmail.com designates 209.85.223.173 as permitted sender)
Received: from [209.85.223.173] (HELO mail-iw0-f173.google.com) (209.85.223.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Sep 2009 09:48:35 +0000
Received: by iwn3 with SMTP id 3so173000iwn.2
        for <general@hadoop.apache.org>; Mon, 07 Sep 2009 02:48:14 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=vD7T+pNUCt8z1nBc1wrjATE9EeBcSRJq3LBwa7up4Nw=;
        b=pMrybZLbPHmHklh0+qRpVNpk+HmeyjS/nl8NelIpLTpLfcV0um2hT+adLPLWHi4iiS
         POWc977oOxSPhS9xRhCiJ/hdN5hEfhoOf50k85yEp5uzBvhRT0oiV2i+Y+I5/7zqqSBe
         E7BgA1An2gV0VvLPDCakST/zyXfBEQ2jtXpG8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=B95uQ6FMEafTy1kwYQG5lx2LKFexpor0OJjn1CN78fE9nrXxdxpe6cOp7n0YAD4GAI
         A+kXB+FccHllTRf/S9+dQJFCkHE830uWcTv3LVoKwjgVnQVTOVl4yuu33inO9lMEBtJd
         qPqjHXPRlXX0f3xRO++LTIlrKQI/k6URDeRHI=
MIME-Version: 1.0
Received: by 10.231.6.164 with SMTP id 36mr11348380ibz.39.1252316893569; Mon, 
	07 Sep 2009 02:48:13 -0700 (PDT)
Date: Mon, 7 Sep 2009 18:48:13 +0900
Message-ID: <e497591c0909070248u347f1ca5j42b7435864fd31ed@mail.gmail.com>
Subject: Type mismatch in value from map: expected org.apache.hadoop.io.Text, 
	recieved org.apache.hadoop.io.IntWritable
From: Harshit Kumar <hkumar.arora@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175ce0689663980472f9bf46
X-Virus-Checked: Checked by ClamAV on apache.org

--0015175ce0689663980472f9bf46
Content-Type: text/plain; charset=UTF-8

I get this error on WordCount program, originally copied from the examples
folder in Hadoop-0.19.2.
I dont understand why I get this error. Everything is set right. Please see
the code and give feedback.

import java.io.IOException;
import java.util.*;
 import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;
 public class WordCount {
    public static class Map extends MapReduceBase implements
Mapper<LongWritable, Text, Text, Text> {
      private Text word = new Text();
      private Text upperWord = new Text();

      public void map(LongWritable key, Text value, OutputCollector<Text,
Text> output, Reporter reporter) throws IOException {
        String line = value.toString();
        StringTokenizer tokenizer = new StringTokenizer(line);
        while (tokenizer.hasMoreTokens()) {
         String temp =tokenizer.nextToken();
          word.set(temp);
          upperWord.set(temp.toUpperCase());
          output.collect(word, upperWord);
        }
      }
    }
    public static class Reduce extends MapReduceBase implements
Reducer<Text, Text, Text, Text> {
      public void reduce(Text key, Iterator<Text> values,
OutputCollector<Text, Text> output, Reporter reporter) throws IOException {
        String sum = new String();
        boolean flag=true;
        while (values.hasNext()) {
         if(!flag)
         sum = sum.concat(", "+values.next().toString());
         else
         sum = values.next().toString();

         flag=false;
        }
        output.collect(key, new Text(sum));
      }
    }
    public static void main(String[] args) throws Exception {
      JobConf conf = new JobConf(WordCount.class);
      conf.setJobName("wordcount");
      conf.setOutputKeyClass(Text.class);
      conf.setOutputValueClass(Text.class);

      conf.setMapOutputKeyClass(Text.class);
      conf.setMapOutputValueClass(Text.class);
      conf.setMapperClass(Map.class);
      //conf.setCombinerClass(Reduce.class);
      conf.setReducerClass(Reduce.class);
      conf.setInputFormat(TextInputFormat.class);
      conf.setOutputFormat(TextOutputFormat.class);
      FileInputFormat.setInputPaths(conf, new Path("In"));
      FileOutputFormat.setOutputPath(conf, new Path("Outer"));
      JobClient.runJob(conf);
    }
}


Thanks
H. Kumar
Phone(Mobile): +82-10-2892-9663
Phone(Office): +82-31-
skype: harshit900
Blog: http://harshitkumar.wordpress.com
Website: http:/kumarharmuscat.tripod.com

--0015175ce0689663980472f9bf46--

From general-return-489-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 07 10:41:06 2009
Return-Path: <general-return-489-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76092 invoked from network); 7 Sep 2009 10:41:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Sep 2009 10:41:05 -0000
Received: (qmail 18607 invoked by uid 500); 7 Sep 2009 10:41:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18468 invoked by uid 500); 7 Sep 2009 10:41:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18440 invoked by uid 99); 7 Sep 2009 10:41:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Sep 2009 10:41:00 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of hkumar.arora@gmail.com designates 209.85.223.173 as permitted sender)
Received: from [209.85.223.173] (HELO mail-iw0-f173.google.com) (209.85.223.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Sep 2009 10:40:52 +0000
Received: by iwn3 with SMTP id 3so179010iwn.2
        for <multiple recipients>; Mon, 07 Sep 2009 03:40:31 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:cc:content-type;
        bh=FlQO013bEp/UozZbSk3ESR5cEMRyY6BvXtMxHVcEaNA=;
        b=Pyour4WbNS7lwDmI1SqNU5UBHUoEN7LoXy4bkmGoBRYqf0iMZguTZDMTV0Nfls4JZk
         TAChETgqVZ5cEDD4gdU8l+/SAoyB+KehGORCZy3W1kK9hn508OZCrtd7YVF7o4QBy2ig
         K6fRAKxYR0IiB1eXuRE6yLWyVGWbbghsnF7hc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:cc:content-type;
        b=uzZcKPPs/WH28Ng5snLQ38plzzHrBngGAp5XocQdfNGwfJuUFu2XhZqZBrrYL46rtf
         DTsAfFuYTCrF+9KYJIqyBZGnhYVnszOsOUpWxyUkCZPoLRBBC2rSsZWkST33vYGN2x62
         zKb9Dey+iLxCc6A/nh4G8MAF86xQcoQv2tTfA=
MIME-Version: 1.0
Received: by 10.231.26.78 with SMTP id d14mr11342557ibc.30.1252320031750; Mon, 
	07 Sep 2009 03:40:31 -0700 (PDT)
Date: Mon, 7 Sep 2009 19:40:29 +0900
Message-ID: <e497591c0909070340t46f7f5afvc86ef66c02da8693@mail.gmail.com>
Subject: Please interpret the output -please help - going mad
From: Harshit Kumar <hkumar.arora@gmail.com>
To: general@hadoop.apache.org
Cc: common-user@hadoop.apache.org, common-dev@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001517740c1aa34a710472fa7a44
X-Virus-Checked: Checked by ClamAV on apache.org

--001517740c1aa34a710472fa7a44
Content-Type: text/plain; charset=UTF-8

Hi
I am going mad, pulling my hairs now, past 2 days, trying to figure out the
output by this map-reduce program. Please help or you can find me in asylum.

in the map function below, output.collect(Text,Duo) - Duo is a custom input
format, map functions trasnfers "hello, duo("hello,hello")" key-value pair
to the reducer.

The reducer ouptuts "<hello,2>" as the key-value pair, however, the output
is actually different.

The input file is
I love you
you love me
we love eachother


The output is
I I love you
we we love eachother
you you love me

Please make me understand,  why the output is not <hello,2>.


---------------------------
Here is the code

public class SplitFile {

public static class Map extends MapReduceBase implements
Mapper<LongWritable, Text, Text, Duo> {


       public void map(LongWritable key, Text value, OutputCollector<Text,
Duo> output, Reporter reporter) throws IOException {
           String property=new String();
           String object = new String();
       String line = value.toString();
           StringTokenizer st = new StringTokenizer(line);
           Text subject = new Text(st.nextToken());
           property = st.nextToken();
   object = st.nextToken();
   Duo d = new Duo(property,object);
               output.collect(new Text("hello"),new Duo("hello","hello"));

       }
    }


    public static class Reduce extends MapReduceBase implements
Reducer<Text, Duo, Text, IntWritable> {

       public void reduce(Text key, Iterator<Duo> values,
OutputCollector<Text, IntWritable> output, Reporter reporter) throws
IOException {
     /*  String finalString=null;
      System.out.println("key"+key);
      Duo d = new Duo();
  while (values.hasNext()) {
           d = values.next();
           System.out.println("duo object received is = "+d.toString());
           finalString = d.getProperty()+ " " + d.getObject();
           //System.out.println("final String ="+finalString);
           }*/
  output.collect(new Text("hello"), new IntWritable(2));
       }
    }


    public static void main(String[] args) throws Exception {

       JobConf conf = new JobConf(SplitFile.class);
       conf.setJobName("SplitFile");

       conf.setMapOutputKeyClass(Text.class);
       conf.setMapOutputValueClass(Duo.class);

       conf.setOutputKeyClass(Text.class);
       conf.setOutputValueClass(IntWritable.class);

       conf.setMapperClass(Map.class);
       //conf.setCombinerClass(Reduce.class);
       conf.setReducerClass(Reduce.class);

       conf.setInputFormat(TextInputFormat.class);
       conf.setOutputFormat(TextOutputFormat.class);

       FileInputFormat.setInputPaths(conf, new Path("In"));
       FileOutputFormat.setOutputPath(conf, new Path("Out"));
       //System.out.println("2");
       JobClient.runJob(conf);
    }

}

Thanks and Regards
H. Kumar

--001517740c1aa34a710472fa7a44--

From general-return-490-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 08 20:30:40 2009
Return-Path: <general-return-490-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 80403 invoked from network); 8 Sep 2009 20:30:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Sep 2009 20:30:40 -0000
Received: (qmail 62866 invoked by uid 500); 8 Sep 2009 20:30:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62807 invoked by uid 500); 8 Sep 2009 20:30:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62797 invoked by uid 99); 8 Sep 2009 20:30:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Sep 2009 20:30:39 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 08 Sep 2009 20:30:35 +0000
Received: (qmail 80193 invoked by uid 99); 8 Sep 2009 20:30:14 -0000
Received: from localhost.apache.org (HELO [192.168.168.105]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Sep 2009 20:30:14 +0000
Message-ID: <4AA6BED4.9050007@apache.org>
Date: Tue, 08 Sep 2009 13:30:12 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: [Fwd: [VOTE] Avro release 1.1.0 (candidate 1)]
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

FYI, I have started a new release vote on avro-dev.

If you have time, please evaluate the candidate and vote there.

Thanks,

Doug

-------- Original Message --------
Subject: [VOTE] Avro release 1.1.0 (candidate 1)
Date: Tue, 08 Sep 2009 13:28:23 -0700
From: Doug Cutting <cutting@apache.org>
To: avro-dev@hadoop.apache.org

I have created a second candidate build for Avro release 1.1.0.

This fixes AVRO-112.

Please download, test and vote by 11 September.

  http://people.apache.org/~cutting/avro-1.1.0-candidate-1/

Thanks!

Doug

From general-return-491-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Sep 09 00:09:23 2009
Return-Path: <general-return-491-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16010 invoked from network); 9 Sep 2009 00:09:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Sep 2009 00:09:23 -0000
Received: (qmail 45687 invoked by uid 500); 9 Sep 2009 00:09:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45551 invoked by uid 500); 9 Sep 2009 00:09:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45459 invoked by uid 99); 9 Sep 2009 00:09:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Sep 2009 00:09:21 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 209.85.221.171 as permitted sender)
Received: from [209.85.221.171] (HELO mail-qy0-f171.google.com) (209.85.221.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Sep 2009 00:09:10 +0000
Received: by qyk1 with SMTP id 1so2973882qyk.22
        for <multiple recipients>; Tue, 08 Sep 2009 17:08:49 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        bh=IZdTJuFHN8E9bH3bf+p67dw0/og8Xu6AqXZx/5oxExg=;
        b=APpeRqDfCtN9TUYupwqvIYHsj7ThFY3hKCVvA7yotlcek2ce3/wiE1qhxpxiP2blj3
         fZbf1kib5jwLTlvajJVsreU4XFvzMKFPWiDoiBv3x3hxINjLI38K4E3RsIOhW0mUX4U5
         uXSh8RxFaVNDZqUnHgk6+um4qsEMnZ2oG/brg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        b=diir3tC5NBHdpZ4BvgCSrEm4soCmr+4YHfhZO0vMi2b083oEA6VWR0SIJNsEgNUYz4
         cQ8LWENaNO1mvMlJIPAPBWfLAAMutHkD2XDInet/o9KnZr1FARpbiSjNydt6yxErYUSY
         hMYgpFxRUIUWZMsGedbFmEVPFYJ1tRSWl93Kc=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
Received: by 10.229.1.65 with SMTP id 1mr4399812qce.20.1252454926276; Tue, 08 
	Sep 2009 17:08:46 -0700 (PDT)
Date: Tue, 8 Sep 2009 17:08:46 -0700
X-Google-Sender-Auth: 52c4c7969d7363e3
Message-ID: <7c962aed0909081708u530477c8m44f223ed04d9df83@mail.gmail.com>
Subject: [ANN] HBase 0.20.0 available for download
From: stack <stack@duboce.net>
To: HBase Dev List <hbase-dev@hadoop.apache.org>, hbase-user@hadoop.apache.org, 
	general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163646c98afa58b4047319e2a1
X-Virus-Checked: Checked by ClamAV on apache.org

--00163646c98afa58b4047319e2a1
Content-Type: text/plain; charset=ISO-8859-1

HBase 0.20.0 is available for download:

   http://hadoop.apache.org/hbase/releases.html

The Release Notes are available here: http://su.pr/2sjrkf

This HBase is faster, slimmer, sweeter smelling, and more robust than
previous versions.  We recommend that all upgrade to this release.

HBase 0.20.0 runs on Hadoop 0.20.0.  A lot has changed since 0.19.x
including configuration fundamentals.  Be sure to read the 'Getting Started'
documentation:
http://hadoop.apache.org/hbase/docs/r0.20.0/api/overview-summary.html#overview_description

If you wish to bring your 0.19.x hbase data forward to 0.20.0, you will need
to run a migration.  See http://wiki.apache.org/hadoop/Hbase/HowToMigrate.
First read the overview and then go to the section, 'From 0.19.x to 0.20.x'.

Thanks to all who contributed to this release

Yours,
The HBasistas

P.S. 0.20.0 Highlights include:

+ Much improved performance
+ Master is no longer SPOF
+ Rolling restarts -- no need to take down whole cluster updating config. or
making minor upgrades
+ A new, more comprehensive API (The old API is still present but
deprecated)
+ Improved mapreduce connectors updated to suit Hadoop 0.20.0 new mapreduce
API
+ New contrib package with updated Transactional HBase (THBase) and Indexed
HBase (ITHBase) as well as a new REST gateway called stargate
+ And, as they say on the radio, "much, much more".

--00163646c98afa58b4047319e2a1--

From general-return-492-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Sep 09 17:15:36 2009
Return-Path: <general-return-492-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 18864 invoked from network); 9 Sep 2009 17:15:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Sep 2009 17:15:36 -0000
Received: (qmail 79697 invoked by uid 500); 9 Sep 2009 17:15:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79598 invoked by uid 500); 9 Sep 2009 17:15:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 79588 invoked by uid 99); 9 Sep 2009 17:15:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Sep 2009 17:15:35 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [209.191.125.37] (HELO web38406.mail.mud.yahoo.com) (209.191.125.37)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 09 Sep 2009 17:15:24 +0000
Received: (qmail 99889 invoked by uid 60001); 9 Sep 2009 17:15:02 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1252516502; bh=bbvl3oXfTwIRJStYCXWxKPCuaxZUyj2ANM5MQmRbZ50=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=XxYZ+L5mxWmtt5VACfR/QBkZUJ26g9LKGlQa15+55XTUR5rvQNn50yj9WdYMcRW724tLhsaYozwBMNjBdCLbcuVJ+tcZEPZsjP2rJlYo5u2t4en8wN0RnyRMmfshMk5IVW26U/SOrFK+pZBXi6b3wJtLrFJFxbk7bSjnHYQWLxE=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=O/Mhvqmb+B+Rx0cd5siV6Ejuae3c+kwFtQDIMWNT1amuKHpkw/rW07hs2wEd637DhaHKQe+XubZNq3CK0kJFW+W0dQkcuzCCBHv7qBNjaBJu6WL3EmjjNMDYL/cggQlPFV1tmyEnD5GKIyiL+abQSWlRpRcTYTAOJ1myEOnUPDU=;
Message-ID: <14015.97592.qm@web38406.mail.mud.yahoo.com>
X-YMail-OSG: G39EPv8VM1ldhcyUqRXXoUaiwCYGaDmsDJ0HDNqKnh4_jkgHCV9CECfVCg5M.0z3GWVEty0..1H6zFjXtXayNoZGI3fNR.CCEelUR6AZGleEccEjCk.u66loDUHqeMt4hV0qyCxvc_6IDZobKOpFVpeg5kdJw_AzdBJYTuzvd0CJxeD1UphmCDOF0nSHrBXHuhxrmRq8RnD7SSQOKx2cR6p8G0uqNPHnMGZNa.R0cYLt783nWZ9tiJEId8wiuoRfjSL7yJOHt2u59tfCQmkKpeX1x3armPeBG6NgAZYk1Q3D.5fP1urMpTv_9kgn3Pa5JzUgzl1CskjYR_F8Zd20dd6khbmKt2r7AMVsylk-
Received: from [129.170.212.235] by web38406.mail.mud.yahoo.com via HTTP; Wed, 09 Sep 2009 10:15:01 PDT
X-Mailer: YahooMailRC/157.15 YahooMailWebService/0.7.338.2
Date: Wed, 9 Sep 2009 10:15:01 -0700 (PDT)
From: himanshu chandola <himanshu_coolguy@yahoo.com>
Subject: multicore node clusters
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,
I've a cluster where every node is a multicore. From doing internet searches I've figured out that I definitely need to change mapred.tasktracker.tasks.maximum according to the number of clusters. But there are definitely other things that I would like to change for example mapred.map.tasks. Can someone point me out the list of things I should change to get the best performance out of my cluster ?

thanks

 Morpheus: Do you believe in fate, Neo?
Neo: No.
Morpheus: Why Not?
Neo: Because I don't like the idea that I'm not in control of my life.



      

From general-return-493-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Sep 09 23:24:52 2009
Return-Path: <general-return-493-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27242 invoked from network); 9 Sep 2009 23:24:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Sep 2009 23:24:51 -0000
Received: (qmail 71702 invoked by uid 500); 9 Sep 2009 23:24:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71628 invoked by uid 500); 9 Sep 2009 23:24:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71618 invoked by uid 99); 9 Sep 2009 23:24:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Sep 2009 23:24:50 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of gmackey@cs.ucf.edu designates 132.170.108.1 as permitted sender)
Received: from [132.170.108.1] (HELO longwood.cs.ucf.edu) (132.170.108.1)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Sep 2009 23:24:39 +0000
Received: from longwood.cs.ucf.edu (localhost [127.0.0.1])
	by longwood.cs.ucf.edu (8.14.1/8.14.1) with ESMTP id n89NOBwx015056
	for <general@hadoop.apache.org>; Wed, 9 Sep 2009 19:24:11 -0400 (EDT)
Received: (from prayer@localhost)
	by longwood.cs.ucf.edu (8.14.1/8.14.1/Submit) id n89NOA85015055
	for general@hadoop.apache.org; Wed, 9 Sep 2009 19:24:10 -0400 (EDT)
X-Authentication-Warning: longwood.cs.ucf.edu: prayer set sender to gmackey@cs.ucf.edu using -f
From: gmackey@cs.ucf.edu
To: general@hadoop.apache.org
Subject: Perceus and Hadoop
Date: 09 Sep 2009 19:24:10 -0400
X-Mailer: Prayer v1.0.16
X-Originating-IP: [192.12.184.2]
Message-ID: <Prayer.1.0.16.0909091924100.14684@longwood.cs.ucf.edu>
Mime-Version: 1.0
Content-Type: text/plain; format=flowed; charset=ISO-8859-1
X-Spam-Level: *
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on gondor.cs.ucf.edu
X-Virus-Scanned: clamav-milter 0.95.2 at longwood
X-Virus-Status: Clean
X-Virus-Checked: Checked by ClamAV on apache.org
X-Old-Spam-Status: No, score=1.6 required=6.0 tests=FH_RELAY_NODNS,RDNS_NONE
	autolearn=no version=3.2.5

Has anyone experimented with using the perceus provisioning system and 
hadoop? I am presently trying to get the hadoop cluster up with this system 
and I am getting really weird results. sometimes the dfs will start 
properly and then other times it will not, and I never change the config 
files.

all of the hadoop components are running stateless, and the dfs is pointed 
to the two local disks on each node.

I don't believe it is my config files, because when I do manage to get the 
dfs up, it works appropriately, from being able to store files to it and 
all of the dfsadmin commands. any ideas?

thanks much all

-- 
--
Grant Mackey
PhD student Computer Engineering
University of Central Florida

From general-return-494-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Sep 10 02:37:29 2009
Return-Path: <general-return-494-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85999 invoked from network); 10 Sep 2009 02:37:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Sep 2009 02:37:28 -0000
Received: (qmail 66339 invoked by uid 500); 10 Sep 2009 02:37:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66246 invoked by uid 500); 10 Sep 2009 02:37:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66236 invoked by uid 99); 10 Sep 2009 02:37:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Sep 2009 02:37:27 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of matthew.kelcey@gmail.com designates 74.125.78.27 as permitted sender)
Received: from [74.125.78.27] (HELO ey-out-2122.google.com) (74.125.78.27)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Sep 2009 02:37:20 +0000
Received: by ey-out-2122.google.com with SMTP id 22so1274775eye.23
        for <general@hadoop.apache.org>; Wed, 09 Sep 2009 19:36:58 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=kaNh4hicnq3WtrYQdauhfmj+E1bM1V5n+WW051UK1DQ=;
        b=Ogw8lUkiry6brRJHyz2ga21N58A5DnNEbRYQ686NGqTTlesbDdG55i5f0V8a/mIRB9
         d4HwAOVUzRA3eHYyzfsQGsjBmOcLwtTjng4ARq5ybyRXa5epVY0KC61N3lcHbVf5/O35
         ltslzrRZoe+jzfqYseKeicHTB63zUbld2B00U=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=QpZwhkKRpXIlpZkRm8JkL6FTuLzqJyGjRLkH9r3SW036j4rhSIokTjt9lS0ttrTMJY
         FMCxxfiMqbWPD3wY0TCw5EPUhctarqeDby8IxivmrJuugZGHq5q+Uv4YKpr5NcwcDN2B
         l2jmhmjh+37eK67GDIPZedQsTkzzyIRZYgaEc=
MIME-Version: 1.0
Received: by 10.211.129.20 with SMTP id g20mr301842ebn.14.1252550218761; Wed, 
	09 Sep 2009 19:36:58 -0700 (PDT)
In-Reply-To: <14015.97592.qm@web38406.mail.mud.yahoo.com>
References: <14015.97592.qm@web38406.mail.mud.yahoo.com>
Date: Thu, 10 Sep 2009 12:36:58 +1000
Message-ID: <93d501de0909091936r5e234e69k80f849fb6a5ed345@mail.gmail.com>
Subject: Re: multicore node clusters
From: Mat Kelcey <matthew.kelcey@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

> I've a cluster where every node is a multicore. From doing internet searc=
hes I've figured out that I definitely need to change mapred.tasktracker.ta=
sks.maximum according to the number of clusters. But there are definitely o=
ther things that I would like to change for example mapred.map.tasks. Can s=
omeone point me out the list of things I should change to get the best perf=
ormance out of my cluster ?

nothing will give you better results than benchmarking with some jobs
indicative to your domain!

From general-return-495-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Sep 10 13:32:04 2009
Return-Path: <general-return-495-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72741 invoked from network); 10 Sep 2009 13:32:04 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Sep 2009 13:32:04 -0000
Received: (qmail 71315 invoked by uid 500); 10 Sep 2009 13:32:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71245 invoked by uid 500); 10 Sep 2009 13:32:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71230 invoked by uid 99); 10 Sep 2009 13:32:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Sep 2009 13:32:03 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cpbhagtani@gmail.com designates 209.85.216.183 as permitted sender)
Received: from [209.85.216.183] (HELO mail-px0-f183.google.com) (209.85.216.183)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Sep 2009 13:31:51 +0000
Received: by pxi13 with SMTP id 13so91420pxi.13
        for <general@hadoop.apache.org>; Thu, 10 Sep 2009 06:31:30 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=WTPDTWWZp7Na9Sc+F6oBBAavPv7ukKNP5mhJzkTBkAE=;
        b=Q6ueKA291/XdxN8Nmv+RWkeVeeOBKpvyDdQreS7QoK5GjyOQEMyYPVYz5pZziAZrVo
         ys02E5psxCBE4/+sPAQh5KGvTXLB0kwUtu29GV/aJd5dPHwvVXZ8OyHQ9SW0BN/3mVb+
         PPV5pyehl1l9lxdrcrb1fkAXZDat/VMHLAr7Y=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Ua8r+zESnEkh7tTFY6GT2JZ6IzD/OxD7KECTf6B6OWGnbeT6rUTOQL+vgmp2PJVbL5
         TZJbzViB3Y/I73HdHCkReefT1icZU0gksQm/CR5++BWvJJOZZbg9Vs6+CJ3P3Hf4p+wt
         8x3euCUBKNsML/oM2yIsrn8O/LDdU0e2ESH24=
MIME-Version: 1.0
Received: by 10.115.149.4 with SMTP id b4mr2929946wao.18.1252589489698; Thu, 
	10 Sep 2009 06:31:29 -0700 (PDT)
In-Reply-To: <93d501de0909091936r5e234e69k80f849fb6a5ed345@mail.gmail.com>
References: <14015.97592.qm@web38406.mail.mud.yahoo.com>
	 <93d501de0909091936r5e234e69k80f849fb6a5ed345@mail.gmail.com>
Date: Thu, 10 Sep 2009 19:01:29 +0530
Message-ID: <4061df20909100631g5b6b52eeh90a3af5d3b9549ef@mail.gmail.com>
Subject: Re: multicore node clusters
From: Chandraprakash Bhagtani <cpbhagtani@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e646a49a953a250473393790
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e646a49a953a250473393790
Content-Type: text/plain; charset=ISO-8859-1

Hi,

You should definitely change mapred.tasktracker.map/reduce.tasks.maximum. If
your tasks are more CPU bound then you should run the tasks equal to the
number of CPU cores otherwise you can run more tasks than cores. You can
determine CPU and memory usage by running "top" command on datanodes. You
should also take care of following configuration parameters to achieve best
performance

*mapred.compress.map.output:* Faster data transfer (from mapper to
reducers), saves disk space, faster disk writing. Extra time in compression
and decompression

*io.sort.mb: *If you have idle physical memory after running all tasks you
can increase this value. But swap space should not be used since it makes it
slow.*

**io.sort.factor: *If your map tasks have large number of spills* *then you
should increase this value.It also helps in merging at reducers.

*mapred.job.reuse.jvm.num.tasks: *The overhead of JVM creation for each task
is around 1 second. So for the tasks which live for seconds or a few minutes
and have lengthy initialization, this value can be increased to gain
performance.

*mapred.reduce.parallel.copies: *For Large jobs (the jobs in which map
output is very large), value of this property can be increased keeping in
mind that it will increase the total CPU usage.*

**mapred.map/reduce.tasks.speculative.execution: *set to false to gain high
throughput.

*dfs.block.size* or *mapred.min.split.size* or *mapred.max.split.size* : to
control the number of maps

On Thu, Sep 10, 2009 at 8:06 AM, Mat Kelcey <matthew.kelcey@gmail.com>wrote:

> > I've a cluster where every node is a multicore. From doing internet
> searches I've figured out that I definitely need to change
> mapred.tasktracker.tasks.maximum according to the number of clusters. But
> there are definitely other things that I would like to change for example
> mapred.map.tasks. Can someone point me out the list of things I should
> change to get the best performance out of my cluster ?
>
> nothing will give you better results than benchmarking with some jobs
> indicative to your domain!
>



-- 
Thanks & Regards,
Chandra Prakash Bhagtani,

--0016e646a49a953a250473393790--

From general-return-496-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Sep 11 16:26:28 2009
Return-Path: <general-return-496-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 11677 invoked from network); 11 Sep 2009 16:26:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Sep 2009 16:26:28 -0000
Received: (qmail 50562 invoked by uid 500); 11 Sep 2009 16:26:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49893 invoked by uid 500); 11 Sep 2009 16:26:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49718 invoked by uid 99); 11 Sep 2009 16:26:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Sep 2009 16:26:24 +0000
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Sep 2009 16:26:12 +0000
Received: from sp1-ex07cas01.ds.corp.yahoo.com (sp1-ex07cas01.ds.corp.yahoo.com [216.252.116.137])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n8BGP64T099316;
	Fri, 11 Sep 2009 09:25:06 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:references:in-reply-to:accept-language:
	content-language:x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:
	content-type:content-transfer-encoding:mime-version;
	b=CrpfHne5u0w6clG27M6Qp8bfnrko9UG93nT7tbuAfQf+NaSf/KDVkW3iYBKERxB4
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas01.ds.corp.yahoo.com ([216.252.116.137]) with mapi; Fri, 11 Sep
 2009 09:25:06 -0700
From: Dekel Tankel <dekel@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>,
        "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Date: Fri, 11 Sep 2009 09:25:05 -0700
Subject: Hadoop User Group (Bay Area) - Sep 23rd at Yahoo!
Thread-Topic: Hadoop User Group (Bay Area) - Sep 23rd at Yahoo!
Thread-Index: AcoSMRRU8X5auVLKR8ufbpFTQi1hLggyrkMg
Message-ID: <46E2672895E8744A97D7577A6110858B4FD3462851@SP1-EX07VS01.ds.corp.yahoo.com>
References: <46E2672895E8744A97D7577A6110858B4E1AC69A91@SP1-EX07VS01.ds.corp.yahoo.com>
In-Reply-To: <46E2672895E8744A97D7577A6110858B4E1AC69A91@SP1-EX07VS01.ds.corp.yahoo.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Hi all,

I'd like to remind everyone that RSVP is open for the next monthly Bay Area=
 Hadoop user group organized by Yahoo!.


Agenda and registration available here

http://www.meetup.com/hadoop/calendar/11166700/

Looking forward to seeing you at September 23rd.

Dekel


From general-return-497-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Sep 11 17:34:51 2009
Return-Path: <general-return-497-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38524 invoked from network); 11 Sep 2009 17:34:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Sep 2009 17:34:51 -0000
Received: (qmail 81758 invoked by uid 500); 11 Sep 2009 17:34:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81684 invoked by uid 500); 11 Sep 2009 17:34:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81674 invoked by uid 99); 11 Sep 2009 17:34:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Sep 2009 17:34:50 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 11 Sep 2009 17:34:48 +0000
Received: (qmail 38357 invoked by uid 99); 11 Sep 2009 17:34:26 -0000
Received: from localhost.apache.org (HELO [192.168.168.105]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Sep 2009 17:34:26 +0000
Message-ID: <4AAA8A21.7040202@apache.org>
Date: Fri, 11 Sep 2009 10:34:25 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: [Fwd: [VOTE] Avro release 1.1.0 (candidate 1)]
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

FYI, I have started a new release vote on avro-dev.

If you have time, please evaluate the candidate and vote there.

Thanks,

Doug

-------- Original Message --------
Subject: [VOTE] Avro release 1.1.0 (candidate 1)
Date: Fri, 11 Sep 2009 10:31:55 -0700
From: Doug Cutting <cutting@apache.org>
To: avro-dev@hadoop.apache.org

I have created a third candidate build for Avro release 1.1.0.

This fixes AVRO-114.

Please download, test and vote by 15 September.

  http://people.apache.org/~cutting/avro-1.1.0-candidate-2/

Thanks!

Doug

From general-return-498-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Sep 11 21:41:48 2009
Return-Path: <general-return-498-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34150 invoked from network); 11 Sep 2009 21:41:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Sep 2009 21:41:48 -0000
Received: (qmail 37124 invoked by uid 500); 11 Sep 2009 21:41:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37039 invoked by uid 500); 11 Sep 2009 21:41:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37029 invoked by uid 99); 11 Sep 2009 21:41:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Sep 2009 21:41:47 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 11 Sep 2009 21:41:45 +0000
Received: (qmail 33644 invoked by uid 99); 11 Sep 2009 21:41:24 -0000
Received: from localhost.apache.org (HELO [192.168.168.105]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Sep 2009 21:41:24 +0000
Message-ID: <4AAAC403.80809@apache.org>
Date: Fri, 11 Sep 2009 14:41:23 -0700
From: Doug Cutting <cutting@apache.org>
Reply-To: general@hadoop.apache.org
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: HTTP transport?
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

I'm considering an HTTP-based transport for Avro as the preferred, 
high-performance option.

HTTP has lots of advantages.  In particular, it already has
  - lots of authentication, authorization and encryption support;
  - highly optimized servers;
  - monitoring, logging, etc.

Tomcat and other servlet containers support async NIO, where a thread is 
not required per connection.  A servlet can process bulk data with a 
single copy to and from the socket (bypassing stream buffers).  Calls 
can be multiplexed over a single HTTP connection using Comet events.

http://tomcat.apache.org/tomcat-6.0-doc/aio.html

Zero copy is not an option for servlets that generate arbitrary data, 
but one can specify a file/start/length tuple and Tomcat will use 
sendfile to write the response.  That means that while HDFS datanode 
file reads could not be done via RPC, they could be done via HTTP with 
zero-copy.  If authentication and authorization are already done in the 
HTTP server, this may not be a big loss.  The HDFS client might make two 
HTTP requests, one to read a files data, and another to read its 
checksums.  The server would then stream the entire block to the client 
using sendfile, using TCP flow control as today.

Thoughts?

Doug

From general-return-499-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Sep 12 03:49:00 2009
Return-Path: <general-return-499-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24640 invoked from network); 12 Sep 2009 03:49:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Sep 2009 03:49:00 -0000
Received: (qmail 97570 invoked by uid 500); 12 Sep 2009 03:48:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97421 invoked by uid 500); 12 Sep 2009 03:48:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97411 invoked by uid 99); 12 Sep 2009 03:48:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Sep 2009 03:48:57 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [209.191.125.32] (HELO web38401.mail.mud.yahoo.com) (209.191.125.32)
    by apache.org (qpsmtpd/0.29) with SMTP; Sat, 12 Sep 2009 03:48:47 +0000
Received: (qmail 72400 invoked by uid 60001); 12 Sep 2009 03:48:25 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1252727305; bh=y2IB/blseocl/BTpwByicTPpR0oOcoFndk/eS7L8qfI=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=HlvCOHmfMnYsYUJTjsyDtzm7pKi6Ms8CQEw6vQ0E+nU7C1266iTHmBQmo/dLAWKmbvFJP7SiTwYEm/uzRolaE2/CSa8sbUBInuHN4EnIRPwt2BkommcHPEFVjuRO8nmVI7KA9MXyXfVzv0SsiRNRDiF0pU7mxlFQKOY1/hFfqXs=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=MQeJpqXPu2n7xuw6ekpRwKRGRSBjOm1h4EUqPC0lwpw5qWlTJSfsJlZtHgxjYvAutTyp3AEth9a4Jt3/izVEFeul39D5/xHiXuN1lf4e+0h02vVonrfmhtz4nvugcJbJAhafip5HSkefYEAJZJteoiF8eTGi7QbJzdQYAKCT/I4=;
Message-ID: <699968.68276.qm@web38401.mail.mud.yahoo.com>
X-YMail-OSG: VSNAUmIVM1maBMB70H9VQPicGHLLqo3TnBIAd6NL2r7ADCw3diJnj_c3qIjXAXyP4mk04J8Lzp7Qi33pQUgOsWyk0AdhiB1NY4PAJpxNVaRIqe44pzYLJ6jmWZZS0YH9GN2ugsYBDR1vaJG0qJBnYK_VaLG66eNMgSByWKTPhqoGd0nZvmsoia_hrjBKZDDq8psx1p678XtuzyzFb30veAeuNCbU8wX3wf6B_Zb3NGr.kX3xcaVlaX_chFWlR7MGxqhyk5nUwHFlTJgut1PvbEpGDwMntb5Xp54byBa92fs28jc-
Received: from [75.69.131.160] by web38401.mail.mud.yahoo.com via HTTP; Fri, 11 Sep 2009 20:48:25 PDT
X-Mailer: YahooMailRC/157.18 YahooMailWebService/0.7.347.2
Date: Fri, 11 Sep 2009 20:48:25 -0700 (PDT)
From: himanshu chandola <himanshu_coolguy@yahoo.com>
Subject: changing map and reduce model
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Ppl,
What would Ive to do to make the following model work for the word count application:
The input is a text file . What I would like to do is that every map task gets a list of lines equal to number of lines/ number of map tasks allowed.

I think its better for the performance of the app, that a map task get a list of records rather than just one record ., so that you save time on starting jvm and the functional overheads.

It seems to me that given that we 

 Morpheus: Do you believe in fate, Neo?
Neo: No.
Morpheus: Why Not?
Neo: Because I don't like the idea that I'm not in control of my life.



      

From general-return-500-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Sep 12 05:02:14 2009
Return-Path: <general-return-500-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 36379 invoked from network); 12 Sep 2009 05:02:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Sep 2009 05:02:14 -0000
Received: (qmail 35644 invoked by uid 500); 12 Sep 2009 05:02:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 35554 invoked by uid 500); 12 Sep 2009 05:02:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 35544 invoked by uid 99); 12 Sep 2009 05:02:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Sep 2009 05:02:13 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.191.125.37] (HELO web38406.mail.mud.yahoo.com) (209.191.125.37)
    by apache.org (qpsmtpd/0.29) with SMTP; Sat, 12 Sep 2009 05:02:04 +0000
Received: (qmail 59071 invoked by uid 60001); 12 Sep 2009 05:01:43 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1252731703; bh=yILGDXczNQyOE7b2K9T9EwbnkI3RJ6wK1VowupsFHcQ=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=a9EyPphafd6PKKlYUX44pOmSoChKS/sOt1kzm3evmPoEuGEJoZkWOMATS98KSXgyVdwJpR3T6kzX85StMEDZ0REAXp6LXLNKrt+4txWhgTxTQQiwaR/7vP4xfIdv0XTYFtX8oJ1Upv7cdI5vQJn7F+cnkDC7S+qJMi5/SLCZl0k=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=Ze1Yb54746gJTKnzUN9RoKVOCIlSGtD8iX2OQFCjO+5cNwwxnDcLwvPl3xHsvxrtkmHRSEqHCzCdzXtdN1DC9jw9ZS1ZlhCBsSCwveDsTyX2pXeDp4C/ZmAibD8R8rWKEMovBjo2cvnpNBf2Bd+4iSMp7R+lEFjK746gxPs/X/g=;
Message-ID: <17178.56716.qm@web38406.mail.mud.yahoo.com>
X-YMail-OSG: IrgAUE8VM1nnZH9OdNP1f.7QJa3o7.nla9MJ3C0qXQ9p7fCh2qP44bcJw6sjVMbryzkuWiYZajDLzeMAbqtJC7y7R.mE8AmaTmXgCfrFwoCkQIHoBx30VDw6RUaDPQAjVXW.UNLzWeNqNVJT9bXYhhLWaspzQe.n.yk3fl6zxPjMb8LhJWyzctTXXVKGYzdp8bsfe0puqs3Wi6r07VRAlWFpOpJM16Kb3XZhW5m3PIKDUtJ65SZvz_dYGDYkqAt3EbMChIlcsbZlGjtVZaZ2GxHKYfiG53UIbCg8NP6yQW4ELHb7ILHA8.EDYc_zXeJW3LQSlAGVmAGAWpxFggU6BQTTmncihvWcaGP_
Received: from [75.69.131.160] by web38406.mail.mud.yahoo.com via HTTP; Fri, 11 Sep 2009 22:01:42 PDT
X-Mailer: YahooMailRC/157.18 YahooMailWebService/0.7.347.2
Date: Fri, 11 Sep 2009 22:01:42 -0700 (PDT)
From: himanshu chandola <himanshu_coolguy@yahoo.com>
Subject: nodes lying idle
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org

Hi everyone,
Ive a cluster of 40 nodes. The input file has 2^18 lines and every line is an input to a map job. Every node is a quad core and hence I've set mapred.tasktracker.map/reduce.tasks.maximum to a value greater than 4. The first 20 nodes are showing hadoop jobs taking 100% but with only one process running while since its a quad core I would've liked to see 4 java processes taking 100% (there are 5 java processes on this system but 4 are idle and only one is taking 100% or 1 cpu). For the last half of the nodes, the cpu usage of hadoop processes is 0. This is really strange since my map tasks are processing in a very slow way and I wouldve liked to use all nodes and all the cores. 

What could possibly be wrong ? It would really help if anyone could suggest .

thanks

H

 Morpheus: Do you believe in fate, Neo?
Neo: No.
Morpheus: Why Not?
Neo: Because I don't like the idea that I'm not in control of my life.



      

From general-return-501-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Sep 12 05:50:13 2009
Return-Path: <general-return-501-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43153 invoked from network); 12 Sep 2009 05:50:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Sep 2009 05:50:13 -0000
Received: (qmail 56322 invoked by uid 500); 12 Sep 2009 05:50:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56232 invoked by uid 500); 12 Sep 2009 05:50:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56221 invoked by uid 99); 12 Sep 2009 05:50:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Sep 2009 05:50:12 +0000
X-ASF-Spam-Status: No, hits=3.7 required=10.0
	tests=HTML_MESSAGE,SPF_PASS,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cpbhagtani@gmail.com designates 209.85.216.204 as permitted sender)
Received: from [209.85.216.204] (HELO mail-px0-f204.google.com) (209.85.216.204)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Sep 2009 05:50:04 +0000
Received: by pxi42 with SMTP id 42so1404555pxi.20
        for <general@hadoop.apache.org>; Fri, 11 Sep 2009 22:49:44 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=TzSgSukWm4ThPOdWuqFOeF4yztmSVJVQBjNWUZkV7O0=;
        b=xfzOWNxKvtFaxnU1b0Wx9MzNKKsFJx9x4HVSQmxnyb6aoXRJNGXM3wQFQavKZrYoX4
         BmrG4SDMJmqNKS5xXCcMXMIzIfTZz4qh5ZXiCol+/CEYTk6bzW8LpNvdb8Cn0nNPex62
         YgaI1MXbJzQj3tWtLBfvLPRpul7rMTqvG7G4M=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=wmn49+9Js4xTGgjWBHO8Si3dgwEwx8L27Pca0l/4pWx58FWiIzubYCrRsT2eomOYn4
         wBHKs1FQJUiPUrUvcbXkohBs9+gr1vl4FIl0IU8tnwv+E078pjrEnA9Kd07MZJQIXeiG
         f+0ZlL+odtwXWq6J8MUrGFGjODbADyd2TUxJs=
MIME-Version: 1.0
Received: by 10.114.116.12 with SMTP id o12mr6975576wac.83.1252734581975; Fri, 
	11 Sep 2009 22:49:41 -0700 (PDT)
In-Reply-To: <17178.56716.qm@web38406.mail.mud.yahoo.com>
References: <17178.56716.qm@web38406.mail.mud.yahoo.com>
Date: Sat, 12 Sep 2009 11:19:41 +0530
Message-ID: <4061df20909112249l1aa9a5cdifa91ebbfd6e3a7e6@mail.gmail.com>
Subject: Re: nodes lying idle
From: Chandraprakash Bhagtani <cpbhagtani@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636b14b3bc1ab4004735aff2e
X-Virus-Checked: Checked by ClamAV on apache.org

--001636b14b3bc1ab4004735aff2e
Content-Type: text/plain; charset=ISO-8859-1

You need to check your cluster's Map/Reduce task  capacity. i.e. how many
Map/Reduce task can run on cluster at once. You can check it on
http://JobtrackerServerIP:50030. You should also check total number of map
tasks in your job. It should be greater than map task capacity of the
cluster.

Intially reduce tasks will be idle till first batch of map task complete.
-- 
Thanks & Regards,
Chandra Prakash Bhagtani,

On Sat, Sep 12, 2009 at 10:31 AM, himanshu chandola <
himanshu_coolguy@yahoo.com> wrote:

> Hi everyone,
> Ive a cluster of 40 nodes. The input file has 2^18 lines and every line is
> an input to a map job. Every node is a quad core and hence I've set
> mapred.tasktracker.map/reduce.tasks.maximum to a value greater than 4. The
> first 20 nodes are showing hadoop jobs taking 100% but with only one process
> running while since its a quad core I would've liked to see 4 java processes
> taking 100% (there are 5 java processes on this system but 4 are idle and
> only one is taking 100% or 1 cpu). For the last half of the nodes, the cpu
> usage of hadoop processes is 0. This is really strange since my map tasks
> are processing in a very slow way and I wouldve liked to use all nodes and
> all the cores.
>
> What could possibly be wrong ? It would really help if anyone could suggest
> .
>
> thanks
>
> H
>
>  Morpheus: Do you believe in fate, Neo?
> Neo: No.
> Morpheus: Why Not?
> Neo: Because I don't like the idea that I'm not in control of my life.
>
>
>
>
>

--001636b14b3bc1ab4004735aff2e--

From general-return-502-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Sep 12 06:55:51 2009
Return-Path: <general-return-502-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94644 invoked from network); 12 Sep 2009 06:55:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Sep 2009 06:55:51 -0000
Received: (qmail 76168 invoked by uid 500); 12 Sep 2009 06:55:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76099 invoked by uid 500); 12 Sep 2009 06:55:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76089 invoked by uid 99); 12 Sep 2009 06:55:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Sep 2009 06:55:50 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=SPF_PASS,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [209.191.125.39] (HELO web38408.mail.mud.yahoo.com) (209.191.125.39)
    by apache.org (qpsmtpd/0.29) with SMTP; Sat, 12 Sep 2009 06:55:39 +0000
Received: (qmail 50117 invoked by uid 60001); 12 Sep 2009 06:55:16 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1252738516; bh=Ik9BBY8fVnB5BAjG3aq5rjP4G0mxyscER42ZasQ3ZUI=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=1524owITrXlcPbEz/YYbI0Pf1OskCj4x1368RilNFbXaQ1Wf89ks9g9Cd3oB1H3QD4DLpyOfKLjV6214Zj5Fje3ImnruhGSZszy7MFxYdYS2Y6YIvTgyzeHUgczFejVoE7VV5ShFgajENOZmHx60TN04L2V3i8UAGHWnP4zqC24=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=1pUR7uLNFxZJVYVi+mhlSRipcE6dy6dkvShnJIfaK4ITkZ002H8kTiCgzMTzoaUQxm+auDqVHXKxOEcqYQDwB8JOijkxo6eNtiB5h/VS8oaSYwcPtRTpHE+SgPlJUCmxnHHT8TD07lHcE+MDD30pEHeIrFUPi9JLyrdbbxkcPCw=;
Message-ID: <442272.48575.qm@web38408.mail.mud.yahoo.com>
X-YMail-OSG: 2iyMPTcVM1kBRCcQzYPyqbAPXoTxkTgHES67oBvzr_NPmQDeIY1qL_nVxmNHUwxXNJ9ON0LCTsOO.MoWlNpQAS2cjucVMmXyNI_KtyN5_KPxjox0bgzuRdALYz2AGUv6.V329UUWyzVnQOpJPTjmSoX8w1Kna0hqe6.9.TfOyPdP2lYrs5vQcKOkm7ZV8CDoREPo0slQPg5QwQFSMv2PmrhggbPreBUhATSrwRT320G1UQOft1ChSs4oU_mkBJiTTfIDNL2DrNx6oAn_JfWTprWEKQA8MERUYerJvqrlF9YQEGxubDAFyS4gyM6ElFAjf65o0afr3LdVvqJalhei.bde0A0o8L0JSWka3xPzWvR_zjp5rLOPhEvPu4sLvN6vPigf
Received: from [75.69.131.160] by web38408.mail.mud.yahoo.com via HTTP; Fri, 11 Sep 2009 23:55:16 PDT
X-Mailer: YahooMailRC/157.18 YahooMailWebService/0.7.347.2
References: <17178.56716.qm@web38406.mail.mud.yahoo.com> <4061df20909112249l1aa9a5cdifa91ebbfd6e3a7e6@mail.gmail.com>
Date: Fri, 11 Sep 2009 23:55:16 -0700 (PDT)
From: himanshu chandola <himanshu_coolguy@yahoo.com>
Subject: Re: nodes lying idle
To: general@hadoop.apache.org
In-Reply-To: <4061df20909112249l1aa9a5cdifa91ebbfd6e3a7e6@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks for the tip. 
So is the value mapred.tasktracker.map/reduce.tasks.maximum for the entire cluster ? I had set the map.tasks.maximum to 6 and hitting the web interface it shows up that total map tasks for my job is just 6. My tasks are cpu intensive and hence I would like each of my quad core nodes to be running 4 hadoop map tasks atleast . The whole cluster is running just 6 and each of these 6 nodes is  running 1 each.

 Morpheus: Do you believe in fate, Neo?
Neo: No.
Morpheus: Why Not?
Neo: Because I don't like the idea that I'm not in control of my life.



----- Original Message ----
From: Chandraprakash Bhagtani <cpbhagtani@gmail.com>
To: general@hadoop.apache.org
Sent: Saturday, September 12, 2009 1:49:41 AM
Subject: Re: nodes lying idle

You need to check your cluster's Map/Reduce task  capacity. i.e. how many
Map/Reduce task can run on cluster at once. You can check it on
http://JobtrackerServerIP:50030. You should also check total number of map
tasks in your job. It should be greater than map task capacity of the
cluster.

Intially reduce tasks will be idle till first batch of map task complete.
-- 
Thanks & Regards,
Chandra Prakash Bhagtani,

On Sat, Sep 12, 2009 at 10:31 AM, himanshu chandola <
himanshu_coolguy@yahoo.com> wrote:

> Hi everyone,
> Ive a cluster of 40 nodes. The input file has 2^18 lines and every line is
> an input to a map job. Every node is a quad core and hence I've set
> mapred.tasktracker.map/reduce.tasks.maximum to a value greater than 4. The
> first 20 nodes are showing hadoop jobs taking 100% but with only one process
> running while since its a quad core I would've liked to see 4 java processes
> taking 100% (there are 5 java processes on this system but 4 are idle and
> only one is taking 100% or 1 cpu). For the last half of the nodes, the cpu
> usage of hadoop processes is 0. This is really strange since my map tasks
> are processing in a very slow way and I wouldve liked to use all nodes and
> all the cores.
>
> What could possibly be wrong ? It would really help if anyone could suggest
> .
>
> thanks
>
> H
>
>  Morpheus: Do you believe in fate, Neo?
> Neo: No.
> Morpheus: Why Not?
> Neo: Because I don't like the idea that I'm not in control of my life.
>
>
>
>
>



      

From general-return-503-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Sep 12 12:22:42 2009
Return-Path: <general-return-503-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 48879 invoked from network); 12 Sep 2009 12:22:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Sep 2009 12:22:42 -0000
Received: (qmail 5944 invoked by uid 500); 12 Sep 2009 12:22:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5881 invoked by uid 500); 12 Sep 2009 12:22:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5871 invoked by uid 99); 12 Sep 2009 12:22:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Sep 2009 12:22:41 +0000
X-ASF-Spam-Status: No, hits=3.7 required=10.0
	tests=HTML_MESSAGE,SPF_PASS,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cpbhagtani@gmail.com designates 209.85.222.204 as permitted sender)
Received: from [209.85.222.204] (HELO mail-pz0-f204.google.com) (209.85.222.204)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Sep 2009 12:22:33 +0000
Received: by pzk42 with SMTP id 42so1541279pzk.31
        for <general@hadoop.apache.org>; Sat, 12 Sep 2009 05:22:13 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=bLbsPG/0hj0g8dTOSaNCDM1A5l9ZK0/Kk1btxDf1oBQ=;
        b=TEY0Iti17sMM8nM2xYMcCubV3ZOgeVnsvKNsKQhn5qcsWMrj8onpS2SvUGPTGxvqMz
         ARLjoikAB25Bv+yGPg/WQqRLVlin2N/KTj1VZOKSn4ureq6v2dZ9JX4CHjZoBp20QbLN
         paKwk4iU7Pl5vIw76jfrD39w1PrPtVUBbdo+Q=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=KCTmIMR6k0uNHBzwAKUQZ549tyDB/5kF7+Y8MscMx9SLVz4Lwf3h6J7zgKVXBVpFR8
         ygTzt4NM9O5obdJaF8YolmdL+7nQbXKSN9B9+j93tK2nzl8QOz7mBk7Nf1Bfe97U35Nx
         knJKO4hXu2e2+XiCvyYEhMbigOWhaRud5OFaw=
MIME-Version: 1.0
Received: by 10.115.113.4 with SMTP id q4mr7532416wam.54.1252758133077; Sat, 
	12 Sep 2009 05:22:13 -0700 (PDT)
In-Reply-To: <442272.48575.qm@web38408.mail.mud.yahoo.com>
References: <17178.56716.qm@web38406.mail.mud.yahoo.com>
	 <4061df20909112249l1aa9a5cdifa91ebbfd6e3a7e6@mail.gmail.com>
	 <442272.48575.qm@web38408.mail.mud.yahoo.com>
Date: Sat, 12 Sep 2009 17:52:13 +0530
Message-ID: <4061df20909120522w437e0953g1e1524281bf275af@mail.gmail.com>
Subject: Re: nodes lying idle
From: Chandraprakash Bhagtani <cpbhagtani@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64be86282fb970473607b44
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e64be86282fb970473607b44
Content-Type: text/plain; charset=ISO-8859-1

no *mapred.tasktracker.map/reduce.tasks.maximum* value is for single
datanode. i.e. this many mappers and reducers will run on single datanode.
for example if you have set

*mapred.tasktracker.map.tasks.maximum = 4
**mapred.tasktracker.reduce.tasks.maximum = 4*
and no. of datanodes = 40

then entire cluster's map task capacity = 4*40 = 160
*map.tasks.maximum* = 6 means only 6 maps will run for your job which will
definitely not use all of your cluster resources.

what is the size of your data? and what is your cluster specifications?

-- 
Thanks & Regards,
Chandra Prakash Bhagtani,


On Sat, Sep 12, 2009 at 12:25 PM, himanshu chandola <
himanshu_coolguy@yahoo.com> wrote:

> Thanks for the tip.
> So is the value mapred.tasktracker.map/reduce.tasks.maximum for the entire
> cluster ? I had set the map.tasks.maximum to 6 and hitting the web interface
> it shows up that total map tasks for my job is just 6. My tasks are cpu
> intensive and hence I would like each of my quad core nodes to be running 4
> hadoop map tasks atleast . The whole cluster is running just 6 and each of
> these 6 nodes is  running 1 each.
>
>  Morpheus: Do you believe in fate, Neo?
> Neo: No.
> Morpheus: Why Not?
> Neo: Because I don't like the idea that I'm not in control of my life.
>
>
>
> ----- Original Message ----
> From: Chandraprakash Bhagtani <cpbhagtani@gmail.com>
> To: general@hadoop.apache.org
> Sent: Saturday, September 12, 2009 1:49:41 AM
> Subject: Re: nodes lying idle
>
> You need to check your cluster's Map/Reduce task  capacity. i.e. how many
> Map/Reduce task can run on cluster at once. You can check it on
> http://JobtrackerServerIP:50030. You should also check total number of map
> tasks in your job. It should be greater than map task capacity of the
> cluster.
>
> Intially reduce tasks will be idle till first batch of map task complete.
> --
> Thanks & Regards,
> Chandra Prakash Bhagtani,
>
> On Sat, Sep 12, 2009 at 10:31 AM, himanshu chandola <
> himanshu_coolguy@yahoo.com> wrote:
>
> > Hi everyone,
> > Ive a cluster of 40 nodes. The input file has 2^18 lines and every line
> is
> > an input to a map job. Every node is a quad core and hence I've set
> > mapred.tasktracker.map/reduce.tasks.maximum to a value greater than 4.
> The
> > first 20 nodes are showing hadoop jobs taking 100% but with only one
> process
> > running while since its a quad core I would've liked to see 4 java
> processes
> > taking 100% (there are 5 java processes on this system but 4 are idle and
> > only one is taking 100% or 1 cpu). For the last half of the nodes, the
> cpu
> > usage of hadoop processes is 0. This is really strange since my map tasks
> > are processing in a very slow way and I wouldve liked to use all nodes
> and
> > all the cores.
> >
> > What could possibly be wrong ? It would really help if anyone could
> suggest
> > .
> >
> > thanks
> >
> > H
> >
> >  Morpheus: Do you believe in fate, Neo?
> > Neo: No.
> > Morpheus: Why Not?
> > Neo: Because I don't like the idea that I'm not in control of my life.
> >
> >
> >
> >
> >
>
>
>
>
>

--0016e64be86282fb970473607b44--

From general-return-504-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Sep 12 23:28:31 2009
Return-Path: <general-return-504-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60286 invoked from network); 12 Sep 2009 23:28:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Sep 2009 23:28:31 -0000
Received: (qmail 80622 invoked by uid 500); 12 Sep 2009 23:28:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80553 invoked by uid 500); 12 Sep 2009 23:28:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80543 invoked by uid 99); 12 Sep 2009 23:28:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Sep 2009 23:28:30 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of deepak.halale@gmail.com designates 209.85.216.204 as permitted sender)
Received: from [209.85.216.204] (HELO mail-px0-f204.google.com) (209.85.216.204)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Sep 2009 23:28:21 +0000
Received: by pxi42 with SMTP id 42so1765138pxi.20
        for <general@hadoop.apache.org>; Sat, 12 Sep 2009 16:28:00 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=wSKZxT+M4UYIMdHC0zdv2bWUfCidMcISBc3YitU3Uks=;
        b=feg0ajYOaEb856ihVo13W/9nnpn/HLx5R2+ueBI9unNdZkUziYHyo9FsCd3Wj9FCR3
         HvrHKNt1KpaLFrjB1rvbKLvhV7oLz46ZXTzmZnQjahD7DKOCpoOZ2BNabXEk08QW51wR
         lZ8K5yVqoZVP/A1qZJYhXabXtRI+8zR+OZZLA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=knFO3UmCvcrrH6ls9FWaLD6qWmf5lxuaX0b7hXvVEFbPR4R5XBS2J5Oh6Ec86Yidd7
         Ew5S2cqJyKzJukyIS09cznpUEAy9ItX+tKLCbmx4QpkoO1AEqh5BHSfVUQzyC8h17xFd
         D8zdnjpVY2k6rpGnshPcKEibqDjlZN+3L3Bmw=
MIME-Version: 1.0
Received: by 10.115.115.14 with SMTP id s14mr8297604wam.189.1252798079673; 
	Sat, 12 Sep 2009 16:27:59 -0700 (PDT)
Date: Sat, 12 Sep 2009 19:27:59 -0400
Message-ID: <15678300909121627w7a9c2244rc10c60e8bdbdd468@mail.gmail.com>
Subject: Hadoop Map/Reduce and Hive clarification
From: Deepak Halale <deepak.halale@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e648f5de83a775047369c817
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e648f5de83a775047369c817
Content-Type: text/plain; charset=ISO-8859-1

Hi,
I am new to Hadoop , need some clarifications
a) how to automate executing Map/Reduce jobs and also automating loading
data in Hive, do I need to create  a cron job or is there a better way.

b) I have 2 tables as the source for M/R jobs
1) Order Master and Order detail
OrderMaster has order header columns
(OrderId,CustId,PaymentMethod,DeliveryMethod etc)
OrderDetail has orders' item level information (viz.
OrderId,ItemId,Quantity,SalesPrice,CostPrice,DeliveryAddress, Delivery
State,DeliveryZip,DeliveryCountry)
The relation between Master and Detail is 1 to many and OrderId is the key.

If I generate a tab delimited file from each table, how does Reduce  is
going to aggregate the data from OrderDetail example
If I have to sum the OrderRevenue by Order.


Thanks

Deepak

--0016e648f5de83a775047369c817--

From general-return-505-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Sep 13 04:30:58 2009
Return-Path: <general-return-505-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 2733 invoked from network); 13 Sep 2009 04:30:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Sep 2009 04:30:58 -0000
Received: (qmail 76146 invoked by uid 500); 13 Sep 2009 04:30:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76065 invoked by uid 500); 13 Sep 2009 04:30:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76055 invoked by uid 99); 13 Sep 2009 04:30:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Sep 2009 04:30:56 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=SPF_PASS,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.191.125.32] (HELO web38401.mail.mud.yahoo.com) (209.191.125.32)
    by apache.org (qpsmtpd/0.29) with SMTP; Sun, 13 Sep 2009 04:30:47 +0000
Received: (qmail 60614 invoked by uid 60001); 13 Sep 2009 04:30:26 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1252816226; bh=kVQe3mvehjks9CTqYtkSAaqnky96GX8PF76Ub1PB83M=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=BZFIeicgCY0zvO271P3vdBfcfWC7vIPTre/yyDjdGqR8QUpdXTnG2mh+DJykZCrEojxHUKVvjPE1YpXVjgkvh7eZwi0dO0la8HiIDvbqS4IJoPuaDN2PPTdJsaGxEx7XoMnkK7IFRPWU9LbX1FzM4fxu3+ZWco+SxhdDQSrr2CM=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=K+ZixPQwmXE6f75/ZwrDBJgUrEh+cEf8/Zywzi6SmdeXV474R2XLxTgyh+M8uxaZxUIzk1QRiyFF0TPbnrQHMigyG1Qyltf8fXlAhx8VE76BDZ9xV61MxpCbag6QtwNv9lQIu/56xUZZcEV2Cn9pRfmdG3/uLh9uEUDGWiW9CVs=;
Message-ID: <285037.58660.qm@web38401.mail.mud.yahoo.com>
X-YMail-OSG: R79w06IVM1lrZIwC05dg4f_HSaJNRV41t5bn5e6AiAAcq07EBIm3dYonAkInvo0qeknRSIPy5DRB_w92uAkpYMQflue4w1Ec_Sx7.ozjeE9vBrOPQl6pnOd.W1hBUD7cXgETTOgaCtufudylBM3eMB7R2c_Z3YX.F72OShRK0wBJaT5ycVqW56EvDdWY9X5M1PitnhGBNNJAxgJ9DqFs__y4w5XmwSRw22DzUJgnhN3HFAw6Nt_9bDuBOBDQit9ca7opIS91k2KcGbHIRo8uW2nn.wJriu4iHBfCek239DeGYd3gdcOqYF6QEyMy5A5O4G906Hdq9n_mGhwrUPuqvJUFMjg80f.dOeTdUSbSFvr8EgsMkUZJbjn8VzY1T9afWmdP
Received: from [75.69.131.160] by web38401.mail.mud.yahoo.com via HTTP; Sat, 12 Sep 2009 21:30:26 PDT
X-Mailer: YahooMailRC/157.18 YahooMailWebService/0.7.347.2
References: <17178.56716.qm@web38406.mail.mud.yahoo.com>  <4061df20909112249l1aa9a5cdifa91ebbfd6e3a7e6@mail.gmail.com>  <442272.48575.qm@web38408.mail.mud.yahoo.com> <4061df20909120522w437e0953g1e1524281bf275af@mail.gmail.com>
Date: Sat, 12 Sep 2009 21:30:26 -0700 (PDT)
From: himanshu chandola <himanshu_coolguy@yahoo.com>
Subject: Re: nodes lying idle
To: general@hadoop.apache.org
In-Reply-To: <4061df20909120522w437e0953g1e1524281bf275af@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org

So, I used JobConf.setNumMapTasks and it worked. I used setNumMapTasks(40) and I ended up with 100 maps rather than the 6 I had initially. 
The size of my data is 32 mb but every line is converted into an object and the computations are cpu intensive so I would like to have as many map jobs as there are cores.
There is no xml entry of the type map.tasks.maximum. I'm using cloudera's distribution 0.18.3-14.

 Morpheus: Do you believe in fate, Neo?
Neo: No.
Morpheus: Why Not?
Neo: Because I don't like the idea that I'm not in control of my life.



----- Original Message ----
From: Chandraprakash Bhagtani <cpbhagtani@gmail.com>
To: general@hadoop.apache.org
Sent: Saturday, September 12, 2009 8:22:13 AM
Subject: Re: nodes lying idle

no *mapred.tasktracker.map/reduce.tasks.maximum* value is for single
datanode. i.e. this many mappers and reducers will run on single datanode.
for example if you have set

*mapred.tasktracker.map.tasks.maximum = 4
**mapred.tasktracker.reduce.tasks.maximum = 4*
and no. of datanodes = 40

then entire cluster's map task capacity = 4*40 = 160
*map.tasks.maximum* = 6 means only 6 maps will run for your job which will
definitely not use all of your cluster resources.

what is the size of your data? and what is your cluster specifications?

-- 
Thanks & Regards,
Chandra Prakash Bhagtani,


On Sat, Sep 12, 2009 at 12:25 PM, himanshu chandola <
himanshu_coolguy@yahoo.com> wrote:

> Thanks for the tip.
> So is the value mapred.tasktracker.map/reduce.tasks.maximum for the entire
> cluster ? I had set the map.tasks.maximum to 6 and hitting the web interface
> it shows up that total map tasks for my job is just 6. My tasks are cpu
> intensive and hence I would like each of my quad core nodes to be running 4
> hadoop map tasks atleast . The whole cluster is running just 6 and each of
> these 6 nodes is  running 1 each.
>
>  Morpheus: Do you believe in fate, Neo?
> Neo: No.
> Morpheus: Why Not?
> Neo: Because I don't like the idea that I'm not in control of my life.
>
>
>
> ----- Original Message ----
> From: Chandraprakash Bhagtani <cpbhagtani@gmail.com>
> To: general@hadoop.apache.org
> Sent: Saturday, September 12, 2009 1:49:41 AM
> Subject: Re: nodes lying idle
>
> You need to check your cluster's Map/Reduce task  capacity. i.e. how many
> Map/Reduce task can run on cluster at once. You can check it on
> http://JobtrackerServerIP:50030. You should also check total number of map
> tasks in your job. It should be greater than map task capacity of the
> cluster.
>
> Intially reduce tasks will be idle till first batch of map task complete.
> --
> Thanks & Regards,
> Chandra Prakash Bhagtani,
>
> On Sat, Sep 12, 2009 at 10:31 AM, himanshu chandola <
> himanshu_coolguy@yahoo.com> wrote:
>
> > Hi everyone,
> > Ive a cluster of 40 nodes. The input file has 2^18 lines and every line
> is
> > an input to a map job. Every node is a quad core and hence I've set
> > mapred.tasktracker.map/reduce.tasks.maximum to a value greater than 4.
> The
> > first 20 nodes are showing hadoop jobs taking 100% but with only one
> process
> > running while since its a quad core I would've liked to see 4 java
> processes
> > taking 100% (there are 5 java processes on this system but 4 are idle and
> > only one is taking 100% or 1 cpu). For the last half of the nodes, the
> cpu
> > usage of hadoop processes is 0. This is really strange since my map tasks
> > are processing in a very slow way and I wouldve liked to use all nodes
> and
> > all the cores.
> >
> > What could possibly be wrong ? It would really help if anyone could
> suggest
> > .
> >
> > thanks
> >
> > H
> >
> >  Morpheus: Do you believe in fate, Neo?
> > Neo: No.
> > Morpheus: Why Not?
> > Neo: Because I don't like the idea that I'm not in control of my life.
> >
> >
> >
> >
> >
>
>
>
>
>



      

From general-return-506-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 14 02:27:31 2009
Return-Path: <general-return-506-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1548 invoked from network); 14 Sep 2009 02:27:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Sep 2009 02:27:31 -0000
Received: (qmail 37909 invoked by uid 500); 14 Sep 2009 02:27:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37839 invoked by uid 500); 14 Sep 2009 02:27:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37829 invoked by uid 99); 14 Sep 2009 02:27:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Sep 2009 02:27:30 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.92.26] (HELO qw-out-2122.google.com) (74.125.92.26)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Sep 2009 02:27:19 +0000
Received: by qw-out-2122.google.com with SMTP id 9so795982qwb.35
        for <general@hadoop.apache.org>; Sun, 13 Sep 2009 19:26:58 -0700 (PDT)
MIME-Version: 1.0
Sender: edward@udanax.org
Received: by 10.224.13.205 with SMTP id d13mr4753281qaa.217.1252895218333; 
	Sun, 13 Sep 2009 19:26:58 -0700 (PDT)
Date: Mon, 14 Sep 2009 11:26:58 +0900
X-Google-Sender-Auth: 16ac46d4d8fd0eea
Message-ID: <eb4706e0909131926m48ddb6f8g2138ab3f76b9170@mail.gmail.com>
Subject: Discussion about Hamburg (provisional name) open sourcing
From: "Edward J. Yoon" <edwardyoon@apache.org>
To: general@hadoop.apache.org, hama-dev@incubator.apache.org, 
	hamburg-dev@googlegroups.com
Cc: Paolo Castagna <paolo.castagna@hp.com>
Content-Type: text/plain; charset=UTF-8
X-Virus-Checked: Checked by ClamAV on apache.org

Hello communities,

I'm the one of the Hamburg (provisional name), which is the graph
computing framework on Hadoop sponsor. Now we're working on the
perfection of our prototype project, and we'll propose the Hamburg
project soon.

- http://wiki.apache.org/hadoop/Hamburg, a wiki page
- http://throb.googlecode.com/, a prototype project

BTW, before we decide to propose, we need time just to consider where
it belongs to.

Since it aims to create a "general graph computing framework" on
Hadoop, I'd like to propose it as a sub-project of Hadoop. On the
other hand, since the matrix and graph are both in the domain of
scientific computing and BSP model could be used for matrix
computation areas, I think this project also can be integrated with
the Hama project.

WDYT? Any advices are welcome.

-- 
Best Regards, Edward J. Yoon @ NHN, corp.
edwardyoon@apache.org
http://blog.udanax.org

From general-return-507-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 14 04:10:09 2009
Return-Path: <general-return-507-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33061 invoked from network); 14 Sep 2009 04:10:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Sep 2009 04:10:09 -0000
Received: (qmail 99870 invoked by uid 500); 14 Sep 2009 04:10:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99682 invoked by uid 500); 14 Sep 2009 04:10:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 99657 invoked by uid 99); 14 Sep 2009 04:10:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Sep 2009 04:10:07 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [192.101.109.35] (HELO emailgw04.pnl.gov) (192.101.109.35)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Sep 2009 04:09:54 +0000
X-IronPort-AV: E=Sophos;i="4.44,381,1249282800"; 
   d="scan'208";a="12136164"
Received: from emailbh02.pnl.gov ([130.20.128.50])
  by emailgw04.pnl.gov with ESMTP; 13 Sep 2009 21:09:32 -0700
Received: from EMAIL02.pnl.gov ([130.20.128.221]) by emailbh02.pnl.gov with Microsoft SMTPSVC(6.0.3790.3959);
	 Sun, 13 Sep 2009 21:09:32 -0700
x-mimeole: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: RE: Discussion about Hamburg (provisional name) open sourcing
Date: Sun, 13 Sep 2009 21:09:31 -0700
Message-ID: <12A8FC4D0563D1468AD23AAAED861A3EDE081D@EMAIL02.pnl.gov>
In-Reply-To: <eb4706e0909131926m48ddb6f8g2138ab3f76b9170@mail.gmail.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Discussion about Hamburg (provisional name) open sourcing
Thread-Index: Aco04uzokcSvL0NYTm2SCXEpGPryPQABCFUA
References: <eb4706e0909131926m48ddb6f8g2138ab3f76b9170@mail.gmail.com>
From: "Taylor, Ronald C" <ronald.taylor@pnl.gov>
To: <general@hadoop.apache.org>,
	<hama-dev@incubator.apache.org>,
	<hamburg-dev@googlegroups.com>
Cc: "Paolo Castagna" <paolo.castagna@hp.com>,
	"Taylor, Ronald C" <ronald.taylor@pnl.gov>
X-OriginalArrivalTime: 14 Sep 2009 04:09:32.0370 (UTC) FILETIME=[29A19B20:01CA34F1]
X-Virus-Checked: Checked by ClamAV on apache.org

Hello Mr. Yoon,

I was delighted to hear of your proposed Hamburg project. I am a new
user of Hadoop (and Hbase). It looks like that I will be spending a
substantial amount of time working in this environment over the next
couple years, for both DOE bioinformatics work (my primary field) and
for work funded by DoD. I am enthusiastic about using Hadoop, Hive,
Hbase. Also am quite interested in the Mahout project.

While I cannot offer advice as to where to place your new project within
the Apache framework, I did want to offer my support. I believe that it
could well be of value in the coming years both to me, for my
bioinformatics research, and to other researchers here at PNNL working
in the areas of social networks (in our national security directorate)
and in a set of projects directed toward making the electrical grid
"smarter". I would not be able to contribute any code until I found
funding from current or new projects for my time. But if Hamburg moves
forward and can demonstrate its usefulness, that might be a real
possibility.=20

And in regards to funding for getting you some help: if you can find a
collaborator based at a university or non-profit, said collaborator
could well apply for a grant from the US National Science Foundation for
open source Hadoop-based development of graph computing / mining
algorithms. The NSF Computer and Information Science and Engineering
Directorate is awarding grants specifically devoted to the area of graph
mining (at least this year - hopefully will continue next year - anyway,
NSF gives money for algorithm and tool development in general - friendly
to that). I can't apply (at least not directly) - NSF does not like to
give money to other US government labs. But I would think you could find
someone in academia - perhaps someone already working with the Mahout
group. It would appear a natural fit. I presume there are a number of
people associated with the Apache org who know something about the NSF
and could offer further advice in that direction.

I look forward to hearing more about Hamburg, as it progresses.

 Best,
  Ron Taylor

___________________________________________
Ronald Taylor, Ph.D.
Computational Biology & Bioinformatics Group
Pacific Northwest National Laboratory
902 Battelle Boulevard
P.O. Box 999, MSIN K7-90
Richland, WA  99352 USA
Office:  509-372-6568
Email: ronald.taylor@pnl.gov
www.pnl.gov


-----Original Message-----
From: edward@udanax.org [mailto:edward@udanax.org] On Behalf Of Edward
J. Yoon
Sent: Sunday, September 13, 2009 7:27 PM
To: general@hadoop.apache.org; hama-dev@incubator.apache.org;
hamburg-dev@googlegroups.com
Cc: Paolo Castagna
Subject: Discussion about Hamburg (provisional name) open sourcing

Hello communities,

I'm the one of the Hamburg (provisional name), which is the graph
computing framework on Hadoop sponsor. Now we're working on the
perfection of our prototype project, and we'll propose the Hamburg
project soon.

- http://wiki.apache.org/hadoop/Hamburg, a wiki page
- http://throb.googlecode.com/, a prototype project

BTW, before we decide to propose, we need time just to consider where it
belongs to.

Since it aims to create a "general graph computing framework" on Hadoop,
I'd like to propose it as a sub-project of Hadoop. On the other hand,
since the matrix and graph are both in the domain of scientific
computing and BSP model could be used for matrix computation areas, I
think this project also can be integrated with the Hama project.

WDYT? Any advices are welcome.

--
Best Regards, Edward J. Yoon @ NHN, corp.
edwardyoon@apache.org
http://blog.udanax.org

From general-return-508-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 14 17:07:08 2009
Return-Path: <general-return-508-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 90055 invoked from network); 14 Sep 2009 17:07:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Sep 2009 17:07:08 -0000
Received: (qmail 90649 invoked by uid 500); 14 Sep 2009 17:07:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90575 invoked by uid 500); 14 Sep 2009 17:07:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90562 invoked by uid 99); 14 Sep 2009 17:07:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Sep 2009 17:07:07 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cabiwan@gmail.com designates 209.85.220.218 as permitted sender)
Received: from [209.85.220.218] (HELO mail-fx0-f218.google.com) (209.85.220.218)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Sep 2009 17:06:59 +0000
Received: by fxm18 with SMTP id 18so2350752fxm.29
        for <general@hadoop.apache.org>; Mon, 14 Sep 2009 10:06:38 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=Y7ThtrYUcMNZqp4jDmVuXSvbX+uGkrydHRL7aPlZm/s=;
        b=ntgARpIBaB7M+8Pn/6wQ/JJdoi9FC73ZA4rr8FWpJ5xijcNIYklZrFyXOlZQVYmbDc
         pLnceY8FhjhMEqXEOSSjazCjk4FVOtje9KDqZn5XMaIbfvMfsrPfJJw5PVeL1p5p0VhF
         gHqDgxmLqO9a8Sq5ZdbcLvKYfdyboBYLa3k6g=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=ayuMMoxbeX9myunyy4gtof+MeeFzGELDx/sTH2KLuF8qYaArc7r+naoxnZpwNLi3Lt
         E9UNlwsgJMQfN1Ib/9Ywg4nPr5ygQ+YXA/iaJRVKUaazrDExg3OQf4/3/NSdYrWyAZFP
         nxtsDlOdJlPlI8dHWKVi1kgxPyvJ1SkBF+cME=
MIME-Version: 1.0
Received: by 10.103.48.20 with SMTP id a20mr2753493muk.121.1252947998093; Mon, 
	14 Sep 2009 10:06:38 -0700 (PDT)
Date: Mon, 14 Sep 2009 19:06:38 +0200
Message-ID: <309f76d00909141006i2afea592mbca25fa6605ecf73@mail.gmail.com>
Subject: Problems with hadoop 0.18.0 plugin and Eclipse 3.5 (Galileo)
From: Alberto Luengo Cabanillas <cabiwan@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6546f3e59327104738cb0d3
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6546f3e59327104738cb0d3
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi people! I=B4m a newbie of Hadoop and I=B4d like to have an integrated
environment for Hadoop and Pig in the same Eclipse Galileo distribution. Th=
e
Pig plugin (PigPen) works properly but the Hadoop plugin only lets me brows=
e
the DFS server location (in a remote Virtual Machine) and not sending jobs.
Could anyone help me with this problem?
Thanks a lot.

--=20
Alberto

--0016e6546f3e59327104738cb0d3--

From general-return-509-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 14 17:34:06 2009
Return-Path: <general-return-509-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1265 invoked from network); 14 Sep 2009 17:34:06 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Sep 2009 17:34:06 -0000
Received: (qmail 34773 invoked by uid 500); 14 Sep 2009 17:34:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34719 invoked by uid 500); 14 Sep 2009 17:34:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34709 invoked by uid 99); 14 Sep 2009 17:34:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Sep 2009 17:34:05 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jonathan.holloway@gmail.com designates 209.85.210.204 as permitted sender)
Received: from [209.85.210.204] (HELO mail-yx0-f204.google.com) (209.85.210.204)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Sep 2009 17:33:57 +0000
Received: by yxe42 with SMTP id 42so4346975yxe.31
        for <general@hadoop.apache.org>; Mon, 14 Sep 2009 10:33:36 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=6l+KsVjmliGvaTmW2EVXUDlim9Vd8hC8zTLjTqmiAYQ=;
        b=iVdXeDjQHa4kj+3IodG8Cpwr9kBpNm6ejQ+ARzoyhXuNZLa9Q/1+ZR3pZ4amYmE9Qb
         cAiMJpE8HLVx5n/oAHF1q9FN+PL9ilKhE/JiTwMVWqkdvZd64Zl302qMpAgmyKvPswaV
         RxM5ffmoWWXGhy3OLxf4jqUDaoNNfgMV8p19g=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=k1nVXVe0w4tBUB3KL/k+LS+qBZUaEoEYPgOegJEbaUupgCfuKQHe73Pt+NZbA0kwuW
         JHb8WFnYi8N+rV5oxeZt2ZjCkHftF/B0q5VaNy17awGsIKJaV/NyxffYVCHJjm1Gi6eo
         eVuteIUiOzv+CDGmSf4TR02MeUhMlO+2qdyc8=
MIME-Version: 1.0
Received: by 10.101.209.23 with SMTP id l23mr6403513anq.173.1252949616059; 
	Mon, 14 Sep 2009 10:33:36 -0700 (PDT)
Date: Mon, 14 Sep 2009 10:33:36 -0700
Message-ID: <11c4fb4d0909141033n2491565dt9128b86e35ae1cf2@mail.gmail.com>
Subject: Hadoop and Small Files
From: Jonathan Holloway <jonathan.holloway@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d3cf14c966f104738d10a0
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d3cf14c966f104738d10a0
Content-Type: text/plain; charset=UTF-8

Hi all,

I'm new to Hadoop and currently looking at it for a project where there is
around a few TB of data that needs to be stored
in a format suitable for MapReduce functions.  The problem is that I'm
dealing with small text files (including metadata)
of 10Kb in size (and upwards to a few MB) that need to be stored in some
format.  The files need to be accessed randomly
with very low latency.  I've been through the docs and previous posts on the
mailing list, and looked at the following options:

* HDFS - not suitable "as is" because of the 64MB block size
* HAR (Hadoop Archives) - not sure about random access to files within
format
* Sequence Files - slow to convert into this format, can't randomly access
the files
* CombineFileInputFormat - assuming you still can't access the files
randomly https://issues.apache.org/jira/browse/HADOOP-4565
* MapFile - looks good... but not sure about latency
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/MapFile.html
* HBase (or similar distributed key-value store) - not sure about latency,
has this improved with the 0.20 release?

Please correct if I'm wrong re: the assumptions above.  Which is the most
appropriate option here?

Many thanks...
Jon.

--0016e6d3cf14c966f104738d10a0--

From general-return-510-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 14 19:10:34 2009
Return-Path: <general-return-510-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39030 invoked from network); 14 Sep 2009 19:10:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Sep 2009 19:10:34 -0000
Received: (qmail 98789 invoked by uid 500); 14 Sep 2009 19:10:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98734 invoked by uid 500); 14 Sep 2009 19:10:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98724 invoked by uid 99); 14 Sep 2009 19:10:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Sep 2009 19:10:33 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.220.218] (HELO mail-fx0-f218.google.com) (209.85.220.218)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Sep 2009 19:10:24 +0000
Received: by fxm18 with SMTP id 18so2435015fxm.29
        for <general@hadoop.apache.org>; Mon, 14 Sep 2009 12:10:04 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.86.163.5 with SMTP id l5mr5358965fge.3.1252955404094; Mon, 14 
	Sep 2009 12:10:04 -0700 (PDT)
In-Reply-To: <309f76d00909141006i2afea592mbca25fa6605ecf73@mail.gmail.com>
References: <309f76d00909141006i2afea592mbca25fa6605ecf73@mail.gmail.com>
From: Philip Zeyliger <philip@cloudera.com>
Date: Mon, 14 Sep 2009 12:09:44 -0700
Message-ID: <15da8a100909141209y6f10fc02y9f75378cffeaacc1@mail.gmail.com>
Subject: Re: Problems with hadoop 0.18.0 plugin and Eclipse 3.5 (Galileo)
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636025e9ac7cc1904738e6943
X-Virus-Checked: Checked by ClamAV on apache.org

--001636025e9ac7cc1904738e6943
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

You might want to chime in at
http://issues.apache.org/jira/browse/HADOOP-3744 .  People have seen that
patch fix it for them.

-- Philip

On Mon, Sep 14, 2009 at 10:06 AM, Alberto Luengo Cabanillas <
cabiwan@gmail.com> wrote:

> Hi people! I=B4m a newbie of Hadoop and I=B4d like to have an integrated
> environment for Hadoop and Pig in the same Eclipse Galileo distribution.
> The
> Pig plugin (PigPen) works properly but the Hadoop plugin only lets me
> browse
> the DFS server location (in a remote Virtual Machine) and not sending job=
s.
> Could anyone help me with this problem?
> Thanks a lot.
>
> --
> Alberto
>

--001636025e9ac7cc1904738e6943--

From general-return-511-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 14 19:16:10 2009
Return-Path: <general-return-511-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 40263 invoked from network); 14 Sep 2009 19:16:10 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Sep 2009 19:16:10 -0000
Received: (qmail 5464 invoked by uid 500); 14 Sep 2009 19:16:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5376 invoked by uid 500); 14 Sep 2009 19:16:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5366 invoked by uid 99); 14 Sep 2009 19:16:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Sep 2009 19:16:09 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cabiwan@gmail.com designates 209.85.218.214 as permitted sender)
Received: from [209.85.218.214] (HELO mail-bw0-f214.google.com) (209.85.218.214)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Sep 2009 19:16:01 +0000
Received: by bwz10 with SMTP id 10so2424184bwz.29
        for <general@hadoop.apache.org>; Mon, 14 Sep 2009 12:15:37 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=uzvbpEWYuxLXeMlC8JKeYhajOUCy5Qo7qUrhY8wkw38=;
        b=gVuFcuiWSXciuMJEPjiLQIPggTmkt4QVS7DD5FqclJy4BXg2EpS7UU6WVr/xvYyxpO
         dK6qo61Qu5MZ6x+/AiK0hpmE/c44bKgyRwDMlAWV6c9pIHfas2s+Yd2GgEfGuVEYedGQ
         1VE/dPZLcoqWo2HP4lcFqI/uYwB2irJQ/RuKk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=REf7SSuAdLAKQRAJNLPKeIZNVvlEcRPDLq47I9fh68AA8EjyZvNwNMsQuB1E1pATwA
         /gnKhvcklkeyssFXRAssALz0iobj+Wj2Lz7uakfjQ3ddWyhJQAmhlLuxMT4RmRK/1jYh
         2MKNOsSu3BuC4BhP4qQ6EO4jQ2YFev65ZlPAc=
MIME-Version: 1.0
Received: by 10.102.226.17 with SMTP id y17mr2844161mug.67.1252955737815; Mon, 
	14 Sep 2009 12:15:37 -0700 (PDT)
In-Reply-To: <15da8a100909141209y6f10fc02y9f75378cffeaacc1@mail.gmail.com>
References: <309f76d00909141006i2afea592mbca25fa6605ecf73@mail.gmail.com>
	 <15da8a100909141209y6f10fc02y9f75378cffeaacc1@mail.gmail.com>
Date: Mon, 14 Sep 2009 21:15:37 +0200
Message-ID: <309f76d00909141215g4917ac1fg1468ba473348611e@mail.gmail.com>
Subject: Re: Problems with hadoop 0.18.0 plugin and Eclipse 3.5 (Galileo)
From: Alberto Luengo Cabanillas <cabiwan@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364d205babfb7c04738e7d60
X-Virus-Checked: Checked by ClamAV on apache.org

--0016364d205babfb7c04738e7d60
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Thanks a lot for the quick reply. I=B4ll take a look.

2009/9/14 Philip Zeyliger <philip@cloudera.com>

> You might want to chime in at
> http://issues.apache.org/jira/browse/HADOOP-3744 .  People have seen that
> patch fix it for them.
>
> -- Philip
>
> On Mon, Sep 14, 2009 at 10:06 AM, Alberto Luengo Cabanillas <
> cabiwan@gmail.com> wrote:
>
> > Hi people! I=B4m a newbie of Hadoop and I=B4d like to have an integrate=
d
> > environment for Hadoop and Pig in the same Eclipse Galileo distribution=
.
> > The
> > Pig plugin (PigPen) works properly but the Hadoop plugin only lets me
> > browse
> > the DFS server location (in a remote Virtual Machine) and not sending
> jobs.
> > Could anyone help me with this problem?
> > Thanks a lot.
> >
> > --
> > Alberto
> >
>



--=20
Alberto

--0016364d205babfb7c04738e7d60--

From general-return-512-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 14 20:19:37 2009
Return-Path: <general-return-512-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 64642 invoked from network); 14 Sep 2009 20:19:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Sep 2009 20:19:36 -0000
Received: (qmail 15335 invoked by uid 500); 14 Sep 2009 20:19:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15260 invoked by uid 500); 14 Sep 2009 20:19:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15250 invoked by uid 99); 14 Sep 2009 20:19:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Sep 2009 20:19:35 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [66.46.182.51] (HELO relay.ihostexchange.net) (66.46.182.51)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Sep 2009 20:19:25 +0000
Received: from VMBX115.ihostexchange.net ([192.168.30.15]) by
 HUB101.ihostexchange.net ([66.46.182.51]) with mapi; Mon, 14 Sep 2009
 16:19:03 -0400
From: Sam Baskinger <sam.baskinger@networkedinsights.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 14 Sep 2009 16:19:00 -0400
Subject: Re: Hadoop and Small Files
Thread-Topic: Hadoop and Small Files
Thread-Index: Aco1YZOTp2CN7GQcQ7q9ULm5KVTNpwAFwSpS
Message-ID: <C6D40F64.12CF%sam.baskinger@networkedinsights.com>
In-Reply-To: <11c4fb4d0909141033n2491565dt9128b86e35ae1cf2@mail.gmail.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C6D40F6412CFsambaskingernetworkedinsightscom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C6D40F6412CFsambaskingernetworkedinsightscom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Hey Jon,

I don't know how many seconds would constitute low latency for your applica=
tion, but I would guess that Hadoop simply will not cut it. I would recomme=
nd something closer to Grid-SQL.

If you absolutely must process all the files using MapReduce, perhaps you c=
an split them all up into 1GB files and process them as a fraction of the l=
arger problem? Just a thought.

Sam


On 9/14/09 12:33 PM, "Jonathan Holloway" <jonathan.holloway@gmail.com> wrot=
e:

Hi all,

I'm new to Hadoop and currently looking at it for a project where there is
around a few TB of data that needs to be stored
in a format suitable for MapReduce functions.  The problem is that I'm
dealing with small text files (including metadata)
of 10Kb in size (and upwards to a few MB) that need to be stored in some
format.  The files need to be accessed randomly
with very low latency.  I've been through the docs and previous posts on th=
e
mailing list, and looked at the following options:

* HDFS - not suitable "as is" because of the 64MB block size
* HAR (Hadoop Archives) - not sure about random access to files within
format
* Sequence Files - slow to convert into this format, can't randomly access
the files
* CombineFileInputFormat - assuming you still can't access the files
randomly https://issues.apache.org/jira/browse/HADOOP-4565
* MapFile - looks good... but not sure about latency
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/MapFi=
le.html
* HBase (or similar distributed key-value store) - not sure about latency,
has this improved with the 0.20 release?

Please correct if I'm wrong re: the assumptions above.  Which is the most
appropriate option here?

Many thanks...
Jon.


Sam Baskinger
Software Engineer
Networked Insights, Inc. <http://www.networkedinsights.com/>

This e-mail message and any attachments are for the sole use of the intende=
d recipient(s) and may contain confidential and privileged information.  An=
y unauthorized review, use, disclosure, duplication or distribution is proh=
ibited.  If you received this message in error, please notify me by phone o=
r return email, do not forward to any other person and permanently delete t=
he foregoing message, all attachments and all copies immediately.


--_000_C6D40F6412CFsambaskingernetworkedinsightscom_--

From general-return-513-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 15 01:16:08 2009
Return-Path: <general-return-513-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60775 invoked from network); 15 Sep 2009 01:16:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Sep 2009 01:16:08 -0000
Received: (qmail 6573 invoked by uid 500); 15 Sep 2009 01:16:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6507 invoked by uid 500); 15 Sep 2009 01:16:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6497 invoked by uid 99); 15 Sep 2009 01:16:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Sep 2009 01:16:06 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of matthew.kelcey@gmail.com designates 209.85.219.208 as permitted sender)
Received: from [209.85.219.208] (HELO mail-ew0-f208.google.com) (209.85.219.208)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Sep 2009 01:15:55 +0000
Received: by ewy4 with SMTP id 4so3626743ewy.36
        for <general@hadoop.apache.org>; Mon, 14 Sep 2009 18:14:34 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=S8fMgrlgFuWNPj8yI82ifni/RD4Ba7BweSbA3KRj0tY=;
        b=RpaxG3bQrtctXPVbirxfc2pVle4KzKCqpWCfsJ7fbYX+rEo5QqY3FX9ES0dOtlO/3a
         9f5aMIaq6wMF+SL7lXvqFraLGgLUVjE6qYtMZckckFr6/NAzATD4KFF2IYeO0MyXtAaf
         pckG0qUtjHP4C8oPWqHwT9XE5nOw6CHZPSu8k=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=vVdP/vmJjlVOCOzbhqbFtFkQTkOCb1bEzll0EH3RmdV90iW+mzjAmAbS2c2CruIl2B
         C+FzuQorYsuztQswzurbe2uzpLlv5b9C9lm5cyCZR4rJJWXY3Y5FPySkQXw8xMX2F+b1
         /HixJSp1vcS0VFGdgZhVOn9OnPRW7yDaHoIAw=
MIME-Version: 1.0
Received: by 10.211.131.39 with SMTP id i39mr7685597ebn.98.1252977274426; Mon, 
	14 Sep 2009 18:14:34 -0700 (PDT)
Date: Tue, 15 Sep 2009 11:14:34 +1000
Message-ID: <93d501de0909141814vaa8c9c0wc5a47ee05baae7de@mail.gmail.com>
Subject: hadoop scales but is not performant?
From: Mat Kelcey <matthew.kelcey@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

hi all,

recently i've started playing with hadoop and my first learning
experiment has surprised me

i'm implementing a a text problem; trying to extract infrequent
phrases using a mixture of probabilistic models.
it's a bit of toy problem so the algorithm details aren't super
important, though the implementation might be....

this problem ended up being represented by a dozen or so map reduce
jobs of various types including some aggregate steps and some manual
joins.
(see the bottom of this page for details
http://matpalm.com/sip/take3_markov_chains.html)

i've implemented each step using ruby / streaming, the code is at
http://github.com/matpalm/sip if anyone cares.

i ran some tests using 10 ec2 medium cpu instance across a small
100mb's of gzipped text.
for validation of the results i also reimplemented the entire
algorithm as a single threaded ruby app

my surprise comes from finding that the ruby implementation
outperforms the 10 ec2 instances on this data size...
i ran a few samples of different sizes with the graph at the bottom of
http://matpalm.com/sip/part4_but_does_it_scale.html

so why is this? here are my explanations in order of how confident i am...

a) 100mb is peanuts and hadoop was made for 1000x this size so the
test is invalid.
b) there is a better representation of this problem that uses fewer
map/reduce passes.
c) streaming is too slow and rewriting in java (and making use of
techniques like chaining mappers) would speed things up
d) doing these steps, particularly the joins, in pig would be faster

my next steps are to rewrite some of the steps in pig to sample the difference

does anyone have any high level comments on this?

cheers,
mat

From general-return-514-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 15 05:15:38 2009
Return-Path: <general-return-514-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4503 invoked from network); 15 Sep 2009 05:15:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Sep 2009 05:15:38 -0000
Received: (qmail 49324 invoked by uid 500); 15 Sep 2009 05:15:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49241 invoked by uid 500); 15 Sep 2009 05:15:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49231 invoked by uid 99); 15 Sep 2009 05:15:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Sep 2009 05:15:37 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [209.85.222.204] (HELO mail-pz0-f204.google.com) (209.85.222.204)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Sep 2009 05:15:26 +0000
Received: by pzk42 with SMTP id 42so3186946pzk.31
        for <general@hadoop.apache.org>; Mon, 14 Sep 2009 22:14:04 -0700 (PDT)
Received: by 10.114.165.12 with SMTP id n12mr13032833wae.107.1252991644291;
        Mon, 14 Sep 2009 22:14:04 -0700 (PDT)
Received: from ?192.168.42.100? (75-101-123-136.dsl.static.sonic.net [75.101.123.136])
        by mx.google.com with ESMTPS id 22sm627308pzk.10.2009.09.14.22.13.49
        (version=SSLv3 cipher=RC4-MD5);
        Mon, 14 Sep 2009 22:13:51 -0700 (PDT)
Message-ID: <4AAF2286.4050105@cloudera.com>
Date: Mon, 14 Sep 2009 22:13:42 -0700
From: Amr Awadallah <aaa@cloudera.com>
Organization: Cloudera, Inc.
User-Agent: Thunderbird 2.0.0.23 (Windows/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Hadoop and Small Files
References: <11c4fb4d0909141033n2491565dt9128b86e35ae1cf2@mail.gmail.com>
In-Reply-To: <11c4fb4d0909141033n2491565dt9128b86e35ae1cf2@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

 > The files need to be accessed randomly with very low latency

Then use:

* HBase (or similar distributed key-value store) - not sure about latency,
has this improved with the 0.20 release?

Yes, latency is significantly better with 0.20, see preso from hadoop 
summit on results:

http://devblog.streamy.com/2009/07/24/streamy-hadoop-summit-hbase-goes-realtime/

-- amr

Jonathan Holloway wrote:
> Hi all,
>
> I'm new to Hadoop and currently looking at it for a project where there is
> around a few TB of data that needs to be stored
> in a format suitable for MapReduce functions.  The problem is that I'm
> dealing with small text files (including metadata)
> of 10Kb in size (and upwards to a few MB) that need to be stored in some
> format.  The files need to be accessed randomly
> with very low latency.  I've been through the docs and previous posts on the
> mailing list, and looked at the following options:
>
> * HDFS - not suitable "as is" because of the 64MB block size
> * HAR (Hadoop Archives) - not sure about random access to files within
> format
> * Sequence Files - slow to convert into this format, can't randomly access
> the files
> * CombineFileInputFormat - assuming you still can't access the files
> randomly https://issues.apache.org/jira/browse/HADOOP-4565
> * MapFile - looks good... but not sure about latency
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/MapFile.html
> * HBase (or similar distributed key-value store) - not sure about latency,
> has this improved with the 0.20 release?
>
> Please correct if I'm wrong re: the assumptions above.  Which is the most
> appropriate option here?
>
> Many thanks...
> Jon.
>
>   

From general-return-515-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 15 05:23:12 2009
Return-Path: <general-return-515-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 7337 invoked from network); 15 Sep 2009 05:23:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Sep 2009 05:23:12 -0000
Received: (qmail 55824 invoked by uid 500); 15 Sep 2009 05:23:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55753 invoked by uid 500); 15 Sep 2009 05:23:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55743 invoked by uid 99); 15 Sep 2009 05:23:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Sep 2009 05:23:11 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jonathan.holloway@gmail.com designates 209.85.210.204 as permitted sender)
Received: from [209.85.210.204] (HELO mail-yx0-f204.google.com) (209.85.210.204)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Sep 2009 05:23:00 +0000
Received: by yxe42 with SMTP id 42so5030362yxe.31
        for <general@hadoop.apache.org>; Mon, 14 Sep 2009 22:21:39 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=WPyf6rq4jJgKDgZKqTRMplZUiymgv7OJdzhcdkpmUUo=;
        b=k7fJtXS0q2O89oa28x+yCXRu5OJY1HvuH8adtY+wGxhQ8JessI5ptDi/wp/2wGu3vO
         fBNvA/NOB4ihPKvSORHyB+GjSSYikaNGdm5W5NDU9fp7BDc+tLI7bnKSp2JKbpgonG1f
         iLn1s92NNdVNxOabvDf1k+Y5YGZhDOXiMuBfE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=MP+psXgwDLmnFvRLI+amgCkQ48wPg9P4fjJnUUuDb+ES3ZhuG5wtFhYOxQAgQfw3Sr
         HkP7lJvlDoX1RCJ/xDptpPo/MJ02ncIRYJsSjvXrQ6YObDPrq6wg6+urLqSK7kDsmdD1
         XRrK0bGxUwsX6n6Q+fHHQIUvfFNDDPK7fOeE0=
MIME-Version: 1.0
Received: by 10.101.129.9 with SMTP id g9mr7038331ann.115.1252992097830; Mon, 
	14 Sep 2009 22:21:37 -0700 (PDT)
In-Reply-To: <4AAF2286.4050105@cloudera.com>
References: <11c4fb4d0909141033n2491565dt9128b86e35ae1cf2@mail.gmail.com>
	 <4AAF2286.4050105@cloudera.com>
Date: Mon, 14 Sep 2009 22:21:37 -0700
Message-ID: <11c4fb4d0909142221j6a94f19eu7cfe5a41279b868b@mail.gmail.com>
Subject: Re: Hadoop and Small Files
From: Jonathan Holloway <jonathan.holloway@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016368e2b7ce5c7c4047396f4eb
X-Virus-Checked: Checked by ClamAV on apache.org

--0016368e2b7ce5c7c4047396f4eb
Content-Type: text/plain; charset=UTF-8

Hi all...

Many thanks for your help and the responses, currently investigating HBase
0.20 as a potential option...

2009/9/14 Amr Awadallah <aaa@cloudera.com>

> > The files need to be accessed randomly with very low latency
>
> Then use:
>
> * HBase (or similar distributed key-value store) - not sure about latency,
> has this improved with the 0.20 release?
>
> Yes, latency is significantly better with 0.20, see preso from hadoop
> summit on results:
>
>
> http://devblog.streamy.com/2009/07/24/streamy-hadoop-summit-hbase-goes-realtime/
>
> -- amr
>
>
> Jonathan Holloway wrote:
>
>> Hi all,
>>
>> I'm new to Hadoop and currently looking at it for a project where there is
>> around a few TB of data that needs to be stored
>> in a format suitable for MapReduce functions.  The problem is that I'm
>> dealing with small text files (including metadata)
>> of 10Kb in size (and upwards to a few MB) that need to be stored in some
>> format.  The files need to be accessed randomly
>> with very low latency.  I've been through the docs and previous posts on
>> the
>> mailing list, and looked at the following options:
>>
>> * HDFS - not suitable "as is" because of the 64MB block size
>> * HAR (Hadoop Archives) - not sure about random access to files within
>> format
>> * Sequence Files - slow to convert into this format, can't randomly access
>> the files
>> * CombineFileInputFormat - assuming you still can't access the files
>> randomly https://issues.apache.org/jira/browse/HADOOP-4565
>> * MapFile - looks good... but not sure about latency
>>
>> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/io/MapFile.html
>> * HBase (or similar distributed key-value store) - not sure about latency,
>> has this improved with the 0.20 release?
>>
>> Please correct if I'm wrong re: the assumptions above.  Which is the most
>> appropriate option here?
>>
>> Many thanks...
>> Jon.
>>
>>
>>
>


-- 
Design and Tech-noogly
Web: http://www.oogly.co.uk
Mail:  jonathan.holloway@oogly.co.uk
IM:jonathan_philip_holloway@hotmail.com<IM%3Ajonathan_philip_holloway@hotmail.com>

--0016368e2b7ce5c7c4047396f4eb--

From general-return-516-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 15 08:33:31 2009
Return-Path: <general-return-516-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83395 invoked from network); 15 Sep 2009 08:33:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Sep 2009 08:33:31 -0000
Received: (qmail 40813 invoked by uid 500); 15 Sep 2009 08:33:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40731 invoked by uid 500); 15 Sep 2009 08:33:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40721 invoked by uid 99); 15 Sep 2009 08:33:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Sep 2009 08:33:30 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.92.25] (HELO qw-out-2122.google.com) (74.125.92.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Sep 2009 08:33:18 +0000
Received: by qw-out-2122.google.com with SMTP id 9so1123973qwb.35
        for <general@hadoop.apache.org>; Tue, 15 Sep 2009 01:31:56 -0700 (PDT)
MIME-Version: 1.0
Sender: edward@udanax.org
Received: by 10.224.12.141 with SMTP id x13mr5924774qax.337.1253003156829; 
	Tue, 15 Sep 2009 01:25:56 -0700 (PDT)
In-Reply-To: <12A8FC4D0563D1468AD23AAAED861A3EDE081D@EMAIL02.pnl.gov>
References: <eb4706e0909131926m48ddb6f8g2138ab3f76b9170@mail.gmail.com>
	 <12A8FC4D0563D1468AD23AAAED861A3EDE081D@EMAIL02.pnl.gov>
Date: Tue, 15 Sep 2009 17:25:56 +0900
X-Google-Sender-Auth: 40b425b73b1cf80a
Message-ID: <eb4706e0909150125v5281428ay43820c9fb9e7d150@mail.gmail.com>
Subject: Re: Discussion about Hamburg (provisional name) open sourcing
From: "Edward J. Yoon" <edwardyoon@apache.org>
To: general@hadoop.apache.org
Cc: hama-dev@incubator.apache.org, hamburg-dev@googlegroups.com, 
	Paolo Castagna <paolo.castagna@hp.com>, "Taylor, Ronald C" <ronald.taylor@pnl.gov>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks for interesting information.

With hindsight, IMO, the Hama project is a best to incubate Hamburg
project. and, we can consider again when prepare to graduate from
incubator.

On Mon, Sep 14, 2009 at 1:09 PM, Taylor, Ronald C <ronald.taylor@pnl.gov> w=
rote:
> Hello Mr. Yoon,
>
> I was delighted to hear of your proposed Hamburg project. I am a new
> user of Hadoop (and Hbase). It looks like that I will be spending a
> substantial amount of time working in this environment over the next
> couple years, for both DOE bioinformatics work (my primary field) and
> for work funded by DoD. I am enthusiastic about using Hadoop, Hive,
> Hbase. Also am quite interested in the Mahout project.
>
> While I cannot offer advice as to where to place your new project within
> the Apache framework, I did want to offer my support. I believe that it
> could well be of value in the coming years both to me, for my
> bioinformatics research, and to other researchers here at PNNL working
> in the areas of social networks (in our national security directorate)
> and in a set of projects directed toward making the electrical grid
> "smarter". I would not be able to contribute any code until I found
> funding from current or new projects for my time. But if Hamburg moves
> forward and can demonstrate its usefulness, that might be a real
> possibility.
>
> And in regards to funding for getting you some help: if you can find a
> collaborator based at a university or non-profit, said collaborator
> could well apply for a grant from the US National Science Foundation for
> open source Hadoop-based development of graph computing / mining
> algorithms. The NSF Computer and Information Science and Engineering
> Directorate is awarding grants specifically devoted to the area of graph
> mining (at least this year - hopefully will continue next year - anyway,
> NSF gives money for algorithm and tool development in general - friendly
> to that). I can't apply (at least not directly) - NSF does not like to
> give money to other US government labs. But I would think you could find
> someone in academia - perhaps someone already working with the Mahout
> group. It would appear a natural fit. I presume there are a number of
> people associated with the Apache org who know something about the NSF
> and could offer further advice in that direction.
>
> I look forward to hearing more about Hamburg, as it progresses.
>
> =C2=A0Best,
> =C2=A0Ron Taylor
>
> ___________________________________________
> Ronald Taylor, Ph.D.
> Computational Biology & Bioinformatics Group
> Pacific Northwest National Laboratory
> 902 Battelle Boulevard
> P.O. Box 999, MSIN K7-90
> Richland, WA =C2=A099352 USA
> Office: =C2=A0509-372-6568
> Email: ronald.taylor@pnl.gov
> www.pnl.gov
>
>
> -----Original Message-----
> From: edward@udanax.org [mailto:edward@udanax.org] On Behalf Of Edward
> J. Yoon
> Sent: Sunday, September 13, 2009 7:27 PM
> To: general@hadoop.apache.org; hama-dev@incubator.apache.org;
> hamburg-dev@googlegroups.com
> Cc: Paolo Castagna
> Subject: Discussion about Hamburg (provisional name) open sourcing
>
> Hello communities,
>
> I'm the one of the Hamburg (provisional name), which is the graph
> computing framework on Hadoop sponsor. Now we're working on the
> perfection of our prototype project, and we'll propose the Hamburg
> project soon.
>
> - http://wiki.apache.org/hadoop/Hamburg, a wiki page
> - http://throb.googlecode.com/, a prototype project
>
> BTW, before we decide to propose, we need time just to consider where it
> belongs to.
>
> Since it aims to create a "general graph computing framework" on Hadoop,
> I'd like to propose it as a sub-project of Hadoop. On the other hand,
> since the matrix and graph are both in the domain of scientific
> computing and BSP model could be used for matrix computation areas, I
> think this project also can be integrated with the Hama project.
>
> WDYT? Any advices are welcome.
>
> --
> Best Regards, Edward J. Yoon @ NHN, corp.
> edwardyoon@apache.org
> http://blog.udanax.org
>



--=20
Best Regards, Edward J. Yoon @ NHN, corp.
edwardyoon@apache.org
http://blog.udanax.org

From general-return-517-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 15 13:01:02 2009
Return-Path: <general-return-517-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69949 invoked from network); 15 Sep 2009 13:01:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Sep 2009 13:01:01 -0000
Received: (qmail 12102 invoked by uid 500); 15 Sep 2009 13:01:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12028 invoked by uid 500); 15 Sep 2009 13:01:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12018 invoked by uid 99); 15 Sep 2009 13:01:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Sep 2009 13:01:01 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Sep 2009 13:00:48 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id B9BEFB7D67
	for <general@hadoop.apache.org>; Tue, 15 Sep 2009 14:00:27 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id QL8k8uc3ZyDH for <general@hadoop.apache.org>;
	Tue, 15 Sep 2009 14:00:20 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id A53EAB7CE4
	for <general@hadoop.apache.org>; Tue, 15 Sep 2009 14:00:20 +0100 (BST)
MailScanner-NULL-Check: 1253624407.58742@mdYF2cbz0rs/TrYdxfiVFw
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id n8FD06Ca018869
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Tue, 15 Sep 2009 14:00:07 +0100 (BST)
Message-ID: <4AAF8FD6.7010703@apache.org>
Date: Tue, 15 Sep 2009 14:00:06 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Hadoop and universities/research
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: n8FD06Ca018869
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org


I've stuck up a little slideset with some thoughts on what we could be 
doing to get tighter engagement between universities and the Hadoop codebase
http://www.slideshare.net/steve_l/hadoop-and-universities

There are various levels of engagement

- teaching MapReduce and other datacentre-scale coding techiques.
   This could be done by getting involved with the undergrad/grad 
teachers, helping with the lectures and the coursework.
   Remember , most universities do welcome outside lecturers giving 
guest talks to the students.

- encourage scientific computation to be done on top of Hadoop. There's 
a small problem there, cluster time, which means that people with access 
to datacentres with CPU and storage to spare need to lift a hand here. 
Or we help get Hadoop up on the existing clusters the physicists run.  [ 
see http://www.slideshare.net/steve_l/hadoop-hep for some plans here]

- encourage people doing maths and CS work to do it on Hadoop.
   The plugins for scheduling and placement are a good low-risk place we 
can get people involved, the other area of interest is stuff on top, 
such as graphs and other leading-edge stuff.

- get some of the people doing work in this area to talk at apachecon. 
Come on, you want to know how do debug a particle accelerator experiment 
that generates 1PB/month of data but which only 50 events/year are 
actually interesting.

Term-time is rushing up, so now might be a good time for Apache to get 
ready to bring the universities on board.

I'm going to start by proposing we create a hadoop-research list, for 
people doing more researchy stuff on top of and inside hadoop. we can 
then start identifying who is interested in this area, which academic 
and industrial people are involved, where they are, start meeting up. We 
had a good little workshop at Bristol University last month 
http://wiki.apache.org/hadoop/BristolHadoopWorkshop ; it was good to get 
together the different groups.

Thoughts?

-steve


From general-return-518-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 15 16:22:11 2009
Return-Path: <general-return-518-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 51142 invoked from network); 15 Sep 2009 16:22:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Sep 2009 16:22:11 -0000
Received: (qmail 46388 invoked by uid 500); 15 Sep 2009 16:22:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46324 invoked by uid 500); 15 Sep 2009 16:22:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46314 invoked by uid 99); 15 Sep 2009 16:22:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Sep 2009 16:22:10 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Sep 2009 16:21:58 +0000
Received: from [192.168.0.197] (snvvpn4-10-72-168-c143.hq.corp.yahoo.com [10.72.168.143])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n8FGIdHq057237
	for <general@hadoop.apache.org>; Tue, 15 Sep 2009 09:18:39 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=Jb3H9gzMhyfauF3zfqWDRmvWvOaPX8YEz4pyUa0hKgbdpxmFvv9KFsQiCIPBLZYa
Message-Id: <216DCE46-71A1-4FFA-B7AA-1028EB6EBEF5@yahoo-inc.com>
From: Alan Gates <gates@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <93d501de0909141814vaa8c9c0wc5a47ee05baae7de@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: hadoop scales but is not performant?
Date: Tue, 15 Sep 2009 09:18:39 -0700
References: <93d501de0909141814vaa8c9c0wc5a47ee05baae7de@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

Some thoughts, inlined.

Alan.

On Sep 14, 2009, at 6:14 PM, Mat Kelcey wrote:

> hi all,
>
> recently i've started playing with hadoop and my first learning
> experiment has surprised me
>
> i'm implementing a a text problem; trying to extract infrequent
> phrases using a mixture of probabilistic models.
> it's a bit of toy problem so the algorithm details aren't super
> important, though the implementation might be....
>
> this problem ended up being represented by a dozen or so map reduce
> jobs of various types including some aggregate steps and some manual
> joins.
> (see the bottom of this page for details
> http://matpalm.com/sip/take3_markov_chains.html)
>
> i've implemented each step using ruby / streaming, the code is at
> http://github.com/matpalm/sip if anyone cares.
>
> i ran some tests using 10 ec2 medium cpu instance across a small
> 100mb's of gzipped text.
> for validation of the results i also reimplemented the entire
> algorithm as a single threaded ruby app
>
> my surprise comes from finding that the ruby implementation
> outperforms the 10 ec2 instances on this data size...
> i ran a few samples of different sizes with the graph at the bottom of
> http://matpalm.com/sip/part4_but_does_it_scale.html
>
> so why is this? here are my explanations in order of how confident i  
> am...
>
> a) 100mb is peanuts and hadoop was made for 1000x this size so the
> test is invalid.
Definitely the case.  At this size your start costs will swamp any  
performance benefits of parallelism.

> b) there is a better representation of this problem that uses fewer
> map/reduce passes.
> c) streaming is too slow and rewriting in java (and making use of
> techniques like chaining mappers) would speed things up
Maybe.  Streaming itself is a little slower than using java.  I don't  
know what the penalty of using java versus ruby is.

> d) doing these steps, particularly the joins, in pig would be faster
Writing your joins in Pig will definitely be faster to code.  They  
won't be faster to execute unless you are able to use one of Pig's  
specialized join algorithms (fragment-replicate, merge, skew).  At  
100mb its hard to see that any of those will make a big difference.

>
> my next steps are to rewrite some of the steps in pig to sample the  
> difference
>
> does anyone have any high level comments on this?
>
> cheers,
> mat


From general-return-519-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 15 21:06:10 2009
Return-Path: <general-return-519-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 64449 invoked from network); 15 Sep 2009 21:06:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Sep 2009 21:06:09 -0000
Received: (qmail 75525 invoked by uid 500); 15 Sep 2009 21:06:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75486 invoked by uid 500); 15 Sep 2009 21:06:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75476 invoked by uid 99); 15 Sep 2009 21:06:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Sep 2009 21:06:08 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of matthew.kelcey@gmail.com designates 74.125.78.26 as permitted sender)
Received: from [74.125.78.26] (HELO ey-out-2122.google.com) (74.125.78.26)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Sep 2009 21:06:00 +0000
Received: by ey-out-2122.google.com with SMTP id 22so814245eye.23
        for <general@hadoop.apache.org>; Tue, 15 Sep 2009 14:05:39 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=MhOTplPqJIsl8etc468PU7J6yQISHH4ihEVqV8IVvhI=;
        b=ql4Eam3E59MgVrwlMu5wW3SRxYLruxIS0Kvk+Ne8Emig5qXeJY+nTe3rtUSmhJNifh
         EAaLw2eyd4n5Os3GlTdTz9y3m7NzIkPH1/T2laQoP9aKLnhnqMT0qByiw5mZaWEIYWLA
         jntKg+Ciw7TahusR+uiDRu61qpwtyk9RZ0QbM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=TkCeRExQD1tN80/oTtsDknPMD7bNhUI0WDlWB0qN5hkOm8Spr75zxLQ5ksR/C1a7v+
         7dX9H2QCM5Xl6+aMQ0/qdEB5CmlfFJv4p+cOwu1PIrwOc+e6j7XYgDP0ovU5F/Kt4mnI
         Htr/5G8Wip4dhWyLp3447n9OoiK0UaE8xTLeM=
MIME-Version: 1.0
Received: by 10.211.147.26 with SMTP id z26mr1355300ebn.73.1253048739568; Tue, 
	15 Sep 2009 14:05:39 -0700 (PDT)
In-Reply-To: <216DCE46-71A1-4FFA-B7AA-1028EB6EBEF5@yahoo-inc.com>
References: <93d501de0909141814vaa8c9c0wc5a47ee05baae7de@mail.gmail.com>
	 <216DCE46-71A1-4FFA-B7AA-1028EB6EBEF5@yahoo-inc.com>
Date: Wed, 16 Sep 2009 07:05:39 +1000
Message-ID: <93d501de0909151405t39c1f919h1629d373fa033466@mail.gmail.com>
Subject: Re: hadoop scales but is not performant?
From: Mat Kelcey <matthew.kelcey@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

>>[me]
>> a) 100mb is peanuts and hadoop was made for 1000x this size so the
>> test is invalid.

>[Alan]
> Definitely the case. =A0At this size your start costs will swamp any
> performance benefits of parallelism.

cheers, thanks alan
was pretty sure this was the case.
mat

From general-return-520-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 15 21:41:58 2009
Return-Path: <general-return-520-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76522 invoked from network); 15 Sep 2009 21:41:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Sep 2009 21:41:58 -0000
Received: (qmail 19434 invoked by uid 500); 15 Sep 2009 21:41:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19314 invoked by uid 500); 15 Sep 2009 21:41:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19211 invoked by uid 99); 15 Sep 2009 21:41:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Sep 2009 21:41:53 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.210.204] (HELO mail-yx0-f204.google.com) (209.85.210.204)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Sep 2009 21:41:41 +0000
Received: by yxe42 with SMTP id 42so5976979yxe.31
        for <multiple recipients>; Tue, 15 Sep 2009 14:41:19 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.91.76.5 with SMTP id d5mr4870203agl.68.1253050879535; Tue, 15 
	Sep 2009 14:41:19 -0700 (PDT)
From: Christophe Bisciglia <christophe@cloudera.com>
Date: Tue, 15 Sep 2009 14:40:59 -0700
Message-ID: <69035570909151440i4f3c78baxfb78ba730212ba68@mail.gmail.com>
Subject: Hadoop World: NYC - Speakers and Schedule Up - New: Lightning Talks 
	at Welcome Reception!
To: common-user@hadoop.apache.org, general@hadoop.apache.org, 
	hive-user@hadoop.apache.org, pig-user@hadoop.apache.org, 
	hbase-user <hbase-user@hadoop.apache.org>, zookeeper-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hadoop Fans, we're getting down to the wire for Hadoop World. We
couldn't be happier with how the schedule has come together.

For a full list of speakers and registration details, see
http://www.cloudera.com/hadoop-world-nyc - and if you plan to attend,
register this week, as prices go up $100 after September 21st.

We received nearly 50 submissions, and it was hard to trim this down,
even after creating a 3rd track. To provide more speaking
opportunities and time for the community to share their personal
experiences, Cloudera engineering is hosting a welcome reception with
lightning talks, an open bar, and live entertainment between talks.
There will be no sales or marketing from Cloudera or anyone else, and
talks are limited to 5 minutes to keep things lively. You'll find the
welcome reception in the Roosevelt Grill on the main floor, just off
the hotel lobby.

We'll get started at 6PM on October 1st. We'll also let you
pre-register for the conference so you can skip the line in the
morning.

At the main event, we're particularly excited to hear from Yahoo!
about upcoming API enhancements and security work. The whole team over
there is doing great stuff in this department, and it's features like
this that will help pave the way for serious enterprise adoption of
Hadoop. Facebook is also going to do a deep dive into HDFS
improvements and Hive - other key features for delivering a scalable
platform for a wide variety of data intensive applications.

You'll also hear from leading techologists at traditional enterprises
using Hadoop to solve real business problems today. Of particular note
are VISA on Large Scale Transaction Analysis, IBM on Ad-hoc Web Scale
Analytics, JP Morgan Chase on applications in Financial Services, Booz
Allen on Protein Alignment and Amazon Web Services on Bioinformatics.
These are in addition to all the usual suspects crunching big data in
the web space, and demonstrate how much closer we are to the beginning
of the Hadoop upswing than a plateau. These are exciting times, and we
hope you can all join us.

Cheers,
Christophe and the Cloudera Team

-- 
get hadoop: cloudera.com/hadoop
online training: cloudera.com/hadoop-training
blog: cloudera.com/blog
twitter: twitter.com/cloudera

From general-return-521-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 15 23:03:19 2009
Return-Path: <general-return-521-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6456 invoked from network); 15 Sep 2009 23:03:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Sep 2009 23:03:19 -0000
Received: (qmail 37852 invoked by uid 500); 15 Sep 2009 23:03:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37762 invoked by uid 500); 15 Sep 2009 23:03:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37752 invoked by uid 99); 15 Sep 2009 23:03:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Sep 2009 23:03:18 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [64.78.17.16] (HELO EXHUB018-1.exch018.msoutlookonline.net) (64.78.17.16)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Sep 2009 23:03:05 +0000
Received: from EXVMBX018-1.exch018.msoutlookonline.net ([64.78.17.47]) by
 EXHUB018-1.exch018.msoutlookonline.net ([64.78.17.16]) with mapi; Tue, 15 Sep
 2009 16:02:43 -0700
From: Scott Carey <scott@richrelevance.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 15 Sep 2009 16:02:40 -0700
Subject: Re: hadoop scales but is not performant?
Thread-Topic: hadoop scales but is not performant?
Thread-Index: Aco1ohy4R0PEAxULTzSBNeEZ+4RQ7AAtoMcS
Message-ID: <C6D56B20.1180B%scott@richrelevance.com>
In-Reply-To: <93d501de0909141814vaa8c9c0wc5a47ee05baae7de@mail.gmail.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C6D56B201180Bscottrichrelevancecom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C6D56B201180Bscottrichrelevancecom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Its not entirely invalid at this scale, though the dataset generally should=
 be (in memory) closer to the size of the available RAM of the cluster or m=
ore to be a better test.

Hadoop should still do well at this size relative to a single process.  If =
it is taking 45 minutes on a 10 machine cluster, it should be faster than a=
ny single process.  If this was limited primarily by startup costs, then in=
creasing the data size wouldn't almost linearly increase the time.

It can probably be significantly tuned.
Some tips for jobs this size:
Watch the CPU / RAM / IO on a node while it is running.  Is it being well u=
tilized?
Watch the jobs in the HTML UI on the jobtracker.  Are the total number of s=
lots configured being used most of the time?   Or is it spending a lot of c=
lock time with a lot of empty slots?  If so, which slots?  Are Maps or Redu=
ces taking the most time?

Hadoop currently is a slow scheduler for tasks (can schecule at most 1 task=
 per second or two per node).  Using the Fair Scheduler and enabling some o=
ptions can turn this up to 1 map and 1 reduce to schedule per 'tick'.  Ther=
e are some big changes in the schedulers due out in 0.21 that will signific=
antly help here.  For smaller low latency jobs this can make a big differen=
ce.

Hadoop also has some flaws in the shuffle phase that affect clusters of all=
 sizes, but can hurt small clusters with many small map jobs.   Look at the=
 log file output of your reduce jobs (in the jobtracker UI) and see how lon=
g the shuffle phase is taking.
There is a big change due for 0.21 that makes this a LOT faster for some ca=
ses, and low latency smaller jobs will benefit a lot too.
https://issues.apache.org/jira/browse/MAPREDUCE-318

See my comment in that ticket from the 10th of June in relation to shuffle =
times.  A one line change in 0.19.x and 0.20.x cut times on smaller low lat=
ency jobs on smaller clusters in half when limited by shuffle inefficiencie=
s (specifically, fetching no more than one map output from one host every f=
ew seconds for no good reason).

Tuning several hadoop parameters might help a great deal as well.   Clouder=
a's configuration wizard is a good starting point to help you find which kn=
obs are more important.


You probably also want to do some work to figure out what is taking time on=
 your local tests.  There might be something wrong there.  It seems suspici=
ous to me, even at the one document size, that it would take 2 + minutes ve=
rsus less than one second.  Startup costs aren't that big for 2 maps and 2 =
reduces - on the order of a few seconds not minutes.  A few seconds times a=
 few jobs is still not that much.



On 9/14/09 6:14 PM, "Mat Kelcey" <matthew.kelcey@gmail.com> wrote:

hi all,

recently i've started playing with hadoop and my first learning
experiment has surprised me

i'm implementing a a text problem; trying to extract infrequent
phrases using a mixture of probabilistic models.
it's a bit of toy problem so the algorithm details aren't super
important, though the implementation might be....

this problem ended up being represented by a dozen or so map reduce
jobs of various types including some aggregate steps and some manual
joins.
(see the bottom of this page for details
http://matpalm.com/sip/take3_markov_chains.html)

i've implemented each step using ruby / streaming, the code is at
http://github.com/matpalm/sip if anyone cares.

i ran some tests using 10 ec2 medium cpu instance across a small
100mb's of gzipped text.
for validation of the results i also reimplemented the entire
algorithm as a single threaded ruby app

my surprise comes from finding that the ruby implementation
outperforms the 10 ec2 instances on this data size...
i ran a few samples of different sizes with the graph at the bottom of
http://matpalm.com/sip/part4_but_does_it_scale.html

so why is this? here are my explanations in order of how confident i am...

a) 100mb is peanuts and hadoop was made for 1000x this size so the
test is invalid.
b) there is a better representation of this problem that uses fewer
map/reduce passes.
c) streaming is too slow and rewriting in java (and making use of
techniques like chaining mappers) would speed things up
d) doing these steps, particularly the joins, in pig would be faster

my next steps are to rewrite some of the steps in pig to sample the differe=
nce

does anyone have any high level comments on this?

cheers,
mat


--_000_C6D56B201180Bscottrichrelevancecom_--

From general-return-522-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Sep 16 00:01:37 2009
Return-Path: <general-return-522-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23478 invoked from network); 16 Sep 2009 00:01:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Sep 2009 00:01:37 -0000
Received: (qmail 82018 invoked by uid 500); 16 Sep 2009 00:01:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81950 invoked by uid 500); 16 Sep 2009 00:01:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81940 invoked by uid 99); 16 Sep 2009 00:01:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Sep 2009 00:01:36 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [64.78.17.18] (HELO EXHUB018-3.exch018.msoutlookonline.net) (64.78.17.18)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Sep 2009 00:01:22 +0000
Received: from EXVMBX018-1.exch018.msoutlookonline.net ([64.78.17.47]) by
 EXHUB018-3.exch018.msoutlookonline.net ([64.78.17.18]) with mapi; Tue, 15 Sep
 2009 16:59:51 -0700
From: Scott Carey <scott@richrelevance.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 15 Sep 2009 16:59:45 -0700
Subject: Re: hadoop scales but is not performant?
Thread-Topic: hadoop scales but is not performant?
Thread-Index: Aco1ohy4R0PEAxULTzSBNeEZ+4RQ7AAtoMcSAAH+XeQ=
Message-ID: <C6D57881.11824%scott@richrelevance.com>
In-Reply-To: <C6D56B20.1180B%scott@richrelevance.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C6D5788111824scottrichrelevancecom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C6D5788111824scottrichrelevancecom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Some more comments:

How are you controlling your graph of different map reduce dependencies?  D=
oing this linearly will leave the cluster rather underutilized.  Using Pig =
or Cascading or the job control interface to schedule this all (which can s=
ubmit parallel independent jobs), combined with the Fair Scheduler, will us=
ually have very good results on smaller clusters in reducing overall batch =
latency and increasing cluster utilization.  This often allows setting the =
number of reduces for each job more optimally as well.

Although Pig will be a bit less performant than an optimized M/R, for tasks=
 like yours it can possibly go much faster with a lot less code by optimizi=
ng your M/R passes, scheduling, and providing some alternate join strategie=
s out of the box if used right.


On 9/15/09 4:02 PM, "Scott Carey" <scott@richrelevance.com> wrote:

Its not entirely invalid at this scale, though the dataset generally should=
 be (in memory) closer to the size of the available RAM of the cluster or m=
ore to be a better test.

Hadoop should still do well at this size relative to a single process.  If =
it is taking 45 minutes on a 10 machine cluster, it should be faster than a=
ny single process.  If this was limited primarily by startup costs, then in=
creasing the data size wouldn't almost linearly increase the time.

It can probably be significantly tuned.
Some tips for jobs this size:
Watch the CPU / RAM / IO on a node while it is running.  Is it being well u=
tilized?
Watch the jobs in the HTML UI on the jobtracker.  Are the total number of s=
lots configured being used most of the time?   Or is it spending a lot of c=
lock time with a lot of empty slots?  If so, which slots?  Are Maps or Redu=
ces taking the most time?

Hadoop currently is a slow scheduler for tasks (can schecule at most 1 task=
 per second or two per node).  Using the Fair Scheduler and enabling some o=
ptions can turn this up to 1 map and 1 reduce to schedule per 'tick'.  Ther=
e are some big changes in the schedulers due out in 0.21 that will signific=
antly help here.  For smaller low latency jobs this can make a big differen=
ce.

Hadoop also has some flaws in the shuffle phase that affect clusters of all=
 sizes, but can hurt small clusters with many small map jobs.   Look at the=
 log file output of your reduce jobs (in the jobtracker UI) and see how lon=
g the shuffle phase is taking.
There is a big change due for 0.21 that makes this a LOT faster for some ca=
ses, and low latency smaller jobs will benefit a lot too.
https://issues.apache.org/jira/browse/MAPREDUCE-318

See my comment in that ticket from the 10th of June in relation to shuffle =
times.  A one line change in 0.19.x and 0.20.x cut times on smaller low lat=
ency jobs on smaller clusters in half when limited by shuffle inefficiencie=
s (specifically, fetching no more than one map output from one host every f=
ew seconds for no good reason).

Tuning several hadoop parameters might help a great deal as well.   Clouder=
a's configuration wizard is a good starting point to help you find which kn=
obs are more important.


You probably also want to do some work to figure out what is taking time on=
 your local tests.  There might be something wrong there.  It seems suspici=
ous to me, even at the one document size, that it would take 2 + minutes ve=
rsus less than one second.  Startup costs aren't that big for 2 maps and 2 =
reduces - on the order of a few seconds not minutes.  A few seconds times a=
 few jobs is still not that much.



On 9/14/09 6:14 PM, "Mat Kelcey" <matthew.kelcey@gmail.com> wrote:

hi all,

recently i've started playing with hadoop and my first learning
experiment has surprised me

i'm implementing a a text problem; trying to extract infrequent
phrases using a mixture of probabilistic models.
it's a bit of toy problem so the algorithm details aren't super
important, though the implementation might be....

this problem ended up being represented by a dozen or so map reduce
jobs of various types including some aggregate steps and some manual
joins.
(see the bottom of this page for details
http://matpalm.com/sip/take3_markov_chains.html)

i've implemented each step using ruby / streaming, the code is at
http://github.com/matpalm/sip if anyone cares.

i ran some tests using 10 ec2 medium cpu instance across a small
100mb's of gzipped text.
for validation of the results i also reimplemented the entire
algorithm as a single threaded ruby app

my surprise comes from finding that the ruby implementation
outperforms the 10 ec2 instances on this data size...
i ran a few samples of different sizes with the graph at the bottom of
http://matpalm.com/sip/part4_but_does_it_scale.html

so why is this? here are my explanations in order of how confident i am...

a) 100mb is peanuts and hadoop was made for 1000x this size so the
test is invalid.
b) there is a better representation of this problem that uses fewer
map/reduce passes.
c) streaming is too slow and rewriting in java (and making use of
techniques like chaining mappers) would speed things up
d) doing these steps, particularly the joins, in pig would be faster

my next steps are to rewrite some of the steps in pig to sample the differe=
nce

does anyone have any high level comments on this?

cheers,
mat



--_000_C6D5788111824scottrichrelevancecom_--

From general-return-523-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Sep 16 02:11:17 2009
Return-Path: <general-return-523-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55500 invoked from network); 16 Sep 2009 02:11:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Sep 2009 02:11:17 -0000
Received: (qmail 68138 invoked by uid 500); 16 Sep 2009 02:11:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68090 invoked by uid 500); 16 Sep 2009 02:11:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68080 invoked by uid 99); 16 Sep 2009 02:11:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Sep 2009 02:11:12 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.92.24] (HELO qw-out-2122.google.com) (74.125.92.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Sep 2009 02:11:00 +0000
Received: by qw-out-2122.google.com with SMTP id 9so1364153qwb.35
        for <general@hadoop.apache.org>; Tue, 15 Sep 2009 19:10:37 -0700 (PDT)
MIME-Version: 1.0
Sender: edward@udanax.org
Received: by 10.224.58.15 with SMTP id e15mr6773757qah.221.1253067037452; Tue, 
	15 Sep 2009 19:10:37 -0700 (PDT)
In-Reply-To: <4b2c7b610909150210q7ff32de2s664079275f89fe2a@mail.gmail.com>
References: <eb4706e0909131926m48ddb6f8g2138ab3f76b9170@mail.gmail.com>
	 <12A8FC4D0563D1468AD23AAAED861A3EDE081D@EMAIL02.pnl.gov>
	 <eb4706e0909150125v5281428ay43820c9fb9e7d150@mail.gmail.com>
	 <4b2c7b610909150210q7ff32de2s664079275f89fe2a@mail.gmail.com>
Date: Wed, 16 Sep 2009 11:10:37 +0900
X-Google-Sender-Auth: 6931b21f6accd5c2
Message-ID: <eb4706e0909151910u1d3c139ao2d33c1cdd7ae03f6@mail.gmail.com>
Subject: Re: Discussion about Hamburg (provisional name) open sourcing
From: "Edward J. Yoon" <edwardyoon@apache.org>
To: hamburg-dev@googlegroups.com
Cc: general@hadoop.apache.org, hama-dev@incubator.apache.org, 
	Paolo Castagna <paolo.castagna@hp.com>, "Taylor, Ronald C" <ronald.taylor@pnl.gov>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Hyunsik,

I would suggest, we are to independently library-ize BSP component so
that we can commonly use it for an bulk-synchronous algorithms in a
matrix computational package, and development of BSP based graph
computing framework. Then, IMO, the top-level package, roughly, could
be as described below:

org.apache.hama.bsp
org.apache.hama.matrix
org.apache.hama.graph
org.apache.hama.examples

Let's discuss more details in the hama-dev@ and hamburg-dev@.

Thanks. ;)

On Tue, Sep 15, 2009 at 6:10 PM, Hyunsik Choi <c0d3h4ck@gmail.com> wrote:
>
> Hi,
>
> As you know, graph is a very useful data model. Matrix is the great
> tool to store graph data and process them. Therefore, I think that
> they will give many benefits each other. I agree with your opinion.
>
> If we do so, how can we integrate them? What do you think about that?
>
> Best regards,
> --
> Hyunsik Choi
> Database & Information Systems Group, Korea University
> http://diveintodata.org
>
>
>
> On Tue, Sep 15, 2009 at 5:25 PM, Edward J. Yoon <edwardyoon@apache.org> w=
rote:
>>
>> Thanks for interesting information.
>>
>> With hindsight, IMO, the Hama project is a best to incubate Hamburg
>> project. and, we can consider again when prepare to graduate from
>> incubator.
>>
>> On Mon, Sep 14, 2009 at 1:09 PM, Taylor, Ronald C <ronald.taylor@pnl.gov=
> wrote:
>>> Hello Mr. Yoon,
>>>
>>> I was delighted to hear of your proposed Hamburg project. I am a new
>>> user of Hadoop (and Hbase). It looks like that I will be spending a
>>> substantial amount of time working in this environment over the next
>>> couple years, for both DOE bioinformatics work (my primary field) and
>>> for work funded by DoD. I am enthusiastic about using Hadoop, Hive,
>>> Hbase. Also am quite interested in the Mahout project.
>>>
>>> While I cannot offer advice as to where to place your new project withi=
n
>>> the Apache framework, I did want to offer my support. I believe that it
>>> could well be of value in the coming years both to me, for my
>>> bioinformatics research, and to other researchers here at PNNL working
>>> in the areas of social networks (in our national security directorate)
>>> and in a set of projects directed toward making the electrical grid
>>> "smarter". I would not be able to contribute any code until I found
>>> funding from current or new projects for my time. But if Hamburg moves
>>> forward and can demonstrate its usefulness, that might be a real
>>> possibility.
>>>
>>> And in regards to funding for getting you some help: if you can find a
>>> collaborator based at a university or non-profit, said collaborator
>>> could well apply for a grant from the US National Science Foundation fo=
r
>>> open source Hadoop-based development of graph computing / mining
>>> algorithms. The NSF Computer and Information Science and Engineering
>>> Directorate is awarding grants specifically devoted to the area of grap=
h
>>> mining (at least this year - hopefully will continue next year - anyway=
,
>>> NSF gives money for algorithm and tool development in general - friendl=
y
>>> to that). I can't apply (at least not directly) - NSF does not like to
>>> give money to other US government labs. But I would think you could fin=
d
>>> someone in academia - perhaps someone already working with the Mahout
>>> group. It would appear a natural fit. I presume there are a number of
>>> people associated with the Apache org who know something about the NSF
>>> and could offer further advice in that direction.
>>>
>>> I look forward to hearing more about Hamburg, as it progresses.
>>>
>>> =C2=A0Best,
>>> =C2=A0Ron Taylor
>>>
>>> ___________________________________________
>>> Ronald Taylor, Ph.D.
>>> Computational Biology & Bioinformatics Group
>>> Pacific Northwest National Laboratory
>>> 902 Battelle Boulevard
>>> P.O. Box 999, MSIN K7-90
>>> Richland, WA =C2=A099352 USA
>>> Office: =C2=A0509-372-6568
>>> Email: ronald.taylor@pnl.gov
>>> www.pnl.gov
>>>
>>>
>>> -----Original Message-----
>>> From: edward@udanax.org [mailto:edward@udanax.org] On Behalf Of Edward
>>> J. Yoon
>>> Sent: Sunday, September 13, 2009 7:27 PM
>>> To: general@hadoop.apache.org; hama-dev@incubator.apache.org;
>>> hamburg-dev@googlegroups.com
>>> Cc: Paolo Castagna
>>> Subject: Discussion about Hamburg (provisional name) open sourcing
>>>
>>> Hello communities,
>>>
>>> I'm the one of the Hamburg (provisional name), which is the graph
>>> computing framework on Hadoop sponsor. Now we're working on the
>>> perfection of our prototype project, and we'll propose the Hamburg
>>> project soon.
>>>
>>> - http://wiki.apache.org/hadoop/Hamburg, a wiki page
>>> - http://throb.googlecode.com/, a prototype project
>>>
>>> BTW, before we decide to propose, we need time just to consider where i=
t
>>> belongs to.
>>>
>>> Since it aims to create a "general graph computing framework" on Hadoop=
,
>>> I'd like to propose it as a sub-project of Hadoop. On the other hand,
>>> since the matrix and graph are both in the domain of scientific
>>> computing and BSP model could be used for matrix computation areas, I
>>> think this project also can be integrated with the Hama project.
>>>
>>> WDYT? Any advices are welcome.
>>>
>>> --
>>> Best Regards, Edward J. Yoon @ NHN, corp.
>>> edwardyoon@apache.org
>>> http://blog.udanax.org
>>>
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon @ NHN, corp.
>> edwardyoon@apache.org
>> http://blog.udanax.org
>>
>



--=20
Best Regards, Edward J. Yoon @ NHN, corp.
edwardyoon@apache.org
http://blog.udanax.org

From general-return-524-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Sep 16 03:38:26 2009
Return-Path: <general-return-524-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 74359 invoked from network); 16 Sep 2009 03:38:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Sep 2009 03:38:26 -0000
Received: (qmail 15601 invoked by uid 500); 16 Sep 2009 03:38:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15463 invoked by uid 500); 16 Sep 2009 03:38:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15453 invoked by uid 99); 16 Sep 2009 03:38:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Sep 2009 03:38:25 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of matthew.kelcey@gmail.com designates 209.85.219.208 as permitted sender)
Received: from [209.85.219.208] (HELO mail-ew0-f208.google.com) (209.85.219.208)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Sep 2009 03:38:16 +0000
Received: by ewy4 with SMTP id 4so4702939ewy.36
        for <general@hadoop.apache.org>; Tue, 15 Sep 2009 20:37:55 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=FtQjYUM/nWaDFHXle2KG904d1zTMmu5nGPOz2/W1DCE=;
        b=ow37oHlOT21wfZWS1dFCSbRnRm+Drdj+Qme1XsHpiEFI7b+Z+BSYHzZuRFR4B3exac
         xFhoU2nXasTejUIrnEVenZK74tkgkep9yhhIGiwZqD1+kDN7FSnqGX4tF9/2QZjnAU2q
         2OhFCbD7ONaiN1Jsm++uqXS8rw5YuYcERjbpk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Yw6fyiQgvyPReZhCVbHO+Qs3O9qoUhFbYwecJcjymzl3fojMkeiqDnAY06zOUKYOeb
         11vQXbEoSLDsUkFoNCcXD4KLZIhdWLI2ZGgcjDNa5pfRlvlALPxGVqa1q5kZkE2Z5m+t
         rm0Y6DEFTvsz3S8JIruQ4g2BOEx5JL7toNKnk=
MIME-Version: 1.0
Received: by 10.211.128.9 with SMTP id f9mr9217582ebn.93.1253072275607; Tue, 
	15 Sep 2009 20:37:55 -0700 (PDT)
In-Reply-To: <C6D57881.11824%scott@richrelevance.com>
References: <C6D56B20.1180B%scott@richrelevance.com>
	 <C6D57881.11824%scott@richrelevance.com>
Date: Wed, 16 Sep 2009 13:37:55 +1000
Message-ID: <93d501de0909152037j4e6a53b8t4f2e972cf7783813@mail.gmail.com>
Subject: Re: hadoop scales but is not performant?
From: Mat Kelcey <matthew.kelcey@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

thanks scott, some great things to think about!

the only "tuning" i did was to set mapred.reduce.tasks and
mapred.map.tasks to 30 to correspond to the capability specified by
the html ui. i admit i did this without a deep understanding what it
meant, i do know that when i did not specify these then only a few
mappers would be utilised (due to the same input data size)

in relation to scheduling i was taking the simple approach of running
the streaming jobs sequentially with the default scheduler. even from
watching output scroll past it is obvious that a _lot_ of time is
being taken up in setup related activities. this is most apparent in
the single document case. something is just not right...

i had read in http://issues.apache.org/jira/browse/HADOOP-2721 "Use
job control for tasks (and therefore for pipes and streaming)"  that
jobcontrol (specifically representing job dependencies) was not yet
available for streaming. as such i dismissed any scheduling changes.
i'll revisit this to make sure i understand what i can and can't do in
streaming. if nothing else i can try fairscheduling with my own rolled
version of dependencies. i'm orchestrating the job runs from rake and
i've got my own homebrew libraries for this type of dependency
management, though i'm also loath to roll my own versions of things.

so lots of ideas and things to check, i'll rerun trying some of the
things you've mentioned.

thanks again for the feedback!
mat

From general-return-525-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Sep 16 03:41:15 2009
Return-Path: <general-return-525-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 74764 invoked from network); 16 Sep 2009 03:41:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Sep 2009 03:41:15 -0000
Received: (qmail 16590 invoked by uid 500); 16 Sep 2009 03:41:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16470 invoked by uid 500); 16 Sep 2009 03:41:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16460 invoked by uid 99); 16 Sep 2009 03:41:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Sep 2009 03:41:14 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of matthew.kelcey@gmail.com designates 209.85.219.208 as permitted sender)
Received: from [209.85.219.208] (HELO mail-ew0-f208.google.com) (209.85.219.208)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Sep 2009 03:41:05 +0000
Received: by ewy4 with SMTP id 4so4704173ewy.36
        for <general@hadoop.apache.org>; Tue, 15 Sep 2009 20:40:44 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=bgWv35SkaiucEFU3JI7sQD6AaDq2OqFdQN7xkph+X9U=;
        b=efLy6CnwHwFQgrf78ayxXbAqlnbO2ZOabJ3vlGeOLhmut5SQz2G25mJe2HMGIa0/dB
         Lc/DGoc3h1ZJgbvwW/RoVRtxjt13QDR1SdIUcBriYqUyk7WPjPyQ9cmrNmzoO7zimrpw
         BOro0ybkQtnmz+4lauXr1qCePAqqM2ya3wWd0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=BSDcqhEcZUALkxbIwORxCcXHrz6tZwSKEab/py3NIo9CFkzCUV7t+jG1MYaCpBoItY
         BJXi7dsK/HDrz34Fwr1VSgdXvg6MWXlea8UgEEOYVnVz6/hTNxIK8aqNjLZYjsPMoIXc
         WNEtIIau3KkR+NvJADj/AjWZqxN3RXtGrVPLk=
MIME-Version: 1.0
Received: by 10.211.129.8 with SMTP id g8mr9302657ebn.71.1253072444645; Tue, 
	15 Sep 2009 20:40:44 -0700 (PDT)
In-Reply-To: <93d501de0909152037j4e6a53b8t4f2e972cf7783813@mail.gmail.com>
References: <C6D56B20.1180B%scott@richrelevance.com>
	 <C6D57881.11824%scott@richrelevance.com>
	 <93d501de0909152037j4e6a53b8t4f2e972cf7783813@mail.gmail.com>
Date: Wed, 16 Sep 2009 13:40:44 +1000
Message-ID: <93d501de0909152040w1de486bw55a8138709d452e0@mail.gmail.com>
Subject: Re: hadoop scales but is not performant?
From: Mat Kelcey <matthew.kelcey@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

> mappers would be utilised (due to the same input data size)
whoops, i mean "small input data size"

From general-return-526-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Sep 16 17:51:39 2009
Return-Path: <general-return-526-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 28962 invoked from network); 16 Sep 2009 17:51:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Sep 2009 17:51:39 -0000
Received: (qmail 63326 invoked by uid 500); 16 Sep 2009 17:51:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63251 invoked by uid 500); 16 Sep 2009 17:51:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63241 invoked by uid 99); 16 Sep 2009 17:51:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Sep 2009 17:51:38 +0000
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.30.48] (HELO QMTA05.emeryville.ca.mail.comcast.net) (76.96.30.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Sep 2009 17:51:27 +0000
Received: from OMTA17.emeryville.ca.mail.comcast.net ([76.96.30.73])
	by QMTA05.emeryville.ca.mail.comcast.net with comcast
	id hUz81c0041afHeLA5Vr6nG; Wed, 16 Sep 2009 17:51:06 +0000
Received: from wlan-guest-112.corp.yahoo.com ([209.131.62.144])
	by OMTA17.emeryville.ca.mail.comcast.net with comcast
	id hVqw1c00336jmhM8dVqyGr; Wed, 16 Sep 2009 17:51:02 +0000
Message-Id: <D17D77C2-76AF-4367-A71E-EC8855885FEB@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: [ANNOUNCE] Hadoop Core 0.20.1 is released
Date: Wed, 16 Sep 2009 10:50:54 -0700
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

Hadoop Core 0.20.1 has been released and the website is now published.  
Get them while they are hot! Hadoop 0.20.1 is now considered the  
stable release and is recommended for new installations. There are a  
lot of important fixes that have been made over 0.20.0. Download the  
new release from:

http://www.apache.org/dyn/closer.cgi/hadoop/core/

The list of bug fixes that were addressed in the release are:

http://hadoop.apache.org/common/docs/r0.20.1/releasenotes.html

-- Owen

From general-return-527-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Sep 16 18:08:36 2009
Return-Path: <general-return-527-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 36131 invoked from network); 16 Sep 2009 18:08:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Sep 2009 18:08:36 -0000
Received: (qmail 88013 invoked by uid 500); 16 Sep 2009 18:08:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87931 invoked by uid 500); 16 Sep 2009 18:08:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87921 invoked by uid 99); 16 Sep 2009 18:08:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Sep 2009 18:08:35 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cabiwan@gmail.com designates 209.85.220.218 as permitted sender)
Received: from [209.85.220.218] (HELO mail-fx0-f218.google.com) (209.85.220.218)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Sep 2009 18:08:25 +0000
Received: by fxm18 with SMTP id 18so3872954fxm.29
        for <general@hadoop.apache.org>; Wed, 16 Sep 2009 11:08:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=TtIMgGhh4TbN38eD06eTPRXneNGz1ZO4D4I6Kzh1Kts=;
        b=WdFre/ND/J34nylao92OH4tRj+lpTVfOqW36FcgC2eEmFpJ/LZfwsaBNo7Z9ASpcrI
         bBxn7CgwzPEbBAyznHEIh31P5JH+H+t08GisIAKqXgxD9uxRXD6lCgKu0h3GM5Yi1L5+
         cRbefWuJavFpqMabh06i50U3j+qhv7+58wXeY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=RkeQgLOg/cFpliNy/PSPdtpqI/5VDVmT1gN38jZBzZSbr2v6MEH75i8svCm6hg7A65
         fBrfc9SF4tJkf6kIcssFXhL2A34bvSVWJj/Qv6nQhDvNcfzgV04KhOZUEcC60Br5QEJ+
         a2gyVTFwcoP3hn5vLrvesMclcBZG6tleUeT2M=
MIME-Version: 1.0
Received: by 10.103.76.12 with SMTP id d12mr3978487mul.25.1253124485693; Wed, 
	16 Sep 2009 11:08:05 -0700 (PDT)
Date: Wed, 16 Sep 2009 20:08:05 +0200
Message-ID: <309f76d00909161108t4de1bf2fvd771ae76cdff3bbd@mail.gmail.com>
Subject: Question about Eclipse plugin 0.18.0 and latest stable release 0.20.1
From: Alberto Luengo Cabanillas <cabiwan@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e65b611cd441540473b5c766
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e65b611cd441540473b5c766
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi all!
20 minutes ago, the latest stable version of Hadoop core has been released,
and that made me think about a clean reinstallation of my old Hadoop 0.18.0
core (running in a VMWare image). The question is that I access to that cor=
e
through the Eclipse plugin (in its 0.18.1 version) and if I "upgrade" the
HDFS to this latest version, how would the plugin for Eclipse work?. I=B4d
like to know if anyone upgraded the core, but left the old plugin working a=
s
usual (or with some workaround)...
Thanks a lot.

--=20
Alberto

--0016e65b611cd441540473b5c766--

From general-return-528-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Sep 17 21:08:38 2009
Return-Path: <general-return-528-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84098 invoked from network); 17 Sep 2009 21:08:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Sep 2009 21:08:37 -0000
Received: (qmail 74976 invoked by uid 500); 17 Sep 2009 21:08:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74917 invoked by uid 500); 17 Sep 2009 21:08:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74900 invoked by uid 99); 17 Sep 2009 21:08:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Sep 2009 21:08:36 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.191.125.35] (HELO web38404.mail.mud.yahoo.com) (209.191.125.35)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 17 Sep 2009 21:08:27 +0000
Received: (qmail 93979 invoked by uid 60001); 17 Sep 2009 21:08:05 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1253221685; bh=9sJDHh5f4YP8Eq5Jo07KwO3ARUQao6gMKU9VMZKh9EE=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=IY8S2f7bIuk2LCPQtVWyxQzBz2JVvPuZVakJOj2a2A4JqK118UWurGyKHiek9PxDbJb9F5govQnRF6xLl2yLbqPogolPDv7EG4BXEFvw6OfVZHjOBgpfXe2gOfFvpRnlrtdeUJ2Mgdt4lVMFS+2qUnXFGTjSMDpFoM8m8zzKKNw=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=ZSF8vBSo3P6pm5HvXbdJSx8fkyxi0meDoEWJOqG+FGAG6qlRr6ya0aPVAP743UnJYVMpprY/QAc3yrKXFekSmCMYTlADdJR9ajyK8jTZrtxFCjQE9oRb/79Eh0njMBf7QrE3IQMW5RxnZUHO0g+rjUjH1+xBUqGFtZQjOaOym+I=;
Message-ID: <904767.92694.qm@web38404.mail.mud.yahoo.com>
X-YMail-OSG: 2sT5kSIVM1n9IlnBgBx2GU_xfPPdV9fkYxCZrm29KXKdEGXmEIrl5YvtkYBt7hiIVGnWFsmAJ93IKHxXUocHR4_JZTmX_v3TnlcHJdhH26nNORDz3eyuyda7Noegyv7d4n7IyPkaX_OryUNiv9mUeRHeHV5EtgdX..ntlEodtZ2mPYFSo83ThiAdUin.QyVriUjNTi8xAXvnUq4oQ9a0yXX9yg575E05t5pE7AUnleK_ReJjvmXTz9wmQiqfenm7fnvOOcROC2yYMWJ9ttqxQGOCWGRfuTPn1XfALg931Ity7sQaxePOFi17KFmUI2XaBgQ7ejspvbf0hi7pYUn155wqgZuGaoEiMhT_Jp6ihpLfCVc0_dPHt2VXl0buyQboXtm47EYMA5VPhm70ykTep4zD
Received: from [129.170.212.235] by web38404.mail.mud.yahoo.com via HTTP; Thu, 17 Sep 2009 14:08:05 PDT
X-Mailer: YahooMailRC/157.18 YahooMailWebService/0.7.347.2
Date: Thu, 17 Sep 2009 14:08:05 -0700 (PDT)
From: himanshu chandola <himanshu_coolguy@yahoo.com>
Subject: hadoop hangs on reduce
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,
Has anyone seen hadoop getting stuck on reduces? 
I'm using a compiled version of hadoop from cloudera:

Hadoop 0.18.3-14.cloudera.CH0_3
Subversion  -r HEAD
Compiled by root on Mon Jul  6 15:02:31 EDT 2009

I've a map reduce job and hadoop gets stuck at 96.49% with no progress since the last 1 hour. I've tried to look into the logs and there isn't anything interesting there. Here's the log from the last  1 hour:

>>>>
2009-09-17 14:51:47,033 ERROR org.apache.hadoop.dfs.DataNode: DatanodeRegistration(10.42.1.1:50010, storageID=DS-1052225239-129.170.192.42-50010-1252112893659, infoPort=50075, ipcPort=50020):DataXceiver: java.net.SocketTimeoutException: 480000 millis timeout while waiting for channel to be ready for write. ch : java.nio.channels.SocketChannel[connected local=/10.42.1.1:50010 remote=/10.42.255.247:41915]
    at org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:246)
    at org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159)
    at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198)
    at org.apache.hadoop.dfs.DataNode$BlockSender.sendChunks(DataNode.java:1938)
    at org.apache.hadoop.dfs.DataNode$BlockSender.sendBlock(DataNode.java:2032)
    at org.apache.hadoop.dfs.DataNode$DataXceiver.readBlock(DataNode.java:1159)
    at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:1087)
    at java.lang.Thread.run(Thread.java:619)

2009-09-17 14:52:29,916 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 40 blocks got processed in 7 msecs
2009-09-17 15:15:06,586 INFO org.apache.hadoop.dfs.DataBlockScanner: Verification succeeded for blk_4472685249728744796_2145
2009-09-17 15:52:28,686 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 40 blocks got processed in 8 msecs
2009-09-17 16:52:30,397 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 40 blocks got processed in 8 msecs
>>>>
while the hadoop reduce is stuck at 96.49 for the last 1 hr:
09/09/17 15:49:49 INFO mapred.JobClient:  map 100% reduce 96%

This is the second time I've tried to run this code and both of the times I've seen it hitting a barrier on reduce. My reduce step is just aggregating all of the keys and dumping them into files using MultipleTextOutputFormat. I've run a problem of similar size before where the input to reduce was of the same size as this one.

Any help would be greatly appreciated. I can't seem to find any reason why this is happening.

Thanks

Himanshu

 Morpheus: Do you believe in fate, Neo?
Neo: No.
Morpheus: Why Not?
Neo: Because I don't like the idea that I'm not in control of my life.



      

From general-return-529-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Sep 18 03:51:46 2009
Return-Path: <general-return-529-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 92280 invoked from network); 18 Sep 2009 03:51:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Sep 2009 03:51:45 -0000
Received: (qmail 11106 invoked by uid 500); 18 Sep 2009 03:51:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11009 invoked by uid 500); 18 Sep 2009 03:51:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 80318 invoked by uid 99); 15 Sep 2009 09:11:55 -0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of c0d3h4ck@gmail.com designates 209.85.223.193 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:cc:content-type
         :content-transfer-encoding;
        bh=rAFWTxBueXT1iJNI7czdWCWD6I2tqSCjBvYJHMTZMyQ=;
        b=CpIf1cSGCEhAJo6m3PR30cbtGl3+w2nS7+ulWui359gnGpKYKANvfK073W2j6nLHJ3
         O8+C4HP2oCrvlBqtEoTXmtWUkPqMYtikVnQGrOrccktKM5JtiOGJnJKqVkRQwBqqSLAZ
         UwkxAueahKxDC7LSey0fWU3sKZGrRX+il8DYc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type:content-transfer-encoding;
        b=dPzrpbUXAJydNZlp+XlL+IoySaFzl30oqH6ft8gdee38G3pX+aU5x2ABSWKzmzYvT+
         G2MEq6PS11yVYzf+c584EHpR2QVPGbRDbfqkIPnKb30c7AgrWFDg3TWVe+HL3dnOgDXV
         gubyQt2tF4hhmOXGJKudnI/vzJsDgGGcgolJE=
MIME-Version: 1.0
In-Reply-To: <eb4706e0909150125v5281428ay43820c9fb9e7d150@mail.gmail.com>
References: <eb4706e0909131926m48ddb6f8g2138ab3f76b9170@mail.gmail.com>
	 <12A8FC4D0563D1468AD23AAAED861A3EDE081D@EMAIL02.pnl.gov>
	 <eb4706e0909150125v5281428ay43820c9fb9e7d150@mail.gmail.com>
Date: Tue, 15 Sep 2009 18:10:19 +0900
Message-ID: <4b2c7b610909150210q7ff32de2s664079275f89fe2a@mail.gmail.com>
Subject: Re: Discussion about Hamburg (provisional name) open sourcing
From: Hyunsik Choi <c0d3h4ck@gmail.com>
To: hamburg-dev@googlegroups.com
Cc: general@hadoop.apache.org, hama-dev@incubator.apache.org, 
	Paolo Castagna <paolo.castagna@hp.com>, "Taylor, Ronald C" <ronald.taylor@pnl.gov>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

As you know, graph is a very useful data model. Matrix is the great
tool to store graph data and process them. Therefore, I think that
they will give many benefits each other. I agree with your opinion.

If we do so, how can we integrate them? What do you think about that?

Best regards,
--
Hyunsik Choi
Database & Information Systems Group, Korea University
http://diveintodata.org



On Tue, Sep 15, 2009 at 5:25 PM, Edward J. Yoon <edwardyoon@apache.org> wro=
te:
>
> Thanks for interesting information.
>
> With hindsight, IMO, the Hama project is a best to incubate Hamburg
> project. and, we can consider again when prepare to graduate from
> incubator.
>
> On Mon, Sep 14, 2009 at 1:09 PM, Taylor, Ronald C <ronald.taylor@pnl.gov>=
 wrote:
>> Hello Mr. Yoon,
>>
>> I was delighted to hear of your proposed Hamburg project. I am a new
>> user of Hadoop (and Hbase). It looks like that I will be spending a
>> substantial amount of time working in this environment over the next
>> couple years, for both DOE bioinformatics work (my primary field) and
>> for work funded by DoD. I am enthusiastic about using Hadoop, Hive,
>> Hbase. Also am quite interested in the Mahout project.
>>
>> While I cannot offer advice as to where to place your new project within
>> the Apache framework, I did want to offer my support. I believe that it
>> could well be of value in the coming years both to me, for my
>> bioinformatics research, and to other researchers here at PNNL working
>> in the areas of social networks (in our national security directorate)
>> and in a set of projects directed toward making the electrical grid
>> "smarter". I would not be able to contribute any code until I found
>> funding from current or new projects for my time. But if Hamburg moves
>> forward and can demonstrate its usefulness, that might be a real
>> possibility.
>>
>> And in regards to funding for getting you some help: if you can find a
>> collaborator based at a university or non-profit, said collaborator
>> could well apply for a grant from the US National Science Foundation for
>> open source Hadoop-based development of graph computing / mining
>> algorithms. The NSF Computer and Information Science and Engineering
>> Directorate is awarding grants specifically devoted to the area of graph
>> mining (at least this year - hopefully will continue next year - anyway,
>> NSF gives money for algorithm and tool development in general - friendly
>> to that). I can't apply (at least not directly) - NSF does not like to
>> give money to other US government labs. But I would think you could find
>> someone in academia - perhaps someone already working with the Mahout
>> group. It would appear a natural fit. I presume there are a number of
>> people associated with the Apache org who know something about the NSF
>> and could offer further advice in that direction.
>>
>> I look forward to hearing more about Hamburg, as it progresses.
>>
>> =A0Best,
>> =A0Ron Taylor
>>
>> ___________________________________________
>> Ronald Taylor, Ph.D.
>> Computational Biology & Bioinformatics Group
>> Pacific Northwest National Laboratory
>> 902 Battelle Boulevard
>> P.O. Box 999, MSIN K7-90
>> Richland, WA =A099352 USA
>> Office: =A0509-372-6568
>> Email: ronald.taylor@pnl.gov
>> www.pnl.gov
>>
>>
>> -----Original Message-----
>> From: edward@udanax.org [mailto:edward@udanax.org] On Behalf Of Edward
>> J. Yoon
>> Sent: Sunday, September 13, 2009 7:27 PM
>> To: general@hadoop.apache.org; hama-dev@incubator.apache.org;
>> hamburg-dev@googlegroups.com
>> Cc: Paolo Castagna
>> Subject: Discussion about Hamburg (provisional name) open sourcing
>>
>> Hello communities,
>>
>> I'm the one of the Hamburg (provisional name), which is the graph
>> computing framework on Hadoop sponsor. Now we're working on the
>> perfection of our prototype project, and we'll propose the Hamburg
>> project soon.
>>
>> - http://wiki.apache.org/hadoop/Hamburg, a wiki page
>> - http://throb.googlecode.com/, a prototype project
>>
>> BTW, before we decide to propose, we need time just to consider where it
>> belongs to.
>>
>> Since it aims to create a "general graph computing framework" on Hadoop,
>> I'd like to propose it as a sub-project of Hadoop. On the other hand,
>> since the matrix and graph are both in the domain of scientific
>> computing and BSP model could be used for matrix computation areas, I
>> think this project also can be integrated with the Hama project.
>>
>> WDYT? Any advices are welcome.
>>
>> --
>> Best Regards, Edward J. Yoon @ NHN, corp.
>> edwardyoon@apache.org
>> http://blog.udanax.org
>>
>
>
>
> --
> Best Regards, Edward J. Yoon @ NHN, corp.
> edwardyoon@apache.org
> http://blog.udanax.org
>

From general-return-530-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Sep 18 18:35:07 2009
Return-Path: <general-return-530-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82798 invoked from network); 18 Sep 2009 18:35:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Sep 2009 18:35:07 -0000
Received: (qmail 73651 invoked by uid 500); 18 Sep 2009 18:35:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73566 invoked by uid 500); 18 Sep 2009 18:35:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73555 invoked by uid 99); 18 Sep 2009 18:35:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Sep 2009 18:35:06 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [209.191.125.36] (HELO web38405.mail.mud.yahoo.com) (209.191.125.36)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 18 Sep 2009 18:34:55 +0000
Received: (qmail 12267 invoked by uid 60001); 18 Sep 2009 18:34:34 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1253298874; bh=60ed4Ds68Mj2XZ8ZLO7sphamlvxRbkuHDyQCj8TMJ48=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=JpxguBpY8YESBOjmjK8/aheEq/SnaA1eZQtDPsSwtiItbfTh1MZxsVcHes8y2W1OjMHPEVrUAPLPYdAppV760HTo0Px+1CBc5Dzz7ADWj5QX2mkdQM0N5VOfu2pHVU16T9d6bB+dwMh2/cQOmJnn/sNyvctQx9yjbPD70muwMqA=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=VwxfQBYxOfkT6P+M0U6Hvnazodf1VeSHVHmWz2AYtDHLHJZAtbckYUp7TTJPlWZ1DJjl4SUGLlWH32XhB7IAoGXWLZzt2qU32lCpPhDojlB8PRHrHi2/mJBFg8wCBPBoi1ripZxXPzQHTJg/aoIa+CJJkmX7Hucx4KIROhCs7rw=;
Message-ID: <232426.10791.qm@web38405.mail.mud.yahoo.com>
X-YMail-OSG: t6L1P04VM1lX_O5.XrxxJpx6vPn1VuLqOu3zZJCyAYMcvFwr1Ye5RMh0EXHj7ZCPLKL0VCa9rVTi5rEpEj5N_pTJ3ebeJpZ7Vh861ZZWcvxhq4kAcOrDQaPVuUNdheu9X2U68aAYytCpUksSjgcI60jugcpU6mgwail7PhjzoFNRfFDmnWwFCm4LflYRKZeGZhJFP62vSSzlF1JqKAMIFuw_TM0MxI7nc..AHvtm4OgpMVQ7YQL4IkjaeeYmsG_pNAZV6qX8hmcQ8dS1WX6UnjE3tZ4ySr4dBJMrQcKnqSeyX.zIq3qz1JzOWiFKhrY5zdYm8Td9OxujVVNZmk6gxlIBEy5Y0ddPYTZx8VXKBTR37IS0nzix9Xt9FgSHmfStcOmVZYM-
Received: from [129.170.212.235] by web38405.mail.mud.yahoo.com via HTTP; Fri, 18 Sep 2009 11:34:34 PDT
X-Mailer: YahooMailRC/157.18 YahooMailWebService/0.7.347.2
References: <904767.92694.qm@web38404.mail.mud.yahoo.com>
Date: Fri, 18 Sep 2009 11:34:34 -0700 (PDT)
From: himanshu chandola <himanshu_coolguy@yahoo.com>
Subject: Re: hadoop hangs on reduce
To: general@hadoop.apache.org
In-Reply-To: <904767.92694.qm@web38404.mail.mud.yahoo.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org

Just as an update. I made a dummy map job so that the map outputs a unique key for every input and hence the input to reduce is unique too. Still my reduce jobs hang at 76.02 % now (I've added a few nodes into my cluster so I suspect what was earlier 96.49 is 76.02). So this is definitely not a memory or io issue. 

Do I restart my task trackers ? (ive tried once but didnt help)

thanks

 Morpheus: Do you believe in fate, Neo?
Neo: No.
Morpheus: Why Not?
Neo: Because I don't like the idea that I'm not in control of my life.



----- Original Message ----
From: himanshu chandola <himanshu_coolguy@yahoo.com>
To: general@hadoop.apache.org
Sent: Thursday, September 17, 2009 5:08:05 PM
Subject: hadoop hangs on reduce

Hi,
Has anyone seen hadoop getting stuck on reduces? 
I'm using a compiled version of hadoop from cloudera:

Hadoop 0.18.3-14.cloudera.CH0_3
Subversion  -r HEAD
Compiled by root on Mon Jul  6 15:02:31 EDT 2009

I've a map reduce job and hadoop gets stuck at 96.49% with no progress since the last 1 hour. I've tried to look into the logs and there isn't anything interesting there. Here's the log from the last  1 hour:

>>>>
2009-09-17 14:51:47,033 ERROR org.apache.hadoop.dfs.DataNode: DatanodeRegistration(10.42.1.1:50010, storageID=DS-1052225239-129.170.192.42-50010-1252112893659, infoPort=50075, ipcPort=50020):DataXceiver: java.net.SocketTimeoutException: 480000 millis timeout while waiting for channel to be ready for write. ch : java.nio.channels.SocketChannel[connected local=/10.42.1.1:50010 remote=/10.42.255.247:41915]
    at org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:246)
    at org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159)
    at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198)
    at org.apache.hadoop.dfs.DataNode$BlockSender.sendChunks(DataNode.java:1938)
    at org.apache.hadoop.dfs.DataNode$BlockSender.sendBlock(DataNode.java:2032)
    at org.apache.hadoop.dfs.DataNode$DataXceiver.readBlock(DataNode.java:1159)
    at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:1087)
    at java.lang.Thread.run(Thread.java:619)

2009-09-17 14:52:29,916 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 40 blocks got processed in 7 msecs
2009-09-17 15:15:06,586 INFO org.apache.hadoop.dfs.DataBlockScanner: Verification succeeded for blk_4472685249728744796_2145
2009-09-17 15:52:28,686 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 40 blocks got processed in 8 msecs
2009-09-17 16:52:30,397 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 40 blocks got processed in 8 msecs
>>>>
while the hadoop reduce is stuck at 96.49 for the last 1 hr:
09/09/17 15:49:49 INFO mapred.JobClient:  map 100% reduce 96%

This is the second time I've tried to run this code and both of the times I've seen it hitting a barrier on reduce. My reduce step is just aggregating all of the keys and dumping them into files using MultipleTextOutputFormat. I've run a problem of similar size before where the input to reduce was of the same size as this one.

Any help would be greatly appreciated. I can't seem to find any reason why this is happening.

Thanks

Himanshu

Morpheus: Do you believe in fate, Neo?
Neo: No.
Morpheus: Why Not?
Neo: Because I don't like the idea that I'm not in control of my life.


      

From general-return-531-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 21 10:10:33 2009
Return-Path: <general-return-531-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91844 invoked from network); 21 Sep 2009 10:10:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Sep 2009 10:10:33 -0000
Received: (qmail 82773 invoked by uid 500); 21 Sep 2009 10:10:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82686 invoked by uid 500); 21 Sep 2009 10:10:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82676 invoked by uid 99); 21 Sep 2009 10:10:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Sep 2009 10:10:32 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Sep 2009 10:10:20 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 1DDB7B7CD1
	for <general@hadoop.apache.org>; Mon, 21 Sep 2009 11:09:59 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id A4Hv+x6LImDF for <general@hadoop.apache.org>;
	Mon, 21 Sep 2009 11:09:53 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 3FF78B7CCF
	for <general@hadoop.apache.org>; Mon, 21 Sep 2009 11:09:53 +0100 (BST)
MailScanner-NULL-Check: 1254132580.76735@ZTxUz5Q6IUlaVMRp/KAQgg
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id n8LA9eHH008689
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Mon, 21 Sep 2009 11:09:40 +0100 (BST)
Message-ID: <4AB750E4.70300@apache.org>
Date: Mon, 21 Sep 2009 11:09:40 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: hadoop hangs on reduce
References: <904767.92694.qm@web38404.mail.mud.yahoo.com> <232426.10791.qm@web38405.mail.mud.yahoo.com>
In-Reply-To: <232426.10791.qm@web38405.mail.mud.yahoo.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: n8LA9eHH008689
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

himanshu chandola wrote:
> Just as an update. I made a dummy map job so that the map outputs a unique key for every input and hence the input to reduce is unique too. Still my reduce jobs hang at 76.02 % now (I've added a few nodes into my cluster so I suspect what was earlier 96.49 is 76.02). So this is definitely not a memory or io issue. 
> 
> Do I restart my task trackers ? (ive tried once but didnt help)
> 

I see reduce hangs when the TT's cant talk to each other, when they 
can't get data from the other TTs

check the value of mapred.task.tracker.report.address , that it is on an 
external address (not 127.0.0.1) and that the port in use is open on all 
the machines.

-steve


From general-return-532-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 22 23:22:12 2009
Return-Path: <general-return-532-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68106 invoked from network); 22 Sep 2009 23:22:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Sep 2009 23:22:12 -0000
Received: (qmail 20560 invoked by uid 500); 22 Sep 2009 23:22:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20512 invoked by uid 500); 22 Sep 2009 23:22:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 20502 invoked by uid 99); 22 Sep 2009 23:22:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Sep 2009 23:22:10 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cabiwan@gmail.com designates 209.85.218.214 as permitted sender)
Received: from [209.85.218.214] (HELO mail-bw0-f214.google.com) (209.85.218.214)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Sep 2009 23:22:01 +0000
Received: by bwz10 with SMTP id 10so178101bwz.29
        for <general@hadoop.apache.org>; Tue, 22 Sep 2009 16:21:41 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=KS4QdGaioB8NQYWiPSSTT7mCgEt9WkpBS3muKA17Sp4=;
        b=vodjWrP+pTzL7ob3SYH0aaEHYLMowrg94xttRPWx/75w6H9ycFEjEpfKxImN0LqbZZ
         FgmdGCVwkAW0JSj9rekYUI55gWTKqgFtEf6sUhg2aiPSXzZa9cMd1HgJaDTXoolWr6M8
         lBGDKP8QGgrTLyPbzA/zcsyh5z0pY8zl+ELpM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=b1motqY3q6G1KFgyOTXdo5xh772a6RHH435LotKMaxR6REFXMq5L8pi+1kDWHDQgAF
         k4j4KvEqqV1YT3afbakc3Vhct4uTGhJWwg9vZHZ6IkbHcdtoHUEifArIkkdKevvMb8IL
         Wp66/2nFxF0nV9yHrbD6t6t2ulifnrK3Dc42U=
MIME-Version: 1.0
Received: by 10.102.211.35 with SMTP id j35mr686076mug.35.1253661700993; Tue, 
	22 Sep 2009 16:21:40 -0700 (PDT)
Date: Wed, 23 Sep 2009 01:21:40 +0200
Message-ID: <309f76d00909221621x603129dch2c026b61c7373dde@mail.gmail.com>
Subject: Question about remote execution with Hadoop
From: Alberto Luengo Cabanillas <cabiwan@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163641781f5b34d2047432dcee
X-Virus-Checked: Checked by ClamAV on apache.org

--00163641781f5b34d2047432dcee
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi everyone!
I=B4m actually working on a project involving Map-Reduce and genetics
algorithms (with an iterative approach). I=B4m developing this project with
Eclipse 3.5 (with PigPen and Hadoop 0.18.1 plugins integrated). There=B4s a
class called "Client" which is actually the entry point to the MapReduce
Runtime System (user data introduction, etc). There=B4s also a "MRPGAMaster=
"
file, which works as the Driver for the Hadoop framework. In this class
there=B4s a "main" method where I declare the job to submit to Map-Reduce (=
as
usual) so as the Mapper, Reducer, etc, but I=B4m having problems finding a =
way
to call this "main" method from the "main" method in the "Client" class.
I=B4ve tried this:

MRPGAMaster master =3D new MRPGAMaster();
master.main();

but the job in MRPGAMaster is executed locally and not in the HDFS (which i=
s
running in a Virtual Machine).
What I=B4m trying to do is possible? Does anyone have any idea?.
Thanks a lot!
--=20
Alberto

--00163641781f5b34d2047432dcee--

From general-return-533-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Sep 25 08:14:24 2009
Return-Path: <general-return-533-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83727 invoked from network); 25 Sep 2009 08:14:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Sep 2009 08:14:24 -0000
Received: (qmail 81697 invoked by uid 500); 25 Sep 2009 08:14:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81622 invoked by uid 500); 25 Sep 2009 08:14:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81612 invoked by uid 99); 25 Sep 2009 08:14:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Sep 2009 08:14:22 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.85.211.185] (HELO mail-yw0-f185.google.com) (209.85.211.185)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Sep 2009 08:14:11 +0000
Received: by ywh15 with SMTP id 15so3008105ywh.5
        for <general@hadoop.apache.org>; Fri, 25 Sep 2009 01:13:47 -0700 (PDT)
Received: by 10.90.144.10 with SMTP id r10mr580030agd.70.1253866427646;
        Fri, 25 Sep 2009 01:13:47 -0700 (PDT)
Received: from nsf264fa356e37 ([202.51.92.14])
        by mx.google.com with ESMTPS id 5sm1303109agc.47.2009.09.25.01.13.45
        (version=SSLv3 cipher=RC4-MD5);
        Fri, 25 Sep 2009 01:13:47 -0700 (PDT)
From: "Chandan Tamrakar" <chandan.tamrakar@nepasoft.com>
To: <general@hadoop.apache.org>
Subject: Integrating Lucene in Hadoop
Date: Fri, 25 Sep 2009 13:58:41 +0545
Message-ID: <4abc7bbb.05035a0a.55c7.ffffd84c@mx.google.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_NextPart_000_0030_01CA3DE8.4C675D70"
X-Mailer: Microsoft Office Outlook 12.0
Thread-Index: Aco9uBewVT+JU953TmSoQkzEl1fo6Q==
Content-Language: en-us
X-Virus-Checked: Checked by ClamAV on apache.org

------=_NextPart_000_0030_01CA3DE8.4C675D70
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit

I was doing few R&D on  integrating hadoop and  Lucene . I could create
lucene indexes into HDFS using  index contribution provided for Hadoop

 

Is this a proper way to create Lucene index how much it is different from a
project KATTA ?

 

Please suggest what would be the better approach to create distributed
lucene indexes 

 

Thanks in advance


------=_NextPart_000_0030_01CA3DE8.4C675D70--


From general-return-534-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Sep 26 00:37:10 2009
Return-Path: <general-return-534-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81938 invoked from network); 26 Sep 2009 00:37:10 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Sep 2009 00:37:10 -0000
Received: (qmail 18066 invoked by uid 500); 26 Sep 2009 00:37:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17977 invoked by uid 500); 26 Sep 2009 00:37:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17967 invoked by uid 99); 26 Sep 2009 00:37:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Sep 2009 00:37:08 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [64.78.17.19] (HELO EXHUB018-4.exch018.msoutlookonline.net) (64.78.17.19)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Sep 2009 00:36:57 +0000
Received: from EXVMBX018-1.exch018.msoutlookonline.net ([64.78.17.47]) by
 EXHUB018-4.exch018.msoutlookonline.net ([64.78.17.19]) with mapi; Fri, 25 Sep
 2009 17:36:35 -0700
From: Scott Carey <scott@richrelevance.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>,
	"general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 25 Sep 2009 17:36:25 -0700
Subject: Re: HTTP transport?
Thread-Topic: HTTP transport?
Thread-Index: AcozKK1fBLyodrspRxSIuJsYqlOv9QLGLNZL
Message-ID: <C6E2B019.127CC%scott@richrelevance.com>
In-Reply-To: <4AAAC403.80809@apache.org>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C6E2B019127CCscottrichrelevancecom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C6E2B019127CCscottrichrelevancecom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Ok, I have some thoughts on this.  I might be misinterpreting some use case=
s here however.

HTTP is very useful and typically performs very well.  It has lots of thing=
s built-in too.  In addition to what you mention, it  has a caching mechani=
sm built-in, range queries, and all sorts of ways to tag along state if nee=
ded.  To top it off there are a lot of testing and debugging tools availabl=
e for it.  So from that front using it is very attractive.

However, In my experience zero-copy is not going to be much of a gain perfo=
rmance-wise for this sort of application, and will limit what can be done. =
 As long as a servlet doesn't transcode data and mostly copies, it will be =
very fast - many multiples of gigabit ethernet speed per CPU - far more tha=
n most disk setups will handle for a while.  Furthermore, it is easier to o=
ptimize disk requests to be 'sequentially chunky' if it goes through the JV=
M.  And I suspect that for many use cases, optimizing disk I/O is more valu=
able than a little bit of extra CPU spent copying data into and out of the =
process.
Additionally, I'm not sure CRC checking should occur on the client.  TCP/IP=
 already checksums packets, so network data corruption over HTTP is not a p=
rimary concern.   The big concern is silent data corruption on the disk.  F=
or the DataNode use case, it should find such errors as early as possible, =
and not rely on clients discovering errors.  Then it can coordinate with th=
e NameNode on fixing the block or discarding it.  So if it has to check the=
 file integrity anyway, there is no reason to worry about zero-copy.  Avoid=
ing the extra request for the CRC data at least partially counters the loss=
 of zero-copy.

Additionally, embedding Tomcat tends to be more tricky than Jetty, though t=
hat can be overcome.  One might argue that we don't even want a servlet con=
tainer, we just want an HTTP connector.  The Servlet API is familiar, but f=
or a high performance transport it might just be overhead and restrictive. =
 Direct access to Tomcat's NIO connector might be significantly lighter-wei=
ght and more flexible.
Tomcat's NIO connector implementation works great and I have had great succ=
ess with up to 10K connections with the pure Java connector using ordinary =
byte buffers and about 20 servlet threads.  But if a large number of open c=
onnections are not needed (less than about 5x the number of CPU core thread=
s) then thread-per-connection servlet containers work ok too.  These sort o=
f implementation details can evolve over time however.

Just my 2c

-Scott



On 9/11/09 2:41 PM, "Doug Cutting" <cutting@apache.org> wrote:

I'm considering an HTTP-based transport for Avro as the preferred,
high-performance option.

HTTP has lots of advantages.  In particular, it already has
  - lots of authentication, authorization and encryption support;
  - highly optimized servers;
  - monitoring, logging, etc.

Tomcat and other servlet containers support async NIO, where a thread is
not required per connection.  A servlet can process bulk data with a
single copy to and from the socket (bypassing stream buffers).  Calls
can be multiplexed over a single HTTP connection using Comet events.

http://tomcat.apache.org/tomcat-6.0-doc/aio.html

Zero copy is not an option for servlets that generate arbitrary data,
but one can specify a file/start/length tuple and Tomcat will use
sendfile to write the response.  That means that while HDFS datanode
file reads could not be done via RPC, they could be done via HTTP with
zero-copy.  If authentication and authorization are already done in the
HTTP server, this may not be a big loss.  The HDFS client might make two
HTTP requests, one to read a files data, and another to read its
checksums.  The server would then stream the entire block to the client
using sendfile, using TCP flow control as today.

Thoughts?

Doug


--_000_C6E2B019127CCscottrichrelevancecom_--

From general-return-535-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 28 04:30:45 2009
Return-Path: <general-return-535-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25481 invoked from network); 28 Sep 2009 04:30:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Sep 2009 04:30:45 -0000
Received: (qmail 60783 invoked by uid 500); 28 Sep 2009 04:30:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60688 invoked by uid 500); 28 Sep 2009 04:30:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60678 invoked by uid 99); 28 Sep 2009 04:30:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Sep 2009 04:30:43 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jason.hadoop@gmail.com designates 209.85.212.186 as permitted sender)
Received: from [209.85.212.186] (HELO mail-vw0-f186.google.com) (209.85.212.186)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Sep 2009 04:30:35 +0000
Received: by vws16 with SMTP id 16so2764439vws.2
        for <general@hadoop.apache.org>; Sun, 27 Sep 2009 21:30:14 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=1TFhVdpQ2gfU7AvDMBs/yK+xCXuUMmAbUQtwu3qJCTw=;
        b=EaQ0WkSViSJX9E23QaRrxnETWqc1y0hDtHsdN/mWXy4xFXkKJ2IwLLIPQHQr5qX5Im
         rSQwnGT1Gfg+hpsPRi8YP7XihEg1jkyAARPbBQaTu5Go0bBTTbs1tezXm1WHk6yrgiwz
         FSRCIKhZJhXlG7tImQ7RsC/VJlsgTRcWwzBic=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=cQotK9z7F3bmLBEoj3EC/aKTbrAofYJZ/zJ0CEyckvKgTTTI5Ellx+3j9YJ8S/tDVA
         Xg5VmllJTloxwXIsdZsp9e+v8JLY11gDBRG5pz3SkvI7ZBZ/zHk8qgSeafHbhzre/Kvh
         8dC3vx6jmCdyV9V4caCsCSUMMMFk/noE2XiRg=
MIME-Version: 1.0
Received: by 10.220.11.9 with SMTP id r9mr5067241vcr.58.1254112213723; Sun, 27 
	Sep 2009 21:30:13 -0700 (PDT)
In-Reply-To: <4abc7bbb.05035a0a.55c7.ffffd84c@mx.google.com>
References: <Aco9uBewVT+JU953TmSoQkzEl1fo6Q==>
	 <4abc7bbb.05035a0a.55c7.ffffd84c@mx.google.com>
Date: Sun, 27 Sep 2009 21:30:13 -0700
Message-ID: <314098690909272130n4f22e636re90a067e47caffa2@mail.gmail.com>
Subject: Re: Integrating Lucene in Hadoop
From: Jason Venner <jason.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485e7ca2001eaa304749bc1b9
X-Virus-Checked: Checked by ClamAV on apache.org

--001485e7ca2001eaa304749bc1b9
Content-Type: text/plain; charset=ISO-8859-1

Katta is a tool for managing distributed search, which happens by default to
use lucene as it' search engine.
Katta is able to read indexes from hdfs, or s3, that katta then deploys onto
the local disk of the katta nodes.

The contrib directory of the hadoop installation contains a tool for
building lucene indexes, and patch solr-1395, provides an integration of
solr with katta, and patch solr-1301 provides a simple way of building solr
indexes in hadoop (as a parallel to the contrib lucene index build tool).



On Fri, Sep 25, 2009 at 1:13 AM, Chandan Tamrakar <
chandan.tamrakar@nepasoft.com> wrote:

> I was doing few R&D on  integrating hadoop and  Lucene . I could create
> lucene indexes into HDFS using  index contribution provided for Hadoop
>
>
>
> Is this a proper way to create Lucene index how much it is different from a
> project KATTA ?
>
>
>
> Please suggest what would be the better approach to create distributed
> lucene indexes
>
>
>
> Thanks in advance
>
>


-- 
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals

--001485e7ca2001eaa304749bc1b9--

From general-return-536-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 28 14:36:04 2009
Return-Path: <general-return-536-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 75288 invoked from network); 28 Sep 2009 14:36:04 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Sep 2009 14:36:04 -0000
Received: (qmail 27859 invoked by uid 500); 28 Sep 2009 14:36:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27761 invoked by uid 500); 28 Sep 2009 14:36:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27751 invoked by uid 99); 28 Sep 2009 14:36:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Sep 2009 14:36:02 +0000
X-ASF-Spam-Status: No, hits=4.0 required=10.0
	tests=HTML_MESSAGE,SPF_PASS,SUBJ_ALL_CAPS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cabiwan@gmail.com designates 209.85.220.222 as permitted sender)
Received: from [209.85.220.222] (HELO mail-fx0-f222.google.com) (209.85.220.222)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Sep 2009 14:35:53 +0000
Received: by fxm22 with SMTP id 22so36807fxm.36
        for <general@hadoop.apache.org>; Mon, 28 Sep 2009 07:35:33 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=dDcpDeH1x4wBc8+o5ia3CeoR4GHC3+OqwhiaVa6YLco=;
        b=tKXsPGOsc+ZCWw0EUbnPdL7krPa+Ho3ADwldiMT6O25F1EvykrmdWBmU9GQ8s22/Ii
         aQg32geTeIDvO3+X3btAxd3c6cTZMYPg3W0UX1yssMReIrMe30wDIygLChdt3xTXjvvD
         jIdXaVDX4zZuj8sNN6EsA7bt4i8UtBBtoRYEY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=OfG40bQG3kKxDyq4Rvxbu8VkQ55BQ6AmWZc2SMMGUrvdZr1V/8eDEIhyIehL4jbM2A
         U8X5/10fsjXrnE4vt+gYdSpmr10XaysSmbwvCDz/vrQBZ1wdTbDOS+8QZB2B3L1QqSag
         4x2uJySRX/rMm/LJQ3ylmZy3RhAJSVDVElJBQ=
MIME-Version: 1.0
Received: by 10.103.122.5 with SMTP id z5mr1292237mum.11.1254148533304; Mon, 
	28 Sep 2009 07:35:33 -0700 (PDT)
Date: Mon, 28 Sep 2009 16:35:33 +0200
Message-ID: <309f76d00909280735m5f23442bvf699256b343f085f@mail.gmail.com>
Subject: ECLIPSE PLUGIN FOR HADOOP 0.20.1
From: Alberto Luengo Cabanillas <cabiwan@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e659faeed2bd260474a435d9
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e659faeed2bd260474a435d9
Content-Type: text/plain; charset=ISO-8859-1

Hi everyone! Does anyone managed to generate the Eclipse plugin JAR file
with the latest Hadoop stable version (0.20.1) under Windows? Could anyone
help me?
Thanks in advance.

-- 
Alberto

--0016e659faeed2bd260474a435d9--

From general-return-537-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 28 17:01:45 2009
Return-Path: <general-return-537-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22230 invoked from network); 28 Sep 2009 17:01:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Sep 2009 17:01:44 -0000
Received: (qmail 16023 invoked by uid 500); 28 Sep 2009 17:01:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15943 invoked by uid 500); 28 Sep 2009 17:01:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15933 invoked by uid 99); 28 Sep 2009 17:01:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Sep 2009 17:01:43 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 28 Sep 2009 17:01:40 +0000
Received: (qmail 21139 invoked by uid 99); 28 Sep 2009 17:01:18 -0000
Received: from localhost.apache.org (HELO [192.168.168.108]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Sep 2009 17:01:18 +0000
Message-ID: <4AC0EBDD.9070207@apache.org>
Date: Mon, 28 Sep 2009 10:01:17 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: HTTP transport?
References: <C6E2B019.127CC%scott@richrelevance.com>
In-Reply-To: <C6E2B019.127CC%scott@richrelevance.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Scott Carey wrote:
> HTTP is very useful and typically performs very well.  It has lots of
> things built-in too.  In addition to what you mention, it  has a
> caching mechanism built-in, range queries, and all sorts of ways to
> tag along state if needed.  To top it off there are a lot of testing
> and debugging tools available for it.  So from that front using it is
> very attractive.

Glad you agree!

> However, In my experience zero-copy is not going to be much of a gain
> performance-wise for this sort of application, and will limit what
> can be done.  As long as a servlet doesn't transcode data and mostly
> copies, it will be very fast - many multiples of gigabit ethernet
> speed per CPU - far more than most disk setups will handle for a
> while.

In MapReduce, datanodes are also running map and reduce tasks, so we'd 
like it if datanodes not only keep up with disks and networks, but also 
use minimal CPU to do so.  Zerocopy on the datanode has been shown to 
help significantly MapReduce benchmarks.  That said, zero copy may or 
may not be significantly better than one-copy.  I intend to benchmark 
that.  But the important thing to measure is not just throughput but 
also idle CPU.

> Additionally, I'm not sure CRC checking should occur on the
> client.  TCP/IP already checksums packets, so network data corruption
> over HTTP is not a primary concern.   The big concern is silent data
> corruption on the disk.

I believe that disks are the largest source of data corruption, but I am 
not confident they are the only source.  HDFS uses end-to-end checksums. 
  As data is written to HDFS it is immediately checksummed on the 
client.  This checksum then lives with the data and is validated on the 
client immediately before the data is returned to the application.  The 
goal is to catch corruption wherever it may occur, on disks, on the 
network, or while buffered in memory.  In addition, the checksum is 
validated after data is transmitted to datanodes but before before 
blocks are stored, so that initial network and memory corruptions are 
caught early and the writing process fails, rather than permitting an 
application to write corrupt data.  Finally, datanodes periodically scan 
for corrupt blocks on disks, replacing them with non-corrupt replicas, 
decreasing the chance that over time all replicas become corrupt.

> Additionally, embedding Tomcat tends to be more tricky than Jetty,
> though that can be overcome.  One might argue that we don't even want
> a servlet container, we just want an HTTP connector.  The Servlet API
> is familiar, but for a high performance transport it might just be
> overhead and restrictive.  Direct access to Tomcat's NIO connector
> might be significantly lighter-weight and more flexible. Tomcat's NIO
> connector implementation works great and I have had great success
> with up to 10K connections with the pure Java connector using
> ordinary byte buffers and about 20 servlet threads.

I hope to start benchmarking bulk data RPC over the next few weeks. 
I'll probably start with a servlet using Jetty, then see if I can 
increase throughput and decrease CPU utilization through the use of 
things like Tomcat's NIO connector, Grizzly, etc.

Doug

From general-return-538-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 28 18:00:00 2009
Return-Path: <general-return-538-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45135 invoked from network); 28 Sep 2009 17:59:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Sep 2009 17:59:59 -0000
Received: (qmail 99467 invoked by uid 500); 28 Sep 2009 17:59:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99423 invoked by uid 500); 28 Sep 2009 17:59:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 99345 invoked by uid 99); 28 Sep 2009 17:59:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Sep 2009 17:59:57 +0000
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.30.56] (HELO QMTA06.emeryville.ca.mail.comcast.net) (76.96.30.56)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Sep 2009 17:59:47 +0000
Received: from OMTA19.emeryville.ca.mail.comcast.net ([76.96.30.76])
	by QMTA06.emeryville.ca.mail.comcast.net with comcast
	id mD8P1c0041eYJf8A6HzU24; Mon, 28 Sep 2009 17:59:28 +0000
Received: from wlan-guest-114.corp.yahoo.com ([209.131.62.144])
	by OMTA19.emeryville.ca.mail.comcast.net with comcast
	id mHzK1c00A36jmhM01HzMX2; Mon, 28 Sep 2009 17:59:25 +0000
Message-Id: <1CB467B9-B60C-4ED0-BA82-7BB571E724C6@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <4AAAC403.80809@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: HTTP transport?
Date: Mon, 28 Sep 2009 10:59:17 -0700
References: <4AAAC403.80809@apache.org>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


On Sep 11, 2009, at 2:41 PM, Doug Cutting wrote:

> I'm considering an HTTP-based transport for Avro as the preferred,  
> high-performance option.

I've got concerns about this. Both tactical and strategic. The  
tactical problem is that I need to get security (both Kerberos and  
token) in to 0.22. I'd really like to get Avro RPC into 0.22. I'd like  
both to be done roughly in 5 months. If you switch off of the current  
RPC code base to a completely new RPC code base, I don't see that  
happening.

> HTTP has lots of advantages.  In particular, it already has
> - lots of authentication, authorization and encryption support;
> - highly optimized servers;
> - monitoring, logging, etc.

It also has a couple of disadvantages:
   - poor integration with kerberos
   - very expensive on the wire encryption (ssl)

> Tomcat and other servlet containers support async NIO, where a  
> thread is not required per connection.

I'm also concerned about the weight of Tomcat. Everything I've read  
about it says that it take a lot more memory and cpu than Jetty. I  
think a solution that requires Tomcat may be problematic...

-- Owen

From general-return-539-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 28 20:14:40 2009
Return-Path: <general-return-539-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95734 invoked from network); 28 Sep 2009 20:14:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Sep 2009 20:14:40 -0000
Received: (qmail 12104 invoked by uid 500); 28 Sep 2009 20:14:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12026 invoked by uid 500); 28 Sep 2009 20:14:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12016 invoked by uid 99); 28 Sep 2009 20:14:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Sep 2009 20:14:39 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Sep 2009 20:14:27 +0000
Received: from oceanfarearth-lm.corp.yahoo.com (oceanfarearth-lm.corp.yahoo.com [10.72.113.156])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n8SKD42o001720
	for <general@hadoop.apache.org>; Mon, 28 Sep 2009 13:13:04 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:mime-version:
	subject:date:references:x-mailer;
	b=yFWMrq/7IUlyqH+eKsPNrVGcYXZTWKD+Hf4YqY7QydmSuyr9UKICLBGE4Wo50Ilr
Message-Id: <E462AEA2-0EFB-460B-9E99-F1E0002AE3C2@yahoo-inc.com>
From: Sanjay Radia <sradia@yahoo-inc.com>
To: <general@hadoop.apache.org>
In-Reply-To: <4AAAC403.80809@apache.org>
Content-Type: multipart/alternative; boundary=Apple-Mail-36-983677215
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: HTTP transport?
Date: Mon, 28 Sep 2009 13:13:04 -0700
References: <4AAAC403.80809@apache.org>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-36-983677215
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


On Sep 11, 2009, at 2:41 PM, Doug Cutting wrote:

> I'm considering an HTTP-based transport for Avro as the preferred,
> high-performance option.
>
> HTTP has lots of advantages.  In particular, it already has
>   - lots of authentication, authorization and encryption support;
>   - highly optimized servers;
>   - monitoring, logging, etc.
>

Q. Is this to replace the client-DN  data-transfer protocol or for ALL  
Hadoop rpc?

Q. Was authentication one of your main motivation?
The current plans for authentication is centered around kerberos.
HTTP does not fit in too well in that picture.


sanjay
>
> Tomcat and other servlet containers support async NIO, where a  
> thread is
> not required per connection.  A servlet can process bulk data with a
> single copy to and from the socket (bypassing stream buffers).  Calls
> can be multiplexed over a single HTTP connection using Comet events.
>
> http://tomcat.apache.org/tomcat-6.0-doc/aio.html
>
> Zero copy is not an option for servlets that generate arbitrary data,
> but one can specify a file/start/length tuple and Tomcat will use
> sendfile to write the response.  That means that while HDFS datanode
> file reads could not be done via RPC, they could be done via HTTP with
> zero-copy.  If authentication and authorization are already done in  
> the
> HTTP server, this may not be a big loss.  The HDFS client might make  
> two
> HTTP requests, one to read a files data, and another to read its
> checksums.  The server would then stream the entire block to the  
> client
> using sendfile, using TCP flow control as today.
>
> Thoughts?
>
> Doug
>


--Apple-Mail-36-983677215--

From general-return-540-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 28 22:31:14 2009
Return-Path: <general-return-540-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35456 invoked from network); 28 Sep 2009 22:31:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Sep 2009 22:31:13 -0000
Received: (qmail 69525 invoked by uid 500); 28 Sep 2009 22:31:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69453 invoked by uid 500); 28 Sep 2009 22:31:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69443 invoked by uid 99); 28 Sep 2009 22:31:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Sep 2009 22:31:13 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of anubhav.hanjura@convergys.com designates 167.1.158.29 as permitted sender)
Received: from [167.1.158.29] (HELO cvgmx2.convergys.com) (167.1.158.29)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Sep 2009 22:31:02 +0000
Received: from CDC5NW41.na.convergys.com (CDC5NW41.na.convergys.com [155.90.28.181])
	by cvgmx2.convergys.com (Postfix) with ESMTP id E73054D049
	for <general@hadoop.apache.org>; Mon, 28 Sep 2009 22:30:40 +0000 (UTC)
Subject: Anubhav Hanjura is out of the office.
From: anubhav.hanjura@convergys.com
To: general@hadoop.apache.org
Message-ID: <OF3277769F.590346ED-ON6525763F.007BA68F-6525763F.007BA68F@nd.convergys.com>
Date: Tue, 29 Sep 2009 04:00:35 +0530
X-MIMETrack: Serialize by Router on CVGSMTP1/SRVR/CVG(Release 6.5.5FP3|March 22, 2007) at
 09/28/2009 06:30:40 PM
MIME-Version: 1.0
Content-type: multipart/alternative; 
	Boundary="0__=EABBFCACDFE8201F8f9e8a93df938690918cEABBFCACDFE8201F"
Content-Disposition: inline
X-Virus-Checked: Checked by ClamAV on apache.org

--0__=EABBFCACDFE8201F8f9e8a93df938690918cEABBFCACDFE8201F
Content-type: text/plain; charset=US-ASCII
Content-transfer-encoding: quoted-printable






I will be out of the office starting  09/29/2009 and will not return un=
til
10/05/2009.


--
"NOTICE:  The information contained in this electronic mail transmissio=
n is
intended by Convergys Corporation for the use of the named individual o=
r
entity to which it is directed and may contain information that is
privileged or otherwise confidential.  If you have received this electr=
onic
mail transmission in error, please delete it from your system without
copying or forwarding it, and notify the sender of the error by reply e=
mail
or by telephone (collect), so that the sender's address records can be
corrected."=

--0__=EABBFCACDFE8201F8f9e8a93df938690918cEABBFCACDFE8201F--


From general-return-541-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 28 22:42:42 2009
Return-Path: <general-return-541-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37364 invoked from network); 28 Sep 2009 22:42:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Sep 2009 22:42:41 -0000
Received: (qmail 75861 invoked by uid 500); 28 Sep 2009 22:42:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75788 invoked by uid 500); 28 Sep 2009 22:42:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75778 invoked by uid 99); 28 Sep 2009 22:42:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Sep 2009 22:42:40 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 28 Sep 2009 22:42:38 +0000
Received: (qmail 37330 invoked by uid 99); 28 Sep 2009 22:42:16 -0000
Received: from localhost.apache.org (HELO [192.168.168.108]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Sep 2009 22:42:16 +0000
Message-ID: <4AC13BC7.6020902@apache.org>
Date: Mon, 28 Sep 2009 15:42:15 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: HTTP transport?
References: <4AAAC403.80809@apache.org> <1CB467B9-B60C-4ED0-BA82-7BB571E724C6@apache.org>
In-Reply-To: <1CB467B9-B60C-4ED0-BA82-7BB571E724C6@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Owen O'Malley wrote:
> I've got concerns about this. Both tactical and strategic. The tactical 
> problem is that I need to get security (both Kerberos and token) in to 
> 0.22. I'd really like to get Avro RPC into 0.22. I'd like both to be 
> done roughly in 5 months. If you switch off of the current RPC code base 
> to a completely new RPC code base, I don't see that happening.

What transport do you expect to use with Avro?  If wire-compatibility is 
a goal, and that includes access from languages besides Java, then we 
must use a transport that's well-specified and Java-independent.  HTTP 
is both of these.  The existing Hadoop RPC protocol is not.

We could adapting Hadoop's existing RPC transport to be well-specified 
and language independent.  This is perhaps not a huge task, but it feels 
to me a bit like re-inventing much of what's already in HTTP clients and 
servers these days: connection-pooling, async servers, etc.  Plus we 
take on the onus of fully specifying the transport, so that it may be 
implemented in other languages, and we need to provide some alternate 
implementations to demonstrate this.

Do you feel our existing RPC framework's transport is actually more 
scalable and reliable than, say, Jetty?  Do you think it would be 
substantially harder to add, e.g., token-based security to Jetty than to 
a homegrown server?

> [ HTTP ] also has a couple of disadvantages:
>   - poor integration with kerberos

Do you think it would be substantially harder to integrate Kerberos with 
Jetty than with a homegrown protocol and server?

>   - very expensive on the wire encryption (ssl)

If we don't use HTTP, will we be providing on-wire encryption?  If not, 
this is moot.

Finally, need to have secure HTTP-based access anyway, right?  If we use 
HTTP as our RPC transport mightn't we reuse most of that effort?

Doug

From general-return-542-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 29 05:12:38 2009
Return-Path: <general-return-542-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30019 invoked from network); 29 Sep 2009 05:12:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Sep 2009 05:12:37 -0000
Received: (qmail 30185 invoked by uid 500); 29 Sep 2009 05:12:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30107 invoked by uid 500); 29 Sep 2009 05:12:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30097 invoked by uid 99); 29 Sep 2009 05:12:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Sep 2009 05:12:36 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 209.85.221.188 as permitted sender)
Received: from [209.85.221.188] (HELO mail-qy0-f188.google.com) (209.85.221.188)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Sep 2009 05:12:27 +0000
Received: by qyk26 with SMTP id 26so4228557qyk.5
        for <general@hadoop.apache.org>; Mon, 28 Sep 2009 22:12:06 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:message-id:subject:from:to
         :content-type;
        bh=DNjcdieNS1XjTe8gdSru14wqFZjGrmMj0pLJrdsY9LE=;
        b=BS4F3Aw8Nu1m4qhLGqgnV1iLxEP65+kVKzWBj2NWZ0hzFDgSY1DnHhC4HbiszepBAr
         j2Ip17RGEmawLlosm4A6I9UxhvvUW62jBgM3IaQg9s0aeyakUwqwlcGTZVidHr5TXycv
         7+FyMZ84QtkAXui62cPfllIeCvxdpRxKGxB0E=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=GBbeS4NOhsHzWgMOYZeX7ug6otDO5u0Y/G5kkW38zCrmQzlOQkn2YOnJsZ9VCXyIzq
         Up2I3Ss28qBNATadbxtoa6W+3GKK48IAOrOdy2ZpNqFar9IRLh52COppEOZ9jz1DWPwi
         II0OCqRu30xuEBpp00pgzxF5nnnboVSZmvS/U=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
Received: by 10.229.45.19 with SMTP id c19mr503938qcf.0.1254201126613; Mon, 28 
	Sep 2009 22:12:06 -0700 (PDT)
In-Reply-To: <2B8EEA26-46D4-4D77-997A-A3394790613F@apache.org>
References: <2B8EEA26-46D4-4D77-997A-A3394790613F@apache.org>
Date: Mon, 28 Sep 2009 22:12:06 -0700
X-Google-Sender-Auth: 85cc9fb4d4a197ad
Message-ID: <7c962aed0909282212p77dbc9cex80923e08cb7f837@mail.gmail.com>
Subject: Re: [VOTE] Create hbase-issues email list
From: stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016363ba12ea0f40d0474b074c5
X-Virus-Checked: Checked by ClamAV on apache.org

--0016363ba12ea0f40d0474b074c5
Content-Type: text/plain; charset=ISO-8859-1

Thanks Owen for running the vote.  It looks like infrastructure created the
list (it seems to work -- I can subscribe).

I do not seem to be able to change hbase jira to send email notifications
via the new mailing list  According to the JIRA manual there is supposed to
be a "Edit Configuration" for the "Mail Configuration" entry in the project
detail list but I don't have such an option.  Is this something you have
privileges for?

Thanks boss,
St.Ack

On Fri, Aug 21, 2009 at 11:32 PM, Owen O'Malley <omalley@apache.org> wrote:

> I think we should create an hbase-issues lists to track that detailed jira
> traffic and reduce the traffic on hbase-dev. It will replicate the structure
> that we have in common, hdfs, and mapreduce. Clearly, I'm +1.
>
> -- Owen
>

--0016363ba12ea0f40d0474b074c5--

From general-return-543-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 29 18:52:58 2009
Return-Path: <general-return-543-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 8145 invoked from network); 29 Sep 2009 18:52:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Sep 2009 18:52:58 -0000
Received: (qmail 96537 invoked by uid 500); 29 Sep 2009 18:52:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96442 invoked by uid 500); 29 Sep 2009 18:52:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96415 invoked by uid 99); 29 Sep 2009 18:52:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Sep 2009 18:52:56 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Sep 2009 18:52:44 +0000
Received: from oceanfarearth-lm.corp.yahoo.com (oceanfarearth-lm.corp.yahoo.com [10.72.113.156])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n8TIq91Z011458
	for <general@hadoop.apache.org>; Tue, 29 Sep 2009 11:52:09 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:mime-version:
	subject:date:references:x-mailer;
	b=GHmsbPVLkQWwomtSK1yYoF7pFzYjLazX82GrdfOZ8Nq2A25JijOqlPUSJJ6TbrjJ
Message-Id: <B38E3438-CC8F-48B8-B415-549165996ED5@yahoo-inc.com>
From: Sanjay Radia <sradia@yahoo-inc.com>
To: <general@hadoop.apache.org>
In-Reply-To: <4AC13BC7.6020902@apache.org>
Content-Type: multipart/alternative; boundary=Apple-Mail-42-1065222992
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: HTTP transport?
Date: Tue, 29 Sep 2009 11:52:09 -0700
References: <4AAAC403.80809@apache.org> <1CB467B9-B60C-4ED0-BA82-7BB571E724C6@apache.org> <4AC13BC7.6020902@apache.org>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-42-1065222992
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


On Sep 28, 2009, at 3:42 PM, Doug Cutting wrote:

> Owen O'Malley wrote:
> > I've got concerns about this. Both tactical and strategic. The  
> tactical
> > problem is that I need to get security (both Kerberos and token)  
> in to
> > 0.22. I'd really like to get Avro RPC into 0.22. I'd like both to be
> > done roughly in 5 months. If you switch off of the current RPC  
> code base
> > to a completely new RPC code base, I don't see that happening.
>
> What transport do you expect to use with Avro?  If wire- 
> compatibility is
> a goal, and that includes access from languages besides Java, then we
> must use a transport that's well-specified and Java-independent.  HTTP
> is both of these.  The existing Hadoop RPC protocol is not.
>
> We could adapting Hadoop's existing RPC transport to be well-specified
> and language independent.  This is perhaps not a huge task, but it  
> feels
> to me a bit like re-inventing much of what's already in HTTP clients  
> and
> servers these days: connection-pooling, async servers, etc.
>


Wrt  connection pooling/async servers: Can't we use the same libraries  
that Jetty and Tomcat use?
  Grizzly?

> grate Kerberos with
> Jetty than with a homegrown protocol and server?
>
>
> >   - very expensive on the wire encryption (ssl)
>
> If we don't use HTTP, will we be providing on-wire encryption?  If  
> not,
> this is moot.
>

Yes we are expecting to use encryption down the road.
>
> Finally, need to have secure HTTP-based access anyway, right?  If we  
> use
> HTTP as our RPC transport mightn't we reuse most of that effort?
>
> Doug
>


--Apple-Mail-42-1065222992--

From general-return-544-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 29 19:43:51 2009
Return-Path: <general-return-544-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33690 invoked from network); 29 Sep 2009 19:43:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Sep 2009 19:43:50 -0000
Received: (qmail 73166 invoked by uid 500); 29 Sep 2009 19:43:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73103 invoked by uid 500); 29 Sep 2009 19:43:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73093 invoked by uid 99); 29 Sep 2009 19:43:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Sep 2009 19:43:49 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 29 Sep 2009 19:43:47 +0000
Received: (qmail 33604 invoked by uid 99); 29 Sep 2009 19:43:25 -0000
Received: from localhost.apache.org (HELO [192.168.168.108]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Sep 2009 19:43:25 +0000
Message-ID: <4AC2635C.20902@apache.org>
Date: Tue, 29 Sep 2009 12:43:24 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: HTTP transport?
References: <4AAAC403.80809@apache.org> <1CB467B9-B60C-4ED0-BA82-7BB571E724C6@apache.org> <4AC13BC7.6020902@apache.org> <B38E3438-CC8F-48B8-B415-549165996ED5@yahoo-inc.com>
In-Reply-To: <B38E3438-CC8F-48B8-B415-549165996ED5@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Sanjay Radia wrote:
> Wrt  connection pooling/async servers: Can't we use the same libraries 
> that Jetty and Tomcat use?
>  Grizzly?

Grizzly also supports HTTP.  Choosing Grizzly is independent of choosing 
HTTP as a wire transport or choosing a server.

The question I'm asking now is about the wire format, whether we wish to 
precede each RPC request with something like "GET 
/avro/org.apache.hadoop.hdfs.NameNode HTTP/1.1\n" and each response with 
"HTTP/1.1 200 OK\n", plus a couple of other headers in each case (e.g., 
Content-Type and Content-Length).  I think there are great benefits to 
using a single, standard protocol on the wire.  Which server and client 
implementations we use will be determined by performance, features, etc. 
  But using a standard wire format will greatly simplify things as we 
attempt to support multiple languages.  Since we want to provide browser 
access, we're compelled to support HTTP.  So the question is, are there 
compelling reasons why HTTP should not be used for other, non-browser, 
access?

> Yes we are expecting to use encryption down the road.

Do we expect to use something different from TLS?  With its 'resume' 
feature, is TLS performance unacceptable?  Would we implement some other 
encryption protocol, or use a non-standards-based encryption protocol?

Doug

From general-return-545-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 29 20:39:29 2009
Return-Path: <general-return-545-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46455 invoked from network); 29 Sep 2009 20:39:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Sep 2009 20:39:28 -0000
Received: (qmail 36373 invoked by uid 500); 29 Sep 2009 20:39:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36309 invoked by uid 500); 29 Sep 2009 20:39:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36299 invoked by uid 99); 29 Sep 2009 20:39:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Sep 2009 20:39:27 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 74.125.92.24 as permitted sender)
Received: from [74.125.92.24] (HELO qw-out-2122.google.com) (74.125.92.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Sep 2009 20:39:19 +0000
Received: by qw-out-2122.google.com with SMTP id 5so1859937qwd.35
        for <general@hadoop.apache.org>; Tue, 29 Sep 2009 13:38:59 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:message-id:subject:from:to
         :content-type;
        bh=GbpIaegajObWsSreHVIu5f3NdJq2hziMUixOYSBV0VU=;
        b=hRtvXYHN6bPn6fYEbfCqYDOe9Q8J5t/aeQB1hpidPU1J105bqsfLFN0FcfFSvoDlBX
         /srBnc7B5mHYwdJx1xBD9Mk/OQkNTLlGBHF/dpGIS3bab3X4mhN7bosvR9MKxtE2ojGK
         hsJOCLOq8SFkkkVvM7sNR9b5XnANTZNJjamvg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=m020aSYEewQLmi9njIJ4LQ9c4r12UHGOZnhxOaRDo+ZrrqmnqE9GLh2aw76mVcrboU
         y41zldvyhYBILeb2LfPMIBVrEuQ9uaMsObHRUSJZsMiM1in5in2LJJiaTTm2J8fomrTj
         U+A9qnfjg70eWnFLrI6lUsw1v2LVD2oHFdEbI=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
Received: by 10.229.69.83 with SMTP id y19mr2335603qci.50.1254256738954; Tue, 
	29 Sep 2009 13:38:58 -0700 (PDT)
In-Reply-To: <4AC2635C.20902@apache.org>
References: <4AAAC403.80809@apache.org>
	 <1CB467B9-B60C-4ED0-BA82-7BB571E724C6@apache.org>
	 <4AC13BC7.6020902@apache.org>
	 <B38E3438-CC8F-48B8-B415-549165996ED5@yahoo-inc.com>
	 <4AC2635C.20902@apache.org>
Date: Tue, 29 Sep 2009 13:38:57 -0700
X-Google-Sender-Auth: a9124d647c5ac91c
Message-ID: <7c962aed0909291338o7f09601ci102186e23fd4d464@mail.gmail.com>
Subject: Re: HTTP transport?
From: stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00032557f4ba61ecc90474bd677d
X-Virus-Checked: Checked by ClamAV on apache.org

--00032557f4ba61ecc90474bd677d
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Sep 29, 2009 at 12:43 PM, Doug Cutting <cutting@apache.org> wrote:

>
> The question I'm asking now is about the wire format, whether we wish to
> precede each RPC request with something like "GET
> /avro/org.apache.hadoop.hdfs.NameNode HTTP/1.1\n" and each response with
> "HTTP/1.1 200 OK\n", plus a couple of other headers in each case (e.g.,
> Content-Type and Content-Length).  I think there are great benefits to using
> a single, standard protocol on the wire.  Which server and client
> implementations we use will be determined by performance, features, etc.
>  But using a standard wire format will greatly simplify things as we attempt
> to support multiple languages.  Since we want to provide browser access,
> we're compelled to support HTTP.  So the question is, are there compelling
> reasons why HTTP should not be used for other, non-browser, access?



I like the idea of using a proven transport.

The HTTP request and response header verbiage seems profligate if whats
being passed is small.

What do you think the path on the first line look like? Will it be a method
name or will it be customizable? (In hbase, it might be nice to have path be
/tablename/row/family/qualifier etc).

St.Ack

--00032557f4ba61ecc90474bd677d--

From general-return-546-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 29 21:09:11 2009
Return-Path: <general-return-546-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55389 invoked from network); 29 Sep 2009 21:09:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Sep 2009 21:09:11 -0000
Received: (qmail 72302 invoked by uid 500); 29 Sep 2009 21:09:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72239 invoked by uid 500); 29 Sep 2009 21:09:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72229 invoked by uid 99); 29 Sep 2009 21:09:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Sep 2009 21:09:10 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 29 Sep 2009 21:09:07 +0000
Received: (qmail 55301 invoked by uid 99); 29 Sep 2009 21:08:45 -0000
Received: from localhost.apache.org (HELO [192.168.168.108]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Sep 2009 21:08:45 +0000
Message-ID: <4AC2775C.2000302@apache.org>
Date: Tue, 29 Sep 2009 14:08:44 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: HTTP transport?
References: <4AAAC403.80809@apache.org>	 <1CB467B9-B60C-4ED0-BA82-7BB571E724C6@apache.org>	 <4AC13BC7.6020902@apache.org>	 <B38E3438-CC8F-48B8-B415-549165996ED5@yahoo-inc.com>	 <4AC2635C.20902@apache.org> <7c962aed0909291338o7f09601ci102186e23fd4d464@mail.gmail.com>
In-Reply-To: <7c962aed0909291338o7f09601ci102186e23fd4d464@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

stack wrote:
> What do you think the path on the first line look like? Will it be a method
> name or will it be customizable?

Avro RPC currently includes the message name in the payload, so, unless 
that changes, for Avro RPC, we'd probably use a different URL per 
protocol.  As a convention we might use the namespace-qualified protocol 
name as the URL path.

Alternately, we could try to make Avro's RPC more HTTP-friendly, and 
pull stuff out of Avro's payload into HTTP headers.  The downside of 
that would be that, if we still wish to support non-HTTP transports, 
we'd end up with duplicated logic.

If we fully embraced HTTP as Avro's primary RPC transport then it might 
make sense to move the message name to the URL and to use the HTTP 
return code to determine whether the response is an error or not. 
Avro's RPC payload also currently includes request and response 
metadata, which are functionally redundant with HTTP headers.

> (In hbase, it might be nice to have path be /tablename/row/family/qualifier etc).

It sounds like you'd perhaps like to be able to put RPC request 
parameters into the URL?  I don't see that being done automatically in a 
general way for arbitrary parameter types without the URLs getting 
really ugly and adding a lot of complexity.  For this it might be better 
to write a servlet filter that constructs the appropriate Avro-format 
request and forwards it to the RPC url.

Doug

From general-return-547-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 29 21:58:03 2009
Return-Path: <general-return-547-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77021 invoked from network); 29 Sep 2009 21:58:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Sep 2009 21:58:03 -0000
Received: (qmail 51895 invoked by uid 500); 29 Sep 2009 21:58:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51836 invoked by uid 500); 29 Sep 2009 21:58:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51826 invoked by uid 99); 29 Sep 2009 21:58:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Sep 2009 21:58:02 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 209.85.221.188 as permitted sender)
Received: from [209.85.221.188] (HELO mail-qy0-f188.google.com) (209.85.221.188)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Sep 2009 21:57:54 +0000
Received: by qyk26 with SMTP id 26so4816693qyk.5
        for <general@hadoop.apache.org>; Tue, 29 Sep 2009 14:57:33 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:message-id:subject:from:to
         :content-type;
        bh=Uc9uXoakxbsrBA3Fa+qz+ZqZuvDXqoKBlGQlmmOnhOM=;
        b=wDMEzE+ZC5j1Os9J+Gy7t4KWwY5jc3BwzIkQFkLebEahqpc3RrGwGZHkKbQlmBRrQ4
         kCzgydJUnEOKFvom6S/PysLDJ3orRm/F3YiQ7ZgU306oxXnLZpDUssdMULZbRa2JzCN/
         NVLT3zJ1RjScNhnLqA7kA1RTtJTsrOnkLNBI0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=OtNGKYEAOL+bDhcnbiXYoMefv9QNT2wXsz5Q2R13p5qV3MD7eQ0DS7qj+15ayLzi5j
         zLrDEGBAsZhgy60z31xYfQhQvVBAovTEZWcc9AcekGv5XI4pTcIzHGPapZmQ0cuqHYuE
         bvEhJqWS6jYVrGwoXBJeE2gFe4kDQz6qRVW4g=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
Received: by 10.229.43.68 with SMTP id v4mr2423141qce.95.1254261453147; Tue, 
	29 Sep 2009 14:57:33 -0700 (PDT)
In-Reply-To: <4AC2775C.2000302@apache.org>
References: <4AAAC403.80809@apache.org>
	 <1CB467B9-B60C-4ED0-BA82-7BB571E724C6@apache.org>
	 <4AC13BC7.6020902@apache.org>
	 <B38E3438-CC8F-48B8-B415-549165996ED5@yahoo-inc.com>
	 <4AC2635C.20902@apache.org>
	 <7c962aed0909291338o7f09601ci102186e23fd4d464@mail.gmail.com>
	 <4AC2775C.2000302@apache.org>
Date: Tue, 29 Sep 2009 14:57:33 -0700
X-Google-Sender-Auth: ad395697f8c8bd10
Message-ID: <7c962aed0909291457o48baee08hfa3b6f4eb16213de@mail.gmail.com>
Subject: Re: HTTP transport?
From: stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f6c3605ecfec0474be8070
X-Virus-Checked: Checked by ClamAV on apache.org

--001485f6c3605ecfec0474be8070
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Sep 29, 2009 at 2:08 PM, Doug Cutting <cutting@apache.org> wrote:

>
> Alternately, we could try to make Avro's RPC more HTTP-friendly, and pull
> stuff out of Avro's payload into HTTP headers.  The downside of that would
> be that, if we still wish to support non-HTTP transports, we'd end up with
> duplicated logic.
>


There would be loads of upside I'd imagine if there was a natural mapping of
avro payload specifiers and metadata up into http headers in terms of
visibility


So, are we're talking about doing something like following for a
request/response:

 GET /avro/org.apache.hadoop.hbase.RegionServer HTTP/1.1
 Host: www.example.com


 HTTP/1.1 200 OK
 Date: Mon, 23 May 2005 22:38:34 GMT
 Server: Apache/1.3.3.7 (Unix)  (Red-Hat/Linux)
 Last-Modified: Wed, 08 Jan 2003 23:11:55 GMT
 Etag: "3f80f-1b6-3e1cb03b"
 Accept-Ranges: bytes
 Content-Length: 438
 Connection: close
 Content-Type: X-avro/binary


... or some variation on above on each and every RPC?

St.Ack

--001485f6c3605ecfec0474be8070--

From general-return-548-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 29 22:02:34 2009
Return-Path: <general-return-548-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78385 invoked from network); 29 Sep 2009 22:02:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Sep 2009 22:02:34 -0000
Received: (qmail 57804 invoked by uid 500); 29 Sep 2009 22:02:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57748 invoked by uid 500); 29 Sep 2009 22:02:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57738 invoked by uid 99); 29 Sep 2009 22:02:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Sep 2009 22:02:33 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [67.195.15.168] (HELO web111407.mail.gq1.yahoo.com) (67.195.15.168)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 29 Sep 2009 22:02:23 +0000
Received: (qmail 74931 invoked by uid 60001); 29 Sep 2009 22:02:01 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1254261720; bh=FEGSooHVNSpN7SPwfYMSgGb+ARXO8RgWvmtWqamudP4=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=gkdsJLisxdUKbvfJthtob23VXYkTYL6h1JzY6eHBFS65DISiz+mUsN/IAXY3V4Mi2wyB33bSAYM3CIWzo8G1gaC7HZqqO3w81H1FPMtfb4MqMRm18EMP7D/jBxWB96gMrCbXBtf+kig79SnbGGxJ9CgHcz/2ETylrcsWHD6jCaw=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=wzNQQ+eZLMFQAJ2zie/Xq/6Z7rKiRzuG3hcir/qVJJCt1l0vd49TnvLNmLwe8HqTtSIqXUb+X8wTuVrh/TKpvVlgUyCafXTSQA19p7zJchw/Pp7S5FHb+gqCaWEqsbH1jDzd19n/QG01kNBiCju3t8+wG/lMtnCZgsiQlreh5Tg=;
Message-ID: <931352.74710.qm@web111407.mail.gq1.yahoo.com>
X-YMail-OSG: nUEvM3IVM1mPuTfSRuq7QEXk8cDLJIjLL4lGEElwQuxC7b3QE3h.QuLysvg5g.ts3.DQAPGVYBdworgF29lcKNt1LUFfK.1BT7LflpEgUQjtUqIs56AQuVCVlLcK8SQPNpJ38iyQaLTNyhhG07mqanQFwU7NqEGKvjz_SJqRY50h6jwRiiutXAAjDqVI2qG.BIX.RNnOX0e5kTbF9AbgnkOXCvZjkIvJveg9x_JwAokVzRfzRhl.tWIeZQViWi0MfL2uEREQG3avcloAGkjx9rI6Asvie7Zo5BYsQ.cRM2orHUCali41abDL7JpOZQv80PF8
Received: from [216.239.45.4] by web111407.mail.gq1.yahoo.com via HTTP; Tue, 29 Sep 2009 15:02:00 PDT
X-Mailer: YahooMailClassic/7.0.14 YahooMailWebService/0.7.347.3
Date: Tue, 29 Sep 2009 15:02:00 -0700 (PDT)
From: Something Something <luckyguy2050@yahoo.com>
Subject: ArrayList as a column value?
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-794499716-1254261720=:74710"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-794499716-1254261720=:74710
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Hello,
=0A
=0AHow do I add values from an ArrayList to a column in a HTable?
=0A
=0ALet's say I have a "Product" table with columns:=A0 name, desc.=A0 Now I=
=0Awant to add to each row a list of categories that a product belongs=0Ato=
.=A0 I tried the following:
=0A
=0A=A0=A0=A0=A0=A0=A0=A0 for (String category : categories) {
=0A=A0=A0=A0=A0=A0=A0=A0=A0=A0 put.add(Bytes.toBytes("info"), Bytes.toBytes=
("categories"), Bytes.toBytes(category));
=0A=A0=A0=A0=A0=A0=A0=A0 }
=0A
=0AThis doesn't work as expected.=A0 It only keeps one (last) category.=A0 =
The=0Arest get overwritten.=A0 I tried using 'version', but that didn't wor=
k=0Aeither.=A0 There's a put(KeyValue) method.=A0 Is that the one I am supp=
osed=0Ato use?
=0A
=0AAny help in this regard will be greatly appreciated.=A0 Thanks.
=0A
=0A=0A=0A      
--0-794499716-1254261720=:74710--

From general-return-549-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 29 22:12:53 2009
Return-Path: <general-return-549-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 80774 invoked from network); 29 Sep 2009 22:12:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Sep 2009 22:12:53 -0000
Received: (qmail 72888 invoked by uid 500); 29 Sep 2009 22:12:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72812 invoked by uid 500); 29 Sep 2009 22:12:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72799 invoked by uid 99); 29 Sep 2009 22:12:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Sep 2009 22:12:52 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Sep 2009 22:12:41 +0000
Received: from [10.72.114.189] (specialwarm-lm.corp.yahoo.com [10.72.114.189])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n8TMBRhE074133
	for <general@hadoop.apache.org>; Tue, 29 Sep 2009 15:11:27 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=1kWttednZqneThxfyy3tbRA/LcdFPoXhy5e/Nr/GoR9x5mwRKsOmokCsBUHoGrK1
Message-ID: <4AC2860F.1030801@yahoo-inc.com>
Date: Tue, 29 Sep 2009 15:11:27 -0700
From: Raghu Angadi <rangadi@yahoo-inc.com>
User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: HTTP transport?
References: <4AAAC403.80809@apache.org>	 <1CB467B9-B60C-4ED0-BA82-7BB571E724C6@apache.org>	 <4AC13BC7.6020902@apache.org>	 <B38E3438-CC8F-48B8-B415-549165996ED5@yahoo-inc.com>	 <4AC2635C.20902@apache.org> <7c962aed0909291338o7f09601ci102186e23fd4d464@mail.gmail.com> <4AC2775C.2000302@apache.org>
In-Reply-To: <4AC2775C.2000302@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Doug Cutting wrote:
> stack wrote:
>> What do you think the path on the first line look like? Will it be a 
>> method
>> name or will it be customizable?
> 
> Avro RPC currently includes the message name in the payload, so, unless 
> that changes, for Avro RPC, we'd probably use a different URL per 
> protocol.  As a convention we might use the namespace-qualified protocol 
> name as the URL path.
> 
> Alternately, we could try to make Avro's RPC more HTTP-friendly, and 
> pull stuff out of Avro's payload into HTTP headers.  The downside of 
> that would be that, if we still wish to support non-HTTP transports, 
> we'd end up with duplicated logic.

Keeping Avro payload independent of transport seems pretty useful, at 
least for now. As understand Avro payload is Avro 'proper' (i.e. it 
issupported in all the languages supported by Avro... and other goodies).

I just noticed AVRO-129 and it seems like a great example of using HTTP 
transport.

Does this mean current Avro RPC transport (an improved version of Hadoop 
RPC) can still exist as long as it supported by developers?

Where does security lie : Avro or Transport layer?
If it is part of Avro : transport layer does not matter for security.
If it is part of transport : How does an app get hold of required 
information (e.g. user identity).
May be 'transceiver' can have an interface that can transfer security 
information between transport layer and Avro.

Raghu.

> If we fully embraced HTTP as Avro's primary RPC transport then it might 
> make sense to move the message name to the URL and to use the HTTP 
> return code to determine whether the response is an error or not. Avro's 
> RPC payload also currently includes request and response metadata, which 
> are functionally redundant with HTTP headers.
> 
>> (In hbase, it might be nice to have path be 
>> /tablename/row/family/qualifier etc).
> 
> It sounds like you'd perhaps like to be able to put RPC request 
> parameters into the URL?  I don't see that being done automatically in a 
> general way for arbitrary parameter types without the URLs getting 
> really ugly and adding a lot of complexity.  For this it might be better 
> to write a servlet filter that constructs the appropriate Avro-format 
> request and forwards it to the RPC url.
> 
> Doug


From general-return-550-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 29 23:17:27 2009
Return-Path: <general-return-550-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98698 invoked from network); 29 Sep 2009 23:17:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Sep 2009 23:17:27 -0000
Received: (qmail 54666 invoked by uid 500); 29 Sep 2009 23:17:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54579 invoked by uid 500); 29 Sep 2009 23:17:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54569 invoked by uid 99); 29 Sep 2009 23:17:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Sep 2009 23:17:26 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 29 Sep 2009 23:17:24 +0000
Received: (qmail 98620 invoked by uid 99); 29 Sep 2009 23:17:04 -0000
Received: from localhost.apache.org (HELO [192.168.168.108]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Sep 2009 23:17:04 +0000
Message-ID: <4AC2956F.6020303@apache.org>
Date: Tue, 29 Sep 2009 16:17:03 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: HTTP transport?
References: <4AAAC403.80809@apache.org>	 <1CB467B9-B60C-4ED0-BA82-7BB571E724C6@apache.org>	 <4AC13BC7.6020902@apache.org>	 <B38E3438-CC8F-48B8-B415-549165996ED5@yahoo-inc.com>	 <4AC2635C.20902@apache.org>	 <7c962aed0909291338o7f09601ci102186e23fd4d464@mail.gmail.com>	 <4AC2775C.2000302@apache.org> <7c962aed0909291457o48baee08hfa3b6f4eb16213de@mail.gmail.com>
In-Reply-To: <7c962aed0909291457o48baee08hfa3b6f4eb16213de@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

stack wrote:
> So, are we're talking about doing something like following for a
> request/response:
> 
>  GET /avro/org.apache.hadoop.hbase.RegionServer HTTP/1.1
>  Host: www.example.com
> 
> 
>  HTTP/1.1 200 OK
>  Date: Mon, 23 May 2005 22:38:34 GMT
>  Server: Apache/1.3.3.7 (Unix)  (Red-Hat/Linux)
>  Last-Modified: Wed, 08 Jan 2003 23:11:55 GMT
>  Etag: "3f80f-1b6-3e1cb03b"
>  Accept-Ranges: bytes
>  Content-Length: 438
>  Connection: close
>  Content-Type: X-avro/binary
> 
> 
> ... or some variation on above on each and every RPC?

More or less.  Except we can probably arrange to omit most of those 
response headers except Content-Length.  Are any others strictly required?

I today implemented a simple HTTP-based transport for Avro:

   https://issues.apache.org/jira/browse/AVRO-129

In some simple benchmarks I am able to make over 5000 sequential 
RPCs/second, each with ~100 bytes of response payload.  Increasing 
response payloads to 100kB slows this to around 2500 RPCs/second, giving 
throughput of 250MB/second, or 2.5Gbit/s.  This is with both client and 
server running on my laptop.  The client is java.net.URLConnection and 
the server is Jetty with its default configuration.

Doug

From general-return-551-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 29 23:35:40 2009
Return-Path: <general-return-551-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5679 invoked from network); 29 Sep 2009 23:35:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Sep 2009 23:35:40 -0000
Received: (qmail 70741 invoked by uid 500); 29 Sep 2009 23:35:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70662 invoked by uid 500); 29 Sep 2009 23:35:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70652 invoked by uid 99); 29 Sep 2009 23:35:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Sep 2009 23:35:39 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 29 Sep 2009 23:35:36 +0000
Received: (qmail 5621 invoked by uid 99); 29 Sep 2009 23:35:15 -0000
Received: from localhost.apache.org (HELO [192.168.168.108]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Sep 2009 23:35:15 +0000
Message-ID: <4AC299B2.8050400@apache.org>
Date: Tue, 29 Sep 2009 16:35:14 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: HTTP transport?
References: <4AAAC403.80809@apache.org>	 <1CB467B9-B60C-4ED0-BA82-7BB571E724C6@apache.org>	 <4AC13BC7.6020902@apache.org>	 <B38E3438-CC8F-48B8-B415-549165996ED5@yahoo-inc.com>	 <4AC2635C.20902@apache.org> <7c962aed0909291338o7f09601ci102186e23fd4d464@mail.gmail.com> <4AC2775C.2000302@apache.org> <4AC2860F.1030801@yahoo-inc.com>
In-Reply-To: <4AC2860F.1030801@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Raghu Angadi wrote:
> Does this mean current Avro RPC transport (an improved version of Hadoop 
> RPC) can still exist as long as it supported by developers?

Sure, folks can create new transports for Avro.  There is, for example, 
in Hadoop Common some code that tunnels Avro RPCs inside Hadoop RPCs.

> Where does security lie : Avro or Transport layer?

That's not yet clear.  If we settle on HTTP as the preferred transport, 
then the transport should probably handle security, since many security 
standards already exist for HTTP and many HTTP servers and clients 
already support adding new security mechanisms.  I'd rather not 
re-invent all this in Avro if we can avoid it.

> If it is part of transport : How does an app get hold of required 
> information (e.g. user identity).

Perhaps the way we currently do this in the RPC server, with thread 
locals?  For example, the Avro RPC servlet could have a static method 
that returns that returns the value of 
HttpServletRequest#getUserPrincipal().

> May be 'transceiver' can have an interface that can transfer security 
> information between transport layer and Avro.

Yes, we could add methods like getPrincipal() to Transciever, but we'd 
still probably need to use a thread local accessed by a static method to 
get the Transciever if we continue to use reflection for server 
implementations.  Or we could stray from reflection, and make services 
implement an interface through which we can pass them things like the 
principal.

Doug

From general-return-552-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Sep 29 23:58:53 2009
Return-Path: <general-return-552-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 11756 invoked from network); 29 Sep 2009 23:58:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Sep 2009 23:58:53 -0000
Received: (qmail 86239 invoked by uid 500); 29 Sep 2009 23:58:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86137 invoked by uid 500); 29 Sep 2009 23:58:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86127 invoked by uid 99); 29 Sep 2009 23:58:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Sep 2009 23:58:52 +0000
X-ASF-Spam-Status: No, hits=4.2 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Sep 2009 23:58:41 +0000
Received: from sp1-ex07cas03.ds.corp.yahoo.com (sp1-ex07cas03.ds.corp.yahoo.com [216.252.116.151])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n8TNvYcx021942
	for <general@hadoop.apache.org>; Tue, 29 Sep 2009 16:57:34 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:in-reply-to:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=xnK6zAWp8In3PbZakCVUYO5X2dG3Js/PvF0mjJLdx0TTrXxSvg1knw+uh9BI8p5X
Received: from SP1-EX07VS02.ds.corp.yahoo.com ([216.252.116.135]) by
 sp1-ex07cas03.ds.corp.yahoo.com ([216.252.116.151]) with mapi; Tue, 29 Sep
 2009 16:57:33 -0700
From: Devaraj Das <ddas@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 29 Sep 2009 16:57:31 -0700
Subject: Re: HTTP transport?
Thread-Topic: HTTP transport?
Thread-Index: AcpBWywlZmAutc7dTNSnntYSTy7nawABW8OE
Message-ID: <C6E7ECFB.D5F0%ddas@yahoo-inc.com>
In-Reply-To: <4AC2956F.6020303@apache.org>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C6E7ECFBD5F0ddasyahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C6E7ECFBD5F0ddasyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Out of curiosity, do we have such numbers for the current hadoop RPC?


On 9/29/09 4:17 PM, "Doug Cutting" <cutting@apache.org> wrote:

stack wrote:
> So, are we're talking about doing something like following for a
> request/response:
>
>  GET /avro/org.apache.hadoop.hbase.RegionServer HTTP/1.1
>  Host: www.example.com
>
>
>  HTTP/1.1 200 OK
>  Date: Mon, 23 May 2005 22:38:34 GMT
>  Server: Apache/1.3.3.7 (Unix)  (Red-Hat/Linux)
>  Last-Modified: Wed, 08 Jan 2003 23:11:55 GMT
>  Etag: "3f80f-1b6-3e1cb03b"
>  Accept-Ranges: bytes
>  Content-Length: 438
>  Connection: close
>  Content-Type: X-avro/binary
>
>
> ... or some variation on above on each and every RPC?

More or less.  Except we can probably arrange to omit most of those
response headers except Content-Length.  Are any others strictly required?

I today implemented a simple HTTP-based transport for Avro:

   https://issues.apache.org/jira/browse/AVRO-129

In some simple benchmarks I am able to make over 5000 sequential
RPCs/second, each with ~100 bytes of response payload.  Increasing
response payloads to 100kB slows this to around 2500 RPCs/second, giving
throughput of 250MB/second, or 2.5Gbit/s.  This is with both client and
server running on my laptop.  The client is java.net.URLConnection and
the server is Jetty with its default configuration.

Doug


--_000_C6E7ECFBD5F0ddasyahooinccom_--

From general-return-553-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Sep 30 01:38:18 2009
Return-Path: <general-return-553-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 51650 invoked from network); 30 Sep 2009 01:38:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Sep 2009 01:38:18 -0000
Received: (qmail 55262 invoked by uid 500); 30 Sep 2009 01:38:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55189 invoked by uid 500); 30 Sep 2009 01:38:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55171 invoked by uid 99); 30 Sep 2009 01:38:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Sep 2009 01:38:17 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [64.78.17.16] (HELO EXHUB018-1.exch018.msoutlookonline.net) (64.78.17.16)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Sep 2009 01:38:06 +0000
Received: from EXVMBX018-1.exch018.msoutlookonline.net ([64.78.17.47]) by
 EXHUB018-1.exch018.msoutlookonline.net ([64.78.17.16]) with mapi; Tue, 29 Sep
 2009 18:37:45 -0700
From: Scott Carey <scott@richrelevance.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 29 Sep 2009 18:37:35 -0700
Subject: Re: HTTP transport?
Thread-Topic: HTTP transport?
Thread-Index: AcpBWwSLrypcEXd4TlmQxIIxaVoQUgAE5FS/
Message-ID: <C6E8046F.12C6A%scott@richrelevance.com>
In-Reply-To: <4AC2956F.6020303@apache.org>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C6E8046F12C6Ascottrichrelevancecom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C6E8046F12C6Ascottrichrelevancecom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

BTW, java.net.UrlConnection is the likely bottleneck there - it stinks perf=
ormance-wise.  The Apache commons http client is much faster.  Try out usin=
g Jmeter and switch from one connector to the other for an example.


On 9/29/09 4:17 PM, "Doug Cutting" <cutting@apache.org> wrote:

stack wrote:
> So, are we're talking about doing something like following for a
> request/response:
>
>  GET /avro/org.apache.hadoop.hbase.RegionServer HTTP/1.1
>  Host: www.example.com
>
>
>  HTTP/1.1 200 OK
>  Date: Mon, 23 May 2005 22:38:34 GMT
>  Server: Apache/1.3.3.7 (Unix)  (Red-Hat/Linux)
>  Last-Modified: Wed, 08 Jan 2003 23:11:55 GMT
>  Etag: "3f80f-1b6-3e1cb03b"
>  Accept-Ranges: bytes
>  Content-Length: 438
>  Connection: close
>  Content-Type: X-avro/binary
>
>
> ... or some variation on above on each and every RPC?

More or less.  Except we can probably arrange to omit most of those
response headers except Content-Length.  Are any others strictly required?

I today implemented a simple HTTP-based transport for Avro:

   https://issues.apache.org/jira/browse/AVRO-129

In some simple benchmarks I am able to make over 5000 sequential
RPCs/second, each with ~100 bytes of response payload.  Increasing
response payloads to 100kB slows this to around 2500 RPCs/second, giving
throughput of 250MB/second, or 2.5Gbit/s.  This is with both client and
server running on my laptop.  The client is java.net.URLConnection and
the server is Jetty with its default configuration.

Doug


--_000_C6E8046F12C6Ascottrichrelevancecom_--

From general-return-554-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Sep 30 03:07:04 2009
Return-Path: <general-return-554-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76964 invoked from network); 30 Sep 2009 03:07:04 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Sep 2009 03:07:04 -0000
Received: (qmail 99962 invoked by uid 500); 30 Sep 2009 03:07:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99826 invoked by uid 500); 30 Sep 2009 03:07:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 99816 invoked by uid 99); 30 Sep 2009 03:07:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Sep 2009 03:07:03 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [64.78.17.18] (HELO EXHUB018-3.exch018.msoutlookonline.net) (64.78.17.18)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Sep 2009 03:06:53 +0000
Received: from EXVMBX018-1.exch018.msoutlookonline.net ([64.78.17.47]) by
 EXHUB018-3.exch018.msoutlookonline.net ([64.78.17.18]) with mapi; Tue, 29 Sep
 2009 20:06:33 -0700
From: Scott Carey <scott@richrelevance.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 29 Sep 2009 20:06:22 -0700
Subject: Re: HTTP transport?
Thread-Topic: HTTP transport?
Thread-Index: AcpBT+192SUEA7aMSymjKr7qp00t4AAKw+Fi
Message-ID: <C6E8193E.12C83%scott@richrelevance.com>
In-Reply-To: <7c962aed0909291457o48baee08hfa3b6f4eb16213de@mail.gmail.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org




On 9/29/09 2:57 PM, "stack" <stack@duboce.net> wrote:

> On Tue, Sep 29, 2009 at 2:08 PM, Doug Cutting <cutting@apache.org> wrote:
>=20
>>=20
>> Alternately, we could try to make Avro's RPC more HTTP-friendly, and pul=
l
>> stuff out of Avro's payload into HTTP headers.  The downside of that wou=
ld
>> be that, if we still wish to support non-HTTP transports, we'd end up wi=
th
>> duplicated logic.
>>=20
>=20
>=20
> There would be loads of upside I'd imagine if there was a natural mapping=
 of
> avro payload specifiers and metadata up into http headers in terms of
> visibility
>=20

There are some very serious disadvantages to headers if overused.

I highly advise keeping what goes into the URL and headers very specific to
support well defined features for this specific transport type.  Otherwise,
put it in the data payload for all transports.

A couple header disadvantages:
* Limited character set allowed.  You can't put any data in there you want,
and you can end up with an inefficient encoding mess that is not easy to
read.
* Headers don't take advantage of other transport features.  For example,
Content-Encoding:gzip provides gzip compression support for the data
payload, but you can't compress the headers in HTTP.

On the other hand, Custom headers can be handy ways to implement transport
specific handshakes or advertize capabilities (which helps build in
cross-version compatibility).
But browsers only work with the standard ones, so whatever 'browser
requirement' is out there is going to be a limited subset no matter how you
do it.

This thread brings up the security features.  Payload encryption does not
seem to be a transport feature -- but it could be done via something like
Content-Encoding (X-Avro-Content-Encrypted?).  It seems to fit better IMO
within the payload itself, or at the socket / network level via SSH or a
secure tunnel.

Authentication is a better fit for the transport layer -- but as mentioned
elsewhere if it has to be done for all transports, could it fit in the
payload somehow?=20

>=20
> So, are we're talking about doing something like following for a
> request/response:
>=20
>  GET /avro/org.apache.hadoop.hbase.RegionServer HTTP/1.1
>  Host: www.example.com
>=20
>=20
>  HTTP/1.1 200 OK
>  Date: Mon, 23 May 2005 22:38:34 GMT
>  Server: Apache/1.3.3.7 (Unix)  (Red-Hat/Linux)
>  Last-Modified: Wed, 08 Jan 2003 23:11:55 GMT
>  Etag: "3f80f-1b6-3e1cb03b"
>  Accept-Ranges: bytes
>  Content-Length: 438
>  Connection: close
>  Content-Type: X-avro/binary
>=20

Its acceptable to drop a lot of the headers above.  Some of them are useful
to implement extended functionality -- the Etag can be used for caching if
that were desired, for example.  Keep-Alive connections and chunked
responses are nice built-ins too.

>=20
> ... or some variation on above on each and every RPC?
>=20
> St.Ack
>=20


From general-return-555-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Sep 30 03:21:15 2009
Return-Path: <general-return-555-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81897 invoked from network); 30 Sep 2009 03:21:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Sep 2009 03:21:15 -0000
Received: (qmail 10122 invoked by uid 500); 30 Sep 2009 03:21:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 10024 invoked by uid 500); 30 Sep 2009 03:21:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 10014 invoked by uid 99); 30 Sep 2009 03:21:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Sep 2009 03:21:14 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ryanobjc@gmail.com designates 209.85.212.186 as permitted sender)
Received: from [209.85.212.186] (HELO mail-vw0-f186.google.com) (209.85.212.186)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Sep 2009 03:21:04 +0000
Received: by vws16 with SMTP id 16so3859189vws.2
        for <general@hadoop.apache.org>; Tue, 29 Sep 2009 20:20:43 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=6/nNEJPLJn7szlBU0LWThtLVBIDJyBjsK2MPYypEhss=;
        b=bGBAyxFH6AKCdkb1diP94s2f7eoXOowz+Y97J7AMUxjKrjm+bRUMMIuUOvTflC9HcA
         mEgyJ68gs+jQg87k/zh00+sb70ULGVhSCDNwfQLlfsVTXbdhheq8b3TWu9J5Hgfno10r
         SGqS3DrbSv8Lj66xeJWQQJmWC4Y98lzhxSfho=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=kZHhLAQsZK+su7Jmu8v8xn7+saCYgdFX1ubGDwzCXokDaSxif5BGpsDjZHQ4oyuCdE
         GIgLWHgnw6Xu5kgElrNM0wNearDh7cqSmzudGUso+/HeE7dcr+DFgDRkhqH8ctUloCyR
         amWcjqJzqHMMXYCSuPQaXVWAnNRqiTvuAZyOs=
MIME-Version: 1.0
Received: by 10.220.69.21 with SMTP id x21mr9512569vci.9.1254280843184; Tue, 
	29 Sep 2009 20:20:43 -0700 (PDT)
In-Reply-To: <C6E8193E.12C83%scott@richrelevance.com>
References: <7c962aed0909291457o48baee08hfa3b6f4eb16213de@mail.gmail.com>
	 <C6E8193E.12C83%scott@richrelevance.com>
Date: Tue, 29 Sep 2009 20:20:43 -0700
Message-ID: <78568af10909292020r21f2c765jbe091941d5003355@mail.gmail.com>
Subject: Re: HTTP transport?
From: Ryan Rawson <ryanobjc@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

I wanted to chime in on a few things, since avro is a candidate for
the HBase RPC.

I am not sure that "browser compatibility" is a legitimate requirement
for this kind of thing. It is at odds with high performance in a
number of areas, and isn't the driving factor for using HTTP anyways.

Security - you can get the advantage of security standards, such as
the X.509 SSL cert, without actually using HTTPS.

Headers - I don't really think providing a caching mechanism built
into the RPC layer is a top requirement.  We'd then have to build in a
GET/POST idempotent flag into the Avro IDL, and everyone would have to
get it right, etc.

Considering my top requirement is "make bulk data access and RPC
rate/sec as high as possible", I'm not sure caching fits in here and
can work against that.


On Tue, Sep 29, 2009 at 8:06 PM, Scott Carey <scott@richrelevance.com> wrot=
e:
>
>
>
> On 9/29/09 2:57 PM, "stack" <stack@duboce.net> wrote:
>
>> On Tue, Sep 29, 2009 at 2:08 PM, Doug Cutting <cutting@apache.org> wrote=
:
>>
>>>
>>> Alternately, we could try to make Avro's RPC more HTTP-friendly, and pu=
ll
>>> stuff out of Avro's payload into HTTP headers. =A0The downside of that =
would
>>> be that, if we still wish to support non-HTTP transports, we'd end up w=
ith
>>> duplicated logic.
>>>
>>
>>
>> There would be loads of upside I'd imagine if there was a natural mappin=
g of
>> avro payload specifiers and metadata up into http headers in terms of
>> visibility
>>
>
> There are some very serious disadvantages to headers if overused.
>
> I highly advise keeping what goes into the URL and headers very specific =
to
> support well defined features for this specific transport type. =A0Otherw=
ise,
> put it in the data payload for all transports.
>
> A couple header disadvantages:
> * Limited character set allowed. =A0You can't put any data in there you w=
ant,
> and you can end up with an inefficient encoding mess that is not easy to
> read.
> * Headers don't take advantage of other transport features. =A0For exampl=
e,
> Content-Encoding:gzip provides gzip compression support for the data
> payload, but you can't compress the headers in HTTP.
>
> On the other hand, Custom headers can be handy ways to implement transpor=
t
> specific handshakes or advertize capabilities (which helps build in
> cross-version compatibility).
> But browsers only work with the standard ones, so whatever 'browser
> requirement' is out there is going to be a limited subset no matter how y=
ou
> do it.
>
> This thread brings up the security features. =A0Payload encryption does n=
ot
> seem to be a transport feature -- but it could be done via something like
> Content-Encoding (X-Avro-Content-Encrypted?). =A0It seems to fit better I=
MO
> within the payload itself, or at the socket / network level via SSH or a
> secure tunnel.
>
> Authentication is a better fit for the transport layer -- but as mentione=
d
> elsewhere if it has to be done for all transports, could it fit in the
> payload somehow?
>
>>
>> So, are we're talking about doing something like following for a
>> request/response:
>>
>> =A0GET /avro/org.apache.hadoop.hbase.RegionServer HTTP/1.1
>> =A0Host: www.example.com
>>
>>
>> =A0HTTP/1.1 200 OK
>> =A0Date: Mon, 23 May 2005 22:38:34 GMT
>> =A0Server: Apache/1.3.3.7 (Unix) =A0(Red-Hat/Linux)
>> =A0Last-Modified: Wed, 08 Jan 2003 23:11:55 GMT
>> =A0Etag: "3f80f-1b6-3e1cb03b"
>> =A0Accept-Ranges: bytes
>> =A0Content-Length: 438
>> =A0Connection: close
>> =A0Content-Type: X-avro/binary
>>
>
> Its acceptable to drop a lot of the headers above. =A0Some of them are us=
eful
> to implement extended functionality -- the Etag can be used for caching i=
f
> that were desired, for example. =A0Keep-Alive connections and chunked
> responses are nice built-ins too.
>
>>
>> ... or some variation on above on each and every RPC?
>>
>> St.Ack
>>
>
>

From general-return-556-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Sep 30 16:22:42 2009
Return-Path: <general-return-556-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27452 invoked from network); 30 Sep 2009 16:22:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Sep 2009 16:22:42 -0000
Received: (qmail 98131 invoked by uid 500); 30 Sep 2009 16:22:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98087 invoked by uid 500); 30 Sep 2009 16:22:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98077 invoked by uid 99); 30 Sep 2009 16:22:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Sep 2009 16:22:41 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [67.195.15.204] (HELO web111413.mail.gq1.yahoo.com) (67.195.15.204)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 30 Sep 2009 16:22:31 +0000
Received: (qmail 65192 invoked by uid 60001); 30 Sep 2009 16:22:09 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1254327729; bh=dIBJ4FouLfHPyOJ/OB3C53vrkZtezcccRK100EKufIA=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=MyJ62+WSc4KD7N/wpOmSZfUugyh0Lx+kOAy9EHAidnjhNHmqGsP6m6Jm46kh7v+L84DhiIymA0VYyvEF0de7rDVvDazp9bJNRdVb3e2u+1zqArnRsq7gkrV13FKdJbns8MJfClOu/BrB5xBEHURAyAAjYRiBhd6G+36nDHR/cj4=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=SAztRV58DV1m8tdVIMNooPyQ10lMwdzRaQOZoUGjSzApGG6iGJWZuQlVcP9j6NmNmNIuAvetY/woFWPhgAVmnwFMHZODRjT1unGtndDLyuPQ9rbHDqAQb6g2RKlN3UqR3oRMCNeiMaKvVmfh3zpoxU+pOkUj0QJLEkOZ0waauws=;
Message-ID: <440561.64930.qm@web111413.mail.gq1.yahoo.com>
X-YMail-OSG: _aEegsQVM1ksqeAznsVzVZno2EN3UTuI0Z.0Jk4XHteTVFDhKZ76cfmKElYz0iavAqAqnj4Kwalz.2mKZqYkjCFNeu5ZZRv9GQw1VYKVJ0Jb7ayfOKgTYizt8c6hdZZdMk9fhDOaNdNqbHgFfEHG7KCDj8jLLTKme6s.pZ_LOtVlU86p_v2wsqU7.pMYM2cwTp9hfbTh7XJsWWOHdvTlgfXmUJxMgFk1HCDdfIW11R_Ea8japz2HtaAvJeDba9q6Tpvegyy8LbISNBOmIOQeE2TPrho7Z6G1RQ8G.THrXNe68FxtIwBXwm52niS2.3XNB732
Received: from [216.239.45.4] by web111413.mail.gq1.yahoo.com via HTTP; Wed, 30 Sep 2009 09:22:09 PDT
X-Mailer: YahooMailClassic/7.0.14 YahooMailWebService/0.7.347.3
Date: Wed, 30 Sep 2009 09:22:09 -0700 (PDT)
From: Something Something <luckyguy2050@yahoo.com>
Subject: HBase configuration...
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-113412847-1254327729=:64930"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-113412847-1254327729=:64930
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Hello,

I noticed that if I start HBase in "standalone" mode it creates a table in =
memory.=A0 In other words, after rebooting the machine, the table goes away=
.=A0 I would like to persist the table to a local file system.=A0 How can I=
 do that?=A0 Do I have to use "Psuedo Distribution" mode for this?

Please help.=A0 Thanks.

=0A=0A=0A      
--0-113412847-1254327729=:64930--

From general-return-557-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Sep 30 21:43:00 2009
Return-Path: <general-return-557-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 99819 invoked from network); 30 Sep 2009 21:43:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Sep 2009 21:43:00 -0000
Received: (qmail 8838 invoked by uid 500); 30 Sep 2009 21:42:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8739 invoked by uid 500); 30 Sep 2009 21:42:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8729 invoked by uid 99); 30 Sep 2009 21:42:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Sep 2009 21:42:59 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 209.85.221.188 as permitted sender)
Received: from [209.85.221.188] (HELO mail-qy0-f188.google.com) (209.85.221.188)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Sep 2009 21:42:48 +0000
Received: by qyk26 with SMTP id 26so5748058qyk.5
        for <general@hadoop.apache.org>; Wed, 30 Sep 2009 14:42:25 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:message-id:subject:from:to
         :content-type;
        bh=xHTtXMuWiIrxCo4G0Dp0Z43UXxKC888lC66OIn83Dxw=;
        b=pVSIZt03/uXhjZytRcTJHg2C5JVfmaC4UBk3nBAqVrmBuXkdoLOgiDC+Q1EZ/Gdh4p
         22S2S0UyYRoEsH45rMfbuSsFju10p5b3Qo4w4L6qz9cISl5VyA15K9hOF/MPDoDdJ9B0
         bYxNwRuuw8Sq5XrFzgBXn6gfjlohz8u6/pgKY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=XC+mj0gcwSY06AUFwq0zEchjBR0Ldvtf42rlvszU2ZpZha6qxRE9cOX1Y6yVhjpilF
         CbY5EjD01nDKgcpm8el4IHraH6r6BKkBiRJh80S3WdEku2BNLPtYr0Akqm0k03WEzHNs
         6cQ1RwQ+rfLlogBk41S0IHpjtGe1ruStZu15Y=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
Received: by 10.229.31.211 with SMTP id z19mr254437qcc.24.1254346945616; Wed, 
	30 Sep 2009 14:42:25 -0700 (PDT)
In-Reply-To: <440561.64930.qm@web111413.mail.gq1.yahoo.com>
References: <440561.64930.qm@web111413.mail.gq1.yahoo.com>
Date: Wed, 30 Sep 2009 14:42:25 -0700
X-Google-Sender-Auth: 4410559c9aa9cec5
Message-ID: <7c962aed0909301442g57a7b4ceg62a84ccb0ff22102@mail.gmail.com>
Subject: Re: HBase configuration...
From: stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364eeba81e5b8d0474d268c8
X-Virus-Checked: Checked by ClamAV on apache.org

--0016364eeba81e5b8d0474d268c8
Content-Type: text/plain; charset=ISO-8859-1

Shut it down cleanly (./bin/stop-hbase.sh).  My guess is that you are
killing it before it has chance to flush its in-memory state.

HBase questions will get more timely response if posted to the hbase lists
(see hbase.org).

Yours,
St.Ack St.Ack


On Wed, Sep 30, 2009 at 9:22 AM, Something Something <luckyguy2050@yahoo.com
> wrote:

> Hello,
>
> I noticed that if I start HBase in "standalone" mode it creates a table in
> memory.  In other words, after rebooting the machine, the table goes away.
> I would like to persist the table to a local file system.  How can I do
> that?  Do I have to use "Psuedo Distribution" mode for this?
>
> Please help.  Thanks.
>
>
>
>
>

--0016364eeba81e5b8d0474d268c8--

From general-return-558-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Sep 30 22:33:34 2009
Return-Path: <general-return-558-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16541 invoked from network); 30 Sep 2009 22:33:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Sep 2009 22:33:34 -0000
Received: (qmail 69132 invoked by uid 500); 30 Sep 2009 22:33:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69056 invoked by uid 500); 30 Sep 2009 22:33:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69046 invoked by uid 99); 30 Sep 2009 22:33:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Sep 2009 22:33:33 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [67.195.15.210] (HELO web111414.mail.gq1.yahoo.com) (67.195.15.210)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 30 Sep 2009 22:33:21 +0000
Received: (qmail 69337 invoked by uid 60001); 30 Sep 2009 22:33:00 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1254349980; bh=MFBzsm/EQiUOkwMmKVpnwZon9uBn8ustxLzTAyFLjeA=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=MHn+Ctvf/B9EKcd8IyELtP3xA9WKGmKRZSzM5uav7m+AROYHYFBJV7dMeNn2PLjDlrdF4JZyM0oteQBW591LHME4sJZP8hR4QrsEcPQzVc/+n91PBJH5sRkp7OBBMoDv9l5i60qCGvCn/ZFb7x0DxVNNSd/DApLOzR2KwCgy75c=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type;
  b=4hV1ExUYq7eIvpMFv7IUk4f15MXsSvKezMfNqHizDTdPSmKkzEAXJTwm7ZgqyQrFWPxjECXW7wj2hIDlK9D22wP34NjekmA5EF8PbpFvPQqRuBs3QMekPzf7Ulz4HrUXLB4cyg7ThEAJeKQXVArhVsdQq/1c57DVdnPOcBHoIQE=;
Message-ID: <231420.69323.qm@web111414.mail.gq1.yahoo.com>
X-YMail-OSG: lQqNL0AVM1nKwMJZWYihByPDrp4g6ukHdAiiArzvmvEDi1e5fsBKCkBYQCxB63uPobKtjj2wTyVTTLID0LmliEAc5YDB7Jg2IefXAD_qLmc62CkflI21yZxsAPpq.cUVoezkr34I5GvmqWAyhyCPAoYUSyKlps0mghoKx3hzRQWv3Ze1tF5iyaPk4nHZH2BbeGIwj6M3nXoH7y0vF9cvEQcRL7rDF9qkMcsecIzW6Aol7v6X6i4ilEm.Vyx5QsUZWMJ_EKh8UBmNK8i6rjh.pxVYpM3hVKpV5GTHmxQVaSXVKw--
Received: from [75.54.226.161] by web111414.mail.gq1.yahoo.com via HTTP; Wed, 30 Sep 2009 15:33:00 PDT
X-Mailer: YahooMailClassic/7.0.14 YahooMailWebService/0.7.347.2
Date: Wed, 30 Sep 2009 15:33:00 -0700 (PDT)
From: Something Something <luckyguy2050@yahoo.com>
Subject: Re: HBase configuration...
To: general@hadoop.apache.org
Cc: hbase-user@hadoop.apache.org
In-Reply-To: <7c962aed0909301442g57a7b4ceg62a84ccb0ff22102@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1358326848-1254349980=:69323"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-1358326848-1254349980=:69323
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Hmmm.. Interesting... I could swear the 2nd time around I had run stop-hbas=
e.sh before rebooting. =A0But may be I didn't wait long enough. =A0My job a=
dds over 100 million rows to a table. =A0I have plenty of free space on my =
hard drive, so I am assuming this will all work in "standalone" mode, corre=
ct? =A0(Just doing a proof of concept for now.)

In any case, thanks for the reply. =A0It sounds like the tables should get =
persisted to disk. =A0Will try once again. =A0Thanks.

--- On Wed, 9/30/09, stack <stack@duboce.net> wrote:

From: stack <stack@duboce.net>
Subject: Re: HBase configuration...
To: general@hadoop.apache.org
Date: Wednesday, September 30, 2009, 2:42 PM

Shut it down cleanly (./bin/stop-hbase.sh).=A0 My guess is that you are
killing it before it has chance to flush its in-memory state.

HBase questions will get more timely response if posted to the hbase lists
(see hbase.org).

Yours,
St.Ack St.Ack


On Wed, Sep 30, 2009 at 9:22 AM, Something Something <luckyguy2050@yahoo.co=
m
> wrote:

> Hello,
>
> I noticed that if I start HBase in "standalone" mode it creates a table i=
n
> memory.=A0 In other words, after rebooting the machine, the table goes aw=
ay.
> I would like to persist the table to a local file system.=A0 How can I do
> that?=A0 Do I have to use "Psuedo Distribution" mode for this?
>
> Please help.=A0 Thanks.
>
>
>
>
>
=0A=0A=0A      
--0-1358326848-1254349980=:69323--

From general-return-559-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Sep 30 23:05:43 2009
Return-Path: <general-return-559-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25583 invoked from network); 30 Sep 2009 23:05:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Sep 2009 23:05:43 -0000
Received: (qmail 12097 invoked by uid 500); 30 Sep 2009 23:05:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12019 invoked by uid 500); 30 Sep 2009 23:05:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12009 invoked by uid 99); 30 Sep 2009 23:05:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Sep 2009 23:05:42 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Sep 2009 23:05:30 +0000
Received: from [172.16.132.152] (snvvpn1-10-72-244-c195.hq.corp.yahoo.com [10.72.244.195])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n8UN3FkM018908
	for <general@hadoop.apache.org>; Wed, 30 Sep 2009 16:04:11 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:mime-version:
	subject:date:references:x-mailer;
	b=KeWgJR/B8xqV1Wi6aRN/kox0R3DCYtKj2ejN7sfBQcUZi4yyeW7COs8a8sUALpMr
Message-Id: <076526D4-2D47-4841-99E0-9D3678ED7F99@yahoo-inc.com>
From: Sanjay Radia <sradia@yahoo-inc.com>
To: <general@hadoop.apache.org>
In-Reply-To: <4AC2775C.2000302@apache.org>
Content-Type: multipart/alternative; boundary=Apple-Mail-45--980739934
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: HTTP transport?
Date: Wed, 30 Sep 2009 16:04:10 -0700
References: <4AAAC403.80809@apache.org>	 <1CB467B9-B60C-4ED0-BA82-7BB571E724C6@apache.org>	 <4AC13BC7.6020902@apache.org>	 <B38E3438-CC8F-48B8-B415-549165996ED5@yahoo-inc.com>	 <4AC2635C.20902@apache.org> <7c962aed0909291338o7f09601ci102186e23fd4d464@mail.gmail.com> <4AC2775C.2000302@apache.org>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-45--980739934
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


On Sep 29, 2009, at 2:08 PM, Doug Cutting wrote:

> ...
>
> Alternately, we could try to make Avro's RPC more HTTP-friendly, and
> pull stuff out of Avro's payload into HTTP headers.  The downside of
> that would be that, if we still wish to support non-HTTP transports,
> we'd end up with duplicated logic.
>

I would prefer to retain layer independence so that we can use other  
transports.
(I am still not sold on HTTP as a transport so far but am listening  
with an open mind).
>
>


--Apple-Mail-45--980739934--

From general-return-560-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Sep 30 23:08:44 2009
Return-Path: <general-return-560-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25970 invoked from network); 30 Sep 2009 23:08:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Sep 2009 23:08:44 -0000
Received: (qmail 13716 invoked by uid 500); 30 Sep 2009 23:08:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13633 invoked by uid 500); 30 Sep 2009 23:08:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13623 invoked by uid 99); 30 Sep 2009 23:08:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Sep 2009 23:08:43 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 209.85.222.198 is neither permitted nor denied by domain of kpeterson@biz360.com)
Received: from [209.85.222.198] (HELO mail-pz0-f198.google.com) (209.85.222.198)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Sep 2009 23:08:32 +0000
Received: by pzk36 with SMTP id 36so2566872pzk.5
        for <general@hadoop.apache.org>; Wed, 30 Sep 2009 16:08:10 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.142.210.17 with SMTP id i17mr33077wfg.29.1254352090705; Wed, 
	30 Sep 2009 16:08:10 -0700 (PDT)
In-Reply-To: <231420.69323.qm@web111414.mail.gq1.yahoo.com>
References: <7c962aed0909301442g57a7b4ceg62a84ccb0ff22102@mail.gmail.com>
	 <231420.69323.qm@web111414.mail.gq1.yahoo.com>
Date: Wed, 30 Sep 2009 16:08:10 -0700
Message-ID: <f27b3fc0909301608mbc71980sa7fe78765201f9dc@mail.gmail.com>
Subject: Re: HBase configuration...
From: Kevin Peterson <kpeterson@biz360.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd32ed6ca30620474d39a5e
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd32ed6ca30620474d39a5e
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Sep 30, 2009 at 3:33 PM, Something Something <luckyguy2050@yahoo.com
> wrote:

> Hmmm.. Interesting... I could swear the 2nd time around I had run
> stop-hbase.sh before rebooting.  But may be I didn't wait long enough.  My
> job adds over 100 million rows to a table.  I have plenty of free space on
> my hard drive, so I am assuming this will all work in "standalone" mode,
> correct?  (Just doing a proof of concept for now.)
>
> By default it gets saved into /tmp, which on many machines gets cleared out
every night or after every boot.

Set hbase.rootdir to something else (writable) to ensure it isn't cleared on
reboot.

--000e0cd32ed6ca30620474d39a5e--

From general-return-2920-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 01 00:28:06 2011
Return-Path: <general-return-2920-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42222 invoked from network); 1 Feb 2011 00:28:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Feb 2011 00:28:05 -0000
Received: (qmail 96786 invoked by uid 500); 1 Feb 2011 00:28:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96694 invoked by uid 500); 1 Feb 2011 00:28:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96686 invoked by uid 99); 1 Feb 2011 00:28:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 00:28:03 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of shv.hadoop@gmail.com designates 209.85.218.48 as permitted sender)
Received: from [209.85.218.48] (HELO mail-yi0-f48.google.com) (209.85.218.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 00:27:56 +0000
Received: by yib17 with SMTP id 17so2450891yib.35
        for <general@hadoop.apache.org>; Mon, 31 Jan 2011 16:27:35 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=VzFaMlzInQpReSO5S0QQ8mOsn8V99RQqAacg1bLWkwk=;
        b=Bf0/zdX42B06K/PXsW5QqBrWaxmtYdubCqYJR8NiMfa99aUS1vCeUsCyW8nr2ULUo8
         aPZHWsq1Z5Kw1ZkkRaoqDLeafFDRlag0AfyxFm2RRdx6CWXvQeDd9C68Y+46AdshQCq/
         pCeJbmyFmVpj0co0153PI5qzYgHLde4WngIv8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=aTXtJB4PBUkqjXPJr2adAD8vjtVbW0DqtjstMaRHJ350ZA7eV190jYpovH9U2BcMvP
         8CNbe8lFhAAX6uTTs0cJgFORPu6z4ZWy8MEuXEDs6Ix7Jb87WiTx27WtAOY1cwZriQia
         142rO9QQyrkVN09FeYq4QPf7MhWeWyegfZos4=
MIME-Version: 1.0
Received: by 10.236.108.43 with SMTP id p31mr14127738yhg.22.1296520054330;
 Mon, 31 Jan 2011 16:27:34 -0800 (PST)
Received: by 10.236.108.35 with HTTP; Mon, 31 Jan 2011 16:27:34 -0800 (PST)
In-Reply-To: <7DC70A70-5DE9-4288-A44B-B5EF9C454D9B@yahoo-inc.com>
References: <AANLkTinB+8KRByZFhRT-y6MiqA0urtKNs8c_XSrhP7RH@mail.gmail.com>
	<AANLkTimmcYh0hY-6ht6vaQ+S=pTGHaFxS6k9qEv=q8Cf@mail.gmail.com>
	<AANLkTinr_Exrr8VPShGKkBSMzmWX1B9nVALLLTMmvYzd@mail.gmail.com>
	<AANLkTinrheVYQm-y9n=k7TRK_BdzuZYKAWG+qMWR88ct@mail.gmail.com>
	<AANLkTi=cAhCOkqZ=n9NpBMoPzechoJmfB6KO2yL3QcVA@mail.gmail.com>
	<AANLkTi=+81g8dFMGMqP5Zc6aAkg5n0dD2QpCVg7ABzd6@mail.gmail.com>
	<7DC70A70-5DE9-4288-A44B-B5EF9C454D9B@yahoo-inc.com>
Date: Mon, 31 Jan 2011 16:27:34 -0800
Message-ID: <AANLkTin9fsb3bgsa98bma+xx_C9K06C_YNKJ6vaA7=TO@mail.gmail.com>
Subject: Re: Hadoop-common-trunk-Commit is failing since 01/19/2011
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba4fbe2e4865ea049b2d99c4
X-Virus-Checked: Checked by ClamAV on apache.org

--90e6ba4fbe2e4865ea049b2d99c4
Content-Type: text/plain; charset=ISO-8859-1

Giri
looks like the last run you started failed the same way as previous ones.
Any thoughts on what's going on?
Thanks,
--Konstantin

On Mon, Jan 31, 2011 at 3:33 PM, Giridharan Kesavan
<gkesavan@yahoo-inc.com>wrote:

> ant mvn-deploy would publish snapshot artifact to the apache maven
> repository as long you have the right credentials in ~/.m2/settings.xml.
>
> For settings.xml template pls look at
> http://wiki.apache.org/hadoop/HowToRelease
>
> I'm pushing the latest common artifacts now.
>
> -Giri
>
>
>
> On Jan 31, 2011, at 3:11 PM, Jakob Homan wrote:
>
> > By manually installing a new core jar into the cache, I can compile
> > trunk.  Looks like we just need to kick a new Core into maven.  Are
> > there instructions somewhere for committers to do this?  I know Nigel
> > and Owen know how, but I don't know if the knowledge is diffused past
> > them.
> > -Jakob
> >
> >
> > On Mon, Jan 31, 2011 at 1:57 PM, Konstantin Shvachko
> > <shv.hadoop@gmail.com> wrote:
> >> Current trunk for HDFS and MapReduce are not compiling at the moment.
> Try to
> >> build trunk.
> >> This is the result of that changes to common api introduced by
> HADOOP-6904
> >> are not promoted to HDFS and MR trunks.
> >> HDFS-1335 and MAPREDUCE-2263 depend on these changes.
> >>
> >> Common is not promoted to HDFS and MR because Hadoop-Common-trunk-Commit
> >> build is broken. See here.
> >>
> https://hudson.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/
> >>
> >> As I see the last successful build was on 01/19, which integrated
> >> HADOOP-6864.
> >> I think this is when JNI changes were introduced, which cannot be
> digested
> >> by Hudson since then.
> >>
> >> Anybody with gcc active could you please verify if the problem is caused
> by
> >> HADOOP-6864.
> >>
> >> Thanks,
> >> --Konstantin
> >>
> >> On Mon, Jan 31, 2011 at 1:36 PM, Ted Dunning <tdunning@maprtech.com>
> wrote:
> >>
> >>> The has been a problem with more than one build failing (Mahout is the
> one
> >>> that I saw first) due to a change in maven version which meant that the
> >>> clover license isn't being found properly.  At least, that is the tale
> I
> >>> heard from infra.
> >>>
> >>> On Mon, Jan 31, 2011 at 1:31 PM, Eli Collins <eli@cloudera.com> wrote:
> >>>
> >>>> Hey Konstantin,
> >>>>
> >>>> The only build breakage I saw from HADOOP-6904 is MAPREDUCE-2290,
> >>>> which was fixed.  Trees from trunk are compiling against each other
> >>>> for me (eg each installed to a local maven repo), perhaps the upstream
> >>>> maven repo hasn't been updated with the latest bits yet.
> >>>>
> >>>> Thanks,
> >>>> Eli
> >>>>
> >>>> On Mon, Jan 31, 2011 at 12:14 PM, Konstantin Shvachko
> >>>> <shv.hadoop@gmail.com> wrote:
> >>>>> Sending this to general to attract urgent attention.
> >>>>> Both HDFS and MapReduce are not compiling since
> >>>>> HADOOP-6904 and its hdfs and MP counterparts were committed.
> >>>>> The problem is not with this patch as described below, but I think
> >>> those
> >>>>> commits should be reversed if Common integration build cannot be
> >>>>> restored promptly.
> >>>>>
> >>>>> Thanks,
> >>>>> --Konstantin
> >>>>>
> >>>>>
> >>>>> On Fri, Jan 28, 2011 at 5:53 PM, Konstantin Shvachko
> >>>>> <shv.hadoop@gmail.com>wrote:
> >>>>>
> >>>>>> I see Hadoop-common-trunk-Commit is failing and not sending any
> >>> emails.
> >>>>>> It times out on native compilation and aborts.
> >>>>>> Therefore changes are not integrated, and now it lead to hdfs and
> >>>> mapreduce
> >>>>>> both not compiling.
> >>>>>> Can somebody please take a look at this.
> >>>>>> The last few lines of the build are below.
> >>>>>>
> >>>>>> Thanks
> >>>>>> --Konstantin
> >>>>>>
> >>>>>>     [javah] [Loaded
> >>>>
> >>>
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/build/classes/org/apache/hadoop/security/JniBasedUnixGroupsMapping.class]
> >>>>>>
> >>>>>>     [javah] [Loaded
> >>>>
> >>>
> /homes/hudson/tools/java/jdk1.6.0_11-32/jre/lib/rt.jar(java/lang/Object.class)]
> >>>>>>     [javah] [Forcefully writing file
> >>>>
> >>>
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/build/native/Linux-i386-32/src/org/apache/hadoop/security/org_apache_hadoop_security_JniBasedUnixGroupsNetgroupMapping.h]
> >>>>>>
> >>>>>>      [exec] checking for gcc... gcc
> >>>>>>      [exec] checking whether the C compiler works... yes
> >>>>>>      [exec] checking for C compiler default output file name...
> a.out
> >>>>>>      [exec] checking for suffix of executables...
> >>>>>>
> >>>>>> Build timed out. Aborting
> >>>>>> Build was aborted
> >>>>>> [FINDBUGS] Skipping publisher since build result is ABORTED
> >>>>>> Publishing Javadoc
> >>>>>> Archiving artifacts
> >>>>>> Recording test results
> >>>>>> No test report files were found. Configuration error?
> >>>>>>
> >>>>>> Recording fingerprints
> >>>>>>      [exec] Terminated
> >>>>>> Publishing Clover coverage report...
> >>>>>> No Clover report will be published due to a Build Failure
> >>>>>> No emails were triggered.
> >>>>>> Finished: ABORTED
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>

--90e6ba4fbe2e4865ea049b2d99c4--

From general-return-2921-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 01 00:41:12 2011
Return-Path: <general-return-2921-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46822 invoked from network); 1 Feb 2011 00:41:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Feb 2011 00:41:12 -0000
Received: (qmail 11262 invoked by uid 500); 1 Feb 2011 00:41:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11201 invoked by uid 500); 1 Feb 2011 00:41:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11193 invoked by uid 99); 1 Feb 2011 00:41:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 00:41:10 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 00:41:04 +0000
Received: from SP2-EX07CAS03.ds.corp.yahoo.com (sp2-ex07cas03.corp.sp2.yahoo.com [98.137.59.35])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p110eT6i054994
	for <general@hadoop.apache.org>; Mon, 31 Jan 2011 16:40:29 -0800 (PST)
Received: from SP2-EX07VS02.ds.corp.yahoo.com ([98.137.59.30]) by
 SP2-EX07CAS03.ds.corp.yahoo.com ([98.137.59.35]) with mapi; Mon, 31 Jan 2011
 16:40:29 -0800
From: "Giridharan  Kesavan" <gkesavan@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 31 Jan 2011 16:40:28 -0800
Subject: Re: Hadoop-common-trunk-Commit is failing since 01/19/2011
Thread-Topic: Hadoop-common-trunk-Commit is failing since 01/19/2011
Thread-Index: AcvBqJ/6VhKWdNn+RhyDr1/eI3M8aA==
Message-ID: <F3847AE0-3FEF-4E85-8757-FA8A7B204A64@yahoo-inc.com>
References: <AANLkTinB+8KRByZFhRT-y6MiqA0urtKNs8c_XSrhP7RH@mail.gmail.com>
 <AANLkTimmcYh0hY-6ht6vaQ+S=pTGHaFxS6k9qEv=q8Cf@mail.gmail.com>
 <AANLkTinr_Exrr8VPShGKkBSMzmWX1B9nVALLLTMmvYzd@mail.gmail.com>
 <AANLkTinrheVYQm-y9n=k7TRK_BdzuZYKAWG+qMWR88ct@mail.gmail.com>
 <AANLkTi=cAhCOkqZ=n9NpBMoPzechoJmfB6KO2yL3QcVA@mail.gmail.com>
 <AANLkTi=+81g8dFMGMqP5Zc6aAkg5n0dD2QpCVg7ABzd6@mail.gmail.com>
 <7DC70A70-5DE9-4288-A44B-B5EF9C454D9B@yahoo-inc.com>
 <AANLkTin9fsb3bgsa98bma+xx_C9K06C_YNKJ6vaA7=TO@mail.gmail.com>
In-Reply-To: <AANLkTin9fsb3bgsa98bma+xx_C9K06C_YNKJ6vaA7=TO@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Konstantin,

I think I need to restart the slave which is running the commit build. For =
now I have published the common artifact manually from commandline.

Thanks,
Giri

On Jan 31, 2011, at 4:27 PM, Konstantin Shvachko wrote:

> Giri
> looks like the last run you started failed the same way as previous ones.
> Any thoughts on what's going on?
> Thanks,
> --Konstantin
>=20
> On Mon, Jan 31, 2011 at 3:33 PM, Giridharan Kesavan
> <gkesavan@yahoo-inc.com>wrote:
>=20
>> ant mvn-deploy would publish snapshot artifact to the apache maven
>> repository as long you have the right credentials in ~/.m2/settings.xml.
>>=20
>> For settings.xml template pls look at
>> http://wiki.apache.org/hadoop/HowToRelease
>>=20
>> I'm pushing the latest common artifacts now.
>>=20
>> -Giri
>>=20
>>=20
>>=20
>> On Jan 31, 2011, at 3:11 PM, Jakob Homan wrote:
>>=20
>>> By manually installing a new core jar into the cache, I can compile
>>> trunk.  Looks like we just need to kick a new Core into maven.  Are
>>> there instructions somewhere for committers to do this?  I know Nigel
>>> and Owen know how, but I don't know if the knowledge is diffused past
>>> them.
>>> -Jakob
>>>=20
>>>=20
>>> On Mon, Jan 31, 2011 at 1:57 PM, Konstantin Shvachko
>>> <shv.hadoop@gmail.com> wrote:
>>>> Current trunk for HDFS and MapReduce are not compiling at the moment.
>> Try to
>>>> build trunk.
>>>> This is the result of that changes to common api introduced by
>> HADOOP-6904
>>>> are not promoted to HDFS and MR trunks.
>>>> HDFS-1335 and MAPREDUCE-2263 depend on these changes.
>>>>=20
>>>> Common is not promoted to HDFS and MR because Hadoop-Common-trunk-Comm=
it
>>>> build is broken. See here.
>>>>=20
>> https://hudson.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-Common-=
trunk-Commit/
>>>>=20
>>>> As I see the last successful build was on 01/19, which integrated
>>>> HADOOP-6864.
>>>> I think this is when JNI changes were introduced, which cannot be
>> digested
>>>> by Hudson since then.
>>>>=20
>>>> Anybody with gcc active could you please verify if the problem is caus=
ed
>> by
>>>> HADOOP-6864.
>>>>=20
>>>> Thanks,
>>>> --Konstantin
>>>>=20
>>>> On Mon, Jan 31, 2011 at 1:36 PM, Ted Dunning <tdunning@maprtech.com>
>> wrote:
>>>>=20
>>>>> The has been a problem with more than one build failing (Mahout is th=
e
>> one
>>>>> that I saw first) due to a change in maven version which meant that t=
he
>>>>> clover license isn't being found properly.  At least, that is the tal=
e
>> I
>>>>> heard from infra.
>>>>>=20
>>>>> On Mon, Jan 31, 2011 at 1:31 PM, Eli Collins <eli@cloudera.com> wrote=
:
>>>>>=20
>>>>>> Hey Konstantin,
>>>>>>=20
>>>>>> The only build breakage I saw from HADOOP-6904 is MAPREDUCE-2290,
>>>>>> which was fixed.  Trees from trunk are compiling against each other
>>>>>> for me (eg each installed to a local maven repo), perhaps the upstre=
am
>>>>>> maven repo hasn't been updated with the latest bits yet.
>>>>>>=20
>>>>>> Thanks,
>>>>>> Eli
>>>>>>=20
>>>>>> On Mon, Jan 31, 2011 at 12:14 PM, Konstantin Shvachko
>>>>>> <shv.hadoop@gmail.com> wrote:
>>>>>>> Sending this to general to attract urgent attention.
>>>>>>> Both HDFS and MapReduce are not compiling since
>>>>>>> HADOOP-6904 and its hdfs and MP counterparts were committed.
>>>>>>> The problem is not with this patch as described below, but I think
>>>>> those
>>>>>>> commits should be reversed if Common integration build cannot be
>>>>>>> restored promptly.
>>>>>>>=20
>>>>>>> Thanks,
>>>>>>> --Konstantin
>>>>>>>=20
>>>>>>>=20
>>>>>>> On Fri, Jan 28, 2011 at 5:53 PM, Konstantin Shvachko
>>>>>>> <shv.hadoop@gmail.com>wrote:
>>>>>>>=20
>>>>>>>> I see Hadoop-common-trunk-Commit is failing and not sending any
>>>>> emails.
>>>>>>>> It times out on native compilation and aborts.
>>>>>>>> Therefore changes are not integrated, and now it lead to hdfs and
>>>>>> mapreduce
>>>>>>>> both not compiling.
>>>>>>>> Can somebody please take a look at this.
>>>>>>>> The last few lines of the build are below.
>>>>>>>>=20
>>>>>>>> Thanks
>>>>>>>> --Konstantin
>>>>>>>>=20
>>>>>>>>    [javah] [Loaded
>>>>>>=20
>>>>>=20
>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/b=
uild/classes/org/apache/hadoop/security/JniBasedUnixGroupsMapping.class]
>>>>>>>>=20
>>>>>>>>    [javah] [Loaded
>>>>>>=20
>>>>>=20
>> /homes/hudson/tools/java/jdk1.6.0_11-32/jre/lib/rt.jar(java/lang/Object.=
class)]
>>>>>>>>    [javah] [Forcefully writing file
>>>>>>=20
>>>>>=20
>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/b=
uild/native/Linux-i386-32/src/org/apache/hadoop/security/org_apache_hadoop_=
security_JniBasedUnixGroupsNetgroupMapping.h]
>>>>>>>>=20
>>>>>>>>     [exec] checking for gcc... gcc
>>>>>>>>     [exec] checking whether the C compiler works... yes
>>>>>>>>     [exec] checking for C compiler default output file name...
>> a.out
>>>>>>>>     [exec] checking for suffix of executables...
>>>>>>>>=20
>>>>>>>> Build timed out. Aborting
>>>>>>>> Build was aborted
>>>>>>>> [FINDBUGS] Skipping publisher since build result is ABORTED
>>>>>>>> Publishing Javadoc
>>>>>>>> Archiving artifacts
>>>>>>>> Recording test results
>>>>>>>> No test report files were found. Configuration error?
>>>>>>>>=20
>>>>>>>> Recording fingerprints
>>>>>>>>     [exec] Terminated
>>>>>>>> Publishing Clover coverage report...
>>>>>>>> No Clover report will be published due to a Build Failure
>>>>>>>> No emails were triggered.
>>>>>>>> Finished: ABORTED
>>>>>>>>=20
>>>>>>>>=20
>>>>>>>>=20
>>>>>>>>=20
>>>>>>>=20
>>>>>>=20
>>>>>=20
>>>>=20
>>=20
>>=20


From general-return-2922-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 01 00:58:31 2011
Return-Path: <general-return-2922-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 51889 invoked from network); 1 Feb 2011 00:58:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Feb 2011 00:58:31 -0000
Received: (qmail 28562 invoked by uid 500); 1 Feb 2011 00:58:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28508 invoked by uid 500); 1 Feb 2011 00:58:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28500 invoked by uid 99); 1 Feb 2011 00:58:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 00:58:29 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of shv.hadoop@gmail.com designates 209.85.213.176 as permitted sender)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 00:58:24 +0000
Received: by yxm8 with SMTP id 8so2451823yxm.35
        for <general@hadoop.apache.org>; Mon, 31 Jan 2011 16:58:03 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=cyljU9HljYwCm1QQiAyVIs9t6E0lMU68UJjak0gcQVs=;
        b=NGAavLJ8azmcSB8YF+h/7VseJWQ9r7yTBEZcS1gx8Js91VRJm79gVHGUbfJABspb2p
         hh7MvT8xkn2Pdhcej2eXYPeJ2pQzF0zEWGYRi+vJYwaim1XmQeaOh6v8deXN+swgkRe6
         Mfxw8ZOHw1U6RAoxPjOypUwN4ANgemZERKJnk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=QSC3xBdSrnWbF7UzQpZ+KGrn1suwmc3g62BID8dpizx++4wdNuX9w0fPdm4j6Z/d8x
         EHiKt8CCG4XSIYQpoByzypW1gJkV+7J6T5onsDQRO1tYYfS8AwCBJtOaI3OHG/pPI73a
         61Cpn1mNsQSYypYLP7Zr6+Z/69IcEKJB2Bm9w=
MIME-Version: 1.0
Received: by 10.236.105.161 with SMTP id k21mr14018670yhg.87.1296521883101;
 Mon, 31 Jan 2011 16:58:03 -0800 (PST)
Received: by 10.236.108.35 with HTTP; Mon, 31 Jan 2011 16:58:03 -0800 (PST)
In-Reply-To: <F3847AE0-3FEF-4E85-8757-FA8A7B204A64@yahoo-inc.com>
References: <AANLkTinB+8KRByZFhRT-y6MiqA0urtKNs8c_XSrhP7RH@mail.gmail.com>
	<AANLkTimmcYh0hY-6ht6vaQ+S=pTGHaFxS6k9qEv=q8Cf@mail.gmail.com>
	<AANLkTinr_Exrr8VPShGKkBSMzmWX1B9nVALLLTMmvYzd@mail.gmail.com>
	<AANLkTinrheVYQm-y9n=k7TRK_BdzuZYKAWG+qMWR88ct@mail.gmail.com>
	<AANLkTi=cAhCOkqZ=n9NpBMoPzechoJmfB6KO2yL3QcVA@mail.gmail.com>
	<AANLkTi=+81g8dFMGMqP5Zc6aAkg5n0dD2QpCVg7ABzd6@mail.gmail.com>
	<7DC70A70-5DE9-4288-A44B-B5EF9C454D9B@yahoo-inc.com>
	<AANLkTin9fsb3bgsa98bma+xx_C9K06C_YNKJ6vaA7=TO@mail.gmail.com>
	<F3847AE0-3FEF-4E85-8757-FA8A7B204A64@yahoo-inc.com>
Date: Mon, 31 Jan 2011 16:58:03 -0800
Message-ID: <AANLkTi=o15MxMp6PPbr1WsmdmGpE+X7g+Sa1DV0TitTz@mail.gmail.com>
Subject: Re: Hadoop-common-trunk-Commit is failing since 01/19/2011
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba25e53b492bbd049b2e060c

--90e6ba25e53b492bbd049b2e060c
Content-Type: text/plain; charset=ISO-8859-1

Thanks, Giri.
--Konst

On Mon, Jan 31, 2011 at 4:40 PM, Giridharan Kesavan
<gkesavan@yahoo-inc.com>wrote:

> Konstantin,
>
> I think I need to restart the slave which is running the commit build. For
> now I have published the common artifact manually from commandline.
>
> Thanks,
> Giri
>
> On Jan 31, 2011, at 4:27 PM, Konstantin Shvachko wrote:
>
> > Giri
> > looks like the last run you started failed the same way as previous ones.
> > Any thoughts on what's going on?
> > Thanks,
> > --Konstantin
> >
> > On Mon, Jan 31, 2011 at 3:33 PM, Giridharan Kesavan
> > <gkesavan@yahoo-inc.com>wrote:
> >
> >> ant mvn-deploy would publish snapshot artifact to the apache maven
> >> repository as long you have the right credentials in ~/.m2/settings.xml.
> >>
> >> For settings.xml template pls look at
> >> http://wiki.apache.org/hadoop/HowToRelease
> >>
> >> I'm pushing the latest common artifacts now.
> >>
> >> -Giri
> >>
> >>
> >>
> >> On Jan 31, 2011, at 3:11 PM, Jakob Homan wrote:
> >>
> >>> By manually installing a new core jar into the cache, I can compile
> >>> trunk.  Looks like we just need to kick a new Core into maven.  Are
> >>> there instructions somewhere for committers to do this?  I know Nigel
> >>> and Owen know how, but I don't know if the knowledge is diffused past
> >>> them.
> >>> -Jakob
> >>>
> >>>
> >>> On Mon, Jan 31, 2011 at 1:57 PM, Konstantin Shvachko
> >>> <shv.hadoop@gmail.com> wrote:
> >>>> Current trunk for HDFS and MapReduce are not compiling at the moment.
> >> Try to
> >>>> build trunk.
> >>>> This is the result of that changes to common api introduced by
> >> HADOOP-6904
> >>>> are not promoted to HDFS and MR trunks.
> >>>> HDFS-1335 and MAPREDUCE-2263 depend on these changes.
> >>>>
> >>>> Common is not promoted to HDFS and MR because
> Hadoop-Common-trunk-Commit
> >>>> build is broken. See here.
> >>>>
> >>
> https://hudson.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/
> >>>>
> >>>> As I see the last successful build was on 01/19, which integrated
> >>>> HADOOP-6864.
> >>>> I think this is when JNI changes were introduced, which cannot be
> >> digested
> >>>> by Hudson since then.
> >>>>
> >>>> Anybody with gcc active could you please verify if the problem is
> caused
> >> by
> >>>> HADOOP-6864.
> >>>>
> >>>> Thanks,
> >>>> --Konstantin
> >>>>
> >>>> On Mon, Jan 31, 2011 at 1:36 PM, Ted Dunning <tdunning@maprtech.com>
> >> wrote:
> >>>>
> >>>>> The has been a problem with more than one build failing (Mahout is
> the
> >> one
> >>>>> that I saw first) due to a change in maven version which meant that
> the
> >>>>> clover license isn't being found properly.  At least, that is the
> tale
> >> I
> >>>>> heard from infra.
> >>>>>
> >>>>> On Mon, Jan 31, 2011 at 1:31 PM, Eli Collins <eli@cloudera.com>
> wrote:
> >>>>>
> >>>>>> Hey Konstantin,
> >>>>>>
> >>>>>> The only build breakage I saw from HADOOP-6904 is MAPREDUCE-2290,
> >>>>>> which was fixed.  Trees from trunk are compiling against each other
> >>>>>> for me (eg each installed to a local maven repo), perhaps the
> upstream
> >>>>>> maven repo hasn't been updated with the latest bits yet.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Eli
> >>>>>>
> >>>>>> On Mon, Jan 31, 2011 at 12:14 PM, Konstantin Shvachko
> >>>>>> <shv.hadoop@gmail.com> wrote:
> >>>>>>> Sending this to general to attract urgent attention.
> >>>>>>> Both HDFS and MapReduce are not compiling since
> >>>>>>> HADOOP-6904 and its hdfs and MP counterparts were committed.
> >>>>>>> The problem is not with this patch as described below, but I think
> >>>>> those
> >>>>>>> commits should be reversed if Common integration build cannot be
> >>>>>>> restored promptly.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> --Konstantin
> >>>>>>>
> >>>>>>>
> >>>>>>> On Fri, Jan 28, 2011 at 5:53 PM, Konstantin Shvachko
> >>>>>>> <shv.hadoop@gmail.com>wrote:
> >>>>>>>
> >>>>>>>> I see Hadoop-common-trunk-Commit is failing and not sending any
> >>>>> emails.
> >>>>>>>> It times out on native compilation and aborts.
> >>>>>>>> Therefore changes are not integrated, and now it lead to hdfs and
> >>>>>> mapreduce
> >>>>>>>> both not compiling.
> >>>>>>>> Can somebody please take a look at this.
> >>>>>>>> The last few lines of the build are below.
> >>>>>>>>
> >>>>>>>> Thanks
> >>>>>>>> --Konstantin
> >>>>>>>>
> >>>>>>>>    [javah] [Loaded
> >>>>>>
> >>>>>
> >>
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/build/classes/org/apache/hadoop/security/JniBasedUnixGroupsMapping.class]
> >>>>>>>>
> >>>>>>>>    [javah] [Loaded
> >>>>>>
> >>>>>
> >>
> /homes/hudson/tools/java/jdk1.6.0_11-32/jre/lib/rt.jar(java/lang/Object.class)]
> >>>>>>>>    [javah] [Forcefully writing file
> >>>>>>
> >>>>>
> >>
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/build/native/Linux-i386-32/src/org/apache/hadoop/security/org_apache_hadoop_security_JniBasedUnixGroupsNetgroupMapping.h]
> >>>>>>>>
> >>>>>>>>     [exec] checking for gcc... gcc
> >>>>>>>>     [exec] checking whether the C compiler works... yes
> >>>>>>>>     [exec] checking for C compiler default output file name...
> >> a.out
> >>>>>>>>     [exec] checking for suffix of executables...
> >>>>>>>>
> >>>>>>>> Build timed out. Aborting
> >>>>>>>> Build was aborted
> >>>>>>>> [FINDBUGS] Skipping publisher since build result is ABORTED
> >>>>>>>> Publishing Javadoc
> >>>>>>>> Archiving artifacts
> >>>>>>>> Recording test results
> >>>>>>>> No test report files were found. Configuration error?
> >>>>>>>>
> >>>>>>>> Recording fingerprints
> >>>>>>>>     [exec] Terminated
> >>>>>>>> Publishing Clover coverage report...
> >>>>>>>> No Clover report will be published due to a Build Failure
> >>>>>>>> No emails were triggered.
> >>>>>>>> Finished: ABORTED
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> >>
>
>

--90e6ba25e53b492bbd049b2e060c--

From general-return-2923-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 01 03:28:34 2011
Return-Path: <general-return-2923-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22010 invoked from network); 1 Feb 2011 03:28:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Feb 2011 03:28:34 -0000
Received: (qmail 22900 invoked by uid 500); 1 Feb 2011 03:28:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22406 invoked by uid 500); 1 Feb 2011 03:28:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22397 invoked by uid 99); 1 Feb 2011 03:28:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 03:28:28 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 03:28:22 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p113RmAM075650
	for <general@hadoop.apache.org>; Mon, 31 Jan 2011 19:27:48 -0800 (PST)
Received: from [10.0.1.4] (10.73.152.246) by SP2-EX07CAS01.ds.corp.yahoo.com
 (98.137.59.4) with Microsoft SMTP Server (TLS) id 8.3.137.0; Mon, 31 Jan 2011
 19:27:47 -0800
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: [ANNOUNCEMENT] Yahoo focusing on Apache Hadoop, discontinuing "The Yahoo Distribution of Hadoop"
Date: Mon, 31 Jan 2011 19:27:45 -0800
Message-ID: <55BA39F5-5017-42B1-BE25-A643CCEADC75@yahoo-inc.com>
To: <general@hadoop.apache.org>
MIME-Version: 1.0 (Apple Message framework v1082)
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Folks,

I'm pleased to announce that after some reflection, Yahoo! has decided =
to discontinue the  "The Yahoo Distribution of Hadoop" and focus on =
Apache Hadoop.  We plan to remove all references to a Yahoo distribution =
from our website (developer.yahoo.com/hadoop), close our github repo =
(yahoo.github.com/hadoop-common) and focus on working more closely with =
the Apache community.  Our intent is to return to helping Apache produce =
binary releases of Apache Hadoop that are so bullet proof that Yahoo and =
other production Hadoop users can run them unpatched on their clusters.

Until Hadoop 0.20, Yahoo committers worked as release masters to produce =
binary Apache Hadoop releases that the entire community used on their =
clusters.    As the community grew, we have experiment with using the =
"Yahoo! Distribution of Hadoop" as the vehicle to share our work.  =
Unfortunately, Apache is no longer the obvious place to go for Hadoop =
releases.  The Yahoo! team wants to return to a world where anyone can =
download and directly use releases of Hadoop from Apache.  We want to =
contribute to the stabilization and testing of those releases.  We also =
want to share our regular program of sustaining engineering that =
backports minor feature enhancements into new dot releases on a regular =
basis, so that the world sees regular improvements coming from Apache =
every few months, not years.

Recently the Apache Hadoop community has been very turbulent.  Over the =
last few months we have been developing Hadoop enhancements in our =
internal git repository while doing a complete review of our options. =
Our commitment to open sourcing our work was never in doubt (see =
http://yhoo.it/e8p3Dd), but the future of the "Yahoo distribution of =
Hadoop" was far from clear.  We've concluded that focusing on Apache =
Hadoop is the way forward.  We believe that more focus on communicating =
our goals to the Apache Hadoop community, and more willingness to =
compromise on how we get to those goals, will help us get back to making =
Hadoop even better.

Unfortunately, we now have to sort out how to contribute several =
person-years worth of work to Apache to let us unwind the Yahoo! git =
repositories.  We currently run two lines of Hadoop development, our =
sustaining program (hadoop-0.20-sustaining) and hadoop-future.  =
Hadoop-0.20-sustaining is the stable version of Hadoop we currently run =
on Yahoo's 40,000 nodes.  It contains a series of fixes and enhancements =
that are all backwards compatible with our "Hadoop 0.20 with security".  =
It is our most stable and high performance release of Hadoop ever.  =
We've expended a lot of energy finding and fixing bugs in it this year. =
We have initiated the process of contributing this work to Apache in the =
branch: hadoop/common/branches/branch-0.20-security.  We've proposed =
calling this the 20.100 release.  Once folks have had a chance to try =
this out and we've had a chance to respond to their feedback, we plan to =
create 20.100 release candidates and ask the community to vote on making =
them Apache releases.=20

Hadoop-future is our new feature branch.  We are working on a set of new =
features for Hadoop to improve its availability, scalability and =
interoperability to make Hadoop more usable in mission critical =
deployments. You're going to see another burst of email activity from us =
as we work to get hadoop-future patches socialized, reviewed and checked =
in.  These bulk checkins are exceptional.  They are the result of us =
striving to be more transparent.  Once we've merged our hadoop-future =
and hadoop-0.20-sustaining work back into Apache, folks can expect us to =
return to our regular development cadence.  Looking forward, we plan to =
socialize our roadmaps regularly, actively synchronize our work with =
other active Hadoop contributors and develop our code collaboratively, =
directly in Apache.

In summary, our decision to discontinue the "Yahoo! Distribution of =
Hadoop" is a commitment to working more effectively with the Apache =
Hadoop community.  Our goal is to make Apache Hadoop THE open source =
platform for big data.

Thanks,

E14

--

PS Here is a draft list of key features in hadoop-future:

* HDFS-1052 - Federation, the ability to support much more storage per =
Hadoop cluster.

* HADOOP-6728 - A the new metrics framework

* MAPREDUCE-1220 - Optimizations for small jobs

---
PPS This is cross-posted on our blog: http://yhoo.it/i9Ww8W=

From general-return-2924-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 01 03:44:34 2011
Return-Path: <general-return-2924-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 26996 invoked from network); 1 Feb 2011 03:44:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Feb 2011 03:44:34 -0000
Received: (qmail 33729 invoked by uid 500); 1 Feb 2011 03:44:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33285 invoked by uid 500); 1 Feb 2011 03:44:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33270 invoked by uid 99); 1 Feb 2011 03:44:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 03:44:28 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of hammer@cloudera.com designates 209.85.161.176 as permitted sender)
Received: from [209.85.161.176] (HELO mail-gx0-f176.google.com) (209.85.161.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 03:44:24 +0000
Received: by gxk4 with SMTP id 4so2532287gxk.35
        for <general@hadoop.apache.org>; Mon, 31 Jan 2011 19:44:03 -0800 (PST)
MIME-Version: 1.0
Received: by 10.150.57.20 with SMTP id f20mr1876178yba.446.1296531842749; Mon,
 31 Jan 2011 19:44:02 -0800 (PST)
Received: by 10.151.83.9 with HTTP; Mon, 31 Jan 2011 19:44:02 -0800 (PST)
In-Reply-To: <55BA39F5-5017-42B1-BE25-A643CCEADC75@yahoo-inc.com>
References: <55BA39F5-5017-42B1-BE25-A643CCEADC75@yahoo-inc.com>
Date: Mon, 31 Jan 2011 19:44:02 -0800
Message-ID: <AANLkTikW1OEoT5DSzYnpWyyuaraLZ_mjf6p7O_pRaqRW@mail.gmail.com>
Subject: Re: [ANNOUNCEMENT] Yahoo focusing on Apache Hadoop, discontinuing
 "The Yahoo Distribution of Hadoop"
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd307c6ed54e3049b3057f6

--000e0cd307c6ed54e3049b3057f6
Content-Type: text/plain; charset=ISO-8859-1

Excellent news! Will you also make Howl, Oozie, and Yarn Apache projects as
well?

On Mon, Jan 31, 2011 at 7:27 PM, Eric Baldeschwieler
<eric14@yahoo-inc.com>wrote:

> Hi Folks,
>
> I'm pleased to announce that after some reflection, Yahoo! has decided to
> discontinue the  "The Yahoo Distribution of Hadoop" and focus on Apache
> Hadoop.  We plan to remove all references to a Yahoo distribution from our
> website (developer.yahoo.com/hadoop), close our github repo (
> yahoo.github.com/hadoop-common) and focus on working more closely with the
> Apache community.  Our intent is to return to helping Apache produce binary
> releases of Apache Hadoop that are so bullet proof that Yahoo and other
> production Hadoop users can run them unpatched on their clusters.
>
> Until Hadoop 0.20, Yahoo committers worked as release masters to produce
> binary Apache Hadoop releases that the entire community used on their
> clusters.    As the community grew, we have experiment with using the
> "Yahoo! Distribution of Hadoop" as the vehicle to share our work.
>  Unfortunately, Apache is no longer the obvious place to go for Hadoop
> releases.  The Yahoo! team wants to return to a world where anyone can
> download and directly use releases of Hadoop from Apache.  We want to
> contribute to the stabilization and testing of those releases.  We also want
> to share our regular program of sustaining engineering that backports minor
> feature enhancements into new dot releases on a regular basis, so that the
> world sees regular improvements coming from Apache every few months, not
> years.
>
> Recently the Apache Hadoop community has been very turbulent.  Over the
> last few months we have been developing Hadoop enhancements in our internal
> git repository while doing a complete review of our options. Our commitment
> to open sourcing our work was never in doubt (see http://yhoo.it/e8p3Dd),
> but the future of the "Yahoo distribution of Hadoop" was far from clear.
>  We've concluded that focusing on Apache Hadoop is the way forward.  We
> believe that more focus on communicating our goals to the Apache Hadoop
> community, and more willingness to compromise on how we get to those goals,
> will help us get back to making Hadoop even better.
>
> Unfortunately, we now have to sort out how to contribute several
> person-years worth of work to Apache to let us unwind the Yahoo! git
> repositories.  We currently run two lines of Hadoop development, our
> sustaining program (hadoop-0.20-sustaining) and hadoop-future.
>  Hadoop-0.20-sustaining is the stable version of Hadoop we currently run on
> Yahoo's 40,000 nodes.  It contains a series of fixes and enhancements that
> are all backwards compatible with our "Hadoop 0.20 with security".  It is
> our most stable and high performance release of Hadoop ever.  We've expended
> a lot of energy finding and fixing bugs in it this year. We have initiated
> the process of contributing this work to Apache in the branch:
> hadoop/common/branches/branch-0.20-security.  We've proposed calling this
> the 20.100 release.  Once folks have had a chance to try this out and we've
> had a chance to respond to their feedback, we plan to create 20.100 release
> candidates and ask the community to vote on making them Apache releases.
>
> Hadoop-future is our new feature branch.  We are working on a set of new
> features for Hadoop to improve its availability, scalability and
> interoperability to make Hadoop more usable in mission critical deployments.
> You're going to see another burst of email activity from us as we work to
> get hadoop-future patches socialized, reviewed and checked in.  These bulk
> checkins are exceptional.  They are the result of us striving to be more
> transparent.  Once we've merged our hadoop-future and hadoop-0.20-sustaining
> work back into Apache, folks can expect us to return to our regular
> development cadence.  Looking forward, we plan to socialize our roadmaps
> regularly, actively synchronize our work with other active Hadoop
> contributors and develop our code collaboratively, directly in Apache.
>
> In summary, our decision to discontinue the "Yahoo! Distribution of Hadoop"
> is a commitment to working more effectively with the Apache Hadoop
> community.  Our goal is to make Apache Hadoop THE open source platform for
> big data.
>
> Thanks,
>
> E14
>
> --
>
> PS Here is a draft list of key features in hadoop-future:
>
> * HDFS-1052 - Federation, the ability to support much more storage per
> Hadoop cluster.
>
> * HADOOP-6728 - A the new metrics framework
>
> * MAPREDUCE-1220 - Optimizations for small jobs
>
> ---
> PPS This is cross-posted on our blog: http://yhoo.it/i9Ww8W

--000e0cd307c6ed54e3049b3057f6--

From general-return-2925-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 01 05:37:36 2011
Return-Path: <general-return-2925-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84253 invoked from network); 1 Feb 2011 05:37:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Feb 2011 05:37:35 -0000
Received: (qmail 14830 invoked by uid 500); 1 Feb 2011 05:37:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14348 invoked by uid 500); 1 Feb 2011 05:37:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14340 invoked by uid 99); 1 Feb 2011 05:37:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 05:37:30 +0000
X-ASF-Spam-Status: No, hits=3.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of akimball83@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 05:37:23 +0000
Received: by bwz8 with SMTP id 8so6739279bwz.35
        for <general@hadoop.apache.org>; Mon, 31 Jan 2011 21:37:02 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type;
        bh=Fk1dd4RHSmAiGwYYf5UlbUexJBaQpdJPHdJTobfJfZQ=;
        b=ZWmeV1R3bNU8nLEIGNuJV8djp1CjFuATVI0XpeLamqdEG7kqXSo/tQWS2Aln6H134M
         mraPkp2hUL/V6rfkXNP+05gKI7Dscy3PO+bxX+LJOfyvsU5BTuuB6ed1AnhTpHYa39t8
         ZM01MpBtv+uxriEEZIMlka6r+1r5Mff8fonT8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=sVRQNQCjZ8EFOwJe/psIhuI0tKNZKimdGRL9tUkFRP4RNvOe1aODfz0wKNY2GgCyYi
         nq+tdFQh/A8F7Zgze9Pga8B70ms6A+J1tTuSXzt98lmMxYEWujXq7G0yLTCN/eslp3No
         5SkS6f7x/XD/wYBZZ/YQrUaoRloLiMvl3tP/s=
Received: by 10.204.117.77 with SMTP id p13mr6302927bkq.19.1296538622086; Mon,
 31 Jan 2011 21:37:02 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.54.72 with HTTP; Mon, 31 Jan 2011 21:36:41 -0800 (PST)
In-Reply-To: <5E01B48E-BFE9-45BA-B729-50E692B74F46@mac.com>
References: <AANLkTi=fOv-5sKrP9jMDC=mJuQjZgh+Tz4LWyDjYiwic@mail.gmail.com>
 <AANLkTi=9kZgxwRWyBcDJcjno5WY-uT7gQx=FJ9jF=3=M@mail.gmail.com> <5E01B48E-BFE9-45BA-B729-50E692B74F46@mac.com>
From: Aaron Kimball <akimball83@gmail.com>
Date: Mon, 31 Jan 2011 21:36:41 -0800
Message-ID: <AANLkTi=zdtxxJveKeKjNqXooCWTfetFOB55Qzttm2_bE@mail.gmail.com>
Subject: Re: MRUnit
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d647db01cf22049b31eca9

--0016e6d647db01cf22049b31eca9
Content-Type: text/plain; charset=ISO-8859-1

+1 - thanks for taking the initiative to clean this up!

You should file a JIRA with a patch that removes src/contrib/mrunit and
removes it from src/contrib/build.xml.

- Aaron




On Sun, Jan 30, 2011 at 7:59 PM, Nigel Daley <ndaley@mac.com> wrote:

> +1.  I just started a thread on moving all components out of contrib.
>
> nige
>
> On Jan 30, 2011, at 6:50 PM, Jeff Hammerbacher wrote:
>
> >>
> >> I think it makes sense to remove mrunit from contrib (after a
> >> deprecation period) but I'm curious to hear others' opinions.
> >
> >
> > Oh please do take things out of contrib. I'd love to see all of it move
> to
> > Github. Thanks for putting in the time for MRUnit.
>
>

--0016e6d647db01cf22049b31eca9--

From general-return-2926-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 01 05:42:39 2011
Return-Path: <general-return-2926-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88962 invoked from network); 1 Feb 2011 05:42:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Feb 2011 05:42:38 -0000
Received: (qmail 16325 invoked by uid 500); 1 Feb 2011 05:42:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16120 invoked by uid 500); 1 Feb 2011 05:42:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16110 invoked by uid 99); 1 Feb 2011 05:42:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 05:42:34 +0000
X-ASF-Spam-Status: No, hits=3.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of akimball83@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 05:42:29 +0000
Received: by bwz8 with SMTP id 8so6741486bwz.35
        for <general@hadoop.apache.org>; Mon, 31 Jan 2011 21:42:07 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type;
        bh=P/GBWX/OGYVgWqEw3AEGi0c/7Es2H9hQlnAmSwxMLH0=;
        b=LBTA4U/hg3yYqNBEZMz8xCrvMVYLYruJdaGqz66b1RwpAlOesj3CnL1r1d5xpLHAvx
         Xj5YmLI/H2fG1nsoy80nGY/KHvYJVPvZrpXyoeZRmpvE60HPDhOzDzLVlkZYoUf/axhm
         jy7uG/VZTvE2ypvMPUy+2Svw/d1ZcglSWnZaM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=EoS/8UKnZg+G8S5uUeLm1lBHUIksZXqyAQtvgoKE3Yf6xgFoKJqz39RqIelfkfz2uW
         w9tZas24WlyZMuJKvqrRZjY4TFS2EQd82KFUAoL1e5KcuEWi2opQTbwF9piZ3c4jfkFr
         MhgoJjBh9MKGAijNWw3HxIxpOeaxBk569DtBk=
Received: by 10.204.84.32 with SMTP id h32mr6294281bkl.181.1296538927072; Mon,
 31 Jan 2011 21:42:07 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.54.72 with HTTP; Mon, 31 Jan 2011 21:41:46 -0800 (PST)
In-Reply-To: <AANLkTikYHbb935h5Qo+YbsYyrAYPSYYoyjrL5APy_unC@mail.gmail.com>
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com> <FD5BC1D3-881A-4D45-BA2C-C6084C884418@apache.org>
 <AANLkTikYHbb935h5Qo+YbsYyrAYPSYYoyjrL5APy_unC@mail.gmail.com>
From: Aaron Kimball <akimball83@gmail.com>
Date: Mon, 31 Jan 2011 21:41:46 -0800
Message-ID: <AANLkTin3T9Yh5pUN4QNvjkFdt37Z4vAuTUdC8eJUN+ZU@mail.gmail.com>
Subject: Re: [DISCUSS] Move common, hdfs, mapreduce contrib components to
 apache-extras.org or elsewhere
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d7df5d2f869a049b31fe15

--0016e6d7df5d2f869a049b31fe15
Content-Type: text/plain; charset=ISO-8859-1

+1 to this process in general.

In particular, tools like MRUnit can benefit from having an independent
release due to where they are used in a project's lifecycle. MRUnit should
be specified as a test dependency, whereas Hadoop itself is a
compile/runtime dependency.  As it stands, there isn't an easy way to manage
this. This would increase flexibility for this tool, probably for others as
well.

- Aaron

On Mon, Jan 31, 2011 at 3:23 PM, Todd Lipcon <todd@cloudera.com> wrote:

> On Sun, Jan 30, 2011 at 11:19 PM, Owen O'Malley <omalley@apache.org>
> wrote:
>
> >
> > Also note that pushing code out of Hadoop has a high cost. There are at
> > least 3 forks of the hadoop-gpl-compression code. That creates a lot of
> > confusion for the users. A lot of users never go to the work to figure
> out
> > which fork and branch of hadoop-gpl-compression work with the version of
> > Hadoop they installed.
> >
> >
> Indeed it creates confusion, but in my opinion it has been very successful
> modulo that confusion.
>
> In particular, Kevin and I (who each have a repo on github but basically
> co-maintain a branch) have done about 8 bugfix releases of LZO in the last
> year. The ability to take a bug and turn it around into a release within a
> few days has been very beneficial to the users. If it were part of core
> Hadoop, people would be forced to live with these blocker bugs for months
> at
> a time between dot releases.
>
> IMO the more we can take non-core components and move them to separate
> release timelines, the better. Yes, it is harder for users, but it also is
> easier for them when they hit a bug - they don't have to wait months for a
> wholesale upgrade which might contain hundreds of other changes to core
> components. I think this will also help the situation where people have set
> up shop on branches -- a lot of the value of these branches comes from the
> frequency of backports and bugfixes to "non-core" components. If the
> non-core stuff were on a faster timeline upstream, we could maintain core
> stability while also offering people the latest and greatest libraries,
> tools, codecs, etc.
>
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

--0016e6d7df5d2f869a049b31fe15--

From general-return-2927-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 01 08:41:27 2011
Return-Path: <general-return-2927-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60459 invoked from network); 1 Feb 2011 08:41:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Feb 2011 08:41:27 -0000
Received: (qmail 63834 invoked by uid 500); 1 Feb 2011 08:41:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63488 invoked by uid 500); 1 Feb 2011 08:41:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63480 invoked by uid 99); 1 Feb 2011 08:41:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 08:41:22 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of shv.hadoop@gmail.com designates 209.85.218.48 as permitted sender)
Received: from [209.85.218.48] (HELO mail-yi0-f48.google.com) (209.85.218.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 08:41:15 +0000
Received: by yib17 with SMTP id 17so2592407yib.35
        for <general@hadoop.apache.org>; Tue, 01 Feb 2011 00:40:54 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=9dHFuPsOIcZT68Yl587Q+j6GhEJXr1Csct8J0qRp4QA=;
        b=OsCfTFLVOwWRaBf9BmatLJBVrSonhnuRf+3yIk5SQbACAnjANNiLFwc6GJTs75gKBb
         mb3ceqYBdUsy7aH/Cy8IPFNLiJI1KrOHYFAWLXBb4or/PhSc1tmNtU1Kqy30KXhgl04r
         uDvmgQWaN7WfgbSc5FnJPOPT5OsWrdrwtbevg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=A/1KaPcQJ+4f3AqKefTOklxHA3Ud1APCfbrkYPE75ljhOKGngapFgmis008uD5QXgI
         UIXRVZKLAD+dwaKx7hbzvZFy/lo6kaRb7nWDN7MQd9gcfDY80CeOxuOQ+BpGJ116uEni
         oJNQoRqFd7vCEa+nK2Jb/n0Gc1cpe2vzJd/J8=
MIME-Version: 1.0
Received: by 10.236.109.168 with SMTP id s28mr14991203yhg.74.1296549654483;
 Tue, 01 Feb 2011 00:40:54 -0800 (PST)
Received: by 10.236.108.35 with HTTP; Tue, 1 Feb 2011 00:40:54 -0800 (PST)
In-Reply-To: <F3847AE0-3FEF-4E85-8757-FA8A7B204A64@yahoo-inc.com>
References: <AANLkTinB+8KRByZFhRT-y6MiqA0urtKNs8c_XSrhP7RH@mail.gmail.com>
	<AANLkTimmcYh0hY-6ht6vaQ+S=pTGHaFxS6k9qEv=q8Cf@mail.gmail.com>
	<AANLkTinr_Exrr8VPShGKkBSMzmWX1B9nVALLLTMmvYzd@mail.gmail.com>
	<AANLkTinrheVYQm-y9n=k7TRK_BdzuZYKAWG+qMWR88ct@mail.gmail.com>
	<AANLkTi=cAhCOkqZ=n9NpBMoPzechoJmfB6KO2yL3QcVA@mail.gmail.com>
	<AANLkTi=+81g8dFMGMqP5Zc6aAkg5n0dD2QpCVg7ABzd6@mail.gmail.com>
	<7DC70A70-5DE9-4288-A44B-B5EF9C454D9B@yahoo-inc.com>
	<AANLkTin9fsb3bgsa98bma+xx_C9K06C_YNKJ6vaA7=TO@mail.gmail.com>
	<F3847AE0-3FEF-4E85-8757-FA8A7B204A64@yahoo-inc.com>
Date: Tue, 1 Feb 2011 00:40:54 -0800
Message-ID: <AANLkTikCOHYJx4Qjt=gajGTcspPai0GmsxWQFogua92Q@mail.gmail.com>
Subject: Re: Hadoop-common-trunk-Commit is failing since 01/19/2011
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0023547c8b4396d647049b347d88
X-Virus-Checked: Checked by ClamAV on apache.org

--0023547c8b4396d647049b347d88
Content-Type: text/plain; charset=ISO-8859-1

Giri,

Looking at configuration of Hadoop-Common-trunk-Commit/
There seems to be errors in the Post-build Actions.
It is complaining that
'trunk' exists but not 'trunk/artifacts/...'
Is it possible that this misconfiguration is the reason of failures?

--Konstantin


On Mon, Jan 31, 2011 at 4:40 PM, Giridharan Kesavan
<gkesavan@yahoo-inc.com>wrote:

> Konstantin,
>
> I think I need to restart the slave which is running the commit build. For
> now I have published the common artifact manually from commandline.
>
> Thanks,
> Giri
>
> On Jan 31, 2011, at 4:27 PM, Konstantin Shvachko wrote:
>
> > Giri
> > looks like the last run you started failed the same way as previous ones.
> > Any thoughts on what's going on?
> > Thanks,
> > --Konstantin
> >
> > On Mon, Jan 31, 2011 at 3:33 PM, Giridharan Kesavan
> > <gkesavan@yahoo-inc.com>wrote:
> >
> >> ant mvn-deploy would publish snapshot artifact to the apache maven
> >> repository as long you have the right credentials in ~/.m2/settings.xml.
> >>
> >> For settings.xml template pls look at
> >> http://wiki.apache.org/hadoop/HowToRelease
> >>
> >> I'm pushing the latest common artifacts now.
> >>
> >> -Giri
> >>
> >>
> >>
> >> On Jan 31, 2011, at 3:11 PM, Jakob Homan wrote:
> >>
> >>> By manually installing a new core jar into the cache, I can compile
> >>> trunk.  Looks like we just need to kick a new Core into maven.  Are
> >>> there instructions somewhere for committers to do this?  I know Nigel
> >>> and Owen know how, but I don't know if the knowledge is diffused past
> >>> them.
> >>> -Jakob
> >>>
> >>>
> >>> On Mon, Jan 31, 2011 at 1:57 PM, Konstantin Shvachko
> >>> <shv.hadoop@gmail.com> wrote:
> >>>> Current trunk for HDFS and MapReduce are not compiling at the moment.
> >> Try to
> >>>> build trunk.
> >>>> This is the result of that changes to common api introduced by
> >> HADOOP-6904
> >>>> are not promoted to HDFS and MR trunks.
> >>>> HDFS-1335 and MAPREDUCE-2263 depend on these changes.
> >>>>
> >>>> Common is not promoted to HDFS and MR because
> Hadoop-Common-trunk-Commit
> >>>> build is broken. See here.
> >>>>
> >>
> https://hudson.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/
> >>>>
> >>>> As I see the last successful build was on 01/19, which integrated
> >>>> HADOOP-6864.
> >>>> I think this is when JNI changes were introduced, which cannot be
> >> digested
> >>>> by Hudson since then.
> >>>>
> >>>> Anybody with gcc active could you please verify if the problem is
> caused
> >> by
> >>>> HADOOP-6864.
> >>>>
> >>>> Thanks,
> >>>> --Konstantin
> >>>>
> >>>> On Mon, Jan 31, 2011 at 1:36 PM, Ted Dunning <tdunning@maprtech.com>
> >> wrote:
> >>>>
> >>>>> The has been a problem with more than one build failing (Mahout is
> the
> >> one
> >>>>> that I saw first) due to a change in maven version which meant that
> the
> >>>>> clover license isn't being found properly.  At least, that is the
> tale
> >> I
> >>>>> heard from infra.
> >>>>>
> >>>>> On Mon, Jan 31, 2011 at 1:31 PM, Eli Collins <eli@cloudera.com>
> wrote:
> >>>>>
> >>>>>> Hey Konstantin,
> >>>>>>
> >>>>>> The only build breakage I saw from HADOOP-6904 is MAPREDUCE-2290,
> >>>>>> which was fixed.  Trees from trunk are compiling against each other
> >>>>>> for me (eg each installed to a local maven repo), perhaps the
> upstream
> >>>>>> maven repo hasn't been updated with the latest bits yet.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Eli
> >>>>>>
> >>>>>> On Mon, Jan 31, 2011 at 12:14 PM, Konstantin Shvachko
> >>>>>> <shv.hadoop@gmail.com> wrote:
> >>>>>>> Sending this to general to attract urgent attention.
> >>>>>>> Both HDFS and MapReduce are not compiling since
> >>>>>>> HADOOP-6904 and its hdfs and MP counterparts were committed.
> >>>>>>> The problem is not with this patch as described below, but I think
> >>>>> those
> >>>>>>> commits should be reversed if Common integration build cannot be
> >>>>>>> restored promptly.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> --Konstantin
> >>>>>>>
> >>>>>>>
> >>>>>>> On Fri, Jan 28, 2011 at 5:53 PM, Konstantin Shvachko
> >>>>>>> <shv.hadoop@gmail.com>wrote:
> >>>>>>>
> >>>>>>>> I see Hadoop-common-trunk-Commit is failing and not sending any
> >>>>> emails.
> >>>>>>>> It times out on native compilation and aborts.
> >>>>>>>> Therefore changes are not integrated, and now it lead to hdfs and
> >>>>>> mapreduce
> >>>>>>>> both not compiling.
> >>>>>>>> Can somebody please take a look at this.
> >>>>>>>> The last few lines of the build are below.
> >>>>>>>>
> >>>>>>>> Thanks
> >>>>>>>> --Konstantin
> >>>>>>>>
> >>>>>>>>    [javah] [Loaded
> >>>>>>
> >>>>>
> >>
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/build/classes/org/apache/hadoop/security/JniBasedUnixGroupsMapping.class]
> >>>>>>>>
> >>>>>>>>    [javah] [Loaded
> >>>>>>
> >>>>>
> >>
> /homes/hudson/tools/java/jdk1.6.0_11-32/jre/lib/rt.jar(java/lang/Object.class)]
> >>>>>>>>    [javah] [Forcefully writing file
> >>>>>>
> >>>>>
> >>
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/build/native/Linux-i386-32/src/org/apache/hadoop/security/org_apache_hadoop_security_JniBasedUnixGroupsNetgroupMapping.h]
> >>>>>>>>
> >>>>>>>>     [exec] checking for gcc... gcc
> >>>>>>>>     [exec] checking whether the C compiler works... yes
> >>>>>>>>     [exec] checking for C compiler default output file name...
> >> a.out
> >>>>>>>>     [exec] checking for suffix of executables...
> >>>>>>>>
> >>>>>>>> Build timed out. Aborting
> >>>>>>>> Build was aborted
> >>>>>>>> [FINDBUGS] Skipping publisher since build result is ABORTED
> >>>>>>>> Publishing Javadoc
> >>>>>>>> Archiving artifacts
> >>>>>>>> Recording test results
> >>>>>>>> No test report files were found. Configuration error?
> >>>>>>>>
> >>>>>>>> Recording fingerprints
> >>>>>>>>     [exec] Terminated
> >>>>>>>> Publishing Clover coverage report...
> >>>>>>>> No Clover report will be published due to a Build Failure
> >>>>>>>> No emails were triggered.
> >>>>>>>> Finished: ABORTED
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> >>
>
>

--0023547c8b4396d647049b347d88--

From general-return-2928-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 01 09:01:48 2011
Return-Path: <general-return-2928-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 65540 invoked from network); 1 Feb 2011 09:01:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Feb 2011 09:01:48 -0000
Received: (qmail 85652 invoked by uid 500); 1 Feb 2011 09:01:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85262 invoked by uid 500); 1 Feb 2011 09:01:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85254 invoked by uid 99); 1 Feb 2011 09:01:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 09:01:42 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 09:01:37 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=YbTTAQxtLaRem/4VmZulRSeCgGpm3gspfZyOjse++LZvDGZeqaJ4nft9
   mLBUgybtpO7vGo0aAXkyZ/R/A0nnufPdPCsfzozQxXqPNhZ/cH+inwfV9
   16NWxAJVSgaWfSw;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1296550895; x=1328086895;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20[DISCUSS]=20Move=20common,=20hdfs,=20ma
   preduce=20contrib=20components=20to=0D=0A=20apache-extras
   .org=20or=20elsewhere|Date:=20Tue,=201=20Feb=202011=2009:
   02:26=20+0000|Message-ID:=20<A9EDE11D-0CE9-4D04-A821-4DDE
   E685055A@linkedin.com>|To:=20"<general@hadoop.apache.org>
   "=20<general@hadoop.apache.org>|MIME-Version:=201.0
   |Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<A3DC37BBEE34924AAD8EB63776A79EFC@linkedin
   .com>|In-Reply-To:=20<AANLkTikYHbb935h5Qo+YbsYyrAYPSYYoyj
   rL5APy_unC@mail.gmail.com>|References:=20<FBD8F805-4345-4
   51E-AEA3-F19A2545C387@mac.com>=0D=0A=20<FD5BC1D3-881A-4D4
   5-BA2C-C6084C884418@apache.org>=0D=0A=20<AANLkTikYHbb935h
   5Qo+YbsYyrAYPSYYoyjrL5APy_unC@mail.gmail.com>;
  bh=+r1zeD3otkmm9dy7ZBv2AXeM0MKaW1CmgeNnibBPH+w=;
  b=kDIn/QwQEXeU2PVamhS7AUfFcLy6muenQwV+WZRhKYq9ZaPC1DrG4dPm
   mLWgtKZ0pfZJbpqR6qobOzkLPmBvPeho420G0J7/kA9mrOnAvU5BvynBr
   pzy0c/NHv7tvwph;
X-IronPort-AV: E=Sophos;i="4.60,409,1291622400"; 
   d="scan'208";a="19688688"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi id 14.01.0251.000; Tue, 1
 Feb 2011 01:02:27 -0800
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: [DISCUSS] Move common, hdfs, mapreduce contrib components to
 apache-extras.org or elsewhere
Thread-Topic: [DISCUSS] Move common, hdfs, mapreduce contrib components to
 apache-extras.org or elsewhere
Thread-Index: AQHLwPkf34Q5Zjm8NkummeRbbY5xTJPrMnYAgAENWQCAAKGEgA==
Date: Tue, 1 Feb 2011 09:02:26 +0000
Message-ID: <A9EDE11D-0CE9-4D04-A821-4DDEE685055A@linkedin.com>
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com>
 <FD5BC1D3-881A-4D45-BA2C-C6084C884418@apache.org>
 <AANLkTikYHbb935h5Qo+YbsYyrAYPSYYoyjrL5APy_unC@mail.gmail.com>
In-Reply-To: <AANLkTikYHbb935h5Qo+YbsYyrAYPSYYoyjrL5APy_unC@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <A3DC37BBEE34924AAD8EB63776A79EFC@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


On Jan 31, 2011, at 3:23 PM, Todd Lipcon wrote:

> On Sun, Jan 30, 2011 at 11:19 PM, Owen O'Malley <omalley@apache.org> wrot=
e:
>=20
>>=20
>> Also note that pushing code out of Hadoop has a high cost. There are at
>> least 3 forks of the hadoop-gpl-compression code. That creates a lot of
>> confusion for the users. A lot of users never go to the work to figure o=
ut
>> which fork and branch of hadoop-gpl-compression work with the version of
>> Hadoop they installed.
>>=20
>>=20
> Indeed it creates confusion, but in my opinion it has been very successfu=
l
> modulo that confusion.

	I'm not sure how the above works with what you wrote below:

> In particular, Kevin and I (who each have a repo on github but basically
> co-maintain a branch) have done about 8 bugfix releases of LZO in the las=
t
> year. The ability to take a bug and turn it around into a release within =
a
> few days has been very beneficial to the users. If it were part of core
> Hadoop, people would be forced to live with these blocker bugs for months=
 at
> a time between dot releases.

	So is the expectation that users would have to follow bread crumbs to the =
github dumping ground, then try to figure out which repo is the 'better' ch=
oice for their usage?   Using LZO as an example, it appears we have a choic=
e of kevin's, your's, or the master without even taking into consideration =
any tags. That sounds like a recipe for disaster that's even worse than wha=
t we have today.


> IMO the more we can take non-core components and move them to separate
> release timelines, the better. Yes, it is harder for users, but it also i=
s
> easier for them when they hit a bug - they don't have to wait months for =
a
> wholesale upgrade which might contain hundreds of other changes to core
> components.

	I'd agree except for one thing:  even when users do provide patches to con=
trib components we ignore them.  How long have those patches for HOD been s=
itting there in the patch queue?  So of course they wait months/years--beca=
use we seemingly ignore anything that isn't important to us.  Unfortunately=
, that covers a large chunk of contrib. :(



From general-return-2929-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 01 09:15:15 2011
Return-Path: <general-return-2929-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77804 invoked from network); 1 Feb 2011 09:15:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Feb 2011 09:15:15 -0000
Received: (qmail 4919 invoked by uid 500); 1 Feb 2011 09:15:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4529 invoked by uid 500); 1 Feb 2011 09:15:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4521 invoked by uid 99); 1 Feb 2011 09:15:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 09:15:09 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [85.128.214.242] (HELO magdalenammc.nazwa.pl) (85.128.214.242)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 09:15:01 +0000
Received: from Pavilionl (unknown [62.233.208.194])
	by magdalenammc.nazwa.pl (Postfix) with ESMTP id 6A00A30CA4F6
	for <general@hadoop.apache.org>; Tue,  1 Feb 2011 10:14:40 +0100 (CET)
From: =?iso-8859-2?Q?Magdalena_Moll-Musia=B3?= <office@mm-consulting.com.pl>
To: <general@hadoop.apache.org>
Subject: Job opportunities - 2 roles with Hadoop
Date: Tue, 1 Feb 2011 10:14:39 +0100
Message-ID: <003d01cbc1f0$7436eea0$5ca4cbe0$@com.pl>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_NextPart_000_003E_01CBC1F8.D5FB56A0"
X-Mailer: Microsoft Office Outlook 12.0
Thread-Index: AcvB8HQDJHO3GhBXRj2MhndJcoGyRQ==
Content-Language: pl
X-Virus-Checked: Checked by ClamAV on apache.org

------=_NextPart_000_003E_01CBC1F8.D5FB56A0
Content-Type: text/plain;
	charset="iso-8859-2"
Content-Transfer-Encoding: quoted-printable

Dear All,

=20

I am looking for experienced candidates to join Nokia in Berlin.

=20

Please follow the links to read the descriptions of the roles:

=20
<http://www.mm-consulting.com.pl/senior-software-engineer---machine-learn=
ing
.html>
http://www.mm-consulting.com.pl/senior-software-engineer---machine-learni=
ng.
html

http://www.mm-consulting.com.pl/analitics.html

=20

Your CVs are more than welcome J=20

Pease provide me with your CV in English and please suggest when I could
call you to discuss the opportunity (15 minutes conversation).

=20

In case of questions, please let me know.

=20

Best regards,

Magda=20

=20

______________________________________

mmConsulting Magdalena Moll-Musia=B3

recruitment partner=20

+48 516 340 127

 <mailto:office@mm-consulting.com.pl> office@mm-consulting.com.pl

 <http://www.mm-consulting.com.pl/> www.mm-consulting.com.pl

=20


------=_NextPart_000_003E_01CBC1F8.D5FB56A0--


From general-return-2930-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 01 14:04:51 2011
Return-Path: <general-return-2930-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 51888 invoked from network); 1 Feb 2011 14:04:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Feb 2011 14:04:50 -0000
Received: (qmail 15979 invoked by uid 500); 1 Feb 2011 14:04:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15539 invoked by uid 500); 1 Feb 2011 14:04:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15531 invoked by uid 99); 1 Feb 2011 14:04:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 14:04:43 +0000
X-ASF-Spam-Status: No, hits=3.5 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL,URIBL_BLACK
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 14:04:38 +0000
Received: by qwe4 with SMTP id 4so6935947qwe.35
        for <general@hadoop.apache.org>; Tue, 01 Feb 2011 06:04:17 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=mMpmDxE3BDd70BDsoNLHZRtdbNJTzqxhMfUqDe0K01o=;
        b=Ig/HW01BmxYbAf+N2fC+stTdnAeXZHHT3GfZ8zPT5MLPkqlf21FB3xvTYPPcigF24q
         ag6nJNsrYhfPonaU0TRl63K3cLIkrF+OmRxBAGSZsYNkLWCErtTfEr/Wo6P7SCpJ/aBA
         HaQZ492tcbSVUGkVCMmqGhSNKg7wFXNMIjQA4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=TBaku7Fq0bEQf0AL+GIggHGyF3DISikm3fbLgDzlx1fQHYFdacyZViLBej9iroWIrs
         pqlNacPXkrZdkQrfjwfn/MpHj0N6ZclW7hYfUvhm6G/vbGra/rfMAlVH7ByDPt9KxMxe
         C7zjS4gy8dZ389Sq72NcwKRFwQMLVmxwYcl3c=
Received: by 10.229.235.202 with SMTP id kh10mr5429976qcb.194.1296569057426;
        Tue, 01 Feb 2011 06:04:17 -0800 (PST)
Received: from [10.180.150.9] (h-64-236-128-41.nat.aol.com [64.236.128.41])
        by mx.google.com with ESMTPS id p13sm15615108qcu.17.2011.02.01.06.04.15
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Tue, 01 Feb 2011 06:04:16 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [ANNOUNCEMENT] Yahoo focusing on Apache Hadoop, discontinuing "The Yahoo Distribution of Hadoop"
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <55BA39F5-5017-42B1-BE25-A643CCEADC75@yahoo-inc.com>
Date: Tue, 1 Feb 2011 09:04:15 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <0C3025FB-DDCF-4572-B90B-6AD07A134293@Holsman.NET>
References: <55BA39F5-5017-42B1-BE25-A643CCEADC75@yahoo-inc.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

Congratulations Eric.
this is fantastic news.
On Jan 31, 2011, at 10:27 PM, Eric Baldeschwieler wrote:

> Hi Folks,
>=20
> I'm pleased to announce that after some reflection, Yahoo! has decided =
to discontinue the  "The Yahoo Distribution of Hadoop" and focus on =
Apache Hadoop.  We plan to remove all references to a Yahoo distribution =
from our website (developer.yahoo.com/hadoop), close our github repo =
(yahoo.github.com/hadoop-common) and focus on working more closely with =
the Apache community.  Our intent is to return to helping Apache produce =
binary releases of Apache Hadoop that are so bullet proof that Yahoo and =
other production Hadoop users can run them unpatched on their clusters.
>=20
> Until Hadoop 0.20, Yahoo committers worked as release masters to =
produce binary Apache Hadoop releases that the entire community used on =
their clusters.    As the community grew, we have experiment with using =
the "Yahoo! Distribution of Hadoop" as the vehicle to share our work.  =
Unfortunately, Apache is no longer the obvious place to go for Hadoop =
releases.  The Yahoo! team wants to return to a world where anyone can =
download and directly use releases of Hadoop from Apache.  We want to =
contribute to the stabilization and testing of those releases.  We also =
want to share our regular program of sustaining engineering that =
backports minor feature enhancements into new dot releases on a regular =
basis, so that the world sees regular improvements coming from Apache =
every few months, not years.
>=20
> Recently the Apache Hadoop community has been very turbulent.  Over =
the last few months we have been developing Hadoop enhancements in our =
internal git repository while doing a complete review of our options. =
Our commitment to open sourcing our work was never in doubt (see =
http://yhoo.it/e8p3Dd), but the future of the "Yahoo distribution of =
Hadoop" was far from clear.  We've concluded that focusing on Apache =
Hadoop is the way forward.  We believe that more focus on communicating =
our goals to the Apache Hadoop community, and more willingness to =
compromise on how we get to those goals, will help us get back to making =
Hadoop even better.
>=20
> Unfortunately, we now have to sort out how to contribute several =
person-years worth of work to Apache to let us unwind the Yahoo! git =
repositories.  We currently run two lines of Hadoop development, our =
sustaining program (hadoop-0.20-sustaining) and hadoop-future.  =
Hadoop-0.20-sustaining is the stable version of Hadoop we currently run =
on Yahoo's 40,000 nodes.  It contains a series of fixes and enhancements =
that are all backwards compatible with our "Hadoop 0.20 with security".  =
It is our most stable and high performance release of Hadoop ever.  =
We've expended a lot of energy finding and fixing bugs in it this year. =
We have initiated the process of contributing this work to Apache in the =
branch: hadoop/common/branches/branch-0.20-security.  We've proposed =
calling this the 20.100 release.  Once folks have had a chance to try =
this out and we've had a chance to respond to their feedback, we plan to =
create 20.100 release candidates and ask the community to vote on making =
them Apache releases.=20
>=20
> Hadoop-future is our new feature branch.  We are working on a set of =
new features for Hadoop to improve its availability, scalability and =
interoperability to make Hadoop more usable in mission critical =
deployments. You're going to see another burst of email activity from us =
as we work to get hadoop-future patches socialized, reviewed and checked =
in.  These bulk checkins are exceptional.  They are the result of us =
striving to be more transparent.  Once we've merged our hadoop-future =
and hadoop-0.20-sustaining work back into Apache, folks can expect us to =
return to our regular development cadence.  Looking forward, we plan to =
socialize our roadmaps regularly, actively synchronize our work with =
other active Hadoop contributors and develop our code collaboratively, =
directly in Apache.
>=20
> In summary, our decision to discontinue the "Yahoo! Distribution of =
Hadoop" is a commitment to working more effectively with the Apache =
Hadoop community.  Our goal is to make Apache Hadoop THE open source =
platform for big data.
>=20
> Thanks,
>=20
> E14
>=20
> --
>=20
> PS Here is a draft list of key features in hadoop-future:
>=20
> * HDFS-1052 - Federation, the ability to support much more storage per =
Hadoop cluster.
>=20
> * HADOOP-6728 - A the new metrics framework
>=20
> * MAPREDUCE-1220 - Optimizations for small jobs
>=20
> ---
> PPS This is cross-posted on our blog: http://yhoo.it/i9Ww8W


From general-return-2931-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 01 16:07:30 2011
Return-Path: <general-return-2931-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38498 invoked from network); 1 Feb 2011 16:07:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Feb 2011 16:07:30 -0000
Received: (qmail 32460 invoked by uid 500); 1 Feb 2011 16:07:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32019 invoked by uid 500); 1 Feb 2011 16:07:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32011 invoked by uid 99); 1 Feb 2011 16:07:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 16:07:23 +0000
X-ASF-Spam-Status: No, hits=4.2 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL,URIBL_BLACK
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 16:07:17 +0000
Received: from [192.168.0.199] (snvvpn2-10-73-152-c156.hq.corp.yahoo.com [10.73.152.156])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p11G6M4R084229
	for <general@hadoop.apache.org>; Tue, 1 Feb 2011 08:06:23 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1296576390;
	bh=kEKjDdHb/78L6bwya9E00YaPq/pBIxGokz52rkQ/Ey0=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=sZYnuaQtQhO9Cy2/zQ712N7ImqducHMldTcf+sLCELYhfNmaqe+4O0msluS42UoBR
	 zYl8uhFfJwAn3GY34rrAs74m5lIKxU0Iny45oY6aY/ww3Z1S3aYBSfE3VrCOcca2XE
	 xcBB2fhX+t+WTufIT07ZdRJ4aWNLIqyLpjoyetPo=
Message-Id: <F3BFC5F7-19FC-4446-9C70-D30B8CC3B9F3@yahoo-inc.com>
From: Alan Gates <gates@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <AANLkTikW1OEoT5DSzYnpWyyuaraLZ_mjf6p7O_pRaqRW@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [ANNOUNCEMENT] Yahoo focusing on Apache Hadoop, discontinuing "The Yahoo Distribution of Hadoop"
Date: Tue, 1 Feb 2011 08:06:22 -0800
References: <55BA39F5-5017-42B1-BE25-A643CCEADC75@yahoo-inc.com> <AANLkTikW1OEoT5DSzYnpWyyuaraLZ_mjf6p7O_pRaqRW@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

We will be proposing Howl as an Incubator project soon.

Alan.

On Jan 31, 2011, at 7:44 PM, Jeff Hammerbacher wrote:

> Excellent news! Will you also make Howl, Oozie, and Yarn Apache  
> projects as
> well?
>
> On Mon, Jan 31, 2011 at 7:27 PM, Eric Baldeschwieler
> <eric14@yahoo-inc.com>wrote:
>
>> Hi Folks,
>>
>> I'm pleased to announce that after some reflection, Yahoo! has  
>> decided to
>> discontinue the  "The Yahoo Distribution of Hadoop" and focus on  
>> Apache
>> Hadoop.  We plan to remove all references to a Yahoo distribution  
>> from our
>> website (developer.yahoo.com/hadoop), close our github repo (
>> yahoo.github.com/hadoop-common) and focus on working more closely  
>> with the
>> Apache community.  Our intent is to return to helping Apache  
>> produce binary
>> releases of Apache Hadoop that are so bullet proof that Yahoo and  
>> other
>> production Hadoop users can run them unpatched on their clusters.
>>
>> Until Hadoop 0.20, Yahoo committers worked as release masters to  
>> produce
>> binary Apache Hadoop releases that the entire community used on their
>> clusters.    As the community grew, we have experiment with using the
>> "Yahoo! Distribution of Hadoop" as the vehicle to share our work.
>> Unfortunately, Apache is no longer the obvious place to go for Hadoop
>> releases.  The Yahoo! team wants to return to a world where anyone  
>> can
>> download and directly use releases of Hadoop from Apache.  We want to
>> contribute to the stabilization and testing of those releases.  We  
>> also want
>> to share our regular program of sustaining engineering that  
>> backports minor
>> feature enhancements into new dot releases on a regular basis, so  
>> that the
>> world sees regular improvements coming from Apache every few  
>> months, not
>> years.
>>
>> Recently the Apache Hadoop community has been very turbulent.  Over  
>> the
>> last few months we have been developing Hadoop enhancements in our  
>> internal
>> git repository while doing a complete review of our options. Our  
>> commitment
>> to open sourcing our work was never in doubt (see http://yhoo.it/e8p3Dd) 
>> ,
>> but the future of the "Yahoo distribution of Hadoop" was far from  
>> clear.
>> We've concluded that focusing on Apache Hadoop is the way forward.   
>> We
>> believe that more focus on communicating our goals to the Apache  
>> Hadoop
>> community, and more willingness to compromise on how we get to  
>> those goals,
>> will help us get back to making Hadoop even better.
>>
>> Unfortunately, we now have to sort out how to contribute several
>> person-years worth of work to Apache to let us unwind the Yahoo! git
>> repositories.  We currently run two lines of Hadoop development, our
>> sustaining program (hadoop-0.20-sustaining) and hadoop-future.
>> Hadoop-0.20-sustaining is the stable version of Hadoop we currently  
>> run on
>> Yahoo's 40,000 nodes.  It contains a series of fixes and  
>> enhancements that
>> are all backwards compatible with our "Hadoop 0.20 with security".   
>> It is
>> our most stable and high performance release of Hadoop ever.  We've  
>> expended
>> a lot of energy finding and fixing bugs in it this year. We have  
>> initiated
>> the process of contributing this work to Apache in the branch:
>> hadoop/common/branches/branch-0.20-security.  We've proposed  
>> calling this
>> the 20.100 release.  Once folks have had a chance to try this out  
>> and we've
>> had a chance to respond to their feedback, we plan to create 20.100  
>> release
>> candidates and ask the community to vote on making them Apache  
>> releases.
>>
>> Hadoop-future is our new feature branch.  We are working on a set  
>> of new
>> features for Hadoop to improve its availability, scalability and
>> interoperability to make Hadoop more usable in mission critical  
>> deployments.
>> You're going to see another burst of email activity from us as we  
>> work to
>> get hadoop-future patches socialized, reviewed and checked in.   
>> These bulk
>> checkins are exceptional.  They are the result of us striving to be  
>> more
>> transparent.  Once we've merged our hadoop-future and hadoop-0.20- 
>> sustaining
>> work back into Apache, folks can expect us to return to our regular
>> development cadence.  Looking forward, we plan to socialize our  
>> roadmaps
>> regularly, actively synchronize our work with other active Hadoop
>> contributors and develop our code collaboratively, directly in  
>> Apache.
>>
>> In summary, our decision to discontinue the "Yahoo! Distribution of  
>> Hadoop"
>> is a commitment to working more effectively with the Apache Hadoop
>> community.  Our goal is to make Apache Hadoop THE open source  
>> platform for
>> big data.
>>
>> Thanks,
>>
>> E14
>>
>> --
>>
>> PS Here is a draft list of key features in hadoop-future:
>>
>> * HDFS-1052 - Federation, the ability to support much more storage  
>> per
>> Hadoop cluster.
>>
>> * HADOOP-6728 - A the new metrics framework
>>
>> * MAPREDUCE-1220 - Optimizations for small jobs
>>
>> ---
>> PPS This is cross-posted on our blog: http://yhoo.it/i9Ww8W


From general-return-2932-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 01 16:11:37 2011
Return-Path: <general-return-2932-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39772 invoked from network); 1 Feb 2011 16:11:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Feb 2011 16:11:37 -0000
Received: (qmail 43237 invoked by uid 500); 1 Feb 2011 16:11:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42966 invoked by uid 500); 1 Feb 2011 16:11:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42953 invoked by uid 99); 1 Feb 2011 16:11:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 16:11:31 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_HELO_PASS,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [212.227.17.10] (HELO moutng.kundenserver.de) (212.227.17.10)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 16:11:25 +0000
Received: from office1.lan.sumadev.de (dslb-084-059-076-183.pools.arcor-ip.net [84.59.76.183])
	by mrelayeu.kundenserver.de (node=mreu0) with ESMTP (Nemesis)
	id 0LkknQ-1QKk203v76-00aRdN; Tue, 01 Feb 2011 17:10:57 +0100
Received: from [192.168.5.106] (dslb-084-059-076-183.pools.arcor-ip.net [84.59.76.183])
	by office1.lan.sumadev.de (Postfix) with ESMTPA id BDF455F701
	for <general@hadoop.apache.org>; Tue,  1 Feb 2011 17:10:56 +0100 (CET)
Subject: [ANN] Plasma MapReduce, PlasmaFS, version 0.3
From: Gerd Stolpmann <info@gerd-stolpmann.de>
To: general@hadoop.apache.org
Content-Type: text/plain; charset="UTF-8"
Date: Tue, 01 Feb 2011 17:10:55 +0100
Message-ID: <1296576655.24058.101.camel@thinkpad>
Mime-Version: 1.0
X-Mailer: Evolution 2.28.1 
Content-Transfer-Encoding: 7bit
X-Provags-ID: V02:K0:VzzLd84NUSy8Aze4b5btkGPUdU/ffsbaAcE+ELYn2XG
 2sIauN9WiGoUCGUCZWmNWXjyidpHKAkAtXQGQUY5YrHJmerTgq
 RnJqDo+dx6lQscw2vQ1exh1fv7o3qr/uZeJvWMpWMQSoHBkSIa
 ME1QJhUzh9qgyPpG1OhPxPm2By/kT7OwZmsnH6/4W3oEaGppN0
 vVm2G2cobAmKkY1+G9ubQ==

Hi,

This is about the release of Plasma-0.3, an alternate and independent
implementation of map/reduce with its own dfs. This might also be
interesting for Hadoop users and developers, because this project
incorporates a number of new ideas. So far, Plasma works on smaller
clusters and shows good signs of being scalable. HA support is still
very incomplete.

--

Plasma consists of two parts (for now), namely Plasma MapReduce, a
map/reduce compute framework, and PlasmaFS, the underlying distributed
filesystem.

Major changes in version 0.3 :

      * Optimized blocklist representation (extent-based)
      * Improved block allocator to minimize disk seeks
      * Allocating datanode access tickets in advance
      * Sophisticated RAM management
      * The command-line utility "plasma" supports wildcards

Of course, there are also numerous bug fixes and performance
improvements.

Plasma MapReduce is a distributed implementation of the map/reduce
algorithm scheme written in Ocaml. PlasmaFS is the underlying
distributed filesystem, also written in Ocaml. Especially the PlasmaFS
approach has numerous differences compared to HDFS:

      * Data blocks are preallocated, and PlasmaFS takes care of block
        placement
      * Blocklists are extent-based
      * Metadata is stored in a PostgreSQL db
      * 2-phase commit is used to distribute the metadata db
      * the full set of file access functions is supported, including
        random writes
      * file accesses can be transaction-based
      * shared memory can be used for speeding up the data path to
        locally stored data blocks
      * we _think_ it is not possible to corrupt the namenode by
        accident or by crashes
      * PlasmaFS volumes can be directly mounted via NFS
      * PlasmaFS uses ONCRPC as protocol and not home-grown protocols
        (and one of the next releases will add security via GSS-API)
      * We got rid of multi-threading

There is no need that user programs are written in Ocaml, as Plasma also
support a streaming mode.

Both pieces of software are bundled together in one download. The
project page with further links is

http://projects.camlcity.org/projects/plasma.html

There is now also a homepage at

http://plasma.camlcity.org

This is an early alpha release (0.3). A lot of things work already, and
you can already run distributed map/reduce jobs. However, it is in no
way complete.

Plasma is installable via GODI for Ocaml 3.12.

For discussions on specifics of Plasma there is a separate mailing list:

https://godirepo.camlcity.org/mailman/listinfo/plasma-list

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann, Bad Nauheimer Str.3, 64289 Darmstadt,Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Phone: +49-6151-153855                  Fax: +49-6151-997714
------------------------------------------------------------



From general-return-2933-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 01 16:29:49 2011
Return-Path: <general-return-2933-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 48322 invoked from network); 1 Feb 2011 16:29:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Feb 2011 16:29:48 -0000
Received: (qmail 88098 invoked by uid 500); 1 Feb 2011 16:29:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87736 invoked by uid 500); 1 Feb 2011 16:29:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87727 invoked by uid 99); 1 Feb 2011 16:29:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 16:29:44 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [98.139.53.202] (HELO nm1-vm0.bullet.mail.ac4.yahoo.com) (98.139.53.202)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 01 Feb 2011 16:29:35 +0000
Received: from [98.139.52.191] by nm1.bullet.mail.ac4.yahoo.com with NNFMP; 01 Feb 2011 16:29:13 -0000
Received: from [98.139.52.138] by tm4.bullet.mail.ac4.yahoo.com with NNFMP; 01 Feb 2011 16:29:13 -0000
Received: from [127.0.0.1] by omp1021.mail.ac4.yahoo.com with NNFMP; 01 Feb 2011 16:29:13 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 755594.7991.bm@omp1021.mail.ac4.yahoo.com
Received: (qmail 23456 invoked by uid 60001); 1 Feb 2011 16:29:13 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1296577753; bh=Kj4O3evh0hKXzIH7ESZ0SJbY2vL8QXqylDiuoekSMK0=; h=Message-ID:X-YMail-OSG:Received:X-RocketYMMF:X-Mailer:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=zLKnpqceNnIc9fzlREmzpZ7RvIF/mYAZDIgAZIYw24eQw0+0KIE90viEsqG5ZWNcWijMymv5k8JtFHegTrPggEI9zZ76wr6wTNBeRMSZ1UCfMr7XpPTqFueTB4251ebmLeEgJ+jrbiF9HtrG6rJ4+WRgyFwhxVc5w+Cl27tMnsA=
Message-ID: <446576.23201.qm@web65501.mail.ac4.yahoo.com>
X-YMail-OSG: 1xPbKKkVM1mVeGod0XAJs.VqZQ.sZYHKZJkjOMo5QMYt25I
 rAhVgqkuvoT2SFWDcSojKMB9sVGam.90GNhRA8k6eIh9BwaC409Jg33AmlWw
 snjfj2GEBzt8qwWVP7qagdK66IfnlJxItqH4TPDKlQa.q4Nf6P56oAOpP7OV
 FY5.uEf8AiK8tAnx_cqxPE2UlpBHaFNyIJSwxbmPmHe6UbRfn.irCUiafUXd
 xUgVFKp4Fm46mZkw1KxxNAOT7utKSX.KA0BKFY63rxHuEpSMOjYf3YZ7OKQg
 WEibBucq87CNFUnKE__jywxAsz3qkFFVbE62uWb3cGlK5Uvb0tL1LaABW7Lh
 By36ANpGsse_jhpj_0tu0BxjTsRqf9VaR._L2KZJ70x8i.X3mx7nxWbon8fm
 MlbcCfcUAvnc-
Received: from [68.123.236.97] by web65501.mail.ac4.yahoo.com via HTTP; Tue, 01 Feb 2011 08:29:12 PST
X-RocketYMMF: apurtell
X-Mailer: YahooMailClassic/11.4.20 YahooMailWebService/0.8.107.285259
Date: Tue, 1 Feb 2011 08:29:12 -0800 (PST)
From: Andrew Purtell <apurtell@apache.org>
Reply-To: apurtell@apache.org
Subject: Re: [ANNOUNCEMENT] Yahoo focusing on Apache Hadoop, discontinuing "The Yahoo Distribution of Hadoop"
To: general@hadoop.apache.org
In-Reply-To: <F3BFC5F7-19FC-4446-9C70-D30B8CC3B9F3@yahoo-inc.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org

> From: Alan Gates <gates@yahoo-inc.com>
>
> We will be proposing Howl as an Incubator project soon.

That would be excellent.

Best regards,

    - Andy

Problems worthy of attack prove their worth by hitting back.
  - Piet Hein (via Tom White)



      

From general-return-2934-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 01 17:37:44 2011
Return-Path: <general-return-2934-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83485 invoked from network); 1 Feb 2011 17:37:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Feb 2011 17:37:44 -0000
Received: (qmail 21814 invoked by uid 500); 1 Feb 2011 17:37:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21400 invoked by uid 500); 1 Feb 2011 17:37:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21382 invoked by uid 99); 1 Feb 2011 17:37:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 17:37:39 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of tom@cloudera.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 17:37:33 +0000
Received: by wye20 with SMTP id 20so7246211wye.35
        for <general@hadoop.apache.org>; Tue, 01 Feb 2011 09:37:13 -0800 (PST)
MIME-Version: 1.0
Received: by 10.216.19.139 with SMTP id n11mr5973052wen.78.1296581831258; Tue,
 01 Feb 2011 09:37:11 -0800 (PST)
Received: by 10.216.38.16 with HTTP; Tue, 1 Feb 2011 09:37:10 -0800 (PST)
In-Reply-To: <A9EDE11D-0CE9-4D04-A821-4DDEE685055A@linkedin.com>
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com>
	<FD5BC1D3-881A-4D45-BA2C-C6084C884418@apache.org>
	<AANLkTikYHbb935h5Qo+YbsYyrAYPSYYoyjrL5APy_unC@mail.gmail.com>
	<A9EDE11D-0CE9-4D04-A821-4DDEE685055A@linkedin.com>
Date: Tue, 1 Feb 2011 09:37:10 -0800
Message-ID: <AANLkTimnXKkAZTBCYoA7vqSK=5q-s21u=fMOz_NK0gKx@mail.gmail.com>
Subject: Re: [DISCUSS] Move common, hdfs, mapreduce contrib components to
 apache-extras.org or elsewhere
From: Tom White <tom@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

+1 for the reasons already cited: independent release cycles,
testing/build problems, lack of maintenance, etc. I think we should
strongly discourage new contrib components in favour of Apache Extras
or github, remove inactive contrib components, and also allow
maintainers to move components out if they volunteer to.

HBase moved all its contrib components out of the main tree a few
months back - can anyone comment how that worked out?

I agree that we should move streaming (MAPREDUCE-602) and the
schedulers to the main codebase. With work like MAPREDUCE-1478 we can
put these components into a library tree so that the libraries can
depend on core, but core doesn't depend on the libraries.

Milind: Record IO is in Common (in the main tree, not a contrib
component), and was deprecated in 0.21.0. We could remove it in a
future release.

Cheers,
Tom

On Tue, Feb 1, 2011 at 1:02 AM, Allen Wittenauer
<awittenauer@linkedin.com> wrote:
>
> On Jan 31, 2011, at 3:23 PM, Todd Lipcon wrote:
>
>> On Sun, Jan 30, 2011 at 11:19 PM, Owen O'Malley <omalley@apache.org> wro=
te:
>>
>>>
>>> Also note that pushing code out of Hadoop has a high cost. There are at
>>> least 3 forks of the hadoop-gpl-compression code. That creates a lot of
>>> confusion for the users. A lot of users never go to the work to figure =
out
>>> which fork and branch of hadoop-gpl-compression work with the version o=
f
>>> Hadoop they installed.
>>>
>>>
>> Indeed it creates confusion, but in my opinion it has been very successf=
ul
>> modulo that confusion.
>
> =A0 =A0 =A0 =A0I'm not sure how the above works with what you wrote below=
:
>
>> In particular, Kevin and I (who each have a repo on github but basically
>> co-maintain a branch) have done about 8 bugfix releases of LZO in the la=
st
>> year. The ability to take a bug and turn it around into a release within=
 a
>> few days has been very beneficial to the users. If it were part of core
>> Hadoop, people would be forced to live with these blocker bugs for month=
s at
>> a time between dot releases.
>
> =A0 =A0 =A0 =A0So is the expectation that users would have to follow brea=
d crumbs to the github dumping ground, then try to figure out which repo is=
 the 'better' choice for their usage? =A0 Using LZO as an example, it appea=
rs we have a choice of kevin's, your's, or the master without even taking i=
nto consideration any tags. That sounds like a recipe for disaster that's e=
ven worse than what we have today.
>
>
>> IMO the more we can take non-core components and move them to separate
>> release timelines, the better. Yes, it is harder for users, but it also =
is
>> easier for them when they hit a bug - they don't have to wait months for=
 a
>> wholesale upgrade which might contain hundreds of other changes to core
>> components.
>
> =A0 =A0 =A0 =A0I'd agree except for one thing: =A0even when users do prov=
ide patches to contrib components we ignore them. =A0How long have those pa=
tches for HOD been sitting there in the patch queue? =A0So of course they w=
ait months/years--because we seemingly ignore anything that isn't important=
 to us. =A0Unfortunately, that covers a large chunk of contrib. :(
>
>
>

From general-return-2935-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 01 17:51:00 2011
Return-Path: <general-return-2935-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 89218 invoked from network); 1 Feb 2011 17:50:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Feb 2011 17:50:59 -0000
Received: (qmail 60244 invoked by uid 500); 1 Feb 2011 17:50:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59970 invoked by uid 500); 1 Feb 2011 17:50:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59959 invoked by uid 99); 1 Feb 2011 17:50:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 17:50:55 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of tom@cloudera.com designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 17:50:49 +0000
Received: by ewy20 with SMTP id 20so3241693ewy.35
        for <general@hadoop.apache.org>; Tue, 01 Feb 2011 09:50:27 -0800 (PST)
MIME-Version: 1.0
Received: by 10.216.25.136 with SMTP id z8mr909866wez.93.1296582627660; Tue,
 01 Feb 2011 09:50:27 -0800 (PST)
Received: by 10.216.38.16 with HTTP; Tue, 1 Feb 2011 09:50:27 -0800 (PST)
In-Reply-To: <4D46DC4A.7040700@apache.org>
References: <4D46B6AD.2020802@apache.org>
	<AANLkTik2WDc7Y7Kd_708TSJVb5wjAcad6hQsBEg0FtqK@mail.gmail.com>
	<4D46DC4A.7040700@apache.org>
Date: Tue, 1 Feb 2011 09:50:27 -0800
Message-ID: <AANLkTin0omQU0he1zjPxEkXtYB0i+uLjRzGj4zJh51JB@mail.gmail.com>
Subject: Re: Defining Compatibility
From: Tom White <tom@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

FWIW the FileSystemContractBaseTest class and the FileContext*BaseTest
classes (and their concrete subclasses) are probably the closest thing
we have to compatibility tests for FileSystem and FileContext
implementations in Hadoop.

Tom

On Mon, Jan 31, 2011 at 7:59 AM, Steve Loughran <stevel@apache.org> wrote:
> On 31/01/11 14:32, Chris Douglas wrote:
>>
>> Steve-
>>
>> It's hard to answer without more concrete criteria. Is this a
>> trademark question affecting the marketing of a product? A
>> cross-compatibility taxonomy for users? The minimum criteria to
>> publish a paper/release a product without eye-rolling? The particular
>> compatibility claims made by a system will be nuanced and specific; a
>> runtime that executes MapReduce jobs as they would run in Hadoop can
>> simply make that claim, whether it uses parts of MapReduce, HDFS, or
>> neither.
>
> No, I'm thinking more about what large scale tests are needed to be run
> against the codebase before you can say "it works", and then how to say some
> changes means that it still works.
>
>>
>> For the various distributions "Powered by Apache Hadoop," one would
>> assume that compatibility will vary depending on the featureset and
>> the audience. A distribution that runs MapReduce applications
>> as-written for Apache Hadoop may be incompatible with a user's
>> deployed metrics/monitoring system. Some random script to scrape the
>> UI may not work. The product may only scale to 20 nodes. Whether these
>> are "compatible with Apache Hadoop" is awkward to answer generally,
>> unless we want to define the semantics of that phrase by policy.
>>
>> To put it bluntly, why would we bother to define such a policy? One
>> could assert that a fully-compatible system would implement all the
>> public/stable APIs as defined in HADOOP-5073, but who would that help?
>> And though interoperability is certainly relevant to systems built on
>> top of Hadoop, is there a reason the Apache project needs to be
>> involved in defining the standards for compatibility among them?
>
> Agreed, I'm just thinking about namings and definitions. Even with the
> stable/unstable internal/external split, there's still the question as to
> what the semantics of operations are, both explicit (this operation does X)
> and implicit (and it takes less than Y seconds to do it). It's those
> implicit things that always catch you out (indeed, they are the argument
> points in things like Java and Java EE compatibility test kits)
>

From general-return-2936-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 01 19:28:08 2011
Return-Path: <general-return-2936-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 65661 invoked from network); 1 Feb 2011 19:28:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Feb 2011 19:28:08 -0000
Received: (qmail 80685 invoked by uid 500); 1 Feb 2011 19:28:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80578 invoked by uid 500); 1 Feb 2011 19:28:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80566 invoked by uid 99); 1 Feb 2011 19:28:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 19:28:05 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 19:27:59 +0000
Received: from SP2-EX07CAS05.ds.corp.yahoo.com (sp2-ex07cas05.corp.sp2.yahoo.com [98.137.59.39])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p11JRR0U082137
	for <general@hadoop.apache.org>; Tue, 1 Feb 2011 11:27:27 -0800 (PST)
Received: from SP2-EX07VS02.ds.corp.yahoo.com ([98.137.59.30]) by
 SP2-EX07CAS05.ds.corp.yahoo.com ([98.137.59.39]) with mapi; Tue, 1 Feb 2011
 11:27:27 -0800
From: "Giridharan  Kesavan" <gkesavan@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 1 Feb 2011 11:27:25 -0800
Subject: Re: Hadoop-common-trunk-Commit is failing since 01/19/2011
Thread-Topic: Hadoop-common-trunk-Commit is failing since 01/19/2011
Thread-Index: AcvCRg7fOtFpj/qCSJa4hg5ZsiYl7g==
Message-ID: <AE3E9DCF-B2D6-4EA7-A166-DB139CD76BFD@yahoo-inc.com>
References: <AANLkTinB+8KRByZFhRT-y6MiqA0urtKNs8c_XSrhP7RH@mail.gmail.com>
 <AANLkTimmcYh0hY-6ht6vaQ+S=pTGHaFxS6k9qEv=q8Cf@mail.gmail.com>
 <AANLkTinr_Exrr8VPShGKkBSMzmWX1B9nVALLLTMmvYzd@mail.gmail.com>
 <AANLkTinrheVYQm-y9n=k7TRK_BdzuZYKAWG+qMWR88ct@mail.gmail.com>
 <AANLkTi=cAhCOkqZ=n9NpBMoPzechoJmfB6KO2yL3QcVA@mail.gmail.com>
 <AANLkTi=+81g8dFMGMqP5Zc6aAkg5n0dD2QpCVg7ABzd6@mail.gmail.com>
 <7DC70A70-5DE9-4288-A44B-B5EF9C454D9B@yahoo-inc.com>
 <AANLkTin9fsb3bgsa98bma+xx_C9K06C_YNKJ6vaA7=TO@mail.gmail.com>
 <F3847AE0-3FEF-4E85-8757-FA8A7B204A64@yahoo-inc.com>
 <AANLkTikCOHYJx4Qjt=gajGTcspPai0GmsxWQFogua92Q@mail.gmail.com>
In-Reply-To: <AANLkTikCOHYJx4Qjt=gajGTcspPai0GmsxWQFogua92Q@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Konstantin,

trunk/artifacts gets populated when the jar and the tar ant target are succ=
essful.

The main reason for the build failure so far is the build abort time config=
uration. It was set to 30mins.
I have increased the build abort time and the builds are going on fine=20
https://hudson.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-Common-tru=
nk-Commit


Thanks,
Giri

On Feb 1, 2011, at 12:40 AM, Konstantin Shvachko wrote:

> Giri,
>=20
> Looking at configuration of Hadoop-Common-trunk-Commit/
> There seems to be errors in the Post-build Actions.
> It is complaining that
> 'trunk' exists but not 'trunk/artifacts/...'
> Is it possible that this misconfiguration is the reason of failures?
>=20
> --Konstantin
>=20
>=20
> On Mon, Jan 31, 2011 at 4:40 PM, Giridharan Kesavan
> <gkesavan@yahoo-inc.com>wrote:
>=20
>> Konstantin,
>>=20
>> I think I need to restart the slave which is running the commit build. F=
or
>> now I have published the common artifact manually from commandline.
>>=20
>> Thanks,
>> Giri
>>=20
>> On Jan 31, 2011, at 4:27 PM, Konstantin Shvachko wrote:
>>=20
>>> Giri
>>> looks like the last run you started failed the same way as previous one=
s.
>>> Any thoughts on what's going on?
>>> Thanks,
>>> --Konstantin
>>>=20
>>> On Mon, Jan 31, 2011 at 3:33 PM, Giridharan Kesavan
>>> <gkesavan@yahoo-inc.com>wrote:
>>>=20
>>>> ant mvn-deploy would publish snapshot artifact to the apache maven
>>>> repository as long you have the right credentials in ~/.m2/settings.xm=
l.
>>>>=20
>>>> For settings.xml template pls look at
>>>> http://wiki.apache.org/hadoop/HowToRelease
>>>>=20
>>>> I'm pushing the latest common artifacts now.
>>>>=20
>>>> -Giri
>>>>=20
>>>>=20
>>>>=20
>>>> On Jan 31, 2011, at 3:11 PM, Jakob Homan wrote:
>>>>=20
>>>>> By manually installing a new core jar into the cache, I can compile
>>>>> trunk.  Looks like we just need to kick a new Core into maven.  Are
>>>>> there instructions somewhere for committers to do this?  I know Nigel
>>>>> and Owen know how, but I don't know if the knowledge is diffused past
>>>>> them.
>>>>> -Jakob
>>>>>=20
>>>>>=20
>>>>> On Mon, Jan 31, 2011 at 1:57 PM, Konstantin Shvachko
>>>>> <shv.hadoop@gmail.com> wrote:
>>>>>> Current trunk for HDFS and MapReduce are not compiling at the moment=
.
>>>> Try to
>>>>>> build trunk.
>>>>>> This is the result of that changes to common api introduced by
>>>> HADOOP-6904
>>>>>> are not promoted to HDFS and MR trunks.
>>>>>> HDFS-1335 and MAPREDUCE-2263 depend on these changes.
>>>>>>=20
>>>>>> Common is not promoted to HDFS and MR because
>> Hadoop-Common-trunk-Commit
>>>>>> build is broken. See here.
>>>>>>=20
>>>>=20
>> https://hudson.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-Common-=
trunk-Commit/
>>>>>>=20
>>>>>> As I see the last successful build was on 01/19, which integrated
>>>>>> HADOOP-6864.
>>>>>> I think this is when JNI changes were introduced, which cannot be
>>>> digested
>>>>>> by Hudson since then.
>>>>>>=20
>>>>>> Anybody with gcc active could you please verify if the problem is
>> caused
>>>> by
>>>>>> HADOOP-6864.
>>>>>>=20
>>>>>> Thanks,
>>>>>> --Konstantin
>>>>>>=20
>>>>>> On Mon, Jan 31, 2011 at 1:36 PM, Ted Dunning <tdunning@maprtech.com>
>>>> wrote:
>>>>>>=20
>>>>>>> The has been a problem with more than one build failing (Mahout is
>> the
>>>> one
>>>>>>> that I saw first) due to a change in maven version which meant that
>> the
>>>>>>> clover license isn't being found properly.  At least, that is the
>> tale
>>>> I
>>>>>>> heard from infra.
>>>>>>>=20
>>>>>>> On Mon, Jan 31, 2011 at 1:31 PM, Eli Collins <eli@cloudera.com>
>> wrote:
>>>>>>>=20
>>>>>>>> Hey Konstantin,
>>>>>>>>=20
>>>>>>>> The only build breakage I saw from HADOOP-6904 is MAPREDUCE-2290,
>>>>>>>> which was fixed.  Trees from trunk are compiling against each othe=
r
>>>>>>>> for me (eg each installed to a local maven repo), perhaps the
>> upstream
>>>>>>>> maven repo hasn't been updated with the latest bits yet.
>>>>>>>>=20
>>>>>>>> Thanks,
>>>>>>>> Eli
>>>>>>>>=20
>>>>>>>> On Mon, Jan 31, 2011 at 12:14 PM, Konstantin Shvachko
>>>>>>>> <shv.hadoop@gmail.com> wrote:
>>>>>>>>> Sending this to general to attract urgent attention.
>>>>>>>>> Both HDFS and MapReduce are not compiling since
>>>>>>>>> HADOOP-6904 and its hdfs and MP counterparts were committed.
>>>>>>>>> The problem is not with this patch as described below, but I thin=
k
>>>>>>> those
>>>>>>>>> commits should be reversed if Common integration build cannot be
>>>>>>>>> restored promptly.
>>>>>>>>>=20
>>>>>>>>> Thanks,
>>>>>>>>> --Konstantin
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>> On Fri, Jan 28, 2011 at 5:53 PM, Konstantin Shvachko
>>>>>>>>> <shv.hadoop@gmail.com>wrote:
>>>>>>>>>=20
>>>>>>>>>> I see Hadoop-common-trunk-Commit is failing and not sending any
>>>>>>> emails.
>>>>>>>>>> It times out on native compilation and aborts.
>>>>>>>>>> Therefore changes are not integrated, and now it lead to hdfs an=
d
>>>>>>>> mapreduce
>>>>>>>>>> both not compiling.
>>>>>>>>>> Can somebody please take a look at this.
>>>>>>>>>> The last few lines of the build are below.
>>>>>>>>>>=20
>>>>>>>>>> Thanks
>>>>>>>>>> --Konstantin
>>>>>>>>>>=20
>>>>>>>>>>   [javah] [Loaded
>>>>>>>>=20
>>>>>>>=20
>>>>=20
>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/b=
uild/classes/org/apache/hadoop/security/JniBasedUnixGroupsMapping.class]
>>>>>>>>>>=20
>>>>>>>>>>   [javah] [Loaded
>>>>>>>>=20
>>>>>>>=20
>>>>=20
>> /homes/hudson/tools/java/jdk1.6.0_11-32/jre/lib/rt.jar(java/lang/Object.=
class)]
>>>>>>>>>>   [javah] [Forcefully writing file
>>>>>>>>=20
>>>>>>>=20
>>>>=20
>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/b=
uild/native/Linux-i386-32/src/org/apache/hadoop/security/org_apache_hadoop_=
security_JniBasedUnixGroupsNetgroupMapping.h]
>>>>>>>>>>=20
>>>>>>>>>>    [exec] checking for gcc... gcc
>>>>>>>>>>    [exec] checking whether the C compiler works... yes
>>>>>>>>>>    [exec] checking for C compiler default output file name...
>>>> a.out
>>>>>>>>>>    [exec] checking for suffix of executables...
>>>>>>>>>>=20
>>>>>>>>>> Build timed out. Aborting
>>>>>>>>>> Build was aborted
>>>>>>>>>> [FINDBUGS] Skipping publisher since build result is ABORTED
>>>>>>>>>> Publishing Javadoc
>>>>>>>>>> Archiving artifacts
>>>>>>>>>> Recording test results
>>>>>>>>>> No test report files were found. Configuration error?
>>>>>>>>>>=20
>>>>>>>>>> Recording fingerprints
>>>>>>>>>>    [exec] Terminated
>>>>>>>>>> Publishing Clover coverage report...
>>>>>>>>>> No Clover report will be published due to a Build Failure
>>>>>>>>>> No emails were triggered.
>>>>>>>>>> Finished: ABORTED
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>=20
>>>>>>>=20
>>>>>>=20
>>>>=20
>>>>=20
>>=20
>>=20


From general-return-2937-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 01 19:45:04 2011
Return-Path: <general-return-2937-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72138 invoked from network); 1 Feb 2011 19:45:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Feb 2011 19:45:03 -0000
Received: (qmail 7128 invoked by uid 500); 1 Feb 2011 19:45:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6921 invoked by uid 500); 1 Feb 2011 19:45:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6913 invoked by uid 99); 1 Feb 2011 19:45:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 19:45:01 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of shv.hadoop@gmail.com designates 74.125.83.48 as permitted sender)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 19:44:54 +0000
Received: by gwj15 with SMTP id 15so2922579gwj.35
        for <general@hadoop.apache.org>; Tue, 01 Feb 2011 11:44:33 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=disa3L/y0GzF+xVYZS25feS1FoOwvSkblGulAvTcWzY=;
        b=bVICb4r6ypazY3KSzikg6wGwseQ+aAUzUrSORAuUumqwA61bXSD6Ilb9ftva7cz9Fw
         VGRXapxhaGk4Hzixve0Q51jaJjwh8YmtRUrIQjWU+luQySP/vfkisqocFdXprAnaXjVT
         KmpENiOw83O+ka7q8KujYSQpubu0b4ndABlSQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=a85oki+hbp1QhMdQDv4gWAETq93adW/GzrnLVpGIIYXLRzCmsrPuJFcnZI9LvPgOE1
         RlShyfDbYVwhmGJhUsrYvvpjOImAGCmdwnMVC7qTcENjHpFAVwELh/lEJeqJN7++cAnW
         zJ+MknmFCxk5avMnN0FzZDL/FKdxysjgi2kSs=
MIME-Version: 1.0
Received: by 10.236.103.180 with SMTP id f40mr16748733yhg.0.1296589473385;
 Tue, 01 Feb 2011 11:44:33 -0800 (PST)
Received: by 10.236.108.35 with HTTP; Tue, 1 Feb 2011 11:44:33 -0800 (PST)
In-Reply-To: <AE3E9DCF-B2D6-4EA7-A166-DB139CD76BFD@yahoo-inc.com>
References: <AANLkTinB+8KRByZFhRT-y6MiqA0urtKNs8c_XSrhP7RH@mail.gmail.com>
	<AANLkTimmcYh0hY-6ht6vaQ+S=pTGHaFxS6k9qEv=q8Cf@mail.gmail.com>
	<AANLkTinr_Exrr8VPShGKkBSMzmWX1B9nVALLLTMmvYzd@mail.gmail.com>
	<AANLkTinrheVYQm-y9n=k7TRK_BdzuZYKAWG+qMWR88ct@mail.gmail.com>
	<AANLkTi=cAhCOkqZ=n9NpBMoPzechoJmfB6KO2yL3QcVA@mail.gmail.com>
	<AANLkTi=+81g8dFMGMqP5Zc6aAkg5n0dD2QpCVg7ABzd6@mail.gmail.com>
	<7DC70A70-5DE9-4288-A44B-B5EF9C454D9B@yahoo-inc.com>
	<AANLkTin9fsb3bgsa98bma+xx_C9K06C_YNKJ6vaA7=TO@mail.gmail.com>
	<F3847AE0-3FEF-4E85-8757-FA8A7B204A64@yahoo-inc.com>
	<AANLkTikCOHYJx4Qjt=gajGTcspPai0GmsxWQFogua92Q@mail.gmail.com>
	<AE3E9DCF-B2D6-4EA7-A166-DB139CD76BFD@yahoo-inc.com>
Date: Tue, 1 Feb 2011 11:44:33 -0800
Message-ID: <AANLkTik7t1Gv4aJOSEGKt9pG1B9EYa4eGbOF9F_nCSq4@mail.gmail.com>
Subject: Re: Hadoop-common-trunk-Commit is failing since 01/19/2011
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0023543a2748fb173f049b3dc226
X-Virus-Checked: Checked by ClamAV on apache.org

--0023543a2748fb173f049b3dc226
Content-Type: text/plain; charset=ISO-8859-1

Giri,
Thanks a lot for fixing this.
I see it is working now.
--Konstantin

On Tue, Feb 1, 2011 at 11:27 AM, Giridharan Kesavan
<gkesavan@yahoo-inc.com>wrote:

> Konstantin,
>
> trunk/artifacts gets populated when the jar and the tar ant target are
> successful.
>
> The main reason for the build failure so far is the build abort time
> configuration. It was set to 30mins.
> I have increased the build abort time and the builds are going on fine
>
> https://hudson.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit
>
>
> Thanks,
> Giri
>
> On Feb 1, 2011, at 12:40 AM, Konstantin Shvachko wrote:
>
> > Giri,
> >
> > Looking at configuration of Hadoop-Common-trunk-Commit/
> > There seems to be errors in the Post-build Actions.
> > It is complaining that
> > 'trunk' exists but not 'trunk/artifacts/...'
> > Is it possible that this misconfiguration is the reason of failures?
> >
> > --Konstantin
> >
> >
> > On Mon, Jan 31, 2011 at 4:40 PM, Giridharan Kesavan
> > <gkesavan@yahoo-inc.com>wrote:
> >
> >> Konstantin,
> >>
> >> I think I need to restart the slave which is running the commit build.
> For
> >> now I have published the common artifact manually from commandline.
> >>
> >> Thanks,
> >> Giri
> >>
> >> On Jan 31, 2011, at 4:27 PM, Konstantin Shvachko wrote:
> >>
> >>> Giri
> >>> looks like the last run you started failed the same way as previous
> ones.
> >>> Any thoughts on what's going on?
> >>> Thanks,
> >>> --Konstantin
> >>>
> >>> On Mon, Jan 31, 2011 at 3:33 PM, Giridharan Kesavan
> >>> <gkesavan@yahoo-inc.com>wrote:
> >>>
> >>>> ant mvn-deploy would publish snapshot artifact to the apache maven
> >>>> repository as long you have the right credentials in
> ~/.m2/settings.xml.
> >>>>
> >>>> For settings.xml template pls look at
> >>>> http://wiki.apache.org/hadoop/HowToRelease
> >>>>
> >>>> I'm pushing the latest common artifacts now.
> >>>>
> >>>> -Giri
> >>>>
> >>>>
> >>>>
> >>>> On Jan 31, 2011, at 3:11 PM, Jakob Homan wrote:
> >>>>
> >>>>> By manually installing a new core jar into the cache, I can compile
> >>>>> trunk.  Looks like we just need to kick a new Core into maven.  Are
> >>>>> there instructions somewhere for committers to do this?  I know Nigel
> >>>>> and Owen know how, but I don't know if the knowledge is diffused past
> >>>>> them.
> >>>>> -Jakob
> >>>>>
> >>>>>
> >>>>> On Mon, Jan 31, 2011 at 1:57 PM, Konstantin Shvachko
> >>>>> <shv.hadoop@gmail.com> wrote:
> >>>>>> Current trunk for HDFS and MapReduce are not compiling at the
> moment.
> >>>> Try to
> >>>>>> build trunk.
> >>>>>> This is the result of that changes to common api introduced by
> >>>> HADOOP-6904
> >>>>>> are not promoted to HDFS and MR trunks.
> >>>>>> HDFS-1335 and MAPREDUCE-2263 depend on these changes.
> >>>>>>
> >>>>>> Common is not promoted to HDFS and MR because
> >> Hadoop-Common-trunk-Commit
> >>>>>> build is broken. See here.
> >>>>>>
> >>>>
> >>
> https://hudson.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/
> >>>>>>
> >>>>>> As I see the last successful build was on 01/19, which integrated
> >>>>>> HADOOP-6864.
> >>>>>> I think this is when JNI changes were introduced, which cannot be
> >>>> digested
> >>>>>> by Hudson since then.
> >>>>>>
> >>>>>> Anybody with gcc active could you please verify if the problem is
> >> caused
> >>>> by
> >>>>>> HADOOP-6864.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> --Konstantin
> >>>>>>
> >>>>>> On Mon, Jan 31, 2011 at 1:36 PM, Ted Dunning <tdunning@maprtech.com
> >
> >>>> wrote:
> >>>>>>
> >>>>>>> The has been a problem with more than one build failing (Mahout is
> >> the
> >>>> one
> >>>>>>> that I saw first) due to a change in maven version which meant that
> >> the
> >>>>>>> clover license isn't being found properly.  At least, that is the
> >> tale
> >>>> I
> >>>>>>> heard from infra.
> >>>>>>>
> >>>>>>> On Mon, Jan 31, 2011 at 1:31 PM, Eli Collins <eli@cloudera.com>
> >> wrote:
> >>>>>>>
> >>>>>>>> Hey Konstantin,
> >>>>>>>>
> >>>>>>>> The only build breakage I saw from HADOOP-6904 is MAPREDUCE-2290,
> >>>>>>>> which was fixed.  Trees from trunk are compiling against each
> other
> >>>>>>>> for me (eg each installed to a local maven repo), perhaps the
> >> upstream
> >>>>>>>> maven repo hasn't been updated with the latest bits yet.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Eli
> >>>>>>>>
> >>>>>>>> On Mon, Jan 31, 2011 at 12:14 PM, Konstantin Shvachko
> >>>>>>>> <shv.hadoop@gmail.com> wrote:
> >>>>>>>>> Sending this to general to attract urgent attention.
> >>>>>>>>> Both HDFS and MapReduce are not compiling since
> >>>>>>>>> HADOOP-6904 and its hdfs and MP counterparts were committed.
> >>>>>>>>> The problem is not with this patch as described below, but I
> think
> >>>>>>> those
> >>>>>>>>> commits should be reversed if Common integration build cannot be
> >>>>>>>>> restored promptly.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> --Konstantin
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Fri, Jan 28, 2011 at 5:53 PM, Konstantin Shvachko
> >>>>>>>>> <shv.hadoop@gmail.com>wrote:
> >>>>>>>>>
> >>>>>>>>>> I see Hadoop-common-trunk-Commit is failing and not sending any
> >>>>>>> emails.
> >>>>>>>>>> It times out on native compilation and aborts.
> >>>>>>>>>> Therefore changes are not integrated, and now it lead to hdfs
> and
> >>>>>>>> mapreduce
> >>>>>>>>>> both not compiling.
> >>>>>>>>>> Can somebody please take a look at this.
> >>>>>>>>>> The last few lines of the build are below.
> >>>>>>>>>>
> >>>>>>>>>> Thanks
> >>>>>>>>>> --Konstantin
> >>>>>>>>>>
> >>>>>>>>>>   [javah] [Loaded
> >>>>>>>>
> >>>>>>>
> >>>>
> >>
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/build/classes/org/apache/hadoop/security/JniBasedUnixGroupsMapping.class]
> >>>>>>>>>>
> >>>>>>>>>>   [javah] [Loaded
> >>>>>>>>
> >>>>>>>
> >>>>
> >>
> /homes/hudson/tools/java/jdk1.6.0_11-32/jre/lib/rt.jar(java/lang/Object.class)]
> >>>>>>>>>>   [javah] [Forcefully writing file
> >>>>>>>>
> >>>>>>>
> >>>>
> >>
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/build/native/Linux-i386-32/src/org/apache/hadoop/security/org_apache_hadoop_security_JniBasedUnixGroupsNetgroupMapping.h]
> >>>>>>>>>>
> >>>>>>>>>>    [exec] checking for gcc... gcc
> >>>>>>>>>>    [exec] checking whether the C compiler works... yes
> >>>>>>>>>>    [exec] checking for C compiler default output file name...
> >>>> a.out
> >>>>>>>>>>    [exec] checking for suffix of executables...
> >>>>>>>>>>
> >>>>>>>>>> Build timed out. Aborting
> >>>>>>>>>> Build was aborted
> >>>>>>>>>> [FINDBUGS] Skipping publisher since build result is ABORTED
> >>>>>>>>>> Publishing Javadoc
> >>>>>>>>>> Archiving artifacts
> >>>>>>>>>> Recording test results
> >>>>>>>>>> No test report files were found. Configuration error?
> >>>>>>>>>>
> >>>>>>>>>> Recording fingerprints
> >>>>>>>>>>    [exec] Terminated
> >>>>>>>>>> Publishing Clover coverage report...
> >>>>>>>>>> No Clover report will be published due to a Build Failure
> >>>>>>>>>> No emails were triggered.
> >>>>>>>>>> Finished: ABORTED
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
> >>
>
>

--0023543a2748fb173f049b3dc226--

From general-return-2938-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 01 19:47:24 2011
Return-Path: <general-return-2938-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72941 invoked from network); 1 Feb 2011 19:47:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Feb 2011 19:47:23 -0000
Received: (qmail 11642 invoked by uid 500); 1 Feb 2011 19:47:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11587 invoked by uid 500); 1 Feb 2011 19:47:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11579 invoked by uid 99); 1 Feb 2011 19:47:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 19:47:21 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates 209.85.210.176 as permitted sender)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 19:47:17 +0000
Received: by iyb26 with SMTP id 26so7048815iyb.35
        for <general@hadoop.apache.org>; Tue, 01 Feb 2011 11:46:56 -0800 (PST)
Received: by 10.231.207.84 with SMTP id fx20mr8736689ibb.62.1296589616524;
 Tue, 01 Feb 2011 11:46:56 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.153.19 with HTTP; Tue, 1 Feb 2011 11:46:35 -0800 (PST)
In-Reply-To: <A9EDE11D-0CE9-4D04-A821-4DDEE685055A@linkedin.com>
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com> <FD5BC1D3-881A-4D45-BA2C-C6084C884418@apache.org>
 <AANLkTikYHbb935h5Qo+YbsYyrAYPSYYoyjrL5APy_unC@mail.gmail.com> <A9EDE11D-0CE9-4D04-A821-4DDEE685055A@linkedin.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Tue, 1 Feb 2011 11:46:35 -0800
Message-ID: <AANLkTinjVGUSU6V_2NO6TcQmt9t0nuHdw+OAysL0sVJU@mail.gmail.com>
Subject: Re: [DISCUSS] Move common, hdfs, mapreduce contrib components to
 apache-extras.org or elsewhere
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba4fc514832ec9049b3dcb5a

--90e6ba4fc514832ec9049b3dcb5a
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Feb 1, 2011 at 1:02 AM, Allen Wittenauer
<awittenauer@linkedin.com>wrote:

>
>
>         So is the expectation that users would have to follow bread crumbs
> to the github dumping ground, then try to figure out which repo is the
> 'better' choice for their usage?   Using LZO as an example, it appears we
> have a choice of kevin's, your's, or the master without even taking into
> consideration any tags. That sounds like a recipe for disaster that's even
> worse than what we have today.
>
>
Kevin's and mine are currently identical
(0e7005136e4160ed4cc157c4ddd7f4f1c6e11ffa)

Not sure who "the master" is -- maybe you're referring to the Google Code
repo? The reason we started working on github over a year ago is that the
bugs we reported (and provided diffs for) in the Google Code project were
ignored. For example:
http://code.google.com/p/hadoop-gpl-compression/issues/detail?id=17

In fact this repo hasn't been updated since Sep '09:
http://code.google.com/p/hadoop-gpl-compression/source/list

Github provided an excellent place to collaborate on the project, make
progress, fix bugs, and provide a better product for the users.

As for "dumping ground," I don't quite follow your point - we develop in the
open, accept pull requests from users, and code review each others' changes.
Since October every commit has either been contributed by or fixes a bug
reported by a user completely outside of the organizations where Kevin and I
work.

I agree that it's a bit of "breadcrumb following" to find the repo, though.
We do at least have a link on the wiki:
http://wiki.apache.org/hadoop/UsingLzoCompression which points to Kevin's
repo.

Perhaps the best solution here is to add a page to the official Hadoop site
(not just the wiki) with links to actively maintained contrib projects?


>
> > IMO the more we can take non-core components and move them to separate
> > release timelines, the better. Yes, it is harder for users, but it also
> is
> > easier for them when they hit a bug - they don't have to wait months for
> a
> > wholesale upgrade which might contain hundreds of other changes to core
> > components.
>
>         I'd agree except for one thing:  even when users do provide patches
> to contrib components we ignore them.  How long have those patches for HOD
> been sitting there in the patch queue?  So of course they wait
> months/years--because we seemingly ignore anything that isn't important to
> us.  Unfortunately, that covers a large chunk of contrib. :(
>

True - we ignore them because the core contributors generally have little
clue about the contrib components, so don't feel qualified to review. I'll
happily admit that I've never run failmon, index, dynamic-scheduler,
eclipse-plugin, data_join, mumak, or vertica contribs. Wouldn't you rather
these components lived on github so the people who wrote them could update
them as they wished without having to wait on committers who have little to
no clue about how to evaluate the changes?

-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

--90e6ba4fc514832ec9049b3dcb5a--

From general-return-2939-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 01 19:54:56 2011
Return-Path: <general-return-2939-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 75726 invoked from network); 1 Feb 2011 19:54:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Feb 2011 19:54:56 -0000
Received: (qmail 21408 invoked by uid 500); 1 Feb 2011 19:54:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21160 invoked by uid 500); 1 Feb 2011 19:54:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21147 invoked by uid 99); 1 Feb 2011 19:54:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 19:54:54 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 19:54:50 +0000
Received: by iwn2 with SMTP id 2so7604866iwn.35
        for <general@hadoop.apache.org>; Tue, 01 Feb 2011 11:54:29 -0800 (PST)
Received: by 10.231.207.84 with SMTP id fx20mr8746499ibb.62.1296590068666;
 Tue, 01 Feb 2011 11:54:28 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.153.19 with HTTP; Tue, 1 Feb 2011 11:54:08 -0800 (PST)
In-Reply-To: <AANLkTimnXKkAZTBCYoA7vqSK=5q-s21u=fMOz_NK0gKx@mail.gmail.com>
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com> <FD5BC1D3-881A-4D45-BA2C-C6084C884418@apache.org>
 <AANLkTikYHbb935h5Qo+YbsYyrAYPSYYoyjrL5APy_unC@mail.gmail.com>
 <A9EDE11D-0CE9-4D04-A821-4DDEE685055A@linkedin.com> <AANLkTimnXKkAZTBCYoA7vqSK=5q-s21u=fMOz_NK0gKx@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Tue, 1 Feb 2011 11:54:08 -0800
Message-ID: <AANLkTim6KCdv1jkc1JT=ZSE2uGAbF3iHbS2NtT6nR5xO@mail.gmail.com>
Subject: Re: [DISCUSS] Move common, hdfs, mapreduce contrib components to
 apache-extras.org or elsewhere
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba4fc514765287049b3de63b

--90e6ba4fc514765287049b3de63b
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Feb 1, 2011 at 9:37 AM, Tom White <tom@cloudera.com> wrote:

>
> HBase moved all its contrib components out of the main tree a few
> months back - can anyone comment how that worked out?
>
>
Sure. For each contrib:

ec2: no longer exists, and now has been integrated into Whirr and much
improved. Whirr has made several releases in the time that HBase has made
one. The whirr contributors know way more about cloud deployment than the
HBase contributors (except where they happen to overlap). Strong net
positive.

mdc_replication: pulled into core since it's developed by core committers
and also needs a fair amount of tight integration with core components

stargate: pulled into core - it was only in contrib as a sort of staging
ground - it's really an improved/new version of the "rest" interface we
already had in core.

transactional: moved to github - this has languished a bit on github because
only one person was actively maintaining it. However, it had already been
"languishing" as part of contrib - even though it compiled, it never really
worked very well in HBase trunk. So, moving it to a place where it's
languished has just made it more obvious what was already true - that it
isn't a well supported component (yet). Recently it's been taken back up by
the author of it - if it develops a large user base it can move quickly and
evolve without waiting on our release. Net: probably a wash

So, overall, I'd say it was a good decision. Though we never had the same
number of contribs that Hadoop seems to have sprouted.

-Todd


>
> On Tue, Feb 1, 2011 at 1:02 AM, Allen Wittenauer
> <awittenauer@linkedin.com> wrote:
> >
> > On Jan 31, 2011, at 3:23 PM, Todd Lipcon wrote:
> >
> >> On Sun, Jan 30, 2011 at 11:19 PM, Owen O'Malley <omalley@apache.org>
> wrote:
> >>
> >>>
> >>> Also note that pushing code out of Hadoop has a high cost. There are at
> >>> least 3 forks of the hadoop-gpl-compression code. That creates a lot of
> >>> confusion for the users. A lot of users never go to the work to figure
> out
> >>> which fork and branch of hadoop-gpl-compression work with the version
> of
> >>> Hadoop they installed.
> >>>
> >>>
> >> Indeed it creates confusion, but in my opinion it has been very
> successful
> >> modulo that confusion.
> >
> >        I'm not sure how the above works with what you wrote below:
> >
> >> In particular, Kevin and I (who each have a repo on github but basically
> >> co-maintain a branch) have done about 8 bugfix releases of LZO in the
> last
> >> year. The ability to take a bug and turn it around into a release within
> a
> >> few days has been very beneficial to the users. If it were part of core
> >> Hadoop, people would be forced to live with these blocker bugs for
> months at
> >> a time between dot releases.
> >
> >        So is the expectation that users would have to follow bread crumbs
> to the github dumping ground, then try to figure out which repo is the
> 'better' choice for their usage?   Using LZO as an example, it appears we
> have a choice of kevin's, your's, or the master without even taking into
> consideration any tags. That sounds like a recipe for disaster that's even
> worse than what we have today.
> >
> >
> >> IMO the more we can take non-core components and move them to separate
> >> release timelines, the better. Yes, it is harder for users, but it also
> is
> >> easier for them when they hit a bug - they don't have to wait months for
> a
> >> wholesale upgrade which might contain hundreds of other changes to core
> >> components.
> >
> >        I'd agree except for one thing:  even when users do provide
> patches to contrib components we ignore them.  How long have those patches
> for HOD been sitting there in the patch queue?  So of course they wait
> months/years--because we seemingly ignore anything that isn't important to
> us.  Unfortunately, that covers a large chunk of contrib. :(
> >
> >
> >
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--90e6ba4fc514765287049b3de63b--

From general-return-2940-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 01 22:03:42 2011
Return-Path: <general-return-2940-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53033 invoked from network); 1 Feb 2011 22:03:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Feb 2011 22:03:42 -0000
Received: (qmail 96756 invoked by uid 500); 1 Feb 2011 22:03:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96632 invoked by uid 500); 1 Feb 2011 22:03:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96624 invoked by uid 99); 1 Feb 2011 22:03:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 22:03:39 +0000
X-ASF-Spam-Status: No, hits=3.3 required=5.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 22:03:31 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p11M2sPg032056
	for <general@hadoop.apache.org>; Tue, 1 Feb 2011 14:02:54 -0800 (PST)
Received: from SP2-EX07VS07.ds.corp.yahoo.com ([98.137.59.26]) by
 SP2-EX07CAS01.ds.corp.yahoo.com ([98.137.59.37]) with mapi; Tue, 1 Feb 2011
 14:02:54 -0800
From: Todd Papaioannou <toddp@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 1 Feb 2011 14:02:51 -0800
Subject: Re: [ANNOUNCEMENT] Yahoo focusing on Apache Hadoop, discontinuing
 "The Yahoo Distribution of Hadoop"
Thread-Topic: [ANNOUNCEMENT] Yahoo focusing on Apache Hadoop, discontinuing
 "The Yahoo Distribution of Hadoop"
Thread-Index: AcvCW8aSawLxVZd6Q4i7ZYySA7QPOQ==
Message-ID: <C96DC125.467C%toddp@yahoo-inc.com>
In-Reply-To: <AANLkTikW1OEoT5DSzYnpWyyuaraLZ_mjf6p7O_pRaqRW@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-MacOutlook/14.2.0.101115
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C96DC125467Ctoddpyahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C96DC125467Ctoddpyahooinccom_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Yes. We have been and continue to be firm believers in Apache and the value=
 of Open Source software, as you can see from our track record to date of c=
ontributing heavily to Hadoop and donating Pig, ZooKeeper, Avro, etc. We ar=
e excited about their potential and we hope others will find them useful to=
o.

ToddP

On 1/31/11 7:44 PM, "Jeff Hammerbacher" <hammer@cloudera.com<mailto:hammer@=
cloudera.com>> wrote:

Excellent news! Will you also make Howl, Oozie, and Yarn Apache projects as
well?

On Mon, Jan 31, 2011 at 7:27 PM, Eric Baldeschwieler
<eric14@yahoo-inc.com<mailto:eric14@yahoo-inc.com>>wrote:

Hi Folks,

I'm pleased to announce that after some reflection, Yahoo! has decided to
discontinue the  "The Yahoo Distribution of Hadoop" and focus on Apache
Hadoop.  We plan to remove all references to a Yahoo distribution from our
website (developer.yahoo.com/hadoop), close our github repo (
yahoo.github.com/hadoop-common) and focus on working more closely with the
Apache community.  Our intent is to return to helping Apache produce binary
releases of Apache Hadoop that are so bullet proof that Yahoo and other
production Hadoop users can run them unpatched on their clusters.

Until Hadoop 0.20, Yahoo committers worked as release masters to produce
binary Apache Hadoop releases that the entire community used on their
clusters.    As the community grew, we have experiment with using the
"Yahoo! Distribution of Hadoop" as the vehicle to share our work.
  Unfortunately, Apache is no longer the obvious place to go for Hadoop
releases.  The Yahoo! team wants to return to a world where anyone can
download and directly use releases of Hadoop from Apache.  We want to
contribute to the stabilization and testing of those releases.  We also wan=
t
to share our regular program of sustaining engineering that backports minor
feature enhancements into new dot releases on a regular basis, so that the
world sees regular improvements coming from Apache every few months, not
years.

Recently the Apache Hadoop community has been very turbulent.  Over the
last few months we have been developing Hadoop enhancements in our internal
git repository while doing a complete review of our options. Our commitment
to open sourcing our work was never in doubt (see http://yhoo.it/e8p3Dd),
but the future of the "Yahoo distribution of Hadoop" was far from clear.
  We've concluded that focusing on Apache Hadoop is the way forward.  We
believe that more focus on communicating our goals to the Apache Hadoop
community, and more willingness to compromise on how we get to those goals,
will help us get back to making Hadoop even better.

Unfortunately, we now have to sort out how to contribute several
person-years worth of work to Apache to let us unwind the Yahoo! git
repositories.  We currently run two lines of Hadoop development, our
sustaining program (hadoop-0.20-sustaining) and hadoop-future.
  Hadoop-0.20-sustaining is the stable version of Hadoop we currently run o=
n
Yahoo's 40,000 nodes.  It contains a series of fixes and enhancements that
are all backwards compatible with our "Hadoop 0.20 with security".  It is
our most stable and high performance release of Hadoop ever.  We've expende=
d
a lot of energy finding and fixing bugs in it this year. We have initiated
the process of contributing this work to Apache in the branch:
hadoop/common/branches/branch-0.20-security.  We've proposed calling this
the 20.100 release.  Once folks have had a chance to try this out and we've
had a chance to respond to their feedback, we plan to create 20.100 release
candidates and ask the community to vote on making them Apache releases.

Hadoop-future is our new feature branch.  We are working on a set of new
features for Hadoop to improve its availability, scalability and
interoperability to make Hadoop more usable in mission critical deployments=
.
You're going to see another burst of email activity from us as we work to
get hadoop-future patches socialized, reviewed and checked in.  These bulk
checkins are exceptional.  They are the result of us striving to be more
transparent.  Once we've merged our hadoop-future and hadoop-0.20-sustainin=
g
work back into Apache, folks can expect us to return to our regular
development cadence.  Looking forward, we plan to socialize our roadmaps
regularly, actively synchronize our work with other active Hadoop
contributors and develop our code collaboratively, directly in Apache.

In summary, our decision to discontinue the "Yahoo! Distribution of Hadoop"
is a commitment to working more effectively with the Apache Hadoop
community.  Our goal is to make Apache Hadoop THE open source platform for
big data.

Thanks,

E14

--

PS Here is a draft list of key features in hadoop-future:

* HDFS-1052 - Federation, the ability to support much more storage per
Hadoop cluster.

* HADOOP-6728 - A the new metrics framework

* MAPREDUCE-1220 - Optimizations for small jobs

---
PPS This is cross-posted on our blog: http://yhoo.it/i9Ww8W


--_000_C96DC125467Ctoddpyahooinccom_--

From general-return-2941-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 02 03:26:05 2011
Return-Path: <general-return-2941-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 114 invoked from network); 2 Feb 2011 03:26:04 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Feb 2011 03:26:04 -0000
Received: (qmail 38244 invoked by uid 500); 2 Feb 2011 03:26:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37929 invoked by uid 500); 2 Feb 2011 03:26:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37917 invoked by uid 99); 2 Feb 2011 03:25:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Feb 2011 03:25:59 +0000
X-ASF-Spam-Status: No, hits=-0.1 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of chenqibj@cn.ibm.com designates 202.81.31.142 as permitted sender)
Received: from [202.81.31.142] (HELO e23smtp09.au.ibm.com) (202.81.31.142)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Feb 2011 03:25:49 +0000
Received: from d23relay05.au.ibm.com (d23relay05.au.ibm.com [202.81.31.247])
	by e23smtp09.au.ibm.com (8.14.4/8.13.1) with ESMTP id p123PPjr031028
	for <general@hadoop.apache.org>; Wed, 2 Feb 2011 14:25:25 +1100
Received: from d23av04.au.ibm.com (d23av04.au.ibm.com [9.190.235.139])
	by d23relay05.au.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id p123PPoD2523156
	for <general@hadoop.apache.org>; Wed, 2 Feb 2011 14:25:25 +1100
Received: from d23av04.au.ibm.com (loopback [127.0.0.1])
	by d23av04.au.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id p123PPao022778
	for <general@hadoop.apache.org>; Wed, 2 Feb 2011 14:25:25 +1100
Received: from d23m0037.cn.ibm.com (d23m0037.cn.ibm.com [9.181.2.105])
	by d23av04.au.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id p123POnl022769
	for <general@hadoop.apache.org>; Wed, 2 Feb 2011 14:25:25 +1100
Subject: Keith Chen is out of office for Spring Festival holiday
Auto-Submitted: auto-generated
From: Qi BJ Chen <chenqibj@cn.ibm.com>
To: general@hadoop.apache.org
Message-ID: <OF1F942A80.FC350621-ON4825782B.0012CDBF-4825782B.0012CDBF@cn.ibm.com>
Date: Wed, 2 Feb 2011 11:25:23 +0800
X-MIMETrack: Serialize by Router on D23M0037/23/M/IBM(Release 8.5.1FP4|July 25, 2010) at
 02/02/2011 11:25:23
MIME-Version: 1.0
Content-type: multipart/alternative; 
	Boundary="0__=C7BBF2B8DF814B2F8f9e8a93df938690918cC7BBF2B8DF814B2F"
Content-Disposition: inline
X-Virus-Checked: Checked by ClamAV on apache.org

--0__=C7BBF2B8DF814B2F8f9e8a93df938690918cC7BBF2B8DF814B2F
Content-type: text/plain; charset=US-ASCII


I will be out of the office starting  2011-02-02 and will not return until
2011-02-09.

--0__=C7BBF2B8DF814B2F8f9e8a93df938690918cC7BBF2B8DF814B2F--


From general-return-2942-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 02 13:44:46 2011
Return-Path: <general-return-2942-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73264 invoked from network); 2 Feb 2011 13:44:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Feb 2011 13:44:46 -0000
Received: (qmail 15490 invoked by uid 500); 2 Feb 2011 13:44:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15083 invoked by uid 500); 2 Feb 2011 13:44:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15069 invoked by uid 99); 2 Feb 2011 13:44:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Feb 2011 13:44:40 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of erlfilho@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Feb 2011 13:44:33 +0000
Received: by fxm2 with SMTP id 2so8635071fxm.35
        for <general@hadoop.apache.org>; Wed, 02 Feb 2011 05:44:13 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=fBjvU2yGi682FisbJ7Un0/AAByHkcT6sjGtxJw6kKzM=;
        b=cW53EpJw6i+x66jn7SGRtMG80MIZK2FCzI27J3oHwIp8YrmjccXK8V7vS7/R7uT4VQ
         q58z8KUrqTK3DM/mJbjH7FC2Vi27WpP6e2RZ/+Q5vXZE1gxPSzbX/i7yZQbpooAvvXJQ
         ZNTVFapSUCpXALjIO2vs+s/BrS9hxYqFZWyY0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=A6l6FwNAEjdR0BG6bIDFgMJTOdVszaHu39Renccr1cmBoB40yuGplLlojwegv3Ujms
         K3u5xMRNl4c6wbf37bGs3LNTSLllVRfCh0uTFmZg3GEEsm7tZgo5Y+FM0NNVpo+m/fgC
         gwPX0U2hp4oSh+6uNGN5x+zo3y7GqP6kr8PyQ=
MIME-Version: 1.0
Received: by 10.223.83.144 with SMTP id f16mr4983797fal.4.1296654250546; Wed,
 02 Feb 2011 05:44:10 -0800 (PST)
Received: by 10.223.78.134 with HTTP; Wed, 2 Feb 2011 05:44:10 -0800 (PST)
Date: Wed, 2 Feb 2011 11:44:10 -0200
Message-ID: <AANLkTinP7OoCQVQvfu-q3xekwpp4vLekFh-7pC-oM-t0@mail.gmail.com>
Subject: MRUnit and Herriot
From: Edson Ramiro <erlfilho@gmail.com>
To: Hadoop General <general@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=20cf3054a73700156c049b4cd82d
X-Virus-Checked: Checked by ClamAV on apache.org

--20cf3054a73700156c049b4cd82d
Content-Type: text/plain; charset=ISO-8859-1

Hi all,

Plz, could you explain me the difference between MRUnit and Herriot?

I've read the documentation of both and they seem very similar to me.

Is Herriot an evolution of MRUnit?

What can Herriot do that MRUnit can't?

Thanks in Advance

--
Edson Ramiro Lucas Filho
{skype, twitter, gtalk}: erlfilho
http://www.inf.ufpr.br/erlf07/

--20cf3054a73700156c049b4cd82d--

From general-return-2943-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 02 16:47:25 2011
Return-Path: <general-return-2943-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68413 invoked from network); 2 Feb 2011 16:47:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Feb 2011 16:47:25 -0000
Received: (qmail 99038 invoked by uid 500); 2 Feb 2011 16:47:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98507 invoked by uid 500); 2 Feb 2011 16:47:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98494 invoked by uid 99); 2 Feb 2011 16:47:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Feb 2011 16:47:18 +0000
X-ASF-Spam-Status: No, hits=-1997.8 required=5.0
	tests=ALL_TRUSTED,HTML_MESSAGE,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 02 Feb 2011 16:47:16 +0000
Received: (qmail 68258 invoked by uid 99); 2 Feb 2011 16:46:54 -0000
Received: from localhost.apache.org (HELO mail-bw0-f48.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Feb 2011 16:46:54 +0000
Received: by bwz8 with SMTP id 8so772202bwz.35
        for <multiple recipients>; Wed, 02 Feb 2011 08:46:52 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.119.133 with SMTP id z5mr8462721bkq.72.1296665204755; Wed,
 02 Feb 2011 08:46:44 -0800 (PST)
Received: by 10.204.100.134 with HTTP; Wed, 2 Feb 2011 08:46:44 -0800 (PST)
In-Reply-To: <AANLkTinP7OoCQVQvfu-q3xekwpp4vLekFh-7pC-oM-t0@mail.gmail.com>
References: <AANLkTinP7OoCQVQvfu-q3xekwpp4vLekFh-7pC-oM-t0@mail.gmail.com>
Date: Wed, 2 Feb 2011 08:46:44 -0800
Message-ID: <AANLkTimLjpf0ze1kgWRLa9RVJrP0sLkBR0n08jZK1Owi@mail.gmail.com>
Subject: Re: MRUnit and Herriot
From: "Owen O'Malley" <omalley@apache.org>
To: common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c5a81cec0d74049b4f645d
X-Virus-Checked: Checked by ClamAV on apache.org

--001636c5a81cec0d74049b4f645d
Content-Type: text/plain; charset=UTF-8

Please keep user questions off of general and use the user lists instead.
This is defined here <http://hadoop.apache.org/mailing_lists.html>.

MRUnit is for testing user's MapReduce applications. Herriot is for testing
the framework in the presence of failures.

-- Owen

On Wed, Feb 2, 2011 at 5:44 AM, Edson Ramiro <erlfilho@gmail.com> wrote:

> Hi all,
>
> Plz, could you explain me the difference between MRUnit and Herriot?
>
> I've read the documentation of both and they seem very similar to me.
>
> Is Herriot an evolution of MRUnit?
>
> What can Herriot do that MRUnit can't?
>
> Thanks in Advance
>
> --
> Edson Ramiro Lucas Filho
> {skype, twitter, gtalk}: erlfilho
> http://www.inf.ufpr.br/erlf07/
>

--001636c5a81cec0d74049b4f645d--

From general-return-2944-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 04 16:59:34 2011
Return-Path: <general-return-2944-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 7516 invoked from network); 4 Feb 2011 16:59:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Feb 2011 16:59:34 -0000
Received: (qmail 98801 invoked by uid 500); 4 Feb 2011 16:59:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98236 invoked by uid 500); 4 Feb 2011 16:59:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98222 invoked by uid 99); 4 Feb 2011 16:59:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Feb 2011 16:59:27 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of michael_segel@hotmail.com designates 65.55.34.21 as permitted sender)
Received: from [65.55.34.21] (HELO col0-omc1-s11.col0.hotmail.com) (65.55.34.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Feb 2011 16:59:21 +0000
Received: from COL117-W56 ([65.55.34.8]) by col0-omc1-s11.col0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Fri, 4 Feb 2011 08:59:00 -0800
Message-ID: <COL117-W560583A56E3934479FF7478FE60@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_94f9c8d2-c36d-47c6-abb2-1537c19d907b_"
X-Originating-IP: [65.167.11.254]
From: Michael Segel <michael_segel@hotmail.com>
To: <general@hadoop.apache.org>
Subject: Next CHUG (Chicago Hadoop User Group) meeting is Feb 16th @
 Jefferson Tap
Date: Fri, 4 Feb 2011 10:59:00 -0600
Importance: Normal
MIME-Version: 1.0
X-OriginalArrivalTime: 04 Feb 2011 16:59:00.0451 (UTC) FILETIME=[D1CCB330:01CBC48C]

--_94f9c8d2-c36d-47c6-abb2-1537c19d907b_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


At our last meeting we had a great turnout for Chris Cooper's talk on Avro!

The next CHUG meeting will be a departure from our normal routine. Instead =
of having a speaker=2C we're meeting at Jefferson Tap (upstairs) in a more =
informal 'networking' session.=20

There is no set agenda and the meeting is a way for people to ask others qu=
estions about what they are doing or how to do things in the Hadoop environ=
ment. (Pig=2C Hive=2C EC2=2C HBase schema design=2C hardware choices=2C mis=
c scripts=2C etc...)

So if you're in Chicago and didn't know that we had a meetup group=2C now's=
 the time to come downtown and meet your fellow Hadoopers. :-)

For more information check out our meetup page: http://www.meetup.com/Chica=
go-area-Hadoop-User-Group-CHUG/#calendar

Please join the group and RSVP so I can get some sort of a head count.=20

Thx=20

-Mike

PS. For those who aren't in Chicago=2C you're really missing out on some be=
autiful weather. If you're planning a trip to Chicago=2C check out our meet=
ing dates and times.=20
 		 	   		  =

--_94f9c8d2-c36d-47c6-abb2-1537c19d907b_--

From general-return-2945-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Feb 05 01:57:08 2011
Return-Path: <general-return-2945-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54573 invoked from network); 5 Feb 2011 01:57:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Feb 2011 01:57:08 -0000
Received: (qmail 84208 invoked by uid 500); 5 Feb 2011 01:57:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84152 invoked by uid 500); 5 Feb 2011 01:57:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84144 invoked by uid 99); 5 Feb 2011 01:57:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Feb 2011 01:57:05 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mbhandarkar@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Feb 2011 01:57:00 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:Accept-Language:
   Content-Language:X-MS-Has-Attach:X-MS-TNEF-Correlator:
   x-originating-ip:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version;
  b=vo8Ugbbap/pb6xd6XM1Lsn6iOP+7XEMcH0a+y99NgqaBmO+SFf475fMZ
   KMT9zU8w6jmS8sGlAQJ/igcYwswtJibgmy1OqrKWkasB13zTi1WnMsDFX
   KNx2Nd/2UCD/+ek;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=mbhandarkar@linkedin.com; q=dns/txt;
  s=proddkim; t=1296871015; x=1328407015;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Milind=20Bhandarkar=20<mbhandarkar@linkedin.com>
   |Subject:=20blog.hadoop.com|Date:=20Sat,=205=20Feb=202011
   =2001:57:53=20+0000|Message-ID:=20<C1B49EB9-2D39-4674-988
   9-494FF98EAEA6@linkedin.com>|To:=20"general@hadoop.apache
   .org"=20<general@hadoop.apache.org>|MIME-Version:=201.0
   |Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<3DC9FB43EDD7FA41A763B97517EB8DE0@linkedin
   .com>;
  bh=Wi70zs7AVoX0BE6zJHWk8J+J2JtnZITe+IuW8BdeFh4=;
  b=mvQiTOb/QOX/ZwxPfZUywtMNZBjOxg+p1pdROurJ3JH/9fy3SQm1L+JR
   n6+mKtUnfeEi9e3yzFUS7YCaWJdK2UL9T1B1Zb8+Nb2CfilbBV+bNxcAd
   Omw4ku+YRi8Ik8K;
X-IronPort-AV: E=Sophos;i="4.60,429,1291622400"; 
   d="scan'208";a="20431390"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi id 14.01.0251.000; Fri, 4
 Feb 2011 17:57:53 -0800
From: Milind Bhandarkar <mbhandarkar@linkedin.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: blog.hadoop.com
Thread-Topic: blog.hadoop.com
Thread-Index: AQHLxNgZ20aoUO0cUE2q49XSbf1HsQ==
Date: Sat, 5 Feb 2011 01:57:53 +0000
Message-ID: <C1B49EB9-2D39-4674-9889-494FF98EAEA6@linkedin.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <3DC9FB43EDD7FA41A763B97517EB8DE0@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Hey Folks,

Back in 2007, when I was part of the Hadoop team at Yahoo, and Hadoop usage=
 had not yet exploded, I had created hadoop.wordpress.com. Also, the subdom=
ain blog.hadoop.com points to hadoop.wordpress.com.

Despite initial enthusiasm of the small hadoop community (i.e. 8-9 cubicles=
 around my cubicle :-) then, there were no posts on that blog.

Does anyone want to take it over and create something useful ?

Please let me know. If you have an account on wordpress.com, I can transfer=
 it to you easily.

Or does one need permission from Apache legal folks to do this ?

- milind


P.S. Whois indicates that some one called "Cutting, Doug  dns@lucene.com" o=
wns hadoop.com, and so blog.hadoop.com pointing to hadoop.wordpress.com is =
Doug's doing. You will have to negotiate with him whether after the admin c=
hange, the cname still remains :-)

---
Milind Bhandarkar
mbhandarkar@linkedin.com




From general-return-2946-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Feb 05 03:07:37 2011
Return-Path: <general-return-2946-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88641 invoked from network); 5 Feb 2011 03:07:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Feb 2011 03:07:36 -0000
Received: (qmail 62377 invoked by uid 500); 5 Feb 2011 03:07:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61982 invoked by uid 500); 5 Feb 2011 03:07:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61974 invoked by uid 99); 5 Feb 2011 03:07:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Feb 2011 03:07:31 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Feb 2011 03:07:25 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p1536rFZ016648
	for <general@hadoop.apache.org>; Fri, 4 Feb 2011 19:06:53 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1296875213;
	bh=bxR3SLaModcsdVHabvS7gxax+MgEZKBE5Nfq3IOBVXo=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=sZzkEkhuXKX395i4mV+T8ap2YJATZhc1rlrD0iYmH4HZe0WJindmny+2WqjpDb7FT
	 dk2s1CBROAmM9V2lhwwQuKa1ljZY20XudzrCklR51I/0+JhRVpkqyXDsLLW5UiB/yL
	 0pH1wdMTKUPZQWnZ6qsq6C2AakVOIDbntvQOYzLo=
Message-Id: <33F071FF-D0D0-40D1-9767-F16D44A29B70@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <C1B49EB9-2D39-4674-9889-494FF98EAEA6@linkedin.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: blog.hadoop.com
Date: Fri, 4 Feb 2011 19:06:53 -0800
References: <C1B49EB9-2D39-4674-9889-494FF98EAEA6@linkedin.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

Yes, I'd be interested assuming the legal stuff doesn't bite me. :)

On Feb 4, 2011, at 5:57 PM, Milind Bhandarkar wrote:

> Hey Folks,
>
> Back in 2007, when I was part of the Hadoop team at Yahoo, and  
> Hadoop usage had not yet exploded, I had created  
> hadoop.wordpress.com. Also, the subdomain blog.hadoop.com points to  
> hadoop.wordpress.com.
>
> Despite initial enthusiasm of the small hadoop community (i.e. 8-9  
> cubicles around my cubicle :-) then, there were no posts on that blog.
>
> Does anyone want to take it over and create something useful ?
>
> Please let me know. If you have an account on wordpress.com, I can  
> transfer it to you easily.
>
> Or does one need permission from Apache legal folks to do this ?
>
> - milind
>
>
> P.S. Whois indicates that some one called "Cutting, Doug  dns@lucene.com 
> " owns hadoop.com, and so blog.hadoop.com pointing to  
> hadoop.wordpress.com is Doug's doing. You will have to negotiate  
> with him whether after the admin change, the cname still remains :-)
>
> ---
> Milind Bhandarkar
> mbhandarkar@linkedin.com
>
>
>


From general-return-2947-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Feb 05 03:15:05 2011
Return-Path: <general-return-2947-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 89970 invoked from network); 5 Feb 2011 03:15:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Feb 2011 03:15:05 -0000
Received: (qmail 68878 invoked by uid 500); 5 Feb 2011 03:15:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68689 invoked by uid 500); 5 Feb 2011 03:15:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68681 invoked by uid 99); 5 Feb 2011 03:15:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Feb 2011 03:15:00 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of gaur.vbagga@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Feb 2011 03:14:55 +0000
Received: by wwd20 with SMTP id 20so2967758wwd.29
        for <general@hadoop.apache.org>; Fri, 04 Feb 2011 19:14:34 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type:content-transfer-encoding;
        bh=0Z5A9P3iX7C14fprHdJUPTgEZNDK7UYRMckNY7fWPkA=;
        b=EaoEkWYyObkmDMeRQykbqjIZWSJFKL5+Q/bx7lHDYHkXRUhby8XKqWxtwN9xczUsVR
         gJyCoGOxnHq6yRyiIaw8XIQK15DuE3oOoq8AhNxHBD1K0maU5WFuwhYkLIf3AkCFvW5V
         tGoVUypLBcZB696dZ+Bbb2L2l+AWUhg6hWp5M=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=yHwOLJq+ZCsUKCbuI+mRa6uZ86RxEhRYfGRuKFnOre0Xyh8c38JmvvXYRtoQO/vRCW
         xtaZ8eNVX9BgNTmuRyZTlEBMMEnMCQy92zbagpQtF3mrYSIiM78iqXPoSBAFPKm/QpTb
         Vf3aLsS2PNjKfMpKE837TsOWduRBT7THsxO6c=
Received: by 10.216.62.212 with SMTP id y62mr13541408wec.9.1296875674428; Fri,
 04 Feb 2011 19:14:34 -0800 (PST)
MIME-Version: 1.0
Received: by 10.216.9.68 with HTTP; Fri, 4 Feb 2011 19:14:13 -0800 (PST)
In-Reply-To: <33F071FF-D0D0-40D1-9767-F16D44A29B70@yahoo-inc.com>
References: <C1B49EB9-2D39-4674-9889-494FF98EAEA6@linkedin.com> <33F071FF-D0D0-40D1-9767-F16D44A29B70@yahoo-inc.com>
From: gaurav bagga <gaur.vbagga@gmail.com>
Date: Fri, 4 Feb 2011 19:14:13 -0800
Message-ID: <AANLkTimysaq0cBXZbTmpAxLc9=OjLJu4k-g28Kku99oe@mail.gmail.com>
Subject: Re: blog.hadoop.com
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Arun

I can help you with proof reading and other stuff if you want.
secondary help to who ever takes on to the blog.

Gaurav

On Fri, Feb 4, 2011 at 7:06 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:
> Yes, I'd be interested assuming the legal stuff doesn't bite me. :)
>
> On Feb 4, 2011, at 5:57 PM, Milind Bhandarkar wrote:
>
>> Hey Folks,
>>
>> Back in 2007, when I was part of the Hadoop team at Yahoo, and Hadoop
>> usage had not yet exploded, I had created hadoop.wordpress.com. Also, th=
e
>> subdomain blog.hadoop.com points to hadoop.wordpress.com.
>>
>> Despite initial enthusiasm of the small hadoop community (i.e. 8-9
>> cubicles around my cubicle :-) then, there were no posts on that blog.
>>
>> Does anyone want to take it over and create something useful ?
>>
>> Please let me know. If you have an account on wordpress.com, I can
>> transfer it to you easily.
>>
>> Or does one need permission from Apache legal folks to do this ?
>>
>> - milind
>>
>>
>> P.S. Whois indicates that some one called "Cutting, Doug =A0dns@lucene.c=
om"
>> owns hadoop.com, and so blog.hadoop.com pointing to hadoop.wordpress.com=
 is
>> Doug's doing. You will have to negotiate with him whether after the admi=
n
>> change, the cname still remains :-)
>>
>> ---
>> Milind Bhandarkar
>> mbhandarkar@linkedin.com
>>
>>
>>
>
>

From general-return-2948-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Feb 05 13:53:40 2011
Return-Path: <general-return-2948-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 442 invoked from network); 5 Feb 2011 13:53:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Feb 2011 13:53:40 -0000
Received: (qmail 71898 invoked by uid 500); 5 Feb 2011 13:53:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71451 invoked by uid 500); 5 Feb 2011 13:53:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71443 invoked by uid 99); 5 Feb 2011 13:53:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Feb 2011 13:53:34 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [98.139.91.88] (HELO nm18.bullet.mail.sp2.yahoo.com) (98.139.91.88)
    by apache.org (qpsmtpd/0.29) with SMTP; Sat, 05 Feb 2011 13:53:24 +0000
Received: from [98.139.91.66] by nm18.bullet.mail.sp2.yahoo.com with NNFMP; 05 Feb 2011 13:53:02 -0000
Received: from [98.139.91.8] by tm6.bullet.mail.sp2.yahoo.com with NNFMP; 05 Feb 2011 13:53:02 -0000
Received: from [127.0.0.1] by omp1008.mail.sp2.yahoo.com with NNFMP; 05 Feb 2011 13:53:02 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 660368.29497.bm@omp1008.mail.sp2.yahoo.com
Received: (qmail 89507 invoked by uid 60001); 5 Feb 2011 13:53:02 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1296913982; bh=Om7yTqDWWC/7qnTX8au1tReBoWQ2we5KRKOv/2ZxJeY=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=xdk7WGZMotxOFa/aCqWRNHOkaMYnHcH1KyzZoCAq97l2cyOy3SJdyMueWBBXUibc8R95HjWnXhfiVu2hHtJ5PWItksKAir45eOLS395UMPh8mKEISyR7HYzCjgvvEt6evDqLvdON6B/lkuAFSywUiq+xoEh973jw0ECXDaZHMIc=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=GQUyRyc5XqpOnHU+HD2uKRMwM/3ODf2F8wTCBBaFANiyRs6/HMaTqNHqmduTVvPyVk6dpQBbRsTysCtEuJEymNyr1xb1bwU3bA4OAVUiVFb6X8M+p7DXRsOtvmd7ctB+oN6WDSkFhIKXebUy3yES3YRfZ3+wDlB3ktmQYa4hkHc=;
Message-ID: <329811.88501.qm@web112114.mail.gq1.yahoo.com>
X-YMail-OSG: JQKuTbEVM1mGGOHgkIdg1er4eH5LWOLeYUi4W0sAfl3Dflf
 UMS2TyXYH3xmsQ7.7uxlc9I_d.JhsC9Ad3rhYeCjNPR9PUgLvWlE1PiEjIuJ
 6ZsgjDQwqDzxOfCSqXB9nza1.GAPmzwDTYArqLzXZjSqgehwRUIKhVu9VwmT
 tRwd2QWET0NItVONvRDb2.6N4G2ZSCb6VbtF3v9UmOEe9mDIZlK0XPO3bQp1
 2kaF72ELpKthPt2v8MAH9eM.PglDvi1bBNjeEhvWcKsEODGvH5xj5s2qqc4m
 UAg4XHfyLpJmy0DL0XM7vZqY.alGfbCnqkgKV57RFVjouz5avlwWzzcZ5YYO
 wXSOe23XQyIiEiLMyOtZ_N4w7DKJL3FBPP3_7iLQoLv9x0Fptw5mCTEWh_lX
 b1AY6lzjMaIo-
Received: from [129.132.85.169] by web112114.mail.gq1.yahoo.com via HTTP; Sat, 05 Feb 2011 05:53:02 PST
X-Mailer: YahooMailClassic/11.4.20 YahooMailWebService/0.8.108.291010
Date: Sat, 5 Feb 2011 05:53:02 -0800 (PST)
From: Grandl Robert <rgrandl@yahoo.com>
Subject: Multiple Queue question
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-179121995-1296913982=:88501"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-179121995-1296913982=:88501
Content-Type: text/plain; charset=us-ascii

Hi all,

I am trying to submit jobs to different queues in hadoop-0.20.2

I configured conf/mapred-site.xml
<property>
<name>mapred.queue.names</name>
<value>queue1, queue2</value>
</property>

Then I was starting wordcount job with following configuration:

<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>queue1</value>
</property>
</configuration>

I am trying to run Hadoop with default FIFO scheduler.

However the wordcount job appears on 
http://localhost:50030/jobtracker.jsp that is was submitted
to both queue1 and queue2.

Did I forgot to configure something ?

Thank you very much,
Robert





      
--0-179121995-1296913982=:88501--

From general-return-2949-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Feb 05 15:36:38 2011
Return-Path: <general-return-2949-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 36023 invoked from network); 5 Feb 2011 15:36:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Feb 2011 15:36:38 -0000
Received: (qmail 37587 invoked by uid 500); 5 Feb 2011 15:36:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36679 invoked by uid 500); 5 Feb 2011 15:36:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36663 invoked by uid 99); 5 Feb 2011 15:36:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Feb 2011 15:36:31 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Feb 2011 15:36:25 +0000
Received: from [192.168.1.71] (snvvpn1-10-72-244-c61.hq.corp.yahoo.com [10.72.244.61])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p15FZleZ026124;
	Sat, 5 Feb 2011 07:35:47 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1296920147;
	bh=pYptUOIAC9MotsnzGq5J+8jDMGKCuanPp/9GVLfzPX0=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Subject:Mime-Version:Date:References;
	b=NqBaM5vf1LqoAMpBlfnVHTRJWottjBWDrSdU6uoDgNJwLwu7LeLobTxSSHvmbx1Cd
	 hLRwaYp3j/4CMOotaaIO0+M0yDqcuqFqTsvbKy2XG7mijcrnU/EJ/t4jtveMtqLpla
	 PXoKYXBqipIxd6zJmzmdAA8mTVVCPWvTCsskRFlc=
Message-Id: <574B3B67-3B40-4279-A8F0-75CD808ED7C9@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: mapreduce-user@hadoop.apache.org
In-Reply-To: <329811.88501.qm@web112114.mail.gq1.yahoo.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Subject: Re: Multiple Queue question
Mime-Version: 1.0 (Apple Message framework v936)
Date: Sat, 5 Feb 2011 07:35:47 -0800
References: <329811.88501.qm@web112114.mail.gq1.yahoo.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

Moving to mapreduce-user@, bcc general@. Please use project-specific  
lists.

The default FIFO scheduler doesn't support queues. You'll need to use  
the CapacityScheduler.

Some details:
http://hadoop.apache.org/common/docs/r0.20.0/capacity_scheduler.html

Arun

On Feb 5, 2011, at 5:53 AM, Grandl Robert wrote:

> Hi all,
>
> I am trying to submit jobs to different queues in hadoop-0.20.2
>
> I configured conf/mapred-site.xml
> <property>
> <name>mapred.queue.names</name>
> <value>queue1, queue2</value>
> </property>
>
> Then I was starting wordcount job with following configuration:
>
> <configuration>
> <property>
> <name>mapred.job.queue.name</name>
> <value>queue1</value>
> </property>
> </configuration>
>
> I am trying to run Hadoop with default FIFO scheduler.
>
> However the wordcount job appears on
> http://localhost:50030/jobtracker.jsp that is was submitted
> to both queue1 and queue2.
>
> Did I forgot to configure something ?
>
> Thank you very much,
> Robert
>
>
>
>
>


From general-return-2950-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 07 16:26:02 2011
Return-Path: <general-return-2950-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 75994 invoked from network); 7 Feb 2011 16:26:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Feb 2011 16:26:01 -0000
Received: (qmail 14917 invoked by uid 500); 7 Feb 2011 16:26:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14462 invoked by uid 500); 7 Feb 2011 16:25:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14454 invoked by uid 99); 7 Feb 2011 16:25:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Feb 2011 16:25:55 +0000
X-ASF-Spam-Status: No, hits=4.0 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,HTML_OBFUSCATE_20_30,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of binhara@gmail.com designates 74.125.82.42 as permitted sender)
Received: from [74.125.82.42] (HELO mail-ww0-f42.google.com) (74.125.82.42)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Feb 2011 16:25:45 +0000
Received: by wwi17 with SMTP id 17so2972130wwi.5
        for <general@hadoop.apache.org>; Mon, 07 Feb 2011 08:25:25 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=HvHzB3H7HrfX5sH30VF6Xz3CpDpWwEdkmDTbOqAz/xo=;
        b=lG2OluEJpMTgR7SORhKTjGrl16CvYbDaDxw6Dw7dUV0l1EXhW582n91+SA8ZKL7fU2
         NK0P5lLIOiZXwxM8oQGwRJVOnL9DlQME1iVBoA5i9axP9BVT6wFmHx8noWtNDLv9ubW+
         GqE/1oZeVUjjPOwjtJvDECz793yTZrOpg+34g=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=ZPSNid9CiXyY8Az5OWMUvEfLaW95gITBdbXYHlXC5sV06M7ZBfIc7XIHc3sIJW+LYj
         unqhLhQhNqdl20shoCNOnn3vTfGKfZM1FiPDPHftJvJ95tVrucph3vy0ULPNDfrOed/P
         dGYzl10X70qWVw7kpVtryGa4apqpz3aCAAQyc=
MIME-Version: 1.0
Received: by 10.216.220.219 with SMTP id o69mr137929wep.57.1297095924791; Mon,
 07 Feb 2011 08:25:24 -0800 (PST)
Received: by 10.216.17.146 with HTTP; Mon, 7 Feb 2011 08:25:24 -0800 (PST)
Date: Mon, 7 Feb 2011 14:25:24 -0200
Message-ID: <AANLkTi=pgp7rM1go0Co+TkQ1+jgsHA=D25fZ+6KKOdc5@mail.gmail.com>
Subject: Programming book to mapreduce with hadoop
From: Alessandro Binhara <binhara@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364c7b91d63ad6049bb3adf9
X-Virus-Checked: Checked by ClamAV on apache.org

--0016364c7b91d63ad6049bb3adf9
Content-Type: text/plain; charset=ISO-8859-1

hello all ...

I need a indication of books for MapReduce programming with Hadoop ..
thank s

--0016364c7b91d63ad6049bb3adf9--

From general-return-2951-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 07 16:53:13 2011
Return-Path: <general-return-2951-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 86303 invoked from network); 7 Feb 2011 16:53:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Feb 2011 16:53:13 -0000
Received: (qmail 53583 invoked by uid 500); 7 Feb 2011 16:53:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53217 invoked by uid 500); 7 Feb 2011 16:53:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53208 invoked by uid 99); 7 Feb 2011 16:53:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Feb 2011 16:53:07 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Feb 2011 16:53:01 +0000
Received: by fxm2 with SMTP id 2so5278239fxm.35
        for <general@hadoop.apache.org>; Mon, 07 Feb 2011 08:52:41 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type;
        bh=/Ph46ynNhUqMiRUNHlmToZ3lQ0DzwKHIZnLDFkvTwIc=;
        b=PSkO+p83pS2hLcfKXUeIuEjTb7BM8i+sl/6eWlFtE7RqUMjUGYzp0JhGGij7xLl8s3
         LxMijBk0cupzgObbeZBZP3Lss9yizYClQUISRsrcwilqcVXz11hDt+i4269/HNOfDxdJ
         dmuVCPOTYAOTwqfJojJqi4mSqLBULWCjqNSN8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=FRJwwj0Ur4gNm/ufUeayLwGf3r+javpDH3/Y+u0MWVmQtfAcRH5ACdpz9YB6jQSNcT
         GfX2O3RggKqcCGzeJZ7mjYweUfml/nnUl90ZEeeJJROia7ZLjOg8dFLMBUm9LkJ65Mzx
         07dot6L9ZtUy5vrxVXY4L3kHMpt/EG3P+GSJI=
Received: by 10.223.116.1 with SMTP id k1mr4820961faq.51.1297097561054; Mon,
 07 Feb 2011 08:52:41 -0800 (PST)
MIME-Version: 1.0
Received: by 10.223.124.200 with HTTP; Mon, 7 Feb 2011 08:52:20 -0800 (PST)
In-Reply-To: <AANLkTi=pgp7rM1go0Co+TkQ1+jgsHA=D25fZ+6KKOdc5@mail.gmail.com>
References: <AANLkTi=pgp7rM1go0Co+TkQ1+jgsHA=D25fZ+6KKOdc5@mail.gmail.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Mon, 7 Feb 2011 22:22:20 +0530
Message-ID: <AANLkTikPhoEGVPkUCn7uH_bbruTQFp1EoU0jWKQB+mTW@mail.gmail.com>
Subject: Re: Programming book to mapreduce with hadoop
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

A list is available on Hadoop's wiki: http://wiki.apache.org/hadoop/Books

On Mon, Feb 7, 2011 at 9:55 PM, Alessandro Binhara <binhara@gmail.com> wrote:
> hello all ...
>
> I need a indication of books for MapReduce programming with Hadoop ..
> thank s
>



-- 
Harsh J
www.harshj.com

From general-return-2952-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 07 18:28:29 2011
Return-Path: <general-return-2952-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35183 invoked from network); 7 Feb 2011 18:28:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Feb 2011 18:28:29 -0000
Received: (qmail 7437 invoked by uid 500); 7 Feb 2011 18:28:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7030 invoked by uid 500); 7 Feb 2011 18:28:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7021 invoked by uid 99); 7 Feb 2011 18:28:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Feb 2011 18:28:24 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=NORMAL_HTTP_TO_IP,SPF_PASS,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of phaidinyak@local.com designates 70.183.28.5 as permitted sender)
Received: from [70.183.28.5] (HELO mail11.local.com) (70.183.28.5)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Feb 2011 18:28:18 +0000
Received: from IRV1HUBCAS01.eLiberation.com (10.1.190.40) by mail11.local.com
 (10.1.230.41) with Microsoft SMTP Server (TLS) id 8.3.83.0; Mon, 7 Feb 2011
 10:27:50 -0800
Received: from IRV1EXMB01.eLiberation.com ([fe80::895:3aed:8948:77ab]) by
 IRV1HUBCAS01.eLiberation.com ([fe80::7853:3c40:9449:94df%13]) with mapi; Mon,
 7 Feb 2011 10:27:58 -0800
From: Peter Haidinyak <phaidinyak@local.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 7 Feb 2011 10:27:54 -0800
Subject: Errors in the log
Thread-Topic: Errors in the log
Thread-Index: AcvG54IypxqBJFAwRY6G8o57/s6m5AACd4aQ
Message-ID: <321C2E54215EEB41A581FDD9DAECBC5DFCD945F6@IRV1EXMB01.eLiberation.com>
References: <AANLkTi=pgp7rM1go0Co+TkQ1+jgsHA=D25fZ+6KKOdc5@mail.gmail.com>
 <AANLkTikPhoEGVPkUCn7uH_bbruTQFp1EoU0jWKQB+mTW@mail.gmail.com>
In-Reply-To: <AANLkTikPhoEGVPkUCn7uH_bbruTQFp1EoU0jWKQB+mTW@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

HBase 0.89.20100924+28
Hadoop 0.20.2+737

During my import process I'm starting to see various warning and errors in =
my Hadoop logs. This just started to happen, the import process has been wo=
rking for awhile. I tried to put some of the errors from the logs on variou=
s machines here to see if this is a know problem.

Thanks

-Pete

Datanode log

ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration=
(172.16.2.224:50010, storageID=3DDS-118625752-172.16.2.224-50010-1294851626=
750, infoPort=3D50075,=20
ipcPort=3D50020):DataXceiver
java.io.EOFException: while trying to read 65557 bytes

ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration=
(172.16.2.224:50010, storageID=3DDS-118625752-172.16.2.224-50010-1294851626=
750, infoPort=3D50075,=20
ipcPort=3D50020):DataXceiver
java.io.IOException: Interrupted receiveBlock

WARN org.apache.hadoop.hdfs.server.protocol.InterDatanodeProtocol: Failed t=
o getBlockMetaDataInfo for block (=3Dblk_2012842016347254862_70849) from da=
tanode (=3D172.16.2.224
:50010)
java.io.IOException: Block blk_2012842016347254862_70849 length is 16906240=
 does not match block file length 16971264

WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in BlockR=
eceiver.run():=20
java.io.IOException: Broken pipe

WARN org.apache.hadoop.hdfs.server.datanode.DataNode: checkDiskError: excep=
tion:=20
java.io.IOException: Broken pipe

WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in BlockR=
eceiver.run():=20
java.io.IOException: The stream is closed



Namenode log

WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed. java=
.io.IOException: Content-Length header is not provided by the namenode when=
 trying to fetch http://0.0.0.0:50090/getimage?getimage=3D1


Secondary name node log

ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception i=
n doCheckpoint:=20
2011-02-07 08:51:15,062=20
ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: java.io.Fil=
eNotFoundException: http://caiss01a:50070/getimage?putimage=3D1&port=3D5009=
0&machine=3D0.0.0.0&token=3D-19:84946961:0:1297097472
000:1297097169213

From general-return-2953-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 07 19:42:22 2011
Return-Path: <general-return-2953-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67254 invoked from network); 7 Feb 2011 19:42:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Feb 2011 19:42:21 -0000
Received: (qmail 27181 invoked by uid 500); 7 Feb 2011 19:42:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27083 invoked by uid 500); 7 Feb 2011 19:42:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27017 invoked by uid 99); 7 Feb 2011 19:42:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Feb 2011 19:42:19 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 07 Feb 2011 19:42:18 +0000
Received: (qmail 66982 invoked by uid 99); 7 Feb 2011 19:41:57 -0000
Received: from localhost.apache.org (HELO [192.168.168.110]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Feb 2011 19:41:57 +0000
Message-ID: <4D504B06.8090200@apache.org>
Date: Mon, 07 Feb 2011 11:41:58 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: blog.hadoop.com
References: <C1B49EB9-2D39-4674-9889-494FF98EAEA6@linkedin.com> <33F071FF-D0D0-40D1-9767-F16D44A29B70@yahoo-inc.com>
In-Reply-To: <33F071FF-D0D0-40D1-9767-F16D44A29B70@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Should we instead just blog things at http://blogs.apache.org/?  Lots of 
projects have a blog there.  It should be easy to get one that permits, 
e.g., any PMC member to blog on behalf of the project.

Doug

On 02/04/2011 07:06 PM, Arun C Murthy wrote:
> Yes, I'd be interested assuming the legal stuff doesn't bite me. :)
>
> On Feb 4, 2011, at 5:57 PM, Milind Bhandarkar wrote:
>
>> Hey Folks,
>>
>> Back in 2007, when I was part of the Hadoop team at Yahoo, and Hadoop
>> usage had not yet exploded, I had created hadoop.wordpress.com. Also,
>> the subdomain blog.hadoop.com points to hadoop.wordpress.com.
>>
>> Despite initial enthusiasm of the small hadoop community (i.e. 8-9
>> cubicles around my cubicle :-) then, there were no posts on that blog.
>>
>> Does anyone want to take it over and create something useful ?
>>
>> Please let me know. If you have an account on wordpress.com, I can
>> transfer it to you easily.
>>
>> Or does one need permission from Apache legal folks to do this ?
>>
>> - milind
>>
>>
>> P.S. Whois indicates that some one called "Cutting, Doug
>> dns@lucene.com" owns hadoop.com, and so blog.hadoop.com pointing to
>> hadoop.wordpress.com is Doug's doing. You will have to negotiate with
>> him whether after the admin change, the cname still remains :-)
>>
>> ---
>> Milind Bhandarkar
>> mbhandarkar@linkedin.com
>>
>>
>>
>

From general-return-2954-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 07 20:37:55 2011
Return-Path: <general-return-2954-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95954 invoked from network); 7 Feb 2011 20:37:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Feb 2011 20:37:55 -0000
Received: (qmail 16645 invoked by uid 500); 7 Feb 2011 20:37:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16585 invoked by uid 500); 7 Feb 2011 20:37:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16577 invoked by uid 99); 7 Feb 2011 20:37:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Feb 2011 20:37:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 07 Feb 2011 20:37:52 +0000
Received: (qmail 95716 invoked by uid 99); 7 Feb 2011 20:37:32 -0000
Received: from localhost.apache.org (HELO mail-iy0-f176.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Feb 2011 20:37:32 +0000
Received: by iyb26 with SMTP id 26so5441480iyb.35
        for <general@hadoop.apache.org>; Mon, 07 Feb 2011 12:37:31 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.31.196 with SMTP id z4mr17636538ibc.199.1297111051494;
 Mon, 07 Feb 2011 12:37:31 -0800 (PST)
Received: by 10.231.148.82 with HTTP; Mon, 7 Feb 2011 12:37:31 -0800 (PST)
In-Reply-To: <4D504B06.8090200@apache.org>
References: <C1B49EB9-2D39-4674-9889-494FF98EAEA6@linkedin.com>
	<33F071FF-D0D0-40D1-9767-F16D44A29B70@yahoo-inc.com>
	<4D504B06.8090200@apache.org>
Date: Mon, 7 Feb 2011 12:37:31 -0800
Message-ID: <AANLkTim8Q4gFrNUMby1NBE3VhruQVRWtm40+EbGeaHQt@mail.gmail.com>
Subject: Re: blog.hadoop.com
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Mon, Feb 7, 2011 at 11:41 AM, Doug Cutting <cutting@apache.org> wrote:
> Should we instead just blog things at http://blogs.apache.org/? =A0Lots o=
f
> projects have a blog there. =A0It should be easy to get one that permits,
> e.g., any PMC member to blog on behalf of the project.

+1 for adding a project blog at Apache.

Is the implication that blog.hadoop.com would point there? -C

From general-return-2955-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 07 21:06:43 2011
Return-Path: <general-return-2955-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19777 invoked from network); 7 Feb 2011 21:06:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Feb 2011 21:06:43 -0000
Received: (qmail 74521 invoked by uid 500); 7 Feb 2011 21:06:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74449 invoked by uid 500); 7 Feb 2011 21:06:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74441 invoked by uid 99); 7 Feb 2011 21:06:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Feb 2011 21:06:40 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 07 Feb 2011 21:06:40 +0000
Received: (qmail 19512 invoked by uid 99); 7 Feb 2011 21:06:18 -0000
Received: from localhost.apache.org (HELO [192.168.168.110]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Feb 2011 21:06:18 +0000
Message-ID: <4D505ECB.3070802@apache.org>
Date: Mon, 07 Feb 2011 13:06:19 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: blog.hadoop.com
References: <C1B49EB9-2D39-4674-9889-494FF98EAEA6@linkedin.com>	<33F071FF-D0D0-40D1-9767-F16D44A29B70@yahoo-inc.com>	<4D504B06.8090200@apache.org> <AANLkTim8Q4gFrNUMby1NBE3VhruQVRWtm40+EbGeaHQt@mail.gmail.com>
In-Reply-To: <AANLkTim8Q4gFrNUMby1NBE3VhruQVRWtm40+EbGeaHQt@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 02/07/2011 12:37 PM, Chris Douglas wrote:
> Is the implication that blog.hadoop.com would point there? -C

Sure, I'd be happy to redirect that CNAME.  The other redirects for 
hadoop.com already point to hadoop.apache.org.  I grabbed the domain 
long ago so that squatters wouldn't abuse it as happened with nutch.com.

Doug

From general-return-2956-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 07 21:19:42 2011
Return-Path: <general-return-2956-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24719 invoked from network); 7 Feb 2011 21:19:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Feb 2011 21:19:42 -0000
Received: (qmail 96108 invoked by uid 500); 7 Feb 2011 21:19:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96050 invoked by uid 500); 7 Feb 2011 21:19:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96042 invoked by uid 99); 7 Feb 2011 21:19:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Feb 2011 21:19:39 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Feb 2011 21:19:34 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p17LJ615006930
	for <general@hadoop.apache.org>; Mon, 7 Feb 2011 13:19:06 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1297113546;
	bh=Ca82b9LGX1fYH0UNyEhkQejWMhwsYsVEQocoE8jYElI=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=QJRwcmfM5n8QdNPv6TTMiXG+Sz3MVxPVgdVlHtsvovO+K61kBTkM1/8mpGulbDyD1
	 jfC4cllannVmbJZSwvsOEnjo8jpWpWdvrOF0g2lW2eUJB1N9/RBhyVXyQlEUEFOMp1
	 /MbUYkmdRlzvNf/u6N6pHjLHDJME03PGZ1ga0bQo=
Message-Id: <34F0EE58-557D-4131-AB0E-78128E9CBFF9@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <4D504B06.8090200@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: blog.hadoop.com
Date: Mon, 7 Feb 2011 13:19:06 -0800
References: <C1B49EB9-2D39-4674-9889-494FF98EAEA6@linkedin.com> <33F071FF-D0D0-40D1-9767-F16D44A29B70@yahoo-inc.com> <4D504B06.8090200@apache.org>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

On Feb 7, 2011, at 11:41 AM, Doug Cutting wrote:

> Should we instead just blog things at http://blogs.apache.org/?   
> Lots of
> projects have a blog there.  It should be easy to get one that  
> permits,
> e.g., any PMC member to blog on behalf of the project.
>

+1

Arun



From general-return-2957-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 07 21:29:17 2011
Return-Path: <general-return-2957-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27078 invoked from network); 7 Feb 2011 21:29:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Feb 2011 21:29:17 -0000
Received: (qmail 7162 invoked by uid 500); 7 Feb 2011 21:29:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7084 invoked by uid 500); 7 Feb 2011 21:29:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7076 invoked by uid 99); 7 Feb 2011 21:29:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Feb 2011 21:29:15 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Feb 2011 21:29:11 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=XEtbEWl7vDCmK/iIj4J9vVzbM7eCGl1/LjpwZaZYLNngk1Q5GvP0u3QS
   DhTt2E5EyGDaeRpir+dNHWjxOHvSDZayWD3eLfSvl99+F3c0SIfuZStHb
   Kc3PHZjkBCY+QaU;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1297114146; x=1328650146;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20blog.hadoop.com|Date:=20Mon,=207=20Feb
   =202011=2021:30:09=20+0000|Message-ID:=20<79DED3D5-2F6C-4
   942-964D-669F9DCC61FE@linkedin.com>|To:=20"<general@hadoo
   p.apache.org>"=20<general@hadoop.apache.org>
   |MIME-Version:=201.0|Content-Transfer-Encoding:=20quoted-
   printable|Content-ID:=20<E70737D16D58364490B696E5EA885249
   @linkedin.com>|In-Reply-To:=20<4D505ECB.3070802@apache.or
   g>|References:=20<C1B49EB9-2D39-4674-9889-494FF98EAEA6@li
   nkedin.com>=0D=0A=09<33F071FF-D0D0-40D1-9767-F16D44A29B70
   @yahoo-inc.com>=0D=0A=09<4D504B06.8090200@apache.org>=0D
   =0A=20<AANLkTim8Q4gFrNUMby1NBE3VhruQVRWtm40+EbGeaHQt@mail
   .gmail.com>=0D=0A=20<4D505ECB.3070802@apache.org>;
  bh=5V8gXvZv2swQwiPNVgka0W+RN4jiytfGcDGnG844qMo=;
  b=GPXWWT1DQxiZQ/J3B1AGiVKBptUPAvf+oGfMHSqJNl0TGsxOqRyYThlf
   ci83qehZ2cPLMMQRHcoiC8n0y7jEN/AQnHcHs15lF3NW/8QwC2imL0CyC
   lY1mzebPonoj7Mu;
X-IronPort-AV: E=Sophos;i="4.60,438,1291622400"; 
   d="scan'208";a="20487558"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi id 14.01.0251.000; Mon, 7
 Feb 2011 13:30:09 -0800
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: blog.hadoop.com
Thread-Topic: blog.hadoop.com
Thread-Index: AQHLxNgZ20aoUO0cUE2q49XSbf1HsZPyv+aAgAQ6rwCAAA+GgIAACAyAgAAGSYA=
Date: Mon, 7 Feb 2011 21:30:09 +0000
Message-ID: <79DED3D5-2F6C-4942-964D-669F9DCC61FE@linkedin.com>
References: <C1B49EB9-2D39-4674-9889-494FF98EAEA6@linkedin.com>
	<33F071FF-D0D0-40D1-9767-F16D44A29B70@yahoo-inc.com>
	<4D504B06.8090200@apache.org>
 <AANLkTim8Q4gFrNUMby1NBE3VhruQVRWtm40+EbGeaHQt@mail.gmail.com>
 <4D505ECB.3070802@apache.org>
In-Reply-To: <4D505ECB.3070802@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <E70737D16D58364490B696E5EA885249@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


On Feb 7, 2011, at 1:06 PM, Doug Cutting wrote:

> On 02/07/2011 12:37 PM, Chris Douglas wrote:
>> Is the implication that blog.hadoop.com would point there? -C
>=20
> Sure, I'd be happy to redirect that CNAME.  The other redirects for hadoo=
p.com already point to hadoop.apache.org.  I grabbed the domain long ago so=
 that squatters wouldn't abuse it as happened with nutch.com.


Too bad about mapreduce.org.

;)



From general-return-2958-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 07 21:59:03 2011
Return-Path: <general-return-2958-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37461 invoked from network); 7 Feb 2011 21:59:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Feb 2011 21:59:03 -0000
Received: (qmail 62290 invoked by uid 500); 7 Feb 2011 21:59:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62180 invoked by uid 500); 7 Feb 2011 21:59:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62172 invoked by uid 99); 7 Feb 2011 21:59:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Feb 2011 21:59:01 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of atm@cloudera.com designates 74.125.83.48 as permitted sender)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Feb 2011 21:58:55 +0000
Received: by gwj15 with SMTP id 15so2113514gwj.35
        for <general@hadoop.apache.org>; Mon, 07 Feb 2011 13:58:33 -0800 (PST)
Received: by 10.150.51.8 with SMTP id y8mr4122852yby.213.1297115605246; Mon,
 07 Feb 2011 13:53:25 -0800 (PST)
MIME-Version: 1.0
Received: by 10.150.57.13 with HTTP; Mon, 7 Feb 2011 13:53:05 -0800 (PST)
In-Reply-To: <321C2E54215EEB41A581FDD9DAECBC5DFCD945F6@IRV1EXMB01.eLiberation.com>
References: <AANLkTi=pgp7rM1go0Co+TkQ1+jgsHA=D25fZ+6KKOdc5@mail.gmail.com>
 <AANLkTikPhoEGVPkUCn7uH_bbruTQFp1EoU0jWKQB+mTW@mail.gmail.com> <321C2E54215EEB41A581FDD9DAECBC5DFCD945F6@IRV1EXMB01.eLiberation.com>
From: "Aaron T. Myers" <atm@cloudera.com>
Date: Mon, 7 Feb 2011 13:53:05 -0800
Message-ID: <AANLkTim8EWNy43xMa45rkt5-x6NtFuqvv1=V9VxK6Zxi@mail.gmail.com>
Subject: Re: Errors in the log
To: cdh-user@cloudera.org
Content-Type: multipart/alternative; boundary=000e0cd6a850e224a5049bb842ec
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd6a850e224a5049bb842ec
Content-Type: text/plain; charset=ISO-8859-1

+cdh-user@cloudera.org
bcc: general@hadoop.apache.org

Hey Pete,

The general@hadoop.apache.org list is for high-level discussion of the
Apache Hadoop community (usually votes and governance issues.) A question
like this is more appropriate for a *-user list, and since judging by the
version numbers you're using CDH3b3, I've added cdh-user@cloudera.org.

Though I can't comment on the errors you're seeing in the DN log, I do
recognize both errors in your 2NN and NN logs. Those are due to a known bug
in CDH3b3 wherein the 2NN incorrectly determines its own host name during a
checkpoint, and so tells the NN it can be found at 0.0.0.0. (The
"&machine=0.0.0.0" is the giveaway.) This bug will be fixed in the next
release of CDH, but in the mean time the solution is just to configure
the "dfs.secondary.http.address"
to a valid machine name or IP address which will resolve to your 2NN.

--
Aaron T. Myers
Software Engineer, Cloudera



On Mon, Feb 7, 2011 at 10:27 AM, Peter Haidinyak <phaidinyak@local.com>wrote:

> HBase 0.89.20100924+28
> Hadoop 0.20.2+737
>
> During my import process I'm starting to see various warning and errors in
> my Hadoop logs. This just started to happen, the import process has been
> working for awhile. I tried to put some of the errors from the logs on
> various machines here to see if this is a know problem.
>
> Thanks
>
> -Pete
>
> Datanode log
>
> ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(172.16.2.224:50010,
> storageID=DS-118625752-172.16.2.224-50010-1294851626750, infoPort=50075,
> ipcPort=50020):DataXceiver
> java.io.EOFException: while trying to read 65557 bytes
>
> ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(172.16.2.224:50010,
> storageID=DS-118625752-172.16.2.224-50010-1294851626750, infoPort=50075,
> ipcPort=50020):DataXceiver
> java.io.IOException: Interrupted receiveBlock
>
> WARN org.apache.hadoop.hdfs.server.protocol.InterDatanodeProtocol: Failed
> to getBlockMetaDataInfo for block (=blk_2012842016347254862_70849) from
> datanode (=172.16.2.224
> :50010)
> java.io.IOException: Block blk_2012842016347254862_70849 length is 16906240
> does not match block file length 16971264
>
> WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in
> BlockReceiver.run():
> java.io.IOException: Broken pipe
>
> WARN org.apache.hadoop.hdfs.server.datanode.DataNode: checkDiskError:
> exception:
> java.io.IOException: Broken pipe
>
> WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in
> BlockReceiver.run():
> java.io.IOException: The stream is closed
>
>
>
> Namenode log
>
> WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed.
> java.io.IOException: Content-Length header is not provided by the namenode
> when trying to fetch http://0.0.0.0:50090/getimage?getimage=1
>
>
> Secondary name node log
>
> ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception
> in doCheckpoint:
> 2011-02-07 08:51:15,062
> ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
> java.io.FileNotFoundException:
> http://caiss01a:50070/getimage?putimage=1&port=50090&machine=0.0.0.0&token=-19:84946961:0:1297097472
> 000:1297097169213
>

--000e0cd6a850e224a5049bb842ec--

From general-return-2959-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 07 22:01:04 2011
Return-Path: <general-return-2959-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39336 invoked from network); 7 Feb 2011 22:01:04 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Feb 2011 22:01:04 -0000
Received: (qmail 66457 invoked by uid 500); 7 Feb 2011 22:01:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66390 invoked by uid 500); 7 Feb 2011 22:01:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66382 invoked by uid 99); 7 Feb 2011 22:01:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Feb 2011 22:01:01 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=NORMAL_HTTP_TO_IP,SPF_PASS,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of phaidinyak@local.com designates 70.183.28.5 as permitted sender)
Received: from [70.183.28.5] (HELO mail11.local.com) (70.183.28.5)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Feb 2011 22:00:57 +0000
Received: from IRV1HUBCAS01.eLiberation.com (10.1.190.40) by mail11.local.com
 (10.1.230.41) with Microsoft SMTP Server (TLS) id 8.3.83.0; Mon, 7 Feb 2011
 14:00:28 -0800
Received: from IRV1EXMB01.eLiberation.com ([fe80::895:3aed:8948:77ab]) by
 IRV1HUBCAS01.eLiberation.com ([fe80::7853:3c40:9449:94df%13]) with mapi; Mon,
 7 Feb 2011 14:00:35 -0800
From: Peter Haidinyak <phaidinyak@local.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 7 Feb 2011 14:00:34 -0800
Subject: RE: Errors in the log
Thread-Topic: Errors in the log
Thread-Index: AcvHEjvY23gxAHvQRSif2NDrAAonoQAACnmw
Message-ID: <321C2E54215EEB41A581FDD9DAECBC5DFCD9466E@IRV1EXMB01.eLiberation.com>
References: <AANLkTi=pgp7rM1go0Co+TkQ1+jgsHA=D25fZ+6KKOdc5@mail.gmail.com>
 <AANLkTikPhoEGVPkUCn7uH_bbruTQFp1EoU0jWKQB+mTW@mail.gmail.com>
 <321C2E54215EEB41A581FDD9DAECBC5DFCD945F6@IRV1EXMB01.eLiberation.com>
 <AANLkTim8EWNy43xMa45rkt5-x6NtFuqvv1=V9VxK6Zxi@mail.gmail.com>
In-Reply-To: <AANLkTim8EWNy43xMa45rkt5-x6NtFuqvv1=V9VxK6Zxi@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Thanks, I'll change lists in the future.

-Pete

-----Original Message-----
From: Aaron T. Myers [mailto:atm@cloudera.com]=20
Sent: Monday, February 07, 2011 1:53 PM
To: cdh-user@cloudera.org
Subject: Re: Errors in the log

+cdh-user@cloudera.org
bcc: general@hadoop.apache.org

Hey Pete,

The general@hadoop.apache.org list is for high-level discussion of the
Apache Hadoop community (usually votes and governance issues.) A question
like this is more appropriate for a *-user list, and since judging by the
version numbers you're using CDH3b3, I've added cdh-user@cloudera.org.

Though I can't comment on the errors you're seeing in the DN log, I do
recognize both errors in your 2NN and NN logs. Those are due to a known bug
in CDH3b3 wherein the 2NN incorrectly determines its own host name during a
checkpoint, and so tells the NN it can be found at 0.0.0.0. (The
"&machine=3D0.0.0.0" is the giveaway.) This bug will be fixed in the next
release of CDH, but in the mean time the solution is just to configure
the "dfs.secondary.http.address"
to a valid machine name or IP address which will resolve to your 2NN.

--
Aaron T. Myers
Software Engineer, Cloudera



On Mon, Feb 7, 2011 at 10:27 AM, Peter Haidinyak <phaidinyak@local.com>wrot=
e:

> HBase 0.89.20100924+28
> Hadoop 0.20.2+737
>
> During my import process I'm starting to see various warning and errors i=
n
> my Hadoop logs. This just started to happen, the import process has been
> working for awhile. I tried to put some of the errors from the logs on
> various machines here to see if this is a know problem.
>
> Thanks
>
> -Pete
>
> Datanode log
>
> ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(172.16.2.224:50010,
> storageID=3DDS-118625752-172.16.2.224-50010-1294851626750, infoPort=3D500=
75,
> ipcPort=3D50020):DataXceiver
> java.io.EOFException: while trying to read 65557 bytes
>
> ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(172.16.2.224:50010,
> storageID=3DDS-118625752-172.16.2.224-50010-1294851626750, infoPort=3D500=
75,
> ipcPort=3D50020):DataXceiver
> java.io.IOException: Interrupted receiveBlock
>
> WARN org.apache.hadoop.hdfs.server.protocol.InterDatanodeProtocol: Failed
> to getBlockMetaDataInfo for block (=3Dblk_2012842016347254862_70849) from
> datanode (=3D172.16.2.224
> :50010)
> java.io.IOException: Block blk_2012842016347254862_70849 length is 169062=
40
> does not match block file length 16971264
>
> WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in
> BlockReceiver.run():
> java.io.IOException: Broken pipe
>
> WARN org.apache.hadoop.hdfs.server.datanode.DataNode: checkDiskError:
> exception:
> java.io.IOException: Broken pipe
>
> WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in
> BlockReceiver.run():
> java.io.IOException: The stream is closed
>
>
>
> Namenode log
>
> WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed.
> java.io.IOException: Content-Length header is not provided by the namenod=
e
> when trying to fetch http://0.0.0.0:50090/getimage?getimage=3D1
>
>
> Secondary name node log
>
> ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception
> in doCheckpoint:
> 2011-02-07 08:51:15,062
> ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
> java.io.FileNotFoundException:
> http://caiss01a:50070/getimage?putimage=3D1&port=3D50090&machine=3D0.0.0.=
0&token=3D-19:84946961:0:1297097472
> 000:1297097169213
>

From general-return-2960-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 08 16:33:05 2011
Return-Path: <general-return-2960-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84938 invoked from network); 8 Feb 2011 16:33:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Feb 2011 16:33:05 -0000
Received: (qmail 67523 invoked by uid 500); 8 Feb 2011 16:33:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67039 invoked by uid 500); 8 Feb 2011 16:33:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67019 invoked by uid 99); 8 Feb 2011 16:33:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Feb 2011 16:32:59 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.103 as permitted sender)
Received: from [17.148.16.103] (HELO asmtpout028.mac.com) (17.148.16.103)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Feb 2011 16:32:54 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp028.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LGB00KKQ4M91C50@asmtp028.mac.com>
 for general@hadoop.apache.org; Tue, 08 Feb 2011 08:32:34 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-02-08_05:2011-02-08,2011-02-08,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1102080075
Subject: Re: Hadoop-common-trunk-Commit is failing since 01/19/2011
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <AE3E9DCF-B2D6-4EA7-A166-DB139CD76BFD@yahoo-inc.com>
Date: Tue, 08 Feb 2011 08:32:29 -0800
Message-id: <07AA15F8-B4F2-4B5B-8A65-600B31D96D83@mac.com>
References: <AANLkTinB+8KRByZFhRT-y6MiqA0urtKNs8c_XSrhP7RH@mail.gmail.com>
 <AANLkTimmcYh0hY-6ht6vaQ+S=pTGHaFxS6k9qEv=q8Cf@mail.gmail.com>
 <AANLkTinr_Exrr8VPShGKkBSMzmWX1B9nVALLLTMmvYzd@mail.gmail.com>
 <AANLkTinrheVYQm-y9n=k7TRK_BdzuZYKAWG+qMWR88ct@mail.gmail.com>
 <AANLkTi=cAhCOkqZ=n9NpBMoPzechoJmfB6KO2yL3QcVA@mail.gmail.com>
 <AANLkTi=+81g8dFMGMqP5Zc6aAkg5n0dD2QpCVg7ABzd6@mail.gmail.com>
 <7DC70A70-5DE9-4288-A44B-B5EF9C454D9B@yahoo-inc.com>
 <AANLkTin9fsb3bgsa98bma+xx_C9K06C_YNKJ6vaA7=TO@mail.gmail.com>
 <F3847AE0-3FEF-4E85-8757-FA8A7B204A64@yahoo-inc.com>
 <AANLkTikCOHYJx4Qjt=gajGTcspPai0GmsxWQFogua92Q@mail.gmail.com>
 <AE3E9DCF-B2D6-4EA7-A166-DB139CD76BFD@yahoo-inc.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

Hmm, haven't seen Hudson post build failures to the common-dev list lately.  

Ian, can you check that hudson@hudson.apache.org is still subscribed to common-dev@.  If not, please add it.

Thx,
Nige


On Feb 1, 2011, at 11:27 AM, Giridharan Kesavan wrote:

> Konstantin,
> 
> trunk/artifacts gets populated when the jar and the tar ant target are successful.
> 
> The main reason for the build failure so far is the build abort time configuration. It was set to 30mins.
> I have increased the build abort time and the builds are going on fine 
> https://hudson.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit
> 
> 
> Thanks,
> Giri
> 
> On Feb 1, 2011, at 12:40 AM, Konstantin Shvachko wrote:
> 
>> Giri,
>> 
>> Looking at configuration of Hadoop-Common-trunk-Commit/
>> There seems to be errors in the Post-build Actions.
>> It is complaining that
>> 'trunk' exists but not 'trunk/artifacts/...'
>> Is it possible that this misconfiguration is the reason of failures?
>> 
>> --Konstantin
>> 
>> 
>> On Mon, Jan 31, 2011 at 4:40 PM, Giridharan Kesavan
>> <gkesavan@yahoo-inc.com>wrote:
>> 
>>> Konstantin,
>>> 
>>> I think I need to restart the slave which is running the commit build. For
>>> now I have published the common artifact manually from commandline.
>>> 
>>> Thanks,
>>> Giri
>>> 
>>> On Jan 31, 2011, at 4:27 PM, Konstantin Shvachko wrote:
>>> 
>>>> Giri
>>>> looks like the last run you started failed the same way as previous ones.
>>>> Any thoughts on what's going on?
>>>> Thanks,
>>>> --Konstantin
>>>> 
>>>> On Mon, Jan 31, 2011 at 3:33 PM, Giridharan Kesavan
>>>> <gkesavan@yahoo-inc.com>wrote:
>>>> 
>>>>> ant mvn-deploy would publish snapshot artifact to the apache maven
>>>>> repository as long you have the right credentials in ~/.m2/settings.xml.
>>>>> 
>>>>> For settings.xml template pls look at
>>>>> http://wiki.apache.org/hadoop/HowToRelease
>>>>> 
>>>>> I'm pushing the latest common artifacts now.
>>>>> 
>>>>> -Giri
>>>>> 
>>>>> 
>>>>> 
>>>>> On Jan 31, 2011, at 3:11 PM, Jakob Homan wrote:
>>>>> 
>>>>>> By manually installing a new core jar into the cache, I can compile
>>>>>> trunk.  Looks like we just need to kick a new Core into maven.  Are
>>>>>> there instructions somewhere for committers to do this?  I know Nigel
>>>>>> and Owen know how, but I don't know if the knowledge is diffused past
>>>>>> them.
>>>>>> -Jakob
>>>>>> 
>>>>>> 
>>>>>> On Mon, Jan 31, 2011 at 1:57 PM, Konstantin Shvachko
>>>>>> <shv.hadoop@gmail.com> wrote:
>>>>>>> Current trunk for HDFS and MapReduce are not compiling at the moment.
>>>>> Try to
>>>>>>> build trunk.
>>>>>>> This is the result of that changes to common api introduced by
>>>>> HADOOP-6904
>>>>>>> are not promoted to HDFS and MR trunks.
>>>>>>> HDFS-1335 and MAPREDUCE-2263 depend on these changes.
>>>>>>> 
>>>>>>> Common is not promoted to HDFS and MR because
>>> Hadoop-Common-trunk-Commit
>>>>>>> build is broken. See here.
>>>>>>> 
>>>>> 
>>> https://hudson.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/
>>>>>>> 
>>>>>>> As I see the last successful build was on 01/19, which integrated
>>>>>>> HADOOP-6864.
>>>>>>> I think this is when JNI changes were introduced, which cannot be
>>>>> digested
>>>>>>> by Hudson since then.
>>>>>>> 
>>>>>>> Anybody with gcc active could you please verify if the problem is
>>> caused
>>>>> by
>>>>>>> HADOOP-6864.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> --Konstantin
>>>>>>> 
>>>>>>> On Mon, Jan 31, 2011 at 1:36 PM, Ted Dunning <tdunning@maprtech.com>
>>>>> wrote:
>>>>>>> 
>>>>>>>> The has been a problem with more than one build failing (Mahout is
>>> the
>>>>> one
>>>>>>>> that I saw first) due to a change in maven version which meant that
>>> the
>>>>>>>> clover license isn't being found properly.  At least, that is the
>>> tale
>>>>> I
>>>>>>>> heard from infra.
>>>>>>>> 
>>>>>>>> On Mon, Jan 31, 2011 at 1:31 PM, Eli Collins <eli@cloudera.com>
>>> wrote:
>>>>>>>> 
>>>>>>>>> Hey Konstantin,
>>>>>>>>> 
>>>>>>>>> The only build breakage I saw from HADOOP-6904 is MAPREDUCE-2290,
>>>>>>>>> which was fixed.  Trees from trunk are compiling against each other
>>>>>>>>> for me (eg each installed to a local maven repo), perhaps the
>>> upstream
>>>>>>>>> maven repo hasn't been updated with the latest bits yet.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Eli
>>>>>>>>> 
>>>>>>>>> On Mon, Jan 31, 2011 at 12:14 PM, Konstantin Shvachko
>>>>>>>>> <shv.hadoop@gmail.com> wrote:
>>>>>>>>>> Sending this to general to attract urgent attention.
>>>>>>>>>> Both HDFS and MapReduce are not compiling since
>>>>>>>>>> HADOOP-6904 and its hdfs and MP counterparts were committed.
>>>>>>>>>> The problem is not with this patch as described below, but I think
>>>>>>>> those
>>>>>>>>>> commits should be reversed if Common integration build cannot be
>>>>>>>>>> restored promptly.
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> --Konstantin
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Fri, Jan 28, 2011 at 5:53 PM, Konstantin Shvachko
>>>>>>>>>> <shv.hadoop@gmail.com>wrote:
>>>>>>>>>> 
>>>>>>>>>>> I see Hadoop-common-trunk-Commit is failing and not sending any
>>>>>>>> emails.
>>>>>>>>>>> It times out on native compilation and aborts.
>>>>>>>>>>> Therefore changes are not integrated, and now it lead to hdfs and
>>>>>>>>> mapreduce
>>>>>>>>>>> both not compiling.
>>>>>>>>>>> Can somebody please take a look at this.
>>>>>>>>>>> The last few lines of the build are below.
>>>>>>>>>>> 
>>>>>>>>>>> Thanks
>>>>>>>>>>> --Konstantin
>>>>>>>>>>> 
>>>>>>>>>>>  [javah] [Loaded
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/build/classes/org/apache/hadoop/security/JniBasedUnixGroupsMapping.class]
>>>>>>>>>>> 
>>>>>>>>>>>  [javah] [Loaded
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>> /homes/hudson/tools/java/jdk1.6.0_11-32/jre/lib/rt.jar(java/lang/Object.class)]
>>>>>>>>>>>  [javah] [Forcefully writing file
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/build/native/Linux-i386-32/src/org/apache/hadoop/security/org_apache_hadoop_security_JniBasedUnixGroupsNetgroupMapping.h]
>>>>>>>>>>> 
>>>>>>>>>>>   [exec] checking for gcc... gcc
>>>>>>>>>>>   [exec] checking whether the C compiler works... yes
>>>>>>>>>>>   [exec] checking for C compiler default output file name...
>>>>> a.out
>>>>>>>>>>>   [exec] checking for suffix of executables...
>>>>>>>>>>> 
>>>>>>>>>>> Build timed out. Aborting
>>>>>>>>>>> Build was aborted
>>>>>>>>>>> [FINDBUGS] Skipping publisher since build result is ABORTED
>>>>>>>>>>> Publishing Javadoc
>>>>>>>>>>> Archiving artifacts
>>>>>>>>>>> Recording test results
>>>>>>>>>>> No test report files were found. Configuration error?
>>>>>>>>>>> 
>>>>>>>>>>> Recording fingerprints
>>>>>>>>>>>   [exec] Terminated
>>>>>>>>>>> Publishing Clover coverage report...
>>>>>>>>>>> No Clover report will be published due to a Build Failure
>>>>>>>>>>> No emails were triggered.
>>>>>>>>>>> Finished: ABORTED
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>> 
>>> 
> 


From general-return-2961-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 08 19:31:57 2011
Return-Path: <general-return-2961-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70716 invoked from network); 8 Feb 2011 19:31:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Feb 2011 19:31:57 -0000
Received: (qmail 95404 invoked by uid 500); 8 Feb 2011 19:31:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95039 invoked by uid 500); 8 Feb 2011 19:31:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95030 invoked by uid 99); 8 Feb 2011 19:31:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Feb 2011 19:31:52 +0000
X-ASF-Spam-Status: No, hits=2.0 required=5.0
	tests=RCVD_IN_BL_SPAMCOP_NET,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Feb 2011 19:31:47 +0000
Received: by qwe4 with SMTP id 4so4566693qwe.35
        for <general@hadoop.apache.org>; Tue, 08 Feb 2011 11:31:27 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=hfg+F9bGXn0H/kRL7zeuN/EGT70NITI0oO//POnImSk=;
        b=PxVwIlZ/43KwSJMxQmNxJgCo3hH6JSZ/LmfkexxC/5hBz+FIJ6i7YMxwWOQQnW18oP
         tDz6NJAcT56aJaZw9WAHqrx/FLwWfy573umpPk4HdfaNtbm9RMuTNoQj859v6bArR+ct
         NwJ6Wb5LX31+5neHIPMSmBg8pQQPfg322wRqI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=bOWbnxtNqKTWjqQPNkT9TaB+omEcyboYxUkaqlkqgK4QoVlX353Ng2VtESlATyBps5
         RJkYVfLbGgQbsKuDFI4iQEhgWDqIsRHWS+YA1WG7a9QSoOwmK52WC6Q82VuI2CcPRilU
         K+bQVgah09Jnu0ACr1M0Lw2rgxoDNSuMZgG2U=
Received: by 10.224.37.81 with SMTP id w17mr16028964qad.132.1297193485223;
        Tue, 08 Feb 2011 11:31:25 -0800 (PST)
Received: from [10.172.2.87] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id p13sm3859797qcu.17.2011.02.08.11.31.16
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Tue, 08 Feb 2011 11:31:24 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Hadoop-common-trunk-Commit is failing since 01/19/2011
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <07AA15F8-B4F2-4B5B-8A65-600B31D96D83@mac.com>
Date: Wed, 9 Feb 2011 06:31:12 +1100
Content-Transfer-Encoding: quoted-printable
Message-Id: <2E13A2FF-8D87-4319-8751-B1884B6B934F@Holsman.NET>
References: <AANLkTinB+8KRByZFhRT-y6MiqA0urtKNs8c_XSrhP7RH@mail.gmail.com> <AANLkTimmcYh0hY-6ht6vaQ+S=pTGHaFxS6k9qEv=q8Cf@mail.gmail.com> <AANLkTinr_Exrr8VPShGKkBSMzmWX1B9nVALLLTMmvYzd@mail.gmail.com> <AANLkTinrheVYQm-y9n=k7TRK_BdzuZYKAWG+qMWR88ct@mail.gmail.com> <AANLkTi=cAhCOkqZ=n9NpBMoPzechoJmfB6KO2yL3QcVA@mail.gmail.com> <AANLkTi=+81g8dFMGMqP5Zc6aAkg5n0dD2QpCVg7ABzd6@mail.gmail.com> <7DC70A70-5DE9-4288-A44B-B5EF9C454D9B@yahoo-inc.com> <AANLkTin9fsb3bgsa98bma+xx_C9K06C_YNKJ6vaA7=TO@mail.gmail.com> <F3847AE0-3FEF-4E85-8757-FA8A7B204A64@yahoo-inc.com> <AANLkTikCOHYJx4Qjt=gajGTcspPai0GmsxWQFogua92Q@mail.gmail.com> <AE3E9DCF-B2D6-4EA7-A166-DB139CD76BFD@yahoo-inc.com> <07AA15F8-B4F2-4B5B-8A65-600B31D96D83@mac.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

Hi Nige.
I've subscribed it now. as it's wasn't on the subscriber list. we do =
have hudson@lucene.zones.apache.org.

but I haven't seen any moderation notices for it either... so I'm not =
sure it is generating emails.

On Feb 9, 2011, at 3:32 AM, Nigel Daley wrote:

> Hmm, haven't seen Hudson post build failures to the common-dev list =
lately. =20
>=20
> Ian, can you check that hudson@hudson.apache.org is still subscribed =
to common-dev@.  If not, please add it.
>=20
> Thx,
> Nige
>=20
>=20
> On Feb 1, 2011, at 11:27 AM, Giridharan Kesavan wrote:
>=20
>> Konstantin,
>>=20
>> trunk/artifacts gets populated when the jar and the tar ant target =
are successful.
>>=20
>> The main reason for the build failure so far is the build abort time =
configuration. It was set to 30mins.
>> I have increased the build abort time and the builds are going on =
fine=20
>> =
https://hudson.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-Common-tr=
unk-Commit
>>=20
>>=20
>> Thanks,
>> Giri
>>=20
>> On Feb 1, 2011, at 12:40 AM, Konstantin Shvachko wrote:
>>=20
>>> Giri,
>>>=20
>>> Looking at configuration of Hadoop-Common-trunk-Commit/
>>> There seems to be errors in the Post-build Actions.
>>> It is complaining that
>>> 'trunk' exists but not 'trunk/artifacts/...'
>>> Is it possible that this misconfiguration is the reason of failures?
>>>=20
>>> --Konstantin
>>>=20
>>>=20
>>> On Mon, Jan 31, 2011 at 4:40 PM, Giridharan Kesavan
>>> <gkesavan@yahoo-inc.com>wrote:
>>>=20
>>>> Konstantin,
>>>>=20
>>>> I think I need to restart the slave which is running the commit =
build. For
>>>> now I have published the common artifact manually from commandline.
>>>>=20
>>>> Thanks,
>>>> Giri
>>>>=20
>>>> On Jan 31, 2011, at 4:27 PM, Konstantin Shvachko wrote:
>>>>=20
>>>>> Giri
>>>>> looks like the last run you started failed the same way as =
previous ones.
>>>>> Any thoughts on what's going on?
>>>>> Thanks,
>>>>> --Konstantin
>>>>>=20
>>>>> On Mon, Jan 31, 2011 at 3:33 PM, Giridharan Kesavan
>>>>> <gkesavan@yahoo-inc.com>wrote:
>>>>>=20
>>>>>> ant mvn-deploy would publish snapshot artifact to the apache =
maven
>>>>>> repository as long you have the right credentials in =
~/.m2/settings.xml.
>>>>>>=20
>>>>>> For settings.xml template pls look at
>>>>>> http://wiki.apache.org/hadoop/HowToRelease
>>>>>>=20
>>>>>> I'm pushing the latest common artifacts now.
>>>>>>=20
>>>>>> -Giri
>>>>>>=20
>>>>>>=20
>>>>>>=20
>>>>>> On Jan 31, 2011, at 3:11 PM, Jakob Homan wrote:
>>>>>>=20
>>>>>>> By manually installing a new core jar into the cache, I can =
compile
>>>>>>> trunk.  Looks like we just need to kick a new Core into maven.  =
Are
>>>>>>> there instructions somewhere for committers to do this?  I know =
Nigel
>>>>>>> and Owen know how, but I don't know if the knowledge is diffused =
past
>>>>>>> them.
>>>>>>> -Jakob
>>>>>>>=20
>>>>>>>=20
>>>>>>> On Mon, Jan 31, 2011 at 1:57 PM, Konstantin Shvachko
>>>>>>> <shv.hadoop@gmail.com> wrote:
>>>>>>>> Current trunk for HDFS and MapReduce are not compiling at the =
moment.
>>>>>> Try to
>>>>>>>> build trunk.
>>>>>>>> This is the result of that changes to common api introduced by
>>>>>> HADOOP-6904
>>>>>>>> are not promoted to HDFS and MR trunks.
>>>>>>>> HDFS-1335 and MAPREDUCE-2263 depend on these changes.
>>>>>>>>=20
>>>>>>>> Common is not promoted to HDFS and MR because
>>>> Hadoop-Common-trunk-Commit
>>>>>>>> build is broken. See here.
>>>>>>>>=20
>>>>>>=20
>>>> =
https://hudson.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-Common-tr=
unk-Commit/
>>>>>>>>=20
>>>>>>>> As I see the last successful build was on 01/19, which =
integrated
>>>>>>>> HADOOP-6864.
>>>>>>>> I think this is when JNI changes were introduced, which cannot =
be
>>>>>> digested
>>>>>>>> by Hudson since then.
>>>>>>>>=20
>>>>>>>> Anybody with gcc active could you please verify if the problem =
is
>>>> caused
>>>>>> by
>>>>>>>> HADOOP-6864.
>>>>>>>>=20
>>>>>>>> Thanks,
>>>>>>>> --Konstantin
>>>>>>>>=20
>>>>>>>> On Mon, Jan 31, 2011 at 1:36 PM, Ted Dunning =
<tdunning@maprtech.com>
>>>>>> wrote:
>>>>>>>>=20
>>>>>>>>> The has been a problem with more than one build failing =
(Mahout is
>>>> the
>>>>>> one
>>>>>>>>> that I saw first) due to a change in maven version which meant =
that
>>>> the
>>>>>>>>> clover license isn't being found properly.  At least, that is =
the
>>>> tale
>>>>>> I
>>>>>>>>> heard from infra.
>>>>>>>>>=20
>>>>>>>>> On Mon, Jan 31, 2011 at 1:31 PM, Eli Collins =
<eli@cloudera.com>
>>>> wrote:
>>>>>>>>>=20
>>>>>>>>>> Hey Konstantin,
>>>>>>>>>>=20
>>>>>>>>>> The only build breakage I saw from HADOOP-6904 is =
MAPREDUCE-2290,
>>>>>>>>>> which was fixed.  Trees from trunk are compiling against each =
other
>>>>>>>>>> for me (eg each installed to a local maven repo), perhaps the
>>>> upstream
>>>>>>>>>> maven repo hasn't been updated with the latest bits yet.
>>>>>>>>>>=20
>>>>>>>>>> Thanks,
>>>>>>>>>> Eli
>>>>>>>>>>=20
>>>>>>>>>> On Mon, Jan 31, 2011 at 12:14 PM, Konstantin Shvachko
>>>>>>>>>> <shv.hadoop@gmail.com> wrote:
>>>>>>>>>>> Sending this to general to attract urgent attention.
>>>>>>>>>>> Both HDFS and MapReduce are not compiling since
>>>>>>>>>>> HADOOP-6904 and its hdfs and MP counterparts were committed.
>>>>>>>>>>> The problem is not with this patch as described below, but I =
think
>>>>>>>>> those
>>>>>>>>>>> commits should be reversed if Common integration build =
cannot be
>>>>>>>>>>> restored promptly.
>>>>>>>>>>>=20
>>>>>>>>>>> Thanks,
>>>>>>>>>>> --Konstantin
>>>>>>>>>>>=20
>>>>>>>>>>>=20
>>>>>>>>>>> On Fri, Jan 28, 2011 at 5:53 PM, Konstantin Shvachko
>>>>>>>>>>> <shv.hadoop@gmail.com>wrote:
>>>>>>>>>>>=20
>>>>>>>>>>>> I see Hadoop-common-trunk-Commit is failing and not sending =
any
>>>>>>>>> emails.
>>>>>>>>>>>> It times out on native compilation and aborts.
>>>>>>>>>>>> Therefore changes are not integrated, and now it lead to =
hdfs and
>>>>>>>>>> mapreduce
>>>>>>>>>>>> both not compiling.
>>>>>>>>>>>> Can somebody please take a look at this.
>>>>>>>>>>>> The last few lines of the build are below.
>>>>>>>>>>>>=20
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> --Konstantin
>>>>>>>>>>>>=20
>>>>>>>>>>>> [javah] [Loaded
>>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>=20
>>>> =
/grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/bui=
ld/classes/org/apache/hadoop/security/JniBasedUnixGroupsMapping.class]
>>>>>>>>>>>>=20
>>>>>>>>>>>> [javah] [Loaded
>>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>=20
>>>> =
/homes/hudson/tools/java/jdk1.6.0_11-32/jre/lib/rt.jar(java/lang/Object.cl=
ass)]
>>>>>>>>>>>> [javah] [Forcefully writing file
>>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>=20
>>>> =
/grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/bui=
ld/native/Linux-i386-32/src/org/apache/hadoop/security/org_apache_hadoop_s=
ecurity_JniBasedUnixGroupsNetgroupMapping.h]
>>>>>>>>>>>>=20
>>>>>>>>>>>>  [exec] checking for gcc... gcc
>>>>>>>>>>>>  [exec] checking whether the C compiler works... yes
>>>>>>>>>>>>  [exec] checking for C compiler default output file name...
>>>>>> a.out
>>>>>>>>>>>>  [exec] checking for suffix of executables...
>>>>>>>>>>>>=20
>>>>>>>>>>>> Build timed out. Aborting
>>>>>>>>>>>> Build was aborted
>>>>>>>>>>>> [FINDBUGS] Skipping publisher since build result is ABORTED
>>>>>>>>>>>> Publishing Javadoc
>>>>>>>>>>>> Archiving artifacts
>>>>>>>>>>>> Recording test results
>>>>>>>>>>>> No test report files were found. Configuration error?
>>>>>>>>>>>>=20
>>>>>>>>>>>> Recording fingerprints
>>>>>>>>>>>>  [exec] Terminated
>>>>>>>>>>>> Publishing Clover coverage report...
>>>>>>>>>>>> No Clover report will be published due to a Build Failure
>>>>>>>>>>>> No emails were triggered.
>>>>>>>>>>>> Finished: ABORTED
>>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>=20
>>>>>>=20
>>>>>>=20
>>>>=20
>>>>=20
>>=20
>=20


From general-return-2962-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 08 19:36:24 2011
Return-Path: <general-return-2962-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72126 invoked from network); 8 Feb 2011 19:36:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Feb 2011 19:36:24 -0000
Received: (qmail 1166 invoked by uid 500); 8 Feb 2011 19:36:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1109 invoked by uid 500); 8 Feb 2011 19:36:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1101 invoked by uid 99); 8 Feb 2011 19:36:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Feb 2011 19:36:21 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.96 as permitted sender)
Received: from [17.148.16.96] (HELO asmtpout021.mac.com) (17.148.16.96)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Feb 2011 19:36:16 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [172.29.12.221] ([38.102.147.105])
 by asmtp021.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LGB009PJD3K3PE0@asmtp021.mac.com>
 for general@hadoop.apache.org; Tue, 08 Feb 2011 11:35:47 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-02-08_05:2011-02-08,2011-02-08,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1102080099
Subject: Re: Hadoop-common-trunk-Commit is failing since 01/19/2011
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <2E13A2FF-8D87-4319-8751-B1884B6B934F@Holsman.NET>
Date: Tue, 08 Feb 2011 11:35:44 -0800
Message-id: <23956D01-5586-4FE2-95C6-4AAD78A1F6BB@mac.com>
References: <AANLkTinB+8KRByZFhRT-y6MiqA0urtKNs8c_XSrhP7RH@mail.gmail.com>
 <AANLkTimmcYh0hY-6ht6vaQ+S=pTGHaFxS6k9qEv=q8Cf@mail.gmail.com>
 <AANLkTinr_Exrr8VPShGKkBSMzmWX1B9nVALLLTMmvYzd@mail.gmail.com>
 <AANLkTinrheVYQm-y9n=k7TRK_BdzuZYKAWG+qMWR88ct@mail.gmail.com>
 <AANLkTi=cAhCOkqZ=n9NpBMoPzechoJmfB6KO2yL3QcVA@mail.gmail.com>
 <AANLkTi=+81g8dFMGMqP5Zc6aAkg5n0dD2QpCVg7ABzd6@mail.gmail.com>
 <7DC70A70-5DE9-4288-A44B-B5EF9C454D9B@yahoo-inc.com>
 <AANLkTin9fsb3bgsa98bma+xx_C9K06C_YNKJ6vaA7=TO@mail.gmail.com>
 <F3847AE0-3FEF-4E85-8757-FA8A7B204A64@yahoo-inc.com>
 <AANLkTikCOHYJx4Qjt=gajGTcspPai0GmsxWQFogua92Q@mail.gmail.com>
 <AE3E9DCF-B2D6-4EA7-A166-DB139CD76BFD@yahoo-inc.com>
 <07AA15F8-B4F2-4B5B-8A65-600B31D96D83@mac.com>
 <2E13A2FF-8D87-4319-8751-B1884B6B934F@Holsman.NET>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

ah, that's the old address.  I wonder how many of our lists have the old one...

nige

On Feb 8, 2011, at 11:31 AM, Ian Holsman wrote:

> Hi Nige.
> I've subscribed it now. as it's wasn't on the subscriber list. we do have hudson@lucene.zones.apache.org.
> 
> but I haven't seen any moderation notices for it either... so I'm not sure it is generating emails.
> 
> On Feb 9, 2011, at 3:32 AM, Nigel Daley wrote:
> 
>> Hmm, haven't seen Hudson post build failures to the common-dev list lately.  
>> 
>> Ian, can you check that hudson@hudson.apache.org is still subscribed to common-dev@.  If not, please add it.
>> 
>> Thx,
>> Nige
>> 
>> 
>> On Feb 1, 2011, at 11:27 AM, Giridharan Kesavan wrote:
>> 
>>> Konstantin,
>>> 
>>> trunk/artifacts gets populated when the jar and the tar ant target are successful.
>>> 
>>> The main reason for the build failure so far is the build abort time configuration. It was set to 30mins.
>>> I have increased the build abort time and the builds are going on fine 
>>> https://hudson.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit
>>> 
>>> 
>>> Thanks,
>>> Giri
>>> 
>>> On Feb 1, 2011, at 12:40 AM, Konstantin Shvachko wrote:
>>> 
>>>> Giri,
>>>> 
>>>> Looking at configuration of Hadoop-Common-trunk-Commit/
>>>> There seems to be errors in the Post-build Actions.
>>>> It is complaining that
>>>> 'trunk' exists but not 'trunk/artifacts/...'
>>>> Is it possible that this misconfiguration is the reason of failures?
>>>> 
>>>> --Konstantin
>>>> 
>>>> 
>>>> On Mon, Jan 31, 2011 at 4:40 PM, Giridharan Kesavan
>>>> <gkesavan@yahoo-inc.com>wrote:
>>>> 
>>>>> Konstantin,
>>>>> 
>>>>> I think I need to restart the slave which is running the commit build. For
>>>>> now I have published the common artifact manually from commandline.
>>>>> 
>>>>> Thanks,
>>>>> Giri
>>>>> 
>>>>> On Jan 31, 2011, at 4:27 PM, Konstantin Shvachko wrote:
>>>>> 
>>>>>> Giri
>>>>>> looks like the last run you started failed the same way as previous ones.
>>>>>> Any thoughts on what's going on?
>>>>>> Thanks,
>>>>>> --Konstantin
>>>>>> 
>>>>>> On Mon, Jan 31, 2011 at 3:33 PM, Giridharan Kesavan
>>>>>> <gkesavan@yahoo-inc.com>wrote:
>>>>>> 
>>>>>>> ant mvn-deploy would publish snapshot artifact to the apache maven
>>>>>>> repository as long you have the right credentials in ~/.m2/settings.xml.
>>>>>>> 
>>>>>>> For settings.xml template pls look at
>>>>>>> http://wiki.apache.org/hadoop/HowToRelease
>>>>>>> 
>>>>>>> I'm pushing the latest common artifacts now.
>>>>>>> 
>>>>>>> -Giri
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Jan 31, 2011, at 3:11 PM, Jakob Homan wrote:
>>>>>>> 
>>>>>>>> By manually installing a new core jar into the cache, I can compile
>>>>>>>> trunk.  Looks like we just need to kick a new Core into maven.  Are
>>>>>>>> there instructions somewhere for committers to do this?  I know Nigel
>>>>>>>> and Owen know how, but I don't know if the knowledge is diffused past
>>>>>>>> them.
>>>>>>>> -Jakob
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Mon, Jan 31, 2011 at 1:57 PM, Konstantin Shvachko
>>>>>>>> <shv.hadoop@gmail.com> wrote:
>>>>>>>>> Current trunk for HDFS and MapReduce are not compiling at the moment.
>>>>>>> Try to
>>>>>>>>> build trunk.
>>>>>>>>> This is the result of that changes to common api introduced by
>>>>>>> HADOOP-6904
>>>>>>>>> are not promoted to HDFS and MR trunks.
>>>>>>>>> HDFS-1335 and MAPREDUCE-2263 depend on these changes.
>>>>>>>>> 
>>>>>>>>> Common is not promoted to HDFS and MR because
>>>>> Hadoop-Common-trunk-Commit
>>>>>>>>> build is broken. See here.
>>>>>>>>> 
>>>>>>> 
>>>>> https://hudson.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/
>>>>>>>>> 
>>>>>>>>> As I see the last successful build was on 01/19, which integrated
>>>>>>>>> HADOOP-6864.
>>>>>>>>> I think this is when JNI changes were introduced, which cannot be
>>>>>>> digested
>>>>>>>>> by Hudson since then.
>>>>>>>>> 
>>>>>>>>> Anybody with gcc active could you please verify if the problem is
>>>>> caused
>>>>>>> by
>>>>>>>>> HADOOP-6864.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> --Konstantin
>>>>>>>>> 
>>>>>>>>> On Mon, Jan 31, 2011 at 1:36 PM, Ted Dunning <tdunning@maprtech.com>
>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> The has been a problem with more than one build failing (Mahout is
>>>>> the
>>>>>>> one
>>>>>>>>>> that I saw first) due to a change in maven version which meant that
>>>>> the
>>>>>>>>>> clover license isn't being found properly.  At least, that is the
>>>>> tale
>>>>>>> I
>>>>>>>>>> heard from infra.
>>>>>>>>>> 
>>>>>>>>>> On Mon, Jan 31, 2011 at 1:31 PM, Eli Collins <eli@cloudera.com>
>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hey Konstantin,
>>>>>>>>>>> 
>>>>>>>>>>> The only build breakage I saw from HADOOP-6904 is MAPREDUCE-2290,
>>>>>>>>>>> which was fixed.  Trees from trunk are compiling against each other
>>>>>>>>>>> for me (eg each installed to a local maven repo), perhaps the
>>>>> upstream
>>>>>>>>>>> maven repo hasn't been updated with the latest bits yet.
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Eli
>>>>>>>>>>> 
>>>>>>>>>>> On Mon, Jan 31, 2011 at 12:14 PM, Konstantin Shvachko
>>>>>>>>>>> <shv.hadoop@gmail.com> wrote:
>>>>>>>>>>>> Sending this to general to attract urgent attention.
>>>>>>>>>>>> Both HDFS and MapReduce are not compiling since
>>>>>>>>>>>> HADOOP-6904 and its hdfs and MP counterparts were committed.
>>>>>>>>>>>> The problem is not with this patch as described below, but I think
>>>>>>>>>> those
>>>>>>>>>>>> commits should be reversed if Common integration build cannot be
>>>>>>>>>>>> restored promptly.
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> --Konstantin
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Fri, Jan 28, 2011 at 5:53 PM, Konstantin Shvachko
>>>>>>>>>>>> <shv.hadoop@gmail.com>wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> I see Hadoop-common-trunk-Commit is failing and not sending any
>>>>>>>>>> emails.
>>>>>>>>>>>>> It times out on native compilation and aborts.
>>>>>>>>>>>>> Therefore changes are not integrated, and now it lead to hdfs and
>>>>>>>>>>> mapreduce
>>>>>>>>>>>>> both not compiling.
>>>>>>>>>>>>> Can somebody please take a look at this.
>>>>>>>>>>>>> The last few lines of the build are below.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>> --Konstantin
>>>>>>>>>>>>> 
>>>>>>>>>>>>> [javah] [Loaded
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>> 
>>>>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/build/classes/org/apache/hadoop/security/JniBasedUnixGroupsMapping.class]
>>>>>>>>>>>>> 
>>>>>>>>>>>>> [javah] [Loaded
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>> 
>>>>> /homes/hudson/tools/java/jdk1.6.0_11-32/jre/lib/rt.jar(java/lang/Object.class)]
>>>>>>>>>>>>> [javah] [Forcefully writing file
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>> 
>>>>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/build/native/Linux-i386-32/src/org/apache/hadoop/security/org_apache_hadoop_security_JniBasedUnixGroupsNetgroupMapping.h]
>>>>>>>>>>>>> 
>>>>>>>>>>>>> [exec] checking for gcc... gcc
>>>>>>>>>>>>> [exec] checking whether the C compiler works... yes
>>>>>>>>>>>>> [exec] checking for C compiler default output file name...
>>>>>>> a.out
>>>>>>>>>>>>> [exec] checking for suffix of executables...
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Build timed out. Aborting
>>>>>>>>>>>>> Build was aborted
>>>>>>>>>>>>> [FINDBUGS] Skipping publisher since build result is ABORTED
>>>>>>>>>>>>> Publishing Javadoc
>>>>>>>>>>>>> Archiving artifacts
>>>>>>>>>>>>> Recording test results
>>>>>>>>>>>>> No test report files were found. Configuration error?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Recording fingerprints
>>>>>>>>>>>>> [exec] Terminated
>>>>>>>>>>>>> Publishing Clover coverage report...
>>>>>>>>>>>>> No Clover report will be published due to a Build Failure
>>>>>>>>>>>>> No emails were triggered.
>>>>>>>>>>>>> Finished: ABORTED
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>> 
>> 
> 


From general-return-2963-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 08 23:21:52 2011
Return-Path: <general-return-2963-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 75304 invoked from network); 8 Feb 2011 23:21:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Feb 2011 23:21:52 -0000
Received: (qmail 17533 invoked by uid 500); 8 Feb 2011 23:21:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17431 invoked by uid 500); 8 Feb 2011 23:21:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17423 invoked by uid 99); 8 Feb 2011 23:21:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Feb 2011 23:21:49 +0000
X-ASF-Spam-Status: No, hits=2.0 required=5.0
	tests=RCVD_IN_BL_SPAMCOP_NET,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Feb 2011 23:21:45 +0000
Received: by qwe4 with SMTP id 4so4743415qwe.35
        for <general@hadoop.apache.org>; Tue, 08 Feb 2011 15:21:24 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=yGEHQEPCdmIypZHtCemcRfYMLkB2ayjQsN3Cc0bUXCo=;
        b=XZGL5y5zo6MBRVJQjE1W0Ahzi6OeB61wWFHueOHyFjqZ8SrmWH3juXaDgyVPKiNt0O
         Fg2qiZ9HcymgTvCP8HCcdZ75IxgakEH7CqR3lcIK8AvnqnZ36ET8fDPIVVTfic0XCZga
         OF6majKxOUnq+jlWlUPOh2U7LNKGnicGqJZts=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=bfOCWcFPQ/dHto0eQrBuzuADGJy1wUTltC0IF7WCed+bV6GILcve4NxaOdpZMly7ug
         23Abkt1Rh/t9uGjcEaVmA1R7ji3z8s2hJltefyIpEht+7DLsIFlUWBNEOOCm//dpoJjL
         dG0MsuwdwTv11gdGrCIpd/rEKsJh97cE8Zzh4=
Received: by 10.224.10.195 with SMTP id q3mr15837560qaq.35.1297207282386;
        Tue, 08 Feb 2011 15:21:22 -0800 (PST)
Received: from [10.172.2.87] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id s10sm5745qco.11.2011.02.08.15.21.13
        (version=TLSv1/SSLv3 cipher=OTHER);
        Tue, 08 Feb 2011 15:21:21 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Hadoop-common-trunk-Commit is failing since 01/19/2011
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <23956D01-5586-4FE2-95C6-4AAD78A1F6BB@mac.com>
Date: Wed, 9 Feb 2011 10:21:09 +1100
Content-Transfer-Encoding: quoted-printable
Message-Id: <229E1339-4B9E-4FBD-B6A6-EF9EFAA8FFD0@Holsman.NET>
References: <AANLkTinB+8KRByZFhRT-y6MiqA0urtKNs8c_XSrhP7RH@mail.gmail.com> <AANLkTimmcYh0hY-6ht6vaQ+S=pTGHaFxS6k9qEv=q8Cf@mail.gmail.com> <AANLkTinr_Exrr8VPShGKkBSMzmWX1B9nVALLLTMmvYzd@mail.gmail.com> <AANLkTinrheVYQm-y9n=k7TRK_BdzuZYKAWG+qMWR88ct@mail.gmail.com> <AANLkTi=cAhCOkqZ=n9NpBMoPzechoJmfB6KO2yL3QcVA@mail.gmail.com> <AANLkTi=+81g8dFMGMqP5Zc6aAkg5n0dD2QpCVg7ABzd6@mail.gmail.com> <7DC70A70-5DE9-4288-A44B-B5EF9C454D9B@yahoo-inc.com> <AANLkTin9fsb3bgsa98bma+xx_C9K06C_YNKJ6vaA7=TO@mail.gmail.com> <F3847AE0-3FEF-4E85-8757-FA8A7B204A64@yahoo-inc.com> <AANLkTikCOHYJx4Qjt=gajGTcspPai0GmsxWQFogua92Q@mail.gmail.com> <AE3E9DCF-B2D6-4EA7-A166-DB139CD76BFD@yahoo-inc.com> <07AA15F8-B4F2-4B5B-8A65-600B31D96D83@mac.com> <2E13A2FF-8D87-4319-8751-B1884B6B934F@Holsman.NET> <23956D01-5586-4FE2-95C6-4AAD78A1F6BB@mac.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

I'll check it out later on today.
even if it is sending from the wrong email address, I should be able to =
moderate them through regardless. (I haven't seen any recently)

can you see if hudson is working on common-dev first?

regards
Ian
On Feb 9, 2011, at 6:35 AM, Nigel Daley wrote:

> ah, that's the old address.  I wonder how many of our lists have the =
old one...
>=20
> nige
>=20
> On Feb 8, 2011, at 11:31 AM, Ian Holsman wrote:
>=20
>> Hi Nige.
>> I've subscribed it now. as it's wasn't on the subscriber list. we do =
have hudson@lucene.zones.apache.org.
>>=20
>> but I haven't seen any moderation notices for it either... so I'm not =
sure it is generating emails.
>>=20
>> On Feb 9, 2011, at 3:32 AM, Nigel Daley wrote:
>>=20
>>> Hmm, haven't seen Hudson post build failures to the common-dev list =
lately. =20
>>>=20
>>> Ian, can you check that hudson@hudson.apache.org is still subscribed =
to common-dev@.  If not, please add it.
>>>=20
>>> Thx,
>>> Nige
>>>=20
>>>=20
>>> On Feb 1, 2011, at 11:27 AM, Giridharan Kesavan wrote:
>>>=20
>>>> Konstantin,
>>>>=20
>>>> trunk/artifacts gets populated when the jar and the tar ant target =
are successful.
>>>>=20
>>>> The main reason for the build failure so far is the build abort =
time configuration. It was set to 30mins.
>>>> I have increased the build abort time and the builds are going on =
fine=20
>>>> =
https://hudson.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-Common-tr=
unk-Commit
>>>>=20
>>>>=20
>>>> Thanks,
>>>> Giri
>>>>=20
>>>> On Feb 1, 2011, at 12:40 AM, Konstantin Shvachko wrote:
>>>>=20
>>>>> Giri,
>>>>>=20
>>>>> Looking at configuration of Hadoop-Common-trunk-Commit/
>>>>> There seems to be errors in the Post-build Actions.
>>>>> It is complaining that
>>>>> 'trunk' exists but not 'trunk/artifacts/...'
>>>>> Is it possible that this misconfiguration is the reason of =
failures?
>>>>>=20
>>>>> --Konstantin
>>>>>=20
>>>>>=20
>>>>> On Mon, Jan 31, 2011 at 4:40 PM, Giridharan Kesavan
>>>>> <gkesavan@yahoo-inc.com>wrote:
>>>>>=20
>>>>>> Konstantin,
>>>>>>=20
>>>>>> I think I need to restart the slave which is running the commit =
build. For
>>>>>> now I have published the common artifact manually from =
commandline.
>>>>>>=20
>>>>>> Thanks,
>>>>>> Giri
>>>>>>=20
>>>>>> On Jan 31, 2011, at 4:27 PM, Konstantin Shvachko wrote:
>>>>>>=20
>>>>>>> Giri
>>>>>>> looks like the last run you started failed the same way as =
previous ones.
>>>>>>> Any thoughts on what's going on?
>>>>>>> Thanks,
>>>>>>> --Konstantin
>>>>>>>=20
>>>>>>> On Mon, Jan 31, 2011 at 3:33 PM, Giridharan Kesavan
>>>>>>> <gkesavan@yahoo-inc.com>wrote:
>>>>>>>=20
>>>>>>>> ant mvn-deploy would publish snapshot artifact to the apache =
maven
>>>>>>>> repository as long you have the right credentials in =
~/.m2/settings.xml.
>>>>>>>>=20
>>>>>>>> For settings.xml template pls look at
>>>>>>>> http://wiki.apache.org/hadoop/HowToRelease
>>>>>>>>=20
>>>>>>>> I'm pushing the latest common artifacts now.
>>>>>>>>=20
>>>>>>>> -Giri
>>>>>>>>=20
>>>>>>>>=20
>>>>>>>>=20
>>>>>>>> On Jan 31, 2011, at 3:11 PM, Jakob Homan wrote:
>>>>>>>>=20
>>>>>>>>> By manually installing a new core jar into the cache, I can =
compile
>>>>>>>>> trunk.  Looks like we just need to kick a new Core into maven. =
 Are
>>>>>>>>> there instructions somewhere for committers to do this?  I =
know Nigel
>>>>>>>>> and Owen know how, but I don't know if the knowledge is =
diffused past
>>>>>>>>> them.
>>>>>>>>> -Jakob
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>> On Mon, Jan 31, 2011 at 1:57 PM, Konstantin Shvachko
>>>>>>>>> <shv.hadoop@gmail.com> wrote:
>>>>>>>>>> Current trunk for HDFS and MapReduce are not compiling at the =
moment.
>>>>>>>> Try to
>>>>>>>>>> build trunk.
>>>>>>>>>> This is the result of that changes to common api introduced =
by
>>>>>>>> HADOOP-6904
>>>>>>>>>> are not promoted to HDFS and MR trunks.
>>>>>>>>>> HDFS-1335 and MAPREDUCE-2263 depend on these changes.
>>>>>>>>>>=20
>>>>>>>>>> Common is not promoted to HDFS and MR because
>>>>>> Hadoop-Common-trunk-Commit
>>>>>>>>>> build is broken. See here.
>>>>>>>>>>=20
>>>>>>>>=20
>>>>>> =
https://hudson.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-Common-tr=
unk-Commit/
>>>>>>>>>>=20
>>>>>>>>>> As I see the last successful build was on 01/19, which =
integrated
>>>>>>>>>> HADOOP-6864.
>>>>>>>>>> I think this is when JNI changes were introduced, which =
cannot be
>>>>>>>> digested
>>>>>>>>>> by Hudson since then.
>>>>>>>>>>=20
>>>>>>>>>> Anybody with gcc active could you please verify if the =
problem is
>>>>>> caused
>>>>>>>> by
>>>>>>>>>> HADOOP-6864.
>>>>>>>>>>=20
>>>>>>>>>> Thanks,
>>>>>>>>>> --Konstantin
>>>>>>>>>>=20
>>>>>>>>>> On Mon, Jan 31, 2011 at 1:36 PM, Ted Dunning =
<tdunning@maprtech.com>
>>>>>>>> wrote:
>>>>>>>>>>=20
>>>>>>>>>>> The has been a problem with more than one build failing =
(Mahout is
>>>>>> the
>>>>>>>> one
>>>>>>>>>>> that I saw first) due to a change in maven version which =
meant that
>>>>>> the
>>>>>>>>>>> clover license isn't being found properly.  At least, that =
is the
>>>>>> tale
>>>>>>>> I
>>>>>>>>>>> heard from infra.
>>>>>>>>>>>=20
>>>>>>>>>>> On Mon, Jan 31, 2011 at 1:31 PM, Eli Collins =
<eli@cloudera.com>
>>>>>> wrote:
>>>>>>>>>>>=20
>>>>>>>>>>>> Hey Konstantin,
>>>>>>>>>>>>=20
>>>>>>>>>>>> The only build breakage I saw from HADOOP-6904 is =
MAPREDUCE-2290,
>>>>>>>>>>>> which was fixed.  Trees from trunk are compiling against =
each other
>>>>>>>>>>>> for me (eg each installed to a local maven repo), perhaps =
the
>>>>>> upstream
>>>>>>>>>>>> maven repo hasn't been updated with the latest bits yet.
>>>>>>>>>>>>=20
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Eli
>>>>>>>>>>>>=20
>>>>>>>>>>>> On Mon, Jan 31, 2011 at 12:14 PM, Konstantin Shvachko
>>>>>>>>>>>> <shv.hadoop@gmail.com> wrote:
>>>>>>>>>>>>> Sending this to general to attract urgent attention.
>>>>>>>>>>>>> Both HDFS and MapReduce are not compiling since
>>>>>>>>>>>>> HADOOP-6904 and its hdfs and MP counterparts were =
committed.
>>>>>>>>>>>>> The problem is not with this patch as described below, but =
I think
>>>>>>>>>>> those
>>>>>>>>>>>>> commits should be reversed if Common integration build =
cannot be
>>>>>>>>>>>>> restored promptly.
>>>>>>>>>>>>>=20
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> --Konstantin
>>>>>>>>>>>>>=20
>>>>>>>>>>>>>=20
>>>>>>>>>>>>> On Fri, Jan 28, 2011 at 5:53 PM, Konstantin Shvachko
>>>>>>>>>>>>> <shv.hadoop@gmail.com>wrote:
>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> I see Hadoop-common-trunk-Commit is failing and not =
sending any
>>>>>>>>>>> emails.
>>>>>>>>>>>>>> It times out on native compilation and aborts.
>>>>>>>>>>>>>> Therefore changes are not integrated, and now it lead to =
hdfs and
>>>>>>>>>>>> mapreduce
>>>>>>>>>>>>>> both not compiling.
>>>>>>>>>>>>>> Can somebody please take a look at this.
>>>>>>>>>>>>>> The last few lines of the build are below.
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>> --Konstantin
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> [javah] [Loaded
>>>>>>>>>>>>=20
>>>>>>>>>>>=20
>>>>>>>>=20
>>>>>> =
/grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/bui=
ld/classes/org/apache/hadoop/security/JniBasedUnixGroupsMapping.class]
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> [javah] [Loaded
>>>>>>>>>>>>=20
>>>>>>>>>>>=20
>>>>>>>>=20
>>>>>> =
/homes/hudson/tools/java/jdk1.6.0_11-32/jre/lib/rt.jar(java/lang/Object.cl=
ass)]
>>>>>>>>>>>>>> [javah] [Forcefully writing file
>>>>>>>>>>>>=20
>>>>>>>>>>>=20
>>>>>>>>=20
>>>>>> =
/grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/bui=
ld/native/Linux-i386-32/src/org/apache/hadoop/security/org_apache_hadoop_s=
ecurity_JniBasedUnixGroupsNetgroupMapping.h]
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> [exec] checking for gcc... gcc
>>>>>>>>>>>>>> [exec] checking whether the C compiler works... yes
>>>>>>>>>>>>>> [exec] checking for C compiler default output file =
name...
>>>>>>>> a.out
>>>>>>>>>>>>>> [exec] checking for suffix of executables...
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> Build timed out. Aborting
>>>>>>>>>>>>>> Build was aborted
>>>>>>>>>>>>>> [FINDBUGS] Skipping publisher since build result is =
ABORTED
>>>>>>>>>>>>>> Publishing Javadoc
>>>>>>>>>>>>>> Archiving artifacts
>>>>>>>>>>>>>> Recording test results
>>>>>>>>>>>>>> No test report files were found. Configuration error?
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> Recording fingerprints
>>>>>>>>>>>>>> [exec] Terminated
>>>>>>>>>>>>>> Publishing Clover coverage report...
>>>>>>>>>>>>>> No Clover report will be published due to a Build Failure
>>>>>>>>>>>>>> No emails were triggered.
>>>>>>>>>>>>>> Finished: ABORTED
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>=20
>>>>>>>>=20
>>>>>>=20
>>>>>>=20
>>>>=20
>>>=20
>>=20
>=20


From general-return-2964-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 09 17:12:20 2011
Return-Path: <general-return-2964-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 89230 invoked from network); 9 Feb 2011 17:12:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Feb 2011 17:12:20 -0000
Received: (qmail 15612 invoked by uid 500); 9 Feb 2011 17:12:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15290 invoked by uid 500); 9 Feb 2011 17:12:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15280 invoked by uid 99); 9 Feb 2011 17:12:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Feb 2011 17:12:14 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.97 as permitted sender)
Received: from [17.148.16.97] (HELO asmtpout022.mac.com) (17.148.16.97)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Feb 2011 17:12:06 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp022.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LGD007N513K36A0@asmtp022.mac.com>
 for general@hadoop.apache.org; Wed, 09 Feb 2011 09:11:45 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-02-09_06:2011-02-09,2011-02-09,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1102090095
Subject: Re: [DISCUSS] Move common, hdfs,
 mapreduce contrib components to apache-extras.org or elsewhere
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com>
Date: Wed, 09 Feb 2011 09:11:43 -0800
Message-id: <E6B510A3-D141-4490-9978-712681195C37@mac.com>
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

After considering the feedback, I will move forward with calling a separate vote for each contrib (or small groups of contribs). The vote will ask the PMC to abandon the given contrib or move it to core.  For those we agree to abandon, I will setup an Attic wiki that will point to the last SVN revision of the contrib.  I will also start a Related Projects wiki (if we don't already have one) with pointers to the contrib modules that folks have volunteered to keep developing elsewhere.

Cheers,
Nige

On Jan 30, 2011, at 7:42 PM, Nigel Daley wrote:

> Folks,
> 
> Now that http://apache-extras.org is launched (https://blogs.apache.org/foundation/entry/the_apache_software_foundation_launches) I'd like to start a discussion on moving contrib components out of common, mapreduce, and hdfs.  
> 
> These contrib components complicate the builds, cause test failures that nobody seems to care about, have releases that are tied to Hadoop's long release cycles, etc.  Most folks I've talked with agree that these contrib components would be better served by being pulled out of Hadoop and hosted elsewhere. The new apache-extras code hosting site seems like a natural *default* location for migrating these contrib projects.  Perhaps some should graduate from contrib to src (ie from contrib to core of the project they're included in).  If folks agree, we'll need to come up with a mapping of contrib component to it's final destination and file a jira.
> 
> Here are the contrib components by project (hopefully I didn't miss any).
> 
> Common Contrib:
>  failmon
>  hod
>  test
> 
> 
> MapReduce Contrib:
>  capacity-scheduler -- move to MR core?
>  data_join
>  dynamic-scheduler
>  eclipse-plugin
>  fairscheduler -- move to MR core?
>  gridmix
>  index
>  mrunit
>  mumak
>  raid
>  sqoop
>  streaming -- move to MR core?
>  vaidya
>  vertica
> 
> 
> HDFS Contrib:
>  fuse-dfs
>  hdfsproxy
>  thriftfs
> 
> 
> Cheers,
> Nige


From general-return-2965-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 09 17:19:06 2011
Return-Path: <general-return-2965-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91687 invoked from network); 9 Feb 2011 17:19:06 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Feb 2011 17:19:06 -0000
Received: (qmail 35178 invoked by uid 500); 9 Feb 2011 17:19:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34674 invoked by uid 500); 9 Feb 2011 17:19:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34665 invoked by uid 99); 9 Feb 2011 17:19:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Feb 2011 17:19:00 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.100 as permitted sender)
Received: from [17.148.16.100] (HELO asmtpout025.mac.com) (17.148.16.100)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Feb 2011 17:18:51 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp025.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LGD004FH1EG5R00@asmtp025.mac.com>
 for general@hadoop.apache.org; Wed, 09 Feb 2011 09:18:18 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-02-09_06:2011-02-09,2011-02-09,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1102090096
From: Nigel Daley <ndaley@mac.com>
Subject: [VOTE] Abandon failmon Common contrib
Date: Wed, 09 Feb 2011 09:18:16 -0800
Message-id: <C98FD122-F11B-4A3F-8D34-3C547392F881@mac.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

I think the PMC should abandon the failmon common contrib component.  It's last meaningful contribution was it's original commit in August of 2008:

HADOOP-3585. FailMon package for hardware failure monitoring and analysis of anomalies. (Ioannis Koltsidas via dhruba)

Here is my +1.

Nige

From general-return-2966-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 09 19:31:04 2011
Return-Path: <general-return-2966-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77057 invoked from network); 9 Feb 2011 19:31:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Feb 2011 19:31:03 -0000
Received: (qmail 96504 invoked by uid 500); 9 Feb 2011 19:31:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96122 invoked by uid 500); 9 Feb 2011 19:30:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96114 invoked by uid 99); 9 Feb 2011 19:30:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Feb 2011 19:30:59 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of eli@cloudera.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Feb 2011 19:30:51 +0000
Received: by qwe4 with SMTP id 4so410044qwe.35
        for <general@hadoop.apache.org>; Wed, 09 Feb 2011 11:30:30 -0800 (PST)
MIME-Version: 1.0
Received: by 10.229.239.142 with SMTP id kw14mr16440794qcb.38.1297279830640;
 Wed, 09 Feb 2011 11:30:30 -0800 (PST)
Received: by 10.229.215.2 with HTTP; Wed, 9 Feb 2011 11:30:30 -0800 (PST)
In-Reply-To: <C98FD122-F11B-4A3F-8D34-3C547392F881@mac.com>
References: <C98FD122-F11B-4A3F-8D34-3C547392F881@mac.com>
Date: Wed, 9 Feb 2011 11:30:30 -0800
Message-ID: <AANLkTi=i5jkgQv9Bas5u82hsDxoCpoTOEm0WJ3L_cDkb@mail.gmail.com>
Subject: Re: [VOTE] Abandon failmon Common contrib
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

+1

On Wed, Feb 9, 2011 at 9:18 AM, Nigel Daley <ndaley@mac.com> wrote:
> I think the PMC should abandon the failmon common contrib component. =A0I=
t's last meaningful contribution was it's original commit in August of 2008=
:
>
> HADOOP-3585. FailMon package for hardware failure monitoring and analysis=
 of anomalies. (Ioannis Koltsidas via dhruba)
>
> Here is my +1.
>
> Nige
>

From general-return-2967-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 09 19:50:15 2011
Return-Path: <general-return-2967-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84684 invoked from network); 9 Feb 2011 19:50:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Feb 2011 19:50:14 -0000
Received: (qmail 39246 invoked by uid 500); 9 Feb 2011 19:50:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38793 invoked by uid 500); 9 Feb 2011 19:50:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 38784 invoked by uid 99); 9 Feb 2011 19:50:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Feb 2011 19:50:10 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Feb 2011 19:50:05 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p19JnOLb010794
	for <general@hadoop.apache.org>; Wed, 9 Feb 2011 11:49:25 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1297280965;
	bh=fcplp8q3NvLfEkOxBp6IQjTzCnt3S3RFx4hKua4dYCk=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=uMTdNFa5FDcduVHyEz5Kp70NoDUNrCUmVx1kT3gI1LwaU5FXuNCpPbBIBSbzAc/8z
	 tjL5JA9qkmYuuW18hkpkCsGHC/fTQvCxxt4HsIOJVux7m6in7E2EjblYJqJnEJofey
	 AdTBXrIEzXHf2NO5rmc1FlwQ4aTNdsrvMxP/rDkY=
Message-Id: <B175D96C-396D-42DE-BF74-08542C60319A@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <C98FD122-F11B-4A3F-8D34-3C547392F881@mac.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Abandon failmon Common contrib
Date: Wed, 9 Feb 2011 11:49:24 -0800
References: <C98FD122-F11B-4A3F-8D34-3C547392F881@mac.com>
X-Mailer: Apple Mail (2.936)

+1

Arun

On Feb 9, 2011, at 9:18 AM, Nigel Daley wrote:

> I think the PMC should abandon the failmon common contrib  
> component.  It's last meaningful contribution was it's original  
> commit in August of 2008:
>
> HADOOP-3585. FailMon package for hardware failure monitoring and  
> analysis of anomalies. (Ioannis Koltsidas via dhruba)
>
> Here is my +1.
>
> Nige


From general-return-2968-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 10 01:55:50 2011
Return-Path: <general-return-2968-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 86357 invoked from network); 10 Feb 2011 01:55:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Feb 2011 01:55:50 -0000
Received: (qmail 89278 invoked by uid 500); 10 Feb 2011 01:55:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89192 invoked by uid 500); 10 Feb 2011 01:55:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89184 invoked by uid 99); 10 Feb 2011 01:55:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Feb 2011 01:55:47 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of tom.e.white@gmail.com designates 74.125.82.42 as permitted sender)
Received: from [74.125.82.42] (HELO mail-ww0-f42.google.com) (74.125.82.42)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Feb 2011 01:55:40 +0000
Received: by wwi17 with SMTP id 17so2481772wwi.5
        for <general@hadoop.apache.org>; Wed, 09 Feb 2011 17:55:19 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type:content-transfer-encoding;
        bh=oLuMGaMxcFBpOMYWfTOkHTgKct8yESOsRH8ExCyKjV0=;
        b=jsR5pM1Eepvqwkszvx/gCJDUPE05rySwFkpzEzmrUvLdiWMnunIUCQbmROe6n1qDPj
         +UymwyopkkPvmcGsnlqPlyiVhkUYE/qBpE/VJ2vI6IGptLxszMaXVTi1e7y+TC9ADETd
         uZnGil3pJIQfZNcKOq35XX1q3jvWOhod6rN+A=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=HkhAKzMWxxV/TmhLhU8mkuSYggIJjRqW1hfqXfKIpI/dkxLxvfWd1QvfrXfQ3Ru+af
         VS56p1gA3diX0mac8/gFfEYoe/nPsLThd1rDv2X4/h9SFQ+k8C+fJrNgjSo14fArf6Wv
         aAt+/x8LXaqu7dOCdG9fIRRDmgaAiy0AMAc/E=
Received: by 10.216.173.147 with SMTP id v19mr18938757wel.102.1297302751640;
 Wed, 09 Feb 2011 17:52:31 -0800 (PST)
MIME-Version: 1.0
Received: by 10.216.159.193 with HTTP; Wed, 9 Feb 2011 17:23:40 -0800 (PST)
In-Reply-To: <C98FD122-F11B-4A3F-8D34-3C547392F881@mac.com>
References: <C98FD122-F11B-4A3F-8D34-3C547392F881@mac.com>
From: Tom White <tom.e.white@gmail.com>
Date: Wed, 9 Feb 2011 17:23:40 -0800
Message-ID: <AANLkTikMgUY367cxGPvr0okSdS5TnVMc58C2qsy491Po@mail.gmail.com>
Subject: Re: [VOTE] Abandon failmon Common contrib
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Tom

On Wed, Feb 9, 2011 at 9:18 AM, Nigel Daley <ndaley@mac.com> wrote:
> I think the PMC should abandon the failmon common contrib component. =A0I=
t's last meaningful contribution was it's original commit in August of 2008=
:
>
> HADOOP-3585. FailMon package for hardware failure monitoring and analysis=
 of anomalies. (Ioannis Koltsidas via dhruba)
>
> Here is my +1.
>
> Nige
>

From general-return-2969-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 10 02:46:22 2011
Return-Path: <general-return-2969-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6843 invoked from network); 10 Feb 2011 02:46:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Feb 2011 02:46:21 -0000
Received: (qmail 22480 invoked by uid 500); 10 Feb 2011 02:46:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22148 invoked by uid 500); 10 Feb 2011 02:46:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22140 invoked by uid 99); 10 Feb 2011 02:46:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Feb 2011 02:46:18 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Feb 2011 02:46:11 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=rqg51Cb/R0I+HjD+ceJa7jWz7J/WKQgxn9BDMIufnnTd6zVliqq35mNs
   9uMXaluFR+vbgD4VySxSqM2/kxlYEmCYfSDSuXFew7YvXAZCkJOOoNp4p
   Idg0NzNQ3AHoreD;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1297305967; x=1328841967;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20[VOTE]=20Abandon=20failmon=20Common=20c
   ontrib|Date:=20Thu,=2010=20Feb=202011=2002:47:12=20+0000
   |Message-ID:=20<04D296EB-2D0B-4C1E-B094-2881508AFD42@link
   edin.com>|To:=20"<general@hadoop.apache.org>"=20<general@
   hadoop.apache.org>|MIME-Version:=201.0
   |Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<1C5D17072608F443979915D41C0D080B@linkedin
   .com>|In-Reply-To:=20<C98FD122-F11B-4A3F-8D34-3C547392F88
   1@mac.com>|References:=20<C98FD122-F11B-4A3F-8D34-3C54739
   2F881@mac.com>;
  bh=wJHOj/h+m9GU9O1+c/+hAYpbD17Ms9HvXpMeqE3JDWc=;
  b=sKFZHqRfJm68wANLWHQgIZ9YU640VphnIyPy5be2oKJQ0hlmBgBqrsmG
   Gq7a1CiT4oPzhiqsLSXf5S/lasbJIDVVdN2EenlEffpDAm6p3lppOj94k
   CzeKBvUle8he0jb;
X-IronPort-AV: E=Sophos;i="4.60,449,1291622400"; 
   d="scan'208";a="20594044"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi id 14.01.0251.000; Wed, 9
 Feb 2011 18:47:12 -0800
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: [VOTE] Abandon failmon Common contrib
Thread-Topic: [VOTE] Abandon failmon Common contrib
Thread-Index: AQHLyH2mCiwdAds8qkuATBAp8rE3DJP6jmEA
Date: Thu, 10 Feb 2011 02:47:12 +0000
Message-ID: <04D296EB-2D0B-4C1E-B094-2881508AFD42@linkedin.com>
References: <C98FD122-F11B-4A3F-8D34-3C547392F881@mac.com>
In-Reply-To: <C98FD122-F11B-4A3F-8D34-3C547392F881@mac.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <1C5D17072608F443979915D41C0D080B@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


On Feb 9, 2011, at 9:18 AM, Nigel Daley wrote:

> I think the PMC should abandon the failmon common contrib component.  It'=
s last meaningful contribution was it's original commit in August of 2008:

	Are there any patches in the patch queue?

	When it comes to contrib, last commit is pretty useless IMO.=

From general-return-2970-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 10 03:38:13 2011
Return-Path: <general-return-2970-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31566 invoked from network); 10 Feb 2011 03:38:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Feb 2011 03:38:13 -0000
Received: (qmail 60256 invoked by uid 500); 10 Feb 2011 03:38:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59979 invoked by uid 500); 10 Feb 2011 03:38:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59971 invoked by uid 99); 10 Feb 2011 03:38:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Feb 2011 03:38:08 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [128.149.139.105] (HELO mail.jpl.nasa.gov) (128.149.139.105)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Feb 2011 03:38:03 +0000
Received: from mail.jpl.nasa.gov (altvirehtstap02.jpl.nasa.gov [128.149.137.73])
	by smtp.jpl.nasa.gov (Switch-3.4.3/Switch-3.4.3) with ESMTP id p1A3bfUF029499
	(using TLSv1/SSLv3 with cipher RC4-MD5 (128 bits) verified FAIL)
	for <general@hadoop.apache.org>; Wed, 9 Feb 2011 19:37:42 -0800
Received: from ALTPHYEMBEVSP20.RES.AD.JPL ([128.149.137.82]) by
 ALTVIREHTSTAP02.RES.AD.JPL ([128.149.137.73]) with mapi; Wed, 9 Feb 2011
 19:37:41 -0800
From: "Mattmann, Chris A (388J)" <chris.a.mattmann@jpl.nasa.gov>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 9 Feb 2011 19:37:40 -0800
Subject: Re: [DISCUSS] Move common, hdfs, mapreduce contrib components to
 apache-extras.org or elsewhere
Thread-Topic: [DISCUSS] Move common, hdfs, mapreduce contrib components to
 apache-extras.org or elsewhere
Thread-Index: AcvI0961R/NdGOFIRWyT5DZTUjL0nw==
Message-ID: <84E62479-A4EE-4AB5-8148-F038FE4532B5@jpl.nasa.gov>
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com>
 <E6B510A3-D141-4490-9978-712681195C37@mac.com>
In-Reply-To: <E6B510A3-D141-4490-9978-712681195C37@mac.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Source-IP: altvirehtstap02.jpl.nasa.gov [128.149.137.73]
X-Source-Sender: chris.a.mattmann@jpl.nasa.gov
X-AUTH: Authorized

Hi Nigel,

My 2 cents -- why is Hadoop re-creating its own mini-Attic? The point of th=
e umbrella project shakedown over the past year at Apache is to stop projec=
ts from recreating things like the Incubator (and the Attic) inside of the =
project.

One suggestion: for your [VOTE] thread that you are calling -- if the resul=
t is to attic the project -- move it to the "real" Apache Attic, via a boar=
d resolution (you could do a single board resolution from the Hadoop PMC to=
 the Apache Board containing the set of projects to Attic).=20

Cheers,
Chris

On Feb 9, 2011, at 9:11 AM, Nigel Daley wrote:

> After considering the feedback, I will move forward with calling a separa=
te vote for each contrib (or small groups of contribs). The vote will ask t=
he PMC to abandon the given contrib or move it to core.  For those we agree=
 to abandon, I will setup an Attic wiki that will point to the last SVN rev=
ision of the contrib.  I will also start a Related Projects wiki (if we don=
't already have one) with pointers to the contrib modules that folks have v=
olunteered to keep developing elsewhere.
>=20
> Cheers,
> Nige
>=20
> On Jan 30, 2011, at 7:42 PM, Nigel Daley wrote:
>=20
>> Folks,
>>=20
>> Now that http://apache-extras.org is launched (https://blogs.apache.org/=
foundation/entry/the_apache_software_foundation_launches) I'd like to start=
 a discussion on moving contrib components out of common, mapreduce, and hd=
fs. =20
>>=20
>> These contrib components complicate the builds, cause test failures that=
 nobody seems to care about, have releases that are tied to Hadoop's long r=
elease cycles, etc.  Most folks I've talked with agree that these contrib c=
omponents would be better served by being pulled out of Hadoop and hosted e=
lsewhere. The new apache-extras code hosting site seems like a natural *def=
ault* location for migrating these contrib projects.  Perhaps some should g=
raduate from contrib to src (ie from contrib to core of the project they're=
 included in).  If folks agree, we'll need to come up with a mapping of con=
trib component to it's final destination and file a jira.
>>=20
>> Here are the contrib components by project (hopefully I didn't miss any)=
.
>>=20
>> Common Contrib:
>> failmon
>> hod
>> test
>>=20
>>=20
>> MapReduce Contrib:
>> capacity-scheduler -- move to MR core?
>> data_join
>> dynamic-scheduler
>> eclipse-plugin
>> fairscheduler -- move to MR core?
>> gridmix
>> index
>> mrunit
>> mumak
>> raid
>> sqoop
>> streaming -- move to MR core?
>> vaidya
>> vertica
>>=20
>>=20
>> HDFS Contrib:
>> fuse-dfs
>> hdfsproxy
>> thriftfs
>>=20
>>=20
>> Cheers,
>> Nige
>=20


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


From general-return-2971-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 10 06:36:31 2011
Return-Path: <general-return-2971-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 11061 invoked from network); 10 Feb 2011 06:36:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Feb 2011 06:36:30 -0000
Received: (qmail 60712 invoked by uid 500); 10 Feb 2011 06:36:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60257 invoked by uid 500); 10 Feb 2011 06:36:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60244 invoked by uid 99); 10 Feb 2011 06:36:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Feb 2011 06:36:24 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.96 as permitted sender)
Received: from [17.148.16.96] (HELO asmtpout021.mac.com) (17.148.16.96)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Feb 2011 06:36:19 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp021.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LGE00BKN2B1O220@asmtp021.mac.com>
 for general@hadoop.apache.org; Wed, 09 Feb 2011 22:35:26 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-02-10_02:2011-02-10,2011-02-10,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1102090243
Subject: Re: [DISCUSS] Move common, hdfs,
 mapreduce contrib components to apache-extras.org or elsewhere
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <84E62479-A4EE-4AB5-8148-F038FE4532B5@jpl.nasa.gov>
Date: Wed, 09 Feb 2011 22:35:25 -0800
Message-id: <F4A665FC-82EF-42B0-9BA1-01FAE419805D@mac.com>
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com>
 <E6B510A3-D141-4490-9978-712681195C37@mac.com>
 <84E62479-A4EE-4AB5-8148-F038FE4532B5@jpl.nasa.gov>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

Hi Chris,

You use the word 'project' over and over.  These contrib modules are not Apache projects.  They are source code directories within the various Hadoop subprojects. Reading thru attic.apache.org, I don't see how it directly relates.  Surely not every 'svn remove <dir>' should be replaced with a move to attic.apache.org.  

Cheers,
Nige


On Feb 9, 2011, at 7:37 PM, Mattmann, Chris A (388J) wrote:

> Hi Nigel,
> 
> My 2 cents -- why is Hadoop re-creating its own mini-Attic? The point of the umbrella project shakedown over the past year at Apache is to stop projects from recreating things like the Incubator (and the Attic) inside of the project.
> 
> One suggestion: for your [VOTE] thread that you are calling -- if the result is to attic the project -- move it to the "real" Apache Attic, via a board resolution (you could do a single board resolution from the Hadoop PMC to the Apache Board containing the set of projects to Attic). 
> 
> Cheers,
> Chris
> 
> On Feb 9, 2011, at 9:11 AM, Nigel Daley wrote:
> 
>> After considering the feedback, I will move forward with calling a separate vote for each contrib (or small groups of contribs). The vote will ask the PMC to abandon the given contrib or move it to core.  For those we agree to abandon, I will setup an Attic wiki that will point to the last SVN revision of the contrib.  I will also start a Related Projects wiki (if we don't already have one) with pointers to the contrib modules that folks have volunteered to keep developing elsewhere.
>> 
>> Cheers,
>> Nige
>> 
>> On Jan 30, 2011, at 7:42 PM, Nigel Daley wrote:
>> 
>>> Folks,
>>> 
>>> Now that http://apache-extras.org is launched (https://blogs.apache.org/foundation/entry/the_apache_software_foundation_launches) I'd like to start a discussion on moving contrib components out of common, mapreduce, and hdfs.  
>>> 
>>> These contrib components complicate the builds, cause test failures that nobody seems to care about, have releases that are tied to Hadoop's long release cycles, etc.  Most folks I've talked with agree that these contrib components would be better served by being pulled out of Hadoop and hosted elsewhere. The new apache-extras code hosting site seems like a natural *default* location for migrating these contrib projects.  Perhaps some should graduate from contrib to src (ie from contrib to core of the project they're included in).  If folks agree, we'll need to come up with a mapping of contrib component to it's final destination and file a jira.
>>> 
>>> Here are the contrib components by project (hopefully I didn't miss any).
>>> 
>>> Common Contrib:
>>> failmon
>>> hod
>>> test
>>> 
>>> 
>>> MapReduce Contrib:
>>> capacity-scheduler -- move to MR core?
>>> data_join
>>> dynamic-scheduler
>>> eclipse-plugin
>>> fairscheduler -- move to MR core?
>>> gridmix
>>> index
>>> mrunit
>>> mumak
>>> raid
>>> sqoop
>>> streaming -- move to MR core?
>>> vaidya
>>> vertica
>>> 
>>> 
>>> HDFS Contrib:
>>> fuse-dfs
>>> hdfsproxy
>>> thriftfs
>>> 
>>> 
>>> Cheers,
>>> Nige
>> 
> 
> 
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW:   http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 


From general-return-2972-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 10 06:54:18 2011
Return-Path: <general-return-2972-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16245 invoked from network); 10 Feb 2011 06:54:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Feb 2011 06:54:17 -0000
Received: (qmail 73194 invoked by uid 500); 10 Feb 2011 06:54:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72723 invoked by uid 500); 10 Feb 2011 06:54:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72715 invoked by uid 99); 10 Feb 2011 06:54:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Feb 2011 06:54:12 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [128.149.139.105] (HELO mail.jpl.nasa.gov) (128.149.139.105)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Feb 2011 06:54:06 +0000
Received: from mail.jpl.nasa.gov (altvirehtstap02.jpl.nasa.gov [128.149.137.73])
	by smtp.jpl.nasa.gov (Switch-3.4.3/Switch-3.4.3) with ESMTP id p1A6rfZv015401
	(using TLSv1/SSLv3 with cipher RC4-MD5 (128 bits) verified FAIL)
	for <general@hadoop.apache.org>; Wed, 9 Feb 2011 22:53:45 -0800
Received: from ALTPHYEMBEVSP20.RES.AD.JPL ([128.149.137.82]) by
 ALTVIREHTSTAP02.RES.AD.JPL ([128.149.137.73]) with mapi; Wed, 9 Feb 2011
 22:53:36 -0800
From: "Mattmann, Chris A (388J)" <chris.a.mattmann@jpl.nasa.gov>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 9 Feb 2011 22:53:33 -0800
Subject: Re: [DISCUSS] Move common, hdfs, mapreduce contrib components to
 apache-extras.org or elsewhere
Thread-Topic: [DISCUSS] Move common, hdfs, mapreduce contrib components to
 apache-extras.org or elsewhere
Thread-Index: AcvI7z02x9ouEYTURaGkieyFL5YU8g==
Message-ID: <E5A12B8B-1D39-4FCB-9FC0-7A3758A0007B@jpl.nasa.gov>
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com>
 <E6B510A3-D141-4490-9978-712681195C37@mac.com>
 <84E62479-A4EE-4AB5-8148-F038FE4532B5@jpl.nasa.gov>
 <F4A665FC-82EF-42B0-9BA1-01FAE419805D@mac.com>
In-Reply-To: <F4A665FC-82EF-42B0-9BA1-01FAE419805D@mac.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Source-IP: altvirehtstap02.jpl.nasa.gov [128.149.137.73]
X-Source-Sender: chris.a.mattmann@jpl.nasa.gov
X-AUTH: Authorized

Hi Nige,

Thanks. I think in fact it does directly relate -- to use your verbage belo=
w -- not ever svn remove <dir> requires a [DISCUSS] thread before svn remov=
ing it, right?

Hence, my knee-jerk reaction on reading this thread is that contrib modules=
 are =3D contrib modules in other umbrella projects -- some are small, and =
likely yes should be part of the core of Hadoop; others are not, and are in=
 fact, "mini projects" that have been baking up in Hadoop for a while. In t=
he case of the former, I agree, +1, those should be moved as part of the Ha=
doop core "project". In the case of the latter, I do not agree that simply =
the Hadoop project should create a wiki page and declare by VOTE that there=
 won't be anymore development on them. In fact, they are perfect candidates=
 for moving to the Attic where someone besides the Hadoop PMC might want to=
 pick up on them at a later point in time.

There is no hard and fast rule for the size of a TLP too btw: utilities tha=
t run on top of Hadoop could go through e.g., Incubation, and eventually gr=
aduate to TLPs.

Cheers,
Chris

On Feb 9, 2011, at 10:35 PM, Nigel Daley wrote:

> Hi Chris,
>=20
> You use the word 'project' over and over.  These contrib modules are not =
Apache projects.  They are source code directories within the various Hadoo=
p subprojects. Reading thru attic.apache.org, I don't see how it directly r=
elates.  Surely not every 'svn remove <dir>' should be replaced with a move=
 to attic.apache.org. =20
>=20
> Cheers,
> Nige
>=20
>=20
> On Feb 9, 2011, at 7:37 PM, Mattmann, Chris A (388J) wrote:
>=20
>> Hi Nigel,
>>=20
>> My 2 cents -- why is Hadoop re-creating its own mini-Attic? The point of=
 the umbrella project shakedown over the past year at Apache is to stop pro=
jects from recreating things like the Incubator (and the Attic) inside of t=
he project.
>>=20
>> One suggestion: for your [VOTE] thread that you are calling -- if the re=
sult is to attic the project -- move it to the "real" Apache Attic, via a b=
oard resolution (you could do a single board resolution from the Hadoop PMC=
 to the Apache Board containing the set of projects to Attic).=20
>>=20
>> Cheers,
>> Chris
>>=20
>> On Feb 9, 2011, at 9:11 AM, Nigel Daley wrote:
>>=20
>>> After considering the feedback, I will move forward with calling a sepa=
rate vote for each contrib (or small groups of contribs). The vote will ask=
 the PMC to abandon the given contrib or move it to core.  For those we agr=
ee to abandon, I will setup an Attic wiki that will point to the last SVN r=
evision of the contrib.  I will also start a Related Projects wiki (if we d=
on't already have one) with pointers to the contrib modules that folks have=
 volunteered to keep developing elsewhere.
>>>=20
>>> Cheers,
>>> Nige
>>>=20
>>> On Jan 30, 2011, at 7:42 PM, Nigel Daley wrote:
>>>=20
>>>> Folks,
>>>>=20
>>>> Now that http://apache-extras.org is launched (https://blogs.apache.or=
g/foundation/entry/the_apache_software_foundation_launches) I'd like to sta=
rt a discussion on moving contrib components out of common, mapreduce, and =
hdfs. =20
>>>>=20
>>>> These contrib components complicate the builds, cause test failures th=
at nobody seems to care about, have releases that are tied to Hadoop's long=
 release cycles, etc.  Most folks I've talked with agree that these contrib=
 components would be better served by being pulled out of Hadoop and hosted=
 elsewhere. The new apache-extras code hosting site seems like a natural *d=
efault* location for migrating these contrib projects.  Perhaps some should=
 graduate from contrib to src (ie from contrib to core of the project they'=
re included in).  If folks agree, we'll need to come up with a mapping of c=
ontrib component to it's final destination and file a jira.
>>>>=20
>>>> Here are the contrib components by project (hopefully I didn't miss an=
y).
>>>>=20
>>>> Common Contrib:
>>>> failmon
>>>> hod
>>>> test
>>>>=20
>>>>=20
>>>> MapReduce Contrib:
>>>> capacity-scheduler -- move to MR core?
>>>> data_join
>>>> dynamic-scheduler
>>>> eclipse-plugin
>>>> fairscheduler -- move to MR core?
>>>> gridmix
>>>> index
>>>> mrunit
>>>> mumak
>>>> raid
>>>> sqoop
>>>> streaming -- move to MR core?
>>>> vaidya
>>>> vertica
>>>>=20
>>>>=20
>>>> HDFS Contrib:
>>>> fuse-dfs
>>>> hdfsproxy
>>>> thriftfs
>>>>=20
>>>>=20
>>>> Cheers,
>>>> Nige
>>>=20
>>=20
>>=20
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW:   http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>=20
>=20


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


From general-return-2973-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 10 07:13:34 2011
Return-Path: <general-return-2973-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31930 invoked from network); 10 Feb 2011 07:13:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Feb 2011 07:13:34 -0000
Received: (qmail 92017 invoked by uid 500); 10 Feb 2011 07:13:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91814 invoked by uid 500); 10 Feb 2011 07:13:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91806 invoked by uid 99); 10 Feb 2011 07:13:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Feb 2011 07:13:29 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of dhruba@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Feb 2011 07:13:23 +0000
Received: by wye20 with SMTP id 20so1047859wye.35
        for <general@hadoop.apache.org>; Wed, 09 Feb 2011 23:13:02 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=BbeEbMUXOi9+IvbWJkktq+KsJPKpFw/DnafBjfD2YKA=;
        b=TXQEkGj1XbmSb7iW2D7/muNAgXTrVpD92aiV2wcBjbB7SJsrbNTQPDtKj25A/npDa0
         99BbYh5MgtOGsgldozc3ujCunEcs3tAJyZ2F6cmrovZrbEoPrbQ5m2n9J0kM069LVKYq
         Cnqh60zivrCIvNWb42XNMkX0+zDUffyutO47g=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=nsW/auMYzQvx1JrjmrU068O8Z0hXwRUiILH6RslCWdkWw848wt+bUU0XBf+gIzQiLW
         Kyz6Rj5QQHOFNnrn5KN3yYD1wEr03ZSpB+et/x0P6iyXrpjuX2BwHjKYcQimlVmI8t+1
         F+o8fwdNhH86poHeyE3dbbzU6A4YAWok7yGBA=
MIME-Version: 1.0
Received: by 10.227.154.130 with SMTP id o2mr5928331wbw.137.1297321982835;
 Wed, 09 Feb 2011 23:13:02 -0800 (PST)
Received: by 10.216.185.135 with HTTP; Wed, 9 Feb 2011 23:13:02 -0800 (PST)
In-Reply-To: <AANLkTikMgUY367cxGPvr0okSdS5TnVMc58C2qsy491Po@mail.gmail.com>
References: <C98FD122-F11B-4A3F-8D34-3C547392F881@mac.com>
	<AANLkTikMgUY367cxGPvr0okSdS5TnVMc58C2qsy491Po@mail.gmail.com>
Date: Wed, 9 Feb 2011 23:13:02 -0800
Message-ID: <AANLkTinSiqyBYydQpSX1QTw7HxR5CSdGsoPvNrgU2cFk@mail.gmail.com>
Subject: Re: [VOTE] Abandon failmon Common contrib
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e657b052f23b2c049be84f0d
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e657b052f23b2c049be84f0d
Content-Type: text/plain; charset=ISO-8859-1

+1

On Wed, Feb 9, 2011 at 5:23 PM, Tom White <tom.e.white@gmail.com> wrote:

> +1
>
> Tom
>
> On Wed, Feb 9, 2011 at 9:18 AM, Nigel Daley <ndaley@mac.com> wrote:
> > I think the PMC should abandon the failmon common contrib component.
>  It's last meaningful contribution was it's original commit in August of
> 2008:
> >
> > HADOOP-3585. FailMon package for hardware failure monitoring and analysis
> of anomalies. (Ioannis Koltsidas via dhruba)
> >
> > Here is my +1.
> >
> > Nige
> >
>



-- 
Connect to me at http://www.facebook.com/dhruba

--0016e657b052f23b2c049be84f0d--

From general-return-2974-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 10 09:41:43 2011
Return-Path: <general-return-2974-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9524 invoked from network); 10 Feb 2011 09:41:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Feb 2011 09:41:32 -0000
Received: (qmail 88883 invoked by uid 500); 10 Feb 2011 09:41:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88747 invoked by uid 500); 10 Feb 2011 09:41:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88739 invoked by uid 99); 10 Feb 2011 09:41:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Feb 2011 09:41:26 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Feb 2011 09:41:18 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 7386E1BA75C
	for <general@hadoop.apache.org>; Thu, 10 Feb 2011 09:40:56 +0000 (GMT)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id FJruB7J7Mjgr for <general@hadoop.apache.org>;
	Thu, 10 Feb 2011 09:40:56 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by colossus.hpl.hp.com (Postfix) with ESMTPS id C949C1BA741
	for <general@hadoop.apache.org>; Thu, 10 Feb 2011 09:40:55 +0000 (GMT)
MailScanner-NULL-Check: 1297935641.27782@Crs+uuqUq/ffkFPNGeuVBQ
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p1A9efAt021270
	for <general@hadoop.apache.org>; Thu, 10 Feb 2011 09:40:41 GMT
Message-ID: <4D53B299.30206@apache.org>
Date: Thu, 10 Feb 2011 09:40:41 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Abandon failmon Common contrib
References: <C98FD122-F11B-4A3F-8D34-3C547392F881@mac.com>
In-Reply-To: <C98FD122-F11B-4A3F-8D34-3C547392F881@mac.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p1A9efAt021270
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 09/02/11 17:18, Nigel Daley wrote:
> I think the PMC should abandon the failmon common contrib component.  It's last meaningful contribution was it's original commit in August of 2008:
>
> HADOOP-3585. FailMon package for hardware failure monitoring and analysis of anomalies. (Ioannis Koltsidas via dhruba)
>
> Here is my +1.
>
> Nige

=0

I'm not sure that apache-extras is the right place. I'd more interested 
in putting it to the apache attic and making sure that the svn to git 
bridge works for them, so anyone who wants to fork it can do it by 
typing git clone

* keeps the code under the ASF, makes it easier to go live
* makes it easier for anyone with commit rights to pull in the source 
and work with it
* if someone wants to take it live, they will do it within apache, and 
so feed back into the code base.

we have an attic, so let's move it there.

From general-return-2975-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 10 09:45:39 2011
Return-Path: <general-return-2975-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12763 invoked from network); 10 Feb 2011 09:45:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Feb 2011 09:45:38 -0000
Received: (qmail 95655 invoked by uid 500); 10 Feb 2011 09:45:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94994 invoked by uid 500); 10 Feb 2011 09:45:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94976 invoked by uid 99); 10 Feb 2011 09:45:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Feb 2011 09:45:31 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [88.198.2.104] (HELO koch.ro) (88.198.2.104)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Feb 2011 09:45:25 +0000
Received: from 35-126.77-83.cust.bluewin.ch ([83.77.126.35] helo=jona.localnet)
	by koch.ro with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
	(Exim 4.69)
	(envelope-from <thomas@koch.ro>)
	id 1PnT5E-0006UO-Ix; Thu, 10 Feb 2011 10:45:04 +0100
From: Thomas Koch <thomas@koch.ro>
Reply-To: thomas@koch.ro
To: infrastructure-dev@apache.org,
 general@hadoop.apache.org,
 dev@zookeeper.apache.org
Subject: Blog post on Gerrit (GIT review tool)
Date: Thu, 10 Feb 2011 10:44:54 +0100
User-Agent: KMail/1.13.5 (Linux/2.6.32-4-amd64; KDE/4.4.5; x86_64; ; )
MIME-Version: 1.0
Content-Type: Text/Plain;
  charset="us-ascii"
Content-Transfer-Encoding: 7bit
Message-Id: <201102101044.54634.thomas@koch.ro>

http://alblue.bandlem.com/2011/02/someday.html

Very nice writeup of the advantages of Gerrit and integration with Jenkins.

Gerrit is a code review plattform on top of GIT developed by Google for the 
Android project and in use at the Eclipse Foundation.

How do you think in general about GIT for the ASF? It puzzles me, that there 
doesn't seem to be much of an interest to get official GIT repositories 
despite the pain of working with Subversion. The discussion on infra@a.o is 
stale for more then 2 months now.

Best regards,

Thomas Koch, http://www.koch.ro

From general-return-2976-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 10 16:57:24 2011
Return-Path: <general-return-2976-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58137 invoked from network); 10 Feb 2011 16:57:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Feb 2011 16:57:23 -0000
Received: (qmail 57527 invoked by uid 500); 10 Feb 2011 16:57:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57298 invoked by uid 500); 10 Feb 2011 16:57:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57289 invoked by uid 99); 10 Feb 2011 16:57:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Feb 2011 16:57:18 +0000
X-ASF-Spam-Status: No, hits=-1997.8 required=5.0
	tests=ALL_TRUSTED,HTML_MESSAGE,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 10 Feb 2011 16:57:16 +0000
Received: (qmail 57997 invoked by uid 99); 10 Feb 2011 16:56:54 -0000
Received: from localhost.apache.org (HELO mail-ey0-f176.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Feb 2011 16:56:54 +0000
Received: by eyz10 with SMTP id 10so842075eyz.35
        for <general@hadoop.apache.org>; Thu, 10 Feb 2011 08:56:52 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.52.134 with SMTP id i6mr6078832bkg.36.1297357012536; Thu,
 10 Feb 2011 08:56:52 -0800 (PST)
Received: by 10.204.97.133 with HTTP; Thu, 10 Feb 2011 08:56:52 -0800 (PST)
In-Reply-To: <4D53B299.30206@apache.org>
References: <C98FD122-F11B-4A3F-8D34-3C547392F881@mac.com>
	<4D53B299.30206@apache.org>
Date: Thu, 10 Feb 2011 08:56:52 -0800
Message-ID: <AANLkTik1m8Zg28+O=omyZ+L4nOAvtU=w1q19pf0txHPN@mail.gmail.com>
Subject: Re: [VOTE] Abandon failmon Common contrib
From: "Owen O'Malley" <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c5bfdde10f1c049bf077e7
X-Virus-Checked: Checked by ClamAV on apache.org

--001636c5bfdde10f1c049bf077e7
Content-Type: text/plain; charset=UTF-8

On Thu, Feb 10, 2011 at 1:40 AM, Steve Loughran <stevel@apache.org> wrote:

> On 09/02/11 17:18, Nigel Daley wrote:
>
>> I think the PMC should abandon the failmon common contrib component.  It's
>> last meaningful contribution was it's original commit in August of 2008:
>>
>
+1


> I'm not sure that apache-extras is the right place. I'd more interested in
>> putting it to the apache attic and making sure that the svn to git bridge
>> works for them, so anyone who wants to fork it can do it by typing git clone
>
>
This is just a vote to remove it from Hadoop. If someone decides to adopt
it, they can choose where to host it (github, sourceforge, google code, or
apache extras, etc.). Since the code was in subversion, it will always be
available in the revision from before it was deleted.


> * keeps the code under the ASF, makes it easier to go live
>

Apache extras isn't under the ASF in any meaningful sense. Projects under
Apache extras do not have any restrictions on their governance model or even
software license. In particular, there are no PMCs over in Apache extras.


> we have an attic, so let's move it there.
>

Attic is for projects that have been retired. This is just a contrib
component that we are voting on retiring.

-- Owen

--001636c5bfdde10f1c049bf077e7--

From general-return-2977-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 10 21:59:52 2011
Return-Path: <general-return-2977-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 8560 invoked from network); 10 Feb 2011 21:59:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Feb 2011 21:59:52 -0000
Received: (qmail 58764 invoked by uid 500); 10 Feb 2011 21:59:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58687 invoked by uid 500); 10 Feb 2011 21:59:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58679 invoked by uid 99); 10 Feb 2011 21:59:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Feb 2011 21:59:49 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [208.97.132.83] (HELO homiemail-a24.g.dreamhost.com) (208.97.132.83)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Feb 2011 21:59:44 +0000
Received: from homiemail-a24.g.dreamhost.com (localhost [127.0.0.1])
	by homiemail-a24.g.dreamhost.com (Postfix) with ESMTP id B8E662C8075
	for <general@hadoop.apache.org>; Thu, 10 Feb 2011 13:59:23 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; q=dns; s=
	gbiv.com; b=L9k/ZN4NQt/w0OGc6OILnSKbXuXi4UutirOnbmkgbt3MRJ+HxojM
	08av/5lBs952IsBGRQS6jLVy+6h9aJhLltBoRRjTAdlkv/x3xvP0HzVWKHR4RJh7
	VwKUE3wVW+GRFsyjI4Z6Hx6fvUxuhDnWorWX/6ttQ9Qy9bMKm2soRXk=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; s=gbiv.com;
	 bh=Zh/GW1rWhFM776XnW7hx7hGwKtI=; b=Xv+cpLP6hHcIkM6RsNjk1c89+oLe
	9HYxFXEfvCJeVmRGU30jpON3KZNhOtGkOC9hWLlTf73I8WWAPNPNGV7liGvyBrmw
	MDAstlaBmNRVZvwnVIFNhUpMRajtSRO7KLb+eoPbp9LllctZ7Evquy6UuohJJGrm
	v3k0q2QG5VTruMY=
Received: from [192.168.1.84] (99-21-208-82.lightspeed.irvnca.sbcglobal.net [99.21.208.82])
	(using TLSv1 with cipher AES128-SHA (128/128 bits))
	(No client certificate requested)
	(Authenticated sender: fielding@gbiv.com)
	by homiemail-a24.g.dreamhost.com (Postfix) with ESMTPSA id A44D82C8057
	for <general@hadoop.apache.org>; Thu, 10 Feb 2011 13:59:23 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [DISCUSS] Move common, hdfs, mapreduce contrib components to apache-extras.org or elsewhere
From: "Roy T. Fielding" <fielding@gbiv.com>
In-Reply-To: <E5A12B8B-1D39-4FCB-9FC0-7A3758A0007B@jpl.nasa.gov>
Date: Thu, 10 Feb 2011 13:59:23 -0800
Content-Transfer-Encoding: 7bit
Message-Id: <7632FD7B-A1E0-4C35-BDF5-EDFF01458CFE@gbiv.com>
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com> <E6B510A3-D141-4490-9978-712681195C37@mac.com> <84E62479-A4EE-4AB5-8148-F038FE4532B5@jpl.nasa.gov> <F4A665FC-82EF-42B0-9BA1-01FAE419805D@mac.com> <E5A12B8B-1D39-4FCB-9FC0-7A3758A0007B@jpl.nasa.gov>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

The Apache Attic exists for products that have been released by
the ASF but for which there is no longer a community capable of
making decisions like "do we release a fix to this security
issue".  contrib stuff does not need to go to the Attic -- it
can just be svn removed, or svn moved to something like a
"sandbox" area that doesn't get released.

....Roy


From general-return-2978-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 02:46:08 2011
Return-Path: <general-return-2978-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42566 invoked from network); 11 Feb 2011 02:46:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 02:46:07 -0000
Received: (qmail 59687 invoked by uid 500); 11 Feb 2011 02:46:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59277 invoked by uid 500); 11 Feb 2011 02:46:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59268 invoked by uid 99); 11 Feb 2011 02:46:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 02:46:03 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.100 as permitted sender)
Received: from [17.148.16.100] (HELO asmtpout025.mac.com) (17.148.16.100)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 02:45:56 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp025.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LGF00H4AMBXZN00@asmtp025.mac.com>
 for general@hadoop.apache.org; Thu, 10 Feb 2011 18:45:34 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-02-10_08:2011-02-10,2011-02-10,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1102100217
Subject: Re: [VOTE] Abandon failmon Common contrib
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <04D296EB-2D0B-4C1E-B094-2881508AFD42@linkedin.com>
Date: Thu, 10 Feb 2011 18:45:33 -0800
Message-id: <8101A02A-4B82-4856-A033-73E93DF47447@mac.com>
References: <C98FD122-F11B-4A3F-8D34-3C547392F881@mac.com>
 <04D296EB-2D0B-4C1E-B094-2881508AFD42@linkedin.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)


On Feb 9, 2011, at 6:47 PM, Allen Wittenauer wrote:

> 
> On Feb 9, 2011, at 9:18 AM, Nigel Daley wrote:
> 
>> I think the PMC should abandon the failmon common contrib component.  It's last meaningful contribution was it's original commit in August of 2008:
> 
> 	Are there any patches in the patch queue?

Not that I could find.  (answering your own question would have been helpful :-)

> 	When it comes to contrib, last commit is pretty useless IMO.

I'll include that info in the remaining votes.

Nige

From general-return-2979-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 02:52:34 2011
Return-Path: <general-return-2979-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43462 invoked from network); 11 Feb 2011 02:52:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 02:52:33 -0000
Received: (qmail 62796 invoked by uid 500); 11 Feb 2011 02:52:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62613 invoked by uid 500); 11 Feb 2011 02:52:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62605 invoked by uid 99); 11 Feb 2011 02:52:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 02:52:30 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.105 as permitted sender)
Received: from [17.148.16.105] (HELO asmtpout030.mac.com) (17.148.16.105)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 02:52:21 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp030.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LGF00KJ2MLWTF60@asmtp030.mac.com>
 for general@hadoop.apache.org; Thu, 10 Feb 2011 18:51:33 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-02-10_08:2011-02-10,2011-02-10,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1102100219
From: Nigel Daley <ndaley@mac.com>
Subject: [VOTE] Abandon hod Common contrib
Date: Thu, 10 Feb 2011 18:51:32 -0800
Message-id: <AC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

I think the PMC should abandon the hod common contrib component.  It's last meaningful contribution was February 2009:

HADOOP-2898. Provide an option to specify a port range for Hadoop services provisioned by HOD. Contributed by Peeyush Bishnoi.

There are 44 unresolved contrib/hod issues in Jira, 1 of them Patch Available since November 2009 (HADOOP-6369).

Here is my +1.

Nige

From general-return-2980-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 02:59:31 2011
Return-Path: <general-return-2980-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44695 invoked from network); 11 Feb 2011 02:59:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 02:59:31 -0000
Received: (qmail 69946 invoked by uid 500); 11 Feb 2011 02:59:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69735 invoked by uid 500); 11 Feb 2011 02:59:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69722 invoked by uid 99); 11 Feb 2011 02:59:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 02:59:27 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.102 as permitted sender)
Received: from [17.148.16.102] (HELO asmtpout027.mac.com) (17.148.16.102)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 02:59:18 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp027.mac.com (Oracle Communications Messaging Exchange Server 7u4-18.01
 64bit (built Jul 15 2010)) with ESMTPSA id <0LGF00DDHMXBHM10@asmtp027.mac.com>
 for general@hadoop.apache.org; Thu, 10 Feb 2011 18:58:24 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-02-10_08:2011-02-10,2011-02-10,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1102100219
From: Nigel Daley <ndaley@mac.com>
Subject: [VOTE] Abandon fuse-dfs HDFS contrib
Date: Thu, 10 Feb 2011 18:58:22 -0800
Message-id: <E43E21C9-4267-440D-A0C8-9CB21E9DBADA@mac.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

I think the PMC should abandon the fuse-dfs HDFS contrib component.  It's last meaningful contribution was March 2010:

HDFS-961. dfs_readdir incorrectly parses paths. Contributed by Eli Collins.

There are 18 unresolved contrib/fuse-dfs issues in Jira, none of them Patch Available.

Here is my +1.

Nige


From general-return-2981-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 03:18:58 2011
Return-Path: <general-return-2981-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56967 invoked from network); 11 Feb 2011 03:18:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 03:18:58 -0000
Received: (qmail 81987 invoked by uid 500); 11 Feb 2011 03:18:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81537 invoked by uid 500); 11 Feb 2011 03:18:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81524 invoked by uid 99); 11 Feb 2011 03:18:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 03:18:51 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 03:18:46 +0000
Received: by wye20 with SMTP id 20so2081177wye.35
        for <general@hadoop.apache.org>; Thu, 10 Feb 2011 19:18:25 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=lRmEnvkCFSGRwSOEL7819duOEbIyjmemwjAJbTtQyYg=;
        b=JqL1yaZgT1vBarNxmZA3LTHDPC+QP9gMsYBx7YWP7VHR6D1/vrPk0zZ4gqL1O5V4lK
         VpAlzs5wVffs2N97W4e1rIEKwxkku1hcYMRqUgX9zSuKmSlVztkQaMMr4CvUSh4lkpiF
         SDD8daK9vtytKHf/vMtFCq+e/ZYRYyik9ZxWk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=PHalwdmZbdWw01G2b4wpkxU/3kfEjlMlx7Ocq/6NbJFIhxIP8pFHSdtYFYyurI7kC7
         ipuz7JEgpm24VDfDT0xzW6HRrzZdkMad0pmvYXkMXE+pgcpdAucNr/E1YWasawYH7orh
         HTxchMt3byM2v+v3ytr4wY0G5RHo0iCVTbqIU=
MIME-Version: 1.0
Received: by 10.216.55.145 with SMTP id k17mr243622wec.48.1297394305342; Thu,
 10 Feb 2011 19:18:25 -0800 (PST)
Received: by 10.216.185.135 with HTTP; Thu, 10 Feb 2011 19:18:25 -0800 (PST)
In-Reply-To: <AC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com>
References: <AC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com>
Date: Thu, 10 Feb 2011 19:18:25 -0800
Message-ID: <AANLkTikNEJjdOtf2MWgC3jzs_W8ZoHhnONLWcNESB8Qs@mail.gmail.com>
Subject: Re: [VOTE] Abandon hod Common contrib
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6dab4dfb41c73049bf926ac

--0016e6dab4dfb41c73049bf926ac
Content-Type: text/plain; charset=ISO-8859-1

+1

On Thu, Feb 10, 2011 at 6:51 PM, Nigel Daley <ndaley@mac.com> wrote:

> I think the PMC should abandon the hod common contrib component.  It's last
> meaningful contribution was February 2009:
>
> HADOOP-2898. Provide an option to specify a port range for Hadoop services
> provisioned by HOD. Contributed by Peeyush Bishnoi.
>
> There are 44 unresolved contrib/hod issues in Jira, 1 of them Patch
> Available since November 2009 (HADOOP-6369).
>
> Here is my +1.
>
> Nige
>



-- 
Connect to me at http://www.facebook.com/dhruba

--0016e6dab4dfb41c73049bf926ac--

From general-return-2982-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 03:20:38 2011
Return-Path: <general-return-2982-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57126 invoked from network); 11 Feb 2011 03:20:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 03:20:38 -0000
Received: (qmail 83180 invoked by uid 500); 11 Feb 2011 03:20:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83048 invoked by uid 500); 11 Feb 2011 03:20:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83040 invoked by uid 99); 11 Feb 2011 03:20:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 03:20:34 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 03:20:29 +0000
Received: by wwd20 with SMTP id 20so2036306wwd.29
        for <general@hadoop.apache.org>; Thu, 10 Feb 2011 19:20:08 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=IRA73qKy/MrP29qmpV0KSHhqSZxrOZC8sZsitZcIMHs=;
        b=KtGzWWjAUdH1hEuD9ihe4uEPXJNTHFQ6dS5DJpT9cLVo51cvxh7+aYqZmbRkzEvg5R
         ba6CTrcye3bTYXmLcV3rYoKDhrviKfcOpdPpJOmBrP+f+nm4XCLSyy98iloEM/2irD9F
         zlNnACbMyp1CW1ocLOSZ/lga1AU1mal9R1M1w=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=LhvDD4ue/jJ1l5bKRSlLQpm5SSy8NbyQ5C9J8um8PIrGFs73hMxK1gdEO+WsYsA4C8
         vn3e2YYdBKGaHNHhDJKteJhOkZMP4zYfUoe+i5O1i9GH7lE0mEYTQE/kCqLcxzAJnat3
         JfB1c/Ezzz82QxdreTvuF+ygX0FY+Fmagzy3E=
MIME-Version: 1.0
Received: by 10.216.20.141 with SMTP id p13mr228265wep.102.1297394408001; Thu,
 10 Feb 2011 19:20:08 -0800 (PST)
Received: by 10.216.185.135 with HTTP; Thu, 10 Feb 2011 19:20:07 -0800 (PST)
In-Reply-To: <E43E21C9-4267-440D-A0C8-9CB21E9DBADA@mac.com>
References: <E43E21C9-4267-440D-A0C8-9CB21E9DBADA@mac.com>
Date: Thu, 10 Feb 2011 19:20:07 -0800
Message-ID: <AANLkTinjKp50gu_tFKq=B+XYvvxso=501YhQx0qQDVFo@mail.gmail.com>
Subject: Re: [VOTE] Abandon fuse-dfs HDFS contrib
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163646da74d293fd049bf92cd4

--00163646da74d293fd049bf92cd4
Content-Type: text/plain; charset=ISO-8859-1

I am guessing that this is in moderate use by people and seems like a useful
thing. I do not use it myself.

-dhruba

On Thu, Feb 10, 2011 at 6:58 PM, Nigel Daley <ndaley@mac.com> wrote:

> I think the PMC should abandon the fuse-dfs HDFS contrib component.  It's
> last meaningful contribution was March 2010:
>
> HDFS-961. dfs_readdir incorrectly parses paths. Contributed by Eli Collins.
>
> There are 18 unresolved contrib/fuse-dfs issues in Jira, none of them Patch
> Available.
>
> Here is my +1.
>
> Nige
>
>


-- 
Connect to me at http://www.facebook.com/dhruba

--00163646da74d293fd049bf92cd4--

From general-return-2983-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 03:41:15 2011
Return-Path: <general-return-2983-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60819 invoked from network); 11 Feb 2011 03:41:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 03:41:14 -0000
Received: (qmail 95296 invoked by uid 500); 11 Feb 2011 03:41:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94894 invoked by uid 500); 11 Feb 2011 03:41:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94886 invoked by uid 99); 11 Feb 2011 03:41:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 03:41:09 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.95 as permitted sender)
Received: from [17.148.16.95] (HELO asmtpout020.mac.com) (17.148.16.95)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 03:41:01 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp020.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LGF00IWAOVT5530@asmtp020.mac.com>
 for general@hadoop.apache.org; Thu, 10 Feb 2011 19:40:41 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-02-11_01:2011-02-10,2011-02-11,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=1 spamscore=1
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1102100233
From: Nigel Daley <ndaley@mac.com>
Subject: [VOTE] Abandon hdfsproxy HDFS contrib
Date: Thu, 10 Feb 2011 19:40:40 -0800
Message-id: <D97CF74A-147E-4573-B188-2F9C48CFC482@mac.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

I think the PMC should abandon the hdfsproxy HDFS contrib component.  It's last meaningful contribution was August 2010:

HDFS-1340. A null delegation token is appended to the url if security is disabled when browsing filesystem.

There are 7 unresolved contrib/hdfsproxy issues in Jira, none of them Patch Available.

Here is my +1.

Nige

From general-return-2984-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 03:45:29 2011
Return-Path: <general-return-2984-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 61399 invoked from network); 11 Feb 2011 03:45:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 03:45:28 -0000
Received: (qmail 99858 invoked by uid 500); 11 Feb 2011 03:45:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99493 invoked by uid 500); 11 Feb 2011 03:45:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 99482 invoked by uid 99); 11 Feb 2011 03:45:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 03:45:23 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.100 as permitted sender)
Received: from [17.148.16.100] (HELO asmtpout025.mac.com) (17.148.16.100)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 03:45:15 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp025.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LGF00H60P1NZN70@asmtp025.mac.com>
 for general@hadoop.apache.org; Thu, 10 Feb 2011 19:44:12 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-02-11_01:2011-02-10,2011-02-11,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1102100235
From: Nigel Daley <ndaley@mac.com>
Subject: [VOTE] Abandon thriftfs HDFS contrib
Date: Thu, 10 Feb 2011 19:44:11 -0800
Message-id: <EF44CFE2-692F-4956-8B33-D125D05E2B53@mac.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

I think the PMC should abandon the thriftfs HDFS contrib component.  It's last meaningful contribution was it's original commit in August 2008:

HADOOP-3754. Add a thrift interface to access HDFS. (dhruba via omalley)

There are 5 unresolved contrib/thriftfs issues in Jira, 1 of them Patch Available since November 2009 (HDFS-789).

Here is my +1.

Nige

From general-return-2985-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 04:06:50 2011
Return-Path: <general-return-2985-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72837 invoked from network); 11 Feb 2011 04:06:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 04:06:49 -0000
Received: (qmail 10730 invoked by uid 500); 11 Feb 2011 04:06:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 10275 invoked by uid 500); 11 Feb 2011 04:06:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 10177 invoked by uid 99); 11 Feb 2011 04:06:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 04:06:44 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 04:06:36 +0000
Received: by wye20 with SMTP id 20so2106334wye.35
        for <general@hadoop.apache.org>; Thu, 10 Feb 2011 20:06:16 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=Xhm6EYz0LTusfPO7kldFXi+ToBpmewEr0rxgU1qPnd8=;
        b=Dr91FvxBsFleXedyA4JNz8M+xqAmwfktmNbaHlVHrdTfgVwmUFJU4/Q6OIBcVb4NWW
         SkrKd0Bug5sQXd8Rj4TO7eCKawuLyWdGdRTCEqEDXK4+tb6IuxK3Tw5po6PM9S/KWnPZ
         PGYuoqxZO/hWUa7sM8UvI4V90QUbRpFzEILk0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=NdmqrG3uopFmWkj5pphc5EoKfOoLkkFcU2MM3q+e1QJKMLHiOUHWhmTnUCkuXNt3py
         /wKL3Piynkth6sTXS32EuPVgpLXkocIHqec1sKIew6i09vRsZsOa9jYs4U/4oz4r0RMm
         3SS1KTyIX7BwmMkJ4iVX7xXrR8lfNXrJ2VoUg=
MIME-Version: 1.0
Received: by 10.216.7.205 with SMTP id 55mr4226wep.96.1297397176214; Thu, 10
 Feb 2011 20:06:16 -0800 (PST)
Sender: saint.ack@gmail.com
Received: by 10.216.49.11 with HTTP; Thu, 10 Feb 2011 20:06:16 -0800 (PST)
In-Reply-To: <EF44CFE2-692F-4956-8B33-D125D05E2B53@mac.com>
References: <EF44CFE2-692F-4956-8B33-D125D05E2B53@mac.com>
Date: Thu, 10 Feb 2011 20:06:16 -0800
X-Google-Sender-Auth: uddpioFo1WQQCTXP4inskPv0wwc
Message-ID: <AANLkTi=FKf_1FZznkZvEWC3qBGO6NPGB8mL0e2okjOFN@mail.gmail.com>
Subject: Re: [VOTE] Abandon thriftfs HDFS contrib
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

+1

On Thu, Feb 10, 2011 at 7:44 PM, Nigel Daley <ndaley@mac.com> wrote:
> I think the PMC should abandon the thriftfs HDFS contrib component. =A0It=
's last meaningful contribution was it's original commit in August 2008:
>
> HADOOP-3754. Add a thrift interface to access HDFS. (dhruba via omalley)
>
> There are 5 unresolved contrib/thriftfs issues in Jira, 1 of them Patch A=
vailable since November 2009 (HDFS-789).
>
> Here is my +1.
>
> Nige
>

From general-return-2986-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 04:07:23 2011
Return-Path: <general-return-2986-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72961 invoked from network); 11 Feb 2011 04:07:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 04:07:22 -0000
Received: (qmail 11877 invoked by uid 500); 11 Feb 2011 04:07:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11726 invoked by uid 500); 11 Feb 2011 04:07:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11717 invoked by uid 99); 11 Feb 2011 04:07:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 04:07:18 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 04:07:10 +0000
Received: by wwd20 with SMTP id 20so2060309wwd.29
        for <general@hadoop.apache.org>; Thu, 10 Feb 2011 20:06:50 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=PgF0ewK/opHEkykP+cTejrbnrCatS3tibBIDrCA0jrU=;
        b=BIas9Wvc/hICdduCV62gsluZQ1ebcNSd67QrBrlqvdNNrKZdN4ldj6299WY4VntxeR
         pXKgCRyS9y2xaSp5ZSINsH8Bk0Vn4ASFZ68OT3pOIehKut5+CUQY/QSjuI7BuC6aeBc7
         1/NiFciALb6g+M2lcMUZGp0035LNsnySusdSk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=IyDu6qXplWMUDO/Q5Fa4EYVrBG7qDBc+VBuTcx225DUUGyDS65nl8rxipjayB6adJY
         NeIIEiqoOwR9QllN4uqafLGI6tKmfPclePHqTNflcBzynjqpUjEOIRqKQPfVsUif+EC9
         TtNOPhiYs206AiIGZmD1DDxez2z/oEs/RImgk=
MIME-Version: 1.0
Received: by 10.216.163.203 with SMTP id a53mr89825wel.104.1297397210108; Thu,
 10 Feb 2011 20:06:50 -0800 (PST)
Sender: saint.ack@gmail.com
Received: by 10.216.49.11 with HTTP; Thu, 10 Feb 2011 20:06:50 -0800 (PST)
In-Reply-To: <D97CF74A-147E-4573-B188-2F9C48CFC482@mac.com>
References: <D97CF74A-147E-4573-B188-2F9C48CFC482@mac.com>
Date: Thu, 10 Feb 2011 20:06:50 -0800
X-Google-Sender-Auth: Nz_Rlcqc6MpGW-EZd81TFsxkUmc
Message-ID: <AANLkTikQE6xCPz_Lzp=Lgp+hp=EV6S93D4N0Voe6i+wG@mail.gmail.com>
Subject: Re: [VOTE] Abandon hdfsproxy HDFS contrib
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

+1

On Thu, Feb 10, 2011 at 7:40 PM, Nigel Daley <ndaley@mac.com> wrote:
> I think the PMC should abandon the hdfsproxy HDFS contrib component. =A0I=
t's last meaningful contribution was August 2010:
>
> HDFS-1340. A null delegation token is appended to the url if security is =
disabled when browsing filesystem.
>
> There are 7 unresolved contrib/hdfsproxy issues in Jira, none of them Pat=
ch Available.
>
> Here is my +1.
>
> Nige
>

From general-return-2987-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 04:08:21 2011
Return-Path: <general-return-2987-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73096 invoked from network); 11 Feb 2011 04:08:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 04:08:21 -0000
Received: (qmail 13395 invoked by uid 500); 11 Feb 2011 04:08:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13282 invoked by uid 500); 11 Feb 2011 04:08:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13274 invoked by uid 99); 11 Feb 2011 04:08:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 04:08:16 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 04:08:09 +0000
Received: by wye20 with SMTP id 20so2107188wye.35
        for <general@hadoop.apache.org>; Thu, 10 Feb 2011 20:07:48 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=mOZ176HrMZf9JXDUtQrXAWARzYILodDfOf75a8ZxBHI=;
        b=bCfJIN84OpK9CGT+kH+b/kW3053sLgwmzcdRX41mWdMOPNDwXrj+hrPOpQX+Ca6iF8
         OxCxw/BjyaJUS3xJSslEDGiSoVHCF1EiKoTOiEV+qwvQQ3Zqbd3vOco6vRzgYBw6ugdR
         KSB/VWOcZadPgyU1ZB2U/WOcpOJ6DltHpCR8w=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=qN1iCazzO6WNLe3pN2PmXPptdYWpcsP/3+fxCA3XPxvwrBI4bVC4WiMVfB56h6S94O
         GAb0qrTPGvlBHO+gks3MRdFcFXnv80I7WJLQ+JZB+tpehC5S940C2c0AD/v5IHIhik6s
         jv06WRkQ6655rYaaivzb32NpgKiy3eX695M3M=
MIME-Version: 1.0
Received: by 10.216.25.210 with SMTP id z60mr2947wez.104.1297397268505; Thu,
 10 Feb 2011 20:07:48 -0800 (PST)
Sender: saint.ack@gmail.com
Received: by 10.216.49.11 with HTTP; Thu, 10 Feb 2011 20:07:48 -0800 (PST)
In-Reply-To: <E43E21C9-4267-440D-A0C8-9CB21E9DBADA@mac.com>
References: <E43E21C9-4267-440D-A0C8-9CB21E9DBADA@mac.com>
Date: Thu, 10 Feb 2011 20:07:48 -0800
X-Google-Sender-Auth: 46fL5HRVRsf_-TTZFP33rhm0NZw
Message-ID: <AANLkTi=qhmTRdJZHnfR0macSarCPUBcmoTVXehY7qs9R@mail.gmail.com>
Subject: Re: [VOTE] Abandon fuse-dfs HDFS contrib
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

+1 on punting it.  Its a drag on core.  Move it elsewhere.
St.Ack


On Thu, Feb 10, 2011 at 6:58 PM, Nigel Daley <ndaley@mac.com> wrote:
> I think the PMC should abandon the fuse-dfs HDFS contrib component. =A0It=
's last meaningful contribution was March 2010:
>
> HDFS-961. dfs_readdir incorrectly parses paths. Contributed by Eli Collin=
s.
>
> There are 18 unresolved contrib/fuse-dfs issues in Jira, none of them Pat=
ch Available.
>
> Here is my +1.
>
> Nige
>
>

From general-return-2988-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 04:09:02 2011
Return-Path: <general-return-2988-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73316 invoked from network); 11 Feb 2011 04:09:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 04:09:02 -0000
Received: (qmail 15331 invoked by uid 500); 11 Feb 2011 04:09:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15239 invoked by uid 500); 11 Feb 2011 04:08:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15231 invoked by uid 99); 11 Feb 2011 04:08:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 04:08:57 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 04:08:50 +0000
Received: by mail-wy0-f176.google.com with SMTP id 20so2107188wye.35
        for <general@hadoop.apache.org>; Thu, 10 Feb 2011 20:08:30 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=y36QNXwuVNOzpq62vNmjzuCIQN3L8qnuPD3v+gYK5zc=;
        b=I8bVZKH9jCVf8W0uydkIigLmTGE850l1f+9LndEhZT+OLl+GzWHQwYBz0hGg8UBzKo
         p6d3HnMxMFdozme6ngzzhNQ72xA5xP0jp3coCZGcGVCxwo0R/j8LRAcRED6k30sieAou
         TqMwVe3nJdOmrb558vJ+0QF86OC1FXOaFaiZU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=KT/IONDfjKdp93PeLjjwgQgc+4NbqAIbnCOiARU+2pQTorWFIvqVeAoJTiLSPBuOex
         CC3xpC5UE6Pf4UR2ntoydfgodHrFfDO6ppxIF7pMNeuFl5h+EbOK/+8ccX3xUAbgEESk
         xyYtHWplpcenEwDMZ0MWz6IVA7uQaFjizF9j4=
MIME-Version: 1.0
Received: by 10.216.24.207 with SMTP id x57mr429wex.89.1297397309977; Thu, 10
 Feb 2011 20:08:29 -0800 (PST)
Sender: saint.ack@gmail.com
Received: by 10.216.49.11 with HTTP; Thu, 10 Feb 2011 20:08:29 -0800 (PST)
In-Reply-To: <AC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com>
References: <AC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com>
Date: Thu, 10 Feb 2011 20:08:29 -0800
X-Google-Sender-Auth: VSHD3nht1HCQfdfJvTRnA4CZnkA
Message-ID: <AANLkTimrVa7kv4NQJ2VnknP5v1cEMWjaLEiq5cK8wsEZ@mail.gmail.com>
Subject: Re: [VOTE] Abandon hod Common contrib
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

+1

On Thu, Feb 10, 2011 at 6:51 PM, Nigel Daley <ndaley@mac.com> wrote:
> I think the PMC should abandon the hod common contrib component. =A0It's =
last meaningful contribution was February 2009:
>
> HADOOP-2898. Provide an option to specify a port range for Hadoop service=
s provisioned by HOD. Contributed by Peeyush Bishnoi.
>
> There are 44 unresolved contrib/hod issues in Jira, 1 of them Patch Avail=
able since November 2009 (HADOOP-6369).
>
> Here is my +1.
>
> Nige
>

From general-return-2989-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 05:22:40 2011
Return-Path: <general-return-2989-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94993 invoked from network); 11 Feb 2011 05:22:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 05:22:40 -0000
Received: (qmail 52070 invoked by uid 500); 11 Feb 2011 05:22:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51503 invoked by uid 500); 11 Feb 2011 05:22:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51495 invoked by uid 99); 11 Feb 2011 05:22:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 05:22:35 +0000
X-ASF-Spam-Status: No, hits=-1997.8 required=5.0
	tests=ALL_TRUSTED,HTML_MESSAGE,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 11 Feb 2011 05:22:33 +0000
Received: (qmail 94962 invoked by uid 99); 11 Feb 2011 05:22:11 -0000
Received: from localhost.apache.org (HELO mail-bw0-f48.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 05:22:11 +0000
Received: by bwz8 with SMTP id 8so3021079bwz.35
        for <general@hadoop.apache.org>; Thu, 10 Feb 2011 21:22:09 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.52.134 with SMTP id i6mr66873bkg.36.1297401728954; Thu, 10
 Feb 2011 21:22:08 -0800 (PST)
Received: by 10.204.97.133 with HTTP; Thu, 10 Feb 2011 21:22:08 -0800 (PST)
In-Reply-To: <E43E21C9-4267-440D-A0C8-9CB21E9DBADA@mac.com>
References: <E43E21C9-4267-440D-A0C8-9CB21E9DBADA@mac.com>
Date: Thu, 10 Feb 2011 21:22:08 -0800
Message-ID: <AANLkTimMinEh_Ko7JS3VAknFJWFOgeEtbr95KUHBPo5e@mail.gmail.com>
Subject: Re: [VOTE] Abandon fuse-dfs HDFS contrib
From: "Owen O'Malley" <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c5bfdd2f7338049bfae1b6
X-Virus-Checked: Checked by ClamAV on apache.org

--001636c5bfdd2f7338049bfae1b6
Content-Type: text/plain; charset=UTF-8

On Thu, Feb 10, 2011 at 6:58 PM, Nigel Daley <ndaley@mac.com> wrote:

> I think the PMC should abandon the fuse-dfs HDFS contrib component.  It's
> last meaningful contribution was March 2010:
>

I think we should hold on to this one until there is a reasonable
replacement. -1

-- Owen

--001636c5bfdd2f7338049bfae1b6--

From general-return-2990-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 05:23:18 2011
Return-Path: <general-return-2990-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95156 invoked from network); 11 Feb 2011 05:23:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 05:23:18 -0000
Received: (qmail 53111 invoked by uid 500); 11 Feb 2011 05:23:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52981 invoked by uid 500); 11 Feb 2011 05:23:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52973 invoked by uid 99); 11 Feb 2011 05:23:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 05:23:14 +0000
X-ASF-Spam-Status: No, hits=-1997.8 required=5.0
	tests=ALL_TRUSTED,HTML_MESSAGE,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 11 Feb 2011 05:23:12 +0000
Received: (qmail 95088 invoked by uid 99); 11 Feb 2011 05:22:50 -0000
Received: from localhost.apache.org (HELO mail-bw0-f48.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 05:22:50 +0000
Received: by bwz8 with SMTP id 8so3021472bwz.35
        for <general@hadoop.apache.org>; Thu, 10 Feb 2011 21:22:48 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.98.65 with SMTP id p1mr22045353bkn.198.1297401768275; Thu,
 10 Feb 2011 21:22:48 -0800 (PST)
Received: by 10.204.97.133 with HTTP; Thu, 10 Feb 2011 21:22:48 -0800 (PST)
In-Reply-To: <AANLkTimrVa7kv4NQJ2VnknP5v1cEMWjaLEiq5cK8wsEZ@mail.gmail.com>
References: <AC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com>
	<AANLkTimrVa7kv4NQJ2VnknP5v1cEMWjaLEiq5cK8wsEZ@mail.gmail.com>
Date: Thu, 10 Feb 2011 21:22:48 -0800
Message-ID: <AANLkTin3UKcwtWey-JfMCXJBKfAq6TpVn3sTpy_5Y0-G@mail.gmail.com>
Subject: Re: [VOTE] Abandon hod Common contrib
From: "Owen O'Malley" <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016363b84b0876db8049bfae3e0
X-Virus-Checked: Checked by ClamAV on apache.org

--0016363b84b0876db8049bfae3e0
Content-Type: text/plain; charset=UTF-8

+1

--0016363b84b0876db8049bfae3e0--

From general-return-2991-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 05:24:34 2011
Return-Path: <general-return-2991-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95302 invoked from network); 11 Feb 2011 05:24:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 05:24:34 -0000
Received: (qmail 54498 invoked by uid 500); 11 Feb 2011 05:24:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54410 invoked by uid 500); 11 Feb 2011 05:24:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54402 invoked by uid 99); 11 Feb 2011 05:24:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 05:24:29 +0000
X-ASF-Spam-Status: No, hits=-1997.8 required=5.0
	tests=ALL_TRUSTED,HTML_MESSAGE,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 11 Feb 2011 05:24:28 +0000
Received: (qmail 95238 invoked by uid 99); 11 Feb 2011 05:24:08 -0000
Received: from localhost.apache.org (HELO mail-bw0-f48.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 05:24:08 +0000
Received: by bwz8 with SMTP id 8so3022087bwz.35
        for <general@hadoop.apache.org>; Thu, 10 Feb 2011 21:24:06 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.14.6 with SMTP id e6mr63384bka.9.1297401846143; Thu, 10
 Feb 2011 21:24:06 -0800 (PST)
Received: by 10.204.97.133 with HTTP; Thu, 10 Feb 2011 21:24:06 -0800 (PST)
In-Reply-To: <D97CF74A-147E-4573-B188-2F9C48CFC482@mac.com>
References: <D97CF74A-147E-4573-B188-2F9C48CFC482@mac.com>
Date: Thu, 10 Feb 2011 21:24:06 -0800
Message-ID: <AANLkTi=F2eBsACBfjYSVava+VwFcsRpqZdRRN+OHXNRJ@mail.gmail.com>
Subject: Re: [VOTE] Abandon hdfsproxy HDFS contrib
From: "Owen O'Malley" <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0003255578762b9a89049bfae8d3

--0003255578762b9a89049bfae8d3
Content-Type: text/plain; charset=UTF-8

On Thu, Feb 10, 2011 at 7:40 PM, Nigel Daley <ndaley@mac.com> wrote:

> I think the PMC should abandon the hdfsproxy HDFS contrib component.  It's
> last meaningful contribution was August 2010:
>

-1 we still use and are maintaining this.

-- Owen

--0003255578762b9a89049bfae8d3--

From general-return-2992-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 05:58:41 2011
Return-Path: <general-return-2992-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14584 invoked from network); 11 Feb 2011 05:58:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 05:58:41 -0000
Received: (qmail 73307 invoked by uid 500); 11 Feb 2011 05:58:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72981 invoked by uid 500); 11 Feb 2011 05:58:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72973 invoked by uid 99); 11 Feb 2011 05:58:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 05:58:36 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.99 as permitted sender)
Received: from [17.148.16.99] (HELO asmtpout024.mac.com) (17.148.16.99)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 05:58:27 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp024.mac.com (Oracle Communications Messaging Exchange Server 7u4-18.01
 64bit (built Jul 15 2010)) with ESMTPSA id <0LGF0098WV837F20@asmtp024.mac.com>
 for general@hadoop.apache.org; Thu, 10 Feb 2011 21:57:40 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-02-11_02:2011-02-10,2011-02-11,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1102100268
Subject: Re: [VOTE] Abandon fuse-dfs HDFS contrib
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <AANLkTimMinEh_Ko7JS3VAknFJWFOgeEtbr95KUHBPo5e@mail.gmail.com>
Date: Thu, 10 Feb 2011 21:57:39 -0800
Message-id: <0437FB76-82C0-472D-94B5-1E01B8E94E77@mac.com>
References: <E43E21C9-4267-440D-A0C8-9CB21E9DBADA@mac.com>
 <AANLkTimMinEh_Ko7JS3VAknFJWFOgeEtbr95KUHBPo5e@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org


On Feb 10, 2011, at 9:22 PM, Owen O'Malley wrote:

> On Thu, Feb 10, 2011 at 6:58 PM, Nigel Daley <ndaley@mac.com> wrote:
> 
>> I think the PMC should abandon the fuse-dfs HDFS contrib component.  It's
>> last meaningful contribution was March 2010:
>> 
> 
> I think we should hold on to this one until there is a reasonable
> replacement. -1

If a reasonable replacement was developed elsewhere, would you change your vote?  I believe some folks were interested in developing this elsewhere.

Nige


From general-return-2993-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 06:12:42 2011
Return-Path: <general-return-2993-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 26187 invoked from network); 11 Feb 2011 06:12:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 06:12:41 -0000
Received: (qmail 86553 invoked by uid 500); 11 Feb 2011 06:12:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86107 invoked by uid 500); 11 Feb 2011 06:12:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86099 invoked by uid 99); 11 Feb 2011 06:12:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 06:12:35 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.100 as permitted sender)
Received: from [17.148.16.100] (HELO asmtpout025.mac.com) (17.148.16.100)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 06:12:28 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp025.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LGF00MTMVVTMV60@asmtp025.mac.com>
 for general@hadoop.apache.org; Thu, 10 Feb 2011 22:11:54 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-02-11_02:2011-02-10,2011-02-11,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1102100270
Subject: Re: [VOTE] Abandon hdfsproxy HDFS contrib
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <AANLkTi=F2eBsACBfjYSVava+VwFcsRpqZdRRN+OHXNRJ@mail.gmail.com>
Date: Thu, 10 Feb 2011 22:11:52 -0800
Message-id: <823262F0-EE37-4A0A-8F8C-D9422541D67B@mac.com>
References: <D97CF74A-147E-4573-B188-2F9C48CFC482@mac.com>
 <AANLkTi=F2eBsACBfjYSVava+VwFcsRpqZdRRN+OHXNRJ@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)


On Feb 10, 2011, at 9:24 PM, Owen O'Malley wrote:

> On Thu, Feb 10, 2011 at 7:40 PM, Nigel Daley <ndaley@mac.com> wrote:
> 
>> I think the PMC should abandon the hdfsproxy HDFS contrib component.  It's
>> last meaningful contribution was August 2010:
>> 
> 
> -1 we still use and are maintaining this.


Who's the 'we'?  Looking at HDFS-1164 it looks like hdfsproxy was failing a unit test for 7 months.  This is exactly the reason we should thoughtfully consider whether it has a future within Hadoop.

Perhaps Y! uses a different version of hdfsproxy? 

Nige


From general-return-2994-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 06:34:33 2011
Return-Path: <general-return-2994-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30739 invoked from network); 11 Feb 2011 06:34:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 06:34:33 -0000
Received: (qmail 97869 invoked by uid 500); 11 Feb 2011 06:34:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97399 invoked by uid 500); 11 Feb 2011 06:34:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97391 invoked by uid 99); 11 Feb 2011 06:34:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 06:34:27 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 06:34:21 +0000
Received: by qyk10 with SMTP id 10so1694565qyk.14
        for <general@hadoop.apache.org>; Thu, 10 Feb 2011 22:34:00 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=hSjIGLVL99dwX1hxEN6p9jeu/dEmAIkPwTCmD4sEWow=;
        b=h6MAi3UC6DKkCGotU/bp9N0EGz5ULDlHYjuKt2vSRBX0OXAv61ElpoyItEsEBp1l2/
         aZFsXP09mFt1j3/ZXtFFTyua/mamJ3ePkiJusFdbySOzqfagXY5WepP/4rMRmNy0ZLRJ
         1uDi7SuIIr0ek/Xk/8+u9prrY+Ixdd4TAs70k=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=fQAxj88E4ulq2sUxnDnWRJNE/1/lrDxWXGCOi1NuJ7QFPK2BWqpJQxHbwAM9qqAW1Y
         FOxVbLvadIfJnTX4pvT9AkPM5MVf0X0ioNYGfTTcTnKxNodpVOKF7NnBWW+UlLwJdjWq
         hVKLVQRgek8mieb+L4nwOnsLsve8bxUKUNubE=
Received: by 10.229.250.130 with SMTP id mo2mr127605qcb.163.1297406040388;
        Thu, 10 Feb 2011 22:34:00 -0800 (PST)
Received: from [10.172.1.130] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id l12sm362315qcu.7.2011.02.10.22.33.58
        (version=TLSv1/SSLv3 cipher=OTHER);
        Thu, 10 Feb 2011 22:33:59 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [VOTE] Abandon hdfsproxy HDFS contrib
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <823262F0-EE37-4A0A-8F8C-D9422541D67B@mac.com>
Date: Fri, 11 Feb 2011 17:33:55 +1100
Content-Transfer-Encoding: quoted-printable
Message-Id: <0CBAFDBB-AC2A-495D-8936-4DF4D6731B69@Holsman.NET>
References: <D97CF74A-147E-4573-B188-2F9C48CFC482@mac.com> <AANLkTi=F2eBsACBfjYSVava+VwFcsRpqZdRRN+OHXNRJ@mail.gmail.com> <823262F0-EE37-4A0A-8F8C-D9422541D67B@mac.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)


On Feb 11, 2011, at 5:11 PM, Nigel Daley wrote:

>=20
> On Feb 10, 2011, at 9:24 PM, Owen O'Malley wrote:
>=20
>> On Thu, Feb 10, 2011 at 7:40 PM, Nigel Daley <ndaley@mac.com> wrote:
>>=20
>>> I think the PMC should abandon the hdfsproxy HDFS contrib component. =
 It's
>>> last meaningful contribution was August 2010:
>>>=20
>>=20
>> -1 we still use and are maintaining this.
>=20
>=20
> Who's the 'we'?  Looking at HDFS-1164 it looks like hdfsproxy was =
failing a unit test for 7 months.  This is exactly the reason we should =
thoughtfully consider whether it has a future within Hadoop.
>=20
> Perhaps Y! uses a different version of hdfsproxy?=20

They probably have patched it, and mistakenly forgot to submit them.. =
any chance of doing a diff on your version and submitting it?=20

Regards
Ian

>=20
> Nige
>=20


From general-return-2995-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 06:50:55 2011
Return-Path: <general-return-2995-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39610 invoked from network); 11 Feb 2011 06:50:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 06:50:55 -0000
Received: (qmail 9322 invoked by uid 500); 11 Feb 2011 06:50:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8992 invoked by uid 500); 11 Feb 2011 06:50:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8984 invoked by uid 99); 11 Feb 2011 06:50:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 06:50:49 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.96 as permitted sender)
Received: from [17.148.16.96] (HELO asmtpout021.mac.com) (17.148.16.96)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 06:50:43 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp021.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LGF00GTVXMKI170@asmtp021.mac.com>
 for general@hadoop.apache.org; Thu, 10 Feb 2011 22:49:33 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-02-11_02:2011-02-10,2011-02-11,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1102100275
Subject: Re: Hadoop-common-trunk-Commit is failing since 01/19/2011
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <229E1339-4B9E-4FBD-B6A6-EF9EFAA8FFD0@Holsman.NET>
Date: Thu, 10 Feb 2011 22:49:30 -0800
Message-id: <BF58237A-C978-47D6-AE2E-DE968DB2429C@mac.com>
References: <AANLkTinB+8KRByZFhRT-y6MiqA0urtKNs8c_XSrhP7RH@mail.gmail.com>
 <AANLkTimmcYh0hY-6ht6vaQ+S=pTGHaFxS6k9qEv=q8Cf@mail.gmail.com>
 <AANLkTinr_Exrr8VPShGKkBSMzmWX1B9nVALLLTMmvYzd@mail.gmail.com>
 <AANLkTinrheVYQm-y9n=k7TRK_BdzuZYKAWG+qMWR88ct@mail.gmail.com>
 <AANLkTi=cAhCOkqZ=n9NpBMoPzechoJmfB6KO2yL3QcVA@mail.gmail.com>
 <AANLkTi=+81g8dFMGMqP5Zc6aAkg5n0dD2QpCVg7ABzd6@mail.gmail.com>
 <7DC70A70-5DE9-4288-A44B-B5EF9C454D9B@yahoo-inc.com>
 <AANLkTin9fsb3bgsa98bma+xx_C9K06C_YNKJ6vaA7=TO@mail.gmail.com>
 <F3847AE0-3FEF-4E85-8757-FA8A7B204A64@yahoo-inc.com>
 <AANLkTikCOHYJx4Qjt=gajGTcspPai0GmsxWQFogua92Q@mail.gmail.com>
 <AE3E9DCF-B2D6-4EA7-A166-DB139CD76BFD@yahoo-inc.com>
 <07AA15F8-B4F2-4B5B-8A65-600B31D96D83@mac.com>
 <2E13A2FF-8D87-4319-8751-B1884B6B934F@Holsman.NET>
 <23956D01-5586-4FE2-95C6-4AAD78A1F6BB@mac.com>
 <229E1339-4B9E-4FBD-B6A6-EF9EFAA8FFD0@Holsman.NET>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

> can you see if hudson is working on common-dev first?

confirmed.

nige

On Feb 8, 2011, at 3:21 PM, Ian Holsman wrote:

> I'll check it out later on today.
> even if it is sending from the wrong email address, I should be able to moderate them through regardless. (I haven't seen any recently)
> 
> can you see if hudson is working on common-dev first?
> 
> regards
> Ian
> On Feb 9, 2011, at 6:35 AM, Nigel Daley wrote:
> 
>> ah, that's the old address.  I wonder how many of our lists have the old one...
>> 
>> nige
>> 
>> On Feb 8, 2011, at 11:31 AM, Ian Holsman wrote:
>> 
>>> Hi Nige.
>>> I've subscribed it now. as it's wasn't on the subscriber list. we do have hudson@lucene.zones.apache.org.
>>> 
>>> but I haven't seen any moderation notices for it either... so I'm not sure it is generating emails.
>>> 
>>> On Feb 9, 2011, at 3:32 AM, Nigel Daley wrote:
>>> 
>>>> Hmm, haven't seen Hudson post build failures to the common-dev list lately.  
>>>> 
>>>> Ian, can you check that hudson@hudson.apache.org is still subscribed to common-dev@.  If not, please add it.
>>>> 
>>>> Thx,
>>>> Nige
>>>> 
>>>> 
>>>> On Feb 1, 2011, at 11:27 AM, Giridharan Kesavan wrote:
>>>> 
>>>>> Konstantin,
>>>>> 
>>>>> trunk/artifacts gets populated when the jar and the tar ant target are successful.
>>>>> 
>>>>> The main reason for the build failure so far is the build abort time configuration. It was set to 30mins.
>>>>> I have increased the build abort time and the builds are going on fine 
>>>>> https://hudson.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit
>>>>> 
>>>>> 
>>>>> Thanks,
>>>>> Giri
>>>>> 
>>>>> On Feb 1, 2011, at 12:40 AM, Konstantin Shvachko wrote:
>>>>> 
>>>>>> Giri,
>>>>>> 
>>>>>> Looking at configuration of Hadoop-Common-trunk-Commit/
>>>>>> There seems to be errors in the Post-build Actions.
>>>>>> It is complaining that
>>>>>> 'trunk' exists but not 'trunk/artifacts/...'
>>>>>> Is it possible that this misconfiguration is the reason of failures?
>>>>>> 
>>>>>> --Konstantin
>>>>>> 
>>>>>> 
>>>>>> On Mon, Jan 31, 2011 at 4:40 PM, Giridharan Kesavan
>>>>>> <gkesavan@yahoo-inc.com>wrote:
>>>>>> 
>>>>>>> Konstantin,
>>>>>>> 
>>>>>>> I think I need to restart the slave which is running the commit build. For
>>>>>>> now I have published the common artifact manually from commandline.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Giri
>>>>>>> 
>>>>>>> On Jan 31, 2011, at 4:27 PM, Konstantin Shvachko wrote:
>>>>>>> 
>>>>>>>> Giri
>>>>>>>> looks like the last run you started failed the same way as previous ones.
>>>>>>>> Any thoughts on what's going on?
>>>>>>>> Thanks,
>>>>>>>> --Konstantin
>>>>>>>> 
>>>>>>>> On Mon, Jan 31, 2011 at 3:33 PM, Giridharan Kesavan
>>>>>>>> <gkesavan@yahoo-inc.com>wrote:
>>>>>>>> 
>>>>>>>>> ant mvn-deploy would publish snapshot artifact to the apache maven
>>>>>>>>> repository as long you have the right credentials in ~/.m2/settings.xml.
>>>>>>>>> 
>>>>>>>>> For settings.xml template pls look at
>>>>>>>>> http://wiki.apache.org/hadoop/HowToRelease
>>>>>>>>> 
>>>>>>>>> I'm pushing the latest common artifacts now.
>>>>>>>>> 
>>>>>>>>> -Giri
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Jan 31, 2011, at 3:11 PM, Jakob Homan wrote:
>>>>>>>>> 
>>>>>>>>>> By manually installing a new core jar into the cache, I can compile
>>>>>>>>>> trunk.  Looks like we just need to kick a new Core into maven.  Are
>>>>>>>>>> there instructions somewhere for committers to do this?  I know Nigel
>>>>>>>>>> and Owen know how, but I don't know if the knowledge is diffused past
>>>>>>>>>> them.
>>>>>>>>>> -Jakob
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Mon, Jan 31, 2011 at 1:57 PM, Konstantin Shvachko
>>>>>>>>>> <shv.hadoop@gmail.com> wrote:
>>>>>>>>>>> Current trunk for HDFS and MapReduce are not compiling at the moment.
>>>>>>>>> Try to
>>>>>>>>>>> build trunk.
>>>>>>>>>>> This is the result of that changes to common api introduced by
>>>>>>>>> HADOOP-6904
>>>>>>>>>>> are not promoted to HDFS and MR trunks.
>>>>>>>>>>> HDFS-1335 and MAPREDUCE-2263 depend on these changes.
>>>>>>>>>>> 
>>>>>>>>>>> Common is not promoted to HDFS and MR because
>>>>>>> Hadoop-Common-trunk-Commit
>>>>>>>>>>> build is broken. See here.
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> https://hudson.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/
>>>>>>>>>>> 
>>>>>>>>>>> As I see the last successful build was on 01/19, which integrated
>>>>>>>>>>> HADOOP-6864.
>>>>>>>>>>> I think this is when JNI changes were introduced, which cannot be
>>>>>>>>> digested
>>>>>>>>>>> by Hudson since then.
>>>>>>>>>>> 
>>>>>>>>>>> Anybody with gcc active could you please verify if the problem is
>>>>>>> caused
>>>>>>>>> by
>>>>>>>>>>> HADOOP-6864.
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> --Konstantin
>>>>>>>>>>> 
>>>>>>>>>>> On Mon, Jan 31, 2011 at 1:36 PM, Ted Dunning <tdunning@maprtech.com>
>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> The has been a problem with more than one build failing (Mahout is
>>>>>>> the
>>>>>>>>> one
>>>>>>>>>>>> that I saw first) due to a change in maven version which meant that
>>>>>>> the
>>>>>>>>>>>> clover license isn't being found properly.  At least, that is the
>>>>>>> tale
>>>>>>>>> I
>>>>>>>>>>>> heard from infra.
>>>>>>>>>>>> 
>>>>>>>>>>>> On Mon, Jan 31, 2011 at 1:31 PM, Eli Collins <eli@cloudera.com>
>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hey Konstantin,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The only build breakage I saw from HADOOP-6904 is MAPREDUCE-2290,
>>>>>>>>>>>>> which was fixed.  Trees from trunk are compiling against each other
>>>>>>>>>>>>> for me (eg each installed to a local maven repo), perhaps the
>>>>>>> upstream
>>>>>>>>>>>>> maven repo hasn't been updated with the latest bits yet.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Eli
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Mon, Jan 31, 2011 at 12:14 PM, Konstantin Shvachko
>>>>>>>>>>>>> <shv.hadoop@gmail.com> wrote:
>>>>>>>>>>>>>> Sending this to general to attract urgent attention.
>>>>>>>>>>>>>> Both HDFS and MapReduce are not compiling since
>>>>>>>>>>>>>> HADOOP-6904 and its hdfs and MP counterparts were committed.
>>>>>>>>>>>>>> The problem is not with this patch as described below, but I think
>>>>>>>>>>>> those
>>>>>>>>>>>>>> commits should be reversed if Common integration build cannot be
>>>>>>>>>>>>>> restored promptly.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> --Konstantin
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Fri, Jan 28, 2011 at 5:53 PM, Konstantin Shvachko
>>>>>>>>>>>>>> <shv.hadoop@gmail.com>wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I see Hadoop-common-trunk-Commit is failing and not sending any
>>>>>>>>>>>> emails.
>>>>>>>>>>>>>>> It times out on native compilation and aborts.
>>>>>>>>>>>>>>> Therefore changes are not integrated, and now it lead to hdfs and
>>>>>>>>>>>>> mapreduce
>>>>>>>>>>>>>>> both not compiling.
>>>>>>>>>>>>>>> Can somebody please take a look at this.
>>>>>>>>>>>>>>> The last few lines of the build are below.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>> --Konstantin
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> [javah] [Loaded
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/build/classes/org/apache/hadoop/security/JniBasedUnixGroupsMapping.class]
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> [javah] [Loaded
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> /homes/hudson/tools/java/jdk1.6.0_11-32/jre/lib/rt.jar(java/lang/Object.class)]
>>>>>>>>>>>>>>> [javah] [Forcefully writing file
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/build/native/Linux-i386-32/src/org/apache/hadoop/security/org_apache_hadoop_security_JniBasedUnixGroupsNetgroupMapping.h]
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> [exec] checking for gcc... gcc
>>>>>>>>>>>>>>> [exec] checking whether the C compiler works... yes
>>>>>>>>>>>>>>> [exec] checking for C compiler default output file name...
>>>>>>>>> a.out
>>>>>>>>>>>>>>> [exec] checking for suffix of executables...
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Build timed out. Aborting
>>>>>>>>>>>>>>> Build was aborted
>>>>>>>>>>>>>>> [FINDBUGS] Skipping publisher since build result is ABORTED
>>>>>>>>>>>>>>> Publishing Javadoc
>>>>>>>>>>>>>>> Archiving artifacts
>>>>>>>>>>>>>>> Recording test results
>>>>>>>>>>>>>>> No test report files were found. Configuration error?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Recording fingerprints
>>>>>>>>>>>>>>> [exec] Terminated
>>>>>>>>>>>>>>> Publishing Clover coverage report...
>>>>>>>>>>>>>>> No Clover report will be published due to a Build Failure
>>>>>>>>>>>>>>> No emails were triggered.
>>>>>>>>>>>>>>> Finished: ABORTED
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 


From general-return-2996-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 07:09:23 2011
Return-Path: <general-return-2996-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50012 invoked from network); 11 Feb 2011 07:09:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 07:09:22 -0000
Received: (qmail 18918 invoked by uid 500); 11 Feb 2011 07:09:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18466 invoked by uid 500); 11 Feb 2011 07:09:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18454 invoked by uid 99); 11 Feb 2011 07:09:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 07:09:16 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.101 as permitted sender)
Received: from [17.148.16.101] (HELO asmtpout026.mac.com) (17.148.16.101)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 07:09:09 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp026.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LGF005R1YIM3U10@asmtp026.mac.com>
 for general@hadoop.apache.org; Thu, 10 Feb 2011 23:08:47 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-02-11_02:2011-02-10,2011-02-11,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1102100277
From: Nigel Daley <ndaley@mac.com>
Subject: [VOTE] Abandon mrunit MapReduce contrib
Date: Thu, 10 Feb 2011 23:08:46 -0800
Message-id: <77405974-6771-4604-926B-976240743F3C@mac.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

I think the PMC should abandon the mrunit MapReduce contrib component.  The originator of mrunit and primary maintainer (Aaron Kimball) is moving the active development elsewhere. 

There are 2 unresolved contrib/mrunit issues in Jira, none of them Patch Available.

Here is my +1.

Nige

From general-return-2997-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 07:34:10 2011
Return-Path: <general-return-2997-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54806 invoked from network); 11 Feb 2011 07:34:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 07:34:09 -0000
Received: (qmail 35192 invoked by uid 500); 11 Feb 2011 07:34:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34809 invoked by uid 500); 11 Feb 2011 07:34:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34801 invoked by uid 99); 11 Feb 2011 07:34:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 07:34:04 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.97 as permitted sender)
Received: from [17.148.16.97] (HELO asmtpout022.mac.com) (17.148.16.97)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 07:33:56 +0000
MIME-version: 1.0
Content-type: multipart/alternative;
 boundary="Boundary_(ID_aq5ekLtfrQ+vWRfb9UlgKA)"
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp022.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LGF006U2ZNZ1D00@asmtp022.mac.com>
 for general@hadoop.apache.org; Thu, 10 Feb 2011 23:33:36 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-02-11_02:2011-02-10,2011-02-11,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1102100280
From: Nigel Daley <ndaley@mac.com>
Subject: Hadoop 0.22 Blockers
Date: Thu, 10 Feb 2011 23:33:35 -0800
Message-id: <57A4D9DB-2CD2-4838-AD65-7DEBA1F3F0EA@mac.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

--Boundary_(ID_aq5ekLtfrQ+vWRfb9UlgKA)
Content-type: text/plain; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT

Tom has created a public Jira filter for 0.22 blockers (thanks Tom!):
https://issues.apache.org/jira/secure/IssueNavigator.jspa?mode=hide&requestId=12313687
With the new version of Jira, this single query across projects & versions is now possible.

Looks like 26 issues marked blocker.  9 of them unassigned -- would love to get those assigned.  I'll start reviewing these issues more closely over the weekend.

Cheers,
Nige

--Boundary_(ID_aq5ekLtfrQ+vWRfb9UlgKA)--

From general-return-2998-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 07:37:31 2011
Return-Path: <general-return-2998-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55652 invoked from network); 11 Feb 2011 07:37:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 07:37:30 -0000
Received: (qmail 38815 invoked by uid 500); 11 Feb 2011 07:37:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38614 invoked by uid 500); 11 Feb 2011 07:37:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 38606 invoked by uid 99); 11 Feb 2011 07:37:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 07:37:25 +0000
X-ASF-Spam-Status: No, hits=3.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of akimball83@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 07:37:18 +0000
Received: by bwz8 with SMTP id 8so3095445bwz.35
        for <general@hadoop.apache.org>; Thu, 10 Feb 2011 23:36:57 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type;
        bh=bgO1wRt+TG6j2Mqv9efWpyYnqTwHF5hf5PCinGAB/1Q=;
        b=QjMUF9Gd3U487yl5rixzW+yjhkbhiuzd79u/SkS16HQGzC8mhN5gB3go/1hS9Oonrd
         UYhUh0NYzVAbCNaeDie8vGgcNz8EmaVowjfYkApRqhGxuiElCVwymcJtK6JqYw3HwwmP
         OZ+r3MChOkGOZA9DUeFB9y3B7HUNO7GEd+QNo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=MFIcPssfUMxjNlEONrv/2656Y/qSS9MDGLlphPdzWgKhuOmJB+ehNw5goLsW4x2tCJ
         FvXlfh2j20yEHmcD63Iv3Z7uMFaKFvVNuWEjUYffFzjKJAAUu7NgziAIa95uziytWcgd
         xxqpm5KwASD1zZo5OC8BVpg8DcGoNumu5yxkk=
Received: by 10.204.97.193 with SMTP id m1mr157979bkn.46.1297409817539; Thu,
 10 Feb 2011 23:36:57 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.54.72 with HTTP; Thu, 10 Feb 2011 23:36:37 -0800 (PST)
In-Reply-To: <77405974-6771-4604-926B-976240743F3C@mac.com>
References: <77405974-6771-4604-926B-976240743F3C@mac.com>
From: Aaron Kimball <akimball83@gmail.com>
Date: Thu, 10 Feb 2011 23:36:37 -0800
Message-ID: <AANLkTim80KJ3f14iM6Ltzy7UG613myhN-AiwQ3pCmVdi@mail.gmail.com>
Subject: Re: [VOTE] Abandon mrunit MapReduce contrib
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636457ade4d7519049bfcc365

--001636457ade4d7519049bfcc365
Content-Type: text/plain; charset=ISO-8859-1

+1. Eric Sammer and I will be working on this via github. (Come join us!)

- Aaron

On Thu, Feb 10, 2011 at 11:08 PM, Nigel Daley <ndaley@mac.com> wrote:

> I think the PMC should abandon the mrunit MapReduce contrib component.  The
> originator of mrunit and primary maintainer (Aaron Kimball) is moving the
> active development elsewhere.
>
> There are 2 unresolved contrib/mrunit issues in Jira, none of them Patch
> Available.
>
> Here is my +1.
>
> Nige
>

--001636457ade4d7519049bfcc365--

From general-return-2999-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 07:52:54 2011
Return-Path: <general-return-2999-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58973 invoked from network); 11 Feb 2011 07:52:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 07:52:53 -0000
Received: (qmail 51941 invoked by uid 500); 11 Feb 2011 07:52:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51509 invoked by uid 500); 11 Feb 2011 07:52:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51501 invoked by uid 99); 11 Feb 2011 07:52:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 07:52:47 +0000
X-ASF-Spam-Status: No, hits=-1997.8 required=5.0
	tests=ALL_TRUSTED,HTML_MESSAGE,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 11 Feb 2011 07:52:47 +0000
Received: (qmail 58921 invoked by uid 99); 11 Feb 2011 07:52:26 -0000
Received: from localhost.apache.org (HELO mail-bw0-f48.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 07:52:26 +0000
Received: by bwz8 with SMTP id 8so3104435bwz.35
        for <general@hadoop.apache.org>; Thu, 10 Feb 2011 23:52:24 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.14.6 with SMTP id e6mr160602bka.9.1297410744610; Thu, 10
 Feb 2011 23:52:24 -0800 (PST)
Received: by 10.204.97.133 with HTTP; Thu, 10 Feb 2011 23:52:24 -0800 (PST)
In-Reply-To: <AANLkTim80KJ3f14iM6Ltzy7UG613myhN-AiwQ3pCmVdi@mail.gmail.com>
References: <77405974-6771-4604-926B-976240743F3C@mac.com>
	<AANLkTim80KJ3f14iM6Ltzy7UG613myhN-AiwQ3pCmVdi@mail.gmail.com>
Date: Thu, 10 Feb 2011 23:52:24 -0800
Message-ID: <AANLkTiknkrWK=GzGMVj4_Z6p5Jn8TQu1P5TkaFSgORJk@mail.gmail.com>
Subject: Re: [VOTE] Abandon mrunit MapReduce contrib
From: "Owen O'Malley" <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0003255578768f6ff3049bfcfa45

--0003255578768f6ff3049bfcfa45
Content-Type: text/plain; charset=UTF-8

On Thu, Feb 10, 2011 at 11:36 PM, Aaron Kimball <akimball83@gmail.com>wrote:

> +1. Eric Sammer and I will be working on this via github. (Come join us!)


Votes to remove code should be because the PMC doesn't think the code is
worth maintaining any more. I don't think that applies in this case. Aaron
is a committer in Hadoop now and I'd strongly encourage you both to give
your changes back to Apache instead of forking it into GitHub.

-- Owen

--0003255578768f6ff3049bfcfa45--

From general-return-3000-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 10:04:29 2011
Return-Path: <general-return-3000-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 11876 invoked from network); 11 Feb 2011 10:04:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 10:04:29 -0000
Received: (qmail 13094 invoked by uid 500); 11 Feb 2011 10:04:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13041 invoked by uid 500); 11 Feb 2011 10:04:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13033 invoked by uid 99); 11 Feb 2011 10:04:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 10:04:23 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bernd.fondermann@googlemail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 10:04:18 +0000
Received: by iwn2 with SMTP id 2so2392297iwn.35
        for <general@hadoop.apache.org>; Fri, 11 Feb 2011 02:03:57 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type:content-transfer-encoding;
        bh=aiCo7NoBfekbfQiJrSyCjzcpPwQv9f3u87UEUUoxQCI=;
        b=X/ILtNpWE/691A8uh/+TXzJaDUqvt9VjWFdK1BfyCAbCO5J+3GvnVYGF2c1Fj3pjN+
         4Jbp7A0p3W/lmnD3vMknMbDUqyGRlE5xx1Pfpdvk9cwEA8joRM+ZD3IzvskeiCLGa9Wr
         0tS2yxKX/V+Tdh0iY3n3B8vuyX20zeOVwKl2c=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=gQbgN9Yzgc/oI9K+TbT7vHv4HHbrTI9ErDuI+Atm6zIx1Gd540U0tsPeigy/PfAJMT
         TZY6BwIPzdrIu/sDxRIIdWuMpO9wtS1+F75WvlO4DtxRvJTnyEoOQ/OMeTIjHphn1/V7
         ibKfYTEaczq0ZKUs/f5tAoPYlnRY0KtZRvdY8=
MIME-Version: 1.0
Received: by 10.42.178.193 with SMTP id bn1mr393634icb.14.1297418637452; Fri,
 11 Feb 2011 02:03:57 -0800 (PST)
Received: by 10.42.8.149 with HTTP; Fri, 11 Feb 2011 02:03:57 -0800 (PST)
In-Reply-To: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com>
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com>
Date: Fri, 11 Feb 2011 11:03:57 +0100
Message-ID: <AANLkTi=75e6oyBuGVivXP3is+76gOODSc1NcLKfg=VxF@mail.gmail.com>
Subject: Re: [DISCUSS] Move common, hdfs, mapreduce contrib components to
 apache-extras.org or elsewhere
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

-1.

Move it away from TRUNK so it doesn't affect builds is a much better
(and simpler) solution. If someone wants to revive it, he can within
the bounds of Apache Hadoop and will become a part of the Hadoop
community, which would be good.
If you'd move it off-site, if the code ever gets new contributors,
it's hard to integrate them (code and contributors) into Hadoop again.

AFAIUI, apache-extras is for placing non-Apache code closer to the
related Apache projects, not for moving our code away from our own
premises.

  Bernd

On Mon, Jan 31, 2011 at 04:42, Nigel Daley <ndaley@mac.com> wrote:
> Folks,
>
> Now that http://apache-extras.org is launched (https://blogs.apache.org/f=
oundation/entry/the_apache_software_foundation_launches) I'd like to start =
a discussion on moving contrib components out of common, mapreduce, and hdf=
s.
>
> These contrib components complicate the builds, cause test failures that =
nobody seems to care about, have releases that are tied to Hadoop's long re=
lease cycles, etc. =A0Most folks I've talked with agree that these contrib =
components would be better served by being pulled out of Hadoop and hosted =
elsewhere. The new apache-extras code hosting site seems like a natural *de=
fault* location for migrating these contrib projects. =A0Perhaps some shoul=
d graduate from contrib to src (ie from contrib to core of the project they=
're included in). =A0If folks agree, we'll need to come up with a mapping o=
f contrib component to it's final destination and file a jira.
>
> Here are the contrib components by project (hopefully I didn't miss any).
>
> Common Contrib:
> =A0failmon
> =A0hod
> =A0test
>
>
> MapReduce Contrib:
> =A0capacity-scheduler -- move to MR core?
> =A0data_join
> =A0dynamic-scheduler
> =A0eclipse-plugin
> =A0fairscheduler -- move to MR core?
> =A0gridmix
> =A0index
> =A0mrunit
> =A0mumak
> =A0raid
> =A0sqoop
> =A0streaming -- move to MR core?
> =A0vaidya
> =A0vertica
>
>
> HDFS Contrib:
> =A0fuse-dfs
> =A0hdfsproxy
> =A0thriftfs
>
>
> Cheers,
> Nige
>

From general-return-3001-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 10:29:29 2011
Return-Path: <general-return-3001-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27391 invoked from network); 11 Feb 2011 10:29:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 10:29:28 -0000
Received: (qmail 46693 invoked by uid 500); 11 Feb 2011 10:29:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46310 invoked by uid 500); 11 Feb 2011 10:29:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46302 invoked by uid 99); 11 Feb 2011 10:29:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 10:29:23 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bernd.fondermann@googlemail.com designates 209.85.210.176 as permitted sender)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 10:29:19 +0000
Received: by iyb26 with SMTP id 26so2419624iyb.35
        for <general@hadoop.apache.org>; Fri, 11 Feb 2011 02:28:58 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type:content-transfer-encoding;
        bh=Mp4q3U4JzK1NYRhyWOEuNZQeJwBxL22LdeQFAHlIJoE=;
        b=V7HW2hJH/hJH0nBjAPQ6AK2V5nJsbcx4W/7QhkNpLztF135LcMPu/eYWz/Z1+zRyAA
         O7jOO71q72i0UqXWhTiCANmWMyn2yD8oLY5Y3Hq2c6gqOIWLKiY/bS/5h3v1D/8p0IpX
         2aIA3aJL4e5rc+9vQ9Lq7Mwxzd9LzscF8p/Jg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=CKr0zXGBPzq36XkIVWXM4OW/bqDJeGLjmSBqTJeVT8SrwfKTFHjWXfXZ/pjNmc7MQv
         WKC0t4xYzEX/mb6jHtr4wWHw6stTafWy0qlA+R0/845hE41KCEuqf6oABcq2XszU1XIu
         ZOi4eu8v6UTcovpqGqpyDdWB89dwYWP9mes94=
MIME-Version: 1.0
Received: by 10.42.230.137 with SMTP id jm9mr386804icb.282.1297420138292; Fri,
 11 Feb 2011 02:28:58 -0800 (PST)
Received: by 10.42.8.149 with HTTP; Fri, 11 Feb 2011 02:28:58 -0800 (PST)
In-Reply-To: <0CBAFDBB-AC2A-495D-8936-4DF4D6731B69@Holsman.NET>
References: <D97CF74A-147E-4573-B188-2F9C48CFC482@mac.com>
	<AANLkTi=F2eBsACBfjYSVava+VwFcsRpqZdRRN+OHXNRJ@mail.gmail.com>
	<823262F0-EE37-4A0A-8F8C-D9422541D67B@mac.com>
	<0CBAFDBB-AC2A-495D-8936-4DF4D6731B69@Holsman.NET>
Date: Fri, 11 Feb 2011 11:28:58 +0100
Message-ID: <AANLkTimUhDbTnwn6Hk2oYXFJMts+A=FnL-q-veXqX1m0@mail.gmail.com>
Subject: Re: [VOTE] Abandon hdfsproxy HDFS contrib
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Fri, Feb 11, 2011 at 07:33, Ian Holsman <hadoop@holsman.net> wrote:
>
> On Feb 11, 2011, at 5:11 PM, Nigel Daley wrote:
>
>>
>> On Feb 10, 2011, at 9:24 PM, Owen O'Malley wrote:
>>
>>> On Thu, Feb 10, 2011 at 7:40 PM, Nigel Daley <ndaley@mac.com> wrote:
>>>
>>>> I think the PMC should abandon the hdfsproxy HDFS contrib component. =
=A0It's
>>>> last meaningful contribution was August 2010:
>>>>
>>>
>>> -1 we still use and are maintaining this.
>>
>>
>> Who's the 'we'? =A0Looking at HDFS-1164 it looks like hdfsproxy was fail=
ing a unit test for 7 months. =A0This is exactly the reason we should thoug=
htfully consider whether it has a future within Hadoop.
>>
>> Perhaps Y! uses a different version of hdfsproxy?
>
> They probably have patched it, and mistakenly forgot to submit them.. any=
 chance of doing a diff on your version and submitting it?

Please keep in mind: The original author(s) would need to submit it -
not a proxy.

  Bernd

From general-return-3002-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 12:47:24 2011
Return-Path: <general-return-3002-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93588 invoked from network); 11 Feb 2011 12:47:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 12:47:24 -0000
Received: (qmail 4888 invoked by uid 500); 11 Feb 2011 12:47:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4330 invoked by uid 500); 11 Feb 2011 12:47:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4321 invoked by uid 99); 11 Feb 2011 12:47:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 12:47:17 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 12:47:08 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id C8A99B7DB8
	for <general@hadoop.apache.org>; Fri, 11 Feb 2011 12:46:46 +0000 (GMT)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id GKboa9bCPPvA for <general@hadoop.apache.org>;
	Fri, 11 Feb 2011 12:46:36 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id E78D6B7D3B
	for <general@hadoop.apache.org>; Fri, 11 Feb 2011 12:46:35 +0000 (GMT)
MailScanner-NULL-Check: 1298033181.61416@ZaOJysoEZnjwnnC7pHgrzQ
Received: from [16.24.234.180] ([16.24.234.180])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p1BCkLcY027883
	for <general@hadoop.apache.org>; Fri, 11 Feb 2011 12:46:21 GMT
Message-ID: <4D552F97.8090309@apache.org>
Date: Fri, 11 Feb 2011 12:46:15 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-GB; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Abandon hod Common contrib
References: <AC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com>
In-Reply-To: <AC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p1BCkLcY027883
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 11/02/2011 02:51, Nigel Daley wrote:
> I think the PMC should abandon the hod common contrib component.  It's last meaningful contribution was February 2009:
>
> HADOOP-2898. Provide an option to specify a port range for Hadoop services provisioned by HOD. Contributed by Peeyush Bishnoi.
>
> There are 44 unresolved contrib/hod issues in Jira, 1 of them Patch Available since November 2009 (HADOOP-6369).
>
> Here is my +1.
>
> Nige

+1

From general-return-3003-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 14:16:21 2011
Return-Path: <general-return-3003-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35632 invoked from network); 11 Feb 2011 14:16:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 14:16:20 -0000
Received: (qmail 33194 invoked by uid 500); 11 Feb 2011 14:16:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32769 invoked by uid 500); 11 Feb 2011 14:16:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32761 invoked by uid 99); 11 Feb 2011 14:16:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 14:16:14 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [129.93.165.11] (HELO cse-mail.unl.edu) (129.93.165.11)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 14:16:03 +0000
Received: from cse-barracuda.cse.unl.edu (cse-barracuda.unl.edu [129.93.164.185])
	(authenticated bits=0)
	by cse-mail.unl.edu (8.14.3/8.14.3) with ESMTP id p1BEFbm9014461
	for <general@hadoop.apache.org>; Fri, 11 Feb 2011 08:15:42 -0600 (CST)
X-ASG-Debug-ID: 1297433737-4b5c0dbe3e3d-IjwG86
Received: from cse.unl.edu (cse.unl.edu [129.93.165.2]) by cse-barracuda.cse.unl.edu with ESMTP id 91iE92kbqhLpiM87 for <general@hadoop.apache.org>; Fri, 11 Feb 2011 08:15:37 -0600 (CST)
X-Barracuda-Envelope-From: bbockelm@cse.unl.edu
X-Barracuda-RBL-Trusted-Forwarder: 129.93.165.2
Received: from pcp088890pcs.unl.edu (pcp088890pcs.unl.edu [129.93.158.5])
	(authenticated bits=0)
	by cse.unl.edu (8.14.4/8.14.3) with ESMTP id p1BEFaqS008530
	(version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO)
	for <general@hadoop.apache.org>; Fri, 11 Feb 2011 08:15:36 -0600
From: Brian Bockelman <bbockelm@cse.unl.edu>
X-Barracuda-Apparent-Source-IP: 129.93.158.5
Content-Type: multipart/signed; boundary=Apple-Mail-1285--848526120; protocol="application/pkcs7-signature"; micalg=sha1
Subject: Re: Abandon fuse-dfs HDFS contrib
Date: Fri, 11 Feb 2011 08:15:37 -0600
X-ASG-Orig-Subj: Re: Abandon fuse-dfs HDFS contrib
Message-Id: <4269965E-D7A3-44BE-B85D-E1F4B6E97638@cse.unl.edu>
To: general@hadoop.apache.org
Mime-Version: 1.0 (Apple Message framework v1082)
X-Mailer: Apple Mail (2.1082)
X-Barracuda-Connect: cse.unl.edu[129.93.165.2]
X-Barracuda-Start-Time: 1297433737
X-Barracuda-URL: http://cse-barracuda.unl.edu:8000/cgi-mod/mark.cgi
X-Virus-Scanned: clamav-milter 0.97 at cse-mail
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using global scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=9.0 tests=
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.2.55028
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0.1 (cse-mail.unl.edu [129.93.165.11]); Fri, 11 Feb 2011 08:15:42 -0600 (CST)
X-Virus-Status: Clean
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-1285--848526120
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

(Sorry, I apparently wasn't subscribed to general@hadoop... so this =
response is going to muck up your email client's threading)

We use fuse-dfs heavily at approximately 10 sites for the LHC; at least =
3 of them at the >1 petabyte level to deliver multiple terabytes of data =
per day.

However, as things have tended to be pretty stable in Hadoop-land, we =
tend to lag significantly in versions, and I haven't noticed the JIRA =
issues pile up.

I volunteer to attempt to "tidy up house" in this contrib, propose the =
vote be tabled for 2 months to allow me time for sufficient progress.  =
If I haven't been able to fix 18 lousy bugs by then, then I suppose I =
deserve what I get ;).

Brian=

--Apple-Mail-1285--848526120
Content-Disposition: attachment;
	filename=smime.p7s
Content-Type: application/pkcs7-signature;
	name=smime.p7s
Content-Transfer-Encoding: base64

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIIKDCCA/gw
ggLgoAMCAQICASkwDQYJKoZIhvcNAQEFBQAwdTETMBEGCgmSJomT8ixkARkWA25ldDESMBAGCgmS
JomT8ixkARkWAkVTMQ4wDAYDVQQKEwVFU25ldDEgMB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9y
aXRpZXMxGDAWBgNVBAMTD0VTbmV0IFJvb3QgQ0EgMTAeFw0wMjEyMDUwODAwMDBaFw0xMzAxMjUw
ODAwMDBaMGkxEzARBgoJkiaJk/IsZAEZFgNvcmcxGDAWBgoJkiaJk/IsZAEZFghET0VHcmlkczEg
MB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9yaXRpZXMxFjAUBgNVBAMTDURPRUdyaWRzIENBIDEw
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC09dYjYaPbCD5mtbiQb7Ka3y1qAm0ZcqKC
FciWcfe8Kwcuy9tjHuIsLf9ZItdkDW4xy8sua9nJlx3KlwjtumTMtOtg35KZCknUd8KM4VGTSFdL
VG9AbNayef76caVCGM1+jyF0Lq03kauGOPTcNfZe1TZa3e1c9rc8ljV5OSWa/mfsCACyS5zFIWu0
yIDNyJdf+n0hwaPN53wllpJ30taD+JBjQ7h2k4xRWzeaznLOb9OztZVRA/1sVze+iczFh2xwa4Vd
Gy0eIIPw1pfvYwxO36rm0S109qvbsNlaroPRbxerPKakQLpKe034Xcx7gBPqUk/FxoRRWin5EWN3
rz9LAgMBAAGjgZ4wgZswDgYDVR0PAQH/BAQDAgGGMBEGCWCGSAGG+EIBAQQEAwIAhzAdBgNVHQ4E
FgQUyhkdEo5upDhdQtQxDgjb2Y0XDV0wHwYDVR0jBBgwFoAUvF1NSC/4NZRZq1yJSz7RsjoUAeow
DwYDVR0TAQH/BAUwAwEB/zAlBgNVHREEHjAcgRpET0VHcmlkcy1DQS0xQGRvZWdyaWRzLm9yZzAN
BgkqhkiG9w0BAQUFAAOCAQEAZNVrIDLqe39CEOiJt7Q7EpBPhAihMvDTSf/42u0SMbUmChww4mLm
ph5DBghZUVF8Yn59kRZMn1QLOtO1HzLqvAvPITacZVPlJgG2IXzlR636YghZFAycbIUEOJDBHR4v
tQO1KDxgZwvAbtmKIoxvhUCq2xsfFt9kCBBn+JYtQ6O5LsBJq3PmuubeMcc7mbQAfJZ7h/3Qghgk
FIhmE1+LBXPJbkuP8vgfg6h2BKoAf5TFfZECgGZKimfN110tBvfedGZwYYd3/GsJc83B0JN1gny0
gqNVPm392UchXGeBRrHnm2gkhIkr48Oq6EmNGV9/a6XfbplQW/JWbtPVPWkaizCCBCgwggMQoAMC
AQICAwCvNzANBgkqhkiG9w0BAQUFADBpMRMwEQYKCZImiZPyLGQBGRYDb3JnMRgwFgYKCZImiZPy
LGQBGRYIRE9FR3JpZHMxIDAeBgNVBAsTF0NlcnRpZmljYXRlIEF1dGhvcml0aWVzMRYwFAYDVQQD
Ew1ET0VHcmlkcyBDQSAxMB4XDTEwMDUyNDE4MTg0OVoXDTExMDUyNDE4MTg0OVowYTETMBEGCgmS
JomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCGRvZWdyaWRzMQ8wDQYDVQQLEwZQZW9wbGUx
HzAdBgNVBAMTFkJyaWFuIEJvY2tlbG1hbiA1MDQzMDcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw
ggEKAoIBAQCwiNKRMFRMWG+AJyo/G5bYisPndrH+44JqRHdxDzMaQP59gxrBO2koRwg/13gINe3e
J8QL6bX4ANz/y1uglsytmoJwK5J9fxNYgJgbja83kO0j0HNL6oIyBhGStluaNTbiHrk8GA95M9VR
XRqpYiwKvT1F0KS2r7sZ+PWevbAek787eTqg51yuvUUlIBPgTm1kV3vZs21oeIZUuw7wPGXBKN49
XqIDsamUIqiFARwPgqKR9eo6itlYy2NrHo0hHLXew37rEOcKv/0g4pI4J/y4+1qB7fN3nMkIMack
FWfAQTngcnH/JpKmh8fmXdkeVv8EKYUXIgkUI5pb+Ak105olAgMBAAGjgeAwgd0wEQYJYIZIAYb4
QgEBBAQDAgWgMA4GA1UdDwEB/wQEAwIF4DA2BgNVHSAELzAtMA0GCyqGSIb3TAMHAQMBMAwGCiqG
SIb3TAUCAgEwDgYMKoZIhvdMBQIDAgEBMD4GA1UdHwQ3MDUwM6AxoC+GLWh0dHA6Ly9jcmwuZG9l
Z3JpZHMub3JnLzFjM2YyY2E4LzFjM2YyY2E4LmNybDAfBgNVHREEGDAWgRRiYm9ja2VsbUBjc2Uu
dW5sLmVkdTAfBgNVHSMEGDAWgBTKGR0Sjm6kOF1C1DEOCNvZjRcNXTANBgkqhkiG9w0BAQUFAAOC
AQEAlQSD+8Cvb0GxWqD4xhXd8Sl5MJRr1uJxMeGoMA4RZAJuyvVlBUx8v5moqY0XHMfNI+FulyMx
wgOoNfvF3dluz3J4C/u5NvzfNqikLj++sL4XDaZoxSHLo9cJxVTcM15Gogct+kvIF1+msEsqLlNR
/lqUVE/o8ANdD6PVx/044f/Dzi6s+6jZmBz/vWPI77ymT1EHaAkaHDqoNIlItPQrAHdkJWY67v1z
s6mDrKaspF/2ThDdYax208o1oLFd8wY8kQUdTBlMmUAbchQnjOC9vH17w6meDc8VxD+pEL3vAiG2
JN2vzQ3IJCYTCTmagUyiLWHFEudH8Brn43NY0/HwKTGCAv0wggL5AgEBMHAwaTETMBEGCgmSJomT
8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0
ZSBBdXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3JpZHMgQ0EgMQIDAK83MAkGBSsOAwIaBQCgggFi
MBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTExMDIxMTE0MTUzN1ow
IwYJKoZIhvcNAQkEMRYEFIQCbO1lxenni1oV+L6IFZGUO5JbMH8GCSsGAQQBgjcQBDFyMHAwaTET
MBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERPRUdyaWRzMSAwHgYDVQQLExdD
ZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3JpZHMgQ0EgMQIDAK83MIGBBgsq
hkiG9w0BCRACCzFyoHAwaTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERP
RUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3Jp
ZHMgQ0EgMQIDAK83MA0GCSqGSIb3DQEBAQUABIIBAHzBjI2iu0t3lmw88IH4npqs8c9s0ddgXanp
VIIXyJXiwmuc2EOj3m5oc1zReraLgP9BTzOyGA7kQhaRP2K5pJ1zsZ9Pqtuc8XxvQ/+F8VXkCrIu
gOD/BlMY7xalgIhBKI1GPP/QA9Y2GyFYY/prko4tfdRSdNXreYPGoblOYkF4HEqZcMjFYKCp9vqf
R2s4ShIuaLuJwMqo/N1VVlhELM6f17SGAghNRdVXqOXEXkWtkKUGHZgQWwQuL5zfwxzG/pv5mHkL
ab6UJV6YmqbGDojjR2pqgdNo6wpPgxvOTYCZv2s3MRPCkjCgwgyhsunRpIw8nKYUMnzDJDqgvQGy
XjAAAAAAAAA=

--Apple-Mail-1285--848526120--

From general-return-3004-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 15:13:21 2011
Return-Path: <general-return-3004-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63500 invoked from network); 11 Feb 2011 15:13:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 15:13:21 -0000
Received: (qmail 40599 invoked by uid 500); 11 Feb 2011 15:13:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40155 invoked by uid 500); 11 Feb 2011 15:13:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40137 invoked by uid 99); 11 Feb 2011 15:13:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 15:13:13 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [128.149.139.106] (HELO mail.jpl.nasa.gov) (128.149.139.106)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 15:13:03 +0000
Received: from mail.jpl.nasa.gov (altvirehtstap02.jpl.nasa.gov [128.149.137.73])
	by smtp.jpl.nasa.gov (Switch-3.4.3/Switch-3.4.3) with ESMTP id p1BFCgeo015560
	(using TLSv1/SSLv3 with cipher RC4-MD5 (128 bits) verified FAIL)
	for <general@hadoop.apache.org>; Fri, 11 Feb 2011 07:12:43 -0800
Received: from ALTPHYEMBEVSP20.RES.AD.JPL ([128.149.137.82]) by
 ALTVIREHTSTAP02.RES.AD.JPL ([128.149.137.73]) with mapi; Fri, 11 Feb 2011
 07:12:43 -0800
From: "Mattmann, Chris A (388J)" <chris.a.mattmann@jpl.nasa.gov>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 11 Feb 2011 07:12:42 -0800
Subject: Re: [VOTE] Abandon mrunit MapReduce contrib
Thread-Topic: [VOTE] Abandon mrunit MapReduce contrib
Thread-Index: AcvJ/iFX00ZSNrPKTLiUKWpoe5V8rw==
Message-ID: <22EA923E-F75A-42AD-84D2-D75D25AC94FC@jpl.nasa.gov>
References: <77405974-6771-4604-926B-976240743F3C@mac.com>
 <AANLkTim80KJ3f14iM6Ltzy7UG613myhN-AiwQ3pCmVdi@mail.gmail.com>
 <AANLkTiknkrWK=GzGMVj4_Z6p5Jn8TQu1P5TkaFSgORJk@mail.gmail.com>
In-Reply-To: <AANLkTiknkrWK=GzGMVj4_Z6p5Jn8TQu1P5TkaFSgORJk@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Source-IP: altvirehtstap02.jpl.nasa.gov [128.149.137.73]
X-Source-Sender: chris.a.mattmann@jpl.nasa.gov
X-AUTH: Authorized

Hi All,

> Votes to remove code should be because the PMC doesn't think the code is
> worth maintaining any more. I don't think that applies in this case. Aaro=
n
> is a committer in Hadoop now and I'd strongly encourage you both to give
> your changes back to Apache instead of forking it into GitHub.

+1.

Cheers,
Chris

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


From general-return-3005-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 15:34:13 2011
Return-Path: <general-return-3005-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72414 invoked from network); 11 Feb 2011 15:34:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 15:34:13 -0000
Received: (qmail 86273 invoked by uid 500); 11 Feb 2011 15:34:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85831 invoked by uid 500); 11 Feb 2011 15:34:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85822 invoked by uid 99); 11 Feb 2011 15:34:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 15:34:06 +0000
X-ASF-Spam-Status: No, hits=-1997.8 required=5.0
	tests=ALL_TRUSTED,HTML_MESSAGE,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 11 Feb 2011 15:34:03 +0000
Received: (qmail 72252 invoked by uid 99); 11 Feb 2011 15:33:41 -0000
Received: from localhost.apache.org (HELO mail-bw0-f48.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 15:33:41 +0000
Received: by bwz8 with SMTP id 8so3491323bwz.35
        for <general@hadoop.apache.org>; Fri, 11 Feb 2011 07:33:39 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.113.75 with SMTP id z11mr598362bkp.90.1297438419218; Fri,
 11 Feb 2011 07:33:39 -0800 (PST)
Received: by 10.204.97.133 with HTTP; Fri, 11 Feb 2011 07:33:39 -0800 (PST)
In-Reply-To: <0437FB76-82C0-472D-94B5-1E01B8E94E77@mac.com>
References: <E43E21C9-4267-440D-A0C8-9CB21E9DBADA@mac.com>
	<AANLkTimMinEh_Ko7JS3VAknFJWFOgeEtbr95KUHBPo5e@mail.gmail.com>
	<0437FB76-82C0-472D-94B5-1E01B8E94E77@mac.com>
Date: Fri, 11 Feb 2011 07:33:39 -0800
Message-ID: <AANLkTinQ22K9467t2P4qXdt9fgQJb2TT+OxJjvn1yotS@mail.gmail.com>
Subject: Re: [VOTE] Abandon fuse-dfs HDFS contrib
From: "Owen O'Malley" <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d64473187103049c036c0f
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d64473187103049c036c0f
Content-Type: text/plain; charset=UTF-8

On Thu, Feb 10, 2011 at 9:57 PM, Nigel Daley <ndaley@mac.com> wrote:

If a reasonable replacement was developed elsewhere, would you change your
> vote?  I believe some folks were interested in developing this elsewhere.
>

It would depend. As I have said before, our goal as a PMC is to produce
useful Hadoop releases. Pulling out functionality that users require makes
the Hadoop releases less useful. Making users go to 20 different sites to
download compatible versions of the different components, although it helps
aggregating companies like RedHat, is an anti-goal.

That said, if projects are active enough splitting them out into other
Apache projects makes sense. Chukwa, HBase, Hive, Hive, Pig, and Zookeeper
are wonderful examples of related projects where we've moved them out of
Hadoop and their communities are healthier for the move. Moving actively
used components into non-Apache managed projects, including Apache Extras,
is an anti-goal.

-- Owen

--0016e6d64473187103049c036c0f--

From general-return-3006-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 16:03:07 2011
Return-Path: <general-return-3006-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 89181 invoked from network); 11 Feb 2011 16:03:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 16:03:07 -0000
Received: (qmail 52397 invoked by uid 500); 11 Feb 2011 16:03:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52030 invoked by uid 500); 11 Feb 2011 16:03:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52020 invoked by uid 99); 11 Feb 2011 16:03:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 16:03:03 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of esammer@cloudera.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 16:02:55 +0000
Received: by qwe4 with SMTP id 4so1681866qwe.35
        for <general@hadoop.apache.org>; Fri, 11 Feb 2011 08:02:34 -0800 (PST)
MIME-Version: 1.0
Received: by 10.220.187.69 with SMTP id cv5mr792709vcb.200.1297440154517; Fri,
 11 Feb 2011 08:02:34 -0800 (PST)
Received: by 10.220.210.70 with HTTP; Fri, 11 Feb 2011 08:02:34 -0800 (PST)
In-Reply-To: <AANLkTiknkrWK=GzGMVj4_Z6p5Jn8TQu1P5TkaFSgORJk@mail.gmail.com>
References: <77405974-6771-4604-926B-976240743F3C@mac.com>
	<AANLkTim80KJ3f14iM6Ltzy7UG613myhN-AiwQ3pCmVdi@mail.gmail.com>
	<AANLkTiknkrWK=GzGMVj4_Z6p5Jn8TQu1P5TkaFSgORJk@mail.gmail.com>
Date: Fri, 11 Feb 2011 11:02:34 -0500
Message-ID: <AANLkTin0TguLYHi9X0uL3ZF7tCz5xTHBYa7056oKF2Ae@mail.gmail.com>
Subject: Re: [VOTE] Abandon mrunit MapReduce contrib
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba53a538870317049c03d3bd
X-Virus-Checked: Checked by ClamAV on apache.org

--90e6ba53a538870317049c03d3bd
Content-Type: text/plain; charset=ISO-8859-1

Owen:

I think you make a fair point. The reason I think it still makes sense to
bring mrunit out of Hadoop contrib is to:

- start to simplify the build by breaking projects that are only clients of
Hadoop libs out of contrib.
- allow mrunit to have its own release cycle. This is, I think, the most
important.

I would actually prefer to move it to Extras or Incubator and leave this
within the ASF. Right now, I picked github because of the ability to easily
collaborate with others (and to use git).

Thanks!

On Fri, Feb 11, 2011 at 2:52 AM, Owen O'Malley <omalley@apache.org> wrote:

> On Thu, Feb 10, 2011 at 11:36 PM, Aaron Kimball <akimball83@gmail.com
> >wrote:
>
> > +1. Eric Sammer and I will be working on this via github. (Come join us!)
>
>
> Votes to remove code should be because the PMC doesn't think the code is
> worth maintaining any more. I don't think that applies in this case. Aaron
> is a committer in Hadoop now and I'd strongly encourage you both to give
> your changes back to Apache instead of forking it into GitHub.
>
> -- Owen
>



-- 
Eric Sammer
twitter: esammer
data: www.cloudera.com

--90e6ba53a538870317049c03d3bd--

From general-return-3007-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 16:48:50 2011
Return-Path: <general-return-3007-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21833 invoked from network); 11 Feb 2011 16:48:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 16:48:50 -0000
Received: (qmail 42624 invoked by uid 500); 11 Feb 2011 16:48:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42334 invoked by uid 500); 11 Feb 2011 16:48:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42321 invoked by uid 99); 11 Feb 2011 16:48:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 16:48:45 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.30.24] (HELO qmta02.emeryville.ca.mail.comcast.net) (76.96.30.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 16:48:35 +0000
Received: from omta06.emeryville.ca.mail.comcast.net ([76.96.30.51])
	by qmta02.emeryville.ca.mail.comcast.net with comcast
	id 6gSw1g00A16AWCUA2goDlQ; Fri, 11 Feb 2011 16:48:13 +0000
Received: from [10.72.184.42] ([209.131.62.113])
	by omta06.emeryville.ca.mail.comcast.net with comcast
	id 6go41g01T2SbwD58Sgo7bo; Fri, 11 Feb 2011 16:48:11 +0000
Message-Id: <93644D3B-4830-4813-92BE-809738969CF9@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <AANLkTin0TguLYHi9X0uL3ZF7tCz5xTHBYa7056oKF2Ae@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Abandon mrunit MapReduce contrib
Date: Fri, 11 Feb 2011 08:48:03 -0800
References: <77405974-6771-4604-926B-976240743F3C@mac.com> <AANLkTim80KJ3f14iM6Ltzy7UG613myhN-AiwQ3pCmVdi@mail.gmail.com> <AANLkTiknkrWK=GzGMVj4_Z6p5Jn8TQu1P5TkaFSgORJk@mail.gmail.com> <AANLkTin0TguLYHi9X0uL3ZF7tCz5xTHBYa7056oKF2Ae@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

On Feb 11, 2011, at 8:02 AM, Eric Sammer wrote:

> - allow mrunit to have its own release cycle. This is, I think, the  
> most

> important.

If you submit your work to Apache we can evaluate it for inclusion in  
the 0.20.100 branch to get your changes released in a timely manner.

> I would actually prefer to move it to Extras or Incubator and leave  
> this
> within the ASF.

Extras is **NOT** inside of the ASF. Extras is a source hosting system  
for non-Apache projects that are related to Apache projects.

> Right now, I picked github because of the ability to easily
> collaborate with others (and to use git).

I agree that it is unfortunate that Apache doesn't yet support read- 
write git access. However, you'll find that building a community is  
easier at Apache than at github.

-- Owen

From general-return-3008-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 16:51:35 2011
Return-Path: <general-return-3008-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23064 invoked from network); 11 Feb 2011 16:51:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 16:51:34 -0000
Received: (qmail 47101 invoked by uid 500); 11 Feb 2011 16:51:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46852 invoked by uid 500); 11 Feb 2011 16:51:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46840 invoked by uid 99); 11 Feb 2011 16:51:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 16:51:30 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.104 as permitted sender)
Received: from [17.148.16.104] (HELO asmtpout029.mac.com) (17.148.16.104)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 16:51:22 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp029.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LGG00KPIPGGJI60@asmtp029.mac.com>
 for general@hadoop.apache.org; Fri, 11 Feb 2011 08:50:42 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-02-11_06:2011-02-11,2011-02-11,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1102110074
Subject: Re: [VOTE] Abandon fuse-dfs HDFS contrib
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <AANLkTinQ22K9467t2P4qXdt9fgQJb2TT+OxJjvn1yotS@mail.gmail.com>
Date: Fri, 11 Feb 2011 08:50:39 -0800
Message-id: <94C00012-C67B-437F-A905-012934F6B498@mac.com>
References: <E43E21C9-4267-440D-A0C8-9CB21E9DBADA@mac.com>
 <AANLkTimMinEh_Ko7JS3VAknFJWFOgeEtbr95KUHBPo5e@mail.gmail.com>
 <0437FB76-82C0-472D-94B5-1E01B8E94E77@mac.com>
 <AANLkTinQ22K9467t2P4qXdt9fgQJb2TT+OxJjvn1yotS@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)


On Feb 11, 2011, at 7:33 AM, Owen O'Malley wrote:

> On Thu, Feb 10, 2011 at 9:57 PM, Nigel Daley <ndaley@mac.com> wrote:
> 
> If a reasonable replacement was developed elsewhere, would you change your
>> vote?  I believe some folks were interested in developing this elsewhere.
>> 
> 
> It would depend. As I have said before, our goal as a PMC is to produce
> useful Hadoop releases. Pulling out functionality that users require makes
> the Hadoop releases less useful. Making users go to 20 different sites to
> download compatible versions of the different components, although it helps
> aggregating companies like RedHat, is an anti-goal.

I wasn't aware that RedHat was producing an aggregated release.

If these components are required, why are they in contrib?  Feel free to start a vote to move this component to core or incubator.

Why is this an anti-goal of Hadoop?  Did you mean non-goal?  Hadoop is NOT the Big Data Stack.  We're not responsible for integrating Pig, HBase, Hive, ZK, and all these contribs into one package.  If you'd like to propose a Big Data Stack integration project to the incubator I suspect you'll have a lot of support from folks in this community. That could be a really compelling project.  Until that happens, there's obvious value for users to use an aggregated, integrated Big Data Stack type package from companies like Cloudera.

> That said, if projects are active enough splitting them out into other
> Apache projects makes sense. Chukwa, HBase, Hive, Hive, Pig, and Zookeeper
> are wonderful examples of related projects where we've moved them out of
> Hadoop and their communities are healthier for the move. Moving actively
> used components into non-Apache managed projects, including Apache Extras,
> is an anti-goal.

We seem to have trouble creating releases of core Hadoop.  Why saddle ourselves with these ancillary components that are not core?  Sure, I'd love to see some of these move to incubator -- but I'm not signing up to do that.

Cheers,
Nige

From general-return-3009-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 16:59:36 2011
Return-Path: <general-return-3009-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 26102 invoked from network); 11 Feb 2011 16:59:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 16:59:35 -0000
Received: (qmail 59309 invoked by uid 500); 11 Feb 2011 16:59:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59000 invoked by uid 500); 11 Feb 2011 16:59:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58986 invoked by uid 99); 11 Feb 2011 16:59:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 16:59:30 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mcsrivas@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 16:59:22 +0000
Received: by vws18 with SMTP id 18so1702016vws.35
        for <general@hadoop.apache.org>; Fri, 11 Feb 2011 08:59:01 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=OU0vcfBngX3cBAXMPqLnlb1bVrdvzTna6sk70JvCYrs=;
        b=ws0v7kSvsiiEC+5Fy+9vddOUFuGK0esQqVC/jaoqcB391K8Rn5G+9U8tRa6iXA0G3g
         nn/WXdNJwHOtbkPq7pFxsA6ECt+Lu58qfwkM1NhwFtUzH9+L0nTrD2dG8i0K35EHFJa0
         CBAGBOVolbIL3x8YgXowSuffh22dWuMHFjDwo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Co32kkYAcmi0hIPtXt0HLbiMxuujnCFYGGMCQWMLas/0X6LnfXfJMzxUh8ApdKqHRF
         2NON73NIF/8aOIQzNmL1sBhWnApdFC7hqxDn4L52Ab1VPcRFPqlxBh61gA2+j/8kaHPB
         o+Lt1PBePRGII6EeKKLKGUzjiGhB0/ZTNHEeU=
MIME-Version: 1.0
Received: by 10.220.186.3 with SMTP id cq3mr903996vcb.20.1297443540316; Fri,
 11 Feb 2011 08:59:00 -0800 (PST)
Received: by 10.220.193.1 with HTTP; Fri, 11 Feb 2011 08:59:00 -0800 (PST)
In-Reply-To: <0437FB76-82C0-472D-94B5-1E01B8E94E77@mac.com>
References: <E43E21C9-4267-440D-A0C8-9CB21E9DBADA@mac.com>
	<AANLkTimMinEh_Ko7JS3VAknFJWFOgeEtbr95KUHBPo5e@mail.gmail.com>
	<0437FB76-82C0-472D-94B5-1E01B8E94E77@mac.com>
Date: Fri, 11 Feb 2011 08:59:00 -0800
Message-ID: <AANLkTimSHBQrWsXyq-7+fLmzE4jUtRR-U4sD5V-g88yt@mail.gmail.com>
Subject: Re: [VOTE] Abandon fuse-dfs HDFS contrib
From: "M. C. Srivas" <mcsrivas@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba53aa5e56321a049c049d18
X-Virus-Checked: Checked by ClamAV on apache.org

--90e6ba53aa5e56321a049c049d18
Content-Type: text/plain; charset=ISO-8859-1

On Thu, Feb 10, 2011 at 9:57 PM, Nigel Daley <ndaley@mac.com> wrote:

>
> On Feb 10, 2011, at 9:22 PM, Owen O'Malley wrote:
>
> > On Thu, Feb 10, 2011 at 6:58 PM, Nigel Daley <ndaley@mac.com> wrote:
> >
> >> I think the PMC should abandon the fuse-dfs HDFS contrib component.
>  It's
> >> last meaningful contribution was March 2010:
> >>
> >
> > I think we should hold on to this one until there is a reasonable
> > replacement. -1
>
> If a reasonable replacement was developed elsewhere, would you change your
> vote?  I believe some folks were interested in developing this elsewhere.
>

Looks like lot of folks are using the current one, so why not wait until new
one is submitted (and it works)?

Just because there are bugs against fuse-dfs that haven't been fixed in a
while doesn't make it useless. There are bugs open against Hadoop that
haven't been worked on for a long time either.










>
> Nige
>
>

--90e6ba53aa5e56321a049c049d18--

From general-return-3010-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 17:05:27 2011
Return-Path: <general-return-3010-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34984 invoked from network); 11 Feb 2011 17:05:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 17:05:26 -0000
Received: (qmail 69213 invoked by uid 500); 11 Feb 2011 17:05:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68916 invoked by uid 500); 11 Feb 2011 17:05:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68904 invoked by uid 99); 11 Feb 2011 17:05:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 17:05:22 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of esammer@cloudera.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 17:05:16 +0000
Received: by vws18 with SMTP id 18so1707088vws.35
        for <general@hadoop.apache.org>; Fri, 11 Feb 2011 09:04:55 -0800 (PST)
MIME-Version: 1.0
Received: by 10.220.184.2 with SMTP id ci2mr888553vcb.162.1297443888558; Fri,
 11 Feb 2011 09:04:48 -0800 (PST)
Received: by 10.220.5.80 with HTTP; Fri, 11 Feb 2011 09:04:48 -0800 (PST)
In-Reply-To: <93644D3B-4830-4813-92BE-809738969CF9@apache.org>
References: <77405974-6771-4604-926B-976240743F3C@mac.com>
	<AANLkTim80KJ3f14iM6Ltzy7UG613myhN-AiwQ3pCmVdi@mail.gmail.com>
	<AANLkTiknkrWK=GzGMVj4_Z6p5Jn8TQu1P5TkaFSgORJk@mail.gmail.com>
	<AANLkTin0TguLYHi9X0uL3ZF7tCz5xTHBYa7056oKF2Ae@mail.gmail.com>
	<93644D3B-4830-4813-92BE-809738969CF9@apache.org>
Date: Fri, 11 Feb 2011 12:04:48 -0500
Message-ID: <AANLkTi=dbGSLhtFA+Rf9LgaX05v45HDLoQDE_2C5ULAf@mail.gmail.com>
Subject: Re: [VOTE] Abandon mrunit MapReduce contrib
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba4fbdf4188c20049c04b24d
X-Virus-Checked: Checked by ClamAV on apache.org

--90e6ba4fbdf4188c20049c04b24d
Content-Type: text/plain; charset=ISO-8859-1

On Fri, Feb 11, 2011 at 11:48 AM, Owen O'Malley <omalley@apache.org> wrote:

> On Feb 11, 2011, at 8:02 AM, Eric Sammer wrote:
>
>  - allow mrunit to have its own release cycle. This is, I think, the most
>>
>
>  important.
>>
>
> If you submit your work to Apache we can evaluate it for inclusion in the
> 0.20.100 branch to get your changes released in a timely manner.


I'm thinking in general (beyond the next immediate release). Independent of
where mrunit goes, I think it should leave the contrib tree to facilitate
light weight releases (the dependency on Hadoop proper is a public facing
API - a pure client). I think most projects could benefit from this with the
exception of things that are tightly coupled to Hadoop releases or touch
non-public APIs.


>  I would actually prefer to move it to Extras or Incubator and leave this
>> within the ASF.
>>
>
> Extras is **NOT** inside of the ASF. Extras is a source hosting system for
> non-Apache projects that are related to Apache projects.


Got it. Thanks for correcting me. I only mentioned it because someone
suggested it to me initially.


> Right now, I picked github because of the ability to easily
> collaborate with others (and to use git).
>

I agree that it is unfortunate that Apache doesn't yet support read-write
> git access. However, you'll find that building a community is easier at
> Apache than at github.
>

> -- Owen
>



-- 
Eric Sammer
twitter: esammer
data: www.cloudera.com

--90e6ba4fbdf4188c20049c04b24d--

From general-return-3011-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 17:38:46 2011
Return-Path: <general-return-3011-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50470 invoked from network); 11 Feb 2011 17:38:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 17:38:46 -0000
Received: (qmail 17710 invoked by uid 500); 11 Feb 2011 17:38:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17433 invoked by uid 500); 11 Feb 2011 17:38:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17425 invoked by uid 99); 11 Feb 2011 17:38:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 17:38:41 +0000
X-ASF-Spam-Status: No, hits=-0.1 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [129.93.165.11] (HELO cse-mail.unl.edu) (129.93.165.11)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 17:38:33 +0000
Received: from cse-barracuda.cse.unl.edu (cse-barracuda.unl.edu [129.93.164.185])
	(authenticated bits=0)
	by cse-mail.unl.edu (8.14.3/8.14.3) with ESMTP id p1BHc7r0007032
	for <general@hadoop.apache.org>; Fri, 11 Feb 2011 11:38:12 -0600 (CST)
X-ASG-Debug-ID: 1297445886-4b5c0dbe46da-IjwG86
Received: from cse.unl.edu (cse.unl.edu [129.93.165.2]) by cse-barracuda.cse.unl.edu with ESMTP id NmvTwLabBhGZy1Qa for <general@hadoop.apache.org>; Fri, 11 Feb 2011 11:38:06 -0600 (CST)
X-Barracuda-Envelope-From: bbockelm@cse.unl.edu
X-Barracuda-RBL-Trusted-Forwarder: 129.93.165.2
Received: from pcp088890pcs.unl.edu (pcp088890pcs.unl.edu [129.93.158.5])
	(authenticated bits=0)
	by cse.unl.edu (8.14.4/8.14.3) with ESMTP id p1BHc6SX008418
	(version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO)
	for <general@hadoop.apache.org>; Fri, 11 Feb 2011 11:38:06 -0600
From: Brian Bockelman <bbockelm@cse.unl.edu>
X-Barracuda-Apparent-Source-IP: 129.93.158.5
Mime-Version: 1.0 (Apple Message framework v1082)
Content-Type: multipart/signed; boundary=Apple-Mail-1289--836377044; protocol="application/pkcs7-signature"; micalg=sha1
Subject: Re: [VOTE] Abandon fuse-dfs HDFS contrib
Date: Fri, 11 Feb 2011 11:38:06 -0600
X-ASG-Orig-Subj: Re: [VOTE] Abandon fuse-dfs HDFS contrib
In-Reply-To: <AANLkTimSHBQrWsXyq-7+fLmzE4jUtRR-U4sD5V-g88yt@mail.gmail.com>
To: general@hadoop.apache.org
References: <E43E21C9-4267-440D-A0C8-9CB21E9DBADA@mac.com> <AANLkTimMinEh_Ko7JS3VAknFJWFOgeEtbr95KUHBPo5e@mail.gmail.com> <0437FB76-82C0-472D-94B5-1E01B8E94E77@mac.com> <AANLkTimSHBQrWsXyq-7+fLmzE4jUtRR-U4sD5V-g88yt@mail.gmail.com>
Message-Id: <9A9B8BB2-DBB6-4715-9234-496E90D44EF9@cse.unl.edu>
X-Mailer: Apple Mail (2.1082)
X-Barracuda-Connect: cse.unl.edu[129.93.165.2]
X-Barracuda-Start-Time: 1297445886
X-Barracuda-URL: http://cse-barracuda.unl.edu:8000/cgi-mod/mark.cgi
X-Virus-Scanned: clamav-milter 0.97 at cse-mail
X-Barracuda-Spam-Score: 0.00
X-Barracuda-Spam-Status: No, SCORE=0.00 using global scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=9.0 tests=HTML_MESSAGE
X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.2.55042
	Rule breakdown below
	 pts rule name              description
	---- ---------------------- --------------------------------------------------
	0.00 HTML_MESSAGE           BODY: HTML included in message
X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0.1 (cse-mail.unl.edu [129.93.165.11]); Fri, 11 Feb 2011 11:38:12 -0600 (CST)
X-Virus-Status: Clean

--Apple-Mail-1289--836377044
Content-Type: multipart/alternative;
	boundary=Apple-Mail-1288--836377101


--Apple-Mail-1288--836377101
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii


On Feb 11, 2011, at 10:59 AM, M. C. Srivas wrote:

> On Thu, Feb 10, 2011 at 9:57 PM, Nigel Daley <ndaley@mac.com> wrote:
>=20
>>=20
>> On Feb 10, 2011, at 9:22 PM, Owen O'Malley wrote:
>>=20
>>> On Thu, Feb 10, 2011 at 6:58 PM, Nigel Daley <ndaley@mac.com> wrote:
>>>=20
>>>> I think the PMC should abandon the fuse-dfs HDFS contrib component.
>> It's
>>>> last meaningful contribution was March 2010:
>>>>=20
>>>=20
>>> I think we should hold on to this one until there is a reasonable
>>> replacement. -1
>>=20
>> If a reasonable replacement was developed elsewhere, would you change =
your
>> vote?  I believe some folks were interested in developing this =
elsewhere.
>>=20
>=20
> Looks like lot of folks are using the current one, so why not wait =
until new
> one is submitted (and it works)?
>=20
> Just because there are bugs against fuse-dfs that haven't been fixed =
in a
> while doesn't make it useless. There are bugs open against Hadoop that
> haven't been worked on for a long time either.
>=20

Indeed - and I would volunteer to clean house if given the time.

I've been browsing through the open issues.  A lot could be knocked out =
easily; most are improvement requests.

The contrib is used by folks in the community; it works well for them.  =
Typically, it only will need to be updated when the libhdfs header =
changes or when bugs are found.  It can use some TLC, not removal.

Brian


--Apple-Mail-1288--836377101
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=us-ascii

<html><head></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space; =
"><br><div><div>On Feb 11, 2011, at 10:59 AM, M. C. Srivas =
wrote:</div><br class=3D"Apple-interchange-newline"><blockquote =
type=3D"cite"><div>On Thu, Feb 10, 2011 at 9:57 PM, Nigel Daley &lt;<a =
href=3D"mailto:ndaley@mac.com">ndaley@mac.com</a>&gt; =
wrote:<br><br><blockquote type=3D"cite"><br></blockquote><blockquote =
type=3D"cite">On Feb 10, 2011, at 9:22 PM, Owen O'Malley =
wrote:<br></blockquote><blockquote =
type=3D"cite"><br></blockquote><blockquote type=3D"cite"><blockquote =
type=3D"cite">On Thu, Feb 10, 2011 at 6:58 PM, Nigel Daley &lt;<a =
href=3D"mailto:ndaley@mac.com">ndaley@mac.com</a>&gt; =
wrote:<br></blockquote></blockquote><blockquote type=3D"cite"><blockquote =
type=3D"cite"><br></blockquote></blockquote><blockquote =
type=3D"cite"><blockquote type=3D"cite"><blockquote type=3D"cite">I =
think the PMC should abandon the fuse-dfs HDFS contrib =
component.<br></blockquote></blockquote></blockquote><blockquote =
type=3D"cite"> It's<br></blockquote><blockquote type=3D"cite"><blockquote =
type=3D"cite"><blockquote type=3D"cite">last meaningful contribution was =
March 2010:<br></blockquote></blockquote></blockquote><blockquote =
type=3D"cite"><blockquote type=3D"cite"><blockquote =
type=3D"cite"><br></blockquote></blockquote></blockquote><blockquote =
type=3D"cite"><blockquote =
type=3D"cite"><br></blockquote></blockquote><blockquote =
type=3D"cite"><blockquote type=3D"cite">I think we should hold on to =
this one until there is a =
reasonable<br></blockquote></blockquote><blockquote =
type=3D"cite"><blockquote type=3D"cite">replacement. =
-1<br></blockquote></blockquote><blockquote =
type=3D"cite"><br></blockquote><blockquote type=3D"cite">If a reasonable =
replacement was developed elsewhere, would you change =
your<br></blockquote><blockquote type=3D"cite">vote? &nbsp;I believe =
some folks were interested in developing this =
elsewhere.<br></blockquote><blockquote =
type=3D"cite"><br></blockquote><br>Looks like lot of folks are using the =
current one, so why not wait until new<br>one is submitted (and it =
works)?<br><br>Just because there are bugs against fuse-dfs that haven't =
been fixed in a<br>while doesn't make it useless. There are bugs open =
against Hadoop that<br>haven't been worked on for a long time =
either.<br><font class=3D"Apple-style-span" color=3D"#000000"><font =
class=3D"Apple-style-span" =
color=3D"#144FAE"><br></font></font></div></blockquote><br></div><div>Inde=
ed - and I would volunteer to clean house if given the =
time.</div><div><br></div><div>I've been browsing through the open =
issues. &nbsp;A lot could be knocked out easily; most are improvement =
requests.</div><div><br></div><div>The contrib is used by folks in the =
community; it works well for them. &nbsp;Typically, it only will need to =
be updated when the libhdfs header changes or when bugs are found. =
&nbsp;It can use some TLC, not =
removal.</div><div><br></div><div>Brian</div><div><br></div></body></html>=

--Apple-Mail-1288--836377101--

--Apple-Mail-1289--836377044
Content-Disposition: attachment;
	filename=smime.p7s
Content-Type: application/pkcs7-signature;
	name=smime.p7s
Content-Transfer-Encoding: base64

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIIKDCCA/gw
ggLgoAMCAQICASkwDQYJKoZIhvcNAQEFBQAwdTETMBEGCgmSJomT8ixkARkWA25ldDESMBAGCgmS
JomT8ixkARkWAkVTMQ4wDAYDVQQKEwVFU25ldDEgMB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9y
aXRpZXMxGDAWBgNVBAMTD0VTbmV0IFJvb3QgQ0EgMTAeFw0wMjEyMDUwODAwMDBaFw0xMzAxMjUw
ODAwMDBaMGkxEzARBgoJkiaJk/IsZAEZFgNvcmcxGDAWBgoJkiaJk/IsZAEZFghET0VHcmlkczEg
MB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9yaXRpZXMxFjAUBgNVBAMTDURPRUdyaWRzIENBIDEw
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC09dYjYaPbCD5mtbiQb7Ka3y1qAm0ZcqKC
FciWcfe8Kwcuy9tjHuIsLf9ZItdkDW4xy8sua9nJlx3KlwjtumTMtOtg35KZCknUd8KM4VGTSFdL
VG9AbNayef76caVCGM1+jyF0Lq03kauGOPTcNfZe1TZa3e1c9rc8ljV5OSWa/mfsCACyS5zFIWu0
yIDNyJdf+n0hwaPN53wllpJ30taD+JBjQ7h2k4xRWzeaznLOb9OztZVRA/1sVze+iczFh2xwa4Vd
Gy0eIIPw1pfvYwxO36rm0S109qvbsNlaroPRbxerPKakQLpKe034Xcx7gBPqUk/FxoRRWin5EWN3
rz9LAgMBAAGjgZ4wgZswDgYDVR0PAQH/BAQDAgGGMBEGCWCGSAGG+EIBAQQEAwIAhzAdBgNVHQ4E
FgQUyhkdEo5upDhdQtQxDgjb2Y0XDV0wHwYDVR0jBBgwFoAUvF1NSC/4NZRZq1yJSz7RsjoUAeow
DwYDVR0TAQH/BAUwAwEB/zAlBgNVHREEHjAcgRpET0VHcmlkcy1DQS0xQGRvZWdyaWRzLm9yZzAN
BgkqhkiG9w0BAQUFAAOCAQEAZNVrIDLqe39CEOiJt7Q7EpBPhAihMvDTSf/42u0SMbUmChww4mLm
ph5DBghZUVF8Yn59kRZMn1QLOtO1HzLqvAvPITacZVPlJgG2IXzlR636YghZFAycbIUEOJDBHR4v
tQO1KDxgZwvAbtmKIoxvhUCq2xsfFt9kCBBn+JYtQ6O5LsBJq3PmuubeMcc7mbQAfJZ7h/3Qghgk
FIhmE1+LBXPJbkuP8vgfg6h2BKoAf5TFfZECgGZKimfN110tBvfedGZwYYd3/GsJc83B0JN1gny0
gqNVPm392UchXGeBRrHnm2gkhIkr48Oq6EmNGV9/a6XfbplQW/JWbtPVPWkaizCCBCgwggMQoAMC
AQICAwCvNzANBgkqhkiG9w0BAQUFADBpMRMwEQYKCZImiZPyLGQBGRYDb3JnMRgwFgYKCZImiZPy
LGQBGRYIRE9FR3JpZHMxIDAeBgNVBAsTF0NlcnRpZmljYXRlIEF1dGhvcml0aWVzMRYwFAYDVQQD
Ew1ET0VHcmlkcyBDQSAxMB4XDTEwMDUyNDE4MTg0OVoXDTExMDUyNDE4MTg0OVowYTETMBEGCgmS
JomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCGRvZWdyaWRzMQ8wDQYDVQQLEwZQZW9wbGUx
HzAdBgNVBAMTFkJyaWFuIEJvY2tlbG1hbiA1MDQzMDcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw
ggEKAoIBAQCwiNKRMFRMWG+AJyo/G5bYisPndrH+44JqRHdxDzMaQP59gxrBO2koRwg/13gINe3e
J8QL6bX4ANz/y1uglsytmoJwK5J9fxNYgJgbja83kO0j0HNL6oIyBhGStluaNTbiHrk8GA95M9VR
XRqpYiwKvT1F0KS2r7sZ+PWevbAek787eTqg51yuvUUlIBPgTm1kV3vZs21oeIZUuw7wPGXBKN49
XqIDsamUIqiFARwPgqKR9eo6itlYy2NrHo0hHLXew37rEOcKv/0g4pI4J/y4+1qB7fN3nMkIMack
FWfAQTngcnH/JpKmh8fmXdkeVv8EKYUXIgkUI5pb+Ak105olAgMBAAGjgeAwgd0wEQYJYIZIAYb4
QgEBBAQDAgWgMA4GA1UdDwEB/wQEAwIF4DA2BgNVHSAELzAtMA0GCyqGSIb3TAMHAQMBMAwGCiqG
SIb3TAUCAgEwDgYMKoZIhvdMBQIDAgEBMD4GA1UdHwQ3MDUwM6AxoC+GLWh0dHA6Ly9jcmwuZG9l
Z3JpZHMub3JnLzFjM2YyY2E4LzFjM2YyY2E4LmNybDAfBgNVHREEGDAWgRRiYm9ja2VsbUBjc2Uu
dW5sLmVkdTAfBgNVHSMEGDAWgBTKGR0Sjm6kOF1C1DEOCNvZjRcNXTANBgkqhkiG9w0BAQUFAAOC
AQEAlQSD+8Cvb0GxWqD4xhXd8Sl5MJRr1uJxMeGoMA4RZAJuyvVlBUx8v5moqY0XHMfNI+FulyMx
wgOoNfvF3dluz3J4C/u5NvzfNqikLj++sL4XDaZoxSHLo9cJxVTcM15Gogct+kvIF1+msEsqLlNR
/lqUVE/o8ANdD6PVx/044f/Dzi6s+6jZmBz/vWPI77ymT1EHaAkaHDqoNIlItPQrAHdkJWY67v1z
s6mDrKaspF/2ThDdYax208o1oLFd8wY8kQUdTBlMmUAbchQnjOC9vH17w6meDc8VxD+pEL3vAiG2
JN2vzQ3IJCYTCTmagUyiLWHFEudH8Brn43NY0/HwKTGCAv0wggL5AgEBMHAwaTETMBEGCgmSJomT
8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0
ZSBBdXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3JpZHMgQ0EgMQIDAK83MAkGBSsOAwIaBQCgggFi
MBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTExMDIxMTE3MzgwNlow
IwYJKoZIhvcNAQkEMRYEFEvT1driIBwO9w0Z+eZx/7lEXOncMH8GCSsGAQQBgjcQBDFyMHAwaTET
MBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERPRUdyaWRzMSAwHgYDVQQLExdD
ZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3JpZHMgQ0EgMQIDAK83MIGBBgsq
hkiG9w0BCRACCzFyoHAwaTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERP
RUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3Jp
ZHMgQ0EgMQIDAK83MA0GCSqGSIb3DQEBAQUABIIBAEZo+PRdAoXTrtL4osRwXKqxP63XlyL1X1/R
mA3yPwZQxybdAMX97gNnld937NqEsFSCxdAqJEM3BxxYx6oaRSumZzh/prdOzccixKoDg/KC2iQj
Qze35HW2B2nkAZJRmI5rl9VlHjTGWOuWCmqp/Y4lt6ol6R/50Lb80x5ooX9H9zmNdyCe5ZYOD8t5
zKpH9zdgutjAINrOVK1/c1U9r+mugJwzqIuh0b8AqtZkzmXFvIZk1fRm8Buy/4+tTjCNCwXM1adR
VvrKzKNNzjMW9B2jizf1/gGB5GrTAbaoslWUaIA5xedykMLR1A4pl2cPvpCblYZ1Z/1NEwbuLvce
jxAAAAAAAAA=

--Apple-Mail-1289--836377044--

From general-return-3012-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 17:41:01 2011
Return-Path: <general-return-3012-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 51123 invoked from network); 11 Feb 2011 17:41:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 17:41:01 -0000
Received: (qmail 22841 invoked by uid 500); 11 Feb 2011 17:40:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22748 invoked by uid 500); 11 Feb 2011 17:40:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22740 invoked by uid 99); 11 Feb 2011 17:40:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 17:40:57 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 17:40:51 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=maTPG8evfkq2UyyhuGsTx9RJxXD7VAWCtngCMJF/m8AGFhdcuDbnLBOm
   yGqJUMXtq4mIy+gcUIcOf1VDHqMHfU0SBNWmZbOnKK6SjQHRChQzJBNDA
   uTNO8mG+Q9GWSZQ;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1297446048; x=1328982048;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20Hadoop=200.22=20Blockers|Date:=20Fri,
   =2011=20Feb=202011=2017:41:20=20+0000|Message-ID:=20<3D9E
   A200-342B-4B1F-A2E8-6FC159BFDE38@linkedin.com>|To:=20"<ge
   neral@hadoop.apache.org>"=20<general@hadoop.apache.org>
   |MIME-Version:=201.0|Content-Transfer-Encoding:=20quoted-
   printable|Content-ID:=20<65133C54BF6D854D9057A3E553F5C4C4
   @linkedin.com>|In-Reply-To:=20<57A4D9DB-2CD2-4838-AD65-7D
   EBA1F3F0EA@mac.com>|References:=20<57A4D9DB-2CD2-4838-AD6
   5-7DEBA1F3F0EA@mac.com>;
  bh=5eADLt+FAuJEs22YTeUN/6hB67LuLBeZmtFQlq2p2No=;
  b=w1xjMhqyVrZby/Bk+eOiUs7tp0JmznX2z5twfz10NA2+SpIeLb/D0SIk
   KHRQU9Ia+zdwnfoPUm5rPya3HXL5zMj9pADwzarV3kO4/LKap++iU5AHu
   zs2GLSbvN34OdP4;
X-IronPort-AV: E=Sophos;i="4.60,456,1291622400"; 
   d="scan'208";a="20083137"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi id 14.01.0251.000; Fri,
 11 Feb 2011 09:41:21 -0800
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: Hadoop 0.22 Blockers
Thread-Topic: Hadoop 0.22 Blockers
Thread-Index: AQHLyb5E13M7p8Xx00aNwoGFZtJq/pP9GAOA
Date: Fri, 11 Feb 2011 17:41:20 +0000
Message-ID: <3D9EA200-342B-4B1F-A2E8-6FC159BFDE38@linkedin.com>
References: <57A4D9DB-2CD2-4838-AD65-7DEBA1F3F0EA@mac.com>
In-Reply-To: <57A4D9DB-2CD2-4838-AD65-7DEBA1F3F0EA@mac.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <65133C54BF6D854D9057A3E553F5C4C4@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


On Feb 10, 2011, at 11:33 PM, Nigel Daley wrote:

> Tom has created a public Jira filter for 0.22 blockers (thanks Tom!):
> https://issues.apache.org/jira/secure/IssueNavigator.jspa?mode=3Dhide&req=
uestId=3D12313687
> With the new version of Jira, this single query across projects & version=
s is now possible.
>=20
> Looks like 26 issues marked blocker.  9 of them unassigned -- would love =
to get those assigned.  I'll start reviewing these issues more closely over=
 the weekend.

	Does this include the ones you unmarked as blockers for some mysterious re=
ason?


From general-return-3013-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 17:44:45 2011
Return-Path: <general-return-3013-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 52501 invoked from network); 11 Feb 2011 17:44:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 17:44:44 -0000
Received: (qmail 30336 invoked by uid 500); 11 Feb 2011 17:44:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30125 invoked by uid 500); 11 Feb 2011 17:44:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30117 invoked by uid 99); 11 Feb 2011 17:44:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 17:44:40 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.104 as permitted sender)
Received: from [17.148.16.104] (HELO asmtpout029.mac.com) (17.148.16.104)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 17:44:31 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [172.29.12.221] ([38.102.147.105])
 by asmtp029.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LGG0035MRXEFI20@asmtp029.mac.com>
 for general@hadoop.apache.org; Fri, 11 Feb 2011 09:44:02 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-02-11_06:2011-02-11,2011-02-11,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1102110081
Subject: Re: Hadoop 0.22 Blockers
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <3D9EA200-342B-4B1F-A2E8-6FC159BFDE38@linkedin.com>
Date: Fri, 11 Feb 2011 09:44:02 -0800
Message-id: <03B93C0F-FDB4-4704-8CF3-57B8DF00E9B8@mac.com>
References: <57A4D9DB-2CD2-4838-AD65-7DEBA1F3F0EA@mac.com>
 <3D9EA200-342B-4B1F-A2E8-6FC159BFDE38@linkedin.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org


On Feb 11, 2011, at 9:41 AM, Allen Wittenauer wrote:

> 
> On Feb 10, 2011, at 11:33 PM, Nigel Daley wrote:
> 
>> Tom has created a public Jira filter for 0.22 blockers (thanks Tom!):
>> https://issues.apache.org/jira/secure/IssueNavigator.jspa?mode=hide&requestId=12313687
>> With the new version of Jira, this single query across projects & versions is now possible.
>> 
>> Looks like 26 issues marked blocker.  9 of them unassigned -- would love to get those assigned.  I'll start reviewing these issues more closely over the weekend.
> 
> 	Does this include the ones you unmarked as blockers for some mysterious reason?

To which issues are you referring exactly?


From general-return-3014-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 17:44:59 2011
Return-Path: <general-return-3014-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 52598 invoked from network); 11 Feb 2011 17:44:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 17:44:59 -0000
Received: (qmail 31609 invoked by uid 500); 11 Feb 2011 17:44:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31512 invoked by uid 500); 11 Feb 2011 17:44:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31504 invoked by uid 99); 11 Feb 2011 17:44:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 17:44:55 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [128.149.139.109] (HELO mail.jpl.nasa.gov) (128.149.139.109)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 17:44:48 +0000
Received: from mail.jpl.nasa.gov (altvirehtstap01.jpl.nasa.gov [128.149.137.72])
	by smtp.jpl.nasa.gov (Switch-3.4.3/Switch-3.4.3) with ESMTP id p1BHiPtd000771
	(using TLSv1/SSLv3 with cipher RC4-MD5 (128 bits) verified FAIL)
	for <general@hadoop.apache.org>; Fri, 11 Feb 2011 09:44:25 -0800
Received: from ALTPHYEMBEVSP20.RES.AD.JPL ([128.149.137.82]) by
 ALTVIREHTSTAP01.RES.AD.JPL ([128.149.137.72]) with mapi; Fri, 11 Feb 2011
 09:44:25 -0800
From: "Mattmann, Chris A (388J)" <chris.a.mattmann@jpl.nasa.gov>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 11 Feb 2011 09:44:24 -0800
Subject: Re: [VOTE] Abandon mrunit MapReduce contrib
Thread-Topic: [VOTE] Abandon mrunit MapReduce contrib
Thread-Index: AcvKE1LnpNwZY2ONTaS1jQcAsBWSCw==
Message-ID: <65311075-A30E-4971-A2BA-C75409C6486E@jpl.nasa.gov>
References: <77405974-6771-4604-926B-976240743F3C@mac.com>
 <AANLkTim80KJ3f14iM6Ltzy7UG613myhN-AiwQ3pCmVdi@mail.gmail.com>
 <AANLkTiknkrWK=GzGMVj4_Z6p5Jn8TQu1P5TkaFSgORJk@mail.gmail.com>
 <AANLkTin0TguLYHi9X0uL3ZF7tCz5xTHBYa7056oKF2Ae@mail.gmail.com>
 <93644D3B-4830-4813-92BE-809738969CF9@apache.org>
 <AANLkTi=dbGSLhtFA+Rf9LgaX05v45HDLoQDE_2C5ULAf@mail.gmail.com>
In-Reply-To: <AANLkTi=dbGSLhtFA+Rf9LgaX05v45HDLoQDE_2C5ULAf@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Source-IP: altvirehtstap01.jpl.nasa.gov [128.149.137.72]
X-Source-Sender: chris.a.mattmann@jpl.nasa.gov
X-AUTH: Authorized
X-Virus-Checked: Checked by ClamAV on apache.org

Guys, BTW, if you need help or a mentor in Apache Incubator-ville for MRUni=
t, I would be happy to help.

Cheers,
Chris

On Feb 11, 2011, at 9:04 AM, Eric Sammer wrote:

> On Fri, Feb 11, 2011 at 11:48 AM, Owen O'Malley <omalley@apache.org> wrot=
e:
>=20
>> On Feb 11, 2011, at 8:02 AM, Eric Sammer wrote:
>>=20
>> - allow mrunit to have its own release cycle. This is, I think, the most
>>>=20
>>=20
>> important.
>>>=20
>>=20
>> If you submit your work to Apache we can evaluate it for inclusion in th=
e
>> 0.20.100 branch to get your changes released in a timely manner.
>=20
>=20
> I'm thinking in general (beyond the next immediate release). Independent =
of
> where mrunit goes, I think it should leave the contrib tree to facilitate
> light weight releases (the dependency on Hadoop proper is a public facing
> API - a pure client). I think most projects could benefit from this with =
the
> exception of things that are tightly coupled to Hadoop releases or touch
> non-public APIs.
>=20
>=20
>> I would actually prefer to move it to Extras or Incubator and leave this
>>> within the ASF.
>>>=20
>>=20
>> Extras is **NOT** inside of the ASF. Extras is a source hosting system f=
or
>> non-Apache projects that are related to Apache projects.
>=20
>=20
> Got it. Thanks for correcting me. I only mentioned it because someone
> suggested it to me initially.
>=20
>=20
>> Right now, I picked github because of the ability to easily
>> collaborate with others (and to use git).
>>=20
>=20
> I agree that it is unfortunate that Apache doesn't yet support read-write
>> git access. However, you'll find that building a community is easier at
>> Apache than at github.
>>=20
>=20
>> -- Owen
>>=20
>=20
>=20
>=20
> --=20
> Eric Sammer
> twitter: esammer
> data: www.cloudera.com


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


From general-return-3015-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 17:47:33 2011
Return-Path: <general-return-3015-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 52972 invoked from network); 11 Feb 2011 17:47:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 17:47:33 -0000
Received: (qmail 37584 invoked by uid 500); 11 Feb 2011 17:47:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36452 invoked by uid 500); 11 Feb 2011 17:47:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36194 invoked by uid 99); 11 Feb 2011 17:47:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 17:47:24 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 17:47:18 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=nb+qmdTqCNsVbdNVU+U4lBEUw1iuf5yfUIkLnO7Xbk6i7wQuidnsigBQ
   fRMRvTYN5ZO3QEIo7f5l50e+PdjkEOgqUDfl2uWDLt45+B0vSiDMxkwYK
   XEWBF5eA4MZ8GGM;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1297446433; x=1328982433;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20[VOTE]=20Abandon=20hod=20Common=20contr
   ib|Date:=20Fri,=2011=20Feb=202011=2017:48:22=20+0000
   |Message-ID:=20<84B97117-5DA4-4A32-912A-8DBFCE3D063E@link
   edin.com>|To:=20"<general@hadoop.apache.org>"=20<general@
   hadoop.apache.org>|MIME-Version:=201.0
   |Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<93ACD9280A292747AE4F56B05B826CE2@linkedin
   .com>|In-Reply-To:=20<AC35A7EF-1D68-4055-8D47-EDA2FCF8C2F
   6@mac.com>|References:=20<AC35A7EF-1D68-4055-8D47-EDA2FCF
   8C2F6@mac.com>;
  bh=t+RWGsXiIe2fWhxkSBk+uuNE7e2Lfc6L6pgEhsMXP5k=;
  b=kmFCnkTsyosw9kja2Fw2eyoBkVb9mzgbnTKkOefKTt30Kuxn2C+sIBcr
   5H2HqyPPtHDmnZxn45n4ozG1CPBIMMZq72oMrvXDqoGZZjI9MVrTuNKdO
   l0ghE1SgBElt8Kx;
X-IronPort-AV: E=Sophos;i="4.60,456,1291622400"; 
   d="scan'208";a="20658572"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi id 14.01.0251.000; Fri,
 11 Feb 2011 09:48:23 -0800
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: [VOTE] Abandon hod Common contrib
Thread-Topic: [VOTE] Abandon hod Common contrib
Thread-Index: AQHLyZbt98XqvxdoQkW+Mjf3xnP4gZP9GkiA
Date: Fri, 11 Feb 2011 17:48:22 +0000
Message-ID: <84B97117-5DA4-4A32-912A-8DBFCE3D063E@linkedin.com>
References: <AC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com>
In-Reply-To: <AC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <93ACD9280A292747AE4F56B05B826CE2@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


On Feb 10, 2011, at 6:51 PM, Nigel Daley wrote:

> I think the PMC should abandon the hod common contrib component.  It's la=
st meaningful contribution was February 2009:
>=20
> HADOOP-2898. Provide an option to specify a port range for Hadoop service=
s provisioned by HOD. Contributed by Peeyush Bishnoi.
>=20
> There are 44 unresolved contrib/hod issues in Jira, 1 of them Patch Avail=
able since November 2009 (HADOOP-6369).

-1

a) I don't think hod is actually part of any unit tests, so including it wo=
uld likely only be a burden on the tarball size.
b) The edu community uses this quite extensively, evidenced by the topic co=
ming up on the mailing lists at least once every two months or so and has f=
or years.  Can't say that about the other contrib modules other than the sc=
hedulers and streaming.
c) The community that does use it has even submitted a patch that we've ign=
ored.
d) With SGE in flux, there is no equivalent functionality as far as I'm awa=
re.




From general-return-3016-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 17:55:00 2011
Return-Path: <general-return-3016-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54832 invoked from network); 11 Feb 2011 17:55:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 17:55:00 -0000
Received: (qmail 51865 invoked by uid 500); 11 Feb 2011 17:54:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51587 invoked by uid 500); 11 Feb 2011 17:54:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51571 invoked by uid 99); 11 Feb 2011 17:54:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 17:54:55 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of tom.e.white@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 17:54:49 +0000
Received: by wwd20 with SMTP id 20so2730374wwd.29
        for <general@hadoop.apache.org>; Fri, 11 Feb 2011 09:54:27 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type:content-transfer-encoding;
        bh=YWdwO6cTNg2/9ERPFoEf4rpWbqOhoGSfimENE2bwrko=;
        b=CvfuE+TWWFRAOwM+sHNUGHvnekWh6qa4GQa8kztjToUH1pzJhF3kGkTbGbtzgs4nQu
         Clq6nO2W8PmnbKO7KIJAkVQr5CNu033dFphkf1ucBad7GtkTu2fBOyAOJ93tdliuYa6M
         pZoeIuO7cnAZsNrRoTeh+wbwpmFLBal1sJzVA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=Ec89c+bBAhCAnxc0Zu/sYS+4p9rOy6dTJzbU8ZsAPyy8uKO/hufLqmZ77godZ2bOXD
         P5+Qg6KS3cMFYSfojJYYra4+oVq6K94q26qor+Qr5myWWmmj9TXeJwpgBHEZmetWJcp5
         Mdbz9fUQ17gYl976VyuhjVuH0TB9eJw/xoA/M=
Received: by 10.216.165.204 with SMTP id e54mr815110wel.48.1297446867194; Fri,
 11 Feb 2011 09:54:27 -0800 (PST)
MIME-Version: 1.0
Received: by 10.216.159.193 with HTTP; Fri, 11 Feb 2011 09:54:06 -0800 (PST)
In-Reply-To: <AC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com>
References: <AC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com>
From: Tom White <tom.e.white@gmail.com>
Date: Fri, 11 Feb 2011 09:54:06 -0800
Message-ID: <AANLkTinGfL3z37pNPXOTRwuE=17QHBNO3AqPH+Ar7evf@mail.gmail.com>
Subject: Re: [VOTE] Abandon hod Common contrib
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

+1

Tom

On Thu, Feb 10, 2011 at 6:51 PM, Nigel Daley <ndaley@mac.com> wrote:
> I think the PMC should abandon the hod common contrib component. =A0It's =
last meaningful contribution was February 2009:
>
> HADOOP-2898. Provide an option to specify a port range for Hadoop service=
s provisioned by HOD. Contributed by Peeyush Bishnoi.
>
> There are 44 unresolved contrib/hod issues in Jira, 1 of them Patch Avail=
able since November 2009 (HADOOP-6369).
>
> Here is my +1.
>
> Nige
>

From general-return-3017-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 18:12:53 2011
Return-Path: <general-return-3017-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68876 invoked from network); 11 Feb 2011 18:12:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 18:12:52 -0000
Received: (qmail 93470 invoked by uid 500); 11 Feb 2011 18:12:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93338 invoked by uid 500); 11 Feb 2011 18:12:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93324 invoked by uid 99); 11 Feb 2011 18:12:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 18:12:48 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of tom.e.white@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 18:12:42 +0000
Received: by wye20 with SMTP id 20so2767591wye.35
        for <general@hadoop.apache.org>; Fri, 11 Feb 2011 10:12:21 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type:content-transfer-encoding;
        bh=DMkSha7jqNaVHffk/Je6C8y3oCcL6M+hdrYKFSrRxPA=;
        b=vp38Y0mT5RvfexgCI6n9j/POFRh0nv+zXTF2G85c6/RRFWiX4mVC7tt6tXU7+bw2ay
         uE1Ur5eYg8tGNw+QjjfHaTXNYE3bs9WtDdSvVaFh65+SWCi55GQq60UI9IKNOLykULsv
         wGe0r6Qmp4Apfg+a4djq02ZBzh8ttwXJc2k+U=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=rlbh0WZKBs0nxNFiod1VmKvWiia5njgLXXBo5YTc4FHfAvmSWJF5xQEjvMPUoPXdn+
         xmECGVDq4PhJ5pTzotzXnQo5x9F7nA+Or+xTjaHwXWMGADkjlZvlUIBbwZeuxFE8//NF
         GefD9uaN6rkLmxdbsaV1q8Ioyqfvcza6doM0o=
Received: by 10.216.55.145 with SMTP id k17mr832868wec.48.1297447941593; Fri,
 11 Feb 2011 10:12:21 -0800 (PST)
MIME-Version: 1.0
Received: by 10.216.159.193 with HTTP; Fri, 11 Feb 2011 10:12:01 -0800 (PST)
In-Reply-To: <AANLkTi=75e6oyBuGVivXP3is+76gOODSc1NcLKfg=VxF@mail.gmail.com>
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com> <AANLkTi=75e6oyBuGVivXP3is+76gOODSc1NcLKfg=VxF@mail.gmail.com>
From: Tom White <tom.e.white@gmail.com>
Date: Fri, 11 Feb 2011 10:12:01 -0800
Message-ID: <AANLkTi=HQ1cn++JO6g+vp-cJAtAjB00Ok-FW2VrfTiSO@mail.gmail.com>
Subject: Re: [DISCUSS] Move common, hdfs, mapreduce contrib components to
 apache-extras.org or elsewhere
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

For contrib components that we elect not to remove, I suggest that we
remove them from the main build, so that failures in a contrib
component don't hinder the main build and release. This is what
ZooKeeper does, for example.

Tom

On Fri, Feb 11, 2011 at 2:03 AM, Bernd Fondermann
<bernd.fondermann@googlemail.com> wrote:
> -1.
>
> Move it away from TRUNK so it doesn't affect builds is a much better
> (and simpler) solution. If someone wants to revive it, he can within
> the bounds of Apache Hadoop and will become a part of the Hadoop
> community, which would be good.
> If you'd move it off-site, if the code ever gets new contributors,
> it's hard to integrate them (code and contributors) into Hadoop again.
>
> AFAIUI, apache-extras is for placing non-Apache code closer to the
> related Apache projects, not for moving our code away from our own
> premises.
>
> =A0Bernd
>
> On Mon, Jan 31, 2011 at 04:42, Nigel Daley <ndaley@mac.com> wrote:
>> Folks,
>>
>> Now that http://apache-extras.org is launched (https://blogs.apache.org/=
foundation/entry/the_apache_software_foundation_launches) I'd like to start=
 a discussion on moving contrib components out of common, mapreduce, and hd=
fs.
>>
>> These contrib components complicate the builds, cause test failures that=
 nobody seems to care about, have releases that are tied to Hadoop's long r=
elease cycles, etc. =A0Most folks I've talked with agree that these contrib=
 components would be better served by being pulled out of Hadoop and hosted=
 elsewhere. The new apache-extras code hosting site seems like a natural *d=
efault* location for migrating these contrib projects. =A0Perhaps some shou=
ld graduate from contrib to src (ie from contrib to core of the project the=
y're included in). =A0If folks agree, we'll need to come up with a mapping =
of contrib component to it's final destination and file a jira.
>>
>> Here are the contrib components by project (hopefully I didn't miss any)=
.
>>
>> Common Contrib:
>> =A0failmon
>> =A0hod
>> =A0test
>>
>>
>> MapReduce Contrib:
>> =A0capacity-scheduler -- move to MR core?
>> =A0data_join
>> =A0dynamic-scheduler
>> =A0eclipse-plugin
>> =A0fairscheduler -- move to MR core?
>> =A0gridmix
>> =A0index
>> =A0mrunit
>> =A0mumak
>> =A0raid
>> =A0sqoop
>> =A0streaming -- move to MR core?
>> =A0vaidya
>> =A0vertica
>>
>>
>> HDFS Contrib:
>> =A0fuse-dfs
>> =A0hdfsproxy
>> =A0thriftfs
>>
>>
>> Cheers,
>> Nige
>>
>

From general-return-3018-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 18:21:16 2011
Return-Path: <general-return-3018-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70563 invoked from network); 11 Feb 2011 18:21:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 18:21:16 -0000
Received: (qmail 8146 invoked by uid 500); 11 Feb 2011 18:21:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7716 invoked by uid 500); 11 Feb 2011 18:21:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7708 invoked by uid 99); 11 Feb 2011 18:21:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 18:21:11 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of tom.e.white@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 18:21:04 +0000
Received: by wwd20 with SMTP id 20so2755703wwd.29
        for <general@hadoop.apache.org>; Fri, 11 Feb 2011 10:20:43 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type:content-transfer-encoding;
        bh=pRLDeaRkHFzblmgw3tvWYOBz9Uf8eABJHA1mO9UQe88=;
        b=U7YYXqcAmSg7NuiXTRjNuP2r4mfyGI5aewtkHYYiefXjUnoO3NI8WKXSslpOEj78ms
         SYnSEZzAdmnBhzGNEnQhKLoCQ9FoNNH+Y77zUZhhY8ZwD2TOeXz350kGysBTs45/JSX0
         NJAt24eG4XRqiyFRHHUrGhgPwxCdVNRi686ig=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=j1oABDu9YTKLWI2sbdPUQS1u/mzn+imETV/43DqVrmtQ6EediF9L4x0Z9WCGc0kfXt
         whNb0L2Zjn8dgsUH/casuug53mEZM8SHygls3gdiqyVAW3RX47Ur0Qj7UebVqrRZG/NG
         vXp4Nqn64B7XeC91J45rDTnNbuMfRWEPH6SH0=
Received: by 10.216.173.147 with SMTP id v19mr792892wel.102.1297446546459;
 Fri, 11 Feb 2011 09:49:06 -0800 (PST)
MIME-Version: 1.0
Received: by 10.216.159.193 with HTTP; Fri, 11 Feb 2011 09:48:45 -0800 (PST)
In-Reply-To: <EF44CFE2-692F-4956-8B33-D125D05E2B53@mac.com>
References: <EF44CFE2-692F-4956-8B33-D125D05E2B53@mac.com>
From: Tom White <tom.e.white@gmail.com>
Date: Fri, 11 Feb 2011 09:48:45 -0800
Message-ID: <AANLkTikouS9=ksDypNXa2LGUdqTbzAUbjEW4QeJszPAk@mail.gmail.com>
Subject: Re: [VOTE] Abandon thriftfs HDFS contrib
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Tom

On Thu, Feb 10, 2011 at 7:44 PM, Nigel Daley <ndaley@mac.com> wrote:
> I think the PMC should abandon the thriftfs HDFS contrib component. =A0It=
's last meaningful contribution was it's original commit in August 2008:
>
> HADOOP-3754. Add a thrift interface to access HDFS. (dhruba via omalley)
>
> There are 5 unresolved contrib/thriftfs issues in Jira, 1 of them Patch A=
vailable since November 2009 (HDFS-789).
>
> Here is my +1.
>
> Nige
>

From general-return-3019-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 18:26:57 2011
Return-Path: <general-return-3019-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78086 invoked from network); 11 Feb 2011 18:26:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 18:26:56 -0000
Received: (qmail 12867 invoked by uid 500); 11 Feb 2011 18:26:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12710 invoked by uid 500); 11 Feb 2011 18:26:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12673 invoked by uid 99); 11 Feb 2011 18:26:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 18:26:52 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of wugarrett@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 18:26:46 +0000
Received: by bwz8 with SMTP id 8so3671492bwz.35
        for <general@hadoop.apache.org>; Fri, 11 Feb 2011 10:26:26 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=r+0anfbvbUCXsylHfC5gAgVlOEDE47ymELQ0jOZy6t4=;
        b=luyyJXt1uRfrLChn50n7P90mpbgMVA/otnBV1XTsLXHQklqetRZVuiQYDN5sR26lB1
         2r1iFNAkf6o0TrGNMEETOFY8mII+7mQxrNWXNsscaZ70MJxA6fwyMKc47ckamX/ztJh2
         Ce4QK5CNAEh0nVXGNmhHgIC1+DWAPYWUrC0uQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=BNyhG9Ks203v0+WdYeitdWfcqfnuT+eMdbYsUDQyiojq5PNFD5hQuG5ORI6a86Cnxc
         NH4un5VM1Z6uUGlUK8QQCO2gyqCaLTHbigsct0kYnK3Hzf0hLFS7FehY6OD+CGfdRVd+
         TO3I4DJe8iI6HwTo+sdcxSS/bSbD0oHGX4D7o=
MIME-Version: 1.0
Received: by 10.204.76.207 with SMTP id d15mr13035070bkk.104.1297448785756;
 Fri, 11 Feb 2011 10:26:25 -0800 (PST)
Received: by 10.204.80.6 with HTTP; Fri, 11 Feb 2011 10:26:25 -0800 (PST)
In-Reply-To: <AANLkTi=dbGSLhtFA+Rf9LgaX05v45HDLoQDE_2C5ULAf@mail.gmail.com>
References: <77405974-6771-4604-926B-976240743F3C@mac.com>
	<AANLkTim80KJ3f14iM6Ltzy7UG613myhN-AiwQ3pCmVdi@mail.gmail.com>
	<AANLkTiknkrWK=GzGMVj4_Z6p5Jn8TQu1P5TkaFSgORJk@mail.gmail.com>
	<AANLkTin0TguLYHi9X0uL3ZF7tCz5xTHBYa7056oKF2Ae@mail.gmail.com>
	<93644D3B-4830-4813-92BE-809738969CF9@apache.org>
	<AANLkTi=dbGSLhtFA+Rf9LgaX05v45HDLoQDE_2C5ULAf@mail.gmail.com>
Date: Fri, 11 Feb 2011 10:26:25 -0800
Message-ID: <AANLkTikBQ1aDpq_PYQe5Sv6a0jOw03851XGt00D27QCU@mail.gmail.com>
Subject: Re: [VOTE] Abandon mrunit MapReduce contrib
From: Garrett Wu <wugarrett@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636499889fd4201049c05d5ef
X-Virus-Checked: Checked by ClamAV on apache.org

--001636499889fd4201049c05d5ef
Content-Type: text/plain; charset=ISO-8859-1

On Fri, Feb 11, 2011 at 9:04 AM, Eric Sammer <esammer@cloudera.com> wrote:

> On Fri, Feb 11, 2011 at 11:48 AM, Owen O'Malley <omalley@apache.org>
> wrote:
>
> > On Feb 11, 2011, at 8:02 AM, Eric Sammer wrote:
> >
> >  - allow mrunit to have its own release cycle. This is, I think, the most
> >>
> >
> >  important.
> >>
> >
> > If you submit your work to Apache we can evaluate it for inclusion in the
> > 0.20.100 branch to get your changes released in a timely manner.
>
>
> I'm thinking in general (beyond the next immediate release). Independent of
> where mrunit goes, I think it should leave the contrib tree to facilitate
> light weight releases (the dependency on Hadoop proper is a public facing
> API - a pure client). I think most projects could benefit from this with
> the
> exception of things that are tightly coupled to Hadoop releases or touch
> non-public APIs.
>
>
+1 for a faster release cycle and using git.  I have a couple of patches for
mrunit, and it would be nice to get those in separately from hadoop
releases.


>
> >  I would actually prefer to move it to Extras or Incubator and leave this
> >> within the ASF.
> >>
> >
> > Extras is **NOT** inside of the ASF. Extras is a source hosting system
> for
> > non-Apache projects that are related to Apache projects.
>
>
> Got it. Thanks for correcting me. I only mentioned it because someone
> suggested it to me initially.
>
>
> > Right now, I picked github because of the ability to easily
> > collaborate with others (and to use git).
> >
>
> I agree that it is unfortunate that Apache doesn't yet support read-write
> > git access. However, you'll find that building a community is easier at
> > Apache than at github.
> >
>
> > -- Owen
> >
>
>
>
> --
> Eric Sammer
> twitter: esammer
> data: www.cloudera.com
>

--001636499889fd4201049c05d5ef--

From general-return-3020-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 19:38:08 2011
Return-Path: <general-return-3020-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 7342 invoked from network); 11 Feb 2011 19:38:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 19:38:08 -0000
Received: (qmail 95176 invoked by uid 500); 11 Feb 2011 19:38:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95072 invoked by uid 500); 11 Feb 2011 19:38:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95062 invoked by uid 99); 11 Feb 2011 19:38:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 19:38:05 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of gmackey@cs.ucf.edu designates 132.170.216.154 as permitted sender)
Received: from [132.170.216.154] (HELO longwood.cs.ucf.edu) (132.170.216.154)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 19:37:58 +0000
Received: from longwood.cs.ucf.edu (localhost [127.0.0.1])
	by longwood.cs.ucf.edu (8.14.1/8.14.1) with ESMTP id p1BJbZkQ003092
	for <general@hadoop.apache.org>; Fri, 11 Feb 2011 14:37:36 -0500 (EST)
Received: (from apache@localhost)
	by longwood.cs.ucf.edu (8.14.1/8.14.4/Submit) id p1BJbZLZ003091
	for general@hadoop.apache.org; Fri, 11 Feb 2011 14:37:35 -0500 (EST)
X-Authentication-Warning: longwood.cs.ucf.edu: apache set sender to gmackey@cs.ucf.edu using -f
Received: from 10.173.214.212 ([10.173.214.212]) by mail.cs.ucf.edu (Horde
 Framework) with HTTP; Fri, 11 Feb 2011 14:37:35 -0500
Message-ID: <20110211143735.38602fmir0zmqi3o@mail.cs.ucf.edu>
Date: Fri, 11 Feb 2011 14:37:35 -0500
From: Grant Mackey <gmackey@cs.ucf.edu>
To: general@hadoop.apache.org
Subject: Re: [VOTE] Abandon fuse-dfs HDFS contrib
References: <E43E21C9-4267-440D-A0C8-9CB21E9DBADA@mac.com>
In-Reply-To: <E43E21C9-4267-440D-A0C8-9CB21E9DBADA@mac.com>
MIME-Version: 1.0
Content-Type: text/plain;
 charset=ISO-8859-1;
 DelSp="Yes";
 format="flowed"
Content-Disposition: inline
Content-Transfer-Encoding: 7bit
User-Agent: Internet Messaging Program (IMP) 4.3.3
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on gondor.cs.ucf.edu
X-Virus-Scanned: clamav-milter 0.96.5 at longwood
X-Virus-Status: Clean
X-Virus-Checked: Checked by ClamAV on apache.org
X-Old-Spam-Status: No, score=-1.4 required=6.0 tests=ALL_TRUSTED autolearn=failed
	version=3.2.5

As a university researcher for file systems, having the fuse-dfs code  
makes life easy when trying to run i/o benchmark apps. I would hate to  
see it go. Here's my -1

  - Grant

Quoting Nigel Daley <ndaley@mac.com>:

> I think the PMC should abandon the fuse-dfs HDFS contrib component.   
> It's last meaningful contribution was March 2010:
>
> HDFS-961. dfs_readdir incorrectly parses paths. Contributed by Eli Collins.
>
> There are 18 unresolved contrib/fuse-dfs issues in Jira, none of  
> them Patch Available.
>
> Here is my +1.
>
> Nige
>



Grant Mackey
UCF Research Assistant
Engineering III
Rm 238 Cubicle 1

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.


From general-return-3021-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 19:46:45 2011
Return-Path: <general-return-3021-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9436 invoked from network); 11 Feb 2011 19:46:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 19:46:45 -0000
Received: (qmail 5646 invoked by uid 500); 11 Feb 2011 19:46:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5386 invoked by uid 500); 11 Feb 2011 19:46:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5378 invoked by uid 99); 11 Feb 2011 19:46:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 19:46:42 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of esammer@cloudera.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 19:46:38 +0000
Received: by vws18 with SMTP id 18so1832094vws.35
        for <general@hadoop.apache.org>; Fri, 11 Feb 2011 11:46:17 -0800 (PST)
MIME-Version: 1.0
Received: by 10.220.186.195 with SMTP id ct3mr1120299vcb.57.1297453576743;
 Fri, 11 Feb 2011 11:46:16 -0800 (PST)
Received: by 10.220.5.80 with HTTP; Fri, 11 Feb 2011 11:46:16 -0800 (PST)
In-Reply-To: <20110211143735.38602fmir0zmqi3o@mail.cs.ucf.edu>
References: <E43E21C9-4267-440D-A0C8-9CB21E9DBADA@mac.com>
	<20110211143735.38602fmir0zmqi3o@mail.cs.ucf.edu>
Date: Fri, 11 Feb 2011 14:46:16 -0500
Message-ID: <AANLkTi=nDNm7kogT3ooTKZmEjMBJEUphy3NhKkvzta5B@mail.gmail.com>
Subject: Re: [VOTE] Abandon fuse-dfs HDFS contrib
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba53a8968deb3b049c06f385

--90e6ba53a8968deb3b049c06f385
Content-Type: text/plain; charset=ISO-8859-1

I see these votes as a desire to simply not develop this code under the
Hadoop project, not to kill the code base. I believe the general intention
is to "clean up" the project so it passes the "do one thing and do it well"
rule (or "develop one thing and develop it well"). Projects that are related
such as fuse-dfs, mrunit, and so on should and must exist. The question (and
vote) is simply should it be within Hadoop proper.

I think they can thrive in the incubator or wherever the majority of active
contributors wish to host it (although Owen has made the good point about
visibility and community development features of the ASF). My (non-binding)
vote is +1.

On Fri, Feb 11, 2011 at 2:37 PM, Grant Mackey <gmackey@cs.ucf.edu> wrote:

> As a university researcher for file systems, having the fuse-dfs code makes
> life easy when trying to run i/o benchmark apps. I would hate to see it go.
> Here's my -1
>
>  - Grant
>
>
> Quoting Nigel Daley <ndaley@mac.com>:
>
>  I think the PMC should abandon the fuse-dfs HDFS contrib component.  It's
>> last meaningful contribution was March 2010:
>>
>> HDFS-961. dfs_readdir incorrectly parses paths. Contributed by Eli
>> Collins.
>>
>> There are 18 unresolved contrib/fuse-dfs issues in Jira, none of them
>> Patch Available.
>>
>> Here is my +1.
>>
>> Nige
>>
>>
>
>
> Grant Mackey
> UCF Research Assistant
> Engineering III
> Rm 238 Cubicle 1
>
> ----------------------------------------------------------------
> This message was sent using IMP, the Internet Messaging Program.
>
>


-- 
Eric Sammer
twitter: esammer
data: www.cloudera.com

--90e6ba53a8968deb3b049c06f385--

From general-return-3022-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 19:46:48 2011
Return-Path: <general-return-3022-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9521 invoked from network); 11 Feb 2011 19:46:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 19:46:47 -0000
Received: (qmail 6868 invoked by uid 500); 11 Feb 2011 19:46:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6782 invoked by uid 500); 11 Feb 2011 19:46:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6774 invoked by uid 99); 11 Feb 2011 19:46:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 19:46:45 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [128.149.139.109] (HELO mail.jpl.nasa.gov) (128.149.139.109)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 19:46:38 +0000
Received: from mail.jpl.nasa.gov (altvirehtstap02.jpl.nasa.gov [128.149.137.73])
	by smtp.jpl.nasa.gov (Switch-3.4.3/Switch-3.4.3) with ESMTP id p1BJkGv6018705
	(using TLSv1/SSLv3 with cipher RC4-MD5 (128 bits) verified FAIL)
	for <general@hadoop.apache.org>; Fri, 11 Feb 2011 11:46:16 -0800
Received: from ALTPHYEMBEVSP20.RES.AD.JPL ([128.149.137.82]) by
 ALTVIREHTSTAP02.RES.AD.JPL ([128.149.137.73]) with mapi; Fri, 11 Feb 2011
 11:46:16 -0800
From: "Mattmann, Chris A (388J)" <chris.a.mattmann@jpl.nasa.gov>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 11 Feb 2011 11:46:15 -0800
Subject: Re: [VOTE] Abandon fuse-dfs HDFS contrib
Thread-Topic: [VOTE] Abandon fuse-dfs HDFS contrib
Thread-Index: AcvKJFhzszlhVGf9Ryy1fQbpTsmB7Q==
Message-ID: <8DFFD4D7-918D-499F-A814-8A2DBD0D6823@jpl.nasa.gov>
References: <E43E21C9-4267-440D-A0C8-9CB21E9DBADA@mac.com>
 <20110211143735.38602fmir0zmqi3o@mail.cs.ucf.edu>
In-Reply-To: <20110211143735.38602fmir0zmqi3o@mail.cs.ucf.edu>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Source-IP: altvirehtstap02.jpl.nasa.gov [128.149.137.73]
X-Source-Sender: chris.a.mattmann@jpl.nasa.gov
X-AUTH: Authorized
X-Virus-Checked: Checked by ClamAV on apache.org

-1 (not binding) as well.

Cheers,
Chris

On Feb 11, 2011, at 11:37 AM, Grant Mackey wrote:

> As a university researcher for file systems, having the fuse-dfs code =20
> makes life easy when trying to run i/o benchmark apps. I would hate to =20
> see it go. Here's my -1
>=20
>  - Grant
>=20
> Quoting Nigel Daley <ndaley@mac.com>:
>=20
>> I think the PMC should abandon the fuse-dfs HDFS contrib component.  =20
>> It's last meaningful contribution was March 2010:
>>=20
>> HDFS-961. dfs_readdir incorrectly parses paths. Contributed by Eli Colli=
ns.
>>=20
>> There are 18 unresolved contrib/fuse-dfs issues in Jira, none of =20
>> them Patch Available.
>>=20
>> Here is my +1.
>>=20
>> Nige
>>=20
>=20
>=20
>=20
> Grant Mackey
> UCF Research Assistant
> Engineering III
> Rm 238 Cubicle 1
>=20
> ----------------------------------------------------------------
> This message was sent using IMP, the Internet Messaging Program.
>=20


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


From general-return-3023-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 19:52:54 2011
Return-Path: <general-return-3023-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 10929 invoked from network); 11 Feb 2011 19:52:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 19:52:53 -0000
Received: (qmail 15269 invoked by uid 500); 11 Feb 2011 19:52:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15220 invoked by uid 500); 11 Feb 2011 19:52:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15212 invoked by uid 99); 11 Feb 2011 19:52:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 19:52:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 11 Feb 2011 19:52:49 +0000
Received: (qmail 10712 invoked by uid 99); 11 Feb 2011 19:52:27 -0000
Received: from localhost.apache.org (HELO mail-iw0-f176.google.com) (127.0.0.1)
  (smtp-auth username phunt, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 19:52:27 +0000
Received: by iwn2 with SMTP id 2so2923860iwn.35
        for <general@hadoop.apache.org>; Fri, 11 Feb 2011 11:52:27 -0800 (PST)
MIME-Version: 1.0
Received: by 10.42.177.74 with SMTP id bh10mr1187234icb.159.1297453946960;
 Fri, 11 Feb 2011 11:52:26 -0800 (PST)
Received: by 10.42.174.193 with HTTP; Fri, 11 Feb 2011 11:52:26 -0800 (PST)
In-Reply-To: <65311075-A30E-4971-A2BA-C75409C6486E@jpl.nasa.gov>
References: <77405974-6771-4604-926B-976240743F3C@mac.com>
	<AANLkTim80KJ3f14iM6Ltzy7UG613myhN-AiwQ3pCmVdi@mail.gmail.com>
	<AANLkTiknkrWK=GzGMVj4_Z6p5Jn8TQu1P5TkaFSgORJk@mail.gmail.com>
	<AANLkTin0TguLYHi9X0uL3ZF7tCz5xTHBYa7056oKF2Ae@mail.gmail.com>
	<93644D3B-4830-4813-92BE-809738969CF9@apache.org>
	<AANLkTi=dbGSLhtFA+Rf9LgaX05v45HDLoQDE_2C5ULAf@mail.gmail.com>
	<65311075-A30E-4971-A2BA-C75409C6486E@jpl.nasa.gov>
Date: Fri, 11 Feb 2011 11:52:26 -0800
Message-ID: <AANLkTi=XbpVUmg_TpH7eqEn8w7Gcu9KiwtJmKMLa5w5n@mail.gmail.com>
Subject: Re: [VOTE] Abandon mrunit MapReduce contrib
From: Patrick Hunt <phunt@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Fri, Feb 11, 2011 at 9:44 AM, Mattmann, Chris A (388J)
<chris.a.mattmann@jpl.nasa.gov> wrote:
> Guys, BTW, if you need help or a mentor in Apache Incubator-ville for MRU=
nit, I would be happy to help.

I was going to suggest the same thing (mrunit to incubator). I would
also be happy to be a mentor.

Patrick

>
> On Feb 11, 2011, at 9:04 AM, Eric Sammer wrote:
>
>> On Fri, Feb 11, 2011 at 11:48 AM, Owen O'Malley <omalley@apache.org> wro=
te:
>>
>>> On Feb 11, 2011, at 8:02 AM, Eric Sammer wrote:
>>>
>>> - allow mrunit to have its own release cycle. This is, I think, the mos=
t
>>>>
>>>
>>> important.
>>>>
>>>
>>> If you submit your work to Apache we can evaluate it for inclusion in t=
he
>>> 0.20.100 branch to get your changes released in a timely manner.
>>
>>
>> I'm thinking in general (beyond the next immediate release). Independent=
 of
>> where mrunit goes, I think it should leave the contrib tree to facilitat=
e
>> light weight releases (the dependency on Hadoop proper is a public facin=
g
>> API - a pure client). I think most projects could benefit from this with=
 the
>> exception of things that are tightly coupled to Hadoop releases or touch
>> non-public APIs.
>>
>>
>>> I would actually prefer to move it to Extras or Incubator and leave thi=
s
>>>> within the ASF.
>>>>
>>>
>>> Extras is **NOT** inside of the ASF. Extras is a source hosting system =
for
>>> non-Apache projects that are related to Apache projects.
>>
>>
>> Got it. Thanks for correcting me. I only mentioned it because someone
>> suggested it to me initially.
>>
>>
>>> Right now, I picked github because of the ability to easily
>>> collaborate with others (and to use git).
>>>
>>
>> I agree that it is unfortunate that Apache doesn't yet support read-writ=
e
>>> git access. However, you'll find that building a community is easier at
>>> Apache than at github.
>>>
>>
>>> -- Owen
>>>
>>
>>
>>
>> --
>> Eric Sammer
>> twitter: esammer
>> data: www.cloudera.com
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: =A0 http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>

From general-return-3024-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 19:58:16 2011
Return-Path: <general-return-3024-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 11992 invoked from network); 11 Feb 2011 19:58:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 19:58:15 -0000
Received: (qmail 20064 invoked by uid 500); 11 Feb 2011 19:58:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20020 invoked by uid 500); 11 Feb 2011 19:58:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 20012 invoked by uid 99); 11 Feb 2011 19:58:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 19:58:13 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [128.149.139.109] (HELO mail.jpl.nasa.gov) (128.149.139.109)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 19:58:05 +0000
Received: from mail.jpl.nasa.gov (altvirehtstap02.jpl.nasa.gov [128.149.137.73])
	by smtp.jpl.nasa.gov (Switch-3.4.3/Switch-3.4.3) with ESMTP id p1BJvgXd026150
	(using TLSv1/SSLv3 with cipher RC4-MD5 (128 bits) verified FAIL)
	for <general@hadoop.apache.org>; Fri, 11 Feb 2011 11:57:43 -0800
Received: from ALTPHYEMBEVSP20.RES.AD.JPL ([128.149.137.82]) by
 ALTVIREHTSTAP02.RES.AD.JPL ([128.149.137.73]) with mapi; Fri, 11 Feb 2011
 11:57:43 -0800
From: "Mattmann, Chris A (388J)" <chris.a.mattmann@jpl.nasa.gov>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 11 Feb 2011 11:57:41 -0800
Subject: Re: [VOTE] Abandon mrunit MapReduce contrib
Thread-Topic: [VOTE] Abandon mrunit MapReduce contrib
Thread-Index: AcvKJfEofw+8XlXUSzekMZNhhEnanQ==
Message-ID: <FAE19BAA-47A7-4747-BDD8-5AA42C259EB6@jpl.nasa.gov>
References: <77405974-6771-4604-926B-976240743F3C@mac.com>
 <AANLkTim80KJ3f14iM6Ltzy7UG613myhN-AiwQ3pCmVdi@mail.gmail.com>
 <AANLkTiknkrWK=GzGMVj4_Z6p5Jn8TQu1P5TkaFSgORJk@mail.gmail.com>
 <AANLkTin0TguLYHi9X0uL3ZF7tCz5xTHBYa7056oKF2Ae@mail.gmail.com>
 <93644D3B-4830-4813-92BE-809738969CF9@apache.org>
 <AANLkTi=dbGSLhtFA+Rf9LgaX05v45HDLoQDE_2C5ULAf@mail.gmail.com>
 <65311075-A30E-4971-A2BA-C75409C6486E@jpl.nasa.gov>
 <AANLkTi=XbpVUmg_TpH7eqEn8w7Gcu9KiwtJmKMLa5w5n@mail.gmail.com>
In-Reply-To: <AANLkTi=XbpVUmg_TpH7eqEn8w7Gcu9KiwtJmKMLa5w5n@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Source-IP: altvirehtstap02.jpl.nasa.gov [128.149.137.73]
X-Source-Sender: chris.a.mattmann@jpl.nasa.gov
X-AUTH: Authorized
X-Virus-Checked: Checked by ClamAV on apache.org

Awesome Patrick, we'd probably need one more active mentor. Any takers?

After we get that, then we cook up a proposal on the Incubator wiki here [1=
], and follow the process here [2] to get started...

Cheers,
Chris

[1] http://wiki.apache.org/incubator/MRUnitProposal
[2] http://incubator.apache.org/guides/proposal.html

On Feb 11, 2011, at 11:52 AM, Patrick Hunt wrote:

> On Fri, Feb 11, 2011 at 9:44 AM, Mattmann, Chris A (388J)
> <chris.a.mattmann@jpl.nasa.gov> wrote:
>> Guys, BTW, if you need help or a mentor in Apache Incubator-ville for MR=
Unit, I would be happy to help.
>=20
> I was going to suggest the same thing (mrunit to incubator). I would
> also be happy to be a mentor.
>=20
> Patrick
>=20
>>=20
>> On Feb 11, 2011, at 9:04 AM, Eric Sammer wrote:
>>=20
>>> On Fri, Feb 11, 2011 at 11:48 AM, Owen O'Malley <omalley@apache.org> wr=
ote:
>>>=20
>>>> On Feb 11, 2011, at 8:02 AM, Eric Sammer wrote:
>>>>=20
>>>> - allow mrunit to have its own release cycle. This is, I think, the mo=
st
>>>>>=20
>>>>=20
>>>> important.
>>>>>=20
>>>>=20
>>>> If you submit your work to Apache we can evaluate it for inclusion in =
the
>>>> 0.20.100 branch to get your changes released in a timely manner.
>>>=20
>>>=20
>>> I'm thinking in general (beyond the next immediate release). Independen=
t of
>>> where mrunit goes, I think it should leave the contrib tree to facilita=
te
>>> light weight releases (the dependency on Hadoop proper is a public faci=
ng
>>> API - a pure client). I think most projects could benefit from this wit=
h the
>>> exception of things that are tightly coupled to Hadoop releases or touc=
h
>>> non-public APIs.
>>>=20
>>>=20
>>>> I would actually prefer to move it to Extras or Incubator and leave th=
is
>>>>> within the ASF.
>>>>>=20
>>>>=20
>>>> Extras is **NOT** inside of the ASF. Extras is a source hosting system=
 for
>>>> non-Apache projects that are related to Apache projects.
>>>=20
>>>=20
>>> Got it. Thanks for correcting me. I only mentioned it because someone
>>> suggested it to me initially.
>>>=20
>>>=20
>>>> Right now, I picked github because of the ability to easily
>>>> collaborate with others (and to use git).
>>>>=20
>>>=20
>>> I agree that it is unfortunate that Apache doesn't yet support read-wri=
te
>>>> git access. However, you'll find that building a community is easier a=
t
>>>> Apache than at github.
>>>>=20
>>>=20
>>>> -- Owen
>>>>=20
>>>=20
>>>=20
>>>=20
>>> --
>>> Eric Sammer
>>> twitter: esammer
>>> data: www.cloudera.com
>>=20
>>=20
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW:   http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>=20
>>=20


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


From general-return-3025-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 21:04:30 2011
Return-Path: <general-return-3025-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 41451 invoked from network); 11 Feb 2011 21:04:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 21:04:30 -0000
Received: (qmail 28849 invoked by uid 500); 11 Feb 2011 21:04:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28790 invoked by uid 500); 11 Feb 2011 21:04:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28780 invoked by uid 99); 11 Feb 2011 21:04:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 21:04:28 +0000
X-ASF-Spam-Status: No, hits=3.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of akimball83@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 21:04:21 +0000
Received: by bwz8 with SMTP id 8so3824458bwz.35
        for <general@hadoop.apache.org>; Fri, 11 Feb 2011 13:04:01 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type;
        bh=+l3f6l//mBHlp2njVtXVJvbF9YdO61n0d+r0c1o16Q4=;
        b=JFlUNjFblPEpI7w9i+92tJFlNW1URkg+6AqxFbHsXC35LbL6uR7yW4jcn3sTa8Wv5b
         bAHtnmHose1DdGLNodhAYixuuH0zd6Rl5Gkzn7qQddjD6yjB1kvbBBA6XWoSWC3NDQOu
         Py27MghAszquM/Zp8FNgVYM3Ze7CyWowLQsFE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=CM4EBzGDIhDm4xhUkuHxEpV9B6Zf4AoXA608di0UrjowkEfeX32qPmRuksBYu8HimH
         namWEBPTUGyj1GyeStOUGtDf/kBpV/N9p/I40dgCTg0T4WYSuNljAFu0+t7TxkVZsOxM
         CBMmywiwWFDbMPEwDEB5eBMkth3sUP2waSeec=
Received: by 10.204.117.77 with SMTP id p13mr927512bkq.19.1297458241223; Fri,
 11 Feb 2011 13:04:01 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.54.72 with HTTP; Fri, 11 Feb 2011 13:03:41 -0800 (PST)
In-Reply-To: <AANLkTi=HQ1cn++JO6g+vp-cJAtAjB00Ok-FW2VrfTiSO@mail.gmail.com>
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com> <AANLkTi=75e6oyBuGVivXP3is+76gOODSc1NcLKfg=VxF@mail.gmail.com>
 <AANLkTi=HQ1cn++JO6g+vp-cJAtAjB00Ok-FW2VrfTiSO@mail.gmail.com>
From: Aaron Kimball <akimball83@gmail.com>
Date: Fri, 11 Feb 2011 13:03:41 -0800
Message-ID: <AANLkTimitemnV+qfNx+T0BAwZJx9qvaH8r1RMYL27hHu@mail.gmail.com>
Subject: Re: [DISCUSS] Move common, hdfs, mapreduce contrib components to
 apache-extras.org or elsewhere
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d647db943c46049c0809ec
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d647db943c46049c0809ec
Content-Type: text/plain; charset=ISO-8859-1

Tom,

How do these contrib components get released then? If the intent of having
the code is to eventually produce release artifacts that people can use,
then allowing them to further degrade in releasability seems antithetical to
the point of keeping the source around. I think users who download Hadoop
would be surprised to find that tools/projects under contrib do not operate
as advertised.

If the point of retaining the code would be instead to have it as an example
for future developers to reference, then maybe it would be better to move
them into an "attic" or "unmaintaned" directory so it is clear that we do
not expect these to stay up to date. If we do expect them to work, and
believe that they belong in the project, then I think we need to keep them
building -- and yes, that adds to the burden associated with the release of
Hadoop as a whole.

- Aaron

On Fri, Feb 11, 2011 at 10:12 AM, Tom White <tom.e.white@gmail.com> wrote:

> For contrib components that we elect not to remove, I suggest that we
> remove them from the main build, so that failures in a contrib
> component don't hinder the main build and release. This is what
> ZooKeeper does, for example.
>
> Tom
>
> On Fri, Feb 11, 2011 at 2:03 AM, Bernd Fondermann
> <bernd.fondermann@googlemail.com> wrote:
> > -1.
> >
> > Move it away from TRUNK so it doesn't affect builds is a much better
> > (and simpler) solution. If someone wants to revive it, he can within
> > the bounds of Apache Hadoop and will become a part of the Hadoop
> > community, which would be good.
> > If you'd move it off-site, if the code ever gets new contributors,
> > it's hard to integrate them (code and contributors) into Hadoop again.
> >
> > AFAIUI, apache-extras is for placing non-Apache code closer to the
> > related Apache projects, not for moving our code away from our own
> > premises.
> >
> >  Bernd
> >
> > On Mon, Jan 31, 2011 at 04:42, Nigel Daley <ndaley@mac.com> wrote:
> >> Folks,
> >>
> >> Now that http://apache-extras.org is launched (
> https://blogs.apache.org/foundation/entry/the_apache_software_foundation_launches)
> I'd like to start a discussion on moving contrib components out of common,
> mapreduce, and hdfs.
> >>
> >> These contrib components complicate the builds, cause test failures that
> nobody seems to care about, have releases that are tied to Hadoop's long
> release cycles, etc.  Most folks I've talked with agree that these contrib
> components would be better served by being pulled out of Hadoop and hosted
> elsewhere. The new apache-extras code hosting site seems like a natural
> *default* location for migrating these contrib projects.  Perhaps some
> should graduate from contrib to src (ie from contrib to core of the project
> they're included in).  If folks agree, we'll need to come up with a mapping
> of contrib component to it's final destination and file a jira.
> >>
> >> Here are the contrib components by project (hopefully I didn't miss
> any).
> >>
> >> Common Contrib:
> >>  failmon
> >>  hod
> >>  test
> >>
> >>
> >> MapReduce Contrib:
> >>  capacity-scheduler -- move to MR core?
> >>  data_join
> >>  dynamic-scheduler
> >>  eclipse-plugin
> >>  fairscheduler -- move to MR core?
> >>  gridmix
> >>  index
> >>  mrunit
> >>  mumak
> >>  raid
> >>  sqoop
> >>  streaming -- move to MR core?
> >>  vaidya
> >>  vertica
> >>
> >>
> >> HDFS Contrib:
> >>  fuse-dfs
> >>  hdfsproxy
> >>  thriftfs
> >>
> >>
> >> Cheers,
> >> Nige
> >>
> >
>

--0016e6d647db943c46049c0809ec--

From general-return-3026-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 22:10:59 2011
Return-Path: <general-return-3026-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88934 invoked from network); 11 Feb 2011 22:10:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 22:10:59 -0000
Received: (qmail 8354 invoked by uid 500); 11 Feb 2011 22:10:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8298 invoked by uid 500); 11 Feb 2011 22:10:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8290 invoked by uid 99); 11 Feb 2011 22:10:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 22:10:56 +0000
X-ASF-Spam-Status: No, hits=3.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of akimball83@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 22:10:50 +0000
Received: by bwz8 with SMTP id 8so3880588bwz.35
        for <general@hadoop.apache.org>; Fri, 11 Feb 2011 14:10:26 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type;
        bh=UW0NPCWCGV6W+aqwhKyImfO0NzbSP4kstpf3kHpvxdA=;
        b=AvmQRRhVuYc4zUJGAmeqZb5/pq28D0yPXDN4G/AsxOY8ooXwFuEWIXcSQt78/vH8CE
         oKkbxVMeIzN3s0kjDwN1Z1HfJravf7f3FJQyrqwiD2otpC/pv1+H8swAuS+55dbpyDBq
         GbGUVREKsPZR7AFCvN+DfwElPVdEfOv1/OipY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=Yt9ZDpURG/9NnZFfOHgQGpB6DiMIn6gyMsJBO7erz9uawn9qefLfgVDyQ5fv2cgz+D
         lTdUL+g38ersq/CbIenXj4QG7NS164cnIp+hOXNhFh2FZGDB0wPMKO9/o7Fap/JJditG
         eVVM8+gKFLnVfeBNcH0DRasNQzwinDQEGNqT4=
Received: by 10.204.82.84 with SMTP id a20mr11415865bkl.154.1297462225809;
 Fri, 11 Feb 2011 14:10:25 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.54.72 with HTTP; Fri, 11 Feb 2011 14:10:04 -0800 (PST)
In-Reply-To: <FAE19BAA-47A7-4747-BDD8-5AA42C259EB6@jpl.nasa.gov>
References: <77405974-6771-4604-926B-976240743F3C@mac.com> <AANLkTim80KJ3f14iM6Ltzy7UG613myhN-AiwQ3pCmVdi@mail.gmail.com>
 <AANLkTiknkrWK=GzGMVj4_Z6p5Jn8TQu1P5TkaFSgORJk@mail.gmail.com>
 <AANLkTin0TguLYHi9X0uL3ZF7tCz5xTHBYa7056oKF2Ae@mail.gmail.com>
 <93644D3B-4830-4813-92BE-809738969CF9@apache.org> <AANLkTi=dbGSLhtFA+Rf9LgaX05v45HDLoQDE_2C5ULAf@mail.gmail.com>
 <65311075-A30E-4971-A2BA-C75409C6486E@jpl.nasa.gov> <AANLkTi=XbpVUmg_TpH7eqEn8w7Gcu9KiwtJmKMLa5w5n@mail.gmail.com>
 <FAE19BAA-47A7-4747-BDD8-5AA42C259EB6@jpl.nasa.gov>
From: Aaron Kimball <akimball83@gmail.com>
Date: Fri, 11 Feb 2011 14:10:04 -0800
Message-ID: <AANLkTin2JvOGdZT5g5tmTGeCpD3Qz2soZWcTQtHMsSza@mail.gmail.com>
Subject: Re: [VOTE] Abandon mrunit MapReduce contrib
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d7e355143344049c08f71a
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d7e355143344049c08f71a
Content-Type: text/plain; charset=ISO-8859-1

The main reason I am interested in removing MRUnit from Hadoop is that I
believe that MRUnit deserves its own release cycle. I think this is in the
best interest of its users.

MRUnit is valuable to users of several different versions of Hadoop. But
MRUnit has only ever been committed to version 0.21 and above -- even though
in practice, the majority (dare I say--all) of its users are running on
0.20. The only place today to get a version of MRUnit compatible with 0.20
has been through a Cloudera release, which backported the entire MRUnit
patchset.

My thoughts on MRUnit in 0.20.100 resonate with Eric's. There will be
further fixes to MRUnit and its lightweight codebase can be released far
more rapidly than whenever the next 0.20.1xx release of Hadoop would occur.
Given that MRUnit has already been in the repository since April 2009 (see
https://issues.apache.org/jira/browse/HADOOP-5518) and has yet to see an
Apache 0.20-based release, I do not think it is in the best interest of the
library's userbase to couple MRUnit's release cycle to that of Hadoop
itself.

Perhaps more importantly, access to new features in MRUnit should not
require upgrading one's entire Hadoop deployment; this is a client library
that depends only on Hadoop's public APIs.

My primary concern is to move MRUnit to a place where the community can
derive the most benefit from it. The Apache Incubator could fulfill this
role; given the presence of individuals willing to mentor this project, I
believe this would be a successful way to release MRUnit more quickly and
continue to work to grow the MRUnit community.

Regards,
- Aaron


On Fri, Feb 11, 2011 at 11:57 AM, Mattmann, Chris A (388J) <
chris.a.mattmann@jpl.nasa.gov> wrote:

> Awesome Patrick, we'd probably need one more active mentor. Any takers?
>
> After we get that, then we cook up a proposal on the Incubator wiki here
> [1], and follow the process here [2] to get started...
>
> Cheers,
> Chris
>
> [1] http://wiki.apache.org/incubator/MRUnitProposal
> [2] http://incubator.apache.org/guides/proposal.html
>
> On Feb 11, 2011, at 11:52 AM, Patrick Hunt wrote:
>
> > On Fri, Feb 11, 2011 at 9:44 AM, Mattmann, Chris A (388J)
> > <chris.a.mattmann@jpl.nasa.gov> wrote:
> >> Guys, BTW, if you need help or a mentor in Apache Incubator-ville for
> MRUnit, I would be happy to help.
> >
> > I was going to suggest the same thing (mrunit to incubator). I would
> > also be happy to be a mentor.
> >
> > Patrick
> >
> >>
> >> On Feb 11, 2011, at 9:04 AM, Eric Sammer wrote:
> >>
> >>> On Fri, Feb 11, 2011 at 11:48 AM, Owen O'Malley <omalley@apache.org>
> wrote:
> >>>
> >>>> On Feb 11, 2011, at 8:02 AM, Eric Sammer wrote:
> >>>>
> >>>> - allow mrunit to have its own release cycle. This is, I think, the
> most
> >>>>>
> >>>>
> >>>> important.
> >>>>>
> >>>>
> >>>> If you submit your work to Apache we can evaluate it for inclusion in
> the
> >>>> 0.20.100 branch to get your changes released in a timely manner.
> >>>
> >>>
> >>> I'm thinking in general (beyond the next immediate release).
> Independent of
> >>> where mrunit goes, I think it should leave the contrib tree to
> facilitate
> >>> light weight releases (the dependency on Hadoop proper is a public
> facing
> >>> API - a pure client). I think most projects could benefit from this
> with the
> >>> exception of things that are tightly coupled to Hadoop releases or
> touch
> >>> non-public APIs.
> >>>
> >>>
> >>>> I would actually prefer to move it to Extras or Incubator and leave
> this
> >>>>> within the ASF.
> >>>>>
> >>>>
> >>>> Extras is **NOT** inside of the ASF. Extras is a source hosting system
> for
> >>>> non-Apache projects that are related to Apache projects.
> >>>
> >>>
> >>> Got it. Thanks for correcting me. I only mentioned it because someone
> >>> suggested it to me initially.
> >>>
> >>>
> >>>> Right now, I picked github because of the ability to easily
> >>>> collaborate with others (and to use git).
> >>>>
> >>>
> >>> I agree that it is unfortunate that Apache doesn't yet support
> read-write
> >>>> git access. However, you'll find that building a community is easier
> at
> >>>> Apache than at github.
> >>>>
> >>>
> >>>> -- Owen
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Eric Sammer
> >>> twitter: esammer
> >>> data: www.cloudera.com
> >>
> >>
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> Chris Mattmann, Ph.D.
> >> Senior Computer Scientist
> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >> Office: 171-266B, Mailstop: 171-246
> >> Email: chris.a.mattmann@nasa.gov
> >> WWW:   http://sunset.usc.edu/~mattmann/
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> Adjunct Assistant Professor, Computer Science Department
> >> University of Southern California, Los Angeles, CA 90089 USA
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>
> >>
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW:   http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>

--0016e6d7e355143344049c08f71a--

From general-return-3027-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 22:27:19 2011
Return-Path: <general-return-3027-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98147 invoked from network); 11 Feb 2011 22:27:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 22:27:19 -0000
Received: (qmail 27141 invoked by uid 500); 11 Feb 2011 22:27:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26556 invoked by uid 500); 11 Feb 2011 22:27:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26548 invoked by uid 99); 11 Feb 2011 22:27:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 22:27:14 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of esammer@cloudera.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 22:27:10 +0000
Received: by vws18 with SMTP id 18so1935168vws.35
        for <general@hadoop.apache.org>; Fri, 11 Feb 2011 14:26:49 -0800 (PST)
MIME-Version: 1.0
Received: by 10.220.73.194 with SMTP id r2mr1316889vcj.267.1297463208731; Fri,
 11 Feb 2011 14:26:48 -0800 (PST)
Received: by 10.220.5.80 with HTTP; Fri, 11 Feb 2011 14:26:48 -0800 (PST)
In-Reply-To: <AANLkTin2JvOGdZT5g5tmTGeCpD3Qz2soZWcTQtHMsSza@mail.gmail.com>
References: <77405974-6771-4604-926B-976240743F3C@mac.com>
	<AANLkTim80KJ3f14iM6Ltzy7UG613myhN-AiwQ3pCmVdi@mail.gmail.com>
	<AANLkTiknkrWK=GzGMVj4_Z6p5Jn8TQu1P5TkaFSgORJk@mail.gmail.com>
	<AANLkTin0TguLYHi9X0uL3ZF7tCz5xTHBYa7056oKF2Ae@mail.gmail.com>
	<93644D3B-4830-4813-92BE-809738969CF9@apache.org>
	<AANLkTi=dbGSLhtFA+Rf9LgaX05v45HDLoQDE_2C5ULAf@mail.gmail.com>
	<65311075-A30E-4971-A2BA-C75409C6486E@jpl.nasa.gov>
	<AANLkTi=XbpVUmg_TpH7eqEn8w7Gcu9KiwtJmKMLa5w5n@mail.gmail.com>
	<FAE19BAA-47A7-4747-BDD8-5AA42C259EB6@jpl.nasa.gov>
	<AANLkTin2JvOGdZT5g5tmTGeCpD3Qz2soZWcTQtHMsSza@mail.gmail.com>
Date: Fri, 11 Feb 2011 17:26:48 -0500
Message-ID: <AANLkTim+FQuoS+mR7YuTf0kw1KVxCdy14wDjf2u_cZdB@mail.gmail.com>
Subject: Re: [VOTE] Abandon mrunit MapReduce contrib
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016363107b3aa6c75049c09315b

--0016363107b3aa6c75049c09315b
Content-Type: text/plain; charset=ISO-8859-1

Just to add to the option of going to incubator, I'm fine with that as well.
Github was an easy thing to get started and I was under the impression we
needed some greater degree of committer diversity and, frankly, a bigger
project. If mrunit is a candidate, keeping this under the ASF umbrella is
more than fine with me.

On Fri, Feb 11, 2011 at 5:10 PM, Aaron Kimball <akimball83@gmail.com> wrote:

> The main reason I am interested in removing MRUnit from Hadoop is that I
> believe that MRUnit deserves its own release cycle. I think this is in the
> best interest of its users.
>
> MRUnit is valuable to users of several different versions of Hadoop. But
> MRUnit has only ever been committed to version 0.21 and above -- even
> though
> in practice, the majority (dare I say--all) of its users are running on
> 0.20. The only place today to get a version of MRUnit compatible with 0.20
> has been through a Cloudera release, which backported the entire MRUnit
> patchset.
>
> My thoughts on MRUnit in 0.20.100 resonate with Eric's. There will be
> further fixes to MRUnit and its lightweight codebase can be released far
> more rapidly than whenever the next 0.20.1xx release of Hadoop would occur.
> Given that MRUnit has already been in the repository since April 2009 (see
> https://issues.apache.org/jira/browse/HADOOP-5518) and has yet to see an
> Apache 0.20-based release, I do not think it is in the best interest of the
> library's userbase to couple MRUnit's release cycle to that of Hadoop
> itself.
>
> Perhaps more importantly, access to new features in MRUnit should not
> require upgrading one's entire Hadoop deployment; this is a client library
> that depends only on Hadoop's public APIs.
>
> My primary concern is to move MRUnit to a place where the community can
> derive the most benefit from it. The Apache Incubator could fulfill this
> role; given the presence of individuals willing to mentor this project, I
> believe this would be a successful way to release MRUnit more quickly and
> continue to work to grow the MRUnit community.
>
> Regards,
> - Aaron
>
>
> On Fri, Feb 11, 2011 at 11:57 AM, Mattmann, Chris A (388J) <
> chris.a.mattmann@jpl.nasa.gov> wrote:
>
> > Awesome Patrick, we'd probably need one more active mentor. Any takers?
> >
> > After we get that, then we cook up a proposal on the Incubator wiki here
> > [1], and follow the process here [2] to get started...
> >
> > Cheers,
> > Chris
> >
> > [1] http://wiki.apache.org/incubator/MRUnitProposal
> > [2] http://incubator.apache.org/guides/proposal.html
> >
> > On Feb 11, 2011, at 11:52 AM, Patrick Hunt wrote:
> >
> > > On Fri, Feb 11, 2011 at 9:44 AM, Mattmann, Chris A (388J)
> > > <chris.a.mattmann@jpl.nasa.gov> wrote:
> > >> Guys, BTW, if you need help or a mentor in Apache Incubator-ville for
> > MRUnit, I would be happy to help.
> > >
> > > I was going to suggest the same thing (mrunit to incubator). I would
> > > also be happy to be a mentor.
> > >
> > > Patrick
> > >
> > >>
> > >> On Feb 11, 2011, at 9:04 AM, Eric Sammer wrote:
> > >>
> > >>> On Fri, Feb 11, 2011 at 11:48 AM, Owen O'Malley <omalley@apache.org>
> > wrote:
> > >>>
> > >>>> On Feb 11, 2011, at 8:02 AM, Eric Sammer wrote:
> > >>>>
> > >>>> - allow mrunit to have its own release cycle. This is, I think, the
> > most
> > >>>>>
> > >>>>
> > >>>> important.
> > >>>>>
> > >>>>
> > >>>> If you submit your work to Apache we can evaluate it for inclusion
> in
> > the
> > >>>> 0.20.100 branch to get your changes released in a timely manner.
> > >>>
> > >>>
> > >>> I'm thinking in general (beyond the next immediate release).
> > Independent of
> > >>> where mrunit goes, I think it should leave the contrib tree to
> > facilitate
> > >>> light weight releases (the dependency on Hadoop proper is a public
> > facing
> > >>> API - a pure client). I think most projects could benefit from this
> > with the
> > >>> exception of things that are tightly coupled to Hadoop releases or
> > touch
> > >>> non-public APIs.
> > >>>
> > >>>
> > >>>> I would actually prefer to move it to Extras or Incubator and leave
> > this
> > >>>>> within the ASF.
> > >>>>>
> > >>>>
> > >>>> Extras is **NOT** inside of the ASF. Extras is a source hosting
> system
> > for
> > >>>> non-Apache projects that are related to Apache projects.
> > >>>
> > >>>
> > >>> Got it. Thanks for correcting me. I only mentioned it because someone
> > >>> suggested it to me initially.
> > >>>
> > >>>
> > >>>> Right now, I picked github because of the ability to easily
> > >>>> collaborate with others (and to use git).
> > >>>>
> > >>>
> > >>> I agree that it is unfortunate that Apache doesn't yet support
> > read-write
> > >>>> git access. However, you'll find that building a community is easier
> > at
> > >>>> Apache than at github.
> > >>>>
> > >>>
> > >>>> -- Owen
> > >>>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Eric Sammer
> > >>> twitter: esammer
> > >>> data: www.cloudera.com
> > >>
> > >>
> > >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >> Chris Mattmann, Ph.D.
> > >> Senior Computer Scientist
> > >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > >> Office: 171-266B, Mailstop: 171-246
> > >> Email: chris.a.mattmann@nasa.gov
> > >> WWW:   http://sunset.usc.edu/~mattmann/
> > >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >> Adjunct Assistant Professor, Computer Science Department
> > >> University of Southern California, Los Angeles, CA 90089 USA
> > >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >>
> > >>
> >
> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Chris Mattmann, Ph.D.
> > Senior Computer Scientist
> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > Office: 171-266B, Mailstop: 171-246
> > Email: chris.a.mattmann@nasa.gov
> > WWW:   http://sunset.usc.edu/~mattmann/
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Adjunct Assistant Professor, Computer Science Department
> > University of Southern California, Los Angeles, CA 90089 USA
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >
> >
>



-- 
Eric Sammer
twitter: esammer
data: www.cloudera.com

--0016363107b3aa6c75049c09315b--

From general-return-3028-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 22:44:24 2011
Return-Path: <general-return-3028-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 2627 invoked from network); 11 Feb 2011 22:44:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 22:44:24 -0000
Received: (qmail 50169 invoked by uid 500); 11 Feb 2011 22:44:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50037 invoked by uid 500); 11 Feb 2011 22:44:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50019 invoked by uid 99); 11 Feb 2011 22:44:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 22:44:22 +0000
X-ASF-Spam-Status: No, hits=3.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of akimball83@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 22:44:15 +0000
Received: by bwz8 with SMTP id 8so3906109bwz.35
        for <multiple recipients>; Fri, 11 Feb 2011 14:43:55 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:from:date:message-id:subject:to
         :content-type;
        bh=75PvP40dK7zS4bE+7DlElkdiSyZwA+uHcZIzdMK4mY4=;
        b=Xb+JA3XJYdOoKlyVlrdfxeBJmOtZiP99bNu4CcPTIIVK1mkQcvpv1ukNmzThI3bOjg
         M+G3WG/2UsM6hB9iGsHMrXzI7clEINfpkCFUKM/8XKKpef5nBp7suHiqxkwr+/muJH1G
         CAmhxyN7jjmAl12pBdNym3RROu1MNjXT7M2EA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type;
        b=rO4Gd0kSpG3yWJlYlQ7Y92JG2/SgHhR3wZqpnq0btKS18oxovGq7DjX7Q2etPA1O5s
         +z/C+U9EZIBs8VQOJqUwWoCq3gK/gkXDLCj4c8C9SIP3iC7XdhKLITewVNUqAkEz5Kgz
         RZbfZBHNQgc8+mm9a5grKDodol6+OM7pzPs0E=
Received: by 10.204.59.72 with SMTP id k8mr2377489bkh.208.1297464235133; Fri,
 11 Feb 2011 14:43:55 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.54.72 with HTTP; Fri, 11 Feb 2011 14:43:35 -0800 (PST)
From: Aaron Kimball <akimball83@gmail.com>
Date: Fri, 11 Feb 2011 14:43:35 -0800
Message-ID: <AANLkTi=96WcZuUYTM=S8wWUoQ27vg6Sk+Y6fSD=fzq4k@mail.gmail.com>
Subject: SF Hadoop meetup report
To: general@hadoop.apache.org, core-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c5b174d80dd8049c096ec7
X-Virus-Checked: Checked by ClamAV on apache.org

--001636c5b174d80dd8049c096ec7
Content-Type: text/plain; charset=ISO-8859-1

Hadoop fans,

This week we held the second SF Hadoop meetup. About 70 individuals
attended, and we again had several great discussions in parallel, using the
same unconference format as the first meetup in January.

The following break-out sessions were held:
* Hadoop and Workflow Systems
* Feeding Hadoop
* Contributing to Hadoop and Related Projects
* Hive Q&A
* Hadoop + RDBMS
* Hadoop and Predictive Analytics
* Hadoop in Amazon
* HBase: What is the sweet spot?
* Hardware for Hadoop
* Hadoop Security

For some discussions, summary notes are available; these will be posted at
http://www.meetup.com/hadoopsf/messages/boards/

Thanks again to our hosts at CBS Interactive for providing the space for
this meetup. I would also like to thank the discussion facilitators, and all
the attendees for contributing to the success of the discussions.

Stay tuned for the next SF Hadoop meetup announcement! Sign up at
http://www.meetup.com/hadoopsf/

Regards,
- Aaron Kimball

--001636c5b174d80dd8049c096ec7--

From general-return-3029-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 11 22:57:00 2011
Return-Path: <general-return-3029-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5531 invoked from network); 11 Feb 2011 22:56:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2011 22:56:59 -0000
Received: (qmail 66245 invoked by uid 500); 11 Feb 2011 22:56:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66208 invoked by uid 500); 11 Feb 2011 22:56:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66200 invoked by uid 99); 11 Feb 2011 22:56:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 22:56:57 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.59.243] (HELO qmta13.westchester.pa.mail.comcast.net) (76.96.59.243)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Feb 2011 22:56:49 +0000
Received: from omta15.westchester.pa.mail.comcast.net ([76.96.62.87])
	by qmta13.westchester.pa.mail.comcast.net with comcast
	id 6mYZ1g0031swQuc5DmwVNf; Fri, 11 Feb 2011 22:56:29 +0000
Received: from [10.72.184.42] ([209.131.62.113])
	by omta15.westchester.pa.mail.comcast.net with comcast
	id 6mwK1g01m2SbwD53bmwNqR; Fri, 11 Feb 2011 22:56:27 +0000
Message-Id: <354259E0-9D22-47B5-B247-4187182B60DF@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <55BA39F5-5017-42B1-BE25-A643CCEADC75@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [ANNOUNCEMENT] Yahoo focusing on Apache Hadoop, discontinuing "The Yahoo Distribution of Hadoop"
Date: Fri, 11 Feb 2011 14:56:18 -0800
References: <55BA39F5-5017-42B1-BE25-A643CCEADC75@yahoo-inc.com>
X-Mailer: Apple Mail (2.936)


On Jan 31, 2011, at 7:27 PM, Eric Baldeschwieler wrote:

> Unfortunately, we now have to sort out how to contribute several  
> person-years worth of work to Apache to let us unwind the Yahoo! git  
> repositories.  We currently run two lines of Hadoop development, our  
> sustaining program (hadoop-0.20-sustaining) and hadoop-future.

As Eric mentioned, we have several person years worth of development  
to contribute to Apache. Arun has started that process by creating the  
branch-0.20-security branch. I plan on pushing the individual patches  
to branch-0.20-security-patches and then when it is identical with  
Arun's branch, I'll rename mine to branch-0.20-security.

I also plan to start pushing the hadoop-future work into a branch  
called yahoo-merge(?)  as individual commits from our internal git  
repository. The goal of creating the branch is to enable faster review  
and discussion. These patches will be individually run through the  
jira, review, and commit process to be added to trunk.

-- Owen

From general-return-3030-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Feb 12 00:56:47 2011
Return-Path: <general-return-3030-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60218 invoked from network); 12 Feb 2011 00:56:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Feb 2011 00:56:40 -0000
Received: (qmail 75890 invoked by uid 500); 12 Feb 2011 00:56:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75828 invoked by uid 500); 12 Feb 2011 00:56:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75820 invoked by uid 99); 12 Feb 2011 00:56:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Feb 2011 00:56:38 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of tom.e.white@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Feb 2011 00:56:33 +0000
Received: by wwd20 with SMTP id 20so3059623wwd.29
        for <general@hadoop.apache.org>; Fri, 11 Feb 2011 16:56:12 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type:content-transfer-encoding;
        bh=54RbqBJnNbB7VM+Pgp5YMHIiMZZTC7/xNbbbpvgak08=;
        b=RjL3KUVkXzLNGCNhEd+dkw1xDfHW23sJFjSD2wyB7hO8iuYjgM8pcoW2ym0d6ufzWR
         D04uuLWF04b5HfzzUhshlt7STEsU8NAb89p+yIfb808U1jmHmiE88eVxiMcI7igZKzHP
         RP/GNh0PG9tipPrZDoVNJVKm9c54DNLuDZyeg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=srWyRse5h+QNecF1fQjN3MqxzDnjMQLMw5ovdUogRUew0ZUnHD/8rExHIm49xPP1Nt
         V3rCfEExgzhU4En6DlYhf9CbNYRaDhCAfIOT5PUgeXi3q+8TTV+RKzGCFKSpCStzAz0w
         Agdxfi7rr75wJZDeSoysRdlXZVsVJgw91TMsI=
Received: by 10.216.173.147 with SMTP id v19mr1290602wel.102.1297472171472;
 Fri, 11 Feb 2011 16:56:11 -0800 (PST)
MIME-Version: 1.0
Received: by 10.216.159.193 with HTTP; Fri, 11 Feb 2011 16:55:51 -0800 (PST)
In-Reply-To: <AANLkTimitemnV+qfNx+T0BAwZJx9qvaH8r1RMYL27hHu@mail.gmail.com>
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com> <AANLkTi=75e6oyBuGVivXP3is+76gOODSc1NcLKfg=VxF@mail.gmail.com>
 <AANLkTi=HQ1cn++JO6g+vp-cJAtAjB00Ok-FW2VrfTiSO@mail.gmail.com> <AANLkTimitemnV+qfNx+T0BAwZJx9qvaH8r1RMYL27hHu@mail.gmail.com>
From: Tom White <tom.e.white@gmail.com>
Date: Fri, 11 Feb 2011 16:55:51 -0800
Message-ID: <AANLkTik0wsLWmmuiMxSLG7u8QYJi09VXKmZFcJvEzbQM@mail.gmail.com>
Subject: Re: [DISCUSS] Move common, hdfs, mapreduce contrib components to
 apache-extras.org or elsewhere
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Fri, Feb 11, 2011 at 1:03 PM, Aaron Kimball <akimball83@gmail.com> wrote=
:
> Tom,
>
> How do these contrib components get released then?

In source form - this is what ZooKeeper does.

> If the intent of having
> the code is to eventually produce release artifacts that people can use,
> then allowing them to further degrade in releasability seems antithetical=
 to
> the point of keeping the source around. I think users who download Hadoop
> would be surprised to find that tools/projects under contrib do not opera=
te
> as advertised.
>
> If the point of retaining the code would be instead to have it as an exam=
ple
> for future developers to reference, then maybe it would be better to move
> them into an "attic" or "unmaintaned" directory so it is clear that we do
> not expect these to stay up to date. If we do expect them to work, and
> believe that they belong in the project, then I think we need to keep the=
m
> building -- and yes, that adds to the burden associated with the release =
of
> Hadoop as a whole.

Maybe. We haven't always done a great job of this though.

>
> - Aaron
>
> On Fri, Feb 11, 2011 at 10:12 AM, Tom White <tom.e.white@gmail.com> wrote=
:
>
>> For contrib components that we elect not to remove, I suggest that we
>> remove them from the main build, so that failures in a contrib
>> component don't hinder the main build and release. This is what
>> ZooKeeper does, for example.
>>
>> Tom
>>
>> On Fri, Feb 11, 2011 at 2:03 AM, Bernd Fondermann
>> <bernd.fondermann@googlemail.com> wrote:
>> > -1.
>> >
>> > Move it away from TRUNK so it doesn't affect builds is a much better
>> > (and simpler) solution. If someone wants to revive it, he can within
>> > the bounds of Apache Hadoop and will become a part of the Hadoop
>> > community, which would be good.
>> > If you'd move it off-site, if the code ever gets new contributors,
>> > it's hard to integrate them (code and contributors) into Hadoop again.
>> >
>> > AFAIUI, apache-extras is for placing non-Apache code closer to the
>> > related Apache projects, not for moving our code away from our own
>> > premises.
>> >
>> > =A0Bernd
>> >
>> > On Mon, Jan 31, 2011 at 04:42, Nigel Daley <ndaley@mac.com> wrote:
>> >> Folks,
>> >>
>> >> Now that http://apache-extras.org is launched (
>> https://blogs.apache.org/foundation/entry/the_apache_software_foundation=
_launches)
>> I'd like to start a discussion on moving contrib components out of commo=
n,
>> mapreduce, and hdfs.
>> >>
>> >> These contrib components complicate the builds, cause test failures t=
hat
>> nobody seems to care about, have releases that are tied to Hadoop's long
>> release cycles, etc. =A0Most folks I've talked with agree that these con=
trib
>> components would be better served by being pulled out of Hadoop and host=
ed
>> elsewhere. The new apache-extras code hosting site seems like a natural
>> *default* location for migrating these contrib projects. =A0Perhaps some
>> should graduate from contrib to src (ie from contrib to core of the proj=
ect
>> they're included in). =A0If folks agree, we'll need to come up with a ma=
pping
>> of contrib component to it's final destination and file a jira.
>> >>
>> >> Here are the contrib components by project (hopefully I didn't miss
>> any).
>> >>
>> >> Common Contrib:
>> >> =A0failmon
>> >> =A0hod
>> >> =A0test
>> >>
>> >>
>> >> MapReduce Contrib:
>> >> =A0capacity-scheduler -- move to MR core?
>> >> =A0data_join
>> >> =A0dynamic-scheduler
>> >> =A0eclipse-plugin
>> >> =A0fairscheduler -- move to MR core?
>> >> =A0gridmix
>> >> =A0index
>> >> =A0mrunit
>> >> =A0mumak
>> >> =A0raid
>> >> =A0sqoop
>> >> =A0streaming -- move to MR core?
>> >> =A0vaidya
>> >> =A0vertica
>> >>
>> >>
>> >> HDFS Contrib:
>> >> =A0fuse-dfs
>> >> =A0hdfsproxy
>> >> =A0thriftfs
>> >>
>> >>
>> >> Cheers,
>> >> Nige
>> >>
>> >
>>
>

From general-return-3031-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Feb 12 02:11:29 2011
Return-Path: <general-return-3031-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95035 invoked from network); 12 Feb 2011 02:11:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Feb 2011 02:11:28 -0000
Received: (qmail 37531 invoked by uid 500); 12 Feb 2011 02:11:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37420 invoked by uid 500); 12 Feb 2011 02:11:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37412 invoked by uid 99); 12 Feb 2011 02:11:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Feb 2011 02:11:26 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.99 as permitted sender)
Received: from [17.148.16.99] (HELO asmtpout024.mac.com) (17.148.16.99)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Feb 2011 02:11:17 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp024.mac.com (Oracle Communications Messaging Exchange Server 7u4-18.01
 64bit (built Jul 15 2010)) with ESMTPSA id <0LGH00LKGFDV0300@asmtp024.mac.com>
 for general@hadoop.apache.org; Fri, 11 Feb 2011 18:10:44 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-02-11_09:2011-02-11,2011-02-11,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1102110167
Subject: Re: [VOTE] Abandon failmon Common contrib
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <C98FD122-F11B-4A3F-8D34-3C547392F881@mac.com>
Date: Fri, 11 Feb 2011 18:10:43 -0800
Message-id: <57A8F207-B566-48FB-8D84-E115C593DED2@mac.com>
References: <C98FD122-F11B-4A3F-8D34-3C547392F881@mac.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

This component has been removed with Jira HADOOP-7136.

Thanks,
Nige

On Feb 9, 2011, at 9:18 AM, Nigel Daley wrote:

> I think the PMC should abandon the failmon common contrib component.  It's last meaningful contribution was it's original commit in August of 2008:
> 
> HADOOP-3585. FailMon package for hardware failure monitoring and analysis of anomalies. (Ioannis Koltsidas via dhruba)
> 
> Here is my +1.
> 
> Nige


From general-return-3032-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Feb 12 02:18:52 2011
Return-Path: <general-return-3032-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95810 invoked from network); 12 Feb 2011 02:18:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Feb 2011 02:18:52 -0000
Received: (qmail 41170 invoked by uid 500); 12 Feb 2011 02:18:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40961 invoked by uid 500); 12 Feb 2011 02:18:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40953 invoked by uid 99); 12 Feb 2011 02:18:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Feb 2011 02:18:50 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.95 as permitted sender)
Received: from [17.148.16.95] (HELO asmtpout020.mac.com) (17.148.16.95)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Feb 2011 02:18:42 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp020.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LGH005IIFPRUP70@asmtp020.mac.com>
 for general@hadoop.apache.org; Fri, 11 Feb 2011 18:17:51 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-02-11_09:2011-02-11,2011-02-11,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1102110168
Subject: Re: [VOTE] Abandon hod Common contrib
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <84B97117-5DA4-4A32-912A-8DBFCE3D063E@linkedin.com>
Date: Fri, 11 Feb 2011 18:17:50 -0800
Message-id: <535D05CA-3321-48BA-82C4-C18FC8983C3A@mac.com>
References: <AC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com>
 <84B97117-5DA4-4A32-912A-8DBFCE3D063E@linkedin.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)


On Feb 11, 2011, at 9:48 AM, Allen Wittenauer wrote:

> 
> On Feb 10, 2011, at 6:51 PM, Nigel Daley wrote:
> 
>> I think the PMC should abandon the hod common contrib component.  It's last meaningful contribution was February 2009:
>> 
>> HADOOP-2898. Provide an option to specify a port range for Hadoop services provisioned by HOD. Contributed by Peeyush Bishnoi.
>> 
>> There are 44 unresolved contrib/hod issues in Jira, 1 of them Patch Available since November 2009 (HADOOP-6369).
> 
> -1
> 
> a) I don't think hod is actually part of any unit tests, so including it would likely only be a burden on the tarball size.

Not true.  HOD has python unit tests and is the reason our builds have dependencies on python.

> b) The edu community uses this quite extensively, evidenced by the topic coming up on the mailing lists at least once every two months or so and has for years.  Can't say that about the other contrib modules other than the schedulers and streaming.

Then they are using old version of Hadoop.  AFAICT HOD does not work with 0.20 or beyond.

> c) The community that does use it has even submitted a patch that we've ignored.

Which means the committers of this project gave up on it long ago.

Nige

From general-return-3033-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Feb 12 03:28:45 2011
Return-Path: <general-return-3033-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 13849 invoked from network); 12 Feb 2011 03:28:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Feb 2011 03:28:45 -0000
Received: (qmail 78548 invoked by uid 500); 12 Feb 2011 03:28:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78056 invoked by uid 500); 12 Feb 2011 03:28:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78048 invoked by uid 99); 12 Feb 2011 03:28:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Feb 2011 03:28:40 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of dhruba@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Feb 2011 03:28:33 +0000
Received: by wwd20 with SMTP id 20so3119390wwd.29
        for <general@hadoop.apache.org>; Fri, 11 Feb 2011 19:28:13 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=NLDgwJb0EWHN5yZnRw2kUu2UYYhbzK4QFqz51m5wZMU=;
        b=OImZAuf9DejzPpSEKD88S7QvjxQDyUZ/ujY7E+C+Kvw6vI42YBzLuRJxezhht3GWiJ
         yxEkmpIJRlTwi7ecWeNICzpk+9vYqNKlY3za9X3TtMz2kNsNf8rReobSwzrNPQI8R69S
         SPibQ0tEViWvsYuT7mtB0ByAxtbvy2ynWNo9U=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=mbGNmdC+nCjzw1NMgcZOqK+8vcKOlm7vDwFpB29UzuOl0aUmCB84EmGE9fXVIo7D2z
         MlXMF78XOBeOZB71hq2RR8xr81v0ReV7vWOv+M5H21xoZPIBZw7H9YsF1UlHx8PjXbpQ
         32pcB5unZxTBy/ONFEGCLCSoBIhsz991yW9vE=
MIME-Version: 1.0
Received: by 10.216.20.141 with SMTP id p13mr1398474wep.102.1297481293375;
 Fri, 11 Feb 2011 19:28:13 -0800 (PST)
Received: by 10.216.185.135 with HTTP; Fri, 11 Feb 2011 19:28:13 -0800 (PST)
In-Reply-To: <AANLkTikouS9=ksDypNXa2LGUdqTbzAUbjEW4QeJszPAk@mail.gmail.com>
References: <EF44CFE2-692F-4956-8B33-D125D05E2B53@mac.com>
	<AANLkTikouS9=ksDypNXa2LGUdqTbzAUbjEW4QeJszPAk@mail.gmail.com>
Date: Fri, 11 Feb 2011 19:28:13 -0800
Message-ID: <AANLkTinB4YRLpQNRZu7vneUd+u1ggmSkP_TfL+u6mAnQ@mail.gmail.com>
Subject: Re: [VOTE] Abandon thriftfs HDFS contrib
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163646da74982a80049c0d679f
X-Virus-Checked: Checked by ClamAV on apache.org

--00163646da74982a80049c0d679f
Content-Type: text/plain; charset=ISO-8859-1

+0.

we are using this in production, but sadly I never had any time to keep it
updated. I have a plan of integrating the generated Thrift server-stubs with
the namenode itself, so that the NameNode can expose Hadoop RPC as well as
Thrift RPC. But this is something for the future, so I am fine either way if
we decide to remove it or to keep it as it is.

thanks,
dhruba

On Fri, Feb 11, 2011 at 9:48 AM, Tom White <tom.e.white@gmail.com> wrote:

> +1
>
> Tom
>
> On Thu, Feb 10, 2011 at 7:44 PM, Nigel Daley <ndaley@mac.com> wrote:
> > I think the PMC should abandon the thriftfs HDFS contrib component.  It's
> last meaningful contribution was it's original commit in August 2008:
> >
> > HADOOP-3754. Add a thrift interface to access HDFS. (dhruba via omalley)
> >
> > There are 5 unresolved contrib/thriftfs issues in Jira, 1 of them Patch
> Available since November 2009 (HDFS-789).
> >
> > Here is my +1.
> >
> > Nige
> >
>



-- 
Connect to me at http://www.facebook.com/dhruba

--00163646da74982a80049c0d679f--

From general-return-3034-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Feb 12 05:16:31 2011
Return-Path: <general-return-3034-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45848 invoked from network); 12 Feb 2011 05:16:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Feb 2011 05:16:30 -0000
Received: (qmail 9260 invoked by uid 500); 12 Feb 2011 05:16:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8710 invoked by uid 500); 12 Feb 2011 05:16:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8697 invoked by uid 99); 12 Feb 2011 05:16:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Feb 2011 05:16:24 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.27.212] (HELO qmta14.emeryville.ca.mail.comcast.net) (76.96.27.212)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Feb 2011 05:16:14 +0000
Received: from omta16.emeryville.ca.mail.comcast.net ([76.96.30.72])
	by qmta14.emeryville.ca.mail.comcast.net with comcast
	id 6tEC1g0041ZMdJ4AEtFsMS; Sat, 12 Feb 2011 05:15:52 +0000
Received: from [192.168.1.50] ([209.131.62.115])
	by omta16.emeryville.ca.mail.comcast.net with comcast
	id 6tFj1g0032VBGtd8ctFl5M; Sat, 12 Feb 2011 05:15:50 +0000
Message-Id: <AC425E88-8B0E-4982-84FF-D55C7C2C7E3D@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <535D05CA-3321-48BA-82C4-C18FC8983C3A@mac.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Abandon hod Common contrib
Date: Fri, 11 Feb 2011 21:15:42 -0800
References: <AC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com> <84B97117-5DA4-4A32-912A-8DBFCE3D063E@linkedin.com> <535D05CA-3321-48BA-82C4-C18FC8983C3A@mac.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


On Feb 11, 2011, at 6:17 PM, Nigel Daley wrote:

>> a) I don't think hod is actually part of any unit tests, so  
>> including it would likely only be a burden on the tarball size.
>
> Not true.  HOD has python unit tests and is the reason our builds  
> have dependencies on python.

But Allen's point is that I don't recall ever seeing HOD test failures  
causing the build to fail.

>> b) The edu community uses this quite extensively, evidenced by the  
>> topic coming up on the mailing lists at least once every two months  
>> or so and has for years.  Can't say that about the other contrib  
>> modules other than the schedulers and streaming.
>
> Then they are using old version of Hadoop.  AFAICT HOD does not work  
> with 0.20 or beyond.

Out of curiosity, what goes wrong? Clearly nothing major has changed  
in starting up a mapreduce cluster in a very long time.

>> c) The community that does use it has even submitted a patch that  
>> we've ignored.
>
> Which means the committers of this project gave up on it long ago.

There are also some patches on core Hadoop that have been sitting for  
a long time, so I don't think that is a valid inference.

I would love to hear some of the people who are using HOD speak up and  
give us their feedback.

-- Owen

From general-return-3035-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Feb 12 05:51:40 2011
Return-Path: <general-return-3035-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63172 invoked from network); 12 Feb 2011 05:51:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Feb 2011 05:51:39 -0000
Received: (qmail 25986 invoked by uid 500); 12 Feb 2011 05:51:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25920 invoked by uid 500); 12 Feb 2011 05:51:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25912 invoked by uid 99); 12 Feb 2011 05:51:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Feb 2011 05:51:33 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.101 as permitted sender)
Received: from [17.148.16.101] (HELO asmtpout026.mac.com) (17.148.16.101)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Feb 2011 05:51:24 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp026.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LGH00H0HPKNFB20@asmtp026.mac.com>
 for general@hadoop.apache.org; Fri, 11 Feb 2011 21:50:48 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-02-12_02:2011-02-11,2011-02-12,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1102110182
Subject: Re: [VOTE] Abandon hod Common contrib
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <AC425E88-8B0E-4982-84FF-D55C7C2C7E3D@apache.org>
Date: Fri, 11 Feb 2011 21:50:47 -0800
Message-id: <BF846B83-D48C-46B5-8192-19B09EBEA595@mac.com>
References: <AC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com>
 <84B97117-5DA4-4A32-912A-8DBFCE3D063E@linkedin.com>
 <535D05CA-3321-48BA-82C4-C18FC8983C3A@mac.com>
 <AC425E88-8B0E-4982-84FF-D55C7C2C7E3D@apache.org>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org


On Feb 11, 2011, at 9:15 PM, Owen O'Malley wrote:

> 
> On Feb 11, 2011, at 6:17 PM, Nigel Daley wrote:
> 
>>> a) I don't think hod is actually part of any unit tests, so including it would likely only be a burden on the tarball size.
>> 
>> Not true.  HOD has python unit tests and is the reason our builds have dependencies on python.
> 
> But Allen's point is that I don't recall ever seeing HOD test failures causing the build to fail.

Ah, looks like we haven't been testing it via Hudson for a long time.

Nige


From general-return-3036-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Feb 12 05:57:51 2011
Return-Path: <general-return-3036-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 64013 invoked from network); 12 Feb 2011 05:57:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Feb 2011 05:57:51 -0000
Received: (qmail 28857 invoked by uid 500); 12 Feb 2011 05:57:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28670 invoked by uid 500); 12 Feb 2011 05:57:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28662 invoked by uid 99); 12 Feb 2011 05:57:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Feb 2011 05:57:45 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.95 as permitted sender)
Received: from [17.148.16.95] (HELO asmtpout020.mac.com) (17.148.16.95)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Feb 2011 05:57:39 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp020.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LGH00C43PVHRT10@asmtp020.mac.com>
 for general@hadoop.apache.org; Fri, 11 Feb 2011 21:57:18 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-02-12_02:2011-02-11,2011-02-12,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1102110183
Subject: Re: [VOTE] Abandon mrunit MapReduce contrib
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <AANLkTim+FQuoS+mR7YuTf0kw1KVxCdy14wDjf2u_cZdB@mail.gmail.com>
Date: Fri, 11 Feb 2011 21:57:17 -0800
Message-id: <33A12FDB-820A-4CCC-9AB7-0521B41952B1@mac.com>
References: <77405974-6771-4604-926B-976240743F3C@mac.com>
 <AANLkTim80KJ3f14iM6Ltzy7UG613myhN-AiwQ3pCmVdi@mail.gmail.com>
 <AANLkTiknkrWK=GzGMVj4_Z6p5Jn8TQu1P5TkaFSgORJk@mail.gmail.com>
 <AANLkTin0TguLYHi9X0uL3ZF7tCz5xTHBYa7056oKF2Ae@mail.gmail.com>
 <93644D3B-4830-4813-92BE-809738969CF9@apache.org>
 <AANLkTi=dbGSLhtFA+Rf9LgaX05v45HDLoQDE_2C5ULAf@mail.gmail.com>
 <65311075-A30E-4971-A2BA-C75409C6486E@jpl.nasa.gov>
 <AANLkTi=XbpVUmg_TpH7eqEn8w7Gcu9KiwtJmKMLa5w5n@mail.gmail.com>
 <FAE19BAA-47A7-4747-BDD8-5AA42C259EB6@jpl.nasa.gov>
 <AANLkTin2JvOGdZT5g5tmTGeCpD3Qz2soZWcTQtHMsSza@mail.gmail.com>
 <AANLkTim+FQuoS+mR7YuTf0kw1KVxCdy14wDjf2u_cZdB@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

This is great!  So we'll leave mrunit in contrib until it can be moved to incubator.

Nige

On Feb 11, 2011, at 2:26 PM, Eric Sammer wrote:

> Just to add to the option of going to incubator, I'm fine with that as well.
> Github was an easy thing to get started and I was under the impression we
> needed some greater degree of committer diversity and, frankly, a bigger
> project. If mrunit is a candidate, keeping this under the ASF umbrella is
> more than fine with me.
> 
> On Fri, Feb 11, 2011 at 5:10 PM, Aaron Kimball <akimball83@gmail.com> wrote:
> 
>> The main reason I am interested in removing MRUnit from Hadoop is that I
>> believe that MRUnit deserves its own release cycle. I think this is in the
>> best interest of its users.
>> 
>> MRUnit is valuable to users of several different versions of Hadoop. But
>> MRUnit has only ever been committed to version 0.21 and above -- even
>> though
>> in practice, the majority (dare I say--all) of its users are running on
>> 0.20. The only place today to get a version of MRUnit compatible with 0.20
>> has been through a Cloudera release, which backported the entire MRUnit
>> patchset.
>> 
>> My thoughts on MRUnit in 0.20.100 resonate with Eric's. There will be
>> further fixes to MRUnit and its lightweight codebase can be released far
>> more rapidly than whenever the next 0.20.1xx release of Hadoop would occur.
>> Given that MRUnit has already been in the repository since April 2009 (see
>> https://issues.apache.org/jira/browse/HADOOP-5518) and has yet to see an
>> Apache 0.20-based release, I do not think it is in the best interest of the
>> library's userbase to couple MRUnit's release cycle to that of Hadoop
>> itself.
>> 
>> Perhaps more importantly, access to new features in MRUnit should not
>> require upgrading one's entire Hadoop deployment; this is a client library
>> that depends only on Hadoop's public APIs.
>> 
>> My primary concern is to move MRUnit to a place where the community can
>> derive the most benefit from it. The Apache Incubator could fulfill this
>> role; given the presence of individuals willing to mentor this project, I
>> believe this would be a successful way to release MRUnit more quickly and
>> continue to work to grow the MRUnit community.
>> 
>> Regards,
>> - Aaron
>> 
>> 
>> On Fri, Feb 11, 2011 at 11:57 AM, Mattmann, Chris A (388J) <
>> chris.a.mattmann@jpl.nasa.gov> wrote:
>> 
>>> Awesome Patrick, we'd probably need one more active mentor. Any takers?
>>> 
>>> After we get that, then we cook up a proposal on the Incubator wiki here
>>> [1], and follow the process here [2] to get started...
>>> 
>>> Cheers,
>>> Chris
>>> 
>>> [1] http://wiki.apache.org/incubator/MRUnitProposal
>>> [2] http://incubator.apache.org/guides/proposal.html
>>> 
>>> On Feb 11, 2011, at 11:52 AM, Patrick Hunt wrote:
>>> 
>>>> On Fri, Feb 11, 2011 at 9:44 AM, Mattmann, Chris A (388J)
>>>> <chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>> Guys, BTW, if you need help or a mentor in Apache Incubator-ville for
>>> MRUnit, I would be happy to help.
>>>> 
>>>> I was going to suggest the same thing (mrunit to incubator). I would
>>>> also be happy to be a mentor.
>>>> 
>>>> Patrick
>>>> 
>>>>> 
>>>>> On Feb 11, 2011, at 9:04 AM, Eric Sammer wrote:
>>>>> 
>>>>>> On Fri, Feb 11, 2011 at 11:48 AM, Owen O'Malley <omalley@apache.org>
>>> wrote:
>>>>>> 
>>>>>>> On Feb 11, 2011, at 8:02 AM, Eric Sammer wrote:
>>>>>>> 
>>>>>>> - allow mrunit to have its own release cycle. This is, I think, the
>>> most
>>>>>>>> 
>>>>>>> 
>>>>>>> important.
>>>>>>>> 
>>>>>>> 
>>>>>>> If you submit your work to Apache we can evaluate it for inclusion
>> in
>>> the
>>>>>>> 0.20.100 branch to get your changes released in a timely manner.
>>>>>> 
>>>>>> 
>>>>>> I'm thinking in general (beyond the next immediate release).
>>> Independent of
>>>>>> where mrunit goes, I think it should leave the contrib tree to
>>> facilitate
>>>>>> light weight releases (the dependency on Hadoop proper is a public
>>> facing
>>>>>> API - a pure client). I think most projects could benefit from this
>>> with the
>>>>>> exception of things that are tightly coupled to Hadoop releases or
>>> touch
>>>>>> non-public APIs.
>>>>>> 
>>>>>> 
>>>>>>> I would actually prefer to move it to Extras or Incubator and leave
>>> this
>>>>>>>> within the ASF.
>>>>>>>> 
>>>>>>> 
>>>>>>> Extras is **NOT** inside of the ASF. Extras is a source hosting
>> system
>>> for
>>>>>>> non-Apache projects that are related to Apache projects.
>>>>>> 
>>>>>> 
>>>>>> Got it. Thanks for correcting me. I only mentioned it because someone
>>>>>> suggested it to me initially.
>>>>>> 
>>>>>> 
>>>>>>> Right now, I picked github because of the ability to easily
>>>>>>> collaborate with others (and to use git).
>>>>>>> 
>>>>>> 
>>>>>> I agree that it is unfortunate that Apache doesn't yet support
>>> read-write
>>>>>>> git access. However, you'll find that building a community is easier
>>> at
>>>>>>> Apache than at github.
>>>>>>> 
>>>>>> 
>>>>>>> -- Owen
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Eric Sammer
>>>>>> twitter: esammer
>>>>>> data: www.cloudera.com
>>>>> 
>>>>> 
>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> Chris Mattmann, Ph.D.
>>>>> Senior Computer Scientist
>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>> Office: 171-266B, Mailstop: 171-246
>>>>> Email: chris.a.mattmann@nasa.gov
>>>>> WWW:   http://sunset.usc.edu/~mattmann/
>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> 
>>>>> 
>>> 
>>> 
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Chris Mattmann, Ph.D.
>>> Senior Computer Scientist
>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> Office: 171-266B, Mailstop: 171-246
>>> Email: chris.a.mattmann@nasa.gov
>>> WWW:   http://sunset.usc.edu/~mattmann/
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Adjunct Assistant Professor, Computer Science Department
>>> University of Southern California, Los Angeles, CA 90089 USA
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> 
>>> 
>> 
> 
> 
> 
> -- 
> Eric Sammer
> twitter: esammer
> data: www.cloudera.com


From general-return-3037-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Feb 12 06:02:10 2011
Return-Path: <general-return-3037-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 64936 invoked from network); 12 Feb 2011 06:02:10 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Feb 2011 06:02:10 -0000
Received: (qmail 31892 invoked by uid 500); 12 Feb 2011 06:02:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31664 invoked by uid 500); 12 Feb 2011 06:02:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31656 invoked by uid 99); 12 Feb 2011 06:02:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Feb 2011 06:02:05 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.105 as permitted sender)
Received: from [17.148.16.105] (HELO asmtpout030.mac.com) (17.148.16.105)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Feb 2011 06:01:58 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp030.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LGH002PWQ29HS60@asmtp030.mac.com>
 for general@hadoop.apache.org; Fri, 11 Feb 2011 22:01:22 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-02-12_02:2011-02-11,2011-02-12,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1102110183
Subject: Re: [DISCUSS] Move common, hdfs,
 mapreduce contrib components to apache-extras.org or elsewhere
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <AANLkTi=HQ1cn++JO6g+vp-cJAtAjB00Ok-FW2VrfTiSO@mail.gmail.com>
Date: Fri, 11 Feb 2011 22:01:21 -0800
Message-id: <5E398D0F-ABCC-497B-939A-9213A5B3C7C4@mac.com>
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com>
 <AANLkTi=75e6oyBuGVivXP3is+76gOODSc1NcLKfg=VxF@mail.gmail.com>
 <AANLkTi=HQ1cn++JO6g+vp-cJAtAjB00Ok-FW2VrfTiSO@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

+1.  And only include them as source in releases.

Nige 

On Feb 11, 2011, at 10:12 AM, Tom White wrote:

> For contrib components that we elect not to remove, I suggest that we
> remove them from the main build, so that failures in a contrib
> component don't hinder the main build and release. This is what
> ZooKeeper does, for example.
> 
> Tom
> 
> On Fri, Feb 11, 2011 at 2:03 AM, Bernd Fondermann
> <bernd.fondermann@googlemail.com> wrote:
>> -1.
>> 
>> Move it away from TRUNK so it doesn't affect builds is a much better
>> (and simpler) solution. If someone wants to revive it, he can within
>> the bounds of Apache Hadoop and will become a part of the Hadoop
>> community, which would be good.
>> If you'd move it off-site, if the code ever gets new contributors,
>> it's hard to integrate them (code and contributors) into Hadoop again.
>> 
>> AFAIUI, apache-extras is for placing non-Apache code closer to the
>> related Apache projects, not for moving our code away from our own
>> premises.
>> 
>>  Bernd
>> 
>> On Mon, Jan 31, 2011 at 04:42, Nigel Daley <ndaley@mac.com> wrote:
>>> Folks,
>>> 
>>> Now that http://apache-extras.org is launched (https://blogs.apache.org/foundation/entry/the_apache_software_foundation_launches) I'd like to start a discussion on moving contrib components out of common, mapreduce, and hdfs.
>>> 
>>> These contrib components complicate the builds, cause test failures that nobody seems to care about, have releases that are tied to Hadoop's long release cycles, etc.  Most folks I've talked with agree that these contrib components would be better served by being pulled out of Hadoop and hosted elsewhere. The new apache-extras code hosting site seems like a natural *default* location for migrating these contrib projects.  Perhaps some should graduate from contrib to src (ie from contrib to core of the project they're included in).  If folks agree, we'll need to come up with a mapping of contrib component to it's final destination and file a jira.
>>> 
>>> Here are the contrib components by project (hopefully I didn't miss any).
>>> 
>>> Common Contrib:
>>>  failmon
>>>  hod
>>>  test
>>> 
>>> 
>>> MapReduce Contrib:
>>>  capacity-scheduler -- move to MR core?
>>>  data_join
>>>  dynamic-scheduler
>>>  eclipse-plugin
>>>  fairscheduler -- move to MR core?
>>>  gridmix
>>>  index
>>>  mrunit
>>>  mumak
>>>  raid
>>>  sqoop
>>>  streaming -- move to MR core?
>>>  vaidya
>>>  vertica
>>> 
>>> 
>>> HDFS Contrib:
>>>  fuse-dfs
>>>  hdfsproxy
>>>  thriftfs
>>> 
>>> 
>>> Cheers,
>>> Nige
>>> 
>> 


From general-return-3038-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Feb 12 08:18:44 2011
Return-Path: <general-return-3038-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16870 invoked from network); 12 Feb 2011 08:18:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Feb 2011 08:18:43 -0000
Received: (qmail 88484 invoked by uid 500); 12 Feb 2011 08:18:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87911 invoked by uid 500); 12 Feb 2011 08:18:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87903 invoked by uid 99); 12 Feb 2011 08:18:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Feb 2011 08:18:39 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [208.97.132.207] (HELO homiemail-a73.g.dreamhost.com) (208.97.132.207)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Feb 2011 08:18:31 +0000
Received: from homiemail-a73.g.dreamhost.com (localhost [127.0.0.1])
	by homiemail-a73.g.dreamhost.com (Postfix) with ESMTP id 6411B1F0080
	for <general@hadoop.apache.org>; Sat, 12 Feb 2011 00:18:10 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; q=dns; s=
	gbiv.com; b=c00i7ge+x1A2r9KHYqLRyhD+KSMiKuUVGPMaI+ea3GdVMYBpL3LV
	ir7OWAIBeqRxhV28xnmnHsNR7B3NR4gqR0II7ULDsoP3Ngj5s49bj9ujrzxhlH0S
	/NK/xozlGxDwLsQNuA/RwE1xORW5TQ/HR4rZgOPQ7FhVBpJ3HF7P5fk=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; s=gbiv.com;
	 bh=OFuff24D+2Y/YM0WmMOfWpU+5x0=; b=SfSsQKSvxcAF12uxC9vDr3hANez9
	9fdova3egxZkIQFv5iPj1r1OqCHGY6HmTveP3MLw+p2ApToKVxPczzShdQ1t6Pof
	e/a6I+Tw0ETtoytL4EtCL9R4Va6Ujg9+kABr0U6vN9YRoNJ5MvAkVi+UvJ+H+do5
	xE62IRHpoZXAZz8=
Received: from [192.168.1.84] (99-21-208-82.lightspeed.irvnca.sbcglobal.net [99.21.208.82])
	(using TLSv1 with cipher AES128-SHA (128/128 bits))
	(No client certificate requested)
	(Authenticated sender: fielding@gbiv.com)
	by homiemail-a73.g.dreamhost.com (Postfix) with ESMTPSA id 32F111F007C
	for <general@hadoop.apache.org>; Sat, 12 Feb 2011 00:18:10 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [VOTE] Abandon hdfsproxy HDFS contrib
From: "Roy T. Fielding" <fielding@gbiv.com>
In-Reply-To: <AANLkTimUhDbTnwn6Hk2oYXFJMts+A=FnL-q-veXqX1m0@mail.gmail.com>
Date: Sat, 12 Feb 2011 00:18:09 -0800
Content-Transfer-Encoding: quoted-printable
Message-Id: <F2EC1B1B-A2BB-4FC8-9842-3904A25338EE@gbiv.com>
References: <D97CF74A-147E-4573-B188-2F9C48CFC482@mac.com> <AANLkTi=F2eBsACBfjYSVava+VwFcsRpqZdRRN+OHXNRJ@mail.gmail.com> <823262F0-EE37-4A0A-8F8C-D9422541D67B@mac.com> <0CBAFDBB-AC2A-495D-8936-4DF4D6731B69@Holsman.NET> <AANLkTimUhDbTnwn6Hk2oYXFJMts+A=FnL-q-veXqX1m0@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

On Feb 11, 2011, at 2:28 AM, Bernd Fondermann wrote:
> On Fri, Feb 11, 2011 at 07:33, Ian Holsman <hadoop@holsman.net> wrote:
>> They probably have patched it, and mistakenly forgot to submit them.. =
any chance of doing a diff on your version and submitting it?
>=20
> Please keep in mind: The original author(s) would need to submit it -
> not a proxy.

No, anyone with permission of the copyright owner can submit it
if it is a separately copyrightable work.  If it is just a repair,
then anyone can submit it, since repairs are not copyrightable.
The original author should be noted in any credit given for the
fix.

....Roy


From general-return-3039-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Feb 12 22:43:28 2011
Return-Path: <general-return-3039-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29665 invoked from network); 12 Feb 2011 22:43:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Feb 2011 22:43:28 -0000
Received: (qmail 95549 invoked by uid 500); 12 Feb 2011 22:43:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95466 invoked by uid 500); 12 Feb 2011 22:43:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95458 invoked by uid 99); 12 Feb 2011 22:43:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Feb 2011 22:43:25 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Feb 2011 22:43:19 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=WVM4D2tglxIwiDhTM2IKTGUwvpI5ZBpS1TOMBrA5siDEoAgIqROs7HDI
   AebjoMbriYFfpfehvgk72/8xfhxDaGw14l663E8PBQGXBUIRI4SZMaMlt
   /qv5wquI2i5sQKe;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1297550596; x=1329086596;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20Hadoop=200.22=20Blockers|Date:=20Sat,
   =2012=20Feb=202011=2022:44:23=20+0000|Message-ID:=20<EF94
   155D-5148-4A74-85CD-669BA842C3CC@linkedin.com>|To:=20"<ge
   neral@hadoop.apache.org>"=20<general@hadoop.apache.org>
   |MIME-Version:=201.0|Content-Transfer-Encoding:=20quoted-
   printable|Content-ID:=20<A7336EE13BB75D40A260188C8E499D68
   @linkedin.com>|In-Reply-To:=20<03B93C0F-FDB4-4704-8CF3-57
   B8DF00E9B8@mac.com>|References:=20<57A4D9DB-2CD2-4838-AD6
   5-7DEBA1F3F0EA@mac.com>=0D=0A=20<3D9EA200-342B-4B1F-A2E8-
   6FC159BFDE38@linkedin.com>=0D=0A=20<03B93C0F-FDB4-4704-8C
   F3-57B8DF00E9B8@mac.com>;
  bh=hDGzG1WLjZ5FZ/Hob4qNtRhXNoOcFckVBz0RfGEWe/E=;
  b=eKC39P3uSBU2XyGj05FrH30iV8OceNYpmAL6GaSEmscJOsgOcYRSxBEi
   8JjXbI/Fy7F0Fts8cMYO9iYGtptg0MWw2p2jl9YQLl3Ie67802y7B37ef
   xs9o9cz8sFKwPn3;
X-IronPort-AV: E=Sophos;i="4.60,463,1291622400"; 
   d="scan'208";a="20116030"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi id 14.01.0251.000; Sat,
 12 Feb 2011 14:44:24 -0800
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: Hadoop 0.22 Blockers
Thread-Topic: Hadoop 0.22 Blockers
Thread-Index: AQHLyb5E13M7p8Xx00aNwoGFZtJq/pP9GAOAgAABJgCAAeXYAA==
Date: Sat, 12 Feb 2011 22:44:23 +0000
Message-ID: <EF94155D-5148-4A74-85CD-669BA842C3CC@linkedin.com>
References: <57A4D9DB-2CD2-4838-AD65-7DEBA1F3F0EA@mac.com>
 <3D9EA200-342B-4B1F-A2E8-6FC159BFDE38@linkedin.com>
 <03B93C0F-FDB4-4704-8CF3-57B8DF00E9B8@mac.com>
In-Reply-To: <03B93C0F-FDB4-4704-8CF3-57B8DF00E9B8@mac.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <A7336EE13BB75D40A260188C8E499D68@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


On Feb 11, 2011, at 9:44 AM, Nigel Daley wrote:

>=20
> On Feb 11, 2011, at 9:41 AM, Allen Wittenauer wrote:
>=20
>>=20
>> On Feb 10, 2011, at 11:33 PM, Nigel Daley wrote:
>>=20
>>> Tom has created a public Jira filter for 0.22 blockers (thanks Tom!):
>>> https://issues.apache.org/jira/secure/IssueNavigator.jspa?mode=3Dhide&r=
equestId=3D12313687
>>> With the new version of Jira, this single query across projects & versi=
ons is now possible.
>>>=20
>>> Looks like 26 issues marked blocker.  9 of them unassigned -- would lov=
e to get those assigned.  I'll start reviewing these issues more closely ov=
er the weekend.
>>=20
>> 	Does this include the ones you unmarked as blockers for some mysterious=
 reason?
>=20
> To which issues are you referring exactly?
>=20


	I'm thinking specifically of the one that prevents datanodes from being pr=
operly decommissioned.  But you've changed so many it is hard to say what t=
he full list is.=

From general-return-3040-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Feb 13 00:57:34 2011
Return-Path: <general-return-3040-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84320 invoked from network); 13 Feb 2011 00:57:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Feb 2011 00:57:33 -0000
Received: (qmail 43897 invoked by uid 500); 13 Feb 2011 00:57:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 43820 invoked by uid 500); 13 Feb 2011 00:57:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 43811 invoked by uid 99); 13 Feb 2011 00:57:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Feb 2011 00:57:31 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.105 as permitted sender)
Received: from [17.148.16.105] (HELO asmtpout030.mac.com) (17.148.16.105)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Feb 2011 00:57:22 +0000
MIME-version: 1.0
Content-type: multipart/alternative;
 boundary="Boundary_(ID_cbRb3sXTUT84iHOEy9NlCQ)"
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp030.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LGJ003VF6M17Z00@asmtp030.mac.com>
 for general@hadoop.apache.org; Sat, 12 Feb 2011 16:56:26 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-02-13_01:2011-02-11,2011-02-13,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1102120085
From: Nigel Daley <ndaley@mac.com>
Subject: Re: Hadoop 0.22 Blockers
Date: Sat, 12 Feb 2011 16:56:25 -0800
In-reply-to: <EF94155D-5148-4A74-85CD-669BA842C3CC@linkedin.com>
To: general@hadoop.apache.org
References: <57A4D9DB-2CD2-4838-AD65-7DEBA1F3F0EA@mac.com>
 <3D9EA200-342B-4B1F-A2E8-6FC159BFDE38@linkedin.com>
 <03B93C0F-FDB4-4704-8CF3-57B8DF00E9B8@mac.com>
 <EF94155D-5148-4A74-85CD-669BA842C3CC@linkedin.com>
Message-id: <768EC263-4F4F-4377-8561-2769C07318C3@mac.com>
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

--Boundary_(ID_cbRb3sXTUT84iHOEy9NlCQ)
Content-type: text/plain; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT


On Feb 12, 2011, at 2:44 PM, Allen Wittenauer wrote:

> 
> On Feb 11, 2011, at 9:44 AM, Nigel Daley wrote:
> 
>> 
>> On Feb 11, 2011, at 9:41 AM, Allen Wittenauer wrote:
>> 
>>> 
>>> On Feb 10, 2011, at 11:33 PM, Nigel Daley wrote:
>>> 
>>>> Tom has created a public Jira filter for 0.22 blockers (thanks Tom!):
>>>> https://issues.apache.org/jira/secure/IssueNavigator.jspa?mode=hide&requestId=12313687
>>>> With the new version of Jira, this single query across projects & versions is now possible.
>>>> 
>>>> Looks like 26 issues marked blocker.  9 of them unassigned -- would love to get those assigned.  I'll start reviewing these issues more closely over the weekend.
>>> 
>>> 	Does this include the ones you unmarked as blockers for some mysterious reason?
>> 
>> To which issues are you referring exactly?
>> 
> 
> 
> 	I'm thinking specifically of the one that prevents datanodes from being properly decommissioned.  But you've changed so many it is hard to say what the full list is.

Something from here perhaps? https://spreadsheets.google.com/ccc?key=t39bSGEkCmAqKkXJwyHVFJw&hl=en#gid=3
These were the original Hadoop 0.22 Jira's that weren't blockers so I unassigned them, as per release SOP. 

I want to make sure what you're referring to was inadvertent. Please provide issue numbers and then I can respond.

Nige

--Boundary_(ID_cbRb3sXTUT84iHOEy9NlCQ)--

From general-return-3041-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 14 06:16:30 2011
Return-Path: <general-return-3041-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95777 invoked from network); 14 Feb 2011 06:16:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Feb 2011 06:16:29 -0000
Received: (qmail 24015 invoked by uid 500); 14 Feb 2011 06:16:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23654 invoked by uid 500); 14 Feb 2011 06:16:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23641 invoked by uid 99); 14 Feb 2011 06:16:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Feb 2011 06:16:23 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Feb 2011 06:16:17 +0000
Received: from [10.66.72.163] (sightbusy-lx.eglbp.corp.yahoo.com [10.66.72.163])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p1E6FjXv032594;
	Sun, 13 Feb 2011 22:15:45 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1297664146;
	bh=87jj+TVOEXmuqCxuxUzWgZ9org91LfYuptYNavggZAg=;
	h=Message-ID:Date:From:MIME-Version:To:CC:Subject:References:
	 In-Reply-To:Content-Type:Content-Transfer-Encoding;
	b=mn1digmGbnLU5TUTFA1ONARCJkZmeQWJxs+DizkEiVdC50pNMTnaZ8zMDpToWrbPs
	 OoI6uG4SQWSIIZ0/RwDJ6kUV2kOjyV2inTjY9pWZvZ9rbEpOe2R4F6uA++EnTstqUG
	 rdTYoF5v8FBi9iNrfg4rI7RjZxcmVkGzZGiuOaB0=
Message-ID: <4D58C890.5030601@yahoo-inc.com>
Date: Mon, 14 Feb 2011 11:45:44 +0530
From: Vinod KV <vinodkv@yahoo-inc.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101208 Lightning/1.0b2 Thunderbird/3.1.7
MIME-Version: 1.0
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "Owen O'Malley" <omalley@apache.org>
Subject: Re: [VOTE] Abandon hod Common contrib
References: <AC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com> <84B97117-5DA4-4A32-912A-8DBFCE3D063E@linkedin.com> <535D05CA-3321-48BA-82C4-C18FC8983C3A@mac.com> <AC425E88-8B0E-4982-84FF-D55C7C2C7E3D@apache.org>
In-Reply-To: <AC425E88-8B0E-4982-84FF-D55C7C2C7E3D@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit


Hemanth Yamijala and myself were the main contributors to HOD.

Answers to few of the questions:
  - IIRC, Hod needed python 2.5 (or some such latest) which was not 
available on Hudson. The tests gracefully run only when that version is 
available. We always used to run these tests on dev boxes before pushing 
patches. May be the versions upgraded after that, but I am not sure.
  - HOD wasn't tested extensively with Hadoop 0.20 and beyond. But I 
could successfully run HOD clusters with Hadoop 20. Thing that could 
affect the run are the configuration properties, environment variables 
and the scripts which at that time were backwards compatible. If some 
properties were removed in 21/22 breaking compatibility with pre 20 
versions (very much possible), that will warrant fixing HOD. Otherwise, 
things should work as they are.

Sure would like to know any interest with the users.

Thanks,
+Vinod


On Saturday 12 February 2011 10:45 AM, Owen O'Malley wrote:
> On Feb 11, 2011, at 6:17 PM, Nigel Daley wrote:
>
>>> a) I don't think hod is actually part of any unit tests, so
>>> including it would likely only be a burden on the tarball size.
>> Not true.  HOD has python unit tests and is the reason our builds
>> have dependencies on python.
> But Allen's point is that I don't recall ever seeing HOD test failures
> causing the build to fail.
>
>>> b) The edu community uses this quite extensively, evidenced by the
>>> topic coming up on the mailing lists at least once every two months
>>> or so and has for years.  Can't say that about the other contrib
>>> modules other than the schedulers and streaming.
>> Then they are using old version of Hadoop.  AFAICT HOD does not work
>> with 0.20 or beyond.
> Out of curiosity, what goes wrong? Clearly nothing major has changed
> in starting up a mapreduce cluster in a very long time.
>
>>> c) The community that does use it has even submitted a patch that
>>> we've ignored.
>> Which means the committers of this project gave up on it long ago.
> There are also some patches on core Hadoop that have been sitting for
> a long time, so I don't think that is a valid inference.
>
> I would love to hear some of the people who are using HOD speak up and
> give us their feedback.
>
> -- Owen
>


From general-return-3042-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 14 21:34:37 2011
Return-Path: <general-return-3042-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14880 invoked from network); 14 Feb 2011 21:34:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Feb 2011 21:34:37 -0000
Received: (qmail 12759 invoked by uid 500); 14 Feb 2011 21:34:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12666 invoked by uid 500); 14 Feb 2011 21:34:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12658 invoked by uid 99); 14 Feb 2011 21:34:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Feb 2011 21:34:34 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Feb 2011 21:34:29 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p1ELY3f4075720
	for <general@hadoop.apache.org>; Mon, 14 Feb 2011 13:34:03 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1297719244;
	bh=GPJW3ZIWh6bWaykOtnbhyzSFd7lDBvSnz7AYWcvy98s=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=VkBJcVSsKSfGOfMO9fKH12IdSfa32PxrljNiMBqT/SFYDDSCNp0S4PoIArA3BM55e
	 y34inBA26Z6N7frmiSGiphC7Jp78Mh2WLCqnhqp4MhJCrTgSUztduSO7dr+ldR0G3O
	 l9iGQW6uuOUKwjDCg7r7QEyhxo/yYEYgPIuMh9r8=
Message-Id: <75192454-0158-4C84-9F79-337A16AA913E@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <354259E0-9D22-47B5-B247-4187182B60DF@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [ANNOUNCEMENT] Yahoo focusing on Apache Hadoop, discontinuing "The Yahoo Distribution of Hadoop"
Date: Mon, 14 Feb 2011 13:34:03 -0800
References: <55BA39F5-5017-42B1-BE25-A643CCEADC75@yahoo-inc.com> <354259E0-9D22-47B5-B247-4187182B60DF@apache.org>
X-Mailer: Apple Mail (2.936)


On Feb 11, 2011, at 2:56 PM, Owen O'Malley wrote:

>
> On Jan 31, 2011, at 7:27 PM, Eric Baldeschwieler wrote:
>
>> Unfortunately, we now have to sort out how to contribute several
>> person-years worth of work to Apache to let us unwind the Yahoo! git
>> repositories.  We currently run two lines of Hadoop development, our
>> sustaining program (hadoop-0.20-sustaining) and hadoop-future.
>
> I also plan to start pushing the hadoop-future work into a branch
> called yahoo-merge(?)  as individual commits from our internal git
> repository. The goal of creating the branch is to enable faster review
> and discussion. These patches will be individually run through the
> jira, review, and commit process to be added to trunk.

As the final installment in this process, I've started a discussion on  
us contributing a re-factor of Map-Reduce in https://issues.apache.org/jira/browse/MAPREDUCE-279 
.

We have a prototype we'd like to commit to a branch soon, where we  
look forward to feedback. From there on, we would love to collaborate  
to get it committed to trunk.

thanks,
Arun



From general-return-3043-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 15 11:14:35 2011
Return-Path: <general-return-3043-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 74677 invoked from network); 15 Feb 2011 11:14:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Feb 2011 11:14:34 -0000
Received: (qmail 15453 invoked by uid 500); 15 Feb 2011 11:14:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15151 invoked by uid 500); 15 Feb 2011 11:14:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15140 invoked by uid 99); 15 Feb 2011 11:14:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Feb 2011 11:14:29 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Feb 2011 11:14:19 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 22377B7D3E
	for <general@hadoop.apache.org>; Tue, 15 Feb 2011 11:13:57 +0000 (GMT)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id H6VPR+yJ9yAL for <general@hadoop.apache.org>;
	Tue, 15 Feb 2011 11:13:47 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 04AE2B7CB2
	for <general@hadoop.apache.org>; Tue, 15 Feb 2011 11:13:46 +0000 (GMT)
MailScanner-NULL-Check: 1298373215.03588@1A6sSnFDGNi0/WjZNFnu7w
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p1FBDXcF008960
	for <general@hadoop.apache.org>; Tue, 15 Feb 2011 11:13:33 GMT
Message-ID: <4D5A5FDD.70600@apache.org>
Date: Tue, 15 Feb 2011 11:13:33 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Abandon mrunit MapReduce contrib
References: <77405974-6771-4604-926B-976240743F3C@mac.com>	<AANLkTim80KJ3f14iM6Ltzy7UG613myhN-AiwQ3pCmVdi@mail.gmail.com>	<AANLkTiknkrWK=GzGMVj4_Z6p5Jn8TQu1P5TkaFSgORJk@mail.gmail.com>	<AANLkTin0TguLYHi9X0uL3ZF7tCz5xTHBYa7056oKF2Ae@mail.gmail.com>	<93644D3B-4830-4813-92BE-809738969CF9@apache.org>	<AANLkTi=dbGSLhtFA+Rf9LgaX05v45HDLoQDE_2C5ULAf@mail.gmail.com>	<65311075-A30E-4971-A2BA-C75409C6486E@jpl.nasa.gov>	<AANLkTi=XbpVUmg_TpH7eqEn8w7Gcu9KiwtJmKMLa5w5n@mail.gmail.com>	<FAE19BAA-47A7-4747-BDD8-5AA42C259EB6@jpl.nasa.gov>	<AANLkTin2JvOGdZT5g5tmTGeCpD3Qz2soZWcTQtHMsSza@mail.gmail.com> <AANLkTim+FQuoS+mR7YuTf0kw1KVxCdy14wDjf2u_cZdB@mail.gmail.com>
In-Reply-To: <AANLkTim+FQuoS+mR7YuTf0kw1KVxCdy14wDjf2u_cZdB@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p1FBDXcF008960
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 11/02/11 22:26, Eric Sammer wrote:
> Just to add to the option of going to incubator, I'm fine with that as well.
> Github was an easy thing to get started and I was under the impression we
> needed some greater degree of committer diversity and, frankly, a bigger
> project. If mrunit is a candidate, keeping this under the ASF umbrella is
> more than fine with me.
>

There is a git repository coming up at Apache, with global read and 
LDAP-authenticated write:
https://issues.apache.org/jira/browse/INFRA-3165

I know a lot of the in house Hadoop teams use Git for SCM, so I think 
having the big Hadoop projects work on Git too makes sense -it would 
certainly help Owen and team get their changes over if their branches 
could be merged.

Starting off with a small but active Hadoop family project would be the 
way to start this, and MRUnit does appeal as a first step
  -active development
  -fast and decoupled release cycle
  -off the critical path

I'll volunteer to go down as a committer with the usual caveat that I am 
over-committed and unless I have a direct personal need to touch the 
code I'll be effectively offline. Tom White can vouch for that, given my 
engagement in Whirr.

Here then are my votes

  +1 to move Mrunit into incubator
  +1 to experiment with mrunit being live in Apache's Git repository.

-steve

From general-return-3044-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 15 18:37:35 2011
Return-Path: <general-return-3044-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27055 invoked from network); 15 Feb 2011 18:37:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Feb 2011 18:37:35 -0000
Received: (qmail 29311 invoked by uid 500); 15 Feb 2011 18:37:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29056 invoked by uid 500); 15 Feb 2011 18:37:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29048 invoked by uid 99); 15 Feb 2011 18:37:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Feb 2011 18:37:29 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.100 as permitted sender)
Received: from [17.148.16.100] (HELO asmtpout025.mac.com) (17.148.16.100)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Feb 2011 18:37:24 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [172.29.12.221] ([173.164.176.117])
 by asmtp025.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LGO009QI90SMJ40@asmtp025.mac.com>
 for general@hadoop.apache.org; Tue, 15 Feb 2011 10:36:29 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-02-15_06:2011-02-15,2011-02-15,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1102150107
Subject: Re: [VOTE] Abandon mrunit MapReduce contrib
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <AANLkTi=XbpVUmg_TpH7eqEn8w7Gcu9KiwtJmKMLa5w5n@mail.gmail.com>
Date: Tue, 15 Feb 2011 10:36:28 -0800
Message-id: <AB8246D6-23FF-40FC-9AE5-2777BE14A544@mac.com>
References: <77405974-6771-4604-926B-976240743F3C@mac.com>
 <AANLkTim80KJ3f14iM6Ltzy7UG613myhN-AiwQ3pCmVdi@mail.gmail.com>
 <AANLkTiknkrWK=GzGMVj4_Z6p5Jn8TQu1P5TkaFSgORJk@mail.gmail.com>
 <AANLkTin0TguLYHi9X0uL3ZF7tCz5xTHBYa7056oKF2Ae@mail.gmail.com>
 <93644D3B-4830-4813-92BE-809738969CF9@apache.org>
 <AANLkTi=dbGSLhtFA+Rf9LgaX05v45HDLoQDE_2C5ULAf@mail.gmail.com>
 <65311075-A30E-4971-A2BA-C75409C6486E@jpl.nasa.gov>
 <AANLkTi=XbpVUmg_TpH7eqEn8w7Gcu9KiwtJmKMLa5w5n@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

I'm happy to help mentor as well.

Cheers,
Nige

On Feb 11, 2011, at 11:52 AM, Patrick Hunt wrote:

> On Fri, Feb 11, 2011 at 9:44 AM, Mattmann, Chris A (388J)
> <chris.a.mattmann@jpl.nasa.gov> wrote:
>> Guys, BTW, if you need help or a mentor in Apache Incubator-ville for MRUnit, I would be happy to help.
> 
> I was going to suggest the same thing (mrunit to incubator). I would
> also be happy to be a mentor.
> 
> Patrick
> 
>> 
>> On Feb 11, 2011, at 9:04 AM, Eric Sammer wrote:
>> 
>>> On Fri, Feb 11, 2011 at 11:48 AM, Owen O'Malley <omalley@apache.org> wrote:
>>> 
>>>> On Feb 11, 2011, at 8:02 AM, Eric Sammer wrote:
>>>> 
>>>> - allow mrunit to have its own release cycle. This is, I think, the most
>>>>> 
>>>> 
>>>> important.
>>>>> 
>>>> 
>>>> If you submit your work to Apache we can evaluate it for inclusion in the
>>>> 0.20.100 branch to get your changes released in a timely manner.
>>> 
>>> 
>>> I'm thinking in general (beyond the next immediate release). Independent of
>>> where mrunit goes, I think it should leave the contrib tree to facilitate
>>> light weight releases (the dependency on Hadoop proper is a public facing
>>> API - a pure client). I think most projects could benefit from this with the
>>> exception of things that are tightly coupled to Hadoop releases or touch
>>> non-public APIs.
>>> 
>>> 
>>>> I would actually prefer to move it to Extras or Incubator and leave this
>>>>> within the ASF.
>>>>> 
>>>> 
>>>> Extras is **NOT** inside of the ASF. Extras is a source hosting system for
>>>> non-Apache projects that are related to Apache projects.
>>> 
>>> 
>>> Got it. Thanks for correcting me. I only mentioned it because someone
>>> suggested it to me initially.
>>> 
>>> 
>>>> Right now, I picked github because of the ability to easily
>>>> collaborate with others (and to use git).
>>>> 
>>> 
>>> I agree that it is unfortunate that Apache doesn't yet support read-write
>>>> git access. However, you'll find that building a community is easier at
>>>> Apache than at github.
>>>> 
>>> 
>>>> -- Owen
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Eric Sammer
>>> twitter: esammer
>>> data: www.cloudera.com
>> 
>> 
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW:   http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> 
>> 


From general-return-3045-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 15 18:44:39 2011
Return-Path: <general-return-3045-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29324 invoked from network); 15 Feb 2011 18:44:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Feb 2011 18:44:39 -0000
Received: (qmail 42868 invoked by uid 500); 15 Feb 2011 18:44:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42542 invoked by uid 500); 15 Feb 2011 18:44:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42533 invoked by uid 99); 15 Feb 2011 18:44:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Feb 2011 18:44:34 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of esammer@cloudera.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Feb 2011 18:44:28 +0000
Received: by vws18 with SMTP id 18so337375vws.35
        for <general@hadoop.apache.org>; Tue, 15 Feb 2011 10:44:07 -0800 (PST)
MIME-Version: 1.0
Received: by 10.220.186.195 with SMTP id ct3mr1346472vcb.57.1297795446767;
 Tue, 15 Feb 2011 10:44:06 -0800 (PST)
Received: by 10.220.5.80 with HTTP; Tue, 15 Feb 2011 10:44:06 -0800 (PST)
In-Reply-To: <AB8246D6-23FF-40FC-9AE5-2777BE14A544@mac.com>
References: <77405974-6771-4604-926B-976240743F3C@mac.com>
	<AANLkTim80KJ3f14iM6Ltzy7UG613myhN-AiwQ3pCmVdi@mail.gmail.com>
	<AANLkTiknkrWK=GzGMVj4_Z6p5Jn8TQu1P5TkaFSgORJk@mail.gmail.com>
	<AANLkTin0TguLYHi9X0uL3ZF7tCz5xTHBYa7056oKF2Ae@mail.gmail.com>
	<93644D3B-4830-4813-92BE-809738969CF9@apache.org>
	<AANLkTi=dbGSLhtFA+Rf9LgaX05v45HDLoQDE_2C5ULAf@mail.gmail.com>
	<65311075-A30E-4971-A2BA-C75409C6486E@jpl.nasa.gov>
	<AANLkTi=XbpVUmg_TpH7eqEn8w7Gcu9KiwtJmKMLa5w5n@mail.gmail.com>
	<AB8246D6-23FF-40FC-9AE5-2777BE14A544@mac.com>
Date: Tue, 15 Feb 2011 13:44:06 -0500
Message-ID: <AANLkTimAJ6d9ACxmFoXCJYQNfiNU7oVeZiaXDfyZniUF@mail.gmail.com>
Subject: Re: [VOTE] Abandon mrunit MapReduce contrib
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba53a896988589049c568ce7
X-Virus-Checked: Checked by ClamAV on apache.org

--90e6ba53a896988589049c568ce7
Content-Type: text/plain; charset=ISO-8859-1

I've started the wiki page proposal for Incubator for mrunit. I'll ping
people off list for mentoring. Much appreciated for all the help!

On Tue, Feb 15, 2011 at 1:36 PM, Nigel Daley <ndaley@mac.com> wrote:

> I'm happy to help mentor as well.
>
> Cheers,
> Nige
>
> On Feb 11, 2011, at 11:52 AM, Patrick Hunt wrote:
>
> > On Fri, Feb 11, 2011 at 9:44 AM, Mattmann, Chris A (388J)
> > <chris.a.mattmann@jpl.nasa.gov> wrote:
> >> Guys, BTW, if you need help or a mentor in Apache Incubator-ville for
> MRUnit, I would be happy to help.
> >
> > I was going to suggest the same thing (mrunit to incubator). I would
> > also be happy to be a mentor.
> >
> > Patrick
> >
> >>
> >> On Feb 11, 2011, at 9:04 AM, Eric Sammer wrote:
> >>
> >>> On Fri, Feb 11, 2011 at 11:48 AM, Owen O'Malley <omalley@apache.org>
> wrote:
> >>>
> >>>> On Feb 11, 2011, at 8:02 AM, Eric Sammer wrote:
> >>>>
> >>>> - allow mrunit to have its own release cycle. This is, I think, the
> most
> >>>>>
> >>>>
> >>>> important.
> >>>>>
> >>>>
> >>>> If you submit your work to Apache we can evaluate it for inclusion in
> the
> >>>> 0.20.100 branch to get your changes released in a timely manner.
> >>>
> >>>
> >>> I'm thinking in general (beyond the next immediate release).
> Independent of
> >>> where mrunit goes, I think it should leave the contrib tree to
> facilitate
> >>> light weight releases (the dependency on Hadoop proper is a public
> facing
> >>> API - a pure client). I think most projects could benefit from this
> with the
> >>> exception of things that are tightly coupled to Hadoop releases or
> touch
> >>> non-public APIs.
> >>>
> >>>
> >>>> I would actually prefer to move it to Extras or Incubator and leave
> this
> >>>>> within the ASF.
> >>>>>
> >>>>
> >>>> Extras is **NOT** inside of the ASF. Extras is a source hosting system
> for
> >>>> non-Apache projects that are related to Apache projects.
> >>>
> >>>
> >>> Got it. Thanks for correcting me. I only mentioned it because someone
> >>> suggested it to me initially.
> >>>
> >>>
> >>>> Right now, I picked github because of the ability to easily
> >>>> collaborate with others (and to use git).
> >>>>
> >>>
> >>> I agree that it is unfortunate that Apache doesn't yet support
> read-write
> >>>> git access. However, you'll find that building a community is easier
> at
> >>>> Apache than at github.
> >>>>
> >>>
> >>>> -- Owen
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Eric Sammer
> >>> twitter: esammer
> >>> data: www.cloudera.com
> >>
> >>
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> Chris Mattmann, Ph.D.
> >> Senior Computer Scientist
> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >> Office: 171-266B, Mailstop: 171-246
> >> Email: chris.a.mattmann@nasa.gov
> >> WWW:   http://sunset.usc.edu/~mattmann/
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> Adjunct Assistant Professor, Computer Science Department
> >> University of Southern California, Los Angeles, CA 90089 USA
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>
> >>
>
>


-- 
Eric Sammer
twitter: esammer
data: www.cloudera.com

--90e6ba53a896988589049c568ce7--

From general-return-3046-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 15 19:02:29 2011
Return-Path: <general-return-3046-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 36695 invoked from network); 15 Feb 2011 19:02:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Feb 2011 19:02:29 -0000
Received: (qmail 68869 invoked by uid 500); 15 Feb 2011 19:02:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68475 invoked by uid 500); 15 Feb 2011 19:02:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68083 invoked by uid 99); 15 Feb 2011 19:02:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Feb 2011 19:02:23 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.42] (HELO mail-ww0-f42.google.com) (74.125.82.42)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Feb 2011 19:02:18 +0000
Received: by wwi17 with SMTP id 17so3354241wwi.5
        for <general@hadoop.apache.org>; Tue, 15 Feb 2011 11:01:57 -0800 (PST)
Received: by 10.217.2.73 with SMTP id o51mr1144870wes.66.1297796516786; Tue,
 15 Feb 2011 11:01:56 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.82.194 with HTTP; Tue, 15 Feb 2011 11:01:36 -0800 (PST)
In-Reply-To: <AB8246D6-23FF-40FC-9AE5-2777BE14A544@mac.com>
References: <77405974-6771-4604-926B-976240743F3C@mac.com> <AANLkTim80KJ3f14iM6Ltzy7UG613myhN-AiwQ3pCmVdi@mail.gmail.com>
 <AANLkTiknkrWK=GzGMVj4_Z6p5Jn8TQu1P5TkaFSgORJk@mail.gmail.com>
 <AANLkTin0TguLYHi9X0uL3ZF7tCz5xTHBYa7056oKF2Ae@mail.gmail.com>
 <93644D3B-4830-4813-92BE-809738969CF9@apache.org> <AANLkTi=dbGSLhtFA+Rf9LgaX05v45HDLoQDE_2C5ULAf@mail.gmail.com>
 <65311075-A30E-4971-A2BA-C75409C6486E@jpl.nasa.gov> <AANLkTi=XbpVUmg_TpH7eqEn8w7Gcu9KiwtJmKMLa5w5n@mail.gmail.com>
 <AB8246D6-23FF-40FC-9AE5-2777BE14A544@mac.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Tue, 15 Feb 2011 11:01:36 -0800
X-Google-Sender-Auth: hvldm39sfNsXpZ1A5gtr2Zg-LWs
Message-ID: <AANLkTim8RWasPYFOS6UpAoU+teoRYb1czqRsOXRRDUtW@mail.gmail.com>
Subject: Re: [VOTE] Abandon mrunit MapReduce contrib
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I am up to help as a committer on this project.
--
=A0 Take care,
Konstantin (Cos) Boudnik


On Tue, Feb 15, 2011 at 10:36, Nigel Daley <ndaley@mac.com> wrote:
> I'm happy to help mentor as well.
>
> Cheers,
> Nige
>
> On Feb 11, 2011, at 11:52 AM, Patrick Hunt wrote:
>
>> On Fri, Feb 11, 2011 at 9:44 AM, Mattmann, Chris A (388J)
>> <chris.a.mattmann@jpl.nasa.gov> wrote:
>>> Guys, BTW, if you need help or a mentor in Apache Incubator-ville for M=
RUnit, I would be happy to help.
>>
>> I was going to suggest the same thing (mrunit to incubator). I would
>> also be happy to be a mentor.
>>
>> Patrick
>>
>>>
>>> On Feb 11, 2011, at 9:04 AM, Eric Sammer wrote:
>>>
>>>> On Fri, Feb 11, 2011 at 11:48 AM, Owen O'Malley <omalley@apache.org> w=
rote:
>>>>
>>>>> On Feb 11, 2011, at 8:02 AM, Eric Sammer wrote:
>>>>>
>>>>> - allow mrunit to have its own release cycle. This is, I think, the m=
ost
>>>>>>
>>>>>
>>>>> important.
>>>>>>
>>>>>
>>>>> If you submit your work to Apache we can evaluate it for inclusion in=
 the
>>>>> 0.20.100 branch to get your changes released in a timely manner.
>>>>
>>>>
>>>> I'm thinking in general (beyond the next immediate release). Independe=
nt of
>>>> where mrunit goes, I think it should leave the contrib tree to facilit=
ate
>>>> light weight releases (the dependency on Hadoop proper is a public fac=
ing
>>>> API - a pure client). I think most projects could benefit from this wi=
th the
>>>> exception of things that are tightly coupled to Hadoop releases or tou=
ch
>>>> non-public APIs.
>>>>
>>>>
>>>>> I would actually prefer to move it to Extras or Incubator and leave t=
his
>>>>>> within the ASF.
>>>>>>
>>>>>
>>>>> Extras is **NOT** inside of the ASF. Extras is a source hosting syste=
m for
>>>>> non-Apache projects that are related to Apache projects.
>>>>
>>>>
>>>> Got it. Thanks for correcting me. I only mentioned it because someone
>>>> suggested it to me initially.
>>>>
>>>>
>>>>> Right now, I picked github because of the ability to easily
>>>>> collaborate with others (and to use git).
>>>>>
>>>>
>>>> I agree that it is unfortunate that Apache doesn't yet support read-wr=
ite
>>>>> git access. However, you'll find that building a community is easier =
at
>>>>> Apache than at github.
>>>>>
>>>>
>>>>> -- Owen
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Eric Sammer
>>>> twitter: esammer
>>>> data: www.cloudera.com
>>>
>>>
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Chris Mattmann, Ph.D.
>>> Senior Computer Scientist
>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> Office: 171-266B, Mailstop: 171-246
>>> Email: chris.a.mattmann@nasa.gov
>>> WWW: =A0 http://sunset.usc.edu/~mattmann/
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Adjunct Assistant Professor, Computer Science Department
>>> University of Southern California, Los Angeles, CA 90089 USA
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>
>>>
>
>

From general-return-3047-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 15 21:13:03 2011
Return-Path: <general-return-3047-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 15455 invoked from network); 15 Feb 2011 21:13:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Feb 2011 21:13:02 -0000
Received: (qmail 91322 invoked by uid 500); 15 Feb 2011 21:13:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91240 invoked by uid 500); 15 Feb 2011 21:13:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91232 invoked by uid 99); 15 Feb 2011 21:12:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Feb 2011 21:12:59 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Feb 2011 21:12:53 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p1FLBsYa001896
	for <general@hadoop.apache.org>; Tue, 15 Feb 2011 13:11:54 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1297804314;
	bh=ukYQhpad+b0QfsNItplIv+36Rm0t+yjmBuYEh8kxn5A=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=bywJkJIXhs+Jgvt8KrAvL5pmQ0TNJQIU9GNaINsGv+Jnf4M8ITNKW/4i5v95YzbpB
	 L8gOB0UIeM+kzufkvfT+xexbnQoUEcWhgBwiuKeA5z537k2iLoq21Q7jsKEfz1kY+3
	 2EiW/NLHSSNlHT+Owk7Z+7Ug8JluMOymnzOQWhMc=
Message-Id: <E9441B93-BE33-4383-AFAF-0565402A3E53@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <33A12FDB-820A-4CCC-9AB7-0521B41952B1@mac.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Abandon mrunit MapReduce contrib
Date: Tue, 15 Feb 2011 13:11:53 -0800
References: <77405974-6771-4604-926B-976240743F3C@mac.com> <AANLkTim80KJ3f14iM6Ltzy7UG613myhN-AiwQ3pCmVdi@mail.gmail.com> <AANLkTiknkrWK=GzGMVj4_Z6p5Jn8TQu1P5TkaFSgORJk@mail.gmail.com> <AANLkTin0TguLYHi9X0uL3ZF7tCz5xTHBYa7056oKF2Ae@mail.gmail.com> <93644D3B-4830-4813-92BE-809738969CF9@apache.org> <AANLkTi=dbGSLhtFA+Rf9LgaX05v45HDLoQDE_2C5ULAf@mail.gmail.com> <65311075-A30E-4971-A2BA-C75409C6486E@jpl.nasa.gov> <AANLkTi=XbpVUmg_TpH7eqEn8w7Gcu9KiwtJmKMLa5w5n@mail.gmail.com> <FAE19BAA-47A7-4747-BDD8-5AA42C259EB6@jpl.nasa.gov> <AANLkTin2JvOGdZT5g5tmTGeCpD3Qz2soZWcTQtHMsSza@mail.gmail.com> <AANLkTim+FQuoS+mR7YuTf0kw1KVxCdy14wDjf2u_cZdB@mail.gmail.com> <33A12FDB-820A-4CCC-9AB7-0521B41952B1@mac.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

On Feb 11, 2011, at 9:57 PM, Nigel Daley wrote:

> This is great!  So we'll leave mrunit in contrib until it can be  
> moved to incubator.
>

+1



From general-return-3048-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 15 21:59:15 2011
Return-Path: <general-return-3048-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29218 invoked from network); 15 Feb 2011 21:59:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Feb 2011 21:59:15 -0000
Received: (qmail 45375 invoked by uid 500); 15 Feb 2011 21:59:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45046 invoked by uid 500); 15 Feb 2011 21:59:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45032 invoked by uid 99); 15 Feb 2011 21:59:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Feb 2011 21:59:13 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Feb 2011 21:59:09 +0000
Received: by wye20 with SMTP id 20so667307wye.35
        for <general@hadoop.apache.org>; Tue, 15 Feb 2011 13:58:47 -0800 (PST)
Received: by 10.216.24.132 with SMTP id x4mr1327956wex.81.1297807127585; Tue,
 15 Feb 2011 13:58:47 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.82.194 with HTTP; Tue, 15 Feb 2011 13:58:27 -0800 (PST)
From: Konstantin Boudnik <cos@apache.org>
Date: Tue, 15 Feb 2011 13:58:27 -0800
X-Google-Sender-Auth: ukTjFoxD4LQBlTgdfbzmbvjZdyQ
Message-ID: <AANLkTi=EvhSotauym5K080wy0+UORW+W2Nuh4by5JgdZ@mail.gmail.com>
Subject: Hadoop testing project [Was: [VOTE] Abandon mrunit MapReduce contrib]
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

While MrUnit discussion draws to its natural conclusion I would like
to bring up another point which might be well aligned with that
discussion. Patrick Hunt has brought up this idea earlier today and I
believe it has to be elaborated further.

A number of testing projects both for Hadoop and Hadoop-related
component were brought to life over last year or two. Among those are
MRUnit, PigUnit, YCSB, Herriot, and perhaps a few more. They all
focusing on more or less the same problem e.g. validation of Hadoop or
on-top-of-Hadoop components, or application level testing for Hadoop.
However, the fact that they all are spread across a wide variety of
projects seems to confuse/mislead Hadoop users.

How about incubating a bigger Hadoop (Pig, Oozie, HBase) testing
project which will take care about development and support of common
(where's possible) tools, frameworks and the like? Please feel free to
share your thoughts :)
--
  Take care,
Konstantin (Cos) Boudnik


On Tue, Feb 15, 2011 at 10:44, Eric Sammer <esammer@cloudera.com> wrote:
> I've started the wiki page proposal for Incubator for mrunit. I'll ping
> people off list for mentoring. Much appreciated for all the help!
>
> On Tue, Feb 15, 2011 at 1:36 PM, Nigel Daley <ndaley@mac.com> wrote:
>
>> I'm happy to help mentor as well.
>>
>> Cheers,
>> Nige
>>
>> On Feb 11, 2011, at 11:52 AM, Patrick Hunt wrote:
>>
>> > On Fri, Feb 11, 2011 at 9:44 AM, Mattmann, Chris A (388J)
>> > <chris.a.mattmann@jpl.nasa.gov> wrote:
>> >> Guys, BTW, if you need help or a mentor in Apache Incubator-ville for
>> MRUnit, I would be happy to help.
>> >
>> > I was going to suggest the same thing (mrunit to incubator). I would
>> > also be happy to be a mentor.
>> >
>> > Patrick

From general-return-3049-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 15 22:14:45 2011
Return-Path: <general-return-3049-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42547 invoked from network); 15 Feb 2011 22:14:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Feb 2011 22:14:44 -0000
Received: (qmail 62871 invoked by uid 500); 15 Feb 2011 22:14:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62763 invoked by uid 500); 15 Feb 2011 22:14:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62754 invoked by uid 99); 15 Feb 2011 22:14:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Feb 2011 22:14:42 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [128.149.139.109] (HELO mail.jpl.nasa.gov) (128.149.139.109)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Feb 2011 22:14:35 +0000
Received: from mail.jpl.nasa.gov (altvirehtstap01.jpl.nasa.gov [128.149.137.72])
	by smtp.jpl.nasa.gov (Switch-3.4.3/Switch-3.4.3) with ESMTP id p1FMEC66009049
	(using TLSv1/SSLv3 with cipher RC4-MD5 (128 bits) verified FAIL)
	for <general@hadoop.apache.org>; Tue, 15 Feb 2011 14:14:12 -0800
Received: from ALTPHYEMBEVSP20.RES.AD.JPL ([128.149.137.82]) by
 ALTVIREHTSTAP01.RES.AD.JPL ([128.149.137.72]) with mapi; Tue, 15 Feb 2011
 14:14:11 -0800
From: "Mattmann, Chris A (388J)" <chris.a.mattmann@jpl.nasa.gov>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 15 Feb 2011 14:14:03 -0800
Subject: Re: Hadoop testing project [Was: [VOTE] Abandon mrunit MapReduce
 contrib]
Thread-Topic: Hadoop testing project [Was: [VOTE] Abandon mrunit MapReduce
 contrib]
Thread-Index: AcvNXaxJTZGJm0h5QtCPJlVpzb4q0w==
Message-ID: <2BF0E28E-32C9-4EA1-844D-035588928B53@jpl.nasa.gov>
References: <AANLkTi=EvhSotauym5K080wy0+UORW+W2Nuh4by5JgdZ@mail.gmail.com>
In-Reply-To: <AANLkTi=EvhSotauym5K080wy0+UORW+W2Nuh4by5JgdZ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Source-IP: altvirehtstap01.jpl.nasa.gov [128.149.137.72]
X-Source-Sender: chris.a.mattmann@jpl.nasa.gov
X-AUTH: Authorized
X-Virus-Checked: Checked by ClamAV on apache.org

Sounds good to me, Cos. I'm fine to help/mentor with either one that ends u=
p standing when the dust clears :)

Cheers,
Chris

On Feb 15, 2011, at 1:58 PM, Konstantin Boudnik wrote:

> While MrUnit discussion draws to its natural conclusion I would like
> to bring up another point which might be well aligned with that
> discussion. Patrick Hunt has brought up this idea earlier today and I
> believe it has to be elaborated further.
>=20
> A number of testing projects both for Hadoop and Hadoop-related
> component were brought to life over last year or two. Among those are
> MRUnit, PigUnit, YCSB, Herriot, and perhaps a few more. They all
> focusing on more or less the same problem e.g. validation of Hadoop or
> on-top-of-Hadoop components, or application level testing for Hadoop.
> However, the fact that they all are spread across a wide variety of
> projects seems to confuse/mislead Hadoop users.
>=20
> How about incubating a bigger Hadoop (Pig, Oozie, HBase) testing
> project which will take care about development and support of common
> (where's possible) tools, frameworks and the like? Please feel free to
> share your thoughts :)
> --
>  Take care,
> Konstantin (Cos) Boudnik
>=20
>=20
> On Tue, Feb 15, 2011 at 10:44, Eric Sammer <esammer@cloudera.com> wrote:
>> I've started the wiki page proposal for Incubator for mrunit. I'll ping
>> people off list for mentoring. Much appreciated for all the help!
>>=20
>> On Tue, Feb 15, 2011 at 1:36 PM, Nigel Daley <ndaley@mac.com> wrote:
>>=20
>>> I'm happy to help mentor as well.
>>>=20
>>> Cheers,
>>> Nige
>>>=20
>>> On Feb 11, 2011, at 11:52 AM, Patrick Hunt wrote:
>>>=20
>>>> On Fri, Feb 11, 2011 at 9:44 AM, Mattmann, Chris A (388J)
>>>> <chris.a.mattmann@jpl.nasa.gov> wrote:
>>>>> Guys, BTW, if you need help or a mentor in Apache Incubator-ville for
>>> MRUnit, I would be happy to help.
>>>>=20
>>>> I was going to suggest the same thing (mrunit to incubator). I would
>>>> also be happy to be a mentor.
>>>>=20
>>>> Patrick


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


From general-return-3050-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 15 22:15:44 2011
Return-Path: <general-return-3050-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42911 invoked from network); 15 Feb 2011 22:15:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Feb 2011 22:15:44 -0000
Received: (qmail 64404 invoked by uid 500); 15 Feb 2011 22:15:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 64344 invoked by uid 500); 15 Feb 2011 22:15:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 64336 invoked by uid 99); 15 Feb 2011 22:15:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Feb 2011 22:15:41 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of esammer@cloudera.com designates 209.85.220.176 as permitted sender)
Received: from [209.85.220.176] (HELO mail-vx0-f176.google.com) (209.85.220.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Feb 2011 22:15:35 +0000
Received: by vxb37 with SMTP id 37so493957vxb.35
        for <general@hadoop.apache.org>; Tue, 15 Feb 2011 14:15:14 -0800 (PST)
MIME-Version: 1.0
Received: by 10.220.181.138 with SMTP id by10mr1642299vcb.92.1297808113497;
 Tue, 15 Feb 2011 14:15:13 -0800 (PST)
Received: by 10.220.5.80 with HTTP; Tue, 15 Feb 2011 14:15:13 -0800 (PST)
In-Reply-To: <AANLkTi=EvhSotauym5K080wy0+UORW+W2Nuh4by5JgdZ@mail.gmail.com>
References: <AANLkTi=EvhSotauym5K080wy0+UORW+W2Nuh4by5JgdZ@mail.gmail.com>
Date: Tue, 15 Feb 2011 17:15:13 -0500
Message-ID: <AANLkTinQ9iVjdjYr7eeLbXHts4+sFNA1-h4hY7KDw+wk@mail.gmail.com>
Subject: Re: Hadoop testing project [Was: [VOTE] Abandon mrunit MapReduce contrib]
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba4fc3b697ddd5049c597fb0
X-Virus-Checked: Checked by ClamAV on apache.org

--90e6ba4fc3b697ddd5049c597fb0
Content-Type: text/plain; charset=ISO-8859-1

I think this is a good idea. The only thing I think is that it may make
sense to split such an effort into two components: one for the testing of
Hadoop and the projects themselves and one to test end user applications and
libraries. Performance testing tools like YCSB are probably more in the
former camp where mrunit is the latter, as a for instance. I think it's
important to have separate artifacts to minimize uber-jar issues (or
contrib-like situations where release cycles are coupled).

On Tue, Feb 15, 2011 at 4:58 PM, Konstantin Boudnik <cos@apache.org> wrote:

> While MrUnit discussion draws to its natural conclusion I would like
> to bring up another point which might be well aligned with that
> discussion. Patrick Hunt has brought up this idea earlier today and I
> believe it has to be elaborated further.
>
> A number of testing projects both for Hadoop and Hadoop-related
> component were brought to life over last year or two. Among those are
> MRUnit, PigUnit, YCSB, Herriot, and perhaps a few more. They all
> focusing on more or less the same problem e.g. validation of Hadoop or
> on-top-of-Hadoop components, or application level testing for Hadoop.
> However, the fact that they all are spread across a wide variety of
> projects seems to confuse/mislead Hadoop users.
>
> How about incubating a bigger Hadoop (Pig, Oozie, HBase) testing
> project which will take care about development and support of common
> (where's possible) tools, frameworks and the like? Please feel free to
> share your thoughts :)
> --
>  Take care,
> Konstantin (Cos) Boudnik
>
>
> On Tue, Feb 15, 2011 at 10:44, Eric Sammer <esammer@cloudera.com> wrote:
> > I've started the wiki page proposal for Incubator for mrunit. I'll ping
> > people off list for mentoring. Much appreciated for all the help!
> >
> > On Tue, Feb 15, 2011 at 1:36 PM, Nigel Daley <ndaley@mac.com> wrote:
> >
> >> I'm happy to help mentor as well.
> >>
> >> Cheers,
> >> Nige
> >>
> >> On Feb 11, 2011, at 11:52 AM, Patrick Hunt wrote:
> >>
> >> > On Fri, Feb 11, 2011 at 9:44 AM, Mattmann, Chris A (388J)
> >> > <chris.a.mattmann@jpl.nasa.gov> wrote:
> >> >> Guys, BTW, if you need help or a mentor in Apache Incubator-ville for
> >> MRUnit, I would be happy to help.
> >> >
> >> > I was going to suggest the same thing (mrunit to incubator). I would
> >> > also be happy to be a mentor.
> >> >
> >> > Patrick
>



-- 
Eric Sammer
twitter: esammer
data: www.cloudera.com

--90e6ba4fc3b697ddd5049c597fb0--

From general-return-3051-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 15 23:55:24 2011
Return-Path: <general-return-3051-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 75651 invoked from network); 15 Feb 2011 23:55:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Feb 2011 23:55:24 -0000
Received: (qmail 1936 invoked by uid 500); 15 Feb 2011 23:55:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1874 invoked by uid 500); 15 Feb 2011 23:55:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1805 invoked by uid 99); 15 Feb 2011 23:55:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Feb 2011 23:55:22 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Feb 2011 23:55:17 +0000
Received: by wwd20 with SMTP id 20so786131wwd.29
        for <general@hadoop.apache.org>; Tue, 15 Feb 2011 15:54:55 -0800 (PST)
Received: by 10.216.24.132 with SMTP id x4mr1408374wex.81.1297814095447; Tue,
 15 Feb 2011 15:54:55 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.82.194 with HTTP; Tue, 15 Feb 2011 15:54:35 -0800 (PST)
In-Reply-To: <AANLkTinQ9iVjdjYr7eeLbXHts4+sFNA1-h4hY7KDw+wk@mail.gmail.com>
References: <AANLkTi=EvhSotauym5K080wy0+UORW+W2Nuh4by5JgdZ@mail.gmail.com> <AANLkTinQ9iVjdjYr7eeLbXHts4+sFNA1-h4hY7KDw+wk@mail.gmail.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Tue, 15 Feb 2011 15:54:35 -0800
X-Google-Sender-Auth: fnnnr2F39cp4crvIsC4McnJcowc
Message-ID: <AANLkTi=RmQKWiG5fA+CSmF3p3Xn-S02zCT+Zhd1T2dgP@mail.gmail.com>
Subject: Re: Hadoop testing project [Was: [VOTE] Abandon mrunit MapReduce contrib]
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Tue, Feb 15, 2011 at 14:15, Eric Sammer <esammer@cloudera.com> wrote:
> I think this is a good idea. The only thing I think is that it may make
> sense to split such an effort into two components: one for the testing of
> Hadoop and the projects themselves and one to test end user applications =
and

I expect to see even greater number of component, to be honest. E.g. a
harness to run stacks testing (which as has been discussed with HBase
folks might utilize YCSB artifacts). Which doesn't invalidate the
purpose of central Hadoop testing project or whatever we might call
it.

> libraries. Performance testing tools like YCSB are probably more in the
> former camp where mrunit is the latter, as a for instance. I think it's
> important to have separate artifacts to minimize uber-jar issues (or
> contrib-like situations where release cycles are coupled).

Having separate artifacts/release cycles would be pretty important for
another reason too: test artifacts might undergo significant changes
between releases of a product. Thus requiring using different versions
of such validating artifacts for differently composed Hadoop stacks.
Uber-jar are proven to be inflexible and pain to deal with.

Cos

> On Tue, Feb 15, 2011 at 4:58 PM, Konstantin Boudnik <cos@apache.org> wrot=
e:
>
>> While MrUnit discussion draws to its natural conclusion I would like
>> to bring up another point which might be well aligned with that
>> discussion. Patrick Hunt has brought up this idea earlier today and I
>> believe it has to be elaborated further.
>>
>> A number of testing projects both for Hadoop and Hadoop-related
>> component were brought to life over last year or two. Among those are
>> MRUnit, PigUnit, YCSB, Herriot, and perhaps a few more. They all
>> focusing on more or less the same problem e.g. validation of Hadoop or
>> on-top-of-Hadoop components, or application level testing for Hadoop.
>> However, the fact that they all are spread across a wide variety of
>> projects seems to confuse/mislead Hadoop users.
>>
>> How about incubating a bigger Hadoop (Pig, Oozie, HBase) testing
>> project which will take care about development and support of common
>> (where's possible) tools, frameworks and the like? Please feel free to
>> share your thoughts :)
>> --
>> =A0Take care,
>> Konstantin (Cos) Boudnik
>>
>>
>> On Tue, Feb 15, 2011 at 10:44, Eric Sammer <esammer@cloudera.com> wrote:
>> > I've started the wiki page proposal for Incubator for mrunit. I'll pin=
g
>> > people off list for mentoring. Much appreciated for all the help!
>> >
>> > On Tue, Feb 15, 2011 at 1:36 PM, Nigel Daley <ndaley@mac.com> wrote:
>> >
>> >> I'm happy to help mentor as well.
>> >>
>> >> Cheers,
>> >> Nige
>> >>
>> >> On Feb 11, 2011, at 11:52 AM, Patrick Hunt wrote:
>> >>
>> >> > On Fri, Feb 11, 2011 at 9:44 AM, Mattmann, Chris A (388J)
>> >> > <chris.a.mattmann@jpl.nasa.gov> wrote:
>> >> >> Guys, BTW, if you need help or a mentor in Apache Incubator-ville =
for
>> >> MRUnit, I would be happy to help.
>> >> >
>> >> > I was going to suggest the same thing (mrunit to incubator). I woul=
d
>> >> > also be happy to be a mentor.
>> >> >
>> >> > Patrick
>>
>
>
>
> --
> Eric Sammer
> twitter: esammer
> data: www.cloudera.com
>

From general-return-3052-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 16 11:38:10 2011
Return-Path: <general-return-3052-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 75816 invoked from network); 16 Feb 2011 11:38:10 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Feb 2011 11:38:10 -0000
Received: (qmail 35230 invoked by uid 500); 16 Feb 2011 11:38:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34812 invoked by uid 500); 16 Feb 2011 11:38:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34801 invoked by uid 99); 16 Feb 2011 11:38:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Feb 2011 11:38:02 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Feb 2011 11:37:54 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 4E190B7DD1
	for <general@hadoop.apache.org>; Wed, 16 Feb 2011 11:37:32 +0000 (GMT)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id VPBUhMdLSxR3 for <general@hadoop.apache.org>;
	Wed, 16 Feb 2011 11:37:21 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 5A581B7DC1
	for <general@hadoop.apache.org>; Wed, 16 Feb 2011 11:37:20 +0000 (GMT)
MailScanner-NULL-Check: 1298461028.36992@5x2sSncegTd3NQ21xZC26g
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p1GBb7vO013393
	for <general@hadoop.apache.org>; Wed, 16 Feb 2011 11:37:07 GMT
Message-ID: <4D5BB6E4.8040403@apache.org>
Date: Wed, 16 Feb 2011 11:37:08 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Hadoop testing project [Was: [VOTE] Abandon mrunit MapReduce
 contrib]
References: <AANLkTi=EvhSotauym5K080wy0+UORW+W2Nuh4by5JgdZ@mail.gmail.com>
In-Reply-To: <AANLkTi=EvhSotauym5K080wy0+UORW+W2Nuh4by5JgdZ@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p1GBb7vO013393
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 15/02/11 21:58, Konstantin Boudnik wrote:
> While MrUnit discussion draws to its natural conclusion I would like
> to bring up another point which might be well aligned with that
> discussion. Patrick Hunt has brought up this idea earlier today and I
> believe it has to be elaborated further.
>
> A number of testing projects both for Hadoop and Hadoop-related
> component were brought to life over last year or two. Among those are
> MRUnit, PigUnit, YCSB, Herriot, and perhaps a few more. They all
> focusing on more or less the same problem e.g. validation of Hadoop or
> on-top-of-Hadoop components, or application level testing for Hadoop.
> However, the fact that they all are spread across a wide variety of
> projects seems to confuse/mislead Hadoop users.
>
> How about incubating a bigger Hadoop (Pig, Oozie, HBase) testing
> project which will take care about development and support of common
> (where's possible) tools, frameworks and the like? Please feel free to
> share your thoughts :)
> --

I think it would be good though specific projects will need/have their 
own testing needs -I'd expect more focus for testing redistributables to 
be on helping Hadoop users test their stuff against subsets of data, 
rather than the hadoop-*-dev problem of "stressing the hadoop stack once 
your latest patch is applied".

That said, the whole problem of qualifying an OS, Java release and 
cluster is something we'd expect most end user teams to have to do 
-right now terasort is the main stress test.

From general-return-3053-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 16 18:23:15 2011
Return-Path: <general-return-3053-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 26431 invoked from network); 16 Feb 2011 18:23:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Feb 2011 18:23:14 -0000
Received: (qmail 31664 invoked by uid 500); 16 Feb 2011 18:23:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31307 invoked by uid 500); 16 Feb 2011 18:23:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31299 invoked by uid 99); 16 Feb 2011 18:23:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Feb 2011 18:23:09 +0000
X-ASF-Spam-Status: No, hits=-5.0 required=5.0
	tests=RCVD_IN_DNSWL_HI
X-Spam-Check-By: apache.org
Received-SPF: unknown (nike.apache.org: error in processing during lookup of jrottinghuis@ebay.com)
Received: from [216.33.244.7] (HELO rhv-mipot-002.corp.ebay.com) (216.33.244.7)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Feb 2011 18:23:04 +0000
DomainKey-Signature: s=corp; d=ebay.com; c=nofws; q=dns;
  h=X-EBay-Corp:X-IronPort-AV:Received:Received:From:To:Date:
   Subject:Thread-Topic:Thread-Index:Message-ID:References:
   In-Reply-To:Accept-Language:Content-Language:
   X-MS-Has-Attach:X-MS-TNEF-Correlator:acceptlanguage:
   x-ems-proccessed:x-ems-stamp:Content-Type:
   Content-Transfer-Encoding:MIME-Version:X-CFilter;
  b=StNSjyu6ELnBkcjrMLMQd38rERqj2t4w6/Oh7FJkMJCJd1QS91Fkx54K
   Hb1D+VbjWwA6EQtoKPxaaHD4i1nic1iYJXv6uMu7xmFr1vYnvGphNNVr8
   n70Ldn3MwEwAnFP;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=ebay.com; i=jrottinghuis@ebay.com; q=dns/txt; s=corp;
  t=1297880584; x=1329416584;
  h=from:to:date:subject:message-id:references:in-reply-to:
   content-transfer-encoding:mime-version;
  z=From:=20"Rottinghuis,=20Joep"=20<jrottinghuis@ebay.com>
   |To:=20"general@hadoop.apache.org"=20<general@hadoop.apac
   he.org>|Date:=20Wed,=2016=20Feb=202011=2010:19:21=20-0800
   |Subject:=20RE:=20Hadoop=20testing=20project=20[Was:=20[V
   OTE]=20Abandon=20mrunit=20MapReduce=0D=0A=20contrib]
   |Message-ID:=20<F4FCE59AE46B2243922F224426BC869B01AA87C5F
   F@RHV-MEXMS-002.corp.ebay.com>|References:=20<AANLkTi=3DE
   vhSotauym5K080wy0+UORW+W2Nuh4by5JgdZ@mail.gmail.com>
   |In-Reply-To:=20<AANLkTi=3DEvhSotauym5K080wy0+UORW+W2Nuh4
   by5JgdZ@mail.gmail.com>|Content-Transfer-Encoding:=20quot
   ed-printable|MIME-Version:=201.0;
  bh=mZkHYMGK7qbymBEooUn22UIP4L0uo/poC5xHSEm7kJQ=;
  b=3XsCcNsuuQuL2X5a4zWRbtBcDHMxnJ+CXhY9ViJjDrb1Q43qdnQ31+8E
   EIWYFrH0hBfAgmTjbhnKk9QFHZ9cRjfIjVgHpw183n0hRRPXoHklVoSTP
   JBH37Ei0Z2qLwiO;
X-EBay-Corp: Yes
X-IronPort-AV: E=Sophos;i="4.60,480,1291622400"; 
   d="scan'208";a="8310883"
Received: from rhv-vtenf-001.corp.ebay.com (HELO RHV-MEXHT-001.corp.ebay.com) ([10.112.113.52])
  by rhv-mipot-002.corp.ebay.com with ESMTP; 16 Feb 2011 10:22:40 -0800
Received: from RHV-MEXMS-002.corp.ebay.com ([10.245.17.114]) by
 RHV-MEXHT-001.corp.ebay.com ([10.245.24.100]) with mapi; Wed, 16 Feb 2011
 10:22:39 -0800
From: "Rottinghuis, Joep" <jrottinghuis@ebay.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 16 Feb 2011 10:19:21 -0800
Subject: RE: Hadoop testing project [Was: [VOTE] Abandon mrunit MapReduce
 contrib]
Thread-Topic: Hadoop testing project [Was: [VOTE] Abandon mrunit MapReduce
 contrib]
Thread-Index: AcvNW5f4MNkIM+lgQ+aUgeRxC0GldQAqnCE5
Message-ID: <F4FCE59AE46B2243922F224426BC869B01AA87C5FF@RHV-MEXMS-002.corp.ebay.com>
References: <AANLkTi=EvhSotauym5K080wy0+UORW+W2Nuh4by5JgdZ@mail.gmail.com>
In-Reply-To: <AANLkTi=EvhSotauym5K080wy0+UORW+W2Nuh4by5JgdZ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
x-ems-proccessed: 10SqDH0iR7ekR7SRpKqm5A==
x-ems-stamp: qp9wncopcho0EvOFWQ9ePw==
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-CFilter: Scanned
X-Virus-Checked: Checked by ClamAV on apache.org

+1
Having a coherent approach for system level testing increases confidence in=
 the various Hadoop releases and and will reduce the effort to take any (se=
t of ) changes from development into production.
The more automated and formalized system testing, the better!

Thanks,

Joep
________________________________________
From: cos@boudnik.org [cos@boudnik.org] On Behalf Of Konstantin Boudnik [co=
s@apache.org]
Sent: Tuesday, February 15, 2011 1:58 PM
To: general@hadoop.apache.org
Subject: Hadoop testing project [Was: [VOTE] Abandon mrunit MapReduce contr=
ib]

While MrUnit discussion draws to its natural conclusion I would like
to bring up another point which might be well aligned with that
discussion. Patrick Hunt has brought up this idea earlier today and I
believe it has to be elaborated further.

A number of testing projects both for Hadoop and Hadoop-related
component were brought to life over last year or two. Among those are
MRUnit, PigUnit, YCSB, Herriot, and perhaps a few more. They all
focusing on more or less the same problem e.g. validation of Hadoop or
on-top-of-Hadoop components, or application level testing for Hadoop.
However, the fact that they all are spread across a wide variety of
projects seems to confuse/mislead Hadoop users.

How about incubating a bigger Hadoop (Pig, Oozie, HBase) testing
project which will take care about development and support of common
(where's possible) tools, frameworks and the like? Please feel free to
share your thoughts :)
--
  Take care,
Konstantin (Cos) Boudnik


On Tue, Feb 15, 2011 at 10:44, Eric Sammer <esammer@cloudera.com> wrote:
> I've started the wiki page proposal for Incubator for mrunit. I'll ping
> people off list for mentoring. Much appreciated for all the help!
>
> On Tue, Feb 15, 2011 at 1:36 PM, Nigel Daley <ndaley@mac.com> wrote:
>
>> I'm happy to help mentor as well.
>>
>> Cheers,
>> Nige
>>
>> On Feb 11, 2011, at 11:52 AM, Patrick Hunt wrote:
>>
>> > On Fri, Feb 11, 2011 at 9:44 AM, Mattmann, Chris A (388J)
>> > <chris.a.mattmann@jpl.nasa.gov> wrote:
>> >> Guys, BTW, if you need help or a mentor in Apache Incubator-ville for
>> MRUnit, I would be happy to help.
>> >
>> > I was going to suggest the same thing (mrunit to incubator). I would
>> > also be happy to be a mentor.
>> >
>> > Patrick=

From general-return-3054-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 16 19:50:47 2011
Return-Path: <general-return-3054-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 89171 invoked from network); 16 Feb 2011 19:50:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Feb 2011 19:50:47 -0000
Received: (qmail 36721 invoked by uid 500); 16 Feb 2011 19:50:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36494 invoked by uid 500); 16 Feb 2011 19:50:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36486 invoked by uid 99); 16 Feb 2011 19:50:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Feb 2011 19:50:44 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.27.212] (HELO qmta14.emeryville.ca.mail.comcast.net) (76.96.27.212)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Feb 2011 19:50:36 +0000
Received: from omta14.emeryville.ca.mail.comcast.net ([76.96.30.60])
	by qmta14.emeryville.ca.mail.comcast.net with comcast
	id 8jjK1g0021HpZEsAEjqEKZ; Wed, 16 Feb 2011 19:50:14 +0000
Received: from fs ([24.4.185.157])
	by omta14.emeryville.ca.mail.comcast.net with comcast
	id 8jqD1g0073QAh8g8ajqEQc; Wed, 16 Feb 2011 19:50:14 +0000
Received: from mail.boudnik.org (localhost [127.0.0.1])
	by fs (8.14.3/8.14.3/Debian-5+lenny1) with ESMTP id p1GJoDrG029705
	for <general@hadoop.apache.org>; Wed, 16 Feb 2011 11:50:13 -0800
Received: (from cos@localhost)
	by mail.boudnik.org (8.14.3/8.14.3/Submit) id p1GJoCq1029704
	for general@hadoop.apache.org; Wed, 16 Feb 2011 11:50:12 -0800
X-Authentication-Warning: mail.boudnik.org: cos set sender to cos@apache.org using -f
Date: Wed, 16 Feb 2011 11:50:12 -0800
From: Konstantin Boudnik <cos@apache.org>
To: general@hadoop.apache.org
Subject: Re: Hadoop testing project
Message-ID: <20110216195012.GA29594@linspire.com>
References: <AANLkTi=EvhSotauym5K080wy0+UORW+W2Nuh4by5JgdZ@mail.gmail.com> <4D5BB6E4.8040403@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <4D5BB6E4.8040403@apache.org>
X-Organization: It's something of 'Cos
User-Agent: Mutt/1.5.18 (2008-05-17)
X-Virus-Checked: Checked by ClamAV on apache.org

Steve.

If the project under discussion will provide a common harness where such a test
artifact (think of a Maven artifact for example) will click and will be
executed automatically with all needed tools and dependencies resolved for you
- would it be appealing for end-users' cause?

As Joep said this "...will reduce the effort to take any (set of ) changes
from development into production." Take it one step further: when your cluster
is 'assembled' you need to validate it (on top of a concrete OS, etc.); is it
desirable to follow N-steps process to bring about whatever testing work-load
you need or you'd prefer to simply do something like:

    wget http://workloads.internal.mydomain.com/stackValidations/v12.4.pom \
        && mvn verify

and check the results later on?

These gonna be the same tools that dev. use for their tasks although worksets
will be different. So what?

Cos

On Wed, Feb 16, 2011 at 11:37AM, Steve Loughran wrote:
> On 15/02/11 21:58, Konstantin Boudnik wrote:
>> While MrUnit discussion draws to its natural conclusion I would like
>> to bring up another point which might be well aligned with that
>> discussion. Patrick Hunt has brought up this idea earlier today and I
>> believe it has to be elaborated further.
>>
>> A number of testing projects both for Hadoop and Hadoop-related
>> component were brought to life over last year or two. Among those are
>> MRUnit, PigUnit, YCSB, Herriot, and perhaps a few more. They all
>> focusing on more or less the same problem e.g. validation of Hadoop or
>> on-top-of-Hadoop components, or application level testing for Hadoop.
>> However, the fact that they all are spread across a wide variety of
>> projects seems to confuse/mislead Hadoop users.
>>
>> How about incubating a bigger Hadoop (Pig, Oozie, HBase) testing
>> project which will take care about development and support of common
>> (where's possible) tools, frameworks and the like? Please feel free to
>> share your thoughts :)
>> --
>
> I think it would be good though specific projects will need/have their  
> own testing needs -I'd expect more focus for testing redistributables to  
> be on helping Hadoop users test their stuff against subsets of data,  
> rather than the hadoop-*-dev problem of "stressing the hadoop stack once  
> your latest patch is applied".
>
> That said, the whole problem of qualifying an OS, Java release and  
> cluster is something we'd expect most end user teams to have to do  
> -right now terasort is the main stress test.

From general-return-3055-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 17 04:26:40 2011
Return-Path: <general-return-3055-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24548 invoked from network); 17 Feb 2011 04:26:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Feb 2011 04:26:40 -0000
Received: (qmail 85839 invoked by uid 500); 17 Feb 2011 04:26:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85387 invoked by uid 500); 17 Feb 2011 04:26:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85374 invoked by uid 99); 17 Feb 2011 04:26:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 04:26:36 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 04:26:31 +0000
Received: from traveling-laptop-66-59.eglbp.corp.yahoo.com (traveling-laptop-66-59.eglbp.corp.yahoo.com [10.66.66.59])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p1H4PpFM096613
	for <general@hadoop.apache.org>; Wed, 16 Feb 2011 20:25:52 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1297916753;
	bh=KFZEXKduCdhbxyzZtj1W7GmU4uJBWmdW2bCU8hUbE6k=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=PbGMnItxE6op8zndCS7sRn/bTERUrfz4rbxoJvx6TUs5OlMBem/vzsEm3PdTLwkyN
	 f5000cvsq1L5Q0xq6fP552jS4qYPZBScosMuMK32+uLJBfsLBtKCtlnATZc97lHZ9c
	 4QpT/aogpsMCWk1KiqkI3R2AvT2gcuTDuc18vvD0=
Message-Id: <A9878555-698E-4CD9-A5CE-765D9C36520A@yahoo-inc.com>
From: Sanjay Radia <sradia@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <AANLkTim_dNOdj=HUrsx7ZhvG3_QXWerB_EVaakMuT68w@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSS] Move common, hdfs, mapreduce contrib components to apache-extras.org or elsewhere
Date: Thu, 17 Feb 2011 09:55:52 +0530
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com> <FD5BC1D3-881A-4D45-BA2C-C6084C884418@apache.org> <AANLkTim_dNOdj=HUrsx7ZhvG3_QXWerB_EVaakMuT68w@mail.gmail.com>
X-Mailer: Apple Mail (2.936)


On Jan 31, 2011, at 10:19 PM, Konstantin Boudnik wrote:

> On Sun, Jan 30, 2011 at 23:19, Owen O'Malley <omalley@apache.org>  
> wrote:
>>
>> On Jan 30, 2011, at 7:42 PM, Nigel Daley wrote:
>>
>>> Now that http://apache-extras.org is launched
>>> (https://blogs.apache.org/foundation/entry/the_apache_software_foundation_launches 
>>> )
>>> I'd like to start a discussion on moving contrib components out of  
>>> common,
>>> mapreduce, and hdfs.
>>
>> The PMC can't "move" code to Apache extras. It can only choose to  
>> abandon
>> code that it doesn't want to support any longer. As a separate  
>> action some
>> group of developers may create projects in Apache Extras based on  
>> the code
>> from Hadoop.
>>
>> Therefore the question is really what if any code Hadoop wants to  
>> abandon.
>> That is a good question and one that we should ask ourselves  
>> occasionally.
>>
>> After a quick consideration, my personal list would look like:
>>
>> failmon
>> fault injection
>
> This is the best way to kill a project as tightly coupled with the
> core code as fault injection.
>
> So, if you really want to kill it - then move it.


Nigel/Owen did not say "kill it". Folks were simply listing potential  
projects to move out.
If you feel that it should stay in then simply say so and give the  
reasons -- looks like your reason is "tight coupling".


sanjay

>
>> fuse-dfs
>> hod
>> kfs
>>
>> Also note that pushing code out of Hadoop has a high cost. There  
>> are at
>> least 3 forks of the hadoop-gpl-compression code. That creates a  
>> lot of
>> confusion for the users. A lot of users never go to the work to  
>> figure out
>> which fork and branch of hadoop-gpl-compression work with the  
>> version of
>> Hadoop they installed.
>>
>> -- Owen
>>
>>


From general-return-3056-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 17 04:32:39 2011
Return-Path: <general-return-3056-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25309 invoked from network); 17 Feb 2011 04:32:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Feb 2011 04:32:38 -0000
Received: (qmail 88861 invoked by uid 500); 17 Feb 2011 04:32:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88551 invoked by uid 500); 17 Feb 2011 04:32:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88543 invoked by uid 99); 17 Feb 2011 04:32:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 04:32:34 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 04:32:27 +0000
Received: from traveling-laptop-66-59.eglbp.corp.yahoo.com (traveling-laptop-66-59.eglbp.corp.yahoo.com [10.66.66.59])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p1H4VgDG098126
	for <general@hadoop.apache.org>; Wed, 16 Feb 2011 20:31:43 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1297917104;
	bh=/k4/66507HjDWBbZYPkXppodVvP4B17iDESvPImQEZM=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=fXl9qaFJmsi7cGAGwIJMVM4YgX0sryFvawkkXa0cRElnwJfguPM5kHgR+tzKnI4zH
	 SGOVfClypZWCkiZR4gbYqKxZmVodtC16zhza952w/ws+AizFUSSEvksrZl6le7hrds
	 WWrwNS9Hdza9tC7gmrgv9zneyTRK5GaD64g/y0jU=
Message-Id: <B61B4A67-CB97-4638-AEF8-C7FAE09001BC@yahoo-inc.com>
From: Sanjay Radia <sradia@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <0CBAFDBB-AC2A-495D-8936-4DF4D6731B69@Holsman.NET>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Abandon hdfsproxy HDFS contrib
Date: Thu, 17 Feb 2011 10:01:43 +0530
References: <D97CF74A-147E-4573-B188-2F9C48CFC482@mac.com> <AANLkTi=F2eBsACBfjYSVava+VwFcsRpqZdRRN+OHXNRJ@mail.gmail.com> <823262F0-EE37-4A0A-8F8C-D9422541D67B@mac.com> <0CBAFDBB-AC2A-495D-8936-4DF4D6731B69@Holsman.NET>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


On Feb 11, 2011, at 12:03 PM, Ian Holsman wrote:

>
> On Feb 11, 2011, at 5:11 PM, Nigel Daley wrote:
>
>>
>> On Feb 10, 2011, at 9:24 PM, Owen O'Malley wrote:
>>
>>> On Thu, Feb 10, 2011 at 7:40 PM, Nigel Daley <ndaley@mac.com> wrote:
>>>
>>>> I think the PMC should abandon the hdfsproxy HDFS contrib  
>>>> component.  It's
>>>> last meaningful contribution was August 2010:
>>>>
>>>
>>> -1 we still use and are maintaining this.
>>
>>
>> Who's the 'we'?  Looking at HDFS-1164 it looks like hdfsproxy was  
>> failing a unit test for 7 months.  This is exactly the reason we  
>> should thoughtfully consider whether it has a future within Hadoop.
>>
>> Perhaps Y! uses a different version of hdfsproxy?
>
> They probably have patched it, and mistakenly forgot to submit them
Yes, we have some updates that we haven;t got around to pushing around

> .. any chance of doing a diff on your version and submitting it?


I will check into this and get back.
We are also working to incorporate some of the features of the proxy  
into hdfs proper -- if that work completes and is accepted by the  
community then hdfs proxy may become
less useful.

Hence, for now,  my vote is -1 for removing hdfs proxy.

sanjay

sanjay
>
> Regards
> Ian
>
>>
>> Nige
>>
>


From general-return-3057-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 17 05:30:55 2011
Return-Path: <general-return-3057-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 41991 invoked from network); 17 Feb 2011 05:30:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Feb 2011 05:30:55 -0000
Received: (qmail 14691 invoked by uid 500); 17 Feb 2011 05:30:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14237 invoked by uid 500); 17 Feb 2011 05:30:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14227 invoked by uid 99); 17 Feb 2011 05:30:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 05:30:49 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 05:30:44 +0000
Received: by wwd20 with SMTP id 20so2135151wwd.29
        for <general@hadoop.apache.org>; Wed, 16 Feb 2011 21:30:23 -0800 (PST)
Received: by 10.216.79.80 with SMTP id h58mr1287129wee.111.1297920622420; Wed,
 16 Feb 2011 21:30:22 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.82.194 with HTTP; Wed, 16 Feb 2011 21:30:01 -0800 (PST)
In-Reply-To: <A9878555-698E-4CD9-A5CE-765D9C36520A@yahoo-inc.com>
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com> <FD5BC1D3-881A-4D45-BA2C-C6084C884418@apache.org>
 <AANLkTim_dNOdj=HUrsx7ZhvG3_QXWerB_EVaakMuT68w@mail.gmail.com> <A9878555-698E-4CD9-A5CE-765D9C36520A@yahoo-inc.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Wed, 16 Feb 2011 21:30:01 -0800
X-Google-Sender-Auth: xp-srOeTU5ZNvB4kFqzZfqvRMTA
Message-ID: <AANLkTik9a=WW=aeGeaLLTA1RgvVn7n+sJHxQ9y2A=5xx@mail.gmail.com>
Subject: Re: [DISCUSS] Move common, hdfs, mapreduce contrib components to
 apache-extras.org or elsewhere
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Yes, Sanjay. The reason is 'tight coupling'. In fact this was I who
was opposing it - not Nigel. I guess you misread the thread ;)

On Wed, Feb 16, 2011 at 20:25, Sanjay Radia <sradia@yahoo-inc.com> wrote:
>
> On Jan 31, 2011, at 10:19 PM, Konstantin Boudnik wrote:
>
>> On Sun, Jan 30, 2011 at 23:19, Owen O'Malley <omalley@apache.org> wrote:
>>>
>>> On Jan 30, 2011, at 7:42 PM, Nigel Daley wrote:
>>>
>>>> Now that http://apache-extras.org is launched
>>>>
>>>> (https://blogs.apache.org/foundation/entry/the_apache_software_foundation_launches)
>>>> I'd like to start a discussion on moving contrib components out of
>>>> common,
>>>> mapreduce, and hdfs.
>>>
>>> The PMC can't "move" code to Apache extras. It can only choose to abandon
>>> code that it doesn't want to support any longer. As a separate action
>>> some
>>> group of developers may create projects in Apache Extras based on the
>>> code
>>> from Hadoop.
>>>
>>> Therefore the question is really what if any code Hadoop wants to
>>> abandon.
>>> That is a good question and one that we should ask ourselves
>>> occasionally.
>>>
>>> After a quick consideration, my personal list would look like:
>>>
>>> failmon
>>> fault injection
>>
>> This is the best way to kill a project as tightly coupled with the
>> core code as fault injection.
>>
>> So, if you really want to kill it - then move it.
>
>
> Nigel/Owen did not say "kill it". Folks were simply listing potential
> projects to move out.
> If you feel that it should stay in then simply say so and give the reasons
> -- looks like your reason is "tight coupling".
>
>
> sanjay
>
>>
>>> fuse-dfs
>>> hod
>>> kfs
>>>
>>> Also note that pushing code out of Hadoop has a high cost. There are at
>>> least 3 forks of the hadoop-gpl-compression code. That creates a lot of
>>> confusion for the users. A lot of users never go to the work to figure
>>> out
>>> which fork and branch of hadoop-gpl-compression work with the version of
>>> Hadoop they installed.
>>>
>>> -- Owen
>>>
>>>
>
>

From general-return-3058-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 17 12:44:19 2011
Return-Path: <general-return-3058-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47889 invoked from network); 17 Feb 2011 12:44:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Feb 2011 12:44:19 -0000
Received: (qmail 29353 invoked by uid 500); 17 Feb 2011 12:44:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28878 invoked by uid 500); 17 Feb 2011 12:44:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28866 invoked by uid 99); 17 Feb 2011 12:44:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 12:44:13 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bernd.fondermann@googlemail.com designates 209.85.210.176 as permitted sender)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 12:44:08 +0000
Received: by iyb26 with SMTP id 26so2428773iyb.35
        for <general@hadoop.apache.org>; Thu, 17 Feb 2011 04:43:48 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type:content-transfer-encoding;
        bh=+LAqN2j3zeqH3S8YLhUA6B20ENw2g/1umDnkO8zRmsc=;
        b=PNwaL+ZqJiSWkWWKIGO4y6EeFBYvJpkoRdxrVj+1xNkBtZ2Ua/u4ydLU5WFphfQSYU
         iV2So4X7TcdTpS8fhWM7BZPoVdg5QBe/4DS9GuaMewRSaZMFzyZTl5vg4a9WE4BiKnFA
         +G4k/XYABGh2jZLUT9232ZVpJatQaE3AGEAXw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=iu6TBhMXoUAaMMAUU/durKUt+eHEcernHcxLbHWKUHw7XR903Xzp1NsrtVi9a9IASq
         JcE1mN90aQZ7KrHWl3qHrtJFUdyK6HhV1dLY3iuZiJh24o7+BXUNcGxmSXCFVqbeoePV
         LvzSZA1jTlJ24nu2PW+yP/C6Hdve9ZjbgPQDY=
MIME-Version: 1.0
Received: by 10.42.239.200 with SMTP id kx8mr2562758icb.416.1297946627951;
 Thu, 17 Feb 2011 04:43:47 -0800 (PST)
Received: by 10.42.8.149 with HTTP; Thu, 17 Feb 2011 04:43:47 -0800 (PST)
In-Reply-To: <F2EC1B1B-A2BB-4FC8-9842-3904A25338EE@gbiv.com>
References: <D97CF74A-147E-4573-B188-2F9C48CFC482@mac.com>
	<AANLkTi=F2eBsACBfjYSVava+VwFcsRpqZdRRN+OHXNRJ@mail.gmail.com>
	<823262F0-EE37-4A0A-8F8C-D9422541D67B@mac.com>
	<0CBAFDBB-AC2A-495D-8936-4DF4D6731B69@Holsman.NET>
	<AANLkTimUhDbTnwn6Hk2oYXFJMts+A=FnL-q-veXqX1m0@mail.gmail.com>
	<F2EC1B1B-A2BB-4FC8-9842-3904A25338EE@gbiv.com>
Date: Thu, 17 Feb 2011 13:43:47 +0100
Message-ID: <AANLkTikVcGcKHnh9EgUDQr2JGLPvkBvBRgn4my3uJfOc@mail.gmail.com>
Subject: Re: [VOTE] Abandon hdfsproxy HDFS contrib
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Sat, Feb 12, 2011 at 09:18, Roy T. Fielding <fielding@gbiv.com> wrote:
> On Feb 11, 2011, at 2:28 AM, Bernd Fondermann wrote:
>> On Fri, Feb 11, 2011 at 07:33, Ian Holsman <hadoop@holsman.net> wrote:
>>> They probably have patched it, and mistakenly forgot to submit them.. a=
ny chance of doing a diff on your version and submitting it?
>>
>> Please keep in mind: The original author(s) would need to submit it -
>> not a proxy.
>
> No, anyone with permission of the copyright owner can submit it
> if it is a separately copyrightable work. =A0If it is just a repair,
> then anyone can submit it, since repairs are not copyrightable.
> The original author should be noted in any credit given for the
> fix.

We have the very unfortunate situation here at Hadoop where Apache
Hadoop is not the primary and foremost place of Hadoop development.
Instead, code is developed internally at Yahoo and then contributed in
(smaller or larger) chunks to Hadoop.
This is open source development upside down.
It is not ok for people to diff ASF svn against their internal code
and provide the diff as a patch without reviewing IP first for every
line of code changed.
For larger chunks I'd suggest to even go via the Incubator IP clearance pro=
cess.
Only then will we force committers to primarily work here in the open
and return to what I'd consider a healthy project.

To be honest: Hadoop is in the process of falling apart.
Contrib Code gets moved out of Apache instead of being maintained here.
Discussions are seldom consense-driven.
Release branches stagnate.
Downstream projects like HBase don't get proper support.
Production setups are made from 3rd party distributions.
Development is not happening here, but elsewhere behind corporate doors.
Discussion about future developments are started on corporate blogs (
http://developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextgen/
) instead of on the proper mailing list.
Hurdles for committing are way too high.
On the bright side, new committers and PMC members are added, this is
an improvement.

I'd suggest to move away from relying on large code dumps from
corporations, and move back to the ASF-proven "individual committer
commits on trunk"-model where more committers can get involved.
If that means not to support high end cluster sizes for some months,
well, so be it. Average committers cannot run - e.g. test - on high
end cluster sizes. If that would mean they cannot participate, then
the open source project better concentrate on small and medium sized
cluster instead.

  Bernd

From general-return-3059-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 17 13:32:18 2011
Return-Path: <general-return-3059-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66692 invoked from network); 17 Feb 2011 13:32:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Feb 2011 13:32:18 -0000
Received: (qmail 89228 invoked by uid 500); 17 Feb 2011 13:32:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88783 invoked by uid 500); 17 Feb 2011 13:32:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88759 invoked by uid 99); 17 Feb 2011 13:32:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 13:32:12 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bernd.fondermann@googlemail.com designates 209.85.210.176 as permitted sender)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 13:32:04 +0000
Received: by iyb26 with SMTP id 26so2475526iyb.35
        for <general@hadoop.apache.org>; Thu, 17 Feb 2011 05:31:43 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=tPS4ivaPGMavR47NJPDtVjjfu2Tv1We1s0u24NSChgQ=;
        b=CQwYQJfGhr91QtjdGI3fh/oL3rKpioM57ozztye6/MgNVcqWxJzAFdfdK5rQO5k/ac
         MtXp1SwGQXQJrFZnnZ3Weapjf35NA/d+Y/2UaNCZKWs6VnmL/qCVDx+cnjxPZeBYH7Z9
         BabkazI5IjePDVeMJatvpNG319465O7quH30M=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=utgDPV1sbGDQP1r04y4+viflGPav/Di4mVAnslobkTfVZaraH2ZEUCssGAY9Lt/4l1
         ORb5nWhAPScud6ZolzYDrzXLblVWesQ5MQKT5Q1DBG403ck5e5cWQAdLMWpMozHP4NVX
         VdYhny7hvv+XB2O83/XX6vNHke9Atm97Qrymk=
MIME-Version: 1.0
Received: by 10.42.213.66 with SMTP id gv2mr2708658icb.81.1297949503671; Thu,
 17 Feb 2011 05:31:43 -0800 (PST)
Received: by 10.42.8.149 with HTTP; Thu, 17 Feb 2011 05:31:43 -0800 (PST)
In-Reply-To: <AANLkTin2JvOGdZT5g5tmTGeCpD3Qz2soZWcTQtHMsSza@mail.gmail.com>
References: <77405974-6771-4604-926B-976240743F3C@mac.com>
	<AANLkTim80KJ3f14iM6Ltzy7UG613myhN-AiwQ3pCmVdi@mail.gmail.com>
	<AANLkTiknkrWK=GzGMVj4_Z6p5Jn8TQu1P5TkaFSgORJk@mail.gmail.com>
	<AANLkTin0TguLYHi9X0uL3ZF7tCz5xTHBYa7056oKF2Ae@mail.gmail.com>
	<93644D3B-4830-4813-92BE-809738969CF9@apache.org>
	<AANLkTi=dbGSLhtFA+Rf9LgaX05v45HDLoQDE_2C5ULAf@mail.gmail.com>
	<65311075-A30E-4971-A2BA-C75409C6486E@jpl.nasa.gov>
	<AANLkTi=XbpVUmg_TpH7eqEn8w7Gcu9KiwtJmKMLa5w5n@mail.gmail.com>
	<FAE19BAA-47A7-4747-BDD8-5AA42C259EB6@jpl.nasa.gov>
	<AANLkTin2JvOGdZT5g5tmTGeCpD3Qz2soZWcTQtHMsSza@mail.gmail.com>
Date: Thu, 17 Feb 2011 14:31:43 +0100
Message-ID: <AANLkTi=iGCBrp6+dsK66-+fWF7UoLPSqMOhoNUnjbMHY@mail.gmail.com>
Subject: Re: [VOTE] Abandon mrunit MapReduce contrib
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Fri, Feb 11, 2011 at 23:10, Aaron Kimball <akimball83@gmail.com> wrote:
> The main reason I am interested in removing MRUnit from Hadoop is that I
> believe that MRUnit deserves its own release cycle. I think this is in the
> best interest of its users.

Not in mine, at least. (I'm writing MR unit tests.)
Many projects release more than one product. I'd rather get MRUnit
from the same source where I get my MR from.
Separate release cylcles would be ok for me, though.

> Perhaps more importantly, access to new features in MRUnit should not
> require upgrading one's entire Hadoop deployment; this is a client library
> that depends only on Hadoop's public APIs.

+1.

> My primary concern is to move MRUnit to a place where the community can
> derive the most benefit from it. The Apache Incubator could fulfill this
> role; given the presence of individuals willing to mentor this project, I
> believe this would be a successful way to release MRUnit more quickly and
> continue to work to grow the MRUnit community.

What are your expectations what MRUnit would become, software-wise?
Wouldn't the MRUnit community be largely the same as the Hadoop-MR
community?

  Bernd

From general-return-3060-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 17 13:46:33 2011
Return-Path: <general-return-3060-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70331 invoked from network); 17 Feb 2011 13:46:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Feb 2011 13:46:32 -0000
Received: (qmail 13410 invoked by uid 500); 17 Feb 2011 13:46:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12799 invoked by uid 500); 17 Feb 2011 13:46:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12774 invoked by uid 99); 17 Feb 2011 13:46:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 13:46:27 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 13:46:19 +0000
Received: by vws18 with SMTP id 18so1100896vws.35
        for <general@hadoop.apache.org>; Thu, 17 Feb 2011 05:45:58 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=uit4PO8G3r1v2l3+96EBICsGEPhTGx1EkBE13+g/rsM=;
        b=i++UGGyqG0IAv/S1t1qjjDFOfY1x/XKw9lXZjapaS4q/PDoVV28dTXTP78I+P4VrE+
         XURkPUUuTcCeIPfJUHdE/A83++bbjJeW9jslwZWWStYe18P+xA/R4M2r765C91jgSWim
         lJWGzlYxxSr0Vdr2eCauoZIDdhAScKsuWwgCE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=eIYnTd4WKeW6MfVpYAUKqTMvw8gSpHwMlCnE+dL88uMWz8yr06SzTCOOyzppfvOBdn
         rgMviAfRjC0KVDb3GQbaHpz9F69VRZBhvA4pgprOuKUOEi/IJuNZ5TKhAS5t/5gWFPCG
         eH2pIm9IcoXMgPGT7eU+rTNHWasBs9mLe0zcQ=
Received: by 10.52.159.161 with SMTP id xd1mr149321vdb.23.1297950357868;
        Thu, 17 Feb 2011 05:45:57 -0800 (PST)
Received: from [10.180.150.79] (h-64-236-128-43.nat.aol.com [64.236.128.43])
        by mx.google.com with ESMTPS id e10sm418868vch.43.2011.02.17.05.45.56
        (version=TLSv1/SSLv3 cipher=OTHER);
        Thu, 17 Feb 2011 05:45:56 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Hadoop testing project
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <20110216195012.GA29594@linspire.com>
Date: Thu, 17 Feb 2011 08:45:55 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <19DCD585-D5DC-43B9-8CA5-DAE6F949F5DC@Holsman.NET>
References: <AANLkTi=EvhSotauym5K080wy0+UORW+W2Nuh4by5JgdZ@mail.gmail.com> <4D5BB6E4.8040403@apache.org> <20110216195012.GA29594@linspire.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

I'm not sure it makes sense to  all the testing packages under a =
different umbrella that covers the code they test.
While there might be commonalities building a test harness, I would =
think that each testing tool would need to have deep knowledge of the =
tool's internals that it is testing. as such it would need someone with =
the experience to code it.

I don't see what advantage combining PigUnit & say 'MRUnit' would be for =
example.=20
--I
On Feb 16, 2011, at 2:50 PM, Konstantin Boudnik wrote:

> Steve.
>=20
> If the project under discussion will provide a common harness where =
such a test
> artifact (think of a Maven artifact for example) will click and will =
be
> executed automatically with all needed tools and dependencies resolved =
for you
> - would it be appealing for end-users' cause?
>=20
> As Joep said this "...will reduce the effort to take any (set of ) =
changes
> from development into production." Take it one step further: when your =
cluster
> is 'assembled' you need to validate it (on top of a concrete OS, =
etc.); is it
> desirable to follow N-steps process to bring about whatever testing =
work-load
> you need or you'd prefer to simply do something like:
>=20
>    wget =
http://workloads.internal.mydomain.com/stackValidations/v12.4.pom \
>        && mvn verify
>=20
> and check the results later on?
>=20
> These gonna be the same tools that dev. use for their tasks although =
worksets
> will be different. So what?
>=20
> Cos
>=20
> On Wed, Feb 16, 2011 at 11:37AM, Steve Loughran wrote:
>> On 15/02/11 21:58, Konstantin Boudnik wrote:
>>> While MrUnit discussion draws to its natural conclusion I would like
>>> to bring up another point which might be well aligned with that
>>> discussion. Patrick Hunt has brought up this idea earlier today and =
I
>>> believe it has to be elaborated further.
>>>=20
>>> A number of testing projects both for Hadoop and Hadoop-related
>>> component were brought to life over last year or two. Among those =
are
>>> MRUnit, PigUnit, YCSB, Herriot, and perhaps a few more. They all
>>> focusing on more or less the same problem e.g. validation of Hadoop =
or
>>> on-top-of-Hadoop components, or application level testing for =
Hadoop.
>>> However, the fact that they all are spread across a wide variety of
>>> projects seems to confuse/mislead Hadoop users.
>>>=20
>>> How about incubating a bigger Hadoop (Pig, Oozie, HBase) testing
>>> project which will take care about development and support of common
>>> (where's possible) tools, frameworks and the like? Please feel free =
to
>>> share your thoughts :)
>>> --
>>=20
>> I think it would be good though specific projects will need/have =
their =20
>> own testing needs -I'd expect more focus for testing redistributables =
to =20
>> be on helping Hadoop users test their stuff against subsets of data, =20=

>> rather than the hadoop-*-dev problem of "stressing the hadoop stack =
once =20
>> your latest patch is applied".
>>=20
>> That said, the whole problem of qualifying an OS, Java release and =20=

>> cluster is something we'd expect most end user teams to have to do =20=

>> -right now terasort is the main stress test.


From general-return-3061-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 17 13:59:31 2011
Return-Path: <general-return-3061-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73714 invoked from network); 17 Feb 2011 13:59:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Feb 2011 13:59:30 -0000
Received: (qmail 40212 invoked by uid 500); 17 Feb 2011 13:59:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39822 invoked by uid 500); 17 Feb 2011 13:59:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39813 invoked by uid 99); 17 Feb 2011 13:59:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 13:59:25 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 13:59:19 +0000
Received: by qwe4 with SMTP id 4so2507636qwe.35
        for <general@hadoop.apache.org>; Thu, 17 Feb 2011 05:58:58 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=wlbDA1hxSXRnC7F3hHEPYex7HQgcZ90D1B89toYy8Is=;
        b=XRkhk7FsyTfPRNXYQ0Si9PifqhTlcb9UvGuEoakqEXmqk3gUkKMU2CcvhqpBYLaNRN
         YP8T5v6aOOCWt8g+qaz+aFuRzCw0S/XNZseenu3SdDxYlJhybX21L0r0Bo4Dmq/5jJcF
         bLNewCvPf1vXAM6mp0TOt4hhS1oVDUDLW+kio=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=Vh1FHllcfMlCwBMQ4haL+7kD3CEfH1VZwk4JCymhbkgVpUut9oFe8CTxkjvLqpnRpb
         xxOEM+gn2IAD3S+Cd/C+V3eUe1zwDO8tdzrvJF1LNrG0O8C0vPqBTAvLCi+KHvlxEcKE
         Vrfix/GORk598uUXCanhqxI1aHvdEH6b3Qs8I=
Received: by 10.224.80.143 with SMTP id t15mr2599118qak.41.1297951137949;
        Thu, 17 Feb 2011 05:58:57 -0800 (PST)
Received: from [10.180.150.79] (h-64-236-128-43.nat.aol.com [64.236.128.43])
        by mx.google.com with ESMTPS id l12sm705378qcu.7.2011.02.17.05.58.56
        (version=TLSv1/SSLv3 cipher=OTHER);
        Thu, 17 Feb 2011 05:58:57 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [VOTE] Abandon hdfsproxy HDFS contrib
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <AANLkTikVcGcKHnh9EgUDQr2JGLPvkBvBRgn4my3uJfOc@mail.gmail.com>
Date: Thu, 17 Feb 2011 08:58:56 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <864B55E7-8A46-458F-8B16-8252A2011835@Holsman.NET>
References: <D97CF74A-147E-4573-B188-2F9C48CFC482@mac.com> <AANLkTi=F2eBsACBfjYSVava+VwFcsRpqZdRRN+OHXNRJ@mail.gmail.com> <823262F0-EE37-4A0A-8F8C-D9422541D67B@mac.com> <0CBAFDBB-AC2A-495D-8936-4DF4D6731B69@Holsman.NET> <AANLkTimUhDbTnwn6Hk2oYXFJMts+A=FnL-q-veXqX1m0@mail.gmail.com> <F2EC1B1B-A2BB-4FC8-9842-3904A25338EE@gbiv.com> <AANLkTikVcGcKHnh9EgUDQr2JGLPvkBvBRgn4my3uJfOc@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

Hi Bernd.

On Feb 17, 2011, at 7:43 AM, Bernd Fondermann wrote:
>=20
> We have the very unfortunate situation here at Hadoop where Apache
> Hadoop is not the primary and foremost place of Hadoop development.
> Instead, code is developed internally at Yahoo and then contributed in
> (smaller or larger) chunks to Hadoop.

This has been the situation in the past,
but as you can see in the last month, this has changed.

Yahoo! has publicly committed to move their development into the main =
code base, and you can see they have started doing this with the 20.100 =
branch,
and their recent commits to trunk.=20
Combine this with Nige taking on the 0.22 release branch, (and =
sheperding it into a stable release) and I think we have are addressing =
your concerns.

They have also started bringing the discussions back on the list, see =
the recent discussion about Jobtracker-nextgen Arun has re-started in =
MAPREDUCE-279.

I'm not saying it's perfect, but I think the major players understand =
there is an issue, and they are *ALL* moving in the right direction.



> This is open source development upside down.
> It is not ok for people to diff ASF svn against their internal code
> and provide the diff as a patch without reviewing IP first for every
> line of code changed.
> For larger chunks I'd suggest to even go via the Incubator IP =
clearance process.
> Only then will we force committers to primarily work here in the open
> and return to what I'd consider a healthy project.
>=20
> To be honest: Hadoop is in the process of falling apart.
> Contrib Code gets moved out of Apache instead of being maintained =
here.
> Discussions are seldom consense-driven.
> Release branches stagnate.

True. releases do take a long time. This is mainly due to it being =
extremely hard to test and verify that a release is stable.
It's not enough to just run the thing on 4 machines, you need at least =
50 to test some of the major problems. This requires some serious $ for =
someone to verify.

> Downstream projects like HBase don't get proper support.
> Production setups are made from 3rd party distributions.
> Development is not happening here, but elsewhere behind corporate =
doors.
> Discussion about future developments are started on corporate blogs (
> =
http://developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextgen/
> ) instead of on the proper mailing list.
> Hurdles for committing are way too high.
> On the bright side, new committers and PMC members are added, this is
> an improvement.
>=20
> I'd suggest to move away from relying on large code dumps from
> corporations, and move back to the ASF-proven "individual committer
> commits on trunk"-model where more committers can get involved.
> If that means not to support high end cluster sizes for some months,
> well, so be it.

> Average committers cannot run - e.g. test - on high
> end cluster sizes. If that would mean they cannot participate, then
> the open source project better concentrate on small and medium sized
> cluster instead.


Well.. that's one approach.. but there are several companies out there =
who rely on apache's hadoop to power their large clusters, so I'd hate =
to see hadoop become something that only runs well on 10-nodes.. as I =
don't think that will help anyone either.



>=20
>  Bernd


From general-return-3062-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 17 15:23:37 2011
Return-Path: <general-return-3062-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 3703 invoked from network); 17 Feb 2011 15:23:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Feb 2011 15:23:36 -0000
Received: (qmail 94200 invoked by uid 500); 17 Feb 2011 15:23:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93780 invoked by uid 500); 17 Feb 2011 15:23:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93766 invoked by uid 99); 17 Feb 2011 15:23:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 15:23:30 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of bernd.fondermann@googlemail.com designates 209.85.161.176 as permitted sender)
Received: from [209.85.161.176] (HELO mail-gx0-f176.google.com) (209.85.161.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 15:23:24 +0000
Received: by gxk4 with SMTP id 4so1143289gxk.35
        for <general@hadoop.apache.org>; Thu, 17 Feb 2011 07:23:03 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=gHqP0CGrzxWnbxeXBNyrGBh0i2FORQ7MnNIOXl2hh5g=;
        b=ThGRDonz1xvrFpdxXp0dNxhI19CUv5H5u2XK7GlwVH/uSwnY9q7Em+pvSf2osn2+vb
         TIo8fsTt72yrlIS9ZsW/7dlsXMu6iSqz8FgpW9mE/fME9GlSo/gmBTr9mUcJxvu87p/N
         4nIBHSgkYUqiKVKl7Kuku5LvS/Jyt2sOnbMqw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=QGM3H3YuMzZI9L3nSm9+U+MvSdvHa/kUMxQlTTqactPJ7xsxxsHKJCbx1cKnDf+JJU
         GXPqZqBObXmvidDBheHTWb/pzsGqArv5RRQExvtyaPm3uSFeRM8VQM6WyWUdoAPvH5iN
         mA89Yb4xpRkNmmK0YMnCe8RIIC6PT7AJwgysQ=
MIME-Version: 1.0
Received: by 10.42.223.65 with SMTP id ij1mr2938715icb.349.1297956182976; Thu,
 17 Feb 2011 07:23:02 -0800 (PST)
Received: by 10.42.8.149 with HTTP; Thu, 17 Feb 2011 07:23:02 -0800 (PST)
In-Reply-To: <864B55E7-8A46-458F-8B16-8252A2011835@Holsman.NET>
References: <D97CF74A-147E-4573-B188-2F9C48CFC482@mac.com>
	<AANLkTi=F2eBsACBfjYSVava+VwFcsRpqZdRRN+OHXNRJ@mail.gmail.com>
	<823262F0-EE37-4A0A-8F8C-D9422541D67B@mac.com>
	<0CBAFDBB-AC2A-495D-8936-4DF4D6731B69@Holsman.NET>
	<AANLkTimUhDbTnwn6Hk2oYXFJMts+A=FnL-q-veXqX1m0@mail.gmail.com>
	<F2EC1B1B-A2BB-4FC8-9842-3904A25338EE@gbiv.com>
	<AANLkTikVcGcKHnh9EgUDQr2JGLPvkBvBRgn4my3uJfOc@mail.gmail.com>
	<864B55E7-8A46-458F-8B16-8252A2011835@Holsman.NET>
Date: Thu, 17 Feb 2011 16:23:02 +0100
Message-ID: <AANLkTikhe6YjUBGY8LMc3OG=b7VXWz7WhyNKGkJRu_7K@mail.gmail.com>
Subject: Re: [VOTE] Abandon hdfsproxy HDFS contrib
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Thu, Feb 17, 2011 at 14:58, Ian Holsman <hadoop@holsman.net> wrote:
> Hi Bernd.
>
> On Feb 17, 2011, at 7:43 AM, Bernd Fondermann wrote:
>>
>> We have the very unfortunate situation here at Hadoop where Apache
>> Hadoop is not the primary and foremost place of Hadoop development.
>> Instead, code is developed internally at Yahoo and then contributed in
>> (smaller or larger) chunks to Hadoop.
>
> This has been the situation in the past,
> but as you can see in the last month, this has changed.
>
> Yahoo! has publicly committed to move their development into the main code base, and you can see they have started doing this with the 20.100 branch,
> and their recent commits to trunk.
> Combine this with Nige taking on the 0.22 release branch, (and sheperding it into a stable release) and I think we have are addressing your concerns.
>
> They have also started bringing the discussions back on the list, see the recent discussion about Jobtracker-nextgen Arun has re-started in MAPREDUCE-279.
>
> I'm not saying it's perfect, but I think the major players understand there is an issue, and they are *ALL* moving in the right direction.

I enthusiastically would like to see your optimism be verified.
Maybe I'm misreading the statements issued publicly, but I don't think
that this is fully understood. I agree though that it's a move into
the right direction.

>> This is open source development upside down.
>> It is not ok for people to diff ASF svn against their internal code
>> and provide the diff as a patch without reviewing IP first for every
>> line of code changed.
>> For larger chunks I'd suggest to even go via the Incubator IP clearance process.
>> Only then will we force committers to primarily work here in the open
>> and return to what I'd consider a healthy project.
>>
>> To be honest: Hadoop is in the process of falling apart.
>> Contrib Code gets moved out of Apache instead of being maintained here.
>> Discussions are seldom consense-driven.
>> Release branches stagnate.
>
> True. releases do take a long time. This is mainly due to it being extremely hard to test and verify that a release is stable.
> It's not enough to just run the thing on 4 machines, you need at least 50 to test some of the major problems. This requires some serious $ for someone to verify.

It has been proposed on the list before, IIRC. Don't know how to get
there, but the project seriously needs access to a cluster of this
size.

>> Downstream projects like HBase don't get proper support.
>> Production setups are made from 3rd party distributions.
>> Development is not happening here, but elsewhere behind corporate doors.
>> Discussion about future developments are started on corporate blogs (
>> http://developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextgen/
>> ) instead of on the proper mailing list.
>> Hurdles for committing are way too high.
>> On the bright side, new committers and PMC members are added, this is
>> an improvement.
>>
>> I'd suggest to move away from relying on large code dumps from
>> corporations, and move back to the ASF-proven "individual committer
>> commits on trunk"-model where more committers can get involved.
>> If that means not to support high end cluster sizes for some months,
>> well, so be it.
>
>> Average committers cannot run - e.g. test - on high
>> end cluster sizes. If that would mean they cannot participate, then
>> the open source project better concentrate on small and medium sized
>> cluster instead.
>
>
> Well.. that's one approach.. but there are several companies out there who rely on apache's hadoop to power their large clusters, so I'd hate to see hadoop become something that only runs well on
> 10-nodes.. as I don't think that will help anyone either.

But only looking at high-end scale doesn't help either.

Lets face the fact that Hadoop is now moving from early adaptors phase
into a much broader market. I predict that small to medium sized
clusters will be the majority of Hadoop deployments in a few month
time. 4000, or even 500 machines is the high-end range. If the open
source project Hadoop cannot support those users adequately (without
becoming defunct), the committership might be better off to focus on
the low-end and medium sized users.

I'm not suggesting to turn away from the handfull (?) of high-end
users. They certainly have most valuable input. But also, *they*
obviously have the resources in terms of larger clusters and
developers to deal with their specific setups. Obviously, they don't
need to rely on the open source project to make releases. In fact,
they *do* work on their own Hadoop derivatives.
All the other users, the hundreds of boring small cluster users, don't
have that choice. They *depend* on the open source releases.

Hadoop is an Apache project, to provide HDFS and MR free of charge to
the general public. Not only to me - nor to only one or two big
companies either.
Focus on all the users.

  Bernd

From general-return-3063-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 17 17:14:29 2011
Return-Path: <general-return-3063-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77765 invoked from network); 17 Feb 2011 17:14:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Feb 2011 17:14:29 -0000
Received: (qmail 51983 invoked by uid 500); 17 Feb 2011 17:14:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51476 invoked by uid 500); 17 Feb 2011 17:14:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51465 invoked by uid 99); 17 Feb 2011 17:14:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 17:14:22 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 17:14:12 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=hrz+57osGwpaedvHsYt0BsRShds8hyRR5nOMyT4ye17QrnvXZW6p5Jjp
   TwlUR83PC79QDSp3pzxUGGzquKtwsg79puR/8mduf2xpa2gLfmubw1LM5
   11LUCrSE9LLgrFd;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1297962847; x=1329498847;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20Hadoop=20testing=20project|Date:=20Thu,
   =2017=20Feb=202011=2017:15:23=20+0000|Message-ID:=20<4224
   A6E6-D5A1-46FD-890D-4AAF4C14AEF7@linkedin.com>|To:=20"<ge
   neral@hadoop.apache.org>"=20<general@hadoop.apache.org>
   |MIME-Version:=201.0|Content-Transfer-Encoding:=20quoted-
   printable|Content-ID:=20<9460C078A739844A9E6F8BA3DA5A1FFB
   @linkedin.com>|In-Reply-To:=20<20110216195012.GA29594@lin
   spire.com>|References:=20<AANLkTi=3DEvhSotauym5K080wy0+UO
   RW+W2Nuh4by5JgdZ@mail.gmail.com>=0D=0A=20<4D5BB6E4.804040
   3@apache.org>=20<20110216195012.GA29594@linspire.com>;
  bh=Cgm1jGrsgyAgS7bvo0ogCRtL2dK000ErnCMacC9bN/0=;
  b=HpWdDdegNwYttfEUM75S+Cwk5MIWJPwq3Mc6K5reHYC4NArvJbg1n7nN
   goBGdKvkT6SSK+l1t3XJSxr9ze0ZmQMlSfV9G+6ipgHGKJllvyp9txG5S
   ojXOPEIPiSAsDAZ;
X-IronPort-AV: E=Sophos;i="4.60,487,1291622400"; 
   d="scan'208";a="20833958"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi id 14.01.0251.000; Thu,
 17 Feb 2011 09:15:24 -0800
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: Hadoop testing project
Thread-Topic: Hadoop testing project
Thread-Index: AQHLzhMOHwhWDZrkTkqV8PVHPGb3EJQGdgwA
Date: Thu, 17 Feb 2011 17:15:23 +0000
Message-ID: <4224A6E6-D5A1-46FD-890D-4AAF4C14AEF7@linkedin.com>
References: <AANLkTi=EvhSotauym5K080wy0+UORW+W2Nuh4by5JgdZ@mail.gmail.com>
 <4D5BB6E4.8040403@apache.org> <20110216195012.GA29594@linspire.com>
In-Reply-To: <20110216195012.GA29594@linspire.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <9460C078A739844A9E6F8BA3DA5A1FFB@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


On Feb 16, 2011, at 11:50 AM, Konstantin Boudnik wrote:
> As Joep said this "...will reduce the effort to take any (set of ) change=
s
> from development into production." Take it one step further: when your cl=
uster
> is 'assembled' you need to validate it (on top of a concrete OS, etc.); i=
s it
> desirable to follow N-steps process to bring about whatever testing work-=
load
> you need or you'd prefer to simply do something like:
>=20
>    wget http://workloads.internal.mydomain.com/stackValidations/v12.4.pom=
 \
>        && mvn verify
>=20
> and check the results later on?

	We need a definition of what 'validate' means.  Is it "does stuff run"?  I=
n practice, that's useless.  Even horribly misconfigured grids can usually =
run something at least once.=20


From general-return-3064-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 17 17:21:50 2011
Return-Path: <general-return-3064-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 80896 invoked from network); 17 Feb 2011 17:21:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Feb 2011 17:21:50 -0000
Received: (qmail 67738 invoked by uid 500); 17 Feb 2011 17:21:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67277 invoked by uid 500); 17 Feb 2011 17:21:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67269 invoked by uid 99); 17 Feb 2011 17:21:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 17:21:44 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 17:21:37 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=FPb9wrxgCk/rCEKpwRWWb/rnd9c2P0spwDY532qrTASwMZychhdHGaS9
   SfFsdiOnWL7WyjhfZYCaUroUQ39L9MidO+yB2dkl10QC58iAto3jjWAdr
   IauVGsjHKVL/S1b;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1297963292; x=1329499292;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20[VOTE]=20Abandon=20hdfsproxy=20HDFS=20c
   ontrib|Date:=20Thu,=2017=20Feb=202011=2017:22:49=20+0000
   |Message-ID:=20<AF7F40DC-BF8D-43DF-A5E7-4A83A9DABC44@link
   edin.com>|To:=20"<general@hadoop.apache.org>"=20<general@
   hadoop.apache.org>|MIME-Version:=201.0
   |Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<B87DEC09A213B74CA65655DC8DB033B4@linkedin
   .com>|In-Reply-To:=20<AANLkTikVcGcKHnh9EgUDQr2JGLPvkBvBRg
   n4my3uJfOc@mail.gmail.com>|References:=20<D97CF74A-147E-4
   573-B188-2F9C48CFC482@mac.com>=0D=0A=20<AANLkTi=3DF2eBsAC
   BfjYSVava+VwFcsRpqZdRRN+OHXNRJ@mail.gmail.com>=0D=0A=20<8
   23262F0-EE37-4A0A-8F8C-D9422541D67B@mac.com>=0D=0A=20<0CB
   AFDBB-AC2A-495D-8936-4DF4D6731B69@Holsman.NET>=0D=0A=20<A
   ANLkTimUhDbTnwn6Hk2oYXFJMts+A=3DFnL-q-veXqX1m0@mail.gmail
   .com>=0D=0A=20<F2EC1B1B-A2BB-4FC8-9842-3904A25338EE@gbiv.
   com>=0D=0A=20<AANLkTikVcGcKHnh9EgUDQr2JGLPvkBvBRgn4my3uJf
   Oc@mail.gmail.com>;
  bh=ZdESL3j2EcvViUolETjzzhSEae640PYtwKFu5+FUIP0=;
  b=HCo3oaflskkkLY30LFbKOxiho/tqt7MLoJwT5B9iRGS/NDWgVFWk9nb3
   qCdzyix07uvrmvsI/PIl9D2f4XpJBQnEFavhMSMKTPQz0LP/YyJt0daLF
   31QZOyxZKOcvheM;
X-IronPort-AV: E=Sophos;i="4.60,487,1291622400"; 
   d="scan'208";a="20834215"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi id 14.01.0251.000; Thu,
 17 Feb 2011 09:22:49 -0800
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: [VOTE] Abandon hdfsproxy HDFS contrib
Thread-Topic: [VOTE] Abandon hdfsproxy HDFS contrib
Thread-Index: AQHLyZ27ICuajETxE02yFg9O1wZLWpP8Sq4AgAANWQCAAAYpgIAAQawAgAFtyICACCXggIAATYcA
Date: Thu, 17 Feb 2011 17:22:49 +0000
Message-ID: <AF7F40DC-BF8D-43DF-A5E7-4A83A9DABC44@linkedin.com>
References: <D97CF74A-147E-4573-B188-2F9C48CFC482@mac.com>
 <AANLkTi=F2eBsACBfjYSVava+VwFcsRpqZdRRN+OHXNRJ@mail.gmail.com>
 <823262F0-EE37-4A0A-8F8C-D9422541D67B@mac.com>
 <0CBAFDBB-AC2A-495D-8936-4DF4D6731B69@Holsman.NET>
 <AANLkTimUhDbTnwn6Hk2oYXFJMts+A=FnL-q-veXqX1m0@mail.gmail.com>
 <F2EC1B1B-A2BB-4FC8-9842-3904A25338EE@gbiv.com>
 <AANLkTikVcGcKHnh9EgUDQr2JGLPvkBvBRgn4my3uJfOc@mail.gmail.com>
In-Reply-To: <AANLkTikVcGcKHnh9EgUDQr2JGLPvkBvBRgn4my3uJfOc@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <B87DEC09A213B74CA65655DC8DB033B4@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


On Feb 17, 2011, at 4:43 AM, Bernd Fondermann wrote:
> To be honest: Hadoop is in the process of falling apart.

	We can thank the Apache Board for helping there as well.  Their high hande=
d interference basically set the project back 6 mos to a year; we're still =
recovering from the general mistrust that their actions have instilled in t=
he community.  (If we recover.  I'm not sure we will.)=

From general-return-3065-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 17 17:34:51 2011
Return-Path: <general-return-3065-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84599 invoked from network); 17 Feb 2011 17:34:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Feb 2011 17:34:50 -0000
Received: (qmail 1181 invoked by uid 500); 17 Feb 2011 17:34:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 810 invoked by uid 500); 17 Feb 2011 17:34:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 802 invoked by uid 99); 17 Feb 2011 17:34:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 17:34:45 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 17:34:39 +0000
Received: by ewy20 with SMTP id 20so1086324ewy.35
        for <general@hadoop.apache.org>; Thu, 17 Feb 2011 09:34:18 -0800 (PST)
Received: by 10.216.170.149 with SMTP id p21mr2019913wel.81.1297964058721;
 Thu, 17 Feb 2011 09:34:18 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.82.194 with HTTP; Thu, 17 Feb 2011 09:33:57 -0800 (PST)
In-Reply-To: <19DCD585-D5DC-43B9-8CA5-DAE6F949F5DC@Holsman.NET>
References: <AANLkTi=EvhSotauym5K080wy0+UORW+W2Nuh4by5JgdZ@mail.gmail.com>
 <4D5BB6E4.8040403@apache.org> <20110216195012.GA29594@linspire.com> <19DCD585-D5DC-43B9-8CA5-DAE6F949F5DC@Holsman.NET>
From: Konstantin Boudnik <cos@apache.org>
Date: Thu, 17 Feb 2011 09:33:57 -0800
X-Google-Sender-Auth: Oh1UlMGCYe1XUhAoeSJ1hOweu20
Message-ID: <AANLkTin7YOraMb5UXktcYJWLueLwe3o+_bEcQF0r+fse@mail.gmail.com>
Subject: Re: Hadoop testing project
To: general@hadoop.apache.org
Cc: Ian Holsman <hadoop@holsman.net>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Thu, Feb 17, 2011 at 05:45, Ian Holsman <hadoop@holsman.net> wrote:
> I'm not sure it makes sense to =A0all the testing packages under a differ=
ent umbrella that covers the code they test.
> While there might be commonalities building a test harness, I would think=
 that each testing tool would need to have deep knowledge of the tool's int=
ernals that it is testing. as such it would need someone with the experienc=
e to code it.

That's pretty much true indeed if you are talking about tests for a
project or closely tightened projects such as Herriot in Hadoop.
Speaking of tools there are some benefits though. Say, PigUnit and
MRUnit are both xUnit frameworks. The former allows you to run Pig
jobs in local and cluster mode. The latter is to validate MB jobs
without a need to fire up a cluster.

> I don't see what advantage combining PigUnit & say 'MRUnit' would be for =
example.

Don't you think Pig user would benefit if Pig scripts can be tested
against MRUnit which gives you a flavor of cluster environment without
one? Now, do you think it is likely that someone will go great lengths
to make such an effort and build such a bridge right now?

Cos

> --I
> On Feb 16, 2011, at 2:50 PM, Konstantin Boudnik wrote:
>
>> Steve.
>>
>> If the project under discussion will provide a common harness where such=
 a test
>> artifact (think of a Maven artifact for example) will click and will be
>> executed automatically with all needed tools and dependencies resolved f=
or you
>> - would it be appealing for end-users' cause?
>>
>> As Joep said this "...will reduce the effort to take any (set of ) chang=
es
>> from development into production." Take it one step further: when your c=
luster
>> is 'assembled' you need to validate it (on top of a concrete OS, etc.); =
is it
>> desirable to follow N-steps process to bring about whatever testing work=
-load
>> you need or you'd prefer to simply do something like:
>>
>> =A0 =A0wget http://workloads.internal.mydomain.com/stackValidations/v12.=
4.pom \
>> =A0 =A0 =A0 =A0&& mvn verify
>>
>> and check the results later on?
>>
>> These gonna be the same tools that dev. use for their tasks although wor=
ksets
>> will be different. So what?
>>
>> Cos
>>
>> On Wed, Feb 16, 2011 at 11:37AM, Steve Loughran wrote:
>>> On 15/02/11 21:58, Konstantin Boudnik wrote:
>>>> While MrUnit discussion draws to its natural conclusion I would like
>>>> to bring up another point which might be well aligned with that
>>>> discussion. Patrick Hunt has brought up this idea earlier today and I
>>>> believe it has to be elaborated further.
>>>>
>>>> A number of testing projects both for Hadoop and Hadoop-related
>>>> component were brought to life over last year or two. Among those are
>>>> MRUnit, PigUnit, YCSB, Herriot, and perhaps a few more. They all
>>>> focusing on more or less the same problem e.g. validation of Hadoop or
>>>> on-top-of-Hadoop components, or application level testing for Hadoop.
>>>> However, the fact that they all are spread across a wide variety of
>>>> projects seems to confuse/mislead Hadoop users.
>>>>
>>>> How about incubating a bigger Hadoop (Pig, Oozie, HBase) testing
>>>> project which will take care about development and support of common
>>>> (where's possible) tools, frameworks and the like? Please feel free to
>>>> share your thoughts :)
>>>> --
>>>
>>> I think it would be good though specific projects will need/have their
>>> own testing needs -I'd expect more focus for testing redistributables t=
o
>>> be on helping Hadoop users test their stuff against subsets of data,
>>> rather than the hadoop-*-dev problem of "stressing the hadoop stack onc=
e
>>> your latest patch is applied".
>>>
>>> That said, the whole problem of qualifying an OS, Java release and
>>> cluster is something we'd expect most end user teams to have to do
>>> -right now terasort is the main stress test.
>
>

From general-return-3066-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 17 18:28:10 2011
Return-Path: <general-return-3066-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 18175 invoked from network); 17 Feb 2011 18:28:10 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Feb 2011 18:28:10 -0000
Received: (qmail 98655 invoked by uid 500); 17 Feb 2011 18:28:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98282 invoked by uid 500); 17 Feb 2011 18:28:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98273 invoked by uid 99); 17 Feb 2011 18:28:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 18:28:05 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 18:27:58 +0000
Received: by qyk10 with SMTP id 10so2760990qyk.14
        for <general@hadoop.apache.org>; Thu, 17 Feb 2011 10:27:38 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=16fkgIJOhfB/umMin0p6eapsA2WF9CbSOU/DllMaXZE=;
        b=ui1Es1vIEkr891ARBLgFk9d//5L1ST194MizncL0xvOoMVYFxu4jZK67kEO16f162l
         LRPMp/YENzcD0evtgJHT6R7DnjG+yLzPxQqWkwF957v3gHIn0vJ+y2Sc5AwK82USkeyO
         2YlfIq+gSpnz2ZcItPMuGDe18BljOVFJmQzh8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=HYrYmhdtNE9GxqSUiW2X47CFEXieKWiyvrXF3ZuzT+xpo/I2GNq36APjXMRqFQTpHe
         8zImHWloBfGM0u4+FDbGae6/Whc9/xHJRCCSiTuTcjmCfpfPD9vGe1uQLfUAwn5bxXiX
         NN6jmnQw5fCkCXG9JiJ4e33jLjvuKSYfHfKRE=
Received: by 10.229.79.135 with SMTP id p7mr2618629qck.154.1297967257540;
        Thu, 17 Feb 2011 10:27:37 -0800 (PST)
Received: from [10.180.150.79] (h-64-236-128-43.nat.aol.com [64.236.128.43])
        by mx.google.com with ESMTPS id t7sm886763qcs.4.2011.02.17.10.27.36
        (version=TLSv1/SSLv3 cipher=OTHER);
        Thu, 17 Feb 2011 10:27:36 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [VOTE] Abandon hdfsproxy HDFS contrib
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <AANLkTikhe6YjUBGY8LMc3OG=b7VXWz7WhyNKGkJRu_7K@mail.gmail.com>
Date: Thu, 17 Feb 2011 13:27:35 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <0BEDB5ED-D1A2-4651-8BFB-7EE0911B8509@Holsman.NET>
References: <D97CF74A-147E-4573-B188-2F9C48CFC482@mac.com> <AANLkTi=F2eBsACBfjYSVava+VwFcsRpqZdRRN+OHXNRJ@mail.gmail.com> <823262F0-EE37-4A0A-8F8C-D9422541D67B@mac.com> <0CBAFDBB-AC2A-495D-8936-4DF4D6731B69@Holsman.NET> <AANLkTimUhDbTnwn6Hk2oYXFJMts+A=FnL-q-veXqX1m0@mail.gmail.com> <F2EC1B1B-A2BB-4FC8-9842-3904A25338EE@gbiv.com> <AANLkTikVcGcKHnh9EgUDQr2JGLPvkBvBRgn4my3uJfOc@mail.gmail.com> <864B55E7-8A46-458F-8B16-8252A2011835@Holsman.NET> <AANLkTikhe6YjUBGY8LMc3OG=b7VXWz7WhyNKGkJRu_7K@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)


On Feb 17, 2011, at 10:23 AM, Bernd Fondermann wrote:

> On Thu, Feb 17, 2011 at 14:58, Ian Holsman <hadoop@holsman.net> wrote:
>> Hi Bernd.
>>=20
>> On Feb 17, 2011, at 7:43 AM, Bernd Fondermann wrote:
>>>=20
>>> We have the very unfortunate situation here at Hadoop where Apache
>>> Hadoop is not the primary and foremost place of Hadoop development.
>>> Instead, code is developed internally at Yahoo and then contributed =
in
>>> (smaller or larger) chunks to Hadoop.
>>=20
>> This has been the situation in the past,
>> but as you can see in the last month, this has changed.
>>=20
>> Yahoo! has publicly committed to move their development into the main =
code base, and you can see they have started doing this with the 20.100 =
branch,
>> and their recent commits to trunk.
>> Combine this with Nige taking on the 0.22 release branch, (and =
sheperding it into a stable release) and I think we have are addressing =
your concerns.
>>=20
>> They have also started bringing the discussions back on the list, see =
the recent discussion about Jobtracker-nextgen Arun has re-started in =
MAPREDUCE-279.
>>=20
>> I'm not saying it's perfect, but I think the major players understand =
there is an issue, and they are *ALL* moving in the right direction.
>=20
> I enthusiastically would like to see your optimism be verified.
> Maybe I'm misreading the statements issued publicly, but I don't think
> that this is fully understood. I agree though that it's a move into
> the right direction.

I also hope to see more as well... and hopefully we will start seeing =
more and more of this from everyone.


>=20
>  Bernd


From general-return-3067-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 17 18:37:33 2011
Return-Path: <general-return-3067-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21950 invoked from network); 17 Feb 2011 18:37:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Feb 2011 18:37:33 -0000
Received: (qmail 16417 invoked by uid 500); 17 Feb 2011 18:37:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16101 invoked by uid 500); 17 Feb 2011 18:37:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16093 invoked by uid 99); 17 Feb 2011 18:37:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 18:37:29 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 18:37:21 +0000
Received: from SP2-EX07CAS05.ds.corp.yahoo.com (sp2-ex07cas05.corp.sp2.yahoo.com [98.137.59.39])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p1HIaatw046252;
	Thu, 17 Feb 2011 10:36:37 -0800 (PST)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by
 SP2-EX07CAS05.ds.corp.yahoo.com ([98.137.59.39]) with mapi; Thu, 17 Feb 2011
 10:36:36 -0800
From: Eric Yang <eyang@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: Ian Holsman <hadoop@holsman.net>
Date: Thu, 17 Feb 2011 10:36:35 -0800
Subject: Re: Hadoop testing project
Thread-Topic: Hadoop testing project
Thread-Index: AcvOySafP89cVP/JQsyA1di6+2nFcQACHQgN
Message-ID: <C982AAB3.FACF%eyang@yahoo-inc.com>
In-Reply-To: <AANLkTin7YOraMb5UXktcYJWLueLwe3o+_bEcQF0r+fse@mail.gmail.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

The biggest hurtle in hadoop adoption is that there is no easy way to setup
a pseudo cluster on developer's machine.  People are steering off course to
build additional simulation tools and validation tools.  In practice, those
tools don't provide nearly enough insight in things that could go wrong in =
a
real cluster.  For example, if a pig job uses HBaseStorage for accessing
data.  There is not a single hint that, the hbase-site.xml needs to be in a
jar file in the pig class path for pig job to distribute the hbase
environment to the MR cluster for the job to work.  Regardless of how good
the simulation tools, they are limited to the silo environment.  What we ca=
n
do to improve the integration is to have a set of installable packages that
can integrate well across hadoop ecosystems on the developer's machine.

This is similar situation that mass majority of developer on LAMP stack,
people don't start with compiling their own apache server and mysql server
to start development and testing.  They start by installing binary packages
and getting their work tested on real software.

Hence, hadoop developers bearing the responsibility of testing release
packages, while end user developer should be responsible for certifying the
integrated system on his/her own cluster.  There are already a list of tool=
s
to validate a cluster, like Terasort or GridMix 1,2,3.

I think the bigger concern is that Hadoop ecosystem does not have a standar=
d
method in linking dependencies.  Hbase depends on Zookeeper, and Pig depend=
s
on Hadoop and Hbase.  Then pig decided to put hadoop-core jar in it's own
jar file.  Chukwa depends on pig + hbase + hadoop and zookeeper.  The
version incompatibility is probably what driving people nuts.  Hence, there
is a new proposal on how to integrate among hadoop ecosystem.  I urge
project owners to review the proposal and provide feedbacks.

The proposal is located at:

https://issues.apache.org/jira/secure/attachment/12470823/deployment.pdf

The related jiras are:

https://issues.apache.org/jira/browse/HADOOP-6255
https://issues.apache.org/jira/browse/PIG-1857

There are plans to file more jiras for related projects.  The integration
would also be a lot easier if all related projects are using maven for
dependency management.

Regards,
Eric

On 2/17/11 9:33 AM, "Konstantin Boudnik" <cos@apache.org> wrote:

> On Thu, Feb 17, 2011 at 05:45, Ian Holsman <hadoop@holsman.net> wrote:
>> I'm not sure it makes sense to =A0all the testing packages under a diffe=
rent
>> umbrella that covers the code they test.
>> While there might be commonalities building a test harness, I would thin=
k
>> that each testing tool would need to have deep knowledge of the tool's
>> internals that it is testing. as such it would need someone with the
>> experience to code it.
>=20
> That's pretty much true indeed if you are talking about tests for a
> project or closely tightened projects such as Herriot in Hadoop.
> Speaking of tools there are some benefits though. Say, PigUnit and
> MRUnit are both xUnit frameworks. The former allows you to run Pig
> jobs in local and cluster mode. The latter is to validate MB jobs
> without a need to fire up a cluster.
>=20
>> I don't see what advantage combining PigUnit & say 'MRUnit' would be for
>> example.
>=20
> Don't you think Pig user would benefit if Pig scripts can be tested
> against MRUnit which gives you a flavor of cluster environment without
> one? Now, do you think it is likely that someone will go great lengths
> to make such an effort and build such a bridge right now?
>=20
> Cos
>=20
>> --I
>> On Feb 16, 2011, at 2:50 PM, Konstantin Boudnik wrote:
>>=20
>>> Steve.
>>>=20
>>> If the project under discussion will provide a common harness where suc=
h a
>>> test
>>> artifact (think of a Maven artifact for example) will click and will be
>>> executed automatically with all needed tools and dependencies resolved =
for
>>> you
>>> - would it be appealing for end-users' cause?
>>>=20
>>> As Joep said this "...will reduce the effort to take any (set of ) chan=
ges
>>> from development into production." Take it one step further: when your
>>> cluster
>>> is 'assembled' you need to validate it (on top of a concrete OS, etc.);=
 is
>>> it
>>> desirable to follow N-steps process to bring about whatever testing
>>> work-load
>>> you need or you'd prefer to simply do something like:
>>>=20
>>> =A0 =A0wget http://workloads.internal.mydomain.com/stackValidations/v12=
.4.pom \
>>> =A0 =A0 =A0 =A0&& mvn verify
>>>=20
>>> and check the results later on?
>>>=20
>>> These gonna be the same tools that dev. use for their tasks although
>>> worksets
>>> will be different. So what?
>>>=20
>>> Cos
>>>=20
>>> On Wed, Feb 16, 2011 at 11:37AM, Steve Loughran wrote:
>>>> On 15/02/11 21:58, Konstantin Boudnik wrote:
>>>>> While MrUnit discussion draws to its natural conclusion I would like
>>>>> to bring up another point which might be well aligned with that
>>>>> discussion. Patrick Hunt has brought up this idea earlier today and I
>>>>> believe it has to be elaborated further.
>>>>>=20
>>>>> A number of testing projects both for Hadoop and Hadoop-related
>>>>> component were brought to life over last year or two. Among those are
>>>>> MRUnit, PigUnit, YCSB, Herriot, and perhaps a few more. They all
>>>>> focusing on more or less the same problem e.g. validation of Hadoop o=
r
>>>>> on-top-of-Hadoop components, or application level testing for Hadoop.
>>>>> However, the fact that they all are spread across a wide variety of
>>>>> projects seems to confuse/mislead Hadoop users.
>>>>>=20
>>>>> How about incubating a bigger Hadoop (Pig, Oozie, HBase) testing
>>>>> project which will take care about development and support of common
>>>>> (where's possible) tools, frameworks and the like? Please feel free t=
o
>>>>> share your thoughts :)
>>>>> --
>>>>=20
>>>> I think it would be good though specific projects will need/have their
>>>> own testing needs -I'd expect more focus for testing redistributables =
to
>>>> be on helping Hadoop users test their stuff against subsets of data,
>>>> rather than the hadoop-*-dev problem of "stressing the hadoop stack on=
ce
>>>> your latest patch is applied".
>>>>=20
>>>> That said, the whole problem of qualifying an OS, Java release and
>>>> cluster is something we'd expect most end user teams to have to do
>>>> -right now terasort is the main stress test.
>>=20
>>=20
>=20


From general-return-3068-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 17 18:44:13 2011
Return-Path: <general-return-3068-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23658 invoked from network); 17 Feb 2011 18:44:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Feb 2011 18:44:12 -0000
Received: (qmail 35035 invoked by uid 500); 17 Feb 2011 18:44:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34856 invoked by uid 500); 17 Feb 2011 18:44:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34848 invoked by uid 99); 17 Feb 2011 18:44:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 18:44:08 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [208.97.132.66] (HELO homiemail-a70.g.dreamhost.com) (208.97.132.66)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 18:44:02 +0000
Received: from homiemail-a70.g.dreamhost.com (localhost [127.0.0.1])
	by homiemail-a70.g.dreamhost.com (Postfix) with ESMTP id 3E33E76805C
	for <general@hadoop.apache.org>; Thu, 17 Feb 2011 10:43:42 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; q=dns; s=
	gbiv.com; b=xJ0Z92kpkoYmqht4+0N8MH8TX3xRP57liGiqKkarVzJupfucCCCk
	k1/bJqR1hXMpE9tE8yzxk0zX44ehTc6psBAa/1z+cYJ3sV9D80V+1fdW7jxEr7zi
	PJOqfE9effhdMdoIabrNuv+QEIxgitB/UmKytgoCCi4ZqX487R0Vjtg=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; s=gbiv.com;
	 bh=u/1lSHzqIi7ABGa1zkerj0rmSSE=; b=1yrH3btddSOainbdKafzFZCLmm46
	qGO6vEPFC4M6l27P7sIJ8pfwufDT0yum002JHU16z00yw+ajYbSb1PFMinU4rRXN
	RZL/Vmer+nK3I3NSzQ3xl2IxWnOvjJzgz87mdV8smqtr6M+YTssPyfqFba8vdnaV
	XkGx8wf6UNiDjKQ=
Received: from [192.168.1.84] (99-21-208-82.lightspeed.irvnca.sbcglobal.net [99.21.208.82])
	(using TLSv1 with cipher AES128-SHA (128/128 bits))
	(No client certificate requested)
	(Authenticated sender: fielding@gbiv.com)
	by homiemail-a70.g.dreamhost.com (Postfix) with ESMTPSA id BAF74768061
	for <general@hadoop.apache.org>; Thu, 17 Feb 2011 10:43:41 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [VOTE] Abandon hdfsproxy HDFS contrib
From: "Roy T. Fielding" <fielding@gbiv.com>
In-Reply-To: <AANLkTikVcGcKHnh9EgUDQr2JGLPvkBvBRgn4my3uJfOc@mail.gmail.com>
Date: Thu, 17 Feb 2011 10:43:46 -0800
Content-Transfer-Encoding: quoted-printable
Message-Id: <1FEA58C1-AA9A-4A6F-8873-C995057CDD19@gbiv.com>
References: <D97CF74A-147E-4573-B188-2F9C48CFC482@mac.com> <AANLkTi=F2eBsACBfjYSVava+VwFcsRpqZdRRN+OHXNRJ@mail.gmail.com> <823262F0-EE37-4A0A-8F8C-D9422541D67B@mac.com> <0CBAFDBB-AC2A-495D-8936-4DF4D6731B69@Holsman.NET> <AANLkTimUhDbTnwn6Hk2oYXFJMts+A=FnL-q-veXqX1m0@mail.gmail.com> <F2EC1B1B-A2BB-4FC8-9842-3904A25338EE@gbiv.com> <AANLkTikVcGcKHnh9EgUDQr2JGLPvkBvBRgn4my3uJfOc@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

On Feb 17, 2011, at 4:43 AM, Bernd Fondermann wrote:
> We have the very unfortunate situation here at Hadoop where Apache
> Hadoop is not the primary and foremost place of Hadoop development.
> Instead, code is developed internally at Yahoo and then contributed in
> (smaller or larger) chunks to Hadoop.
> This is open source development upside down.

It is not open development.  The development community can do better,
but it has to make up for past mistakes first.

> It is not ok for people to diff ASF svn against their internal code
> and provide the diff as a patch without reviewing IP first for every
> line of code changed.

That is simply untrue.  If the code came from one company's employees
and they all signed an employment agreement with their employer and
the employer approves of the contribution and the committer knows that
when they commit (and logs the authors of the patch when committed),
then all necessary IP clearance has been done.  Committers are =
responsible
for ensuring that they have permission to commit under their own CLA.

> For larger chunks I'd suggest to even go via the Incubator IP =
clearance process.

Nonsense (with director hat on).

> Only then will we force committers to primarily work here in the open
> and return to what I'd consider a healthy project.

No, you'll force people to work on the open by actually collaborating
with them as they work and veto a patch for any technical faults it
may contain.  Pestering them about your personal view of the Apache Way
of development is not a contribution.

....Roy


From general-return-3069-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 17 18:48:29 2011
Return-Path: <general-return-3069-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24673 invoked from network); 17 Feb 2011 18:48:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Feb 2011 18:48:28 -0000
Received: (qmail 41095 invoked by uid 500); 17 Feb 2011 18:48:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40883 invoked by uid 500); 17 Feb 2011 18:48:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40874 invoked by uid 99); 17 Feb 2011 18:48:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 18:48:24 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of binhara@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 18:48:17 +0000
Received: by wye20 with SMTP id 20so2844498wye.35
        for <general@hadoop.apache.org>; Thu, 17 Feb 2011 10:47:56 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=TuxYLXkkoM+3DVi7D5LloxKPWgxmnc4DOQOxIV0IrJs=;
        b=SpSjDWJY2/ro9Ft+zoIhcrvHo3zd9bJRWSqBeY0I/9Teud8/oe70QToEnKqwMpL241
         EP2MoWw5B1IPPVp+J3RD10Rj3HXl5zJjSsMIf2/q7jcZeIQyB8wGtlj7N0N/d3arEmVs
         R6lOKGPdcdrPVlIBCqXzi64O6KxVBGM3MRZxM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=q7yEMZUC68a1XR+pvCQZzRUTsJwjI1EJ3Wp4Rx9om8gOllA6UEEIGjjdAm8AZ9+oBs
         Mav/sVgQtuwtyfcTJlTRFaoSo+tLhVscH/rzTJMdZdfEZx1jXRw/lO2Xu3tDte45DLF/
         hbHZDiin7B8MjfVMaFHW3x0B9/dy5Y4oegvTQ=
MIME-Version: 1.0
Received: by 10.216.51.130 with SMTP id b2mr1959718wec.42.1297968476768; Thu,
 17 Feb 2011 10:47:56 -0800 (PST)
Received: by 10.216.17.146 with HTTP; Thu, 17 Feb 2011 10:47:56 -0800 (PST)
Date: Thu, 17 Feb 2011 16:47:56 -0200
Message-ID: <AANLkTimw9+b=gtfSEfh=capMb83rpAeBU6EbpDbqqUYK@mail.gmail.com>
Subject: simple map reduce
From: Alessandro Binhara <binhara@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6dd8c8dfcc99d049c7ed552
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6dd8c8dfcc99d049c7ed552
Content-Type: text/plain; charset=ISO-8859-1

Helo all..

i create a simple map reduce .. to sum a valeu from a input file like :

key, value
993, 3
993, 2
333, 2

etc .. like this..

public void map(LongWritable key, Text value,
    OutputCollector<Text, IntWritable> output, Reporter reporter)
    throws IOException
    {
 String line = value.toString();
String fields[] = line.split(",");
output.collect(new Text(fields[0]), new
IntWritable(Integer.parseInt(fields[1])));
 }

 public void reduce(Text key, Iterator<IntWritable> values,
OutputCollector<Text, IntWritable> output, Reporter reporter)
      throws IOException
      {
   int maxValue = 0;
   while (values.hasNext()) {
   maxValue +=  values.next().get();
   }
   output.collect(key, new IntWritable(maxValue));
      }


Well.. my question is..

I had a another structure .. of input file
key, value1, value2, value3
993, 3,2,3
993, 2,1,1
333, 2,2,1

How i can sendo to reduce a list of values, ? To process a list numers and
not only one number?
values must be added separately..

thanks for all ..

--0016e6dd8c8dfcc99d049c7ed552--

From general-return-3070-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 17 19:12:22 2011
Return-Path: <general-return-3070-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39108 invoked from network); 17 Feb 2011 19:12:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Feb 2011 19:12:22 -0000
Received: (qmail 70900 invoked by uid 500); 17 Feb 2011 19:12:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70568 invoked by uid 500); 17 Feb 2011 19:12:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70558 invoked by uid 99); 17 Feb 2011 19:12:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 19:12:17 +0000
X-ASF-Spam-Status: No, hits=3.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of akimball83@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 19:12:11 +0000
Received: by bwz8 with SMTP id 8so3007237bwz.35
        for <general@hadoop.apache.org>; Thu, 17 Feb 2011 11:11:50 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:cc:content-type;
        bh=vR2nVdHiuFEUzIN8+VMRjdjcHCoRbhwUFiLZ2cBua8s=;
        b=Fh/4kUDwpMxe44NQk1l0a8RcHj2ObK3bEeacdWzxRLXwe43gs/8JAmMOCe5HDfXI03
         xQYHVSKbEhKCdUHgKh6H4on3og8f+tWuN0KMN7xmF/X7E2vMtn6AQvR5aKMsSVxWDb4q
         LYDWrERcwXXCcSPTcia0iopR2n6AqAgT2np7s=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :cc:content-type;
        b=kmB7natCnu0vACwxPzDYGySuk+XQBZGeB8nMutvSDX+zUahAqsOEKZce57Xcs5he0K
         tcgiN0dfYofPXiLjEbREuH0FEdUpqCStfQBEMNziHIST8eC8bi/JFyQjERh/F1RgbfxR
         r65AiIiWw3kyLP80L9062ZjZ/ORioHlzBZBYI=
Received: by 10.204.70.137 with SMTP id d9mr2071343bkj.141.1297969910357; Thu,
 17 Feb 2011 11:11:50 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.151.215 with HTTP; Thu, 17 Feb 2011 11:11:30 -0800 (PST)
In-Reply-To: <AANLkTi=iGCBrp6+dsK66-+fWF7UoLPSqMOhoNUnjbMHY@mail.gmail.com>
References: <77405974-6771-4604-926B-976240743F3C@mac.com> <AANLkTim80KJ3f14iM6Ltzy7UG613myhN-AiwQ3pCmVdi@mail.gmail.com>
 <AANLkTiknkrWK=GzGMVj4_Z6p5Jn8TQu1P5TkaFSgORJk@mail.gmail.com>
 <AANLkTin0TguLYHi9X0uL3ZF7tCz5xTHBYa7056oKF2Ae@mail.gmail.com>
 <93644D3B-4830-4813-92BE-809738969CF9@apache.org> <AANLkTi=dbGSLhtFA+Rf9LgaX05v45HDLoQDE_2C5ULAf@mail.gmail.com>
 <65311075-A30E-4971-A2BA-C75409C6486E@jpl.nasa.gov> <AANLkTi=XbpVUmg_TpH7eqEn8w7Gcu9KiwtJmKMLa5w5n@mail.gmail.com>
 <FAE19BAA-47A7-4747-BDD8-5AA42C259EB6@jpl.nasa.gov> <AANLkTin2JvOGdZT5g5tmTGeCpD3Qz2soZWcTQtHMsSza@mail.gmail.com>
 <AANLkTi=iGCBrp6+dsK66-+fWF7UoLPSqMOhoNUnjbMHY@mail.gmail.com>
From: Aaron Kimball <akimball83@gmail.com>
Date: Thu, 17 Feb 2011 11:11:30 -0800
Message-ID: <AANLkTimXXt3e4fL==fxhU9MJ5v=hPeEa746-3CV4x4Pc@mail.gmail.com>
Subject: Re: [VOTE] Abandon mrunit MapReduce contrib
To: general@hadoop.apache.org
Cc: Bernd Fondermann <bernd.fondermann@googlemail.com>
Content-Type: multipart/alternative; boundary=001636c5aa556f9e07049c7f2bcc

--001636c5aa556f9e07049c7f2bcc
Content-Type: text/plain; charset=ISO-8859-1

The MRUnit community is a specific subset of the Hadoop community: Engineers
writing Java code running on Hadoop. The Hadoop community also includes
IT/ops staff who maintain Hadoop clusters, data scientists who use tools
such as Pig & Hive, as well as those written by the aforementioned
engineers, etc.

The Hadoop project has long recognized that tools aimed at a specific subset
of the Hadoop community, with separate release cycles, can more successfully
reach their aims by splitting into incubator projects. Hive, Pig, and HBase,
for example, have all gone this path.

A "current" version of MRUnit would need to compile against multiple
versions of Hadoop itself. This is not possible if it is in the same source
tree as Hadoop.

- Aaron

On Thu, Feb 17, 2011 at 5:31 AM, Bernd Fondermann <
bernd.fondermann@googlemail.com> wrote:

> On Fri, Feb 11, 2011 at 23:10, Aaron Kimball <akimball83@gmail.com> wrote:
> > The main reason I am interested in removing MRUnit from Hadoop is that I
> > believe that MRUnit deserves its own release cycle. I think this is in
> the
> > best interest of its users.
>
> Not in mine, at least. (I'm writing MR unit tests.)
> Many projects release more than one product. I'd rather get MRUnit
> from the same source where I get my MR from.
> Separate release cylcles would be ok for me, though.
>
> > Perhaps more importantly, access to new features in MRUnit should not
> > require upgrading one's entire Hadoop deployment; this is a client
> library
> > that depends only on Hadoop's public APIs.
>
> +1.
>
> > My primary concern is to move MRUnit to a place where the community can
> > derive the most benefit from it. The Apache Incubator could fulfill this
> > role; given the presence of individuals willing to mentor this project, I
> > believe this would be a successful way to release MRUnit more quickly and
> > continue to work to grow the MRUnit community.
>
> What are your expectations what MRUnit would become, software-wise?
> Wouldn't the MRUnit community be largely the same as the Hadoop-MR
> community?
>
>  Bernd
>

--001636c5aa556f9e07049c7f2bcc--

From general-return-3071-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 17 19:21:54 2011
Return-Path: <general-return-3071-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 41569 invoked from network); 17 Feb 2011 19:21:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Feb 2011 19:21:53 -0000
Received: (qmail 79052 invoked by uid 500); 17 Feb 2011 19:21:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78593 invoked by uid 500); 17 Feb 2011 19:21:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78585 invoked by uid 99); 17 Feb 2011 19:21:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 19:21:49 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 19:21:43 +0000
Received: by wye20 with SMTP id 20so2877792wye.35
        for <general@hadoop.apache.org>; Thu, 17 Feb 2011 11:21:22 -0800 (PST)
Received: by 10.216.170.149 with SMTP id p21mr2127198wel.81.1297970482344;
 Thu, 17 Feb 2011 11:21:22 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.82.194 with HTTP; Thu, 17 Feb 2011 11:21:02 -0800 (PST)
In-Reply-To: <C982AAB3.FACF%eyang@yahoo-inc.com>
References: <AANLkTin7YOraMb5UXktcYJWLueLwe3o+_bEcQF0r+fse@mail.gmail.com> <C982AAB3.FACF%eyang@yahoo-inc.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Thu, 17 Feb 2011 11:21:02 -0800
X-Google-Sender-Auth: 9fedkSwpQufm2MDbacG1smNltsk
Message-ID: <AANLkTikK6KR8j-a2D-1DUsTeDXexXnQ89zRRCRQ6W17t@mail.gmail.com>
Subject: Re: Hadoop testing project
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Eric.

I am sure that packaging of Hadoop and other application working
directly with Hadoop is a highly needed thing (although there's always
a tricky question how many platforms you plan to provide packaging
for, etc.). What we are discussing here is software testing, not
packaging nor integration issues between packaged bits.

If you want to - please start a separate discussion to avoid steering
this thread away and not mixing the issues.

Cos

> I think the bigger concern is that Hadoop ecosystem does not have a stand=
ard
> method in linking dependencies. =A0Hbase depends on Zookeeper, and Pig de=
pends
> on Hadoop and Hbase. =A0Then pig decided to put hadoop-core jar in it's o=
wn
> jar file. =A0Chukwa depends on pig + hbase + hadoop and zookeeper. =A0The
> version incompatibility is probably what driving people nuts. =A0Hence, t=
here
> is a new proposal on how to integrate among hadoop ecosystem. =A0I urge
> project owners to review the proposal and provide feedbacks.
>
> The proposal is located at:
>
> https://issues.apache.org/jira/secure/attachment/12470823/deployment.pdf
>
> The related jiras are:
>
> https://issues.apache.org/jira/browse/HADOOP-6255
> https://issues.apache.org/jira/browse/PIG-1857
>
> There are plans to file more jiras for related projects. =A0The integrati=
on
> would also be a lot easier if all related projects are using maven for
> dependency management.
>
> Regards,
> Eric
>
> On 2/17/11 9:33 AM, "Konstantin Boudnik" <cos@apache.org> wrote:
>
>> On Thu, Feb 17, 2011 at 05:45, Ian Holsman <hadoop@holsman.net> wrote:
>>> I'm not sure it makes sense to =A0all the testing packages under a diff=
erent
>>> umbrella that covers the code they test.
>>> While there might be commonalities building a test harness, I would thi=
nk
>>> that each testing tool would need to have deep knowledge of the tool's
>>> internals that it is testing. as such it would need someone with the
>>> experience to code it.
>>
>> That's pretty much true indeed if you are talking about tests for a
>> project or closely tightened projects such as Herriot in Hadoop.
>> Speaking of tools there are some benefits though. Say, PigUnit and
>> MRUnit are both xUnit frameworks. The former allows you to run Pig
>> jobs in local and cluster mode. The latter is to validate MB jobs
>> without a need to fire up a cluster.
>>
>>> I don't see what advantage combining PigUnit & say 'MRUnit' would be fo=
r
>>> example.
>>
>> Don't you think Pig user would benefit if Pig scripts can be tested
>> against MRUnit which gives you a flavor of cluster environment without
>> one? Now, do you think it is likely that someone will go great lengths
>> to make such an effort and build such a bridge right now?
>>
>> Cos
>>
>>> --I
>>> On Feb 16, 2011, at 2:50 PM, Konstantin Boudnik wrote:
>>>
>>>> Steve.
>>>>
>>>> If the project under discussion will provide a common harness where su=
ch a
>>>> test
>>>> artifact (think of a Maven artifact for example) will click and will b=
e
>>>> executed automatically with all needed tools and dependencies resolved=
 for
>>>> you
>>>> - would it be appealing for end-users' cause?
>>>>
>>>> As Joep said this "...will reduce the effort to take any (set of ) cha=
nges
>>>> from development into production." Take it one step further: when your
>>>> cluster
>>>> is 'assembled' you need to validate it (on top of a concrete OS, etc.)=
; is
>>>> it
>>>> desirable to follow N-steps process to bring about whatever testing
>>>> work-load
>>>> you need or you'd prefer to simply do something like:
>>>>
>>>> =A0 =A0wget http://workloads.internal.mydomain.com/stackValidations/v1=
2.4.pom \
>>>> =A0 =A0 =A0 =A0&& mvn verify
>>>>
>>>> and check the results later on?
>>>>
>>>> These gonna be the same tools that dev. use for their tasks although
>>>> worksets
>>>> will be different. So what?
>>>>
>>>> Cos
>>>>
>>>> On Wed, Feb 16, 2011 at 11:37AM, Steve Loughran wrote:
>>>>> On 15/02/11 21:58, Konstantin Boudnik wrote:
>>>>>> While MrUnit discussion draws to its natural conclusion I would like
>>>>>> to bring up another point which might be well aligned with that
>>>>>> discussion. Patrick Hunt has brought up this idea earlier today and =
I
>>>>>> believe it has to be elaborated further.
>>>>>>
>>>>>> A number of testing projects both for Hadoop and Hadoop-related
>>>>>> component were brought to life over last year or two. Among those ar=
e
>>>>>> MRUnit, PigUnit, YCSB, Herriot, and perhaps a few more. They all
>>>>>> focusing on more or less the same problem e.g. validation of Hadoop =
or
>>>>>> on-top-of-Hadoop components, or application level testing for Hadoop=
.
>>>>>> However, the fact that they all are spread across a wide variety of
>>>>>> projects seems to confuse/mislead Hadoop users.
>>>>>>
>>>>>> How about incubating a bigger Hadoop (Pig, Oozie, HBase) testing
>>>>>> project which will take care about development and support of common
>>>>>> (where's possible) tools, frameworks and the like? Please feel free =
to
>>>>>> share your thoughts :)
>>>>>> --
>>>>>
>>>>> I think it would be good though specific projects will need/have thei=
r
>>>>> own testing needs -I'd expect more focus for testing redistributables=
 to
>>>>> be on helping Hadoop users test their stuff against subsets of data,
>>>>> rather than the hadoop-*-dev problem of "stressing the hadoop stack o=
nce
>>>>> your latest patch is applied".
>>>>>
>>>>> That said, the whole problem of qualifying an OS, Java release and
>>>>> cluster is something we'd expect most end user teams to have to do
>>>>> -right now terasort is the main stress test.
>>>
>>>
>>
>
>

From general-return-3072-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 17 19:28:16 2011
Return-Path: <general-return-3072-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42813 invoked from network); 17 Feb 2011 19:28:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Feb 2011 19:28:16 -0000
Received: (qmail 86803 invoked by uid 500); 17 Feb 2011 19:28:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86575 invoked by uid 500); 17 Feb 2011 19:28:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86561 invoked by uid 99); 17 Feb 2011 19:28:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 19:28:12 +0000
X-ASF-Spam-Status: No, hits=3.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of akimball83@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 19:28:06 +0000
Received: by bwz8 with SMTP id 8so3022031bwz.35
        for <general@hadoop.apache.org>; Thu, 17 Feb 2011 11:27:45 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type;
        bh=Ix7Eb32bt4zcySoYxulTiPyqSOhYguiJcXnYhTOfXLo=;
        b=yEsE8En5fl+4/GElrOjqfbrROg0zR9IsUT+ktIFVUlPra9J+9VJ/giu7irxRn08sip
         gxlUBCNboEJCMY2Y+nykeGT5YxBagZBYZ7feyJDtDtu2hes2jCnKo7nGT2HKA6CFqARG
         3+W1PSHXjbuEpCQdGsuKtqn0/C5HjN4S036TA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=ooZpLrNaMoQ2kRmmaqVm3/3i7gRPWwzCF9rtgNn0pOhnN5bwLOc1Oy5rB80H5RRILf
         9MxG/NK/i+CUF4M2pn4viLFvvgRIn79jZtpgyBQK1SEQhLBScilpxTgWuN+70gNWOHnD
         3N8/pi68WZWnJp1F7jRYYwz0Hzda3cGUT9X4I=
Received: by 10.204.70.137 with SMTP id d9mr2086176bkj.141.1297970865771; Thu,
 17 Feb 2011 11:27:45 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.151.215 with HTTP; Thu, 17 Feb 2011 11:27:13 -0800 (PST)
In-Reply-To: <F4FCE59AE46B2243922F224426BC869B01AA87C5FF@RHV-MEXMS-002.corp.ebay.com>
References: <AANLkTi=EvhSotauym5K080wy0+UORW+W2Nuh4by5JgdZ@mail.gmail.com> <F4FCE59AE46B2243922F224426BC869B01AA87C5FF@RHV-MEXMS-002.corp.ebay.com>
From: Aaron Kimball <akimball83@gmail.com>
Date: Thu, 17 Feb 2011 11:27:13 -0800
Message-ID: <AANLkTi=9onmmpbzU=t7rvcDPGypRLCfrtMmrdaOdt6sW@mail.gmail.com>
Subject: Re: Hadoop testing project [Was: [VOTE] Abandon mrunit MapReduce contrib]
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c5aa55621419049c7f6469
X-Virus-Checked: Checked by ClamAV on apache.org

--001636c5aa55621419049c7f6469
Content-Type: text/plain; charset=ISO-8859-1

Working to develop code as a client of Hadoop is a path full of landmines.
The more tools we can provide to users to improve the quality of their code,
the better. I think it is important, though, to draw a clear distinction
between tools intended for different audiences. Talking about system testing
tools for Hadoop release/QA processes is good, but one of the benefits I see
of calling MRUnit (designed for client app developers) out of the Hadoop
project at large is to increase its usability. Conflating it with a system
testing tool (for release engineers) would not fulfill that need.

As long as the new project can release several distinct artifacts in a way
that makes their intent clear to the user community, I'm in favor of
gathering as many perspectives on Hadoop testing under one "roof" as
possible.

- Aaron

On Wed, Feb 16, 2011 at 10:19 AM, Rottinghuis, Joep
<jrottinghuis@ebay.com>wrote:

> +1
> Having a coherent approach for system level testing increases confidence in
> the various Hadoop releases and and will reduce the effort to take any (set
> of ) changes from development into production.
> The more automated and formalized system testing, the better!
>
> Thanks,
>
> Joep
> ________________________________________
> From: cos@boudnik.org [cos@boudnik.org] On Behalf Of Konstantin Boudnik [
> cos@apache.org]
> Sent: Tuesday, February 15, 2011 1:58 PM
> To: general@hadoop.apache.org
> Subject: Hadoop testing project [Was: [VOTE] Abandon mrunit MapReduce
> contrib]
>
> While MrUnit discussion draws to its natural conclusion I would like
> to bring up another point which might be well aligned with that
> discussion. Patrick Hunt has brought up this idea earlier today and I
> believe it has to be elaborated further.
>
> A number of testing projects both for Hadoop and Hadoop-related
> component were brought to life over last year or two. Among those are
> MRUnit, PigUnit, YCSB, Herriot, and perhaps a few more. They all
> focusing on more or less the same problem e.g. validation of Hadoop or
> on-top-of-Hadoop components, or application level testing for Hadoop.
> However, the fact that they all are spread across a wide variety of
> projects seems to confuse/mislead Hadoop users.
>
> How about incubating a bigger Hadoop (Pig, Oozie, HBase) testing
> project which will take care about development and support of common
> (where's possible) tools, frameworks and the like? Please feel free to
> share your thoughts :)
> --
>  Take care,
> Konstantin (Cos) Boudnik
>
>
> On Tue, Feb 15, 2011 at 10:44, Eric Sammer <esammer@cloudera.com> wrote:
> > I've started the wiki page proposal for Incubator for mrunit. I'll ping
> > people off list for mentoring. Much appreciated for all the help!
> >
> > On Tue, Feb 15, 2011 at 1:36 PM, Nigel Daley <ndaley@mac.com> wrote:
> >
> >> I'm happy to help mentor as well.
> >>
> >> Cheers,
> >> Nige
> >>
> >> On Feb 11, 2011, at 11:52 AM, Patrick Hunt wrote:
> >>
> >> > On Fri, Feb 11, 2011 at 9:44 AM, Mattmann, Chris A (388J)
> >> > <chris.a.mattmann@jpl.nasa.gov> wrote:
> >> >> Guys, BTW, if you need help or a mentor in Apache Incubator-ville for
> >> MRUnit, I would be happy to help.
> >> >
> >> > I was going to suggest the same thing (mrunit to incubator). I would
> >> > also be happy to be a mentor.
> >> >
> >> > Patrick
>

--001636c5aa55621419049c7f6469--

From general-return-3073-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 17 19:31:52 2011
Return-Path: <general-return-3073-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43842 invoked from network); 17 Feb 2011 19:31:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Feb 2011 19:31:52 -0000
Received: (qmail 1744 invoked by uid 500); 17 Feb 2011 19:31:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1671 invoked by uid 500); 17 Feb 2011 19:31:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1663 invoked by uid 99); 17 Feb 2011 19:31:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 19:31:49 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [128.149.139.106] (HELO mail.jpl.nasa.gov) (128.149.139.106)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 19:31:41 +0000
Received: from mail.jpl.nasa.gov (altvirehtstap02.jpl.nasa.gov [128.149.137.73])
	by smtp.jpl.nasa.gov (Switch-3.4.3/Switch-3.4.3) with ESMTP id p1HJVFVS015478
	(using TLSv1/SSLv3 with cipher RC4-MD5 (128 bits) verified FAIL);
	Thu, 17 Feb 2011 11:31:16 -0800
Received: from ALTPHYEMBEVSP20.RES.AD.JPL ([128.149.137.82]) by
 ALTVIREHTSTAP02.RES.AD.JPL ([128.149.137.73]) with mapi; Thu, 17 Feb 2011
 11:31:09 -0800
From: "Mattmann, Chris A (388J)" <chris.a.mattmann@jpl.nasa.gov>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: Bernd Fondermann <bernd.fondermann@googlemail.com>
Date: Thu, 17 Feb 2011 11:31:06 -0800
Subject: Re: [VOTE] Abandon mrunit MapReduce contrib
Thread-Topic: [VOTE] Abandon mrunit MapReduce contrib
Thread-Index: AcvO2ToMCBnqejDYTC26veHnO5fsSg==
Message-ID: <C1C69515-B440-4C3F-9918-9E9767E4804D@jpl.nasa.gov>
References: <77405974-6771-4604-926B-976240743F3C@mac.com>
 <AANLkTim80KJ3f14iM6Ltzy7UG613myhN-AiwQ3pCmVdi@mail.gmail.com>
 <AANLkTiknkrWK=GzGMVj4_Z6p5Jn8TQu1P5TkaFSgORJk@mail.gmail.com>
 <AANLkTin0TguLYHi9X0uL3ZF7tCz5xTHBYa7056oKF2Ae@mail.gmail.com>
 <93644D3B-4830-4813-92BE-809738969CF9@apache.org>
 <AANLkTi=dbGSLhtFA+Rf9LgaX05v45HDLoQDE_2C5ULAf@mail.gmail.com>
 <65311075-A30E-4971-A2BA-C75409C6486E@jpl.nasa.gov>
 <AANLkTi=XbpVUmg_TpH7eqEn8w7Gcu9KiwtJmKMLa5w5n@mail.gmail.com>
 <FAE19BAA-47A7-4747-BDD8-5AA42C259EB6@jpl.nasa.gov>
 <AANLkTin2JvOGdZT5g5tmTGeCpD3Qz2soZWcTQtHMsSza@mail.gmail.com>
 <AANLkTi=iGCBrp6+dsK66-+fWF7UoLPSqMOhoNUnjbMHY@mail.gmail.com>
 <AANLkTimXXt3e4fL==fxhU9MJ5v=hPeEa746-3CV4x4Pc@mail.gmail.com>
In-Reply-To: <AANLkTimXXt3e4fL==fxhU9MJ5v=hPeEa746-3CV4x4Pc@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Source-IP: altvirehtstap02.jpl.nasa.gov [128.149.137.73]
X-Source-Sender: chris.a.mattmann@jpl.nasa.gov
X-AUTH: Authorized
X-Virus-Checked: Checked by ClamAV on apache.org

Hey Guys,

FYI on this: Eric has mentioned he is going to start the Incubator proposal=
 for MRUnit. Let's start small and then grow big (as needed). It seems like=
 we've achieved enough consensus for the required mentors and critical mass=
 to make an MRUnit Incubator proposal and then to have the Incubator commun=
ity weigh in. If that expands to include other testing projects/etc., we ca=
n address that over the Incubation process, and as needed.=20

Eric: as soon as that wiki page is up, I'd be happy to add my name to it as=
 a mentor and /kick the can on this.

Cheers,
Chris

On Feb 17, 2011, at 11:11 AM, Aaron Kimball wrote:

> The MRUnit community is a specific subset of the Hadoop community: Engine=
ers
> writing Java code running on Hadoop. The Hadoop community also includes
> IT/ops staff who maintain Hadoop clusters, data scientists who use tools
> such as Pig & Hive, as well as those written by the aforementioned
> engineers, etc.
>=20
> The Hadoop project has long recognized that tools aimed at a specific sub=
set
> of the Hadoop community, with separate release cycles, can more successfu=
lly
> reach their aims by splitting into incubator projects. Hive, Pig, and HBa=
se,
> for example, have all gone this path.
>=20
> A "current" version of MRUnit would need to compile against multiple
> versions of Hadoop itself. This is not possible if it is in the same sour=
ce
> tree as Hadoop.
>=20
> - Aaron
>=20
> On Thu, Feb 17, 2011 at 5:31 AM, Bernd Fondermann <
> bernd.fondermann@googlemail.com> wrote:
>=20
>> On Fri, Feb 11, 2011 at 23:10, Aaron Kimball <akimball83@gmail.com> wrot=
e:
>>> The main reason I am interested in removing MRUnit from Hadoop is that =
I
>>> believe that MRUnit deserves its own release cycle. I think this is in
>> the
>>> best interest of its users.
>>=20
>> Not in mine, at least. (I'm writing MR unit tests.)
>> Many projects release more than one product. I'd rather get MRUnit
>> from the same source where I get my MR from.
>> Separate release cylcles would be ok for me, though.
>>=20
>>> Perhaps more importantly, access to new features in MRUnit should not
>>> require upgrading one's entire Hadoop deployment; this is a client
>> library
>>> that depends only on Hadoop's public APIs.
>>=20
>> +1.
>>=20
>>> My primary concern is to move MRUnit to a place where the community can
>>> derive the most benefit from it. The Apache Incubator could fulfill thi=
s
>>> role; given the presence of individuals willing to mentor this project,=
 I
>>> believe this would be a successful way to release MRUnit more quickly a=
nd
>>> continue to work to grow the MRUnit community.
>>=20
>> What are your expectations what MRUnit would become, software-wise?
>> Wouldn't the MRUnit community be largely the same as the Hadoop-MR
>> community?
>>=20
>> Bernd
>>=20


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


From general-return-3074-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 17 20:03:53 2011
Return-Path: <general-return-3074-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53021 invoked from network); 17 Feb 2011 20:03:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Feb 2011 20:03:52 -0000
Received: (qmail 58540 invoked by uid 500); 17 Feb 2011 20:03:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58208 invoked by uid 500); 17 Feb 2011 20:03:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58197 invoked by uid 99); 17 Feb 2011 20:03:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 20:03:49 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 20:03:43 +0000
Received: by wwd20 with SMTP id 20so2895404wwd.29
        for <general@hadoop.apache.org>; Thu, 17 Feb 2011 12:03:22 -0800 (PST)
Received: by 10.216.24.132 with SMTP id x4mr804972wex.81.1297973002318; Thu,
 17 Feb 2011 12:03:22 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.82.194 with HTTP; Thu, 17 Feb 2011 12:03:01 -0800 (PST)
In-Reply-To: <AANLkTi=9onmmpbzU=t7rvcDPGypRLCfrtMmrdaOdt6sW@mail.gmail.com>
References: <AANLkTi=EvhSotauym5K080wy0+UORW+W2Nuh4by5JgdZ@mail.gmail.com>
 <F4FCE59AE46B2243922F224426BC869B01AA87C5FF@RHV-MEXMS-002.corp.ebay.com> <AANLkTi=9onmmpbzU=t7rvcDPGypRLCfrtMmrdaOdt6sW@mail.gmail.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Thu, 17 Feb 2011 12:03:01 -0800
X-Google-Sender-Auth: 6xd6zYCj-Jupm22HEX7ceANyx2Y
Message-ID: <AANLkTi=XEvYgSX2VEQHBMtB9vaX1-oiO5dzCct+ZwUkB@mail.gmail.com>
Subject: Re: Hadoop testing project [Was: [VOTE] Abandon mrunit MapReduce contrib]
To: general@hadoop.apache.org
Cc: Aaron Kimball <akimball83@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Thu, Feb 17, 2011 at 11:27, Aaron Kimball <akimball83@gmail.com> wrote:
> Working to develop code as a client of Hadoop is a path full of landmines=
.
> The more tools we can provide to users to improve the quality of their co=
de,
> the better. I think it is important, though, to draw a clear distinction
> between tools intended for different audiences. Talking about system test=
ing
> tools for Hadoop release/QA processes is good, but one of the benefits I =
see
> of calling MRUnit (designed for client app developers) out of the Hadoop
> project at large is to increase its usability. Conflating it with a syste=
m
> testing tool (for release engineers) would not fulfill that need.

Yup, they are different all right.

> As long as the new project can release several distinct artifacts in a wa=
y
> that makes their intent clear to the user community, I'm in favor of
> gathering as many perspectives on Hadoop testing under one "roof" as
> possible.

That's the goal.

> - Aaron
>
> On Wed, Feb 16, 2011 at 10:19 AM, Rottinghuis, Joep
> <jrottinghuis@ebay.com>wrote:
>
>> +1
>> Having a coherent approach for system level testing increases confidence=
 in
>> the various Hadoop releases and and will reduce the effort to take any (=
set
>> of ) changes from development into production.
>> The more automated and formalized system testing, the better!
>>
>> Thanks,
>>
>> Joep
>> ________________________________________
>> From: cos@boudnik.org [cos@boudnik.org] On Behalf Of Konstantin Boudnik =
[
>> cos@apache.org]
>> Sent: Tuesday, February 15, 2011 1:58 PM
>> To: general@hadoop.apache.org
>> Subject: Hadoop testing project [Was: [VOTE] Abandon mrunit MapReduce
>> contrib]
>>
>> While MrUnit discussion draws to its natural conclusion I would like
>> to bring up another point which might be well aligned with that
>> discussion. Patrick Hunt has brought up this idea earlier today and I
>> believe it has to be elaborated further.
>>
>> A number of testing projects both for Hadoop and Hadoop-related
>> component were brought to life over last year or two. Among those are
>> MRUnit, PigUnit, YCSB, Herriot, and perhaps a few more. They all
>> focusing on more or less the same problem e.g. validation of Hadoop or
>> on-top-of-Hadoop components, or application level testing for Hadoop.
>> However, the fact that they all are spread across a wide variety of
>> projects seems to confuse/mislead Hadoop users.
>>
>> How about incubating a bigger Hadoop (Pig, Oozie, HBase) testing
>> project which will take care about development and support of common
>> (where's possible) tools, frameworks and the like? Please feel free to
>> share your thoughts :)
>> --
>> =A0Take care,
>> Konstantin (Cos) Boudnik
>>
>>
>> On Tue, Feb 15, 2011 at 10:44, Eric Sammer <esammer@cloudera.com> wrote:
>> > I've started the wiki page proposal for Incubator for mrunit. I'll pin=
g
>> > people off list for mentoring. Much appreciated for all the help!
>> >
>> > On Tue, Feb 15, 2011 at 1:36 PM, Nigel Daley <ndaley@mac.com> wrote:
>> >
>> >> I'm happy to help mentor as well.
>> >>
>> >> Cheers,
>> >> Nige
>> >>
>> >> On Feb 11, 2011, at 11:52 AM, Patrick Hunt wrote:
>> >>
>> >> > On Fri, Feb 11, 2011 at 9:44 AM, Mattmann, Chris A (388J)
>> >> > <chris.a.mattmann@jpl.nasa.gov> wrote:
>> >> >> Guys, BTW, if you need help or a mentor in Apache Incubator-ville =
for
>> >> MRUnit, I would be happy to help.
>> >> >
>> >> > I was going to suggest the same thing (mrunit to incubator). I woul=
d
>> >> > also be happy to be a mentor.
>> >> >
>> >> > Patrick
>>
>

From general-return-3075-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 17 21:00:56 2011
Return-Path: <general-return-3075-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77528 invoked from network); 17 Feb 2011 21:00:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Feb 2011 21:00:55 -0000
Received: (qmail 55567 invoked by uid 500); 17 Feb 2011 21:00:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55436 invoked by uid 500); 17 Feb 2011 21:00:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55425 invoked by uid 99); 17 Feb 2011 21:00:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 21:00:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 17 Feb 2011 21:00:51 +0000
Received: (qmail 77006 invoked by uid 99); 17 Feb 2011 21:00:31 -0000
Received: from localhost.apache.org (HELO mail-iw0-f176.google.com) (127.0.0.1)
  (smtp-auth username phunt, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 21:00:31 +0000
Received: by iwn2 with SMTP id 2so3014315iwn.35
        for <general@hadoop.apache.org>; Thu, 17 Feb 2011 13:00:30 -0800 (PST)
MIME-Version: 1.0
Received: by 10.42.179.132 with SMTP id bq4mr3461636icb.293.1297976430276;
 Thu, 17 Feb 2011 13:00:30 -0800 (PST)
Received: by 10.42.174.193 with HTTP; Thu, 17 Feb 2011 13:00:30 -0800 (PST)
In-Reply-To: <C1C69515-B440-4C3F-9918-9E9767E4804D@jpl.nasa.gov>
References: <77405974-6771-4604-926B-976240743F3C@mac.com>
	<AANLkTim80KJ3f14iM6Ltzy7UG613myhN-AiwQ3pCmVdi@mail.gmail.com>
	<AANLkTiknkrWK=GzGMVj4_Z6p5Jn8TQu1P5TkaFSgORJk@mail.gmail.com>
	<AANLkTin0TguLYHi9X0uL3ZF7tCz5xTHBYa7056oKF2Ae@mail.gmail.com>
	<93644D3B-4830-4813-92BE-809738969CF9@apache.org>
	<AANLkTi=dbGSLhtFA+Rf9LgaX05v45HDLoQDE_2C5ULAf@mail.gmail.com>
	<65311075-A30E-4971-A2BA-C75409C6486E@jpl.nasa.gov>
	<AANLkTi=XbpVUmg_TpH7eqEn8w7Gcu9KiwtJmKMLa5w5n@mail.gmail.com>
	<FAE19BAA-47A7-4747-BDD8-5AA42C259EB6@jpl.nasa.gov>
	<AANLkTin2JvOGdZT5g5tmTGeCpD3Qz2soZWcTQtHMsSza@mail.gmail.com>
	<AANLkTi=iGCBrp6+dsK66-+fWF7UoLPSqMOhoNUnjbMHY@mail.gmail.com>
	<AANLkTimXXt3e4fL==fxhU9MJ5v=hPeEa746-3CV4x4Pc@mail.gmail.com>
	<C1C69515-B440-4C3F-9918-9E9767E4804D@jpl.nasa.gov>
Date: Thu, 17 Feb 2011 13:00:30 -0800
Message-ID: <AANLkTimBS+KLnknfh8hGzF3s3xdqN5fVjEexy3cpXzE7@mail.gmail.com>
Subject: Re: [VOTE] Abandon mrunit MapReduce contrib
From: Patrick Hunt <phunt@apache.org>
To: "Mattmann, Chris A (388J)" <chris.a.mattmann@jpl.nasa.gov>
Cc: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Chris a page is up (still being created by Eric afaict):
http://wiki.apache.org/incubator/MRUnitProposal

I took the liberty of listing us as mentors.

Patrick

On Thu, Feb 17, 2011 at 11:31 AM, Mattmann, Chris A (388J)
<chris.a.mattmann@jpl.nasa.gov> wrote:
> Hey Guys,
>
> FYI on this: Eric has mentioned he is going to start the Incubator propos=
al for MRUnit. Let's start small and then grow big (as needed). It seems li=
ke we've achieved enough consensus for the required mentors and critical ma=
ss to make an MRUnit Incubator proposal and then to have the Incubator comm=
unity weigh in. If that expands to include other testing projects/etc., we =
can address that over the Incubation process, and as needed.
>
> Eric: as soon as that wiki page is up, I'd be happy to add my name to it =
as a mentor and /kick the can on this.
>
> Cheers,
> Chris
>
> On Feb 17, 2011, at 11:11 AM, Aaron Kimball wrote:
>
>> The MRUnit community is a specific subset of the Hadoop community: Engin=
eers
>> writing Java code running on Hadoop. The Hadoop community also includes
>> IT/ops staff who maintain Hadoop clusters, data scientists who use tools
>> such as Pig & Hive, as well as those written by the aforementioned
>> engineers, etc.
>>
>> The Hadoop project has long recognized that tools aimed at a specific su=
bset
>> of the Hadoop community, with separate release cycles, can more successf=
ully
>> reach their aims by splitting into incubator projects. Hive, Pig, and HB=
ase,
>> for example, have all gone this path.
>>
>> A "current" version of MRUnit would need to compile against multiple
>> versions of Hadoop itself. This is not possible if it is in the same sou=
rce
>> tree as Hadoop.
>>
>> - Aaron
>>
>> On Thu, Feb 17, 2011 at 5:31 AM, Bernd Fondermann <
>> bernd.fondermann@googlemail.com> wrote:
>>
>>> On Fri, Feb 11, 2011 at 23:10, Aaron Kimball <akimball83@gmail.com> wro=
te:
>>>> The main reason I am interested in removing MRUnit from Hadoop is that=
 I
>>>> believe that MRUnit deserves its own release cycle. I think this is in
>>> the
>>>> best interest of its users.
>>>
>>> Not in mine, at least. (I'm writing MR unit tests.)
>>> Many projects release more than one product. I'd rather get MRUnit
>>> from the same source where I get my MR from.
>>> Separate release cylcles would be ok for me, though.
>>>
>>>> Perhaps more importantly, access to new features in MRUnit should not
>>>> require upgrading one's entire Hadoop deployment; this is a client
>>> library
>>>> that depends only on Hadoop's public APIs.
>>>
>>> +1.
>>>
>>>> My primary concern is to move MRUnit to a place where the community ca=
n
>>>> derive the most benefit from it. The Apache Incubator could fulfill th=
is
>>>> role; given the presence of individuals willing to mentor this project=
, I
>>>> believe this would be a successful way to release MRUnit more quickly =
and
>>>> continue to work to grow the MRUnit community.
>>>
>>> What are your expectations what MRUnit would become, software-wise?
>>> Wouldn't the MRUnit community be largely the same as the Hadoop-MR
>>> community?
>>>
>>> Bernd
>>>
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: =A0 http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>

From general-return-3076-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 17 21:22:26 2011
Return-Path: <general-return-3076-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 92497 invoked from network); 17 Feb 2011 21:22:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Feb 2011 21:22:26 -0000
Received: (qmail 83304 invoked by uid 500); 17 Feb 2011 21:22:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83233 invoked by uid 500); 17 Feb 2011 21:22:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83222 invoked by uid 99); 17 Feb 2011 21:22:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 21:22:24 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of shv.hadoop@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 21:22:19 +0000
Received: by wye20 with SMTP id 20so2999947wye.35
        for <general@hadoop.apache.org>; Thu, 17 Feb 2011 13:21:57 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=ynExP/Su4oHiIX/AkGboMALODLLUBXQSEMegDMfK5oY=;
        b=DNPrVO6pn8DECFqZrGmnXLvUCyjJqf0AXHs/XOBl9yZWGYmgk1Ul0THcQY2SE4JZBg
         8jzi6KKU6W0WXr2sHsWYiBK6bzEUaZIy17bgWhQ6UcGQ1QXhjGcNcN8xjsKu7zZOhOOt
         0bRqHbccRwxQomHORRL5DQsEG30JBAd6HMKts=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Fp7geDvOyXoHeNDjhVBcAXV/GyY10Pi2wX4IzSjsPBppbsvEKmW8pSnUlO9ldreyQH
         6A/UhYhOFA23B/tUCKWmwQ4ZH9jZ9n35jjPY/eQIx1Vf69zyE0izfw561Kg+T/mQ9PDE
         R2DBeF0mjQqC+vPsUPpR/TU4BtRxDHGtNMYTM=
MIME-Version: 1.0
Received: by 10.227.136.84 with SMTP id q20mr2127311wbt.72.1297977717653; Thu,
 17 Feb 2011 13:21:57 -0800 (PST)
Received: by 10.227.72.133 with HTTP; Thu, 17 Feb 2011 13:21:57 -0800 (PST)
In-Reply-To: <B61B4A67-CB97-4638-AEF8-C7FAE09001BC@yahoo-inc.com>
References: <D97CF74A-147E-4573-B188-2F9C48CFC482@mac.com>
	<AANLkTi=F2eBsACBfjYSVava+VwFcsRpqZdRRN+OHXNRJ@mail.gmail.com>
	<823262F0-EE37-4A0A-8F8C-D9422541D67B@mac.com>
	<0CBAFDBB-AC2A-495D-8936-4DF4D6731B69@Holsman.NET>
	<B61B4A67-CB97-4638-AEF8-C7FAE09001BC@yahoo-inc.com>
Date: Thu, 17 Feb 2011 13:21:57 -0800
Message-ID: <AANLkTi=POJgv0Pc4XBQ29ahAhU-ZoLcf1hK_CvzgRPW_@mail.gmail.com>
Subject: Re: [VOTE] Abandon hdfsproxy HDFS contrib
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64974e0c9813e049c80fc0c

--0016e64974e0c9813e049c80fc0c
Content-Type: text/plain; charset=ISO-8859-1

hdfsproxy is a wrapper around hftpFileSystem (in its current state).
So you can always replace hdfsproxy with hftpFileSystem.
Also it uses pure FileSystem api, so it can successfully be maintained
outside of hdfs.

Therefore I am +1 removing it from hdfs/contrib.

What is the use case for hdfsproxy anyways?
Thanks,
--Konstantin


On Wed, Feb 16, 2011 at 8:31 PM, Sanjay Radia <sradia@yahoo-inc.com> wrote:

>
> On Feb 11, 2011, at 12:03 PM, Ian Holsman wrote:
>
>
>> On Feb 11, 2011, at 5:11 PM, Nigel Daley wrote:
>>
>>
>>> On Feb 10, 2011, at 9:24 PM, Owen O'Malley wrote:
>>>
>>>  On Thu, Feb 10, 2011 at 7:40 PM, Nigel Daley <ndaley@mac.com> wrote:
>>>>
>>>>  I think the PMC should abandon the hdfsproxy HDFS contrib component.
>>>>>  It's
>>>>> last meaningful contribution was August 2010:
>>>>>
>>>>>
>>>> -1 we still use and are maintaining this.
>>>>
>>>
>>>
>>> Who's the 'we'?  Looking at HDFS-1164 it looks like hdfsproxy was failing
>>> a unit test for 7 months.  This is exactly the reason we should thoughtfully
>>> consider whether it has a future within Hadoop.
>>>
>>> Perhaps Y! uses a different version of hdfsproxy?
>>>
>>
>> They probably have patched it, and mistakenly forgot to submit them
>>
> Yes, we have some updates that we haven;t got around to pushing around
>
>  .. any chance of doing a diff on your version and submitting it?
>>
>
>
> I will check into this and get back.
> We are also working to incorporate some of the features of the proxy into
> hdfs proper -- if that work completes and is accepted by the community then
> hdfs proxy may become
> less useful.
>
> Hence, for now,  my vote is -1 for removing hdfs proxy.
>
> sanjay
>
> sanjay
>
>>
>> Regards
>> Ian
>>
>>
>>> Nige
>>>
>>>
>>
>

--0016e64974e0c9813e049c80fc0c--

From general-return-3077-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 17 21:25:50 2011
Return-Path: <general-return-3077-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93609 invoked from network); 17 Feb 2011 21:25:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Feb 2011 21:25:50 -0000
Received: (qmail 88722 invoked by uid 500); 17 Feb 2011 21:25:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88629 invoked by uid 500); 17 Feb 2011 21:25:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88621 invoked by uid 99); 17 Feb 2011 21:25:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 21:25:48 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 21:25:42 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=h1LDvivOuWFlbJt/pY1xpplV5AlShp5891johTnMriVwNCbxNVfMlD4l
   TPOf1lDJ4J8GroSAE+4pPgeblQqpxLgSVJ2Vs67PnQxOoL4/Zw11lfOAN
   KKOanOmKWvJJd1+;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1297977939; x=1329513939;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20[VOTE]=20Abandon=20hdfsproxy=20HDFS=20c
   ontrib|Date:=20Thu,=2017=20Feb=202011=2021:26:54=20+0000
   |Message-ID:=20<71308072-C3D4-4A90-A4E7-588C91C5D088@link
   edin.com>|To:=20"<general@hadoop.apache.org>"=20<general@
   hadoop.apache.org>|MIME-Version:=201.0
   |Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<504A3A45A83EF14083C24E6167EF9191@linkedin
   .com>|In-Reply-To:=20<AANLkTi=3DPOJgv0Pc4XBQ29ahAhU-ZoLcf
   1hK_CvzgRPW_@mail.gmail.com>|References:=20<D97CF74A-147E
   -4573-B188-2F9C48CFC482@mac.com>=0D=0A=20<AANLkTi=3DF2eBs
   ACBfjYSVava+VwFcsRpqZdRRN+OHXNRJ@mail.gmail.com>=0D=0A=20
   <823262F0-EE37-4A0A-8F8C-D9422541D67B@mac.com>=0D=0A=20<0
   CBAFDBB-AC2A-495D-8936-4DF4D6731B69@Holsman.NET>=0D=0A=20
   <B61B4A67-CB97-4638-AEF8-C7FAE09001BC@yahoo-inc.com>=0D
   =0A=20<AANLkTi=3DPOJgv0Pc4XBQ29ahAhU-ZoLcf1hK_CvzgRPW_@ma
   il.gmail.com>;
  bh=XD5zD7I3Vb3vcr7goloisqsVXb64wJFMqKBDYO3nPOY=;
  b=aB+gdM4ND1LgLD22wxEcFTkLGBhG6VHZ670spVWpMMlO5pOvnUDnq9h9
   1y6Hr9lTwgpYvNTOu/nMZfrkXgKpHauzsyWeeAKsGT51cZanBAt7slc7g
   DMMSbkGftboBAOm;
X-IronPort-AV: E=Sophos;i="4.62,182,1297065600"; 
   d="scan'208";a="20287068"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi id 14.01.0251.000; Thu,
 17 Feb 2011 13:26:54 -0800
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: [VOTE] Abandon hdfsproxy HDFS contrib
Thread-Topic: [VOTE] Abandon hdfsproxy HDFS contrib
Thread-Index: AQHLyZ27ICuajETxE02yFg9O1wZLWpP8Sq4AgAANWQCAAAYpgIAJS9iAgAEaQoCAAADyAA==
Date: Thu, 17 Feb 2011 21:26:54 +0000
Message-ID: <71308072-C3D4-4A90-A4E7-588C91C5D088@linkedin.com>
References: <D97CF74A-147E-4573-B188-2F9C48CFC482@mac.com>
 <AANLkTi=F2eBsACBfjYSVava+VwFcsRpqZdRRN+OHXNRJ@mail.gmail.com>
 <823262F0-EE37-4A0A-8F8C-D9422541D67B@mac.com>
 <0CBAFDBB-AC2A-495D-8936-4DF4D6731B69@Holsman.NET>
 <B61B4A67-CB97-4638-AEF8-C7FAE09001BC@yahoo-inc.com>
 <AANLkTi=POJgv0Pc4XBQ29ahAhU-ZoLcf1hK_CvzgRPW_@mail.gmail.com>
In-Reply-To: <AANLkTi=POJgv0Pc4XBQ29ahAhU-ZoLcf1hK_CvzgRPW_@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <504A3A45A83EF14083C24E6167EF9191@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


On Feb 17, 2011, at 1:21 PM, Konstantin Shvachko wrote:

> hdfsproxy is a wrapper around hftpFileSystem (in its current state).
> So you can always replace hdfsproxy with hftpFileSystem.
> Also it uses pure FileSystem api, so it can successfully be maintained
> outside of hdfs.
>=20
> Therefore I am +1 removing it from hdfs/contrib.
>=20
> What is the use case for hdfsproxy anyways?

	A stable, secure http get interface for files.  (No, the normal web ui is =
not good enough.  Think firewalls.).




From general-return-3078-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 17 21:52:47 2011
Return-Path: <general-return-3078-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5003 invoked from network); 17 Feb 2011 21:52:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Feb 2011 21:52:47 -0000
Received: (qmail 40200 invoked by uid 500); 17 Feb 2011 21:52:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40088 invoked by uid 500); 17 Feb 2011 21:52:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40072 invoked by uid 99); 17 Feb 2011 21:52:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 21:52:45 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [128.149.139.106] (HELO mail.jpl.nasa.gov) (128.149.139.106)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Feb 2011 21:52:37 +0000
Received: from mail.jpl.nasa.gov (altvirehtstap02.jpl.nasa.gov [128.149.137.73])
	by smtp.jpl.nasa.gov (Switch-3.4.3/Switch-3.4.3) with ESMTP id p1HLqCOL006089
	(using TLSv1/SSLv3 with cipher RC4-MD5 (128 bits) verified FAIL)
	for <general@hadoop.apache.org>; Thu, 17 Feb 2011 13:52:13 -0800
Received: from ALTPHYEMBEVSP20.RES.AD.JPL ([128.149.137.82]) by
 ALTVIREHTSTAP02.RES.AD.JPL ([128.149.137.73]) with mapi; Thu, 17 Feb 2011
 13:52:12 -0800
From: "Mattmann, Chris A (388J)" <chris.a.mattmann@jpl.nasa.gov>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 17 Feb 2011 13:52:09 -0800
Subject: Re: [VOTE] Abandon mrunit MapReduce contrib
Thread-Topic: [VOTE] Abandon mrunit MapReduce contrib
Thread-Index: AcvO7O5pLUWIIUuZTh+5q5F8HHyjHw==
Message-ID: <5C5BE47B-6159-4678-8455-219CF1F41863@jpl.nasa.gov>
References: <77405974-6771-4604-926B-976240743F3C@mac.com>
 <AANLkTim80KJ3f14iM6Ltzy7UG613myhN-AiwQ3pCmVdi@mail.gmail.com>
 <AANLkTiknkrWK=GzGMVj4_Z6p5Jn8TQu1P5TkaFSgORJk@mail.gmail.com>
 <AANLkTin0TguLYHi9X0uL3ZF7tCz5xTHBYa7056oKF2Ae@mail.gmail.com>
 <93644D3B-4830-4813-92BE-809738969CF9@apache.org>
 <AANLkTi=dbGSLhtFA+Rf9LgaX05v45HDLoQDE_2C5ULAf@mail.gmail.com>
 <65311075-A30E-4971-A2BA-C75409C6486E@jpl.nasa.gov>
 <AANLkTi=XbpVUmg_TpH7eqEn8w7Gcu9KiwtJmKMLa5w5n@mail.gmail.com>
 <FAE19BAA-47A7-4747-BDD8-5AA42C259EB6@jpl.nasa.gov>
 <AANLkTin2JvOGdZT5g5tmTGeCpD3Qz2soZWcTQtHMsSza@mail.gmail.com>
 <AANLkTi=iGCBrp6+dsK66-+fWF7UoLPSqMOhoNUnjbMHY@mail.gmail.com>
 <AANLkTimXXt3e4fL==fxhU9MJ5v=hPeEa746-3CV4x4Pc@mail.gmail.com>
 <C1C69515-B440-4C3F-9918-9E9767E4804D@jpl.nasa.gov>
 <AANLkTimBS+KLnknfh8hGzF3s3xdqN5fVjEexy3cpXzE7@mail.gmail.com>
In-Reply-To: <AANLkTimBS+KLnknfh8hGzF3s3xdqN5fVjEexy3cpXzE7@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Source-IP: altvirehtstap02.jpl.nasa.gov [128.149.137.73]
X-Source-Sender: chris.a.mattmann@jpl.nasa.gov
X-AUTH: Authorized
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks Patrick.

Sounds good. Besides mentors, we need a champion, who must be an ASF member=
. So, either me, you or Nigel can do it. I'm already championing Gora, and =
would be happy to champion MRUnit, but am open to either one of you guys do=
ing it too (or even another ASF member). Just let me know. I'll head over t=
o the wiki and add my info. Thanks for getting this started Eric+Patrick!

Cheers,
Chris

On Feb 17, 2011, at 1:00 PM, Patrick Hunt wrote:

> Chris a page is up (still being created by Eric afaict):
> http://wiki.apache.org/incubator/MRUnitProposal
>=20
> I took the liberty of listing us as mentors.
>=20
> Patrick
>=20
> On Thu, Feb 17, 2011 at 11:31 AM, Mattmann, Chris A (388J)
> <chris.a.mattmann@jpl.nasa.gov> wrote:
>> Hey Guys,
>>=20
>> FYI on this: Eric has mentioned he is going to start the Incubator propo=
sal for MRUnit. Let's start small and then grow big (as needed). It seems l=
ike we've achieved enough consensus for the required mentors and critical m=
ass to make an MRUnit Incubator proposal and then to have the Incubator com=
munity weigh in. If that expands to include other testing projects/etc., we=
 can address that over the Incubation process, and as needed.
>>=20
>> Eric: as soon as that wiki page is up, I'd be happy to add my name to it=
 as a mentor and /kick the can on this.
>>=20
>> Cheers,
>> Chris
>>=20
>> On Feb 17, 2011, at 11:11 AM, Aaron Kimball wrote:
>>=20
>>> The MRUnit community is a specific subset of the Hadoop community: Engi=
neers
>>> writing Java code running on Hadoop. The Hadoop community also includes
>>> IT/ops staff who maintain Hadoop clusters, data scientists who use tool=
s
>>> such as Pig & Hive, as well as those written by the aforementioned
>>> engineers, etc.
>>>=20
>>> The Hadoop project has long recognized that tools aimed at a specific s=
ubset
>>> of the Hadoop community, with separate release cycles, can more success=
fully
>>> reach their aims by splitting into incubator projects. Hive, Pig, and H=
Base,
>>> for example, have all gone this path.
>>>=20
>>> A "current" version of MRUnit would need to compile against multiple
>>> versions of Hadoop itself. This is not possible if it is in the same so=
urce
>>> tree as Hadoop.
>>>=20
>>> - Aaron
>>>=20
>>> On Thu, Feb 17, 2011 at 5:31 AM, Bernd Fondermann <
>>> bernd.fondermann@googlemail.com> wrote:
>>>=20
>>>> On Fri, Feb 11, 2011 at 23:10, Aaron Kimball <akimball83@gmail.com> wr=
ote:
>>>>> The main reason I am interested in removing MRUnit from Hadoop is tha=
t I
>>>>> believe that MRUnit deserves its own release cycle. I think this is i=
n
>>>> the
>>>>> best interest of its users.
>>>>=20
>>>> Not in mine, at least. (I'm writing MR unit tests.)
>>>> Many projects release more than one product. I'd rather get MRUnit
>>>> from the same source where I get my MR from.
>>>> Separate release cylcles would be ok for me, though.
>>>>=20
>>>>> Perhaps more importantly, access to new features in MRUnit should not
>>>>> require upgrading one's entire Hadoop deployment; this is a client
>>>> library
>>>>> that depends only on Hadoop's public APIs.
>>>>=20
>>>> +1.
>>>>=20
>>>>> My primary concern is to move MRUnit to a place where the community c=
an
>>>>> derive the most benefit from it. The Apache Incubator could fulfill t=
his
>>>>> role; given the presence of individuals willing to mentor this projec=
t, I
>>>>> believe this would be a successful way to release MRUnit more quickly=
 and
>>>>> continue to work to grow the MRUnit community.
>>>>=20
>>>> What are your expectations what MRUnit would become, software-wise?
>>>> Wouldn't the MRUnit community be largely the same as the Hadoop-MR
>>>> community?
>>>>=20
>>>> Bernd
>>>>=20
>>=20
>>=20
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW:   http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>=20
>>=20


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


From general-return-3079-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 18 03:16:18 2011
Return-Path: <general-return-3079-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56898 invoked from network); 18 Feb 2011 03:16:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2011 03:16:18 -0000
Received: (qmail 66777 invoked by uid 500); 18 Feb 2011 03:16:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66159 invoked by uid 500); 18 Feb 2011 03:16:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66151 invoked by uid 99); 18 Feb 2011 03:16:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 03:16:12 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 03:16:07 +0000
Received: by fxm2 with SMTP id 2so3261882fxm.35
        for <general@hadoop.apache.org>; Thu, 17 Feb 2011 19:15:46 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type;
        bh=aGFnWBN28iKEjpTUEMJz242edcRcZXEaD842M4aFisM=;
        b=tkb50Js4HMisw5OzTGs0Fyw8BT8Cyoa0uY/BOsYI+SPuOnEYxdd4bUnr9Yw51XH896
         EnuZxUokMaqbFeu+Duf66r38VRVfQLDe2MnZRezrzdUi969FuWyefh03SPSHVoMQsSKw
         kWQNOyH1yTtaVj+nHGXx+Fqh2PtPfgKpjSi+s=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=LZwSbVN2DcTM7YJwqpIls32r9BK3yeAKpgl5zvJbKJTqZwZWykGsECJWDSZYcVys/u
         /Kqf2uE3eqkUKIr2ygIUI1LpBoq9u6/LtniCRLX8R201S4jnHFi4qayvUB/vZiIImMvN
         i+DD99uvimE+anG1EUaXxjop1xT+aIqQ172lg=
Received: by 10.223.83.194 with SMTP id g2mr253455fal.75.1297998945077; Thu,
 17 Feb 2011 19:15:45 -0800 (PST)
MIME-Version: 1.0
Received: by 10.223.122.83 with HTTP; Thu, 17 Feb 2011 19:15:25 -0800 (PST)
In-Reply-To: <AANLkTimw9+b=gtfSEfh=capMb83rpAeBU6EbpDbqqUYK@mail.gmail.com>
References: <AANLkTimw9+b=gtfSEfh=capMb83rpAeBU6EbpDbqqUYK@mail.gmail.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Fri, 18 Feb 2011 08:45:25 +0530
Message-ID: <AANLkTimUPJ__9WODxthGrWEc5C6JJBcVeYOUT=mmum9W@mail.gmail.com>
Subject: Re: simple map reduce
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hello,

On Fri, Feb 18, 2011 at 12:17 AM, Alessandro Binhara <binhara@gmail.com> wrote:
> I had a another structure .. of input file
> key, value1, value2, value3
> 993, 3,2,3
> 993, 2,1,1
> 333, 2,2,1
>
> How i can sendo to reduce a list of values, ? To process a list numers and
> not only one number?
> values must be added separately..

An IntWritable can't obviously hold a sequence of integers. So you
need a different data structure to hold a sequence of integers. From
the available bank of Writables in Hadoop, you can perhaps use an
ArrayWritable initialized for IntWritable members.

-- 
Harsh J
www.harshj.com

From general-return-3080-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 18 06:11:27 2011
Return-Path: <general-return-3080-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30375 invoked from network); 18 Feb 2011 06:11:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2011 06:11:27 -0000
Received: (qmail 66609 invoked by uid 500); 18 Feb 2011 06:11:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66182 invoked by uid 500); 18 Feb 2011 06:11:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66171 invoked by uid 99); 18 Feb 2011 06:11:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 06:11:21 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 06:11:16 +0000
Received: by iwn2 with SMTP id 2so3417109iwn.35
        for <general@hadoop.apache.org>; Thu, 17 Feb 2011 22:10:55 -0800 (PST)
Received: by 10.231.39.73 with SMTP id f9mr259853ibe.79.1298009455173; Thu, 17
 Feb 2011 22:10:55 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.42.201 with HTTP; Thu, 17 Feb 2011 22:10:34 -0800 (PST)
From: Todd Lipcon <todd@cloudera.com>
Date: Thu, 17 Feb 2011 22:10:34 -0800
Message-ID: <AANLkTi=37eY=sUBPX8TtjoFge=KnUEEjQ8iOqjMvqDFa@mail.gmail.com>
Subject: [ANN] HBase 0.90.1 available for download
To: general@hadoop.apache.org, user@hbase.apache.org
Content-Type: multipart/alternative; boundary=0003255752fe7da23b049c886054

--0003255752fe7da23b049c886054
Content-Type: text/plain; charset=ISO-8859-1

The Apache HBase team is happy to announce the general availability of HBase
0.90.1, available from your Apache mirror of choice:

http://www.apache.org/dyn/closer.cgi/hbase/
[at the time of this writing, not all mirrors have updated yet -- please
pick a different mirror if your first choice does not show 0.90.1]

HBase 0.90.1 is a maintenance release that fixes several important bugs
since version 0.90.0, while retaining API and data compatibility. The
release notes may be found on the Apache JIRA:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12315548

Users upgrading from HBase 0.90.0 may upgrade clients and servers
separately, though it is recommended that both be upgraded. If upgrading
from a version of HBase prior to 0.90.0, please read the notes accompanying
that release:
http://osdir.com/ml/general-hadoop-apache/2011-01/msg00208.html

As always, many thanks to those who contributed to this release!

-The HBase Team

--0003255752fe7da23b049c886054--

From general-return-3081-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 18 10:12:31 2011
Return-Path: <general-return-3081-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 20773 invoked from network); 18 Feb 2011 10:12:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2011 10:12:30 -0000
Received: (qmail 73631 invoked by uid 500); 18 Feb 2011 10:12:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73080 invoked by uid 500); 18 Feb 2011 10:12:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73072 invoked by uid 99); 18 Feb 2011 10:12:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 10:12:23 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bernd.fondermann@googlemail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 10:12:19 +0000
Received: by iwn2 with SMTP id 2so3585966iwn.35
        for <general@hadoop.apache.org>; Fri, 18 Feb 2011 02:11:58 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type:content-transfer-encoding;
        bh=PNMA50ktIQxKHR5wLRwpzNxP46y1BENja+FY/x0qBc0=;
        b=ZTBIxzkqQBmuafwKJNXr7oNmYoQYh/7Rb4UbclltOcFan4hZO+f0MW7MWh6VSE/yGc
         UiGPtrGD1dp2u5vuZhI25VbwG8l/o0+xghAFv1xlGwfI/Bo5u7j5sLW5CNgUZmYM/qc6
         sVOxOB8hVEb03KPQrQPiP7Dp8KOwlVGtGeFSo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=hAtDBK5mE8095X/Y3alYDAkkHT2Cd2C3Qjn0gmg/ZkIN4uoeuo3RCWfv+JJOV9HuMj
         wrWwp0962NHapXgFq1iV/Q3M5O4b5Lvtlopzk/NnPgZ+qFKesr5py2xuih0uNSZTnMUM
         zR4dUFr58eVcoWTLKmryhi7uAMkyrNE4vZjxw=
MIME-Version: 1.0
Received: by 10.42.213.66 with SMTP id gv2mr670456icb.81.1298023918638; Fri,
 18 Feb 2011 02:11:58 -0800 (PST)
Received: by 10.42.8.149 with HTTP; Fri, 18 Feb 2011 02:11:58 -0800 (PST)
In-Reply-To: <1FEA58C1-AA9A-4A6F-8873-C995057CDD19@gbiv.com>
References: <D97CF74A-147E-4573-B188-2F9C48CFC482@mac.com>
	<AANLkTi=F2eBsACBfjYSVava+VwFcsRpqZdRRN+OHXNRJ@mail.gmail.com>
	<823262F0-EE37-4A0A-8F8C-D9422541D67B@mac.com>
	<0CBAFDBB-AC2A-495D-8936-4DF4D6731B69@Holsman.NET>
	<AANLkTimUhDbTnwn6Hk2oYXFJMts+A=FnL-q-veXqX1m0@mail.gmail.com>
	<F2EC1B1B-A2BB-4FC8-9842-3904A25338EE@gbiv.com>
	<AANLkTikVcGcKHnh9EgUDQr2JGLPvkBvBRgn4my3uJfOc@mail.gmail.com>
	<1FEA58C1-AA9A-4A6F-8873-C995057CDD19@gbiv.com>
Date: Fri, 18 Feb 2011 11:11:58 +0100
Message-ID: <AANLkTik1LFE4z0PGoOEudbFJ93e+LCrwTGLPmHybo-AL@mail.gmail.com>
Subject: Re: [VOTE] Abandon hdfsproxy HDFS contrib
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Thu, Feb 17, 2011 at 19:43, Roy T. Fielding <fielding@gbiv.com> wrote:
> On Feb 17, 2011, at 4:43 AM, Bernd Fondermann wrote:
>> We have the very unfortunate situation here at Hadoop where Apache
>> Hadoop is not the primary and foremost place of Hadoop development.
>> Instead, code is developed internally at Yahoo and then contributed in
>> (smaller or larger) chunks to Hadoop.
>> This is open source development upside down.
>
> It is not open development. =A0The development community can do better,
> but it has to make up for past mistakes first.
>
>> It is not ok for people to diff ASF svn against their internal code
>> and provide the diff as a patch without reviewing IP first for every
>> line of code changed.
>
> That is simply untrue. =A0If the code came from one company's employees
> and they all signed an employment agreement with their employer and
> the employer approves of the contribution and the committer knows that
> when they commit (and logs the authors of the patch when committed),
> then all necessary IP clearance has been done.

I don't know how many Y-employees are working on H internally. Only
the contributors can sort that out.

> Committers are responsible
> for ensuring that they have permission to commit under their own CLA.

I just wanted to point that out, not stop someone from contributing.
Of course, your words are much more precise than mine.

>> For larger chunks I'd suggest to even go via the Incubator IP clearance =
process.
>
> Nonsense (with director hat on).
>
>> Only then will we force committers to primarily work here in the open
>> and return to what I'd consider a healthy project.
>
> No, you'll force people to work on the open by actually collaborating
> with them as they work and veto a patch for any technical faults it
> may contain. =A0Pestering them about your personal view of the Apache Way
> of development is not a contribution.

Getting personal is not a valuable contribution either.

  Bernd

From general-return-3082-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 18 10:43:21 2011
Return-Path: <general-return-3082-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27307 invoked from network); 18 Feb 2011 10:43:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2011 10:43:21 -0000
Received: (qmail 4620 invoked by uid 500); 18 Feb 2011 10:43:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4124 invoked by uid 500); 18 Feb 2011 10:43:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4116 invoked by uid 99); 18 Feb 2011 10:43:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 10:43:15 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of binhara@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 10:43:10 +0000
Received: by wye20 with SMTP id 20so3532131wye.35
        for <general@hadoop.apache.org>; Fri, 18 Feb 2011 02:42:49 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:cc:content-type;
        bh=81zbayNj19YL8aqox2zpW0KFx6RFNLTNbYh7TUtHVDI=;
        b=AM4tyvkcOq8m79RULyfoL5apemwzV1d/ALudlp5gRTNA7HbfwGfBwoRSJPggPh78Jn
         e61/R3sUZOzY2mqgvfp2QqHscwH3cvcSFaZRevRW9mgRGZg8JX+lxQjScXpj2BWmhco4
         +EykdatSVrhXnIQG1u0T4Dcoz4dOArszgsRfs=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=wlAfelTlt6YTLCzN6yy7k2hmU8/14ZCU9ARRCF/VckBG3JfORlZK6pRs+tz9qPKhI/
         q1QrZ2Pm7X8tHRSKEriq0eq/Z1i8APQpc9V/U2r7kHFKRgThhrWfeQ/S7vLfJEaBo+wr
         MtiXhm0quJv56yeTz9c/oA8SvA0OBaSOWk4m8=
MIME-Version: 1.0
Received: by 10.216.165.85 with SMTP id d63mr465685wel.12.1298025769258; Fri,
 18 Feb 2011 02:42:49 -0800 (PST)
Received: by 10.216.17.146 with HTTP; Fri, 18 Feb 2011 02:42:49 -0800 (PST)
In-Reply-To: <AANLkTimUPJ__9WODxthGrWEc5C6JJBcVeYOUT=mmum9W@mail.gmail.com>
References: <AANLkTimw9+b=gtfSEfh=capMb83rpAeBU6EbpDbqqUYK@mail.gmail.com>
	<AANLkTimUPJ__9WODxthGrWEc5C6JJBcVeYOUT=mmum9W@mail.gmail.com>
Date: Fri, 18 Feb 2011 08:42:49 -0200
Message-ID: <AANLkTi=DaSGz7637_7mw32EP-bMsyXXYo1Fs+5waSvQR@mail.gmail.com>
Subject: Re: simple map reduce
From: Alessandro Binhara <binhara@gmail.com>
To: general@hadoop.apache.org
Cc: Harsh J <qwertymaniac@gmail.com>
Content-Type: multipart/alternative; boundary=0016e6469722e2ce6c049c8c2c79

--0016e6469722e2ce6c049c8c2c79
Content-Type: text/plain; charset=ISO-8859-1

I need extend ArrayWritable ??? like this :

public class IntArrayWritable extends ArrayWritable { public
IntArrayWritable() { super(IntWritable.class); } }

On Fri, Feb 18, 2011 at 1:15 AM, Harsh J <qwertymaniac@gmail.com> wrote:

> Hello,
>
> On Fri, Feb 18, 2011 at 12:17 AM, Alessandro Binhara <binhara@gmail.com>
> wrote:
> > I had a another structure .. of input file
> > key, value1, value2, value3
> > 993, 3,2,3
> > 993, 2,1,1
> > 333, 2,2,1
> >
> > How i can sendo to reduce a list of values, ? To process a list numers
> and
> > not only one number?
> > values must be added separately..
>
> An IntWritable can't obviously hold a sequence of integers. So you
> need a different data structure to hold a sequence of integers. From
> the available bank of Writables in Hadoop, you can perhaps use an
> ArrayWritable initialized for IntWritable members.
>
> --
> Harsh J
> www.harshj.com
>

--0016e6469722e2ce6c049c8c2c79--

From general-return-3083-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 18 11:29:37 2011
Return-Path: <general-return-3083-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47701 invoked from network); 18 Feb 2011 11:29:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2011 11:29:36 -0000
Received: (qmail 57312 invoked by uid 500); 18 Feb 2011 11:29:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56905 invoked by uid 500); 18 Feb 2011 11:29:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56897 invoked by uid 99); 18 Feb 2011 11:29:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 11:29:31 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 11:29:23 +0000
Received: from SP2-EX07CAS04.ds.corp.yahoo.com (sp2-ex07cas04.corp.sp2.yahoo.com [98.137.59.5])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p1IBSAlA029083;
	Fri, 18 Feb 2011 03:28:10 -0800 (PST)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by
 SP2-EX07CAS04.ds.corp.yahoo.com ([98.137.59.5]) with mapi; Fri, 18 Feb 2011
 03:28:10 -0800
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Date: Fri, 18 Feb 2011 03:28:06 -0800
Subject: Re: [VOTE] Abandon hdfsproxy HDFS contrib
Thread-Topic: [VOTE] Abandon hdfsproxy HDFS contrib
Thread-Index: AcvPXuvCx9X6ZAWyRGaLq9e8FwlbKg==
Message-ID: <89BF364A-2A1F-49B7-9A91-F25C6E8A83F0@yahoo-inc.com>
References: <D97CF74A-147E-4573-B188-2F9C48CFC482@mac.com>
 <AANLkTi=F2eBsACBfjYSVava+VwFcsRpqZdRRN+OHXNRJ@mail.gmail.com>
 <823262F0-EE37-4A0A-8F8C-D9422541D67B@mac.com>
 <0CBAFDBB-AC2A-495D-8936-4DF4D6731B69@Holsman.NET>
 <B61B4A67-CB97-4638-AEF8-C7FAE09001BC@yahoo-inc.com>
 <AANLkTi=POJgv0Pc4XBQ29ahAhU-ZoLcf1hK_CvzgRPW_@mail.gmail.com>
 <71308072-C3D4-4A90-A4E7-588C91C5D088@linkedin.com>
In-Reply-To: <71308072-C3D4-4A90-A4E7-588C91C5D088@linkedin.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Well put Allen.=20

My thinking is to move this API into HDFS removing the need for HDFS proxy.=
 Then you can use the http proxy of your choice if you have needs for a pro=
xy (we do this for external bandwidth management, security and ip space pre=
servation reasons).=20

I'm +1 for removing it.=20

We can always revive the project in extras if the API changes hit a wall.=20

---
E14 - via iPhone

On Feb 17, 2011, at 1:26 PM, "Allen Wittenauer" <awittenauer@linkedin.com> =
wrote:

>=20
> On Feb 17, 2011, at 1:21 PM, Konstantin Shvachko wrote:
>=20
>> hdfsproxy is a wrapper around hftpFileSystem (in its current state).
>> So you can always replace hdfsproxy with hftpFileSystem.
>> Also it uses pure FileSystem api, so it can successfully be maintained
>> outside of hdfs.
>>=20
>> Therefore I am +1 removing it from hdfs/contrib.
>>=20
>> What is the use case for hdfsproxy anyways?
>=20
>    A stable, secure http get interface for files.  (No, the normal web ui=
 is not good enough.  Think firewalls.).
>=20
>=20
>=20

From general-return-3084-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 18 12:25:16 2011
Return-Path: <general-return-3084-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69666 invoked from network); 18 Feb 2011 12:25:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2011 12:25:16 -0000
Received: (qmail 8133 invoked by uid 500); 18 Feb 2011 12:25:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7596 invoked by uid 500); 18 Feb 2011 12:25:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7582 invoked by uid 99); 18 Feb 2011 12:25:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 12:25:09 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of magawake@gmail.com designates 209.85.210.176 as permitted sender)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 12:25:03 +0000
Received: by iyb26 with SMTP id 26so3595008iyb.35
        for <general@hadoop.apache.org>; Fri, 18 Feb 2011 04:24:42 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:cc:content-type;
        bh=fQG8Fe/NTt0TUMN7kw3d8bjINPOC1mD/EH/fc63hJ98=;
        b=eLmZRtpOJ9oz7CMvFLQS247MLl7zry96v8NWeaOtpxnOTr/u/tcQYOpDHbpAaWpYdT
         hdM1HVcbz1KYmbZdeCYitOW2xMVIRw8wmn0WUROenEUS+D89g/B86QfFaIo0xfknQn3u
         yqvKwsTkBE3a9gHvmmAqyhelYrBV0GgarCBsU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=ljJdcaRigrqmRXkzllt0FXaZArXTJ6b2jIU3vUQcXH6rdGWsSuwHLXZRzkW0RVJeIy
         Azf/AA/jcH+jWFN7tT8e7okTzKZzkfGCxyCRlH0HE0coLBl83GifY8uqlElC9QMAjXGZ
         sh0CbfUcxUAb7B3+u9lnSH2PQfTGPt+CeEMn4=
MIME-Version: 1.0
Received: by 10.42.225.68 with SMTP id ir4mr787730icb.291.1298031882490; Fri,
 18 Feb 2011 04:24:42 -0800 (PST)
Received: by 10.42.170.197 with HTTP; Fri, 18 Feb 2011 04:24:42 -0800 (PST)
In-Reply-To: <AANLkTikxP0Y3y2RR-5PzAPn5pjpuQY8wZXptXmp1sML3@mail.gmail.com>
References: <AANLkTi=cyTOYxSbvn_YqRTuHPW0b74=EVWhky+oqxSGM@mail.gmail.com>
	<AANLkTikJMzXhYA1wJwxYm=9Zc=PdAtz6-5mOoUQEF5Dh@mail.gmail.com>
	<AANLkTi=6jLDBd7g2TF5cD_7b_v63n+m_eG+z9t31FW-g@mail.gmail.com>
	<AANLkTikxP0Y3y2RR-5PzAPn5pjpuQY8wZXptXmp1sML3@mail.gmail.com>
Date: Fri, 18 Feb 2011 07:24:42 -0500
Message-ID: <AANLkTinma0Dpfv3Z6LjWpTckMVq57hrZgWRDjZsu71AW@mail.gmail.com>
Subject: Re: dfs.datanode.du.pct question
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Cc: "Aaron T. Myers" <atm@cloudera.com>
Content-Type: text/plain; charset=UTF-8
X-Virus-Checked: Checked by ClamAV on apache.org

Aaron,

Thanks for following up on this issue for us.

Has dfs.datanode.du.pct been depreciated?

On Mon, Jan 3, 2011 at 6:06 PM, Aaron T. Myers <atm@cloudera.com> wrote:
> Filed https://issues.apache.org/jira/browse/HDFS-1564
>
> --
> Aaron T. Myers
> Software Engineer, Cloudera
>
>
>
> On Sun, Jan 2, 2011 at 3:19 PM, Mag Gam <magawake@gmail.com> wrote:
>
>> Thanks for the response.
>>
>> It would be nice to have a feature like this. It will make our data
>> nodes much more flexible.
>>
>>
>>
>>
>>
>>
>>
>> On Fri, Dec 31, 2010 at 5:56 PM, Harsh J <qwertymaniac@gmail.com> wrote:
>> > Hi,
>> >
>> > Isn't this property a legacy one? 0.18 is the last release I see with
>> > this property. I've always used dfs.datanode.du.reserved instead,
>> > reserving constant space across all dfs.data.dir volumes.
>> >
>> > On Fri, Dec 31, 2010 at 9:32 PM, Mag Gam <magawake@gmail.com> wrote:
>> >> On some of my data nodes I have 2 filesystems. /fs1 and /fs2. Is it
>> >> possible to set the du pct so, for fs1 its 80% usage and /fs2 its 20%
>> >> usage?
>> >
>> > According to the release's code: No.
>> >
>> > (Unless two DataNodes are run off the same machine, which is not
>> recommended.)
>> >
>> > --
>> > Harsh J
>> > www.harshj.com
>> >
>>
>

From general-return-3085-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 18 12:47:08 2011
Return-Path: <general-return-3085-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82579 invoked from network); 18 Feb 2011 12:47:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2011 12:47:08 -0000
Received: (qmail 33598 invoked by uid 500); 18 Feb 2011 12:47:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32841 invoked by uid 500); 18 Feb 2011 12:47:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32833 invoked by uid 99); 18 Feb 2011 12:47:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 12:47:02 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 12:46:54 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p1ICkCUS074757
	for <general@hadoop.apache.org>; Fri, 18 Feb 2011 04:46:12 -0800 (PST)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by
 SP2-EX07CAS01.ds.corp.yahoo.com ([98.137.59.37]) with mapi; Fri, 18 Feb 2011
 04:46:12 -0800
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 18 Feb 2011 04:46:06 -0800
Subject: Re: [VOTE] Abandon hdfsproxy HDFS contrib
Thread-Topic: [VOTE] Abandon hdfsproxy HDFS contrib
Thread-Index: AcvPadI6Be8tLPCgTTeEGkYt3V5Hew==
Message-ID: <FB7E6664-F292-480F-8C4F-2B3CCAB9FFB7@yahoo-inc.com>
References: <D97CF74A-147E-4573-B188-2F9C48CFC482@mac.com>
 <AANLkTi=F2eBsACBfjYSVava+VwFcsRpqZdRRN+OHXNRJ@mail.gmail.com>
 <823262F0-EE37-4A0A-8F8C-D9422541D67B@mac.com>
 <0CBAFDBB-AC2A-495D-8936-4DF4D6731B69@Holsman.NET>
 <AANLkTimUhDbTnwn6Hk2oYXFJMts+A=FnL-q-veXqX1m0@mail.gmail.com>
 <F2EC1B1B-A2BB-4FC8-9842-3904A25338EE@gbiv.com>
 <AANLkTikVcGcKHnh9EgUDQr2JGLPvkBvBRgn4my3uJfOc@mail.gmail.com>
 <864B55E7-8A46-458F-8B16-8252A2011835@Holsman.NET>
 <AANLkTikhe6YjUBGY8LMc3OG=b7VXWz7WhyNKGkJRu_7K@mail.gmail.com>
In-Reply-To: <AANLkTikhe6YjUBGY8LMc3OG=b7VXWz7WhyNKGkJRu_7K@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Bernd,

Apache Hadoop is about scale. Most clusters will always be small, but Hadoo=
p is going mainstream precisely because it scales to huge data and cluster =
sizes.=20

There are lots of systems that work well on 10 node clusters. People select=
   Hadoop because they are confident that as their business / problem grows=
, Hadoop can grow with it.=20

---
E14 - via iPhone

On Feb 17, 2011, at 7:25 AM, "Bernd Fondermann" <bernd.fondermann@googlemai=
l.com> wrote:

> On Thu, Feb 17, 2011 at 14:58, Ian Holsman <hadoop@holsman.net> wrote:
>> Hi Bernd.
>>=20
>> On Feb 17, 2011, at 7:43 AM, Bernd Fondermann wrote:
>>>=20
>>> We have the very unfortunate situation here at Hadoop where Apache
>>> Hadoop is not the primary and foremost place of Hadoop development.
>>> Instead, code is developed internally at Yahoo and then contributed in
>>> (smaller or larger) chunks to Hadoop.
>>=20
>> This has been the situation in the past,
>> but as you can see in the last month, this has changed.
>>=20
>> Yahoo! has publicly committed to move their development into the main co=
de base, and you can see they have started doing this with the 20.100 branc=
h,
>> and their recent commits to trunk.
>> Combine this with Nige taking on the 0.22 release branch, (and sheperdin=
g it into a stable release) and I think we have are addressing your concern=
s.
>>=20
>> They have also started bringing the discussions back on the list, see th=
e recent discussion about Jobtracker-nextgen Arun has re-started in MAPREDU=
CE-279.
>>=20
>> I'm not saying it's perfect, but I think the major players understand th=
ere is an issue, and they are *ALL* moving in the right direction.
>=20
> I enthusiastically would like to see your optimism be verified.
> Maybe I'm misreading the statements issued publicly, but I don't think
> that this is fully understood. I agree though that it's a move into
> the right direction.
>=20
>>> This is open source development upside down.
>>> It is not ok for people to diff ASF svn against their internal code
>>> and provide the diff as a patch without reviewing IP first for every
>>> line of code changed.
>>> For larger chunks I'd suggest to even go via the Incubator IP clearance=
 process.
>>> Only then will we force committers to primarily work here in the open
>>> and return to what I'd consider a healthy project.
>>>=20
>>> To be honest: Hadoop is in the process of falling apart.
>>> Contrib Code gets moved out of Apache instead of being maintained here.
>>> Discussions are seldom consense-driven.
>>> Release branches stagnate.
>>=20
>> True. releases do take a long time. This is mainly due to it being extre=
mely hard to test and verify that a release is stable.
>> It's not enough to just run the thing on 4 machines, you need at least 5=
0 to test some of the major problems. This requires some serious $ for some=
one to verify.
>=20
> It has been proposed on the list before, IIRC. Don't know how to get
> there, but the project seriously needs access to a cluster of this
> size.
>=20
>>> Downstream projects like HBase don't get proper support.
>>> Production setups are made from 3rd party distributions.
>>> Development is not happening here, but elsewhere behind corporate doors=
.
>>> Discussion about future developments are started on corporate blogs (
>>> http://developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextgen=
/
>>> ) instead of on the proper mailing list.
>>> Hurdles for committing are way too high.
>>> On the bright side, new committers and PMC members are added, this is
>>> an improvement.
>>>=20
>>> I'd suggest to move away from relying on large code dumps from
>>> corporations, and move back to the ASF-proven "individual committer
>>> commits on trunk"-model where more committers can get involved.
>>> If that means not to support high end cluster sizes for some months,
>>> well, so be it.
>>=20
>>> Average committers cannot run - e.g. test - on high
>>> end cluster sizes. If that would mean they cannot participate, then
>>> the open source project better concentrate on small and medium sized
>>> cluster instead.
>>=20
>>=20
>> Well.. that's one approach.. but there are several companies out there w=
ho rely on apache's hadoop to power their large clusters, so I'd hate to se=
e hadoop become something that only runs well on
>> 10-nodes.. as I don't think that will help anyone either.
>=20
> But only looking at high-end scale doesn't help either.
>=20
> Lets face the fact that Hadoop is now moving from early adaptors phase
> into a much broader market. I predict that small to medium sized
> clusters will be the majority of Hadoop deployments in a few month
> time. 4000, or even 500 machines is the high-end range. If the open
> source project Hadoop cannot support those users adequately (without
> becoming defunct), the committership might be better off to focus on
> the low-end and medium sized users.
>=20
> I'm not suggesting to turn away from the handfull (?) of high-end
> users. They certainly have most valuable input. But also, *they*
> obviously have the resources in terms of larger clusters and
> developers to deal with their specific setups. Obviously, they don't
> need to rely on the open source project to make releases. In fact,
> they *do* work on their own Hadoop derivatives.
> All the other users, the hundreds of boring small cluster users, don't
> have that choice. They *depend* on the open source releases.
>=20
> Hadoop is an Apache project, to provide HDFS and MR free of charge to
> the general public. Not only to me - nor to only one or two big
> companies either.
> Focus on all the users.
>=20
>  Bernd

From general-return-3086-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 18 13:21:17 2011
Return-Path: <general-return-3086-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98710 invoked from network); 18 Feb 2011 13:21:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2011 13:21:16 -0000
Received: (qmail 68325 invoked by uid 500); 18 Feb 2011 13:21:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67859 invoked by uid 500); 18 Feb 2011 13:21:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67851 invoked by uid 99); 18 Feb 2011 13:21:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 13:21:11 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bernd.fondermann@googlemail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 13:21:07 +0000
Received: by iwn2 with SMTP id 2so3741295iwn.35
        for <general@hadoop.apache.org>; Fri, 18 Feb 2011 05:20:46 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type:content-transfer-encoding;
        bh=W936eqJIUYN1ShV49bb+8RSaCYHFkUWvtstY04RJfyk=;
        b=KC1+PACb81MgJrzOmkxHNd4JZ6/aLOICkRX/Qo467rkIe8nnkUs/irgfz9jIPUti7s
         TvhEmlFyab1xr5/0+7YF4lJxnWRW32Rv/UtxiHSE2uZf6ccykfXEGScknX8Q/2b67NQr
         rlk9KZbdq0jFt1C6iWvMcwUFOJJrvQldvAfNc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=U4EuzTkMg2wOtRKL+frAdH7hr8br2LztE6IMBHbxFBPN65Qs8EfwNwe92NGNgm+P4X
         2lWdxfwseA98ZT0iqftYUJUhYDTUbN0oVYy7Ud8QFrpUrkOUQT0T0xKi1mLO/8BOh4xU
         CZHp+Zz9S3U1YSWOPR19nHopP8aDI8b+2Pa3g=
MIME-Version: 1.0
Received: by 10.42.117.130 with SMTP id t2mr872697icq.148.1298035246317; Fri,
 18 Feb 2011 05:20:46 -0800 (PST)
Received: by 10.42.8.149 with HTTP; Fri, 18 Feb 2011 05:20:46 -0800 (PST)
In-Reply-To: <FB7E6664-F292-480F-8C4F-2B3CCAB9FFB7@yahoo-inc.com>
References: <D97CF74A-147E-4573-B188-2F9C48CFC482@mac.com>
	<AANLkTi=F2eBsACBfjYSVava+VwFcsRpqZdRRN+OHXNRJ@mail.gmail.com>
	<823262F0-EE37-4A0A-8F8C-D9422541D67B@mac.com>
	<0CBAFDBB-AC2A-495D-8936-4DF4D6731B69@Holsman.NET>
	<AANLkTimUhDbTnwn6Hk2oYXFJMts+A=FnL-q-veXqX1m0@mail.gmail.com>
	<F2EC1B1B-A2BB-4FC8-9842-3904A25338EE@gbiv.com>
	<AANLkTikVcGcKHnh9EgUDQr2JGLPvkBvBRgn4my3uJfOc@mail.gmail.com>
	<864B55E7-8A46-458F-8B16-8252A2011835@Holsman.NET>
	<AANLkTikhe6YjUBGY8LMc3OG=b7VXWz7WhyNKGkJRu_7K@mail.gmail.com>
	<FB7E6664-F292-480F-8C4F-2B3CCAB9FFB7@yahoo-inc.com>
Date: Fri, 18 Feb 2011 14:20:46 +0100
Message-ID: <AANLkTik_gZ-_6=2Om=9JSDMSq4T-jNycOUGSNCrD2Gd6@mail.gmail.com>
Subject: Re: [VOTE] Abandon hdfsproxy HDFS contrib
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Eric,

On Fri, Feb 18, 2011 at 13:46, Eric Baldeschwieler <eric14@yahoo-inc.com> w=
rote:
> Hi Bernd,
>
> Apache Hadoop is about scale. Most clusters will always be small, but Had=
oop is going mainstream precisely because it scales to huge data and cluste=
r sizes.
>
> There are lots of systems that work well on 10 node clusters. People sele=
ct =A0 Hadoop because they are confident that as their business / problem g=
rows, Hadoop can grow with it.

Please note that I did not say that Hadoop should not scale.
I know that winning Sorting contests is a great achievement and a huge
selling point.

I'm thinking along the lines of: How much scalability would the
majority of users be willing to trade for
a. more active committers (guess: 0%)
b. more regular releases
c. more non-scalability features (hot standby NN, security, younameit)

I for myself as a low-scale user *would* trade a few percent for b. and c.

Thanks,

  Bernd

> ---
> E14 - via iPhone
>
> On Feb 17, 2011, at 7:25 AM, "Bernd Fondermann" <bernd.fondermann@googlem=
ail.com> wrote:
>
>> On Thu, Feb 17, 2011 at 14:58, Ian Holsman <hadoop@holsman.net> wrote:
>>> Hi Bernd.
>>>
>>> On Feb 17, 2011, at 7:43 AM, Bernd Fondermann wrote:
>>>>
>>>> We have the very unfortunate situation here at Hadoop where Apache
>>>> Hadoop is not the primary and foremost place of Hadoop development.
>>>> Instead, code is developed internally at Yahoo and then contributed in
>>>> (smaller or larger) chunks to Hadoop.
>>>
>>> This has been the situation in the past,
>>> but as you can see in the last month, this has changed.
>>>
>>> Yahoo! has publicly committed to move their development into the main c=
ode base, and you can see they have started doing this with the 20.100 bran=
ch,
>>> and their recent commits to trunk.
>>> Combine this with Nige taking on the 0.22 release branch, (and sheperdi=
ng it into a stable release) and I think we have are addressing your concer=
ns.
>>>
>>> They have also started bringing the discussions back on the list, see t=
he recent discussion about Jobtracker-nextgen Arun has re-started in MAPRED=
UCE-279.
>>>
>>> I'm not saying it's perfect, but I think the major players understand t=
here is an issue, and they are *ALL* moving in the right direction.
>>
>> I enthusiastically would like to see your optimism be verified.
>> Maybe I'm misreading the statements issued publicly, but I don't think
>> that this is fully understood. I agree though that it's a move into
>> the right direction.
>>
>>>> This is open source development upside down.
>>>> It is not ok for people to diff ASF svn against their internal code
>>>> and provide the diff as a patch without reviewing IP first for every
>>>> line of code changed.
>>>> For larger chunks I'd suggest to even go via the Incubator IP clearanc=
e process.
>>>> Only then will we force committers to primarily work here in the open
>>>> and return to what I'd consider a healthy project.
>>>>
>>>> To be honest: Hadoop is in the process of falling apart.
>>>> Contrib Code gets moved out of Apache instead of being maintained here=
.
>>>> Discussions are seldom consense-driven.
>>>> Release branches stagnate.
>>>
>>> True. releases do take a long time. This is mainly due to it being extr=
emely hard to test and verify that a release is stable.
>>> It's not enough to just run the thing on 4 machines, you need at least =
50 to test some of the major problems. This requires some serious $ for som=
eone to verify.
>>
>> It has been proposed on the list before, IIRC. Don't know how to get
>> there, but the project seriously needs access to a cluster of this
>> size.
>>
>>>> Downstream projects like HBase don't get proper support.
>>>> Production setups are made from 3rd party distributions.
>>>> Development is not happening here, but elsewhere behind corporate door=
s.
>>>> Discussion about future developments are started on corporate blogs (
>>>> http://developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextge=
n/
>>>> ) instead of on the proper mailing list.
>>>> Hurdles for committing are way too high.
>>>> On the bright side, new committers and PMC members are added, this is
>>>> an improvement.
>>>>
>>>> I'd suggest to move away from relying on large code dumps from
>>>> corporations, and move back to the ASF-proven "individual committer
>>>> commits on trunk"-model where more committers can get involved.
>>>> If that means not to support high end cluster sizes for some months,
>>>> well, so be it.
>>>
>>>> Average committers cannot run - e.g. test - on high
>>>> end cluster sizes. If that would mean they cannot participate, then
>>>> the open source project better concentrate on small and medium sized
>>>> cluster instead.
>>>
>>>
>>> Well.. that's one approach.. but there are several companies out there =
who rely on apache's hadoop to power their large clusters, so I'd hate to s=
ee hadoop become something that only runs well on
>>> 10-nodes.. as I don't think that will help anyone either.
>>
>> But only looking at high-end scale doesn't help either.
>>
>> Lets face the fact that Hadoop is now moving from early adaptors phase
>> into a much broader market. I predict that small to medium sized
>> clusters will be the majority of Hadoop deployments in a few month
>> time. 4000, or even 500 machines is the high-end range. If the open
>> source project Hadoop cannot support those users adequately (without
>> becoming defunct), the committership might be better off to focus on
>> the low-end and medium sized users.
>>
>> I'm not suggesting to turn away from the handfull (?) of high-end
>> users. They certainly have most valuable input. But also, *they*
>> obviously have the resources in terms of larger clusters and
>> developers to deal with their specific setups. Obviously, they don't
>> need to rely on the open source project to make releases. In fact,
>> they *do* work on their own Hadoop derivatives.
>> All the other users, the hundreds of boring small cluster users, don't
>> have that choice. They *depend* on the open source releases.
>>
>> Hadoop is an Apache project, to provide HDFS and MR free of charge to
>> the general public. Not only to me - nor to only one or two big
>> companies either.
>> Focus on all the users.
>>
>> =A0Bernd
>

From general-return-3087-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 18 14:41:07 2011
Return-Path: <general-return-3087-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24393 invoked from network); 18 Feb 2011 14:41:06 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2011 14:41:06 -0000
Received: (qmail 78169 invoked by uid 500); 18 Feb 2011 14:41:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77706 invoked by uid 500); 18 Feb 2011 14:41:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77689 invoked by uid 99); 18 Feb 2011 14:41:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 14:41:01 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [156.148.72.33] (HELO raffaello.crs4.it) (156.148.72.33)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 14:40:53 +0000
Received: from [156.148.71.202] (neuron.crs4.it [156.148.71.202])
	by raffaello.crs4.it (Postfix) with ESMTP id 0259D790177
	for <general@hadoop.apache.org>; Fri, 18 Feb 2011 15:40:33 +0100 (CET)
Message-ID: <4D5E84DC.6050200@crs4.it>
Date: Fri, 18 Feb 2011 15:40:28 +0100
From: Simone Leo <simone.leo@crs4.it>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101216 Lightning/1.0b3pre Thunderbird/3.1.7
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Abandon hod Common contrib
References: <AC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com> <84B97117-5DA4-4A32-912A-8DBFCE3D063E@linkedin.com> <535D05CA-3321-48BA-82C4-C18FC8983C3A@mac.com> <AC425E88-8B0E-4982-84FF-D55C7C2C7E3D@apache.org>
In-Reply-To: <AC425E88-8B0E-4982-84FF-D55C7C2C7E3D@apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

I am the co-author (with Gianluigi Zanetti) of HADOOP-6369 -- add Grid
Engine support to HOD. At CRS4 we've been using (our patched version of)
HOD since 2008 and we still use it in production. We use Hadoop 0.20.2
since it was released one year ago.

Simone

On 02/12/11 06:15, Owen O'Malley wrote:
> 
> On Feb 11, 2011, at 6:17 PM, Nigel Daley wrote:
> 
>>> a) I don't think hod is actually part of any unit tests, so including
>>> it would likely only be a burden on the tarball size.
>>
>> Not true.  HOD has python unit tests and is the reason our builds have
>> dependencies on python.
> 
> But Allen's point is that I don't recall ever seeing HOD test failures
> causing the build to fail.
> 
>>> b) The edu community uses this quite extensively, evidenced by the
>>> topic coming up on the mailing lists at least once every two months
>>> or so and has for years.  Can't say that about the other contrib
>>> modules other than the schedulers and streaming.
>>
>> Then they are using old version of Hadoop.  AFAICT HOD does not work
>> with 0.20 or beyond.
> 
> Out of curiosity, what goes wrong? Clearly nothing major has changed in
> starting up a mapreduce cluster in a very long time.
> 
>>> c) The community that does use it has even submitted a patch that
>>> we've ignored.
>>
>> Which means the committers of this project gave up on it long ago.
> 
> There are also some patches on core Hadoop that have been sitting for a
> long time, so I don't think that is a valid inference.
> 
> I would love to hear some of the people who are using HOD speak up and
> give us their feedback.
> 
> -- Owen


-- 
Simone Leo
Data Fusion - Distributed Computing
CRS4
POLARIS - Building #1
Piscina Manna
I-09010 Pula (CA) - Italy
e-mail: simone.leo@crs4.it
http://www.crs4.it

From general-return-3088-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 18 16:11:36 2011
Return-Path: <general-return-3088-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70158 invoked from network); 18 Feb 2011 16:11:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2011 16:11:35 -0000
Received: (qmail 26312 invoked by uid 500); 18 Feb 2011 16:11:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24788 invoked by uid 500); 18 Feb 2011 16:11:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24780 invoked by uid 99); 18 Feb 2011 16:11:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 16:11:30 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 16:11:25 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=tE1zIGWLusT49xEaPADyUvyaEcxbe7OzED6+Tjl2haXcKIXbfsg7H+gl
   LwTWy+5IsoiHgmPjPNjU2HajWWurwiQW00HH/ipJahC44VAiwVI0Wzghy
   RiWmNoW2xMuq1OW;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1298045482; x=1329581482;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20[VOTE]=20Abandon=20hdfsproxy=20HDFS=20c
   ontrib|Date:=20Fri,=2018=20Feb=202011=2016:11:38=20+0000
   |Message-ID:=20<BF4E6121-8323-4B5F-A10E-2434B2FC0CE2@link
   edin.com>|To:=20"<general@hadoop.apache.org>"=20<general@
   hadoop.apache.org>|MIME-Version:=201.0
   |Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<A1070D4F24AA834BAA658CBD6D868EEC@linkedin
   .com>|In-Reply-To:=20<AANLkTik1LFE4z0PGoOEudbFJ93e+LCrwTG
   LPmHybo-AL@mail.gmail.com>|References:=20<D97CF74A-147E-4
   573-B188-2F9C48CFC482@mac.com>=0D=0A=20<AANLkTi=3DF2eBsAC
   BfjYSVava+VwFcsRpqZdRRN+OHXNRJ@mail.gmail.com>=0D=0A=20<8
   23262F0-EE37-4A0A-8F8C-D9422541D67B@mac.com>=0D=0A=20<0CB
   AFDBB-AC2A-495D-8936-4DF4D6731B69@Holsman.NET>=0D=0A=20<A
   ANLkTimUhDbTnwn6Hk2oYXFJMts+A=3DFnL-q-veXqX1m0@mail.gmail
   .com>=0D=0A=20<F2EC1B1B-A2BB-4FC8-9842-3904A25338EE@gbiv.
   com>=0D=0A=20<AANLkTikVcGcKHnh9EgUDQr2JGLPvkBvBRgn4my3uJf
   Oc@mail.gmail.com>=0D=0A=20<1FEA58C1-AA9A-4A6F-8873-C9950
   57CDD19@gbiv.com>=0D=0A=20<AANLkTik1LFE4z0PGoOEudbFJ93e+L
   CrwTGLPmHybo-AL@mail.gmail.com>;
  bh=Y3LM5rprKKoMY5fH4kbG7nuv68pUybNRMdRMC9h+TSI=;
  b=ET6zWQHVbBlmFUCp0/tH61izMzBbkLrjG8n5zoLjcG7Epw7D+a0aYRUQ
   1YJ/B7Mi3zV8tVmXC1QPQJVwLWYR7k5St1V4/sq+E4y3RXUddYzX2OSkC
   FpzsSoe+vsvwVWS;
X-IronPort-AV: E=Sophos;i="4.62,187,1297065600"; 
   d="scan'208";a="20310592"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi id 14.01.0251.000; Fri,
 18 Feb 2011 08:11:39 -0800
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: [VOTE] Abandon hdfsproxy HDFS contrib
Thread-Topic: [VOTE] Abandon hdfsproxy HDFS contrib
Thread-Index: AQHLyZ27ICuajETxE02yFg9O1wZLWpP8Sq4AgAANWQCAAAYpgIAAQawAgAFtyICACCXggIAAZJQAgAEDVgCAAGQLAA==
Date: Fri, 18 Feb 2011 16:11:38 +0000
Message-ID: <BF4E6121-8323-4B5F-A10E-2434B2FC0CE2@linkedin.com>
References: <D97CF74A-147E-4573-B188-2F9C48CFC482@mac.com>
 <AANLkTi=F2eBsACBfjYSVava+VwFcsRpqZdRRN+OHXNRJ@mail.gmail.com>
 <823262F0-EE37-4A0A-8F8C-D9422541D67B@mac.com>
 <0CBAFDBB-AC2A-495D-8936-4DF4D6731B69@Holsman.NET>
 <AANLkTimUhDbTnwn6Hk2oYXFJMts+A=FnL-q-veXqX1m0@mail.gmail.com>
 <F2EC1B1B-A2BB-4FC8-9842-3904A25338EE@gbiv.com>
 <AANLkTikVcGcKHnh9EgUDQr2JGLPvkBvBRgn4my3uJfOc@mail.gmail.com>
 <1FEA58C1-AA9A-4A6F-8873-C995057CDD19@gbiv.com>
 <AANLkTik1LFE4z0PGoOEudbFJ93e+LCrwTGLPmHybo-AL@mail.gmail.com>
In-Reply-To: <AANLkTik1LFE4z0PGoOEudbFJ93e+LCrwTGLPmHybo-AL@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <A1070D4F24AA834BAA658CBD6D868EEC@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


On Feb 18, 2011, at 2:11 AM, Bernd Fondermann wrote:
> I don't know how many Y-employees are working on H internally. Only
> the contributors can sort that out.

	Did Carol Bartz run over your puppy or something?  You don't appear to rea=
lize that pretty much all the major companies that house the major committe=
rs are doing essentially the same thing:  work on a major feature patch wit=
h their internal teams then offer up the patch mostly complete, having deal=
t with all of the minor nits on the way due to having field tested it for r=
eal.   The patches still get discussed before getting committed.  In some c=
ases, this has resulted in either the patch getting rejected/not committed =
or reworked.

	I sometimes wonder if the people who troll the team realize that Hadoop is=
 essentially a distributed, networked operating system.  That requires a hi=
gher level of quality control than a simple framework.=20


From general-return-3089-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 18 16:26:37 2011
Return-Path: <general-return-3089-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77447 invoked from network); 18 Feb 2011 16:26:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2011 16:26:37 -0000
Received: (qmail 54645 invoked by uid 500); 18 Feb 2011 16:26:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54085 invoked by uid 500); 18 Feb 2011 16:26:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54077 invoked by uid 99); 18 Feb 2011 16:26:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 16:26:31 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 16:26:23 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 30CCAB7D7E
	for <general@hadoop.apache.org>; Fri, 18 Feb 2011 16:26:01 +0000 (GMT)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id jIRSyn0SL+8y for <general@hadoop.apache.org>;
	Fri, 18 Feb 2011 16:25:50 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 32F9CB7DA6
	for <general@hadoop.apache.org>; Fri, 18 Feb 2011 16:24:38 +0000 (GMT)
MailScanner-NULL-Check: 1298651065.8959@BqKvN1FIx0ZTkKF1hOgh7Q
Received: from [16.24.210.8] ([16.24.210.8])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p1IGOPaa022745
	for <general@hadoop.apache.org>; Fri, 18 Feb 2011 16:24:25 GMT
Message-ID: <4D5E9D33.5040003@apache.org>
Date: Fri, 18 Feb 2011 16:24:19 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-GB; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Hadoop testing project
References: <C982AAB3.FACF%eyang@yahoo-inc.com>
In-Reply-To: <C982AAB3.FACF%eyang@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p1IGOPaa022745
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 17/02/2011 18:36, Eric Yang wrote:

> I think the bigger concern is that Hadoop ecosystem does not have a standard
> method in linking dependencies.  Hbase depends on Zookeeper, and Pig depends
> on Hadoop and Hbase.  Then pig decided to put hadoop-core jar in it's own
> jar file.  Chukwa depends on pig + hbase + hadoop and zookeeper.  The
> version incompatibility is probably what driving people nuts.  Hence, there
> is a new proposal on how to integrate among hadoop ecosystem.  I urge
> project owners to review the proposal and provide feedbacks.
>
> The proposal is located at:
>
> https://issues.apache.org/jira/secure/attachment/12470823/deployment.pdf
>
> The related jiras are:
>
> https://issues.apache.org/jira/browse/HADOOP-6255
> https://issues.apache.org/jira/browse/PIG-1857
>
> There are plans to file more jiras for related projects.

I think a focus on RPMs/debs and then say "local VM for testing", 
because that gets people used to deploying on Linux from day one, rather 
than developing on windows and being surprised that things work 
differently in production. You still have the problem of debugging 
remotely to a VM in the VLAN, but that's tractable.

> The integration
> would also be a lot easier if all related projects are using maven for
> dependency management.

Do you mean Maven the tool or the maven repo format?

From general-return-3090-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 18 16:40:37 2011
Return-Path: <general-return-3090-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84159 invoked from network); 18 Feb 2011 16:40:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2011 16:40:36 -0000
Received: (qmail 85116 invoked by uid 500); 18 Feb 2011 16:40:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84764 invoked by uid 500); 18 Feb 2011 16:40:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84752 invoked by uid 99); 18 Feb 2011 16:40:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 16:40:32 +0000
X-ASF-Spam-Status: No, hits=3.5 required=5.0
	tests=TO_NO_BRKTS_DIRECT
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 18 Feb 2011 16:40:31 +0000
Received: (qmail 84056 invoked by uid 99); 18 Feb 2011 16:40:10 -0000
Received: from localhost.apache.org (HELO [192.168.168.144]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 16:40:10 +0000
Message-ID: <4D5EA0E9.5070103@apache.org>
Date: Fri, 18 Feb 2011 08:40:09 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: pdf design documents
X-Enigmail-Version: 1.1.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Owen has recently submitted several PDF design documents, e.g.:

https://issues.apache.org/jira/browse/HADOOP-6255
https://issues.apache.org/jira/browse/HADOOP-7150

I feel PDF discourages collaboration.  Either plain text or HTML would
permit folks to more easily provide feedback, either as patches or as
alternate drafts.  Some projects use their wikis to collaboratively
evolve designs, so everyone can see the document evolve from the start.
 Some projects use the dev mailing list directly for design.  Any of
these would be preferable to me to PDF, which only the original
submitter can easily edit.

What do others think?

Thanks,

Doug

From general-return-3091-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 18 16:59:57 2011
Return-Path: <general-return-3091-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91209 invoked from network); 18 Feb 2011 16:59:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2011 16:59:57 -0000
Received: (qmail 21877 invoked by uid 500); 18 Feb 2011 16:59:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20937 invoked by uid 500); 18 Feb 2011 16:59:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 20921 invoked by uid 99); 18 Feb 2011 16:59:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 16:59:50 +0000
X-ASF-Spam-Status: No, hits=2.9 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.59.227] (HELO qmta12.westchester.pa.mail.comcast.net) (76.96.59.227)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 16:59:42 +0000
Received: from omta07.westchester.pa.mail.comcast.net ([76.96.62.59])
	by qmta12.westchester.pa.mail.comcast.net with comcast
	id 9UzB1g0051GhbT85CUzNdl; Fri, 18 Feb 2011 16:59:22 +0000
Received: from [10.72.184.42] ([209.131.62.113])
	by omta07.westchester.pa.mail.comcast.net with comcast
	id 9UzA1g02E2SbwD53TUzDD7; Fri, 18 Feb 2011 16:59:20 +0000
Message-Id: <778FC6F7-8658-4E01-8710-2F384F22452F@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: mapreduce-user@hadoop.apache.org
In-Reply-To: <AANLkTi=DaSGz7637_7mw32EP-bMsyXXYo1Fs+5waSvQR@mail.gmail.com>
Content-Type: multipart/alternative; boundary=Apple-Mail-60--233915219
Subject: Re: simple map reduce
Mime-Version: 1.0 (Apple Message framework v936)
Date: Fri, 18 Feb 2011 08:59:08 -0800
References: <AANLkTimw9+b=gtfSEfh=capMb83rpAeBU6EbpDbqqUYK@mail.gmail.com> <AANLkTimUPJ__9WODxthGrWEc5C6JJBcVeYOUT=mmum9W@mail.gmail.com> <AANLkTi=DaSGz7637_7mw32EP-bMsyXXYo1Fs+5waSvQR@mail.gmail.com>
X-Mailer: Apple Mail (2.936)

--Apple-Mail-60--233915219
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit

Please keep user questions on mapreduce-user as per http://hadoop.apache.org/mailing_lists.html 
  .

On Feb 18, 2011, at 2:42 AM, Alessandro Binhara wrote:

> I need extend ArrayWritable ??? like this :
>
> public class IntArrayWritable extends ArrayWritable { public
> IntArrayWritable() { super(IntWritable.class); } }

Yes.

-- Owen


--Apple-Mail-60--233915219--

From general-return-3092-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 18 17:13:29 2011
Return-Path: <general-return-3092-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4718 invoked from network); 18 Feb 2011 17:13:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2011 17:13:28 -0000
Received: (qmail 47135 invoked by uid 500); 18 Feb 2011 17:13:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46771 invoked by uid 500); 18 Feb 2011 17:13:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46763 invoked by uid 99); 18 Feb 2011 17:13:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 17:13:23 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.30.32] (HELO qmta03.emeryville.ca.mail.comcast.net) (76.96.30.32)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 17:13:13 +0000
Received: from omta01.emeryville.ca.mail.comcast.net ([76.96.30.11])
	by qmta03.emeryville.ca.mail.comcast.net with comcast
	id 9V771g0070EPchoA3VCrzw; Fri, 18 Feb 2011 17:12:51 +0000
Received: from [10.72.184.42] ([209.131.62.113])
	by omta01.emeryville.ca.mail.comcast.net with comcast
	id 9VCi1g0182SbwD58MVClh2; Fri, 18 Feb 2011 17:12:49 +0000
Message-Id: <B3FD7F41-EFC3-4C3A-BD66-DDA3B21BEADB@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <4D5EA0E9.5070103@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: pdf design documents
Date: Fri, 18 Feb 2011 09:12:41 -0800
References: <4D5EA0E9.5070103@apache.org>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


On Feb 18, 2011, at 8:40 AM, Doug Cutting wrote:

> I feel PDF discourages collaboration.  Either plain text or HTML would
> permit folks to more easily provide feedback, either as patches or as
> alternate drafts.

Ok, I've upload the latex files.

-- Owen

From general-return-3093-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 18 17:48:53 2011
Return-Path: <general-return-3093-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14167 invoked from network); 18 Feb 2011 17:48:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2011 17:48:53 -0000
Received: (qmail 90907 invoked by uid 500); 18 Feb 2011 17:48:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90517 invoked by uid 500); 18 Feb 2011 17:48:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90507 invoked by uid 99); 18 Feb 2011 17:48:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 17:48:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 18 Feb 2011 17:48:48 +0000
Received: (qmail 14085 invoked by uid 99); 18 Feb 2011 17:48:28 -0000
Received: from localhost.apache.org (HELO mail-ww0-f48.google.com) (127.0.0.1)
  (smtp-auth username mahadev, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 17:48:28 +0000
Received: by wwd20 with SMTP id 20so3887454wwd.29
        for <general@hadoop.apache.org>; Fri, 18 Feb 2011 09:48:26 -0800 (PST)
MIME-Version: 1.0
Received: by 10.216.7.137 with SMTP id 9mr1739620wep.97.1298051306210; Fri, 18
 Feb 2011 09:48:26 -0800 (PST)
Received: by 10.216.50.134 with HTTP; Fri, 18 Feb 2011 09:48:26 -0800 (PST)
In-Reply-To: <B3FD7F41-EFC3-4C3A-BD66-DDA3B21BEADB@apache.org>
References: <4D5EA0E9.5070103@apache.org>
	<B3FD7F41-EFC3-4C3A-BD66-DDA3B21BEADB@apache.org>
Date: Fri, 18 Feb 2011 09:48:26 -0800
Message-ID: <AANLkTimeLqAyUX+2r52Oc0uKXMd0_t2eEinbp1DDFesR@mail.gmail.com>
Subject: Re: pdf design documents
From: Mahadev Konar <mahadev@apache.org>
To: general@hadoop.apache.org
Cc: "Owen O'Malley" <omalley@apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I am unsure why this is being discussed on general. I havent seen any
contributor to ignore request for source of the design document. If
someone wants to edit or submit a new one, just commenting on the jira
and asking for it should be enough. no?

thanks
mahadev

On Fri, Feb 18, 2011 at 9:12 AM, Owen O'Malley <omalley@apache.org> wrote:
>
> On Feb 18, 2011, at 8:40 AM, Doug Cutting wrote:
>
>> I feel PDF discourages collaboration. =A0Either plain text or HTML would
>> permit folks to more easily provide feedback, either as patches or as
>> alternate drafts.
>
> Ok, I've upload the latex files.
>
> -- Owen
>

From general-return-3094-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 18 17:49:56 2011
Return-Path: <general-return-3094-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14339 invoked from network); 18 Feb 2011 17:49:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2011 17:49:56 -0000
Received: (qmail 92338 invoked by uid 500); 18 Feb 2011 17:49:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92268 invoked by uid 500); 18 Feb 2011 17:49:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 92259 invoked by uid 99); 18 Feb 2011 17:49:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 17:49:51 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of atm@cloudera.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 17:49:46 +0000
Received: by qwe4 with SMTP id 4so3671908qwe.35
        for <general@hadoop.apache.org>; Fri, 18 Feb 2011 09:49:25 -0800 (PST)
Received: by 10.224.6.79 with SMTP id 15mr878295qay.168.1298051365362; Fri, 18
 Feb 2011 09:49:25 -0800 (PST)
MIME-Version: 1.0
Received: by 10.229.228.138 with HTTP; Fri, 18 Feb 2011 09:49:04 -0800 (PST)
In-Reply-To: <AANLkTinma0Dpfv3Z6LjWpTckMVq57hrZgWRDjZsu71AW@mail.gmail.com>
References: <AANLkTi=cyTOYxSbvn_YqRTuHPW0b74=EVWhky+oqxSGM@mail.gmail.com>
 <AANLkTikJMzXhYA1wJwxYm=9Zc=PdAtz6-5mOoUQEF5Dh@mail.gmail.com>
 <AANLkTi=6jLDBd7g2TF5cD_7b_v63n+m_eG+z9t31FW-g@mail.gmail.com>
 <AANLkTikxP0Y3y2RR-5PzAPn5pjpuQY8wZXptXmp1sML3@mail.gmail.com> <AANLkTinma0Dpfv3Z6LjWpTckMVq57hrZgWRDjZsu71AW@mail.gmail.com>
From: "Aaron T. Myers" <atm@cloudera.com>
Date: Fri, 18 Feb 2011 09:49:04 -0800
Message-ID: <AANLkTimE5Aq9EcGafnT1WxfZEk32wZyBLsy+_REcoTxh@mail.gmail.com>
Subject: Re: dfs.datanode.du.pct question
To: Mag Gam <magawake@gmail.com>
Cc: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175cb06c885c40049c922242

--0015175cb06c885c40049c922242
Content-Type: text/plain; charset=ISO-8859-1

Hi Mag,

Per http://hadoop.apache.org/mailing_lists.html, general@ is not the best
place for this discussion. Let's continue this on the actual JIRA:
https://issues.apache.org/jira/browse/HDFS-1564

--
Aaron T. Myers
Software Engineer, Cloudera



On Fri, Feb 18, 2011 at 4:24 AM, Mag Gam <magawake@gmail.com> wrote:

> Aaron,
>
> Thanks for following up on this issue for us.
>
> Has dfs.datanode.du.pct been depreciated?
>
> On Mon, Jan 3, 2011 at 6:06 PM, Aaron T. Myers <atm@cloudera.com> wrote:
> > Filed https://issues.apache.org/jira/browse/HDFS-1564
> >
> > --
> > Aaron T. Myers
> > Software Engineer, Cloudera
> >
> >
> >
> > On Sun, Jan 2, 2011 at 3:19 PM, Mag Gam <magawake@gmail.com> wrote:
> >
> >> Thanks for the response.
> >>
> >> It would be nice to have a feature like this. It will make our data
> >> nodes much more flexible.
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Fri, Dec 31, 2010 at 5:56 PM, Harsh J <qwertymaniac@gmail.com>
> wrote:
> >> > Hi,
> >> >
> >> > Isn't this property a legacy one? 0.18 is the last release I see with
> >> > this property. I've always used dfs.datanode.du.reserved instead,
> >> > reserving constant space across all dfs.data.dir volumes.
> >> >
> >> > On Fri, Dec 31, 2010 at 9:32 PM, Mag Gam <magawake@gmail.com> wrote:
> >> >> On some of my data nodes I have 2 filesystems. /fs1 and /fs2. Is it
> >> >> possible to set the du pct so, for fs1 its 80% usage and /fs2 its 20%
> >> >> usage?
> >> >
> >> > According to the release's code: No.
> >> >
> >> > (Unless two DataNodes are run off the same machine, which is not
> >> recommended.)
> >> >
> >> > --
> >> > Harsh J
> >> > www.harshj.com
> >> >
> >>
> >
>

--0015175cb06c885c40049c922242--

From general-return-3095-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 18 17:50:26 2011
Return-Path: <general-return-3095-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14472 invoked from network); 18 Feb 2011 17:50:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2011 17:50:26 -0000
Received: (qmail 93859 invoked by uid 500); 18 Feb 2011 17:50:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93664 invoked by uid 500); 18 Feb 2011 17:50:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93656 invoked by uid 99); 18 Feb 2011 17:50:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 17:50:21 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 17:50:11 +0000
Received: from SP2-EX07CAS03.ds.corp.yahoo.com (sp2-ex07cas03.corp.sp2.yahoo.com [98.137.59.35])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p1IHnElf058087
	for <general@hadoop.apache.org>; Fri, 18 Feb 2011 09:49:34 -0800 (PST)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by
 SP2-EX07CAS03.ds.corp.yahoo.com ([98.137.59.35]) with mapi; Fri, 18 Feb 2011
 09:49:28 -0800
From: Eric Yang <eyang@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 18 Feb 2011 09:49:27 -0800
Subject: Re: Hadoop testing project
Thread-Topic: Hadoop testing project
Thread-Index: AcvPiLHk7gmOLvkMQFyo8weY8+9nlAAC32ky
Message-ID: <C983F127.FBCB%eyang@yahoo-inc.com>
In-Reply-To: <4D5E9D33.5040003@apache.org>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org




On 2/18/11 8:24 AM, "Steve Loughran" <stevel@apache.org> wrote:

> On 17/02/2011 18:36, Eric Yang wrote:
>=20
>> I think the bigger concern is that Hadoop ecosystem does not have a stan=
dard
>> method in linking dependencies.  Hbase depends on Zookeeper, and Pig dep=
ends
>> on Hadoop and Hbase.  Then pig decided to put hadoop-core jar in it's ow=
n
>> jar file.  Chukwa depends on pig + hbase + hadoop and zookeeper.  The
>> version incompatibility is probably what driving people nuts.  Hence, th=
ere
>> is a new proposal on how to integrate among hadoop ecosystem.  I urge
>> project owners to review the proposal and provide feedbacks.
>>=20
>> The proposal is located at:
>>=20
>> https://issues.apache.org/jira/secure/attachment/12470823/deployment.pdf
>>=20
>> The related jiras are:
>>=20
>> https://issues.apache.org/jira/browse/HADOOP-6255
>> https://issues.apache.org/jira/browse/PIG-1857
>>=20
>> There are plans to file more jiras for related projects.
>=20
> I think a focus on RPMs/debs and then say "local VM for testing",
> because that gets people used to deploying on Linux from day one, rather
> than developing on windows and being surprised that things work
> differently in production. You still have the problem of debugging
> remotely to a VM in the VLAN, but that's tractable.
>=20

Packaging, VM testing, scale, then deployment automation.  My goal is
"deployment automation", and this requires polishing each steps along the
way.  For hadoop testing project, it depends on the type of testing that yo=
u
are focus on.  It will be easier to understand the scope and apply
implementation.

>> The integration
>> would also be a lot easier if all related projects are using maven for
>> dependency management.
>=20
> Do you mean Maven the tool or the maven repo format?
>=20

I meant both, it is a little easier to manage dependencies in maven build
tools than ivy, and deploying jar files to maven repository is also an
essential mean for integrate project more closely.

Regards,
Eric


From general-return-3096-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 18 18:00:39 2011
Return-Path: <general-return-3096-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17455 invoked from network); 18 Feb 2011 18:00:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2011 18:00:39 -0000
Received: (qmail 8873 invoked by uid 500); 18 Feb 2011 18:00:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8772 invoked by uid 500); 18 Feb 2011 18:00:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8764 invoked by uid 99); 18 Feb 2011 18:00:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 18:00:33 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 18:00:23 +0000
Received: from SP2-EX07CAS03.ds.corp.yahoo.com (sp2-ex07cas03.corp.sp2.yahoo.com [98.137.59.35])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p1IHxmGF034051
	for <general@hadoop.apache.org>; Fri, 18 Feb 2011 09:59:48 -0800 (PST)
Received: from SP2-EX07VS07.ds.corp.yahoo.com ([98.137.59.26]) by
 SP2-EX07CAS03.ds.corp.yahoo.com ([98.137.59.35]) with mapi; Fri, 18 Feb 2011
 09:59:49 -0800
From: Todd Papaioannou <toddp@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 18 Feb 2011 09:59:38 -0800
Subject: Re: pdf design documents
Thread-Topic: pdf design documents
Thread-Index: AcvPlaGMGMf3USiJRdiuvTIQVvOZ5g==
Message-ID: <11E7A42F-1DBF-4249-BD41-CC2148FFC3A2@yahoo-inc.com>
References: <4D5EA0E9.5070103@apache.org>
 <B3FD7F41-EFC3-4C3A-BD66-DDA3B21BEADB@apache.org>
 <AANLkTimeLqAyUX+2r52Oc0uKXMd0_t2eEinbp1DDFesR@mail.gmail.com>
In-Reply-To: <AANLkTimeLqAyUX+2r52Oc0uKXMd0_t2eEinbp1DDFesR@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Last time I checked, ctrl-c ctrl-v works just fine with PDF documents too..=
.

Thanks,

Todd

On Feb 18, 2011, at 11:19 PM, "Mahadev Konar" <mahadev@apache.org> wrote:

> I am unsure why this is being discussed on general. I havent seen any
> contributor to ignore request for source of the design document. If
> someone wants to edit or submit a new one, just commenting on the jira
> and asking for it should be enough. no?
>=20
> thanks
> mahadev
>=20
> On Fri, Feb 18, 2011 at 9:12 AM, Owen O'Malley <omalley@apache.org> wrote=
:
>>=20
>> On Feb 18, 2011, at 8:40 AM, Doug Cutting wrote:
>>=20
>>> I feel PDF discourages collaboration.  Either plain text or HTML would
>>> permit folks to more easily provide feedback, either as patches or as
>>> alternate drafts.
>>=20
>> Ok, I've upload the latex files.
>>=20
>> -- Owen
>>=20

From general-return-3097-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 18 18:34:06 2011
Return-Path: <general-return-3097-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 41567 invoked from network); 18 Feb 2011 18:34:06 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2011 18:34:06 -0000
Received: (qmail 54224 invoked by uid 500); 18 Feb 2011 18:34:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53620 invoked by uid 500); 18 Feb 2011 18:34:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53610 invoked by uid 99); 18 Feb 2011 18:34:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 18:34:01 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 18 Feb 2011 18:34:00 +0000
Received: (qmail 41445 invoked by uid 99); 18 Feb 2011 18:33:40 -0000
Received: from localhost.apache.org (HELO [192.168.168.144]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 18:33:40 +0000
Message-ID: <4D5EBB83.30405@apache.org>
Date: Fri, 18 Feb 2011 10:33:39 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: pdf design documents
References: <4D5EA0E9.5070103@apache.org> <B3FD7F41-EFC3-4C3A-BD66-DDA3B21BEADB@apache.org>
In-Reply-To: <B3FD7F41-EFC3-4C3A-BD66-DDA3B21BEADB@apache.org>
X-Enigmail-Version: 1.1.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On 02/18/2011 09:12 AM, Owen O'Malley wrote:
> Ok, I've upload the latex files.

Thanks, Owen!

Doug

From general-return-3098-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 18 18:40:16 2011
Return-Path: <general-return-3098-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43828 invoked from network); 18 Feb 2011 18:40:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2011 18:40:15 -0000
Received: (qmail 69269 invoked by uid 500); 18 Feb 2011 18:40:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68863 invoked by uid 500); 18 Feb 2011 18:40:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68504 invoked by uid 99); 18 Feb 2011 18:40:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 18:40:11 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 18 Feb 2011 18:40:09 +0000
Received: (qmail 43477 invoked by uid 99); 18 Feb 2011 18:39:47 -0000
Received: from localhost.apache.org (HELO [192.168.168.144]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 18:39:47 +0000
Message-ID: <4D5EBCF2.1060800@apache.org>
Date: Fri, 18 Feb 2011 10:39:46 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: pdf design documents
References: <4D5EA0E9.5070103@apache.org>	<B3FD7F41-EFC3-4C3A-BD66-DDA3B21BEADB@apache.org> <AANLkTimeLqAyUX+2r52Oc0uKXMd0_t2eEinbp1DDFesR@mail.gmail.com>
In-Reply-To: <AANLkTimeLqAyUX+2r52Oc0uKXMd0_t2eEinbp1DDFesR@mail.gmail.com>
X-Enigmail-Version: 1.1.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 02/18/2011 09:48 AM, Mahadev Konar wrote:
> I am unsure why this is being discussed on general.

I feel this is an issue of general interest and wanted to find out how
others felt about it, not just those following those two particular
issues.  Over the years, on several projects, I've expressed my
preference that project documents be posted in open, editable formats.
I don't recall this being discussed on Hadoop, but, if it has, it's been
a while and seems to have been forgotten, so I thought I'd raise it as
an issue of general policy so that we can all come to some agreement on
this.

Doug

From general-return-3099-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 18 19:10:14 2011
Return-Path: <general-return-3099-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 62753 invoked from network); 18 Feb 2011 19:10:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2011 19:10:14 -0000
Received: (qmail 22373 invoked by uid 500); 18 Feb 2011 19:10:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21965 invoked by uid 500); 18 Feb 2011 19:10:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21957 invoked by uid 99); 18 Feb 2011 19:10:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 19:10:09 +0000
X-ASF-Spam-Status: No, hits=3.3 required=5.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 19:09:59 +0000
Received: from SP2-EX07CAS02.ds.corp.yahoo.com (sp2-ex07cas02.corp.sp2.yahoo.com [98.137.59.38])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p1IJ9QiZ014503
	for <general@hadoop.apache.org>; Fri, 18 Feb 2011 11:09:26 -0800 (PST)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by
 SP2-EX07CAS02.ds.corp.yahoo.com ([98.137.59.38]) with mapi; Fri, 18 Feb 2011
 11:09:26 -0800
From: Eric Yang <eyang@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 18 Feb 2011 11:09:25 -0800
Subject: Re: pdf design documents
Thread-Topic: pdf design documents
Thread-Index: AcvPm19lPhV3tRv8RsGdtqDhWdPzzwAA/v68
Message-ID: <C98403E5.FBE0%eyang@yahoo-inc.com>
In-Reply-To: <4D5EBCF2.1060800@apache.org>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C98403E5FBE0eyangyahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C98403E5FBE0eyangyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

FYI, Owen's design doc for HADOOP-6255 is also available in xml format in t=
he proposed patch.  I think as long as there is a mean to edit/revise the d=
esign doc, it should be fine right?

Regards,
Eric

On 2/18/11 10:39 AM, "Doug Cutting" <cutting@apache.org> wrote:

On 02/18/2011 09:48 AM, Mahadev Konar wrote:
> I am unsure why this is being discussed on general.

I feel this is an issue of general interest and wanted to find out how
others felt about it, not just those following those two particular
issues.  Over the years, on several projects, I've expressed my
preference that project documents be posted in open, editable formats.
I don't recall this being discussed on Hadoop, but, if it has, it's been
a while and seems to have been forgotten, so I thought I'd raise it as
an issue of general policy so that we can all come to some agreement on
this.

Doug


--_000_C98403E5FBE0eyangyahooinccom_--

From general-return-3100-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 18 19:50:56 2011
Return-Path: <general-return-3100-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77364 invoked from network); 18 Feb 2011 19:50:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2011 19:50:56 -0000
Received: (qmail 94614 invoked by uid 500); 18 Feb 2011 19:50:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94288 invoked by uid 500); 18 Feb 2011 19:50:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94280 invoked by uid 99); 18 Feb 2011 19:50:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 19:50:53 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 18 Feb 2011 19:50:53 +0000
Received: (qmail 77238 invoked by uid 99); 18 Feb 2011 19:50:32 -0000
Received: from localhost.apache.org (HELO [192.168.168.144]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 19:50:32 +0000
Message-ID: <4D5ECD86.6080104@apache.org>
Date: Fri, 18 Feb 2011 11:50:30 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: pdf design documents
References: <C98403E5.FBE0%eyang@yahoo-inc.com>
In-Reply-To: <C98403E5.FBE0%eyang@yahoo-inc.com>
X-Enigmail-Version: 1.1.2
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On 02/18/2011 11:09 AM, Eric Yang wrote:
> FYI, Owen's design doc for HADOOP-6255 is also available in xml
> format in the proposed patch.  I think as long as there is a mean to
> edit/revise the design doc, it should be fine right?

Yes, that looks great to me.  I am just proposing that, when a design is
first presented as a proposal intended for collaborative feedback prior
to implementation that it be provided in a format that's easy for all to
modify.  Does that seem unreasonable?

Doug

From general-return-3101-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 18 19:55:43 2011
Return-Path: <general-return-3101-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78628 invoked from network); 18 Feb 2011 19:55:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2011 19:55:42 -0000
Received: (qmail 8201 invoked by uid 500); 18 Feb 2011 19:55:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7992 invoked by uid 500); 18 Feb 2011 19:55:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7983 invoked by uid 99); 18 Feb 2011 19:55:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 19:55:40 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [81.169.146.161] (HELO mo-p00-ob.rzone.de) (81.169.146.161)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 19:55:33 +0000
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; t=1298058910; l=1404;
	s=domk; d=brainlounge.de;
	h=Content-Transfer-Encoding:Content-Type:In-Reply-To:References:
	Subject:To:MIME-Version:From:Date:X-RZG-CLASS-ID:X-RZG-AUTH;
	bh=thvBk0NH0cfPEGsqz6sMsHrdWTk=;
	b=fSKFF0AlmmT096n9kUB4qmGlm59k1RRjcuEI+GcbcYYBwn90DDtYyGYSCCqEX87HOE4
	F9CwLJcrG3anuNwrhUXLk4RLwP66Kc325xHvJBBnMivkFk5gpcohMDcOLB6W9WxXfJFwc
	dZXVO5wc3EIhUkSnXeBjbOtsvs8P30uBKq0=
X-RZG-AUTH: :LmkWe0TmffCene0uRkJY6AqDWBjdcSs8EVOKK4t9JJpSMjflHqg6LsBmWxPH/k40rz9aRX/I1Oxv
X-RZG-CLASS-ID: mo00
Received: from vitamin.local (e180164016.adsl.alicedsl.de [85.180.164.16])
	by post.strato.de (fruni mo61) (RZmta 25.5)
	with ESMTPA id o03027n1IJ5VQs for <general@hadoop.apache.org>;
	Fri, 18 Feb 2011 20:55:09 +0100 (MET)
Message-ID: <4D5ECE9D.9030603@brainlounge.de>
Date: Fri, 18 Feb 2011 20:55:09 +0100
From: Bernd Fondermann <bf_jak@brainlounge.de>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.13) Gecko/20101207 Lightning/1.0b2 Thunderbird/3.1.7
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Abandon hdfsproxy HDFS contrib
References: <D97CF74A-147E-4573-B188-2F9C48CFC482@mac.com> <AANLkTi=F2eBsACBfjYSVava+VwFcsRpqZdRRN+OHXNRJ@mail.gmail.com> <823262F0-EE37-4A0A-8F8C-D9422541D67B@mac.com> <0CBAFDBB-AC2A-495D-8936-4DF4D6731B69@Holsman.NET> <AANLkTimUhDbTnwn6Hk2oYXFJMts+A=FnL-q-veXqX1m0@mail.gmail.com> <F2EC1B1B-A2BB-4FC8-9842-3904A25338EE@gbiv.com> <AANLkTikVcGcKHnh9EgUDQr2JGLPvkBvBRgn4my3uJfOc@mail.gmail.com> <1FEA58C1-AA9A-4A6F-8873-C995057CDD19@gbiv.com> <AANLkTik1LFE4z0PGoOEudbFJ93e+LCrwTGLPmHybo-AL@mail.gmail.com> <BF4E6121-8323-4B5F-A10E-2434B2FC0CE2@linkedin.com>
In-Reply-To: <BF4E6121-8323-4B5F-A10E-2434B2FC0CE2@linkedin.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 18.02.11 17:11, Allen Wittenauer wrote:
>
> On Feb 18, 2011, at 2:11 AM, Bernd Fondermann wrote:
>> I don't know how many Y-employees are working on H internally. Only
>> the contributors can sort that out.
>
> Did Carol Bartz run over your puppy or something?

Something, definitively. I assume EOR.

 > You don't appear to realize that pretty much all the
> major companies that house the major committers
 > are doing essentially the same thing:
> work on a major feature patch with their internal
> teams then offer up the patch mostly complete, having
> dealt with all of the minor nits on the way due to
> having field tested it for real.

I realize that. I even critized it: The fact that a closed group 
develops code until they donate it to the project in smaller or larger 
chunks. Whoever these closed groups are, I don't make a difference.

> The patches still get discussed before getting committed.
> In some cases, this has resulted in either the patch
> getting rejected/not committed or reworked.

Sounds good.

> I sometimes wonder if the people who troll the team
> realize that Hadoop is essentially a distributed, networked
> operating system.

I realize that.

> That requires a higher level of quality control than a simple framework.

Which again is only accessible to a closed group. Which excludes the 
rest of the committership.

   Bernd

From general-return-3102-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 18 19:58:57 2011
Return-Path: <general-return-3102-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 79065 invoked from network); 18 Feb 2011 19:58:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2011 19:58:57 -0000
Received: (qmail 12134 invoked by uid 500); 18 Feb 2011 19:58:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12056 invoked by uid 500); 18 Feb 2011 19:58:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12047 invoked by uid 99); 18 Feb 2011 19:58:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 19:58:54 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of dhruba@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 19:58:48 +0000
Received: by wye20 with SMTP id 20so4070666wye.35
        for <general@hadoop.apache.org>; Fri, 18 Feb 2011 11:58:28 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=6kNZqHA7GN4p+l9yhDTrX5HbUjkMY1YxaE0k61EBqU8=;
        b=tBFO72RzamjpPxEpw+/cuYEzY5xApV6bwR9dX9Sq6PTQ8DVuw2Kl9Z2X7Em9d89FPa
         LBRXUBG1DxxwCQrb13L29NKVUKTolkyUAgPO0B1FEMxHGktWJtZRo0xsbGncgwsPyfmM
         ySNu71cRtTeJAIP53OtgaYEKsldq+9QfotyqU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=OE7/HJqwLtOWZpI/m0exa//AzlXtjCffQClpkxdE5pt4lzO6reyVjp/s0rXyqvN6+K
         /Y4/GyCsAZ3N+QdRIX57db7co9uvKjwvEqsstT1+9/b3LJrUYPM2OAF540dpzoLnIkO8
         jQY3xoE1cJiau5iN9PiwaOPPfwV/PD0HOmprU=
MIME-Version: 1.0
Received: by 10.216.173.147 with SMTP id v19mr1109145wel.102.1298059097513;
 Fri, 18 Feb 2011 11:58:17 -0800 (PST)
Received: by 10.216.255.147 with HTTP; Fri, 18 Feb 2011 11:58:17 -0800 (PST)
In-Reply-To: <4D5ECD86.6080104@apache.org>
References: <C98403E5.FBE0%eyang@yahoo-inc.com>
	<4D5ECD86.6080104@apache.org>
Date: Fri, 18 Feb 2011 11:58:17 -0800
Message-ID: <AANLkTimwqe0GsNaqz3dKm=KXkZH5XdXjg4orT8o3Mfu5@mail.gmail.com>
Subject: Re: pdf design documents
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016367b6440679fa3049c93ef2f
X-Virus-Checked: Checked by ClamAV on apache.org

--0016367b6440679fa3049c93ef2f
Content-Type: text/plain; charset=ISO-8859-1

I would request that a pdf version be provided too (along with the other
editable formats), it just makes it easier for me to read it without much
effort. It helps me keep up with new designs that are being proposed, even
though I am not competent enough to comment/collaborate on those designs.

thanks,
dhruba

On Fri, Feb 18, 2011 at 11:50 AM, Doug Cutting <cutting@apache.org> wrote:

> On 02/18/2011 11:09 AM, Eric Yang wrote:
> > FYI, Owen's design doc for HADOOP-6255 is also available in xml
> > format in the proposed patch.  I think as long as there is a mean to
> > edit/revise the design doc, it should be fine right?
>
> Yes, that looks great to me.  I am just proposing that, when a design is
> first presented as a proposal intended for collaborative feedback prior
> to implementation that it be provided in a format that's easy for all to
> modify.  Does that seem unreasonable?
>
> Doug
>



-- 
Connect to me at http://www.facebook.com/dhruba

--0016367b6440679fa3049c93ef2f--

From general-return-3103-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 18 22:50:18 2011
Return-Path: <general-return-3103-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46689 invoked from network); 18 Feb 2011 22:50:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2011 22:50:17 -0000
Received: (qmail 96507 invoked by uid 500); 18 Feb 2011 22:50:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96286 invoked by uid 500); 18 Feb 2011 22:50:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96189 invoked by uid 99); 18 Feb 2011 22:50:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 22:50:14 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Feb 2011 22:50:06 +0000
Received: from SP2-EX07CAS02.ds.corp.yahoo.com (sp2-ex07cas02.corp.sp2.yahoo.com [98.137.59.38])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p1IMnLnQ081974
	for <general@hadoop.apache.org>; Fri, 18 Feb 2011 14:49:21 -0800 (PST)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by
 SP2-EX07CAS02.ds.corp.yahoo.com ([98.137.59.38]) with mapi; Fri, 18 Feb 2011
 14:49:21 -0800
From: Eric Yang <eyang@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 18 Feb 2011 14:49:19 -0800
Subject: Re: pdf design documents
Thread-Topic: pdf design documents
Thread-Index: AcvPpTDCWh5GU1GQTH6VxIqF8YPHIQAGOLYN
Message-ID: <C984376F.FC01%eyang@yahoo-inc.com>
In-Reply-To: <4D5ECD86.6080104@apache.org>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

The editable proposal is good advice.  It would be nice to present both
editable version and read-only version to track the history of the proposal=
.
If it comes down to streamline efficiency, having editable proposal is
preferred.  Hence, I will use editable document next time.

Regards,
Eric

On 2/18/11 11:50 AM, "Doug Cutting" <cutting@apache.org> wrote:

> On 02/18/2011 11:09 AM, Eric Yang wrote:
>> FYI, Owen's design doc for HADOOP-6255 is also available in xml
>> format in the proposed patch.  I think as long as there is a mean to
>> edit/revise the design doc, it should be fine right?
>=20
> Yes, that looks great to me.  I am just proposing that, when a design is
> first presented as a proposal intended for collaborative feedback prior
> to implementation that it be provided in a format that's easy for all to
> modify.  Does that seem unreasonable?
>=20
> Doug
>=20


From general-return-3104-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Feb 20 02:53:36 2011
Return-Path: <general-return-3104-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85349 invoked from network); 20 Feb 2011 02:53:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Feb 2011 02:53:35 -0000
Received: (qmail 83057 invoked by uid 500); 20 Feb 2011 02:53:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82694 invoked by uid 500); 20 Feb 2011 02:53:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82686 invoked by uid 99); 20 Feb 2011 02:53:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 Feb 2011 02:53:31 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [128.149.139.106] (HELO mail.jpl.nasa.gov) (128.149.139.106)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 Feb 2011 02:53:23 +0000
Received: from mail.jpl.nasa.gov (altvirehtstap02.jpl.nasa.gov [128.149.137.73])
	by smtp.jpl.nasa.gov (Switch-3.4.3/Switch-3.4.3) with ESMTP id p1K2qx5N027040
	(using TLSv1/SSLv3 with cipher RC4-MD5 (128 bits) verified FAIL)
	for <general@hadoop.apache.org>; Sat, 19 Feb 2011 18:53:00 -0800
Received: from ALTPHYEMBEVSP20.RES.AD.JPL ([128.149.137.82]) by
 ALTVIREHTSTAP02.RES.AD.JPL ([128.149.137.73]) with mapi; Sat, 19 Feb 2011
 18:53:00 -0800
From: "Mattmann, Chris A (388J)" <chris.a.mattmann@jpl.nasa.gov>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Sat, 19 Feb 2011 18:52:58 -0800
Subject: Re: [VOTE] Abandon mrunit MapReduce contrib
Thread-Topic: [VOTE] Abandon mrunit MapReduce contrib
Thread-Index: AcvQqUhm+SfJyMwYTW2glxPixqv2qQ==
Message-ID: <70074956-0C23-4CA2-A414-6F1C2C6453F2@jpl.nasa.gov>
References: <77405974-6771-4604-926B-976240743F3C@mac.com>
 <AANLkTim80KJ3f14iM6Ltzy7UG613myhN-AiwQ3pCmVdi@mail.gmail.com>
 <AANLkTiknkrWK=GzGMVj4_Z6p5Jn8TQu1P5TkaFSgORJk@mail.gmail.com>
 <AANLkTin0TguLYHi9X0uL3ZF7tCz5xTHBYa7056oKF2Ae@mail.gmail.com>
 <93644D3B-4830-4813-92BE-809738969CF9@apache.org>
 <AANLkTi=dbGSLhtFA+Rf9LgaX05v45HDLoQDE_2C5ULAf@mail.gmail.com>
 <65311075-A30E-4971-A2BA-C75409C6486E@jpl.nasa.gov>
 <AANLkTi=XbpVUmg_TpH7eqEn8w7Gcu9KiwtJmKMLa5w5n@mail.gmail.com>
 <FAE19BAA-47A7-4747-BDD8-5AA42C259EB6@jpl.nasa.gov>
 <AANLkTin2JvOGdZT5g5tmTGeCpD3Qz2soZWcTQtHMsSza@mail.gmail.com>
 <AANLkTi=iGCBrp6+dsK66-+fWF7UoLPSqMOhoNUnjbMHY@mail.gmail.com>
 <AANLkTimXXt3e4fL==fxhU9MJ5v=hPeEa746-3CV4x4Pc@mail.gmail.com>
 <C1C69515-B440-4C3F-9918-9E9767E4804D@jpl.nasa.gov>
 <AANLkTimBS+KLnknfh8hGzF3s3xdqN5fVjEexy3cpXzE7@mail.gmail.com>
 <5C5BE47B-6159-4678-8455-219CF1F41863@jpl.nasa.gov>
In-Reply-To: <5C5BE47B-6159-4678-8455-219CF1F41863@jpl.nasa.gov>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Source-IP: altvirehtstap02.jpl.nasa.gov [128.149.137.73]
X-Source-Sender: chris.a.mattmann@jpl.nasa.gov
X-AUTH: Authorized
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Guys,

I took a pass at the MRUnit proposal [1]. It's far from complete, but just =
letting you know I took a swipe :)

Cheers,
Chris

[1] http://wiki.apache.org/incubator/MRUnitProposal

On Feb 17, 2011, at 1:52 PM, Mattmann, Chris A (388J) wrote:

> Thanks Patrick.
>=20
> Sounds good. Besides mentors, we need a champion, who must be an ASF memb=
er. So, either me, you or Nigel can do it. I'm already championing Gora, an=
d would be happy to champion MRUnit, but am open to either one of you guys =
doing it too (or even another ASF member). Just let me know. I'll head over=
 to the wiki and add my info. Thanks for getting this started Eric+Patrick!
>=20
> Cheers,
> Chris
>=20
> On Feb 17, 2011, at 1:00 PM, Patrick Hunt wrote:
>=20
>> Chris a page is up (still being created by Eric afaict):
>> http://wiki.apache.org/incubator/MRUnitProposal
>>=20
>> I took the liberty of listing us as mentors.
>>=20
>> Patrick
>>=20
>> On Thu, Feb 17, 2011 at 11:31 AM, Mattmann, Chris A (388J)
>> <chris.a.mattmann@jpl.nasa.gov> wrote:
>>> Hey Guys,
>>>=20
>>> FYI on this: Eric has mentioned he is going to start the Incubator prop=
osal for MRUnit. Let's start small and then grow big (as needed). It seems =
like we've achieved enough consensus for the required mentors and critical =
mass to make an MRUnit Incubator proposal and then to have the Incubator co=
mmunity weigh in. If that expands to include other testing projects/etc., w=
e can address that over the Incubation process, and as needed.
>>>=20
>>> Eric: as soon as that wiki page is up, I'd be happy to add my name to i=
t as a mentor and /kick the can on this.
>>>=20
>>> Cheers,
>>> Chris
>>>=20
>>> On Feb 17, 2011, at 11:11 AM, Aaron Kimball wrote:
>>>=20
>>>> The MRUnit community is a specific subset of the Hadoop community: Eng=
ineers
>>>> writing Java code running on Hadoop. The Hadoop community also include=
s
>>>> IT/ops staff who maintain Hadoop clusters, data scientists who use too=
ls
>>>> such as Pig & Hive, as well as those written by the aforementioned
>>>> engineers, etc.
>>>>=20
>>>> The Hadoop project has long recognized that tools aimed at a specific =
subset
>>>> of the Hadoop community, with separate release cycles, can more succes=
sfully
>>>> reach their aims by splitting into incubator projects. Hive, Pig, and =
HBase,
>>>> for example, have all gone this path.
>>>>=20
>>>> A "current" version of MRUnit would need to compile against multiple
>>>> versions of Hadoop itself. This is not possible if it is in the same s=
ource
>>>> tree as Hadoop.
>>>>=20
>>>> - Aaron
>>>>=20
>>>> On Thu, Feb 17, 2011 at 5:31 AM, Bernd Fondermann <
>>>> bernd.fondermann@googlemail.com> wrote:
>>>>=20
>>>>> On Fri, Feb 11, 2011 at 23:10, Aaron Kimball <akimball83@gmail.com> w=
rote:
>>>>>> The main reason I am interested in removing MRUnit from Hadoop is th=
at I
>>>>>> believe that MRUnit deserves its own release cycle. I think this is =
in
>>>>> the
>>>>>> best interest of its users.
>>>>>=20
>>>>> Not in mine, at least. (I'm writing MR unit tests.)
>>>>> Many projects release more than one product. I'd rather get MRUnit
>>>>> from the same source where I get my MR from.
>>>>> Separate release cylcles would be ok for me, though.
>>>>>=20
>>>>>> Perhaps more importantly, access to new features in MRUnit should no=
t
>>>>>> require upgrading one's entire Hadoop deployment; this is a client
>>>>> library
>>>>>> that depends only on Hadoop's public APIs.
>>>>>=20
>>>>> +1.
>>>>>=20
>>>>>> My primary concern is to move MRUnit to a place where the community =
can
>>>>>> derive the most benefit from it. The Apache Incubator could fulfill =
this
>>>>>> role; given the presence of individuals willing to mentor this proje=
ct, I
>>>>>> believe this would be a successful way to release MRUnit more quickl=
y and
>>>>>> continue to work to grow the MRUnit community.
>>>>>=20
>>>>> What are your expectations what MRUnit would become, software-wise?
>>>>> Wouldn't the MRUnit community be largely the same as the Hadoop-MR
>>>>> community?
>>>>>=20
>>>>> Bernd
>>>>>=20
>>>=20
>>>=20
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Chris Mattmann, Ph.D.
>>> Senior Computer Scientist
>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> Office: 171-266B, Mailstop: 171-246
>>> Email: chris.a.mattmann@nasa.gov
>>> WWW:   http://sunset.usc.edu/~mattmann/
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Adjunct Assistant Professor, Computer Science Department
>>> University of Southern California, Los Angeles, CA 90089 USA
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>=20
>>>=20
>=20
>=20
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW:   http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>=20


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


From general-return-3105-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 21 06:03:00 2011
Return-Path: <general-return-3105-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59942 invoked from network); 21 Feb 2011 06:03:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Feb 2011 06:03:00 -0000
Received: (qmail 67406 invoked by uid 500); 21 Feb 2011 06:02:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66896 invoked by uid 500); 21 Feb 2011 06:02:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66888 invoked by uid 99); 21 Feb 2011 06:02:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Feb 2011 06:02:54 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.96 as permitted sender)
Received: from [17.148.16.96] (HELO asmtpout021.mac.com) (17.148.16.96)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Feb 2011 06:02:45 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp021.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LGY007ADE40GB10@asmtp021.mac.com>
 for general@hadoop.apache.org; Sun, 20 Feb 2011 22:02:25 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-02-20_08:2011-02-19,2011-02-20,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1102200156
From: Nigel Daley <ndaley@mac.com>
Subject: precommit testing gone crazy
Date: Sun, 20 Feb 2011 22:02:24 -0800
Message-id: <784E9D98-E8EE-4DCC-824B-94F63B8973CE@mac.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

Apologies.  Looks like the precommit testing control file got corrupted over the weekend and most Patch Available issues got run thru Hudson again.

Nige

From general-return-3106-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 21 15:31:51 2011
Return-Path: <general-return-3106-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 7153 invoked from network); 21 Feb 2011 15:31:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Feb 2011 15:31:50 -0000
Received: (qmail 94808 invoked by uid 500); 21 Feb 2011 15:31:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92377 invoked by uid 500); 21 Feb 2011 15:31:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 92358 invoked by uid 99); 21 Feb 2011 15:31:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Feb 2011 15:31:43 +0000
X-ASF-Spam-Status: No, hits=4.4 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 74.201.117.226 is neither permitted nor denied by domain of kiddlee1989@gmail.com)
Received: from [74.201.117.226] (HELO smtp.stumbleupon.com) (74.201.117.226)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Feb 2011 15:31:37 +0000
Received: from www.stumbleupon.com (sv4web54 [10.4.21.44])
	by smtp.stumbleupon.com (Postfix) with ESMTP id CC6976E1A
	for <general@hadoop.apache.org>; Mon, 21 Feb 2011 07:31:15 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=stumbleupon.com;
	s=mail; t=1298302275;
	bh=6skzqyc1a+vuGBoMD56I3rngtYmSzOtJi8LewnQQrRo=;
	h=Date:To:From:Reply-to:Subject:Message-ID:Sender:MIME-Version:
	 Content-Type;
	b=SQz2Dr0oO9ZYS07GDjTI7lZIO/DqZT7DZyYXhu9nf4ow7g16kEAhjWW1R4Ys0dd+S
	 Jiv5iFQiHzcp3m5eKSL+3BXeV1ojepGzrJrfY7znqyXVcJOSJv2OSc4BCg/IBlu5s4
	 bNuDLPwlPReEYDO3ylz6Z+OvNsELVSlkHLQFrAz0=
Date: Mon, 21 Feb 2011 07:31:15 -0800
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
From: kiddlee <kiddlee1989@gmail.com>
Reply-to: kiddlee <kiddlee1989@gmail.com>
Subject: Join me on StumbleUpon!
Message-ID: <d1eaea1bb8d7747992288877ae9ab4a1@www.stumbleupon.com>
X-Priority: 3
X-Mailer: PHPMailer 5.1 (phpmailer.sourceforge.net)
Sender: service@stumbleupon.com
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="b1_d1eaea1bb8d7747992288877ae9ab4a1"
X-Virus-Checked: Checked by ClamAV on apache.org

--b1_d1eaea1bb8d7747992288877ae9ab4a1
Content-Type: text/plain; charset = "utf-8"
Content-Transfer-Encoding: 8bit

[StumbleUpon]
(http://www.stumbleupon.com/to/e/849181372:VISJZDHUSKXUA8LI/invite-4/home1/https%253A%252F%252Fwww.stumbleupon.com%252F%253Fpre%253Dinvite%2526u%253Duf1q) 
Join for free (http://www.stumbleupon.com/to/e/849181372:VISJZDHUSKXUA8LI/invite-4/home2/https%253A%252F%252Fwww.stumbleupon.com%252F%253Fpre%253Dinvite%2526u%253Duf1q) 

I've been discovering great stuff on the web with StumbleUpon and I think you'll love it!
Join now so we can see and share each other's favorite web pages.  It only takes a minute - check it out!
- kiddlee


StumbleUpon is a discovery engine that finds the best of the web, recommended just for you. 
Learn more (http://www.stumbleupon.com/to/e/849181372:VISJZDHUSKXUA8LI/invite-4/hf/http%253A%252F%252Fwww.stumbleupon.com%252F) 




If you do not wish to receive emails sent by StumbleUpon, please
click here (http://www.stumbleupon.com/to/e/849181372:VISJZDHUSKXUA8LI/invite-4/oo/__eNrLKCkpsNLXLy8v1ysuKc1NykktLcjP00vOz9XPyy_JTMtMTizJzM8r1ivIKLBPzU3MzEnOT0m1DfMM9opy8QgN9o4IdbTw8VQrqSxItfXMK8ssSQVcMCpDHrk%252C) 


(c) StumbleUpon 2001 - 2011


--b1_d1eaea1bb8d7747992288877ae9ab4a1--



From general-return-3107-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 21 15:32:48 2011
Return-Path: <general-return-3107-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 7655 invoked from network); 21 Feb 2011 15:32:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Feb 2011 15:32:48 -0000
Received: (qmail 99484 invoked by uid 500); 21 Feb 2011 15:32:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97991 invoked by uid 500); 21 Feb 2011 15:32:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97671 invoked by uid 99); 21 Feb 2011 15:32:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Feb 2011 15:32:43 +0000
X-ASF-Spam-Status: No, hits=4.4 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 74.201.117.226 is neither permitted nor denied by domain of kiddlee1989@gmail.com)
Received: from [74.201.117.226] (HELO smtp.stumbleupon.com) (74.201.117.226)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Feb 2011 15:32:36 +0000
Received: from www.stumbleupon.com (sv4web7 [10.4.27.36])
	by smtp.stumbleupon.com (Postfix) with ESMTP id CDB1972E8
	for <general@hadoop.apache.org>; Mon, 21 Feb 2011 07:31:15 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=stumbleupon.com;
	s=mail; t=1298302275;
	bh=5MoYH+CVRaxNHnGoyI76RdwjUxqB8Z7p2jFgIhUmqTk=;
	h=Date:To:From:Reply-to:Subject:Message-ID:Sender:MIME-Version:
	 Content-Type;
	b=GtlNKug+3wU+kVDqH8TsBhcu2KD/QHw2I0HuxBh1Zxgk14D0JFZWRjQFSxC0Udbrt
	 /G+0YBAp3emVMnbV0JD4ItjlnQF2k+9UACa/ZDejxPkutr1Bv3k2gRUqIpdkYsDkQF
	 LGbE0vKalU4Y2IwC6qGoRj7TQn0RCHpz441X4r3c=
Date: Mon, 21 Feb 2011 07:31:15 -0800
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
From: kiddlee <kiddlee1989@gmail.com>
Reply-to: kiddlee <kiddlee1989@gmail.com>
Subject: Join me on StumbleUpon!
Message-ID: <059927cba83651ee30d3f7ce1a664271@www.stumbleupon.com>
X-Priority: 3
X-Mailer: PHPMailer 5.1 (phpmailer.sourceforge.net)
Sender: service@stumbleupon.com
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="b1_059927cba83651ee30d3f7ce1a664271"
X-Virus-Checked: Checked by ClamAV on apache.org

--b1_059927cba83651ee30d3f7ce1a664271
Content-Type: text/plain; charset = "utf-8"
Content-Transfer-Encoding: 8bit

[StumbleUpon]
(http://www.stumbleupon.com/to/e/849181372:VISJZDHUSKXUA8LI/invite-4/home1/https%253A%252F%252Fwww.stumbleupon.com%252F%253Fpre%253Dinvite%2526u%253Duf1q) 
Join for free (http://www.stumbleupon.com/to/e/849181372:VISJZDHUSKXUA8LI/invite-4/home2/https%253A%252F%252Fwww.stumbleupon.com%252F%253Fpre%253Dinvite%2526u%253Duf1q) 

I've been discovering great stuff on the web with StumbleUpon and I think you'll love it!
Join now so we can see and share each other's favorite web pages.  It only takes a minute - check it out!
- kiddlee


StumbleUpon is a discovery engine that finds the best of the web, recommended just for you. 
Learn more (http://www.stumbleupon.com/to/e/849181372:VISJZDHUSKXUA8LI/invite-4/hf/http%253A%252F%252Fwww.stumbleupon.com%252F) 




If you do not wish to receive emails sent by StumbleUpon, please
click here (http://www.stumbleupon.com/to/e/849181372:VISJZDHUSKXUA8LI/invite-4/oo/__eNrLKCkpsNLXLy8v1ysuKc1NykktLcjP00vOz9XPyy_JTMtMTizJzM8r1ivIKLBPzU3MzEnOT0m1DfMM9opy8QgN9o4IdbTw8VQrqSxItfXMK8ssSQVcMCpDHrk%252C) 


(c) StumbleUpon 2001 - 2011


--b1_059927cba83651ee30d3f7ce1a664271--



From general-return-3108-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 21 15:50:09 2011
Return-Path: <general-return-3108-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 13533 invoked from network); 21 Feb 2011 15:50:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Feb 2011 15:50:08 -0000
Received: (qmail 22183 invoked by uid 500); 21 Feb 2011 15:50:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21684 invoked by uid 500); 21 Feb 2011 15:50:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21676 invoked by uid 99); 21 Feb 2011 15:50:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Feb 2011 15:50:03 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jaybooth@gmail.com designates 209.85.220.176 as permitted sender)
Received: from [209.85.220.176] (HELO mail-vx0-f176.google.com) (209.85.220.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Feb 2011 15:49:57 +0000
Received: by vxc41 with SMTP id 41so1172674vxc.35
        for <general@hadoop.apache.org>; Mon, 21 Feb 2011 07:49:36 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type:content-transfer-encoding;
        bh=DyQdSpvujpF3L40X+OREiKL2xuopYTA6ICTXZK+xWTQ=;
        b=MjPEk4WSFXWw0H5F31zfhQADb4qIl2BEqyUQsiggFqZK9ELlkLdn0oQyhMiZvDikXg
         mOD7jfkFjt+lPXFaCs9nntySefLWZLRwl7E0sxIM/yKxn7WojDB11vcBP45djeQq/nHI
         jVezqG1Fkg1TciNbkZTVw1gBLHZzv6yTShqnE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=QiJNciX8lh6qfSlZOI1qgM4/aJjiD55L45l0UrFIj9pJ7KZj/F0Pb0or5abn2U24Rv
         rwnYjCk+MPsNdCAg3cBVK0jlWQb753S4fEbCP2clvtoh8bsW/HoYLAzXA1XvQnXcztff
         koMFdibdarYuZhOBXyNuBwEymYUZnFpvX3Yhk=
MIME-Version: 1.0
Received: by 10.52.167.72 with SMTP id zm8mr2112796vdb.172.1298303376251; Mon,
 21 Feb 2011 07:49:36 -0800 (PST)
Received: by 10.220.167.132 with HTTP; Mon, 21 Feb 2011 07:49:36 -0800 (PST)
In-Reply-To: <059927cba83651ee30d3f7ce1a664271@www.stumbleupon.com>
References: <059927cba83651ee30d3f7ce1a664271@www.stumbleupon.com>
Date: Mon, 21 Feb 2011 10:49:36 -0500
Message-ID: <AANLkTinGboCN-S__qZFNKOANjO2jrcAk69cRFq7-=4wX@mail.gmail.com>
Subject: Re: Join me on StumbleUpon!
From: Jay Booth <jaybooth@gmail.com>
To: general@hadoop.apache.org, kiddlee <kiddlee1989@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

lol

On Mon, Feb 21, 2011 at 10:31 AM, kiddlee <kiddlee1989@gmail.com> wrote:
> [StumbleUpon]
> (http://www.stumbleupon.com/to/e/849181372:VISJZDHUSKXUA8LI/invite-4/home=
1/https%253A%252F%252Fwww.stumbleupon.com%252F%253Fpre%253Dinvite%2526u%253=
Duf1q)
> Join for free (http://www.stumbleupon.com/to/e/849181372:VISJZDHUSKXUA8LI=
/invite-4/home2/https%253A%252F%252Fwww.stumbleupon.com%252F%253Fpre%253Din=
vite%2526u%253Duf1q)
>
> I've been discovering great stuff on the web with StumbleUpon and I think=
 you'll love it!
> Join now so we can see and share each other's favorite web pages. =A0It o=
nly takes a minute - check it out!
> - kiddlee
>
>
> StumbleUpon is a discovery engine that finds the best of the web, recomme=
nded just for you.
> Learn more (http://www.stumbleupon.com/to/e/849181372:VISJZDHUSKXUA8LI/in=
vite-4/hf/http%253A%252F%252Fwww.stumbleupon.com%252F)
>
>
>
>
> If you do not wish to receive emails sent by StumbleUpon, please
> click here (http://www.stumbleupon.com/to/e/849181372:VISJZDHUSKXUA8LI/in=
vite-4/oo/__eNrLKCkpsNLXLy8v1ysuKc1NykktLcjP00vOz9XPyy_JTMtMTizJzM8r1ivIKLB=
PzU3MzEnOT0m1DfMM9opy8QgN9o4IdbTw8VQrqSxItfXMK8ssSQVcMCpDHrk%252C)
>
>
> (c) StumbleUpon 2001 - 2011
>
>

From general-return-3109-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 21 21:33:46 2011
Return-Path: <general-return-3109-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67382 invoked from network); 21 Feb 2011 21:33:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Feb 2011 21:33:45 -0000
Received: (qmail 7023 invoked by uid 500); 21 Feb 2011 21:33:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6145 invoked by uid 500); 21 Feb 2011 21:33:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6098 invoked by uid 99); 21 Feb 2011 21:33:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Feb 2011 21:33:35 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [85.214.115.60] (HELO h1349259.stratoserver.net) (85.214.115.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Feb 2011 21:33:30 +0000
Received: from isabel-drost.de ([85.214.115.60] helo=motokokusanaghi.localnet)
	by h1349259.stratoserver.net with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
	(Exim 4.69)
	(envelope-from <isabel@apache.org>)
	id 1PrdNR-0004he-25; Mon, 21 Feb 2011 21:33:05 +0000
From: Isabel Drost <isabel@apache.org>
Reply-To: isabel@apache.org
To: isabel@apache.org
Subject: Call for Presentations Berlin Buzzwords - one more week to go
Date: Mon, 21 Feb 2011 13:19:01 -0800
User-Agent: KMail/1.13.5 (Linux/2.6.32-5-amd64; KDE/4.4.5; x86_64; ; )
MIME-Version: 1.0
Message-Id: <201102211319.02523.isabel@apache.org>
Content-Type: multipart/signed;
  boundary="nextPart6564839.KPnVJzyG1M";
  protocol="application/pgp-signature";
  micalg=pgp-sha1
Content-Transfer-Encoding: 7bit

--nextPart6564839.KPnVJzyG1M
Content-Type: text/plain;
  charset="utf-8"
Content-Transfer-Encoding: quoted-printable

As a reminder: the call for presentations of Berlin Buzzwords will
close next week on Tuesday, March 1st. Submissions on scalable search, data
storage and analysis are all welcome. We are looking for presentations on
the core technologies such as Apache Hadoop, CouchDB, Lucene, Redis,
Voldemort but also talks on interesting use cases and system architectures:

http://berlinbuzzwords.de/content/call-presentations

We are very proud to announce that Doug Cutting as well as Ted Dunning have=
=20
agreed to join us as keynote speakers. Several very interesting talk propos=
als=20
have been submitted already. The list of accepted speakers will be publishe=
d=20
soon after the CfP closes. Tickets are out for sale - don=E2=80=99t wait to=
o long to get=20
your early bird ticket.

In addition we are in the process of drafting some additional packages for
those of you who would like to bring their non-tech spouse to the conference
or need day care facilities for their children. If you are interested in
either package please provide feedback online:

http://berlinbuzzwords.de/content/one-big-family

http://berlinbuzzwords.de/content/children-welcome

Looking forward to seeing you in Berlin this summer,
Isabel

--nextPart6564839.KPnVJzyG1M
Content-Type: application/pgp-signature; name=signature.asc 
Content-Description: This is a digitally signed message part.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEABECAAYFAk1i1sYACgkQPJpCBAwIhbSTmACfc2Q8huWjg+qwzSgXhQ6/uBV9
ooMAn0JbLNm4ogR6WZGsoxlJgpvq4KUw
=Uo0t
-----END PGP SIGNATURE-----

--nextPart6564839.KPnVJzyG1M--

From general-return-3110-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 22 01:55:31 2011
Return-Path: <general-return-3110-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69775 invoked from network); 22 Feb 2011 01:55:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Feb 2011 01:55:31 -0000
Received: (qmail 97316 invoked by uid 500); 22 Feb 2011 01:55:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97208 invoked by uid 500); 22 Feb 2011 01:55:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97200 invoked by uid 99); 22 Feb 2011 01:55:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Feb 2011 01:55:29 +0000
X-ASF-Spam-Status: No, hits=3.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of akimball83@gmail.com designates 209.85.214.50 as permitted sender)
Received: from [209.85.214.50] (HELO mail-bw0-f50.google.com) (209.85.214.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Feb 2011 01:55:22 +0000
Received: by bwz2 with SMTP id 2so1893895bwz.37
        for <general@hadoop.apache.org>; Mon, 21 Feb 2011 17:55:02 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:cc:content-type;
        bh=0EbUmCw8xP1EpiEBxbIVxCkFgYEHDZhcKMW6WOFVV40=;
        b=n6pekDXNIKI7Gvr4Zu94gEZf4W2LUdkrgVytw7d5JebD++2r0IIXDwfbrFEALW+OjM
         3k+ajunacMeDS5TnpUNazVVuqBw/rn76by1LVfvfZSAnxMhM8TKAWAXlc/bDxORazHOJ
         cJnjilVEb2H6zm2CixRWyyep+mTqWe1dm29LY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :cc:content-type;
        b=f+doKXkPuvWrC97xdnLxwhuRnRxdnXMQ79qt+/ADaANSJNAsWwu7JLt0BBtAZ92pUj
         fxPf08b4Gbng2Ewlc5VrQtkXg2aNU73oQgw/kgTKHOkIDFFugv6VcIZ8aVZaweadcuEx
         ZtlEhDR3M+ZMlnN4BaPGVDVZZ6tK9cRZTA3m8=
Received: by 10.204.75.142 with SMTP id y14mr1982311bkj.114.1298339702087;
 Mon, 21 Feb 2011 17:55:02 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.151.215 with HTTP; Mon, 21 Feb 2011 17:54:41 -0800 (PST)
In-Reply-To: <70074956-0C23-4CA2-A414-6F1C2C6453F2@jpl.nasa.gov>
References: <77405974-6771-4604-926B-976240743F3C@mac.com> <AANLkTim80KJ3f14iM6Ltzy7UG613myhN-AiwQ3pCmVdi@mail.gmail.com>
 <AANLkTiknkrWK=GzGMVj4_Z6p5Jn8TQu1P5TkaFSgORJk@mail.gmail.com>
 <AANLkTin0TguLYHi9X0uL3ZF7tCz5xTHBYa7056oKF2Ae@mail.gmail.com>
 <93644D3B-4830-4813-92BE-809738969CF9@apache.org> <AANLkTi=dbGSLhtFA+Rf9LgaX05v45HDLoQDE_2C5ULAf@mail.gmail.com>
 <65311075-A30E-4971-A2BA-C75409C6486E@jpl.nasa.gov> <AANLkTi=XbpVUmg_TpH7eqEn8w7Gcu9KiwtJmKMLa5w5n@mail.gmail.com>
 <FAE19BAA-47A7-4747-BDD8-5AA42C259EB6@jpl.nasa.gov> <AANLkTin2JvOGdZT5g5tmTGeCpD3Qz2soZWcTQtHMsSza@mail.gmail.com>
 <AANLkTi=iGCBrp6+dsK66-+fWF7UoLPSqMOhoNUnjbMHY@mail.gmail.com>
 <AANLkTimXXt3e4fL==fxhU9MJ5v=hPeEa746-3CV4x4Pc@mail.gmail.com>
 <C1C69515-B440-4C3F-9918-9E9767E4804D@jpl.nasa.gov> <AANLkTimBS+KLnknfh8hGzF3s3xdqN5fVjEexy3cpXzE7@mail.gmail.com>
 <5C5BE47B-6159-4678-8455-219CF1F41863@jpl.nasa.gov> <70074956-0C23-4CA2-A414-6F1C2C6453F2@jpl.nasa.gov>
From: Aaron Kimball <akimball83@gmail.com>
Date: Mon, 21 Feb 2011 17:54:41 -0800
Message-ID: <AANLkTi==+bXFytzih-Jsfiv5dfT31zwW5yQDUAgtJqAi@mail.gmail.com>
Subject: Re: [VOTE] Abandon mrunit MapReduce contrib
To: general@hadoop.apache.org
Cc: "Mattmann, Chris A (388J)" <chris.a.mattmann@jpl.nasa.gov>
Content-Type: multipart/alternative; boundary=0016e6d7efd7bda03b049cd54466
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d7efd7bda03b049cd54466
Content-Type: text/plain; charset=ISO-8859-1

I have made some revisions and improvements here too..
- Aaron

On Sat, Feb 19, 2011 at 6:52 PM, Mattmann, Chris A (388J) <
chris.a.mattmann@jpl.nasa.gov> wrote:

> Hi Guys,
>
> I took a pass at the MRUnit proposal [1]. It's far from complete, but just
> letting you know I took a swipe :)
>
> Cheers,
> Chris
>
> [1] http://wiki.apache.org/incubator/MRUnitProposal
>
> On Feb 17, 2011, at 1:52 PM, Mattmann, Chris A (388J) wrote:
>
> > Thanks Patrick.
> >
> > Sounds good. Besides mentors, we need a champion, who must be an ASF
> member. So, either me, you or Nigel can do it. I'm already championing Gora,
> and would be happy to champion MRUnit, but am open to either one of you guys
> doing it too (or even another ASF member). Just let me know. I'll head over
> to the wiki and add my info. Thanks for getting this started Eric+Patrick!
> >
> > Cheers,
> > Chris
> >
> > On Feb 17, 2011, at 1:00 PM, Patrick Hunt wrote:
> >
> >> Chris a page is up (still being created by Eric afaict):
> >> http://wiki.apache.org/incubator/MRUnitProposal
> >>
> >> I took the liberty of listing us as mentors.
> >>
> >> Patrick
> >>
> >> On Thu, Feb 17, 2011 at 11:31 AM, Mattmann, Chris A (388J)
> >> <chris.a.mattmann@jpl.nasa.gov> wrote:
> >>> Hey Guys,
> >>>
> >>> FYI on this: Eric has mentioned he is going to start the Incubator
> proposal for MRUnit. Let's start small and then grow big (as needed). It
> seems like we've achieved enough consensus for the required mentors and
> critical mass to make an MRUnit Incubator proposal and then to have the
> Incubator community weigh in. If that expands to include other testing
> projects/etc., we can address that over the Incubation process, and as
> needed.
> >>>
> >>> Eric: as soon as that wiki page is up, I'd be happy to add my name to
> it as a mentor and /kick the can on this.
> >>>
> >>> Cheers,
> >>> Chris
> >>>
> >>> On Feb 17, 2011, at 11:11 AM, Aaron Kimball wrote:
> >>>
> >>>> The MRUnit community is a specific subset of the Hadoop community:
> Engineers
> >>>> writing Java code running on Hadoop. The Hadoop community also
> includes
> >>>> IT/ops staff who maintain Hadoop clusters, data scientists who use
> tools
> >>>> such as Pig & Hive, as well as those written by the aforementioned
> >>>> engineers, etc.
> >>>>
> >>>> The Hadoop project has long recognized that tools aimed at a specific
> subset
> >>>> of the Hadoop community, with separate release cycles, can more
> successfully
> >>>> reach their aims by splitting into incubator projects. Hive, Pig, and
> HBase,
> >>>> for example, have all gone this path.
> >>>>
> >>>> A "current" version of MRUnit would need to compile against multiple
> >>>> versions of Hadoop itself. This is not possible if it is in the same
> source
> >>>> tree as Hadoop.
> >>>>
> >>>> - Aaron
> >>>>
> >>>> On Thu, Feb 17, 2011 at 5:31 AM, Bernd Fondermann <
> >>>> bernd.fondermann@googlemail.com> wrote:
> >>>>
> >>>>> On Fri, Feb 11, 2011 at 23:10, Aaron Kimball <akimball83@gmail.com>
> wrote:
> >>>>>> The main reason I am interested in removing MRUnit from Hadoop is
> that I
> >>>>>> believe that MRUnit deserves its own release cycle. I think this is
> in
> >>>>> the
> >>>>>> best interest of its users.
> >>>>>
> >>>>> Not in mine, at least. (I'm writing MR unit tests.)
> >>>>> Many projects release more than one product. I'd rather get MRUnit
> >>>>> from the same source where I get my MR from.
> >>>>> Separate release cylcles would be ok for me, though.
> >>>>>
> >>>>>> Perhaps more importantly, access to new features in MRUnit should
> not
> >>>>>> require upgrading one's entire Hadoop deployment; this is a client
> >>>>> library
> >>>>>> that depends only on Hadoop's public APIs.
> >>>>>
> >>>>> +1.
> >>>>>
> >>>>>> My primary concern is to move MRUnit to a place where the community
> can
> >>>>>> derive the most benefit from it. The Apache Incubator could fulfill
> this
> >>>>>> role; given the presence of individuals willing to mentor this
> project, I
> >>>>>> believe this would be a successful way to release MRUnit more
> quickly and
> >>>>>> continue to work to grow the MRUnit community.
> >>>>>
> >>>>> What are your expectations what MRUnit would become, software-wise?
> >>>>> Wouldn't the MRUnit community be largely the same as the Hadoop-MR
> >>>>> community?
> >>>>>
> >>>>> Bernd
> >>>>>
> >>>
> >>>
> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> Chris Mattmann, Ph.D.
> >>> Senior Computer Scientist
> >>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>> Office: 171-266B, Mailstop: 171-246
> >>> Email: chris.a.mattmann@nasa.gov
> >>> WWW:   http://sunset.usc.edu/~mattmann/
> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>> Adjunct Assistant Professor, Computer Science Department
> >>> University of Southern California, Los Angeles, CA 90089 USA
> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>>
> >>>
> >
> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Chris Mattmann, Ph.D.
> > Senior Computer Scientist
> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > Office: 171-266B, Mailstop: 171-246
> > Email: chris.a.mattmann@nasa.gov
> > WWW:   http://sunset.usc.edu/~mattmann/
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Adjunct Assistant Professor, Computer Science Department
> > University of Southern California, Los Angeles, CA 90089 USA
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW:   http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>

--0016e6d7efd7bda03b049cd54466--

From general-return-3111-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 22 03:52:59 2011
Return-Path: <general-return-3111-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 11239 invoked from network); 22 Feb 2011 03:52:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Feb 2011 03:52:59 -0000
Received: (qmail 65290 invoked by uid 500); 22 Feb 2011 03:52:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 64820 invoked by uid 500); 22 Feb 2011 03:52:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 64806 invoked by uid 99); 22 Feb 2011 03:52:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Feb 2011 03:52:53 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Feb 2011 03:52:49 +0000
Received: from dishnailfind-dx.eglbp.corp.yahoo.com (dishnailfind-dx.eglbp.corp.yahoo.com [10.66.77.50])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p1M3qM5A009608
	for <general@hadoop.apache.org>; Mon, 21 Feb 2011 19:52:23 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1298346744;
	bh=nIN+GEgiyIit31hVg3hVzFacWTRpmicpqUtVtFh324I=;
	h=Message-ID:Date:From:MIME-Version:To:Subject:References:
	 In-Reply-To:Content-Type:Content-Transfer-Encoding;
	b=cIgUfhJ/41Rl5S9Xq5hwfAT5o/VYEylx7ZhU1XrnIZC3nPcJhX4ALC+rj+BOBluqu
	 5do6n/4wH4bpGFXXsChxKjpXK8qJ+XVpg3jLQ5rMNChrdZwxxEJFRGqbG+VuOh+/Rk
	 Fr+K7Q7mdAknaka1Ei2NAZ1RKY9RBaEL+6jHglEE=
Message-ID: <4D6332F6.8030200@yahoo-inc.com>
Date: Tue, 22 Feb 2011 09:22:22 +0530
From: Ranjit Mathew <ranjit@yahoo-inc.com>
Organization: Yahoo
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.9.2.13) Gecko/20101207 Lightning/1.0b2 Thunderbird/3.1.7
MIME-Version: 1.0
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: precommit testing gone crazy
References: <784E9D98-E8EE-4DCC-824B-94F63B8973CE@mac.com>
In-Reply-To: <784E9D98-E8EE-4DCC-824B-94F63B8973CE@mac.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On Monday 21 February 2011 11:32 AM, Nigel Daley wrote:
> Apologies.  Looks like the precommit testing control file got corrupted over the weekend and
 > most Patch Available issues got run thru Hudson again.

...and when it does run, it cannot seem to find "/usr/bin/patch":

   https://issues.apache.org/jira/browse/HADOOP-7152

:-/

Ranjit

From general-return-3112-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 22 19:19:41 2011
Return-Path: <general-return-3112-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31400 invoked from network); 22 Feb 2011 19:19:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Feb 2011 19:19:41 -0000
Received: (qmail 26906 invoked by uid 500); 22 Feb 2011 19:19:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26668 invoked by uid 500); 22 Feb 2011 19:19:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26657 invoked by uid 99); 22 Feb 2011 19:19:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Feb 2011 19:19:38 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.220.176] (HELO mail-vx0-f176.google.com) (209.85.220.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Feb 2011 19:19:32 +0000
Received: by vxc41 with SMTP id 41so2303100vxc.35
        for <general@hadoop.apache.org>; Tue, 22 Feb 2011 11:19:11 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:from:content-type:content-transfer-encoding
         :subject:date:message-id:to:mime-version:x-mailer;
        bh=wTfYIJc/uSFdQzYwvtiaG9O6UdkQYLc4iOmxU9QNC8U=;
        b=YzTX1uQEAR8ur4iEtB/qmwjYoq7ox7d5ESmcIWp1ukcFaL2mjrw5abIr5fXFd5Dtnv
         LfeZx+fE/37+MU6muouLc4IOQS+0uTHuZ9r8eZVWzZ3iD9kVtI9A2Y1a/CKvrz8jPQ09
         NISVZ+f9xluckfJOtScs2vcDn1aTraf6Yfl9w=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=from:content-type:content-transfer-encoding:subject:date:message-id
         :to:mime-version:x-mailer;
        b=dl878hF3i+dE63NrM7EbY78RGZTEiUexl1Dk0lC7Tjdk/2u1rd5bIFPIZqafKdBZjP
         iBiN6k0DixYJIjps+GGdLYtRm6dvLWPKkcFYDNnaquiFI+5S3YuWc3FpRC/Y4s9Kpd9N
         KzsnYspbemqstNsrCP5X+7GqNzdvrE7JOQI1I=
Received: by 10.52.159.225 with SMTP id xf1mr4655111vdb.55.1298402351321;
        Tue, 22 Feb 2011 11:19:11 -0800 (PST)
Received: from [10.180.150.31] (h-64-236-128-43.nat.aol.com [64.236.128.43])
        by mx.google.com with ESMTPS id l6sm2506301vcp.38.2011.02.22.11.19.09
        (version=SSLv3 cipher=OTHER);
        Tue, 22 Feb 2011 11:19:10 -0800 (PST)
From: Ian Holsman <ian@holsman.net>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: Please Welcome Todd to Hadoop's PMC
Date: Tue, 22 Feb 2011 14:19:09 -0500
Message-Id: <030F14BB-D7D5-4606-A570-9F3A94132C55@holsman.net>
To: general@hadoop.apache.org
Mime-Version: 1.0 (Apple Message framework v1082)
X-Mailer: Apple Mail (2.1082)

It gives me great pride to announce that Todd has accepted a position on =
Hadoop's PMC.

The Hadoop PMC is tasked with the oversight of Hadoop and its =
sub-projects, as well as voting new committers.

Well Done Todd!

--
Ian Holsman
Ian@Holsman.net
PH: +1-703 879-3128 AOLIM: ianholsman Skype:iholsman

Never explain - your friends do not need it and your enemies will not =
believe you anyway. Elbert Hubbard





From general-return-3113-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 22 19:25:26 2011
Return-Path: <general-return-3113-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33058 invoked from network); 22 Feb 2011 19:25:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Feb 2011 19:25:26 -0000
Received: (qmail 36091 invoked by uid 500); 22 Feb 2011 19:25:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36049 invoked by uid 500); 22 Feb 2011 19:25:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36041 invoked by uid 99); 22 Feb 2011 19:25:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Feb 2011 19:25:24 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Feb 2011 19:25:16 +0000
Received: by vws20 with SMTP id 20so3162880vws.35
        for <general@hadoop.apache.org>; Tue, 22 Feb 2011 11:24:55 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:subject:mime-version:content-type:from
         :in-reply-to:date:cc:content-transfer-encoding:message-id:references
         :to:x-mailer;
        bh=jJzVHocbUECTkYxwXPvxVsO16tDAZd1R0t4l8K5a+k4=;
        b=g75I4CG4W4RUL1Blbbmr6tZtn5/chCZIhj7Yxy5zWVolbKUlXrGza+th2yUfuIxl8/
         ihXgsVaZ+Rz1ZH/ZdOCVxRdodzUfSrAxZAPAQBqKuNzCA9JmrNX9qL1+21+E78Iu2j/K
         /QTeDapiK6t8LyEXLOlUK8ICn8kbn24QtzGa4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=subject:mime-version:content-type:from:in-reply-to:date:cc
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=PeV+cQZsw7Hv5qPsHyIgRncg0Wf5ArY8AFX5CibuADnxukJh/4ik88lTekUy9Znhd3
         gcshyGVDKt8JZa3aVeshT8AFxtg+oibs1j83rL5eGIQxyvbjOeSWOGZ7B7yMfKG+53bk
         f7MWLZStGdxDjiyBujizqM5hnnyOrHFOfc58s=
Received: by 10.52.167.164 with SMTP id zp4mr4581059vdb.305.1298402675525;
        Tue, 22 Feb 2011 11:24:35 -0800 (PST)
Received: from [10.180.150.31] (h-64-236-128-43.nat.aol.com [64.236.128.43])
        by mx.google.com with ESMTPS id u4sm668098vch.36.2011.02.22.11.24.34
        (version=SSLv3 cipher=OTHER);
        Tue, 22 Feb 2011 11:24:34 -0800 (PST)
Subject: Re: Please Welcome Todd Lipcon to Hadoop's PMC
Mime-Version: 1.0 (Apple Message framework v1082)
Content-Type: text/plain; charset=us-ascii
From: Ian Holsman <ian@holsman.net>
In-Reply-To: <030F14BB-D7D5-4606-A570-9F3A94132C55@holsman.net>
Date: Tue, 22 Feb 2011 14:24:33 -0500
Cc: general@hadoop.apache.org
Content-Transfer-Encoding: quoted-printable
Message-Id: <16A6620E-9998-4D45-A163-6E47D3C9F9CC@Holsman.net>
References: <030F14BB-D7D5-4606-A570-9F3A94132C55@holsman.net>
To: Ian Holsman <ian@holsman.net>
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

Sorry for the confusion..=20
I meant Todd Lipcon.

On Feb 22, 2011, at 2:19 PM, Ian Holsman wrote:

> It gives me great pride to announce that Todd has accepted a position =
on Hadoop's PMC.
>=20
> The Hadoop PMC is tasked with the oversight of Hadoop and its =
sub-projects, as well as voting new committers.
>=20
> Well Done Todd!
>=20
> --
> Ian Holsman
> Ian@Holsman.net
> PH: +1-703 879-3128 AOLIM: ianholsman Skype:iholsman
>=20
> Never explain - your friends do not need it and your enemies will not =
believe you anyway. Elbert Hubbard
>=20
>=20
>=20
>=20

--
Ian Holsman
Ian@Holsman.net
PH: +1-703 879-3128 AOLIM: ianholsman Skype:iholsman

We are not afraid of the truth, in fact we plan on taking the truth out =
for a nice meal while we persuade it to adopt our views


From general-return-3114-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 23 05:28:29 2011
Return-Path: <general-return-3114-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76607 invoked from network); 23 Feb 2011 05:28:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Feb 2011 05:28:29 -0000
Received: (qmail 54830 invoked by uid 500); 23 Feb 2011 05:28:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54442 invoked by uid 500); 23 Feb 2011 05:28:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54432 invoked by uid 99); 23 Feb 2011 05:28:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Feb 2011 05:28:23 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.105 as permitted sender)
Received: from [17.148.16.105] (HELO asmtpout030.mac.com) (17.148.16.105)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Feb 2011 05:28:16 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp030.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LH200MX51TCBZ30@asmtp030.mac.com>
 for general@hadoop.apache.org; Tue, 22 Feb 2011 21:27:13 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-02-23_02:2011-02-23,2011-02-23,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1102220212
Subject: Re: [VOTE] Abandon hdfsproxy HDFS contrib
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <FB7E6664-F292-480F-8C4F-2B3CCAB9FFB7@yahoo-inc.com>
Date: Tue, 22 Feb 2011 21:27:13 -0800
Message-id: <D3254862-5A94-4121-9B0B-9AE276D0955C@mac.com>
References: <D97CF74A-147E-4573-B188-2F9C48CFC482@mac.com>
 <AANLkTi=F2eBsACBfjYSVava+VwFcsRpqZdRRN+OHXNRJ@mail.gmail.com>
 <823262F0-EE37-4A0A-8F8C-D9422541D67B@mac.com>
 <0CBAFDBB-AC2A-495D-8936-4DF4D6731B69@Holsman.NET>
 <AANLkTimUhDbTnwn6Hk2oYXFJMts+A=FnL-q-veXqX1m0@mail.gmail.com>
 <F2EC1B1B-A2BB-4FC8-9842-3904A25338EE@gbiv.com>
 <AANLkTikVcGcKHnh9EgUDQr2JGLPvkBvBRgn4my3uJfOc@mail.gmail.com>
 <864B55E7-8A46-458F-8B16-8252A2011835@Holsman.NET>
 <AANLkTikhe6YjUBGY8LMc3OG=b7VXWz7WhyNKGkJRu_7K@mail.gmail.com>
 <FB7E6664-F292-480F-8C4F-2B3CCAB9FFB7@yahoo-inc.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

For closure, this vote fails due to a couple binding -1 votes.

Nige

On Feb 18, 2011, at 4:46 AM, Eric Baldeschwieler wrote:

> Hi Bernd,
> 
> Apache Hadoop is about scale. Most clusters will always be small, but Hadoop is going mainstream precisely because it scales to huge data and cluster sizes. 
> 
> There are lots of systems that work well on 10 node clusters. People select   Hadoop because they are confident that as their business / problem grows, Hadoop can grow with it. 
> 
> ---
> E14 - via iPhone
> 
> On Feb 17, 2011, at 7:25 AM, "Bernd Fondermann" <bernd.fondermann@googlemail.com> wrote:
> 
>> On Thu, Feb 17, 2011 at 14:58, Ian Holsman <hadoop@holsman.net> wrote:
>>> Hi Bernd.
>>> 
>>> On Feb 17, 2011, at 7:43 AM, Bernd Fondermann wrote:
>>>> 
>>>> We have the very unfortunate situation here at Hadoop where Apache
>>>> Hadoop is not the primary and foremost place of Hadoop development.
>>>> Instead, code is developed internally at Yahoo and then contributed in
>>>> (smaller or larger) chunks to Hadoop.
>>> 
>>> This has been the situation in the past,
>>> but as you can see in the last month, this has changed.
>>> 
>>> Yahoo! has publicly committed to move their development into the main code base, and you can see they have started doing this with the 20.100 branch,
>>> and their recent commits to trunk.
>>> Combine this with Nige taking on the 0.22 release branch, (and sheperding it into a stable release) and I think we have are addressing your concerns.
>>> 
>>> They have also started bringing the discussions back on the list, see the recent discussion about Jobtracker-nextgen Arun has re-started in MAPREDUCE-279.
>>> 
>>> I'm not saying it's perfect, but I think the major players understand there is an issue, and they are *ALL* moving in the right direction.
>> 
>> I enthusiastically would like to see your optimism be verified.
>> Maybe I'm misreading the statements issued publicly, but I don't think
>> that this is fully understood. I agree though that it's a move into
>> the right direction.
>> 
>>>> This is open source development upside down.
>>>> It is not ok for people to diff ASF svn against their internal code
>>>> and provide the diff as a patch without reviewing IP first for every
>>>> line of code changed.
>>>> For larger chunks I'd suggest to even go via the Incubator IP clearance process.
>>>> Only then will we force committers to primarily work here in the open
>>>> and return to what I'd consider a healthy project.
>>>> 
>>>> To be honest: Hadoop is in the process of falling apart.
>>>> Contrib Code gets moved out of Apache instead of being maintained here.
>>>> Discussions are seldom consense-driven.
>>>> Release branches stagnate.
>>> 
>>> True. releases do take a long time. This is mainly due to it being extremely hard to test and verify that a release is stable.
>>> It's not enough to just run the thing on 4 machines, you need at least 50 to test some of the major problems. This requires some serious $ for someone to verify.
>> 
>> It has been proposed on the list before, IIRC. Don't know how to get
>> there, but the project seriously needs access to a cluster of this
>> size.
>> 
>>>> Downstream projects like HBase don't get proper support.
>>>> Production setups are made from 3rd party distributions.
>>>> Development is not happening here, but elsewhere behind corporate doors.
>>>> Discussion about future developments are started on corporate blogs (
>>>> http://developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextgen/
>>>> ) instead of on the proper mailing list.
>>>> Hurdles for committing are way too high.
>>>> On the bright side, new committers and PMC members are added, this is
>>>> an improvement.
>>>> 
>>>> I'd suggest to move away from relying on large code dumps from
>>>> corporations, and move back to the ASF-proven "individual committer
>>>> commits on trunk"-model where more committers can get involved.
>>>> If that means not to support high end cluster sizes for some months,
>>>> well, so be it.
>>> 
>>>> Average committers cannot run - e.g. test - on high
>>>> end cluster sizes. If that would mean they cannot participate, then
>>>> the open source project better concentrate on small and medium sized
>>>> cluster instead.
>>> 
>>> 
>>> Well.. that's one approach.. but there are several companies out there who rely on apache's hadoop to power their large clusters, so I'd hate to see hadoop become something that only runs well on
>>> 10-nodes.. as I don't think that will help anyone either.
>> 
>> But only looking at high-end scale doesn't help either.
>> 
>> Lets face the fact that Hadoop is now moving from early adaptors phase
>> into a much broader market. I predict that small to medium sized
>> clusters will be the majority of Hadoop deployments in a few month
>> time. 4000, or even 500 machines is the high-end range. If the open
>> source project Hadoop cannot support those users adequately (without
>> becoming defunct), the committership might be better off to focus on
>> the low-end and medium sized users.
>> 
>> I'm not suggesting to turn away from the handfull (?) of high-end
>> users. They certainly have most valuable input. But also, *they*
>> obviously have the resources in terms of larger clusters and
>> developers to deal with their specific setups. Obviously, they don't
>> need to rely on the open source project to make releases. In fact,
>> they *do* work on their own Hadoop derivatives.
>> All the other users, the hundreds of boring small cluster users, don't
>> have that choice. They *depend* on the open source releases.
>> 
>> Hadoop is an Apache project, to provide HDFS and MR free of charge to
>> the general public. Not only to me - nor to only one or two big
>> companies either.
>> Focus on all the users.
>> 
>> Bernd


From general-return-3115-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 23 05:39:23 2011
Return-Path: <general-return-3115-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91162 invoked from network); 23 Feb 2011 05:39:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Feb 2011 05:39:23 -0000
Received: (qmail 66796 invoked by uid 500); 23 Feb 2011 05:39:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66464 invoked by uid 500); 23 Feb 2011 05:39:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66455 invoked by uid 99); 23 Feb 2011 05:39:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Feb 2011 05:39:18 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.101 as permitted sender)
Received: from [17.148.16.101] (HELO asmtpout026.mac.com) (17.148.16.101)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Feb 2011 05:39:10 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp026.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LH200MDT2C5GR30@asmtp026.mac.com>
 for general@hadoop.apache.org; Tue, 22 Feb 2011 21:38:30 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-02-23_02:2011-02-23,2011-02-23,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1102220215
Subject: Re: [VOTE] Abandon hod Common contrib
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <4D5E84DC.6050200@crs4.it>
Date: Tue, 22 Feb 2011 21:38:30 -0800
Message-id: <D3AD1243-5B2E-4CBC-BEC3-FB0B36A85FB6@mac.com>
References: <AC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com>
 <84B97117-5DA4-4A32-912A-8DBFCE3D063E@linkedin.com>
 <535D05CA-3321-48BA-82C4-C18FC8983C3A@mac.com>
 <AC425E88-8B0E-4982-84FF-D55C7C2C7E3D@apache.org> <4D5E84DC.6050200@crs4.it>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

For closure, this vote fails due to a binding -1 vote.

Nige

On Feb 18, 2011, at 6:40 AM, Simone Leo wrote:

> I am the co-author (with Gianluigi Zanetti) of HADOOP-6369 -- add Grid
> Engine support to HOD. At CRS4 we've been using (our patched version of)
> HOD since 2008 and we still use it in production. We use Hadoop 0.20.2
> since it was released one year ago.
> 
> Simone
> 
> On 02/12/11 06:15, Owen O'Malley wrote:
>> 
>> On Feb 11, 2011, at 6:17 PM, Nigel Daley wrote:
>> 
>>>> a) I don't think hod is actually part of any unit tests, so including
>>>> it would likely only be a burden on the tarball size.
>>> 
>>> Not true.  HOD has python unit tests and is the reason our builds have
>>> dependencies on python.
>> 
>> But Allen's point is that I don't recall ever seeing HOD test failures
>> causing the build to fail.
>> 
>>>> b) The edu community uses this quite extensively, evidenced by the
>>>> topic coming up on the mailing lists at least once every two months
>>>> or so and has for years.  Can't say that about the other contrib
>>>> modules other than the schedulers and streaming.
>>> 
>>> Then they are using old version of Hadoop.  AFAICT HOD does not work
>>> with 0.20 or beyond.
>> 
>> Out of curiosity, what goes wrong? Clearly nothing major has changed in
>> starting up a mapreduce cluster in a very long time.
>> 
>>>> c) The community that does use it has even submitted a patch that
>>>> we've ignored.
>>> 
>>> Which means the committers of this project gave up on it long ago.
>> 
>> There are also some patches on core Hadoop that have been sitting for a
>> long time, so I don't think that is a valid inference.
>> 
>> I would love to hear some of the people who are using HOD speak up and
>> give us their feedback.
>> 
>> -- Owen
> 
> 
> -- 
> Simone Leo
> Data Fusion - Distributed Computing
> CRS4
> POLARIS - Building #1
> Piscina Manna
> I-09010 Pula (CA) - Italy
> e-mail: simone.leo@crs4.it
> http://www.crs4.it


From general-return-3116-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 23 22:08:28 2011
Return-Path: <general-return-3116-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72118 invoked from network); 23 Feb 2011 22:08:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Feb 2011 22:08:27 -0000
Received: (qmail 51714 invoked by uid 500); 23 Feb 2011 22:08:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51505 invoked by uid 500); 23 Feb 2011 22:08:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51490 invoked by uid 99); 23 Feb 2011 22:08:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Feb 2011 22:08:21 +0000
X-ASF-Spam-Status: No, hits=3.7 required=5.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of akimball83@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Feb 2011 22:08:16 +0000
Received: by bwz8 with SMTP id 8so627665bwz.35
        for <multiple recipients>; Wed, 23 Feb 2011 14:07:55 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:from:date:message-id:subject:to
         :content-type;
        bh=kJq3pJdHsHXiov9x7Sr/udyzvPR76O+VmItty+PPOAU=;
        b=LHHdD3MLM/iitNMT/Sj+SMjMrYShcROZ7NlKduy8VmbAT7WZ9/vPuGICQ/97p6E4E7
         vYJwnAnM9davKPsV2or4xw29MUzqveKlOKAq/bLXE/d2PNCmdqbiTKe80a0OGByeU6bQ
         /u8RT0+GSz61H53RF546twOb2c4OM6ILOMGRc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type;
        b=GQ52yNCMBhEo26Uh6DrOeun2i7ri8RqZMqs+x+mP3wkwCgrrVMjrKpFjV/oUrGViEK
         P+QEaoWbz7wOB8XzwFzHs2qkCN5gLiPZWaQDQOcdL+8BY1VpQoQaqZdYYq5V+iabpgdX
         Dp951LuQrUTeyaeCetCwnWQ+Pmw1i45SLhIqU=
Received: by 10.204.123.7 with SMTP id n7mr55603bkr.33.1298498868270; Wed, 23
 Feb 2011 14:07:48 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.151.215 with HTTP; Wed, 23 Feb 2011 14:07:28 -0800 (PST)
From: Aaron Kimball <akimball83@gmail.com>
Date: Wed, 23 Feb 2011 14:07:28 -0800
Message-ID: <AANLkTi=C0rywVY7te3FT6aWZ7m2C5-+mPnWJGjkff5iU@mail.gmail.com>
Subject: March 2011 San Francisco Hadoop User Meetup ("integration")
To: general@hadoop.apache.org, common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502d60cc8d799049cfa5395

--00504502d60cc8d799049cfa5395
Content-Type: text/plain; charset=ISO-8859-1

Hadoop fans,

I'm pleased to announce that the third SF Hadoop meetup will be held
Wednesday, March 9th, from 6pm to 8pm. (We will hopefully continue using the
2nd Wednesday of the month for successive meetups).

This meetup will be hosted by the good folks at Yelp. Their office is at 706
Mission St, San Francisco (3rd and Mission). We have ample space available
for a large gathering of Hadoop enthusiasts.

As per our emerging tradition, we will continue to use the discussion-based
"unconference" format. At the beginning of the meetup we will
collaboratively construct an agenda consisting of several discussion
breakout groups. All participants may propose a topic and volunteer to
facilitate a discussion. All members of the Hadoop community are welcome to
attend. While all Hadoop-related subjects are on topic, I'd like to seed
this month's discussion with the theme of "integration."

Yelp has asked that all attendees RSVP in advance, to comply with their
security policy. Please join the meetup group and RSVP at
http://www.meetup.com/hadoopsf/events/16678757/

Refreshments will be provided.

Regards,
- Aaron Kimball

--00504502d60cc8d799049cfa5395--

From general-return-3117-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 24 01:04:47 2011
Return-Path: <general-return-3117-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 32608 invoked from network); 24 Feb 2011 01:04:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Feb 2011 01:04:47 -0000
Received: (qmail 56360 invoked by uid 500); 24 Feb 2011 01:04:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56259 invoked by uid 500); 24 Feb 2011 01:04:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56251 invoked by uid 99); 24 Feb 2011 01:04:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Feb 2011 01:04:45 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Feb 2011 01:04:39 +0000
Received: by iwr19 with SMTP id 19so33418iwr.35
        for <general@hadoop.apache.org>; Wed, 23 Feb 2011 17:04:19 -0800 (PST)
Received: by 10.231.31.131 with SMTP id y3mr283192ibc.179.1298509459096; Wed,
 23 Feb 2011 17:04:19 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.42.201 with HTTP; Wed, 23 Feb 2011 17:03:59 -0800 (PST)
In-Reply-To: <4D5E84DC.6050200@crs4.it>
References: <AC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com> <84B97117-5DA4-4A32-912A-8DBFCE3D063E@linkedin.com>
 <535D05CA-3321-48BA-82C4-C18FC8983C3A@mac.com> <AC425E88-8B0E-4982-84FF-D55C7C2C7E3D@apache.org>
 <4D5E84DC.6050200@crs4.it>
From: Todd Lipcon <todd@cloudera.com>
Date: Wed, 23 Feb 2011 17:03:59 -0800
Message-ID: <AANLkTikDf_KgEsCPJ+sgr=BpizP6fd9=uXh-vVY=wmHK@mail.gmail.com>
Subject: Re: [VOTE] Abandon hod Common contrib
To: general@hadoop.apache.org
Cc: Simone Leo <simone.leo@crs4.it>
Content-Type: multipart/alternative; boundary=00032557a34e0c0510049cfccbc8

--00032557a34e0c0510049cfccbc8
Content-Type: text/plain; charset=ISO-8859-1

Can any committer with knowledge of HOD please review this patch?

If there are no committers with such knowledge, I would encourage us to
either (a) add a committer to maintain hod, or (b) reconsider the vote to
abandon it as an official contrib. Perhaps Simone and Gianluigi could move
it to a separate incubator project?

-Todd

On Fri, Feb 18, 2011 at 6:40 AM, Simone Leo <simone.leo@crs4.it> wrote:

> I am the co-author (with Gianluigi Zanetti) of HADOOP-6369 -- add Grid
> Engine support to HOD. At CRS4 we've been using (our patched version of)
> HOD since 2008 and we still use it in production. We use Hadoop 0.20.2
> since it was released one year ago.
>
> Simone
>
> On 02/12/11 06:15, Owen O'Malley wrote:
> >
> > On Feb 11, 2011, at 6:17 PM, Nigel Daley wrote:
> >
> >>> a) I don't think hod is actually part of any unit tests, so including
> >>> it would likely only be a burden on the tarball size.
> >>
> >> Not true.  HOD has python unit tests and is the reason our builds have
> >> dependencies on python.
> >
> > But Allen's point is that I don't recall ever seeing HOD test failures
> > causing the build to fail.
> >
> >>> b) The edu community uses this quite extensively, evidenced by the
> >>> topic coming up on the mailing lists at least once every two months
> >>> or so and has for years.  Can't say that about the other contrib
> >>> modules other than the schedulers and streaming.
> >>
> >> Then they are using old version of Hadoop.  AFAICT HOD does not work
> >> with 0.20 or beyond.
> >
> > Out of curiosity, what goes wrong? Clearly nothing major has changed in
> > starting up a mapreduce cluster in a very long time.
> >
> >>> c) The community that does use it has even submitted a patch that
> >>> we've ignored.
> >>
> >> Which means the committers of this project gave up on it long ago.
> >
> > There are also some patches on core Hadoop that have been sitting for a
> > long time, so I don't think that is a valid inference.
> >
> > I would love to hear some of the people who are using HOD speak up and
> > give us their feedback.
> >
> > -- Owen
>
>
> --
> Simone Leo
> Data Fusion - Distributed Computing
> CRS4
> POLARIS - Building #1
> Piscina Manna
> I-09010 Pula (CA) - Italy
> e-mail: simone.leo@crs4.it
> http://www.crs4.it
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--00032557a34e0c0510049cfccbc8--

From general-return-3118-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 24 18:46:17 2011
Return-Path: <general-return-3118-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81779 invoked from network); 24 Feb 2011 18:46:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Feb 2011 18:46:16 -0000
Received: (qmail 42338 invoked by uid 500); 24 Feb 2011 18:46:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42073 invoked by uid 500); 24 Feb 2011 18:46:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42057 invoked by uid 99); 24 Feb 2011 18:46:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Feb 2011 18:46:10 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of binhara@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Feb 2011 18:46:03 +0000
Received: by wwi14 with SMTP id 14so681420wwi.29
        for <general@hadoop.apache.org>; Thu, 24 Feb 2011 10:45:42 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=UUdzCtvlsNYiwVfsa8tMBGe9a2zfTJJEZ5JhdQpXcAw=;
        b=O8pok3zrfJiUuyD//XCUpW/pJJTYjuhEL1CWy9gaA7yNFuyEpyf6zu5J0TSeVfTdkp
         3DLqOdvgUEjYe8eYokzhZFeGEwc3g2EjNTvqQeImeckKVoMZ6MdySuq1RIKlejqqm5+m
         OsIRKdb61Jfk0kXjzjPzADID1svbkA/OrJIyQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=Syi6DVOPys6tkzp5Ol8qNpzWmmI93jgLqSOHNfjgayfl4fAW5ujZoqZOHRNlantjeP
         A4lsK+lGNeW2mfrhGE/qsPUMZEkgHcSOhU9Ib79Og76tpAmBkepb95RmEBGChmKs0qTG
         tqP9VbMdm7TCTo6jlQSQqtliVG0eMdZxCPDQk=
MIME-Version: 1.0
Received: by 10.216.52.143 with SMTP id e15mr6188579wec.44.1298573142316; Thu,
 24 Feb 2011 10:45:42 -0800 (PST)
Received: by 10.216.17.146 with HTTP; Thu, 24 Feb 2011 10:45:42 -0800 (PST)
Date: Thu, 24 Feb 2011 15:45:42 -0300
Message-ID: <AANLkTin+S17z-5LQc8ijxQ0-75e_ujjLa4X+Hu=FD2nd@mail.gmail.com>
Subject: JAVA API to HDFS
From: Alessandro Binhara <binhara@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6dbdf5ddcd652049d0b9e2e

--0016e6dbdf5ddcd652049d0b9e2e
Content-Type: text/plain; charset=ISO-8859-1

 How to copy a file from a HDS to local file system with a JAVA API ?

where i can find a documentation and example about it?


thanks

--0016e6dbdf5ddcd652049d0b9e2e--

From general-return-3119-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 24 18:53:35 2011
Return-Path: <general-return-3119-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83890 invoked from network); 24 Feb 2011 18:53:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Feb 2011 18:53:35 -0000
Received: (qmail 61514 invoked by uid 500); 24 Feb 2011 18:53:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61186 invoked by uid 500); 24 Feb 2011 18:53:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61173 invoked by uid 99); 24 Feb 2011 18:53:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Feb 2011 18:53:31 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Feb 2011 18:53:25 +0000
Received: by wwi14 with SMTP id 14so690846wwi.29
        for <general@hadoop.apache.org>; Thu, 24 Feb 2011 10:53:03 -0800 (PST)
Received: by 10.216.10.208 with SMTP id 58mr1182577wev.14.1298573583579; Thu,
 24 Feb 2011 10:53:03 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.180.70 with HTTP; Thu, 24 Feb 2011 10:52:32 -0800 (PST)
In-Reply-To: <AANLkTin+S17z-5LQc8ijxQ0-75e_ujjLa4X+Hu=FD2nd@mail.gmail.com>
References: <AANLkTin+S17z-5LQc8ijxQ0-75e_ujjLa4X+Hu=FD2nd@mail.gmail.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Thu, 24 Feb 2011 10:52:32 -0800
X-Google-Sender-Auth: q9rPT9o0Y9wlJ88bthFGIvhcLLs
Message-ID: <AANLkTi=k_Prfp8MCLhfKE=-6Bxc6X4bM7HWBZrDq_t6V@mail.gmail.com>
Subject: Re: JAVA API to HDFS
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Please send HDFS specific questions to hdfs-user@hadoop.apache.org
--
=A0 Take care,
Konstantin (Cos) Boudnik

On Thu, Feb 24, 2011 at 10:45, Alessandro Binhara <binhara@gmail.com> wrote=
:
> =A0How to copy a file from a HDS to local file system with a JAVA API ?
>
> where i can find a documentation and example about it?
>
>
> thanks
>

From general-return-3120-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 24 23:11:12 2011
Return-Path: <general-return-3120-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 74361 invoked from network); 24 Feb 2011 23:11:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Feb 2011 23:11:12 -0000
Received: (qmail 21508 invoked by uid 500); 24 Feb 2011 23:11:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21431 invoked by uid 500); 24 Feb 2011 23:11:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21423 invoked by uid 99); 24 Feb 2011 23:11:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Feb 2011 23:11:09 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.161.176] (HELO mail-gx0-f176.google.com) (209.85.161.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Feb 2011 23:11:02 +0000
Received: by gxk20 with SMTP id 20so686433gxk.35
        for <general@hadoop.apache.org>; Thu, 24 Feb 2011 15:10:40 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:from:content-type:content-transfer-encoding
         :subject:date:message-id:to:mime-version:x-mailer;
        bh=xHxuJ392Y4EVLCpi0HgGxWc1IxM9LkOgU2P1uOG00Xg=;
        b=rUDNPyIu8dAfPmLVTzk/XoKlhEr9xGLzEjD1wahPiEQR3vv7xIkNOBFxpayEoc/ssk
         n7yenbL5pGdQlz7dGsbbI7ZIvQNoNNMKc3QlqknTBLGsb0/yIYJE0GyoVuJLv1q//dej
         bMnH53zNqzmHXKj6uiPS082NHvUq2vybfe23Q=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=from:content-type:content-transfer-encoding:subject:date:message-id
         :to:mime-version:x-mailer;
        b=Dm99XL508hqriiGDid5IT6dLfqsUvQ5YR6k+TwO9ceDAqJG4U+FCAqkA4LGsyl8809
         JbN8M/9xYZ+rCRlKCH+6doMssWPMr0m6iOJPKjsjWs5ZotjO2vwQ0CVNZV37DkPc7H8X
         bnncLAFW1P/uUitgYiFWv3Ho+xGdfXDwaCBRk=
Received: by 10.150.140.15 with SMTP id n15mr2733143ybd.80.1298589025918;
        Thu, 24 Feb 2011 15:10:25 -0800 (PST)
Received: from [172.16.1.10] ([12.50.76.194])
        by mx.google.com with ESMTPS id v39sm101299yba.18.2011.02.24.15.10.24
        (version=TLSv1/SSLv3 cipher=OTHER);
        Thu, 24 Feb 2011 15:10:25 -0800 (PST)
From: Ian Holsman <hadoop@holsman.net>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: DRAFT: Giving credit where it is due - how we would like to be acknowledged page
Date: Thu, 24 Feb 2011 18:10:23 -0500
Message-Id: <869583EB-E0F5-447F-BE05-DABFBB0B3A5C@Holsman.NET>
To: general@hadoop.apache.org
Mime-Version: 1.0 (Apple Message framework v1082)
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

Hey guys.

I'd like to create a link on hadoop.apache.org which describes how we =
(Apache Hadoop) would like to be recognized by the vendors & =
applications that make use of our software.

I'm not a wordsmith, and would naturally appreciate your feedback on=20
a) should we do this
b) what it should say

here is my initial stab at it.
--

Apache Hadoop powers hundreds of sites, and the ecosystem of =
applications and vendors that the hadoop platform supports.

If you are part of that ecosystem, we don't ask for license fees or that =
you give us a free lunch when we are in your neck of the woods
but we would appreciate that when you do reference us, that you =
reference us properly.

We have a page describing our trademark policy here - =
http://www.apache.org/foundation/marks/, and if you are holding event =
about Apache Hadoop, you might want to look at =
http://www.apache.org/foundation/marks/events.html.



From general-return-3121-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 25 02:10:44 2011
Return-Path: <general-return-3121-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 52293 invoked from network); 25 Feb 2011 02:10:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Feb 2011 02:10:43 -0000
Received: (qmail 81773 invoked by uid 500); 25 Feb 2011 02:10:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81573 invoked by uid 500); 25 Feb 2011 02:10:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81565 invoked by uid 99); 25 Feb 2011 02:10:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Feb 2011 02:10:41 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mbhandarkar@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Feb 2011 02:10:36 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=PieEoRbkVZ73SW0gqWwfXFskCxC7h4CZPWUT2Byudy6i7dAKbkws52an
   cEkbb6cXUxEckb8F4nfR8iNP/j13txp27W00Z0b7ESdknvzN6WcKNstT8
   cpSIkErq5YrK1Al;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=mbhandarkar@linkedin.com; q=dns/txt;
  s=proddkim; t=1298599833; x=1330135833;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Milind=20Bhandarkar=20<mbhandarkar@linkedin.com>
   |Subject:=20Re:=20DRAFT:=20Giving=20credit=20where=20it
   =20is=20due=20-=20how=20we=20would=20like=20to=20be=0D=0A
   =20acknowledged=20page|Date:=20Fri,=2025=20Feb=202011=200
   2:10:15=20+0000|Message-ID:=20<CE472070-C9CE-40ED-AA7E-43
   8442AA370E@linkedin.com>|To:=20"<general@hadoop.apache.or
   g>"=20<general@hadoop.apache.org>|MIME-Version:=201.0
   |Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<33DD5698C299064193F322E0BA4E2F6F@linkedin
   .com>|In-Reply-To:=20<869583EB-E0F5-447F-BE05-DABFBB0B3A5
   C@Holsman.NET>|References:=20<869583EB-E0F5-447F-BE05-DAB
   FBB0B3A5C@Holsman.NET>;
  bh=WNRKPU8+PmLC4upbXmF14taVq68bYPLWIdgSwYbfBmU=;
  b=qFVveU4G9ATgH9wP3dp43KDwj4A9QwgdVfOetL45XzCaMifaagYZ/zxl
   RclzPJR0pF5ddjnzFObCuWU+thxZIyW8jBpi/Bj6trpV2a57KgIX2HE5X
   q0t8A9ZONfgEpS6;
X-IronPort-AV: E=Sophos;i="4.62,221,1297065600"; 
   d="scan'208";a="20548069"
Received: from ESV4-EXC02.linkedin.biz ([fe80::4d74:48bd:e0bd:13ee]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi id 14.01.0251.000; Thu,
 24 Feb 2011 18:12:00 -0800
From: Milind Bhandarkar <mbhandarkar@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: DRAFT: Giving credit where it is due - how we would like to be
 acknowledged page
Thread-Topic: DRAFT: Giving credit where it is due - how we would like to be
 acknowledged page
Thread-Index: AQHL1Hhe9+QBqsSYpkKeZ7vyyersNpQR/3OA
Date: Fri, 25 Feb 2011 02:10:15 +0000
Message-ID: <CE472070-C9CE-40ED-AA7E-438442AA370E@linkedin.com>
References: <869583EB-E0F5-447F-BE05-DABFBB0B3A5C@Holsman.NET>
In-Reply-To: <869583EB-E0F5-447F-BE05-DABFBB0B3A5C@Holsman.NET>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <33DD5698C299064193F322E0BA4E2F6F@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hey Ian,

is it okay to tweet with #hadoop hashtag or should it be #apachehadoop ?

- milind

On Feb 24, 2011, at 3:10 PM, Ian Holsman wrote:

> Hey guys.
>=20
> I'd like to create a link on hadoop.apache.org which describes how we (Ap=
ache Hadoop) would like to be recognized by the vendors & applications that=
 make use of our software.
>=20
> I'm not a wordsmith, and would naturally appreciate your feedback on=20
> a) should we do this
> b) what it should say
>=20
> here is my initial stab at it.
> --
>=20
> Apache Hadoop powers hundreds of sites, and the ecosystem of applications=
 and vendors that the hadoop platform supports.
>=20
> If you are part of that ecosystem, we don't ask for license fees or that =
you give us a free lunch when we are in your neck of the woods
> but we would appreciate that when you do reference us, that you reference=
 us properly.
>=20
> We have a page describing our trademark policy here - http://www.apache.o=
rg/foundation/marks/, and if you are holding event about Apache Hadoop, you=
 might want to look at http://www.apache.org/foundation/marks/events.html.
>=20
>=20

---
Milind Bhandarkar
mbhandarkar@linkedin.com




From general-return-3122-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 25 07:34:47 2011
Return-Path: <general-return-3122-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85595 invoked from network); 25 Feb 2011 07:34:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Feb 2011 07:34:46 -0000
Received: (qmail 8510 invoked by uid 500); 25 Feb 2011 07:34:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7965 invoked by uid 500); 25 Feb 2011 07:34:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7951 invoked by uid 99); 25 Feb 2011 07:34:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Feb 2011 07:34:40 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.102 as permitted sender)
Received: from [17.148.16.102] (HELO asmtpout027.mac.com) (17.148.16.102)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Feb 2011 07:34:30 +0000
MIME-version: 1.0
Content-type: multipart/alternative;
 boundary="Boundary_(ID_c+4beeT6mJ3IKNo4C1Voyw)"
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp027.mac.com (Oracle Communications Messaging Exchange Server 7u4-18.01
 64bit (built Jul 15 2010)) with ESMTPSA id <0LH5007Q0X05R630@asmtp027.mac.com>
 for general@hadoop.apache.org; Thu, 24 Feb 2011 23:33:42 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-02-25_03:2011-02-25,2011-02-25,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1102240204
From: Nigel Daley <ndaley@mac.com>
Subject: Precommit MapReduce testing
Date: Thu, 24 Feb 2011 23:33:41 -0800
Message-id: <5232178A-5B74-4A24-A09C-FFAEF845C5A7@mac.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

--Boundary_(ID_c+4beeT6mJ3IKNo4C1Voyw)
Content-type: text/plain; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT

I've just turned on precommit testing for MAPREDUCE.  The Hadoop build slaves should work thru most of these in the next 24 hours: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/

In addition, I've hopefully fixed the issue with the precommit testing control file getting corrupted causing all the Patch Available issues to be tested again.

Cheers,
Nige

--Boundary_(ID_c+4beeT6mJ3IKNo4C1Voyw)--

From general-return-3123-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Feb 26 03:56:00 2011
Return-Path: <general-return-3123-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94421 invoked from network); 26 Feb 2011 03:56:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Feb 2011 03:56:00 -0000
Received: (qmail 65090 invoked by uid 500); 26 Feb 2011 03:55:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 64469 invoked by uid 500); 26 Feb 2011 03:55:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 64461 invoked by uid 99); 26 Feb 2011 03:55:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Feb 2011 03:55:53 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Feb 2011 03:55:44 +0000
Received: by yxd5 with SMTP id 5so1366329yxd.35
        for <general@hadoop.apache.org>; Fri, 25 Feb 2011 19:55:22 -0800 (PST)
Received: by 10.150.168.4 with SMTP id q4mr4323866ybe.427.1298692522153;
        Fri, 25 Feb 2011 19:55:22 -0800 (PST)
Received: from tp (c-24-4-185-157.hsd1.ca.comcast.net [24.4.185.157])
        by mx.google.com with ESMTPS id j8sm868917yha.1.2011.02.25.19.55.20
        (version=SSLv3 cipher=OTHER);
        Fri, 25 Feb 2011 19:55:21 -0800 (PST)
Sender: Konstantin Boudnik <cos@boudnik.org>
Date: Fri, 25 Feb 2011 19:55:16 -0800
From: Konstantin Boudnik <cos@apache.org>
To: general@hadoop.apache.org
Subject: Build/test infrastructure
Message-ID: <20110226035516.GA20670@tp>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature"; boundary="MGYHOYXEY6WxJCY8"
Content-Disposition: inline
X-Organization: It's something of 'Cos
X-PGP-Key: http://www.boudnik.org/~cos/pubkey.asc
User-Agent: Mutt/1.5.20 (2009-06-14)
X-Virus-Checked: Checked by ClamAV on apache.org

--MGYHOYXEY6WxJCY8
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Looking at re-occurring build/test-patch problems on hadoop? build machines=
 I
thought of a way to make them:
  a) all the same (configuration, installed software wise)
  b) have an effortless system to run upgrades/updates on all of them in a
  controlled fashion.

I would suggest to create Puppet configs (the exact content to be defined)
which we'll be checked in SCM (e.g. SVN), Whenever a build host's software
is needed to be restored/updated a simple run of Puppet across the machines
or change in config and run of Puppet will do the magic for us.

If there are no objections from the community I can put together some
Puppet recipes which might be evolved as we go.

--=20
Take care,
	Cos
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

After all, it is only the mediocre who are always at their best.
		 Jean Giraudoux

--MGYHOYXEY6WxJCY8
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iF4EAREIAAYFAk1oeaQACgkQenyFlstYjhI43wD/RmdCV2npaSlt5lfDVFd/aejx
JGNCXM02EjgYnoN3aNoA/3giugnaLx7j0CYg52nu8Qo1jNKzortCUUHsVS3iSruL
=5ucp
-----END PGP SIGNATURE-----

--MGYHOYXEY6WxJCY8--

From general-return-3124-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Feb 26 07:49:16 2011
Return-Path: <general-return-3124-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82275 invoked from network); 26 Feb 2011 07:49:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Feb 2011 07:49:15 -0000
Received: (qmail 26815 invoked by uid 500); 26 Feb 2011 07:49:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26325 invoked by uid 500); 26 Feb 2011 07:49:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26317 invoked by uid 99); 26 Feb 2011 07:49:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Feb 2011 07:49:09 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.99 as permitted sender)
Received: from [17.148.16.99] (HELO asmtpout024.mac.com) (17.148.16.99)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Feb 2011 07:49:00 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp024.mac.com (Oracle Communications Messaging Exchange Server 7u4-18.01
 64bit (built Jul 15 2010)) with ESMTPSA id <0LH700LSISBKCQ80@asmtp024.mac.com>
 for general@hadoop.apache.org; Fri, 25 Feb 2011 23:47:45 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-02-26_03:2011-02-25,2011-02-26,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1102250246
Subject: Re: Build/test infrastructure
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <20110226035516.GA20670@tp>
Date: Fri, 25 Feb 2011 23:47:44 -0800
Message-id: <A11EC7A5-8C43-403F-8F39-5FF28C1CB356@mac.com>
References: <20110226035516.GA20670@tp>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

+1.  

Once HADOOP-7106 is committed, I'd like to propose we create a directory at the same level of common/hdfs/mapreduce to hold build (and deploy) type scripts and files.  These would then get branches/tagged with the rest of the release.

Nige

On Feb 25, 2011, at 7:55 PM, Konstantin Boudnik wrote:

> Looking at re-occurring build/test-patch problems on hadoop? build machines I
> thought of a way to make them:
>  a) all the same (configuration, installed software wise)
>  b) have an effortless system to run upgrades/updates on all of them in a
>  controlled fashion.
> 
> I would suggest to create Puppet configs (the exact content to be defined)
> which we'll be checked in SCM (e.g. SVN), Whenever a build host's software
> is needed to be restored/updated a simple run of Puppet across the machines
> or change in config and run of Puppet will do the magic for us.
> 
> If there are no objections from the community I can put together some
> Puppet recipes which might be evolved as we go.
> 
> -- 
> Take care,
> 	Cos
> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> 
> After all, it is only the mediocre who are always at their best.
> 		 Jean Giraudoux


From general-return-3125-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Feb 26 19:15:03 2011
Return-Path: <general-return-3125-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9572 invoked from network); 26 Feb 2011 19:15:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Feb 2011 19:15:03 -0000
Received: (qmail 96710 invoked by uid 500); 26 Feb 2011 19:15:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96597 invoked by uid 500); 26 Feb 2011 19:15:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96589 invoked by uid 99); 26 Feb 2011 19:15:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Feb 2011 19:15:00 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Feb 2011 19:14:53 +0000
Received: by wya21 with SMTP id 21so3288030wya.35
        for <general@hadoop.apache.org>; Sat, 26 Feb 2011 11:14:32 -0800 (PST)
Received: by 10.217.1.198 with SMTP id n48mr3317025wes.59.1298747672091; Sat,
 26 Feb 2011 11:14:32 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.180.70 with HTTP; Sat, 26 Feb 2011 11:14:12 -0800 (PST)
In-Reply-To: <A11EC7A5-8C43-403F-8F39-5FF28C1CB356@mac.com>
References: <20110226035516.GA20670@tp> <A11EC7A5-8C43-403F-8F39-5FF28C1CB356@mac.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Sat, 26 Feb 2011 11:14:12 -0800
X-Google-Sender-Auth: YNBfnNYR9FvbULuKWanZBzd_DTs
Message-ID: <AANLkTikkPrX4zd0H1orp-YZ7iUYox1V9jBt5dfBO46bO@mail.gmail.com>
Subject: Re: Build/test infrastructure
To: general@hadoop.apache.org
Cc: Nigel Daley <ndaley@mac.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Fri, Feb 25, 2011 at 23:47, Nigel Daley <ndaley@mac.com> wrote:
> +1.
>
> Once HADOOP-7106 is committed, I'd like to propose we create a directory =
at the same level of common/hdfs/mapreduce to hold build (and deploy) type =
scripts and files. =A0These would then get branches/tagged with the rest of=
 the release.

That makes sense, although I don't see changes of the host
configurations to happen very often.

Cos

> Nige
>
> On Feb 25, 2011, at 7:55 PM, Konstantin Boudnik wrote:
>
>> Looking at re-occurring build/test-patch problems on hadoop? build machi=
nes I
>> thought of a way to make them:
>> =A0a) all the same (configuration, installed software wise)
>> =A0b) have an effortless system to run upgrades/updates on all of them i=
n a
>> =A0controlled fashion.
>>
>> I would suggest to create Puppet configs (the exact content to be define=
d)
>> which we'll be checked in SCM (e.g. SVN), Whenever a build host's softwa=
re
>> is needed to be restored/updated a simple run of Puppet across the machi=
nes
>> or change in config and run of Puppet will do the magic for us.
>>
>> If there are no objections from the community I can put together some
>> Puppet recipes which might be evolved as we go.
>>
>> --
>> Take care,
>> =A0 =A0 =A0 Cos
>> 2CAC 8312 4870 D885 8616 =A06115 220F 6980 1F27 E622
>>
>> After all, it is only the mediocre who are always at their best.
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Jean Giraudoux
>
>

From general-return-3126-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Feb 26 20:44:46 2011
Return-Path: <general-return-3126-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 40271 invoked from network); 26 Feb 2011 20:44:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Feb 2011 20:44:45 -0000
Received: (qmail 86672 invoked by uid 500); 26 Feb 2011 20:44:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86572 invoked by uid 500); 26 Feb 2011 20:44:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86564 invoked by uid 99); 26 Feb 2011 20:44:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Feb 2011 20:44:43 +0000
X-ASF-Spam-Status: No, hits=3.3 required=5.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Feb 2011 20:44:35 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p1QKhvu1022378
	for <general@hadoop.apache.org>; Sat, 26 Feb 2011 12:43:57 -0800 (PST)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by
 SP2-EX07CAS01.ds.corp.yahoo.com ([98.137.59.37]) with mapi; Sat, 26 Feb 2011
 12:43:57 -0800
From: Eric Yang <eyang@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Sat, 26 Feb 2011 12:43:55 -0800
Subject: Re: Build/test infrastructure
Thread-Topic: Build/test infrastructure
Thread-Index: AcvV6oZGAbCdjxR3RwWlrLjoqtP98AAC1v8N
Message-ID: <C98EA60B.FFAC%eyang@yahoo-inc.com>
In-Reply-To: <AANLkTikkPrX4zd0H1orp-YZ7iUYox1V9jBt5dfBO46bO@mail.gmail.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C98EA60BFFACeyangyahooinccom_"
MIME-Version: 1.0

--_000_C98EA60BFFACeyangyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

We should be very careful about the approach that we chosen for build/packa=
ging.  The current state of hadoop is coupled together due to lack of stand=
ardized RPC format.  Once this issue is cleared, the community will want to=
 split hdfs and m/r into separated projects at some point.  It may be bette=
r to ensure project is modularized, and work from the same svn repository. =
 Maven is great for doing this, and most of the build and scripts can be de=
fined in pom.xml.  Deployment/test server configuration can be pass in from=
 hudson.  We should ensure that build and deployment script do not further =
couple the project.

Regards,
Eric

On 2/26/11 11:14 AM, "Konstantin Boudnik" <cos@apache.org> wrote:

On Fri, Feb 25, 2011 at 23:47, Nigel Daley <ndaley@mac.com> wrote:
> +1.
>
> Once HADOOP-7106 is committed, I'd like to propose we create a directory =
at the same level of common/hdfs/mapreduce to hold build (and deploy) type =
scripts and files.  These would then get branches/tagged with the rest of t=
he release.

That makes sense, although I don't see changes of the host
configurations to happen very often.

Cos

> Nige
>
> On Feb 25, 2011, at 7:55 PM, Konstantin Boudnik wrote:
>
>> Looking at re-occurring build/test-patch problems on hadoop? build machi=
nes I
>> thought of a way to make them:
>>  a) all the same (configuration, installed software wise)
>>  b) have an effortless system to run upgrades/updates on all of them in =
a
>>  controlled fashion.
>>
>> I would suggest to create Puppet configs (the exact content to be define=
d)
>> which we'll be checked in SCM (e.g. SVN), Whenever a build host's softwa=
re
>> is needed to be restored/updated a simple run of Puppet across the machi=
nes
>> or change in config and run of Puppet will do the magic for us.
>>
>> If there are no objections from the community I can put together some
>> Puppet recipes which might be evolved as we go.
>>
>> --
>> Take care,
>>       Cos
>> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>>
>> After all, it is only the mediocre who are always at their best.
>>                Jean Giraudoux
>
>


--_000_C98EA60BFFACeyangyahooinccom_--

From general-return-3127-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Feb 26 22:19:30 2011
Return-Path: <general-return-3127-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72261 invoked from network); 26 Feb 2011 22:19:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Feb 2011 22:19:29 -0000
Received: (qmail 47516 invoked by uid 500); 26 Feb 2011 22:19:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47463 invoked by uid 500); 26 Feb 2011 22:19:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47455 invoked by uid 99); 26 Feb 2011 22:19:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Feb 2011 22:19:27 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Feb 2011 22:19:22 +0000
Received: by wya21 with SMTP id 21so3375546wya.35
        for <general@hadoop.apache.org>; Sat, 26 Feb 2011 14:19:01 -0800 (PST)
Received: by 10.216.17.201 with SMTP id j51mr3441644wej.44.1298758741056; Sat,
 26 Feb 2011 14:19:01 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.180.70 with HTTP; Sat, 26 Feb 2011 14:18:41 -0800 (PST)
In-Reply-To: <C98EA60B.FFAC%eyang@yahoo-inc.com>
References: <AANLkTikkPrX4zd0H1orp-YZ7iUYox1V9jBt5dfBO46bO@mail.gmail.com> <C98EA60B.FFAC%eyang@yahoo-inc.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Sat, 26 Feb 2011 14:18:41 -0800
X-Google-Sender-Auth: 9HdhXLMp0soAsMApVoaNT1P07mo
Message-ID: <AANLkTikAM=EFnErEvszsvUUoZYLb1uRp9DFo8zo4quFG@mail.gmail.com>
Subject: Re: Build/test infrastructure
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

This discussion isn't about build of the product nor about packaging
of it. We are discussing patch validation and snapshot build
infrastructure.

On Sat, Feb 26, 2011 at 12:43, Eric Yang <eyang@yahoo-inc.com> wrote:
> We should be very careful about the approach that we chosen for build/pac=
kaging. =A0The current state of hadoop is coupled together due to lack of s=
tandardized RPC format. =A0Once this issue is cleared, the community will w=
ant to split hdfs and m/r into separated projects at some point. =A0It may =
be better to ensure project is modularized, and work from the same svn repo=
sitory. =A0Maven is great for doing this, and most of the build and scripts=
 can be defined in pom.xml. =A0Deployment/test server configuration can be =
pass in from hudson. =A0We should ensure that build and deployment script d=
o not further couple the project.
>
> Regards,
> Eric
>
> On 2/26/11 11:14 AM, "Konstantin Boudnik" <cos@apache.org> wrote:
>
> On Fri, Feb 25, 2011 at 23:47, Nigel Daley <ndaley@mac.com> wrote:
>> +1.
>>
>> Once HADOOP-7106 is committed, I'd like to propose we create a directory=
 at the same level of common/hdfs/mapreduce to hold build (and deploy) type=
 scripts and files. =A0These would then get branches/tagged with the rest o=
f the release.
>
> That makes sense, although I don't see changes of the host
> configurations to happen very often.
>
> Cos
>
>> Nige
>>
>> On Feb 25, 2011, at 7:55 PM, Konstantin Boudnik wrote:
>>
>>> Looking at re-occurring build/test-patch problems on hadoop? build mach=
ines I
>>> thought of a way to make them:
>>> =A0a) all the same (configuration, installed software wise)
>>> =A0b) have an effortless system to run upgrades/updates on all of them =
in a
>>> =A0controlled fashion.
>>>
>>> I would suggest to create Puppet configs (the exact content to be defin=
ed)
>>> which we'll be checked in SCM (e.g. SVN), Whenever a build host's softw=
are
>>> is needed to be restored/updated a simple run of Puppet across the mach=
ines
>>> or change in config and run of Puppet will do the magic for us.
>>>
>>> If there are no objections from the community I can put together some
>>> Puppet recipes which might be evolved as we go.
>>>
>>> --
>>> Take care,
>>> =A0 =A0 =A0 Cos
>>> 2CAC 8312 4870 D885 8616 =A06115 220F 6980 1F27 E622
>>>
>>> After all, it is only the mediocre who are always at their best.
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Jean Giraudoux
>>
>>
>
>

From general-return-3128-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Feb 27 00:04:27 2011
Return-Path: <general-return-3128-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5412 invoked from network); 27 Feb 2011 00:04:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Feb 2011 00:04:27 -0000
Received: (qmail 92994 invoked by uid 500); 27 Feb 2011 00:04:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92794 invoked by uid 500); 27 Feb 2011 00:04:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 92786 invoked by uid 99); 27 Feb 2011 00:04:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Feb 2011 00:04:25 +0000
X-ASF-Spam-Status: No, hits=3.3 required=5.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Feb 2011 00:04:16 +0000
Received: from SP2-EX07CAS05.ds.corp.yahoo.com (sp2-ex07cas05.corp.sp2.yahoo.com [98.137.59.39])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p1R03YLr065474
	for <general@hadoop.apache.org>; Sat, 26 Feb 2011 16:03:35 -0800 (PST)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by
 SP2-EX07CAS05.ds.corp.yahoo.com ([98.137.59.39]) with mapi; Sat, 26 Feb 2011
 16:03:34 -0800
From: Eric Yang <eyang@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Sat, 26 Feb 2011 16:03:33 -0800
Subject: Re: Build/test infrastructure
Thread-Topic: Build/test infrastructure
Thread-Index: AcvWA0CGfgEVAKH8TNSkuin505UthQADoUtj
Message-ID: <C98ED4D5.FFB4%eyang@yahoo-inc.com>
In-Reply-To: <AANLkTikAM=EFnErEvszsvUUoZYLb1uRp9DFo8zo4quFG@mail.gmail.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C98ED4D5FFB4eyangyahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C98ED4D5FFB4eyangyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

The proposed test automation process hasn't been thought through.   Apache =
Hudson has been setup to trigger patch builds, and setup pre-commit test en=
vironment.  Unfortunately, the current setup needs refinement with proper s=
ource code setup to make the builds working again.  Ideally, the test cycle=
 have a commit build which runs simple unit tests, and a secondary build (e=
very 24 hours) to run more through tests on multiple machine setup.  The te=
st cluster should be cleansed after every secondary build, and ideally this=
 is done in a sandbox approach.  However, I don't think bring in puppet env=
ironment setup is making the test system reproducible.  Consequently, it ma=
y be better to have the cluster test setup as part of scripts in maven inte=
gration test phase.  This will enable any hadoop developer to setup his own=
 test cluster without setup puppet master.  I am not fixated on build and p=
ackaging only, but express my opinions on improving build system and making=
 the system easier to reproduce.

Regards,
Eric

On 2/26/11 2:18 PM, "Konstantin Boudnik" <cos@apache.org> wrote:

This discussion isn't about build of the product nor about packaging
of it. We are discussing patch validation and snapshot build
infrastructure.

On Sat, Feb 26, 2011 at 12:43, Eric Yang <eyang@yahoo-inc.com> wrote:
> We should be very careful about the approach that we chosen for build/pac=
kaging.  The current state of hadoop is coupled together due to lack of sta=
ndardized RPC format.  Once this issue is cleared, the community will want =
to split hdfs and m/r into separated projects at some point.  It may be bet=
ter to ensure project is modularized, and work from the same svn repository=
.  Maven is great for doing this, and most of the build and scripts can be =
defined in pom.xml.  Deployment/test server configuration can be pass in fr=
om hudson.  We should ensure that build and deployment script do not furthe=
r couple the project.
>
> Regards,
> Eric
>
> On 2/26/11 11:14 AM, "Konstantin Boudnik" <cos@apache.org> wrote:
>
> On Fri, Feb 25, 2011 at 23:47, Nigel Daley <ndaley@mac.com> wrote:
>> +1.
>>
>> Once HADOOP-7106 is committed, I'd like to propose we create a directory=
 at the same level of common/hdfs/mapreduce to hold build (and deploy) type=
 scripts and files.  These would then get branches/tagged with the rest of =
the release.
>
> That makes sense, although I don't see changes of the host
> configurations to happen very often.
>
> Cos
>
>> Nige
>>
>> On Feb 25, 2011, at 7:55 PM, Konstantin Boudnik wrote:
>>
>>> Looking at re-occurring build/test-patch problems on hadoop? build mach=
ines I
>>> thought of a way to make them:
>>>  a) all the same (configuration, installed software wise)
>>>  b) have an effortless system to run upgrades/updates on all of them in=
 a
>>>  controlled fashion.
>>>
>>> I would suggest to create Puppet configs (the exact content to be defin=
ed)
>>> which we'll be checked in SCM (e.g. SVN), Whenever a build host's softw=
are
>>> is needed to be restored/updated a simple run of Puppet across the mach=
ines
>>> or change in config and run of Puppet will do the magic for us.
>>>
>>> If there are no objections from the community I can put together some
>>> Puppet recipes which might be evolved as we go.
>>>
>>> --
>>> Take care,
>>>       Cos
>>> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>>>
>>> After all, it is only the mediocre who are always at their best.
>>>                Jean Giraudoux
>>
>>
>
>


--_000_C98ED4D5FFB4eyangyahooinccom_--

From general-return-3129-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Feb 27 00:34:49 2011
Return-Path: <general-return-3129-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 26231 invoked from network); 27 Feb 2011 00:34:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Feb 2011 00:34:49 -0000
Received: (qmail 6560 invoked by uid 500); 27 Feb 2011 00:34:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6447 invoked by uid 500); 27 Feb 2011 00:34:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6439 invoked by uid 99); 27 Feb 2011 00:34:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Feb 2011 00:34:47 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Feb 2011 00:34:41 +0000
Received: by iwr19 with SMTP id 19so2772089iwr.35
        for <general@hadoop.apache.org>; Sat, 26 Feb 2011 16:34:20 -0800 (PST)
Received: by 10.42.178.135 with SMTP id bm7mr2876745icb.158.1298766860570;
        Sat, 26 Feb 2011 16:34:20 -0800 (PST)
Received: from tp (c-24-4-185-157.hsd1.ca.comcast.net [24.4.185.157])
        by mx.google.com with ESMTPS id gy41sm2596588ibb.11.2011.02.26.16.34.18
        (version=SSLv3 cipher=OTHER);
        Sat, 26 Feb 2011 16:34:19 -0800 (PST)
Sender: Konstantin Boudnik <cos@boudnik.org>
Date: Sat, 26 Feb 2011 16:34:15 -0800
From: Konstantin Boudnik <cos@apache.org>
To: general@hadoop.apache.org
Subject: Re: Build/test infrastructure
Message-ID: <20110227003415.GA9036@tp>
References: <AANLkTikAM=EFnErEvszsvUUoZYLb1uRp9DFo8zo4quFG@mail.gmail.com>
 <C98ED4D5.FFB4%eyang@yahoo-inc.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature"; boundary="5mCyUwZo2JvN/JJP"
Content-Disposition: inline
In-Reply-To: <C98ED4D5.FFB4%eyang@yahoo-inc.com>
X-Organization: It's something of 'Cos
X-PGP-Key: http://www.boudnik.org/~cos/pubkey.asc
User-Agent: Mutt/1.5.20 (2009-06-14)

--5mCyUwZo2JvN/JJP
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Apparently you are talking about something else, but I will bite...

On Sat, Feb 26, 2011 at 04:03PM, Eric Yang wrote:
> The proposed test automation process hasn't been thought through.   Apache
> Hudson has been setup to trigger patch builds, and setup pre-commit test
> environment.  Unfortunately, the current setup needs refinement with prop=
er
> source code setup to make the builds working again.  Ideally, the test cy=
cle
> have a commit build which runs simple unit tests, and a secondary build
> (every 24 hours) to run more through tests on multiple machine setup.  The
> test cluster should be cleansed after every secondary build, and ideally

We don't have a test cluster for Apache Hadoop validation. All I am focusing
on is build and patch validation infrastructure.

> this is done in a sandbox approach.  However, I don't think bring in pupp=
et
> environment setup is making the test system reproducible.  Consequently, =
it

If a specialized and highly scalable host configuration system such as Pupp=
et
doesn't guarantee configuration reproducibility then I am not sure what else
will. Say, Y! uses that proprietary Igor environment exactly for these
purposes but of course it is highly coupled with yinst format and can't be
used anywhere else.

> may be better to have the cluster test setup as part of scripts in maven
> integration test phase. This will enable any hadoop developer to setup his

Doing deployment from a build system is certainly possible, but is suboptim=
al
because it pollutes the build with HW/OS details, deployment scripts and su=
ch.
Besides, last time I've checked Hadoop was built by Ant.

> own test cluster without setup puppet master. I am not fixated on build a=
nd

You don't need to setup puppet muster in order to bounce a node. Puppet wor=
ks
i a client-only mode just as perfect.=20

Cos

> packaging only, but express my opinions on improving build system and mak=
ing
> the system easier to reproduce.
>=20
> Regards,
> Eric
>=20
> On 2/26/11 2:18 PM, "Konstantin Boudnik" <cos@apache.org> wrote:
>=20
> This discussion isn't about build of the product nor about packaging
> of it. We are discussing patch validation and snapshot build
> infrastructure.
>=20
> On Sat, Feb 26, 2011 at 12:43, Eric Yang <eyang@yahoo-inc.com> wrote:
> > We should be very careful about the approach that we chosen for
> > build/packaging.  The current state of hadoop is coupled together due to
> > lack of standardized RPC format.  Once this issue is cleared, the
> > community will want to split hdfs and m/r into separated projects at so=
me
> > point.  It may be better to ensure project is modularized, and work from
> > the same svn repository.  Maven is great for doing this, and most of the
> > build and scripts can be defined in pom.xml.  Deployment/test server
> > configuration can be pass in from hudson.  We should ensure that build =
and
> > deployment script do not further couple the project.
> >
> > Regards,
> > Eric
> >
> > On 2/26/11 11:14 AM, "Konstantin Boudnik" <cos@apache.org> wrote:
> >
> > On Fri, Feb 25, 2011 at 23:47, Nigel Daley <ndaley@mac.com> wrote:
> >> +1.
> >>
> >> Once HADOOP-7106 is committed, I'd like to propose we create a directo=
ry at the same level of common/hdfs/mapreduce to hold build (and deploy) ty=
pe scripts and files.  These would then get branches/tagged with the rest o=
f the release.
> >
> > That makes sense, although I don't see changes of the host
> > configurations to happen very often.
> >
> > Cos
> >
> >> Nige
> >>
> >> On Feb 25, 2011, at 7:55 PM, Konstantin Boudnik wrote:
> >>
> >>> Looking at re-occurring build/test-patch problems on hadoop? build ma=
chines I
> >>> thought of a way to make them:
> >>>  a) all the same (configuration, installed software wise)
> >>>  b) have an effortless system to run upgrades/updates on all of them =
in a
> >>>  controlled fashion.
> >>>
> >>> I would suggest to create Puppet configs (the exact content to be def=
ined)
> >>> which we'll be checked in SCM (e.g. SVN), Whenever a build host's sof=
tware
> >>> is needed to be restored/updated a simple run of Puppet across the ma=
chines
> >>> or change in config and run of Puppet will do the magic for us.
> >>>
> >>> If there are no objections from the community I can put together some
> >>> Puppet recipes which might be evolved as we go.
> >>>
> >>> --
> >>> Take care,
> >>>       Cos
> >>> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> >>>
> >>> After all, it is only the mediocre who are always at their best.
> >>>                Jean Giraudoux
> >>
> >>
> >
> >
>=20

--5mCyUwZo2JvN/JJP
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iF4EAREIAAYFAk1pnAcACgkQenyFlstYjhJ+swD+LEIMMXnUeJkqryj3nMPBHAbz
kbHT9nYI42AZ+RNUXBMBAMwmjgKCDAQpJ+oicSfLAQNtGNI/tJl5WCgyrr5aonSq
=98Qk
-----END PGP SIGNATURE-----

--5mCyUwZo2JvN/JJP--

From general-return-3130-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Feb 27 01:39:42 2011
Return-Path: <general-return-3130-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44763 invoked from network); 27 Feb 2011 01:39:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Feb 2011 01:39:42 -0000
Received: (qmail 28761 invoked by uid 500); 27 Feb 2011 01:39:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28621 invoked by uid 500); 27 Feb 2011 01:39:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28613 invoked by uid 99); 27 Feb 2011 01:39:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Feb 2011 01:39:40 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Feb 2011 01:39:32 +0000
Received: from SP2-EX07CAS04.ds.corp.yahoo.com (sp2-ex07cas04.corp.sp2.yahoo.com [98.137.59.5])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p1R1cuLq083310
	for <general@hadoop.apache.org>; Sat, 26 Feb 2011 17:38:57 -0800 (PST)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by
 SP2-EX07CAS04.ds.corp.yahoo.com ([98.137.59.5]) with mapi; Sat, 26 Feb 2011
 17:38:56 -0800
From: Eric Yang <eyang@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Sat, 26 Feb 2011 17:38:54 -0800
Subject: Re: Build/test infrastructure
Thread-Topic: Build/test infrastructure
Thread-Index: AcvWFjkKcwTDFM6BTa6TPdanYpa6FAACN6lk
Message-ID: <C98EEB2E.FFBA%eyang@yahoo-inc.com>
In-Reply-To: <20110227003415.GA9036@tp>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

On 2/26/11 4:34 PM, "Konstantin Boudnik" <cos@apache.org> wrote:

> Apparently you are talking about something else, but I will bite...
>=20
> On Sat, Feb 26, 2011 at 04:03PM, Eric Yang wrote:
>> The proposed test automation process hasn't been thought through.   Apac=
he
>> Hudson has been setup to trigger patch builds, and setup pre-commit test
>> environment.  Unfortunately, the current setup needs refinement with pro=
per
>> source code setup to make the builds working again.  Ideally, the test c=
ycle
>> have a commit build which runs simple unit tests, and a secondary build
>> (every 24 hours) to run more through tests on multiple machine setup.  T=
he
>> test cluster should be cleansed after every secondary build, and ideally
>=20
> We don't have a test cluster for Apache Hadoop validation. All I am focus=
ing
> on is build and patch validation infrastructure.

If the plan is using puppet agent without puppet master for configuring the
system locally to test patch builds.  It is probably using the wrong tool
for the job.  The value of puppet is to be able to configure heterogeneous
services across machines in a consistent manner.  Is there plan to deploy
multiple services across machines?  If the purpose is using puppet for
config templates, ant or maven can do the job equally well.

> Doing deployment from a build system is certainly possible, but is subopt=
imal
> because it pollutes the build with HW/OS details, deployment scripts and =
such.
> Besides, last time I've checked Hadoop was built by Ant.

Deploy to remote machine can be as simple as scp tarball, extra, apply
template, and run it.  None of this requires puppet.  Instead of ant +
puppet combination, the patch test build structure could be simplified by
using maven + shell scripts.

Regards,
Eric

> You don't need to setup puppet muster in order to bounce a node. Puppet w=
orks
> i a client-only mode just as perfect.
>=20
> Cos
>=20

>> packaging only, but express my opinions on improving build system and ma=
king
>> the system easier to reproduce.
>>=20
>> Regards,
>> Eric
>>=20
>> On 2/26/11 2:18 PM, "Konstantin Boudnik" <cos@apache.org> wrote:
>>=20
>> This discussion isn't about build of the product nor about packaging
>> of it. We are discussing patch validation and snapshot build
>> infrastructure.
>>=20
>> On Sat, Feb 26, 2011 at 12:43, Eric Yang <eyang@yahoo-inc.com> wrote:
>>> We should be very careful about the approach that we chosen for
>>> build/packaging.  The current state of hadoop is coupled together due t=
o
>>> lack of standardized RPC format.  Once this issue is cleared, the
>>> community will want to split hdfs and m/r into separated projects at so=
me
>>> point.  It may be better to ensure project is modularized, and work fro=
m
>>> the same svn repository.  Maven is great for doing this, and most of th=
e
>>> build and scripts can be defined in pom.xml.  Deployment/test server
>>> configuration can be pass in from hudson.  We should ensure that build =
and
>>> deployment script do not further couple the project.
>>>=20
>>> Regards,
>>> Eric
>>>=20
>>> On 2/26/11 11:14 AM, "Konstantin Boudnik" <cos@apache.org> wrote:
>>>=20
>>> On Fri, Feb 25, 2011 at 23:47, Nigel Daley <ndaley@mac.com> wrote:
>>>> +1.
>>>>=20
>>>> Once HADOOP-7106 is committed, I'd like to propose we create a directo=
ry at
>>>> the same level of common/hdfs/mapreduce to hold build (and deploy) typ=
e
>>>> scripts and files.  These would then get branches/tagged with the rest=
 of
>>>> the release.
>>>=20
>>> That makes sense, although I don't see changes of the host
>>> configurations to happen very often.
>>>=20
>>> Cos
>>>=20
>>>> Nige
>>>>=20
>>>> On Feb 25, 2011, at 7:55 PM, Konstantin Boudnik wrote:
>>>>=20
>>>>> Looking at re-occurring build/test-patch problems on hadoop? build
>>>>> machines I
>>>>> thought of a way to make them:
>>>>>  a) all the same (configuration, installed software wise)
>>>>>  b) have an effortless system to run upgrades/updates on all of them =
in a
>>>>>  controlled fashion.
>>>>>=20
>>>>> I would suggest to create Puppet configs (the exact content to be def=
ined)
>>>>> which we'll be checked in SCM (e.g. SVN), Whenever a build host's sof=
tware
>>>>> is needed to be restored/updated a simple run of Puppet across the
>>>>> machines
>>>>> or change in config and run of Puppet will do the magic for us.
>>>>>=20
>>>>> If there are no objections from the community I can put together some
>>>>> Puppet recipes which might be evolved as we go.
>>>>>=20
>>>>> --
>>>>> Take care,
>>>>>       Cos
>>>>> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>>>>>=20
>>>>> After all, it is only the mediocre who are always at their best.
>>>>>                Jean Giraudoux
>>>>=20
>>>>=20
>>>=20
>>>=20
>>=20


From general-return-3131-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Feb 27 03:11:18 2011
Return-Path: <general-return-3131-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 74805 invoked from network); 27 Feb 2011 03:11:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Feb 2011 03:11:17 -0000
Received: (qmail 60117 invoked by uid 500); 27 Feb 2011 03:11:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59636 invoked by uid 500); 27 Feb 2011 03:11:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59610 invoked by uid 99); 27 Feb 2011 03:11:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Feb 2011 03:11:13 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.83.50] (HELO mail-gw0-f50.google.com) (74.125.83.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Feb 2011 03:11:06 +0000
Received: by gwaa20 with SMTP id a20so1976727gwa.37
        for <general@hadoop.apache.org>; Sat, 26 Feb 2011 19:10:44 -0800 (PST)
Received: by 10.150.49.14 with SMTP id w14mr5575760ybw.241.1298776244331;
        Sat, 26 Feb 2011 19:10:44 -0800 (PST)
Received: from tp (c-24-4-185-157.hsd1.ca.comcast.net [24.4.185.157])
        by mx.google.com with ESMTPS id l2sm1455197ybn.3.2011.02.26.19.10.41
        (version=SSLv3 cipher=OTHER);
        Sat, 26 Feb 2011 19:10:42 -0800 (PST)
Sender: Konstantin Boudnik <cos@boudnik.org>
Date: Sat, 26 Feb 2011 19:10:38 -0800
From: Konstantin Boudnik <cos@apache.org>
To: general@hadoop.apache.org
Subject: Re: Build/test infrastructure
Message-ID: <20110227031037.GA14579@tp>
References: <20110227003415.GA9036@tp>
 <C98EEB2E.FFBA%eyang@yahoo-inc.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature"; boundary="pWyiEgJYm5f9v55/"
Content-Disposition: inline
In-Reply-To: <C98EEB2E.FFBA%eyang@yahoo-inc.com>
X-Organization: It's something of 'Cos
X-PGP-Key: http://www.boudnik.org/~cos/pubkey.asc
User-Agent: Mutt/1.5.20 (2009-06-14)
X-Virus-Checked: Checked by ClamAV on apache.org

--pWyiEgJYm5f9v55/
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Sat, Feb 26, 2011 at 05:38PM, Eric Yang wrote:
> On 2/26/11 4:34 PM, "Konstantin Boudnik" <cos@apache.org> wrote:
>=20
> > Apparently you are talking about something else, but I will bite...
> >=20
> > On Sat, Feb 26, 2011 at 04:03PM, Eric Yang wrote:
> >> The proposed test automation process hasn't been thought through.   Ap=
ache
> >> Hudson has been setup to trigger patch builds, and setup pre-commit te=
st
> >> environment.  Unfortunately, the current setup needs refinement with p=
roper
> >> source code setup to make the builds working again.  Ideally, the test=
 cycle
> >> have a commit build which runs simple unit tests, and a secondary build
> >> (every 24 hours) to run more through tests on multiple machine setup. =
 The
> >> test cluster should be cleansed after every secondary build, and ideal=
ly
> >=20
> > We don't have a test cluster for Apache Hadoop validation. All I am foc=
using
> > on is build and patch validation infrastructure.
>=20
> If the plan is using puppet agent without puppet master for configuring t=
he
> system locally to test patch builds.  It is probably using the wrong tool
> for the job.  The value of puppet is to be able to configure heterogeneous
> services across machines in a consistent manner.  Is there plan to deploy

This is simply not the only value of the tool. It allows to maintain OS
configurations and system packages installation as easy for 1 host as for 1=
000
of them. Here's one of a many examples
    http://hstack.org/hstack-automated-deployment-using-puppet/
BTW, Puppet and Chef recipes are very widely used by all sorts of Ops and
cluster management companies. Perhaps, Maven and shell too - I'm not in a
position to make a judgement call. I'll let Y! Grid Ops to comment on it -
they know everything about sizable clusters configuration management and to=
ols
for the job.

> multiple services across machines?  If the purpose is using puppet for
> config templates, ant or maven can do the job equally well.

Are you suggesting that it is easier to install patch and gcc packages of
version X.X.Z from a Maven build than from Puppet or Chef? If so - please c=
ut
such a patch for the community to review. That'd be great a great
contribution!

Furthermore, my Puppet knowledge is very limited and I am for sure no expert
in maven. I have some concern however:
  - how to provide privileged access
  - how and where to store host configurations (i.e. packages names, versio=
ns,
    which are gonna be different for difference OSes)
  - how to do native packages (see above example) and native dependency
    management from maven? With shell scripting?
  - how to maintain such a construct?
I can continue for a long time, but I'd rather want to solve an issue of
managing build host configurations/package sets in a most efficient and
sustainable manner.

In a properly designed CI system build shouldn't be responsible to configure
its operation environment. It might and should check if everything is in pl=
ace
(and crash/report accordingly). But if my Ant script goes around to downloa=
d,
install and god forbids compiles some chunks of my OS I soon will end up wi=
th
an elegance of Python or some such.

> > Doing deployment from a build system is certainly possible, but is subo=
ptimal
> > because it pollutes the build with HW/OS details, deployment scripts an=
d such.
> > Besides, last time I've checked Hadoop was built by Ant.
>=20
> Deploy to remote machine can be as simple as scp tarball, extra, apply
> template, and run it.  None of this requires puppet.  Instead of ant +
> puppet combination, the patch test build structure could be simplified by
> using maven + shell scripts.

Sorry, but Maven + shell script can be called simplification only in a pipe
dream ;) Maven is a build tool. A relatively good one perhaps, but just a
build tool. Certainly everything can be done with a combination of a shell
scripting + tar balls and a little SSH sugar topping. But I'd rather use a
accurately designed and supported tool (Puppet, Chef, etc.).

And BTW - Hadoop builds aren't maven'ized yet. Which renders most of the
argument a time waste until that problem is solved.

At any rate, HADOOP-7157 is the JIRA for this. Please comment on it.

Cos

> Regards,
> Eric
>=20
> > You don't need to setup puppet muster in order to bounce a node. Puppet=
 works
> > i a client-only mode just as perfect.
> >=20
> > Cos
> >=20
>=20
> >> packaging only, but express my opinions on improving build system and =
making
> >> the system easier to reproduce.
> >>=20
> >> Regards,
> >> Eric
> >>=20
> >> On 2/26/11 2:18 PM, "Konstantin Boudnik" <cos@apache.org> wrote:
> >>=20
> >> This discussion isn't about build of the product nor about packaging
> >> of it. We are discussing patch validation and snapshot build
> >> infrastructure.
> >>=20
> >> On Sat, Feb 26, 2011 at 12:43, Eric Yang <eyang@yahoo-inc.com> wrote:
> >>> We should be very careful about the approach that we chosen for
> >>> build/packaging.  The current state of hadoop is coupled together due=
 to
> >>> lack of standardized RPC format.  Once this issue is cleared, the
> >>> community will want to split hdfs and m/r into separated projects at =
some
> >>> point.  It may be better to ensure project is modularized, and work f=
rom
> >>> the same svn repository.  Maven is great for doing this, and most of =
the
> >>> build and scripts can be defined in pom.xml.  Deployment/test server
> >>> configuration can be pass in from hudson.  We should ensure that buil=
d and
> >>> deployment script do not further couple the project.
> >>>=20
> >>> Regards,
> >>> Eric
> >>>=20
> >>> On 2/26/11 11:14 AM, "Konstantin Boudnik" <cos@apache.org> wrote:
> >>>=20
> >>> On Fri, Feb 25, 2011 at 23:47, Nigel Daley <ndaley@mac.com> wrote:
> >>>> +1.
> >>>>=20
> >>>> Once HADOOP-7106 is committed, I'd like to propose we create a direc=
tory at
> >>>> the same level of common/hdfs/mapreduce to hold build (and deploy) t=
ype
> >>>> scripts and files.  These would then get branches/tagged with the re=
st of
> >>>> the release.
> >>>=20
> >>> That makes sense, although I don't see changes of the host
> >>> configurations to happen very often.
> >>>=20
> >>> Cos
> >>>=20
> >>>> Nige
> >>>>=20
> >>>> On Feb 25, 2011, at 7:55 PM, Konstantin Boudnik wrote:
> >>>>=20
> >>>>> Looking at re-occurring build/test-patch problems on hadoop? build
> >>>>> machines I
> >>>>> thought of a way to make them:
> >>>>>  a) all the same (configuration, installed software wise)
> >>>>>  b) have an effortless system to run upgrades/updates on all of the=
m in a
> >>>>>  controlled fashion.
> >>>>>=20
> >>>>> I would suggest to create Puppet configs (the exact content to be d=
efined)
> >>>>> which we'll be checked in SCM (e.g. SVN), Whenever a build host's s=
oftware
> >>>>> is needed to be restored/updated a simple run of Puppet across the
> >>>>> machines
> >>>>> or change in config and run of Puppet will do the magic for us.
> >>>>>=20
> >>>>> If there are no objections from the community I can put together so=
me
> >>>>> Puppet recipes which might be evolved as we go.
> >>>>>=20
> >>>>> --
> >>>>> Take care,
> >>>>>       Cos
> >>>>> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> >>>>>=20
> >>>>> After all, it is only the mediocre who are always at their best.
> >>>>>                Jean Giraudoux
> >>>>=20
> >>>>=20
> >>>=20
> >>>=20
> >>=20
>=20

--pWyiEgJYm5f9v55/
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iF4EAREIAAYFAk1pwK0ACgkQenyFlstYjhKMnQD+N4hf5dJa6w7JXCJ8q3tlyKwP
8j9e6KA/M8iJl/fjpO0A/02Wqzhus5AnuJcSBFHrJIvtRXTA3xnIZOSpjhntMX6B
=J3Z7
-----END PGP SIGNATURE-----

--pWyiEgJYm5f9v55/--

From general-return-3132-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Feb 27 09:33:12 2011
Return-Path: <general-return-3132-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97842 invoked from network); 27 Feb 2011 09:33:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Feb 2011 09:33:12 -0000
Received: (qmail 77839 invoked by uid 500); 27 Feb 2011 09:33:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77388 invoked by uid 500); 27 Feb 2011 09:33:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77379 invoked by uid 99); 27 Feb 2011 09:33:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Feb 2011 09:33:06 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Feb 2011 09:32:59 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p1R9WTIe066202
	for <general@hadoop.apache.org>; Sun, 27 Feb 2011 01:32:29 -0800 (PST)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by
 SP2-EX07CAS01.ds.corp.yahoo.com ([98.137.59.37]) with mapi; Sun, 27 Feb 2011
 01:32:29 -0800
From: Eric Yang <eyang@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Sun, 27 Feb 2011 01:32:27 -0800
Subject: Re: Build/test infrastructure
Thread-Topic: Build/test infrastructure
Thread-Index: AcvWLAx3/tgbNc9tRtGERFXjPYvM+AANTKx5
Message-ID: <C98F5A2B.FFC3%eyang@yahoo-inc.com>
In-Reply-To: <20110227031037.GA14579@tp>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

On 2/26/11 7:10 PM, "Konstantin Boudnik" <cos@apache.org> wrote:

> On Sat, Feb 26, 2011 at 05:38PM, Eric Yang wrote:
>=20
> Furthermore, my Puppet knowledge is very limited and I am for sure no exp=
ert
> in maven. I have some concern however:
>   - how to provide privileged access
>   - how and where to store host configurations (i.e. packages names, vers=
ions,
>     which are gonna be different for difference OSes)
>   - how to do native packages (see above example) and native dependency
>     management from maven? With shell scripting?
>   - how to maintain such a construct?
> I can continue for a long time, but I'd rather want to solve an issue of
> managing build host configurations/package sets in a most efficient and
> sustainable manner.

Hudson already supports chroot jail environment.  It is easy to setup
privileged access in the jailed environment by giving the hudson running
user sudo access to the jailed environment.  The host configuration can be
mirrored into chroot environment with minimum set of shell commands.

> Sorry, but Maven + shell script can be called simplification only in a pi=
pe
> dream ;) Maven is a build tool. A relatively good one perhaps, but just a
> build tool. Certainly everything can be done with a combination of a shel=
l
> scripting + tar balls and a little SSH sugar topping. But I'd rather use =
a
> accurately designed and supported tool (Puppet, Chef, etc.).

Maven supports various kind of remote deployment plugin.  Exec plugin with
shell script is the easiest one to implement.  There are also plugin like
cargo for more complex container deployment.  There is a plan to write a
deployment framework for hadoop for large scale deployment.  This project i=
s
in planning stage.  The scope is deploying the entire hadoop stack (hdfs,
mr, zookeeper, hbase, pig, hive, and chukwa) to multiple large clusters.
Similar to what you are planning except at the scale that it would make
sense to use puppet+mcollective.  We had done the evaluation, and found per
puppet master would not scale well after 1800 nodes, and multilayer
puppeteer spamming tree to cover all our nodes, is not ideal.  We choose to
use chef-solo for edge deployment.  The rest of the details are to be worke=
d
out.  This is the reason that I am interested on the test environment that
is being planned here.  It will be possible to use "to be invented"
framework in the hudson.  This system is not going to be grown in ant/maven
build script, hence it will be better to keep build system simple for now.

Regards,
Eric

>=20
> And BTW - Hadoop builds aren't maven'ized yet. Which renders most of the
> argument a time waste until that problem is solved.
>=20
> At any rate, HADOOP-7157 is the JIRA for this. Please comment on it.
>=20
> Cos
>=20
>> Regards,
>> Eric
>>=20
>>> You don't need to setup puppet muster in order to bounce a node. Puppet
>>> works
>>> i a client-only mode just as perfect.
>>>=20
>>> Cos
>>>=20
>>=20
>>>> packaging only, but express my opinions on improving build system and
>>>> making
>>>> the system easier to reproduce.
>>>>=20
>>>> Regards,
>>>> Eric
>>>>=20
>>>> On 2/26/11 2:18 PM, "Konstantin Boudnik" <cos@apache.org> wrote:
>>>>=20
>>>> This discussion isn't about build of the product nor about packaging
>>>> of it. We are discussing patch validation and snapshot build
>>>> infrastructure.
>>>>=20
>>>> On Sat, Feb 26, 2011 at 12:43, Eric Yang <eyang@yahoo-inc.com> wrote:
>>>>> We should be very careful about the approach that we chosen for
>>>>> build/packaging.  The current state of hadoop is coupled together due=
 to
>>>>> lack of standardized RPC format.  Once this issue is cleared, the
>>>>> community will want to split hdfs and m/r into separated projects at =
some
>>>>> point.  It may be better to ensure project is modularized, and work f=
rom
>>>>> the same svn repository.  Maven is great for doing this, and most of =
the
>>>>> build and scripts can be defined in pom.xml.  Deployment/test server
>>>>> configuration can be pass in from hudson.  We should ensure that buil=
d and
>>>>> deployment script do not further couple the project.
>>>>>=20
>>>>> Regards,
>>>>> Eric
>>>>>=20
>>>>> On 2/26/11 11:14 AM, "Konstantin Boudnik" <cos@apache.org> wrote:
>>>>>=20
>>>>> On Fri, Feb 25, 2011 at 23:47, Nigel Daley <ndaley@mac.com> wrote:
>>>>>> +1.
>>>>>>=20
>>>>>> Once HADOOP-7106 is committed, I'd like to propose we create a direc=
tory
>>>>>> at
>>>>>> the same level of common/hdfs/mapreduce to hold build (and deploy) t=
ype
>>>>>> scripts and files.  These would then get branches/tagged with the re=
st of
>>>>>> the release.
>>>>>=20
>>>>> That makes sense, although I don't see changes of the host
>>>>> configurations to happen very often.
>>>>>=20
>>>>> Cos
>>>>>=20
>>>>>> Nige
>>>>>>=20
>>>>>> On Feb 25, 2011, at 7:55 PM, Konstantin Boudnik wrote:
>>>>>>=20
>>>>>>> Looking at re-occurring build/test-patch problems on hadoop? build
>>>>>>> machines I
>>>>>>> thought of a way to make them:
>>>>>>>  a) all the same (configuration, installed software wise)
>>>>>>>  b) have an effortless system to run upgrades/updates on all of the=
m in
>>>>>>> a
>>>>>>>  controlled fashion.
>>>>>>>=20
>>>>>>> I would suggest to create Puppet configs (the exact content to be
>>>>>>> defined)
>>>>>>> which we'll be checked in SCM (e.g. SVN), Whenever a build host's
>>>>>>> software
>>>>>>> is needed to be restored/updated a simple run of Puppet across the
>>>>>>> machines
>>>>>>> or change in config and run of Puppet will do the magic for us.
>>>>>>>=20
>>>>>>> If there are no objections from the community I can put together so=
me
>>>>>>> Puppet recipes which might be evolved as we go.
>>>>>>>=20
>>>>>>> --
>>>>>>> Take care,
>>>>>>>       Cos
>>>>>>> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>>>>>>>=20
>>>>>>> After all, it is only the mediocre who are always at their best.
>>>>>>>                Jean Giraudoux
>>>>>>=20
>>>>>>=20
>>>>>=20
>>>>>=20
>>>>=20
>>=20


From general-return-3133-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Feb 27 19:37:44 2011
Return-Path: <general-return-3133-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31503 invoked from network); 27 Feb 2011 19:37:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Feb 2011 19:37:44 -0000
Received: (qmail 54623 invoked by uid 500); 27 Feb 2011 19:37:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54491 invoked by uid 500); 27 Feb 2011 19:37:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54483 invoked by uid 99); 27 Feb 2011 19:37:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Feb 2011 19:37:41 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Feb 2011 19:37:37 +0000
Received: by wwi14 with SMTP id 14so3726798wwi.29
        for <general@hadoop.apache.org>; Sun, 27 Feb 2011 11:37:15 -0800 (PST)
Received: by 10.216.52.143 with SMTP id e15mr1553367wec.44.1298835435096; Sun,
 27 Feb 2011 11:37:15 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.180.70 with HTTP; Sun, 27 Feb 2011 11:36:55 -0800 (PST)
In-Reply-To: <C98F5A2B.FFC3%eyang@yahoo-inc.com>
References: <20110227031037.GA14579@tp> <C98F5A2B.FFC3%eyang@yahoo-inc.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Sun, 27 Feb 2011 11:36:55 -0800
X-Google-Sender-Auth: WgtLDL3rRhsBf2BRBVQKDCxndPM
Message-ID: <AANLkTin7HoHSeW5fVXPSWyt7-ab0N6t5gv0DY+0ddOUF@mail.gmail.com>
Subject: Re: Build/test infrastructure
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

And now jailed environments with Hudson - oh yes, lovely yinst all over aga=
in.

Eric, the glorious plans about a framework to deploy thousand of nodes
by the click of a button are awesome indeed. But you apparently
ignoring the point:
  - this is about synchronization of 10 build machines at the level of
system packaging and operating system configuration.

With warmest regards,
  Cos

On Sun, Feb 27, 2011 at 01:32, Eric Yang <eyang@yahoo-inc.com> wrote:
> On 2/26/11 7:10 PM, "Konstantin Boudnik" <cos@apache.org> wrote:
>
>> On Sat, Feb 26, 2011 at 05:38PM, Eric Yang wrote:
>>
>> Furthermore, my Puppet knowledge is very limited and I am for sure no ex=
pert
>> in maven. I have some concern however:
>> =A0 - how to provide privileged access
>> =A0 - how and where to store host configurations (i.e. packages names, v=
ersions,
>> =A0 =A0 which are gonna be different for difference OSes)
>> =A0 - how to do native packages (see above example) and native dependenc=
y
>> =A0 =A0 management from maven? With shell scripting?
>> =A0 - how to maintain such a construct?
>> I can continue for a long time, but I'd rather want to solve an issue of
>> managing build host configurations/package sets in a most efficient and
>> sustainable manner.
>
> Hudson already supports chroot jail environment. =A0It is easy to setup
> privileged access in the jailed environment by giving the hudson running
> user sudo access to the jailed environment. =A0The host configuration can=
 be
> mirrored into chroot environment with minimum set of shell commands..?
>
>> Sorry, but Maven + shell script can be called simplification only in a p=
ipe
>> dream ;) Maven is a build tool. A relatively good one perhaps, but just =
a
>> build tool. Certainly everything can be done with a combination of a she=
ll
>> scripting + tar balls and a little SSH sugar topping. But I'd rather use=
 a
>> accurately designed and supported tool (Puppet, Chef, etc.).
>
> Maven supports various kind of remote deployment plugin. =A0Exec plugin w=
ith
> shell script is the easiest one to implement. =A0There are also plugin li=
ke
> cargo for more complex container deployment. =A0There is a plan to write =
a
> deployment framework for hadoop for large scale deployment. =A0This proje=
ct is
> in planning stage. =A0The scope is deploying the entire hadoop stack (hdf=
s,
> mr, zookeeper, hbase, pig, hive, and chukwa) to multiple large clusters.
> Similar to what you are planning except at the scale that it would make
> sense to use puppet+mcollective. =A0We had done the evaluation, and found=
 per
> puppet master would not scale well after 1800 nodes, and multilayer
> puppeteer spamming tree to cover all our nodes, is not ideal. =A0We choos=
e to
> use chef-solo for edge deployment. =A0The rest of the details are to be w=
orked
> out. =A0This is the reason that I am interested on the test environment t=
hat
> is being planned here. =A0It will be possible to use "to be invented"
> framework in the hudson. =A0This system is not going to be grown in ant/m=
aven
> build script, hence it will be better to keep build system simple for now=
.
>
> Regards,
> Eric
>
>>
>> And BTW - Hadoop builds aren't maven'ized yet. Which renders most of the
>> argument a time waste until that problem is solved.
>>
>> At any rate, HADOOP-7157 is the JIRA for this. Please comment on it.
>>
>> Cos
>>
>>> Regards,
>>> Eric
>>>
>>>> You don't need to setup puppet muster in order to bounce a node. Puppe=
t
>>>> works
>>>> i a client-only mode just as perfect.
>>>>
>>>> Cos
>>>>
>>>
>>>>> packaging only, but express my opinions on improving build system and
>>>>> making
>>>>> the system easier to reproduce.
>>>>>
>>>>> Regards,
>>>>> Eric
>>>>>
>>>>> On 2/26/11 2:18 PM, "Konstantin Boudnik" <cos@apache.org> wrote:
>>>>>
>>>>> This discussion isn't about build of the product nor about packaging
>>>>> of it. We are discussing patch validation and snapshot build
>>>>> infrastructure.
>>>>>
>>>>> On Sat, Feb 26, 2011 at 12:43, Eric Yang <eyang@yahoo-inc.com> wrote:
>>>>>> We should be very careful about the approach that we chosen for
>>>>>> build/packaging. =A0The current state of hadoop is coupled together =
due to
>>>>>> lack of standardized RPC format. =A0Once this issue is cleared, the
>>>>>> community will want to split hdfs and m/r into separated projects at=
 some
>>>>>> point. =A0It may be better to ensure project is modularized, and wor=
k from
>>>>>> the same svn repository. =A0Maven is great for doing this, and most =
of the
>>>>>> build and scripts can be defined in pom.xml. =A0Deployment/test serv=
er
>>>>>> configuration can be pass in from hudson. =A0We should ensure that b=
uild and
>>>>>> deployment script do not further couple the project.
>>>>>>
>>>>>> Regards,
>>>>>> Eric
>>>>>>
>>>>>> On 2/26/11 11:14 AM, "Konstantin Boudnik" <cos@apache.org> wrote:
>>>>>>
>>>>>> On Fri, Feb 25, 2011 at 23:47, Nigel Daley <ndaley@mac.com> wrote:
>>>>>>> +1.
>>>>>>>
>>>>>>> Once HADOOP-7106 is committed, I'd like to propose we create a dire=
ctory
>>>>>>> at
>>>>>>> the same level of common/hdfs/mapreduce to hold build (and deploy) =
type
>>>>>>> scripts and files. =A0These would then get branches/tagged with the=
 rest of
>>>>>>> the release.
>>>>>>
>>>>>> That makes sense, although I don't see changes of the host
>>>>>> configurations to happen very often.
>>>>>>
>>>>>> Cos
>>>>>>
>>>>>>> Nige
>>>>>>>
>>>>>>> On Feb 25, 2011, at 7:55 PM, Konstantin Boudnik wrote:
>>>>>>>
>>>>>>>> Looking at re-occurring build/test-patch problems on hadoop? build
>>>>>>>> machines I
>>>>>>>> thought of a way to make them:
>>>>>>>> =A0a) all the same (configuration, installed software wise)
>>>>>>>> =A0b) have an effortless system to run upgrades/updates on all of =
them in
>>>>>>>> a
>>>>>>>> =A0controlled fashion.
>>>>>>>>
>>>>>>>> I would suggest to create Puppet configs (the exact content to be
>>>>>>>> defined)
>>>>>>>> which we'll be checked in SCM (e.g. SVN), Whenever a build host's
>>>>>>>> software
>>>>>>>> is needed to be restored/updated a simple run of Puppet across the
>>>>>>>> machines
>>>>>>>> or change in config and run of Puppet will do the magic for us.
>>>>>>>>
>>>>>>>> If there are no objections from the community I can put together s=
ome
>>>>>>>> Puppet recipes which might be evolved as we go.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Take care,
>>>>>>>> =A0 =A0 =A0 Cos
>>>>>>>> 2CAC 8312 4870 D885 8616 =A06115 220F 6980 1F27 E622
>>>>>>>>
>>>>>>>> After all, it is only the mediocre who are always at their best.
>>>>>>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Jean Giraudoux
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>
>

From general-return-3134-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 28 20:07:36 2011
Return-Path: <general-return-3134-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1506 invoked from network); 28 Feb 2011 20:07:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Feb 2011 20:07:36 -0000
Received: (qmail 227 invoked by uid 500); 28 Feb 2011 20:07:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 134 invoked by uid 500); 28 Feb 2011 20:07:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 94048 invoked by uid 99); 28 Feb 2011 20:03:44 -0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo-inc.com; s=ginc1024; t=1298923396; bh=sgta2AOzP/Q0BRWcr+Mwlt8n8UhJoFBI5uFmVBe+vkM=; h=Message-ID:X-YMail-OSG:Received:X-RocketYMMF:X-Mailer:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=aK1qf2gsq79wN9bQORShfm33LsRogAJ15QP0qccVoqVTuXs0w/CokewAxinWOzpunH2J8/v1xjiX8Q69bw+28trBSYWYJDKmCSNM31CxO+1mpnrUGZ75v5Q4pzl7eIQXH255RjW2WlQ3xh5XXmRF9mNlcRT1EYnarODyuKxC0jk=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=ginc1024; d=yahoo-inc.com;
  h=Message-ID:X-YMail-OSG:Received:X-RocketYMMF:X-Mailer:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=Da9jxxhDIag8OM7f/wezgDDQf801FLpdCmZTqR41PhuqfAhaAbAwQBYWTmYMuaXrCnPJlJqNB++zep1wpNrtJmftXfmkcPRZlDEpIYN5G3NSMZB9yS0y7WBhxgxCt6L/C4S39OH8UKXbpEe0Nb49bKX2IJW45/NWsWpMgnJTGg4=;
Message-ID: <912233.26325.qm@web31811.mail.mud.yahoo.com>
X-YMail-OSG: 61kC8UYVM1lzfkI7wNCdOcCC5fEIUsjuNHRasuQ0m7LvU2E
 ezQaD4ssLGQxTjJyRUXz7_jx.VWJ6bdFfrCWLgITyDkGuF.NCvizsIWRgL5M
 nfAg3YmnGDA.qnawaaqq0RJjRdmyp_tJgxJ8k9aSCkJO.KuybfunPD2Eu6xu
 JY_JkSlx80nqaiJRXJKu3YkdOIBVG9wDVy70K_NXqcc2WHtGeMD6FgNqMfkI
 baJ4N5XSOrhXSnZ4f.vaHKjXalP7W5ydN9zCj7lFOhih8NSC7ks2LI1IpMn8
 AnKw3lNwSrKcpmfck_R7J7.I_ADLWTlzYTLABgJ4SXR_KRQEKGg--
X-RocketYMMF: rajive_c
X-Mailer: YahooMailClassic/11.4.20 YahooMailWebService/0.8.109.295617
Date: Mon, 28 Feb 2011 12:03:16 -0800 (PST)
From: Rajiv Chittajallu <rajive@yahoo-inc.com>
Reply-To: rajive@yahoo-inc.com
Subject: Re: Build/test infrastructure
To: general@hadoop.apache.org
In-Reply-To: <AANLkTin7HoHSeW5fVXPSWyt7-ab0N6t5gv0DY+0ddOUF@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii

--- On Sun, 2/27/11, Konstantin Boudnik <cos@apache.org> wrote:

> And now jailed environments with
> Hudson - oh yes, lovely yinst all over again.

>From where did the 'yinst' come in to this discussion? Since you know something about a company's internal tools, don't assume every one posting from that company is talking about it.  



From general-return-3135-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 28 20:22:13 2011
Return-Path: <general-return-3135-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5362 invoked from network); 28 Feb 2011 20:22:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Feb 2011 20:22:13 -0000
Received: (qmail 19524 invoked by uid 500); 28 Feb 2011 20:22:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19479 invoked by uid 500); 28 Feb 2011 20:22:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19471 invoked by uid 99); 28 Feb 2011 20:22:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Feb 2011 20:22:11 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Feb 2011 20:22:06 +0000
Received: by wya21 with SMTP id 21so5100447wya.35
        for <general@hadoop.apache.org>; Mon, 28 Feb 2011 12:21:45 -0800 (PST)
Received: by 10.216.51.135 with SMTP id b7mr2706266wec.29.1298924505158; Mon,
 28 Feb 2011 12:21:45 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.180.70 with HTTP; Mon, 28 Feb 2011 12:21:25 -0800 (PST)
In-Reply-To: <912233.26325.qm@web31811.mail.mud.yahoo.com>
References: <AANLkTin7HoHSeW5fVXPSWyt7-ab0N6t5gv0DY+0ddOUF@mail.gmail.com> <912233.26325.qm@web31811.mail.mud.yahoo.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Mon, 28 Feb 2011 12:21:25 -0800
X-Google-Sender-Auth: DJN9nTxmEQO1JiA0_dQz7QEoSoA
Message-ID: <AANLkTi=7hao9vpwhxaMYO3zmRWw-qOswqd0m7_sy+Fnq@mail.gmail.com>
Subject: Re: Build/test infrastructure
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Feb 28, 2011 at 12:03, Rajiv Chittajallu <rajive@yahoo-inc.com> wrote:
> --- On Sun, 2/27/11, Konstantin Boudnik <cos@apache.org> wrote:
>
>> And now jailed environments with
>> Hudson - oh yes, lovely yinst all over again.
>
> From where did the 'yinst' come in to this discussion? Since you know something about a company's
> internal tools, don't assume every one posting from that company is talking about it.

True, it doesn't imply. It was my assumption based on
chrooting/jailing suggestion above. Of course, it doesn't means yinst
or anything related to it. Sorry if I have offended anyone by this
reference.

Cos

From general-return-3136-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 28 20:49:02 2011
Return-Path: <general-return-3136-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 11364 invoked from network); 28 Feb 2011 20:49:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Feb 2011 20:49:01 -0000
Received: (qmail 63548 invoked by uid 500); 28 Feb 2011 20:49:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63423 invoked by uid 500); 28 Feb 2011 20:48:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63415 invoked by uid 99); 28 Feb 2011 20:48:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Feb 2011 20:48:59 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Feb 2011 20:48:54 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=iQGANoB4OAOnLFhZbugvJfVmiiqWPF8IunHxyx5NT8PaNYVLCIy42Reu
   dicJQ5U9DUYy6+SjyPWeW1bai/XKyWfdiGZ4ArnWtD6koYG60gUaoO3eK
   ghjNaQbAt2zHHSF;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1298926129; x=1330462129;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20Build/test=20infrastructure|Date:=20Mon
   ,=2028=20Feb=202011=2020:50:22=20+0000|Message-ID:=20<A5F
   0A7B8-15EF-4E50-BF2F-141C4D449E69@linkedin.com>|To:=20"<g
   eneral@hadoop.apache.org>"=20<general@hadoop.apache.org>
   |MIME-Version:=201.0|Content-Transfer-Encoding:=20quoted-
   printable|Content-ID:=20<A5645B0E4642B641A9CE2B59F267D466
   @linkedin.com>|In-Reply-To:=20<20110227031037.GA14579@tp>
   |References:=20<20110227003415.GA9036@tp>=20<C98EEB2E.FFB
   A%eyang@yahoo-inc.com>=0D=0A=20<20110227031037.GA14579@tp
   >;
  bh=prQ5DagKOTm9zZS4vcN89KbiymFkMAlDT9BjfjMJd1M=;
  b=crHL9NIrTAY97ZVhHEO0AL68EcepIuj4AduMMvjmSUjhTpG9ZL7N3IVH
   1teoNXlND0ejzvegOneKd6sFHOY8k4M6qwxBUnkl+FUVk+sTQj5s39Xf7
   GJ1YMAmbqDiwoWT;
X-IronPort-AV: E=Sophos;i="4.62,242,1297065600"; 
   d="scan'208";a="21188402"
Received: from ESV4-EXC03.linkedin.biz ([fe80::985a:a6b4:f1e1:23b0]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi id 14.01.0251.000; Mon,
 28 Feb 2011 12:50:23 -0800
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: Build/test infrastructure
Thread-Topic: Build/test infrastructure
Thread-Index: AQHL1WlWfNl4aje6tUuwduC5YcQrXpQT7jEAgAC/zACAABkRgIAAGnqAgAAdTYCAAAiTgIAAEhAAgAAZogCAArnnAA==
Date: Mon, 28 Feb 2011 20:50:22 +0000
Message-ID: <A5F0A7B8-15EF-4E50-BF2F-141C4D449E69@linkedin.com>
References: <20110227003415.GA9036@tp> <C98EEB2E.FFBA%eyang@yahoo-inc.com>
 <20110227031037.GA14579@tp>
In-Reply-To: <20110227031037.GA14579@tp>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <A5645B0E4642B641A9CE2B59F267D466@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


On Feb 26, 2011, at 7:10 PM, Konstantin Boudnik wrote:
> BTW, Puppet and Chef recipes are very widely used by all sorts of Ops and
> cluster management companies. Perhaps, Maven and shell too - I'm not in a
> position to make a judgement call. I'll let Y! Grid Ops to comment on it =
-
> they know everything about sizable clusters configuration management and =
tools
> for the job.

	I'm not in Y! Grid Ops, but from where I sit, it sounds like you are solvi=
ng the wrong problem.

	Getting the build machines to be the same should mostly be a one-time issu=
e.  If non-ops-types are messing with the package load, then that's a privi=
lege and process problem not an automation problem.   The success of a conf=
iguration management utility is directly correlated to the amount of work t=
hat people are willing to put into using them.


From general-return-135-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 02 18:19:46 2009
Return-Path: <general-return-135-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 7158 invoked from network); 2 Mar 2009 18:19:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 2 Mar 2009 18:19:44 -0000
Received: (qmail 52043 invoked by uid 500); 2 Mar 2009 18:19:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52008 invoked by uid 500); 2 Mar 2009 18:19:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51997 invoked by uid 99); 2 Mar 2009 18:19:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 10:19:43 -0800
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.30.80] (HELO QMTA08.emeryville.ca.mail.comcast.net) (76.96.30.80)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 18:19:35 +0000
Received: from OMTA10.emeryville.ca.mail.comcast.net ([76.96.30.28])
	by QMTA08.emeryville.ca.mail.comcast.net with comcast
	id NGcA1b0040cQ2SLA8JKFSg; Mon, 02 Mar 2009 18:19:15 +0000
Received: from battlerock-lm.corp.yahoo.com ([209.131.62.113])
	by OMTA10.emeryville.ca.mail.comcast.net with comcast
	id NJK61b00S2SbwD58WJK8fD; Mon, 02 Mar 2009 18:19:13 +0000
Message-Id: <8E7E08E2-2976-4322-BEB7-DDB415788DE0@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <C5C9AC53.143C4%eyang@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: [RESULT] Chukwa as subproject
Date: Mon, 2 Mar 2009 10:19:05 -0800
References: <C5C9AC53.143C4%eyang@yahoo-inc.com>
X-Mailer: Apple Mail (2.930.3)
X-Virus-Checked: Checked by ClamAV on apache.org

With 6 +1's from the PMC and no -1's, the vote passes. I'll create the  
mailing lists and Jira and schedule the subversion move.

-- Owen

From general-return-136-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 02 20:58:20 2009
Return-Path: <general-return-136-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 79173 invoked from network); 2 Mar 2009 20:58:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 2 Mar 2009 20:58:20 -0000
Received: (qmail 56529 invoked by uid 500); 2 Mar 2009 20:58:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56485 invoked by uid 500); 2 Mar 2009 20:58:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56464 invoked by uid 99); 2 Mar 2009 20:58:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 12:58:11 -0800
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of work.aviad@gmail.com designates 209.85.218.179 as permitted sender)
Received: from [209.85.218.179] (HELO mail-bw0-f179.google.com) (209.85.218.179)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 20:58:01 +0000
Received: by bwz27 with SMTP id 27so2319245bwz.29
        for <multiple recipients>; Mon, 02 Mar 2009 12:57:40 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:message-id:subject:from:to:cc
         :content-type;
        bh=SzD/OkyYkdlRA3qfhTPaRhMvEayhH3sJFIxMp0GJTqs=;
        b=ao++AG0AT0AIx4uNdXmPIhKkIuFyCzib3sr1vBgICinr/lMushvVhDQ3URQdC/Nmca
         dWsXM4iQ6eJ+BEv+yV+C4XOOnjVAABystKd4LsOgJS4uL8F0SWEFX5Acr6SBI1RNyZTh
         0UDwQSbOw2ktcM+IZ4PtrINpPk2EIcUeyeCj4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:cc:content-type;
        b=M1QxT7u/Unp+q3l6Xo37jXiyaZ/rRREhGWAKqc/NyBycSOtQK9kRXCHl/RZ7M+u3qa
         jmOfV0ikmxRrA5RxKaT/FYKP4hI3bqJVvOLc3p4g/H7xyJjjzGHv22f8gL+U084XsEwj
         49h7s1YieY86ihV5oiwLQ45IGytnBBTdQUrVs=
MIME-Version: 1.0
Sender: work.aviad@gmail.com
Received: by 10.181.226.3 with SMTP id d3mr2257346bkr.12.1236027460165; Mon, 
	02 Mar 2009 12:57:40 -0800 (PST)
In-Reply-To: <A1B8D071-8081-47C9-AB8D-CC39E9216AB6@yahoo-inc.com>
References: <C607AEFB-EFD7-4BFA-B5F6-50EB6866BCCB@yahoo-inc.com>
	 <6a3f6a4b0902250227m10d46683m62f5ed7737ca5f5d@mail.gmail.com>
	 <A1B8D071-8081-47C9-AB8D-CC39E9216AB6@yahoo-inc.com>
Date: Mon, 2 Mar 2009 22:57:40 +0200
X-Google-Sender-Auth: 24c4ac5b828dabeb
Message-ID: <6a3f6a4b0903021257r6ec74f54n34b3f287059f8ff3@mail.gmail.com>
Subject: Re: [ANNOUNCE] Hadoop release 0.19.1 available
From: Aviad sela <sela.stam@gmail.com>
To: core-user@hadoop.apache.org
Cc: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c5a3dbb21f08046429112e
X-Virus-Checked: Checked by ClamAV on apache.org

--001636c5a3dbb21f08046429112e
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Nigel Thanks,
I have extracted the new project.

However, I am having problems building the project
I am using Eclipse 3.4
and ant 1.7

I recieve error compiling core classes
*

compile-core-classes*:

BUILD FAILED

<jsp-compile

uriroot="${src.webapps}/task"

outputdir="${build.src}"

package="org.apache.hadoop.mapred"

webxml="${build.webapps}/task/WEB-INF/web.xml">

</jsp-compile>
*

D:\Work\AviadWork\workspace\cur\WSAD\Hadoop_Core_19_1\Hadoop\build.xml:302:
java.lang.ExceptionInInitializerError

*
it points to the the webxml tag



On Fri, Feb 27, 2009 at 9:24 AM, Nigel Daley <ndaley@yahoo-inc.com> wrote:

> Sorry for delay.  Should now be fixed.  Please confirm.
>
> Thx,
> Nige
>
>
> On Feb 25, 2009, at 2:27 AM, Aviad sela wrote:
>
> Nigel,
>> The SVN tag http://svn.apach.org/repos/asf/core/tags/release-0.19.1 include
>> also the branch folder branch-0.19
>> which results with double size project which is not necessary.
>>
>> I need to use this release to apply patch HADOOP-4546 (
>>
>> https://issues.apache.org/jira/browse/HADOOP-4546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel
>> )
>> I don't know how the above branch folder should contribute when building
>> the
>> new eclipse project.
>>
>> Isn't it going to be correct to remove this branch folder from the release
>> tag ?
>>
>> Thanks,
>>
>>
>>
>> On Wed, Feb 25, 2009 at 12:20 AM, Nigel Daley <ndaley@yahoo-inc.com>
>> wrote:
>>
>> Release 0.19.1 fixes many critical bugs in 0.19.0, including ***some data
>>> loss issues***. The release also introduces an ***incompatible change***
>>> by
>>> disabling the file append API (HADOOP-5224) until it can be stabilized.
>>>
>>> For Hadoop release details and downloads, visit:
>>> http://hadoop.apache.org/core/releases.html
>>>
>>> Hadoop 0.19.1 Release Notes are at
>>> http://hadoop.apache.org/core/docs/r0.19.1/releasenotes.html
>>>
>>> Thanks to all who contributed to this release!
>>>
>>> Nigel
>>>
>>>
>>>
>>>
>>>
>

--001636c5a3dbb21f08046429112e--

From general-return-137-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 12 21:36:10 2009
Return-Path: <general-return-137-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30404 invoked from network); 12 Mar 2009 21:36:10 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 12 Mar 2009 21:36:10 -0000
Received: (qmail 43566 invoked by uid 500); 12 Mar 2009 21:35:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 43429 invoked by uid 500); 12 Mar 2009 21:35:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42989 invoked by uid 99); 12 Mar 2009 21:35:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Mar 2009 14:35:57 -0700
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Mar 2009 21:35:45 +0000
Received: from SNV-EXPF01.ds.corp.yahoo.com (snv-expf01.ds.corp.yahoo.com [207.126.227.250])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n2CLYqM9061348;
	Thu, 12 Mar 2009 14:34:52 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:x-mimeole:content-class:mime-version:
	content-type:subject:date:message-id:in-reply-to:x-ms-has-attach:
	x-ms-tnef-correlator:thread-topic:thread-index:references:from:to:return-path:x-originalarrivaltime;
	b=xx9jIAJEiorv7TJbtC5/W5UWuhGqs+n59JkzD5mu3EAex75XZ0l3ofCOsOxyuugh
Received: from SNV-EXVS02.ds.corp.yahoo.com ([216.145.51.196]) by SNV-EXPF01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Thu, 12 Mar 2009 14:34:52 -0700
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C9A35A.4226627E"
Subject: Hadoop User Group Meeting (Bay Area) 3/18
Date: Thu, 12 Mar 2009 14:33:59 -0700
Message-ID: <DD27C9769EA63D43BE3CF4CCCC12DD97023EE393@SNV-EXVS02.ds.corp.yahoo.com>
In-Reply-To: <AD0C16AAE446924586B68FC0EB421C9BAE1548CD@SNV-EXVS02.ds.corp.yahoo.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Hadoop User Group Meeting (Bay Area) 3/18
Thread-Index: AcmNW1p3QQcXNKUgTi6+Qtm5ky8HQAV/ghoA
References: <AD0C16AAE446924586B68FC0EB421C9BAE1548CD@SNV-EXVS02.ds.corp.yahoo.com>
From: "Ajay Anand" <aanand@yahoo-inc.com>
To: "Ajay Anand" <aanand@yahoo-inc.com>,
        "core-user@hadoop.apache.org" <'core-user@hadoop.apache.org'>,
        "general@hadoop.apache.org" <'general@hadoop.apache.org'>,
        "zookeeper-user@hadoop.apache.org" <'zookeeper-user@hadoop.apache.org'>,
        "hbase-user@hadoop.apache.org" <'hbase-user@hadoop.apache.org'>,
        <pig-user@hadoop.apache.org>
X-OriginalArrivalTime: 12 Mar 2009 21:34:52.0611 (UTC) FILETIME=[60F96130:01C9A35A]
X-Virus-Checked: Checked by ClamAV on apache.org

------_=_NextPart_001_01C9A35A.4226627E
Content-Type: text/plain;
	charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable

The next Bay Area Hadoop User Group meeting is scheduled for Wednesday,
March 18th at Yahoo! 2811 Mission College Blvd, Santa Clara, Building 2,
Training Rooms 5 & 6 from 6:00-7:30 pm.

=20

Agenda:
"Performance Enhancement Techniques with Hadoop - a Case Study" - Milind
Bhandarkar
"RPMs for Hadoop Deployment and Configuration Management" - Matt Massie,
Christophe Bisciglia



Registration: http://upcoming.yahoo.com/event/2112338/

=20

Look forward to seeing you there.

Ajay

=20


------_=_NextPart_001_01C9A35A.4226627E--

From general-return-138-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 12 22:09:38 2009
Return-Path: <general-return-138-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49176 invoked from network); 12 Mar 2009 22:09:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 12 Mar 2009 22:09:38 -0000
Received: (qmail 97425 invoked by uid 500); 12 Mar 2009 22:09:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97400 invoked by uid 500); 12 Mar 2009 22:09:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97389 invoked by uid 99); 12 Mar 2009 22:09:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Mar 2009 15:09:37 -0700
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [153.2.232.100] (HELO ups.com) (153.2.232.100)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Mar 2009 22:09:27 +0000
Received: from ([153.2.23.181])
	by magma11.ups.com with ESMTP with TLS id 5503512.60781333;
	Thu, 12 Mar 2009 18:08:46 -0400
Received: from njrarsvr3bee.us.ups.com ([153.2.23.199]) by
 njrarsvr3ab2.us.ups.com ([153.2.23.181]) with mapi; Thu, 12 Mar 2009 18:08:46
 -0400
From: <jlivingstone@ups.com>
To: <general@hadoop.apache.org>
Importance: high
Date: Thu, 12 Mar 2009 18:08:44 -0400
Subject: RE: Hadoop User Group Meeting (Bay Area) 3/18
Thread-Topic: Hadoop User Group Meeting (Bay Area) 3/18
Thread-Index: AcmNW1p3QQcXNKUgTi6+Qtm5ky8HQAV/ghoAAAFg6FA=
Message-ID: <15473C93FCCAB44A8B221DDD97CBC69734AFF072@njrarsvr3bee.us.ups.com>
References: <AD0C16AAE446924586B68FC0EB421C9BAE1548CD@SNV-EXVS02.ds.corp.yahoo.com>
 <DD27C9769EA63D43BE3CF4CCCC12DD97023EE393@SNV-EXVS02.ds.corp.yahoo.com>
In-Reply-To: <DD27C9769EA63D43BE3CF4CCCC12DD97023EE393@SNV-EXVS02.ds.corp.yahoo.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Ajay,
I would love to attend - however, I'm way over on the east coast - NJ to be=
 specific!  Do you think that I can get a copy of presentation?
Thanks,
John

John Livingstone
Enterprise Architecture, United Parcel Service
R-03A-176, Mahwah, New Jersey
* Tel  201.828.7584
* jlivingstone@ups.com

-----Original Message-----
From: Ajay Anand [mailto:aanand@yahoo-inc.com]
Sent: Thursday, March 12, 2009 5:34 PM
To: Ajay Anand; core-user@hadoop.apache.org; general@hadoop.apache.org; zoo=
keeper-user@hadoop.apache.org; hbase-user@hadoop.apache.org; pig-user@hadoo=
p.apache.org
Subject: Hadoop User Group Meeting (Bay Area) 3/18

The next Bay Area Hadoop User Group meeting is scheduled for Wednesday, Mar=
ch 18th at Yahoo! 2811 Mission College Blvd, Santa Clara, Building 2, Train=
ing Rooms 5 & 6 from 6:00-7:30 pm.



Agenda:
"Performance Enhancement Techniques with Hadoop - a Case Study" - Milind Bh=
andarkar "RPMs for Hadoop Deployment and Configuration Management" - Matt M=
assie, Christophe Bisciglia



Registration: http://upcoming.yahoo.com/event/2112338/



Look forward to seeing you there.

Ajay




From general-return-139-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Mar 15 14:41:32 2009
Return-Path: <general-return-139-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78225 invoked from network); 15 Mar 2009 14:41:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 15 Mar 2009 14:41:32 -0000
Received: (qmail 71775 invoked by uid 500); 15 Mar 2009 14:41:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71752 invoked by uid 500); 15 Mar 2009 14:41:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 45521 invoked by uid 99); 15 Mar 2009 13:59:00 -0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of andy2005cst@gmail.com designates 209.85.198.226 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=/SJGRwVtRc1NERwNvdlKJyvX2Fc2iQNrxrz49luJa0w=;
        b=IVigPEP3aZ25uStz0Bjl1nHk8WnV7a2/Y59RBE1QGX1XGAZJUVURZUJuo41EbU/jOb
         C5sS4rpj5N88hK8l8a6ZCxSyiOWTTQVSjQZSpS1BmULAayzL6ECa6zdDzcp3vHqSVb/5
         pkhg0sRzwIcLlV4LLxrU7fJChESKf+O7Pa3T4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=o0CcyWiK2ZV0UxOUzaCOA3PhCjqjqyhjyR7d2n63Kqp1pc5QUA5WKkrjWNVyGe2ZvJ
         2XMrT6kKjNaoJcIKykADeegei5VgauCZcR3KYjhJdf47wG9VoBUyWx3bxld66ZVQAVKZ
         OwjFrH3N2fw13SJE3PVFKo6AiGTmRXItV/7jA=
MIME-Version: 1.0
Date: Sun, 15 Mar 2009 21:58:32 +0800
Message-ID: <17e74dd00903150658n4881efe7p209948cf60a9de49@mail.gmail.com>
Subject: [ask for help]Bad connection to FS. command aborted.
From: Xudong Du <andy2005cst@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636e90dcbb8191f046528baf6
X-Virus-Checked: Checked by ClamAV on apache.org

--001636e90dcbb8191f046528baf6
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hello, everyone. I met a strange problem and I am looking forward to your
help.

I setup a 2 nodes cluster hadoop on 2 mechine runing ubuntu. and 1 is both
master and slave and the other one is slave, and it seems to be functional.
but yesterday, it does not work anymore.

when I type hadoop command such as "bin/hadoop dfs -ls", it shows that "Bad
connection to FS. command aborted."
so, I run "bin/start-all.sh" to see if the dfs is shut, however it shows
that:

(***.***.***.*** stands for the master's ip address; ###.###.### stands for
the other one's ip address)
starting namenode, logging to *****
***.***.***.***: datanode running as process 14800. Stop it first.
***.***.***.***: datanode running as process 9113. Stop it first.
***.***.***.***: starting secondarynamenode, logging to ***
jobtracker running as process 14946. Stop it first.
***.***.***.***: tasktracker running as process 15029. Stop it first.
###.###.###.###: tasktracker running as process 9217. Stop it first.

and i type "bin/hadoop dfs -ls", but again: Bad connection to FS. command
aborted.

then when I type "bin/hadoop dfs -ls", it shows:
(***.***.***.*** stands for the master's ip address; ###.###.### stands for
the other one's ip address)
stopping jobtracker
***.***.***.***: stopping tasktracker
###.###.###: stopping tasktracker
no namenode to stop  <--------this is very strange
***.***.***.***: stopping datanode
###.###.###: stopping datanode
***.***.***.***: no secondarynamenode to stop

when i restart it by "bin/start-all.sh", it still does not work..

can anyone give me some idea? Thanks a lot.

-- 
Yours Sincerely
Xudong Du
Zijing 2# 305A
Tsinghua University, Beijing, China, 100084

--001636e90dcbb8191f046528baf6--

From general-return-140-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Mar 15 14:41:49 2009
Return-Path: <general-return-140-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78405 invoked from network); 15 Mar 2009 14:41:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 15 Mar 2009 14:41:47 -0000
Received: (qmail 72439 invoked by uid 500); 15 Mar 2009 14:41:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72421 invoked by uid 500); 15 Mar 2009 14:41:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 46528 invoked by uid 99); 15 Mar 2009 14:04:22 -0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of andy2005cst@gmail.com designates 209.85.200.168 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=RYToJBi0fCUGv/v/OwaDSPfRKP4vkhYSR7Jy3yBr2ko=;
        b=BcPzHcKdp8WF50jEa8Rori2ce3vKnXkoxs332n5bj0/WY4tpfHSJPNLCyXdWn9E3+e
         E9kDumjerqIUJjJdDha64urqkZ1AOLdRzj9pU2fiWb+NPxLZ2R+K9hs1gS2pvM4jdRZJ
         hvhrtQ6+naxGLNJrY5Mslg2O/JTw2doyM7PdA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=IhsqxX045WF2JCJE3a4CGYqdRtYT/JNWUgBlYnXmMEi0O4DY0J62lxCJolURam7d0o
         xBN4DMewyC+VopGVKIwNIzq8EA+n70KVLD3JovDGh6ddmqweIBoPpgAE/4Zo4BmlboSX
         wrIQnxB9096WJNKVU/NDbKarf96xjqpywra94=
MIME-Version: 1.0
In-Reply-To: <17e74dd00903150658n4881efe7p209948cf60a9de49@mail.gmail.com>
References: <17e74dd00903150658n4881efe7p209948cf60a9de49@mail.gmail.com>
Date: Sun, 15 Mar 2009 22:03:54 +0800
Message-ID: <17e74dd00903150703k7b19d9edg1b74030ab1cf4405@mail.gmail.com>
Subject: Fwd: [ask for help]Bad connection to FS. command aborted.
From: Xudong Du <andy2005cst@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd23e7ceaf9a7046528cdc9
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd23e7ceaf9a7046528cdc9
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

just now, I miswrite a command in the email, it is highlighted in red now.
Sorry. waiting for your help. thanks a lot.

---------- Forwarded message ----------
From: Xudong Du <andy2005cst@gmail.com>
Date: Sun, Mar 15, 2009 at 9:58 PM
Subject: [ask for help]Bad connection to FS. command aborted.
To: general@hadoop.apache.org


Hello, everyone. I met a strange problem and I am looking forward to your
help.

I setup a 2 nodes cluster hadoop on 2 mechine runing ubuntu. and 1 is both
master and slave and the other one is slave, and it seems to be functional.
but yesterday, it does not work anymore.

when I type hadoop command such as "bin/hadoop dfs -ls", it shows that "Bad
connection to FS. command aborted."
so, I run "bin/start-all.sh" to see if the dfs is shut, however it shows
that:

(***.***.***.*** stands for the master's ip address; ###.###.### stands for
the other one's ip address)
starting namenode, logging to *****
***.***.***.***: datanode running as process 14800. Stop it first.
***.***.***.***: datanode running as process 9113. Stop it first.
***.***.***.***: starting secondarynamenode, logging to ***
jobtracker running as process 14946. Stop it first.
***.***.***.***: tasktracker running as process 15029. Stop it first.
###.###.###.###: tasktracker running as process 9217. Stop it first.

and i type "bin/hadoop dfs -ls", but again: Bad connection to FS. command
aborted.

then when I type "bin/stop-all.sh(sorry, I mis-write just now )", it shows:
(***.***.***.*** stands for the master's ip address; ###.###.### stands for
the other one's ip address)
stopping jobtracker
***.***.***.***: stopping tasktracker
###.###.###: stopping tasktracker
no namenode to stop  <--------this is very strange
***.***.***.***: stopping datanode
###.###.###: stopping datanode
***.***.***.***: no secondarynamenode to stop

when i restart it by "bin/start-all.sh", it still does not work..

can anyone give me some idea? Thanks a lot.

-- 
Yours Sincerely
Xudong Du
Zijing 2# 305A
Tsinghua University, Beijing, China, 100084



-- 
Yours Sincerely
Xudong Du
Zijing 2# 305A
Tsinghua University, Beijing, China, 100084

--000e0cd23e7ceaf9a7046528cdc9--

From general-return-141-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 20 17:20:08 2009
Return-Path: <general-return-141-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38029 invoked from network); 20 Mar 2009 17:20:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 20 Mar 2009 17:20:08 -0000
Received: (qmail 44779 invoked by uid 500); 20 Mar 2009 17:20:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44762 invoked by uid 500); 20 Mar 2009 17:20:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44745 invoked by uid 99); 20 Mar 2009 17:20:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Mar 2009 10:20:07 -0700
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [132.239.1.207] (HELO outbound3.ucsd.edu) (132.239.1.207)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Mar 2009 17:19:57 +0000
Received: from iport-c1.ucsd.edu (iport-c1.ucsd.edu [132.239.0.180])
	by outbound3.ucsd.edu (8.13.6/8.13.6) with ESMTP id n2KHJZjj006996
	for <general@hadoop.apache.org>; Fri, 20 Mar 2009 10:19:36 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=2007001; d=ucsd.edu; c=simple; q=dns;
	b=l9jfJYFNSpdkW0vKEJXH7FhbRD2geFRBXDtHcZ15ppAnUQWxGiuDUUmfc0mdQxMQN
	Lqlqqh6K6NwvzX8/SrQ5w==
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AgMFAFttw0mE7wEx/2dsb2JhbACBUMQ/DYYwiE2DfQZh
X-IronPort-AV: E=Sophos;i="4.38,396,1233561600"; 
   d="scan'208";a="35499609"
X-Spam-Level: 
Received: from smtp.ucsd.edu ([132.239.1.49])
  by iport-c1-out.ucsd.edu with ESMTP/TLS/DHE-RSA-AES256-SHA; 20 Mar 2009 10:19:35 -0700
Received: from mouse.localnet (mouse.ucsd.edu [132.239.186.22])
	(authenticated bits=0)
	by smtp.ucsd.edu (8.13.6/8.13.6) with ESMTP id n2KHJYTp019341
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Fri, 20 Mar 2009 10:19:34 -0700 (PDT)
X-Authentication-Warning: smtp.ucsd.edu: Host mouse.ucsd.edu [132.239.186.22] claimed to be mouse.localnet
From: Terrence Martin <tmartin@physics.ucsd.edu>
To: general@hadoop.apache.org
Subject: Java exception when starting the namenode
Date: Fri, 20 Mar 2009 10:19:28 -0700
User-Agent: KMail/1.11.0 (Linux/2.6.27-11-generic; KDE/4.2.0; i686; ; )
MIME-Version: 1.0
Content-Type: text/plain;
  charset="us-ascii"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Message-Id: <200903201019.29128.tmartin@physics.ucsd.edu>
X-Virus-Checked: Checked by ClamAV on apache.org
X-Old-Spam-Status: No

I am seeing the following exception when I attempt to start my Hadoop namenode. This is a namenode that was working and the system had a few TB of files in it. Yesterday we were attempting to upgrade it to a more recent version and things appeared to be working. 

2009-03-20 10:14:02,667 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:  
/************************************************************                               
STARTUP_MSG: Starting NameNode                                                              
STARTUP_MSG:   host = proxy-1.t2.ucsd.edu/169.228.130.63                                    
STARTUP_MSG:   args = []                                                                    
STARTUP_MSG:   version = 0.19.2-dev                                                         
STARTUP_MSG:   build = http://svn.apache.org/repos/asf/hadoop/core/tags/release-0.19.1 -r 748415; compiled by 'sl1-user' on Tue Mar 17 10:49:06 CDT 2009                                                                                                                                
************************************************************/                                                                               
2009-03-20 10:14:02,772 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=NameNode, port=9000           
2009-03-20 10:14:02,776 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Namenode up at: proxy-1.t2.ucsd.edu/169.228.130.63:9000       
2009-03-20 10:14:02,778 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=NameNode, sessionId=Hadoop 
2009-03-20 10:14:02,783 INFO org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics: Initializing NameNodeMeterics using context object:org.apache.hadoop.metrics.spi.NullContext                                                                                               
2009-03-20 10:14:02,832 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=root,root,bin,daemon,sys,adm,disk,wheel           
2009-03-20 10:14:02,832 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup                                     
2009-03-20 10:14:02,832 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled=true                                  
2009-03-20 10:14:02,839 INFO org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics: Initializing FSNamesystemMetrics using context object:org.apache.hadoop.metrics.spi.NullContext                                                                                        
2009-03-20 10:14:02,840 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemStatusMBean                        
2009-03-20 10:14:02,866 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files = 1435                                           
2009-03-20 10:14:03,061 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files under construction = 3                           
2009-03-20 10:14:03,065 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 361956 loaded in 0 seconds.
2009-03-20 10:14:03,074 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.lang.NullPointerException
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addChild(FSDirectory.java:1006)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addNode(FSDirectory.java:982)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedAddFile(FSDirectory.java:194)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:613)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:973)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:793)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:352)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:309)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:288)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:163)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:208)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:194)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:859)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:868)

2009-03-20 10:14:03,075 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at proxy-1.t2.ucsd.edu/169.228.130.63
************************************************************/

Could this be some sort of corruption on loading the image? 

Terrence

From general-return-142-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 23 18:07:34 2009
Return-Path: <general-return-142-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31334 invoked from network); 23 Mar 2009 18:07:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Mar 2009 18:07:34 -0000
Received: (qmail 86123 invoked by uid 500); 23 Mar 2009 18:07:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86080 invoked by uid 500); 23 Mar 2009 18:07:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86070 invoked by uid 99); 23 Mar 2009 18:07:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Mar 2009 18:07:34 +0000
X-ASF-Spam-Status: No, hits=4.2 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.62.56] (HELO QMTA06.westchester.pa.mail.comcast.net) (76.96.62.56)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Mar 2009 18:07:23 +0000
Received: from OMTA09.westchester.pa.mail.comcast.net ([76.96.62.20])
	by QMTA06.westchester.pa.mail.comcast.net with comcast
	id WcvW1b0060SCNGk56i6rxp; Mon, 23 Mar 2009 18:06:51 +0000
Received: from battlerock-lm.corp.yahoo.com ([209.131.62.113])
	by OMTA09.westchester.pa.mail.comcast.net with comcast
	id Wi6Z1b00R2SbwD53Vi6f6i; Mon, 23 Mar 2009 18:06:54 +0000
Message-Id: <297EBC6C-D1E4-4070-8010-C349F6F3624C@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=Apple-Mail-1--321129169
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: ApacheCon EU 2009 this week
Date: Mon, 23 Mar 2009 11:06:32 -0700
Cc: core-user@hadoop.apache.org
X-Mailer: Apple Mail (2.930.3)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-1--321129169
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit

  ApacheCon EU 2009 is in Amsterdam this week, with a lot of talks on  
Hadoop. There are also going to be a lot of the committers there,  
including Doug Cutting. There is no word yet whether he is bringing  
the "original" Hadoop as seen in the NY Times.

This year the live video streaming includes the Hadoop track. The  
video of the keynote and lunch talks are free, but there is a charge  
for the Hadoop track.

The Hadoop track this year includes:

     * Opening Keynote - Data Management in the Cloud - Raghu  
Ramakrishnan
     * Introduction to Hadoop - Owen O'Malley
     * Hadoop Map-Reduce: Tuning and Debugging - Arun Murthy
     * Pig: Making Hadoop Easy - Olga Natkovich
     * Running Hadoop in the Cloud - Tom White
     * Configuring Hadoop for Grid Services - Allen Wittenauer
     * Dynamic Hadoop Clusters - Steve Loughran

-- Owen


--Apple-Mail-1--321129169--

From general-return-143-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 23 20:31:18 2009
Return-Path: <general-return-143-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 89465 invoked from network); 23 Mar 2009 20:31:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Mar 2009 20:31:17 -0000
Received: (qmail 8991 invoked by uid 500); 23 Mar 2009 20:31:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8880 invoked by uid 500); 23 Mar 2009 20:31:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8808 invoked by uid 99); 23 Mar 2009 20:31:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Mar 2009 20:31:14 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 74.125.44.30 as permitted sender)
Received: from [74.125.44.30] (HELO yx-out-2324.google.com) (74.125.44.30)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Mar 2009 20:31:06 +0000
Received: by yx-out-2324.google.com with SMTP id 3so1512226yxj.29
        for <multiple recipients>; Mon, 23 Mar 2009 13:30:45 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:message-id:subject:from:to:cc
         :content-type;
        bh=XnGBgm3k/u+MIglb9A4Grot+0RAUHxS2sFd7zAp+ghY=;
        b=QRBI2emrVU1u3fQeRZ3CSWp2eQ2fczypXZU0bijvwkS+zLU9W/0Nu1iTgpOpURNYX3
         OCe8EDQIE3T2vB6qzTigoQlGwQh6r41MbHkS5Xu0LPimrxhpGn8USWSZwyTmmoJiKqaU
         w9jqJdXfz3zpRxe2tH2YbJnhKEsRHrTRRkmek=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:cc:content-type;
        b=lWRHAt4t/TJjpZQ5cnbOfkq8MQuldG+/+BVOq070BWNynZpmoB3+jEq7giUFqmMADi
         cLNWq+BFTfTMno56OVhqYh6umsaEd8wLzvdJuadpHI7BX9e32XuuK6lyAHht0o1WTXxf
         XsBz97oRgeEYDoMFwxpSrjFAYZa7AfT5CHK/4=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
Received: by 10.100.133.16 with SMTP id g16mr6989933and.136.1237840245083; 
	Mon, 23 Mar 2009 13:30:45 -0700 (PDT)
In-Reply-To: <297EBC6C-D1E4-4070-8010-C349F6F3624C@apache.org>
References: <297EBC6C-D1E4-4070-8010-C349F6F3624C@apache.org>
Date: Mon, 23 Mar 2009 13:30:45 -0700
X-Google-Sender-Auth: 23f2cc32eedee571
Message-ID: <7c962aed0903231330y6c62663fx42c8eae57e5d847@mail.gmail.com>
Subject: Re: ApacheCon EU 2009 this week
From: stack <stack@duboce.net>
To: core-user@hadoop.apache.org
Cc: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e646526618d8a20465cf2401
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e646526618d8a20465cf2401
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Riding on the below's coat-tails, there'll also be a talk on HBase on the
wednesday at 3pm.
Thanks,
St.Ack

On Mon, Mar 23, 2009 at 11:06 AM, Owen O'Malley <omalley@apache.org> wrote:

>  ApacheCon EU 2009 is in Amsterdam this week, with a lot of talks on
> Hadoop. There are also going to be a lot of the committers there, including
> Doug Cutting. There is no word yet whether he is bringing the "original"
> Hadoop as seen in the NY Times.
>
> This year the live video streaming includes the Hadoop track. The video of
> the keynote and lunch talks are free, but there is a charge for the Hadoop
> track.
>
> The Hadoop track this year includes:
>
>    * Opening Keynote - Data Management in the Cloud - Raghu Ramakrishnan
>    * Introduction to Hadoop - Owen O'Malley
>    * Hadoop Map-Reduce: Tuning and Debugging - Arun Murthy
>    * Pig: Making Hadoop Easy - Olga Natkovich
>    * Running Hadoop in the Cloud - Tom White
>    * Configuring Hadoop for Grid Services - Allen Wittenauer
>    * Dynamic Hadoop Clusters - Steve Loughran
>
> -- Owen
>
>

--0016e646526618d8a20465cf2401--

From general-return-144-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 24 21:45:07 2009
Return-Path: <general-return-144-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72216 invoked from network); 24 Mar 2009 21:45:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2009 21:45:07 -0000
Received: (qmail 57045 invoked by uid 500); 24 Mar 2009 21:45:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56996 invoked by uid 500); 24 Mar 2009 21:45:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56986 invoked by uid 99); 24 Mar 2009 21:45:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Mar 2009 21:45:07 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of gmackey@cs.ucf.edu designates 132.170.108.1 as permitted sender)
Received: from [132.170.108.1] (HELO longwood.cs.ucf.edu) (132.170.108.1)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Mar 2009 21:44:58 +0000
Received: from longwood.cs.ucf.edu (localhost [127.0.0.1])
	by longwood.cs.ucf.edu (8.14.1/8.14.1) with ESMTP id n2OLi3tH020480
	for <general@hadoop.apache.org>; Tue, 24 Mar 2009 17:44:26 -0400 (EDT)
Received: (from apache@localhost)
	by longwood.cs.ucf.edu (8.14.1/8.14.1/Submit) id n2OLi3H2020479
	for general@hadoop.apache.org; Tue, 24 Mar 2009 17:44:03 -0400 (EDT)
X-Authentication-Warning: longwood.cs.ucf.edu: apache set sender to gmackey@cs.ucf.edu using -f
Received: from 10.173.214.212 ([10.173.214.212]) by mail.cs.ucf.edu (Horde
 Framework) with HTTP; Tue, 24 Mar 2009 17:44:03 -0400
Message-ID: <20090324174403.83006e4jos7t6tk0@mail.cs.ucf.edu>
Date: Tue, 24 Mar 2009 17:44:03 -0400
From: Grant Mackey <gmackey@cs.ucf.edu>
To: general@hadoop.apache.org
Subject: Re: Fwd: [ask for help]Bad connection to FS. command aborted.
References: <17e74dd00903150658n4881efe7p209948cf60a9de49@mail.gmail.com>
 <17e74dd00903150703k7b19d9edg1b74030ab1cf4405@mail.gmail.com>
In-Reply-To: <17e74dd00903150703k7b19d9edg1b74030ab1cf4405@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain;
 charset=ISO-8859-1;
 DelSp="Yes";
 format="flowed"
Content-Disposition: inline
Content-Transfer-Encoding: 7bit
User-Agent: Internet Messaging Program (IMP) 4.3.3
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on gondor.cs.ucf.edu
X-Virus-Checked: Checked by ClamAV on apache.org
X-Old-Spam-Status: No, score=-0.8 required=6.0 tests=ALL_TRUSTED,J_CHICKENPOX_43
	autolearn=no version=3.2.5

Is this your first time using Hadoop? or was this cluster running and  
then broke? were you trying to update hadoop?

If this is the first time you've run it, then did you format the  
namenode? if not then run bin/hadoop namenode -format.

start the cluster up using start-dfs.sh, that way you don't have to  
worry abou the job/tasktrackers...

log into whatever node you have established as the master and simply  
run bin/hadoop namenode to see what error reporting you get. generally  
this will give you a better idea of what is happening. let me know the  
output of the namenode before I can help.

thanks

  - Grant

Quoting Xudong Du <andy2005cst@gmail.com>:

> just now, I miswrite a command in the email, it is highlighted in red now.
> Sorry. waiting for your help. thanks a lot.
>
> ---------- Forwarded message ----------
> From: Xudong Du <andy2005cst@gmail.com>
> Date: Sun, Mar 15, 2009 at 9:58 PM
> Subject: [ask for help]Bad connection to FS. command aborted.
> To: general@hadoop.apache.org
>
>
> Hello, everyone. I met a strange problem and I am looking forward to your
> help.
>
> I setup a 2 nodes cluster hadoop on 2 mechine runing ubuntu. and 1 is both
> master and slave and the other one is slave, and it seems to be functional.
> but yesterday, it does not work anymore.
>
> when I type hadoop command such as "bin/hadoop dfs -ls", it shows that "Bad
> connection to FS. command aborted."
> so, I run "bin/start-all.sh" to see if the dfs is shut, however it shows
> that:
>
> (***.***.***.*** stands for the master's ip address; ###.###.### stands for
> the other one's ip address)
> starting namenode, logging to *****
> ***.***.***.***: datanode running as process 14800. Stop it first.
> ***.***.***.***: datanode running as process 9113. Stop it first.
> ***.***.***.***: starting secondarynamenode, logging to ***
> jobtracker running as process 14946. Stop it first.
> ***.***.***.***: tasktracker running as process 15029. Stop it first.
> ###.###.###.###: tasktracker running as process 9217. Stop it first.
>
> and i type "bin/hadoop dfs -ls", but again: Bad connection to FS. command
> aborted.
>
> then when I type "bin/stop-all.sh(sorry, I mis-write just now )", it shows:
> (***.***.***.*** stands for the master's ip address; ###.###.### stands for
> the other one's ip address)
> stopping jobtracker
> ***.***.***.***: stopping tasktracker
> ###.###.###: stopping tasktracker
> no namenode to stop  <--------this is very strange
> ***.***.***.***: stopping datanode
> ###.###.###: stopping datanode
> ***.***.***.***: no secondarynamenode to stop
>
> when i restart it by "bin/start-all.sh", it still does not work..
>
> can anyone give me some idea? Thanks a lot.
>
> --
> Yours Sincerely
> Xudong Du
> Zijing 2# 305A
> Tsinghua University, Beijing, China, 100084
>
>
>
> --
> Yours Sincerely
> Xudong Du
> Zijing 2# 305A
> Tsinghua University, Beijing, China, 100084
>



Grant Mackey
UCF Research Assistant
Engineering III
Rm 238 Cubicle 1

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.


From general-return-145-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 24 21:49:43 2009
Return-Path: <general-return-145-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 75728 invoked from network); 24 Mar 2009 21:49:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2009 21:49:43 -0000
Received: (qmail 67391 invoked by uid 500); 24 Mar 2009 21:49:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67331 invoked by uid 500); 24 Mar 2009 21:49:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67321 invoked by uid 99); 24 Mar 2009 21:49:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Mar 2009 21:49:42 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [96.9.130.69] (HELO sophia.codeminders.com) (96.9.130.69)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Mar 2009 21:49:33 +0000
Received: from [172.16.1.53] (pool-71-116-87-51.snfcca.dsl-w.verizon.net [71.116.87.51])
	(using TLSv1 with cipher AES128-SHA (128/128 bits))
	(No client certificate requested)
	by sophia.codeminders.com (Postfix) with ESMTPSA id 3770281865
	for <general@hadoop.apache.org>; Tue, 24 Mar 2009 14:50:39 -0700 (PDT)
Message-Id: <AB80A6F7-24B5-44C8-82C7-89F7452838A7@codeminders.com>
From: Vadim Zaliva <lord@codeminders.com>
To: general@hadoop.apache.org
In-Reply-To: <20090324174403.83006e4jos7t6tk0@mail.cs.ucf.edu>
Content-Type: multipart/signed; boundary=Apple-Mail-31--221370935; micalg=sha1; protocol="application/pkcs7-signature"
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: Re: [ask for help]Bad connection to FS. command aborted.
Date: Tue, 24 Mar 2009 14:49:10 -0700
References: <17e74dd00903150658n4881efe7p209948cf60a9de49@mail.gmail.com> <17e74dd00903150703k7b19d9edg1b74030ab1cf4405@mail.gmail.com> <20090324174403.83006e4jos7t6tk0@mail.cs.ucf.edu>
X-Mailer: Apple Mail (2.930.3)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-31--221370935
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


On Mar 24, 2009, at 14:44 , Grant Mackey wrote:

> Is this your first time using Hadoop? or was this cluster running  
> and then broke? were you trying to update hadoop?


Grant,

The cluster is running for a while and I have quite a few tasks which  
I am executing regularly. It is just and
intermittent problem which I observe only with PIG on some of tasks.

 From googling it seems like some deadlock bug in Java threading which  
people reported on some other java programs,
not only pig/hadoop.

Vadim

--
"La perfection est atteinte non quand il ne reste rien a ajouter, mais
quand il ne reste rien a enlever."  (Antoine de Saint-Exupery)




--Apple-Mail-31--221370935
Content-Disposition: attachment;
	filename=smime.p7s
Content-Type: application/pkcs7-signature;
	name=smime.p7s
Content-Transfer-Encoding: base64

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIGQDCCAvkw
ggJioAMCAQICEERMoY4Lfo7a7sLhgKLeLcswDQYJKoZIhvcNAQEFBQAwYjELMAkGA1UEBhMCWkEx
JTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQ
ZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA4MDgwNDE4MDY1MloXDTA5MDgwNDE4MDY1
MlowXTEPMA0GA1UEBBMGWmFsaXZhMQ4wDAYDVQQqEwVWYWRpbTEVMBMGA1UEAxMMVmFkaW0gWmFs
aXZhMSMwIQYJKoZIhvcNAQkBFhRsb3JkQGNvZGVtaW5kZXJzLmNvbTCCASIwDQYJKoZIhvcNAQEB
BQADggEPADCCAQoCggEBAJp/hSCd9qKL3z67jLQ/rBeXMRgkL43ZL7wTXjUnS8OfVGYeRT2jLN06
JEouCV0ndiQwDT1Am8xq2IdZC1h/CEZJGrvj663uvrxFCadh2pqT2pFnq7VCiBsekFyc/uhE8NOF
It8doKZmzKrQQJ2iX94EdhEIP/mACO3+n9j0TBqSa7z4hfn8DYNW6MkZX4VGlcAm/d3ueTlcoiQ5
lrcTw3i5aK4SBTiReLaHQKoyU2FYvk0A3728MQjcU4sIJDpUTKCnzLMLmJHaSgdq9Ey7nDSUKb0F
CHVjsgp+dr/6ozRgmvIcs+WfEo1ELBuFfi+U6ESXlOy26qH3coOS7LXT9gsCAwEAAaMxMC8wHwYD
VR0RBBgwFoEUbG9yZEBjb2RlbWluZGVycy5jb20wDAYDVR0TAQH/BAIwADANBgkqhkiG9w0BAQUF
AAOBgQCbuIPK+DCnk/9WGG2Wv1kdmqj41IjBR7oXQ5GsWfDi6uKGWdzz/Z628YcirynXqr3bgJze
IvJ5XbkFuGe0SHl5HsXtKBeTNjqBS+lZQ8+eXkxuJTrFGzvn72EhY1E1NmYntVaCblizQmZIu/aq
zScU5ibqWqntjn8P7Iu3GSE9QTCCAz8wggKooAMCAQICAQ0wDQYJKoZIhvcNAQEFBQAwgdExCzAJ
BgNVBAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJuIENhcGUxEjAQBgNVBAcTCUNhcGUgVG93bjEaMBgG
A1UEChMRVGhhd3RlIENvbnN1bHRpbmcxKDAmBgNVBAsTH0NlcnRpZmljYXRpb24gU2VydmljZXMg
RGl2aXNpb24xJDAiBgNVBAMTG1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBDQTErMCkGCSqGSIb3
DQEJARYccGVyc29uYWwtZnJlZW1haWxAdGhhd3RlLmNvbTAeFw0wMzA3MTcwMDAwMDBaFw0xMzA3
MTYyMzU5NTlaMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5
KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBDQTCBnzAN
BgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAxKY8VXNV+065yplaHmjAdQRwnd/p/6Me7L3N9VvyGna9
fww6YfK/Uc4B1OVQCjDXAmNaLIkVcI7dyfArhVqqP3FWy688Cwfn8R+RNiQqE88r1fOCdz0Dviv+
uxg+B79AgAJk16emu59l0cUqVIUPSAR/p7bRPGEEQB5kGXJgt/sCAwEAAaOBlDCBkTASBgNVHRMB
Af8ECDAGAQH/AgEAMEMGA1UdHwQ8MDowOKA2oDSGMmh0dHA6Ly9jcmwudGhhd3RlLmNvbS9UaGF3
dGVQZXJzb25hbEZyZWVtYWlsQ0EuY3JsMAsGA1UdDwQEAwIBBjApBgNVHREEIjAgpB4wHDEaMBgG
A1UEAxMRUHJpdmF0ZUxhYmVsMi0xMzgwDQYJKoZIhvcNAQEFBQADgYEASIzRUIPqCy7MDaNmrGcP
f6+svsIXoUOWlJ1/TCG4+DYfqi2fNi/A9BxQIJNwPP2t4WFiw9k6GX6EsZkbAMUaC4J0niVQlGLH
2ydxVyWN3amcOY6MIE9lX5Xa9/eH1sYITq726jTlEBpbNU1341YheILcIRk13iSx0x1G/11fZU8x
ggMQMIIDDAIBATB2MGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAo
UHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBDQQIQ
REyhjgt+jtruwuGAot4tyzAJBgUrDgMCGgUAoIIBbzAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcB
MBwGCSqGSIb3DQEJBTEPFw0wOTAzMjQyMTQ5MTBaMCMGCSqGSIb3DQEJBDEWBBQ1J0t6V/CYnXPs
eTb0piJz3/nNvDCBhQYJKwYBBAGCNxAEMXgwdjBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhh
d3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVt
YWlsIElzc3VpbmcgQ0ECEERMoY4Lfo7a7sLhgKLeLcswgYcGCyqGSIb3DQEJEAILMXigdjBiMQsw
CQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UE
AxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0ECEERMoY4Lfo7a7sLhgKLeLcsw
DQYJKoZIhvcNAQEBBQAEggEAF37XECorc9hNCovIvFnn2KtWIJzGUPe9DTebL7e6QO9DiMh6QWbg
9L5AQW1roTUzhoDazY+MGI1apZl5Vm2Q0LmbWTYy1HxHhpz0b6id0MCpMkqLhp2rc500RHU0aNMr
E4AOw191KeNMXc8PPcMCZptTbMUQj69x0BjvXNi26Vrnd9LQlcj1h0sBOfWax2VlKFod74ZeKlON
jpGT9xkYF1NaFe773hQbGtOJC7NKzKgds1NGKdfmoWhZNnxKWZ8BiaM7zfybi81Idhaa7lRcUjA5
u98vXYN1QzNoVYSOOWcIf4nZWK1maUf86/5i8Gy8C1zAGfVfSx5z9m/vAHquWAAAAAAAAA==

--Apple-Mail-31--221370935--

From general-return-146-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 27 18:00:59 2009
Return-Path: <general-return-146-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 40946 invoked from network); 27 Mar 2009 18:00:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Mar 2009 18:00:59 -0000
Received: (qmail 80536 invoked by uid 500); 27 Mar 2009 18:00:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80192 invoked by uid 500); 27 Mar 2009 18:00:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 79822 invoked by uid 99); 27 Mar 2009 18:00:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Mar 2009 18:00:56 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [207.126.228.150] (HELO rsmtp2.corp.yahoo.com) (207.126.228.150)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Mar 2009 18:00:48 +0000
Received: from [172.21.148.171] (wlanvpn-mc2e-246-171.corp.yahoo.com [172.21.148.171])
	(authenticated bits=0)
	by rsmtp2.corp.yahoo.com (8.13.8/8.13.8/y.rout) with ESMTP id n2RI0KVI065014
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Fri, 27 Mar 2009 11:00:21 -0700 (PDT)
Message-ID: <49CD1434.2040502@apache.org>
Date: Fri, 27 Mar 2009 11:00:20 -0700
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.21 (X11/20090318)
MIME-Version: 1.0
To: announce@apache.org,
        "zookeeper-dev@hadoop.apache.org" <zookeeper-dev@hadoop.apache.org>,
        "zookeeper-user@hadoop.apache.org" <zookeeper-user@hadoop.apache.org>,
        core-user@hadoop.apache.org, general@hadoop.apache.org
Subject: [ANNOUNCE] Apache ZooKeeper 3.1.1
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

The Apache ZooKeeper team is proud to announce Apache ZooKeeper version 
3.1.1.

ZooKeeper is a high-performance coordination service for distributed 
applications. It exposes common services - such as naming, configuration 
management, synchronization, and group services - in a simple interface 
so you don't have to write them from scratch. You can use it 
off-the-shelf to implement consensus, group management, leader election, 
and presence protocols. And you can build on it for your own, specific 
needs.

If you are upgrading from version 2.2.1 on SourceForge be sure to review 
the 3.0.1 release notes for migration instructions.

For ZooKeeper release details and downloads, visit:
http://hadoop.apache.org/zookeeper/releases.html

ZooKeeper 3.1.1 Release Notes are at:
http://hadoop.apache.org/zookeeper/docs/r3.1.1/releasenotes.html

Regards,

The ZooKeeper Team

From general-return-147-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 31 14:31:53 2009
Return-Path: <general-return-147-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38358 invoked from network); 31 Mar 2009 14:31:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2009 14:31:52 -0000
Received: (qmail 76115 invoked by uid 500); 31 Mar 2009 14:31:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76066 invoked by uid 500); 31 Mar 2009 14:31:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 99372 invoked by uid 99); 31 Mar 2009 03:50:04 -0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of amaltasb@gmail.com designates 209.85.218.176 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=j+b46bmN9eLRGGVB6+jya+/tQ7W/htfHvYc2AihQW+8=;
        b=Ou18G5ZbQ8TD5mQMNPsTXIMZ0OAGCZ7xWBWyqSMsR5dRI9UCaWncWiKdWAlv4niwZo
         b7KB3HPnhr7YVrYMeRHAC5GKkhGRheXyef/19MwIPSfM3fg2z/2PHJkTjx2vjsdviZW9
         105J3iDxftlrcXyR9UcaQphSTvgAP2AgSofVo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=PZwyR3tyBEc0BMsbZhR9DYziFSyjh/kVRQdSTVQ655JxgNM4FAGj5PbOwnVT0TvEy+
         OV7dFLSKEzfzDWD2dxqSQ9dg7PHH2ntrGTZxGhtQeicCoVKmLhOA/ta23wsqalVYB1y/
         L4eHbuTQ28Nj90DZHEiob2fYypEO+LywbY8bo=
MIME-Version: 1.0
Date: Tue, 31 Mar 2009 09:19:36 +0530
Message-ID: <1a2551840903302049u28744c78t2c94ef8d29c8562b@mail.gmail.com>
Subject: Berkeley DB as Hadoop Input
From: Amal <amaltasb@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502de86763f54046662167f
X-Virus-Checked: Checked by ClamAV on apache.org

--00504502de86763f54046662167f
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

I have around 3 million rows of data stored in Berkeley DB Java Edition. Is
it possible to use this database as a MapReduce Input to Hadoop. If not,
what is the alternative storage engine, mainly for storing key/value pairs
(no relational features or transactional support needed), so that I can run
mapreduce on it.

Thanks

--00504502de86763f54046662167f--

From general-return-1007-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 01 21:44:24 2010
Return-Path: <general-return-1007-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22040 invoked from network); 1 Feb 2010 21:44:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Feb 2010 21:44:23 -0000
Received: (qmail 88280 invoked by uid 500); 1 Feb 2010 21:44:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87507 invoked by uid 500); 1 Feb 2010 21:44:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87221 invoked by uid 99); 1 Feb 2010 21:44:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 21:44:20 +0000
X-ASF-Spam-Status: No, hits=-1.8 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of swatt@us.ibm.com designates 32.97.110.153 as permitted sender)
Received: from [32.97.110.153] (HELO e35.co.us.ibm.com) (32.97.110.153)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 21:44:08 +0000
Received: from d03relay01.boulder.ibm.com (d03relay01.boulder.ibm.com [9.17.195.226])
	by e35.co.us.ibm.com (8.14.3/8.13.1) with ESMTP id o11LTaOV007374;
	Mon, 1 Feb 2010 14:29:36 -0700
Received: from d03av06.boulder.ibm.com (d03av06.boulder.ibm.com [9.17.195.245])
	by d03relay01.boulder.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id o11Lhgu3170848;
	Mon, 1 Feb 2010 14:43:42 -0700
Received: from d03av06.boulder.ibm.com (loopback [127.0.0.1])
	by d03av06.boulder.ibm.com (8.14.3/8.13.1/NCO v10.0 AVout) with ESMTP id o11LjrL7028174;
	Mon, 1 Feb 2010 14:45:53 -0700
Received: from d03nm123.boulder.ibm.com (d03nm123.boulder.ibm.com [9.17.195.149])
	by d03av06.boulder.ibm.com (8.14.3/8.13.1/NCO v10.0 AVin) with ESMTP id o11LjrVJ028171;
	Mon, 1 Feb 2010 14:45:53 -0700
To: general@hadoop.apache.org, common-user@hadoop.apache.org
MIME-Version: 1.0
Subject: First Official Austin Hadoop User Group - March 18th
X-KeepSent: F3BD88CD:7E5F1E7B-862576BD:0075427B;
 type=4; name=$KeepSent
X-Mailer: Lotus Notes Release 8.5 December 05, 2008
Message-ID: <OFF3BD88CD.7E5F1E7B-ON862576BD.0075427B-862576BD.00775B15@us.ibm.com>
From: Stephen Watt <swatt@us.ibm.com>
Date: Mon, 1 Feb 2010 15:43:41 -0600
X-MIMETrack: Serialize by Router on D03NM123/03/M/IBM(Release 8.0.1|February 07, 2008) at
 02/01/2010 14:43:42,
	Serialize complete at 02/01/2010 14:43:42
Content-Type: multipart/alternative; boundary="=_alternative 00775B14862576BD_="
X-Virus-Checked: Checked by ClamAV on apache.org

--=_alternative 00775B14862576BD_=
Content-Type: text/plain; charset="US-ASCII"

Hi Folks

This note is to let you know that we'll be kicking off the inaugural 
Austin Hadoop User Group on March the 18th. At present, we have speakers 
lined up from IBM and Rackspace and will cover quite a wide variety of 
topics along with a few demos. This event will follow directly on the 
heels of the SXSW Interactive conference. Pizza, beer and soft drinks will 
be provided BUT we are still looking for a location to host the meeting 
somewhere downtown or reasonably close to it. Do you work for a company 
that would be interesting in sponsoring a location ? I believe we're 
likely to have within 15 to 30 attendees initially so we'd need a large 
room with a projector.

The initial sentiment, based on discussions I've had with people 
interested in attending, is that the group will cover both Hadoop and Big 
Data concerns in general. If you'd like to attend, present a topic or are 
interested in sponsorship, please send me a note directly so I can put you 
on the distribution list (and it will also help me with catering). I'll 
also be posting the event on the door64.com event calendar.

Kind regards
Steve Watt
swatt@us.ibm.com
--=_alternative 00775B14862576BD_=--

From general-return-1008-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 02 11:13:40 2010
Return-Path: <general-return-1008-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95353 invoked from network); 2 Feb 2010 11:13:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Feb 2010 11:13:40 -0000
Received: (qmail 68304 invoked by uid 500); 2 Feb 2010 11:13:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68233 invoked by uid 500); 2 Feb 2010 11:13:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68223 invoked by uid 99); 2 Feb 2010 11:13:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Feb 2010 11:13:38 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Feb 2010 11:13:26 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 4D07EB7E03
	for <general@hadoop.apache.org>; Tue,  2 Feb 2010 11:13:05 +0000 (GMT)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id VM6-a1vZyqLI for <general@hadoop.apache.org>;
	Tue,  2 Feb 2010 11:12:59 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 6F999B7D88
	for <general@hadoop.apache.org>; Tue,  2 Feb 2010 11:12:59 +0000 (GMT)
MailScanner-NULL-Check: 1265713968.93393@tLPYYLOHCUebq0mQtJtkuQ
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o12BClEV024092
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Tue, 2 Feb 2010 11:12:48 GMT
Message-ID: <4B6808AE.9000106@apache.org>
Date: Tue, 02 Feb 2010 11:12:46 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090812)
MIME-Version: 1.0
To: Hadoop General <general@hadoop.apache.org>
Subject: HPLabs Palo Alto internship opportunity
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o12BClEV024092
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org


I am pleased to forward on this announcement of an opportunity for a 
student to spend some time at HPLaboratories in California doing fun 
things. This isn't the bit of HPLabs where I work (I'm based in the EU), 
but these are some of my colleagues who are doing some really 
interesting stuff.

As it's not my group, I'm not in a position to field questions about it, 
-please follow the link at the bottom of the posting

-Steve

----------

In HP Labs at Palo Alto (http://www.hpl.hp.com/), we are currently 
seeking qualified self-motivated MS and PhD students for the following 
intern position in software and systems research.

This position is in the area of large-scale data management, with the 
focus of developing a scalable platform as a cloud service to store and 
manage data and enforce data management policies. This cloud service 
aims to help service providers protect outsourced customer data and meet 
compliance requirements. The involved data policies include data 
retention, data availability, data integrity, data migration, and data 
access control, among others.

This position requires deep knowledge and hands-on experience with 
distributed systems and service-oriented computing, which include 
database, transactional and parallel processing, high availability and 
fault tolerance, and other related disciplines. This position also 
requires strong object-oriented programming and web development skills 
(Java related platforms preferred). Experience with Hadoop and 
Map/Reduce and knowledge of secure computing (e.g., identity management, 
access control) are desired but not required.  Qualified candidates must 
have excellent communication skills and the ability to work 
independently and in a team.

To apply to this position, please visit 
https://hp.taleo.net/careersection/2/jobsearch.ftl?lang=en, and at the 
"Job Number" field, enter the number "385468". Then press the button 
"Search for Jobs". The search result will bring up this specific 
position related page, from which the online application to this 
position can be started.


From general-return-1009-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 02 16:51:56 2010
Return-Path: <general-return-1009-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21139 invoked from network); 2 Feb 2010 16:51:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Feb 2010 16:51:55 -0000
Received: (qmail 44817 invoked by uid 500); 2 Feb 2010 16:51:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44245 invoked by uid 500); 2 Feb 2010 16:51:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44056 invoked by uid 99); 2 Feb 2010 16:51:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Feb 2010 16:51:52 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [88.198.2.104] (HELO koch.ro) (88.198.2.104)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Feb 2010 16:51:41 +0000
Received: from 84-72-85-88.dclient.hispeed.ch ([84.72.85.88] helo=jona.localnet)
	by koch.ro with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
	(Exim 4.69)
	(envelope-from <thomas@koch.ro>)
	id 1NcLo5-0008RD-PY; Tue, 02 Feb 2010 17:40:53 +0100
From: Thomas Koch <thomas@koch.ro>
Reply-To: thomas@koch.ro
To: general@hadoop.apache.org,
 katta-developer@lists.sourceforge.net,
 debian-java@lists.debian.org,
 common-user@hadoop.apache.org
Subject: [RFH][Announce] hadoop on its way into Debian
Date: Tue, 2 Feb 2010 17:51:14 +0100
User-Agent: KMail/1.12.4 (Linux/2.6.31-1-amd64; KDE/4.3.4; x86_64; ; )
MIME-Version: 1.0
Content-Type: Text/Plain;
  charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Message-Id: <201002021751.14884.thomas@koch.ro>
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

there's an attempt, to get hadoop into the Debian Linux distribution. For n=
ow,=20
this is more a pre-announce, since the package still has to pass some revie=
w.=20
But you may already want to add your review and comments to the current sta=
te:

Packaging repository:
http://git.debian.org/?p=3Dpkg-java/hadoop.git

Source package:
http://mentors.debian.net/debian/pool/main/h/hadoop

No binary package for now. If you do not now, how to build the binary packa=
ge,=20
you may most likely want to wait until other have reviewed the packaging.

=46rom the TODO list/wishlist:

* Package native libraries
* Package more contribs
* Work with upstream on a build system, that can be reused for packaging
* Test the package for reliability
* Cleanup the shell scripts
* Add logrotation to everything in /var/log/hadoop

You can meet me next weekend at fosdem, if you like:
http://www.koch.ro/blog/index.php?/archives/136-Going-to-FOSDEM-2010.html

Thanks to cloudera for their initial work on debian packaging!

Best regards,

Thomas Koch, http://www.koch.ro

From general-return-1010-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 02 23:46:27 2010
Return-Path: <general-return-1010-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34796 invoked from network); 2 Feb 2010 23:46:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Feb 2010 23:46:27 -0000
Received: (qmail 35622 invoked by uid 500); 2 Feb 2010 23:46:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 35262 invoked by uid 500); 2 Feb 2010 23:46:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 35192 invoked by uid 99); 2 Feb 2010 23:46:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Feb 2010 23:46:23 +0000
X-ASF-Spam-Status: No, hits=4.0 required=10.0
	tests=HTML_MESSAGE,SPF_PASS,SUBJ_ALL_CAPS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cabiwan@gmail.com designates 209.85.220.224 as permitted sender)
Received: from [209.85.220.224] (HELO mail-fx0-f224.google.com) (209.85.220.224)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Feb 2010 23:46:17 +0000
Received: by fxm24 with SMTP id 24so71222fxm.11
        for <general@hadoop.apache.org>; Tue, 02 Feb 2010 15:45:55 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=t6LkpfpMdX1KLS8ifCuIqURDr8qTYJQdOPJcdgPNE3Q=;
        b=YPHCtZJ7NufgByeu3BwDiSICyaUClRZQz//N96UETQWcFcGRbwpwQX+Vq9Ut1/LSZ7
         QpUZJniX5ZZESv7HGnGRDAH+9plfI3shQEgE2dZC6Kq/6faq/qjfvHJgIy5OOdHuSi4b
         CKmxxRGLne8UDMpxwlQy0ctJ+CE5iFd7YVACo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=wgV6gNviyU8yJNqyNMD6uie/4DN+V6VGKzyaoPM7DCkh5L8uYYxodBsxiXy+zW2SaV
         mIZYAlTXR4Xfjw2Qg1guJgzlaCsF3FCBE7+DIcbmqD06UCkg4tSGi6ZpH8AUcKGYDpEl
         jPzbuPc6v3CDSXtsUNpscZMzn/WMm9vJy8sFw=
MIME-Version: 1.0
Received: by 10.103.48.38 with SMTP id a38mr4568570muk.37.1265154355812; Tue, 
	02 Feb 2010 15:45:55 -0800 (PST)
Date: Wed, 3 Feb 2010 00:45:55 +0100
Message-ID: <309f76d01002021545n4170af2dr2120bb45b9c208aa@mail.gmail.com>
Subject: READING A FILE AND PUSHING CONTENTS IN A STRUCTURE
From: Alberto Luengo Cabanillas <cabiwan@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e659f46cf6d90a047ea6b374

--0016e659f46cf6d90a047ea6b374
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi people! For the last two days I=B4m facing this problem: In the setup
function of  Reducer class, I=B4m trying to read a file (with a value per
line) located in HDFS and putting them into a Hashtable the following way:
@Override
protected void setup(Context cont) throws IOException{
        try {
            FileSystem hdfs =3D FileSystem.get(new Configuration());
            Path path =3D new Path(HDFS_REDUCER_CONFIGURATION_FILE);

            //Validation
            if (!hdfs.exists(path))
            {
                throw new IOException("File"
+HDFS_REDUCER_CONFIGURATION_FILE + "doesnt exist");
            }

            if (!hdfs.isFile(path))
            {
                throw new IOException("File"
+HDFS_REDUCER_CONFIGURATION_FILE + "doesnt exist");
            }

            FSDataInputStream dis =3D hdfs.open(path);
            String input=3D"";

            while ((input =3D dis.readUTF()) !=3D null) {
                String numPop =3D dis.readUTF();
                parameters.put("numPopulation", Integer.getInteger(numPop))=
;
                String maxIter =3D dis.readUTF();
                parameters.put("maxIterations",
Integer.getInteger(maxIter));
            }
            dis.close();
        } catch (IOException ioe) {
            System.err.println("REDUCER: IOException reading Reducer
configuration file");
            System.err.println(ioe.toString());
        }
    }

But when I try to access the elements of the structure (parameters) I get
null values, caused-probably- because I=B4m not reading well those values.
What I=B4m doing wrong? Another way to do this?
Thanks a lot in advance.

PD: I forgot to mention that I=B4m working in pseudo-distributed mode.




--=20
Alberto

--0016e659f46cf6d90a047ea6b374--

From general-return-1011-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 03 12:35:25 2010
Return-Path: <general-return-1011-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97159 invoked from network); 3 Feb 2010 12:35:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Feb 2010 12:35:25 -0000
Received: (qmail 19377 invoked by uid 500); 3 Feb 2010 12:35:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19280 invoked by uid 500); 3 Feb 2010 12:35:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19267 invoked by uid 99); 3 Feb 2010 12:35:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Feb 2010 12:35:23 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of nipen.mark@gmail.com designates 209.85.210.197 as permitted sender)
Received: from [209.85.210.197] (HELO mail-yx0-f197.google.com) (209.85.210.197)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Feb 2010 12:35:15 +0000
Received: by yxe35 with SMTP id 35so1211422yxe.2
        for <general@hadoop.apache.org>; Wed, 03 Feb 2010 04:34:52 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=CXAn2YeYQ0Qor4OlIOebO1MpEofOyLNNRrR608VXo68=;
        b=jewgaOt7vUqD2KEYNU4vmMPdUR/Sor0mIBjHno4h0fsO871E+61i22EXM4INYj8FrV
         id1t/EmX1RhixEC+wZQC7hN9fcf6XNPQ/BttFlZczuNxvOh0mPgG47C7f1AZ52XlWo0s
         A8YnlAgJunQHFBkJ38YDghlbV14pdPg/Ul8/I=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=C5Wo5g73Xff/oSPRQE08bgrjQ6L1jfdbkGLAdCaIgYlQAYSceusm9aPBiSNhJkP/k2
         PjLFmZHOtYUOHxeUqWOdM18kU8Fncq0k18pSlZCpeENQ9kLuMmthGlJyNGTsck5KPt6L
         vSIGejGnZCJKd6K74YhuDR41zFLWoIdrcPLjI=
MIME-Version: 1.0
Received: by 10.91.84.17 with SMTP id m17mr4573942agl.15.1265200491245; Wed, 
	03 Feb 2010 04:34:51 -0800 (PST)
Date: Wed, 3 Feb 2010 18:19:51 +0545
Message-ID: <b45b7eee1002030434t439040c1s12935c34ac873938@mail.gmail.com>
Subject: Job Tracker questions
From: Mark N <nipen.mark@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f92baad9b1b3047eb171cd
X-Virus-Checked: Checked by ClamAV on apache.org

--001485f92baad9b1b3047eb171cd
Content-Type: text/plain; charset=ISO-8859-1

We have a hadoop job running and have used custom counters to track  few
counters ( like no of successfully processed documents matching certain
conditions)


Since we need to get this counters even while the Hadoop job is running , we
wrote another Java program to read these counters


*Counter reader  program *will do the following :


1)      List all the running jobs.

2)   Get the running job using Job name

2)     Get all the counter for individual running jobs

3)      Set this counters in variables.
        We could successfully read these counters  , but since we need to
show these counters to custom UI , how can we show these counters?

        we looked into various options to read these counters to show in UI
as following :

      1. Dump these counters to database , however this may be overhead
      2. Write web service   and UI will invoke the functions from these
service to show in UI ( However since we need to run "*Counter reader
program "  *with Hadoop command it might not be feasible to write web
service ?   )

      so the question is can we achive to read the counters using simple
Java APIs ? Does anyone have idea how does the default jobtracker JSP works
? we wanted to built something similar to this

thanks



-- 
Nipen Mark

--001485f92baad9b1b3047eb171cd--

From general-return-1012-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 03 14:48:56 2010
Return-Path: <general-return-1012-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54035 invoked from network); 3 Feb 2010 14:48:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Feb 2010 14:48:56 -0000
Received: (qmail 83947 invoked by uid 500); 3 Feb 2010 14:48:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83861 invoked by uid 500); 3 Feb 2010 14:48:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83851 invoked by uid 99); 3 Feb 2010 14:48:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Feb 2010 14:48:54 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of zjffdu@gmail.com designates 209.85.216.204 as permitted sender)
Received: from [209.85.216.204] (HELO mail-px0-f204.google.com) (209.85.216.204)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Feb 2010 14:48:46 +0000
Received: by pxi42 with SMTP id 42so2432031pxi.5
        for <general@hadoop.apache.org>; Wed, 03 Feb 2010 06:48:25 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=smaTWCsGqSQUDtSWoBzyKqyCV4HCMyQ4Tca3rUWNS/o=;
        b=X9dIn9bkTgcOBKrpWDWlYxgIlDqSAjFrXKtth2fOuw1xuCxK5mhj/yNhineZvLyXZF
         RMvPpx+2JgqVulqBsLkCM/zObj5dI6sc4vIBhfdgSeY/dNFlE2MVDSPI9ykQni8xsZej
         NDsCyUSlBxWU3dbzAm6J9kADTlcf/FuZAlxkw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=hTvVurnbOAkczR65DZ92M9xlg5HNOgl4sRTMAtr2QMUay4kbSKCogEbIEYFKE76BOT
         SQmjD6kppNHJsj/lOHmSmYht7f8PF2sN7yjqI1vB0PjTtJQbq39iuDZQsVtHj4XJ6lVa
         bPVkApjtZqJAsCR1ae454zZfmDyTQSpBLUz44=
MIME-Version: 1.0
Received: by 10.143.154.2 with SMTP id g2mr1539132wfo.114.1265208505036; Wed, 
	03 Feb 2010 06:48:25 -0800 (PST)
In-Reply-To: <b45b7eee1002030434t439040c1s12935c34ac873938@mail.gmail.com>
References: <b45b7eee1002030434t439040c1s12935c34ac873938@mail.gmail.com>
Date: Wed, 3 Feb 2010 06:48:24 -0800
Message-ID: <8211a1321002030648h565edc02yf8e41bdc2ca51cc@mail.gmail.com>
Subject: Re: Job Tracker questions
From: Jeff Zhang <zjffdu@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636e0a6c08271b1047eb34fdc
X-Virus-Checked: Checked by ClamAV on apache.org

--001636e0a6c08271b1047eb34fdc
Content-Type: text/plain; charset=UTF-8

I think you can use JobClient to get the counters in your web service.
If you look at the shell script bin/hadoop, you will find that actually this
shell use the JobClient to get the counters.



On Wed, Feb 3, 2010 at 4:34 AM, Mark N <nipen.mark@gmail.com> wrote:

> We have a hadoop job running and have used custom counters to track  few
> counters ( like no of successfully processed documents matching certain
> conditions)
>
>
> Since we need to get this counters even while the Hadoop job is running ,
> we
> wrote another Java program to read these counters
>
>
> *Counter reader  program *will do the following :
>
>
> 1)      List all the running jobs.
>
> 2)   Get the running job using Job name
>
> 2)     Get all the counter for individual running jobs
>
> 3)      Set this counters in variables.
>        We could successfully read these counters  , but since we need to
> show these counters to custom UI , how can we show these counters?
>
>        we looked into various options to read these counters to show in UI
> as following :
>
>      1. Dump these counters to database , however this may be overhead
>      2. Write web service   and UI will invoke the functions from these
> service to show in UI ( However since we need to run "*Counter reader
> program "  *with Hadoop command it might not be feasible to write web
> service ?   )
>
>      so the question is can we achive to read the counters using simple
> Java APIs ? Does anyone have idea how does the default jobtracker JSP works
> ? we wanted to built something similar to this
>
> thanks
>
>
>
> --
> Nipen Mark
>



-- 
Best Regards

Jeff Zhang

--001636e0a6c08271b1047eb34fdc--

From general-return-1013-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 03 21:53:47 2010
Return-Path: <general-return-1013-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42609 invoked from network); 3 Feb 2010 21:53:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Feb 2010 21:53:46 -0000
Received: (qmail 13678 invoked by uid 500); 3 Feb 2010 21:53:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13624 invoked by uid 500); 3 Feb 2010 21:53:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13611 invoked by uid 99); 3 Feb 2010 21:53:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Feb 2010 21:53:45 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of krystiankichewko@gmail.com designates 209.85.223.193 as permitted sender)
Received: from [209.85.223.193] (HELO mail-iw0-f193.google.com) (209.85.223.193)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Feb 2010 21:53:36 +0000
Received: by iwn33 with SMTP id 33so403461iwn.20
        for <general@hadoop.apache.org>; Wed, 03 Feb 2010 13:53:15 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=TCsr1W1dsC/U4rDMUNAxmDn7q0ttx0CIcRaFpGX7z6U=;
        b=DHyhxKyNKT+/oaXSQfqOPYuJvP2epChAfgTPfrrOLcp1JDXuI8iBOMxlSaMIr9cKUQ
         /UIeWt89igQc0qeiHBuUsscLwP+HYQv0Ka/sGN3xGESIjeMOb3kabxydAr1E6fDo0RX0
         h3cQjB0CoIPY4ILN/3g29Bc9UY97ngZU+Ji3U=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=d2s6R9/GbGN+gIPPjMjJ5WMpMgrFpBn8UPE5Mkfpqsknf53+7YYFiBCnq8z0KG4PeT
         fXoGQgUR5W6uFWrWgJU1Dr4n42FaGdlrS12Q4OHhE2ry55UlEFLwDIQjt7r4jpQDudgn
         pVae1EvGlaLQ15sc0O+mtDJOKgRORoKYS9uOI=
MIME-Version: 1.0
Received: by 10.231.159.207 with SMTP id k15mr342387ibx.75.1265233995681; Wed, 
	03 Feb 2010 13:53:15 -0800 (PST)
Date: Wed, 3 Feb 2010 22:53:15 +0100
Message-ID: <cd744ded1002031353h17f188b9q147a4ac99a9311a6@mail.gmail.com>
Subject: Hadoop in maven repository,
From: Krystian Kichewko <krystiankichewko@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hello!

I have a question:

Is hadoop available in maven repositories?

After a little bit of search I found this thread:

http://www.mail-archive.com/core-user@hadoop.apache.org/msg00241.html

It is a bit old 2yrs, and I still can't find official hadoop jars in
maven repository. There are jars from apache mahout project but I'm
not sure if this is pure hadoop core or with some kind of
pathes/changes. I'm using hdfs client in my project which is built
using maven 2, and it would be alot easier if I could get official
hadoop core jar from maven repository.

Is there any chance of pushing hadoop into maven repositories?

Thanks in advance!

Krystian Kichewko

From general-return-1014-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 03 22:50:10 2010
Return-Path: <general-return-1014-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69951 invoked from network); 3 Feb 2010 22:50:10 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Feb 2010 22:50:10 -0000
Received: (qmail 91820 invoked by uid 500); 3 Feb 2010 22:50:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91729 invoked by uid 500); 3 Feb 2010 22:50:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91719 invoked by uid 99); 3 Feb 2010 22:50:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Feb 2010 22:50:08 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of drew.farris@gmail.com designates 209.85.223.193 as permitted sender)
Received: from [209.85.223.193] (HELO mail-iw0-f193.google.com) (209.85.223.193)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Feb 2010 22:49:57 +0000
Received: by iwn33 with SMTP id 33so460416iwn.20
        for <general@hadoop.apache.org>; Wed, 03 Feb 2010 14:49:37 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=s7cWtApt9HHsXmra8MvZIC+jDbmjlmgRbvXcu3VlYmg=;
        b=weraPY5rlqnueOYFTTF/LqIjmR45OhMsq1YRo6OjD9SIpujhLFSLAJFHPU6bGDtcCv
         Er0xfdGekaM1af9Tykn/FxFLmgmlU3Gm7Xdl//DueLqVSo63TwTWy79y37/VvXO91gWe
         IgrrLI/JfL2Wm6dDTm08MvjCbPeKUlf4VKfM8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=xheAYRbHmRBQgmhhcWO44jCpy0b5xVFDQSypD0LNJXBPefIpzObakFqc7HTdGSnhw0
         2KBvG595kAK8oQXTk3BdqS1dm5k2jPRex7J0YbQNLH42/3pfqhwIsJP2jxZ26Oqirqo0
         +iqd4343TWPfvq+4QT+yU/5bCbv2UB+figqtE=
MIME-Version: 1.0
Received: by 10.231.154.197 with SMTP id p5mr519284ibw.28.1265237376835; Wed, 
	03 Feb 2010 14:49:36 -0800 (PST)
In-Reply-To: <8f8e14c41002031446h21efec4csf8f1ec695d19fd9f@mail.gmail.com>
References: <cd744ded1002031353h17f188b9q147a4ac99a9311a6@mail.gmail.com>
	 <8f8e14c41002031446h21efec4csf8f1ec695d19fd9f@mail.gmail.com>
Date: Wed, 3 Feb 2010 17:49:36 -0500
Message-ID: <8f8e14c41002031449i252f1817hed2b48afaa3b3910@mail.gmail.com>
Subject: Re: Hadoop in maven repository,
From: Drew Farris <drew.farris@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c92c896720c4047eba0809
X-Virus-Checked: Checked by ClamAV on apache.org

--001636c92c896720c4047eba0809
Content-Type: text/plain; charset=ISO-8859-1

Krystian,

repository.apache.org contains some of the hadoop artifacts. That is the
best place to start.

Drew

On Feb 3, 2010 4:53 PM, "Krystian Kichewko" <krystiankichewko@gmail.com>
wrote:

Hello!

I have a question:

Is hadoop available in maven repositories?

After a little bit of search I found this thread:

http://www.mail-archive.com/core-user@hadoop.apache.org/msg00241.html

It is a bit old 2yrs, and I still can't find official hadoop jars in
maven repository. There are jars from apache mahout project but I'm
not sure if this is pure hadoop core or with some kind of
pathes/changes. I'm using hdfs client in my project which is built
using maven 2, and it would be alot easier if I could get official
hadoop core jar from maven repository.

Is there any chance of pushing hadoop into maven repositories?

Thanks in advance!

Krystian Kichewko

--001636c92c896720c4047eba0809--

From general-return-1015-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 04 02:36:18 2010
Return-Path: <general-return-1015-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82931 invoked from network); 4 Feb 2010 02:36:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Feb 2010 02:36:18 -0000
Received: (qmail 90871 invoked by uid 500); 4 Feb 2010 02:36:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90617 invoked by uid 500); 4 Feb 2010 02:36:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90597 invoked by uid 99); 4 Feb 2010 02:36:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Feb 2010 02:36:13 +0000
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Feb 2010 02:36:02 +0000
Received: from sp1-ex07cas03.ds.corp.yahoo.com (sp1-ex07cas03.ds.corp.yahoo.com [216.252.116.151])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o142ZJto013215;
	Wed, 3 Feb 2010 18:35:19 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:references:in-reply-to:accept-language:
	content-language:x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:
	content-type:content-transfer-encoding:mime-version;
	b=erKgseSsYPVBOXibtLiR4Is/5hwRm+RHCPInjRWs9RvVnktMUR005HWmNRd2SkZp
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas03.ds.corp.yahoo.com ([216.252.116.151]) with mapi; Wed, 3 Feb
 2010 18:35:19 -0800
From: Dekel Tankel <dekel@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>,
        "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Date: Wed, 3 Feb 2010 18:35:17 -0800
Subject: Hadoop User Group (Bay Area) - Feb 17th at Yahoo!
Thread-Topic: Hadoop User Group (Bay Area) - Feb 17th at Yahoo!
Thread-Index: AcoSMRRU8X5auVLKR8ufbpFTQi1hLgQXaiyQCpfO4pAWCVl24A==
Message-ID: <46E2672895E8744A97D7577A6110858B4FDBF8B10A@SP1-EX07VS01.ds.corp.yahoo.com>
References: <46E2672895E8744A97D7577A6110858B4E1AC69A91@SP1-EX07VS01.ds.corp.yahoo.com>
  <46E2672895E8744A97D7577A6110858B4FD39F6EDF@SP1-EX07VS01.ds.corp.yahoo.com>
In-Reply-To: <46E2672895E8744A97D7577A6110858B4FD39F6EDF@SP1-EX07VS01.ds.corp.yahoo.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


Hi all,

RSVP is open for the next monthly Bay Area Hadoop user group at the Yahoo! =
Sunnyvale Campus, Wednesday, Feb 17th, 6PM

Registration and Agenda are available here
http://www.meetup.com/hadoop/calendar/12497904/

Looking forward to seeing you there!

Dekel


From general-return-1016-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 04 03:52:35 2010
Return-Path: <general-return-1016-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 2569 invoked from network); 4 Feb 2010 03:52:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Feb 2010 03:52:34 -0000
Received: (qmail 42713 invoked by uid 500); 4 Feb 2010 03:52:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42529 invoked by uid 500); 4 Feb 2010 03:52:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42500 invoked by uid 99); 4 Feb 2010 03:52:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Feb 2010 03:52:31 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Feb 2010 03:52:20 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o143p3Lh044681;
	Wed, 3 Feb 2010 19:51:03 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:content-type:
	content-transfer-encoding:mime-version:subject:date:cc:x-mailer;
	b=UQdyN6m829qrt5KTt6fbnBN8GnzK1KnWbSQbqnBZggHcepv4/YugWhKl0aIYd5Cl
Message-Id: <715AE598-D615-4E75-B42A-6EF685AF3F0A@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: mapreduce-dev@hadoop.apache.org, mapreduce-user@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: New Hadoop Map-Reduce Committer: Vinod Kumar V.
Date: Wed, 3 Feb 2010 19:51:03 -0800
Cc: general@hadoop.apache.org
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

The Hadoop PMC has voted to make Vinod Kumar V. a committer on Hadoop  
Map-Reduce.

Congratulations, Vinod.

Thanks again for all your valuable contributions, we look forward to  
more!

Arun

From general-return-1017-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 04 05:17:28 2010
Return-Path: <general-return-1017-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 20496 invoked from network); 4 Feb 2010 05:17:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Feb 2010 05:17:27 -0000
Received: (qmail 1766 invoked by uid 500); 4 Feb 2010 05:17:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1538 invoked by uid 500); 4 Feb 2010 05:17:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1528 invoked by uid 99); 4 Feb 2010 05:17:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Feb 2010 05:17:26 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of zqmaster@gmail.com designates 209.85.217.209 as permitted sender)
Received: from [209.85.217.209] (HELO mail-gx0-f209.google.com) (209.85.217.209)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Feb 2010 05:17:16 +0000
Received: by gxk1 with SMTP id 1so2155175gxk.11
        for <general@hadoop.apache.org>; Wed, 03 Feb 2010 21:16:55 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from
         :user-agent:mime-version:to:subject:content-type
         :content-transfer-encoding;
        bh=x0+N7NNVySfdyHyVffJRObJMDOh/Dp3BUPfKCbnKeBE=;
        b=LHPCTgyRCKCF6zHhHlcxXBpVZlOujAESVEfD0pZwUFKmBWEWNecLnnzDw4QRRysVnE
         1B6m9sx2rd0ZmYkfZb7HWikJNe/d1CUKt1jNmEGhZNU3X5PEZeEYpko0lWfRRF9Ye3wy
         KH1R3AkOtvr8Tl/gBDQ2c9k3T7MRIc/4uckXc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject
         :content-type:content-transfer-encoding;
        b=OyGO2I9QUpi8Nms+863gvsUpa/6MASsbkCUP2b30upf1R+Tces+4yuPs7FZ3cjTmXl
         a/EK5/dQuuhMPxSqGUzqDed1S01ezyO4eBvTQDbALOWd89QmtM/7ZK5ShNJ3uMjggRRy
         bVg9f3k8UCTlHXO2skc+jH7JxMt6jgCVBwJaM=
Received: by 10.100.245.29 with SMTP id s29mr707610anh.146.1265260615732;
        Wed, 03 Feb 2010 21:16:55 -0800 (PST)
Received: from ?192.168.1.101? (cpe-174-096-108-211.carolina.res.rr.com [174.96.108.211])
        by mx.google.com with ESMTPS id 7sm2806837ywf.40.2010.02.03.21.16.54
        (version=SSLv3 cipher=RC4-MD5);
        Wed, 03 Feb 2010 21:16:54 -0800 (PST)
Message-ID: <4B6A5844.2090505@gmail.com>
Date: Thu, 04 Feb 2010 00:16:52 -0500
From: "Ma, Zhiqiang" <zqmaster@gmail.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.7) Gecko/20100111 Thunderbird/3.0.1
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: help for beginner
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hi All,

I am a beginner of Hadoop. I modified the Inverted Index Code in Yahoo's 
Tutorial  
(http://developer.yahoo.com/hadoop/tutorial/module4.html#solution), but 
I always get errors of "java.io.IOException: Type mismatch in key from 
map: expected org.apache.hadoop.io.Text, recieved 
org.apache.hadoop.io.LongWritable". Could some people tell me what is 
wrong in my code? Thanks a million!

Zhiqiang

----------------------------code 
starts--------------------------------------
import java.io.IOException;
import java.util.Iterator;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.input.FileSplit;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;

public class YahooIndex {

   public static class MyMapper extends Mapper<LongWritable, Text, Text, 
Text> {

     private final static Text word = new Text();
     private final static Text location = new Text();

     public void map(LongWritable key, Text val,
         OutputCollector<Text, Text> output, Reporter reporter)
         throws IOException {

       FileSplit fileSplit = (FileSplit)reporter.getInputSplit();
       String fileName = fileSplit.getPath().getName();
       location.set(fileName);

       String line = val.toString();
       StringTokenizer itr = new StringTokenizer(line.toLowerCase());
       while (itr.hasMoreTokens()) {
         word.set(itr.nextToken());
         output.collect(word, location);
       }
     }
   }



   public static class MyReducer extends Reducer<Text, Text, Text, Text> {

     public void reduce(Text key, Iterator<Text> values,
         OutputCollector<Text, Text> output, Reporter reporter)
         throws IOException {

       boolean first = true;
       StringBuilder toReturn = new StringBuilder();
       while (values.hasNext()){
         if (!first)
           toReturn.append(", ");
         first=false;
         toReturn.append(values.next().toString());
       }

       output.collect(key, new Text(toReturn.toString()));
     }
   }


   public static void main(String[] args) throws IOException, 
InterruptedException, ClassNotFoundException {
           Configuration conf = new Configuration();
         Job job = new Job(conf, "Example Hadoop 0.20.1 WordCount");
         job.setJarByClass(YahooIndex.class);
         job.setMapperClass(MyMapper.class);
         job.setReducerClass(MyReducer.class);
         job.setOutputKeyClass(Text.class);
         job.setOutputValueClass(Text.class);
         FileInputFormat.addInputPath(job, new Path("input"));
         FileOutputFormat.setOutputPath(job, new Path("output"));
         System.exit(job.waitForCompletion(true) ? 0 : 1);
   }
}
-----------------------------------code 
ends-----------------------------------------------------------------------------------

From general-return-1018-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 04 08:26:18 2010
Return-Path: <general-return-1018-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 90967 invoked from network); 4 Feb 2010 08:26:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Feb 2010 08:26:17 -0000
Received: (qmail 85818 invoked by uid 500); 4 Feb 2010 08:26:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85742 invoked by uid 500); 4 Feb 2010 08:26:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85732 invoked by uid 99); 4 Feb 2010 08:26:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Feb 2010 08:26:16 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of hkumar.arora@gmail.com designates 209.85.223.194 as permitted sender)
Received: from [209.85.223.194] (HELO mail-iw0-f194.google.com) (209.85.223.194)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Feb 2010 08:26:04 +0000
Received: by iwn34 with SMTP id 34so846241iwn.21
        for <general@hadoop.apache.org>; Thu, 04 Feb 2010 00:25:43 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=MOPqmHBmjM7RPrFzKLVUUbyTqyBx4MHypozMy3glWbA=;
        b=I5Gufbz6NuoJ8pitMQGEOPVgBctzfNioVYlUQdTHMGtTbesMM/4Qnocd3rPGwzhwlj
         7e+cmx6169KX3c0PITHxDX2VezDgXUOxAAwv5qmUipGitdJeXYvWpbNWieT1KOqCniTK
         RXnxPGjZ8z1W7UWAPFAkCNNwkjw7fWCJ0mk/o=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=a9TUyEe6DksAlZbADyAUfo8L50GsF3oUhcitSVyhVJbTyTjnk5D7TCLq8FXi0XrfEm
         vfawKrSZgcoIrGSDOlmjHNeEmYnGihmMDm5Yq4mHVex9Q5mFGL7NvTzK50/C2ABuk/X8
         WwOrM+b/MDmXJnU0tVnerBowtmLXrA+LSgyBw=
MIME-Version: 1.0
Received: by 10.231.161.138 with SMTP id r10mr1464434ibx.34.1265271942293; 
	Thu, 04 Feb 2010 00:25:42 -0800 (PST)
Date: Thu, 4 Feb 2010 17:25:42 +0900
Message-ID: <e497591c1002040025l4fa56b73oa0a9bbb5dcc5a809@mail.gmail.com>
Subject: =?UTF-8?B?amF2YS5pby5JT0V4Y2VwdGlvbjogQ2Fubm90IG9wZW4gZmlsZW5hbWUgL3VzZXIvcm9vdA==?=
	=?UTF-8?B?L++/vXPvv71077+9Ze+/vXDvv70x77+9L++/vXDvv71h77+9cu+/vXTvv70t77+9MO+/vTDvv70w77+9?=
	=?UTF-8?B?MO+/vTA=?=
From: Harshit Kumar <hkumar.arora@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c5971baa2fa4047ec214a9
X-Virus-Checked: Checked by ClamAV on apache.org

--001636c5971baa2fa4047ec214a9
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi

I dont understand the reason for this error.

java.io.IOException: Cannot open filename
/user/root/=EF=BF=BDs=EF=BF=BDt=EF=BF=BDe=EF=BF=BDp=EF=BF=BD1=EF=BF=BD/=EF=
=BF=BDp=EF=BF=BDa=EF=BF=BDr=EF=BF=BDt=EF=BF=BD-=EF=BF=BD0=EF=BF=BD0=EF=BF=
=BD0=EF=BF=BD0=EF=BF=BD0 at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:139=
4)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.(DFSClient.java:1385) at
org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:338) at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.jav=
a:171)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:359) at
org.bike.MakeNPairReduce.reduce(MakeNPairReduce.java:40) at
org.bike.MakeNPairReduce.reduce(MakeNPairReduce.java:1) at
org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436) at
org.apache.hadoop.mapred.Child.main(Child.java:158)


I have a code that scans a folder step0 to find name of files generated in
the previous map-reduce phase. Then create another file with the entries fo=
r
ex:
if scanning finds that there are 2 files produced by 1st map-reduce phase,
then new created file will have 2 entries step1/part00000 and
step1/part00001 i.e. one entry for each file.

Now, when I read this file in another map-reduce job, each line is read as
/user/root/=EF=BF=BDs=EF=BF=BDt=EF=BF=BDe=EF=BF=BDp=EF=BF=BD1=EF=BF=BD/=EF=
=BF=BDp=EF=BF=BDa=EF=BF=BDr=EF=BF=BDt=EF=BF=BD-=EF=BF=BD0=EF=BF=BD0=EF=BF=
=BD0=EF=BF=BD0=EF=BF=BD0 . What it seems like, a string
inserted by my code, when read by FSDataInputStream prefix each character o=
f
the string by a question mark (?). Why is that so?

The file name part-00000 do exist inside folder step1, but reading this
filename, /user/root/=EF=BF=BDs=EF=BF=BDt=EF=BF=BDe=EF=BF=BDp=EF=BF=BD1=EF=
=BF=BD/=EF=BF=BDp=EF=BF=BDa=EF=BF=BDr=EF=BF=BDt=EF=BF=BD-=EF=BF=BD0=EF=BF=
=BD0=EF=BF=BD0=EF=BF=BD0=EF=BF=BD0 , throws IOException
which I can undersand that there is no such filename, but why are these ?'s
infiltraded before each letter.

Really appreciate if some one can help me solve this riddle?

Thanks and Regards
H. Kumar
skype: harshit900
Blog: http://harshitkumar.wordpress.com
Website: http:/kumarharmuscat.tripod.com

--001636c5971baa2fa4047ec214a9--

From general-return-1019-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 04 11:00:13 2010
Return-Path: <general-return-1019-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49345 invoked from network); 4 Feb 2010 11:00:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Feb 2010 11:00:12 -0000
Received: (qmail 72771 invoked by uid 500); 4 Feb 2010 11:00:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72210 invoked by uid 500); 4 Feb 2010 11:00:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72149 invoked by uid 99); 4 Feb 2010 11:00:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Feb 2010 11:00:09 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of nipen.mark@gmail.com designates 209.85.217.209 as permitted sender)
Received: from [209.85.217.209] (HELO mail-gx0-f209.google.com) (209.85.217.209)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Feb 2010 11:00:02 +0000
Received: by gxk1 with SMTP id 1so2312276gxk.11
        for <multiple recipients>; Thu, 04 Feb 2010 02:59:41 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:cc:content-type;
        bh=cXwWGokMwVbpBJSvrXSy0AaMMmcxz8S1KhUZcnOiG/I=;
        b=oafuYCINu2TqOV79fXTs/y9JTzmCGJZlzuJtVNFSO3SdAtd06HMYwgHRv+3zTF5LRf
         wyV/D2qegMJfC6t816O/0P9IAIivgC/Bc/FsPmReYtOlTBhoeRRAloA3Z/Kyw9X+SIU9
         4KIHP9wu23YD3D/bPbPWDYDm6EB1ve47Yvv9c=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=ndyQGNZYAuUmWy+DMTWked5a6tk8PyUeIgzOtpjCp2YujLJyjpwFd7oHypWHXrGGce
         yZtlg9CEI0D9cfPLhuFVUtMkqHsWCTyyhVgofJP6VRSF+SNdm5eU4U3B0kCUTMctx57l
         qA56i0eKa3NRu5MkdvLr5F3c1HzrLiCRxv3JA=
MIME-Version: 1.0
Received: by 10.91.164.39 with SMTP id r39mr1007306ago.113.1265281179432; Thu, 
	04 Feb 2010 02:59:39 -0800 (PST)
In-Reply-To: <8211a1321002030648h565edc02yf8e41bdc2ca51cc@mail.gmail.com>
References: <b45b7eee1002030434t439040c1s12935c34ac873938@mail.gmail.com>
	 <8211a1321002030648h565edc02yf8e41bdc2ca51cc@mail.gmail.com>
Date: Thu, 4 Feb 2010 16:44:39 +0545
Message-ID: <b45b7eee1002040259y423e7776x60ce1e0f9c53cc67@mail.gmail.com>
Subject: Re: Job Tracker questions
From: Mark N <nipen.mark@gmail.com>
To: general@hadoop.apache.org
Cc: common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f5ea343db781047ec43b19

--001485f5ea343db781047ec43b19
Content-Type: text/plain; charset=ISO-8859-1

Ye currently am using jobclient to read these counters.

But We are not able to use *webservices *because the jar which is used to
read the counters from  running hadoop job  is itself a Hadoop program

If we could have pure Java Api which is run without hadoop command then we
could return the counter variable into webservices and show in UI.

Any help  or technique to show thsese counters in the UI would be
appreciated  ( not necessarily using web service )


I am using webservices because I am having .net VB client

thanks



On Wed, Feb 3, 2010 at 8:33 PM, Jeff Zhang <zjffdu@gmail.com> wrote:

> I think you can use JobClient to get the counters in your web service.
> If you look at the shell script bin/hadoop, you will find that actually
> this
> shell use the JobClient to get the counters.
>
>
>
> On Wed, Feb 3, 2010 at 4:34 AM, Mark N <nipen.mark@gmail.com> wrote:
>
> > We have a hadoop job running and have used custom counters to track  few
> > counters ( like no of successfully processed documents matching certain
> > conditions)
> >
> >
> > Since we need to get this counters even while the Hadoop job is running ,
> > we
> > wrote another Java program to read these counters
> >
> >
> > *Counter reader  program *will do the following :
> >
> >
> > 1)      List all the running jobs.
> >
> > 2)   Get the running job using Job name
> >
> > 2)     Get all the counter for individual running jobs
> >
> > 3)      Set this counters in variables.
> >        We could successfully read these counters  , but since we need to
> > show these counters to custom UI , how can we show these counters?
> >
> >        we looked into various options to read these counters to show in
> UI
> > as following :
> >
> >      1. Dump these counters to database , however this may be overhead
> >      2. Write web service   and UI will invoke the functions from these
> > service to show in UI ( However since we need to run "*Counter reader
> > program "  *with Hadoop command it might not be feasible to write web
> > service ?   )
> >
> >      so the question is can we achive to read the counters using simple
> > Java APIs ? Does anyone have idea how does the default jobtracker JSP
> works
> > ? we wanted to built something similar to this
> >
> > thanks
> >
> >
> >
> > --
> > Nipen Mark
> >
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>



-- 
Nipen Mark

--001485f5ea343db781047ec43b19--

From general-return-1020-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 05 18:50:05 2010
Return-Path: <general-return-1020-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60188 invoked from network); 5 Feb 2010 18:50:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Feb 2010 18:50:05 -0000
Received: (qmail 57060 invoked by uid 500); 5 Feb 2010 18:50:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56971 invoked by uid 500); 5 Feb 2010 18:50:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56961 invoked by uid 99); 5 Feb 2010 18:50:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Feb 2010 18:50:03 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.62.48] (HELO qmta05.westchester.pa.mail.comcast.net) (76.96.62.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Feb 2010 18:49:54 +0000
Received: from omta03.westchester.pa.mail.comcast.net ([76.96.62.27])
	by qmta05.westchester.pa.mail.comcast.net with comcast
	id eG1i1d01K0bG4ec55Jom6x; Fri, 05 Feb 2010 18:48:46 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta03.westchester.pa.mail.comcast.net with comcast
	id eJpQ1d00R2SbwD53PJpUeC; Fri, 05 Feb 2010 18:49:32 +0000
Message-Id: <E41C08B5-BB31-4CD4-A95A-7ED3754A617E@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=Apple-Mail-13--674245159
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Welcome 
Date: Fri, 5 Feb 2010 10:49:23 -0800
X-Mailer: Apple Mail (2.936)

--Apple-Mail-13--674245159
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed
Content-Transfer-Encoding: 7bit

The Hadoop PMC has added the following committers to its ranks:

Daniel Dai
Pradeep Kamath
Tsz Wo (Nicholas) Sze

Thanks for all of your hard work on Hadoop!

-- Owen
--Apple-Mail-13--674245159--

From general-return-1021-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 08 14:51:07 2010
Return-Path: <general-return-1021-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95796 invoked from network); 8 Feb 2010 14:51:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Feb 2010 14:51:07 -0000
Received: (qmail 44006 invoked by uid 500); 8 Feb 2010 14:51:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 43932 invoked by uid 500); 8 Feb 2010 14:51:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 43922 invoked by uid 99); 8 Feb 2010 14:51:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Feb 2010 14:51:05 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jason.hadoop@gmail.com designates 209.85.222.175 as permitted sender)
Received: from [209.85.222.175] (HELO mail-pz0-f175.google.com) (209.85.222.175)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Feb 2010 14:50:55 +0000
Received: by pzk5 with SMTP id 5so6162232pzk.29
        for <general@hadoop.apache.org>; Mon, 08 Feb 2010 06:50:33 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=xGSQue9Iw4ZwskDSGHeI1lia+gRBrINm05XzeRLmrUE=;
        b=I9jfWdCCmEELmby7BH4tvrCx8wU3aBSDe48irIzWbsGj9C+tkGGpU9HuaWkkxEQJ5Z
         AkH+PF28KOyAdV1Ik98F4tcQ0lr+I+XEbNa4Rva0EHJK5Izq6nuvBSJFY0sw0lBPStoA
         pmxo1LJ7uM4mS2zf3ScHs8AITsovmHEhV7dhM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=rkuBcnqkP8ktzblC0s/juJwBfO0T6nf3UGIotTzxNPY0OF1d2W5RJnnawHe1vpXXiw
         gm6t8IDZlcl8Paaklx/vTcUC0iSMI14PiwzD6IqrMRYpQGLwoTzJtyyOk6QDasscnj8c
         xeIMZn3kD3eQPSWmrsfk9UDdoa79+aOKGbAxg=
MIME-Version: 1.0
Received: by 10.141.105.12 with SMTP id h12mr4591865rvm.64.1265640632114; Mon, 
	08 Feb 2010 06:50:32 -0800 (PST)
In-Reply-To: <4B6A5844.2090505@gmail.com>
References: <4B6A5844.2090505@gmail.com>
Date: Mon, 8 Feb 2010 06:50:32 -0800
Message-ID: <314098691002080650me3ca315gf05894d92d28a0ce@mail.gmail.com>
Subject: Re: help for beginner
From: Jason Venner <jason.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd1389e4a5f35047f17ecea
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd1389e4a5f35047f17ecea
Content-Type: text/plain; charset=ISO-8859-1

This class of problem has a pretty standard answer

http://www.prohadoopbook.com/forum/topics/type-mismatch-error

On Wed, Feb 3, 2010 at 9:16 PM, Ma, Zhiqiang <zqmaster@gmail.com> wrote:

> Hi All,
>
> I am a beginner of Hadoop. I modified the Inverted Index Code in Yahoo's
> Tutorial  (
> http://developer.yahoo.com/hadoop/tutorial/module4.html#solution), but I
> always get errors of "java.io.IOException: Type mismatch in key from map:
> expected org.apache.hadoop.io.Text, recieved
> org.apache.hadoop.io.LongWritable". Could some people tell me what is wrong
> in my code? Thanks a million!
>
> Zhiqiang
>
> ----------------------------code
> starts--------------------------------------
> import java.io.IOException;
> import java.util.Iterator;
> import java.util.StringTokenizer;
>
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.io.LongWritable;
> import org.apache.hadoop.io.Text;
> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
> import org.apache.hadoop.mapreduce.lib.input.FileSplit;
> import org.apache.hadoop.mapred.OutputCollector;
> import org.apache.hadoop.mapred.Reporter;
> import org.apache.hadoop.mapreduce.Job;
> import org.apache.hadoop.mapreduce.Mapper;
> import org.apache.hadoop.mapreduce.Reducer;
>
> public class YahooIndex {
>
>  public static class MyMapper extends Mapper<LongWritable, Text, Text,
> Text> {
>
>    private final static Text word = new Text();
>    private final static Text location = new Text();
>
>    public void map(LongWritable key, Text val,
>        OutputCollector<Text, Text> output, Reporter reporter)
>        throws IOException {
>
>      FileSplit fileSplit = (FileSplit)reporter.getInputSplit();
>      String fileName = fileSplit.getPath().getName();
>      location.set(fileName);
>
>      String line = val.toString();
>      StringTokenizer itr = new StringTokenizer(line.toLowerCase());
>      while (itr.hasMoreTokens()) {
>        word.set(itr.nextToken());
>        output.collect(word, location);
>      }
>    }
>  }
>
>
>
>  public static class MyReducer extends Reducer<Text, Text, Text, Text> {
>
>    public void reduce(Text key, Iterator<Text> values,
>        OutputCollector<Text, Text> output, Reporter reporter)
>        throws IOException {
>
>      boolean first = true;
>      StringBuilder toReturn = new StringBuilder();
>      while (values.hasNext()){
>        if (!first)
>          toReturn.append(", ");
>        first=false;
>        toReturn.append(values.next().toString());
>      }
>
>      output.collect(key, new Text(toReturn.toString()));
>    }
>  }
>
>
>  public static void main(String[] args) throws IOException,
> InterruptedException, ClassNotFoundException {
>          Configuration conf = new Configuration();
>        Job job = new Job(conf, "Example Hadoop 0.20.1 WordCount");
>        job.setJarByClass(YahooIndex.class);
>        job.setMapperClass(MyMapper.class);
>        job.setReducerClass(MyReducer.class);
>        job.setOutputKeyClass(Text.class);
>        job.setOutputValueClass(Text.class);
>        FileInputFormat.addInputPath(job, new Path("input"));
>        FileOutputFormat.setOutputPath(job, new Path("output"));
>        System.exit(job.waitForCompletion(true) ? 0 : 1);
>  }
> }
> -----------------------------------code
> ends-----------------------------------------------------------------------------------
>



-- 
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals

--000e0cd1389e4a5f35047f17ecea--

From general-return-1022-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 08 16:04:06 2010
Return-Path: <general-return-1022-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 18354 invoked from network); 8 Feb 2010 16:04:06 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Feb 2010 16:04:06 -0000
Received: (qmail 78619 invoked by uid 500); 8 Feb 2010 16:04:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78518 invoked by uid 500); 8 Feb 2010 16:04:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78508 invoked by uid 99); 8 Feb 2010 16:04:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Feb 2010 16:04:04 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of gautam.singaraju@gmail.com designates 209.85.217.228 as permitted sender)
Received: from [209.85.217.228] (HELO mail-gx0-f228.google.com) (209.85.217.228)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Feb 2010 16:03:58 +0000
Received: by gxk28 with SMTP id 28so5301949gxk.9
        for <general@hadoop.apache.org>; Mon, 08 Feb 2010 08:03:37 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=defmw3k6rjSBJV9Z+NCuXAN0T4VE7Kr5yNRSmTHoN7w=;
        b=CcScJ9EjdhMmESJbaO3RQAkSjHPPTxrcI2whmTXQQp31Hj2S7/J/ApIVyk2OzXI6IN
         CShNWYK8OHQF3+wI5zT1XkjgtGiaiYJLwhHFT4suT1FsO2OpntGUbHZJk9fTteZZgyQ8
         pEsXrzFiE+4Q6kd+zYK6tRmKPvemAqUPhbQq8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=nINawdRPBBJ4BdXokC/wCKYM99iOl3iXZD2Ea9RcgD/rOLHVxeShFxkq2eQspBbOKB
         aY6ItOs6fxlx20MWIkb9uCk8xB5pb12pJbcUtEL2Qo2i1zz89rbwTrVY7tpUeY586iHe
         vwiOvZCaeyv1wWOOGkWrsa7Os9AML5k68zFXM=
MIME-Version: 1.0
Received: by 10.150.120.8 with SMTP id s8mr2038365ybc.127.1265645017208; Mon, 
	08 Feb 2010 08:03:37 -0800 (PST)
Date: Mon, 8 Feb 2010 11:03:37 -0500
Message-ID: <f8b2ccd41002080803g7b1df1f7t2f7b0a1813aa7a91@mail.gmail.com>
Subject: Hadoop on Demand Infrastructure
From: Gautam Singaraju <gautam.singaraju@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

We are deploying Hadoop on Demand for a university research
environment. Due to the nature of infrastructure, where other
computationally intensive processes apart from HOD might be running on
each machine. Each of this machine has a limited amount of disk
storage available.

Would large cluster each machine with a single dedicated core be more
efficient in comparison to multiple cores on fewer machines? Since it
is HOD, I think network IO would be the same for both.

Any help would be appreciated.

From general-return-1023-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 10 02:32:46 2010
Return-Path: <general-return-1023-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39389 invoked from network); 10 Feb 2010 02:32:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Feb 2010 02:32:46 -0000
Received: (qmail 23141 invoked by uid 500); 10 Feb 2010 02:32:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22988 invoked by uid 500); 10 Feb 2010 02:32:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22978 invoked by uid 99); 10 Feb 2010 02:32:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Feb 2010 02:32:43 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of zookog@gmail.com designates 209.85.223.189 as permitted sender)
Received: from [209.85.223.189] (HELO mail-iw0-f189.google.com) (209.85.223.189)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Feb 2010 02:32:34 +0000
Received: by iwn27 with SMTP id 27so1090907iwn.20
        for <general@hadoop.apache.org>; Tue, 09 Feb 2010 18:32:13 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=26/91oybHpxMNV72HPx9bDb52S7G1FvtvvNUR34jpyc=;
        b=xka466AuvCa2ZgyzydHa1rnLWCSRM0gO8r5r3Su94celiv03QhyP5ZrP/K5T9nJN6k
         dEAR8iT+ICtuZMWV0SM4/GowNaSORCmRyL6Tu5s1wH0nS+VhMW2oWoL/Ut9uxi9VtILZ
         dkrBwLWJEwljlY/mF+c9uW/1dK7Y9B3jZ844s=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=LO6aZkmNo4IUzPvkYWLW2Sf34sfCbnkoRjznTyiKo0mdD/pNQrp1oz62HUj3W9ekO6
         /nC8KlOLhXkXKHEdVXoJ1TUEH0EsybQ60ezG9N45yxGt8CBOHd1/fvS8Lm5wERLa41HU
         B7tJ1viAhMjwZfdjWWqtIa1LKCQKqctxoe0CQ=
MIME-Version: 1.0
Received: by 10.231.148.83 with SMTP id o19mr502135ibv.39.1265769133137; Tue, 
	09 Feb 2010 18:32:13 -0800 (PST)
Date: Tue, 9 Feb 2010 19:32:13 -0700
Message-ID: <cd6401a1002091832m585520aw8529172aef3bea4e@mail.gmail.com>
Subject: announcing Tahoe-LAFS v1.6
From: "Zooko O'Whielacronx" <zookog@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
X-Virus-Checked: Checked by ClamAV on apache.org

Folks:

We've just released a new stable release of Tahoe-LAFS. Tahoe-LAFS is
an extremely reliable cloud storage system which includes integrated
encryption, erasure-coding and digital signatures. I like to think
that Tahoe-LAFS is a cloud storage system that "scales up" in terms of
the number of people, computers, and organizations that you can
connect to your data while maintaining integrity and access-control.

There is a Hadoop plugin named hadoop-lafs [1] which makes it possible
to use Tahoe-LAFS instead of HDFS. This provides strong security and
integrity features as well as erasure-coding (a la HDFS-503 [2]) while
maintaining, according to Aaron Cordova's HadoopWorld presentation
[3], good performance for at least some uses.

The release announcement follows.

Regards,

Zooko Wilcox-O'Hearn

[1] http://code.google.com/p/hadoop-lafs
[2] http://issues.apache.org/jira/browse/HDFS-503
[3] http://docs.google.com/fileview?id=0B9iCsXQ_HuEpN2NlNDgwMTMtNWRjOC00YzMwLWExYjMtYzVmM2FiZGRhNjA4&hl=en

ANNOUNCING Tahoe, the Least-Authority File System, v1.6

We are pleased to announce the immediate availability of
version 1.6 of Tahoe-LAFS, an extremely reliable distributed
data store.

Tahoe-LAFS is the first cloud storage system which offers
"provider-independent security" -- meaning that not even your
cloud service provider can read or alter your data without your
consent.  Here is the one-page explanation of its unique
security and fault-tolerance properties:

http://allmydata.org/source/tahoe/trunk/docs/about.html

Tahoe-LAFS v1.6.0 is the successor to v1.5.0, which was
released August 1, 2009 [1]. This release includes major
performance improvements, usability improvements, and one major
new feature: deep-immutable directories (cryptographically
unalterable permanent snapshots). See the NEWS file [2] for
details.


WHAT IS IT GOOD FOR?

With Tahoe-LAFS, you spread your filesystem across multiple
servers, and even if some of the servers fail or are taken over
by an attacker, the entire filesystem continues to work
correctly, and continues to preserve your privacy and
security. You can easily and securely share chosen files and
directories with others.

In addition to the core storage system itself, volunteers have
developed related projects to integrate it with other
tools. These include frontends for Windows, Macintosh,
JavaScript, and iPhone, and plugins for Hadoop, bzr, duplicity,
TiddlyWiki, and more. As of this release, contributors have
added an Android frontend and a working read-only FUSE
frontend. See the Related Projects page on the wiki [3].

We believe that the combination of erasure coding, strong
encryption, Free/Open Source Software and careful engineering
make Tahoe-LAFS safer than RAID, removable drive, tape, on-line
backup or other Cloud storage systems.

This software is developed under thorough unit tests, and there
are no known bugs or security flaws which would compromise
confidentiality or data integrity under normal use. (For all
currently known issues please see the known_issues.txt file
[4].)


COMPATIBILITY

This release is fully compatible with the version 1 series of
Tahoe-LAFS. Clients from this release can write files and
directories in the format used by clients of all versions back
to v1.0 (which was released March 25, 2008). Clients from this
release can read files and directories produced by clients of
all versions since v1.0. Servers from this release can serve
clients of all versions back to v1.0 and clients from this
release can use servers of all versions back to v1.0.

This is the seventh release in the version 1 series. The
version 1 series of Tahoe-LAFS will be actively supported and
maintained for the forseeable future, and future versions of
Tahoe-LAFS will retain the ability to read and write files
compatible with Tahoe-LAFS v1.

In addition, version 1.6 improves forward-compatibility with
planned future directory formats, allowing updates to a
directory containing both current and future links, without
loss of information.


LICENCE

You may use this package under the GNU General Public License,
version 2 or, at your option, any later version. See the file
"COPYING.GPL" [5] for the terms of the GNU General Public
License, version 2.

You may use this package under the Transitive Grace Period
Public Licence, version 1 or, at your option, any later
version. (The Transitive Grace Period Public Licence has
requirements similar to the GPL except that it allows you to
wait for up to twelve months after you redistribute a derived
work before releasing the source code of your derived work.)
See the file "COPYING.TGPPL.html" [6] for the terms of the
Transitive Grace Period Public Licence, version 1.

(You may choose to use this package under the terms of either
licence, at your option.)


INSTALLATION

Tahoe-LAFS works on Linux, Mac OS X, Windows, Cygwin, Solaris,
*BSD, and probably most other systems. Start with
"docs/install.html" [7].


HACKING AND COMMUNITY

Please join us on the mailing list [8]. Patches are gratefully
accepted -- the RoadMap page [9] shows the next improvements
that we plan to make and CREDITS [10] lists the names of people
who've contributed to the project. The Dev page [11] contains
resources for hackers.


SPONSORSHIP

Tahoe-LAFS was originally developed thanks to the sponsorship
of Allmydata, Inc. [12], a provider of commercial backup
services. Allmydata founded the Tahoe-LAFS project and
contributed hardware, software, ideas, bug reports,
suggestions, demands, and they employed several Tahoe-LAFS
hackers and instructed them to spend part of their work time on
this Free Software project. Also they awarded customized
t-shirts to hackers who found security flaws in Tahoe-LAFS (see
the Hack Tahoe-LAFS Hall Of Fame [13]). After discontinuing
funding of Tahoe-LAFS R&D in early 2009, Allmydata, Inc. has
continued to provide servers, co-lo space, bandwidth, and small
personal gifts as tokens of appreciation. (Also they continue
to provide bug reports.) Thank you to Allmydata, Inc. for their
generous and public-spirited support.

This is the third release of Tahoe-LAFS to be created solely as
a labor of love by volunteers. Thank you very much to the
dedicated team of "hackers in the public interest" who make
Tahoe-LAFS possible.

Zooko Wilcox-O'Hearn
on behalf of the Tahoe-LAFS team

February 1, 2010
Boulder, Colorado, USA

[1] http://allmydata.org/trac/tahoe/browser/relnotes.txt?rev=4042
[2] http://allmydata.org/trac/tahoe/browser/NEWS?rev=4189
[3] http://allmydata.org/trac/tahoe/wiki/RelatedProjects
[4] http://allmydata.org/trac/tahoe/browser/docs/known_issues.txt
[5] http://allmydata.org/trac/tahoe/browser/COPYING.GPL
[6] http://allmydata.org/source/tahoe/trunk/COPYING.TGPPL.html
[7] http://allmydata.org/source/tahoe/trunk/docs/install.html
[8] http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev
[9] http://allmydata.org/trac/tahoe/roadmap
[10] http://allmydata.org/trac/tahoe/browser/CREDITS?rev=4186
[11] http://allmydata.org/trac/tahoe/wiki/Dev
[12] http://allmydata.com
[13] http://hacktahoe.org

From general-return-1024-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 10 23:07:47 2010
Return-Path: <general-return-1024-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 48401 invoked from network); 10 Feb 2010 23:07:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Feb 2010 23:07:47 -0000
Received: (qmail 97500 invoked by uid 500); 10 Feb 2010 23:07:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97426 invoked by uid 500); 10 Feb 2010 23:07:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97416 invoked by uid 99); 10 Feb 2010 23:07:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Feb 2010 23:07:45 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Feb 2010 23:07:34 +0000
Received: from wlan-snve-154-213.corp.yahoo.com (snvvpn4-10-72-168-c194.hq.corp.yahoo.com [10.72.168.194])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o1AN5lLw057406
	for <general@hadoop.apache.org>; Wed, 10 Feb 2010 15:05:48 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:content-type:
	content-transfer-encoding:mime-version:subject:date:x-mailer;
	b=NZ4kpy/hyFjBYhNXa5y/wjEwQoCgct+ROZsYMyJsbm0FmxqTkB+XtQvT1BWtSJ1W
Message-Id: <6800345F-07A4-4824-BDC2-87023707A793@yahoo-inc.com>
From: "Owen O'Malley" <oom@yahoo-inc.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: [VOTE] Release candidate for Hadoop 0.20.2
Date: Wed, 10 Feb 2010 15:05:47 -0800
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

PMC,
     I've rolled a release candidate for 0.20.2 and uploaded it to http://people.apache.org/~omalley/0.20.2/ 
. Please download it and test it out.

Thanks,
    Owen

From general-return-1025-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 11 06:52:07 2010
Return-Path: <general-return-1025-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35067 invoked from network); 11 Feb 2010 06:52:06 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2010 06:52:06 -0000
Received: (qmail 70236 invoked by uid 500); 11 Feb 2010 06:52:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70125 invoked by uid 500); 11 Feb 2010 06:52:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70115 invoked by uid 99); 11 Feb 2010 06:52:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Feb 2010 06:52:03 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.85.222.181] (HELO mail-pz0-f181.google.com) (209.85.222.181)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Feb 2010 06:51:56 +0000
Received: by pzk11 with SMTP id 11so1132986pzk.30
        for <general@hadoop.apache.org>; Wed, 10 Feb 2010 22:51:36 -0800 (PST)
MIME-Version: 1.0
Received: by 10.142.250.3 with SMTP id x3mr877922wfh.347.1265871096143; Wed, 
	10 Feb 2010 22:51:36 -0800 (PST)
In-Reply-To: <6800345F-07A4-4824-BDC2-87023707A793@yahoo-inc.com>
References: <6800345F-07A4-4824-BDC2-87023707A793@yahoo-inc.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Wed, 10 Feb 2010 22:51:16 -0800
Message-ID: <45f85f71002102251w1b84594bq84c232f218b09acc@mail.gmail.com>
Subject: Re: [VOTE] Release candidate for Hadoop 0.20.2
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

-0 on this particular tarball (md5sum
9759e01d7426c9bbe14758bf9ab69012). Trying to compile from the tarball
with ant -Dcompile.c++=3Dtrue -Dcompile.native=3Dtrue -Dlibhdfs=3Dtrue
bin-package I ran into these issues on my karmic box at home:

1) The configure scripts are not all executable.

BUILD FAILED
/tmp/hadoop-0.20.2/build.xml:1418: Execute failed: java.io.IOException: Can=
not r
un program "/tmp/hadoop-0.20.2/src/c++/pipes/configure" (in directory "/tmp=
/hado
op-0.20.2/build/c++-build/Linux-amd64-64/pipes"): java.io.IOException: erro=
r=3D13,
 Permission denied

I applied HADOOP-5612 to fix this, though I think creating a tarball
after chmod 755ing the configure scripts would also be correct.

Buildfile: build.xml

2) native build failed due to missing header includes in c++ code

I applied the patches from HADOOP-5611 and MAPREDUCE-1251 to get past this

These same problems are present as seen in the branch-20 Hudson build:
http://hudson.zones.apache.org/hudson/job/Hadoop-20-Build/67/console

3) It would be nice if the tarball had libhdfs included in c++/lib in
addition to the pipes libraries. libhdfs wasn't in 0.20.1 either,
though, so I don't think it's a big deal. If you end up doing a
rebuild, people would probably be happy to see -Dlibhdfs=3Dtrue in the
release.

I say -0 instead of -1 because none of these issues are regressions
from 0.20.1. When we did 0.20.1 we didn't notice some all of these,
since several are problems that only show up on some newer distros
(like karmic). Since we're doing a new release, and the patches apply
cleanly and are pretty trivial, I think we should include them.

Thanks
-Todd


On Wed, Feb 10, 2010 at 3:05 PM, Owen O'Malley <oom@yahoo-inc.com> wrote:
> PMC,
> =A0 =A0I've rolled a release candidate for 0.20.2 and uploaded it to
> http://people.apache.org/~omalley/0.20.2/. Please download it and test it
> out.
>
> Thanks,
> =A0 Owen
>

From general-return-1026-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 11 17:00:58 2010
Return-Path: <general-return-1026-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67409 invoked from network); 11 Feb 2010 17:00:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2010 17:00:57 -0000
Received: (qmail 94108 invoked by uid 500); 11 Feb 2010 17:00:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94048 invoked by uid 500); 11 Feb 2010 17:00:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94031 invoked by uid 99); 11 Feb 2010 17:00:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Feb 2010 17:00:55 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.59.211] (HELO QMTA11.westchester.pa.mail.comcast.net) (76.96.59.211)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Feb 2010 17:00:47 +0000
Received: from omta14.westchester.pa.mail.comcast.net ([76.96.62.60])
	by QMTA11.westchester.pa.mail.comcast.net with comcast
	id gejH1d00B1HzFnQ5Bh0Txp; Thu, 11 Feb 2010 17:00:27 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta14.westchester.pa.mail.comcast.net with comcast
	id gh0H1d00F2SbwD53ah0Ldr; Thu, 11 Feb 2010 17:00:25 +0000
Message-Id: <88756F25-FCA8-475B-8E97-F2821911B7CA@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <45f85f71002102251w1b84594bq84c232f218b09acc@mail.gmail.com>
Content-Type: multipart/alternative; boundary=Apple-Mail-21--162391534
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Release candidate for Hadoop 0.20.2
Date: Thu, 11 Feb 2010 09:00:17 -0800
References: <6800345F-07A4-4824-BDC2-87023707A793@yahoo-inc.com> <45f85f71002102251w1b84594bq84c232f218b09acc@mail.gmail.com>
X-Mailer: Apple Mail (2.936)

--Apple-Mail-21--162391534
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


On Feb 10, 2010, at 10:51 PM, Todd Lipcon wrote:

> I applied HADOOP-5612 to fix this, though I think creating a tarball
> after chmod 755ing the configure scripts would also be correct.

*Sigh* You can't blame a release for not containing one of your  
patches, if you didn't ask for it to be applied to the relevant  
branch. If you want it to be included in the next 0.20 release, reopen  
the jira and ask for it to be applied back to branch-0.20.

> These same problems are present as seen in the branch-20 Hudson build:
> http://hudson.zones.apache.org/hudson/job/Hadoop-20-Build/67/console

The problem on Hudson seems to be that automake 1.9 is not installed.  
That seems like a different issue.

> 3) It would be nice if the tarball had libhdfs included in c++/lib

It does have the .so for libhdfs.

  hadoop-0.20.2/c++/Linux-i386-32/lib/libhdfs.so

I assume you were looking for the 64 bit version. Did someone get the  
64 bit libhdfs working? If so, we should update the HowToRelease twiki  
to include it.

-- Owen
--Apple-Mail-21--162391534--

From general-return-1027-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 11 17:18:56 2010
Return-Path: <general-return-1027-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78294 invoked from network); 11 Feb 2010 17:18:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2010 17:18:56 -0000
Received: (qmail 27325 invoked by uid 500); 11 Feb 2010 17:18:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27243 invoked by uid 500); 11 Feb 2010 17:18:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27233 invoked by uid 99); 11 Feb 2010 17:18:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Feb 2010 17:18:55 +0000
X-ASF-Spam-Status: No, hits=-2.5 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Feb 2010 17:18:46 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:User-Agent:Date:Subject:From:To:
   Message-ID:Thread-Topic:Thread-Index:In-Reply-To:
   Mime-version:Content-type:Content-transfer-encoding;
  b=lYXBd5kUjvpraeQkRE3XCp4PZQWNwo/wkzSSUN4lDNVtYpRZUdLjTu1t
   GJrdjE5Hzcfm8qJ/KzEzJowFHIuLavpic+Q4eiYsqAygi7ZQMAisBy40p
   nX7HjOdGJ/b9LcG;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1265908726; x=1297444726;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20[VOTE]=20Release=20candidate=20for=20Ha
   doop=200.20.2|Date:=20Thu,=2011=20Feb=202010=2009:18:00
   =20-0800|Message-ID:=20<C7997BC8.2F58%awittenauer@linkedi
   n.com>|To:=20<general@hadoop.apache.org>|Mime-version:=20
   1.0|Content-transfer-encoding:=207bit|In-Reply-To:=20<887
   56F25-FCA8-475B-8E97-F2821911B7CA@apache.org>;
  bh=pwHhgnkRxCdCveEAg641hjNmGHYcCiyX75I3O3tV+Ss=;
  b=tZ5RtirpqldE+xKIZj+o8GyLqUnOKHzd2U8+mI30bGtKqL2D6INP68/I
   cSljqY/Z2Z9g4aP8pW3a1CG5h3fJBAc9rkYnXomJNgSAy73WjxsFtdxru
   29KsrRW0z8FtZ2Z;
X-IronPort-AV: E=Sophos;i="4.49,453,1262592000"; 
   d="scan'208";a="11162974"
Received: from 172.16.19.141 ([172.16.19.141]) by CORP-MAIL.linkedin.biz ([172.18.46.135]) via Exchange Front-End Server mail-access.linkedin.biz ([172.18.46.133]) with Microsoft Exchange Server HTTP-DAV ;
 Thu, 11 Feb 2010 17:18:04 +0000
User-Agent: Microsoft-Entourage/12.10.0.080409
Date: Thu, 11 Feb 2010 09:18:00 -0800
Subject: Re: [VOTE] Release candidate for Hadoop 0.20.2
From: Allen Wittenauer <awittenauer@linkedin.com>
To: <general@hadoop.apache.org>
Message-ID: <C7997BC8.2F58%awittenauer@linkedin.com>
Thread-Topic: [VOTE] Release candidate for Hadoop 0.20.2
Thread-Index: AcqrPikjdG6pJwJeg0enhiq5WmWxyg==
In-Reply-To: <88756F25-FCA8-475B-8E97-F2821911B7CA@apache.org>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit




On 2/11/10 9:00 AM, "Owen O'Malley" <omalley@apache.org> wrote:

>> These same problems are present as seen in the branch-20 Hudson build:
>> http://hudson.zones.apache.org/hudson/job/Hadoop-20-Build/67/console
> 
> The problem on Hudson seems to be that automake 1.9 is not installed.
> That seems like a different issue.

     [exec] 
/grid/0/hudson/hudson-slave/workspace/Hadoop-20-Build/trunk/src/c++/utils/im
pl/StringUtils.cc: In function 'uint64_t HadoopUtils::getCurrentMillis()':
     [exec] 
/grid/0/hudson/hudson-slave/workspace/Hadoop-20-Build/trunk/src/c++/utils/im
pl/StringUtils.cc:74: error: 'strerror' was not declared in this scope
     [exec] 
/grid/0/hudson/hudson-slave/workspace/Hadoop-20-Build/trunk/src/c++/utils/im
pl/StringUtils.cc: In function 'std::string HadoopUtils::quoteString(const
std::string&, const char*)':
     [exec] 
/grid/0/hudson/hudson-slave/workspace/Hadoop-20-Build/trunk/src/c++/utils/im
pl/StringUtils.cc:103: error: 'strchr' was not declared in this scope
     [exec] 
/grid/0/hudson/hudson-slave/workspace/Hadoop-20-Build/trunk/src/c++/utils/im
pl/StringUtils.cc: In function 'std::string HadoopUtils::unquoteString(const
std::string&)':
     [exec] 
/grid/0/hudson/hudson-slave/workspace/Hadoop-20-Build/trunk/src/c++/utils/im
pl/StringUtils.cc:144: error: 'strtol' was not declared in this scope
     
... has more to do with missing string header files than automake. :)


> 
>> 3) It would be nice if the tarball had libhdfs included in c++/lib
> 
> It does have the .so for libhdfs.
> 
>   hadoop-0.20.2/c++/Linux-i386-32/lib/libhdfs.so
> 
> I assume you were looking for the 64 bit version. Did someone get the
> 64 bit libhdfs working? If so, we should update the HowToRelease twiki
> to include it.
> 
> -- Owen


From general-return-1028-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 11 17:21:31 2010
Return-Path: <general-return-1028-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 79472 invoked from network); 11 Feb 2010 17:21:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Feb 2010 17:21:30 -0000
Received: (qmail 34294 invoked by uid 500); 11 Feb 2010 17:21:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34227 invoked by uid 500); 11 Feb 2010 17:21:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34217 invoked by uid 99); 11 Feb 2010 17:21:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Feb 2010 17:21:29 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.92.26] (HELO qw-out-2122.google.com) (74.125.92.26)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Feb 2010 17:21:23 +0000
Received: by qw-out-2122.google.com with SMTP id 9so311211qwb.35
        for <general@hadoop.apache.org>; Thu, 11 Feb 2010 09:21:02 -0800 (PST)
MIME-Version: 1.0
Received: by 10.143.21.36 with SMTP id y36mr98917wfi.160.1265908862164; Thu, 
	11 Feb 2010 09:21:02 -0800 (PST)
In-Reply-To: <88756F25-FCA8-475B-8E97-F2821911B7CA@apache.org>
References: <6800345F-07A4-4824-BDC2-87023707A793@yahoo-inc.com> 
	<45f85f71002102251w1b84594bq84c232f218b09acc@mail.gmail.com> 
	<88756F25-FCA8-475B-8E97-F2821911B7CA@apache.org>
From: Todd Lipcon <todd@cloudera.com>
Date: Thu, 11 Feb 2010 09:20:42 -0800
Message-ID: <45f85f71002110920r567d73e1n7ab649f01bba72dc@mail.gmail.com>
Subject: Re: [VOTE] Release candidate for Hadoop 0.20.2
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Thu, Feb 11, 2010 at 9:00 AM, Owen O'Malley <omalley@apache.org> wrote:
>
> On Feb 10, 2010, at 10:51 PM, Todd Lipcon wrote:
>
>> I applied HADOOP-5612 to fix this, though I think creating a tarball
>> after chmod 755ing the configure scripts would also be correct.
>
> *Sigh* You can't blame a release for not containing one of your patches, =
if
> you didn't ask for it to be applied to the relevant branch. If you want i=
t
> to be included in the next 0.20 release, reopen the jira and ask for it t=
o
> be applied back to branch-0.20.
>

Yes, I should have asked at the time to put it in 20. I had newly
started working on Hadoop when I did that patch and didn't know the
proper JIRA protocol at the time. Will do so now. Like I said in my
mail, I'm not trying to block a release. I'm just reporting the errors
I ran into trying to use the release candidate - if the rest of the
community or PMC thinks it's not worth rolling a new one, I'll leave
it up to them.

Regardless, it has nothing to do with whether it's "my patch" - I
think it reflects poorly on the project when the releases fail to
build on a common system configuration.

>> These same problems are present as seen in the branch-20 Hudson build:
>> http://hudson.zones.apache.org/hudson/job/Hadoop-20-Build/67/console
>
> The problem on Hudson seems to be that automake 1.9 is not installed. Tha=
t
> seems like a different issue.
>

automake should not be a build time requirement for released artifacts
in pretty much any sane project. autotools "make dist" targets always
include a configure script, which means automake was already run by
the developer. After applying the patches mentioned above that fix
missing header includes, I built just fine on my box that has no
automake-1.9.

>> 3) It would be nice if the tarball had libhdfs included in c++/lib
>
> It does have the .so for libhdfs.
>
> =A0hadoop-0.20.2/c++/Linux-i386-32/lib/libhdfs.so
>
> I assume you were looking for the 64 bit version. Did someone get the 64 =
bit
> libhdfs working? If so, we should update the HowToRelease twiki to includ=
e
> it.

After applying the patches I mentioned, the ant command quoted above
succeeded in building libhdfs just fine:
build/c++/Linux-amd64-64/lib/libhdfs.so.0.0.0: ELF 64-bit LSB shared
object, x86-64, version 1 (SYSV), dynamically linked, not stripped


If others think these are blockers, I'm happy to roll a new candidate
tarball later this week once they've been pushed into branch-20.
Although I don't have a binding vote in the release process, I am
pretty sure the ASF policies are fine with anyone rolling an rc and
calling a vote. If no one else thinks these patches are worth stalling
for, that's fine too.

Thanks
-Todd

From general-return-1029-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 12 10:42:07 2010
Return-Path: <general-return-1029-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42792 invoked from network); 12 Feb 2010 10:42:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Feb 2010 10:42:07 -0000
Received: (qmail 46650 invoked by uid 500); 12 Feb 2010 10:42:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46542 invoked by uid 500); 12 Feb 2010 10:42:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46532 invoked by uid 99); 12 Feb 2010 10:42:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Feb 2010 10:42:04 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 12 Feb 2010 10:42:04 +0000
Received: (qmail 42627 invoked by uid 99); 12 Feb 2010 10:41:43 -0000
Received: from localhost.apache.org (HELO mail-bw0-f209.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Feb 2010 10:41:43 +0000
Received: by bwz1 with SMTP id 1so1881131bwz.32
        for <general@hadoop.apache.org>; Fri, 12 Feb 2010 02:41:41 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.138.81 with SMTP id z17mr753601bkt.49.1265971301412; Fri, 
	12 Feb 2010 02:41:41 -0800 (PST)
In-Reply-To: <45f85f71002110920r567d73e1n7ab649f01bba72dc@mail.gmail.com>
References: <6800345F-07A4-4824-BDC2-87023707A793@yahoo-inc.com>
	 <45f85f71002102251w1b84594bq84c232f218b09acc@mail.gmail.com>
	 <88756F25-FCA8-475B-8E97-F2821911B7CA@apache.org>
	 <45f85f71002110920r567d73e1n7ab649f01bba72dc@mail.gmail.com>
Date: Fri, 12 Feb 2010 02:41:41 -0800
Message-ID: <1267dd3b1002120241u78569b05v2bc9b3b60cca2077@mail.gmail.com>
Subject: Re: [VOTE] Release candidate for Hadoop 0.20.2
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I committed HADOOP-5611 to 0.20 and MAPREDUCE-1251 to 0.21 and 0.20. I
am suggesting slight modifications to HADOOP-5612 but think it should
also be included (btw: the original patch was *not* licensed to the
ASF; if the author re-posts the patch with permission granted, would
that be sufficient?). A backport of MAPREDUCE-623 should also be
applied to 0.20; the javac warnings are pretty ugly, if benign.

None of these issues are critical, though; if no other issues are
found with this RC, I'm +1 on releasing it, but would like to include
these if another candidate is rolled. -C

On Thu, Feb 11, 2010 at 9:20 AM, Todd Lipcon <todd@cloudera.com> wrote:
> On Thu, Feb 11, 2010 at 9:00 AM, Owen O'Malley <omalley@apache.org> wrote=
:
>>
>> On Feb 10, 2010, at 10:51 PM, Todd Lipcon wrote:
>>
>>> I applied HADOOP-5612 to fix this, though I think creating a tarball
>>> after chmod 755ing the configure scripts would also be correct.
>>
>> *Sigh* You can't blame a release for not containing one of your patches,=
 if
>> you didn't ask for it to be applied to the relevant branch. If you want =
it
>> to be included in the next 0.20 release, reopen the jira and ask for it =
to
>> be applied back to branch-0.20.
>>
>
> Yes, I should have asked at the time to put it in 20. I had newly
> started working on Hadoop when I did that patch and didn't know the
> proper JIRA protocol at the time. Will do so now. Like I said in my
> mail, I'm not trying to block a release. I'm just reporting the errors
> I ran into trying to use the release candidate - if the rest of the
> community or PMC thinks it's not worth rolling a new one, I'll leave
> it up to them.
>
> Regardless, it has nothing to do with whether it's "my patch" - I
> think it reflects poorly on the project when the releases fail to
> build on a common system configuration.
>
>>> These same problems are present as seen in the branch-20 Hudson build:
>>> http://hudson.zones.apache.org/hudson/job/Hadoop-20-Build/67/console
>>
>> The problem on Hudson seems to be that automake 1.9 is not installed. Th=
at
>> seems like a different issue.
>>
>
> automake should not be a build time requirement for released artifacts
> in pretty much any sane project. autotools "make dist" targets always
> include a configure script, which means automake was already run by
> the developer. After applying the patches mentioned above that fix
> missing header includes, I built just fine on my box that has no
> automake-1.9.
>
>>> 3) It would be nice if the tarball had libhdfs included in c++/lib
>>
>> It does have the .so for libhdfs.
>>
>> =A0hadoop-0.20.2/c++/Linux-i386-32/lib/libhdfs.so
>>
>> I assume you were looking for the 64 bit version. Did someone get the 64=
 bit
>> libhdfs working? If so, we should update the HowToRelease twiki to inclu=
de
>> it.
>
> After applying the patches I mentioned, the ant command quoted above
> succeeded in building libhdfs just fine:
> build/c++/Linux-amd64-64/lib/libhdfs.so.0.0.0: ELF 64-bit LSB shared
> object, x86-64, version 1 (SYSV), dynamically linked, not stripped
>
>
> If others think these are blockers, I'm happy to roll a new candidate
> tarball later this week once they've been pushed into branch-20.
> Although I don't have a binding vote in the release process, I am
> pretty sure the ASF policies are fine with anyone rolling an rc and
> calling a vote. If no one else thinks these patches are worth stalling
> for, that's fine too.
>
> Thanks
> -Todd
>

From general-return-1030-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 16 19:23:28 2010
Return-Path: <general-return-1030-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 13766 invoked from network); 16 Feb 2010 19:23:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Feb 2010 19:23:28 -0000
Received: (qmail 4336 invoked by uid 500); 16 Feb 2010 19:23:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4263 invoked by uid 500); 16 Feb 2010 19:23:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4247 invoked by uid 99); 16 Feb 2010 19:23:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Feb 2010 19:23:26 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Feb 2010 19:23:16 +0000
Received: from [192.168.0.198] (snvvpn1-10-72-244-c242.hq.corp.yahoo.com [10.72.244.242])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o1GJMq2E000778
	for <general@hadoop.apache.org>; Tue, 16 Feb 2010 11:22:52 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:content-type:
	content-transfer-encoding:mime-version:subject:date:x-mailer;
	b=WfQoZ4pKrUiL4bOb4nguExsLGWme53ljPv4Z09VJaG8aYDQanPdXfpNxl8x/8e0I
Message-Id: <3316F847-FBB7-4D41-B1A7-C5CD3FB1A40D@yahoo-inc.com>
From: Alan Gates <gates@yahoo-inc.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: [VOTE] Release candidate for Pig 0.6.0
Date: Tue, 16 Feb 2010 11:22:52 -0800
X-Mailer: Apple Mail (2.936)

All,

I have rolled a release candidate for Pig 0.6.0 and uploaded it to http://people.apache.org/~gates/pig-0.6.0-candidate-1/
A description of what is new and different in this release is included  
in the release notes, which I have also uploaded to that directory.   
Please download the candidate, test it, and vote.

Alan.

From general-return-1031-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 16 22:12:39 2010
Return-Path: <general-return-1031-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83417 invoked from network); 16 Feb 2010 22:12:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Feb 2010 22:12:39 -0000
Received: (qmail 77370 invoked by uid 500); 16 Feb 2010 22:12:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77290 invoked by uid 500); 16 Feb 2010 22:12:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77280 invoked by uid 99); 16 Feb 2010 22:12:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Feb 2010 22:12:38 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.83.176] (HELO mail-pv0-f48.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 Feb 2010 22:12:30 +0000
Received: by pva18 with SMTP id 18so8681pva.35
        for <general@hadoop.apache.org>; Tue, 16 Feb 2010 14:12:08 -0800 (PST)
MIME-Version: 1.0
Received: by 10.142.210.17 with SMTP id i17mr4761838wfg.146.1266358328332; 
	Tue, 16 Feb 2010 14:12:08 -0800 (PST)
Date: Tue, 16 Feb 2010 14:12:08 -0800
Message-ID: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com>
Subject: Release plans
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hey guys,

What are the current plans for the 21 release?

The roadmap on the wiki [1] says "minor releases are made regularly,
every few months." The 21 branch was created 5 months ago, and the
release dates in jira for the various core projects have all passed. I
know some projects, eg HBase, would like to get users on more recent
bits. What are the next steps for rolling a release?

Thanks,
Eli

[1] http://wiki.apache.org/hadoop/Roadmap

From general-return-1032-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 17 00:13:38 2010
Return-Path: <general-return-1032-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37449 invoked from network); 17 Feb 2010 00:13:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Feb 2010 00:13:38 -0000
Received: (qmail 22300 invoked by uid 500); 17 Feb 2010 00:13:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22237 invoked by uid 500); 17 Feb 2010 00:13:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22227 invoked by uid 99); 17 Feb 2010 00:13:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Feb 2010 00:13:37 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 17 Feb 2010 00:13:36 +0000
Received: (qmail 37348 invoked by uid 99); 17 Feb 2010 00:13:16 -0000
Received: from localhost.apache.org (HELO [192.168.168.104]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Feb 2010 00:13:16 +0000
Message-ID: <4B7B349B.1090106@apache.org>
Date: Tue, 16 Feb 2010 16:13:15 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: [Fwd: [VOTE] Avro release 1.3.0 (RC 0)]
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

FYI, a release vote has started on avro-dev.

Please vote on that list, not here.

Thanks,

Doug

-------- Original Message --------
Subject: [VOTE] Avro release 1.3.0 (RC 0)
Date: Tue, 16 Feb 2010 16:07:49 -0800
From: Doug Cutting <cutting@apache.org>
Reply-To: avro-dev@hadoop.apache.org
To: avro-dev@hadoop.apache.org

I have created a candidate build for Avro release 1.3.0.

In this release:
  - the Avro file format has been revised and simplified;
  - the source tree and release artifacts have been restructured;
  - the Python port has been largely rewritten;
  - the C port is now considerably more complete, supporting data files;
  - a Ruby port has been added, supporting data files and RPC.

Please download, test, and vote by 19 February.

http://people.apache.org/~cutting/avro-1.3.0-RC0/

Thanks,

Doug

From general-return-1033-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 18 01:25:41 2010
Return-Path: <general-return-1033-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22062 invoked from network); 18 Feb 2010 01:25:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2010 01:25:41 -0000
Received: (qmail 73709 invoked by uid 500); 18 Feb 2010 01:25:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73622 invoked by uid 500); 18 Feb 2010 01:25:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73612 invoked by uid 99); 18 Feb 2010 01:25:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 01:25:39 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.30.96] (HELO qmta09.emeryville.ca.mail.comcast.net) (76.96.30.96)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 01:25:29 +0000
Received: from omta13.emeryville.ca.mail.comcast.net ([76.96.30.52])
	by qmta09.emeryville.ca.mail.comcast.net with comcast
	id jD4d1d00317UAYkA9DR9sb; Thu, 18 Feb 2010 01:25:09 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta13.emeryville.ca.mail.comcast.net with comcast
	id jDR01d00K2SbwD58ZDR2hu; Thu, 18 Feb 2010 01:25:06 +0000
Message-Id: <BE5B5A4B-CDBF-473F-B760-DBC1E1ED33C2@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: [VOTE] Should we release Hadoop 0.20.2 rc2?
Date: Wed, 17 Feb 2010 17:24:59 -0800
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

Ok, I've rolled a new tarball with Todd's suggested updates to the  
branch. Please try it out and vote.

http://people.apache.org/~omalley/0.20.2/

-- Owen

From general-return-1034-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 18 03:29:06 2010
Return-Path: <general-return-1034-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68200 invoked from network); 18 Feb 2010 03:29:06 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2010 03:29:06 -0000
Received: (qmail 41377 invoked by uid 500); 18 Feb 2010 03:29:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 41220 invoked by uid 500); 18 Feb 2010 03:29:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 41210 invoked by uid 99); 18 Feb 2010 03:29:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 03:29:03 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.85.222.195] (HELO mail-pz0-f195.google.com) (209.85.222.195)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 03:28:57 +0000
Received: by pzk33 with SMTP id 33so3944883pzk.2
        for <general@hadoop.apache.org>; Wed, 17 Feb 2010 19:28:36 -0800 (PST)
MIME-Version: 1.0
Received: by 10.141.100.5 with SMTP id c5mr5945836rvm.3.1266463716250; Wed, 17 
	Feb 2010 19:28:36 -0800 (PST)
In-Reply-To: <BE5B5A4B-CDBF-473F-B760-DBC1E1ED33C2@apache.org>
References: <BE5B5A4B-CDBF-473F-B760-DBC1E1ED33C2@apache.org>
From: Todd Lipcon <todd@cloudera.com>
Date: Wed, 17 Feb 2010 19:28:16 -0800
Message-ID: <45f85f71002171928w1e8aeba7u4562f8255871b61c@mail.gmail.com>
Subject: Re: [VOTE] Should we release Hadoop 0.20.2 rc2?
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hey Owen,

Thanks for rolling the second rc!

Looks like that file needs a chmod, getting 404 Forbidden:

[todd@minotaur:/home/omalley/public_html/0.20.2]$ ls -l
total 40664
-rw-r-----  1 omalley  omalley  41662994 Feb 18 01:21 hadoop-0.20.2.tar.gz
-rw-r--r--  1 omalley  omalley       195 Feb 18 01:21 hadoop-0.20.2.tar.gz.asc
-rw-r--r--  1 omalley  omalley      1034 Feb 18 01:21 hadoop-0.20.2.tar.gz.mds

-Todd


On Wed, Feb 17, 2010 at 5:24 PM, Owen O'Malley <omalley@apache.org> wrote:
> Ok, I've rolled a new tarball with Todd's suggested updates to the branch.
> Please try it out and vote.
>
> http://people.apache.org/~omalley/0.20.2/
>
> -- Owen
>

From general-return-1035-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 18 04:10:20 2010
Return-Path: <general-return-1035-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76643 invoked from network); 18 Feb 2010 04:10:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2010 04:10:20 -0000
Received: (qmail 65273 invoked by uid 500); 18 Feb 2010 04:10:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65100 invoked by uid 500); 18 Feb 2010 04:10:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65090 invoked by uid 99); 18 Feb 2010 04:10:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 04:10:17 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 74.125.92.26 as permitted sender)
Received: from [74.125.92.26] (HELO qw-out-2122.google.com) (74.125.92.26)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 04:10:11 +0000
Received: by qw-out-2122.google.com with SMTP id 5so1185207qwi.35
        for <general@hadoop.apache.org>; Wed, 17 Feb 2010 20:09:50 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        bh=U1nCQIbFvdjJJmWZ6CJWfhQ1UIjzB8NQPvkZQhHrF5g=;
        b=VSR+9Rs+m9Xxxu5WpyI+zqJdtEoSdpjxtW7xmL7NjnZrILwgf5enjQ6QSeZ49irPm6
         OAyVFhvbNN8EEitkVVJuk4xEiJuMqbchtd1EoWHzbh+d13+0Vll2y4CBj3n33vLrt4Tm
         mHOEHHJ2xKlNmh1YyxQ7Gwpj9l01Uz7dA6hzc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=f87MWOa8V1OWFuRMvJudSC5aRac99VZyoIIc7/RK75kkTp8uHxkRKxVvdvZgKemnsh
         UrocvbBzV/qPr/nz68biU3vJLC7UhXMvjy3+qQPX+eCFaPDweLq6Mf0x3dUJRpztVQ00
         Cpxj750IOxsmaYJ7v/Xaek8dmgfztdQ9xhP4Y=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
Received: by 10.229.241.82 with SMTP id ld18mr676392qcb.60.1266466189929; Wed, 
	17 Feb 2010 20:09:49 -0800 (PST)
In-Reply-To: <45f85f71002171928w1e8aeba7u4562f8255871b61c@mail.gmail.com>
References: <BE5B5A4B-CDBF-473F-B760-DBC1E1ED33C2@apache.org>
	 <45f85f71002171928w1e8aeba7u4562f8255871b61c@mail.gmail.com>
Date: Wed, 17 Feb 2010 20:09:49 -0800
X-Google-Sender-Auth: 8b8d134678cb853b
Message-ID: <7c962aed1002172009g1a4d4978v472c16d1c2bdcf6b@mail.gmail.com>
Subject: Re: [VOTE] Should we release Hadoop 0.20.2 rc2?
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Yeah, me too...

$ wget http://people.apache.org/~omalley/0.20.2/hadoop-0.20.2.tar.gz
--04:09:10--  http://people.apache.org/~omalley/0.20.2/hadoop-0.20.2.tar.gz
Resolving people.apache.org... 140.211.11.9
Connecting to people.apache.org|140.211.11.9|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
04:09:10 ERROR 403: Forbidden.

St.Ack

On Wed, Feb 17, 2010 at 7:28 PM, Todd Lipcon <todd@cloudera.com> wrote:
> Hey Owen,
>
> Thanks for rolling the second rc!
>
> Looks like that file needs a chmod, getting 404 Forbidden:
>
> [todd@minotaur:/home/omalley/public_html/0.20.2]$ ls -l
> total 40664
> -rw-r----- =A01 omalley =A0omalley =A041662994 Feb 18 01:21 hadoop-0.20.2=
.tar.gz
> -rw-r--r-- =A01 omalley =A0omalley =A0 =A0 =A0 195 Feb 18 01:21 hadoop-0.=
20.2.tar.gz.asc
> -rw-r--r-- =A01 omalley =A0omalley =A0 =A0 =A01034 Feb 18 01:21 hadoop-0.=
20.2.tar.gz.mds
>
> -Todd
>
>
> On Wed, Feb 17, 2010 at 5:24 PM, Owen O'Malley <omalley@apache.org> wrote=
:
>> Ok, I've rolled a new tarball with Todd's suggested updates to the branc=
h.
>> Please try it out and vote.
>>
>> http://people.apache.org/~omalley/0.20.2/
>>
>> -- Owen
>>
>

From general-return-1036-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 18 05:59:48 2010
Return-Path: <general-return-1036-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4686 invoked from network); 18 Feb 2010 05:59:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2010 05:59:48 -0000
Received: (qmail 11531 invoked by uid 500); 18 Feb 2010 05:59:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11337 invoked by uid 500); 18 Feb 2010 05:59:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11327 invoked by uid 99); 18 Feb 2010 05:59:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 05:59:44 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of owen.omalley@gmail.com designates 209.85.222.195 as permitted sender)
Received: from [209.85.222.195] (HELO mail-pz0-f195.google.com) (209.85.222.195)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 05:59:36 +0000
Received: by pzk33 with SMTP id 33so4092863pzk.2
        for <general@hadoop.apache.org>; Wed, 17 Feb 2010 21:59:15 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=nn6ZFHCg5QrcLOT05QT9z4tOxsAMUaT5tULrgsiMSJk=;
        b=j+OOYUNk4vyGshbUQxRhEii19VPqk/EKme7x65ZcaHWfB6p29ZTZdQASUhRuWU654p
         K87yM7m6eJzIjnBNc8jtefO8Ju1iIaMq+xCNtTW2kQtqZnJwEb9wvbN8wPrivWbnZr7w
         IyQo6jRQiJk9Ehu6lWlmb1+WL7EOGR9TpecrU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=a9DYrAXNKPCEcuIbkuR55Bjkpw3gsP7XZ6xSXJksRNQq6RRH2yWRmOc9Tc+6twA0Tc
         xaKlksgOYeIIxSmN0f5Sg5b2XLd3JzBuJMi6PGEBuCJYWEGFEhaMVCMhoJeA5GiQoKk1
         PN3DsSka5lZ6tfIhVSgahKGlElh6ZEHjqRauk=
MIME-Version: 1.0
Received: by 10.141.23.12 with SMTP id a12mr6052830rvj.161.1266472755302; Wed, 
	17 Feb 2010 21:59:15 -0800 (PST)
In-Reply-To: <7c962aed1002172009g1a4d4978v472c16d1c2bdcf6b@mail.gmail.com>
References: <BE5B5A4B-CDBF-473F-B760-DBC1E1ED33C2@apache.org>
	 <45f85f71002171928w1e8aeba7u4562f8255871b61c@mail.gmail.com>
	 <7c962aed1002172009g1a4d4978v472c16d1c2bdcf6b@mail.gmail.com>
Date: Wed, 17 Feb 2010 21:59:15 -0800
Message-ID: <5f7f740b1002172159x4792d7e0n1999def6f8998d04@mail.gmail.com>
Subject: Re: [VOTE] Should we release Hadoop 0.20.2 rc2?
From: "Owen O'Malley" <owen.omalley@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd1708eb2932d047fd9aae0
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd1708eb2932d047fd9aae0
Content-Type: text/plain; charset=UTF-8

Ok, the permission problem should be fixed. Please try now.

-- Owen

--000e0cd1708eb2932d047fd9aae0--

From general-return-1037-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 18 07:22:28 2010
Return-Path: <general-return-1037-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 41579 invoked from network); 18 Feb 2010 07:22:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2010 07:22:28 -0000
Received: (qmail 56049 invoked by uid 500); 18 Feb 2010 07:22:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55998 invoked by uid 500); 18 Feb 2010 07:22:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55988 invoked by uid 99); 18 Feb 2010 07:22:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 07:22:26 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 07:22:18 +0000
Received: by pwi6 with SMTP id 6so316295pwi.35
        for <general@hadoop.apache.org>; Wed, 17 Feb 2010 23:21:57 -0800 (PST)
MIME-Version: 1.0
Received: by 10.141.106.13 with SMTP id i13mr6103647rvm.179.1266477717176; 
	Wed, 17 Feb 2010 23:21:57 -0800 (PST)
In-Reply-To: <5f7f740b1002172159x4792d7e0n1999def6f8998d04@mail.gmail.com>
References: <BE5B5A4B-CDBF-473F-B760-DBC1E1ED33C2@apache.org> 
	<45f85f71002171928w1e8aeba7u4562f8255871b61c@mail.gmail.com> 
	<7c962aed1002172009g1a4d4978v472c16d1c2bdcf6b@mail.gmail.com> 
	<5f7f740b1002172159x4792d7e0n1999def6f8998d04@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Wed, 17 Feb 2010 23:21:37 -0800
Message-ID: <45f85f71002172321q6b30b4dax20a95080659d3d3@mail.gmail.com>
Subject: Re: [VOTE] Should we release Hadoop 0.20.2 rc2?
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

All good. +1 from me:

- Verified Owen's GPG signature
- Verified the md5 sum 7b0ae9bbeb5bc932ad028932d5e7b5d5
- Unpacked, built java and native code both on 64-bit, karmic
- Started a pseudodistributed cluster and ran a couple of sleep jobs.

The only one tiny complaint is that there is still no 64-bit libhdfs.
But if there was an issue with that it's no big deal.

In case any crypto-happy people care, here's my signature on the tarball:
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEABECAAYFAkt86kwACgkQXkPKua7Hfq/ujwCfeu6pD7PwD9RtGbTHiXmlf+bp
4I8AoM/fv8q6ejJbxy9ApyB3RBXVpfNd
=V8Qd
-----END PGP SIGNATURE-----

Thanks again for the release work

-Todd

On Wed, Feb 17, 2010 at 9:59 PM, Owen O'Malley <owen.omalley@gmail.com> wrote:
> Ok, the permission problem should be fixed. Please try now.
>
> -- Owen
>

From general-return-1038-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 18 09:06:27 2010
Return-Path: <general-return-1038-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66632 invoked from network); 18 Feb 2010 09:06:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2010 09:06:27 -0000
Received: (qmail 62098 invoked by uid 500); 18 Feb 2010 09:06:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62013 invoked by uid 500); 18 Feb 2010 09:06:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62003 invoked by uid 99); 18 Feb 2010 09:06:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 09:06:25 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 18 Feb 2010 09:06:23 +0000
Received: (qmail 66591 invoked by uid 99); 18 Feb 2010 09:06:01 -0000
Received: from localhost.apache.org (HELO mail-fx0-f218.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 09:06:01 +0000
Received: by fxm10 with SMTP id 10so8554211fxm.29
        for <general@hadoop.apache.org>; Thu, 18 Feb 2010 01:05:59 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.14.84 with SMTP id f20mr449372bka.209.1266483959261; Thu, 
	18 Feb 2010 01:05:59 -0800 (PST)
In-Reply-To: <BE5B5A4B-CDBF-473F-B760-DBC1E1ED33C2@apache.org>
References: <BE5B5A4B-CDBF-473F-B760-DBC1E1ED33C2@apache.org>
Date: Thu, 18 Feb 2010 01:05:59 -0800
Message-ID: <1267dd3b1002180105y5339e19elf9712ed719a6500e@mail.gmail.com>
Subject: Re: [VOTE] Should we release Hadoop 0.20.2 rc2?
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

*sigh* The tests currently do not compile. HADOOP-6506 added a call in
the build:

+    <antcall target="run-test-core-fault-inject"/>

which does not exist in 0.20. -C

On Wed, Feb 17, 2010 at 5:24 PM, Owen O'Malley <omalley@apache.org> wrote:
> Ok, I've rolled a new tarball with Todd's suggested updates to the branch.
> Please try it out and vote.
>
> http://people.apache.org/~omalley/0.20.2/
>
> -- Owen
>

From general-return-1039-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 18 09:17:26 2010
Return-Path: <general-return-1039-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70193 invoked from network); 18 Feb 2010 09:17:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2010 09:17:26 -0000
Received: (qmail 74743 invoked by uid 500); 18 Feb 2010 09:17:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74673 invoked by uid 500); 18 Feb 2010 09:17:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74663 invoked by uid 99); 18 Feb 2010 09:17:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 09:17:25 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 18 Feb 2010 09:17:23 +0000
Received: (qmail 70166 invoked by uid 99); 18 Feb 2010 09:17:01 -0000
Received: from localhost.apache.org (HELO fg-out-1718.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 09:17:01 +0000
Received: by fg-out-1718.google.com with SMTP id e21so729515fga.11
        for <general@hadoop.apache.org>; Thu, 18 Feb 2010 01:16:59 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.14.84 with SMTP id f20mr451398bka.209.1266484619130; Thu, 
	18 Feb 2010 01:16:59 -0800 (PST)
In-Reply-To: <1267dd3b1002180105y5339e19elf9712ed719a6500e@mail.gmail.com>
References: <BE5B5A4B-CDBF-473F-B760-DBC1E1ED33C2@apache.org>
	 <1267dd3b1002180105y5339e19elf9712ed719a6500e@mail.gmail.com>
Date: Thu, 18 Feb 2010 01:16:59 -0800
Message-ID: <1267dd3b1002180116n32c7d0d6ucc5b9342dfc9da1b@mail.gmail.com>
Subject: Re: [VOTE] Should we release Hadoop 0.20.2 rc2?
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Filed HADOOP-6575.

On the bright side, MAPREDUCE-623 can also be part of this release. -C

On Thu, Feb 18, 2010 at 1:05 AM, Chris Douglas <cdouglas@apache.org> wrote:
> *sigh* The tests currently do not compile. HADOOP-6506 added a call in
> the build:
>
> + =A0 =A0<antcall target=3D"run-test-core-fault-inject"/>
>
> which does not exist in 0.20. -C
>
> On Wed, Feb 17, 2010 at 5:24 PM, Owen O'Malley <omalley@apache.org> wrote=
:
>> Ok, I've rolled a new tarball with Todd's suggested updates to the branc=
h.
>> Please try it out and vote.
>>
>> http://people.apache.org/~omalley/0.20.2/
>>
>> -- Owen
>>
>

From general-return-1040-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 18 18:55:24 2010
Return-Path: <general-return-1040-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 62480 invoked from network); 18 Feb 2010 18:55:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2010 18:55:24 -0000
Received: (qmail 34213 invoked by uid 500); 18 Feb 2010 18:55:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34136 invoked by uid 500); 18 Feb 2010 18:55:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34126 invoked by uid 99); 18 Feb 2010 18:55:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 18:55:22 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 74.125.92.25 as permitted sender)
Received: from [74.125.92.25] (HELO qw-out-2122.google.com) (74.125.92.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 18:55:14 +0000
Received: by qw-out-2122.google.com with SMTP id 5so47117qwi.35
        for <general@hadoop.apache.org>; Thu, 18 Feb 2010 10:54:53 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:message-id:subject:from:to
         :content-type;
        bh=8p7y8MzY0YXx1YMBzhL4ZX0s3qQgxOAl/JiLxafkTvQ=;
        b=l9PpKDuNyCGebvzKwIl3VTjbmxoIpTuGmYE39e7uAzWQxGL3YQtb8K9qFLq2RWeA+5
         5EG1e92pr7hlqzO3Zcn3DI6ZPKxcLpPx4v7xD6uW+OxlrlF1HD50NisKI+sjFL038dJf
         und001pmqgv17Q30XK51usBWtQh4Z2tMfOC8Q=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=TNC4mGoKR5Y4429w3tpW6VwoXUM6NRBniWhmPlCboxYeFZrnsoyQnXQr9X1LKNs3R+
         0a2xSEq1AqYLY8vMum8sR5kWxn9t/+9uhTAewMSxx1ajDuLIHU/1a4s7chH1ZC3OLrMl
         qXqnj8JOU4a7tH6s7ykl6xUC1BiCPDjziTUZU=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
Received: by 10.229.224.133 with SMTP id io5mr2512668qcb.37.1266519292660; 
	Thu, 18 Feb 2010 10:54:52 -0800 (PST)
In-Reply-To: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com>
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com>
Date: Thu, 18 Feb 2010 10:54:52 -0800
X-Google-Sender-Auth: 90f555ea047c51b9
Message-ID: <7c962aed1002181054w3cdb9302y93f607f3b53f0d18@mail.gmail.com>
Subject: Re: Release plans
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Tue, Feb 16, 2010 at 2:12 PM, Eli Collins <eli@cloudera.com> wrote:
> Hey guys,
>
> What are the current plans for the 21 release?
>
Doesn't seem like there is much interest going by the deafening silence Eli.

Did it come up last night at the HUG?

> The roadmap on the wiki [1] says "minor releases are made regularly,
> every few months." The 21 branch was created 5 months ago, and the
> release dates in jira for the various core projects have all passed. I
> know some projects, eg HBase, would like to get users on more recent
> bits. What are the next steps for rolling a release?
>

Taking a look at blockers -- assuming to make a release, we just need
to clear blockers and roll a candidate -- it seems like common/core
and hdfs have but a few but mapreduce has a bunch.  Too many.

Does anyone detect activity in the blocker issues?  I've not made a
review.  Perhaps someone already has?

St.Ack

From general-return-1041-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 18 19:29:00 2010
Return-Path: <general-return-1041-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 80160 invoked from network); 18 Feb 2010 19:29:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2010 19:29:00 -0000
Received: (qmail 79087 invoked by uid 500); 18 Feb 2010 19:28:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79009 invoked by uid 500); 18 Feb 2010 19:28:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78999 invoked by uid 99); 18 Feb 2010 19:28:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 19:28:59 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.85.222.195] (HELO mail-pz0-f195.google.com) (209.85.222.195)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 19:28:52 +0000
Received: by pzk33 with SMTP id 33so4913085pzk.2
        for <general@hadoop.apache.org>; Thu, 18 Feb 2010 11:28:32 -0800 (PST)
MIME-Version: 1.0
Received: by 10.142.62.34 with SMTP id k34mr6672885wfa.282.1266521311896; Thu, 
	18 Feb 2010 11:28:31 -0800 (PST)
In-Reply-To: <7c962aed1002181054w3cdb9302y93f607f3b53f0d18@mail.gmail.com>
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com>
	 <7c962aed1002181054w3cdb9302y93f607f3b53f0d18@mail.gmail.com>
Date: Thu, 18 Feb 2010 11:28:31 -0800
Message-ID: <dfe484f01002181128v4b788bd6o9c394f31ae9121c5@mail.gmail.com>
Subject: Re: Release plans
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Thu, Feb 18, 2010 at 10:54 AM, Stack <stack@duboce.net> wrote:
> On Tue, Feb 16, 2010 at 2:12 PM, Eli Collins <eli@cloudera.com> wrote:
>> Hey guys,
>>
>> What are the current plans for the 21 release?
>>
> Doesn't seem like there is much interest going by the deafening silence E=
li.
>
> Did it come up last night at the HUG?
>

It did, Owen gave an update in his talk and there was some follow on
discussion. The question of whether it makes sense to rebase 21 on
trunk came up, since 21 has gotten really out of date since it was
branched. What do you and others think of rebasing the 21 branch on
trunk once security is feature complete next month, and then targeting
an initial release a month after that? The rationale being that 21
branch is old, has not been getting the backports it needs, and the
community should be releasing relatively recent bits. In the mean time
we can focus on 21/trunk blockers. It doesn't look like there are a
ton more trunk blockers than 21 blockers. I know people, HBase in
particular, have been waiting for a 21 release for a while so delaying
further is a pain, and we should consider rolling what we've got, the
hope would be that the 21 branch would be more widely useful if it has
more recent features and fixes.

>> The roadmap on the wiki [1] says "minor releases are made regularly,
>> every few months." The 21 branch was created 5 months ago, and the
>> release dates in jira for the various core projects have all passed. I
>> know some projects, eg HBase, would like to get users on more recent
>> bits. What are the next steps for rolling a release?
>>
>
> Taking a look at blockers -- assuming to make a release, we just need
> to clear blockers and roll a candidate -- it seems like common/core
> and hdfs have but a few but mapreduce has a bunch. =A0Too many.
>
> Does anyone detect activity in the blocker issues? =A0I've not made a
> review. =A0Perhaps someone already has?

A decent number of the MR ones are patch available and quite a few
others are documentation related, nothing too scary. The core/hdfs
ones are pretty doable. We should make another pass to confirm that
they're all actually blockers. I'm happy to start working on the
blockers if that will help push the release forward.

Thanks,
Eli

From general-return-1042-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 18 19:29:31 2010
Return-Path: <general-return-1042-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 80275 invoked from network); 18 Feb 2010 19:29:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2010 19:29:31 -0000
Received: (qmail 80021 invoked by uid 500); 18 Feb 2010 19:29:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79948 invoked by uid 500); 18 Feb 2010 19:29:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 79938 invoked by uid 99); 18 Feb 2010 19:29:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 19:29:30 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 18 Feb 2010 19:29:29 +0000
Received: (qmail 80242 invoked by uid 99); 18 Feb 2010 19:29:08 -0000
Received: from localhost.apache.org (HELO [192.168.168.107]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 19:29:08 +0000
Message-ID: <4B7D9503.4080205@apache.org>
Date: Thu, 18 Feb 2010 11:29:07 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Release plans
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com>
In-Reply-To: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Eli Collins wrote:
> What are the current plans for the 21 release?

Personally, I'd rather re-branch 21 from trunk in early March.  I 
believe Y!'s security changes will be feature-complete in trunk by then. 
  Symlinks and end-to-end Avro should also hopefully be committed by then.

I'd then propose we roll a 0.21.0 (alpha) release one month later, in 
early April.

Henceforth, I'd like to see the project:
  - set branch dates at 6-month intervals
  - roll alpha releases one month after branch dates
  - roll bugfix releases as needed thereafter

At the onset of each six-month period we'd also need to decide what sort 
of release we'll make, major or minor.  (Minor releases allow new 
features, but remove no deprecated APIs.  Only major releases are 
permitted to remove previously deprecated APIs.  All pre-1.0 releases 
are major.)

We'd also designate a release master for each release, to drive the 
process: making the branches, rolling the artifacts, and calling the 
votes, etc.  Do we have any release master candidate volunteers for 0.21?

Doug

From general-return-1043-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 18 21:19:05 2010
Return-Path: <general-return-1043-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 32717 invoked from network); 18 Feb 2010 21:19:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2010 21:19:05 -0000
Received: (qmail 72612 invoked by uid 500); 18 Feb 2010 21:19:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72524 invoked by uid 500); 18 Feb 2010 21:19:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72514 invoked by uid 99); 18 Feb 2010 21:19:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 21:19:03 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 21:18:52 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:User-Agent:Date:Subject:From:To:
   Message-ID:Thread-Topic:Thread-Index:In-Reply-To:
   Mime-version:Content-type:Content-transfer-encoding;
  b=DHwtvFb+imPRbdyYikUmjPOZWUd6VEgK7SbW8TAgN1fkNOx33ikZWVC7
   +ZYqfH+NEKlcFK3gf6fDgkmAQTHVDSAEWBdu7u4GFkb3ntbDAAXi/Ekg1
   EQtwuTpL8QLZHkj;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1266527932; x=1298063932;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20Release=20plans|Date:=20Thu,=2018=20Feb
   =202010=2013:17:50=20-0800|Message-ID:=20<C7A2EE7E.317E%a
   wittenauer@linkedin.com>|To:=20<general@hadoop.apache.org
   >|Mime-version:=201.0|Content-transfer-encoding:=207bit
   |In-Reply-To:=20<4B7D9503.4080205@apache.org>;
  bh=O3INq+T8onxNfkx6hZ4VLBGrKRjgof+n5I6Zsqk0PaY=;
  b=qNInKnQLSqJmdnyxEZjtumMWNDavSjxew8ppcujkJHqvbvVCbPiiQHYt
   wlAEVNfCu1qqzCodxqdLt01/HA2cn7JS1B/hm6+40Oxup3FN3TXfhpcyz
   UyEjwXIR/JXiuaS;
X-IronPort-AV: E=Sophos;i="4.49,499,1262592000"; 
   d="scan'208";a="11280471"
Received: from 172.16.19.141 ([172.16.19.141]) by CORP-MAIL.linkedin.biz ([172.18.46.135]) via Exchange Front-End Server mail-access.linkedin.biz ([172.18.46.133]) with Microsoft Exchange Server HTTP-DAV ;
 Thu, 18 Feb 2010 21:17:53 +0000
User-Agent: Microsoft-Entourage/12.10.0.080409
Date: Thu, 18 Feb 2010 13:17:50 -0800
Subject: Re: Release plans
From: Allen Wittenauer <awittenauer@linkedin.com>
To: <general@hadoop.apache.org>
Message-ID: <C7A2EE7E.317E%awittenauer@linkedin.com>
Thread-Topic: Release plans
Thread-Index: Acqw39MjJ7pdek8Y0EqS482uI4MRsg==
In-Reply-To: <4B7D9503.4080205@apache.org>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org




On 2/18/10 11:29 AM, "Doug Cutting" <cutting@apache.org> wrote:

> Eli Collins wrote:
>> What are the current plans for the 21 release?
> 
> Personally, I'd rather re-branch 21 from trunk in early March.  I
> believe Y!'s security changes will be feature-complete in trunk by then.
>   Symlinks and end-to-end Avro should also hopefully be committed by then.
> 
> I'd then propose we roll a 0.21.0 (alpha) release one month later, in
> early April.

+1


From general-return-1044-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 18 21:28:50 2010
Return-Path: <general-return-1044-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 36576 invoked from network); 18 Feb 2010 21:28:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2010 21:28:50 -0000
Received: (qmail 83987 invoked by uid 500); 18 Feb 2010 21:28:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83919 invoked by uid 500); 18 Feb 2010 21:28:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83909 invoked by uid 99); 18 Feb 2010 21:28:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 21:28:49 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 209.85.211.199 as permitted sender)
Received: from [209.85.211.199] (HELO mail-yw0-f199.google.com) (209.85.211.199)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 21:28:39 +0000
Received: by ywh37 with SMTP id 37so1565788ywh.2
        for <general@hadoop.apache.org>; Thu, 18 Feb 2010 13:28:19 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=JlcNVoALLI1p67VP5vjJo0sY4+X2n+OWKbJJqDXuqQU=;
        b=rw/4FPU9bVnDBnvzUhl2rmPl/NCzM+7/BgD7z3FMBFT3EQAP1vk8dHAp2FN+r7y1FG
         pLOAIQwPqB+n4q5j/u9h34b0n1GLvehazAPrHydBwuWvfRKJdMs8GxrUwERZaMqZtXx2
         S0KScsNY+XgE1K8PLsKeBSvcySoXGvXIQ7xl0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=ketPgAkjFw6pMk+TaUatzeImv0F7o98TrD1HVmug2ZfIvHvfFfFe9XfLS3bYf1XFzX
         0UKoVdwOqjRTjZX6Jc7peBpIlaIuj5lFyXCzHRmE6l+pljX9OgcH4C8WiqmcqFkgciGA
         wbs9g3iPahJEOpZhlLhIfXw29y25ysySfFGts=
MIME-Version: 1.0
Received: by 10.150.241.41 with SMTP id o41mr14644910ybh.68.1266528498601; 
	Thu, 18 Feb 2010 13:28:18 -0800 (PST)
In-Reply-To: <4B7D9503.4080205@apache.org>
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com>
	 <4B7D9503.4080205@apache.org>
Date: Thu, 18 Feb 2010 13:28:18 -0800
Message-ID: <4aa34eb71002181328l5e40037eo891b1cda4288cd9c@mail.gmail.com>
Subject: Re: Release plans
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd51bca41cdf5047fe6a52c

--000e0cd51bca41cdf5047fe6a52c
Content-Type: text/plain; charset=ISO-8859-1

I have seen/heard about people using the 0.21 branch. Won't it disturbing to
them that a new 0.21 branch that they might sync to in the future will have
a whole bunch of new code/features that they were not testing with earlier?

I thought that there is some benefit in rolling a 0.22 sometime in the
future and then marking 0.21 as "not for production usage".

thanks,
dhruba

On Thu, Feb 18, 2010 at 11:29 AM, Doug Cutting <cutting@apache.org> wrote:

> Eli Collins wrote:
>
>> What are the current plans for the 21 release?
>>
>
> Personally, I'd rather re-branch 21 from trunk in early March.  I believe
> Y!'s security changes will be feature-complete in trunk by then.  Symlinks
> and end-to-end Avro should also hopefully be committed by then.
>
> I'd then propose we roll a 0.21.0 (alpha) release one month later, in early
> April.
>
> Henceforth, I'd like to see the project:
>  - set branch dates at 6-month intervals
>  - roll alpha releases one month after branch dates
>  - roll bugfix releases as needed thereafter
>
> At the onset of each six-month period we'd also need to decide what sort of
> release we'll make, major or minor.  (Minor releases allow new features, but
> remove no deprecated APIs.  Only major releases are permitted to remove
> previously deprecated APIs.  All pre-1.0 releases are major.)
>
> We'd also designate a release master for each release, to drive the
> process: making the branches, rolling the artifacts, and calling the votes,
> etc.  Do we have any release master candidate volunteers for 0.21?
>
> Doug
>



-- 
Connect to me at http://www.facebook.com/dhruba

--000e0cd51bca41cdf5047fe6a52c--

From general-return-1045-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 18 21:55:40 2010
Return-Path: <general-return-1045-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53360 invoked from network); 18 Feb 2010 21:55:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2010 21:55:40 -0000
Received: (qmail 36080 invoked by uid 500); 18 Feb 2010 21:55:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36007 invoked by uid 500); 18 Feb 2010 21:55:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 35997 invoked by uid 99); 18 Feb 2010 21:55:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 21:55:39 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.30.80] (HELO qmta08.emeryville.ca.mail.comcast.net) (76.96.30.80)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 21:55:30 +0000
Received: from omta14.emeryville.ca.mail.comcast.net ([76.96.30.60])
	by qmta08.emeryville.ca.mail.comcast.net with comcast
	id jTHZ1d0011HpZEsA8ZvAG2; Thu, 18 Feb 2010 21:55:10 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta14.emeryville.ca.mail.comcast.net with comcast
	id jZv21d0032SbwD58aZv4L8; Thu, 18 Feb 2010 21:55:08 +0000
Message-Id: <DA9E7EF1-9A82-414E-AA5C-DD1088869CD2@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <4B7D9503.4080205@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Release plans
Date: Thu, 18 Feb 2010 13:55:00 -0800
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com> <4B7D9503.4080205@apache.org>
X-Mailer: Apple Mail (2.936)


On Feb 18, 2010, at 11:29 AM, Doug Cutting wrote:

My presentation basically said that:
   * Yahoo is currently running Yahoo's 0.20.8 on all 25,000 of our  
nodes.
   * We have made substantial numbers of feature back ports from trunk  
into our branch. Including:
      * improved Capacity scheduler
      * run MapReduce tasks as the real user
   * Hadoop 0.21 isn't stable yet and still has 28 blockers.
   * Furthermore, there are missing back ports that have been fixed in  
trunk but not 0.21.
   * We have a very tight timeline for adding security, which we can't  
slip
      * feature complete in february
      * deploy to first integration cluster in april
      * completely deployed on all clusters in august
   * To reduce risk, we are back porting security in to a branch of  
Yahoo's 0.20 branch.
   * By the time we are ready for the next version after security, the  
current 0.21 will be too stale to be interesting
   * So Yahoo will probably skip straight from Yahoo 0.20 with  
security to 0.22 in 2011.
   * Someone outside of Yahoo needs to step up and start addressing  
the blockers to get 0.21 released.

> Personally, I'd rather re-branch 21 from trunk in early March.

Rather than set a date, I think it is time to move to feature-based  
releases. We'd need to vote on the feature set, but looking for  
security, end-to-end avro, and symlinks seems like a reasonable list.  
That will avoid the large rush of commits the last week of the  
deadline, which has been counter productive.

-- Owen

From general-return-1046-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 18 23:06:34 2010
Return-Path: <general-return-1046-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 92960 invoked from network); 18 Feb 2010 23:06:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2010 23:06:34 -0000
Received: (qmail 29027 invoked by uid 500); 18 Feb 2010 23:06:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28959 invoked by uid 500); 18 Feb 2010 23:06:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28949 invoked by uid 99); 18 Feb 2010 23:06:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 23:06:32 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.92.27] (HELO qw-out-2122.google.com) (74.125.92.27)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 23:06:26 +0000
Received: by qw-out-2122.google.com with SMTP id 5so110313qwi.35
        for <general@hadoop.apache.org>; Thu, 18 Feb 2010 15:06:04 -0800 (PST)
MIME-Version: 1.0
Received: by 10.229.112.2 with SMTP id u2mr5136828qcp.0.1266534364701; Thu, 18 
	Feb 2010 15:06:04 -0800 (PST)
In-Reply-To: <DA9E7EF1-9A82-414E-AA5C-DD1088869CD2@apache.org>
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com>
	 <4B7D9503.4080205@apache.org>
	 <DA9E7EF1-9A82-414E-AA5C-DD1088869CD2@apache.org>
Date: Thu, 18 Feb 2010 15:06:00 -0800
Message-ID: <c9118d4c1002181506l7ec9840dq3690158ebc70c02@mail.gmail.com>
Subject: Re: Release plans
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0023544714c0e77794047fe8025d

--0023544714c0e77794047fe8025d
Content-Type: text/plain; charset=ISO-8859-1

>
>  * Someone outside of Yahoo needs to step up and start addressing the
> blockers to get 0.21 released.
>

Thanks for the update on the state of the world at Yahoo!, Owen--very
helpful. I think the community will step up to knock down some of the
blockers once we resolve what should be in the 0.21 release: the current
branch, or a rebase on trunk. Do you/Yahoo! have a preference on that front?

Rather than set a date, I think it is time to move to feature-based
> releases. We'd need to vote on the feature set, but looking for security,
> end-to-end avro, and symlinks seems like a reasonable list. That will avoid
> the large rush of commits the last week of the deadline, which has been
> counter productive.
>

Could someone give an example of a successful open source project that
follows a feature-based release cycle? From the research I've done, the
regular drumbeat of time-based releases seem to be more conducive to project
health. Always interested to hear otherwise. The example Owen cites of "the
large rush of commits the last week of the deadline" is certainly a good
argument in favor of feature-based releases; I'm curious to hear more.

Thanks,
Jeff

--0023544714c0e77794047fe8025d--

From general-return-1047-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 18 23:19:07 2010
Return-Path: <general-return-1047-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96640 invoked from network); 18 Feb 2010 23:19:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2010 23:19:07 -0000
Received: (qmail 59568 invoked by uid 500); 18 Feb 2010 23:19:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59512 invoked by uid 500); 18 Feb 2010 23:19:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59502 invoked by uid 99); 18 Feb 2010 23:19:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 23:19:06 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 23:18:57 +0000
Received: from oceanfarearth-lm.corp.yahoo.com (oceanfarearth-lm.corp.yahoo.com [10.72.113.156])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o1INI5pn009485
	for <general@hadoop.apache.org>; Thu, 18 Feb 2010 15:18:06 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=DCm+MW1pr3aU4I66DBMcgj6vrXOJrEnnuIbrmkMmQP4EI4f5UT9F3XvKQd3n13W1
Message-Id: <82FD77AF-E1E2-49E2-8641-35C8697B8973@yahoo-inc.com>
From: Sanjay Radia <sradia@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <dfe484f01002181128v4b788bd6o9c394f31ae9121c5@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Release plans
Date: Thu, 18 Feb 2010 15:18:05 -0800
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com> <7c962aed1002181054w3cdb9302y93f607f3b53f0d18@mail.gmail.com> <dfe484f01002181128v4b788bd6o9c394f31ae9121c5@mail.gmail.com>
X-Mailer: Apple Mail (2.936)


On Feb 18, 2010, at 11:28 AM, Eli Collins wrote:

> On Thu, Feb 18, 2010 at 10:54 AM, Stack <stack@duboce.net> wrote:
>> On Tue, Feb 16, 2010 at 2:12 PM, Eli Collins <eli@cloudera.com>  
>> wrote:
>>> Hey guys,
>>>
>>> What are the current plans for the 21 release?
>>>
>> Doesn't seem like there is much interest going by the deafening  
>> silence Eli.
>>
>> Did it come up last night at the HUG?
>>
>
> It did, Owen gave an update in his talk and there was some follow on
> discussion. The question of whether it makes sense to rebase 21 on
> trunk came up, since 21 has gotten really out of date since it was
> branched. What do you and others think of rebasing the 21 branch on
> trunk once security is feature complete next month, and then targeting
> an initial release a month after that? The rationale being that 21
> branch is old, has not been getting the backports it needs, and the
> community should be releasing relatively recent bits. In the mean time
> we can focus on 21/trunk blockers. It doesn't look like there are a
> ton more trunk blockers than 21 blockers. I know people, HBase in
> particular, have been waiting for a 21 release for a while so delaying
> further is a pain, and we should consider rolling what we've got, the
> hope would be that the 21 branch would be more widely useful if it has
> more recent features and fixes.



If the community want the append features fast then we  are better off  
using the original 21 branch.
Basing a 21 off trunk will mean that there are more features with a  
new set of blockers and  and hence will take longer to fix.


sanjay

From general-return-1048-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 18 23:21:03 2010
Return-Path: <general-return-1048-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97246 invoked from network); 18 Feb 2010 23:21:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Feb 2010 23:21:03 -0000
Received: (qmail 61990 invoked by uid 500); 18 Feb 2010 23:21:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61920 invoked by uid 500); 18 Feb 2010 23:21:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61910 invoked by uid 99); 18 Feb 2010 23:21:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 23:21:02 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 18 Feb 2010 23:20:57 +0000
Received: (qmail 97194 invoked by uid 99); 18 Feb 2010 23:20:37 -0000
Received: from localhost.apache.org (HELO [192.168.168.107]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Feb 2010 23:20:37 +0000
Message-ID: <4B7DCB44.20506@apache.org>
Date: Thu, 18 Feb 2010 15:20:36 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Release plans
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com> <4B7D9503.4080205@apache.org> <DA9E7EF1-9A82-414E-AA5C-DD1088869CD2@apache.org>
In-Reply-To: <DA9E7EF1-9A82-414E-AA5C-DD1088869CD2@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Owen O'Malley wrote:
> Rather than set a date, I think it is time to move to feature-based 
> releases.

The problem with feature-based releases is that one feature can delay 
releases arbitrarily, so a single feature can hold all others hostage. 
Rather I think we need to be willing to remove or disable a feature in a 
branch if it proves problematic.  If an addition causes regressions that 
are not resolved promptly then that addition might be removed or 
disabled.  This levels the playing field: features that doesn't cause 
regressions can be made available in a release in a timely, predictable 
manner and are not held back by more problematic features.

Doug

From general-return-1049-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 19 01:02:46 2010
Return-Path: <general-return-1049-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34902 invoked from network); 19 Feb 2010 01:02:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Feb 2010 01:02:45 -0000
Received: (qmail 66726 invoked by uid 500); 19 Feb 2010 01:02:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66650 invoked by uid 500); 19 Feb 2010 01:02:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66640 invoked by uid 99); 19 Feb 2010 01:02:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 01:02:44 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.62.16] (HELO qmta01.westchester.pa.mail.comcast.net) (76.96.62.16)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 01:02:36 +0000
Received: from omta09.westchester.pa.mail.comcast.net ([76.96.62.20])
	by qmta01.westchester.pa.mail.comcast.net with comcast
	id jQEo1d0020SCNGk51d2Gdz; Fri, 19 Feb 2010 01:02:16 +0000
Received: from [172.21.155.74] ([209.131.62.146])
	by omta09.westchester.pa.mail.comcast.net with comcast
	id jd261d00G39K7Mt3Vd293t; Fri, 19 Feb 2010 01:02:14 +0000
Message-Id: <EFFFAD8D-1998-4C19-9BD3-D80B3B79577A@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <c9118d4c1002181506l7ec9840dq3690158ebc70c02@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Release plans
Date: Thu, 18 Feb 2010 17:02:04 -0800
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com> <4B7D9503.4080205@apache.org> <DA9E7EF1-9A82-414E-AA5C-DD1088869CD2@apache.org> <c9118d4c1002181506l7ec9840dq3690158ebc70c02@mail.gmail.com>
X-Mailer: Apple Mail (2.936)

On Feb 18, 2010, at 3:06 PM, Jeff Hammerbacher wrote:

> I think the community will step up to knock down some of the
> blockers once we resolve what should be in the 0.21 release: the  
> current
> branch, or a rebase on trunk. Do you/Yahoo! have a preference on  
> that front?

Yahoo doesn't care. Even if we rebase the 0.21 branch, because of the  
timing, Yahoo will probably never deploy it. (It take months to push a  
release through QA and production testing, as I wrote the security  
release will hit the pipeline this year (code complete in february,  
first integration cluster in april, on all production clusters by  
august). Yahoo can't handle another big release until january 2011.

Personally, I'd prefer to rebase 0.21, especially after we have the  
Maven story straightened out. Generating good poms would be a huge win  
for downstream projects.

One big concern is that backwards incompatibility is a big cost.  
Especially if 0.21 (like 0.19) never gets wide deployment, I'd like to  
start a vote that we don't make any API incompatible in 0.22 relative  
to 0.20.

-- Owen

From general-return-1050-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 19 01:20:01 2010
Return-Path: <general-return-1050-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 41505 invoked from network); 19 Feb 2010 01:20:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Feb 2010 01:20:00 -0000
Received: (qmail 77284 invoked by uid 500); 19 Feb 2010 01:20:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77188 invoked by uid 500); 19 Feb 2010 01:19:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77178 invoked by uid 99); 19 Feb 2010 01:19:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 01:19:59 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.221.184] (HELO mail-qy0-f184.google.com) (209.85.221.184)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 01:19:51 +0000
Received: by qyk14 with SMTP id 14so912733qyk.9
        for <general@hadoop.apache.org>; Thu, 18 Feb 2010 17:19:30 -0800 (PST)
MIME-Version: 1.0
Received: by 10.229.224.17 with SMTP id im17mr2708322qcb.41.1266542369936; 
	Thu, 18 Feb 2010 17:19:29 -0800 (PST)
In-Reply-To: <EFFFAD8D-1998-4C19-9BD3-D80B3B79577A@apache.org>
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com>
	 <4B7D9503.4080205@apache.org>
	 <DA9E7EF1-9A82-414E-AA5C-DD1088869CD2@apache.org>
	 <c9118d4c1002181506l7ec9840dq3690158ebc70c02@mail.gmail.com>
	 <EFFFAD8D-1998-4C19-9BD3-D80B3B79577A@apache.org>
Date: Thu, 18 Feb 2010 17:19:29 -0800
Message-ID: <c9118d4c1002181719v52a11b0cu55159f344558a78e@mail.gmail.com>
Subject: Re: Release plans
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364ed7840d961c047fe9e015
X-Virus-Checked: Checked by ClamAV on apache.org

--0016364ed7840d961c047fe9e015
Content-Type: text/plain; charset=ISO-8859-1

Thanks Owen, that's useful information. It sounds like the API
incompatibility vote can be a separate issue.

Do we have consensus around rebasing on 0.21? Anyone already testing on 0.21
who would be upset if the current branch were to be retired?

On Thu, Feb 18, 2010 at 5:02 PM, Owen O'Malley <omalley@apache.org> wrote:

> On Feb 18, 2010, at 3:06 PM, Jeff Hammerbacher wrote:
>
>  I think the community will step up to knock down some of the
>> blockers once we resolve what should be in the 0.21 release: the current
>> branch, or a rebase on trunk. Do you/Yahoo! have a preference on that
>> front?
>>
>
> Yahoo doesn't care. Even if we rebase the 0.21 branch, because of the
> timing, Yahoo will probably never deploy it. (It take months to push a
> release through QA and production testing, as I wrote the security release
> will hit the pipeline this year (code complete in february, first
> integration cluster in april, on all production clusters by august). Yahoo
> can't handle another big release until january 2011.
>
> Personally, I'd prefer to rebase 0.21, especially after we have the Maven
> story straightened out. Generating good poms would be a huge win for
> downstream projects.
>
> One big concern is that backwards incompatibility is a big cost. Especially
> if 0.21 (like 0.19) never gets wide deployment, I'd like to start a vote
> that we don't make any API incompatible in 0.22 relative to 0.20.
>
> -- Owen
>

--0016364ed7840d961c047fe9e015--

From general-return-1051-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 19 01:27:00 2010
Return-Path: <general-return-1051-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43342 invoked from network); 19 Feb 2010 01:26:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Feb 2010 01:26:59 -0000
Received: (qmail 82722 invoked by uid 500); 19 Feb 2010 01:26:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82647 invoked by uid 500); 19 Feb 2010 01:26:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82637 invoked by uid 99); 19 Feb 2010 01:26:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 01:26:58 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.85.222.195] (HELO mail-pz0-f195.google.com) (209.85.222.195)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 01:26:51 +0000
Received: by pzk33 with SMTP id 33so5277907pzk.2
        for <general@hadoop.apache.org>; Thu, 18 Feb 2010 17:26:31 -0800 (PST)
MIME-Version: 1.0
Received: by 10.141.4.4 with SMTP id g4mr6931891rvi.275.1266542791193; Thu, 18 
	Feb 2010 17:26:31 -0800 (PST)
In-Reply-To: <c9118d4c1002181719v52a11b0cu55159f344558a78e@mail.gmail.com>
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com> 
	<4B7D9503.4080205@apache.org> <DA9E7EF1-9A82-414E-AA5C-DD1088869CD2@apache.org> 
	<c9118d4c1002181506l7ec9840dq3690158ebc70c02@mail.gmail.com> 
	<EFFFAD8D-1998-4C19-9BD3-D80B3B79577A@apache.org> <c9118d4c1002181719v52a11b0cu55159f344558a78e@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Thu, 18 Feb 2010 17:26:11 -0800
Message-ID: <45f85f71002181726m5e29dd65o14e6a6bb8bd4d0a1@mail.gmail.com>
Subject: Re: Release plans
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Thu, Feb 18, 2010 at 5:19 PM, Jeff Hammerbacher <hammer@cloudera.com> wr=
ote:
> Thanks Owen, that's useful information. It sounds like the API
> incompatibility vote can be a separate issue.
>
> Do we have consensus around rebasing on 0.21? Anyone already testing on 0=
.21
> who would be upset if the current branch were to be retired?
>

One question I have is what to do with the JIRA tickets - we probably
need some kind of bulk edit, right? Anything that's currently resolved
with fix version 22 needs to be changed over to fix version 21?

> On Thu, Feb 18, 2010 at 5:02 PM, Owen O'Malley <omalley@apache.org> wrote=
:
>
>> On Feb 18, 2010, at 3:06 PM, Jeff Hammerbacher wrote:
>>
>> =A0I think the community will step up to knock down some of the
>>> blockers once we resolve what should be in the 0.21 release: the curren=
t
>>> branch, or a rebase on trunk. Do you/Yahoo! have a preference on that
>>> front?
>>>
>>
>> Yahoo doesn't care. Even if we rebase the 0.21 branch, because of the
>> timing, Yahoo will probably never deploy it. (It take months to push a
>> release through QA and production testing, as I wrote the security relea=
se
>> will hit the pipeline this year (code complete in february, first
>> integration cluster in april, on all production clusters by august). Yah=
oo
>> can't handle another big release until january 2011.
>>
>> Personally, I'd prefer to rebase 0.21, especially after we have the Mave=
n
>> story straightened out. Generating good poms would be a huge win for
>> downstream projects.
>>
>> One big concern is that backwards incompatibility is a big cost. Especia=
lly
>> if 0.21 (like 0.19) never gets wide deployment, I'd like to start a vote
>> that we don't make any API incompatible in 0.22 relative to 0.20.
>>
>> -- Owen
>>
>

From general-return-1052-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 19 01:30:39 2010
Return-Path: <general-return-1052-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44604 invoked from network); 19 Feb 2010 01:30:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Feb 2010 01:30:39 -0000
Received: (qmail 85372 invoked by uid 500); 19 Feb 2010 01:30:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85288 invoked by uid 500); 19 Feb 2010 01:30:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85278 invoked by uid 99); 19 Feb 2010 01:30:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 01:30:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 19 Feb 2010 01:30:36 +0000
Received: (qmail 44481 invoked by uid 99); 19 Feb 2010 01:30:14 -0000
Received: from localhost.apache.org (HELO mail-bw0-f211.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 01:30:14 +0000
Received: by bwz3 with SMTP id 3so858265bwz.29
        for <general@hadoop.apache.org>; Thu, 18 Feb 2010 17:30:12 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.33.137 with SMTP id h9mr620004bkd.134.1266543012408; Thu, 
	18 Feb 2010 17:30:12 -0800 (PST)
In-Reply-To: <1267dd3b1002180116n32c7d0d6ucc5b9342dfc9da1b@mail.gmail.com>
References: <BE5B5A4B-CDBF-473F-B760-DBC1E1ED33C2@apache.org>
	 <1267dd3b1002180105y5339e19elf9712ed719a6500e@mail.gmail.com>
	 <1267dd3b1002180116n32c7d0d6ucc5b9342dfc9da1b@mail.gmail.com>
Date: Thu, 18 Feb 2010 17:30:12 -0800
Message-ID: <1267dd3b1002181730h613d7565j15b7e495021871fd@mail.gmail.com>
Subject: Re: [VOTE] Should we release Hadoop 0.20.2 rc2?
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

I rolled a new release candidate, but TestStreamingStatus is failing
consistently after HADOOP-5623. The test passes when the changes are
reverted. -C

On Thu, Feb 18, 2010 at 1:16 AM, Chris Douglas <cdouglas@apache.org> wrote:
> Filed HADOOP-6575.
>
> On the bright side, MAPREDUCE-623 can also be part of this release. -C
>
> On Thu, Feb 18, 2010 at 1:05 AM, Chris Douglas <cdouglas@apache.org> wrot=
e:
>> *sigh* The tests currently do not compile. HADOOP-6506 added a call in
>> the build:
>>
>> + =A0 =A0<antcall target=3D"run-test-core-fault-inject"/>
>>
>> which does not exist in 0.20. -C
>>
>> On Wed, Feb 17, 2010 at 5:24 PM, Owen O'Malley <omalley@apache.org> wrot=
e:
>>> Ok, I've rolled a new tarball with Todd's suggested updates to the bran=
ch.
>>> Please try it out and vote.
>>>
>>> http://people.apache.org/~omalley/0.20.2/
>>>
>>> -- Owen
>>>
>>
>

From general-return-1053-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 19 01:31:15 2010
Return-Path: <general-return-1053-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44865 invoked from network); 19 Feb 2010 01:31:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Feb 2010 01:31:15 -0000
Received: (qmail 86320 invoked by uid 500); 19 Feb 2010 01:31:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86237 invoked by uid 500); 19 Feb 2010 01:31:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86227 invoked by uid 99); 19 Feb 2010 01:31:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 01:31:14 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 19 Feb 2010 01:31:11 +0000
Received: (qmail 44717 invoked by uid 99); 19 Feb 2010 01:30:50 -0000
Received: from localhost.apache.org (HELO [192.168.168.107]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 01:30:50 +0000
Message-ID: <4B7DE9C9.5090707@apache.org>
Date: Thu, 18 Feb 2010 17:30:49 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Release plans
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com> 	<4B7D9503.4080205@apache.org> <DA9E7EF1-9A82-414E-AA5C-DD1088869CD2@apache.org> 	<c9118d4c1002181506l7ec9840dq3690158ebc70c02@mail.gmail.com> 	<EFFFAD8D-1998-4C19-9BD3-D80B3B79577A@apache.org> <c9118d4c1002181719v52a11b0cu55159f344558a78e@mail.gmail.com> <45f85f71002181726m5e29dd65o14e6a6bb8bd4d0a1@mail.gmail.com>
In-Reply-To: <45f85f71002181726m5e29dd65o14e6a6bb8bd4d0a1@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Todd Lipcon wrote:
> One question I have is what to do with the JIRA tickets - we probably
> need some kind of bulk edit, right? Anything that's currently resolved
> with fix version 22 needs to be changed over to fix version 21?

Yes, that should be done if we decide to rebase the 0.21 branch.

Doug

From general-return-1054-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 19 01:32:31 2010
Return-Path: <general-return-1054-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45461 invoked from network); 19 Feb 2010 01:32:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Feb 2010 01:32:31 -0000
Received: (qmail 87378 invoked by uid 500); 19 Feb 2010 01:32:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87316 invoked by uid 500); 19 Feb 2010 01:32:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87306 invoked by uid 99); 19 Feb 2010 01:32:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 01:32:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 19 Feb 2010 01:32:28 +0000
Received: (qmail 45316 invoked by uid 99); 19 Feb 2010 01:32:06 -0000
Received: from localhost.apache.org (HELO mail-bw0-f211.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 01:32:06 +0000
Received: by bwz3 with SMTP id 3so859580bwz.29
        for <general@hadoop.apache.org>; Thu, 18 Feb 2010 17:32:04 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.34.3 with SMTP id j3mr3536356bkd.23.1266543124728; Thu, 18 
	Feb 2010 17:32:04 -0800 (PST)
In-Reply-To: <1267dd3b1002181730h613d7565j15b7e495021871fd@mail.gmail.com>
References: <BE5B5A4B-CDBF-473F-B760-DBC1E1ED33C2@apache.org>
	 <1267dd3b1002180105y5339e19elf9712ed719a6500e@mail.gmail.com>
	 <1267dd3b1002180116n32c7d0d6ucc5b9342dfc9da1b@mail.gmail.com>
	 <1267dd3b1002181730h613d7565j15b7e495021871fd@mail.gmail.com>
Date: Thu, 18 Feb 2010 17:32:04 -0800
Message-ID: <1267dd3b1002181732oa3c1805o38a4b8493193be8d@mail.gmail.com>
Subject: Re: [VOTE] Should we release Hadoop 0.20.2 rc2?
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Filed HADOOP-6576 -C

On Thu, Feb 18, 2010 at 5:30 PM, Chris Douglas <cdouglas@apache.org> wrote:
> I rolled a new release candidate, but TestStreamingStatus is failing
> consistently after HADOOP-5623. The test passes when the changes are
> reverted. -C
>
> On Thu, Feb 18, 2010 at 1:16 AM, Chris Douglas <cdouglas@apache.org> wrot=
e:
>> Filed HADOOP-6575.
>>
>> On the bright side, MAPREDUCE-623 can also be part of this release. -C
>>
>> On Thu, Feb 18, 2010 at 1:05 AM, Chris Douglas <cdouglas@apache.org> wro=
te:
>>> *sigh* The tests currently do not compile. HADOOP-6506 added a call in
>>> the build:
>>>
>>> + =A0 =A0<antcall target=3D"run-test-core-fault-inject"/>
>>>
>>> which does not exist in 0.20. -C
>>>
>>> On Wed, Feb 17, 2010 at 5:24 PM, Owen O'Malley <omalley@apache.org> wro=
te:
>>>> Ok, I've rolled a new tarball with Todd's suggested updates to the bra=
nch.
>>>> Please try it out and vote.
>>>>
>>>> http://people.apache.org/~omalley/0.20.2/
>>>>
>>>> -- Owen
>>>>
>>>
>>
>

From general-return-1055-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 19 02:09:09 2010
Return-Path: <general-return-1055-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59787 invoked from network); 19 Feb 2010 02:09:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Feb 2010 02:09:09 -0000
Received: (qmail 5473 invoked by uid 500); 19 Feb 2010 02:09:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5390 invoked by uid 500); 19 Feb 2010 02:09:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5379 invoked by uid 99); 19 Feb 2010 02:09:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 02:09:08 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 02:08:59 +0000
Received: from [127.0.0.1] (gentlepaint-lx.corp.yahoo.com [10.72.185.127])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o1J289AA080128
	for <general@hadoop.apache.org>; Thu, 18 Feb 2010 18:08:17 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=HOEQH268sF6mVIrrv/SxmZNr/Bz27L60t5BMEHVweNfcswQ8idBdnpN9ua9YFRMA
Message-ID: <4B7DF289.5060304@yahoo-inc.com>
Date: Thu, 18 Feb 2010 18:08:09 -0800
From: Konstantin Shvachko <shv@yahoo-inc.com>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.7) Gecko/20100111 Thunderbird/3.0.1
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Release plans
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com>	 <4B7D9503.4080205@apache.org>	 <DA9E7EF1-9A82-414E-AA5C-DD1088869CD2@apache.org>	 <c9118d4c1002181506l7ec9840dq3690158ebc70c02@mail.gmail.com>	 <EFFFAD8D-1998-4C19-9BD3-D80B3B79577A@apache.org> <c9118d4c1002181719v52a11b0cu55159f344558a78e@mail.gmail.com>
In-Reply-To: <c9118d4c1002181719v52a11b0cu55159f344558a78e@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 2/18/2010 5:19 PM, Jeff Hammerbacher wrote:
> Do we have consensus around rebasing on 0.21? Anyone already testing on 0.21
> who would be upset if the current branch were to be retired?

Rebasing 0.21 will further delay the release.
In current 0.21 branch there is some 28 blockers,
which will take a couple of weeks to fix.
The rebased 0.21 will add to this more issues, and therefore
more time. Based on the experience I had time to stabilize
the release is measured in months rather than weeks.

--Konstantin


>
> On Thu, Feb 18, 2010 at 5:02 PM, Owen O'Malley<omalley@apache.org>  wrote:
>
>> >  On Feb 18, 2010, at 3:06 PM, Jeff Hammerbacher wrote:
>> >
>> >    I think the community will step up to knock down some of the
>>> >>  blockers once we resolve what should be in the 0.21 release: the current
>>> >>  branch, or a rebase on trunk. Do you/Yahoo! have a preference on that
>>> >>  front?
>>> >>
>> >
>> >  Yahoo doesn't care. Even if we rebase the 0.21 branch, because of the
>> >  timing, Yahoo will probably never deploy it. (It take months to push a
>> >  release through QA and production testing, as I wrote the security release
>> >  will hit the pipeline this year (code complete in february, first
>> >  integration cluster in april, on all production clusters by august). Yahoo
>> >  can't handle another big release until january 2011.
>> >
>> >  Personally, I'd prefer to rebase 0.21, especially after we have the Maven
>> >  story straightened out. Generating good poms would be a huge win for
>> >  downstream projects.
>> >
>> >  One big concern is that backwards incompatibility is a big cost. Especially
>> >  if 0.21 (like 0.19) never gets wide deployment, I'd like to start a vote
>> >  that we don't make any API incompatible in 0.22 relative to 0.20.
>> >
>> >  -- Owen
>> >


From general-return-1056-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 19 02:13:20 2010
Return-Path: <general-return-1056-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60497 invoked from network); 19 Feb 2010 02:13:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Feb 2010 02:13:19 -0000
Received: (qmail 7107 invoked by uid 500); 19 Feb 2010 02:13:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7029 invoked by uid 500); 19 Feb 2010 02:13:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7014 invoked by uid 99); 19 Feb 2010 02:13:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 02:13:18 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 02:13:10 +0000
Received: by pwi6 with SMTP id 6so682909pwi.35
        for <general@hadoop.apache.org>; Thu, 18 Feb 2010 18:12:49 -0800 (PST)
MIME-Version: 1.0
Received: by 10.141.213.27 with SMTP id p27mr6828428rvq.176.1266545569189; 
	Thu, 18 Feb 2010 18:12:49 -0800 (PST)
In-Reply-To: <4B7DF289.5060304@yahoo-inc.com>
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com> 
	<4B7D9503.4080205@apache.org> <DA9E7EF1-9A82-414E-AA5C-DD1088869CD2@apache.org> 
	<c9118d4c1002181506l7ec9840dq3690158ebc70c02@mail.gmail.com> 
	<EFFFAD8D-1998-4C19-9BD3-D80B3B79577A@apache.org> <c9118d4c1002181719v52a11b0cu55159f344558a78e@mail.gmail.com> 
	<4B7DF289.5060304@yahoo-inc.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Thu, 18 Feb 2010 18:12:29 -0800
Message-ID: <45f85f71002181812s66f154fan551840c3b70bad39@mail.gmail.com>
Subject: Re: Release plans
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Thu, Feb 18, 2010 at 6:08 PM, Konstantin Shvachko <shv@yahoo-inc.com> wr=
ote:
> On 2/18/2010 5:19 PM, Jeff Hammerbacher wrote:
>>
>> Do we have consensus around rebasing on 0.21? Anyone already testing on
>> 0.21
>> who would be upset if the current branch were to be retired?
>
> Rebasing 0.21 will further delay the release.
> In current 0.21 branch there is some 28 blockers,
> which will take a couple of weeks to fix.
> The rebased 0.21 will add to this more issues, and therefore
> more time. Based on the experience I had time to stabilize
> the release is measured in months rather than weeks.

I agree that we're probably talking months rather than weeks. However,
I see a lot of the stabilization time as a fixed cost regardless of
the number of changes. Certainly there is an O(n) component too, but I
don't think the stabilization time of a rebased 21 is double the
stabilization time of current 21. Maybe more like 30% more? Do you
disagree?

-Todd


> --Konstantin
>
>
>>
>> On Thu, Feb 18, 2010 at 5:02 PM, Owen O'Malley<omalley@apache.org> =A0wr=
ote:
>>
>>> > =A0On Feb 18, 2010, at 3:06 PM, Jeff Hammerbacher wrote:
>>> >
>>> > =A0 =A0I think the community will step up to knock down some of the
>>>>
>>>> >> =A0blockers once we resolve what should be in the 0.21 release: the
>>>> >> current
>>>> >> =A0branch, or a rebase on trunk. Do you/Yahoo! have a preference on
>>>> >> that
>>>> >> =A0front?
>>>> >>
>>>
>>> >
>>> > =A0Yahoo doesn't care. Even if we rebase the 0.21 branch, because of =
the
>>> > =A0timing, Yahoo will probably never deploy it. (It take months to pu=
sh a
>>> > =A0release through QA and production testing, as I wrote the security
>>> > release
>>> > =A0will hit the pipeline this year (code complete in february, first
>>> > =A0integration cluster in april, on all production clusters by august=
).
>>> > Yahoo
>>> > =A0can't handle another big release until january 2011.
>>> >
>>> > =A0Personally, I'd prefer to rebase 0.21, especially after we have th=
e
>>> > Maven
>>> > =A0story straightened out. Generating good poms would be a huge win f=
or
>>> > =A0downstream projects.
>>> >
>>> > =A0One big concern is that backwards incompatibility is a big cost.
>>> > Especially
>>> > =A0if 0.21 (like 0.19) never gets wide deployment, I'd like to start =
a
>>> > vote
>>> > =A0that we don't make any API incompatible in 0.22 relative to 0.20.
>>> >
>>> > =A0-- Owen
>>> >
>
>

From general-return-1057-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 19 03:43:08 2010
Return-Path: <general-return-1057-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88546 invoked from network); 19 Feb 2010 03:43:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Feb 2010 03:43:08 -0000
Received: (qmail 49806 invoked by uid 500); 19 Feb 2010 03:43:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49732 invoked by uid 500); 19 Feb 2010 03:43:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49722 invoked by uid 99); 19 Feb 2010 03:43:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 03:43:06 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [209.85.222.181] (HELO mail-pz0-f181.google.com) (209.85.222.181)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 03:42:56 +0000
Received: by pzk11 with SMTP id 11so370074pzk.30
        for <general@hadoop.apache.org>; Thu, 18 Feb 2010 19:42:34 -0800 (PST)
MIME-Version: 1.0
Received: by 10.141.213.42 with SMTP id p42mr6882701rvq.168.1266550953186; 
	Thu, 18 Feb 2010 19:42:33 -0800 (PST)
X-Originating-IP: [128.12.163.130]
In-Reply-To: <45f85f71002181812s66f154fan551840c3b70bad39@mail.gmail.com>
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com>
	 <4B7D9503.4080205@apache.org>
	 <DA9E7EF1-9A82-414E-AA5C-DD1088869CD2@apache.org>
	 <c9118d4c1002181506l7ec9840dq3690158ebc70c02@mail.gmail.com>
	 <EFFFAD8D-1998-4C19-9BD3-D80B3B79577A@apache.org>
	 <c9118d4c1002181719v52a11b0cu55159f344558a78e@mail.gmail.com>
	 <4B7DF289.5060304@yahoo-inc.com>
	 <45f85f71002181812s66f154fan551840c3b70bad39@mail.gmail.com>
Date: Thu, 18 Feb 2010 19:42:33 -0800
Message-ID: <ae78cd001002181942y289bf985i231316ad101b7ca2@mail.gmail.com>
Subject: Re: Release plans
From: Tomer Shiran <tshiran@maprtech.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd1b3e0a79778047febdf3c
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd1b3e0a79778047febdf3c
Content-Type: text/plain; charset=ISO-8859-1

If we include symlinks, security and Avro in 0.21, then what's the feature
set for 0.22? Do we have any big items planned?

Thanks,
Tomer

On Thu, Feb 18, 2010 at 6:12 PM, Todd Lipcon <todd@cloudera.com> wrote:

> On Thu, Feb 18, 2010 at 6:08 PM, Konstantin Shvachko <shv@yahoo-inc.com>
> wrote:
> > On 2/18/2010 5:19 PM, Jeff Hammerbacher wrote:
> >>
> >> Do we have consensus around rebasing on 0.21? Anyone already testing on
> >> 0.21
> >> who would be upset if the current branch were to be retired?
> >
> > Rebasing 0.21 will further delay the release.
> > In current 0.21 branch there is some 28 blockers,
> > which will take a couple of weeks to fix.
> > The rebased 0.21 will add to this more issues, and therefore
> > more time. Based on the experience I had time to stabilize
> > the release is measured in months rather than weeks.
>
> I agree that we're probably talking months rather than weeks. However,
> I see a lot of the stabilization time as a fixed cost regardless of
> the number of changes. Certainly there is an O(n) component too, but I
> don't think the stabilization time of a rebased 21 is double the
> stabilization time of current 21. Maybe more like 30% more? Do you
> disagree?
>
> -Todd
>
>
> > --Konstantin
> >
> >
> >>
> >> On Thu, Feb 18, 2010 at 5:02 PM, Owen O'Malley<omalley@apache.org>
>  wrote:
> >>
> >>> >  On Feb 18, 2010, at 3:06 PM, Jeff Hammerbacher wrote:
> >>> >
> >>> >    I think the community will step up to knock down some of the
> >>>>
> >>>> >>  blockers once we resolve what should be in the 0.21 release: the
> >>>> >> current
> >>>> >>  branch, or a rebase on trunk. Do you/Yahoo! have a preference on
> >>>> >> that
> >>>> >>  front?
> >>>> >>
> >>>
> >>> >
> >>> >  Yahoo doesn't care. Even if we rebase the 0.21 branch, because of
> the
> >>> >  timing, Yahoo will probably never deploy it. (It take months to push
> a
> >>> >  release through QA and production testing, as I wrote the security
> >>> > release
> >>> >  will hit the pipeline this year (code complete in february, first
> >>> >  integration cluster in april, on all production clusters by august).
> >>> > Yahoo
> >>> >  can't handle another big release until january 2011.
> >>> >
> >>> >  Personally, I'd prefer to rebase 0.21, especially after we have the
> >>> > Maven
> >>> >  story straightened out. Generating good poms would be a huge win for
> >>> >  downstream projects.
> >>> >
> >>> >  One big concern is that backwards incompatibility is a big cost.
> >>> > Especially
> >>> >  if 0.21 (like 0.19) never gets wide deployment, I'd like to start a
> >>> > vote
> >>> >  that we don't make any API incompatible in 0.22 relative to 0.20.
> >>> >
> >>> >  -- Owen
> >>> >
> >
> >
>



-- 
Tomer Shiran
Director of Product Management | MapR Technologies (www.mapr.com) |
650-804-8657

--000e0cd1b3e0a79778047febdf3c--

From general-return-1058-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 19 04:37:16 2010
Return-Path: <general-return-1058-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 2843 invoked from network); 19 Feb 2010 04:37:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Feb 2010 04:37:16 -0000
Received: (qmail 72854 invoked by uid 500); 19 Feb 2010 04:37:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72708 invoked by uid 500); 19 Feb 2010 04:37:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72698 invoked by uid 99); 19 Feb 2010 04:37:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 04:37:14 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 209.85.221.184 as permitted sender)
Received: from [209.85.221.184] (HELO mail-qy0-f184.google.com) (209.85.221.184)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 04:37:08 +0000
Received: by qyk14 with SMTP id 14so1030374qyk.9
        for <general@hadoop.apache.org>; Thu, 18 Feb 2010 20:36:47 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:message-id:subject:from:to
         :content-type;
        bh=UW/wdlszPfBFkRjm91tC2C/cM5COH3xF8RYlV6WYIZk=;
        b=Nm+ElHvvD0vstmyW0b2JWZf6KnzNA0kGwzO7EOnr/l+n6xsSGRYEb4nwlmFuDMlo/U
         rBEf6y4h+K1NA3ngQ95lttcM+/I7AsFf+ar6rgNtTBZlGrf6lWb/5wiFRoPkCOquvUND
         z8nZJt0y6UieHoNkrfXlK7GakSjPQJYube/7Y=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=dZ8SnCGqXF46uyHCpMEVSWB5+LNavDHS6l+/8VwL9wqpuymqXVjbS+f3ffMa1xJ3Wv
         FwocwzYSt95lwtx289nJgFL/GwAzTXTI9quY2MM2d462rd+Gnc5IO4w2vqAX/mbiAq5v
         S84ZJQmYFQhF/YTbH9oKvTQ7YlNvwNAHgDVGY=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
Received: by 10.229.33.70 with SMTP id g6mr12053qcd.75.1266554207422; Thu, 18 
	Feb 2010 20:36:47 -0800 (PST)
In-Reply-To: <EFFFAD8D-1998-4C19-9BD3-D80B3B79577A@apache.org>
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com>
	 <4B7D9503.4080205@apache.org>
	 <DA9E7EF1-9A82-414E-AA5C-DD1088869CD2@apache.org>
	 <c9118d4c1002181506l7ec9840dq3690158ebc70c02@mail.gmail.com>
	 <EFFFAD8D-1998-4C19-9BD3-D80B3B79577A@apache.org>
Date: Thu, 18 Feb 2010 20:36:47 -0800
X-Google-Sender-Auth: b40400ed77101547
Message-ID: <7c962aed1002182036l7db697ahcdc581fde3619405@mail.gmail.com>
Subject: Re: Release plans
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Thu, Feb 18, 2010 at 5:02 PM, Owen O'Malley <omalley@apache.org> wrote:
> Yahoo doesn't care. Even if we rebase the 0.21 branch, because of the
> timing, Yahoo will probably never deploy it. (It take months to push a
> release through QA and production testing, as I wrote the security release
> will hit the pipeline this year (code complete in february, first
> integration cluster in april, on all production clusters by august). Yahoo
> can't handle another big release until january 2011.

I haven't done a survey but my guess is that the HBase crew would
favor putting out the current 0.21 branch as 0.21 rather than
rebasing.  To us, hdfs and its flush, in testing, runs great.  I'm
with Konstantin and Sanjay that a release is pretty close and that a
rebasing pulling in new features means more months even allowing for
the Todd stabilizing constant.

If the project is not getting any heavy-duty Y!-fu till ~January 2011,
that sounds like hadoop 0.23 time, rather than 0.22, to me, that is if
we want to get the project back on 6-month cycles again (> 1 year
between major releases ain't healthy).

St.Ack

From general-return-1059-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 19 09:08:09 2010
Return-Path: <general-return-1059-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93303 invoked from network); 19 Feb 2010 09:08:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Feb 2010 09:08:09 -0000
Received: (qmail 80720 invoked by uid 500); 19 Feb 2010 09:08:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80626 invoked by uid 500); 19 Feb 2010 09:08:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80616 invoked by uid 99); 19 Feb 2010 09:08:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 09:08:07 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 19 Feb 2010 09:08:05 +0000
Received: (qmail 93005 invoked by uid 99); 19 Feb 2010 09:07:43 -0000
Received: from localhost.apache.org (HELO mail-bw0-f211.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 09:07:43 +0000
Received: by bwz3 with SMTP id 3so1050279bwz.29
        for <general@hadoop.apache.org>; Fri, 19 Feb 2010 01:07:41 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.131.210 with SMTP id y18mr458217bks.100.1266570460462; 
	Fri, 19 Feb 2010 01:07:40 -0800 (PST)
Date: Fri, 19 Feb 2010 01:07:40 -0800
Message-ID: <1267dd3b1002190107y65801a7iab7c5c33cad2daf@mail.gmail.com>
Subject: [VOTE] Should we release 0.20.2-rc4 ?
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

There are now only two consistently failing testcases in my
environment, both in the capacity-scheduler contrib module:

org.apache.hadoop.mapred.TestJobInitialization
org.apache.hadoop.mapred.TestQueueCapacities

neither of which is a regression from 0.20.1

http://people.apache.org/~cdouglas/0.20.2-rc4

Please try it out. -C

From general-return-1060-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 19 16:53:29 2010
Return-Path: <general-return-1060-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 79709 invoked from network); 19 Feb 2010 16:53:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Feb 2010 16:53:29 -0000
Received: (qmail 32314 invoked by uid 500); 19 Feb 2010 16:53:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32244 invoked by uid 500); 19 Feb 2010 16:53:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32234 invoked by uid 99); 19 Feb 2010 16:53:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 16:53:28 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ashutosh.chauhan@gmail.com designates 209.85.219.218 as permitted sender)
Received: from [209.85.219.218] (HELO mail-ew0-f218.google.com) (209.85.219.218)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 16:53:21 +0000
Received: by ewy10 with SMTP id 10so294984ewy.11
        for <general@hadoop.apache.org>; Fri, 19 Feb 2010 08:53:00 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :from:date:message-id:subject:to:content-type;
        bh=+BiC5rMGg9IeQGBKEPhxOKcrYTCeBhiPhL2Lt+0DWOY=;
        b=uVaun29i/JXKh50ZlDxwq+aU6fsvaACRwbLSd1WGAeqvPAvF+wCaZgUFMxL6vFPQr/
         2HAkwD8w24G4FLGzyo1agn+T88ih3Sk/I26lGiKuze42UXq9MEFV6rn+tOTpgRlPgQQH
         DIXS8N9dQ5VaUlk6Duq5/NP/dIMNNSJUhXMDY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=wDq6AOU0dlM6YRv/DamVTe4u8qge3tuoF6dLqV4HZlbJbyt/3UvnlrxpKP/vj88zaC
         tHaPE9F+gMb2qQQXC6+xUbHljJ0Ofj350ES6vQ3glWeRO+vxzWXWJT8/ABl70FEsE14B
         Kz6MdDk9yGkEEXnPYIitNtX0ki358/OvqlJK0=
MIME-Version: 1.0
Received: by 10.216.87.79 with SMTP id x57mr3325245wee.83.1266598379911; Fri, 
	19 Feb 2010 08:52:59 -0800 (PST)
In-Reply-To: <3316F847-FBB7-4D41-B1A7-C5CD3FB1A40D@yahoo-inc.com>
References: <3316F847-FBB7-4D41-B1A7-C5CD3FB1A40D@yahoo-inc.com>
From: Ashutosh Chauhan <ashutosh.chauhan@gmail.com>
Date: Fri, 19 Feb 2010 08:52:38 -0800
Message-ID: <54de02ae1002190852re61625ehf3c477b1975836d1@mail.gmail.com>
Subject: Re: [VOTE] Release candidate for Pig 0.6.0
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d7dfa781f94e047ff6eaf8

--0016e6d7dfa781f94e047ff6eaf8
Content-Type: text/plain; charset=ISO-8859-1

 Downloaded the tar ball. Build the jar. Ran tests, small scripts and few
dfs commands through grunt. All looks good.
+1

Ashutosh

On Tue, Feb 16, 2010 at 11:22, Alan Gates <gates@yahoo-inc.com> wrote:

> All,
>
> I have rolled a release candidate for Pig 0.6.0 and uploaded it to
> http://people.apache.org/~gates/pig-0.6.0-candidate-1/<http://people.apache.org/%7Egates/pig-0.6.0-candidate-1/>
> A description of what is new and different in this release is included in
> the release notes, which I have also uploaded to that directory.  Please
> download the candidate, test it, and vote.
>
> Alan.
>

--0016e6d7dfa781f94e047ff6eaf8--

From general-return-1061-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 19 19:08:29 2010
Return-Path: <general-return-1061-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57771 invoked from network); 19 Feb 2010 19:08:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Feb 2010 19:08:29 -0000
Received: (qmail 16554 invoked by uid 500); 19 Feb 2010 19:08:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16467 invoked by uid 500); 19 Feb 2010 19:08:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16457 invoked by uid 99); 19 Feb 2010 19:08:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 19:08:27 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 74.125.92.25 as permitted sender)
Received: from [74.125.92.25] (HELO qw-out-2122.google.com) (74.125.92.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 19:08:20 +0000
Received: by qw-out-2122.google.com with SMTP id 5so72968qwi.35
        for <general@hadoop.apache.org>; Fri, 19 Feb 2010 11:08:00 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:message-id:subject:from:to
         :content-type;
        bh=CdeBb/yPjpb9vv2Kc7zmf+3prhMLcEStXNn3h5+dGG8=;
        b=OQCPmFeJ/mkYxZ8rzBForOOgahWgeL8+jmhNoqa4LCqc/Puie5yLxRU4sDJINRTrmw
         MH3NZ7/9sxVafydTrOQ7Pej56DYeyaTcaruVu4ekJPAkNlivrHm7nT4W+Fd7ialoUdqw
         A8Qv/4sDjhzogYQywL07Myrr400l1Xs9ZZxc0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=s/IVXFFBO3GJSar1jVrKzHAbmX2NztfbSfbCdJNGar5+h9WFmq//ci5PP64vRQMzgq
         YvlwolEzDPuGyucnB4IJvpGy6vs/qNeBItFtEjxnf+wAy/GixweQlvJKC0OJco9mVWjA
         /9Sl+e49/H7rdxDvklZrNLYSlWdfK3WQ43Hl8=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
Received: by 10.229.230.84 with SMTP id jl20mr4980728qcb.88.1266606480208; 
	Fri, 19 Feb 2010 11:08:00 -0800 (PST)
In-Reply-To: <1267dd3b1002190107y65801a7iab7c5c33cad2daf@mail.gmail.com>
References: <1267dd3b1002190107y65801a7iab7c5c33cad2daf@mail.gmail.com>
Date: Fri, 19 Feb 2010 11:08:00 -0800
X-Google-Sender-Auth: ddc1077375cf35de
Message-ID: <7c962aed1002191108n76516c82q106e7d3e5aeb9428@mail.gmail.com>
Subject: Re: [VOTE] Should we release 0.20.2-rc4 ?
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

+1

I put up on a small cluster under load.  Seems to work fine.  Trolled
logs a while.  Nothing out of the ordinary.  Checked docs.  They look
grand.

St.Ack

On Fri, Feb 19, 2010 at 1:07 AM, Chris Douglas <cdouglas@apache.org> wrote:
> There are now only two consistently failing testcases in my
> environment, both in the capacity-scheduler contrib module:
>
> org.apache.hadoop.mapred.TestJobInitialization
> org.apache.hadoop.mapred.TestQueueCapacities
>
> neither of which is a regression from 0.20.1
>
> http://people.apache.org/~cdouglas/0.20.2-rc4
>
> Please try it out. -C
>

From general-return-1062-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 19 21:44:39 2010
Return-Path: <general-return-1062-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 11000 invoked from network); 19 Feb 2010 21:44:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Feb 2010 21:44:39 -0000
Received: (qmail 22522 invoked by uid 500); 19 Feb 2010 21:44:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22442 invoked by uid 500); 19 Feb 2010 21:44:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22432 invoked by uid 99); 19 Feb 2010 21:44:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 21:44:37 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 21:44:31 +0000
Received: by pwi6 with SMTP id 6so511574pwi.35
        for <general@hadoop.apache.org>; Fri, 19 Feb 2010 13:44:10 -0800 (PST)
MIME-Version: 1.0
Received: by 10.142.67.7 with SMTP id p7mr7730105wfa.120.1266615850068; Fri, 
	19 Feb 2010 13:44:10 -0800 (PST)
In-Reply-To: <7c962aed1002182036l7db697ahcdc581fde3619405@mail.gmail.com>
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com>
	 <4B7D9503.4080205@apache.org>
	 <DA9E7EF1-9A82-414E-AA5C-DD1088869CD2@apache.org>
	 <c9118d4c1002181506l7ec9840dq3690158ebc70c02@mail.gmail.com>
	 <EFFFAD8D-1998-4C19-9BD3-D80B3B79577A@apache.org>
	 <7c962aed1002182036l7db697ahcdc581fde3619405@mail.gmail.com>
Date: Fri, 19 Feb 2010 13:44:09 -0800
Message-ID: <dfe484f01002191344u27c3c24eoe5fb5c5aa5701658@mail.gmail.com>
Subject: Re: Release plans
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Thu, Feb 18, 2010 at 8:36 PM, Stack <stack@duboce.net> wrote:
> On Thu, Feb 18, 2010 at 5:02 PM, Owen O'Malley <omalley@apache.org> wrote=
:
>> Yahoo doesn't care. Even if we rebase the 0.21 branch, because of the
>> timing, Yahoo will probably never deploy it. (It take months to push a
>> release through QA and production testing, as I wrote the security relea=
se
>> will hit the pipeline this year (code complete in february, first
>> integration cluster in april, on all production clusters by august). Yah=
oo
>> can't handle another big release until january 2011.
>
> I haven't done a survey but my guess is that the HBase crew would
> favor putting out the current 0.21 branch as 0.21 rather than
> rebasing. =A0To us, hdfs and its flush, in testing, runs great. =A0I'm
> with Konstantin and Sanjay that a release is pretty close and that a
> rebasing pulling in new features means more months even allowing for
> the Todd stabilizing constant.

Given that Yahoo and others plan to skip 21, and the HBase crowd could
actually use 21 and have invested in the current branch's contents, it
sounds like we should try to get the current branch released. Anyone
object?

Thanks,
Eli

From general-return-1063-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 19 21:48:51 2010
Return-Path: <general-return-1063-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12721 invoked from network); 19 Feb 2010 21:48:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Feb 2010 21:48:51 -0000
Received: (qmail 28773 invoked by uid 500); 19 Feb 2010 21:48:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28685 invoked by uid 500); 19 Feb 2010 21:48:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28675 invoked by uid 99); 19 Feb 2010 21:48:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 21:48:49 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [209.85.216.182] (HELO mail-px0-f182.google.com) (209.85.216.182)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 21:48:41 +0000
Received: by pxi12 with SMTP id 12so299513pxi.33
        for <general@hadoop.apache.org>; Fri, 19 Feb 2010 13:48:20 -0800 (PST)
MIME-Version: 1.0
Received: by 10.141.124.18 with SMTP id b18mr995324rvn.151.1266616100158; Fri, 
	19 Feb 2010 13:48:20 -0800 (PST)
In-Reply-To: <dfe484f01002191344u27c3c24eoe5fb5c5aa5701658@mail.gmail.com>
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com> 
	<4B7D9503.4080205@apache.org> <DA9E7EF1-9A82-414E-AA5C-DD1088869CD2@apache.org> 
	<c9118d4c1002181506l7ec9840dq3690158ebc70c02@mail.gmail.com> 
	<EFFFAD8D-1998-4C19-9BD3-D80B3B79577A@apache.org> <7c962aed1002182036l7db697ahcdc581fde3619405@mail.gmail.com> 
	<dfe484f01002191344u27c3c24eoe5fb5c5aa5701658@mail.gmail.com>
From: Aaron Kimball <aaron@cloudera.com>
Date: Fri, 19 Feb 2010 13:48:00 -0800
Message-ID: <d6d7c4411002191348sbcbe6d9s859200a6c2cc98ea@mail.gmail.com>
Subject: Re: Release plans
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000325562e1eb77c12047ffb0aba
X-Virus-Checked: Checked by ClamAV on apache.org

--000325562e1eb77c12047ffb0aba
Content-Type: text/plain; charset=ISO-8859-1

+1 from me.

I agree with Stack; faster release cycles will help keep the project focused
and get code testing and soaking in more environments.

- Aaron

On Fri, Feb 19, 2010 at 1:44 PM, Eli Collins <eli@cloudera.com> wrote:

> On Thu, Feb 18, 2010 at 8:36 PM, Stack <stack@duboce.net> wrote:
> > On Thu, Feb 18, 2010 at 5:02 PM, Owen O'Malley <omalley@apache.org>
> wrote:
> >> Yahoo doesn't care. Even if we rebase the 0.21 branch, because of the
> >> timing, Yahoo will probably never deploy it. (It take months to push a
> >> release through QA and production testing, as I wrote the security
> release
> >> will hit the pipeline this year (code complete in february, first
> >> integration cluster in april, on all production clusters by august).
> Yahoo
> >> can't handle another big release until january 2011.
> >
> > I haven't done a survey but my guess is that the HBase crew would
> > favor putting out the current 0.21 branch as 0.21 rather than
> > rebasing.  To us, hdfs and its flush, in testing, runs great.  I'm
> > with Konstantin and Sanjay that a release is pretty close and that a
> > rebasing pulling in new features means more months even allowing for
> > the Todd stabilizing constant.
>
> Given that Yahoo and others plan to skip 21, and the HBase crowd could
> actually use 21 and have invested in the current branch's contents, it
> sounds like we should try to get the current branch released. Anyone
> object?
>
> Thanks,
> Eli
>

--000325562e1eb77c12047ffb0aba--

From general-return-1064-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 19 21:58:56 2010
Return-Path: <general-return-1064-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17765 invoked from network); 19 Feb 2010 21:58:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Feb 2010 21:58:56 -0000
Received: (qmail 49138 invoked by uid 500); 19 Feb 2010 21:58:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49075 invoked by uid 500); 19 Feb 2010 21:58:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49065 invoked by uid 99); 19 Feb 2010 21:58:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 21:58:54 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 19 Feb 2010 21:58:52 +0000
Received: (qmail 17592 invoked by uid 99); 19 Feb 2010 21:58:30 -0000
Received: from localhost.apache.org (HELO [192.168.168.107]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 21:58:30 +0000
Message-ID: <4B7F0985.3010000@apache.org>
Date: Fri, 19 Feb 2010 13:58:29 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Release plans
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com>	 <4B7D9503.4080205@apache.org>	 <DA9E7EF1-9A82-414E-AA5C-DD1088869CD2@apache.org>	 <c9118d4c1002181506l7ec9840dq3690158ebc70c02@mail.gmail.com>	 <EFFFAD8D-1998-4C19-9BD3-D80B3B79577A@apache.org>	 <7c962aed1002182036l7db697ahcdc581fde3619405@mail.gmail.com> <dfe484f01002191344u27c3c24eoe5fb5c5aa5701658@mail.gmail.com>
In-Reply-To: <dfe484f01002191344u27c3c24eoe5fb5c5aa5701658@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Eli Collins wrote:
> Given that Yahoo and others plan to skip 21, and the HBase crowd could
> actually use 21 and have invested in the current branch's contents, it
> sounds like we should try to get the current branch released.

My concern is more about when the next branch from trunk will be made. 
If we embrace a six-month trunk branch cycle, then the next branch from 
trunk should be created soon, regardless of whether we call it 21 or 22.

Also note that this decision has compatibility implications, unless we 
decide to, post-fact, declare that 0.20 was actually a major release 
(1.0) and that 0.21 (1.1) and 0.22 (1.2) are minor, feature releases, 
that may add deprecations and features but may not remove any or 
otherwise change APIs incompatibly.

Doug

From general-return-1065-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 19 21:59:09 2010
Return-Path: <general-return-1065-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17884 invoked from network); 19 Feb 2010 21:59:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Feb 2010 21:59:09 -0000
Received: (qmail 50047 invoked by uid 500); 19 Feb 2010 21:59:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49979 invoked by uid 500); 19 Feb 2010 21:59:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49965 invoked by uid 99); 19 Feb 2010 21:59:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 21:59:07 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 21:58:58 +0000
Received: from pugholehand-lm.corp.yahoo.com (pugholehand-lm.corp.yahoo.com [10.72.115.44])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o1JLvq07004999;
	Fri, 19 Feb 2010 13:57:52 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=from:to:in-reply-to:subject:x-priority:references:
	message-id:content-type:content-transfer-encoding:mime-version:date:cc:x-mailer;
	b=KQu6yvWXssBy4z2tV0jTlYEkirYAn45ai2PdbT3vcMA8H3qaBb3v3hleqgRo57Mf
From: Alan Gates <gates@yahoo-inc.com>
To: private@hadoop.apache.org
In-Reply-To: <1E049E93BEE249D59D88B0DDACEF9106@DanielLaptop>
Subject: Re: [VOTE] Release candidate for Pig 0.6.0
X-Priority: 3
References: <3316F847-FBB7-4D41-B1A7-C5CD3FB1A40D@yahoo-inc.com> <14B7BB1E-F459-4FBC-AEBD-120B822281E3@yahoo-inc.com> <1E049E93BEE249D59D88B0DDACEF9106@DanielLaptop>
Message-Id: <466AB331-82A3-4111-BA34-75F44A31DBCF@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Date: Fri, 19 Feb 2010 13:57:51 -0800
Cc: general@hadoop.apache.org
X-Mailer: Apple Mail (2.936)

Alright, I'll fix that and reroll the release.

Alan.

On Feb 19, 2010, at 1:35 PM, Daniel Dai wrote:

> "pig -version" should report 0.6.0 instead of 0.6.1. +1 otherwise.
>
>
> --------------------------------------------------
> From: "Alan Gates" <gates@yahoo-inc.com>
> Sent: Friday, February 19, 2010 11:58 AM
> To: <private@hadoop.apache.org>
> Subject: Fwd: [VOTE] Release candidate for Pig 0.6.0
>
>> PMC members,
>>
>> Please take a look and vote.  Thanks.
>>
>> Alan.
>>
>> Begin forwarded message:
>>
>>> From: Alan Gates <gates@yahoo-inc.com>
>>> Date: February 16, 2010 11:22:52 AM PST
>>> To: <general@hadoop.apache.org>
>>> Subject: [VOTE] Release candidate for Pig 0.6.0
>>> Reply-To: general@hadoop.apache.org
>>>
>>> All,
>>>
>>> I have rolled a release candidate for Pig 0.6.0 and uploaded it to http://people.apache.org/~gates/pig-0.6.0-candidate-1/
>>> A description of what is new and different in this release is   
>>> included in the release notes, which I have also uploaded to that   
>>> directory. Please download the candidate, test it, and vote.
>>>
>>> Alan.


From general-return-1066-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 19 22:20:30 2010
Return-Path: <general-return-1066-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 32154 invoked from network); 19 Feb 2010 22:20:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Feb 2010 22:20:30 -0000
Received: (qmail 84418 invoked by uid 500); 19 Feb 2010 22:20:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84348 invoked by uid 500); 19 Feb 2010 22:20:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84338 invoked by uid 99); 19 Feb 2010 22:20:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 22:20:29 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 22:20:21 +0000
Received: by pwi6 with SMTP id 6so539717pwi.35
        for <general@hadoop.apache.org>; Fri, 19 Feb 2010 14:20:00 -0800 (PST)
MIME-Version: 1.0
Received: by 10.142.118.3 with SMTP id q3mr2819803wfc.199.1266617999971; Fri, 
	19 Feb 2010 14:19:59 -0800 (PST)
In-Reply-To: <4B7F0985.3010000@apache.org>
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com>
	 <4B7D9503.4080205@apache.org>
	 <DA9E7EF1-9A82-414E-AA5C-DD1088869CD2@apache.org>
	 <c9118d4c1002181506l7ec9840dq3690158ebc70c02@mail.gmail.com>
	 <EFFFAD8D-1998-4C19-9BD3-D80B3B79577A@apache.org>
	 <7c962aed1002182036l7db697ahcdc581fde3619405@mail.gmail.com>
	 <dfe484f01002191344u27c3c24eoe5fb5c5aa5701658@mail.gmail.com>
	 <4B7F0985.3010000@apache.org>
Date: Fri, 19 Feb 2010 14:19:59 -0800
Message-ID: <dfe484f01002191419g691e7651xe589a0a4a1307918@mail.gmail.com>
Subject: Re: Release plans
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Fri, Feb 19, 2010 at 1:58 PM, Doug Cutting <cutting@apache.org> wrote:
> Eli Collins wrote:
>>
>> Given that Yahoo and others plan to skip 21, and the HBase crowd could
>> actually use 21 and have invested in the current branch's contents, it
>> sounds like we should try to get the current branch released.
>
> My concern is more about when the next branch from trunk will be made. If we
> embrace a six-month trunk branch cycle, then the next branch from trunk
> should be created soon, regardless of whether we call it 21 or 22.
>

Does there need to be a dependency? We could still branch 22 once
security, avro, etc are in, even if that's only a couple months from
now. Minor releases are supposed to come out in a matter of months
anyway, since we haven't released a major version yet.

> Also note that this decision has compatibility implications, unless we
> decide to, post-fact, declare that 0.20 was actually a major release (1.0)
> and that 0.21 (1.1) and 0.22 (1.2) are minor, feature releases, that may add
> deprecations and features but may not remove any or otherwise change APIs
> incompatibly.

Could the vote on whether 22 is backwards compatible with 20 be
independent of what we call 1.0? Guess it depends on what type of
backwards compatibility we're talking about (eg API vs wire). Given
that already branched 21 a while back, it seems like we should be able
to release it, regardless of how a compatibility vote goes.

Thanks,
Eli

From general-return-1067-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 19 22:49:55 2010
Return-Path: <general-return-1067-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47147 invoked from network); 19 Feb 2010 22:49:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Feb 2010 22:49:55 -0000
Received: (qmail 30867 invoked by uid 500); 19 Feb 2010 22:49:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30765 invoked by uid 500); 19 Feb 2010 22:49:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30755 invoked by uid 99); 19 Feb 2010 22:49:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 22:49:54 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 19 Feb 2010 22:49:53 +0000
Received: (qmail 46966 invoked by uid 99); 19 Feb 2010 22:49:33 -0000
Received: from localhost.apache.org (HELO [192.168.168.107]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 22:49:33 +0000
Message-ID: <4B7F157C.1090305@apache.org>
Date: Fri, 19 Feb 2010 14:49:32 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Release plans
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com>	 <4B7D9503.4080205@apache.org>	 <DA9E7EF1-9A82-414E-AA5C-DD1088869CD2@apache.org>	 <c9118d4c1002181506l7ec9840dq3690158ebc70c02@mail.gmail.com>	 <EFFFAD8D-1998-4C19-9BD3-D80B3B79577A@apache.org>	 <7c962aed1002182036l7db697ahcdc581fde3619405@mail.gmail.com>	 <dfe484f01002191344u27c3c24eoe5fb5c5aa5701658@mail.gmail.com>	 <4B7F0985.3010000@apache.org> <dfe484f01002191419g691e7651xe589a0a4a1307918@mail.gmail.com>
In-Reply-To: <dfe484f01002191419g691e7651xe589a0a4a1307918@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Eli Collins wrote:
>> My concern is more about when the next branch from trunk will be made.
> 
> Does there need to be a dependency?

No.  I just wanted to note that, from my point of view, releasing from 
the existing 0.21 branch is not sufficient, that, regardless, we still 
need to release from trunk soon, and we need a schedule going forward 
for when future branches from trunk will be made.

Aside from resolving compatibility issues (more on that below) what we 
perhaps need are one or two release master volunteers, for a release 
based on trunk and perhaps for a release based on the 0.21 branch.

> Could the vote on whether 22 is backwards compatible with 20 be
> independent of what we call 1.0?

We minimally need to declare whether releases are major or minor.  The 
convention has been that all 0.X releases are major, 1.0 is major, 1.1, 
1.2, etc. are minor, 2.0 is major, 2.1, 2.2, etc are minor and so on. 
We could amend this convention, but things might get confusing.  For 
example, we could declare that we just have a sequence of numerically 
increasing releases, some major, some minor, but that you can't tell 
which is which from the number.  Or we could, like Sun did with Java, 
move the decimal point, and say that 0.20 is just 2.0, and that, instead 
of 0.21 our next release is 2.1.  Or we could, as I suggested in my 
previous message, retroactively consider 0.20 to be 1.0, and then make 
our next release 1.1.  None are particularly attractive options.

Doug

From general-return-1068-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 19 22:54:09 2010
Return-Path: <general-return-1068-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49370 invoked from network); 19 Feb 2010 22:54:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Feb 2010 22:54:09 -0000
Received: (qmail 37692 invoked by uid 500); 19 Feb 2010 22:54:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37617 invoked by uid 500); 19 Feb 2010 22:54:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37607 invoked by uid 99); 19 Feb 2010 22:54:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 22:54:08 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jaybooth@gmail.com designates 209.85.210.197 as permitted sender)
Received: from [209.85.210.197] (HELO mail-yx0-f197.google.com) (209.85.210.197)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 22:53:59 +0000
Received: by yxe35 with SMTP id 35so605125yxe.2
        for <general@hadoop.apache.org>; Fri, 19 Feb 2010 14:53:39 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=YNwQTOL40LjgN8uS0aUm8HWipOMxzoh1gSQparehvAE=;
        b=Ogz7ieE0dv0wWabI8dexsMukJBnlJaIO7JgbcOlVb+jRbwk715joRnKWmeHfs7Qgcr
         4qRO/YoyzNcar7w98cBUyOHZ4c+eW1XKGaHLRsNlzjLLD0UCeVTIMUAuReCx7i05lsCQ
         rMG1MpvuZu5Dpp8b8sWRkqcRNUGRBTNsKpEJU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=xap9UifRfgkelGr+e6m/r+FOgP4CrtVcSdRbD/y3FGkHcbCiZoITUePhm3SQ4fVekf
         GKiMcLv/7taVYow0WkPWSPy97mem7MrQUa53otDE8Vz9KBsUZ/+p8R08XD5R14/GU8wj
         xKxJhpK+1CFxBOfQQoYlyjVbwy2UiOLISPNnQ=
MIME-Version: 1.0
Received: by 10.101.213.2 with SMTP id p2mr2789673anq.229.1266620018867; Fri, 
	19 Feb 2010 14:53:38 -0800 (PST)
In-Reply-To: <4B7F157C.1090305@apache.org>
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com>
	 <4B7D9503.4080205@apache.org>
	 <DA9E7EF1-9A82-414E-AA5C-DD1088869CD2@apache.org>
	 <c9118d4c1002181506l7ec9840dq3690158ebc70c02@mail.gmail.com>
	 <EFFFAD8D-1998-4C19-9BD3-D80B3B79577A@apache.org>
	 <7c962aed1002182036l7db697ahcdc581fde3619405@mail.gmail.com>
	 <dfe484f01002191344u27c3c24eoe5fb5c5aa5701658@mail.gmail.com>
	 <4B7F0985.3010000@apache.org>
	 <dfe484f01002191419g691e7651xe589a0a4a1307918@mail.gmail.com>
	 <4B7F157C.1090305@apache.org>
Date: Fri, 19 Feb 2010 17:53:38 -0500
Message-ID: <54dc3c51002191453j98b9055vdf85531a5d202bad@mail.gmail.com>
Subject: Re: Release plans
From: Jay Booth <jaybooth@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c9252b4a3e6b047ffbf4c3
X-Virus-Checked: Checked by ClamAV on apache.org

--001636c9252b4a3e6b047ffbf4c3
Content-Type: text/plain; charset=ISO-8859-1

Hadoop 2 EE Version 5?

On Fri, Feb 19, 2010 at 5:49 PM, Doug Cutting <cutting@apache.org> wrote:

> Eli Collins wrote:
>
>> My concern is more about when the next branch from trunk will be made.
>>>
>>
>> Does there need to be a dependency?
>>
>
> No.  I just wanted to note that, from my point of view, releasing from the
> existing 0.21 branch is not sufficient, that, regardless, we still need to
> release from trunk soon, and we need a schedule going forward for when
> future branches from trunk will be made.
>
> Aside from resolving compatibility issues (more on that below) what we
> perhaps need are one or two release master volunteers, for a release based
> on trunk and perhaps for a release based on the 0.21 branch.
>
>
>  Could the vote on whether 22 is backwards compatible with 20 be
>> independent of what we call 1.0?
>>
>
> We minimally need to declare whether releases are major or minor.  The
> convention has been that all 0.X releases are major, 1.0 is major, 1.1, 1.2,
> etc. are minor, 2.0 is major, 2.1, 2.2, etc are minor and so on. We could
> amend this convention, but things might get confusing.  For example, we
> could declare that we just have a sequence of numerically increasing
> releases, some major, some minor, but that you can't tell which is which
> from the number.  Or we could, like Sun did with Java, move the decimal
> point, and say that 0.20 is just 2.0, and that, instead of 0.21 our next
> release is 2.1.  Or we could, as I suggested in my previous message,
> retroactively consider 0.20 to be 1.0, and then make our next release 1.1.
>  None are particularly attractive options.
>
> Doug
>

--001636c9252b4a3e6b047ffbf4c3--

From general-return-1069-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 19 23:15:27 2010
Return-Path: <general-return-1069-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57637 invoked from network); 19 Feb 2010 23:15:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Feb 2010 23:15:26 -0000
Received: (qmail 65810 invoked by uid 500); 19 Feb 2010 23:15:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65757 invoked by uid 500); 19 Feb 2010 23:15:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65747 invoked by uid 99); 19 Feb 2010 23:15:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 23:15:25 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 23:15:18 +0000
Received: by pwi6 with SMTP id 6so579935pwi.35
        for <general@hadoop.apache.org>; Fri, 19 Feb 2010 15:14:58 -0800 (PST)
MIME-Version: 1.0
Received: by 10.142.55.8 with SMTP id d8mr7755268wfa.81.1266621298583; Fri, 19 
	Feb 2010 15:14:58 -0800 (PST)
In-Reply-To: <4B7F157C.1090305@apache.org>
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com>
	 <4B7D9503.4080205@apache.org>
	 <DA9E7EF1-9A82-414E-AA5C-DD1088869CD2@apache.org>
	 <c9118d4c1002181506l7ec9840dq3690158ebc70c02@mail.gmail.com>
	 <EFFFAD8D-1998-4C19-9BD3-D80B3B79577A@apache.org>
	 <7c962aed1002182036l7db697ahcdc581fde3619405@mail.gmail.com>
	 <dfe484f01002191344u27c3c24eoe5fb5c5aa5701658@mail.gmail.com>
	 <4B7F0985.3010000@apache.org>
	 <dfe484f01002191419g691e7651xe589a0a4a1307918@mail.gmail.com>
	 <4B7F157C.1090305@apache.org>
Date: Fri, 19 Feb 2010 15:14:58 -0800
Message-ID: <dfe484f01002191514i5fe5b31cg1e884693737a3c93@mail.gmail.com>
Subject: Re: Release plans
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Fri, Feb 19, 2010 at 2:49 PM, Doug Cutting <cutting@apache.org> wrote:
> Eli Collins wrote:
>>>
>>> My concern is more about when the next branch from trunk will be made.
>>
>> Does there need to be a dependency?
>
> No. =A0I just wanted to note that, from my point of view, releasing from =
the
> existing 0.21 branch is not sufficient, that, regardless, we still need t=
o
> release from trunk soon, and we need a schedule going forward for when
> future branches from trunk will be made.
>

Agreed. Releasing a 21 won't settle the release process question or
tell us when the first major release is. Maybe we could get 21 behind
us and have a separate discussion covering those.

>> Could the vote on whether 22 is backwards compatible with 20 be
>> independent of what we call 1.0?
>
> We minimally need to declare whether releases are major or minor.

I was assuming 21 would be another minor release, didn't hear
otherwise when it was branched. I didn't interpret the compatibility
vote as a suggestion that we should retroactively consider 20 to be
the first major release, but rather a suggestion for voting that we
don't remove APIs in 22 that were deprecated in 20. Owen, please
correct me if I'm wrong.

Thanks,
Eli

From general-return-1070-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 19 23:39:00 2010
Return-Path: <general-return-1070-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 64247 invoked from network); 19 Feb 2010 23:39:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Feb 2010 23:39:00 -0000
Received: (qmail 90741 invoked by uid 500); 19 Feb 2010 23:38:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90663 invoked by uid 500); 19 Feb 2010 23:38:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90653 invoked by uid 99); 19 Feb 2010 23:38:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 23:38:59 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 19 Feb 2010 23:38:56 +0000
Received: (qmail 64217 invoked by uid 99); 19 Feb 2010 23:38:35 -0000
Received: from localhost.apache.org (HELO [192.168.168.107]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Feb 2010 23:38:35 +0000
Message-ID: <4B7F20FA.2050704@apache.org>
Date: Fri, 19 Feb 2010 15:38:34 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Release plans
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com>	 <4B7D9503.4080205@apache.org>	 <DA9E7EF1-9A82-414E-AA5C-DD1088869CD2@apache.org>	 <c9118d4c1002181506l7ec9840dq3690158ebc70c02@mail.gmail.com>	 <EFFFAD8D-1998-4C19-9BD3-D80B3B79577A@apache.org>	 <7c962aed1002182036l7db697ahcdc581fde3619405@mail.gmail.com>	 <dfe484f01002191344u27c3c24eoe5fb5c5aa5701658@mail.gmail.com>	 <4B7F0985.3010000@apache.org>	 <dfe484f01002191419g691e7651xe589a0a4a1307918@mail.gmail.com>	 <4B7F157C.1090305@apache.org> <dfe484f01002191514i5fe5b31cg1e884693737a3c93@mail.gmail.com>
In-Reply-To: <dfe484f01002191514i5fe5b31cg1e884693737a3c93@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Eli Collins wrote:
> Maybe we could get 21 behind
> us and have a separate discussion covering those.

I'm personally more interested in that discussion than in what's 
currently in the 0.21 branch, but others, like Stack, may reasonably be 
of the opposite persuasion.  So rather than putting one before the 
other, can we pursue both in parallel?

> I was assuming 21 would be another minor release, didn't hear
> otherwise when it was branched.

We've never officially had a minor release.  All of our releases have 
been either major or bugfix.  A minor release adds features but does not 
remove any deprecated APIs.  So making 0.21 a minor release would be a 
change in the rules.

Have no deprecations in fact been removed between the 0.20 and 0.21 
branches?

Perhaps we can agree that, after 0.20, no deprecations will be removed 
until 1.0, that all pre-1.0 releases after 0.20 are now minor instead of 
major, that 0.20 was effectively 1.0 but we're not going to rename it, 
and that 1.0 will effectively be 2.0, etc.  Could that work?

Doug

From general-return-1071-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Feb 20 00:19:10 2010
Return-Path: <general-return-1071-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 75954 invoked from network); 20 Feb 2010 00:19:10 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Feb 2010 00:19:10 -0000
Received: (qmail 21548 invoked by uid 500); 20 Feb 2010 00:19:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21478 invoked by uid 500); 20 Feb 2010 00:19:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21468 invoked by uid 99); 20 Feb 2010 00:19:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 20 Feb 2010 00:19:08 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 20 Feb 2010 00:18:59 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:User-Agent:Date:Subject:From:To:
   Message-ID:Thread-Topic:Thread-Index:In-Reply-To:
   Mime-version:Content-type:Content-transfer-encoding;
  b=dpyiqKJYWEoTwMJwzlaxilEhyOLPhPQPGtsCHUlmFXPVCpCCYSqsnnzc
   CbM5Nv0W82mLJhGY6vBrYWV4dnnqOaNZMeRH3o8538rQYAt7o9Z9TJXPG
   HE5f23sAAnSM7Y7;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1266625139; x=1298161139;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20Release=20plans|Date:=20Fri,=2019=20Feb
   =202010=2016:18:23=20-0800|Message-ID:=20<C7A46A4F.3233%a
   wittenauer@linkedin.com>|To:=20<general@hadoop.apache.org
   >|Mime-version:=201.0|Content-transfer-encoding:=207bit
   |In-Reply-To:=20<4B7F20FA.2050704@apache.org>;
  bh=INnM7cn7Dh2GLTWXXfoRTXMexyCNu8LfiGpA4VJTfgs=;
  b=iFBNHHpun83KM3Z0V6tKJFsTk9DyubGOC3eb7Kt3FYpOuYvx7xD0JbKt
   Piy4p791S/LGurXY5I17uBYjExNv8WPbEdYjVR5Nidx3mOmEtfkkovhtd
   4k2e9CI+Cd4NxfE;
X-IronPort-AV: E=Sophos;i="4.49,506,1262592000"; 
   d="scan'208";a="10974159"
Received: from 172.16.22.176 ([172.16.22.176]) by CORP-MAIL.linkedin.biz ([172.18.46.135]) via Exchange Front-End Server mail-access.linkedin.biz ([172.18.46.133]) with Microsoft Exchange Server HTTP-DAV ;
 Sat, 20 Feb 2010 00:18:26 +0000
User-Agent: Microsoft-Entourage/12.10.0.080409
Date: Fri, 19 Feb 2010 16:18:23 -0800
Subject: Re: Release plans
From: Allen Wittenauer <awittenauer@linkedin.com>
To: <general@hadoop.apache.org>
Message-ID: <C7A46A4F.3233%awittenauer@linkedin.com>
Thread-Topic: Release plans
Thread-Index: AcqxwjaFG2b/fCDTjESXxU4jmgi6jg==
In-Reply-To: <4B7F20FA.2050704@apache.org>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit




On 2/19/10 3:38 PM, "Doug Cutting" <cutting@apache.org> wrote:
> Perhaps we can agree that, after 0.20, no deprecations will be removed
> until 1.0, that all pre-1.0 releases after 0.20 are now minor instead of
> major, that 0.20 was effectively 1.0 but we're not going to rename it,
> and that 1.0 will effectively be 2.0, etc.  Could that work?

FWIW: I'm still very much opposed to any 1.0 that isn't ABI-wire compatible
going forward.


From general-return-1072-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Feb 20 00:24:21 2010
Return-Path: <general-return-1072-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77482 invoked from network); 20 Feb 2010 00:24:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Feb 2010 00:24:21 -0000
Received: (qmail 25122 invoked by uid 500); 20 Feb 2010 00:24:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25044 invoked by uid 500); 20 Feb 2010 00:24:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25034 invoked by uid 99); 20 Feb 2010 00:24:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 20 Feb 2010 00:24:20 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 20 Feb 2010 00:24:13 +0000
Received: by pwi6 with SMTP id 6so624263pwi.35
        for <general@hadoop.apache.org>; Fri, 19 Feb 2010 16:23:53 -0800 (PST)
MIME-Version: 1.0
Received: by 10.142.55.8 with SMTP id d8mr7777723wfa.81.1266625432940; Fri, 19 
	Feb 2010 16:23:52 -0800 (PST)
In-Reply-To: <4B7F20FA.2050704@apache.org>
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com>
	 <c9118d4c1002181506l7ec9840dq3690158ebc70c02@mail.gmail.com>
	 <EFFFAD8D-1998-4C19-9BD3-D80B3B79577A@apache.org>
	 <7c962aed1002182036l7db697ahcdc581fde3619405@mail.gmail.com>
	 <dfe484f01002191344u27c3c24eoe5fb5c5aa5701658@mail.gmail.com>
	 <4B7F0985.3010000@apache.org>
	 <dfe484f01002191419g691e7651xe589a0a4a1307918@mail.gmail.com>
	 <4B7F157C.1090305@apache.org>
	 <dfe484f01002191514i5fe5b31cg1e884693737a3c93@mail.gmail.com>
	 <4B7F20FA.2050704@apache.org>
Date: Fri, 19 Feb 2010 16:23:52 -0800
Message-ID: <dfe484f01002191623i6b70021et9fea198575460a68@mail.gmail.com>
Subject: Re: Release plans
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Fri, Feb 19, 2010 at 3:38 PM, Doug Cutting <cutting@apache.org> wrote:
> Eli Collins wrote:
>>
>> Maybe we could get 21 behind
>> us and have a separate discussion covering those.
>
> I'm personally more interested in that discussion than in what's currentl=
y
> in the 0.21 branch, but others, like Stack, may reasonably be of the
> opposite persuasion. =A0So rather than putting one before the other, can =
we
> pursue both in parallel?
>

Don't want to hold up the release process, version, compatibility
discussion, or have it hold up work on the 21 release (most 21
blockers are probably trunk blockers so probably not much wasted
effort either way).

>> I was assuming 21 would be another minor release, didn't hear
>> otherwise when it was branched.
>
> We've never officially had a minor release. =A0All of our releases have b=
een
> either major or bugfix. =A0A minor release adds features but does not rem=
ove
> any deprecated APIs. =A0So making 0.21 a minor release would be a change =
in
> the rules.
>

Good to know, the versioning scheme (major.minor.point) outlined on
the wiki makes it seem like there's only been minor and point
releases.

Can we make a decision on basing 21 on the current branch and if it's
decided that 22 can't remove stuff that was in 20 we'll go back and do
the necessary additions on 21 and trunk? Suspect that decision will
take a lot more back and forth, but needs to conclude before 21 is
released.

Thanks,
Eli

From general-return-1073-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Feb 20 22:04:44 2010
Return-Path: <general-return-1073-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 7047 invoked from network); 20 Feb 2010 22:04:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Feb 2010 22:04:44 -0000
Received: (qmail 72570 invoked by uid 500); 20 Feb 2010 22:04:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72468 invoked by uid 500); 20 Feb 2010 22:04:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72458 invoked by uid 99); 20 Feb 2010 22:04:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 20 Feb 2010 22:04:42 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 74.125.92.26 as permitted sender)
Received: from [74.125.92.26] (HELO qw-out-2122.google.com) (74.125.92.26)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 20 Feb 2010 22:04:34 +0000
Received: by qw-out-2122.google.com with SMTP id 5so235871qwi.35
        for <general@hadoop.apache.org>; Sat, 20 Feb 2010 14:04:13 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:message-id:subject:from:to
         :content-type;
        bh=8qDzSNMS7BgeElDeSBeo23ssya2hi/XcfinHr5snJmE=;
        b=EdsduFLkr9n37168j/kgezO49aE9c9MNNXuObFr1Mhf1MwwUaUmeQ8EwFfb37fbbY6
         FicLdShrUxdyxrfW5tZjkKVbdHZHArhriRJO6HHNDmoCJJCvikQ7n6feozymHESJFAXz
         v0TXkOl80lu20x/vPvsJd+8RIE+8QDvNyP3/g=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=Ob2xbtaHDUG7aUMVnXkPuPQW7ip+kuIJ8ybNKFvJES6Z/R84XYnBuVbPSMFxFQJnX7
         qJwqPLx8bYppvYGn0Plj9630feLqoZ3Kg8gefbaxCFt0K/M6GgfsT1paLLBAVLl3dVmZ
         D269b3NnyQNj/czTaYCYGDxPgWzHe24opGEmI=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
Received: by 10.229.107.29 with SMTP id z29mr629458qco.42.1266703450479; Sat, 
	20 Feb 2010 14:04:10 -0800 (PST)
In-Reply-To: <dfe484f01002191623i6b70021et9fea198575460a68@mail.gmail.com>
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com>
	 <EFFFAD8D-1998-4C19-9BD3-D80B3B79577A@apache.org>
	 <7c962aed1002182036l7db697ahcdc581fde3619405@mail.gmail.com>
	 <dfe484f01002191344u27c3c24eoe5fb5c5aa5701658@mail.gmail.com>
	 <4B7F0985.3010000@apache.org>
	 <dfe484f01002191419g691e7651xe589a0a4a1307918@mail.gmail.com>
	 <4B7F157C.1090305@apache.org>
	 <dfe484f01002191514i5fe5b31cg1e884693737a3c93@mail.gmail.com>
	 <4B7F20FA.2050704@apache.org>
	 <dfe484f01002191623i6b70021et9fea198575460a68@mail.gmail.com>
Date: Sat, 20 Feb 2010 14:04:10 -0800
X-Google-Sender-Auth: d2dbac5606b200cd
Message-ID: <7c962aed1002201404q71c4b820x1fdf16d8f473be88@mail.gmail.com>
Subject: Re: Release plans
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Fri, Feb 19, 2010 at 4:23 PM, Eli Collins <eli@cloudera.com> wrote:
> Can we make a decision on basing 21 on the current branch and if it's
> decided that 22 can't remove stuff that was in 20 we'll go back and do
> the necessary additions on 21 and trunk? Suspect that decision will
> take a lot more back and forth, but needs to conclude before 21 is
> released.
>

Lets.

Regards 0.21/current-branch release, as has been suggested above,
first we need to figure the release master.  No release master, no
release.  If we have a release master, then I suggest we vote on
current branch being released as 0.21 as soon as the blockers are
cleared.

I don't think we need muddy the above vote with whether or not 0.21
maintains API combatibility with 0.20.  IMO, it must (because Y! want
to have the 0.20 API in place when January 2011 rolls around).  This
makes 0.21 a "minor" release -- something we've not done before (For
the record, I also had a misunderstanding that what we were doing up
to this was major and patch only).  So, part of the release process
would involve ensuring no removed deprecations, etc.

As DC has been saying, this requirement that releases between now and
January 2011 not change APIs makes 0.20 retroactively into a "major"
release.  0.20 is the release where major shifted left in our
versioning scheme and minor releases came into play.  0.21 and 0.22
will be minor releases. Can we just acknowledge this fact, that there
was a step at version 0.20, update the wiki around versioning -- its
currently wrong anyways as Elis' points out -- and just move on?
Going back and calling 0.20 a 1.0 seems more apt to create confusion
and besides, I'm with Allen that hadoop 1.0 needs wire compatibility
before the 1.0 can roll around.

Thanks,
St.Ack
P.S. +1 on branching as soon as avro and security are in, etc.

From general-return-1074-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Feb 21 02:54:05 2010
Return-Path: <general-return-1074-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 577 invoked from network); 21 Feb 2010 02:54:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Feb 2010 02:54:05 -0000
Received: (qmail 4693 invoked by uid 500); 21 Feb 2010 02:54:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4553 invoked by uid 500); 21 Feb 2010 02:54:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4543 invoked by uid 99); 21 Feb 2010 02:54:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 21 Feb 2010 02:54:02 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jason.hadoop@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 21 Feb 2010 02:53:52 +0000
Received: by pwi6 with SMTP id 6so1356703pwi.35
        for <general@hadoop.apache.org>; Sat, 20 Feb 2010 18:53:32 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=yK+bV9yYZn3Mi0VOqZyZaH1CfV6qlLrNRGdiVFzukWE=;
        b=TtkAW24pDmOzuXVH6hmsiP0pWGKrji1/Ue747pgj6nTOOQpVw4uvnSd38D/Fmc+THp
         UFtqLgpoy2tMUDeZ7zsRb/f9lieTnDeqW44iTykPfM7X2YSIyjKoMSzr3cCqd7kSGW4K
         HhTPBDdI0IdhaitRvFrtQEVlL8gvUznEdicYw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=sWIvsxfLcbNf4aBKUZMUdXS34ds+aCBcpSj0iixQXHCFAFvi5AHrx08WFnJCEs18T4
         QeC4qpblll/VuvNSIHfM+CxeDrKdX21c4zcHGnC3ZNNNKlv2ot8nuyw3wRp8g4XdD+au
         GXnTNWe++VCg5uvJipqg8qAJWYuLiwv8d1Izs=
MIME-Version: 1.0
Received: by 10.141.125.11 with SMTP id c11mr3224133rvn.34.1266720812111; Sat, 
	20 Feb 2010 18:53:32 -0800 (PST)
In-Reply-To: <e497591c1002040025l4fa56b73oa0a9bbb5dcc5a809@mail.gmail.com>
References: <e497591c1002040025l4fa56b73oa0a9bbb5dcc5a809@mail.gmail.com>
Date: Sat, 20 Feb 2010 18:53:32 -0800
Message-ID: <314098691002201853x669559bcnd44a6b5822bd461a@mail.gmail.com>
Subject: =?UTF-8?B?UmU6IGphdmEuaW8uSU9FeGNlcHRpb246IENhbm5vdCBvcGVuIGZpbGVuYW1lIC91c2VyLw==?=
	=?UTF-8?B?cm9vdC/vv71z77+9dO+/vWXvv71w77+9Me+/vS/vv71w77+9Ye+/vXLvv71077+9Le+/vTDvv70w77+9?=
	=?UTF-8?B?MO+/vTDvv70w?=
From: Jason Venner <jason.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

something is not doing character set conversion correctly somewhere in
your code path.
If the text was passed in the email correctly each ascii letter is
prefixed by 3 bytes
0xEF,0xBF,0xBD, which is the encoding for \u0FFFD, the utf 8 character
that utf8 decoders use to replace an illegal utf8 byte sequence. see
http://en.wikipedia.org/wiki/Unicode_Specials

It looks like the file data is written in a double byte format,
perhaps utf16 and the reader is not able to correctly recognize the
character encoding, and the first byte of each double byte pair is
being replaced by the replacement character \u0FFFD, while the second
byte, the actual ascii character in your path is passed forward by the
stream reader.

On Thu, Feb 4, 2010 at 12:25 AM, Harshit Kumar <hkumar.arora@gmail.com> wro=
te:
>
> Hi
>
> I dont understand the reason for this error.
>
> java.io.IOException: Cannot open filename
> /user/root/=EF=BF=BDs=EF=BF=BDt=EF=BF=BDe=EF=BF=BDp=EF=BF=BD1=EF=BF=BD/=
=EF=BF=BDp=EF=BF=BDa=EF=BF=BDr=EF=BF=BDt=EF=BF=BD-=EF=BF=BD0=EF=BF=BD0=EF=
=BF=BD0=EF=BF=BD0=EF=BF=BD0 at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1=
394)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.(DFSClient.java:1385) =
at
> org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:338) at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.j=
ava:171)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:359) at
> org.bike.MakeNPairReduce.reduce(MakeNPairReduce.java:40) at
> org.bike.MakeNPairReduce.reduce(MakeNPairReduce.java:1) at
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436) at
> org.apache.hadoop.mapred.Child.main(Child.java:158)
>
>
> I have a code that scans a folder step0 to find name of files generated i=
n
> the previous map-reduce phase. Then create another file with the entries =
for
> ex:
> if scanning finds that there are 2 files produced by 1st map-reduce phase=
,
> then new created file will have 2 entries step1/part00000 and
> step1/part00001 i.e. one entry for each file.
>
> Now, when I read this file in another map-reduce job, each line is read a=
s
> /user/root/=EF=BF=BDs=EF=BF=BDt=EF=BF=BDe=EF=BF=BDp=EF=BF=BD1=EF=BF=BD/=
=EF=BF=BDp=EF=BF=BDa=EF=BF=BDr=EF=BF=BDt=EF=BF=BD-=EF=BF=BD0=EF=BF=BD0=EF=
=BF=BD0=EF=BF=BD0=EF=BF=BD0 . What it seems like, a string
> inserted by my code, when read by FSDataInputStream prefix each character=
 of
> the string by a question mark (?). Why is that so?
>
> The file name part-00000 do exist inside folder step1, but reading this
> filename, /user/root/=EF=BF=BDs=EF=BF=BDt=EF=BF=BDe=EF=BF=BDp=EF=BF=BD1=
=EF=BF=BD/=EF=BF=BDp=EF=BF=BDa=EF=BF=BDr=EF=BF=BDt=EF=BF=BD-=EF=BF=BD0=EF=
=BF=BD0=EF=BF=BD0=EF=BF=BD0=EF=BF=BD0 , throws IOException
> which I can undersand that there is no such filename, but why are these ?=
's
> infiltraded before each letter.
>
> Really appreciate if some one can help me solve this riddle?
>
> Thanks and Regards
> H. Kumar
> skype: harshit900
> Blog: http://harshitkumar.wordpress.com
> Website: http:/kumarharmuscat.tripod.com



--
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=3Djewlerymall
www.prohadoopbook.com a community for Hadoop Professionals

From general-return-1075-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 22 02:43:51 2010
Return-Path: <general-return-1075-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38497 invoked from network); 22 Feb 2010 02:43:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Feb 2010 02:43:51 -0000
Received: (qmail 6441 invoked by uid 500); 22 Feb 2010 02:43:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6370 invoked by uid 500); 22 Feb 2010 02:43:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6360 invoked by uid 99); 22 Feb 2010 02:43:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Feb 2010 02:43:48 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 209.85.217.212 as permitted sender)
Received: from [209.85.217.212] (HELO mail-gx0-f212.google.com) (209.85.217.212)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Feb 2010 02:43:37 +0000
Received: by gxk4 with SMTP id 4so2486055gxk.5
        for <general@hadoop.apache.org>; Sun, 21 Feb 2010 18:43:17 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=2fboe8UhvJDnrdaH31NqJcmm+kLqSH7vEg27t5YH2y4=;
        b=mQEmYLkQwHP/GIogjMIMgKZLDJry1pGiGzSpDHeUhtqWcnZ5N7gcJ6jALQRoC/9HlV
         c+9IpY8cS3VUNNjJnKREnWTOCb9dR1EzCZDB5Uw5pfS3tuYnk68/hf+A/F/KybEcEINn
         1mJkbNPF6XWbVYpczabGHGwf8go8h1NXBmCQ4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=umqDE45PqQf0Wgr6WWu7LubMe4R7QbSpekl6alXgOMW/uyUw/r8PXHADfij0hh4+wG
         PgUM/7FhfVh2rlQaqdrz3L9tuSFUi01W135e1g4ZOhtY03YCBzlxn5PEyk4w4FU4ew5T
         TTogTFER5s4nyECKXzU+H6wW4U2yUVCXXPHBM=
MIME-Version: 1.0
Received: by 10.150.184.14 with SMTP id h14mr3928840ybf.51.1266806596942; Sun, 
	21 Feb 2010 18:43:16 -0800 (PST)
In-Reply-To: <314098691002201853x669559bcnd44a6b5822bd461a@mail.gmail.com>
References: <e497591c1002040025l4fa56b73oa0a9bbb5dcc5a809@mail.gmail.com>
	 <314098691002201853x669559bcnd44a6b5822bd461a@mail.gmail.com>
Date: Sun, 21 Feb 2010 18:43:16 -0800
Message-ID: <4aa34eb71002211843j74a0f963xb2118c184f02b022@mail.gmail.com>
Subject: =?UTF-8?B?UmU6IGphdmEuaW8uSU9FeGNlcHRpb246IENhbm5vdCBvcGVuIGZpbGVuYW1lIC91c2VyLw==?=
	=?UTF-8?B?cm9vdC/vv71z77+9dO+/vWXvv71w77+9Me+/vS/vv71w77+9Ye+/vXLvv71077+9Le+/vTDvv70w77+9?=
	=?UTF-8?B?MO+/vTDvv70w?=
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd6ed6a35bc310480276530

--000e0cd6ed6a35bc310480276530
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Is this, by any chance, related to

http://issues.apache.org/jira/browse/HDFS-983?

You should try if you can recreate your problem with the patch from
http://issues.apache.org/jira/browse/HADOOP-6522?

thanks,
dhruba


On Sat, Feb 20, 2010 at 6:53 PM, Jason Venner <jason.hadoop@gmail.com>wrote=
:

> something is not doing character set conversion correctly somewhere in
> your code path.
> If the text was passed in the email correctly each ascii letter is
> prefixed by 3 bytes
> 0xEF,0xBF,0xBD, which is the encoding for \u0FFFD, the utf 8 character
> that utf8 decoders use to replace an illegal utf8 byte sequence. see
> http://en.wikipedia.org/wiki/Unicode_Specials
>
> It looks like the file data is written in a double byte format,
> perhaps utf16 and the reader is not able to correctly recognize the
> character encoding, and the first byte of each double byte pair is
> being replaced by the replacement character \u0FFFD, while the second
> byte, the actual ascii character in your path is passed forward by the
> stream reader.
>
> On Thu, Feb 4, 2010 at 12:25 AM, Harshit Kumar <hkumar.arora@gmail.com>
> wrote:
> >
> > Hi
> >
> > I dont understand the reason for this error.
> >
> > java.io.IOException: Cannot open filename
> > /user/root/=EF=BF=BDs=EF=BF=BDt=EF=BF=BDe=EF=BF=BDp=EF=BF=BD1=EF=BF=BD/=
=EF=BF=BDp=EF=BF=BDa=EF=BF=BDr=EF=BF=BDt=EF=BF=BD-=EF=BF=BD0=EF=BF=BD0=EF=
=BF=BD0=EF=BF=BD0=EF=BF=BD0 at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1=
394)
> > at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.(DFSClient.java:1385=
)
> at
> > org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:338) at
> >
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.j=
ava:171)
> > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:359) at
> > org.bike.MakeNPairReduce.reduce(MakeNPairReduce.java:40) at
> > org.bike.MakeNPairReduce.reduce(MakeNPairReduce.java:1) at
> > org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:436) at
> > org.apache.hadoop.mapred.Child.main(Child.java:158)
> >
> >
> > I have a code that scans a folder step0 to find name of files generated
> in
> > the previous map-reduce phase. Then create another file with the entrie=
s
> for
> > ex:
> > if scanning finds that there are 2 files produced by 1st map-reduce
> phase,
> > then new created file will have 2 entries step1/part00000 and
> > step1/part00001 i.e. one entry for each file.
> >
> > Now, when I read this file in another map-reduce job, each line is read
> as
> > /user/root/=EF=BF=BDs=EF=BF=BDt=EF=BF=BDe=EF=BF=BDp=EF=BF=BD1=EF=BF=BD/=
=EF=BF=BDp=EF=BF=BDa=EF=BF=BDr=EF=BF=BDt=EF=BF=BD-=EF=BF=BD0=EF=BF=BD0=EF=
=BF=BD0=EF=BF=BD0=EF=BF=BD0 . What it seems like, a
> string
> > inserted by my code, when read by FSDataInputStream prefix each charact=
er
> of
> > the string by a question mark (?). Why is that so?
> >
> > The file name part-00000 do exist inside folder step1, but reading this
> > filename, /user/root/=EF=BF=BDs=EF=BF=BDt=EF=BF=BDe=EF=BF=BDp=EF=BF=BD1=
=EF=BF=BD/=EF=BF=BDp=EF=BF=BDa=EF=BF=BDr=EF=BF=BDt=EF=BF=BD-=EF=BF=BD0=EF=
=BF=BD0=EF=BF=BD0=EF=BF=BD0=EF=BF=BD0 , throws
> IOException
> > which I can undersand that there is no such filename, but why are these
> ?'s
> > infiltraded before each letter.
> >
> > Really appreciate if some one can help me solve this riddle?
> >
> > Thanks and Regards
> > H. Kumar
> > skype: harshit900
> > Blog: http://harshitkumar.wordpress.com
> > Website: http:/kumarharmuscat.tripod.com
>
>
>
> --
> Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> http://www.amazon.com/dp/1430219424?tag=3Djewlerymall
> www.prohadoopbook.com a community for Hadoop Professionals
>



--=20
Connect to me at http://www.facebook.com/dhruba

--000e0cd6ed6a35bc310480276530--

From general-return-1076-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 22 04:21:58 2010
Return-Path: <general-return-1076-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78131 invoked from network); 22 Feb 2010 04:21:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Feb 2010 04:21:58 -0000
Received: (qmail 69754 invoked by uid 500); 22 Feb 2010 04:21:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69606 invoked by uid 500); 22 Feb 2010 04:21:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69591 invoked by uid 99); 22 Feb 2010 04:21:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Feb 2010 04:21:55 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yangxiao9901@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Feb 2010 04:21:47 +0000
Received: by pwi6 with SMTP id 6so2042010pwi.35
        for <general@hadoop.apache.org>; Sun, 21 Feb 2010 20:20:25 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=TtJJwHs8r6pJwDKQVEFSDN8e9qK9Vz6AYHRvgsQn+18=;
        b=tgndu/HHVBEcOw07LgBwzKmcGAIf1DW+ld5WjF5jRlWNiZJ5NOEKia1IWF0sFwwhMR
         9bgZuQ7aYxtQ478itu6yyyAqDldRDSHrmhzSHnYnhT20dIGiYM0XodaOPtI4fnbnXJxg
         5KKXcrJHDGGKVhNPN5KzSG6k44hKXQG9c5DR0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=QtuT8rHcYn92GZdMrMaMuNxXM7hpnxh+q3KE63T4VyM75Ve+uL+HSBsxv6N+3xTIrU
         M/dUp/B6auoYulJGOg8wV9JlpAfMbkO6O/9cjD5e1y4qMLxTc406BT6ppEvzYQerMAOB
         2oekEzQocKpPFYYr5ORTatMXR9wZHXj7f2zPw=
MIME-Version: 1.0
Received: by 10.141.100.20 with SMTP id c20mr8656512rvm.143.1266812425833; 
	Sun, 21 Feb 2010 20:20:25 -0800 (PST)
Date: Mon, 22 Feb 2010 12:20:25 +0800
Message-ID: <5a921af21002212020u5677fd1tabc549322787c41f@mail.gmail.com>
Subject: Namenode data loss
From: xiao yang <yangxiao9901@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

My namenode failed, and I can't find data back on this machine.
I have look into the data directory in secondary namenode, but the
"name" directory is empty.
What should I do?

Thanks!
Xiao

From general-return-1077-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 22 04:56:54 2010
Return-Path: <general-return-1077-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87086 invoked from network); 22 Feb 2010 04:56:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Feb 2010 04:56:54 -0000
Received: (qmail 95107 invoked by uid 500); 22 Feb 2010 04:56:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95004 invoked by uid 500); 22 Feb 2010 04:56:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94994 invoked by uid 99); 22 Feb 2010 04:56:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Feb 2010 04:56:51 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jaybooth@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Feb 2010 04:56:41 +0000
Received: by vws12 with SMTP id 12so408970vws.35
        for <general@hadoop.apache.org>; Sun, 21 Feb 2010 20:55:20 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:references:message-id:from:to
         :in-reply-to:content-type:content-transfer-encoding:x-mailer
         :mime-version:subject:date:cc;
        bh=CFDimYN8TdznufT58pRB0tXuQwzb0parylGlcJUAwUo=;
        b=LzqSTUVutLhcCIsN0ai1YDE45bd61OfNburzz3j7pRF1l7V12f0dBXCMpLHmeet8Cb
         v567haAqDpABeS+jLdHiPkI+gIo6YE3cCUSLOTBPca/TkkFbPOa2FSMdI1tl4MSe+D8B
         Fq2puqyjUq+19M58RHjk+LY3sLJziXOlblT+g=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=references:message-id:from:to:in-reply-to:content-type
         :content-transfer-encoding:x-mailer:mime-version:subject:date:cc;
        b=f8RB4+TUWgJlwDBVKt93lV+RoTSlBQFukRFSZv+alljEql3AdAVj13Pm/yaBoLGj/R
         9sNu/4eF62tGvemOnvjzNkkEX2wM18C4XHgXC0GS+tJYtGyaj7HApZ3E/tUVvz7qxxdq
         HyGZpqiX8uTLVcSUoC0VN382i8wBN4CH0fkjg=
Received: by 10.220.123.21 with SMTP id n21mr7154255vcr.134.1266814520245;
        Sun, 21 Feb 2010 20:55:20 -0800 (PST)
Received: from ?10.146.181.200? ([32.137.184.37])
        by mx.google.com with ESMTPS id 22sm28934497vws.14.2010.02.21.20.55.17
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Sun, 21 Feb 2010 20:55:18 -0800 (PST)
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com> <EFFFAD8D-1998-4C19-9BD3-D80B3B79577A@apache.org> <7c962aed1002182036l7db697ahcdc581fde3619405@mail.gmail.com> <dfe484f01002191344u27c3c24eoe5fb5c5aa5701658@mail.gmail.com> <4B7F0985.3010000@apache.org> <dfe484f01002191419g691e7651xe589a0a4a1307918@mail.gmail.com> <4B7F157C.1090305@apache.org> <dfe484f01002191514i5fe5b31cg1e884693737a3c93@mail.gmail.com> <4B7F20FA.2050704@apache.org> <dfe484f01002191623i6b70021et9fea198575460a68@mail.gmail.com> <7c962aed1002201404q71c4b820x1fdf16d8f473be88@mail.gmail.com>
Message-Id: <63C0685C-E1BF-4223-91A8-4F58266A6B5E@gmail.com>
From: Jay Booth <jaybooth@gmail.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <7c962aed1002201404q71c4b820x1fdf16d8f473be88@mail.gmail.com>
Content-Type: text/plain;
	charset=us-ascii;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit
X-Mailer: iPhone Mail (5H11)
Mime-Version: 1.0 (iPhone Mail 5H11)
Subject: Re: Release plans
Date: Sun, 21 Feb 2010 23:55:09 -0500
Cc: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

Well, since someone has to get the ball rolling as far as release  
masters, I'll nominate Stack and/or someone hbase related for 0.21  
with the primary goal of being "soon"?  They get a big win from append  
and others will gain from the expanded mapreduce lib, better  
schedulers, etc. There are a lot of new features and some major  
changes (project split) already in the 0.21 branch, so IMO it's worth  
considering a release with minimal backports, rather than make binding  
decisions about 0.22 before 0.21 is even in the wild.

-Jay

PS sorry Stack

On Feb 20, 2010, at 5:04 PM, Stack <stack@duboce.net> wrote:

> On Fri, Feb 19, 2010 at 4:23 PM, Eli Collins <eli@cloudera.com> wrote:
>> Can we make a decision on basing 21 on the current branch and if it's
>> decided that 22 can't remove stuff that was in 20 we'll go back and  
>> do
>> the necessary additions on 21 and trunk? Suspect that decision will
>> take a lot more back and forth, but needs to conclude before 21 is
>> released.
>>
>
> Lets.
>
> Regards 0.21/current-branch release, as has been suggested above,
> first we need to figure the release master.  No release master, no
> release.  If we have a release master, then I suggest we vote on
> current branch being released as 0.21 as soon as the blockers are
> cleared.
>
> I don't think we need muddy the above vote with whether or not 0.21
> maintains API combatibility with 0.20.  IMO, it must (because Y! want
> to have the 0.20 API in place when January 2011 rolls around).  This
> makes 0.21 a "minor" release -- something we've not done before (For
> the record, I also had a misunderstanding that what we were doing up
> to this was major and patch only).  So, part of the release process
> would involve ensuring no removed deprecations, etc.
>
> As DC has been saying, this requirement that releases between now and
> January 2011 not change APIs makes 0.20 retroactively into a "major"
> release.  0.20 is the release where major shifted left in our
> versioning scheme and minor releases came into play.  0.21 and 0.22
> will be minor releases. Can we just acknowledge this fact, that there
> was a step at version 0.20, update the wiki around versioning -- its
> currently wrong anyways as Elis' points out -- and just move on?
> Going back and calling 0.20 a 1.0 seems more apt to create confusion
> and besides, I'm with Allen that hadoop 1.0 needs wire compatibility
> before the 1.0 can roll around.
>
> Thanks,
> St.Ack
> P.S. +1 on branching as soon as avro and security are in, etc.

From general-return-1078-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 22 08:41:03 2010
Return-Path: <general-return-1078-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 71191 invoked from network); 22 Feb 2010 08:41:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Feb 2010 08:41:03 -0000
Received: (qmail 37167 invoked by uid 500); 22 Feb 2010 08:41:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37092 invoked by uid 500); 22 Feb 2010 08:41:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37082 invoked by uid 99); 22 Feb 2010 08:41:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Feb 2010 08:41:01 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [145.100.8.32] (HELO planck.ka.sara.nl) (145.100.8.32)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Feb 2010 08:40:54 +0000
Received: from planck.ka.sara.nl ([145.100.8.32]) by planck.ka.sara.nl
 ([145.100.8.32]) with mapi; Mon, 22 Feb 2010 09:40:31 +0100
From: Evert Lammerts <Evert.Lammerts@sara.nl>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 22 Feb 2010 09:40:30 +0100
Subject: RE: Release plans
Thread-Topic: Release plans
Thread-Index: Acqze3xOn+NAZdG3TmO0bwZQzKBd/QAGu78w
Message-ID: <ADF94D8555C7A246B86A633685E0178AB867CCCDEB@planck.ka.sara.nl>
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com>
 <EFFFAD8D-1998-4C19-9BD3-D80B3B79577A@apache.org>
 <7c962aed1002182036l7db697ahcdc581fde3619405@mail.gmail.com>
 <dfe484f01002191344u27c3c24eoe5fb5c5aa5701658@mail.gmail.com>
 <4B7F0985.3010000@apache.org>
 <dfe484f01002191419g691e7651xe589a0a4a1307918@mail.gmail.com>
 <4B7F157C.1090305@apache.org>
 <dfe484f01002191514i5fe5b31cg1e884693737a3c93@mail.gmail.com>
 <4B7F20FA.2050704@apache.org>
 <dfe484f01002191623i6b70021et9fea198575460a68@mail.gmail.com>
 <7c962aed1002201404q71c4b820x1fdf16d8f473be88@mail.gmail.com>
 <63C0685C-E1BF-4223-91A8-4F58266A6B5E@gmail.com>
In-Reply-To: <63C0685C-E1BF-4223-91A8-4F58266A6B5E@gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/signed; protocol="application/x-pkcs7-signature";
	micalg=SHA1; boundary="----=_NextPart_000_003A_01CAB3A3.12F57DC0"
MIME-Version: 1.0

------=_NextPart_000_003A_01CAB3A3.12F57DC0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit

I'm sorry for breaking into your discussion as an outsider, but I'm very
curious about the security features you are planning to roll out in March.
Where can I find information about this?

Best regards,
Evert Lammerts

> -----Original Message-----
> From: Jay Booth [mailto:jaybooth@gmail.com]
> Sent: maandag 22 februari 2010 5:55
> To: general@hadoop.apache.org
> Cc: general@hadoop.apache.org
> Subject: Re: Release plans
> 
> Well, since someone has to get the ball rolling as far as release
> masters, I'll nominate Stack and/or someone hbase related for 0.21
> with the primary goal of being "soon"?  They get a big win from append
> and others will gain from the expanded mapreduce lib, better
> schedulers, etc. There are a lot of new features and some major
> changes (project split) already in the 0.21 branch, so IMO it's worth
> considering a release with minimal backports, rather than make binding
> decisions about 0.22 before 0.21 is even in the wild.
> 
> -Jay
> 
> PS sorry Stack
> 
> On Feb 20, 2010, at 5:04 PM, Stack <stack@duboce.net> wrote:
> 
> > On Fri, Feb 19, 2010 at 4:23 PM, Eli Collins <eli@cloudera.com>
> wrote:
> >> Can we make a decision on basing 21 on the current branch and if
> it's
> >> decided that 22 can't remove stuff that was in 20 we'll go back and
> >> do
> >> the necessary additions on 21 and trunk? Suspect that decision will
> >> take a lot more back and forth, but needs to conclude before 21 is
> >> released.
> >>
> >
> > Lets.
> >
> > Regards 0.21/current-branch release, as has been suggested above,
> > first we need to figure the release master.  No release master, no
> > release.  If we have a release master, then I suggest we vote on
> > current branch being released as 0.21 as soon as the blockers are
> > cleared.
> >
> > I don't think we need muddy the above vote with whether or not 0.21
> > maintains API combatibility with 0.20.  IMO, it must (because Y! want
> > to have the 0.20 API in place when January 2011 rolls around).  This
> > makes 0.21 a "minor" release -- something we've not done before (For
> > the record, I also had a misunderstanding that what we were doing up
> > to this was major and patch only).  So, part of the release process
> > would involve ensuring no removed deprecations, etc.
> >
> > As DC has been saying, this requirement that releases between now and
> > January 2011 not change APIs makes 0.20 retroactively into a "major"
> > release.  0.20 is the release where major shifted left in our
> > versioning scheme and minor releases came into play.  0.21 and 0.22
> > will be minor releases. Can we just acknowledge this fact, that there
> > was a step at version 0.20, update the wiki around versioning -- its
> > currently wrong anyways as Elis' points out -- and just move on?
> > Going back and calling 0.20 a 1.0 seems more apt to create confusion
> > and besides, I'm with Allen that hadoop 1.0 needs wire compatibility
> > before the 1.0 can roll around.
> >
> > Thanks,
> > St.Ack
> > P.S. +1 on branching as soon as avro and security are in, etc.

------=_NextPart_000_003A_01CAB3A3.12F57DC0
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIPUDCCBDIw
ggMaoAMCAQICAQEwDQYJKoZIhvcNAQEFBQAwezELMAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0
ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0
ZWQxITAfBgNVBAMMGEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczAeFw0wNDAxMDEwMDAwMDBaFw0y
ODEyMzEyMzU5NTlaMHsxCzAJBgNVBAYTAkdCMRswGQYDVQQIDBJHcmVhdGVyIE1hbmNoZXN0ZXIx
EDAOBgNVBAcMB1NhbGZvcmQxGjAYBgNVBAoMEUNvbW9kbyBDQSBMaW1pdGVkMSEwHwYDVQQDDBhB
QUEgQ2VydGlmaWNhdGUgU2VydmljZXMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC+
QJ30buHqdoccTUVEjr5GyIMGncEq/hgfjuQC+vOrXVCKFjELmgbQxXAizUktVGPMtm5oRgtT6stM
JMC8ck7q8RWu9FSaEgrDerIzYOLaiVXzIljz3tzP74OGooyUT59o8piQRoQnx3a/48w1LIteB2Rl
gsBIsKiR+WGfdiBQqJHHZrXreGIDVvCKGhPqMaMeoJn9OPb2JzJYbwf1a7j7FCuvt6rM1mNfc4za
BZmoOKjLF3g2UazpnvR4Oo3PD9lC4pgMqy+fDgHe75+ZSfEt36x0TRuYtUfF5SnR+ZAYx2KcvoPH
Jns+iiXHwN2d5jVoECCdj9je0sOEnA1e6C/JAgMBAAGjgcAwgb0wHQYDVR0OBBYEFKARCiM+lvEH
7OKvKe+CpX/QMKS0MA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MHsGA1UdHwR0MHIw
OKA2oDSGMmh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0FBQUNlcnRpZmljYXRlU2VydmljZXMuY3Js
MDagNKAyhjBodHRwOi8vY3JsLmNvbW9kby5uZXQvQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmww
DQYJKoZIhvcNAQEFBQADggEBAAhW/ALwm+j/pPrWe8ZEgM5PxMX2AFjMpra8FEloBHbo5u5d7AIP
YNaNUBhPJk4B4+awpe6/vHRUQb/9/BK4x09a9IlgBX9gtwVK8/bxwr/EuXSGti19a8zS80bdL8bg
asPDNAMsfZbdWsIOpwqZwQWLqwwv81w6z2w3VQmH3lNAbFjv/LarZW4E9hvcPOBaFcae2fFZSDAh
ZQNs7Okhc+ybA6HgN62gFRiP+roCzqcsqRATLNTlCCarIpdg+JBedNSimlO98qlo4KJuwtdssaMP
nr/raOdW8q7y4ys4OgmBtWuF174t7T8at7Jj4vViLILUagBBUPE5g5+V6TaWmG4wggTdMIIDxaAD
AgECAhBxkvvmGV+sTRKFdHE0ohinMA0GCSqGSIb3DQEBBQUAMHsxCzAJBgNVBAYTAkdCMRswGQYD
VQQIDBJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcMB1NhbGZvcmQxGjAYBgNVBAoMEUNvbW9k
byBDQSBMaW1pdGVkMSEwHwYDVQQDDBhBQUEgQ2VydGlmaWNhdGUgU2VydmljZXMwHhcNMDQwMTAx
MDAwMDAwWhcNMjgxMjMxMjM1OTU5WjCBrjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYD
VQQHEw5TYWx0IExha2UgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYD
VQQLExhodHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1VU0VSRmlyc3QtQ2xp
ZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBFbWFpbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBALI5haTyfatBO2JGN67NwWB1vDll+UoaR6K5zEjMapjVTTUZuaRC5c5J4oovHnzSMQfHTrSD
ZJ0uKdWiZMSFvYVRNXmkTmiQexx6pJKoF/KYFfKTzMmkMpW7DE8wvZigC4vlbhuiRvp4vKJvq1le
pS/Pytptqi/rrKGzaqq3Lmc1i3nhHmmI4uZGzaCl6r4LznY6eg6b6vzaJ1s9cx8i5khhxkzzabGo
Lhu21DEgLLyCio6kDqXXiUP8FlqvHXHXEVnauocNr/rz4cLwpMVnjNbWVDreCqS6A3ezZcj9HtN0
YqoYymiTHqGFfvVHZcv4TVcodNI0/zC27vZiMBSMLOsCAwEAAaOCAScwggEjMB8GA1UdIwQYMBaA
FKARCiM+lvEH7OKvKe+CpX/QMKS0MB0GA1UdDgQWBBSJgmd9xJ0mcABLtFBIfN49rgRufTAOBgNV
HQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUH
AwQwEQYDVR0gBAowCDAGBgRVHSAAMHsGA1UdHwR0MHIwOKA2oDSGMmh0dHA6Ly9jcmwuY29tb2Rv
Y2EuY29tL0FBQUNlcnRpZmljYXRlU2VydmljZXMuY3JsMDagNKAyhjBodHRwOi8vY3JsLmNvbW9k
by5uZXQvQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwEQYJYIZIAYb4QgEBBAQDAgEGMA0GCSqG
SIb3DQEBBQUAA4IBAQCdlcs8uH6lCcQevwvCx3aOOTyUxhCqTwzJ4KuEXYlU4GU7820cfDcsJVRf
liH8N4SRnRXcFE+Bz1Qda2xFYMct+ZdRTPlmyjyggoymyPDi6dRK+ew/VsnddozDggFPbADzHhph
dARHA6nGQFeRvGUixSdnT1fbZFrZjR+6hi/0Bq6cae3p9M8pF9jgSp8aIC+XTFG7RgfEijdOIOMJ
MWjHnsSLneh+EbwyaBCWEZhE2CpRYE2I63Q630MGMsg5Vow6EVLTQaRDA/Tt7zMn2zngFE4mydj1
OeKJuJNdtykmQeqzm66D/Hd1yujKtf7iZUpjPkTE0MNeh3OpmByvfxV/MIIGNTCCBR2gAwIBAgIR
APUHLR99vIaqc6YYFoh++L8wDQYJKoZIhvcNAQEFBQAwga4xCzAJBgNVBAYTAlVTMQswCQYDVQQI
EwJVVDEXMBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNVBAoTFVRoZSBVU0VSVFJVU1QgTmV0
d29yazEhMB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3QuY29tMTYwNAYDVQQDEy1VVE4tVVNF
UkZpcnN0LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1haWwwHhcNMTAwMjA4MDAwMDAwWhcN
MTEwMjA4MjM1OTU5WjCB4DE1MDMGA1UECxMsQ29tb2RvIFRydXN0IE5ldHdvcmsgLSBQRVJTT05B
IE5PVCBWQUxJREFURUQxRjBEBgNVBAsTPVRlcm1zIGFuZCBDb25kaXRpb25zIG9mIHVzZTogaHR0
cDovL3d3dy5jb21vZG8ubmV0L3JlcG9zaXRvcnkxHzAdBgNVBAsTFihjKTIwMDMgQ29tb2RvIExp
bWl0ZWQxFzAVBgNVBAMTDkV2ZXJ0IExhbW1lcnRzMSUwIwYJKoZIhvcNAQkBFhZldmVydC5sYW1t
ZXJ0c0BzYXJhLm5sMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtYAj4sXuBqEwPxYY
BMkYLPGrnqT35USIERoWs26xtSY+dLVc7HqaHDZ1lU5m9wFx/Jqph1urlRuAcCj03kGSS23Ta6kt
HTRMYRCHI3nOLfEEbH4POT+wDytORUHNeEKp/PrBn1kmKk+dk+bJI7MNdy2XkiEeO+OqKmLiBGkT
glIxngOeqgqR2IvmVv2hLAyzq8Ax20inwveZsDMa7QdQLSVUaX1kSRSihwz4Jw9X/K+6oA3ZLGp9
pZYnFHBNTHM2DR/C7dNwQgbZS7TY/Jb1kObYNhwD2JzzJlkoe0blR17MeJkF7/+Y414j0AdPFB7I
ggZ4Tm7Maheer9cIQSsvIQIDAQABo4ICGDCCAhQwHwYDVR0jBBgwFoAUiYJnfcSdJnAAS7RQSHze
Pa4Ebn0wHQYDVR0OBBYEFNTyUAqBNg9JXuiPHa9jfLvRM3IyMA4GA1UdDwEB/wQEAwIFoDAMBgNV
HRMBAf8EAjAAMCAGA1UdJQQZMBcGCCsGAQUFBwMEBgsrBgEEAbIxAQMFAjARBglghkgBhvhCAQEE
BAMCBSAwRgYDVR0gBD8wPTA7BgwrBgEEAbIxAQIBAQEwKzApBggrBgEFBQcCARYdaHR0cHM6Ly9z
ZWN1cmUuY29tb2RvLm5ldC9DUFMwgaUGA1UdHwSBnTCBmjBMoEqgSIZGaHR0cDovL2NybC5jb21v
ZG9jYS5jb20vVVROLVVTRVJGaXJzdC1DbGllbnRBdXRoZW50aWNhdGlvbmFuZEVtYWlsLmNybDBK
oEigRoZEaHR0cDovL2NybC5jb21vZG8ubmV0L1VUTi1VU0VSRmlyc3QtQ2xpZW50QXV0aGVudGlj
YXRpb25hbmRFbWFpbC5jcmwwbAYIKwYBBQUHAQEEYDBeMDYGCCsGAQUFBzAChipodHRwOi8vY3J0
LmNvbW9kb2NhLmNvbS9VVE5BQUFDbGllbnRDQS5jcnQwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3Nw
LmNvbW9kb2NhLmNvbTAhBgNVHREEGjAYgRZldmVydC5sYW1tZXJ0c0BzYXJhLm5sMA0GCSqGSIb3
DQEBBQUAA4IBAQBaJvuiBPgqdq9MEYH9dj2vW8hprAIBEnOOvfXdoP604kBhSbj3s/l0ZQyQY35W
YVAltuJQ365uJKigPQHxS+slpUOAJPEVNmwPUIW0GAjxCPxgLdJgAc4jOV8f7oF3pnfI4ukCmjPN
LpaylZzMhFt+t6zDWiMtUqyxK/4on62LoLp2D88lxwxI3q3i00bPlV0wZ+mjzS7vrQGcUgDPWEBL
FD0I3pR+Z1EH1qjelBmBJV9paVzfzmoU93LIDJA7/MEUXd3JiMZGEoVd2qE5qIZO4fNJ+Cyo6tu1
ECSQxoL6qN+sBwekiFA9Q/zlUp0HfU6Jt8WmtZ44SOh67LcmCL4ZMYIEaDCCBGQCAQEwgcQwga4x
CzAJBgNVBAYTAlVTMQswCQYDVQQIEwJVVDEXMBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNV
BAoTFVRoZSBVU0VSVFJVU1QgTmV0d29yazEhMB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3Qu
Y29tMTYwNAYDVQQDEy1VVE4tVVNFUkZpcnN0LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1h
aWwCEQD1By0ffbyGqnOmGBaIfvi/MAkGBSsOAwIaBQCgggJ4MBgGCSqGSIb3DQEJAzELBgkqhkiG
9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTEwMDIyMjA4NDAzMVowIwYJKoZIhvcNAQkEMRYEFO9qyUsH
urYB3V0r+3w8xeu9BtwGMGcGCSqGSIb3DQEJDzFaMFgwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwIC
AgCAMA0GCCqGSIb3DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMAcGBSsOAwIaMAoGCCqG
SIb3DQIFMIHVBgkrBgEEAYI3EAQxgccwgcQwga4xCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJVVDEX
MBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNVBAoTFVRoZSBVU0VSVFJVU1QgTmV0d29yazEh
MB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3QuY29tMTYwNAYDVQQDEy1VVE4tVVNFUkZpcnN0
LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1haWwCEQD1By0ffbyGqnOmGBaIfvi/MIHXBgsq
hkiG9w0BCRACCzGBx6CBxDCBrjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5T
YWx0IExha2UgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYDVQQLExho
dHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1VU0VSRmlyc3QtQ2xpZW50IEF1
dGhlbnRpY2F0aW9uIGFuZCBFbWFpbAIRAPUHLR99vIaqc6YYFoh++L8wDQYJKoZIhvcNAQEBBQAE
ggEAqNuavFkJJoCE1dqsfTg8fSApcgG7m9per4jZ9OdM5GQdAuHRQrG13DYvNERwfOd5iRffrsi5
KPwslXQDM9oBsLWK9+TV8e2RnbhjAgM1jmMZFNYbc0bgjzQpfPG0Ezdn2NaYfHHtZ619q8yAkJX1
kybINGCDcxH4/bh7beZ5yBVPHPbFvBaVQA/7KQdc99tYqHiOdn0s6GDi9SAHKAFNXUja0C4whwEH
dJERpKaKVsEKmOewywnaVYg6lGHF66Rh1H0Jwnts0phuv2dTJdSBWc6Sua8SMy0G4kHTZbJ2eMbZ
rxSfVwDqk97rgh6OuHD7LED3I8/YRSE9PJ6CtCTXUgAAAAAAAA==

------=_NextPart_000_003A_01CAB3A3.12F57DC0--

From general-return-1079-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 22 16:20:08 2010
Return-Path: <general-return-1079-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 61629 invoked from network); 22 Feb 2010 16:20:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Feb 2010 16:20:08 -0000
Received: (qmail 65159 invoked by uid 500); 22 Feb 2010 16:20:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65096 invoked by uid 500); 22 Feb 2010 16:20:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65086 invoked by uid 99); 22 Feb 2010 16:20:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Feb 2010 16:20:07 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.92.25] (HELO qw-out-2122.google.com) (74.125.92.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Feb 2010 16:20:00 +0000
Received: by qw-out-2122.google.com with SMTP id 5so488619qwi.35
        for <general@hadoop.apache.org>; Mon, 22 Feb 2010 08:19:39 -0800 (PST)
MIME-Version: 1.0
Received: by 10.229.227.5 with SMTP id iy5mr351098qcb.29.1266855579246; Mon, 
	22 Feb 2010 08:19:39 -0800 (PST)
Date: Mon, 22 Feb 2010 08:19:39 -0800
Message-ID: <c9118d4c1002220819h1d995bcap794ef0ef86218d5b@mail.gmail.com>
Subject: Security roadmap [Was: Release plans]
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016363b82f0c862bd048032cce7

--0016363b82f0c862bd048032cce7
Content-Type: text/plain; charset=ISO-8859-1

https://issues.apache.org/jira/browse/HADOOP-4487. Please start a new thread
next time.

On Mon, Feb 22, 2010 at 12:40 AM, Evert Lammerts <Evert.Lammerts@sara.nl>wrote:

> I'm sorry for breaking into your discussion as an outsider, but I'm very
> curious about the security features you are planning to roll out in March.
> Where can I find information about this?
>
> Best regards,
> Evert Lammerts
>
> > -----Original Message-----
> > From: Jay Booth [mailto:jaybooth@gmail.com]
> > Sent: maandag 22 februari 2010 5:55
> > To: general@hadoop.apache.org
> > Cc: general@hadoop.apache.org
> > Subject: Re: Release plans
> >
> > Well, since someone has to get the ball rolling as far as release
> > masters, I'll nominate Stack and/or someone hbase related for 0.21
> > with the primary goal of being "soon"?  They get a big win from append
> > and others will gain from the expanded mapreduce lib, better
> > schedulers, etc. There are a lot of new features and some major
> > changes (project split) already in the 0.21 branch, so IMO it's worth
> > considering a release with minimal backports, rather than make binding
> > decisions about 0.22 before 0.21 is even in the wild.
> >
> > -Jay
> >
> > PS sorry Stack
> >
> > On Feb 20, 2010, at 5:04 PM, Stack <stack@duboce.net> wrote:
> >
> > > On Fri, Feb 19, 2010 at 4:23 PM, Eli Collins <eli@cloudera.com>
> > wrote:
> > >> Can we make a decision on basing 21 on the current branch and if
> > it's
> > >> decided that 22 can't remove stuff that was in 20 we'll go back and
> > >> do
> > >> the necessary additions on 21 and trunk? Suspect that decision will
> > >> take a lot more back and forth, but needs to conclude before 21 is
> > >> released.
> > >>
> > >
> > > Lets.
> > >
> > > Regards 0.21/current-branch release, as has been suggested above,
> > > first we need to figure the release master.  No release master, no
> > > release.  If we have a release master, then I suggest we vote on
> > > current branch being released as 0.21 as soon as the blockers are
> > > cleared.
> > >
> > > I don't think we need muddy the above vote with whether or not 0.21
> > > maintains API combatibility with 0.20.  IMO, it must (because Y! want
> > > to have the 0.20 API in place when January 2011 rolls around).  This
> > > makes 0.21 a "minor" release -- something we've not done before (For
> > > the record, I also had a misunderstanding that what we were doing up
> > > to this was major and patch only).  So, part of the release process
> > > would involve ensuring no removed deprecations, etc.
> > >
> > > As DC has been saying, this requirement that releases between now and
> > > January 2011 not change APIs makes 0.20 retroactively into a "major"
> > > release.  0.20 is the release where major shifted left in our
> > > versioning scheme and minor releases came into play.  0.21 and 0.22
> > > will be minor releases. Can we just acknowledge this fact, that there
> > > was a step at version 0.20, update the wiki around versioning -- its
> > > currently wrong anyways as Elis' points out -- and just move on?
> > > Going back and calling 0.20 a 1.0 seems more apt to create confusion
> > > and besides, I'm with Allen that hadoop 1.0 needs wire compatibility
> > > before the 1.0 can roll around.
> > >
> > > Thanks,
> > > St.Ack
> > > P.S. +1 on branching as soon as avro and security are in, etc.
>

--0016363b82f0c862bd048032cce7--

From general-return-1080-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 22 17:56:35 2010
Return-Path: <general-return-1080-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6668 invoked from network); 22 Feb 2010 17:56:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Feb 2010 17:56:35 -0000
Received: (qmail 44020 invoked by uid 500); 22 Feb 2010 17:56:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 43951 invoked by uid 500); 22 Feb 2010 17:56:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 43941 invoked by uid 99); 22 Feb 2010 17:56:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Feb 2010 17:56:33 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.59.243] (HELO qmta13.westchester.pa.mail.comcast.net) (76.96.59.243)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Feb 2010 17:56:24 +0000
Received: from omta16.westchester.pa.mail.comcast.net ([76.96.62.88])
	by qmta13.westchester.pa.mail.comcast.net with comcast
	id l3Bd1d0041uE5Es5D5w3mK; Mon, 22 Feb 2010 17:56:04 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta16.westchester.pa.mail.comcast.net with comcast
	id l5yH1d0132SbwD53c5yLGT; Mon, 22 Feb 2010 17:58:25 +0000
Message-Id: <8A9082CB-8E27-4904-AADA-7894C0190177@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <c9118d4c1002220819h1d995bcap794ef0ef86218d5b@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Security roadmap [Was: Release plans]
Date: Mon, 22 Feb 2010 09:55:52 -0800
References: <c9118d4c1002220819h1d995bcap794ef0ef86218d5b@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


Evert Lammerts wrote:

> I'm sorry for breaking into your discussion as an outsider, but I'm  
> very
> curious about the security features you are planning to roll out in  
> March.

At a high level, all of the interactions with HDFS and MapReduce will  
be authenticated. Kerberos will be used as the base authentication,  
but there are mechanisms to ensure that MapReduce jobs launching won't  
cause a DOS against the Kerberos KDC.

Yahoo is pushing hard for getting this deployed this summer. We are  
backporting all of the work on to our Yahoo 0.20 branch and will be  
posting it to github.

-- Owen


From general-return-1081-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 23 01:46:40 2010
Return-Path: <general-return-1081-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54929 invoked from network); 23 Feb 2010 01:46:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Feb 2010 01:46:40 -0000
Received: (qmail 57150 invoked by uid 500); 23 Feb 2010 01:46:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57079 invoked by uid 500); 23 Feb 2010 01:46:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57069 invoked by uid 99); 23 Feb 2010 01:46:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 01:46:39 +0000
X-ASF-Spam-Status: No, hits=4.3 required=10.0
	tests=FS_START_DOYOU2,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 01:46:29 +0000
Received: from thickbeside-lm.corp.yahoo.com (thickbeside-lm.corp.yahoo.com [10.72.109.129])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o1N1k6dK043928
	for <general@hadoop.apache.org>; Mon, 22 Feb 2010 17:46:06 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:content-type:
	content-transfer-encoding:mime-version:subject:date:x-mailer;
	b=Qc8y1/6aP3e0T8UpAqD+UHQLjIEtrvZm8AwmN2B7f2vEnwGu9s/57wP5wei99kPG
Message-Id: <F7524368-35C3-4D70-B53B-D17F39F1ECF1@yahoo-inc.com>
From: Nigel Daley <ndaley@yahoo-inc.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Do you have what it takes to test Hadoop?
Date: Mon, 22 Feb 2010 17:46:05 -0800
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

Folks, we are looking for outstanding software engineers and  
architects who are interested in quality and testing.

Please spread the word.  Details on how to apply are here:

http://developer.yahoo.net/blogs/hadoop/2010/02/do_you_have_what_it_takes_to_t.html

Cheers,
Nigel


From general-return-1082-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 23 06:10:37 2010
Return-Path: <general-return-1082-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49313 invoked from network); 23 Feb 2010 06:10:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Feb 2010 06:10:36 -0000
Received: (qmail 10410 invoked by uid 500); 23 Feb 2010 06:10:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 10310 invoked by uid 500); 23 Feb 2010 06:10:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 10290 invoked by uid 99); 23 Feb 2010 06:10:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 06:10:35 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 23 Feb 2010 06:10:33 +0000
Received: (qmail 49107 invoked by uid 99); 23 Feb 2010 06:10:11 -0000
Received: from localhost.apache.org (HELO mail-bw0-f211.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 06:10:11 +0000
Received: by bwz3 with SMTP id 3so2367552bwz.29
        for <multiple recipients>; Mon, 22 Feb 2010 22:10:09 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.0.84 with SMTP id 20mr3478099bka.5.1266905409726; Mon, 22 
	Feb 2010 22:10:09 -0800 (PST)
In-Reply-To: <1267dd3b1002190107y65801a7iab7c5c33cad2daf@mail.gmail.com>
References: <1267dd3b1002190107y65801a7iab7c5c33cad2daf@mail.gmail.com>
Date: Mon, 22 Feb 2010 22:10:09 -0800
Message-ID: <1267dd3b1002222210h154cc911gd6c9ee992a09f50e@mail.gmail.com>
Subject: Re: [VOTE] Should we release 0.20.2-rc4 ?
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Cc: private@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Sending a gentle reminder to PMC members to vote on this release and
to devs and users to test it out. -C

On Fri, Feb 19, 2010 at 1:07 AM, Chris Douglas <cdouglas@apache.org> wrote:
> There are now only two consistently failing testcases in my
> environment, both in the capacity-scheduler contrib module:
>
> org.apache.hadoop.mapred.TestJobInitialization
> org.apache.hadoop.mapred.TestQueueCapacities
>
> neither of which is a regression from 0.20.1
>
> http://people.apache.org/~cdouglas/0.20.2-rc4
>
> Please try it out. -C
>

From general-return-1083-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 23 07:53:13 2010
Return-Path: <general-return-1083-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96467 invoked from network); 23 Feb 2010 07:53:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Feb 2010 07:53:12 -0000
Received: (qmail 2365 invoked by uid 500); 23 Feb 2010 07:53:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2295 invoked by uid 500); 23 Feb 2010 07:53:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2285 invoked by uid 99); 23 Feb 2010 07:53:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 07:53:11 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of dhruba@gmail.com designates 74.125.83.48 as permitted sender)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 07:53:00 +0000
Received: by gwaa11 with SMTP id a11so391021gwa.35
        for <general@hadoop.apache.org>; Mon, 22 Feb 2010 23:52:38 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=/d+dPg19eZo7CDnAhDUuTpHe8bqjdNyxXnYSnEX2fco=;
        b=QYQHXyEM0XUzup6WiYBE+MOW+fjJb1gXSdzh8M7RbXfNpPemfCwmyp/gDUdwqCG/oX
         I9II3AUzVmtgDkc3gVraVMGPb4N9Woh1O+4I/cuGq2Xu2UyFB4GEmxXuMwPUk4sBOm+M
         pzIn5UXBoJQ0gxC6FiAy8gvaHwLUU3r80rmno=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Ixmm9bGLuuMobqAwX0yIaqkuqCMwqGbjC29BbG2HeOsiNfEnQ+sJncX8QFuR/4LQ2X
         60jGh1Q2hojGRUV61MpdcLnxAGgexyATRgnPIqTWOgMViLV9FZF/gOSASp1xoOo2YXsr
         iaolGDqAHge9mBAOQalaCWIyjnGscqc8TQRJc=
MIME-Version: 1.0
Received: by 10.150.117.26 with SMTP id p26mr4846065ybc.348.1266911558292; 
	Mon, 22 Feb 2010 23:52:38 -0800 (PST)
In-Reply-To: <7c962aed1002191108n76516c82q106e7d3e5aeb9428@mail.gmail.com>
References: <1267dd3b1002190107y65801a7iab7c5c33cad2daf@mail.gmail.com>
	 <7c962aed1002191108n76516c82q106e7d3e5aeb9428@mail.gmail.com>
Date: Mon, 22 Feb 2010 23:52:38 -0800
Message-ID: <4aa34eb71002222352l394b9002jd3fd95b4ef0c06c6@mail.gmail.com>
Subject: Re: [VOTE] Should we release 0.20.2-rc4 ?
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd72a1864d57304803fd5a4
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd72a1864d57304803fd5a4
Content-Type: text/plain; charset=ISO-8859-1

+1. Looks good to me. Ran unit tests.

-dhruba


On Fri, Feb 19, 2010 at 11:08 AM, Stack <stack@duboce.net> wrote:

> +1
>
> I put up on a small cluster under load.  Seems to work fine.  Trolled
> logs a while.  Nothing out of the ordinary.  Checked docs.  They look
> grand.
>
> St.Ack
>
> On Fri, Feb 19, 2010 at 1:07 AM, Chris Douglas <cdouglas@apache.org>
> wrote:
> > There are now only two consistently failing testcases in my
> > environment, both in the capacity-scheduler contrib module:
> >
> > org.apache.hadoop.mapred.TestJobInitialization
> > org.apache.hadoop.mapred.TestQueueCapacities
> >
> > neither of which is a regression from 0.20.1
> >
> > http://people.apache.org/~cdouglas/0.20.2-rc4
> >
> > Please try it out. -C
> >
>



-- 
Connect to me at http://www.facebook.com/dhruba

--000e0cd72a1864d57304803fd5a4--

From general-return-1084-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 23 18:50:59 2010
Return-Path: <general-return-1084-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 13459 invoked from network); 23 Feb 2010 18:50:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Feb 2010 18:50:59 -0000
Received: (qmail 99074 invoked by uid 500); 23 Feb 2010 18:50:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98993 invoked by uid 500); 23 Feb 2010 18:50:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98983 invoked by uid 99); 23 Feb 2010 18:50:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 18:50:57 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 18:50:48 +0000
Received: by pvd12 with SMTP id 12so425502pvd.35
        for <general@hadoop.apache.org>; Tue, 23 Feb 2010 10:50:26 -0800 (PST)
MIME-Version: 1.0
Received: by 10.141.213.39 with SMTP id p39mr4258037rvq.100.1266951026393; 
	Tue, 23 Feb 2010 10:50:26 -0800 (PST)
In-Reply-To: <4aa34eb71002222352l394b9002jd3fd95b4ef0c06c6@mail.gmail.com>
References: <1267dd3b1002190107y65801a7iab7c5c33cad2daf@mail.gmail.com> 
	<7c962aed1002191108n76516c82q106e7d3e5aeb9428@mail.gmail.com> 
	<4aa34eb71002222352l394b9002jd3fd95b4ef0c06c6@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Tue, 23 Feb 2010 10:50:06 -0800
Message-ID: <45f85f71002231050r53cc0c9bk57d61be742ba98@mail.gmail.com>
Subject: Re: [VOTE] Should we release 0.20.2-rc4 ?
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Tested download with md5: 8f40198ed18bef28aeea1401ec536cb9
Tried to verify the GPG signature, but Chris is not in
http://download.nextag.com/apache/hadoop/core/KEYS - he should be
added there if he is going to sign releases.

I ran unit tests on my machine at home - TestStreamingExitStatus
failed with an OOME. I think it's exactly MAPREDUCE-587. Aside from
that, all unit tests passed. I also ran a few jobs on a
pseudo-distributed cluster and it worked fine.

Since this is just a test bug, and in contrib, I think we should
release anyway. Meanwhile let's commit MAPREDUCE-587 to branch-20
before the next release. I'll reopen that JIRA.

So, [non-binding] +1 from me. Thanks for the hard work, Chris.

-Todd

On Mon, Feb 22, 2010 at 11:52 PM, Dhruba Borthakur <dhruba@gmail.com> wrote=
:
> +1. Looks good to me. Ran unit tests.
>
> -dhruba
>
>
> On Fri, Feb 19, 2010 at 11:08 AM, Stack <stack@duboce.net> wrote:
>
>> +1
>>
>> I put up on a small cluster under load. =A0Seems to work fine. =A0Trolle=
d
>> logs a while. =A0Nothing out of the ordinary. =A0Checked docs. =A0They l=
ook
>> grand.
>>
>> St.Ack
>>
>> On Fri, Feb 19, 2010 at 1:07 AM, Chris Douglas <cdouglas@apache.org>
>> wrote:
>> > There are now only two consistently failing testcases in my
>> > environment, both in the capacity-scheduler contrib module:
>> >
>> > org.apache.hadoop.mapred.TestJobInitialization
>> > org.apache.hadoop.mapred.TestQueueCapacities
>> >
>> > neither of which is a regression from 0.20.1
>> >
>> > http://people.apache.org/~cdouglas/0.20.2-rc4
>> >
>> > Please try it out. -C
>> >
>>
>
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
>

From general-return-1085-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 23 20:07:39 2010
Return-Path: <general-return-1085-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53075 invoked from network); 23 Feb 2010 20:07:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Feb 2010 20:07:39 -0000
Received: (qmail 25595 invoked by uid 500); 23 Feb 2010 20:07:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25497 invoked by uid 500); 23 Feb 2010 20:07:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25477 invoked by uid 99); 23 Feb 2010 20:07:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 20:07:37 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 20:07:29 +0000
Received: from [192.168.0.198] (snvvpn4-10-72-168-c187.hq.corp.yahoo.com [10.72.168.187])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o1NK5iVH051144;
	Tue, 23 Feb 2010 12:05:45 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=from:to:in-reply-to:subject:x-priority:references:
	message-id:content-type:content-transfer-encoding:mime-version:date:cc:x-mailer;
	b=y7LKgFrFvXS1QISp9AN35KwngL8lOBoAZRE1nPBB2pHtGz7EDyINEyB5kAaYoWt/
From: Alan Gates <gates@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <466AB331-82A3-4111-BA34-75F44A31DBCF@yahoo-inc.com>
Subject: Re: [VOTE] Release candidate for Pig 0.6.0
X-Priority: 3
References: <3316F847-FBB7-4D41-B1A7-C5CD3FB1A40D@yahoo-inc.com> <14B7BB1E-F459-4FBC-AEBD-120B822281E3@yahoo-inc.com> <1E049E93BEE249D59D88B0DDACEF9106@DanielLaptop> <466AB331-82A3-4111-BA34-75F44A31DBCF@yahoo-inc.com>
Message-Id: <E744EA5B-ECE1-44CD-B4F7-E75A53FCBA19@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Date: Tue, 23 Feb 2010 12:05:44 -0800
Cc: <private@hadoop.apache.org>
X-Mailer: Apple Mail (2.936)

I have rolled a new version of the tarball that has the version  
command correct.  I did not make any code changes, I just needed to  
run ant with the correct properties set.  So I have not made this a  
new release candidate.  So the candidate is at the same place as  
before, http://people.apache.org/~gates/pig-0.6.0-candidate-1/

Please try it out and vote by Friday, February 26th.  Thanks.

Alan.

On Feb 19, 2010, at 1:57 PM, Alan Gates wrote:

> Alright, I'll fix that and reroll the release.
>
> Alan.
>
> On Feb 19, 2010, at 1:35 PM, Daniel Dai wrote:
>
>> "pig -version" should report 0.6.0 instead of 0.6.1. +1 otherwise.
>>
>>
>> --------------------------------------------------
>> From: "Alan Gates" <gates@yahoo-inc.com>
>> Sent: Friday, February 19, 2010 11:58 AM
>> To: <private@hadoop.apache.org>
>> Subject: Fwd: [VOTE] Release candidate for Pig 0.6.0
>>
>>> PMC members,
>>>
>>> Please take a look and vote.  Thanks.
>>>
>>> Alan.
>>>
>>> Begin forwarded message:
>>>
>>>> From: Alan Gates <gates@yahoo-inc.com>
>>>> Date: February 16, 2010 11:22:52 AM PST
>>>> To: <general@hadoop.apache.org>
>>>> Subject: [VOTE] Release candidate for Pig 0.6.0
>>>> Reply-To: general@hadoop.apache.org
>>>>
>>>> All,
>>>>
>>>> I have rolled a release candidate for Pig 0.6.0 and uploaded it  
>>>> to http://people.apache.org/~gates/pig-0.6.0-candidate-1/
>>>> A description of what is new and different in this release is   
>>>> included in the release notes, which I have also uploaded to  
>>>> that  directory. Please download the candidate, test it, and vote.
>>>>
>>>> Alan.
>


From general-return-1086-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 23 20:56:22 2010
Return-Path: <general-return-1086-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68165 invoked from network); 23 Feb 2010 20:56:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Feb 2010 20:56:22 -0000
Received: (qmail 11902 invoked by uid 500); 23 Feb 2010 20:56:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11815 invoked by uid 500); 23 Feb 2010 20:56:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11805 invoked by uid 99); 23 Feb 2010 20:56:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 20:56:21 +0000
X-ASF-Spam-Status: No, hits=4.2 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 20:56:11 +0000
Received: from sp1-ex07cas01.ds.corp.yahoo.com (sp1-ex07cas01.ds.corp.yahoo.com [216.252.116.137])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o1NKruuk075365
	for <general@hadoop.apache.org>; Tue, 23 Feb 2010 12:53:56 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:in-reply-to:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=CF5XNKikZb1vG0hfkOpcgRNK9yDQ8wPjOP9a2J3RxZBwxRcoi5cAVNm8I0YBlpAz
Received: from SP1-EX07VS02.ds.corp.yahoo.com ([216.252.116.135]) by
 sp1-ex07cas01.ds.corp.yahoo.com ([216.252.116.137]) with mapi; Tue, 23 Feb
 2010 12:53:56 -0800
From: Ravi Phulari <rphulari@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 23 Feb 2010 12:53:54 -0800
Subject: Re: [VOTE] Should we release 0.20.2-rc4 ?
Thread-Topic: [VOTE] Should we release 0.20.2-rc4 ?
Thread-Index: Acq0XW5lIPPM8dk9QaioNEUfwFmHDgAbODgp
Message-ID: <C7A98062.16DAB%rphulari@yahoo-inc.com>
In-Reply-To: <4aa34eb71002222352l394b9002jd3fd95b4ef0c06c6@mail.gmail.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C7A9806216DABrphulariyahooinccom_"
MIME-Version: 1.0

--_000_C7A9806216DABrphulariyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

+1 Release looks good.
Successfully ran unit tests , Executed Mapred word count program , verified=
 user documentation.

-
Ravi

On 2/22/10 11:52 PM, "Dhruba Borthakur" <dhruba@gmail.com> wrote:

+1. Looks good to me. Ran unit tests.

-dhruba


On Fri, Feb 19, 2010 at 11:08 AM, Stack <stack@duboce.net> wrote:

> +1
>
> I put up on a small cluster under load.  Seems to work fine.  Trolled
> logs a while.  Nothing out of the ordinary.  Checked docs.  They look
> grand.
>
> St.Ack
>
> On Fri, Feb 19, 2010 at 1:07 AM, Chris Douglas <cdouglas@apache.org>
> wrote:
> > There are now only two consistently failing testcases in my
> > environment, both in the capacity-scheduler contrib module:
> >
> > org.apache.hadoop.mapred.TestJobInitialization
> > org.apache.hadoop.mapred.TestQueueCapacities
> >
> > neither of which is a regression from 0.20.1
> >
> > http://people.apache.org/~cdouglas/0.20.2-rc4
> >
> > Please try it out. -C
> >
>



--
Connect to me at http://www.facebook.com/dhruba


Ravi
--


--_000_C7A9806216DABrphulariyahooinccom_--

From general-return-1087-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 23 21:33:27 2010
Return-Path: <general-return-1087-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 90142 invoked from network); 23 Feb 2010 21:33:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Feb 2010 21:33:27 -0000
Received: (qmail 86660 invoked by uid 500); 23 Feb 2010 21:33:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86589 invoked by uid 500); 23 Feb 2010 21:33:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86579 invoked by uid 99); 23 Feb 2010 21:33:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 21:33:26 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 23 Feb 2010 21:33:17 +0000
Received: (qmail 89926 invoked by uid 99); 23 Feb 2010 21:32:57 -0000
Received: from localhost.apache.org (HELO mail-bw0-f211.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 21:32:57 +0000
Received: by bwz3 with SMTP id 3so3125605bwz.29
        for <general@hadoop.apache.org>; Tue, 23 Feb 2010 13:32:55 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.13.69 with SMTP id b5mr3245166bka.196.1266960775297; Tue, 
	23 Feb 2010 13:32:55 -0800 (PST)
In-Reply-To: <45f85f71002231050r53cc0c9bk57d61be742ba98@mail.gmail.com>
References: <1267dd3b1002190107y65801a7iab7c5c33cad2daf@mail.gmail.com>
	 <7c962aed1002191108n76516c82q106e7d3e5aeb9428@mail.gmail.com>
	 <4aa34eb71002222352l394b9002jd3fd95b4ef0c06c6@mail.gmail.com>
	 <45f85f71002231050r53cc0c9bk57d61be742ba98@mail.gmail.com>
Date: Tue, 23 Feb 2010 13:32:54 -0800
Message-ID: <1267dd3b1002231332oe60997egdf2626c3b6536573@mail.gmail.com>
Subject: Re: [VOTE] Should we release 0.20.2-rc4 ?
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I looked around, and found this repository of public keys:

http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS

I'd be happy to upload my key elsewhere if necessary; I found no
documentation on that point.

We'll inevitably roll a 0.20.3, which I agree, should include MAPREDUCE-587=
 -C

On Tue, Feb 23, 2010 at 10:50 AM, Todd Lipcon <todd@cloudera.com> wrote:
> Tested download with md5: 8f40198ed18bef28aeea1401ec536cb9
> Tried to verify the GPG signature, but Chris is not in
> http://download.nextag.com/apache/hadoop/core/KEYS - he should be
> added there if he is going to sign releases.
>
> I ran unit tests on my machine at home - TestStreamingExitStatus
> failed with an OOME. I think it's exactly MAPREDUCE-587. Aside from
> that, all unit tests passed. I also ran a few jobs on a
> pseudo-distributed cluster and it worked fine.
>
> Since this is just a test bug, and in contrib, I think we should
> release anyway. Meanwhile let's commit MAPREDUCE-587 to branch-20
> before the next release. I'll reopen that JIRA.
>
> So, [non-binding] +1 from me. Thanks for the hard work, Chris.
>
> -Todd
>
> On Mon, Feb 22, 2010 at 11:52 PM, Dhruba Borthakur <dhruba@gmail.com> wro=
te:
>> +1. Looks good to me. Ran unit tests.
>>
>> -dhruba
>>
>>
>> On Fri, Feb 19, 2010 at 11:08 AM, Stack <stack@duboce.net> wrote:
>>
>>> +1
>>>
>>> I put up on a small cluster under load. =A0Seems to work fine. =A0Troll=
ed
>>> logs a while. =A0Nothing out of the ordinary. =A0Checked docs. =A0They =
look
>>> grand.
>>>
>>> St.Ack
>>>
>>> On Fri, Feb 19, 2010 at 1:07 AM, Chris Douglas <cdouglas@apache.org>
>>> wrote:
>>> > There are now only two consistently failing testcases in my
>>> > environment, both in the capacity-scheduler contrib module:
>>> >
>>> > org.apache.hadoop.mapred.TestJobInitialization
>>> > org.apache.hadoop.mapred.TestQueueCapacities
>>> >
>>> > neither of which is a regression from 0.20.1
>>> >
>>> > http://people.apache.org/~cdouglas/0.20.2-rc4
>>> >
>>> > Please try it out. -C
>>> >
>>>
>>
>>
>>
>> --
>> Connect to me at http://www.facebook.com/dhruba
>>
>

From general-return-1088-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 23 21:37:22 2010
Return-Path: <general-return-1088-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91844 invoked from network); 23 Feb 2010 21:37:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Feb 2010 21:37:22 -0000
Received: (qmail 93649 invoked by uid 500); 23 Feb 2010 21:37:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93588 invoked by uid 500); 23 Feb 2010 21:37:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93578 invoked by uid 99); 23 Feb 2010 21:37:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 21:37:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 23 Feb 2010 21:37:20 +0000
Received: (qmail 91703 invoked by uid 99); 23 Feb 2010 21:37:00 -0000
Received: from localhost.apache.org (HELO mail-bw0-f211.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 21:37:00 +0000
Received: by bwz3 with SMTP id 3so3128697bwz.29
        for <multiple recipients>; Tue, 23 Feb 2010 13:36:57 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.155.69 with SMTP id r5mr955420bkw.0.1266961017783; Tue, 23 
	Feb 2010 13:36:57 -0800 (PST)
In-Reply-To: <E744EA5B-ECE1-44CD-B4F7-E75A53FCBA19@yahoo-inc.com>
References: <3316F847-FBB7-4D41-B1A7-C5CD3FB1A40D@yahoo-inc.com>
	 <14B7BB1E-F459-4FBC-AEBD-120B822281E3@yahoo-inc.com>
	 <1E049E93BEE249D59D88B0DDACEF9106@DanielLaptop>
	 <466AB331-82A3-4111-BA34-75F44A31DBCF@yahoo-inc.com>
	 <E744EA5B-ECE1-44CD-B4F7-E75A53FCBA19@yahoo-inc.com>
Date: Tue, 23 Feb 2010 13:36:57 -0800
Message-ID: <1267dd3b1002231336m657adf5oac3734bd36f3b323@mail.gmail.com>
Subject: Re: [VOTE] Release candidate for Pig 0.6.0
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Cc: private@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

+1

I ran the unit tests on the first version of the RC and all passed.
MD5 and signature of this build match. -C

On Tue, Feb 23, 2010 at 12:05 PM, Alan Gates <gates@yahoo-inc.com> wrote:
> I have rolled a new version of the tarball that has the version command
> correct. =A0I did not make any code changes, I just needed to run ant wit=
h the
> correct properties set. =A0So I have not made this a new release candidat=
e.
> =A0So the candidate is at the same place as before,
> http://people.apache.org/~gates/pig-0.6.0-candidate-1/
>
> Please try it out and vote by Friday, February 26th. =A0Thanks.
>
> Alan.
>
> On Feb 19, 2010, at 1:57 PM, Alan Gates wrote:
>
>> Alright, I'll fix that and reroll the release.
>>
>> Alan.
>>
>> On Feb 19, 2010, at 1:35 PM, Daniel Dai wrote:
>>
>>> "pig -version" should report 0.6.0 instead of 0.6.1. +1 otherwise.
>>>
>>>
>>> --------------------------------------------------
>>> From: "Alan Gates" <gates@yahoo-inc.com>
>>> Sent: Friday, February 19, 2010 11:58 AM
>>> To: <private@hadoop.apache.org>
>>> Subject: Fwd: [VOTE] Release candidate for Pig 0.6.0
>>>
>>>> PMC members,
>>>>
>>>> Please take a look and vote. =A0Thanks.
>>>>
>>>> Alan.
>>>>
>>>> Begin forwarded message:
>>>>
>>>>> From: Alan Gates <gates@yahoo-inc.com>
>>>>> Date: February 16, 2010 11:22:52 AM PST
>>>>> To: <general@hadoop.apache.org>
>>>>> Subject: [VOTE] Release candidate for Pig 0.6.0
>>>>> Reply-To: general@hadoop.apache.org
>>>>>
>>>>> All,
>>>>>
>>>>> I have rolled a release candidate for Pig 0.6.0 and uploaded it to
>>>>> http://people.apache.org/~gates/pig-0.6.0-candidate-1/
>>>>> A description of what is new and different in this release is =A0incl=
uded
>>>>> in the release notes, which I have also uploaded to that =A0directory=
. Please
>>>>> download the candidate, test it, and vote.
>>>>>
>>>>> Alan.
>>
>
>

From general-return-1089-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 23 22:09:24 2010
Return-Path: <general-return-1089-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 8785 invoked from network); 23 Feb 2010 22:09:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Feb 2010 22:09:24 -0000
Received: (qmail 42362 invoked by uid 500); 23 Feb 2010 22:09:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42293 invoked by uid 500); 23 Feb 2010 22:09:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42283 invoked by uid 99); 23 Feb 2010 22:09:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 22:09:23 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of swatt@us.ibm.com designates 32.97.110.160 as permitted sender)
Received: from [32.97.110.160] (HELO e39.co.us.ibm.com) (32.97.110.160)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 22:09:12 +0000
Received: from d03relay01.boulder.ibm.com (d03relay01.boulder.ibm.com [9.17.195.226])
	by e39.co.us.ibm.com (8.14.3/8.13.1) with ESMTP id o1NM1A51023705
	for <general@hadoop.apache.org>; Tue, 23 Feb 2010 15:01:10 -0700
Received: from d03av04.boulder.ibm.com (d03av04.boulder.ibm.com [9.17.195.170])
	by d03relay01.boulder.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id o1NM8YYu034552
	for <general@hadoop.apache.org>; Tue, 23 Feb 2010 15:08:36 -0700
Received: from d03av04.boulder.ibm.com (loopback [127.0.0.1])
	by d03av04.boulder.ibm.com (8.14.3/8.13.1/NCO v10.0 AVout) with ESMTP id o1NM8XHU026758
	for <general@hadoop.apache.org>; Tue, 23 Feb 2010 15:08:33 -0700
Received: from d03nm123.boulder.ibm.com (d03nm123.boulder.ibm.com [9.17.195.149])
	by d03av04.boulder.ibm.com (8.14.3/8.13.1/NCO v10.0 AVin) with ESMTP id o1NM8Xml026739
	for <general@hadoop.apache.org>; Tue, 23 Feb 2010 15:08:33 -0700
In-Reply-To: <1267dd3b1002231332oe60997egdf2626c3b6536573@mail.gmail.com>
References: <1267dd3b1002190107y65801a7iab7c5c33cad2daf@mail.gmail.com>	 <7c962aed1002191108n76516c82q106e7d3e5aeb9428@mail.gmail.com>
	 <4aa34eb71002222352l394b9002jd3fd95b4ef0c06c6@mail.gmail.com>	 <45f85f71002231050r53cc0c9bk57d61be742ba98@mail.gmail.com> <1267dd3b1002231332oe60997egdf2626c3b6536573@mail.gmail.com>
To: general@hadoop.apache.org
MIME-Version: 1.0
Subject: Re: [VOTE] Should we release 0.20.2-rc4 ?
X-KeepSent: F388BED4:4E9FA0BC-862576D3:0079591D;
 type=4; name=$KeepSent
X-Mailer: Lotus Notes Release 8.5 December 05, 2008
Message-ID: <OFF388BED4.4E9FA0BC-ON862576D3.0079591D-862576D3.0079A187@us.ibm.com>
From: Stephen Watt <swatt@us.ibm.com>
Date: Tue, 23 Feb 2010 16:08:31 -0600
X-MIMETrack: Serialize by Router on D03NM123/03/M/IBM(Release 8.0.1|February 07, 2008) at
 02/23/2010 15:08:32,
	Serialize complete at 02/23/2010 15:08:32
Content-Type: multipart/alternative; boundary="=_alternative 0079A185862576D3_="
X-Virus-Checked: Checked by ClamAV on apache.org

--=_alternative 0079A185862576D3_=
Content-Type: text/plain; charset="US-ASCII"

Bit of noob question here, but I haven't been around long enough to have 
observed the full lifecycle of a point release candidate yet. Given the 
generally positive feedback about 20.2 rc4, how close are we to actually 
releasing it? Is it 2 days, 1 week, 2 weeks ? 

Kind regards
Steve Watt



From:
Chris Douglas <cdouglas@apache.org>
To:
general@hadoop.apache.org
Date:
02/23/2010 03:34 PM
Subject:
Re: [VOTE] Should we release 0.20.2-rc4 ?



I looked around, and found this repository of public keys:

http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS

I'd be happy to upload my key elsewhere if necessary; I found no
documentation on that point.

We'll inevitably roll a 0.20.3, which I agree, should include 
MAPREDUCE-587 -C

On Tue, Feb 23, 2010 at 10:50 AM, Todd Lipcon <todd@cloudera.com> wrote:
> Tested download with md5: 8f40198ed18bef28aeea1401ec536cb9
> Tried to verify the GPG signature, but Chris is not in
> http://download.nextag.com/apache/hadoop/core/KEYS - he should be
> added there if he is going to sign releases.
>
> I ran unit tests on my machine at home - TestStreamingExitStatus
> failed with an OOME. I think it's exactly MAPREDUCE-587. Aside from
> that, all unit tests passed. I also ran a few jobs on a
> pseudo-distributed cluster and it worked fine.
>
> Since this is just a test bug, and in contrib, I think we should
> release anyway. Meanwhile let's commit MAPREDUCE-587 to branch-20
> before the next release. I'll reopen that JIRA.
>
> So, [non-binding] +1 from me. Thanks for the hard work, Chris.
>
> -Todd
>
> On Mon, Feb 22, 2010 at 11:52 PM, Dhruba Borthakur <dhruba@gmail.com> 
wrote:
>> +1. Looks good to me. Ran unit tests.
>>
>> -dhruba
>>
>>
>> On Fri, Feb 19, 2010 at 11:08 AM, Stack <stack@duboce.net> wrote:
>>
>>> +1
>>>
>>> I put up on a small cluster under load.  Seems to work fine.  Trolled
>>> logs a while.  Nothing out of the ordinary.  Checked docs.  They look
>>> grand.
>>>
>>> St.Ack
>>>
>>> On Fri, Feb 19, 2010 at 1:07 AM, Chris Douglas <cdouglas@apache.org>
>>> wrote:
>>> > There are now only two consistently failing testcases in my
>>> > environment, both in the capacity-scheduler contrib module:
>>> >
>>> > org.apache.hadoop.mapred.TestJobInitialization
>>> > org.apache.hadoop.mapred.TestQueueCapacities
>>> >
>>> > neither of which is a regression from 0.20.1
>>> >
>>> > http://people.apache.org/~cdouglas/0.20.2-rc4
>>> >
>>> > Please try it out. -C
>>> >
>>>
>>
>>
>>
>> --
>> Connect to me at http://www.facebook.com/dhruba
>>
>



--=_alternative 0079A185862576D3_=--

From general-return-1090-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 23 22:13:02 2010
Return-Path: <general-return-1090-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 10666 invoked from network); 23 Feb 2010 22:13:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Feb 2010 22:13:02 -0000
Received: (qmail 49906 invoked by uid 500); 23 Feb 2010 22:13:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49829 invoked by uid 500); 23 Feb 2010 22:13:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49819 invoked by uid 99); 23 Feb 2010 22:13:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 22:13:01 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 22:12:52 +0000
Received: by pwi6 with SMTP id 6so3843634pwi.35
        for <general@hadoop.apache.org>; Tue, 23 Feb 2010 14:12:32 -0800 (PST)
MIME-Version: 1.0
Received: by 10.141.107.3 with SMTP id j3mr554802rvm.271.1266963152143; Tue, 
	23 Feb 2010 14:12:32 -0800 (PST)
In-Reply-To: <OFF388BED4.4E9FA0BC-ON862576D3.0079591D-862576D3.0079A187@us.ibm.com>
References: <1267dd3b1002190107y65801a7iab7c5c33cad2daf@mail.gmail.com> 
	<7c962aed1002191108n76516c82q106e7d3e5aeb9428@mail.gmail.com> 
	<4aa34eb71002222352l394b9002jd3fd95b4ef0c06c6@mail.gmail.com> 
	<45f85f71002231050r53cc0c9bk57d61be742ba98@mail.gmail.com> 
	<1267dd3b1002231332oe60997egdf2626c3b6536573@mail.gmail.com> 
	<OFF388BED4.4E9FA0BC-ON862576D3.0079591D-862576D3.0079A187@us.ibm.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Tue, 23 Feb 2010 14:12:12 -0800
Message-ID: <45f85f71002231412i51d3c79fre9c67436b721dcf1@mail.gmail.com>
Subject: Re: [VOTE] Should we release 0.20.2-rc4 ?
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Steve,

I believe the process is that there need to be 3 "+1" votes from PMC
members. I'm not sure if the one who rolled the release counts as one
of the binding +1 votes. If so, we should have the requisite number,
and we just need to wait a few days to be sure there are no -1s before
closing the vote. If not, we need one more PMC +1.

So, to answer your question, I would guess < 1 week.

Thanks
-Todd


On Tue, Feb 23, 2010 at 2:08 PM, Stephen Watt <swatt@us.ibm.com> wrote:
> Bit of noob question here, but I haven't been around long enough to have
> observed the full lifecycle of a point release candidate yet. Given the
> generally positive feedback about 20.2 rc4, how close are we to actually
> releasing it? Is it 2 days, 1 week, 2 weeks ?
>
> Kind regards
> Steve Watt
>
>
>
> From:
> Chris Douglas <cdouglas@apache.org>
> To:
> general@hadoop.apache.org
> Date:
> 02/23/2010 03:34 PM
> Subject:
> Re: [VOTE] Should we release 0.20.2-rc4 ?
>
>
>
> I looked around, and found this repository of public keys:
>
> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>
> I'd be happy to upload my key elsewhere if necessary; I found no
> documentation on that point.
>
> We'll inevitably roll a 0.20.3, which I agree, should include
> MAPREDUCE-587 -C
>
> On Tue, Feb 23, 2010 at 10:50 AM, Todd Lipcon <todd@cloudera.com> wrote:
>> Tested download with md5: 8f40198ed18bef28aeea1401ec536cb9
>> Tried to verify the GPG signature, but Chris is not in
>> http://download.nextag.com/apache/hadoop/core/KEYS - he should be
>> added there if he is going to sign releases.
>>
>> I ran unit tests on my machine at home - TestStreamingExitStatus
>> failed with an OOME. I think it's exactly MAPREDUCE-587. Aside from
>> that, all unit tests passed. I also ran a few jobs on a
>> pseudo-distributed cluster and it worked fine.
>>
>> Since this is just a test bug, and in contrib, I think we should
>> release anyway. Meanwhile let's commit MAPREDUCE-587 to branch-20
>> before the next release. I'll reopen that JIRA.
>>
>> So, [non-binding] +1 from me. Thanks for the hard work, Chris.
>>
>> -Todd
>>
>> On Mon, Feb 22, 2010 at 11:52 PM, Dhruba Borthakur <dhruba@gmail.com>
> wrote:
>>> +1. Looks good to me. Ran unit tests.
>>>
>>> -dhruba
>>>
>>>
>>> On Fri, Feb 19, 2010 at 11:08 AM, Stack <stack@duboce.net> wrote:
>>>
>>>> +1
>>>>
>>>> I put up on a small cluster under load. =A0Seems to work fine. =A0Trol=
led
>>>> logs a while. =A0Nothing out of the ordinary. =A0Checked docs. =A0They=
 look
>>>> grand.
>>>>
>>>> St.Ack
>>>>
>>>> On Fri, Feb 19, 2010 at 1:07 AM, Chris Douglas <cdouglas@apache.org>
>>>> wrote:
>>>> > There are now only two consistently failing testcases in my
>>>> > environment, both in the capacity-scheduler contrib module:
>>>> >
>>>> > org.apache.hadoop.mapred.TestJobInitialization
>>>> > org.apache.hadoop.mapred.TestQueueCapacities
>>>> >
>>>> > neither of which is a regression from 0.20.1
>>>> >
>>>> > http://people.apache.org/~cdouglas/0.20.2-rc4
>>>> >
>>>> > Please try it out. -C
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> Connect to me at http://www.facebook.com/dhruba
>>>
>>
>
>
>

From general-return-1091-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 23 22:15:37 2010
Return-Path: <general-return-1091-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 11537 invoked from network); 23 Feb 2010 22:15:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Feb 2010 22:15:37 -0000
Received: (qmail 59715 invoked by uid 500); 23 Feb 2010 22:15:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59665 invoked by uid 500); 23 Feb 2010 22:15:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59655 invoked by uid 99); 23 Feb 2010 22:15:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 22:15:35 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 22:15:27 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o1NMEgb6010814
	for <general@hadoop.apache.org>; Tue, 23 Feb 2010 14:14:43 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=hcwXRyhubncqjOrULu9ohB6YBbkCYBvFxHVnXSIlJfQOntHjUrSahQ8hM+ufx0XO
Message-Id: <E8BBF35A-70A4-47BE-BA50-79F17AC57C7E@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <1267dd3b1002190107y65801a7iab7c5c33cad2daf@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Should we release 0.20.2-rc4 ?
Date: Tue, 23 Feb 2010 14:14:42 -0800
References: <1267dd3b1002190107y65801a7iab7c5c33cad2daf@mail.gmail.com>
X-Mailer: Apple Mail (2.936)

+1

Downloaded, verified checksums and ran tests. The usual suspects  
failed, none of which are blockers.

Arun

On Feb 19, 2010, at 1:07 AM, Chris Douglas wrote:

> There are now only two consistently failing testcases in my
> environment, both in the capacity-scheduler contrib module:
>
> org.apache.hadoop.mapred.TestJobInitialization
> org.apache.hadoop.mapred.TestQueueCapacities
>
> neither of which is a regression from 0.20.1
>
> http://people.apache.org/~cdouglas/0.20.2-rc4
>
> Please try it out. -C


From general-return-1092-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 23 22:22:55 2010
Return-Path: <general-return-1092-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14985 invoked from network); 23 Feb 2010 22:22:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Feb 2010 22:22:54 -0000
Received: (qmail 70942 invoked by uid 500); 23 Feb 2010 22:22:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70866 invoked by uid 500); 23 Feb 2010 22:22:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70856 invoked by uid 99); 23 Feb 2010 22:22:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 22:22:53 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.27.243] (HELO qmta13.emeryville.ca.mail.comcast.net) (76.96.27.243)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 22:22:44 +0000
Received: from omta23.emeryville.ca.mail.comcast.net ([76.96.30.90])
	by qmta13.emeryville.ca.mail.comcast.net with comcast
	id lVxv1d0051wfjNsADaNRoQ; Tue, 23 Feb 2010 22:22:25 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta23.emeryville.ca.mail.comcast.net with comcast
	id laN91d00P2SbwD58jaNCYC; Tue, 23 Feb 2010 22:22:19 +0000
Message-Id: <E9D34FF3-57A3-401B-A5FC-C85DC9C5D12E@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <E8BBF35A-70A4-47BE-BA50-79F17AC57C7E@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [VOTE] Should we release 0.20.2-rc4 ?
Date: Tue, 23 Feb 2010 14:22:07 -0800
References: <1267dd3b1002190107y65801a7iab7c5c33cad2daf@mail.gmail.com> <E8BBF35A-70A4-47BE-BA50-79F17AC57C7E@yahoo-inc.com>
X-Mailer: Apple Mail (2.936)

+1. Downloaded, verified the md5 and gpg signed. Ran word count on a 1  
node cluster.

-- Owen

From general-return-1093-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 23 22:28:26 2010
Return-Path: <general-return-1093-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17742 invoked from network); 23 Feb 2010 22:28:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Feb 2010 22:28:26 -0000
Received: (qmail 77109 invoked by uid 500); 23 Feb 2010 22:28:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77049 invoked by uid 500); 23 Feb 2010 22:28:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77039 invoked by uid 99); 23 Feb 2010 22:28:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 22:28:25 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 23 Feb 2010 22:28:16 +0000
Received: (qmail 17465 invoked by uid 99); 23 Feb 2010 22:27:56 -0000
Received: from localhost.apache.org (HELO mail-bw0-f211.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 22:27:56 +0000
Received: by bwz3 with SMTP id 3so3164296bwz.29
        for <general@hadoop.apache.org>; Tue, 23 Feb 2010 14:27:54 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.39.211 with SMTP id h19mr1519719bke.164.1266964074458; 
	Tue, 23 Feb 2010 14:27:54 -0800 (PST)
In-Reply-To: <45f85f71002231412i51d3c79fre9c67436b721dcf1@mail.gmail.com>
References: <1267dd3b1002190107y65801a7iab7c5c33cad2daf@mail.gmail.com>
	 <7c962aed1002191108n76516c82q106e7d3e5aeb9428@mail.gmail.com>
	 <4aa34eb71002222352l394b9002jd3fd95b4ef0c06c6@mail.gmail.com>
	 <45f85f71002231050r53cc0c9bk57d61be742ba98@mail.gmail.com>
	 <1267dd3b1002231332oe60997egdf2626c3b6536573@mail.gmail.com>
	 <OFF388BED4.4E9FA0BC-ON862576D3.0079591D-862576D3.0079A187@us.ibm.com>
	 <45f85f71002231412i51d3c79fre9c67436b721dcf1@mail.gmail.com>
Date: Tue, 23 Feb 2010 14:27:54 -0800
Message-ID: <1267dd3b1002231427i5f1d54c4n3f3fad20f19e6cf2@mail.gmail.com>
Subject: Re: [VOTE] Should we release 0.20.2-rc4 ?
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

The rules are here:

http://www.apache.org/foundation/voting.html#ReleaseVotes

This says that one cannot veto a release, but 3 days should be
sufficient for everyone to test it out and find problems. If nobody
finds anything sufficient to block the release, I'll try to push it on
Wednesday or Thursday. -C

On Tue, Feb 23, 2010 at 2:12 PM, Todd Lipcon <todd@cloudera.com> wrote:
> Hi Steve,
>
> I believe the process is that there need to be 3 "+1" votes from PMC
> members. I'm not sure if the one who rolled the release counts as one
> of the binding +1 votes. If so, we should have the requisite number,
> and we just need to wait a few days to be sure there are no -1s before
> closing the vote. If not, we need one more PMC +1.
>
> So, to answer your question, I would guess < 1 week.
>
> Thanks
> -Todd
>
>
> On Tue, Feb 23, 2010 at 2:08 PM, Stephen Watt <swatt@us.ibm.com> wrote:
>> Bit of noob question here, but I haven't been around long enough to have
>> observed the full lifecycle of a point release candidate yet. Given the
>> generally positive feedback about 20.2 rc4, how close are we to actually
>> releasing it? Is it 2 days, 1 week, 2 weeks ?
>>
>> Kind regards
>> Steve Watt
>>
>>
>>
>> From:
>> Chris Douglas <cdouglas@apache.org>
>> To:
>> general@hadoop.apache.org
>> Date:
>> 02/23/2010 03:34 PM
>> Subject:
>> Re: [VOTE] Should we release 0.20.2-rc4 ?
>>
>>
>>
>> I looked around, and found this repository of public keys:
>>
>> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>>
>> I'd be happy to upload my key elsewhere if necessary; I found no
>> documentation on that point.
>>
>> We'll inevitably roll a 0.20.3, which I agree, should include
>> MAPREDUCE-587 -C
>>
>> On Tue, Feb 23, 2010 at 10:50 AM, Todd Lipcon <todd@cloudera.com> wrote:
>>> Tested download with md5: 8f40198ed18bef28aeea1401ec536cb9
>>> Tried to verify the GPG signature, but Chris is not in
>>> http://download.nextag.com/apache/hadoop/core/KEYS - he should be
>>> added there if he is going to sign releases.
>>>
>>> I ran unit tests on my machine at home - TestStreamingExitStatus
>>> failed with an OOME. I think it's exactly MAPREDUCE-587. Aside from
>>> that, all unit tests passed. I also ran a few jobs on a
>>> pseudo-distributed cluster and it worked fine.
>>>
>>> Since this is just a test bug, and in contrib, I think we should
>>> release anyway. Meanwhile let's commit MAPREDUCE-587 to branch-20
>>> before the next release. I'll reopen that JIRA.
>>>
>>> So, [non-binding] +1 from me. Thanks for the hard work, Chris.
>>>
>>> -Todd
>>>
>>> On Mon, Feb 22, 2010 at 11:52 PM, Dhruba Borthakur <dhruba@gmail.com>
>> wrote:
>>>> +1. Looks good to me. Ran unit tests.
>>>>
>>>> -dhruba
>>>>
>>>>
>>>> On Fri, Feb 19, 2010 at 11:08 AM, Stack <stack@duboce.net> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> I put up on a small cluster under load. =A0Seems to work fine. =A0Tro=
lled
>>>>> logs a while. =A0Nothing out of the ordinary. =A0Checked docs. =A0The=
y look
>>>>> grand.
>>>>>
>>>>> St.Ack
>>>>>
>>>>> On Fri, Feb 19, 2010 at 1:07 AM, Chris Douglas <cdouglas@apache.org>
>>>>> wrote:
>>>>> > There are now only two consistently failing testcases in my
>>>>> > environment, both in the capacity-scheduler contrib module:
>>>>> >
>>>>> > org.apache.hadoop.mapred.TestJobInitialization
>>>>> > org.apache.hadoop.mapred.TestQueueCapacities
>>>>> >
>>>>> > neither of which is a regression from 0.20.1
>>>>> >
>>>>> > http://people.apache.org/~cdouglas/0.20.2-rc4
>>>>> >
>>>>> > Please try it out. -C
>>>>> >
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Connect to me at http://www.facebook.com/dhruba
>>>>
>>>
>>
>>
>>
>

From general-return-1094-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 23 22:52:54 2010
Return-Path: <general-return-1094-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30005 invoked from network); 23 Feb 2010 22:52:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Feb 2010 22:52:54 -0000
Received: (qmail 8161 invoked by uid 500); 23 Feb 2010 22:52:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8118 invoked by uid 500); 23 Feb 2010 22:52:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8108 invoked by uid 99); 23 Feb 2010 22:52:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 22:52:53 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 22:52:43 +0000
Received: from oceanfarearth-lm.corp.yahoo.com (oceanfarearth-lm.corp.yahoo.com [10.72.113.156])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o1NMpTPA027315
	for <general@hadoop.apache.org>; Tue, 23 Feb 2010 14:51:29 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=h86R7iaX1ezARQQqtA3W6m4dH+2WUaoGWeLNQHQUDet7TFs4+Nb2sZB/DZfy7U5s
Message-Id: <C98531C5-694A-4224-A9C0-C15A8ED5FF76@yahoo-inc.com>
From: Sanjay Radia <sradia@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <dfe484f01002191514i5fe5b31cg1e884693737a3c93@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Release plans
Date: Tue, 23 Feb 2010 14:51:29 -0800
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com> <4B7D9503.4080205@apache.org> <DA9E7EF1-9A82-414E-AA5C-DD1088869CD2@apache.org> <c9118d4c1002181506l7ec9840dq3690158ebc70c02@mail.gmail.com> <EFFFAD8D-1998-4C19-9BD3-D80B3B79577A@apache.org> <7c962aed1002182036l7db697ahcdc581fde3619405@mail.gmail.com> <dfe484f01002191344u27c3c24eoe5fb5c5aa5701658@mail.gmail.com> <4B7F0985.3010000@apache.org> <dfe484f01002191419g691e7651xe589a0a4a1307918@mail.gmail.com> <4B7F157C.1090305@apache.org> <dfe484f01002191514i5fe5b31cg1e884693737a3c93@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


On Feb 19, 2010, at 3:14 PM, Eli Collins wrote:

> On Fri, Feb 19, 2010 at 2:49 PM, Doug Cutting <cutting@apache.org>  
> wrote:
>> Eli Collins wrote:
>>>>
>>>> My concern is more about when the next branch from trunk will be  
>>>> made.
>>>
>>> Does there need to be a dependency?
>>
>> No.  I just wanted to note that, from my point of view, releasing  
>> from the
>> existing 0.21 branch is not sufficient, that, regardless, we still  
>> need to
>> release from trunk soon, and we need a schedule going forward for  
>> when
>> future branches from trunk will be made.
>>
>
> Agreed. Releasing a 21 won't settle the release process question or
> tell us when the first major release is. Maybe we could get 21 behind
> us and have a separate discussion covering those.
>
>>> Could the vote on whether 22 is backwards compatible with 20 be
>>> independent of what we call 1.0?
>>
>> We minimally need to declare whether releases are major or minor.
>
> I was assuming 21 would be another minor release, didn't hear
> otherwise when it was branched. I didn't interpret the compatibility
> vote as a suggestion that we should retroactively consider 20 to be
> the first major release, but rather a suggestion for voting that we
> don't remove APIs in 22 that were deprecated in 20.
Agreed.
Rather than renaming versions, lets vote on the compatibility rules  
for 22: 22 is API compatible with 20.


> Owen, please
> correct me if I'm wrong.
>
> Thanks,
> Eli


From general-return-1095-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 23 23:35:18 2010
Return-Path: <general-return-1095-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 48635 invoked from network); 23 Feb 2010 23:35:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Feb 2010 23:35:18 -0000
Received: (qmail 65338 invoked by uid 500); 23 Feb 2010 23:35:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65256 invoked by uid 500); 23 Feb 2010 23:35:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65246 invoked by uid 99); 23 Feb 2010 23:35:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 23:35:17 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 23 Feb 2010 23:35:15 +0000
Received: (qmail 48576 invoked by uid 99); 23 Feb 2010 23:34:53 -0000
Received: from localhost.apache.org (HELO [192.168.168.105]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Feb 2010 23:34:53 +0000
Message-ID: <4B84661C.1000600@apache.org>
Date: Tue, 23 Feb 2010 15:34:52 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Release plans
References: <dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com> <4B7D9503.4080205@apache.org> <DA9E7EF1-9A82-414E-AA5C-DD1088869CD2@apache.org> <c9118d4c1002181506l7ec9840dq3690158ebc70c02@mail.gmail.com> <EFFFAD8D-1998-4C19-9BD3-D80B3B79577A@apache.org> <7c962aed1002182036l7db697ahcdc581fde3619405@mail.gmail.com> <dfe484f01002191344u27c3c24eoe5fb5c5aa5701658@mail.gmail.com> <4B7F0985.3010000@apache.org> <dfe484f01002191419g691e7651xe589a0a4a1307918@mail.gmail.com> <4B7F157C.1090305@apache.org> <dfe484f01002191514i5fe5b31cg1e884693737a3c93@mail.gmail.com> <C98531C5-694A-4224-A9C0-C15A8ED5FF76@yahoo-inc.com>
In-Reply-To: <C98531C5-694A-4224-A9C0-C15A8ED5FF76@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Sanjay Radia wrote:
> Rather than renaming versions, lets vote on the compatibility rules for 
> 22: 22 is API compatible with 20.

I think if we elect to go this way then we should make the rule more 
general, i.e.: all releases after 0.20 are to be API-compatible with 
0.20 until we state otherwise (probably 1.0).  If we both release 0.21 
as currently branched and we branch 0.22 from trunk in a few weeks, then 
we might want to have an API-compatible 0.23 too.

Doug

From general-return-1096-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 24 05:27:01 2010
Return-Path: <general-return-1096-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 86620 invoked from network); 24 Feb 2010 05:27:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Feb 2010 05:27:01 -0000
Received: (qmail 78888 invoked by uid 500); 24 Feb 2010 05:27:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78781 invoked by uid 500); 24 Feb 2010 05:27:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 12024 invoked by uid 99); 23 Feb 2010 22:53:58 -0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1266965607; bh=q9YlvEMaYdSQ2+tocvqL1GwxX9Dh9A2WOIc2UwHOAkE=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=LEuP8XQzKDdeetX41XCh7D0OVwE70jzckMIkt9EQZmdDk7Vj7hp1gCHrQVLqrNh3RlbAVRokV0mFvSfpBm3HWVtypuHkqR2LLoObZ7uxNvONSssx49xsVkW8fVdeshfLjRqgNNPr5zLcfWC1K8zzs7sNvEgWP0DK17o9n5xPCSc=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type;
  b=tTKFXBUk3hpCOL5L5l3c/AW8ewNB/uuPipX7Fe7bp2GJuUMA902aYoj3zCedqpPSm5Gu+EtSbFmK4A0avj4ArlHxTQBkVoBAJGE15NnuCuAhrzrjAVxdE8vJTx/bIHsdmYoB7it38rcoAVNDS7oudIEoQvGVKIUMUR/LL4/iDyU=;
Message-ID: <311992.9657.qm@web58701.mail.re1.yahoo.com>
X-YMail-OSG: 3Eerq4wVM1nwO07t_CqlvI6OZ_eJHF6ELaa4AnQN2DLkZ4B
 aJEPrErmp3hn12g3GHb4R82bxKSLMeIl4C9EYA9N_O2vGeUKelGLRGfiRbz0
 F7FdRGLsycUoLAaHI_DHukOJyKucyQPcHQcGTMM37C4zFwyjBxmqZZV2Ky5C
 SNNu5dkWJLgx4SYmQTz9qybud9m5D5LmCAKmW9fbDwhpwukuRtzpgbvo7wrb
 3iVgh8UnDJb9bsCty0QYJq6NW3IX5hbqFN2QXj54hH_jL1eK2iI3hkec2NXz
 ASH.NFOuvw9PzysQvSGev9cVcIWlt2As.wE94BOunuxJ7utLJgFcLYJ.WWzn
 rI.rD09OnaLH0aGXfLNcYbXEAxVAjdCWbrUX9DKalc38fIr6D
X-Mailer: YahooMailRC/300.3 YahooMailWebService/0.8.100.260964
References: <3316F847-FBB7-4D41-B1A7-C5CD3FB1A40D@yahoo-inc.com> <14B7BB1E-F459-4FBC-AEBD-120B822281E3@yahoo-inc.com> <1E049E93BEE249D59D88B0DDACEF9106@DanielLaptop> <466AB331-82A3-4111-BA34-75F44A31DBCF@yahoo-inc.com> <E744EA5B-ECE1-44CD-B4F7-E75A53FCBA19@yahoo-inc.com>
Date: Tue, 23 Feb 2010 14:53:27 -0800 (PST)
From: Pradeep Kamath <pradeepkth@yahoo.com>
Subject: Re: [VOTE] Release candidate for Pig 0.6.0
To: private@hadoop.apache.org, general@hadoop.apache.org
Cc: private@hadoop.apache.org
In-Reply-To: <E744EA5B-ECE1-44CD-B4F7-E75A53FCBA19@yahoo-inc.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-231140451-1266965607=:9657"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-231140451-1266965607=:9657
Content-Type: text/plain; charset=us-ascii

+1, md5 matches - extracted and ran simple scripts in local and map reduce mode - ran "ant test-commit" sucessfully.




________________________________
From: Alan Gates <gates@yahoo-inc.com>
To: general@hadoop.apache.org
Cc: private@hadoop.apache.org
Sent: Tue, February 23, 2010 12:05:44 PM
Subject: Re: [VOTE] Release candidate for Pig 0.6.0

I have rolled a new version of the tarball that has the version command correct.  I did not make any code changes, I just needed to run ant with the correct properties set.  So I have not made this a new release candidate.  So the candidate is at the same place as before, http://people.apache.org/~gates/pig-0.6.0-candidate-1/

Please try it out and vote by Friday, February 26th.  Thanks.

Alan.

On Feb 19, 2010, at 1:57 PM, Alan Gates wrote:

> Alright, I'll fix that and reroll the release.
> 
> Alan.
> 
> On Feb 19, 2010, at 1:35 PM, Daniel Dai wrote:
> 
>> "pig -version" should report 0.6.0 instead of 0.6.1. +1 otherwise.
>> 
>> 
>> --------------------------------------------------
>> From: "Alan Gates" <gates@yahoo-inc.com>
>> Sent: Friday, February 19, 2010 11:58 AM
>> To: <private@hadoop.apache.org>
>> Subject: Fwd: [VOTE] Release candidate for Pig 0.6.0
>> 
>>> PMC members,
>>> 
>>> Please take a look and vote.  Thanks.
>>> 
>>> Alan.
>>> 
>>> Begin forwarded message:
>>> 
>>>> From: Alan Gates <gates@yahoo-inc.com>
>>>> Date: February 16, 2010 11:22:52 AM PST
>>>> To: <general@hadoop.apache.org>
>>>> Subject: [VOTE] Release candidate for Pig 0.6.0
>>>> Reply-To: general@hadoop.apache.org
>>>> 
>>>> All,
>>>> 
>>>> I have rolled a release candidate for Pig 0.6.0 and uploaded it to http://people.apache.org/~gates/pig-0.6.0-candidate-1/
>>>> A description of what is new and different in this release is  included in the release notes, which I have also uploaded to that  directory. Please download the candidate, test it, and vote.
>>>> 
>>>> Alan.
> 


      
--0-231140451-1266965607=:9657--

From general-return-1097-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 24 08:58:59 2010
Return-Path: <general-return-1097-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66525 invoked from network); 24 Feb 2010 08:58:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Feb 2010 08:58:58 -0000
Received: (qmail 70158 invoked by uid 500); 24 Feb 2010 08:58:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70081 invoked by uid 500); 24 Feb 2010 08:58:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69750 invoked by uid 99); 24 Feb 2010 08:58:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Feb 2010 08:58:55 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [88.198.2.104] (HELO koch.ro) (88.198.2.104)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Feb 2010 08:58:46 +0000
Received: from 84-72-85-88.dclient.hispeed.ch ([84.72.85.88] helo=jona.localnet)
	by koch.ro with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
	(Exim 4.69)
	(envelope-from <thomas@koch.ro>)
	id 1NkD3A-0000pb-Jl
	for general@hadoop.apache.org; Wed, 24 Feb 2010 09:56:56 +0100
From: Thomas Koch <thomas@koch.ro>
Reply-To: thomas@koch.ro
To: general@hadoop.apache.org
Subject: additional source only release tarball
Date: Wed, 24 Feb 2010 09:53:02 +0100
User-Agent: KMail/1.12.4 (Linux/2.6.32-2-amd64; KDE/4.3.4; x86_64; ; )
MIME-Version: 1.0
Content-Type: text/plain;
  charset="us-ascii"
Content-Transfer-Encoding: 7bit
Message-Id: <201002240953.02613.thomas@koch.ro>
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

I've packaged hadoop for Debian. (It's still in the upload queue for 
verification[1]).
One common annoyance when packaging java applications for a Free Software 
distribution is the necessity to repackage the upstream tarball. The 
repackaging is necessary, because Debian may only distribute binary files 
build from source that's also available from Debian.
So we build the jar/war files ourselfes to make sure there's nothing we don't 
have the sources for.
It would take one (annoying and time consuming) step less for packagers, if 
java upstream projects would release an additional tarball without any binary 
files or third party code.
I'm asking you first, because many other projects (like zookeeper) took or 
take hadoop as an example for their build infrastructure.
For your orientation, these are the patterns that I used to filter the hadoop 
tarball: (Usable with tar --exclude)

"*.jar",
"uming.*",
"prototype.js",
"config.sub",
"config.guess",
"ltmain.sh",
"Makefile.in",
"configure",
"aclocal.m4",
"config.h.in",
"install-sh",
"autom4te.cache",
"depcomp",
"missing",
"pipes/compile",
"src/contrib/eclipse-plugin/resources/*.jpg",
"src/contrib/eclipse-plugin/resources/*.png",
"src/contrib/eclipse-plugin/resources/*.gif",
"hadoop-0.20.1/src/core/org/apache/hadoop/record/compiler/generated/*.java",
"hadoop-0.20.1/src/docs/cn/build",
"hadoop-0.20.1/c++",
"hadoop-0.20.1/contrib",
"hadoop-0.20.1/lib/native",
"hadoop-0.20.1/librecordio",
"hadoop-0.20.1/src/contrib/thriftfs/gen-*",
"hadoop-0.20.1/docs",

There were different reasons why stuff needed to be filtered:
- unclear license (uming.*)
- unclear origin (images in the eclipse plugin)
- precompiled documentation / code / hadoop binaries
- pregenerated C/C++ automake files
- third party libraries (prototype.js, lib/*.jar)
 
If you'd be willing to release an additional tarball like this, I'd provide 
the necessary patch to your build.xml

[1] http://ftp-master.debian.org/new.html

Best regards,

Thomas Koch, http://www.koch.ro

From general-return-1098-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 24 16:29:02 2010
Return-Path: <general-return-1098-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 75876 invoked from network); 24 Feb 2010 16:29:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Feb 2010 16:29:02 -0000
Received: (qmail 24162 invoked by uid 500); 24 Feb 2010 16:29:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24104 invoked by uid 500); 24 Feb 2010 16:29:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24094 invoked by uid 99); 24 Feb 2010 16:29:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Feb 2010 16:29:00 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Feb 2010 16:28:50 +0000
Received: from [192.168.1.71] (snvvpn4-10-72-168-c173.hq.corp.yahoo.com [10.72.168.173])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o1OGSOvV031403;
	Wed, 24 Feb 2010 08:28:24 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=amTB+dTwMeRD2gC39UHr6D3K6u8JMJrAr+1IPt0ea9Wy23lvtDxG105mA3hFLbaZ
Message-Id: <E11411EB-3536-4AD3-876E-DF8B7501BE65@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: <general@hadoop.apache.org>, <thomas@koch.ro>
In-Reply-To: <201002240953.02613.thomas@koch.ro>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: additional source only release tarball
Date: Wed, 24 Feb 2010 08:28:23 -0800
References: <201002240953.02613.thomas@koch.ro>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

  This seems reasonable, anyone else?

  Thomas, could you, please, open a jira and attach your patch to  
build.xml there? I have some comments on some of your exclude  
patterns, but we can have that discussion on the jira.

thanks,
Arun

On Feb 24, 2010, at 12:53 AM, Thomas Koch wrote:

> Hi,
>
> I've packaged hadoop for Debian. (It's still in the upload queue for
> verification[1]).
> One common annoyance when packaging java applications for a Free  
> Software
> distribution is the necessity to repackage the upstream tarball. The
> repackaging is necessary, because Debian may only distribute binary  
> files
> build from source that's also available from Debian.
> So we build the jar/war files ourselfes to make sure there's nothing  
> we don't
> have the sources for.
> It would take one (annoying and time consuming) step less for  
> packagers, if
> java upstream projects would release an additional tarball without  
> any binary
> files or third party code.
> I'm asking you first, because many other projects (like zookeeper)  
> took or
> take hadoop as an example for their build infrastructure.
> For your orientation, these are the patterns that I used to filter  
> the hadoop
> tarball: (Usable with tar --exclude)
>
> "*.jar",
> "uming.*",
> "prototype.js",
> "config.sub",
> "config.guess",
> "ltmain.sh",
> "Makefile.in",
> "configure",
> "aclocal.m4",
> "config.h.in",
> "install-sh",
> "autom4te.cache",
> "depcomp",
> "missing",
> "pipes/compile",
> "src/contrib/eclipse-plugin/resources/*.jpg",
> "src/contrib/eclipse-plugin/resources/*.png",
> "src/contrib/eclipse-plugin/resources/*.gif",
> "hadoop-0.20.1/src/core/org/apache/hadoop/record/compiler/generated/ 
> *.java",
> "hadoop-0.20.1/src/docs/cn/build",
> "hadoop-0.20.1/c++",
> "hadoop-0.20.1/contrib",
> "hadoop-0.20.1/lib/native",
> "hadoop-0.20.1/librecordio",
> "hadoop-0.20.1/src/contrib/thriftfs/gen-*",
> "hadoop-0.20.1/docs",
>
> There were different reasons why stuff needed to be filtered:
> - unclear license (uming.*)
> - unclear origin (images in the eclipse plugin)
> - precompiled documentation / code / hadoop binaries
> - pregenerated C/C++ automake files
> - third party libraries (prototype.js, lib/*.jar)
>
> If you'd be willing to release an additional tarball like this, I'd  
> provide
> the necessary patch to your build.xml
>
> [1] http://ftp-master.debian.org/new.html
>
> Best regards,
>
> Thomas Koch, http://www.koch.ro


From general-return-1099-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 24 21:54:56 2010
Return-Path: <general-return-1099-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 48380 invoked from network); 24 Feb 2010 21:54:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Feb 2010 21:54:56 -0000
Received: (qmail 27950 invoked by uid 500); 24 Feb 2010 21:54:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27874 invoked by uid 500); 24 Feb 2010 21:54:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27864 invoked by uid 99); 24 Feb 2010 21:54:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Feb 2010 21:54:54 +0000
X-ASF-Spam-Status: No, hits=-1.8 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [192.101.109.31] (HELO emailgw03.pnl.gov) (192.101.109.31)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Feb 2010 21:54:46 +0000
X-IronPort-AV: E=Sophos;i="4.49,534,1262592000"; 
   d="scan'208,217";a="16110821"
Received: from emailhub02.pnl.gov ([130.20.251.62])
  by emailgw03.pnl.gov with ESMTP/TLS/AES128-SHA; 24 Feb 2010 13:54:26 -0800
Received: from Email05.pnl.gov ([169.254.2.77]) by emailhub02.pnl.gov
 ([130.20.251.62]) with mapi; Wed, 24 Feb 2010 13:54:25 -0800
From: "Taylor, Ronald C" <ronald.taylor@pnl.gov>
To: "'general@hadoop.apache.org'" <general@hadoop.apache.org>
CC: "Taylor, Ronald C" <ronald.taylor@pnl.gov>, "Trease, Harold E"
	<het@pnl.gov>
Date: Wed, 24 Feb 2010 13:54:25 -0800
Subject: Is there Hadoop support for parallel reading of a compressed video
 file  - say as *.tar or *.zip file?
Thread-Topic: Is there Hadoop support for parallel reading of a compressed
 video file  - say as *.tar or *.zip file?
Thread-Index: Acq1m+2OgYjg/iBNS2isS3Y+FSOfQg==
Message-ID: <A632B3F93D861843A3C966F3962C3F84010639E135CB@EMAIL05.pnl.gov>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_A632B3F93D861843A3C966F3962C3F84010639E135CBEMAIL05pnlg_"
MIME-Version: 1.0

--_000_A632B3F93D861843A3C966F3962C3F84010639E135CBEMAIL05pnlg_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable


Hello Hadoop list,

We need to process huge files of video quickly in Hadoop. Think, for exampl=
e, of a 100 Gb YouTube video file being uploaded to our cluster  in *.tar o=
r *.zip format. We need a reader, I presume some variant on LineRecordReade=
r, that can automatically split the file appropriately between a set of Map=
pers, one Mapper per node.

I did a quick Google search and found something about work being done on a =
BZip2 compressed text file reader - the email mentions this work as issue H=
ADOOP-4012. Could anybody tell me the state of such work, or of similar wor=
k for *.zip and *.tar files? Is there any working code available?

Also: more generally, we need Hadoop-based code that can automatically spli=
t (compressed) video files at appropriate boundaries for processing in para=
llel. Our group would deeply appreciate any guidance pointing toward curren=
t Hadoop work in this area.

  Cheers,
    Ron Taylor

___________________________________________
Ronald Taylor, Ph.D.
Computational Biology & Bioinformatics Group
Pacific Northwest National Laboratory
902 Battelle Boulevard
P.O. Box 999, Mail Stop J4-33
Richland, WA  99352 USA
Office:  509-372-6568
Email: ronald.taylor@pnl.gov



--_000_A632B3F93D861843A3C966F3962C3F84010639E135CBEMAIL05pnlg_--

From general-return-1100-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 25 13:21:29 2010
Return-Path: <general-return-1100-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30988 invoked from network); 25 Feb 2010 13:21:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Feb 2010 13:21:29 -0000
Received: (qmail 39481 invoked by uid 500); 25 Feb 2010 13:21:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39426 invoked by uid 500); 25 Feb 2010 13:21:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39418 invoked by uid 99); 25 Feb 2010 13:21:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Feb 2010 13:21:27 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Feb 2010 13:21:16 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 8B71CB7E9E
	for <general@hadoop.apache.org>; Thu, 25 Feb 2010 13:20:55 +0000 (GMT)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id hOZFv3nj7pJW for <general@hadoop.apache.org>;
	Thu, 25 Feb 2010 13:20:54 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 73CDCB7E9D
	for <general@hadoop.apache.org>; Thu, 25 Feb 2010 13:20:54 +0000 (GMT)
MailScanner-NULL-Check: 1267708843.9569@raricJWinZ9uo6WVx+WQ5A
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o1PDKfkn014480
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Thu, 25 Feb 2010 13:20:43 GMT
Message-ID: <4B867929.2030702@apache.org>
Date: Thu, 25 Feb 2010 13:20:41 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: additional source only release tarball
References: <201002240953.02613.thomas@koch.ro> <E11411EB-3536-4AD3-876E-DF8B7501BE65@yahoo-inc.com>
In-Reply-To: <E11411EB-3536-4AD3-876E-DF8B7501BE65@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o1PDKfkn014480
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

Arun C Murthy wrote:
>  This seems reasonable, anyone else?
> 
>  Thomas, could you, please, open a jira and attach your patch to 
> build.xml there? I have some comments on some of your exclude patterns, 
> but we can have that discussion on the jira.
> 
> thanks,
> Arun

One thing I've discussed before is the whole notion of having some 
hadoop-redist project whose aim is to produce useful 
packaging/installation of the artifacts of the main Hadoop projects.

  -RPMs
  -Debs
  -stuff to help the various CM tools work w/ Hadoop
  -things to help is deploy and test at scale


This may seem trivial, but it isn't. Right now my test process is
  * create my own RPMs
  * scp them to a machine with some scripts (that I don't run) to
  * ssh to that machine, run some scripts and create a VM disk image 
containing the RPMs and the installer scripts
  * bring up the controller front end to the infrastructure

Then I run a junit test
  * Have htmlunit start talking to a web ui
  * through it, ask for 20 VMs
  * go get my 16th coffee of the day, assuming this is test run #14 
(first two coffees are at home, one before the school run, the other 
while doing the first round of emails)
  * have the controller wait for the VMs to come up
  * as they do, for each one, run a thread that SSHs in  and actually 
pushes out the Hadoop config based on the hostnames given to the 
different VMs
  * run terasort
  * grab the logs
  * teardown
  * merge the logs in with the rest of the junit output
  * try and work out why the test failed

oh, and everything from the 20 vms onwards is started by junit test

To make life more fun, that bit of [1] has to spin waiting for the DNS 
names to resolve, the machines to boot enough for SSH to work, then SSH 
in and spin there waiting for the installed programs to be on the path 
and then the CM tooling to be live. And feed the log of this not just to 
the local log4j log but (somehow) stream it back over the (remote) web 
ui that has to then live update the web page (this is surprisingly hard)

Here's a bit of it, excluding the coffees

            PortUtils.waitForHostnameToResolve(hostname, 
factory.getPortConnectTimeout(),
                     SLEEP_TIME_FOR_HOSTNAME_RESOLUTION);
           PortUtils.checkPort(hostname, factory.getPort(), 
factory.getPortConnectTimeout());

	//some stuff creating SSH sessions, SCP-ing over files skipped

             //make a few attempts to find the startup command
             for (int i = 0; i < STARTUP_LOCATE_ATTEMPTS; i++) {
                 commandsList.add("which " + SF_START + " || sleep " + 
STARTUP_SLEEP_TIME);
             }
             commandsList.add("which " + SF_START + " || echo " + 
ERROR_NO_EXECUTABLE + SF_START);
             String sfPing = makeSFCommand(SF_PING) + " " + "localhost";
             for (int i = 0; i < STARTUP_PING_ATTEMPTS; i++) {
                 commandsList.add(sfPing + " || sleep " + 
STARTUP_PING_SLEEP_TIME);
             }
             commandsList.add(sshCommand);
             commandsList.add("sleep " + SLEEP_TIME);
             if (!factory.isKeepFiles()) {
                 commandsList.add("rm " + desttempfile);
             }
             commandsList.add("exit");
             sshExec(session, commandsList);

See that? fun. JUnit test cases that fail in interesting ways. Really 
interesting ways. Sometimes it's the infrastructure, sometimes it's my 
code. Sometimes, because I am working with SVN_HEAD, it's Hadoop. 
Problem is, it takes a while to track down, especially if the VMs are no 
longer there at the end of the run.

Right now one of the problems is that none of Hadoop's JSP pages work. I 
can submit jobs, talk to the FS, etc, but all the JSP pages are failing 
and the root of every web is /webapps ; I think it's something that 
Todd's also seeing, but I can' t make it go away, not without fixing how 
Hadoop locates/registers webapps.

Returning to the idea of a deployment project, having more tests that 
can be run against production clusters would be good: all the tests 
should be redistributables that you can deploy/run. There is something 
needed to QA a cluster, and ideally identify the approximate area where 
problems lie (e.g. DNS playing up, rDNS not working, clocks out of sync, 
etc)

thoughts?

-steve

From general-return-1101-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 25 15:10:50 2010
Return-Path: <general-return-1101-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88376 invoked from network); 25 Feb 2010 15:10:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Feb 2010 15:10:50 -0000
Received: (qmail 57856 invoked by uid 500); 25 Feb 2010 15:10:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57812 invoked by uid 500); 25 Feb 2010 15:10:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57800 invoked by uid 99); 25 Feb 2010 15:10:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Feb 2010 15:10:49 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [88.198.2.104] (HELO koch.ro) (88.198.2.104)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Feb 2010 15:10:39 +0000
Received: from 245-214.76-83.cust.bluewin.ch ([83.76.214.245] helo=jona.localnet)
	by koch.ro with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
	(Exim 4.69)
	(envelope-from <thomas@koch.ro>)
	id 1NkfKV-000350-6t; Thu, 25 Feb 2010 16:08:43 +0100
From: Thomas Koch <thomas@koch.ro>
Reply-To: thomas@koch.ro
To: general@hadoop.apache.org
Subject: Re: additional source only release tarball
Date: Thu, 25 Feb 2010 16:10:11 +0100
User-Agent: KMail/1.12.4 (Linux/2.6.32-2-amd64; KDE/4.3.4; x86_64; ; )
Cc: Steve Loughran <stevel@apache.org>
References: <201002240953.02613.thomas@koch.ro> <E11411EB-3536-4AD3-876E-DF8B7501BE65@yahoo-inc.com> <4B867929.2030702@apache.org>
In-Reply-To: <4B867929.2030702@apache.org>
MIME-Version: 1.0
Content-Type: Text/Plain;
  charset="us-ascii"
Content-Transfer-Encoding: 7bit
Message-Id: <201002251610.11798.thomas@koch.ro>
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Steve,

> One thing I've discussed before is the whole notion of having some
> hadoop-redist project whose aim is to produce useful
> packaging/installation of the artifacts of the main Hadoop projects.
you may be interested in project-builder.org. It's a tool to integrate the 
building of distribution packages in a continuous integration workflow. There 
has been a presentation at the European Free Software conference FOSDEM:

Continuous Packaging with Project-Builder.org
http://meetings-archive.debian.net/pub/debian-meetings/2010/fosdem10/

Best regards,

Thomas Koch, http://www.koch.ro

From general-return-1102-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 25 16:40:01 2010
Return-Path: <general-return-1102-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22578 invoked from network); 25 Feb 2010 16:40:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Feb 2010 16:40:01 -0000
Received: (qmail 21290 invoked by uid 500); 25 Feb 2010 16:40:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21223 invoked by uid 500); 25 Feb 2010 16:40:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21213 invoked by uid 99); 25 Feb 2010 16:40:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Feb 2010 16:40:00 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Feb 2010 16:39:51 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 7814E1BA760
	for <general@hadoop.apache.org>; Thu, 25 Feb 2010 16:39:29 +0000 (GMT)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id 9br9zM8CO0Qs for <general@hadoop.apache.org>;
	Thu, 25 Feb 2010 16:39:28 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by colossus.hpl.hp.com (Postfix) with ESMTPS id B58B61BA75F
	for <general@hadoop.apache.org>; Thu, 25 Feb 2010 16:39:28 +0000 (GMT)
MailScanner-NULL-Check: 1267720758.20624@PKyasK0NPu7vVg+x6JyOcw
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o1PGdGlj019600
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Thu, 25 Feb 2010 16:39:17 GMT
Message-ID: <4B86A7B5.9000301@apache.org>
Date: Thu, 25 Feb 2010 16:39:17 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: additional source only release tarball
References: <201002240953.02613.thomas@koch.ro> <E11411EB-3536-4AD3-876E-DF8B7501BE65@yahoo-inc.com> <4B867929.2030702@apache.org> <201002251610.11798.thomas@koch.ro>
In-Reply-To: <201002251610.11798.thomas@koch.ro>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o1PGdGlj019600
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

Thomas Koch wrote:
> Hi Steve,
> 
>> One thing I've discussed before is the whole notion of having some
>> hadoop-redist project whose aim is to produce useful
>> packaging/installation of the artifacts of the main Hadoop projects.
> you may be interested in project-builder.org. It's a tool to integrate the 
> building of distribution packages in a continuous integration workflow. There 
> has been a presentation at the European Free Software conference FOSDEM:
> 
> Continuous Packaging with Project-Builder.org
> http://meetings-archive.debian.net/pub/debian-meetings/2010/fosdem10/
> 
> Best regards,
> 
> Thomas Koch, http://www.koch.ro

thanks, I'll look at it. I do have the CI tools creating RPMs right now, 
scp-ing to VMs, though the fact that the rpm tools (especially rpm -qf 
to verify that a file exists/is managed) doesn't return error/success 
codes, so I have to grab the ssh output and scan for error strings, 
which we all think is wrong.

From general-return-1103-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 25 17:26:55 2010
Return-Path: <general-return-1103-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44631 invoked from network); 25 Feb 2010 17:26:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Feb 2010 17:26:55 -0000
Received: (qmail 7557 invoked by uid 500); 25 Feb 2010 17:26:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7466 invoked by uid 500); 25 Feb 2010 17:26:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 59994 invoked by uid 99); 25 Feb 2010 06:40:24 -0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of susheel.varma@gmail.com designates 209.85.210.173 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type:content-transfer-encoding;
        bh=TRqXDMnXE+TEIiosI5cxg+tSNsmST4plzvx5IdSrhaM=;
        b=D1QBBpto4g//0nI9X50UlHUFO/Xo6HNsT16JcmFBJqTaMNrnwhPOIQ18/rhhCfL7Oe
         b1Fieka5Y1UvGLrB+fr5Sp1gLWkH74VDWzICRaigd8awu6TlPRkSnWqwv/xPnSdjJjxv
         b8UpXvCJR/GswNF1+6GT+pbfSgWdW8pxp1iAA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=k+OGiPiUtoY1DJCEdA2vgtAuSR/HW2uwoDg/Ul8hVdm0G7H8iu/3C5r/8IPevfbwc4
         MsX6SYok+Z2a55dy/D75MR/cVQ086yLanDGviHSlEDYZHduyexh9mdEtCaPTlFsSoKso
         Lajs1t542cwBh7omSfTGR20DYsK4HqM3InzWo=
MIME-Version: 1.0
Date: Thu, 25 Feb 2010 06:39:56 +0000
Message-ID: <7488e7731002242239u10b7d870r17983d03efed647e@mail.gmail.com>
Subject: Metadata, Daemons, Benchmarks, Web UI
From: Susheel Varma <susheel.varma@gmail.com>
To: general <general@hadoop.apache.org>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

We are trying evaluate a small set of distributed data management
solutions(iRODS, HDFS, Lustre) for our project. We don't really have a
need for scalable computation, but rather our focus is more on
redundancy, reliability and security. Although small bits computation
would be needed at some level. We have just begun an evaluation of
HDFS, however this has thrown up a few questions(a good thing, I
guess):

1. Metadata & Links
a. Is there a way to add/update/remove metadata held on the NameNode? Examp=
les?
b. Is there a way to get hold of the metadata held on the NameNode?
I'd like to allow users to search the HDFS using the the metadata, and
then resolve the query to actual link to the file.

If a or b is not possible, I would have to resort to using an HBase to
store the custom metadata.=C2=A0Examples?

2. Daemons
a. Is there a way to setup a daemon job(map-reduce, or otherwise) to
listen for filesystem events and trigger actions that need to be
performed on these files? Examples?
b. If not, Is there an FSEvent API I could use? Examples?

3. Benchmarks
a. Are there any un/published HDFS filesystem benchmarks using IOZone,
PostMark etc.=C2=A0I know almost all FS benchmarks must be taken with a
pinch of salt, but I'd really like to see the quantitative comparisons
with Lustre for example.

4. Web UI
a. Is there a simple way to augment the NameNode Web UI to allow users
to login, search and download the files on the backend filesystem?
Examples?
b. If not, could you show me examples where users have combined a web
application to serve files store on the HDFS?

Thanks
Susheel

From general-return-1104-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 25 17:32:07 2010
Return-Path: <general-return-1104-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47866 invoked from network); 25 Feb 2010 17:32:06 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Feb 2010 17:32:06 -0000
Received: (qmail 16795 invoked by uid 500); 25 Feb 2010 17:32:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16715 invoked by uid 500); 25 Feb 2010 17:32:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16702 invoked by uid 99); 25 Feb 2010 17:32:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Feb 2010 17:32:06 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Feb 2010 17:31:56 +0000
Received: from [10.73.135.249] (wifi-e-135-249.corp.yahoo.com [10.73.135.249])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o1PHVXGh011197;
	Thu, 25 Feb 2010 09:31:33 -0800 (PST)
Message-ID: <4B86B3F5.60504@apache.org>
Date: Thu, 25 Feb 2010 09:31:33 -0800
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: additional source only release tarball
References: <201002240953.02613.thomas@koch.ro> <E11411EB-3536-4AD3-876E-DF8B7501BE65@yahoo-inc.com> <4B867929.2030702@apache.org> <201002251610.11798.thomas@koch.ro> <4B86A7B5.9000301@apache.org>
In-Reply-To: <4B86A7B5.9000301@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Thomas, not sure if this solves all your issues but have you considered 
using SVN rather than the release tarball? The latest versions of Hadoop 
and ZooKeeper have moved to ant/ivy, while HBase has moved to Maven (and 
ZooKeeper has been considering moving to Maven at some point). As a 
result I'm not sure how much overlap there will be going forward wrt a 
common build...

If you use the release tag from SVN you are guaranteed to get the same 
source as is contained in the release artifact. However the release 
artifact contains additional things like lib dependencies (typically the 
results of "ant package" or "ant tar")

By moving to SVN you would guarantee that you have the same source but 
since ivy/maven are used to pull the dependencies (for the most part, 
there may be some exceptions) you'd essentially get what you are looking 
for.

Patrick

Steve Loughran wrote:
> Thomas Koch wrote:
>> Hi Steve,
>>
>>> One thing I've discussed before is the whole notion of having some
>>> hadoop-redist project whose aim is to produce useful
>>> packaging/installation of the artifacts of the main Hadoop projects.
>> you may be interested in project-builder.org. It's a tool to integrate 
>> the building of distribution packages in a continuous integration 
>> workflow. There has been a presentation at the European Free Software 
>> conference FOSDEM:
>>
>> Continuous Packaging with Project-Builder.org
>> http://meetings-archive.debian.net/pub/debian-meetings/2010/fosdem10/
>>
>> Best regards,
>>
>> Thomas Koch, http://www.koch.ro
> 
> thanks, I'll look at it. I do have the CI tools creating RPMs right now, 
> scp-ing to VMs, though the fact that the rpm tools (especially rpm -qf 
> to verify that a file exists/is managed) doesn't return error/success 
> codes, so I have to grab the ssh output and scan for error strings, 
> which we all think is wrong.

From general-return-1105-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 25 17:57:44 2010
Return-Path: <general-return-1105-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 61255 invoked from network); 25 Feb 2010 17:57:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Feb 2010 17:57:44 -0000
Received: (qmail 65630 invoked by uid 500); 25 Feb 2010 17:57:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65447 invoked by uid 500); 25 Feb 2010 17:57:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65426 invoked by uid 99); 25 Feb 2010 17:57:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Feb 2010 17:57:42 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Feb 2010 17:57:35 +0000
Received: by pwi6 with SMTP id 6so5329056pwi.35
        for <general@hadoop.apache.org>; Thu, 25 Feb 2010 09:57:15 -0800 (PST)
MIME-Version: 1.0
Received: by 10.141.4.8 with SMTP id g8mr752650rvi.182.1267120635088; Thu, 25 
	Feb 2010 09:57:15 -0800 (PST)
In-Reply-To: <7488e7731002242239u10b7d870r17983d03efed647e@mail.gmail.com>
References: <7488e7731002242239u10b7d870r17983d03efed647e@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Thu, 25 Feb 2010 09:56:54 -0800
Message-ID: <45f85f71002250956l102e5396gef52f6e175d0f359@mail.gmail.com>
Subject: Re: Metadata, Daemons, Benchmarks, Web UI
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd0eba257914304807083c1

--000e0cd0eba257914304807083c1
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Feb 24, 2010 at 10:39 PM, Susheel Varma <susheel.varma@gmail.com>wrote:

> Hi,
>
> We are trying evaluate a small set of distributed data management
> solutions(iRODS, HDFS, Lustre) for our project. We don't really have a
> need for scalable computation, but rather our focus is more on
> redundancy, reliability and security. Although small bits computation
> would be needed at some level. We have just begun an evaluation of
> HDFS, however this has thrown up a few questions(a good thing, I
> guess):
>
> 1. Metadata & Links
> a. Is there a way to add/update/remove metadata held on the NameNode?
> Examples?
> b. Is there a way to get hold of the metadata held on the NameNode?
> I'd like to allow users to search the HDFS using the the metadata, and
> then resolve the query to actual link to the file.
>
> If a or b is not possible, I would have to resort to using an HBase to

store the custom metadata. Examples?
>

Can you please explain what you mean by metadata? Do you mean arbitrary
attributes on files? (like extended attributes in linux?)


>
> 2. Daemons
> a. Is there a way to setup a daemon job(map-reduce, or otherwise) to
> listen for filesystem events and trigger actions that need to be
> performed on these files? Examples?
> b. If not, Is there an FSEvent API I could use? Examples?
>

Nope. I think someone may have opened a JIRA for an inotify-like API, but
this does not exist currently. Polling is the best bet. Alternatively, you
could run a daemon on the same host as the NN which tails the audit log (or
write a log4j appender for the audit log) to watch for events on the trigger
files. This would be a nice contribution.


>
> 3. Benchmarks
> a. Are there any un/published HDFS filesystem benchmarks using IOZone,
> PostMark etc. I know almost all FS benchmarks must be taken with a
> pinch of salt, but I'd really like to see the quantitative comparisons
> with Lustre for example.
>
>
I don't think so - most of the existing benchmark tools require a POSIX
filesystem (eg they test random write). There was a thread some time back
about running iozone (minus the random write benchmarks) on HDFS, but I'm
not sure what the result was.

My hunch is that you will find it is competitive for sequential workloads
and less competitive for random workloads. That said, I know Brian Bockelman
(pretty active on this list) uses HDFS heavily with a primarily random-read
workload on a huge scientific dataset with good results.


> 4. Web UI
> a. Is there a simple way to augment the NameNode Web UI to allow users
> to login, search and download the files on the backend filesystem?
> Examples?
>

The existing NN UI has a "browse the filesystem" link. From there you can
navigate to a file and click "Download this file".

Search is not provided, as there is no efficient indexing and an O(number of
files) walk of the filesystem is prohibitively expensive for large NNs.


> b. If not, could you show me examples where users have combined a web
> application to serve files store on the HDFS?
>
> Thanks
> Susheel
>

-Todd

--000e0cd0eba257914304807083c1--

From general-return-1106-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 25 18:02:57 2010
Return-Path: <general-return-1106-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 65036 invoked from network); 25 Feb 2010 18:02:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Feb 2010 18:02:57 -0000
Received: (qmail 81843 invoked by uid 500); 25 Feb 2010 18:02:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81702 invoked by uid 500); 25 Feb 2010 18:02:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81692 invoked by uid 99); 25 Feb 2010 18:02:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Feb 2010 18:02:56 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 25 Feb 2010 18:02:55 +0000
Received: (qmail 64899 invoked by uid 99); 25 Feb 2010 18:02:35 -0000
Received: from localhost.apache.org (HELO [192.168.168.105]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Feb 2010 18:02:35 +0000
Message-ID: <4B86BB3A.9080103@apache.org>
Date: Thu, 25 Feb 2010 10:02:34 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: additional source only release tarball
References: <201002240953.02613.thomas@koch.ro> <E11411EB-3536-4AD3-876E-DF8B7501BE65@yahoo-inc.com> <4B867929.2030702@apache.org> <201002251610.11798.thomas@koch.ro> <4B86A7B5.9000301@apache.org> <4B86B3F5.60504@apache.org>
In-Reply-To: <4B86B3F5.60504@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Patrick Hunt wrote:
> Thomas, not sure if this solves all your issues but have you considered 
> using SVN rather than the release tarball?

FWIW, in Avro we've moved away from a single release tarball that 
includes dependencies.  Rather the primary release artifact is a source 
tarball generated with 'svn export'.  In addition we distribute derived 
artifacts as conveniences, like a documentation tarball, a java jar & 
pom for upload to maven, a java executable jar for command-line tools, a 
python egg, a ruby gem, C & C++ pristine tarballs, etc.

> If you use the release tag from SVN you are guaranteed to get the
> same source as is contained in the release artifact.

"Guarantee" is a strong word.  You certainly should get the same source, 
but the source that's voted on, that's checksummed and that's signed is 
the release artifact, not the SVN tag.  It's a bug if this is not the 
same as the release tag, but not impossible.

Doug


From general-return-1107-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 25 18:25:08 2010
Return-Path: <general-return-1107-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 79973 invoked from network); 25 Feb 2010 18:25:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Feb 2010 18:25:08 -0000
Received: (qmail 39745 invoked by uid 500); 25 Feb 2010 18:25:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39609 invoked by uid 500); 25 Feb 2010 18:25:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39597 invoked by uid 99); 25 Feb 2010 18:25:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Feb 2010 18:25:07 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Feb 2010 18:24:59 +0000
Received: from [10.73.135.249] (wifi-e-135-249.corp.yahoo.com [10.73.135.249])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o1PINrD3015957;
	Thu, 25 Feb 2010 10:23:53 -0800 (PST)
Message-ID: <4B86C039.2090309@apache.org>
Date: Thu, 25 Feb 2010 10:23:53 -0800
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: additional source only release tarball
References: <201002240953.02613.thomas@koch.ro> <E11411EB-3536-4AD3-876E-DF8B7501BE65@yahoo-inc.com> <4B867929.2030702@apache.org> <201002251610.11798.thomas@koch.ro> <4B86A7B5.9000301@apache.org> <4B86B3F5.60504@apache.org> <4B86BB3A.9080103@apache.org>
In-Reply-To: <4B86BB3A.9080103@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org


Doug Cutting wrote:
> Patrick Hunt wrote:
>> If you use the release tag from SVN you are guaranteed to get the
>> same source as is contained in the release artifact.
> 
> "Guarantee" is a strong word.  You certainly should get the same source, 
> but the source that's voted on, that's checksummed and that's signed is 
> the release artifact, not the SVN tag.  It's a bug if this is not the 
> same as the release tag, but not impossible.

Ah, thanks for clarify that Doug. To take it a bit further, when you say 
"bug" you really mean "serious breach of Apache process/rules", would 
that be valid? i.e. it would be something that the responsible Apache 
team should work to address with highest of priority.

Patrick

ps. I do like this Avro idea of a separate src artifact (re the signing 
esp).

From general-return-1108-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 25 18:43:22 2010
Return-Path: <general-return-1108-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95725 invoked from network); 25 Feb 2010 18:43:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Feb 2010 18:43:22 -0000
Received: (qmail 68284 invoked by uid 500); 25 Feb 2010 18:43:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68219 invoked by uid 500); 25 Feb 2010 18:43:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68211 invoked by uid 99); 25 Feb 2010 18:43:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Feb 2010 18:43:21 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 25 Feb 2010 18:43:20 +0000
Received: (qmail 95457 invoked by uid 99); 25 Feb 2010 18:42:58 -0000
Received: from localhost.apache.org (HELO [192.168.168.105]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Feb 2010 18:42:58 +0000
Message-ID: <4B86C4B1.9050206@apache.org>
Date: Thu, 25 Feb 2010 10:42:57 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: additional source only release tarball
References: <201002240953.02613.thomas@koch.ro> <E11411EB-3536-4AD3-876E-DF8B7501BE65@yahoo-inc.com> <4B867929.2030702@apache.org> <201002251610.11798.thomas@koch.ro> <4B86A7B5.9000301@apache.org> <4B86B3F5.60504@apache.org> <4B86BB3A.9080103@apache.org> <4B86C039.2090309@apache.org>
In-Reply-To: <4B86C039.2090309@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Patrick Hunt wrote:
> Ah, thanks for clarify that Doug. To take it a bit further, when you say 
> "bug" you really mean "serious breach of Apache process/rules", would 
> that be valid? i.e. it would be something that the responsible Apache 
> team should work to address with highest of priority.

To some degree that depends on the Apache project.  I don't know of a 
project that does not create release tags and that would accept an 
incorrect one lightly.  That said, release tags are not required nor 
authoritative: the thing that counts is the signed artifact.

I'd certainly encourage developers to leverage tags when convenient 
e.g., for automated testing against and comparison with prior releases, 
for IDE source browsing, etc.  But if someone wants to package an 
alternate distribution of an Apache release, I think they're better 
starting from the release artifact than the tag.  The artifact can be 
validated against the signature at http://www.apache.org/dist/, while 
there's currently no good means of validating the contents of a tag.  I 
suppose one could rebuild the tarball from the tag and try to validate 
its checksum against that at http://www.apache.org/dist/, but that seems 
both fragile and less secure.

Doug

From general-return-1109-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 25 18:45:54 2010
Return-Path: <general-return-1109-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96736 invoked from network); 25 Feb 2010 18:45:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Feb 2010 18:45:54 -0000
Received: (qmail 70417 invoked by uid 500); 25 Feb 2010 18:45:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70385 invoked by uid 500); 25 Feb 2010 18:45:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70371 invoked by uid 99); 25 Feb 2010 18:45:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Feb 2010 18:45:53 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Feb 2010 18:45:46 +0000
Received: from [10.73.135.249] (wifi-e-135-249.corp.yahoo.com [10.73.135.249])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o1PIj5nQ021810;
	Thu, 25 Feb 2010 10:45:05 -0800 (PST)
Message-ID: <4B86C52E.8090808@apache.org>
Date: Thu, 25 Feb 2010 10:45:02 -0800
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: additional source only release tarball
References: <201002240953.02613.thomas@koch.ro> <E11411EB-3536-4AD3-876E-DF8B7501BE65@yahoo-inc.com> <4B867929.2030702@apache.org> <201002251610.11798.thomas@koch.ro> <4B86A7B5.9000301@apache.org> <4B86B3F5.60504@apache.org> <4B86BB3A.9080103@apache.org> <4B86C039.2090309@apache.org> <4B86C4B1.9050206@apache.org>
In-Reply-To: <4B86C4B1.9050206@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Ok, that's good to know. Thanks.

Patrick

Doug Cutting wrote:
> Patrick Hunt wrote:
>> Ah, thanks for clarify that Doug. To take it a bit further, when you 
>> say "bug" you really mean "serious breach of Apache process/rules", 
>> would that be valid? i.e. it would be something that the responsible 
>> Apache team should work to address with highest of priority.
> 
> To some degree that depends on the Apache project.  I don't know of a 
> project that does not create release tags and that would accept an 
> incorrect one lightly.  That said, release tags are not required nor 
> authoritative: the thing that counts is the signed artifact.
> 
> I'd certainly encourage developers to leverage tags when convenient 
> e.g., for automated testing against and comparison with prior releases, 
> for IDE source browsing, etc.  But if someone wants to package an 
> alternate distribution of an Apache release, I think they're better 
> starting from the release artifact than the tag.  The artifact can be 
> validated against the signature at http://www.apache.org/dist/, while 
> there's currently no good means of validating the contents of a tag.  I 
> suppose one could rebuild the tarball from the tag and try to validate 
> its checksum against that at http://www.apache.org/dist/, but that seems 
> both fragile and less secure.
> 
> Doug

From general-return-1110-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 26 06:28:17 2010
Return-Path: <general-return-1110-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1413 invoked from network); 26 Feb 2010 06:28:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Feb 2010 06:28:17 -0000
Received: (qmail 98226 invoked by uid 500); 26 Feb 2010 06:28:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98173 invoked by uid 500); 26 Feb 2010 06:28:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98165 invoked by uid 99); 26 Feb 2010 06:28:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Feb 2010 06:28:15 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW
X-Spam-Check-By: apache.org
Received-SPF: unknown (athena.apache.org: error in processing during lookup of dkumar@qasource.com)
Received: from [161.58.59.72] (HELO qasource.com) (161.58.59.72)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Feb 2010 06:28:07 +0000
Received: from [192.16.17.127] ([203.193.132.44])
	(authenticated bits=0)
	by qasource.com (8.13.6.20060614/8.13.6) with ESMTP id o1Q6Rio1024283
	for <general@hadoop.apache.org>; Fri, 26 Feb 2010 06:27:46 GMT
Message-ID: <4B876A2E.7080903@qasource.com>
Date: Fri, 26 Feb 2010 11:59:02 +0530
From: Devender <dkumar@qasource.com>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.7) Gecko/20100111 Lightning/1.0b1 Thunderbird/3.0.1
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Automation/Performance Testing on Hadoop
References: <201002240953.02613.thomas@koch.ro> <E11411EB-3536-4AD3-876E-DF8B7501BE65@yahoo-inc.com> <4B867929.2030702@apache.org> <201002251610.11798.thomas@koch.ro> <4B86A7B5.9000301@apache.org> <4B86B3F5.60504@apache.org> <4B86BB3A.9080103@apache.org> <4B86C039.2090309@apache.org> <4B86C4B1.9050206@apache.org> <4B86C52E.8090808@apache.org>
In-Reply-To: <4B86C52E.8090808@apache.org>
Content-Type: multipart/alternative;
 boundary="------------000605000104000404060103"

--------------000605000104000404060103
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi,

I had few questions on Hadoop,

1. Is there any way to extract data from Hadoop (data will be in XML) 
and building customer usage models?
2. Creating tests using those models to drive HTTP, SMTP, etc.
3. Automating the running of those tests on a variety of test matrices 
at varying load levels to determine system maxima while monitoring 
system resource parameters to identify bottlenecks.
4.  producing graphs/charts based on Excel, perl, web GUI, etc.

5. Which performance tools can be used for this?
6. Which Automation tools can be used for this?

Kindly provide your feedback if any one have any information?

Regards,
Dev



--------------000605000104000404060103--

From general-return-1111-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 26 09:29:27 2010
Return-Path: <general-return-1111-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 74459 invoked from network); 26 Feb 2010 09:29:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Feb 2010 09:29:27 -0000
Received: (qmail 97403 invoked by uid 500); 26 Feb 2010 09:29:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97372 invoked by uid 500); 26 Feb 2010 09:29:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97363 invoked by uid 99); 26 Feb 2010 09:29:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Feb 2010 09:29:26 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 26 Feb 2010 09:29:17 +0000
Received: (qmail 74277 invoked by uid 99); 26 Feb 2010 09:28:57 -0000
Received: from localhost.apache.org (HELO mail-bw0-f211.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Feb 2010 09:28:57 +0000
Received: by bwz3 with SMTP id 3so5334436bwz.29
        for <general@hadoop.apache.org>; Fri, 26 Feb 2010 01:28:55 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.36.202 with SMTP id u10mr93930bkd.65.1267176535666; Fri, 
	26 Feb 2010 01:28:55 -0800 (PST)
In-Reply-To: <1267dd3b1002231427i5f1d54c4n3f3fad20f19e6cf2@mail.gmail.com>
References: <1267dd3b1002190107y65801a7iab7c5c33cad2daf@mail.gmail.com>
	 <7c962aed1002191108n76516c82q106e7d3e5aeb9428@mail.gmail.com>
	 <4aa34eb71002222352l394b9002jd3fd95b4ef0c06c6@mail.gmail.com>
	 <45f85f71002231050r53cc0c9bk57d61be742ba98@mail.gmail.com>
	 <1267dd3b1002231332oe60997egdf2626c3b6536573@mail.gmail.com>
	 <OFF388BED4.4E9FA0BC-ON862576D3.0079591D-862576D3.0079A187@us.ibm.com>
	 <45f85f71002231412i51d3c79fre9c67436b721dcf1@mail.gmail.com>
	 <1267dd3b1002231427i5f1d54c4n3f3fad20f19e6cf2@mail.gmail.com>
Date: Fri, 26 Feb 2010 01:28:55 -0800
Message-ID: <1267dd3b1002260128h642c36a3p7779f4749c779f4c@mail.gmail.com>
Subject: Re: [VOTE] Should we release 0.20.2-rc4 ?
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

With 4 +1s, the vote passes. I'll push out the release. -C

On Tue, Feb 23, 2010 at 2:27 PM, Chris Douglas <cdouglas@apache.org> wrote:
> The rules are here:
>
> http://www.apache.org/foundation/voting.html#ReleaseVotes
>
> This says that one cannot veto a release, but 3 days should be
> sufficient for everyone to test it out and find problems. If nobody
> finds anything sufficient to block the release, I'll try to push it on
> Wednesday or Thursday. -C
>
> On Tue, Feb 23, 2010 at 2:12 PM, Todd Lipcon <todd@cloudera.com> wrote:
>> Hi Steve,
>>
>> I believe the process is that there need to be 3 "+1" votes from PMC
>> members. I'm not sure if the one who rolled the release counts as one
>> of the binding +1 votes. If so, we should have the requisite number,
>> and we just need to wait a few days to be sure there are no -1s before
>> closing the vote. If not, we need one more PMC +1.
>>
>> So, to answer your question, I would guess < 1 week.
>>
>> Thanks
>> -Todd
>>
>>
>> On Tue, Feb 23, 2010 at 2:08 PM, Stephen Watt <swatt@us.ibm.com> wrote:
>>> Bit of noob question here, but I haven't been around long enough to hav=
e
>>> observed the full lifecycle of a point release candidate yet. Given the
>>> generally positive feedback about 20.2 rc4, how close are we to actuall=
y
>>> releasing it? Is it 2 days, 1 week, 2 weeks ?
>>>
>>> Kind regards
>>> Steve Watt
>>>
>>>
>>>
>>> From:
>>> Chris Douglas <cdouglas@apache.org>
>>> To:
>>> general@hadoop.apache.org
>>> Date:
>>> 02/23/2010 03:34 PM
>>> Subject:
>>> Re: [VOTE] Should we release 0.20.2-rc4 ?
>>>
>>>
>>>
>>> I looked around, and found this repository of public keys:
>>>
>>> http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS
>>>
>>> I'd be happy to upload my key elsewhere if necessary; I found no
>>> documentation on that point.
>>>
>>> We'll inevitably roll a 0.20.3, which I agree, should include
>>> MAPREDUCE-587 -C
>>>
>>> On Tue, Feb 23, 2010 at 10:50 AM, Todd Lipcon <todd@cloudera.com> wrote=
:
>>>> Tested download with md5: 8f40198ed18bef28aeea1401ec536cb9
>>>> Tried to verify the GPG signature, but Chris is not in
>>>> http://download.nextag.com/apache/hadoop/core/KEYS - he should be
>>>> added there if he is going to sign releases.
>>>>
>>>> I ran unit tests on my machine at home - TestStreamingExitStatus
>>>> failed with an OOME. I think it's exactly MAPREDUCE-587. Aside from
>>>> that, all unit tests passed. I also ran a few jobs on a
>>>> pseudo-distributed cluster and it worked fine.
>>>>
>>>> Since this is just a test bug, and in contrib, I think we should
>>>> release anyway. Meanwhile let's commit MAPREDUCE-587 to branch-20
>>>> before the next release. I'll reopen that JIRA.
>>>>
>>>> So, [non-binding] +1 from me. Thanks for the hard work, Chris.
>>>>
>>>> -Todd
>>>>
>>>> On Mon, Feb 22, 2010 at 11:52 PM, Dhruba Borthakur <dhruba@gmail.com>
>>> wrote:
>>>>> +1. Looks good to me. Ran unit tests.
>>>>>
>>>>> -dhruba
>>>>>
>>>>>
>>>>> On Fri, Feb 19, 2010 at 11:08 AM, Stack <stack@duboce.net> wrote:
>>>>>
>>>>>> +1
>>>>>>
>>>>>> I put up on a small cluster under load. =A0Seems to work fine. =A0Tr=
olled
>>>>>> logs a while. =A0Nothing out of the ordinary. =A0Checked docs. =A0Th=
ey look
>>>>>> grand.
>>>>>>
>>>>>> St.Ack
>>>>>>
>>>>>> On Fri, Feb 19, 2010 at 1:07 AM, Chris Douglas <cdouglas@apache.org>
>>>>>> wrote:
>>>>>> > There are now only two consistently failing testcases in my
>>>>>> > environment, both in the capacity-scheduler contrib module:
>>>>>> >
>>>>>> > org.apache.hadoop.mapred.TestJobInitialization
>>>>>> > org.apache.hadoop.mapred.TestQueueCapacities
>>>>>> >
>>>>>> > neither of which is a regression from 0.20.1
>>>>>> >
>>>>>> > http://people.apache.org/~cdouglas/0.20.2-rc4
>>>>>> >
>>>>>> > Please try it out. -C
>>>>>> >
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Connect to me at http://www.facebook.com/dhruba
>>>>>
>>>>
>>>
>>>
>>>
>>
>

From general-return-1112-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 26 18:10:56 2010
Return-Path: <general-return-1112-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95833 invoked from network); 26 Feb 2010 18:10:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Feb 2010 18:10:55 -0000
Received: (qmail 50786 invoked by uid 500); 26 Feb 2010 18:10:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50744 invoked by uid 500); 26 Feb 2010 18:10:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50686 invoked by uid 99); 26 Feb 2010 18:10:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Feb 2010 18:10:53 +0000
X-ASF-Spam-Status: No, hits=4.7 required=10.0
	tests=SPF_NEUTRAL,URIBL_BLACK
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.13.13.86] (HELO n2d.bullet.mail.ac4.yahoo.com) (76.13.13.86)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 26 Feb 2010 18:10:43 +0000
Received: from [74.6.228.94] by n2.bullet.mail.ac4.yahoo.com with NNFMP; 26 Feb 2010 18:10:22 -0000
Received: from [76.13.10.167] by t1.bullet.mail.ac4.yahoo.com with NNFMP; 26 Feb 2010 18:10:22 -0000
Received: from [127.0.0.1] by omp108.mail.ac4.yahoo.com with NNFMP; 26 Feb 2010 18:10:22 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 180180.13144.bm@omp108.mail.ac4.yahoo.com
Received: (qmail 11373 invoked by uid 60001); 26 Feb 2010 18:10:22 -0000
Message-ID: <850621.10792.qm@web65510.mail.ac4.yahoo.com>
X-YMail-OSG: _5jUdv4VM1k9KTf.Y4Al.DS4hnr.CmVrkJrVgkjkhISduUbdP2XxTAhuYCy3kwIVjGRHWbox.xkzbGqufadynFtmKS6LFRFrnQKX_8EtaxiTm7zjyze27N2x2nI73XKUblwrUKgptdtBYB7ukoCFWm3TC7DhTphUIWSTP.C39o6E5bDr65plBmQmXvfPXRj9XteC2Fq595yYnSfJfV8SQqAzApKxNdD.FMwn0zLufLAYDty0HTY0Uq.KbritbobZGlS7fVm9uwkBr7mflIY8AlSPSK3Xh6gRRya5ShCbNFCRzRjDublfSTpuqnbeSC97zRNJQd02jL9gkJBjqhk-
Received: from [69.111.188.126] by web65510.mail.ac4.yahoo.com via HTTP; Fri, 26 Feb 2010 10:10:21 PST
X-RocketYMMF: apurtell
X-Mailer: YahooMailRC/300.3 YahooMailWebService/0.8.100.260964
Date: Fri, 26 Feb 2010 10:10:21 -0800 (PST)
From: Andrew Purtell <apurtell@apache.org>
Subject: HUG9 @ Mozilla March 10th, register at http://su.pr/4pe8Of
To: hbase-user@hadoop.apache.org, general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii

To any interested parties,

The 9th edition of the HBase Users' Group  is kindly being sponsored and hosted by Mozilla in Mountain View (California) on March 10th.  
Register at http://su.pr/4pe8Of 

Yours,

The HBaseistas


      


From general-return-1113-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 26 21:56:44 2010
Return-Path: <general-return-1113-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77485 invoked from network); 26 Feb 2010 21:56:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Feb 2010 21:56:44 -0000
Received: (qmail 30931 invoked by uid 500); 26 Feb 2010 21:56:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30629 invoked by uid 500); 26 Feb 2010 21:56:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30590 invoked by uid 99); 26 Feb 2010 21:56:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Feb 2010 21:56:42 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of zshao9@gmail.com designates 209.85.223.193 as permitted sender)
Received: from [209.85.223.193] (HELO mail-iw0-f193.google.com) (209.85.223.193)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Feb 2010 21:56:33 +0000
Received: by iwn31 with SMTP id 31so625881iwn.30
        for <multiple recipients>; Fri, 26 Feb 2010 13:56:13 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=XzENaykAvSgdyhPPKrB0B5lZ7moQDAfCK6+ZcR0bWOU=;
        b=Xg9SKxAF1jCNxuWw0OB5FLkzMyeYXZC30xKszulqLn+lAJYUUuIlU+d0lRc5kw5wL3
         IQPnPj4MU1z6sUF+0f2RvLN0r8cfx5+R+jR7w2/n7pCA8bctaORhvoGJpcqlIob//885
         zu7wtY9SELKCGl0WkiTleOkm4akCWxC1IgmX8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=c9j9Ne/FbP0uBYZ1NNDScm6NOwY2HYhssa/ojGXi/dB/l34kg+AAu59ajVx9MD09F0
         jUbTh4GP4F+XPa5gjpilXgNy0Y/GOi37rtKFsotWJBygOcHcvwRA7Rqkxb1nguxKiHQF
         zVnq8cUEBnpTwh8ks3XUTMczEslQGxUwm3J6A=
MIME-Version: 1.0
Received: by 10.231.157.83 with SMTP id a19mr764859ibx.41.1267221372918; Fri, 
	26 Feb 2010 13:56:12 -0800 (PST)
Date: Fri, 26 Feb 2010 13:56:12 -0800
Message-ID: <34fd060d1002261356u1f37e824x8f879d20ba1ebdb6@mail.gmail.com>
Subject: Hive User Group Meeting 3/18/2010 7pm at Facebook
From: Zheng Shao <zshao9@gmail.com>
To: hive-user@hadoop.apache.org, hive-dev@hadoop.apache.org, 
	core-user@hadoop.apache.org, general@hadoop.apache.org, 
	common-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hi all,

We are going to hold the second Hive User Group Meeting at 7PM on
3/18/2010 Thursday.

The agenda will be:

* Hive Tutorial: 20 min
* Hive User Case Study: 20 min
* New Features and API: 25 min
 JDBC/ODBC and CTAS
 UDF/UDAF/UDTF
 Create View/HBaseInputFormat
 Hive Join Strategy
 SerDe

The audience is beginner to intermediate Hive users/developers.

*** The details are here: http://www.facebook.com/event.php?eid=319237846974 ***
*** Please RSVP so we can schedule logistics accordingly. ***

-- 
Yours,
Zheng

From general-return-240-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 09 01:31:40 2009
Return-Path: <general-return-240-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17910 invoked from network); 9 Jun 2009 01:31:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jun 2009 01:31:40 -0000
Received: (qmail 61590 invoked by uid 500); 9 Jun 2009 01:31:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61364 invoked by uid 500); 9 Jun 2009 01:31:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61221 invoked by uid 99); 9 Jun 2009 01:31:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Jun 2009 01:31:50 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [207.126.228.149] (HELO rsmtp1.corp.yahoo.com) (207.126.228.149)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Jun 2009 01:31:42 +0000
Received: from [10.73.153.22] (snvvpn1-10-73-153-c22.hq.corp.yahoo.com [10.73.153.22])
	(authenticated bits=0)
	by rsmtp1.corp.yahoo.com (8.13.8/8.13.8/y.rout) with ESMTP id n591VDuX086034
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Mon, 8 Jun 2009 18:31:14 -0700 (PDT)
Message-ID: <4A2DBB61.5010707@apache.org>
Date: Mon, 08 Jun 2009 18:31:13 -0700
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.21 (X11/20090318)
MIME-Version: 1.0
To: "zookeeper-user@hadoop.apache.org" <zookeeper-user@hadoop.apache.org>,
        "zookeeper-dev@hadoop.apache.org" <zookeeper-dev@hadoop.apache.org>,
        general@hadoop.apache.org
Subject: Show your ZooKeeper pride!
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

The Hadoop summit is Wednesday. If you're attending please feel free to 
say hi -- Mahadev is presenting @4, Ben and I will be attending as well.

Also, regardless of whether you're attending or not we'd appreciate any 
updates to the "powered by" page, if you're too busy to update it 
yourself send us a snippet and we'll update it for you ;-)

http://wiki.apache.org/hadoop/ZooKeeper/PoweredBy

Regards,

Patrick

From general-return-241-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 09 02:07:13 2009
Return-Path: <general-return-241-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35326 invoked from network); 9 Jun 2009 02:07:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jun 2009 02:07:12 -0000
Received: (qmail 94064 invoked by uid 500); 9 Jun 2009 02:07:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93949 invoked by uid 500); 9 Jun 2009 02:07:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93921 invoked by uid 99); 9 Jun 2009 02:07:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Jun 2009 02:07:20 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of nitayj@gmail.com designates 74.125.46.29 as permitted sender)
Received: from [74.125.46.29] (HELO yw-out-2324.google.com) (74.125.46.29)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Jun 2009 02:07:09 +0000
Received: by yw-out-2324.google.com with SMTP id 9so1945095ywe.29
        for <multiple recipients>; Mon, 08 Jun 2009 19:06:48 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:cc:content-type;
        bh=qywph0KMWmf25LAIBnw8yXHQRRXeemxTut1yB05vUF0=;
        b=kFkX8bizh4GLmPwm9hHjFZJE94ACPfI/C3WLSDY6sp8i93O2rcJl2/n4b17w3onGQx
         8vzG7V1Rnk/L2Nb3cccBNxO78K2MhCgmsKECZXiV91lcvEJyKJvK0C9nUGo6cGjATbw7
         9e9e+93xqepOBpToBmJrSRLbk+jKL/+THRR5U=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=mkdOgPgNhkqsq7irAo4gTPRpmLrXBLCU6xCYZbBURqXRUc6GvSsdugSKLiFtIcOK8i
         4NhSExjGuL/CD7eugLOCWrpLqnyKyhvOkCzFWxyDtYqSGW67NMz7Llq5zOPWv6EoNvBv
         UsnLWyoMb7xpWEDjlJs1NZBPCkWqOsBErCH64=
MIME-Version: 1.0
Received: by 10.151.144.13 with SMTP id w13mr13812089ybn.281.1244513208395; 
	Mon, 08 Jun 2009 19:06:48 -0700 (PDT)
In-Reply-To: <c7d45fc70906081901p7c2d8f3eibbe513939a643e96@mail.gmail.com>
References: <4A2DBB61.5010707@apache.org>
	 <c7d45fc70906081901p7c2d8f3eibbe513939a643e96@mail.gmail.com>
Date: Mon, 8 Jun 2009 19:06:48 -0700
Message-ID: <82b0992a0906081906p67cb7877pc1f6212dc0d87c6f@mail.gmail.com>
Subject: Re: Show your ZooKeeper pride!
From: Nitay <nitayj@gmail.com>
To: zookeeper-user@hadoop.apache.org
Cc: "zookeeper-dev@hadoop.apache.org" <zookeeper-dev@hadoop.apache.org>, general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00151750dcacb467ef046be0cf91
X-Virus-Checked: Checked by ClamAV on apache.org

--00151750dcacb467ef046be0cf91
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Added HBase.

On Mon, Jun 8, 2009 at 7:01 PM, Ted Dunning <ted.dunning@gmail.com> wrote:

>  How come Yahoo isn't listed?
>
> On Mon, Jun 8, 2009 at 6:31 PM, Patrick Hunt <phunt@apache.org> wrote:
>
> > The Hadoop summit is Wednesday. If you're attending please feel free to
> say
> > hi -- Mahadev is presenting @4, Ben and I will be attending as well.
> >
> > Also, regardless of whether you're attending or not we'd appreciate any
> > updates to the "powered by" page, if you're too busy to update it
> yourself
> > send us a snippet and we'll update it for you ;-)
> >
> > http://wiki.apache.org/hadoop/ZooKeeper/PoweredBy
> >
> > Regards,
> >
> > Patrick
> >
>
>
>
> --
> Ted Dunning, CTO
> DeepDyve
>
> 111 West Evelyn Ave. Ste. 202
> Sunnyvale, CA 94086
> http://www.deepdyve.com
> 858-414-0013 (m)
> 408-773-0220 (fax)
>

--00151750dcacb467ef046be0cf91--

From general-return-242-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 09 03:34:45 2009
Return-Path: <general-return-242-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58687 invoked from network); 9 Jun 2009 03:34:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jun 2009 03:34:45 -0000
Received: (qmail 46948 invoked by uid 500); 9 Jun 2009 03:34:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46847 invoked by uid 500); 9 Jun 2009 03:34:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 90034 invoked by uid 99); 9 Jun 2009 02:01:59 -0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ted.dunning@gmail.com designates 74.125.46.28 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :from:date:message-id:subject:to:cc:content-type;
        bh=NaVsAlPDjnfhwh7UoFF2WsR9+BahoZDccuz6iZ1N/oI=;
        b=DjTGdQ2goYfTSVfHhqqdWaF3x0HtOMCGG09Fmrte03fbExMaH6TXG5ydxdMzZUnfw9
         EbJB3D46/4ioEwO/i0HZS7ZoobDC6grw8ERmDJBqkOsBr+dbRXwEpE3PkezysEwD3nhL
         kZffx07tEBar2oUTUW+uEfa752f9gXbpL/+PA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :cc:content-type;
        b=USh+C5bXq3iK9dv2PfWjFls5qOoFkCUqC9pFQBCyNmo9iXiXZ28Jm8JBS4OMY3VqQJ
         Yyeu+2mHHv/HL7kmftba8TqTULM2PEAVQk49Va0UNS2hCAP7LW+BiL4SKSPxm2YLlfLf
         Kp1WLbZwS7L4UH8aNOeCwr3B2E/tdXDfWI/ds=
MIME-Version: 1.0
In-Reply-To: <4A2DBB61.5010707@apache.org>
References: <4A2DBB61.5010707@apache.org>
From: Ted Dunning <ted.dunning@gmail.com>
Date: Mon, 8 Jun 2009 19:01:07 -0700
Message-ID: <c7d45fc70906081901p7c2d8f3eibbe513939a643e96@mail.gmail.com>
Subject: Re: Show your ZooKeeper pride!
To: zookeeper-user@hadoop.apache.org
Cc: "zookeeper-dev@hadoop.apache.org" <zookeeper-dev@hadoop.apache.org>, general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001517573df28e3c04046be0bc43
X-Virus-Checked: Checked by ClamAV on apache.org

--001517573df28e3c04046be0bc43
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

 How come Yahoo isn't listed?

On Mon, Jun 8, 2009 at 6:31 PM, Patrick Hunt <phunt@apache.org> wrote:

> The Hadoop summit is Wednesday. If you're attending please feel free to say
> hi -- Mahadev is presenting @4, Ben and I will be attending as well.
>
> Also, regardless of whether you're attending or not we'd appreciate any
> updates to the "powered by" page, if you're too busy to update it yourself
> send us a snippet and we'll update it for you ;-)
>
> http://wiki.apache.org/hadoop/ZooKeeper/PoweredBy
>
> Regards,
>
> Patrick
>



-- 
Ted Dunning, CTO
DeepDyve

111 West Evelyn Ave. Ste. 202
Sunnyvale, CA 94086
http://www.deepdyve.com
858-414-0013 (m)
408-773-0220 (fax)

--001517573df28e3c04046be0bc43--

From general-return-243-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 09 03:34:57 2009
Return-Path: <general-return-243-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58805 invoked from network); 9 Jun 2009 03:34:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jun 2009 03:34:57 -0000
Received: (qmail 47893 invoked by uid 500); 9 Jun 2009 03:35:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47823 invoked by uid 500); 9 Jun 2009 03:35:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 96086 invoked by uid 99); 9 Jun 2009 02:10:20 -0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:cc:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:x-originalarrivaltime;
	b=k8C0RhxEJ0ksin6H1lRe523p9GKRLRt6uw4u0gjvIt99d7or/xh6zmwluJ9RwBsP
User-Agent: Microsoft-Entourage/12.17.0.090302
Date: Mon, 08 Jun 2009 19:09:28 -0700
Subject: Re: Show your ZooKeeper pride!
From: Mahadev Konar <mahadev@yahoo-inc.com>
To: <zookeeper-user@hadoop.apache.org>,
        "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "zookeeper-dev@hadoop.apache.org" <zookeeper-dev@hadoop.apache.org>
Message-ID: <C6531268.1E1F2%mahadev@yahoo-inc.com>
Thread-Topic: Show your ZooKeeper pride!
Thread-Index: Acnop1Fs7VYAde60k0+RtUgGtkXLIg==
In-Reply-To: <c7d45fc70906081901p7c2d8f3eibbe513939a643e96@mail.gmail.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-OriginalArrivalTime: 09 Jun 2009 02:09:29.0536 (UTC) FILETIME=[52569C00:01C9E8A7]
X-Virus-Checked: Checked by ClamAV on apache.org

Its just that Zookeeper mostly is used in applications which are proprietary
(to Yahoo!) and so its harder to update the wiki with specifics on how it is
being used.

thanks
mahadev

On 6/8/09 7:01 PM, "Ted Dunning" <ted.dunning@gmail.com> wrote:

>  How come Yahoo isn't listed?
> 
> On Mon, Jun 8, 2009 at 6:31 PM, Patrick Hunt <phunt@apache.org> wrote:
> 
>> The Hadoop summit is Wednesday. If you're attending please feel free to say
>> hi -- Mahadev is presenting @4, Ben and I will be attending as well.
>> 
>> Also, regardless of whether you're attending or not we'd appreciate any
>> updates to the "powered by" page, if you're too busy to update it yourself
>> send us a snippet and we'll update it for you ;-)
>> 
>> http://wiki.apache.org/hadoop/ZooKeeper/PoweredBy
>> 
>> Regards,
>> 
>> Patrick
>> 
> 
> 


From general-return-244-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 09 06:54:11 2009
Return-Path: <general-return-244-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 62249 invoked from network); 9 Jun 2009 06:54:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jun 2009 06:54:11 -0000
Received: (qmail 66230 invoked by uid 500); 9 Jun 2009 06:54:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66154 invoked by uid 500); 9 Jun 2009 06:54:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66144 invoked by uid 99); 9 Jun 2009 06:54:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Jun 2009 06:54:22 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of miryalavignesh@gmail.com designates 209.85.216.189 as permitted sender)
Received: from [209.85.216.189] (HELO mail-px0-f189.google.com) (209.85.216.189)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Jun 2009 06:54:14 +0000
Received: by pxi27 with SMTP id 27so351484pxi.5
        for <general@hadoop.apache.org>; Mon, 08 Jun 2009 23:53:54 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:from:date:message-id
         :subject:to:content-type:content-transfer-encoding;
        bh=4J7YxtI5KePpRG9TBs5iI5XMPa4D7H+LXtXpT5HOo/Y=;
        b=DsIJW4+Voe/KWOKlDd80gqqzXggQCdu8syiSj5NjJHFBeoiQo8wluVcBfNEBS8ClpI
         lWS0jhTvc7ey+UHtfGfU4N5JmIWVQO6kAt2jwBFzO8l6M43hDEJpdrQ4dmwTnUf0a0dJ
         bQRi+AVKfMwQZk7Z0iihajbIcdw05LRHsbLxE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        b=ECSKQoDMxgPO8RvpciarV06hxZAxI/Aaciv2VeU0O97y0OxzgpmHIwLwarCvRK6jI1
         G+V5QZeBzPt0HJ1Nw7qSjrNesxNwyptMnp42teq3jUEj+gTKWYc+FH4ne6pEylcOeD8B
         3ipmIX2RxPgU/1nJ4ixhWiKmqpjQfRsFB3sWQ=
MIME-Version: 1.0
Received: by 10.142.214.12 with SMTP id m12mr2883643wfg.119.1244530434045; 
	Mon, 08 Jun 2009 23:53:54 -0700 (PDT)
From: miryala vignesh <miryalavignesh@gmail.com>
Date: Tue, 9 Jun 2009 12:23:34 +0530
Message-ID: <46a7d8de0906082353g24338861i68e6ad053c7df6a4@mail.gmail.com>
Subject: Implementing join on hadoop
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Did any one work on any project related to databases (simple things
like join) on   hadoop ?
Can any one tell me where can I find hbase programming tutorials
(expecting something other than official website )?

I am not sure if this is the right place for this message , please
direct me to the apt mailing list or forum if this isn't the place .

From general-return-245-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 09 16:08:49 2009
Return-Path: <general-return-245-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96153 invoked from network); 9 Jun 2009 16:08:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jun 2009 16:08:49 -0000
Received: (qmail 47927 invoked by uid 500); 9 Jun 2009 16:09:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47852 invoked by uid 500); 9 Jun 2009 16:09:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 24550 invoked by uid 99); 9 Jun 2009 07:37:00 -0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of matei@eecs.berkeley.edu designates 169.229.60.87 as permitted sender)
Message-Id: <C7B3DF26-CB1A-496D-B1EE-8472BBE02545@eecs.berkeley.edu>
From: Matei Zaharia <matei@eecs.berkeley.edu>
To: general@hadoop.apache.org
In-Reply-To: <46a7d8de0906082353g24338861i68e6ad053c7df6a4@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v935.3)
Subject: Re: Implementing join on hadoop
Date: Tue, 9 Jun 2009 00:36:23 -0700
References: <46a7d8de0906082353g24338861i68e6ad053c7df6a4@mail.gmail.com>
X-Mailer: Apple Mail (2.935.3)
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Miryala,

In general you should use core-user@hadoop.apache.org (subscribe here: http://hadoop.apache.org/core/mailing_lists.html) 
  for questions about using Hadoop. Anyway, joins and other SQL  
operators are implemented by both the Hive and Pig Hadoop sub-projects  
(linked at the top of http://hadoop.apache.org/) and by the DataJoin  
library in Hadoop. Hive is the closest thing to a relational database  
on Hadoop, supporting table schemas, a centralized metadata  
repository, and a SQL-like query language. Pig is a high-level  
language for writing Hadoop workflows that includes SQL operators.

Matei

On Jun 8, 2009, at 11:53 PM, miryala vignesh wrote:

> Did any one work on any project related to databases (simple things
> like join) on   hadoop ?
> Can any one tell me where can I find hbase programming tutorials
> (expecting something other than official website )?
>
> I am not sure if this is the right place for this message , please
> direct me to the apt mailing list or forum if this isn't the place .


From general-return-246-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 17 18:12:23 2009
Return-Path: <general-return-246-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 41055 invoked from network); 17 Jun 2009 18:12:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jun 2009 18:12:12 -0000
Received: (qmail 78369 invoked by uid 500); 17 Jun 2009 18:12:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78323 invoked by uid 500); 17 Jun 2009 18:12:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78311 invoked by uid 99); 17 Jun 2009 18:12:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2009 18:12:22 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [208.65.144.75] (HELO p02c11o142.mxlogic.net) (208.65.144.75)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2009 18:12:11 +0000
Received: from unknown [63.251.237.3] (EHLO mtiexch01.mti.com)
	by p02c11o142.mxlogic.net (mxl_mta-6.2.0-4)
	with ESMTP id 5e1393a4.2586471312.636695.00-006.p02c11o142.mxlogic.net (envelope-from <satish@mellanox.com>);
	Wed, 17 Jun 2009 12:11:51 -0600 (MDT)
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
X-MimeOLE: Produced By Microsoft Exchange V6.5
Subject: RE: Hadoop User Group (Bay Area) May 20th
Date: Wed, 17 Jun 2009 11:11:49 -0700
Message-ID: <9FA59C95FFCBB34EA5E42C1A8573784F01DDF27B@mtiexch01.mti.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Hadoop User Group (Bay Area) May 20th
Thread-Index: AcnNxepSpOFHRec4RX+Djty21JFqUgGP/ssQAVhVC7AFg+0HkA==
References: <AD0C16AAE446924586B68FC0EB421C9BAE155806@SNV-EXVS02.ds.corp.yahoo.com> <AD0C16AAE446924586B68FC0EB421C9BAE155864@SNV-EXVS02.ds.corp.yahoo.com> <DD27C9769EA63D43BE3CF4CCCC12DD970262B33A@SNV-EXVS02.ds.corp.yahoo.com>
From: "Satish Kikkeri" <Satish@Mellanox.com>
To: <general@hadoop.apache.org>
Cc: <aanand@yahoo-inc.com>
X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2009052001)]
X-MAIL-FROM: <satish@mellanox.com>
X-SOURCE-IP: [63.251.237.3]
X-AnalysisOut: [v=1.0 c=1 a=4iyT6hJci8cA:10 a=xupnbh4h0YLOHZnncC45HQ==:17 ]
X-AnalysisOut: [a=CbDCq_QkAAAA:8 a=AnZFoEh2AAAA:8 a=mV9VRH-2AAAA:8 a=CjxXg]
X-AnalysisOut: [O3LAAAA:8 a=htzNLyBbAAAA:8 a=X7BGoOqpqRD5uP4gtdMA:9 a=9h-b]
X-AnalysisOut: [tLetWG7E2Ap4n5gYeDKE_ngA:4 a=6bbpDATN3xAA:10 a=gQ22TatrZ4o]
X-AnalysisOut: [A:10 a=mqKVMgZcPAkA:10 a=-NOWlJdPJ3YA:10]
X-Virus-Checked: Checked by ClamAV on apache.org

Is the next Hadoop UG meeting today (6/18) or next week (6/25)

Thanks=20

________________________________________________________________________
_______________

=20

Satish Kikkeri

Mellanox Technologies

350 Oakmead Parkway, Sunnyvale, CA 94085

408-916-0045 (Direct)

408-807-9903 (Mobile)

www.mellanox.com

=20

=20


-----Original Message-----
From: Ajay Anand [mailto:aanand@yahoo-inc.com]=20
Sent: Wednesday, May 20, 2009 9:28 AM
To: core-user@hadoop.apache.org; general@hadoop.apache.org;
zookeeper-user@hadoop.apache.org; hbase-user@hadoop.apache.org;
pig-user@hadoop.apache.org
Subject: RE: Hadoop User Group (Bay Area) May 20th

Reminder: the Bay Area Hadoop User Group meeting is today at 6 pm at the
Yahoo! Sunnyvale campus - http://upcoming.yahoo.com/event/2659418/

=20

________________________________

From: Ajay Anand=20
Sent: Wednesday, May 13, 2009 1:12 PM
To: 'core-user@hadoop.apache.org'; 'general@hadoop.apache.org';
'zookeeper-user@hadoop.apache.org'; 'hbase-user@hadoop.apache.org';
'pig-user@hadoop.apache.org'
Subject: Hadoop User Group (Bay Area) May 20th

=20

The next Bay Area Hadoop User Group meeting is scheduled for Wednesday,
May 20th at Yahoo! 700 First Avenue, Sunnyvale, Building E, Class room
10 from 6:00-7:30 pm. Please note that the location has changed from our
usual meeting place in Santa Clara.=20

Registration: http://upcoming.yahoo.com/event/2659418/

=20

Agenda:

- Hadoop Summit 2009 preview (registration open at
http://hadoopsummit09.eventbrite.com/ )
- Hadoop with GPFS - Prasenjit Sarkar, IBM
- SQoop - Cloudera

Look forward to seeing you there!

Ajay

=20

=20


From general-return-247-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 17 18:29:17 2009
Return-Path: <general-return-247-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68939 invoked from network); 17 Jun 2009 18:29:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jun 2009 18:29:14 -0000
Received: (qmail 20324 invoked by uid 500); 17 Jun 2009 18:29:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20258 invoked by uid 500); 17 Jun 2009 18:29:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 20233 invoked by uid 99); 17 Jun 2009 18:29:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2009 18:29:24 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2009 18:29:12 +0000
Received: from battlerock-lm.corp.yahoo.com (battlerock-lm.corp.yahoo.com [10.72.112.37])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n5HISCVN024240
	for <general@hadoop.apache.org>; Wed, 17 Jun 2009 11:28:12 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:content-type:
	content-transfer-encoding:mime-version:subject:date:x-mailer;
	b=Br4V39n5sT7nRoTIEJfknEsTCrnYECCsnNf++J2Da7R+6QKJj8sim1jKpDHTDZ2L
Message-Id: <094B4654-8D07-4251-B521-9EAFAAFBCB4E@yahoo-inc.com>
From: "Owen O'Malley" <oom@yahoo-inc.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v935.3)
Subject: [VOTE] Rename Core to Common
Date: Wed, 17 Jun 2009 11:28:12 -0700
X-Mailer: Apple Mail (2.935.3)
X-Virus-Checked: Checked by ClamAV on apache.org

I'd like to propose that we rename Core to Common. This will better  
reflect the fact that it is no longer the "core" of Hadoop, but rather  
just the common libraries.

-- Owen

From general-return-248-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 17 18:31:29 2009
Return-Path: <general-return-248-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 71158 invoked from network); 17 Jun 2009 18:31:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jun 2009 18:31:29 -0000
Received: (qmail 22629 invoked by uid 500); 17 Jun 2009 18:31:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22566 invoked by uid 500); 17 Jun 2009 18:31:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22556 invoked by uid 99); 17 Jun 2009 18:31:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2009 18:31:40 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.218.222] (HELO mail-bw0-f222.google.com) (209.85.218.222)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2009 18:31:31 +0000
Received: by bwz22 with SMTP id 22so681906bwz.29
        for <general@hadoop.apache.org>; Wed, 17 Jun 2009 11:31:10 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.112.1 with SMTP id u1mr425881bkp.37.1245263470527; Wed, 17 
	Jun 2009 11:31:10 -0700 (PDT)
In-Reply-To: <094B4654-8D07-4251-B521-9EAFAAFBCB4E@yahoo-inc.com>
References: <094B4654-8D07-4251-B521-9EAFAAFBCB4E@yahoo-inc.com>
From: Philip Zeyliger <philip@cloudera.com>
Date: Wed, 17 Jun 2009 11:30:50 -0700
Message-ID: <15da8a100906171130k77c8726el710686e1eb968b4@mail.gmail.com>
Subject: Re: [VOTE] Rename Core to Common
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d64515d068b4046c8f7e70
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d64515d068b4046c8f7e70
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

+1

On Wed, Jun 17, 2009 at 11:28 AM, Owen O'Malley <oom@yahoo-inc.com> wrote:

> I'd like to propose that we rename Core to Common. This will better reflect
> the fact that it is no longer the "core" of Hadoop, but rather just the
> common libraries.
>
> -- Owen
>

--0016e6d64515d068b4046c8f7e70--

From general-return-249-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 17 18:34:09 2009
Return-Path: <general-return-249-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73160 invoked from network); 17 Jun 2009 18:34:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jun 2009 18:34:09 -0000
Received: (qmail 28438 invoked by uid 500); 17 Jun 2009 18:34:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28372 invoked by uid 500); 17 Jun 2009 18:34:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28362 invoked by uid 99); 17 Jun 2009 18:34:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2009 18:34:18 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.218.222] (HELO mail-bw0-f222.google.com) (209.85.218.222)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2009 18:34:09 +0000
Received: by bwz22 with SMTP id 22so684008bwz.29
        for <general@hadoop.apache.org>; Wed, 17 Jun 2009 11:33:48 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.60.194 with SMTP id q2mr393501bkh.150.1245263627758; Wed, 
	17 Jun 2009 11:33:47 -0700 (PDT)
In-Reply-To: <15da8a100906171130k77c8726el710686e1eb968b4@mail.gmail.com>
References: <094B4654-8D07-4251-B521-9EAFAAFBCB4E@yahoo-inc.com>
	 <15da8a100906171130k77c8726el710686e1eb968b4@mail.gmail.com>
Date: Wed, 17 Jun 2009 11:33:47 -0700
Message-ID: <ac79ea400906171133n61d59073l338c88597f32826@mail.gmail.com>
Subject: Re: [VOTE] Rename Core to Common
From: Tom White <tom@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Tom

On Wed, Jun 17, 2009 at 11:30 AM, Philip Zeyliger<philip@cloudera.com> wrote:
> +1
>
> On Wed, Jun 17, 2009 at 11:28 AM, Owen O'Malley <oom@yahoo-inc.com> wrote:
>
>> I'd like to propose that we rename Core to Common. This will better reflect
>> the fact that it is no longer the "core" of Hadoop, but rather just the
>> common libraries.
>>
>> -- Owen
>>
>

From general-return-250-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 17 18:35:27 2009
Return-Path: <general-return-250-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73814 invoked from network); 17 Jun 2009 18:35:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jun 2009 18:35:27 -0000
Received: (qmail 34338 invoked by uid 500); 17 Jun 2009 18:35:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34284 invoked by uid 500); 17 Jun 2009 18:35:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34259 invoked by uid 99); 17 Jun 2009 18:35:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2009 18:35:38 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2009 18:35:27 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n5HIYeoN055955
	for <general@hadoop.apache.org>; Wed, 17 Jun 2009 11:34:40 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=nEjLWIWr+kPvMMP19W4gps4vvu8oV5IL6ZjggfM3PbLARRfblxFglINIgASIDY2f
Message-Id: <EAE9387C-C581-44E0-920C-79D55475AACA@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <094B4654-8D07-4251-B521-9EAFAAFBCB4E@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v935.3)
Subject: Re: [VOTE] Rename Core to Common
Date: Wed, 17 Jun 2009 11:34:39 -0700
References: <094B4654-8D07-4251-B521-9EAFAAFBCB4E@yahoo-inc.com>
X-Mailer: Apple Mail (2.935.3)
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Arun

On Jun 17, 2009, at 11:28 AM, Owen O'Malley wrote:

> I'd like to propose that we rename Core to Common. This will better  
> reflect the fact that it is no longer the "core" of Hadoop, but  
> rather just the common libraries.
>
> -- Owen


From general-return-251-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 17 18:39:59 2009
Return-Path: <general-return-251-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76856 invoked from network); 17 Jun 2009 18:39:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jun 2009 18:39:59 -0000
Received: (qmail 48159 invoked by uid 500); 17 Jun 2009 18:40:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48106 invoked by uid 500); 17 Jun 2009 18:40:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48096 invoked by uid 99); 17 Jun 2009 18:40:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2009 18:40:10 +0000
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.62.80] (HELO QMTA08.westchester.pa.mail.comcast.net) (76.96.62.80)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2009 18:39:59 +0000
Received: from OMTA04.westchester.pa.mail.comcast.net ([76.96.62.35])
	by QMTA08.westchester.pa.mail.comcast.net with comcast
	id 51sD1c0020ldTLk586fffV; Wed, 17 Jun 2009 18:39:39 +0000
Received: from battlerock-lm.corp.yahoo.com ([209.131.62.113])
	by OMTA04.westchester.pa.mail.comcast.net with comcast
	id 56fT1c0052SbwD53Q6fWlh; Wed, 17 Jun 2009 18:39:36 +0000
Cc: Dekel Tankel <dekel@yahoo-inc.com>
Message-Id: <94E98768-027B-464D-A738-5B74069D9291@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <9FA59C95FFCBB34EA5E42C1A8573784F01DDF27B@mtiexch01.mti.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v935.3)
Subject: Re: Hadoop User Group (Bay Area) May 20th
Date: Wed, 17 Jun 2009 11:39:25 -0700
References: <AD0C16AAE446924586B68FC0EB421C9BAE155806@SNV-EXVS02.ds.corp.yahoo.com> <AD0C16AAE446924586B68FC0EB421C9BAE155864@SNV-EXVS02.ds.corp.yahoo.com> <DD27C9769EA63D43BE3CF4CCCC12DD970262B33A@SNV-EXVS02.ds.corp.yahoo.com> <9FA59C95FFCBB34EA5E42C1A8573784F01DDF27B@mtiexch01.mti.com>
X-Mailer: Apple Mail (2.935.3)
X-Virus-Checked: Checked by ClamAV on apache.org


On Jun 17, 2009, at 11:11 AM, Satish Kikkeri wrote:

> Is the next Hadoop UG meeting today (6/18) or next week (6/25)

We just had the Hadoop Summit last week, so let's skip this month. I  
expect that the Hadoop User Group meetings will resume next month.

-- Owen

From general-return-252-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 17 18:52:37 2009
Return-Path: <general-return-252-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87097 invoked from network); 17 Jun 2009 18:52:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jun 2009 18:52:37 -0000
Received: (qmail 81403 invoked by uid 500); 17 Jun 2009 18:52:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81343 invoked by uid 500); 17 Jun 2009 18:52:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81333 invoked by uid 99); 17 Jun 2009 18:52:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2009 18:52:48 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [216.252.110.217] (HELO web56208.mail.re3.yahoo.com) (216.252.110.217)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 17 Jun 2009 18:52:38 +0000
Received: (qmail 81546 invoked by uid 60001); 17 Jun 2009 18:52:15 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1245264735; bh=uEB+24h54con6PGYLRD9tQ37RfcNAzH41b6LuIPYlhQ=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=KefAyHw6YRczsn2KW70vY5bAJLLo+5uBr8TULJwYQyX+tWpLqsXsq4hgauDoOVf4B9Pp+cJPgNR+bDZVL8h0N3aqKpumMYbNBx8RuIj+T7WkNHTINeMK8bC1Jukf8WIIqwZFo22o5WREFrJR4JCLQrgrgfgp2ocE6CKPrWXsRvo=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=Qs2+fHblQPZcO9lRDvuoepawHGcB6DPldZ48BhrFXeYQSvrqizOW6ELvdAlxNhJ5ZwspcsCLZctHAfqNqfXPKJ+ZW4l91XvAsSbCL7rhJkjLYtl1ea1zmG4Fkyp87HJsdOnSvbfA2CSwkRgDubr47qGZYAUEaM9/FFtYCtQ5qGg=;
Message-ID: <183794.81398.qm@web56208.mail.re3.yahoo.com>
X-YMail-OSG: X.wJaLEVM1mfeGEd1jn2oOXJ_jSoeZfD78zyW13zKvejEMk9CHw380Xe5LxBbwnBUkBvm1OecPMwmt48eI0r5Kaoc45J5QFqjhN06Is26IfEBw9pH_MrfmC.8mzpYOHXFqdRoRHI3GuVWijGy72Pb4noB3zJJTEDYidalVOGmz_EXhfGOmTVN96IymfR3XADXcQ2VbTQWGy.OC_6W2u6qiRRb2rO9z8Mug3oWTLJvU2.TkMmDhniXxXyQmXqcRgoclDZSFENRmJ.6v6Ch1HG2RncWXSRMTCDzNrk2HPnR5AM4zIgEjCLaC.SLIFcdM7L4QoEH5LxlXw-
Received: from [216.145.49.5] by web56208.mail.re3.yahoo.com via HTTP; Wed, 17 Jun 2009 11:52:14 PDT
X-Mailer: YahooMailRC/1277.43 YahooMailWebService/0.7.289.15
References: <094B4654-8D07-4251-B521-9EAFAAFBCB4E@yahoo-inc.com>
Date: Wed, 17 Jun 2009 11:52:14 -0700 (PDT)
From: "Tsz Wo \(Nicholas\), Sze" <s29752-hadoopgeneral@yahoo.com>
Subject: Re: [VOTE] Rename Core to Common
To: general@hadoop.apache.org
In-Reply-To: <094B4654-8D07-4251-B521-9EAFAAFBCB4E@yahoo-inc.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org


+1

Nicholas Sze




----- Original Message ----
> From: Owen O'Malley <oom@yahoo-inc.com>
> To: general@hadoop.apache.org
> Sent: Wednesday, June 17, 2009 11:28:12 AM
> Subject: [VOTE] Rename Core to Common
> 
> I'd like to propose that we rename Core to Common. This will better reflect the 
> fact that it is no longer the "core" of Hadoop, but rather just the common 
> libraries.
> 
> -- Owen


From general-return-253-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 17 19:20:19 2009
Return-Path: <general-return-253-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9466 invoked from network); 17 Jun 2009 19:20:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jun 2009 19:20:19 -0000
Received: (qmail 30792 invoked by uid 500); 17 Jun 2009 19:20:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30699 invoked by uid 500); 17 Jun 2009 19:20:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30689 invoked by uid 99); 17 Jun 2009 19:20:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2009 19:20:29 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cutting@gmail.com designates 209.85.200.173 as permitted sender)
Received: from [209.85.200.173] (HELO wf-out-1314.google.com) (209.85.200.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2009 19:20:19 +0000
Received: by wf-out-1314.google.com with SMTP id 23so217232wfg.2
        for <general@hadoop.apache.org>; Wed, 17 Jun 2009 12:19:58 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:sender:message-id:date:from
         :user-agent:mime-version:to:subject:references:in-reply-to
         :content-type:content-transfer-encoding;
        bh=q5frm1c99ZHP8HDe9IgaOmwbw/5AlMMq1sHYnl2EDLk=;
        b=CYzYlwNO3XQ1h0iVXSzj6hoBWeT91ZUc59vvi1orIz5fGxgtN5Kegx25oj8K0KKKFI
         FB0aMckGr+8+xSnnrwdRHcl0QB677v/OzGurdV1bbYbQgwS7Z2tC8zM2ugxhY17Ok6NP
         6SfXN5MmetHBr8SDySeX6HQ2DWl0L0ngrcz+Y=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=sender:message-id:date:from:user-agent:mime-version:to:subject
         :references:in-reply-to:content-type:content-transfer-encoding;
        b=EO2kJICEnxPdEcsdR89VjGuJNBZQ5qA7+NTmaIA1L+XwKH6GxGCv8IXoAuE/OUO5+V
         3kLrFTmn5yzcPSipqyUisLjBPYHxUCsPdqnh/+mEv6kOPu/qOCOfloBzd5ovtKvrgPHa
         VbDcBfaRyIHLB7ZDdr6z5fFhu+sD731cYswgs=
Received: by 10.142.169.4 with SMTP id r4mr502921wfe.262.1245266398133;
        Wed, 17 Jun 2009 12:19:58 -0700 (PDT)
Received: from ?192.168.168.16? (c-76-103-176-123.hsd1.ca.comcast.net [76.103.176.123])
        by mx.google.com with ESMTPS id 30sm460579wfg.10.2009.06.17.12.19.56
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Wed, 17 Jun 2009 12:19:57 -0700 (PDT)
Sender: Doug Cutting <cutting@gmail.com>
Message-ID: <4A3941DB.30707@apache.org>
Date: Wed, 17 Jun 2009 12:19:55 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.21 (X11/20090409)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Rename Core to Common
References: <094B4654-8D07-4251-B521-9EAFAAFBCB4E@yahoo-inc.com>
In-Reply-To: <094B4654-8D07-4251-B521-9EAFAAFBCB4E@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Owen O'Malley wrote:
> I'd like to propose that we rename Core to Common. This will better 
> reflect the fact that it is no longer the "core" of Hadoop, but rather 
> just the common libraries.
> 
> -- Owen

From general-return-254-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 17 20:08:32 2009
Return-Path: <general-return-254-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 41604 invoked from network); 17 Jun 2009 20:08:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jun 2009 20:08:32 -0000
Received: (qmail 58685 invoked by uid 500); 17 Jun 2009 20:08:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58622 invoked by uid 500); 17 Jun 2009 20:08:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58612 invoked by uid 99); 17 Jun 2009 20:08:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2009 20:08:42 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2009 20:08:30 +0000
Received: from SNV-EXBH01.ds.corp.yahoo.com (snv-exbh01.ds.corp.yahoo.com [207.126.227.249])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n5HK7ekr065696
	for <general@hadoop.apache.org>; Wed, 17 Jun 2009 13:07:40 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:x-originalarrivaltime;
	b=RmB8jXe8euWJDl9kBmeBiXopCNKzmp5lQmrVrQVlnrf4Rn7xRK2BGtFfLjmRSz0N
Received: from SNV-EXVS09.ds.corp.yahoo.com ([207.126.227.87]) by SNV-EXBH01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Wed, 17 Jun 2009 13:07:40 -0700
Received: from 10.73.146.106 ([10.73.146.106]) by SNV-EXVS09.ds.corp.yahoo.com ([207.126.227.84]) via Exchange Front-End Server snv-webmail.corp.yahoo.com ([207.126.227.60]) with Microsoft Exchange Server HTTP-DAV ;
 Wed, 17 Jun 2009 20:07:39 +0000
User-Agent: Microsoft-Entourage/12.19.0.090515
Date: Wed, 17 Jun 2009 13:07:39 -0700
Subject: Re: [VOTE] Rename Core to Common
From: Mahadev Konar <mahadev@yahoo-inc.com>
To: <general@hadoop.apache.org>
Message-ID: <C65E9B1B.1EA3D%mahadev@yahoo-inc.com>
Thread-Topic: [VOTE] Rename Core to Common
Thread-Index: Acnvh0OR64puutHZgU+XAV2PwkU8JA==
In-Reply-To: <4A3941DB.30707@apache.org>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-OriginalArrivalTime: 17 Jun 2009 20:07:40.0400 (UTC) FILETIME=[44671700:01C9EF87]
X-Virus-Checked: Checked by ClamAV on apache.org

+1

mahadev


On 6/17/09 12:19 PM, "Doug Cutting" <cutting@apache.org> wrote:

> +1
> 
> Owen O'Malley wrote:
>> I'd like to propose that we rename Core to Common. This will better
>> reflect the fact that it is no longer the "core" of Hadoop, but rather
>> just the common libraries.
>> 
>> -- Owen


From general-return-255-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 17 20:10:59 2009
Return-Path: <general-return-255-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42147 invoked from network); 17 Jun 2009 20:10:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jun 2009 20:10:59 -0000
Received: (qmail 61342 invoked by uid 500); 17 Jun 2009 20:11:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61285 invoked by uid 500); 17 Jun 2009 20:11:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61264 invoked by uid 99); 17 Jun 2009 20:11:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2009 20:11:05 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of matei@eecs.berkeley.edu designates 169.229.60.87 as permitted sender)
Received: from [169.229.60.87] (HELO gateway0.EECS.Berkeley.EDU) (169.229.60.87)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2009 20:10:54 +0000
Received: from [192.168.1.3] (c-76-21-11-236.hsd1.ca.comcast.net [76.21.11.236])
	(authenticated bits=0)
	by gateway0.EECS.Berkeley.EDU (8.14.3/8.13.5) with ESMTP id n5HKAUq9028875
	(version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT)
	for <general@hadoop.apache.org>; Wed, 17 Jun 2009 13:10:32 -0700 (PDT)
Message-Id: <C09186BA-16BD-44FC-A5CB-8E55C11C45C0@eecs.berkeley.edu>
From: Matei Zaharia <matei@eecs.berkeley.edu>
To: general@hadoop.apache.org
In-Reply-To: <094B4654-8D07-4251-B521-9EAFAAFBCB4E@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v935.3)
Subject: Re: [VOTE] Rename Core to Common
X-Priority: 5 (Lowest)
Date: Wed, 17 Jun 2009 13:10:30 -0700
References: <094B4654-8D07-4251-B521-9EAFAAFBCB4E@yahoo-inc.com>
X-Mailer: Apple Mail (2.935.3)
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Matei

On Jun 17, 2009, at 11:28 AM, Owen O'Malley wrote:

> I'd like to propose that we rename Core to Common. This will better  
> reflect the fact that it is no longer the "core" of Hadoop, but  
> rather just the common libraries.
>
> -- Owen


From general-return-256-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 17 20:13:46 2009
Return-Path: <general-return-256-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43898 invoked from network); 17 Jun 2009 20:13:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jun 2009 20:13:46 -0000
Received: (qmail 66936 invoked by uid 500); 17 Jun 2009 20:13:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66874 invoked by uid 500); 17 Jun 2009 20:13:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66864 invoked by uid 99); 17 Jun 2009 20:13:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2009 20:13:57 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2009 20:13:46 +0000
Received: from wlanvpn-mc2e-246-34.corp.yahoo.com (wlanvpn-mc2e-246-34.corp.yahoo.com [172.21.148.34])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n5HKCRY5067628
	for <general@hadoop.apache.org>; Wed, 17 Jun 2009 13:12:28 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=Y19bt/4XHtrIT2CKyV8W2d4n2Cc9hgGHN803hd1l2p7BzL7H6Yoy4UrdhF9oK2DJ
Message-Id: <D95BE8DB-7C83-4483-9E0D-43733CDBBD10@yahoo-inc.com>
From: Nigel Daley <ndaley@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <094B4654-8D07-4251-B521-9EAFAAFBCB4E@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: Re: [VOTE] Rename Core to Common
Date: Wed, 17 Jun 2009 13:12:27 -0700
References: <094B4654-8D07-4251-B521-9EAFAAFBCB4E@yahoo-inc.com>
X-Mailer: Apple Mail (2.930.3)
X-Virus-Checked: Checked by ClamAV on apache.org

+1

On Jun 17, 2009, at 11:28 AM, Owen O'Malley wrote:

> I'd like to propose that we rename Core to Common. This will better  
> reflect the fact that it is no longer the "core" of Hadoop, but  
> rather just the common libraries.
>
> -- Owen


From general-return-257-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 17 21:04:14 2009
Return-Path: <general-return-257-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 65614 invoked from network); 17 Jun 2009 21:04:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jun 2009 21:04:14 -0000
Received: (qmail 60499 invoked by uid 500); 17 Jun 2009 21:04:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60438 invoked by uid 500); 17 Jun 2009 21:04:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60428 invoked by uid 99); 17 Jun 2009 21:04:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2009 21:04:21 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.222.200] (HELO mail-pz0-f200.google.com) (209.85.222.200)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2009 21:04:12 +0000
Received: by pzk38 with SMTP id 38so571937pzk.5
        for <general@hadoop.apache.org>; Wed, 17 Jun 2009 14:03:51 -0700 (PDT)
Received: by 10.114.149.2 with SMTP id w2mr788452wad.182.1245272631434;
        Wed, 17 Jun 2009 14:03:51 -0700 (PDT)
Received: from ?192.168.42.100? (75-101-123-136.dsl.static.sonic.net [75.101.123.136])
        by mx.google.com with ESMTPS id d20sm1910547waa.47.2009.06.17.14.03.49
        (version=SSLv3 cipher=RC4-MD5);
        Wed, 17 Jun 2009 14:03:50 -0700 (PDT)
Message-ID: <4A395A13.9030809@cloudera.com>
Date: Wed, 17 Jun 2009 14:03:15 -0700
From: Amr Awadallah <aaa@cloudera.com>
Organization: Cloudera, Inc.
User-Agent: Thunderbird 2.0.0.21 (Windows/20090302)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Rename Core to Common
References: <094B4654-8D07-4251-B521-9EAFAAFBCB4E@yahoo-inc.com>	 <15da8a100906171130k77c8726el710686e1eb968b4@mail.gmail.com> <ac79ea400906171133n61d59073l338c88597f32826@mail.gmail.com>
In-Reply-To: <ac79ea400906171133n61d59073l338c88597f32826@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

+1

From general-return-258-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 17 22:09:20 2009
Return-Path: <general-return-258-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9734 invoked from network); 17 Jun 2009 22:09:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jun 2009 22:09:20 -0000
Received: (qmail 87794 invoked by uid 500); 17 Jun 2009 22:09:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87725 invoked by uid 500); 17 Jun 2009 22:09:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87715 invoked by uid 99); 17 Jun 2009 22:09:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2009 22:09:30 +0000
X-ASF-Spam-Status: No, hits=-8.0 required=10.0
	tests=RCVD_IN_DNSWL_HI,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of Jim.Kellerman@microsoft.com designates 131.107.115.214 as permitted sender)
Received: from [131.107.115.214] (HELO smtp.microsoft.com) (131.107.115.214)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2009 22:09:20 +0000
Received: from TK5EX14MLTC102.redmond.corp.microsoft.com (157.54.79.180) by
 TK5-EXGWY-E803.partners.extranet.microsoft.com (10.251.56.169) with Microsoft
 SMTP Server (TLS) id 8.2.99.4; Wed, 17 Jun 2009 15:08:58 -0700
Received: from TK5EX14MBXC120.redmond.corp.microsoft.com ([157.54.91.69]) by
 TK5EX14MLTC102.redmond.corp.microsoft.com ([157.54.79.180]) with mapi; Wed,
 17 Jun 2009 14:23:21 -0700
From: "Jim Kellerman (POWERSET)" <Jim.Kellerman@microsoft.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: RE: [VOTE] Rename Core to Common
Thread-Topic: [VOTE] Rename Core to Common
Thread-Index: AQHJ75G77t8RWR/NNU23Lq1uQDj/9JBLRIMA
Date: Wed, 17 Jun 2009 21:23:21 +0000
Message-ID: <E9C6F5B134FDA3468879CB4E533CF1C8029213@TK5EX14MBXC120.redmond.corp.microsoft.com>
References: <094B4654-8D07-4251-B521-9EAFAAFBCB4E@yahoo-inc.com>
In-Reply-To: <094B4654-8D07-4251-B521-9EAFAAFBCB4E@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

+1

> -----Original Message-----
> From: Owen O'Malley [mailto:oom@yahoo-inc.com]
> Sent: Wednesday, June 17, 2009 11:28 AM
> To: general@hadoop.apache.org
> Subject: [VOTE] Rename Core to Common
>
> I'd like to propose that we rename Core to Common. This will better
> reflect the fact that it is no longer the "core" of Hadoop, but rather
> just the common libraries.
>
> -- Owen


From general-return-259-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 17 22:31:11 2009
Return-Path: <general-return-259-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27644 invoked from network); 17 Jun 2009 22:31:10 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jun 2009 22:31:10 -0000
Received: (qmail 32083 invoked by uid 500); 17 Jun 2009 22:31:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31967 invoked by uid 500); 17 Jun 2009 22:31:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31947 invoked by uid 99); 17 Jun 2009 22:31:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2009 22:31:20 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of anubhav.hanjura@convergys.com designates 167.1.158.29 as permitted sender)
Received: from [167.1.158.29] (HELO cvgmx2.convergys.com) (167.1.158.29)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Jun 2009 22:31:10 +0000
Received: from CDC5NW41.na.convergys.com (CDC5NW41.na.convergys.com [155.90.28.181])
	by cvgmx2.convergys.com (Postfix) with ESMTP id 06991152E5E2
	for <general@hadoop.apache.org>; Wed, 17 Jun 2009 22:29:24 +0000 (UTC)
Subject: Anubhav Hanjura is out of the office.
From: anubhav.hanjura@convergys.com
To: general@hadoop.apache.org
Message-ID: <OF3324B0A6.B746E442-ON652575D8.007B9F31-652575D8.007B9F31@nd.convergys.com>
Date: Thu, 18 Jun 2009 04:00:16 +0530
X-MIMETrack: Serialize by Router on CVGSMTP1/SRVR/CVG(Release 6.5.5FP3|March 22, 2007) at
 06/17/2009 06:31:14 PM
MIME-Version: 1.0
Content-type: text/plain; charset=US-ASCII
X-Virus-Checked: Checked by ClamAV on apache.org





I will be out of the office starting  06/15/2009 and will not return until
06/29/2009.

Please contact Nishi Kant for general operability issues, Dileep for 5632
related questions, Shashank/Tarun/Kishore for OLC Lab related issues.

--
"NOTICE:  The information contained in this electronic mail transmission is
intended by Convergys Corporation for the use of the named individual or
entity to which it is directed and may contain information that is
privileged or otherwise confidential.  If you have received this electronic
mail transmission in error, please delete it from your system without
copying or forwarding it, and notify the sender of the error by reply email
or by telephone (collect), so that the sender's address records can be
corrected."


From general-return-260-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 18 00:32:24 2009
Return-Path: <general-return-260-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 74099 invoked from network); 18 Jun 2009 00:32:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jun 2009 00:32:24 -0000
Received: (qmail 63985 invoked by uid 500); 18 Jun 2009 00:32:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63924 invoked by uid 500); 18 Jun 2009 00:32:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63914 invoked by uid 99); 18 Jun 2009 00:32:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Jun 2009 00:32:35 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 74.125.46.30 as permitted sender)
Received: from [74.125.46.30] (HELO yw-out-2324.google.com) (74.125.46.30)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Jun 2009 00:32:27 +0000
Received: by yw-out-2324.google.com with SMTP id 9so323819ywe.29
        for <general@hadoop.apache.org>; Wed, 17 Jun 2009 17:32:06 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=PTL85vE1ULKA8WPw8TGm2+RwmPXD3D1Xz1OdA/odmWM=;
        b=Gk5O9ykmzHd9qd0WIsq47cMuk7yoj9tUbNp73LT4AeCjCabAaRKlio2ktyRNMC6uET
         DBQ5C69YCQanXRwMLEp5YY/QQkA0HG5KPHXDSw+mo1CRUGTkNyDLiP+V7bybGQ/ygLYQ
         ZiDENN8kqosvD0Vgfs+VNZAPNu4wazrHGvUIY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=YcEU8O20Q8CYF3UL0zJsutt3JDBC6qPgdE46/gpz1xZIdSp2QPoLl4dYg9LnbqyGJR
         f7jvrHaATYGarT/Tp9/c1kZ06Kl6tLIflJBGvBdRCe8bEbiUcVPolrblczF23NiPu0Aw
         0MA3btCB13avAoA4pIl4wbBsIhc8DjAjz+SR0=
MIME-Version: 1.0
Received: by 10.151.50.13 with SMTP id c13mr2325056ybk.79.1245284752447; Wed, 
	17 Jun 2009 17:25:52 -0700 (PDT)
In-Reply-To: <E9C6F5B134FDA3468879CB4E533CF1C8029213@TK5EX14MBXC120.redmond.corp.microsoft.com>
References: <094B4654-8D07-4251-B521-9EAFAAFBCB4E@yahoo-inc.com>
	 <E9C6F5B134FDA3468879CB4E533CF1C8029213@TK5EX14MBXC120.redmond.corp.microsoft.com>
Date: Wed, 17 Jun 2009 17:25:52 -0700
Message-ID: <4aa34eb70906171725y3ece018dm5b003800ecd03f36@mail.gmail.com>
Subject: Re: [VOTE] Rename Core to Common
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00151750eed0505a2b046c947301
X-Virus-Checked: Checked by ClamAV on apache.org

--00151750eed0505a2b046c947301
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

+1

On Wed, Jun 17, 2009 at 2:23 PM, Jim Kellerman (POWERSET) <
Jim.Kellerman@microsoft.com> wrote:

> +1
>
> > -----Original Message-----
> > From: Owen O'Malley [mailto:oom@yahoo-inc.com]
> > Sent: Wednesday, June 17, 2009 11:28 AM
> > To: general@hadoop.apache.org
> > Subject: [VOTE] Rename Core to Common
> >
> > I'd like to propose that we rename Core to Common. This will better
> > reflect the fact that it is no longer the "core" of Hadoop, but rather
> > just the common libraries.
> >
> > -- Owen
>
>

--00151750eed0505a2b046c947301--

From general-return-261-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 18 01:11:45 2009
Return-Path: <general-return-261-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88107 invoked from network); 18 Jun 2009 01:11:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jun 2009 01:11:45 -0000
Received: (qmail 96299 invoked by uid 500); 18 Jun 2009 01:11:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96242 invoked by uid 500); 18 Jun 2009 01:11:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96232 invoked by uid 99); 18 Jun 2009 01:11:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Jun 2009 01:11:56 +0000
X-ASF-Spam-Status: No, hits=4.2 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Jun 2009 01:11:43 +0000
Received: from sp1-ex07cas01.ds.corp.yahoo.com (sp1-ex07cas01.ds.corp.yahoo.com [216.252.116.137])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n5I1AMdT083888
	for <general@hadoop.apache.org>; Wed, 17 Jun 2009 18:10:22 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:in-reply-to:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=zompbFYfot0LB5plQrMY1HqfVyrlfjkaFk7tmp/moMBU5kTOy97cM3zq62j+jCue
Received: from SP1-EX07VS02.ds.corp.yahoo.com ([216.252.116.135]) by
 sp1-ex07cas01.ds.corp.yahoo.com ([216.252.116.137]) with mapi; Wed, 17 Jun
 2009 18:10:22 -0700
From: Devaraj Das <ddas@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 17 Jun 2009 18:10:19 -0700
Subject: Re: [VOTE] Rename Core to Common
Thread-Topic: [VOTE] Rename Core to Common
Thread-Index: AcnveZ8IvE1E+64WQ3uw/tn6W1H9wgAN+y8I
Message-ID: <C65F91D3.5EB9D%ddas@yahoo-inc.com>
In-Reply-To: <094B4654-8D07-4251-B521-9EAFAAFBCB4E@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C65F91D35EB9Dddasyahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C65F91D35EB9Dddasyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

+1


On 6/17/09 11:58 PM, "Owen O'Malley" <oom@yahoo-inc.com> wrote:

I'd like to propose that we rename Core to Common. This will better
reflect the fact that it is no longer the "core" of Hadoop, but rather
just the common libraries.

-- Owen


--_000_C65F91D35EB9Dddasyahooinccom_--

From general-return-262-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 18 19:33:06 2009
Return-Path: <general-return-262-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85759 invoked from network); 18 Jun 2009 19:33:06 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jun 2009 19:33:06 -0000
Received: (qmail 69635 invoked by uid 500); 18 Jun 2009 19:33:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69398 invoked by uid 500); 18 Jun 2009 19:33:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69332 invoked by uid 99); 18 Jun 2009 19:33:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Jun 2009 19:33:16 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Jun 2009 19:33:03 +0000
Received: from SNV-EXPF01.ds.corp.yahoo.com (snv-expf01.ds.corp.yahoo.com [207.126.227.250])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n5IJVHqd046002;
	Thu, 18 Jun 2009 12:31:18 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:x-mimeole:content-class:mime-version:
	content-type:subject:date:message-id:x-ms-has-attach:
	x-ms-tnef-correlator:thread-topic:thread-index:from:to:x-originalarrivaltime;
	b=UOOfeLnB7t4XUgHFnpMqMsGfeJvDEwFQnFVkEdoBCu6yUZ12UCM91Zb5fmHi3gv7
Received: from SNV-EXVS09.ds.corp.yahoo.com ([207.126.227.87]) by SNV-EXPF01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Thu, 18 Jun 2009 12:31:16 -0700
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C9F04B.376C558E"
Subject: [VOTE] Release Pig 0.3.0 (candidate 0)
Date: Thu, 18 Jun 2009 12:30:19 -0700
Message-ID: <236D5828CF7F5749AF1BCB7F87596DE103BA3E9D@SNV-EXVS09.ds.corp.yahoo.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: [VOTE] Release Pig 0.3.0 (candidate 0)
Thread-Index: AcnwSzdeHvsmuCnUSiKzZJzKb8waJw==
From: "Olga Natkovich" <olgan@yahoo-inc.com>
To: <pig-dev@hadoop.apache.org>, <private@hadoop.apache.org>,
        <general@hadoop.apache.org>
X-OriginalArrivalTime: 18 Jun 2009 19:31:16.0902 (UTC) FILETIME=[59597C60:01C9F04B]
X-Virus-Checked: Checked by ClamAV on apache.org

------_=_NextPart_001_01C9F04B.376C558E
Content-Type: text/plain;
	charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable

Hi,
=20
I created a candidate build for Pig 0.3.0 release. The main feature of
this release is support for multiquery which allows to share computation
across multiple queries within the same script. We see significant
performance improvements (up to order of magnitude) as the result of
this optimization.
=20
I ran the rat report and made sure that all the source files contain
proper headers. (Not attaching the report since it caused trouble with
the last release.)
=20
Keys used to sign the release candidate are at
http://svn.apache.org/viewvc/hadoop/pig/trunk/KEYS.
=20
Please, download and try the release candidate:
http://people.apache.org/~olga/pig-0.3.0-candidate-0/.
=20
Please, vote by Wednesday, June 24th.
=20
Olga
=20

------_=_NextPart_001_01C9F04B.376C558E--

From general-return-263-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 19 16:11:34 2009
Return-Path: <general-return-263-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 13512 invoked from network); 19 Jun 2009 16:11:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jun 2009 16:11:34 -0000
Received: (qmail 92144 invoked by uid 500); 19 Jun 2009 16:11:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92067 invoked by uid 500); 19 Jun 2009 16:11:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 92057 invoked by uid 99); 19 Jun 2009 16:11:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Jun 2009 16:11:44 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [192.100.122.233] (HELO mgw-mx06.nokia.com) (192.100.122.233)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Jun 2009 16:11:34 +0000
Received: from vaebh106.NOE.Nokia.com (vaebh106.europe.nokia.com [10.160.244.32])
	by mgw-mx06.nokia.com (Switch-3.3.3/Switch-3.3.3) with ESMTP id n5JGAmY6013323
	for <general@hadoop.apache.org>; Fri, 19 Jun 2009 19:11:05 +0300
Received: from esebh102.NOE.Nokia.com ([172.21.138.183]) by vaebh106.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Fri, 19 Jun 2009 19:11:02 +0300
Received: from mgw-sa01.ext.nokia.com ([147.243.1.47]) by esebh102.NOE.Nokia.com over TLS secured channel with Microsoft SMTPSVC(6.0.3790.3959);
	 Fri, 19 Jun 2009 19:11:01 +0300
Received: from [172.19.167.46] (bsdhcp16746.americas.nokia.com [172.19.167.46])
	by mgw-sa01.ext.nokia.com (Switch-3.2.6/Switch-3.2.6) with ESMTP id n5JGAxxC019441
	for <general@hadoop.apache.org>; Fri, 19 Jun 2009 19:10:59 +0300
Subject: HDFS name node HA roadmap
From: Andrew Wharton <andrew.wharton@nokia.com>
Reply-To: andrew.wharton@nokia.com
To: general@hadoop.apache.org
Content-Type: text/plain
Organization: Nokia, Inc.
Date: Fri, 19 Jun 2009 12:10:58 -0400
Message-Id: <1245427858.5132.2.camel@1USL14830.NOE.Nokia.com>
Mime-Version: 1.0
X-Mailer: Evolution 2.26.1 
Content-Transfer-Encoding: 7bit
X-OriginalArrivalTime: 19 Jun 2009 16:11:01.0745 (UTC) FILETIME=[8A2BC610:01C9F0F8]
X-Nokia-AV: Clean
X-Virus-Checked: Checked by ClamAV on apache.org

Hello,

My understanding is that the HDFS name node single point of failure will
be removed in Hadoop 0.21. Is that still the case? Also, what is the
expected delivery date of that release?

Thanks.

-- Andrew


From general-return-264-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 19 16:37:50 2009
Return-Path: <general-return-264-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 20459 invoked from network); 19 Jun 2009 16:37:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jun 2009 16:37:50 -0000
Received: (qmail 22747 invoked by uid 500); 19 Jun 2009 16:24:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22726 invoked by uid 500); 19 Jun 2009 16:24:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22716 invoked by uid 99); 19 Jun 2009 16:24:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Jun 2009 16:24:28 +0000
X-ASF-Spam-Status: No, hits=-1.8 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of qiujie@cn.ibm.com designates 202.81.31.141 as permitted sender)
Received: from [202.81.31.141] (HELO e23smtp08.au.ibm.com) (202.81.31.141)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Jun 2009 16:24:17 +0000
Received: from d23relay02.au.ibm.com (d23relay02.au.ibm.com [202.81.31.244])
	by e23smtp08.au.ibm.com (8.13.1/8.13.1) with ESMTP id n5K2KpcC001466
	for <general@hadoop.apache.org>; Sat, 20 Jun 2009 12:20:51 +1000
Received: from d23av02.au.ibm.com (d23av02.au.ibm.com [9.190.235.138])
	by d23relay02.au.ibm.com (8.13.8/8.13.8/NCO v9.2) with ESMTP id n5JGNsUq835806
	for <general@hadoop.apache.org>; Sat, 20 Jun 2009 02:23:55 +1000
Received: from d23av02.au.ibm.com (loopback [127.0.0.1])
	by d23av02.au.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id n5JGNse5019533
	for <general@hadoop.apache.org>; Sat, 20 Jun 2009 02:23:54 +1000
Received: from d23m0037.cn.ibm.com (d23m0037.cn.ibm.com [9.181.2.105])
	by d23av02.au.ibm.com (8.12.11.20060308/8.12.11) with ESMTP id n5JGNrhQ019512;
	Sat, 20 Jun 2009 02:23:54 +1000
In-Reply-To: <1245427858.5132.2.camel@1USL14830.NOE.Nokia.com>
To: general@hadoop.apache.org
Cc: shv@yahoo-inc.com, andrew.wharton@nokia.com
MIME-Version: 1.0
Subject: Re: HDFS name node HA roadmap
X-Mailer: Lotus Notes Release 7.0 HF277 June 21, 2006
Message-ID: <OF09D9C8A4.DE5DA975-ON482575DA.0059ACC8-482575DA.005A1346@cn.ibm.com>
From: Jie Qiu <qiujie@cn.ibm.com>
Date: Sat, 20 Jun 2009 00:23:59 +0800
X-MIMETrack: Serialize by Router on D23M0037/23/M/IBM(Release 8.0.1|February 07, 2008) at
 06/20/2009 00:24:00,
	Serialize complete at 06/20/2009 00:24:00
Content-Type: multipart/alternative; boundary="=_alternative 005A1343482575DA_="
X-Virus-Checked: Checked by ClamAV on apache.org

--=_alternative 005A1343482575DA_=
Content-Type: text/plain; charset="US-ASCII"

Hi Andrew,

Yes, our IBM is implementing HA for Hadoop which will be open sourced 
soon. I think shv from Yahoo also post JIRA to describe this issue.

However, our approach may be different with shv. Let me discuss with shv 
about how to combine our thoughts and implementation together. 

==================================================
True insight comes from within!

Qiu Jie

Autonomic Middleware and Service Delivery
IBM China Research Lab.
Tel: 8610-58748086  Tieline: 905-8086
FAX: 8610-58748330 
Add.: Building 19 Zhongguancun Software Park, 8 Dongbeiwang WestRoad, 
Haidian District, Beijing,P.R.C.100094
Email: qiujie@cn.ibm.com
==================================================



Andrew Wharton <andrew.wharton@nokia.com> 
2009-06-20 00:10
Please respond to
general@hadoop.apache.org


To
general@hadoop.apache.org
cc

Subject
HDFS name node HA roadmap






Hello,

My understanding is that the HDFS name node single point of failure will
be removed in Hadoop 0.21. Is that still the case? Also, what is the
expected delivery date of that release?

Thanks.

-- Andrew



--=_alternative 005A1343482575DA_=--

From general-return-265-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 19 17:16:49 2009
Return-Path: <general-return-265-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37285 invoked from network); 19 Jun 2009 17:16:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jun 2009 17:16:49 -0000
Received: (qmail 11816 invoked by uid 500); 19 Jun 2009 17:17:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11761 invoked by uid 500); 19 Jun 2009 17:17:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11751 invoked by uid 99); 19 Jun 2009 17:17:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Jun 2009 17:17:00 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of owen.omalley@gmail.com designates 74.125.46.28 as permitted sender)
Received: from [74.125.46.28] (HELO yw-out-2324.google.com) (74.125.46.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Jun 2009 17:16:52 +0000
Received: by yw-out-2324.google.com with SMTP id 9so854266ywe.29
        for <general@hadoop.apache.org>; Fri, 19 Jun 2009 10:16:31 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=kG0vzUVzNS3dPQnwVtQiQoycJhPcWze0sVuaUBHbn2k=;
        b=NOF0YY/bmkgl89qRZgBLK6gXzmqbdXwmZeEm7hyHuXwvRv8KewRhc8qVYmYZPq6aRP
         TtT25HxVcsgGftCTF5UK/XHaTVbN+QDnfqW+Sze1YchCPYE8XR/0gtLqmUsIA0IchyQD
         MBEWrBNzz3hKKmBXQvtnNGD2cy809YosuoJhc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=S8bcGcUm4Ss/OiFhJH1aVAS3lsoikGFY3wQ6ag2GXU2NwrIJpLEto2RJwoKtgjt0RM
         3sBeJG+sO5BgEQkBZDe3Tu3/vYEf1KQV4QnxkQFhLVwEPsJf47lqvKDjywNEKaQa57m9
         yL1QtxQ43QDw7+ia2ezOf49GE/oq8vJ0enBnA=
MIME-Version: 1.0
Received: by 10.151.135.11 with SMTP id m11mr6155850ybn.228.1245431791394; 
	Fri, 19 Jun 2009 10:16:31 -0700 (PDT)
In-Reply-To: <OF09D9C8A4.DE5DA975-ON482575DA.0059ACC8-482575DA.005A1346@cn.ibm.com>
References: <1245427858.5132.2.camel@1USL14830.NOE.Nokia.com>
	 <OF09D9C8A4.DE5DA975-ON482575DA.0059ACC8-482575DA.005A1346@cn.ibm.com>
Date: Fri, 19 Jun 2009 10:16:31 -0700
Message-ID: <5f7f740b0906191016m3e1862ebs26e4fc5f5f0635e0@mail.gmail.com>
Subject: Re: HDFS name node HA roadmap
From: "Owen O'Malley" <owen.omalley@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001e680f0938849eee046cb6afd9
X-Virus-Checked: Checked by ClamAV on apache.org

--001e680f0938849eee046cb6afd9
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

On Fri, Jun 19, 2009 at 9:23 AM, Jie Qiu <qiujie@cn.ibm.com> wrote:

However, our approach may be different with shv. Let me discuss with shv
> about how to combine our thoughts and implementation together.


If you want a major change like this to go into the real code base, you need
to open a jira, and let people comment on your architecture, design and test
plans. If you present it as a huge code drop, it is unlikely to go in.

-- Owen

--001e680f0938849eee046cb6afd9--

From general-return-266-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 19 17:19:54 2009
Return-Path: <general-return-266-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38725 invoked from network); 19 Jun 2009 17:19:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jun 2009 17:19:54 -0000
Received: (qmail 15975 invoked by uid 500); 19 Jun 2009 17:20:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15911 invoked by uid 500); 19 Jun 2009 17:20:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15900 invoked by uid 99); 19 Jun 2009 17:20:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Jun 2009 17:20:05 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 209.85.217.215 as permitted sender)
Received: from [209.85.217.215] (HELO mail-gx0-f215.google.com) (209.85.217.215)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Jun 2009 17:19:57 +0000
Received: by gxk11 with SMTP id 11so2758173gxk.5
        for <general@hadoop.apache.org>; Fri, 19 Jun 2009 10:19:36 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:cc:content-type;
        bh=P1FN1jWZkAbokDGgcFsRW0AP917AwHhQXQGUWGkMV4k=;
        b=UhRLbd81Fb956q9ff+q7ChzXiVMPe59S9S6R6qY3NcaGNaKGXucWhkQPpy0BiIx/g1
         FmnSKIMuO6qF/XNp/gUsjRQ+0pTW9wSp87Okip0LP0AqATofC+3xrMy2fmHRmWJfz68F
         +Cf6XZw/X0x1sFflc/Lf+y81xONh2TvY0F0Yw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=OOqWzdR1KpB8HcCmpTTlv8Z0J7U9vKmiuVHwOBfqHZVULbtWSdzVg78Av+XKoOOOPB
         NpBpn8XhyZ7PIMpZVNPGlNIZmu+/NCSk+K0N9H6iNAjDZdUMjf7qd5FPfIXDllDHPlRy
         gadJSMz60p6aDrEGLclfTOFLPjafkg6JSRrPo=
MIME-Version: 1.0
Received: by 10.151.134.2 with SMTP id l2mr6160180ybn.163.1245431975886; Fri, 
	19 Jun 2009 10:19:35 -0700 (PDT)
In-Reply-To: <OF09D9C8A4.DE5DA975-ON482575DA.0059ACC8-482575DA.005A1346@cn.ibm.com>
References: <1245427858.5132.2.camel@1USL14830.NOE.Nokia.com>
	 <OF09D9C8A4.DE5DA975-ON482575DA.0059ACC8-482575DA.005A1346@cn.ibm.com>
Date: Fri, 19 Jun 2009 10:19:35 -0700
Message-ID: <4aa34eb70906191019t31618bf0jd6a7ee1a92f55525@mail.gmail.com>
Subject: Re: HDFS name node HA roadmap
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Cc: shv@yahoo-inc.com, andrew.wharton@nokia.com
Content-Type: multipart/alternative; boundary=001b24be338c83c13d046cb6ba3d
X-Virus-Checked: Checked by ClamAV on apache.org

--001b24be338c83c13d046cb6ba3d
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi Hie,

If you are working on this one, it woudl be nice if you can open a JIRA and
let the community comment/review your design. This process usually takes a
while and it is good to get a an early start on it.

thanks,
dhruba


On Fri, Jun 19, 2009 at 9:23 AM, Jie Qiu <qiujie@cn.ibm.com> wrote:

> Hi Andrew,
>
> Yes, our IBM is implementing HA for Hadoop which will be open sourced
> soon. I think shv from Yahoo also post JIRA to describe this issue.
>
> However, our approach may be different with shv. Let me discuss with shv
> about how to combine our thoughts and implementation together.
>
> ==================================================
> True insight comes from within!
>
> Qiu Jie
>
> Autonomic Middleware and Service Delivery
> IBM China Research Lab.
> Tel: 8610-58748086  Tieline: 905-8086
> FAX: 8610-58748330
> Add.: Building 19 Zhongguancun Software Park, 8 Dongbeiwang WestRoad,
> Haidian District, Beijing,P.R.C.100094
> Email: qiujie@cn.ibm.com
> ==================================================
>
>
>
> Andrew Wharton <andrew.wharton@nokia.com>
> 2009-06-20 00:10
> Please respond to
> general@hadoop.apache.org
>
>
> To
> general@hadoop.apache.org
> cc
>
> Subject
> HDFS name node HA roadmap
>
>
>
>
>
>
> Hello,
>
> My understanding is that the HDFS name node single point of failure will
> be removed in Hadoop 0.21. Is that still the case? Also, what is the
> expected delivery date of that release?
>
> Thanks.
>
> -- Andrew
>
>
>

--001b24be338c83c13d046cb6ba3d--

From general-return-267-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 19 17:29:19 2009
Return-Path: <general-return-267-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 48780 invoked from network); 19 Jun 2009 17:29:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jun 2009 17:29:19 -0000
Received: (qmail 29558 invoked by uid 500); 19 Jun 2009 17:29:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29480 invoked by uid 500); 19 Jun 2009 17:29:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29470 invoked by uid 99); 19 Jun 2009 17:29:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Jun 2009 17:29:29 +0000
X-ASF-Spam-Status: No, hits=-1.8 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of qiujie@cn.ibm.com designates 202.81.31.148 as permitted sender)
Received: from [202.81.31.148] (HELO e23smtp06.au.ibm.com) (202.81.31.148)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Jun 2009 17:29:17 +0000
Received: from d23relay01.au.ibm.com (d23relay01.au.ibm.com [202.81.31.243])
	by e23smtp06.au.ibm.com (8.13.1/8.13.1) with ESMTP id n5JHSkZn014762
	for <general@hadoop.apache.org>; Sat, 20 Jun 2009 03:28:46 +1000
Received: from d23av01.au.ibm.com (d23av01.au.ibm.com [9.190.234.96])
	by d23relay01.au.ibm.com (8.13.8/8.13.8/NCO v9.2) with ESMTP id n5JHSrm7536868
	for <general@hadoop.apache.org>; Sat, 20 Jun 2009 03:28:53 +1000
Received: from d23av01.au.ibm.com (loopback [127.0.0.1])
	by d23av01.au.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id n5JHSrGg030260
	for <general@hadoop.apache.org>; Sat, 20 Jun 2009 03:28:53 +1000
Received: from d23m0037.cn.ibm.com (d23m0037.cn.ibm.com [9.181.2.105])
	by d23av01.au.ibm.com (8.12.11.20060308/8.12.11) with ESMTP id n5JHSqP4030257;
	Sat, 20 Jun 2009 03:28:52 +1000
In-Reply-To: <4aa34eb70906191019t31618bf0jd6a7ee1a92f55525@mail.gmail.com>
To: general@hadoop.apache.org
Cc: andrew.wharton@nokia.com, general@hadoop.apache.org, shv@yahoo-inc.com
MIME-Version: 1.0
Subject: Re: HDFS name node HA roadmap
X-Mailer: Lotus Notes Release 7.0 HF277 June 21, 2006
Message-ID: <OF2C3916CB.40CA1B26-ON482575DA.005F9060-482575DA.00600626@cn.ibm.com>
From: Jie Qiu <qiujie@cn.ibm.com>
Date: Sat, 20 Jun 2009 01:28:57 +0800
X-MIMETrack: Serialize by Router on D23M0037/23/M/IBM(Release 8.0.1|February 07, 2008) at
 06/20/2009 01:28:59,
	Serialize complete at 06/20/2009 01:28:59
Content-Type: multipart/alternative; boundary="=_alternative 00600623482575DA_="
X-Virus-Checked: Checked by ClamAV on apache.org

--=_alternative 00600623482575DA_=
Content-Type: text/plain; charset="US-ASCII"

Hi, Dhruba,

Yes, we plan to open a JIRA to let community to review our implementation 
and give comments. Recently, we are going with IBM internal process before 
submitting to the community.

I also want to discuss with shv whether we need to another separate JIRA 
or work on the same integrated JIRA.

Do you any suggestion on this?

==================================================
True insight comes from within!

Qiu Jie

Autonomic Middleware and Service Delivery
IBM China Research Lab.
Tel: 8610-58748086  Tieline: 905-8086
FAX: 8610-58748330 
Add.: Building 19 Zhongguancun Software Park, 8 Dongbeiwang WestRoad, 
Haidian District, Beijing,P.R.C.100094
Email: qiujie@cn.ibm.com
==================================================



Dhruba Borthakur <dhruba@gmail.com> 
2009-06-20 01:19
Please respond to
general@hadoop.apache.org


To
general@hadoop.apache.org
cc
shv@yahoo-inc.com, andrew.wharton@nokia.com
Subject
Re: HDFS name node HA roadmap






Hi Hie,

If you are working on this one, it woudl be nice if you can open a JIRA 
and
let the community comment/review your design. This process usually takes a
while and it is good to get a an early start on it.

thanks,
dhruba


On Fri, Jun 19, 2009 at 9:23 AM, Jie Qiu <qiujie@cn.ibm.com> wrote:

> Hi Andrew,
>
> Yes, our IBM is implementing HA for Hadoop which will be open sourced
> soon. I think shv from Yahoo also post JIRA to describe this issue.
>
> However, our approach may be different with shv. Let me discuss with shv
> about how to combine our thoughts and implementation together.
>
> ==================================================
> True insight comes from within!
>
> Qiu Jie
>
> Autonomic Middleware and Service Delivery
> IBM China Research Lab.
> Tel: 8610-58748086  Tieline: 905-8086
> FAX: 8610-58748330
> Add.: Building 19 Zhongguancun Software Park, 8 Dongbeiwang WestRoad,
> Haidian District, Beijing,P.R.C.100094
> Email: qiujie@cn.ibm.com
> ==================================================
>
>
>
> Andrew Wharton <andrew.wharton@nokia.com>
> 2009-06-20 00:10
> Please respond to
> general@hadoop.apache.org
>
>
> To
> general@hadoop.apache.org
> cc
>
> Subject
> HDFS name node HA roadmap
>
>
>
>
>
>
> Hello,
>
> My understanding is that the HDFS name node single point of failure will
> be removed in Hadoop 0.21. Is that still the case? Also, what is the
> expected delivery date of that release?
>
> Thanks.
>
> -- Andrew
>
>
>


--=_alternative 00600623482575DA_=--

From general-return-268-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 19 17:46:13 2009
Return-Path: <general-return-268-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58085 invoked from network); 19 Jun 2009 17:46:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jun 2009 17:46:13 -0000
Received: (qmail 48889 invoked by uid 500); 19 Jun 2009 17:46:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48818 invoked by uid 500); 19 Jun 2009 17:46:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48808 invoked by uid 99); 19 Jun 2009 17:46:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Jun 2009 17:46:24 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Jun 2009 17:46:12 +0000
Received: from SNV-EXPF01.ds.corp.yahoo.com (snv-expf01.ds.corp.yahoo.com [207.126.227.250])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n5JHjIYW019066;
	Fri, 19 Jun 2009 10:45:19 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:cc:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:x-originalarrivaltime;
	b=sgzFsloTusNfouIFrD0tWbE9PJXeu2y50TWAKtQw/GzV9MjVfh4DRksnrNCKMANO
Received: from SNV-EXVS09.ds.corp.yahoo.com ([207.126.227.87]) by SNV-EXPF01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Fri, 19 Jun 2009 10:45:18 -0700
Received: from 10.73.146.106 ([10.73.146.106]) by SNV-EXVS09.ds.corp.yahoo.com ([207.126.227.84]) via Exchange Front-End Server snv-webmail.corp.yahoo.com ([207.126.227.60]) with Microsoft Exchange Server HTTP-DAV ;
 Fri, 19 Jun 2009 17:44:55 +0000
User-Agent: Microsoft-Entourage/12.19.0.090515
Date: Fri, 19 Jun 2009 10:44:54 -0700
Subject: Re: HDFS name node HA roadmap
From: Mahadev Konar <mahadev@yahoo-inc.com>
To: <general@hadoop.apache.org>
CC: <andrew.wharton@nokia.com>, <shv@yahoo-inc.com>
Message-ID: <C6611CA6.1ECB4%mahadev@yahoo-inc.com>
Thread-Topic: HDFS name node HA roadmap
Thread-Index: AcnxBadBYwaT3PkG+02cKROvsirQFg==
In-Reply-To: <OF2C3916CB.40CA1B26-ON482575DA.005F9060-482575DA.00600626@cn.ibm.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-OriginalArrivalTime: 19 Jun 2009 17:45:18.0938 (UTC) FILETIME=[B61EEBA0:01C9F105]
X-Virus-Checked: Checked by ClamAV on apache.org

Does anyone know the jira that shv (konstantin) has opened?

mahadev


On 6/19/09 10:28 AM, "Jie Qiu" <qiujie@cn.ibm.com> wrote:

> Hi, Dhruba,
> 
> Yes, we plan to open a JIRA to let community to review our implementation
> and give comments. Recently, we are going with IBM internal process before
> submitting to the community.
> 
> I also want to discuss with shv whether we need to another separate JIRA
> or work on the same integrated JIRA.
> 
> Do you any suggestion on this?
> 
> ==================================================
> True insight comes from within!
> 
> Qiu Jie
> 
> Autonomic Middleware and Service Delivery
> IBM China Research Lab.
> Tel: 8610-58748086  Tieline: 905-8086
> FAX: 8610-58748330
> Add.: Building 19 Zhongguancun Software Park, 8 Dongbeiwang WestRoad,
> Haidian District, Beijing,P.R.C.100094
> Email: qiujie@cn.ibm.com
> ==================================================
> 
> 
> 
> Dhruba Borthakur <dhruba@gmail.com>
> 2009-06-20 01:19
> Please respond to
> general@hadoop.apache.org
> 
> 
> To
> general@hadoop.apache.org
> cc
> shv@yahoo-inc.com, andrew.wharton@nokia.com
> Subject
> Re: HDFS name node HA roadmap
> 
> 
> 
> 
> 
> 
> Hi Hie,
> 
> If you are working on this one, it woudl be nice if you can open a JIRA
> and
> let the community comment/review your design. This process usually takes a
> while and it is good to get a an early start on it.
> 
> thanks,
> dhruba
> 
> 
> On Fri, Jun 19, 2009 at 9:23 AM, Jie Qiu <qiujie@cn.ibm.com> wrote:
> 
>> Hi Andrew,
>> 
>> Yes, our IBM is implementing HA for Hadoop which will be open sourced
>> soon. I think shv from Yahoo also post JIRA to describe this issue.
>> 
>> However, our approach may be different with shv. Let me discuss with shv
>> about how to combine our thoughts and implementation together.
>> 
>> ==================================================
>> True insight comes from within!
>> 
>> Qiu Jie
>> 
>> Autonomic Middleware and Service Delivery
>> IBM China Research Lab.
>> Tel: 8610-58748086  Tieline: 905-8086
>> FAX: 8610-58748330
>> Add.: Building 19 Zhongguancun Software Park, 8 Dongbeiwang WestRoad,
>> Haidian District, Beijing,P.R.C.100094
>> Email: qiujie@cn.ibm.com
>> ==================================================
>> 
>> 
>> 
>> Andrew Wharton <andrew.wharton@nokia.com>
>> 2009-06-20 00:10
>> Please respond to
>> general@hadoop.apache.org
>> 
>> 
>> To
>> general@hadoop.apache.org
>> cc
>> 
>> Subject
>> HDFS name node HA roadmap
>> 
>> 
>> 
>> 
>> 
>> 
>> Hello,
>> 
>> My understanding is that the HDFS name node single point of failure will
>> be removed in Hadoop 0.21. Is that still the case? Also, what is the
>> expected delivery date of that release?
>> 
>> Thanks.
>> 
>> -- Andrew
>> 
>> 
>> 
> 


From general-return-269-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 19 17:51:24 2009
Return-Path: <general-return-269-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60453 invoked from network); 19 Jun 2009 17:51:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jun 2009 17:51:23 -0000
Received: (qmail 55050 invoked by uid 500); 19 Jun 2009 17:51:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55000 invoked by uid 500); 19 Jun 2009 17:51:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54990 invoked by uid 99); 19 Jun 2009 17:51:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Jun 2009 17:51:34 +0000
X-ASF-Spam-Status: No, hits=-1.8 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of qiujie@cn.ibm.com designates 202.81.31.148 as permitted sender)
Received: from [202.81.31.148] (HELO e23smtp06.au.ibm.com) (202.81.31.148)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Jun 2009 17:51:24 +0000
Received: from d23relay02.au.ibm.com (d23relay02.au.ibm.com [202.81.31.244])
	by e23smtp06.au.ibm.com (8.13.1/8.13.1) with ESMTP id n5JHorRj031900
	for <general@hadoop.apache.org>; Sat, 20 Jun 2009 03:50:53 +1000
Received: from d23av04.au.ibm.com (d23av04.au.ibm.com [9.190.235.139])
	by d23relay02.au.ibm.com (8.13.8/8.13.8/NCO v9.2) with ESMTP id n5JHp0Kc643220
	for <general@hadoop.apache.org>; Sat, 20 Jun 2009 03:51:00 +1000
Received: from d23av04.au.ibm.com (loopback [127.0.0.1])
	by d23av04.au.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id n5JHp0RD013192
	for <general@hadoop.apache.org>; Sat, 20 Jun 2009 03:51:00 +1000
Received: from d23m0037.cn.ibm.com (d23m0037.cn.ibm.com [9.181.2.105])
	by d23av04.au.ibm.com (8.12.11.20060308/8.12.11) with ESMTP id n5JHouIn013159;
	Sat, 20 Jun 2009 03:50:57 +1000
In-Reply-To: <C6611CA6.1ECB4%mahadev@yahoo-inc.com>
To: general@hadoop.apache.org
Cc: andrew.wharton@nokia.com, general@hadoop.apache.org, shv@yahoo-inc.com
MIME-Version: 1.0
Subject: Re: HDFS name node HA roadmap
X-Mailer: Lotus Notes Release 7.0 HF277 June 21, 2006
Message-ID: <OF17430CF6.93C10C0D-ON482575DA.0061EF28-482575DA.00620B06@cn.ibm.com>
From: Jie Qiu <qiujie@cn.ibm.com>
Date: Sat, 20 Jun 2009 01:51:01 +0800
X-MIMETrack: Serialize by Router on D23M0037/23/M/IBM(Release 8.0.1|February 07, 2008) at
 06/20/2009 01:51:03,
	Serialize complete at 06/20/2009 01:51:03
Content-Type: multipart/related; boundary="=_related 00620AFB482575DA_="
X-Virus-Checked: Checked by ClamAV on apache.org

--=_related 00620AFB482575DA_=
Content-Type: multipart/alternative; boundary="=_alternative 00620AFB482575DA_="


--=_alternative 00620AFB482575DA_=
Content-Type: text/plain; charset="US-ASCII"

https://issues.apache.org/jira/browse/HADOOP-5651



==================================================
True insight comes from within!

Qiu Jie

Autonomic Middleware and Service Delivery
IBM China Research Lab.
Tel: 8610-58748086  Tieline: 905-8086
FAX: 8610-58748330 
Add.: Building 19 Zhongguancun Software Park, 8 Dongbeiwang WestRoad, 
Haidian District, Beijing,P.R.C.100094
Email: qiujie@cn.ibm.com
==================================================



Mahadev Konar <mahadev@yahoo-inc.com> 
2009-06-20 01:44
Please respond to
general@hadoop.apache.org


To
<general@hadoop.apache.org>
cc
<andrew.wharton@nokia.com>, <shv@yahoo-inc.com>
Subject
Re: HDFS name node HA roadmap






Does anyone know the jira that shv (konstantin) has opened?

mahadev


On 6/19/09 10:28 AM, "Jie Qiu" <qiujie@cn.ibm.com> wrote:

> Hi, Dhruba,
> 
> Yes, we plan to open a JIRA to let community to review our 
implementation
> and give comments. Recently, we are going with IBM internal process 
before
> submitting to the community.
> 
> I also want to discuss with shv whether we need to another separate JIRA
> or work on the same integrated JIRA.
> 
> Do you any suggestion on this?
> 
> ==================================================
> True insight comes from within!
> 
> Qiu Jie
> 
> Autonomic Middleware and Service Delivery
> IBM China Research Lab.
> Tel: 8610-58748086  Tieline: 905-8086
> FAX: 8610-58748330
> Add.: Building 19 Zhongguancun Software Park, 8 Dongbeiwang WestRoad,
> Haidian District, Beijing,P.R.C.100094
> Email: qiujie@cn.ibm.com
> ==================================================
> 
> 
> 
> Dhruba Borthakur <dhruba@gmail.com>
> 2009-06-20 01:19
> Please respond to
> general@hadoop.apache.org
> 
> 
> To
> general@hadoop.apache.org
> cc
> shv@yahoo-inc.com, andrew.wharton@nokia.com
> Subject
> Re: HDFS name node HA roadmap
> 
> 
> 
> 
> 
> 
> Hi Hie,
> 
> If you are working on this one, it woudl be nice if you can open a JIRA
> and
> let the community comment/review your design. This process usually takes 
a
> while and it is good to get a an early start on it.
> 
> thanks,
> dhruba
> 
> 
> On Fri, Jun 19, 2009 at 9:23 AM, Jie Qiu <qiujie@cn.ibm.com> wrote:
> 
>> Hi Andrew,
>> 
>> Yes, our IBM is implementing HA for Hadoop which will be open sourced
>> soon. I think shv from Yahoo also post JIRA to describe this issue.
>> 
>> However, our approach may be different with shv. Let me discuss with 
shv
>> about how to combine our thoughts and implementation together.
>> 
>> ==================================================
>> True insight comes from within!
>> 
>> Qiu Jie
>> 
>> Autonomic Middleware and Service Delivery
>> IBM China Research Lab.
>> Tel: 8610-58748086  Tieline: 905-8086
>> FAX: 8610-58748330
>> Add.: Building 19 Zhongguancun Software Park, 8 Dongbeiwang WestRoad,
>> Haidian District, Beijing,P.R.C.100094
>> Email: qiujie@cn.ibm.com
>> ==================================================
>> 
>> 
>> 
>> Andrew Wharton <andrew.wharton@nokia.com>
>> 2009-06-20 00:10
>> Please respond to
>> general@hadoop.apache.org
>> 
>> 
>> To
>> general@hadoop.apache.org
>> cc
>> 
>> Subject
>> HDFS name node HA roadmap
>> 
>> 
>> 
>> 
>> 
>> 
>> Hello,
>> 
>> My understanding is that the HDFS name node single point of failure 
will
>> be removed in Hadoop 0.21. Is that still the case? Also, what is the
>> expected delivery date of that release?
>> 
>> Thanks.
>> 
>> -- Andrew
>> 
>> 
>> 
> 



--=_alternative 00620AFB482575DA_=
Content-Type: text/html; charset="US-ASCII"


<br><a href="https://issues.apache.org/jira/browse/HADOOP-5651"><font size=2 color=blue face="sans-serif">https://issues.apache.org/jira/browse/HADOOP-5651</font></a>
<br>
<br><img src=cid:_1_0304CEC80304CAE000620AFB482575DA>
<br>
<br><font size=2 face="sans-serif">==================================================<br>
True insight comes from within!<br>
<br>
Qiu Jie<br>
<br>
Autonomic Middleware and Service Delivery<br>
IBM China Research Lab.<br>
Tel: 8610-58748086 &nbsp;Tieline: 905-8086<br>
FAX: 8610-58748330 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;<br>
Add.: Building 19 Zhongguancun Software Park, 8 Dongbeiwang WestRoad, Haidian
District, Beijing,P.R.C.100094<br>
Email: qiujie@cn.ibm.com<br>
==================================================</font>
<br>
<br>
<br>
<table width=100%>
<tr valign=top>
<td width=40%><font size=1 face="sans-serif"><b>Mahadev Konar &lt;mahadev@yahoo-inc.com&gt;</b>
</font>
<p><font size=1 face="sans-serif">2009-06-20 01:44</font>
<table border>
<tr valign=top>
<td bgcolor=white>
<div align=center><font size=1 face="sans-serif">Please respond to<br>
general@hadoop.apache.org</font></div></table>
<br>
<td width=59%>
<table width=100%>
<tr valign=top>
<td>
<div align=right><font size=1 face="sans-serif">To</font></div>
<td><font size=1 face="sans-serif">&lt;general@hadoop.apache.org&gt;</font>
<tr valign=top>
<td>
<div align=right><font size=1 face="sans-serif">cc</font></div>
<td><font size=1 face="sans-serif">&lt;andrew.wharton@nokia.com&gt;, &lt;shv@yahoo-inc.com&gt;</font>
<tr valign=top>
<td>
<div align=right><font size=1 face="sans-serif">Subject</font></div>
<td><font size=1 face="sans-serif">Re: HDFS name node HA roadmap</font></table>
<br>
<table>
<tr valign=top>
<td>
<td></table>
<br></table>
<br>
<br>
<br><tt><font size=2>Does anyone know the jira that shv (konstantin) has
opened?<br>
<br>
mahadev<br>
<br>
<br>
On 6/19/09 10:28 AM, &quot;Jie Qiu&quot; &lt;qiujie@cn.ibm.com&gt; wrote:<br>
<br>
&gt; Hi, Dhruba,<br>
&gt; <br>
&gt; Yes, we plan to open a JIRA to let community to review our implementation<br>
&gt; and give comments. Recently, we are going with IBM internal process
before<br>
&gt; submitting to the community.<br>
&gt; <br>
&gt; I also want to discuss with shv whether we need to another separate
JIRA<br>
&gt; or work on the same integrated JIRA.<br>
&gt; <br>
&gt; Do you any suggestion on this?<br>
&gt; <br>
&gt; ==================================================<br>
&gt; True insight comes from within!<br>
&gt; <br>
&gt; Qiu Jie<br>
&gt; <br>
&gt; Autonomic Middleware and Service Delivery<br>
&gt; IBM China Research Lab.<br>
&gt; Tel: 8610-58748086 &nbsp;Tieline: 905-8086<br>
&gt; FAX: 8610-58748330<br>
&gt; Add.: Building 19 Zhongguancun Software Park, 8 Dongbeiwang WestRoad,<br>
&gt; Haidian District, Beijing,P.R.C.100094<br>
&gt; Email: qiujie@cn.ibm.com<br>
&gt; ==================================================<br>
&gt; <br>
&gt; <br>
&gt; <br>
&gt; Dhruba Borthakur &lt;dhruba@gmail.com&gt;<br>
&gt; 2009-06-20 01:19<br>
&gt; Please respond to<br>
&gt; general@hadoop.apache.org<br>
&gt; <br>
&gt; <br>
&gt; To<br>
&gt; general@hadoop.apache.org<br>
&gt; cc<br>
&gt; shv@yahoo-inc.com, andrew.wharton@nokia.com<br>
&gt; Subject<br>
&gt; Re: HDFS name node HA roadmap<br>
&gt; <br>
&gt; <br>
&gt; <br>
&gt; <br>
&gt; <br>
&gt; <br>
&gt; Hi Hie,<br>
&gt; <br>
&gt; If you are working on this one, it woudl be nice if you can open a
JIRA<br>
&gt; and<br>
&gt; let the community comment/review your design. This process usually
takes a<br>
&gt; while and it is good to get a an early start on it.<br>
&gt; <br>
&gt; thanks,<br>
&gt; dhruba<br>
&gt; <br>
&gt; <br>
&gt; On Fri, Jun 19, 2009 at 9:23 AM, Jie Qiu &lt;qiujie@cn.ibm.com&gt;
wrote:<br>
&gt; <br>
&gt;&gt; Hi Andrew,<br>
&gt;&gt; <br>
&gt;&gt; Yes, our IBM is implementing HA for Hadoop which will be open
sourced<br>
&gt;&gt; soon. I think shv from Yahoo also post JIRA to describe this issue.<br>
&gt;&gt; <br>
&gt;&gt; However, our approach may be different with shv. Let me discuss
with shv<br>
&gt;&gt; about how to combine our thoughts and implementation together.<br>
&gt;&gt; <br>
&gt;&gt; ==================================================<br>
&gt;&gt; True insight comes from within!<br>
&gt;&gt; <br>
&gt;&gt; Qiu Jie<br>
&gt;&gt; <br>
&gt;&gt; Autonomic Middleware and Service Delivery<br>
&gt;&gt; IBM China Research Lab.<br>
&gt;&gt; Tel: 8610-58748086 &nbsp;Tieline: 905-8086<br>
&gt;&gt; FAX: 8610-58748330<br>
&gt;&gt; Add.: Building 19 Zhongguancun Software Park, 8 Dongbeiwang WestRoad,<br>
&gt;&gt; Haidian District, Beijing,P.R.C.100094<br>
&gt;&gt; Email: qiujie@cn.ibm.com<br>
&gt;&gt; ==================================================<br>
&gt;&gt; <br>
&gt;&gt; <br>
&gt;&gt; <br>
&gt;&gt; Andrew Wharton &lt;andrew.wharton@nokia.com&gt;<br>
&gt;&gt; 2009-06-20 00:10<br>
&gt;&gt; Please respond to<br>
&gt;&gt; general@hadoop.apache.org<br>
&gt;&gt; <br>
&gt;&gt; <br>
&gt;&gt; To<br>
&gt;&gt; general@hadoop.apache.org<br>
&gt;&gt; cc<br>
&gt;&gt; <br>
&gt;&gt; Subject<br>
&gt;&gt; HDFS name node HA roadmap<br>
&gt;&gt; <br>
&gt;&gt; <br>
&gt;&gt; <br>
&gt;&gt; <br>
&gt;&gt; <br>
&gt;&gt; <br>
&gt;&gt; Hello,<br>
&gt;&gt; <br>
&gt;&gt; My understanding is that the HDFS name node single point of failure
will<br>
&gt;&gt; be removed in Hadoop 0.21. Is that still the case? Also, what
is the<br>
&gt;&gt; expected delivery date of that release?<br>
&gt;&gt; <br>
&gt;&gt; Thanks.<br>
&gt;&gt; <br>
&gt;&gt; -- Andrew<br>
&gt;&gt; <br>
&gt;&gt; <br>
&gt;&gt; <br>
&gt; <br>
<br>
</font></tt>
<br>
--=_alternative 00620AFB482575DA_=--
--=_related 00620AFB482575DA_=--

From general-return-270-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 19 17:55:09 2009
Return-Path: <general-return-270-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 62435 invoked from network); 19 Jun 2009 17:55:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jun 2009 17:55:09 -0000
Received: (qmail 58584 invoked by uid 500); 19 Jun 2009 17:55:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58509 invoked by uid 500); 19 Jun 2009 17:55:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 47087 invoked by uid 99); 19 Jun 2009 17:45:23 -0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:cc:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=Tu1VvEkpQVHd3pmYhPEfIuoPY44MPpoTmkHP/9UTpmrWFfnVAilxYLAR6C5V6lKl
Message-ID: <4A3BCE76.5070503@yahoo-inc.com>
Date: Fri, 19 Jun 2009 10:44:22 -0700
From: Konstantin Shvachko <shv@yahoo-inc.com>
User-Agent: Thunderbird 2.0.0.21 (Windows/20090302)
MIME-Version: 1.0
To: Jie Qiu <qiujie@cn.ibm.com>
CC: general@hadoop.apache.org, andrew.wharton@nokia.com
Subject: Re: HDFS name node HA roadmap
References: <OF2C3916CB.40CA1B26-ON482575DA.005F9060-482575DA.00600626@cn.ibm.com>
In-Reply-To: <OF2C3916CB.40CA1B26-ON482575DA.005F9060-482575DA.00600626@cn.ibm.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Jie Qiu,

It would be interesting to see what your approach to HA is.
Indeed, I had some ideas on how to implement it.
And I would be happy to talk to you about it.

Which jira do you refer to?

Thanks,
--Konstantin

Jie Qiu wrote:
> 
> Hi, Dhruba,
> 
> Yes, we plan to open a JIRA to let community to review our 
> implementation and give comments. Recently, we are going with IBM 
> internal process before submitting to the community.
> 
> I also want to discuss with shv whether we need to another separate JIRA 
> or work on the same integrated JIRA.
> 
> Do you any suggestion on this?
> 
> ==================================================
> True insight comes from within!
> 
> Qiu Jie
> 
> Autonomic Middleware and Service Delivery
> IBM China Research Lab.
> Tel: 8610-58748086  Tieline: 905-8086
> FAX: 8610-58748330          
> Add.: Building 19 Zhongguancun Software Park, 8 Dongbeiwang WestRoad, 
> Haidian District, Beijing,P.R.C.100094
> Email: qiujie@cn.ibm.com
> ==================================================
> 
> 
> *Dhruba Borthakur <dhruba@gmail.com>*
> 
> 2009-06-20 01:19
> Please respond to
> general@hadoop.apache.org
> 
> 
> 	
> To
> 	general@hadoop.apache.org
> cc
> 	shv@yahoo-inc.com, andrew.wharton@nokia.com
> Subject
> 	Re: HDFS name node HA roadmap
> 
> 
> 	
> 
> 
> 
> 
> 
> Hi Hie,
> 
> If you are working on this one, it woudl be nice if you can open a JIRA and
> let the community comment/review your design. This process usually takes a
> while and it is good to get a an early start on it.
> 
> thanks,
> dhruba
> 
> 
> On Fri, Jun 19, 2009 at 9:23 AM, Jie Qiu <qiujie@cn.ibm.com> wrote:
> 
>  > Hi Andrew,
>  >
>  > Yes, our IBM is implementing HA for Hadoop which will be open sourced
>  > soon. I think shv from Yahoo also post JIRA to describe this issue.
>  >
>  > However, our approach may be different with shv. Let me discuss with shv
>  > about how to combine our thoughts and implementation together.
>  >
>  > ==================================================
>  > True insight comes from within!
>  >
>  > Qiu Jie
>  >
>  > Autonomic Middleware and Service Delivery
>  > IBM China Research Lab.
>  > Tel: 8610-58748086  Tieline: 905-8086
>  > FAX: 8610-58748330
>  > Add.: Building 19 Zhongguancun Software Park, 8 Dongbeiwang WestRoad,
>  > Haidian District, Beijing,P.R.C.100094
>  > Email: qiujie@cn.ibm.com
>  > ==================================================
>  >
>  >
>  >
>  > Andrew Wharton <andrew.wharton@nokia.com>
>  > 2009-06-20 00:10
>  > Please respond to
>  > general@hadoop.apache.org
>  >
>  >
>  > To
>  > general@hadoop.apache.org
>  > cc
>  >
>  > Subject
>  > HDFS name node HA roadmap
>  >
>  >
>  >
>  >
>  >
>  >
>  > Hello,
>  >
>  > My understanding is that the HDFS name node single point of failure will
>  > be removed in Hadoop 0.21. Is that still the case? Also, what is the
>  > expected delivery date of that release?
>  >
>  > Thanks.
>  >
>  > -- Andrew
>  >
>  >
>  >
> 

From general-return-271-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 19 18:55:18 2009
Return-Path: <general-return-271-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95420 invoked from network); 19 Jun 2009 18:55:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jun 2009 18:55:16 -0000
Received: (qmail 27615 invoked by uid 500); 19 Jun 2009 18:55:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27542 invoked by uid 500); 19 Jun 2009 18:55:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27532 invoked by uid 99); 19 Jun 2009 18:55:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Jun 2009 18:55:27 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [192.100.122.230] (HELO mgw-mx03.nokia.com) (192.100.122.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Jun 2009 18:55:18 +0000
Received: from vaebh106.NOE.Nokia.com (vaebh106.europe.nokia.com [10.160.244.32])
	by mgw-mx03.nokia.com (Switch-3.3.3/Switch-3.3.3) with ESMTP id n5JIsl0O018601
	for <general@hadoop.apache.org>; Fri, 19 Jun 2009 21:54:52 +0300
Received: from esebh102.NOE.Nokia.com ([172.21.138.183]) by vaebh106.NOE.Nokia.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Fri, 19 Jun 2009 21:54:53 +0300
Received: from mgw-da01.ext.nokia.com ([147.243.128.24]) by esebh102.NOE.Nokia.com over TLS secured channel with Microsoft SMTPSVC(6.0.3790.3959);
	 Fri, 19 Jun 2009 21:54:52 +0300
Received: from [172.19.167.46] (bsdhcp16746.americas.nokia.com [172.19.167.46])
	by mgw-da01.ext.nokia.com (Switch-3.2.6/Switch-3.2.6) with ESMTP id n5JIsm1E014353
	for <general@hadoop.apache.org>; Fri, 19 Jun 2009 21:54:49 +0300
Subject: Re: HDFS name node HA roadmap
From: Andrew Wharton <andrew.wharton@nokia.com>
Reply-To: andrew.wharton@nokia.com
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <OF17430CF6.93C10C0D-ON482575DA.0061EF28-482575DA.00620B06@cn.ibm.com>
References: 
	 <OF17430CF6.93C10C0D-ON482575DA.0061EF28-482575DA.00620B06@cn.ibm.com>
Content-Type: text/plain
Organization: Nokia, Inc.
Date: Fri, 19 Jun 2009 14:54:49 -0400
Message-Id: <1245437689.5132.10.camel@1USL14830.NOE.Nokia.com>
Mime-Version: 1.0
X-Mailer: Evolution 2.26.1 
Content-Transfer-Encoding: 7bit
X-OriginalArrivalTime: 19 Jun 2009 18:54:52.0775 (UTC) FILETIME=[6DEBE770:01C9F10F]
X-Nokia-AV: Clean
X-Virus-Checked: Checked by ClamAV on apache.org

I asked on the HBase list and they pointed me to this ticket, which is
more to my liking: https://issues.apache.org/jira/browse/HADOOP-4539


From general-return-272-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 22 14:56:27 2009
Return-Path: <general-return-272-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23999 invoked from network); 22 Jun 2009 14:56:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jun 2009 14:56:27 -0000
Received: (qmail 58134 invoked by uid 500); 22 Jun 2009 14:56:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58066 invoked by uid 500); 22 Jun 2009 14:56:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58056 invoked by uid 99); 22 Jun 2009 14:56:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Jun 2009 14:56:37 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.30.24] (HELO QMTA02.emeryville.ca.mail.comcast.net) (76.96.30.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Jun 2009 14:56:26 +0000
Received: from OMTA03.emeryville.ca.mail.comcast.net ([76.96.30.27])
	by QMTA02.emeryville.ca.mail.comcast.net with comcast
	id 70Cb1c0020b6N64A22w5rn; Mon, 22 Jun 2009 14:56:05 +0000
Received: from [11.0.0.135] ([209.131.62.115])
	by OMTA03.emeryville.ca.mail.comcast.net with comcast
	id 72vw1c00K2VBGtd8P2vzqv; Mon, 22 Jun 2009 14:56:03 +0000
Message-Id: <C2E26C3B-3F10-4F3A-B779-A2AF2FD5C000@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <094B4654-8D07-4251-B521-9EAFAAFBCB4E@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v935.3)
Subject: [RESULT] Rename Core to Common
Date: Mon, 22 Jun 2009 07:55:56 -0700
References: <094B4654-8D07-4251-B521-9EAFAAFBCB4E@yahoo-inc.com>
X-Mailer: Apple Mail (2.935.3)
X-Virus-Checked: Checked by ClamAV on apache.org


On Jun 17, 2009, at 11:28 AM, Owen O'Malley wrote:

> I'd like to propose that we rename Core to Common. This will better  
> reflect the fact that it is no longer the "core" of Hadoop, but  
> rather just the common libraries.

With 13 +1's and no -1's, the vote passes. I'll file the  
infrastructure ticket to rename the core-* mailing lists.

-- Owen

From general-return-273-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 22 16:31:27 2009
Return-Path: <general-return-273-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69777 invoked from network); 22 Jun 2009 16:31:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jun 2009 16:31:25 -0000
Received: (qmail 72598 invoked by uid 500); 22 Jun 2009 16:31:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72510 invoked by uid 500); 22 Jun 2009 16:31:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72492 invoked by uid 99); 22 Jun 2009 16:31:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Jun 2009 16:31:34 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Jun 2009 16:31:23 +0000
Received: from [192.168.0.198] (snvvpn1-10-73-152-c147.hq.corp.yahoo.com [10.73.152.147])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n5MGU7ML016726;
	Mon, 22 Jun 2009 09:30:07 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=cc:message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=nK00Umo7E4+ThnBj+TJf09saD9JDyrzO4ODrlWMRvJVldEphWDxut8p0pFLWa6UL
Cc: <pig-dev@hadoop.apache.org>, <general@hadoop.apache.org>
Message-Id: <E3985969-8DEA-45E0-B071-C06E9471602B@yahoo-inc.com>
From: Alan Gates <gates@yahoo-inc.com>
To: private@hadoop.apache.org
In-Reply-To: <236D5828CF7F5749AF1BCB7F87596DE103BA3E9D@SNV-EXVS09.ds.corp.yahoo.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v935.3)
Subject: Re: [VOTE] Release Pig 0.3.0 (candidate 0)
Date: Mon, 22 Jun 2009 09:30:06 -0700
References: <236D5828CF7F5749AF1BCB7F87596DE103BA3E9D@SNV-EXVS09.ds.corp.yahoo.com>
X-Mailer: Apple Mail (2.935.3)
X-Virus-Checked: Checked by ClamAV on apache.org

Downloaded, ran, ran tutorial, built piggybank.  All looks good.

+1

Alan.

On Jun 18, 2009, at 12:30 PM, Olga Natkovich wrote:

> Hi,
>
> I created a candidate build for Pig 0.3.0 release. The main feature of
> this release is support for multiquery which allows to share  
> computation
> across multiple queries within the same script. We see significant
> performance improvements (up to order of magnitude) as the result of
> this optimization.
>
> I ran the rat report and made sure that all the source files contain
> proper headers. (Not attaching the report since it caused trouble with
> the last release.)
>
> Keys used to sign the release candidate are at
> http://svn.apache.org/viewvc/hadoop/pig/trunk/KEYS.
>
> Please, download and try the release candidate:
> http://people.apache.org/~olga/pig-0.3.0-candidate-0/.
>
> Please, vote by Wednesday, June 24th.
>
> Olga
>


From general-return-274-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 22 17:47:08 2009
Return-Path: <general-return-274-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6075 invoked from network); 22 Jun 2009 17:47:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jun 2009 17:47:07 -0000
Received: (qmail 1720 invoked by uid 500); 22 Jun 2009 17:47:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1618 invoked by uid 500); 22 Jun 2009 17:47:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1587 invoked by uid 99); 22 Jun 2009 17:47:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Jun 2009 17:47:17 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Jun 2009 17:47:06 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n5MHkadQ078642;
	Mon, 22 Jun 2009 10:46:36 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=cc:message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=r936DlctCt35XOzpPMBLKW2tcuOTAVFoIoJlGIyxGCM4FNXB297zwUL4qjoe4gvL
Cc: <pig-dev@hadoop.apache.org>, <general@hadoop.apache.org>
Message-Id: <AE36757B-60AE-4D62-8941-881B095AB9A9@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: private@hadoop.apache.org
In-Reply-To: <236D5828CF7F5749AF1BCB7F87596DE103BA3E9D@SNV-EXVS09.ds.corp.yahoo.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v935.3)
Subject: Re: [VOTE] Release Pig 0.3.0 (candidate 0)
Date: Mon, 22 Jun 2009 10:46:36 -0700
References: <236D5828CF7F5749AF1BCB7F87596DE103BA3E9D@SNV-EXVS09.ds.corp.yahoo.com>
X-Mailer: Apple Mail (2.935.3)
X-Virus-Checked: Checked by ClamAV on apache.org


On Jun 18, 2009, at 12:30 PM, Olga Natkovich wrote:

> Hi,
>
> I created a candidate build for Pig 0.3.0 release. The main feature of
> this release is support for multiquery which allows to share  
> computation
> across multiple queries within the same script. We see significant
> performance improvements (up to order of magnitude) as the result of
> this optimization.
>

+1

I downloaded the release, validated checksums and ran the unit-tests  
successfully.

Arun


From general-return-275-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 22 17:50:38 2009
Return-Path: <general-return-275-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9218 invoked from network); 22 Jun 2009 17:50:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jun 2009 17:50:38 -0000
Received: (qmail 8430 invoked by uid 500); 22 Jun 2009 17:50:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8349 invoked by uid 500); 22 Jun 2009 17:50:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8330 invoked by uid 99); 22 Jun 2009 17:50:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Jun 2009 17:50:48 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Jun 2009 17:50:36 +0000
Received: from thickbeside-lm.corp.yahoo.com (thickbeside-lm.corp.yahoo.com [10.72.109.129])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n5MHmNor056432;
	Mon, 22 Jun 2009 10:48:24 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=cc:message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=lxRYjJPND7msLJDOUQb00XeFdNz2idtN7zD+kYvsRkrrmxUQyaO3MK9kplGrXeiY
Cc: pig-dev@hadoop.apache.org
Message-Id: <41E637DE-95E5-411D-9245-577A8775ED50@yahoo-inc.com>
From: Nigel Daley <ndaley@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <236D5828CF7F5749AF1BCB7F87596DE103BA3E9D@SNV-EXVS09.ds.corp.yahoo.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v935.3)
Subject: Re: [VOTE] Release Pig 0.3.0 (candidate 0)
Date: Mon, 22 Jun 2009 10:48:23 -0700
References: <236D5828CF7F5749AF1BCB7F87596DE103BA3E9D@SNV-EXVS09.ds.corp.yahoo.com>
X-Mailer: Apple Mail (2.935.3)
X-Virus-Checked: Checked by ClamAV on apache.org

+1.

On Jun 18, 2009, at 12:30 PM, Olga Natkovich wrote:

> Hi,
>
> I created a candidate build for Pig 0.3.0 release. The main feature of
> this release is support for multiquery which allows to share  
> computation
> across multiple queries within the same script. We see significant
> performance improvements (up to order of magnitude) as the result of
> this optimization.
>
> I ran the rat report and made sure that all the source files contain
> proper headers. (Not attaching the report since it caused trouble with
> the last release.)
>
> Keys used to sign the release candidate are at
> http://svn.apache.org/viewvc/hadoop/pig/trunk/KEYS.
>
> Please, download and try the release candidate:
> http://people.apache.org/~olga/pig-0.3.0-candidate-0/.
>
> Please, vote by Wednesday, June 24th.
>
> Olga
>


From general-return-276-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 22 17:51:44 2009
Return-Path: <general-return-276-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 10133 invoked from network); 22 Jun 2009 17:51:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jun 2009 17:51:44 -0000
Received: (qmail 10302 invoked by uid 500); 22 Jun 2009 17:51:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 10206 invoked by uid 500); 22 Jun 2009 17:51:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 10123 invoked by uid 99); 22 Jun 2009 17:51:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Jun 2009 17:51:54 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Jun 2009 17:51:43 +0000
Received: from SNV-EXPF01.ds.corp.yahoo.com (snv-expf01.ds.corp.yahoo.com [207.126.227.250])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n5MHp0uB086457;
	Mon, 22 Jun 2009 10:51:02 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:x-mimeole:content-class:mime-version:
	content-type:content-transfer-encoding:subject:date:message-id:
	in-reply-to:x-ms-has-attach:x-ms-tnef-correlator:thread-topic:
	thread-index:references:from:to:cc:x-originalarrivaltime;
	b=ycgKolX/LpQY26G9HDKFhsYfYK5/ndfnNdAOXH/4xjwEp1iH9Xq6CeUVKD53+2jY
Received: from SNV-EXVS06.ds.corp.yahoo.com ([207.126.227.234]) by SNV-EXPF01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Mon, 22 Jun 2009 10:51:01 -0700
x-mimeole: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: RE: [VOTE] Release Pig 0.3.0 (candidate 0)
Date: Mon, 22 Jun 2009 10:52:22 -0700
Message-ID: <11ED50FC7C760F4EA9F44109C531617D03F57BD8@SNV-EXVS06.ds.corp.yahoo.com>
In-Reply-To: <E3985969-8DEA-45E0-B071-C06E9471602B@yahoo-inc.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: [VOTE] Release Pig 0.3.0 (candidate 0)
Thread-Index: AcnzV/Hj+K85wPcbSnOP7XUOod/72QACSIqw
References: <236D5828CF7F5749AF1BCB7F87596DE103BA3E9D@SNV-EXVS09.ds.corp.yahoo.com> <E3985969-8DEA-45E0-B071-C06E9471602B@yahoo-inc.com>
From: "Pradeep Kamath" <pradeepk@yahoo-inc.com>
To: <pig-dev@hadoop.apache.org>, <private@hadoop.apache.org>
Cc: <general@hadoop.apache.org>
X-OriginalArrivalTime: 22 Jun 2009 17:51:01.0237 (UTC) FILETIME=[0162DA50:01C9F362]
X-Virus-Checked: Checked by ClamAV on apache.org

+1 for release.

-Pradeep

-----Original Message-----
From: Alan Gates [mailto:gates@yahoo-inc.com]=20
Sent: Monday, June 22, 2009 9:30 AM
To: private@hadoop.apache.org
Cc: pig-dev@hadoop.apache.org; general@hadoop.apache.org
Subject: Re: [VOTE] Release Pig 0.3.0 (candidate 0)

Downloaded, ran, ran tutorial, built piggybank.  All looks good.

+1

Alan.

On Jun 18, 2009, at 12:30 PM, Olga Natkovich wrote:

> Hi,
>
> I created a candidate build for Pig 0.3.0 release. The main feature of
> this release is support for multiquery which allows to share =20
> computation
> across multiple queries within the same script. We see significant
> performance improvements (up to order of magnitude) as the result of
> this optimization.
>
> I ran the rat report and made sure that all the source files contain
> proper headers. (Not attaching the report since it caused trouble with
> the last release.)
>
> Keys used to sign the release candidate are at
> http://svn.apache.org/viewvc/hadoop/pig/trunk/KEYS.
>
> Please, download and try the release candidate:
> http://people.apache.org/~olga/pig-0.3.0-candidate-0/.
>
> Please, vote by Wednesday, June 24th.
>
> Olga
>


From general-return-277-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 22 22:31:13 2009
Return-Path: <general-return-277-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44752 invoked from network); 22 Jun 2009 22:31:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jun 2009 22:31:13 -0000
Received: (qmail 57525 invoked by uid 500); 22 Jun 2009 22:31:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57445 invoked by uid 500); 22 Jun 2009 22:31:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57435 invoked by uid 99); 22 Jun 2009 22:31:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Jun 2009 22:31:23 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of lenz.grimmer@googlemail.com designates 209.85.221.178 as permitted sender)
Received: from [209.85.221.178] (HELO mail-qy0-f178.google.com) (209.85.221.178)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Jun 2009 22:31:11 +0000
Received: by qyk8 with SMTP id 8so4435213qyk.5
        for <general@hadoop.apache.org>; Mon, 22 Jun 2009 15:30:51 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:received:received:sender:message-id:date:from
         :user-agent:mime-version:to:subject:x-enigmail-version:openpgp
         :content-type;
        bh=nZCChYp3KgJMJdMMPL54e6VdpYX3UQeqnHBvTo8LRU4=;
        b=npaF6qii6VzQ9l1MKlnoiheTTOcwcojSjiftwLBvwyrCubzRjPVloRTUJuz5l1DGo+
         l4r8Osx7/tZDJVtjfxo6WFHU82bBohMFuFoehLrcWZ8j+yP8HBD92UlbiRf9EV0SG0CN
         wNpMNY0OFIgiv8QSOQ2qVIwVoyhxOLkpNr+P4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=sender:message-id:date:from:user-agent:mime-version:to:subject
         :x-enigmail-version:openpgp:content-type;
        b=PFbf5NoY10QEEiL4BvPOaJ+NeBSEba5JrblOSUPywRthFeMMvNX8PfInRUwG0UMFIg
         DCezp7vugQ9gDVVAFyvw5TA8HU8RfyWPOnnqLHbVp6kTnuYhg0JtimxfJVk5wcnvtA4D
         OBnv9xkH0/5zheHqqIbbgkGjtIRSup+YnxJns=
Received: by 10.224.67.196 with SMTP id s4mr5526195qai.198.1245709850970;
        Mon, 22 Jun 2009 15:30:50 -0700 (PDT)
Received: from ?192.168.2.107? (c172212.adsl.hansenet.de [213.39.172.212])
        by mx.google.com with ESMTPS id 5sm1041968qwg.5.2009.06.22.15.30.49
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Mon, 22 Jun 2009 15:30:49 -0700 (PDT)
Sender: Lenz Grimmer <lenz.grimmer@googlemail.com>
Message-ID: <4A400607.4050102@grimmer.com>
Date: Tue, 23 Jun 2009 00:30:31 +0200
From: Lenz Grimmer <lenz@grimmer.com>
User-Agent: Thunderbird 2.0.0.21 (X11/20090310)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Call for Participation: OpenSQL Camp at FrOSCon (St. Augustin, Germany,
 22+23 August 2009)
X-Enigmail-Version: 0.95.7
OpenPGP: id=B27291F2
Content-Type: multipart/signed; micalg=pgp-sha1;
 protocol="application/pgp-signature";
 boundary="------------enig40065C80C96752B5A0CFEAC8"
X-Virus-Checked: Checked by ClamAV on apache.org

--------------enig40065C80C96752B5A0CFEAC8
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi there,

OpenSQL Camp is a free conference of, by, and for the open-source databas=
e
community of users and developers. The OpenSQLCamp 2009, European Edition=

(http://opensqlcamp.org/) will take part in parallel to the Free and Open=

Source Conference 2009 (http://froscon.org/) on Saturday 22nd and Sunday =
23rd
August in St. Augustin, Germany, which is located close to Bonn and Colog=
ne.

The goal of this event is to spread the word about the vibrant communitie=
s and
large ecosystems that exist around Open Source Databases and to educate t=
he
attendees about possible alternatives to commercial databases. It is a pl=
ace
where people come to learn, to participate and to contribute.

We would like to invite your project to participate in this event.

We've set up a call for participation
(http://opensqlcamp.org/Events/2009/Call_for_Participation) - the deadlin=
e for
submitting your proposal is *July 19th*.

We are seeking talks related to Open Source Databases of all kind, not ju=
st
relational databases! Submission about tools and technologies related to =
OSS
databases (e.g. connectors/APIs) are also welcome.

Some ideas and for submissions:

 * An introduction/overview about a certain database project/product or
   related tool
 * Providing "best practices" information for administrators
 * A deeply technical and developer-centric session about some project's
   internals or an API used to connect to a database

We look forward to your contribution! Please don't hesitate to contact us=
 via
IRC (#opensqlcamp on FreeNode) or our Discussion Group
(http://groups.google.com/group/opensqlcamp)

Thanks!

Bye,
	LenZ
--=20
 Lenz Grimmer <lenz@grimmer.com> - http://www.lenzg.net/









--------------enig40065C80C96752B5A0CFEAC8
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org

iD8DBQFKQAYHSVDhKrJykfIRAjDGAJ4vaxF9EjfuIRSE/KThla/rKJ9FbQCdGN4x
HJ6Y8VG6WIUw/B3oAqjswNg=
=cWKI
-----END PGP SIGNATURE-----

--------------enig40065C80C96752B5A0CFEAC8--

From general-return-278-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 24 19:24:15 2009
Return-Path: <general-return-278-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47524 invoked from network); 24 Jun 2009 19:24:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jun 2009 19:24:15 -0000
Received: (qmail 2155 invoked by uid 500); 24 Jun 2009 19:24:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2078 invoked by uid 500); 24 Jun 2009 19:24:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1837 invoked by uid 99); 24 Jun 2009 19:24:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Jun 2009 19:24:24 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Jun 2009 19:24:12 +0000
Received: from SNV-EXPF01.ds.corp.yahoo.com (snv-expf01.ds.corp.yahoo.com [207.126.227.250])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n5OJMQJE032223;
	Wed, 24 Jun 2009 12:22:28 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:x-mimeole:content-class:mime-version:
	content-type:content-transfer-encoding:subject:date:message-id:
	in-reply-to:x-ms-has-attach:x-ms-tnef-correlator:thread-topic:
	thread-index:references:from:to:x-originalarrivaltime;
	b=OTXgfzY/jNjpY4QMEvIdFEjHT5mETWT4krcOyteV3nihIQJdv1Schz2B0u8fPqU8
Received: from SNV-EXVS09.ds.corp.yahoo.com ([207.126.227.87]) by SNV-EXPF01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Wed, 24 Jun 2009 12:22:26 -0700
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable
Subject: RE: [VOTE] Release Pig 0.3.0 (candidate 0)
Date: Wed, 24 Jun 2009 12:21:40 -0700
Message-ID: <236D5828CF7F5749AF1BCB7F87596DE103CBD0CE@SNV-EXVS09.ds.corp.yahoo.com>
In-Reply-To: <236D5828CF7F5749AF1BCB7F87596DE103BA3E9D@SNV-EXVS09.ds.corp.yahoo.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: [VOTE] Release Pig 0.3.0 (candidate 0)
Thread-Index: AcnwSzdeHvsmuCnUSiKzZJzKb8waJwEtL+zQ
References: <236D5828CF7F5749AF1BCB7F87596DE103BA3E9D@SNV-EXVS09.ds.corp.yahoo.com>
From: "Olga Natkovich" <olgan@yahoo-inc.com>
To: <private@hadoop.apache.org>, <pig-dev@hadoop.apache.org>,
        <general@hadoop.apache.org>
X-OriginalArrivalTime: 24 Jun 2009 19:22:26.0122 (UTC) FILETIME=[1B7562A0:01C9F501]
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,
=20
Thanks for everybody who voted!

We have four +1 binding votes from PMC members Arun Murthy, Nigel Daley,
Alan Gates, and Olga Natkovich. We have three +1 non-binding votes from
Pig Committers Pradeep Kamath, Daniel Dai, and Santhosh Srinivasan

There are no -1 votes. Also sufficient time has passed to make the
release official.
=20
Unless I hear otherwise, I am going to start publishing the release
shortly.
=20
Olga=20

> -----Original Message-----
> From: Olga Natkovich [mailto:olgan@yahoo-inc.com]=20
> Sent: Thursday, June 18, 2009 12:30 PM
> To: pig-dev@hadoop.apache.org; private@hadoop.apache.org;=20
> general@hadoop.apache.org
> Subject: [VOTE] Release Pig 0.3.0 (candidate 0)
>=20
> Hi,
> =20
> I created a candidate build for Pig 0.3.0 release. The main=20
> feature of this release is support for multiquery which=20
> allows to share computation across multiple queries within=20
> the same script. We see significant performance improvements=20
> (up to order of magnitude) as the result of this optimization.
> =20
> I ran the rat report and made sure that all the source files=20
> contain proper headers. (Not attaching the report since it=20
> caused trouble with the last release.)
> =20
> Keys used to sign the release candidate are at=20
> http://svn.apache.org/viewvc/hadoop/pig/trunk/KEYS.
> =20
> Please, download and try the release candidate:
> http://people.apache.org/~olga/pig-0.3.0-candidate-0/.
> =20
> Please, vote by Wednesday, June 24th.
> =20
> Olga
> =20
>=20

From general-return-279-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 27 04:04:18 2009
Return-Path: <general-return-279-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47884 invoked from network); 27 Jun 2009 04:04:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jun 2009 04:04:18 -0000
Received: (qmail 83664 invoked by uid 500); 27 Jun 2009 04:04:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83577 invoked by uid 500); 27 Jun 2009 04:04:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83567 invoked by uid 99); 27 Jun 2009 04:04:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 27 Jun 2009 04:04:27 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.59.211] (HELO QMTA11.westchester.pa.mail.comcast.net) (76.96.59.211)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 27 Jun 2009 04:04:16 +0000
Received: from OMTA16.westchester.pa.mail.comcast.net ([76.96.62.88])
	by QMTA11.westchester.pa.mail.comcast.net with comcast
	id 8rcG1c0051uE5Es5Bs3wRa; Sat, 27 Jun 2009 04:03:56 +0000
Received: from [11.0.0.135] ([209.131.62.115])
	by OMTA16.westchester.pa.mail.comcast.net with comcast
	id 8s4o1c0072VBGtd3cs4shy; Sat, 27 Jun 2009 04:04:57 +0000
Message-Id: <7934684D-B188-428E-B817-6B188CB977A2@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <094B4654-8D07-4251-B521-9EAFAAFBCB4E@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v935.3)
Subject: Re: [VOTE] Rename Core to Common
Date: Fri, 26 Jun 2009 21:03:44 -0700
References: <094B4654-8D07-4251-B521-9EAFAAFBCB4E@yahoo-inc.com>
X-Mailer: Apple Mail (2.935.3)
X-Virus-Checked: Checked by ClamAV on apache.org


On Jun 17, 2009, at 11:28 AM, Owen O'Malley wrote:

> I'd like to propose that we rename Core to Common. This will better  
> reflect the fact that it is no longer the "core" of Hadoop, but  
> rather just the common libraries.

With 12 +1's and no -1's, the vote passes. I've filed INFRA-2116 to  
rename the mailing lists. The subversion rename I did as part of the  
project split.

-- Owen

From general-return-280-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 30 07:15:35 2009
Return-Path: <general-return-280-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54728 invoked from network); 30 Jun 2009 07:15:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Jun 2009 07:15:35 -0000
Received: (qmail 50360 invoked by uid 500); 30 Jun 2009 07:15:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50291 invoked by uid 500); 30 Jun 2009 07:15:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50281 invoked by uid 99); 30 Jun 2009 07:15:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Jun 2009 07:15:44 +0000
X-ASF-Spam-Status: No, hits=3.7 required=10.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of xine.jar@googlemail.com designates 209.85.217.159 as permitted sender)
Received: from [209.85.217.159] (HELO mail-gx0-f159.google.com) (209.85.217.159)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Jun 2009 07:15:34 +0000
Received: by gxk3 with SMTP id 3so353808gxk.5
        for <general@hadoop.apache.org>; Tue, 30 Jun 2009 00:15:13 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=fcx1Od0IbEBsv0p21AGz9Zw5aj1PtVN1D3UvRqZw/w4=;
        b=TMD/dpU/tfGfX3wJyAuJ+JEQcilMAltPZSmaP5KfI+c9i+arX+Ki7lHvensDrMjA0z
         tw1o9lqwmtwy7k4pm2riNAN8Di5UngYSCov0drp5QnCHQKZ7/qLtJhHVjpwtSCOLXxL0
         dRm3Qqgl4TLCk1ZMuSdHjxvaNNktTng5alZUk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=Z7wRf3f7XOoYp/29e9sp3tkCTzmvIXjyJiHcEuyCErDWPfUKOrsTknKnkq4wDCM4QE
         lW0QFSkA7t6uKcVCeUC/G89WyhCK5RLoU26RPiCXcC0AxZZQP6FxhuYDgcYpxbycfVwS
         7fUsVhtS8jUHz/ELVMP7WTG57lnuSglFr3zKw=
MIME-Version: 1.0
Received: by 10.231.16.138 with SMTP id o10mr1186611iba.13.1246346111615; Tue, 
	30 Jun 2009 00:15:11 -0700 (PDT)
Date: Tue, 30 Jun 2009 09:15:11 +0200
Message-ID: <aa41f20e0906300015s42558e37s9128b0e827083b24@mail.gmail.com>
Subject: problem with running WordCode v0 with a distributed operation
From: C J <xine.jar@googlemail.com>
To: general@hadoop.apache.org, C J <xine.jar@googlemail.com>
Content-Type: multipart/alternative; boundary=000325574d2a400486046d8b9107
X-Virus-Checked: Checked by ClamAV on apache.org

--000325574d2a400486046d8b9107
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Help help help,

 I am a new user, it has been already 2-3 weeks that I am trying to run
Hadoop 0.18.3.


 *My settings:*

   1.

   I am on LINUX
   2.

   Using the Java 1.6.0
   3.

   I have unpacked the hadoop-0.18.3 folder directly on the desktop


*Working steps:*

   1.

   I have succeeded to run it on the local
   2.

   I went through the quick start tutorial and managed to operate

   Hadoop in the standalone and the Pseudo-distributed motes
   3.

   I started going through the map/reduce tutorial and managed to

   run the WordCount v1.0 with a standalone operation.

*Current status:*

   1.

   I would like to run the WordCount v1.0 example on a distributed operatio=
n
   2.

   I went through the steps in the cluster setup tutorial and did the
   following:


   -

   On a distant server I am running 4 virtual machines:

   134.130.223.58:1

   134.130.223.72:1

   134.130.223.85:1

   134.130.223.92:1
   -

   My own machine has the following ip address

   134.130.222.54
   -

   The hadoop-en.sh file is the same on all the five machines:

   *export HADOOP_HEAPSIZE=3D500*

   *export JAVA_HOME=3D"/usr/java/jdk1.6.0_14"*

   *export HADOOP_NAMENODE_OPTS=3D"-Dcom.sun.management.jmxremote
   $HADOOP_NAMENODE_OPTS"*

   *export HADOOP_SECONDARYNAMENODE_OPTS=3D"-Dcom.sun.management.jmxremote
   $HADOOP_SECONDARYNAMENODE_OPTS"*

   *export HADOOP_DATANODE_OPTS=3D"-Dcom.sun.management.jmxremote *

   *$HADOOP_DATANODE_OPTS"*

   *export HADOOP_BALANCER_OPTS=3D"-Dcom.sun.management.jmxremote *

   *$HADOOP_BALANCER_OPTS"*

   *export HADOOP_JOBTRACKER_OPTS=3D"-Dcom.sun.management.jmxremote *

   *$HADOOP_JOBTRACKER_OPTS"*

   *export HADOOP_HOME=3D"/root/Desktop/hadoop-0.18.3"*

   *export HADOOP_VERSION=3D"0.18.3"*

   *export HADOOP_LOG_DIR=3D${HADOOP_HOME}/logs*
   -

   The hadoop-site.xml is the same on all the five machines:

   *<?xml version=3D"1.0"?>*

   *<?xml-stylesheet type=3D"text/xsl" href=3D"configuration.xsl"?>*

   *<!-- Put site-specific property overrides in this file. -->*

   *<configuration>*

   * <property>*

   * Hadoop Quickstart*

   * Page 3*

   * Copyright =A9 2007 The Apache Software Foundation. All rights reserved=
.*

   * <name>fs.default.name</name>*

   * <value>134.130.223.85:9000</value>*

   * </property>*

   * <property>*

   * <name>mapred.job.tracker</name>*

   * <value>134.130.223.58:1</value>*

   * </property>*

   * <property>*

   * <name>dfs.name.dir</name>*

   * <value>/root/Desktop/hadoop-0.18.3/logstina</value>*

   * </property>*

   * <property>*

   * <name>dfs.data.dir</name>*

   * <value>/root/Desktop/hadoop-0.18.3/blockstina</value>*

   * </property>*

   * <property>*

   * <name>mapred.system.dir</name>*

   * <value>systemtina</value>*

   * </property>*

   * <property>*

   * <name>mapred.local.dir</name>*

   * <value>/root/Desktop/hadoop-0.18.3/tempMapReducetina</value>*

   * </property>*

   *</configuration> *





   -

   The slaves file is the same on all five machines and contains:

   134.130.223.72

   134.130.222.54

   134.130.223.92
   -

   from the Namenode machine 134.130.223.85 I have formatted a new namenode=
,
   started the hdfs (bin/start-dfs.sh) and started the Map-reduce
   (bin/start-mapred.sh)


 *Problem:*

Si      Since the jar file of the WordCount was already created (by the
local operation) in the folder

us     usr/tina, I tried running directly the application similarly to loca=
l
operation by typing

   *$bin/hadoop jar usr/tina/wordcount.jar org.myorg.WordCount
usr/tina/wordcount/input usr/tina/wordcount/output.*

  Then I got the following error:



 *09/06/29 19:38:05 WARN fs.FileSystem: "134.130.223.85:9000" is a
deprecated filesystem name. Use "hdfs://134.130.223.85:9000/" instead.*

*09/06/29 19:38:06 INFO ipc.Client: Retrying connect to server: /
134.130.223.85:9000. Already tried 0 time(s).*

*09/06/29 19:38:07 INFO ipc.Client: Retrying connect to server: /
134.130.223.85:9000. Already tried 1 time(s).*

*09/06/29 19:38:08 INFO ipc.Client: Retrying connect to server: /
134.130.223.85:9000. Already tried 2 time(s).*

*09/06/29 19:38:09 INFO ipc.Client: Retrying connect to server: /
134.130.223.85:9000. Already tried 3 time(s).*

*09/06/29 19:38:10 INFO ipc.Client: Retrying connect to server: /
134.130.223.85:9000. Already tried 4 time(s).*

*09/06/29 19:38:11 INFO ipc.Client: Retrying connect to server: /
134.130.223.85:9000. Already tried 5 time(s).*

*09/06/29 19:38:12 INFO ipc.Client: Retrying connect to server: /
134.130.223.85:9000. Already tried 6 time(s).*

*09/06/29 19:38:13 INFO ipc.Client: Retrying connect to server: /
134.130.223.85:9000. Already tried 7 time(s).*

*09/06/29 19:38:14 INFO ipc.Client: Retrying connect to server: /
134.130.223.85:9000. Already tried 8 time(s).*

*09/06/29 19:38:15 INFO ipc.Client: Retrying connect to server: /
134.130.223.85:9000. Already tried 9 time(s).*

*java.lang.RuntimeException: java.net.ConnectException: Call to /
134.130.223.85:9000 failed on connection exception:
java.net.ConnectException: Connection refused*

*at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:358)*

*at
org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java=
:377)
*

*at org.myorg.WordCount.main(WordCount.java:53)*

*at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)*

*at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:3=
9)
*

*at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp=
l.java:25)
*

*at java.lang.reflect.Method.invoke(Method.java:597)*

*at org.apache.hadoop.util.RunJar.main(RunJar.java:155)*

*at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)*

*at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)*

*at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)*

*at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)*

*Caused by: java.net.ConnectException: Call to /134.130.223.85:9000 failed
on connection exception: java.net.ConnectException: Connection refused*

*at org.apache.hadoop.ipc.Client.wrapException(Client.java:743)*

*at org.apache.hadoop.ipc.Client.call(Client.java:719)*

*at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)*

*at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)*

*at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:348)*

*at org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:103)*

*at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:172)*

*at
org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSyste=
m.java:67)
*

*at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1339)*

*at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)*

*at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1351)*

*at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:213)*

*at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:118)*

*at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:354)*

*... 11 more*

*Caused by: java.net.ConnectException: Connection refused*

*at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)*

*at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)*

*at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)*

*at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:301)=
*

*at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:178)*

*at org.apache.hadoop.ipc.Client.getConnection(Client.java:820)*

*at org.apache.hadoop.ipc.Client.call(Client.java:705)*

*... 23 more*




 * Questions*

   1.

   Can someone help me in solving/debugging this problem?

   P.S: I have tried to stop the HDFS with bin/stop-dfs.sh before

   starting new ones.


 Thank you,

C.J

--000325574d2a400486046d8b9107--

From general-return-281-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 30 13:24:03 2009
Return-Path: <general-return-281-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85999 invoked from network); 30 Jun 2009 13:24:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Jun 2009 13:24:03 -0000
Received: (qmail 95570 invoked by uid 500); 30 Jun 2009 13:24:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95478 invoked by uid 500); 30 Jun 2009 13:24:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95468 invoked by uid 99); 30 Jun 2009 13:24:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Jun 2009 13:24:13 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yeahyf@gmail.com designates 74.125.92.25 as permitted sender)
Received: from [74.125.92.25] (HELO qw-out-2122.google.com) (74.125.92.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Jun 2009 13:24:05 +0000
Received: by qw-out-2122.google.com with SMTP id 5so59146qwd.35
        for <general@hadoop.apache.org>; Tue, 30 Jun 2009 06:23:44 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=G1hxWOXKv/Lz3wLdTQ+PRQxo2elNwEvWMWTE8TXr+f4=;
        b=x9i6cb+SCxJm5gxol1gq4O8gAJzC6jHwWvNLzDo2KjvZEL4Hd7mk99fUznuA1N9Hw3
         VZxUHj2GtPbhU/5rD/PKhuu/u11YnGYp03NvQJlOgXfWLF52PE67TwlGXLQVd9pE4lF3
         HpeW51V1jMODXRb9mMLuAG5gn/faZvBll40t8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=QxrBJwIqaGM0iDWlNPWqAMSByX2mgnbaG0xD6HBqFWfZhRgrfQrEIeStjv/tweRQKB
         oB1Ta9QblSOrHDEqNK9hS7nTOhPAxMgu5sHykhz0/+TEk7I/xYj1TqSCm+jkUM9yT9rr
         70fY+bHnd4qaORKBAVexhfWWRLFnq8qwCVzvQ=
MIME-Version: 1.0
Received: by 10.229.89.202 with SMTP id f10mr1937545qcm.71.1246368223536; Tue, 
	30 Jun 2009 06:23:43 -0700 (PDT)
In-Reply-To: <7934684D-B188-428E-B817-6B188CB977A2@apache.org>
References: <094B4654-8D07-4251-B521-9EAFAAFBCB4E@yahoo-inc.com>
	 <7934684D-B188-428E-B817-6B188CB977A2@apache.org>
Date: Tue, 30 Jun 2009 21:23:43 +0800
Message-ID: <3b13e4710906300623g3b042ff9h7da21cb6dd2a62a4@mail.gmail.com>
Subject: Re: [VOTE] Rename Core to Common
From: =?GB2312?B?0e634Q==?= <yeahyf@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364ee0c43982b5046d90b7a3
X-Virus-Checked: Checked by ClamAV on apache.org

--0016364ee0c43982b5046d90b7a3
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

+1

2009/6/27 Owen O'Malley <omalley@apache.org>

>
> On Jun 17, 2009, at 11:28 AM, Owen O'Malley wrote:
>
>  I'd like to propose that we rename Core to Common. This will better
>> reflect the fact that it is no longer the "core" of Hadoop, but rather just
>> the common libraries.
>>
>
> With 12 +1's and no -1's, the vote passes. I've filed INFRA-2116 to rename
> the mailing lists. The subversion rename I did as part of the project split.
>
> -- Owen
>

--0016364ee0c43982b5046d90b7a3--

From general-return-282-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 30 15:06:56 2009
Return-Path: <general-return-282-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39674 invoked from network); 30 Jun 2009 15:06:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Jun 2009 15:06:56 -0000
Received: (qmail 32042 invoked by uid 500); 30 Jun 2009 15:06:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32000 invoked by uid 500); 30 Jun 2009 15:06:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31963 invoked by uid 99); 30 Jun 2009 15:06:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Jun 2009 15:06:32 +0000
X-ASF-Spam-Status: No, hits=3.7 required=10.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of vermaabhishekp@gmail.com designates 209.85.222.196 as permitted sender)
Received: from [209.85.222.196] (HELO mail-pz0-f196.google.com) (209.85.222.196)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Jun 2009 15:06:21 +0000
Received: by pzk34 with SMTP id 34so164651pzk.5
        for <general@hadoop.apache.org>; Tue, 30 Jun 2009 08:05:59 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=JCAoeRlLNLNRASMeIpv9zCGeBtF6UIRUwJpqQ0IhUAw=;
        b=ksmzCTMF64uhSLYvWLcDkn58W15wPEMzB+bEccM6oHzlH6bk3B7XMnsE9lCWBfFBxF
         Aubn8EKtcBGaHvrAqvpKeTQ/7OqcYQeuOUh2ND8y3qto4Dn1h8H4om4X0oki5p60Cpav
         H1UzXJQ5mWDmRQpS6glMcvynZlV3FSxqti5FI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Dkdg8kEG4hdStKz04QJkLBasZGEipRFAF9OHHp+W8xE97LuXipvGKOkV7PpiK14/ar
         tZTtzmn1j08rjjos2vcp1L4s9kBsEhpX1zI2tHkDnZfqlIK2yzTZw/owM6m4+IChIp/6
         SLfgED0vX01keVJDjz1sjMcspsXvkEAGAHkEI=
MIME-Version: 1.0
Received: by 10.141.26.19 with SMTP id d19mr1773091rvj.100.1246374359765; Tue, 
	30 Jun 2009 08:05:59 -0700 (PDT)
In-Reply-To: <aa41f20e0906300015s42558e37s9128b0e827083b24@mail.gmail.com>
References: <aa41f20e0906300015s42558e37s9128b0e827083b24@mail.gmail.com>
Date: Tue, 30 Jun 2009 10:05:59 -0500
Message-ID: <3c682ecd0906300805s1f6f5bd7x3eedfe4360413c10@mail.gmail.com>
Subject: Re: problem with running WordCode v0 with a distributed operation
From: Abhishek Verma <vermaabhishekp@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd14deef89225046d9224ec
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd14deef89225046d9224ec
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Did you format the namenode:

$bin/hadoop namenode -format

before starting all
$bin/start-all.sh?
http://hadoop.apache.org/core/docs/current/quickstart.html

On Tue, Jun 30, 2009 at 2:15 AM, C J <xine.jar@googlemail.com> wrote:

> Help help help,
>
>  I am a new user, it has been already 2-3 weeks that I am trying to run
> Hadoop 0.18.3.
>
>
>  *My settings:*
>
>   1.
>
>   I am on LINUX
>   2.
>
>   Using the Java 1.6.0
>   3.
>
>   I have unpacked the hadoop-0.18.3 folder directly on the desktop
>
>
> *Working steps:*
>
>   1.
>
>   I have succeeded to run it on the local
>   2.
>
>   I went through the quick start tutorial and managed to operate
>
>   Hadoop in the standalone and the Pseudo-distributed motes
>   3.
>
>   I started going through the map/reduce tutorial and managed to
>
>   run the WordCount v1.0 with a standalone operation.
>
> *Current status:*
>
>   1.
>
>   I would like to run the WordCount v1.0 example on a distributed operati=
on
>   2.
>
>   I went through the steps in the cluster setup tutorial and did the
>   following:
>
>
>   -
>
>   On a distant server I am running 4 virtual machines:
>
>   134.130.223.58:1
>
>   134.130.223.72:1
>
>   134.130.223.85:1
>
>   134.130.223.92:1
>   -
>
>   My own machine has the following ip address
>
>   134.130.222.54
>   -
>
>   The hadoop-en.sh file is the same on all the five machines:
>
>   *export HADOOP_HEAPSIZE=3D500*
>
>   *export JAVA_HOME=3D"/usr/java/jdk1.6.0_14"*
>
>   *export HADOOP_NAMENODE_OPTS=3D"-Dcom.sun.management.jmxremote
>   $HADOOP_NAMENODE_OPTS"*
>
>   *export HADOOP_SECONDARYNAMENODE_OPTS=3D"-Dcom.sun.management.jmxremote
>   $HADOOP_SECONDARYNAMENODE_OPTS"*
>
>   *export HADOOP_DATANODE_OPTS=3D"-Dcom.sun.management.jmxremote *
>
>   *$HADOOP_DATANODE_OPTS"*
>
>   *export HADOOP_BALANCER_OPTS=3D"-Dcom.sun.management.jmxremote *
>
>   *$HADOOP_BALANCER_OPTS"*
>
>   *export HADOOP_JOBTRACKER_OPTS=3D"-Dcom.sun.management.jmxremote *
>
>   *$HADOOP_JOBTRACKER_OPTS"*
>
>   *export HADOOP_HOME=3D"/root/Desktop/hadoop-0.18.3"*
>
>   *export HADOOP_VERSION=3D"0.18.3"*
>
>   *export HADOOP_LOG_DIR=3D${HADOOP_HOME}/logs*
>   -
>
>   The hadoop-site.xml is the same on all the five machines:
>
>   *<?xml version=3D"1.0"?>*
>
>   *<?xml-stylesheet type=3D"text/xsl" href=3D"configuration.xsl"?>*
>
>   *<!-- Put site-specific property overrides in this file. -->*
>
>   *<configuration>*
>
>   * <property>*
>
>   * Hadoop Quickstart*
>
>   * Page 3*
>
>   * Copyright =A9 2007 The Apache Software Foundation. All rights reserve=
d.*
>
>   * <name>fs.default.name</name>*
>
>   * <value>134.130.223.85:9000</value>*
>
>   * </property>*
>
>   * <property>*
>
>   * <name>mapred.job.tracker</name>*
>
>   * <value>134.130.223.58:1</value>*
>
>   * </property>*
>
>   * <property>*
>
>   * <name>dfs.name.dir</name>*
>
>   * <value>/root/Desktop/hadoop-0.18.3/logstina</value>*
>
>   * </property>*
>
>   * <property>*
>
>   * <name>dfs.data.dir</name>*
>
>   * <value>/root/Desktop/hadoop-0.18.3/blockstina</value>*
>
>   * </property>*
>
>   * <property>*
>
>   * <name>mapred.system.dir</name>*
>
>   * <value>systemtina</value>*
>
>   * </property>*
>
>   * <property>*
>
>   * <name>mapred.local.dir</name>*
>
>   * <value>/root/Desktop/hadoop-0.18.3/tempMapReducetina</value>*
>
>   * </property>*
>
>   *</configuration> *
>
>
>
>
>
>   -
>
>   The slaves file is the same on all five machines and contains:
>
>   134.130.223.72
>
>   134.130.222.54
>
>   134.130.223.92
>   -
>
>   from the Namenode machine 134.130.223.85 I have formatted a new namenod=
e,
>   started the hdfs (bin/start-dfs.sh) and started the Map-reduce
>   (bin/start-mapred.sh)
>
>
>  *Problem:*
>
> Si      Since the jar file of the WordCount was already created (by the
> local operation) in the folder
>
> us     usr/tina, I tried running directly the application similarly to
> local
> operation by typing
>
>   *$bin/hadoop jar usr/tina/wordcount.jar org.myorg.WordCount
> usr/tina/wordcount/input usr/tina/wordcount/output.*
>
>  Then I got the following error:
>
>
>
>  *09/06/29 19:38:05 WARN fs.FileSystem: "134.130.223.85:9000" is a
> deprecated filesystem name. Use "hdfs://134.130.223.85:9000/" instead.*
>
> *09/06/29 19:38:06 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 0 time(s).*
>
> *09/06/29 19:38:07 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 1 time(s).*
>
> *09/06/29 19:38:08 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 2 time(s).*
>
> *09/06/29 19:38:09 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 3 time(s).*
>
> *09/06/29 19:38:10 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 4 time(s).*
>
> *09/06/29 19:38:11 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 5 time(s).*
>
> *09/06/29 19:38:12 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 6 time(s).*
>
> *09/06/29 19:38:13 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 7 time(s).*
>
> *09/06/29 19:38:14 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 8 time(s).*
>
> *09/06/29 19:38:15 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 9 time(s).*
>
> *java.lang.RuntimeException: java.net.ConnectException: Call to /
> 134.130.223.85:9000 failed on connection exception:
> java.net.ConnectException: Connection refused*
>
> *at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:358=
)*
>
> *at
>
> org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.ja=
va:377)
> *
>
> *at org.myorg.WordCount.main(WordCount.java:53)*
>
> *at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)*
>
> *at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java=
:39)
> *
>
> *at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI=
mpl.java:25)
> *
>
> *at java.lang.reflect.Method.invoke(Method.java:597)*
>
> *at org.apache.hadoop.util.RunJar.main(RunJar.java:155)*
>
> *at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)*
>
> *at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)*
>
> *at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)*
>
> *at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)*
>
> *Caused by: java.net.ConnectException: Call to /134.130.223.85:9000 faile=
d
> on connection exception: java.net.ConnectException: Connection refused*
>
> *at org.apache.hadoop.ipc.Client.wrapException(Client.java:743)*
>
> *at org.apache.hadoop.ipc.Client.call(Client.java:719)*
>
> *at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)*
>
> *at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)*
>
> *at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:348)*
>
> *at org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:103)=
*
>
> *at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:172)*
>
> *at
>
> org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSys=
tem.java:67)
> *
>
> *at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1339=
)*
>
> *at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)*
>
> *at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1351)*
>
> *at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:213)*
>
> *at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:118)*
>
> *at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:354=
)*
>
> *... 11 more*
>
> *Caused by: java.net.ConnectException: Connection refused*
>
> *at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)*
>
> *at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574=
)*
>
> *at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)*
>
> *at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:301)*
>
> *at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:178)*
>
> *at org.apache.hadoop.ipc.Client.getConnection(Client.java:820)*
>
> *at org.apache.hadoop.ipc.Client.call(Client.java:705)*
>
> *... 23 more*
>
>
>
>
>  * Questions*
>
>   1.
>
>   Can someone help me in solving/debugging this problem?
>
>   P.S: I have tried to stop the HDFS with bin/stop-dfs.sh before
>
>   starting new ones.
>
>
>  Thank you,
>
> C.J
>

--000e0cd14deef89225046d9224ec--

From general-return-283-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 30 15:30:53 2009
Return-Path: <general-return-283-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49997 invoked from network); 30 Jun 2009 15:30:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Jun 2009 15:30:52 -0000
Received: (qmail 4469 invoked by uid 500); 30 Jun 2009 15:31:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4433 invoked by uid 500); 30 Jun 2009 15:31:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4423 invoked by uid 99); 30 Jun 2009 15:31:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Jun 2009 15:31:03 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [12.110.209.161] (HELO usausmgw01.spansion.com) (12.110.209.161)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Jun 2009 15:30:53 +0000
X-IronPort-AV: E=McAfee;i="5300,2777,5661"; a="5251915"
Received: from usausexbh2.spansion.com ([10.248.26.116])
  by usausmgw01.spansion.com with ESMTP; 30 Jun 2009 08:30:31 -0700
Received: from USAUSEXMBPF2.spansion.com ([10.248.26.56]) by usausexbh2.spansion.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Tue, 30 Jun 2009 10:30:31 -0500
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable
X-MimeOLE: Produced By Microsoft Exchange V6.5
Subject: RE: problem with running WordCode v0 with a distributed operation
Date: Tue, 30 Jun 2009 10:30:31 -0500
Message-ID: <AF176AD83CF4724EAAD1AE6F1BAE02043DB2CD@USAUSEXMBPF2.spansion.com>
In-Reply-To: <aa41f20e0906300015s42558e37s9128b0e827083b24@mail.gmail.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: problem with running WordCode v0 with a distributed operation
thread-index: Acn5UpbgRxUobXuTT6qUIW9qyw4lIwAQ6XFw
References: <aa41f20e0906300015s42558e37s9128b0e827083b24@mail.gmail.com>
From: "Gross, Danny" <Danny.Gross@spansion.com>
To: <general@hadoop.apache.org>
X-OriginalArrivalTime: 30 Jun 2009 15:30:31.0339 (UTC) FILETIME=[B4147FB0:01C9F997]
X-Virus-Checked: Checked by ClamAV on apache.org

Hello,

I suggest that you check iptables on your systems.  At one time, one of
my nodes showed a similar error, and this was the culprit.

Good luck,

Danny

-----Original Message-----
From: C J [mailto:xine.jar@googlemail.com]=20
Sent: Tuesday, June 30, 2009 2:15 AM
To: general@hadoop.apache.org; C J
Subject: problem with running WordCode v0 with a distributed operation

Help help help,

 I am a new user, it has been already 2-3 weeks that I am trying to run
Hadoop 0.18.3.


 *My settings:*

   1.

   I am on LINUX
   2.

   Using the Java 1.6.0
   3.

   I have unpacked the hadoop-0.18.3 folder directly on the desktop


*Working steps:*

   1.

   I have succeeded to run it on the local
   2.

   I went through the quick start tutorial and managed to operate

   Hadoop in the standalone and the Pseudo-distributed motes
   3.

   I started going through the map/reduce tutorial and managed to

   run the WordCount v1.0 with a standalone operation.

*Current status:*

   1.

   I would like to run the WordCount v1.0 example on a distributed
operation
   2.

   I went through the steps in the cluster setup tutorial and did the
   following:


   -

   On a distant server I am running 4 virtual machines:

   134.130.223.58:1

   134.130.223.72:1

   134.130.223.85:1

   134.130.223.92:1
   -

   My own machine has the following ip address

   134.130.222.54
   -

   The hadoop-en.sh file is the same on all the five machines:

   *export HADOOP_HEAPSIZE=3D500*

   *export JAVA_HOME=3D"/usr/java/jdk1.6.0_14"*

   *export HADOOP_NAMENODE_OPTS=3D"-Dcom.sun.management.jmxremote
   $HADOOP_NAMENODE_OPTS"*

   *export =
HADOOP_SECONDARYNAMENODE_OPTS=3D"-Dcom.sun.management.jmxremote
   $HADOOP_SECONDARYNAMENODE_OPTS"*

   *export HADOOP_DATANODE_OPTS=3D"-Dcom.sun.management.jmxremote *

   *$HADOOP_DATANODE_OPTS"*

   *export HADOOP_BALANCER_OPTS=3D"-Dcom.sun.management.jmxremote *

   *$HADOOP_BALANCER_OPTS"*

   *export HADOOP_JOBTRACKER_OPTS=3D"-Dcom.sun.management.jmxremote *

   *$HADOOP_JOBTRACKER_OPTS"*

   *export HADOOP_HOME=3D"/root/Desktop/hadoop-0.18.3"*

   *export HADOOP_VERSION=3D"0.18.3"*

   *export HADOOP_LOG_DIR=3D${HADOOP_HOME}/logs*
   -

   The hadoop-site.xml is the same on all the five machines:

   *<?xml version=3D"1.0"?>*

   *<?xml-stylesheet type=3D"text/xsl" href=3D"configuration.xsl"?>*

   *<!-- Put site-specific property overrides in this file. -->*

   *<configuration>*

   * <property>*

   * Hadoop Quickstart*

   * Page 3*

   * Copyright (c) 2007 The Apache Software Foundation. All rights
reserved.*

   * <name>fs.default.name</name>*

   * <value>134.130.223.85:9000</value>*

   * </property>*

   * <property>*

   * <name>mapred.job.tracker</name>*

   * <value>134.130.223.58:1</value>*

   * </property>*

   * <property>*

   * <name>dfs.name.dir</name>*

   * <value>/root/Desktop/hadoop-0.18.3/logstina</value>*

   * </property>*

   * <property>*

   * <name>dfs.data.dir</name>*

   * <value>/root/Desktop/hadoop-0.18.3/blockstina</value>*

   * </property>*

   * <property>*

   * <name>mapred.system.dir</name>*

   * <value>systemtina</value>*

   * </property>*

   * <property>*

   * <name>mapred.local.dir</name>*

   * <value>/root/Desktop/hadoop-0.18.3/tempMapReducetina</value>*

   * </property>*

   *</configuration> *





   -

   The slaves file is the same on all five machines and contains:

   134.130.223.72

   134.130.222.54

   134.130.223.92
   -

   from the Namenode machine 134.130.223.85 I have formatted a new
namenode,
   started the hdfs (bin/start-dfs.sh) and started the Map-reduce
   (bin/start-mapred.sh)


 *Problem:*

Si      Since the jar file of the WordCount was already created (by the
local operation) in the folder

us     usr/tina, I tried running directly the application similarly to
local
operation by typing

   *$bin/hadoop jar usr/tina/wordcount.jar org.myorg.WordCount
usr/tina/wordcount/input usr/tina/wordcount/output.*

  Then I got the following error:



 *09/06/29 19:38:05 WARN fs.FileSystem: "134.130.223.85:9000" is a
deprecated filesystem name. Use "hdfs://134.130.223.85:9000/" instead.*

*09/06/29 19:38:06 INFO ipc.Client: Retrying connect to server: /
134.130.223.85:9000. Already tried 0 time(s).*

*09/06/29 19:38:07 INFO ipc.Client: Retrying connect to server: /
134.130.223.85:9000. Already tried 1 time(s).*

*09/06/29 19:38:08 INFO ipc.Client: Retrying connect to server: /
134.130.223.85:9000. Already tried 2 time(s).*

*09/06/29 19:38:09 INFO ipc.Client: Retrying connect to server: /
134.130.223.85:9000. Already tried 3 time(s).*

*09/06/29 19:38:10 INFO ipc.Client: Retrying connect to server: /
134.130.223.85:9000. Already tried 4 time(s).*

*09/06/29 19:38:11 INFO ipc.Client: Retrying connect to server: /
134.130.223.85:9000. Already tried 5 time(s).*

*09/06/29 19:38:12 INFO ipc.Client: Retrying connect to server: /
134.130.223.85:9000. Already tried 6 time(s).*

*09/06/29 19:38:13 INFO ipc.Client: Retrying connect to server: /
134.130.223.85:9000. Already tried 7 time(s).*

*09/06/29 19:38:14 INFO ipc.Client: Retrying connect to server: /
134.130.223.85:9000. Already tried 8 time(s).*

*09/06/29 19:38:15 INFO ipc.Client: Retrying connect to server: /
134.130.223.85:9000. Already tried 9 time(s).*

*java.lang.RuntimeException: java.net.ConnectException: Call to /
134.130.223.85:9000 failed on connection exception:
java.net.ConnectException: Connection refused*

*at
org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:358)*

*at
org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.j
ava:377)
*

*at org.myorg.WordCount.main(WordCount.java:53)*

*at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)*

*at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
*

*at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
*

*at java.lang.reflect.Method.invoke(Method.java:597)*

*at org.apache.hadoop.util.RunJar.main(RunJar.java:155)*

*at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)*

*at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)*

*at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)*

*at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)*

*Caused by: java.net.ConnectException: Call to /134.130.223.85:9000
failed
on connection exception: java.net.ConnectException: Connection refused*

*at org.apache.hadoop.ipc.Client.wrapException(Client.java:743)*

*at org.apache.hadoop.ipc.Client.call(Client.java:719)*

*at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)*

*at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)*

*at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:348)*

*at
org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:103)*

*at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:172)*

*at
org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSy
stem.java:67)
*

*at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1339)*

*at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)*

*at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1351)*

*at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:213)*

*at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:118)*

*at
org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:354)*

*... 11 more*

*Caused by: java.net.ConnectException: Connection refused*

*at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)*

*at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)*

*at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)*

*at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:301)*

*at
org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:178)*

*at org.apache.hadoop.ipc.Client.getConnection(Client.java:820)*

*at org.apache.hadoop.ipc.Client.call(Client.java:705)*

*... 23 more*




 * Questions*

   1.

   Can someone help me in solving/debugging this problem?

   P.S: I have tried to stop the HDFS with bin/stop-dfs.sh before

   starting new ones.


 Thank you,

C.J

From general-return-284-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 30 17:19:29 2009
Return-Path: <general-return-284-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 453 invoked from network); 30 Jun 2009 17:19:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Jun 2009 17:19:29 -0000
Received: (qmail 28477 invoked by uid 500); 30 Jun 2009 17:19:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28397 invoked by uid 500); 30 Jun 2009 17:19:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28387 invoked by uid 99); 30 Jun 2009 17:19:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Jun 2009 17:19:39 +0000
X-ASF-Spam-Status: No, hits=3.7 required=10.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of xine.jar@googlemail.com designates 209.85.221.189 as permitted sender)
Received: from [209.85.221.189] (HELO mail-qy0-f189.google.com) (209.85.221.189)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Jun 2009 17:19:27 +0000
Received: by qyk27 with SMTP id 27so316416qyk.5
        for <general@hadoop.apache.org>; Tue, 30 Jun 2009 10:19:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=Q/0u6BWFS8gdOfNrGt80OlMf6DyUtb2aT5GmEDIVorU=;
        b=Lu3tKilu7zL7MHPkq7asxoya+nGCi0eZctWapFuRHxv/NcWxmyj/4b8IAy6D5CR4eE
         hm0eFFD1eoKf3lzlJ2gzgwEPmnPRKFlx8x9uUN+gB03M5KRBuBwaKrG2WGm0mKMyk60n
         zjCLL7DRN9u778e3bxy7IT5CoxzKYnXQxHP/A=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Z8B7y9ypLVrMrNVk1gVg4Njz3tuWpItUFz1I644UTh5kQPTRU4JBxL8ZEg/K16Vwgy
         j/WuQzO0N0LYfIDZDX5/+RnXi0+HHdNTe2C0IaxNx/V84dGuPUvWKEoG4z8ITSD70rlo
         6DNIHdda+zt2KOQohF/MgU7oXi7clilM9MNJo=
MIME-Version: 1.0
Received: by 10.231.32.11 with SMTP id a11mr1459214ibd.38.1246382345016; Tue, 
	30 Jun 2009 10:19:05 -0700 (PDT)
In-Reply-To: <3c682ecd0906300805s1f6f5bd7x3eedfe4360413c10@mail.gmail.com>
References: <aa41f20e0906300015s42558e37s9128b0e827083b24@mail.gmail.com>
	 <3c682ecd0906300805s1f6f5bd7x3eedfe4360413c10@mail.gmail.com>
Date: Tue, 30 Jun 2009 19:19:04 +0200
Message-ID: <aa41f20e0906301019i12ce2ebai6906eb0516fe075c@mail.gmail.com>
Subject: Re: problem with running WordCode v0 with a distributed operation
From: C J <xine.jar@googlemail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0022152d6d75ede519046d940013
X-Virus-Checked: Checked by ClamAV on apache.org

--0022152d6d75ede519046d940013
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Yes of course I did,
but the error is the same. What else shall I try?

On Tue, Jun 30, 2009 at 5:05 PM, Abhishek Verma <vermaabhishekp@gmail.com>w=
rote:

> Did you format the namenode:
>
> $bin/hadoop namenode -format
>
> before starting all
> $bin/start-all.sh?
> http://hadoop.apache.org/core/docs/current/quickstart.html
>
> On Tue, Jun 30, 2009 at 2:15 AM, C J <xine.jar@googlemail.com> wrote:
>
> > Help help help,
> >
> >  I am a new user, it has been already 2-3 weeks that I am trying to run
> > Hadoop 0.18.3.
> >
> >
> >  *My settings:*
> >
> >   1.
> >
> >   I am on LINUX
> >   2.
> >
> >   Using the Java 1.6.0
> >   3.
> >
> >   I have unpacked the hadoop-0.18.3 folder directly on the desktop
> >
> >
> > *Working steps:*
> >
> >   1.
> >
> >   I have succeeded to run it on the local
> >   2.
> >
> >   I went through the quick start tutorial and managed to operate
> >
> >   Hadoop in the standalone and the Pseudo-distributed motes
> >   3.
> >
> >   I started going through the map/reduce tutorial and managed to
> >
> >   run the WordCount v1.0 with a standalone operation.
> >
> > *Current status:*
> >
> >   1.
> >
> >   I would like to run the WordCount v1.0 example on a distributed
> operation
> >   2.
> >
> >   I went through the steps in the cluster setup tutorial and did the
> >   following:
> >
> >
> >   -
> >
> >   On a distant server I am running 4 virtual machines:
> >
> >   134.130.223.58:1
> >
> >   134.130.223.72:1
> >
> >   134.130.223.85:1
> >
> >   134.130.223.92:1
> >   -
> >
> >   My own machine has the following ip address
> >
> >   134.130.222.54
> >   -
> >
> >   The hadoop-en.sh file is the same on all the five machines:
> >
> >   *export HADOOP_HEAPSIZE=3D500*
> >
> >   *export JAVA_HOME=3D"/usr/java/jdk1.6.0_14"*
> >
> >   *export HADOOP_NAMENODE_OPTS=3D"-Dcom.sun.management.jmxremote
> >   $HADOOP_NAMENODE_OPTS"*
> >
> >   *export HADOOP_SECONDARYNAMENODE_OPTS=3D"-Dcom.sun.management.jmxremo=
te
> >   $HADOOP_SECONDARYNAMENODE_OPTS"*
> >
> >   *export HADOOP_DATANODE_OPTS=3D"-Dcom.sun.management.jmxremote *
> >
> >   *$HADOOP_DATANODE_OPTS"*
> >
> >   *export HADOOP_BALANCER_OPTS=3D"-Dcom.sun.management.jmxremote *
> >
> >   *$HADOOP_BALANCER_OPTS"*
> >
> >   *export HADOOP_JOBTRACKER_OPTS=3D"-Dcom.sun.management.jmxremote *
> >
> >   *$HADOOP_JOBTRACKER_OPTS"*
> >
> >   *export HADOOP_HOME=3D"/root/Desktop/hadoop-0.18.3"*
> >
> >   *export HADOOP_VERSION=3D"0.18.3"*
> >
> >   *export HADOOP_LOG_DIR=3D${HADOOP_HOME}/logs*
> >   -
> >
> >   The hadoop-site.xml is the same on all the five machines:
> >
> >   *<?xml version=3D"1.0"?>*
> >
> >   *<?xml-stylesheet type=3D"text/xsl" href=3D"configuration.xsl"?>*
> >
> >   *<!-- Put site-specific property overrides in this file. -->*
> >
> >   *<configuration>*
> >
> >   * <property>*
> >
> >   * Hadoop Quickstart*
> >
> >   * Page 3*
> >
> >   * Copyright =A9 2007 The Apache Software Foundation. All rights
> reserved.*
> >
> >   * <name>fs.default.name</name>*
> >
> >   * <value>134.130.223.85:9000</value>*
> >
> >   * </property>*
> >
> >   * <property>*
> >
> >   * <name>mapred.job.tracker</name>*
> >
> >   * <value>134.130.223.58:1</value>*
> >
> >   * </property>*
> >
> >   * <property>*
> >
> >   * <name>dfs.name.dir</name>*
> >
> >   * <value>/root/Desktop/hadoop-0.18.3/logstina</value>*
> >
> >   * </property>*
> >
> >   * <property>*
> >
> >   * <name>dfs.data.dir</name>*
> >
> >   * <value>/root/Desktop/hadoop-0.18.3/blockstina</value>*
> >
> >   * </property>*
> >
> >   * <property>*
> >
> >   * <name>mapred.system.dir</name>*
> >
> >   * <value>systemtina</value>*
> >
> >   * </property>*
> >
> >   * <property>*
> >
> >   * <name>mapred.local.dir</name>*
> >
> >   * <value>/root/Desktop/hadoop-0.18.3/tempMapReducetina</value>*
> >
> >   * </property>*
> >
> >   *</configuration> *
> >
> >
> >
> >
> >
> >   -
> >
> >   The slaves file is the same on all five machines and contains:
> >
> >   134.130.223.72
> >
> >   134.130.222.54
> >
> >   134.130.223.92
> >   -
> >
> >   from the Namenode machine 134.130.223.85 I have formatted a new
> namenode,
> >   started the hdfs (bin/start-dfs.sh) and started the Map-reduce
> >   (bin/start-mapred.sh)
> >
> >
> >  *Problem:*
> >
> > Si      Since the jar file of the WordCount was already created (by the
> > local operation) in the folder
> >
> > us     usr/tina, I tried running directly the application similarly to
> > local
> > operation by typing
> >
> >   *$bin/hadoop jar usr/tina/wordcount.jar org.myorg.WordCount
> > usr/tina/wordcount/input usr/tina/wordcount/output.*
> >
> >  Then I got the following error:
> >
> >
> >
> >  *09/06/29 19:38:05 WARN fs.FileSystem: "134.130.223.85:9000" is a
> > deprecated filesystem name. Use "hdfs://134.130.223.85:9000/" instead.*
> >
> > *09/06/29 19:38:06 INFO ipc.Client: Retrying connect to server: /
> > 134.130.223.85:9000. Already tried 0 time(s).*
> >
> > *09/06/29 19:38:07 INFO ipc.Client: Retrying connect to server: /
> > 134.130.223.85:9000. Already tried 1 time(s).*
> >
> > *09/06/29 19:38:08 INFO ipc.Client: Retrying connect to server: /
> > 134.130.223.85:9000. Already tried 2 time(s).*
> >
> > *09/06/29 19:38:09 INFO ipc.Client: Retrying connect to server: /
> > 134.130.223.85:9000. Already tried 3 time(s).*
> >
> > *09/06/29 19:38:10 INFO ipc.Client: Retrying connect to server: /
> > 134.130.223.85:9000. Already tried 4 time(s).*
> >
> > *09/06/29 19:38:11 INFO ipc.Client: Retrying connect to server: /
> > 134.130.223.85:9000. Already tried 5 time(s).*
> >
> > *09/06/29 19:38:12 INFO ipc.Client: Retrying connect to server: /
> > 134.130.223.85:9000. Already tried 6 time(s).*
> >
> > *09/06/29 19:38:13 INFO ipc.Client: Retrying connect to server: /
> > 134.130.223.85:9000. Already tried 7 time(s).*
> >
> > *09/06/29 19:38:14 INFO ipc.Client: Retrying connect to server: /
> > 134.130.223.85:9000. Already tried 8 time(s).*
> >
> > *09/06/29 19:38:15 INFO ipc.Client: Retrying connect to server: /
> > 134.130.223.85:9000. Already tried 9 time(s).*
> >
> > *java.lang.RuntimeException: java.net.ConnectException: Call to /
> > 134.130.223.85:9000 failed on connection exception:
> > java.net.ConnectException: Connection refused*
> >
> > *at
> org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:358)*
> >
> > *at
> >
> >
> org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.ja=
va:377)
> > *
> >
> > *at org.myorg.WordCount.main(WordCount.java:53)*
> >
> > *at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)*
> >
> > *at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java=
:39)
> > *
> >
> > *at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI=
mpl.java:25)
> > *
> >
> > *at java.lang.reflect.Method.invoke(Method.java:597)*
> >
> > *at org.apache.hadoop.util.RunJar.main(RunJar.java:155)*
> >
> > *at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)*
> >
> > *at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)*
> >
> > *at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)*
> >
> > *at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)*
> >
> > *Caused by: java.net.ConnectException: Call to /134.130.223.85:9000fail=
ed
> > on connection exception: java.net.ConnectException: Connection refused*
> >
> > *at org.apache.hadoop.ipc.Client.wrapException(Client.java:743)*
> >
> > *at org.apache.hadoop.ipc.Client.call(Client.java:719)*
> >
> > *at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)*
> >
> > *at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)*
> >
> > *at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:348)*
> >
> > *at
> org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:103)*
> >
> > *at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:172)*
> >
> > *at
> >
> >
> org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSys=
tem.java:67)
> > *
> >
> > *at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1339)*
> >
> > *at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)*
> >
> > *at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1351)*
> >
> > *at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:213)*
> >
> > *at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:118)*
> >
> > *at
> org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:354)*
> >
> > *... 11 more*
> >
> > *Caused by: java.net.ConnectException: Connection refused*
> >
> > *at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)*
> >
> > *at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)*
> >
> > *at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)*
> >
> > *at
> > org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:301)=
*
> >
> > *at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:178=
)*
> >
> > *at org.apache.hadoop.ipc.Client.getConnection(Client.java:820)*
> >
> > *at org.apache.hadoop.ipc.Client.call(Client.java:705)*
> >
> > *... 23 more*
> >
> >
> >
> >
> >  * Questions*
> >
> >   1.
> >
> >   Can someone help me in solving/debugging this problem?
> >
> >   P.S: I have tried to stop the HDFS with bin/stop-dfs.sh before
> >
> >   starting new ones.
> >
> >
> >  Thank you,
> >
> > C.J
> >
>

--0022152d6d75ede519046d940013--

From general-return-285-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 30 17:25:16 2009
Return-Path: <general-return-285-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5138 invoked from network); 30 Jun 2009 17:25:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Jun 2009 17:25:16 -0000
Received: (qmail 36681 invoked by uid 500); 30 Jun 2009 17:25:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36636 invoked by uid 500); 30 Jun 2009 17:25:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36626 invoked by uid 99); 30 Jun 2009 17:25:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Jun 2009 17:25:26 +0000
X-ASF-Spam-Status: No, hits=3.7 required=10.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of xine.jar@googlemail.com designates 209.85.217.159 as permitted sender)
Received: from [209.85.217.159] (HELO mail-gx0-f159.google.com) (209.85.217.159)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Jun 2009 17:25:14 +0000
Received: by gxk3 with SMTP id 3so18114gxk.5
        for <general@hadoop.apache.org>; Tue, 30 Jun 2009 10:24:53 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=12H9RtgBdKBMYt3oz8clWZ2/t24ibJaUIM3BfcpCpGI=;
        b=o6Uuj2M0HCXMSx4TltjOHFxcuIaPJvr22aenTn068pigSXq78WERH6cfsrS4fSN2lJ
         +Q3xGtR5Xwe33nrTXabA4eSSxR9a0+nM8itxUHEcPFE+aOdJ/uY9oXN8v1cdq046kqNA
         nffNmRx2CljtanPODNIDL3ufkeE+l/i6Sflng=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=FidCaLqRWiQ+yZNjWmgL64PMWN2Sotju2EFSwyQZMdDiUHyfBjsZ+S/SPwwTGOaHkC
         lAuCBSnNSDnQot6BDJWmWJV1kxLF7FuVv4RPh2xRk7OHUBcrdFZXaZ8qrLHztpm2hBp8
         fhw+WVUOViZsC2+yPRACP1ns4l5Qg34CdRzKc=
MIME-Version: 1.0
Received: by 10.231.32.141 with SMTP id c13mr1319471ibd.25.1246382691033; Tue, 
	30 Jun 2009 10:24:51 -0700 (PDT)
In-Reply-To: <AF176AD83CF4724EAAD1AE6F1BAE02043DB2CD@USAUSEXMBPF2.spansion.com>
References: <aa41f20e0906300015s42558e37s9128b0e827083b24@mail.gmail.com>
	 <AF176AD83CF4724EAAD1AE6F1BAE02043DB2CD@USAUSEXMBPF2.spansion.com>
Date: Tue, 30 Jun 2009 19:24:49 +0200
Message-ID: <aa41f20e0906301024o736adaa9s5daa9a1ce014a856@mail.gmail.com>
Subject: Re: problem with running WordCode v0 with a distributed operation
From: C J <xine.jar@googlemail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0022152d5cf98ded6d046d94150c
X-Virus-Checked: Checked by ClamAV on apache.org

--0022152d5cf98ded6d046d94150c
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi Danny,
can you be a bit more precise what was wrong in your iptable?
thank you

On Tue, Jun 30, 2009 at 5:30 PM, Gross, Danny <Danny.Gross@spansion.com>wrote:

> Hello,
>
> I suggest that you check iptables on your systems.  At one time, one of
> my nodes showed a similar error, and this was the culprit.
>
> Good luck,
>
> Danny
>
> -----Original Message-----
> From: C J [mailto:xine.jar@googlemail.com]
> Sent: Tuesday, June 30, 2009 2:15 AM
> To: general@hadoop.apache.org; C J
> Subject: problem with running WordCode v0 with a distributed operation
>
> Help help help,
>
>  I am a new user, it has been already 2-3 weeks that I am trying to run
> Hadoop 0.18.3.
>
>
>  *My settings:*
>
>   1.
>
>   I am on LINUX
>   2.
>
>    Using the Java 1.6.0
>    3.
>
>   I have unpacked the hadoop-0.18.3 folder directly on the desktop
>
>
> *Working steps:*
>
>   1.
>
>   I have succeeded to run it on the local
>    2.
>
>   I went through the quick start tutorial and managed to operate
>
>   Hadoop in the standalone and the Pseudo-distributed motes
>    3.
>
>   I started going through the map/reduce tutorial and managed to
>
>   run the WordCount v1.0 with a standalone operation.
>
> *Current status:*
>
>   1.
>
>   I would like to run the WordCount v1.0 example on a distributed
> operation
>    2.
>
>   I went through the steps in the cluster setup tutorial and did the
>   following:
>
>
>   -
>
>   On a distant server I am running 4 virtual machines:
>
>   134.130.223.58:1
>
>   134.130.223.72:1
>
>   134.130.223.85:1
>
>   134.130.223.92:1
>   -
>
>   My own machine has the following ip address
>
>   134.130.222.54
>   -
>
>   The hadoop-en.sh file is the same on all the five machines:
>
>   *export HADOOP_HEAPSIZE=500*
>
>   *export JAVA_HOME="/usr/java/jdk1.6.0_14"*
>
>   *export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote
>   $HADOOP_NAMENODE_OPTS"*
>
>   *export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote
>   $HADOOP_SECONDARYNAMENODE_OPTS"*
>
>   *export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote *
>
>   *$HADOOP_DATANODE_OPTS"*
>
>   *export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote *
>
>   *$HADOOP_BALANCER_OPTS"*
>
>   *export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote *
>
>   *$HADOOP_JOBTRACKER_OPTS"*
>
>   *export HADOOP_HOME="/root/Desktop/hadoop-0.18.3"*
>
>   *export HADOOP_VERSION="0.18.3"*
>
>   *export HADOOP_LOG_DIR=${HADOOP_HOME}/logs*
>   -
>
>   The hadoop-site.xml is the same on all the five machines:
>
>   *<?xml version="1.0"?>*
>
>   *<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>*
>
>   *<!-- Put site-specific property overrides in this file. -->*
>
>   *<configuration>*
>
>   * <property>*
>
>   * Hadoop Quickstart*
>
>   * Page 3*
>
>    * Copyright (c) 2007 The Apache Software Foundation. All rights
> reserved.*
>
>   * <name>fs.default.name</name>*
>
>   * <value>134.130.223.85:9000</value>*
>
>   * </property>*
>
>   * <property>*
>
>   * <name>mapred.job.tracker</name>*
>
>   * <value>134.130.223.58:1</value>*
>
>   * </property>*
>
>   * <property>*
>
>   * <name>dfs.name.dir</name>*
>
>   * <value>/root/Desktop/hadoop-0.18.3/logstina</value>*
>
>   * </property>*
>
>   * <property>*
>
>   * <name>dfs.data.dir</name>*
>
>   * <value>/root/Desktop/hadoop-0.18.3/blockstina</value>*
>
>   * </property>*
>
>   * <property>*
>
>   * <name>mapred.system.dir</name>*
>
>   * <value>systemtina</value>*
>
>   * </property>*
>
>   * <property>*
>
>   * <name>mapred.local.dir</name>*
>
>   * <value>/root/Desktop/hadoop-0.18.3/tempMapReducetina</value>*
>
>   * </property>*
>
>   *</configuration> *
>
>
>
>
>
>   -
>
>   The slaves file is the same on all five machines and contains:
>
>   134.130.223.72
>
>   134.130.222.54
>
>   134.130.223.92
>   -
>
>   from the Namenode machine 134.130.223.85 I have formatted a new
> namenode,
>   started the hdfs (bin/start-dfs.sh) and started the Map-reduce
>   (bin/start-mapred.sh)
>
>
>  *Problem:*
>
> Si      Since the jar file of the WordCount was already created (by the
> local operation) in the folder
>
> us     usr/tina, I tried running directly the application similarly to
> local
> operation by typing
>
>   *$bin/hadoop jar usr/tina/wordcount.jar org.myorg.WordCount
> usr/tina/wordcount/input usr/tina/wordcount/output.*
>
>  Then I got the following error:
>
>
>
>  *09/06/29 19:38:05 WARN fs.FileSystem: "134.130.223.85:9000" is a
> deprecated filesystem name. Use "hdfs://134.130.223.85:9000/" instead.*
>
> *09/06/29 19:38:06 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 0 time(s).*
>
> *09/06/29 19:38:07 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 1 time(s).*
>
> *09/06/29 19:38:08 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 2 time(s).*
>
> *09/06/29 19:38:09 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 3 time(s).*
>
> *09/06/29 19:38:10 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 4 time(s).*
>
> *09/06/29 19:38:11 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 5 time(s).*
>
> *09/06/29 19:38:12 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 6 time(s).*
>
> *09/06/29 19:38:13 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 7 time(s).*
>
> *09/06/29 19:38:14 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 8 time(s).*
>
> *09/06/29 19:38:15 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 9 time(s).*
>
> *java.lang.RuntimeException: java.net.ConnectException: Call to /
> 134.130.223.85:9000 failed on connection exception:
> java.net.ConnectException: Connection refused*
>
> *at
> org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:358)*
>
> *at
> org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.j
> ava:377)
> *
>
> *at org.myorg.WordCount.main(WordCount.java:53)*
>
> *at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)*
>
> *at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
> a:39)
> *
>
> *at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:25)
> *
>
> *at java.lang.reflect.Method.invoke(Method.java:597)*
>
> *at org.apache.hadoop.util.RunJar.main(RunJar.java:155)*
>
> *at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)*
>
> *at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)*
>
> *at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)*
>
> *at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)*
>
> *Caused by: java.net.ConnectException: Call to /134.130.223.85:9000
> failed
> on connection exception: java.net.ConnectException: Connection refused*
>
> *at org.apache.hadoop.ipc.Client.wrapException(Client.java:743)*
>
> *at org.apache.hadoop.ipc.Client.call(Client.java:719)*
>
> *at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)*
>
> *at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)*
>
> *at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:348)*
>
> *at
> org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:103)*
>
> *at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:172)*
>
> *at
> org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSy
> stem.java:67)
> *
>
> *at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1339)*
>
> *at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)*
>
> *at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1351)*
>
> *at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:213)*
>
> *at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:118)*
>
> *at
> org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:354)*
>
> *... 11 more*
>
> *Caused by: java.net.ConnectException: Connection refused*
>
> *at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)*
>
> *at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)*
>
> *at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)*
>
> *at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:301)*
>
> *at
> org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:178)*
>
> *at org.apache.hadoop.ipc.Client.getConnection(Client.java:820)*
>
> *at org.apache.hadoop.ipc.Client.call(Client.java:705)*
>
> *... 23 more*
>
>
>
>
>  * Questions*
>
>   1.
>
>   Can someone help me in solving/debugging this problem?
>
>   P.S: I have tried to stop the HDFS with bin/stop-dfs.sh before
>
>   starting new ones.
>
>
>  Thank you,
>
> C.J
>

--0022152d5cf98ded6d046d94150c--

From general-return-286-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 30 17:55:33 2009
Return-Path: <general-return-286-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24451 invoked from network); 30 Jun 2009 17:55:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Jun 2009 17:55:33 -0000
Received: (qmail 76129 invoked by uid 500); 30 Jun 2009 17:55:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76043 invoked by uid 500); 30 Jun 2009 17:55:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76031 invoked by uid 99); 30 Jun 2009 17:55:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Jun 2009 17:55:42 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Jun 2009 17:55:32 +0000
Received: from battlerock-lm.corp.yahoo.com (battlerock-lm.corp.yahoo.com [10.72.112.37])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n5UHrTvh045000
	for <general@hadoop.apache.org>; Tue, 30 Jun 2009 10:53:30 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:content-type:
	content-transfer-encoding:mime-version:subject:date:x-mailer;
	b=iR79m6lW69oPC4UBwdp3k4q3wQ2syV3yqFPQnXALkscG9tqOWXytEw4IqNA1KAOQ
Message-Id: <C1F6B4F2-B91D-49E5-81E1-4F7FBF83BBA6@yahoo-inc.com>
From: "Owen O'Malley" <oom@yahoo-inc.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v935.3)
Subject: jira notifications on common, hdfs, and mapreduce
Date: Tue, 30 Jun 2009 10:53:29 -0700
X-Mailer: Apple Mail (2.935.3)
X-Virus-Checked: Checked by ClamAV on apache.org

All,
    As we voted on core-dev, I just changed the jira notifications to:

create/resolved/reopen events are sent to {common,mapreduce,hdfs}-dev
all jira events are sent to {common,mapreduce,hdfs}-issues

-- Owen

From general-return-287-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 30 18:55:43 2009
Return-Path: <general-return-287-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58968 invoked from network); 30 Jun 2009 18:55:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Jun 2009 18:55:43 -0000
Received: (qmail 56306 invoked by uid 500); 30 Jun 2009 18:55:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56241 invoked by uid 500); 30 Jun 2009 18:55:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56231 invoked by uid 99); 30 Jun 2009 18:55:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Jun 2009 18:55:53 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [12.110.209.161] (HELO usausmgw01.spansion.com) (12.110.209.161)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Jun 2009 18:55:42 +0000
X-IronPort-AV: E=McAfee;i="5300,2777,5662"; a="5254637"
Received: from usausexbh1.spansion.com ([10.248.26.58])
  by usausmgw01.spansion.com with ESMTP; 30 Jun 2009 11:55:08 -0700
Received: from USAUSEXMBPF2.spansion.com ([10.248.26.56]) by USAUSEXBH1.spansion.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Tue, 30 Jun 2009 13:55:08 -0500
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable
X-MimeOLE: Produced By Microsoft Exchange V6.5
Subject: RE: problem with running WordCode v0 with a distributed operation
Date: Tue, 30 Jun 2009 13:55:08 -0500
Message-ID: <AF176AD83CF4724EAAD1AE6F1BAE02043DB2CE@USAUSEXMBPF2.spansion.com>
In-Reply-To: <aa41f20e0906301024o736adaa9s5daa9a1ce014a856@mail.gmail.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: problem with running WordCode v0 with a distributed operation
thread-index: Acn5p8Nqp8fKi3I5SOi++zWBapY+qwACeW2A
References: <aa41f20e0906300015s42558e37s9128b0e827083b24@mail.gmail.com> <AF176AD83CF4724EAAD1AE6F1BAE02043DB2CD@USAUSEXMBPF2.spansion.com> <aa41f20e0906301024o736adaa9s5daa9a1ce014a856@mail.gmail.com>
From: "Gross, Danny" <Danny.Gross@spansion.com>
To: <general@hadoop.apache.org>
X-OriginalArrivalTime: 30 Jun 2009 18:55:08.0612 (UTC) FILETIME=[49E7B040:01C9F9B4]
X-Virus-Checked: Checked by ClamAV on apache.org

Hi CJ,

One of my CentOS systems was built with the full installation.  This
installed a base iptables ruleset (sorry, I didn't keep the output from
that).  All my other systems looked like this:

[root@msddl01 ~]# iptables --list
Chain INPUT (policy ACCEPT)
target     prot opt source               destination        =20

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination        =20

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

I cleared the iptables for my problem system with the following:

/sbin/iptables -P INPUT ACCEPT
/sbin/iptables -P OUTPUT ACCEPT
/sbin/iptables -P FORWARD ACCEPT
/sbin/iptables -Z
/sbin/iptables -F

Once that was done, my node responded to the cluster, accepted jobs,
etc.

One thing I'm not clear on from your description below:  Do your HDFS
and map/reduce management web interfaces come up once you start dfs and
mapred?

My habit is:

- hadoop namenode -format (1st time)
- start-dfs.sh
- start-mapred.sh
- Check HDFS web interface -- status of HDFS nodes.=20
- Check map/reduce web interface -- status of task trackers.
- Make sure that both HDFS and map/reduce interfaces show as up and
running before kicking off jobs.  HDFS will go into standby mode, and
mapreduce will show initializing at times.

Hope it helps.

Best regards,

Danny

-----Original Message-----
From: C J [mailto:xine.jar@googlemail.com]=20
Sent: Tuesday, June 30, 2009 12:25 PM
To: general@hadoop.apache.org
Subject: Re: problem with running WordCode v0 with a distributed
operation

Hi Danny,
can you be a bit more precise what was wrong in your iptable?
thank you

On Tue, Jun 30, 2009 at 5:30 PM, Gross, Danny
<Danny.Gross@spansion.com>wrote:

> Hello,
>
> I suggest that you check iptables on your systems.  At one time, one
of
> my nodes showed a similar error, and this was the culprit.
>
> Good luck,
>
> Danny
>
> -----Original Message-----
> From: C J [mailto:xine.jar@googlemail.com]
> Sent: Tuesday, June 30, 2009 2:15 AM
> To: general@hadoop.apache.org; C J
> Subject: problem with running WordCode v0 with a distributed operation
>
> Help help help,
>
>  I am a new user, it has been already 2-3 weeks that I am trying to
run
> Hadoop 0.18.3.
>
>
>  *My settings:*
>
>   1.
>
>   I am on LINUX
>   2.
>
>    Using the Java 1.6.0
>    3.
>
>   I have unpacked the hadoop-0.18.3 folder directly on the desktop
>
>
> *Working steps:*
>
>   1.
>
>   I have succeeded to run it on the local
>    2.
>
>   I went through the quick start tutorial and managed to operate
>
>   Hadoop in the standalone and the Pseudo-distributed motes
>    3.
>
>   I started going through the map/reduce tutorial and managed to
>
>   run the WordCount v1.0 with a standalone operation.
>
> *Current status:*
>
>   1.
>
>   I would like to run the WordCount v1.0 example on a distributed
> operation
>    2.
>
>   I went through the steps in the cluster setup tutorial and did the
>   following:
>
>
>   -
>
>   On a distant server I am running 4 virtual machines:
>
>   134.130.223.58:1
>
>   134.130.223.72:1
>
>   134.130.223.85:1
>
>   134.130.223.92:1
>   -
>
>   My own machine has the following ip address
>
>   134.130.222.54
>   -
>
>   The hadoop-en.sh file is the same on all the five machines:
>
>   *export HADOOP_HEAPSIZE=3D500*
>
>   *export JAVA_HOME=3D"/usr/java/jdk1.6.0_14"*
>
>   *export HADOOP_NAMENODE_OPTS=3D"-Dcom.sun.management.jmxremote
>   $HADOOP_NAMENODE_OPTS"*
>
>   *export
HADOOP_SECONDARYNAMENODE_OPTS=3D"-Dcom.sun.management.jmxremote
>   $HADOOP_SECONDARYNAMENODE_OPTS"*
>
>   *export HADOOP_DATANODE_OPTS=3D"-Dcom.sun.management.jmxremote *
>
>   *$HADOOP_DATANODE_OPTS"*
>
>   *export HADOOP_BALANCER_OPTS=3D"-Dcom.sun.management.jmxremote *
>
>   *$HADOOP_BALANCER_OPTS"*
>
>   *export HADOOP_JOBTRACKER_OPTS=3D"-Dcom.sun.management.jmxremote *
>
>   *$HADOOP_JOBTRACKER_OPTS"*
>
>   *export HADOOP_HOME=3D"/root/Desktop/hadoop-0.18.3"*
>
>   *export HADOOP_VERSION=3D"0.18.3"*
>
>   *export HADOOP_LOG_DIR=3D${HADOOP_HOME}/logs*
>   -
>
>   The hadoop-site.xml is the same on all the five machines:
>
>   *<?xml version=3D"1.0"?>*
>
>   *<?xml-stylesheet type=3D"text/xsl" href=3D"configuration.xsl"?>*
>
>   *<!-- Put site-specific property overrides in this file. -->*
>
>   *<configuration>*
>
>   * <property>*
>
>   * Hadoop Quickstart*
>
>   * Page 3*
>
>    * Copyright (c) 2007 The Apache Software Foundation. All rights
> reserved.*
>
>   * <name>fs.default.name</name>*
>
>   * <value>134.130.223.85:9000</value>*
>
>   * </property>*
>
>   * <property>*
>
>   * <name>mapred.job.tracker</name>*
>
>   * <value>134.130.223.58:1</value>*
>
>   * </property>*
>
>   * <property>*
>
>   * <name>dfs.name.dir</name>*
>
>   * <value>/root/Desktop/hadoop-0.18.3/logstina</value>*
>
>   * </property>*
>
>   * <property>*
>
>   * <name>dfs.data.dir</name>*
>
>   * <value>/root/Desktop/hadoop-0.18.3/blockstina</value>*
>
>   * </property>*
>
>   * <property>*
>
>   * <name>mapred.system.dir</name>*
>
>   * <value>systemtina</value>*
>
>   * </property>*
>
>   * <property>*
>
>   * <name>mapred.local.dir</name>*
>
>   * <value>/root/Desktop/hadoop-0.18.3/tempMapReducetina</value>*
>
>   * </property>*
>
>   *</configuration> *
>
>
>
>
>
>   -
>
>   The slaves file is the same on all five machines and contains:
>
>   134.130.223.72
>
>   134.130.222.54
>
>   134.130.223.92
>   -
>
>   from the Namenode machine 134.130.223.85 I have formatted a new
> namenode,
>   started the hdfs (bin/start-dfs.sh) and started the Map-reduce
>   (bin/start-mapred.sh)
>
>
>  *Problem:*
>
> Si      Since the jar file of the WordCount was already created (by
the
> local operation) in the folder
>
> us     usr/tina, I tried running directly the application similarly to
> local
> operation by typing
>
>   *$bin/hadoop jar usr/tina/wordcount.jar org.myorg.WordCount
> usr/tina/wordcount/input usr/tina/wordcount/output.*
>
>  Then I got the following error:
>
>
>
>  *09/06/29 19:38:05 WARN fs.FileSystem: "134.130.223.85:9000" is a
> deprecated filesystem name. Use "hdfs://134.130.223.85:9000/"
instead.*
>
> *09/06/29 19:38:06 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 0 time(s).*
>
> *09/06/29 19:38:07 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 1 time(s).*
>
> *09/06/29 19:38:08 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 2 time(s).*
>
> *09/06/29 19:38:09 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 3 time(s).*
>
> *09/06/29 19:38:10 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 4 time(s).*
>
> *09/06/29 19:38:11 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 5 time(s).*
>
> *09/06/29 19:38:12 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 6 time(s).*
>
> *09/06/29 19:38:13 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 7 time(s).*
>
> *09/06/29 19:38:14 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 8 time(s).*
>
> *09/06/29 19:38:15 INFO ipc.Client: Retrying connect to server: /
> 134.130.223.85:9000. Already tried 9 time(s).*
>
> *java.lang.RuntimeException: java.net.ConnectException: Call to /
> 134.130.223.85:9000 failed on connection exception:
> java.net.ConnectException: Connection refused*
>
> *at
>
org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:358)*
>
> *at
>
org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.j
> ava:377)
> *
>
> *at org.myorg.WordCount.main(WordCount.java:53)*
>
> *at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)*
>
> *at
>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
> a:39)
> *
>
> *at
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:25)
> *
>
> *at java.lang.reflect.Method.invoke(Method.java:597)*
>
> *at org.apache.hadoop.util.RunJar.main(RunJar.java:155)*
>
> *at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)*
>
> *at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)*
>
> *at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)*
>
> *at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)*
>
> *Caused by: java.net.ConnectException: Call to /134.130.223.85:9000
> failed
> on connection exception: java.net.ConnectException: Connection
refused*
>
> *at org.apache.hadoop.ipc.Client.wrapException(Client.java:743)*
>
> *at org.apache.hadoop.ipc.Client.call(Client.java:719)*
>
> *at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)*
>
> *at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)*
>
> *at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:348)*
>
> *at
> org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:103)*
>
> *at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:172)*
>
> *at
>
org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSy
> stem.java:67)
> *
>
> *at
>
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1339)*
>
> *at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)*
>
> *at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1351)*
>
> *at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:213)*
>
> *at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:118)*
>
> *at
>
org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:354)*
>
> *... 11 more*
>
> *Caused by: java.net.ConnectException: Connection refused*
>
> *at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)*
>
> *at
>
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)*
>
> *at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)*
>
> *at
>
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:301)*
>
> *at
> org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:178)*
>
> *at org.apache.hadoop.ipc.Client.getConnection(Client.java:820)*
>
> *at org.apache.hadoop.ipc.Client.call(Client.java:705)*
>
> *... 23 more*
>
>
>
>
>  * Questions*
>
>   1.
>
>   Can someone help me in solving/debugging this problem?
>
>   P.S: I have tried to stop the HDFS with bin/stop-dfs.sh before
>
>   starting new ones.
>
>
>  Thank you,
>
> C.J
>

From general-return-2647-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jan 02 23:20:08 2011
Return-Path: <general-return-2647-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35317 invoked from network); 2 Jan 2011 23:20:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jan 2011 23:20:08 -0000
Received: (qmail 37691 invoked by uid 500); 2 Jan 2011 23:20:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37475 invoked by uid 500); 2 Jan 2011 23:20:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37467 invoked by uid 99); 2 Jan 2011 23:20:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 02 Jan 2011 23:20:05 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of magawake@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 02 Jan 2011 23:19:59 +0000
Received: by vws18 with SMTP id 18so5265641vws.35
        for <general@hadoop.apache.org>; Sun, 02 Jan 2011 15:19:38 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=rGWghuD+E62mBN4ETFLpEkwfAY+WRXW/V12FX/PdHAE=;
        b=Jj+lrY88YL0PSfXJtma/jlktT5hjm3QuuXDLEdiwP7hWUooIRAlfwHu4JhfbTro5N2
         fs21JngTsy5YuK92RFj1BfgNQfJp3QA/v+1VRwGiK6lKl4O25wkDsIRryZNtcdRc5Eks
         k4aizJ9sS9M4mSDK6APkyQsXFKHISp5SgRzBg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=VFu07l6It8KjhY9AXw0sxcj3k7Q2NPPk1P/o4hLcPmvLWyXyKgr0eGeUtyomV2XIyo
         2oxn57CbKAuFzFiaWWHbzKC0w/v1OEzLMr+6P4fMojAq2vWe6IWMbVwacT6TTmUPnTqc
         uoK+MuSExK7AqGLdDbfKfZrlULuRhyFjd3tFM=
MIME-Version: 1.0
Received: by 10.220.193.8 with SMTP id ds8mr2485890vcb.159.1294010376512; Sun,
 02 Jan 2011 15:19:36 -0800 (PST)
Received: by 10.220.118.20 with HTTP; Sun, 2 Jan 2011 15:19:36 -0800 (PST)
In-Reply-To: <AANLkTikJMzXhYA1wJwxYm=9Zc=PdAtz6-5mOoUQEF5Dh@mail.gmail.com>
References: <AANLkTi=cyTOYxSbvn_YqRTuHPW0b74=EVWhky+oqxSGM@mail.gmail.com>
	<AANLkTikJMzXhYA1wJwxYm=9Zc=PdAtz6-5mOoUQEF5Dh@mail.gmail.com>
Date: Sun, 2 Jan 2011 18:19:36 -0500
Message-ID: <AANLkTi=6jLDBd7g2TF5cD_7b_v63n+m_eG+z9t31FW-g@mail.gmail.com>
Subject: Re: dfs.datanode.du.pct question
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks for the response.

It would be nice to have a feature like this. It will make our data
nodes much more flexible.







On Fri, Dec 31, 2010 at 5:56 PM, Harsh J <qwertymaniac@gmail.com> wrote:
> Hi,
>
> Isn't this property a legacy one? 0.18 is the last release I see with
> this property. I've always used dfs.datanode.du.reserved instead,
> reserving constant space across all dfs.data.dir volumes.
>
> On Fri, Dec 31, 2010 at 9:32 PM, Mag Gam <magawake@gmail.com> wrote:
>> On some of my data nodes I have 2 filesystems. /fs1 and /fs2. Is it
>> possible to set the du pct so, for fs1 its 80% usage and /fs2 its 20%
>> usage?
>
> According to the release's code: No.
>
> (Unless two DataNodes are run off the same machine, which is not recommended.)
>
> --
> Harsh J
> www.harshj.com
>

From general-return-2648-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jan 02 23:48:50 2011
Return-Path: <general-return-2648-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 41483 invoked from network); 2 Jan 2011 23:48:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jan 2011 23:48:50 -0000
Received: (qmail 56412 invoked by uid 500); 2 Jan 2011 23:48:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56364 invoked by uid 500); 2 Jan 2011 23:48:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56356 invoked by uid 99); 2 Jan 2011 23:48:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 02 Jan 2011 23:48:48 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [209.191.69.23] (HELO web30006.mail.mud.yahoo.com) (209.191.69.23)
    by apache.org (qpsmtpd/0.29) with SMTP; Sun, 02 Jan 2011 23:48:40 +0000
Received: (qmail 26956 invoked by uid 60001); 2 Jan 2011 23:48:18 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1294012098; bh=k02A2RmxG5YdHlm+FOAgIYAVadISeudPoYt0liIbTMI=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type; b=VJUrVhDLmEnCNSmjRmbvANcrWXIwWsNI2MDb4SkrEGReEi14rwkXH8WBXMrHsQV71Uj3k9lfuKtCDQRoWAu23TfykRWWFQ5h8UFUR/bfo0iV/o87DTg7ySGqh1B6xiOcgpAcuf9fnt0r5uzvke+PvQbhVhD3WKyOCOPo2pApcTA=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type;
  b=rtGl545D+3bWe2LaXiKpYH5wT7NNiSHVNzFdMANakJ7/ix5CR7WNCKL9tgRLJPN0QKP5kti34RHVObbl/9rp8V3XqzrbqzDPZbXwpocjD3uVYZ8omvZiQHyc+gsXE5oeuhu84PleN454R/DN2PtajpUkginEWa15q2rxlxxOCsk=;
Message-ID: <615288.25718.qm@web30006.mail.mud.yahoo.com>
X-YMail-OSG: S0pxn5UVM1mXhF0jLLt3_MtV9.q5UZwG3oQcZ8rxlYAID27
 ubG_slbid6KUc4_n1sHf2rLVmnD3lHyuIbxwLWyZkx4b1UH4hjnquSqutEGX
 c6YtuIsPcUijzk8X2F6sqIyeyxYFUOfVYwsDX5eOj6RVMFowjJHeUVyLeaYC
 Eoyxr0NI3Ll0sXR_QtHvdKNlbq82mTFoVgi2nhw4jSVkoMnyxSdQT0Kwx19u
 pxoz065mUnixskXf9iW0wXoqCOoqr.eEHHe_fb3uSD6MENOry5fKYluE8Hw5
 QCXs4IMVBQtWmK6Ke47_uqrCQM56IDPmK4ywTeD_ABsR.MhqJY0iOKg4ZcWw
 djj95m2U4tv8H1lck_bNsh7aStpflKrsYfxte.EyZXTqL_HgQIUjYHb.5EY9
 iwMA9sKAbHak-
Received: from [76.126.156.12] by web30006.mail.mud.yahoo.com via HTTP; Sun, 02 Jan 2011 15:48:18 PST
X-Mailer: YahooMailClassic/11.4.20 YahooMailWebService/0.8.107.285259
Date: Sun, 2 Jan 2011 15:48:18 -0800 (PST)
From: whorl1quote-re@yahoo.com
Reply-To: whorl1quote-re@yahoo.com
Subject: fair scheduler on memory usage
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1857806086-1294012098=:25718"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-1857806086-1294012098=:25718
Content-Type: text/plain; charset=us-ascii

Can fair scheduler set an upper limit on a job's memory consumption? How do I use fair/capacity scheduler to make sure a long running (memory drain) job won't impact the service level of other higher priority jobs?

--0-1857806086-1294012098=:25718--

From general-return-2649-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 03 17:59:19 2011
Return-Path: <general-return-2649-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37054 invoked from network); 3 Jan 2011 17:59:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jan 2011 17:59:19 -0000
Received: (qmail 13944 invoked by uid 500); 3 Jan 2011 17:59:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13738 invoked by uid 500); 3 Jan 2011 17:59:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 17299 invoked by uid 99); 3 Jan 2011 11:01:43 -0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1294052435;
	bh=ORylPbyWpN68zLMA6nl0CS76YgXPL1xMNwUfQueWCZI=;
	h=Message-ID:Date:From:Reply-To:MIME-Version:To:Subject:
	 Content-Type;
	b=ob21XefM9/qZD71CSd1ujr2ad5EDMQwTEu8MQTPPn0YOB2kOkVao0oJHDrM5TPbLu
	 E6gJ+dIQJA0lh01SlAuvoEUU4mZblaoSanbQ2rMMVSF8FOp7YPIKfcmVWNkQnmlqSZ
	 PMUFFFTOQxN7A/faYQ7ZazGgJmt2i2y/cgOYlW2U=
Message-ID: <4D21AC48.2060200@yahoo-inc.com>
Date: Mon, 03 Jan 2011 16:30:24 +0530
From: Basant Verma <bverma@yahoo-inc.com>
Reply-To: yahoohadoop@yahoo-inc.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.13) Gecko/20101207 Lightning/1.0b2 Thunderbird/3.1.7
MIME-Version: 1.0
To: common-user@hadoop.apache.org, user@pig.apache.org, user@hive.apache.org,
        mapreduce-user@hadoop.apache.org, hdfs-user@hadoop.apache.org,
        general@hadoop.apache.org
Subject: Hadoop India Summit 2011
Content-Type: multipart/mixed;
 boundary="------------050608010503050709020404"
X-Virus-Checked: Checked by ClamAV on apache.org

--------------050608010503050709020404
Content-Type: multipart/alternative;
 boundary="------------020100060701060208070900"


--------------020100060701060208070900
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi Hadoop enthusiasts,

Apache Hadoop has become the de-facto platform for developing 
large-scale data-intensive applications. It has been used actively in 
academia and Industry for research and data mining. Hadoop Summit 
provides an opportunity for understanding the latest trends and roadmap 
for Hadoop Platform and its ecosystem and how Hadoop is leveraged in 
various domains.

Yahoo! India R&D is proud to host second India Hadoop Summit which will 
take place in Bangalore, India on February 16th or 17th, 2011.

Please look at the attached information if you'd like to present in the 
summit. Stay tuned for summit registration info which we will publish in 
early January.

Best Regards,
Basant Verma
Yahoo, Hadoop

--------------020100060701060208070900
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
  <head>

    <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
  </head>
  <body bgcolor="#ffffff" text="#000000">
    <font face="Calibri, Verdana, Helvetica, Arial"><span
        style="font-size: 11pt;">Hi Hadoop enthusiasts,<br>
      </span><br>
    </font><font face="Calibri, Verdana, Helvetica, Arial"><span
        style="font-size: 11pt;">Apache Hadoop has become the de-facto
        platform for developing large-scale data-intensive applications.
        It has been used actively in academia and Industry for research
        and data mining. Hadoop Summit provides an opportunity for
        understanding the latest trends and roadmap for Hadoop Platform
        and its ecosystem and how Hadoop is leveraged in various
        domains.<br>
        <br>
        Yahoo! India R&amp;D is proud to host second India Hadoop Summit
        which will take place in Bangalore, India on February 16th or
        17th, 2011. <br>
        <br>
        Please look at the attached information if you&#8217;d like to present
        in the summit. Stay tuned for summit registration info which we
        will publish in early January.<br>
        <br>
        Best Regards,<br>
        Basant Verma<br>
        Yahoo, Hadoop<br>
      </span></font>
  </body>
</html>

--------------020100060701060208070900--

--------------050608010503050709020404--


From general-return-2650-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 03 20:23:00 2011
Return-Path: <general-return-2650-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19755 invoked from network); 3 Jan 2011 20:22:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jan 2011 20:22:59 -0000
Received: (qmail 27611 invoked by uid 500); 3 Jan 2011 20:22:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27546 invoked by uid 500); 3 Jan 2011 20:22:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 22665 invoked by uid 99); 3 Jan 2011 20:21:37 -0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of buttler1@llnl.gov designates 128.115.41.83 as permitted sender)
X-Attachments: None
From: "Buttler, David" <buttler1@llnl.gov>
To: "user@hbase.apache.org" <user@hbase.apache.org>,
	"'general@hadoop.apache.org'" <general@hadoop.apache.org>
CC: "Fox, Kevin M" <kevin.fox@pnl.gov>, "Brown, David M JR"
	<david.brown@pnl.gov>
Date: Mon, 3 Jan 2011 12:20:53 -0800
Subject: RE: What is the fastest way to get a large amount of data into the
  Hadoop HDFS file system  (or Hbase)?
Thread-Topic: What is the fastest way to get a large amount of data into the
  Hadoop HDFS file system  (or Hbase)?
Thread-Index: Acul2gQhGBzjuFTiQL2qQO67SCiwKwA/gmuwASqbeAA=
Message-ID: <2D6136772A13B84E95DF6DA79E85A9F00134476BEFDC@NSPEXMBX-A.the-lab.llnl.gov>
References: <590321.36008.qm@web31813.mail.mud.yahoo.com>
 <536061.77419.qm@web130101.mail.mud.yahoo.com>
 <A632B3F93D861843A3C966F3962C3F84010822D5000C@EMAIL05.pnl.gov>
 <4D18AF52.9080800@mozilla.com>
 <A632B3F93D861843A3C966F3962C3F84010822D50026@EMAIL05.pnl.gov>
In-Reply-To: <A632B3F93D861843A3C966F3962C3F84010822D50026@EMAIL05.pnl.gov>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Ron,
Loading into HDFS and HBase are two different issues. =20

HDFS: if you have a large number of files to load from your nfs file system=
 into HDFS it is not clear that parallelizing the load will help.  You have=
 two sources of bottlenecks: the nfs file system and the HDFS file system. =
 In your parallel example, you will likely saturate your nfs file system fi=
rst.  If they are actually local files, then loading them via M/R is a non-=
starter as you have no control over which machine will get a map task.  Unl=
ess all of the machines have files in the same directory and you are just g=
oing to look in that directory to upload.  Then, it sounds like more of a j=
ob for a parallel shell command and less of a map/reduce command.

HBase: So far my strategy has been to get the files into HDFS first, and th=
en write a Map job to load them into HBase.  You can try to do this and see=
 if direct inserts into hbase are fast enough for your use case.  But, if y=
ou are going to TBs/week then you will likely want to investigate the bulk =
load features.  I haven't yet incorporated that into my workflow so I can't=
 offer much advice there. Just be sure your cluster is sized appropriately.=
  E.g., with your compression turned on in hbase, see how much a 1 GB input=
 file expands to inside hbase / hdfs.  That should give you a feeling for h=
ow much space you will need for your expected data load.

Dave


-----Original Message-----
From: Taylor, Ronald C [mailto:ronald.taylor@pnl.gov]=20
Sent: Tuesday, December 28, 2010 2:05 PM
To: 'user@hbase.apache.org'; 'general@hadoop.apache.org'
Cc: Taylor, Ronald C; Fox, Kevin M; Brown, David M JR
Subject: What is the fastest way to get a large amount of data into the Had=
oop HDFS file system (or Hbase)?


Folks,

We plan on uploading large amounts of data on a regular basis onto a Hadoop=
 cluster, with Hbase operating on top of Hadoop. Figure eventually on the o=
rder of multiple terabytes per week. So - we are concerned about doing the =
uploads themselves as fast as possible from our native Linux file system in=
to HDFS. Figure files will be in, roughly, the 1 to 300 GB range.=20

Off the top of my head, I'm thinking that doing this in parallel using a Ja=
va MapReduce program would work fastest. So my idea would be to have a file=
 listing all the data files (full paths) to be uploaded, one per line, and =
then use that listing file as input to a MapReduce program.=20

Each Mapper would then upload one of the data files (using "hadoop fs -copy=
FromLocal <source> <dest>") in parallel with all the other Mappers, with th=
e Mappers operating on all the nodes of the cluster, spreading out the file=
 upload across the nodes.

Does that sound like a wise way to approach this? Are there better methods?=
 Anything else out there for doing automated upload in parallel? We would v=
ery much appreciate advice in this area, since we believe upload speed migh=
t become a bottleneck.

  - Ron Taylor

___________________________________________
Ronald Taylor, Ph.D.
Computational Biology & Bioinformatics Group

Pacific Northwest National Laboratory
902 Battelle Boulevard
P.O. Box 999, Mail Stop J4-33
Richland, WA  99352 USA
Office:  509-372-6568
Email: ronald.taylor@pnl.gov



From general-return-2651-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 03 20:44:27 2011
Return-Path: <general-return-2651-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 28164 invoked from network); 3 Jan 2011 20:44:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jan 2011 20:44:27 -0000
Received: (qmail 51866 invoked by uid 500); 3 Jan 2011 20:44:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51754 invoked by uid 500); 3 Jan 2011 20:44:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51739 invoked by uid 99); 3 Jan 2011 20:44:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Jan 2011 20:44:23 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [192.101.109.31] (HELO emailgw03.pnl.gov) (192.101.109.31)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Jan 2011 20:44:16 +0000
X-IronPort-AV: E=Sophos;i="4.60,268,1291622400"; 
   d="scan'208";a="35972043"
Received: from emailhub02.pnl.gov ([130.20.251.62])
  by emailgw03.pnl.gov with ESMTP/TLS/AES128-SHA; 03 Jan 2011 12:43:56 -0800
Received: from Email05.pnl.gov ([130.20.251.69]) by emailhub02.pnl.gov
 ([130.20.251.62]) with mapi; Mon, 3 Jan 2011 12:43:55 -0800
From: "Taylor, Ronald C" <ronald.taylor@pnl.gov>
To: "user@hbase.apache.org" <user@hbase.apache.org>,
	"'general@hadoop.apache.org'" <general@hadoop.apache.org>
CC: "Fox, Kevin M" <kevin.fox@pnl.gov>, "Brown, David M JR"
	<david.brown@pnl.gov>, "Taylor, Ronald C" <ronald.taylor@pnl.gov>
Date: Mon, 3 Jan 2011 12:43:55 -0800
Subject: RE: What is the fastest way to get a large amount of data into the
  Hadoop HDFS file system  (or Hbase)?
Thread-Topic: What is the fastest way to get a large amount of data into the
  Hadoop HDFS file system  (or Hbase)?
Thread-Index: Acul2gQhGBzjuFTiQL2qQO67SCiwKwA/gmuwASqbeAAAAIt4AA==
Message-ID: <A632B3F93D861843A3C966F3962C3F84010822D5005B@EMAIL05.pnl.gov>
References: <590321.36008.qm@web31813.mail.mud.yahoo.com>
 <536061.77419.qm@web130101.mail.mud.yahoo.com>
 <A632B3F93D861843A3C966F3962C3F84010822D5000C@EMAIL05.pnl.gov>
 <4D18AF52.9080800@mozilla.com>
 <A632B3F93D861843A3C966F3962C3F84010822D50026@EMAIL05.pnl.gov>
 <2D6136772A13B84E95DF6DA79E85A9F00134476BEFDC@NSPEXMBX-A.the-lab.llnl.gov>
In-Reply-To: <2D6136772A13B84E95DF6DA79E85A9F00134476BEFDC@NSPEXMBX-A.the-lab.llnl.gov>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi Dave,

Thanks for the suggestions. Glad to hear from a fellow DOE national lab per=
son!=20

We are just starting to explore all this here at Pacific Northwest Nat Lab,=
 and what will be going into Hbase and what will be left as files in HDFS i=
s an open question, to be empirically determined over the coming year. It w=
ill depend on upon what instrument data gets put in, how the users want to =
analyze the data, what turns out to be practical for future growth and main=
tenance, etc. My lab colleagues Kevin Fox and David Brown have a lot more e=
xperience handling massive amount of data - they are already handling hundr=
eds of TBs in the archive cluster for EMSL, our national user facility (lot=
s of mass spec, NMR, microscopy, and next gen sequencing machines for biolo=
gy and chemistry, as you may already know). And they have much better grip =
on the hardware and OS side of things. So I imagine you & the list will be =
hearing directly from them fairly often as questions arise.

 Ron

-----Original Message-----
From: Buttler, David [mailto:buttler1@llnl.gov]=20
Sent: Monday, January 03, 2011 12:21 PM
To: user@hbase.apache.org; 'general@hadoop.apache.org'
Cc: Fox, Kevin M; Brown, David M JR
Subject: RE: What is the fastest way to get a large amount of data into the=
 Hadoop HDFS file system (or Hbase)?

Hi Ron,
Loading into HDFS and HBase are two different issues. =20

HDFS: if you have a large number of files to load from your nfs file system=
 into HDFS it is not clear that parallelizing the load will help.  You have=
 two sources of bottlenecks: the nfs file system and the HDFS file system. =
 In your parallel example, you will likely saturate your nfs file system fi=
rst.  If they are actually local files, then loading them via M/R is a non-=
starter as you have no control over which machine will get a map task.  Unl=
ess all of the machines have files in the same directory and you are just g=
oing to look in that directory to upload.  Then, it sounds like more of a j=
ob for a parallel shell command and less of a map/reduce command.

HBase: So far my strategy has been to get the files into HDFS first, and th=
en write a Map job to load them into HBase.  You can try to do this and see=
 if direct inserts into hbase are fast enough for your use case.  But, if y=
ou are going to TBs/week then you will likely want to investigate the bulk =
load features.  I haven't yet incorporated that into my workflow so I can't=
 offer much advice there. Just be sure your cluster is sized appropriately.=
  E.g., with your compression turned on in hbase, see how much a 1 GB input=
 file expands to inside hbase / hdfs.  That should give you a feeling for h=
ow much space you will need for your expected data load.

Dave


-----Original Message-----
From: Taylor, Ronald C [mailto:ronald.taylor@pnl.gov]=20
Sent: Tuesday, December 28, 2010 2:05 PM
To: 'user@hbase.apache.org'; 'general@hadoop.apache.org'
Cc: Taylor, Ronald C; Fox, Kevin M; Brown, David M JR
Subject: What is the fastest way to get a large amount of data into the Had=
oop HDFS file system (or Hbase)?


Folks,

We plan on uploading large amounts of data on a regular basis onto a Hadoop=
 cluster, with Hbase operating on top of Hadoop. Figure eventually on the o=
rder of multiple terabytes per week. So - we are concerned about doing the =
uploads themselves as fast as possible from our native Linux file system in=
to HDFS. Figure files will be in, roughly, the 1 to 300 GB range.=20

Off the top of my head, I'm thinking that doing this in parallel using a Ja=
va MapReduce program would work fastest. So my idea would be to have a file=
 listing all the data files (full paths) to be uploaded, one per line, and =
then use that listing file as input to a MapReduce program.=20

Each Mapper would then upload one of the data files (using "hadoop fs -copy=
FromLocal <source> <dest>") in parallel with all the other Mappers, with th=
e Mappers operating on all the nodes of the cluster, spreading out the file=
 upload across the nodes.

Does that sound like a wise way to approach this? Are there better methods?=
 Anything else out there for doing automated upload in parallel? We would v=
ery much appreciate advice in this area, since we believe upload speed migh=
t become a bottleneck.

  - Ron Taylor

___________________________________________
Ronald Taylor, Ph.D.
Computational Biology & Bioinformatics Group

Pacific Northwest National Laboratory
902 Battelle Boulevard
P.O. Box 999, Mail Stop J4-33
Richland, WA  99352 USA
Office:  509-372-6568
Email: ronald.taylor@pnl.gov



From general-return-2652-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 03 21:46:40 2011
Return-Path: <general-return-2652-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55372 invoked from network); 3 Jan 2011 21:46:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jan 2011 21:46:39 -0000
Received: (qmail 38282 invoked by uid 500); 3 Jan 2011 21:46:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38043 invoked by uid 500); 3 Jan 2011 21:46:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 12771 invoked by uid 99); 3 Jan 2011 21:38:59 -0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
From: "Mattmann, Chris A (388J)" <chris.a.mattmann@jpl.nasa.gov>
To: "dev@oodt.apache.org" <dev@oodt.apache.org>,
        "general@hadoop.apache.org"
	<general@hadoop.apache.org>,
        "common-user@hadoop.apache.org"
	<common-user@hadoop.apache.org>
CC: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org>,
        "user@oodt.apache.org" <user@oodt.apache.org>,
        "general@incubator.apache.org"
	<general@incubator.apache.org>,
        "dev@tika.apache.org" <dev@tika.apache.org>,
        "user@tika.apache.org" <user@tika.apache.org>,
        "whirr-dev@incubator.apache.org" <whirr-dev@incubator.apache.org>,
        "gora-dev@incubator.apache.org" <gora-dev@incubator.apache.org>,
        "sis-dev@incubator.apache.org" <sis-dev@incubator.apache.org>
Date: Mon, 3 Jan 2011 13:38:24 -0800
Subject: [Call for Papers] ICSE Software Engineering for Cloud Computing
 (SECLOUD) Workshop
Thread-Topic: [Call for Papers] ICSE Software Engineering for Cloud
 Computing (SECLOUD) Workshop
Thread-Index: Acurjo0U1eus/GuLQlSXr8nNmcFIBw==
Message-ID: <DC6E34C8-7ECE-42B8-A55E-2A7DA11EB5BD@jpl.nasa.gov>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Source-IP: altvirehtstap01.jpl.nasa.gov [128.149.137.72]
X-Source-Sender: chris.a.mattmann@jpl.nasa.gov
X-AUTH: Authorized
X-Virus-Checked: Checked by ClamAV on apache.org

(apologies for the cross posting)

Please consider submitting a paper to the ICSE 2011 Software Engineering fo=
r Cloud Computing (SECLOUD) Workshop to be held Sunday, May 22, 2011, at th=
e Hilton Hawaiian Village Resort in Waikiki, Honolulu, HI.

This workshop focuses on identifying the grand challenges that lay before u=
s in the realm of software engineering for cloud computing. We will debate =
existing notions of SE for the construction of cloud services and for their=
 deployment. We will discuss and evangelize individual successes in the fie=
ld and attempt to identify the key ingredients that enabled that success. P=
articipants in this workshop will take a role in helping to formulate a lon=
g-term, concrete software engineering agenda for cloud and will have an opp=
ortunity to share in and contribute to the existing =93tribal knowledge=94 =
for cloud-related software engineering.

We invite high quality submissions of technical and research papers, as wel=
l as research demonstrations describing original and unpublished results of=
 research relevant to software engineering for cloud. Authors are asked to =
reserve a section in their submitted papers for identifying their suggested=
 near-term and long-term challenge areas for the field.

The workshop seeks to focus discussion around the ways that the disruptive =
effect of cloud computing is engendering a new set of principles and approa=
ches to software engineering. Specific topics of interest include, but are =
not limited to:

Topic Areas of Interest:

- Agile Software development on the cloud
- Architecting Applications using the cloud
- Benchmarking and Performance Metrics in the cloud
- Cloud Application Reliability
- Distributed collaboration on the cloud
- Interoperability and Portability challenges in multi-cloud environments
- Open Source Cloud Environments: Hadoop, Eucalyptus and other programming =
paradigms, languages and environments for the Cloud
- Requirements Elicitation effectiveness when using cloud service abstracti=
ons
- Secure Software Engineering in the Cloud
- Software Engineering Practices using popular cloud vendor services
- Software Engineering tools
- Software Project Estimations, Testing,Verification and Validation over cl=
oud service abstractions

Submission Guidelines:

Submissions may take one of two acceptable forms. Research paper contributi=
ons may be submitted with a maximum length of 7 pages, and must conform to =
ICSE 2011 paper formatting guidelines [1]. Research demonstration submissio=
ns are limited to 1 page in the conference proceedings and will be given du=
ring he two 15-minute breakout sessions.

All submissions must be in English and in PDF format [2]. An abstract must =
accompany each research paper contribution. The workshop website [3] will b=
e updated to include a link to the CyberChairPRO online submission system o=
nce it has been set up. All accepted papers will be published and dissemina=
ted on the workshop website after the conclusion of the workshop.

Review and Evaluation Criteria:

Each submission will be reviewed by at least two members of the Program Com=
mittee and will be assessed on the basis of originality, importance and rel=
evance of contribution to SE for cloud, soundness, quality of presentation,=
 and an appropriate comparison to related work. The program committee as a =
whole will make final decisions about acceptance.

Workshop Organizers and PC Chairs:

Nenad Medvidovic , University of Southern California
T.S. Mohan, Infosys Technologies
Chris A. Mattmann, NASA Jet Propulsion Laboratory
Owen O=92Malley, Yahoo, Inc.

Important Dates:

Paper Submission: 21.January.2011
Paper Notification: 21.February.2011
Camera Ready: 10.March.2011

Website: http://sites.google.com/site/icsecloud2011

Program Committee:

Andrew F. Hart, NASA JPL, United States
Ben Reed, Yahoo, United States
Craig Lee, Aerospace Corp., United States
Daniel J. Crichton, NASA JPL, United States
David C. Kale, Children's Hospital LA, United States
David M. Woollard, Project WBS, United States
Dennis Kubes, Bit.ly, United States
Eric Dashofy, Aerospace Corp., United States
Jacek Becla, Stanford National Accelerator Lab, United States
Justin Erenkrantz, Project WBS/Apache, United States
Luca Cinquini, NASA JPL, United States
Mohanakrishna, Infosys, India
Nabor Mendonca, Univ. of Fortaleza, Brazil
Sanjay Radia, Yahoo, United States
Schahram Dustdar, TU Wien, Vienna
Srinivas Padmanabhuni, Infosys, India
Stefan Tai, Karlsruhe Institute of Technology, Germany
YN Srikant, Indian Institute of Science, Bangalore, India

Links:

[1] ICSE-2011 Format: http://2011.icse-conferences.org/content-submission-g=
uidelines
[2] ACM Submission Policies: http://www.acm.org/sigs/publications/proceedin=
gs-templates
[3] Workshop Website: http://sites.google.com/site/icsecloud2011


About ICSE:=20

The International Conference on Software Engineering (ICSE) is the premier =
software engineering conference, providing a forum for researchers, practit=
ioners and educators to present and discuss the most recent innovations, tr=
ends, experiences and concerns in the field of software engineering.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


From general-return-2653-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 03 22:58:42 2011
Return-Path: <general-return-2653-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 90561 invoked from network); 3 Jan 2011 22:58:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jan 2011 22:58:42 -0000
Received: (qmail 5279 invoked by uid 500); 3 Jan 2011 22:58:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5001 invoked by uid 500); 3 Jan 2011 22:58:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4993 invoked by uid 99); 3 Jan 2011 22:58:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Jan 2011 22:58:40 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [192.101.109.31] (HELO emailgw03.pnl.gov) (192.101.109.31)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Jan 2011 22:58:32 +0000
X-IronPort-AV: E=Sophos;i="4.60,268,1291622400"; 
   d="scan'208";a="35983849"
Received: from sledge.emsl.pnl.gov (HELO [130.20.231.68]) ([130.20.231.68])
  by emailgw03.pnl.gov with ESMTP; 03 Jan 2011 14:58:09 -0800
Subject: RE: What is the fastest way to get a large amount of data into the
 Hadoop HDFS file system  (or Hbase)?
From: Kevin Fox <Kevin.Fox@pnl.gov>
To: "Buttler, David" <buttler1@llnl.gov>
Cc: "user@hbase.apache.org" <user@hbase.apache.org>,  "'general@hadoop.apache.org'" <general@hadoop.apache.org>, "Brown, David M JR" <david.brown@pnl.gov>
In-Reply-To: <2D6136772A13B84E95DF6DA79E85A9F00134476BEFDC@NSPEXMBX-A.the-lab.llnl.gov>
References: <590321.36008.qm@web31813.mail.mud.yahoo.com>
	 <536061.77419.qm@web130101.mail.mud.yahoo.com>
	 <A632B3F93D861843A3C966F3962C3F84010822D5000C@EMAIL05.pnl.gov>
	 <4D18AF52.9080800@mozilla.com>
	 <A632B3F93D861843A3C966F3962C3F84010822D50026@EMAIL05.pnl.gov>
	 <2D6136772A13B84E95DF6DA79E85A9F00134476BEFDC@NSPEXMBX-A.the-lab.llnl.gov>
Content-Type: text/plain; charset="UTF-8"
Date: Mon, 03 Jan 2011 14:58:09 -0800
Message-ID: <1294095489.5455.195.camel@sledge.emsl.pnl.gov>
Mime-Version: 1.0
X-Mailer: Evolution 2.28.3 
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On Mon, 2011-01-03 at 12:20 -0800, Buttler, David wrote:
> Hi Ron,
> Loading into HDFS and HBase are two different issues.  
> 
> HDFS: if you have a large number of files to load from your nfs file system into HDFS it is not clear that parallelizing the load will help. 

Its not nfs. Its a parallel file system.

>  You have two sources of bottlenecks: the nfs file system and the HDFS file system.  In your parallel example, you will likely saturate your nfs file system first.

Unlikely in this case. We're in the unusual position of our archive
cluster being faster then our hadoop cluster.

>   If they are actually local files, then loading them via M/R is a non-starter as you have no control over which machine will get a map task.

If the same files are "local" on each node, does it matter? Shouldn't
the map jobs all be scheduled in a way as to spread out the load?

Thanks,
Kevin

>   Unless all of the machines have files in the same directory and you are just going to look in that directory to upload.  Then, it sounds like more of a job for a parallel shell command and less of a map/reduce command.
> 
> HBase: So far my strategy has been to get the files into HDFS first, and then write a Map job to load them into HBase.  You can try to do this and see if direct inserts into hbase are fast enough for your use case.  But, if you are going to TBs/week then you will likely want to investigate the bulk load features.  I haven't yet incorporated that into my workflow so I can't offer much advice there. Just be sure your cluster is sized appropriately.  E.g., with your compression turned on in hbase, see how much a 1 GB input file expands to inside hbase / hdfs.  That should give you a feeling for how much space you will need for your expected data load.
> 
> Dave
> 
> 
> -----Original Message-----
> From: Taylor, Ronald C [mailto:ronald.taylor@pnl.gov] 
> Sent: Tuesday, December 28, 2010 2:05 PM
> To: 'user@hbase.apache.org'; 'general@hadoop.apache.org'
> Cc: Taylor, Ronald C; Fox, Kevin M; Brown, David M JR
> Subject: What is the fastest way to get a large amount of data into the Hadoop HDFS file system (or Hbase)?
> 
> 
> Folks,
> 
> We plan on uploading large amounts of data on a regular basis onto a Hadoop cluster, with Hbase operating on top of Hadoop. Figure eventually on the order of multiple terabytes per week. So - we are concerned about doing the uploads themselves as fast as possible from our native Linux file system into HDFS. Figure files will be in, roughly, the 1 to 300 GB range. 
> 
> Off the top of my head, I'm thinking that doing this in parallel using a Java MapReduce program would work fastest. So my idea would be to have a file listing all the data files (full paths) to be uploaded, one per line, and then use that listing file as input to a MapReduce program. 
> 
> Each Mapper would then upload one of the data files (using "hadoop fs -copyFromLocal <source> <dest>") in parallel with all the other Mappers, with the Mappers operating on all the nodes of the cluster, spreading out the file upload across the nodes.
> 
> Does that sound like a wise way to approach this? Are there better methods? Anything else out there for doing automated upload in parallel? We would very much appreciate advice in this area, since we believe upload speed might become a bottleneck.
> 
>   - Ron Taylor
> 
> ___________________________________________
> Ronald Taylor, Ph.D.
> Computational Biology & Bioinformatics Group
> 
> Pacific Northwest National Laboratory
> 902 Battelle Boulevard
> P.O. Box 999, Mail Stop J4-33
> Richland, WA  99352 USA
> Office:  509-372-6568
> Email: ronald.taylor@pnl.gov
> 
> 



From general-return-2654-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 03 23:07:04 2011
Return-Path: <general-return-2654-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5552 invoked from network); 3 Jan 2011 23:07:04 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jan 2011 23:07:04 -0000
Received: (qmail 21162 invoked by uid 500); 3 Jan 2011 23:07:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21102 invoked by uid 500); 3 Jan 2011 23:07:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21094 invoked by uid 99); 3 Jan 2011 23:07:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Jan 2011 23:07:02 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Jan 2011 23:06:56 +0000
Received: by qyk10 with SMTP id 10so14389560qyk.14
        for <general@hadoop.apache.org>; Mon, 03 Jan 2011 15:06:35 -0800 (PST)
Received: by 10.229.80.77 with SMTP id s13mr18269792qck.242.1294095995019;
 Mon, 03 Jan 2011 15:06:35 -0800 (PST)
MIME-Version: 1.0
Received: by 10.229.82.17 with HTTP; Mon, 3 Jan 2011 15:06:14 -0800 (PST)
In-Reply-To: <AANLkTi=6jLDBd7g2TF5cD_7b_v63n+m_eG+z9t31FW-g@mail.gmail.com>
References: <AANLkTi=cyTOYxSbvn_YqRTuHPW0b74=EVWhky+oqxSGM@mail.gmail.com>
 <AANLkTikJMzXhYA1wJwxYm=9Zc=PdAtz6-5mOoUQEF5Dh@mail.gmail.com> <AANLkTi=6jLDBd7g2TF5cD_7b_v63n+m_eG+z9t31FW-g@mail.gmail.com>
From: "Aaron T. Myers" <atm@cloudera.com>
Date: Mon, 3 Jan 2011 15:06:14 -0800
Message-ID: <AANLkTikxP0Y3y2RR-5PzAPn5pjpuQY8wZXptXmp1sML3@mail.gmail.com>
Subject: Re: dfs.datanode.du.pct question
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e646a53a16a7430498f9343a
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e646a53a16a7430498f9343a
Content-Type: text/plain; charset=ISO-8859-1

Filed https://issues.apache.org/jira/browse/HDFS-1564

--
Aaron T. Myers
Software Engineer, Cloudera



On Sun, Jan 2, 2011 at 3:19 PM, Mag Gam <magawake@gmail.com> wrote:

> Thanks for the response.
>
> It would be nice to have a feature like this. It will make our data
> nodes much more flexible.
>
>
>
>
>
>
>
> On Fri, Dec 31, 2010 at 5:56 PM, Harsh J <qwertymaniac@gmail.com> wrote:
> > Hi,
> >
> > Isn't this property a legacy one? 0.18 is the last release I see with
> > this property. I've always used dfs.datanode.du.reserved instead,
> > reserving constant space across all dfs.data.dir volumes.
> >
> > On Fri, Dec 31, 2010 at 9:32 PM, Mag Gam <magawake@gmail.com> wrote:
> >> On some of my data nodes I have 2 filesystems. /fs1 and /fs2. Is it
> >> possible to set the du pct so, for fs1 its 80% usage and /fs2 its 20%
> >> usage?
> >
> > According to the release's code: No.
> >
> > (Unless two DataNodes are run off the same machine, which is not
> recommended.)
> >
> > --
> > Harsh J
> > www.harshj.com
> >
>

--0016e646a53a16a7430498f9343a--

From general-return-2655-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 04 01:06:28 2011
Return-Path: <general-return-2655-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66293 invoked from network); 4 Jan 2011 01:06:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Jan 2011 01:06:28 -0000
Received: (qmail 52669 invoked by uid 500); 4 Jan 2011 01:06:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52599 invoked by uid 500); 4 Jan 2011 01:06:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52591 invoked by uid 99); 4 Jan 2011 01:06:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Jan 2011 01:06:26 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of buttler1@llnl.gov designates 128.115.41.82 as permitted sender)
Received: from [128.115.41.82] (HELO nspiron-2.llnl.gov) (128.115.41.82)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Jan 2011 01:06:17 +0000
X-Attachments: None
Received: from nspexhub-2.llnl.gov (HELO nspexhub-2.the-lab.llnl.gov) ([128.115.54.114])
  by nspiron-2.llnl.gov with ESMTP; 03 Jan 2011 17:05:55 -0800
Received: from NSPEXMBX-A.the-lab.llnl.gov ([128.115.54.105]) by
 nspexhub-2.the-lab.llnl.gov ([172.16.54.114]) with mapi; Mon, 3 Jan 2011
 17:05:55 -0800
From: "Buttler, David" <buttler1@llnl.gov>
To: Kevin Fox <Kevin.Fox@pnl.gov>
CC: "user@hbase.apache.org" <user@hbase.apache.org>,
	"'general@hadoop.apache.org'" <general@hadoop.apache.org>, "Brown, David M
 JR" <david.brown@pnl.gov>
Date: Mon, 3 Jan 2011 17:05:54 -0800
Subject: RE: What is the fastest way to get a large amount of data into the
 Hadoop HDFS file system  (or Hbase)?
Thread-Topic: What is the fastest way to get a large amount of data into the
 Hadoop HDFS file system  (or Hbase)?
Thread-Index: AcurmbIMPrJ07f2rTE2CM3g4zhkKfAAEQG/w
Message-ID: <2D6136772A13B84E95DF6DA79E85A9F00134476BF12C@NSPEXMBX-A.the-lab.llnl.gov>
References: <590321.36008.qm@web31813.mail.mud.yahoo.com>
	 <536061.77419.qm@web130101.mail.mud.yahoo.com>
	 <A632B3F93D861843A3C966F3962C3F84010822D5000C@EMAIL05.pnl.gov>
	 <4D18AF52.9080800@mozilla.com>
	 <A632B3F93D861843A3C966F3962C3F84010822D50026@EMAIL05.pnl.gov>
	 <2D6136772A13B84E95DF6DA79E85A9F00134476BEFDC@NSPEXMBX-A.the-lab.llnl.gov>
 <1294095489.5455.195.camel@sledge.emsl.pnl.gov>
In-Reply-To: <1294095489.5455.195.camel@sledge.emsl.pnl.gov>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

UmlnaHQsIEkgc2hvdWxkIGhhdmUgcmVhbGl6ZWQgdGhhdCB5b3UgZ3V5cyB3b3VsZCBiZSB1c2lu
ZyBhIGdvb2QgcGFyYWxsZWwgZmlsZSBzeXN0ZW0uICBJbiB0aGF0IGNhc2UgYSBNL1Igam9iIHdp
bGwgYmUgZ3JlYXQgZm9yIG1vdmluZyB0aGUgZGF0YSAtLSBhcyBsb25nIGFzIHlvdSBkb24ndCBv
dmVybG9hZCB0aGUgbmV0d29yay4gIEFuZCBpZiB5b3UgYXJlIGdvaW5nIHRvIGhhdmUgdGhlIGRh
dGEgZW5kIHVwIGluIEhCYXNlIHlvdSBtYXkganVzdCB3cml0ZSBhIG1hcCBqb2IgdG8gZGlyZWN0
bHkgaW5zZXJ0IGludG8gaGJhc2UsIGVpdGhlciB0aHJvdWdoIHN0YW5kYXJkIGluc2VydHMgb3Ig
YnVsayBsb2FkcyAod2hpY2ggc2hvdWxkIGJlIDEweCBmYXN0ZXIpLg0KDQpJZiB5b3UgaGF2ZSB0
aW1lIHRvIHBsYXksIGl0IG1pZ2h0IGJlIGludGVyZXN0aW5nIHRvIHNlZSBob3cgaGJhc2UgcnVu
cyBvdmVyIHlvdXIgY3VycmVudCBmaWxlIHN5c3RlbS4gIEp1c3QgdXNlIHRoZSBmaWxlOi8vPHBh
dGggdG8gc2hhcmVkIGRpcmVjdG9yeT4gaW5zdGVhZCBvZiB0aGUgaGRmczovLyB1cmwgaW4gdGhl
IGhiYXNlLXNpdGUueG1sIGZpbGUuICBJIHdvdWxkIHN0cmVzcyB0ZXN0IHRoYXQgZmlyc3Qgd2l0
aCBib3RoIGhiYXNlIGFuZCBtL3Igam9icyBqdXN0IHRvIG1ha2Ugc3VyZSB0aGF0IGl0IGJlaGF2
ZXMgd2VsbCwgYnV0IHRoZXJlIHdvdWxkIGJlIG1hbnkgcGVvcGxlIHdobyB3b3VsZCBsb3ZlIHRv
IGhlYXIgYWJvdXQgeW91ciBleHBlcmllbmNlIGhlcmUuICBJIGZvciBvbmUgd291bGQgbGlrZSB0
byBrbm93IGlmIEkgY2FuIHJ1biBoYmFzZSBvdmVyIGx1c3RlciBzaW5jZSB3ZSBsb3ZlIGx1c3Rl
ciBhcm91bmQgaGVyZS4NCg0KRGF2ZQ0KDQotLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KRnJv
bTogS2V2aW4gRm94IFttYWlsdG86S2V2aW4uRm94QHBubC5nb3ZdIA0KU2VudDogTW9uZGF5LCBK
YW51YXJ5IDAzLCAyMDExIDI6NTggUE0NClRvOiBCdXR0bGVyLCBEYXZpZA0KQ2M6IHVzZXJAaGJh
c2UuYXBhY2hlLm9yZzsgJ2dlbmVyYWxAaGFkb29wLmFwYWNoZS5vcmcnOyBCcm93biwgRGF2aWQg
TSBKUg0KU3ViamVjdDogUkU6IFdoYXQgaXMgdGhlIGZhc3Rlc3Qgd2F5IHRvIGdldCBhIGxhcmdl
IGFtb3VudCBvZiBkYXRhIGludG8gdGhlIEhhZG9vcCBIREZTIGZpbGUgc3lzdGVtIChvciBIYmFz
ZSk/DQoNCk9uIE1vbiwgMjAxMS0wMS0wMyBhdCAxMjoyMCAtMDgwMCwgQnV0dGxlciwgRGF2aWQg
d3JvdGU6DQo+IEhpIFJvbiwNCj4gTG9hZGluZyBpbnRvIEhERlMgYW5kIEhCYXNlIGFyZSB0d28g
ZGlmZmVyZW50IGlzc3Vlcy4gIA0KPiANCj4gSERGUzogaWYgeW91IGhhdmUgYSBsYXJnZSBudW1i
ZXIgb2YgZmlsZXMgdG8gbG9hZCBmcm9tIHlvdXIgbmZzIGZpbGUgc3lzdGVtIGludG8gSERGUyBp
dCBpcyBub3QgY2xlYXIgdGhhdCBwYXJhbGxlbGl6aW5nIHRoZSBsb2FkIHdpbGwgaGVscC4gDQoN
Ckl0cyBub3QgbmZzLiBJdHMgYSBwYXJhbGxlbCBmaWxlIHN5c3RlbS4NCg0KPiAgWW91IGhhdmUg
dHdvIHNvdXJjZXMgb2YgYm90dGxlbmVja3M6IHRoZSBuZnMgZmlsZSBzeXN0ZW0gYW5kIHRoZSBI
REZTIGZpbGUgc3lzdGVtLiAgSW4geW91ciBwYXJhbGxlbCBleGFtcGxlLCB5b3Ugd2lsbCBsaWtl
bHkgc2F0dXJhdGUgeW91ciBuZnMgZmlsZSBzeXN0ZW0gZmlyc3QuDQoNClVubGlrZWx5IGluIHRo
aXMgY2FzZS4gV2UncmUgaW4gdGhlIHVudXN1YWwgcG9zaXRpb24gb2Ygb3VyIGFyY2hpdmUNCmNs
dXN0ZXIgYmVpbmcgZmFzdGVyIHRoZW4gb3VyIGhhZG9vcCBjbHVzdGVyLg0KDQo+ICAgSWYgdGhl
eSBhcmUgYWN0dWFsbHkgbG9jYWwgZmlsZXMsIHRoZW4gbG9hZGluZyB0aGVtIHZpYSBNL1IgaXMg
YSBub24tc3RhcnRlciBhcyB5b3UgaGF2ZSBubyBjb250cm9sIG92ZXIgd2hpY2ggbWFjaGluZSB3
aWxsIGdldCBhIG1hcCB0YXNrLg0KDQpJZiB0aGUgc2FtZSBmaWxlcyBhcmUgImxvY2FsIiBvbiBl
YWNoIG5vZGUsIGRvZXMgaXQgbWF0dGVyPyBTaG91bGRuJ3QNCnRoZSBtYXAgam9icyBhbGwgYmUg
c2NoZWR1bGVkIGluIGEgd2F5IGFzIHRvIHNwcmVhZCBvdXQgdGhlIGxvYWQ/DQoNClRoYW5rcywN
CktldmluDQoNCj4gICBVbmxlc3MgYWxsIG9mIHRoZSBtYWNoaW5lcyBoYXZlIGZpbGVzIGluIHRo
ZSBzYW1lIGRpcmVjdG9yeSBhbmQgeW91IGFyZSBqdXN0IGdvaW5nIHRvIGxvb2sgaW4gdGhhdCBk
aXJlY3RvcnkgdG8gdXBsb2FkLiAgVGhlbiwgaXQgc291bmRzIGxpa2UgbW9yZSBvZiBhIGpvYiBm
b3IgYSBwYXJhbGxlbCBzaGVsbCBjb21tYW5kIGFuZCBsZXNzIG9mIGEgbWFwL3JlZHVjZSBjb21t
YW5kLg0KPiANCj4gSEJhc2U6IFNvIGZhciBteSBzdHJhdGVneSBoYXMgYmVlbiB0byBnZXQgdGhl
IGZpbGVzIGludG8gSERGUyBmaXJzdCwgYW5kIHRoZW4gd3JpdGUgYSBNYXAgam9iIHRvIGxvYWQg
dGhlbSBpbnRvIEhCYXNlLiAgWW91IGNhbiB0cnkgdG8gZG8gdGhpcyBhbmQgc2VlIGlmIGRpcmVj
dCBpbnNlcnRzIGludG8gaGJhc2UgYXJlIGZhc3QgZW5vdWdoIGZvciB5b3VyIHVzZSBjYXNlLiAg
QnV0LCBpZiB5b3UgYXJlIGdvaW5nIHRvIFRCcy93ZWVrIHRoZW4geW91IHdpbGwgbGlrZWx5IHdh
bnQgdG8gaW52ZXN0aWdhdGUgdGhlIGJ1bGsgbG9hZCBmZWF0dXJlcy4gIEkgaGF2ZW4ndCB5ZXQg
aW5jb3Jwb3JhdGVkIHRoYXQgaW50byBteSB3b3JrZmxvdyBzbyBJIGNhbid0IG9mZmVyIG11Y2gg
YWR2aWNlIHRoZXJlLiBKdXN0IGJlIHN1cmUgeW91ciBjbHVzdGVyIGlzIHNpemVkIGFwcHJvcHJp
YXRlbHkuICBFLmcuLCB3aXRoIHlvdXIgY29tcHJlc3Npb24gdHVybmVkIG9uIGluIGhiYXNlLCBz
ZWUgaG93IG11Y2ggYSAxIEdCIGlucHV0IGZpbGUgZXhwYW5kcyB0byBpbnNpZGUgaGJhc2UgLyBo
ZGZzLiAgVGhhdCBzaG91bGQgZ2l2ZSB5b3UgYSBmZWVsaW5nIGZvciBob3cgbXVjaCBzcGFjZSB5
b3Ugd2lsbCBuZWVkIGZvciB5b3VyIGV4cGVjdGVkIGRhdGEgbG9hZC4NCj4gDQo+IERhdmUNCj4g
DQo+IA0KPiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBUYXlsb3IsIFJvbmFs
ZCBDIFttYWlsdG86cm9uYWxkLnRheWxvckBwbmwuZ292XSANCj4gU2VudDogVHVlc2RheSwgRGVj
ZW1iZXIgMjgsIDIwMTAgMjowNSBQTQ0KPiBUbzogJ3VzZXJAaGJhc2UuYXBhY2hlLm9yZyc7ICdn
ZW5lcmFsQGhhZG9vcC5hcGFjaGUub3JnJw0KPiBDYzogVGF5bG9yLCBSb25hbGQgQzsgRm94LCBL
ZXZpbiBNOyBCcm93biwgRGF2aWQgTSBKUg0KPiBTdWJqZWN0OiBXaGF0IGlzIHRoZSBmYXN0ZXN0
IHdheSB0byBnZXQgYSBsYXJnZSBhbW91bnQgb2YgZGF0YSBpbnRvIHRoZSBIYWRvb3AgSERGUyBm
aWxlIHN5c3RlbSAob3IgSGJhc2UpPw0KPiANCj4gDQo+IEZvbGtzLA0KPiANCj4gV2UgcGxhbiBv
biB1cGxvYWRpbmcgbGFyZ2UgYW1vdW50cyBvZiBkYXRhIG9uIGEgcmVndWxhciBiYXNpcyBvbnRv
IGEgSGFkb29wIGNsdXN0ZXIsIHdpdGggSGJhc2Ugb3BlcmF0aW5nIG9uIHRvcCBvZiBIYWRvb3Au
IEZpZ3VyZSBldmVudHVhbGx5IG9uIHRoZSBvcmRlciBvZiBtdWx0aXBsZSB0ZXJhYnl0ZXMgcGVy
IHdlZWsuIFNvIC0gd2UgYXJlIGNvbmNlcm5lZCBhYm91dCBkb2luZyB0aGUgdXBsb2FkcyB0aGVt
c2VsdmVzIGFzIGZhc3QgYXMgcG9zc2libGUgZnJvbSBvdXIgbmF0aXZlIExpbnV4IGZpbGUgc3lz
dGVtIGludG8gSERGUy4gRmlndXJlIGZpbGVzIHdpbGwgYmUgaW4sIHJvdWdobHksIHRoZSAxIHRv
IDMwMCBHQiByYW5nZS4gDQo+IA0KPiBPZmYgdGhlIHRvcCBvZiBteSBoZWFkLCBJJ20gdGhpbmtp
bmcgdGhhdCBkb2luZyB0aGlzIGluIHBhcmFsbGVsIHVzaW5nIGEgSmF2YSBNYXBSZWR1Y2UgcHJv
Z3JhbSB3b3VsZCB3b3JrIGZhc3Rlc3QuIFNvIG15IGlkZWEgd291bGQgYmUgdG8gaGF2ZSBhIGZp
bGUgbGlzdGluZyBhbGwgdGhlIGRhdGEgZmlsZXMgKGZ1bGwgcGF0aHMpIHRvIGJlIHVwbG9hZGVk
LCBvbmUgcGVyIGxpbmUsIGFuZCB0aGVuIHVzZSB0aGF0IGxpc3RpbmcgZmlsZSBhcyBpbnB1dCB0
byBhIE1hcFJlZHVjZSBwcm9ncmFtLiANCj4gDQo+IEVhY2ggTWFwcGVyIHdvdWxkIHRoZW4gdXBs
b2FkIG9uZSBvZiB0aGUgZGF0YSBmaWxlcyAodXNpbmcgImhhZG9vcCBmcyAtY29weUZyb21Mb2Nh
bCA8c291cmNlPiA8ZGVzdD4iKSBpbiBwYXJhbGxlbCB3aXRoIGFsbCB0aGUgb3RoZXIgTWFwcGVy
cywgd2l0aCB0aGUgTWFwcGVycyBvcGVyYXRpbmcgb24gYWxsIHRoZSBub2RlcyBvZiB0aGUgY2x1
c3Rlciwgc3ByZWFkaW5nIG91dCB0aGUgZmlsZSB1cGxvYWQgYWNyb3NzIHRoZSBub2Rlcy4NCj4g
DQo+IERvZXMgdGhhdCBzb3VuZCBsaWtlIGEgd2lzZSB3YXkgdG8gYXBwcm9hY2ggdGhpcz8gQXJl
IHRoZXJlIGJldHRlciBtZXRob2RzPyBBbnl0aGluZyBlbHNlIG91dCB0aGVyZSBmb3IgZG9pbmcg
YXV0b21hdGVkIHVwbG9hZCBpbiBwYXJhbGxlbD8gV2Ugd291bGQgdmVyeSBtdWNoIGFwcHJlY2lh
dGUgYWR2aWNlIGluIHRoaXMgYXJlYSwgc2luY2Ugd2UgYmVsaWV2ZSB1cGxvYWQgc3BlZWQgbWln
aHQgYmVjb21lIGEgYm90dGxlbmVjay4NCj4gDQo+ICAgLSBSb24gVGF5bG9yDQo+IA0KPiBfX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fDQo+IFJvbmFsZCBUYXlsb3Is
IFBoLkQuDQo+IENvbXB1dGF0aW9uYWwgQmlvbG9neSAmIEJpb2luZm9ybWF0aWNzIEdyb3VwDQo+
IA0KPiBQYWNpZmljIE5vcnRod2VzdCBOYXRpb25hbCBMYWJvcmF0b3J5DQo+IDkwMiBCYXR0ZWxs
ZSBCb3VsZXZhcmQNCj4gUC5PLiBCb3ggOTk5LCBNYWlsIFN0b3AgSjQtMzMNCj4gUmljaGxhbmQs
IFdBICA5OTM1MiBVU0ENCj4gT2ZmaWNlOiAgNTA5LTM3Mi02NTY4DQo+IEVtYWlsOiByb25hbGQu
dGF5bG9yQHBubC5nb3YNCj4gDQo+IA0KDQoNCg==

From general-return-2656-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 04 02:58:00 2011
Return-Path: <general-return-2656-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95540 invoked from network); 4 Jan 2011 02:58:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Jan 2011 02:58:00 -0000
Received: (qmail 23230 invoked by uid 500); 4 Jan 2011 02:57:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22587 invoked by uid 500); 4 Jan 2011 02:57:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22527 invoked by uid 99); 4 Jan 2011 02:57:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Jan 2011 02:57:55 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Jan 2011 02:57:48 +0000
Received: from [127.0.0.1] (vpn-client-104-6.eglbp.corp.yahoo.com [10.66.104.6])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p042ukO6043438;
	Mon, 3 Jan 2011 18:56:54 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1294109817;
	bh=Rqqop5Pu1Qbk0Uds3Ol///eO+Gi35+134xgxQ5skhxw=;
	h=Message-ID:Date:From:MIME-Version:To:Subject:Content-Type;
	b=CZGaaC2e/cw8jtH5edOcEq2qwHVSCuI5yYB9vjCaEy7YrubIMYzTkbOCRdN9NYoZv
	 eeGi9A2XgIwqtqeUU8r5bTuCjU3+fsKa03QMnEdwMG+6gjrqqd+SgxKR3i6Gi67CnV
	 SmFq6Ih3uQP49iHb5DM3j32E+vB6Gzxgf2hcDD7g=
Message-ID: <4D228C6D.5000507@yahoo-inc.com>
Date: Tue, 04 Jan 2011 08:26:45 +0530
From: Basant Verma <bverma@yahoo-inc.com>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.13) Gecko/20101207 Lightning/1.0b2 Thunderbird/3.1.7
MIME-Version: 1.0
To: common-user@hadoop.apache.org, user@pig.apache.org, user@hive.apache.org,
        general@hadoop.apache.org
Subject: Hadoop India Summit 2011 - Call for Papers now Open
Content-Type: multipart/alternative;
 boundary="------------030706090007090900010202"
X-Virus-Checked: Checked by ClamAV on apache.org

--------------030706090007090900010202
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi Hadoop enthusiasts,

Apache Hadoop has become the de-facto platform for developing 
large-scale data-intensive applications. It has been used actively in 
academia and Industry for research and data mining. Hadoop Summit 
provides an opportunity for understanding the latest trends and roadmap 
for Hadoop Platform and its ecosystem and how Hadoop is leveraged in 
various domains.

Yahoo! India R&D is proud to host second India Hadoop Summit which will 
take place in Bangalore, India on February 16th or 17th, 2011.

Please look at 
http://developer.yahoo.com/blogs/ydn/posts/2011/01/hadoop-india-summit-2011-call-for-papers/  
for more information information if you'd like to present in the summit. 
Stay tuned for summit registration info which we will publish in early 
January.

Best Regards,
Basant Verma
Yahoo India R&D, Hadoop

--------------030706090007090900010202--


From general-return-2657-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 04 22:49:03 2011
Return-Path: <general-return-2657-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25986 invoked from network); 4 Jan 2011 22:49:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Jan 2011 22:49:02 -0000
Received: (qmail 51920 invoked by uid 500); 4 Jan 2011 22:49:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51711 invoked by uid 500); 4 Jan 2011 22:49:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 59203 invoked by uid 99); 4 Jan 2011 22:11:25 -0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of iphulari@gmail.com designates 209.85.210.48 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=vNLwlS/41FzgFx7cHR7lH5C5VeAPlscc4jGPhcvefhE=;
        b=F4wi74maiJVi/qgvXW1P1jThgsC999RlLUSQeno+NVh15b3TfQqgAjKod90zXH2Qzl
         QJZbaUYPiC1YbeQqz107Z739/V4cpquK5WiVfB2bkkUyyHdKEh1jMJ/aD3/D6Yx0BgVb
         xb3eQs/RFJu/250/m5YbBF/RGpxcGXbG3k6a0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=mSpACDvcOE+62faBCJQ/lPoV9VckqGN1XdRdUusp2wELYicnnH8Lac2KSJ8aYy13sV
         FdK8dggP+pycAF1ntcOSB1mWf5Yz5x+VimU1DFOv75weIqIN2mfemf3OSGsXF1AwUpqY
         cNRbgGMuhi0wwqG2bRq0+amSEyX9AODZ5QRig=
MIME-Version: 1.0
Date: Tue, 4 Jan 2011 14:10:55 -0800
Message-ID: <AANLkTi=FP8cjr7MfmJsDdkR0Rohfer6KM7VQmO5CVTGB@mail.gmail.com>
Subject: HFTP based distcp failing.
From: Ravi Phulari <iphulari@gmail.com>
To: hdfs-user@hadoop.apache.org, user@hadoop.apache.org, 
	general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016367be957e6575804990c8ac8
X-Virus-Checked: Checked by ClamAV on apache.org

--0016367be957e6575804990c8ac8
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Hello Hadoopers,
  I need to distcp data across two clusters. For security reasons I can not
use hdfs based distcp.
HFTP based distcp is failing  with following Ioexception.

Stack trace.

Copy failed: java.io.IOException: Not supported
    at org.apache.hadoop.hdfs.HftpFileSystem.delete(HftpFileSystem.java:360=
)
    at org.apache.hadoop.tools.DistCp.fullyDelete(DistCp.java:939)
    at org.apache.hadoop.tools.DistCp.copy(DistCp.java:655)
    at org.apache.hadoop.tools.DistCp.run(DistCp.java:857)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
    at org.apache.hadoop.tools.DistCp.main(DistCp.java:884)

I am using following command for distcp.

Hadoop distcp  hftp://nn1.hadoop1:50070/data
hftp://nn2.hadoop2:50070/user/hadoop/
Hadoop distcp  /data/logs    hftp://nn2.hadoop2:50070/user/hadoop/

Any idea why this distcp could be failing.
I don=92t see any logs in JT and NN.

Any help will be greatly appreciated.

-
Thanks,
Ravi

--0016367be957e6575804990c8ac8--

From general-return-2658-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 04 22:57:54 2011
Return-Path: <general-return-2658-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31872 invoked from network); 4 Jan 2011 22:57:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Jan 2011 22:57:54 -0000
Received: (qmail 68783 invoked by uid 500); 4 Jan 2011 22:57:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68574 invoked by uid 500); 4 Jan 2011 22:57:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68559 invoked by uid 99); 4 Jan 2011 22:57:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Jan 2011 22:57:52 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of lars.george@gmail.com designates 209.85.215.176 as permitted sender)
Received: from [209.85.215.176] (HELO mail-ey0-f176.google.com) (209.85.215.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Jan 2011 22:57:44 +0000
Received: by eyz10 with SMTP id 10so6222735eyz.35
        for <multiple recipients>; Tue, 04 Jan 2011 14:57:24 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:cc:content-type
         :content-transfer-encoding;
        bh=5v/yT/d8HgWRRNvpN7kgZFG4Lj3C4zI3Lc+5WW1tkg0=;
        b=fn4zdla/ekDdQLvFdZgmJaKxVGE/tPMh317AHuZ+c2kw0PB3n0wi1a+rPFSoNlkigk
         7qVnOA4r9o3PUXU0dIOt1qgZsex5oN4zd1mr7cFLrvfOswGGDaDODEugtyjX7sSukBM8
         B/FVzA+QQ0Nf92bU+OO4dSReqdyfmV4dQSPPk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type:content-transfer-encoding;
        b=GhdBbcpuKUnjKESNkh4xRpcfD7TdUm+P4+RIJB5GfhKyse1s6b1RKIkb3QP1ji8G7Z
         aZxLuIZOz/3WFx1XrqnF533oq89Xvu8Oflfsb5ajMsq97OOJsaMPMiLzo+LJJ3ED5DZP
         35qgkFhS2plTDxmB2DE1TpAykN5LBCJG98Ot0=
MIME-Version: 1.0
Received: by 10.213.8.69 with SMTP id g5mr15144764ebg.70.1294181844111; Tue,
 04 Jan 2011 14:57:24 -0800 (PST)
Received: by 10.213.35.140 with HTTP; Tue, 4 Jan 2011 14:57:24 -0800 (PST)
In-Reply-To: <AANLkTi=FP8cjr7MfmJsDdkR0Rohfer6KM7VQmO5CVTGB@mail.gmail.com>
References: <AANLkTi=FP8cjr7MfmJsDdkR0Rohfer6KM7VQmO5CVTGB@mail.gmail.com>
Date: Tue, 4 Jan 2011 23:57:24 +0100
Message-ID: <AANLkTi=7FVx5Xef_KdASKXHPwJmEovaB7-9VjW-AUeD+@mail.gmail.com>
Subject: Re: HFTP based distcp failing.
From: Lars George <lars.george@gmail.com>
To: general@hadoop.apache.org
Cc: hdfs-user@hadoop.apache.org, user@hadoop.apache.org
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Ravi,

With distcp you usually run it on the target cluster and specify the
source with the "hftp://" URI and the target with "hdfs://". hftp is a
read only, http based access to the data.

Lars

On Tue, Jan 4, 2011 at 11:10 PM, Ravi Phulari <iphulari@gmail.com> wrote:
> Hello Hadoopers,
> =A0I need to distcp data across two clusters. For security reasons I can =
not
> use hdfs based distcp.
> HFTP based distcp is failing =A0with following Ioexception.
>
> Stack trace.
>
> Copy failed: java.io.IOException: Not supported
> =A0 =A0at org.apache.hadoop.hdfs.HftpFileSystem.delete(HftpFileSystem.jav=
a:360)
> =A0 =A0at org.apache.hadoop.tools.DistCp.fullyDelete(DistCp.java:939)
> =A0 =A0at org.apache.hadoop.tools.DistCp.copy(DistCp.java:655)
> =A0 =A0at org.apache.hadoop.tools.DistCp.run(DistCp.java:857)
> =A0 =A0at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> =A0 =A0at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> =A0 =A0at org.apache.hadoop.tools.DistCp.main(DistCp.java:884)
>
> I am using following command for distcp.
>
> Hadoop distcp =A0hftp://nn1.hadoop1:50070/data
> hftp://nn2.hadoop2:50070/user/hadoop/
> Hadoop distcp =A0/data/logs =A0 =A0hftp://nn2.hadoop2:50070/user/hadoop/
>
> Any idea why this distcp could be failing.
> I don=92t see any logs in JT and NN.
>
> Any help will be greatly appreciated.
>
> -
> Thanks,
> Ravi
>

From general-return-2659-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 05 04:12:29 2011
Return-Path: <general-return-2659-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 64016 invoked from network); 5 Jan 2011 04:12:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Jan 2011 04:12:28 -0000
Received: (qmail 51579 invoked by uid 500); 5 Jan 2011 04:12:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51272 invoked by uid 500); 5 Jan 2011 04:12:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51264 invoked by uid 99); 5 Jan 2011 04:12:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jan 2011 04:12:24 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of venkatesh.kowluru@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jan 2011 04:12:16 +0000
Received: by wye20 with SMTP id 20so15851369wye.35
        for <general@hadoop.apache.org>; Tue, 04 Jan 2011 20:11:56 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=Ac5eOt+qcu4kqTsoAR/gYKlU9car4r5grFnTKmf0tDY=;
        b=V4KFhGz+M7/P7vf7Jxk0hoZj91VK4eSviouH95wtAQT9Vtjz2N2HZtIQYGqo69yBtj
         +Xwtgk3KdzW/qG+fsF8IDKIqhKuIc91c2ubsri18J+9LwueJHL7ko64n7peiGoI9VBEm
         bZXglegj8rYjZxtpns9cimaKSuUd8SVWXRySA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=KPicMdnkAzyeZ6VGSB3PRQJ1d+zMAwwVB+XJfRoue4C+K07VVgVOBgK+Z0i1YTV5ba
         blpxXEF56i93dRMhjmDbrkrXGu3AeVDhtLLvS30btuz6+tPnKoYVLxHh9VcVxFY4G7OI
         1nSrncBXHSJX+mES5NvWJ2oBGOF6lGTUt/y70=
MIME-Version: 1.0
Received: by 10.227.98.94 with SMTP id p30mr4634444wbn.152.1294200716188; Tue,
 04 Jan 2011 20:11:56 -0800 (PST)
Received: by 10.227.151.129 with HTTP; Tue, 4 Jan 2011 20:11:56 -0800 (PST)
In-Reply-To: <1294200654.50927.ezmlm@hadoop.apache.org>
References: <1294200654.50927.ezmlm@hadoop.apache.org>
Date: Tue, 4 Jan 2011 20:11:56 -0800
Message-ID: <AANLkTik_DVhnYYJhA_Es9LejjUnkrDuuORPki=mJdXO_@mail.gmail.com>
Subject: Re: Already subscribed to general@hadoop.apache.org
From: venkatesh kavuluri <venkatesh.kowluru@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd30782f4dddf04991195bd
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd30782f4dddf04991195bd
Content-Type: text/plain; charset=ISO-8859-1

Hi All,

I am trying to override the TextOutputFormat class default key-value
separator ("\t") with a control character using the configuration parameter
below.

conf.set("mapred.textoutputformat.separator", "\u0008"); and also like
conf.set("mapred.textoutputformat.separator", "\b");

I get the error below,

[Fatal Error] :40:68: Character reference "&#8" is an invalid XML character.
10/12/22 14:39:13 FATAL conf.Configuration: error parsing conf file:
org.xml.sax.SAXParseException: Character reference "&#8" is an invalid XML
character.
Exception in thread "main" java.lang.RuntimeException:
org.xml.sax.SAXParseException: Character reference "&#8" is an invalid XML
character.
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1168)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1040)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:980)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:382)
at
org.apache.hadoop.mapred.JobConf.checkAndWarnDeprecation(JobConf.java:1662)
at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:215)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:93)
at
org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:373)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:800)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:432)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:447)
at com.ebay.ewa2.aggregation.BotVisits.run(BotVisits.java:181)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at com.ebay.ewa2.aggregation.BotVisits.main(BotVisits.java:185)
Caused by: org.xml.sax.SAXParseException: Character reference "&#8" is an
invalid XML character.
at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
at javax.xml.parsers.DocumentBuilder.parse(Unknown Source)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1092)
... 14 more

Looks like the jobConf object is not being deserialized properly. Is there
any work around without needing to override the TextOutputFormat's
RecordWriter method. I am using Hadoop 0.20.2 API.

Thanks,
Venkatesh

--000e0cd30782f4dddf04991195bd--

From general-return-2660-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 05 06:37:38 2011
Return-Path: <general-return-2660-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35842 invoked from network); 5 Jan 2011 06:37:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Jan 2011 06:37:37 -0000
Received: (qmail 56087 invoked by uid 500); 5 Jan 2011 06:37:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55747 invoked by uid 500); 5 Jan 2011 06:37:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55733 invoked by uid 99); 5 Jan 2011 06:37:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jan 2011 06:37:35 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jan 2011 06:37:28 +0000
Received: by fxm2 with SMTP id 2so14455789fxm.35
        for <general@hadoop.apache.org>; Tue, 04 Jan 2011 22:37:08 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type;
        bh=+lpsTd2Z8O7uHOq/+m6C8YLu0Xtf5Pwg/H9xPfqlkyo=;
        b=qgDyCqvUPeczRbmUYKulJf5WDYZEpzuCD0hiyhWLgwaM+zEs742nvaf4rSZ6b6Hbkv
         pyvFYulsqWU84E+Qy8TpHl11lalFrYU6aTdMVTfFVrU4mh0+imiEPZT+ktlBRjXTnsCP
         SGdrtOBIHKh6CK065VExPgssavEkRPEQIJXKo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=VjhJAgzMsqmKNJWVQTGq8bfuHwLhwvOqjxgRngGS13Uf8yCT/Se08Ck+HynnK2hhA3
         xySp3Pnp7d+ql4JxvxAsIZxxwrMY7z/8Sz37GYECHVm4J8mO6qwYRsGZOL8WxPu/Fs7l
         YNUSabQ53sDFpk0C8F8d9orSrEtsq85jFnv98=
Received: by 10.223.124.129 with SMTP id u1mr494636far.112.1294209428439; Tue,
 04 Jan 2011 22:37:08 -0800 (PST)
MIME-Version: 1.0
Received: by 10.223.120.14 with HTTP; Tue, 4 Jan 2011 22:36:48 -0800 (PST)
In-Reply-To: <AANLkTik_DVhnYYJhA_Es9LejjUnkrDuuORPki=mJdXO_@mail.gmail.com>
References: <1294200654.50927.ezmlm@hadoop.apache.org> <AANLkTik_DVhnYYJhA_Es9LejjUnkrDuuORPki=mJdXO_@mail.gmail.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Wed, 5 Jan 2011 12:06:48 +0530
Message-ID: <AANLkTimbToUo7M+BGzPv6qH37JEOsbceypaQNkZF5h4y@mail.gmail.com>
Subject: Re: Already subscribed to general@hadoop.apache.org
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

I don't think the XML spec (the serialized configuration for a Job, is
in XML) accepts a backspace character. You should add an escaped
version of it instead.

http://www.w3.org/TR/REC-xml/#charsets

-- 
Harsh J
www.harshj.com

From general-return-2661-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 05 11:56:24 2011
Return-Path: <general-return-2661-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96727 invoked from network); 5 Jan 2011 11:56:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Jan 2011 11:56:24 -0000
Received: (qmail 96328 invoked by uid 500); 5 Jan 2011 11:56:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96077 invoked by uid 500); 5 Jan 2011 11:56:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96069 invoked by uid 99); 5 Jan 2011 11:56:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jan 2011 11:56:21 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of magawake@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jan 2011 11:56:15 +0000
Received: by qwh6 with SMTP id 6so16155817qwh.35
        for <general@hadoop.apache.org>; Wed, 05 Jan 2011 03:55:54 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=OBatY5gAcDgAMiQmKCfwTd8AYLr4lWmc2ZAX8CfueoA=;
        b=c0f1ctiJVYcYyUkHHrt6MQWQwJwyGCrRJKk59LvYiV70nPq/lBQw1ZY7KM5YcFZBUm
         77RdowIiKpBsOjWGuH8jSi0v3kgUJe51I+3GnHlk+0DdiQ2AwmsyAyJTXzjGVTLE2ghD
         rA75Jj0yOPKyzalRIetJvlXBU3UhELF/vB/DQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=gS2wOy6u8c0s1/lVHTEh0VfFFXDGFlzIGgk18zZbDFLX/iG4d/hsC3T1Zxjhee6OOe
         QsKPXcWB79IsUnIbjEl6bMjgesLoGe+tPrWF8rK08PQ7EZ2Z32dfg1kp8tid4i6s3K2A
         MR29aCK3vsEeUH8w3OeZdXdoAE55pII/W6KCk=
MIME-Version: 1.0
Received: by 10.229.219.136 with SMTP id hu8mr8044278qcb.249.1294228554252;
 Wed, 05 Jan 2011 03:55:54 -0800 (PST)
Received: by 10.220.118.20 with HTTP; Wed, 5 Jan 2011 03:55:54 -0800 (PST)
Date: Wed, 5 Jan 2011 06:55:54 -0500
Message-ID: <AANLkTikN5ZkqMN9NrDgq_ohB6L_Qo_4TTjAiOzuS=bPt@mail.gmail.com>
Subject: decommission a rack
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8

Using 0.21

I have a server which was in rack3. It was the only server in this
rack so I decommissioned the server. fsck shows only 2 racks however
topology shows 3 racks. How can I fix this?

From general-return-2662-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 05 15:02:21 2011
Return-Path: <general-return-2662-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84388 invoked from network); 5 Jan 2011 15:02:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Jan 2011 15:02:21 -0000
Received: (qmail 43126 invoked by uid 500); 5 Jan 2011 15:02:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42930 invoked by uid 500); 5 Jan 2011 15:02:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42918 invoked by uid 99); 5 Jan 2011 15:02:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jan 2011 15:02:18 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of msegel@navteq.com designates 65.167.11.240 as permitted sender)
Received: from [65.167.11.240] (HELO xmailchi.navteq.com) (65.167.11.240)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jan 2011 15:02:10 +0000
Received: from chicago-spamfw.navteq.com (imailchi [10.8.5.161])
	by xmailchi.navteq.com (8.13.8/8.13.6) with ESMTP id p05EVsbS021294
	for <general@hadoop.apache.org>; Wed, 5 Jan 2011 08:31:54 -0600
Received: from imailchi.hq.navteq.com (localhost [127.0.0.1])
	by chicago-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id C6883432DA74
	for <general@hadoop.apache.org>; Wed,  5 Jan 2011 09:01:48 -0600 (CST)
Received: from imailchi.hq.navteq.com (imailchi.hq.navteq.com [10.8.120.104]) by chicago-spamfw.navteq.com with ESMTP id wOhlRRpdCNEWzi5s for <general@hadoop.apache.org>; Wed, 05 Jan 2011 09:01:48 -0600 (CST)
Received: from hq-ex-ht01.ad.navteq.com (hq-ex-ht01.ad.navteq.com [10.8.222.51])
	by imailchi.hq.navteq.com (8.12.9/8.12.9) with ESMTP id p05F1mtf005405
	for <general@hadoop.apache.org>; Wed, 5 Jan 2011 09:01:48 -0600
Received: from hq-ex-mb02.ad.navteq.com ([fe80::880d:318e:e860:a8db]) by
 hq-ex-ht01.ad.navteq.com ([fe80::55a6:5b94:5c00:fe39%12]) with mapi; Wed, 5
 Jan 2011 09:01:48 -0600
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 5 Jan 2011 09:01:47 -0600
Subject: RE: decommission a rack
Thread-Topic: decommission a rack
Thread-Index: Acusz5Q8xTAoxjaMRzWQNq+Rh/T30AAGGEfQ
Message-ID: <ABC24175AFD3BE4DA15F4CD375ED413D063B0BD673@hq-ex-mb02.ad.navteq.com>
References: <AANLkTikN5ZkqMN9NrDgq_ohB6L_Qo_4TTjAiOzuS=bPt@mail.gmail.com>
In-Reply-To: <AANLkTikN5ZkqMN9NrDgq_ohB6L_Qo_4TTjAiOzuS=bPt@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1
X-Virus-Checked: Checked by ClamAV on apache.org

Change your topology script?

One thing about rack awareness. You never want to have an unbalanced configuration. The caveat is if your uplinks between the ToR Switch  are 'fat' enough. (4 10GBe should do it.)

If you watched your system with the one machine on a separate rack (assuming it was also on a separate switch) you'd see a major network bottle neck. (We use Ganglia)

If you're seeing the rack count wrong, you could just bounce the cloud and it should show only two.

Note: Deprecating a machine can take a long time. You may want to consider just killing the node, waiting till the cluster recognizes that its down and I believe just kicking off an FSCK or maybe running the balancer program will clean that up faster than deprecating. (YMMV)

HTH

-Mike


-----Original Message-----
From: Mag Gam [mailto:magawake@gmail.com] 
Sent: Wednesday, January 05, 2011 5:56 AM
To: general@hadoop.apache.org
Subject: decommission a rack

Using 0.21

I have a server which was in rack3. It was the only server in this
rack so I decommissioned the server. fsck shows only 2 racks however
topology shows 3 racks. How can I fix this?


The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above.  If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited.  If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.

From general-return-2663-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 05 22:46:15 2011
Return-Path: <general-return-2663-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25907 invoked from network); 5 Jan 2011 22:46:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Jan 2011 22:46:14 -0000
Received: (qmail 36553 invoked by uid 500); 5 Jan 2011 22:46:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36488 invoked by uid 500); 5 Jan 2011 22:46:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36480 invoked by uid 99); 5 Jan 2011 22:46:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jan 2011 22:46:12 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jan 2011 22:46:08 +0000
Received: by iyb26 with SMTP id 26so17071602iyb.35
        for <general@hadoop.apache.org>; Wed, 05 Jan 2011 14:45:47 -0800 (PST)
Received: by 10.231.59.149 with SMTP id l21mr23489582ibh.196.1294267547346;
 Wed, 05 Jan 2011 14:45:47 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.115.8 with HTTP; Wed, 5 Jan 2011 14:45:26 -0800 (PST)
In-Reply-To: <7B140F50-27E0-47C7-8EF8-B897D26CEE49@mac.com>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com> <20101020195420.GG2075@tp>
 <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com> <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
 <7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com> <AANLkTimLXA+CvUs-09LMN3ob6GZqmhxiyrKO7pzWEfse@mail.gmail.com>
 <4CE46119.2030509@yahoo-inc.com> <8617BECB-78B4-42B6-B592-D7FC1F8DA923@mac.com>
 <4CE47C88.5050203@yahoo-inc.com> <AANLkTinTM5h-3NupRSsFh-DnyvFYw+_OsXpASxcQUSDn@mail.gmail.com>
 <AANLkTimSOxaXZi-VLjwuTB5Ch5tRt0bKozc5L+Nx=ZrQ@mail.gmail.com>
 <AANLkTineU2v=Sx-=oC=6ROt6Os8TEbnS3L8xz1ZkoxCc@mail.gmail.com>
 <AANLkTin+oucXdCdOo7LQTtqndTf1axiT0gQntiOZ_mQJ@mail.gmail.com>
 <AANLkTinGn4Pb6_LkrY4-kK56PdHX4C1Zfu7A2PVfUoub@mail.gmail.com>
 <41FB0800-3703-49C1-8069-DEB74FFE6FAC@mac.com> <AANLkTik_rNLMHj3JT6Ekdop-NU_qzcB=VdsX+k4uFBtf@mail.gmail.com>
 <AANLkTin9=FPxv+0EydK4mWp_1zJbgGTddevvF9MrHW80@mail.gmail.com>
 <AANLkTiku_OvZuuOMWuXMcVf4UxB+W-SfW1WWV6rU0cXB@mail.gmail.com> <7B140F50-27E0-47C7-8EF8-B897D26CEE49@mac.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Wed, 5 Jan 2011 14:45:26 -0800
Message-ID: <AANLkTi=9wEVmjZqXN+Cw2+cW3iNfuMsUJG=OeXnMfg0f@mail.gmail.com>
Subject: Re: Patch testing
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485ebea386769b4049921253d

--001485ebea386769b4049921253d
Content-Type: text/plain; charset=ISO-8859-1

Hi Nigel,

MAPREDUCE-2172 has been fixed for a while. Are there any other particular
JIRAs you think need to be fixed before the MR test-patch queue gets
enabled? I have a lot of outstanding patches and doing all the test-patch
turnaround manually on 3 different boxes is a real headache.

Thanks
-Todd

On Tue, Dec 21, 2010 at 1:33 PM, Nigel Daley <ndaley@mac.com> wrote:

> Ok, HDFS is now enabled.  You'll see a stream of updates shortly on the ~30
> Patch Available HDFS issues.
>
> Nige
>
> On Dec 20, 2010, at 12:42 PM, Jakob Homan wrote:
>
> > I committed HDFS-1511 this morning.  We should be good to go.  I can
> > haz snooty robot butler?
> >
> > On Fri, Dec 17, 2010 at 8:31 PM, Konstantin Boudnik <cos@apache.org>
> wrote:
> >> Thanks Jacob. I am wasted already but I can do it on Sun, I think,
> >> unless it is done earlier.
> >> --
> >>   Take care,
> >> Konstantin (Cos) Boudnik
> >>
> >>
> >>
> >> On Fri, Dec 17, 2010 at 19:41, Jakob Homan <jghoman@gmail.com> wrote:
> >>> Ok.  I'll get a patch out for 1511 tomorrow, unless someone wants to
> >>> whip one up tonight.
> >>>
> >>>
> >>> On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley <ndaley@mac.com> wrote:
> >>>> I agree with Cos on fixing HDFS-1511 first. Once that is done I'll
> enable hdfs patch testing.
> >>>>
> >>>> Cheers,
> >>>> Nige
> >>>>
> >>>> Sent from my iPhone4
> >>>>
> >>>> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <cos@apache.org>
> wrote:
> >>>>
> >>>>> One more issue needs to be addressed before test-patch is turned on
> HDFS is
> >>>>>  https://issues.apache.org/jira/browse/HDFS-1511
> >>>>> --
> >>>>>   Take care,
> >>>>> Konstantin (Cos) Boudnik
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <cos@apache.org>
> wrote:
> >>>>>> Considering that because of these 4 faulty cases every patch will be
> >>>>>> -1'ed a patch author will still have to look at it and make a
> comment
> >>>>>> why this particular -1 isn't valid. Lesser work, perhaps, but
> messier
> >>>>>> IMO. I'm not blocking it - I just feel like there's a better way.
> >>>>>>
> >>>>>> --
> >>>>>>   Take care,
> >>>>>> Konstantin (Cos) Boudnik
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jghoman@gmail.com>
> wrote:
> >>>>>>>> If HDFS is added to the test-patch queue right now we get
> >>>>>>>> nothing but dozens of -1'ed patches.
> >>>>>>> There aren't dozens of patches being submitted currently.  The -1
> >>>>>>> isn't the important thing, it's the grunt work of actually running
> >>>>>>> (and waiting) for the tests, test-patch, etc. that Hudson does so
> that
> >>>>>>> the developer doesn't have to.
> >>>>>>>
> >>>>>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <
> dhruba@gmail.com> wrote:
> >>>>>>>> +1, thanks for doing this.
> >>>>>>>>
> >>>>>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jghoman@gmail.com>
> wrote:
> >>>>>>>>
> >>>>>>>>> So, with test-patch updated to show the failing tests, saving the
> >>>>>>>>> developers the need to go and verify that the failed tests are
> all
> >>>>>>>>> known, how do people feel about turning on test-patch again for
> HDFS
> >>>>>>>>> and mapred?  I think it'll help prevent any more tests from
> entering
> >>>>>>>>> the "yeah, we know" category.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> jg
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <
> jhoman@yahoo-inc.com> wrote:
> >>>>>>>>>> True, each patch would get a -1 and the failing tests would need
> to be
> >>>>>>>>>> verified as those known bad (BTW, it would be great if Hudson
> could list
> >>>>>>>>>> which tests failed in the message it posts to JIRA).  But that's
> still
> >>>>>>>>> quite
> >>>>>>>>>> a bit less error-prone work than if the developer runs the tests
> and
> >>>>>>>>>> test-patch themselves.  Also, with 22 being cut, there are a lot
> of
> >>>>>>>>> patches
> >>>>>>>>>> up in the air and several developers are juggling multiple
> patches.  The
> >>>>>>>>>> more automation we can have, even if it's not perfect, will
> decrease
> >>>>>>>>> errors
> >>>>>>>>>> we may make.
> >>>>>>>>>> -jg
> >>>>>>>>>>
> >>>>>>>>>> Nigel Daley wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>>> It's also ready to run on MapReduce and HDFS but we won't
> turn it on
> >>>>>>>>>>>>> until these projects build and test cleanly.  Looks like both
> these
> >>>>>>>>> projects
> >>>>>>>>>>>>> currently have test failures.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Assuming the projects are compiling and building, is there a
> reason to
> >>>>>>>>>>>> not turn it on despite the test failures? Hudson is invaluable
> to
> >>>>>>>>> developers
> >>>>>>>>>>>> who then don't have to run the tests and test-patch
> themselves.  We
> >>>>>>>>> didn't
> >>>>>>>>>>>> turn Hudson off when it was working previously and there were
> known
> >>>>>>>>>>>> failures.  I think one of the reasons we have more failing
> tests now is
> >>>>>>>>> the
> >>>>>>>>>>>> higher cost of doing Hudson's work (not a great excuse I
> know).  This
> >>>>>>>>> is
> >>>>>>>>>>>> particularly true now because several of the failing tests
> involve
> >>>>>>>>> tests
> >>>>>>>>>>>> timing out, making the whole testing regime even longer.
> >>>>>>>>>>>
> >>>>>>>>>>> Every single patch would get a -1 and need investigation.
>  Currently,
> >>>>>>>>> that
> >>>>>>>>>>> would be about 83 investigations between MR and HDFS issues
> that are in
> >>>>>>>>>>> patch available state.  Shouldn't we focus on getting these
> tests fixed
> >>>>>>>>> or
> >>>>>>>>>>> removed/?  Also, I need to get MAPREDUCE-2172 fixed (applies to
> HDFS as
> >>>>>>>>>>> well) before I turn this on.
> >>>>>>>>>>>
> >>>>>>>>>>> Cheers,
> >>>>>>>>>>> Nige
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Connect to me at http://www.facebook.com/dhruba
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>
> >>
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

--001485ebea386769b4049921253d--

From general-return-2664-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 05 23:59:29 2011
Return-Path: <general-return-2664-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57028 invoked from network); 5 Jan 2011 23:59:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Jan 2011 23:59:28 -0000
Received: (qmail 30751 invoked by uid 500); 5 Jan 2011 23:59:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30492 invoked by uid 500); 5 Jan 2011 23:59:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30476 invoked by uid 99); 5 Jan 2011 23:59:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jan 2011 23:59:25 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jan 2011 23:59:16 +0000
Received: by vws18 with SMTP id 18so6268172vws.35
        for <multiple recipients>; Wed, 05 Jan 2011 15:58:55 -0800 (PST)
Received: by 10.220.191.2 with SMTP id dk2mr208774vcb.251.1294271934165;
        Wed, 05 Jan 2011 15:58:54 -0800 (PST)
Received: from [10.172.5.187] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id d14sm5123889vcx.23.2011.01.05.15.58.50
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Wed, 05 Jan 2011 15:58:52 -0800 (PST)
Subject: Re: svn commit: r1055684 - /hadoop/common/branches/branch-0.20/CHANGES.txt
Mime-Version: 1.0 (Apple Message framework v1082)
Content-Type: text/plain; charset=us-ascii
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <20110105231249.66E8023889E0@eris.apache.org>
Date: Thu, 6 Jan 2011 10:58:46 +1100
Cc: general@hadoop.apache.org
Content-Transfer-Encoding: quoted-printable
Message-Id: <DC2DBF18-1437-4330-A035-9EDCDF288ADA@Holsman.NET>
References: <20110105231249.66E8023889E0@eris.apache.org>
To: common-dev@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

Is 20.3 a 'dead' release ?

I haven't seen any discussion of this on the apache lists about creating =
a 20.3 release, and kind of goes against all the discussion that we =
recently had with StAck about creating a 'append' release on 0.20.


I'm not against 20.3, but I would like to see some discussion and not =
have things reverted out of it without discussion.



On Jan 6, 2011, at 10:12 AM, omalley@apache.org wrote:

> Author: omalley
> Date: Wed Jan  5 23:12:49 2011
> New Revision: 1055684
>=20
> URL: http://svn.apache.org/viewvc?rev=3D1055684&view=3Drev
> Log:
> HADOOP-1734. Move to release 0.20.4 since I already made the tag =
0.20.3.
>=20
> Modified:
>    hadoop/common/branches/branch-0.20/CHANGES.txt
>=20
> Modified: hadoop/common/branches/branch-0.20/CHANGES.txt
> URL: =
http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20/CHANGES.tx=
t?rev=3D1055684&r1=3D1055683&r2=3D1055684&view=3Ddiff
> =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D
> --- hadoop/common/branches/branch-0.20/CHANGES.txt (original)
> +++ hadoop/common/branches/branch-0.20/CHANGES.txt Wed Jan  5 23:12:49 =
2011
> @@ -8,6 +8,9 @@ Release 0.20.4 - Unreleased
>=20
>   IMPROVEMENTS
>=20
> +    MAPREDUCE-1734. Un-deprecate the old MapReduce API in the 0.20 =
branch.
> +    (todd)
> +
> Release 0.20.3 - 2011-1-5
>=20
>   NEW FEATURES
> @@ -89,9 +92,6 @@ Release 0.20.3 - 2011-1-5
>=20
>     MAPREDUCE-1832. Allow file sizes less than 1MB in DFSIO benchmark. =
(shv)
>=20
> -    MAPREDUCE-1734. Un-deprecate the old MapReduce API in the 0.20 =
branch.
> -    (todd)
> -
> Release 0.20.2 - 2010-2-19
>=20
>   NEW FEATURES
>=20
>=20


From general-return-2665-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 06 00:13:50 2011
Return-Path: <general-return-2665-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67894 invoked from network); 6 Jan 2011 00:13:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2011 00:13:49 -0000
Received: (qmail 40909 invoked by uid 500); 6 Jan 2011 00:13:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40846 invoked by uid 500); 6 Jan 2011 00:13:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40838 invoked by uid 99); 6 Jan 2011 00:13:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 00:13:48 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.97 as permitted sender)
Received: from [17.148.16.97] (HELO asmtpout022.mac.com) (17.148.16.97)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 00:13:39 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp022.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LEK003J0R9HCV10@asmtp022.mac.com>
 for general@hadoop.apache.org; Wed, 05 Jan 2011 16:12:54 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-05_12:2011-01-06,2011-01-05,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101050128
From: Nigel Daley <ndaley@mac.com>
Subject: Hadoop 0.22 Release Preparation
Date: Wed, 05 Jan 2011 16:12:53 -0800
Message-id: <741040A1-C034-4BF9-9522-5C4E377EE6CC@mac.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

Folks,

Happy New Year!  As Hadoop 0.22 release manager I'd like to start moving towards a release.

There are 2 main lists of issues that I'd like to review with folks.  I propose doing this on the Hadoop IRC channel, reviewing issues one-on-one with issue assignee (or reporter if no assignee) at a designated time.  Of course anyone else is free to chime in during the IRC discussion.  Decisions and rationale will be recorded on the respective Jiras.  The 2 lists are:

1) All COMMON, HDFS, and MAPREDUCE issues that are currently marked BLOCKER against any release:
http://tinyurl.com/2a4sh6s
The goal is to correctly set FIX VERSION on these issues and try to assign most of these issues.

2) All COMMON, HDFS, and MAPREDUCE issues that are BUGS and CRITICAL against any release:
http://tinyurl.com/29w9bxw
The goal is to assess whether the issues should be moved to BLOCKER state for Hadoop 0.22

If you have other BUG issues that you believe are blockers but are not in the BLOCKER or CRITICAL Jira state, then please mark them as such before this review occurs so that they can be evaluated.  ***I will be removing the 0.22 FIX VERSION for all non-blocker issues before these reviews start.***

As always, test and doc IMPROVEMENTS are always eligible for commit to 0.22.  They should be marked as BLOCKER with FIX VERSION set to 0.22.  I may downgrade them from BLOCKER once we have the above reviews complete.

I know this is a volunteer effort, but it doesn't hurt to have some goals.  I'd like to start the review of these issues early next week and complete the review by Jan 21.  I'd love to see blockers fixed by Feb 21 and have our first release candidate shortly thereafter.  I'll try to coordinate test resources closer to that time.

Thoughts on the above?

Thanks,
Nige



From general-return-2666-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 06 00:20:01 2011
Return-Path: <general-return-2666-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69231 invoked from network); 6 Jan 2011 00:20:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2011 00:20:00 -0000
Received: (qmail 46941 invoked by uid 500); 6 Jan 2011 00:19:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46900 invoked by uid 500); 6 Jan 2011 00:19:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46891 invoked by uid 99); 6 Jan 2011 00:19:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 00:19:58 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.102 as permitted sender)
Received: from [17.148.16.102] (HELO asmtpout027.mac.com) (17.148.16.102)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 00:19:50 +0000
MIME-version: 1.0
Content-type: multipart/alternative;
 boundary="Boundary_(ID_KMnKR6wtSY5LljuUSASQZQ)"
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp027.mac.com (Oracle Communications Messaging Exchange Server 7u4-18.01
 64bit (built Jul 15 2010)) with ESMTPSA id <0LEK00E7MRKF9U60@asmtp027.mac.com>
 for general@hadoop.apache.org; Wed, 05 Jan 2011 16:19:28 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-05_12:2011-01-06,2011-01-05,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101050129
Sun-Java-System-SMTP-Warning: Lines longer than SMTP allows found and wrapped.
From: Nigel Daley <ndaley@mac.com>
Subject: Re: Patch testing
Date: Wed, 05 Jan 2011 16:19:27 -0800
In-reply-to: <AANLkTi=9wEVmjZqXN+Cw2+cW3iNfuMsUJG=OeXnMfg0f@mail.gmail.com>
To: general@hadoop.apache.org
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com>
 <20101020195420.GG2075@tp>
 <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com>
 <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
 <7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com>
 <AANLkTimLXA+CvUs-09LMN3ob6GZqmhxiyrKO7pzWEfse@mail.gmail.com>
 <4CE46119.2030509@yahoo-inc.com>
 <8617BECB-78B4-42B6-B592-D7FC1F8DA923@mac.com>
 <4CE47C88.5050203@yahoo-inc.com>
 <AANLkTinTM5h-3NupRSsFh-DnyvFYw+_OsXpASxcQUSDn@mail.gmail.com>
 <AANLkTimSOxaXZi-VLjwuTB5Ch5tRt0bKozc5L+Nx=ZrQ@mail.gmail.com>
 <AANLkTineU2v=Sx-=oC=6ROt6Os8TEbnS3L8xz1ZkoxCc@mail.gmail.com>
 <AANLkTin+oucXdCdOo7LQTtqndTf1axiT0gQntiOZ_mQJ@mail.gmail.com>
 <AANLkTinGn4Pb6_LkrY4-kK56PdHX4C1Zfu7A2PVfUoub@mail.gmail.com>
 <41FB0800-3703-49C1-8069-DEB74FFE6FAC@mac.com>
 <AANLkTik_rNLMHj3JT6Ekdop-NU_qzcB=VdsX+k4uFBtf@mail.gmail.com>
 <AANLkTin9=FPxv+0EydK4mWp_1zJbgGTddevvF9MrHW80@mail.gmail.com>
 <AANLkTiku_OvZuuOMWuXMcVf4UxB+W-SfW1WWV6rU0cXB@mail.gmail.com>
 <7B140F50-27E0-47C7-8EF8-B897D26CEE49@mac.com>
 <AANLkTi=9wEVmjZqXN+Cw2+cW3iNfuMsUJG=OeXnMfg0f@mail.gmail.com>
Message-id: <394BBDCC-9561-4E07-8EBA-AE3A92814E5A@mac.com>
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

--Boundary_(ID_KMnKR6wtSY5LljuUSASQZQ)
Content-type: text/plain; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT

Todd, would love to get https://issues.apache.org/jira/browse/MAPREDUCE-2121 fixed first since this is failing every night on trunk. 

Nige

On Jan 5, 2011, at 2:45 PM, Todd Lipcon wrote:

> Hi Nigel,
> 
> MAPREDUCE-2172 has been fixed for a while. Are there any other particular
> JIRAs you think need to be fixed before the MR test-patch queue gets
> enabled? I have a lot of outstanding patches and doing all the test-patch
> turnaround manually on 3 different boxes is a real headache.
> 
> Thanks
> -Todd
> 
> On Tue, Dec 21, 2010 at 1:33 PM, Nigel Daley <ndaley@mac.com> wrote:
> 
>> Ok, HDFS is now enabled.  You'll see a stream of updates shortly on the ~30
>> Patch Available HDFS issues.
>> 
>> Nige
>> 
>> On Dec 20, 2010, at 12:42 PM, Jakob Homan wrote:
>> 
>>> I committed HDFS-1511 this morning.  We should be good to go.  I can
>>> haz snooty robot butler?
>>> 
>>> On Fri, Dec 17, 2010 at 8:31 PM, Konstantin Boudnik <cos@apache.org>
>> wrote:
>>>> Thanks Jacob. I am wasted already but I can do it on Sun, I think,
>>>> unless it is done earlier.
>>>> --
>>>>  Take care,
>>>> Konstantin (Cos) Boudnik
>>>> 
>>>> 
>>>> 
>>>> On Fri, Dec 17, 2010 at 19:41, Jakob Homan <jghoman@gmail.com> wrote:
>>>>> Ok.  I'll get a patch out for 1511 tomorrow, unless someone wants to
>>>>> whip one up tonight.
>>>>> 
>>>>> 
>>>>> On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley <ndaley@mac.com> wrote:
>>>>>> I agree with Cos on fixing HDFS-1511 first. Once that is done I'll
>> enable hdfs patch testing.
>>>>>> 
>>>>>> Cheers,
>>>>>> Nige
>>>>>> 
>>>>>> Sent from my iPhone4
>>>>>> 
>>>>>> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <cos@apache.org>
>> wrote:
>>>>>> 
>>>>>>> One more issue needs to be addressed before test-patch is turned on
>> HDFS is
>>>>>>> https://issues.apache.org/jira/browse/HDFS-1511
>>>>>>> --
>>>>>>>  Take care,
>>>>>>> Konstantin (Cos) Boudnik
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <cos@apache.org>
>> wrote:
>>>>>>>> Considering that because of these 4 faulty cases every patch will be
>>>>>>>> -1'ed a patch author will still have to look at it and make a
>> comment
>>>>>>>> why this particular -1 isn't valid. Lesser work, perhaps, but
>> messier
>>>>>>>> IMO. I'm not blocking it - I just feel like there's a better way.
>>>>>>>> 
>>>>>>>> --
>>>>>>>>  Take care,
>>>>>>>> Konstantin (Cos) Boudnik
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jghoman@gmail.com>
>> wrote:
>>>>>>>>>> If HDFS is added to the test-patch queue right now we get
>>>>>>>>>> nothing but dozens of -1'ed patches.
>>>>>>>>> There aren't dozens of patches being submitted currently.  The -1
>>>>>>>>> isn't the important thing, it's the grunt work of actually running
>>>>>>>>> (and waiting) for the tests, test-patch, etc. that Hudson does so
>> that
>>>>>>>>> the developer doesn't have to.
>>>>>>>>> 
>>>>>>>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <
>> dhruba@gmail.com> wrote:
>>>>>>>>>> +1, thanks for doing this.
>>>>>>>>>> 
>>>>>>>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jghoman@gmail.com>
>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> So, with test-patch updated to show the failing tests, saving the
>>>>>>>>>>> developers the need to go and verify that the failed tests are
>> all
>>>>>>>>>>> known, how do people feel about turning on test-patch again for
>> HDFS
>>>>>>>>>>> and mapred?  I think it'll help prevent any more tests from
>> entering
>>>>>>>>>>> the "yeah, we know" category.
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> jg
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <
>> jhoman@yahoo-inc.com> wrote:
>>>>>>>>>>>> True, each patch would get a -1 and the failing tests would need
>> to be
>>>>>>>>>>>> verified as those known bad (BTW, it would be great if Hudson
>> could list
>>>>>>>>>>>> which tests failed in the message it posts to JIRA).  But that's
>> still
>>>>>>>>>>> quite
>>>>>>>>>>>> a bit less error-prone work than if the developer runs the tests
>> and
>>>>>>>>>>>> test-patch themselves.  Also, with 22 being cut, there are a lot
>> of
>>>>>>>>>>> patches
>>>>>>>>>>>> up in the air and several developers are juggling multiple
>> patches.  The
>>>>>>>>>>>> more automation we can have, even if it's not perfect, will
>> decrease
>>>>>>>>>>> errors
>>>>>>>>>>>> we may make.
>>>>>>>>>>>> -jg
>>>>>>>>>>>> 
>>>>>>>>>>>> Nigel Daley wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> It's also ready to run on MapReduce and HDFS but we won't
>> turn it on
>>>>>>>>>>>>>>> until these projects build and test cleanly.  Looks like both
>> these
>>>>>>>>>>> projects
>>>>>>>>>>>>>>> currently have test failures.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Assuming the projects are compiling and building, is there a
>> reason to
>>>>>>>>>>>>>> not turn it on despite the test failures? Hudson is invaluable
>> to
>>>>>>>>>>> developers
>>>>>>>>>>>>>> who then don't have to run the tests and test-patch
>> themselves.  We
>>>>>>>>>>> didn't
>>>>>>>>>>>>>> turn Hudson off when it was working previously and there were
>> known
>>>>>>>>>>>>>> failures.  I think one of the reasons we have more failing
>> tests now is
>>>>>>>>>>> the
>>>>>>>>>>>>>> higher cost of doing Hudson's work (not a great excuse I
>> know).  This
>>>>>>>>>>> is
>>>>>>>>>>>>>> particularly true now because several of the failing tests
>> involve
>>>>>>>>>>> tests
>>>>>>>>>>>>>> timing out, making the whole testing regime even longer.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Every single patch would get a -1 and need investigation.
>> Currently,
>>>>>>>>>>> that
>>>>>>>>>>>>> would be about 83 investigations between MR and HDFS issues
>> that are in
>>>>>>>>>>>>> patch available state.  Shouldn't we focus on getting these
>> tests fixed
>>>>>>>>>>> or
>>>>>>>>>>>>> removed/?  Also, I need to get MAPREDUCE-2172 fixed (applies to
>> HDFS as
>>>>>>>>>>>>> well) before I turn this on.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Nige
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> Connect to me at http://www.facebook.com/dhruba
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> 
>> 
> 
> 
> -- 
> Todd Lipcon
> Software Engineer, Cloudera


--Boundary_(ID_KMnKR6wtSY5LljuUSASQZQ)--

From general-return-2667-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 06 00:34:15 2011
Return-Path: <general-return-2667-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78225 invoked from network); 6 Jan 2011 00:34:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2011 00:34:14 -0000
Received: (qmail 61063 invoked by uid 500); 6 Jan 2011 00:34:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60964 invoked by uid 500); 6 Jan 2011 00:34:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60955 invoked by uid 99); 6 Jan 2011 00:34:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 00:34:13 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 00:34:06 +0000
Received: by iyb26 with SMTP id 26so17149839iyb.35
        for <general@hadoop.apache.org>; Wed, 05 Jan 2011 16:33:45 -0800 (PST)
Received: by 10.231.31.139 with SMTP id y11mr7515511ibc.96.1294274025264; Wed,
 05 Jan 2011 16:33:45 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.115.8 with HTTP; Wed, 5 Jan 2011 16:33:25 -0800 (PST)
In-Reply-To: <394BBDCC-9561-4E07-8EBA-AE3A92814E5A@mac.com>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com> <20101020195420.GG2075@tp>
 <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com> <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
 <7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com> <AANLkTimLXA+CvUs-09LMN3ob6GZqmhxiyrKO7pzWEfse@mail.gmail.com>
 <4CE46119.2030509@yahoo-inc.com> <8617BECB-78B4-42B6-B592-D7FC1F8DA923@mac.com>
 <4CE47C88.5050203@yahoo-inc.com> <AANLkTinTM5h-3NupRSsFh-DnyvFYw+_OsXpASxcQUSDn@mail.gmail.com>
 <AANLkTimSOxaXZi-VLjwuTB5Ch5tRt0bKozc5L+Nx=ZrQ@mail.gmail.com>
 <AANLkTineU2v=Sx-=oC=6ROt6Os8TEbnS3L8xz1ZkoxCc@mail.gmail.com>
 <AANLkTin+oucXdCdOo7LQTtqndTf1axiT0gQntiOZ_mQJ@mail.gmail.com>
 <AANLkTinGn4Pb6_LkrY4-kK56PdHX4C1Zfu7A2PVfUoub@mail.gmail.com>
 <41FB0800-3703-49C1-8069-DEB74FFE6FAC@mac.com> <AANLkTik_rNLMHj3JT6Ekdop-NU_qzcB=VdsX+k4uFBtf@mail.gmail.com>
 <AANLkTin9=FPxv+0EydK4mWp_1zJbgGTddevvF9MrHW80@mail.gmail.com>
 <AANLkTiku_OvZuuOMWuXMcVf4UxB+W-SfW1WWV6rU0cXB@mail.gmail.com>
 <7B140F50-27E0-47C7-8EF8-B897D26CEE49@mac.com> <AANLkTi=9wEVmjZqXN+Cw2+cW3iNfuMsUJG=OeXnMfg0f@mail.gmail.com>
 <394BBDCC-9561-4E07-8EBA-AE3A92814E5A@mac.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Wed, 5 Jan 2011 16:33:25 -0800
Message-ID: <AANLkTinKeADQRv1hXMDxqsaws+u2p=b7ntHrJGPOFSRo@mail.gmail.com>
Subject: Re: Patch testing
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00235433303a8495b9049922a7c7
X-Virus-Checked: Checked by ClamAV on apache.org

--00235433303a8495b9049922a7c7
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Jan 5, 2011 at 4:19 PM, Nigel Daley <ndaley@mac.com> wrote:

> Todd, would love to get
> https://issues.apache.org/jira/browse/MAPREDUCE-2121 fixed first since
> this is failing every night on trunk.
>

What if we disable that test, move that issue to 0.22 blocker, and then
enable the test-patch? I'll also look into that one today, but if it's
something that will take a while to fix, I don't think we should hold off
the useful testing for all the other patches.

 -Todd

On Jan 5, 2011, at 2:45 PM, Todd Lipcon wrote:
>
> > Hi Nigel,
> >
> > MAPREDUCE-2172 has been fixed for a while. Are there any other particular
> > JIRAs you think need to be fixed before the MR test-patch queue gets
> > enabled? I have a lot of outstanding patches and doing all the test-patch
> > turnaround manually on 3 different boxes is a real headache.
> >
> > Thanks
> > -Todd
> >
> > On Tue, Dec 21, 2010 at 1:33 PM, Nigel Daley <ndaley@mac.com> wrote:
> >
> >> Ok, HDFS is now enabled.  You'll see a stream of updates shortly on the
> ~30
> >> Patch Available HDFS issues.
> >>
> >> Nige
> >>
> >> On Dec 20, 2010, at 12:42 PM, Jakob Homan wrote:
> >>
> >>> I committed HDFS-1511 this morning.  We should be good to go.  I can
> >>> haz snooty robot butler?
> >>>
> >>> On Fri, Dec 17, 2010 at 8:31 PM, Konstantin Boudnik <cos@apache.org>
> >> wrote:
> >>>> Thanks Jacob. I am wasted already but I can do it on Sun, I think,
> >>>> unless it is done earlier.
> >>>> --
> >>>>  Take care,
> >>>> Konstantin (Cos) Boudnik
> >>>>
> >>>>
> >>>>
> >>>> On Fri, Dec 17, 2010 at 19:41, Jakob Homan <jghoman@gmail.com> wrote:
> >>>>> Ok.  I'll get a patch out for 1511 tomorrow, unless someone wants to
> >>>>> whip one up tonight.
> >>>>>
> >>>>>
> >>>>> On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley <ndaley@mac.com> wrote:
> >>>>>> I agree with Cos on fixing HDFS-1511 first. Once that is done I'll
> >> enable hdfs patch testing.
> >>>>>>
> >>>>>> Cheers,
> >>>>>> Nige
> >>>>>>
> >>>>>> Sent from my iPhone4
> >>>>>>
> >>>>>> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <cos@apache.org>
> >> wrote:
> >>>>>>
> >>>>>>> One more issue needs to be addressed before test-patch is turned on
> >> HDFS is
> >>>>>>> https://issues.apache.org/jira/browse/HDFS-1511
> >>>>>>> --
> >>>>>>>  Take care,
> >>>>>>> Konstantin (Cos) Boudnik
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <cos@apache.org>
> >> wrote:
> >>>>>>>> Considering that because of these 4 faulty cases every patch will
> be
> >>>>>>>> -1'ed a patch author will still have to look at it and make a
> >> comment
> >>>>>>>> why this particular -1 isn't valid. Lesser work, perhaps, but
> >> messier
> >>>>>>>> IMO. I'm not blocking it - I just feel like there's a better way.
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>>  Take care,
> >>>>>>>> Konstantin (Cos) Boudnik
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jghoman@gmail.com>
> >> wrote:
> >>>>>>>>>> If HDFS is added to the test-patch queue right now we get
> >>>>>>>>>> nothing but dozens of -1'ed patches.
> >>>>>>>>> There aren't dozens of patches being submitted currently.  The -1
> >>>>>>>>> isn't the important thing, it's the grunt work of actually
> running
> >>>>>>>>> (and waiting) for the tests, test-patch, etc. that Hudson does so
> >> that
> >>>>>>>>> the developer doesn't have to.
> >>>>>>>>>
> >>>>>>>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <
> >> dhruba@gmail.com> wrote:
> >>>>>>>>>> +1, thanks for doing this.
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jghoman@gmail.com
> >
> >> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> So, with test-patch updated to show the failing tests, saving
> the
> >>>>>>>>>>> developers the need to go and verify that the failed tests are
> >> all
> >>>>>>>>>>> known, how do people feel about turning on test-patch again for
> >> HDFS
> >>>>>>>>>>> and mapred?  I think it'll help prevent any more tests from
> >> entering
> >>>>>>>>>>> the "yeah, we know" category.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> jg
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <
> >> jhoman@yahoo-inc.com> wrote:
> >>>>>>>>>>>> True, each patch would get a -1 and the failing tests would
> need
> >> to be
> >>>>>>>>>>>> verified as those known bad (BTW, it would be great if Hudson
> >> could list
> >>>>>>>>>>>> which tests failed in the message it posts to JIRA).  But
> that's
> >> still
> >>>>>>>>>>> quite
> >>>>>>>>>>>> a bit less error-prone work than if the developer runs the
> tests
> >> and
> >>>>>>>>>>>> test-patch themselves.  Also, with 22 being cut, there are a
> lot
> >> of
> >>>>>>>>>>> patches
> >>>>>>>>>>>> up in the air and several developers are juggling multiple
> >> patches.  The
> >>>>>>>>>>>> more automation we can have, even if it's not perfect, will
> >> decrease
> >>>>>>>>>>> errors
> >>>>>>>>>>>> we may make.
> >>>>>>>>>>>> -jg
> >>>>>>>>>>>>
> >>>>>>>>>>>> Nigel Daley wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>> It's also ready to run on MapReduce and HDFS but we won't
> >> turn it on
> >>>>>>>>>>>>>>> until these projects build and test cleanly.  Looks like
> both
> >> these
> >>>>>>>>>>> projects
> >>>>>>>>>>>>>>> currently have test failures.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Assuming the projects are compiling and building, is there a
> >> reason to
> >>>>>>>>>>>>>> not turn it on despite the test failures? Hudson is
> invaluable
> >> to
> >>>>>>>>>>> developers
> >>>>>>>>>>>>>> who then don't have to run the tests and test-patch
> >> themselves.  We
> >>>>>>>>>>> didn't
> >>>>>>>>>>>>>> turn Hudson off when it was working previously and there
> were
> >> known
> >>>>>>>>>>>>>> failures.  I think one of the reasons we have more failing
> >> tests now is
> >>>>>>>>>>> the
> >>>>>>>>>>>>>> higher cost of doing Hudson's work (not a great excuse I
> >> know).  This
> >>>>>>>>>>> is
> >>>>>>>>>>>>>> particularly true now because several of the failing tests
> >> involve
> >>>>>>>>>>> tests
> >>>>>>>>>>>>>> timing out, making the whole testing regime even longer.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Every single patch would get a -1 and need investigation.
> >> Currently,
> >>>>>>>>>>> that
> >>>>>>>>>>>>> would be about 83 investigations between MR and HDFS issues
> >> that are in
> >>>>>>>>>>>>> patch available state.  Shouldn't we focus on getting these
> >> tests fixed
> >>>>>>>>>>> or
> >>>>>>>>>>>>> removed/?  Also, I need to get MAPREDUCE-2172 fixed (applies
> to
> >> HDFS as
> >>>>>>>>>>>>> well) before I turn this on.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>> Nige
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> Connect to me at http://www.facebook.com/dhruba
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> >>
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

--00235433303a8495b9049922a7c7--

From general-return-2668-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 06 00:40:35 2011
Return-Path: <general-return-2668-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 79260 invoked from network); 6 Jan 2011 00:40:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2011 00:40:35 -0000
Received: (qmail 67081 invoked by uid 500); 6 Jan 2011 00:40:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67034 invoked by uid 500); 6 Jan 2011 00:40:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67026 invoked by uid 99); 6 Jan 2011 00:40:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 00:40:33 +0000
X-ASF-Spam-Status: No, hits=1.8 required=10.0
	tests=FREEMAIL_FROM,FREEMAIL_REPLY,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.97 as permitted sender)
Received: from [17.148.16.97] (HELO asmtpout022.mac.com) (17.148.16.97)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 00:40:26 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp022.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LEK001QUSHWWW90@asmtp022.mac.com>
 for general@hadoop.apache.org; Wed, 05 Jan 2011 16:39:32 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-05_12:2011-01-06,2011-01-05,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101050132
Sun-Java-System-SMTP-Warning: Lines longer than SMTP allows found and wrapped.
Subject: Re: Patch testing
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <AANLkTinKeADQRv1hXMDxqsaws+u2p=b7ntHrJGPOFSRo@mail.gmail.com>
Date: Wed, 05 Jan 2011 16:39:31 -0800
Message-id: <F5C6F56C-74DF-43B6-9846-EE60C16E04C7@mac.com>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com>
 <20101020195420.GG2075@tp>
 <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com>
 <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
 <7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com>
 <AANLkTimLXA+CvUs-09LMN3ob6GZqmhxiyrKO7pzWEfse@mail.gmail.com>
 <4CE46119.2030509@yahoo-inc.com>
 <8617BECB-78B4-42B6-B592-D7FC1F8DA923@mac.com>
 <4CE47C88.5050203@yahoo-inc.com>
 <AANLkTinTM5h-3NupRSsFh-DnyvFYw+_OsXpASxcQUSDn@mail.gmail.com>
 <AANLkTimSOxaXZi-VLjwuTB5Ch5tRt0bKozc5L+Nx=ZrQ@mail.gmail.com>
 <AANLkTineU2v=Sx-=oC=6ROt6Os8TEbnS3L8xz1ZkoxCc@mail.gmail.com>
 <AANLkTin+oucXdCdOo7LQTtqndTf1axiT0gQntiOZ_mQJ@mail.gmail.com>
 <AANLkTinGn4Pb6_LkrY4-kK56PdHX4C1Zfu7A2PVfUoub@mail.gmail.com>
 <41FB0800-3703-49C1-8069-DEB74FFE6FAC@mac.com>
 <AANLkTik_rNLMHj3JT6Ekdop-NU_qzcB=VdsX+k4uFBtf@mail.gmail.com>
 <AANLkTin9=FPxv+0EydK4mWp_1zJbgGTddevvF9MrHW80@mail.gmail.com>
 <AANLkTiku_OvZuuOMWuXMcVf4UxB+W-SfW1WWV6rU0cXB@mail.gmail.com>
 <7B140F50-27E0-47C7-8EF8-B897D26CEE49@mac.com>
 <AANLkTi=9wEVmjZqXN+Cw2+cW3iNfuMsUJG=OeXnMfg0f@mail.gmail.com>
 <394BBDCC-9561-4E07-8EBA-AE3A92814E5A@mac.com>
 <AANLkTinKeADQRv1hXMDxqsaws+u2p=b7ntHrJGPOFSRo@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks for looking into it Todd.  Let's first see if you think it can be fixed quickly.  Let me know.

Thanks,
Nige

On Jan 5, 2011, at 4:33 PM, Todd Lipcon wrote:

> On Wed, Jan 5, 2011 at 4:19 PM, Nigel Daley <ndaley@mac.com> wrote:
> 
>> Todd, would love to get
>> https://issues.apache.org/jira/browse/MAPREDUCE-2121 fixed first since
>> this is failing every night on trunk.
>> 
> 
> What if we disable that test, move that issue to 0.22 blocker, and then
> enable the test-patch? I'll also look into that one today, but if it's
> something that will take a while to fix, I don't think we should hold off
> the useful testing for all the other patches.
> 
> -Todd
> 
> On Jan 5, 2011, at 2:45 PM, Todd Lipcon wrote:
>> 
>>> Hi Nigel,
>>> 
>>> MAPREDUCE-2172 has been fixed for a while. Are there any other particular
>>> JIRAs you think need to be fixed before the MR test-patch queue gets
>>> enabled? I have a lot of outstanding patches and doing all the test-patch
>>> turnaround manually on 3 different boxes is a real headache.
>>> 
>>> Thanks
>>> -Todd
>>> 
>>> On Tue, Dec 21, 2010 at 1:33 PM, Nigel Daley <ndaley@mac.com> wrote:
>>> 
>>>> Ok, HDFS is now enabled.  You'll see a stream of updates shortly on the
>> ~30
>>>> Patch Available HDFS issues.
>>>> 
>>>> Nige
>>>> 
>>>> On Dec 20, 2010, at 12:42 PM, Jakob Homan wrote:
>>>> 
>>>>> I committed HDFS-1511 this morning.  We should be good to go.  I can
>>>>> haz snooty robot butler?
>>>>> 
>>>>> On Fri, Dec 17, 2010 at 8:31 PM, Konstantin Boudnik <cos@apache.org>
>>>> wrote:
>>>>>> Thanks Jacob. I am wasted already but I can do it on Sun, I think,
>>>>>> unless it is done earlier.
>>>>>> --
>>>>>> Take care,
>>>>>> Konstantin (Cos) Boudnik
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Fri, Dec 17, 2010 at 19:41, Jakob Homan <jghoman@gmail.com> wrote:
>>>>>>> Ok.  I'll get a patch out for 1511 tomorrow, unless someone wants to
>>>>>>> whip one up tonight.
>>>>>>> 
>>>>>>> 
>>>>>>> On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley <ndaley@mac.com> wrote:
>>>>>>>> I agree with Cos on fixing HDFS-1511 first. Once that is done I'll
>>>> enable hdfs patch testing.
>>>>>>>> 
>>>>>>>> Cheers,
>>>>>>>> Nige
>>>>>>>> 
>>>>>>>> Sent from my iPhone4
>>>>>>>> 
>>>>>>>> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <cos@apache.org>
>>>> wrote:
>>>>>>>> 
>>>>>>>>> One more issue needs to be addressed before test-patch is turned on
>>>> HDFS is
>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-1511
>>>>>>>>> --
>>>>>>>>> Take care,
>>>>>>>>> Konstantin (Cos) Boudnik
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <cos@apache.org>
>>>> wrote:
>>>>>>>>>> Considering that because of these 4 faulty cases every patch will
>> be
>>>>>>>>>> -1'ed a patch author will still have to look at it and make a
>>>> comment
>>>>>>>>>> why this particular -1 isn't valid. Lesser work, perhaps, but
>>>> messier
>>>>>>>>>> IMO. I'm not blocking it - I just feel like there's a better way.
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> Take care,
>>>>>>>>>> Konstantin (Cos) Boudnik
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jghoman@gmail.com>
>>>> wrote:
>>>>>>>>>>>> If HDFS is added to the test-patch queue right now we get
>>>>>>>>>>>> nothing but dozens of -1'ed patches.
>>>>>>>>>>> There aren't dozens of patches being submitted currently.  The -1
>>>>>>>>>>> isn't the important thing, it's the grunt work of actually
>> running
>>>>>>>>>>> (and waiting) for the tests, test-patch, etc. that Hudson does so
>>>> that
>>>>>>>>>>> the developer doesn't have to.
>>>>>>>>>>> 
>>>>>>>>>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <
>>>> dhruba@gmail.com> wrote:
>>>>>>>>>>>> +1, thanks for doing this.
>>>>>>>>>>>> 
>>>>>>>>>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <jghoman@gmail.com
>>> 
>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> So, with test-patch updated to show the failing tests, saving
>> the
>>>>>>>>>>>>> developers the need to go and verify that the failed tests are
>>>> all
>>>>>>>>>>>>> known, how do people feel about turning on test-patch again for
>>>> HDFS
>>>>>>>>>>>>> and mapred?  I think it'll help prevent any more tests from
>>>> entering
>>>>>>>>>>>>> the "yeah, we know" category.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> jg
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <
>>>> jhoman@yahoo-inc.com> wrote:
>>>>>>>>>>>>>> True, each patch would get a -1 and the failing tests would
>> need
>>>> to be
>>>>>>>>>>>>>> verified as those known bad (BTW, it would be great if Hudson
>>>> could list
>>>>>>>>>>>>>> which tests failed in the message it posts to JIRA).  But
>> that's
>>>> still
>>>>>>>>>>>>> quite
>>>>>>>>>>>>>> a bit less error-prone work than if the developer runs the
>> tests
>>>> and
>>>>>>>>>>>>>> test-patch themselves.  Also, with 22 being cut, there are a
>> lot
>>>> of
>>>>>>>>>>>>> patches
>>>>>>>>>>>>>> up in the air and several developers are juggling multiple
>>>> patches.  The
>>>>>>>>>>>>>> more automation we can have, even if it's not perfect, will
>>>> decrease
>>>>>>>>>>>>> errors
>>>>>>>>>>>>>> we may make.
>>>>>>>>>>>>>> -jg
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Nigel Daley wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> It's also ready to run on MapReduce and HDFS but we won't
>>>> turn it on
>>>>>>>>>>>>>>>>> until these projects build and test cleanly.  Looks like
>> both
>>>> these
>>>>>>>>>>>>> projects
>>>>>>>>>>>>>>>>> currently have test failures.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Assuming the projects are compiling and building, is there a
>>>> reason to
>>>>>>>>>>>>>>>> not turn it on despite the test failures? Hudson is
>> invaluable
>>>> to
>>>>>>>>>>>>> developers
>>>>>>>>>>>>>>>> who then don't have to run the tests and test-patch
>>>> themselves.  We
>>>>>>>>>>>>> didn't
>>>>>>>>>>>>>>>> turn Hudson off when it was working previously and there
>> were
>>>> known
>>>>>>>>>>>>>>>> failures.  I think one of the reasons we have more failing
>>>> tests now is
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> higher cost of doing Hudson's work (not a great excuse I
>>>> know).  This
>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> particularly true now because several of the failing tests
>>>> involve
>>>>>>>>>>>>> tests
>>>>>>>>>>>>>>>> timing out, making the whole testing regime even longer.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Every single patch would get a -1 and need investigation.
>>>> Currently,
>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> would be about 83 investigations between MR and HDFS issues
>>>> that are in
>>>>>>>>>>>>>>> patch available state.  Shouldn't we focus on getting these
>>>> tests fixed
>>>>>>>>>>>>> or
>>>>>>>>>>>>>>> removed/?  Also, I need to get MAPREDUCE-2172 fixed (applies
>> to
>>>> HDFS as
>>>>>>>>>>>>>>> well) before I turn this on.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>> Nige
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> --
>>>>>>>>>>>> Connect to me at http://www.facebook.com/dhruba
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>> 
>> 
> 
> 
> -- 
> Todd Lipcon
> Software Engineer, Cloudera


From general-return-2669-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 06 01:43:42 2011
Return-Path: <general-return-2669-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1363 invoked from network); 6 Jan 2011 01:43:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2011 01:43:41 -0000
Received: (qmail 2798 invoked by uid 500); 6 Jan 2011 01:43:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2747 invoked by uid 500); 6 Jan 2011 01:43:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2739 invoked by uid 99); 6 Jan 2011 01:43:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 01:43:40 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 01:43:33 +0000
Received: by iyb26 with SMTP id 26so17194261iyb.35
        for <general@hadoop.apache.org>; Wed, 05 Jan 2011 17:43:11 -0800 (PST)
Received: by 10.231.208.17 with SMTP id ga17mr14942486ibb.121.1294278191505;
 Wed, 05 Jan 2011 17:43:11 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.115.8 with HTTP; Wed, 5 Jan 2011 17:42:51 -0800 (PST)
In-Reply-To: <F5C6F56C-74DF-43B6-9846-EE60C16E04C7@mac.com>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com> <20101020195420.GG2075@tp>
 <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com> <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
 <7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com> <AANLkTimLXA+CvUs-09LMN3ob6GZqmhxiyrKO7pzWEfse@mail.gmail.com>
 <4CE46119.2030509@yahoo-inc.com> <8617BECB-78B4-42B6-B592-D7FC1F8DA923@mac.com>
 <4CE47C88.5050203@yahoo-inc.com> <AANLkTinTM5h-3NupRSsFh-DnyvFYw+_OsXpASxcQUSDn@mail.gmail.com>
 <AANLkTimSOxaXZi-VLjwuTB5Ch5tRt0bKozc5L+Nx=ZrQ@mail.gmail.com>
 <AANLkTineU2v=Sx-=oC=6ROt6Os8TEbnS3L8xz1ZkoxCc@mail.gmail.com>
 <AANLkTin+oucXdCdOo7LQTtqndTf1axiT0gQntiOZ_mQJ@mail.gmail.com>
 <AANLkTinGn4Pb6_LkrY4-kK56PdHX4C1Zfu7A2PVfUoub@mail.gmail.com>
 <41FB0800-3703-49C1-8069-DEB74FFE6FAC@mac.com> <AANLkTik_rNLMHj3JT6Ekdop-NU_qzcB=VdsX+k4uFBtf@mail.gmail.com>
 <AANLkTin9=FPxv+0EydK4mWp_1zJbgGTddevvF9MrHW80@mail.gmail.com>
 <AANLkTiku_OvZuuOMWuXMcVf4UxB+W-SfW1WWV6rU0cXB@mail.gmail.com>
 <7B140F50-27E0-47C7-8EF8-B897D26CEE49@mac.com> <AANLkTi=9wEVmjZqXN+Cw2+cW3iNfuMsUJG=OeXnMfg0f@mail.gmail.com>
 <394BBDCC-9561-4E07-8EBA-AE3A92814E5A@mac.com> <AANLkTinKeADQRv1hXMDxqsaws+u2p=b7ntHrJGPOFSRo@mail.gmail.com>
 <F5C6F56C-74DF-43B6-9846-EE60C16E04C7@mac.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Wed, 5 Jan 2011 17:42:51 -0800
Message-ID: <AANLkTim_o2HDjsxK36pWQn76D9FsJN97SACCHNMEiS7X@mail.gmail.com>
Subject: Re: Patch testing
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba5bc923d8615f0499239f11
X-Virus-Checked: Checked by ClamAV on apache.org

--90e6ba5bc923d8615f0499239f11
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Jan 5, 2011 at 4:39 PM, Nigel Daley <ndaley@mac.com> wrote:

> Thanks for looking into it Todd.  Let's first see if you think it can be
> fixed quickly.  Let me know.
>
>
No problem, it wasn't too bad after all. Patch up on HADOOP-7087 which fixes
this test timeout for me.

-Todd


>  On Jan 5, 2011, at 4:33 PM, Todd Lipcon wrote:
>
> > On Wed, Jan 5, 2011 at 4:19 PM, Nigel Daley <ndaley@mac.com> wrote:
> >
> >> Todd, would love to get
> >> https://issues.apache.org/jira/browse/MAPREDUCE-2121 fixed first since
> >> this is failing every night on trunk.
> >>
> >
> > What if we disable that test, move that issue to 0.22 blocker, and then
> > enable the test-patch? I'll also look into that one today, but if it's
> > something that will take a while to fix, I don't think we should hold off
> > the useful testing for all the other patches.
> >
> > -Todd
> >
> > On Jan 5, 2011, at 2:45 PM, Todd Lipcon wrote:
> >>
> >>> Hi Nigel,
> >>>
> >>> MAPREDUCE-2172 has been fixed for a while. Are there any other
> particular
> >>> JIRAs you think need to be fixed before the MR test-patch queue gets
> >>> enabled? I have a lot of outstanding patches and doing all the
> test-patch
> >>> turnaround manually on 3 different boxes is a real headache.
> >>>
> >>> Thanks
> >>> -Todd
> >>>
> >>> On Tue, Dec 21, 2010 at 1:33 PM, Nigel Daley <ndaley@mac.com> wrote:
> >>>
> >>>> Ok, HDFS is now enabled.  You'll see a stream of updates shortly on
> the
> >> ~30
> >>>> Patch Available HDFS issues.
> >>>>
> >>>> Nige
> >>>>
> >>>> On Dec 20, 2010, at 12:42 PM, Jakob Homan wrote:
> >>>>
> >>>>> I committed HDFS-1511 this morning.  We should be good to go.  I can
> >>>>> haz snooty robot butler?
> >>>>>
> >>>>> On Fri, Dec 17, 2010 at 8:31 PM, Konstantin Boudnik <cos@apache.org>
> >>>> wrote:
> >>>>>> Thanks Jacob. I am wasted already but I can do it on Sun, I think,
> >>>>>> unless it is done earlier.
> >>>>>> --
> >>>>>> Take care,
> >>>>>> Konstantin (Cos) Boudnik
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Fri, Dec 17, 2010 at 19:41, Jakob Homan <jghoman@gmail.com>
> wrote:
> >>>>>>> Ok.  I'll get a patch out for 1511 tomorrow, unless someone wants
> to
> >>>>>>> whip one up tonight.
> >>>>>>>
> >>>>>>>
> >>>>>>> On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley <ndaley@mac.com>
> wrote:
> >>>>>>>> I agree with Cos on fixing HDFS-1511 first. Once that is done I'll
> >>>> enable hdfs patch testing.
> >>>>>>>>
> >>>>>>>> Cheers,
> >>>>>>>> Nige
> >>>>>>>>
> >>>>>>>> Sent from my iPhone4
> >>>>>>>>
> >>>>>>>> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <cos@apache.org>
> >>>> wrote:
> >>>>>>>>
> >>>>>>>>> One more issue needs to be addressed before test-patch is turned
> on
> >>>> HDFS is
> >>>>>>>>> https://issues.apache.org/jira/browse/HDFS-1511
> >>>>>>>>> --
> >>>>>>>>> Take care,
> >>>>>>>>> Konstantin (Cos) Boudnik
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <
> cos@apache.org>
> >>>> wrote:
> >>>>>>>>>> Considering that because of these 4 faulty cases every patch
> will
> >> be
> >>>>>>>>>> -1'ed a patch author will still have to look at it and make a
> >>>> comment
> >>>>>>>>>> why this particular -1 isn't valid. Lesser work, perhaps, but
> >>>> messier
> >>>>>>>>>> IMO. I'm not blocking it - I just feel like there's a better
> way.
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> Take care,
> >>>>>>>>>> Konstantin (Cos) Boudnik
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jghoman@gmail.com>
> >>>> wrote:
> >>>>>>>>>>>> If HDFS is added to the test-patch queue right now we get
> >>>>>>>>>>>> nothing but dozens of -1'ed patches.
> >>>>>>>>>>> There aren't dozens of patches being submitted currently.  The
> -1
> >>>>>>>>>>> isn't the important thing, it's the grunt work of actually
> >> running
> >>>>>>>>>>> (and waiting) for the tests, test-patch, etc. that Hudson does
> so
> >>>> that
> >>>>>>>>>>> the developer doesn't have to.
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <
> >>>> dhruba@gmail.com> wrote:
> >>>>>>>>>>>> +1, thanks for doing this.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <
> jghoman@gmail.com
> >>>
> >>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> So, with test-patch updated to show the failing tests, saving
> >> the
> >>>>>>>>>>>>> developers the need to go and verify that the failed tests
> are
> >>>> all
> >>>>>>>>>>>>> known, how do people feel about turning on test-patch again
> for
> >>>> HDFS
> >>>>>>>>>>>>> and mapred?  I think it'll help prevent any more tests from
> >>>> entering
> >>>>>>>>>>>>> the "yeah, we know" category.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>> jg
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <
> >>>> jhoman@yahoo-inc.com> wrote:
> >>>>>>>>>>>>>> True, each patch would get a -1 and the failing tests would
> >> need
> >>>> to be
> >>>>>>>>>>>>>> verified as those known bad (BTW, it would be great if
> Hudson
> >>>> could list
> >>>>>>>>>>>>>> which tests failed in the message it posts to JIRA).  But
> >> that's
> >>>> still
> >>>>>>>>>>>>> quite
> >>>>>>>>>>>>>> a bit less error-prone work than if the developer runs the
> >> tests
> >>>> and
> >>>>>>>>>>>>>> test-patch themselves.  Also, with 22 being cut, there are a
> >> lot
> >>>> of
> >>>>>>>>>>>>> patches
> >>>>>>>>>>>>>> up in the air and several developers are juggling multiple
> >>>> patches.  The
> >>>>>>>>>>>>>> more automation we can have, even if it's not perfect, will
> >>>> decrease
> >>>>>>>>>>>>> errors
> >>>>>>>>>>>>>> we may make.
> >>>>>>>>>>>>>> -jg
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Nigel Daley wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> It's also ready to run on MapReduce and HDFS but we won't
> >>>> turn it on
> >>>>>>>>>>>>>>>>> until these projects build and test cleanly.  Looks like
> >> both
> >>>> these
> >>>>>>>>>>>>> projects
> >>>>>>>>>>>>>>>>> currently have test failures.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Assuming the projects are compiling and building, is there
> a
> >>>> reason to
> >>>>>>>>>>>>>>>> not turn it on despite the test failures? Hudson is
> >> invaluable
> >>>> to
> >>>>>>>>>>>>> developers
> >>>>>>>>>>>>>>>> who then don't have to run the tests and test-patch
> >>>> themselves.  We
> >>>>>>>>>>>>> didn't
> >>>>>>>>>>>>>>>> turn Hudson off when it was working previously and there
> >> were
> >>>> known
> >>>>>>>>>>>>>>>> failures.  I think one of the reasons we have more failing
> >>>> tests now is
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> higher cost of doing Hudson's work (not a great excuse I
> >>>> know).  This
> >>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>> particularly true now because several of the failing tests
> >>>> involve
> >>>>>>>>>>>>> tests
> >>>>>>>>>>>>>>>> timing out, making the whole testing regime even longer.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Every single patch would get a -1 and need investigation.
> >>>> Currently,
> >>>>>>>>>>>>> that
> >>>>>>>>>>>>>>> would be about 83 investigations between MR and HDFS issues
> >>>> that are in
> >>>>>>>>>>>>>>> patch available state.  Shouldn't we focus on getting these
> >>>> tests fixed
> >>>>>>>>>>>>> or
> >>>>>>>>>>>>>>> removed/?  Also, I need to get MAPREDUCE-2172 fixed
> (applies
> >> to
> >>>> HDFS as
> >>>>>>>>>>>>>>> well) before I turn this on.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>> Nige
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> Connect to me at http://www.facebook.com/dhruba
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> Todd Lipcon
> >>> Software Engineer, Cloudera
> >>
> >>
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

--90e6ba5bc923d8615f0499239f11--

From general-return-2670-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 06 02:25:25 2011
Return-Path: <general-return-2670-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 18459 invoked from network); 6 Jan 2011 02:25:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2011 02:25:25 -0000
Received: (qmail 37398 invoked by uid 500); 6 Jan 2011 02:25:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37352 invoked by uid 500); 6 Jan 2011 02:25:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37344 invoked by uid 99); 6 Jan 2011 02:25:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 02:25:23 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of magawake@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 02:25:18 +0000
Received: by qwh6 with SMTP id 6so16908465qwh.35
        for <general@hadoop.apache.org>; Wed, 05 Jan 2011 18:24:57 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=hjoxvIamRAjHl4FKD2FS0C8Q6oLYqF+Hgh+BMi5lI+8=;
        b=Zs3X+BBgLGHB61HPfrcbfhwRz+9TGtUc4KDDCthTffUetgxk/EpKDZrLU2uzMkRYA3
         dGoIUkRuYKbIhdy8eNVmBamPmnah+83nxJfiQxS1tsWMBJWURtPvBH+ZXvF6doyikJz7
         lYIpZ/tX3S94P0BtrX6WdcedBDlRnvjz50794=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=Q1HVdgnWkD+19wu8tOiN6yvG1zJnDQOvbEKiiaDjfbWdlvTe1/fwLsTz+MlyyW+1q9
         l1/1BWjWgNKd0whvpiVPC9MgzgPEp/eGpRPxuAtL7uPpiruTCDVT83PwJeSRpAE4GOVV
         ovYLVITmuo73a3SukfMgNZVNShsOAjpIBWmOQ=
MIME-Version: 1.0
Received: by 10.224.199.6 with SMTP id eq6mr22127078qab.272.1294280697675;
 Wed, 05 Jan 2011 18:24:57 -0800 (PST)
Received: by 10.220.118.20 with HTTP; Wed, 5 Jan 2011 18:24:57 -0800 (PST)
In-Reply-To: <ABC24175AFD3BE4DA15F4CD375ED413D063B0BD673@hq-ex-mb02.ad.navteq.com>
References: <AANLkTikN5ZkqMN9NrDgq_ohB6L_Qo_4TTjAiOzuS=bPt@mail.gmail.com>
	<ABC24175AFD3BE4DA15F4CD375ED413D063B0BD673@hq-ex-mb02.ad.navteq.com>
Date: Wed, 5 Jan 2011 21:24:57 -0500
Message-ID: <AANLkTimSF48e+hbN3sMhLMkkoNfsJdT37RvM-BFt_38P@mail.gmail.com>
Subject: Re: decommission a rack
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Thanks for the response.

I did decommission the server and it took couple of minutes. However,
I see the server in my "Dead Datanotes" page. How can I completely
forget about this box?


On Wed, Jan 5, 2011 at 10:01 AM, Segel, Mike <msegel@navteq.com> wrote:
> Change your topology script?
>
> One thing about rack awareness. You never want to have an unbalanced conf=
iguration. The caveat is if your uplinks between the ToR Switch =C2=A0are '=
fat' enough. (4 10GBe should do it.)
>
> If you watched your system with the one machine on a separate rack (assum=
ing it was also on a separate switch) you'd see a major network bottle neck=
. (We use Ganglia)
>
> If you're seeing the rack count wrong, you could just bounce the cloud an=
d it should show only two.
>
> Note: Deprecating a machine can take a long time. You may want to conside=
r just killing the node, waiting till the cluster recognizes that its down =
and I believe just kicking off an FSCK or maybe running the balancer progra=
m will clean that up faster than deprecating. (YMMV)
>
> HTH
>
> -Mike
>
>
> -----Original Message-----
> From: Mag Gam [mailto:magawake@gmail.com]
> Sent: Wednesday, January 05, 2011 5:56 AM
> To: general@hadoop.apache.org
> Subject: decommission a rack
>
> Using 0.21
>
> I have a server which was in rack3. It was the only server in this
> rack so I decommissioned the server. fsck shows only 2 racks however
> topology shows 3 racks. How can I fix this?
>
>
> The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above. =C2=A0If you a=
re not the intended recipient, you are hereby notified that any disseminati=
on, distribution, or copying of this communication, or any of its contents,=
 is strictly prohibited. =C2=A0If you have received this communication in e=
rror, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.
>

From general-return-2671-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 06 02:30:25 2011
Return-Path: <general-return-2671-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19247 invoked from network); 6 Jan 2011 02:30:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2011 02:30:24 -0000
Received: (qmail 42453 invoked by uid 500); 6 Jan 2011 02:30:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42402 invoked by uid 500); 6 Jan 2011 02:30:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42394 invoked by uid 99); 6 Jan 2011 02:30:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 02:30:22 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of magawake@gmail.com designates 209.85.216.169 as permitted sender)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 02:30:15 +0000
Received: by qyk7 with SMTP id 7so17426582qyk.14
        for <general@hadoop.apache.org>; Wed, 05 Jan 2011 18:29:54 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=lk3VlRY98AuiTTb2zKEKDsow2ISxLeKV+HqUojCZWuw=;
        b=b1AEX9yoQsJgYIk2/17XzJeOpwArKAfrwgGcjHjgkTkXp2mO5aBxed3DqEM9LoMmDt
         euf8kwDiaFToNlS5LG4jLoCKHgSL5ZD34AgzHHhTA5VKWc0t+pGYkoMR0ePxB+EBbiLI
         j9bBg8e4YkPqlAcAtd0cxUAL3wcKk6gu3aJWA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=J8/YniOxi8ATHxtCiDZ9z4DgklQCsVoGzoyDuRbr+4w0sMjtJuSthAJ78FX+h7yJQO
         HNDQTEg8KTCocV1f786gSfGQB/a8gxxk2aoNwf1BhHhCqM0cX9aobzcDZ5CW3rwfrVzz
         BGipwdSGZa6Uf+DUpq+962TRCPv9YaWhsaFqQ=
MIME-Version: 1.0
Received: by 10.224.67.72 with SMTP id q8mr9625250qai.222.1294280994325; Wed,
 05 Jan 2011 18:29:54 -0800 (PST)
Received: by 10.220.118.20 with HTTP; Wed, 5 Jan 2011 18:29:54 -0800 (PST)
Date: Wed, 5 Jan 2011 21:29:54 -0500
Message-ID: <AANLkTi=B1TR1yAmEqrf5PKi-=NRABB12T-7+6Ja_LJws@mail.gmail.com>
Subject: Moving to a new namenode
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
X-Virus-Checked: Checked by ClamAV on apache.org

What is the correct procedure to move to a new namenode? My current
namenode is running on a 2 core with 4gb of memory. I am moving to a
new host,  8 core x 32 GB of memory. My question are: what is the
proper way to move to the new namenode, and what can I do to enhance
the performance of it since the new hardware is much superior than the
old one.

From general-return-2672-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 06 03:40:49 2011
Return-Path: <general-return-2672-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42482 invoked from network); 6 Jan 2011 03:40:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2011 03:40:46 -0000
Received: (qmail 89625 invoked by uid 500); 6 Jan 2011 03:40:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89523 invoked by uid 500); 6 Jan 2011 03:40:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89499 invoked by uid 99); 6 Jan 2011 03:40:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 03:40:43 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 03:40:34 +0000
Received: by vws18 with SMTP id 18so6342222vws.35
        for <multiple recipients>; Wed, 05 Jan 2011 19:40:12 -0800 (PST)
Received: by 10.220.188.201 with SMTP id db9mr6349960vcb.279.1294285212626;
        Wed, 05 Jan 2011 19:40:12 -0800 (PST)
Received: from [10.172.5.187] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id w7sm5206914vch.44.2011.01.05.19.40.09
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Wed, 05 Jan 2011 19:40:11 -0800 (PST)
From: Ian Holsman <hadoop@holsman.net>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: Please join me in welcoming the following people as committers to the Hadoop project
Date: Thu, 6 Jan 2011 14:40:04 +1100
Message-Id: <93F81E22-00DB-451B-B6F2-BDB0EB550998@Holsman.NET>
To: mapreduce-dev@hadoop.apache.org,
 general@hadoop.apache.org,
 hdfs-dev@hadoop.apache.org
Mime-Version: 1.0 (Apple Message framework v1082)
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

On behalf of the Apache Hadoop PMC, I would like to extend a warm =
welcome to the following people,
who have all chosen to accept the role of committers on Hadoop.

In no alphabetical order:
=20
- Aaron Kimball
- Allen Wittenauer=20
- Amar Kamat
- Dmytro Molkov
- Jitendra Pandey
- Kan Zhang
- Ravi Gummadi
- Sreekanth Ramakrishna
- Todd Lipcon

I appreciate all the hard work these people have put into the project so =
far, and look forward to future contributions they will make to Hadoop =
in the future

Well done guys!


--Ian


From general-return-2673-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 06 04:09:51 2011
Return-Path: <general-return-2673-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58569 invoked from network); 6 Jan 2011 04:09:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2011 04:09:50 -0000
Received: (qmail 8505 invoked by uid 500); 6 Jan 2011 04:09:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8168 invoked by uid 500); 6 Jan 2011 04:09:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7773 invoked by uid 99); 6 Jan 2011 04:09:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 04:09:46 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 04:09:41 +0000
Received: by fxm2 with SMTP id 2so15481398fxm.35
        for <multiple recipients>; Wed, 05 Jan 2011 20:09:20 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:cc:content-type;
        bh=exbgC/5acdnE/Ajqq87i6AzKtv3V8z7TTrwecrNq+98=;
        b=L/2O25grF6PxSxxdyQR5RHgtkLdmCY9bJ+BqGGLOFPzaR+CEymTCOQWarVotm3xXMh
         CclG7MjLa33L6HqdLX615vI10NraHfAJVaydg794IWaHhD6DceHsGmIRWXYTiACLxzpW
         WgWGec16bX5mL76KAiNhSanAxKCD/Z4NaDBA8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:cc:content-type;
        b=MaZ7Q49eRvyXw2RMukWq9VzdnmWrxgodKeYdMLZcL8QaTEWUNwd+dOPnAuy+X2ves+
         8AfZ6y/rl715loufekqovhuIN+pgLPy/SWMMFof0JOCCsxjoNCvdigr2AOR/u6Ovzotp
         umTsEIT0VogKwlG3gQziANkY9df9lPDE40sww=
MIME-Version: 1.0
Received: by 10.223.96.199 with SMTP id i7mr758939fan.56.1294286960385; Wed,
 05 Jan 2011 20:09:20 -0800 (PST)
Sender: saint.ack@gmail.com
Received: by 10.223.83.136 with HTTP; Wed, 5 Jan 2011 20:09:20 -0800 (PST)
In-Reply-To: <93F81E22-00DB-451B-B6F2-BDB0EB550998@Holsman.NET>
References: <93F81E22-00DB-451B-B6F2-BDB0EB550998@Holsman.NET>
Date: Wed, 5 Jan 2011 20:09:20 -0800
X-Google-Sender-Auth: vzlBHJ_bY6IBpL6zR-ZjBKxFw3E
Message-ID: <AANLkTi=t3XoeHsyL38+wMzD-mitM8UM6JPBWPPay0pOu@mail.gmail.com>
Subject: Re: Please join me in welcoming the following people as committers to
 the Hadoop project
From: Stack <stack@duboce.net>
To: hdfs-dev@hadoop.apache.org
Cc: mapreduce-dev@hadoop.apache.org, general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Congrats lads.
St.Ack

On Wed, Jan 5, 2011 at 7:40 PM, Ian Holsman <hadoop@holsman.net> wrote:
> On behalf of the Apache Hadoop PMC, I would like to extend a warm welcome to the following people,
> who have all chosen to accept the role of committers on Hadoop.
>
> In no alphabetical order:
>
> - Aaron Kimball
> - Allen Wittenauer
> - Amar Kamat
> - Dmytro Molkov
> - Jitendra Pandey
> - Kan Zhang
> - Ravi Gummadi
> - Sreekanth Ramakrishna
> - Todd Lipcon
>
> I appreciate all the hard work these people have put into the project so far, and look forward to future contributions they will make to Hadoop in the future
>
> Well done guys!
>
>
> --Ian
>
>

From general-return-2674-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 06 04:11:45 2011
Return-Path: <general-return-2674-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59180 invoked from network); 6 Jan 2011 04:11:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2011 04:11:44 -0000
Received: (qmail 10679 invoked by uid 500); 6 Jan 2011 04:11:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 10559 invoked by uid 500); 6 Jan 2011 04:11:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 10536 invoked by uid 99); 6 Jan 2011 04:11:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 04:11:42 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 04:11:35 +0000
Received: by fxm2 with SMTP id 2so15482369fxm.35
        for <general@hadoop.apache.org>; Wed, 05 Jan 2011 20:11:14 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:content-type:content-transfer-encoding;
        bh=O5QTmSS58FPN61V7P9JBie3xzci+ayaoiMjKHmYNk9Y=;
        b=j/SpjnyDPkyG7puWfgHHPWyn6FZweuf1DKe5AGonVJWc5j/qzCVDSbqz1II3WO2P0P
         yY9xxP5EYafAZGZvJl6HmTZ1QcwDVD1EC/YzqvwHR0wc+uO0na7IDQe2n4ra1Mi29ood
         SMtU7LSceAeUHCiaVEU5R6xB3StXkpIJOKUmI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=rXBQ/vg9+x68ggCuB6F3yU9/mHVDvf3oLGARsM1qlgA55hw5+6dYTk5JfK1gtxWa9u
         y234H80jTKxayhBCjQJqIkGhZb19/eD+3grJTZBtCTbkUItUg2Pf7tSpvo4zmLTIi9zT
         U5MccaqvByRF2asMaKQ6b/japseJpyAxHJzOI=
MIME-Version: 1.0
Received: by 10.223.83.11 with SMTP id d11mr3073436fal.37.1294287074538; Wed,
 05 Jan 2011 20:11:14 -0800 (PST)
Sender: saint.ack@gmail.com
Received: by 10.223.83.136 with HTTP; Wed, 5 Jan 2011 20:11:14 -0800 (PST)
In-Reply-To: <0C437F3B-1577-49E1-9C6A-7D4AA53ED9D9@mac.com>
References: <C8E49ED7.447A0%mahadev@yahoo-inc.com>
	<0C437F3B-1577-49E1-9C6A-7D4AA53ED9D9@mac.com>
Date: Wed, 5 Jan 2011 20:11:14 -0800
X-Google-Sender-Auth: 1nGplt5-mcHStbjT-OnxLEurmcE
Message-ID: <AANLkTi=rzF3G-J=2_GB75NJ3PjqWtV_Cms3n+UHJ6GF-@mail.gmail.com>
Subject: Re: Patch testing
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I'll give you $5.00 Nigel if you add HBase to the list below.
St.Ack

On Wed, Oct 20, 2010 at 1:31 PM, Nigel Daley <ndaley@mac.com> wrote:
> I think ZK, PIG, etc will also be included in the update I'm working on.
>
> Cheers,
> Nige
>
> On Oct 20, 2010, at 1:27 PM, Mahadev Konar wrote:
>
>> Great.
>>
>> Nigel,
>> Can you please document in somewhere on how you fixed it? We=B9d like to=
 fix
>> it for ZooKeeper as well.
>>
>> Thanks
>> mahadev
>>
>>
>> On 10/20/10 1:23 PM, "Owen O'Malley" <omalley@apache.org> wrote:
>>
>>>
>>>
>>> On Oct 20, 2010, at 12:47 PM, Nigel Daley wrote:
>>>
>>>> I'm working to get the pre-commit patch testing running again for
>>>> HDFS, HADOOP, and MAPREDUCE patches.
>>>
>>> That would be great, Nigel.
>>>
>>> Thanks,
>>> =A0 =A0Owen
>>>
>>
>
>

From general-return-2675-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 06 04:35:10 2011
Return-Path: <general-return-2675-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63320 invoked from network); 6 Jan 2011 04:35:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2011 04:35:08 -0000
Received: (qmail 22233 invoked by uid 500); 6 Jan 2011 04:35:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22023 invoked by uid 500); 6 Jan 2011 04:35:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22002 invoked by uid 99); 6 Jan 2011 04:35:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 04:35:05 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jaybooth@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 04:34:59 +0000
Received: by qwh6 with SMTP id 6so16982636qwh.35
        for <multiple recipients>; Wed, 05 Jan 2011 20:34:39 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:cc:content-type;
        bh=lyG9fJdUD4k+5lC0b7mXnekCm4beVYWrzoUaFjRdK2w=;
        b=fLjlqH9p4Tg4LwSnwK1VFqjxUM2e0ChbcVoz9ddt+scaTdMJPb5vpogLgVwxiM8QZ0
         MuiZEqkUnZhR0dtX/4qfRmDUrGtEYJ0eskE4syO+kmfAMuu8/LsWlbD8bAYQ51Bupv1H
         zo7UoupNnsXzVOFJhLbPbe6TEOG7vq7emmCzA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=qnF0Yq8qYobYiwLL/xboqBWuj6K0868GzkH0G/a9rVS6YTDFcUcRBnOcnAGlG/TBae
         P4b697fFed+6eILtIQds2W8SKKKER3/I2tQGJEV5BIUSIzYNPVDKw+loa3QsO4FeQ+vH
         ZcT5yo8A0J34fq0r9atnOC/E+0AtAmnsAGgEg=
MIME-Version: 1.0
Received: by 10.229.235.142 with SMTP id kg14mr20948353qcb.133.1294288478926;
 Wed, 05 Jan 2011 20:34:38 -0800 (PST)
Received: by 10.220.185.129 with HTTP; Wed, 5 Jan 2011 20:34:38 -0800 (PST)
In-Reply-To: <AANLkTi=t3XoeHsyL38+wMzD-mitM8UM6JPBWPPay0pOu@mail.gmail.com>
References: <93F81E22-00DB-451B-B6F2-BDB0EB550998@Holsman.NET>
	<AANLkTi=t3XoeHsyL38+wMzD-mitM8UM6JPBWPPay0pOu@mail.gmail.com>
Date: Wed, 5 Jan 2011 23:34:38 -0500
Message-ID: <AANLkTimXUy11wT2he2BL4FfXDN2cTyi8+iMpPnM8djEU@mail.gmail.com>
Subject: Re: Please join me in welcoming the following people as committers to
 the Hadoop project
From: Jay Booth <jaybooth@gmail.com>
To: hdfs-dev@hadoop.apache.org
Cc: mapreduce-dev@hadoop.apache.org, general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64dda7605f92f0499260592
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e64dda7605f92f0499260592
Content-Type: text/plain; charset=ISO-8859-1

Congrats, all!

On Wed, Jan 5, 2011 at 11:09 PM, Stack <stack@duboce.net> wrote:

> Congrats lads.
> St.Ack
>
> On Wed, Jan 5, 2011 at 7:40 PM, Ian Holsman <hadoop@holsman.net> wrote:
> > On behalf of the Apache Hadoop PMC, I would like to extend a warm welcome
> to the following people,
> > who have all chosen to accept the role of committers on Hadoop.
> >
> > In no alphabetical order:
> >
> > - Aaron Kimball
> > - Allen Wittenauer
> > - Amar Kamat
> > - Dmytro Molkov
> > - Jitendra Pandey
> > - Kan Zhang
> > - Ravi Gummadi
> > - Sreekanth Ramakrishna
> > - Todd Lipcon
> >
> > I appreciate all the hard work these people have put into the project so
> far, and look forward to future contributions they will make to Hadoop in
> the future
> >
> > Well done guys!
> >
> >
> > --Ian
> >
> >
>

--0016e64dda7605f92f0499260592--

From general-return-2676-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 06 04:36:37 2011
Return-Path: <general-return-2676-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63789 invoked from network); 6 Jan 2011 04:36:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2011 04:36:36 -0000
Received: (qmail 25103 invoked by uid 500); 6 Jan 2011 04:36:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24470 invoked by uid 500); 6 Jan 2011 04:36:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24444 invoked by uid 99); 6 Jan 2011 04:36:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 04:36:29 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 04:36:23 +0000
Received: by iyb26 with SMTP id 26so17297311iyb.35
        for <multiple recipients>; Wed, 05 Jan 2011 20:36:01 -0800 (PST)
MIME-Version: 1.0
Received: by 10.42.218.6 with SMTP id ho6mr24686786icb.12.1294288561762; Wed,
 05 Jan 2011 20:36:01 -0800 (PST)
Received: by 10.42.167.66 with HTTP; Wed, 5 Jan 2011 20:36:01 -0800 (PST)
In-Reply-To: <AANLkTimXUy11wT2he2BL4FfXDN2cTyi8+iMpPnM8djEU@mail.gmail.com>
References: <93F81E22-00DB-451B-B6F2-BDB0EB550998@Holsman.NET>
	<AANLkTi=t3XoeHsyL38+wMzD-mitM8UM6JPBWPPay0pOu@mail.gmail.com>
	<AANLkTimXUy11wT2he2BL4FfXDN2cTyi8+iMpPnM8djEU@mail.gmail.com>
Date: Wed, 5 Jan 2011 20:36:01 -0800
Message-ID: <AANLkTi=DyyyZ1hHejGUL4TUMJzPAuRJhnVBoOY_GZeKv@mail.gmail.com>
Subject: Re: Please join me in welcoming the following people as committers to
 the Hadoop project
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Cc: hdfs-dev@hadoop.apache.org, mapreduce-dev@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf3054a045f5f1230499260984
X-Virus-Checked: Checked by ClamAV on apache.org

--20cf3054a045f5f1230499260984
Content-Type: text/plain; charset=ISO-8859-1

Congratulations! Great to see the community growing once again.

On Wed, Jan 5, 2011 at 8:34 PM, Jay Booth <jaybooth@gmail.com> wrote:

> Congrats, all!
>
> On Wed, Jan 5, 2011 at 11:09 PM, Stack <stack@duboce.net> wrote:
>
> > Congrats lads.
> > St.Ack
> >
> > On Wed, Jan 5, 2011 at 7:40 PM, Ian Holsman <hadoop@holsman.net> wrote:
> > > On behalf of the Apache Hadoop PMC, I would like to extend a warm
> welcome
> > to the following people,
> > > who have all chosen to accept the role of committers on Hadoop.
> > >
> > > In no alphabetical order:
> > >
> > > - Aaron Kimball
> > > - Allen Wittenauer
> > > - Amar Kamat
> > > - Dmytro Molkov
> > > - Jitendra Pandey
> > > - Kan Zhang
> > > - Ravi Gummadi
> > > - Sreekanth Ramakrishna
> > > - Todd Lipcon
> > >
> > > I appreciate all the hard work these people have put into the project
> so
> > far, and look forward to future contributions they will make to Hadoop in
> > the future
> > >
> > > Well done guys!
> > >
> > >
> > > --Ian
> > >
> > >
> >
>

--20cf3054a045f5f1230499260984--

From general-return-2677-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 06 04:51:01 2011
Return-Path: <general-return-2677-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66674 invoked from network); 6 Jan 2011 04:51:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2011 04:51:01 -0000
Received: (qmail 37725 invoked by uid 500); 6 Jan 2011 04:50:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37548 invoked by uid 500); 6 Jan 2011 04:50:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37540 invoked by uid 99); 6 Jan 2011 04:50:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 04:50:58 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,MIME_QP_LONG_LINE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.99 as permitted sender)
Received: from [17.148.16.99] (HELO asmtpout024.mac.com) (17.148.16.99)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 04:50:52 +0000
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Received: from [10.38.43.202]
 (166-205-139-112.mobile.mymmode.com [166.205.139.112])
 by asmtp024.mac.com (Oracle Communications Messaging Exchange Server 7u4-18.01
 64bit (built Jul 15 2010)) with ESMTPSA id <0LEL009SG420FL20@asmtp024.mac.com>
 for general@hadoop.apache.org; Wed, 05 Jan 2011 20:49:32 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-06_03:2011-01-06,2011-01-06,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101050149
Subject: Re: Patch testing
References: <C8E49ED7.447A0%mahadev@yahoo-inc.com>
 <0C437F3B-1577-49E1-9C6A-7D4AA53ED9D9@mac.com>
 <AANLkTi=rzF3G-J=2_GB75NJ3PjqWtV_Cms3n+UHJ6GF-@mail.gmail.com>
From: Nigel Daley <ndaley@mac.com>
X-Mailer: iPhone Mail (8C148)
In-reply-to: <AANLkTi=rzF3G-J=2_GB75NJ3PjqWtV_Cms3n+UHJ6GF-@mail.gmail.com>
Message-id: <4EF902B7-B44F-4133-A623-6295EA729E9A@mac.com>
Date: Wed, 05 Jan 2011 20:46:54 -0800
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Content-transfer-encoding: quoted-printable

Lol. At least now we know how much it's valued. I could get a foot long from=
 subway.=20

Cheers,
Nige

On Jan 5, 2011, at 8:11 PM, Stack <stack@duboce.net> wrote:

> I'll give you $5.00 Nigel if you add HBase to the list below.
> St.Ack
>=20
> On Wed, Oct 20, 2010 at 1:31 PM, Nigel Daley <ndaley@mac.com> wrote:
>> I think ZK, PIG, etc will also be included in the update I'm working on.
>>=20
>> Cheers,
>> Nige
>>=20
>> On Oct 20, 2010, at 1:27 PM, Mahadev Konar wrote:
>>=20
>>> Great.
>>>=20
>>> Nigel,
>>> Can you please document in somewhere on how you fixed it? We=C2=B9d like=
 to fix
>>> it for ZooKeeper as well.
>>>=20
>>> Thanks
>>> mahadev
>>>=20
>>>=20
>>> On 10/20/10 1:23 PM, "Owen O'Malley" <omalley@apache.org> wrote:
>>>=20
>>>>=20
>>>>=20
>>>> On Oct 20, 2010, at 12:47 PM, Nigel Daley wrote:
>>>>=20
>>>>> I'm working to get the pre-commit patch testing running again for
>>>>> HDFS, HADOOP, and MAPREDUCE patches.
>>>>=20
>>>> That would be great, Nigel.
>>>>=20
>>>> Thanks,
>>>>    Owen
>>>>=20
>>>=20
>>=20
>>=20

From general-return-2678-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 06 04:54:11 2011
Return-Path: <general-return-2678-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67549 invoked from network); 6 Jan 2011 04:54:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2011 04:54:11 -0000
Received: (qmail 40963 invoked by uid 500); 6 Jan 2011 04:54:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40840 invoked by uid 500); 6 Jan 2011 04:54:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40827 invoked by uid 99); 6 Jan 2011 04:54:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 04:54:09 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of li.j2ee@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 04:54:04 +0000
Received: by bwz8 with SMTP id 8so16270152bwz.35
        for <multiple recipients>; Wed, 05 Jan 2011 20:53:42 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:cc:content-type;
        bh=JI6Xyzzajo/CBPgjSH9FDwhrbh5ssyb4CRLuRbZBg0k=;
        b=cqtknkCde1ocWJGRr466hApNJBi4Cq/G+4UgPU1CO59UcxlvYxScpzndEtpf8pNOKb
         L+ObIPqKy0atvtnOqfQjUFcuQsmXpYrzLZJTb6gnNsfQYqheYD9CBEJkOWK7VAjbO99Z
         ECBVpzHWdzy5GmudKR+lxCf2ScWmGBv5JCL2E=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=GHivsj/4SNaO1adOU+njtDK8bFdfAvlBfBtsKVCh56yor7brjExpfEYfjCXPznv64F
         T2SVzkCvYm+KHvKtNAZjX0Wsw1rrFo01CmSvsmS7f8omdea1+zjlKwzCcK1LtnN4VpGD
         UDD/tSRrMl34f56JyVq3/fd0zuIqd1YEGp/Ko=
MIME-Version: 1.0
Received: by 10.204.72.207 with SMTP id n15mr10691621bkj.62.1294289622734;
 Wed, 05 Jan 2011 20:53:42 -0800 (PST)
Received: by 10.204.84.144 with HTTP; Wed, 5 Jan 2011 20:53:42 -0800 (PST)
In-Reply-To: <93F81E22-00DB-451B-B6F2-BDB0EB550998@Holsman.NET>
References: <93F81E22-00DB-451B-B6F2-BDB0EB550998@Holsman.NET>
Date: Thu, 6 Jan 2011 12:53:42 +0800
Message-ID: <AANLkTinDatiqh26X34LoGvw0_rwyVfau4kY7xF-oerk=@mail.gmail.com>
Subject: Re: Please join me in welcoming the following people as committers to
 the Hadoop project
From: li ping <li.j2ee@gmail.com>
To: general@hadoop.apache.org
Cc: mapreduce-dev@hadoop.apache.org, hdfs-dev@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636d34c4433177704992649d0

--001636d34c4433177704992649d0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

congratulations

On Thu, Jan 6, 2011 at 11:40 AM, Ian Holsman <hadoop@holsman.net> wrote:

> On behalf of the Apache Hadoop PMC, I would like to extend a warm welcome
> to the following people,
> who have all chosen to accept the role of committers on Hadoop.
>
> In no alphabetical order:
>
> - Aaron Kimball
> - Allen Wittenauer
> - Amar Kamat
> - Dmytro Molkov
> - Jitendra Pandey
> - Kan Zhang
> - Ravi Gummadi
> - Sreekanth Ramakrishna
> - Todd Lipcon
>
> I appreciate all the hard work these people have put into the project so
> far, and look forward to future contributions they will make to Hadoop in
> the future
>
> Well done guys!
>
>
> --Ian
>
>


--=20
-----=E6=9D=8E=E5=B9=B3

--001636d34c4433177704992649d0--

From general-return-2679-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 06 05:00:06 2011
Return-Path: <general-return-2679-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69171 invoked from network); 6 Jan 2011 05:00:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2011 05:00:05 -0000
Received: (qmail 48396 invoked by uid 500); 6 Jan 2011 05:00:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48341 invoked by uid 500); 6 Jan 2011 05:00:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48319 invoked by uid 99); 6 Jan 2011 05:00:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 05:00:03 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.213.48] (HELO mail-yw0-f48.google.com) (209.85.213.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 05:00:03 +0000
Received: by ywo7 with SMTP id 7so7494138ywo.35
        for <general@hadoop.apache.org>; Wed, 05 Jan 2011 20:59:40 -0800 (PST)
Received: by 10.150.145.7 with SMTP id s7mr23463102ybd.251.1294289979834;
        Wed, 05 Jan 2011 20:59:39 -0800 (PST)
Received: from tp (c-24-4-185-157.hsd1.ca.comcast.net [24.4.185.157])
        by mx.google.com with ESMTPS id v39sm15105817yba.7.2011.01.05.20.59.37
        (version=SSLv3 cipher=RC4-MD5);
        Wed, 05 Jan 2011 20:59:38 -0800 (PST)
Sender: Konstantin Boudnik <cos@boudnik.org>
Date: Wed, 5 Jan 2011 20:59:34 -0800
From: Konstantin Boudnik <cos@apache.org>
To: general@hadoop.apache.org
Subject: Re: Patch testing
Message-ID: <20110106045934.GA14065@tp>
References: <C8E49ED7.447A0%mahadev@yahoo-inc.com>
 <0C437F3B-1577-49E1-9C6A-7D4AA53ED9D9@mac.com>
 <AANLkTi=rzF3G-J=2_GB75NJ3PjqWtV_Cms3n+UHJ6GF-@mail.gmail.com>
 <4EF902B7-B44F-4133-A623-6295EA729E9A@mac.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature"; boundary="sdtB3X0nJg68CQEu"
Content-Disposition: inline
In-Reply-To: <4EF902B7-B44F-4133-A623-6295EA729E9A@mac.com>
X-Organization: It's something of 'Cos
X-PGP-Key: http://www.boudnik.org/~cos/pubkey.asc
User-Agent: Mutt/1.5.20 (2009-06-14)

--sdtB3X0nJg68CQEu
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Wed, Jan 05, 2011 at 08:46PM, Nigel Daley wrote:
> Lol. At least now we know how much it's valued. I could get a foot long f=
rom subway.=20

But hurry Nige - pretty soon it's gonna be worthless ;(

> Cheers,
> Nige
>=20
> On Jan 5, 2011, at 8:11 PM, Stack <stack@duboce.net> wrote:
>=20
> > I'll give you $5.00 Nigel if you add HBase to the list below.
> > St.Ack
> >=20
> > On Wed, Oct 20, 2010 at 1:31 PM, Nigel Daley <ndaley@mac.com> wrote:
> >> I think ZK, PIG, etc will also be included in the update I'm working o=
n.
> >>=20
> >> Cheers,
> >> Nige
> >>=20
> >> On Oct 20, 2010, at 1:27 PM, Mahadev Konar wrote:
> >>=20
> >>> Great.
> >>>=20
> >>> Nigel,
> >>> Can you please document in somewhere on how you fixed it? We=C2=B9d l=
ike to fix
> >>> it for ZooKeeper as well.
> >>>=20
> >>> Thanks
> >>> mahadev
> >>>=20
> >>>=20
> >>> On 10/20/10 1:23 PM, "Owen O'Malley" <omalley@apache.org> wrote:
> >>>=20
> >>>>=20
> >>>>=20
> >>>> On Oct 20, 2010, at 12:47 PM, Nigel Daley wrote:
> >>>>=20
> >>>>> I'm working to get the pre-commit patch testing running again for
> >>>>> HDFS, HADOOP, and MAPREDUCE patches.
> >>>>=20
> >>>> That would be great, Nigel.
> >>>>=20
> >>>> Thanks,
> >>>>    Owen
> >>>>=20
> >>>=20
> >>=20
> >>=20

--sdtB3X0nJg68CQEu
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iF4EAREIAAYFAk0lTDYACgkQenyFlstYjhJAOQD9HuO3zaOmzCBAZm2bqNcO7Cdu
uFZpeWZRC56wXyk/oiwBAJP3rOkuu3Gc3kLFy5Vf/3PX9gh3zGCw/0coikBGcsRh
=6ejt
-----END PGP SIGNATURE-----

--sdtB3X0nJg68CQEu--

From general-return-2680-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 06 05:15:08 2011
Return-Path: <general-return-2680-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82307 invoked from network); 6 Jan 2011 05:15:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2011 05:15:08 -0000
Received: (qmail 61728 invoked by uid 500); 6 Jan 2011 05:15:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61545 invoked by uid 500); 6 Jan 2011 05:15:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61534 invoked by uid 99); 6 Jan 2011 05:15:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 05:15:06 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 05:14:58 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p065E4JG089230
	for <general@hadoop.apache.org>; Wed, 5 Jan 2011 21:14:05 -0800 (PST)
Received: from lifthappen-lm.corp.yahoo.com (10.72.113.229) by
 SP2-EX07CAS01.ds.corp.yahoo.com (98.137.59.4) with Microsoft SMTP Server
 (TLS) id 8.3.106.1; Wed, 5 Jan 2011 21:14:04 -0800
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Moving to a new namenode
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <AANLkTi=B1TR1yAmEqrf5PKi-=NRABB12T-7+6Ja_LJws@mail.gmail.com>
Date: Wed, 5 Jan 2011 21:14:04 -0800
Content-Transfer-Encoding: 7bit
Message-ID: <A1782BC8-D13E-42C1-8AE1-801C8ACE3753@yahoo-inc.com>
References: <AANLkTi=B1TR1yAmEqrf5PKi-=NRABB12T-7+6Ja_LJws@mail.gmail.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)

could you move this traffic to HDFS?

thanks!

On Jan 5, 2011, at 6:29 PM, Mag Gam wrote:

> What is the correct procedure to move to a new namenode? My current
> namenode is running on a 2 core with 4gb of memory. I am moving to a
> new host,  8 core x 32 GB of memory. My question are: what is the
> proper way to move to the new namenode, and what can I do to enhance
> the performance of it since the new hardware is much superior than the
> old one.


From general-return-2681-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 06 07:22:58 2011
Return-Path: <general-return-2681-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47322 invoked from network); 6 Jan 2011 07:22:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2011 07:22:57 -0000
Received: (qmail 56634 invoked by uid 500); 6 Jan 2011 07:22:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56144 invoked by uid 500); 6 Jan 2011 07:22:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56118 invoked by uid 99); 6 Jan 2011 07:22:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 07:22:54 +0000
X-ASF-Spam-Status: No, hits=3.7 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of deepakmca05@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 07:22:49 +0000
Received: by wye20 with SMTP id 20so17021057wye.35
        for <multiple recipients>; Wed, 05 Jan 2011 23:22:27 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:cc:content-type;
        bh=nMmVbQwwiewp4xLvTWWpcCGMndhWuHV87LFrqS+nFgU=;
        b=HWC0aq/Uxjm58Z31CIAYm1QZCf81D/AnHOR/6QkmBJe5Q4XtOnTRNdJkYSVJcFfw5o
         HU89ipk0Aoku0BFqcaDHsHorl+z7TPDrFMgh83f2wBXXBL7KmnV7e0Yg+GRVarhD3aYp
         pTBVuwoCMSCHaWQ6TkYNRtj7+5t1X9fwaoKek=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :cc:content-type;
        b=ofDdp7Qy+3XE/+WB+FAni97PYCX/thIybn7WZbUrABvnRPjxNvO7n+aLEIikTg8TGP
         gzOg+DAAPHEB6REJ/7fig2DvGuDS+bSMj5kCWOgoCVpO5EKvrHhjSEQXs4zOJdLqSHEj
         Q1Di2X6r6yyXtCOPlGGEQrdyL+SlEH0pA5qww=
Received: by 10.227.146.149 with SMTP id h21mr13997561wbv.43.1294298547764;
 Wed, 05 Jan 2011 23:22:27 -0800 (PST)
MIME-Version: 1.0
Received: by 10.227.69.206 with HTTP; Wed, 5 Jan 2011 23:22:07 -0800 (PST)
In-Reply-To: <AANLkTinDatiqh26X34LoGvw0_rwyVfau4kY7xF-oerk=@mail.gmail.com>
References: <93F81E22-00DB-451B-B6F2-BDB0EB550998@Holsman.NET> <AANLkTinDatiqh26X34LoGvw0_rwyVfau4kY7xF-oerk=@mail.gmail.com>
From: Deepak Sharma <deepakmca05@gmail.com>
Date: Thu, 6 Jan 2011 12:52:07 +0530
Message-ID: <AANLkTinG1b5fP7XsBco7DAH5npqA0kECPR830x+tipWN@mail.gmail.com>
Subject: Re: Please join me in welcoming the following people as committers to
 the Hadoop project
To: mapreduce-dev@hadoop.apache.org
Cc: general@hadoop.apache.org, hdfs-dev@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163683394a2c3f010499285de1

--00163683394a2c3f010499285de1
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

congrats all..I hope i'll be committer some day as well:)

On Thu, Jan 6, 2011 at 10:23 AM, li ping <li.j2ee@gmail.com> wrote:

> congratulations
>
> On Thu, Jan 6, 2011 at 11:40 AM, Ian Holsman <hadoop@holsman.net> wrote:
>
> > On behalf of the Apache Hadoop PMC, I would like to extend a warm welco=
me
> > to the following people,
> > who have all chosen to accept the role of committers on Hadoop.
> >
> > In no alphabetical order:
> >
> > - Aaron Kimball
> > - Allen Wittenauer
> > - Amar Kamat
> > - Dmytro Molkov
> > - Jitendra Pandey
> > - Kan Zhang
> > - Ravi Gummadi
> > - Sreekanth Ramakrishna
> > - Todd Lipcon
> >
> > I appreciate all the hard work these people have put into the project s=
o
> > far, and look forward to future contributions they will make to Hadoop =
in
> > the future
> >
> > Well done guys!
> >
> >
> > --Ian
> >
> >
>
>
> --
> -----=E6=9D=8E=E5=B9=B3
>



--=20
Deepak Sharma
http://www.linkedin.com/in/rikindia

--00163683394a2c3f010499285de1--

From general-return-2682-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 06 09:43:07 2011
Return-Path: <general-return-2682-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95113 invoked from network); 6 Jan 2011 09:43:06 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2011 09:43:06 -0000
Received: (qmail 2925 invoked by uid 500); 6 Jan 2011 09:43:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2728 invoked by uid 500); 6 Jan 2011 09:43:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 92290 invoked by uid 99); 6 Jan 2011 09:27:39 -0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of dhaval.makawana@gmail.com designates 209.85.161.48 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:cc:content-type;
        bh=TxyNiS47T9jd0JtJPPrmL4yXZWaK/cqzHbcVojl6f5Q=;
        b=VUDTLMlq1qguvcu5P03nyysKoTDZaL5hJec8UxYde80CkngEYip8ip5sbAtZX5o708
         kv3wRjsz5Oqg+ToAGt56V4UsuJFE60yiQSp+sdWMkw792qwvLHAhvkllw6Pc2PCygzQ8
         GyGeOUx/VpOPsVd2KlsS9KirVKxu0sKDhp+H0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :cc:content-type;
        b=AZyHqu2rZdmb2YNwYQZGB2RAFlcSiW7tc+rvn6xjG8wPO5VFFeKVDU0D9Gb4OR22an
         zYQRNeAhUYJqPqwrRZw4sfyJT2NpfVpWIR2jwWwB9oQEo52se4x4Ls56fSbNGnZrBeAQ
         HJmG/SC7G7kl++RifZMcEyMjzNu/P9Mr6ChLU=
MIME-Version: 1.0
In-Reply-To: <AANLkTi=EPH2VqAAN5stfnyy3tFTnuV18XOCLRHrHE8TT@mail.gmail.com>
References: <AANLkTi=EPH2VqAAN5stfnyy3tFTnuV18XOCLRHrHE8TT@mail.gmail.com>
From: Dhaval Makawana <dhaval.makawana@gmail.com>
Date: Thu, 6 Jan 2011 14:56:51 +0530
Message-ID: <AANLkTinxeQgPap64WejAjPLAJLC0geW4QG7N-jvDBmfn@mail.gmail.com>
Subject: Reducer getting stuck after 66% processing
To: general@hadoop.apache.org
Cc: vinayakh@gmail.com
Content-Type: multipart/alternative; boundary=20cf3054ac85382dfd04992a1bac
X-Virus-Checked: Checked by ClamAV on apache.org

--20cf3054ac85382dfd04992a1bac
Content-Type: text/plain; charset=ISO-8859-1

Hi,



I have 1 GB of input data with 10954103 records going to 7 reducer tasks.
The cluster is set up is such that each reducer gets 1 GB of RAM and 256 MB
out of it is reserved for sorting.  The reducer simply sums up one field of
values and outputs all the values along with the sum value. The problem is
all reducers are getting stuck after around 66% of processing. Tail of task
tracker log reads as below.



2011-01-06 08:39:31,309 INFO org.apache.hadoop.mapred.ReduceTask:
Interleaved on-disk merge complete: 0 files left.

2011-01-06 08:39:31,309 INFO org.apache.hadoop.mapred.ReduceTask: In-memory
merge complete: 2 files left.

2011-01-06 08:39:31,333 INFO org.apache.hadoop.mapred.Merger: Merging 2
sorted segments

2011-01-06 08:39:31,334 INFO org.apache.hadoop.mapred.Merger: Down to the
last merge-pass, with 1 segments left of total size: 25352535 bytes

2011-01-06 08:39:31,343 INFO org.apache.hadoop.io.compress.CodecPool: Got
brand-new compressor

2011-01-06 08:39:31,704 INFO org.apache.hadoop.mapred.ReduceTask: Merged 2
segments, 25352537 bytes to disk to satisfy reduce memory limit

2011-01-06 08:39:31,704 INFO org.apache.hadoop.mapred.ReduceTask: Merging 1
files, 2947959 bytes from disk

2011-01-06 08:39:31,705 INFO org.apache.hadoop.mapred.ReduceTask: Merging 0
segments, 0 bytes from memory into reduce

2011-01-06 08:39:31,705 INFO org.apache.hadoop.mapred.Merger: Merging 1
sorted segments

2011-01-06 08:39:31,709 INFO org.apache.hadoop.mapred.Merger: Down to the
last merge-pass, with 1 segments left of total size: 2947955 bytes


Please help solving the problem.


Regards,

Dhaval

--20cf3054ac85382dfd04992a1bac--

From general-return-2683-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 06 10:05:09 2011
Return-Path: <general-return-2683-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 682 invoked from network); 6 Jan 2011 10:05:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2011 10:05:09 -0000
Received: (qmail 25847 invoked by uid 500); 6 Jan 2011 10:05:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25572 invoked by uid 500); 6 Jan 2011 10:05:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25552 invoked by uid 99); 6 Jan 2011 10:05:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 10:05:06 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of lars.george@gmail.com designates 209.85.215.176 as permitted sender)
Received: from [209.85.215.176] (HELO mail-ey0-f176.google.com) (209.85.215.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 10:05:00 +0000
Received: by eyz10 with SMTP id 10so6777968eyz.35
        for <multiple recipients>; Thu, 06 Jan 2011 02:04:40 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:cc:content-type
         :content-transfer-encoding;
        bh=r/qM5Ds5oZ2fKNi3MyvDqA4edA1Mk6EtIIyeEyace/w=;
        b=GE5k+Vc+uu9LEUHRvxZniRtcg+tjxIJxaImnr+Frtfe9BFRyT9iCd0mO9ew/cEAbtX
         OOcIIyxB9XfqpJFb9vtsGSFy3kaPP9m5VcJRoh6oFIdbhlZO/4UG8LObDm8jmRHaIPHD
         JC9Z2JuKPwKw5y4pCl6JNSORU0zE0uVYB0Gtg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type:content-transfer-encoding;
        b=fo0Yo4gsqHGlISJ/WUN1moa/z1CQMMVO2lEF22zot4xT7ypm1fJMLJ1T2apO4K79jH
         HcThoTKXFpd7y4I91ZCooPaLvNyF4z1eTcqnAUgKe4ycT6IFIeyzQ3PY84J5PZafcocF
         I543k1gpv1bkKTDfipddaJzYd6Zpw6inprkvM=
MIME-Version: 1.0
Received: by 10.213.22.142 with SMTP id n14mr17845656ebb.57.1294308280184;
 Thu, 06 Jan 2011 02:04:40 -0800 (PST)
Received: by 10.213.35.140 with HTTP; Thu, 6 Jan 2011 02:04:40 -0800 (PST)
In-Reply-To: <AANLkTinG1b5fP7XsBco7DAH5npqA0kECPR830x+tipWN@mail.gmail.com>
References: <93F81E22-00DB-451B-B6F2-BDB0EB550998@Holsman.NET>
	<AANLkTinDatiqh26X34LoGvw0_rwyVfau4kY7xF-oerk=@mail.gmail.com>
	<AANLkTinG1b5fP7XsBco7DAH5npqA0kECPR830x+tipWN@mail.gmail.com>
Date: Thu, 6 Jan 2011 11:04:40 +0100
Message-ID: <AANLkTimQ3=1QS627v-zC_1PGDNmRH0ecm3cUJzGgVzSw@mail.gmail.com>
Subject: Re: Please join me in welcoming the following people as committers to
 the Hadoop project
From: Lars George <lars.george@gmail.com>
To: general@hadoop.apache.org
Cc: mapreduce-dev@hadoop.apache.org, hdfs-dev@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Congratulations to all of you!

On Thu, Jan 6, 2011 at 8:22 AM, Deepak Sharma <deepakmca05@gmail.com> wrote=
:
> congrats all..I hope i'll be committer some day as well:)
>
> On Thu, Jan 6, 2011 at 10:23 AM, li ping <li.j2ee@gmail.com> wrote:
>
>> congratulations
>>
>> On Thu, Jan 6, 2011 at 11:40 AM, Ian Holsman <hadoop@holsman.net> wrote:
>>
>> > On behalf of the Apache Hadoop PMC, I would like to extend a warm welc=
ome
>> > to the following people,
>> > who have all chosen to accept the role of committers on Hadoop.
>> >
>> > In no alphabetical order:
>> >
>> > - Aaron Kimball
>> > - Allen Wittenauer
>> > - Amar Kamat
>> > - Dmytro Molkov
>> > - Jitendra Pandey
>> > - Kan Zhang
>> > - Ravi Gummadi
>> > - Sreekanth Ramakrishna
>> > - Todd Lipcon
>> >
>> > I appreciate all the hard work these people have put into the project =
so
>> > far, and look forward to future contributions they will make to Hadoop=
 in
>> > the future
>> >
>> > Well done guys!
>> >
>> >
>> > --Ian
>> >
>> >
>>
>>
>> --
>> -----=E6=9D=8E=E5=B9=B3
>>
>
>
>
> --
> Deepak Sharma
> http://www.linkedin.com/in/rikindia
>

From general-return-2684-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 06 12:35:30 2011
Return-Path: <general-return-2684-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72629 invoked from network); 6 Jan 2011 12:35:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2011 12:35:29 -0000
Received: (qmail 84720 invoked by uid 500); 6 Jan 2011 12:35:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84537 invoked by uid 500); 6 Jan 2011 12:35:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84529 invoked by uid 99); 6 Jan 2011 12:35:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 12:35:26 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of magawake@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 12:35:22 +0000
Received: by qwh6 with SMTP id 6so17286787qwh.35
        for <general@hadoop.apache.org>; Thu, 06 Jan 2011 04:35:01 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=GNKukIFKamPh999rM3aIXj6cWm5wBJBUb+aCNcNPZI0=;
        b=EXr4ugGs79YVSryasVKL1jHqESm/7W3WbifJzVP04snlTGLvzfZHDD0Zb8CiyHvjJB
         iuHajBfiZqUrVRAthVPklpnUrTN8Pg8xD7+lNQP1i5MUhSz3zE06KANG/07Mf1D3rfLO
         ofGEO2XD01XjJuuWRPnyK2l4x9zRnkC3K1KsU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=bNgYRe0o27Nc23aWPnIulUUnPp0xZtr3j+XtNMqPztDjpKC2b2kbopOrKYVvyK3/KL
         dzRAKNz7NOjPibZAQrimyZkYEYn5oUjGztSzK4X4dT9vhRpOIWTRuzzxedEDD5sfgtNG
         XaBNtoTUrgSVqL+vSJR5vSt9MidS1XWC8sQUA=
MIME-Version: 1.0
Received: by 10.224.199.6 with SMTP id eq6mr22596527qab.272.1294317301321;
 Thu, 06 Jan 2011 04:35:01 -0800 (PST)
Received: by 10.220.118.20 with HTTP; Thu, 6 Jan 2011 04:35:01 -0800 (PST)
In-Reply-To: <A1782BC8-D13E-42C1-8AE1-801C8ACE3753@yahoo-inc.com>
References: <AANLkTi=B1TR1yAmEqrf5PKi-=NRABB12T-7+6Ja_LJws@mail.gmail.com>
	<A1782BC8-D13E-42C1-8AE1-801C8ACE3753@yahoo-inc.com>
Date: Thu, 6 Jan 2011 07:35:01 -0500
Message-ID: <AANLkTimwS=ChaqcZtEt8WM0Vf4LUbm-D07BgwGGKYySq@mail.gmail.com>
Subject: Re: Moving to a new namenode
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

The current filesystem is in HDFS.

Do you mean, create a new namenode on the new server and assign some
data nodes to it and start moving the data off?


On Thu, Jan 6, 2011 at 12:14 AM, Eric Baldeschwieler
<eric14@yahoo-inc.com> wrote:
> could you move this traffic to HDFS?
>
> thanks!
>
> On Jan 5, 2011, at 6:29 PM, Mag Gam wrote:
>
>> What is the correct procedure to move to a new namenode? My current
>> namenode is running on a 2 core with 4gb of memory. I am moving to a
>> new host, =C2=A08 core x 32 GB of memory. My question are: what is the
>> proper way to move to the new namenode, and what can I do to enhance
>> the performance of it since the new hardware is much superior than the
>> old one.
>
>

From general-return-2685-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 06 12:43:58 2011
Return-Path: <general-return-2685-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 74548 invoked from network); 6 Jan 2011 12:43:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2011 12:43:58 -0000
Received: (qmail 96857 invoked by uid 500); 6 Jan 2011 12:43:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96750 invoked by uid 500); 6 Jan 2011 12:43:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96738 invoked by uid 99); 6 Jan 2011 12:43:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 12:43:56 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 12:43:50 +0000
Received: by fxm2 with SMTP id 2so15745376fxm.35
        for <multiple recipients>; Thu, 06 Jan 2011 04:43:30 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=QMmNE4I/BuwrRWp8DiD8NZu27Eg1xtKqzulbI/qBlAY=;
        b=BeDdhD3XDe2wCJLLIra85rTFuOPRWwFPRf7TpRk5GlJNAK2/jHfLdU394ZyZ/06Ev4
         5TCHP7LdO8vS7ZDDQtWVfpovOeM2MuLdoiVsBFy3GNjXPTeAQxl8zqJx1I3RDsFmoelU
         sFHwjtho1xv4FenwR1tf/etlp+qOWZHR6zO+0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=xz5FEimIY6/+nlGcCocyeGReIavFwNUKzwdjJ5WEV7gMe762D5rT3ifizQYNla0GGX
         m9KTv/OpWOwF2Iy+ypVNeJeqlvgaHq8RCC8gogJOrLvy1A8IUgxClj7xIJfWKVhLKFsV
         Qg2fg79OYpdUGqGp09LnCDOY3tLeoHVAR45nA=
Received: by 10.223.72.202 with SMTP id n10mr10685214faj.74.1294317808463;
 Thu, 06 Jan 2011 04:43:28 -0800 (PST)
MIME-Version: 1.0
Received: by 10.223.120.14 with HTTP; Thu, 6 Jan 2011 04:43:08 -0800 (PST)
In-Reply-To: <AANLkTimwS=ChaqcZtEt8WM0Vf4LUbm-D07BgwGGKYySq@mail.gmail.com>
References: <AANLkTi=B1TR1yAmEqrf5PKi-=NRABB12T-7+6Ja_LJws@mail.gmail.com>
 <A1782BC8-D13E-42C1-8AE1-801C8ACE3753@yahoo-inc.com> <AANLkTimwS=ChaqcZtEt8WM0Vf4LUbm-D07BgwGGKYySq@mail.gmail.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Thu, 6 Jan 2011 18:13:08 +0530
Message-ID: <AANLkTinSqm4Z1ZO9ETJPv2DMUoojJ0DGUg=e5HL=gXag@mail.gmail.com>
Subject: Re: Moving to a new namenode
To: hdfs-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

No, he meant to move the discussion to the hdfs-user@hadoop.apache.org
list for HDFS queries.

Sending to hdfs-user, bcc'ing to general. [This the right way?]

On Thu, Jan 6, 2011 at 6:05 PM, Mag Gam <magawake@gmail.com> wrote:
> The current filesystem is in HDFS.
>
> Do you mean, create a new namenode on the new server and assign some
> data nodes to it and start moving the data off?
>
>
> On Thu, Jan 6, 2011 at 12:14 AM, Eric Baldeschwieler
> <eric14@yahoo-inc.com> wrote:
>> could you move this traffic to HDFS?
>>
>> thanks!
>>
>> On Jan 5, 2011, at 6:29 PM, Mag Gam wrote:
>>
>>> What is the correct procedure to move to a new namenode? My current
>>> namenode is running on a 2 core with 4gb of memory. I am moving to a
>>> new host, =A08 core x 32 GB of memory. My question are: what is the
>>> proper way to move to the new namenode, and what can I do to enhance
>>> the performance of it since the new hardware is much superior than the
>>> old one.
>>
>>
>



--=20
Harsh J
www.harshj.com

From general-return-2686-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 06 12:57:44 2011
Return-Path: <general-return-2686-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77499 invoked from network); 6 Jan 2011 12:57:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2011 12:57:44 -0000
Received: (qmail 22237 invoked by uid 500); 6 Jan 2011 12:57:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22118 invoked by uid 500); 6 Jan 2011 12:57:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22103 invoked by uid 99); 6 Jan 2011 12:57:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 12:57:41 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of magawake@gmail.com designates 209.85.216.169 as permitted sender)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 12:57:35 +0000
Received: by qyk7 with SMTP id 7so17812140qyk.14
        for <multiple recipients>; Thu, 06 Jan 2011 04:57:15 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:cc:content-type
         :content-transfer-encoding;
        bh=6F79ak9XuL6nxMjo7ALwXWwTf2mDzGaIst9g6bqohj8=;
        b=bibtT0HmvFSIejOCMDpBOhdWbQGKGAEpceQoGdS42pAsGPKYJkuHHNhPYxdgHtjFAz
         +owAOSPt3Ed+VkfvjzYFqCclvvCbnv/uyVTyEDlX2wzA9QRSNnZiTkjkeVNYKVs9i0Nc
         naJSUuZ3DSZisw3cbtIt5rRWemFXXv9cUaEy8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type:content-transfer-encoding;
        b=avS8u5Dz0EV7BWpLGW5IGOgTCSR9paoXNGfRxDMtHyoqk/Fgqo+D/qpS49wAob2hXi
         QSqwk2I8lc2CfjNWFMn2N9xU/BQrAkSIeiRNQ/JCvb4lg3ZcgiviEWA7KI6jvsvlLBL6
         vLYvmz4BoAXO6/R3c6bB9MUfC02PrAnnW5L1U=
MIME-Version: 1.0
Received: by 10.224.67.208 with SMTP id s16mr20176885qai.322.1294318634712;
 Thu, 06 Jan 2011 04:57:14 -0800 (PST)
Received: by 10.220.118.20 with HTTP; Thu, 6 Jan 2011 04:57:14 -0800 (PST)
In-Reply-To: <AANLkTinSqm4Z1ZO9ETJPv2DMUoojJ0DGUg=e5HL=gXag@mail.gmail.com>
References: <AANLkTi=B1TR1yAmEqrf5PKi-=NRABB12T-7+6Ja_LJws@mail.gmail.com>
	<A1782BC8-D13E-42C1-8AE1-801C8ACE3753@yahoo-inc.com>
	<AANLkTimwS=ChaqcZtEt8WM0Vf4LUbm-D07BgwGGKYySq@mail.gmail.com>
	<AANLkTinSqm4Z1ZO9ETJPv2DMUoojJ0DGUg=e5HL=gXag@mail.gmail.com>
Date: Thu, 6 Jan 2011 07:57:14 -0500
Message-ID: <AANLkTikDSq1amxeBGrMpQ=V=yDd-Bnfwju2t-_8c9jSj@mail.gmail.com>
Subject: Re: Moving to a new namenode
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Cc: hdfs-user@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

sorry everyone. I have subscribed to hdfs-users list and will discuss
this there.



On Thu, Jan 6, 2011 at 7:43 AM, Harsh J <qwertymaniac@gmail.com> wrote:
> No, he meant to move the discussion to the hdfs-user@hadoop.apache.org
> list for HDFS queries.
>
> Sending to hdfs-user, bcc'ing to general. [This the right way?]
>
> On Thu, Jan 6, 2011 at 6:05 PM, Mag Gam <magawake@gmail.com> wrote:
>> The current filesystem is in HDFS.
>>
>> Do you mean, create a new namenode on the new server and assign some
>> data nodes to it and start moving the data off?
>>
>>
>> On Thu, Jan 6, 2011 at 12:14 AM, Eric Baldeschwieler
>> <eric14@yahoo-inc.com> wrote:
>>> could you move this traffic to HDFS?
>>>
>>> thanks!
>>>
>>> On Jan 5, 2011, at 6:29 PM, Mag Gam wrote:
>>>
>>>> What is the correct procedure to move to a new namenode? My current
>>>> namenode is running on a 2 core with 4gb of memory. I am moving to a
>>>> new host, =C2=A08 core x 32 GB of memory. My question are: what is the
>>>> proper way to move to the new namenode, and what can I do to enhance
>>>> the performance of it since the new hardware is much superior than the
>>>> old one.
>>>
>>>
>>
>
>
>
> --
> Harsh J
> www.harshj.com
>

From general-return-2687-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 06 15:01:08 2011
Return-Path: <general-return-2687-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38074 invoked from network); 6 Jan 2011 15:01:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2011 15:01:08 -0000
Received: (qmail 36876 invoked by uid 500); 6 Jan 2011 15:01:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36654 invoked by uid 500); 6 Jan 2011 15:01:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36636 invoked by uid 99); 6 Jan 2011 15:01:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 15:01:04 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 15:00:57 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p06F0MBd079144;
	Thu, 6 Jan 2011 07:00:22 -0800 (PST)
Received: from [10.0.1.5] (10.73.152.71) by SP2-EX07CAS01.ds.corp.yahoo.com
 (98.137.59.4) with Microsoft SMTP Server (TLS) id 8.3.106.1; Thu, 6 Jan 2011
 07:00:21 -0800
Subject: Re: Moving to a new namenode
MIME-Version: 1.0 (Apple Message framework v1082)
Content-Type: text/plain; charset="us-ascii"
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <AANLkTikDSq1amxeBGrMpQ=V=yDd-Bnfwju2t-_8c9jSj@mail.gmail.com>
Date: Thu, 6 Jan 2011 07:00:19 -0800
CC: "hdfs-user@hadoop.apache.org" <hdfs-user@hadoop.apache.org>
Content-Transfer-Encoding: 7bit
Message-ID: <30091EA3-7035-40B7-AA83-7934C02894E9@yahoo-inc.com>
References: <AANLkTi=B1TR1yAmEqrf5PKi-=NRABB12T-7+6Ja_LJws@mail.gmail.com> <A1782BC8-D13E-42C1-8AE1-801C8ACE3753@yahoo-inc.com> <AANLkTimwS=ChaqcZtEt8WM0Vf4LUbm-D07BgwGGKYySq@mail.gmail.com> <AANLkTinSqm4Z1ZO9ETJPv2DMUoojJ0DGUg=e5HL=gXag@mail.gmail.com> <AANLkTikDSq1amxeBGrMpQ=V=yDd-Bnfwju2t-_8c9jSj@mail.gmail.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks!  Don't worry, we should make this clear on the site.

I apologize, I didn't mean to spam the list with my suggestion!


On Jan 6, 2011, at 4:57 AM, Mag Gam wrote:

> sorry everyone. I have subscribed to hdfs-users list and will discuss
> this there.
> 
> 
> 
> On Thu, Jan 6, 2011 at 7:43 AM, Harsh J <qwertymaniac@gmail.com> wrote:
>> No, he meant to move the discussion to the hdfs-user@hadoop.apache.org
>> list for HDFS queries.
>> 
>> Sending to hdfs-user, bcc'ing to general. [This the right way?]
>> 
>> On Thu, Jan 6, 2011 at 6:05 PM, Mag Gam <magawake@gmail.com> wrote:
>>> The current filesystem is in HDFS.
>>> 
>>> Do you mean, create a new namenode on the new server and assign some
>>> data nodes to it and start moving the data off?
>>> 
>>> 
>>> On Thu, Jan 6, 2011 at 12:14 AM, Eric Baldeschwieler
>>> <eric14@yahoo-inc.com> wrote:
>>>> could you move this traffic to HDFS?
>>>> 
>>>> thanks!
>>>> 
>>>> On Jan 5, 2011, at 6:29 PM, Mag Gam wrote:
>>>> 
>>>>> What is the correct procedure to move to a new namenode? My current
>>>>> namenode is running on a 2 core with 4gb of memory. I am moving to a
>>>>> new host,  8 core x 32 GB of memory. My question are: what is the
>>>>> proper way to move to the new namenode, and what can I do to enhance
>>>>> the performance of it since the new hardware is much superior than the
>>>>> old one.
>>>> 
>>>> 
>>> 
>> 
>> 
>> 
>> --
>> Harsh J
>> www.harshj.com
>> 


From general-return-2688-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 06 15:04:45 2011
Return-Path: <general-return-2688-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38959 invoked from network); 6 Jan 2011 15:04:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2011 15:04:45 -0000
Received: (qmail 46287 invoked by uid 500); 6 Jan 2011 15:04:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46141 invoked by uid 500); 6 Jan 2011 15:04:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46129 invoked by uid 99); 6 Jan 2011 15:04:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 15:04:43 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 15:04:37 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p06F3xfC051343;
	Thu, 6 Jan 2011 07:03:59 -0800 (PST)
Received: from [10.0.1.5] (10.73.152.71) by SP2-EX07CAS01.ds.corp.yahoo.com
 (98.137.59.4) with Microsoft SMTP Server (TLS) id 8.3.106.1; Thu, 6 Jan 2011
 07:03:58 -0800
Subject: Re: Please join me in welcoming the following people as committers to the Hadoop project
MIME-Version: 1.0 (Apple Message framework v1082)
Content-Type: text/plain; charset="utf-8"
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <AANLkTimQ3=1QS627v-zC_1PGDNmRH0ecm3cUJzGgVzSw@mail.gmail.com>
Date: Thu, 6 Jan 2011 07:03:56 -0800
CC: "mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>,
        "hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org>
Content-Transfer-Encoding: quoted-printable
Message-ID: <A02562FA-465B-4C62-93A6-53571032ADAB@yahoo-inc.com>
References: <93F81E22-00DB-451B-B6F2-BDB0EB550998@Holsman.NET> <AANLkTinDatiqh26X34LoGvw0_rwyVfau4kY7xF-oerk=@mail.gmail.com> <AANLkTinG1b5fP7XsBco7DAH5npqA0kECPR830x+tipWN@mail.gmail.com> <AANLkTimQ3=1QS627v-zC_1PGDNmRH0ecm3cUJzGgVzSw@mail.gmail.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)

congrats all! =20

And thanks for your years and years of contribution to date!

E14

On Jan 6, 2011, at 2:04 AM, Lars George wrote:

> Congratulations to all of you!
>=20
> On Thu, Jan 6, 2011 at 8:22 AM, Deepak Sharma <deepakmca05@gmail.com> =
wrote:
>> congrats all..I hope i'll be committer some day as well:)
>>=20
>> On Thu, Jan 6, 2011 at 10:23 AM, li ping <li.j2ee@gmail.com> wrote:
>>=20
>>> congratulations
>>>=20
>>> On Thu, Jan 6, 2011 at 11:40 AM, Ian Holsman <hadoop@holsman.net> =
wrote:
>>>=20
>>>> On behalf of the Apache Hadoop PMC, I would like to extend a warm =
welcome
>>>> to the following people,
>>>> who have all chosen to accept the role of committers on Hadoop.
>>>>=20
>>>> In no alphabetical order:
>>>>=20
>>>> - Aaron Kimball
>>>> - Allen Wittenauer
>>>> - Amar Kamat
>>>> - Dmytro Molkov
>>>> - Jitendra Pandey
>>>> - Kan Zhang
>>>> - Ravi Gummadi
>>>> - Sreekanth Ramakrishna
>>>> - Todd Lipcon
>>>>=20
>>>> I appreciate all the hard work these people have put into the =
project so
>>>> far, and look forward to future contributions they will make to =
Hadoop in
>>>> the future
>>>>=20
>>>> Well done guys!
>>>>=20
>>>>=20
>>>> --Ian
>>>>=20
>>>>=20
>>>=20
>>>=20
>>> --
>>> -----=E6=9D=8E=E5=B9=B3
>>>=20
>>=20
>>=20
>>=20
>> --
>> Deepak Sharma
>> http://www.linkedin.com/in/rikindia
>>=20


From general-return-2689-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 06 19:06:34 2011
Return-Path: <general-return-2689-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77548 invoked from network); 6 Jan 2011 19:06:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2011 19:06:32 -0000
Received: (qmail 66237 invoked by uid 500); 6 Jan 2011 19:06:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66185 invoked by uid 500); 6 Jan 2011 19:06:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66166 invoked by uid 99); 6 Jan 2011 19:06:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 19:06:31 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [88.198.2.104] (HELO koch.ro) (88.198.2.104)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 19:06:24 +0000
Received: from 201-127.76-83.cust.bluewin.ch ([83.76.127.201] helo=jona.localnet)
	by koch.ro with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
	(Exim 4.69)
	(envelope-from <thomas@koch.ro>)
	id 1Pav9v-0008UD-HL; Thu, 06 Jan 2011 20:06:03 +0100
From: Thomas Koch <thomas@koch.ro>
Reply-To: thomas@koch.ro
To: "Owen O'Malley" <omalley@apache.org>
Subject: Re: Test instance of Gerrit?
Date: Thu, 6 Jan 2011 20:05:53 +0100
User-Agent: KMail/1.13.5 (Linux/2.6.32-4-amd64; KDE/4.4.5; x86_64; ; )
Cc: infrastructure-dev@apache.org,
 general@hadoop.apache.org
References: <201012220856.27550.thomas@koch.ro> <AANLkTimRtoHJQ26111PayuAtQD6mSbXuSuar=C13d++G@mail.gmail.com>
In-Reply-To: <AANLkTimRtoHJQ26111PayuAtQD6mSbXuSuar=C13d++G@mail.gmail.com>
MIME-Version: 1.0
Content-Type: Text/Plain;
  charset="us-ascii"
Content-Transfer-Encoding: 7bit
Message-Id: <201101062005.54067.thomas@koch.ro>

Owen O'Malley:
> On Tue, Dec 21, 2010 at 11:56 PM, Thomas Koch <thomas@koch.ro> wrote:
> 
> Now my question is, whether there's some interest from developers to give
> 
> > Gerrit a try? In this case, I'd setup a private toy instance and import
> > your
> > project.
> 
> I'll give it a try. Could you import hadoop common, hdfs, and mapreduce?
> 
> -- Owen
Hi Owen,

The gerrit instance is running on http://koch.ro:8080
Projects imported are ZooKeeper and Hadoop-(common|hdfs|mapreduce).
Since Gerrit is a Patch review system, you don't "see" anything about the 
projects but only get a list of patches waiting for review.

You can upload a patch like

git push ssh://koch.ro:29418/PROJECT/NAME \
LOCAL_REF:refs/for/DESTINATION-BRANCH/OPTIONAL-TAG

I used:
git push ssh://koch.ro:29418/zookeeper/zookeeper \
ZOOKEEPER-823_on_tlp:refs/for/trunk/ZOOKEEPER-823

you may want to use sth. like:

git push ssh://koch.ro:29418/hadoop/mapreduce \
MY_BRANCH:refs/for/trunk

The project names are
hadoop/common, hadoop/hdfs and hadoop/mapreduce.

I'm available on Freenode as thkoch at CET working hours.

Best regards,

Thomas Koch, http://www.koch.ro

From general-return-2690-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 06 20:54:17 2011
Return-Path: <general-return-2690-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25239 invoked from network); 6 Jan 2011 20:54:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2011 20:54:16 -0000
Received: (qmail 26523 invoked by uid 500); 6 Jan 2011 20:54:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26481 invoked by uid 500); 6 Jan 2011 20:54:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26473 invoked by uid 99); 6 Jan 2011 20:54:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 20:54:15 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of michael_segel@hotmail.com designates 65.55.34.15 as permitted sender)
Received: from [65.55.34.15] (HELO col0-omc1-s5.col0.hotmail.com) (65.55.34.15)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 20:54:08 +0000
Received: from COL117-W27 ([65.55.34.9]) by col0-omc1-s5.col0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Thu, 6 Jan 2011 12:53:47 -0800
Message-ID: <COL117-W2791857F7AB881303F159D8F0A0@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_1ba0912b-7d23-4731-bf62-cfd56bee4e99_"
X-Originating-IP: [65.167.11.254]
From: Michael Segel <michael_segel@hotmail.com>
To: <general@hadoop.apache.org>
Subject: RE: decommission a rack
Date: Thu, 6 Jan 2011 14:53:47 -0600
Importance: Normal
In-Reply-To: <AANLkTimSF48e+hbN3sMhLMkkoNfsJdT37RvM-BFt_38P@mail.gmail.com>
References:
 <AANLkTikN5ZkqMN9NrDgq_ohB6L_Qo_4TTjAiOzuS=bPt@mail.gmail.com>,<ABC24175AFD3BE4DA15F4CD375ED413D063B0BD673@hq-ex-mb02.ad.navteq.com>,<AANLkTimSF48e+hbN3sMhLMkkoNfsJdT37RvM-BFt_38P@mail.gmail.com>
MIME-Version: 1.0
X-OriginalArrivalTime: 06 Jan 2011 20:53:47.0931 (UTC) FILETIME=[D09CCEB0:01CBADE3]

--_1ba0912b-7d23-4731-bf62-cfd56bee4e99_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable



Its ok if its in the dead column.=20
Take it out of your slaves file and when you reboot=2C it will not show up =
at all.=20
Until then its just a foot note in history. :-)

> Date: Wed=2C 5 Jan 2011 21:24:57 -0500
> Subject: Re: decommission a rack
> From: magawake@gmail.com
> To: general@hadoop.apache.org
>=20
> Thanks for the response.
>=20
> I did decommission the server and it took couple of minutes. However=2C
> I see the server in my "Dead Datanotes" page. How can I completely
> forget about this box?
>=20
>=20
> On Wed=2C Jan 5=2C 2011 at 10:01 AM=2C Segel=2C Mike <msegel@navteq.com> =
wrote:
> > Change your topology script?
> >
> > One thing about rack awareness. You never want to have an unbalanced co=
nfiguration. The caveat is if your uplinks between the ToR Switch  are 'fat=
' enough. (4 10GBe should do it.)
> >
> > If you watched your system with the one machine on a separate rack (ass=
uming it was also on a separate switch) you'd see a major network bottle ne=
ck. (We use Ganglia)
> >
> > If you're seeing the rack count wrong=2C you could just bounce the clou=
d and it should show only two.
> >
> > Note: Deprecating a machine can take a long time. You may want to consi=
der just killing the node=2C waiting till the cluster recognizes that its d=
own and I believe just kicking off an FSCK or maybe running the balancer pr=
ogram will clean that up faster than deprecating. (YMMV)
> >
> > HTH
> >
> > -Mike
> >
> >
> > -----Original Message-----
> > From: Mag Gam [mailto:magawake@gmail.com]
> > Sent: Wednesday=2C January 05=2C 2011 5:56 AM
> > To: general@hadoop.apache.org
> > Subject: decommission a rack
> >
> > Using 0.21
> >
> > I have a server which was in rack3. It was the only server in this
> > rack so I decommissioned the server. fsck shows only 2 racks however
> > topology shows 3 racks. How can I fix this?
> >
> >
> > The information contained in this communication may be CONFIDENTIAL and=
 is intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient=2C you are hereby notified that any disseminatio=
n=2C distribution=2C or copying of this communication=2C or any of its cont=
ents=2C is strictly prohibited.  If you have received this communication in=
 error=2C please notify the sender and delete/destroy the original message =
and any copy of it from your computer or paper files.
> >
 		 	   		  =

--_1ba0912b-7d23-4731-bf62-cfd56bee4e99_--

From general-return-2691-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 06 21:00:34 2011
Return-Path: <general-return-2691-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 28668 invoked from network); 6 Jan 2011 21:00:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2011 21:00:34 -0000
Received: (qmail 37018 invoked by uid 500); 6 Jan 2011 21:00:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36936 invoked by uid 500); 6 Jan 2011 21:00:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36928 invoked by uid 99); 6 Jan 2011 21:00:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 21:00:32 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 21:00:26 +0000
Received: from SP1-EX07CAS02.ds.corp.yahoo.com (sp1-ex07cas02.ds.corp.yahoo.com [216.252.116.138])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p06KxOC0015079
	for <general@hadoop.apache.org>; Thu, 6 Jan 2011 12:59:24 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1294347565;
	bh=+9JWtIdVT+s7ppAKx7B+VsrlmqyvhcIqQYS/VPgod5M=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 MIME-Version;
	b=m+I3yRU7R8G63FHCe8peZYD5Ep32dvjhLn/x0injBnBPlecNxT5tyPX/hA2q9klkW
	 CSQCTW12wU8Avwf5iECEZ3jlyDltQoOYWmLx3TVd7jYsiX8ItRBkm9FAKmlqS4fobs
	 zo3kxnyFh6N+gt5xLZghh5/1tk/7i70n54jUb220=
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 SP1-EX07CAS02.ds.corp.yahoo.com ([216.252.116.167]) with mapi; Thu, 6 Jan
 2011 12:59:24 -0800
From: Suresh Srinivas <sureshms@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 6 Jan 2011 12:59:22 -0800
Subject: Re: decommission a rack
Thread-Topic: decommission a rack
Thread-Index: Acut4/KV0HiPsV1BS1uUPFIwrj0rWQAAKUmp
Message-ID: <C94B6D2A.2E46A%sureshms@yahoo-inc.com>
In-Reply-To: <COL117-W2791857F7AB881303F159D8F0A0@phx.gbl>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C94B6D2A2E46Asureshmsyahooinccom_"
MIME-Version: 1.0

--_000_C94B6D2A2E46Asureshmsyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Removing from slaves file only prevents launching datanode process on that =
node.

If you you use dfs.hosts configuration to set include file with list of dat=
anodes, remove it from that as well. This will ensure:

 *   datanode cannot register with the namenode.
 *   datanode will not be shown in the dead list.

Regards,
Suresh


On 1/6/11 12:53 PM, "Michael Segel" <michael_segel@hotmail.com> wrote:




Its ok if its in the dead column.
Take it out of your slaves file and when you reboot, it will not show up at=
 all.
Until then its just a foot note in history. :-)

> Date: Wed, 5 Jan 2011 21:24:57 -0500
> Subject: Re: decommission a rack
> From: magawake@gmail.com
> To: general@hadoop.apache.org
>
> Thanks for the response.
>
> I did decommission the server and it took couple of minutes. However,
> I see the server in my "Dead Datanotes" page. How can I completely
> forget about this box?
>
>
> On Wed, Jan 5, 2011 at 10:01 AM, Segel, Mike <msegel@navteq.com> wrote:
> > Change your topology script?
> >
> > One thing about rack awareness. You never want to have an unbalanced co=
nfiguration. The caveat is if your uplinks between the ToR Switch  are 'fat=
' enough. (4 10GBe should do it.)
> >
> > If you watched your system with the one machine on a separate rack (ass=
uming it was also on a separate switch) you'd see a major network bottle ne=
ck. (We use Ganglia)
> >
> > If you're seeing the rack count wrong, you could just bounce the cloud =
and it should show only two.
> >
> > Note: Deprecating a machine can take a long time. You may want to consi=
der just killing the node, waiting till the cluster recognizes that its dow=
n and I believe just kicking off an FSCK or maybe running the balancer prog=
ram will clean that up faster than deprecating. (YMMV)
> >
> > HTH
> >
> > -Mike
> >
> >
> > -----Original Message-----
> > From: Mag Gam [mailto:magawake@gmail.com]
> > Sent: Wednesday, January 05, 2011 5:56 AM
> > To: general@hadoop.apache.org
> > Subject: decommission a rack
> >
> > Using 0.21
> >
> > I have a server which was in rack3. It was the only server in this
> > rack so I decommissioned the server. fsck shows only 2 racks however
> > topology shows 3 racks. How can I fix this?
> >
> >
> > The information contained in this communication may be CONFIDENTIAL and=
 is intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any dissemination,=
 distribution, or copying of this communication, or any of its contents, is=
 strictly prohibited.  If you have received this communication in error, pl=
ease notify the sender and delete/destroy the original message and any copy=
 of it from your computer or paper files.
> >



--_000_C94B6D2A2E46Asureshmsyahooinccom_--

From general-return-2692-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 06 21:03:37 2011
Return-Path: <general-return-2692-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30196 invoked from network); 6 Jan 2011 21:03:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2011 21:03:37 -0000
Received: (qmail 43281 invoked by uid 500); 6 Jan 2011 21:03:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 43208 invoked by uid 500); 6 Jan 2011 21:03:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 43200 invoked by uid 99); 6 Jan 2011 21:03:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 21:03:35 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of msegel@navteq.com designates 204.120.70.37 as permitted sender)
Received: from [204.120.70.37] (HELO xmailfargo.navteq.com) (204.120.70.37)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 21:03:30 +0000
Received: from fargo-spamfw.navteq.com (navteq-spamfw.navteq.com [10.6.10.131])
	by xmailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id p06KYcGt028982
	for <general@hadoop.apache.org>; Thu, 6 Jan 2011 14:34:38 -0600
Received: from imailfargo.navteq.com (localhost [127.0.0.1])
	by fargo-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id 9078E4808BB8
	for <general@hadoop.apache.org>; Thu,  6 Jan 2011 15:03:09 -0600 (CST)
Received: from imailfargo.navteq.com (imailfargo.navteq.com [10.6.10.130]) by fargo-spamfw.navteq.com with ESMTP id kwOUI00zScj9sblG for <general@hadoop.apache.org>; Thu, 06 Jan 2011 15:03:09 -0600 (CST)
Received: from hq-ex-ht01.ad.navteq.com ([10.8.222.51])
	by imailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id p06L39vR000738
	for <general@hadoop.apache.org>; Thu, 6 Jan 2011 15:03:09 -0600
Received: from hq-ex-mb02.ad.navteq.com ([fe80::880d:318e:e860:a8db]) by
 hq-ex-ht01.ad.navteq.com ([fe80::55a6:5b94:5c00:fe39%12]) with mapi; Thu, 6
 Jan 2011 15:03:09 -0600
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 6 Jan 2011 15:03:07 -0600
Subject: RE: decommission a rack
Thread-Topic: decommission a rack
Thread-Index: Acut4/KV0HiPsV1BS1uUPFIwrj0rWQAAKUmpAAAPvFA=
Message-ID: <ABC24175AFD3BE4DA15F4CD375ED413D063B0BDEB8@hq-ex-mb02.ad.navteq.com>
References: <COL117-W2791857F7AB881303F159D8F0A0@phx.gbl>
 <C94B6D2A.2E46A%sureshms@yahoo-inc.com>
In-Reply-To: <C94B6D2A.2E46A%sureshms@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1

Right.=20
But the point was that after decommissioning a node, it shows up as a dea=
d node. By taking it out of slaves or whatever you set dfs.hosts to point=
 to, the node is effectively out of the cluster. So when you restart the =
cluster, you won't touch the machine and the namenode won't see it or car=
e about it.

Of course you'll need to rebalance the cluster and run fsck to make sure =
you have the correct amount of replications on the rest of the cluster.
=20

-----Original Message-----
From: Suresh Srinivas [mailto:sureshms@yahoo-inc.com]=20
Sent: Thursday, January 06, 2011 2:59 PM
To: general@hadoop.apache.org
Subject: Re: decommission a rack

Removing from slaves file only prevents launching datanode process on tha=
t node.

If you you use dfs.hosts configuration to set include file with list of d=
atanodes, remove it from that as well. This will ensure:

 *   datanode cannot register with the namenode.
 *   datanode will not be shown in the dead list.

Regards,
Suresh


On 1/6/11 12:53 PM, "Michael Segel" <michael_segel@hotmail.com> wrote:




Its ok if its in the dead column.
Take it out of your slaves file and when you reboot, it will not show up =
at all.
Until then its just a foot note in history. :-)

> Date: Wed, 5 Jan 2011 21:24:57 -0500
> Subject: Re: decommission a rack
> From: magawake@gmail.com
> To: general@hadoop.apache.org
>
> Thanks for the response.
>
> I did decommission the server and it took couple of minutes. However,
> I see the server in my "Dead Datanotes" page. How can I completely
> forget about this box?
>
>
> On Wed, Jan 5, 2011 at 10:01 AM, Segel, Mike <msegel@navteq.com> wrote:
> > Change your topology script?
> >
> > One thing about rack awareness. You never want to have an unbalanced =
configuration. The caveat is if your uplinks between the ToR Switch  are =
'fat' enough. (4 10GBe should do it.)
> >
> > If you watched your system with the one machine on a separate rack (a=
ssuming it was also on a separate switch) you'd see a major network bottl=
e neck. (We use Ganglia)
> >
> > If you're seeing the rack count wrong, you could just bounce the clou=
d and it should show only two.
> >
> > Note: Deprecating a machine can take a long time. You may want to con=
sider just killing the node, waiting till the cluster recognizes that its=
 down and I believe just kicking off an FSCK or maybe running the balance=
r program will clean that up faster than deprecating. (YMMV)
> >
> > HTH
> >
> > -Mike
> >
> >
> > -----Original Message-----
> > From: Mag Gam [mailto:magawake@gmail.com]
> > Sent: Wednesday, January 05, 2011 5:56 AM
> > To: general@hadoop.apache.org
> > Subject: decommission a rack
> >
> > Using 0.21
> >
> > I have a server which was in rack3. It was the only server in this
> > rack so I decommissioned the server. fsck shows only 2 racks however
> > topology shows 3 racks. How can I fix this?
> >
> >
> > The information contained in this communication may be CONFIDENTIAL a=
nd is intended only for the use of the recipient(s) named above.  If you =
are not the intended recipient, you are hereby notified that any dissemin=
ation, distribution, or copying of this communication, or any of its cont=
ents, is strictly prohibited.  If you have received this communication in=
 error, please notify the sender and delete/destroy the original message =
and any copy of it from your computer or paper files.
> >




The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.

From general-return-2693-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 06 21:04:09 2011
Return-Path: <general-return-2693-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30471 invoked from network); 6 Jan 2011 21:04:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jan 2011 21:04:09 -0000
Received: (qmail 44847 invoked by uid 500); 6 Jan 2011 21:04:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44782 invoked by uid 500); 6 Jan 2011 21:04:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44774 invoked by uid 99); 6 Jan 2011 21:04:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 21:04:07 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jan 2011 21:04:01 +0000
Received: from SP1-EX07CAS02.ds.corp.yahoo.com (sp1-ex07cas02.ds.corp.yahoo.com [216.252.116.138])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p06L3B90060971
	for <general@hadoop.apache.org>; Thu, 6 Jan 2011 13:03:11 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1294347792;
	bh=AkFUgN7Gp4o94NdF6iURgkv6jF8gt0fhZa3g17EyziE=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 MIME-Version;
	b=T0gp2e+Ey3JTOLzR9QbtFxGOeBum+IwFYXrc8oAhmw/kv3t1OotMlThk63vU7LCFg
	 Qd58m7TooJcNpfK2GkbQgwUqJupvwDQcz2epCLMMqStnz1E6dTUxJY+hV86we8GdVj
	 xhEygVl6eLa8yGzj75b88VjstqpYoSXvC7j8vT9E=
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 SP1-EX07CAS02.ds.corp.yahoo.com ([216.252.116.167]) with mapi; Thu, 6 Jan
 2011 13:03:11 -0800
From: Suresh Srinivas <sureshms@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 6 Jan 2011 13:03:10 -0800
Subject: Re: decommission a rack
Thread-Topic: decommission a rack
Thread-Index: Acut4/KV0HiPsV1BS1uUPFIwrj0rWQAAKUmpAAAh+v0=
Message-ID: <C94B6E0E.2E46C%sureshms@yahoo-inc.com>
In-Reply-To: <C94B6D2A.2E46A%sureshms@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C94B6E0E2E46Csureshmsyahooinccom_"
MIME-Version: 1.0

--_000_C94B6E0E2E46Csureshmsyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

If you still have the datanode in exlude list, remove it from there as well=
...


On 1/6/11 12:59 PM, "Suresh Srinivas" <sureshms@yahoo-inc.com> wrote:

Removing from slaves file only prevents launching datanode process on that =
node.

If you you use dfs.hosts configuration to set include file with list of dat=
anodes, remove it from that as well. This will ensure:

 *   datanode cannot register with the namenode.
 *   datanode will not be shown in the dead list.

Regards,
Suresh


On 1/6/11 12:53 PM, "Michael Segel" <michael_segel@hotmail.com> wrote:




Its ok if its in the dead column.
Take it out of your slaves file and when you reboot, it will not show up at=
 all.
Until then its just a foot note in history. :-)

> Date: Wed, 5 Jan 2011 21:24:57 -0500
> Subject: Re: decommission a rack
> From: magawake@gmail.com
> To: general@hadoop.apache.org
>
> Thanks for the response.
>
> I did decommission the server and it took couple of minutes. However,
> I see the server in my "Dead Datanotes" page. How can I completely
> forget about this box?
>
>
> On Wed, Jan 5, 2011 at 10:01 AM, Segel, Mike <msegel@navteq.com> wrote:
> > Change your topology script?
> >
> > One thing about rack awareness. You never want to have an unbalanced co=
nfiguration. The caveat is if your uplinks between the ToR Switch  are 'fat=
' enough. (4 10GBe should do it.)
> >
> > If you watched your system with the one machine on a separate rack (ass=
uming it was also on a separate switch) you'd see a major network bottle ne=
ck. (We use Ganglia)
> >
> > If you're seeing the rack count wrong, you could just bounce the cloud =
and it should show only two.
> >
> > Note: Deprecating a machine can take a long time. You may want to consi=
der just killing the node, waiting till the cluster recognizes that its dow=
n and I believe just kicking off an FSCK or maybe running the balancer prog=
ram will clean that up faster than deprecating. (YMMV)
> >
> > HTH
> >
> > -Mike
> >
> >
> > -----Original Message-----
> > From: Mag Gam [mailto:magawake@gmail.com]
> > Sent: Wednesday, January 05, 2011 5:56 AM
> > To: general@hadoop.apache.org
> > Subject: decommission a rack
> >
> > Using 0.21
> >
> > I have a server which was in rack3. It was the only server in this
> > rack so I decommissioned the server. fsck shows only 2 racks however
> > topology shows 3 racks. How can I fix this?
> >
> >
> > The information contained in this communication may be CONFIDENTIAL and=
 is intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any dissemination,=
 distribution, or copying of this communication, or any of its contents, is=
 strictly prohibited.  If you have received this communication in error, pl=
ease notify the sender and delete/destroy the original message and any copy=
 of it from your computer or paper files.
> >




--_000_C94B6E0E2E46Csureshmsyahooinccom_--

From general-return-2694-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 07 05:13:38 2011
Return-Path: <general-return-2694-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50239 invoked from network); 7 Jan 2011 05:13:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jan 2011 05:13:38 -0000
Received: (qmail 66583 invoked by uid 500); 7 Jan 2011 05:13:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66410 invoked by uid 500); 7 Jan 2011 05:13:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66402 invoked by uid 99); 7 Jan 2011 05:13:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Jan 2011 05:13:35 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of gautamkowshik@gmail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Jan 2011 05:13:26 +0000
Received: by iwn2 with SMTP id 2so19415478iwn.35
        for <general@hadoop.apache.org>; Thu, 06 Jan 2011 21:13:04 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:references:message-id:from:to
         :in-reply-to:content-type:content-transfer-encoding:x-mailer
         :mime-version:subject:date:cc;
        bh=HkDpjVTuZfcVIuAo8LOmZ9Ls0Lwo+0M9S/2wVBNYh80=;
        b=wC3cGpq+zW1hLH+U5PD7YnVIO8hdF6WWg94jepl9+mGJ0Dk3l5/6ZzKuRS4bvyvBRY
         xgSt3QYO7fNrjHk08gvcM7iQuhY+OPz5M5MQrZK/vXpTZmlxCa8d9Fp0Tc5uClE07Mqe
         QOl6/b2EJFwsI8sAwKoL2d3O9IhAM3zDqVso0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=references:message-id:from:to:in-reply-to:content-type
         :content-transfer-encoding:x-mailer:mime-version:subject:date:cc;
        b=W2LKuaqJ8ZaU6dBrq7ytowec2SOD63wsWe4BM9GuH91z3NQmxhYP7kZSAAz9/uDl1U
         tUxV9egtHlQrFl915C5sEjfC8/+xolptz+EO9uo0kOfTpmBzkk1GcDmMniGEJq2Vtwa3
         NJOFrjpFwegkaiQsQOUTeoVPJcVwIoFpPX4e8=
Received: by 10.42.178.200 with SMTP id bn8mr409092icb.529.1294377184876;
        Thu, 06 Jan 2011 21:13:04 -0800 (PST)
Received: from [10.128.206.164] ([166.205.136.170])
        by mx.google.com with ESMTPS id he5sm315938icb.22.2011.01.06.21.13.00
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Thu, 06 Jan 2011 21:13:02 -0800 (PST)
References: <AANLkTi=EPH2VqAAN5stfnyy3tFTnuV18XOCLRHrHE8TT@mail.gmail.com> <AANLkTinxeQgPap64WejAjPLAJLC0geW4QG7N-jvDBmfn@mail.gmail.com>
Message-Id: <6EDB68F4-7EE2-43AA-B1AB-9AE676155F77@gmail.com>
From: Gautama Kowshik <gautamkowshik@gmail.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <AANLkTinxeQgPap64WejAjPLAJLC0geW4QG7N-jvDBmfn@mail.gmail.com>
Content-Type: text/plain;
	charset=us-ascii;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit
X-Mailer: iPhone Mail (7E18)
Mime-Version: 1.0 (iPhone Mail 7E18)
Subject: Re: Reducer getting stuck after 66% processing
Date: Thu, 6 Jan 2011 21:12:50 -0800
Cc: "general@hadoop.apache.org" <general@hadoop.apache.org>,
 "vinayakh@gmail.com" <vinayakh@gmail.com>
X-Virus-Checked: Checked by ClamAV on apache.org

Reducers are supposed to stream out data. Do not wait for the last  
value to arrive in the reduce to put it out in the collector. Call  
output.collect on each value.

Sent from my iPhone

On Jan 6, 2011, at 1:26 AM, Dhaval Makawana<dhaval.makawana@gmail.com>  
wrote:

> Hi,
>
>
>
> I have 1 GB of input data with 10954103 records going to 7 reducer  
> tasks.
> The cluster is set up is such that each reducer gets 1 GB of RAM and  
> 256 MB
> out of it is reserved for sorting.  The reducer simply sums up one  
> field of
> values and outputs all the values along with the sum value. The  
> problem is
> all reducers are getting stuck after around 66% of processing. Tail  
> of task
> tracker log reads as below.
>
>
>
> 2011-01-06 08:39:31,309 INFO org.apache.hadoop.mapred.ReduceTask:
> Interleaved on-disk merge complete: 0 files left.
>
> 2011-01-06 08:39:31,309 INFO org.apache.hadoop.mapred.ReduceTask: In- 
> memory
> merge complete: 2 files left.
>
> 2011-01-06 08:39:31,333 INFO org.apache.hadoop.mapred.Merger:  
> Merging 2
> sorted segments
>
> 2011-01-06 08:39:31,334 INFO org.apache.hadoop.mapred.Merger: Down  
> to the
> last merge-pass, with 1 segments left of total size: 25352535 bytes
>
> 2011-01-06 08:39:31,343 INFO  
> org.apache.hadoop.io.compress.CodecPool: Got
> brand-new compressor
>
> 2011-01-06 08:39:31,704 INFO org.apache.hadoop.mapred.ReduceTask:  
> Merged 2
> segments, 25352537 bytes to disk to satisfy reduce memory limit
>
> 2011-01-06 08:39:31,704 INFO org.apache.hadoop.mapred.ReduceTask:  
> Merging 1
> files, 2947959 bytes from disk
>
> 2011-01-06 08:39:31,705 INFO org.apache.hadoop.mapred.ReduceTask:  
> Merging 0
> segments, 0 bytes from memory into reduce
>
> 2011-01-06 08:39:31,705 INFO org.apache.hadoop.mapred.Merger:  
> Merging 1
> sorted segments
>
> 2011-01-06 08:39:31,709 INFO org.apache.hadoop.mapred.Merger: Down  
> to the
> last merge-pass, with 1 segments left of total size: 2947955 bytes
>
>
> Please help solving the problem.
>
>
> Regards,
>
> Dhaval

From general-return-2695-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 07 22:11:54 2011
Return-Path: <general-return-2695-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39215 invoked from network); 7 Jan 2011 22:11:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jan 2011 22:11:53 -0000
Received: (qmail 57053 invoked by uid 500); 7 Jan 2011 22:11:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56964 invoked by uid 500); 7 Jan 2011 22:11:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56956 invoked by uid 99); 7 Jan 2011 22:11:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Jan 2011 22:11:51 +0000
X-ASF-Spam-Status: No, hits=4.7 required=10.0
	tests=FREEMAIL_FROM,FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.103 as permitted sender)
Received: from [17.148.16.103] (HELO asmtpout028.mac.com) (17.148.16.103)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Jan 2011 22:11:45 +0000
MIME-version: 1.0
Content-type: multipart/alternative;
 boundary="Boundary_(ID_2EODpMQZAMxQO8NYrj3T/Q)"
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp028.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LEO00J66AXYYZ80@asmtp028.mac.com>
 for general@hadoop.apache.org; Fri, 07 Jan 2011 14:11:10 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-07_11:2011-01-07,2011-01-07,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101070089
Sun-Java-System-SMTP-Warning: Lines longer than SMTP allows found and wrapped.
From: Nigel Daley <ndaley@mac.com>
Subject: Re: Patch testing
Date: Fri, 07 Jan 2011 14:11:08 -0800
In-reply-to: <AANLkTim_o2HDjsxK36pWQn76D9FsJN97SACCHNMEiS7X@mail.gmail.com>
To: general@hadoop.apache.org
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com>
 <20101020195420.GG2075@tp>
 <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com>
 <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
 <7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com>
 <AANLkTimLXA+CvUs-09LMN3ob6GZqmhxiyrKO7pzWEfse@mail.gmail.com>
 <4CE46119.2030509@yahoo-inc.com>
 <8617BECB-78B4-42B6-B592-D7FC1F8DA923@mac.com>
 <4CE47C88.5050203@yahoo-inc.com>
 <AANLkTinTM5h-3NupRSsFh-DnyvFYw+_OsXpASxcQUSDn@mail.gmail.com>
 <AANLkTimSOxaXZi-VLjwuTB5Ch5tRt0bKozc5L+Nx=ZrQ@mail.gmail.com>
 <AANLkTineU2v=Sx-=oC=6ROt6Os8TEbnS3L8xz1ZkoxCc@mail.gmail.com>
 <AANLkTin+oucXdCdOo7LQTtqndTf1axiT0gQntiOZ_mQJ@mail.gmail.com>
 <AANLkTinGn4Pb6_LkrY4-kK56PdHX4C1Zfu7A2PVfUoub@mail.gmail.com>
 <41FB0800-3703-49C1-8069-DEB74FFE6FAC@mac.com>
 <AANLkTik_rNLMHj3JT6Ekdop-NU_qzcB=VdsX+k4uFBtf@mail.gmail.com>
 <AANLkTin9=FPxv+0EydK4mWp_1zJbgGTddevvF9MrHW80@mail.gmail.com>
 <AANLkTiku_OvZuuOMWuXMcVf4UxB+W-SfW1WWV6rU0cXB@mail.gmail.com>
 <7B140F50-27E0-47C7-8EF8-B897D26CEE49@mac.com>
 <AANLkTi=9wEVmjZqXN+Cw2+cW3iNfuMsUJG=OeXnMfg0f@mail.gmail.com>
 <394BBDCC-9561-4E07-8EBA-AE3A92814E5A@mac.com>
 <AANLkTinKeADQRv1hXMDxqsaws+u2p=b7ntHrJGPOFSRo@mail.gmail.com>
 <F5C6F56C-74DF-43B6-9846-EE60C16E04C7@mac.com>
 <AANLkTim_o2HDjsxK36pWQn76D9FsJN97SACCHNMEiS7X@mail.gmail.com>
Message-id: <FCE82CCA-43DC-424C-95E5-B0BD18C96EED@mac.com>
X-Mailer: Apple Mail (2.1082)

--Boundary_(ID_2EODpMQZAMxQO8NYrj3T/Q)
Content-type: text/plain; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT

Hrm, the MR precommit test I'm running has hung (been running for 14 hours so far).  FWIW, 2 HDFS precommit tests are hung too.  I suspect it could be the NFS mounts on the machines.  I forced a thread dump which you can see in the console: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/10/console

Any other ideas why these might be hanging?

Thanks,
Nige

On Jan 5, 2011, at 5:42 PM, Todd Lipcon wrote:

> On Wed, Jan 5, 2011 at 4:39 PM, Nigel Daley <ndaley@mac.com> wrote:
> 
>> Thanks for looking into it Todd.  Let's first see if you think it can be
>> fixed quickly.  Let me know.
>> 
>> 
> No problem, it wasn't too bad after all. Patch up on HADOOP-7087 which fixes
> this test timeout for me.
> 
> -Todd
> 
> 
>> On Jan 5, 2011, at 4:33 PM, Todd Lipcon wrote:
>> 
>>> On Wed, Jan 5, 2011 at 4:19 PM, Nigel Daley <ndaley@mac.com> wrote:
>>> 
>>>> Todd, would love to get
>>>> https://issues.apache.org/jira/browse/MAPREDUCE-2121 fixed first since
>>>> this is failing every night on trunk.
>>>> 
>>> 
>>> What if we disable that test, move that issue to 0.22 blocker, and then
>>> enable the test-patch? I'll also look into that one today, but if it's
>>> something that will take a while to fix, I don't think we should hold off
>>> the useful testing for all the other patches.
>>> 
>>> -Todd
>>> 
>>> On Jan 5, 2011, at 2:45 PM, Todd Lipcon wrote:
>>>> 
>>>>> Hi Nigel,
>>>>> 
>>>>> MAPREDUCE-2172 has been fixed for a while. Are there any other
>> particular
>>>>> JIRAs you think need to be fixed before the MR test-patch queue gets
>>>>> enabled? I have a lot of outstanding patches and doing all the
>> test-patch
>>>>> turnaround manually on 3 different boxes is a real headache.
>>>>> 
>>>>> Thanks
>>>>> -Todd
>>>>> 
>>>>> On Tue, Dec 21, 2010 at 1:33 PM, Nigel Daley <ndaley@mac.com> wrote:
>>>>> 
>>>>>> Ok, HDFS is now enabled.  You'll see a stream of updates shortly on
>> the
>>>> ~30
>>>>>> Patch Available HDFS issues.
>>>>>> 
>>>>>> Nige
>>>>>> 
>>>>>> On Dec 20, 2010, at 12:42 PM, Jakob Homan wrote:
>>>>>> 
>>>>>>> I committed HDFS-1511 this morning.  We should be good to go.  I can
>>>>>>> haz snooty robot butler?
>>>>>>> 
>>>>>>> On Fri, Dec 17, 2010 at 8:31 PM, Konstantin Boudnik <cos@apache.org>
>>>>>> wrote:
>>>>>>>> Thanks Jacob. I am wasted already but I can do it on Sun, I think,
>>>>>>>> unless it is done earlier.
>>>>>>>> --
>>>>>>>> Take care,
>>>>>>>> Konstantin (Cos) Boudnik
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Fri, Dec 17, 2010 at 19:41, Jakob Homan <jghoman@gmail.com>
>> wrote:
>>>>>>>>> Ok.  I'll get a patch out for 1511 tomorrow, unless someone wants
>> to
>>>>>>>>> whip one up tonight.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley <ndaley@mac.com>
>> wrote:
>>>>>>>>>> I agree with Cos on fixing HDFS-1511 first. Once that is done I'll
>>>>>> enable hdfs patch testing.
>>>>>>>>>> 
>>>>>>>>>> Cheers,
>>>>>>>>>> Nige
>>>>>>>>>> 
>>>>>>>>>> Sent from my iPhone4
>>>>>>>>>> 
>>>>>>>>>> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <cos@apache.org>
>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> One more issue needs to be addressed before test-patch is turned
>> on
>>>>>> HDFS is
>>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-1511
>>>>>>>>>>> --
>>>>>>>>>>> Take care,
>>>>>>>>>>> Konstantin (Cos) Boudnik
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <
>> cos@apache.org>
>>>>>> wrote:
>>>>>>>>>>>> Considering that because of these 4 faulty cases every patch
>> will
>>>> be
>>>>>>>>>>>> -1'ed a patch author will still have to look at it and make a
>>>>>> comment
>>>>>>>>>>>> why this particular -1 isn't valid. Lesser work, perhaps, but
>>>>>> messier
>>>>>>>>>>>> IMO. I'm not blocking it - I just feel like there's a better
>> way.
>>>>>>>>>>>> 
>>>>>>>>>>>> --
>>>>>>>>>>>> Take care,
>>>>>>>>>>>> Konstantin (Cos) Boudnik
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jghoman@gmail.com>
>>>>>> wrote:
>>>>>>>>>>>>>> If HDFS is added to the test-patch queue right now we get
>>>>>>>>>>>>>> nothing but dozens of -1'ed patches.
>>>>>>>>>>>>> There aren't dozens of patches being submitted currently.  The
>> -1
>>>>>>>>>>>>> isn't the important thing, it's the grunt work of actually
>>>> running
>>>>>>>>>>>>> (and waiting) for the tests, test-patch, etc. that Hudson does
>> so
>>>>>> that
>>>>>>>>>>>>> the developer doesn't have to.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <
>>>>>> dhruba@gmail.com> wrote:
>>>>>>>>>>>>>> +1, thanks for doing this.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <
>> jghoman@gmail.com
>>>>> 
>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> So, with test-patch updated to show the failing tests, saving
>>>> the
>>>>>>>>>>>>>>> developers the need to go and verify that the failed tests
>> are
>>>>>> all
>>>>>>>>>>>>>>> known, how do people feel about turning on test-patch again
>> for
>>>>>> HDFS
>>>>>>>>>>>>>>> and mapred?  I think it'll help prevent any more tests from
>>>>>> entering
>>>>>>>>>>>>>>> the "yeah, we know" category.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> jg
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <
>>>>>> jhoman@yahoo-inc.com> wrote:
>>>>>>>>>>>>>>>> True, each patch would get a -1 and the failing tests would
>>>> need
>>>>>> to be
>>>>>>>>>>>>>>>> verified as those known bad (BTW, it would be great if
>> Hudson
>>>>>> could list
>>>>>>>>>>>>>>>> which tests failed in the message it posts to JIRA).  But
>>>> that's
>>>>>> still
>>>>>>>>>>>>>>> quite
>>>>>>>>>>>>>>>> a bit less error-prone work than if the developer runs the
>>>> tests
>>>>>> and
>>>>>>>>>>>>>>>> test-patch themselves.  Also, with 22 being cut, there are a
>>>> lot
>>>>>> of
>>>>>>>>>>>>>>> patches
>>>>>>>>>>>>>>>> up in the air and several developers are juggling multiple
>>>>>> patches.  The
>>>>>>>>>>>>>>>> more automation we can have, even if it's not perfect, will
>>>>>> decrease
>>>>>>>>>>>>>>> errors
>>>>>>>>>>>>>>>> we may make.
>>>>>>>>>>>>>>>> -jg
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Nigel Daley wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> It's also ready to run on MapReduce and HDFS but we won't
>>>>>> turn it on
>>>>>>>>>>>>>>>>>>> until these projects build and test cleanly.  Looks like
>>>> both
>>>>>> these
>>>>>>>>>>>>>>> projects
>>>>>>>>>>>>>>>>>>> currently have test failures.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Assuming the projects are compiling and building, is there
>> a
>>>>>> reason to
>>>>>>>>>>>>>>>>>> not turn it on despite the test failures? Hudson is
>>>> invaluable
>>>>>> to
>>>>>>>>>>>>>>> developers
>>>>>>>>>>>>>>>>>> who then don't have to run the tests and test-patch
>>>>>> themselves.  We
>>>>>>>>>>>>>>> didn't
>>>>>>>>>>>>>>>>>> turn Hudson off when it was working previously and there
>>>> were
>>>>>> known
>>>>>>>>>>>>>>>>>> failures.  I think one of the reasons we have more failing
>>>>>> tests now is
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> higher cost of doing Hudson's work (not a great excuse I
>>>>>> know).  This
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> particularly true now because several of the failing tests
>>>>>> involve
>>>>>>>>>>>>>>> tests
>>>>>>>>>>>>>>>>>> timing out, making the whole testing regime even longer.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Every single patch would get a -1 and need investigation.
>>>>>> Currently,
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> would be about 83 investigations between MR and HDFS issues
>>>>>> that are in
>>>>>>>>>>>>>>>>> patch available state.  Shouldn't we focus on getting these
>>>>>> tests fixed
>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>> removed/?  Also, I need to get MAPREDUCE-2172 fixed
>> (applies
>>>> to
>>>>>> HDFS as
>>>>>>>>>>>>>>>>> well) before I turn this on.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>> Nige
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Connect to me at http://www.facebook.com/dhruba
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Todd Lipcon
>>>>> Software Engineer, Cloudera
>>>> 
>>>> 
>>> 
>>> 
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>> 
>> 
> 
> 
> -- 
> Todd Lipcon
> Software Engineer, Cloudera


--Boundary_(ID_2EODpMQZAMxQO8NYrj3T/Q)--

From general-return-2696-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 07 23:34:12 2011
Return-Path: <general-return-2696-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83812 invoked from network); 7 Jan 2011 23:34:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jan 2011 23:34:12 -0000
Received: (qmail 94589 invoked by uid 500); 7 Jan 2011 23:34:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94497 invoked by uid 500); 7 Jan 2011 23:34:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94247 invoked by uid 99); 7 Jan 2011 23:34:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Jan 2011 23:34:10 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [98.139.53.212] (HELO nm19-vm0.bullet.mail.ac4.yahoo.com) (98.139.53.212)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 07 Jan 2011 23:34:02 +0000
Received: from [98.139.52.197] by nm19.bullet.mail.ac4.yahoo.com with NNFMP; 07 Jan 2011 23:33:41 -0000
Received: from [98.139.52.179] by tm10.bullet.mail.ac4.yahoo.com with NNFMP; 07 Jan 2011 23:33:41 -0000
Received: from [127.0.0.1] by omp1062.mail.ac4.yahoo.com with NNFMP; 07 Jan 2011 23:33:41 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 785420.3524.bm@omp1062.mail.ac4.yahoo.com
Received: (qmail 27461 invoked by uid 60001); 7 Jan 2011 23:33:41 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1294443221; bh=ouMbyJ5ZMCDrtJnE0h8W220Pfhmh8jM/xIjxqR4Sq64=; h=Message-ID:X-YMail-OSG:Received:X-RocketYMMF:X-Mailer:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=tbOm9ugzjouxNrnYVHfS1gH/tYktw+qhE1e9atbKdG8dOTYhrbwD4eprQpGRUz8wPPHeMvfjxViHD1HYXdHr7T5ZBZ0zkj95lao3LRZ1NyOAnddsjSPXp2mxtJz6NqtjRvj2hq25mzndIepIX+3ecUrPM3uvByZpgEQ/Obt5d9g=
Message-ID: <533710.16018.qm@web65507.mail.ac4.yahoo.com>
X-YMail-OSG: y078T_oVM1lxgxv0QorkU9rajF0.SNzxrmbPMRJXLBtiKCo
 hwB.CXH6SMi5XGCj.6Z8CdG2zstFxDh.AkioN.dY_8mGRDLvJIDnlDroeVJj
 TKv9qJlX2CC.qldFdB5RNDx1w24a.SWqBBQ0WTZzshRsF.gBen0pFFPpiAlM
 yTkDTpg8TBgV.GwnhKFQCU.sDaNQxinUh8QdlAidOYczBoJE_JuXbP6hFtSG
 2AdI96W0jJZPGKaeYO8H.59pJBuMkx7xT27qT07zDd9b1jcqUurTgTmZ4dgM
 qmR6SQ_HMVGpBRKgZ8E7iVaoUZhQIsiCO0BIXvOioQQ4iSiEXp7H1S3.oNXR
 b0vUwe1uPmYZG9A8P2eTIJWp_LfTyoVnY1.9Zp0f4XULut5uuDPaLXJDCnJz
 3X.t4XSWR7V4y
Received: from [69.106.200.155] by web65507.mail.ac4.yahoo.com via HTTP; Fri, 07 Jan 2011 15:33:41 PST
X-RocketYMMF: apurtell
X-Mailer: YahooMailClassic/11.4.20 YahooMailWebService/0.8.107.285259
Date: Fri, 7 Jan 2011 15:33:41 -0800 (PST)
From: Andrew Purtell <apurtell@apache.org>
Reply-To: apurtell@apache.org
Subject: Re: Please join me in welcoming the following people as committers to the Hadoop project
To: mapreduce-dev@hadoop.apache.org, general@hadoop.apache.org,
  hdfs-dev@hadoop.apache.org
In-Reply-To: <93F81E22-00DB-451B-B6F2-BDB0EB550998@Holsman.NET>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii

Congratulations to all!

Best regards,

    - Andy


--- On Wed, 1/5/11, Ian Holsman <hadoop@holsman.net> wrote:

> From: Ian Holsman <hadoop@holsman.net>
> Subject: Please join me in welcoming the following people as committers to the Hadoop project
> To: mapreduce-dev@hadoop.apache.org, general@hadoop.apache.org, hdfs-dev@hadoop.apache.org
> Date: Wednesday, January 5, 2011, 7:40 PM
> On behalf of the Apache Hadoop PMC, I
> would like to extend a warm welcome to the following
> people,
> who have all chosen to accept the role of committers on
> Hadoop.
> 
> In no alphabetical order:
>  
> - Aaron Kimball
> - Allen Wittenauer 
> - Amar Kamat
> - Dmytro Molkov
> - Jitendra Pandey
> - Kan Zhang
> - Ravi Gummadi
> - Sreekanth Ramakrishna
> - Todd Lipcon
> 
> I appreciate all the hard work these people have put into
> the project so far, and look forward to future contributions
> they will make to Hadoop in the future
> 
> Well done guys!
> 
> 
> --Ian
> 
> 


      

From general-return-2697-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jan 08 01:15:15 2011
Return-Path: <general-return-2697-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39739 invoked from network); 8 Jan 2011 01:15:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jan 2011 01:15:15 -0000
Received: (qmail 61995 invoked by uid 500); 8 Jan 2011 01:15:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61902 invoked by uid 500); 8 Jan 2011 01:15:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61894 invoked by uid 99); 8 Jan 2011 01:15:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 08 Jan 2011 01:15:13 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 08 Jan 2011 01:15:09 +0000
Received: by iyb26 with SMTP id 26so19227645iyb.35
        for <general@hadoop.apache.org>; Fri, 07 Jan 2011 17:14:48 -0800 (PST)
Received: by 10.231.31.139 with SMTP id y11mr9972646ibc.96.1294449288166; Fri,
 07 Jan 2011 17:14:48 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.115.8 with HTTP; Fri, 7 Jan 2011 17:14:28 -0800 (PST)
In-Reply-To: <FCE82CCA-43DC-424C-95E5-B0BD18C96EED@mac.com>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com> <20101020195420.GG2075@tp>
 <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com> <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
 <7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com> <AANLkTimLXA+CvUs-09LMN3ob6GZqmhxiyrKO7pzWEfse@mail.gmail.com>
 <4CE46119.2030509@yahoo-inc.com> <8617BECB-78B4-42B6-B592-D7FC1F8DA923@mac.com>
 <4CE47C88.5050203@yahoo-inc.com> <AANLkTinTM5h-3NupRSsFh-DnyvFYw+_OsXpASxcQUSDn@mail.gmail.com>
 <AANLkTimSOxaXZi-VLjwuTB5Ch5tRt0bKozc5L+Nx=ZrQ@mail.gmail.com>
 <AANLkTineU2v=Sx-=oC=6ROt6Os8TEbnS3L8xz1ZkoxCc@mail.gmail.com>
 <AANLkTin+oucXdCdOo7LQTtqndTf1axiT0gQntiOZ_mQJ@mail.gmail.com>
 <AANLkTinGn4Pb6_LkrY4-kK56PdHX4C1Zfu7A2PVfUoub@mail.gmail.com>
 <41FB0800-3703-49C1-8069-DEB74FFE6FAC@mac.com> <AANLkTik_rNLMHj3JT6Ekdop-NU_qzcB=VdsX+k4uFBtf@mail.gmail.com>
 <AANLkTin9=FPxv+0EydK4mWp_1zJbgGTddevvF9MrHW80@mail.gmail.com>
 <AANLkTiku_OvZuuOMWuXMcVf4UxB+W-SfW1WWV6rU0cXB@mail.gmail.com>
 <7B140F50-27E0-47C7-8EF8-B897D26CEE49@mac.com> <AANLkTi=9wEVmjZqXN+Cw2+cW3iNfuMsUJG=OeXnMfg0f@mail.gmail.com>
 <394BBDCC-9561-4E07-8EBA-AE3A92814E5A@mac.com> <AANLkTinKeADQRv1hXMDxqsaws+u2p=b7ntHrJGPOFSRo@mail.gmail.com>
 <F5C6F56C-74DF-43B6-9846-EE60C16E04C7@mac.com> <AANLkTim_o2HDjsxK36pWQn76D9FsJN97SACCHNMEiS7X@mail.gmail.com>
 <FCE82CCA-43DC-424C-95E5-B0BD18C96EED@mac.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Fri, 7 Jan 2011 17:14:28 -0800
Message-ID: <AANLkTikhwe_uOpnJ_v5UeObfJyD_bDy_g55EJVFWw4ro@mail.gmail.com>
Subject: Re: Patch testing
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00235433303a003c6704994b76fe

--00235433303a003c6704994b76fe
Content-Type: text/plain; charset=ISO-8859-1

On Fri, Jan 7, 2011 at 2:11 PM, Nigel Daley <ndaley@mac.com> wrote:

> Hrm, the MR precommit test I'm running has hung (been running for 14 hours
> so far).  FWIW, 2 HDFS precommit tests are hung too.  I suspect it could be
> the NFS mounts on the machines.  I forced a thread dump which you can see in
> the console:
> https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/10/console
>
>
Strange, haven't seen a hang like that before in handleConnectionFailure. It
should retry for 15 minutes max in that loop.


> Any other ideas why these might be hanging?
>
>
There is an HDFS bug right now that can cause hangs on some tests -
HDFS-1529 - would appreciate if someone can take a look. But I don't think
this is responsible for the MR hang above.

-Todd


> On Jan 5, 2011, at 5:42 PM, Todd Lipcon wrote:
>
> > On Wed, Jan 5, 2011 at 4:39 PM, Nigel Daley <ndaley@mac.com> wrote:
> >
> >> Thanks for looking into it Todd.  Let's first see if you think it can be
> >> fixed quickly.  Let me know.
> >>
> >>
> > No problem, it wasn't too bad after all. Patch up on HADOOP-7087 which
> fixes
> > this test timeout for me.
> >
> > -Todd
> >
> >
> >> On Jan 5, 2011, at 4:33 PM, Todd Lipcon wrote:
> >>
> >>> On Wed, Jan 5, 2011 at 4:19 PM, Nigel Daley <ndaley@mac.com> wrote:
> >>>
> >>>> Todd, would love to get
> >>>> https://issues.apache.org/jira/browse/MAPREDUCE-2121 fixed first
> since
> >>>> this is failing every night on trunk.
> >>>>
> >>>
> >>> What if we disable that test, move that issue to 0.22 blocker, and then
> >>> enable the test-patch? I'll also look into that one today, but if it's
> >>> something that will take a while to fix, I don't think we should hold
> off
> >>> the useful testing for all the other patches.
> >>>
> >>> -Todd
> >>>
> >>> On Jan 5, 2011, at 2:45 PM, Todd Lipcon wrote:
> >>>>
> >>>>> Hi Nigel,
> >>>>>
> >>>>> MAPREDUCE-2172 has been fixed for a while. Are there any other
> >> particular
> >>>>> JIRAs you think need to be fixed before the MR test-patch queue gets
> >>>>> enabled? I have a lot of outstanding patches and doing all the
> >> test-patch
> >>>>> turnaround manually on 3 different boxes is a real headache.
> >>>>>
> >>>>> Thanks
> >>>>> -Todd
> >>>>>
> >>>>> On Tue, Dec 21, 2010 at 1:33 PM, Nigel Daley <ndaley@mac.com> wrote:
> >>>>>
> >>>>>> Ok, HDFS is now enabled.  You'll see a stream of updates shortly on
> >> the
> >>>> ~30
> >>>>>> Patch Available HDFS issues.
> >>>>>>
> >>>>>> Nige
> >>>>>>
> >>>>>> On Dec 20, 2010, at 12:42 PM, Jakob Homan wrote:
> >>>>>>
> >>>>>>> I committed HDFS-1511 this morning.  We should be good to go.  I
> can
> >>>>>>> haz snooty robot butler?
> >>>>>>>
> >>>>>>> On Fri, Dec 17, 2010 at 8:31 PM, Konstantin Boudnik <
> cos@apache.org>
> >>>>>> wrote:
> >>>>>>>> Thanks Jacob. I am wasted already but I can do it on Sun, I think,
> >>>>>>>> unless it is done earlier.
> >>>>>>>> --
> >>>>>>>> Take care,
> >>>>>>>> Konstantin (Cos) Boudnik
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Fri, Dec 17, 2010 at 19:41, Jakob Homan <jghoman@gmail.com>
> >> wrote:
> >>>>>>>>> Ok.  I'll get a patch out for 1511 tomorrow, unless someone wants
> >> to
> >>>>>>>>> whip one up tonight.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley <ndaley@mac.com>
> >> wrote:
> >>>>>>>>>> I agree with Cos on fixing HDFS-1511 first. Once that is done
> I'll
> >>>>>> enable hdfs patch testing.
> >>>>>>>>>>
> >>>>>>>>>> Cheers,
> >>>>>>>>>> Nige
> >>>>>>>>>>
> >>>>>>>>>> Sent from my iPhone4
> >>>>>>>>>>
> >>>>>>>>>> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <cos@apache.org
> >
> >>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> One more issue needs to be addressed before test-patch is
> turned
> >> on
> >>>>>> HDFS is
> >>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-1511
> >>>>>>>>>>> --
> >>>>>>>>>>> Take care,
> >>>>>>>>>>> Konstantin (Cos) Boudnik
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <
> >> cos@apache.org>
> >>>>>> wrote:
> >>>>>>>>>>>> Considering that because of these 4 faulty cases every patch
> >> will
> >>>> be
> >>>>>>>>>>>> -1'ed a patch author will still have to look at it and make a
> >>>>>> comment
> >>>>>>>>>>>> why this particular -1 isn't valid. Lesser work, perhaps, but
> >>>>>> messier
> >>>>>>>>>>>> IMO. I'm not blocking it - I just feel like there's a better
> >> way.
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> Take care,
> >>>>>>>>>>>> Konstantin (Cos) Boudnik
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jghoman@gmail.com
> >
> >>>>>> wrote:
> >>>>>>>>>>>>>> If HDFS is added to the test-patch queue right now we get
> >>>>>>>>>>>>>> nothing but dozens of -1'ed patches.
> >>>>>>>>>>>>> There aren't dozens of patches being submitted currently.
>  The
> >> -1
> >>>>>>>>>>>>> isn't the important thing, it's the grunt work of actually
> >>>> running
> >>>>>>>>>>>>> (and waiting) for the tests, test-patch, etc. that Hudson
> does
> >> so
> >>>>>> that
> >>>>>>>>>>>>> the developer doesn't have to.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <
> >>>>>> dhruba@gmail.com> wrote:
> >>>>>>>>>>>>>> +1, thanks for doing this.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <
> >> jghoman@gmail.com
> >>>>>
> >>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> So, with test-patch updated to show the failing tests,
> saving
> >>>> the
> >>>>>>>>>>>>>>> developers the need to go and verify that the failed tests
> >> are
> >>>>>> all
> >>>>>>>>>>>>>>> known, how do people feel about turning on test-patch again
> >> for
> >>>>>> HDFS
> >>>>>>>>>>>>>>> and mapred?  I think it'll help prevent any more tests from
> >>>>>> entering
> >>>>>>>>>>>>>>> the "yeah, we know" category.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>> jg
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <
> >>>>>> jhoman@yahoo-inc.com> wrote:
> >>>>>>>>>>>>>>>> True, each patch would get a -1 and the failing tests
> would
> >>>> need
> >>>>>> to be
> >>>>>>>>>>>>>>>> verified as those known bad (BTW, it would be great if
> >> Hudson
> >>>>>> could list
> >>>>>>>>>>>>>>>> which tests failed in the message it posts to JIRA).  But
> >>>> that's
> >>>>>> still
> >>>>>>>>>>>>>>> quite
> >>>>>>>>>>>>>>>> a bit less error-prone work than if the developer runs the
> >>>> tests
> >>>>>> and
> >>>>>>>>>>>>>>>> test-patch themselves.  Also, with 22 being cut, there are
> a
> >>>> lot
> >>>>>> of
> >>>>>>>>>>>>>>> patches
> >>>>>>>>>>>>>>>> up in the air and several developers are juggling multiple
> >>>>>> patches.  The
> >>>>>>>>>>>>>>>> more automation we can have, even if it's not perfect,
> will
> >>>>>> decrease
> >>>>>>>>>>>>>>> errors
> >>>>>>>>>>>>>>>> we may make.
> >>>>>>>>>>>>>>>> -jg
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Nigel Daley wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> It's also ready to run on MapReduce and HDFS but we
> won't
> >>>>>> turn it on
> >>>>>>>>>>>>>>>>>>> until these projects build and test cleanly.  Looks
> like
> >>>> both
> >>>>>> these
> >>>>>>>>>>>>>>> projects
> >>>>>>>>>>>>>>>>>>> currently have test failures.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Assuming the projects are compiling and building, is
> there
> >> a
> >>>>>> reason to
> >>>>>>>>>>>>>>>>>> not turn it on despite the test failures? Hudson is
> >>>> invaluable
> >>>>>> to
> >>>>>>>>>>>>>>> developers
> >>>>>>>>>>>>>>>>>> who then don't have to run the tests and test-patch
> >>>>>> themselves.  We
> >>>>>>>>>>>>>>> didn't
> >>>>>>>>>>>>>>>>>> turn Hudson off when it was working previously and there
> >>>> were
> >>>>>> known
> >>>>>>>>>>>>>>>>>> failures.  I think one of the reasons we have more
> failing
> >>>>>> tests now is
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> higher cost of doing Hudson's work (not a great excuse I
> >>>>>> know).  This
> >>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>> particularly true now because several of the failing
> tests
> >>>>>> involve
> >>>>>>>>>>>>>>> tests
> >>>>>>>>>>>>>>>>>> timing out, making the whole testing regime even longer.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Every single patch would get a -1 and need investigation.
> >>>>>> Currently,
> >>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>> would be about 83 investigations between MR and HDFS
> issues
> >>>>>> that are in
> >>>>>>>>>>>>>>>>> patch available state.  Shouldn't we focus on getting
> these
> >>>>>> tests fixed
> >>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>> removed/?  Also, I need to get MAPREDUCE-2172 fixed
> >> (applies
> >>>> to
> >>>>>> HDFS as
> >>>>>>>>>>>>>>>>> well) before I turn this on.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>> Nige
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> --
> >>>>>>>>>>>>>> Connect to me at http://www.facebook.com/dhruba
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Todd Lipcon
> >>>>> Software Engineer, Cloudera
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> Todd Lipcon
> >>> Software Engineer, Cloudera
> >>
> >>
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

--00235433303a003c6704994b76fe--

From general-return-2698-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 11 02:56:13 2011
Return-Path: <general-return-2698-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44615 invoked from network); 11 Jan 2011 02:56:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jan 2011 02:56:12 -0000
Received: (qmail 62577 invoked by uid 500); 11 Jan 2011 02:56:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62492 invoked by uid 500); 11 Jan 2011 02:56:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62484 invoked by uid 99); 11 Jan 2011 02:56:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 02:56:08 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.100 as permitted sender)
Received: from [17.148.16.100] (HELO asmtpout025.mac.com) (17.148.16.100)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 02:55:59 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp025.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LEU009E28420F00@asmtp025.mac.com>
 for general@hadoop.apache.org; Mon, 10 Jan 2011 18:55:15 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-11_02:2011-01-11,2011-01-11,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101100156
From: Nigel Daley <ndaley@mac.com>
Subject: email lists: too many or not enough?
Date: Mon, 10 Jan 2011 18:55:14 -0800
Message-id: <A38987A0-7069-43DF-8422-7742AF8613EE@mac.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

This project has a lot of email lists [*] -- but I feel we're either missing 1 or 2 lists or we have too many.  Given all our lists, it's not clear which one we should use to talk about cross project dev issues and cross project community/policy issues.  general@ has perhaps become a catch all list with user, dev, and community issues inter-mixed.  Is that what we want?  Should we create more lists to break out these topics?  Live with the current inter-mingling? Or consolidate some lists?

Nige

[*] I may have missed some:
general@
security@
private@
common-user@
common-dev@
common-commits@
common-issues@
hdfs-user@
hdfs-dev@
hdfs-commits@
hdfs-issues@
mapreduce-user@
mapreduce-dev@
mapreduce-commits@
mapreduce-issues@

From general-return-2699-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 11 04:06:06 2011
Return-Path: <general-return-2699-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 80643 invoked from network); 11 Jan 2011 04:06:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jan 2011 04:06:05 -0000
Received: (qmail 12769 invoked by uid 500); 11 Jan 2011 04:06:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12422 invoked by uid 500); 11 Jan 2011 04:06:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12414 invoked by uid 99); 11 Jan 2011 04:06:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 04:06:02 +0000
X-ASF-Spam-Status: No, hits=-0.1 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of chenqibj@cn.ibm.com designates 202.81.31.142 as permitted sender)
Received: from [202.81.31.142] (HELO e23smtp09.au.ibm.com) (202.81.31.142)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 04:05:53 +0000
Received: from d23relay04.au.ibm.com (d23relay04.au.ibm.com [202.81.31.246])
	by e23smtp09.au.ibm.com (8.14.4/8.13.1) with ESMTP id p0B45Ve7023836
	for <general@hadoop.apache.org>; Tue, 11 Jan 2011 15:05:31 +1100
Received: from d23av04.au.ibm.com (d23av04.au.ibm.com [9.190.235.139])
	by d23relay04.au.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id p0B45V1E2359506
	for <general@hadoop.apache.org>; Tue, 11 Jan 2011 15:05:31 +1100
Received: from d23av04.au.ibm.com (loopback [127.0.0.1])
	by d23av04.au.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id p0B45VYk024585
	for <general@hadoop.apache.org>; Tue, 11 Jan 2011 15:05:31 +1100
Received: from d23m0037.cn.ibm.com (d23m0037.cn.ibm.com [9.181.2.105])
	by d23av04.au.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id p0B45Uaq024576
	for <general@hadoop.apache.org>; Tue, 11 Jan 2011 15:05:30 +1100
Subject: Keith is  out of office for a business trip
Auto-Submitted: auto-generated
From: Qi BJ Chen <chenqibj@cn.ibm.com>
To: general@hadoop.apache.org
Message-ID: <OF2D3D6200.E9400AF1-ON48257815.001679F9-48257815.001679FA@cn.ibm.com>
Date: Tue, 11 Jan 2011 12:05:30 +0800
X-MIMETrack: Serialize by Router on D23M0037/23/M/IBM(Release 8.5.1FP4|July 25, 2010) at
 01/11/2011 12:05:30
MIME-Version: 1.0
Content-type: multipart/alternative; 
	Boundary="0__=C7BBF286DF85FF698f9e8a93df938690918cC7BBF286DF85FF69"
Content-Disposition: inline

--0__=C7BBF286DF85FF698f9e8a93df938690918cC7BBF286DF85FF69
Content-type: text/plain; charset=US-ASCII


I will be out of the office starting  2011-01-11 and will not return until
2011-01-14.

--0__=C7BBF286DF85FF698f9e8a93df938690918cC7BBF286DF85FF69--


From general-return-2700-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 11 05:24:12 2011
Return-Path: <general-return-2700-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4997 invoked from network); 11 Jan 2011 05:24:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jan 2011 05:24:12 -0000
Received: (qmail 57093 invoked by uid 500); 11 Jan 2011 05:24:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56872 invoked by uid 500); 11 Jan 2011 05:24:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56859 invoked by uid 99); 11 Jan 2011 05:24:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 05:24:08 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 05:24:01 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0B5NVfc004151
	for <general@hadoop.apache.org>; Mon, 10 Jan 2011 21:23:31 -0800 (PST)
Received: from [10.0.1.5] (10.72.76.201) by SP2-EX07CAS01.ds.corp.yahoo.com
 (98.137.59.4) with Microsoft SMTP Server (TLS) id 8.3.106.1; Mon, 10 Jan 2011
 21:23:31 -0800
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0 (Apple Message framework v1082)
Subject: Re: email lists: too many or not enough?
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <A38987A0-7069-43DF-8422-7742AF8613EE@mac.com>
Date: Mon, 10 Jan 2011 21:23:29 -0800
Content-Transfer-Encoding: quoted-printable
Message-ID: <8C43652C-3B79-4583-B69E-C735B0084F10@yahoo-inc.com>
References: <A38987A0-7069-43DF-8422-7742AF8613EE@mac.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)

I'd love to have a list that covers most of what is on general (big =
topics on governance, etc), but does not have usage questions on it.  =
I'd suggest creating a new list for that and letting general become a =
general questions list.

On Jan 10, 2011, at 6:55 PM, Nigel Daley wrote:

> This project has a lot of email lists [*] -- but I feel we're either =
missing 1 or 2 lists or we have too many.  Given all our lists, it's not =
clear which one we should use to talk about cross project dev issues and =
cross project community/policy issues.  general@ has perhaps become a =
catch all list with user, dev, and community issues inter-mixed.  Is =
that what we want?  Should we create more lists to break out these =
topics?  Live with the current inter-mingling? Or consolidate some =
lists?
>=20
> Nige
>=20
> [*] I may have missed some:
> general@
> security@
> private@
> common-user@
> common-dev@
> common-commits@
> common-issues@
> hdfs-user@
> hdfs-dev@
> hdfs-commits@
> hdfs-issues@
> mapreduce-user@
> mapreduce-dev@
> mapreduce-commits@
> mapreduce-issues@


From general-return-2701-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 11 05:52:25 2011
Return-Path: <general-return-2701-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22091 invoked from network); 11 Jan 2011 05:52:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jan 2011 05:52:25 -0000
Received: (qmail 78251 invoked by uid 500); 11 Jan 2011 05:52:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78013 invoked by uid 500); 11 Jan 2011 05:52:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78005 invoked by uid 99); 11 Jan 2011 05:52:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 05:52:22 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 05:52:16 +0000
Received: by wye20 with SMTP id 20so21303264wye.35
        for <general@hadoop.apache.org>; Mon, 10 Jan 2011 21:51:55 -0800 (PST)
Received: by 10.216.7.205 with SMTP id 55mr2428395wep.96.1294725115090; Mon,
 10 Jan 2011 21:51:55 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.59.13 with HTTP; Mon, 10 Jan 2011 21:51:34 -0800 (PST)
In-Reply-To: <8C43652C-3B79-4583-B69E-C735B0084F10@yahoo-inc.com>
References: <A38987A0-7069-43DF-8422-7742AF8613EE@mac.com> <8C43652C-3B79-4583-B69E-C735B0084F10@yahoo-inc.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Mon, 10 Jan 2011 21:51:34 -0800
X-Google-Sender-Auth: -Tj7HpxakCVYudjBiIDi_6Fu5rs
Message-ID: <AANLkTine=OQYB9tuHwV6z1aXvQMN9KzSp0ztfzLAvHV5@mail.gmail.com>
Subject: Re: email lists: too many or not enough?
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Mon, Jan 10, 2011 at 21:23, Eric Baldeschwieler <eric14@yahoo-inc.com> w=
rote:
> I'd love to have a list that covers most of what is on general (big topic=
s on governance, etc), but does not have usage questions on it. =A0I'd sugg=
est creating a new list for that and letting general become a general quest=
ions list.

That what common-user@ is for.

> On Jan 10, 2011, at 6:55 PM, Nigel Daley wrote:
>
>> This project has a lot of email lists [*] -- but I feel we're either mis=
sing 1 or 2 lists or we have too many. =A0Given all our lists, it's not cle=
ar which one we should use to talk about cross project dev issues and cross=
 project community/policy issues. =A0general@ has perhaps become a catch al=
l list with user, dev, and community issues inter-mixed. =A0Is that what we=
 want? =A0Should we create more lists to break out these topics? =A0Live wi=
th the current inter-mingling? Or consolidate some lists?
>>
>> Nige
>>
>> [*] I may have missed some:
>> general@
>> security@
>> private@
>> common-user@
>> common-dev@
>> common-commits@
>> common-issues@
>> hdfs-user@
>> hdfs-dev@
>> hdfs-commits@
>> hdfs-issues@
>> mapreduce-user@
>> mapreduce-dev@
>> mapreduce-commits@
>> mapreduce-issues@
>
>

From general-return-2702-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 11 06:00:54 2011
Return-Path: <general-return-2702-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24018 invoked from network); 11 Jan 2011 06:00:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jan 2011 06:00:54 -0000
Received: (qmail 83846 invoked by uid 500); 11 Jan 2011 06:00:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83634 invoked by uid 500); 11 Jan 2011 06:00:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83626 invoked by uid 99); 11 Jan 2011 06:00:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 06:00:50 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.97 as permitted sender)
Received: from [17.148.16.97] (HELO asmtpout022.mac.com) (17.148.16.97)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 06:00:43 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp022.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LEU001LEGOLEP60@asmtp022.mac.com>
 for general@hadoop.apache.org; Mon, 10 Jan 2011 22:00:22 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-11_03:2011-01-11,2011-01-11,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101100168
Subject: Re: email lists: too many or not enough?
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <AANLkTine=OQYB9tuHwV6z1aXvQMN9KzSp0ztfzLAvHV5@mail.gmail.com>
Date: Mon, 10 Jan 2011 22:00:21 -0800
Message-id: <C74475D4-7BF3-49F8-B910-0E53B20D49D2@mac.com>
References: <A38987A0-7069-43DF-8422-7742AF8613EE@mac.com>
 <8C43652C-3B79-4583-B69E-C735B0084F10@yahoo-inc.com>
 <AANLkTine=OQYB9tuHwV6z1aXvQMN9KzSp0ztfzLAvHV5@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

> That what common-user@ is for.

Is it?  AFAICT we don't try to move user discussions there.  Nor do I see it documented anywhere that common-user@ is for cross hadoop user questions.  Also, FWIW, that list is documented at hadoop.apache.org -> common -> developers -> mailing lists -- not very intuitive.

Nige


On Jan 10, 2011, at 9:51 PM, Konstantin Boudnik wrote:

> On Mon, Jan 10, 2011 at 21:23, Eric Baldeschwieler <eric14@yahoo-inc.com> wrote:
>> I'd love to have a list that covers most of what is on general (big topics on governance, etc), but does not have usage questions on it.  I'd suggest creating a new list for that and letting general become a general questions list.
> 
> That what common-user@ is for.
> 
>> On Jan 10, 2011, at 6:55 PM, Nigel Daley wrote:
>> 
>>> This project has a lot of email lists [*] -- but I feel we're either missing 1 or 2 lists or we have too many.  Given all our lists, it's not clear which one we should use to talk about cross project dev issues and cross project community/policy issues.  general@ has perhaps become a catch all list with user, dev, and community issues inter-mixed.  Is that what we want?  Should we create more lists to break out these topics?  Live with the current inter-mingling? Or consolidate some lists?
>>> 
>>> Nige
>>> 
>>> [*] I may have missed some:
>>> general@
>>> security@
>>> private@
>>> common-user@
>>> common-dev@
>>> common-commits@
>>> common-issues@
>>> hdfs-user@
>>> hdfs-dev@
>>> hdfs-commits@
>>> hdfs-issues@
>>> mapreduce-user@
>>> mapreduce-dev@
>>> mapreduce-commits@
>>> mapreduce-issues@
>> 
>> 


From general-return-2703-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 11 07:12:28 2011
Return-Path: <general-return-2703-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57858 invoked from network); 11 Jan 2011 07:12:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jan 2011 07:12:28 -0000
Received: (qmail 34536 invoked by uid 500); 11 Jan 2011 07:12:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34242 invoked by uid 500); 11 Jan 2011 07:12:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34234 invoked by uid 99); 11 Jan 2011 07:12:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 07:12:23 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 07:12:19 +0000
Received: from [192.168.1.71] (snvvpn1-10-72-244-c4.hq.corp.yahoo.com [10.72.244.4])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0B7BZ6q008856
	for <general@hadoop.apache.org>; Mon, 10 Jan 2011 23:11:36 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1294729896;
	bh=Iaue2uVd7N4CvUy25Laz1uCogle//A7zsL3HEbkoZqk=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:Mime-Version:Subject:
	 Date:References;
	b=ON/WWtlFJc+zBYWqEODTwL2x7IJmxlUKUDL4yxOvcuGME8rpvFQgnEFTIWfjzlQy1
	 Te9sATG+wYFgx3sANPUr+LTSx0UGh/KLy0PStas6GOxwN0cbbniBn2eFu1QX6gl+t2
	 cfNtztPjoM5BnRRKOnEIBE2eyKMMLvYQG+f59wk4=
Message-Id: <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
Content-Type: multipart/alternative; boundary=Apple-Mail-39-742599375
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Date: Mon, 10 Jan 2011 23:11:35 -0800
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
X-Mailer: Apple Mail (2.936)

--Apple-Mail-39-742599375
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit

On 10/15/2010 02:28 PM, Doug Cutting wrote:

> On 08/30/2010 02:14 PM, Arun C Murthy wrote:
> Since most people seemed to think of it as a reasonable idea, I'm  
> going
> to create the hadoop-0.20-security branch and start the necessary  
> work.
>
> I don't yet see this branch.  Are you still intending to do this?
>
> Doug

Things stalled, my apologies. Turns out having a kid is a lot of work,  
who knew! *smile*

I'm back now and plan to start work on this. Hopefully I can get this  
over with quickly, in a couple of weeks, to focus on the next  
release(s).

thanks,
Arun
--Apple-Mail-39-742599375--

From general-return-2704-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 11 07:24:24 2011
Return-Path: <general-return-2704-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59271 invoked from network); 11 Jan 2011 07:24:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jan 2011 07:24:24 -0000
Received: (qmail 40147 invoked by uid 500); 11 Jan 2011 07:24:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39865 invoked by uid 500); 11 Jan 2011 07:24:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39839 invoked by uid 99); 11 Jan 2011 07:24:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 07:24:17 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of iphulari@gmail.com designates 209.85.210.48 as permitted sender)
Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 07:24:12 +0000
Received: by pzk28 with SMTP id 28so3498828pzk.35
        for <multiple recipients>; Mon, 10 Jan 2011 23:23:52 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=wC46xZuuPmI3Ecj9zVN9U/Yi2/qVhio1q3XgHxHlXtw=;
        b=m0PYy+cYYXJ2NdkkLVR0AlSr1RpGALSxqDXSZKsNLJUPkUYIJgFj1yXp1d9HpvcTy5
         pflVTbN1oHGY1rmbjXkf4jBzr6jKB1q9W55SjYxWCbyzxMEqbRDZRf8FuEnf28WI1bPb
         TY+b3af2N17p6A3Y4ZQ3N4yJ8te7tao7FjVUg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=T/unycwOz8YxP82jvVV7PAKfyx3RHw+84oovoHb2Dqx3oS36XMPzDu3oWtAOLmXgu7
         NCEOdIRyu0NRYIJCHtxytv/10PH/CK7QBUJaBHeo4xYdfWQH9gdkPGNDF+2Tu4+iKJRv
         F55T/F65D2RD5lr0MwKrtvjklMdLePSxJwxng=
MIME-Version: 1.0
Received: by 10.142.170.9 with SMTP id s9mr5250467wfe.187.1294730632147; Mon,
 10 Jan 2011 23:23:52 -0800 (PST)
Received: by 10.142.80.17 with HTTP; Mon, 10 Jan 2011 23:23:52 -0800 (PST)
Date: Mon, 10 Jan 2011 23:23:52 -0800
Message-ID: <AANLkTimhHEE=y6so4y=ohZAU9M2ESBpoTj65g0_uajt8@mail.gmail.com>
Subject: Reposting - Hadoop log timestamps & file timestamps not same as
 system time.
From: Ravi Phulari <iphulari@gmail.com>
To: hdfs-user@hadoop.apache.org, mapreduce-user@hadoop.apache.org, 
	general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd32db268b23b04998cf72e

--000e0cd32db268b23b04998cf72e
Content-Type: text/plain; charset=ISO-8859-1

Hello Friends,
 I am seeing Hadoop log timestamps & file timestamps not same as system
time. I found that this problem was discussed on on mailing list earlier but
there was no solution posted.

  http://www.mail-archive.com/hdfs-user@hadoop.apache.org/msg00310.html

I am wondering what is causing this behaviour and how to fix it.

As you can see below system time is in UTC and Hadoop is showing EST.

[hadoop@hadoop~]$ date
Mon Jan 10 21:54:17 UTC 2011

[hadoop@hadoop ~]$ hadoop jar $HADOOP_HOME/*examples.jar teragen  1000
tera-dir
Generating 1000 using 2 maps with step of 500
11/01/10 16:50:33 INFO mapred.JobClient: Running job: job_201101061958_2590
11/01/10 16:50:34 INFO mapred.JobClient:  map 0% reduce 0%
11/01/10 16:50:40 INFO mapred.JobClient:  map 100% reduce 0%
11/01/10 16:50:40 INFO mapred.JobClient: Job complete: job_201101061958_2590
11/01/10 16:50:40 INFO mapred.JobClient: Counters: 6
11/01/10 16:50:40 INFO mapred.JobClient:   Job Counters
11/01/10 16:50:40 INFO mapred.JobClient:     Launched map tasks=2
11/01/10 16:50:40 INFO mapred.JobClient:   FileSystemCounters

I thought possible workaround for this issue to add -Duser.timezone in
conf/hadoop-env.sh but it works only on my localsystem but not on cluster.

Why is hadoop reporting wrong EST time format? From where hadoop reads time?

Any help is greatly appreciated.
Thanks in advance,
Ravi

--000e0cd32db268b23b04998cf72e--

From general-return-2705-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 11 09:09:59 2011
Return-Path: <general-return-2705-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 878 invoked from network); 11 Jan 2011 09:09:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jan 2011 09:09:59 -0000
Received: (qmail 62369 invoked by uid 500); 11 Jan 2011 09:09:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62060 invoked by uid 500); 11 Jan 2011 09:09:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62046 invoked by uid 99); 11 Jan 2011 09:09:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 09:09:53 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [85.128.214.242] (HELO magdalenammc.nazwa.pl) (85.128.214.242)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 09:09:46 +0000
Received: from Pavilionl (unknown [62.233.208.194])
	by magdalenammc.nazwa.pl (Postfix) with ESMTP id 4622F2FA8CA3
	for <general@hadoop.apache.org>; Tue, 11 Jan 2011 10:09:26 +0100 (CET)
From: =?iso-8859-2?Q?Magdalena_Moll-Musia=B3?= <office@mm-consulting.com.pl>
To: <general@hadoop.apache.org>
References: <A38987A0-7069-43DF-8422-7742AF8613EE@mac.com> <8C43652C-3B79-4583-B69E-C735B0084F10@yahoo-inc.com> <AANLkTine=OQYB9tuHwV6z1aXvQMN9KzSp0ztfzLAvHV5@mail.gmail.com>
In-Reply-To: <AANLkTine=OQYB9tuHwV6z1aXvQMN9KzSp0ztfzLAvHV5@mail.gmail.com>
Subject: Job opportunity - Hadoop
Date: Tue, 11 Jan 2011 10:09:24 +0100
Message-ID: <003c01cbb16f$3dd6e1d0$b984a570$@com.pl>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-2"
Content-Transfer-Encoding: quoted-printable
X-Mailer: Microsoft Office Outlook 12.0
Thread-Index: AcuxU7lVzaDt7I7FQVWzEV4twNUtvwAGvcdA
Content-Language: pl
X-Virus-Checked: Checked by ClamAV on apache.org

Hello,=20

I am Recruitment Agency and I am looking for Hadoop specialists for this
role:

http://www.mm-consulting.com.pl/analitics.html

It is a permanent position in Berlin, Germany. Salary - 40-65 k =
depending on
candidate's seniority.
=20
The offer is for candidates from Europe and US.

If you are interested, please send me a copy of your CV and suggest when =
I
can call you via phone or via skype.

Thank you.

Best regards,
Magda=20


______________________________________
mmConsulting Magdalena Moll-Musia=B3
recruitment partner=20
+48 516 340 127
office@mm-consulting.com.pl
www.mm-consulting.com.pl
=20

__________ Informacja programu ESET NOD32 Antivirus, wersja bazy =
sygnatur
wirusow 5776 (20110110) __________

Wiadomosc zostala sprawdzona przez program ESET NOD32 Antivirus.
http://www.eset.pl lub http://www.eset.com=20
=20


From general-return-2706-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 11 17:07:27 2011
Return-Path: <general-return-2706-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93348 invoked from network); 11 Jan 2011 17:07:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jan 2011 17:07:27 -0000
Received: (qmail 86011 invoked by uid 500); 11 Jan 2011 17:07:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85863 invoked by uid 500); 11 Jan 2011 17:07:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85855 invoked by uid 99); 11 Jan 2011 17:07:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 17:07:22 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 17:07:17 +0000
Received: by ewy20 with SMTP id 20so6086875ewy.35
        for <general@hadoop.apache.org>; Tue, 11 Jan 2011 09:06:55 -0800 (PST)
Received: by 10.216.4.82 with SMTP id 60mr3057782wei.89.1294765614980; Tue, 11
 Jan 2011 09:06:54 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.59.13 with HTTP; Tue, 11 Jan 2011 09:06:34 -0800 (PST)
In-Reply-To: <C74475D4-7BF3-49F8-B910-0E53B20D49D2@mac.com>
References: <A38987A0-7069-43DF-8422-7742AF8613EE@mac.com> <8C43652C-3B79-4583-B69E-C735B0084F10@yahoo-inc.com>
 <AANLkTine=OQYB9tuHwV6z1aXvQMN9KzSp0ztfzLAvHV5@mail.gmail.com> <C74475D4-7BF3-49F8-B910-0E53B20D49D2@mac.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Tue, 11 Jan 2011 09:06:34 -0800
X-Google-Sender-Auth: fUZUnKuFt_lXDBOefQjnkplFJEc
Message-ID: <AANLkTimFAo9K_cNwL0cK0zHySkOWRw3_uit0Wjb3UEMx@mail.gmail.com>
Subject: Re: email lists: too many or not enough?
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Mon, Jan 10, 2011 at 22:00, Nigel Daley <ndaley@mac.com> wrote:
>> That what common-user@ is for.
>
> Is it? =A0AFAICT we don't try to move user discussions there. =A0Nor do I=
 see it documented anywhere that common-user@ is for cross hadoop user ques=
tions. =A0Also, FWIW, that list is documented at hadoop.apache.org -> commo=
n -> developers -> mailing lists -- not very intuitive.

While I in agreement with your logic above.. Let's see... If I have a
use question about Hadoop (not HDFS nor MR) where would I go? general@
doesn't sound very convincing... A-ha! common-user@.... *firing up
email client*

Instead of creating new lists to add an insult to the injury perhaps
we shall just make the intention for the existing ones more apparent?
E.g. re-organize 'mailing lists' section or something? I believe
developers, in general, are able to find information on their own.
Users sometimes demonstrate a lack of such ability ;) And we are well
beyond 'dev. only' community as far as I can see.

Cos

> Nige
>
>
> On Jan 10, 2011, at 9:51 PM, Konstantin Boudnik wrote:
>
>> On Mon, Jan 10, 2011 at 21:23, Eric Baldeschwieler <eric14@yahoo-inc.com=
> wrote:
>>> I'd love to have a list that covers most of what is on general (big top=
ics on governance, etc), but does not have usage questions on it. =A0I'd su=
ggest creating a new list for that and letting general become a general que=
stions list.
>>
>> That what common-user@ is for.
>>
>>> On Jan 10, 2011, at 6:55 PM, Nigel Daley wrote:
>>>
>>>> This project has a lot of email lists [*] -- but I feel we're either m=
issing 1 or 2 lists or we have too many. =A0Given all our lists, it's not c=
lear which one we should use to talk about cross project dev issues and cro=
ss project community/policy issues. =A0general@ has perhaps become a catch =
all list with user, dev, and community issues inter-mixed. =A0Is that what =
we want? =A0Should we create more lists to break out these topics? =A0Live =
with the current inter-mingling? Or consolidate some lists?
>>>>
>>>> Nige
>>>>
>>>> [*] I may have missed some:
>>>> general@
>>>> security@
>>>> private@
>>>> common-user@
>>>> common-dev@
>>>> common-commits@
>>>> common-issues@
>>>> hdfs-user@
>>>> hdfs-dev@
>>>> hdfs-commits@
>>>> hdfs-issues@
>>>> mapreduce-user@
>>>> mapreduce-dev@
>>>> mapreduce-commits@
>>>> mapreduce-issues@
>>>
>>>
>
>

From general-return-2707-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 11 17:39:18 2011
Return-Path: <general-return-2707-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9094 invoked from network); 11 Jan 2011 17:39:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jan 2011 17:39:18 -0000
Received: (qmail 49158 invoked by uid 500); 11 Jan 2011 17:39:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48998 invoked by uid 500); 11 Jan 2011 17:39:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48990 invoked by uid 99); 11 Jan 2011 17:39:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 17:39:16 +0000
X-ASF-Spam-Status: No, hits=-1997.8 required=10.0
	tests=ALL_TRUSTED,HTML_MESSAGE,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 11 Jan 2011 17:39:15 +0000
Received: (qmail 8942 invoked by uid 99); 11 Jan 2011 17:38:55 -0000
Received: from localhost.apache.org (HELO mail-bw0-f48.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 17:38:55 +0000
Received: by bwz8 with SMTP id 8so20904079bwz.35
        for <general@hadoop.apache.org>; Tue, 11 Jan 2011 09:38:53 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.127.89 with SMTP id f25mr15347866bks.143.1294767533471;
 Tue, 11 Jan 2011 09:38:53 -0800 (PST)
Received: by 10.204.61.73 with HTTP; Tue, 11 Jan 2011 09:38:53 -0800 (PST)
In-Reply-To: <C74475D4-7BF3-49F8-B910-0E53B20D49D2@mac.com>
References: <A38987A0-7069-43DF-8422-7742AF8613EE@mac.com>
	<8C43652C-3B79-4583-B69E-C735B0084F10@yahoo-inc.com>
	<AANLkTine=OQYB9tuHwV6z1aXvQMN9KzSp0ztfzLAvHV5@mail.gmail.com>
	<C74475D4-7BF3-49F8-B910-0E53B20D49D2@mac.com>
Date: Tue, 11 Jan 2011 09:38:53 -0800
Message-ID: <AANLkTi=49KJ1XZ_S6Qx6QKWww92QdgSQmu+gnuGJMhGJ@mail.gmail.com>
Subject: Re: email lists: too many or not enough?
From: "Owen O'Malley" <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6db2934e63a850499958e1e

--0016e6db2934e63a850499958e1e
Content-Type: text/plain; charset=UTF-8

On Mon, Jan 10, 2011 at 10:00 PM, Nigel Daley <ndaley@mac.com> wrote:

> > That what common-user@ is for.
>
> Is it?  AFAICT we don't try to move user discussions there.


We do try to move the questions off of general. It would be great if someone
updated the website with the intended usage of each of the lists.

Actually, we also need to move Zookeeper to the related projects of the
site.

-- Owen

--0016e6db2934e63a850499958e1e--

From general-return-2708-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 11 17:50:32 2011
Return-Path: <general-return-2708-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14886 invoked from network); 11 Jan 2011 17:50:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jan 2011 17:50:32 -0000
Received: (qmail 71648 invoked by uid 500); 11 Jan 2011 17:50:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71574 invoked by uid 500); 11 Jan 2011 17:50:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71566 invoked by uid 99); 11 Jan 2011 17:50:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 17:50:30 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 17:50:26 +0000
Received: by fxm2 with SMTP id 2so20104822fxm.35
        for <general@hadoop.apache.org>; Tue, 11 Jan 2011 09:50:05 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=nBQgHicJE1go7gPMUMEvX9mDClbqzSjrBrtiKVaTUc4=;
        b=LFLic7cgwfLM0oywunoPSwW/I8XLuLJ9mmKsquNP7Ww76HuHlGQYF7c6OmIovCDI5j
         ksQdhXsUg4sAe0V8VOw1Rrmzd88FoEZtiqeE4wUOP+eZ7cntNNXtgl2cWBI5iWg668+k
         oe63MccAu1b5wbsvjFgrxxuJEz8tAWD6TpvPQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=q2pAqJ57Q26SM3SeN4VkVrWYgQzV+N0lIvNcxfhMD0B6xHhixrRdB0TLTJl1InFmU4
         rKl1lPmw98uvBo2DA4JZ7vFy4T1KTABK39bn/mrqrc+lsN7bhBxWLINp8FnI2xpuzkiW
         g9yDyihnuJ6hG20/PpZYdp2ulL5BRuYCMm4zA=
Received: by 10.223.72.9 with SMTP id k9mr1318530faj.93.1294768205337; Tue, 11
 Jan 2011 09:50:05 -0800 (PST)
MIME-Version: 1.0
Received: by 10.223.120.14 with HTTP; Tue, 11 Jan 2011 09:49:36 -0800 (PST)
In-Reply-To: <AANLkTimhHEE=y6so4y=ohZAU9M2ESBpoTj65g0_uajt8@mail.gmail.com>
References: <AANLkTimhHEE=y6so4y=ohZAU9M2ESBpoTj65g0_uajt8@mail.gmail.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Tue, 11 Jan 2011 23:19:36 +0530
Message-ID: <AANLkTingS2hQtLE7McvPUX9jx27WAL3Od8a5KhFLuwHx@mail.gmail.com>
Subject: Re: Reposting - Hadoop log timestamps & file timestamps not same as
 system time.
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

It is probably your TZ configuration. Your linked archive thread does
have a solution posted in it [exporting TZ to have the proper timezone
in the environment], have you tried that?

On Tue, Jan 11, 2011 at 12:53 PM, Ravi Phulari <iphulari@gmail.com> wrote:
> Hello Friends,
> =A0I am seeing Hadoop log timestamps & file timestamps not same as system
> time. I found that this problem was discussed on on mailing list earlier =
but
> there was no solution posted.
>
> =A0http://www.mail-archive.com/hdfs-user@hadoop.apache.org/msg00310.html
>
> I am wondering what is causing this behaviour and how to fix it.
>
> As you can see below system time is in UTC and Hadoop is showing EST.
>
> [hadoop@hadoop~]$ date
> Mon Jan 10 21:54:17 UTC 2011
>
> [hadoop@hadoop ~]$ hadoop jar $HADOOP_HOME/*examples.jar teragen =A01000
> tera-dir
> Generating 1000 using 2 maps with step of 500
> 11/01/10 16:50:33 INFO mapred.JobClient: Running job: job_201101061958_25=
90
> 11/01/10 16:50:34 INFO mapred.JobClient: =A0map 0% reduce 0%
> 11/01/10 16:50:40 INFO mapred.JobClient: =A0map 100% reduce 0%
> 11/01/10 16:50:40 INFO mapred.JobClient: Job complete: job_201101061958_2=
590
> 11/01/10 16:50:40 INFO mapred.JobClient: Counters: 6
> 11/01/10 16:50:40 INFO mapred.JobClient: =A0 Job Counters
> 11/01/10 16:50:40 INFO mapred.JobClient: =A0 =A0 Launched map tasks=3D2
> 11/01/10 16:50:40 INFO mapred.JobClient: =A0 FileSystemCounters
>
> I thought possible workaround for this issue to add -Duser.timezone in
> conf/hadoop-env.sh but it works only on my localsystem but not on cluster=
.
>
> Why is hadoop reporting wrong EST time format? From where hadoop reads ti=
me?
>
> Any help is greatly appreciated.
> Thanks in advance,
> Ravi
>



--=20
Harsh J
www.harshj.com

From general-return-2709-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 11 18:42:16 2011
Return-Path: <general-return-2709-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60318 invoked from network); 11 Jan 2011 18:42:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jan 2011 18:42:16 -0000
Received: (qmail 84756 invoked by uid 500); 11 Jan 2011 18:42:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84711 invoked by uid 500); 11 Jan 2011 18:42:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84703 invoked by uid 99); 11 Jan 2011 18:42:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 18:42:13 +0000
X-ASF-Spam-Status: No, hits=-1997.8 required=10.0
	tests=ALL_TRUSTED,HTML_MESSAGE,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 11 Jan 2011 18:42:12 +0000
Received: (qmail 60051 invoked by uid 99); 11 Jan 2011 18:41:50 -0000
Received: from localhost.apache.org (HELO mail-bw0-f48.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 18:41:50 +0000
Received: by bwz8 with SMTP id 8so20967758bwz.35
        for <multiple recipients>; Tue, 11 Jan 2011 10:41:46 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.71.77 with SMTP id g13mr2214713bkj.170.1294771306050; Tue,
 11 Jan 2011 10:41:46 -0800 (PST)
Received: by 10.204.61.73 with HTTP; Tue, 11 Jan 2011 10:41:45 -0800 (PST)
In-Reply-To: <AANLkTimhHEE=y6so4y=ohZAU9M2ESBpoTj65g0_uajt8@mail.gmail.com>
References: <AANLkTimhHEE=y6so4y=ohZAU9M2ESBpoTj65g0_uajt8@mail.gmail.com>
Date: Tue, 11 Jan 2011 10:41:45 -0800
Message-ID: <AANLkTi=xx-ddL+ZWucG4iw+UsP7XxKZccHFrLvSFMZbo@mail.gmail.com>
Subject: Re: Reposting - Hadoop log timestamps & file timestamps not same as
 system time.
From: "Owen O'Malley" <omalley@apache.org>
To: mapreduce-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502dd82c335ff0499966f9d
X-Virus-Checked: Checked by ClamAV on apache.org

--00504502dd82c335ff0499966f9d
Content-Type: text/plain; charset=UTF-8

On Mon, Jan 10, 2011 at 11:23 PM, Ravi Phulari <iphulari@gmail.com> wrote:

> Hello Friends,
>  I am seeing Hadoop log timestamps & file timestamps not same as system
> time.


 Please use common-user for general usage questions. General is for
announcements and discussions about the project.

Thanks,
   Owen

--00504502dd82c335ff0499966f9d--

From general-return-2710-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 11 18:50:37 2011
Return-Path: <general-return-2710-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63883 invoked from network); 11 Jan 2011 18:50:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jan 2011 18:50:37 -0000
Received: (qmail 95618 invoked by uid 500); 11 Jan 2011 18:50:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95516 invoked by uid 500); 11 Jan 2011 18:50:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95508 invoked by uid 99); 11 Jan 2011 18:50:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 18:50:35 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of samplise@gmail.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 18:50:27 +0000
Received: by gyf1 with SMTP id 1so9293554gyf.35
        for <general@hadoop.apache.org>; Tue, 11 Jan 2011 10:50:06 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=Y1ESiz+GtotFJxIok/bG7nZbDd1bwawpkgeuCNbtOJU=;
        b=ooqSD0PyyMzh6LupfIM/qpP5rPf8on/07otB3rnJKCjYlbYO4pS2kdKksmPKc2Nlj0
         tFwwN5ltLqZHxvPIU0S+JPQgtZI2qcwsOhi0gSrfiip3SNFDZjHW0FdcluJPuNZ8DNjk
         VRw1gG1Y02bgVpsdZNDoU1OTjuHGaJWIe3Oug=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=j5/8O3hjsgUKKAK2HtB0KuWIe4gDvNHI/IvQlWgbMzwVjoVRZSfVlYnhNqk93j3hVD
         az2fuVxqRxbRZkK+P9UuwAVXrmurnsFR+GlovgdvPGsg5vZR11jf5MuqZopNNTrGRJIZ
         Leakl/BEnXxNqbuyiSTVIJ//UNdDY1bHVxO3Q=
MIME-Version: 1.0
Received: by 10.150.57.3 with SMTP id f3mr548773yba.446.1294771806385; Tue, 11
 Jan 2011 10:50:06 -0800 (PST)
Received: by 10.151.14.3 with HTTP; Tue, 11 Jan 2011 10:50:06 -0800 (PST)
Date: Tue, 11 Jan 2011 07:50:06 -1100
Message-ID: <AANLkTi=wopmzNL3rQQwuL7pp5v7yH=49drgPGqaHDpzN@mail.gmail.com>
Subject: Application for testing
From: Bo Sang <samplise@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd30b1c95b67d0499968d6f
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd30b1c95b67d0499968d6f
Content-Type: text/plain; charset=ISO-8859-1

Hi, guys:

I have deployed a hadoop on our group's nodes. Could you recommend some
typical applications for me? I want to test whether it can really work and
observe its performance.

-- 
Best Regards!

Sincerely
Bo Sang

--000e0cd30b1c95b67d0499968d6f--

From general-return-2711-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 11 19:14:04 2011
Return-Path: <general-return-2711-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82184 invoked from network); 11 Jan 2011 19:14:04 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jan 2011 19:14:04 -0000
Received: (qmail 44373 invoked by uid 500); 11 Jan 2011 19:14:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44100 invoked by uid 500); 11 Jan 2011 19:14:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44083 invoked by uid 99); 11 Jan 2011 19:14:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 19:14:01 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 19:13:52 +0000
Received: by wye20 with SMTP id 20so21986402wye.35
        for <general@hadoop.apache.org>; Tue, 11 Jan 2011 11:13:32 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        bh=NEz7X9slbBgFTT1pKtV1ZPR1SDFw0iOe1lMlxO1jldc=;
        b=shEGZliOx3TAib0Rzer1qY8rCGCowLKNyGlSl/27CMja86vuDSnLL7gAMFGuI9oafr
         sUTbbVtSoQfnRHH7W4zvOh7Nc/O8sLSQxifdO67wulNpMcQwfbew05udf9jbwkR5jCcH
         s37DE+kqPDP7qcrVAC8u5RyOvhO3LrCRZy8fw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=rSAf6WNeyFxyGml+dWxxPn1yhVmTlo4VoKFt9dKL5Azwkk0Zz+dRgJoDBshoo2ekDr
         Ovfd5cW9PFxtecffBTeQ8ZomkpCGPYg7Nft9zZPU1fvOgVlIbf83Ci5SJeihHQwrxBsl
         nhf7Rze5Ze5BgRyKNZsZxhwr8CD0681ahsk6c=
MIME-Version: 1.0
Received: by 10.216.48.5 with SMTP id u5mr114739web.96.1294773212445; Tue, 11
 Jan 2011 11:13:32 -0800 (PST)
Sender: saint.ack@gmail.com
Received: by 10.216.49.11 with HTTP; Tue, 11 Jan 2011 11:13:31 -0800 (PST)
In-Reply-To: <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
	<366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
	<18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
Date: Tue, 11 Jan 2011 11:13:31 -0800
X-Google-Sender-Auth: i7QgO2gPIG3TfkXQUVr1dwDTfD4
Message-ID: <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Mon, Jan 10, 2011 at 11:11 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:
> Things stalled, my apologies. Turns out having a kid is a lot of work, who
> knew! *smile*
>

Really (smile -- congrats Arun).

> I'm back now and plan to start work on this. Hopefully I can get this over
> with quickly, in a couple of weeks, to focus on the next release(s).
>

What you thinking?  What'll you call it?

Good on you,
St.Ack

From general-return-2712-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 11 19:24:03 2011
Return-Path: <general-return-2712-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87235 invoked from network); 11 Jan 2011 19:24:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jan 2011 19:24:02 -0000
Received: (qmail 64409 invoked by uid 500); 11 Jan 2011 19:24:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 64351 invoked by uid 500); 11 Jan 2011 19:24:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 64343 invoked by uid 99); 11 Jan 2011 19:24:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 19:24:01 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 19:23:54 +0000
Received: by fxm2 with SMTP id 2so20196937fxm.35
        for <general@hadoop.apache.org>; Tue, 11 Jan 2011 11:23:34 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:content-type;
        bh=0mO9zRmnnS9XekAI9l2ht/y+CWYbgBIhGwcZw/RMU5Q=;
        b=M8xaIjPzpEca68+NPkrTZcUKwTiJxFwzeSzpIKSHd2yTA8dGbUgi2ihmv/b4xkeFym
         S22pQ2egi6Q6dmKjUbtgQU0NhzJSjBDsqZgOh4eFMawVuHUrkcSlz7CCXWrTD3H4pXi6
         AQSj9ElCrL2RpEpojoT/Hv05lrgX1/9yutrqc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=P1BBs5RWil7I+hT022fUPmNIohSAU864Z1h3N/kszzT0TCqJh34410oKnRYCrWTl0n
         45xb0jW/SdkQ0WGQ3U+FHCvVVEzGDa2uBB1GVSlHIU57UhkF7bsHkgN6A2sMTHNMNcg+
         aK0RcCfSBkKpUKs4wENRhPZKTQtpmvqPk12u8=
Received: by 10.223.72.202 with SMTP id n10mr3511768faj.74.1294773527722; Tue,
 11 Jan 2011 11:18:47 -0800 (PST)
MIME-Version: 1.0
Received: by 10.223.120.14 with HTTP; Tue, 11 Jan 2011 11:18:08 -0800 (PST)
In-Reply-To: <AANLkTi=wopmzNL3rQQwuL7pp5v7yH=49drgPGqaHDpzN@mail.gmail.com>
References: <AANLkTi=wopmzNL3rQQwuL7pp5v7yH=49drgPGqaHDpzN@mail.gmail.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Wed, 12 Jan 2011 00:48:08 +0530
Message-ID: <AANLkTim7uXXA=V7FacbDNxyQbsXttNiLXN=2wYvswZVi@mail.gmail.com>
Subject: Re: Application for testing
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Are you looking to benchmark? You could run Gridmix, if so.
http://hadoop.apache.org/mapreduce/docs/current/gridmix.html

On Wed, Jan 12, 2011 at 12:20 AM, Bo Sang <samplise@gmail.com> wrote:
> Hi, guys:
>
> I have deployed a hadoop on our group's nodes. Could you recommend some
> typical applications for me? I want to test whether it can really work and
> observe its performance.
>
> --
> Best Regards!
>
> Sincerely
> Bo Sang
>



-- 
Harsh J
www.harshj.com

From general-return-2713-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 12 04:36:09 2011
Return-Path: <general-return-2713-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69217 invoked from network); 12 Jan 2011 04:36:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2011 04:36:08 -0000
Received: (qmail 7901 invoked by uid 500); 12 Jan 2011 04:36:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7450 invoked by uid 500); 12 Jan 2011 04:36:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7441 invoked by uid 99); 12 Jan 2011 04:36:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 04:36:02 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.105 as permitted sender)
Received: from [17.148.16.105] (HELO asmtpout030.mac.com) (17.148.16.105)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 04:35:54 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; charset=us-ascii
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp030.mac.com (Sun Java(tm) System Messaging Server 6.3-8.01 (built Dec
 16 2008; 32bit)) with ESMTPSA id <0LEW00JJ67EL2Q10@asmtp030.mac.com> for
 general@hadoop.apache.org; Tue, 11 Jan 2011 20:35:10 -0800 (PST)
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101110167
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-12_02:2011-01-12,2011-01-12,1970-01-01 signatures=0
Subject: Re: email lists: too many or not enough?
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <AANLkTi=49KJ1XZ_S6Qx6QKWww92QdgSQmu+gnuGJMhGJ@mail.gmail.com>
Date: Tue, 11 Jan 2011 20:35:09 -0800
Message-id: <93FCAFF3-8F8F-4BD5-8A46-60771D6CDE6C@mac.com>
References: <A38987A0-7069-43DF-8422-7742AF8613EE@mac.com>
 <8C43652C-3B79-4583-B69E-C735B0084F10@yahoo-inc.com>
 <AANLkTine=OQYB9tuHwV6z1aXvQMN9KzSp0ztfzLAvHV5@mail.gmail.com>
 <C74475D4-7BF3-49F8-B910-0E53B20D49D2@mac.com>
 <AANLkTi=49KJ1XZ_S6Qx6QKWww92QdgSQmu+gnuGJMhGJ@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)


On Jan 11, 2011, at 9:38 AM, Owen O'Malley wrote:

> On Mon, Jan 10, 2011 at 10:00 PM, Nigel Daley <ndaley@mac.com> wrote:
> 
>>> That what common-user@ is for.
>> 
>> Is it?  AFAICT we don't try to move user discussions there.
> 
> 
> We do try to move the questions off of general. It would be great if someone
> updated the website with the intended usage of each of the lists.

Ok, I'll work on a page.

> Actually, we also need to move Zookeeper to the related projects of the
> site.

I'll do that too.  Can't remember -- do we file Jira's and patches for site changes?  I couldn't find a component for site in Hadoop Jira.

Cheers,
Nige


From general-return-2714-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 12 05:10:21 2011
Return-Path: <general-return-2714-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87819 invoked from network); 12 Jan 2011 05:10:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2011 05:10:20 -0000
Received: (qmail 36661 invoked by uid 500); 12 Jan 2011 05:10:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36403 invoked by uid 500); 12 Jan 2011 05:10:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36392 invoked by uid 99); 12 Jan 2011 05:10:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 05:10:16 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 05:10:07 +0000
Received: from SP2-EX07CAS05.ds.corp.yahoo.com (sp2-ex07cas05.corp.sp2.yahoo.com [98.137.59.39])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p0C59WC1031506
	for <general@hadoop.apache.org>; Tue, 11 Jan 2011 21:09:32 -0800 (PST)
Received: from SP2-EX07VS04.ds.corp.yahoo.com ([98.137.59.33]) by
 SP2-EX07CAS05.ds.corp.yahoo.com ([98.137.59.39]) with mapi; Tue, 11 Jan 2011
 21:09:31 -0800
From: Arun C Murthy <arunc@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 11 Jan 2011 21:09:21 -0800
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Thread-Topic: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Thread-Index: AcuyFuUQTxglYTDWRCaST/brpshVPw==
Message-ID: <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
 <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
 <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
 <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
In-Reply-To: <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

On Jan 11, 2011, at 11:14 AM, "Stack" <stack@duboce.net> wrote:

>> I'm back now and plan to start work on this. Hopefully I can get this ov=
er
>> with quickly, in a couple of weeks, to focus on the next release(s).
>>=20
>=20
> What you thinking?  What'll you call it?
>=20
> Good on you,
> St.Ack

Thanks Stack.=20

I'm open to suggestions - how about something like 20.100 to show that it's=
 a big jump? Anything else?

Arun=

From general-return-2715-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 12 05:44:07 2011
Return-Path: <general-return-2715-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 11172 invoked from network); 12 Jan 2011 05:44:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2011 05:44:07 -0000
Received: (qmail 70371 invoked by uid 500); 12 Jan 2011 05:44:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69970 invoked by uid 500); 12 Jan 2011 05:44:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69954 invoked by uid 99); 12 Jan 2011 05:44:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 05:44:01 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 05:44:01 +0000
Received: by yxm8 with SMTP id 8so107879yxm.35
        for <multiple recipients>; Tue, 11 Jan 2011 21:43:40 -0800 (PST)
Received: by 10.236.109.7 with SMTP id r7mr1012366yhg.80.1294811020042;
        Tue, 11 Jan 2011 21:43:40 -0800 (PST)
Received: from tp (c-24-4-185-157.hsd1.ca.comcast.net [24.4.185.157])
        by mx.google.com with ESMTPS id 7sm202448yhl.27.2011.01.11.21.43.37
        (version=SSLv3 cipher=RC4-MD5);
        Tue, 11 Jan 2011 21:43:38 -0800 (PST)
Sender: Konstantin Boudnik <cos@boudnik.org>
Date: Tue, 11 Jan 2011 21:43:34 -0800
From: Konstantin Boudnik <cos@apache.org>
To: common-user@hadoop.apache.org
Subject: Re: Application for testing
Message-ID: <20110112054334.GA21574@tp>
Reply-To: common-user@hadoop.apache.org
References: <AANLkTi=wopmzNL3rQQwuL7pp5v7yH=49drgPGqaHDpzN@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature"; boundary="bp/iNruPH9dso1Pn"
Content-Disposition: inline
In-Reply-To: <AANLkTi=wopmzNL3rQQwuL7pp5v7yH=49drgPGqaHDpzN@mail.gmail.com>
X-Organization: It's something of 'Cos
X-PGP-Key: http://www.boudnik.org/~cos/pubkey.asc
User-Agent: Mutt/1.5.20 (2009-06-14)

--bp/iNruPH9dso1Pn
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

(Moving general@ to Bcc: list)

Bo, you can try to run TeraSort from Hadoop examples: you'll see if the
cluster is up and running and cen compare its performance between upgrades,=
 if
needed.

Also, please don't use general@ for user questions: there's common-user@ li=
st
exactly for these purposes.

With regards,
  Cos

On Tue, Jan 11, 2011 at 07:50AM, Bo Sang wrote:
> Hi, guys:
>=20
> I have deployed a hadoop on our group's nodes. Could you recommend some
> typical applications for me? I want to test whether it can really work and
> observe its performance.
>=20
> --=20
> Best Regards!
>=20
> Sincerely
> Bo Sang

--bp/iNruPH9dso1Pn
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iF4EAREIAAYFAk0tP4UACgkQenyFlstYjhJxgQEAtmXhVttWmpTwH0YdjWnwCvNf
rwt1Wmpph8c45wD/vh4BALqnCNpf5m04mAKJyQ+BuLopHYWCk45V9HK3ewsHcEBS
=3+ZJ
-----END PGP SIGNATURE-----

--bp/iNruPH9dso1Pn--

From general-return-2716-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 12 05:46:44 2011
Return-Path: <general-return-2716-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12131 invoked from network); 12 Jan 2011 05:46:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2011 05:46:44 -0000
Received: (qmail 77945 invoked by uid 500); 12 Jan 2011 05:46:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77747 invoked by uid 500); 12 Jan 2011 05:46:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77739 invoked by uid 99); 12 Jan 2011 05:46:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 05:46:39 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 05:46:33 +0000
Received: by wwd20 with SMTP id 20so214738wwd.29
        for <general@hadoop.apache.org>; Tue, 11 Jan 2011 21:46:12 -0800 (PST)
Received: by 10.216.4.82 with SMTP id 60mr3592952wei.89.1294811172020; Tue, 11
 Jan 2011 21:46:12 -0800 (PST)
MIME-Version: 1.0
Received: by 10.216.59.13 with HTTP; Tue, 11 Jan 2011 21:45:51 -0800 (PST)
In-Reply-To: <93FCAFF3-8F8F-4BD5-8A46-60771D6CDE6C@mac.com>
References: <A38987A0-7069-43DF-8422-7742AF8613EE@mac.com> <8C43652C-3B79-4583-B69E-C735B0084F10@yahoo-inc.com>
 <AANLkTine=OQYB9tuHwV6z1aXvQMN9KzSp0ztfzLAvHV5@mail.gmail.com>
 <C74475D4-7BF3-49F8-B910-0E53B20D49D2@mac.com> <AANLkTi=49KJ1XZ_S6Qx6QKWww92QdgSQmu+gnuGJMhGJ@mail.gmail.com>
 <93FCAFF3-8F8F-4BD5-8A46-60771D6CDE6C@mac.com>
From: Konstantin Boudnik <cos@boudnik.org>
Date: Tue, 11 Jan 2011 21:45:51 -0800
Message-ID: <AANLkTinrXJUo1CR7P9k5vbFLNYScqB1g1FRc8BTJmJbr@mail.gmail.com>
Subject: Re: email lists: too many or not enough?
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Another approach to clean-up general@ is to manually reroute (with
appropriate comment) user questions to common-user@ or other relevant
user lists.

On Tue, Jan 11, 2011 at 20:35, Nigel Daley <ndaley@mac.com> wrote:
>
> On Jan 11, 2011, at 9:38 AM, Owen O'Malley wrote:
>
>> On Mon, Jan 10, 2011 at 10:00 PM, Nigel Daley <ndaley@mac.com> wrote:
>>
>>>> That what common-user@ is for.
>>>
>>> Is it? =A0AFAICT we don't try to move user discussions there.
>>
>>
>> We do try to move the questions off of general. It would be great if som=
eone
>> updated the website with the intended usage of each of the lists.
>
> Ok, I'll work on a page.
>
>> Actually, we also need to move Zookeeper to the related projects of the
>> site.
>
> I'll do that too. =A0Can't remember -- do we file Jira's and patches for =
site changes? =A0I couldn't find a component for site in Hadoop Jira.
>
> Cheers,
> Nige
>
>

From general-return-2717-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 12 06:55:12 2011
Return-Path: <general-return-2717-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 41257 invoked from network); 12 Jan 2011 06:55:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2011 06:55:12 -0000
Received: (qmail 29311 invoked by uid 500); 12 Jan 2011 06:55:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28975 invoked by uid 500); 12 Jan 2011 06:55:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28966 invoked by uid 99); 12 Jan 2011 06:55:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 06:55:06 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 06:54:59 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0C6sGUj093787
	for <general@hadoop.apache.org>; Tue, 11 Jan 2011 22:54:16 -0800 (PST)
Received: from [10.0.1.5] (10.72.244.10) by SP2-EX07CAS01.ds.corp.yahoo.com
 (98.137.59.4) with Microsoft SMTP Server (TLS) id 8.3.106.1; Tue, 11 Jan 2011
 22:54:16 -0800
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0 (Apple Message framework v1082)
Subject: Re: email lists: too many or not enough?
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <93FCAFF3-8F8F-4BD5-8A46-60771D6CDE6C@mac.com>
Date: Tue, 11 Jan 2011 22:54:10 -0800
Content-Transfer-Encoding: quoted-printable
Message-ID: <C4B15A9A-5F5A-4B04-B63E-580BC533CFB2@yahoo-inc.com>
References: <A38987A0-7069-43DF-8422-7742AF8613EE@mac.com> <8C43652C-3B79-4583-B69E-C735B0084F10@yahoo-inc.com> <AANLkTine=OQYB9tuHwV6z1aXvQMN9KzSp0ztfzLAvHV5@mail.gmail.com> <C74475D4-7BF3-49F8-B910-0E53B20D49D2@mac.com> <AANLkTi=49KJ1XZ_S6Qx6QKWww92QdgSQmu+gnuGJMhGJ@mail.gmail.com> <93FCAFF3-8F8F-4BD5-8A46-60771D6CDE6C@mac.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)

Thanks nigel!!

I think some text on this page suggesting various lists for questions =
and a clean definition of what should go to general would go a long way.

http://hadoop.apache.org/mailing_lists.html


On Jan 11, 2011, at 8:35 PM, Nigel Daley wrote:

>=20
> On Jan 11, 2011, at 9:38 AM, Owen O'Malley wrote:
>=20
>> On Mon, Jan 10, 2011 at 10:00 PM, Nigel Daley <ndaley@mac.com> wrote:
>>=20
>>>> That what common-user@ is for.
>>>=20
>>> Is it?  AFAICT we don't try to move user discussions there.
>>=20
>>=20
>> We do try to move the questions off of general. It would be great if =
someone
>> updated the website with the intended usage of each of the lists.
>=20
> Ok, I'll work on a page.
>=20
>> Actually, we also need to move Zookeeper to the related projects of =
the
>> site.
>=20
> I'll do that too.  Can't remember -- do we file Jira's and patches for =
site changes?  I couldn't find a component for site in Hadoop Jira.
>=20
> Cheers,
> Nige
>=20


From general-return-2718-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 12 17:09:13 2011
Return-Path: <general-return-2718-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 20244 invoked from network); 12 Jan 2011 17:09:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2011 17:09:13 -0000
Received: (qmail 20605 invoked by uid 500); 12 Jan 2011 17:09:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20391 invoked by uid 500); 12 Jan 2011 17:09:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 20382 invoked by uid 99); 12 Jan 2011 17:09:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 17:09:08 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 17:09:01 +0000
Received: from SP2-EX07CAS04.ds.corp.yahoo.com (sp2-ex07cas04.corp.sp2.yahoo.com [98.137.59.5])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0CH829W084720
	for <general@hadoop.apache.org>; Wed, 12 Jan 2011 09:08:05 -0800 (PST)
Received: from SP2-EX07VS04.ds.corp.yahoo.com ([98.137.59.33]) by
 SP2-EX07CAS04.ds.corp.yahoo.com ([98.137.59.5]) with mapi; Wed, 12 Jan 2011
 09:08:03 -0800
From: Mahadev Konar <mahadev@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 12 Jan 2011 09:08:02 -0800
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Thread-Topic: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Thread-Index: AcuyFuUQTxglYTDWRCaST/brpshVPwAZGAFB
Message-ID: <C9531FF2.49CE1%mahadev@yahoo-inc.com>
In-Reply-To: <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
Accept-Language: en, en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en, en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

+1. I like the idea of 20.100.

Thanks
mahadev


On 1/11/11 9:09 PM, "Arun C Murthy" <arunc@yahoo-inc.com> wrote:

> On Jan 11, 2011, at 11:14 AM, "Stack" <stack@duboce.net> wrote:
>=20
>>> I'm back now and plan to start work on this. Hopefully I can get this o=
ver
>>> with quickly, in a couple of weeks, to focus on the next release(s).
>>>=20
>>=20
>> What you thinking?  What'll you call it?
>>=20
>> Good on you,
>> St.Ack
>=20
> Thanks Stack.
>=20
> I'm open to suggestions - how about something like 20.100 to show that it=
's a
> big jump? Anything else?
>=20
> Arun=20
>=20


From general-return-2719-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 12 17:16:44 2011
Return-Path: <general-return-2719-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23234 invoked from network); 12 Jan 2011 17:16:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2011 17:16:44 -0000
Received: (qmail 33808 invoked by uid 500); 12 Jan 2011 17:16:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33437 invoked by uid 500); 12 Jan 2011 17:16:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33428 invoked by uid 99); 12 Jan 2011 17:16:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 17:16:40 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 12 Jan 2011 17:16:39 +0000
Received: (qmail 23115 invoked by uid 99); 12 Jan 2011 17:16:19 -0000
Received: from localhost.apache.org (HELO [192.168.168.16]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 17:16:19 +0000
Message-ID: <4D2DE1E2.9030805@apache.org>
Date: Wed, 12 Jan 2011 09:16:18 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: email lists: too many or not enough?
References: <A38987A0-7069-43DF-8422-7742AF8613EE@mac.com> <8C43652C-3B79-4583-B69E-C735B0084F10@yahoo-inc.com> <AANLkTine=OQYB9tuHwV6z1aXvQMN9KzSp0ztfzLAvHV5@mail.gmail.com> <C74475D4-7BF3-49F8-B910-0E53B20D49D2@mac.com> <AANLkTi=49KJ1XZ_S6Qx6QKWww92QdgSQmu+gnuGJMhGJ@mail.gmail.com> <93FCAFF3-8F8F-4BD5-8A46-60771D6CDE6C@mac.com>
In-Reply-To: <93FCAFF3-8F8F-4BD5-8A46-60771D6CDE6C@mac.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 01/11/2011 08:35 PM, Nigel Daley wrote:
> I'll do that too.  Can't remember -- do we file Jira's and patches for site changes?  I couldn't find a component for site in Hadoop Jira.

I've often made trivial site changes without filing an issue.  For 
something that might need review I've filed an issue with 
component=documentation and version=site.  The latter keeps it from 
being associated with any release.  We could also add a 'site' component 
though.

Doug

From general-return-2720-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 12 17:27:40 2011
Return-Path: <general-return-2720-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29496 invoked from network); 12 Jan 2011 17:27:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2011 17:27:39 -0000
Received: (qmail 57358 invoked by uid 500); 12 Jan 2011 17:27:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57148 invoked by uid 500); 12 Jan 2011 17:27:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57140 invoked by uid 99); 12 Jan 2011 17:27:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 17:27:35 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of patrickangeles@gmail.com designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 17:27:30 +0000
Received: by ewy20 with SMTP id 20so382239ewy.35
        for <general@hadoop.apache.org>; Wed, 12 Jan 2011 09:27:09 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        bh=5wPC1MK9oj3JMav2Qr8XaIJhLxZfpcoLPdfR8CaF6VM=;
        b=PkjPdWav/L0Y5W5TqRyC9GwZMEbJIG6IAUvC1DkgpXuswnNHzmvbaoG8rQ4fJv5sxN
         lg6qqfWsZ1V0BhagvONEQWAhnsLPkV8K4nOGgZO5FoyJaZrNXdyhrPOIxWsCSjq7aApT
         KW0BirHn6PedS4MjNdxixeJogL7jfTn1eWpUw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=jY0I9hUHrfnCTze2c3azUo0W7fcDLTLh3Lu8haE1Z12fHejYZNSG0NZ8JGKW49/Hal
         4H6CMev+qjsXnV1YgWWh3Noirut4cY7iTQ0hsqLtMPk/IJ0DkfKLGTzsyoJzW8PqLo27
         qlVKWGpyTI5cwRsD8qLhGPCuKk7OjXTl6AYAw=
MIME-Version: 1.0
Received: by 10.213.32.193 with SMTP id e1mr1432532ebd.74.1294853229187; Wed,
 12 Jan 2011 09:27:09 -0800 (PST)
Sender: patrickangeles@gmail.com
Received: by 10.213.33.8 with HTTP; Wed, 12 Jan 2011 09:27:09 -0800 (PST)
In-Reply-To: <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
	<366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
	<18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
	<AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
	<F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
Date: Wed, 12 Jan 2011 12:27:09 -0500
X-Google-Sender-Auth: _likBWvF29T9SET7wLgT2NS_H1U
Message-ID: <AANLkTikEXfmK1jSv8GJyskH0Be+L5De+uF4KosZAfV6F@mail.gmail.com>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Patrick Angeles <patrick@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015174c3f04c315530499a982db

--0015174c3f04c315530499a982db
Content-Type: text/plain; charset=ISO-8859-1

You're gonna call your kid 20.100?

:)

Congratz

On Wed, Jan 12, 2011 at 12:09 AM, Arun C Murthy <arunc@yahoo-inc.com> wrote:

> On Jan 11, 2011, at 11:14 AM, "Stack" <stack@duboce.net> wrote:
>
> >> I'm back now and plan to start work on this. Hopefully I can get this
> over
> >> with quickly, in a couple of weeks, to focus on the next release(s).
> >>
> >
> > What you thinking?  What'll you call it?
> >
> > Good on you,
> > St.Ack
>
> Thanks Stack.
>
> I'm open to suggestions - how about something like 20.100 to show that it's
> a big jump? Anything else?
>
> Arun

--0015174c3f04c315530499a982db--

From general-return-2721-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 12 17:48:12 2011
Return-Path: <general-return-2721-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35550 invoked from network); 12 Jan 2011 17:48:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2011 17:48:12 -0000
Received: (qmail 86191 invoked by uid 500); 12 Jan 2011 17:48:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86139 invoked by uid 500); 12 Jan 2011 17:48:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86131 invoked by uid 99); 12 Jan 2011 17:48:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 17:48:10 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.103 as permitted sender)
Received: from [17.148.16.103] (HELO asmtpout028.mac.com) (17.148.16.103)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 17:48:00 +0000
MIME-version: 1.0
Content-type: multipart/alternative;
 boundary="Boundary_(ID_JHEPgmpPBxpVmuLHbEDIBA)"
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp028.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LEX0066E83120Y0@asmtp028.mac.com>
 for general@hadoop.apache.org; Wed, 12 Jan 2011 09:47:26 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-12_05:2011-01-12,2011-01-12,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101120090
From: Nigel Daley <ndaley@mac.com>
Subject: Re: email lists: too many or not enough?
Date: Wed, 12 Jan 2011 09:47:25 -0800
In-reply-to: <93FCAFF3-8F8F-4BD5-8A46-60771D6CDE6C@mac.com>
To: general@hadoop.apache.org
References: <A38987A0-7069-43DF-8422-7742AF8613EE@mac.com>
 <8C43652C-3B79-4583-B69E-C735B0084F10@yahoo-inc.com>
 <AANLkTine=OQYB9tuHwV6z1aXvQMN9KzSp0ztfzLAvHV5@mail.gmail.com>
 <C74475D4-7BF3-49F8-B910-0E53B20D49D2@mac.com>
 <AANLkTi=49KJ1XZ_S6Qx6QKWww92QdgSQmu+gnuGJMhGJ@mail.gmail.com>
 <93FCAFF3-8F8F-4BD5-8A46-60771D6CDE6C@mac.com>
Message-id: <4F749041-9774-4968-AE2C-52977651148A@mac.com>
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

--Boundary_(ID_JHEPgmpPBxpVmuLHbEDIBA)
Content-type: text/plain; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT


On Jan 11, 2011, at 8:35 PM, Nigel Daley wrote:
> 
> On Jan 11, 2011, at 9:38 AM, Owen O'Malley wrote:
>> We do try to move the questions off of general. It would be great if someone
>> updated the website with the intended usage of each of the lists.
> 
> Ok, I'll work on a page.

Done. See the revamped http://hadoop.apache.org/mailing_lists.html 
Corrections welcome.  You must now click on general@ to see the subscription instructions (it's a new page).
I couldn't figure out how to add cell padding around the table cells so it looks squished -- any forrest deities out there that can provide the right incantation?

>> Actually, we also need to move Zookeeper to the related projects of the
>> site.
> 
> I'll do that too. 

Done.  But didn't remove the tab or change the links.  They still point to hadoop.apache.org/zookeeper.  Turns out zookeeper.apache.org isn't fully ready according to Patrick.

Cheers,
Nige



--Boundary_(ID_JHEPgmpPBxpVmuLHbEDIBA)--

From general-return-2722-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 12 18:09:37 2011
Return-Path: <general-return-2722-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57313 invoked from network); 12 Jan 2011 18:09:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2011 18:09:36 -0000
Received: (qmail 23326 invoked by uid 500); 12 Jan 2011 18:09:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23267 invoked by uid 500); 12 Jan 2011 18:09:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23259 invoked by uid 99); 12 Jan 2011 18:09:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 18:09:34 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.30.40] (HELO qmta04.emeryville.ca.mail.comcast.net) (76.96.30.40)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 18:09:26 +0000
Received: from omta10.emeryville.ca.mail.comcast.net ([76.96.30.28])
	by qmta04.emeryville.ca.mail.comcast.net with comcast
	id ufVi1f0040cQ2SLA4i95FX; Wed, 12 Jan 2011 18:09:05 +0000
Received: from [10.72.184.42] ([209.131.62.113])
	by omta10.emeryville.ca.mail.comcast.net with comcast
	id ui8w1f0082SbwD58Wi8zdg; Wed, 12 Jan 2011 18:09:03 +0000
Message-Id: <DF3B1523-EB44-45C4-86E3-5B4BB6027294@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <4F749041-9774-4968-AE2C-52977651148A@mac.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: email lists: too many or not enough?
Date: Wed, 12 Jan 2011 10:08:56 -0800
References: <A38987A0-7069-43DF-8422-7742AF8613EE@mac.com> <8C43652C-3B79-4583-B69E-C735B0084F10@yahoo-inc.com> <AANLkTine=OQYB9tuHwV6z1aXvQMN9KzSp0ztfzLAvHV5@mail.gmail.com> <C74475D4-7BF3-49F8-B910-0E53B20D49D2@mac.com> <AANLkTi=49KJ1XZ_S6Qx6QKWww92QdgSQmu+gnuGJMhGJ@mail.gmail.com> <93FCAFF3-8F8F-4BD5-8A46-60771D6CDE6C@mac.com> <4F749041-9774-4968-AE2C-52977651148A@mac.com>
X-Mailer: Apple Mail (2.936)


On Jan 12, 2011, at 9:47 AM, Nigel Daley wrote:

>
> On Jan 11, 2011, at 8:35 PM, Nigel Daley wrote:
>>
>> On Jan 11, 2011, at 9:38 AM, Owen O'Malley wrote:
>>> We do try to move the questions off of general. It would be great  
>>> if someone
>>> updated the website with the intended usage of each of the lists.
>>
>> Ok, I'll work on a page.
>
> Done. See the revamped http://hadoop.apache.org/mailing_lists.html
> Corrections welcome.  You must now click on general@ to see the  
> subscription instructions (it's a new page).

I'd suggest segregating the list into 4 tables based on audience:

user questions:
common-user
hdfs-user
mapreduce-user
security - only for notifying the project of security vulnerabilities

project level announcements and discussions:
general

developer questions:
common-dev
hdfs-dev
mapreduce-dev

jira and subversion tracking for developers:
common-issues
common-commits
hdfs-issues
hdfs-commits

mapreduce-issues
mapreduce-commits

-- Owen

From general-return-2723-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 12 18:45:41 2011
Return-Path: <general-return-2723-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76363 invoked from network); 12 Jan 2011 18:45:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2011 18:45:40 -0000
Received: (qmail 63146 invoked by uid 500); 12 Jan 2011 18:45:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62970 invoked by uid 500); 12 Jan 2011 18:45:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 11381 invoked by uid 99); 12 Jan 2011 18:04:23 -0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of rakeshdav@gmail.com designates 209.85.216.169 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=hR8uIVej8Pgs5RgoyJjvOiLezAkfErJLFkjbEKMK6Fs=;
        b=ZlSuvxsolfZDRt1l9jf864fUvABcL+E3VV6TtpITNaqznzUx/N4ooPJECF3nDulkBy
         53AUlBuWT81XJ+0gK7PgvA/16lz+4ztgJ50yk9R9o6kJX1sQQmSRU86zTOPmd9dLGxB2
         iwMFzm44HTnpzO1LMXEZOo/SrhA0AdyX5E4c4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=n55MYfHQ/p7pguWLUmP8JAwqfLhfJuNG/OK9u3IRq4IEpP+nFxxwcdvrsUzDbfzNoi
         dbGDReCohvFTiwtG6gPC+9qq4xs2zrtEnixAF2bZHvsTlwwiQynae27hP6KITHZ3tXKX
         hpblfmKLLagr9Kxzy8tY8SJo/x8AFxfsmSuXs=
MIME-Version: 1.0
Date: Wed, 12 Jan 2011 23:33:56 +0530
Message-ID: <AANLkTikUYwYR+=ZLboQMU+Dv_9xGs1vuCtfsFFHZSM2B@mail.gmail.com>
Subject: Restricting number of records from map output
From: Rakesh Davanum <rakeshdav@gmail.com>
To: common-issues@hadoop.apache.org, common-user@hadoop.apache.org, 
	general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64c3c3855c8a30499aa06cc

--0016e64c3c3855c8a30499aa06cc
Content-Type: text/plain; charset=ISO-8859-1

Hi,

I have a sort job consisting of only the Mapper (no Reducer) task. I want my
results to contain only the top n records. Is there any way of restricting
the number of records that are emitted by the Mappers?

Basically I am looking to see if there is an equivalent of achieving
the behavior similar to LIMIT in SQL queries.

Thanks & Regards,
Rakesh

--0016e64c3c3855c8a30499aa06cc--

From general-return-2724-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 12 19:53:46 2011
Return-Path: <general-return-2724-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 13605 invoked from network); 12 Jan 2011 19:53:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2011 19:53:46 -0000
Received: (qmail 73539 invoked by uid 500); 12 Jan 2011 19:53:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73458 invoked by uid 500); 12 Jan 2011 19:53:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73450 invoked by uid 99); 12 Jan 2011 19:53:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 19:53:43 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 19:53:37 +0000
Received: by qyk10 with SMTP id 10so977638qyk.14
        for <general@hadoop.apache.org>; Wed, 12 Jan 2011 11:53:16 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=2nZI0eiy5taBvTz6wwg9QO2hEPWN0HI6OKmZrbaZrzA=;
        b=tdwJZTjCRXz1rEDUvbHhvgdl0E493o6mQ1nyRZt+Z+kFRpaePW5BVhkYa3wK/dhWR6
         0ZQc53i95G8LkncGxgqGqIgTyCC8XmYSsSRhBuJwKckBNzz1N1c+KxVGCIKRWzACPrDi
         QN/CoqZ9qlgFdMIlbt3jJSOHZFoZvkb5JJKPs=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=PZfeZ5rqqApuGNpcQ+ni5WJkp5+OamqRqL//uPXh6vRzkej6lQTgjCjXKIb0Hzlvgf
         Vik/l3pbQDzMdZN2zVGxgDg19D/jvQwgtoYLoNZXIsBesgO62WtcRyiKxD+N/miUxDzy
         aZkT7iK44aAiJE/81V13/2hgki/n/ygccYvJU=
Received: by 10.224.135.227 with SMTP id o35mr1291030qat.75.1294861996687;
        Wed, 12 Jan 2011 11:53:16 -0800 (PST)
Received: from [10.180.150.17] (h-64-236-128-41.nat.aol.com [64.236.128.41])
        by mx.google.com with ESMTPS id l12sm722175qcu.19.2011.01.12.11.53.13
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Wed, 12 Jan 2011 11:53:14 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: email lists: too many or not enough?
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <DF3B1523-EB44-45C4-86E3-5B4BB6027294@apache.org>
Date: Wed, 12 Jan 2011 14:53:12 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <796C0A0E-F54A-4B1A-8CD9-5C663F7330AF@Holsman.NET>
References: <A38987A0-7069-43DF-8422-7742AF8613EE@mac.com> <8C43652C-3B79-4583-B69E-C735B0084F10@yahoo-inc.com> <AANLkTine=OQYB9tuHwV6z1aXvQMN9KzSp0ztfzLAvHV5@mail.gmail.com> <C74475D4-7BF3-49F8-B910-0E53B20D49D2@mac.com> <AANLkTi=49KJ1XZ_S6Qx6QKWww92QdgSQmu+gnuGJMhGJ@mail.gmail.com> <93FCAFF3-8F8F-4BD5-8A46-60771D6CDE6C@mac.com> <4F749041-9774-4968-AE2C-52977651148A@mac.com> <DF3B1523-EB44-45C4-86E3-5B4BB6027294@apache.org>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

to add some metrics to this discussion.

we have ~3,500 unique email addresses subscribed to at least one hadoop =
mailing list.


subscribed to:
1 list : 2431
2 lists:  784
3 lists:  238
4 lists:  118
5 lists:   56
6 lists:   66
7 lists:  102=20

(I didn't look at the various 'issues or commits' mailing lists).

So I'm not really sure adding another list will get more focused =
discussions, or further split up the community.

On Jan 12, 2011, at 1:08 PM, Owen O'Malley wrote:

>=20
> On Jan 12, 2011, at 9:47 AM, Nigel Daley wrote:
>=20
>>=20
>> On Jan 11, 2011, at 8:35 PM, Nigel Daley wrote:
>>>=20
>>> On Jan 11, 2011, at 9:38 AM, Owen O'Malley wrote:
>>>> We do try to move the questions off of general. It would be great =
if someone
>>>> updated the website with the intended usage of each of the lists.
>>>=20
>>> Ok, I'll work on a page.
>>=20
>> Done. See the revamped http://hadoop.apache.org/mailing_lists.html
>> Corrections welcome.  You must now click on general@ to see the =
subscription instructions (it's a new page).
>=20
> I'd suggest segregating the list into 4 tables based on audience:
>=20
> user questions:
> common-user
> hdfs-user
> mapreduce-user
> security - only for notifying the project of security vulnerabilities
>=20
> project level announcements and discussions:
> general
>=20
> developer questions:
> common-dev
> hdfs-dev
> mapreduce-dev
>=20
> jira and subversion tracking for developers:
> common-issues
> common-commits
> hdfs-issues
> hdfs-commits
>=20
> mapreduce-issues
> mapreduce-commits
>=20
> -- Owen


From general-return-2725-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 12 20:15:46 2011
Return-Path: <general-return-2725-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31231 invoked from network); 12 Jan 2011 20:15:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2011 20:15:46 -0000
Received: (qmail 6669 invoked by uid 500); 12 Jan 2011 20:15:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6610 invoked by uid 500); 12 Jan 2011 20:15:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6602 invoked by uid 99); 12 Jan 2011 20:15:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 20:15:44 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.100 as permitted sender)
Received: from [17.148.16.100] (HELO asmtpout025.mac.com) (17.148.16.100)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 20:15:35 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp025.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LEX00A1JEWVAT00@asmtp025.mac.com>
 for general@hadoop.apache.org; Wed, 12 Jan 2011 12:14:56 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-12_05:2011-01-12,2011-01-12,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101120115
Subject: Re: email lists: too many or not enough?
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <796C0A0E-F54A-4B1A-8CD9-5C663F7330AF@Holsman.NET>
Date: Wed, 12 Jan 2011 12:14:54 -0800
Message-id: <ABEAFB4A-DE38-4122-B395-666C43C57F59@mac.com>
References: <A38987A0-7069-43DF-8422-7742AF8613EE@mac.com>
 <8C43652C-3B79-4583-B69E-C735B0084F10@yahoo-inc.com>
 <AANLkTine=OQYB9tuHwV6z1aXvQMN9KzSp0ztfzLAvHV5@mail.gmail.com>
 <C74475D4-7BF3-49F8-B910-0E53B20D49D2@mac.com>
 <AANLkTi=49KJ1XZ_S6Qx6QKWww92QdgSQmu+gnuGJMhGJ@mail.gmail.com>
 <93FCAFF3-8F8F-4BD5-8A46-60771D6CDE6C@mac.com>
 <4F749041-9774-4968-AE2C-52977651148A@mac.com>
 <DF3B1523-EB44-45C4-86E3-5B4BB6027294@apache.org>
 <796C0A0E-F54A-4B1A-8CD9-5C663F7330AF@Holsman.NET>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

Ya, if we keep general@ to it's intended purpose (and better document/enforce the use of various lists) then there is no need for a new list.  I'll make the edit's Owen suggested to the revamped mailing_lists.html.

I'm still confused, however, where release discussions should happen.  general@ or common-dev@?  I guess I'll leave them on general@ unless this is enough opposition to that.

Nige

On Jan 12, 2011, at 11:53 AM, Ian Holsman wrote:

> to add some metrics to this discussion.
> 
> we have ~3,500 unique email addresses subscribed to at least one hadoop mailing list.
> 
> 
> subscribed to:
> 1 list : 2431
> 2 lists:  784
> 3 lists:  238
> 4 lists:  118
> 5 lists:   56
> 6 lists:   66
> 7 lists:  102 
> 
> (I didn't look at the various 'issues or commits' mailing lists).
> 
> So I'm not really sure adding another list will get more focused discussions, or further split up the community.
> 
> On Jan 12, 2011, at 1:08 PM, Owen O'Malley wrote:
> 
>> 
>> On Jan 12, 2011, at 9:47 AM, Nigel Daley wrote:
>> 
>>> 
>>> On Jan 11, 2011, at 8:35 PM, Nigel Daley wrote:
>>>> 
>>>> On Jan 11, 2011, at 9:38 AM, Owen O'Malley wrote:
>>>>> We do try to move the questions off of general. It would be great if someone
>>>>> updated the website with the intended usage of each of the lists.
>>>> 
>>>> Ok, I'll work on a page.
>>> 
>>> Done. See the revamped http://hadoop.apache.org/mailing_lists.html
>>> Corrections welcome.  You must now click on general@ to see the subscription instructions (it's a new page).
>> 
>> I'd suggest segregating the list into 4 tables based on audience:
>> 
>> user questions:
>> common-user
>> hdfs-user
>> mapreduce-user
>> security - only for notifying the project of security vulnerabilities
>> 
>> project level announcements and discussions:
>> general
>> 
>> developer questions:
>> common-dev
>> hdfs-dev
>> mapreduce-dev
>> 
>> jira and subversion tracking for developers:
>> common-issues
>> common-commits
>> hdfs-issues
>> hdfs-commits
>> 
>> mapreduce-issues
>> mapreduce-commits
>> 
>> -- Owen
> 


From general-return-2726-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 12 21:02:15 2011
Return-Path: <general-return-2726-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45262 invoked from network); 12 Jan 2011 21:02:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2011 21:02:15 -0000
Received: (qmail 71134 invoked by uid 500); 12 Jan 2011 21:02:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70929 invoked by uid 500); 12 Jan 2011 21:02:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70921 invoked by uid 99); 12 Jan 2011 21:02:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 21:02:12 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 21:02:07 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0CL1Rsp090511
	for <general@hadoop.apache.org>; Wed, 12 Jan 2011 13:01:27 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1294866087;
	bh=oAXonzEV5TBVnTUpFjfqwa7lhDrCN8EfhUXpdGhlOZs=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=AwS+nWUrlzWOGpFeN5Zpc9UP5b0uqokm0CTlh3BuKbqpYvHFCsGEA9CeG3m77bEtY
	 LcxRqiYEEMR5oRmtAGGNWaHBrmBu2xCuagC9nchIaXbgJM7oN2Rs5L8QwDLKCpLxJR
	 /BrkR+eShKOrWadbkBpPYNLsC6825XzbjG4J2tzY=
Message-Id: <03FDF0A8-B2C3-4962-903B-36F5BE2C7FC7@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <ABEAFB4A-DE38-4122-B395-666C43C57F59@mac.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: email lists: too many or not enough?
Date: Wed, 12 Jan 2011 13:01:28 -0800
References: <A38987A0-7069-43DF-8422-7742AF8613EE@mac.com> <8C43652C-3B79-4583-B69E-C735B0084F10@yahoo-inc.com> <AANLkTine=OQYB9tuHwV6z1aXvQMN9KzSp0ztfzLAvHV5@mail.gmail.com> <C74475D4-7BF3-49F8-B910-0E53B20D49D2@mac.com> <AANLkTi=49KJ1XZ_S6Qx6QKWww92QdgSQmu+gnuGJMhGJ@mail.gmail.com> <93FCAFF3-8F8F-4BD5-8A46-60771D6CDE6C@mac.com> <4F749041-9774-4968-AE2C-52977651148A@mac.com> <DF3B1523-EB44-45C4-86E3-5B4BB6027294@apache.org> <796C0A0E-F54A-4B1A-8CD9-5C663F7330AF@Holsman.NET> <ABEAFB4A-DE38-4122-B395-666C43C57F59@mac.com>
X-Mailer: Apple Mail (2.936)

On Jan 12, 2011, at 12:14 PM, Nigel Daley wrote:

> Ya, if we keep general@ to it's intended purpose (and better  
> document/enforce the use of various lists) then there is no need for  
> a new list.  I'll make the edit's Owen suggested to the revamped  
> mailing_lists.html.
>
> I'm still confused, however, where release discussions should  
> happen.  general@ or common-dev@?  I guess I'll leave them on  
> general@ unless this is enough opposition to that.
>

+1 for general@. It seems to have the widest coverage.

Arun

> Nige
>
> On Jan 12, 2011, at 11:53 AM, Ian Holsman wrote:
>
>> to add some metrics to this discussion.
>>
>> we have ~3,500 unique email addresses subscribed to at least one  
>> hadoop mailing list.
>>
>>
>> subscribed to:
>> 1 list : 2431
>> 2 lists:  784
>> 3 lists:  238
>> 4 lists:  118
>> 5 lists:   56
>> 6 lists:   66
>> 7 lists:  102
>>
>> (I didn't look at the various 'issues or commits' mailing lists).
>>
>> So I'm not really sure adding another list will get more focused  
>> discussions, or further split up the community.
>>
>> On Jan 12, 2011, at 1:08 PM, Owen O'Malley wrote:
>>
>>>
>>> On Jan 12, 2011, at 9:47 AM, Nigel Daley wrote:
>>>
>>>>
>>>> On Jan 11, 2011, at 8:35 PM, Nigel Daley wrote:
>>>>>
>>>>> On Jan 11, 2011, at 9:38 AM, Owen O'Malley wrote:
>>>>>> We do try to move the questions off of general. It would be  
>>>>>> great if someone
>>>>>> updated the website with the intended usage of each of the lists.
>>>>>
>>>>> Ok, I'll work on a page.
>>>>
>>>> Done. See the revamped http://hadoop.apache.org/mailing_lists.html
>>>> Corrections welcome.  You must now click on general@ to see the  
>>>> subscription instructions (it's a new page).
>>>
>>> I'd suggest segregating the list into 4 tables based on audience:
>>>
>>> user questions:
>>> common-user
>>> hdfs-user
>>> mapreduce-user
>>> security - only for notifying the project of security  
>>> vulnerabilities
>>>
>>> project level announcements and discussions:
>>> general
>>>
>>> developer questions:
>>> common-dev
>>> hdfs-dev
>>> mapreduce-dev
>>>
>>> jira and subversion tracking for developers:
>>> common-issues
>>> common-commits
>>> hdfs-issues
>>> hdfs-commits
>>>
>>> mapreduce-issues
>>> mapreduce-commits
>>>
>>> -- Owen
>>
>


From general-return-2727-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 12 21:11:41 2011
Return-Path: <general-return-2727-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56886 invoked from network); 12 Jan 2011 21:11:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2011 21:11:40 -0000
Received: (qmail 83600 invoked by uid 500); 12 Jan 2011 21:11:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83491 invoked by uid 500); 12 Jan 2011 21:11:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83483 invoked by uid 99); 12 Jan 2011 21:11:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 21:11:38 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.62.96] (HELO qmta09.westchester.pa.mail.comcast.net) (76.96.62.96)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 21:11:30 +0000
Received: from omta07.westchester.pa.mail.comcast.net ([76.96.62.59])
	by qmta09.westchester.pa.mail.comcast.net with comcast
	id ufSN1f0041GhbT859lBAJE; Wed, 12 Jan 2011 21:11:10 +0000
Received: from [10.72.184.42] ([209.131.62.113])
	by omta07.westchester.pa.mail.comcast.net with comcast
	id ulB01f00m2SbwD53TlB4Dp; Wed, 12 Jan 2011 21:11:08 +0000
Message-Id: <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Date: Wed, 12 Jan 2011 13:10:59 -0800
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com> <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com> <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
X-Mailer: Apple Mail (2.936)


On Jan 11, 2011, at 9:09 PM, Arun C Murthy wrote:

> I'm open to suggestions - how about something like 20.100 to show  
> that it's a big jump? Anything else?


Although I'm not wild about any of the potential release names, this  
patch set is neither a subset or superset of the 0.21 or 0.22  
branches. Given that, I think that a new major release number makes  
the most sense. It is also relatively likely that additional minor  
releases will be made off of this branch while 0.22 is stabilizing.  
We've talked about declaring 0.20 a 1.0 for a long time and this feels  
like backing into the decision, but technically, I believe it to be  
the right name for such a release.

Thoughts?

-- Owen

From general-return-2728-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 12 21:27:10 2011
Return-Path: <general-return-2728-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60512 invoked from network); 12 Jan 2011 21:27:10 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2011 21:27:10 -0000
Received: (qmail 99745 invoked by uid 500); 12 Jan 2011 21:27:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99604 invoked by uid 500); 12 Jan 2011 21:27:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 99595 invoked by uid 99); 12 Jan 2011 21:27:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 21:27:08 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 21:27:00 +0000
Received: by qyk10 with SMTP id 10so1068792qyk.14
        for <general@hadoop.apache.org>; Wed, 12 Jan 2011 13:26:39 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=JxD+ZYWKopVEdKuyiKSOcMXNz23YgRCngR4ZXZp47oY=;
        b=ax/YSkyoMMdyCHzZPqYIgCj/eTp0wORmhY6vRglaYQDFk0pLgBEPxFcITnvIUltGk2
         tQP7exKLCP2Z7SJvXqtTeXOSptWkM1pKvMZ82v50e3kI3ssvKN85S5De4rGuFQoUG6E/
         c2HzhpLbmShR3k+V7LKAqaPPh8Eu4Z0RKnefM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=hLQC2bAYXLP/ueCHqMaTfXivMuTxxJe3Q+idZwr7NK5heUHOaibLeLdI2ZoZ89mf/v
         x5CWvhg8f0q8Nsy5HR4scOelLn0fF91wXAcrbHpkSSD/EugCJUJ7uVspUxS0gMb12s8m
         raBFDNJaeWaTqHs3XDJGmmjizqHezzWB38Yzo=
Received: by 10.224.61.11 with SMTP id r11mr1310611qah.230.1294867598492;
        Wed, 12 Jan 2011 13:26:38 -0800 (PST)
Received: from [10.180.150.23] (h-64-236-128-43.nat.aol.com [64.236.128.43])
        by mx.google.com with ESMTPS id y17sm776916qci.45.2011.01.12.13.26.37
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Wed, 12 Jan 2011 13:26:37 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
Date: Wed, 12 Jan 2011 16:26:36 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <2404862E-9910-4680-9FF9-10F305B56085@Holsman.NET>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com> <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com> <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

so if 0.20 becomes 1.0, what does 0.22 become ?

I'm still not sure if we shouldn't just add security to 0.22, and leave =
the 0.20 in maintenance mode from here on.

On Jan 12, 2011, at 4:10 PM, Owen O'Malley wrote:

>=20
> On Jan 11, 2011, at 9:09 PM, Arun C Murthy wrote:
>=20
>> I'm open to suggestions - how about something like 20.100 to show =
that it's a big jump? Anything else?
>=20
>=20
> Although I'm not wild about any of the potential release names, this =
patch set is neither a subset or superset of the 0.21 or 0.22 branches. =
Given that, I think that a new major release number makes the most =
sense. It is also relatively likely that additional minor releases will =
be made off of this branch while 0.22 is stabilizing. We've talked about =
declaring 0.20 a 1.0 for a long time and this feels like backing into =
the decision, but technically, I believe it to be the right name for =
such a release.
>=20
> Thoughts?
>=20
> -- Owen


From general-return-2729-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 12 21:35:23 2011
Return-Path: <general-return-2729-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63878 invoked from network); 12 Jan 2011 21:35:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2011 21:35:23 -0000
Received: (qmail 17262 invoked by uid 500); 12 Jan 2011 21:35:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17203 invoked by uid 500); 12 Jan 2011 21:35:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17195 invoked by uid 99); 12 Jan 2011 21:35:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 21:35:20 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 21:35:14 +0000
Received: from wlanvpn-snve-246-69.corp.yahoo.com (wlanvpn-snve-246-69.corp.yahoo.com [172.21.148.69])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p0CLYeFT013923
	for <general@hadoop.apache.org>; Wed, 12 Jan 2011 13:34:40 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1294868080;
	bh=NbdLJ5Go3uzg1qmM3LZhG2PnzQUk0yOTUaCb33pOd9w=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=OJ1VlCanwiiGhI1+ahrOjlnCbrXQ0tFSzb/BsxwH1DOTA+kyqNMpDiQ7/8JM+63AG
	 G1MtmneXmKBtShCwXZ4E8dEe7iVQ3C1Tl2FBs5PnadJopX6TcXrGi8TD1VSnA3MKpr
	 S6/H8xTCDID+irKLLdD6eNKoq4ALEJIPu70uQ6cU=
Message-Id: <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Date: Wed, 12 Jan 2011 13:34:41 -0800
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com> <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com> <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

I'm willing to discuss any and all options, for a very short period.

Technically you have a reasonable point, Doug has suggested this in  
the past too. If everyone agrees, fine; if not, I'm do not want hung  
up on a release number. I just *do not* want a controversy.

As I mentioned, I'm looking to finish this up in a couple of weeks;  
so, I could do without a long discussion on the on the critical path.

I'm happy to go with a reasonable compromise, if not, hadoop-0.20.100  
is what I'm priming for.

Heck, if Stack wants to call the append release (not sure how far  
ahead he is) as hadoop-0.20.100, I'm willing to call this  
hadoop-0.20.200.

All I care about is having a distinct release number from 0.20.2 (our  
last stable release). Again, I just want to get a release into the  
hands of our users. Please, let's resolve this quickly. Please.

Arun

On Jan 12, 2011, at 1:10 PM, Owen O'Malley wrote:

>
> On Jan 11, 2011, at 9:09 PM, Arun C Murthy wrote:
>
>> I'm open to suggestions - how about something like 20.100 to show
>> that it's a big jump? Anything else?
>
>
> Although I'm not wild about any of the potential release names, this
> patch set is neither a subset or superset of the 0.21 or 0.22
> branches. Given that, I think that a new major release number makes
> the most sense. It is also relatively likely that additional minor
> releases will be made off of this branch while 0.22 is stabilizing.
> We've talked about declaring 0.20 a 1.0 for a long time and this feels
> like backing into the decision, but technically, I believe it to be
> the right name for such a release.
>
> Thoughts?
>
> -- Owen


From general-return-2730-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 12 21:54:29 2011
Return-Path: <general-return-2730-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70204 invoked from network); 12 Jan 2011 21:54:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2011 21:54:29 -0000
Received: (qmail 65694 invoked by uid 500); 12 Jan 2011 21:54:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65631 invoked by uid 500); 12 Jan 2011 21:54:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65623 invoked by uid 99); 12 Jan 2011 21:54:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 21:54:27 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 21:54:19 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0CLrOHR090210
	for <general@hadoop.apache.org>; Wed, 12 Jan 2011 13:53:24 -0800 (PST)
Received: from lifthappen-lm.corp.yahoo.com (10.72.113.229) by
 SP2-EX07CAS01.ds.corp.yahoo.com (98.137.59.4) with Microsoft SMTP Server
 (TLS) id 8.3.106.1; Wed, 12 Jan 2011 13:53:23 -0800
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
Date: Wed, 12 Jan 2011 13:53:23 -0800
Content-Transfer-Encoding: quoted-printable
Message-ID: <9181E726-4686-4C31-AC8E-232830A0EC5A@yahoo-inc.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com> <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com> <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org> <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)

Let me second arun here.

This is incremental work on 0.20.  We're happy to support any branch =
naming strategy the community likes, but sticking with 20.<minor> seems =
like the right default approach. =20

Let's discuss 1.0 issues on another thread.  Our priority is to get our =
work into other folks hands.

Thanks!
E14

On Jan 12, 2011, at 1:34 PM, Arun C Murthy wrote:

> I'm willing to discuss any and all options, for a very short period.
>=20
> Technically you have a reasonable point, Doug has suggested this in =20=

> the past too. If everyone agrees, fine; if not, I'm do not want hung =20=

> up on a release number. I just *do not* want a controversy.
>=20
> As I mentioned, I'm looking to finish this up in a couple of weeks; =20=

> so, I could do without a long discussion on the on the critical path.
>=20
> I'm happy to go with a reasonable compromise, if not, hadoop-0.20.100 =20=

> is what I'm priming for.
>=20
> Heck, if Stack wants to call the append release (not sure how far =20
> ahead he is) as hadoop-0.20.100, I'm willing to call this =20
> hadoop-0.20.200.
>=20
> All I care about is having a distinct release number from 0.20.2 (our =20=

> last stable release). Again, I just want to get a release into the =20
> hands of our users. Please, let's resolve this quickly. Please.
>=20
> Arun
>=20
> On Jan 12, 2011, at 1:10 PM, Owen O'Malley wrote:
>=20
>>=20
>> On Jan 11, 2011, at 9:09 PM, Arun C Murthy wrote:
>>=20
>>> I'm open to suggestions - how about something like 20.100 to show
>>> that it's a big jump? Anything else?
>>=20
>>=20
>> Although I'm not wild about any of the potential release names, this
>> patch set is neither a subset or superset of the 0.21 or 0.22
>> branches. Given that, I think that a new major release number makes
>> the most sense. It is also relatively likely that additional minor
>> releases will be made off of this branch while 0.22 is stabilizing.
>> We've talked about declaring 0.20 a 1.0 for a long time and this =
feels
>> like backing into the decision, but technically, I believe it to be
>> the right name for such a release.
>>=20
>> Thoughts?
>>=20
>> -- Owen
>=20


From general-return-2731-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 12 21:58:16 2011
Return-Path: <general-return-2731-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 71304 invoked from network); 12 Jan 2011 21:58:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2011 21:58:15 -0000
Received: (qmail 69216 invoked by uid 500); 12 Jan 2011 21:58:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69136 invoked by uid 500); 12 Jan 2011 21:58:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69128 invoked by uid 99); 12 Jan 2011 21:58:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 21:58:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 12 Jan 2011 21:58:13 +0000
Received: (qmail 71242 invoked by uid 99); 12 Jan 2011 21:57:52 -0000
Received: from localhost.apache.org (HELO mail-iy0-f176.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 21:57:52 +0000
Received: by iyb26 with SMTP id 26so1100293iyb.35
        for <general@hadoop.apache.org>; Wed, 12 Jan 2011 13:57:52 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.207.7 with SMTP id fw7mr384083ibb.41.1294869472078; Wed,
 12 Jan 2011 13:57:52 -0800 (PST)
Received: by 10.231.167.199 with HTTP; Wed, 12 Jan 2011 13:57:51 -0800 (PST)
In-Reply-To: <2404862E-9910-4680-9FF9-10F305B56085@Holsman.NET>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
	<366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
	<18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
	<AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
	<F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
	<D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
	<2404862E-9910-4680-9FF9-10F305B56085@Holsman.NET>
Date: Wed, 12 Jan 2011 13:57:51 -0800
Message-ID: <AANLkTikzozRAhzYukUsGFbPCWneGtkPJo6Qpe_fB4pYh@mail.gmail.com>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I had exactly the same reaction when this came up in the past:

http://s.apache.org/l9
http://s.apache.org/5Gv

but our experience with myriad 0.20 variants has demonstrated that
Hadoop can support both a stable branch and a development branch.
Trying to direct effort away from 0.20 by preventing it from happening
in Apache didn't work, and I was wrong to advocate for it. The
interest in a more slow-moving, stable version of Hadoop will exist
whether we give it an outlet in Apache or not, most of us work on both
anyway, so we might as well collaborate in both fora. -C

On Wed, Jan 12, 2011 at 1:26 PM, Ian Holsman <hadoop@holsman.net> wrote:
> so if 0.20 becomes 1.0, what does 0.22 become ?
>
> I'm still not sure if we shouldn't just add security to 0.22, and leave t=
he 0.20 in maintenance mode from here on.
>
> On Jan 12, 2011, at 4:10 PM, Owen O'Malley wrote:
>
>>
>> On Jan 11, 2011, at 9:09 PM, Arun C Murthy wrote:
>>
>>> I'm open to suggestions - how about something like 20.100 to show that =
it's a big jump? Anything else?
>>
>>
>> Although I'm not wild about any of the potential release names, this pat=
ch set is neither a subset or superset of the 0.21 or 0.22 branches. Given =
that, I think that a new major release number makes the most sense. It is a=
lso relatively likely that additional minor releases will be made off of th=
is branch while 0.22 is stabilizing. We've talked about declaring 0.20 a 1.=
0 for a long time and this feels like backing into the decision, but techni=
cally, I believe it to be the right name for such a release.
>>
>> Thoughts?
>>
>> -- Owen
>
>

From general-return-2732-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 12 22:57:04 2011
Return-Path: <general-return-2732-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98095 invoked from network); 12 Jan 2011 22:57:04 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2011 22:57:04 -0000
Received: (qmail 91466 invoked by uid 500); 12 Jan 2011 22:57:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91396 invoked by uid 500); 12 Jan 2011 22:57:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91388 invoked by uid 99); 12 Jan 2011 22:57:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 22:57:02 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.100 as permitted sender)
Received: from [17.148.16.100] (HELO asmtpout025.mac.com) (17.148.16.100)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 22:56:54 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp025.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LEX00I3DMDINRA0@asmtp025.mac.com>
 for general@hadoop.apache.org; Wed, 12 Jan 2011 14:56:07 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-12_05:2011-01-12,2011-01-12,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101120149
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
Date: Wed, 12 Jan 2011 14:56:05 -0800
Message-id: <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
 <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
 <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
 <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
 <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
 <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
 <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

+1 for 0.20.x, where x >= 100.  I agree that the 1.0 moniker would involve more discussion.

Will this be a jumbo patch attached to a Jira and then committed to the branch?  Just curious.

Cheers,
Nige


On Jan 12, 2011, at 1:34 PM, Arun C Murthy wrote:

> I'm willing to discuss any and all options, for a very short period.
> 
> Technically you have a reasonable point, Doug has suggested this in the past too. If everyone agrees, fine; if not, I'm do not want hung up on a release number. I just *do not* want a controversy.
> 
> As I mentioned, I'm looking to finish this up in a couple of weeks; so, I could do without a long discussion on the on the critical path.
> 
> I'm happy to go with a reasonable compromise, if not, hadoop-0.20.100 is what I'm priming for.
> 
> Heck, if Stack wants to call the append release (not sure how far ahead he is) as hadoop-0.20.100, I'm willing to call this hadoop-0.20.200.
> 
> All I care about is having a distinct release number from 0.20.2 (our last stable release). Again, I just want to get a release into the hands of our users. Please, let's resolve this quickly. Please.
> 
> Arun
> 
> On Jan 12, 2011, at 1:10 PM, Owen O'Malley wrote:
> 
>> 
>> On Jan 11, 2011, at 9:09 PM, Arun C Murthy wrote:
>> 
>>> I'm open to suggestions - how about something like 20.100 to show
>>> that it's a big jump? Anything else?
>> 
>> 
>> Although I'm not wild about any of the potential release names, this
>> patch set is neither a subset or superset of the 0.21 or 0.22
>> branches. Given that, I think that a new major release number makes
>> the most sense. It is also relatively likely that additional minor
>> releases will be made off of this branch while 0.22 is stabilizing.
>> We've talked about declaring 0.20 a 1.0 for a long time and this feels
>> like backing into the decision, but technically, I believe it to be
>> the right name for such a release.
>> 
>> Thoughts?
>> 
>> -- Owen
> 


From general-return-2733-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 12 23:03:11 2011
Return-Path: <general-return-2733-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 99558 invoked from network); 12 Jan 2011 23:03:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2011 23:03:11 -0000
Received: (qmail 99703 invoked by uid 500); 12 Jan 2011 23:03:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99491 invoked by uid 500); 12 Jan 2011 23:03:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 99476 invoked by uid 99); 12 Jan 2011 23:03:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 23:03:09 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 23:03:02 +0000
Received: by qwh6 with SMTP id 6so1153533qwh.35
        for <general@hadoop.apache.org>; Wed, 12 Jan 2011 15:02:41 -0800 (PST)
MIME-Version: 1.0
Received: by 10.229.236.193 with SMTP id kl1mr1400768qcb.114.1294873361826;
 Wed, 12 Jan 2011 15:02:41 -0800 (PST)
Received: by 10.229.215.2 with HTTP; Wed, 12 Jan 2011 15:02:41 -0800 (PST)
In-Reply-To: <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
	<366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
	<18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
	<AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
	<F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
	<D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
	<2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
	<5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
Date: Wed, 12 Jan 2011 15:02:41 -0800
Message-ID: <AANLkTimbBt7StzXHS5tVd7sbZOkfiDT7qaAnKachkrkH@mail.gmail.com>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

+1 on 0.20.x   (where x is a J > 3)

Nigel - could we make all the patches in this branch that have not
been committed up stream (that need to be) blockers for 22?   This way
22 is not a regression against 0.20.x.

Thanks,
Eli

On Wed, Jan 12, 2011 at 2:56 PM, Nigel Daley <ndaley@mac.com> wrote:
> +1 for 0.20.x, where x >=3D 100. =A0I agree that the 1.0 moniker would in=
volve more discussion.
>
> Will this be a jumbo patch attached to a Jira and then committed to the b=
ranch? =A0Just curious.
>
> Cheers,
> Nige
>
>
> On Jan 12, 2011, at 1:34 PM, Arun C Murthy wrote:
>
>> I'm willing to discuss any and all options, for a very short period.
>>
>> Technically you have a reasonable point, Doug has suggested this in the =
past too. If everyone agrees, fine; if not, I'm do not want hung up on a re=
lease number. I just *do not* want a controversy.
>>
>> As I mentioned, I'm looking to finish this up in a couple of weeks; so, =
I could do without a long discussion on the on the critical path.
>>
>> I'm happy to go with a reasonable compromise, if not, hadoop-0.20.100 is=
 what I'm priming for.
>>
>> Heck, if Stack wants to call the append release (not sure how far ahead =
he is) as hadoop-0.20.100, I'm willing to call this hadoop-0.20.200.
>>
>> All I care about is having a distinct release number from 0.20.2 (our la=
st stable release). Again, I just want to get a release into the hands of o=
ur users. Please, let's resolve this quickly. Please.
>>
>> Arun
>>
>> On Jan 12, 2011, at 1:10 PM, Owen O'Malley wrote:
>>
>>>
>>> On Jan 11, 2011, at 9:09 PM, Arun C Murthy wrote:
>>>
>>>> I'm open to suggestions - how about something like 20.100 to show
>>>> that it's a big jump? Anything else?
>>>
>>>
>>> Although I'm not wild about any of the potential release names, this
>>> patch set is neither a subset or superset of the 0.21 or 0.22
>>> branches. Given that, I think that a new major release number makes
>>> the most sense. It is also relatively likely that additional minor
>>> releases will be made off of this branch while 0.22 is stabilizing.
>>> We've talked about declaring 0.20 a 1.0 for a long time and this feels
>>> like backing into the decision, but technically, I believe it to be
>>> the right name for such a release.
>>>
>>> Thoughts?
>>>
>>> -- Owen
>>
>
>

From general-return-2734-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 12 23:16:01 2011
Return-Path: <general-return-2734-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12389 invoked from network); 12 Jan 2011 23:16:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2011 23:16:00 -0000
Received: (qmail 22184 invoked by uid 500); 12 Jan 2011 23:15:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22085 invoked by uid 500); 12 Jan 2011 23:15:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22077 invoked by uid 99); 12 Jan 2011 23:15:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 23:15:58 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.99 as permitted sender)
Received: from [17.148.16.99] (HELO asmtpout024.mac.com) (17.148.16.99)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 23:15:51 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp024.mac.com (Oracle Communications Messaging Exchange Server 7u4-18.01
 64bit (built Jul 15 2010)) with ESMTPSA id <0LEX00K0UN7X3V10@asmtp024.mac.com>
 for general@hadoop.apache.org; Wed, 12 Jan 2011 15:14:22 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-12_08:2011-01-12,2011-01-12,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101120151
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <AANLkTimbBt7StzXHS5tVd7sbZOkfiDT7qaAnKachkrkH@mail.gmail.com>
Date: Wed, 12 Jan 2011 15:14:21 -0800
Message-id: <B4B91ACA-E652-4B7D-908B-0766E1E06A51@mac.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
 <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
 <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
 <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
 <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
 <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
 <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
 <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
 <AANLkTimbBt7StzXHS5tVd7sbZOkfiDT7qaAnKachkrkH@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

> Nigel - could we make all the patches in this branch that have not
> been committed up stream (that need to be) blockers for 22?   This way
> 22 is not a regression against 0.20.x.

I sure hope so.  May be difficult to untangle if it's a jumbo patch -- answer is in the details of how it's contributed. 

Cheers,
Nige


On Jan 12, 2011, at 3:02 PM, Eli Collins wrote:

> +1 on 0.20.x   (where x is a J > 3)
> 
> Nigel - could we make all the patches in this branch that have not
> been committed up stream (that need to be) blockers for 22?   This way
> 22 is not a regression against 0.20.x.
> 
> Thanks,
> Eli
> 
> On Wed, Jan 12, 2011 at 2:56 PM, Nigel Daley <ndaley@mac.com> wrote:
>> +1 for 0.20.x, where x >= 100.  I agree that the 1.0 moniker would involve more discussion.
>> 
>> Will this be a jumbo patch attached to a Jira and then committed to the branch?  Just curious.
>> 
>> Cheers,
>> Nige
>> 
>> 
>> On Jan 12, 2011, at 1:34 PM, Arun C Murthy wrote:
>> 
>>> I'm willing to discuss any and all options, for a very short period.
>>> 
>>> Technically you have a reasonable point, Doug has suggested this in the past too. If everyone agrees, fine; if not, I'm do not want hung up on a release number. I just *do not* want a controversy.
>>> 
>>> As I mentioned, I'm looking to finish this up in a couple of weeks; so, I could do without a long discussion on the on the critical path.
>>> 
>>> I'm happy to go with a reasonable compromise, if not, hadoop-0.20.100 is what I'm priming for.
>>> 
>>> Heck, if Stack wants to call the append release (not sure how far ahead he is) as hadoop-0.20.100, I'm willing to call this hadoop-0.20.200.
>>> 
>>> All I care about is having a distinct release number from 0.20.2 (our last stable release). Again, I just want to get a release into the hands of our users. Please, let's resolve this quickly. Please.
>>> 
>>> Arun
>>> 
>>> On Jan 12, 2011, at 1:10 PM, Owen O'Malley wrote:
>>> 
>>>> 
>>>> On Jan 11, 2011, at 9:09 PM, Arun C Murthy wrote:
>>>> 
>>>>> I'm open to suggestions - how about something like 20.100 to show
>>>>> that it's a big jump? Anything else?
>>>> 
>>>> 
>>>> Although I'm not wild about any of the potential release names, this
>>>> patch set is neither a subset or superset of the 0.21 or 0.22
>>>> branches. Given that, I think that a new major release number makes
>>>> the most sense. It is also relatively likely that additional minor
>>>> releases will be made off of this branch while 0.22 is stabilizing.
>>>> We've talked about declaring 0.20 a 1.0 for a long time and this feels
>>>> like backing into the decision, but technically, I believe it to be
>>>> the right name for such a release.
>>>> 
>>>> Thoughts?
>>>> 
>>>> -- Owen
>>> 
>> 
>> 


From general-return-2735-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 12 23:27:05 2011
Return-Path: <general-return-2735-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14730 invoked from network); 12 Jan 2011 23:27:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jan 2011 23:27:05 -0000
Received: (qmail 38231 invoked by uid 500); 12 Jan 2011 23:27:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38153 invoked by uid 500); 12 Jan 2011 23:27:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 38145 invoked by uid 99); 12 Jan 2011 23:27:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 23:27:03 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=MIME_QP_LONG_LINE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Jan 2011 23:26:55 +0000
Received: by qwh6 with SMTP id 6so1175427qwh.35
        for <general@hadoop.apache.org>; Wed, 12 Jan 2011 15:26:35 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:references:in-reply-to:mime-version
         :content-transfer-encoding:content-type:message-id:cc:x-mailer:from
         :subject:date:to;
        bh=SbUSq4r5heZe+OPsdQJ4LSTbWE37yeRIiPZg0jzaupE=;
        b=NR7clcqWjo3F5J0JX1tSYQQO7Tr68AYHiUSIqKnTu4WzsAtUxVk5DTImPHHZKk8dSc
         IVqkTHPk5axHJDMEgTNu/nRE9ZJDaQIcaDpmqm9sRaDZD84v037xdEKJx4aVSjCisUV7
         b2MmKC9oZ6ev4nScUiwR8PVKAgqjkSB5aV1L8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=references:in-reply-to:mime-version:content-transfer-encoding
         :content-type:message-id:cc:x-mailer:from:subject:date:to;
        b=QY3bbZi+UDXbxn6RuWvDXC06+bb30BhaFJjHofti64UxZHwrgihnjhNFXTyudWEj3f
         vGq4pAAcd9ykC14QQXVY/PAklQi+kNDFw62I6vaj5UqNjQuHzJZU9QsCU4d0tW7bRL9D
         wPtdGXxAsBy4f/2hEhJHAwsVr4mv2nEvKjqPc=
Received: by 10.224.6.140 with SMTP id 12mr1421691qaz.331.1294874794872;
        Wed, 12 Jan 2011 15:26:34 -0800 (PST)
Received: from [172.16.1.36] ([12.50.71.162])
        by mx.google.com with ESMTPS id nb15sm864308qcb.14.2011.01.12.15.26.32
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Wed, 12 Jan 2011 15:26:33 -0800 (PST)
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com> <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com> <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org> <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com> <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com> <AANLkTimbBt7StzXHS5tVd7sbZOkfiDT7qaAnKachkrkH@mail.gmail.com>
In-Reply-To: <AANLkTimbBt7StzXHS5tVd7sbZOkfiDT7qaAnKachkrkH@mail.gmail.com>
Mime-Version: 1.0 (iPhone Mail 8C148)
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii
Message-Id: <75282D37-0A3E-4F63-9C6D-C82652615F1C@holsman.net>
Cc: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: iPhone Mail (8C148)
From: Ian Holsman <hadoop@holsman.net>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Date: Wed, 12 Jan 2011 18:26:28 -0500
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

So what is the plan with 20.3 that Owen volunteered to RM?=20
Should we do that, or just integrate the security code with that and call it=
 20.x?

---
Ian Holsman - 703 879-3128

I saw the angel in the marble and carved until I set him free -- Michelangel=
o

On 12/01/2011, at 6:02 PM, Eli Collins <eli@cloudera.com> wrote:

> +1 on 0.20.x   (where x is a J > 3)
>=20
> Nigel - could we make all the patches in this branch that have not
> been committed up stream (that need to be) blockers for 22?   This way
> 22 is not a regression against 0.20.x.
>=20
> Thanks,
> Eli
>=20
> On Wed, Jan 12, 2011 at 2:56 PM, Nigel Daley <ndaley@mac.com> wrote:
>> +1 for 0.20.x, where x >=3D 100.  I agree that the 1.0 moniker would invo=
lve more discussion.
>>=20
>> Will this be a jumbo patch attached to a Jira and then committed to the b=
ranch?  Just curious.
>>=20
>> Cheers,
>> Nige
>>=20
>>=20
>> On Jan 12, 2011, at 1:34 PM, Arun C Murthy wrote:
>>=20
>>> I'm willing to discuss any and all options, for a very short period.
>>>=20
>>> Technically you have a reasonable point, Doug has suggested this in the p=
ast too. If everyone agrees, fine; if not, I'm do not want hung up on a rele=
ase number. I just *do not* want a controversy.
>>>=20
>>> As I mentioned, I'm looking to finish this up in a couple of weeks; so, I=
 could do without a long discussion on the on the critical path.
>>>=20
>>> I'm happy to go with a reasonable compromise, if not, hadoop-0.20.100 is=
 what I'm priming for.
>>>=20
>>> Heck, if Stack wants to call the append release (not sure how far ahead h=
e is) as hadoop-0.20.100, I'm willing to call this hadoop-0.20.200.
>>>=20
>>> All I care about is having a distinct release number from 0.20.2 (our la=
st stable release). Again, I just want to get a release into the hands of ou=
r users. Please, let's resolve this quickly. Please.
>>>=20
>>> Arun
>>>=20
>>> On Jan 12, 2011, at 1:10 PM, Owen O'Malley wrote:
>>>=20
>>>>=20
>>>> On Jan 11, 2011, at 9:09 PM, Arun C Murthy wrote:
>>>>=20
>>>>> I'm open to suggestions - how about something like 20.100 to show
>>>>> that it's a big jump? Anything else?
>>>>=20
>>>>=20
>>>> Although I'm not wild about any of the potential release names, this
>>>> patch set is neither a subset or superset of the 0.21 or 0.22
>>>> branches. Given that, I think that a new major release number makes
>>>> the most sense. It is also relatively likely that additional minor
>>>> releases will be made off of this branch while 0.22 is stabilizing.
>>>> We've talked about declaring 0.20 a 1.0 for a long time and this feels
>>>> like backing into the decision, but technically, I believe it to be
>>>> the right name for such a release.
>>>>=20
>>>> Thoughts?
>>>>=20
>>>> -- Owen
>>>=20
>>=20
>>=20

From general-return-2736-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 13 00:34:48 2011
Return-Path: <general-return-2736-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47973 invoked from network); 13 Jan 2011 00:34:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jan 2011 00:34:47 -0000
Received: (qmail 17191 invoked by uid 500); 13 Jan 2011 00:34:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17101 invoked by uid 500); 13 Jan 2011 00:34:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17093 invoked by uid 99); 13 Jan 2011 00:34:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 00:34:45 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.100 as permitted sender)
Received: from [17.148.16.100] (HELO asmtpout025.mac.com) (17.148.16.100)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 00:34:36 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp025.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LEX00CRVQWKRO70@asmtp025.mac.com>
 for general@hadoop.apache.org; Wed, 12 Jan 2011 16:33:57 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-12_08:2011-01-12,2011-01-12,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101120169
Subject: triggering automated precommit testing
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <5125873.329641294877388152.JavaMail.jira@thor>
Date: Wed, 12 Jan 2011 16:33:55 -0800
Message-id: <D570B102-7A4F-42E2-BB9A-B3CFC72428CC@mac.com>
References: <5125873.329641294877388152.JavaMail.jira@thor>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

> Jakob Homan commented on HDFS-884:
> ----------------------------------
> 
>> Konstantin, if you're trying to kick a new patch build for this you no longer move it to "Open" and back to "Patch Available". Instead, you must upload a new patch. Or, if you have permission, you can kickhttps://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/ and enter the issue number.
> 
> That makes me sad.  Is this a new feature or regression?

[For everyone's benefit, moving this to general@]

Jakob, I referenced the change here: http://tinyurl.com/4crxlvy
The new system is much more robust partial because it no longer relies on watching Jira generated emails to determined when issues move into Patch Available state. There is limited info I can get from the Jira API, thus the triggering mechanism had to change.

Cheers,
Nige

From general-return-2737-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 13 00:41:50 2011
Return-Path: <general-return-2737-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49106 invoked from network); 13 Jan 2011 00:41:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jan 2011 00:41:50 -0000
Received: (qmail 19449 invoked by uid 500); 13 Jan 2011 00:41:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19391 invoked by uid 500); 13 Jan 2011 00:41:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19383 invoked by uid 99); 13 Jan 2011 00:41:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 00:41:48 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of shv.hadoop@gmail.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 00:41:42 +0000
Received: by gyf1 with SMTP id 1so516984gyf.35
        for <general@hadoop.apache.org>; Wed, 12 Jan 2011 16:41:21 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=cmsgGY2sOa8pA/6dsfmWcIi/Yp/MSELvfGVLB2JWazE=;
        b=gamsb/6PhDk9dD6qHplYRUjZqrn6r/Q+MM4x/VsP1BqxROg7qcIt2KIaRG3fDbDT7K
         H/gsHQKeOtPdLJqcmqF+F8q++Oorgc+0a8gSCcC4LlQ5tJ0C8XYynqI6AiVt0jjxJDpm
         LA0tmK0QeicfJ786AECIoPZCMp4mgXdvKiGu0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=YUMOisM4PQc7rVfzkx4nD10dcoSdJfCI3/aFJmXeR2tckd2hSy7Yn7gRR0x044s61j
         rOALHP1CyvGplu48zWhk+7zDmAhwgENa7daG1koHFKoQq23nxoCxKyIQQqJ5Or87OcIl
         zBYLUk52Z3J1SOPC73Wcdn+DwUZcjpFXlxrXA=
MIME-Version: 1.0
Received: by 10.236.108.145 with SMTP id q17mr3522035yhg.70.1294879281148;
 Wed, 12 Jan 2011 16:41:21 -0800 (PST)
Received: by 10.236.105.228 with HTTP; Wed, 12 Jan 2011 16:41:21 -0800 (PST)
In-Reply-To: <D570B102-7A4F-42E2-BB9A-B3CFC72428CC@mac.com>
References: <5125873.329641294877388152.JavaMail.jira@thor>
	<D570B102-7A4F-42E2-BB9A-B3CFC72428CC@mac.com>
Date: Wed, 12 Jan 2011 16:41:21 -0800
Message-ID: <AANLkTik14AD5XC+K-5=XMp-v9bdo=vmSG-9o7x=-qbP-@mail.gmail.com>
Subject: Re: triggering automated precommit testing
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba53a53e9473610499af9312

--90e6ba53a53e9473610499af9312
Content-Type: text/plain; charset=ISO-8859-1

So, who has the permissions. I don't. Should I?
--Konstantin

On Wed, Jan 12, 2011 at 4:33 PM, Nigel Daley <ndaley@mac.com> wrote:

> > Jakob Homan commented on HDFS-884:
> > ----------------------------------
> >
> >> Konstantin, if you're trying to kick a new patch build for this you no
> longer move it to "Open" and back to "Patch Available". Instead, you must
> upload a new patch. Or, if you have permission, you can kickhttps://
> hudson.apache.org/hudson/job/PreCommit-HDFS-Build/ and enter the issue
> number.
> >
> > That makes me sad.  Is this a new feature or regression?
>
> [For everyone's benefit, moving this to general@]
>
> Jakob, I referenced the change here: http://tinyurl.com/4crxlvy
> The new system is much more robust partial because it no longer relies on
> watching Jira generated emails to determined when issues move into Patch
> Available state. There is limited info I can get from the Jira API, thus the
> triggering mechanism had to change.
>
> Cheers,
> Nige
>

--90e6ba53a53e9473610499af9312--

From general-return-2738-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 13 01:13:32 2011
Return-Path: <general-return-2738-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66770 invoked from network); 13 Jan 2011 01:13:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jan 2011 01:13:32 -0000
Received: (qmail 55305 invoked by uid 500); 13 Jan 2011 01:13:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55241 invoked by uid 500); 13 Jan 2011 01:13:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55233 invoked by uid 99); 13 Jan 2011 01:13:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 01:13:30 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.102 as permitted sender)
Received: from [17.148.16.102] (HELO asmtpout027.mac.com) (17.148.16.102)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 01:13:22 +0000
MIME-version: 1.0
Content-type: multipart/alternative;
 boundary="Boundary_(ID_VCVJlt22Mg0kt5Ik3EEI5g)"
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp027.mac.com (Oracle Communications Messaging Exchange Server 7u4-18.01
 64bit (built Jul 15 2010)) with ESMTPSA id <0LEX00BDTSPLMR90@asmtp027.mac.com>
 for general@hadoop.apache.org; Wed, 12 Jan 2011 17:12:58 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-13_01:2011-01-12,2011-01-13,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101120176
From: Nigel Daley <ndaley@mac.com>
Subject: Re: triggering automated precommit testing
Date: Wed, 12 Jan 2011 17:12:57 -0800
In-reply-to: <AANLkTik14AD5XC+K-5=XMp-v9bdo=vmSG-9o7x=-qbP-@mail.gmail.com>
To: general@hadoop.apache.org
References: <5125873.329641294877388152.JavaMail.jira@thor>
 <D570B102-7A4F-42E2-BB9A-B3CFC72428CC@mac.com>
 <AANLkTik14AD5XC+K-5=XMp-v9bdo=vmSG-9o7x=-qbP-@mail.gmail.com>
Message-id: <627F5E4D-DBDC-49BF-A94A-D30BB4FE7F9A@mac.com>
X-Mailer: Apple Mail (2.1082)

--Boundary_(ID_VCVJlt22Mg0kt5Ik3EEI5g)
Content-type: text/plain; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT

I believe ommitters can gain access following this:
http://wiki.apache.org/general/Hudson#How_do_I_get_an_account
Ian would have to run a command on people.apache.org.

Nige

On Jan 12, 2011, at 4:41 PM, Konstantin Shvachko wrote:

> So, who has the permissions. I don't. Should I?
> --Konstantin
> 
> On Wed, Jan 12, 2011 at 4:33 PM, Nigel Daley <ndaley@mac.com> wrote:
> 
>>> Jakob Homan commented on HDFS-884:
>>> ----------------------------------
>>> 
>>>> Konstantin, if you're trying to kick a new patch build for this you no
>> longer move it to "Open" and back to "Patch Available". Instead, you must
>> upload a new patch. Or, if you have permission, you can kickhttps://
>> hudson.apache.org/hudson/job/PreCommit-HDFS-Build/ and enter the issue
>> number.
>>> 
>>> That makes me sad.  Is this a new feature or regression?
>> 
>> [For everyone's benefit, moving this to general@]
>> 
>> Jakob, I referenced the change here: http://tinyurl.com/4crxlvy
>> The new system is much more robust partial because it no longer relies on
>> watching Jira generated emails to determined when issues move into Patch
>> Available state. There is limited info I can get from the Jira API, thus the
>> triggering mechanism had to change.
>> 
>> Cheers,
>> Nige
>> 


--Boundary_(ID_VCVJlt22Mg0kt5Ik3EEI5g)--

From general-return-2739-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 13 01:19:28 2011
Return-Path: <general-return-2739-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68145 invoked from network); 13 Jan 2011 01:19:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jan 2011 01:19:27 -0000
Received: (qmail 59954 invoked by uid 500); 13 Jan 2011 01:19:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59871 invoked by uid 500); 13 Jan 2011 01:19:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59863 invoked by uid 99); 13 Jan 2011 01:19:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 01:19:25 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 01:19:18 +0000
Received: by qyk10 with SMTP id 10so1245427qyk.14
        for <general@hadoop.apache.org>; Wed, 12 Jan 2011 17:18:56 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=XITzQnAqmOl9mZATozosLXxl725BGrRHFt3KAfftXhI=;
        b=UANdBZh1yWqiLymw4XN8TDWUWNgNYyUcw9tvegef0dHMgZkFQaVycR4FT12TLFeUgj
         Mcplh14kLpFEqRk1q9XbAnGTYpeR1xQbq0Tynk5u5zgfgJcIRK/9YVJiiaaISw6pm06d
         SAvlaexFsDAD8Qp5sXU1AhXHmSvcqStoGpn8Y=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=U3XfgmYdoZajuJBdPbOXJ95LmsFIppilzXi6VFU5QIOaG1jbjGkSlfhpOocJ1N8ck6
         vF3eXwY97LQyht7SU5RX8njGJ1r/g59kQJIAQUnjkO9KIiPG1rPUOBKKv/NI5TyRLZQ2
         j7HKxipTiztp3LrX7F6rrwjhJte8T2bVm2m30=
Received: by 10.224.29.20 with SMTP id o20mr1495523qac.377.1294881536345;
        Wed, 12 Jan 2011 17:18:56 -0800 (PST)
Received: from 12-50-71-176.att-inc.com ([12.50.71.176])
        by mx.google.com with ESMTPS id nb15sm940074qcb.2.2011.01.12.17.18.50
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Wed, 12 Jan 2011 17:18:55 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: triggering automated precommit testing
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <627F5E4D-DBDC-49BF-A94A-D30BB4FE7F9A@mac.com>
Date: Wed, 12 Jan 2011 20:18:39 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <8A0FD462-3DEB-498C-9713-9024583983EF@Holsman.NET>
References: <5125873.329641294877388152.JavaMail.jira@thor> <D570B102-7A4F-42E2-BB9A-B3CFC72428CC@mac.com> <AANLkTik14AD5XC+K-5=XMp-v9bdo=vmSG-9o7x=-qbP-@mail.gmail.com> <627F5E4D-DBDC-49BF-A94A-D30BB4FE7F9A@mac.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

and done.
anybody else want access?

On Jan 12, 2011, at 8:12 PM, Nigel Daley wrote:

> I believe ommitters can gain access following this:
> http://wiki.apache.org/general/Hudson#How_do_I_get_an_account
> Ian would have to run a command on people.apache.org.
>=20
> Nige
>=20
> On Jan 12, 2011, at 4:41 PM, Konstantin Shvachko wrote:
>=20
>> So, who has the permissions. I don't. Should I?
>> --Konstantin
>>=20
>> On Wed, Jan 12, 2011 at 4:33 PM, Nigel Daley <ndaley@mac.com> wrote:
>>=20
>>>> Jakob Homan commented on HDFS-884:
>>>> ----------------------------------
>>>>=20
>>>>> Konstantin, if you're trying to kick a new patch build for this =
you no
>>> longer move it to "Open" and back to "Patch Available". Instead, you =
must
>>> upload a new patch. Or, if you have permission, you can kickhttps://
>>> hudson.apache.org/hudson/job/PreCommit-HDFS-Build/ and enter the =
issue
>>> number.
>>>>=20
>>>> That makes me sad.  Is this a new feature or regression?
>>>=20
>>> [For everyone's benefit, moving this to general@]
>>>=20
>>> Jakob, I referenced the change here: http://tinyurl.com/4crxlvy
>>> The new system is much more robust partial because it no longer =
relies on
>>> watching Jira generated emails to determined when issues move into =
Patch
>>> Available state. There is limited info I can get from the Jira API, =
thus the
>>> triggering mechanism had to change.
>>>=20
>>> Cheers,
>>> Nige
>>>=20
>=20


From general-return-2740-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 13 01:26:10 2011
Return-Path: <general-return-2740-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69633 invoked from network); 13 Jan 2011 01:26:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jan 2011 01:26:09 -0000
Received: (qmail 69733 invoked by uid 500); 13 Jan 2011 01:26:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69688 invoked by uid 500); 13 Jan 2011 01:26:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69680 invoked by uid 99); 13 Jan 2011 01:26:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 01:26:07 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jghoman@gmail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 01:26:01 +0000
Received: by iwn2 with SMTP id 2so1195787iwn.35
        for <general@hadoop.apache.org>; Wed, 12 Jan 2011 17:25:40 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type:content-transfer-encoding;
        bh=iRotsibnHiS0frglpRnYymh1uY5FVdEltsUtQpCrlpA=;
        b=prBueZrzouU68jv1AvaIjE56CnJZFxgAwomf13xILN/cb6877zwExdfN+MHtNn4bhr
         qwWaq12nK/RNfsVWRqBmzHpD/yTuUwDFGgSFputJx76Hnu9y5t4P2Iow6Zny6hmCt4fI
         sA9QipH/2Zr4BygSjfUYkDVnXBuR1tuoaGxtQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=hbd/AtetvOd3XxOSOWMWtEg1hh5LnXBI6JfhtJWY2w8dm66qYF35I8KANUdhcnXpkI
         nenFD+Cql9Gq8jqsmGY9sP+aqkMhkeuqDoOlVhRd/t7NQQSP0WRaB8SdbRDrHhJoO8Y0
         u0eE5HwAnwMzo5kBnTR+prdHFPjxX+u85MXqQ=
Received: by 10.231.36.12 with SMTP id r12mr1808884ibd.69.1294881940220; Wed,
 12 Jan 2011 17:25:40 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.25.163 with HTTP; Wed, 12 Jan 2011 17:25:20 -0800 (PST)
In-Reply-To: <8A0FD462-3DEB-498C-9713-9024583983EF@Holsman.NET>
References: <5125873.329641294877388152.JavaMail.jira@thor>
 <D570B102-7A4F-42E2-BB9A-B3CFC72428CC@mac.com> <AANLkTik14AD5XC+K-5=XMp-v9bdo=vmSG-9o7x=-qbP-@mail.gmail.com>
 <627F5E4D-DBDC-49BF-A94A-D30BB4FE7F9A@mac.com> <8A0FD462-3DEB-498C-9713-9024583983EF@Holsman.NET>
From: Jakob Homan <jghoman@gmail.com>
Date: Wed, 12 Jan 2011 17:25:20 -0800
Message-ID: <AANLkTin69JjPz2JSCVxAdMCR1WswDt0cHYHXsY4nXGZr@mail.gmail.com>
Subject: Re: triggering automated precommit testing
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

yes, please.

On Wed, Jan 12, 2011 at 5:18 PM, Ian Holsman <hadoop@holsman.net> wrote:
> and done.
> anybody else want access?
>
> On Jan 12, 2011, at 8:12 PM, Nigel Daley wrote:
>
>> I believe ommitters can gain access following this:
>> http://wiki.apache.org/general/Hudson#How_do_I_get_an_account
>> Ian would have to run a command on people.apache.org.
>>
>> Nige
>>
>> On Jan 12, 2011, at 4:41 PM, Konstantin Shvachko wrote:
>>
>>> So, who has the permissions. I don't. Should I?
>>> --Konstantin
>>>
>>> On Wed, Jan 12, 2011 at 4:33 PM, Nigel Daley <ndaley@mac.com> wrote:
>>>
>>>>> Jakob Homan commented on HDFS-884:
>>>>> ----------------------------------
>>>>>
>>>>>> Konstantin, if you're trying to kick a new patch build for this you =
no
>>>> longer move it to "Open" and back to "Patch Available". Instead, you m=
ust
>>>> upload a new patch. Or, if you have permission, you can kickhttps://
>>>> hudson.apache.org/hudson/job/PreCommit-HDFS-Build/ and enter the issue
>>>> number.
>>>>>
>>>>> That makes me sad. =A0Is this a new feature or regression?
>>>>
>>>> [For everyone's benefit, moving this to general@]
>>>>
>>>> Jakob, I referenced the change here: http://tinyurl.com/4crxlvy
>>>> The new system is much more robust partial because it no longer relies=
 on
>>>> watching Jira generated emails to determined when issues move into Pat=
ch
>>>> Available state. There is limited info I can get from the Jira API, th=
us the
>>>> triggering mechanism had to change.
>>>>
>>>> Cheers,
>>>> Nige
>>>>
>>
>
>

From general-return-2741-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 13 01:44:00 2011
Return-Path: <general-return-2741-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72801 invoked from network); 13 Jan 2011 01:44:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jan 2011 01:44:00 -0000
Received: (qmail 83201 invoked by uid 500); 13 Jan 2011 01:43:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83144 invoked by uid 500); 13 Jan 2011 01:43:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83136 invoked by uid 99); 13 Jan 2011 01:43:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 01:43:58 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.99 as permitted sender)
Received: from [17.148.16.99] (HELO asmtpout024.mac.com) (17.148.16.99)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 01:43:49 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp024.mac.com (Oracle Communications Messaging Exchange Server 7u4-18.01
 64bit (built Jul 15 2010)) with ESMTPSA id <0LEX00MZ8U4EZV60@asmtp024.mac.com>
 for general@hadoop.apache.org; Wed, 12 Jan 2011 17:43:27 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-13_01:2011-01-12,2011-01-13,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101120180
Subject: Re: email lists: too many or not enough?
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <DF3B1523-EB44-45C4-86E3-5B4BB6027294@apache.org>
Date: Wed, 12 Jan 2011 17:43:26 -0800
Message-id: <00A700DF-3485-4289-AC11-EDDBD5EC8769@mac.com>
References: <A38987A0-7069-43DF-8422-7742AF8613EE@mac.com>
 <8C43652C-3B79-4583-B69E-C735B0084F10@yahoo-inc.com>
 <AANLkTine=OQYB9tuHwV6z1aXvQMN9KzSp0ztfzLAvHV5@mail.gmail.com>
 <C74475D4-7BF3-49F8-B910-0E53B20D49D2@mac.com>
 <AANLkTi=49KJ1XZ_S6Qx6QKWww92QdgSQmu+gnuGJMhGJ@mail.gmail.com>
 <93FCAFF3-8F8F-4BD5-8A46-60771D6CDE6C@mac.com>
 <4F749041-9774-4968-AE2C-52977651148A@mac.com>
 <DF3B1523-EB44-45C4-86E3-5B4BB6027294@apache.org>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

Updated w/ Owen's suggestions.  Feel free to tweak further.

Cheers,
Nige

On Jan 12, 2011, at 10:08 AM, Owen O'Malley wrote:

> 
> On Jan 12, 2011, at 9:47 AM, Nigel Daley wrote:
> 
>> 
>> On Jan 11, 2011, at 8:35 PM, Nigel Daley wrote:
>>> 
>>> On Jan 11, 2011, at 9:38 AM, Owen O'Malley wrote:
>>>> We do try to move the questions off of general. It would be great if someone
>>>> updated the website with the intended usage of each of the lists.
>>> 
>>> Ok, I'll work on a page.
>> 
>> Done. See the revamped http://hadoop.apache.org/mailing_lists.html
>> Corrections welcome.  You must now click on general@ to see the subscription instructions (it's a new page).
> 
> I'd suggest segregating the list into 4 tables based on audience:
> 
> user questions:
> common-user
> hdfs-user
> mapreduce-user
> security - only for notifying the project of security vulnerabilities
> 
> project level announcements and discussions:
> general
> 
> developer questions:
> common-dev
> hdfs-dev
> mapreduce-dev
> 
> jira and subversion tracking for developers:
> common-issues
> common-commits
> hdfs-issues
> hdfs-commits
> 
> mapreduce-issues
> mapreduce-commits
> 
> -- Owen


From general-return-2742-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 13 02:30:03 2011
Return-Path: <general-return-2742-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91834 invoked from network); 13 Jan 2011 02:30:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jan 2011 02:30:02 -0000
Received: (qmail 32776 invoked by uid 500); 13 Jan 2011 02:30:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32655 invoked by uid 500); 13 Jan 2011 02:30:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32647 invoked by uid 99); 13 Jan 2011 02:30:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 02:30:00 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 02:29:55 +0000
Received: by wye20 with SMTP id 20so1305557wye.35
        for <general@hadoop.apache.org>; Wed, 12 Jan 2011 18:29:34 -0800 (PST)
Received: by 10.216.163.203 with SMTP id a53mr1411182wel.104.1294885772861;
 Wed, 12 Jan 2011 18:29:32 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.59.13 with HTTP; Wed, 12 Jan 2011 18:29:12 -0800 (PST)
In-Reply-To: <8A0FD462-3DEB-498C-9713-9024583983EF@Holsman.NET>
References: <5125873.329641294877388152.JavaMail.jira@thor>
 <D570B102-7A4F-42E2-BB9A-B3CFC72428CC@mac.com> <AANLkTik14AD5XC+K-5=XMp-v9bdo=vmSG-9o7x=-qbP-@mail.gmail.com>
 <627F5E4D-DBDC-49BF-A94A-D30BB4FE7F9A@mac.com> <8A0FD462-3DEB-498C-9713-9024583983EF@Holsman.NET>
From: Konstantin Boudnik <cos@apache.org>
Date: Wed, 12 Jan 2011 18:29:12 -0800
X-Google-Sender-Auth: Ma7lQhafuKs2AbQylaXdxLNwTXE
Message-ID: <AANLkTinkrZ6Fbm8ytgVz=KPsfGJM4KdHzLtO+_cXp43T@mail.gmail.com>
Subject: Re: triggering automated precommit testing
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Me too, please.
--
=A0 Take care,
Konstantin (Cos) Boudnik



On Wed, Jan 12, 2011 at 17:18, Ian Holsman <hadoop@holsman.net> wrote:
> and done.
> anybody else want access?
>
> On Jan 12, 2011, at 8:12 PM, Nigel Daley wrote:
>
>> I believe ommitters can gain access following this:
>> http://wiki.apache.org/general/Hudson#How_do_I_get_an_account
>> Ian would have to run a command on people.apache.org.
>>
>> Nige
>>
>> On Jan 12, 2011, at 4:41 PM, Konstantin Shvachko wrote:
>>
>>> So, who has the permissions. I don't. Should I?
>>> --Konstantin
>>>
>>> On Wed, Jan 12, 2011 at 4:33 PM, Nigel Daley <ndaley@mac.com> wrote:
>>>
>>>>> Jakob Homan commented on HDFS-884:
>>>>> ----------------------------------
>>>>>
>>>>>> Konstantin, if you're trying to kick a new patch build for this you =
no
>>>> longer move it to "Open" and back to "Patch Available". Instead, you m=
ust
>>>> upload a new patch. Or, if you have permission, you can kickhttps://
>>>> hudson.apache.org/hudson/job/PreCommit-HDFS-Build/ and enter the issue
>>>> number.
>>>>>
>>>>> That makes me sad. =A0Is this a new feature or regression?
>>>>
>>>> [For everyone's benefit, moving this to general@]
>>>>
>>>> Jakob, I referenced the change here: http://tinyurl.com/4crxlvy
>>>> The new system is much more robust partial because it no longer relies=
 on
>>>> watching Jira generated emails to determined when issues move into Pat=
ch
>>>> Available state. There is limited info I can get from the Jira API, th=
us the
>>>> triggering mechanism had to change.
>>>>
>>>> Cheers,
>>>> Nige
>>>>
>>
>
>

From general-return-2743-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 13 02:34:41 2011
Return-Path: <general-return-2743-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93003 invoked from network); 13 Jan 2011 02:34:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jan 2011 02:34:41 -0000
Received: (qmail 39400 invoked by uid 500); 13 Jan 2011 02:34:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39182 invoked by uid 500); 13 Jan 2011 02:34:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39174 invoked by uid 99); 13 Jan 2011 02:34:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 02:34:38 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.103 as permitted sender)
Received: from [17.148.16.103] (HELO asmtpout028.mac.com) (17.148.16.103)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 02:34:30 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp028.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LEX006E6WGDXUG2@asmtp028.mac.com>
 for general@hadoop.apache.org; Wed, 12 Jan 2011 18:33:50 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-13_01:2011-01-12,2011-01-13,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101120182
From: Nigel Daley <ndaley@mac.com>
Subject: [0.22] Removed all non-blockers/non-critical from 0.22 in Jira
Date: Wed, 12 Jan 2011 18:33:49 -0800
Message-id: <4A527256-DEB3-4405-9F3C-A682110FF861@mac.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

Folks,

I have updated Jira to remove "Fix In" 0.22 from all non-blockers and non-critical issues across common, hdfs, and mapreduce.  If you're wondering what used to be marked for 0.22, I put them in this spreadsheet for your reference (there is one sheet for each of the projects):
https://spreadsheets.google.com/ccc?key=0AjXiUl01876EdDM5YlNHRWtDbUFxS2tYSnd5SFZGSnc&hl=en

***If you now mark an issue for 0.22, please set it's priority as accurately as possible (preferably blocker or critical only) and provide a rationale on why it needs to be in 0.22.***

Our next goal is to fix blockers so we can post a release candidate.

Cheers,
Nige


From general-return-2744-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 13 06:11:17 2011
Return-Path: <general-return-2744-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 89020 invoked from network); 13 Jan 2011 06:11:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jan 2011 06:11:17 -0000
Received: (qmail 68251 invoked by uid 500); 13 Jan 2011 06:11:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67937 invoked by uid 500); 13 Jan 2011 06:11:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67850 invoked by uid 99); 13 Jan 2011 06:11:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 06:11:10 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 06:11:04 +0000
Received: by vws18 with SMTP id 18so496199vws.35
        for <general@hadoop.apache.org>; Wed, 12 Jan 2011 22:10:43 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=RYmQNL600ZZApJcvjB8Cq/r20pTVldzrl5whR8FYHkw=;
        b=BErVMn6Y/cCkdV3ZkZcfO90VBaZ1LZ1x12RewZQGZpH+WjFCP+XEhXN5otyjcFFMQL
         /Kps+N3DuCr+taKZuMaZ41WOD2su5lWC2ExwYX0FkyrrkzfgAP8zzk2SyqWdKTzzkfWq
         CM0xdjMyOZI34J0sA2WJEFPOBSstJ0FDN9R8U=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=UthaRS5L0d+utaU65tG6IewikYCldVp8WC3dn5FO8ZmTkNiWZt5+wZc46fat8t0/ui
         8M9L3eIW0g8arkUBpnh4j5v2akAV7R5WqLFuadYTpNp9ATDm2l3ys6DUNmTtNYiqI/Qr
         BwZg4kz26WYwDKacCWuXSLez2bqgYBUa3GDUM=
Received: by 10.220.193.198 with SMTP id dv6mr489626vcb.84.1294899041637;
        Wed, 12 Jan 2011 22:10:41 -0800 (PST)
Received: from 12-50-71-176.att-inc.com ([12.50.71.176])
        by mx.google.com with ESMTPS id d14sm353804vcx.23.2011.01.12.22.10.40
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Wed, 12 Jan 2011 22:10:40 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: triggering automated precommit testing
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <AANLkTinkrZ6Fbm8ytgVz=KPsfGJM4KdHzLtO+_cXp43T@mail.gmail.com>
Date: Thu, 13 Jan 2011 01:10:36 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <10406582-12B3-4774-90EE-3032D686B32B@Holsman.NET>
References: <5125873.329641294877388152.JavaMail.jira@thor> <D570B102-7A4F-42E2-BB9A-B3CFC72428CC@mac.com> <AANLkTik14AD5XC+K-5=XMp-v9bdo=vmSG-9o7x=-qbP-@mail.gmail.com> <627F5E4D-DBDC-49BF-A94A-D30BB4FE7F9A@mac.com> <8A0FD462-3DEB-498C-9713-9024583983EF@Holsman.NET> <AANLkTinkrZ6Fbm8ytgVz=KPsfGJM4KdHzLtO+_cXp43T@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

done.. (for both cos & jghoman).

regards
Ian
On Jan 12, 2011, at 9:29 PM, Konstantin Boudnik wrote:

> Me too, please.
> --
>   Take care,
> Konstantin (Cos) Boudnik
>=20
>=20
>=20
> On Wed, Jan 12, 2011 at 17:18, Ian Holsman <hadoop@holsman.net> wrote:
>> and done.
>> anybody else want access?
>>=20
>> On Jan 12, 2011, at 8:12 PM, Nigel Daley wrote:
>>=20
>>> I believe ommitters can gain access following this:
>>> http://wiki.apache.org/general/Hudson#How_do_I_get_an_account
>>> Ian would have to run a command on people.apache.org.
>>>=20
>>> Nige
>>>=20
>>> On Jan 12, 2011, at 4:41 PM, Konstantin Shvachko wrote:
>>>=20
>>>> So, who has the permissions. I don't. Should I?
>>>> --Konstantin
>>>>=20
>>>> On Wed, Jan 12, 2011 at 4:33 PM, Nigel Daley <ndaley@mac.com> =
wrote:
>>>>=20
>>>>>> Jakob Homan commented on HDFS-884:
>>>>>> ----------------------------------
>>>>>>=20
>>>>>>> Konstantin, if you're trying to kick a new patch build for this =
you no
>>>>> longer move it to "Open" and back to "Patch Available". Instead, =
you must
>>>>> upload a new patch. Or, if you have permission, you can =
kickhttps://
>>>>> hudson.apache.org/hudson/job/PreCommit-HDFS-Build/ and enter the =
issue
>>>>> number.
>>>>>>=20
>>>>>> That makes me sad.  Is this a new feature or regression?
>>>>>=20
>>>>> [For everyone's benefit, moving this to general@]
>>>>>=20
>>>>> Jakob, I referenced the change here: http://tinyurl.com/4crxlvy
>>>>> The new system is much more robust partial because it no longer =
relies on
>>>>> watching Jira generated emails to determined when issues move into =
Patch
>>>>> Available state. There is limited info I can get from the Jira =
API, thus the
>>>>> triggering mechanism had to change.
>>>>>=20
>>>>> Cheers,
>>>>> Nige
>>>>>=20
>>>=20
>>=20
>>=20


From general-return-2745-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 13 07:08:17 2011
Return-Path: <general-return-2745-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27983 invoked from network); 13 Jan 2011 07:08:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jan 2011 07:08:17 -0000
Received: (qmail 48360 invoked by uid 500); 13 Jan 2011 07:08:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47981 invoked by uid 500); 13 Jan 2011 07:08:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47973 invoked by uid 99); 13 Jan 2011 07:08:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 07:08:10 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 07:08:05 +0000
Received: from [192.168.1.71] (snvvpn2-10-73-152-c177.hq.corp.yahoo.com [10.73.152.177])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p0D77VfK075376
	for <general@hadoop.apache.org>; Wed, 12 Jan 2011 23:07:31 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1294902451;
	bh=6bVVYX/0ursYimDHNVw2av16wTBdV/qtSqc1IloP8hM=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:Mime-Version:Subject:
	 Date:References;
	b=Q1Q31LQNIxU+pKb5XiaIB+Na4vazDnnj5GCbZvEPBEsMS/jy+AjB5LF46TCoOsUCF
	 rpFUs03fh5uVYorVecUX+m84XoEQkqs/h7tC256WrKnEpggolljzKLht0dLllP/slG
	 yuljo2ugxh15+gEfAKCAISRUWnJ9DXFEGJXsSWt8=
Message-Id: <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
Content-Type: multipart/alternative; boundary=Apple-Mail-61-915155131
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Date: Wed, 12 Jan 2011 23:07:31 -0800
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com> <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com> <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org> <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com> <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
X-Mailer: Apple Mail (2.936)

--Apple-Mail-61-915155131
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


On Jan 12, 2011, at 2:56 PM, Nigel Daley wrote:

> +1 for 0.20.x, where x >= 100.  I agree that the 1.0 moniker would  
> involve more discussion.

Ok, seems like we are converging; we can continue talking. I've  
created the branch to get the ball rolling.

> Will this be a jumbo patch attached to a Jira and then committed to  
> the branch?  Just curious.

I'm afraid that the svn log of the branch from github Y! branch is  
fairly useless since a single JIRA might have multiple commits in the  
Y! branch (bugfix on top of a bugfix). We have done that in several  
cases (but the patches committed to trunk have a single patch which is  
the result of forward porting a complete feature/bugfix). IAC the this  
branch and 0.22 have diverged so much that almost no non-trivial patch  
would apply without a significant amount of work.

Thus, I think a jumbo patch should suffice. It will also ensure this  
can done quickly so that the community can then concentrate on 0.22  
and beyond.

However, I will (manually) ensure all relevant jiras are referenced in  
the CHANGES.txt and Release Notes for folks to see the contents of the  
release. This is the hardest part of the exercise. Also, this ensures  
that we can track these jiras for 0.22 as Eli suggested.

Does that seem like a reasonable way forward? I'm happy to brainstorm.

thanks,
Arun


--Apple-Mail-61-915155131--

From general-return-2746-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 13 07:10:23 2011
Return-Path: <general-return-2746-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 28475 invoked from network); 13 Jan 2011 07:10:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jan 2011 07:10:22 -0000
Received: (qmail 51063 invoked by uid 500); 13 Jan 2011 07:10:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50980 invoked by uid 500); 13 Jan 2011 07:10:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50966 invoked by uid 99); 13 Jan 2011 07:10:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 07:10:16 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.103 as permitted sender)
Received: from [17.148.16.103] (HELO asmtpout028.mac.com) (17.148.16.103)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 07:10:08 +0000
MIME-version: 1.0
Content-type: multipart/alternative;
 boundary="Boundary_(ID_XSzzMLC15lA3s4HuMDtA9g)"
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp028.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LEY008VT98A9CW2@asmtp028.mac.com>
 for general@hadoop.apache.org; Wed, 12 Jan 2011 23:09:48 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-13_04:2011-01-13,2011-01-13,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101120213
From: Nigel Daley <ndaley@mac.com>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Date: Wed, 12 Jan 2011 23:09:46 -0800
In-reply-to: <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
To: general@hadoop.apache.org
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
 <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
 <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
 <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
 <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
 <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
 <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
 <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
 <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
Message-id: <80C17BAC-A8EA-43A8-8243-87D7BBBB2578@mac.com>
X-Mailer: Apple Mail (2.1082)

--Boundary_(ID_XSzzMLC15lA3s4HuMDtA9g)
Content-type: text/plain; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT


On Jan 12, 2011, at 11:07 PM, Arun C Murthy wrote:

> 
> On Jan 12, 2011, at 2:56 PM, Nigel Daley wrote:
> 
>> +1 for 0.20.x, where x >= 100.  I agree that the 1.0 moniker would involve more discussion.
> 
> Ok, seems like we are converging; we can continue talking. I've created the branch to get the ball rolling.
> 
>> Will this be a jumbo patch attached to a Jira and then committed to the branch?  Just curious.
> 
> I'm afraid that the svn log of the branch from github Y! branch is fairly useless since a single JIRA might have multiple commits in the Y! branch (bugfix on top of a bugfix). We have done that in several cases (but the patches committed to trunk have a single patch which is the result of forward porting a complete feature/bugfix). IAC the this branch and 0.22 have diverged so much that almost no non-trivial patch would apply without a significant amount of work.
> 
> Thus, I think a jumbo patch should suffice. It will also ensure this can done quickly so that the community can then concentrate on 0.22 and beyond.
> 
> However, I will (manually) ensure all relevant jiras are referenced in the CHANGES.txt and Release Notes for folks to see the contents of the release. This is the hardest part of the exercise. Also, this ensures that we can track these jiras for 0.22 as Eli suggested.
> 
> Does that seem like a reasonable way forward? I'm happy to brainstorm.

+1.  If it turns out to be insufficient to figure out how to apply similar changes to trunk/0.22 then we can address that as needed.

Thanks Arun!

Nige


--Boundary_(ID_XSzzMLC15lA3s4HuMDtA9g)--

From general-return-2747-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 13 08:52:58 2011
Return-Path: <general-return-2747-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 64399 invoked from network); 13 Jan 2011 08:52:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jan 2011 08:52:57 -0000
Received: (qmail 60964 invoked by uid 500); 13 Jan 2011 08:52:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60560 invoked by uid 500); 13 Jan 2011 08:52:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60545 invoked by uid 99); 13 Jan 2011 08:52:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 08:52:51 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 08:52:43 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p0D8q4bd005828
	for <general@hadoop.apache.org>; Thu, 13 Jan 2011 00:52:04 -0800 (PST)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by
 SP2-EX07CAS01.ds.corp.yahoo.com ([98.137.59.37]) with mapi; Thu, 13 Jan 2011
 00:52:04 -0800
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 13 Jan 2011 00:51:57 -0800
Subject: Re: email lists: too many or not enough?
Thread-Topic: email lists: too many or not enough?
Thread-Index: Acuy/yajSdjEpkZsRJqh6S/Wbg6cUw==
Message-ID: <D93D9E03-7141-4900-9E76-3B2DDB94931C@yahoo-inc.com>
References: <A38987A0-7069-43DF-8422-7742AF8613EE@mac.com>
 <8C43652C-3B79-4583-B69E-C735B0084F10@yahoo-inc.com>
 <AANLkTine=OQYB9tuHwV6z1aXvQMN9KzSp0ztfzLAvHV5@mail.gmail.com>
 <C74475D4-7BF3-49F8-B910-0E53B20D49D2@mac.com>
 <AANLkTi=49KJ1XZ_S6Qx6QKWww92QdgSQmu+gnuGJMhGJ@mail.gmail.com>
 <93FCAFF3-8F8F-4BD5-8A46-60771D6CDE6C@mac.com>
 <4F749041-9774-4968-AE2C-52977651148A@mac.com>
 <DF3B1523-EB44-45C4-86E3-5B4BB6027294@apache.org>
 <00A700DF-3485-4289-AC11-EDDBD5EC8769@mac.com>
In-Reply-To: <00A700DF-3485-4289-AC11-EDDBD5EC8769@mac.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Much improved. Thanks!

PS please keep branch discussions here.=20

---
E14 - via iPhone

On Jan 12, 2011, at 5:44 PM, "Nigel Daley" <ndaley@mac.com> wrote:

> Updated w/ Owen's suggestions.  Feel free to tweak further.
>=20
> Cheers,
> Nige
>=20
> On Jan 12, 2011, at 10:08 AM, Owen O'Malley wrote:
>=20
>>=20
>> On Jan 12, 2011, at 9:47 AM, Nigel Daley wrote:
>>=20
>>>=20
>>> On Jan 11, 2011, at 8:35 PM, Nigel Daley wrote:
>>>>=20
>>>> On Jan 11, 2011, at 9:38 AM, Owen O'Malley wrote:
>>>>> We do try to move the questions off of general. It would be great if =
someone
>>>>> updated the website with the intended usage of each of the lists.
>>>>=20
>>>> Ok, I'll work on a page.
>>>=20
>>> Done. See the revamped http://hadoop.apache.org/mailing_lists.html
>>> Corrections welcome.  You must now click on general@ to see the subscri=
ption instructions (it's a new page).
>>=20
>> I'd suggest segregating the list into 4 tables based on audience:
>>=20
>> user questions:
>> common-user
>> hdfs-user
>> mapreduce-user
>> security - only for notifying the project of security vulnerabilities
>>=20
>> project level announcements and discussions:
>> general
>>=20
>> developer questions:
>> common-dev
>> hdfs-dev
>> mapreduce-dev
>>=20
>> jira and subversion tracking for developers:
>> common-issues
>> common-commits
>> hdfs-issues
>> hdfs-commits
>>=20
>> mapreduce-issues
>> mapreduce-commits
>>=20
>> -- Owen
>=20

From general-return-2748-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 13 20:23:25 2011
Return-Path: <general-return-2748-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 90295 invoked from network); 13 Jan 2011 20:23:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jan 2011 20:23:24 -0000
Received: (qmail 39388 invoked by uid 500); 13 Jan 2011 20:23:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39313 invoked by uid 500); 13 Jan 2011 20:23:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39300 invoked by uid 99); 13 Jan 2011 20:23:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 20:23:22 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 20:23:16 +0000
Received: from SP1-EX07CAS02.ds.corp.yahoo.com (sp1-ex07cas02.ds.corp.yahoo.com [216.252.116.138])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0DKM1Hx071751
	for <general@hadoop.apache.org>; Thu, 13 Jan 2011 12:22:01 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1294950122;
	bh=8Tw370nkFcFcRybSIPkmzUUdmNGzr+KXEqHljqC25A4=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 MIME-Version;
	b=jRG0my3RRSHihiQoHiZNxizMs8xiw06UmAmEDBwsvA6BJbTAzpIOo+ZmKY6PHhu5r
	 FL9mIKMKhC4mKBGGtyuxle1Ebo9Zfi5nAjDGCk8CSAI4ZUT3/Rj2wycSbUlI5oznWF
	 FaIor+pUEE6TNDMTlNHM3tQRCO5Ne8yzcZXfqsOw=
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 SP1-EX07CAS02.ds.corp.yahoo.com ([216.252.116.167]) with mapi; Thu, 13 Jan
 2011 12:22:00 -0800
From: Suresh Srinivas <sureshms@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 13 Jan 2011 12:21:58 -0800
Subject: Re: triggering automated precommit testing
Thread-Topic: triggering automated precommit testing
Thread-Index: Acuyv/HwLK2LT3iyQBSQT0pewK0EAAAn5Urb
Message-ID: <C9549EE6.2E6A1%sureshms@yahoo-inc.com>
In-Reply-To: <8A0FD462-3DEB-498C-9713-9024583983EF@Holsman.NET>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C9549EE62E6A1sureshmsyahooinccom_"
MIME-Version: 1.0

--_000_C9549EE62E6A1sureshmsyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Please add me...


On 1/12/11 5:18 PM, "Ian Holsman" <hadoop@holsman.net> wrote:

and done.
anybody else want access?

On Jan 12, 2011, at 8:12 PM, Nigel Daley wrote:

> I believe ommitters can gain access following this:
> http://wiki.apache.org/general/Hudson#How_do_I_get_an_account
> Ian would have to run a command on people.apache.org.
>
> Nige
>
> On Jan 12, 2011, at 4:41 PM, Konstantin Shvachko wrote:
>
>> So, who has the permissions. I don't. Should I?
>> --Konstantin
>>
>> On Wed, Jan 12, 2011 at 4:33 PM, Nigel Daley <ndaley@mac.com> wrote:
>>
>>>> Jakob Homan commented on HDFS-884:
>>>> ----------------------------------
>>>>
>>>>> Konstantin, if you're trying to kick a new patch build for this you n=
o
>>> longer move it to "Open" and back to "Patch Available". Instead, you mu=
st
>>> upload a new patch. Or, if you have permission, you can kickhttps://
>>> hudson.apache.org/hudson/job/PreCommit-HDFS-Build/ and enter the issue
>>> number.
>>>>
>>>> That makes me sad.  Is this a new feature or regression?
>>>
>>> [For everyone's benefit, moving this to general@]
>>>
>>> Jakob, I referenced the change here: http://tinyurl.com/4crxlvy
>>> The new system is much more robust partial because it no longer relies =
on
>>> watching Jira generated emails to determined when issues move into Patc=
h
>>> Available state. There is limited info I can get from the Jira API, thu=
s the
>>> triggering mechanism had to change.
>>>
>>> Cheers,
>>> Nige
>>>
>



--_000_C9549EE62E6A1sureshmsyahooinccom_--

From general-return-2749-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 13 22:05:26 2011
Return-Path: <general-return-2749-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59223 invoked from network); 13 Jan 2011 22:05:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jan 2011 22:05:26 -0000
Received: (qmail 74236 invoked by uid 500); 13 Jan 2011 22:05:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73890 invoked by uid 500); 13 Jan 2011 22:05:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73876 invoked by uid 99); 13 Jan 2011 22:05:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 22:05:23 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 22:05:16 +0000
Received: by iwn2 with SMTP id 2so2192408iwn.35
        for <general@hadoop.apache.org>; Thu, 13 Jan 2011 14:04:55 -0800 (PST)
Received: by 10.231.199.196 with SMTP id et4mr2976799ibb.71.1294956295444;
 Thu, 13 Jan 2011 14:04:55 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.115.8 with HTTP; Thu, 13 Jan 2011 14:04:35 -0800 (PST)
In-Reply-To: <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
 <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
 <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
 <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
 <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com> <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
 <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Thu, 13 Jan 2011 14:04:35 -0800
Message-ID: <AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba53a4f4fd5fd40499c181e1
X-Virus-Checked: Checked by ClamAV on apache.org

--90e6ba53a4f4fd5fd40499c181e1
Content-Type: text/plain; charset=ISO-8859-1

Hi Arun, all,

When we merged YDH and CDH for CDH3b3, we went through the effort of
"linearizing" all of the YDH patches and squashing multiple commits into
single ones corresponding to a single JIRA where possible. So, we have a
100% linear set of patches that applies on top of the 0.20.2 source tree and
includes Yahoo 0.20.100.3 as well as almost all the patches from 0.20-append
and a number of other backports.

Since this could be applied as a linear set of patches instead of a big
lump, would there be interest in using this as the 0.20.>100 Apache release?
I can take the time to remove any patches that are cloudera specific or not
yet applied upstream.

Thanks
-Todd


On Wed, Jan 12, 2011 at 11:07 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:

>
> On Jan 12, 2011, at 2:56 PM, Nigel Daley wrote:
>
>  +1 for 0.20.x, where x >= 100.  I agree that the 1.0 moniker would involve
>> more discussion.
>>
>
> Ok, seems like we are converging; we can continue talking. I've created the
> branch to get the ball rolling.
>
>
>  Will this be a jumbo patch attached to a Jira and then committed to the
>> branch?  Just curious.
>>
>
> I'm afraid that the svn log of the branch from github Y! branch is fairly
> useless since a single JIRA might have multiple commits in the Y! branch
> (bugfix on top of a bugfix). We have done that in several cases (but the
> patches committed to trunk have a single patch which is the result of
> forward porting a complete feature/bugfix). IAC the this branch and 0.22
> have diverged so much that almost no non-trivial patch would apply without a
> significant amount of work.
>
> Thus, I think a jumbo patch should suffice. It will also ensure this can
> done quickly so that the community can then concentrate on 0.22 and beyond.
>
> However, I will (manually) ensure all relevant jiras are referenced in the
> CHANGES.txt and Release Notes for folks to see the contents of the release.
> This is the hardest part of the exercise. Also, this ensures that we can
> track these jiras for 0.22 as Eli suggested.
>
> Does that seem like a reasonable way forward? I'm happy to brainstorm.
>
> thanks,
> Arun
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

--90e6ba53a4f4fd5fd40499c181e1--

From general-return-2750-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 13 23:05:57 2011
Return-Path: <general-return-2750-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9181 invoked from network); 13 Jan 2011 23:05:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jan 2011 23:05:57 -0000
Received: (qmail 75304 invoked by uid 500); 13 Jan 2011 23:05:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75113 invoked by uid 500); 13 Jan 2011 23:05:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75105 invoked by uid 99); 13 Jan 2011 23:05:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 23:05:55 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 23:05:48 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0DN54XN047861
	for <general@hadoop.apache.org>; Thu, 13 Jan 2011 15:05:04 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1294959904;
	bh=+OqBxBUo8AXRb6bYaiWKPynUlYjHIKajMM2+KfBwNrs=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=WODVXOlU/jyTZ+myXoQWAdVOd4RzpS14aS7Fo96c9NzqPDbEoYCEuTkA8wlwn2j/Q
	 BbN+Srme2tkONzEA6d1GiywHVaY1XkfxhFYU2BSmQUw3UayaVzod4eT1SjPrP1W4CU
	 ZIRWjj6jtAgjkRC7Sh6gj8K2/+Z5a8K48MAFo3wk=
Message-Id: <388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Date: Thu, 13 Jan 2011 15:05:04 -0800
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com> <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com> <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org> <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com> <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com> <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com> <AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

Todd,


On Jan 13, 2011, at 2:04 PM, Todd Lipcon wrote:

> Hi Arun, all,
>
> When we merged YDH and CDH for CDH3b3, we went through the effort of
> "linearizing" all of the YDH patches and squashing multiple commits  
> into
> single ones corresponding to a single JIRA where possible. So, we  
> have a
> 100% linear set of patches that applies on top of the 0.20.2 source  
> tree and
> includes Yahoo 0.20.100.3 as well as almost all the patches from  
> 0.20-append
> and a number of other backports.
>
> Since this could be applied as a linear set of patches instead of a  
> big
> lump, would there be interest in using this as the 0.20.>100 Apache  
> release?
> I can take the time to remove any patches that are cloudera specific  
> or not
> yet applied upstream.
>

Interesting discussion, thanks.

I'm sure it took you a fair amount of work to squash patches (which I  
tried too, btw). That, plus the fact that we would need to do a  
similar amount of work for the 10 or so releases we have done after  
0.20.100.3 scares me.

As we Nigel and I discussed here, the jumbo  patch and an up-to-date  
CHANGES.txt provides almost all of the benefits we seek and allows all  
of us to get this done very quickly to focus on hadoop-0.22 and beyond.

What do you think?

OTOH, I could get this release done and start squashing patches for  
the sake of completeness as a background activity.

Thoughts?

thanks,
Arun

From general-return-2751-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 13 23:39:12 2011
Return-Path: <general-return-2751-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37834 invoked from network); 13 Jan 2011 23:39:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jan 2011 23:39:12 -0000
Received: (qmail 22102 invoked by uid 500); 13 Jan 2011 23:39:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22041 invoked by uid 500); 13 Jan 2011 23:39:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22033 invoked by uid 99); 13 Jan 2011 23:39:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 23:39:10 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 23:39:05 +0000
Received: by iwn2 with SMTP id 2so2260721iwn.35
        for <general@hadoop.apache.org>; Thu, 13 Jan 2011 15:38:44 -0800 (PST)
Received: by 10.231.11.4 with SMTP id r4mr8358ibr.146.1294961924309; Thu, 13
 Jan 2011 15:38:44 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.115.8 with HTTP; Thu, 13 Jan 2011 15:34:41 -0800 (PST)
In-Reply-To: <388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
 <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
 <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
 <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
 <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com> <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
 <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com> <AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com>
 <388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Thu, 13 Jan 2011 15:34:41 -0800
Message-ID: <AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00221532ca287f036f0499c2d19e

--00221532ca287f036f0499c2d19e
Content-Type: text/plain; charset=ISO-8859-1

On Thu, Jan 13, 2011 at 3:05 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:

> Since this could be applied as a linear set of patches instead of a big
>> lump, would there be interest in using this as the 0.20.>100 Apache
>> release?
>> I can take the time to remove any patches that are cloudera specific or
>> not
>> yet applied upstream.
>>
>>
> Interesting discussion, thanks.
>
> I'm sure it took you a fair amount of work to squash patches (which I tried
> too, btw).


Yep, I had a great summer ;-)


> That, plus the fact that we would need to do a similar amount of work for
> the 10 or so releases we have done after 0.20.100.3 scares me.
>

Sorry, I actually meant 0.20.104.3. Have there been many releases since
then? That's the last version available on the Yahoo github, and that's the
version we incorporated/linearized.

If there is a large sequence of patches after this that you're planning on
including, it would be good to see them in your git repo.



> As we Nigel and I discussed here, the jumbo  patch and an up-to-date
> CHANGES.txt provides almost all of the benefits we seek and allows all of us
> to get this done very quickly to focus on hadoop-0.22 and beyond.
>
>
In my opinion here are the downsides to this plan:

- a mondo "merge" patch is a big pain when trying to do debugging. It may be
sufficient for a user to look at CHANGES.txt, but I find myself using
blame/log/etc on individual files to understand code lineage on a daily
basis. If all of the merge shows up as a big patch it will be very difficult
(at least the way I work with code) to help users debug issues or understand
which JIRA a certain regression may have come from.

- CHANGES.txt traditionally doesn't reference which patch file from a JIRA
was checked in. So we may know that a given JIRA has been included, but
often there are several revisions of patches on the JIRA and it's difficult
to be sure that we have the most up-to-date version. By looking at change
history it's usually easy to pick this out, but if it's one giant patch
apply, this isn't possible.

- the proposal to use the YDH distro certainly solves the Security issue,
but doesn't help out HBase at all. Given HBase has been asking for a long
time to get a real release of the append branch, I think it would be better
to have one 20-based release which has both of these features, rather than
further fragmenting the community into 0.20.2, 0.20.2+security,
0.20.2+append.

I think the first two points could be addressed if you push your git tree
either to github or an apache-hosted git, and then include in SVN as a mondo
patch. It's not ideal, but at least when trying to debug issues and
understand the history of this branch there will be a publicly available
change history to reference.

To clarify my position a bit here - I definitely appreciate your
volunteering to do the work, and wouldn't *block* the proposal as you've put
it forth. I just think it will have limited utility for the community by
being opaque (if contributed as a giant patch) and by not including the sync
feature which is critical for a large segment of users. Given those
downsides I'd rather see the effort diverted towards making a killer 0.22
release that we can all jump on.

Thanks
-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

--00221532ca287f036f0499c2d19e--

From general-return-2752-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 00:04:06 2011
Return-Path: <general-return-2752-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46737 invoked from network); 14 Jan 2011 00:04:06 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 00:04:06 -0000
Received: (qmail 48155 invoked by uid 500); 14 Jan 2011 00:04:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48086 invoked by uid 500); 14 Jan 2011 00:04:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48078 invoked by uid 99); 14 Jan 2011 00:04:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 00:04:04 +0000
X-ASF-Spam-Status: No, hits=-0.1 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of chenqibj@cn.ibm.com designates 202.81.31.142 as permitted sender)
Received: from [202.81.31.142] (HELO e23smtp09.au.ibm.com) (202.81.31.142)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 00:03:53 +0000
Received: from d23relay03.au.ibm.com (d23relay03.au.ibm.com [202.81.31.245])
	by e23smtp09.au.ibm.com (8.14.4/8.13.1) with ESMTP id p0E03Vmj004362
	for <general@hadoop.apache.org>; Fri, 14 Jan 2011 11:03:31 +1100
Received: from d23av03.au.ibm.com (d23av03.au.ibm.com [9.190.234.97])
	by d23relay03.au.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id p0E03V912134148
	for <general@hadoop.apache.org>; Fri, 14 Jan 2011 11:03:31 +1100
Received: from d23av03.au.ibm.com (loopback [127.0.0.1])
	by d23av03.au.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id p0E03UU0022653
	for <general@hadoop.apache.org>; Fri, 14 Jan 2011 11:03:30 +1100
Received: from d23m0037.cn.ibm.com (d23m0037.cn.ibm.com [9.181.2.105])
	by d23av03.au.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id p0E03Unr022611
	for <general@hadoop.apache.org>; Fri, 14 Jan 2011 11:03:30 +1100
Subject: Keith is on vacation
Auto-Submitted: auto-generated
From: Qi BJ Chen <chenqibj@cn.ibm.com>
To: general@hadoop.apache.org
Message-ID: <OF6BF2AB60.86AE6D1A-ON48257818.000051EF-48257818.000051EF@cn.ibm.com>
Date: Fri, 14 Jan 2011 08:03:29 +0800
X-MIMETrack: Serialize by Router on D23M0037/23/M/IBM(Release 8.5.1FP4|July 25, 2010) at
 01/14/2011 08:03:30
MIME-Version: 1.0
Content-type: multipart/alternative; 
	Boundary="0__=C7BBF28BDF93D77F8f9e8a93df938690918cC7BBF28BDF93D77F"
Content-Disposition: inline
X-Virus-Checked: Checked by ClamAV on apache.org

--0__=C7BBF28BDF93D77F8f9e8a93df938690918cC7BBF28BDF93D77F
Content-type: text/plain; charset=US-ASCII


I will be out of the office starting  2011-01-14 and will not return until
2011-01-22.

During this time, I won't access my email.
--0__=C7BBF28BDF93D77F8f9e8a93df938690918cC7BBF28BDF93D77F--


From general-return-2753-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 00:59:13 2011
Return-Path: <general-return-2753-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87699 invoked from network); 14 Jan 2011 00:59:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 00:59:13 -0000
Received: (qmail 1500 invoked by uid 500); 14 Jan 2011 00:59:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1450 invoked by uid 500); 14 Jan 2011 00:59:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1442 invoked by uid 99); 14 Jan 2011 00:59:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 00:59:10 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 00:59:06 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0E0wYHp056778
	for <general@hadoop.apache.org>; Thu, 13 Jan 2011 16:58:34 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1294966714;
	bh=X3W5ZfMJn2uaSRa3ZOYXciLVaHEF2fpvxoIQMASuhWU=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=k5I7vpOMUW2ZYYU/mLvR2EYZR0ERQIRpUI9GwP1nQB1OPqoaxsU/cMsnImMiRXuBB
	 MC2u+Kdu/GqufJoFAoQzazm1ZvwsCu/ydXx85QZx6ncXHmz8IJgGLPZGBu/Ud57kHW
	 v30wCSdjD/PBssY64nsqOYjiqMs6q8M3JXxiT9JQ=
Message-Id: <04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Date: Thu, 13 Jan 2011 16:58:34 -0800
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com> <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com> <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org> <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com> <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com> <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com> <AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com> <388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com> <AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com>
X-Mailer: Apple Mail (2.936)


On Jan 13, 2011, at 3:34 PM, Todd Lipcon wrote:

> On Thu, Jan 13, 2011 at 3:05 PM, Arun C Murthy <acm@yahoo-inc.com>  
> wrote:
>
>> Since this could be applied as a linear set of patches instead of a  
>> big
>>> lump, would there be interest in using this as the 0.20.>100 Apache
>>> release?
>>> I can take the time to remove any patches that are cloudera  
>>> specific or
>>> not
>>> yet applied upstream.
>>>
>>>
>> Interesting discussion, thanks.
>>
>> I'm sure it took you a fair amount of work to squash patches (which  
>> I tried
>> too, btw).
>
>
> Yep, I had a great summer ;-)
>
>
>> That, plus the fact that we would need to do a similar amount of  
>> work for
>> the 10 or so releases we have done after 0.20.100.3 scares me.
>>
>
> Sorry, I actually meant 0.20.104.3. Have there been many releases  
> since
> then? That's the last version available on the Yahoo github, and  
> that's the
> version we incorporated/linearized.

Yep. I had a great summer! ;-)

>>
>> As we Nigel and I discussed here, the jumbo  patch and an up-to-date
>> CHANGES.txt provides almost all of the benefits we seek and allows  
>> all of us
>> to get this done very quickly to focus on hadoop-0.22 and beyond.
>>
>>
> In my opinion here are the downsides to this plan:
>

I agree there are downsides, I think I did point them out at the  
outset! :)

> - a mondo "merge" patch is a big pain when trying to do debugging.  
> It may be
> sufficient for a user to look at CHANGES.txt, but I find myself using
> blame/log/etc on individual files to understand code lineage on a  
> daily
> basis. If all of the merge shows up as a big patch it will be very  
> difficult
> (at least the way I work with code) to help users debug issues or  
> understand
> which JIRA a certain regression may have come from.
>

Right, no question. Which is why I offered to do this as a background  
activity right after... this ensures that the source of truth is  
*always* a branch in Apache subversion.

I feel that we could get a usable release out of door quickly for our  
users. Also, please remember that almost every patch we have committed  
is available on relevant jiras. I understand the devs have a problem  
and I feel we can bear with it for a little while. Again, I agree this  
isn't an ideal solution, I'm just trying to expedite the release for  
the users.

>
> To clarify my position a bit here - I definitely appreciate your
> volunteering to do the work, and wouldn't *block* the proposal as  
> you've put
> it forth. I just think it will have limited utility for the  
> community by
> being opaque (if contributed as a giant patch) and by not including  
> the sync
> feature which is critical for a large segment of users. Given those
> downsides I'd rather see the effort diverted towards making a killer  
> 0.22
> release that we can all jump on.
>

Thanks for understanding.

Again, I completely agree this isn't an ideal situation, but I do hope  
it has a bit more than *limited utility* for our end-users. Who knows,  
I maybe hopelessly deluded! *smile*

Also, I'm trying to do exactly what you suggested - spend very little  
time on this so that everyone, including me, can focus on the future.

thanks,
Arun



From general-return-2754-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 01:35:42 2011
Return-Path: <general-return-2754-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 7274 invoked from network); 14 Jan 2011 01:35:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 01:35:42 -0000
Received: (qmail 36618 invoked by uid 500); 14 Jan 2011 01:35:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36557 invoked by uid 500); 14 Jan 2011 01:35:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36549 invoked by uid 99); 14 Jan 2011 01:35:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 01:35:40 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 01:35:35 +0000
Received: by vws18 with SMTP id 18so846559vws.35
        for <general@hadoop.apache.org>; Thu, 13 Jan 2011 17:35:14 -0800 (PST)
MIME-Version: 1.0
Received: by 10.229.236.14 with SMTP id ki14mr159778qcb.120.1294968913763;
 Thu, 13 Jan 2011 17:35:13 -0800 (PST)
Received: by 10.229.215.2 with HTTP; Thu, 13 Jan 2011 17:35:13 -0800 (PST)
In-Reply-To: <04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
	<366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
	<18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
	<AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
	<F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
	<D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
	<2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
	<5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
	<075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
	<AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com>
	<388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com>
	<AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com>
	<04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com>
Date: Thu, 13 Jan 2011 17:35:13 -0800
Message-ID: <AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Eli Collins <eli@cloudera.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Thursday, January 13, 2011, Arun C Murthy <acm@yahoo-inc.com> wrote:
>
> On Jan 13, 2011, at 3:34 PM, Todd Lipcon wrote:
>
>
> On Thu, Jan 13, 2011 at 3:05 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:
>
>
> Since this could be applied as a linear set of patches instead of a big
>
> lump, would there be interest in using this as the 0.20.>100 Apache
> release?
> I can take the time to remove any patches that are cloudera specific or
> not
> yet applied upstream.
>
>
>
> Interesting discussion, thanks.
>
> I'm sure it took you a fair amount of work to squash patches (which I tri=
ed
> too, btw).
>
>
>
> Yep, I had a great summer ;-)
>
>
>
> That, plus the fact that we would need to do a similar amount of work for
> the 10 or so releases we have done after 0.20.100.3 scares me.
>
>
>
> Sorry, I actually meant 0.20.104.3. Have there been many releases since
> then? That's the last version available on the Yahoo github, and that's t=
he
> version we incorporated/linearized.
>
>
> Yep. I had a great summer! ;-)
>
>
>
> As we Nigel and I discussed here, the jumbo =A0patch and an up-to-date
> CHANGES.txt provides almost all of the benefits we seek and allows all of=
 us
> to get this done very quickly to focus on hadoop-0.22 and beyond.
>
>
>
> In my opinion here are the downsides to this plan:
>
>
>
> I agree there are downsides, I think I did point them out at the outset! =
:)
>
>
> - a mondo "merge" patch is a big pain when trying to do debugging. It may=
 be
> sufficient for a user to look at CHANGES.txt, but I find myself using
> blame/log/etc on individual files to understand code lineage on a daily
> basis. If all of the merge shows up as a big patch it will be very diffic=
ult
> (at least the way I work with code) to help users debug issues or underst=
and
> which JIRA a certain regression may have come from.
>
>
>
> Right, no question. Which is why I offered to do this as a background act=
ivity right after... this ensures that the source of truth is *always* a br=
anch in Apache subversion.
>
> I feel that we could get a usable release out of door quickly for our use=
rs. Also, please remember that almost every patch we have committed is avai=
lable on relevant jiras. I understand the devs have a problem and I feel we=
 can bear with it for a little while. Again, I agree this isn't an ideal so=
lution, I'm just trying to expedite the release for the users.
>
>
>
> To clarify my position a bit here - I definitely appreciate your
> volunteering to do the work, and wouldn't *block* the proposal as you've =
put
> it forth. I just think it will have limited utility for the community by
> being opaque (if contributed as a giant patch) and by not including the s=
ync
> feature which is critical for a large segment of users. Given those
> downsides I'd rather see the effort diverted towards making a killer 0.22
> release that we can all jump on.
>
>
>
> Thanks for understanding.
>
> Again, I completely agree this isn't an ideal situation, but I do hope it=
 has a bit more than *limited utility* for our end-users. Who knows, I mayb=
e hopelessly deluded! *smile*
>
> Also, I'm trying to do exactly what you suggested - spend very little tim=
e on this so that everyone, including me, can focus on the future.
>
> thanks,
> Arun
>

Given that Todd has already done the work to rebase the 0.20.104.3
patch set on 0.20.2, and in a way that doesn't require one big change,
and his patch set includes branch20-append which the HBase guys want
an Apache release of wouldn't it make sense to go this route?  What do
others think? Seems better to have one 0.20.100 release than multiple
ones for security and append.

Thanks,
Eli

From general-return-2755-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 02:13:20 2011
Return-Path: <general-return-2755-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23356 invoked from network); 14 Jan 2011 02:13:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 02:13:20 -0000
Received: (qmail 68181 invoked by uid 500); 14 Jan 2011 02:13:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68129 invoked by uid 500); 14 Jan 2011 02:13:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68121 invoked by uid 99); 14 Jan 2011 02:13:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 02:13:18 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 02:13:13 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p0E2CW8L018122
	for <general@hadoop.apache.org>; Thu, 13 Jan 2011 18:12:32 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1294971152;
	bh=NUaSkJ32IHp7gG5Z1vequBMQ7gUYiJOqjrmeyisD+eA=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=qGFwJc/+W+5h8nd9zI9Zmr1YvrtreVQ9qOoQTEKgAst4qGN9J/8AFJVkK5KE4eU0Y
	 OoK/O5dgoD21N1idRcFei6SPUgGPnroKR2hWR010xOTPXQpIP3kVB/YfR4Xu/omoOo
	 pNQ+7RdIrP3+7ypwPkbSWtrB2ta4zH0/ELDjlatM=
Message-Id: <74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Date: Thu, 13 Jan 2011 18:12:32 -0800
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com> <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com> <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org> <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com> <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com> <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com> <AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com> <388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com> <AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com> <04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com> <AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com>
X-Mailer: Apple Mail (2.936)


On Jan 13, 2011, at 5:35 PM, Eli Collins wrote:
> Given that Todd has already done the work to rebase the 0.20.104.3
> patch set on 0.20.2, and in a way that doesn't require one big change,
> and his patch set includes branch20-append which the HBase guys want
> an Apache release of wouldn't it make sense to go this route?  What do
> others think? Seems better to have one 0.20.100 release than multiple
> ones for security and append.


My concern around 0.20.104.3 is that it has serious security holes  
including a root exploit that we have since fixed. I'm sure you guys  
are aware of them, Todd has helped to fix some.

The version I'm offering to push to the community has fixed all of  
them, *plus* the added benefit of several stability and performance  
fixes we have done since 20.104.3, almost 10 internal releases. This  
is a battle tested and hardened version which we have deployed on  
40,000+ nodes. It is a significant upgrade on 0.20.104.3 which we  
never deployed. I'm pretty sure *some* users will find that valuable. ;)

Also, I've offered to push individual patches as a background activity  
on a branch - that should suffice, no? Or, do you consider this a  
blocker?

Again, my goal in this exercise is to get a stable, improved version  
of Hadoop into the hands of our users asap, and focus on 0.22 and  
beyond.

thanks,
Arun

From general-return-2756-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 02:50:47 2011
Return-Path: <general-return-2756-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33483 invoked from network); 14 Jan 2011 02:50:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 02:50:46 -0000
Received: (qmail 92184 invoked by uid 500); 14 Jan 2011 02:50:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91968 invoked by uid 500); 14 Jan 2011 02:50:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91959 invoked by uid 99); 14 Jan 2011 02:50:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 02:50:43 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 02:50:37 +0000
Received: by qyk7 with SMTP id 7so5706945qyk.14
        for <general@hadoop.apache.org>; Thu, 13 Jan 2011 18:50:15 -0800 (PST)
MIME-Version: 1.0
Received: by 10.229.247.68 with SMTP id mb4mr171361qcb.294.1294973415842; Thu,
 13 Jan 2011 18:50:15 -0800 (PST)
Received: by 10.229.215.2 with HTTP; Thu, 13 Jan 2011 18:50:15 -0800 (PST)
In-Reply-To: <74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
	<366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
	<18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
	<AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
	<F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
	<D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
	<2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
	<5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
	<075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
	<AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com>
	<388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com>
	<AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com>
	<04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com>
	<AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com>
	<74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com>
Date: Thu, 13 Jan 2011 18:50:15 -0800
Message-ID: <AANLkTimLLrfYbaaOzbEV9dcsaMFLeMmdgx73FasdxGhA@mail.gmail.com>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Thu, Jan 13, 2011 at 6:12 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:
>
> On Jan 13, 2011, at 5:35 PM, Eli Collins wrote:
>>
>> Given that Todd has already done the work to rebase the 0.20.104.3
>> patch set on 0.20.2, and in a way that doesn't require one big change,
>> and his patch set includes branch20-append which the HBase guys want
>> an Apache release of wouldn't it make sense to go this route? =A0What do
>> others think? Seems better to have one 0.20.100 release than multiple
>> ones for security and append.
>
>
> My concern around 0.20.104.3 is that it has serious security holes includ=
ing
> a root exploit that we have since fixed. I'm sure you guys are aware of
> them, Todd has helped to fix some.
>

The cdh3 patch set Todd is talking about is not vanilla 104.3, it's
104.3 re-based onto 20.2 plus patches from branch-20 and trunk (the
performance and stability fixes I think you're referring to, at least
the ones that have been posted to Apache jira).

Can you post a pointer to the version you're referring to, eg on
github?  If there isn't a big delta between it and the cdh3 patch set
(which should have the 20-based patches from jira) perhaps you and
Todd could easily merge in the delta to create 0.20.x?

> The version I'm offering to push to the community has fixed all of them,
> *plus* the added benefit of several stability and performance fixes we ha=
ve
> done since 20.104.3, almost 10 internal releases. This is a battle tested
> and hardened version which we have deployed on 40,000+ nodes. It is a
> significant upgrade on 0.20.104.3 which we never deployed. I'm pretty sur=
e
> *some* users will find that valuable. ;)

Definitely, but better to hit two birds with one stone right?  Instead
of a security + enhancements release and an append release we could
have a single security + append + enhancements release and users don't
have to choose.

> Also, I've offered to push individual patches as a background activity on=
 a
> branch - that should suffice, no? Or, do you consider this a blocker?

Definitely not a blocker.

> Again, my goal in this exercise is to get a stable, improved version of
> Hadoop into the hands of our users asap, and focus on 0.22 and beyond.

Agree, that's everyone's goal.  My point is that a release that's
already been re-based on 20.2, doesn't require a separate HBase
release, and doesn't require you spend time on a background task to
break up the big change into smaller ones seems like a faster way
forward.

Thanks,
Eli

From general-return-2757-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 04:23:48 2011
Return-Path: <general-return-2757-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77528 invoked from network); 14 Jan 2011 04:23:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 04:23:48 -0000
Received: (qmail 55123 invoked by uid 500); 14 Jan 2011 04:23:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54816 invoked by uid 500); 14 Jan 2011 04:23:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54808 invoked by uid 99); 14 Jan 2011 04:23:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 04:23:42 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 04:23:37 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p0E4N3ro051002
	for <general@hadoop.apache.org>; Thu, 13 Jan 2011 20:23:03 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1294978983;
	bh=26qvGzxe6kC5zqrGRZ+CBRwxAG0cM/6y0RDoXqvDx90=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:Mime-Version:Subject:
	 Date:References;
	b=HIGVTy8CtMlQWlZMnnRtwaSTxdV8olpfkrzNaOv2lXgIvTEEJelfS2hu5UaTX8FEQ
	 uFRpJQawWAKR3Mrm8ydHGJRatPqsSXRaE/1BI1MhPsJERn66Tw2pRYKDEpVsLnBLhL
	 yFApcH2IkTyD/M7GQRJyT4LzeRxcXpXZimScOzdQ=
Message-Id: <B4C8E6B2-A343-449B-AE63-DE5E3F883ABA@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <AANLkTimLLrfYbaaOzbEV9dcsaMFLeMmdgx73FasdxGhA@mail.gmail.com>
Content-Type: multipart/alternative; boundary=Apple-Mail-78-991687224
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Date: Thu, 13 Jan 2011 20:23:03 -0800
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com> <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com> <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org> <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com> <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com> <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com> <AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com> <388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com> <AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com> <04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com> <AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com> <74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com> <AANLkTimLLrfYbaaOzbEV9dcsaMFLeMmdgx73FasdxGhA@mail.gmail.com>
X-Mailer: Apple Mail (2.936)

--Apple-Mail-78-991687224
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


On Jan 13, 2011, at 6:50 PM, Eli Collins wrote:

> The cdh3 patch set Todd is talking about is not vanilla 104.3, it's
> 104.3 re-based onto 20.2 plus patches from branch-20 and trunk (the
> performance and stability fixes I think you're referring to, at least
> the ones that have been posted to Apache jira).
>
> Can you post a pointer to the version you're referring to, eg on
> github?  If there isn't a big delta between it and the cdh3 patch set
> (which should have the 20-based patches from jira) perhaps you and
> Todd could easily merge in the delta to create 0.20.x?
>

I can guarantee it will need work to merge the enhancements since  
20.104.3, it's over 6 months of development. The enhancements includes  
work on stability such as iterative ls, limits on JT to prevent single  
jobs/users from taking it down etc. and lots of bug-fixes to security.  
So, unfortunately the delta is pretty large.

I'm working on a CHANGES.txt which should reflect all the changes i.e.  
bug-fixes and enhancements.

>> The version I'm offering to push to the community has fixed all of  
>> them,
>> *plus* the added benefit of several stability and performance fixes  
>> we have
>> done since 20.104.3, almost 10 internal releases. This is a battle  
>> tested
>> and hardened version which we have deployed on 40,000+ nodes. It is a
>> significant upgrade on 0.20.104.3 which we never deployed. I'm  
>> pretty sure
>> *some* users will find that valuable. ;)
>
> Definitely, but better to hit two birds with one stone right?  Instead
> of a security + enhancements release and an append release we could
> have a single security + append + enhancements release and users don't
> have to choose.
>


We are discussing two options:
20 + security + enhancements
20 + security + append

I think the value we provide via 20+security+enhancements release is  
that it's stable, tested and deployed at scale. Doing any more work  
merging 6 months of work at Yahoo (again, I guarantee it's a lot of  
work) will need a lots of cycles to validate, test and stabilize.

I feel the alternative is a distraction for me, I'd rather work on 0.22.

I can get 20+security+enhancements done very, very, quickly precisely  
because I don't have to spend cycles testing it.

Does that make sense? Thanks for being patient and bearing with me...

Arun


--Apple-Mail-78-991687224--

From general-return-2758-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 04:59:29 2011
Return-Path: <general-return-2758-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84156 invoked from network); 14 Jan 2011 04:59:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 04:59:28 -0000
Received: (qmail 72482 invoked by uid 500); 14 Jan 2011 04:59:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72107 invoked by uid 500); 14 Jan 2011 04:59:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72099 invoked by uid 99); 14 Jan 2011 04:59:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 04:59:22 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [98.139.53.206] (HELO nm4-vm0.bullet.mail.ac4.yahoo.com) (98.139.53.206)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 14 Jan 2011 04:59:12 +0000
Received: from [98.139.52.197] by nm4.bullet.mail.ac4.yahoo.com with NNFMP; 14 Jan 2011 04:58:50 -0000
Received: from [98.139.52.143] by tm10.bullet.mail.ac4.yahoo.com with NNFMP; 14 Jan 2011 04:58:50 -0000
Received: from [127.0.0.1] by omp1026.mail.ac4.yahoo.com with NNFMP; 14 Jan 2011 04:58:50 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 671344.32362.bm@omp1026.mail.ac4.yahoo.com
Received: (qmail 42460 invoked by uid 60001); 14 Jan 2011 04:58:50 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1294981130; bh=GQRoeyU9v+peFJ8WnufIv6Iz2igzmlT6TwkD1dQTQcw=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=zagABAZByTcFgPGCYKxO+zgoaHe6eIGzqr83JHreeK8uES10mEBJWB6958D2Pci6anEuPJABLzyuQzhTrEdGfldYC2ckNZjIQdZWxqFQWw0H+QgFKgIkoz3FDB1pIHxnx0oxZonx6hDpqEWaulITYkd0M1LQieO/5mSe4C7HwbE=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=myk5NlJZ3ZBKuIoSH52nHPCFzY02WklK6xVdWSEqmCXcGvbzgLNzpgmw2QVandN8yjyu5WP5yR35PYhbZJqubB/bhFPax5fjCTqYJFWGaijdCXzrXi4gkRgpEFZqEVMSNdzjndh8yl98ia5COgHT+5XfAJOfqQDPpbUYobAFils=;
Message-ID: <533709.42262.qm@web56201.mail.re3.yahoo.com>
X-YMail-OSG: CWFoATgVM1l9M1goX.H6wdJ024C1PTYdMtykIY3Yj2slCwK
 _wVOdy5zhEgUWafjO_75JRMRi51KbbHXZkavv4Qu7FNZBH266YYQCIlL4V.X
 fsxyD6KpMHViAYx_delIhmiLKjte0_e8ADfB1jHyqp68VEPg2J8RLMvMucj_
 KB3jjd.J_kU_MkiID7jmRhmqC1g7PhnquCH0M.Ojxj_hOx8Cfd6CeUTwxEAQ
 bu_729jKlFONwQt7Jn5qve.M5H7U9zR4ZbQKTIMtgGv3tygKFo5mi7yUOscE
 XUGI3pqj9BOKM2ytVe2w6NrTQ7ZPOtHMeYdrKIYy7.2HNUunbOMCBbeQcKf6
 9X2RWS2OL
Received: from [209.131.62.113] by web56201.mail.re3.yahoo.com via HTTP; Thu, 13 Jan 2011 20:58:50 PST
X-Mailer: YahooMailRC/553 YahooMailWebService/0.8.107.285259
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com> <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com> <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org> <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com> <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com> <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com> <AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com> <388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com> <AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com> <04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com> <AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com> <74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com> <AANLkTimLLrfYbaaOzbEV9dcsaMFLeMmdgx73FasdxGhA@mail.gmail.com> <B4C8E6B2-A343-449B-AE63-DE5E3F883ABA@yahoo-inc.com>
Date: Thu, 13 Jan 2011 20:58:50 -0800 (PST)
From: "Tsz Wo \(Nicholas\), Sze" <s29752-hadoopgeneral@yahoo.com>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
To: general@hadoop.apache.org
In-Reply-To: <B4C8E6B2-A343-449B-AE63-DE5E3F883ABA@yahoo-inc.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-830671904-1294981130=:42262"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-830671904-1294981130=:42262
Content-Type: text/plain; charset=us-ascii

Below are copied from http://httpd.apache.org/dev/release.html.  Not sure if it 
helps.

What power does the RM yield?
Regarding what makes it into a release, the RM is the unquestioned authority. No 
one can contest what makes it into the release. The community will judge the 
release's quality after it has been issued, but the community can not force the 
RM to include a feature that they feel uncomfortable adding. Remember that this 
document is only a guideline to the community and future RMs - each RM may run a 
release in a different way. If you don't like what an RM is doing, start 
preparing for your own competing release.

Nicholas

--0-830671904-1294981130=:42262--

From general-return-2759-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 05:12:08 2011
Return-Path: <general-return-2759-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94673 invoked from network); 14 Jan 2011 05:12:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 05:12:08 -0000
Received: (qmail 80776 invoked by uid 500); 14 Jan 2011 05:12:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80451 invoked by uid 500); 14 Jan 2011 05:12:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80443 invoked by uid 99); 14 Jan 2011 05:12:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 05:12:02 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 05:11:53 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0E5BKAi030003
	for <general@hadoop.apache.org>; Thu, 13 Jan 2011 21:11:20 -0800 (PST)
Received: from [10.0.1.4] (10.72.76.23) by SP2-EX07CAS01.ds.corp.yahoo.com
 (98.137.59.4) with Microsoft SMTP Server (TLS) id 8.3.106.1; Thu, 13 Jan 2011
 21:11:19 -0800
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <B4C8E6B2-A343-449B-AE63-DE5E3F883ABA@yahoo-inc.com>
Date: Thu, 13 Jan 2011 21:11:16 -0800
Content-Transfer-Encoding: quoted-printable
Message-ID: <AEB7A1C2-199B-4937-B104-5DBC51E64C44@yahoo-inc.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com> <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com> <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org> <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com> <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com> <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com> <AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com> <388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com> <AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com> <04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com> <AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com> <74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com> <AANLkTimLLrfYbaaOzbEV9dcsaMFLeMmdgx73FasdxGhA@mail.gmail.com> <B4C8E6B2-A343-449B-AE63-DE5E3F883ABA@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Eli,

Thanks for the suggestion.

+1 to nigel and arun's proposal.

I completely support the idea of creating a version of 20 with append =
for HBASE.  However, the append issue is very complicated and there does =
not exist any version of append that is certified against a workload as =
diverse as what this branch has been tested against.  I think you are =
trying to cross too many streams here.   If you have resources to help =
integrate any version of Hadoop 0.20 with append, package and test it, I =
fully support you doing so.  But that effort is not aligned with the =
goal of this branch, which is to share a substantial amount of fully =
integrated and tested work.  Members of the community have expressed =
interest in seeing this tested work get checked into Apache and I would =
like to share it.  Mashing it up with other patches would invalidate =
months of testing, defeating the purpose of the exercise.

If you are interested in integrating Append with this branch, why not =
create a 20.200 branch and do so?

Unless you are vetoing the sharing of work as is on a branch (the =
purpose of the branch), I suggest we move on.

Thanks,

E14


On Jan 13, 2011, at 8:23 PM, Arun C Murthy wrote:

>=20
> On Jan 13, 2011, at 6:50 PM, Eli Collins wrote:
>=20
>> The cdh3 patch set Todd is talking about is not vanilla 104.3, it's
>> 104.3 re-based onto 20.2 plus patches from branch-20 and trunk (the
>> performance and stability fixes I think you're referring to, at least
>> the ones that have been posted to Apache jira).
>>=20
>> Can you post a pointer to the version you're referring to, eg on
>> github?  If there isn't a big delta between it and the cdh3 patch set
>> (which should have the 20-based patches from jira) perhaps you and
>> Todd could easily merge in the delta to create 0.20.x?
>>=20
>=20
> I can guarantee it will need work to merge the enhancements since =20
> 20.104.3, it's over 6 months of development. The enhancements includes =
=20
> work on stability such as iterative ls, limits on JT to prevent single =
=20
> jobs/users from taking it down etc. and lots of bug-fixes to security. =
=20
> So, unfortunately the delta is pretty large.
>=20
> I'm working on a CHANGES.txt which should reflect all the changes i.e. =
=20
> bug-fixes and enhancements.
>=20
>>> The version I'm offering to push to the community has fixed all of =20=

>>> them,
>>> *plus* the added benefit of several stability and performance fixes =20=

>>> we have
>>> done since 20.104.3, almost 10 internal releases. This is a battle =20=

>>> tested
>>> and hardened version which we have deployed on 40,000+ nodes. It is =
a
>>> significant upgrade on 0.20.104.3 which we never deployed. I'm =20
>>> pretty sure
>>> *some* users will find that valuable. ;)
>>=20
>> Definitely, but better to hit two birds with one stone right?  =
Instead
>> of a security + enhancements release and an append release we could
>> have a single security + append + enhancements release and users =
don't
>> have to choose.
>>=20
>=20
>=20
> We are discussing two options:
> 20 + security + enhancements
> 20 + security + append
>=20
> I think the value we provide via 20+security+enhancements release is =20=

> that it's stable, tested and deployed at scale. Doing any more work =20=

> merging 6 months of work at Yahoo (again, I guarantee it's a lot of =20=

> work) will need a lots of cycles to validate, test and stabilize.
>=20
> I feel the alternative is a distraction for me, I'd rather work on =
0.22.
>=20
> I can get 20+security+enhancements done very, very, quickly precisely =20=

> because I don't have to spend cycles testing it.
>=20
> Does that make sense? Thanks for being patient and bearing with me...
>=20
> Arun
>=20


From general-return-2760-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 06:07:51 2011
Return-Path: <general-return-2760-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25340 invoked from network); 14 Jan 2011 06:07:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 06:07:50 -0000
Received: (qmail 9509 invoked by uid 500); 14 Jan 2011 06:07:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8975 invoked by uid 500); 14 Jan 2011 06:07:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8962 invoked by uid 99); 14 Jan 2011 06:07:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 06:07:44 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.98 as permitted sender)
Received: from [17.148.16.98] (HELO asmtpout023.mac.com) (17.148.16.98)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 06:07:35 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp023.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LF000D3S101U9B0@asmtp023.mac.com>
 for general@hadoop.apache.org; Thu, 13 Jan 2011 22:07:14 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-14_03:2011-01-14,2011-01-13,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101130227
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <B4C8E6B2-A343-449B-AE63-DE5E3F883ABA@yahoo-inc.com>
Date: Thu, 13 Jan 2011 22:07:13 -0800
Message-id: <36CBB8E6-05D6-4AC0-838C-14B1C2B19CBD@mac.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
 <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
 <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
 <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
 <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
 <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
 <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
 <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
 <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
 <AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com>
 <388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com>
 <AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com>
 <04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com>
 <AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com>
 <74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com>
 <AANLkTimLLrfYbaaOzbEV9dcsaMFLeMmdgx73FasdxGhA@mail.gmail.com>
 <B4C8E6B2-A343-449B-AE63-DE5E3F883ABA@yahoo-inc.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

I say just do it.  Eli said it wasn't a blocker. Sure it ain't perfect, but it's good enough.

Let's move on to 0.22 and beyond.

Nige

On Jan 13, 2011, at 8:23 PM, Arun C Murthy wrote:

> 
> On Jan 13, 2011, at 6:50 PM, Eli Collins wrote:
> 
>> The cdh3 patch set Todd is talking about is not vanilla 104.3, it's
>> 104.3 re-based onto 20.2 plus patches from branch-20 and trunk (the
>> performance and stability fixes I think you're referring to, at least
>> the ones that have been posted to Apache jira).
>> 
>> Can you post a pointer to the version you're referring to, eg on
>> github?  If there isn't a big delta between it and the cdh3 patch set
>> (which should have the 20-based patches from jira) perhaps you and
>> Todd could easily merge in the delta to create 0.20.x?
>> 
> 
> I can guarantee it will need work to merge the enhancements since 20.104.3, it's over 6 months of development. The enhancements includes work on stability such as iterative ls, limits on JT to prevent single jobs/users from taking it down etc. and lots of bug-fixes to security. So, unfortunately the delta is pretty large.
> 
> I'm working on a CHANGES.txt which should reflect all the changes i.e. bug-fixes and enhancements.
> 
>>> The version I'm offering to push to the community has fixed all of them,
>>> *plus* the added benefit of several stability and performance fixes we have
>>> done since 20.104.3, almost 10 internal releases. This is a battle tested
>>> and hardened version which we have deployed on 40,000+ nodes. It is a
>>> significant upgrade on 0.20.104.3 which we never deployed. I'm pretty sure
>>> *some* users will find that valuable. ;)
>> 
>> Definitely, but better to hit two birds with one stone right?  Instead
>> of a security + enhancements release and an append release we could
>> have a single security + append + enhancements release and users don't
>> have to choose.
>> 
> 
> 
> We are discussing two options:
> 20 + security + enhancements
> 20 + security + append
> 
> I think the value we provide via 20+security+enhancements release is that it's stable, tested and deployed at scale. Doing any more work merging 6 months of work at Yahoo (again, I guarantee it's a lot of work) will need a lots of cycles to validate, test and stabilize.
> 
> I feel the alternative is a distraction for me, I'd rather work on 0.22.
> 
> I can get 20+security+enhancements done very, very, quickly precisely because I don't have to spend cycles testing it.
> 
> Does that make sense? Thanks for being patient and bearing with me...
> 
> Arun
> 


From general-return-2761-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 06:18:48 2011
Return-Path: <general-return-2761-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27535 invoked from network); 14 Jan 2011 06:18:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 06:18:48 -0000
Received: (qmail 20411 invoked by uid 500); 14 Jan 2011 06:18:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20073 invoked by uid 500); 14 Jan 2011 06:18:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 20065 invoked by uid 99); 14 Jan 2011 06:18:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 06:18:43 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.99 as permitted sender)
Received: from [17.148.16.99] (HELO asmtpout024.mac.com) (17.148.16.99)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 06:18:34 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp024.mac.com (Oracle Communications Messaging Exchange Server 7u4-18.01
 64bit (built Jul 15 2010)) with ESMTPSA id <0LF0000NO1ICZM30@asmtp024.mac.com>
 for general@hadoop.apache.org; Thu, 13 Jan 2011 22:18:13 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-14_03:2011-01-14,2011-01-13,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101130228
From: Nigel Daley <ndaley@mac.com>
Subject: [DISCUSS] Move project split down a level
Date: Thu, 13 Jan 2011 22:18:12 -0800
Message-id: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

Folks,

As I look more at the impact of the common/MR/HDFS project split on what and how we release Hadoop, I feel like the split needs an adjustment.  Many folks I've talked to agree that the project split has caused us a splitting headache.  I think 1 relatively small change could alleviate some of that.

CURRENT SVN REPO:

hadoop / [common, mapreduce, hdfs] / trunk
hadoop / [common, mapreduce, hdfs] / branches

PROPOSAL:

hadoop / trunk / [common, mapreduce, hdfs]
hadoop / branches / [common, mapreduce, hdfs]

We're a long way from releasing these 3 projects independently.  Given that, they should be branched and released as a unit.  This SVN structure enforces that and provides a more natural place to keep a top level build and pkg scripts that operate across all 3 projects.  

Thoughts?

Cheers,
Nige

From general-return-2762-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 06:22:44 2011
Return-Path: <general-return-2762-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27914 invoked from network); 14 Jan 2011 06:22:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 06:22:44 -0000
Received: (qmail 24332 invoked by uid 500); 14 Jan 2011 06:22:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24077 invoked by uid 500); 14 Jan 2011 06:22:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24069 invoked by uid 99); 14 Jan 2011 06:22:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 06:22:39 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 06:22:28 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0E6LifC069808
	for <general@hadoop.apache.org>; Thu, 13 Jan 2011 22:21:44 -0800 (PST)
Received: from SP2-EX07VS04.ds.corp.yahoo.com ([98.137.59.33]) by
 SP2-EX07CAS01.ds.corp.yahoo.com ([98.137.59.37]) with mapi; Thu, 13 Jan 2011
 22:21:43 -0800
From: Arun C Murthy <arunc@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 13 Jan 2011 22:21:39 -0800
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Thread-Topic: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Thread-Index: Acuzs0/6zNYFKtVfS8ylIlL2Hzktsw==
Message-ID: <00F14279-2802-4CC3-91AC-481AE257FD8B@yahoo-inc.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
 <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
 <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
 <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
 <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
 <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
 <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
 <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
 <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
 <AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com>
 <388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com>
 <AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com>
 <04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com>
 <AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com>
 <74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com>
 <AANLkTimLLrfYbaaOzbEV9dcsaMFLeMmdgx73FasdxGhA@mail.gmail.com>
 <B4C8E6B2-A343-449B-AE63-DE5E3F883ABA@yahoo-inc.com>
 <36CBB8E6-05D6-4AC0-838C-14B1C2B19CBD@mac.com>
In-Reply-To: <36CBB8E6-05D6-4AC0-838C-14B1C2B19CBD@mac.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

*nod* Ok.

Arun

On Jan 13, 2011, at 10:08 PM, "Nigel Daley" <ndaley@mac.com> wrote:

> I say just do it.  Eli said it wasn't a blocker. Sure it ain't perfect, b=
ut it's good enough.
>=20
> Let's move on to 0.22 and beyond.
>=20
> Nige
>=20
> On Jan 13, 2011, at 8:23 PM, Arun C Murthy wrote:
>=20
>>=20
>> On Jan 13, 2011, at 6:50 PM, Eli Collins wrote:
>>=20
>>> The cdh3 patch set Todd is talking about is not vanilla 104.3, it's
>>> 104.3 re-based onto 20.2 plus patches from branch-20 and trunk (the
>>> performance and stability fixes I think you're referring to, at least
>>> the ones that have been posted to Apache jira).
>>>=20
>>> Can you post a pointer to the version you're referring to, eg on
>>> github?  If there isn't a big delta between it and the cdh3 patch set
>>> (which should have the 20-based patches from jira) perhaps you and
>>> Todd could easily merge in the delta to create 0.20.x?
>>>=20
>>=20
>> I can guarantee it will need work to merge the enhancements since 20.104=
.3, it's over 6 months of development. The enhancements includes work on st=
ability such as iterative ls, limits on JT to prevent single jobs/users fro=
m taking it down etc. and lots of bug-fixes to security. So, unfortunately =
the delta is pretty large.
>>=20
>> I'm working on a CHANGES.txt which should reflect all the changes i.e. b=
ug-fixes and enhancements.
>>=20
>>>> The version I'm offering to push to the community has fixed all of the=
m,
>>>> *plus* the added benefit of several stability and performance fixes we=
 have
>>>> done since 20.104.3, almost 10 internal releases. This is a battle tes=
ted
>>>> and hardened version which we have deployed on 40,000+ nodes. It is a
>>>> significant upgrade on 0.20.104.3 which we never deployed. I'm pretty =
sure
>>>> *some* users will find that valuable. ;)
>>>=20
>>> Definitely, but better to hit two birds with one stone right?  Instead
>>> of a security + enhancements release and an append release we could
>>> have a single security + append + enhancements release and users don't
>>> have to choose.
>>>=20
>>=20
>>=20
>> We are discussing two options:
>> 20 + security + enhancements
>> 20 + security + append
>>=20
>> I think the value we provide via 20+security+enhancements release is tha=
t it's stable, tested and deployed at scale. Doing any more work merging 6 =
months of work at Yahoo (again, I guarantee it's a lot of work) will need a=
 lots of cycles to validate, test and stabilize.
>>=20
>> I feel the alternative is a distraction for me, I'd rather work on 0.22.
>>=20
>> I can get 20+security+enhancements done very, very, quickly precisely be=
cause I don't have to spend cycles testing it.
>>=20
>> Does that make sense? Thanks for being patient and bearing with me...
>>=20
>> Arun
>>=20
>=20

From general-return-2763-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 06:29:32 2011
Return-Path: <general-return-2763-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34794 invoked from network); 14 Jan 2011 06:29:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 06:29:31 -0000
Received: (qmail 26744 invoked by uid 500); 14 Jan 2011 06:29:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26497 invoked by uid 500); 14 Jan 2011 06:29:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26489 invoked by uid 99); 14 Jan 2011 06:29:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 06:29:27 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 06:29:22 +0000
Received: by qyk7 with SMTP id 7so5898395qyk.14
        for <general@hadoop.apache.org>; Thu, 13 Jan 2011 22:29:01 -0800 (PST)
MIME-Version: 1.0
Received: by 10.229.239.142 with SMTP id kw14mr413425qcb.38.1294986540924;
 Thu, 13 Jan 2011 22:29:00 -0800 (PST)
Received: by 10.229.215.2 with HTTP; Thu, 13 Jan 2011 22:29:00 -0800 (PST)
In-Reply-To: <00F14279-2802-4CC3-91AC-481AE257FD8B@yahoo-inc.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
	<366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
	<18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
	<AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
	<F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
	<D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
	<2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
	<5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
	<075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
	<AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com>
	<388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com>
	<AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com>
	<04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com>
	<AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com>
	<74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com>
	<AANLkTimLLrfYbaaOzbEV9dcsaMFLeMmdgx73FasdxGhA@mail.gmail.com>
	<B4C8E6B2-A343-449B-AE63-DE5E3F883ABA@yahoo-inc.com>
	<36CBB8E6-05D6-4AC0-838C-14B1C2B19CBD@mac.com>
	<00F14279-2802-4CC3-91AC-481AE257FD8B@yahoo-inc.com>
Date: Thu, 13 Jan 2011 22:29:00 -0800
Message-ID: <AANLkTikqbRrU+3+y2nctu7RzKNb7+P8kd=4az+eQHzKW@mail.gmail.com>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Sorry for rattling you guys, definitely wasn't discussing a veto.  I'm
absolutely not opposed, just thought the alternative Todd raised was
worth a couple emails since users have requested both security and
append, and such a branch that includes both of those plus
enhancements and substantial testing exists.

Arun - I appreciate all the info, looking forward to the release.

Thanks,
Eli

On Thu, Jan 13, 2011 at 10:21 PM, Arun C Murthy <arunc@yahoo-inc.com> wrote=
:
> *nod* Ok.
>
> Arun
>
> On Jan 13, 2011, at 10:08 PM, "Nigel Daley" <ndaley@mac.com> wrote:
>
>> I say just do it. =A0Eli said it wasn't a blocker. Sure it ain't perfect=
, but it's good enough.
>>
>> Let's move on to 0.22 and beyond.
>>
>> Nige
>>
>> On Jan 13, 2011, at 8:23 PM, Arun C Murthy wrote:
>>
>>>
>>> On Jan 13, 2011, at 6:50 PM, Eli Collins wrote:
>>>
>>>> The cdh3 patch set Todd is talking about is not vanilla 104.3, it's
>>>> 104.3 re-based onto 20.2 plus patches from branch-20 and trunk (the
>>>> performance and stability fixes I think you're referring to, at least
>>>> the ones that have been posted to Apache jira).
>>>>
>>>> Can you post a pointer to the version you're referring to, eg on
>>>> github? =A0If there isn't a big delta between it and the cdh3 patch se=
t
>>>> (which should have the 20-based patches from jira) perhaps you and
>>>> Todd could easily merge in the delta to create 0.20.x?
>>>>
>>>
>>> I can guarantee it will need work to merge the enhancements since 20.10=
4.3, it's over 6 months of development. The enhancements includes work on s=
tability such as iterative ls, limits on JT to prevent single jobs/users fr=
om taking it down etc. and lots of bug-fixes to security. So, unfortunately=
 the delta is pretty large.
>>>
>>> I'm working on a CHANGES.txt which should reflect all the changes i.e. =
bug-fixes and enhancements.
>>>
>>>>> The version I'm offering to push to the community has fixed all of th=
em,
>>>>> *plus* the added benefit of several stability and performance fixes w=
e have
>>>>> done since 20.104.3, almost 10 internal releases. This is a battle te=
sted
>>>>> and hardened version which we have deployed on 40,000+ nodes. It is a
>>>>> significant upgrade on 0.20.104.3 which we never deployed. I'm pretty=
 sure
>>>>> *some* users will find that valuable. ;)
>>>>
>>>> Definitely, but better to hit two birds with one stone right? =A0Inste=
ad
>>>> of a security + enhancements release and an append release we could
>>>> have a single security + append + enhancements release and users don't
>>>> have to choose.
>>>>
>>>
>>>
>>> We are discussing two options:
>>> 20 + security + enhancements
>>> 20 + security + append
>>>
>>> I think the value we provide via 20+security+enhancements release is th=
at it's stable, tested and deployed at scale. Doing any more work merging 6=
 months of work at Yahoo (again, I guarantee it's a lot of work) will need =
a lots of cycles to validate, test and stabilize.
>>>
>>> I feel the alternative is a distraction for me, I'd rather work on 0.22=
.
>>>
>>> I can get 20+security+enhancements done very, very, quickly precisely b=
ecause I don't have to spend cycles testing it.
>>>
>>> Does that make sense? Thanks for being patient and bearing with me...
>>>
>>> Arun
>>>
>>
>

From general-return-2764-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 06:37:44 2011
Return-Path: <general-return-2764-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37118 invoked from network); 14 Jan 2011 06:37:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 06:37:44 -0000
Received: (qmail 30729 invoked by uid 500); 14 Jan 2011 06:37:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30566 invoked by uid 500); 14 Jan 2011 06:37:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30553 invoked by uid 99); 14 Jan 2011 06:37:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 06:37:39 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 06:37:33 +0000
Received: by iwn2 with SMTP id 2so2509646iwn.35
        for <general@hadoop.apache.org>; Thu, 13 Jan 2011 22:37:12 -0800 (PST)
Received: by 10.231.59.149 with SMTP id l21mr322549ibh.196.1294987031933; Thu,
 13 Jan 2011 22:37:11 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.115.8 with HTTP; Thu, 13 Jan 2011 22:36:51 -0800 (PST)
In-Reply-To: <AANLkTikqbRrU+3+y2nctu7RzKNb7+P8kd=4az+eQHzKW@mail.gmail.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
 <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
 <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
 <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
 <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com> <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
 <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com> <AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com>
 <388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com> <AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com>
 <04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com> <AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com>
 <74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com> <AANLkTimLLrfYbaaOzbEV9dcsaMFLeMmdgx73FasdxGhA@mail.gmail.com>
 <B4C8E6B2-A343-449B-AE63-DE5E3F883ABA@yahoo-inc.com> <36CBB8E6-05D6-4AC0-838C-14B1C2B19CBD@mac.com>
 <00F14279-2802-4CC3-91AC-481AE257FD8B@yahoo-inc.com> <AANLkTikqbRrU+3+y2nctu7RzKNb7+P8kd=4az+eQHzKW@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Thu, 13 Jan 2011 22:36:51 -0800
Message-ID: <AANLkTinmeabTaF0ua5Bb=NbWNqV+Ws841q3HmMgbGA72@mail.gmail.com>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485ebea3806f1a60499c8aa6e
X-Virus-Checked: Checked by ClamAV on apache.org

--001485ebea3806f1a60499c8aa6e
Content-Type: text/plain; charset=ISO-8859-1

On Thu, Jan 13, 2011 at 10:29 PM, Eli Collins <eli@cloudera.com> wrote:

> Sorry for rattling you guys, definitely wasn't discussing a veto.  I'm
> absolutely not opposed, just thought the alternative Todd raised was
> worth a couple emails since users have requested both security and
> append, and such a branch that includes both of those plus
> enhancements and substantial testing exists.
>
> Arun - I appreciate all the info, looking forward to the release.
>
>
Same here.

Back to the patch queue for me! 0.22 here we come.

-Todd


>  On Thu, Jan 13, 2011 at 10:21 PM, Arun C Murthy <arunc@yahoo-inc.com>
> wrote:
> > *nod* Ok.
> >
> > Arun
> >
> > On Jan 13, 2011, at 10:08 PM, "Nigel Daley" <ndaley@mac.com> wrote:
> >
> >> I say just do it.  Eli said it wasn't a blocker. Sure it ain't perfect,
> but it's good enough.
> >>
> >> Let's move on to 0.22 and beyond.
> >>
> >> Nige
> >>
> >> On Jan 13, 2011, at 8:23 PM, Arun C Murthy wrote:
> >>
> >>>
> >>> On Jan 13, 2011, at 6:50 PM, Eli Collins wrote:
> >>>
> >>>> The cdh3 patch set Todd is talking about is not vanilla 104.3, it's
> >>>> 104.3 re-based onto 20.2 plus patches from branch-20 and trunk (the
> >>>> performance and stability fixes I think you're referring to, at least
> >>>> the ones that have been posted to Apache jira).
> >>>>
> >>>> Can you post a pointer to the version you're referring to, eg on
> >>>> github?  If there isn't a big delta between it and the cdh3 patch set
> >>>> (which should have the 20-based patches from jira) perhaps you and
> >>>> Todd could easily merge in the delta to create 0.20.x?
> >>>>
> >>>
> >>> I can guarantee it will need work to merge the enhancements since
> 20.104.3, it's over 6 months of development. The enhancements includes work
> on stability such as iterative ls, limits on JT to prevent single jobs/users
> from taking it down etc. and lots of bug-fixes to security. So,
> unfortunately the delta is pretty large.
> >>>
> >>> I'm working on a CHANGES.txt which should reflect all the changes i.e.
> bug-fixes and enhancements.
> >>>
> >>>>> The version I'm offering to push to the community has fixed all of
> them,
> >>>>> *plus* the added benefit of several stability and performance fixes
> we have
> >>>>> done since 20.104.3, almost 10 internal releases. This is a battle
> tested
> >>>>> and hardened version which we have deployed on 40,000+ nodes. It is a
> >>>>> significant upgrade on 0.20.104.3 which we never deployed. I'm pretty
> sure
> >>>>> *some* users will find that valuable. ;)
> >>>>
> >>>> Definitely, but better to hit two birds with one stone right?  Instead
> >>>> of a security + enhancements release and an append release we could
> >>>> have a single security + append + enhancements release and users don't
> >>>> have to choose.
> >>>>
> >>>
> >>>
> >>> We are discussing two options:
> >>> 20 + security + enhancements
> >>> 20 + security + append
> >>>
> >>> I think the value we provide via 20+security+enhancements release is
> that it's stable, tested and deployed at scale. Doing any more work merging
> 6 months of work at Yahoo (again, I guarantee it's a lot of work) will need
> a lots of cycles to validate, test and stabilize.
> >>>
> >>> I feel the alternative is a distraction for me, I'd rather work on
> 0.22.
> >>>
> >>> I can get 20+security+enhancements done very, very, quickly precisely
> because I don't have to spend cycles testing it.
> >>>
> >>> Does that make sense? Thanks for being patient and bearing with me...
> >>>
> >>> Arun
> >>>
> >>
> >
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--001485ebea3806f1a60499c8aa6e--

From general-return-2765-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 06:37:50 2011
Return-Path: <general-return-2765-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37215 invoked from network); 14 Jan 2011 06:37:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 06:37:50 -0000
Received: (qmail 32035 invoked by uid 500); 14 Jan 2011 06:37:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31961 invoked by uid 500); 14 Jan 2011 06:37:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31951 invoked by uid 99); 14 Jan 2011 06:37:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 06:37:45 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 06:37:39 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0E6b80H052330
	for <general@hadoop.apache.org>; Thu, 13 Jan 2011 22:37:08 -0800 (PST)
Received: from SP2-EX07VS04.ds.corp.yahoo.com ([98.137.59.33]) by
 SP2-EX07CAS01.ds.corp.yahoo.com ([98.137.59.37]) with mapi; Thu, 13 Jan 2011
 22:37:07 -0800
From: Arun C Murthy <arunc@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 13 Jan 2011 22:37:03 -0800
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Thread-Topic: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Thread-Index: AcuztXaectlR6qMeRFybFnI8dG8VBg==
Message-ID: <58FF3525-B734-44A0-A5C6-45282A23F06F@yahoo-inc.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
 <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
 <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
 <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
 <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
 <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
 <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
 <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
 <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
 <AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com>
 <388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com>
 <AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com>
 <04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com>
 <AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com>
 <74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com>
 <AANLkTimLLrfYbaaOzbEV9dcsaMFLeMmdgx73FasdxGhA@mail.gmail.com>
 <B4C8E6B2-A343-449B-AE63-DE5E3F883ABA@yahoo-inc.com>
 <36CBB8E6-05D6-4AC0-838C-14B1C2B19CBD@mac.com>
 <00F14279-2802-4CC3-91AC-481AE257FD8B@yahoo-inc.com>
 <AANLkTikqbRrU+3+y2nctu7RzKNb7+P8kd=4az+eQHzKW@mail.gmail.com>
In-Reply-To: <AANLkTikqbRrU+3+y2nctu7RzKNb7+P8kd=4az+eQHzKW@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

No worries. Thanks to both Eli & Todd for the discussion.=20

I look forward to getting this done and moving ahead to 0.22 and beyond.

thanks,
Arun

On Jan 13, 2011, at 10:29 PM, "Eli Collins" <eli@cloudera.com> wrote:

> Sorry for rattling you guys, definitely wasn't discussing a veto.  I'm
> absolutely not opposed, just thought the alternative Todd raised was
> worth a couple emails since users have requested both security and
> append, and such a branch that includes both of those plus
> enhancements and substantial testing exists.
>=20
> Arun - I appreciate all the info, looking forward to the release.
>=20
> Thanks,
> Eli
>=20
> On Thu, Jan 13, 2011 at 10:21 PM, Arun C Murthy <arunc@yahoo-inc.com> wro=
te:
>> *nod* Ok.
>>=20
>> Arun
>>=20
>> On Jan 13, 2011, at 10:08 PM, "Nigel Daley" <ndaley@mac.com> wrote:
>>=20
>>> I say just do it.  Eli said it wasn't a blocker. Sure it ain't perfect,=
 but it's good enough.
>>>=20
>>> Let's move on to 0.22 and beyond.
>>>=20
>>> Nige
>>>=20
>>> On Jan 13, 2011, at 8:23 PM, Arun C Murthy wrote:
>>>=20
>>>>=20
>>>> On Jan 13, 2011, at 6:50 PM, Eli Collins wrote:
>>>>=20
>>>>> The cdh3 patch set Todd is talking about is not vanilla 104.3, it's
>>>>> 104.3 re-based onto 20.2 plus patches from branch-20 and trunk (the
>>>>> performance and stability fixes I think you're referring to, at least
>>>>> the ones that have been posted to Apache jira).
>>>>>=20
>>>>> Can you post a pointer to the version you're referring to, eg on
>>>>> github?  If there isn't a big delta between it and the cdh3 patch set
>>>>> (which should have the 20-based patches from jira) perhaps you and
>>>>> Todd could easily merge in the delta to create 0.20.x?
>>>>>=20
>>>>=20
>>>> I can guarantee it will need work to merge the enhancements since 20.1=
04.3, it's over 6 months of development. The enhancements includes work on =
stability such as iterative ls, limits on JT to prevent single jobs/users f=
rom taking it down etc. and lots of bug-fixes to security. So, unfortunatel=
y the delta is pretty large.
>>>>=20
>>>> I'm working on a CHANGES.txt which should reflect all the changes i.e.=
 bug-fixes and enhancements.
>>>>=20
>>>>>> The version I'm offering to push to the community has fixed all of t=
hem,
>>>>>> *plus* the added benefit of several stability and performance fixes =
we have
>>>>>> done since 20.104.3, almost 10 internal releases. This is a battle t=
ested
>>>>>> and hardened version which we have deployed on 40,000+ nodes. It is =
a
>>>>>> significant upgrade on 0.20.104.3 which we never deployed. I'm prett=
y sure
>>>>>> *some* users will find that valuable. ;)
>>>>>=20
>>>>> Definitely, but better to hit two birds with one stone right?  Instea=
d
>>>>> of a security + enhancements release and an append release we could
>>>>> have a single security + append + enhancements release and users don'=
t
>>>>> have to choose.
>>>>>=20
>>>>=20
>>>>=20
>>>> We are discussing two options:
>>>> 20 + security + enhancements
>>>> 20 + security + append
>>>>=20
>>>> I think the value we provide via 20+security+enhancements release is t=
hat it's stable, tested and deployed at scale. Doing any more work merging =
6 months of work at Yahoo (again, I guarantee it's a lot of work) will need=
 a lots of cycles to validate, test and stabilize.
>>>>=20
>>>> I feel the alternative is a distraction for me, I'd rather work on 0.2=
2.
>>>>=20
>>>> I can get 20+security+enhancements done very, very, quickly precisely =
because I don't have to spend cycles testing it.
>>>>=20
>>>> Does that make sense? Thanks for being patient and bearing with me...
>>>>=20
>>>> Arun
>>>>=20
>>>=20
>>=20

From general-return-2766-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 07:00:27 2011
Return-Path: <general-return-2766-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 40742 invoked from network); 14 Jan 2011 07:00:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 07:00:27 -0000
Received: (qmail 46076 invoked by uid 500); 14 Jan 2011 07:00:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45691 invoked by uid 500); 14 Jan 2011 07:00:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45683 invoked by uid 99); 14 Jan 2011 07:00:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 07:00:20 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 07:00:14 +0000
Received: by wye20 with SMTP id 20so2578084wye.35
        for <general@hadoop.apache.org>; Thu, 13 Jan 2011 22:59:52 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        bh=4wc2/K6gHnEKr7nEdxqtJS8QVx3M1k4ArDcpIVuwXdU=;
        b=Dnu4G5P/2MZpbPiuPRvoGKZjeCeQVbP89rsCqaGSm3pQKnQpA+NTOXf+DDFzszSgTQ
         iUxDgZD0r8LQWZvclZik7ebuXEMACFkiKrTAKw+0XKFBu8k/1GAfshUl2KThKiX65Jlf
         lwoCvCsZXjzvZYzT5KeuuydtiSFd1ms+cNROE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=A5mzzYck9Li6RKxBJG2VZsNmMBS3hq7pEfVJGvoN/tY2NQAx7QuUA3yKDosKmymrbN
         UpbFbr1GypFjRGqHNDZKix4WNl7amJi06gQ8G2Tz3CqHUMsJhL8VzW4meSO+ZOyygH2O
         w61sQVMODNQE40xTIdTp903D5C3bPK5xXsqiY=
MIME-Version: 1.0
Received: by 10.216.7.205 with SMTP id 55mr1322764wep.96.1294988391731; Thu,
 13 Jan 2011 22:59:51 -0800 (PST)
Sender: saint.ack@gmail.com
Received: by 10.216.49.11 with HTTP; Thu, 13 Jan 2011 22:59:51 -0800 (PST)
In-Reply-To: <58FF3525-B734-44A0-A5C6-45282A23F06F@yahoo-inc.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
	<366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
	<18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
	<AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
	<F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
	<D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
	<2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
	<5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
	<075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
	<AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com>
	<388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com>
	<AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com>
	<04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com>
	<AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com>
	<74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com>
	<AANLkTimLLrfYbaaOzbEV9dcsaMFLeMmdgx73FasdxGhA@mail.gmail.com>
	<B4C8E6B2-A343-449B-AE63-DE5E3F883ABA@yahoo-inc.com>
	<36CBB8E6-05D6-4AC0-838C-14B1C2B19CBD@mac.com>
	<00F14279-2802-4CC3-91AC-481AE257FD8B@yahoo-inc.com>
	<AANLkTikqbRrU+3+y2nctu7RzKNb7+P8kd=4az+eQHzKW@mail.gmail.com>
	<58FF3525-B734-44A0-A5C6-45282A23F06F@yahoo-inc.com>
Date: Thu, 13 Jan 2011 22:59:51 -0800
X-Google-Sender-Auth: VKCnvQOVy8xi9zOr_eU1WnxlUTI
Message-ID: <AANLkTik-=ZDac0iEbYuDHeBrRosQ3JNGvRbf7MxexQND@mail.gmail.com>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

(Man, it was looking good there for a second when 0.20.100 was about
security+append!)

Good luck w/ the release Arun.

We might be following your 0.20.100 with a 0.20.200 append.

St.Ack

From general-return-2767-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 07:17:26 2011
Return-Path: <general-return-2767-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53163 invoked from network); 14 Jan 2011 07:17:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 07:17:25 -0000
Received: (qmail 57112 invoked by uid 500); 14 Jan 2011 07:17:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56760 invoked by uid 500); 14 Jan 2011 07:17:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56752 invoked by uid 99); 14 Jan 2011 07:17:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 07:17:20 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 07:17:12 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0E7GWHL064163
	for <general@hadoop.apache.org>; Thu, 13 Jan 2011 23:16:32 -0800 (PST)
Received: from [10.0.1.4] (10.72.76.23) by SP2-EX07CAS01.ds.corp.yahoo.com
 (98.137.59.4) with Microsoft SMTP Server (TLS) id 8.3.106.1; Thu, 13 Jan 2011
 23:16:31 -0800
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <AANLkTik-=ZDac0iEbYuDHeBrRosQ3JNGvRbf7MxexQND@mail.gmail.com>
Date: Thu, 13 Jan 2011 23:16:29 -0800
Content-Transfer-Encoding: 7bit
Message-ID: <228FCE4D-1021-473E-A566-7889CA50463A@yahoo-inc.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
 <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
 <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
 <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
 <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
 <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
 <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
 <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
 <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
 <AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com>
 <388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com>
 <AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com>
 <04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com>
 <AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com>
 <74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com>
 <AANLkTimLLrfYbaaOzbEV9dcsaMFLeMmdgx73FasdxGhA@mail.gmail.com>
 <B4C8E6B2-A343-449B-AE63-DE5E3F883ABA@yahoo-inc.com>
 <36CBB8E6-05D6-4AC0-838C-14B1C2B19CBD@mac.com>
 <00F14279-2802-4CC3-91AC-481AE257FD8B@yahoo-inc.com>
 <AANLkTikqbRrU+3+y2nctu7RzKNb7+P8kd=4az+eQHzKW@mail.gmail.com>
 <58FF3525-B734-44A0-A5C6-45282A23F06F@yahoo-inc.com>
 <AANLkTik-=ZDac0iEbYuDHeBrRosQ3JNGvRbf7MxexQND@mail.gmail.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)

I'd love to see that!

On Jan 13, 2011, at 10:59 PM, Stack wrote:

> (Man, it was looking good there for a second when 0.20.100 was about
> security+append!)
> 
> Good luck w/ the release Arun.
> 
> We might be following your 0.20.100 with a 0.20.200 append.
> 
> St.Ack


From general-return-2768-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 07:21:43 2011
Return-Path: <general-return-2768-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54166 invoked from network); 14 Jan 2011 07:21:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 07:21:42 -0000
Received: (qmail 66592 invoked by uid 500); 14 Jan 2011 07:21:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66110 invoked by uid 500); 14 Jan 2011 07:21:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66097 invoked by uid 99); 14 Jan 2011 07:21:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 07:21:36 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 07:21:31 +0000
Received: from [192.168.1.71] (snvvpn4-10-72-168-c97.hq.corp.yahoo.com [10.72.168.97])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p0E7L2c6098319
	for <general@hadoop.apache.org>; Thu, 13 Jan 2011 23:21:03 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1294989663;
	bh=3otjoQNNGxBqL3XY38exrBuqRlu76Y6iQlk4myHY7CQ=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=kGt/QUthNxmAilDUN+2poaKXVZXr/BbNGrwntMeppV1VhwRD6TE4NizQeiKzI6sms
	 IbCYhPM3Ofuud9x8mlH7QvhRy1sdRHrNPKj3nbxTN5uqf6/Hi4kNYlriHxgYcUcElp
	 r0InoIDnQ5B6XYIkmMslEbGPTrUBa0Z8EfDrrrOE=
Message-Id: <7D4397E2-B618-4BD1-8E1E-08B1C598B76F@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <AANLkTik-=ZDac0iEbYuDHeBrRosQ3JNGvRbf7MxexQND@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Date: Thu, 13 Jan 2011 23:21:02 -0800
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com> <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com> <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org> <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com> <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com> <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com> <AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com> <388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com> <AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com> <04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com> <AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com> <74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com> <AANLkTimLLrfYbaaOzbEV9dcsaMFLeMmdgx73FasdxGhA@mail.gmail.com> <B4C8E6B2-A343-449B-AE63-DE5E3F883ABA@yahoo-inc.com> <36CBB8E6-05D6-4AC0-838C-14B1C2B19CBD@!
 mac.com> <00F14279-2802-4CC3-91AC-481AE257FD8B@yahoo-inc.com> <AANLkTikqbRrU+3+y2nctu7RzKNb7+P8kd=4az+eQHzKW@mail.gmail.com> <58FF3525-B734-44A0-A5C6-45282A23F06F@yahoo-inc.com> <AANLkTik-=ZDac0iEbYuDHeBrRosQ3JNGvRbf7MxexQND@mail.gmail.com>
X-Mailer: Apple Mail (2.936)


On Jan 13, 2011, at 10:59 PM, Stack wrote:

> (Man, it was looking good there for a second when 0.20.100 was about
> security+append!)
>
> Good luck w/ the release Arun.
>

Thanks!

> We might be following your 0.20.100 with a 0.20.200 append.
>

Super!

Arun

From general-return-2769-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 07:25:52 2011
Return-Path: <general-return-2769-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54604 invoked from network); 14 Jan 2011 07:25:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 07:25:52 -0000
Received: (qmail 68508 invoked by uid 500); 14 Jan 2011 07:25:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68218 invoked by uid 500); 14 Jan 2011 07:25:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68210 invoked by uid 99); 14 Jan 2011 07:25:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 07:25:47 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 07:25:38 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0E7P620067179
	for <general@hadoop.apache.org>; Thu, 13 Jan 2011 23:25:06 -0800 (PST)
Received: from [10.0.1.4] (10.72.76.23) by SP2-EX07CAS01.ds.corp.yahoo.com
 (98.137.59.4) with Microsoft SMTP Server (TLS) id 8.3.106.1; Thu, 13 Jan 2011
 23:25:06 -0800
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [DISCUSS] Move project split down a level
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com>
Date: Thu, 13 Jan 2011 23:25:04 -0800
Content-Transfer-Encoding: quoted-printable
Message-ID: <91A364F8-480F-4323-92F6-15B47D5A3C32@yahoo-inc.com>
References: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Death to the project split!  Or short of that, anything to tame it.

On Jan 13, 2011, at 10:18 PM, Nigel Daley wrote:

> Folks,
>=20
> As I look more at the impact of the common/MR/HDFS project split on =
what and how we release Hadoop, I feel like the split needs an =
adjustment.  Many folks I've talked to agree that the project split has =
caused us a splitting headache.  I think 1 relatively small change could =
alleviate some of that.
>=20
> CURRENT SVN REPO:
>=20
> hadoop / [common, mapreduce, hdfs] / trunk
> hadoop / [common, mapreduce, hdfs] / branches
>=20
> PROPOSAL:
>=20
> hadoop / trunk / [common, mapreduce, hdfs]
> hadoop / branches / [common, mapreduce, hdfs]
>=20
> We're a long way from releasing these 3 projects independently.  Given =
that, they should be branched and released as a unit.  This SVN =
structure enforces that and provides a more natural place to keep a top =
level build and pkg scripts that operate across all 3 projects. =20
>=20
> Thoughts?
>=20
> Cheers,
> Nige


From general-return-2770-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 07:44:20 2011
Return-Path: <general-return-2770-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58615 invoked from network); 14 Jan 2011 07:44:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 07:44:20 -0000
Received: (qmail 90725 invoked by uid 500); 14 Jan 2011 07:44:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90393 invoked by uid 500); 14 Jan 2011 07:44:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90376 invoked by uid 99); 14 Jan 2011 07:44:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 07:44:14 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 07:44:08 +0000
Received: by iwn2 with SMTP id 2so2549787iwn.35
        for <general@hadoop.apache.org>; Thu, 13 Jan 2011 23:43:47 -0800 (PST)
Received: by 10.231.59.149 with SMTP id l21mr372466ibh.196.1294991027061; Thu,
 13 Jan 2011 23:43:47 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.115.8 with HTTP; Thu, 13 Jan 2011 23:43:26 -0800 (PST)
In-Reply-To: <91A364F8-480F-4323-92F6-15B47D5A3C32@yahoo-inc.com>
References: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com> <91A364F8-480F-4323-92F6-15B47D5A3C32@yahoo-inc.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Thu, 13 Jan 2011 23:43:26 -0800
Message-ID: <AANLkTi=4ZMg2s210hbVkaxDjh5+xNZT7GLkG34=SMk4A@mail.gmail.com>
Subject: Re: [DISCUSS] Move project split down a level
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485ebea3827c3af0499c998d8
X-Virus-Checked: Checked by ClamAV on apache.org

--001485ebea3827c3af0499c998d8
Content-Type: text/plain; charset=ISO-8859-1

Big +1.

Curious how this will map to git, though - do we go back to one git repo?

When we have a patch that is mainly HDFS or MR focused but will need changes
across projects, can we just put up one patch in HDFS/MR or do we still need
to open a parallel common JIRA?

On Thu, Jan 13, 2011 at 11:25 PM, Eric Baldeschwieler
<eric14@yahoo-inc.com>wrote:

> +1
>
> Death to the project split!  Or short of that, anything to tame it.
>
> On Jan 13, 2011, at 10:18 PM, Nigel Daley wrote:
>
> > Folks,
> >
> > As I look more at the impact of the common/MR/HDFS project split on what
> and how we release Hadoop, I feel like the split needs an adjustment.  Many
> folks I've talked to agree that the project split has caused us a splitting
> headache.  I think 1 relatively small change could alleviate some of that.
> >
> > CURRENT SVN REPO:
> >
> > hadoop / [common, mapreduce, hdfs] / trunk
> > hadoop / [common, mapreduce, hdfs] / branches
> >
> > PROPOSAL:
> >
> > hadoop / trunk / [common, mapreduce, hdfs]
> > hadoop / branches / [common, mapreduce, hdfs]
> >
> > We're a long way from releasing these 3 projects independently.  Given
> that, they should be branched and released as a unit.  This SVN structure
> enforces that and provides a more natural place to keep a top level build
> and pkg scripts that operate across all 3 projects.
> >
> > Thoughts?
> >
> > Cheers,
> > Nige
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

--001485ebea3827c3af0499c998d8--

From general-return-2771-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 08:32:44 2011
Return-Path: <general-return-2771-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 79063 invoked from network); 14 Jan 2011 08:32:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 08:32:44 -0000
Received: (qmail 46439 invoked by uid 500); 14 Jan 2011 08:32:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45952 invoked by uid 500); 14 Jan 2011 08:32:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45938 invoked by uid 99); 14 Jan 2011 08:32:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 08:32:38 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=MIME_QP_LONG_LINE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 08:32:31 +0000
Received: by qwh6 with SMTP id 6so2470011qwh.35
        for <general@hadoop.apache.org>; Fri, 14 Jan 2011 00:32:10 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:references:in-reply-to:mime-version
         :content-transfer-encoding:content-type:message-id:cc:x-mailer:from
         :subject:date:to;
        bh=vn4vVSirIjTWIsjUKj727wC+eBAf/I38vrpHuJiapec=;
        b=sOwBI98eVyitMigA1gT9cvcFjFrbt9JT+bERpjqSAjwu/kl8QVyv/Z/t/CGMFx2cJh
         mfINjzUiJH0X71znNmtW6Z+WYT0VdNKTKugu3wndtphOWrEKuLJl9kTyMdV+ma78YmUs
         y2xnrDcTfA8rdLUuxclcrqNoNYwLoByLJkPEc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=references:in-reply-to:mime-version:content-transfer-encoding
         :content-type:message-id:cc:x-mailer:from:subject:date:to;
        b=SNGIPwE8eUumgRewfKNs+w7PvW/vuAUOGTKI3xlyAGgtNFFGGQ4Re0/JkUemumheQz
         SpjSIxqpXA3BOT5UJkJ5cy3JXJTKaZZZmGBuE2yFFpmYsnEXb0C7kq7Mvrmc/q4GO5N2
         rKmK+DERxDT9Is05qsxDpDJJMS1s0vAuY0q2U=
Received: by 10.224.61.10 with SMTP id r10mr437094qah.105.1294993930049;
        Fri, 14 Jan 2011 00:32:10 -0800 (PST)
Received: from [192.168.1.163] ([64.206.95.115])
        by mx.google.com with ESMTPS id nb15sm671705qcb.38.2011.01.14.00.32.08
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Fri, 14 Jan 2011 00:32:08 -0800 (PST)
References: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com> <91A364F8-480F-4323-92F6-15B47D5A3C32@yahoo-inc.com>
In-Reply-To: <91A364F8-480F-4323-92F6-15B47D5A3C32@yahoo-inc.com>
Mime-Version: 1.0 (iPad Mail 8C148)
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii
Message-Id: <88238863-C1CB-4932-9288-40655F225D59@holsman.net>
Cc: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: iPad Mail (8C148)
From: Ian Holsman <hadoop@holsman.net>
Subject: Re: [DISCUSS] Move project split down a level
Date: Fri, 14 Jan 2011 03:32:03 -0500
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

+1 full agreement.=20

I think it will be a pita admin wise (due to how svn authorization is set up=
), so it might slow down creation of a new branch, but its worth it.

---=20
Ian Holsman
AOL Inc
Ian.Holsman@teamaol.com
(703) 879-3128 / AIM:ianholsman=20

it's just a technicality

On Jan 14, 2011, at 2:25 AM, Eric Baldeschwieler <eric14@yahoo-inc.com> wrot=
e:

> +1
>=20
> Death to the project split!  Or short of that, anything to tame it.
>=20
> On Jan 13, 2011, at 10:18 PM, Nigel Daley wrote:
>=20
>> Folks,
>>=20
>> As I look more at the impact of the common/MR/HDFS project split on what a=
nd how we release Hadoop, I feel like the split needs an adjustment.  Many f=
olks I've talked to agree that the project split has caused us a splitting h=
eadache.  I think 1 relatively small change could alleviate some of that.
>>=20
>> CURRENT SVN REPO:
>>=20
>> hadoop / [common, mapreduce, hdfs] / trunk
>> hadoop / [common, mapreduce, hdfs] / branches
>>=20
>> PROPOSAL:
>>=20
>> hadoop / trunk / [common, mapreduce, hdfs]
>> hadoop / branches / [common, mapreduce, hdfs]
>>=20
>> We're a long way from releasing these 3 projects independently.  Given th=
at, they should be branched and released as a unit.  This SVN structure enfo=
rces that and provides a more natural place to keep a top level build and pk=
g scripts that operate across all 3 projects. =20
>>=20
>> Thoughts?
>>=20
>> Cheers,
>> Nige
>=20

From general-return-2772-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 08:40:16 2011
Return-Path: <general-return-2772-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 80749 invoked from network); 14 Jan 2011 08:40:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 08:40:16 -0000
Received: (qmail 55457 invoked by uid 500); 14 Jan 2011 08:40:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55174 invoked by uid 500); 14 Jan 2011 08:40:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55166 invoked by uid 99); 14 Jan 2011 08:40:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 08:40:10 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jghoman@gmail.com designates 209.85.210.176 as permitted sender)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 08:40:06 +0000
Received: by iyb26 with SMTP id 26so2635672iyb.35
        for <general@hadoop.apache.org>; Fri, 14 Jan 2011 00:39:46 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type:content-transfer-encoding;
        bh=4Ty4x45GjPjkXH5+aAJFt3FDoCB1xUY8EZApB0naRrM=;
        b=ZexOgKDjlviZOeH3LsOzoj4KwaXG3oyQVPDbYudF60QSkAjw3PiNvpddKbEa5bGpXQ
         Aao6ZztXqkC44xdNGrlpVnNy/fd4qg3Yi7bQkeJYBbOWqUoRqXQAnJH9FgB+919dsVAL
         JYvDeJkP4gyqUXGXSWExBScSpJi5EqVmuVnlU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=R8mOyOgdjR26QnHrz+b2YK8mgPexQUYoQtXWnB0rVcn3tbYLvWz07qHpj5ZWWrLC+7
         Zt4UPSoPA5Fvc0nRyuztCf9Wt/1JB5cERlUsBRXjzsSrddLLirLoF4WBsOwjcD+B/JNV
         pDIXXwpn+cG8U9BXik7rQBhTXgNZq9tdg9lpE=
Received: by 10.231.16.68 with SMTP id n4mr443005iba.94.1294994385926; Fri, 14
 Jan 2011 00:39:45 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.25.163 with HTTP; Fri, 14 Jan 2011 00:39:25 -0800 (PST)
In-Reply-To: <88238863-C1CB-4932-9288-40655F225D59@holsman.net>
References: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com> <91A364F8-480F-4323-92F6-15B47D5A3C32@yahoo-inc.com>
 <88238863-C1CB-4932-9288-40655F225D59@holsman.net>
From: Jakob Homan <jghoman@gmail.com>
Date: Fri, 14 Jan 2011 00:39:25 -0800
Message-ID: <AANLkTi=tuU0pRo12W-9-6xn7rkb5jSveGz0My=VBGG4z@mail.gmail.com>
Subject: Re: [DISCUSS] Move project split down a level
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

+1. The project split is a lie.

On Fri, Jan 14, 2011 at 12:32 AM, Ian Holsman <hadoop@holsman.net> wrote:
> +1 full agreement.
>
> I think it will be a pita admin wise (due to how svn authorization is set=
 up), so it might slow down creation of a new branch, but its worth it.
>
> ---
> Ian Holsman
> AOL Inc
> Ian.Holsman@teamaol.com
> (703) 879-3128 / AIM:ianholsman
>
> it's just a technicality
>
> On Jan 14, 2011, at 2:25 AM, Eric Baldeschwieler <eric14@yahoo-inc.com> w=
rote:
>
>> +1
>>
>> Death to the project split! =A0Or short of that, anything to tame it.
>>
>> On Jan 13, 2011, at 10:18 PM, Nigel Daley wrote:
>>
>>> Folks,
>>>
>>> As I look more at the impact of the common/MR/HDFS project split on wha=
t and how we release Hadoop, I feel like the split needs an adjustment. =A0=
Many folks I've talked to agree that the project split has caused us a spli=
tting headache. =A0I think 1 relatively small change could alleviate some o=
f that.
>>>
>>> CURRENT SVN REPO:
>>>
>>> hadoop / [common, mapreduce, hdfs] / trunk
>>> hadoop / [common, mapreduce, hdfs] / branches
>>>
>>> PROPOSAL:
>>>
>>> hadoop / trunk / [common, mapreduce, hdfs]
>>> hadoop / branches / [common, mapreduce, hdfs]
>>>
>>> We're a long way from releasing these 3 projects independently. =A0Give=
n that, they should be branched and released as a unit. =A0This SVN structu=
re enforces that and provides a more natural place to keep a top level buil=
d and pkg scripts that operate across all 3 projects.
>>>
>>> Thoughts?
>>>
>>> Cheers,
>>> Nige
>>
>

From general-return-2773-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 08:58:46 2011
Return-Path: <general-return-2773-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85948 invoked from network); 14 Jan 2011 08:58:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 08:58:45 -0000
Received: (qmail 72724 invoked by uid 500); 14 Jan 2011 08:58:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72526 invoked by uid 500); 14 Jan 2011 08:58:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72518 invoked by uid 99); 14 Jan 2011 08:58:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 08:58:40 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 08:58:34 +0000
Received: by iyb26 with SMTP id 26so2647579iyb.35
        for <general@hadoop.apache.org>; Fri, 14 Jan 2011 00:58:13 -0800 (PST)
Received: by 10.231.59.149 with SMTP id l21mr430793ibh.196.1294995493726; Fri,
 14 Jan 2011 00:58:13 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.115.8 with HTTP; Fri, 14 Jan 2011 00:57:52 -0800 (PST)
In-Reply-To: <D570B102-7A4F-42E2-BB9A-B3CFC72428CC@mac.com>
References: <5125873.329641294877388152.JavaMail.jira@thor> <D570B102-7A4F-42E2-BB9A-B3CFC72428CC@mac.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Fri, 14 Jan 2011 00:57:52 -0800
Message-ID: <AANLkTikc_-eeChjjY6GkHJqih3OZMRZ15a=JTvY5Z9mc@mail.gmail.com>
Subject: Re: triggering automated precommit testing
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485ebea3863ab560499caa241

--001485ebea3863ab560499caa241
Content-Type: text/plain; charset=ISO-8859-1

Hey Nigel,

Would there be any way to add a feature where we can make some special
comment on the JIRA that would trigger a hudson retest? There are a lot of
really old patches out on the JIRA that would be worth re-testing against
trunk, and it's a pain to download and re-attach.

I'm thinking a comment with a special token like "@Hudson.Test"

Failing that, Ian, can you add me to the Hudson list?

-Todd

On Wed, Jan 12, 2011 at 4:33 PM, Nigel Daley <ndaley@mac.com> wrote:

> > Jakob Homan commented on HDFS-884:
> > ----------------------------------
> >
> >> Konstantin, if you're trying to kick a new patch build for this you no
> longer move it to "Open" and back to "Patch Available". Instead, you must
> upload a new patch. Or, if you have permission, you can kickhttps://
> hudson.apache.org/hudson/job/PreCommit-HDFS-Build/ and enter the issue
> number.
> >
> > That makes me sad.  Is this a new feature or regression?
>
> [For everyone's benefit, moving this to general@]
>
> Jakob, I referenced the change here: http://tinyurl.com/4crxlvy
> The new system is much more robust partial because it no longer relies on
> watching Jira generated emails to determined when issues move into Patch
> Available state. There is limited info I can get from the Jira API, thus the
> triggering mechanism had to change.
>
> Cheers,
> Nige
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--001485ebea3863ab560499caa241--

From general-return-2774-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 11:58:50 2011
Return-Path: <general-return-2774-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66546 invoked from network); 14 Jan 2011 11:58:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 11:58:49 -0000
Received: (qmail 745 invoked by uid 500); 14 Jan 2011 11:58:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 344 invoked by uid 500); 14 Jan 2011 11:58:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 335 invoked by uid 99); 14 Jan 2011 11:58:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 11:58:43 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [98.139.91.204] (HELO nm5-vm0.bullet.mail.sp2.yahoo.com) (98.139.91.204)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 14 Jan 2011 11:58:34 +0000
Received: from [98.139.91.70] by nm5.bullet.mail.sp2.yahoo.com with NNFMP; 14 Jan 2011 11:58:13 -0000
Received: from [98.139.91.1] by tm10.bullet.mail.sp2.yahoo.com with NNFMP; 14 Jan 2011 11:58:13 -0000
Received: from [127.0.0.1] by omp1001.mail.sp2.yahoo.com with NNFMP; 14 Jan 2011 11:58:13 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 884614.22992.bm@omp1001.mail.sp2.yahoo.com
Received: (qmail 76600 invoked by uid 60001); 14 Jan 2011 11:58:13 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1295006293; bh=kZ4ejDhFONO9VEGRm8aCpTZgRf0jtVnWLQ272wToZZY=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=SfZw2j95inmVZOqlCbHLGZIbgURttvlpP9lCH+uja7cUTDsQVNodRPWDRxeW1XS7CXRdoaAer0c/xFpA8ufFyeKxY6jJI2oaiTvozuRtUPtXihYluky26bF1hp7hsgK5C1ykksVWJpl0MdaU9DTs0xVJhmTITVfW6itKfJmWaic=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=0WGhvXt6J5bQBpnMNCoBVgCWjkrJPwJ8IpM3uaY9tovdGPmQ+81SlTE3ahDrcXSkHpOV52kX00/14SdCUj89HSh6eoL41eit+ZcFsPHQZVS2IvmQ3JzT4eWdV5a9R89ywBvKXAUeNuYunna/a1mAbHWbLUUIqZCoM1o1flgIYJ4=;
Message-ID: <461412.75099.qm@web112109.mail.gq1.yahoo.com>
X-YMail-OSG: AoZ0LyoVM1kt6yTg.hZeatfKXQvWHmshCD03QGyA8rhC8GX
 n54QX2mQW9qrmD_K4IVDfcXLGLvC07QeHXZe1Lo7.OQudA58PCr5nzVYjJew
 OqzIsF1bIFt5nCe.S5.7DMygL57F52YYBMYaC9VXTXm.ghmzzCJh6nPbyGib
 eazyYmnA1vf3d.TDSs4jbjkJ4VRFu_eqOveO2MNOTCHQqn13DZOiCJVttogy
 soNgxIHjz5VzOxuqqS6P4ujGbQwKDQAk9FcG47WVDsmZDgqUmiNiNIw5bdU1
 wJn7ZvLWCDW5iVQtayFf0l1PbVAzWCMKlitZPvqjKibpEh.Ph6dDci3m96FF
 S6ON1kg0rc3qkoASQaijSp__F_4b8YkW7QIoAcRdGUP6vPu73SpwLC1CFGo3
 6E0Su.lJjjfA-
Received: from [129.132.85.169] by web112109.mail.gq1.yahoo.com via HTTP; Fri, 14 Jan 2011 03:58:13 PST
X-Mailer: YahooMailClassic/11.4.20 YahooMailWebService/0.8.107.285259
Date: Fri, 14 Jan 2011 03:58:13 -0800 (PST)
From: Grandl Robert <rgrandl@yahoo.com>
Subject: Ease of development in Hadoop
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-877460507-1295006293=:75099"

--0-877460507-1295006293=:75099
Content-Type: text/plain; charset=us-ascii

Hi all,

I want to make some minor modifications in FairShare scheduler and recompile it such that to use it as before. Is there an easy way to do it without spending much time on recompiling everything or without errors ?

I read here http://wiki.apache.org/hadoop/EclipseEnvironment but I did not manage to really compile. However, I want only to make some slightly changes on the scheduler and rerun with my modifications. 

Also I saw some test directories for each scheduler. So another question follows: how I can test only the scheduler with that tests or others made by me without running all Hadoop platform.

It is also possible to simulate machines locally such that recompiling and deploying the code to not take so much time ? My working scenario has 30 nodes. It is
 really necessary whenever I recompile the scheduler to resubmit the code on the nodes or I can somehow simulate locally the nodes and make faster the process of development ?

Thank you very much for any help,
Robert



      
--0-877460507-1295006293=:75099--

From general-return-2775-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 13:41:49 2011
Return-Path: <general-return-2775-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 28221 invoked from network); 14 Jan 2011 13:41:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 13:41:49 -0000
Received: (qmail 24483 invoked by uid 500); 14 Jan 2011 13:41:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24199 invoked by uid 500); 14 Jan 2011 13:41:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24191 invoked by uid 99); 14 Jan 2011 13:41:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 13:41:43 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 13:41:37 +0000
Received: by vws18 with SMTP id 18so972704vws.35
        for <general@hadoop.apache.org>; Fri, 14 Jan 2011 05:41:16 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=mvigzjs2jT8KuJOtnvKN/F/o65EgCq5JWTaTngPYGWg=;
        b=nCZONdGlj254EHlSRpdcG8x+lfdCRVYQy9LtHVJap+0TjOkQQjjivC2s90ihaT70Cl
         yCiyeKD59nj1rbUHFVvrCSzDgLMxZm99r7EZEPC3hlpt35ydJ92FwHVr8UF8+hqpHeV/
         p7igLCx6srIvVBRw2CLDpZ3fmjIlCZ28KYEzs=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=iXcTs1ks8oa0PYeudHeqvAaZZWGFmqGejWVXata417s6rrmih6uqFyzKgpIoZy4YMm
         M0J8MvIa68L25HH+1AX83gALzwneQ64H8IvJAnH33/cxzUxzYtaWntOzNnHX59JxBjl2
         9WE4NDfN9Mw3hwxuy6JQxP2lnao8serjKaa+c=
Received: by 10.220.71.81 with SMTP id g17mr228454vcj.184.1295012475911;
        Fri, 14 Jan 2011 05:41:15 -0800 (PST)
Received: from [10.172.2.25] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id y14sm400323vch.4.2011.01.14.05.41.14
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Fri, 14 Jan 2011 05:41:14 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: triggering automated precommit testing
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <C9549EE6.2E6A1%sureshms@yahoo-inc.com>
Date: Fri, 14 Jan 2011 08:41:12 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <2482641B-D27F-49BF-85EE-CF9D0DF66DFD@Holsman.NET>
References: <C9549EE6.2E6A1%sureshms@yahoo-inc.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

added.. (and Todd as well).

On Jan 13, 2011, at 3:21 PM, Suresh Srinivas wrote:

> Please add me...
>=20
>=20
> On 1/12/11 5:18 PM, "Ian Holsman" <hadoop@holsman.net> wrote:
>=20
> and done.
> anybody else want access?
>=20
> On Jan 12, 2011, at 8:12 PM, Nigel Daley wrote:
>=20
>> I believe ommitters can gain access following this:
>> http://wiki.apache.org/general/Hudson#How_do_I_get_an_account
>> Ian would have to run a command on people.apache.org.
>>=20
>> Nige
>>=20
>> On Jan 12, 2011, at 4:41 PM, Konstantin Shvachko wrote:
>>=20
>>> So, who has the permissions. I don't. Should I?
>>> --Konstantin
>>>=20
>>> On Wed, Jan 12, 2011 at 4:33 PM, Nigel Daley <ndaley@mac.com> wrote:
>>>=20
>>>>> Jakob Homan commented on HDFS-884:
>>>>> ----------------------------------
>>>>>=20
>>>>>> Konstantin, if you're trying to kick a new patch build for this =
you no
>>>> longer move it to "Open" and back to "Patch Available". Instead, =
you must
>>>> upload a new patch. Or, if you have permission, you can =
kickhttps://
>>>> hudson.apache.org/hudson/job/PreCommit-HDFS-Build/ and enter the =
issue
>>>> number.
>>>>>=20
>>>>> That makes me sad.  Is this a new feature or regression?
>>>>=20
>>>> [For everyone's benefit, moving this to general@]
>>>>=20
>>>> Jakob, I referenced the change here: http://tinyurl.com/4crxlvy
>>>> The new system is much more robust partial because it no longer =
relies on
>>>> watching Jira generated emails to determined when issues move into =
Patch
>>>> Available state. There is limited info I can get from the Jira API, =
thus the
>>>> triggering mechanism had to change.
>>>>=20
>>>> Cheers,
>>>> Nige
>>>>=20
>>=20
>=20
>=20


From general-return-2776-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 13:42:49 2011
Return-Path: <general-return-2776-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 28836 invoked from network); 14 Jan 2011 13:42:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 13:42:49 -0000
Received: (qmail 29012 invoked by uid 500); 14 Jan 2011 13:42:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28925 invoked by uid 500); 14 Jan 2011 13:42:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28916 invoked by uid 99); 14 Jan 2011 13:42:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 13:42:44 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 13:42:37 +0000
Received: by vws18 with SMTP id 18so972811vws.35
        for <general@hadoop.apache.org>; Fri, 14 Jan 2011 05:42:15 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=XkJYBOLFjEw99LY/IqfU1LDKGSQbPN5WaEXGXgFfJL8=;
        b=s/jWplwi0TQj4ga3sNBk+k7T3zZL7OME9X/zhFQN2Rvqk2WxpEdG+WIj0jajrkRAW+
         bdgGU2vvKXDaJ6OCckKyQF7R8r4h4/Q6VdWIZTYBEulCvogSYXS+i+OwwC3IUdvjqmIQ
         9rkYC08XPeliQAnL4Ff2TTr7COIJaLx83iWIQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=OwojQWzVt7cvmYOHDO+MHBH5PAQ2s1osJr2KB/smdDGjfGFXuSEA2Z0sywWVS/Cvl9
         x0k7fJFbOj9Cb9ynQl/xWvVkZS482A/IGqV210JqQNCol/glQlud18ya183i/0vW4CuS
         AuiFyulZJB0z6l3lJY4RmLhvXDE7zZdBTGdRY=
Received: by 10.220.181.2 with SMTP id bw2mr278465vcb.48.1295012535375;
        Fri, 14 Jan 2011 05:42:15 -0800 (PST)
Received: from [10.172.2.25] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id y14sm400323vch.4.2011.01.14.05.42.11
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Fri, 14 Jan 2011 05:42:13 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [DISCUSS] Move project split down a level
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <AANLkTi=tuU0pRo12W-9-6xn7rkb5jSveGz0My=VBGG4z@mail.gmail.com>
Date: Fri, 14 Jan 2011 08:42:11 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <B6BDE780-4A72-4355-9182-8EF6DF44AA2C@Holsman.NET>
References: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com> <91A364F8-480F-4323-92F6-15B47D5A3C32@yahoo-inc.com> <88238863-C1CB-4932-9288-40655F225D59@holsman.net> <AANLkTi=tuU0pRo12W-9-6xn7rkb5jSveGz0My=VBGG4z@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

on that note... I propose we discuss un-splitting the project =
altogether.

On Jan 14, 2011, at 3:39 AM, Jakob Homan wrote:

> +1. The project split is a lie.
>=20
> On Fri, Jan 14, 2011 at 12:32 AM, Ian Holsman <hadoop@holsman.net> =
wrote:
>> +1 full agreement.
>>=20
>> I think it will be a pita admin wise (due to how svn authorization is =
set up), so it might slow down creation of a new branch, but its worth =
it.
>>=20
>> ---
>> Ian Holsman
>> AOL Inc
>> Ian.Holsman@teamaol.com
>> (703) 879-3128 / AIM:ianholsman
>>=20
>> it's just a technicality
>>=20
>> On Jan 14, 2011, at 2:25 AM, Eric Baldeschwieler =
<eric14@yahoo-inc.com> wrote:
>>=20
>>> +1
>>>=20
>>> Death to the project split!  Or short of that, anything to tame it.
>>>=20
>>> On Jan 13, 2011, at 10:18 PM, Nigel Daley wrote:
>>>=20
>>>> Folks,
>>>>=20
>>>> As I look more at the impact of the common/MR/HDFS project split on =
what and how we release Hadoop, I feel like the split needs an =
adjustment.  Many folks I've talked to agree that the project split has =
caused us a splitting headache.  I think 1 relatively small change could =
alleviate some of that.
>>>>=20
>>>> CURRENT SVN REPO:
>>>>=20
>>>> hadoop / [common, mapreduce, hdfs] / trunk
>>>> hadoop / [common, mapreduce, hdfs] / branches
>>>>=20
>>>> PROPOSAL:
>>>>=20
>>>> hadoop / trunk / [common, mapreduce, hdfs]
>>>> hadoop / branches / [common, mapreduce, hdfs]
>>>>=20
>>>> We're a long way from releasing these 3 projects independently.  =
Given that, they should be branched and released as a unit.  This SVN =
structure enforces that and provides a more natural place to keep a top =
level build and pkg scripts that operate across all 3 projects.
>>>>=20
>>>> Thoughts?
>>>>=20
>>>> Cheers,
>>>> Nige
>>>=20
>>=20


From general-return-2777-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 14:15:20 2011
Return-Path: <general-return-2777-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46477 invoked from network); 14 Jan 2011 14:15:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 14:15:20 -0000
Received: (qmail 65900 invoked by uid 500); 14 Jan 2011 14:15:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65637 invoked by uid 500); 14 Jan 2011 14:15:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65629 invoked by uid 99); 14 Jan 2011 14:15:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 14:15:14 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 14:15:07 +0000
Received: by vws18 with SMTP id 18so980487vws.35
        for <general@hadoop.apache.org>; Fri, 14 Jan 2011 06:14:45 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=s5I2UV3THvpjrSEtEppe+lNGN6jmc91z6gepeQWyims=;
        b=NDmrrA2QlRpRALCjXAK1KQFuHy/8McyqfGmgdYGvfsU6VIPA3vTADdoar4eEVmiEnG
         26X9BaKu/Pq4OyVzrFVkNAvDAEvHf0Ns3cS5SA9PBwlMWSHEjZsknSq/Kbck8sZeiI80
         ZHySu7dGS4NIOG028UwCKw1FSfwJLgAsO3JrU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=c4tVWi/WJ3MRIGlWwkhOSZrmjCpIO4DEQ3NV99OWAViAuSHPD98ra9/MA4v5eCEHwc
         IQyA39/BieprEAXjO+hIFclmPh43Ioya8ElAD6S1RGkJwx9cVp5SHQ1IBgGV4ZEezlyb
         bIEpwL5u+NNAAqlrREk0OZrF/WSR+V2GVnNRg=
Received: by 10.220.181.135 with SMTP id by7mr239854vcb.258.1295014485748;
        Fri, 14 Jan 2011 06:14:45 -0800 (PST)
Received: from [10.172.2.25] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id q5sm406821vcr.15.2011.01.14.06.14.42
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Fri, 14 Jan 2011 06:14:45 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <7D4397E2-B618-4BD1-8E1E-08B1C598B76F@yahoo-inc.com>
Date: Fri, 14 Jan 2011 09:14:41 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <10BA9684-51FF-4332-A2FD-5F648E0AAF8C@Holsman.NET>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com> <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com> <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org> <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com> <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com> <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com> <AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com> <388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com> <AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com> <04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com> <AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com> <74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com> <AANLkTimLLrfYbaaOzbEV9dcsaMFLeMmdgx73FasdxGhA@mail.gmail.com> <B4C8E6B2-A343-449B-AE63-DE5E3F883ABA@yahoo-inc.com> <36CBB8E6-05D6-4AC0-838C-14B1C2B19CBD@!
  mac.com> <00F14279-2802-4CC3-91AC-481AE257FD8B@yahoo-inc.com> <AANLkTikqbRrU+3+y2nctu7RzKNb7+P8kd=4az+eQHzKW@mail.gmail.com> <58FF3525-B734-44A0-A5C6-45282A23F06F@yahoo-inc.com> <AANLkTik-=ZDac0iEbYuDHeBrRosQ3JNGvRbf7MxexQND@mail.gmail.com> <7D4397E2-B618-4BD1-8E1E-08B1C598B76F@yahoo-inc.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

(with my Apache hat on)
I'm -0.5 on doing this as one big mega-patch and not including append =
(as opposed to a series of smaller patches).

for the following reasons:

1. It encourages bad behavior. We want discussion (and development) to =
happen on the lists, not in some office. By allowing these large =
code-dumps it condones this behavior, and we will likely see it again =
and again. Like it or not, this is not the apache model of open source =
governance.=20

2. There is a risk that some code that is not in a JIRA or separate =
patch creeps in unwittingly. This isn't a major deal per se, but we =
don't really have the proper paper trail, or the documentation on what =
bug it fixed etc etc.

3. Other groups (Facebook for example) are running with their own set of =
patches. They currently have the luxury of examining each individual =
patch to decide if they want to integrate it (and test it) in their =
environment. We are forcing them to do the work of finding the bits they =
want in this huge patch.

4. By not including the append patch, we are making this release =
unusable for a large portion of our community who run hbase.

5. It makes it very hard to test. While It makes me comfortable that it =
has gone through Yahoo!'s QA and is running in their environments, it =
doesn't mean that it will work in other organizations who have different =
workload mixes and software running on them. With one huge patch it =
makes it all or nothing.. either they take the code-drop and perform a =
large QA-integration effort, or they forgo the whole patch together.


**BUT** we have both the Yahoo! & Cloudera guys happy to do it, and to =
spend their time doing it.. so I think having the code-drop will put us =
in a better place then where we are.


BTW, I'd like to point out a discrepancy here:

On another thread discussing hadoop-0.20-append as a separate branch, =
most people agreed that new features shouldn't be added to 0.20, now we =
have a major feature and we are all gung ho for it..=20

--Ian

On Jan 14, 2011, at 2:21 AM, Arun C Murthy wrote:

>=20
> On Jan 13, 2011, at 10:59 PM, Stack wrote:
>=20
>> (Man, it was looking good there for a second when 0.20.100 was about
>> security+append!)
>>=20
>> Good luck w/ the release Arun.
>>=20
>=20
> Thanks!
>=20
>> We might be following your 0.20.100 with a 0.20.200 append.
>>=20
>=20
> Super!
>=20
> Arun


From general-return-2778-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 16:43:13 2011
Return-Path: <general-return-2778-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19827 invoked from network); 14 Jan 2011 16:43:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 16:43:13 -0000
Received: (qmail 19978 invoked by uid 500); 14 Jan 2011 16:43:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19451 invoked by uid 500); 14 Jan 2011 16:43:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19438 invoked by uid 99); 14 Jan 2011 16:43:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 16:43:08 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.102 as permitted sender)
Received: from [17.148.16.102] (HELO asmtpout027.mac.com) (17.148.16.102)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 16:43:00 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp027.mac.com (Oracle Communications Messaging Exchange Server 7u4-18.01
 64bit (built Jul 15 2010)) with ESMTPSA id <0LF0008UZUE37D20@asmtp027.mac.com>
 for general@hadoop.apache.org; Fri, 14 Jan 2011 08:42:04 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-14_08:2011-01-14,2011-01-14,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101140076
Subject: Re: triggering automated precommit testing
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <2482641B-D27F-49BF-85EE-CF9D0DF66DFD@Holsman.NET>
Date: Fri, 14 Jan 2011 08:42:03 -0800
Message-id: <45973863-6027-4AD0-ABFA-279C3EF8BC48@mac.com>
References: <C9549EE6.2E6A1%sureshms@yahoo-inc.com>
 <2482641B-D27F-49BF-85EE-CF9D0DF66DFD@Holsman.NET>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

Ian, Todd, Cos, Konst, Jakob, Anyone else given this access:

PLEASE do NOT kill any builds from Hudson (little red x box) even if they appear hung.  Hudson does not kill the underlying processes properly so they get left behind and can silently affect subsequent jobs.  Instead, email this list that a job appears to be hung.  Giri or I can then clean up the job properly. 

Thanks,
nige

On Jan 14, 2011, at 5:41 AM, Ian Holsman wrote:

> added.. (and Todd as well).
> 
> On Jan 13, 2011, at 3:21 PM, Suresh Srinivas wrote:
> 
>> Please add me...
>> 
>> 
>> On 1/12/11 5:18 PM, "Ian Holsman" <hadoop@holsman.net> wrote:
>> 
>> and done.
>> anybody else want access?
>> 
>> On Jan 12, 2011, at 8:12 PM, Nigel Daley wrote:
>> 
>>> I believe ommitters can gain access following this:
>>> http://wiki.apache.org/general/Hudson#How_do_I_get_an_account
>>> Ian would have to run a command on people.apache.org.
>>> 
>>> Nige
>>> 
>>> On Jan 12, 2011, at 4:41 PM, Konstantin Shvachko wrote:
>>> 
>>>> So, who has the permissions. I don't. Should I?
>>>> --Konstantin
>>>> 
>>>> On Wed, Jan 12, 2011 at 4:33 PM, Nigel Daley <ndaley@mac.com> wrote:
>>>> 
>>>>>> Jakob Homan commented on HDFS-884:
>>>>>> ----------------------------------
>>>>>> 
>>>>>>> Konstantin, if you're trying to kick a new patch build for this you no
>>>>> longer move it to "Open" and back to "Patch Available". Instead, you must
>>>>> upload a new patch. Or, if you have permission, you can kickhttps://
>>>>> hudson.apache.org/hudson/job/PreCommit-HDFS-Build/ and enter the issue
>>>>> number.
>>>>>> 
>>>>>> That makes me sad.  Is this a new feature or regression?
>>>>> 
>>>>> [For everyone's benefit, moving this to general@]
>>>>> 
>>>>> Jakob, I referenced the change here: http://tinyurl.com/4crxlvy
>>>>> The new system is much more robust partial because it no longer relies on
>>>>> watching Jira generated emails to determined when issues move into Patch
>>>>> Available state. There is limited info I can get from the Jira API, thus the
>>>>> triggering mechanism had to change.
>>>>> 
>>>>> Cheers,
>>>>> Nige
>>>>> 
>>> 
>> 
>> 
> 


From general-return-2779-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 16:45:27 2011
Return-Path: <general-return-2779-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 20508 invoked from network); 14 Jan 2011 16:45:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 16:45:26 -0000
Received: (qmail 23714 invoked by uid 500); 14 Jan 2011 16:45:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23604 invoked by uid 500); 14 Jan 2011 16:45:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23595 invoked by uid 99); 14 Jan 2011 16:45:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 16:45:23 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.102 as permitted sender)
Received: from [17.148.16.102] (HELO asmtpout027.mac.com) (17.148.16.102)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 16:45:14 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp027.mac.com (Oracle Communications Messaging Exchange Server 7u4-18.01
 64bit (built Jul 15 2010)) with ESMTPSA id <0LF00084WUI68330@asmtp027.mac.com>
 for general@hadoop.apache.org; Fri, 14 Jan 2011 08:44:31 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-14_08:2011-01-14,2011-01-14,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101140076
Subject: Re: triggering automated precommit testing
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <AANLkTikc_-eeChjjY6GkHJqih3OZMRZ15a=JTvY5Z9mc@mail.gmail.com>
Date: Fri, 14 Jan 2011 08:44:30 -0800
Message-id: <7B0FFEF0-56CE-4AD6-AD74-291C49AEB0E6@mac.com>
References: <5125873.329641294877388152.JavaMail.jira@thor>
 <D570B102-7A4F-42E2-BB9A-B3CFC72428CC@mac.com>
 <AANLkTikc_-eeChjjY6GkHJqih3OZMRZ15a=JTvY5Z9mc@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

Todd, please file a Jira in Common against test component.  FWIW, I fear the precommit integration with Jira will need some amount of work as Apache moves to Jira 4.2 in the coming weeks.

Nige

On Jan 14, 2011, at 12:57 AM, Todd Lipcon wrote:

> Hey Nigel,
> 
> Would there be any way to add a feature where we can make some special
> comment on the JIRA that would trigger a hudson retest? There are a lot of
> really old patches out on the JIRA that would be worth re-testing against
> trunk, and it's a pain to download and re-attach.
> 
> I'm thinking a comment with a special token like "@Hudson.Test"
> 
> Failing that, Ian, can you add me to the Hudson list?
> 
> -Todd
> 
> On Wed, Jan 12, 2011 at 4:33 PM, Nigel Daley <ndaley@mac.com> wrote:
> 
>>> Jakob Homan commented on HDFS-884:
>>> ----------------------------------
>>> 
>>>> Konstantin, if you're trying to kick a new patch build for this you no
>> longer move it to "Open" and back to "Patch Available". Instead, you must
>> upload a new patch. Or, if you have permission, you can kickhttps://
>> hudson.apache.org/hudson/job/PreCommit-HDFS-Build/ and enter the issue
>> number.
>>> 
>>> That makes me sad.  Is this a new feature or regression?
>> 
>> [For everyone's benefit, moving this to general@]
>> 
>> Jakob, I referenced the change here: http://tinyurl.com/4crxlvy
>> The new system is much more robust partial because it no longer relies on
>> watching Jira generated emails to determined when issues move into Patch
>> Available state. There is limited info I can get from the Jira API, thus the
>> triggering mechanism had to change.
>> 
>> Cheers,
>> Nige
>> 
> 
> 
> 
> -- 
> Todd Lipcon
> Software Engineer, Cloudera


From general-return-2780-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 16:52:08 2011
Return-Path: <general-return-2780-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23041 invoked from network); 14 Jan 2011 16:52:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 16:52:08 -0000
Received: (qmail 36847 invoked by uid 500); 14 Jan 2011 16:52:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36556 invoked by uid 500); 14 Jan 2011 16:52:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36545 invoked by uid 99); 14 Jan 2011 16:52:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 16:52:03 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.59.212] (HELO qmta14.westchester.pa.mail.comcast.net) (76.96.59.212)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 16:51:55 +0000
Received: from omta01.westchester.pa.mail.comcast.net ([76.96.62.11])
	by qmta14.westchester.pa.mail.comcast.net with comcast
	id vUia1f0010EZKEL5EUrbtF; Fri, 14 Jan 2011 16:51:35 +0000
Received: from [10.72.184.42] ([209.131.62.113])
	by omta01.westchester.pa.mail.comcast.net with comcast
	id vUrS1f00H2SbwD53MUrVir; Fri, 14 Jan 2011 16:51:33 +0000
Message-Id: <4A1492FF-C4B5-4949-B197-53B3A3B1BE29@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSS] Move project split down a level
Date: Fri, 14 Jan 2011 08:51:26 -0800
References: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com>
X-Mailer: Apple Mail (2.936)


On Jan 13, 2011, at 10:18 PM, Nigel Daley wrote:

> Folks,
>
> As I look more at the impact of the common/MR/HDFS project split on  
> what and how we release Hadoop, I feel like the split needs an  
> adjustment.  Many folks I've talked to agree that the project split  
> has caused us a splitting headache.  I think 1 relatively small  
> change could alleviate some of that.
>
> CURRENT SVN REPO:
>
> hadoop / [common, mapreduce, hdfs] / trunk
> hadoop / [common, mapreduce, hdfs] / branches
>
> PROPOSAL:
>
> hadoop / trunk / [common, mapreduce, hdfs]
> hadoop / branches / [common, mapreduce, hdfs]


Moving the source trees back together is ok, but will cause a fair  
amount of churn for those of us that depend on the git versions of the  
repository. Using Todd's hack may be able to fix it again at least for  
each individual user.

I assume you meant to propose:

hadoop/ {trunk, branches/*, tags/* } / {common, hdfs, mapreduce}

which means that you can make checkouts, branches and tags with a  
single command. Your proposal as stated would break all of the tools  
that count on standard layouts of subversion repositories, such as the  
subversion to git gateways and eclipse.

We currently have other stuff at the top level of hadoop: hive, logos,  
nightly, pig, site, and zookeeper. Clearly hive, pig, and zookeeper  
should be removed. The others are just versioned and aren't branched.  
I'm fine with leaving them at the top level as "extra" bits, but it  
should be decided.

-- Owen

From general-return-2781-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 16:52:37 2011
Return-Path: <general-return-2781-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23275 invoked from network); 14 Jan 2011 16:52:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 16:52:37 -0000
Received: (qmail 38390 invoked by uid 500); 14 Jan 2011 16:52:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38168 invoked by uid 500); 14 Jan 2011 16:52:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 28334 invoked by uid 99); 14 Jan 2011 16:47:10 -0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
Sender: niels@basj.es
X-Originating-IP: [212.121.118.65]
In-Reply-To: <AANLkTikUYwYR+=ZLboQMU+Dv_9xGs1vuCtfsFFHZSM2B@mail.gmail.com>
References: <AANLkTikUYwYR+=ZLboQMU+Dv_9xGs1vuCtfsFFHZSM2B@mail.gmail.com>
Date: Fri, 14 Jan 2011 17:46:40 +0100
X-Google-Sender-Auth: 9lD8DWE7xdVdFwwCmADYBRi7rt8
Message-ID: <AANLkTinBd7UEjjN7BytDvxqgD7uFZM1HdXS5ZkxbP7zd@mail.gmail.com>
Subject: Re: Restricting number of records from map output
From: Niels Basjes <Niels@basjes.nl>
To: common-user@hadoop.apache.org
Cc: common-issues@hadoop.apache.org, general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

> I have a sort job consisting of only the Mapper (no Reducer) task. I want my
> results to contain only the top n records. Is there any way of restricting
> the number of records that are emitted by the Mappers?
>
> Basically I am looking to see if there is an equivalent of achieving
> the behavior similar to LIMIT in SQL queries.

I think I understand your goal. However the question is toward (what I
think) is the wrong solution.

A mapper gets 1 record as input and only knows about that one record.
There is no way to limit there.

If you implement a simple reducer you can very easily let is stop
reading the input iterator after N records and limit the output in
that way.

Doing it in the reducer also allows you to easily add a concept of
"Top N" by using the "Secondary Sort" trick to sort the input before
it arrives at the reducer.

HTH

Niels Basjes

From general-return-2782-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 16:59:14 2011
Return-Path: <general-return-2782-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25029 invoked from network); 14 Jan 2011 16:59:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 16:59:14 -0000
Received: (qmail 50854 invoked by uid 500); 14 Jan 2011 16:59:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50495 invoked by uid 500); 14 Jan 2011 16:59:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50487 invoked by uid 99); 14 Jan 2011 16:59:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 16:59:10 +0000
X-ASF-Spam-Status: No, hits=-1997.8 required=10.0
	tests=ALL_TRUSTED,HTML_MESSAGE,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 14 Jan 2011 16:59:10 +0000
Received: (qmail 24924 invoked by uid 99); 14 Jan 2011 16:58:49 -0000
Received: from localhost.apache.org (HELO mail-ey0-f176.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 16:58:49 +0000
Received: by eyz10 with SMTP id 10so1479929eyz.35
        for <multiple recipients>; Fri, 14 Jan 2011 08:58:47 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.127.89 with SMTP id f25mr782785bks.143.1295024327451; Fri,
 14 Jan 2011 08:58:47 -0800 (PST)
Received: by 10.204.61.73 with HTTP; Fri, 14 Jan 2011 08:58:47 -0800 (PST)
In-Reply-To: <461412.75099.qm@web112109.mail.gq1.yahoo.com>
References: <461412.75099.qm@web112109.mail.gq1.yahoo.com>
Date: Fri, 14 Jan 2011 08:58:47 -0800
Message-ID: <AANLkTimHCaCFYuNLTC8Wz7smoP-JQHx0Dfwbtkmzt8v3@mail.gmail.com>
Subject: Re: Ease of development in Hadoop
From: "Owen O'Malley" <omalley@apache.org>
To: common-dev@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6db293403640b0499d1595d

--0016e6db293403640b0499d1595d
Content-Type: text/plain; charset=UTF-8

Redirecting to mapreduce-dev. As described in
http://hadoop.apache.org/mailing_lists.html, general isn't for user/dev
questions.

On Fri, Jan 14, 2011 at 3:58 AM, Grandl Robert <rgrandl@yahoo.com> wrote:

> I want to make some minor modifications in FairShare scheduler and
> recompile it such that to use it as before. Is there an easy way to do it
> without spending much time on recompiling everything or without errors ?


Assuming you are working on a 0.20 release, get the source and run "ant
compile". That will create all of the jars for you including the contrib
ones.

-- Owen

--0016e6db293403640b0499d1595d--

From general-return-2783-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 17:18:23 2011
Return-Path: <general-return-2783-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38430 invoked from network); 14 Jan 2011 17:18:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 17:18:22 -0000
Received: (qmail 71366 invoked by uid 500); 14 Jan 2011 17:18:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 71086 invoked by uid 500); 14 Jan 2011 17:18:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 71077 invoked by uid 99); 14 Jan 2011 17:18:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 17:18:17 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 17:18:11 +0000
Received: by iyb26 with SMTP id 26so3098094iyb.35
        for <general@hadoop.apache.org>; Fri, 14 Jan 2011 09:17:50 -0800 (PST)
Received: by 10.231.11.4 with SMTP id r4mr947764ibr.146.1295025470256; Fri, 14
 Jan 2011 09:17:50 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.115.8 with HTTP; Fri, 14 Jan 2011 09:17:30 -0800 (PST)
In-Reply-To: <4A1492FF-C4B5-4949-B197-53B3A3B1BE29@apache.org>
References: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com> <4A1492FF-C4B5-4949-B197-53B3A3B1BE29@apache.org>
From: Todd Lipcon <todd@cloudera.com>
Date: Fri, 14 Jan 2011 09:17:30 -0800
Message-ID: <AANLkTimYUTULEMpiNfNCvqv604JqMeR7a4AdXivxvfic@mail.gmail.com>
Subject: Re: [DISCUSS] Move project split down a level
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00221532ca2821398b0499d19db5

--00221532ca2821398b0499d19db5
Content-Type: text/plain; charset=ISO-8859-1

On Fri, Jan 14, 2011 at 8:51 AM, Owen O'Malley <omalley@apache.org> wrote:

>
> On Jan 13, 2011, at 10:18 PM, Nigel Daley wrote:
>
>  Folks,
>>
>> As I look more at the impact of the common/MR/HDFS project split on what
>> and how we release Hadoop, I feel like the split needs an adjustment.  Many
>> folks I've talked to agree that the project split has caused us a splitting
>> headache.  I think 1 relatively small change could alleviate some of that.
>>
>> CURRENT SVN REPO:
>>
>> hadoop / [common, mapreduce, hdfs] / trunk
>> hadoop / [common, mapreduce, hdfs] / branches
>>
>> PROPOSAL:
>>
>> hadoop / trunk / [common, mapreduce, hdfs]
>> hadoop / branches / [common, mapreduce, hdfs]
>>
>
>
> Moving the source trees back together is ok, but will cause a fair amount
> of churn for those of us that depend on the git versions of the repository.
> Using Todd's hack may be able to fix it again at least for each individual
> user.
>

Yep, I think we can set this up in a reasonable way with grafts. I'm happy
to write a little shell script we can put on the wiki for git users that
would make the history look sane.

Depending on how the git mirrors work, we might even be able to get the
official git.apache.org one to work, too. If we decide to go forward with
this plan I'll talk to the relevant INFRA folks and see about that.

-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

--00221532ca2821398b0499d19db5--

From general-return-2784-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 17:20:59 2011
Return-Path: <general-return-2784-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39227 invoked from network); 14 Jan 2011 17:20:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 17:20:58 -0000
Received: (qmail 74826 invoked by uid 500); 14 Jan 2011 17:20:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74599 invoked by uid 500); 14 Jan 2011 17:20:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74591 invoked by uid 99); 14 Jan 2011 17:20:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 17:20:54 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 17:20:48 +0000
Received: by ewy20 with SMTP id 20so1525613ewy.35
        for <general@hadoop.apache.org>; Fri, 14 Jan 2011 09:20:27 -0800 (PST)
Received: by 10.216.48.5 with SMTP id u5mr758786web.96.1295025627359; Fri, 14
 Jan 2011 09:20:27 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.59.13 with HTTP; Fri, 14 Jan 2011 09:20:06 -0800 (PST)
In-Reply-To: <10BA9684-51FF-4332-A2FD-5F648E0AAF8C@Holsman.NET>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
 <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
 <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
 <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
 <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com> <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
 <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com> <AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com>
 <388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com> <AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com>
 <04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com> <AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com>
 <74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com> <AANLkTimLLrfYbaaOzbEV9dcsaMFLeMmdgx73FasdxGhA@mail.gmail.com>
 <B4C8E6B2-A343-449B-AE63-DE5E3F883ABA@yahoo-inc.com> <00F14279-2802-4CC3-91AC-481AE257FD8B@yahoo-inc.com>
 <AANLkTikqbRrU+3+y2nctu7RzKNb7+P8kd=4az+eQHzKW@mail.gmail.com>
 <58FF3525-B734-44A0-A5C6-45282A23F06F@yahoo-inc.com> <AANLkTik-=ZDac0iEbYuDHeBrRosQ3JNGvRbf7MxexQND@mail.gmail.com>
 <7D4397E2-B618-4BD1-8E1E-08B1C598B76F@yahoo-inc.com> <10BA9684-51FF-4332-A2FD-5F648E0AAF8C@Holsman.NET>
From: Konstantin Boudnik <cos@apache.org>
Date: Fri, 14 Jan 2011 09:20:06 -0800
X-Google-Sender-Auth: gpGz8wA_P6M2sYvO3DrfFP7CvSg
Message-ID: <AANLkTi=0pFkfM+0jMq+f-X7w9tMGd7_oovh0QfUrX6Mj@mail.gmail.com>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

I tend to second most of Ian's points here.

On Fri, Jan 14, 2011 at 06:14, Ian Holsman <hadoop@holsman.net> wrote:
> (with my Apache hat on)
> I'm -0.5 on doing this as one big mega-patch and not including append (as opposed to a series of smaller patches).

#1: we are creating a precedent of a "brain-dump" here. Although, it
isn't the first one in the history of OSS. Infamous Apple "patch" to
OpenBSD is another one ;)

#2: How to spell 'back door' any one?

#5: "almost 10 internal releases" Arun has mentioned above might be,
perhaps, considered as a great quality control effort. Also, not to
mention virtual impossibility to create a test plan to validate a
giant features patch.

> BTW, I'd like to point out a discrepancy here:
>
> On another thread discussing hadoop-0.20-append as a separate branch, most people agreed that new features shouldn't be added to 0.20, now we have a major feature and we are all gung ho for it..

And this ^^^

But, hey I guess it's totally worth it!
  Cos

> --Ian
>
> On Jan 14, 2011, at 2:21 AM, Arun C Murthy wrote:
>
>>
>> On Jan 13, 2011, at 10:59 PM, Stack wrote:
>>
>>> (Man, it was looking good there for a second when 0.20.100 was about
>>> security+append!)
>>>
>>> Good luck w/ the release Arun.
>>>
>>
>> Thanks!
>>
>>> We might be following your 0.20.100 with a 0.20.200 append.
>>>
>>
>> Super!
>>
>> Arun
>
>

From general-return-2785-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 17:33:38 2011
Return-Path: <general-return-2785-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46023 invoked from network); 14 Jan 2011 17:33:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 17:33:38 -0000
Received: (qmail 93415 invoked by uid 500); 14 Jan 2011 17:33:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93045 invoked by uid 500); 14 Jan 2011 17:33:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93026 invoked by uid 99); 14 Jan 2011 17:33:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 17:33:33 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.102 as permitted sender)
Received: from [17.148.16.102] (HELO asmtpout027.mac.com) (17.148.16.102)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 17:33:27 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp027.mac.com (Oracle Communications Messaging Exchange Server 7u4-18.01
 64bit (built Jul 15 2010)) with ESMTPSA id <0LF0009RSWQRPT20@asmtp027.mac.com>
 for general@hadoop.apache.org; Fri, 14 Jan 2011 09:32:53 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-14_08:2011-01-14,2011-01-14,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101140086
Sun-Java-System-SMTP-Warning: Lines longer than SMTP allows found and wrapped.
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <AANLkTi=0pFkfM+0jMq+f-X7w9tMGd7_oovh0QfUrX6Mj@mail.gmail.com>
Date: Fri, 14 Jan 2011 09:32:51 -0800
Message-id: <2C71D87C-D7BF-4993-B0F9-05E455E9CC77@mac.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
 <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
 <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
 <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
 <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
 <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
 <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
 <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
 <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
 <AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com>
 <388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com>
 <AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com>
 <04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com>
 <AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com>
 <74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com>
 <AANLkTimLLrfYbaaOzbEV9dcsaMFLeMmdgx73FasdxGhA@mail.gmail.com>
 <B4C8E6B2-A343-449B-AE63-DE5E3F883ABA@yahoo-inc.com>
 <00F14279-2802-4CC3-91AC-481AE257FD8B@yahoo-inc.com>
 <AANLkTikqbRrU+3+y2nctu7RzKNb7+P8kd=4az+eQHzKW@mail.gmail.com>
 <58FF3525-B734-44A0-A5C6-45282A23F06F@yahoo-inc.com>
 <AANLkTik-=ZDac0iEbYuDHeBrRosQ3JNGvRbf7MxexQND@mail.gmail.com>
 <7D4397E2-B618-4BD1-8E1E-08B1C598B76F@yahoo-inc.com>
 <10BA9684-51FF-4332-A2FD-5F648E0AAF8C@Holsman.NET>
 <AANLkTi=0pFkfM+0jMq+f-X7w9tMGd7_oovh0QfUrX6Mj@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

Yup, I'll say it again.  The process ain't perfect but it's good enough IMO. Thank you Yahoo! for your contribution.

Clearly these patch will need review before commit when going into trunk. 

Let's move on to 0.22.

Nige

On Jan 14, 2011, at 9:20 AM, Konstantin Boudnik wrote:

> I tend to second most of Ian's points here.
> 
> On Fri, Jan 14, 2011 at 06:14, Ian Holsman <hadoop@holsman.net> wrote:
>> (with my Apache hat on)
>> I'm -0.5 on doing this as one big mega-patch and not including append (as opposed to a series of smaller patches).
> 
> #1: we are creating a precedent of a "brain-dump" here. Although, it
> isn't the first one in the history of OSS. Infamous Apple "patch" to
> OpenBSD is another one ;)
> 
> #2: How to spell 'back door' any one?
> 
> #5: "almost 10 internal releases" Arun has mentioned above might be,
> perhaps, considered as a great quality control effort. Also, not to
> mention virtual impossibility to create a test plan to validate a
> giant features patch.
> 
>> BTW, I'd like to point out a discrepancy here:
>> 
>> On another thread discussing hadoop-0.20-append as a separate branch, most people agreed that new features shouldn't be added to 0.20, now we have a major feature and we are all gung ho for it..
> 
> And this ^^^
> 
> But, hey I guess it's totally worth it!
>  Cos
> 
>> --Ian
>> 
>> On Jan 14, 2011, at 2:21 AM, Arun C Murthy wrote:
>> 
>>> 
>>> On Jan 13, 2011, at 10:59 PM, Stack wrote:
>>> 
>>>> (Man, it was looking good there for a second when 0.20.100 was about
>>>> security+append!)
>>>> 
>>>> Good luck w/ the release Arun.
>>>> 
>>> 
>>> Thanks!
>>> 
>>>> We might be following your 0.20.100 with a 0.20.200 append.
>>>> 
>>> 
>>> Super!
>>> 
>>> Arun
>> 
>> 


From general-return-2786-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 17:33:43 2011
Return-Path: <general-return-2786-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46124 invoked from network); 14 Jan 2011 17:33:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 17:33:43 -0000
Received: (qmail 94513 invoked by uid 500); 14 Jan 2011 17:33:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94438 invoked by uid 500); 14 Jan 2011 17:33:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94430 invoked by uid 99); 14 Jan 2011 17:33:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 17:33:40 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 17:33:34 +0000
Received: from SP2-EX07CAS03.ds.corp.yahoo.com (sp2-ex07cas03.corp.sp2.yahoo.com [98.137.59.35])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0EHWg1h040580;
	Fri, 14 Jan 2011 09:32:42 -0800 (PST)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by
 SP2-EX07CAS03.ds.corp.yahoo.com ([98.137.59.35]) with mapi; Fri, 14 Jan 2011
 09:32:42 -0800
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 14 Jan 2011 09:32:33 -0800
Subject: Re: [DISCUSS] Move project split down a level
Thread-Topic: [DISCUSS] Move project split down a level
Thread-Index: Acu0EQvvnMvQxF6KQEGPJHnE+mPMGg==
Message-ID: <669E9C98-C113-49B5-9851-4524EA936CED@yahoo-inc.com>
References: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com>
 <91A364F8-480F-4323-92F6-15B47D5A3C32@yahoo-inc.com>
 <88238863-C1CB-4932-9288-40655F225D59@holsman.net>
 <AANLkTi=tuU0pRo12W-9-6xn7rkb5jSveGz0My=VBGG4z@mail.gmail.com>
 <B6BDE780-4A72-4355-9182-8EF6DF44AA2C@Holsman.NET>
In-Reply-To: <B6BDE780-4A72-4355-9182-8EF6DF44AA2C@Holsman.NET>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

I'm a huge supporter of the idea. On a related note, we've been looking for=
 the right time to mavenize. Maybe we can do both together. We could pitch =
in a bunch of work on both if we could get the timing right.=20

We've got a huge batch of commits in flight now, but if we can find somethi=
ng that satisfied the 22 crowd after we sync in, we'd be happy to pitch in =
on unsplit and/or maven.=20

---
E14 - via iPhone

On Jan 14, 2011, at 5:43 AM, "Ian Holsman" <hadoop@holsman.net> wrote:

> on that note... I propose we discuss un-splitting the project altogether.
>=20
> On Jan 14, 2011, at 3:39 AM, Jakob Homan wrote:
>=20
>> +1. The project split is a lie.
>>=20
>> On Fri, Jan 14, 2011 at 12:32 AM, Ian Holsman <hadoop@holsman.net> wrote=
:
>>> +1 full agreement.
>>>=20
>>> I think it will be a pita admin wise (due to how svn authorization is s=
et up), so it might slow down creation of a new branch, but its worth it.
>>>=20
>>> ---
>>> Ian Holsman
>>> AOL Inc
>>> Ian.Holsman@teamaol.com
>>> (703) 879-3128 / AIM:ianholsman
>>>=20
>>> it's just a technicality
>>>=20
>>> On Jan 14, 2011, at 2:25 AM, Eric Baldeschwieler <eric14@yahoo-inc.com>=
 wrote:
>>>=20
>>>> +1
>>>>=20
>>>> Death to the project split!  Or short of that, anything to tame it.
>>>>=20
>>>> On Jan 13, 2011, at 10:18 PM, Nigel Daley wrote:
>>>>=20
>>>>> Folks,
>>>>>=20
>>>>> As I look more at the impact of the common/MR/HDFS project split on w=
hat and how we release Hadoop, I feel like the split needs an adjustment.  =
Many folks I've talked to agree that the project split has caused us a spli=
tting headache.  I think 1 relatively small change could alleviate some of =
that.
>>>>>=20
>>>>> CURRENT SVN REPO:
>>>>>=20
>>>>> hadoop / [common, mapreduce, hdfs] / trunk
>>>>> hadoop / [common, mapreduce, hdfs] / branches
>>>>>=20
>>>>> PROPOSAL:
>>>>>=20
>>>>> hadoop / trunk / [common, mapreduce, hdfs]
>>>>> hadoop / branches / [common, mapreduce, hdfs]
>>>>>=20
>>>>> We're a long way from releasing these 3 projects independently.  Give=
n that, they should be branched and released as a unit.  This SVN structure=
 enforces that and provides a more natural place to keep a top level build =
and pkg scripts that operate across all 3 projects.
>>>>>=20
>>>>> Thoughts?
>>>>>=20
>>>>> Cheers,
>>>>> Nige
>>>>=20
>>>=20
>=20

From general-return-2787-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 17:34:27 2011
Return-Path: <general-return-2787-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46667 invoked from network); 14 Jan 2011 17:34:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 17:34:26 -0000
Received: (qmail 99255 invoked by uid 500); 14 Jan 2011 17:34:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99186 invoked by uid 500); 14 Jan 2011 17:34:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 99178 invoked by uid 99); 14 Jan 2011 17:34:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 17:34:23 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 17:34:14 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0EHXXYe041892;
	Fri, 14 Jan 2011 09:33:33 -0800 (PST)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by
 SP2-EX07CAS01.ds.corp.yahoo.com ([98.137.59.37]) with mapi; Fri, 14 Jan 2011
 09:33:33 -0800
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 14 Jan 2011 09:33:25 -0800
Subject: Re: [DISCUSS] Move project split down a level
Thread-Topic: [DISCUSS] Move project split down a level
Thread-Index: Acu0ESpXLD2YI+41TTC5wR87X9Z6iQ==
Message-ID: <6E1C8329-82B6-4332-AE30-60AE57D19FF8@yahoo-inc.com>
References: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com>
 <4A1492FF-C4B5-4949-B197-53B3A3B1BE29@apache.org>
 <AANLkTimYUTULEMpiNfNCvqv604JqMeR7a4AdXivxvfic@mail.gmail.com>
In-Reply-To: <AANLkTimYUTULEMpiNfNCvqv604JqMeR7a4AdXivxvfic@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Cool!

---
E14 - via iPhone

On Jan 14, 2011, at 9:18 AM, "Todd Lipcon" <todd@cloudera.com> wrote:

> On Fri, Jan 14, 2011 at 8:51 AM, Owen O'Malley <omalley@apache.org> wrote=
:
>=20
>>=20
>> On Jan 13, 2011, at 10:18 PM, Nigel Daley wrote:
>>=20
>> Folks,
>>>=20
>>> As I look more at the impact of the common/MR/HDFS project split on wha=
t
>>> and how we release Hadoop, I feel like the split needs an adjustment.  =
Many
>>> folks I've talked to agree that the project split has caused us a split=
ting
>>> headache.  I think 1 relatively small change could alleviate some of th=
at.
>>>=20
>>> CURRENT SVN REPO:
>>>=20
>>> hadoop / [common, mapreduce, hdfs] / trunk
>>> hadoop / [common, mapreduce, hdfs] / branches
>>>=20
>>> PROPOSAL:
>>>=20
>>> hadoop / trunk / [common, mapreduce, hdfs]
>>> hadoop / branches / [common, mapreduce, hdfs]
>>>=20
>>=20
>>=20
>> Moving the source trees back together is ok, but will cause a fair amoun=
t
>> of churn for those of us that depend on the git versions of the reposito=
ry.
>> Using Todd's hack may be able to fix it again at least for each individu=
al
>> user.
>>=20
>=20
> Yep, I think we can set this up in a reasonable way with grafts. I'm happ=
y
> to write a little shell script we can put on the wiki for git users that
> would make the history look sane.
>=20
> Depending on how the git mirrors work, we might even be able to get the
> official git.apache.org one to work, too. If we decide to go forward with
> this plan I'll talk to the relevant INFRA folks and see about that.
>=20
> -Todd
> --=20
> Todd Lipcon
> Software Engineer, Cloudera

From general-return-2788-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 17:36:37 2011
Return-Path: <general-return-2788-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47324 invoked from network); 14 Jan 2011 17:36:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 17:36:37 -0000
Received: (qmail 2786 invoked by uid 500); 14 Jan 2011 17:36:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2529 invoked by uid 500); 14 Jan 2011 17:36:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2521 invoked by uid 99); 14 Jan 2011 17:36:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 17:36:33 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of tom.e.white@gmail.com designates 209.85.216.176 as permitted sender)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 17:36:26 +0000
Received: by qyk10 with SMTP id 10so3228905qyk.14
        for <general@hadoop.apache.org>; Fri, 14 Jan 2011 09:36:06 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type:content-transfer-encoding;
        bh=dOW6VGLdAkbelWrBZtCoyy7X3EwMbTuw4BAgiZUQOtE=;
        b=mkYgswduH2yN9Cv/PYlzwpJutoINwIj+NGL9RHAZyAbjnbeM+cYdpiUfXVqHucbTmu
         c1ZBh0m2XOuG63QYmbuQ59xA46FyHIgWj/OnasdKyQAa5BCMJB/KMAzU4tfhAGkyaNzu
         xSBk116+L7nThdx1b1DRexlMlU818N1etL71o=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=trbwTGGNgVUn6X3Qm23iRgE09iOHTvQ0svwOUsfso+/cEEsTbbNKNVIxUtRTsnDUJ8
         n1t8zXUkc83T4fNHg7BwzLAl0b6coWhpWI0+W1dEMd9fPoox9RT7ADLoCYO8Fq+prfpx
         WsZXCPceEy2c3r4qRKIl5/GZXZxd5qRUrv8BU=
Received: by 10.229.110.10 with SMTP id l10mr918244qcp.56.1295026565303; Fri,
 14 Jan 2011 09:36:05 -0800 (PST)
MIME-Version: 1.0
Received: by 10.229.188.69 with HTTP; Fri, 14 Jan 2011 09:35:45 -0800 (PST)
In-Reply-To: <AANLkTimYUTULEMpiNfNCvqv604JqMeR7a4AdXivxvfic@mail.gmail.com>
References: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com> <4A1492FF-C4B5-4949-B197-53B3A3B1BE29@apache.org>
 <AANLkTimYUTULEMpiNfNCvqv604JqMeR7a4AdXivxvfic@mail.gmail.com>
From: Tom White <tom.e.white@gmail.com>
Date: Fri, 14 Jan 2011 09:35:45 -0800
Message-ID: <AANLkTi=WA5aXiK2hZ58ysvQ0md1aP4NNmj+2T7b3mKej@mail.gmail.com>
Subject: Re: [DISCUSS] Move project split down a level
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

+1 to Owen's modified layout.

The current layout means that every svn operation made during a
release has to be carried out three times which increases the amount
of work and the chances of a mistake.

Cheers
Tom

On Fri, Jan 14, 2011 at 9:17 AM, Todd Lipcon <todd@cloudera.com> wrote:
> On Fri, Jan 14, 2011 at 8:51 AM, Owen O'Malley <omalley@apache.org> wrote=
:
>
>>
>> On Jan 13, 2011, at 10:18 PM, Nigel Daley wrote:
>>
>> =A0Folks,
>>>
>>> As I look more at the impact of the common/MR/HDFS project split on wha=
t
>>> and how we release Hadoop, I feel like the split needs an adjustment. =
=A0Many
>>> folks I've talked to agree that the project split has caused us a split=
ting
>>> headache. =A0I think 1 relatively small change could alleviate some of =
that.
>>>
>>> CURRENT SVN REPO:
>>>
>>> hadoop / [common, mapreduce, hdfs] / trunk
>>> hadoop / [common, mapreduce, hdfs] / branches
>>>
>>> PROPOSAL:
>>>
>>> hadoop / trunk / [common, mapreduce, hdfs]
>>> hadoop / branches / [common, mapreduce, hdfs]
>>>
>>
>>
>> Moving the source trees back together is ok, but will cause a fair amoun=
t
>> of churn for those of us that depend on the git versions of the reposito=
ry.
>> Using Todd's hack may be able to fix it again at least for each individu=
al
>> user.
>>
>
> Yep, I think we can set this up in a reasonable way with grafts. I'm happ=
y
> to write a little shell script we can put on the wiki for git users that
> would make the history look sane.
>
> Depending on how the git mirrors work, we might even be able to get the
> official git.apache.org one to work, too. If we decide to go forward with
> this plan I'll talk to the relevant INFRA folks and see about that.
>
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

From general-return-2789-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 17:45:14 2011
Return-Path: <general-return-2789-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50069 invoked from network); 14 Jan 2011 17:45:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 17:45:13 -0000
Received: (qmail 13044 invoked by uid 500); 14 Jan 2011 17:45:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12590 invoked by uid 500); 14 Jan 2011 17:45:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12582 invoked by uid 99); 14 Jan 2011 17:45:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 17:45:09 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 17:45:03 +0000
Received: by wye20 with SMTP id 20so3125981wye.35
        for <general@hadoop.apache.org>; Fri, 14 Jan 2011 09:44:43 -0800 (PST)
Received: by 10.216.7.205 with SMTP id 55mr1850036wep.96.1295027070350; Fri,
 14 Jan 2011 09:44:30 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.59.13 with HTTP; Fri, 14 Jan 2011 09:43:48 -0800 (PST)
In-Reply-To: <669E9C98-C113-49B5-9851-4524EA936CED@yahoo-inc.com>
References: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com> <91A364F8-480F-4323-92F6-15B47D5A3C32@yahoo-inc.com>
 <88238863-C1CB-4932-9288-40655F225D59@holsman.net> <AANLkTi=tuU0pRo12W-9-6xn7rkb5jSveGz0My=VBGG4z@mail.gmail.com>
 <B6BDE780-4A72-4355-9182-8EF6DF44AA2C@Holsman.NET> <669E9C98-C113-49B5-9851-4524EA936CED@yahoo-inc.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Fri, 14 Jan 2011 09:43:48 -0800
X-Google-Sender-Auth: UQl2qT_q6iHhKoShEJnx6GDYCuY
Message-ID: <AANLkTimzvwhg8Rv-pvA-6Rq=bxR55rhxqxAFGbGYh4gw@mail.gmail.com>
Subject: Re: [DISCUSS] Move project split down a level
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Fri, Jan 14, 2011 at 09:32, Eric Baldeschwieler <eric14@yahoo-inc.com> w=
rote:
> I'm a huge supporter of the idea. On a related note, we've been looking f=
or the right time to mavenize. Maybe we can do both together. We could pitc=
h in a bunch of work on both if we could get the timing right.

Adding maveninzation - a great effort by itself with a lot of
potential pitfalls - into the splitting frenzy has a good chances to
be pretty disruptive for individual developers. I well remember how
people were screaming because virtually nothing worked in past-split
time and they had to invent and learn new tricks to merely do their
work.

Being a great tool Maven has a relatively steep learning curve which
won't ease the fact the projects layout have changed again and
everyone will have to adjust their workbenches to address that fact.

I suggest these two events are better be well separated in time.
  Cos

> We've got a huge batch of commits in flight now, but if we can find somet=
hing that satisfied the 22 crowd after we sync in, we'd be happy to pitch i=
n on unsplit and/or maven.
>
> ---
> E14 - via iPhone
>
> On Jan 14, 2011, at 5:43 AM, "Ian Holsman" <hadoop@holsman.net> wrote:
>
>> on that note... I propose we discuss un-splitting the project altogether=
.
>>
>> On Jan 14, 2011, at 3:39 AM, Jakob Homan wrote:
>>
>>> +1. The project split is a lie.
>>>
>>> On Fri, Jan 14, 2011 at 12:32 AM, Ian Holsman <hadoop@holsman.net> wrot=
e:
>>>> +1 full agreement.
>>>>
>>>> I think it will be a pita admin wise (due to how svn authorization is =
set up), so it might slow down creation of a new branch, but its worth it.
>>>>
>>>> ---
>>>> Ian Holsman
>>>> AOL Inc
>>>> Ian.Holsman@teamaol.com
>>>> (703) 879-3128 / AIM:ianholsman
>>>>
>>>> it's just a technicality
>>>>
>>>> On Jan 14, 2011, at 2:25 AM, Eric Baldeschwieler <eric14@yahoo-inc.com=
> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> Death to the project split! =A0Or short of that, anything to tame it.
>>>>>
>>>>> On Jan 13, 2011, at 10:18 PM, Nigel Daley wrote:
>>>>>
>>>>>> Folks,
>>>>>>
>>>>>> As I look more at the impact of the common/MR/HDFS project split on =
what and how we release Hadoop, I feel like the split needs an adjustment. =
=A0Many folks I've talked to agree that the project split has caused us a s=
plitting headache. =A0I think 1 relatively small change could alleviate som=
e of that.
>>>>>>
>>>>>> CURRENT SVN REPO:
>>>>>>
>>>>>> hadoop / [common, mapreduce, hdfs] / trunk
>>>>>> hadoop / [common, mapreduce, hdfs] / branches
>>>>>>
>>>>>> PROPOSAL:
>>>>>>
>>>>>> hadoop / trunk / [common, mapreduce, hdfs]
>>>>>> hadoop / branches / [common, mapreduce, hdfs]
>>>>>>
>>>>>> We're a long way from releasing these 3 projects independently. =A0G=
iven that, they should be branched and released as a unit. =A0This SVN stru=
cture enforces that and provides a more natural place to keep a top level b=
uild and pkg scripts that operate across all 3 projects.
>>>>>>
>>>>>> Thoughts?
>>>>>>
>>>>>> Cheers,
>>>>>> Nige
>>>>>
>>>>
>>
>

From general-return-2790-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 17:52:38 2011
Return-Path: <general-return-2790-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53984 invoked from network); 14 Jan 2011 17:52:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 17:52:38 -0000
Received: (qmail 24812 invoked by uid 500); 14 Jan 2011 17:52:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24649 invoked by uid 500); 14 Jan 2011 17:52:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24641 invoked by uid 99); 14 Jan 2011 17:52:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 17:52:34 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.104 as permitted sender)
Received: from [17.148.16.104] (HELO asmtpout029.mac.com) (17.148.16.104)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 17:52:26 +0000
MIME-version: 1.0
Content-type: multipart/alternative;
 boundary="Boundary_(ID_yDVtP2JNtVYrDRhkCJu2iw)"
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp029.mac.com (Sun Java(tm) System Messaging Server 6.3-7.04 (built Sep
 26 2008; 64bit)) with ESMTPSA id <0LF000480XMTG720@asmtp029.mac.com> for
 general@hadoop.apache.org; Fri, 14 Jan 2011 09:52:06 -0800 (PST)
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101140091
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-14_08:2011-01-14,2011-01-14,1970-01-01 signatures=0
From: Nigel Daley <ndaley@mac.com>
Subject: Re: [DISCUSS] Move project split down a level
Date: Fri, 14 Jan 2011 09:52:05 -0800
In-reply-to: <4A1492FF-C4B5-4949-B197-53B3A3B1BE29@apache.org>
To: general@hadoop.apache.org
References: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com>
 <4A1492FF-C4B5-4949-B197-53B3A3B1BE29@apache.org>
Message-id: <2AC9D86E-764B-4AA7-800F-4FA7E2E814B1@mac.com>
X-Mailer: Apple Mail (2.1082)

--Boundary_(ID_yDVtP2JNtVYrDRhkCJu2iw)
Content-type: text/plain; charset=us-ascii
Content-transfer-encoding: 7BIT


On Jan 14, 2011, at 8:51 AM, Owen O'Malley wrote:

> 
> On Jan 13, 2011, at 10:18 PM, Nigel Daley wrote:
> 
>> Folks,
>> 
>> As I look more at the impact of the common/MR/HDFS project split on what and how we release Hadoop, I feel like the split needs an adjustment.  Many folks I've talked to agree that the project split has caused us a splitting headache.  I think 1 relatively small change could alleviate some of that.
>> 
>> CURRENT SVN REPO:
>> 
>> hadoop / [common, mapreduce, hdfs] / trunk
>> hadoop / [common, mapreduce, hdfs] / branches
>> 
>> PROPOSAL:
>> 
>> hadoop / trunk / [common, mapreduce, hdfs]
>> hadoop / branches / [common, mapreduce, hdfs]
> 
> 
> Moving the source trees back together is ok, but will cause a fair amount of churn for those of us that depend on the git versions of the repository. Using Todd's hack may be able to fix it again at least for each individual user.

Looks like Todd addressed this.  Thanks!

> I assume you meant to propose:
> 
> hadoop/ {trunk, branches/*, tags/* } / {common, hdfs, mapreduce}

Ya.  Thanks for spelling that out more clearly Owen.

> which means that you can make checkouts, branches and tags with a single command. Your proposal as stated would break all of the tools that count on standard layouts of subversion repositories, such as the subversion to git gateways and eclipse.

I'm not much of a git user.
1) subversion to git gateway: break unrepairably?  
2) eclipse: break in what way?  I use eclipse for other projects structure like this without a problem.

> We currently have other stuff at the top level of hadoop: hive, logos, nightly, pig, site, and zookeeper. Clearly hive, pig, and zookeeper should be removed. The others are just versioned and aren't branched. I'm fine with leaving them at the top level as "extra" bits, but it should be decided.

+1 to removing/leaving those extra bits as you propose.  We can always move the remainder later if there is a reason to.

I've filed https://issues.apache.org/jira/browse/HADOOP-7106 for this reorg.

Cheers,
Nige



--Boundary_(ID_yDVtP2JNtVYrDRhkCJu2iw)--

From general-return-2791-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 18:00:45 2011
Return-Path: <general-return-2791-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56186 invoked from network); 14 Jan 2011 18:00:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 18:00:44 -0000
Received: (qmail 36684 invoked by uid 500); 14 Jan 2011 18:00:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36380 invoked by uid 500); 14 Jan 2011 18:00:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36366 invoked by uid 99); 14 Jan 2011 18:00:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 18:00:40 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 18:00:32 +0000
Received: by vws18 with SMTP id 18so1094147vws.35
        for <general@hadoop.apache.org>; Fri, 14 Jan 2011 10:00:11 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=oBY1KtHTyi+H2GK0SAHUf81b97IutBMM1cxqFvvejCA=;
        b=R4MGngD4ToKttg3Aj1PdBYB3aF+mpWDYCqD9Y/S6CuYfCp5udTYbHuSbCm7dCtlPjH
         uRiPCBVJ4khAAAM6Gg8yQF+nrezSSpkTpCDw/KKjgvymy84lj5b4xht6aoDOiIMjoUTn
         YvbOp/3FQ0jLDcMMzW9Vq1RjguXh9MVY+xLjo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=tn+bb0CTaTCkbqiSUwBJmdrK4XWcEsIImfu4+JWn3TBgsMD88stxLPa6nxTdTmFVkL
         a1hpc+csObQkwaolPuq12xZWKdaJF9f+i1FSnZetRqr350xhJ8HflY485v2dYdn4X3II
         ehYTW7qu7vgl7gztAhol62y62V8hwZPI7D6Yk=
Received: by 10.220.189.138 with SMTP id de10mr324315vcb.169.1295028011305;
        Fri, 14 Jan 2011 10:00:11 -0800 (PST)
Received: from [10.172.2.25] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id d14sm498136vcx.23.2011.01.14.10.00.09
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Fri, 14 Jan 2011 10:00:10 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <2C71D87C-D7BF-4993-B0F9-05E455E9CC77@mac.com>
Date: Fri, 14 Jan 2011 13:00:08 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <AFC30DB6-D764-4093-90C9-CD29924A033F@Holsman.NET>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com> <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com> <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org> <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com> <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com> <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com> <AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com> <388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com> <AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com> <04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com> <AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com> <74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com> <AANLkTimLLrfYbaaOzbEV9dcsaMFLeMmdgx73FasdxGhA@mail.gmail.com> <B4C8E6B2-A343-449B-AE63-DE5E3F883ABA@yahoo-inc.com> <00F14279-2802-4CC3-91AC-481AE257FD8B@y
 ahoo-inc.com> <AANLkTikqbRrU+3+y2nctu7RzKNb7+P8kd=4az+eQHzKW@mail.gmail.com> <58FF3525-B734-44A0-A5C6-45282A23F06F@yahoo-inc.com> <AANLkTik-=ZDac0iEbYuDHeBrRosQ3JNGvRbf7MxexQND@mail.gmail.com> <7D4397E2-B618-4BD1-8E1E-08B1C598B76F@yahoo-inc.com> <10BA9684-51FF-4332-A2FD-5F648E0AAF8C@Holsman.NET> <AANLkTi=0pFkfM+0jMq+f-X7w9tMGd7_oovh0QfUrX6Mj@mail.gmail.com> <2C71D87C-D7BF-4993-B0F9-05E455E9CC77@mac.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org


On Jan 14, 2011, at 12:32 PM, Nigel Daley wrote:

> Yup, I'll say it again.  The process ain't perfect but it's good =
enough IMO. Thank you Yahoo! for your contribution.

agree 100%.


From general-return-2792-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 18:01:08 2011
Return-Path: <general-return-2792-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56355 invoked from network); 14 Jan 2011 18:01:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 18:01:08 -0000
Received: (qmail 37974 invoked by uid 500); 14 Jan 2011 18:01:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37888 invoked by uid 500); 14 Jan 2011 18:01:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37880 invoked by uid 99); 14 Jan 2011 18:01:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 18:01:04 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.99 as permitted sender)
Received: from [17.148.16.99] (HELO asmtpout024.mac.com) (17.148.16.99)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 18:00:59 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp024.mac.com (Oracle Communications Messaging Exchange Server 7u4-18.01
 64bit (built Jul 15 2010)) with ESMTPSA id <0LF000IXXXR2X370@asmtp024.mac.com>
 for general@hadoop.apache.org; Fri, 14 Jan 2011 09:54:40 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-14_08:2011-01-14,2011-01-14,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101140091
Subject: Re: [DISCUSS] Move project split down a level
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <669E9C98-C113-49B5-9851-4524EA936CED@yahoo-inc.com>
Date: Fri, 14 Jan 2011 09:54:38 -0800
Message-id: <077B7189-D3D3-4D93-B079-30BBF356C36B@mac.com>
References: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com>
 <91A364F8-480F-4323-92F6-15B47D5A3C32@yahoo-inc.com>
 <88238863-C1CB-4932-9288-40655F225D59@holsman.net>
 <AANLkTi=tuU0pRo12W-9-6xn7rkb5jSveGz0My=VBGG4z@mail.gmail.com>
 <B6BDE780-4A72-4355-9182-8EF6DF44AA2C@Holsman.NET>
 <669E9C98-C113-49B5-9851-4524EA936CED@yahoo-inc.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

Thanks for the offer Eric!  I agree it's the right time to mavenize, but I think we should separate, but order, these two discussions/events.  This first, then mavenization.

Cheers,
Nige

On Jan 14, 2011, at 9:32 AM, Eric Baldeschwieler wrote:

> I'm a huge supporter of the idea. On a related note, we've been looking for the right time to mavenize. Maybe we can do both together. We could pitch in a bunch of work on both if we could get the timing right. 
> 
> We've got a huge batch of commits in flight now, but if we can find something that satisfied the 22 crowd after we sync in, we'd be happy to pitch in on unsplit and/or maven. 
> 
> ---
> E14 - via iPhone
> 
> On Jan 14, 2011, at 5:43 AM, "Ian Holsman" <hadoop@holsman.net> wrote:
> 
>> on that note... I propose we discuss un-splitting the project altogether.
>> 
>> On Jan 14, 2011, at 3:39 AM, Jakob Homan wrote:
>> 
>>> +1. The project split is a lie.
>>> 
>>> On Fri, Jan 14, 2011 at 12:32 AM, Ian Holsman <hadoop@holsman.net> wrote:
>>>> +1 full agreement.
>>>> 
>>>> I think it will be a pita admin wise (due to how svn authorization is set up), so it might slow down creation of a new branch, but its worth it.
>>>> 
>>>> ---
>>>> Ian Holsman
>>>> AOL Inc
>>>> Ian.Holsman@teamaol.com
>>>> (703) 879-3128 / AIM:ianholsman
>>>> 
>>>> it's just a technicality
>>>> 
>>>> On Jan 14, 2011, at 2:25 AM, Eric Baldeschwieler <eric14@yahoo-inc.com> wrote:
>>>> 
>>>>> +1
>>>>> 
>>>>> Death to the project split!  Or short of that, anything to tame it.
>>>>> 
>>>>> On Jan 13, 2011, at 10:18 PM, Nigel Daley wrote:
>>>>> 
>>>>>> Folks,
>>>>>> 
>>>>>> As I look more at the impact of the common/MR/HDFS project split on what and how we release Hadoop, I feel like the split needs an adjustment.  Many folks I've talked to agree that the project split has caused us a splitting headache.  I think 1 relatively small change could alleviate some of that.
>>>>>> 
>>>>>> CURRENT SVN REPO:
>>>>>> 
>>>>>> hadoop / [common, mapreduce, hdfs] / trunk
>>>>>> hadoop / [common, mapreduce, hdfs] / branches
>>>>>> 
>>>>>> PROPOSAL:
>>>>>> 
>>>>>> hadoop / trunk / [common, mapreduce, hdfs]
>>>>>> hadoop / branches / [common, mapreduce, hdfs]
>>>>>> 
>>>>>> We're a long way from releasing these 3 projects independently.  Given that, they should be branched and released as a unit.  This SVN structure enforces that and provides a more natural place to keep a top level build and pkg scripts that operate across all 3 projects.
>>>>>> 
>>>>>> Thoughts?
>>>>>> 
>>>>>> Cheers,
>>>>>> Nige
>>>>> 
>>>> 
>> 


From general-return-2793-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 18:04:34 2011
Return-Path: <general-return-2793-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59391 invoked from network); 14 Jan 2011 18:04:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 18:04:34 -0000
Received: (qmail 46285 invoked by uid 500); 14 Jan 2011 18:04:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45056 invoked by uid 500); 14 Jan 2011 18:04:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45048 invoked by uid 99); 14 Jan 2011 18:04:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 18:04:28 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jghoman@gmail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 18:04:22 +0000
Received: by iwn2 with SMTP id 2so3098066iwn.35
        for <general@hadoop.apache.org>; Fri, 14 Jan 2011 10:04:01 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type:content-transfer-encoding;
        bh=iGtlzfCrlzTEAPVQeTJ+jsTda024iO0FIv7UoNgG4V0=;
        b=PVk5gbNgIY90r63OryrWFsMaDi6cewGajXIm3EmmpqJEx5OXgXVVMc6QGpIVRDeM7P
         xixLmkrA4Tah0tyD3+Jj2zLBf20u+hdD6hfY/yu78qe+xi1n0svbyaB0seylwgqt6mkd
         26jX6euHFxPzs4j9cQXLqA5k3UvBNWFtQtIAk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=Xw0wxrJhHTyq2q8AEPiSX6sJwF7dVGIW0DqAhxGE2nfPYfkfEs7VLufaEdO2Ke0OJW
         YFNbH5WZIf9Yh/4CH4BDpWgqx/5rIRMdNS6LbHnK/C/PZ3y9R05Fiz0CJsopchUvJcPm
         yvF5W6uRhd4ndn+pDhXfSd+IZYJOAo11QbksI=
Received: by 10.231.20.68 with SMTP id e4mr998717ibb.194.1295028240761; Fri,
 14 Jan 2011 10:04:00 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.25.163 with HTTP; Fri, 14 Jan 2011 10:03:40 -0800 (PST)
In-Reply-To: <2C71D87C-D7BF-4993-B0F9-05E455E9CC77@mac.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
 <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
 <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
 <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
 <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com> <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
 <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com> <AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com>
 <388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com> <AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com>
 <04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com> <AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com>
 <74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com> <AANLkTimLLrfYbaaOzbEV9dcsaMFLeMmdgx73FasdxGhA@mail.gmail.com>
 <B4C8E6B2-A343-449B-AE63-DE5E3F883ABA@yahoo-inc.com> <00F14279-2802-4CC3-91AC-481AE257FD8B@yahoo-inc.com>
 <AANLkTikqbRrU+3+y2nctu7RzKNb7+P8kd=4az+eQHzKW@mail.gmail.com>
 <58FF3525-B734-44A0-A5C6-45282A23F06F@yahoo-inc.com> <AANLkTik-=ZDac0iEbYuDHeBrRosQ3JNGvRbf7MxexQND@mail.gmail.com>
 <7D4397E2-B618-4BD1-8E1E-08B1C598B76F@yahoo-inc.com> <10BA9684-51FF-4332-A2FD-5F648E0AAF8C@Holsman.NET>
 <AANLkTi=0pFkfM+0jMq+f-X7w9tMGd7_oovh0QfUrX6Mj@mail.gmail.com> <2C71D87C-D7BF-4993-B0F9-05E455E9CC77@mac.com>
From: Jakob Homan <jghoman@gmail.com>
Date: Fri, 14 Jan 2011 10:03:40 -0800
Message-ID: <AANLkTik_WB518PKGLhBS5qSJHG=RQQBCC5O0L7UWmc5_@mail.gmail.com>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

> On another thread discussing hadoop-0.20-append as a separate branch, mos=
t people agreed that new features shouldn't be added to 0.20, now we have a=
 major feature and we are all gung ho for it..

Not all are.  I'm against it for the all the same reasons I was
against 20 append.  This is also being used as a wedge to get the
append work in as .200.  My position is that every iota effort of
releasing another 20 branch is an iota not spent on getting us a
kick-ass 22.  20 was great, and we had a lot of wonderful times
together, but it's time to move on and see other releases.

But, this is a volunteer effort, and if others want to put the effort
in, they're free to do so.
-jg

On Fri, Jan 14, 2011 at 9:32 AM, Nigel Daley <ndaley@mac.com> wrote:
> Yup, I'll say it again. =A0The process ain't perfect but it's good enough=
 IMO. Thank you Yahoo! for your contribution.
>
> Clearly these patch will need review before commit when going into trunk.
>
> Let's move on to 0.22.
>
> Nige
>
> On Jan 14, 2011, at 9:20 AM, Konstantin Boudnik wrote:
>
>> I tend to second most of Ian's points here.
>>
>> On Fri, Jan 14, 2011 at 06:14, Ian Holsman <hadoop@holsman.net> wrote:
>>> (with my Apache hat on)
>>> I'm -0.5 on doing this as one big mega-patch and not including append (=
as opposed to a series of smaller patches).
>>
>> #1: we are creating a precedent of a "brain-dump" here. Although, it
>> isn't the first one in the history of OSS. Infamous Apple "patch" to
>> OpenBSD is another one ;)
>>
>> #2: How to spell 'back door' any one?
>>
>> #5: "almost 10 internal releases" Arun has mentioned above might be,
>> perhaps, considered as a great quality control effort. Also, not to
>> mention virtual impossibility to create a test plan to validate a
>> giant features patch.
>>
>>> BTW, I'd like to point out a discrepancy here:
>>>
>>> On another thread discussing hadoop-0.20-append as a separate branch, m=
ost people agreed that new features shouldn't be added to 0.20, now we have=
 a major feature and we are all gung ho for it..
>>
>> And this ^^^
>>
>> But, hey I guess it's totally worth it!
>> =A0Cos
>>
>>> --Ian
>>>
>>> On Jan 14, 2011, at 2:21 AM, Arun C Murthy wrote:
>>>
>>>>
>>>> On Jan 13, 2011, at 10:59 PM, Stack wrote:
>>>>
>>>>> (Man, it was looking good there for a second when 0.20.100 was about
>>>>> security+append!)
>>>>>
>>>>> Good luck w/ the release Arun.
>>>>>
>>>>
>>>> Thanks!
>>>>
>>>>> We might be following your 0.20.100 with a 0.20.200 append.
>>>>>
>>>>
>>>> Super!
>>>>
>>>> Arun
>>>
>>>
>
>

From general-return-2794-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 18:16:04 2011
Return-Path: <general-return-2794-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72439 invoked from network); 14 Jan 2011 18:16:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 18:16:03 -0000
Received: (qmail 72346 invoked by uid 500); 14 Jan 2011 18:16:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72261 invoked by uid 500); 14 Jan 2011 18:16:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72250 invoked by uid 99); 14 Jan 2011 18:16:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 18:16:01 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 18:15:54 +0000
Received: from sp1-ex07cas01.ds.corp.yahoo.com (sp1-ex07cas01.ds.corp.yahoo.com [216.252.116.137])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0EICpZR055269
	for <general@hadoop.apache.org>; Fri, 14 Jan 2011 10:12:51 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1295028771;
	bh=KrGlsHL2Tqx8v8o4sVGCVdfetkZ6HU90b9hS+KnbrYM=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 MIME-Version;
	b=GQisGKqx+o+fO6C0LasU+KMMKrcSY/PXICHSvaehpM3RAJWD0rVvW6ZMLAO9c99Ka
	 QKPvroOh8+vRIY7VGParK0ccNnAhM8FN7oDGmdj0xz4mmPZ7YQJnxRtcYaGN6EcfY8
	 mgQ984fcrLBaZf89tz0Mt8eVUj/94+fJfP0xuA9w=
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas01.ds.corp.yahoo.com ([216.252.116.137]) with mapi; Fri, 14 Jan
 2011 10:12:51 -0800
From: Suresh Srinivas <sureshms@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 14 Jan 2011 10:12:49 -0800
Subject: Re: triggering automated precommit testing
Thread-Topic: triggering automated precommit testing
Thread-Index: Acu0CoUYU0JXYvFQQeK57g8LvE1V9QADCGmH
Message-ID: <C955D221.2E72B%sureshms@yahoo-inc.com>
In-Reply-To: <7B0FFEF0-56CE-4AD6-AD74-291C49AEB0E6@mac.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C955D2212E72Bsureshmsyahooinccom_"
MIME-Version: 1.0

--_000_C955D2212E72Bsureshmsyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

It may not be as simple as triggering retest, as Some of the patches could =
be old and may not apply.


On 1/14/11 8:44 AM, "Nigel Daley" <ndaley@mac.com> wrote:

Todd, please file a Jira in Common against test component.  FWIW, I fear th=
e precommit integration with Jira will need some amount of work as Apache m=
oves to Jira 4.2 in the coming weeks.

Nige

On Jan 14, 2011, at 12:57 AM, Todd Lipcon wrote:

> Hey Nigel,
>
> Would there be any way to add a feature where we can make some special
> comment on the JIRA that would trigger a hudson retest? There are a lot o=
f
> really old patches out on the JIRA that would be worth re-testing against
> trunk, and it's a pain to download and re-attach.
>
> I'm thinking a comment with a special token like "@Hudson.Test"
>
> Failing that, Ian, can you add me to the Hudson list?
>
> -Todd
>
> On Wed, Jan 12, 2011 at 4:33 PM, Nigel Daley <ndaley@mac.com> wrote:
>
>>> Jakob Homan commented on HDFS-884:
>>> ----------------------------------
>>>
>>>> Konstantin, if you're trying to kick a new patch build for this you no
>> longer move it to "Open" and back to "Patch Available". Instead, you mus=
t
>> upload a new patch. Or, if you have permission, you can kickhttps://
>> hudson.apache.org/hudson/job/PreCommit-HDFS-Build/ and enter the issue
>> number.
>>>
>>> That makes me sad.  Is this a new feature or regression?
>>
>> [For everyone's benefit, moving this to general@]
>>
>> Jakob, I referenced the change here: http://tinyurl.com/4crxlvy
>> The new system is much more robust partial because it no longer relies o=
n
>> watching Jira generated emails to determined when issues move into Patch
>> Available state. There is limited info I can get from the Jira API, thus=
 the
>> triggering mechanism had to change.
>>
>> Cheers,
>> Nige
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera



--_000_C955D2212E72Bsureshmsyahooinccom_--

From general-return-2795-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 18:26:28 2011
Return-Path: <general-return-2795-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82077 invoked from network); 14 Jan 2011 18:26:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 18:26:27 -0000
Received: (qmail 89295 invoked by uid 500); 14 Jan 2011 18:26:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89213 invoked by uid 500); 14 Jan 2011 18:26:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89202 invoked by uid 99); 14 Jan 2011 18:26:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 18:26:25 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 18:26:17 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0EIPcaD086838;
	Fri, 14 Jan 2011 10:25:38 -0800 (PST)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by
 SP2-EX07CAS01.ds.corp.yahoo.com ([98.137.59.37]) with mapi; Fri, 14 Jan 2011
 10:25:38 -0800
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 14 Jan 2011 10:25:29 -0800
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Thread-Topic: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Thread-Index: Acu0GHDTGqR/wYWwQz6EtfzeWHusfQ==
Message-ID: <4E3C214A-16C6-46FF-908B-8EFB075A24E0@yahoo-inc.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
 <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
 <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
 <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
 <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
 <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
 <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
 <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
 <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
 <AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com>
 <388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com>
 <AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com>
 <04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com>
 <AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com>
 <74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com>
 <AANLkTimLLrfYbaaOzbEV9dcsaMFLeMmdgx73FasdxGhA@mail.gmail.com>
 <B4C8E6B2-A343-449B-AE63-DE5E3F883ABA@yahoo-inc.com>
 <36CBB8E6-05D6-4AC0-838C-14B1C2B19CBD@!  mac.com>
 <00F14279-2802-4CC3-91AC-481AE257FD8B@yahoo-inc.com>
 <AANLkTikqbRrU+3+y2nctu7RzKNb7+P8kd=4az+eQHzKW@mail.gmail.com>
 <58FF3525-B734-44A0-A5C6-45282A23F06F@yahoo-inc.com>
 <AANLkTik-=ZDac0iEbYuDHeBrRosQ3JNGvRbf7MxexQND@mail.gmail.com>
 <7D4397E2-B618-4BD1-8E1E-08B1C598B76F@yahoo-inc.com>
 <10BA9684-51FF-4332-A2FD-5F648E0AAF8C@Holsman.NET>
In-Reply-To: <10BA9684-51FF-4332-A2FD-5F648E0AAF8C@Holsman.NET>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Ian,

Thanks for holding off on that last .5. I've been working in a big email gi=
ving move context on this. Let me preview some issues.=20

Our goal with this branch is two fold: 1) get the code out in a branch quic=
kly so we an collaborate on it with the community. 2) not change the charac=
ter of the code. See testing below. We're happy to compromise any other dim=
ension, as long as we can do 1&2 above.=20

1) I agree this is not a good precedent. We don't support mega-patches in g=
eneral. We are doing this as part of discontinuing the "yahoo distribution =
of Hadoop".  We don't plan to continue doing 30 person year projects outsid=
e apache and then merging them in!!

2) append is hard. It is so hard we rewrote the entire write pipeline (5 pe=
rson-years work) in trunk after giving up on the codeline you are suggestin=
g we merge in. That work is what distinguishes all post 20 releases from 20=
 releases in my mind. I dont trust the 20 append code line. We've been hurt=
 badly by it.  We did the rewrite only after losing a bunch of production d=
ata a bunch of times with the previous code line.  I think the various 20 a=
ppend patch lines may be fine for specialized hbase clusters, but they does=
n't have the rigor behind them to bet your business in them.

3) I think having a very stable recent codeline available for teams coming =
into Hadoop who want to run big business apps and contribute code back is v=
ery helpful. I've been talking to folks in other orgs and they've expressed=
 a huge amount of interest in this work, but begged us to put it into apach=
e, so their oversight bodies will let them use it.=20

4) we're happy to incorporate ideas into how to best merge the work into tr=
unk. Let's find the most cost effective way to preserve the most devel data=
 possible.=20

5) testing. Ian, I think you do us a disservice when you talk about us just=
 testing in our environments. If you look at the history of the project, we=
've been the force behind every stable release of apache Hadoop.  And all t=
he non-apache Hadoop release had been tracking this patch set. We fully sup=
port the community developing independent testing capabilities.  We plan to=
 contribute to that effort.  But we are the organization with far and away =
the best record for testing Hadoop.=20

We are proud of thus release, we want to share it. Help us sort out how.=20

Thanks!

---
E14 - via iPhone

On Jan 14, 2011, at 6:15 AM, "Ian Holsman" <hadoop@holsman.net> wrote:

> (with my Apache hat on)
> I'm -0.5 on doing this as one big mega-patch and not including append (as=
 opposed to a series of smaller patches).
>=20
> for the following reasons:
>=20
> 1. It encourages bad behavior. We want discussion (and development) to ha=
ppen on the lists, not in some office. By allowing these large code-dumps i=
t condones this behavior, and we will likely see it again and again. Like i=
t or not, this is not the apache model of open source governance.=20
>=20
> 2. There is a risk that some code that is not in a JIRA or separate patch=
 creeps in unwittingly. This isn't a major deal per se, but we don't really=
 have the proper paper trail, or the documentation on what bug it fixed etc=
 etc.
>=20
> 3. Other groups (Facebook for example) are running with their own set of =
patches. They currently have the luxury of examining each individual patch =
to decide if they want to integrate it (and test it) in their environment. =
We are forcing them to do the work of finding the bits they want in this hu=
ge patch.
>=20
> 4. By not including the append patch, we are making this release unusable=
 for a large portion of our community who run hbase.
>=20
> 5. It makes it very hard to test. While It makes me comfortable that it h=
as gone through Yahoo!'s QA and is running in their environments, it doesn'=
t mean that it will work in other organizations who have different workload=
 mixes and software running on them. With one huge patch it makes it all or=
 nothing.. either they take the code-drop and perform a large QA-integratio=
n effort, or they forgo the whole patch together.
>=20
>=20
> **BUT** we have both the Yahoo! & Cloudera guys happy to do it, and to sp=
end their time doing it.. so I think having the code-drop will put us in a =
better place then where we are.
>=20
>=20
> BTW, I'd like to point out a discrepancy here:
>=20
> On another thread discussing hadoop-0.20-append as a separate branch, mos=
t people agreed that new features shouldn't be added to 0.20, now we have a=
 major feature and we are all gung ho for it..=20
>=20
> --Ian
>=20
> On Jan 14, 2011, at 2:21 AM, Arun C Murthy wrote:
>=20
>>=20
>> On Jan 13, 2011, at 10:59 PM, Stack wrote:
>>=20
>>> (Man, it was looking good there for a second when 0.20.100 was about
>>> security+append!)
>>>=20
>>> Good luck w/ the release Arun.
>>>=20
>>=20
>> Thanks!
>>=20
>>> We might be following your 0.20.100 with a 0.20.200 append.
>>>=20
>>=20
>> Super!
>>=20
>> Arun
>=20

From general-return-2796-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 18:31:36 2011
Return-Path: <general-return-2796-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84160 invoked from network); 14 Jan 2011 18:31:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 18:31:36 -0000
Received: (qmail 97973 invoked by uid 500); 14 Jan 2011 18:31:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97913 invoked by uid 500); 14 Jan 2011 18:31:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97905 invoked by uid 99); 14 Jan 2011 18:31:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 18:31:34 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 18:31:25 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p0EIUlIT082082;
	Fri, 14 Jan 2011 10:30:47 -0800 (PST)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by
 SP2-EX07CAS01.ds.corp.yahoo.com ([98.137.59.37]) with mapi; Fri, 14 Jan 2011
 10:30:47 -0800
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 14 Jan 2011 10:30:38 -0800
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Thread-Topic: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Thread-Index: Acu0GSk1lRKFr+chTQWn5KsQoIwRSg==
Message-ID: <B5106571-EF43-4FE1-B73E-3A1C13AEC2EE@yahoo-inc.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
 <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
 <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
 <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
 <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
 <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
 <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
 <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
 <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
 <AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com>
 <388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com>
 <AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com>
 <04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com>
 <AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com>
 <74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com>
 <AANLkTimLLrfYbaaOzbEV9dcsaMFLeMmdgx73FasdxGhA@mail.gmail.com>
 <B4C8E6B2-A343-449B-AE63-DE5E3F883ABA@yahoo-inc.com>
 <00F14279-2802-4CC3-91AC-481AE257FD8B@yahoo-inc.com>
 <AANLkTikqbRrU+3+y2nctu7RzKNb7+P8kd=4az+eQHzKW@mail.gmail.com>
 <58FF3525-B734-44A0-A5C6-45282A23F06F@yahoo-inc.com>
 <AANLkTik-=ZDac0iEbYuDHeBrRosQ3JNGvRbf7MxexQND@mail.gmail.com>
 <7D4397E2-B618-4BD1-8E1E-08B1C598B76F@yahoo-inc.com>
 <10BA9684-51FF-4332-A2FD-5F648E0AAF8C@Holsman.NET>
 <AANLkTi=0pFkfM+0jMq+f-X7w9tMGd7_oovh0QfUrX6Mj@mail.gmail.com>
 <2C71D87C-D7BF-4993-B0F9-05E455E9CC77@mac.com>
 <AANLkTik_WB518PKGLhBS5qSJHG=RQQBCC5O0L7UWmc5_@mail.gmail.com>
In-Reply-To: <AANLkTik_WB518PKGLhBS5qSJHG=RQQBCC5O0L7UWmc5_@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Yup. Letting people who want to contribute, do so a good meme!

A stable next release would be great. But orgs do sustaining on stable code=
 releases for a lot of very good reasons.=20

A next Hadoop 21+ of this code quality is almost a year away in my opinion.=
=20

---
E14 - via iPhone

On Jan 14, 2011, at 10:05 AM, "Jakob Homan" <jghoman@gmail.com> wrote:

>> On another thread discussing hadoop-0.20-append as a separate branch, mo=
st people agreed that new features shouldn't be added to 0.20, now we have =
a major feature and we are all gung ho for it..
>=20
> Not all are.  I'm against it for the all the same reasons I was
> against 20 append.  This is also being used as a wedge to get the
> append work in as .200.  My position is that every iota effort of
> releasing another 20 branch is an iota not spent on getting us a
> kick-ass 22.  20 was great, and we had a lot of wonderful times
> together, but it's time to move on and see other releases.
>=20
> But, this is a volunteer effort, and if others want to put the effort
> in, they're free to do so.
> -jg
>=20
> On Fri, Jan 14, 2011 at 9:32 AM, Nigel Daley <ndaley@mac.com> wrote:
>> Yup, I'll say it again.  The process ain't perfect but it's good enough =
IMO. Thank you Yahoo! for your contribution.
>>=20
>> Clearly these patch will need review before commit when going into trunk=
.
>>=20
>> Let's move on to 0.22.
>>=20
>> Nige
>>=20
>> On Jan 14, 2011, at 9:20 AM, Konstantin Boudnik wrote:
>>=20
>>> I tend to second most of Ian's points here.
>>>=20
>>> On Fri, Jan 14, 2011 at 06:14, Ian Holsman <hadoop@holsman.net> wrote:
>>>> (with my Apache hat on)
>>>> I'm -0.5 on doing this as one big mega-patch and not including append =
(as opposed to a series of smaller patches).
>>>=20
>>> #1: we are creating a precedent of a "brain-dump" here. Although, it
>>> isn't the first one in the history of OSS. Infamous Apple "patch" to
>>> OpenBSD is another one ;)
>>>=20
>>> #2: How to spell 'back door' any one?
>>>=20
>>> #5: "almost 10 internal releases" Arun has mentioned above might be,
>>> perhaps, considered as a great quality control effort. Also, not to
>>> mention virtual impossibility to create a test plan to validate a
>>> giant features patch.
>>>=20
>>>> BTW, I'd like to point out a discrepancy here:
>>>>=20
>>>> On another thread discussing hadoop-0.20-append as a separate branch, =
most people agreed that new features shouldn't be added to 0.20, now we hav=
e a major feature and we are all gung ho for it..
>>>=20
>>> And this ^^^
>>>=20
>>> But, hey I guess it's totally worth it!
>>>  Cos
>>>=20
>>>> --Ian
>>>>=20
>>>> On Jan 14, 2011, at 2:21 AM, Arun C Murthy wrote:
>>>>=20
>>>>>=20
>>>>> On Jan 13, 2011, at 10:59 PM, Stack wrote:
>>>>>=20
>>>>>> (Man, it was looking good there for a second when 0.20.100 was about
>>>>>> security+append!)
>>>>>>=20
>>>>>> Good luck w/ the release Arun.
>>>>>>=20
>>>>>=20
>>>>> Thanks!
>>>>>=20
>>>>>> We might be following your 0.20.100 with a 0.20.200 append.
>>>>>>=20
>>>>>=20
>>>>> Super!
>>>>>=20
>>>>> Arun
>>>>=20
>>>>=20
>>=20
>>=20

From general-return-2797-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 18:41:02 2011
Return-Path: <general-return-2797-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88357 invoked from network); 14 Jan 2011 18:41:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 18:41:01 -0000
Received: (qmail 14485 invoked by uid 500); 14 Jan 2011 18:41:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14411 invoked by uid 500); 14 Jan 2011 18:40:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14403 invoked by uid 99); 14 Jan 2011 18:40:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 18:40:59 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.105 as permitted sender)
Received: from [17.148.16.105] (HELO asmtpout030.mac.com) (17.148.16.105)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 18:40:51 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; charset=us-ascii
Received: from [172.16.31.18]
 (adsl-71-146-5-5.dsl.pltn13.sbcglobal.net [71.146.5.5])
 by asmtp030.mac.com (Sun Java(tm) System Messaging Server 6.3-8.01 (built Dec
 16 2008; 32bit)) with ESMTPSA id <0LF00013BZVHD830@asmtp030.mac.com> for
 general@hadoop.apache.org; Fri, 14 Jan 2011 10:40:30 -0800 (PST)
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101140101
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-14_08:2011-01-14,2011-01-14,1970-01-01 signatures=0
Subject: Re: triggering automated precommit testing
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <C955D221.2E72B%sureshms@yahoo-inc.com>
Date: Fri, 14 Jan 2011 10:38:28 -0800
Message-id: <35C64D76-5E29-4B69-B3D7-443392B88F15@mac.com>
References: <C955D221.2E72B%sureshms@yahoo-inc.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

Suresh, FWIW the precommit build fails fast in such cases (by design).

nige

On Jan 14, 2011, at 10:12 AM, Suresh Srinivas wrote:

> It may not be as simple as triggering retest, as Some of the patches could be old and may not apply.
> 
> 
> On 1/14/11 8:44 AM, "Nigel Daley" <ndaley@mac.com> wrote:
> 
> Todd, please file a Jira in Common against test component.  FWIW, I fear the precommit integration with Jira will need some amount of work as Apache moves to Jira 4.2 in the coming weeks.
> 
> Nige
> 
> On Jan 14, 2011, at 12:57 AM, Todd Lipcon wrote:
> 
>> Hey Nigel,
>> 
>> Would there be any way to add a feature where we can make some special
>> comment on the JIRA that would trigger a hudson retest? There are a lot of
>> really old patches out on the JIRA that would be worth re-testing against
>> trunk, and it's a pain to download and re-attach.
>> 
>> I'm thinking a comment with a special token like "@Hudson.Test"
>> 
>> Failing that, Ian, can you add me to the Hudson list?
>> 
>> -Todd
>> 
>> On Wed, Jan 12, 2011 at 4:33 PM, Nigel Daley <ndaley@mac.com> wrote:
>> 
>>>> Jakob Homan commented on HDFS-884:
>>>> ----------------------------------
>>>> 
>>>>> Konstantin, if you're trying to kick a new patch build for this you no
>>> longer move it to "Open" and back to "Patch Available". Instead, you must
>>> upload a new patch. Or, if you have permission, you can kickhttps://
>>> hudson.apache.org/hudson/job/PreCommit-HDFS-Build/ and enter the issue
>>> number.
>>>> 
>>>> That makes me sad.  Is this a new feature or regression?
>>> 
>>> [For everyone's benefit, moving this to general@]
>>> 
>>> Jakob, I referenced the change here: http://tinyurl.com/4crxlvy
>>> The new system is much more robust partial because it no longer relies on
>>> watching Jira generated emails to determined when issues move into Patch
>>> Available state. There is limited info I can get from the Jira API, thus the
>>> triggering mechanism had to change.
>>> 
>>> Cheers,
>>> Nige
>>> 
>> 
>> 
>> 
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
> 
> 


From general-return-2798-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 18:55:39 2011
Return-Path: <general-return-2798-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95239 invoked from network); 14 Jan 2011 18:55:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 18:55:39 -0000
Received: (qmail 37216 invoked by uid 500); 14 Jan 2011 18:55:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37171 invoked by uid 500); 14 Jan 2011 18:55:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37163 invoked by uid 99); 14 Jan 2011 18:55:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 18:55:37 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 18:55:30 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0EIsZpg069919;
	Fri, 14 Jan 2011 10:54:36 -0800 (PST)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by
 SP2-EX07CAS01.ds.corp.yahoo.com ([98.137.59.37]) with mapi; Fri, 14 Jan 2011
 10:54:35 -0800
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 14 Jan 2011 10:54:28 -0800
Subject: Re: [DISCUSS] Move project split down a level
Thread-Topic: [DISCUSS] Move project split down a level
Thread-Index: Acu0HHyGses+x02mQPCxrGpJ14ogmQ==
Message-ID: <F5D81B04-615D-4B81-B38B-7BD854025C30@yahoo-inc.com>
References: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com>
 <91A364F8-480F-4323-92F6-15B47D5A3C32@yahoo-inc.com>
 <88238863-C1CB-4932-9288-40655F225D59@holsman.net>
 <AANLkTi=tuU0pRo12W-9-6xn7rkb5jSveGz0My=VBGG4z@mail.gmail.com>
 <B6BDE780-4A72-4355-9182-8EF6DF44AA2C@Holsman.NET>
 <669E9C98-C113-49B5-9851-4524EA936CED@yahoo-inc.com>
 <077B7189-D3D3-4D93-B079-30BBF356C36B@mac.com>
In-Reply-To: <077B7189-D3D3-4D93-B079-30BBF356C36B@mac.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Cool

---
E14 - via iPhone

On Jan 14, 2011, at 10:01 AM, "Nigel Daley" <ndaley@mac.com> wrote:

> Thanks for the offer Eric!  I agree it's the right time to mavenize, but =
I think we should separate, but order, these two discussions/events.  This =
first, then mavenization.
>=20
> Cheers,
> Nige
>=20
> On Jan 14, 2011, at 9:32 AM, Eric Baldeschwieler wrote:
>=20
>> I'm a huge supporter of the idea. On a related note, we've been looking =
for the right time to mavenize. Maybe we can do both together. We could pit=
ch in a bunch of work on both if we could get the timing right.=20
>>=20
>> We've got a huge batch of commits in flight now, but if we can find some=
thing that satisfied the 22 crowd after we sync in, we'd be happy to pitch =
in on unsplit and/or maven.=20
>>=20
>> ---
>> E14 - via iPhone
>>=20
>> On Jan 14, 2011, at 5:43 AM, "Ian Holsman" <hadoop@holsman.net> wrote:
>>=20
>>> on that note... I propose we discuss un-splitting the project altogethe=
r.
>>>=20
>>> On Jan 14, 2011, at 3:39 AM, Jakob Homan wrote:
>>>=20
>>>> +1. The project split is a lie.
>>>>=20
>>>> On Fri, Jan 14, 2011 at 12:32 AM, Ian Holsman <hadoop@holsman.net> wro=
te:
>>>>> +1 full agreement.
>>>>>=20
>>>>> I think it will be a pita admin wise (due to how svn authorization is=
 set up), so it might slow down creation of a new branch, but its worth it.
>>>>>=20
>>>>> ---
>>>>> Ian Holsman
>>>>> AOL Inc
>>>>> Ian.Holsman@teamaol.com
>>>>> (703) 879-3128 / AIM:ianholsman
>>>>>=20
>>>>> it's just a technicality
>>>>>=20
>>>>> On Jan 14, 2011, at 2:25 AM, Eric Baldeschwieler <eric14@yahoo-inc.co=
m> wrote:
>>>>>=20
>>>>>> +1
>>>>>>=20
>>>>>> Death to the project split!  Or short of that, anything to tame it.
>>>>>>=20
>>>>>> On Jan 13, 2011, at 10:18 PM, Nigel Daley wrote:
>>>>>>=20
>>>>>>> Folks,
>>>>>>>=20
>>>>>>> As I look more at the impact of the common/MR/HDFS project split on=
 what and how we release Hadoop, I feel like the split needs an adjustment.=
  Many folks I've talked to agree that the project split has caused us a sp=
litting headache.  I think 1 relatively small change could alleviate some o=
f that.
>>>>>>>=20
>>>>>>> CURRENT SVN REPO:
>>>>>>>=20
>>>>>>> hadoop / [common, mapreduce, hdfs] / trunk
>>>>>>> hadoop / [common, mapreduce, hdfs] / branches
>>>>>>>=20
>>>>>>> PROPOSAL:
>>>>>>>=20
>>>>>>> hadoop / trunk / [common, mapreduce, hdfs]
>>>>>>> hadoop / branches / [common, mapreduce, hdfs]
>>>>>>>=20
>>>>>>> We're a long way from releasing these 3 projects independently.  Gi=
ven that, they should be branched and released as a unit.  This SVN structu=
re enforces that and provides a more natural place to keep a top level buil=
d and pkg scripts that operate across all 3 projects.
>>>>>>>=20
>>>>>>> Thoughts?
>>>>>>>=20
>>>>>>> Cheers,
>>>>>>> Nige
>>>>>>=20
>>>>>=20
>>>=20
>=20

From general-return-2799-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 19:04:29 2011
Return-Path: <general-return-2799-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 415 invoked from network); 14 Jan 2011 19:04:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 19:04:28 -0000
Received: (qmail 51905 invoked by uid 500); 14 Jan 2011 19:04:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51816 invoked by uid 500); 14 Jan 2011 19:04:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51808 invoked by uid 99); 14 Jan 2011 19:04:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 19:04:26 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of shv.hadoop@gmail.com designates 209.85.213.176 as permitted sender)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 19:04:20 +0000
Received: by yxm8 with SMTP id 8so1432804yxm.35
        for <general@hadoop.apache.org>; Fri, 14 Jan 2011 11:03:59 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=LCuEPM30mTPKJQMc4FbBKvmMfZjnl7HhmBwdkIut5Js=;
        b=wJSGbKY+GwSZHAGj5r19klEjvdwVtvuWYKafT2VAkNhLfPhD8EVHsqKpIE1M3lyD8E
         ERb7dTQMfTy24o7roKQWjfRULlyR2TkZgOsf4vBuDatVMrYawK+TrQgHcPk95ovkrfjX
         EcymQz7KxVnX9hoJfdRGRleD3H96x/MZbwpYs=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=B8LlPXGgEXi7unPdxMGjfEc6Pg9DPZX0PgIjJclwkFHISMjmErang0nKfvGgnCscvi
         8teBhV3RTFJzKkAnUznQPxjfXUMVWfwqR/WDt+eBuPNA6IN0Euafr2f0RdPFuUCMSu/e
         mn6bM08whOsKjesbamiz3Mu0NtHpM3X/bnuF4=
MIME-Version: 1.0
Received: by 10.236.111.38 with SMTP id v26mr333843yhg.18.1295031710490; Fri,
 14 Jan 2011 11:01:50 -0800 (PST)
Received: by 10.236.105.228 with HTTP; Fri, 14 Jan 2011 11:01:50 -0800 (PST)
In-Reply-To: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com>
References: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com>
Date: Fri, 14 Jan 2011 11:01:50 -0800
Message-ID: <AANLkTiknWzZYo0yc3X8sfuzScDF_yREwSaBi9uXsrJi5@mail.gmail.com>
Subject: Re: [DISCUSS] Move project split down a level
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0023547c8e5113a48a0499d311b2

--0023547c8e5113a48a0499d311b2
Content-Type: text/plain; charset=ISO-8859-1

We actually still haven't recovered from the projects split.
We are still fixing HDFS and MR scripts with several jiras open.

If we start this re-split now again before the major release
we risk to get into the same mess, and it will create more work
for the community.

I see Nigel's point that packaging will get easier, and developers
will push less buttons when they commit.
It will delay the release - this is what worries me.

Thanks,
--Konstantin


On Thu, Jan 13, 2011 at 10:18 PM, Nigel Daley <ndaley@mac.com> wrote:

> Folks,
>
> As I look more at the impact of the common/MR/HDFS project split on what
> and how we release Hadoop, I feel like the split needs an adjustment.  Many
> folks I've talked to agree that the project split has caused us a splitting
> headache.  I think 1 relatively small change could alleviate some of that.
>
> CURRENT SVN REPO:
>
> hadoop / [common, mapreduce, hdfs] / trunk
> hadoop / [common, mapreduce, hdfs] / branches
>
> PROPOSAL:
>
> hadoop / trunk / [common, mapreduce, hdfs]
> hadoop / branches / [common, mapreduce, hdfs]
>
> We're a long way from releasing these 3 projects independently.  Given
> that, they should be branched and released as a unit.  This SVN structure
> enforces that and provides a more natural place to keep a top level build
> and pkg scripts that operate across all 3 projects.
>
> Thoughts?
>
> Cheers,
> Nige
>

--0023547c8e5113a48a0499d311b2--

From general-return-2800-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 19:08:42 2011
Return-Path: <general-return-2800-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 11314 invoked from network); 14 Jan 2011 19:08:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 19:08:42 -0000
Received: (qmail 61754 invoked by uid 500); 14 Jan 2011 19:08:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61682 invoked by uid 500); 14 Jan 2011 19:08:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61674 invoked by uid 99); 14 Jan 2011 19:08:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 19:08:40 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.99 as permitted sender)
Received: from [17.148.16.99] (HELO asmtpout024.mac.com) (17.148.16.99)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 19:08:32 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [172.16.31.18]
 (adsl-71-146-5-5.dsl.pltn13.sbcglobal.net [71.146.5.5])
 by asmtp024.mac.com (Oracle Communications Messaging Exchange Server 7u4-18.01
 64bit (built Jul 15 2010)) with ESMTPSA id <0LF100LAY15M0LA0@asmtp024.mac.com>
 for general@hadoop.apache.org; Fri, 14 Jan 2011 11:08:11 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-14_08:2011-01-14,2011-01-14,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=5 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101140105
Subject: Re: [DISCUSS] Move project split down a level
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <AANLkTiknWzZYo0yc3X8sfuzScDF_yREwSaBi9uXsrJi5@mail.gmail.com>
Date: Fri, 14 Jan 2011 11:08:10 -0800
Message-id: <225EAA53-D3F0-4E32-BB1C-6640FA46EA8F@mac.com>
References: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com>
 <AANLkTiknWzZYo0yc3X8sfuzScDF_yREwSaBi9uXsrJi5@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)


On Jan 14, 2011, at 11:01 AM, Konstantin Shvachko wrote:

> We actually still haven't recovered from the projects split.
> We are still fixing HDFS and MR scripts with several jiras open.

Great, so let's do this reorg before we fix those jira's so we don't need to fix them again.  Can you provide the issue numbers you're think of?

Thx,
Nige

> On Thu, Jan 13, 2011 at 10:18 PM, Nigel Daley <ndaley@mac.com> wrote:
> 
>> Folks,
>> 
>> As I look more at the impact of the common/MR/HDFS project split on what
>> and how we release Hadoop, I feel like the split needs an adjustment.  Many
>> folks I've talked to agree that the project split has caused us a splitting
>> headache.  I think 1 relatively small change could alleviate some of that.
>> 
>> CURRENT SVN REPO:
>> 
>> hadoop / [common, mapreduce, hdfs] / trunk
>> hadoop / [common, mapreduce, hdfs] / branches
>> 
>> PROPOSAL:
>> 
>> hadoop / trunk / [common, mapreduce, hdfs]
>> hadoop / branches / [common, mapreduce, hdfs]
>> 
>> We're a long way from releasing these 3 projects independently.  Given
>> that, they should be branched and released as a unit.  This SVN structure
>> enforces that and provides a more natural place to keep a top level build
>> and pkg scripts that operate across all 3 projects.
>> 
>> Thoughts?
>> 
>> Cheers,
>> Nige
>> 


From general-return-2801-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 19:16:58 2011
Return-Path: <general-return-2801-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 13846 invoked from network); 14 Jan 2011 19:16:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 19:16:58 -0000
Received: (qmail 75806 invoked by uid 500); 14 Jan 2011 19:16:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75662 invoked by uid 500); 14 Jan 2011 19:16:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75653 invoked by uid 99); 14 Jan 2011 19:16:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 19:16:56 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [98.139.52.228] (HELO nm7-vm0.bullet.mail.ac4.yahoo.com) (98.139.52.228)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 14 Jan 2011 19:16:45 +0000
Received: from [98.139.52.192] by nm7.bullet.mail.ac4.yahoo.com with NNFMP; 14 Jan 2011 19:16:22 -0000
Received: from [98.139.52.178] by tm5.bullet.mail.ac4.yahoo.com with NNFMP; 14 Jan 2011 19:16:22 -0000
Received: from [127.0.0.1] by omp1061.mail.ac4.yahoo.com with NNFMP; 14 Jan 2011 19:16:22 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 585684.45596.bm@omp1061.mail.ac4.yahoo.com
Received: (qmail 18602 invoked by uid 60001); 14 Jan 2011 19:16:22 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1295032582; bh=PIki+z48q/ePdHQGHMNWp6iCEsDwfx3vO7IJ2TBKjTM=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=1Z6AlFkgVNgLb1+yJ27Vs+wBFjQ5zfRzrmQgsbojgDl8qLgMHWIMqEgy3/aDrmrkGBhHCQ70ItH/5Tsn9XPyUy2ij4inqpoBzZx5ZwbpbSw6RiRsdzjHAEOI9sSZjvax9Wbiu9xhbhPUqUZkIWgjiNlROUsTLjikl9otZ5AF4Tk=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=fPisnWBhyQ1dicvpdGPwmcLmf0kOI2IrV4TG2wKr/gwBYbbUak0sPROtn4SWC+0EgJz7B0Qyh2eaTQCE73giyNdVkgAq6r9vcPIddmhV9tafIxvVBEdL/fcHtuOLju6D9RLzokt/j9H0TnIEizqZcbdWxL8tEOa7YXAezXBzoK4=;
Message-ID: <439416.18126.qm@web56207.mail.re3.yahoo.com>
X-YMail-OSG: 7bAbxbkVM1lOettzdvPRTUIvlp7YXZNVlnG0Fa9KDotEIQM
 5X1MAXGxW7UVPwRrheIcrapv41r2RM_0OjLAmbeINxZPBk8rNYjTbVr0dXva
 HOgizJMHRnNZ6vi.hdKeKOi1cNkjnaXF9oVhikJwzKJm9rttDDjHkxeUXAoT
 A1GZazDJWphFRvMNTft56QhJml65chcyEUPKg0KBJV4t66cFc.1l1Z1FtNXx
 XT7oCeoAF1P0t4moWZpX18wsP8M7xsRkM6vFYfb7SfVi9dIgE4wAWSB.wYbc
 V5MjY1SlWtBbkmVTsQ4SrnngzPKSsO0Gyp0VHU.UHr88umVPwGfDyhB8ZeHj
 e8fy52Zk07wAGjosz0raTPGEZY0Rop1frDakfaB6mr2Mh2.L93aLJdzeU89R
 q8913BhemvUxgTpvx13lhU7DeyurTSyw8
Received: from [209.131.62.113] by web56207.mail.re3.yahoo.com via HTTP; Fri, 14 Jan 2011 11:16:22 PST
X-Mailer: YahooMailRC/555 YahooMailWebService/0.8.107.285259
References: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com> <AANLkTiknWzZYo0yc3X8sfuzScDF_yREwSaBi9uXsrJi5@mail.gmail.com>
Date: Fri, 14 Jan 2011 11:16:22 -0800 (PST)
From: "Tsz Wo \(Nicholas\), Sze" <s29752-hadoopgeneral@yahoo.com>
Subject: Re: [DISCUSS] Move project split down a level
To: general@hadoop.apache.org
In-Reply-To: <AANLkTiknWzZYo0yc3X8sfuzScDF_yREwSaBi9uXsrJi5@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1278697014-1295032582=:18126"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-1278697014-1295032582=:18126
Content-Type: text/plain; charset=us-ascii

Hi Nigel,

> As I look more at the impact of the common/MR/HDFS project split on what
> and how we release Hadoop, I feel like the split needs an adjustment.  Many
> folks I've talked to agree that the project split has caused us a splitting
> headache.  I think 1 relatively small change could alleviate some of that.

Could you elaborate your idea on how the proposed changes would help?  What the 
problems are being addressed?  It is not clear to me.

You are right that the change is small but the impact is huge.  We should first 
understand what we are getting from the changes before doing it.

Thanks.
Nicholas

--0-1278697014-1295032582=:18126--

From general-return-2802-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 19:22:19 2011
Return-Path: <general-return-2802-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14856 invoked from network); 14 Jan 2011 19:22:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 19:22:19 -0000
Received: (qmail 83013 invoked by uid 500); 14 Jan 2011 19:22:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82853 invoked by uid 500); 14 Jan 2011 19:22:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82842 invoked by uid 99); 14 Jan 2011 19:22:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 19:22:17 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.105 as permitted sender)
Received: from [17.148.16.105] (HELO asmtpout030.mac.com) (17.148.16.105)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 19:22:09 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; charset=us-ascii
Received: from [172.16.31.18]
 (adsl-71-146-5-5.dsl.pltn13.sbcglobal.net [71.146.5.5])
 by asmtp030.mac.com (Sun Java(tm) System Messaging Server 6.3-8.01 (built Dec
 16 2008; 32bit)) with ESMTPSA id <0LF10032L1RP0O10@asmtp030.mac.com> for
 general@hadoop.apache.org; Fri, 14 Jan 2011 11:21:27 -0800 (PST)
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101140108
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-14_08:2011-01-14,2011-01-14,1970-01-01 signatures=0
Subject: Re: [DISCUSS] Move project split down a level
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <439416.18126.qm@web56207.mail.re3.yahoo.com>
Date: Fri, 14 Jan 2011 11:21:25 -0800
Message-id: <15CB20F8-469A-457D-BFE2-084785FB9E18@mac.com>
References: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com>
 <AANLkTiknWzZYo0yc3X8sfuzScDF_yREwSaBi9uXsrJi5@mail.gmail.com>
 <439416.18126.qm@web56207.mail.re3.yahoo.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)


On Jan 14, 2011, at 11:16 AM, Tsz Wo (Nicholas), Sze wrote:

> Hi Nigel,
> 
>> As I look more at the impact of the common/MR/HDFS project split on what
>> and how we release Hadoop, I feel like the split needs an adjustment.  Many
>> folks I've talked to agree that the project split has caused us a splitting
>> headache.  I think 1 relatively small change could alleviate some of that.
> 
> Could you elaborate your idea on how the proposed changes would help?  What the 
> problems are being addressed?  It is not clear to me.

Critical in my mind was my statement: "We're a long way from releasing these 3 projects independently.  Given that, they should be branched and released as a unit."  This can not be enforced given the current svn layout. Other's can weigh in with additional thoughts.

> You are right that the change is small but the impact is huge.  We should first 
> understand what we are getting from the changes before doing it.

What do you see as the huge impact?

Nige


From general-return-2803-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 19:23:33 2011
Return-Path: <general-return-2803-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16037 invoked from network); 14 Jan 2011 19:23:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 19:23:32 -0000
Received: (qmail 85255 invoked by uid 500); 14 Jan 2011 19:23:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85206 invoked by uid 500); 14 Jan 2011 19:23:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85198 invoked by uid 99); 14 Jan 2011 19:23:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 19:23:30 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of shv.hadoop@gmail.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 19:23:24 +0000
Received: by gyf1 with SMTP id 1so1434579gyf.35
        for <general@hadoop.apache.org>; Fri, 14 Jan 2011 11:23:03 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=iG3KZ9SYxBKLHbRZjTN6Gy7EeT6mtfFw6W+b4gc09uU=;
        b=u3PgJvAXU1bhT8wm61QZsDWpEl4JMdwqVPLCa+LHmGgIjwgaHw9mR2EdzAwzwtkSvH
         4SXzsFR0WZcQGwSahIJzy7zRny3IB7i/fcT+l5BO7v47J0yN8GaTQ0Eh3xXokWu6VCqV
         XLJlRRwzTkRnTaPV+OCNZp5JqZqC5LaS4A6l8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=bFtTk7VGR8FlRPQx5EHt8jSUEWs7rl/78VugOVwGqaijkqtbaRu8tR4IC+JLr+Px2k
         BUwC0k3hh7Gc4rkZuLnnZjFUbmEKkrrGtX5t0DLRTnNRZx7Dc34Kv6CjSkzrfKtwdyoH
         LIY1Eujs8RRGo1lDc/9lTJOaSc9QXpD/SucPI=
MIME-Version: 1.0
Received: by 10.236.105.240 with SMTP id k76mr2398272yhg.96.1295032978474;
 Fri, 14 Jan 2011 11:22:58 -0800 (PST)
Received: by 10.236.105.228 with HTTP; Fri, 14 Jan 2011 11:22:58 -0800 (PST)
In-Reply-To: <225EAA53-D3F0-4E32-BB1C-6640FA46EA8F@mac.com>
References: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com>
	<AANLkTiknWzZYo0yc3X8sfuzScDF_yREwSaBi9uXsrJi5@mail.gmail.com>
	<225EAA53-D3F0-4E32-BB1C-6640FA46EA8F@mac.com>
Date: Fri, 14 Jan 2011 11:22:58 -0800
Message-ID: <AANLkTim2efnKiFD8C1ujiXbzCVKuJsOeahiQ7zv_aS7C@mail.gmail.com>
Subject: Re: [DISCUSS] Move project split down a level
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00235444776da795fe0499d35ce0

--00235444776da795fe0499d35ce0
Content-Type: text/plain; charset=ISO-8859-1

Well this will generate tens of more jiras, which wont justify closing the
few remaining.
You have been there, it took months to get that thing settled so it was
usable.
I am just saying its a risk we can get the same this time.
--Konstantin

On Fri, Jan 14, 2011 at 11:08 AM, Nigel Daley <ndaley@mac.com> wrote:

>
> On Jan 14, 2011, at 11:01 AM, Konstantin Shvachko wrote:
>
> > We actually still haven't recovered from the projects split.
> > We are still fixing HDFS and MR scripts with several jiras open.
>
> Great, so let's do this reorg before we fix those jira's so we don't need
> to fix them again.  Can you provide the issue numbers you're think of?
>
> Thx,
> Nige
>
> > On Thu, Jan 13, 2011 at 10:18 PM, Nigel Daley <ndaley@mac.com> wrote:
> >
> >> Folks,
> >>
> >> As I look more at the impact of the common/MR/HDFS project split on what
> >> and how we release Hadoop, I feel like the split needs an adjustment.
>  Many
> >> folks I've talked to agree that the project split has caused us a
> splitting
> >> headache.  I think 1 relatively small change could alleviate some of
> that.
> >>
> >> CURRENT SVN REPO:
> >>
> >> hadoop / [common, mapreduce, hdfs] / trunk
> >> hadoop / [common, mapreduce, hdfs] / branches
> >>
> >> PROPOSAL:
> >>
> >> hadoop / trunk / [common, mapreduce, hdfs]
> >> hadoop / branches / [common, mapreduce, hdfs]
> >>
> >> We're a long way from releasing these 3 projects independently.  Given
> >> that, they should be branched and released as a unit.  This SVN
> structure
> >> enforces that and provides a more natural place to keep a top level
> build
> >> and pkg scripts that operate across all 3 projects.
> >>
> >> Thoughts?
> >>
> >> Cheers,
> >> Nige
> >>
>
>

--00235444776da795fe0499d35ce0--

From general-return-2804-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 19:24:51 2011
Return-Path: <general-return-2804-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16263 invoked from network); 14 Jan 2011 19:24:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 19:24:51 -0000
Received: (qmail 87479 invoked by uid 500); 14 Jan 2011 19:24:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87400 invoked by uid 500); 14 Jan 2011 19:24:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87392 invoked by uid 99); 14 Jan 2011 19:24:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 19:24:49 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 74.125.82.194 as permitted sender)
Received: from [74.125.82.194] (HELO mail-wy0-f194.google.com) (74.125.82.194)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 19:24:42 +0000
Received: by wyf23 with SMTP id 23so1021118wyf.5
        for <general@hadoop.apache.org>; Fri, 14 Jan 2011 11:24:21 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=/uSOBseNaGZV9QApbQtQHXhh6JsdlDF8W7saZzhfrKA=;
        b=LOA2ZAsb//8edn6p66hyo8NkxEMuork0kQMf6dkk2bSYjxjrZq1P2Fy0QDKKc7CSs6
         o9WbOYaAU08Y2VLYwMRlT3YTdZ3C4Pxhouhi3PSUCV5mDyfXxsHmLzRZTXgVS3JSxu+r
         FyIRTZtCPmAOWNbgGYcKwkBmjuz9ZD6Zfnm2U=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=wfi8Uhrdm5v/ezW4+Mwj0aCa/OiEeGRA8w1tP2x8DzO7EIc0BeCHaau6RQRh4gJq6q
         ngJQYaF/c3ob6tL8HzvthBbI+2DGMjwklEuRzqSlPGj11HNKEIZ/IV43DtGW2KZq1MuZ
         mitVIX8+tp419bYCXwLi5/KHGrQWWu+xory5M=
MIME-Version: 1.0
Received: by 10.216.30.80 with SMTP id j58mr623500wea.64.1295033061503; Fri,
 14 Jan 2011 11:24:21 -0800 (PST)
Received: by 10.216.63.210 with HTTP; Fri, 14 Jan 2011 11:24:21 -0800 (PST)
In-Reply-To: <4E3C214A-16C6-46FF-908B-8EFB075A24E0@yahoo-inc.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
	<366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
	<18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
	<AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
	<F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
	<D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
	<2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
	<5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
	<075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
	<AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com>
	<388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com>
	<AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com>
	<04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com>
	<AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com>
	<74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com>
	<AANLkTimLLrfYbaaOzbEV9dcsaMFLeMmdgx73FasdxGhA@mail.gmail.com>
	<B4C8E6B2-A343-449B-AE63-DE5E3F883ABA@yahoo-inc.com>
	<00F14279-2802-4CC3-91AC-481AE257FD8B@yahoo-inc.com>
	<AANLkTikqbRrU+3+y2nctu7RzKNb7+P8kd=4az+eQHzKW@mail.gmail.com>
	<58FF3525-B734-44A0-A5C6-45282A23F06F@yahoo-inc.com>
	<AANLkTik-=ZDac0iEbYuDHeBrRosQ3JNGvRbf7MxexQND@mail.gmail.com>
	<7D4397E2-B618-4BD1-8E1E-08B1C598B76F@yahoo-inc.com>
	<10BA9684-51FF-4332-A2FD-5F648E0AAF8C@Holsman.NET>
	<4E3C214A-16C6-46FF-908B-8EFB075A24E0@yahoo-inc.com>
Date: Fri, 14 Jan 2011 11:24:21 -0800
Message-ID: <AANLkTik=KxWZKkwu0FJ0Uej7JfDWe7Crw5+i9_BV9QMX@mail.gmail.com>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6dbe70a9a72af0499d361a0

--0016e6dbe70a9a72af0499d361a0
Content-Type: text/plain; charset=ISO-8859-1

>
>
> 1) I agree this is not a good precedent. We don't support mega-patches in
> general. We are doing this as part of discontinuing the "yahoo distribution
> of Hadoop".  We don't plan to continue doing 30 person year projects outside
> apache and then merging them in!!
>
>
I think this is a very dangerous precedent and completely unwarranted.
mega-patches are bad and is totally not the Apache way to go. I think if you
want to contribute it back to Apache, you should avoid the mega-patch
completely.


  I think the various 20 append patch lines may be fine for specialized
> hbase clusters, but they doesn't have the rigor behind them to bet your
> business in them.
>
>
I think you are completely off-track here and jumping to conclusions. Big
business are already betting on it. HBase is becoming a big user of Hadoop
(dunno whether Y! uses HBase) and I completely agree with Ian that all
business have to anyway test their release themselves before using it,
otherwise you could land up with data loss like the type you mentioned.

 thanks,
dhruba

--0016e6dbe70a9a72af0499d361a0--

From general-return-2805-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 19:54:30 2011
Return-Path: <general-return-2805-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 28210 invoked from network); 14 Jan 2011 19:54:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 19:54:29 -0000
Received: (qmail 28182 invoked by uid 500); 14 Jan 2011 19:54:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28123 invoked by uid 500); 14 Jan 2011 19:54:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28115 invoked by uid 99); 14 Jan 2011 19:54:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 19:54:27 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,LONG_TERM_PRICE,RCVD_IN_DNSWL_NONE,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [98.139.53.193] (HELO nm9-vm1.bullet.mail.ac4.yahoo.com) (98.139.53.193)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 14 Jan 2011 19:54:17 +0000
Received: from [98.139.52.193] by nm9.bullet.mail.ac4.yahoo.com with NNFMP; 14 Jan 2011 19:53:56 -0000
Received: from [98.139.52.162] by tm6.bullet.mail.ac4.yahoo.com with NNFMP; 14 Jan 2011 19:53:56 -0000
Received: from [127.0.0.1] by omp1045.mail.ac4.yahoo.com with NNFMP; 14 Jan 2011 19:53:56 -0000
X-Yahoo-Newman-Property: ymail-5
X-Yahoo-Newman-Id: 315272.7988.bm@omp1045.mail.ac4.yahoo.com
Received: (qmail 32816 invoked by uid 60001); 14 Jan 2011 19:53:56 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1295034836; bh=h99pkVF639/hVsR7ykirYQ2YuYzXothe52yyeWo4VNg=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=Wf25dD+vLntBcnB6Ue0wVgjV3bYWmTUCM5HC1T2BiTcYRbT1z0V+mraFlIEeU4dk6AHoCXs7rjPHsv/OP8KGYtIYAsImmypk+cACES6I85reycwjA3FzEEE5aLJ0rc/6bQ34O5ADzA0pqeY95fuUHi/X8ql/e4MQ2CvxNrPP6MY=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=yWZOTay2tF+2OWkAvbVqg9pcgyhAZ5CMQyPIplvdM8BsMyRhmOWa++ONb+zkOM/T/hRaAWZt6H1fqSBfrHnkEnIXq0mGLi4H/kIiAwNwb5clVB5tJMtoe3a9vmqUl8sDHFaPJIYGHQi8XCg1QjUmRkH7tdl5jS67mk1C5/5x7+c=;
Message-ID: <113546.32737.qm@web56204.mail.re3.yahoo.com>
X-YMail-OSG: L3YZtHUVM1meVjV20QHCofvtZgDLlW.tVJ9RRhqDvHk5gMX
 YwGj6beV7RNMYP3CvIF_SYPnlTGNYhAbmWKMxmtEBC67fLImnYgEfpkGsCfz
 hG7zJq6MJ5P24nav_WeHrbIiQ8mscyPR3pRjlSKwksMIFrDI_rjNFl4prhFM
 ajqOa9MMnrdrUtwB5kJ7lQ_xYHozZXfHbdmH2nTxBmx0yy_yiwpLtLLId7TH
 y898uloNKAUR5CM9ZATF2cvcIakfi8NkJcXDCTxl4JeO1gX6F9vUucpmg.yD
 FE2OCecjarY.fOe7kByTj70a6E6R_vmJGNRg2tAixmZid4qRxtz50eBqXZyv
 pFkYANvMN
Received: from [209.131.62.113] by web56204.mail.re3.yahoo.com via HTTP; Fri, 14 Jan 2011 11:53:55 PST
X-Mailer: YahooMailRC/555 YahooMailWebService/0.8.107.285259
References: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com> <AANLkTiknWzZYo0yc3X8sfuzScDF_yREwSaBi9uXsrJi5@mail.gmail.com> <439416.18126.qm@web56207.mail.re3.yahoo.com> <15CB20F8-469A-457D-BFE2-084785FB9E18@mac.com>
Date: Fri, 14 Jan 2011 11:53:55 -0800 (PST)
From: "Tsz Wo \(Nicholas\), Sze" <s29752-hadoopgeneral@yahoo.com>
Subject: Re: [DISCUSS] Move project split down a level
To: general@hadoop.apache.org
In-Reply-To: <15CB20F8-469A-457D-BFE2-084785FB9E18@mac.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1311263020-1295034835=:32737"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-1311263020-1295034835=:32737
Content-Type: text/plain; charset=us-ascii

This is a kind of an incompatible change: all the developers, QAs, release 
engineers and users have to change their local settings and scripts for this 
change.  Moreover, there are documentations, web pages and existing tools using 
the Apache svn URLs.  So it is a huge impact.  I am conservative on this since, 
as Konstantin mentioned, we risk to get into the same mess, and it will create 
more work for the community.

Why do we want to enforce the releases as a unit, given that the long term 
target is to release these 3 projects independently?

Nicholas





________________________________
From: Nigel Daley <ndaley@mac.com>
To: general@hadoop.apache.org
Sent: Fri, January 14, 2011 11:21:25 AM
Subject: Re: [DISCUSS] Move project split down a level


On Jan 14, 2011, at 11:16 AM, Tsz Wo (Nicholas), Sze wrote:

> Hi Nigel,
> 
>> As I look more at the impact of the common/MR/HDFS project split on what
>> and how we release Hadoop, I feel like the split needs an adjustment.  Many
>> folks I've talked to agree that the project split has caused us a splitting
>> headache.  I think 1 relatively small change could alleviate some of that.
> 
> Could you elaborate your idea on how the proposed changes would help?  What the 
>
> problems are being addressed?  It is not clear to me.

Critical in my mind was my statement: "We're a long way from releasing these 3 
projects independently.  Given that, they should be branched and released as a 
unit."  This can not be enforced given the current svn layout. Other's can weigh 
in with additional thoughts.

> You are right that the change is small but the impact is huge.  We should first 
>
> understand what we are getting from the changes before doing it.

What do you see as the huge impact?

Nige
--0-1311263020-1295034835=:32737--

From general-return-2806-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 19:59:38 2011
Return-Path: <general-return-2806-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29317 invoked from network); 14 Jan 2011 19:59:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 19:59:38 -0000
Received: (qmail 33978 invoked by uid 500); 14 Jan 2011 19:59:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33922 invoked by uid 500); 14 Jan 2011 19:59:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33914 invoked by uid 99); 14 Jan 2011 19:59:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 19:59:36 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SINGLE_HEADER_2K,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mbhandarkar@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 19:59:29 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   In-Reply-To:Accept-Language:Content-Language:
   X-MS-Has-Attach:X-MS-TNEF-Correlator:x-originating-ip:
   Content-Type:Content-ID:Content-Transfer-Encoding:
   MIME-Version;
  b=aVsNu+V/Re8On5200aHDYODuFprKjXrhL5beP4NIMFYVY4rK6VZNqzJt
   /c2NmWMatmo6ARp0BQQo6jSyix2eGtwG0K9uYJ2Fw21iaVJdWQJwLE5kQ
   aTbtYh2UF8f99dD;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=mbhandarkar@linkedin.com; q=dns/txt;
  s=proddkim; t=1295035167; x=1326571167;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Milind=20Bhandarkar=20<mbhandarkar@linkedin.com>
   |Subject:=20Re:=20[DISCUSS]=20Hadoop=20Security=20Release
   =20off=20Yahoo!=20patchset|Date:=20Fri,=2014=20Jan=202011
   =2019:59:52=20+0000|Message-ID:=20<DAC9A3FF-F0C6-4054-A8B
   0-624CB7F9E9BC@linkedin.com>|To:=20"<general@hadoop.apach
   e.org>"=20<general@hadoop.apache.org>|MIME-Version:=201.0
   |Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<E67B0E2339BC294288B5A01874DB547F@linkedin
   .com>|In-Reply-To:=20<AANLkTik=3DKxWZKkwu0FJ0Uej7JfDWe7Cr
   w5+i9_BV9QMX@mail.gmail.com>|References:=20<516684F5-0052
   -4381-805D-760B61DECB16@yahoo-inc.com>=0D=0A=20<366A9E58-
   5BD7-497D-9AE1-229959ED4065@apache.org>=0D=0A=20<18C5C999
   -4680-4684-BC55-A430C40FD746@yahoo-inc.com>=0D=0A=20<AANL
   kTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
   =0D=0A=20<F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.
   com>=0D=0A=20<D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache
   .org>=0D=0A=20<2A07F1E6-7096-493B-B92E-89938689DD50@yahoo
   -inc.com>=0D=0A=20<5CDDF962-5828-459F-87C3-5033EC21E9BF@m
   ac.com>=0D=0A=20<075308A1-129B-4BF7-8924-C04EC6106D3E@yah
   oo-inc.com>=0D=0A=20<AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9
   D03J7gjBg@mail.gmail.com>=0D=0A=20<388582DF-FC85-49D1-A89
   C-1F36CE34A0E2@yahoo-inc.com>=0D=0A=20<AANLkTim=3D+8_WBNb
   xU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com>=0D=0A=20<04
   705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com>=0D=0A
   =20<AANLkTinAk+8YNLHb580uFibMWtNc=3DoYdqq+sOdBc+=3D6U@mai
   l.gmail.com>=0D=0A=20<74BDFA74-DB12-4109-89DF-B353FC7296C
   4@yahoo-inc.com>=0D=0A=20<AANLkTimLLrfYbaaOzbEV9dcsaMFLeM
   mdgx73FasdxGhA@mail.gmail.com>=0D=0A=20<B4C8E6B2-A343-449
   B-AE63-DE5E3F883ABA@yahoo-inc.com>=0D=0A=20<00F14279-2802
   -4CC3-91AC-481AE257FD8B@yahoo-inc.com>=0D=0A=20<AANLkTikq
   bRrU+3+y2nctu7RzKNb7+P8kd=3D4az+eQHzKW@mail.gmail.com>=0D
   =0A=20<58FF3525-B734-44A0-A5C6-45282A23F06F@yahoo-inc.com
   >=0D=0A=20<AANLkTik-=3DZDac0iEbYuDHeBrRosQ3JNGvRbf7MxexQN
   D@mail.gmail.com>=0D=0A=20<7D4397E2-B618-4BD1-8E1E-08B1C5
   98B76F@yahoo-inc.com>=0D=0A=20<10BA9684-51FF-4332-A2FD-5F
   648E0AAF8C@Holsman.NET>=0D=0A=20<4E3C214A-16C6-46FF-908B-
   8EFB075A24E0@yahoo-inc.com>=0D=0A=20<AANLkTik=3DKxWZKkwu0
   FJ0Uej7JfDWe7Crw5+i9_BV9QMX@mail.gmail.com>;
  bh=Jy7YXj53+8vpmMJ12lG0+d7vGftj2gz64Ckxoq0o5AM=;
  b=N4LlDN6o9H5zt7fxa1LXu81LLCt92wCdBH2R5NtPFLaaV83gd8K1ezHv
   SGIQl/f/oSBRy8J76VimHbeH3ycp5SBXXNBIZKw24Wz+38vQDT6E26wEK
   XhIy4MXWJiWpvfx;
X-IronPort-AV: E=Sophos;i="4.60,323,1291622400"; 
   d="scan'208";a="19227423"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi id 14.01.0251.000; Fri,
 14 Jan 2011 11:59:52 -0800
From: Milind Bhandarkar <mbhandarkar@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Thread-Topic: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Thread-Index: AQHLs5XyG4qjhIhUskSpH+TlCvxumJPQZGiAgAAdGoCAAAQJgIAAAg0AgAACQICAAAZfgIAABesAgABzk4CAAEYSgIAAEHOAgAAJtIA=
Date: Fri, 14 Jan 2011 19:59:52 +0000
Message-ID: <DAC9A3FF-F0C6-4054-A8B0-624CB7F9E9BC@linkedin.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
 <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
 <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
 <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
 <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
 <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
 <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
 <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
 <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
 <AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com>
 <388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com>
 <AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com>
 <04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com>
 <AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com>
 <74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com>
 <AANLkTimLLrfYbaaOzbEV9dcsaMFLeMmdgx73FasdxGhA@mail.gmail.com>
 <B4C8E6B2-A343-449B-AE63-DE5E3F883ABA@yahoo-inc.com>
 <00F14279-2802-4CC3-91AC-481AE257FD8B@yahoo-inc.com>
 <AANLkTikqbRrU+3+y2nctu7RzKNb7+P8kd=4az+eQHzKW@mail.gmail.com>
 <58FF3525-B734-44A0-A5C6-45282A23F06F@yahoo-inc.com>
 <AANLkTik-=ZDac0iEbYuDHeBrRosQ3JNGvRbf7MxexQND@mail.gmail.com>
 <7D4397E2-B618-4BD1-8E1E-08B1C598B76F@yahoo-inc.com>
 <10BA9684-51FF-4332-A2FD-5F648E0AAF8C@Holsman.NET>
 <4E3C214A-16C6-46FF-908B-8EFB075A24E0@yahoo-inc.com>
 <AANLkTik=KxWZKkwu0FJ0Uej7JfDWe7Crw5+i9_BV9QMX@mail.gmail.com>
In-Reply-To: <AANLkTik=KxWZKkwu0FJ0Uej7JfDWe7Crw5+i9_BV9QMX@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <E67B0E2339BC294288B5A01874DB547F@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Dhruba,

While I do not think that the releasability of a branch should be determine=
d by the market-cap (either on nasdaq or second-market) of the contributing=
 company, I think a well-tested release is beneficial to the community.

So, I support two releases: 20.100 now, that has security. And 20.200 later=
 that incorporates appends (depending on the 0.22+appends timeline). That w=
ay, a large percentage of community is covered in 2011.

The reasons are these:

1. The proposed 20.100 is perhaps the most tested at scale, out of all 0.20=
 branches. In fact, among *all* hadoop releases in last 5 years. I know fir=
st hand that it causes the least disruption for users, the migration from 0=
.20 to 0.20.10x was the smoothest, while adding a valuable feature.

2. HBase (running on hadoop 0.20 with append) has also been scale tested at=
 Y!, but on much less than 4000 nodes, and certainly not for varied workloa=
ds (where the bugs tend to surface). (To my knowledge, the largest HBase in=
stance is at Y! in production.)

3. Operations folks need to get some experience with raw hadoop first for a=
ny release, before other products on top of hadoop, and then handover the i=
nstallation to users. So, there is still time for HBase+0.20.100, and that =
can be addressed in a separate release.

4. It is not as if the community hasn't had a preview of this mega-patch al=
ready. A large portion of the sub-patches are already in cdh3bx, and many o=
f them have already been committed one-by-one to 0.22.

- Milind

On Jan 14, 2011, at 11:24 AM, Dhruba Borthakur wrote:

>>=20
>>=20
>> 1) I agree this is not a good precedent. We don't support mega-patches i=
n
>> general. We are doing this as part of discontinuing the "yahoo distribut=
ion
>> of Hadoop".  We don't plan to continue doing 30 person year projects out=
side
>> apache and then merging them in!!
>>=20
>>=20
> I think this is a very dangerous precedent and completely unwarranted.
> mega-patches are bad and is totally not the Apache way to go. I think if =
you
> want to contribute it back to Apache, you should avoid the mega-patch
> completely.
>=20
>=20
>  I think the various 20 append patch lines may be fine for specialized
>> hbase clusters, but they doesn't have the rigor behind them to bet your
>> business in them.
>>=20
>>=20
> I think you are completely off-track here and jumping to conclusions. Big
> business are already betting on it. HBase is becoming a big user of Hadoo=
p
> (dunno whether Y! uses HBase) and I completely agree with Ian that all
> business have to anyway test their release themselves before using it,
> otherwise you could land up with data loss like the type you mentioned.
>=20
> thanks,
> dhruba

---
Milind Bhandarkar
mbhandarkar@linkedin.com




From general-return-2807-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 19:59:59 2011
Return-Path: <general-return-2807-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29450 invoked from network); 14 Jan 2011 19:59:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 19:59:59 -0000
Received: (qmail 35394 invoked by uid 500); 14 Jan 2011 19:59:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 35349 invoked by uid 500); 14 Jan 2011 19:59:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 35341 invoked by uid 99); 14 Jan 2011 19:59:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 19:59:57 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,LONG_TERM_PRICE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jaybooth@gmail.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 19:59:50 +0000
Received: by qwh6 with SMTP id 6so3094989qwh.35
        for <general@hadoop.apache.org>; Fri, 14 Jan 2011 11:59:29 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=d/H2sdaVhdkZ6+o1nnewax2gdewCtRXeH2P9MBJPX80=;
        b=HO6leF1ZZXzvlfsY+J/R2DKOM2Am5LEDqlMlweGU48SZrbe0U26eIufsRndHPy9Ij6
         BCXz7FSAcXsTbEAZ1qiw6He48LXhPtlNth6+Q97y6j3hlMhU/uIaJlWlckJ5ftWbkq0u
         cJ+gBurgkQDGyYQTB+6wgfkyGyUolzkUdPtUc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=T4Us6/3uFNgVey4czQE047jZGYYEYMm0ePv3wZ88Dsn1byEFFFEIJEW2QVCFkPnV6Q
         7NHKX9b3lBty9C7KI2YKmlsXoU1Ot4ZgBnAi2rfC2DNv6jtpgB7q/a1V0NKNK7XLJR+t
         dqPFMAYfUWjxrrrV9OZBd2/zkcxG9UBaGy6XA=
MIME-Version: 1.0
Received: by 10.224.19.130 with SMTP id a2mr1073704qab.33.1295035168979; Fri,
 14 Jan 2011 11:59:28 -0800 (PST)
Received: by 10.220.185.129 with HTTP; Fri, 14 Jan 2011 11:59:28 -0800 (PST)
In-Reply-To: <113546.32737.qm@web56204.mail.re3.yahoo.com>
References: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com>
	<AANLkTiknWzZYo0yc3X8sfuzScDF_yREwSaBi9uXsrJi5@mail.gmail.com>
	<439416.18126.qm@web56207.mail.re3.yahoo.com>
	<15CB20F8-469A-457D-BFE2-084785FB9E18@mac.com>
	<113546.32737.qm@web56204.mail.re3.yahoo.com>
Date: Fri, 14 Jan 2011 14:59:28 -0500
Message-ID: <AANLkTimfHvjrYm18yGvY_x0OsAgYNDEt7LDH=KJH9_kW@mail.gmail.com>
Subject: Re: [DISCUSS] Move project split down a level
From: Jay Booth <jaybooth@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175cf74637fa350499d3df66
X-Virus-Checked: Checked by ClamAV on apache.org

--0015175cf74637fa350499d3df66
Content-Type: text/plain; charset=ISO-8859-1

On the flipside, right now it's only the developers, QAs and release
engineers.  There hasn't been much movement to 0.21 yet, and if we're agreed
on the change in general, then pushing out 0.22 without it means making
users change everything twice.

On Fri, Jan 14, 2011 at 2:53 PM, Tsz Wo (Nicholas), Sze <
s29752-hadoopgeneral@yahoo.com> wrote:

> This is a kind of an incompatible change: all the developers, QAs, release
> engineers and users have to change their local settings and scripts for
> this
> change.  Moreover, there are documentations, web pages and existing tools
> using
> the Apache svn URLs.  So it is a huge impact.  I am conservative on this
> since,
> as Konstantin mentioned, we risk to get into the same mess, and it will
> create
> more work for the community.
>
> Why do we want to enforce the releases as a unit, given that the long term
> target is to release these 3 projects independently?
>
> Nicholas
>
>
>
>
>
> ________________________________
> From: Nigel Daley <ndaley@mac.com>
> To: general@hadoop.apache.org
> Sent: Fri, January 14, 2011 11:21:25 AM
> Subject: Re: [DISCUSS] Move project split down a level
>
>
> On Jan 14, 2011, at 11:16 AM, Tsz Wo (Nicholas), Sze wrote:
>
> > Hi Nigel,
> >
> >> As I look more at the impact of the common/MR/HDFS project split on what
> >> and how we release Hadoop, I feel like the split needs an adjustment.
>  Many
> >> folks I've talked to agree that the project split has caused us a
> splitting
> >> headache.  I think 1 relatively small change could alleviate some of
> that.
> >
> > Could you elaborate your idea on how the proposed changes would help?
>  What the
> >
> > problems are being addressed?  It is not clear to me.
>
> Critical in my mind was my statement: "We're a long way from releasing
> these 3
> projects independently.  Given that, they should be branched and released
> as a
> unit."  This can not be enforced given the current svn layout. Other's can
> weigh
> in with additional thoughts.
>
> > You are right that the change is small but the impact is huge.  We should
> first
> >
> > understand what we are getting from the changes before doing it.
>
> What do you see as the huge impact?
>
> Nige
>

--0015175cf74637fa350499d3df66--

From general-return-2808-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 20:01:44 2011
Return-Path: <general-return-2808-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30236 invoked from network); 14 Jan 2011 20:01:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 20:01:43 -0000
Received: (qmail 39498 invoked by uid 500); 14 Jan 2011 20:01:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38849 invoked by uid 500); 14 Jan 2011 20:01:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 38841 invoked by uid 99); 14 Jan 2011 20:01:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 20:01:38 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=HTML_MESSAGE,LONG_TERM_PRICE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 20:01:31 +0000
Received: from SP1-EX07CAS02.ds.corp.yahoo.com (sp1-ex07cas02.ds.corp.yahoo.com [216.252.116.138])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p0EK0xZq009891
	for <general@hadoop.apache.org>; Fri, 14 Jan 2011 12:00:59 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1295035259;
	bh=e/zQMBcgS7OCPipXsi33Vh/yQe4dRJzTNCePqEoty+I=;
	h=From:To:Date:Subject:Message-ID:In-Reply-To:Content-Type:
	 MIME-Version;
	b=B6rhGi/7QWG5mzExlrwg+FSCcJ+YrmxvmAayYsLswWxq+5Yh20g7dyh7E/KfxabWI
	 lDIZLQxm1WLm77FIaCXqc/nqT+Xc2tLFPitLWOLIbh65+f+RdvtIIzZi70wnMSgBCQ
	 22cpXIG21KiWP4OisTUxI/xJ+GfmjHalqKoMMa5U=
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 SP1-EX07CAS02.ds.corp.yahoo.com ([216.252.116.167]) with mapi; Fri, 14 Jan
 2011 12:00:58 -0800
From: Suresh Srinivas <sureshms@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 14 Jan 2011 12:00:58 -0800
Subject: Re: [DISCUSS] Move project split down a level
Thread-Topic: [DISCUSS] Move project split down a level
Thread-Index: Acu0JPE9V+5Y1kmVRK+IyQlw/77BNQAANFBL
Message-ID: <C955EB7A.2E753%sureshms@yahoo-inc.com>
In-Reply-To: <113546.32737.qm@web56204.mail.re3.yahoo.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C955EB7A2E753sureshmsyahooinccom_"
MIME-Version: 1.0

--_000_C955EB7A2E753sureshmsyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

I like the idea of merging projects together. It save a lot of time.

However, I would like to see a detailed proposal on how this will be done a=
nd discussions on it, before moving forward on this. If this work is done, =
need clear messages to the developers on what has changed, and how developm=
ent process is affected. These details were missing when project split was =
done, causing great deal of confusion and pain.

We should also address the following:
# Is project split a goal for hadoop in the future (even though we are not =
ready yet?). If it is, then putting projects back together might result in =
tight dependencies between the project. Ho do we avoid it?
# The committer list for each of the sub project today is different. How do=
 we reconcile them?


On 1/14/11 11:53 AM, "Tsz Wo (Nicholas), Sze" <s29752-hadoopgeneral@yahoo.c=
om> wrote:

This is a kind of an incompatible change: all the developers, QAs, release
engineers and users have to change their local settings and scripts for thi=
s
change.  Moreover, there are documentations, web pages and existing tools u=
sing
the Apache svn URLs.  So it is a huge impact.  I am conservative on this si=
nce,
as Konstantin mentioned, we risk to get into the same mess, and it will cre=
ate
more work for the community.

Why do we want to enforce the releases as a unit, given that the long term
target is to release these 3 projects independently?

Nicholas





________________________________
From: Nigel Daley <ndaley@mac.com>
To: general@hadoop.apache.org
Sent: Fri, January 14, 2011 11:21:25 AM
Subject: Re: [DISCUSS] Move project split down a level


On Jan 14, 2011, at 11:16 AM, Tsz Wo (Nicholas), Sze wrote:

> Hi Nigel,
>
>> As I look more at the impact of the common/MR/HDFS project split on what
>> and how we release Hadoop, I feel like the split needs an adjustment.  M=
any
>> folks I've talked to agree that the project split has caused us a splitt=
ing
>> headache.  I think 1 relatively small change could alleviate some of tha=
t.
>
> Could you elaborate your idea on how the proposed changes would help?  Wh=
at the
>
> problems are being addressed?  It is not clear to me.

Critical in my mind was my statement: "We're a long way from releasing thes=
e 3
projects independently.  Given that, they should be branched and released a=
s a
unit."  This can not be enforced given the current svn layout. Other's can =
weigh
in with additional thoughts.

> You are right that the change is small but the impact is huge.  We should=
 first
>
> understand what we are getting from the changes before doing it.

What do you see as the huge impact?

Nige


--_000_C955EB7A2E753sureshmsyahooinccom_--

From general-return-2809-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 21:39:44 2011
Return-Path: <general-return-2809-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 92215 invoked from network); 14 Jan 2011 21:39:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 21:39:43 -0000
Received: (qmail 54607 invoked by uid 500); 14 Jan 2011 21:39:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54509 invoked by uid 500); 14 Jan 2011 21:39:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54501 invoked by uid 99); 14 Jan 2011 21:39:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 21:39:41 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ryanobjc@gmail.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 21:39:35 +0000
Received: by iwn2 with SMTP id 2so3306983iwn.35
        for <general@hadoop.apache.org>; Fri, 14 Jan 2011 13:39:14 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=+5tuv4Aiqux9Q7a2XGyg0966wV2T/bvdUo5ppO+Am+g=;
        b=o4c7C7Ms6YwvqKb3p1s/s3MmzRylmfZB/7FUmAvx7nUTmlY/yfeSPwGJnqmV0lqA2W
         IUZj7tM1NxEqB0LeZsezsjt9G3sF0r7uzf7+eizsrk4RarTaRL+Z7RsG7MKu/tqj2/sv
         p4OMX8TVDLbWAFlfy04OnhtXob+WxzdDKX63U=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=uDEJPt2408+aG9vKlk2N8P3ukbDkKtaFyFsGuW2qGa9GzpbSNUHeWjhREgOowtwlK+
         dbYKEU9cv1Q/pRE/XpFUlVzllOM6nvhmoFzyGU4ta2gH6XCWCwU31zDlP1uuhIk0gggn
         cB+XdFxSodMbPFyCzm7Gzis99gtaG+1pqgjhY=
MIME-Version: 1.0
Received: by 10.231.166.205 with SMTP id n13mr1256154iby.58.1295041154589;
 Fri, 14 Jan 2011 13:39:14 -0800 (PST)
Received: by 10.231.38.3 with HTTP; Fri, 14 Jan 2011 13:39:14 -0800 (PST)
In-Reply-To: <C955EB7A.2E753%sureshms@yahoo-inc.com>
References: <113546.32737.qm@web56204.mail.re3.yahoo.com>
	<C955EB7A.2E753%sureshms@yahoo-inc.com>
Date: Fri, 14 Jan 2011 13:39:14 -0800
Message-ID: <AANLkTikpfCf7n85NR_t2Krfa4a+h9mk0PEN_yLpQbSJb@mail.gmail.com>
Subject: Re: [DISCUSS] Move project split down a level
From: Ryan Rawson <ryanobjc@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

+1 the post split scripts are the worst things ever.

From general-return-2810-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 14 22:14:44 2011
Return-Path: <general-return-2810-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16856 invoked from network); 14 Jan 2011 22:14:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jan 2011 22:14:44 -0000
Received: (qmail 95855 invoked by uid 500); 14 Jan 2011 22:14:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95625 invoked by uid 500); 14 Jan 2011 22:14:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95609 invoked by uid 99); 14 Jan 2011 22:14:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 22:14:42 +0000
X-ASF-Spam-Status: No, hits=3.7 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of akimball83@gmail.com designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Jan 2011 22:14:34 +0000
Received: by ewy20 with SMTP id 20so1789101ewy.35
        for <general@hadoop.apache.org>; Fri, 14 Jan 2011 14:14:13 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:from:date:message-id:subject:to
         :content-type;
        bh=55X24eyMm7VOEkzVsB5umNLu2LWRg9NZAMFbiYp1kZo=;
        b=ta+bCLpU8cQ+LLnknI1qVThJN2kTRLz/seyD/3F7Xgj9nexY7Q0DVK6lZ0Sc97hayn
         PCM+DBXjNGrKQnJJg2jOGXWWd07nfeTyAjAiaMwxEdYRuigRId17leOlgBWN+NSBdZhW
         GolF8AOx4GX1J4oSJYIdxPNEi48i1U1nOB3fg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type;
        b=eK3bpeYIIV8KmGdLauFLWOKC+LXR4IFbLpG46UCuhNzYb9RbdktthTQxWfizZLodGH
         fW+yVT/Sddc918jC7b556ssh9miLWuGTLMWWA+hyWxeL4Go4nHJx1zV5WMHP37awCm7m
         5aaCEdKkdKGtvn+EuPrKZfl5LSlV0wERiaD74=
Received: by 10.204.81.218 with SMTP id y26mr898682bkk.146.1295043253651; Fri,
 14 Jan 2011 14:14:13 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.61.138 with HTTP; Fri, 14 Jan 2011 14:13:53 -0800 (PST)
From: Aaron Kimball <akimball83@gmail.com>
Date: Fri, 14 Jan 2011 14:13:53 -0800
Message-ID: <AANLkTikGBBaUR-axx9SqtE-sP308kAUPNtLDXdXQb5Wt@mail.gmail.com>
Subject: SF Hadoop meetup report
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d9a3a41a483a0499d5c12e
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d9a3a41a483a0499d5c12e
Content-Type: text/plain; charset=ISO-8859-1

Hadoop fans,

This week we held the inaugural SF Hadoop meetup, and it was a great
success! About forty people attended, and we held a number of great
discussions. After an initial plenary session, an agenda was quickly drawn
up and we identified a number of interesting topics. We spent the rest of
the time in small groups, delving into these subjects in greater depth.

Break-out sessions included:
* Hive and near-time data
* Access to Hadoop for non-technical users
* What are "Hadoopable problems"
* Data ingestion best practices
* Haderp: When Hadoop goes horribly wrong
* HBase schema design best practices
* Care and feeding of your Hadoop cluster

A big thanks to everyone who came to this meeting; it was truly "user
powered" and everyone's efforts contributed to the success we saw. I would
once again like to thank Rapleaf for hosting and Cloudera for sponsoring the
event.

Based on the outcome of this event, we will definitely hold another SF
Hadoop meetup in the near future. I will send out another announcement when
details on time and location are available.

Regards,
- Aaron Kimball

--0016e6d9a3a41a483a0499d5c12e--

From general-return-2811-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jan 15 10:20:54 2011
Return-Path: <general-return-2811-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 20492 invoked from network); 15 Jan 2011 10:20:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jan 2011 10:20:54 -0000
Received: (qmail 74652 invoked by uid 500); 15 Jan 2011 10:20:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 73924 invoked by uid 500); 15 Jan 2011 10:20:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 73902 invoked by uid 99); 15 Jan 2011 10:20:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Jan 2011 10:20:46 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [88.198.2.104] (HELO koch.ro) (88.198.2.104)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Jan 2011 10:20:40 +0000
Received: from 28-176.76-83.cust.bluewin.ch ([83.76.176.28] helo=jona.localnet)
	by koch.ro with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
	(Exim 4.69)
	(envelope-from <thomas@koch.ro>)
	id 1Pe3F2-00082G-7G; Sat, 15 Jan 2011 11:20:16 +0100
From: Thomas Koch <thomas@koch.ro>
Reply-To: thomas@koch.ro
To: infrastructure-dev@apache.org,
 general@hadoop.apache.org,
 dev@zookeeper.apache.org
Subject: Example of patch history in Gerrit
Date: Sat, 15 Jan 2011 11:20:06 +0100
User-Agent: KMail/1.13.5 (Linux/2.6.32-4-amd64; KDE/4.4.5; x86_64; ; )
MIME-Version: 1.0
Content-Type: Text/Plain;
  charset="us-ascii"
Content-Transfer-Encoding: 7bit
Message-Id: <201101151120.06407.thomas@koch.ro>

Hi,

as already written, I've setup a test instance of gerrit[1] (The GIT based 
review and repository management tool) on my server: http://koch.ro:8080

[1] http://gerrit.googlecode.com/svn/documentation/2.1.6/index.html

However if you like, you may get a better impression of gerrit by looking at 
the installation at eclipse:
http://egit.eclipse.org

In particular, I've found an example of a patch with several versions:
http://egit.eclipse.org/r/#change,1320

In the diff view you see "patch history" in the upper left corner. There you 
can select to view the diff between two different versions of the patch.

(Disclaimer: I'm just advertising gerrit as an interested individual. There 
isn't any decision from infra@a.o yet.)

Best regards,

Thomas Koch, http://www.koch.ro

From general-return-2812-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jan 15 16:39:17 2011
Return-Path: <general-return-2812-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 51870 invoked from network); 15 Jan 2011 16:39:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jan 2011 16:39:17 -0000
Received: (qmail 84420 invoked by uid 500); 15 Jan 2011 16:39:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83997 invoked by uid 500); 15 Jan 2011 16:39:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83989 invoked by uid 99); 15 Jan 2011 16:39:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Jan 2011 16:39:11 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of bernd.fondermann@googlemail.com designates 209.85.215.176 as permitted sender)
Received: from [209.85.215.176] (HELO mail-ey0-f176.google.com) (209.85.215.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Jan 2011 16:39:05 +0000
Received: by eyz10 with SMTP id 10so1957561eyz.35
        for <general@hadoop.apache.org>; Sat, 15 Jan 2011 08:38:45 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=b5VvFmqTcb6SwrySO8L7r4g1ZFT0+q2ZPFjtRQ+0Ij8=;
        b=HcLG2uMkfFAa41aO12yOB88qsPJb1GelhDT8v8FLBhi/OedUSRkMG6JBkSQXYh/tNQ
         hoBDygWpH8gzeuia08TKy58rRbB/gdao68U0V9b6yQFZNcAvRStBOSUQmJkbGYByJc75
         oJCVmbQMHE7G9+9zeBDSc2tqCo80VaJIxKrxk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Wn+HR4lQkM2tJo7SHvoIiLaipLWbxQkt7Mkzgd7CY9BNIEJrNhoChxK9IGLFtw17au
         V9Mgd7WBU+KKMpWHLv9ty+uV7SmZmrAl1dHwqhih7w/s8/MraLA4HyI/FwoYjY7veiWO
         a8kQ0l+DDAN6KHEFiJ8r84WdaMEJNgQ2cnSds=
MIME-Version: 1.0
Received: by 10.213.33.69 with SMTP id g5mr790291ebd.23.1295109525088; Sat, 15
 Jan 2011 08:38:45 -0800 (PST)
Received: by 10.213.33.11 with HTTP; Sat, 15 Jan 2011 08:38:45 -0800 (PST)
In-Reply-To: <201101151120.06407.thomas@koch.ro>
References: <201101151120.06407.thomas@koch.ro>
Date: Sat, 15 Jan 2011 17:38:45 +0100
Message-ID: <AANLkTi=ArVZfPnd26x7dtg0haAhLLK86Ex5HwLUbj6Kb@mail.gmail.com>
Subject: Re: Example of patch history in Gerrit
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org, thomas@koch.ro
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On 2011-01-15, Thomas Koch <thomas@koch.ro> wrote:
> Hi,
>
> as already written, I've setup a test instance of gerrit[1] (The GIT based
> review and repository management tool) on my server: http://koch.ro:8080
>
> [1] http://gerrit.googlecode.com/svn/documentation/2.1.6/index.html
>
> However if you like, you may get a better impression of gerrit by looking at
> the installation at eclipse:
> http://egit.eclipse.org
>
> In particular, I've found an example of a patch with several versions:
> http://egit.eclipse.org/r/#change,1320
>
> In the diff view you see "patch history" in the upper left corner. There you
> can select to view the diff between two different versions of the patch.
>
> (Disclaimer: I'm just advertising gerrit as an interested individual. There
> isn't any decision from infra@a.o yet.)
>
> Best regards,
>
> Thomas Koch, http://www.koch.ro
>

-- 
Sent from Google Mail for mobile | mobile.google.com

From general-return-2813-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jan 16 22:58:23 2011
Return-Path: <general-return-2813-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97318 invoked from network); 16 Jan 2011 22:58:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jan 2011 22:58:23 -0000
Received: (qmail 81363 invoked by uid 500); 16 Jan 2011 22:58:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81309 invoked by uid 500); 16 Jan 2011 22:58:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81300 invoked by uid 99); 16 Jan 2011 22:58:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 16 Jan 2011 22:58:20 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 16 Jan 2011 22:58:15 +0000
Received: by wwd20 with SMTP id 20so4695904wwd.29
        for <general@hadoop.apache.org>; Sun, 16 Jan 2011 14:57:54 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=+f83rNGRHeWilpkj+RGiCr+cMYlU1clSl9W6icqtg5k=;
        b=c5F12880u0mBmzEPPBDx36TgqzKNiWUo72R9+0+8bSNKrt5H0Sjg9Fi683BfhdN4To
         z/regKQaOn5hq3ZMklH7C7usoiP7J6izjhzPFlvm36qD3IGjkiYoZEx8j7GB8XAxl3NO
         uuFacXWJqBTxK7gFJmaMQ1jFZLZU9Mnz9ZHQc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=AHh7tHVbJG6ea2kyG9j5DHQGFY6lndgKfMS+2BCvQlITYydGQuJM+a6Hnq/cqXITjE
         gLLcnMTSkbYLOW/Vv/SFgSHz2McZAqz2oa7tmta/S2DUqinBSHVFj/rdNDDxlTG7TLaa
         khdrcKZtv3VXfT95eTqHxgXSziM7tm2eC0VmI=
MIME-Version: 1.0
Received: by 10.216.4.82 with SMTP id 60mr1613876wei.89.1295218673934; Sun, 16
 Jan 2011 14:57:53 -0800 (PST)
Sender: saint.ack@gmail.com
Received: by 10.216.49.11 with HTTP; Sun, 16 Jan 2011 14:57:53 -0800 (PST)
In-Reply-To: <4E3C214A-16C6-46FF-908B-8EFB075A24E0@yahoo-inc.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
	<366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
	<18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
	<AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
	<F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
	<D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
	<2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
	<5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
	<075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
	<AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com>
	<388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com>
	<AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com>
	<04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com>
	<AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com>
	<74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com>
	<AANLkTimLLrfYbaaOzbEV9dcsaMFLeMmdgx73FasdxGhA@mail.gmail.com>
	<B4C8E6B2-A343-449B-AE63-DE5E3F883ABA@yahoo-inc.com>
	<00F14279-2802-4CC3-91AC-481AE257FD8B@yahoo-inc.com>
	<AANLkTikqbRrU+3+y2nctu7RzKNb7+P8kd=4az+eQHzKW@mail.gmail.com>
	<58FF3525-B734-44A0-A5C6-45282A23F06F@yahoo-inc.com>
	<AANLkTik-=ZDac0iEbYuDHeBrRosQ3JNGvRbf7MxexQND@mail.gmail.com>
	<7D4397E2-B618-4BD1-8E1E-08B1C598B76F@yahoo-inc.com>
	<10BA9684-51FF-4332-A2FD-5F648E0AAF8C@Holsman.NET>
	<4E3C214A-16C6-46FF-908B-8EFB075A24E0@yahoo-inc.com>
Date: Sun, 16 Jan 2011 14:57:53 -0800
X-Google-Sender-Auth: YUeeGxiaghr6RbaMLjiOspmvuws
Message-ID: <AANLkTik+oTnp69ZNF7Hgtom1pTTBy+J_m=R8cQ3X+1W0@mail.gmail.com>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Fri, Jan 14, 2011 at 10:25 AM, Eric Baldeschwieler
<eric14@yahoo-inc.com> wrote:
> 2) append is hard. It is so hard we rewrote the entire write pipeline (5 =
person-years work) in trunk after giving up on the codeline you are suggest=
ing we merge in. That work is what distinguishes all post 20 releases from =
20 releases in my mind. I dont trust the 20 append code line. We've been hu=
rt badly by it. =A0We did the rewrite only after losing a bunch of producti=
on data a bunch of times with the previous code line. =A0I think the variou=
s 20 append patch lines may be fine for specialized hbase clusters, but the=
y doesn't have the rigor behind them to bet your business in them.
>

Eric:

A few comments on the above:

+ Append has had a bunch of work done on it since the Y! dataloss of a
few years ago on an ancestor of the branch-0.20-append codebase (IIRC
the issue you refer to in particular -- the 'dataloss' because
partially written blocks were done up in tmp dirs, and on cluster
restart, tmp data was cleared -- has been fixed in
branch-0.20.append).
+ You may not trust 0.20-append (or its close cousin over in CDH) but
a bunch of HBasers do. On the one hand, we have little choice.  Until
the *new* append becomes available in a stable Hadoop the HBase
project has had to sustain itself (What you think?, 3-6 months before
we see 0.22?  HBase project can't hold its breath that long).  On
other hand, the branch-0.20-append work has been carried out by lads
(and lasses!) who know their HDFS.  Its true that it will not have
been tested with Y! rigor but near-derivatives -- CDH or the FB
branches -- already do HDFS-200-based append in production.

St.Ack
P.S. Don't get me wrong.  HBase is looking forward to *new* append.
We just need something to suck on meantime.

From general-return-2814-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 17 20:00:20 2011
Return-Path: <general-return-2814-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91168 invoked from network); 17 Jan 2011 20:00:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jan 2011 20:00:20 -0000
Received: (qmail 28804 invoked by uid 500); 17 Jan 2011 20:00:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28712 invoked by uid 500); 17 Jan 2011 20:00:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28704 invoked by uid 99); 17 Jan 2011 20:00:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Jan 2011 20:00:17 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.99 as permitted sender)
Received: from [17.148.16.99] (HELO asmtpout024.mac.com) (17.148.16.99)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Jan 2011 20:00:11 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13]
 (c-71-198-192-174.hsd1.ca.comcast.net [71.198.192.174])
 by asmtp024.mac.com (Oracle Communications Messaging Exchange Server 7u4-18.01
 64bit (built Jul 15 2010)) with ESMTPSA id <0LF6001GMNH25F70@asmtp024.mac.com>
 for general@hadoop.apache.org; Mon, 17 Jan 2011 11:58:15 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-17_08:2011-01-17,2011-01-17,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=3 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101170123
Sun-Java-System-SMTP-Warning: Lines longer than SMTP allows found and wrapped.
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <B5106571-EF43-4FE1-B73E-3A1C13AEC2EE@yahoo-inc.com>
Date: Mon, 17 Jan 2011 11:58:06 -0800
Message-id: <D0AF4825-358F-472F-9718-07E162355CDC@mac.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
 <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
 <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
 <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
 <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
 <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
 <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
 <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
 <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
 <AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com>
 <388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com>
 <AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com>
 <04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com>
 <AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com>
 <74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com>
 <AANLkTimLLrfYbaaOzbEV9dcsaMFLeMmdgx73FasdxGhA@mail.gmail.com>
 <B4C8E6B2-A343-449B-AE63-DE5E3F883ABA@yahoo-inc.com>
 <00F14279-2802-4CC3-91AC-481AE257FD8B@yahoo-inc.com>
 <AANLkTikqbRrU+3+y2nctu7RzKNb7+P8kd=4az+eQHzKW@mail.gmail.com>
 <58FF3525-B734-44A0-A5C6-45282A23F06F@yahoo-inc.com>
 <AANLkTik-=ZDac0iEbYuDHeBrRosQ3JNGvRbf7MxexQND@mail.gmail.com>
 <7D4397E2-B618-4BD1-8E1E-08B1C598B76F@yahoo-inc.com>
 <10BA9684-51FF-4332-A2FD-5F648E0AAF8C@Holsman.NET>
 <AANLkTi=0pFkfM+0jMq+f-X7w9tMGd7_oovh0QfUrX6Mj@mail.gmail.com>
 <2C71D87C-D7BF-4993-B0F9-05E455E9CC77@mac.com>
 <AANLkTik_WB518PKGLhBS5qSJHG=RQQBCC5O0L7UWmc5_@mail.gmail.com>
 <B5106571-EF43-4FE1-B73E-3A1C13AEC2EE@yahoo-inc.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

Eric, Arun, I'd like to explicitly clarify one aspect of this branch and what you mean by 'release' -- it can have many meanings.

Are you asking to actually create an Apache release from this branch (binary & source)?  Or, as I was assuming, simply commit all this code to this branch and leave it there without a formal release so others can role their own binary if they wish?

Thanks,
Nige


On Jan 14, 2011, at 10:30 AM, Eric Baldeschwieler wrote:

> Yup. Letting people who want to contribute, do so a good meme!
> 
> A stable next release would be great. But orgs do sustaining on stable code releases for a lot of very good reasons. 
> 
> A next Hadoop 21+ of this code quality is almost a year away in my opinion. 
> 
> ---
> E14 - via iPhone
> 
> On Jan 14, 2011, at 10:05 AM, "Jakob Homan" <jghoman@gmail.com> wrote:
> 
>>> On another thread discussing hadoop-0.20-append as a separate branch, most people agreed that new features shouldn't be added to 0.20, now we have a major feature and we are all gung ho for it..
>> 
>> Not all are.  I'm against it for the all the same reasons I was
>> against 20 append.  This is also being used as a wedge to get the
>> append work in as .200.  My position is that every iota effort of
>> releasing another 20 branch is an iota not spent on getting us a
>> kick-ass 22.  20 was great, and we had a lot of wonderful times
>> together, but it's time to move on and see other releases.
>> 
>> But, this is a volunteer effort, and if others want to put the effort
>> in, they're free to do so.
>> -jg
>> 
>> On Fri, Jan 14, 2011 at 9:32 AM, Nigel Daley <ndaley@mac.com> wrote:
>>> Yup, I'll say it again.  The process ain't perfect but it's good enough IMO. Thank you Yahoo! for your contribution.
>>> 
>>> Clearly these patch will need review before commit when going into trunk.
>>> 
>>> Let's move on to 0.22.
>>> 
>>> Nige
>>> 
>>> On Jan 14, 2011, at 9:20 AM, Konstantin Boudnik wrote:
>>> 
>>>> I tend to second most of Ian's points here.
>>>> 
>>>> On Fri, Jan 14, 2011 at 06:14, Ian Holsman <hadoop@holsman.net> wrote:
>>>>> (with my Apache hat on)
>>>>> I'm -0.5 on doing this as one big mega-patch and not including append (as opposed to a series of smaller patches).
>>>> 
>>>> #1: we are creating a precedent of a "brain-dump" here. Although, it
>>>> isn't the first one in the history of OSS. Infamous Apple "patch" to
>>>> OpenBSD is another one ;)
>>>> 
>>>> #2: How to spell 'back door' any one?
>>>> 
>>>> #5: "almost 10 internal releases" Arun has mentioned above might be,
>>>> perhaps, considered as a great quality control effort. Also, not to
>>>> mention virtual impossibility to create a test plan to validate a
>>>> giant features patch.
>>>> 
>>>>> BTW, I'd like to point out a discrepancy here:
>>>>> 
>>>>> On another thread discussing hadoop-0.20-append as a separate branch, most people agreed that new features shouldn't be added to 0.20, now we have a major feature and we are all gung ho for it..
>>>> 
>>>> And this ^^^
>>>> 
>>>> But, hey I guess it's totally worth it!
>>>> Cos
>>>> 
>>>>> --Ian
>>>>> 
>>>>> On Jan 14, 2011, at 2:21 AM, Arun C Murthy wrote:
>>>>> 
>>>>>> 
>>>>>> On Jan 13, 2011, at 10:59 PM, Stack wrote:
>>>>>> 
>>>>>>> (Man, it was looking good there for a second when 0.20.100 was about
>>>>>>> security+append!)
>>>>>>> 
>>>>>>> Good luck w/ the release Arun.
>>>>>>> 
>>>>>> 
>>>>>> Thanks!
>>>>>> 
>>>>>>> We might be following your 0.20.100 with a 0.20.200 append.
>>>>>>> 
>>>>>> 
>>>>>> Super!
>>>>>> 
>>>>>> Arun
>>>>> 
>>>>> 
>>> 
>>> 


From general-return-2815-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 17 20:11:54 2011
Return-Path: <general-return-2815-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 3268 invoked from network); 17 Jan 2011 20:11:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jan 2011 20:11:54 -0000
Received: (qmail 45583 invoked by uid 500); 17 Jan 2011 20:11:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45511 invoked by uid 500); 17 Jan 2011 20:11:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45503 invoked by uid 99); 17 Jan 2011 20:11:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Jan 2011 20:11:52 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 17 Jan 2011 20:11:51 +0000
Received: (qmail 3228 invoked by uid 99); 17 Jan 2011 20:11:31 -0000
Received: from localhost.apache.org (HELO [192.168.168.108]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Jan 2011 20:11:31 +0000
Message-ID: <4D34A270.1060102@apache.org>
Date: Mon, 17 Jan 2011 12:11:28 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com> <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com> <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org> <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com> <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com> <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
In-Reply-To: <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 01/12/2011 11:07 PM, Arun C Murthy wrote:
> Thus, I think a jumbo patch should suffice. It will also ensure this can
> done quickly so that the community can then concentrate on 0.22 and beyond.
>
> However, I will (manually) ensure all relevant jiras are referenced in
> the CHANGES.txt and Release Notes for folks to see the contents of the
> release. This is the hardest part of the exercise. Also, this ensures
> that we can track these jiras for 0.22 as Eli suggested.
>
> Does that seem like a reasonable way forward? I'm happy to brainstorm.

We would not release this until each change in it has been reviewed by 
the community, right?  Otherwise we may end up with changes in a 0.20 
release that don't get approved when they're contributed to trunk and 
cause trunk to regress.  So I don't yet see the point of committing the 
mega patch since the community needs to review each individual change 
anyway, so we might wait until each is reviewed to commit it.

That said, posting the mega patch is useful, so that folks can start to 
pick it apart into separate issues.  Pushing your internal commits to a 
public github branch might also make that review process easier.

Doug

From general-return-2816-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 17 20:22:20 2011
Return-Path: <general-return-2816-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 7765 invoked from network); 17 Jan 2011 20:22:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jan 2011 20:22:19 -0000
Received: (qmail 58584 invoked by uid 500); 17 Jan 2011 20:22:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58519 invoked by uid 500); 17 Jan 2011 20:22:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58511 invoked by uid 99); 17 Jan 2011 20:22:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Jan 2011 20:22:17 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.99 as permitted sender)
Received: from [17.148.16.99] (HELO asmtpout024.mac.com) (17.148.16.99)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Jan 2011 20:22:09 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13]
 (c-71-198-192-174.hsd1.ca.comcast.net [71.198.192.174])
 by asmtp024.mac.com (Oracle Communications Messaging Exchange Server 7u4-18.01
 64bit (built Jul 15 2010)) with ESMTPSA id <0LF600LZSOKBKQQ0@asmtp024.mac.com>
 for general@hadoop.apache.org; Mon, 17 Jan 2011 12:21:49 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-17_08:2011-01-17,2011-01-17,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101170123
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <4D34A270.1060102@apache.org>
Date: Mon, 17 Jan 2011 12:21:46 -0800
Message-id: <3FEB7C1D-AE89-46BE-ABF2-B595B01E222A@mac.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
 <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
 <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
 <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
 <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
 <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
 <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
 <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
 <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
 <4D34A270.1060102@apache.org>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)


On Jan 17, 2011, at 12:11 PM, Doug Cutting wrote:

> On 01/12/2011 11:07 PM, Arun C Murthy wrote:
>> Thus, I think a jumbo patch should suffice. It will also ensure this can
>> done quickly so that the community can then concentrate on 0.22 and beyond.
>> 
>> However, I will (manually) ensure all relevant jiras are referenced in
>> the CHANGES.txt and Release Notes for folks to see the contents of the
>> release. This is the hardest part of the exercise. Also, this ensures
>> that we can track these jiras for 0.22 as Eli suggested.
>> 
>> Does that seem like a reasonable way forward? I'm happy to brainstorm.
> 
> We would not release this until each change in it has been reviewed by the community, right?  Otherwise we may end up with changes in a 0.20 release that don't get approved when they're contributed to trunk and cause trunk to regress.  So I don't yet see the point of committing the mega patch since the community needs to review each individual change anyway, so we might wait until each is reviewed to commit it.

Unless this is a code-only drop into a branch w/ no formal Apache release.  If that's the case then I'm +1 on letting them commit in this way this one time so we can all move on to 0.22.

Nige


From general-return-2817-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 17 22:57:48 2011
Return-Path: <general-return-2817-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4417 invoked from network); 17 Jan 2011 22:57:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jan 2011 22:57:48 -0000
Received: (qmail 60804 invoked by uid 500); 17 Jan 2011 22:57:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60761 invoked by uid 500); 17 Jan 2011 22:57:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60753 invoked by uid 99); 17 Jan 2011 22:57:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Jan 2011 22:57:45 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Jan 2011 22:57:36 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0HMut5D094448
	for <general@hadoop.apache.org>; Mon, 17 Jan 2011 14:56:55 -0800 (PST)
Received: from [10.0.1.4] (10.72.244.133) by SP2-EX07CAS01.ds.corp.yahoo.com
 (98.137.59.4) with Microsoft SMTP Server (TLS) id 8.3.106.1; Mon, 17 Jan 2011
 14:56:54 -0800
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <3FEB7C1D-AE89-46BE-ABF2-B595B01E222A@mac.com>
Date: Mon, 17 Jan 2011 14:56:51 -0800
Content-Transfer-Encoding: quoted-printable
Message-ID: <D160E684-7435-4E53-A766-231CCB38D1B4@yahoo-inc.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com> <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com> <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org> <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com> <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com> <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com> <4D34A270.1060102@apache.org> <3FEB7C1D-AE89-46BE-ABF2-B595B01E222A@mac.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Folks,

We are very interested in sharing what we are doing with the community.  =
I think we can separate this into multiple stages.

1) To doug's point - Yes, absolutely, we want folks to review this.  The =
patch is now available.  Lets work together to get it formatted as folks =
like in subversion and reviewed.  Where there are issues, let's work to =
resolve them.  With luck folks will find this work consistent and =
useful.  Backwards compatibility has been a goal, so with luck we will =
not ID regressions.  As todd mentioned earlier point, a lot of this work =
has already been merged into CDH and all of it has been reviewed by =
several apache committers already.=20

2) This code works, it is the best hadoop we know of.  If you run a =
business on hadoop, I think you would benefit from using it.  Right now =
you don't have the choice of an Apache release if you are looking for a =
stabilized modern version of Hadoop.  We would like to make apache =
releases based on it, source and binary, incorporating bugfixes from =
everyone.  To do that we would of course need to follow the Apache =
Hadoop release process, which requires the release master to produce a =
release candidate and the PMC to vote on the release.  Since that will =
require a formal future vote, no one will be surprised!

3) To nigel's point - I don't think this should distract anyone from =
working on 22 or other Hadoop contributions.  The 22 team will have the =
option of incorporating this work.   We think it will be a better =
release if they do, but that is their choice.  The majority of out =
effort at yahoo is not going into 0.20 (this branch), we are working on =
future features for hadoop.  This is branch is the stable code we use =
while we are waiting for a new release.

Thanks,

E14

On Jan 17, 2011, at 12:21 PM, Nigel Daley wrote:

>=20
> On Jan 17, 2011, at 12:11 PM, Doug Cutting wrote:
>=20
>> On 01/12/2011 11:07 PM, Arun C Murthy wrote:
>>> Thus, I think a jumbo patch should suffice. It will also ensure this =
can
>>> done quickly so that the community can then concentrate on 0.22 and =
beyond.
>>>=20
>>> However, I will (manually) ensure all relevant jiras are referenced =
in
>>> the CHANGES.txt and Release Notes for folks to see the contents of =
the
>>> release. This is the hardest part of the exercise. Also, this =
ensures
>>> that we can track these jiras for 0.22 as Eli suggested.
>>>=20
>>> Does that seem like a reasonable way forward? I'm happy to =
brainstorm.
>>=20
>> We would not release this until each change in it has been reviewed =
by the community, right?  Otherwise we may end up with changes in a 0.20 =
release that don't get approved when they're contributed to trunk and =
cause trunk to regress.  So I don't yet see the point of committing the =
mega patch since the community needs to review each individual change =
anyway, so we might wait until each is reviewed to commit it.
>=20
> Unless this is a code-only drop into a branch w/ no formal Apache =
release.  If that's the case then I'm +1 on letting them commit in this =
way this one time so we can all move on to 0.22.
>=20
> Nige
>=20


From general-return-2818-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 17 23:29:49 2011
Return-Path: <general-return-2818-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22113 invoked from network); 17 Jan 2011 23:29:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jan 2011 23:29:49 -0000
Received: (qmail 85964 invoked by uid 500); 17 Jan 2011 23:29:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85909 invoked by uid 500); 17 Jan 2011 23:29:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85901 invoked by uid 99); 17 Jan 2011 23:29:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Jan 2011 23:29:47 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Jan 2011 23:29:41 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p0HNTEVu007585
	for <general@hadoop.apache.org>; Mon, 17 Jan 2011 15:29:14 -0800 (PST)
Received: from [10.0.1.4] (10.72.244.133) by SP2-EX07CAS01.ds.corp.yahoo.com
 (98.137.59.4) with Microsoft SMTP Server (TLS) id 8.3.106.1; Mon, 17 Jan 2011
 15:29:13 -0800
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <AANLkTik+oTnp69ZNF7Hgtom1pTTBy+J_m=R8cQ3X+1W0@mail.gmail.com>
Date: Mon, 17 Jan 2011 15:29:10 -0800
Content-Transfer-Encoding: quoted-printable
Message-ID: <1BBA9772-9F30-44A1-A979-387D7891903E@yahoo-inc.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
 <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
 <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
 <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
 <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
 <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
 <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
 <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
 <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
 <AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com>
 <388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com>
 <AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com>
 <04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com>
 <AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com>
 <74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com>
 <AANLkTimLLrfYbaaOzbEV9dcsaMFLeMmdgx73FasdxGhA@mail.gmail.com>
 <B4C8E6B2-A343-449B-AE63-DE5E3F883ABA@yahoo-inc.com>
 <00F14279-2802-4CC3-91AC-481AE257FD8B@yahoo-inc.com>
 <AANLkTikqbRrU+3+y2nctu7RzKNb7+P8kd=4az+eQHzKW@mail.gmail.com>
 <58FF3525-B734-44A0-A5C6-45282A23F06F@yahoo-inc.com>
 <AANLkTik-=ZDac0iEbYuDHeBrRosQ3JNGvRbf7MxexQND@mail.gmail.com>
 <7D4397E2-B618-4BD1-8E1E-08B1C598B76F@yahoo-inc.com>
 <10BA9684-51FF-4332-A2FD-5F648E0AAF8C@Holsman.NET>
 <4E3C214A-16C6-46FF-908B-8EFB075A24E0@yahoo-inc.com>
 <AANLkTik+oTnp69ZNF7Hgtom1pTTBy+J_m=R8cQ3X+1W0@mail.gmail.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)

Hi Stack,

I feel your pain.  We're running a 700 node HBASE cluster containing a =
HUGE collections of all web pages.  Both versions of append were started =
by engineers working at yahoo and we've put A LOT of investment into =
both.  I really, really want to see the append issue solved for HBASE!! =20=


My point is simply that we need to separate our concerns.  I would 300% =
support a community of folks building a 0.20 derived version of hadoop =
with append and we know that any new release post 0.20 will contain an =
append solution.  This branch is more backwards facing.  We are simply =
trying to share our last two years of 0.20 experience with the =
community, so that a) folks can use it if they find value in it, b) this =
work can be merged into future hadoop releases (that will have append).

We want to share what we have tested, since we believe that the testing =
is a good chunk of our contribution.

Thanks,

E14

On Jan 16, 2011, at 2:57 PM, Stack wrote:

> On Fri, Jan 14, 2011 at 10:25 AM, Eric Baldeschwieler
> <eric14@yahoo-inc.com> wrote:
>> 2) append is hard. It is so hard we rewrote the entire write pipeline =
(5 person-years work) in trunk after giving up on the codeline you are =
suggesting we merge in. That work is what distinguishes all post 20 =
releases from 20 releases in my mind. I dont trust the 20 append code =
line. We've been hurt badly by it.  We did the rewrite only after losing =
a bunch of production data a bunch of times with the previous code line. =
 I think the various 20 append patch lines may be fine for specialized =
hbase clusters, but they doesn't have the rigor behind them to bet your =
business in them.
>>=20
>=20
> Eric:
>=20
> A few comments on the above:
>=20
> + Append has had a bunch of work done on it since the Y! dataloss of a
> few years ago on an ancestor of the branch-0.20-append codebase (IIRC
> the issue you refer to in particular -- the 'dataloss' because
> partially written blocks were done up in tmp dirs, and on cluster
> restart, tmp data was cleared -- has been fixed in
> branch-0.20.append).
> + You may not trust 0.20-append (or its close cousin over in CDH) but
> a bunch of HBasers do. On the one hand, we have little choice.  Until
> the *new* append becomes available in a stable Hadoop the HBase
> project has had to sustain itself (What you think?, 3-6 months before
> we see 0.22?  HBase project can't hold its breath that long).  On
> other hand, the branch-0.20-append work has been carried out by lads
> (and lasses!) who know their HDFS.  Its true that it will not have
> been tested with Y! rigor but near-derivatives -- CDH or the FB
> branches -- already do HDFS-200-based append in production.
>=20
> St.Ack
> P.S. Don't get me wrong.  HBase is looking forward to *new* append.
> We just need something to suck on meantime.


From general-return-2819-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 17 23:50:23 2011
Return-Path: <general-return-2819-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25930 invoked from network); 17 Jan 2011 23:50:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jan 2011 23:50:22 -0000
Received: (qmail 4305 invoked by uid 500); 17 Jan 2011 23:50:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4193 invoked by uid 500); 17 Jan 2011 23:50:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4185 invoked by uid 99); 17 Jan 2011 23:50:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Jan 2011 23:50:20 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 17 Jan 2011 23:50:18 +0000
Received: (qmail 25870 invoked by uid 99); 17 Jan 2011 23:49:57 -0000
Received: from localhost.apache.org (HELO [192.168.168.108]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Jan 2011 23:49:57 +0000
Message-ID: <4D34D59F.4010802@apache.org>
Date: Mon, 17 Jan 2011 15:49:51 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com> <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com> <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org> <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com> <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com> <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com> <4D34A270.1060102@apache.org> <3FEB7C1D-AE89-46BE-ABF2-B595B01E222A@mac.com> <D160E684-7435-4E53-A766-231CCB38D1B4@yahoo-inc.com>
In-Reply-To: <D160E684-7435-4E53-A766-231CCB38D1B4@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 01/17/2011 02:56 PM, Eric Baldeschwieler wrote:
> 1) To doug's point - Yes, absolutely, we want folks to review this.
> The patch is now available.  Lets work together to get it formatted
> as folks like in subversion and reviewed.  Where there are issues,
> let's work to resolve them.  With luck folks will find this work
> consistent and useful.

The question I was addressing was whether to commit the mega-patch 
as-is, or attempt to linearize it into a sequence of patches, one per 
issue addressed by the mega patch.  I believe that, as-is, it is 
probably too big to review as a unit.  I don't see that merely naming 
the changes in it makes it substantially easier to review.  Rather, each 
issue probably needs to be associated with a distinct patch in order to 
permit independent review.

> Backwards compatibility has been a goal, so
> with luck we will not ID regressions.

My point was that, in addition to back-compatibility with prior 0.20 
releases, we must also consider the forward-compatibility of each change 
with 0.21, 0.22 and trunk.

> As todd mentioned earlier
> point, a lot of this work has already been merged into CDH and all of
> it has been reviewed by several apache committers already.

Right, but review must be in public, so that everyone in the community 
has an equal chance to be involved in the development of each change.

Doug

From general-return-2820-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 01:08:44 2011
Return-Path: <general-return-2820-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67657 invoked from network); 18 Jan 2011 01:08:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 01:08:44 -0000
Received: (qmail 80983 invoked by uid 500); 18 Jan 2011 01:08:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80893 invoked by uid 500); 18 Jan 2011 01:08:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80872 invoked by uid 99); 18 Jan 2011 01:08:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 01:08:41 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of charles.fg@gmail.com designates 209.85.210.176 as permitted sender)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 01:08:35 +0000
Received: by iyb26 with SMTP id 26so6069948iyb.35
        for <general@hadoop.apache.org>; Mon, 17 Jan 2011 17:08:14 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:from:date:message-id:subject:to
         :content-type;
        bh=whay88gx6cx+bCHvU3+OzgrpqlSgouUnCIv9zrZbRCY=;
        b=mM41LZYDC8xa8EnR+GqJkSKKJC+JzQqhM6Aya8d+vtcRBMHUp3DK4nr/Tz8CXBUfvh
         RomLXwSfy0FLtrofaJiDinnCeCb8twxgG/elRNerB147jRhlUqV0L1lig3vhhcwtfTK5
         gfyPx5JXir2Ye+4d783+3VajJZ/MIPt1v6cq4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type;
        b=JEK/dHSP/UXvznyg4zsNPhJ/Rm99PuEqYz/o9ae5F/eeYRKPzSHjB56+4O439kebbv
         C4IS6Wpaa5SispQNUEX3pwCGIYSglGOlPBdjm4P6+3Dbkb1nus+wokUkv7tEx/45I9ds
         l8NAopGjpsw3BkXunp5Ynz8ldSONlLQ4Doi8s=
Received: by 10.42.164.132 with SMTP id g4mr5336965icy.127.1295312894462; Mon,
 17 Jan 2011 17:08:14 -0800 (PST)
MIME-Version: 1.0
Received: by 10.42.167.194 with HTTP; Mon, 17 Jan 2011 17:07:54 -0800 (PST)
From: =?ISO-8859-1?Q?Charles_Gon=E7alves?= <charles.fg@gmail.com>
Date: Mon, 17 Jan 2011 23:07:54 -0200
Message-ID: <AANLkTin8oUnJxouvkSBbdw8yyFcP6qQL2NgDEP_LTvGH@mail.gmail.com>
Subject: Manage a cluster where not all machines are always available
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba6e85f2f2890b049a148811
X-Virus-Checked: Checked by ClamAV on apache.org

--90e6ba6e85f2f2890b049a148811
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Guys,

I'm running a series of pig scripts in a cluster with a dozen of machines.
The problem is that those machines belongs to a lab in my University and
sometimes not all them are available for my use.
What is the best approach to manage the configuration and the data on hdfs
on this enviroment?

Can I simply remove the busy servers from the slaves file and start the hdf=
s
and mapred  and if needed perform a :
hadoop balancer

Can you see a problem in this approach ?
Can anyone see another way!?




--=20
*Charles Ferreira Gon=E7alves *
http://homepages.dcc.ufmg.br/~charles/
UFMG - ICEx - Dcc
Cel.: 55 31 87741485
Tel.:  55 31 34741485
Lab.: 55 31 34095840

--90e6ba6e85f2f2890b049a148811--

From general-return-2821-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 01:34:02 2011
Return-Path: <general-return-2821-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72116 invoked from network); 18 Jan 2011 01:34:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 01:34:02 -0000
Received: (qmail 2832 invoked by uid 500); 18 Jan 2011 01:34:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2780 invoked by uid 500); 18 Jan 2011 01:34:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2772 invoked by uid 99); 18 Jan 2011 01:34:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 01:33:59 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=MIME_BASE64_BLANKS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of msegel@navteq.com designates 204.120.70.37 as permitted sender)
Received: from [204.120.70.37] (HELO xmailfargo.navteq.com) (204.120.70.37)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 01:33:53 +0000
Received: from fargo-spamfw.navteq.com (navteq-spamfw.navteq.com [10.6.10.131])
	by xmailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id p0I14VMx028218
	for <general@hadoop.apache.org>; Mon, 17 Jan 2011 19:04:31 -0600
Received: from imailfargo.navteq.com (localhost [127.0.0.1])
	by fargo-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id 58FA543C4710
	for <general@hadoop.apache.org>; Mon, 17 Jan 2011 19:33:31 -0600 (CST)
Received: from imailfargo.navteq.com (imailfargo.navteq.com [10.6.10.130]) by fargo-spamfw.navteq.com with ESMTP id TXERsDUa9cEDRRX5 for <general@hadoop.apache.org>; Mon, 17 Jan 2011 19:33:31 -0600 (CST)
Received: from hq-ex-ht02.ad.navteq.com ([10.8.222.172])
	by imailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id p0I1XU9H010215
	for <general@hadoop.apache.org>; Mon, 17 Jan 2011 19:33:31 -0600
Received: from hq-ex-mb02.ad.navteq.com ([fe80::880d:318e:e860:a8db]) by
 hq-ex-ht02.ad.navteq.com ([fe80::24f6:ca50:734d:df6e%14]) with mapi; Mon, 17
 Jan 2011 19:32:33 -0600
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 17 Jan 2011 19:33:30 -0600
Subject: RE: Manage a cluster where not all machines are always available
Thread-Topic: Manage a cluster where not all machines are always available
Thread-Index: Acu2rEi++fsfYDAQT1+7nEC5zADXBQAAW2/o
Message-ID: <ABC24175AFD3BE4DA15F4CD375ED413D063A831642@hq-ex-mb02.ad.navteq.com>
References: <AANLkTin8oUnJxouvkSBbdw8yyFcP6qQL2NgDEP_LTvGH@mail.gmail.com>
In-Reply-To: <AANLkTin8oUnJxouvkSBbdw8yyFcP6qQL2NgDEP_LTvGH@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1
X-Virus-Checked: Checked by ClamAV on apache.org

DQpDaGFybGVzLA0KDQpJZiBJIHVuZGVyc3RhbmQgeW91IGNvcnJlY3RseSB5b3Ugd2FudCB0byB0
cmltIHRoZSBjbHVzdGVyIGRvd24gdG8gb25seSB0aG9zZSBtYWNoaW5lcyB0aGF0IHlvdSBjb250
cm9sLi4uDQoNCk9rLi4uIERvIHlvdSBjYXJlIGFib3V0IHRoZSBkYXRhIHRoYXQgaXMgY3VycmVu
dGx5IG9uIHRoZSBjbHVzdGVyPyANCihJcyBhbGwgb2YgdGhlIGRhdGEgeW91cnMsIG9yIHJlcGxh
Y2VhYmxlPykNCg0KQ2FuIHlvdSBlYXNpbHkgY29weSB0aGUgZGF0YSBvZmYgdGhlIGNsdXN0ZXIg
b24gdG8gcGxhaW4gb2xkIHVuaXggZmlsZSBzeXN0ZW0gZGlzayBzcGFjZT8NCg0KSWYgbm90LCB0
aGVuIHlvdSBoYXZlIHRvIGRvIHRoZSBmb2xsb3dpbmcgb24gYSBOT0RFIGJ5IE5PREUgQmFzaXMu
Li4NCkEpIFB1dCB0aGUgbm9kZSBpbiB0aGUgZGZzLmV4bHVkZSBmaWxlIGFuZCByZW1vdmUgZnJv
bSB0aGUgc2xhdmVzIGZpbGUuDQpCKSBBcyByb290IHJ1biBraWxsYWxsIC05IGphdmEgdG8gc3Rv
cCBhbnkgamF2YSBmcm9tIHJ1bm5pbmcuIChJdCB3aWxsIGVuZCB5b3VyIGRhdGFub2RlIGFuZCB0
YXNrdHJhY2tlciBqb2JzLikNCkMpIFdhaXQgMTAgbWlucyB1bnRpbCB0aGUgam9iIHRyYWNrZXIg
YW5kIG5hbWUgbm9kZSBzZWUgdGhlIG5vZGUgYXMgZG93bi4NCkQpIFJ1biBhIGhhZG9vcCBmc2Nr
IC8gdG8gZmluZCBhbGwgb2YgdGhlIGZpbGVzIHRoYXQgYXJlIG5vdyBtaXNzaW5nIGEgcmVwbGlj
YXRpb24uDQpFKSBSdW4gYmFsYW5jZXIgdG8gcmVwbGljYXRlIHRoZSBtaXNzaW5nIGJsb2NrcyBv
biBhIGRpZmZlcmVudCBtYWNoaW5lLg0KDQpPZiBjb3Vyc2UsIGl0IHdvdWxkIGhlbHAgaWYgeW91
IHVwcGVkIHRoZSBiYW5kd2lkdGggdXNlZCBieSB0aGUgYmFsYW5jZXIgdG8gYSBsYXJnZSBudW1i
ZXIuIE5vcm1hbGx5IHRoZSBiYWxhbmNlciBpcyBzdXBwb3NlZCB0byBydW4gaW4gdGhlIGJhY2tn
cm91bmQsIHNvIGJ5IGRlZmF1bHQgaXRzIHNvbWV0aGluZyBsaWtlIDEgTUIvc2VjLiBJZiB5b3Un
dmUgZ290IGEgMUdCIGV0aGVybmV0IGxpbmssIHlvdSBjb3VsZCBlYXNpbHkgcHVzaCB0aGF0IG51
bWJlciB1cCB0byAxMDAgb3IgMjAwIE1CL3NlYy4gVGhlbiB3aGVuIHlvdSBydW4gdGhlIGJhbGFu
Y2VyLCBpdCBtb3ZlcyENCg0KTm90ZTogV2hlbiB3ZSB0cmllZCBkZWNvbW1pc3Npb25pbmcgbm9k
ZXMsIEkgZG9uJ3Qga25vdyBpZiB3ZSBoYWQgY2hhbmdlZCB0aGlzIHBhcmFtZXRlciwgYnV0IGl0
IHdhcyB0YWtpbmcgJ3dlZWtzJyB0byBkZWNvbW1pc3Npb24gYSBub2RlLiAoWW91ciBNaWxlYWdl
IE1heSBWYXJ5KS4gTm90IHN1cmUgaWYgdGhlIGxvbmcgdGltZSB3YXMgZHVlIHRvIHRoaXMgcGFy
YW1ldGVyIGJlaW5nIHNvIGxvdywgb3Igc29tZXRoaW5nIGVsc2UuIA0KDQpXaGF0IEkgbGlzdGVk
IGFib3ZlIHNob3VsZCB3b3JrLiAoRXZlbiBpZiBpdCBpcyBhIGJpdCB1Z2x5LikNCg0KSFRIDQoN
Ci1NaWtlDQoNCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18NCkZyb206
IENoYXJsZXMgR29u52FsdmVzIFtjaGFybGVzLmZnQGdtYWlsLmNvbV0NClNlbnQ6IE1vbmRheSwg
SmFudWFyeSAxNywgMjAxMSA3OjA3IFBNDQpUbzogZ2VuZXJhbEBoYWRvb3AuYXBhY2hlLm9yZw0K
U3ViamVjdDogTWFuYWdlIGEgY2x1c3RlciB3aGVyZSBub3QgYWxsIG1hY2hpbmVzIGFyZSBhbHdh
eXMgYXZhaWxhYmxlDQoNCkhpIEd1eXMsDQoNCkknbSBydW5uaW5nIGEgc2VyaWVzIG9mIHBpZyBz
Y3JpcHRzIGluIGEgY2x1c3RlciB3aXRoIGEgZG96ZW4gb2YgbWFjaGluZXMuDQpUaGUgcHJvYmxl
bSBpcyB0aGF0IHRob3NlIG1hY2hpbmVzIGJlbG9uZ3MgdG8gYSBsYWIgaW4gbXkgVW5pdmVyc2l0
eSBhbmQNCnNvbWV0aW1lcyBub3QgYWxsIHRoZW0gYXJlIGF2YWlsYWJsZSBmb3IgbXkgdXNlLg0K
V2hhdCBpcyB0aGUgYmVzdCBhcHByb2FjaCB0byBtYW5hZ2UgdGhlIGNvbmZpZ3VyYXRpb24gYW5k
IHRoZSBkYXRhIG9uIGhkZnMNCm9uIHRoaXMgZW52aXJvbWVudD8NCg0KQ2FuIEkgc2ltcGx5IHJl
bW92ZSB0aGUgYnVzeSBzZXJ2ZXJzIGZyb20gdGhlIHNsYXZlcyBmaWxlIGFuZCBzdGFydCB0aGUg
aGRmcw0KYW5kIG1hcHJlZCAgYW5kIGlmIG5lZWRlZCBwZXJmb3JtIGEgOg0KaGFkb29wIGJhbGFu
Y2VyDQoNCkNhbiB5b3Ugc2VlIGEgcHJvYmxlbSBpbiB0aGlzIGFwcHJvYWNoID8NCkNhbiBhbnlv
bmUgc2VlIGFub3RoZXIgd2F5IT8NCg0KDQoNCg0KLS0NCipDaGFybGVzIEZlcnJlaXJhIEdvbudh
bHZlcyAqDQpodHRwOi8vaG9tZXBhZ2VzLmRjYy51Zm1nLmJyL35jaGFybGVzLw0KVUZNRyAtIElD
RXggLSBEY2MNCkNlbC46IDU1IDMxIDg3NzQxNDg1DQpUZWwuOiAgNTUgMzEgMzQ3NDE0ODUNCkxh
Yi46IDU1IDMxIDM0MDk1ODQwDQoNCg0KVGhlIGluZm9ybWF0aW9uIGNvbnRhaW5lZCBpbiB0aGlz
IGNvbW11bmljYXRpb24gbWF5IGJlIENPTkZJREVOVElBTCBhbmQgaXMgaW50ZW5kZWQgb25seSBm
b3IgdGhlIHVzZSBvZiB0aGUgcmVjaXBpZW50KHMpIG5hbWVkIGFib3ZlLiAgSWYgeW91IGFyZSBu
b3QgdGhlIGludGVuZGVkIHJlY2lwaWVudCwgeW91IGFyZSBoZXJlYnkgbm90aWZpZWQgdGhhdCBh
bnkgZGlzc2VtaW5hdGlvbiwgZGlzdHJpYnV0aW9uLCBvciBjb3B5aW5nIG9mIHRoaXMgY29tbXVu
aWNhdGlvbiwgb3IgYW55IG9mIGl0cyBjb250ZW50cywgaXMgc3RyaWN0bHkgcHJvaGliaXRlZC4g
IElmIHlvdSBoYXZlIHJlY2VpdmVkIHRoaXMgY29tbXVuaWNhdGlvbiBpbiBlcnJvciwgcGxlYXNl
IG5vdGlmeSB0aGUgc2VuZGVyIGFuZCBkZWxldGUvZGVzdHJveSB0aGUgb3JpZ2luYWwgbWVzc2Fn
ZSBhbmQgYW55IGNvcHkgb2YgaXQgZnJvbSB5b3VyIGNvbXB1dGVyIG9yIHBhcGVyIGZpbGVzLg0K


From general-return-2822-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 01:42:18 2011
Return-Path: <general-return-2822-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73096 invoked from network); 18 Jan 2011 01:42:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 01:42:17 -0000
Received: (qmail 7361 invoked by uid 500); 18 Jan 2011 01:42:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7146 invoked by uid 500); 18 Jan 2011 01:42:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7138 invoked by uid 99); 18 Jan 2011 01:42:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 01:42:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 18 Jan 2011 01:42:15 +0000
Received: (qmail 73006 invoked by uid 99); 18 Jan 2011 01:41:55 -0000
Received: from localhost.apache.org (HELO mail-iw0-f176.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 01:41:55 +0000
Received: by iwn2 with SMTP id 2so5986565iwn.35
        for <general@hadoop.apache.org>; Mon, 17 Jan 2011 17:41:54 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.35.204 with SMTP id q12mr5109624ibd.191.1295314914386;
 Mon, 17 Jan 2011 17:41:54 -0800 (PST)
Received: by 10.231.167.199 with HTTP; Mon, 17 Jan 2011 17:41:54 -0800 (PST)
In-Reply-To: <4D34A270.1060102@apache.org>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
	<366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
	<18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
	<AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
	<F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
	<D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
	<2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
	<5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
	<075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
	<4D34A270.1060102@apache.org>
Date: Mon, 17 Jan 2011 17:41:54 -0800
Message-ID: <AANLkTikqU0xGWq15XEgOP3udEoEBtAzFmyfAb0AUhOpC@mail.gmail.com>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Mon, Jan 17, 2011 at 12:11 PM, Doug Cutting <cutting@apache.org> wrote:
> We would not release this until each change in it has been reviewed by th=
e
> community, right? =A0Otherwise we may end up with changes in a 0.20 relea=
se
> that don't get approved when they're contributed to trunk and cause trunk=
 to
> regress. =A0So I don't yet see the point of committing the mega patch sin=
ce
> the community needs to review each individual change anyway, so we might
> wait until each is reviewed to commit it.

I share this concern. Releasing an omnibus pile of commits in the 0.20
series will create an impossible situation for the mainline. Worse,
the alternative sifts through this pile over months, as the
refinements wrought by consensus require remerging and revalidating of
each issue. Every subsequent issue must also be reconsidered. The
product must then be deployed, tested, and its bugs fixed, just to get
a release as battle-hardened as this one. Signing up for all this work
when most every developer and user would rather see trunk proceed
would be madness.

However, the status quo is also unacceptable. Running any version of
Apache Hadoop is rare, when compared to the popularity of its
variants. We must find a solution to that. Hadoop is not in good shape
right now, and exceptional actions to correct it should not be cast
off lightly by valuing consistency over its future.

To address Nigel and Doug's concerns about compatibility, we should
consider a different release series. We wanted to postpone 1.0
discussions, but that would be one solution. If a secure 0.20 could be
released as 1.0, then if interest in this branch persists, append
could be a 1.1 release on this series,* etc. while 0.22 and its
successors can be 2.0 (as a rare benefit to the project split, one
could argue that "Hadoop" is the unified set, and the Common, HDFS,
and MapReduce projects could continue to release on the 0.x series
until we want to declare those a stable successor to 0.20). Version
numbers are pretty cheap, when compared to our time and focus.

* In the interim, a 0.20-append release would make all kinds of sense,
and fie on the niceties of naming.

> That said, posting the mega patch is useful, so that folks can start to p=
ick
> it apart into separate issues. =A0Pushing your internal commits to a publ=
ic
> github branch might also make that review process easier.

Pushing to github caused this problem. CDH rebased on YDH, and today
Apache Hadoop is considered less stable, less tested, and less usable
than either one of them. Why one would expect things to work
differently this time around is not clear. I assume we all agree it's
a poor outcome.

Arun already volunteered to break up the commits and push individual
patches to the repository, so the history is manageable. We allow CTR
for branches, though it's predicated on the assumption that that work
will be spread over weeks or months; development should not be batched
this way. However, by adding obstacles to an unambiguously positive
outcome, collaborators will be skeptical of engaging more deeply with
this community. Let's focus on making forward progress, not on
ensuring the requisite pain is felt. -C

From general-return-2823-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 02:56:24 2011
Return-Path: <general-return-2823-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95159 invoked from network); 18 Jan 2011 02:56:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 02:56:23 -0000
Received: (qmail 48093 invoked by uid 500); 18 Jan 2011 02:56:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48036 invoked by uid 500); 18 Jan 2011 02:56:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48028 invoked by uid 99); 18 Jan 2011 02:56:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 02:56:20 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 02:56:10 +0000
Received: from SP2-EX07CAS03.ds.corp.yahoo.com (sp2-ex07cas03.corp.sp2.yahoo.com [98.137.59.35])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0I2tOAe045256
	for <general@hadoop.apache.org>; Mon, 17 Jan 2011 18:55:24 -0800 (PST)
Received: from SP2-EX07VS07.ds.corp.yahoo.com ([98.137.59.26]) by
 SP2-EX07CAS03.ds.corp.yahoo.com ([98.137.59.35]) with mapi; Mon, 17 Jan 2011
 18:55:23 -0800
From: Todd Papaioannou <toddp@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 17 Jan 2011 18:55:22 -0800
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Thread-Topic: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Thread-Index: Acu2uyamIJc5VpY2R0ih27883g+MZQ==
Message-ID: <C95A3F70.3A81%toddp@yahoo-inc.com>
In-Reply-To: <4D34D59F.4010802@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-MacOutlook/14.2.0.101115
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C95A3F703A81toddpyahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C95A3F703A81toddpyahooinccom_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

That's only true if you plan to pull forward the changes wholesale into .21=
, .22 and beyond. And that is not what is being proposed.

If the plan is to just land an updated and more stable version of .20 that =
is completely backwards compatible, then this can be done within that code =
line without any impact to the end users. Any changes that the community wi=
sh to pull forward can be identified, isolated and reviewed per the normal =
process. Or they can remain in the .20.100 release for eternity, without an=
y impact on the future.

Either way, the .20 release will be more stable, performant and more useful=
 to our users, and the community at large can focus on releasing .22, which=
 we all believe is the right goal.

ToddP

From: Doug Cutting <cutting@apache.org<mailto:cutting@apache.org>>
Reply-To: "general@hadoop.apache.org<mailto:general@hadoop.apache.org>" <ge=
neral@hadoop.apache.org<mailto:general@hadoop.apache.org>>
Date: Mon, 17 Jan 2011 15:49:51 -0800
To: "general@hadoop.apache.org<mailto:general@hadoop.apache.org>" <general@=
hadoop.apache.org<mailto:general@hadoop.apache.org>>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset


Backwards compatibility has been a goal, so
with luck we will not ID regressions.

My point was that, in addition to back-compatibility with prior 0.20
releases, we must also consider the forward-compatibility of each change
with 0.21, 0.22 and trunk.

--_000_C95A3F703A81toddpyahooinccom_--

From general-return-2824-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 03:05:22 2011
Return-Path: <general-return-2824-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4633 invoked from network); 18 Jan 2011 03:05:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 03:05:22 -0000
Received: (qmail 53677 invoked by uid 500); 18 Jan 2011 03:05:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53397 invoked by uid 500); 18 Jan 2011 03:05:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53389 invoked by uid 99); 18 Jan 2011 03:05:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 03:05:17 +0000
X-ASF-Spam-Status: No, hits=3.7 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of liulei412@gmail.com designates 209.85.213.48 as permitted sender)
Received: from [209.85.213.48] (HELO mail-yw0-f48.google.com) (209.85.213.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 03:05:09 +0000
Received: by ywo7 with SMTP id 7so2171969ywo.35
        for <general@hadoop.apache.org>; Mon, 17 Jan 2011 19:04:49 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=eTQRI7RIajUhCURXURVDn4epunOjpGMXsvi0Mlp/rrE=;
        b=Ktm729JfX7Y0iUZJDdsdZXpjSZnlXWqFsqp7KlRO8kzFvWZOrIenGvUuGXocSSaJh9
         E6qCQrae+ROaW9IFpz5lj1zGdVZGJFP90EioQ/d4WRFrn47cKgMLYsC20+uo6JO0b5un
         TKL23xz/RWC5Wrso1XVBLgGaX3Mv9zL2doMCU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=OIa13SquPOJyK+xeZWw1NsbnwM4wwsiJGYUgxYwGIs6ZvSy9+DKPAHmH9PUNMSDCBf
         sEeALgOdnPUhu/vi6Tpl5not6E6ttyqYe2+Box+GD0SsjqAavwACuy2KIEe8eHBCGXAg
         pg/vdDAi29wGbIiXLWZrAqtiONAEPhHefHLKI=
MIME-Version: 1.0
Received: by 10.150.185.4 with SMTP id i4mr507442ybf.205.1295319888778; Mon,
 17 Jan 2011 19:04:48 -0800 (PST)
Received: by 10.150.201.20 with HTTP; Mon, 17 Jan 2011 19:04:48 -0800 (PST)
Date: Tue, 18 Jan 2011 11:04:48 +0800
Message-ID: <AANLkTik6ycyxketrVd88gTxTzFgpx6DKUybiBJiTnmU1@mail.gmail.com>
Subject: how to know which partition data the reduce receive
From: lei liu <liulei412@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd4d53cd75682049a16291b
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd4d53cd75682049a16291b
Content-Type: text/plain; charset=GB2312
Content-Transfer-Encoding: quoted-printable

There is job that has three reduce tasks that is the map output data
are divided
into three partition,  how can I know every reduce receive which partition=
=A3=BF

--000e0cd4d53cd75682049a16291b--

From general-return-2825-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 03:44:45 2011
Return-Path: <general-return-2825-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 13571 invoked from network); 18 Jan 2011 03:44:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 03:44:45 -0000
Received: (qmail 68834 invoked by uid 500); 18 Jan 2011 03:44:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68541 invoked by uid 500); 18 Jan 2011 03:44:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68533 invoked by uid 99); 18 Jan 2011 03:44:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 03:44:39 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 03:44:33 +0000
Received: by fxm2 with SMTP id 2so7140075fxm.35
        for <general@hadoop.apache.org>; Mon, 17 Jan 2011 19:44:12 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type:content-transfer-encoding;
        bh=PZn3Sh60UfP7fy11nhcje06Q06yI+VyLoupH+orP1zE=;
        b=nQSvY5RK+s2Xrh3OZotc2SjZ95IaRp0uZ4ZXVjqVlIqa7suoL6aDaWLq0IK0dNetdJ
         ZyfUYIC5lPQI35juGnnyPPCsrh1NHjL2BBIN/WaYlJsUt1unjQ/fVw782jZpwGDEyx8M
         qLR/x5cntAM2Wm4q6QzXU7SQkXVtyk1CVrXJg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=FL/9tTfgiJcmVh36W9GcthQPC1fXgsGM043xPN1qV/1aH59lzbwj+5bu0ZTukGeUBU
         oPuuKOMCrkBY3src2187i2wnSMPHHgjaePlsgztbkI02updeyc/4oLso9rogo1hz67UZ
         FedFWoKpv34cpuBNtWpCC4hACHf63OoDhwOFs=
Received: by 10.223.85.203 with SMTP id p11mr5576074fal.108.1295322252702;
 Mon, 17 Jan 2011 19:44:12 -0800 (PST)
MIME-Version: 1.0
Received: by 10.223.124.200 with HTTP; Mon, 17 Jan 2011 19:43:52 -0800 (PST)
In-Reply-To: <AANLkTik6ycyxketrVd88gTxTzFgpx6DKUybiBJiTnmU1@mail.gmail.com>
References: <AANLkTik6ycyxketrVd88gTxTzFgpx6DKUybiBJiTnmU1@mail.gmail.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Tue, 18 Jan 2011 09:13:52 +0530
Message-ID: <AANLkTik3MH5RaooebFux82aO5GQ+TNk-raAgVkyqAp_R@mail.gmail.com>
Subject: Re: how to know which partition data the reduce receive
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

The emitted Partition Number =3D=3D Reducers' ID.

2011/1/18 lei liu <liulei412@gmail.com>:
> There is job that has three reduce tasks that is the map output data
> are divided
> into three partition,  how can I know every reduce receive which partitio=
n=EF=BC=9F
>



--=20
Harsh J
www.harshj.com

From general-return-2826-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 04:31:02 2011
Return-Path: <general-return-2826-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29327 invoked from network); 18 Jan 2011 04:31:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 04:31:01 -0000
Received: (qmail 96282 invoked by uid 500); 18 Jan 2011 04:31:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95936 invoked by uid 500); 18 Jan 2011 04:30:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95928 invoked by uid 99); 18 Jan 2011 04:30:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 04:30:55 +0000
X-ASF-Spam-Status: No, hits=2.8 required=10.0
	tests=FRT_TODAY2,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 04:30:50 +0000
Received: by iwn2 with SMTP id 2so6094186iwn.35
        for <general@hadoop.apache.org>; Mon, 17 Jan 2011 20:30:30 -0800 (PST)
MIME-Version: 1.0
Received: by 10.42.213.74 with SMTP id gv10mr5601045icb.213.1295325029967;
 Mon, 17 Jan 2011 20:30:29 -0800 (PST)
Received: by 10.42.167.66 with HTTP; Mon, 17 Jan 2011 20:30:29 -0800 (PST)
In-Reply-To: <C95A3F70.3A81%toddp@yahoo-inc.com>
References: <4D34D59F.4010802@apache.org>
	<C95A3F70.3A81%toddp@yahoo-inc.com>
Date: Mon, 17 Jan 2011 20:30:29 -0800
Message-ID: <AANLkTimsXa2rdpV0ahoy2pOjoXDzaOaxkYz3snVHR4th@mail.gmail.com>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf3054ac4947a945049a175c33

--20cf3054ac4947a945049a175c33
Content-Type: text/plain; charset=ISO-8859-1

Hey,

We had this exact same discussion about the 0.20-append branch a few weeks
ago. A few organizations have tested that code at scale and feel strongly
that it's stable. We decided not to release it because it does not meet the
Apache guidelines for a release. The Apache process has its pros and cons;
we've all accepted them, so the community moved on and focused its energy on
the 0.22 release.

A few weeks later, we now have another organization claiming that their
0.20-based branch is tested at scale and should be released. It's claimed
that 0.20.100 will be "more stable, performant and more useful to our
users"; the same can be said of the 0.20-append branch. Neither branch,
however, is a bugfix release and thus does not meet the Apache guidelines
for a release. That's too bad; we should work to avoid this situation again
in the future, but let's not try to change the rules because we did a poor
job in the past of getting our work released via Apache.

As Nigel mentions, and as was done with 0.20-append, I would fully support a
"a code-only drop into a branch w/ no formal Apache release". That's fully
compliant with the Apache process.

All of these discussions will be moot once we get 0.22 out the door and stop
arguing over which organization has the most magical 0.20-based bits. I'm
looking forward to seeing all of the Apache Hadoop contributors working full
time on that release process once these bits are committed to the 0.20.100
branch.

Thanks,
Jeff

On Mon, Jan 17, 2011 at 6:55 PM, Todd Papaioannou <toddp@yahoo-inc.com>wrote:

> That's only true if you plan to pull forward the changes wholesale into
> .21, .22 and beyond. And that is not what is being proposed.
>
> If the plan is to just land an updated and more stable version of .20 that
> is completely backwards compatible, then this can be done within that code
> line without any impact to the end users. Any changes that the community
> wish to pull forward can be identified, isolated and reviewed per the normal
> process. Or they can remain in the .20.100 release for eternity, without any
> impact on the future.
>
> Either way, the .20 release will be more stable, performant and more useful
> to our users, and the community at large can focus on releasing .22, which
> we all believe is the right goal.
>
> ToddP
>
> From: Doug Cutting <cutting@apache.org<mailto:cutting@apache.org>>
> Reply-To: "general@hadoop.apache.org<mailto:general@hadoop.apache.org>" <
> general@hadoop.apache.org<mailto:general@hadoop.apache.org>>
> Date: Mon, 17 Jan 2011 15:49:51 -0800
> To: "general@hadoop.apache.org<mailto:general@hadoop.apache.org>" <
> general@hadoop.apache.org<mailto:general@hadoop.apache.org>>
> Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
>
>
> Backwards compatibility has been a goal, so
> with luck we will not ID regressions.
>
> My point was that, in addition to back-compatibility with prior 0.20
> releases, we must also consider the forward-compatibility of each change
> with 0.21, 0.22 and trunk.
>

--20cf3054ac4947a945049a175c33--

From general-return-2827-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 04:32:39 2011
Return-Path: <general-return-2827-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29797 invoked from network); 18 Jan 2011 04:32:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 04:32:39 -0000
Received: (qmail 98565 invoked by uid 500); 18 Jan 2011 04:32:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98304 invoked by uid 500); 18 Jan 2011 04:32:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98296 invoked by uid 99); 18 Jan 2011 04:32:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 04:32:34 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 04:32:29 +0000
Received: from [192.168.1.71] (snvvpn3-10-72-77-c11.hq.corp.yahoo.com [10.72.77.11])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0I4W2CQ077473
	for <general@hadoop.apache.org>; Mon, 17 Jan 2011 20:32:02 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1295325122;
	bh=qnyi6UxZDlFLYjrM1ufevjmAR5JsD28NQx104/BTIAI=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:Mime-Version:Subject:
	 Date:References;
	b=CGU/RKmyoe4HC4ekKEQ8qxrCbX/JNZ7E3culVkM867PWyrBe40y0ODxnzgTmkgd7a
	 aWuWSiOvlNN77Jlaa2UviGeiIwRrd4GDDPlJ6m8gpBoNXPS1yh3ZYZsP+YOFROGtXn
	 FyXCiHVu7nlXZAE1gx/odKW5NIVmb/MnvgxB/B6I=
Message-Id: <6250B06D-2502-47E6-950D-3015485C516B@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <4D34A270.1060102@apache.org>
Content-Type: multipart/alternative; boundary=Apple-Mail-88--809650960
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Date: Mon, 17 Jan 2011 20:32:08 -0800
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com> <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com> <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org> <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com> <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com> <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com> <4D34A270.1060102@apache.org>
X-Mailer: Apple Mail (2.936)

--Apple-Mail-88--809650960
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit

On 1/17/11 12:11 PM, "Doug Cutting" <cutting@apache.org> wrote:
>
> We would not release this until each change in it has been reviewed by
> the community, right?  Otherwise we may end up with changes in a 0.20
> release that don't get approved when they're contributed to trunk and
> cause trunk to regress.  So I don't yet see the point of committing  
> the
> mega patch since the community needs to review each individual change
> anyway, so we might wait until each is reviewed to commit it.

My take is straight-forward:

Apache Hadoop hasn't had a stable, updated release in a while.

As a result there is too much confusion for the user-community. There  
are too many releases done by too many entities and nothing is  
available from Apache, for a long while now. This is a situation we  
need to rectify, urgently!

Engaging in community review of these patches will distract the  
developer community's attention from 0.22 and the future. Not to  
mention, it will take forever and keep users hanging. Yes, the  
mechanics are important - but not more important than the end result.

IAC:
a) The vast majority of these patches are already on jira, and have  
been for several, several months now.
b) The vast majority of these patches have already been committed to  
trunk i.e. 0.22.

Sure, some patches maybe missing from 0.22 or jira; my proposal is not  
ideal and - I don't think anyone is pretending it is.

However, it does remedy the critical problem - a stable, updated  
Apache Hadoop release.

We can remedy backward or forward compatibility by being clever with  
our release versions or names.

An appeal: Let's use a bit of common sense and get the project moving  
forward with a release. Folks are welcome to put forward a append  
release and an append+security release and so forth (I've strongly  
supported that), not to mention 0.22 and beyond. IMHO, more than one  
release is definitely better than none.

Let's get the ball rolling, please!

thanks,
Arun
--Apple-Mail-88--809650960--

From general-return-2828-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 04:40:41 2011
Return-Path: <general-return-2828-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31150 invoked from network); 18 Jan 2011 04:40:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 04:40:41 -0000
Received: (qmail 2605 invoked by uid 500); 18 Jan 2011 04:40:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2379 invoked by uid 500); 18 Jan 2011 04:40:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2370 invoked by uid 99); 18 Jan 2011 04:40:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 04:40:36 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 04:40:29 +0000
Received: by iyb26 with SMTP id 26so6206410iyb.35
        for <general@hadoop.apache.org>; Mon, 17 Jan 2011 20:40:09 -0800 (PST)
MIME-Version: 1.0
Received: by 10.42.172.65 with SMTP id m1mr2855201icz.481.1295325609124; Mon,
 17 Jan 2011 20:40:09 -0800 (PST)
Received: by 10.42.167.66 with HTTP; Mon, 17 Jan 2011 20:40:09 -0800 (PST)
In-Reply-To: <6250B06D-2502-47E6-950D-3015485C516B@yahoo-inc.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
	<366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
	<18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
	<AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
	<F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
	<D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
	<2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
	<5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
	<075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
	<4D34A270.1060102@apache.org>
	<6250B06D-2502-47E6-950D-3015485C516B@yahoo-inc.com>
Date: Mon, 17 Jan 2011 20:40:09 -0800
Message-ID: <AANLkTik5CiL3gEd+nKq5401-S2CZu9LMKFFA3AKvLcXQ@mail.gmail.com>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba613bfccce482049a177e5c

--90e6ba613bfccce482049a177e5c
Content-Type: text/plain; charset=ISO-8859-1

>
> Apache Hadoop hasn't had a stable, updated release in a while.
>

That's what 0.22 is for?

However, it does remedy the critical problem - a stable, updated Apache
> Hadoop release.
>

Again, isn't that what 0.22 is for?


> An appeal: Let's use a bit of common sense and get the project moving
> forward with a release. Folks are welcome to put forward a append release
> and an append+security release and so forth (I've strongly supported that),
> not to mention 0.22 and beyond. IMHO, more than one release is definitely
> better than none.
>

Yes, 0.22 has both append and security. 0.22 also has the nice feature of
following the Apache release guidelines rather than relying on the patch set
of an independent entity, whether it's Cloudera, Facebook, or Yahoo.


> Let's get the ball rolling, please!
>

Agreed! Nigel has done a great job getting the ball rolling on the 0.22
release. I'm looking forward to seeing everyone burn down the blockers that
have been identified.

Regards,
Jeff

--90e6ba613bfccce482049a177e5c--

From general-return-2829-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 04:53:19 2011
Return-Path: <general-return-2829-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 32446 invoked from network); 18 Jan 2011 04:53:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 04:53:19 -0000
Received: (qmail 9425 invoked by uid 500); 18 Jan 2011 04:53:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 9230 invoked by uid 500); 18 Jan 2011 04:53:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 9219 invoked by uid 99); 18 Jan 2011 04:53:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 04:53:14 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.101 as permitted sender)
Received: from [17.148.16.101] (HELO asmtpout026.mac.com) (17.148.16.101)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 04:53:06 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; charset=us-ascii
Received: from [10.0.1.13]
 (c-71-198-192-174.hsd1.ca.comcast.net [71.198.192.174])
 by asmtp026.mac.com (Sun Java(tm) System Messaging Server 6.3-7.04 (built Sep
 26 2008; 64bit)) with ESMTPSA id <0LF7009BCC7TBU70@asmtp026.mac.com> for
 general@hadoop.apache.org; Mon, 17 Jan 2011 20:52:42 -0800 (PST)
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101170234
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-18_01:2011-01-17,2011-01-18,1970-01-01 signatures=0
Subject: Re: [DISCUSS] Move project split down a level
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <AANLkTikpfCf7n85NR_t2Krfa4a+h9mk0PEN_yLpQbSJb@mail.gmail.com>
Date: Mon, 17 Jan 2011 20:52:41 -0800
Message-id: <55BC6A08-29CB-47BF-B71B-9A4A79CF89DD@mac.com>
References: <113546.32737.qm@web56204.mail.re3.yahoo.com>
 <C955EB7A.2E753%sureshms@yahoo-inc.com>
 <AANLkTikpfCf7n85NR_t2Krfa4a+h9mk0PEN_yLpQbSJb@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

Just to be clear, the proposal currently being discussed is NOT a full undo of the split -- it might be better described as a tweak or a bug fix to the (on-going) project split. If someone would like to start a discussion on a complete undo of the project split, please do so under a different thread.

Cheers,
Nige


On Jan 14, 2011, at 1:39 PM, Ryan Rawson wrote:

> +1 the post split scripts are the worst things ever.


From general-return-2830-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 05:05:14 2011
Return-Path: <general-return-2830-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39022 invoked from network); 18 Jan 2011 05:05:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 05:05:12 -0000
Received: (qmail 17868 invoked by uid 500); 18 Jan 2011 05:05:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17152 invoked by uid 500); 18 Jan 2011 05:05:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17135 invoked by uid 99); 18 Jan 2011 05:05:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 05:05:05 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,LONG_TERM_PRICE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.97 as permitted sender)
Received: from [17.148.16.97] (HELO asmtpout022.mac.com) (17.148.16.97)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 05:04:57 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13]
 (c-71-198-192-174.hsd1.ca.comcast.net [71.198.192.174])
 by asmtp022.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LF700MC8CRGQ780@asmtp022.mac.com>
 for general@hadoop.apache.org; Mon, 17 Jan 2011 21:04:29 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-18_01:2011-01-17,2011-01-18,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101170237
Subject: Re: [DISCUSS] Move project split down a level
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <C955EB7A.2E753%sureshms@yahoo-inc.com>
Date: Mon, 17 Jan 2011 21:04:28 -0800
Message-id: <E2932D45-FF26-4579-A975-A98D2578622A@mac.com>
References: <C955EB7A.2E753%sureshms@yahoo-inc.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

Good questions.  Keep them coming!  I'll compile a list so we can start an FAQ on this.

> # Is project split a goal for hadoop in the future (even though we are not ready yet?). If it is, then putting projects back together might result in tight dependencies between the project. Ho do we avoid it?


Note, we're not putting them back together.  This is NOT a cmd-Z (ctrl-Z) on the project split.  It's putting them back under one trunk, but as separate projects underneath that.  IMO this is a relatively small change in the universe of undo-the-project-split possibilities.

> # The committer list for each of the sub project today is different. How do we reconcile them?

I'd like to keep that issue out of this change if at all possible.  I recommend for now we keep the status quo.  Thus, even though all committers may technically have permission to commit to all 3 project trees (can someone confirm that?), we would need to rely on the honor system that committers will only commit to their project trees.

Cheers,
Nige

On Jan 14, 2011, at 12:00 PM, Suresh Srinivas wrote:

> I like the idea of merging projects together. It save a lot of time.
> 
> However, I would like to see a detailed proposal on how this will be done and discussions on it, before moving forward on this. If this work is done, need clear messages to the developers on what has changed, and how development process is affected. These details were missing when project split was done, causing great deal of confusion and pain.
> 
> We should also address the following:
> # Is project split a goal for hadoop in the future (even though we are not ready yet?). If it is, then putting projects back together might result in tight dependencies between the project. Ho do we avoid it?
> # The committer list for each of the sub project today is different. How do we reconcile them?
> 
> 
> On 1/14/11 11:53 AM, "Tsz Wo (Nicholas), Sze" <s29752-hadoopgeneral@yahoo.com> wrote:
> 
> This is a kind of an incompatible change: all the developers, QAs, release
> engineers and users have to change their local settings and scripts for this
> change.  Moreover, there are documentations, web pages and existing tools using
> the Apache svn URLs.  So it is a huge impact.  I am conservative on this since,
> as Konstantin mentioned, we risk to get into the same mess, and it will create
> more work for the community.
> 
> Why do we want to enforce the releases as a unit, given that the long term
> target is to release these 3 projects independently?
> 
> Nicholas
> 
> 
> 
> 
> 
> ________________________________
> From: Nigel Daley <ndaley@mac.com>
> To: general@hadoop.apache.org
> Sent: Fri, January 14, 2011 11:21:25 AM
> Subject: Re: [DISCUSS] Move project split down a level
> 
> 
> On Jan 14, 2011, at 11:16 AM, Tsz Wo (Nicholas), Sze wrote:
> 
>> Hi Nigel,
>> 
>>> As I look more at the impact of the common/MR/HDFS project split on what
>>> and how we release Hadoop, I feel like the split needs an adjustment.  Many
>>> folks I've talked to agree that the project split has caused us a splitting
>>> headache.  I think 1 relatively small change could alleviate some of that.
>> 
>> Could you elaborate your idea on how the proposed changes would help?  What the
>> 
>> problems are being addressed?  It is not clear to me.
> 
> Critical in my mind was my statement: "We're a long way from releasing these 3
> projects independently.  Given that, they should be branched and released as a
> unit."  This can not be enforced given the current svn layout. Other's can weigh
> in with additional thoughts.
> 
>> You are right that the change is small but the impact is huge.  We should first
>> 
>> understand what we are getting from the changes before doing it.
> 
> What do you see as the huge impact?
> 
> Nige
> 


From general-return-2831-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 05:14:34 2011
Return-Path: <general-return-2831-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45750 invoked from network); 18 Jan 2011 05:14:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 05:14:33 -0000
Received: (qmail 21457 invoked by uid 500); 18 Jan 2011 05:14:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21118 invoked by uid 500); 18 Jan 2011 05:14:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21109 invoked by uid 99); 18 Jan 2011 05:14:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 05:14:27 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,LONG_TERM_PRICE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.97 as permitted sender)
Received: from [17.148.16.97] (HELO asmtpout022.mac.com) (17.148.16.97)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 05:14:18 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [10.0.1.13]
 (c-71-198-192-174.hsd1.ca.comcast.net [71.198.192.174])
 by asmtp022.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LF700196D785U30@asmtp022.mac.com>
 for general@hadoop.apache.org; Mon, 17 Jan 2011 21:13:56 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-18_01:2011-01-17,2011-01-18,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101170239
Subject: Re: [DISCUSS] Move project split down a level
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <113546.32737.qm@web56204.mail.re3.yahoo.com>
Date: Mon, 17 Jan 2011 21:13:55 -0800
Message-id: <D02FBCAB-DE76-408D-A137-AA31F1CEAAF1@mac.com>
References: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com>
 <AANLkTiknWzZYo0yc3X8sfuzScDF_yREwSaBi9uXsrJi5@mail.gmail.com>
 <439416.18126.qm@web56207.mail.re3.yahoo.com>
 <15CB20F8-469A-457D-BFE2-084785FB9E18@mac.com>
 <113546.32737.qm@web56204.mail.re3.yahoo.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org


On Jan 14, 2011, at 11:53 AM, Tsz Wo (Nicholas), Sze wrote:

> This is a kind of an incompatible change: all the developers, QAs, release 
> engineers and users have to change their local settings and scripts for this 
> change.  

I have a hard time believing this as I suspect the very small set of folks that test/deploy post-project split releases (0.21/0.22/trunk) have been smashing the 3 projects back together for test/deploy purposes on a cluster.  You *will* have to change your personal and your build machine SVN checkout urls, but beyond that, the projects remain as-is in separate trees.  When we mavenize, that will perhaps cause the disruption you're mentioning, but that is a separate issue/discussion from this one.

> Moreover, there are documentations, web pages and existing tools using 
> the Apache svn URLs.  

I sign up to correct the wiki, site, and Apache Hudson builds and build scripts (although help is gratefully received).

> So it is a huge impact.  I am conservative on this since, 
> as Konstantin mentioned, we risk to get into the same mess, and it will create 
> more work for the community.

I don't believe we have exited from the previous mess.  This could actually help us do that faster.  You may not have noticed that 0.21 was released as a single smashed together tar ball.  0.22 IMO is heading for the same kind of release.

> Why do we want to enforce the releases as a unit, given that the long term  
> target is to release these 3 projects independently?

Because that long term view is currently a fantasy with no real end in sight.

Nige


> ________________________________
> From: Nigel Daley <ndaley@mac.com>
> To: general@hadoop.apache.org
> Sent: Fri, January 14, 2011 11:21:25 AM
> Subject: Re: [DISCUSS] Move project split down a level
> 
> 
> On Jan 14, 2011, at 11:16 AM, Tsz Wo (Nicholas), Sze wrote:
> 
>> Hi Nigel,
>> 
>>> As I look more at the impact of the common/MR/HDFS project split on what
>>> and how we release Hadoop, I feel like the split needs an adjustment.  Many
>>> folks I've talked to agree that the project split has caused us a splitting
>>> headache.  I think 1 relatively small change could alleviate some of that.
>> 
>> Could you elaborate your idea on how the proposed changes would help?  What the 
>> 
>> problems are being addressed?  It is not clear to me.
> 
> Critical in my mind was my statement: "We're a long way from releasing these 3 
> projects independently.  Given that, they should be branched and released as a 
> unit."  This can not be enforced given the current svn layout. Other's can weigh 
> in with additional thoughts.
> 
>> You are right that the change is small but the impact is huge.  We should first 
>> 
>> understand what we are getting from the changes before doing it.
> 
> What do you see as the huge impact?
> 
> Nige


From general-return-2832-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 05:14:48 2011
Return-Path: <general-return-2832-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45841 invoked from network); 18 Jan 2011 05:14:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 05:14:47 -0000
Received: (qmail 22573 invoked by uid 500); 18 Jan 2011 05:14:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22522 invoked by uid 500); 18 Jan 2011 05:14:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22514 invoked by uid 99); 18 Jan 2011 05:14:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 05:14:43 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=LONG_TERM_PRICE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 05:14:36 +0000
Received: by bwz8 with SMTP id 8so3396764bwz.35
        for <general@hadoop.apache.org>; Mon, 17 Jan 2011 21:14:15 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.72.71 with SMTP id l7mr2942734bkj.55.1295327654206; Mon,
 17 Jan 2011 21:14:14 -0800 (PST)
Received: by 10.204.122.5 with HTTP; Mon, 17 Jan 2011 21:14:14 -0800 (PST)
In-Reply-To: <E2932D45-FF26-4579-A975-A98D2578622A@mac.com>
References: <C955EB7A.2E753%sureshms@yahoo-inc.com>
	<E2932D45-FF26-4579-A975-A98D2578622A@mac.com>
Date: Mon, 17 Jan 2011 21:14:14 -0800
Message-ID: <AANLkTikMKWV3383ptWxdvrWemUuT=ZmbqXJ6g0DEngHA@mail.gmail.com>
Subject: Re: [DISCUSS] Move project split down a level
From: Tom White <tom@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Mon, Jan 17, 2011 at 9:04 PM, Nigel Daley <ndaley@mac.com> wrote:
> Good questions. =A0Keep them coming! =A0I'll compile a list so we can sta=
rt an FAQ on this.
>
>> # Is project split a goal for hadoop in the future (even though we are n=
ot ready yet?). If it is, then putting projects back together might result =
in tight dependencies between the project. Ho do we avoid it?
>
>
> Note, we're not putting them back together. =A0This is NOT a cmd-Z (ctrl-=
Z) on the project split. =A0It's putting them back under one trunk, but as =
separate projects underneath that. =A0IMO this is a relatively small change=
 in the universe of undo-the-project-split possibilities.

Yes, the three projects would still build separately. However, it
would be useful to have a small top-level script to produce a single
artifact, like https://issues.apache.org/jira/browse/HADOOP-6342 and
https://issues.apache.org/jira/browse/HADOOP-6846.

>
>> # The committer list for each of the sub project today is different. How=
 do we reconcile them?
>
> I'd like to keep that issue out of this change if at all possible. =A0I r=
ecommend for now we keep the status quo. =A0Thus, even though all committer=
s may technically have permission to commit to all 3 project trees (can som=
eone confirm that?), we would need to rely on the honor system that committ=
ers will only commit to their project trees.

That's right - we don't have fine-grained commit access on the Hadoop
tree, so we don't need to change anything here.

Tom

>
> Cheers,
> Nige
>
> On Jan 14, 2011, at 12:00 PM, Suresh Srinivas wrote:
>
>> I like the idea of merging projects together. It save a lot of time.
>>
>> However, I would like to see a detailed proposal on how this will be don=
e and discussions on it, before moving forward on this. If this work is don=
e, need clear messages to the developers on what has changed, and how devel=
opment process is affected. These details were missing when project split w=
as done, causing great deal of confusion and pain.
>>
>> We should also address the following:
>> # Is project split a goal for hadoop in the future (even though we are n=
ot ready yet?). If it is, then putting projects back together might result =
in tight dependencies between the project. Ho do we avoid it?
>> # The committer list for each of the sub project today is different. How=
 do we reconcile them?
>>
>>
>> On 1/14/11 11:53 AM, "Tsz Wo (Nicholas), Sze" <s29752-hadoopgeneral@yaho=
o.com> wrote:
>>
>> This is a kind of an incompatible change: all the developers, QAs, relea=
se
>> engineers and users have to change their local settings and scripts for =
this
>> change. =A0Moreover, there are documentations, web pages and existing to=
ols using
>> the Apache svn URLs. =A0So it is a huge impact. =A0I am conservative on =
this since,
>> as Konstantin mentioned, we risk to get into the same mess, and it will =
create
>> more work for the community.
>>
>> Why do we want to enforce the releases as a unit, given that the long te=
rm
>> target is to release these 3 projects independently?
>>
>> Nicholas
>>
>>
>>
>>
>>
>> ________________________________
>> From: Nigel Daley <ndaley@mac.com>
>> To: general@hadoop.apache.org
>> Sent: Fri, January 14, 2011 11:21:25 AM
>> Subject: Re: [DISCUSS] Move project split down a level
>>
>>
>> On Jan 14, 2011, at 11:16 AM, Tsz Wo (Nicholas), Sze wrote:
>>
>>> Hi Nigel,
>>>
>>>> As I look more at the impact of the common/MR/HDFS project split on wh=
at
>>>> and how we release Hadoop, I feel like the split needs an adjustment. =
=A0Many
>>>> folks I've talked to agree that the project split has caused us a spli=
tting
>>>> headache. =A0I think 1 relatively small change could alleviate some of=
 that.
>>>
>>> Could you elaborate your idea on how the proposed changes would help? =
=A0What the
>>>
>>> problems are being addressed? =A0It is not clear to me.
>>
>> Critical in my mind was my statement: "We're a long way from releasing t=
hese 3
>> projects independently. =A0Given that, they should be branched and relea=
sed as a
>> unit." =A0This can not be enforced given the current svn layout. Other's=
 can weigh
>> in with additional thoughts.
>>
>>> You are right that the change is small but the impact is huge. =A0We sh=
ould first
>>>
>>> understand what we are getting from the changes before doing it.
>>
>> What do you see as the huge impact?
>>
>> Nige
>>
>
>

From general-return-2833-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 05:36:30 2011
Return-Path: <general-return-2833-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 51741 invoked from network); 18 Jan 2011 05:36:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 05:36:30 -0000
Received: (qmail 35561 invoked by uid 500); 18 Jan 2011 05:36:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 35304 invoked by uid 500); 18 Jan 2011 05:36:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 35296 invoked by uid 99); 18 Jan 2011 05:36:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 05:36:25 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 18 Jan 2011 05:36:22 +0000
Received: (qmail 49702 invoked by uid 99); 18 Jan 2011 05:36:01 -0000
Received: from localhost.apache.org (HELO mail-iy0-f176.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 05:36:01 +0000
Received: by iyb26 with SMTP id 26so6241181iyb.35
        for <general@hadoop.apache.org>; Mon, 17 Jan 2011 21:36:00 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.35.204 with SMTP id q12mr5346921ibd.191.1295328960230;
 Mon, 17 Jan 2011 21:36:00 -0800 (PST)
Received: by 10.231.167.199 with HTTP; Mon, 17 Jan 2011 21:36:00 -0800 (PST)
In-Reply-To: <AANLkTimsXa2rdpV0ahoy2pOjoXDzaOaxkYz3snVHR4th@mail.gmail.com>
References: <4D34D59F.4010802@apache.org>
	<C95A3F70.3A81%toddp@yahoo-inc.com>
	<AANLkTimsXa2rdpV0ahoy2pOjoXDzaOaxkYz3snVHR4th@mail.gmail.com>
Date: Mon, 17 Jan 2011 21:36:00 -0800
Message-ID: <AANLkTikV_PDCACGz8EZTeaSN9u_kbjNiZXJJFRBz_CBE@mail.gmail.com>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Mon, Jan 17, 2011 at 8:30 PM, Jeff Hammerbacher <hammer@cloudera.com> wrote:
> We had this exact same discussion about the 0.20-append branch a few weeks
> ago. A few organizations have tested that code at scale and feel strongly
> that it's stable. We decided not to release it because it does not meet the
> Apache guidelines for a release.

It does meet the guidelines. The last validation and development of
the append work wasn't done openly, but that doesn't prevent it from
being contributed and released. That discussion died out because no
committer stepped up, rolled a release of that branch, and called a
vote. If it gets a majority of votes on the PMC, it could even be
released off the 0.20 branch. Individuals may have their reservations
and vote accordingly, but the PMC has the authority to release
0.20-append and 0.20.100 (or whatever).

> A few weeks later, we now have another organization claiming that their
> 0.20-based branch is tested at scale and should be released. It's claimed
> that 0.20.100 will be "more stable, performant and more useful to our
> users"; the same can be said of the 0.20-append branch. Neither branch,
> however, is a bugfix release and thus does not meet the Apache guidelines
> for a release. That's too bad; we should work to avoid this situation again
> in the future, but let's not try to change the rules because we did a poor
> job in the past of getting our work released via Apache.

The rules from Apache are *far* more flexible than what we've
practiced. Our rigidity has contributed to the current state of the
project by pushing active development behind the walls of contributing
organizations. We aren't obligated to live with a fractured community,
nor do some nebulous Apache guidelines prohibit us from finding a way
forward.

> As Nigel mentions, and as was done with 0.20-append, I would fully support a
> "a code-only drop into a branch w/ no formal Apache release". That's fully
> compliant with the Apache process.

This helps nobody except those who would cut releases outside of
Apache, which is precisely what we're trying to curtail.

> All of these discussions will be moot once we get 0.22 out the door and stop
> arguing over which organization has the most magical 0.20-based bits. I'm
> looking forward to seeing all of the Apache Hadoop contributors working full
> time on that release process once these bits are committed to the 0.20.100
> branch.

+1 I'm sure I'm not alone in turning ill at the thought of working on
a 0.20 branch again. But others may have different priorities,
different interests, and may actually prefer the architectural
decisions in 0.20 to those made since then. There's obviously enough
interest in a stable branch for all the major contributors to solve
those same problems independently. Opening up space in Apache for that
work is what we should have done a year ago. Since we don't yet have a
credible release, it still makes sense today. -C

From general-return-2834-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 05:41:58 2011
Return-Path: <general-return-2834-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63361 invoked from network); 18 Jan 2011 05:41:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 05:41:57 -0000
Received: (qmail 39783 invoked by uid 500); 18 Jan 2011 05:41:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39629 invoked by uid 500); 18 Jan 2011 05:41:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39618 invoked by uid 99); 18 Jan 2011 05:41:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 05:41:52 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=LONG_TERM_PRICE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 05:41:43 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0I5euf4071969
	for <general@hadoop.apache.org>; Mon, 17 Jan 2011 21:40:56 -0800 (PST)
Received: from [10.0.1.4] (10.72.244.133) by SP2-EX07CAS01.ds.corp.yahoo.com
 (98.137.59.4) with Microsoft SMTP Server (TLS) id 8.3.106.1; Mon, 17 Jan 2011
 21:40:55 -0800
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [DISCUSS] Move project split down a level
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <D02FBCAB-DE76-408D-A137-AA31F1CEAAF1@mac.com>
Date: Mon, 17 Jan 2011 21:40:53 -0800
Content-Transfer-Encoding: quoted-printable
Message-ID: <E51D5DEA-73EC-491C-9956-A29342DB9CE5@yahoo-inc.com>
References: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com> <AANLkTiknWzZYo0yc3X8sfuzScDF_yREwSaBi9uXsrJi5@mail.gmail.com> <439416.18126.qm@web56207.mail.re3.yahoo.com> <15CB20F8-469A-457D-BFE2-084785FB9E18@mac.com> <113546.32737.qm@web56204.mail.re3.yahoo.com> <D02FBCAB-DE76-408D-A137-AA31F1CEAAF1@mac.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org


On Jan 17, 2011, at 9:13 PM, Nigel Daley wrote:

>=20
> On Jan 14, 2011, at 11:53 AM, Tsz Wo (Nicholas), Sze wrote:
>>=20
...
>> Why do we want to enforce the releases as a unit, given that the long =
term =20
>> target is to release these 3 projects independently?
>=20
> Because that long term view is currently a fantasy with no real end in =
sight.

** +1 to that.  We release as a unit, branch as a unit, test as a unit, =
deploy as a unit.  I've seen no actual gain from the project split, just =
complexity.  Simplifying things in the short term seems like a good =
goal.

> Nige
>=20
>=20
>> ________________________________
>> From: Nigel Daley <ndaley@mac.com>
>> To: general@hadoop.apache.org
>> Sent: Fri, January 14, 2011 11:21:25 AM
>> Subject: Re: [DISCUSS] Move project split down a level
>>=20
>>=20
>> On Jan 14, 2011, at 11:16 AM, Tsz Wo (Nicholas), Sze wrote:
>>=20
>>> Hi Nigel,
>>>=20
>>>> As I look more at the impact of the common/MR/HDFS project split on =
what
>>>> and how we release Hadoop, I feel like the split needs an =
adjustment.  Many
>>>> folks I've talked to agree that the project split has caused us a =
splitting
>>>> headache.  I think 1 relatively small change could alleviate some =
of that.
>>>=20
>>> Could you elaborate your idea on how the proposed changes would =
help?  What the=20
>>>=20
>>> problems are being addressed?  It is not clear to me.
>>=20
>> Critical in my mind was my statement: "We're a long way from =
releasing these 3=20
>> projects independently.  Given that, they should be branched and =
released as a=20
>> unit."  This can not be enforced given the current svn layout. =
Other's can weigh=20
>> in with additional thoughts.
>>=20
>>> You are right that the change is small but the impact is huge.  We =
should first=20
>>>=20
>>> understand what we are getting from the changes before doing it.
>>=20
>> What do you see as the huge impact?
>>=20
>> Nige
>=20


From general-return-2835-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 05:55:14 2011
Return-Path: <general-return-2835-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66699 invoked from network); 18 Jan 2011 05:55:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 05:55:12 -0000
Received: (qmail 50017 invoked by uid 500); 18 Jan 2011 05:55:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49653 invoked by uid 500); 18 Jan 2011 05:55:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49624 invoked by uid 99); 18 Jan 2011 05:55:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 05:55:05 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=LONG_TERM_PRICE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 05:54:59 +0000
Received: by wye20 with SMTP id 20so5947285wye.35
        for <general@hadoop.apache.org>; Mon, 17 Jan 2011 21:54:38 -0800 (PST)
Received: by 10.216.58.144 with SMTP id q16mr2838104wec.74.1295330078505; Mon,
 17 Jan 2011 21:54:38 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.59.13 with HTTP; Mon, 17 Jan 2011 21:54:18 -0800 (PST)
In-Reply-To: <E51D5DEA-73EC-491C-9956-A29342DB9CE5@yahoo-inc.com>
References: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com> <AANLkTiknWzZYo0yc3X8sfuzScDF_yREwSaBi9uXsrJi5@mail.gmail.com>
 <439416.18126.qm@web56207.mail.re3.yahoo.com> <15CB20F8-469A-457D-BFE2-084785FB9E18@mac.com>
 <113546.32737.qm@web56204.mail.re3.yahoo.com> <D02FBCAB-DE76-408D-A137-AA31F1CEAAF1@mac.com>
 <E51D5DEA-73EC-491C-9956-A29342DB9CE5@yahoo-inc.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Mon, 17 Jan 2011 21:54:18 -0800
X-Google-Sender-Auth: cAOp2fUHoilLgkPDuUNT3IMIgBs
Message-ID: <AANLkTimOpMVAZa+4ues=9+rkv=FNHmpTeKr-Njvyvuqn@mail.gmail.com>
Subject: Re: [DISCUSS] Move project split down a level
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Mon, Jan 17, 2011 at 21:40, Eric Baldeschwieler <eric14@yahoo-inc.com> w=
rote:
>
> On Jan 17, 2011, at 9:13 PM, Nigel Daley wrote:
>
>>
>> On Jan 14, 2011, at 11:53 AM, Tsz Wo (Nicholas), Sze wrote:
>>>
> ...
>>> Why do we want to enforce the releases as a unit, given that the long t=
erm
>>> target is to release these 3 projects independently?
>>
>> Because that long term view is currently a fantasy with no real end in s=
ight.
>
> ** +1 to that. =A0We release as a unit, branch as a unit, test as a unit,=
 deploy as a unit. =A0I've seen no actual gain from the project split, just=
 complexity.

Am I missing something in the latest development of Hadoop, Eric? What
do you mean by 'we... test as a unit'? Is it like we have test
artifacts version'd against Hadoop release proper? Or you are trying
to say something else? It isn't very clear, sorry...

Cos


>> Nige
>>
>>
>>> ________________________________
>>> From: Nigel Daley <ndaley@mac.com>
>>> To: general@hadoop.apache.org
>>> Sent: Fri, January 14, 2011 11:21:25 AM
>>> Subject: Re: [DISCUSS] Move project split down a level
>>>
>>>
>>> On Jan 14, 2011, at 11:16 AM, Tsz Wo (Nicholas), Sze wrote:
>>>
>>>> Hi Nigel,
>>>>
>>>>> As I look more at the impact of the common/MR/HDFS project split on w=
hat
>>>>> and how we release Hadoop, I feel like the split needs an adjustment.=
 =A0Many
>>>>> folks I've talked to agree that the project split has caused us a spl=
itting
>>>>> headache. =A0I think 1 relatively small change could alleviate some o=
f that.
>>>>
>>>> Could you elaborate your idea on how the proposed changes would help? =
=A0What the
>>>>
>>>> problems are being addressed? =A0It is not clear to me.
>>>
>>> Critical in my mind was my statement: "We're a long way from releasing =
these 3
>>> projects independently. =A0Given that, they should be branched and rele=
ased as a
>>> unit." =A0This can not be enforced given the current svn layout. Other'=
s can weigh
>>> in with additional thoughts.
>>>
>>>> You are right that the change is small but the impact is huge. =A0We s=
hould first
>>>>
>>>> understand what we are getting from the changes before doing it.
>>>
>>> What do you see as the huge impact?
>>>
>>> Nige
>>
>
>

From general-return-2836-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 05:57:34 2011
Return-Path: <general-return-2836-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67220 invoked from network); 18 Jan 2011 05:57:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 05:57:34 -0000
Received: (qmail 54816 invoked by uid 500); 18 Jan 2011 05:57:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54610 invoked by uid 500); 18 Jan 2011 05:57:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54602 invoked by uid 99); 18 Jan 2011 05:57:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 05:57:30 +0000
X-ASF-Spam-Status: No, hits=3.5 required=10.0
	tests=FRT_TODAY2,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 05:57:24 +0000
Received: from [192.168.1.71] (snvvpn3-10-72-77-c11.hq.corp.yahoo.com [10.72.77.11])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p0I5uhZ5097561
	for <general@hadoop.apache.org>; Mon, 17 Jan 2011 21:56:43 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1295330204;
	bh=JTn6irNJFvKGHncNkpxdI+nn3hETmuaa6HkXelvEFrs=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:Mime-Version:Subject:
	 Date:References;
	b=j1j9341aqblGqfxPXsG8/sPJKfRM5kjZf+ZynYJcEM8PjL+GEoUZr7sm/4m1MoL/Z
	 XEKNI2Oq5HwYAuy7EWR+vt2p28kCz3jeQrOAlIJQNK0DXvwKiYitXgBkG+P/B0s/iT
	 bM2f5wiEVkfDC/Z68IzNV/7RtJGzPlnxKGKg/PLM=
Message-Id: <728AC0D9-3452-46BB-9E13-405CEDF0843E@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <AANLkTimsXa2rdpV0ahoy2pOjoXDzaOaxkYz3snVHR4th@mail.gmail.com>
Content-Type: multipart/alternative; boundary=Apple-Mail-90--804576159
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Date: Mon, 17 Jan 2011 21:56:43 -0800
References: <4D34D59F.4010802@apache.org>	<C95A3F70.3A81%toddp@yahoo-inc.com> <AANLkTimsXa2rdpV0ahoy2pOjoXDzaOaxkYz3snVHR4th@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-90--804576159
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit

Bringing 'organizations' into this discussion is very disingenuous.

Doug, credit to him, was the first person to propose this release:
http://www.mail-archive.com/general@hadoop.apache.org/msg01427.html

I have supported the append-release:
http://www.mail-archive.com/general@hadoop.apache.org/msg02584.html

So, stop coloring arguments in this manner.

Arun


On Jan 17, 2011, at 8:30 PM, Jeff Hammerbacher wrote:

> Hey,
>
> We had this exact same discussion about the 0.20-append branch a few  
> weeks
> ago. A few organizations have tested that code at scale and feel  
> strongly
> that it's stable. We decided not to release it because it does not  
> meet the
> Apache guidelines for a release. The Apache process has its pros and  
> cons;
> we've all accepted them, so the community moved on and focused its  
> energy on
> the 0.22 release.
>
> A few weeks later, we now have another organization claiming that  
> their
> 0.20-based branch is tested at scale and should be released. It's  
> claimed
> that 0.20.100 will be "more stable, performant and more useful to our
> users"; the same can be said of the 0.20-append branch. Neither  
> branch,
> however, is a bugfix release and thus does not meet the Apache  
> guidelines
> for a release. That's too bad; we should work to avoid this  
> situation again
> in the future, but let's not try to change the rules because we did  
> a poor
> job in the past of getting our work released via Apache.
>
> As Nigel mentions, and as was done with 0.20-append, I would fully  
> support a
> "a code-only drop into a branch w/ no formal Apache release". That's  
> fully
> compliant with the Apache process.
>
> All of these discussions will be moot once we get 0.22 out the door  
> and stop
> arguing over which organization has the most magical 0.20-based  
> bits. I'm
> looking forward to seeing all of the Apache Hadoop contributors  
> working full
> time on that release process once these bits are committed to the  
> 0.20.100
> branch.
>
> Thanks,
> Jeff
>
> On Mon, Jan 17, 2011 at 6:55 PM, Todd Papaioannou <toddp@yahoo- 
> inc.com>wrote:
>
>> That's only true if you plan to pull forward the changes wholesale  
>> into
>> .21, .22 and beyond. And that is not what is being proposed.
>>
>> If the plan is to just land an updated and more stable version of . 
>> 20 that
>> is completely backwards compatible, then this can be done within  
>> that code
>> line without any impact to the end users. Any changes that the  
>> community
>> wish to pull forward can be identified, isolated and reviewed per  
>> the normal
>> process. Or they can remain in the .20.100 release for eternity,  
>> without any
>> impact on the future.
>>
>> Either way, the .20 release will be more stable, performant and  
>> more useful
>> to our users, and the community at large can focus on releasing . 
>> 22, which
>> we all believe is the right goal.
>>
>> ToddP
>>
>> From: Doug Cutting <cutting@apache.org<mailto:cutting@apache.org>>
>> Reply-To:  
>> "general@hadoop.apache.org<mailto:general@hadoop.apache.org>" <
>> general@hadoop.apache.org<mailto:general@hadoop.apache.org>>
>> Date: Mon, 17 Jan 2011 15:49:51 -0800
>> To: "general@hadoop.apache.org<mailto:general@hadoop.apache.org>" <
>> general@hadoop.apache.org<mailto:general@hadoop.apache.org>>
>> Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
>>
>>
>> Backwards compatibility has been a goal, so
>> with luck we will not ID regressions.
>>
>> My point was that, in addition to back-compatibility with prior 0.20
>> releases, we must also consider the forward-compatibility of each  
>> change
>> with 0.21, 0.22 and trunk.
>>


--Apple-Mail-90--804576159--

From general-return-2837-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 05:58:10 2011
Return-Path: <general-return-2837-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67494 invoked from network); 18 Jan 2011 05:58:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 05:58:09 -0000
Received: (qmail 56276 invoked by uid 500); 18 Jan 2011 05:58:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56185 invoked by uid 500); 18 Jan 2011 05:58:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56176 invoked by uid 99); 18 Jan 2011 05:58:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 05:58:05 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 05:58:00 +0000
Received: from [192.168.1.71] (snvvpn3-10-72-77-c11.hq.corp.yahoo.com [10.72.77.11])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p0I5uhZ6097561
	for <general@hadoop.apache.org>; Mon, 17 Jan 2011 21:57:31 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1295330251;
	bh=i+dlT5Nx/yYBhn/Lsr/9tEzKq5TakdgxXVjxQK/z1Z8=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=Xy6LN/v9TLafpQgxB+laBMfftDo9LZsgiTw1lq713x/j0TW8v/BFUxHZS+PzQezzc
	 ybViFCdkJ2+zsMWbsGFWvEyO/rMm8F+oiDXdLsuwg/hWdEe6FfnwBACPM3uTaIFONi
	 EF0zKvMX6HjKsKFT9tHVvXFP0GrutSVMPshEAlss=
Message-Id: <D0DA2DDA-BAB8-475F-AA55-2DB80314A6FB@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <AANLkTik5CiL3gEd+nKq5401-S2CZu9LMKFFA3AKvLcXQ@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Date: Mon, 17 Jan 2011 21:57:30 -0800
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com> <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com> <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org> <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com> <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com> <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com> <4D34A270.1060102@apache.org> <6250B06D-2502-47E6-950D-3015485C516B@yahoo-inc.com> <AANLkTik5CiL3gEd+nKq5401-S2CZu9LMKFFA3AKvLcXQ@mail.gmail.com>
X-Mailer: Apple Mail (2.936)


On Jan 17, 2011, at 8:40 PM, Jeff Hammerbacher wrote:

>>
>> Apache Hadoop hasn't had a stable, updated release in a while.
>>
>
> That's what 0.22 is for?

Every single Hadoop release in the recent past, and I have worked on  
pretty much every single Hadoop release since forever, has taken at  
least 3-4 months to stabilize.

So, we are, at a minimum looking at June, 2011, for 0.22.

This could be a good intermediate release, no? This need not be the  
only one either, please work on 20.append or 20.append+security and  
release it, I fully support it.

IAC, would calling this release something other than 0.20.* be ok?

Arun


From general-return-2838-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 06:17:23 2011
Return-Path: <general-return-2838-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 79943 invoked from network); 18 Jan 2011 06:17:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 06:17:22 -0000
Received: (qmail 66929 invoked by uid 500); 18 Jan 2011 06:17:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66589 invoked by uid 500); 18 Jan 2011 06:17:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66580 invoked by uid 99); 18 Jan 2011 06:17:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 06:17:17 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,LONG_TERM_PRICE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.105 as permitted sender)
Received: from [17.148.16.105] (HELO asmtpout030.mac.com) (17.148.16.105)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 06:17:09 +0000
MIME-version: 1.0
Content-type: multipart/alternative;
 boundary="Boundary_(ID_mEOyl55Y6cSlYPic3Psxzw)"
Received: from [10.0.1.13]
 (c-71-198-192-174.hsd1.ca.comcast.net [71.198.192.174])
 by asmtp030.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LF700EAKG3MOO90@asmtp030.mac.com>
 for general@hadoop.apache.org; Mon, 17 Jan 2011 22:16:35 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-18_01:2011-01-17,2011-01-18,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1010190000 definitions=main-1101170258
From: Nigel Daley <ndaley@mac.com>
Subject: Re: [DISCUSS] Move project split down a level
Date: Mon, 17 Jan 2011 22:16:34 -0800
In-reply-to: <E2932D45-FF26-4579-A975-A98D2578622A@mac.com>
To: general@hadoop.apache.org
References: <C955EB7A.2E753%sureshms@yahoo-inc.com>
 <E2932D45-FF26-4579-A975-A98D2578622A@mac.com>
Message-id: <B98835A6-F4B7-426E-B22F-B2BEBE273C47@mac.com>
X-Mailer: Apple Mail (2.1082)

--Boundary_(ID_mEOyl55Y6cSlYPic3Psxzw)
Content-type: text/plain; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT

Wiki FAQ started here:
http://wiki.apache.org/hadoop/ProjectSplit

Todd, left a note there for you to add in a link to a git-history-fixer-script Jira when the time comes.  If folks find more docs that will need updating, please add them to the list at the end of the FAQ.

We still need to fill in the exact steps that developers will need to perform to change their workspaces once this change is made.

Nige

On Jan 17, 2011, at 9:04 PM, Nigel Daley wrote:

> Good questions.  Keep them coming!  I'll compile a list so we can start an FAQ on this.
> 
>> # Is project split a goal for hadoop in the future (even though we are not ready yet?). If it is, then putting projects back together might result in tight dependencies between the project. Ho do we avoid it?
> 
> 
> Note, we're not putting them back together.  This is NOT a cmd-Z (ctrl-Z) on the project split.  It's putting them back under one trunk, but as separate projects underneath that.  IMO this is a relatively small change in the universe of undo-the-project-split possibilities.
> 
>> # The committer list for each of the sub project today is different. How do we reconcile them?
> 
> I'd like to keep that issue out of this change if at all possible.  I recommend for now we keep the status quo.  Thus, even though all committers may technically have permission to commit to all 3 project trees (can someone confirm that?), we would need to rely on the honor system that committers will only commit to their project trees.
> 
> Cheers,
> Nige
> 
> On Jan 14, 2011, at 12:00 PM, Suresh Srinivas wrote:
> 
>> I like the idea of merging projects together. It save a lot of time.
>> 
>> However, I would like to see a detailed proposal on how this will be done and discussions on it, before moving forward on this. If this work is done, need clear messages to the developers on what has changed, and how development process is affected. These details were missing when project split was done, causing great deal of confusion and pain.
>> 
>> We should also address the following:
>> # Is project split a goal for hadoop in the future (even though we are not ready yet?). If it is, then putting projects back together might result in tight dependencies between the project. Ho do we avoid it?
>> # The committer list for each of the sub project today is different. How do we reconcile them?
>> 
>> 
>> On 1/14/11 11:53 AM, "Tsz Wo (Nicholas), Sze" <s29752-hadoopgeneral@yahoo.com> wrote:
>> 
>> This is a kind of an incompatible change: all the developers, QAs, release
>> engineers and users have to change their local settings and scripts for this
>> change.  Moreover, there are documentations, web pages and existing tools using
>> the Apache svn URLs.  So it is a huge impact.  I am conservative on this since,
>> as Konstantin mentioned, we risk to get into the same mess, and it will create
>> more work for the community.
>> 
>> Why do we want to enforce the releases as a unit, given that the long term
>> target is to release these 3 projects independently?
>> 
>> Nicholas
>> 
>> 
>> 
>> 
>> 
>> ________________________________
>> From: Nigel Daley <ndaley@mac.com>
>> To: general@hadoop.apache.org
>> Sent: Fri, January 14, 2011 11:21:25 AM
>> Subject: Re: [DISCUSS] Move project split down a level
>> 
>> 
>> On Jan 14, 2011, at 11:16 AM, Tsz Wo (Nicholas), Sze wrote:
>> 
>>> Hi Nigel,
>>> 
>>>> As I look more at the impact of the common/MR/HDFS project split on what
>>>> and how we release Hadoop, I feel like the split needs an adjustment.  Many
>>>> folks I've talked to agree that the project split has caused us a splitting
>>>> headache.  I think 1 relatively small change could alleviate some of that.
>>> 
>>> Could you elaborate your idea on how the proposed changes would help?  What the
>>> 
>>> problems are being addressed?  It is not clear to me.
>> 
>> Critical in my mind was my statement: "We're a long way from releasing these 3
>> projects independently.  Given that, they should be branched and released as a
>> unit."  This can not be enforced given the current svn layout. Other's can weigh
>> in with additional thoughts.
>> 
>>> You are right that the change is small but the impact is huge.  We should first
>>> 
>>> understand what we are getting from the changes before doing it.
>> 
>> What do you see as the huge impact?
>> 
>> Nige
>> 
> 


--Boundary_(ID_mEOyl55Y6cSlYPic3Psxzw)--

From general-return-2839-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 06:34:24 2011
Return-Path: <general-return-2839-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 90924 invoked from network); 18 Jan 2011 06:34:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 06:34:23 -0000
Received: (qmail 81012 invoked by uid 500); 18 Jan 2011 06:34:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79384 invoked by uid 500); 18 Jan 2011 06:34:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78535 invoked by uid 99); 18 Jan 2011 06:34:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 06:34:18 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=LONG_TERM_PRICE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 06:34:11 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0I6XbIv008255
	for <general@hadoop.apache.org>; Mon, 17 Jan 2011 22:33:37 -0800 (PST)
Received: from [10.0.1.4] (10.72.244.133) by SP2-EX07CAS01.ds.corp.yahoo.com
 (98.137.59.4) with Microsoft SMTP Server (TLS) id 8.3.106.1; Mon, 17 Jan 2011
 22:33:36 -0800
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [DISCUSS] Move project split down a level
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <AANLkTimOpMVAZa+4ues=9+rkv=FNHmpTeKr-Njvyvuqn@mail.gmail.com>
Date: Mon, 17 Jan 2011 22:33:34 -0800
Content-Transfer-Encoding: quoted-printable
Message-ID: <8AA8E042-F3FA-4382-B8AF-1FE272F8A880@yahoo-inc.com>
References: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com> <AANLkTiknWzZYo0yc3X8sfuzScDF_yREwSaBi9uXsrJi5@mail.gmail.com> <439416.18126.qm@web56207.mail.re3.yahoo.com> <15CB20F8-469A-457D-BFE2-084785FB9E18@mac.com> <113546.32737.qm@web56204.mail.re3.yahoo.com> <D02FBCAB-DE76-408D-A137-AA31F1CEAAF1@mac.com> <E51D5DEA-73EC-491C-9956-A29342DB9CE5@yahoo-inc.com> <AANLkTimOpMVAZa+4ues=9+rkv=FNHmpTeKr-Njvyvuqn@mail.gmail.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

Nigel proposes that in this release (as in previous releases), =
everything should be packaged together.

Our in house experience at yahoo is that this makes a lot of sense.  It =
is how we find it most effective to operate.  The project split has =
introduced a lot of complexity with no return.

Do you see any advantage to the status quo, versus nigel's proposal?

Thanks!  And sorry for any ambiguity.

E14



On Jan 17, 2011, at 9:54 PM, Konstantin Boudnik wrote:

> On Mon, Jan 17, 2011 at 21:40, Eric Baldeschwieler =
<eric14@yahoo-inc.com> wrote:
>>=20
>> On Jan 17, 2011, at 9:13 PM, Nigel Daley wrote:
>>=20
>>>=20
>>> On Jan 14, 2011, at 11:53 AM, Tsz Wo (Nicholas), Sze wrote:
>>>>=20
>> ...
>>>> Why do we want to enforce the releases as a unit, given that the =
long term
>>>> target is to release these 3 projects independently?
>>>=20
>>> Because that long term view is currently a fantasy with no real end =
in sight.
>>=20
>> ** +1 to that.  We release as a unit, branch as a unit, test as a =
unit, deploy as a unit.  I've seen no actual gain from the project =
split, just complexity.
>=20
> Am I missing something in the latest development of Hadoop, Eric? What
> do you mean by 'we... test as a unit'? Is it like we have test
> artifacts version'd against Hadoop release proper? Or you are trying
> to say something else? It isn't very clear, sorry...
>=20
> Cos
>=20
>=20
>>> Nige
>>>=20
>>>=20
>>>> ________________________________
>>>> From: Nigel Daley <ndaley@mac.com>
>>>> To: general@hadoop.apache.org
>>>> Sent: Fri, January 14, 2011 11:21:25 AM
>>>> Subject: Re: [DISCUSS] Move project split down a level
>>>>=20
>>>>=20
>>>> On Jan 14, 2011, at 11:16 AM, Tsz Wo (Nicholas), Sze wrote:
>>>>=20
>>>>> Hi Nigel,
>>>>>=20
>>>>>> As I look more at the impact of the common/MR/HDFS project split =
on what
>>>>>> and how we release Hadoop, I feel like the split needs an =
adjustment.  Many
>>>>>> folks I've talked to agree that the project split has caused us a =
splitting
>>>>>> headache.  I think 1 relatively small change could alleviate some =
of that.
>>>>>=20
>>>>> Could you elaborate your idea on how the proposed changes would =
help?  What the
>>>>>=20
>>>>> problems are being addressed?  It is not clear to me.
>>>>=20
>>>> Critical in my mind was my statement: "We're a long way from =
releasing these 3
>>>> projects independently.  Given that, they should be branched and =
released as a
>>>> unit."  This can not be enforced given the current svn layout. =
Other's can weigh
>>>> in with additional thoughts.
>>>>=20
>>>>> You are right that the change is small but the impact is huge.  We =
should first
>>>>>=20
>>>>> understand what we are getting from the changes before doing it.
>>>>=20
>>>> What do you see as the huge impact?
>>>>=20
>>>> Nige
>>>=20
>>=20
>>=20


From general-return-2840-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 06:53:02 2011
Return-Path: <general-return-2840-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94217 invoked from network); 18 Jan 2011 06:53:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 06:53:02 -0000
Received: (qmail 96787 invoked by uid 500); 18 Jan 2011 06:53:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96460 invoked by uid 500); 18 Jan 2011 06:52:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96452 invoked by uid 99); 18 Jan 2011 06:52:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 06:52:56 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=LONG_TERM_PRICE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 06:52:50 +0000
Received: by wye20 with SMTP id 20so5978840wye.35
        for <general@hadoop.apache.org>; Mon, 17 Jan 2011 22:52:29 -0800 (PST)
Received: by 10.216.143.17 with SMTP id k17mr703509wej.74.1295333549750; Mon,
 17 Jan 2011 22:52:29 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.59.13 with HTTP; Mon, 17 Jan 2011 22:52:08 -0800 (PST)
In-Reply-To: <8AA8E042-F3FA-4382-B8AF-1FE272F8A880@yahoo-inc.com>
References: <D1B0544A-E594-4530-BAEA-46EFFE97676A@mac.com> <AANLkTiknWzZYo0yc3X8sfuzScDF_yREwSaBi9uXsrJi5@mail.gmail.com>
 <439416.18126.qm@web56207.mail.re3.yahoo.com> <15CB20F8-469A-457D-BFE2-084785FB9E18@mac.com>
 <113546.32737.qm@web56204.mail.re3.yahoo.com> <D02FBCAB-DE76-408D-A137-AA31F1CEAAF1@mac.com>
 <E51D5DEA-73EC-491C-9956-A29342DB9CE5@yahoo-inc.com> <AANLkTimOpMVAZa+4ues=9+rkv=FNHmpTeKr-Njvyvuqn@mail.gmail.com>
 <8AA8E042-F3FA-4382-B8AF-1FE272F8A880@yahoo-inc.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Mon, 17 Jan 2011 22:52:08 -0800
X-Google-Sender-Auth: PbAtM50WLuEXxYQwR21xrfmbMCY
Message-ID: <AANLkTikkz1iyUcvsxsxen4Y+mkrkhGiKHZT=e2psKpM6@mail.gmail.com>
Subject: Re: [DISCUSS] Move project split down a level
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Packaging everything together makes sense for unit/functional level of
tests. Since Hadoop in current shape doesn't have any other kinds of
tests (despite of limited system tests implemented within Herriot
framework) there's no objections per se.

And you know my take on Hadoop's stack testing for it proved to be
beneficial  for Y! security release stabilization in particular: stack
testing has to be done by a set of separate component specific testing
artifacts tailored together for the particular stack's validation.

But since we aren't anywhere near the noble goal (as correctly
mentioned by Nigel et all) we don't need to make any changes in out
"bits + functional tests" packaging schema. Thanks for clarification
of your point,
btw.

Cos

On Mon, Jan 17, 2011 at 22:33, Eric Baldeschwieler <eric14@yahoo-inc.com> w=
rote:
> Nigel proposes that in this release (as in previous releases), everything=
 should be packaged together.
>
> Our in house experience at yahoo is that this makes a lot of sense. =A0It=
 is how we find it most effective to operate. =A0The project split has intr=
oduced a lot of complexity with no return.
>
> Do you see any advantage to the status quo, versus nigel's proposal?
>
> Thanks! =A0And sorry for any ambiguity.
>
> E14
>
>
>
> On Jan 17, 2011, at 9:54 PM, Konstantin Boudnik wrote:
>
>> On Mon, Jan 17, 2011 at 21:40, Eric Baldeschwieler <eric14@yahoo-inc.com=
> wrote:
>>>
>>> On Jan 17, 2011, at 9:13 PM, Nigel Daley wrote:
>>>
>>>>
>>>> On Jan 14, 2011, at 11:53 AM, Tsz Wo (Nicholas), Sze wrote:
>>>>>
>>> ...
>>>>> Why do we want to enforce the releases as a unit, given that the long=
 term
>>>>> target is to release these 3 projects independently?
>>>>
>>>> Because that long term view is currently a fantasy with no real end in=
 sight.
>>>
>>> ** +1 to that. =A0We release as a unit, branch as a unit, test as a uni=
t, deploy as a unit. =A0I've seen no actual gain from the project split, ju=
st complexity.
>>
>> Am I missing something in the latest development of Hadoop, Eric? What
>> do you mean by 'we... test as a unit'? Is it like we have test
>> artifacts version'd against Hadoop release proper? Or you are trying
>> to say something else? It isn't very clear, sorry...
>>
>> Cos
>>
>>
>>>> Nige
>>>>
>>>>
>>>>> ________________________________
>>>>> From: Nigel Daley <ndaley@mac.com>
>>>>> To: general@hadoop.apache.org
>>>>> Sent: Fri, January 14, 2011 11:21:25 AM
>>>>> Subject: Re: [DISCUSS] Move project split down a level
>>>>>
>>>>>
>>>>> On Jan 14, 2011, at 11:16 AM, Tsz Wo (Nicholas), Sze wrote:
>>>>>
>>>>>> Hi Nigel,
>>>>>>
>>>>>>> As I look more at the impact of the common/MR/HDFS project split on=
 what
>>>>>>> and how we release Hadoop, I feel like the split needs an adjustmen=
t. =A0Many
>>>>>>> folks I've talked to agree that the project split has caused us a s=
plitting
>>>>>>> headache. =A0I think 1 relatively small change could alleviate some=
 of that.
>>>>>>
>>>>>> Could you elaborate your idea on how the proposed changes would help=
? =A0What the
>>>>>>
>>>>>> problems are being addressed? =A0It is not clear to me.
>>>>>
>>>>> Critical in my mind was my statement: "We're a long way from releasin=
g these 3
>>>>> projects independently. =A0Given that, they should be branched and re=
leased as a
>>>>> unit." =A0This can not be enforced given the current svn layout. Othe=
r's can weigh
>>>>> in with additional thoughts.
>>>>>
>>>>>> You are right that the change is small but the impact is huge. =A0We=
 should first
>>>>>>
>>>>>> understand what we are getting from the changes before doing it.
>>>>>
>>>>> What do you see as the huge impact?
>>>>>
>>>>> Nige
>>>>
>>>
>>>
>
>

From general-return-2841-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 08:02:36 2011
Return-Path: <general-return-2841-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21762 invoked from network); 18 Jan 2011 08:02:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 08:02:35 -0000
Received: (qmail 84240 invoked by uid 500); 18 Jan 2011 08:02:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83860 invoked by uid 500); 18 Jan 2011 08:02:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83842 invoked by uid 99); 18 Jan 2011 08:02:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 08:02:28 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: error (nike.apache.org: local policy)
Received: from [159.226.40.9] (HELO software.ict.ac.cn) (159.226.40.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 18 Jan 2011 08:02:21 +0000
Received: (qmail 15261 invoked from network); 18 Jan 2011 08:01:27 -0000
Received: from unknown (HELO xieys) (10.61.0.2)
  by software.ict.ac.cn with SMTP; 18 Jan 2011 08:01:27 -0000
Date: Tue, 18 Jan 2011 16:00:41 +0800
From: "xieyeshan" <xieyeshan@software.ict.ac.cn>
To: "general" <general@hadoop.apache.org>
Subject: How to use combiner in hadoop streaming
Message-ID: <201101181600405786301@software.ict.ac.cn>
X-mailer: Foxmail 6, 15, 201, 23 [cn]
Mime-Version: 1.0
Content-Type: multipart/alternative;
	boundary="=====003_Dragon117660732750_====="
X-Virus-Checked: Checked by ClamAV on apache.org

--=====003_Dragon117660732750_=====
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit

hi,all

     I want to use combiner in hadoop streaming, but combiner is not the same as mapper or reducer in streaming.How could I do this?
     Thanks!


Yours, 

Yeshan Xie

2011-01-18 



xieyeshan 

--=====003_Dragon117660732750_=====--


From general-return-2842-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 08:07:21 2011
Return-Path: <general-return-2842-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31990 invoked from network); 18 Jan 2011 08:07:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 08:07:21 -0000
Received: (qmail 91828 invoked by uid 500); 18 Jan 2011 08:07:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91635 invoked by uid 500); 18 Jan 2011 08:07:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91624 invoked by uid 99); 18 Jan 2011 08:07:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 08:07:15 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 08:07:09 +0000
Received: by fxm2 with SMTP id 2so7280600fxm.35
        for <general@hadoop.apache.org>; Tue, 18 Jan 2011 00:06:49 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type:content-transfer-encoding;
        bh=VX71sXvG6cvepYdmlvD65LR84NfAs50nLEA9M1Tsa3c=;
        b=Yz5ZZly7f8RlJVyrE6KLwbVCTogkEVq9rXXIcNVr0E3Bn5gxljh32EipsiwvrUEPAh
         yEfUYyZaDCuq6Bi2nHrqEmW941c8JvNtXPmPG75PQyQ+Wn++JIdxiDGwY9z/wafrwahP
         N95aqJMiL2HKzz/q76neGnliOMHTFErucVpVs=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=E6QWYIDD9mGmEGDig5AugahqcgajX0sgmTeFqTcyG09c1qrJa3RfDo9BF7FrwpUXoJ
         if1ZlTh3Qy2NUNQhFwdwglKv3kKY2Icf3OCQTnGDTYORCxKWSIlq6Bwz0Gb2MQLTSH0D
         TaEDiiXR4JNY9eZbmIw7TNmb6seIsayqzhxCo=
Received: by 10.223.85.203 with SMTP id p11mr5806741fal.108.1295338007561;
 Tue, 18 Jan 2011 00:06:47 -0800 (PST)
MIME-Version: 1.0
Received: by 10.223.124.200 with HTTP; Tue, 18 Jan 2011 00:06:27 -0800 (PST)
In-Reply-To: <201101181600405786301@software.ict.ac.cn>
References: <201101181600405786301@software.ict.ac.cn>
From: Harsh J <qwertymaniac@gmail.com>
Date: Tue, 18 Jan 2011 13:36:27 +0530
Message-ID: <AANLkTina1=nYy3qGz6pYmP5NqRMEEXDMrFJ8rUd+3Dyq@mail.gmail.com>
Subject: Re: How to use combiner in hadoop streaming
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Right now, the Combiner has to be a Java Class. A command is
unacceptable, as noted here:
http://wiki.apache.org/hadoop/HadoopStreaming

On Tue, Jan 18, 2011 at 1:30 PM, xieyeshan <xieyeshan@software.ict.ac.cn> w=
rote:
> hi,all
>
> =A0 =A0 I want to use combiner in hadoop streaming, but combiner is not t=
he same as mapper or reducer in streaming.How could I do this?
> =A0 =A0 Thanks!
>
>
> Yours,
>
> Yeshan Xie
>
> 2011-01-18
>
>
>
> xieyeshan
>



--=20
Harsh J
www.harshj.com

From general-return-2843-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 08:15:04 2011
Return-Path: <general-return-2843-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34741 invoked from network); 18 Jan 2011 08:15:04 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 08:15:04 -0000
Received: (qmail 998 invoked by uid 500); 18 Jan 2011 08:15:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 647 invoked by uid 500); 18 Jan 2011 08:14:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 637 invoked by uid 99); 18 Jan 2011 08:14:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 08:14:59 +0000
X-ASF-Spam-Status: No, hits=2.3 required=10.0
	tests=FROM_EXCESS_BASE64,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: error (athena.apache.org: local policy)
Received: from [159.226.40.9] (HELO software.ict.ac.cn) (159.226.40.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 18 Jan 2011 08:14:54 +0000
Received: (qmail 16420 invoked from network); 18 Jan 2011 08:13:59 -0000
Received: from unknown (HELO xieys) (10.61.0.2)
  by software.ict.ac.cn with SMTP; 18 Jan 2011 08:13:59 -0000
Date: Tue, 18 Jan 2011 16:13:13 +0800
From: "=?utf-8?B?eGlleWVzaGFu?=" <xieyeshan@software.ict.ac.cn>
To: "=?utf-8?B?Z2VuZXJhbA==?=" <general@hadoop.apache.org>
References: <201101181600405786301@software.ict.ac.cn>
Subject: =?utf-8?B?UmU6IFJlOiBIb3cgdG8gdXNlIGNvbWJpbmVyIGluIGhhZG9vcCBzdHJlYW1pbmc=?=
Message-ID: <201101181613130466899@software.ict.ac.cn>
X-mailer: Foxmail 6, 15, 201, 23 [cn]
Mime-Version: 1.0
Content-Type: multipart/alternative;
	boundary="=====003_Dragon887860437502_====="

--=====003_Dragon887860437502_=====
Content-Type: text/plain;
	charset="utf-8"
Content-Transfer-Encoding: base64

SGksSGFyc2ggSiANCg0KICAgIEkgdXNlIGEgSmF2YSBDbGFzcyBhcyBjb21iaW5lciBhbmQgZ2V0
IGZvbGxvd2luZyBlcnJvciBpbmZvbWF0aW9uOg0KRXhjZXB0aW9uIGluIHRocmVhZCAibWFpbiIg
amF2YS5sYW5nLk5vQ2xhc3NEZWZGb3VuZEVycm9yOiBvcmcvYXBhY2hlL2hhZG9vcC9zdHJlYW1p
bmcvUGxzYUNvbWJpbmUgKHdyb25nIG5hbWU6IFBsc2FDb21iaW5lKQ0KICAgICAgICBhdCBqYXZh
LmxhbmcuQ2xhc3NMb2FkZXIuZGVmaW5lQ2xhc3MxKE5hdGl2ZSBNZXRob2QpDQogICAgICAgIGF0
IGphdmEubGFuZy5DbGFzc0xvYWRlci5kZWZpbmVDbGFzcyhDbGFzc0xvYWRlci5qYXZhOjYyMCkN
CiAgICAgICAgYXQgamF2YS5zZWN1cml0eS5TZWN1cmVDbGFzc0xvYWRlci5kZWZpbmVDbGFzcyhT
ZWN1cmVDbGFzc0xvYWRlci5qYXZhOjEyNCkNCiAgICAgICAgYXQgamF2YS5uZXQuVVJMQ2xhc3NM
b2FkZXIuZGVmaW5lQ2xhc3MoVVJMQ2xhc3NMb2FkZXIuamF2YToyNjApDQogICAgICAgIGF0IGph
dmEubmV0LlVSTENsYXNzTG9hZGVyLmFjY2VzcyQwMDAoVVJMQ2xhc3NMb2FkZXIuamF2YTo1NikN
CiAgICAgICAgYXQgamF2YS5uZXQuVVJMQ2xhc3NMb2FkZXIkMS5ydW4oVVJMQ2xhc3NMb2FkZXIu
amF2YToxOTUpDQogICAgICAgIGF0IGphdmEuc2VjdXJpdHkuQWNjZXNzQ29udHJvbGxlci5kb1By
aXZpbGVnZWQoTmF0aXZlIE1ldGhvZCkNCiAgICAgICAgYXQgamF2YS5uZXQuVVJMQ2xhc3NMb2Fk
ZXIuZmluZENsYXNzKFVSTENsYXNzTG9hZGVyLmphdmE6MTg4KQ0KICAgICAgICBhdCBqYXZhLmxh
bmcuQ2xhc3NMb2FkZXIubG9hZENsYXNzKENsYXNzTG9hZGVyLmphdmE6MzA2KQ0KICAgICAgICBh
dCBqYXZhLmxhbmcuQ2xhc3NMb2FkZXIubG9hZENsYXNzKENsYXNzTG9hZGVyLmphdmE6MjUxKQ0K
ICAgICAgICBhdCBqYXZhLmxhbmcuQ2xhc3NMb2FkZXIubG9hZENsYXNzSW50ZXJuYWwoQ2xhc3NM
b2FkZXIuamF2YTozMTkpDQogICAgICAgIGF0IGphdmEubGFuZy5DbGFzcy5mb3JOYW1lMChOYXRp
dmUgTWV0aG9kKQ0KICAgICAgICBhdCBqYXZhLmxhbmcuQ2xhc3MuZm9yTmFtZShDbGFzcy5qYXZh
OjI0NykNCiAgICAgICAgYXQgb3JnLmFwYWNoZS5oYWRvb3AuY29uZi5Db25maWd1cmF0aW9uLmdl
dENsYXNzQnlOYW1lKENvbmZpZ3VyYXRpb24uamF2YTo3NjEpDQogICAgICAgIGF0IG9yZy5hcGFj
aGUuaGFkb29wLnN0cmVhbWluZy5TdHJlYW1VdGlsLmdvb2RDbGFzc09yTnVsbChTdHJlYW1VdGls
LmphdmE6NTYpDQogICAgICAgIGF0IG9yZy5hcGFjaGUuaGFkb29wLnN0cmVhbWluZy5TdHJlYW1K
b2Iuc2V0Sm9iQ29uZihTdHJlYW1Kb2IuamF2YTo3MDkpDQogICAgICAgIGF0IG9yZy5hcGFjaGUu
aGFkb29wLnN0cmVhbWluZy5TdHJlYW1Kb2IucnVuKFN0cmVhbUpvYi5qYXZhOjExNykNCiAgICAg
ICAgYXQgb3JnLmFwYWNoZS5oYWRvb3AudXRpbC5Ub29sUnVubmVyLnJ1bihUb29sUnVubmVyLmph
dmE6NjUpDQogICAgICAgIGF0IG9yZy5hcGFjaGUuaGFkb29wLnV0aWwuVG9vbFJ1bm5lci5ydW4o
VG9vbFJ1bm5lci5qYXZhOjc5KQ0KICAgICAgICBhdCBvcmcuYXBhY2hlLmhhZG9vcC5zdHJlYW1p
bmcuSGFkb29wU3RyZWFtaW5nLm1haW4oSGFkb29wU3RyZWFtaW5nLmphdmE6MzIpDQogICAgICAg
IGF0IHN1bi5yZWZsZWN0Lk5hdGl2ZU1ldGhvZEFjY2Vzc29ySW1wbC5pbnZva2UwKE5hdGl2ZSBN
ZXRob2QpDQogICAgICAgIGF0IHN1bi5yZWZsZWN0Lk5hdGl2ZU1ldGhvZEFjY2Vzc29ySW1wbC5p
bnZva2UoTmF0aXZlTWV0aG9kQWNjZXNzb3JJbXBsLmphdmE6MzkpDQogICAgICAgIGF0IHN1bi5y
ZWZsZWN0LkRlbGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcGwuaW52b2tlKERlbGVnYXRpbmdNZXRo
b2RBY2Nlc3NvckltcGwuamF2YToyNSkNCiAgICAgICAgYXQgamF2YS5sYW5nLnJlZmxlY3QuTWV0
aG9kLmludm9rZShNZXRob2QuamF2YTo1OTcpDQogICAgICAgIGF0IG9yZy5hcGFjaGUuaGFkb29w
LnV0aWwuUnVuSmFyLm1haW4oUnVuSmFyLmphdmE6MTU2KQ0KICAgICAgDQoNCg0KVGhhbmtzIQ0K
DQoNCjIwMTEtMDEtMTggDQoNCg0KDQp4aWV5ZXNoYW4gDQoNCg0KDQrlj5Hku7bkurrvvJogSGFy
c2ggSiANCuWPkemAgeaXtumXtO+8miAyMDExLTAxLTE4ICAxNjowNzowNSANCuaUtuS7tuS6uu+8
miBnZW5lcmFsIA0K5oqE6YCB77yaIA0K5Li76aKY77yaIFJlOiBIb3cgdG8gdXNlIGNvbWJpbmVy
IGluIGhhZG9vcCBzdHJlYW1pbmcgDQogDQpSaWdodCBub3csIHRoZSBDb21iaW5lciBoYXMgdG8g
YmUgYSBKYXZhIENsYXNzLiBBIGNvbW1hbmQgaXMNCnVuYWNjZXB0YWJsZSwgYXMgbm90ZWQgaGVy
ZToNCmh0dHA6Ly93aWtpLmFwYWNoZS5vcmcvaGFkb29wL0hhZG9vcFN0cmVhbWluZw0KT24gVHVl
LCBKYW4gMTgsIDIwMTEgYXQgMTozMCBQTSwgeGlleWVzaGFuIDx4aWV5ZXNoYW5Ac29mdHdhcmUu
aWN0LmFjLmNuPiB3cm90ZToNCj4gaGksYWxsDQo+DQo+ICAgICBJIHdhbnQgdG8gdXNlIGNvbWJp
bmVyIGluIGhhZG9vcCBzdHJlYW1pbmcsIGJ1dCBjb21iaW5lciBpcyBub3QgdGhlIHNhbWUgYXMg
bWFwcGVyIG9yIHJlZHVjZXIgaW4gc3RyZWFtaW5nLkhvdyBjb3VsZCBJIGRvIHRoaXM/DQo+ICAg
ICBUaGFua3MhDQo+DQo+DQo+IFlvdXJzLA0KPg0KPiBZZXNoYW4gWGllDQo+DQo+IDIwMTEtMDEt
MTgNCj4NCj4NCj4NCj4geGlleWVzaGFuDQo+DQotLSANCkhhcnNoIEoNCnd3dy5oYXJzaGouY29t
DQo=

--=====003_Dragon887860437502_=====--


From general-return-2844-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 08:21:12 2011
Return-Path: <general-return-2844-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37154 invoked from network); 18 Jan 2011 08:21:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 08:21:12 -0000
Received: (qmail 8211 invoked by uid 500); 18 Jan 2011 08:21:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7872 invoked by uid 500); 18 Jan 2011 08:21:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7860 invoked by uid 99); 18 Jan 2011 08:21:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 08:21:06 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [208.97.132.145] (HELO homiemail-a28.g.dreamhost.com) (208.97.132.145)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 08:21:00 +0000
Received: from homiemail-a28.g.dreamhost.com (localhost [127.0.0.1])
	by homiemail-a28.g.dreamhost.com (Postfix) with ESMTP id 7B8751B4059
	for <general@hadoop.apache.org>; Tue, 18 Jan 2011 00:20:39 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; q=dns; s=
	gbiv.com; b=rKVQ4o0LuvBi8VGHEdPrgY+IxoPw7tgTJRAg2e9EOT/P4k5s6shs
	jFQ8Dt6MDk4ZPsOfnRfGLdKiECbGs+NX0Y6zPr3Y9nxYe3YbGMKjsI/PViBZR+fk
	0MHX4rhBMSYfMLdeT0tDY8kLddSDhjPwRedr36rB6t5DFES8eS5qWP8=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gbiv.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; s=gbiv.com;
	 bh=8/W+nrwShLrbD791RGWTcYZeVpA=; b=FcRQSAFPMyi8uaIr0W4/W4qJ8KVp
	7u5JQkyXNkDMBgHvG/wB0+Sx+ESlxv/u+iG84SMJcKgufyQUzlu/m8XLELY1MCUS
	2EUknklapXuC/6wWUq7Da0sLg8P6QYzgV63kOWwjfq8R2PlA54g7jpL6SX5UHNCx
	mPYkryeLq+/Nh24=
Received: from [192.168.1.84] (99-21-208-82.lightspeed.irvnca.sbcglobal.net [99.21.208.82])
	(Authenticated sender: fielding@gbiv.com)
	by homiemail-a28.g.dreamhost.com (Postfix) with ESMTPA id 3B60D1B4058
	for <general@hadoop.apache.org>; Tue, 18 Jan 2011 00:20:39 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: "Roy T. Fielding" <fielding@gbiv.com>
In-Reply-To: <74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com>
Date: Tue, 18 Jan 2011 00:20:38 -0800
Content-Transfer-Encoding: quoted-printable
Message-Id: <F5C6B0DD-DB20-4324-8A11-A2DE8E7169EB@gbiv.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com> <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com> <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org> <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com> <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com> <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com> <AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com> <388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com> <AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com> <04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com> <AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com> <74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

I thought that this discussion would have reached some sensible
understanding of how Apache projects work by now, but it seems not.

On Jan 13, 2011, at 6:12 PM, Arun C Murthy wrote:
> The version I'm offering to push to the community has fixed all of =
them, *plus* the added benefit of several stability and performance =
fixes we have done since 20.104.3, almost 10 internal releases. This is =
a battle tested and hardened version which we have deployed on 40,000+ =
nodes. It is a significant upgrade on 0.20.104.3 which we never =
deployed. I'm pretty sure *some* users will find that valuable. ;)
>=20
> Also, I've offered to push individual patches as a background activity =
on a branch - that should suffice, no? Or, do you consider this a =
blocker?
>=20
> Again, my goal in this exercise is to get a stable, improved version =
of Hadoop into the hands of our users asap, and focus on 0.22 and =
beyond.

So, you have a bunch of changes that you want to contribute.
Please do so.  There are several ways:

 a) break the changes down into a sequence of patches, create jira
    issues for each one (or append to the existing issue), and then
    provide the group with a list of the issue links so that people
    can quickly +1 each one.  When it seems worthwhile to you, create
    a branch off of some prior Apache release point in svn and commit
    each patch to it until the branch is identical to (or, in your own
    opinion, better than) the source code that you have tested locally.
    Then RM a tarball and start a release vote.  Since all of this is
    being done in jira and svn, others can help you do all but the
    first part (breaking down the big patch).

or

 b) create a branch off of some prior Apache release point in svn
    and replay the internal Y! commits on that branch until the branch
    source code is identical to what you have tested locally.  Then
    RM a tarball based on that branch and start a release vote.
    Since the history is now in svn, others could do the RM bit if
    you don't have time.

or

 c) create a branch off of some prior Apache release point in svn
    and apply one big ugly patch to it.  Then RM a tarball based
    on that branch and ask for a release vote.

You will note that none of the above requires a discussion on this
list prior to the release vote, though (a) would likely result in
more +1s than (b), and (b) would likely receive more +1s than (c).
Regardless, the release vote is a lazy majority decision.

I do not believe that there is any rational reason to apply a
single big patch.  "It takes too long" is nonsense -- you have
already spent far more time discussing it than would be required
to do it.  "It is too hard" is also nonsense -- use your version
control system to extract the set of changes and just replay them
(with appropriate changelog editing).  "It has already been tested
at Y!" is simply irrelevant -- the source code has been tested, not
the order in which the patches have been applied, so all you should
care about is that the final branch code is comparable to the tested
source code (i.e., use diff).

Nevertheless, all contributions at Apache are voluntary.  Do what
you have time for, now, with the understanding that others may or
may not complete the task, and may or may not vote for the release.

You can make a branch, apply the big patch, and stand by
while the rest of the group chooses whether to just accept it
as a big change.  Someone else might create a parallel branch to
apply the specific changes in an orderly matter, or perhaps you'll
discover an easy way to do that a few days from now.  Or it
might just sit there and never be released.

There is no need for the group to agree to a plan up front, just
as there is no need for the group to approve a release just because
someone did the work of RMing a tarball.  Sure, it might save
a lot of time if potential disagreements can be resolved before
work is done, but the fact is that people tend to disagree less
with actual work products than with abstract plans.  After all,
everyone has a plan.  It is also far easier to convince people
to fix their own problems if the problem is right in front of them.

When the release vote happens, encourage folks to test and +1
the release.  If it passes, woohoo!  If not, then listen to the
reasons given by the other PMC members and see if you can make
enough changes to the release to get those extra +1s.

In other words, collaborate.

....Roy=

From general-return-2845-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 10:00:23 2011
Return-Path: <general-return-2845-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 90988 invoked from network); 18 Jan 2011 10:00:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 10:00:21 -0000
Received: (qmail 15418 invoked by uid 500); 18 Jan 2011 10:00:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12527 invoked by uid 500); 18 Jan 2011 10:00:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12517 invoked by uid 99); 18 Jan 2011 10:00:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 10:00:12 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 10:00:07 +0000
Received: from [192.168.1.71] (snvvpn2-10-73-152-c16.hq.corp.yahoo.com [10.73.152.16])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0I9xSHw060344
	for <general@hadoop.apache.org>; Tue, 18 Jan 2011 01:59:28 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1295344769;
	bh=eXe5KYRDLrPfYtZ/wUQx7Hf9vKmDt4RTeLsFQyYdpeQ=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:Mime-Version:Subject:
	 Date:References;
	b=k4pHMe4Aoj0AREQa5fRDYcd5u31Ls+zH4ZV0iAHo6q0HFT5kMMvXMj3ECpLWjNe7D
	 f1PObKcTGRLPIaOj8wMtHdSc19PkogvcUi3pQ+0NHp7uXAcgKPppEn7/5GZVbCDk8m
	 244/FKr/FfJUb0xlxsPjGJXsRS88NeLpms8uCNtI=
Message-Id: <70369193-35EB-4ECD-AB40-99F8F9424C4D@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <F5C6B0DD-DB20-4324-8A11-A2DE8E7169EB@gbiv.com>
Content-Type: multipart/alternative; boundary=Apple-Mail-91--790011745
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Date: Tue, 18 Jan 2011 01:59:27 -0800
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com> <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com> <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org> <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com> <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com> <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com> <AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com> <388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com> <AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com> <04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com> <AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com> <74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com> <F5C6B0DD-DB20-4324-8A11-A2DE8E7169EB@gbiv.com>
X-Mailer: Apple Mail (2.936)

--Apple-Mail-91--790011745
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit

Thanks for the clarifications Roy.

I considered either b) and c).

As I mentioned, the reason I think b) wasn't useful in this context is  
that we have, in several cases, 5-6 patches per jira (bug-fix on on  
top of a bug-fix) and several jiras didn't make sense for trunk since  
the bug didn't exist in trunk etc. etc. Also, I was considering a  
scenario where I would squash relevant patches together to produce a  
minimal set of coherent patches. Then there is work to remove Yahoo!  
specific commits.

IAC, I agree - we've spent too much time talking and too little doing  
actual work. Let me get the job done and folks can then weigh-in on  
the release at later point, folks might be willing to consider this  
more positively once they see the  branch, the change-log etc.

Of course we need to get the small number of remaining patches into  
trunk asap for 0.22 and beyond.

Arun

On Jan 18, 2011, at 12:20 AM, Roy T. Fielding wrote:

> I thought that this discussion would have reached some sensible
> understanding of how Apache projects work by now, but it seems not.
>
> On Jan 13, 2011, at 6:12 PM, Arun C Murthy wrote:
>> The version I'm offering to push to the community has fixed all of  
>> them, *plus* the added benefit of several stability and performance  
>> fixes we have done since 20.104.3, almost 10 internal releases.  
>> This is a battle tested and hardened version which we have deployed  
>> on 40,000+ nodes. It is a significant upgrade on 0.20.104.3 which  
>> we never deployed. I'm pretty sure *some* users will find that  
>> valuable. ;)
>>
>> Also, I've offered to push individual patches as a background  
>> activity on a branch - that should suffice, no? Or, do you consider  
>> this a blocker?
>>
>> Again, my goal in this exercise is to get a stable, improved  
>> version of Hadoop into the hands of our users asap, and focus on  
>> 0.22 and beyond.
>
> So, you have a bunch of changes that you want to contribute.
> Please do so.  There are several ways:
>
> a) break the changes down into a sequence of patches, create jira
>    issues for each one (or append to the existing issue), and then
>    provide the group with a list of the issue links so that people
>    can quickly +1 each one.  When it seems worthwhile to you, create
>    a branch off of some prior Apache release point in svn and commit
>    each patch to it until the branch is identical to (or, in your own
>    opinion, better than) the source code that you have tested locally.
>    Then RM a tarball and start a release vote.  Since all of this is
>    being done in jira and svn, others can help you do all but the
>    first part (breaking down the big patch).
>
> or
>
> b) create a branch off of some prior Apache release point in svn
>    and replay the internal Y! commits on that branch until the branch
>    source code is identical to what you have tested locally.  Then
>    RM a tarball based on that branch and start a release vote.
>    Since the history is now in svn, others could do the RM bit if
>    you don't have time.
>
> or
>
> c) create a branch off of some prior Apache release point in svn
>    and apply one big ugly patch to it.  Then RM a tarball based
>    on that branch and ask for a release vote.
>
> You will note that none of the above requires a discussion on this
> list prior to the release vote, though (a) would likely result in
> more +1s than (b), and (b) would likely receive more +1s than (c).
> Regardless, the release vote is a lazy majority decision.
>
> I do not believe that there is any rational reason to apply a
> single big patch.  "It takes too long" is nonsense -- you have
> already spent far more time discussing it than would be required
> to do it.  "It is too hard" is also nonsense -- use your version
> control system to extract the set of changes and just replay them
> (with appropriate changelog editing).  "It has already been tested
> at Y!" is simply irrelevant -- the source code has been tested, not
> the order in which the patches have been applied, so all you should
> care about is that the final branch code is comparable to the tested
> source code (i.e., use diff).
>
> Nevertheless, all contributions at Apache are voluntary.  Do what
> you have time for, now, with the understanding that others may or
> may not complete the task, and may or may not vote for the release.
>
> You can make a branch, apply the big patch, and stand by
> while the rest of the group chooses whether to just accept it
> as a big change.  Someone else might create a parallel branch to
> apply the specific changes in an orderly matter, or perhaps you'll
> discover an easy way to do that a few days from now.  Or it
> might just sit there and never be released.
>
> There is no need for the group to agree to a plan up front, just
> as there is no need for the group to approve a release just because
> someone did the work of RMing a tarball.  Sure, it might save
> a lot of time if potential disagreements can be resolved before
> work is done, but the fact is that people tend to disagree less
> with actual work products than with abstract plans.  After all,
> everyone has a plan.  It is also far easier to convince people
> to fix their own problems if the problem is right in front of them.
>
> When the release vote happens, encourage folks to test and +1
> the release.  If it passes, woohoo!  If not, then listen to the
> reasons given by the other PMC members and see if you can make
> enough changes to the release to get those extra +1s.
>
> In other words, collaborate.
>
> ....Roy


--Apple-Mail-91--790011745--

From general-return-2846-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 18:51:09 2011
Return-Path: <general-return-2846-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81340 invoked from network); 18 Jan 2011 18:51:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 18:51:09 -0000
Received: (qmail 81209 invoked by uid 500); 18 Jan 2011 18:51:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80949 invoked by uid 500); 18 Jan 2011 18:51:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80940 invoked by uid 99); 18 Jan 2011 18:51:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 18:51:05 +0000
X-ASF-Spam-Status: No, hits=-5.0 required=10.0
	tests=RCVD_IN_DNSWL_HI,SINGLE_HEADER_2K
X-Spam-Check-By: apache.org
Received-SPF: unknown (athena.apache.org: error in processing during lookup of sseverance@ebay.com)
Received: from [216.113.175.153] (HELO den-mipot-002.corp.ebay.com) (216.113.175.153)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 18:51:00 +0000
DomainKey-Signature: s=corp; d=ebay.com; c=nofws; q=dns;
  h=X-EBay-Corp:X-IronPort-AV:Received:Received:From:To:Date:
   Subject:Thread-Topic:Thread-Index:Message-ID:References:
   References:In-Reply-To:Accept-Language:Content-Language:
   X-MS-Has-Attach:X-MS-TNEF-Correlator:acceptlanguage:
   x-ems-proccessed:x-ems-stamp:Content-Type:
   Content-Transfer-Encoding:MIME-Version:X-CFilter;
  b=ECe2Ll2b6RvSPr0Ca3pGDtX5xLTL1tinMicgKaX8cHz/9AjHBPY6uasD
   ZeMjbHyNWu/ayfsAvCTTzttRrPzVvZcXcpf6LRIzlODQLe7qQBDZ06HHj
   aPoV8efSSmzUf+c;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=ebay.com; i=sseverance@ebay.com; q=dns/txt; s=corp;
  t=1295376661; x=1326912661;
  h=from:to:date:subject:message-id:references:in-reply-to:
   content-transfer-encoding:mime-version;
  z=From:=20"Severance,=20Steve"=20<sseverance@ebay.com>|To:
   =20"general@hadoop.apache.org"=20<general@hadoop.apache.o
   rg>|Date:=20Tue,=2018=20Jan=202011=2011:53:32=20-0700
   |Subject:=20RE:=20[DISCUSS]=20Hadoop=20Security=20Release
   =20off=20Yahoo!=20patchset|Message-ID:=20<1BCDF92034F1644
   5AF61BDCCF4520A4EA4D7456A78@DEN-MEXMS-001.corp.ebay.com>
   |References:=20<516684F5-0052-4381-805D-760B61DECB16@yaho
   o-inc.com>=0D=0A=20<366A9E58-5BD7-497D-9AE1-229959ED4065@
   apache.org>=0D=0A=20<18C5C999-4680-4684-BC55-A430C40FD746
   @yahoo-inc.com>=0D=0A=20<AANLkTim+y7CBi8WeV9W+FVckn-1D1SA
   V2hSc6RnEebSn@mail.gmail.com>=0D=0A=20<F8414465-24F2-4F40
   -97F1-B13048820F24@yahoo-inc.com>=0D=0A=20<D7FA8ED2-D9C6-
   4B45-AC3B-9E965ADD9F5D@apache.org>=0D=0A=20<2A07F1E6-7096
   -493B-B92E-89938689DD50@yahoo-inc.com>=0D=0A=20<5CDDF962-
   5828-459F-87C3-5033EC21E9BF@mac.com>=0D=0A=20<075308A1-12
   9B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>=0D=0A=20<AANLkTi
   mbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com>=0D
   =0A=20<388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com
   >=0D=0A=20<AANLkTim=3D+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jj
   z@mail.gmail.com>=0D=0A=20<04705B3C-49A9-46B3-8AA9-5673EF
   BDE544@yahoo-inc.com>=0D=0A=20<AANLkTinAk+8YNLHb580uFibMW
   tNc=3DoYdqq+sOdBc+=3D6U@mail.gmail.com>=0D=0A=20<74BDFA74
   -DB12-4109-89DF-B353FC7296C4@yahoo-inc.com>=0D=0A=20<AANL
   kTimLLrfYbaaOzbEV9dcsaMFLeMmdgx73FasdxGhA@mail.gmail.com>
   =0D=0A=20<B4C8E6B2-A343-449B-AE63-DE5E3F883ABA@yahoo-inc.
   com>=0D=0A=20<36CBB8E6-05D6-4AC0-838C-14B1C2B19CBD@!=20
   =20mac.com>=0D=0A=20<00F14279-2802-4CC3-91AC-481AE257FD8B
   @yahoo-inc.com>=0D=0A=20<AANLkTikqbRrU+3+y2nctu7RzKNb7+P8
   kd=3D4az+eQHzKW@mail.gmail.com>=0D=0A=20<58FF3525-B734-44
   A0-A5C6-45282A23F06F@yahoo-inc.com>=0D=0A=20<AANLkTik-=3D
   ZDac0iEbYuDHeBrRosQ3JNGvRbf7MxexQND@mail.gmail.com>=0D=0A
   =20<7D4397E2-B618-4BD1-8E1E-08B1C598B76F@yahoo-inc.com>
   =0D=0A=20<10BA9684-51FF-4332-A2FD-5F648E0AAF8C@Holsman.NE
   T>=0D=0A=20<4E3C214A-16C6-46FF-908B-8EFB075A24E0@yahoo-in
   c.com>|In-Reply-To:=20<4E3C214A-16C6-46FF-908B-8EFB075A24
   E0@yahoo-inc.com>|Content-Transfer-Encoding:=20quoted-pri
   ntable|MIME-Version:=201.0;
  bh=v/AuRw2+nDq7X/21Lr/bWi7r3AhU7cubzF2I7pjvGB8=;
  b=vmA7tN8Y/cGtqjensk3nD/QA43ob6q0BRH+E6Nwk6ey/JaxYbSzF80MZ
   L0W/lb2yenLU5zoU3FDyi7sBnS9gk4RIIqMUtaXIbO5iBLm9vIsJX5a1m
   BLkraspbG/MqXrB;
X-EBay-Corp: Yes
X-IronPort-AV: E=Sophos;i="4.60,339,1291622400"; 
   d="scan'208";a="844323"
Received: from den-vtenf-001.corp.ebay.com (HELO DEN-MEXHT-003.corp.ebay.com) ([10.101.112.212])
  by den-mipot-002.corp.ebay.com with ESMTP; 18 Jan 2011 10:50:38 -0800
Received: from DEN-MEXMS-001.corp.ebay.com ([10.241.16.225]) by
 DEN-MEXHT-003.corp.ebay.com ([10.241.17.54]) with mapi; Tue, 18 Jan 2011
 11:50:37 -0700
From: "Severance, Steve" <sseverance@ebay.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 18 Jan 2011 11:53:32 -0700
Subject: RE: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Thread-Topic: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Thread-Index: Acu0GHDTGqR/wYWwQz6EtfzeWHusfQDKIF1Q
Message-ID: <1BCDF92034F16445AF61BDCCF4520A4EA4D7456A78@DEN-MEXMS-001.corp.ebay.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
 <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
 <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
 <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
 <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
 <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
 <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
 <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
 <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
 <AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com>
 <388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com>
 <AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com>
 <04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com>
 <AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com>
 <74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com>
 <AANLkTimLLrfYbaaOzbEV9dcsaMFLeMmdgx73FasdxGhA@mail.gmail.com>
 <B4C8E6B2-A343-449B-AE63-DE5E3F883ABA@yahoo-inc.com>
 <36CBB8E6-05D6-4AC0-838C-14B1C2B19CBD@!  mac.com>
 <00F14279-2802-4CC3-91AC-481AE257FD8B@yahoo-inc.com>
 <AANLkTikqbRrU+3+y2nctu7RzKNb7+P8kd=4az+eQHzKW@mail.gmail.com>
 <58FF3525-B734-44A0-A5C6-45282A23F06F@yahoo-inc.com>
 <AANLkTik-=ZDac0iEbYuDHeBrRosQ3JNGvRbf7MxexQND@mail.gmail.com>
 <7D4397E2-B618-4BD1-8E1E-08B1C598B76F@yahoo-inc.com>
 <10BA9684-51FF-4332-A2FD-5F648E0AAF8C@Holsman.NET>
 <4E3C214A-16C6-46FF-908B-8EFB075A24E0@yahoo-inc.com>
In-Reply-To: <4E3C214A-16C6-46FF-908B-8EFB075A24E0@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
x-ems-proccessed: 10SqDH0iR7ekR7SRpKqm5A==
x-ems-stamp: K8q6O4WUDDN5JJklXUq/vQ==
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-CFilter: Scanned

I want to thank Yahoo! for this release. At eBay we are very excited about =
the opportunity to test a build of Hadoop that has already been extensively=
 field tested on large clusters. At eBay we are primarily concerned with cl=
uster availability and throughput so having a build like this available to =
the community is a huge win.

Hats off to Arun, Eric and everyone at Yahoo! for releasing this.

Steve

-----Original Message-----
From: Eric Baldeschwieler [mailto:eric14@yahoo-inc.com]=20
Sent: Friday, January 14, 2011 10:25 AM
To: general@hadoop.apache.org
Cc: general@hadoop.apache.org
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset

Hi Ian,

Thanks for holding off on that last .5. I've been working in a big email gi=
ving move context on this. Let me preview some issues.=20

Our goal with this branch is two fold: 1) get the code out in a branch quic=
kly so we an collaborate on it with the community. 2) not change the charac=
ter of the code. See testing below. We're happy to compromise any other dim=
ension, as long as we can do 1&2 above.=20

1) I agree this is not a good precedent. We don't support mega-patches in g=
eneral. We are doing this as part of discontinuing the "yahoo distribution =
of Hadoop".  We don't plan to continue doing 30 person year projects outsid=
e apache and then merging them in!!

2) append is hard. It is so hard we rewrote the entire write pipeline (5 pe=
rson-years work) in trunk after giving up on the codeline you are suggestin=
g we merge in. That work is what distinguishes all post 20 releases from 20=
 releases in my mind. I dont trust the 20 append code line. We've been hurt=
 badly by it.  We did the rewrite only after losing a bunch of production d=
ata a bunch of times with the previous code line.  I think the various 20 a=
ppend patch lines may be fine for specialized hbase clusters, but they does=
n't have the rigor behind them to bet your business in them.

3) I think having a very stable recent codeline available for teams coming =
into Hadoop who want to run big business apps and contribute code back is v=
ery helpful. I've been talking to folks in other orgs and they've expressed=
 a huge amount of interest in this work, but begged us to put it into apach=
e, so their oversight bodies will let them use it.=20

4) we're happy to incorporate ideas into how to best merge the work into tr=
unk. Let's find the most cost effective way to preserve the most devel data=
 possible.=20

5) testing. Ian, I think you do us a disservice when you talk about us just=
 testing in our environments. If you look at the history of the project, we=
've been the force behind every stable release of apache Hadoop.  And all t=
he non-apache Hadoop release had been tracking this patch set. We fully sup=
port the community developing independent testing capabilities.  We plan to=
 contribute to that effort.  But we are the organization with far and away =
the best record for testing Hadoop.=20

We are proud of thus release, we want to share it. Help us sort out how.=20

Thanks!

---
E14 - via iPhone

On Jan 14, 2011, at 6:15 AM, "Ian Holsman" <hadoop@holsman.net> wrote:

> (with my Apache hat on)
> I'm -0.5 on doing this as one big mega-patch and not including append (as=
 opposed to a series of smaller patches).
>=20
> for the following reasons:
>=20
> 1. It encourages bad behavior. We want discussion (and development) to ha=
ppen on the lists, not in some office. By allowing these large code-dumps i=
t condones this behavior, and we will likely see it again and again. Like i=
t or not, this is not the apache model of open source governance.=20
>=20
> 2. There is a risk that some code that is not in a JIRA or separate patch=
 creeps in unwittingly. This isn't a major deal per se, but we don't really=
 have the proper paper trail, or the documentation on what bug it fixed etc=
 etc.
>=20
> 3. Other groups (Facebook for example) are running with their own set of =
patches. They currently have the luxury of examining each individual patch =
to decide if they want to integrate it (and test it) in their environment. =
We are forcing them to do the work of finding the bits they want in this hu=
ge patch.
>=20
> 4. By not including the append patch, we are making this release unusable=
 for a large portion of our community who run hbase.
>=20
> 5. It makes it very hard to test. While It makes me comfortable that it h=
as gone through Yahoo!'s QA and is running in their environments, it doesn'=
t mean that it will work in other organizations who have different workload=
 mixes and software running on them. With one huge patch it makes it all or=
 nothing.. either they take the code-drop and perform a large QA-integratio=
n effort, or they forgo the whole patch together.
>=20
>=20
> **BUT** we have both the Yahoo! & Cloudera guys happy to do it, and to sp=
end their time doing it.. so I think having the code-drop will put us in a =
better place then where we are.
>=20
>=20
> BTW, I'd like to point out a discrepancy here:
>=20
> On another thread discussing hadoop-0.20-append as a separate branch, mos=
t people agreed that new features shouldn't be added to 0.20, now we have a=
 major feature and we are all gung ho for it..=20
>=20
> --Ian
>=20
> On Jan 14, 2011, at 2:21 AM, Arun C Murthy wrote:
>=20
>>=20
>> On Jan 13, 2011, at 10:59 PM, Stack wrote:
>>=20
>>> (Man, it was looking good there for a second when 0.20.100 was about
>>> security+append!)
>>>=20
>>> Good luck w/ the release Arun.
>>>=20
>>=20
>> Thanks!
>>=20
>>> We might be following your 0.20.100 with a 0.20.200 append.
>>>=20
>>=20
>> Super!
>>=20
>> Arun
>=20

From general-return-2847-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 19:10:07 2011
Return-Path: <general-return-2847-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59451 invoked from network); 18 Jan 2011 19:10:06 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 19:10:06 -0000
Received: (qmail 26204 invoked by uid 500); 18 Jan 2011 19:10:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26037 invoked by uid 500); 18 Jan 2011 19:10:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26025 invoked by uid 99); 18 Jan 2011 19:10:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 19:10:04 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 19:09:57 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=G9mdLWjBL0oRZGVXN6hjotGckByR1p4TVulNcRc2A0EpEuqt1+IYZk6o
   wQqgthqKCVhLDqAzjTUJSRFzR2Vvcyfdg1Mtd0ObTmPSeYjha2SAuX/p8
   gZ0MAKxGNPlrvj5;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1295377794; x=1326913794;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20[DISCUSS]=20Hadoop=20Security=20Release
   =20off=20Yahoo!=20patchset|Date:=20Tue,=2018=20Jan=202011
   =2019:10:26=20+0000|Message-ID:=20<3995AF5C-AA18-475B-938
   4-A532F1CC5D57@linkedin.com>|To:=20"general@hadoop.apache
   .org"=20<general@hadoop.apache.org>|MIME-Version:=201.0
   |Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<E80D31EF0BB1E84FAF8E786EE98C196C@linkedin
   .com>|In-Reply-To:=20<D160E684-7435-4E53-A766-231CCB38D1B
   4@yahoo-inc.com>|References:=20<516684F5-0052-4381-805D-7
   60B61DECB16@yahoo-inc.com>=0D=0A=20<366A9E58-5BD7-497D-9A
   E1-229959ED4065@apache.org>=0D=0A=20<18C5C999-4680-4684-B
   C55-A430C40FD746@yahoo-inc.com>=0D=0A=20<AANLkTim+y7CBi8W
   eV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>=0D=0A=20<F8
   414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>=0D=0A
   =20<D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>=0D
   =0A=20<2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com
   >=0D=0A=20<5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
   =0D=0A=20<075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.
   com>=0D=0A=20<4D34A270.1060102@apache.org>=20<3FEB7C1D-AE
   89-46BE-ABF2-B595B01E222A@mac.com>=0D=0A=20<D160E684-7435
   -4E53-A766-231CCB38D1B4@yahoo-inc.com>;
  bh=6XsB/DnyOvArQOzM0OB0VBHe9dcSOeEI7SdGYySPxAI=;
  b=JsNgsCa2ghWmbz6cCFPbwba0x1dOhSoeHC7Ti1kd3zYZs7HYcRRgpYIS
   06SE60vITSDIoC2BD0phyfFQpiLTqeJSAnA2xEGc4ChvbcuDgPc/jnDTU
   dJfyG0VTWZAWyQB;
X-IronPort-AV: E=Sophos;i="4.60,340,1291622400"; 
   d="scan'208";a="19859802"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi id 14.01.0251.000; Tue,
 18 Jan 2011 11:10:27 -0800
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Thread-Topic: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Thread-Index: AQHLQyNHDr2kMsQuFU+y4yuN2iEG2pL6/9oAgNt/qySAAIijAIAAK1WAgAFS1YA=
Date: Tue, 18 Jan 2011 19:10:26 +0000
Message-ID: <3995AF5C-AA18-475B-9384-A532F1CC5D57@linkedin.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
 <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
 <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
 <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
 <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
 <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
 <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
 <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
 <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
 <4D34A270.1060102@apache.org> <3FEB7C1D-AE89-46BE-ABF2-B595B01E222A@mac.com>
 <D160E684-7435-4E53-A766-231CCB38D1B4@yahoo-inc.com>
In-Reply-To: <D160E684-7435-4E53-A766-231CCB38D1B4@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <E80D31EF0BB1E84FAF8E786EE98C196C@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

On Jan 17, 2011, at 2:56 PM, Eric Baldeschwieler wrote:
>=20
>  Right now you don't have the choice of an Apache release if you are look=
ing for a stabilized modern version of Hadoop. =20


	Can we ratchet down the hyperbole to at least a point where I don't want t=
o vomit? Thanks.

	[For the record, some of us quite like our stable, almost-a-year Apache Ha=
doop 0.20.2 w/3 patches installations, thankyouverymuch.  (Those patches ar=
e for portability and capacity scheduler fixes, since the one in 0.20.2 is =
completely useless.)]=

From general-return-2848-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 19:48:48 2011
Return-Path: <general-return-2848-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21265 invoked from network); 18 Jan 2011 19:48:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 19:48:48 -0000
Received: (qmail 41751 invoked by uid 500); 18 Jan 2011 19:48:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 41556 invoked by uid 500); 18 Jan 2011 19:48:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 41541 invoked by uid 99); 18 Jan 2011 19:48:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 19:48:45 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 19:48:37 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p0IJlvou036945;
	Tue, 18 Jan 2011 11:47:57 -0800 (PST)
Received: from SP2-EX07VS04.ds.corp.yahoo.com ([98.137.59.33]) by
 SP2-EX07CAS01.ds.corp.yahoo.com ([98.137.59.37]) with mapi; Tue, 18 Jan 2011
 11:47:57 -0800
From: Koji Noguchi <knoguchi@yahoo-inc.com>
To: "mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>
Date: Tue, 18 Jan 2011 11:47:55 -0800
Subject: Re: How to use combiner in hadoop streaming
Thread-Topic: How to use combiner in hadoop streaming
Thread-Index: Acu25sRDc5IXtNgIQDSUPLFcOTNATwAYdUuK
Message-ID: <C95B2E6B.62944%knoguchi@yahoo-inc.com>
In-Reply-To: <AANLkTina1=nYy3qGz6pYmP5NqRMEEXDMrFJ8rUd+3Dyq@mail.gmail.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

(Bcc general@. This is for Hadoop project level announcements and
discussions. http://hadoop.apache.org/mailing_lists.html . Using
mapreduce-user@)

Yeshan,=20

Which version of hadoop are you using?

Officially, combiner for non-java command was added in 0.21
https://issues.apache.org/jira/browse/HADOOP-4842

but I=B9d bet some of those 0.20s have this backported.

Koji



On 1/18/11 12:06 AM, "Harsh J" <qwertymaniac@gmail.com> wrote:

> Right now, the Combiner has to be a Java Class. A command is
> unacceptable, as noted here:
> http://wiki.apache.org/hadoop/HadoopStreaming
>=20
> On Tue, Jan 18, 2011 at 1:30 PM, xieyeshan <xieyeshan@software.ict.ac.cn>
> wrote:
>> hi,all
>>=20
>> =A0 =A0 I want to use combiner in hadoop streaming, but combiner is not =
the same
>> as mapper or reducer in streaming.How could I do this?
>> =A0 =A0 Thanks!
>>=20
>>=20
>> Yours,
>>=20
>> Yeshan Xie
>>=20
>> 2011-01-18
>>=20
>>=20
>>=20
>> xieyeshan
>>=20
>=20
>=20
>=20
> --
> Harsh J
> www.harshj.com
>=20


From general-return-2849-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 18 23:23:46 2011
Return-Path: <general-return-2849-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63910 invoked from network); 18 Jan 2011 23:23:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jan 2011 23:23:42 -0000
Received: (qmail 91271 invoked by uid 500); 18 Jan 2011 23:23:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91204 invoked by uid 500); 18 Jan 2011 23:23:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91196 invoked by uid 99); 18 Jan 2011 23:23:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 23:23:39 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Jan 2011 23:23:31 +0000
Received: from SP2-EX07CAS02.ds.corp.yahoo.com (sp2-ex07cas02.corp.sp2.yahoo.com [98.137.59.38])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0INMIkP003143;
	Tue, 18 Jan 2011 15:22:18 -0800 (PST)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([98.137.59.23]) by
 SP2-EX07CAS02.ds.corp.yahoo.com ([98.137.59.38]) with mapi; Tue, 18 Jan 2011
 15:22:18 -0800
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 18 Jan 2011 15:22:09 -0800
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Thread-Topic: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Thread-Index: Acu3ZowKMHpUa0PRR0iyyCLzstAu8A==
Message-ID: <2090582B-B586-4F81-A724-6A0C55820BB5@yahoo-inc.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
 <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
 <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
 <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
 <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
 <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
 <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
 <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
 <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
 <4D34A270.1060102@apache.org> <3FEB7C1D-AE89-46BE-ABF2-B595B01E222A@mac.com>
 <D160E684-7435-4E53-A766-231CCB38D1B4@yahoo-inc.com>
 <3995AF5C-AA18-475B-9384-A532F1CC5D57@linkedin.com>
In-Reply-To: <3995AF5C-AA18-475B-9384-A532F1CC5D57@linkedin.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Sounds like you are agreeing with me in your own way Allen ;-)

We're getting > 2x better throughput and stability from the capacity schedu=
ler in this branch.  I'd love to get you feedback on that.

The more nodes and users in your deployment, the more improvements you will=
 see.=20

---
E14 - via iPhone

On Jan 18, 2011, at 11:10 AM, "Allen Wittenauer" <awittenauer@linkedin.com>=
 wrote:

> On Jan 17, 2011, at 2:56 PM, Eric Baldeschwieler wrote:
>>=20
>> Right now you don't have the choice of an Apache release if you are look=
ing for a stabilized modern version of Hadoop. =20
>=20
>=20
>    Can we ratchet down the hyperbole to at least a point where I don't wa=
nt to vomit? Thanks.
>=20
>    [For the record, some of us quite like our stable, almost-a-year Apach=
e Hadoop 0.20.2 w/3 patches installations, thankyouverymuch.  (Those patche=
s are for portability and capacity scheduler fixes, since the one in 0.20.2=
 is completely useless.)]

From general-return-2850-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 19 07:49:49 2011
Return-Path: <general-return-2850-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47473 invoked from network); 19 Jan 2011 07:49:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jan 2011 07:49:49 -0000
Received: (qmail 66210 invoked by uid 500); 19 Jan 2011 07:49:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66162 invoked by uid 500); 19 Jan 2011 07:49:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66154 invoked by uid 99); 19 Jan 2011 07:49:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Jan 2011 07:49:42 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Jan 2011 07:49:35 +0000
Received: by qyk7 with SMTP id 7so265430qyk.14
        for <general@hadoop.apache.org>; Tue, 18 Jan 2011 23:49:14 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=3FAzO/u316ZKVLsBDgeqfOQc2Cu+RfPEshuKC7UQdCw=;
        b=iEeeJGP+yiG56wqt0V3kvoVQg0xbbOh8pUbfpTfPND0jnoLdqsLnfRSyAXNOv7naY9
         bJPcOtScCDBEKQgLMn22noEtPxKjJTJKb6lBOECk5ZoMgSK6gNO8Dsp+CxI5bk8opHp9
         nS7BzsenfbpQAnvGCHT3OqCafKNqj7bRsxVLc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=pxIAvkadUEF+rXWe10ZaoNacNPQMWaMmUCg9ScRx/HZZH5N5Qe38fMIe1F2gDYp5VY
         //YtdgEi00OUYv7o8YMhh/cfQaAihNJ3D1DKw+75lrWDybUjqfU6yKBIeGJ8XCKv+rau
         +Deb7CjpI9A3iegcXk909b96MMdFpjUj8rQyw=
Received: by 10.229.249.17 with SMTP id mi17mr362538qcb.123.1295423354708;
        Tue, 18 Jan 2011 23:49:14 -0800 (PST)
Received: from [10.172.0.169] (h-64-236-128-62.nat.aol.com [64.236.128.62])
        by mx.google.com with ESMTPS id e29sm4531001qck.3.2011.01.18.23.49.08
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Tue, 18 Jan 2011 23:49:13 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <2090582B-B586-4F81-A724-6A0C55820BB5@yahoo-inc.com>
Date: Wed, 19 Jan 2011 09:49:05 +0200
Content-Transfer-Encoding: quoted-printable
Message-Id: <704BFF6E-AC1D-4769-8856-BFBD125D4D2C@Holsman.NET>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com> <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com> <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org> <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com> <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com> <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com> <4D34A270.1060102@apache.org> <3FEB7C1D-AE89-46BE-ABF2-B595B01E222A@mac.com> <D160E684-7435-4E53-A766-231CCB38D1B4@yahoo-inc.com> <3995AF5C-AA18-475B-9384-A532F1CC5D57@linkedin.com> <2090582B-B586-4F81-A724-6A0C55820BB5@yahoo-inc.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

That's what scares me, and highlights one of the points I made before.

If someone wants to just use the capacity scheduling improvements, and =
not other parts, they will find it hard. I think Roy's suggestion of =
applying the commits individually to the branch from your current =
working branch would help with this.

regards
Ian

On Jan 19, 2011, at 1:22 AM, Eric Baldeschwieler wrote:

> Sounds like you are agreeing with me in your own way Allen ;-)
>=20
> We're getting > 2x better throughput and stability from the capacity =
scheduler in this branch.  I'd love to get you feedback on that.
>=20
> The more nodes and users in your deployment, the more improvements you =
will see.=20
>=20
> ---
> E14 - via iPhone
>=20
> On Jan 18, 2011, at 11:10 AM, "Allen Wittenauer" =
<awittenauer@linkedin.com> wrote:
>=20
>> On Jan 17, 2011, at 2:56 PM, Eric Baldeschwieler wrote:
>>>=20
>>> Right now you don't have the choice of an Apache release if you are =
looking for a stabilized modern version of Hadoop. =20
>>=20
>>=20
>>   Can we ratchet down the hyperbole to at least a point where I don't =
want to vomit? Thanks.
>>=20
>>   [For the record, some of us quite like our stable, almost-a-year =
Apache Hadoop 0.20.2 w/3 patches installations, thankyouverymuch.  =
(Those patches are for portability and capacity scheduler fixes, since =
the one in 0.20.2 is completely useless.)]


From general-return-2851-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 19 18:08:07 2011
Return-Path: <general-return-2851-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39849 invoked from network); 19 Jan 2011 18:08:06 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jan 2011 18:08:06 -0000
Received: (qmail 13454 invoked by uid 500); 19 Jan 2011 18:08:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13142 invoked by uid 500); 19 Jan 2011 18:08:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13132 invoked by uid 99); 19 Jan 2011 18:08:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Jan 2011 18:08:01 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [64.78.17.19] (HELO EXHUB018-4.exch018.msoutlookonline.net) (64.78.17.19)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Jan 2011 18:07:53 +0000
Received: from EXVMBX018-1.exch018.msoutlookonline.net ([64.78.17.47]) by
 EXHUB018-4.exch018.msoutlookonline.net ([64.78.17.19]) with mapi; Wed, 19 Jan
 2011 10:07:33 -0800
From: Scott Carey <scott@richrelevance.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 19 Jan 2011 10:09:41 -0800
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Thread-Topic: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Thread-Index: Acu4A75ppI6F1MqIREK7+QBIPu9T5Q==
Message-ID: <C95C64DD.1A074%scott@richrelevance.com>
In-Reply-To: <AANLkTik=KxWZKkwu0FJ0Uej7JfDWe7Crw5+i9_BV9QMX@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-MacOutlook/14.2.0.101115
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0



On 1/14/11 11:24 AM, "Dhruba Borthakur" <dhruba@gmail.com> wrote:

>>
>>
>> 1) I agree this is not a good precedent. We don't support mega-patches
>>in
>> general. We are doing this as part of discontinuing the "yahoo
>>distribution
>> of Hadoop".  We don't plan to continue doing 30 person year projects
>>outside
>> apache and then merging them in!!
>>
>>
>I think this is a very dangerous precedent and completely unwarranted.
>mega-patches are bad and is totally not the Apache way to go. I think if
>you
>want to contribute it back to Apache, you should avoid the mega-patch
>completely.
>

The mega-patch is not being applied to Trunk, or even the common 0.20.x
branch, so its danger is significantly mitigated.

If there is still a lot of worry about the mega-patch, there is one other
compromise:

* Take Cloudera's linearization of Y! Patches that go from 0.20.2 to
0.20.104.3 and commit them individually.
* Then take a mini-mega patch from there to the latest Y!.

That shouldn't be too hard, and meets Arun's goal of not changing the
character of the code so that testing is minimized/eliminated.  And it
incorporates some hard work on the Cloudera side that will be useful if
debugging on that branch is necessary.


I want to see as much work as possible on 0.22 -- there are major
improvements there that all can share and get the community more unified
again.  One drawback of this release is it could encourage the community
to squat on 0.20 even longer...  But sharing all that work can be seen as
a necessary step to being able let go of 0.20 and move on as well.


From general-return-2852-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 19 18:12:42 2011
Return-Path: <general-return-2852-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 40979 invoked from network); 19 Jan 2011 18:12:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jan 2011 18:12:41 -0000
Received: (qmail 23281 invoked by uid 500); 19 Jan 2011 18:12:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23012 invoked by uid 500); 19 Jan 2011 18:12:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23004 invoked by uid 99); 19 Jan 2011 18:12:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Jan 2011 18:12:37 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of shv.hadoop@gmail.com designates 209.85.215.48 as permitted sender)
Received: from [209.85.215.48] (HELO mail-ew0-f48.google.com) (209.85.215.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Jan 2011 18:12:31 +0000
Received: by ewy20 with SMTP id 20so603829ewy.35
        for <general@hadoop.apache.org>; Wed, 19 Jan 2011 10:12:10 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=hw+u1u/jkpcSyJcigFTBxJx2LHuVXoQuauI5LIi7tWw=;
        b=XPAvzdUqIBKVrxL7UGjpetjumFwp97vIvvcvIWv8jwQyYh0Q+64aHLYWdp2pAHAunn
         aI8hIRWLTdSTjAasPydpseUTmDptXFYUkWRGBxQ14s4PjJsi2aIBHKCrAQE91SvNbWjJ
         O7WnAcYiYKMlFG9VzA+JnJAYQLgcjDK6/HhjY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Dp9ymrAc+OwM2RGHgCRI2jCp+lyobNgHtoXNEan1E8eBOGOpFxBVtflrQwy1yWDo1d
         Mcu0gt1VxUqIOhoBtkmWbk2ykLFYli4PozI4jAlpneoFzERimbTWF5C7Jpl1OUEFE/C7
         CmacBTVLAwD2sGoMN9Ik9+U7pIWNO7SWvPZLk=
MIME-Version: 1.0
Received: by 10.14.127.137 with SMTP id d9mr1233949eei.2.1295460730674; Wed,
 19 Jan 2011 10:12:10 -0800 (PST)
Received: by 10.14.47.140 with HTTP; Wed, 19 Jan 2011 10:12:08 -0800 (PST)
In-Reply-To: <704BFF6E-AC1D-4769-8856-BFBD125D4D2C@Holsman.NET>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com>
	<366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org>
	<18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com>
	<AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com>
	<F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com>
	<D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org>
	<2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com>
	<5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com>
	<075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com>
	<4D34A270.1060102@apache.org>
	<3FEB7C1D-AE89-46BE-ABF2-B595B01E222A@mac.com>
	<D160E684-7435-4E53-A766-231CCB38D1B4@yahoo-inc.com>
	<3995AF5C-AA18-475B-9384-A532F1CC5D57@linkedin.com>
	<2090582B-B586-4F81-A724-6A0C55820BB5@yahoo-inc.com>
	<704BFF6E-AC1D-4769-8856-BFBD125D4D2C@Holsman.NET>
Date: Wed, 19 Jan 2011 10:12:08 -0800
Message-ID: <AANLkTikq-bPfXviWThvh=--cWEDYPfd0iEFSE_m_RbMn@mail.gmail.com>
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba5bb8fbac2734049a36f417
X-Virus-Checked: Checked by ClamAV on apache.org

--90e6ba5bb8fbac2734049a36f417
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Jan 18, 2011 at 11:49 PM, Ian Holsman <hadoop@holsman.net> wrote:

> I think Roy's suggestion of applying the commits individually to the branch
> from your current working branch would help with this.
>

I am sure this is not what Roy suggested. Ian. I think the idea is simple.
If you decide to donate to a non-profit organization you are free to choose
the form of your donation.
There is a tradeoff between its usability to the community and the effort
Softwareput into making it such.
The community can evaluate the quality of the donation and decide on how to
consume it.

Also, this is different from the 0.20-append discussion, imo, as it doesn't
require additional
community resources. I can see many of them are dedicated to building 0.22
as we speak.

Thanks,
--Konstantin

--90e6ba5bb8fbac2734049a36f417--

From general-return-2853-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 19 22:01:03 2011
Return-Path: <general-return-2853-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 10844 invoked from network); 19 Jan 2011 22:01:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jan 2011 22:01:02 -0000
Received: (qmail 36293 invoked by uid 500); 19 Jan 2011 22:00:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 35864 invoked by uid 500); 19 Jan 2011 22:00:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 35852 invoked by uid 99); 19 Jan 2011 22:00:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Jan 2011 22:00:57 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 74.125.83.48 as permitted sender)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Jan 2011 22:00:51 +0000
Received: by gwj15 with SMTP id 15so616636gwj.35
        for <general@hadoop.apache.org>; Wed, 19 Jan 2011 14:00:30 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:date:x-google-sender-auth
         :message-id:subject:from:to:content-type;
        bh=vv+AQ8zLODFfJmULgqQ57xC6CrrC8yYNxK8Cm0l5yUM=;
        b=FwXuNC7/RUsl0LetwqYQ4xI4dE76zRWM1krBrirrLqKRiuQ1wTxI2CeIWnR1Tt/cyL
         fsWXsPYUSNRODZZm4lvNc0pa7I+QV5ShdTBw1ITRH89PPnjh++3xFHxeBCu1dhCt+/S1
         AOBwdCEo9s89vOe3BmMk38OwHdfVS3+yj+rqw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        b=a5QMNQZlROpVbBM87Oqb38gYEFJ4zoBmxrX+w6L3b4IVvD+FD1kegKwKKVZnifj7AW
         CWsgxOHbR1tWwzmEWLdRcbCvXw7mK6+AkJ+/wpgCHFB6E7Pp/IB0kirvo3QV0iSeu7Vg
         i1BxVeds+NLz2/IRD2qgxKAt/gQc6VP8LEieQ=
MIME-Version: 1.0
Received: by 10.216.48.5 with SMTP id u5mr1128762web.96.1295474429825; Wed, 19
 Jan 2011 14:00:29 -0800 (PST)
Sender: saint.ack@gmail.com
Received: by 10.216.49.11 with HTTP; Wed, 19 Jan 2011 14:00:29 -0800 (PST)
Date: Wed, 19 Jan 2011 14:00:29 -0800
X-Google-Sender-Auth: Huowu4lr_xMEnACevqc6kF85R94
Message-ID: <AANLkTikn2ru1VZg+PwNcudz7YdzpiZHDA05EoF3ZpJb-@mail.gmail.com>
Subject: ANN: HBase 0.90.0 available for download
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org, user@hbase.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

The HBase crew are pleased to announce the availability of 0.90.0.

You can download it here:

  http://www.apache.org/dyn/closer.cgi/hbase/

HBase 0.90.0 is the major HBase release to follow HBase-0.20.x and the
fruit of the 0.89.x development release series we've been running over
the last few months.

Over 1k issues have been closed since 0.20.0.  Release notes are
available here: http://su.pr/8LbgvK.  Highlights include new master,
inter-cluster replication, data durability, improved performance,
improved i/o profile, etc.

We recommend all upgrade to this release but ONLY after a close reading
of requirements [1].  In particular, see the section on Hadoop where we
talk about the Hadoop versions and HBase 0.90.0.

There is no migration necessary.  Your data written with HBase 0.20.x
(or with HBase 0.89.x) is readable by HBase 0.90.0.  A shutdown and
restart after putting in place the new HBase should be all thats
involved.  That said, once done, there is no (easy) going back to 0.20.x once
the transition has been made.   HBase 0.90.0 and HBase 0.89.x write
region names differently in the filesystem.  Rolling restart from
0.20.x or 0.89.x to 0.90.0RC1 will not work.

A big thanks goes out to all who contributed to this release.

Yours,
The HBasistas

P.S. For why the version 0.90 and whats new in HBase 0.90, see slides
4-10 in this deck, [2].

1. http://hbase.apache.org/notsoquick.html#requirements
2. http://hbaseblog.com/2010/07/04/hug11-hbase-0-90-preview-wrap-up/

From general-return-2854-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 20 02:39:31 2011
Return-Path: <general-return-2854-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 32188 invoked from network); 20 Jan 2011 02:39:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jan 2011 02:39:30 -0000
Received: (qmail 62680 invoked by uid 500); 20 Jan 2011 02:39:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62455 invoked by uid 500); 20 Jan 2011 02:39:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62446 invoked by uid 99); 20 Jan 2011 02:39:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Jan 2011 02:39:26 +0000
X-ASF-Spam-Status: No, hits=3.7 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of liulei412@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Jan 2011 02:39:18 +0000
Received: by fxm2 with SMTP id 2so110711fxm.35
        for <general@hadoop.apache.org>; Wed, 19 Jan 2011 18:38:55 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=r9DeRFaHcSIOWH+vjT4puithEIIGZYdX0PFXD96yExI=;
        b=l9niiR/Hrq25kgvJxgev2JIHW0ek3bfqhVeosMO9QLZw5qW8/ckli/kSriimKfmITB
         SauhFO02xWXwpqiH5ucE0j/TCbb8wkfu56rhNfJCLEowxnbc1Q+zjNPT6gD9uT3JbPzm
         PrOB5njJnt2mEifsMVzVWtOGQ30jGWX7GRHlM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=IOObcKUbk/GOY9T68H7HhcjQFX8/m6PiZijyf8SoUKTzzcjRBCA2m3gK2PbMcV7Yhr
         a8CIY7y596lm/2yXCpUEwpn/XYoUgDC5avoMTzmB2H5vBESv6ieXelGFQSgQhTsPkuOE
         nFdaDqnIHSAQF73SlEHRQU9dMDkcUWPFLwIxo=
MIME-Version: 1.0
Received: by 10.103.240.6 with SMTP id s6mr980161mur.33.1295491135253; Wed, 19
 Jan 2011 18:38:55 -0800 (PST)
Received: by 10.102.48.15 with HTTP; Wed, 19 Jan 2011 18:38:55 -0800 (PST)
Date: Thu, 20 Jan 2011 10:38:55 +0800
Message-ID: <AANLkTi=WmHZjXOik8UjzzUjGJ_iX=bKV+LSfs64UgVH-@mail.gmail.com>
Subject: use counter to statistics file row number
From: lei liu <liulei412@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016369cff89ed2c74049a3e08eb
X-Virus-Checked: Checked by ClamAV on apache.org

--0016369cff89ed2c74049a3e08eb
Content-Type: text/plain; charset=GB2312
Content-Transfer-Encoding: quoted-printable

I use counter to statistics  file row number in Mapper, example below code:

public void map(LongWritable key, Text value,
            OutputCollector<Text, IntWritable> output, Reporter reporter)
            throws IOException {

        try {

            reporter.incrCounter("row", "num", 1);

        } catch (Throwable e) {
            e.printStackTrace();
            throw new RuntimeException(e);
        }
    }

Could everyone tell me whether there are any risks do so=A3=BF


Thanks=A3=AC

LiuLei

--0016369cff89ed2c74049a3e08eb--

From general-return-2855-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 20 03:33:38 2011
Return-Path: <general-return-2855-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82692 invoked from network); 20 Jan 2011 03:33:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jan 2011 03:33:37 -0000
Received: (qmail 89607 invoked by uid 500); 20 Jan 2011 03:33:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88966 invoked by uid 500); 20 Jan 2011 03:33:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88945 invoked by uid 99); 20 Jan 2011 03:33:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Jan 2011 03:33:31 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of venkatesh.kowluru@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Jan 2011 03:33:23 +0000
Received: by wye20 with SMTP id 20so191252wye.35
        for <multiple recipients>; Wed, 19 Jan 2011 19:33:03 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=yq2UqVJEl6iu3pbIY8CGiWLpBY8rKwzWFBDFO18nDlQ=;
        b=QBO0btcuTcQ2+CBw8y1PDtiMKWS/kK4woA88R0ykLacjSXDfgH9gHHCbQNV3bDABM2
         SM8VSHOzDHbxXNAypwnnetHsI6vASOSWR4Wnpsj542s9yoqKrXvARXTmSO8mHGTFMAg5
         1SR186T3m4uX4YOq6wb8gzj94MEjE1mzNeOlA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=qarwbQ/IQgLrpaHm3ZaexUSbHO7UYpvAURZ/bgQrLN6GS8Cw56ZV6nSEWSFu6wGUqs
         P6cLQDS2iAFqAljdmNIybn7FwRv4alEZ0VgFV7ktS5sorq/rUOnpZft5A5t3sU7oVQdU
         +CfbyYCGFAvnzP4l5X+ky/LB+oqd1uclw711Y=
MIME-Version: 1.0
Received: by 10.227.127.132 with SMTP id g4mr1725488wbs.36.1295494382846; Wed,
 19 Jan 2011 19:33:02 -0800 (PST)
Received: by 10.227.143.141 with HTTP; Wed, 19 Jan 2011 19:33:02 -0800 (PST)
In-Reply-To: <AANLkTi=WmHZjXOik8UjzzUjGJ_iX=bKV+LSfs64UgVH-@mail.gmail.com>
References: <AANLkTi=WmHZjXOik8UjzzUjGJ_iX=bKV+LSfs64UgVH-@mail.gmail.com>
Date: Wed, 19 Jan 2011 19:33:02 -0800
Message-ID: <AANLkTi=XYK_RkNH0whQ8UU+S-+MK3eszJ48Vo-KhBmZi@mail.gmail.com>
Subject: Re: use counter to statistics file row number
From: venkatesh kavuluri <venkatesh.kowluru@gmail.com>
To: mapreduce-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e65a0eea7f827a049a3eca52
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e65a0eea7f827a049a3eca52
Content-Type: text/plain; charset=GB2312
Content-Transfer-Encoding: quoted-printable

(Bcc general@. This is for Hadoop project level discussions. Includingmapre=
duce
-user@)

Liu,

If you want the count of number of records in your input data set, the
map/reduce framework provides a default counter "Map input records". The
only caution to follow regarding the custom counters is to not exceed 20 pe=
r
application as they are very expensive. Generally counters are used to trac=
k
few important pieces of information.

Thanks,
Venkatesh Kavuluri

2011/1/19 lei liu <liulei412@gmail.com>

> I use counter to statistics  file row number in Mapper, example below cod=
e:
>
> public void map(LongWritable key, Text value,
>            OutputCollector<Text, IntWritable> output, Reporter reporter)
>            throws IOException {
>
>        try {
>
>            reporter.incrCounter("row", "num", 1);
>
>        } catch (Throwable e) {
>            e.printStackTrace();
>            throw new RuntimeException(e);
>        }
>    }
>
> Could everyone tell me whether there are any risks do so=A3=BF
>
>
> Thanks=A3=AC
>
> LiuLei
>

--0016e65a0eea7f827a049a3eca52--

From general-return-2856-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 20 06:52:40 2011
Return-Path: <general-return-2856-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 36782 invoked from network); 20 Jan 2011 06:52:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jan 2011 06:52:40 -0000
Received: (qmail 19831 invoked by uid 500); 20 Jan 2011 06:52:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19581 invoked by uid 500); 20 Jan 2011 06:52:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19573 invoked by uid 99); 20 Jan 2011 06:52:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Jan 2011 06:52:34 +0000
X-ASF-Spam-Status: No, hits=3.7 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of liulei412@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Jan 2011 06:52:28 +0000
Received: by fxm2 with SMTP id 2so257993fxm.35
        for <general@hadoop.apache.org>; Wed, 19 Jan 2011 22:52:07 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=IvMNy3sEV5pe+sGidoo5Gi9ZO5rRh8X0b/FYpyzkZVs=;
        b=QKxpEBT4TBkN5HqdzpTeINxJOpoyC5lJK/II9w+10Qepg+qVXLvcPWOl48UBCHqMqQ
         0U97p99I9ePrTxJWJN1xGOslAAH1bM2fiNq0iiMRgMo7XdEzwOpfz0v9j5cDsLIm7nk1
         xGTZCz9T67B9a0TsRGQVKMsEogO5WYYIyZr14=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=KoUGd6+s7J5I38QIgQaMtnGmx6Q+79lWobf8v3zBF3AOMG9mZfjQWyL281YJ6bQfDj
         AJC9ifdF5Q1/SORKO39XRXBs5PgBTXn5M7rwf8OKtRNdFV4faaFB0soBlWSt6IgAcArH
         a25gO4baF846RFzcYLYKOevpPMCn6ITILKMAA=
MIME-Version: 1.0
Received: by 10.103.246.1 with SMTP id y1mr961013mur.69.1295506326969; Wed, 19
 Jan 2011 22:52:06 -0800 (PST)
Received: by 10.102.48.15 with HTTP; Wed, 19 Jan 2011 22:52:06 -0800 (PST)
Date: Thu, 20 Jan 2011 14:52:06 +0800
Message-ID: <AANLkTim8jcfP2YFtjEg51aP1KV=EMoxuvoRU3FDq0PV+@mail.gmail.com>
Subject: can one map instance handle many data of input paths at the same time
From: lei liu <liulei412@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016367ed58f6c5df8049a419213

--0016367ed58f6c5df8049a419213
Content-Type: text/plain; charset=ISO-8859-1

There are two input paths, example: /user/test1/ and /user/test2/ path.  Can
one map instance handle many data of input paths at the same time?


Thanks,

LiuLei

--0016367ed58f6c5df8049a419213--

From general-return-2857-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 20 13:19:51 2011
Return-Path: <general-return-2857-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81908 invoked from network); 20 Jan 2011 13:19:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jan 2011 13:19:51 -0000
Received: (qmail 19908 invoked by uid 500); 20 Jan 2011 13:19:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19578 invoked by uid 500); 20 Jan 2011 13:19:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19570 invoked by uid 99); 20 Jan 2011 13:19:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Jan 2011 13:19:45 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of binhara@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Jan 2011 13:19:39 +0000
Received: by wwd20 with SMTP id 20so584198wwd.29
        for <general@hadoop.apache.org>; Thu, 20 Jan 2011 05:19:18 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=xpVmxUP98wKnzTWlDslr7Xod9LPlGT1YU1Aqj+/R0rk=;
        b=Zfwj+6B51NbkdFkGv27xJgZSKT4tD0g4x11EBTOEWRKg49UwY4P5fj0huVM1ZX+ApV
         bGWhukV8pcgO+GMAxxXPK+I0PZ8VAg1YzRFzghMb3dwoSJbr7V7FJQvM9069f9oQK15R
         6ovcjH0kzRu8t/SH2ZTdObbjSouUADBGYnn3E=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=IFevlY76+dt1HmX+BkBkNUVXStcqq8Mg9QLBhuqSiBSzDLFhGWGJBt4R9aaZ/H1Jjf
         ZFJzx2GGFjNCL5YZr/GiFpBrbbUyvfV9DzaMPRtBFsKlJC2sRJpDjj0+g1CUqI634QDA
         RDz9dIgZfOjrkj2dGX10LCHQBUVRQY94tUuhI=
MIME-Version: 1.0
Received: by 10.216.165.85 with SMTP id d63mr1925094wel.12.1295529556319; Thu,
 20 Jan 2011 05:19:16 -0800 (PST)
Received: by 10.216.17.146 with HTTP; Thu, 20 Jan 2011 05:19:16 -0800 (PST)
Date: Thu, 20 Jan 2011 11:19:16 -0200
Message-ID: <AANLkTikgGewUjmsRse0i_dG=TUa3P54ukbfkSifC3rav@mail.gmail.com>
Subject: HadoopDFSFileReadWrite.java ???
From: Alessandro Binhara <binhara@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6469722002021049a46fb57

--0016e6469722002021049a46fb57
Content-Type: text/plain; charset=ISO-8859-1

Helllo ..

I try access hadoopDFS exemple in follow link :

http://wiki.apache.org/hadoop/HadoopDfsReadWriteExample?action=AttachFile&do=view&target=HadoopDFSFileReadWrite.java

 I got a error:
You are not allowed to do AttachFile on this page.

 I am login at wiki ..

thanks

--0016e6469722002021049a46fb57--

From general-return-2858-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 20 15:35:45 2011
Return-Path: <general-return-2858-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93157 invoked from network); 20 Jan 2011 15:35:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jan 2011 15:35:45 -0000
Received: (qmail 70914 invoked by uid 500); 20 Jan 2011 15:35:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70633 invoked by uid 500); 20 Jan 2011 15:35:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70625 invoked by uid 99); 20 Jan 2011 15:35:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Jan 2011 15:35:39 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of msegel@navteq.com designates 204.120.70.37 as permitted sender)
Received: from [204.120.70.37] (HELO xmailfargo.navteq.com) (204.120.70.37)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Jan 2011 15:35:32 +0000
Received: from fargo-spamfw.navteq.com (navteq-spamfw.navteq.com [10.6.10.131])
	by xmailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id p0KF644o008876
	for <general@hadoop.apache.org>; Thu, 20 Jan 2011 09:06:04 -0600
Received: from imailfargo.navteq.com (localhost [127.0.0.1])
	by fargo-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id 59E6C488A72E
	for <general@hadoop.apache.org>; Thu, 20 Jan 2011 09:35:11 -0600 (CST)
Received: from imailfargo.navteq.com (imailfargo.navteq.com [10.6.10.130]) by fargo-spamfw.navteq.com with ESMTP id Bj92C3vHHGo6u12G for <general@hadoop.apache.org>; Thu, 20 Jan 2011 09:35:11 -0600 (CST)
Received: from hq-ex-ht02.ad.navteq.com ([10.8.222.172])
	by imailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id p0KFZBvm016609
	for <general@hadoop.apache.org>; Thu, 20 Jan 2011 09:35:11 -0600
Received: from hq-ex-mb02.ad.navteq.com ([fe80::880d:318e:e860:a8db]) by
 hq-ex-ht02.ad.navteq.com ([fe80::24f6:ca50:734d:df6e%14]) with mapi; Thu, 20
 Jan 2011 09:35:03 -0600
From: "Segel, Mike" <msegel@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 20 Jan 2011 09:35:09 -0600
Subject: RE: can one map instance handle many data of input paths at the
 same time
Thread-Topic: can one map instance handle many data of input paths at the
 same time
Thread-Index: Acu4bqJ842BI5LKBTomJObN5D4dJvQASLMCA
Message-ID: <ABC24175AFD3BE4DA15F4CD375ED413D063B1BE586@hq-ex-mb02.ad.navteq.com>
References: <AANLkTim8jcfP2YFtjEg51aP1KV=EMoxuvoRU3FDq0PV+@mail.gmail.com>
In-Reply-To: <AANLkTim8jcfP2YFtjEg51aP1KV=EMoxuvoRU3FDq0PV+@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1

First a caveat... I'm going from memory and I haven't had my first cup of=
 coffee yet... :-)

The answer is it depends.
If your input is a directory, then I believe that you will pick up all of=
 the files in the directory.
If you are trying to specify just two files... I don't believe so.

(Now watch. I'm wrong and I do need to get more sleep.)

HTH

-Mike


-----Original Message-----
From: lei liu [mailto:liulei412@gmail.com]=20
Sent: Thursday, January 20, 2011 12:52 AM
To: general@hadoop.apache.org
Subject: can one map instance handle many data of input paths at the same=
 time

There are two input paths, example: /user/test1/ and /user/test2/ path.  =
Can
one map instance handle many data of input paths at the same time?


Thanks,

LiuLei


The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.

From general-return-2859-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 20 15:59:40 2011
Return-Path: <general-return-2859-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73356 invoked from network); 20 Jan 2011 15:59:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jan 2011 15:59:40 -0000
Received: (qmail 16627 invoked by uid 500); 20 Jan 2011 15:59:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16440 invoked by uid 500); 20 Jan 2011 15:59:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16405 invoked by uid 99); 20 Jan 2011 15:59:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Jan 2011 15:59:35 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of binhara@gmail.com designates 209.85.161.176 as permitted sender)
Received: from [209.85.161.176] (HELO mail-gx0-f176.google.com) (209.85.161.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Jan 2011 15:59:28 +0000
Received: by gxk4 with SMTP id 4so151697gxk.35
        for <general@hadoop.apache.org>; Thu, 20 Jan 2011 07:59:07 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=MNIFsyVv8Uo3Ce6KAHl/cWmTU4QItUR/Hqihfk6TiYg=;
        b=aOynoNIPxCwPlnUzkTXwXJhX3CFrssCxxSBH2anZzp9JUNs2PV1ph7r0dbULBtd9+w
         6HFF2cG1YjxPui4vzghmwzAYrEZQR7JIg8h3HeCH3g38I7Kn+sq9AqdHu9AxI8YEEr6e
         G8w3+ecvUwYibjL0TRLS/YwWqqUU7TrFbIh1g=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=ufbOSbZB1K/s6lqc8g5KmbMOF1mkSz5AqMG+S72bRC/YKosbGMO0vr9qsYUaErCQvr
         7l8bFSGkAONdPus19OTRVbbyjzqb0H3CH42JdR7z4tq2ds0VbhBkXRDmFmHNYLyRTOHc
         HeX5zt6lcWjFY7ZSm/CwlOA6X08P1PE+K/T6U=
MIME-Version: 1.0
Received: by 10.150.214.12 with SMTP id m12mr2557028ybg.371.1295539147648;
 Thu, 20 Jan 2011 07:59:07 -0800 (PST)
Received: by 10.151.9.9 with HTTP; Thu, 20 Jan 2011 07:59:07 -0800 (PST)
Date: Thu, 20 Jan 2011 13:59:07 -0200
Message-ID: <AANLkTiks7Hz-cjeEq9Kvh44EkwP+vrkVR=UfgPhE1w-U@mail.gmail.com>
Subject: How to write on HDFS by WEbServer
From: Alessandro Binhara <binhara@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd35714b030dc049a49363c

--000e0cd35714b030dc049a49363c
Content-Type: text/plain; charset=ISO-8859-1

Hello..

I start a first implementation in hadoop.
I see some example how to write on hadoop..

i have a question ...
I have  java code to  write on HDFS ...

i run it calling :
hadoop jar MyJarToWriteOnHdfs

it will run my program and write on HDFS ..ok

I need make this concept :
WebServer ---> MyJarToWriteOnHdfs  ---> HDFS

My question is :
1) I need put a  embedded Jetty in my MyJarToWriteOnHdfs  run it like a job
on Hadoop ?
or
2) I create a webserver , put it on TOMCAT my webserver write a file in
hadoop ?

thanks

--000e0cd35714b030dc049a49363c--

From general-return-2860-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 20 16:19:37 2011
Return-Path: <general-return-2860-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9585 invoked from network); 20 Jan 2011 16:19:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jan 2011 16:19:36 -0000
Received: (qmail 58129 invoked by uid 500); 20 Jan 2011 16:19:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57662 invoked by uid 500); 20 Jan 2011 16:19:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57654 invoked by uid 99); 20 Jan 2011 16:19:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Jan 2011 16:19:30 +0000
X-ASF-Spam-Status: No, hits=2.3 required=10.0
	tests=FROM_EXCESS_BASE64,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: error (athena.apache.org: local policy)
Received: from [159.226.40.9] (HELO software.ict.ac.cn) (159.226.40.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 20 Jan 2011 16:19:25 +0000
Received: (qmail 17724 invoked from network); 20 Jan 2011 16:18:29 -0000
Received: from unknown (HELO xieys) (210.76.197.239)
  by software.ict.ac.cn with SMTP; 20 Jan 2011 16:18:29 -0000
Date: Fri, 21 Jan 2011 00:17:41 +0800
From: "=?utf-8?B?eGlleWVzaGFu?=" <xieyeshan@software.ict.ac.cn>
To: "=?utf-8?B?Z2VuZXJhbA==?=" <general@hadoop.apache.org>,
 "=?utf-8?B?bWFwcmVkdWNlLXVzZXJAaGFkb29wLmFwYWNoZS5vcmc=?=" <mapreduce-user@hadoop.apache.org>
References: <C95B2E6B.62944%knoguchi@yahoo-inc.com>
Subject: =?utf-8?B?UmU6IFJlOiBIb3cgdG8gdXNlIGNvbWJpbmVyIGluIGhhZG9vcCBzdHJlYW1pbmc=?=
Message-ID: <201101210017085620407@software.ict.ac.cn>
X-mailer: Foxmail 6, 15, 201, 23 [cn]
Mime-Version: 1.0
Content-Type: multipart/alternative;
	boundary="=====003_Dragon204443083360_====="

--=====003_Dragon204443083360_=====
Content-Type: text/plain;
	charset="utf-8"
Content-Transfer-Encoding: base64

DQpIYWRvb3AgMC4yMS4wJ3MgY29tYmluZXIgY291bGQgYmUgY21kIGFuZCBpdCBqdXN0IHNvbHZl
cyBteSBwcm9ibGVtLlRoYW5rIGFsbCBvZiB5b3UuVGhhbmtzXl9eIA0KDQoNCjIwMTEtMDEtMjEg
DQoNCg0KDQp4aWV5ZXNoYW4gDQoNCg0KDQrlj5Hku7bkurrvvJogS29qaSBOb2d1Y2hpIA0K5Y+R
6YCB5pe26Ze077yaIDIwMTEtMDEtMTkgIDAzOjQ4OjMxIA0K5pS25Lu25Lq677yaIG1hcHJlZHVj
ZS11c2VyQGhhZG9vcC5hcGFjaGUub3JnIA0K5oqE6YCB77yaIA0K5Li76aKY77yaIFJlOiBIb3cg
dG8gdXNlIGNvbWJpbmVyIGluIGhhZG9vcCBzdHJlYW1pbmcgDQogDQooQmNjIGdlbmVyYWxALiBU
aGlzIGlzIGZvciBIYWRvb3AgcHJvamVjdCBsZXZlbCBhbm5vdW5jZW1lbnRzIGFuZA0KZGlzY3Vz
c2lvbnMuIGh0dHA6Ly9oYWRvb3AuYXBhY2hlLm9yZy9tYWlsaW5nX2xpc3RzLmh0bWwgLiBVc2lu
Zw0KbWFwcmVkdWNlLXVzZXJAKQ0KWWVzaGFuLCANCldoaWNoIHZlcnNpb24gb2YgaGFkb29wIGFy
ZSB5b3UgdXNpbmc/DQpPZmZpY2lhbGx5LCBjb21iaW5lciBmb3Igbm9uLWphdmEgY29tbWFuZCB3
YXMgYWRkZWQgaW4gMC4yMQ0KaHR0cHM6Ly9pc3N1ZXMuYXBhY2hlLm9yZy9qaXJhL2Jyb3dzZS9I
QURPT1AtNDg0Mg0KYnV0IEnCuWQgYmV0IHNvbWUgb2YgdGhvc2UgMC4yMHMgaGF2ZSB0aGlzIGJh
Y2twb3J0ZWQuDQpLb2ppDQpPbiAxLzE4LzExIDEyOjA2IEFNLCAiSGFyc2ggSiIgPHF3ZXJ0eW1h
bmlhY0BnbWFpbC5jb20+IHdyb3RlOg0KPiBSaWdodCBub3csIHRoZSBDb21iaW5lciBoYXMgdG8g
YmUgYSBKYXZhIENsYXNzLiBBIGNvbW1hbmQgaXMNCj4gdW5hY2NlcHRhYmxlLCBhcyBub3RlZCBo
ZXJlOg0KPiBodHRwOi8vd2lraS5hcGFjaGUub3JnL2hhZG9vcC9IYWRvb3BTdHJlYW1pbmcNCj4g
DQo+IE9uIFR1ZSwgSmFuIDE4LCAyMDExIGF0IDE6MzAgUE0sIHhpZXllc2hhbiA8eGlleWVzaGFu
QHNvZnR3YXJlLmljdC5hYy5jbj4NCj4gd3JvdGU6DQo+PiBoaSxhbGwNCj4+IA0KPj4gICAgIEkg
d2FudCB0byB1c2UgY29tYmluZXIgaW4gaGFkb29wIHN0cmVhbWluZywgYnV0IGNvbWJpbmVyIGlz
IG5vdCB0aGUgc2FtZQ0KPj4gYXMgbWFwcGVyIG9yIHJlZHVjZXIgaW4gc3RyZWFtaW5nLkhvdyBj
b3VsZCBJIGRvIHRoaXM/DQo+PiAgICAgVGhhbmtzIQ0KPj4gDQo+PiANCj4+IFlvdXJzLA0KPj4g
DQo+PiBZZXNoYW4gWGllDQo+PiANCj4+IDIwMTEtMDEtMTgNCj4+IA0KPj4gDQo+PiANCj4+IHhp
ZXllc2hhbg0KPj4gDQo+IA0KPiANCj4gDQo+IC0tDQo+IEhhcnNoIEoNCj4gd3d3LmhhcnNoai5j
b20NCj4gDQo=

--=====003_Dragon204443083360_=====--


From general-return-2861-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 20 17:09:25 2011
Return-Path: <general-return-2861-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33859 invoked from network); 20 Jan 2011 17:09:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jan 2011 17:09:25 -0000
Received: (qmail 50386 invoked by uid 500); 20 Jan 2011 17:09:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50105 invoked by uid 500); 20 Jan 2011 17:09:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49972 invoked by uid 99); 20 Jan 2011 17:09:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Jan 2011 17:09:20 +0000
X-ASF-Spam-Status: No, hits=1.3 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_BL_SPAMCOP_NET,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Jan 2011 17:09:13 +0000
Received: by fxm2 with SMTP id 2so836794fxm.35
        for <general@hadoop.apache.org>; Thu, 20 Jan 2011 09:08:53 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type:content-transfer-encoding;
        bh=IHAqa3u23qoOTzFzau9YGhTFfCUEAGEgaN7/SKz7OOQ=;
        b=ayvc9biKk/iY4OzgHvw9CN1gSHutFFN9XxiLCEs7wbmHLn4+PQk1AyMa4S/F/i1O8O
         fkJi4fcIYcRY2Nb2jyhzmLeBKraO7Ybu5IMISUudkAJpLSUdeZh1ZTL7NhlA1obsFJSL
         MnIoHs61Pp3erILw8QaSbTyKpbjrkDT9JaxGM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=ko0YR6Fq3iGsYIY9ZhXhLx2Is9ZESW5aP0GTLJHRHNWuxwmPqfQ9iu42moReKpaqkP
         3RMHuwolteb8WMwWQmvLudFB3o5MHR/Ra/TOSoiUDA2Ys5An+n4Vb+tBYLMw9LeQQ96M
         NdaTGFmcir8gLfmZ2ubflObTDsGUc5bAlY/vk=
Received: by 10.223.116.1 with SMTP id k1mr2328967faq.51.1295543333538; Thu,
 20 Jan 2011 09:08:53 -0800 (PST)
MIME-Version: 1.0
Received: by 10.223.124.200 with HTTP; Thu, 20 Jan 2011 09:08:33 -0800 (PST)
In-Reply-To: <AANLkTiks7Hz-cjeEq9Kvh44EkwP+vrkVR=UfgPhE1w-U@mail.gmail.com>
References: <AANLkTiks7Hz-cjeEq9Kvh44EkwP+vrkVR=UfgPhE1w-U@mail.gmail.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Thu, 20 Jan 2011 22:38:33 +0530
Message-ID: <AANLkTikXJZ5ihrXgejZfzvNO45xSN+kt9RsesUVdzHv6@mail.gmail.com>
Subject: Re: How to write on HDFS by WEbServer
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Well if you are trying only to write files to Hadoop's Distributed
File System, you can simply get that done with (2).

If you are planning to run MapReduce jobs (a good bunch of them), then
you would need something more specific/elaborate to suit your exact
needs (Either managing a set of jars, or configuring jobs live from
the webapp and submitting, etc.).

On Thu, Jan 20, 2011 at 9:29 PM, Alessandro Binhara <binhara@gmail.com> wro=
te:
> Hello..
>
> I start a first implementation in hadoop.
> I see some example how to write on hadoop..
>
> i have a question ...
> I have =A0java code to =A0write on HDFS ...
>
> i run it calling :
> hadoop jar MyJarToWriteOnHdfs
>
> it will run my program and write on HDFS ..ok
>
> I need make this concept :
> WebServer ---> MyJarToWriteOnHdfs =A0---> HDFS
>
> My question is :
> 1) I need put a =A0embedded Jetty in my MyJarToWriteOnHdfs =A0run it like=
 a job
> on Hadoop ?
> or
> 2) I create a webserver , put it on TOMCAT my webserver write a file in
> hadoop ?
>
> thanks
>



--=20
Harsh J
www.harshj.com

From general-return-2862-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 20 17:36:45 2011
Return-Path: <general-return-2862-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44029 invoked from network); 20 Jan 2011 17:36:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jan 2011 17:36:45 -0000
Received: (qmail 42970 invoked by uid 500); 20 Jan 2011 17:36:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42209 invoked by uid 500); 20 Jan 2011 17:36:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42196 invoked by uid 99); 20 Jan 2011 17:36:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Jan 2011 17:36:39 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of binhara@gmail.com designates 74.125.83.48 as permitted sender)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Jan 2011 17:36:34 +0000
Received: by gwj15 with SMTP id 15so270621gwj.35
        for <general@hadoop.apache.org>; Thu, 20 Jan 2011 09:36:14 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=+ADQVwPiq9FxYhBEYkfU/wNnzOJCNLPY6Fyo3JwPpZc=;
        b=LMQlEZMbo7CQBCGvkprikPrFm8YO6ZzROsV7Owimp/k8XUd7ddhANxAkYzSG+i5WlS
         z+IBOK5rJKODVg0tKXJK+9x78dF0gRCjjnGrp86+uuQdeB22R1e5Fftjij07BLref5OI
         a+lIZaWxECHLGLupeKSQF6poWNej4YyrTsBk8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=adIdAuyt0zmJbZz92Rqed6BSKDt20J36VTeVKHT/N7FUFx85oaA2tBQx9sMROFGBdm
         MIeDhGj1KpcL5D6jGoro2BCG8alSpJjYyfHQcPgecnt1VxE5s9fYUeB49cdVaQKK1a3p
         XDK+19YPZaOFAqhBi3OFa6pwvlrZlezKWrj6M=
MIME-Version: 1.0
Received: by 10.150.225.16 with SMTP id x16mr2843256ybg.86.1295544973925; Thu,
 20 Jan 2011 09:36:13 -0800 (PST)
Received: by 10.151.9.9 with HTTP; Thu, 20 Jan 2011 09:36:13 -0800 (PST)
In-Reply-To: <AANLkTikXJZ5ihrXgejZfzvNO45xSN+kt9RsesUVdzHv6@mail.gmail.com>
References: <AANLkTiks7Hz-cjeEq9Kvh44EkwP+vrkVR=UfgPhE1w-U@mail.gmail.com>
	<AANLkTikXJZ5ihrXgejZfzvNO45xSN+kt9RsesUVdzHv6@mail.gmail.com>
Date: Thu, 20 Jan 2011 15:36:13 -0200
Message-ID: <AANLkTikR9Jq8271H_yGR49yRSd2Ggp0CT=2o3cq6xft9@mail.gmail.com>
Subject: Re: How to write on HDFS by WEbServer
From: Alessandro Binhara <binhara@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd47e62f61d01049a4a915f

--000e0cd47e62f61d01049a4a915f
Content-Type: text/plain; charset=ISO-8859-1

Well in second case :
Tomcat --> webserver --> HDFS ..

my example run here from the shell
hadoop jar HahoopHdfsHello.jar HadooHdfsHello

thats ok..
i try run directly in java line.

i think to write on hdfs , it need a enviroment of hadoop.

In a java aplication or webserver it will work ?
i try run my java example.. and a i have a problema with jar

root@master:~# java -jar HahoopHdfsHello.jar
Failed to load Main-Class manifest attribute from
HahoopHdfsHello.jar

thanks for answer ..


On Thu, Jan 20, 2011 at 3:08 PM, Harsh J <qwertymaniac@gmail.com> wrote:

> Well if you are trying only to write files to Hadoop's Distributed
> File System, you can simply get that done with (2).
>
> If you are planning to run MapReduce jobs (a good bunch of them), then
> you would need something more specific/elaborate to suit your exact
> needs (Either managing a set of jars, or configuring jobs live from
> the webapp and submitting, etc.).
>
> On Thu, Jan 20, 2011 at 9:29 PM, Alessandro Binhara <binhara@gmail.com>
> wrote:
> > Hello..
> >
> > I start a first implementation in hadoop.
> > I see some example how to write on hadoop..
> >
> > i have a question ...
> > I have  java code to  write on HDFS ...
> >
> > i run it calling :
> > hadoop jar MyJarToWriteOnHdfs
> >
> > it will run my program and write on HDFS ..ok
> >
> > I need make this concept :
> > WebServer ---> MyJarToWriteOnHdfs  ---> HDFS
> >
> > My question is :
> > 1) I need put a  embedded Jetty in my MyJarToWriteOnHdfs  run it like a
> job
> > on Hadoop ?
> > or
> > 2) I create a webserver , put it on TOMCAT my webserver write a file in
> > hadoop ?
> >
> > thanks
> >
>
>
>
> --
> Harsh J
> www.harshj.com
>

--000e0cd47e62f61d01049a4a915f--

From general-return-2863-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 20 18:21:57 2011
Return-Path: <general-return-2863-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68878 invoked from network); 20 Jan 2011 18:21:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jan 2011 18:21:57 -0000
Received: (qmail 42577 invoked by uid 500); 20 Jan 2011 18:21:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42174 invoked by uid 500); 20 Jan 2011 18:21:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42166 invoked by uid 99); 20 Jan 2011 18:21:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Jan 2011 18:21:52 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Jan 2011 18:21:47 +0000
Received: by gyf1 with SMTP id 1so298935gyf.35
        for <general@hadoop.apache.org>; Thu, 20 Jan 2011 10:21:26 -0800 (PST)
MIME-Version: 1.0
Received: by 10.42.230.9 with SMTP id jk9mr2969069icb.519.1295547686009; Thu,
 20 Jan 2011 10:21:26 -0800 (PST)
Received: by 10.42.168.198 with HTTP; Thu, 20 Jan 2011 10:21:25 -0800 (PST)
In-Reply-To: <AANLkTim8jcfP2YFtjEg51aP1KV=EMoxuvoRU3FDq0PV+@mail.gmail.com>
References: <AANLkTim8jcfP2YFtjEg51aP1KV=EMoxuvoRU3FDq0PV+@mail.gmail.com>
Date: Thu, 20 Jan 2011 13:21:25 -0500
Message-ID: <AANLkTimiH20ywBfBqLEJ5jgYU3ss8ttUmpa+AJ-ORnL3@mail.gmail.com>
Subject: Re: can one map instance handle many data of input paths at the same time
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf3054a0739d3de7049a4b337e

--20cf3054a0739d3de7049a4b337e
Content-Type: text/plain; charset=ISO-8859-1

LiuLei:

Yes. What you're looking for is TextInputFormat.addPath() (assuming you're
talking about text). You can call this multiple times and add multiple input
paths if they are all of the same data format (i.e. text). If you have
multiple paths that contain different format data, you'll need to use
MultipleInputs. See the javadoc for details on usage.

On Thu, Jan 20, 2011 at 1:52 AM, lei liu <liulei412@gmail.com> wrote:

> There are two input paths, example: /user/test1/ and /user/test2/ path.
>  Can
> one map instance handle many data of input paths at the same time?
>
>
> Thanks,
>
> LiuLei
>



-- 
Eric Sammer
twitter: esammer
data: www.cloudera.com

--20cf3054a0739d3de7049a4b337e--

From general-return-2864-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 20 18:33:39 2011
Return-Path: <general-return-2864-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91636 invoked from network); 20 Jan 2011 18:33:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jan 2011 18:33:39 -0000
Received: (qmail 68556 invoked by uid 500); 20 Jan 2011 18:33:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67730 invoked by uid 500); 20 Jan 2011 18:33:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67716 invoked by uid 99); 20 Jan 2011 18:33:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Jan 2011 18:33:34 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.59.211] (HELO QMTA11.westchester.pa.mail.comcast.net) (76.96.59.211)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Jan 2011 18:33:27 +0000
Received: from omta16.westchester.pa.mail.comcast.net ([76.96.62.88])
	by QMTA11.westchester.pa.mail.comcast.net with comcast
	id xuEF1f0051uE5Es5BuZ719; Thu, 20 Jan 2011 18:33:07 +0000
Received: from [10.72.184.42] ([209.131.62.113])
	by omta16.westchester.pa.mail.comcast.net with comcast
	id xuYw1f0012SbwD53cuYyji; Thu, 20 Jan 2011 18:33:05 +0000
Message-Id: <AD37F82F-96D1-4E3F-98EB-7AFAE4470B73@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: mapreduce-user@hadoop.apache.org
In-Reply-To: <AANLkTimiH20ywBfBqLEJ5jgYU3ss8ttUmpa+AJ-ORnL3@mail.gmail.com>
Content-Type: multipart/alternative; boundary=Apple-Mail-39--586405111
Subject: Re: can one map instance handle many data of input paths at the same time
Mime-Version: 1.0 (Apple Message framework v936)
Date: Thu, 20 Jan 2011 10:32:54 -0800
References: <AANLkTim8jcfP2YFtjEg51aP1KV=EMoxuvoRU3FDq0PV+@mail.gmail.com> <AANLkTimiH20ywBfBqLEJ5jgYU3ss8ttUmpa+AJ-ORnL3@mail.gmail.com>
X-Mailer: Apple Mail (2.936)

--Apple-Mail-39--586405111
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit

As defined in http://hadoop.apache.org/mailing_lists.html , please  
send user questions to
mapreduce-user@hadoop.apache.org.

-- Owen

On Jan 20, 2011, at 10:21 AM, Eric Sammer wrote:

> LiuLei:
>
> Yes. What you're looking for is TextInputFormat.addPath() (assuming  
> you're
> talking about text). You can call this multiple times and add  
> multiple input
> paths if they are all of the same data format (i.e. text). If you have
> multiple paths that contain different format data, you'll need to use
> MultipleInputs. See the javadoc for details on usage.
>
> On Thu, Jan 20, 2011 at 1:52 AM, lei liu <liulei412@gmail.com> wrote:
>
>> There are two input paths, example: /user/test1/ and /user/test2/  
>> path.
>> Can
>> one map instance handle many data of input paths at the same time?
>>
>>
>> Thanks,
>>
>> LiuLei
>>
>
>
>
> -- 
> Eric Sammer
> twitter: esammer
> data: www.cloudera.com


--Apple-Mail-39--586405111--

From general-return-2865-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 20 21:03:16 2011
Return-Path: <general-return-2865-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 92827 invoked from network); 20 Jan 2011 21:03:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jan 2011 21:03:16 -0000
Received: (qmail 85732 invoked by uid 500); 20 Jan 2011 21:03:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85675 invoked by uid 500); 20 Jan 2011 21:03:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85667 invoked by uid 99); 20 Jan 2011 21:03:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Jan 2011 21:03:13 +0000
X-ASF-Spam-Status: No, hits=3.7 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of akimball83@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Jan 2011 21:03:08 +0000
Received: by bwz8 with SMTP id 8so951925bwz.35
        for <general@hadoop.apache.org>; Thu, 20 Jan 2011 13:02:47 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:from:date:message-id:subject:to
         :content-type;
        bh=i9zH9SaP+2h++V55Ej+RVaq22drwXEyGZf2eq6Qr83o=;
        b=BqERhZCC4yy/WxqaKkxxyKyO0DivdWA6coMlanin2uNLeNhWn5gcxABm6ha6vl15al
         4tKoCN8vvnUmC+HsDZe6GNjbqtpAjb18l6Gbnpyv9Fs5al5rc5XyYAKJWBNQ+CKpq4pi
         zQbMr2e7YYGkM4Do49bQ5reaAm/WNeB3pz03A=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type;
        b=H76RWLMb+zrKwpr7Mzfss0nigJOCQ3jNjGvOcMLG2V/uCB6avcn5JHk5yAythl4kSe
         zEmt//0rlojm/7+F0KuGKydblHRwvfe7QIqCKjYj+0wNNpYw5UpbxghBoPGZqOyGEwuw
         76HVVuGo5dfybQCDSGJwvdkxNlAn77rNEkC5Y=
Received: by 10.204.62.73 with SMTP id w9mr2351075bkh.11.1295557260809; Thu,
 20 Jan 2011 13:01:00 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.61.138 with HTTP; Thu, 20 Jan 2011 12:54:48 -0800 (PST)
From: Aaron Kimball <akimball83@gmail.com>
Date: Thu, 20 Jan 2011 12:54:48 -0800
Message-ID: <AANLkTimsVF_81MgQ53xyUBQahT2Zkd8mdDOtpvf3XRFr@mail.gmail.com>
Subject: February SF Hadoop Meetup -- Feb 9, 2011
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c923ad511664049a4d6e53

--001636c923ad511664049a4d6e53
Content-Type: text/plain; charset=ISO-8859-1

Hadoop fans,

After the successful first edition of the SF Hadoop meetup, we plan to keep
the momentum rolling with regular meetups. The second SF Hadoop meetup will
be held Wednesday, February 9, 2011, from 6pm to 8pm.

CBS Interactive has offered to host the meetup at their office at 235 2nd
St, San Francisco. This space is larger than the one used in January; we
hope this will better accommodate the number of people who'd like to attend.

Like the previous meetup, we will continue to use the discussion-based
"unconference" format. At the beginning of the meetup we will
collaboratively construct an agenda consisting of several discussion
breakout groups. All participants may propose a topic and volunteer to
facilitate a discussion. All members of the Hadoop community are welcome to
attend.

Please RSVP at http://www.meetup.com/hadoopsf/calendar/16123193/ so we know
how many people to expect. Refreshments will be provided.

Regards,
- Aaron Kimball

--001636c923ad511664049a4d6e53--

From general-return-2866-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 21 05:58:08 2011
Return-Path: <general-return-2866-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 11019 invoked from network); 21 Jan 2011 05:58:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Jan 2011 05:58:08 -0000
Received: (qmail 21436 invoked by uid 500); 21 Jan 2011 05:58:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21148 invoked by uid 500); 21 Jan 2011 05:58:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21139 invoked by uid 99); 21 Jan 2011 05:58:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Jan 2011 05:58:01 +0000
X-ASF-Spam-Status: No, hits=3.7 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of liulei412@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Jan 2011 05:57:54 +0000
Received: by fxm2 with SMTP id 2so1516533fxm.35
        for <general@hadoop.apache.org>; Thu, 20 Jan 2011 21:57:34 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=1DKPDKPZN0DbfhHxXe0yH3+hgGMbDBvYVsD4RSTq0W4=;
        b=mf76sOk1CTZXu8vldzo7jjk9l37CIi0B0OVYFJtvmQoe/hv0pKc8Ym3p8ki+ToLt3X
         1x7cPMSMc4FPGGBEMHtn0ZdrLEG1k+0/9ApUKe/XFLSzq3lx7N/+n1GKIe2XokBffXy2
         /EQEASBdlAH7jU0kFwDjAFmIGC71pcF9yraPs=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=SPHOBFnxoQcx8t6Bo/l4gGRkXFl7LYlyiXv1Tk1JjwyfM/xX/rmI6YCqaSSwlL3J8W
         oBhFDp8NlilshrHpOLWU/7UuxblWssswuyAxNFpqiKStmA9QtosvNXv+f1CQ2Rez93f9
         /Fl5s5RinYmAF25m48fThuq+flIky8htbk4Sk=
MIME-Version: 1.0
Received: by 10.103.168.5 with SMTP id v5mr12474muo.82.1295589453696; Thu, 20
 Jan 2011 21:57:33 -0800 (PST)
Received: by 10.103.228.4 with HTTP; Thu, 20 Jan 2011 21:57:33 -0800 (PST)
In-Reply-To: <AANLkTimiH20ywBfBqLEJ5jgYU3ss8ttUmpa+AJ-ORnL3@mail.gmail.com>
References: <AANLkTim8jcfP2YFtjEg51aP1KV=EMoxuvoRU3FDq0PV+@mail.gmail.com>
	<AANLkTimiH20ywBfBqLEJ5jgYU3ss8ttUmpa+AJ-ORnL3@mail.gmail.com>
Date: Fri, 21 Jan 2011 13:57:33 +0800
Message-ID: <AANLkTinLhnhRV-6faLYojhw4B=DDTLztyLY7KtFm6ruD@mail.gmail.com>
Subject: Re: can one map instance handle many data of input paths at the same time
From: lei liu <liulei412@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016369896eb298ee4049a54ed20
X-Virus-Checked: Checked by ClamAV on apache.org

--0016369896eb298ee4049a54ed20
Content-Type: text/plain; charset=GB2312
Content-Transfer-Encoding: quoted-printable

Thanks everyone=A3=AC

I detailed describe my question.  There are two input
direcoties:/user/test1/ and /user/test2/ path, I want to join the two
direcoties content, in order to join the two directories, I need to identit=
y
the content from which directory, so I use below code in mapper:

    private int tag =3D -1;
    @Override
    public void configure(JobConf conf) {
        try {

            this.conf =3D conf;
            String pathsToAliasStr =3D conf.get("paths.to.alias");//example=
:
conf.set("paths.to.alias", "0=3D/user/test1/,1=3D/user/test2/"
            String[] pathsToAlias =3D pathsToAliasStr.split(",");

            Path fpath =3D new Path((new
Path(conf.get("map.input.file"))).toUri().getPath());
            String path =3D fpath.toUri().toString();

            for (int i =3D 0; i < pathsToAlias.length; i++) {
                String[] pathToAlias =3D pathsToAlias[i].split("=3D");
                if (path.startsWith(pathToAlias[1])) {
                    tag =3D Integer.valueOf(pathToAlias[0].trim());//identi=
ty
current map instatnce are handling which directory content.
                }
            }
        } catch (Throwable e) {
            e.printStackTrace();
            throw new RuntimeException(e);
        }

    }

So when map method  run=A3=AC  the content are handled by the mapper are
identified for same direcoty.

I want to know whether one mapper instatnce only handle content of one
directory at same time.


Thanks

LiuLei






2011/1/21 Eric Sammer <esammer@cloudera.com>

> LiuLei:
>
> Yes. What you're looking for is TextInputFormat.addPath() (assuming you'r=
e
> talking about text). You can call this multiple times and add multiple
> input
> paths if they are all of the same data format (i.e. text). If you have
> multiple paths that contain different format data, you'll need to use
> MultipleInputs. See the javadoc for details on usage.
>
> On Thu, Jan 20, 2011 at 1:52 AM, lei liu <liulei412@gmail.com> wrote:
>
> > There are two input paths, example: /user/test1/ and /user/test2/ path.
> >  Can
> > one map instance handle many data of input paths at the same time?
> >
> >
> > Thanks,
> >
> > LiuLei
> >
>
>
>
> --
> Eric Sammer
> twitter: esammer
> data: www.cloudera.com
>

--0016369896eb298ee4049a54ed20--

From general-return-2867-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 21 07:48:41 2011
Return-Path: <general-return-2867-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60258 invoked from network); 21 Jan 2011 07:48:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Jan 2011 07:48:40 -0000
Received: (qmail 33998 invoked by uid 500); 21 Jan 2011 07:48:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33635 invoked by uid 500); 21 Jan 2011 07:48:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33621 invoked by uid 99); 21 Jan 2011 07:48:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Jan 2011 07:48:34 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Jan 2011 07:48:24 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0L7lT56031837;
	Thu, 20 Jan 2011 23:47:30 -0800 (PST)
Received: from [10.0.1.4] (10.73.153.37) by SP2-EX07CAS01.ds.corp.yahoo.com
 (98.137.59.4) with Microsoft SMTP Server (TLS) id 8.3.106.1; Thu, 20 Jan 2011
 23:47:29 -0800
Subject: Re: use counter to statistics file row number
MIME-Version: 1.0 (Apple Message framework v1082)
Content-Type: text/plain; charset="GB2312"
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <AANLkTi=XYK_RkNH0whQ8UU+S-+MK3eszJ48Vo-KhBmZi@mail.gmail.com>
Date: Thu, 20 Jan 2011 23:47:27 -0800
CC: "mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>
Content-Transfer-Encoding: quoted-printable
Message-ID: <12CD2F7A-2CC1-41B9-A43C-02CD52EF8DF6@yahoo-inc.com>
References: <AANLkTi=WmHZjXOik8UjzzUjGJ_iX=bKV+LSfs64UgVH-@mail.gmail.com> <AANLkTi=XYK_RkNH0whQ8UU+S-+MK3eszJ48Vo-KhBmZi@mail.gmail.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

Hi ,

mapreduce-user seems like a great place for this discussion.  general =
does not.

Thanks!
E14

On Jan 19, 2011, at 7:33 PM, venkatesh kavuluri wrote:

> (Bcc general@. This is for Hadoop project level discussions. =
Includingmapreduce
> -user@)
>=20
> Liu,
>=20
> If you want the count of number of records in your input data set, the
> map/reduce framework provides a default counter "Map input records". =
The
> only caution to follow regarding the custom counters is to not exceed =
20 per
> application as they are very expensive. Generally counters are used to =
track
> few important pieces of information.
>=20
> Thanks,
> Venkatesh Kavuluri
>=20
> 2011/1/19 lei liu <liulei412@gmail.com>
>=20
>> I use counter to statistics  file row number in Mapper, example below =
code:
>>=20
>> public void map(LongWritable key, Text value,
>>           OutputCollector<Text, IntWritable> output, Reporter =
reporter)
>>           throws IOException {
>>=20
>>       try {
>>=20
>>           reporter.incrCounter("row", "num", 1);
>>=20
>>       } catch (Throwable e) {
>>           e.printStackTrace();
>>           throw new RuntimeException(e);
>>       }
>>   }
>>=20
>> Could everyone tell me whether there are any risks do so=A3=BF
>>=20
>>=20
>> Thanks=A3=AC
>>=20
>> LiuLei
>>=20


From general-return-2868-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 21 17:31:26 2011
Return-Path: <general-return-2868-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85311 invoked from network); 21 Jan 2011 17:31:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Jan 2011 17:31:26 -0000
Received: (qmail 39316 invoked by uid 500); 21 Jan 2011 17:31:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39052 invoked by uid 500); 21 Jan 2011 17:31:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39037 invoked by uid 99); 21 Jan 2011 17:31:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Jan 2011 17:31:22 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [64.78.17.18] (HELO EXHUB018-3.exch018.msoutlookonline.net) (64.78.17.18)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Jan 2011 17:31:14 +0000
Received: from EXVMBX018-1.exch018.msoutlookonline.net ([64.78.17.47]) by
 EXHUB018-3.exch018.msoutlookonline.net ([64.78.17.18]) with mapi; Fri, 21 Jan
 2011 09:30:53 -0800
From: Scott Carey <scott@richrelevance.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 21 Jan 2011 09:33:03 -0800
Subject: Re: [DISCUSS] Move project split down a level
Thread-Topic: [DISCUSS] Move project split down a level
Thread-Index: Acu5kPR2iOv28KACSOGkb1qd5xdmBA==
Message-ID: <C95F025C.1B82C%scott@richrelevance.com>
In-Reply-To: <AANLkTiknWzZYo0yc3X8sfuzScDF_yREwSaBi9uXsrJi5@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-MacOutlook/14.2.0.101115
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0



On 1/14/11 11:01 AM, "Konstantin Shvachko" <shv.hadoop@gmail.com> wrote:

>We actually still haven't recovered from the projects split.
>We are still fixing HDFS and MR scripts with several jiras open.
>
>If we start this re-split now again before the major release
>we risk to get into the same mess, and it will create more work
>for the community.
>
>I see Nigel's point that packaging will get easier, and developers
>will push less buttons when they commit.
>It will delay the release - this is what worries me.
>
>Thanks,
>--Konstantin

I don't see how it would change any of those scripts.  All three projects
would still be isolated with the same internal path structures, their
paths relative to each other in svn would be the only difference.

For developers, this might mean simply using 'svn switch' once and nothing
more.=20


From general-return-2869-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jan 22 00:10:42 2011
Return-Path: <general-return-2869-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 3153 invoked from network); 22 Jan 2011 00:10:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jan 2011 00:10:41 -0000
Received: (qmail 76728 invoked by uid 500); 22 Jan 2011 00:10:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76469 invoked by uid 500); 22 Jan 2011 00:10:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 86002 invoked by uid 99); 21 Jan 2011 05:22:24 -0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
From: "Mattmann, Chris A (388J)" <chris.a.mattmann@jpl.nasa.gov>
To: "dev@oodt.apache.org" <dev@oodt.apache.org>,
        "general@hadoop.apache.org"
	<general@hadoop.apache.org>,
        "common-user@hadoop.apache.org"
	<common-user@hadoop.apache.org>
CC: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org>,
        "user@oodt.apache.org" <user@oodt.apache.org>,
        "general@incubator.apache.org"
	<general@incubator.apache.org>,
        "dev@tika.apache.org" <dev@tika.apache.org>,
        "user@tika.apache.org" <user@tika.apache.org>,
        "whirr-dev@incubator.apache.org" <whirr-dev@incubator.apache.org>,
        "gora-dev@incubator.apache.org" <gora-dev@incubator.apache.org>,
        "sis-dev@incubator.apache.org" <sis-dev@incubator.apache.org>
Date: Thu, 20 Jan 2011 21:21:51 -0800
Subject: [Call for Papers] ICSE Software Engineering for Cloud Computing
 (SECLOUD) Workshop
Thread-Topic: [Call for Papers] ICSE Software Engineering for Cloud
 Computing (SECLOUD) Workshop
Thread-Index: Acu5KxxM39jWOK4WQrylk0TCV1w0Uw==
Message-ID: <DFE898A2-B230-49FD-AFFB-AE7827EBEE0D@jpl.nasa.gov>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Source-IP: altvirehtstap01.jpl.nasa.gov [128.149.137.72]
X-Source-Sender: chris.a.mattmann@jpl.nasa.gov
X-AUTH: Authorized
X-Virus-Checked: Checked by ClamAV on apache.org

(apologies for the cross posting)

*** PLEASE NOTE - the deadline for submitting papers has been extended by 1=
 week to 1/28/2011! ***

Please consider submitting a paper to the ICSE 2011 Software Engineering fo=
r Cloud Computing (SECLOUD) Workshop to be held Sunday, May 22, 2011, at th=
e Hilton Hawaiian Village Resort in Waikiki, Honolulu, HI.

This workshop focuses on identifying the grand challenges that lay before u=
s in the realm of software engineering for cloud computing. We will debate =
existing notions of SE for the construction of cloud services and for their=
 deployment. We will discuss and evangelize individual successes in the fie=
ld and attempt to identify the key ingredients that enabled that success. P=
articipants in this workshop will take a role in helping to formulate a lon=
g-term, concrete software engineering agenda for cloud and will have an opp=
ortunity to share in and contribute to the existing =93tribal knowledge=94 =
for cloud-related software engineering.

We invite high quality submissions of technical and research papers, as wel=
l as research demonstrations describing original and unpublished results of=
 research relevant to software engineering for cloud. Authors are asked to =
reserve a section in their submitted papers for identifying their suggested=
 near-term and long-term challenge areas for the field.

The workshop seeks to focus discussion around the ways that the disruptive =
effect of cloud computing is engendering a new set of principles and approa=
ches to software engineering. Specific topics of interest include, but are =
not limited to:

Topic Areas of Interest:

- Agile Software development on the cloud
- Architecting Applications using the cloud
- Benchmarking and Performance Metrics in the cloud
- Cloud Application Reliability
- Distributed collaboration on the cloud
- Interoperability and Portability challenges in multi-cloud environments
- Open Source Cloud Environments: Hadoop, Eucalyptus and other programming =
paradigms, languages and environments for the Cloud
- Requirements Elicitation effectiveness when using cloud service abstracti=
ons
- Secure Software Engineering in the Cloud
- Software Engineering Practices using popular cloud vendor services
- Software Engineering tools
- Software Project Estimations, Testing,Verification and Validation over cl=
oud service abstractions

Submission Guidelines:

Submissions may take one of two acceptable forms. Research paper contributi=
ons may be submitted with a maximum length of 7 pages, and must conform to =
ICSE 2011 paper formatting guidelines [1]. Research demonstration submissio=
ns are limited to 1 page in the conference proceedings and will be given du=
ring he two 15-minute breakout sessions.

All submissions must be in English and in PDF format [2]. An abstract must =
accompany each research paper contribution. The workshop website [3] will b=
e updated to include a link to the CyberChairPRO online submission system o=
nce it has been set up. All accepted papers will be published and dissemina=
ted on the workshop website after the conclusion of the workshop.

Review and Evaluation Criteria:

Each submission will be reviewed by at least two members of the Program Com=
mittee and will be assessed on the basis of originality, importance and rel=
evance of contribution to SE for cloud, soundness, quality of presentation,=
 and an appropriate comparison to related work. The program committee as a =
whole will make final decisions about acceptance.

Workshop Organizers and PC Chairs:

Nenad Medvidovic , University of Southern California
T.S. Mohan, Infosys Technologies
Chris A. Mattmann, NASA Jet Propulsion Laboratory
Owen O=92Malley, Yahoo, Inc.

Important Dates:

Paper Submission: 21.January.2011
Paper Notification: 21.February.2011
Camera Ready: 10.March.2011

Website: http://sites.google.com/site/icsecloud2011

Program Committee:

Andrew F. Hart, NASA JPL, United States
Ben Reed, Yahoo, United States
Craig Lee, Aerospace Corp., United States
Daniel J. Crichton, NASA JPL, United States
David C. Kale, Children's Hospital LA, United States
David M. Woollard, Project WBS, United States
Dennis Kubes, Bit.ly, United States
Eric Dashofy, Aerospace Corp., United States
Jacek Becla, Stanford National Accelerator Lab, United States
Justin Erenkrantz, Project WBS/Apache, United States
Luca Cinquini, NASA JPL, United States
Mohanakrishna, Infosys, India
Nabor Mendonca, Univ. of Fortaleza, Brazil
Sanjay Radia, Yahoo, United States
Schahram Dustdar, TU Wien, Vienna
Srinivas Padmanabhuni, Infosys, India
Stefan Tai, Karlsruhe Institute of Technology, Germany
YN Srikant, Indian Institute of Science, Bangalore, India

Links:

[1] ICSE-2011 Format: http://2011.icse-conferences.org/content-submission-g=
uidelines
[2] ACM Submission Policies: http://www.acm.org/sigs/publications/proceedin=
gs-templates
[3] Workshop Website: http://sites.google.com/site/icsecloud2011


About ICSE:=20

The International Conference on Software Engineering (ICSE) is the premier =
software engineering conference, providing a forum for researchers, practit=
ioners and educators to present and discuss the most recent innovations, tr=
ends, experiences and concerns in the field of software engineering.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++=

From general-return-2870-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jan 22 07:26:48 2011
Return-Path: <general-return-2870-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39574 invoked from network); 22 Jan 2011 07:26:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jan 2011 07:26:47 -0000
Received: (qmail 27634 invoked by uid 500); 22 Jan 2011 07:26:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27339 invoked by uid 500); 22 Jan 2011 07:26:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27327 invoked by uid 99); 22 Jan 2011 07:26:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 Jan 2011 07:26:41 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 Jan 2011 07:26:35 +0000
Received: from [192.168.1.71] (snvvpn1-10-72-244-c159.hq.corp.yahoo.com [10.72.244.159])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0M7Q0kn046708
	for <general@hadoop.apache.org>; Fri, 21 Jan 2011 23:26:00 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1295681161;
	bh=dT6xwKl7Y46GQ3iLz5d+Ot7HOpJMagEL4x40SGXDBVo=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:Mime-Version:Subject:
	 Date:References;
	b=RLGphB+wugKDTVw+knd5ezJXF6kIgqxsi3eQK3YwwieXwHWThOHEByiJCoU2pP0ba
	 GeMInyhfIW6ZwCoZ+todfZLWhZKZnhU1tAeIoL86c7wtSLoNjOBnnJGy2AGF/XDtIA
	 FNeSQhTNG1WbE5IHmuOg0TJnDtK6XNjFcpKaBMsQ=
Message-Id: <0A3C433F-DABA-4CF5-B464-4E890036EB75@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <70369193-35EB-4ECD-AB40-99F8F9424C4D@yahoo-inc.com>
Content-Type: multipart/alternative; boundary=Apple-Mail-120--453619100
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Date: Fri, 21 Jan 2011 23:26:00 -0800
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com> <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com> <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org> <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com> <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com> <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com> <AANLkTimbXjRTTmEXiY1wweFfNGKmc5aWKw9D03J7gjBg@mail.gmail.com> <388582DF-FC85-49D1-A89C-1F36CE34A0E2@yahoo-inc.com> <AANLkTim=+8_WBNbxU43xwH50WT8UQwbac_mR8FQ+0jjz@mail.gmail.com> <04705B3C-49A9-46B3-8AA9-5673EFBDE544@yahoo-inc.com> <AANLkTinAk+8YNLHb580uFibMWtNc=oYdqq+sOdBc+=6U@mail.gmail.com> <74BDFA74-DB12-4109-89DF-B353FC7296C4@yahoo-inc.com> <F5C6B0DD-DB20-4324-8A11-A2DE8E7169EB@gbiv.com> <70369193-35EB-4ECD-AB40-99F8F9424C4D@yahoo-inc.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-120--453619100
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


On Jan 18, 2011, at 1:59 AM, Arun C Murthy wrote:
>
> IAC, I agree - we've spent too much time talking and too little doing
> actual work. Let me get the job done and folks can then weigh-in on
> the release at later point, folks might be willing to consider this
> more positively once they see the  branch, the change-log etc.
>
> Of course we need to get the small number of remaining patches into
> trunk asap for 0.22 and beyond.


FYI - I've merged changes to Common's branch-0.20-security (http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security 
).

Some statistics:
# Total of 441 jiras are covered across Common (164), HDFS (90) and  
Map-Reduce (187).
# 413 jiras out of the 441 are already committed to trunk.
# 28 open jiras: Common (6), HDFS (1), Map-Reduce (21).
# Of the 28 open jiras 7 are Patch Available.
# I've ensured all commits have a jira - I had to open 3 jiras (one  
each in all sub-projects), they are included in the above stats.

Arun


--Apple-Mail-120--453619100--

From general-return-2871-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Jan 22 14:22:57 2011
Return-Path: <general-return-2871-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96799 invoked from network); 22 Jan 2011 14:22:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jan 2011 14:22:57 -0000
Received: (qmail 32606 invoked by uid 500); 22 Jan 2011 14:22:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32241 invoked by uid 500); 22 Jan 2011 14:22:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32233 invoked by uid 99); 22 Jan 2011 14:22:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 Jan 2011 14:22:50 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 Jan 2011 14:22:43 +0000
Received: by qyk7 with SMTP id 7so1402844qyk.14
        for <general@hadoop.apache.org>; Sat, 22 Jan 2011 06:22:22 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=GKd1xREhWjuOFqdEhPSXc6ORPpKxvJgRtHvS2hj97VA=;
        b=Rcc7cGEQa4sRRtUHDzqKK59uaAhGCSFNQ31sgmkjvkXGKGPp55PXw4+qP2kPIO6gdC
         5cypqQBlq2FeRwN/A9FPS8swvcHT6OovCTpLgLv8qug8u8s9nMAkixpeuyTx58LB0Ivh
         LQj5ULscIcccbHuUM752YdQoD4lBe2vkw9C94=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=b29brjDPoJHwVISpFiN+FxcXUeg0s1/RPI1Onv1+nxKMxezxj+7V0nxIk9GDST/JBV
         kY1xglfzrMej/YTRKrIyuNM55AUfOX72sp0CcuQeiOgEBC7TqL3Jlxwofg/yan+15TQO
         CrYRAK7ZLI5NyoAboUbwtPPEg5aVGWqwTdLQQ=
Received: by 10.229.99.76 with SMTP id t12mr1693254qcn.275.1295706142450;
        Sat, 22 Jan 2011 06:22:22 -0800 (PST)
Received: from [172.16.1.13] ([12.50.71.162])
        by mx.google.com with ESMTPS id y17sm7529321qci.9.2011.01.22.06.22.20
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Sat, 22 Jan 2011 06:22:21 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <AANLkTikq-bPfXviWThvh=--cWEDYPfd0iEFSE_m_RbMn@mail.gmail.com>
Date: Sat, 22 Jan 2011 09:22:18 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <B35E1DF9-ECC7-4A0D-9BFF-2F5A45CED68D@Holsman.NET>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com> <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com> <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org> <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com> <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com> <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com> <4D34A270.1060102@apache.org> <3FEB7C1D-AE89-46BE-ABF2-B595B01E222A@mac.com> <D160E684-7435-4E53-A766-231CCB38D1B4@yahoo-inc.com> <3995AF5C-AA18-475B-9384-A532F1CC5D57@linkedin.com> <2090582B-B586-4F81-A724-6A0C55820BB5@yahoo-inc.com> <704BFF6E-AC1D-4769-8856-BFBD125D4D2C@Holsman.NET> <AANLkTikq-bPfXviWThvh=--cWEDYPfd0iEFSE_m_RbMn@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)


On Jan 19, 2011, at 1:12 PM, Konstantin Shvachko wrote:

> On Tue, Jan 18, 2011 at 11:49 PM, Ian Holsman <hadoop@holsman.net> =
wrote:
>=20
>> I think Roy's suggestion of applying the commits individually to the =
branch
>> from your current working branch would help with this.
>>=20
>=20
> I am sure this is not what Roy suggested. Ian. I think the idea is =
simple.

to Quote Roy:
b) create a branch off of some prior Apache release point in svn
   and replay the internal Y! commits on that branch until the branch
   source code is identical to what you have tested locally.  Then
   RM a tarball based on that branch and start a release vote.
   Since the history is now in svn, others could do the RM bit if
   you don't have time.


Arun has chosen option (c), that Roy also mentioned as a valid way of =
doing it.

> If you decide to donate to a non-profit organization you are free to =
choose
> the form of your donation.


I think you are confusing a non-profit for a dumping ground.=20
Any organization (non-profit or for-profit) has responsibilities, and =
their is always a tradeoff between features and risk. Any organization =
can choose to not accept a donation. It comes down to give-and-take


As Roy also mentioned, option (c) will be harder for others to test, and =
get consensus about weather it is release worthy, let alone merge into =
0.22.


>=20
> Thanks,
> --Konstantin


From general-return-2872-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jan 23 15:16:11 2011
Return-Path: <general-return-2872-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76918 invoked from network); 23 Jan 2011 15:16:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jan 2011 15:16:11 -0000
Received: (qmail 91188 invoked by uid 500); 23 Jan 2011 15:16:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89865 invoked by uid 500); 23 Jan 2011 15:16:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89835 invoked by uid 99); 23 Jan 2011 15:15:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 23 Jan 2011 15:15:59 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 23 Jan 2011 15:15:52 +0000
Received: from EGL-EX07CAS02.ds.corp.yahoo.com (egl-ex07cas02.eglbp.corp.yahoo.com [203.83.248.209])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0NFFCQ5009551;
	Sun, 23 Jan 2011 07:15:12 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1295795713;
	bh=ecE1S+uHaJno+Bw9I6PWKR2Sgauada42c7lIQJLo2j4=;
	h=From:To:Date:Subject:Message-ID:References:In-Reply-To:
	 Content-Type:MIME-Version;
	b=vMWgh1LTpBYI+wSoYGJf1VpoZjee5W3yQm/DzZew+/Swdu60v2MsshEnpFYATi5qL
	 mrMezNK+Qr1mO8hHV3kiBHsePSwpArDaaFLsyJROq7+xv4ifzQS3yfystoTGhMEsQH
	 e5ib1TpJFuaa5hzNYuuxZMJF6Xnxa7P75jfGlnXY=
Received: from EGL-EX07VS01.ds.corp.yahoo.com ([203.83.248.205]) by
 EGL-EX07CAS02.ds.corp.yahoo.com ([203.83.248.216]) with mapi; Sun, 23 Jan
 2011 20:45:11 +0530
From: yahoohadoop <yahoohadoop@yahoo-inc.com>
To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>,
        "user@pig.apache.org" <user@pig.apache.org>,
        "user@hive.apache.org"
	<user@hive.apache.org>,
        "general@hadoop.apache.org"
	<general@hadoop.apache.org>
Date: Sun, 23 Jan 2011 20:45:08 +0530
Subject: Apache Hadoop India Summit 2011
Thread-Topic: Apache Hadoop India Summit 2011
Thread-Index: Acu7C9VNYFfkSjAATb2ocBMGxkVBugAAmlxQ
Message-ID: <9C9FC731510CC94A8CDBA912E060F62ED1E454C1AE@EGL-EX07VS01.ds.corp.yahoo.com>
References: <1295793756.521925@eventbrite.com>
In-Reply-To: <1295793756.521925@eventbrite.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_9C9FC731510CC94A8CDBA912E060F62ED1E454C1AEEGLEX07VS01ds_"
MIME-Version: 1.0

--_000_9C9FC731510CC94A8CDBA912E060F62ED1E454C1AEEGLEX07VS01ds_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi Hadoop enthusiasts,

Yahoo! India R&D is proud to host second India Hadoop Summit which will tak=
e place in Bangalore, India on February 16th, 2011.

Participate in the biggest annual cloud event to learn about latest trends,=
 interact with experts, exchange ideas and see what large companies and sta=
rtups are doing in the exciting space of Apache Hadoop and cloud computing.

Please look at http://developer.yahoo.com/blogs/ydn/posts/2011/01/hadoop-in=
dia-summit-2011/   for more information.

Best Regards,
Basant Verma
Yahoo India R&D, Hadoop


--_000_9C9FC731510CC94A8CDBA912E060F62ED1E454C1AEEGLEX07VS01ds_--

From general-return-2873-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 24 11:37:08 2011
Return-Path: <general-return-2873-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91291 invoked from network); 24 Jan 2011 11:37:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jan 2011 11:37:08 -0000
Received: (qmail 51412 invoked by uid 500); 24 Jan 2011 11:37:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51092 invoked by uid 500); 24 Jan 2011 11:37:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51073 invoked by uid 99); 24 Jan 2011 11:37:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Jan 2011 11:37:03 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of binhara@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Jan 2011 11:36:56 +0000
Received: by wwd20 with SMTP id 20so4012805wwd.29
        for <general@hadoop.apache.org>; Mon, 24 Jan 2011 03:36:35 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=uvyaV3g7znCkesab+sbpruIH9HC41x/jCHoNjdowEEQ=;
        b=nTN+vL2+B3FRBXJmHroj1gJFeTflrRRtTTwgZsc1HNRSIfzCOfGtTo4JTDelhkQ69f
         r639fZkwLCWlxotWYIaz7/lYtNPQQpk6tCNcHNw3oUskJ5KL0ATPfYphLJmxYlopumeS
         hbIanOYturO2Eog66Rj2neaGZz8c/b5tfV+BI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=bRYIKJulQkXWCyR1BpEr4Ku5yWgLIG2V4rnqfY475MpxhcvuSsOzQ5ahxkNRmLUh+h
         PtcN7bnRoyfHWAG3EshV1vFkAKYbMtXw/+CJp7+u0UeRyBZB0qn5BiToBMaFHGUJI6MC
         3RZkvZEU5j45jyJbYbWtmmgAQcYxAVELXbIYU=
MIME-Version: 1.0
Received: by 10.216.220.219 with SMTP id o69mr3520579wep.57.1295868995124;
 Mon, 24 Jan 2011 03:36:35 -0800 (PST)
Received: by 10.216.17.146 with HTTP; Mon, 24 Jan 2011 03:36:35 -0800 (PST)
Date: Mon, 24 Jan 2011 09:36:35 -0200
Message-ID: <AANLkTinXOcu16MSGUkahUCEXrz94v8Ym+c6BiTvu-67X@mail.gmail.com>
Subject: Problem write on HDFS
From: Alessandro Binhara <binhara@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364c7b91213dd8049a960397

--0016364c7b91213dd8049a960397
Content-Type: text/plain; charset=ISO-8859-1

Hello ..

i solve problem in jar..
i put a hadoop-core-0.20.2.jar   in same jar dir.

i configure a class path
export CLASSPATH=.:$JAVA_HOME

i got this erro in shell

root:~# java -jar HahoopHdfsHello.jar
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/conf/Configuration
        at HadooHdfsHello.main(HadooHdfsHello.java:18)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.conf.Configuration
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
        ... 1 more


What is the problem?

thanks

--0016364c7b91213dd8049a960397--

From general-return-2874-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 24 14:55:07 2011
Return-Path: <general-return-2874-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97593 invoked from network); 24 Jan 2011 14:55:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jan 2011 14:55:07 -0000
Received: (qmail 43348 invoked by uid 500); 24 Jan 2011 14:55:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42906 invoked by uid 500); 24 Jan 2011 14:55:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42884 invoked by uid 99); 24 Jan 2011 14:55:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Jan 2011 14:55:00 +0000
X-ASF-Spam-Status: No, hits=1.3 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_BL_SPAMCOP_NET,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of qwertymaniac@gmail.com designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Jan 2011 14:54:53 +0000
Received: by fxm2 with SMTP id 2so4645370fxm.35
        for <general@hadoop.apache.org>; Mon, 24 Jan 2011 06:54:33 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type:content-transfer-encoding;
        bh=j5pUXknRjXX7FXDQ2ww2kk4efi9DALgZBQlazYBlnbI=;
        b=lUnLrcPxVEoU8axdadPsQnc+iv1opUa0u0zwcDsihDdhqtP9rU0eewgEZgZ4bFBQ1p
         0Dm64xvz/8oZmcJbXGvHj64CR061nHHS9ew1aR/OhoThwpr5IXfU8KmxjbbN8Fwm2fZs
         eTOjKryXvaAR8P29EZ6G6okQHDgE2fK5tOdT0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=xF0/z8i5G3shPWea0uVqwv88bq8iIp4IjMjGA8PKoxdxJozppzwQYroIWpD/rdeMJt
         P5ads5PIQWfYjfqRzei9nsSHvx+n9jqbiAOn4qo8jy4Bx4hTYuUhqSQyhXqBZxgflYCi
         Q7kmho0eFH/k/7WGXXakCDL8y0vXRxDLYPr3o=
Received: by 10.223.95.202 with SMTP id e10mr4358535fan.32.1295880873131; Mon,
 24 Jan 2011 06:54:33 -0800 (PST)
MIME-Version: 1.0
Received: by 10.223.124.200 with HTTP; Mon, 24 Jan 2011 06:54:11 -0800 (PST)
In-Reply-To: <AANLkTinXOcu16MSGUkahUCEXrz94v8Ym+c6BiTvu-67X@mail.gmail.com>
References: <AANLkTinXOcu16MSGUkahUCEXrz94v8Ym+c6BiTvu-67X@mail.gmail.com>
From: Harsh J <qwertymaniac@gmail.com>
Date: Mon, 24 Jan 2011 20:24:11 +0530
Message-ID: <AANLkTim08e3Np6SBxE1FSpRUpW40xoFv6TfhiDXaGNP+@mail.gmail.com>
Subject: Re: Problem write on HDFS
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

The issue would definitely lie with your CLASSPATH.

Ideally, while beginning development using Hadoop 0.20, it is better
to use the `hadoop jar` command to launch jars of any kind that
require Hadoop libraries; be it MapReduce or not. The command will
ensure that all the classpath requirements for Hadoop-side libraries
are satisfied, so you don't have to worry.

Anyhow, try launching it this way:
$ java -classpath hadoop-0.20.2-core.jar -jar HadoopHdfsHello.jar; #
This should run just fine.

On Mon, Jan 24, 2011 at 5:06 PM, Alessandro Binhara <binhara@gmail.com> wro=
te:
> Hello ..
>
> i solve problem in jar..
> i put a hadoop-core-0.20.2.jar =A0 in same jar dir.
>
> i configure a class path
> export CLASSPATH=3D.:$JAVA_HOME
>
> i got this erro in shell
>
> root:~# java -jar HahoopHdfsHello.jar
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/conf/Configuration
> =A0 =A0 =A0 =A0at HadooHdfsHello.main(HadooHdfsHello.java:18)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.conf.Configuration
> =A0 =A0 =A0 =A0at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> =A0 =A0 =A0 =A0at java.security.AccessController.doPrivileged(Native Meth=
od)
> =A0 =A0 =A0 =A0at java.net.URLClassLoader.findClass(URLClassLoader.java:1=
90)
> =A0 =A0 =A0 =A0at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> =A0 =A0 =A0 =A0at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.jav=
a:301)
> =A0 =A0 =A0 =A0at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> =A0 =A0 =A0 =A0... 1 more
>
>
> What is the problem?
>
> thanks
>



--=20
Harsh J
www.harshj.com

From general-return-2875-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 24 15:02:22 2011
Return-Path: <general-return-2875-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 99923 invoked from network); 24 Jan 2011 15:02:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jan 2011 15:02:21 -0000
Received: (qmail 54384 invoked by uid 500); 24 Jan 2011 15:02:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54032 invoked by uid 500); 24 Jan 2011 15:02:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53497 invoked by uid 99); 24 Jan 2011 15:02:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Jan 2011 15:02:15 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of binhara@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Jan 2011 15:02:08 +0000
Received: by wwd20 with SMTP id 20so4216374wwd.29
        for <general@hadoop.apache.org>; Mon, 24 Jan 2011 07:01:48 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=qXf/ljHbQmicoTrwaIN1e3VQYZEnKjItOtoLjQrXSHE=;
        b=EwFGcvnWZlTKD9Gp3M3aouXHoMeHr6qt4xI/SwMnfSpk2FlhL0ArgwydqHBA6tW3jn
         fArPnk0qAdqxs4YIxRtFy2GjI7dHghzHTacsLHqi6OWdtkxAn7XQOSZhGdErxpgfv/Ub
         xs4Ft7FxjONCN1FqR59xbqQvvTobPgEeJQu98=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=wAsgYEQhR4ACvfk6ZuTJoYY2jTAVu897EoDZHcOLzcCa4YHiAyRYH50e+75EVIDvO6
         MnZU6X0X0zBbOXKGEIx+qh9JSkTAb7kE2leFxGUds4sw4p6/YwjnbFvwiQI6D8m4M5GX
         RBDtz3JumUcGwl8PPHGZ8LndJXnd5Umf2Fs04=
MIME-Version: 1.0
Received: by 10.216.51.130 with SMTP id b2mr3782979wec.42.1295881308142; Mon,
 24 Jan 2011 07:01:48 -0800 (PST)
Received: by 10.216.17.146 with HTTP; Mon, 24 Jan 2011 07:01:48 -0800 (PST)
In-Reply-To: <AANLkTim08e3Np6SBxE1FSpRUpW40xoFv6TfhiDXaGNP+@mail.gmail.com>
References: <AANLkTinXOcu16MSGUkahUCEXrz94v8Ym+c6BiTvu-67X@mail.gmail.com>
	<AANLkTim08e3Np6SBxE1FSpRUpW40xoFv6TfhiDXaGNP+@mail.gmail.com>
Date: Mon, 24 Jan 2011 13:01:48 -0200
Message-ID: <AANLkTins_HUzpVu25foOSrfGqFDC2gMyp48O83ep2rHb@mail.gmail.com>
Subject: Re: Problem write on HDFS
From: Alessandro Binhara <binhara@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6dd8c8d0afd78049a98e172
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6dd8c8d0afd78049a98e172
Content-Type: text/plain; charset=ISO-8859-1

..i try
 java -classpath hadoop-core-0.20.1.jar -jar HahoopHdfsHello.jar

i got a same error..
i will try build a servlet and run on tomcat...
i try many issues to config a classpath... all fail..

thanks


On Mon, Jan 24, 2011 at 12:54 PM, Harsh J <qwertymaniac@gmail.com> wrote:

> The issue would definitely lie with your CLASSPATH.
>
> Ideally, while beginning development using Hadoop 0.20, it is better
> to use the `hadoop jar` command to launch jars of any kind that
> require Hadoop libraries; be it MapReduce or not. The command will
> ensure that all the classpath requirements for Hadoop-side libraries
> are satisfied, so you don't have to worry.
>
> Anyhow, try launching it this way:
> $ java -classpath hadoop-0.20.2-core.jar -jar HadoopHdfsHello.jar; #
> This should run just fine.
>
> On Mon, Jan 24, 2011 at 5:06 PM, Alessandro Binhara <binhara@gmail.com>
> wrote:
> > Hello ..
> >
> > i solve problem in jar..
> > i put a hadoop-core-0.20.2.jar   in same jar dir.
> >
> > i configure a class path
> > export CLASSPATH=.:$JAVA_HOME
> >
> > i got this erro in shell
> >
> > root:~# java -jar HahoopHdfsHello.jar
> > Exception in thread "main" java.lang.NoClassDefFoundError:
> > org/apache/hadoop/conf/Configuration
> >        at HadooHdfsHello.main(HadooHdfsHello.java:18)
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.hadoop.conf.Configuration
> >        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >        at java.security.AccessController.doPrivileged(Native Method)
> >        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> >        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> >        ... 1 more
> >
> >
> > What is the problem?
> >
> > thanks
> >
>
>
>
> --
> Harsh J
> www.harshj.com
>

--0016e6dd8c8d0afd78049a98e172--

From general-return-2876-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 25 10:05:59 2011
Return-Path: <general-return-2876-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1716 invoked from network); 25 Jan 2011 10:05:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Jan 2011 10:05:58 -0000
Received: (qmail 12321 invoked by uid 500); 25 Jan 2011 10:05:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11971 invoked by uid 500); 25 Jan 2011 10:05:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11957 invoked by uid 99); 25 Jan 2011 10:05:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Jan 2011 10:05:53 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Jan 2011 10:05:45 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0PA53DR011922
	for <general@hadoop.apache.org>; Tue, 25 Jan 2011 02:05:03 -0800 (PST)
Received: from [10.0.1.4] (10.72.76.163) by SP2-EX07CAS01.ds.corp.yahoo.com
 (98.137.59.4) with Microsoft SMTP Server (TLS) id 8.3.137.0; Tue, 25 Jan 2011
 02:05:03 -0800
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <B35E1DF9-ECC7-4A0D-9BFF-2F5A45CED68D@Holsman.NET>
Date: Tue, 25 Jan 2011 02:05:00 -0800
Content-Transfer-Encoding: quoted-printable
Message-ID: <5101FEE7-4C57-49C7-841C-2A9BC7BE0464@yahoo-inc.com>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com> <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com> <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org> <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com> <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com> <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com> <4D34A270.1060102@apache.org> <3FEB7C1D-AE89-46BE-ABF2-B595B01E222A@mac.com> <D160E684-7435-4E53-A766-231CCB38D1B4@yahoo-inc.com> <3995AF5C-AA18-475B-9384-A532F1CC5D57@linkedin.com> <2090582B-B586-4F81-A724-6A0C55820BB5@yahoo-inc.com> <704BFF6E-AC1D-4769-8856-BFBD125D4D2C@Holsman.NET> <AANLkTikq-bPfXviWThvh=--cWEDYPfd0iEFSE_m_RbMn@mail.gmail.com> <B35E1DF9-ECC7-4A0D-9BFF-2F5A45CED68D@Holsman.NET>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Ian,

No votes have been called for at the moment.  Right now all arun's done =
is create a branch and ask for feedback from folks who want to try it.  =
Most of the forward ports are already either committed or in a patch =
available state, as arun mentioned.  We'll work through the others as =
individual JIRAs to allow everyone to kick the tires.  That should avoid =
issues with 0.22. =20

I don't anticipate votes etc, unless folks do want to try it and do =
provide feedback.  This is what runs at yahoo at the moment.  I hope =
people think it is worth trying.

Thanks,

E14

On Jan 22, 2011, at 6:22 AM, Ian Holsman wrote:

>=20
> On Jan 19, 2011, at 1:12 PM, Konstantin Shvachko wrote:
>=20
>> On Tue, Jan 18, 2011 at 11:49 PM, Ian Holsman <hadoop@holsman.net> =
wrote:
>>=20
>>> I think Roy's suggestion of applying the commits individually to the =
branch
>>> from your current working branch would help with this.
>>>=20
>>=20
>> I am sure this is not what Roy suggested. Ian. I think the idea is =
simple.
>=20
> to Quote Roy:
> b) create a branch off of some prior Apache release point in svn
>   and replay the internal Y! commits on that branch until the branch
>   source code is identical to what you have tested locally.  Then
>   RM a tarball based on that branch and start a release vote.
>   Since the history is now in svn, others could do the RM bit if
>   you don't have time.
>=20
>=20
> Arun has chosen option (c), that Roy also mentioned as a valid way of =
doing it.
>=20
>> If you decide to donate to a non-profit organization you are free to =
choose
>> the form of your donation.
>=20
>=20
> I think you are confusing a non-profit for a dumping ground.=20
> Any organization (non-profit or for-profit) has responsibilities, and =
their is always a tradeoff between features and risk. Any organization =
can choose to not accept a donation. It comes down to give-and-take
>=20
>=20
> As Roy also mentioned, option (c) will be harder for others to test, =
and get consensus about weather it is release worthy, let alone merge =
into 0.22.
>=20
>=20
>>=20
>> Thanks,
>> --Konstantin
>=20


From general-return-2877-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 25 11:27:59 2011
Return-Path: <general-return-2877-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31618 invoked from network); 25 Jan 2011 11:27:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Jan 2011 11:27:59 -0000
Received: (qmail 22219 invoked by uid 500); 25 Jan 2011 11:27:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21935 invoked by uid 500); 25 Jan 2011 11:27:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21927 invoked by uid 99); 25 Jan 2011 11:27:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Jan 2011 11:27:55 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of binhara@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Jan 2011 11:27:50 +0000
Received: by wye20 with SMTP id 20so5539134wye.35
        for <general@hadoop.apache.org>; Tue, 25 Jan 2011 03:27:28 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=zMo4Q8mJrPYrweZjtJLkAoFeWlG0LIZqe8sHe05Knbo=;
        b=psnSgAXwyltjZTp4bFSSA7cPnxn/N8/r4s2eravGX4KUlRJXMFNZnu0tRVhHHERyNc
         ZravW3sMYSPYAlS2YxMNRQ0Y4hJvnxtNR2MNP6bR3t4Gqt1cAyC4yp7YJBp6P2pm5yHy
         gNKj8Jt/nGxw/xSUs2GTcPjeJszYLyEeQI0OA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=V3cX9pbiQ+nRjhBAYEeISrSWcCycJ1VhEBaTbzAxBNcGq5wfZ9Vx+UqouZogakBK5C
         zVys9IQGonCTMGdlez+L+qx19fMPdFGdOl9dUYvarfwd+eaPjmbStqbMrLzJ7qG8cpSC
         uXS7KE3uXPfOOF5IpRl30cgHBvNiFy+QjFM0I=
MIME-Version: 1.0
Received: by 10.216.155.75 with SMTP id i53mr3524034wek.27.1295954847734; Tue,
 25 Jan 2011 03:27:27 -0800 (PST)
Received: by 10.216.17.146 with HTTP; Tue, 25 Jan 2011 03:27:27 -0800 (PST)
In-Reply-To: <AANLkTins_HUzpVu25foOSrfGqFDC2gMyp48O83ep2rHb@mail.gmail.com>
References: <AANLkTinXOcu16MSGUkahUCEXrz94v8Ym+c6BiTvu-67X@mail.gmail.com>
	<AANLkTim08e3Np6SBxE1FSpRUpW40xoFv6TfhiDXaGNP+@mail.gmail.com>
	<AANLkTins_HUzpVu25foOSrfGqFDC2gMyp48O83ep2rHb@mail.gmail.com>
Date: Tue, 25 Jan 2011 09:27:27 -0200
Message-ID: <AANLkTikBvezqxrmjS8wBnnTebhgELsFf2=Y=zH81CdWb@mail.gmail.com>
Subject: Re: Problem write on HDFS
From: Alessandro Binhara <binhara@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e65b535a581a05049aaa0091

--0016e65b535a581a05049aaa0091
Content-Type: text/plain; charset=ISO-8859-1

I build a servlet with a hadoop...
i think that tomcat enviroment will be find a hadoop-core-0.20.2.jar .. but
a get a same error

*ype* Exception report

*message***

*description* *The server encountered an internal error () that prevented it
from fulfilling this request.*

*exception*

javax.servlet.ServletException: Servlet execution threw an exception

*root cause*

java.lang.NoClassDefFoundError: Could not initialize class
org.apache.hadoop.conf.Configuration
	HadoopWriterLib.HadoopWriter.OpenFileSystem(HadoopWriter.java:22)
	HadoopWriterLib.HadoopWriter.<init>(HadoopWriter.java:16)
	HadoopServletTest.doGet(HadoopServletTest.java:35)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:722)

*note* *The full stack trace of the root cause is available in the Apache
Tomcat/7.0.6 logs.*

The error can be a problem on my ubuntu server ?

thanks

On Mon, Jan 24, 2011 at 1:01 PM, Alessandro Binhara <binhara@gmail.com>wrote:

> ..i try
>  java -classpath hadoop-core-0.20.1.jar -jar HahoopHdfsHello.jar
>
> i got a same error..
> i will try build a servlet and run on tomcat...
> i try many issues to config a classpath... all fail..
>
> thanks
>
>
> On Mon, Jan 24, 2011 at 12:54 PM, Harsh J <qwertymaniac@gmail.com> wrote:
>
>> The issue would definitely lie with your CLASSPATH.
>>
>> Ideally, while beginning development using Hadoop 0.20, it is better
>> to use the `hadoop jar` command to launch jars of any kind that
>> require Hadoop libraries; be it MapReduce or not. The command will
>> ensure that all the classpath requirements for Hadoop-side libraries
>> are satisfied, so you don't have to worry.
>>
>> Anyhow, try launching it this way:
>> $ java -classpath hadoop-0.20.2-core.jar -jar HadoopHdfsHello.jar; #
>> This should run just fine.
>>
>> On Mon, Jan 24, 2011 at 5:06 PM, Alessandro Binhara <binhara@gmail.com>
>> wrote:
>> > Hello ..
>> >
>> > i solve problem in jar..
>> > i put a hadoop-core-0.20.2.jar   in same jar dir.
>> >
>> > i configure a class path
>> > export CLASSPATH=.:$JAVA_HOME
>> >
>> > i got this erro in shell
>> >
>> > root:~# java -jar HahoopHdfsHello.jar
>> > Exception in thread "main" java.lang.NoClassDefFoundError:
>> > org/apache/hadoop/conf/Configuration
>> >        at HadooHdfsHello.main(HadooHdfsHello.java:18)
>> > Caused by: java.lang.ClassNotFoundException:
>> > org.apache.hadoop.conf.Configuration
>> >        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> >        at java.security.AccessController.doPrivileged(Native Method)
>> >        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> >        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>> >        ... 1 more
>> >
>> >
>> > What is the problem?
>> >
>> > thanks
>> >
>>
>>
>>
>> --
>> Harsh J
>> www.harshj.com
>>
>
>

--0016e65b535a581a05049aaa0091--

From general-return-2878-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 25 14:52:07 2011
Return-Path: <general-return-2878-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19275 invoked from network); 25 Jan 2011 14:52:06 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Jan 2011 14:52:06 -0000
Received: (qmail 62592 invoked by uid 500); 25 Jan 2011 14:52:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62215 invoked by uid 500); 25 Jan 2011 14:52:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62206 invoked by uid 99); 25 Jan 2011 14:52:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Jan 2011 14:52:01 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Jan 2011 14:51:55 +0000
Received: by vws18 with SMTP id 18so2197860vws.35
        for <general@hadoop.apache.org>; Tue, 25 Jan 2011 06:51:33 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=XGRRBllNi9/V5K7Tog1ap1abFd6HOZwTd8qU9tkb5ic=;
        b=rCeVUvsBA3IpPwXpxN56dEJ3EnzRLilIemG3TQYz1w8RlDH2btoqszuRx5zeZu2Ocd
         GP/UiyROTkcsSmaYW55IKFe9QWQcmx0zZCQ3DWLjT9LueXKDV95LsqXcruGmYiNcA4B+
         nd1ycYpSMe6gRKv6KPb/chsUhjZa+BTB1QOBU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=fIc59HVmt3UBYwQ0cqMr760piJdMIKBd6rHCHCM9kNt7QAYsOtIq9iDnaVPxO26eQ2
         tbHDZXNs6pp3ijDLzikSXDrKOQIqV9t67M10R7mMzijzdMoN50Hsy8SryfJZzrsQ0iAB
         tm0Euct7Qr21CoW/7HWJhzm92cSs0Xbdd5PGU=
Received: by 10.229.82.14 with SMTP id z14mr4845903qck.257.1295967093521;
        Tue, 25 Jan 2011 06:51:33 -0800 (PST)
Received: from [10.172.33.1] (h-64-236-138-3.aoltw.net [64.236.138.3])
        by mx.google.com with ESMTPS id g28sm10256331qck.13.2011.01.25.06.51.29
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Tue, 25 Jan 2011 06:51:32 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <5101FEE7-4C57-49C7-841C-2A9BC7BE0464@yahoo-inc.com>
Date: Tue, 25 Jan 2011 06:51:24 -0800
Content-Transfer-Encoding: quoted-printable
Message-Id: <2E5193CC-8A01-4FA8-BE48-F59AD3750E41@Holsman.NET>
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com> <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com> <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org> <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com> <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com> <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com> <4D34A270.1060102@apache.org> <3FEB7C1D-AE89-46BE-ABF2-B595B01E222A@mac.com> <D160E684-7435-4E53-A766-231CCB38D1B4@yahoo-inc.com> <3995AF5C-AA18-475B-9384-A532F1CC5D57@linkedin.com> <2090582B-B586-4F81-A724-6A0C55820BB5@yahoo-inc.com> <704BFF6E-AC1D-4769-8856-BFBD125D4D2C@Holsman.NET> <AANLkTikq-bPfXviWThvh=--cWEDYPfd0iEFSE_m_RbMn@mail.gmail.com> <B35E1DF9-ECC7-4A0D-9BFF-2F5A45CED68D@Holsman.NET> <5101FEE7-4C57-49C7-841C-2A9BC7BE0464@yahoo-inc.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

Arun also mentioned that he has created a separate branch ( =
http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20-security-p=
atches/ ) for the individual patches.. so he is doing both.. so everyone =
should be happy.

--I
On Jan 25, 2011, at 2:05 AM, Eric Baldeschwieler wrote:

> Hi Ian,
>=20
> No votes have been called for at the moment.  Right now all arun's =
done is create a branch and ask for feedback from folks who want to try =
it.  Most of the forward ports are already either committed or in a =
patch available state, as arun mentioned.  We'll work through the others =
as individual JIRAs to allow everyone to kick the tires.  That should =
avoid issues with 0.22. =20
>=20
> I don't anticipate votes etc, unless folks do want to try it and do =
provide feedback.  This is what runs at yahoo at the moment.  I hope =
people think it is worth trying.
>=20
> Thanks,
>=20
> E14
>=20
> On Jan 22, 2011, at 6:22 AM, Ian Holsman wrote:
>=20
>>=20
>> On Jan 19, 2011, at 1:12 PM, Konstantin Shvachko wrote:
>>=20
>>> On Tue, Jan 18, 2011 at 11:49 PM, Ian Holsman <hadoop@holsman.net> =
wrote:
>>>=20
>>>> I think Roy's suggestion of applying the commits individually to =
the branch
>>>> from your current working branch would help with this.
>>>>=20
>>>=20
>>> I am sure this is not what Roy suggested. Ian. I think the idea is =
simple.
>>=20
>> to Quote Roy:
>> b) create a branch off of some prior Apache release point in svn
>>  and replay the internal Y! commits on that branch until the branch
>>  source code is identical to what you have tested locally.  Then
>>  RM a tarball based on that branch and start a release vote.
>>  Since the history is now in svn, others could do the RM bit if
>>  you don't have time.
>>=20
>>=20
>> Arun has chosen option (c), that Roy also mentioned as a valid way of =
doing it.
>>=20
>>> If you decide to donate to a non-profit organization you are free to =
choose
>>> the form of your donation.
>>=20
>>=20
>> I think you are confusing a non-profit for a dumping ground.=20
>> Any organization (non-profit or for-profit) has responsibilities, and =
their is always a tradeoff between features and risk. Any organization =
can choose to not accept a donation. It comes down to give-and-take
>>=20
>>=20
>> As Roy also mentioned, option (c) will be harder for others to test, =
and get consensus about weather it is release worthy, let alone merge =
into 0.22.
>>=20
>>=20
>>>=20
>>> Thanks,
>>> --Konstantin
>>=20
>=20


From general-return-2879-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 25 18:07:53 2011
Return-Path: <general-return-2879-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 80615 invoked from network); 25 Jan 2011 18:07:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Jan 2011 18:07:53 -0000
Received: (qmail 41420 invoked by uid 500); 25 Jan 2011 18:07:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 41332 invoked by uid 500); 25 Jan 2011 18:07:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 41324 invoked by uid 99); 25 Jan 2011 18:07:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Jan 2011 18:07:50 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of binhara@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Jan 2011 18:07:43 +0000
Received: by wwd20 with SMTP id 20so58398wwd.29
        for <general@hadoop.apache.org>; Tue, 25 Jan 2011 10:07:23 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=w2OGkYupPW5qzHbrfg9Rhfco8h6SsRs0xAx60OuZ1n4=;
        b=PbduD774GHDx1TPXppIuTv4nY0MjkUxtlxehznVIlTjZYkpVdZozPYPZbSiHNXEx93
         ViiRpEkP5zRY/+3Nt79PCgZEN+eh/yuUOdeKIBx5dhBvac+wpSB2RZFHkL2Gmjz4deB3
         u/K0AVvuGGxi5Jsz1ErECoNe+Q3knsLGBpSMg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=RWPECJioEPKYxvStc4L6wXF1kPOrIlRyQLOJisirONHU2C78kedMVvZqVgDCmb2uES
         Pio694vvUMQwEZkrX2qIN/Km8qcX4dbi4BFNNCMxiXWZYxW1VVextXwke/r5xFFUZpW+
         1MgdWSWPx/KpEEeS0j9MEtupryFnWjYunTAMo=
MIME-Version: 1.0
Received: by 10.216.150.134 with SMTP id z6mr139315wej.27.1295978843262; Tue,
 25 Jan 2011 10:07:23 -0800 (PST)
Received: by 10.216.17.146 with HTTP; Tue, 25 Jan 2011 10:07:23 -0800 (PST)
Date: Tue, 25 Jan 2011 16:07:23 -0200
Message-ID: <AANLkTik3ekxLZH8qC0Sm+yfjaq7HEObMYdrOfAwv3JtY@mail.gmail.com>
Subject: Re: Problem write on HDFS - SOLUTION
From: Alessandro Binhara <binhara@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6daa8a596d183049aaf96e3
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6daa8a596d183049aaf96e3
Content-Type: text/plain; charset=ISO-8859-1

Hello all..

i found the problem . A Hadoop jar file from a hadoop website has a
problem.
I copy a jar file from a Cloudera distribuition and error desapear ..

If have a commiter from hadoop website.. please check the problem ..

thanks

On Tue, Jan 25, 2011 at 9:27 AM, Alessandro Binhara <binhara@gmail.com>wrote:

> I build a servlet with a hadoop...
> i think that tomcat enviroment will be find a hadoop-core-0.20.2.jar ..
> but a get a same error
>
> *ype* Exception report
>
> *message***
>
> *description* *The server encountered an internal error () that prevented
> it from fulfilling this request.*
>
> *exception*
>
> javax.servlet.ServletException: Servlet execution threw an exception
>
>  *root cause*
>
> java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.conf.Configuration
> 	HadoopWriterLib.HadoopWriter.OpenFileSystem(HadoopWriter.java:22)
> 	HadoopWriterLib.HadoopWriter.<init>(HadoopWriter.java:16)
> 	HadoopServletTest.doGet(HadoopServletTest.java:35)
> 	javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> 	javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
>
>  *note* *The full stack trace of the root cause is available in the Apache
> Tomcat/7.0.6 logs.*
>
> The error can be a problem on my ubuntu server ?
>
> thanks
>
> On Mon, Jan 24, 2011 at 1:01 PM, Alessandro Binhara <binhara@gmail.com>wrote:
>
>> ..i try
>>  java -classpath hadoop-core-0.20.1.jar -jar HahoopHdfsHello.jar
>>
>> i got a same error..
>> i will try build a servlet and run on tomcat...
>> i try many issues to config a classpath... all fail..
>>
>> thanks
>>
>>
>> On Mon, Jan 24, 2011 at 12:54 PM, Harsh J <qwertymaniac@gmail.com> wrote:
>>
>>> The issue would definitely lie with your CLASSPATH.
>>>
>>> Ideally, while beginning development using Hadoop 0.20, it is better
>>> to use the `hadoop jar` command to launch jars of any kind that
>>> require Hadoop libraries; be it MapReduce or not. The command will
>>> ensure that all the classpath requirements for Hadoop-side libraries
>>> are satisfied, so you don't have to worry.
>>>
>>> Anyhow, try launching it this way:
>>> $ java -classpath hadoop-0.20.2-core.jar -jar HadoopHdfsHello.jar; #
>>> This should run just fine.
>>>
>>> On Mon, Jan 24, 2011 at 5:06 PM, Alessandro Binhara <binhara@gmail.com>
>>> wrote:
>>> > Hello ..
>>> >
>>> > i solve problem in jar..
>>> > i put a hadoop-core-0.20.2.jar   in same jar dir.
>>> >
>>> > i configure a class path
>>> > export CLASSPATH=.:$JAVA_HOME
>>> >
>>> > i got this erro in shell
>>> >
>>> > root:~# java -jar HahoopHdfsHello.jar
>>> > Exception in thread "main" java.lang.NoClassDefFoundError:
>>> > org/apache/hadoop/conf/Configuration
>>> >        at HadooHdfsHello.main(HadooHdfsHello.java:18)
>>> > Caused by: java.lang.ClassNotFoundException:
>>> > org.apache.hadoop.conf.Configuration
>>> >        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> >        at java.security.AccessController.doPrivileged(Native Method)
>>> >        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>> >        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>> >        ... 1 more
>>> >
>>> >
>>> > What is the problem?
>>> >
>>> > thanks
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>> www.harshj.com
>>>
>>
>>
>

--0016e6daa8a596d183049aaf96e3--

From general-return-2880-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 25 20:54:59 2011
Return-Path: <general-return-2880-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55413 invoked from network); 25 Jan 2011 20:54:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Jan 2011 20:54:57 -0000
Received: (qmail 8809 invoked by uid 500); 25 Jan 2011 20:54:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7514 invoked by uid 500); 25 Jan 2011 20:54:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7457 invoked by uid 99); 25 Jan 2011 20:54:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Jan 2011 20:54:38 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [85.214.115.60] (HELO h1349259.stratoserver.net) (85.214.115.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Jan 2011 20:54:31 +0000
Received: from isabel-drost.de ([85.214.115.60] helo=motokokusanagi.localnet)
	by h1349259.stratoserver.net with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
	(Exim 4.69)
	(envelope-from <isabel@apache.org>)
	id 1Phptu-0001Ql-CV; Tue, 25 Jan 2011 20:54:07 +0000
From: Isabel Drost <isabel@apache.org>
Reply-To: isabel@apache.org
To: user@mahout.apache.org
Subject: CFP - Berlin Buzzwords 2011 - Search, Score, Scale
Date: Tue, 25 Jan 2011 21:53:28 +0100
User-Agent: KMail/1.13.5 (Linux/2.6.32-5-amd64; KDE/4.4.5; x86_64; ; )
Cc: general@lucene.apache.org,
 general@hadoop.apache.org,
 user@hbase.apache.org,
 solr-user@lucene.apache.org,
 java-user@lucene.apache.org,
 user@nutch.apache.org
X-KMail-Markup: true
MIME-Version: 1.0
Content-Type: multipart/signed;
  boundary="nextPart2073183.1OyNNgtYSL";
  protocol="application/pgp-signature";
  micalg=pgp-sha1
Content-Transfer-Encoding: 7bit
Message-Id: <201101252153.30628.isabel@apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

--nextPart2073183.1OyNNgtYSL
Content-Type: multipart/alternative;
  boundary="Boundary-01=_KhzPNKWdhpyvYFX"
Content-Transfer-Encoding: 7bit

--Boundary-01=_KhzPNKWdhpyvYFX
Content-Type: text/plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

This is to announce the Berlin Buzzwords 2011. The second edition of the=20
successful conference on scalable and open search, data processing and data=
=20
storage in Germany, taking place in Berlin.

                    Call for Presentations Berlin Buzzwords
                           http://berlinbuzzwords.de
                  Berlin Buzzwords 2011 - Search, Store, Scale
                                6/7 June 2011

The event will comprise presentations on scalable data processing. We invit=
e you=20
to submit talks on the topics:

   * IR / Search - Lucene, Solr, katta or comparable solutions
   * NoSQL - like CouchDB, MongoDB, Jackrabbit, HBase and others
   * Hadoop - Hadoop itself, MapReduce, Cascading or Pig and relatives
   * Closely related topics not explicitly listed above are welcome. We are
     looking for presentations on the implementation of the systems themsel=
ves,
     real world applications and case studies.

Important Dates (all dates in GMT +2)
   * Submission deadline: March 1st 2011, 23:59 MEZ
   * Notification of accepted speakers: March 22th, 2011, MEZ.
   * Publication of final schedule: April 5th, 2011.
   * Conference: June 6/7. 2011

High quality, technical submissions are called for, ranging from principles=
 to=20
practice. We are looking for real world use cases, background on the=20
architecture of specific projects and a deep dive into architectures built =
on=20
top of e.g. Hadoop clusters.

Proposals should be submitted at http://berlinbuzzwords.de/content/cfp-0 no=
=20
later than March 1st, 2011. Acceptance notifications will be sent out soon =
after=20
the submission deadline. Please include your name, bio and email, the title=
 of=20
the talk, a brief abstract in English language. Please indicate whether you=
 want=20
to give a lightning (10min), short (20min) or long (40min) presentation and=
=20
indicate the level of experience with the topic your audience should have (=
e.g.=20
whether your talk will be suitable for newbies or is targeted for experienc=
ed=20
users.) If you'd like to pitch your brand new product in your talk, please =
let=20
us know as well - there will be extra space for presenting new ideas, aweso=
me=20
products and great new projects.

The presentation format is short. We will be enforcing the schedule rigorou=
sly.

If you are interested in sponsoring the event (e.g. we would be happy to pr=
ovide=20
videos after the event, free drinks for attendees as well as an after-show=
=20
party), please contact us.

=46ollow @hadoopberlin on Twitter for updates. Tickets, news on the confere=
nce,=20
and the final schedule are be published at http://berlinbuzzwords.de.

Program Chairs: Isabel Drost, Jan Lehnardt, and Simon Willnauer.

Please re-distribute this CfP to people who might be interested.

If you are local and wish to meet us earlier, please note that this Thursda=
y=20
evening there will be an Apache Hadoop Get Together (videos kindly sponsore=
d by=20
Cloudera, venue kindly provided for free by Zanox) featuring talks on Apach=
e=20
Hadoop in production as well as news on current Apache Lucene developments.

Contact us at:

newthinking communications=20
GmbH Sch=F6nhauser Allee 6/7=20
10119 Berlin,=20
Germany=20

Julia Gem=E4hlich
Isabel Drost=20

+49(0)30-9210 596

--Boundary-01=_KhzPNKWdhpyvYFX
Content-Type: text/html;
  charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN" "http://www.w3.org/TR/REC-=
html40/strict.dtd">
<html><head><meta name=3D"qrichtext" content=3D"1" /><style type=3D"text/cs=
s">
p, li { white-space: pre-wrap; }
</style></head><body style=3D" font-family:'Monospace'; font-size:8pt; font=
=2Dweight:400; font-style:normal;">
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;">This is to announce the Berlin Buzzw=
ords 2011. The second edition of the successful conference on scalable and =
open search, data processing and data storage in Germany, taking place in B=
erlin.</span></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;"><br />                    Call for P=
resentations Berlin Buzzwords</span></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;">                           http://be=
rlinbuzzwords.de</span></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;">                  Berlin Buzzwords 2=
011 - Search, Store, Scale</span></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;">                                6/7 =
June 2011</span></p>
<p style=3D"-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; ma=
rgin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-u=
ser-state:0; font-size:medium; color:#141312;"></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;">The event will comprise presentation=
s on scalable data processing. We invite you to submit talks on the topics:=
</span></p>
<p style=3D"-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; ma=
rgin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-u=
ser-state:0; font-size:medium; color:#141312;"></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;">   * IR / Search - Lucene, Solr, kat=
ta or comparable solutions</span></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;">   * NoSQL - like CouchDB, MongoDB, =
Jackrabbit, HBase and others</span></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;">   * Hadoop - Hadoop itself, MapRedu=
ce, Cascading or Pig and relatives</span></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;">   * Closely related topics not expl=
icitly listed above are welcome. We are</span></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;">     looking for presentations on th=
e implementation of the systems themselves,</span></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;">     real world applications and cas=
e studies.</span></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;"><br />Important Dates (all dates in =
GMT +2)</span></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;">   * Submission deadline: March 1st =
2011, 23:59 MEZ</span></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;">   * Notification of accepted speake=
rs: March 22th, 2011, MEZ.</span></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;">   * Publication of final schedule: =
April 5th, 2011.</span></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;">   * Conference: June 6/7. 2011</spa=
n></p>
<p style=3D"-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; ma=
rgin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-u=
ser-state:0; font-size:medium; color:#141312;"></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;">High quality, technical submissions =
are called for, ranging from principles to practice. We are looking for rea=
l world use cases, background on the architecture of specific projects and =
a deep dive into architectures built on top of e.g. Hadoop clusters.</span>=
</p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;"><br />Proposals should be submitted =
at http://berlinbuzzwords.de/content/cfp-0 no later than March 1st, 2011. A=
cceptance notifications will be sent out soon after the submission deadline=
=2E Please include your name, bio and email, the title of the talk, a brief=
 abstract in English language. Please indicate whether you want to give a l=
ightning (10min), short (20min) or long (40min) presentation and indicate t=
he level of experience with the topic your audience should have (e.g. wheth=
er your talk will be suitable for newbies or is targeted for experienced us=
ers.) If you'd like to pitch your brand new product in your talk, please le=
t us know as well - there will be extra space for presenting new ideas, awe=
some products and great new projects.</span></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;"><br />The presentation format is sho=
rt. We will be enforcing the schedule rigorously.</span></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;"><br />If you are interested in spons=
oring the event (e.g. we would be happy to provide videos after the event, =
free drinks for attendees as well as an after-show party), please contact u=
s.</span></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;"><br />Follow @hadoopberlin on Twitte=
r for updates. Tickets, news on the conference, and the final schedule are =
be published at http://berlinbuzzwords.de.</span></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;"><br />Program Chairs: Isabel Drost, =
Jan Lehnardt, and Simon Willnauer.</span></p>
<p style=3D"-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; ma=
rgin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-u=
ser-state:0; font-size:medium; color:#141312;"></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;">Please re-distribute this CfP to peo=
ple who might be interested.</span></p>
<p style=3D"-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; ma=
rgin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-u=
ser-state:0; font-size:medium; color:#141312;"></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" color:#141312;">If you are local and wish to meet us earlier, please n=
ote that this Thursday evening there will be an Apache Hadoop Get Together =
(videos kindly sponsored by Cloudera, venue kindly provided for free by Zan=
ox) featuring talks on Apache Hadoop in production as well as news on curre=
nt Apache Lucene developments.</span></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;"><br />Contact us at:</span></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;"><br />newthinking communications </s=
pan></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;">GmbH Sch=F6nhauser Allee 6/7 </span>=
</p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;">10119 Berlin, </span></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;">Germany </span></p>
<p style=3D"-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; ma=
rgin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-u=
ser-state:0; font-size:medium; color:#141312;"></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;">Julia Gem=E4hlich</span></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;">Isabel Drost </span></p>
<p style=3D"-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; ma=
rgin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-u=
ser-state:0; font-size:medium; color:#141312;"></p>
<p style=3D" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-rig=
ht:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=
=3D" font-size:medium; color:#141312;">+49(0)30-9210 596</span></p></body><=
/html>
--Boundary-01=_KhzPNKWdhpyvYFX--

--nextPart2073183.1OyNNgtYSL
Content-Type: application/pgp-signature; name=signature.asc 
Content-Description: This is a digitally signed message part.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEABECAAYFAk0/OEoACgkQPJpCBAwIhbRqCwCfS6MZ0C/CgOnz8zcSaVRix3/A
nIMAmQH/1wVnJbGAJ/Lvu07DqoUxFyvq
=je5I
-----END PGP SIGNATURE-----

--nextPart2073183.1OyNNgtYSL--

From general-return-2881-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 26 03:50:39 2011
Return-Path: <general-return-2881-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49809 invoked from network); 26 Jan 2011 03:50:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Jan 2011 03:50:38 -0000
Received: (qmail 45512 invoked by uid 500); 26 Jan 2011 03:50:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45273 invoked by uid 500); 26 Jan 2011 03:50:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45265 invoked by uid 99); 26 Jan 2011 03:50:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Jan 2011 03:50:31 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of vermaabhishekp@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Jan 2011 03:50:24 +0000
Received: by wwd20 with SMTP id 20so520370wwd.29
        for <general@hadoop.apache.org>; Tue, 25 Jan 2011 19:50:03 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:date:x-google-sender-auth
         :message-id:subject:from:to:content-type;
        bh=MsZs3fNpcTLI91Jmzs45bFrFPSRR41ayUitUOVZHpeE=;
        b=jTfivHouTX8dEbjQ7uadeAYzKVuNXOvCvoaiBpZxDaI9csyPiDMxFzxlyOZff17ykO
         kIuBjZ6kvjE9+MYAPeqdOTmvKaZ+hyLAoV7+KQbWDfRsDCSslcVWJCr4LachkO1U3cDU
         KKmvdxJODfzqv6mR2mt4jGjKEUBxYt9VM63dk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        b=XxGI6lK/cjLSiGhbrkSC450S5b/wsmtbNkb2ZHbZVcDqEFb8PhE8PoEliYGpOVdtqL
         SRKsU9h7IJayTXi0iZZ7lCqU8b802PwaE0MD2UbIo3dOZ6BVquogR9s5BEdVKbq+uI/M
         K18gyweMcpuZP1tGn7QeuBRbzM5ZU/VjofN9Y=
MIME-Version: 1.0
Received: by 10.216.143.2 with SMTP id k2mr659892wej.66.1296013803704; Tue, 25
 Jan 2011 19:50:03 -0800 (PST)
Sender: vermaabhishekp@gmail.com
Received: by 10.216.246.68 with HTTP; Tue, 25 Jan 2011 19:50:03 -0800 (PST)
Date: Tue, 25 Jan 2011 21:50:03 -0600
X-Google-Sender-Auth: qd0sbrKBqPopUBScBjHXiU6OqCE
Message-ID: <AANLkTinxwQXxkbNk1Cy4T=P5pHj6mxuTrORc-7nJT279@mail.gmail.com>
Subject: Hadoop job logs for research
From: Abhishek Verma <verma7@illinois.edu>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6deddc864d12f049ab7ba22
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6deddc864d12f049ab7ba22
Content-Type: text/plain; charset=ISO-8859-1

Hi fellow Hadoop users and developers,

I am a third year PhD student at the University of Illinois and am working
on improving workload management and scheduling in Hadoop. I have tested
some of my ideas on synthetic workloads, GridMix, hadoop-examples and a few
of my own applications. I am looking for real workloads that are executed in
the industry. Specifically, I am interested in the job logs (stored by
default on the JobTracker) of real workloads. If people are concerned about
the confidentiality of the application, I would like to mention that these
logs contain very little information about the processed data or the
application itself. Anonymizing the job names (and their submission times,
etc.) would not be too much of a problem.

I would love to collaborate with folks from the industry in understanding
these workloads. I sincerely hope that the research that I am conducting
will benefit everybody.

Thanks a lot.
-Abhishek.

--0016e6deddc864d12f049ab7ba22--

From general-return-2882-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 26 07:21:16 2011
Return-Path: <general-return-2882-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 41853 invoked from network); 26 Jan 2011 07:21:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Jan 2011 07:21:16 -0000
Received: (qmail 56138 invoked by uid 500); 26 Jan 2011 07:21:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55788 invoked by uid 500); 26 Jan 2011 07:21:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55777 invoked by uid 99); 26 Jan 2011 07:21:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Jan 2011 07:21:10 +0000
X-ASF-Spam-Status: No, hits=4.7 required=10.0
	tests=FREEMAIL_FROM,FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.96 as permitted sender)
Received: from [17.148.16.96] (HELO asmtpout021.mac.com) (17.148.16.96)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Jan 2011 07:21:04 +0000
MIME-version: 1.0
Content-type: multipart/alternative;
 boundary="Boundary_(ID_y2Da1pZ/IPAr4ktkxAgzUg)"
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp021.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LFM009CECC3WTE0@asmtp021.mac.com>
 for general@hadoop.apache.org; Tue, 25 Jan 2011 23:19:16 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-26_04:2011-01-26,2011-01-26,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1101250263
Sun-Java-System-SMTP-Warning: Lines longer than SMTP allows found and wrapped.
From: Nigel Daley <ndaley@mac.com>
Subject: Re: Patch testing
Date: Tue, 25 Jan 2011 23:19:14 -0800
In-reply-to: <AANLkTikhwe_uOpnJ_v5UeObfJyD_bDy_g55EJVFWw4ro@mail.gmail.com>
To: general@hadoop.apache.org
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com>
 <20101020195420.GG2075@tp>
 <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com>
 <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
 <7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com>
 <AANLkTimLXA+CvUs-09LMN3ob6GZqmhxiyrKO7pzWEfse@mail.gmail.com>
 <4CE46119.2030509@yahoo-inc.com>
 <8617BECB-78B4-42B6-B592-D7FC1F8DA923@mac.com>
 <4CE47C88.5050203@yahoo-inc.com>
 <AANLkTinTM5h-3NupRSsFh-DnyvFYw+_OsXpASxcQUSDn@mail.gmail.com>
 <AANLkTimSOxaXZi-VLjwuTB5Ch5tRt0bKozc5L+Nx=ZrQ@mail.gmail.com>
 <AANLkTineU2v=Sx-=oC=6ROt6Os8TEbnS3L8xz1ZkoxCc@mail.gmail.com>
 <AANLkTin+oucXdCdOo7LQTtqndTf1axiT0gQntiOZ_mQJ@mail.gmail.com>
 <AANLkTinGn4Pb6_LkrY4-kK56PdHX4C1Zfu7A2PVfUoub@mail.gmail.com>
 <41FB0800-3703-49C1-8069-DEB74FFE6FAC@mac.com>
 <AANLkTik_rNLMHj3JT6Ekdop-NU_qzcB=VdsX+k4uFBtf@mail.gmail.com>
 <AANLkTin9=FPxv+0EydK4mWp_1zJbgGTddevvF9MrHW80@mail.gmail.com>
 <AANLkTiku_OvZuuOMWuXMcVf4UxB+W-SfW1WWV6rU0cXB@mail.gmail.com>
 <7B140F50-27E0-47C7-8EF8-B897D26CEE49@mac.com>
 <AANLkTi=9wEVmjZqXN+Cw2+cW3iNfuMsUJG=OeXnMfg0f@mail.gmail.com>
 <394BBDCC-9561-4E07-8EBA-AE3A92814E5A@mac.com>
 <AANLkTinKeADQRv1hXMDxqsaws+u2p=b7ntHrJGPOFSRo@mail.gmail.com>
 <F5C6F56C-74DF-43B6-9846-EE60C16E04C7@mac.com>
 <AANLkTim_o2HDjsxK36pWQn76D9FsJN97SACCHNMEiS7X@mail.gmail.com>
 <FCE82CCA-43DC-424C-95E5-B0BD18C96EED@mac.com>
 <AANLkTikhwe_uOpnJ_v5UeObfJyD_bDy_g55EJVFWw4ro@mail.gmail.com>
Message-id: <AB972BA8-7E80-4428-8934-A61F53631619@mac.com>
X-Mailer: Apple Mail (2.1082)

--Boundary_(ID_y2Da1pZ/IPAr4ktkxAgzUg)
Content-type: text/plain; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT

Started another trial run of MR precommit testing:
https://hudson.apache.org/hudson/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/17/

Let's see if 17th time is a charm...

Nige

On Jan 7, 2011, at 5:14 PM, Todd Lipcon wrote:

> On Fri, Jan 7, 2011 at 2:11 PM, Nigel Daley <ndaley@mac.com> wrote:
> 
>> Hrm, the MR precommit test I'm running has hung (been running for 14 hours
>> so far).  FWIW, 2 HDFS precommit tests are hung too.  I suspect it could be
>> the NFS mounts on the machines.  I forced a thread dump which you can see in
>> the console:
>> https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/10/console
>> 
>> 
> Strange, haven't seen a hang like that before in handleConnectionFailure. It
> should retry for 15 minutes max in that loop.
> 
> 
>> Any other ideas why these might be hanging?
>> 
>> 
> There is an HDFS bug right now that can cause hangs on some tests -
> HDFS-1529 - would appreciate if someone can take a look. But I don't think
> this is responsible for the MR hang above.
> 
> -Todd
> 
> 
>> On Jan 5, 2011, at 5:42 PM, Todd Lipcon wrote:
>> 
>>> On Wed, Jan 5, 2011 at 4:39 PM, Nigel Daley <ndaley@mac.com> wrote:
>>> 
>>>> Thanks for looking into it Todd.  Let's first see if you think it can be
>>>> fixed quickly.  Let me know.
>>>> 
>>>> 
>>> No problem, it wasn't too bad after all. Patch up on HADOOP-7087 which
>> fixes
>>> this test timeout for me.
>>> 
>>> -Todd
>>> 
>>> 
>>>> On Jan 5, 2011, at 4:33 PM, Todd Lipcon wrote:
>>>> 
>>>>> On Wed, Jan 5, 2011 at 4:19 PM, Nigel Daley <ndaley@mac.com> wrote:
>>>>> 
>>>>>> Todd, would love to get
>>>>>> https://issues.apache.org/jira/browse/MAPREDUCE-2121 fixed first
>> since
>>>>>> this is failing every night on trunk.
>>>>>> 
>>>>> 
>>>>> What if we disable that test, move that issue to 0.22 blocker, and then
>>>>> enable the test-patch? I'll also look into that one today, but if it's
>>>>> something that will take a while to fix, I don't think we should hold
>> off
>>>>> the useful testing for all the other patches.
>>>>> 
>>>>> -Todd
>>>>> 
>>>>> On Jan 5, 2011, at 2:45 PM, Todd Lipcon wrote:
>>>>>> 
>>>>>>> Hi Nigel,
>>>>>>> 
>>>>>>> MAPREDUCE-2172 has been fixed for a while. Are there any other
>>>> particular
>>>>>>> JIRAs you think need to be fixed before the MR test-patch queue gets
>>>>>>> enabled? I have a lot of outstanding patches and doing all the
>>>> test-patch
>>>>>>> turnaround manually on 3 different boxes is a real headache.
>>>>>>> 
>>>>>>> Thanks
>>>>>>> -Todd
>>>>>>> 
>>>>>>> On Tue, Dec 21, 2010 at 1:33 PM, Nigel Daley <ndaley@mac.com> wrote:
>>>>>>> 
>>>>>>>> Ok, HDFS is now enabled.  You'll see a stream of updates shortly on
>>>> the
>>>>>> ~30
>>>>>>>> Patch Available HDFS issues.
>>>>>>>> 
>>>>>>>> Nige
>>>>>>>> 
>>>>>>>> On Dec 20, 2010, at 12:42 PM, Jakob Homan wrote:
>>>>>>>> 
>>>>>>>>> I committed HDFS-1511 this morning.  We should be good to go.  I
>> can
>>>>>>>>> haz snooty robot butler?
>>>>>>>>> 
>>>>>>>>> On Fri, Dec 17, 2010 at 8:31 PM, Konstantin Boudnik <
>> cos@apache.org>
>>>>>>>> wrote:
>>>>>>>>>> Thanks Jacob. I am wasted already but I can do it on Sun, I think,
>>>>>>>>>> unless it is done earlier.
>>>>>>>>>> --
>>>>>>>>>> Take care,
>>>>>>>>>> Konstantin (Cos) Boudnik
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Fri, Dec 17, 2010 at 19:41, Jakob Homan <jghoman@gmail.com>
>>>> wrote:
>>>>>>>>>>> Ok.  I'll get a patch out for 1511 tomorrow, unless someone wants
>>>> to
>>>>>>>>>>> whip one up tonight.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley <ndaley@mac.com>
>>>> wrote:
>>>>>>>>>>>> I agree with Cos on fixing HDFS-1511 first. Once that is done
>> I'll
>>>>>>>> enable hdfs patch testing.
>>>>>>>>>>>> 
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> Nige
>>>>>>>>>>>> 
>>>>>>>>>>>> Sent from my iPhone4
>>>>>>>>>>>> 
>>>>>>>>>>>> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <cos@apache.org
>>> 
>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> One more issue needs to be addressed before test-patch is
>> turned
>>>> on
>>>>>>>> HDFS is
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-1511
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Take care,
>>>>>>>>>>>>> Konstantin (Cos) Boudnik
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <
>>>> cos@apache.org>
>>>>>>>> wrote:
>>>>>>>>>>>>>> Considering that because of these 4 faulty cases every patch
>>>> will
>>>>>> be
>>>>>>>>>>>>>> -1'ed a patch author will still have to look at it and make a
>>>>>>>> comment
>>>>>>>>>>>>>> why this particular -1 isn't valid. Lesser work, perhaps, but
>>>>>>>> messier
>>>>>>>>>>>>>> IMO. I'm not blocking it - I just feel like there's a better
>>>> way.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Take care,
>>>>>>>>>>>>>> Konstantin (Cos) Boudnik
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jghoman@gmail.com
>>> 
>>>>>>>> wrote:
>>>>>>>>>>>>>>>> If HDFS is added to the test-patch queue right now we get
>>>>>>>>>>>>>>>> nothing but dozens of -1'ed patches.
>>>>>>>>>>>>>>> There aren't dozens of patches being submitted currently.
>> The
>>>> -1
>>>>>>>>>>>>>>> isn't the important thing, it's the grunt work of actually
>>>>>> running
>>>>>>>>>>>>>>> (and waiting) for the tests, test-patch, etc. that Hudson
>> does
>>>> so
>>>>>>>> that
>>>>>>>>>>>>>>> the developer doesn't have to.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <
>>>>>>>> dhruba@gmail.com> wrote:
>>>>>>>>>>>>>>>> +1, thanks for doing this.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <
>>>> jghoman@gmail.com
>>>>>>> 
>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> So, with test-patch updated to show the failing tests,
>> saving
>>>>>> the
>>>>>>>>>>>>>>>>> developers the need to go and verify that the failed tests
>>>> are
>>>>>>>> all
>>>>>>>>>>>>>>>>> known, how do people feel about turning on test-patch again
>>>> for
>>>>>>>> HDFS
>>>>>>>>>>>>>>>>> and mapred?  I think it'll help prevent any more tests from
>>>>>>>> entering
>>>>>>>>>>>>>>>>> the "yeah, we know" category.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> jg
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <
>>>>>>>> jhoman@yahoo-inc.com> wrote:
>>>>>>>>>>>>>>>>>> True, each patch would get a -1 and the failing tests
>> would
>>>>>> need
>>>>>>>> to be
>>>>>>>>>>>>>>>>>> verified as those known bad (BTW, it would be great if
>>>> Hudson
>>>>>>>> could list
>>>>>>>>>>>>>>>>>> which tests failed in the message it posts to JIRA).  But
>>>>>> that's
>>>>>>>> still
>>>>>>>>>>>>>>>>> quite
>>>>>>>>>>>>>>>>>> a bit less error-prone work than if the developer runs the
>>>>>> tests
>>>>>>>> and
>>>>>>>>>>>>>>>>>> test-patch themselves.  Also, with 22 being cut, there are
>> a
>>>>>> lot
>>>>>>>> of
>>>>>>>>>>>>>>>>> patches
>>>>>>>>>>>>>>>>>> up in the air and several developers are juggling multiple
>>>>>>>> patches.  The
>>>>>>>>>>>>>>>>>> more automation we can have, even if it's not perfect,
>> will
>>>>>>>> decrease
>>>>>>>>>>>>>>>>> errors
>>>>>>>>>>>>>>>>>> we may make.
>>>>>>>>>>>>>>>>>> -jg
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Nigel Daley wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> It's also ready to run on MapReduce and HDFS but we
>> won't
>>>>>>>> turn it on
>>>>>>>>>>>>>>>>>>>>> until these projects build and test cleanly.  Looks
>> like
>>>>>> both
>>>>>>>> these
>>>>>>>>>>>>>>>>> projects
>>>>>>>>>>>>>>>>>>>>> currently have test failures.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Assuming the projects are compiling and building, is
>> there
>>>> a
>>>>>>>> reason to
>>>>>>>>>>>>>>>>>>>> not turn it on despite the test failures? Hudson is
>>>>>> invaluable
>>>>>>>> to
>>>>>>>>>>>>>>>>> developers
>>>>>>>>>>>>>>>>>>>> who then don't have to run the tests and test-patch
>>>>>>>> themselves.  We
>>>>>>>>>>>>>>>>> didn't
>>>>>>>>>>>>>>>>>>>> turn Hudson off when it was working previously and there
>>>>>> were
>>>>>>>> known
>>>>>>>>>>>>>>>>>>>> failures.  I think one of the reasons we have more
>> failing
>>>>>>>> tests now is
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> higher cost of doing Hudson's work (not a great excuse I
>>>>>>>> know).  This
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> particularly true now because several of the failing
>> tests
>>>>>>>> involve
>>>>>>>>>>>>>>>>> tests
>>>>>>>>>>>>>>>>>>>> timing out, making the whole testing regime even longer.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Every single patch would get a -1 and need investigation.
>>>>>>>> Currently,
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>> would be about 83 investigations between MR and HDFS
>> issues
>>>>>>>> that are in
>>>>>>>>>>>>>>>>>>> patch available state.  Shouldn't we focus on getting
>> these
>>>>>>>> tests fixed
>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>> removed/?  Also, I need to get MAPREDUCE-2172 fixed
>>>> (applies
>>>>>> to
>>>>>>>> HDFS as
>>>>>>>>>>>>>>>>>>> well) before I turn this on.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>> Nige
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Connect to me at http://www.facebook.com/dhruba
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> Todd Lipcon
>>>>>>> Software Engineer, Cloudera
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Todd Lipcon
>>>>> Software Engineer, Cloudera
>>>> 
>>>> 
>>> 
>>> 
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>> 
>> 
> 
> 
> -- 
> Todd Lipcon
> Software Engineer, Cloudera


--Boundary_(ID_y2Da1pZ/IPAr4ktkxAgzUg)--

From general-return-2883-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 26 11:24:42 2011
Return-Path: <general-return-2883-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44991 invoked from network); 26 Jan 2011 11:24:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Jan 2011 11:24:42 -0000
Received: (qmail 27294 invoked by uid 500); 26 Jan 2011 11:24:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26940 invoked by uid 500); 26 Jan 2011 11:24:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26932 invoked by uid 99); 26 Jan 2011 11:24:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Jan 2011 11:24:36 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of binhara@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Jan 2011 11:24:31 +0000
Received: by wwd20 with SMTP id 20so794063wwd.29
        for <general@hadoop.apache.org>; Wed, 26 Jan 2011 03:24:10 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=7mGG/tmlElPU2xhxDl/gTRf2UMu16j4OBbreLNSRVok=;
        b=CEZcTviCUoYY4/lm1ZF0lpyNTqoM4tOodwxJSe2epp1glGRD9DYYk3PcgriJiJh8+n
         0vK2TAuPHEIrXCBu0XokKcPE3dH4HFFl46iweFv4RMQV5y8tCWcrH3yH1O3CbtrtViyV
         uMynARPRre/975DossO+U4rqoU6fYDykoxVmU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=mPtFNFuwZR/bbK8nen0nAp/8UcU2QtB5JMueqlM2XlxSVZj9wIeuxiOiijSnnuINQp
         PVGSyFOix1ARMeWm5HpiVLKr/YeR9aaoqhKqdNeKKCVUnkE6ChO5yI33pBP+tZbQpPqI
         WYIU/6zvDqSR5E8rZnCjPNqCjelZ2nCJ3AboA=
MIME-Version: 1.0
Received: by 10.216.59.193 with SMTP id s43mr4925008wec.42.1296041050031; Wed,
 26 Jan 2011 03:24:10 -0800 (PST)
Received: by 10.216.17.146 with HTTP; Wed, 26 Jan 2011 03:24:10 -0800 (PST)
Date: Wed, 26 Jan 2011 09:24:10 -0200
Message-ID: <AANLkTimVtXOoiDh_ADcZbtT+Mq_S+fLyrGTO9js8zNxf@mail.gmail.com>
Subject: Hadoop datanote question
From: Alessandro Binhara <binhara@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0ce0d74066c86b049abe12a7

--000e0ce0d74066c86b049abe12a7
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hello..

I'm confused about the architecture of HDFS
I have a client java write a files on a master node. It=B4s work perfect!..
well i see a HDFS architecture
http://hadoop.apache.org/common/docs/r0.20.0/images/hdfsarchitecture.gif
some client access a data note..

i try use my java client write direct on slave node. I change IP on client
configuration:
                Configuration conf =3D new Configuration();
conf.set("fs.default.name","hdfs://192.168.254.25:54310/"); //slave ip

I got this error. .
26/01/2011 09:19:52 org.apache.hadoop.ipc.Client$Connection
handleConnectionFailure
It dont work ..

In architecture picture show me that all datanode can conectable. That=B4s
correct ?

How it work ?
I can only conect to write on HDFS on master node ??

thank s

--000e0ce0d74066c86b049abe12a7--

From general-return-2884-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 26 18:06:39 2011
Return-Path: <general-return-2884-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33179 invoked from network); 26 Jan 2011 18:06:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Jan 2011 18:06:39 -0000
Received: (qmail 54711 invoked by uid 500); 26 Jan 2011 18:06:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54273 invoked by uid 500); 26 Jan 2011 18:06:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54265 invoked by uid 99); 26 Jan 2011 18:06:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Jan 2011 18:06:32 +0000
X-ASF-Spam-Status: No, hits=4.7 required=10.0
	tests=FREEMAIL_FROM,FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.96 as permitted sender)
Received: from [17.148.16.96] (HELO asmtpout021.mac.com) (17.148.16.96)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Jan 2011 18:06:21 +0000
MIME-version: 1.0
Content-type: multipart/alternative;
 boundary="Boundary_(ID_HEiLJUPsR4bQt8nzfCXyqQ)"
Received: from [10.0.1.13] ([71.198.192.174])
 by asmtp021.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LFN00IMP69JC9I0@asmtp021.mac.com>
 for general@hadoop.apache.org; Wed, 26 Jan 2011 10:05:45 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-26_08:2011-01-26,2011-01-26,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1101260111
Sun-Java-System-SMTP-Warning: Lines longer than SMTP allows found and wrapped.
From: Nigel Daley <ndaley@mac.com>
Subject: Re: Patch testing
Date: Wed, 26 Jan 2011 10:05:42 -0800
In-reply-to: <AB972BA8-7E80-4428-8934-A61F53631619@mac.com>
To: general@hadoop.apache.org
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com>
 <20101020195420.GG2075@tp>
 <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com>
 <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
 <7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com>
 <AANLkTimLXA+CvUs-09LMN3ob6GZqmhxiyrKO7pzWEfse@mail.gmail.com>
 <4CE46119.2030509@yahoo-inc.com>
 <8617BECB-78B4-42B6-B592-D7FC1F8DA923@mac.com>
 <4CE47C88.5050203@yahoo-inc.com>
 <AANLkTinTM5h-3NupRSsFh-DnyvFYw+_OsXpASxcQUSDn@mail.gmail.com>
 <AANLkTimSOxaXZi-VLjwuTB5Ch5tRt0bKozc5L+Nx=ZrQ@mail.gmail.com>
 <AANLkTineU2v=Sx-=oC=6ROt6Os8TEbnS3L8xz1ZkoxCc@mail.gmail.com>
 <AANLkTin+oucXdCdOo7LQTtqndTf1axiT0gQntiOZ_mQJ@mail.gmail.com>
 <AANLkTinGn4Pb6_LkrY4-kK56PdHX4C1Zfu7A2PVfUoub@mail.gmail.com>
 <41FB0800-3703-49C1-8069-DEB74FFE6FAC@mac.com>
 <AANLkTik_rNLMHj3JT6Ekdop-NU_qzcB=VdsX+k4uFBtf@mail.gmail.com>
 <AANLkTin9=FPxv+0EydK4mWp_1zJbgGTddevvF9MrHW80@mail.gmail.com>
 <AANLkTiku_OvZuuOMWuXMcVf4UxB+W-SfW1WWV6rU0cXB@mail.gmail.com>
 <7B140F50-27E0-47C7-8EF8-B897D26CEE49@mac.com>
 <AANLkTi=9wEVmjZqXN+Cw2+cW3iNfuMsUJG=OeXnMfg0f@mail.gmail.com>
 <394BBDCC-9561-4E07-8EBA-AE3A92814E5A@mac.com>
 <AANLkTinKeADQRv1hXMDxqsaws+u2p=b7ntHrJGPOFSRo@mail.gmail.com>
 <F5C6F56C-74DF-43B6-9846-EE60C16E04C7@mac.com>
 <AANLkTim_o2HDjsxK36pWQn76D9FsJN97SACCHNMEiS7X@mail.gmail.com>
 <FCE82CCA-43DC-424C-95E5-B0BD18C96EED@mac.com>
 <AANLkTikhwe_uOpnJ_v5UeObfJyD_bDy_g55EJVFWw4ro@mail.gmail.com>
 <AB972BA8-7E80-4428-8934-A61F53631619@mac.com>
Message-id: <804715D0-C9D4-41F4-B209-4A022053DEBD@mac.com>
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

--Boundary_(ID_HEiLJUPsR4bQt8nzfCXyqQ)
Content-type: text/plain; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT

raid (contrib) test hanging: TestBlockFixer

I forced 2 thread dumps.  Both hung in the same place.  Filed https://issues.apache.org/jira/browse/MAPREDUCE-2283  This is a blocker for turning on MR precommit.

Cheers,
Nige

On Jan 25, 2011, at 11:19 PM, Nigel Daley wrote:

> Started another trial run of MR precommit testing:
> https://hudson.apache.org/hudson/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/17/
> 
> Let's see if 17th time is a charm...
> 
> Nige
> 
> On Jan 7, 2011, at 5:14 PM, Todd Lipcon wrote:
> 
>> On Fri, Jan 7, 2011 at 2:11 PM, Nigel Daley <ndaley@mac.com> wrote:
>> 
>>> Hrm, the MR precommit test I'm running has hung (been running for 14 hours
>>> so far).  FWIW, 2 HDFS precommit tests are hung too.  I suspect it could be
>>> the NFS mounts on the machines.  I forced a thread dump which you can see in
>>> the console:
>>> https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/10/console
>>> 
>>> 
>> Strange, haven't seen a hang like that before in handleConnectionFailure. It
>> should retry for 15 minutes max in that loop.
>> 
>> 
>>> Any other ideas why these might be hanging?
>>> 
>>> 
>> There is an HDFS bug right now that can cause hangs on some tests -
>> HDFS-1529 - would appreciate if someone can take a look. But I don't think
>> this is responsible for the MR hang above.
>> 
>> -Todd
>> 
>> 
>>> On Jan 5, 2011, at 5:42 PM, Todd Lipcon wrote:
>>> 
>>>> On Wed, Jan 5, 2011 at 4:39 PM, Nigel Daley <ndaley@mac.com> wrote:
>>>> 
>>>>> Thanks for looking into it Todd.  Let's first see if you think it can be
>>>>> fixed quickly.  Let me know.
>>>>> 
>>>>> 
>>>> No problem, it wasn't too bad after all. Patch up on HADOOP-7087 which
>>> fixes
>>>> this test timeout for me.
>>>> 
>>>> -Todd
>>>> 
>>>> 
>>>>> On Jan 5, 2011, at 4:33 PM, Todd Lipcon wrote:
>>>>> 
>>>>>> On Wed, Jan 5, 2011 at 4:19 PM, Nigel Daley <ndaley@mac.com> wrote:
>>>>>> 
>>>>>>> Todd, would love to get
>>>>>>> https://issues.apache.org/jira/browse/MAPREDUCE-2121 fixed first
>>> since
>>>>>>> this is failing every night on trunk.
>>>>>>> 
>>>>>> 
>>>>>> What if we disable that test, move that issue to 0.22 blocker, and then
>>>>>> enable the test-patch? I'll also look into that one today, but if it's
>>>>>> something that will take a while to fix, I don't think we should hold
>>> off
>>>>>> the useful testing for all the other patches.
>>>>>> 
>>>>>> -Todd
>>>>>> 
>>>>>> On Jan 5, 2011, at 2:45 PM, Todd Lipcon wrote:
>>>>>>> 
>>>>>>>> Hi Nigel,
>>>>>>>> 
>>>>>>>> MAPREDUCE-2172 has been fixed for a while. Are there any other
>>>>> particular
>>>>>>>> JIRAs you think need to be fixed before the MR test-patch queue gets
>>>>>>>> enabled? I have a lot of outstanding patches and doing all the
>>>>> test-patch
>>>>>>>> turnaround manually on 3 different boxes is a real headache.
>>>>>>>> 
>>>>>>>> Thanks
>>>>>>>> -Todd
>>>>>>>> 
>>>>>>>> On Tue, Dec 21, 2010 at 1:33 PM, Nigel Daley <ndaley@mac.com> wrote:
>>>>>>>> 
>>>>>>>>> Ok, HDFS is now enabled.  You'll see a stream of updates shortly on
>>>>> the
>>>>>>> ~30
>>>>>>>>> Patch Available HDFS issues.
>>>>>>>>> 
>>>>>>>>> Nige
>>>>>>>>> 
>>>>>>>>> On Dec 20, 2010, at 12:42 PM, Jakob Homan wrote:
>>>>>>>>> 
>>>>>>>>>> I committed HDFS-1511 this morning.  We should be good to go.  I
>>> can
>>>>>>>>>> haz snooty robot butler?
>>>>>>>>>> 
>>>>>>>>>> On Fri, Dec 17, 2010 at 8:31 PM, Konstantin Boudnik <
>>> cos@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>>> Thanks Jacob. I am wasted already but I can do it on Sun, I think,
>>>>>>>>>>> unless it is done earlier.
>>>>>>>>>>> --
>>>>>>>>>>> Take care,
>>>>>>>>>>> Konstantin (Cos) Boudnik
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Fri, Dec 17, 2010 at 19:41, Jakob Homan <jghoman@gmail.com>
>>>>> wrote:
>>>>>>>>>>>> Ok.  I'll get a patch out for 1511 tomorrow, unless someone wants
>>>>> to
>>>>>>>>>>>> whip one up tonight.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley <ndaley@mac.com>
>>>>> wrote:
>>>>>>>>>>>>> I agree with Cos on fixing HDFS-1511 first. Once that is done
>>> I'll
>>>>>>>>> enable hdfs patch testing.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Nige
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Sent from my iPhone4
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <cos@apache.org
>>>> 
>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> One more issue needs to be addressed before test-patch is
>>> turned
>>>>> on
>>>>>>>>> HDFS is
>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-1511
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Take care,
>>>>>>>>>>>>>> Konstantin (Cos) Boudnik
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <
>>>>> cos@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>> Considering that because of these 4 faulty cases every patch
>>>>> will
>>>>>>> be
>>>>>>>>>>>>>>> -1'ed a patch author will still have to look at it and make a
>>>>>>>>> comment
>>>>>>>>>>>>>>> why this particular -1 isn't valid. Lesser work, perhaps, but
>>>>>>>>> messier
>>>>>>>>>>>>>>> IMO. I'm not blocking it - I just feel like there's a better
>>>>> way.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Take care,
>>>>>>>>>>>>>>> Konstantin (Cos) Boudnik
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <jghoman@gmail.com
>>>> 
>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> If HDFS is added to the test-patch queue right now we get
>>>>>>>>>>>>>>>>> nothing but dozens of -1'ed patches.
>>>>>>>>>>>>>>>> There aren't dozens of patches being submitted currently.
>>> The
>>>>> -1
>>>>>>>>>>>>>>>> isn't the important thing, it's the grunt work of actually
>>>>>>> running
>>>>>>>>>>>>>>>> (and waiting) for the tests, test-patch, etc. that Hudson
>>> does
>>>>> so
>>>>>>>>> that
>>>>>>>>>>>>>>>> the developer doesn't have to.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <
>>>>>>>>> dhruba@gmail.com> wrote:
>>>>>>>>>>>>>>>>> +1, thanks for doing this.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <
>>>>> jghoman@gmail.com
>>>>>>>> 
>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> So, with test-patch updated to show the failing tests,
>>> saving
>>>>>>> the
>>>>>>>>>>>>>>>>>> developers the need to go and verify that the failed tests
>>>>> are
>>>>>>>>> all
>>>>>>>>>>>>>>>>>> known, how do people feel about turning on test-patch again
>>>>> for
>>>>>>>>> HDFS
>>>>>>>>>>>>>>>>>> and mapred?  I think it'll help prevent any more tests from
>>>>>>>>> entering
>>>>>>>>>>>>>>>>>> the "yeah, we know" category.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> jg
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <
>>>>>>>>> jhoman@yahoo-inc.com> wrote:
>>>>>>>>>>>>>>>>>>> True, each patch would get a -1 and the failing tests
>>> would
>>>>>>> need
>>>>>>>>> to be
>>>>>>>>>>>>>>>>>>> verified as those known bad (BTW, it would be great if
>>>>> Hudson
>>>>>>>>> could list
>>>>>>>>>>>>>>>>>>> which tests failed in the message it posts to JIRA).  But
>>>>>>> that's
>>>>>>>>> still
>>>>>>>>>>>>>>>>>> quite
>>>>>>>>>>>>>>>>>>> a bit less error-prone work than if the developer runs the
>>>>>>> tests
>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> test-patch themselves.  Also, with 22 being cut, there are
>>> a
>>>>>>> lot
>>>>>>>>> of
>>>>>>>>>>>>>>>>>> patches
>>>>>>>>>>>>>>>>>>> up in the air and several developers are juggling multiple
>>>>>>>>> patches.  The
>>>>>>>>>>>>>>>>>>> more automation we can have, even if it's not perfect,
>>> will
>>>>>>>>> decrease
>>>>>>>>>>>>>>>>>> errors
>>>>>>>>>>>>>>>>>>> we may make.
>>>>>>>>>>>>>>>>>>> -jg
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Nigel Daley wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> It's also ready to run on MapReduce and HDFS but we
>>> won't
>>>>>>>>> turn it on
>>>>>>>>>>>>>>>>>>>>>> until these projects build and test cleanly.  Looks
>>> like
>>>>>>> both
>>>>>>>>> these
>>>>>>>>>>>>>>>>>> projects
>>>>>>>>>>>>>>>>>>>>>> currently have test failures.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Assuming the projects are compiling and building, is
>>> there
>>>>> a
>>>>>>>>> reason to
>>>>>>>>>>>>>>>>>>>>> not turn it on despite the test failures? Hudson is
>>>>>>> invaluable
>>>>>>>>> to
>>>>>>>>>>>>>>>>>> developers
>>>>>>>>>>>>>>>>>>>>> who then don't have to run the tests and test-patch
>>>>>>>>> themselves.  We
>>>>>>>>>>>>>>>>>> didn't
>>>>>>>>>>>>>>>>>>>>> turn Hudson off when it was working previously and there
>>>>>>> were
>>>>>>>>> known
>>>>>>>>>>>>>>>>>>>>> failures.  I think one of the reasons we have more
>>> failing
>>>>>>>>> tests now is
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> higher cost of doing Hudson's work (not a great excuse I
>>>>>>>>> know).  This
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> particularly true now because several of the failing
>>> tests
>>>>>>>>> involve
>>>>>>>>>>>>>>>>>> tests
>>>>>>>>>>>>>>>>>>>>> timing out, making the whole testing regime even longer.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Every single patch would get a -1 and need investigation.
>>>>>>>>> Currently,
>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> would be about 83 investigations between MR and HDFS
>>> issues
>>>>>>>>> that are in
>>>>>>>>>>>>>>>>>>>> patch available state.  Shouldn't we focus on getting
>>> these
>>>>>>>>> tests fixed
>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>> removed/?  Also, I need to get MAPREDUCE-2172 fixed
>>>>> (applies
>>>>>>> to
>>>>>>>>> HDFS as
>>>>>>>>>>>>>>>>>>>> well) before I turn this on.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>> Nige
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Connect to me at http://www.facebook.com/dhruba
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> Todd Lipcon
>>>>>>>> Software Engineer, Cloudera
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Todd Lipcon
>>>>>> Software Engineer, Cloudera
>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> --
>>>> Todd Lipcon
>>>> Software Engineer, Cloudera
>>> 
>>> 
>> 
>> 
>> -- 
>> Todd Lipcon
>> Software Engineer, Cloudera
> 


--Boundary_(ID_HEiLJUPsR4bQt8nzfCXyqQ)--

From general-return-2885-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 26 18:33:08 2011
Return-Path: <general-return-2885-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 48912 invoked from network); 26 Jan 2011 18:33:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Jan 2011 18:33:08 -0000
Received: (qmail 3991 invoked by uid 500); 26 Jan 2011 18:33:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3907 invoked by uid 500); 26 Jan 2011 18:33:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3899 invoked by uid 99); 26 Jan 2011 18:33:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Jan 2011 18:33:05 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates 209.85.210.176 as permitted sender)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Jan 2011 18:33:01 +0000
Received: by iyb26 with SMTP id 26so702782iyb.35
        for <general@hadoop.apache.org>; Wed, 26 Jan 2011 10:32:40 -0800 (PST)
Received: by 10.231.34.9 with SMTP id j9mr8537274ibd.137.1296066760264; Wed,
 26 Jan 2011 10:32:40 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.153.19 with HTTP; Wed, 26 Jan 2011 10:32:20 -0800 (PST)
In-Reply-To: <804715D0-C9D4-41F4-B209-4A022053DEBD@mac.com>
References: <B118A94D-1254-4909-9C75-116E3E2A99FE@mac.com> <20101020195420.GG2075@tp>
 <53F363B9-E865-4E63-907A-7F341A246235@yahoo-inc.com> <4D646D78-621B-4C50-9420-6B5EC7F49B54@mac.com>
 <7B1CE23C-E15A-4BA5-8D96-62163A56E23C@mac.com> <AANLkTimLXA+CvUs-09LMN3ob6GZqmhxiyrKO7pzWEfse@mail.gmail.com>
 <4CE46119.2030509@yahoo-inc.com> <8617BECB-78B4-42B6-B592-D7FC1F8DA923@mac.com>
 <4CE47C88.5050203@yahoo-inc.com> <AANLkTinTM5h-3NupRSsFh-DnyvFYw+_OsXpASxcQUSDn@mail.gmail.com>
 <AANLkTimSOxaXZi-VLjwuTB5Ch5tRt0bKozc5L+Nx=ZrQ@mail.gmail.com>
 <AANLkTineU2v=Sx-=oC=6ROt6Os8TEbnS3L8xz1ZkoxCc@mail.gmail.com>
 <AANLkTin+oucXdCdOo7LQTtqndTf1axiT0gQntiOZ_mQJ@mail.gmail.com>
 <AANLkTinGn4Pb6_LkrY4-kK56PdHX4C1Zfu7A2PVfUoub@mail.gmail.com>
 <41FB0800-3703-49C1-8069-DEB74FFE6FAC@mac.com> <AANLkTik_rNLMHj3JT6Ekdop-NU_qzcB=VdsX+k4uFBtf@mail.gmail.com>
 <AANLkTin9=FPxv+0EydK4mWp_1zJbgGTddevvF9MrHW80@mail.gmail.com>
 <AANLkTiku_OvZuuOMWuXMcVf4UxB+W-SfW1WWV6rU0cXB@mail.gmail.com>
 <7B140F50-27E0-47C7-8EF8-B897D26CEE49@mac.com> <AANLkTi=9wEVmjZqXN+Cw2+cW3iNfuMsUJG=OeXnMfg0f@mail.gmail.com>
 <394BBDCC-9561-4E07-8EBA-AE3A92814E5A@mac.com> <AANLkTinKeADQRv1hXMDxqsaws+u2p=b7ntHrJGPOFSRo@mail.gmail.com>
 <F5C6F56C-74DF-43B6-9846-EE60C16E04C7@mac.com> <AANLkTim_o2HDjsxK36pWQn76D9FsJN97SACCHNMEiS7X@mail.gmail.com>
 <FCE82CCA-43DC-424C-95E5-B0BD18C96EED@mac.com> <AANLkTikhwe_uOpnJ_v5UeObfJyD_bDy_g55EJVFWw4ro@mail.gmail.com>
 <AB972BA8-7E80-4428-8934-A61F53631619@mac.com> <804715D0-C9D4-41F4-B209-4A022053DEBD@mac.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Wed, 26 Jan 2011 10:32:20 -0800
Message-ID: <AANLkTinHw=KoN2mDVbo9Unz6MfrgVGpTx+u_tPNSBFVH@mail.gmail.com>
Subject: Re: Patch testing
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00221534c927d9cce9049ac40e7f

--00221534c927d9cce9049ac40e7f
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Jan 26, 2011 at 10:05 AM, Nigel Daley <ndaley@mac.com> wrote:

> raid (contrib) test hanging: TestBlockFixer
>
> I forced 2 thread dumps.  Both hung in the same place.  Filed
> https://issues.apache.org/jira/browse/MAPREDUCE-2283  This is a blocker
> for turning on MR precommit.
>

Since this is contrib, I'd like to suggest just disabling this test
temporarily. We can re-enable it once it's fixed.

Not having MR pre-commit working has been pretty painful.

-Todd


> On Jan 25, 2011, at 11:19 PM, Nigel Daley wrote:
>
> > Started another trial run of MR precommit testing:
> >
> https://hudson.apache.org/hudson/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/17/
> >
> > Let's see if 17th time is a charm...
> >
> > Nige
> >
> > On Jan 7, 2011, at 5:14 PM, Todd Lipcon wrote:
> >
> >> On Fri, Jan 7, 2011 at 2:11 PM, Nigel Daley <ndaley@mac.com> wrote:
> >>
> >>> Hrm, the MR precommit test I'm running has hung (been running for 14
> hours
> >>> so far).  FWIW, 2 HDFS precommit tests are hung too.  I suspect it
> could be
> >>> the NFS mounts on the machines.  I forced a thread dump which you can
> see in
> >>> the console:
> >>>
> https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/10/console
> >>>
> >>>
> >> Strange, haven't seen a hang like that before in
> handleConnectionFailure. It
> >> should retry for 15 minutes max in that loop.
> >>
> >>
> >>> Any other ideas why these might be hanging?
> >>>
> >>>
> >> There is an HDFS bug right now that can cause hangs on some tests -
> >> HDFS-1529 - would appreciate if someone can take a look. But I don't
> think
> >> this is responsible for the MR hang above.
> >>
> >> -Todd
> >>
> >>
> >>> On Jan 5, 2011, at 5:42 PM, Todd Lipcon wrote:
> >>>
> >>>> On Wed, Jan 5, 2011 at 4:39 PM, Nigel Daley <ndaley@mac.com> wrote:
> >>>>
> >>>>> Thanks for looking into it Todd.  Let's first see if you think it can
> be
> >>>>> fixed quickly.  Let me know.
> >>>>>
> >>>>>
> >>>> No problem, it wasn't too bad after all. Patch up on HADOOP-7087 which
> >>> fixes
> >>>> this test timeout for me.
> >>>>
> >>>> -Todd
> >>>>
> >>>>
> >>>>> On Jan 5, 2011, at 4:33 PM, Todd Lipcon wrote:
> >>>>>
> >>>>>> On Wed, Jan 5, 2011 at 4:19 PM, Nigel Daley <ndaley@mac.com> wrote:
> >>>>>>
> >>>>>>> Todd, would love to get
> >>>>>>> https://issues.apache.org/jira/browse/MAPREDUCE-2121 fixed first
> >>> since
> >>>>>>> this is failing every night on trunk.
> >>>>>>>
> >>>>>>
> >>>>>> What if we disable that test, move that issue to 0.22 blocker, and
> then
> >>>>>> enable the test-patch? I'll also look into that one today, but if
> it's
> >>>>>> something that will take a while to fix, I don't think we should
> hold
> >>> off
> >>>>>> the useful testing for all the other patches.
> >>>>>>
> >>>>>> -Todd
> >>>>>>
> >>>>>> On Jan 5, 2011, at 2:45 PM, Todd Lipcon wrote:
> >>>>>>>
> >>>>>>>> Hi Nigel,
> >>>>>>>>
> >>>>>>>> MAPREDUCE-2172 has been fixed for a while. Are there any other
> >>>>> particular
> >>>>>>>> JIRAs you think need to be fixed before the MR test-patch queue
> gets
> >>>>>>>> enabled? I have a lot of outstanding patches and doing all the
> >>>>> test-patch
> >>>>>>>> turnaround manually on 3 different boxes is a real headache.
> >>>>>>>>
> >>>>>>>> Thanks
> >>>>>>>> -Todd
> >>>>>>>>
> >>>>>>>> On Tue, Dec 21, 2010 at 1:33 PM, Nigel Daley <ndaley@mac.com>
> wrote:
> >>>>>>>>
> >>>>>>>>> Ok, HDFS is now enabled.  You'll see a stream of updates shortly
> on
> >>>>> the
> >>>>>>> ~30
> >>>>>>>>> Patch Available HDFS issues.
> >>>>>>>>>
> >>>>>>>>> Nige
> >>>>>>>>>
> >>>>>>>>> On Dec 20, 2010, at 12:42 PM, Jakob Homan wrote:
> >>>>>>>>>
> >>>>>>>>>> I committed HDFS-1511 this morning.  We should be good to go.  I
> >>> can
> >>>>>>>>>> haz snooty robot butler?
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Dec 17, 2010 at 8:31 PM, Konstantin Boudnik <
> >>> cos@apache.org>
> >>>>>>>>> wrote:
> >>>>>>>>>>> Thanks Jacob. I am wasted already but I can do it on Sun, I
> think,
> >>>>>>>>>>> unless it is done earlier.
> >>>>>>>>>>> --
> >>>>>>>>>>> Take care,
> >>>>>>>>>>> Konstantin (Cos) Boudnik
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, Dec 17, 2010 at 19:41, Jakob Homan <jghoman@gmail.com>
> >>>>> wrote:
> >>>>>>>>>>>> Ok.  I'll get a patch out for 1511 tomorrow, unless someone
> wants
> >>>>> to
> >>>>>>>>>>>> whip one up tonight.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Dec 17, 2010 at 7:22 PM, Nigel Daley <ndaley@mac.com>
> >>>>> wrote:
> >>>>>>>>>>>>> I agree with Cos on fixing HDFS-1511 first. Once that is done
> >>> I'll
> >>>>>>>>> enable hdfs patch testing.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>> Nige
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Sent from my iPhone4
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Dec 17, 2010, at 7:01 PM, Konstantin Boudnik <
> cos@apache.org
> >>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> One more issue needs to be addressed before test-patch is
> >>> turned
> >>>>> on
> >>>>>>>>> HDFS is
> >>>>>>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-1511
> >>>>>>>>>>>>>> --
> >>>>>>>>>>>>>> Take care,
> >>>>>>>>>>>>>> Konstantin (Cos) Boudnik
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 16:17, Konstantin Boudnik <
> >>>>> cos@apache.org>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>> Considering that because of these 4 faulty cases every
> patch
> >>>>> will
> >>>>>>> be
> >>>>>>>>>>>>>>> -1'ed a patch author will still have to look at it and make
> a
> >>>>>>>>> comment
> >>>>>>>>>>>>>>> why this particular -1 isn't valid. Lesser work, perhaps,
> but
> >>>>>>>>> messier
> >>>>>>>>>>>>>>> IMO. I'm not blocking it - I just feel like there's a
> better
> >>>>> way.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>> Take care,
> >>>>>>>>>>>>>>> Konstantin (Cos) Boudnik
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 15:55, Jakob Homan <
> jghoman@gmail.com
> >>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>> If HDFS is added to the test-patch queue right now we get
> >>>>>>>>>>>>>>>>> nothing but dozens of -1'ed patches.
> >>>>>>>>>>>>>>>> There aren't dozens of patches being submitted currently.
> >>> The
> >>>>> -1
> >>>>>>>>>>>>>>>> isn't the important thing, it's the grunt work of actually
> >>>>>>> running
> >>>>>>>>>>>>>>>> (and waiting) for the tests, test-patch, etc. that Hudson
> >>> does
> >>>>> so
> >>>>>>>>> that
> >>>>>>>>>>>>>>>> the developer doesn't have to.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 3:48 PM, Dhruba Borthakur <
> >>>>>>>>> dhruba@gmail.com> wrote:
> >>>>>>>>>>>>>>>>> +1, thanks for doing this.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Fri, Dec 17, 2010 at 3:19 PM, Jakob Homan <
> >>>>> jghoman@gmail.com
> >>>>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> So, with test-patch updated to show the failing tests,
> >>> saving
> >>>>>>> the
> >>>>>>>>>>>>>>>>>> developers the need to go and verify that the failed
> tests
> >>>>> are
> >>>>>>>>> all
> >>>>>>>>>>>>>>>>>> known, how do people feel about turning on test-patch
> again
> >>>>> for
> >>>>>>>>> HDFS
> >>>>>>>>>>>>>>>>>> and mapred?  I think it'll help prevent any more tests
> from
> >>>>>>>>> entering
> >>>>>>>>>>>>>>>>>> the "yeah, we know" category.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>> jg
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Wed, Nov 17, 2010 at 5:08 PM, Jakob Homan <
> >>>>>>>>> jhoman@yahoo-inc.com> wrote:
> >>>>>>>>>>>>>>>>>>> True, each patch would get a -1 and the failing tests
> >>> would
> >>>>>>> need
> >>>>>>>>> to be
> >>>>>>>>>>>>>>>>>>> verified as those known bad (BTW, it would be great if
> >>>>> Hudson
> >>>>>>>>> could list
> >>>>>>>>>>>>>>>>>>> which tests failed in the message it posts to JIRA).
>  But
> >>>>>>> that's
> >>>>>>>>> still
> >>>>>>>>>>>>>>>>>> quite
> >>>>>>>>>>>>>>>>>>> a bit less error-prone work than if the developer runs
> the
> >>>>>>> tests
> >>>>>>>>> and
> >>>>>>>>>>>>>>>>>>> test-patch themselves.  Also, with 22 being cut, there
> are
> >>> a
> >>>>>>> lot
> >>>>>>>>> of
> >>>>>>>>>>>>>>>>>> patches
> >>>>>>>>>>>>>>>>>>> up in the air and several developers are juggling
> multiple
> >>>>>>>>> patches.  The
> >>>>>>>>>>>>>>>>>>> more automation we can have, even if it's not perfect,
> >>> will
> >>>>>>>>> decrease
> >>>>>>>>>>>>>>>>>> errors
> >>>>>>>>>>>>>>>>>>> we may make.
> >>>>>>>>>>>>>>>>>>> -jg
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Nigel Daley wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Nov 17, 2010, at 3:11 PM, Jakob Homan wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> It's also ready to run on MapReduce and HDFS but we
> >>> won't
> >>>>>>>>> turn it on
> >>>>>>>>>>>>>>>>>>>>>> until these projects build and test cleanly.  Looks
> >>> like
> >>>>>>> both
> >>>>>>>>> these
> >>>>>>>>>>>>>>>>>> projects
> >>>>>>>>>>>>>>>>>>>>>> currently have test failures.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Assuming the projects are compiling and building, is
> >>> there
> >>>>> a
> >>>>>>>>> reason to
> >>>>>>>>>>>>>>>>>>>>> not turn it on despite the test failures? Hudson is
> >>>>>>> invaluable
> >>>>>>>>> to
> >>>>>>>>>>>>>>>>>> developers
> >>>>>>>>>>>>>>>>>>>>> who then don't have to run the tests and test-patch
> >>>>>>>>> themselves.  We
> >>>>>>>>>>>>>>>>>> didn't
> >>>>>>>>>>>>>>>>>>>>> turn Hudson off when it was working previously and
> there
> >>>>>>> were
> >>>>>>>>> known
> >>>>>>>>>>>>>>>>>>>>> failures.  I think one of the reasons we have more
> >>> failing
> >>>>>>>>> tests now is
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> higher cost of doing Hudson's work (not a great
> excuse I
> >>>>>>>>> know).  This
> >>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>> particularly true now because several of the failing
> >>> tests
> >>>>>>>>> involve
> >>>>>>>>>>>>>>>>>> tests
> >>>>>>>>>>>>>>>>>>>>> timing out, making the whole testing regime even
> longer.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Every single patch would get a -1 and need
> investigation.
> >>>>>>>>> Currently,
> >>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>> would be about 83 investigations between MR and HDFS
> >>> issues
> >>>>>>>>> that are in
> >>>>>>>>>>>>>>>>>>>> patch available state.  Shouldn't we focus on getting
> >>> these
> >>>>>>>>> tests fixed
> >>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>> removed/?  Also, I need to get MAPREDUCE-2172 fixed
> >>>>> (applies
> >>>>>>> to
> >>>>>>>>> HDFS as
> >>>>>>>>>>>>>>>>>>>> well) before I turn this on.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>> Nige
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>> Connect to me at http://www.facebook.com/dhruba
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Todd Lipcon
> >>>>>>>> Software Engineer, Cloudera
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Todd Lipcon
> >>>>>> Software Engineer, Cloudera
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Todd Lipcon
> >>>> Software Engineer, Cloudera
> >>>
> >>>
> >>
> >>
> >> --
> >> Todd Lipcon
> >> Software Engineer, Cloudera
> >
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

--00221534c927d9cce9049ac40e7f--

From general-return-2886-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 27 04:12:50 2011
Return-Path: <general-return-2886-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72915 invoked from network); 27 Jan 2011 04:12:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jan 2011 04:12:49 -0000
Received: (qmail 29052 invoked by uid 500); 27 Jan 2011 04:12:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28842 invoked by uid 500); 27 Jan 2011 04:12:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28818 invoked by uid 99); 27 Jan 2011 04:12:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Jan 2011 04:12:42 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Jan 2011 04:12:35 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0R4BlKk061506
	for <general@hadoop.apache.org>; Wed, 26 Jan 2011 20:11:47 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com;
	s=cobra; t=1296101507;
	bh=a++WZ0qqYil5MyAJ3ud4EHF5sKRY1jrOftwwmII9Sx4=;
	h=Message-Id:From:To:In-Reply-To:Content-Type:
	 Content-Transfer-Encoding:Mime-Version:Subject:Date:References;
	b=hypnoGywhXRZvYPQRRK3DQWUmt8iYREXEfLzIbHSoJ4mQhdbbloaMLOwWpg4iXr++
	 WY5kt128Ufso/Brkyq5IdUWAhMW/flq2wlMi+qfpNV/TpnTcrHtFAe1tba7/DRfI4h
	 5kwgXrtGqrPGiD8Zucp/nFpdPcPiGPXwKQzVZFJk=
Message-Id: <3539CB30-3574-4B3A-A327-D9AAAC36314B@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <5101FEE7-4C57-49C7-841C-2A9BC7BE0464@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSS] Hadoop Security Release off Yahoo! patchset
Date: Wed, 26 Jan 2011 20:11:46 -0800
References: <516684F5-0052-4381-805D-760B61DECB16@yahoo-inc.com> <366A9E58-5BD7-497D-9AE1-229959ED4065@apache.org> <18C5C999-4680-4684-BC55-A430C40FD746@yahoo-inc.com> <AANLkTim+y7CBi8WeV9W+FVckn-1D1SAV2hSc6RnEebSn@mail.gmail.com> <F8414465-24F2-4F40-97F1-B13048820F24@yahoo-inc.com> <D7FA8ED2-D9C6-4B45-AC3B-9E965ADD9F5D@apache.org> <2A07F1E6-7096-493B-B92E-89938689DD50@yahoo-inc.com> <5CDDF962-5828-459F-87C3-5033EC21E9BF@mac.com> <075308A1-129B-4BF7-8924-C04EC6106D3E@yahoo-inc.com> <4D34A270.1060102@apache.org> <3FEB7C1D-AE89-46BE-ABF2-B595B01E222A@mac.com> <D160E684-7435-4E53-A766-231CCB38D1B4@yahoo-inc.com> <3995AF5C-AA18-475B-9384-A532F1CC5D57@linkedin.com> <2090582B-B586-4F81-A724-6A0C55820BB5@yahoo-inc.com> <704BFF6E-AC1D-4769-8856-BFBD125D4D2C@Holsman.NET> <AANLkTikq-bPfXviWThvh=--cWEDYPfd0iEFSE_m_RbMn@mail.gmail.com> <B35E1DF9-ECC7-4A0D-9BFF-2F5A45CED68D@Holsman.NET> <5101FEE7-4C57-49C7-841C-2A9BC7BE0464@yahoo-inc.com>
X-Mailer: Apple Mail (2.936)

On Jan 25, 2011, at 2:05 AM, Eric Baldeschwieler wrote:
>
> I don't anticipate votes etc, unless folks do want to try it and do  
> provide feedback.  This is what runs at yahoo at the moment.  I hope  
> people think it is worth trying.
>

I've put up the bits at http://people.apache.org/~acmurthy/hadoop-0.20.100-rc0/ 
  for interested folks. Please do provide feedback if you try it.

thanks,
Arun


From general-return-2887-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 27 04:52:11 2011
Return-Path: <general-return-2887-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78697 invoked from network); 27 Jan 2011 04:52:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jan 2011 04:52:11 -0000
Received: (qmail 57510 invoked by uid 500); 27 Jan 2011 04:52:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57025 invoked by uid 500); 27 Jan 2011 04:52:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57017 invoked by uid 99); 27 Jan 2011 04:52:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Jan 2011 04:52:05 +0000
X-ASF-Spam-Status: No, hits=-1997.8 required=10.0
	tests=ALL_TRUSTED,HTML_MESSAGE,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 27 Jan 2011 04:52:04 +0000
Received: (qmail 78644 invoked by uid 99); 27 Jan 2011 04:51:44 -0000
Received: from localhost.apache.org (HELO mail-bw0-f48.google.com) (127.0.0.1)
  (smtp-auth username omalley, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Jan 2011 04:51:44 +0000
Received: by bwz8 with SMTP id 8so2226758bwz.35
        for <multiple recipients>; Wed, 26 Jan 2011 20:51:41 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.71.141 with SMTP id h13mr1141757bkj.180.1296103901440;
 Wed, 26 Jan 2011 20:51:41 -0800 (PST)
Received: by 10.204.100.134 with HTTP; Wed, 26 Jan 2011 20:51:41 -0800 (PST)
In-Reply-To: <AANLkTikBvezqxrmjS8wBnnTebhgELsFf2=Y=zH81CdWb@mail.gmail.com>
References: <AANLkTinXOcu16MSGUkahUCEXrz94v8Ym+c6BiTvu-67X@mail.gmail.com>
	<AANLkTim08e3Np6SBxE1FSpRUpW40xoFv6TfhiDXaGNP+@mail.gmail.com>
	<AANLkTins_HUzpVu25foOSrfGqFDC2gMyp48O83ep2rHb@mail.gmail.com>
	<AANLkTikBvezqxrmjS8wBnnTebhgELsFf2=Y=zH81CdWb@mail.gmail.com>
Date: Wed, 26 Jan 2011 20:51:41 -0800
Message-ID: <AANLkTinxHRB+_P-jw8xmu9SyBm+tZRFD=CyiEvmkKAF5@mail.gmail.com>
Subject: Re: Problem write on HDFS
From: "Owen O'Malley" <omalley@apache.org>
To: common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6dd8942a32c25049accb498

--0016e6dd8942a32c25049accb498
Content-Type: text/plain; charset=UTF-8

Please direct user questions to common-user@hadoop.apache.org.

-- Owen

On Tue, Jan 25, 2011 at 3:27 AM, Alessandro Binhara <binhara@gmail.com>wrote:

> I build a servlet with a hadoop...
> i think that tomcat enviroment will be find a hadoop-core-0.20.2.jar .. but
> a get a same error
>
> *ype* Exception report
>
> *message***
>
> *description* *The server encountered an internal error () that prevented
> it
> from fulfilling this request.*
>
> *exception*
>
> javax.servlet.ServletException: Servlet execution threw an exception
>
> *root cause*
>
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.conf.Configuration
>         HadoopWriterLib.HadoopWriter.OpenFileSystem(HadoopWriter.java:22)
>        HadoopWriterLib.HadoopWriter.<init>(HadoopWriter.java:16)
>        HadoopServletTest.doGet(HadoopServletTest.java:35)
>        javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
>        javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
>
> *note* *The full stack trace of the root cause is available in the Apache
> Tomcat/7.0.6 logs.*
>
> The error can be a problem on my ubuntu server ?
>
> thanks
>
> On Mon, Jan 24, 2011 at 1:01 PM, Alessandro Binhara <binhara@gmail.com
> >wrote:
>
> > ..i try
> >  java -classpath hadoop-core-0.20.1.jar -jar HahoopHdfsHello.jar
> >
> > i got a same error..
> > i will try build a servlet and run on tomcat...
> > i try many issues to config a classpath... all fail..
> >
> > thanks
> >
> >
> > On Mon, Jan 24, 2011 at 12:54 PM, Harsh J <qwertymaniac@gmail.com>
> wrote:
> >
> >> The issue would definitely lie with your CLASSPATH.
> >>
> >> Ideally, while beginning development using Hadoop 0.20, it is better
> >> to use the `hadoop jar` command to launch jars of any kind that
> >> require Hadoop libraries; be it MapReduce or not. The command will
> >> ensure that all the classpath requirements for Hadoop-side libraries
> >> are satisfied, so you don't have to worry.
> >>
> >> Anyhow, try launching it this way:
> >> $ java -classpath hadoop-0.20.2-core.jar -jar HadoopHdfsHello.jar; #
> >> This should run just fine.
> >>
> >> On Mon, Jan 24, 2011 at 5:06 PM, Alessandro Binhara <binhara@gmail.com>
> >> wrote:
> >> > Hello ..
> >> >
> >> > i solve problem in jar..
> >> > i put a hadoop-core-0.20.2.jar   in same jar dir.
> >> >
> >> > i configure a class path
> >> > export CLASSPATH=.:$JAVA_HOME
> >> >
> >> > i got this erro in shell
> >> >
> >> > root:~# java -jar HahoopHdfsHello.jar
> >> > Exception in thread "main" java.lang.NoClassDefFoundError:
> >> > org/apache/hadoop/conf/Configuration
> >> >        at HadooHdfsHello.main(HadooHdfsHello.java:18)
> >> > Caused by: java.lang.ClassNotFoundException:
> >> > org.apache.hadoop.conf.Configuration
> >> >        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >> >        at java.security.AccessController.doPrivileged(Native Method)
> >> >        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> >> >        at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >> >        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> >> >        ... 1 more
> >> >
> >> >
> >> > What is the problem?
> >> >
> >> > thanks
> >> >
> >>
> >>
> >>
> >> --
> >> Harsh J
> >> www.harshj.com
> >>
> >
> >
>

--0016e6dd8942a32c25049accb498--

From general-return-2888-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 27 05:40:08 2011
Return-Path: <general-return-2888-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 10455 invoked from network); 27 Jan 2011 05:40:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jan 2011 05:40:07 -0000
Received: (qmail 9439 invoked by uid 500); 27 Jan 2011 05:40:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 9107 invoked by uid 500); 27 Jan 2011 05:40:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 9099 invoked by uid 99); 27 Jan 2011 05:40:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Jan 2011 05:40:01 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of li.j2ee@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Jan 2011 05:39:54 +0000
Received: by bwz8 with SMTP id 8so2248823bwz.35
        for <general@hadoop.apache.org>; Wed, 26 Jan 2011 21:39:34 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=cm5ezs8DhwtuVsmAd3BRkDN5cD27/jFuaqKk+q12blg=;
        b=Kho0a7zQaspadbELIo1MwMf1JeBGv8e6gzRULW1uGt2E1bsyXGsBWHX95VxM9wsDO+
         due9lwWTNPmNnZ62QYg8TM7la/EVrMPpcgAjenjee/5GUVhLqTAF+oV3ftfiF+fM8qdD
         WdHp70tw+Y0zK3VUMUwNJNdIPz9PQxdR5TCtA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=fMNSUelKE2q2z2PuEEKwtjmu9vmW0WsMz13AbOSVdaWjLj/DjkZ2kk0Zsg9TlOGdrY
         OTVgl7WduwuuQn07kz8rCHAJjrvFXRoKtG8YVZAarJ8bKkYbPyTp4bZlyn0S+KQf2wQh
         v6yEEpJGJOjQf1vtQrnySMhQN8JMePbo7pWcs=
MIME-Version: 1.0
Received: by 10.204.76.18 with SMTP id a18mr1196139bkk.9.1296106772272; Wed,
 26 Jan 2011 21:39:32 -0800 (PST)
Received: by 10.204.16.146 with HTTP; Wed, 26 Jan 2011 21:39:32 -0800 (PST)
In-Reply-To: <AANLkTimVtXOoiDh_ADcZbtT+Mq_S+fLyrGTO9js8zNxf@mail.gmail.com>
References: <AANLkTimVtXOoiDh_ADcZbtT+Mq_S+fLyrGTO9js8zNxf@mail.gmail.com>
Date: Thu, 27 Jan 2011 13:39:32 +0800
Message-ID: <AANLkTimFCwbhzrhGbrXht9Yiwr1nXGX-5ypw8_svUb+k@mail.gmail.com>
Subject: Re: Hadoop datanote question
From: li ping <li.j2ee@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f46e5ac0982c049acd5f3b
X-Virus-Checked: Checked by ClamAV on apache.org

--001485f46e5ac0982c049acd5f3b
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

I think you must misunderstanding the architecture.
I believe the "master node" that you talk about is actually a NameNode, the
client must talk with NameNode. the namenode will check if the file that yo=
u
try to write existed or not. if the file doesn't exist, NN will create a
record in NN and return a DFSOutputSteam. the data will be writed in this
OutputStream. The DFSOutputStream will handle which datanode the data will
be writed to.

So, the only entry point is the NameNode. you can not connect to the other
node to write the file.

On Wed, Jan 26, 2011 at 7:24 PM, Alessandro Binhara <binhara@gmail.com>wrot=
e:

> Hello..
>
> I'm confused about the architecture of HDFS
> I have a client java write a files on a master node. It=C2=B4s work perfe=
ct!..
> well i see a HDFS architecture
> http://hadoop.apache.org/common/docs/r0.20.0/images/hdfsarchitecture.gif
> some client access a data note..
>
> i try use my java client write direct on slave node. I change IP on clien=
t
> configuration:
>                Configuration conf =3D new Configuration();
> conf.set("fs.default.name","hdfs://192.168.254.25:54310/"); //slave ip
>
> I got this error. .
> 26/01/2011 09:19:52 org.apache.hadoop.ipc.Client$Connection
> handleConnectionFailure
> It dont work ..
>
> In architecture picture show me that all datanode can conectable. That=C2=
=B4s
> correct ?
>
> How it work ?
> I can only conect to write on HDFS on master node ??
>
> thank s
>



--=20
-----=E6=9D=8E=E5=B9=B3

--001485f46e5ac0982c049acd5f3b--

From general-return-2889-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 28 21:49:17 2011
Return-Path: <general-return-2889-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 65054 invoked from network); 28 Jan 2011 21:49:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Jan 2011 21:49:16 -0000
Received: (qmail 48192 invoked by uid 500); 28 Jan 2011 21:49:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48109 invoked by uid 500); 28 Jan 2011 21:49:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48101 invoked by uid 99); 28 Jan 2011 21:49:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Jan 2011 21:49:14 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jcoveney@gmail.com designates 209.85.214.48 as permitted sender)
Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Jan 2011 21:49:06 +0000
Received: by bwz8 with SMTP id 8so4077599bwz.35
        for <general@hadoop.apache.org>; Fri, 28 Jan 2011 13:48:46 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=rmRYEuMBVlxqnj+Q/Nqz/B1o5TEH/t75PNC9udT6qis=;
        b=XWHztKNxsQ3JWnB8lxpnkPdpoDc7bfOZN0W0r286b2MMkO/5VRnHUxd03LxbRYHxME
         uSqnmXrPUCSM3t7Xlz3zO19mGOtUeN4BRNNHn+QgYIroibRXcSQXJOLM4fvAYC3CaD1f
         KpNhkWVBZn4gbDGqDyJmmnPMxYddn2OOWyF80=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=YOqjVagyp2Dfwgmwec9xauiulTZn96qr7ZoOyTMvDnJqZdtICi+hDFySPiWL11j5j4
         yb3WpE70aZm4BInV0Mc0PKP8FJYtPtR5xDF1zB0pJ8wUlAs00PwEQu0M8500HdPYYw08
         +vqmDhj1rG3Qhpr7YvssAyXiQweELpfv2hNT8=
MIME-Version: 1.0
Received: by 10.204.85.84 with SMTP id n20mr2905780bkl.210.1296251326042; Fri,
 28 Jan 2011 13:48:46 -0800 (PST)
Received: by 10.204.16.135 with HTTP; Fri, 28 Jan 2011 13:48:46 -0800 (PST)
Date: Fri, 28 Jan 2011 16:48:46 -0500
Message-ID: <AANLkTi=_Y7hj=hKq1CcCi1Cn45LfFin41Q=woNXTy5ax@mail.gmail.com>
Subject: Dump question about initializing log4j...
From: Jonathan Coveney <jcoveney@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6dab4fdd41441049aef07a7
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6dab4fdd41441049aef07a7
Content-Type: text/plain; charset=ISO-8859-1

I am trying to run hadoop in standalone mode (mainly to get it working with
pig).

I get this error:

log4j:WARN No appenders could be found for logger
(org.apache.hadoop.conf.Configuration).
log4j:WARN Please initialize the log4j system properly.

Looking around, I haven't been able to find a straightforward "this is what
you need to do," and I know very little about these things. I'd be very
thankful if someone could enlighten me...

Thank you
Jon

--0016e6dab4fdd41441049aef07a7--

From general-return-2890-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Jan 30 23:26:40 2011
Return-Path: <general-return-2890-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1826 invoked from network); 30 Jan 2011 23:26:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Jan 2011 23:26:40 -0000
Received: (qmail 63020 invoked by uid 500); 30 Jan 2011 23:26:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62878 invoked by uid 500); 30 Jan 2011 23:26:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62870 invoked by uid 99); 30 Jan 2011 23:26:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 30 Jan 2011 23:26:37 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of esammer@cloudera.com designates 209.85.210.176 as permitted sender)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 30 Jan 2011 23:26:33 +0000
Received: by iyb26 with SMTP id 26so4881513iyb.35
        for <general@hadoop.apache.org>; Sun, 30 Jan 2011 15:26:12 -0800 (PST)
MIME-Version: 1.0
Received: by 10.42.220.67 with SMTP id hx3mr7018507icb.350.1296429970953; Sun,
 30 Jan 2011 15:26:10 -0800 (PST)
Received: by 10.42.220.70 with HTTP; Sun, 30 Jan 2011 15:26:10 -0800 (PST)
Date: Sun, 30 Jan 2011 18:26:10 -0500
Message-ID: <AANLkTi=fOv-5sKrP9jMDC=mJuQjZgh+Tz4LWyDjYiwic@mail.gmail.com>
Subject: MRUnit
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf30549ef7e51554049b189f2a

--20cf30549ef7e51554049b189f2a
Content-Type: text/plain; charset=ISO-8859-1

All:

I've created a fork of mrunit from Hadoop's contrib directory at
https://github.com/esammer/mrunit

Why:

* MRUnit could benefit from a separate development and release cycle than
that of Hadoop itself. This would allow MRUnit to evolve independently of
Hadoop.
* Moving projects out of contrib simplify Hadoop's build and the artifacts
that need to be maintained.
* I've heard rumblings that contrib is generally frowned upon by some of the
committers. I agree.

What is already done:

* Build system ported to Maven.
* No more build warnings re: generics or deprecation.

I think it makes sense to remove mrunit from contrib (after a deprecation
period) but I'm curious to hear others' opinions. If the community at large
decides it makes sense to retain mrunit in contrib I'm happy to provide
contributions back. I'd also like to offer commit status on the fork to any
current Hadoop committers who are interested. The project is being hosted on
Github to ease collaboration. Pull requests are greatly appreciated.

Regards.
-- 
Eric Sammer
twitter: esammer
data: www.cloudera.com

--20cf30549ef7e51554049b189f2a--

From general-return-2891-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 31 02:50:58 2011
Return-Path: <general-return-2891-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83413 invoked from network); 31 Jan 2011 02:50:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jan 2011 02:50:58 -0000
Received: (qmail 81357 invoked by uid 500); 31 Jan 2011 02:50:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81043 invoked by uid 500); 31 Jan 2011 02:50:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81035 invoked by uid 99); 31 Jan 2011 02:50:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 02:50:55 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of hammer@cloudera.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 02:50:49 +0000
Received: by iwn2 with SMTP id 2so5539781iwn.35
        for <general@hadoop.apache.org>; Sun, 30 Jan 2011 18:50:28 -0800 (PST)
MIME-Version: 1.0
Received: by 10.42.176.68 with SMTP id bd4mr7159342icb.408.1296442228247; Sun,
 30 Jan 2011 18:50:28 -0800 (PST)
Received: by 10.42.173.7 with HTTP; Sun, 30 Jan 2011 18:50:28 -0800 (PST)
In-Reply-To: <AANLkTi=fOv-5sKrP9jMDC=mJuQjZgh+Tz4LWyDjYiwic@mail.gmail.com>
References: <AANLkTi=fOv-5sKrP9jMDC=mJuQjZgh+Tz4LWyDjYiwic@mail.gmail.com>
Date: Sun, 30 Jan 2011 18:50:28 -0800
Message-ID: <AANLkTi=9kZgxwRWyBcDJcjno5WY-uT7gQx=FJ9jF=3=M@mail.gmail.com>
Subject: Re: MRUnit
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba6e8ef07c8cca049b1b7a03

--90e6ba6e8ef07c8cca049b1b7a03
Content-Type: text/plain; charset=ISO-8859-1

>
> I think it makes sense to remove mrunit from contrib (after a
> deprecation period) but I'm curious to hear others' opinions.


Oh please do take things out of contrib. I'd love to see all of it move to
Github. Thanks for putting in the time for MRUnit.

--90e6ba6e8ef07c8cca049b1b7a03--

From general-return-2892-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 31 03:43:01 2011
Return-Path: <general-return-2892-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4898 invoked from network); 31 Jan 2011 03:43:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jan 2011 03:43:01 -0000
Received: (qmail 2181 invoked by uid 500); 31 Jan 2011 03:43:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1743 invoked by uid 500); 31 Jan 2011 03:42:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1715 invoked by uid 99); 31 Jan 2011 03:42:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 03:42:55 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.100 as permitted sender)
Received: from [17.148.16.100] (HELO asmtpout025.mac.com) (17.148.16.100)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 03:42:47 +0000
MIME-version: 1.0
Content-type: multipart/alternative;
 boundary="Boundary_(ID_AxP92tquXRHGlvdMq3akGg)"
Received: from [172.20.1.124] ([173.164.176.117])
 by asmtp025.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LFV0034LBMPVD70@asmtp025.mac.com>
 for general@hadoop.apache.org; Sun, 30 Jan 2011 19:42:26 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-30_08:2011-01-28,2011-01-30,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1101300159
From: Nigel Daley <ndaley@mac.com>
Subject: [DISCUSS] Move common, hdfs,
 mapreduce contrib components to apache-extras.org or elsewhere
Date: Sun, 30 Jan 2011 19:42:25 -0800
Message-id: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)
X-Virus-Checked: Checked by ClamAV on apache.org

--Boundary_(ID_AxP92tquXRHGlvdMq3akGg)
Content-type: text/plain; CHARSET=US-ASCII
Content-transfer-encoding: 7BIT

Folks,

Now that http://apache-extras.org is launched (https://blogs.apache.org/foundation/entry/the_apache_software_foundation_launches) I'd like to start a discussion on moving contrib components out of common, mapreduce, and hdfs.  

These contrib components complicate the builds, cause test failures that nobody seems to care about, have releases that are tied to Hadoop's long release cycles, etc.  Most folks I've talked with agree that these contrib components would be better served by being pulled out of Hadoop and hosted elsewhere. The new apache-extras code hosting site seems like a natural *default* location for migrating these contrib projects.  Perhaps some should graduate from contrib to src (ie from contrib to core of the project they're included in).  If folks agree, we'll need to come up with a mapping of contrib component to it's final destination and file a jira.

Here are the contrib components by project (hopefully I didn't miss any).

Common Contrib:
  failmon
  hod
  test


MapReduce Contrib:
  capacity-scheduler -- move to MR core?
  data_join
  dynamic-scheduler
  eclipse-plugin
  fairscheduler -- move to MR core?
  gridmix
  index
  mrunit
  mumak
  raid
  sqoop
  streaming -- move to MR core?
  vaidya
  vertica


HDFS Contrib:
  fuse-dfs
  hdfsproxy
  thriftfs


Cheers,
Nige

--Boundary_(ID_AxP92tquXRHGlvdMq3akGg)--

From general-return-2893-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 31 04:00:39 2011
Return-Path: <general-return-2893-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 13403 invoked from network); 31 Jan 2011 04:00:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jan 2011 04:00:38 -0000
Received: (qmail 18161 invoked by uid 500); 31 Jan 2011 04:00:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17536 invoked by uid 500); 31 Jan 2011 04:00:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17527 invoked by uid 99); 31 Jan 2011 04:00:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 04:00:32 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ndaley@mac.com designates 17.148.16.95 as permitted sender)
Received: from [17.148.16.95] (HELO asmtpout020.mac.com) (17.148.16.95)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 04:00:24 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII
Received: from [172.20.1.124] ([173.164.176.117])
 by asmtp020.mac.com (Oracle Communications Messaging Exchange Server 7u4-20.01
 64bit (built Nov 21 2010)) with ESMTPSA id <0LFV00H1GCFROV50@asmtp020.mac.com>
 for general@hadoop.apache.org; Sun, 30 Jan 2011 19:59:52 -0800 (PST)
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=2.50.10432:5.2.15,1.0.148,0.0.0000
 definitions=2011-01-30_08:2011-01-28,2011-01-30,1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 suspectscore=1 phishscore=0 bulkscore=0 adultscore=0 classifier=spam
 adjust=0 reason=mlx engine=6.0.2-1012030000 definitions=main-1101300160
Subject: Re: MRUnit
From: Nigel Daley <ndaley@mac.com>
In-reply-to: <AANLkTi=9kZgxwRWyBcDJcjno5WY-uT7gQx=FJ9jF=3=M@mail.gmail.com>
Date: Sun, 30 Jan 2011 19:59:51 -0800
Message-id: <5E01B48E-BFE9-45BA-B729-50E692B74F46@mac.com>
References: <AANLkTi=fOv-5sKrP9jMDC=mJuQjZgh+Tz4LWyDjYiwic@mail.gmail.com>
 <AANLkTi=9kZgxwRWyBcDJcjno5WY-uT7gQx=FJ9jF=3=M@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

+1.  I just started a thread on moving all components out of contrib.

nige

On Jan 30, 2011, at 6:50 PM, Jeff Hammerbacher wrote:

>> 
>> I think it makes sense to remove mrunit from contrib (after a
>> deprecation period) but I'm curious to hear others' opinions.
> 
> 
> Oh please do take things out of contrib. I'd love to see all of it move to
> Github. Thanks for putting in the time for MRUnit.


From general-return-2894-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 31 04:23:04 2011
Return-Path: <general-return-2894-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27479 invoked from network); 31 Jan 2011 04:23:04 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jan 2011 04:23:04 -0000
Received: (qmail 44368 invoked by uid 500); 31 Jan 2011 04:23:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 43869 invoked by uid 500); 31 Jan 2011 04:22:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 43861 invoked by uid 99); 31 Jan 2011 04:22:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 04:22:57 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of esammer@cloudera.com designates 209.85.210.176 as permitted sender)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 04:22:53 +0000
Received: by iyb26 with SMTP id 26so5052025iyb.35
        for <general@hadoop.apache.org>; Sun, 30 Jan 2011 20:22:32 -0800 (PST)
MIME-Version: 1.0
Received: by 10.42.222.9 with SMTP id ie9mr7247496icb.417.1296447752128; Sun,
 30 Jan 2011 20:22:32 -0800 (PST)
Received: by 10.42.220.70 with HTTP; Sun, 30 Jan 2011 20:22:32 -0800 (PST)
In-Reply-To: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com>
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com>
Date: Sun, 30 Jan 2011 23:22:32 -0500
Message-ID: <AANLkTinwTvLhXtBcwHwDwm2comw_i0jNoY5fF8=owhgo@mail.gmail.com>
Subject: Re: [DISCUSS] Move common, hdfs, mapreduce contrib components to
 apache-extras.org or elsewhere
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=20cf3054a033bc47ee049b1cc36b

--20cf3054a033bc47ee049b1cc36b
Content-Type: text/plain; charset=ISO-8859-1

Huge +1.

Sqoop has continued development at https://github.com/cloudera/sqoop
MRUnit is at https://github.com/esammer/mrunit

I think the former can be easily removed. The latter, per my previous email,
I think could be removed as well.

On Sun, Jan 30, 2011 at 10:42 PM, Nigel Daley <ndaley@mac.com> wrote:

> Folks,
>
> Now that http://apache-extras.org is launched (
> https://blogs.apache.org/foundation/entry/the_apache_software_foundation_launches)
> I'd like to start a discussion on moving contrib components out of common,
> mapreduce, and hdfs.
>
> These contrib components complicate the builds, cause test failures that
> nobody seems to care about, have releases that are tied to Hadoop's long
> release cycles, etc.  Most folks I've talked with agree that these contrib
> components would be better served by being pulled out of Hadoop and hosted
> elsewhere. The new apache-extras code hosting site seems like a natural
> *default* location for migrating these contrib projects.  Perhaps some
> should graduate from contrib to src (ie from contrib to core of the project
> they're included in).  If folks agree, we'll need to come up with a mapping
> of contrib component to it's final destination and file a jira.
>
> Here are the contrib components by project (hopefully I didn't miss any).
>
> Common Contrib:
>  failmon
>  hod
>  test
>
>
> MapReduce Contrib:
>  capacity-scheduler -- move to MR core?
>  data_join
>  dynamic-scheduler
>  eclipse-plugin
>  fairscheduler -- move to MR core?
>  gridmix
>  index
>  mrunit
>  mumak
>  raid
>  sqoop
>  streaming -- move to MR core?
>  vaidya
>  vertica
>
>
> HDFS Contrib:
>  fuse-dfs
>  hdfsproxy
>  thriftfs
>
>
> Cheers,
> Nige
>



-- 
Eric Sammer
twitter: esammer
data: www.cloudera.com

--20cf3054a033bc47ee049b1cc36b--

From general-return-2895-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 31 05:25:27 2011
Return-Path: <general-return-2895-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 52374 invoked from network); 31 Jan 2011 05:25:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jan 2011 05:25:27 -0000
Received: (qmail 81899 invoked by uid 500); 31 Jan 2011 05:25:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81564 invoked by uid 500); 31 Jan 2011 05:25:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81556 invoked by uid 99); 31 Jan 2011 05:25:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 05:25:21 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 05:25:15 +0000
Received: by wye20 with SMTP id 20so5379400wye.35
        for <general@hadoop.apache.org>; Sun, 30 Jan 2011 21:24:54 -0800 (PST)
Received: by 10.216.163.203 with SMTP id a53mr5314220wel.104.1296451494365;
 Sun, 30 Jan 2011 21:24:54 -0800 (PST)
MIME-Version: 1.0
Received: by 10.216.59.13 with HTTP; Sun, 30 Jan 2011 21:24:34 -0800 (PST)
In-Reply-To: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com>
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com>
From: Konstantin Boudnik <cos@boudnik.org>
Date: Sun, 30 Jan 2011 21:24:34 -0800
Message-ID: <AANLkTikLiLjj5_ghvFaW+XApAJ9SrsqZKEJO0+UPFcgW@mail.gmail.com>
Subject: Re: [DISCUSS] Move common, hdfs, mapreduce contrib components to
 apache-extras.org or elsewhere
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

While this seems to be a very good and long awaited advancement (btw
thanks to Eric Sammer for making a first project on the move out of
MRUnit!) I have a concern about lack of Git support on
http://apache-extras.org. I am sure that hosting decision was deeply
considered by Apache board (or the whole Apache community) however
SVN/Mercurial seems like a real drag for most of Hadoop developers.

Basically, if contribs are moved to apache-extras.org then we are
likely to face the same situation as with the core Hadoop: lotta
development is done in personal Git repos forked from a Git/SVN mirror
and then is committed back to the man SVN repository creating an extra
cycle.

Shall we not dictate a location of contrib projects once they are
moved of Hadoop? If ppl feel like they are better be served by GitHub
perhaps they should have an option to get hosted there?
--
=A0 Take care,
Konstantin (Cos) Boudnik


On Sun, Jan 30, 2011 at 19:42, Nigel Daley <ndaley@mac.com> wrote:
> Folks,
>
> Now that http://apache-extras.org is launched (https://blogs.apache.org/f=
oundation/entry/the_apache_software_foundation_launches) I'd like to start =
a discussion on moving contrib components out of common, mapreduce, and hdf=
s.
>
> These contrib components complicate the builds, cause test failures that =
nobody seems to care about, have releases that are tied to Hadoop's long re=
lease cycles, etc. =A0Most folks I've talked with agree that these contrib =
components would be better served by being pulled out of Hadoop and hosted =
elsewhere. The new apache-extras code hosting site seems like a natural *de=
fault* location for migrating these contrib projects. =A0Perhaps some shoul=
d graduate from contrib to src (ie from contrib to core of the project they=
're included in). =A0If folks agree, we'll need to come up with a mapping o=
f contrib component to it's final destination and file a jira.
>
> Here are the contrib components by project (hopefully I didn't miss any).
>
> Common Contrib:
> =A0failmon
> =A0hod
> =A0test
>
>
> MapReduce Contrib:
> =A0capacity-scheduler -- move to MR core?
> =A0data_join
> =A0dynamic-scheduler
> =A0eclipse-plugin
> =A0fairscheduler -- move to MR core?
> =A0gridmix
> =A0index
> =A0mrunit
> =A0mumak
> =A0raid
> =A0sqoop
> =A0streaming -- move to MR core?
> =A0vaidya
> =A0vertica
>
>
> HDFS Contrib:
> =A0fuse-dfs
> =A0hdfsproxy
> =A0thriftfs
>
>
> Cheers,
> Nige
>

From general-return-2896-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 31 05:38:34 2011
Return-Path: <general-return-2896-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67555 invoked from network); 31 Jan 2011 05:38:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jan 2011 05:38:34 -0000
Received: (qmail 87375 invoked by uid 500); 31 Jan 2011 05:38:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86939 invoked by uid 500); 31 Jan 2011 05:38:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86924 invoked by uid 99); 31 Jan 2011 05:38:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 05:38:29 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 05:38:23 +0000
Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0V5bqaV071764
	for <general@hadoop.apache.org>; Sun, 30 Jan 2011 21:37:52 -0800 (PST)
Received: from [10.0.1.4] (10.72.168.149) by SP2-EX07CAS01.ds.corp.yahoo.com
 (98.137.59.4) with Microsoft SMTP Server (TLS) id 8.3.137.0; Sun, 30 Jan 2011
 21:37:51 -0800
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0 (Apple Message framework v1082)
Subject: Re: [DISCUSS] Move common, hdfs, mapreduce contrib components to apache-extras.org or elsewhere
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
In-Reply-To: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com>
Date: Sun, 30 Jan 2011 21:37:48 -0800
Content-Transfer-Encoding: quoted-printable
Message-ID: <1FA5AC45-7586-4F4C-9EF6-7A8F41199E2A@yahoo-inc.com>
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
X-Mailer: Apple Mail (2.1082)

+1 - A really good idea to clean up the builds and ownership issues.

I think it is good to have a default location for related projects and =
apache-extras.org does seem like a logical place.

We should also probably add a prominent wiki section to support use of =
apache-extras.org and to provide links to projects that want to host =
elsewhere.

On Jan 30, 2011, at 7:42 PM, Nigel Daley wrote:

> Folks,
>=20
> Now that http://apache-extras.org is launched =
(https://blogs.apache.org/foundation/entry/the_apache_software_foundation_=
launches) I'd like to start a discussion on moving contrib components =
out of common, mapreduce, and hdfs. =20
>=20
> These contrib components complicate the builds, cause test failures =
that nobody seems to care about, have releases that are tied to Hadoop's =
long release cycles, etc.  Most folks I've talked with agree that these =
contrib components would be better served by being pulled out of Hadoop =
and hosted elsewhere. The new apache-extras code hosting site seems like =
a natural *default* location for migrating these contrib projects.  =
Perhaps some should graduate from contrib to src (ie from contrib to =
core of the project they're included in).  If folks agree, we'll need to =
come up with a mapping of contrib component to it's final destination =
and file a jira.
>=20
> Here are the contrib components by project (hopefully I didn't miss =
any).
>=20
> Common Contrib:
>  failmon
>  hod
>  test
>=20
>=20
> MapReduce Contrib:
>  capacity-scheduler -- move to MR core?
>  data_join
>  dynamic-scheduler
>  eclipse-plugin
>  fairscheduler -- move to MR core?
>  gridmix
>  index
>  mrunit
>  mumak
>  raid
>  sqoop
>  streaming -- move to MR core?
>  vaidya
>  vertica
>=20
>=20
> HDFS Contrib:
>  fuse-dfs
>  hdfsproxy
>  thriftfs
>=20
>=20
> Cheers,
> Nige


From general-return-2897-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 31 06:02:58 2011
Return-Path: <general-return-2897-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 71853 invoked from network); 31 Jan 2011 06:02:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jan 2011 06:02:58 -0000
Received: (qmail 97607 invoked by uid 500); 31 Jan 2011 06:02:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97082 invoked by uid 500); 31 Jan 2011 06:02:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97074 invoked by uid 99); 31 Jan 2011 06:02:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 06:02:52 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of eli@cloudera.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 06:02:47 +0000
Received: by qwe4 with SMTP id 4so5285139qwe.35
        for <general@hadoop.apache.org>; Sun, 30 Jan 2011 22:02:26 -0800 (PST)
MIME-Version: 1.0
Received: by 10.229.247.68 with SMTP id mb4mr5448590qcb.294.1296453744900;
 Sun, 30 Jan 2011 22:02:24 -0800 (PST)
Received: by 10.229.215.2 with HTTP; Sun, 30 Jan 2011 22:02:24 -0800 (PST)
In-Reply-To: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com>
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com>
Date: Sun, 30 Jan 2011 22:02:24 -0800
Message-ID: <AANLkTimew9N9mfq_tdrgQ5t1hoVt+XogKHFXTCSenk3W@mail.gmail.com>
Subject: Re: [DISCUSS] Move common, hdfs, mapreduce contrib components to
 apache-extras.org or elsewhere
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

+1

Agree w Eric's comments around defaults and documenting them.

How about setting a date where the projects are created on apache
extras and removed from contrib unless someone volunteers to maintain
them?  I agree apache extras should be the default unless the person
volunteering to maintain the project wants to host it elsewhere.

Btw I think raid should also probably move to core, it's actively worked on=
.

I'll volunteer to create a project for fuse-dfs.

Thanks,
Eli

On Sun, Jan 30, 2011 at 7:42 PM, Nigel Daley <ndaley@mac.com> wrote:
> Folks,
>
> Now that http://apache-extras.org is launched (https://blogs.apache.org/f=
oundation/entry/the_apache_software_foundation_launches) I'd like to start =
a discussion on moving contrib components out of common, mapreduce, and hdf=
s.
>
> These contrib components complicate the builds, cause test failures that =
nobody seems to care about, have releases that are tied to Hadoop's long re=
lease cycles, etc. =A0Most folks I've talked with agree that these contrib =
components would be better served by being pulled out of Hadoop and hosted =
elsewhere. The new apache-extras code hosting site seems like a natural *de=
fault* location for migrating these contrib projects. =A0Perhaps some shoul=
d graduate from contrib to src (ie from contrib to core of the project they=
're included in). =A0If folks agree, we'll need to come up with a mapping o=
f contrib component to it's final destination and file a jira.
>
> Here are the contrib components by project (hopefully I didn't miss any).
>
> Common Contrib:
> =A0failmon
> =A0hod
> =A0test
>
>
> MapReduce Contrib:
> =A0capacity-scheduler -- move to MR core?
> =A0data_join
> =A0dynamic-scheduler
> =A0eclipse-plugin
> =A0fairscheduler -- move to MR core?
> =A0gridmix
> =A0index
> =A0mrunit
> =A0mumak
> =A0raid
> =A0sqoop
> =A0streaming -- move to MR core?
> =A0vaidya
> =A0vertica
>
>
> HDFS Contrib:
> =A0fuse-dfs
> =A0hdfsproxy
> =A0thriftfs
>
>
> Cheers,
> Nige
>

From general-return-2898-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 31 07:19:52 2011
Return-Path: <general-return-2898-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 10687 invoked from network); 31 Jan 2011 07:19:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jan 2011 07:19:51 -0000
Received: (qmail 43183 invoked by uid 500); 31 Jan 2011 07:19:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42686 invoked by uid 500); 31 Jan 2011 07:19:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42678 invoked by uid 99); 31 Jan 2011 07:19:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 07:19:46 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.27.228] (HELO qmta15.emeryville.ca.mail.comcast.net) (76.96.27.228)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 07:19:39 +0000
Received: from omta03.emeryville.ca.mail.comcast.net ([76.96.30.27])
	by qmta15.emeryville.ca.mail.comcast.net with comcast
	id 27Jm1g0060b6N64AF7KJo2; Mon, 31 Jan 2011 07:19:18 +0000
Received: from [10.0.0.52] ([209.131.62.115])
	by omta03.emeryville.ca.mail.comcast.net with comcast
	id 27K91g0092VBGtd8P7KCZm; Mon, 31 Jan 2011 07:19:16 +0000
Message-Id: <FD5BC1D3-881A-4D45-BA2C-C6084C884418@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: [DISCUSS] Move common, hdfs, mapreduce contrib components to apache-extras.org or elsewhere
Date: Sun, 30 Jan 2011 23:19:08 -0800
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com>
X-Mailer: Apple Mail (2.936)


On Jan 30, 2011, at 7:42 PM, Nigel Daley wrote:

> Now that http://apache-extras.org is launched (https://blogs.apache.org/foundation/entry/the_apache_software_foundation_launches 
> ) I'd like to start a discussion on moving contrib components out of  
> common, mapreduce, and hdfs.

The PMC can't "move" code to Apache extras. It can only choose to  
abandon code that it doesn't want to support any longer. As a separate  
action some group of developers may create projects in Apache Extras  
based on the code from Hadoop.

Therefore the question is really what if any code Hadoop wants to  
abandon. That is a good question and one that we should ask ourselves  
occasionally.

After a quick consideration, my personal list would look like:

failmon
fault injection
fuse-dfs
hod
kfs

Also note that pushing code out of Hadoop has a high cost. There are  
at least 3 forks of the hadoop-gpl-compression code. That creates a  
lot of confusion for the users. A lot of users never go to the work to  
figure out which fork and branch of hadoop-gpl-compression work with  
the version of Hadoop they installed.

-- Owen


From general-return-2899-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 31 08:41:42 2011
Return-Path: <general-return-2899-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38128 invoked from network); 31 Jan 2011 08:41:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jan 2011 08:41:41 -0000
Received: (qmail 13703 invoked by uid 500); 31 Jan 2011 08:41:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13367 invoked by uid 500); 31 Jan 2011 08:41:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13359 invoked by uid 99); 31 Jan 2011 08:41:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 08:41:35 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of dhruba@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 08:41:28 +0000
Received: by wye20 with SMTP id 20so5493420wye.35
        for <general@hadoop.apache.org>; Mon, 31 Jan 2011 00:41:08 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=C0UZOfMw0kUnVfV4A0kb/EPLM8uQEPBBGPjH2NTAKIY=;
        b=prTGen80hRwEXeft/gQJd9wWiJ3D1ACabG5PCsanROuE3aVc9A6REgGaNAcs62UZpL
         FBWI9cw7mvzzapSzhGTuwG26KHBI7gjYXjvW0/QRwiM6Frvovc7p20A7hJ1QXV35qKXN
         yfYUCwj2hpjji05Ttaoux9UHN8tKFcPw/ujeM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=nWBIQJueXj06fR2TAhFZKOBnxolpxZWWjvCRgdQ4X/cpuhtiOHx9NahXvCKXMQAE+w
         JBDGEZeWz/cVGuk0eIysA/5AVk74FuCGrD58lQOsMkL6jArarENu+dJ9NJdxLkJBK0bE
         jUmHLr09NY46FndoO0SyUaNy8SsU1a/5zL60w=
MIME-Version: 1.0
Received: by 10.216.183.195 with SMTP id q45mr5504409wem.94.1296463268625;
 Mon, 31 Jan 2011 00:41:08 -0800 (PST)
Received: by 10.216.159.5 with HTTP; Mon, 31 Jan 2011 00:41:08 -0800 (PST)
In-Reply-To: <FD5BC1D3-881A-4D45-BA2C-C6084C884418@apache.org>
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com>
	<FD5BC1D3-881A-4D45-BA2C-C6084C884418@apache.org>
Date: Mon, 31 Jan 2011 00:41:08 -0800
Message-ID: <AANLkTin=VRU8xUrOV4qGBRRsmDxnmKoEMhZfAwbC9DRh@mail.gmail.com>
Subject: Re: [DISCUSS] Move common, hdfs, mapreduce contrib components to
 apache-extras.org or elsewhere
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f1e2d4973e23049b206029
X-Virus-Checked: Checked by ClamAV on apache.org

--001485f1e2d4973e23049b206029
Content-Type: text/plain; charset=ISO-8859-1

I agree with Owen. If we move code out of the contrib project, then it is
more likely to create confusion among users, especially when multiple
versions of the code base float around.

But I agree that we should purge contrib code that is not being used or not
being actively developed.

thanks,
dhruba


On Sun, Jan 30, 2011 at 11:19 PM, Owen O'Malley <omalley@apache.org> wrote:

>
> On Jan 30, 2011, at 7:42 PM, Nigel Daley wrote:
>
>  Now that http://apache-extras.org is launched (
>> https://blogs.apache.org/foundation/entry/the_apache_software_foundation_launches)
>> I'd like to start a discussion on moving contrib components out of common,
>> mapreduce, and hdfs.
>>
>
> The PMC can't "move" code to Apache extras. It can only choose to abandon
> code that it doesn't want to support any longer. As a separate action some
> group of developers may create projects in Apache Extras based on the code
> from Hadoop.
>
> Therefore the question is really what if any code Hadoop wants to abandon.
> That is a good question and one that we should ask ourselves occasionally.
>
> After a quick consideration, my personal list would look like:
>
> failmon
> fault injection
> fuse-dfs
> hod
> kfs
>
> Also note that pushing code out of Hadoop has a high cost. There are at
> least 3 forks of the hadoop-gpl-compression code. That creates a lot of
> confusion for the users. A lot of users never go to the work to figure out
> which fork and branch of hadoop-gpl-compression work with the version of
> Hadoop they installed.
>
> -- Owen
>
>


-- 
Connect to me at http://www.facebook.com/dhruba

--001485f1e2d4973e23049b206029--

From general-return-2900-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 31 11:29:21 2011
Return-Path: <general-return-2900-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 99436 invoked from network); 31 Jan 2011 11:29:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jan 2011 11:29:20 -0000
Received: (qmail 63547 invoked by uid 500); 31 Jan 2011 11:29:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63175 invoked by uid 500); 31 Jan 2011 11:29:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63167 invoked by uid 99); 31 Jan 2011 11:29:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 11:29:16 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of binhara@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 11:29:10 +0000
Received: by wye20 with SMTP id 20so5627449wye.35
        for <general@hadoop.apache.org>; Mon, 31 Jan 2011 03:28:48 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:date:message-id:subject:from:to
         :content-type;
        bh=uhYBnECLuxvU9EH40NnS5uMUQCUsvz0Vlh8p98/dbC8=;
        b=V0EzbH0dfN83ctbOl6YbgiFg1Mw8CtPqIltVWEYAnBAk9MivqTG+tLAzV0Tzf6yRTD
         Dx/GhEY8YfteGJYkFctEHinlCGxBcmACa3F5LSGYPcOebACnBCWZ3oM0J4CHPO0/plwL
         eSLcBgwq4NdRbMMM6Cb5dGuGq7D5CjZZS0Z8w=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=tRJBjbtXZuMK5eVgaqzdI6rWxu+OXqUYxXl2Ur2nYO/VOt4CDpmANcoa/7Ls4rifsR
         F1PB23LM1Pubc31Yi6qzx/WIDLEy7wMcvfgGBnNZX5C/g+OB6nug4dElQ2Bna5guwfYd
         1EsyntPNKjZj2MpDt/zZwOSsCdoaUqaSE59Ak=
MIME-Version: 1.0
Received: by 10.216.150.134 with SMTP id z6mr5889501wej.27.1296473328692; Mon,
 31 Jan 2011 03:28:48 -0800 (PST)
Received: by 10.216.17.146 with HTTP; Mon, 31 Jan 2011 03:28:48 -0800 (PST)
Date: Mon, 31 Jan 2011 09:28:48 -0200
Message-ID: <AANLkTi=ECyVDDNtX=cphZQSEdoGJ+43z3evGqATJdwOf@mail.gmail.com>
Subject: Append file is working in on hadoop 0.20
From: Alessandro Binhara <binhara@gmail.com>
To: general@hadoop.apache.org, cdh-user@cloudera.org
Content-Type: multipart/alternative; boundary=0016e6daa8a537ac17049b22b8d5

--0016e6daa8a537ac17049b22b8d5
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hello ...

I need append file in HDFS ..
I see many forum on internet talking about problems in HDFS to append
files.
That=B4s is correct ?

Append file is working in on hadoop 0.20 ?

thank =B4s

--0016e6daa8a537ac17049b22b8d5--

From general-return-2901-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 31 11:44:35 2011
Return-Path: <general-return-2901-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5672 invoked from network); 31 Jan 2011 11:44:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jan 2011 11:44:35 -0000
Received: (qmail 74728 invoked by uid 500); 31 Jan 2011 11:44:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74474 invoked by uid 500); 31 Jan 2011 11:44:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74466 invoked by uid 99); 31 Jan 2011 11:44:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 11:44:30 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 11:44:22 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 7200BB7D2A
	for <general@hadoop.apache.org>; Mon, 31 Jan 2011 11:44:01 +0000 (GMT)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id NAi5mnkuKTxM for <general@hadoop.apache.org>;
	Mon, 31 Jan 2011 11:44:00 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id A8E64B7D23
	for <general@hadoop.apache.org>; Mon, 31 Jan 2011 11:44:00 +0000 (GMT)
MailScanner-NULL-Check: 1297079026.78107@kxgWFDbwxWOHjEcQY9XgCQ
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p0VBhkCZ016252
	for <general@hadoop.apache.org>; Mon, 31 Jan 2011 11:43:46 GMT
Message-ID: <4D46A072.4080801@apache.org>
Date: Mon, 31 Jan 2011 11:43:46 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSS] Move common, hdfs, mapreduce contrib components to
 apache-extras.org or elsewhere
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com> <AANLkTikLiLjj5_ghvFaW+XApAJ9SrsqZKEJO0+UPFcgW@mail.gmail.com>
In-Reply-To: <AANLkTikLiLjj5_ghvFaW+XApAJ9SrsqZKEJO0+UPFcgW@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p0VBhkCZ016252
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 31/01/11 05:24, Konstantin Boudnik wrote:
> Shall we not dictate a location of contrib projects once they are
> moved of Hadoop? If ppl feel like they are better be served by GitHub
> perhaps they should have an option to get hosted there?


-I see discussions about Git at the ASF infra mailing lists

-the stuff in contrib is code contributed to apache, should still live 
there if we can keep it going. Which means people have to step up, or we 
put it in some attic


From general-return-2902-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 31 11:48:26 2011
Return-Path: <general-return-2902-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6411 invoked from network); 31 Jan 2011 11:48:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jan 2011 11:48:25 -0000
Received: (qmail 81838 invoked by uid 500); 31 Jan 2011 11:48:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81472 invoked by uid 500); 31 Jan 2011 11:48:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81464 invoked by uid 99); 31 Jan 2011 11:48:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 11:48:20 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 11:48:13 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id BBCA2B7D26
	for <general@hadoop.apache.org>; Mon, 31 Jan 2011 11:47:51 +0000 (GMT)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id NELvTByMr6Mf for <general@hadoop.apache.org>;
	Mon, 31 Jan 2011 11:47:51 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id AC121B7D23
	for <general@hadoop.apache.org>; Mon, 31 Jan 2011 11:47:50 +0000 (GMT)
MailScanner-NULL-Check: 1297079252.47102@jgPAvKrTM9w+gzdDGad+5Q
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p0VBlWC0016371
	for <general@hadoop.apache.org>; Mon, 31 Jan 2011 11:47:32 GMT
Message-ID: <4D46A154.2070900@apache.org>
Date: Mon, 31 Jan 2011 11:47:32 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSS] Move common, hdfs, mapreduce contrib components to
 apache-extras.org or elsewhere
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com>
In-Reply-To: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p0VBlWC0016371
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

On 31/01/11 03:42, Nigel Daley wrote:
> Folks,
>
> Now that http://apache-extras.org is launched (https://blogs.apache.org/foundation/entry/the_apache_software_foundation_launches) I'd like to start a discussion on moving contrib components out of common, mapreduce, and hdfs.
>
> These contrib components complicate the builds, cause test failures that nobody seems to care about, have releases that are tied to Hadoop's long release cycles, etc.  Most folks I've talked with agree that these contrib components would be better served by being pulled out of Hadoop and hosted elsewhere. The new apache-extras code hosting site seems like a natural *default* location for migrating these contrib projects.  Perhaps some should graduate from contrib to src (ie from contrib to core of the project they're included in).  If folks agree, we'll need to come up with a mapping of contrib component to it's final destination and file a jira.
>
> Here are the contrib components by project (hopefully I didn't miss any).
>
> Common Contrib:
>    failmon
>    hod
>    test
>
>
> MapReduce Contrib:
>    capacity-scheduler -- move to MR core?
>    data_join
>    dynamic-scheduler
>    eclipse-plugin
>    fairscheduler -- move to MR core?
>    gridmix
>    index
>    mrunit
>    mumak
>    raid
>    sqoop
>    streaming -- move to MR core?
>    vaidya
>    vertica
>

+1 for the schedulers in core
+1 for streaming


For the "accessories",they are really separate projects that work on 
with Hadoop, but could have separate release schedules

  -move them to incubation, try and staff them.
  -if they aren't resourced, then that means they are dead code

I'm -1 to having any support for filesystems other than Posix and HDFS 
in there, =0 on S3, but it's used widely enough it should stay in, 
especially as amazon do apparently provide some funding for testing.

Because, as nigel points out, testing is the enemy. If you don't have 
the implementation of the filesystem in question, there is no way to be 
sure that some change works, you can't use it, release it saying "it 
works", or field bug reports.

Testing and releasing of filesystem interfaces should be the 
responsibility of the filesystem suppliers or whoever wants to develop 
the bridge from the FS to Hadoop.

This raises another issue which I've been thinking of recently, how do 
you define "compatibility". If, for example, my colleagues and I were to 
stand up say "our FS is compatible with Apache Hadoop", what does that 
mean?

-Steve

From general-return-2903-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 31 13:19:31 2011
Return-Path: <general-return-2903-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55132 invoked from network); 31 Jan 2011 13:19:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jan 2011 13:19:30 -0000
Received: (qmail 4466 invoked by uid 500); 31 Jan 2011 13:19:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3934 invoked by uid 500); 31 Jan 2011 13:19:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3926 invoked by uid 99); 31 Jan 2011 13:19:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 13:19:24 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 13:19:16 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id A249CB7D2F
	for <general@hadoop.apache.org>; Mon, 31 Jan 2011 13:18:54 +0000 (GMT)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id fGhKqw63bKVU for <general@hadoop.apache.org>;
	Mon, 31 Jan 2011 13:18:54 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id D1EC2B7D21
	for <general@hadoop.apache.org>; Mon, 31 Jan 2011 13:18:53 +0000 (GMT)
MailScanner-NULL-Check: 1297084719.86479@NGoOeQktCJGo5B4Xtk3S2A
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p0VDIb9n019042
	for <general@hadoop.apache.org>; Mon, 31 Jan 2011 13:18:38 GMT
Message-ID: <4D46B6AD.2020802@apache.org>
Date: Mon, 31 Jan 2011 13:18:37 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Defining Compatibility
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p0VDIb9n019042
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

what does it mean to be compatible with Hadoop? And how do products that 
consider themselves compatible with Hadoop say it?

We have plugin schedulers and the like, and all is well, and the Apache 
brand people keep an eye on distributions of the Hadoop code and make 
sure that Apache Hadoop is cleanly distinguished from redistributions of 
binaries by third parties.

But then you get distributions, and you have to define what is meant in 
terms of functionality and compatibility

Presumably, everyone who issues their own release has either explicitly 
or implicitly done a lot more testing than is in the unit test suite, 
testing that exists to stress test the code on large clusters -is there 
stuff there that needs to be added to SVN to help say a build is of 
sufficiently quality to be released?

Then there are the questions about

-things that work with specific versions/releases of Hadoop?
-replacement filesystems ?
-replacement of core parts of the system, like the MapReduce Engine?

IBM have have been talking about "Hadoop on GPFS"
http://www.almaden.ibm.com/storagesystems/projects/hadoop/

If this is running the MR layer, should it say "Apache Hadoop MR engine 
on top of IBM GPFS", or what -and how do you define or assess 
compatibility at this point? Is it up to the vendor to say "works with 
Apache Hadoop", and is running the Terasort client code sufficient to 
say "compatible"?

Similarly, if the MapReduce engine gets swapped out, what then? We in HP 
Labs have been funding some exploratory work at universities in Berlin 
on an engine that does more operations than just map and reduce, but it 
will also handle the existing operations with API compatibility on the 
worker nodes. The goal here is research with an OSS deliverable, but 
while it may support Hadoop jobs, it's not Hadoop.

What to call such things?


-Steve


From general-return-2904-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 31 14:04:16 2011
Return-Path: <general-return-2904-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67975 invoked from network); 31 Jan 2011 14:04:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jan 2011 14:04:16 -0000
Received: (qmail 60791 invoked by uid 500); 31 Jan 2011 14:04:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60498 invoked by uid 500); 31 Jan 2011 14:04:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60490 invoked by uid 99); 31 Jan 2011 14:04:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 14:04:10 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.169] (HELO mail-qy0-f169.google.com) (209.85.216.169)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 14:04:04 +0000
Received: by qyk7 with SMTP id 7so3066132qyk.14
        for <general@hadoop.apache.org>; Mon, 31 Jan 2011 06:03:43 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=zDUQ2fzszh6CH1d1qTZI4i3IwHhIK7LRndmP/5Y8jF0=;
        b=JDmllXaXapV7uVNRUgdaTjs3DTo5s4qVIZ8lgd2Z2Oh7947rKcvV//GbcqpUhoQALp
         6t1/QrAKX+DQh+cLbXjavmIU6hx/Z3EjdSDiCpS1xmJ+WRhzG6e6/nrL6qiiFNQOMUZB
         irBa5Ae44NAZ1KKDcnqqSHsIKL/+Wbv9X/aaA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=FwYfetqZEzR1losbfMMOij+17RcpNmltDFRxTlA8415tz9Z4yzA6E16Yldb7SP+U1G
         T9Th1jozWI4Qao1zrSBo9sMalLwxTE13KlOLDXBx9IkbNnaV2cu34mlGJa63DhQdT3oG
         L89NVyUlBh+TZZp42yRnW1F2/mQU5WhpQmmig=
Received: by 10.224.37.82 with SMTP id w18mr6537644qad.142.1296482594608;
        Mon, 31 Jan 2011 06:03:14 -0800 (PST)
Received: from [10.180.150.9] (h-64-236-128-41.nat.aol.com [64.236.128.41])
        by mx.google.com with ESMTPS id t7sm14809021qcs.40.2011.01.31.06.03.13
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Mon, 31 Jan 2011 06:03:13 -0800 (PST)
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Append file is working in on hadoop 0.20
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <AANLkTi=ECyVDDNtX=cphZQSEdoGJ+43z3evGqATJdwOf@mail.gmail.com>
Date: Mon, 31 Jan 2011 09:03:12 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <F1A75AB4-66AE-4C5C-8B07-ACFD1C917944@Holsman.NET>
References: <AANLkTi=ECyVDDNtX=cphZQSEdoGJ+43z3evGqATJdwOf@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)

Hi Alessandro.

There is a debate among the experts if the 0.20 version of append is =
stable, and technically the best way of doing it.=20

Saying that there are lots of people and companies who are using it in =
their production clusters with no reported hassles.

I suggest you test it out in your own environment to make sure it works =
for you.=20
This is something you should be doing anyway.=20
=20
On Jan 31, 2011, at 6:28 AM, Alessandro Binhara wrote:

> Hello ...
>=20
> I need append file in HDFS ..
> I see many forum on internet talking about problems in HDFS to append
> files.
> That=B4s is correct ?
>=20
> Append file is working in on hadoop 0.20 ?
>=20
> thank =B4s


From general-return-2905-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 31 14:32:38 2011
Return-Path: <general-return-2905-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85657 invoked from network); 31 Jan 2011 14:32:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jan 2011 14:32:37 -0000
Received: (qmail 2809 invoked by uid 500); 31 Jan 2011 14:32:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2608 invoked by uid 500); 31 Jan 2011 14:32:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2599 invoked by uid 99); 31 Jan 2011 14:32:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 14:32:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 31 Jan 2011 14:32:32 +0000
Received: (qmail 85568 invoked by uid 99); 31 Jan 2011 14:32:11 -0000
Received: from localhost.apache.org (HELO mail-iw0-f176.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 14:32:11 +0000
Received: by iwn2 with SMTP id 2so6059856iwn.35
        for <general@hadoop.apache.org>; Mon, 31 Jan 2011 06:32:10 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.11.130 with SMTP id t2mr382184ibt.49.1296484330597; Mon,
 31 Jan 2011 06:32:10 -0800 (PST)
Received: by 10.231.148.82 with HTTP; Mon, 31 Jan 2011 06:32:10 -0800 (PST)
In-Reply-To: <4D46B6AD.2020802@apache.org>
References: <4D46B6AD.2020802@apache.org>
Date: Mon, 31 Jan 2011 06:32:10 -0800
Message-ID: <AANLkTik2WDc7Y7Kd_708TSJVb5wjAcad6hQsBEg0FtqK@mail.gmail.com>
Subject: Re: Defining Compatibility
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Steve-

It's hard to answer without more concrete criteria. Is this a
trademark question affecting the marketing of a product? A
cross-compatibility taxonomy for users? The minimum criteria to
publish a paper/release a product without eye-rolling? The particular
compatibility claims made by a system will be nuanced and specific; a
runtime that executes MapReduce jobs as they would run in Hadoop can
simply make that claim, whether it uses parts of MapReduce, HDFS, or
neither.

For the various distributions "Powered by Apache Hadoop," one would
assume that compatibility will vary depending on the featureset and
the audience. A distribution that runs MapReduce applications
as-written for Apache Hadoop may be incompatible with a user's
deployed metrics/monitoring system. Some random script to scrape the
UI may not work. The product may only scale to 20 nodes. Whether these
are "compatible with Apache Hadoop" is awkward to answer generally,
unless we want to define the semantics of that phrase by policy.

To put it bluntly, why would we bother to define such a policy? One
could assert that a fully-compatible system would implement all the
public/stable APIs as defined in HADOOP-5073, but who would that help?
And though interoperability is certainly relevant to systems built on
top of Hadoop, is there a reason the Apache project needs to be
involved in defining the standards for compatibility among them?

Compatibility matters, but I'm not clear on the objective of this discussion. -C

On Mon, Jan 31, 2011 at 5:18 AM, Steve Loughran <stevel@apache.org> wrote:
> what does it mean to be compatible with Hadoop? And how do products that
> consider themselves compatible with Hadoop say it?
>
> We have plugin schedulers and the like, and all is well, and the Apache
> brand people keep an eye on distributions of the Hadoop code and make sure
> that Apache Hadoop is cleanly distinguished from redistributions of binaries
> by third parties.
>
> But then you get distributions, and you have to define what is meant in
> terms of functionality and compatibility
>
> Presumably, everyone who issues their own release has either explicitly or
> implicitly done a lot more testing than is in the unit test suite, testing
> that exists to stress test the code on large clusters -is there stuff there
> that needs to be added to SVN to help say a build is of sufficiently quality
> to be released?
>
> Then there are the questions about
>
> -things that work with specific versions/releases of Hadoop?
> -replacement filesystems ?
> -replacement of core parts of the system, like the MapReduce Engine?
>
> IBM have have been talking about "Hadoop on GPFS"
> http://www.almaden.ibm.com/storagesystems/projects/hadoop/
>
> If this is running the MR layer, should it say "Apache Hadoop MR engine on
> top of IBM GPFS", or what -and how do you define or assess compatibility at
> this point? Is it up to the vendor to say "works with Apache Hadoop", and is
> running the Terasort client code sufficient to say "compatible"?
>
> Similarly, if the MapReduce engine gets swapped out, what then? We in HP
> Labs have been funding some exploratory work at universities in Berlin on an
> engine that does more operations than just map and reduce, but it will also
> handle the existing operations with API compatibility on the worker nodes.
> The goal here is research with an OSS deliverable, but while it may support
> Hadoop jobs, it's not Hadoop.
>
> What to call such things?
>
>
> -Steve
>
>

From general-return-2906-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 31 15:41:19 2011
Return-Path: <general-return-2906-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29834 invoked from network); 31 Jan 2011 15:41:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jan 2011 15:41:18 -0000
Received: (qmail 21490 invoked by uid 500); 31 Jan 2011 15:41:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21221 invoked by uid 500); 31 Jan 2011 15:41:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21061 invoked by uid 99); 31 Jan 2011 15:41:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 15:41:12 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 15:41:06 +0000
Received: by qwe4 with SMTP id 4so5774503qwe.35
        for <general@hadoop.apache.org>; Mon, 31 Jan 2011 07:40:44 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=holsman.net; s=google;
        h=domainkey-signature:content-type:mime-version:subject:from
         :in-reply-to:date:content-transfer-encoding:message-id:references:to
         :x-mailer;
        bh=15rK05aFowO3CJMEmdtjU45gfuPBPJOhjdidPOmpM0Q=;
        b=eD0Ue3CkBgaO5KXfABu84Hl5N6EeL08GAdT5GT09WCUC0RLLWa+nW4QuA61jQLa0lY
         56m3kMn0xRkftrd6T8eh7L8AznQqPLYQ0dJa7NxVD7O3vxXgdPNU3gXA+clbSJEi6+HV
         gbYQPmRptS6w/NMpnHDv966A7kUyI6xPSO104=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=holsman.net; s=google;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=Fv25fxip09ZfE+2GMw5/F2MrtciXbvtSVDVwV1WMdhB/Ruv7VvH3W97L8Mi9BRqSQs
         EL55SXrrAUVC8w7Of/4wTPyvvpXt2v6VkmhRcGGNf3KKON973p2yLo4LRJ67riISm2eL
         MEPetZjchShwXOsvJE33m2p9YNMZORVJ7vmwg=
Received: by 10.224.29.21 with SMTP id o21mr6579032qac.144.1296488443402;
        Mon, 31 Jan 2011 07:40:43 -0800 (PST)
Received: from [10.180.150.9] (h-64-236-128-41.nat.aol.com [64.236.128.41])
        by mx.google.com with ESMTPS id l12sm14861430qcu.31.2011.01.31.07.40.42
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Mon, 31 Jan 2011 07:40:42 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
Subject: Re: Defining Compatibility
From: Ian Holsman <hadoop@holsman.net>
In-Reply-To: <4D46B6AD.2020802@apache.org>
Date: Mon, 31 Jan 2011 10:40:41 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <290C79C1-117B-4C8C-84EA-3535CE3DA0B6@Holsman.NET>
References: <4D46B6AD.2020802@apache.org>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1082)


On Jan 31, 2011, at 8:18 AM, Steve Loughran wrote:

> what does it mean to be compatible with Hadoop? And how do products =
that consider themselves compatible with Hadoop say it?

I would like to define it in terms of API's and core functionality.

A product (say hive or pig) will run against a set of well defined APIs =
for a given version.
regardless of who implements the API, it should perform as promised, so =
switching between distributions or implementations (say HDFS over GFS) =
should not give the end user any surprises.

Saying that.. HDFS over GFS may expose a superset of APIs that a tool =
may utilize.=20
If the tool requires those APIs it is no longer compatible with Apache =
Hadoop, and should not be called such.

For example, early versions of HUE required the Thrift API to be =
present. So it would clearly not be compatible version Apache Hadoop =
0.20.

What still perplexes me is what to do when some core functionality (say =
the append patch) that has a identical API to the end-user is promoted =
as compatible...=20
I classify this change as a 'end user surprise', *BUT* HDFS over GFS =
would also have similar surprises, where the API is the same, but =
implemented very differently.

So I'm still not sure if you would classify it as being 0.20 compatible.

>=20
> We have plugin schedulers and the like, and all is well, and the =
Apache brand people keep an eye on distributions of the Hadoop code and =
make sure that Apache Hadoop is cleanly distinguished from =
redistributions of binaries by third parties.
>=20
> But then you get distributions, and you have to define what is meant =
in terms of functionality and compatibility
>=20
> Presumably, everyone who issues their own release has either =
explicitly or implicitly done a lot more testing than is in the unit =
test suite, testing that exists to stress test the code on large =
clusters -is there stuff there that needs to be added to SVN to help say =
a build is of sufficiently quality to be released?
>=20
> Then there are the questions about
>=20
> -things that work with specific versions/releases of Hadoop?
> -replacement filesystems ?
> -replacement of core parts of the system, like the MapReduce Engine?
>=20
> IBM have have been talking about "Hadoop on GPFS"
> http://www.almaden.ibm.com/storagesystems/projects/hadoop/
>=20
> If this is running the MR layer, should it say "Apache Hadoop MR =
engine on top of IBM GPFS", or what -and how do you define or assess =
compatibility at this point? Is it up to the vendor to say "works with =
Apache Hadoop", and is running the Terasort client code sufficient to =
say "compatible"?
>=20
> Similarly, if the MapReduce engine gets swapped out, what then? We in =
HP Labs have been funding some exploratory work at universities in =
Berlin on an engine that does more operations than just map and reduce, =
but it will also handle the existing operations with API compatibility =
on the worker nodes. The goal here is research with an OSS deliverable, =
but while it may support Hadoop jobs, it's not Hadoop.
>=20
> What to call such things?
>=20
>=20
> -Steve
>=20


From general-return-2907-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 31 15:59:58 2011
Return-Path: <general-return-2907-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37516 invoked from network); 31 Jan 2011 15:59:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jan 2011 15:59:57 -0000
Received: (qmail 60827 invoked by uid 500); 31 Jan 2011 15:59:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60075 invoked by uid 500); 31 Jan 2011 15:59:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59598 invoked by uid 99); 31 Jan 2011 15:59:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 15:59:51 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 15:59:41 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id C8E7C1BA427
	for <general@hadoop.apache.org>; Mon, 31 Jan 2011 15:59:20 +0000 (GMT)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id DQ3VeEtd6wvQ for <general@hadoop.apache.org>;
	Mon, 31 Jan 2011 15:59:20 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by colossus.hpl.hp.com (Postfix) with ESMTPS id 580451BA418
	for <general@hadoop.apache.org>; Mon, 31 Jan 2011 15:59:20 +0000 (GMT)
MailScanner-NULL-Check: 1297094346.52959@DoMIeDZN5AQM8svjB2bCaA
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id p0VFx61P023970
	for <general@hadoop.apache.org>; Mon, 31 Jan 2011 15:59:06 GMT
Message-ID: <4D46DC4A.7040700@apache.org>
Date: Mon, 31 Jan 2011 15:59:06 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Defining Compatibility
References: <4D46B6AD.2020802@apache.org> <AANLkTik2WDc7Y7Kd_708TSJVb5wjAcad6hQsBEg0FtqK@mail.gmail.com>
In-Reply-To: <AANLkTik2WDc7Y7Kd_708TSJVb5wjAcad6hQsBEg0FtqK@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: p0VFx61P023970
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

On 31/01/11 14:32, Chris Douglas wrote:
> Steve-
>
> It's hard to answer without more concrete criteria. Is this a
> trademark question affecting the marketing of a product? A
> cross-compatibility taxonomy for users? The minimum criteria to
> publish a paper/release a product without eye-rolling? The particular
> compatibility claims made by a system will be nuanced and specific; a
> runtime that executes MapReduce jobs as they would run in Hadoop can
> simply make that claim, whether it uses parts of MapReduce, HDFS, or
> neither.

No, I'm thinking more about what large scale tests are needed to be run 
against the codebase before you can say "it works", and then how to say 
some changes means that it still works.

>
> For the various distributions "Powered by Apache Hadoop," one would
> assume that compatibility will vary depending on the featureset and
> the audience. A distribution that runs MapReduce applications
> as-written for Apache Hadoop may be incompatible with a user's
> deployed metrics/monitoring system. Some random script to scrape the
> UI may not work. The product may only scale to 20 nodes. Whether these
> are "compatible with Apache Hadoop" is awkward to answer generally,
> unless we want to define the semantics of that phrase by policy.
>
> To put it bluntly, why would we bother to define such a policy? One
> could assert that a fully-compatible system would implement all the
> public/stable APIs as defined in HADOOP-5073, but who would that help?
> And though interoperability is certainly relevant to systems built on
> top of Hadoop, is there a reason the Apache project needs to be
> involved in defining the standards for compatibility among them?

Agreed, I'm just thinking about namings and definitions. Even with the 
stable/unstable internal/external split, there's still the question as 
to what the semantics of operations are, both explicit (this operation 
does X) and implicit (and it takes less than Y seconds to do it). It's 
those implicit things that always catch you out (indeed, they are the 
argument points in things like Java and Java EE compatibility test kits)

From general-return-2908-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 31 16:49:57 2011
Return-Path: <general-return-2908-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 63652 invoked from network); 31 Jan 2011 16:49:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jan 2011 16:49:56 -0000
Received: (qmail 48752 invoked by uid 500); 31 Jan 2011 16:49:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48337 invoked by uid 500); 31 Jan 2011 16:49:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48328 invoked by uid 99); 31 Jan 2011 16:49:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 16:49:49 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 16:49:43 +0000
Received: by wye20 with SMTP id 20so5949170wye.35
        for <general@hadoop.apache.org>; Mon, 31 Jan 2011 08:49:22 -0800 (PST)
Received: by 10.216.7.205 with SMTP id 55mr10423837wep.96.1296492561967; Mon,
 31 Jan 2011 08:49:21 -0800 (PST)
MIME-Version: 1.0
Sender: cos@boudnik.org
Received: by 10.216.59.13 with HTTP; Mon, 31 Jan 2011 08:49:01 -0800 (PST)
In-Reply-To: <FD5BC1D3-881A-4D45-BA2C-C6084C884418@apache.org>
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com> <FD5BC1D3-881A-4D45-BA2C-C6084C884418@apache.org>
From: Konstantin Boudnik <cos@apache.org>
Date: Mon, 31 Jan 2011 08:49:01 -0800
X-Google-Sender-Auth: 9ARYDeF7agxudJpfo9gJ-ft4RrE
Message-ID: <AANLkTim_dNOdj=HUrsx7ZhvG3_QXWerB_EVaakMuT68w@mail.gmail.com>
Subject: Re: [DISCUSS] Move common, hdfs, mapreduce contrib components to
 apache-extras.org or elsewhere
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Sun, Jan 30, 2011 at 23:19, Owen O'Malley <omalley@apache.org> wrote:
>
> On Jan 30, 2011, at 7:42 PM, Nigel Daley wrote:
>
>> Now that http://apache-extras.org is launched
>> (https://blogs.apache.org/foundation/entry/the_apache_software_foundation_launches)
>> I'd like to start a discussion on moving contrib components out of common,
>> mapreduce, and hdfs.
>
> The PMC can't "move" code to Apache extras. It can only choose to abandon
> code that it doesn't want to support any longer. As a separate action some
> group of developers may create projects in Apache Extras based on the code
> from Hadoop.
>
> Therefore the question is really what if any code Hadoop wants to abandon.
> That is a good question and one that we should ask ourselves occasionally.
>
> After a quick consideration, my personal list would look like:
>
> failmon
> fault injection

This is the best way to kill a project as tightly coupled with the
core code as fault injection.

So, if you really want to kill it - then move it.

> fuse-dfs
> hod
> kfs
>
> Also note that pushing code out of Hadoop has a high cost. There are at
> least 3 forks of the hadoop-gpl-compression code. That creates a lot of
> confusion for the users. A lot of users never go to the work to figure out
> which fork and branch of hadoop-gpl-compression work with the version of
> Hadoop they installed.
>
> -- Owen
>
>

From general-return-2909-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 31 16:52:10 2011
Return-Path: <general-return-2909-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 64566 invoked from network); 31 Jan 2011 16:52:10 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jan 2011 16:52:10 -0000
Received: (qmail 55457 invoked by uid 500); 31 Jan 2011 16:52:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55195 invoked by uid 500); 31 Jan 2011 16:52:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55187 invoked by uid 99); 31 Jan 2011 16:52:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 16:52:05 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 16:52:00 +0000
Received: by wwd20 with SMTP id 20so5744262wwd.29
        for <general@hadoop.apache.org>; Mon, 31 Jan 2011 08:51:38 -0800 (PST)
Received: by 10.216.24.207 with SMTP id x57mr6011373wex.89.1296492698116; Mon,
 31 Jan 2011 08:51:38 -0800 (PST)
MIME-Version: 1.0
Received: by 10.216.59.13 with HTTP; Mon, 31 Jan 2011 08:51:17 -0800 (PST)
In-Reply-To: <4D46A072.4080801@apache.org>
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com> <AANLkTikLiLjj5_ghvFaW+XApAJ9SrsqZKEJO0+UPFcgW@mail.gmail.com>
 <4D46A072.4080801@apache.org>
From: Konstantin Boudnik <cos@boudnik.org>
Date: Mon, 31 Jan 2011 08:51:17 -0800
Message-ID: <AANLkTinVd96QZHvbe7pSqY8qWhvUXJeBfZKREmtY-n0-@mail.gmail.com>
Subject: Re: [DISCUSS] Move common, hdfs, mapreduce contrib components to
 apache-extras.org or elsewhere
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Jan 31, 2011 at 03:43, Steve Loughran <stevel@apache.org> wrote:
> On 31/01/11 05:24, Konstantin Boudnik wrote:
>>
>> Shall we not dictate a location of contrib projects once they are
>> moved of Hadoop? If ppl feel like they are better be served by GitHub
>> perhaps they should have an option to get hosted there?
>
>
> -I see discussions about Git at the ASF infra mailing lists

Then I withdraw my earlier opinion about github vs. *-extras

> -the stuff in contrib is code contributed to apache, should still live there
> if we can keep it going. Which means people have to step up, or we put it in
> some attic
>
>

From general-return-2910-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 31 20:15:26 2011
Return-Path: <general-return-2910-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 80644 invoked from network); 31 Jan 2011 20:15:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jan 2011 20:15:25 -0000
Received: (qmail 11527 invoked by uid 500); 31 Jan 2011 20:15:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11356 invoked by uid 500); 31 Jan 2011 20:15:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11340 invoked by uid 99); 31 Jan 2011 20:15:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 20:15:20 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of shv.hadoop@gmail.com designates 209.85.213.176 as permitted sender)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 20:15:12 +0000
Received: by yxm8 with SMTP id 8so2347601yxm.35
        for <multiple recipients>; Mon, 31 Jan 2011 12:14:51 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type;
        bh=feRghFqcEzLFo8GphP8SBjJNHLlPeaoUEOWRZ5Q0qBk=;
        b=thZhWw+Z/tXd+opmECjTz2xVh179yvUzv90PbjdLpH717IwJpC8bjpbgMt0fmH1uhV
         T5poqvhOgrce/QjD4xnzT6l81rOyyu08abqYGzCYtu0K7ywRW5vVociK8kS1aJWDei2r
         4P4Uw7AtJy6MrYha7qNYTPFol2gPOMbuBk85I=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Rf5oFdeWjcYoYdCESMCcbAkMSLbtt5Haxi8aZJI3jO3kZbspGvl2jJu7FLBojh+JIh
         ybT4CwGIXFAvktUKK5WfTg0Vjd7tLt9uDlo9ANRPpEBuAJHxMujrrUEBOM39BgKQMYQ3
         Cfot0rEHHwDQrg5fDG9T+gK77MQT5BEdXhdn4=
MIME-Version: 1.0
Received: by 10.236.102.179 with SMTP id d39mr13450022yhg.48.1296504866302;
 Mon, 31 Jan 2011 12:14:26 -0800 (PST)
Received: by 10.236.108.35 with HTTP; Mon, 31 Jan 2011 12:14:26 -0800 (PST)
In-Reply-To: <AANLkTinB+8KRByZFhRT-y6MiqA0urtKNs8c_XSrhP7RH@mail.gmail.com>
References: <AANLkTinB+8KRByZFhRT-y6MiqA0urtKNs8c_XSrhP7RH@mail.gmail.com>
Date: Mon, 31 Jan 2011 12:14:26 -0800
Message-ID: <AANLkTimmcYh0hY-6ht6vaQ+S=pTGHaFxS6k9qEv=q8Cf@mail.gmail.com>
Subject: Re: Hadoop-common-trunk-Commit is failing since 01/19/2011
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: common-dev@hadoop.apache.org, general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00235448f283016e0a049b2a106b
X-Virus-Checked: Checked by ClamAV on apache.org

--00235448f283016e0a049b2a106b
Content-Type: text/plain; charset=ISO-8859-1

Sending this to general to attract urgent attention.
Both HDFS and MapReduce are not compiling since
HADOOP-6904 and its hdfs and MP counterparts were committed.
The problem is not with this patch as described below, but I think those
commits should be reversed if Common integration build cannot be
restored promptly.

Thanks,
--Konstantin


On Fri, Jan 28, 2011 at 5:53 PM, Konstantin Shvachko
<shv.hadoop@gmail.com>wrote:

> I see Hadoop-common-trunk-Commit is failing and not sending any emails.
> It times out on native compilation and aborts.
> Therefore changes are not integrated, and now it lead to hdfs and mapreduce
> both not compiling.
> Can somebody please take a look at this.
> The last few lines of the build are below.
>
> Thanks
> --Konstantin
>
>     [javah] [Loaded /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/build/classes/org/apache/hadoop/security/JniBasedUnixGroupsMapping.class]
>
>     [javah] [Loaded /homes/hudson/tools/java/jdk1.6.0_11-32/jre/lib/rt.jar(java/lang/Object.class)]
>     [javah] [Forcefully writing file /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/build/native/Linux-i386-32/src/org/apache/hadoop/security/org_apache_hadoop_security_JniBasedUnixGroupsNetgroupMapping.h]
>
>      [exec] checking for gcc... gcc
>      [exec] checking whether the C compiler works... yes
>      [exec] checking for C compiler default output file name... a.out
>      [exec] checking for suffix of executables...
>
> Build timed out. Aborting
> Build was aborted
> [FINDBUGS] Skipping publisher since build result is ABORTED
> Publishing Javadoc
> Archiving artifacts
> Recording test results
> No test report files were found. Configuration error?
>
> Recording fingerprints
>      [exec] Terminated
> Publishing Clover coverage report...
> No Clover report will be published due to a Build Failure
> No emails were triggered.
> Finished: ABORTED
>
>
>
>

--00235448f283016e0a049b2a106b--

From general-return-2911-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 31 20:27:50 2011
Return-Path: <general-return-2911-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85455 invoked from network); 31 Jan 2011 20:27:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jan 2011 20:27:50 -0000
Received: (qmail 34690 invoked by uid 500); 31 Jan 2011 20:27:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34305 invoked by uid 500); 31 Jan 2011 20:27:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34176 invoked by uid 99); 31 Jan 2011 20:27:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 20:27:45 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 20:27:40 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=Tr+w2wEtZzlBXlFBkGQovadaDnLI/3q/CK9tM3BziqHzu7vL9oCfkhQ7
   32g3IiHFBrp+Fm1cloOQ+I1ASmzQtX/+lXuLuXdyXIsvKtniAfHNR73lO
   ZV+GMEEPlo+VeKG;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1296505656; x=1328041656;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20MRUnit|Date:=20Mon,=2031=20Jan=202011
   =2020:28:28=20+0000|Message-ID:=20<A5E67639-4F81-4AC4-B47
   F-B8DF2884F8A4@linkedin.com>|To:=20"<general@hadoop.apach
   e.org>"=20<general@hadoop.apache.org>|MIME-Version:=201.0
   |Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<0343FB866B8FBA43A230B8D80DF51FED@linkedin
   .com>|In-Reply-To:=20<5E01B48E-BFE9-45BA-B729-50E692B74F4
   6@mac.com>|References:=20<AANLkTi=3DfOv-5sKrP9jMDC=3DmJuQ
   jZgh+Tz4LWyDjYiwic@mail.gmail.com>=0D=0A=20<AANLkTi=3D9kZ
   gxwRWyBcDJcjno5WY-uT7gQx=3DFJ9jF=3D3=3DM@mail.gmail.com>
   =0D=0A=20<5E01B48E-BFE9-45BA-B729-50E692B74F46@mac.com>;
  bh=z8v7lOY8/NPDFhLjPHrPbh4AXDkesVV+ScMN+D4qrw8=;
  b=TA7LCArmbA6+71fcVYoyM8Vibk7p6TCEH1Mjlkvt97ROzhOfn1LUl2+7
   4J2oqYpKdEItG8zSrifSlqzvMeb9RRrn3v5vBiHHzNcIclwauR8NmieSN
   BxdA1gYlBT8+l2K;
X-IronPort-AV: E=Sophos;i="4.60,405,1291622400"; 
   d="scan'208";a="20245436"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi id 14.01.0251.000; Mon,
 31 Jan 2011 12:28:29 -0800
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: MRUnit
Thread-Topic: MRUnit
Thread-Index: AQHLwNVOLs/1E8t4xUiBZrChL+Rz9JPq560AgAATY4CAARPkAA==
Date: Mon, 31 Jan 2011 20:28:28 +0000
Message-ID: <A5E67639-4F81-4AC4-B47F-B8DF2884F8A4@linkedin.com>
References: <AANLkTi=fOv-5sKrP9jMDC=mJuQjZgh+Tz4LWyDjYiwic@mail.gmail.com>
 <AANLkTi=9kZgxwRWyBcDJcjno5WY-uT7gQx=FJ9jF=3=M@mail.gmail.com>
 <5E01B48E-BFE9-45BA-B729-50E692B74F46@mac.com>
In-Reply-To: <5E01B48E-BFE9-45BA-B729-50E692B74F46@mac.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <0343FB866B8FBA43A230B8D80DF51FED@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


=09
On Jan 30, 2011, at 7:59 PM, Nigel Daley wrote:

> +1.  I just started a thread on moving all components out of contrib.


	Pushing them randomly to the web 2.0 version of Sourceforge (where they wi=
ll never be seen from again) doesn't sound like a decent, long term strateg=
y.


From general-return-2912-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 31 21:31:56 2011
Return-Path: <general-return-2912-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17843 invoked from network); 31 Jan 2011 21:31:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jan 2011 21:31:55 -0000
Received: (qmail 41775 invoked by uid 500); 31 Jan 2011 21:31:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 41572 invoked by uid 500); 31 Jan 2011 21:31:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 41557 invoked by uid 99); 31 Jan 2011 21:31:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 21:31:51 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of eli@cloudera.com designates 209.85.216.48 as permitted sender)
Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 21:31:45 +0000
Received: by qwe4 with SMTP id 4so6130090qwe.35
        for <multiple recipients>; Mon, 31 Jan 2011 13:31:24 -0800 (PST)
MIME-Version: 1.0
Received: by 10.229.82.80 with SMTP id a16mr6209619qcl.76.1296509484048; Mon,
 31 Jan 2011 13:31:24 -0800 (PST)
Received: by 10.229.215.2 with HTTP; Mon, 31 Jan 2011 13:31:23 -0800 (PST)
In-Reply-To: <AANLkTimmcYh0hY-6ht6vaQ+S=pTGHaFxS6k9qEv=q8Cf@mail.gmail.com>
References: <AANLkTinB+8KRByZFhRT-y6MiqA0urtKNs8c_XSrhP7RH@mail.gmail.com>
	<AANLkTimmcYh0hY-6ht6vaQ+S=pTGHaFxS6k9qEv=q8Cf@mail.gmail.com>
Date: Mon, 31 Jan 2011 13:31:23 -0800
Message-ID: <AANLkTinr_Exrr8VPShGKkBSMzmWX1B9nVALLLTMmvYzd@mail.gmail.com>
Subject: Re: Hadoop-common-trunk-Commit is failing since 01/19/2011
From: Eli Collins <eli@cloudera.com>
To: common-dev@hadoop.apache.org
Cc: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hey Konstantin,

The only build breakage I saw from HADOOP-6904 is MAPREDUCE-2290,
which was fixed.  Trees from trunk are compiling against each other
for me (eg each installed to a local maven repo), perhaps the upstream
maven repo hasn't been updated with the latest bits yet.

Thanks,
Eli

On Mon, Jan 31, 2011 at 12:14 PM, Konstantin Shvachko
<shv.hadoop@gmail.com> wrote:
> Sending this to general to attract urgent attention.
> Both HDFS and MapReduce are not compiling since
> HADOOP-6904 and its hdfs and MP counterparts were committed.
> The problem is not with this patch as described below, but I think those
> commits should be reversed if Common integration build cannot be
> restored promptly.
>
> Thanks,
> --Konstantin
>
>
> On Fri, Jan 28, 2011 at 5:53 PM, Konstantin Shvachko
> <shv.hadoop@gmail.com>wrote:
>
>> I see Hadoop-common-trunk-Commit is failing and not sending any emails.
>> It times out on native compilation and aborts.
>> Therefore changes are not integrated, and now it lead to hdfs and mapred=
uce
>> both not compiling.
>> Can somebody please take a look at this.
>> The last few lines of the build are below.
>>
>> Thanks
>> --Konstantin
>>
>> =A0 =A0 [javah] [Loaded /grid/0/hudson/hudson-slave/workspace/Hadoop-Com=
mon-trunk-Commit/trunk/build/classes/org/apache/hadoop/security/JniBasedUni=
xGroupsMapping.class]
>>
>> =A0 =A0 [javah] [Loaded /homes/hudson/tools/java/jdk1.6.0_11-32/jre/lib/=
rt.jar(java/lang/Object.class)]
>> =A0 =A0 [javah] [Forcefully writing file /grid/0/hudson/hudson-slave/wor=
kspace/Hadoop-Common-trunk-Commit/trunk/build/native/Linux-i386-32/src/org/=
apache/hadoop/security/org_apache_hadoop_security_JniBasedUnixGroupsNetgrou=
pMapping.h]
>>
>> =A0 =A0 =A0[exec] checking for gcc... gcc
>> =A0 =A0 =A0[exec] checking whether the C compiler works... yes
>> =A0 =A0 =A0[exec] checking for C compiler default output file name... a.=
out
>> =A0 =A0 =A0[exec] checking for suffix of executables...
>>
>> Build timed out. Aborting
>> Build was aborted
>> [FINDBUGS] Skipping publisher since build result is ABORTED
>> Publishing Javadoc
>> Archiving artifacts
>> Recording test results
>> No test report files were found. Configuration error?
>>
>> Recording fingerprints
>> =A0 =A0 =A0[exec] Terminated
>> Publishing Clover coverage report...
>> No Clover report will be published due to a Build Failure
>> No emails were triggered.
>> Finished: ABORTED
>>
>>
>>
>>
>

From general-return-2913-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 31 21:37:24 2011
Return-Path: <general-return-2913-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21077 invoked from network); 31 Jan 2011 21:37:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jan 2011 21:37:23 -0000
Received: (qmail 47851 invoked by uid 500); 31 Jan 2011 21:37:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47578 invoked by uid 500); 31 Jan 2011 21:37:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47562 invoked by uid 99); 31 Jan 2011 21:37:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 21:37:19 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.176] (HELO mail-qy0-f176.google.com) (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 21:37:12 +0000
Received: by qyk10 with SMTP id 10so6051927qyk.14
        for <multiple recipients>; Mon, 31 Jan 2011 13:36:51 -0800 (PST)
Received: by 10.229.99.130 with SMTP id u2mr6490338qcn.87.1296509811338; Mon,
 31 Jan 2011 13:36:51 -0800 (PST)
MIME-Version: 1.0
Received: by 10.220.75.83 with HTTP; Mon, 31 Jan 2011 13:36:31 -0800 (PST)
X-Originating-IP: [64.105.168.204]
In-Reply-To: <AANLkTinr_Exrr8VPShGKkBSMzmWX1B9nVALLLTMmvYzd@mail.gmail.com>
References: <AANLkTinB+8KRByZFhRT-y6MiqA0urtKNs8c_XSrhP7RH@mail.gmail.com>
 <AANLkTimmcYh0hY-6ht6vaQ+S=pTGHaFxS6k9qEv=q8Cf@mail.gmail.com> <AANLkTinr_Exrr8VPShGKkBSMzmWX1B9nVALLLTMmvYzd@mail.gmail.com>
From: Ted Dunning <tdunning@maprtech.com>
Date: Mon, 31 Jan 2011 13:36:31 -0800
Message-ID: <AANLkTinrheVYQm-y9n=k7TRK_BdzuZYKAWG+qMWR88ct@mail.gmail.com>
Subject: Re: Hadoop-common-trunk-Commit is failing since 01/19/2011
To: common-dev@hadoop.apache.org
Cc: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016363b845cc0ad47049b2b365e

--0016363b845cc0ad47049b2b365e
Content-Type: text/plain; charset=ISO-8859-1

The has been a problem with more than one build failing (Mahout is the one
that I saw first) due to a change in maven version which meant that the
clover license isn't being found properly.  At least, that is the tale I
heard from infra.

On Mon, Jan 31, 2011 at 1:31 PM, Eli Collins <eli@cloudera.com> wrote:

> Hey Konstantin,
>
> The only build breakage I saw from HADOOP-6904 is MAPREDUCE-2290,
> which was fixed.  Trees from trunk are compiling against each other
> for me (eg each installed to a local maven repo), perhaps the upstream
> maven repo hasn't been updated with the latest bits yet.
>
> Thanks,
> Eli
>
> On Mon, Jan 31, 2011 at 12:14 PM, Konstantin Shvachko
> <shv.hadoop@gmail.com> wrote:
> > Sending this to general to attract urgent attention.
> > Both HDFS and MapReduce are not compiling since
> > HADOOP-6904 and its hdfs and MP counterparts were committed.
> > The problem is not with this patch as described below, but I think those
> > commits should be reversed if Common integration build cannot be
> > restored promptly.
> >
> > Thanks,
> > --Konstantin
> >
> >
> > On Fri, Jan 28, 2011 at 5:53 PM, Konstantin Shvachko
> > <shv.hadoop@gmail.com>wrote:
> >
> >> I see Hadoop-common-trunk-Commit is failing and not sending any emails.
> >> It times out on native compilation and aborts.
> >> Therefore changes are not integrated, and now it lead to hdfs and
> mapreduce
> >> both not compiling.
> >> Can somebody please take a look at this.
> >> The last few lines of the build are below.
> >>
> >> Thanks
> >> --Konstantin
> >>
> >>     [javah] [Loaded
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/build/classes/org/apache/hadoop/security/JniBasedUnixGroupsMapping.class]
> >>
> >>     [javah] [Loaded
> /homes/hudson/tools/java/jdk1.6.0_11-32/jre/lib/rt.jar(java/lang/Object.class)]
> >>     [javah] [Forcefully writing file
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/build/native/Linux-i386-32/src/org/apache/hadoop/security/org_apache_hadoop_security_JniBasedUnixGroupsNetgroupMapping.h]
> >>
> >>      [exec] checking for gcc... gcc
> >>      [exec] checking whether the C compiler works... yes
> >>      [exec] checking for C compiler default output file name... a.out
> >>      [exec] checking for suffix of executables...
> >>
> >> Build timed out. Aborting
> >> Build was aborted
> >> [FINDBUGS] Skipping publisher since build result is ABORTED
> >> Publishing Javadoc
> >> Archiving artifacts
> >> Recording test results
> >> No test report files were found. Configuration error?
> >>
> >> Recording fingerprints
> >>      [exec] Terminated
> >> Publishing Clover coverage report...
> >> No Clover report will be published due to a Build Failure
> >> No emails were triggered.
> >> Finished: ABORTED
> >>
> >>
> >>
> >>
> >
>

--0016363b845cc0ad47049b2b365e--

From general-return-2914-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 31 21:57:34 2011
Return-Path: <general-return-2914-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 32226 invoked from network); 31 Jan 2011 21:57:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jan 2011 21:57:33 -0000
Received: (qmail 91703 invoked by uid 500); 31 Jan 2011 21:57:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91563 invoked by uid 500); 31 Jan 2011 21:57:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91548 invoked by uid 99); 31 Jan 2011 21:57:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 21:57:29 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of shv.hadoop@gmail.com designates 209.85.213.176 as permitted sender)
Received: from [209.85.213.176] (HELO mail-yx0-f176.google.com) (209.85.213.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 21:57:23 +0000
Received: by yxm8 with SMTP id 8so2389450yxm.35
        for <multiple recipients>; Mon, 31 Jan 2011 13:57:02 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:cc:content-type;
        bh=0Czt1ROGjTwaHKxGa8XWk0zoMNyrbdA9WOeA4PUjK2c=;
        b=ULXC3nJoo4zDyiewoZeti6dmxmjhjy+reUYkcRuA6NyeGVDqjC1zTq+uryX28jW+zf
         vMPNdEI5U4xj6fYFXEkm0BGYad6YG4NcJBwEhlnUgPKfX8BltOIqffp31exKrEcO5wUZ
         u2RYw1u/+6KQW5OqFyFOTGb3qP3KgHpLudVPw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=wllqvoxlGrWVYB7KJt7t6Mh1eo0n4yFrIVaGhH2mYuQ1HqjcGO55w/OJKJBnz5Yqmr
         4r+eIwrfRxkrr+DLTdeOJKRmtKMA0nQ/66758m/aPWuC7vTSHLgQ+JdlmjdPtgbLuSAZ
         g6K/qo97gg3X3vkpf8gY+XzTdobKWUsfJNz0E=
MIME-Version: 1.0
Received: by 10.236.95.17 with SMTP id o17mr13694300yhf.35.1296511021757; Mon,
 31 Jan 2011 13:57:01 -0800 (PST)
Received: by 10.236.108.35 with HTTP; Mon, 31 Jan 2011 13:57:01 -0800 (PST)
In-Reply-To: <AANLkTinrheVYQm-y9n=k7TRK_BdzuZYKAWG+qMWR88ct@mail.gmail.com>
References: <AANLkTinB+8KRByZFhRT-y6MiqA0urtKNs8c_XSrhP7RH@mail.gmail.com>
	<AANLkTimmcYh0hY-6ht6vaQ+S=pTGHaFxS6k9qEv=q8Cf@mail.gmail.com>
	<AANLkTinr_Exrr8VPShGKkBSMzmWX1B9nVALLLTMmvYzd@mail.gmail.com>
	<AANLkTinrheVYQm-y9n=k7TRK_BdzuZYKAWG+qMWR88ct@mail.gmail.com>
Date: Mon, 31 Jan 2011 13:57:01 -0800
Message-ID: <AANLkTi=cAhCOkqZ=n9NpBMoPzechoJmfB6KO2yL3QcVA@mail.gmail.com>
Subject: Re: Hadoop-common-trunk-Commit is failing since 01/19/2011
From: Konstantin Shvachko <shv.hadoop@gmail.com>
To: common-dev@hadoop.apache.org
Cc: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0023544fb8dce63645049b2b7ed5
X-Virus-Checked: Checked by ClamAV on apache.org

--0023544fb8dce63645049b2b7ed5
Content-Type: text/plain; charset=ISO-8859-1

Current trunk for HDFS and MapReduce are not compiling at the moment. Try to
build trunk.
This is the result of that changes to common api introduced by HADOOP-6904
are not promoted to HDFS and MR trunks.
HDFS-1335 and MAPREDUCE-2263 depend on these changes.

Common is not promoted to HDFS and MR because Hadoop-Common-trunk-Commit
build is broken. See here.
https://hudson.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-Common-trunk-Commit/

As I see the last successful build was on 01/19, which integrated
HADOOP-6864.
I think this is when JNI changes were introduced, which cannot be digested
by Hudson since then.

Anybody with gcc active could you please verify if the problem is caused by
HADOOP-6864.

Thanks,
--Konstantin

On Mon, Jan 31, 2011 at 1:36 PM, Ted Dunning <tdunning@maprtech.com> wrote:

> The has been a problem with more than one build failing (Mahout is the one
> that I saw first) due to a change in maven version which meant that the
> clover license isn't being found properly.  At least, that is the tale I
> heard from infra.
>
> On Mon, Jan 31, 2011 at 1:31 PM, Eli Collins <eli@cloudera.com> wrote:
>
> > Hey Konstantin,
> >
> > The only build breakage I saw from HADOOP-6904 is MAPREDUCE-2290,
> > which was fixed.  Trees from trunk are compiling against each other
> > for me (eg each installed to a local maven repo), perhaps the upstream
> > maven repo hasn't been updated with the latest bits yet.
> >
> > Thanks,
> > Eli
> >
> > On Mon, Jan 31, 2011 at 12:14 PM, Konstantin Shvachko
> > <shv.hadoop@gmail.com> wrote:
> > > Sending this to general to attract urgent attention.
> > > Both HDFS and MapReduce are not compiling since
> > > HADOOP-6904 and its hdfs and MP counterparts were committed.
> > > The problem is not with this patch as described below, but I think
> those
> > > commits should be reversed if Common integration build cannot be
> > > restored promptly.
> > >
> > > Thanks,
> > > --Konstantin
> > >
> > >
> > > On Fri, Jan 28, 2011 at 5:53 PM, Konstantin Shvachko
> > > <shv.hadoop@gmail.com>wrote:
> > >
> > >> I see Hadoop-common-trunk-Commit is failing and not sending any
> emails.
> > >> It times out on native compilation and aborts.
> > >> Therefore changes are not integrated, and now it lead to hdfs and
> > mapreduce
> > >> both not compiling.
> > >> Can somebody please take a look at this.
> > >> The last few lines of the build are below.
> > >>
> > >> Thanks
> > >> --Konstantin
> > >>
> > >>     [javah] [Loaded
> >
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/build/classes/org/apache/hadoop/security/JniBasedUnixGroupsMapping.class]
> > >>
> > >>     [javah] [Loaded
> >
> /homes/hudson/tools/java/jdk1.6.0_11-32/jre/lib/rt.jar(java/lang/Object.class)]
> > >>     [javah] [Forcefully writing file
> >
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/build/native/Linux-i386-32/src/org/apache/hadoop/security/org_apache_hadoop_security_JniBasedUnixGroupsNetgroupMapping.h]
> > >>
> > >>      [exec] checking for gcc... gcc
> > >>      [exec] checking whether the C compiler works... yes
> > >>      [exec] checking for C compiler default output file name... a.out
> > >>      [exec] checking for suffix of executables...
> > >>
> > >> Build timed out. Aborting
> > >> Build was aborted
> > >> [FINDBUGS] Skipping publisher since build result is ABORTED
> > >> Publishing Javadoc
> > >> Archiving artifacts
> > >> Recording test results
> > >> No test report files were found. Configuration error?
> > >>
> > >> Recording fingerprints
> > >>      [exec] Terminated
> > >> Publishing Clover coverage report...
> > >> No Clover report will be published due to a Build Failure
> > >> No emails were triggered.
> > >> Finished: ABORTED
> > >>
> > >>
> > >>
> > >>
> > >
> >
>

--0023544fb8dce63645049b2b7ed5--

From general-return-2915-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 31 23:11:20 2011
Return-Path: <general-return-2915-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77215 invoked from network); 31 Jan 2011 23:11:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jan 2011 23:11:19 -0000
Received: (qmail 1934 invoked by uid 500); 31 Jan 2011 23:11:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1647 invoked by uid 500); 31 Jan 2011 23:11:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1632 invoked by uid 99); 31 Jan 2011 23:11:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 23:11:17 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates 209.85.214.176 as permitted sender)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 23:11:08 +0000
Received: by iwn2 with SMTP id 2so6573416iwn.35
        for <multiple recipients>; Mon, 31 Jan 2011 15:10:47 -0800 (PST)
Received: by 10.231.36.133 with SMTP id t5mr7349639ibd.12.1296515444839; Mon,
 31 Jan 2011 15:10:44 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.153.19 with HTTP; Mon, 31 Jan 2011 15:10:23 -0800 (PST)
In-Reply-To: <AANLkTi=cAhCOkqZ=n9NpBMoPzechoJmfB6KO2yL3QcVA@mail.gmail.com>
References: <AANLkTinB+8KRByZFhRT-y6MiqA0urtKNs8c_XSrhP7RH@mail.gmail.com>
 <AANLkTimmcYh0hY-6ht6vaQ+S=pTGHaFxS6k9qEv=q8Cf@mail.gmail.com>
 <AANLkTinr_Exrr8VPShGKkBSMzmWX1B9nVALLLTMmvYzd@mail.gmail.com>
 <AANLkTinrheVYQm-y9n=k7TRK_BdzuZYKAWG+qMWR88ct@mail.gmail.com> <AANLkTi=cAhCOkqZ=n9NpBMoPzechoJmfB6KO2yL3QcVA@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 31 Jan 2011 15:10:24 -0800
Message-ID: <AANLkTinkjDpj0Od+3Vhba7xHjNHtfNu2H9yHuJiTOmqL@mail.gmail.com>
Subject: Re: Hadoop-common-trunk-Commit is failing since 01/19/2011
To: general@hadoop.apache.org
Cc: common-dev@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00221540147e891775049b2c868f

--00221540147e891775049b2c868f
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Jan 31, 2011 at 1:57 PM, Konstantin Shvachko
<shv.hadoop@gmail.com>wrote:

>
> Anybody with gcc active could you please verify if the problem is caused by
> HADOOP-6864.
>

I can build common trunk just fine on CentOS 5.5 including native.

I think the issue is somehow isolated to the build machines. Anyone know
what OS they've got? Or can I swing an account on the box where the failures
are happening?

-Todd


> On Mon, Jan 31, 2011 at 1:36 PM, Ted Dunning <tdunning@maprtech.com>
> wrote:
>
> > The has been a problem with more than one build failing (Mahout is the
> one
> > that I saw first) due to a change in maven version which meant that the
> > clover license isn't being found properly.  At least, that is the tale I
> > heard from infra.
> >
> > On Mon, Jan 31, 2011 at 1:31 PM, Eli Collins <eli@cloudera.com> wrote:
> >
> > > Hey Konstantin,
> > >
> > > The only build breakage I saw from HADOOP-6904 is MAPREDUCE-2290,
> > > which was fixed.  Trees from trunk are compiling against each other
> > > for me (eg each installed to a local maven repo), perhaps the upstream
> > > maven repo hasn't been updated with the latest bits yet.
> > >
> > > Thanks,
> > > Eli
> > >
> > > On Mon, Jan 31, 2011 at 12:14 PM, Konstantin Shvachko
> > > <shv.hadoop@gmail.com> wrote:
> > > > Sending this to general to attract urgent attention.
> > > > Both HDFS and MapReduce are not compiling since
> > > > HADOOP-6904 and its hdfs and MP counterparts were committed.
> > > > The problem is not with this patch as described below, but I think
> > those
> > > > commits should be reversed if Common integration build cannot be
> > > > restored promptly.
> > > >
> > > > Thanks,
> > > > --Konstantin
> > > >
> > > >
> > > > On Fri, Jan 28, 2011 at 5:53 PM, Konstantin Shvachko
> > > > <shv.hadoop@gmail.com>wrote:
> > > >
> > > >> I see Hadoop-common-trunk-Commit is failing and not sending any
> > emails.
> > > >> It times out on native compilation and aborts.
> > > >> Therefore changes are not integrated, and now it lead to hdfs and
> > > mapreduce
> > > >> both not compiling.
> > > >> Can somebody please take a look at this.
> > > >> The last few lines of the build are below.
> > > >>
> > > >> Thanks
> > > >> --Konstantin
> > > >>
> > > >>     [javah] [Loaded
> > >
> >
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/build/classes/org/apache/hadoop/security/JniBasedUnixGroupsMapping.class]
> > > >>
> > > >>     [javah] [Loaded
> > >
> >
> /homes/hudson/tools/java/jdk1.6.0_11-32/jre/lib/rt.jar(java/lang/Object.class)]
> > > >>     [javah] [Forcefully writing file
> > >
> >
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/build/native/Linux-i386-32/src/org/apache/hadoop/security/org_apache_hadoop_security_JniBasedUnixGroupsNetgroupMapping.h]
> > > >>
> > > >>      [exec] checking for gcc... gcc
> > > >>      [exec] checking whether the C compiler works... yes
> > > >>      [exec] checking for C compiler default output file name...
> a.out
> > > >>      [exec] checking for suffix of executables...
> > > >>
> > > >> Build timed out. Aborting
> > > >> Build was aborted
> > > >> [FINDBUGS] Skipping publisher since build result is ABORTED
> > > >> Publishing Javadoc
> > > >> Archiving artifacts
> > > >> Recording test results
> > > >> No test report files were found. Configuration error?
> > > >>
> > > >> Recording fingerprints
> > > >>      [exec] Terminated
> > > >> Publishing Clover coverage report...
> > > >> No Clover report will be published due to a Build Failure
> > > >> No emails were triggered.
> > > >> Finished: ABORTED
> > > >>
> > > >>
> > > >>
> > > >>
> > > >
> > >
> >
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--00221540147e891775049b2c868f--

From general-return-2916-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 31 23:12:06 2011
Return-Path: <general-return-2916-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77504 invoked from network); 31 Jan 2011 23:12:06 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jan 2011 23:12:06 -0000
Received: (qmail 4443 invoked by uid 500); 31 Jan 2011 23:12:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4373 invoked by uid 500); 31 Jan 2011 23:12:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4365 invoked by uid 99); 31 Jan 2011 23:12:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 23:12:04 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jghoman@gmail.com designates 209.85.210.176 as permitted sender)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 23:11:59 +0000
Received: by iyb26 with SMTP id 26so6029553iyb.35
        for <general@hadoop.apache.org>; Mon, 31 Jan 2011 15:11:38 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type:content-transfer-encoding;
        bh=xEUlHosdM3wxNOZMS2ZP4kviAlUsfCz0gYNqZRDtRvI=;
        b=tkXkSVS+VGVVJ2V7y3a0/+Yulk1Hf4P23/uBk3Bs0ntGK9khd+xQ3xX4iPARL/X0jm
         cT3ZKIr3HO0X3zcfPUQa/36afIe3Hzp74jpDuujC/phSy5+IaH1NWxEi4m+Nbvimt34V
         A4HLvIBBwwDlaxXPbdS80HHIzaZf2ezb1EV8E=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=ex27jF6nxEnADAup+RsetHg4uWRY1yIo1Ip6SdsFuXaYYxJCVzJYciQAcMB59FrZDt
         9qdjQ9Gbckptych9SNGRIG+RjR2c8wHsFppQxruwmv86Hrv0OCD1R4wI5Mp02LV5rdyF
         JpMLmFE5qA6U6nETwAB03R4xXXQh/kbMzlZFE=
Received: by 10.231.36.12 with SMTP id r12mr7452755ibd.69.1296515498404; Mon,
 31 Jan 2011 15:11:38 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.25.163 with HTTP; Mon, 31 Jan 2011 15:11:18 -0800 (PST)
In-Reply-To: <AANLkTi=cAhCOkqZ=n9NpBMoPzechoJmfB6KO2yL3QcVA@mail.gmail.com>
References: <AANLkTinB+8KRByZFhRT-y6MiqA0urtKNs8c_XSrhP7RH@mail.gmail.com>
 <AANLkTimmcYh0hY-6ht6vaQ+S=pTGHaFxS6k9qEv=q8Cf@mail.gmail.com>
 <AANLkTinr_Exrr8VPShGKkBSMzmWX1B9nVALLLTMmvYzd@mail.gmail.com>
 <AANLkTinrheVYQm-y9n=k7TRK_BdzuZYKAWG+qMWR88ct@mail.gmail.com> <AANLkTi=cAhCOkqZ=n9NpBMoPzechoJmfB6KO2yL3QcVA@mail.gmail.com>
From: Jakob Homan <jghoman@gmail.com>
Date: Mon, 31 Jan 2011 15:11:18 -0800
Message-ID: <AANLkTi=+81g8dFMGMqP5Zc6aAkg5n0dD2QpCVg7ABzd6@mail.gmail.com>
Subject: Re: Hadoop-common-trunk-Commit is failing since 01/19/2011
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

By manually installing a new core jar into the cache, I can compile
trunk.  Looks like we just need to kick a new Core into maven.  Are
there instructions somewhere for committers to do this?  I know Nigel
and Owen know how, but I don't know if the knowledge is diffused past
them.
-Jakob


On Mon, Jan 31, 2011 at 1:57 PM, Konstantin Shvachko
<shv.hadoop@gmail.com> wrote:
> Current trunk for HDFS and MapReduce are not compiling at the moment. Try=
 to
> build trunk.
> This is the result of that changes to common api introduced by HADOOP-690=
4
> are not promoted to HDFS and MR trunks.
> HDFS-1335 and MAPREDUCE-2263 depend on these changes.
>
> Common is not promoted to HDFS and MR because Hadoop-Common-trunk-Commit
> build is broken. See here.
> https://hudson.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-Common-t=
runk-Commit/
>
> As I see the last successful build was on 01/19, which integrated
> HADOOP-6864.
> I think this is when JNI changes were introduced, which cannot be digeste=
d
> by Hudson since then.
>
> Anybody with gcc active could you please verify if the problem is caused =
by
> HADOOP-6864.
>
> Thanks,
> --Konstantin
>
> On Mon, Jan 31, 2011 at 1:36 PM, Ted Dunning <tdunning@maprtech.com> wrot=
e:
>
>> The has been a problem with more than one build failing (Mahout is the o=
ne
>> that I saw first) due to a change in maven version which meant that the
>> clover license isn't being found properly. =A0At least, that is the tale=
 I
>> heard from infra.
>>
>> On Mon, Jan 31, 2011 at 1:31 PM, Eli Collins <eli@cloudera.com> wrote:
>>
>> > Hey Konstantin,
>> >
>> > The only build breakage I saw from HADOOP-6904 is MAPREDUCE-2290,
>> > which was fixed. =A0Trees from trunk are compiling against each other
>> > for me (eg each installed to a local maven repo), perhaps the upstream
>> > maven repo hasn't been updated with the latest bits yet.
>> >
>> > Thanks,
>> > Eli
>> >
>> > On Mon, Jan 31, 2011 at 12:14 PM, Konstantin Shvachko
>> > <shv.hadoop@gmail.com> wrote:
>> > > Sending this to general to attract urgent attention.
>> > > Both HDFS and MapReduce are not compiling since
>> > > HADOOP-6904 and its hdfs and MP counterparts were committed.
>> > > The problem is not with this patch as described below, but I think
>> those
>> > > commits should be reversed if Common integration build cannot be
>> > > restored promptly.
>> > >
>> > > Thanks,
>> > > --Konstantin
>> > >
>> > >
>> > > On Fri, Jan 28, 2011 at 5:53 PM, Konstantin Shvachko
>> > > <shv.hadoop@gmail.com>wrote:
>> > >
>> > >> I see Hadoop-common-trunk-Commit is failing and not sending any
>> emails.
>> > >> It times out on native compilation and aborts.
>> > >> Therefore changes are not integrated, and now it lead to hdfs and
>> > mapreduce
>> > >> both not compiling.
>> > >> Can somebody please take a look at this.
>> > >> The last few lines of the build are below.
>> > >>
>> > >> Thanks
>> > >> --Konstantin
>> > >>
>> > >> =A0 =A0 [javah] [Loaded
>> >
>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/b=
uild/classes/org/apache/hadoop/security/JniBasedUnixGroupsMapping.class]
>> > >>
>> > >> =A0 =A0 [javah] [Loaded
>> >
>> /homes/hudson/tools/java/jdk1.6.0_11-32/jre/lib/rt.jar(java/lang/Object.=
class)]
>> > >> =A0 =A0 [javah] [Forcefully writing file
>> >
>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/b=
uild/native/Linux-i386-32/src/org/apache/hadoop/security/org_apache_hadoop_=
security_JniBasedUnixGroupsNetgroupMapping.h]
>> > >>
>> > >> =A0 =A0 =A0[exec] checking for gcc... gcc
>> > >> =A0 =A0 =A0[exec] checking whether the C compiler works... yes
>> > >> =A0 =A0 =A0[exec] checking for C compiler default output file name.=
.. a.out
>> > >> =A0 =A0 =A0[exec] checking for suffix of executables...
>> > >>
>> > >> Build timed out. Aborting
>> > >> Build was aborted
>> > >> [FINDBUGS] Skipping publisher since build result is ABORTED
>> > >> Publishing Javadoc
>> > >> Archiving artifacts
>> > >> Recording test results
>> > >> No test report files were found. Configuration error?
>> > >>
>> > >> Recording fingerprints
>> > >> =A0 =A0 =A0[exec] Terminated
>> > >> Publishing Clover coverage report...
>> > >> No Clover report will be published due to a Build Failure
>> > >> No emails were triggered.
>> > >> Finished: ABORTED
>> > >>
>> > >>
>> > >>
>> > >>
>> > >
>> >
>>
>

From general-return-2917-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 31 23:23:19 2011
Return-Path: <general-return-2917-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81075 invoked from network); 31 Jan 2011 23:23:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jan 2011 23:23:19 -0000
Received: (qmail 23833 invoked by uid 500); 31 Jan 2011 23:23:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23713 invoked by uid 500); 31 Jan 2011 23:23:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23705 invoked by uid 99); 31 Jan 2011 23:23:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 23:23:17 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=5.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mbhandarkar@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 23:23:12 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:From:To:Subject:Thread-Topic:
   Thread-Index:Date:Message-ID:References:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:x-originating-ip:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version;
  b=jIKrlNpe42cpW2c6lNiW0Net5Racxjf3A1x7/VGAuViUGPYUPYHwslZ/
   TbRDPPYeG22VBhoAiaHTJsttoksCGp4+rYZzIa7qfkk2m9yAEfa8oL47D
   kik8xpXv2sl37O0;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  d=linkedin.com; i=mbhandarkar@linkedin.com; q=dns/txt;
  s=proddkim; t=1296516188; x=1328052188;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Milind=20Bhandarkar=20<mbhandarkar@linkedin.com>
   |Subject:=20Re:=20[DISCUSS]=20Move=20common,=20hdfs,=20ma
   preduce=20contrib=20components=20to=0D=0A=20apache-extras
   .org=20or=20elsewhere|Date:=20Mon,=2031=20Jan=202011=2023
   :24:01=20+0000|Message-ID:=20<E0D47808-02B7-45F7-8A63-3E2
   E4D681EEB@linkedin.com>|To:=20"<general@hadoop.apache.org
   >"=20<general@hadoop.apache.org>|MIME-Version:=201.0
   |Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<C019BFC299A44B4F8F0B630B854BBBCE@linkedin
   .com>|In-Reply-To:=20<FD5BC1D3-881A-4D45-BA2C-C6084C88441
   8@apache.org>|References:=20<FBD8F805-4345-451E-AEA3-F19A
   2545C387@mac.com>=0D=0A=20<FD5BC1D3-881A-4D45-BA2C-C6084C
   884418@apache.org>;
  bh=6TjRZCmxx4wpO32ncpe5LowzSywmwjvWn14uotpAF8Q=;
  b=xcuk49MOmGbrdnxvNYDbiHOMTHgstT+pupoUL+ZXqopeGTEcgRxA5Q+B
   utVMWi4WV6ILpktAUEJW7AZ5hhPAstjW3QHX9kxmpt95OT9YvE6GqTUOp
   DqMG15enOoJ0BsH;
X-IronPort-AV: E=Sophos;i="4.60,406,1291622400"; 
   d="scan'208";a="20253152"
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi id 14.01.0251.000; Mon,
 31 Jan 2011 15:24:01 -0800
From: Milind Bhandarkar <mbhandarkar@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: [DISCUSS] Move common, hdfs, mapreduce contrib components to
 apache-extras.org or elsewhere
Thread-Topic: [DISCUSS] Move common, hdfs, mapreduce contrib components to
 apache-extras.org or elsewhere
Thread-Index: AQHLwPke33yZZGIxw0G/PYBJkE27F5PrMnYAgAENQoA=
Date: Mon, 31 Jan 2011 23:24:01 +0000
Message-ID: <E0D47808-02B7-45F7-8A63-3E2E4D681EEB@linkedin.com>
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com>
 <FD5BC1D3-881A-4D45-BA2C-C6084C884418@apache.org>
In-Reply-To: <FD5BC1D3-881A-4D45-BA2C-C6084C884418@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [172.18.46.247]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <C019BFC299A44B4F8F0B630B854BBBCE@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Owen,

I am surprised to not see jute (aka hadoop recordio) on this list.

- milind

On Jan 30, 2011, at 11:19 PM, Owen O'Malley wrote:

>=20
> On Jan 30, 2011, at 7:42 PM, Nigel Daley wrote:
>=20
>> Now that http://apache-extras.org is launched (https://blogs.apache.org/=
foundation/entry/the_apache_software_foundation_launches) I'd like to start=
 a discussion on moving contrib components out of common, mapreduce, and hd=
fs.
>=20
> The PMC can't "move" code to Apache extras. It can only choose to abandon=
 code that it doesn't want to support any longer. As a separate action some=
 group of developers may create projects in Apache Extras based on the code=
 from Hadoop.
>=20
> Therefore the question is really what if any code Hadoop wants to abandon=
. That is a good question and one that we should ask ourselves occasionally=
.
>=20
> After a quick consideration, my personal list would look like:
>=20
> failmon
> fault injection
> fuse-dfs
> hod
> kfs
>=20
> Also note that pushing code out of Hadoop has a high cost. There are at l=
east 3 forks of the hadoop-gpl-compression code. That creates a lot of conf=
usion for the users. A lot of users never go to the work to figure out whic=
h fork and branch of hadoop-gpl-compression work with the version of Hadoop=
 they installed.
>=20
> -- Owen
>=20

---
Milind Bhandarkar
mbhandarkar@linkedin.com




From general-return-2918-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 31 23:24:00 2011
Return-Path: <general-return-2918-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81276 invoked from network); 31 Jan 2011 23:24:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jan 2011 23:24:00 -0000
Received: (qmail 25778 invoked by uid 500); 31 Jan 2011 23:23:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25716 invoked by uid 500); 31 Jan 2011 23:23:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25708 invoked by uid 99); 31 Jan 2011 23:23:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 23:23:58 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of todd@cloudera.com designates 209.85.210.176 as permitted sender)
Received: from [209.85.210.176] (HELO mail-iy0-f176.google.com) (209.85.210.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 23:23:51 +0000
Received: by iyb26 with SMTP id 26so6038396iyb.35
        for <general@hadoop.apache.org>; Mon, 31 Jan 2011 15:23:30 -0800 (PST)
Received: by 10.231.207.84 with SMTP id fx20mr7337587ibb.62.1296516210476;
 Mon, 31 Jan 2011 15:23:30 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.153.19 with HTTP; Mon, 31 Jan 2011 15:23:10 -0800 (PST)
In-Reply-To: <FD5BC1D3-881A-4D45-BA2C-C6084C884418@apache.org>
References: <FBD8F805-4345-451E-AEA3-F19A2545C387@mac.com> <FD5BC1D3-881A-4D45-BA2C-C6084C884418@apache.org>
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 31 Jan 2011 15:23:10 -0800
Message-ID: <AANLkTikYHbb935h5Qo+YbsYyrAYPSYYoyjrL5APy_unC@mail.gmail.com>
Subject: Re: [DISCUSS] Move common, hdfs, mapreduce contrib components to
 apache-extras.org or elsewhere
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=90e6ba4fc5142bc6e4049b2cb420
X-Virus-Checked: Checked by ClamAV on apache.org

--90e6ba4fc5142bc6e4049b2cb420
Content-Type: text/plain; charset=ISO-8859-1

On Sun, Jan 30, 2011 at 11:19 PM, Owen O'Malley <omalley@apache.org> wrote:

>
> Also note that pushing code out of Hadoop has a high cost. There are at
> least 3 forks of the hadoop-gpl-compression code. That creates a lot of
> confusion for the users. A lot of users never go to the work to figure out
> which fork and branch of hadoop-gpl-compression work with the version of
> Hadoop they installed.
>
>
Indeed it creates confusion, but in my opinion it has been very successful
modulo that confusion.

In particular, Kevin and I (who each have a repo on github but basically
co-maintain a branch) have done about 8 bugfix releases of LZO in the last
year. The ability to take a bug and turn it around into a release within a
few days has been very beneficial to the users. If it were part of core
Hadoop, people would be forced to live with these blocker bugs for months at
a time between dot releases.

IMO the more we can take non-core components and move them to separate
release timelines, the better. Yes, it is harder for users, but it also is
easier for them when they hit a bug - they don't have to wait months for a
wholesale upgrade which might contain hundreds of other changes to core
components. I think this will also help the situation where people have set
up shop on branches -- a lot of the value of these branches comes from the
frequency of backports and bugfixes to "non-core" components. If the
non-core stuff were on a faster timeline upstream, we could maintain core
stability while also offering people the latest and greatest libraries,
tools, codecs, etc.

-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera

--90e6ba4fc5142bc6e4049b2cb420--

From general-return-2919-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 31 23:34:39 2011
Return-Path: <general-return-2919-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84087 invoked from network); 31 Jan 2011 23:34:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jan 2011 23:34:39 -0000
Received: (qmail 34219 invoked by uid 500); 31 Jan 2011 23:34:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34146 invoked by uid 500); 31 Jan 2011 23:34:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34138 invoked by uid 99); 31 Jan 2011 23:34:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 23:34:37 +0000
X-ASF-Spam-Status: No, hits=1.1 required=5.0
	tests=NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Jan 2011 23:34:31 +0000
Received: from SP2-EX07CAS02.ds.corp.yahoo.com (sp2-ex07cas02.corp.sp2.yahoo.com [98.137.59.38])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p0VNXlEJ096189
	for <general@hadoop.apache.org>; Mon, 31 Jan 2011 15:33:47 -0800 (PST)
Received: from SP2-EX07VS02.ds.corp.yahoo.com ([98.137.59.30]) by
 SP2-EX07CAS02.ds.corp.yahoo.com ([98.137.59.38]) with mapi; Mon, 31 Jan 2011
 15:33:47 -0800
From: "Giridharan  Kesavan" <gkesavan@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 31 Jan 2011 15:33:46 -0800
Subject: Re: Hadoop-common-trunk-Commit is failing since 01/19/2011
Thread-Topic: Hadoop-common-trunk-Commit is failing since 01/19/2011
Thread-Index: AcvBn05jdVFXLDX2QMyu6Iz0pmLN5g==
Message-ID: <7DC70A70-5DE9-4288-A44B-B5EF9C454D9B@yahoo-inc.com>
References: <AANLkTinB+8KRByZFhRT-y6MiqA0urtKNs8c_XSrhP7RH@mail.gmail.com>
 <AANLkTimmcYh0hY-6ht6vaQ+S=pTGHaFxS6k9qEv=q8Cf@mail.gmail.com>
 <AANLkTinr_Exrr8VPShGKkBSMzmWX1B9nVALLLTMmvYzd@mail.gmail.com>
 <AANLkTinrheVYQm-y9n=k7TRK_BdzuZYKAWG+qMWR88ct@mail.gmail.com>
 <AANLkTi=cAhCOkqZ=n9NpBMoPzechoJmfB6KO2yL3QcVA@mail.gmail.com>
 <AANLkTi=+81g8dFMGMqP5Zc6aAkg5n0dD2QpCVg7ABzd6@mail.gmail.com>
In-Reply-To: <AANLkTi=+81g8dFMGMqP5Zc6aAkg5n0dD2QpCVg7ABzd6@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

ant mvn-deploy would publish snapshot artifact to the apache maven reposito=
ry as long you have the right credentials in ~/.m2/settings.xml.

For settings.xml template pls look at http://wiki.apache.org/hadoop/HowToRe=
lease

I'm pushing the latest common artifacts now.

-Giri



On Jan 31, 2011, at 3:11 PM, Jakob Homan wrote:

> By manually installing a new core jar into the cache, I can compile
> trunk.  Looks like we just need to kick a new Core into maven.  Are
> there instructions somewhere for committers to do this?  I know Nigel
> and Owen know how, but I don't know if the knowledge is diffused past
> them.
> -Jakob
>=20
>=20
> On Mon, Jan 31, 2011 at 1:57 PM, Konstantin Shvachko
> <shv.hadoop@gmail.com> wrote:
>> Current trunk for HDFS and MapReduce are not compiling at the moment. Tr=
y to
>> build trunk.
>> This is the result of that changes to common api introduced by HADOOP-69=
04
>> are not promoted to HDFS and MR trunks.
>> HDFS-1335 and MAPREDUCE-2263 depend on these changes.
>>=20
>> Common is not promoted to HDFS and MR because Hadoop-Common-trunk-Commit
>> build is broken. See here.
>> https://hudson.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-Common-=
trunk-Commit/
>>=20
>> As I see the last successful build was on 01/19, which integrated
>> HADOOP-6864.
>> I think this is when JNI changes were introduced, which cannot be digest=
ed
>> by Hudson since then.
>>=20
>> Anybody with gcc active could you please verify if the problem is caused=
 by
>> HADOOP-6864.
>>=20
>> Thanks,
>> --Konstantin
>>=20
>> On Mon, Jan 31, 2011 at 1:36 PM, Ted Dunning <tdunning@maprtech.com> wro=
te:
>>=20
>>> The has been a problem with more than one build failing (Mahout is the =
one
>>> that I saw first) due to a change in maven version which meant that the
>>> clover license isn't being found properly.  At least, that is the tale =
I
>>> heard from infra.
>>>=20
>>> On Mon, Jan 31, 2011 at 1:31 PM, Eli Collins <eli@cloudera.com> wrote:
>>>=20
>>>> Hey Konstantin,
>>>>=20
>>>> The only build breakage I saw from HADOOP-6904 is MAPREDUCE-2290,
>>>> which was fixed.  Trees from trunk are compiling against each other
>>>> for me (eg each installed to a local maven repo), perhaps the upstream
>>>> maven repo hasn't been updated with the latest bits yet.
>>>>=20
>>>> Thanks,
>>>> Eli
>>>>=20
>>>> On Mon, Jan 31, 2011 at 12:14 PM, Konstantin Shvachko
>>>> <shv.hadoop@gmail.com> wrote:
>>>>> Sending this to general to attract urgent attention.
>>>>> Both HDFS and MapReduce are not compiling since
>>>>> HADOOP-6904 and its hdfs and MP counterparts were committed.
>>>>> The problem is not with this patch as described below, but I think
>>> those
>>>>> commits should be reversed if Common integration build cannot be
>>>>> restored promptly.
>>>>>=20
>>>>> Thanks,
>>>>> --Konstantin
>>>>>=20
>>>>>=20
>>>>> On Fri, Jan 28, 2011 at 5:53 PM, Konstantin Shvachko
>>>>> <shv.hadoop@gmail.com>wrote:
>>>>>=20
>>>>>> I see Hadoop-common-trunk-Commit is failing and not sending any
>>> emails.
>>>>>> It times out on native compilation and aborts.
>>>>>> Therefore changes are not integrated, and now it lead to hdfs and
>>>> mapreduce
>>>>>> both not compiling.
>>>>>> Can somebody please take a look at this.
>>>>>> The last few lines of the build are below.
>>>>>>=20
>>>>>> Thanks
>>>>>> --Konstantin
>>>>>>=20
>>>>>>     [javah] [Loaded
>>>>=20
>>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/=
build/classes/org/apache/hadoop/security/JniBasedUnixGroupsMapping.class]
>>>>>>=20
>>>>>>     [javah] [Loaded
>>>>=20
>>> /homes/hudson/tools/java/jdk1.6.0_11-32/jre/lib/rt.jar(java/lang/Object=
.class)]
>>>>>>     [javah] [Forcefully writing file
>>>>=20
>>> /grid/0/hudson/hudson-slave/workspace/Hadoop-Common-trunk-Commit/trunk/=
build/native/Linux-i386-32/src/org/apache/hadoop/security/org_apache_hadoop=
_security_JniBasedUnixGroupsNetgroupMapping.h]
>>>>>>=20
>>>>>>      [exec] checking for gcc... gcc
>>>>>>      [exec] checking whether the C compiler works... yes
>>>>>>      [exec] checking for C compiler default output file name... a.ou=
t
>>>>>>      [exec] checking for suffix of executables...
>>>>>>=20
>>>>>> Build timed out. Aborting
>>>>>> Build was aborted
>>>>>> [FINDBUGS] Skipping publisher since build result is ABORTED
>>>>>> Publishing Javadoc
>>>>>> Archiving artifacts
>>>>>> Recording test results
>>>>>> No test report files were found. Configuration error?
>>>>>>=20
>>>>>> Recording fingerprints
>>>>>>      [exec] Terminated
>>>>>> Publishing Clover coverage report...
>>>>>> No Clover report will be published due to a Build Failure
>>>>>> No emails were triggered.
>>>>>> Finished: ABORTED
>>>>>>=20
>>>>>>=20
>>>>>>=20
>>>>>>=20
>>>>>=20
>>>>=20
>>>=20
>>=20


From general-return-113-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 05 19:26:26 2009
Return-Path: <general-return-113-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 75142 invoked from network); 5 Feb 2009 19:26:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 5 Feb 2009 19:26:25 -0000
Received: (qmail 11157 invoked by uid 500); 5 Feb 2009 19:26:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11026 invoked by uid 500); 5 Feb 2009 19:26:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 10929 invoked by uid 99); 5 Feb 2009 19:26:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Feb 2009 11:26:14 -0800
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Feb 2009 19:26:04 +0000
Received: from SNV-EXPF01.ds.corp.yahoo.com (snv-expf01.ds.corp.yahoo.com [207.126.227.250])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n15JOxCZ054669;
	Thu, 5 Feb 2009 11:25:08 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:x-mimeole:content-class:mime-version:
	content-type:subject:date:message-id:x-ms-has-attach:
	x-ms-tnef-correlator:thread-topic:thread-index:from:to:return-path:x-originalarrivaltime;
	b=sUIQ/XZ50J8TlBSreY1fE9LDEEf85QEL8DRorIQuMHHN2tElarSVSwWBZDtB9yO3
Received: from SNV-EXVS02.ds.corp.yahoo.com ([216.145.51.196]) by SNV-EXPF01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Thu, 5 Feb 2009 11:25:04 -0800
x-mimeole: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C987C7.71C09061"
Subject: Hadoop Summit 2009 - Call for submissions
Date: Thu, 5 Feb 2009 11:25:03 -0800
Message-ID: <DD27C9769EA63D43BE3CF4CCCC12DD9702248A0C@SNV-EXVS02.ds.corp.yahoo.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Hadoop Summit 2009 - Call for submissions
Thread-Index: AcmHx3Gv8JhZBWiDTGWZWJ+ZECcFuQ==
From: "Ajay Anand" <aanand@yahoo-inc.com>
To: "core-user@hadoop.apache.org" <'core-user@hadoop.apache.org'>,
        "general@hadoop.apache.org" <'general@hadoop.apache.org'>,
        "zookeeper-user@hadoop.apache.org" <'zookeeper-user@hadoop.apache.org'>,
        "hbase-user@hadoop.apache.org" <'hbase-user@hadoop.apache.org'>,
        <pig-user@hadoop.apache.org>
X-OriginalArrivalTime: 05 Feb 2009 19:25:04.0283 (UTC) FILETIME=[724F96B0:01C987C7]
X-Virus-Checked: Checked by ClamAV on apache.org

------_=_NextPart_001_01C987C7.71C09061
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

We are planning the 2009 Hadoop Summit, to be held the second week of
June in Santa Clara, CA.

=20

Please send me (aanand@yahoo-inc.com) your presentation proposals and
suggested topics.

=20

Areas we plan to cover include:

-          Core Hadoop - areas of development and key contributions

-          Subprojects and related projects: including Pig, Hive, Hbase,
Mahout, Zookeeper and others

-          Administration: Monitoring tools and framework, services,
experiences with administering Hadoop clusters

-          Adoption: Deployment examples, hosted services

-          Applications on Hadoop: examples of innovative applications
being developed and deployed on Hadoop

-          Research: Research topics being explored, university research
examples

=20

Thanks, and looking forward to a strong event,

Ajay


------_=_NextPart_001_01C987C7.71C09061--

From general-return-114-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 11 16:54:49 2009
Return-Path: <general-return-114-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33565 invoked from network); 11 Feb 2009 16:54:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 11 Feb 2009 16:54:49 -0000
Received: (qmail 69340 invoked by uid 500); 11 Feb 2009 16:54:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69186 invoked by uid 500); 11 Feb 2009 16:54:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69173 invoked by uid 99); 11 Feb 2009 16:54:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Feb 2009 08:54:48 -0800
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Feb 2009 16:54:38 +0000
Received: from [10.0.1.195] (snvvpn1-10-72-72-c33.hq.corp.yahoo.com [10.72.72.33])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n1BGrSvf068762
	for <general@hadoop.apache.org>; Wed, 11 Feb 2009 08:53:29 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:content-type:
	content-transfer-encoding:mime-version:subject:date:x-mailer;
	b=zwynOc5sVsSjwxgauL8Jd6E4pWfhcb8NVVO04iIxIWkwYFheF8dpLLWjSvQRwCvX
Message-Id: <2FF6742D-7C82-4EAE-8266-A2E48160B728@yahoo-inc.com>
From: Nigel Daley <ndaley@yahoo-inc.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: Stale committer lists for most Hadoop projects
Date: Wed, 11 Feb 2009 08:53:29 -0800
X-Mailer: Apple Mail (2.930.3)
X-Virus-Checked: Checked by ClamAV on apache.org

It looks like some Hadoop subprojects have stale committer lists.   
Please check your list and have it updated.  I know that Hive is out  
of date.  Others may be as well.

http://hadoop.apache.org/core/credits.html
http://hadoop.apache.org/hbase/credits.html
http://hadoop.apache.org/hive/credits.html
http://hadoop.apache.org/pig/whoweare.html (why isn't this  
credits.html?)
http://hadoop.apache.org/zookeeper/credits.html

Cheers,
Nige

From general-return-115-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 11 18:37:34 2009
Return-Path: <general-return-115-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97729 invoked from network); 11 Feb 2009 18:37:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 11 Feb 2009 18:37:34 -0000
Received: (qmail 84350 invoked by uid 500); 11 Feb 2009 18:37:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84327 invoked by uid 500); 11 Feb 2009 18:37:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84314 invoked by uid 99); 11 Feb 2009 18:37:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Feb 2009 10:37:33 -0800
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 209.85.220.14 as permitted sender)
Received: from [209.85.220.14] (HELO mail-fx0-f14.google.com) (209.85.220.14)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Feb 2009 18:37:26 +0000
Received: by fxm7 with SMTP id 7so640537fxm.5
        for <general@hadoop.apache.org>; Wed, 11 Feb 2009 10:37:05 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=HHMtGe4TqNX38+1rwrnprHCRUtJIM7U0NiMgOSrKmzY=;
        b=EGGgcxJMXP1O1HZEd9QBbaXfy4OzB8TMTWN4+j24XJSE1hLXTuGoNXzo0uwM+RK33j
         YazM2rc92XAr5WaqrctgNLb/ugpTa314f3Zux1wPHUONHXjp281prf/79UwWkyNEaliJ
         tjLFMSymkKqIQiqtGqOtEtseWc6C1hW/hdYHo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=W2AFZy7lgmXEcvxYQkYsVzPm9V8FnYVPxYBYRlbZgjRzwHXRr4+fKewln92512wtpC
         IjR9G6gzPn0aPHJy4jBbab3cMwJ0zEo0WqNmaCwgpvNEi+JlHapYKDRf9CTDRJjYUZVU
         KuP4sb/Ne909OKHqdDcbjPi8r787mwFwQDYfk=
MIME-Version: 1.0
Received: by 10.223.105.208 with SMTP id u16mr236875fao.14.1234377424541; Wed, 
	11 Feb 2009 10:37:04 -0800 (PST)
In-Reply-To: <2FF6742D-7C82-4EAE-8266-A2E48160B728@yahoo-inc.com>
References: <2FF6742D-7C82-4EAE-8266-A2E48160B728@yahoo-inc.com>
Date: Wed, 11 Feb 2009 10:36:58 -0800
Message-ID: <4aa34eb70902111036j2e923045se25388319164262f@mail.gmail.com>
Subject: Re: Stale committer lists for most Hadoop projects
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c599f3e899af0462a8e39e
X-Virus-Checked: Checked by ClamAV on apache.org

--001636c599f3e899af0462a8e39e
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi Nigel,

The Hive committer list is very much upto date. There are two more persons
whose VOTES have been approved but their apache accounts have not yet been
created, that's why you do not see them (yet) in the list.

thanks,
dhruba


On Wed, Feb 11, 2009 at 8:53 AM, Nigel Daley <ndaley@yahoo-inc.com> wrote:

> It looks like some Hadoop subprojects have stale committer lists.  Please
> check your list and have it updated.  I know that Hive is out of date.
>  Others may be as well.
>
> http://hadoop.apache.org/core/credits.html
> http://hadoop.apache.org/hbase/credits.html
> http://hadoop.apache.org/hive/credits.html
> http://hadoop.apache.org/pig/whoweare.html (why isn't this credits.html?)
> http://hadoop.apache.org/zookeeper/credits.html
>
> Cheers,
> Nige
>

--001636c599f3e899af0462a8e39e--

From general-return-116-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 12 21:47:58 2009
Return-Path: <general-return-116-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 62411 invoked from network); 12 Feb 2009 21:47:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 12 Feb 2009 21:47:58 -0000
Received: (qmail 66612 invoked by uid 500); 12 Feb 2009 21:47:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66501 invoked by uid 500); 12 Feb 2009 21:47:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66389 invoked by uid 99); 12 Feb 2009 21:47:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Feb 2009 13:47:43 -0800
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Feb 2009 21:47:33 +0000
Received: from SNV-EXPF01.ds.corp.yahoo.com (snv-expf01.ds.corp.yahoo.com [207.126.227.250])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n1CLkeWB090826;
	Thu, 12 Feb 2009 13:46:40 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:x-mimeole:content-class:mime-version:
	content-type:subject:date:message-id:x-ms-has-attach:
	x-ms-tnef-correlator:thread-topic:thread-index:from:to:cc:return-path:x-originalarrivaltime;
	b=NoZK174FV7V6ZFC7ZAhVgHBrMJy5tppeDYrFd2X2nkCL97fFHkq7+Muqxx8DTezC
Received: from SNV-EXVS02.ds.corp.yahoo.com ([216.145.51.196]) by SNV-EXPF01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Thu, 12 Feb 2009 13:46:40 -0800
x-mimeole: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C98D5B.59AA6101"
Subject: Hadoop User Group Meeting (Bay Area) 2/18
Date: Thu, 12 Feb 2009 13:46:25 -0800
Message-ID: <DD27C9769EA63D43BE3CF4CCCC12DD9702248C4D@SNV-EXVS02.ds.corp.yahoo.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Hadoop User Group Meeting (Bay Area) 2/18
Thread-Index: AcmNW1p3QQcXNKUgTi6+Qtm5ky8HQA==
From: "Ajay Anand" <aanand@yahoo-inc.com>
To: "core-user@hadoop.apache.org" <'core-user@hadoop.apache.org'>,
        "general@hadoop.apache.org" <'general@hadoop.apache.org'>,
        "zookeeper-user@hadoop.apache.org" <'zookeeper-user@hadoop.apache.org'>,
        "hbase-user@hadoop.apache.org" <'hbase-user@hadoop.apache.org'>,
        <pig-user@hadoop.apache.org>
Cc: <matei@eecs.berkeley.edu>, "Aaron Kimball" <aaron@cloudera.com>
X-OriginalArrivalTime: 12 Feb 2009 21:46:40.0211 (UTC) FILETIME=[632B9230:01C98D5B]
X-Virus-Checked: Checked by ClamAV on apache.org

------_=_NextPart_001_01C98D5B.59AA6101
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

The next Bay Area Hadoop User Group meeting is scheduled for Wednesday,
February 18th at Yahoo! 2811 Mission College Blvd, Santa Clara, Building
2, Training Rooms 5 & 6 from 6:00-7:30 pm.

=20

Agenda:

Fair Scheduler for Hadoop - Matei Zaharia

Interfacing with MySQL - Aaron Kimball

=20

Registration: http://upcoming.yahoo.com/event/1776616/

=20

As always, suggestions for topics for future meetings are welcome.
Please send them to me directly at aanand@yahoo-inc.com

=20

Look forward to seeing you there!

Ajay

=20


------_=_NextPart_001_01C98D5B.59AA6101--

From general-return-117-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 13 21:51:31 2009
Return-Path: <general-return-117-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 61456 invoked from network); 13 Feb 2009 21:51:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 13 Feb 2009 21:51:31 -0000
Received: (qmail 15773 invoked by uid 500); 13 Feb 2009 21:51:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14703 invoked by uid 500); 13 Feb 2009 21:51:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14439 invoked by uid 99); 13 Feb 2009 21:51:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Feb 2009 13:51:20 -0800
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [207.126.228.150] (HELO rsmtp2.corp.yahoo.com) (207.126.228.150)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Feb 2009 21:51:12 +0000
Received: from [172.21.148.114] (wlanvpn-mc2e-246-114.corp.yahoo.com [172.21.148.114])
	(authenticated bits=0)
	by rsmtp2.corp.yahoo.com (8.13.8/8.13.8/y.rout) with ESMTP id n1DLollp095468
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Fri, 13 Feb 2009 13:50:47 -0800 (PST)
Message-ID: <4995EB37.6090603@apache.org>
Date: Fri, 13 Feb 2009 13:50:47 -0800
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.19 (X11/20090105)
MIME-Version: 1.0
To: announce@apache.org,
        "zookeeper-dev@hadoop.apache.org" <zookeeper-dev@hadoop.apache.org>,
        "zookeeper-user@hadoop.apache.org" <zookeeper-user@hadoop.apache.org>,
        core-user@hadoop.apache.org, general@hadoop.apache.org
Subject: [ANNOUNCE] Apache ZooKeeper 3.1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

The Apache ZooKeeper team is proud to announce Apache ZooKeeper version 
3.1.0.

ZooKeeper is a high-performance coordination service for distributed 
applications. It exposes common services - such as naming, configuration 
management, synchronization, and group services - in a simple interface 
so you don't have to write them from scratch. You can use it 
off-the-shelf to implement consensus, group management, leader election, 
and presence protocols. And you can build on it for your own, specific 
needs.

Key features of the 3.1.0 release:
* Quota support
* BookKeeper - a system to reliably log streams of records
* JMX for server management
* many fixes, improvements, improved documentation, etc...

A bit about BookKeeper: a system to reliably log streams of records. In 
BookKeeper, servers are "bookies", log streams are "ledgers", and each 
unit of a log (aka record) is a "ledger entry". BookKeeper is designed 
to be reliable; bookies, the servers that store ledgers can be 
byzantine, which means that some subset of the bookies can fail, corrupt 
data, discard data, but as long as there are enough correctly behaving 
servers the service as a whole behaves correctly; the meta data for 
BookKeeper is stored in ZooKeeper.


For ZooKeeper release details and downloads, visit:
http://hadoop.apache.org/zookeeper/releases.html

ZooKeeper 3.1.0 Release Notes are at:
http://hadoop.apache.org/zookeeper/docs/r3.1.0/releasenotes.html

Regards,

The ZooKeeper Team

From general-return-118-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Feb 16 10:41:30 2009
Return-Path: <general-return-118-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33927 invoked from network); 16 Feb 2009 10:41:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 16 Feb 2009 10:41:30 -0000
Received: (qmail 40357 invoked by uid 500); 16 Feb 2009 10:41:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40159 invoked by uid 500); 16 Feb 2009 10:41:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40148 invoked by uid 99); 16 Feb 2009 10:41:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Feb 2009 02:41:28 -0800
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.219.15] (HELO mail-ew0-f15.google.com) (209.85.219.15)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Feb 2009 10:41:19 +0000
Received: by ewy8 with SMTP id 8so1710209ewy.5
        for <general@hadoop.apache.org>; Mon, 16 Feb 2009 02:40:57 -0800 (PST)
Received: by 10.210.45.17 with SMTP id s17mr4264619ebs.126.1234780857362;
        Mon, 16 Feb 2009 02:40:57 -0800 (PST)
Received: from ?10.180.255.165? (hq.last.fm [195.24.233.122])
        by mx.google.com with ESMTPS id j8sm6830989gvb.9.2009.02.16.02.40.55
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Mon, 16 Feb 2009 02:40:56 -0800 (PST)
Message-ID: <499942B2.3000205@oskarsson.nu>
Date: Mon, 16 Feb 2009 10:40:50 +0000
From: Johan Oskarsson <johan@oskarsson.nu>
User-Agent: Thunderbird 2.0.0.19 (X11/20090105)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Hive builds on Hudson - Cobertura or Clover?
X-Enigmail-Version: 0.95.7
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hi.

I've recently setup nightly builds of Apache Hive on Hudson.
The setup is fairly basic and we'd like to add a code coverage tool to
the build process. The options right now are Cobertura or Clover.

I'm leaning towards Cobertura, since that's what I have used before and
it is free for anyone to use. Even if Apache have a Clover license I
assume developers can't install that locally. On the other hand it would
make sense to use the same software as Hadoop does.

I am a bit curious about the Cobertura license though,
http://cobertura.sourceforge.net/license.html
It seems the software is GPL2 but the Ant tasks Apache v1.1

Would this stop us from installing it on the Hudson machine?

http://cobertura.sourceforge.net/
http://www.atlassian.com/software/clover/

/Johan

From general-return-119-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 24 21:29:30 2009
Return-Path: <general-return-119-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 80670 invoked from network); 24 Feb 2009 21:29:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 24 Feb 2009 21:29:30 -0000
Received: (qmail 25329 invoked by uid 500); 24 Feb 2009 21:29:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25309 invoked by uid 500); 24 Feb 2009 21:29:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25298 invoked by uid 99); 24 Feb 2009 21:29:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Feb 2009 13:29:29 -0800
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.30.48] (HELO QMTA05.emeryville.ca.mail.comcast.net) (76.96.30.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Feb 2009 21:29:19 +0000
Received: from OMTA06.emeryville.ca.mail.comcast.net ([76.96.30.51])
	by QMTA05.emeryville.ca.mail.comcast.net with comcast
	id KnqK1b00D16AWCUA5xUxt4; Tue, 24 Feb 2009 21:28:57 +0000
Received: from battlerock-lm.corp.yahoo.com ([209.131.62.113])
	by OMTA06.emeryville.ca.mail.comcast.net with comcast
	id KxUo1b00U2SbwD58SxUr8W; Tue, 24 Feb 2009 21:28:55 +0000
Message-Id: <FEF378EF-190B-4354-BD92-594B06906789@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: [VOTE] Chukwa as subproject
Date: Tue, 24 Feb 2009 13:28:47 -0800
X-Mailer: Apple Mail (2.930.3)
X-Virus-Checked: Checked by ClamAV on apache.org

I'd like to propose Chukwa as a new Hadoop subproject. It is currently  
in Hadoop Core's contrib directory. The initial committers will be:  
Eric Yang, and Ari Rabkin. Clearly, I'm +1. Chukwa is still a new  
project and both of the committers are new, so it will still need some  
hand-holding, but I think it is moving fast enough that it should be  
on its own.

-- Owen

From general-return-120-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 24 21:54:36 2009
Return-Path: <general-return-120-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1650 invoked from network); 24 Feb 2009 21:54:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 24 Feb 2009 21:54:35 -0000
Received: (qmail 53952 invoked by uid 500); 24 Feb 2009 21:54:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53928 invoked by uid 500); 24 Feb 2009 21:54:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53917 invoked by uid 99); 24 Feb 2009 21:54:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Feb 2009 13:54:35 -0800
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Feb 2009 21:54:24 +0000
Received: from SNV-EXPF01.ds.corp.yahoo.com (snv-expf01.ds.corp.yahoo.com [207.126.227.250])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n1OLrG4G068970
	for <general@hadoop.apache.org>; Tue, 24 Feb 2009 13:53:25 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:return-path:x-originalarrivaltime;
	b=nKoSbhZlCPobRb5S1loMhFUNhXx1/BpqR3UCRdVWGOWje6fkmwxaQhkiviMVNa6H
Received: from SNV-EXVS06.ds.corp.yahoo.com ([207.126.227.234]) by SNV-EXPF01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Tue, 24 Feb 2009 13:53:24 -0800
Received: from 10.72.111.153 ([10.72.111.153]) by SNV-EXVS06.ds.corp.yahoo.com ([207.126.227.82]) with Microsoft Exchange Server HTTP-DAV ;
 Tue, 24 Feb 2009 21:53:23 +0000
User-Agent: Microsoft-Entourage/12.15.0.081119
Date: Tue, 24 Feb 2009 13:53:23 -0800
Subject: Re: [VOTE] Chukwa as subproject
From: Eric Yang <eyang@yahoo-inc.com>
To: <general@hadoop.apache.org>
Message-ID: <C5C9AC53.143C4%eyang@yahoo-inc.com>
Thread-Topic: [VOTE] Chukwa as subproject
Thread-Index: AcmWylA1V7PHPREJWkuYWsqMXuJwKQ==
In-Reply-To: <FEF378EF-190B-4354-BD92-594B06906789@apache.org>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-OriginalArrivalTime: 24 Feb 2009 21:53:24.0419 (UTC) FILETIME=[510DD130:01C996CA]
X-Virus-Checked: Checked by ClamAV on apache.org

+1 

This is great for Chukwa to manage it's own release schedule and branches.

Regards,
Eric

On 2/24/09 1:28 PM, "Owen O'Malley" <omalley@apache.org> wrote:

> I'd like to propose Chukwa as a new Hadoop subproject. It is currently
> in Hadoop Core's contrib directory. The initial committers will be:
> Eric Yang, and Ari Rabkin. Clearly, I'm +1. Chukwa is still a new
> project and both of the committers are new, so it will still need some
> hand-holding, but I think it is moving fast enough that it should be
> on its own.
> 
> -- Owen


From general-return-121-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 24 22:22:46 2009
Return-Path: <general-return-121-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27125 invoked from network); 24 Feb 2009 22:22:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 24 Feb 2009 22:22:46 -0000
Received: (qmail 2410 invoked by uid 500); 24 Feb 2009 22:22:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2359 invoked by uid 500); 24 Feb 2009 22:22:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2302 invoked by uid 99); 24 Feb 2009 22:22:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Feb 2009 14:22:38 -0800
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Feb 2009 22:22:28 +0000
Received: from wlan-mc2-146-156.corp.yahoo.com (snvvpn1-10-72-72-c112.hq.corp.yahoo.com [10.72.72.112])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n1OMKqeh062848;
	Tue, 24 Feb 2009 14:20:53 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:content-type:
	content-transfer-encoding:mime-version:subject:date:x-mailer;
	b=1awvUFn2WYnUHPgK76bVLNQUPL9kYXeNaNh9b8OUK1Ra5ZBmxYd9lzorStIiJxZ6
Message-Id: <C607AEFB-EFD7-4BFA-B5F6-50EB6866BCCB@yahoo-inc.com>
From: Nigel Daley <ndaley@yahoo-inc.com>
To: general@hadoop.apache.org, core-user@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: [ANNOUNCE] Hadoop release 0.19.1 available
Date: Tue, 24 Feb 2009 14:20:51 -0800
X-Mailer: Apple Mail (2.930.3)
X-Virus-Checked: Checked by ClamAV on apache.org

Release 0.19.1 fixes many critical bugs in 0.19.0, including ***some  
data loss issues***. The release also introduces an ***incompatible  
change*** by disabling the file append API (HADOOP-5224) until it can  
be stabilized.

For Hadoop release details and downloads, visit:
http://hadoop.apache.org/core/releases.html

Hadoop 0.19.1 Release Notes are at
http://hadoop.apache.org/core/docs/r0.19.1/releasenotes.html

Thanks to all who contributed to this release!

Nigel




From general-return-122-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 24 22:23:20 2009
Return-Path: <general-return-122-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27863 invoked from network); 24 Feb 2009 22:23:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 24 Feb 2009 22:23:20 -0000
Received: (qmail 5507 invoked by uid 500); 24 Feb 2009 22:23:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5484 invoked by uid 500); 24 Feb 2009 22:23:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5473 invoked by uid 99); 24 Feb 2009 22:23:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Feb 2009 14:23:19 -0800
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Feb 2009 22:23:09 +0000
Received: from wlan-mc2-146-156.corp.yahoo.com (snvvpn1-10-72-72-c112.hq.corp.yahoo.com [10.72.72.112])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n1OMKqei062848
	for <general@hadoop.apache.org>; Tue, 24 Feb 2009 14:21:45 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=eCK8ThjfF0WaoYbBB9gLd8Nc733qUwYa+z0U6fsdC0Bi5QNpGNPsF8v6VKqn8L97
Message-Id: <DB389354-A0FE-4832-9637-A745AF5A6DA6@yahoo-inc.com>
From: Nigel Daley <ndaley@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <C5C9AC53.143C4%eyang@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: Re: [VOTE] Chukwa as subproject
Date: Tue, 24 Feb 2009 14:21:43 -0800
References: <C5C9AC53.143C4%eyang@yahoo-inc.com>
X-Mailer: Apple Mail (2.930.3)
X-Virus-Checked: Checked by ClamAV on apache.org

+1.

On Feb 24, 2009, at 1:53 PM, Eric Yang wrote:

> +1
>
> This is great for Chukwa to manage it's own release schedule and  
> branches.
>
> Regards,
> Eric
>
> On 2/24/09 1:28 PM, "Owen O'Malley" <omalley@apache.org> wrote:
>
>> I'd like to propose Chukwa as a new Hadoop subproject. It is  
>> currently
>> in Hadoop Core's contrib directory. The initial committers will be:
>> Eric Yang, and Ari Rabkin. Clearly, I'm +1. Chukwa is still a new
>> project and both of the committers are new, so it will still need  
>> some
>> hand-holding, but I think it is moving fast enough that it should be
>> on its own.
>>
>> -- Owen
>


From general-return-123-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 24 22:32:45 2009
Return-Path: <general-return-123-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35015 invoked from network); 24 Feb 2009 22:32:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 24 Feb 2009 22:32:44 -0000
Received: (qmail 20228 invoked by uid 500); 24 Feb 2009 22:32:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20094 invoked by uid 500); 24 Feb 2009 22:32:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 20083 invoked by uid 99); 24 Feb 2009 22:32:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Feb 2009 14:32:44 -0800
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [206.190.37.39] (HELO web53606.mail.re2.yahoo.com) (206.190.37.39)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 24 Feb 2009 22:32:35 +0000
Received: (qmail 64436 invoked by uid 60001); 24 Feb 2009 22:32:13 -0000
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:MIME-Version:Content-Type:Message-ID;
  b=zshS73NFxM2W7RQGyFLN3b8fiLZmXv9Wen5e7l6l+5xC4V0ehIsW7Zp8dBj6qoskGNdFEaSxNJbwDoJM/odeF0YNUlDLtSgZrltZEnu5trg/uajuSpUGkQXbyEf5YD2ACx0EX/YqPoIu+PBN/bAHkund5zunchC7d5tOFf1DlwU=;
X-YMail-OSG: FYLGpjkVM1kCrMnj1QYMUOwnTmH2ymYBlDpNXiaUlghZYstReLIzrRYZX7Is0BD7oEFmXcY30QJmV0OqOuYED97GlN6KpQJ2XD8Ous4.n91SToBGruEChkwDb2VKhIs1huotm4yXRPrVlFShLKKmoV1NkpDUUUlZnGM9YdAblCyN6mTsp5tS9Lg00LqbGGEXJTyb2mjp6sjT5s.V2QmX8OoHCGoMcQ--
Received: from [4.78.243.126] by web53606.mail.re2.yahoo.com via HTTP; Tue, 24 Feb 2009 14:32:13 PST
X-Mailer: YahooMailRC/1156.82 YahooMailWebService/0.7.260.1
References: <C5C9AC53.143C4%eyang@yahoo-inc.com>
Date: Tue, 24 Feb 2009 14:32:13 -0800 (PST)
From: lohit <lohit_bv@yahoo.com>
Subject: Re: [VOTE] Chukwa as subproject
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Message-ID: <311828.61796.qm@web53606.mail.re2.yahoo.com>
X-Virus-Checked: Checked by ClamAV on apache.org

+1.
Thanks,
Lohit



----- Original Message ----
From: Eric Yang <eyang@yahoo-inc.com>
To: general@hadoop.apache.org
Sent: Tuesday, February 24, 2009 1:53:23 PM
Subject: Re: [VOTE] Chukwa as subproject

+1 

This is great for Chukwa to manage it's own release schedule and branches.

Regards,
Eric

On 2/24/09 1:28 PM, "Owen O'Malley" <omalley@apache.org> wrote:

> I'd like to propose Chukwa as a new Hadoop subproject. It is currently
> in Hadoop Core's contrib directory. The initial committers will be:
> Eric Yang, and Ari Rabkin. Clearly, I'm +1. Chukwa is still a new
> project and both of the committers are new, so it will still need some
> hand-holding, but I think it is moving fast enough that it should be
> on its own.
> 
> -- Owen

From general-return-124-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 24 22:39:36 2009
Return-Path: <general-return-124-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37911 invoked from network); 24 Feb 2009 22:39:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 24 Feb 2009 22:39:36 -0000
Received: (qmail 28367 invoked by uid 500); 24 Feb 2009 22:39:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28334 invoked by uid 500); 24 Feb 2009 22:39:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28323 invoked by uid 99); 24 Feb 2009 22:39:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Feb 2009 14:39:36 -0800
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.220.168] (HELO mail-fx0-f168.google.com) (209.85.220.168)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Feb 2009 22:39:27 +0000
Received: by fxm12 with SMTP id 12so3587020fxm.5
        for <general@hadoop.apache.org>; Tue, 24 Feb 2009 14:39:05 -0800 (PST)
Received: by 10.223.114.208 with SMTP id f16mr168223faq.91.1235515145332;
        Tue, 24 Feb 2009 14:39:05 -0800 (PST)
Received: from ?192.168.1.100? (93-96-139-213.zone4.bethere.co.uk [93.96.139.213])
        by mx.google.com with ESMTPS id 23sm190334eya.36.2009.02.24.14.39.03
        (version=SSLv3 cipher=RC4-MD5);
        Tue, 24 Feb 2009 14:39:04 -0800 (PST)
Message-ID: <49A47703.3060300@oskarsson.nu>
Date: Tue, 24 Feb 2009 22:38:59 +0000
From: Johan Oskarsson <johan@oskarsson.nu>
User-Agent: Thunderbird 2.0.0.19 (Windows/20081209)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Chukwa as subproject
References: <FEF378EF-190B-4354-BD92-594B06906789@apache.org>
In-Reply-To: <FEF378EF-190B-4354-BD92-594B06906789@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Owen O'Malley wrote:
> I'd like to propose Chukwa as a new Hadoop subproject. It is currently 
> in Hadoop Core's contrib directory. The initial committers will be: 
> Eric Yang, and Ari Rabkin. Clearly, I'm +1. Chukwa is still a new 
> project and both of the committers are new, so it will still need some 
> hand-holding, but I think it is moving fast enough that it should be 
> on its own.
>
> -- Owen


From general-return-125-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 24 23:58:03 2009
Return-Path: <general-return-125-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81417 invoked from network); 24 Feb 2009 23:58:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 24 Feb 2009 23:58:02 -0000
Received: (qmail 33274 invoked by uid 500); 24 Feb 2009 23:58:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33062 invoked by uid 500); 24 Feb 2009 23:58:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 53136 invoked by uid 99); 24 Feb 2009 21:52:30 -0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of asrabkin@gmail.com designates 209.85.198.226 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type:content-transfer-encoding;
        bh=kUZYQkjiBXOD/BtaQby7BzyR4MQpxKhwmHAj0cVqIOE=;
        b=hbyn8UjdIr5hDsoDW/7dlAkgk5UkVcHEX8bISCpkXFvejeOLK6w7DP2KVbfyWAZsqa
         +cVP7XgPixZ9AYVW9QXzmM0WRNBEuKvuvcbChjwRkrrJR4T8hc75NkEjOdzz1leVUfkf
         wJ21se6X2tNQ05mKUiv7QclvXBnNMa0+me5bA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=uNGMWaHorv5KYxBgpRS84e11qD/amerTgIMDgnF4C4TYKC6ulvkSqUX5yP8eeeVU9G
         rRPys2e8q7y7Q2pmsL+Z5CJfZWUXG6o+cOL3yAS0594ejGmcgauipZ99+y/FLO5xPWVM
         krILzPT/61jdlWs/V3sI362g0LsGRViU/cWyY=
MIME-Version: 1.0
Date: Tue, 24 Feb 2009 13:52:01 -0800
Message-ID: <39b0afc00902241352j2ab5374fw3589fc3f7fd64db9@mail.gmail.com>
Subject: [VOTE] Chukwa as subproject
From: Ariel Rabkin <asrabkin@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Being a subproject would be good; we're pretty loosely coupled with
Hadoop at this point.

+1.

--Ari

-- 
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department

From general-return-126-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 24 23:58:13 2009
Return-Path: <general-return-126-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81505 invoked from network); 24 Feb 2009 23:58:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 24 Feb 2009 23:58:13 -0000
Received: (qmail 33524 invoked by uid 500); 24 Feb 2009 23:58:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33327 invoked by uid 500); 24 Feb 2009 23:58:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 36064 invoked by uid 99); 24 Feb 2009 22:46:36 -0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 74.125.44.28 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:message-id:subject:from:to
         :content-type;
        bh=8qgKy72UvWC3qkedGOUBwcJnvHH0uz6MqeLg+HG8EKE=;
        b=IVTaqqmRaZIfaMJphTZGVfu11PfHvpRhg5ldTjmsm2x4tkLx/e/xncf6g+tjj6eSmx
         mEon4ttymQtZiexJbMHcq7nALEG/9Bk3wlDq3Bi6vkNkTXuUj/yPxY02G7XoumCso/O9
         wrCzQf6fVJlhB/rEiSmkkz6dIf+nJHwpS+Alw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=ojoc/mNy+P0UQEcxE+/WVt2NKBLSMuVMSEWFd9ZZuU01lLihNEHnTH37b8adR3ov07
         Db7VVX1vfkXbFNosc7CBZvBFqi0a/ypdomp5a03UQFSMECXr5W6ScUBC79vFnMH64oGf
         Lh8b4MLHqomWMXzpCZGSqOtyUJx9zQ8bZ2uS4=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
In-Reply-To: <FEF378EF-190B-4354-BD92-594B06906789@apache.org>
References: <FEF378EF-190B-4354-BD92-594B06906789@apache.org>
Date: Tue, 24 Feb 2009 14:46:08 -0800
X-Google-Sender-Auth: b1e8330f4f750ca7
Message-ID: <7c962aed0902241446x6ed2bc2bx4e75928cb70447aa@mail.gmail.com>
Subject: Re: [VOTE] Chukwa as subproject
From: stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64418dc9a4f260463b1e2a9
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e64418dc9a4f260463b1e2a9
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

+1

On Tue, Feb 24, 2009 at 1:28 PM, Owen O'Malley <omalley@apache.org> wrote:

> I'd like to propose Chukwa as a new Hadoop subproject. It is currently in
> Hadoop Core's contrib directory. The initial committers will be: Eric Yang,
> and Ari Rabkin. Clearly, I'm +1. Chukwa is still a new project and both of
> the committers are new, so it will still need some hand-holding, but I think
> it is moving fast enough that it should be on its own.
>
> -- Owen
>

--0016e64418dc9a4f260463b1e2a9--

From general-return-127-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Feb 24 23:58:27 2009
Return-Path: <general-return-127-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81707 invoked from network); 24 Feb 2009 23:58:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 24 Feb 2009 23:58:27 -0000
Received: (qmail 33854 invoked by uid 500); 24 Feb 2009 23:58:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33829 invoked by uid 500); 24 Feb 2009 23:58:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33818 invoked by uid 99); 24 Feb 2009 23:58:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Feb 2009 15:58:26 -0800
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of tom.e.white@gmail.com designates 209.85.217.170 as permitted sender)
Received: from [209.85.217.170] (HELO mail-gx0-f170.google.com) (209.85.217.170)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Feb 2009 23:58:19 +0000
Received: by gxk18 with SMTP id 18so7877330gxk.5
        for <general@hadoop.apache.org>; Tue, 24 Feb 2009 15:57:58 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=epZbqVAqXRKSMhmDel8eWwEQbWo7fCzcubw/zgFiEU8=;
        b=jQPkkFCtXoDSoAjNdmw0mIqig49X6IRLt+Tig5v/yGSOIBASwpmBva8tjNomYb16Hc
         3I9erjGVTrdDj8udLNeThCYa727k2L0bDdLF7PxAwFFdowXSwPs7oIKHhBt/39FCA6Fg
         Y9aSKFme157VISFJXqdCd7Px3xF+tDkqqhgK8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=x9mOoNVp5ZA/kAQRdl5ICzDbv+ilthJVkR9a/ysV3iA8kusoMP9YposCX3mVzP8uAV
         73iQ4jbAxGb0q+pLk96Jqs7HDR5x92vd/Df1xxPt4rsX3PQqScb2WRkg2d3I+HPJVYRi
         gB9iL/6JPnNKnHHVWkwSpLTSvApxft98/d9gI=
MIME-Version: 1.0
Received: by 10.231.19.204 with SMTP id c12mr383908ibb.20.1235519877920; Tue, 
	24 Feb 2009 15:57:57 -0800 (PST)
In-Reply-To: <49A47703.3060300@oskarsson.nu>
References: <FEF378EF-190B-4354-BD92-594B06906789@apache.org>
	 <49A47703.3060300@oskarsson.nu>
Date: Tue, 24 Feb 2009 15:57:57 -0800
Message-ID: <b23e01b50902241557l149c5a5agbb10c2e8da8babac@mail.gmail.com>
Subject: Re: [VOTE] Chukwa as subproject
From: Tom White <tom.e.white@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Tom

On Tue, Feb 24, 2009 at 2:38 PM, Johan Oskarsson <johan@oskarsson.nu> wrote:
> +1
>
> Owen O'Malley wrote:
>>
>> I'd like to propose Chukwa as a new Hadoop subproject. It is currently in
>> Hadoop Core's contrib directory. The initial committers will be: Eric Yang,
>> and Ari Rabkin. Clearly, I'm +1. Chukwa is still a new project and both of
>> the committers are new, so it will still need some hand-holding, but I think
>> it is moving fast enough that it should be on its own.
>>
>> -- Owen
>
>

From general-return-128-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 25 00:32:00 2009
Return-Path: <general-return-128-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 2806 invoked from network); 25 Feb 2009 00:31:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 25 Feb 2009 00:31:59 -0000
Received: (qmail 64361 invoked by uid 500); 25 Feb 2009 00:31:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 64328 invoked by uid 500); 25 Feb 2009 00:31:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 64317 invoked by uid 99); 25 Feb 2009 00:31:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Feb 2009 16:31:59 -0800
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 209.85.217.170 as permitted sender)
Received: from [209.85.217.170] (HELO mail-gx0-f170.google.com) (209.85.217.170)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Feb 2009 00:31:52 +0000
Received: by gxk18 with SMTP id 18so7921561gxk.5
        for <general@hadoop.apache.org>; Tue, 24 Feb 2009 16:31:31 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=GFh2BwaI31zrmh3xPd8afMdZTdpjfUwr1cZxpRpPcFM=;
        b=G7aZG6mUlYLwGxJeJ6mqFGot46fd9d3g+QEH5+KRFHyZkbQY7netQ8+AA0yr61ne91
         Pd7sP6IhELKgC2b71qJ+E2+KSwev5NJQKeAtnnbKhlz7IxnOVEOOdJi2/HfyuPz/z4+y
         YcpBrbvAES57NFb/J3cgvadqWaLonb2bX2Vg8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=gpom/KqWZEzmUa+2NyY/Zd6xkYSKtxbwPNDHvZTur4Qqi6Ekwj2He/ERYkhowYNbdo
         XsCdtM/4vgQW6JQYYY/fK6mMvQEpmqBYCyDIaV+d7iAaWrEkqnqY9qWEXGjC8WNLI+Pr
         9l2g2gRHwTojVXpEG7ZndDTW1Kg8lrQHl2D2I=
MIME-Version: 1.0
Received: by 10.90.53.5 with SMTP id b5mr2514170aga.91.1235521891497; Tue, 24 
	Feb 2009 16:31:31 -0800 (PST)
In-Reply-To: <b23e01b50902241557l149c5a5agbb10c2e8da8babac@mail.gmail.com>
References: <FEF378EF-190B-4354-BD92-594B06906789@apache.org>
	 <49A47703.3060300@oskarsson.nu>
	 <b23e01b50902241557l149c5a5agbb10c2e8da8babac@mail.gmail.com>
Date: Tue, 24 Feb 2009 16:31:31 -0800
Message-ID: <4aa34eb70902241631p7ff89d67kefa32ff269f6af61@mail.gmail.com>
Subject: Re: [VOTE] Chukwa as subproject
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636283b0a7479210463b35b37
X-Virus-Checked: Checked by ClamAV on apache.org

--001636283b0a7479210463b35b37
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

+1.

On Tue, Feb 24, 2009 at 3:57 PM, Tom White <tom.e.white@gmail.com> wrote:

> +1
>
> Tom
>
> On Tue, Feb 24, 2009 at 2:38 PM, Johan Oskarsson <johan@oskarsson.nu>
> wrote:
> > +1
> >
> > Owen O'Malley wrote:
> >>
> >> I'd like to propose Chukwa as a new Hadoop subproject. It is currently
> in
> >> Hadoop Core's contrib directory. The initial committers will be: Eric
> Yang,
> >> and Ari Rabkin. Clearly, I'm +1. Chukwa is still a new project and both
> of
> >> the committers are new, so it will still need some hand-holding, but I
> think
> >> it is moving fast enough that it should be on its own.
> >>
> >> -- Owen
> >
> >
>

--001636283b0a7479210463b35b37--

From general-return-129-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 25 06:17:40 2009
Return-Path: <general-return-129-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22043 invoked from network); 25 Feb 2009 06:17:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 25 Feb 2009 06:17:39 -0000
Received: (qmail 96773 invoked by uid 500); 25 Feb 2009 06:17:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96746 invoked by uid 500); 25 Feb 2009 06:17:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96735 invoked by uid 99); 25 Feb 2009 06:17:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Feb 2009 22:17:39 -0800
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of angsuman@taragana.com designates 72.36.134.170 as permitted sender)
Received: from [72.36.134.170] (HELO mail.taragana.com) (72.36.134.170)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Feb 2009 06:17:30 +0000
Received: from jaguar.taragana.org (unknown [59.162.191.56])
	by mail.taragana.com (Postfix) with ESMTP id 1DD9B1ED0001
	for <general@hadoop.apache.org>; Wed, 25 Feb 2009 00:16:22 -0600 (CST)
Message-ID: <49A4E262.7080508@taragana.com>
Date: Wed, 25 Feb 2009 11:47:06 +0530
From: Angsuman Chakraborty <angsuman@taragana.com>
User-Agent: Thunderbird 2.0.0.14 (X11/20080421)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [VOTE] Chukwa as subproject
References: <FEF378EF-190B-4354-BD92-594B06906789@apache.org>
In-Reply-To: <FEF378EF-190B-4354-BD92-594B06906789@apache.org>
Content-Type: multipart/mixed;
 boundary="------------010801090907090303060503"
X-Virus-Checked: Checked by ClamAV on apache.org

--------------010801090907090303060503
Content-Type: multipart/alternative;
 boundary="------------090406070009050209050405"


--------------090406070009050209050405
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

+1

Angsuman Chakraborty

Owen O'Malley wrote:
> I'd like to propose Chukwa as a new Hadoop subproject. It is currently 
> in Hadoop Core's contrib directory. The initial committers will be: 
> Eric Yang, and Ari Rabkin. Clearly, I'm +1. Chukwa is still a new 
> project and both of the committers are new, so it will still need some 
> hand-holding, but I think it is moving fast enough that it should be 
> on its own.
>
> -- Owen
>

-- 
Angsuman Chakraborty
CEO Taragana Inc.
http://www.taragana.com | http://blog.taragana.com | http://angsuman.taragana.net

You can follow me on twitter - http://twitter.com/angsuman
AIM: angsuman, Yahoo IM: angsuman


--------------090406070009050209050405
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
  <title></title>
</head>
<body bgcolor="#ffffff" text="#000099">
+1<br>
<br>
Angsuman Chakraborty<br>
<br>
Owen O'Malley wrote:
<blockquote cite="mid:FEF378EF-190B-4354-BD92-594B06906789@apache.org"
 type="cite">I'd like to propose Chukwa as a new Hadoop subproject. It
is currently in Hadoop Core's contrib directory. The initial committers
will be: Eric Yang, and Ari Rabkin. Clearly, I'm +1. Chukwa is still a
new project and both of the committers are new, so it will still need
some hand-holding, but I think it is moving fast enough that it should
be on its own.
  <br>
  <br>
-- Owen
  <br>
  <br>
</blockquote>
<br>
<pre class="moz-signature" cols="72">-- 
Angsuman Chakraborty
CEO Taragana Inc.
<a class="moz-txt-link-freetext" href="http://www.taragana.com">http://www.taragana.com</a> | <a class="moz-txt-link-freetext" href="http://blog.taragana.com">http://blog.taragana.com</a> | <a class="moz-txt-link-freetext" href="http://angsuman.taragana.net">http://angsuman.taragana.net</a>

You can follow me on twitter - <a class="moz-txt-link-freetext" href="http://twitter.com/angsuman">http://twitter.com/angsuman</a>
AIM: angsuman, Yahoo IM: angsuman
</pre>
</body>
</html>

--------------090406070009050209050405--

--------------010801090907090303060503--

From general-return-130-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Feb 25 17:07:26 2009
Return-Path: <general-return-130-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 74167 invoked from network); 25 Feb 2009 17:07:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 25 Feb 2009 17:07:26 -0000
Received: (qmail 45927 invoked by uid 500); 25 Feb 2009 17:07:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45890 invoked by uid 500); 25 Feb 2009 17:07:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 29524 invoked by uid 99); 25 Feb 2009 10:27:42 -0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of work.aviad@gmail.com designates 209.85.218.167 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:message-id:subject:from:to:cc
         :content-type;
        bh=JuQ1oyXNUe4iuZ3P5Y/csNesu/xoxdH6Yr72668jPPc=;
        b=oxNJlS7HqbZv0fclovKKCEkiAvyvSJKKBqpRxhnfAApMai9GOPtmCX5sA5YSmtLvWU
         TvOVzr9+Tl4PVq7VUAjXpNsDkhFxp4x9pI/OJxDEr5xP3b7EYR+4RP388Fk8cqc5TdJS
         UEbMLbj7if6uIiizJBxCoxHIAjiSAvaWIA2HY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:cc:content-type;
        b=sf8Xk/efbuTq4UWX4W7pL1GxvzHXzqu9AeinlJJIJ6BTN57pCwN2rHbdMA8z+Nw8lc
         tBgm/qb/xc/Unj9kB4V8voHrfqdR6TvwH12Jh+htDZSyhnEYBZT+wKPa/Y42nBbl2Vyn
         5aHwpazvLPHi6r36z++ImVeUEuscz5L6KBBjY=
MIME-Version: 1.0
Sender: work.aviad@gmail.com
In-Reply-To: <C607AEFB-EFD7-4BFA-B5F6-50EB6866BCCB@yahoo-inc.com>
References: <C607AEFB-EFD7-4BFA-B5F6-50EB6866BCCB@yahoo-inc.com>
Date: Wed, 25 Feb 2009 12:27:10 +0200
X-Google-Sender-Auth: c027fa681f8c8d06
Message-ID: <6a3f6a4b0902250227m10d46683m62f5ed7737ca5f5d@mail.gmail.com>
Subject: Re: [ANNOUNCE] Hadoop release 0.19.1 available
From: Aviad sela <sela.stam@gmail.com>
To: core-user@hadoop.apache.org
Cc: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f7d272ac4e150463bbad72
X-Virus-Checked: Checked by ClamAV on apache.org

--001485f7d272ac4e150463bbad72
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Nigel,
The SVN tag http://svn.apach.org/repos/asf/core/tags/release-0.19.1  include
also the branch folder branch-0.19
which results with double size project which is not necessary.

I need to use this release to apply patch HADOOP-4546 (
https://issues.apache.org/jira/browse/HADOOP-4546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel
)
I don't know how the above branch folder should contribute when building the
new eclipse project.

Isn't it going to be correct to remove this branch folder from the release
tag ?

Thanks,



On Wed, Feb 25, 2009 at 12:20 AM, Nigel Daley <ndaley@yahoo-inc.com> wrote:

> Release 0.19.1 fixes many critical bugs in 0.19.0, including ***some data
> loss issues***. The release also introduces an ***incompatible change*** by
> disabling the file append API (HADOOP-5224) until it can be stabilized.
>
> For Hadoop release details and downloads, visit:
> http://hadoop.apache.org/core/releases.html
>
> Hadoop 0.19.1 Release Notes are at
> http://hadoop.apache.org/core/docs/r0.19.1/releasenotes.html
>
> Thanks to all who contributed to this release!
>
> Nigel
>
>
>
>

--001485f7d272ac4e150463bbad72--

From general-return-131-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Feb 26 23:45:26 2009
Return-Path: <general-return-131-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58533 invoked from network); 26 Feb 2009 23:45:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 26 Feb 2009 23:45:21 -0000
Received: (qmail 76713 invoked by uid 500); 26 Feb 2009 23:45:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76685 invoked by uid 500); 26 Feb 2009 23:45:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76670 invoked by uid 99); 26 Feb 2009 23:45:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Feb 2009 15:45:20 -0800
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Feb 2009 23:45:09 +0000
Received: from sp1-ex07cas01.ds.corp.yahoo.com (sp1-ex07cas01.ds.corp.yahoo.com [216.252.116.137])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n1QNiY20088869
	for <general@hadoop.apache.org>; Thu, 26 Feb 2009 15:44:34 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:accept-language:content-language:x-ms-has-attach:
	x-ms-tnef-correlator:acceptlanguage:content-type:
	content-transfer-encoding:mime-version;
	b=JkNnEZkoT0dM7MJtcuQIv3bGS4OZZyBgcT+qKl22RnaBlzCG9eNSGL91Kljg/mpm
Received: from SP1-EX07VS02.ds.corp.yahoo.com ([216.252.116.135]) by
 sp1-ex07cas01.ds.corp.yahoo.com ([216.252.116.137]) with mapi; Thu, 26 Feb
 2009 15:44:34 -0800
From: Jerome Boulon <jboulon@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 26 Feb 2009 15:44:32 -0800
Subject: Re: [VOTE] Chukwa as subproject
Thread-Topic: [VOTE] Chukwa as subproject
Thread-Index: AcmYbCwR1qnrY0wx2E6GYihgATjduA==
Message-ID: <C5CC6960.17942%jboulon@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

+1

Owen O'Malley wrote:

I'd like to propose Chukwa as a new Hadoop subproject. It is currently in
Hadoop Core's contrib directory. The initial committers will be: Eric Yang,
and Ari Rabkin. Clearly, I'm +1. Chukwa is still a new project and both of
the committers are new, so it will still need some hand-holding, but I thin=
k
it is moving fast enough that it should be on its own.

-- Owen



From general-return-132-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 27 05:56:30 2009
Return-Path: <general-return-132-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84221 invoked from network); 27 Feb 2009 05:56:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 27 Feb 2009 05:56:30 -0000
Received: (qmail 55334 invoked by uid 500); 27 Feb 2009 05:56:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55298 invoked by uid 500); 27 Feb 2009 05:56:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55287 invoked by uid 99); 27 Feb 2009 05:56:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Feb 2009 21:56:29 -0800
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of 1989.gaurav@googlemail.com designates 74.125.44.30 as permitted sender)
Received: from [74.125.44.30] (HELO yx-out-2324.google.com) (74.125.44.30)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Feb 2009 05:56:20 +0000
Received: by yx-out-2324.google.com with SMTP id 8so628630yxb.29
        for <general@hadoop.apache.org>; Thu, 26 Feb 2009 21:55:59 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:reply-to:date:message-id
         :subject:from:to:content-type;
        bh=BnhdAA7JI0wgH4hOClNbbp2j66lb/mJ5nYDfwaOmRUQ=;
        b=EgbSDW4FTYRzuO67s20WdATFiW3Wr+yqqkvpfSmKd+OTyybT/bOQv4MQdbE270i3Rf
         STvPyS/Zj3kLAsjNxgxLWjBDEw062uQgXOJt4oXw7ZH6hrWe/ZzbOfhtuTWugU9Eq0gE
         +gXQZ/yiQKEHP9qSxIRjC44WImnKR9by7gDog=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:reply-to:date:message-id:subject:from:to:content-type;
        b=pmxRBiSd0Qq+RKljvLAziz14b0HiIh1jKK5ZZmP9ykele42myORAlrepNoBKbkgbf/
         JxeF7LP03o/63LPHOR6CoinubkYdGb7lVDx+2etJ/LQCwHtNf/HHy3cA6viSkwx6Anb9
         2fIax8S9PAyjzWwurZJUsp5fBiJymibuvBB7c=
MIME-Version: 1.0
Received: by 10.110.73.19 with SMTP id v19mr3096907tia.40.1235714158418; Thu, 
	26 Feb 2009 21:55:58 -0800 (PST)
Reply-To: 1989.gaurav@gmail.com
Date: Fri, 27 Feb 2009 11:25:58 +0530
Message-ID: <f4e1c520902262155y59202f91u8b87a31e1b24119a@mail.gmail.com>
Subject: Regarding How to start contributing for Hadoop
From: gaurav gupta <1989.gaurav@googlemail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e65138b674dbdf0463e01f5f
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e65138b674dbdf0463e01f5f
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hello everyone,

I am new to FOSS and I wan to start contributing in hadoop. I came across a
project "hadoop-talend-integration" which was in list of ideas proposed for
GSOC-2008. I wan to know what is the basic requirement and how should I
start for this project. Can anyone mentor me for this project so i can start
contributing in Hadoop. Waiting for reply.

Your's sincerely,
GAURAV GUPTA
B.Tech III Yr. , Department of Computer Science & Engineering
IT BHU , Varanasi
Contacts
Phone No: +91-99569-49491

e-mail :
gaurav.gupta@acm.org
gaurav.gupta.cse06@itbhu.ac.in

--0016e65138b674dbdf0463e01f5f--

From general-return-133-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 27 07:26:03 2009
Return-Path: <general-return-133-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34401 invoked from network); 27 Feb 2009 07:26:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 27 Feb 2009 07:26:03 -0000
Received: (qmail 8783 invoked by uid 500); 27 Feb 2009 07:25:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8738 invoked by uid 500); 27 Feb 2009 07:25:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8717 invoked by uid 99); 27 Feb 2009 07:25:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Feb 2009 23:25:56 -0800
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Feb 2009 07:25:45 +0000
Received: from [10.0.1.195] (snvvpn1-10-72-72-c3.hq.corp.yahoo.com [10.72.72.3])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n1R7OvNN024242;
	Thu, 26 Feb 2009 23:24:57 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=cc:message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=RF6xUQw0hxYIPCuVi0aymW3HCuVsIq9Cu3tN6Mx7uyTqN4zcRiM40R5VeE8D7vVd
Cc: <core-user@hadoop.apache.org>
Message-Id: <A1B8D071-8081-47C9-AB8D-CC39E9216AB6@yahoo-inc.com>
From: Nigel Daley <ndaley@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <6a3f6a4b0902250227m10d46683m62f5ed7737ca5f5d@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: Re: [ANNOUNCE] Hadoop release 0.19.1 available
Date: Thu, 26 Feb 2009 23:24:57 -0800
References: <C607AEFB-EFD7-4BFA-B5F6-50EB6866BCCB@yahoo-inc.com> <6a3f6a4b0902250227m10d46683m62f5ed7737ca5f5d@mail.gmail.com>
X-Mailer: Apple Mail (2.930.3)
X-Virus-Checked: Checked by ClamAV on apache.org

Sorry for delay.  Should now be fixed.  Please confirm.

Thx,
Nige

On Feb 25, 2009, at 2:27 AM, Aviad sela wrote:

> Nigel,
> The SVN tag http://svn.apach.org/repos/asf/core/tags/release-0.19.1   
> include
> also the branch folder branch-0.19
> which results with double size project which is not necessary.
>
> I need to use this release to apply patch HADOOP-4546 (
> https://issues.apache.org/jira/browse/HADOOP-4546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel
> )
> I don't know how the above branch folder should contribute when  
> building the
> new eclipse project.
>
> Isn't it going to be correct to remove this branch folder from the  
> release
> tag ?
>
> Thanks,
>
>
>
> On Wed, Feb 25, 2009 at 12:20 AM, Nigel Daley <ndaley@yahoo-inc.com>  
> wrote:
>
>> Release 0.19.1 fixes many critical bugs in 0.19.0, including  
>> ***some data
>> loss issues***. The release also introduces an ***incompatible  
>> change*** by
>> disabling the file append API (HADOOP-5224) until it can be  
>> stabilized.
>>
>> For Hadoop release details and downloads, visit:
>> http://hadoop.apache.org/core/releases.html
>>
>> Hadoop 0.19.1 Release Notes are at
>> http://hadoop.apache.org/core/docs/r0.19.1/releasenotes.html
>>
>> Thanks to all who contributed to this release!
>>
>> Nigel
>>
>>
>>
>>


From general-return-134-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Feb 27 17:26:34 2009
Return-Path: <general-return-134-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6795 invoked from network); 27 Feb 2009 17:26:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 27 Feb 2009 17:26:34 -0000
Received: (qmail 81166 invoked by uid 500); 27 Feb 2009 17:26:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81139 invoked by uid 500); 27 Feb 2009 17:26:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81128 invoked by uid 99); 27 Feb 2009 17:26:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Feb 2009 09:26:33 -0800
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Feb 2009 17:26:24 +0000
Received: from [192.168.1.64] (snvvpn2-10-72-76-c28.hq.corp.yahoo.com [10.72.76.28])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n1RHPm6d095441
	for <general@hadoop.apache.org>; Fri, 27 Feb 2009 09:25:48 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=FfnGizAn4OUlhBP/MYrxWgVptjuTHHsRDkiE7H0IAHPuu4OGKkTYEfAGpU9TMrAM
Message-Id: <B67465F4-567B-4A07-A633-7A62E3DED1A7@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <FEF378EF-190B-4354-BD92-594B06906789@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: Re: [VOTE] Chukwa as subproject
Date: Fri, 27 Feb 2009 09:25:48 -0800
References: <FEF378EF-190B-4354-BD92-594B06906789@apache.org>
X-Mailer: Apple Mail (2.930.3)
X-Virus-Checked: Checked by ClamAV on apache.org


On Feb 24, 2009, at 1:28 PM, Owen O'Malley wrote:

> I'd like to propose Chukwa as a new Hadoop subproject. It is  
> currently in Hadoop Core's contrib directory. The initial committers  
> will be: Eric Yang, and Ari Rabkin. Clearly, I'm +1. Chukwa is still  
> a new project and both of the committers are new, so it will still  
> need some hand-holding, but I think it is moving fast enough that it  
> should be on its own.
>

+1

Arun

From general-return-1-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 19 07:34:26 2008
Return-Path: <general-return-1-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 78653 invoked from network); 19 Jun 2008 07:34:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 19 Jun 2008 07:34:25 -0000
Received: (qmail 57809 invoked by uid 500); 19 Jun 2008 07:34:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57775 invoked by uid 500); 19 Jun 2008 07:34:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 56433 invoked by uid 99); 19 Jun 2008 07:31:57 -0000
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of afro.systems@gmail.com designates 209.85.146.183 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:sender
         :to:subject:mime-version:content-type:x-google-sender-auth;
        bh=80QkXLWP+32unRBHbGtltTjgyrXicF2lcaA6mOgf4p8=;
        b=OOpRPakNlmTUEYheb4gH/Ab8lPZCaF4Y0pcBA/vihu/DE9+THs0LVu9QSlFsxkthIg
         bYHJn/vypeavhZLzEG9ZjMlIlwK3eK9hwSMU3B+gpvrgItbC10IgfiMwgu9OTRHvqlWI
         jvgK4u8XzSc3R2GkNYyXkaUWycwJI4v5/bPPk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:sender:to:subject:mime-version:content-type
         :x-google-sender-auth;
        b=CM3tpaUfF1SKn5nh5nDK5dLB4RRJzjKUse6LsgPA2pw+dVnUPweDhO4GWPz9uUIgRp
         8iuqfbImVMo24BRPXVAxQ5obSdjeyv8zb4rKlkw6sed3LxwWH+QTzwQhivuCgrhot0cd
         70DAshh2gjE6cZ0//mygAMaXaEAiomnfMu4tQ=
Message-ID: <2f5ea7490806190031n7b030d85u4a3871977211f335@mail.gmail.com>
Date: Thu, 19 Jun 2008 10:31:27 +0300
From: "Tzury Bar Yochay" <tzury.by@gmail.com>
Sender: afro.systems@gmail.com
To: general@hadoop.apache.org
Subject: hdfs appending writes
MIME-Version: 1.0
Content-Type: multipart/alternative; 
	boundary="----=_Part_3215_31766083.1213860687414"
X-Google-Sender-Auth: c9d0ea83a551901f
X-Virus-Checked: Checked by ClamAV on apache.org

------=_Part_3215_31766083.1213860687414
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Hi,

quoting from the documentation at
http://hadoop.apache.org/core/docs/current/hdfs_design.html

Simple Coherency Model

HDFS applications need a write-once-read-many access model for files. A file
once created, written, and closed need not be changed. This assumption
simplifies data coherency issues and enables high throughput data access. A
MapReduce application or a web crawler application fits perfectly with this
model. There is a plan to support appending-writes to files in the future.


I was wondering whether this plan ever implemented.

appending-writes are must in my application. and reliable,searchable and
mapreduable distributed system is a must as well.
thus I looked at hadoop thinking it might be the right platform for the
project.

-- 
Tzury Bar Yochay

------=_Part_3215_31766083.1213860687414--

From general-return-2-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 19 07:41:40 2008
Return-Path: <general-return-2-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 80689 invoked from network); 19 Jun 2008 07:41:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 19 Jun 2008 07:41:38 -0000
Received: (qmail 64145 invoked by uid 500); 19 Jun 2008 07:41:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 64131 invoked by uid 500); 19 Jun 2008 07:41:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 64120 invoked by uid 99); 19 Jun 2008 07:41:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Jun 2008 00:41:40 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 209.85.128.191 as permitted sender)
Received: from [209.85.128.191] (HELO fk-out-0910.google.com) (209.85.128.191)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Jun 2008 07:40:51 +0000
Received: by fk-out-0910.google.com with SMTP id 26so656800fkx.13
        for <general@hadoop.apache.org>; Thu, 19 Jun 2008 00:41:08 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:to
         :subject:in-reply-to:mime-version:content-type
         :content-transfer-encoding:content-disposition:references;
        bh=o0T3x8lRqcVA5ZpxJFJpw3agyhkYpR+RU5Urd9rcpvI=;
        b=rLLCUwBs7ya5HpwxV9iCy3Wi0enEo/61rR5I+S1pTXN2jWsvVz+8kEQCr9OMzSAWZ1
         69EMoJX5MooToyyY+1GVIuAieDHlrdzGXYQRxBkSY2895EzBjPCySX3AKRO4j/f7vJs4
         2v7u0UNZoqd9mB8PKufPINEh9D3IJMBct2rMA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version
         :content-type:content-transfer-encoding:content-disposition
         :references;
        b=JCWNDh6vlKN7x5j4xuvNzpxgvUrYKXWFZuVTHEtbLjbkD74F5KrE8LhZqJcBATTn29
         5yhEa7eHl4qoRsbhQ55bUGDqniHDd9/io6ULTs9EPgxsQ/kcgvpYF/jgPPN4kMNHqBIl
         r/XZ/n7DUflJHDNS/qOLBewbXvUuSTwrDmHRQ=
Received: by 10.82.126.1 with SMTP id y1mr104009buc.20.1213861268427;
        Thu, 19 Jun 2008 00:41:08 -0700 (PDT)
Received: by 10.82.178.17 with HTTP; Thu, 19 Jun 2008 00:41:08 -0700 (PDT)
Message-ID: <4aa34eb70806190041k2669796i170faeb9030382b7@mail.gmail.com>
Date: Thu, 19 Jun 2008 00:41:08 -0700
From: "Dhruba Borthakur" <dhruba@gmail.com>
To: general@hadoop.apache.org
Subject: Re: hdfs appending writes
In-Reply-To: <2f5ea7490806190031n7b030d85u4a3871977211f335@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <2f5ea7490806190031n7b030d85u4a3871977211f335@mail.gmail.com>
X-Virus-Checked: Checked by ClamAV on apache.org

The "append" feature is being worked on. It will probably get into the
0.19 release. details here
http://issues.apache.org/jira/browse/HADOOP-1700

thanks,
dhruba

On Thu, Jun 19, 2008 at 12:31 AM, Tzury Bar Yochay <tzury.by@gmail.com> wrote:
> Hi,
>
> quoting from the documentation at
> http://hadoop.apache.org/core/docs/current/hdfs_design.html
>
> Simple Coherency Model
>
> HDFS applications need a write-once-read-many access model for files. A file
> once created, written, and closed need not be changed. This assumption
> simplifies data coherency issues and enables high throughput data access. A
> MapReduce application or a web crawler application fits perfectly with this
> model. There is a plan to support appending-writes to files in the future.
>
>
> I was wondering whether this plan ever implemented.
>
> appending-writes are must in my application. and reliable,searchable and
> mapreduable distributed system is a must as well.
> thus I looked at hadoop thinking it might be the right platform for the
> project.
>
> --
> Tzury Bar Yochay
>

From general-return-561-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 01 07:48:11 2009
Return-Path: <general-return-561-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56796 invoked from network); 1 Oct 2009 07:48:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Oct 2009 07:48:11 -0000
Received: (qmail 19723 invoked by uid 500); 1 Oct 2009 07:48:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19658 invoked by uid 500); 1 Oct 2009 07:48:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19648 invoked by uid 99); 1 Oct 2009 07:48:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Oct 2009 07:48:10 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [140.203.201.100] (HELO mx1.nuigalway.ie) (140.203.201.100)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Oct 2009 07:47:58 +0000
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: ApsEANr6w0oKhJ0L/2dsb2JhbACPQccThCgE
X-IronPort-AV: E=Sophos;i="4.44,486,1249254000"; 
   d="scan'208";a="37864496"
Received: from unknown (HELO EVS1.ac.nuigalway.ie) ([10.132.157.11])
  by mx1.nuigalway.ie with ESMTP; 01 Oct 2009 08:47:38 +0100
Received: from EVS1.ac.nuigalway.ie ([10.132.157.14]) by EVS1.ac.nuigalway.ie with Microsoft SMTPSVC(6.0.3790.3959);
	 Thu, 1 Oct 2009 08:47:37 +0100
Received: from [10.2.18.121] ([140.203.154.11]) by EVS1.ac.nuigalway.ie over TLS secured channel with Microsoft SMTPSVC(6.0.3790.3959);
	 Thu, 1 Oct 2009 08:47:37 +0100
Message-ID: <4AC45E97.20409@deri.org>
Date: Thu, 01 Oct 2009 08:47:35 +0100
From: stephen mulcahy <stephen.mulcahy@deri.org>
User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: HBase configuration...
References: <440561.64930.qm@web111413.mail.gq1.yahoo.com> <7c962aed0909301442g57a7b4ceg62a84ccb0ff22102@mail.gmail.com>
In-Reply-To: <7c962aed0909301442g57a7b4ceg62a84ccb0ff22102@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-OriginalArrivalTime: 01 Oct 2009 07:47:37.0766 (UTC) FILETIME=[72285460:01CA426B]
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

I could be way off the mark here - but if you are running HBase on top 
of Hadoop in the default configuration then it may be using /tmp for 
HDFS. This gets wiped on reboot which would cause the problems you are 
seeing.

-stephen

stack wrote:
> Shut it down cleanly (./bin/stop-hbase.sh).  My guess is that you are
> killing it before it has chance to flush its in-memory state.
> 
> HBase questions will get more timely response if posted to the hbase lists
> (see hbase.org).
> 
> Yours,
> St.Ack St.Ack
> 
> 
> On Wed, Sep 30, 2009 at 9:22 AM, Something Something <luckyguy2050@yahoo.com
>> wrote:
> 
>> Hello,
>>
>> I noticed that if I start HBase in "standalone" mode it creates a table in
>> memory.  In other words, after rebooting the machine, the table goes away.
>> I would like to persist the table to a local file system.  How can I do
>> that?  Do I have to use "Psuedo Distribution" mode for this?
>>
>> Please help.  Thanks.
>>
>>
>>
>>
>>
> 


-- 
Stephen Mulcahy, DI2, Digital Enterprise Research Institute,
NUI Galway, IDA Business Park, Lower Dangan, Galway, Ireland
http://di2.deri.ie    http://webstar.deri.ie    http://sindice.com

From general-return-562-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 01 07:52:57 2009
Return-Path: <general-return-562-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58129 invoked from network); 1 Oct 2009 07:52:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Oct 2009 07:52:57 -0000
Received: (qmail 26016 invoked by uid 500); 1 Oct 2009 07:52:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25960 invoked by uid 500); 1 Oct 2009 07:52:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25950 invoked by uid 99); 1 Oct 2009 07:52:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Oct 2009 07:52:56 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ryanobjc@gmail.com designates 209.85.210.197 as permitted sender)
Received: from [209.85.210.197] (HELO mail-yx0-f197.google.com) (209.85.210.197)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Oct 2009 07:52:45 +0000
Received: by yxe35 with SMTP id 35so3846271yxe.2
        for <general@hadoop.apache.org>; Thu, 01 Oct 2009 00:52:24 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=nanE5e/+1KUXTDDXi7c5HicA50qwm5ysmS7htPfvrnE=;
        b=b7JjLeCBU3L+Ymz8h7nT8VNe3DyVmUO2cvIEZXH1FDQq/FupeOzJnYFUTJMUQ1frzF
         WYcb5z2mMJDJIKPWv6lF+J6KwjYXwwfn3f+SLvUn+8n0No2XMNh6+sfxPfr5frJ9N7ka
         Mf2qoo73J898Y8dZ4gYrkL1pYbUmuuW6ErQds=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=udvr78CPTH88ecZGrdXu6CuU0wqIvQ+n6miQaKOyt9zXwew7F2nCiCfNA+E+Zs3blM
         EAP8nvtL5eFI2uHGKsncasHDfvLEleNayOT1gWkVqSJ14CZB/2sw2T1jq3mgfPiQM04r
         1/E8TGJu8/GKO3g+fduf/fy1AADo2MiPLb3ug=
MIME-Version: 1.0
Received: by 10.150.141.2 with SMTP id o2mr1736803ybd.49.1254383544195; Thu, 
	01 Oct 2009 00:52:24 -0700 (PDT)
In-Reply-To: <4AC45E97.20409@deri.org>
References: <440561.64930.qm@web111413.mail.gq1.yahoo.com>
	 <7c962aed0909301442g57a7b4ceg62a84ccb0ff22102@mail.gmail.com>
	 <4AC45E97.20409@deri.org>
Date: Thu, 1 Oct 2009 00:52:24 -0700
Message-ID: <78568af10910010052gb67910am3b9c4e183eac0deb@mail.gmail.com>
Subject: Re: HBase configuration...
From: Ryan Rawson <ryanobjc@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

yes, the standalone version uses file:///tmp to store data, to allow
people to get up and running without HDFS.  And you are exactly
correct, with /tmp being cleaned on reboot on many distros.  Grep for
/tmp in hbase-default.xml to find the setting.

-ryan

On Thu, Oct 1, 2009 at 12:47 AM, stephen mulcahy
<stephen.mulcahy@deri.org> wrote:
> Hi,
>
> I could be way off the mark here - but if you are running HBase on top of
> Hadoop in the default configuration then it may be using /tmp for HDFS. T=
his
> gets wiped on reboot which would cause the problems you are seeing.
>
> -stephen
>
> stack wrote:
>>
>> Shut it down cleanly (./bin/stop-hbase.sh). =A0My guess is that you are
>> killing it before it has chance to flush its in-memory state.
>>
>> HBase questions will get more timely response if posted to the hbase lis=
ts
>> (see hbase.org).
>>
>> Yours,
>> St.Ack St.Ack
>>
>>
>> On Wed, Sep 30, 2009 at 9:22 AM, Something Something
>> <luckyguy2050@yahoo.com
>>>
>>> wrote:
>>
>>> Hello,
>>>
>>> I noticed that if I start HBase in "standalone" mode it creates a table
>>> in
>>> memory. =A0In other words, after rebooting the machine, the table goes
>>> away.
>>> I would like to persist the table to a local file system. =A0How can I =
do
>>> that? =A0Do I have to use "Psuedo Distribution" mode for this?
>>>
>>> Please help. =A0Thanks.
>>>
>>>
>>>
>>>
>>>
>>
>
>
> --
> Stephen Mulcahy, DI2, Digital Enterprise Research Institute,
> NUI Galway, IDA Business Park, Lower Dangan, Galway, Ireland
> http://di2.deri.ie =A0 =A0http://webstar.deri.ie =A0 =A0http://sindice.co=
m
>

From general-return-563-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 01 08:06:11 2009
Return-Path: <general-return-563-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 62244 invoked from network); 1 Oct 2009 08:06:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Oct 2009 08:06:11 -0000
Received: (qmail 49276 invoked by uid 500); 1 Oct 2009 08:06:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49202 invoked by uid 500); 1 Oct 2009 08:06:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49192 invoked by uid 99); 1 Oct 2009 08:06:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Oct 2009 08:06:09 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [140.203.201.100] (HELO mx1.nuigalway.ie) (140.203.201.100)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Oct 2009 08:05:58 +0000
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: ApoEAE7/w0oKhJ0L/2dsb2JhbADWVIQoBA
X-IronPort-AV: E=Sophos;i="4.44,486,1249254000"; 
   d="scan'208";a="37872480"
Received: from unknown (HELO EVS1.ac.nuigalway.ie) ([10.132.157.11])
  by mx1.nuigalway.ie with ESMTP; 01 Oct 2009 09:05:38 +0100
Received: from EVS1.ac.nuigalway.ie ([10.132.157.14]) by EVS1.ac.nuigalway.ie with Microsoft SMTPSVC(6.0.3790.3959);
	 Thu, 1 Oct 2009 09:05:37 +0100
Received: from [10.2.18.121] ([140.203.154.11]) by EVS1.ac.nuigalway.ie over TLS secured channel with Microsoft SMTPSVC(6.0.3790.3959);
	 Thu, 1 Oct 2009 09:05:37 +0100
Message-ID: <4AC462CF.70806@deri.org>
Date: Thu, 01 Oct 2009 09:05:35 +0100
From: stephen mulcahy <stephen.mulcahy@deri.org>
User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: HBase configuration...
References: <440561.64930.qm@web111413.mail.gq1.yahoo.com>	 <7c962aed0909301442g57a7b4ceg62a84ccb0ff22102@mail.gmail.com>	 <4AC45E97.20409@deri.org> <78568af10910010052gb67910am3b9c4e183eac0deb@mail.gmail.com>
In-Reply-To: <78568af10910010052gb67910am3b9c4e183eac0deb@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-OriginalArrivalTime: 01 Oct 2009 08:05:37.0497 (UTC) FILETIME=[F5BA3490:01CA426D]
X-Virus-Checked: Checked by ClamAV on apache.org

Ryan Rawson wrote:
> yes, the standalone version uses file:///tmp to store data, to allow
> people to get up and running without HDFS.  And you are exactly
> correct, with /tmp being cleaned on reboot on many distros.  Grep for
> /tmp in hbase-default.xml to find the setting.

Would it be better to default this directory to /var/tmp which would 
achieve the same goal of allowing people to quickly get up and running 
but might avoid some of the confusion of their HDFS disappearing on 
reboot (speaking as one recently confused in such a manner ;)

-stephen

-- 
Stephen Mulcahy, DI2, Digital Enterprise Research Institute,
NUI Galway, IDA Business Park, Lower Dangan, Galway, Ireland
http://di2.deri.ie    http://webstar.deri.ie    http://sindice.com

From general-return-564-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 01 08:14:36 2009
Return-Path: <general-return-564-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 64028 invoked from network); 1 Oct 2009 08:14:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Oct 2009 08:14:35 -0000
Received: (qmail 57613 invoked by uid 500); 1 Oct 2009 08:14:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57539 invoked by uid 500); 1 Oct 2009 08:14:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57529 invoked by uid 99); 1 Oct 2009 08:14:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Oct 2009 08:14:35 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ryanobjc@gmail.com designates 209.85.210.197 as permitted sender)
Received: from [209.85.210.197] (HELO mail-yx0-f197.google.com) (209.85.210.197)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Oct 2009 08:14:23 +0000
Received: by yxe35 with SMTP id 35so3858231yxe.2
        for <general@hadoop.apache.org>; Thu, 01 Oct 2009 01:14:03 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=o5/JTevH3QttvxEd1ZwmuzonBxUUG7B4MeBaUoGhDNI=;
        b=rVhuPnEjlkQ0wCy1yeWP8BOZsJa8rBkaxGQ6PFUVzUd51/T7EuFLjeThRgNVv84IZw
         U8BqnHZ68NF/Hz22Deyla41eQCmXdLZ1OuRn3t+EkCRFKMLM6PMPcvj2hJkFqM/XKZG+
         j0kB5Hfx6trpd+8XuFKHisBxmge2N9sLQVpPM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=j5RHsi6o0VwJyYuKTHIN1pzUN9W31+yVl+kRvIL0fIHr02Ew/w4VPJ+p/tlgdiiq5m
         1o10zlV64+afHDZ7dCsaIN09lFCZ67EAFuVIx1yaQJRt81twPHwcqa0gCfyRHX7NA4dm
         TeH6F0/OswoeYW09AflwpXBXQ3cgKiuPsJztI=
MIME-Version: 1.0
Received: by 10.150.130.39 with SMTP id c39mr1683738ybd.338.1254384842999; 
	Thu, 01 Oct 2009 01:14:02 -0700 (PDT)
In-Reply-To: <4AC462CF.70806@deri.org>
References: <440561.64930.qm@web111413.mail.gq1.yahoo.com>
	 <7c962aed0909301442g57a7b4ceg62a84ccb0ff22102@mail.gmail.com>
	 <4AC45E97.20409@deri.org>
	 <78568af10910010052gb67910am3b9c4e183eac0deb@mail.gmail.com>
	 <4AC462CF.70806@deri.org>
Date: Thu, 1 Oct 2009 01:14:02 -0700
Message-ID: <78568af10910010114m12e9dff8ve17c22351b788934@mail.gmail.com>
Subject: Re: HBase configuration...
From: Ryan Rawson <ryanobjc@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

That seems reasonable, although the nice thing about /tmp is how
universal it is :-)



On Thu, Oct 1, 2009 at 1:05 AM, stephen mulcahy
<stephen.mulcahy@deri.org> wrote:
> Ryan Rawson wrote:
>>
>> yes, the standalone version uses file:///tmp to store data, to allow
>> people to get up and running without HDFS. =A0And you are exactly
>> correct, with /tmp being cleaned on reboot on many distros. =A0Grep for
>> /tmp in hbase-default.xml to find the setting.
>
> Would it be better to default this directory to /var/tmp which would achi=
eve
> the same goal of allowing people to quickly get up and running but might
> avoid some of the confusion of their HDFS disappearing on reboot (speakin=
g
> as one recently confused in such a manner ;)
>
> -stephen
>
> --
> Stephen Mulcahy, DI2, Digital Enterprise Research Institute,
> NUI Galway, IDA Business Park, Lower Dangan, Galway, Ireland
> http://di2.deri.ie =A0 =A0http://webstar.deri.ie =A0 =A0http://sindice.co=
m
>

From general-return-565-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 01 17:50:15 2009
Return-Path: <general-return-565-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87299 invoked from network); 1 Oct 2009 17:50:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Oct 2009 17:50:15 -0000
Received: (qmail 18891 invoked by uid 500); 1 Oct 2009 17:50:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18836 invoked by uid 500); 1 Oct 2009 17:50:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18826 invoked by uid 99); 1 Oct 2009 17:50:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Oct 2009 17:50:14 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 216.145.54.172 is neither permitted nor denied by domain of phunt@apache.org)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Oct 2009 17:50:04 +0000
Received: from [10.73.135.251] (wifi-e-135-251.corp.yahoo.com [10.73.135.251])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n91HlwP8088140;
	Thu, 1 Oct 2009 10:47:58 -0700 (PDT)
Message-ID: <4AC4EB4E.4030708@apache.org>
Date: Thu, 01 Oct 2009 10:47:58 -0700
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: question on slf4j and Hadoop
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

I was watching this recently, it's very interesting:
http://www.java-tv.com/2009/09/24/slf4j-and-logback-projects/
slf4j has some very nice properties, not the least that it solves some 
problems like simplifying integration of apps that use different logging 
systems.

I had noticed that Avro uses slf4j, and that Hadoop in general has it as 
part of the ivy config (although the code is still log4j based). Is the 
intent to move all of Hadoop to slf4j over time?

Patrick

From general-return-566-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 01 20:00:41 2009
Return-Path: <general-return-566-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 40180 invoked from network); 1 Oct 2009 20:00:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Oct 2009 20:00:41 -0000
Received: (qmail 21198 invoked by uid 500); 1 Oct 2009 20:00:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21106 invoked by uid 500); 1 Oct 2009 20:00:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21091 invoked by uid 99); 1 Oct 2009 20:00:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Oct 2009 20:00:39 +0000
X-ASF-Spam-Status: No, hits=4.0 required=10.0
	tests=HTML_MESSAGE,SPF_PASS,SUBJ_ALL_CAPS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cabiwan@gmail.com designates 209.85.220.222 as permitted sender)
Received: from [209.85.220.222] (HELO mail-fx0-f222.google.com) (209.85.220.222)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Oct 2009 20:00:30 +0000
Received: by fxm22 with SMTP id 22so495949fxm.36
        for <general@hadoop.apache.org>; Thu, 01 Oct 2009 13:00:10 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=wpb6bAlHPHQrVDJ5Cf05PDKEg4aUyeBC8/JVcufzxlc=;
        b=EI28otesQ7oNzRR2JosuKjq2r+0g38mjsCZ//ygb9pnCJwY1Z7W3JPZ0btDbcOlBhE
         O6UxtKJONkCx9Mq4wJ2IV3mog5Fu+W9hC9CirTEi0V7UGXB1w5OzQ9oW9PgiV+9At8ur
         2gd78pGoPT5mhE5GhmJu4zpzqJpcOaZaRGjSE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=iplu/J/1vMIsLC1lMyiHgrXHnjLTva4sKyKxVrAeRko4PUGYQCh9tLlFcmOVCdslFx
         9/idTi16XgWb5hXYnP8kNgTfCKGlEtACyam9V2lMG50BnYQgI1W3ZskNpGilcZfbuTF6
         WNkorFQu3FVxxQWYO9Hf7Ovahc4BauywCXRHE=
MIME-Version: 1.0
Received: by 10.103.76.5 with SMTP id d5mr587872mul.131.1254427209671; Thu, 01 
	Oct 2009 13:00:09 -0700 (PDT)
Date: Thu, 1 Oct 2009 22:00:09 +0200
Message-ID: <309f76d00910011300o14702a51h2422ad9397eec34d@mail.gmail.com>
Subject: CHANGING FINAL OUTPUT FILE NAME
From: Alberto Luengo Cabanillas <cabiwan@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e65aeee83aa5000474e518b3
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e65aeee83aa5000474e518b3
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi everyone! I have a newbie question: I=B4m actually using Hadoop 0.20.1 a=
nd
I=B4d like to know how can I change the name of the resulting file with the
one I want (i.e from "part-r-00000" to "myoutput"). I=B4ve found something
related in JIRA (https://issues.apache.org/jira/browse/MAPREDUCE-370) but I
don=B4t know for sure i that is my problem too. In this case, do I apply th=
e
patch over the affected file and I=B4m ready to go or do I need to do
something more later?
Thanks a lot!

--=20
Alberto

--0016e65aeee83aa5000474e518b3--

From general-return-567-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 02 01:05:38 2009
Return-Path: <general-return-567-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47188 invoked from network); 2 Oct 2009 01:05:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Oct 2009 01:05:37 -0000
Received: (qmail 66731 invoked by uid 500); 2 Oct 2009 01:05:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66660 invoked by uid 500); 2 Oct 2009 01:05:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66650 invoked by uid 99); 2 Oct 2009 01:05:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Oct 2009 01:05:36 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jason.hadoop@gmail.com designates 209.85.212.198 as permitted sender)
Received: from [209.85.212.198] (HELO mail-vw0-f198.google.com) (209.85.212.198)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Oct 2009 01:05:26 +0000
Received: by vws36 with SMTP id 36so375636vws.29
        for <general@hadoop.apache.org>; Thu, 01 Oct 2009 18:05:04 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=aAKNQ+ApbLKxXHmPKZRzrWQvqNd/4tj6zkq/ZJpAwvU=;
        b=hzpRxC1Td8LC5sQbetgl1urIThAAQcCnMaa9jK2npdH400YuKaRdiLSOxUhrNH+MLJ
         dhjd73odr+M/dGKBuQQpYP8+0F+AHEjXdBbEeWSKgwkfKlJNapwEoyIEc7Qp0sVK/JUu
         e0nJDCImDnLuhAkeIet2lsf4dvBttUm8iO+4w=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=uYKYP9BgjXJsUT+quO3iHmsZb3IqE2dyf8BSapn++pxA2o4TxpeV8D32ZPnqWIN/kz
         V+yR19EK3aD89lElnXBhb0W/BHgYfLOuSktzO1MULqkRWwaoU6VJSz5diEfVkeqjl2kF
         XBj6vp/GXNER9hjlE3mCruXZ/QkNLw9XLLEls=
MIME-Version: 1.0
Received: by 10.220.69.169 with SMTP id z41mr3569245vci.31.1254445504654; Thu, 
	01 Oct 2009 18:05:04 -0700 (PDT)
In-Reply-To: <309f76d00910011300o14702a51h2422ad9397eec34d@mail.gmail.com>
References: <309f76d00910011300o14702a51h2422ad9397eec34d@mail.gmail.com>
Date: Thu, 1 Oct 2009 18:05:04 -0700
Message-ID: <314098690910011805v5b0c9246q221cd51695d7b6e8@mail.gmail.com>
Subject: Re: CHANGING FINAL OUTPUT FILE NAME
From: Jason Venner <jason.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6475626b1ed620474e95ae1
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6475626b1ed620474e95ae1
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I see these ways to go here.

   1.  The one I know to work is to create a recordwriter in the configure
   method of your task, in the per task work/output directory, and then ren=
ame
   it to your chosen name in the close. your task calls write on the
   recordwriter directly instead of output.collect
   2. Use the multi output format
   3. in the close method of the task, rename the part-xxx to your name. I
   am not certain that this is safe in the close method of the task
   4. define a custom OutputCommitter class which renames the file to your
   chosen name.




On Thu, Oct 1, 2009 at 1:00 PM, Alberto Luengo Cabanillas <cabiwan@gmail.co=
m
> wrote:

> Hi everyone! I have a newbie question: I=B4m actually using Hadoop 0.20.1=
 and
> I=B4d like to know how can I change the name of the resulting file with t=
he
> one I want (i.e from "part-r-00000" to "myoutput"). I=B4ve found somethin=
g
> related in JIRA (https://issues.apache.org/jira/browse/MAPREDUCE-370) but
> I
> don=B4t know for sure i that is my problem too. In this case, do I apply =
the
> patch over the affected file and I=B4m ready to go or do I need to do
> something more later?
> Thanks a lot!
>
> --
> Alberto
>



--=20
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=3Djewlerymall
www.prohadoopbook.com a community for Hadoop Professionals

--0016e6475626b1ed620474e95ae1--

From general-return-568-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 02 07:50:02 2009
Return-Path: <general-return-568-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34581 invoked from network); 2 Oct 2009 07:50:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Oct 2009 07:50:02 -0000
Received: (qmail 8138 invoked by uid 500); 2 Oct 2009 07:50:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8076 invoked by uid 500); 2 Oct 2009 07:50:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8066 invoked by uid 99); 2 Oct 2009 07:50:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Oct 2009 07:50:01 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Oct 2009 07:49:49 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 636CA1BA448
	for <general@hadoop.apache.org>; Fri,  2 Oct 2009 08:49:27 +0100 (BST)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id Mfn6Cq74t2Cj for <general@hadoop.apache.org>;
	Fri,  2 Oct 2009 08:49:27 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by colossus.hpl.hp.com (Postfix) with ESMTPS id DB6BF1BA2F2
	for <general@hadoop.apache.org>; Fri,  2 Oct 2009 08:49:26 +0100 (BST)
MailScanner-NULL-Check: 1255074554.29463@yzWtbVzO16YiKIWarRUKIw
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id n927nC7d003412
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Fri, 2 Oct 2009 08:49:13 +0100 (BST)
Message-ID: <4AC5B078.30601@apache.org>
Date: Fri, 02 Oct 2009 08:49:12 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: question on slf4j and Hadoop
References: <4AC4EB4E.4030708@apache.org>
In-Reply-To: <4AC4EB4E.4030708@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: n927nC7d003412
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

Patrick Hunt wrote:
> I was watching this recently, it's very interesting:
> http://www.java-tv.com/2009/09/24/slf4j-and-logback-projects/
> slf4j has some very nice properties, not the least that it solves some 
> problems like simplifying integration of apps that use different logging 
> systems.
> 
> I had noticed that Avro uses slf4j, and that Hadoop in general has it as 
> part of the ivy config (although the code is still log4j based). Is the 
> intent to move all of Hadoop to slf4j over time?
> 
> Patrick

SLF4J is there because jetty uses it. If it had stayed with 
commons-logging it would be that much easier to run different logging 
back ends, as it is I have to have two different bridge libraries to 
integrate hadoop logging with where the data goes in my clusters.

From general-return-569-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 02 08:15:02 2009
Return-Path: <general-return-569-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43003 invoked from network); 2 Oct 2009 08:15:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Oct 2009 08:15:02 -0000
Received: (qmail 40246 invoked by uid 500); 2 Oct 2009 08:15:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40173 invoked by uid 500); 2 Oct 2009 08:15:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40163 invoked by uid 99); 2 Oct 2009 08:15:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Oct 2009 08:15:01 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cabiwan@gmail.com designates 209.85.218.222 as permitted sender)
Received: from [209.85.218.222] (HELO mail-bw0-f222.google.com) (209.85.218.222)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Oct 2009 08:14:51 +0000
Received: by bwz22 with SMTP id 22so737112bwz.29
        for <general@hadoop.apache.org>; Fri, 02 Oct 2009 01:14:30 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=GQVzeXTE34JVg+p+afkiBpmAurSTP6Tz/GYqNMHWTok=;
        b=ikqRT8XEnTuBqpq0EV27NaoWXWDkaB2/0dqgqgumGlqsm94zIUWpf5qXMK7c7WfD2k
         Xi5aXHEKPsqchRONtcDUq5vKAgZNNz2aWP5Xc80x8xbvn+i/jisdwo+YDEk4kANyMa1O
         G0rLiaLQv45ssk93IxNAi5r0B6Nbxyrb3u96A=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=MmvdTO32nsrzdNhMRR5x0Ai2NS+ReclIUijYzBx4Hr5udH0VESA/FJM8t8Jhd4QYDz
         RA/cY/d/a7adPd18S5TVaR3AV7rce5wan1MBbp/X9FC0gAAWhhfqKph6dP3KtAnnXwwb
         Zz1tBm6vonReF09WqMjZAzg5xAaFwnb9m5Ivc=
MIME-Version: 1.0
Received: by 10.102.178.11 with SMTP id a11mr849822muf.129.1254471270575; Fri, 
	02 Oct 2009 01:14:30 -0700 (PDT)
In-Reply-To: <314098690910011805v5b0c9246q221cd51695d7b6e8@mail.gmail.com>
References: <309f76d00910011300o14702a51h2422ad9397eec34d@mail.gmail.com>
	 <314098690910011805v5b0c9246q221cd51695d7b6e8@mail.gmail.com>
Date: Fri, 2 Oct 2009 10:14:30 +0200
Message-ID: <309f76d00910020114j6ca5979ag7dd20b26e586cbe0@mail.gmail.com>
Subject: Re: CHANGING FINAL OUTPUT FILE NAME
From: Alberto Luengo Cabanillas <cabiwan@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163641797d76adae0474ef5a2f
X-Virus-Checked: Checked by ClamAV on apache.org

--00163641797d76adae0474ef5a2f
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Thanks for the reply Jason. I=B4ll try these solutions.

2009/10/2 Jason Venner <jason.hadoop@gmail.com>

> I see these ways to go here.
>
>   1.  The one I know to work is to create a recordwriter in the configure
>   method of your task, in the per task work/output directory, and then
> rename
>   it to your chosen name in the close. your task calls write on the
>   recordwriter directly instead of output.collect
>   2. Use the multi output format
>   3. in the close method of the task, rename the part-xxx to your name. I
>   am not certain that this is safe in the close method of the task
>   4. define a custom OutputCommitter class which renames the file to your
>   chosen name.
>
>
>
>
> On Thu, Oct 1, 2009 at 1:00 PM, Alberto Luengo Cabanillas <
> cabiwan@gmail.com
> > wrote:
>
> > Hi everyone! I have a newbie question: I=B4m actually using Hadoop 0.20=
.1
> and
> > I=B4d like to know how can I change the name of the resulting file with=
 the
> > one I want (i.e from "part-r-00000" to "myoutput"). I=B4ve found someth=
ing
> > related in JIRA (https://issues.apache.org/jira/browse/MAPREDUCE-370)
> but
> > I
> > don=B4t know for sure i that is my problem too. In this case, do I appl=
y
> the
> > patch over the affected file and I=B4m ready to go or do I need to do
> > something more later?
> > Thanks a lot!
> >
> > --
> > Alberto
> >
>
>
>
> --
> Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> http://www.amazon.com/dp/1430219424?tag=3Djewlerymall
> www.prohadoopbook.com a community for Hadoop Professionals
>



--=20
Alberto

--00163641797d76adae0474ef5a2f--

From general-return-570-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 02 08:28:31 2009
Return-Path: <general-return-570-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45519 invoked from network); 2 Oct 2009 08:28:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Oct 2009 08:28:31 -0000
Received: (qmail 57017 invoked by uid 500); 2 Oct 2009 08:28:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56934 invoked by uid 500); 2 Oct 2009 08:28:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56924 invoked by uid 99); 2 Oct 2009 08:28:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Oct 2009 08:28:30 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of timrobertson100@gmail.com designates 209.85.218.222 as permitted sender)
Received: from [209.85.218.222] (HELO mail-bw0-f222.google.com) (209.85.218.222)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Oct 2009 08:28:20 +0000
Received: by bwz22 with SMTP id 22so744755bwz.29
        for <general@hadoop.apache.org>; Fri, 02 Oct 2009 01:27:59 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=uIvl9EtltiKZ4mprJiUbuGhe3kK6oD32MoBk1/cwweE=;
        b=vZ+kG5lsyCInlPJK0TODz3NV0fxxin6JCYMhzsNFH+XZA/8aec7PBv1tBEXNXMfw9o
         CNxw/L9iBHNhmKMQaa/wM5AId3tgCmILQDfcK2dfVWHPMPjw/asd89pig3zQVBO5fqaa
         giKOMnyY85sD0YbZzdZtIaHIOU79pNVH+WrNw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=mG6sqDkq5/VTfAYA7SHNSvmtpQL39+xgZLLP+G/r9sxFjCoxyhrz1tlNeGmpE7pg1o
         v5BOBozmsP6GcUeA7vS0ZQotYtN8UZGA7AS1GZRY8oDZuXlbgk1XZRjXojUgdWoCo5Rk
         aNHqyZWbu65ysq7Ojn4mgmu1RRKeXymHUOpyo=
MIME-Version: 1.0
Received: by 10.223.63.202 with SMTP id c10mr682286fai.73.1254472079429; Fri, 
	02 Oct 2009 01:27:59 -0700 (PDT)
In-Reply-To: <4AC5B078.30601@apache.org>
References: <4AC4EB4E.4030708@apache.org> <4AC5B078.30601@apache.org>
Date: Fri, 2 Oct 2009 10:27:59 +0200
Message-ID: <32120a6a0910020127p7703e695l1882c3e6274a08b9@mail.gmail.com>
Subject: Re: question on slf4j and Hadoop
From: tim robertson <timrobertson100@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

I could be wrong on this, but something to take into account with slf4j...

I have seen a few projects that logged a custom MessageObject which
different appenders do differing things with.  E.g. a file appender
will toString() it, but a custom DB appender might use the object
properties and a prepared statement.  I believe slf4j will call the
toString() before it reachers your appenders so it breaks this
functionality.

Disclaimer: I really may be wrong, but I certainly saw this behavior
and reverted to commons logging on those projects, and it could be
that slf4j now respects the original method signatures are (e.g.)
log.info(Object) and not log.info(String)





On Fri, Oct 2, 2009 at 9:49 AM, Steve Loughran <stevel@apache.org> wrote:
> Patrick Hunt wrote:
>>
>> I was watching this recently, it's very interesting:
>> http://www.java-tv.com/2009/09/24/slf4j-and-logback-projects/
>> slf4j has some very nice properties, not the least that it solves some
>> problems like simplifying integration of apps that use different logging
>> systems.
>>
>> I had noticed that Avro uses slf4j, and that Hadoop in general has it as
>> part of the ivy config (although the code is still log4j based). Is the
>> intent to move all of Hadoop to slf4j over time?
>>
>> Patrick
>
> SLF4J is there because jetty uses it. If it had stayed with commons-logging
> it would be that much easier to run different logging back ends, as it is I
> have to have two different bridge libraries to integrate hadoop logging with
> where the data goes in my clusters.
>

From general-return-571-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 02 15:27:45 2009
Return-Path: <general-return-571-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70920 invoked from network); 2 Oct 2009 15:27:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Oct 2009 15:27:45 -0000
Received: (qmail 32980 invoked by uid 500); 2 Oct 2009 15:27:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32895 invoked by uid 500); 2 Oct 2009 15:27:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32885 invoked by uid 99); 2 Oct 2009 15:27:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Oct 2009 15:27:44 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cabiwan@gmail.com designates 209.85.218.222 as permitted sender)
Received: from [209.85.218.222] (HELO mail-bw0-f222.google.com) (209.85.218.222)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Oct 2009 15:27:33 +0000
Received: by bwz22 with SMTP id 22so1066836bwz.29
        for <general@hadoop.apache.org>; Fri, 02 Oct 2009 08:27:13 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=XcDiQ47gXpu2jwB9D6a8Hl3JjmAg1dLvwtFaIbM42rA=;
        b=jodj9JBPSOH4oFqJdSCHKeQG3XWo2fSbXnZl9QmRIkhbO+ZJB3jA8wdp5xLlVUlc88
         mtac+NuH8ef3JxhxuO0IXLfr9vvVmoo6gbDLQrnWcjU3ENWuY9YsfxyOBpwT0NhKAmJU
         9ZIJw8g+3/GPLnjUHn++JQmBOIw6lSGKGT1EE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=T+zD6AcFYZFVCjYaQqk7MiwxYHT0eVKwU2XhTQfzl3NkbklnLx170dx1LAac1iOwQE
         pzmHy8GPratexMmF7ZIbsu2Cvuoq+38Yy6qFZirvhOBQJPSr+R/OOybwj/ilqzlegHd9
         YHE9PD7EkVTcu/kePy9wtL7DsKU+dJKt+Dvyk=
MIME-Version: 1.0
Received: by 10.102.178.11 with SMTP id a11mr1045632muf.129.1254497232751; 
	Fri, 02 Oct 2009 08:27:12 -0700 (PDT)
In-Reply-To: <314098690910011805v5b0c9246q221cd51695d7b6e8@mail.gmail.com>
References: <309f76d00910011300o14702a51h2422ad9397eec34d@mail.gmail.com>
	 <314098690910011805v5b0c9246q221cd51695d7b6e8@mail.gmail.com>
Date: Fri, 2 Oct 2009 17:27:12 +0200
Message-ID: <309f76d00910020827h6f8214fdv4a2166d35ab58f0a@mail.gmail.com>
Subject: Re: CHANGING FINAL OUTPUT FILE NAME
From: Alberto Luengo Cabanillas <cabiwan@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163641797dee0b300474f5653a
X-Virus-Checked: Checked by ClamAV on apache.org

--00163641797dee0b300474f5653a
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi again! I=B4ve been trying to apply the first of the solutions Jason
proposed two mails ago, but I have some questions about it:

1.-In Hadoop 0.20.1 "RecordWriter" is an abstract class, so it can=B4t be
instantiated.
2.- The "configure" and "close" methods of my task you refer are the
"configure" and "close" methods used in the Reducer?

It would help a lot if you=B4d put some code instructions (only the ones
reffering to this issue) so I can have a better point of view.

Finally, I=B4d like to show my close method in my reducer class, because I =
use
a different approach and I can=B4t see why it fails:

    *@Override
    protected void cleanup(Context cont) throws IOException {

      //write output to a file
      Configuration conf =3D new Configuration();
      JobContext jCont =3D new JobContext(conf, null);
      FileSystem fs =3D FileSystem.get(jCont.getConfiguration());
      Path outDir =3D new Path("/user/hadoop-user/output", "output");
      Path outFile =3D new Path(outDir, "reduce-out");
      SequenceFile.Writer writer =3D SequenceFile.createWriter(fs, conf,
          outFile, LongWritable.class, LongWritable.class,
          CompressionType.NONE);
      writer.append(new Text(keyword), new IntWritable(fitnessValue));
      writer.close();
    }
*
Thanks a lot in advance!

2009/10/2 Jason Venner <jason.hadoop@gmail.com>

> I see these ways to go here.
>
>   1.  The one I know to work is to create a recordwriter in the configure
>   method of your task, in the per task work/output directory, and then
> rename
>   it to your chosen name in the close. your task calls write on the
>   recordwriter directly instead of output.collect
>   2. Use the multi output format
>   3. in the close method of the task, rename the part-xxx to your name. I
>   am not certain that this is safe in the close method of the task
>   4. define a custom OutputCommitter class which renames the file to your
>   chosen name.
>
>
>
>
> On Thu, Oct 1, 2009 at 1:00 PM, Alberto Luengo Cabanillas <
> cabiwan@gmail.com
> > wrote:
>
> > Hi everyone! I have a newbie question: I=B4m actually using Hadoop 0.20=
.1
> and
> > I=B4d like to know how can I change the name of the resulting file with=
 the
> > one I want (i.e from "part-r-00000" to "myoutput"). I=B4ve found someth=
ing
> > related in JIRA (https://issues.apache.org/jira/browse/MAPREDUCE-370)
> but
> > I
> > don=B4t know for sure i that is my problem too. In this case, do I appl=
y
> the
> > patch over the affected file and I=B4m ready to go or do I need to do
> > something more later?
> > Thanks a lot!
> >
> > --
> > Alberto
> >
>
>
>
> --
> Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> http://www.amazon.com/dp/1430219424?tag=3Djewlerymall
> www.prohadoopbook.com a community for Hadoop Professionals
>



--=20
Alberto

--00163641797dee0b300474f5653a--

From general-return-572-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 02 21:41:51 2009
Return-Path: <general-return-572-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 90023 invoked from network); 2 Oct 2009 21:41:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Oct 2009 21:41:51 -0000
Received: (qmail 98297 invoked by uid 500); 2 Oct 2009 21:41:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98145 invoked by uid 500); 2 Oct 2009 21:41:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98062 invoked by uid 99); 2 Oct 2009 21:41:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Oct 2009 21:41:48 +0000
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Oct 2009 21:41:35 +0000
Received: from sp1-ex07cas03.ds.corp.yahoo.com (sp1-ex07cas03.ds.corp.yahoo.com [216.252.116.151])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n92Lf5ix040365;
	Fri, 2 Oct 2009 14:41:10 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:references:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:
	content-transfer-encoding:mime-version;
	b=RvzyT/WRwZ4WSSNMMeCWhZWBrjUrYP7suZW8q8y8nZOwUi934HqPkTvPYHjSiE4L
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas03.ds.corp.yahoo.com ([216.252.116.151]) with mapi; Fri, 2 Oct
 2009 14:41:08 -0700
From: Dekel Tankel <dekel@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>,
        "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Date: Fri, 2 Oct 2009 14:41:05 -0700
Subject: Hadoop User Group (Bay Area) - Oct 21st at Yahoo!
Thread-Topic: Hadoop User Group (Bay Area) - Oct 21st at Yahoo!
Thread-Index: AcoSMRRU8X5auVLKR8ufbpFTQi1hLgQXaiyQCEZs4pA=
Message-ID: <46E2672895E8744A97D7577A6110858B4FD376F380@SP1-EX07VS01.ds.corp.yahoo.com>
References: <46E2672895E8744A97D7577A6110858B4E1AC69A91@SP1-EX07VS01.ds.corp.yahoo.com> 
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


Hi all,

RSVP is now open for the next monthly Bay Area Hadoop user group at the Yah=
oo! Sunnyvale Campus, planed for Oct 21st.

Registration is available here
http://www.meetup.com/hadoop/calendar/11532125/

We would be interested to explore additional Hadoop and/or related technolo=
gies use-cases like Ken provided us in the last event (thank you Ken, very =
impressive!) allowing all of us to learn from each other.

If you are interested to present, please send me an email with a short desc=
ription.

Looking forward to seeing you at Oct 21st


Dekel


From general-return-573-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Oct 04 10:21:38 2009
Return-Path: <general-return-573-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 61339 invoked from network); 4 Oct 2009 10:21:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Oct 2009 10:21:37 -0000
Received: (qmail 27090 invoked by uid 500); 4 Oct 2009 10:21:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27015 invoked by uid 500); 4 Oct 2009 10:21:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27005 invoked by uid 99); 4 Oct 2009 10:21:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 04 Oct 2009 10:21:36 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [209.85.210.194] (HELO mail-yx0-f194.google.com) (209.85.210.194)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 04 Oct 2009 10:21:24 +0000
Received: by yxe32 with SMTP id 32so2530145yxe.5
        for <general@hadoop.apache.org>; Sun, 04 Oct 2009 03:20:02 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.101.102.16 with SMTP id e16mr4440427anm.84.1254651602266; Sun, 
	04 Oct 2009 03:20:02 -0700 (PDT)
In-Reply-To: <314098690909272130n4f22e636re90a067e47caffa2@mail.gmail.com>
References: <4abc7bbb.05035a0a.55c7.ffffd84c@mx.google.com> 
	<314098690909272130n4f22e636re90a067e47caffa2@mail.gmail.com>
From: Chandan Tamrakar <chandan.tamrakar@nepasoft.com>
Date: Sun, 4 Oct 2009 16:04:42 +0545
Message-ID: <95f8b1a90910040319o3bdf0ca2j705a504edaa71588@mail.gmail.com>
Subject: Re: Integrating Lucene in Hadoop
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636ed706311eaaf04751957f8
X-Virus-Checked: Checked by ClamAV on apache.org

--001636ed706311eaaf04751957f8
Content-Type: text/plain; charset=ISO-8859-1

thanks jason

In my case lucene indexes needs to be accessed thru .net  code ( simple
search interfaces)
so should I use solr-1301 patch to index the documents and then move the
indexes  from HDFS to Local file system so that .net interface could access
and query the indexes directly ?

Will this be a better approach ? alternatively  we might need some sort of
web service  that could read/query from index in hdfs and would be called by
.net front end UI.

will very much appreciate the suggestions
thanks


On Mon, Sep 28, 2009 at 10:15 AM, Jason Venner <jason.hadoop@gmail.com>wrote:

> Katta is a tool for managing distributed search, which happens by default
> to
> use lucene as it' search engine.
> Katta is able to read indexes from hdfs, or s3, that katta then deploys
> onto
> the local disk of the katta nodes.
>
> The contrib directory of the hadoop installation contains a tool for
> building lucene indexes, and patch solr-1395, provides an integration of
> solr with katta, and patch solr-1301 provides a simple way of building solr
> indexes in hadoop (as a parallel to the contrib lucene index build tool).
>
>
>
> On Fri, Sep 25, 2009 at 1:13 AM, Chandan Tamrakar <
> chandan.tamrakar@nepasoft.com> wrote:
>
> > I was doing few R&D on  integrating hadoop and  Lucene . I could create
> > lucene indexes into HDFS using  index contribution provided for Hadoop
> >
> >
> >
> > Is this a proper way to create Lucene index how much it is different from
> a
> > project KATTA ?
> >
> >
> >
> > Please suggest what would be the better approach to create distributed
> > lucene indexes
> >
> >
> >
> > Thanks in advance
> >
> >
>
>
> --
> Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> http://www.amazon.com/dp/1430219424?tag=jewlerymall
> www.prohadoopbook.com a community for Hadoop Professionals
>



-- 
Chandan Tamrakar

--001636ed706311eaaf04751957f8--

From general-return-574-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Oct 04 11:20:12 2009
Return-Path: <general-return-574-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68411 invoked from network); 4 Oct 2009 11:20:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Oct 2009 11:20:12 -0000
Received: (qmail 56509 invoked by uid 500); 4 Oct 2009 11:20:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56427 invoked by uid 500); 4 Oct 2009 11:20:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56417 invoked by uid 99); 4 Oct 2009 11:20:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 04 Oct 2009 11:20:11 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [209.85.222.195] (HELO mail-pz0-f195.google.com) (209.85.222.195)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 04 Oct 2009 11:20:00 +0000
Received: by pzk33 with SMTP id 33so2606922pzk.2
        for <general@hadoop.apache.org>; Sun, 04 Oct 2009 04:19:39 -0700 (PDT)
Received: by 10.114.253.23 with SMTP id a23mr6970885wai.155.1254655179546;
        Sun, 04 Oct 2009 04:19:39 -0700 (PDT)
Received: from nsf264fa356e37 ([124.41.206.51])
        by mx.google.com with ESMTPS id 23sm1899034pzk.8.2009.10.04.04.19.37
        (version=SSLv3 cipher=RC4-MD5);
        Sun, 04 Oct 2009 04:19:39 -0700 (PDT)
From: "Chandan Tamrakar" <chandan.tamrakar@nepasoft.com>
To: <general@hadoop.apache.org>
Subject: katta and hadoop index contrib
Date: Sun, 4 Oct 2009 17:04:18 +0545
Message-ID: <4ac884cb.9713f30a.3f10.ffff9c65@mx.google.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_NextPart_000_002D_01CA4514.B8813B60"
X-Mailer: Microsoft Office Outlook 12.0
Thread-Index: AcpE5INtMBUYiVxMTsSECZRm1EyVEQ==
Content-Language: en-us
X-Virus-Checked: Checked by ClamAV on apache.org

------=_NextPart_000_002D_01CA4514.B8813B60
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit

I was using the hadoop.contrib.index code and  was able to make a
distributed Lucene index

Could then search over that index while it is still in hdfs

 

Does the Katta project uses same  hadoop index-contrib to index the
documents ?

 

I found that there are also contributions  solr-1395 and solr-1301 ( solr
-hadoop ) what would be the best approach to begin with 

 

 

 

Thanks

 

 


------=_NextPart_000_002D_01CA4514.B8813B60--


From general-return-575-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Oct 04 11:28:19 2009
Return-Path: <general-return-575-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69696 invoked from network); 4 Oct 2009 11:28:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Oct 2009 11:28:18 -0000
Received: (qmail 60771 invoked by uid 500); 4 Oct 2009 11:28:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60704 invoked by uid 500); 4 Oct 2009 11:28:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60692 invoked by uid 99); 4 Oct 2009 11:28:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 04 Oct 2009 11:28:18 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.197] (HELO mail-px0-f197.google.com) (209.85.216.197)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 04 Oct 2009 11:28:07 +0000
Received: by pxi35 with SMTP id 35so2353070pxi.2
        for <general@hadoop.apache.org>; Sun, 04 Oct 2009 04:27:45 -0700 (PDT)
Received: by 10.114.253.23 with SMTP id a23mr6970885wai.155.1254655179546;
        Sun, 04 Oct 2009 04:19:39 -0700 (PDT)
Received: from nsf264fa356e37 ([124.41.206.51])
        by mx.google.com with ESMTPS id 23sm1899034pzk.8.2009.10.04.04.19.37
        (version=SSLv3 cipher=RC4-MD5);
        Sun, 04 Oct 2009 04:19:39 -0700 (PDT)
From: "Chandan Tamrakar" <chandan.tamrakar@nepasoft.com>
To: <general@hadoop.apache.org>
Subject: katta and hadoop index contrib
Date: Sun, 4 Oct 2009 17:04:18 +0545
Message-ID: <4ac884cb.9713f30a.3f10.ffff9c65@mx.google.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_NextPart_000_002D_01CA4514.B8813B60"
X-Mailer: Microsoft Office Outlook 12.0
Thread-Index: AcpE5INtMBUYiVxMTsSECZRm1EyVEQ==
Content-Language: en-us
X-Virus-Checked: Checked by ClamAV on apache.org

------=_NextPart_000_002D_01CA4514.B8813B60
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit

I was using the hadoop.contrib.index code and  was able to make a
distributed Lucene index

Could then search over that index while it is still in hdfs

 

Does the Katta project uses same  hadoop index-contrib to index the
documents ?

 

I found that there are also contributions  solr-1395 and solr-1301 ( solr
-hadoop ) what would be the best approach to begin with 

 

 

 

Thanks

 

 


------=_NextPart_000_002D_01CA4514.B8813B60--


From general-return-576-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 05 03:48:54 2009
Return-Path: <general-return-576-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 52961 invoked from network); 5 Oct 2009 03:48:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Oct 2009 03:48:54 -0000
Received: (qmail 67049 invoked by uid 500); 5 Oct 2009 03:48:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66924 invoked by uid 500); 5 Oct 2009 03:48:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66914 invoked by uid 99); 5 Oct 2009 03:48:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Oct 2009 03:48:53 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jason.hadoop@gmail.com designates 209.85.212.198 as permitted sender)
Received: from [209.85.212.198] (HELO mail-vw0-f198.google.com) (209.85.212.198)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Oct 2009 03:48:42 +0000
Received: by vws36 with SMTP id 36so1567126vws.29
        for <general@hadoop.apache.org>; Sun, 04 Oct 2009 20:47:21 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=lUU6qXjxvs+3p6p+PL9Xz/NtvPsNKXueTBwOcvWZwJs=;
        b=QorQh0hw3Id1uf0b4ent35FfKbBh+5W8DNuFD43h1PA99FEM6R2y5q9LFSh4181QAc
         oK2rdZ7qZEzVQToVRKSgoOYCono8hifHg2GVYNuEju2p0/3ojAU7kPyDIwZPusafs/PR
         XqcpqFW5koNos7JPGVtka+0B+bZ6HyKIH3Qgg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Npl48W7CT3lveSGvxLQLPHUr8aRuJZ1sEHBaOjIOrhcW9YePhFt6jz9nRe0c2LeBb9
         k7zyt2JmpD5BCw3EzlMDfnAw8o2HU3IfLpSCFPaPVtXGlCNE7I+lRiZcznR8biO/xm6K
         /WmroJ8Co4EM3CPtPGFlU0umPhZ0pa9NzCuos=
MIME-Version: 1.0
Received: by 10.220.69.169 with SMTP id z41mr8994574vci.31.1254714441368; Sun, 
	04 Oct 2009 20:47:21 -0700 (PDT)
In-Reply-To: <4ac884cb.9713f30a.3f10.ffff9c65@mx.google.com>
References: <4ac884cb.9713f30a.3f10.ffff9c65@mx.google.com>
Date: Sun, 4 Oct 2009 20:47:21 -0700
Message-ID: <314098690910042047r79355f3ewc0b140a8b524ee64@mail.gmail.com>
Subject: Re: katta and hadoop index contrib
From: Jason Venner <jason.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64756269285ea047527f857
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e64756269285ea047527f857
Content-Type: text/plain; charset=ISO-8859-1

Currently Katta, will pull the index out of HDFS and deploy it on local
disk.
HDFS is not known for low latency random access, and index lookups generally
require low latency random access.

I seem to remember that someone had a patch that allowed lucene to directly
access files in hdfs for the index, for readon ly access, but I do not
remember the reference.

On Sun, Oct 4, 2009 at 4:19 AM, Chandan Tamrakar <
chandan.tamrakar@nepasoft.com> wrote:

> I was using the hadoop.contrib.index code and  was able to make a
> distributed Lucene index
>
> Could then search over that index while it is still in hdfs
>
>
>
> Does the Katta project uses same  hadoop index-contrib to index the
> documents ?
>
>
>
> I found that there are also contributions  solr-1395 and solr-1301 ( solr
> -hadoop ) what would be the best approach to begin with
>
>
>
>
>
>
>
> Thanks
>
>
>
>
>
>


-- 
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals

--0016e64756269285ea047527f857--

From general-return-577-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 05 03:49:44 2009
Return-Path: <general-return-577-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53150 invoked from network); 5 Oct 2009 03:49:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Oct 2009 03:49:44 -0000
Received: (qmail 67692 invoked by uid 500); 5 Oct 2009 03:49:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67602 invoked by uid 500); 5 Oct 2009 03:49:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67592 invoked by uid 99); 5 Oct 2009 03:49:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Oct 2009 03:49:43 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jason.hadoop@gmail.com designates 209.85.212.198 as permitted sender)
Received: from [209.85.212.198] (HELO mail-vw0-f198.google.com) (209.85.212.198)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Oct 2009 03:49:32 +0000
Received: by mail-vw0-f198.google.com with SMTP id 36so1567126vws.29
        for <general@hadoop.apache.org>; Sun, 04 Oct 2009 20:49:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=uPoQClyPiCy1s92AnhrGjyqNuZWzJnQb2JWNmObyhAM=;
        b=ud0uxMUvAorrHX6hWW7KSTMGdUiGwe7vx/QLvOdvMak0jnpYLPuUv3yMEwG18PzceC
         X6mydF5vsCnI55UHrlSeA2GcAPFWfwv6MKP7ZlSifcfuy00ONdVW16UShm8YzEMbbnTE
         MmAZUgBG+zCJE+JxX3ZicjB3HSYIVjZ8QFDo8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=bQtxoFxUGgbDzrR1Ig3QR5c6antFwaBucpwOa7/hRba+uDRpamgrib+ynSSa8OTnQ6
         I1+827Xb3VAjO4nQuM2zC2OEJvE1z82H9Rh+0NDZgf9rLQxtOAkGYJHlrbaHd6NSJNzf
         kHGFtZyVc1PPAplyrjUYhbLsSk9ZTzHv0K4ls=
MIME-Version: 1.0
Received: by 10.220.16.83 with SMTP id n19mr9150666vca.3.1254714551941; Sun, 
	04 Oct 2009 20:49:11 -0700 (PDT)
In-Reply-To: <95f8b1a90910040319o3bdf0ca2j705a504edaa71588@mail.gmail.com>
References: <4abc7bbb.05035a0a.55c7.ffffd84c@mx.google.com>
	 <314098690909272130n4f22e636re90a067e47caffa2@mail.gmail.com>
	 <95f8b1a90910040319o3bdf0ca2j705a504edaa71588@mail.gmail.com>
Date: Sun, 4 Oct 2009 20:49:11 -0700
Message-ID: <314098690910042049r2e359ce9v93a86f136fc074b6@mail.gmail.com>
Subject: Re: Integrating Lucene in Hadoop
From: Jason Venner <jason.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485ea86b429b8eb047527fff1
X-Virus-Checked: Checked by ClamAV on apache.org

--001485ea86b429b8eb047527fff1
Content-Type: text/plain; charset=ISO-8859-1

So, solr-1301 builds the solr lucene indexes, it is over kill if you are
just using lucene. The contrib index build code with hadoop is sufficient.

Moving the indexes to the local file system is the most simple strategy, you
could try using fuse to mount hdfs as a local file system, for access as an
alternative.

On Sun, Oct 4, 2009 at 3:19 AM, Chandan Tamrakar <
chandan.tamrakar@nepasoft.com> wrote:

> thanks jason
>
> In my case lucene indexes needs to be accessed thru .net  code ( simple
> search interfaces)
> so should I use solr-1301 patch to index the documents and then move the
> indexes  from HDFS to Local file system so that .net interface could access
> and query the indexes directly ?
>
> Will this be a better approach ? alternatively  we might need some sort of
> web service  that could read/query from index in hdfs and would be called
> by
> .net front end UI.
>
> will very much appreciate the suggestions
> thanks
>
>
> On Mon, Sep 28, 2009 at 10:15 AM, Jason Venner <jason.hadoop@gmail.com
> >wrote:
>
> > Katta is a tool for managing distributed search, which happens by default
> > to
> > use lucene as it' search engine.
> > Katta is able to read indexes from hdfs, or s3, that katta then deploys
> > onto
> > the local disk of the katta nodes.
> >
> > The contrib directory of the hadoop installation contains a tool for
> > building lucene indexes, and patch solr-1395, provides an integration of
> > solr with katta, and patch solr-1301 provides a simple way of building
> solr
> > indexes in hadoop (as a parallel to the contrib lucene index build tool).
> >
> >
> >
> > On Fri, Sep 25, 2009 at 1:13 AM, Chandan Tamrakar <
> > chandan.tamrakar@nepasoft.com> wrote:
> >
> > > I was doing few R&D on  integrating hadoop and  Lucene . I could create
> > > lucene indexes into HDFS using  index contribution provided for Hadoop
> > >
> > >
> > >
> > > Is this a proper way to create Lucene index how much it is different
> from
> > a
> > > project KATTA ?
> > >
> > >
> > >
> > > Please suggest what would be the better approach to create distributed
> > > lucene indexes
> > >
> > >
> > >
> > > Thanks in advance
> > >
> > >
> >
> >
> > --
> > Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> > http://www.amazon.com/dp/1430219424?tag=jewlerymall
> > www.prohadoopbook.com a community for Hadoop Professionals
> >
>
>
>
> --
> Chandan Tamrakar
>



-- 
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals

--001485ea86b429b8eb047527fff1--

From general-return-578-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 05 16:44:00 2009
Return-Path: <general-return-578-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37203 invoked from network); 5 Oct 2009 16:44:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Oct 2009 16:44:00 -0000
Received: (qmail 61926 invoked by uid 500); 5 Oct 2009 16:43:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61875 invoked by uid 500); 5 Oct 2009 16:43:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61865 invoked by uid 99); 5 Oct 2009 16:43:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Oct 2009 16:43:58 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Oct 2009 16:43:46 +0000
Received: from oceanfarearth-lm.corp.yahoo.com (oceanfarearth-lm.corp.yahoo.com [10.72.113.156])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n95Gf9ki078106
	for <general@hadoop.apache.org>; Mon, 5 Oct 2009 09:41:09 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:mime-version:
	subject:date:references:x-mailer;
	b=yvBnWdEDiWG2YsUSRvaplW1MwvUmwixhPBldQvY2VvwmA13+ngZmKZxeT5vo/0Mi
Message-Id: <52336B40-7CA1-486D-ADF0-1F9B99AB1808@yahoo-inc.com>
From: Sanjay Radia <sradia@yahoo-inc.com>
To: <general@hadoop.apache.org>
In-Reply-To: <4AC2635C.20902@apache.org>
Content-Type: multipart/alternative; boundary=Apple-Mail-1--571721242
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: HTTP transport?
Date: Mon, 5 Oct 2009 09:41:09 -0700
References: <4AAAC403.80809@apache.org> <1CB467B9-B60C-4ED0-BA82-7BB571E724C6@apache.org> <4AC13BC7.6020902@apache.org> <B38E3438-CC8F-48B8-B415-549165996ED5@yahoo-inc.com> <4AC2635C.20902@apache.org>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-1--571721242
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


On Sep 29, 2009, at 12:43 PM, Doug Cutting wrote:

> Sanjay Radia wrote:
> > Wrt  connection pooling/async servers: Can't we use the same  
> libraries
> > that Jetty and Tomcat use?
> >  Grizzly?
>
> Grizzly also supports HTTP.  Choosing Grizzly is independent of  
> choosing
> HTTP as a wire transport or choosing a server.
>
Agreed.
Hence the main advantages that remain for http transport are
1)  language independent spec for the protocol. The message headers  
will be in avro so that is easy and the message exchange should be  
fairly straightforward. I see this as a minor advantage for using http  
transport.
2) code to implement the transport in multiple languages.

(2) is a significant advantage.
Once we put in the security modifications, will it remain that  
portable? We should look at that more closely.

What about out of order exchange. Will we be able to support that with  
http transport?

sanjay


--Apple-Mail-1--571721242--

From general-return-579-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 05 20:44:22 2009
Return-Path: <general-return-579-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37827 invoked from network); 5 Oct 2009 20:44:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Oct 2009 20:44:22 -0000
Received: (qmail 87668 invoked by uid 500); 5 Oct 2009 20:44:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87591 invoked by uid 500); 5 Oct 2009 20:44:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87581 invoked by uid 99); 5 Oct 2009 20:44:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Oct 2009 20:44:21 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [166.84.7.136] (HELO vc136.vc.panix.com) (166.84.7.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Oct 2009 20:44:11 +0000
Received: from eric-sammers-macbook-pro.local (cpe-66-108-32-184.nyc.res.rr.com [66.108.32.184])
	by vc136.vc.panix.com (Postfix) with ESMTP id 00DD9DC307
	for <general@hadoop.apache.org>; Mon,  5 Oct 2009 16:43:19 -0400 (EDT)
Message-ID: <4ACA5A63.1040707@lifeless.net>
Date: Mon, 05 Oct 2009 16:43:15 -0400
From: Eric Sammer <eric@lifeless.net>
User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: HTTP transport?
References: <4AAAC403.80809@apache.org>	 <1CB467B9-B60C-4ED0-BA82-7BB571E724C6@apache.org>	 <4AC13BC7.6020902@apache.org>	 <B38E3438-CC8F-48B8-B415-549165996ED5@yahoo-inc.com>	 <4AC2635C.20902@apache.org>	 <7c962aed0909291338o7f09601ci102186e23fd4d464@mail.gmail.com>	 <4AC2775C.2000302@apache.org> <7c962aed0909291457o48baee08hfa3b6f4eb16213de@mail.gmail.com> <4AC2956F.6020303@apache.org>
In-Reply-To: <4AC2956F.6020303@apache.org>
X-Enigmail-Version: 0.96.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Doug Cutting wrote:
> More or less.  Except we can probably arrange to omit most of those
> response headers except Content-Length.  Are any others strictly required?

Content-Type and Server are probably unavoidable. Some of the others are
extremely helpful during development / debugging / etc. It depends on
how "open" you are about HTTP being the transport (i.e. do you let
developers augment these headers to support additional features, etc.).
This may not make sense in the context of something specialized like
Avro transport.

> I today implemented a simple HTTP-based transport for Avro:
> 
>   https://issues.apache.org/jira/browse/AVRO-129
> 
> In some simple benchmarks I am able to make over 5000 sequential
> RPCs/second, each with ~100 bytes of response payload.

Just out of curiousity, were you using HTTP keep alive? During testing
on a project a few years ago, I found a huge difference if Keep Alive is
supported. In retrospect, that should have been obvious. I'd imagine the
usage pattern here would be a large number of repeated calls between the
same client / server within a short period of time; perfect for KA.

Regards.
-- 
Eric Sammer
eric@lifless.net
http://esammer.blogspot.com

From general-return-580-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 05 20:47:34 2009
Return-Path: <general-return-580-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39038 invoked from network); 5 Oct 2009 20:47:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Oct 2009 20:47:34 -0000
Received: (qmail 92632 invoked by uid 500); 5 Oct 2009 20:47:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92566 invoked by uid 500); 5 Oct 2009 20:47:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 92556 invoked by uid 99); 5 Oct 2009 20:47:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Oct 2009 20:47:33 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ryanobjc@gmail.com designates 209.85.217.220 as permitted sender)
Received: from [209.85.217.220] (HELO mail-gx0-f220.google.com) (209.85.217.220)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Oct 2009 20:47:23 +0000
Received: by gxk20 with SMTP id 20so3709942gxk.12
        for <general@hadoop.apache.org>; Mon, 05 Oct 2009 13:47:03 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=mvCVITG6K641M1Czi5SZdST1zk4U/mw9LpQkxRFEIEo=;
        b=IM1cNp66Q7LzjcfvD0D8OL7MhoneBellLkqAymI+0NnJLNOwy4SmMsMMlNlZL1mi29
         BLgWtaKzGDQjSE8lovb5kKS/jZJ3obW7joxqULYvKZavl7QwomnWyKTSHMDlyYcLqJgO
         ypnefDTO69slfMwBbtYMy0zMjaIxobWp/n6qE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=dfj7NuzWiavI78ilpFxX67016eTt7VfG1+7s5wZoKy6Sr2jMrBFLcB5YVo7zSrcHut
         RmmKWL73oVVjX2IlqTTFRFIv/i3GXy4uga9bGpdSGlkeuXprekbtRONgGRNko/HwE4WH
         PBYcDy4S2EnxMDyiKWZMCQyL8V/GE8lMri5cU=
MIME-Version: 1.0
Received: by 10.150.76.7 with SMTP id y7mr764893yba.238.1254775622982; Mon, 05 
	Oct 2009 13:47:02 -0700 (PDT)
In-Reply-To: <4ACA5A63.1040707@lifeless.net>
References: <4AAAC403.80809@apache.org>
	 <1CB467B9-B60C-4ED0-BA82-7BB571E724C6@apache.org>
	 <4AC13BC7.6020902@apache.org>
	 <B38E3438-CC8F-48B8-B415-549165996ED5@yahoo-inc.com>
	 <4AC2635C.20902@apache.org>
	 <7c962aed0909291338o7f09601ci102186e23fd4d464@mail.gmail.com>
	 <4AC2775C.2000302@apache.org>
	 <7c962aed0909291457o48baee08hfa3b6f4eb16213de@mail.gmail.com>
	 <4AC2956F.6020303@apache.org> <4ACA5A63.1040707@lifeless.net>
Date: Mon, 5 Oct 2009 13:47:02 -0700
Message-ID: <78568af10910051347q7ea4c92bj4e27d372fea9588@mail.gmail.com>
Subject: Re: HTTP transport?
From: Ryan Rawson <ryanobjc@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd72d9c47dce60475363706
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd72d9c47dce60475363706
Content-Type: text/plain; charset=ISO-8859-1

I have a question about these headers... will they impact the ability to do
many, but small, rpcs? Imagine you'd need to support 5,000 to 50,000
rpcs/second. Would this help or hinder?

On Oct 5, 2009 4:44 PM, "Eric Sammer" <eric@lifeless.net> wrote:

Doug Cutting wrote: > More or less. Except we can probably arrange to omit
most of those > response...
Content-Type and Server are probably unavoidable. Some of the others are
extremely helpful during development / debugging / etc. It depends on
how "open" you are about HTTP being the transport (i.e. do you let
developers augment these headers to support additional features, etc.).
This may not make sense in the context of something specialized like
Avro transport.

> I today implemented a simple HTTP-based transport for Avro: > >
https://issues.apache.org/jira...
Just out of curiousity, were you using HTTP keep alive? During testing
on a project a few years ago, I found a huge difference if Keep Alive is
supported. In retrospect, that should have been obvious. I'd imagine the
usage pattern here would be a large number of repeated calls between the
same client / server within a short period of time; perfect for KA.

Regards.
--
Eric Sammer
eric@lifless.net
http://esammer.blogspot.com

--000e0cd72d9c47dce60475363706--

From general-return-581-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 05 20:54:06 2009
Return-Path: <general-return-581-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 40485 invoked from network); 5 Oct 2009 20:54:06 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Oct 2009 20:54:06 -0000
Received: (qmail 6286 invoked by uid 500); 5 Oct 2009 20:54:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6221 invoked by uid 500); 5 Oct 2009 20:54:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6211 invoked by uid 99); 5 Oct 2009 20:54:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Oct 2009 20:54:05 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [166.84.7.136] (HELO vc136.vc.panix.com) (166.84.7.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Oct 2009 20:53:55 +0000
Received: from eric-sammers-macbook-pro.local (cpe-66-108-32-184.nyc.res.rr.com [66.108.32.184])
	by vc136.vc.panix.com (Postfix) with ESMTP id AD4F0DC307
	for <general@hadoop.apache.org>; Mon,  5 Oct 2009 16:53:34 -0400 (EDT)
Message-ID: <4ACA5CCE.3040704@lifeless.net>
Date: Mon, 05 Oct 2009 16:53:34 -0400
From: Eric Sammer <eric@lifeless.net>
User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: HTTP transport?
References: <4AAAC403.80809@apache.org>	 <1CB467B9-B60C-4ED0-BA82-7BB571E724C6@apache.org>	 <4AC13BC7.6020902@apache.org>	 <B38E3438-CC8F-48B8-B415-549165996ED5@yahoo-inc.com>	 <4AC2635C.20902@apache.org>	 <7c962aed0909291338o7f09601ci102186e23fd4d464@mail.gmail.com>	 <4AC2775C.2000302@apache.org>	 <7c962aed0909291457o48baee08hfa3b6f4eb16213de@mail.gmail.com>	 <4AC2956F.6020303@apache.org> <4ACA5A63.1040707@lifeless.net> <78568af10910051347q7ea4c92bj4e27d372fea9588@mail.gmail.com>
In-Reply-To: <78568af10910051347q7ea4c92bj4e27d372fea9588@mail.gmail.com>
X-Enigmail-Version: 0.96.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Ryan:

Certainly keep alive will help in this case, if that's what you're
referring to. The server holds the socket for N seconds or M requests,
which ever comes first. What you're saving with KA is the connection
setup / tear down. If you have a lot of cases where the client makes a
single request and goes away, then KA hurts because the server holds the
connection for the KA timeout (N seconds). This *really* helps if you're
using TLS due to the additional connection setup overhead.

It's my opinion and experience that KA helps greatly in the case of many
exchanges between a small to medium number of clients and a server such
as RPC. The anti-example is an ad server or web beacon server, for instance.

Regards.

Ryan Rawson wrote:
> I have a question about these headers... will they impact the ability to do
> many, but small, rpcs? Imagine you'd need to support 5,000 to 50,000
> rpcs/second. Would this help or hinder?
> 
> On Oct 5, 2009 4:44 PM, "Eric Sammer" <eric@lifeless.net> wrote:
> 
> Doug Cutting wrote: > More or less. Except we can probably arrange to omit
> most of those > response...
> Content-Type and Server are probably unavoidable. Some of the others are
> extremely helpful during development / debugging / etc. It depends on
> how "open" you are about HTTP being the transport (i.e. do you let
> developers augment these headers to support additional features, etc.).
> This may not make sense in the context of something specialized like
> Avro transport.
> 
>> I today implemented a simple HTTP-based transport for Avro: > >
> https://issues.apache.org/jira...
> Just out of curiousity, were you using HTTP keep alive? During testing
> on a project a few years ago, I found a huge difference if Keep Alive is
> supported. In retrospect, that should have been obvious. I'd imagine the
> usage pattern here would be a large number of repeated calls between the
> same client / server within a short period of time; perfect for KA.
> 
> Regards.
> --
> Eric Sammer
> eric@lifless.net
> http://esammer.blogspot.com
> 


-- 
Eric Sammer
eric@lifless.net
http://esammer.blogspot.com

From general-return-582-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 05 20:57:54 2009
Return-Path: <general-return-582-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42465 invoked from network); 5 Oct 2009 20:57:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Oct 2009 20:57:54 -0000
Received: (qmail 12635 invoked by uid 500); 5 Oct 2009 20:57:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12552 invoked by uid 500); 5 Oct 2009 20:57:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12542 invoked by uid 99); 5 Oct 2009 20:57:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Oct 2009 20:57:53 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ryanobjc@gmail.com designates 209.85.211.189 as permitted sender)
Received: from [209.85.211.189] (HELO mail-yw0-f189.google.com) (209.85.211.189)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Oct 2009 20:57:43 +0000
Received: by ywh27 with SMTP id 27so7184910ywh.2
        for <general@hadoop.apache.org>; Mon, 05 Oct 2009 13:57:22 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=dSmOhCUKi2/rzxv6nuELiNcCjyYvrUXupqIOBkwJ078=;
        b=iXQU6On+D/NE8Q/XMtrphvJyHuOlVHkLdt7JIQ4utI4TVGpLj4fqnyW8pKPNpO70Qf
         NGXyGkQJlOAHGTTVGE15tmoaJILJLpXmvjnvLFvtam3/TmZab860pmmcvRegAzdCEFX3
         6ttPEoeDGsil6cYkdkE6KfikO3w/2KEHLQoDM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Am9OiOk5Q3KwAMCPzSlN1Ea2ZnVBC6BlO1DIvuHhbWgaBsJS+8QTCOjq1RnHeC3h8H
         9EyGs0PyOvAhXbYGZ9GBaiIA6E64mhskZSVsMRR55YkhZW8DPiXr3JpGHaAsinPhb70B
         6JarUo/yga0Lgmkm01GKXS/4vM5WTmYA+uZrk=
MIME-Version: 1.0
Received: by 10.150.87.1 with SMTP id k1mr857005ybb.11.1254776242728; Mon, 05 
	Oct 2009 13:57:22 -0700 (PDT)
In-Reply-To: <78568af10910051356t4ad5f0d2ld361d10b6d378dba@mail.gmail.com>
References: <4AAAC403.80809@apache.org> <4AC2635C.20902@apache.org>
	 <7c962aed0909291338o7f09601ci102186e23fd4d464@mail.gmail.com>
	 <4AC2775C.2000302@apache.org>
	 <7c962aed0909291457o48baee08hfa3b6f4eb16213de@mail.gmail.com>
	 <4AC2956F.6020303@apache.org> <4ACA5A63.1040707@lifeless.net>
	 <78568af10910051347q7ea4c92bj4e27d372fea9588@mail.gmail.com>
	 <4ACA5CCE.3040704@lifeless.net>
	 <78568af10910051356t4ad5f0d2ld361d10b6d378dba@mail.gmail.com>
Date: Mon, 5 Oct 2009 13:57:22 -0700
Message-ID: <78568af10910051357h1e3b8a62jb995e110eea163e1@mail.gmail.com>
Subject: Re: HTTP transport?
From: Ryan Rawson <ryanobjc@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd5997c386dc80475365c2f
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd5997c386dc80475365c2f
Content-Type: text/plain; charset=ISO-8859-1

That's good to know. I thought ka would help... but I was also talking about
the overhead of a header where the payload is smaller than the framing. Eg:
8 byte requests, excluding which rpc. This seems like we could be hurt since
the headers are potentially 5x the size of our payload/request params.

On Oct 5, 2009 4:54 PM, "Eric Sammer" <eric@lifeless.net> wrote:

Ryan:

Certainly keep alive will help in this case, if that's what you're
referring to. The server holds the socket for N seconds or M requests,
which ever comes first. What you're saving with KA is the connection
setup / tear down. If you have a lot of cases where the client makes a
single request and goes away, then KA hurts because the server holds the
connection for the KA timeout (N seconds). This *really* helps if you're
using TLS due to the additional connection setup overhead.

It's my opinion and experience that KA helps greatly in the case of many
exchanges between a small to medium number of clients and a server such
as RPC. The anti-example is an ad server or web beacon server, for instance.

Regards.

Ryan Rawson wrote: > I have a question about these headers... will they
impact the ability to do > ...
--

Eric Sammer eric@lifless.net http://esammer.blogspot.com

--000e0cd5997c386dc80475365c2f--

From general-return-583-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 05 21:14:53 2009
Return-Path: <general-return-583-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47570 invoked from network); 5 Oct 2009 21:14:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Oct 2009 21:14:53 -0000
Received: (qmail 46618 invoked by uid 500); 5 Oct 2009 21:14:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46538 invoked by uid 500); 5 Oct 2009 21:14:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46528 invoked by uid 99); 5 Oct 2009 21:14:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Oct 2009 21:14:52 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [166.84.7.136] (HELO vc136.vc.panix.com) (166.84.7.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Oct 2009 21:14:42 +0000
Received: from eric-sammers-macbook-pro.local (cpe-66-108-32-184.nyc.res.rr.com [66.108.32.184])
	by vc136.vc.panix.com (Postfix) with ESMTP id B70E3DC307
	for <general@hadoop.apache.org>; Mon,  5 Oct 2009 17:13:51 -0400 (EDT)
Message-ID: <4ACA618F.5040709@lifeless.net>
Date: Mon, 05 Oct 2009 17:13:51 -0400
From: Eric Sammer <eric@lifeless.net>
User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: HTTP transport?
References: <4AAAC403.80809@apache.org> <4AC2635C.20902@apache.org>	 <7c962aed0909291338o7f09601ci102186e23fd4d464@mail.gmail.com>	 <4AC2775C.2000302@apache.org>	 <7c962aed0909291457o48baee08hfa3b6f4eb16213de@mail.gmail.com>	 <4AC2956F.6020303@apache.org> <4ACA5A63.1040707@lifeless.net>	 <78568af10910051347q7ea4c92bj4e27d372fea9588@mail.gmail.com>	 <4ACA5CCE.3040704@lifeless.net>	 <78568af10910051356t4ad5f0d2ld361d10b6d378dba@mail.gmail.com> <78568af10910051357h1e3b8a62jb995e110eea163e1@mail.gmail.com>
In-Reply-To: <78568af10910051357h1e3b8a62jb995e110eea163e1@mail.gmail.com>
X-Enigmail-Version: 0.96.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Ryan Rawson wrote:
> That's good to know. I thought ka would help... but I was also talking about
> the overhead of a header where the payload is smaller than the framing. Eg:
> 8 byte requests, excluding which rpc. This seems like we could be hurt since
> the headers are potentially 5x the size of our payload/request params.
> 

Oh, got you. That's the classic SOAP problem. ;) I think it's possible,
but to what degree I couldn't be sure. HTTP has that kind of overhead
because of the generality. I think you'd get that with anything that
isn't specifically designed to be wire efficient. Of course, you wind up
having to do what Doug originally mentioned; rebuilding and maintaining
the original stuff that HTTP (and the supporting clients) already supports.

If over-optimization is in fact the root of all evil (which I've heard
once or twice) then maybe it makes sense to start simple and iterate if
necessary. In other words, say screw it, use HTTP, strip unnecessary
headers, but design the code such that Avro's transport is interface
based in case it needs to change. I think prototyping an Avro transport
with HTTP, optimizing, and then dropping lower level if necessary is a
better approach than going straight to the latter.

All of that said, I don't have the insight into the code base that some
of the other folks do. This is based on my experience with similar high
throughput systems, but I wouldn't say I'm 100% convinced it applies
here as the payloads in those systems were bigger than 8 bytes.

-- 
Eric Sammer
eric@lifless.net
http://esammer.blogspot.com

From general-return-584-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 05 23:48:50 2009
Return-Path: <general-return-584-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98521 invoked from network); 5 Oct 2009 23:48:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Oct 2009 23:48:50 -0000
Received: (qmail 98846 invoked by uid 500); 5 Oct 2009 23:48:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98773 invoked by uid 500); 5 Oct 2009 23:48:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98763 invoked by uid 99); 5 Oct 2009 23:48:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Oct 2009 23:48:49 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 05 Oct 2009 23:48:47 +0000
Received: (qmail 98493 invoked by uid 99); 5 Oct 2009 23:48:25 -0000
Received: from localhost.apache.org (HELO [192.168.168.102]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Oct 2009 23:48:25 +0000
Message-ID: <4ACA85C8.9000804@apache.org>
Date: Mon, 05 Oct 2009 16:48:24 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: HTTP transport?
References: <4AAAC403.80809@apache.org> <1CB467B9-B60C-4ED0-BA82-7BB571E724C6@apache.org> <4AC13BC7.6020902@apache.org> <B38E3438-CC8F-48B8-B415-549165996ED5@yahoo-inc.com> <4AC2635C.20902@apache.org> <52336B40-7CA1-486D-ADF0-1F9B99AB1808@yahoo-inc.com>
In-Reply-To: <52336B40-7CA1-486D-ADF0-1F9B99AB1808@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Sanjay Radia wrote:
> What about out of order exchange. Will we be able to support that with 
> http transport?

Out-of-order exchange was originally added to Hadoop's RPC when it was a 
part of Nutch.  It's an important optimization for distributed search, 
but it's not clear how important it is currently to Hadoop.

That said, the simple way to deal with this in HTTP is to use a client 
library that pools connections, so that, if a second request to the same 
service is made by another thread in the same client process before the 
first has returned, a second connection is opened.  If this is common, 
the high-water mark of connections on the server will be higher. 
However with an async-io-based server, the number of connections should 
not be a primary bottleneck.  And again, we don't know how common this is.

Doug

From general-return-585-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 06 03:00:08 2009
Return-Path: <general-return-585-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46678 invoked from network); 6 Oct 2009 03:00:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Oct 2009 03:00:08 -0000
Received: (qmail 38876 invoked by uid 500); 6 Oct 2009 03:00:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38803 invoked by uid 500); 6 Oct 2009 03:00:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 38793 invoked by uid 99); 6 Oct 2009 03:00:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Oct 2009 03:00:07 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [64.78.17.17] (HELO EXHUB018-2.exch018.msoutlookonline.net) (64.78.17.17)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Oct 2009 02:59:57 +0000
Received: from EXVMBX018-1.exch018.msoutlookonline.net ([64.78.17.47]) by
 EXHUB018-2.exch018.msoutlookonline.net ([64.78.17.17]) with mapi; Mon, 5 Oct
 2009 19:59:35 -0700
From: Scott Carey <scott@richrelevance.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 5 Oct 2009 19:59:24 -0700
Subject: Re: HTTP transport?
Thread-Topic: HTTP transport?
Thread-Index: AcpF/f1TOLX+HK0OTAeTv/iOfb0/LQAMwUAI
Message-ID: <C6F0009C.1380D%scott@richrelevance.com>
In-Reply-To: <4ACA5CCE.3040704@lifeless.net>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


On 10/5/09 1:53 PM, "Eric Sammer" <eric@lifeless.net> wrote:

> Ryan:
>=20
> Certainly keep alive will help in this case, if that's what you're
> referring to. The server holds the socket for N seconds or M requests,
> which ever comes first. What you're saving with KA is the connection
> setup / tear down. If you have a lot of cases where the client makes a
> single request and goes away, then KA hurts because the server holds the
> connection for the KA timeout (N seconds). This *really* helps if you're
> using TLS due to the additional connection setup overhead.
>=20
> It's my opinion and experience that KA helps greatly in the case of many
> exchanges between a small to medium number of clients and a server such
> as RPC. The anti-example is an ad server or web beacon server, for instan=
ce.
>=20

Even in the beacon case, if the browser is likely to send another request
shortly, it cuts the effective network latency in half.  Establishing a TCP
connection is at minimum one full round trip -- before the request.  If
latency is important KeepAlive is useful as long as a second request is
expected in a short enough time.

As long as the server is not process or thread per connection, one can scal=
e
up connection count rather high (20k) if necessary.

With respect to Avro/Hadoop, I suspect requests from clients to be time
clustered.


> Regards.
>=20
> Ryan Rawson wrote:
>> I have a question about these headers... will they impact the ability to=
 do
>> many, but small, rpcs? Imagine you'd need to support 5,000 to 50,000
>> rpcs/second. Would this help or hinder?
>>=20
>> On Oct 5, 2009 4:44 PM, "Eric Sammer" <eric@lifeless.net> wrote:
>>=20
>> Doug Cutting wrote: > More or less. Except we can probably arrange to om=
it
>> most of those > response...
>> Content-Type and Server are probably unavoidable. Some of the others are
>> extremely helpful during development / debugging / etc. It depends on
>> how "open" you are about HTTP being the transport (i.e. do you let
>> developers augment these headers to support additional features, etc.).
>> This may not make sense in the context of something specialized like
>> Avro transport.
>>=20
>>> I today implemented a simple HTTP-based transport for Avro: > >
>> https://issues.apache.org/jira...
>> Just out of curiousity, were you using HTTP keep alive? During testing
>> on a project a few years ago, I found a huge difference if Keep Alive is
>> supported. In retrospect, that should have been obvious. I'd imagine the
>> usage pattern here would be a large number of repeated calls between the
>> same client / server within a short period of time; perfect for KA.
>>=20
>> Regards.
>> --
>> Eric Sammer
>> eric@lifless.net
>> http://esammer.blogspot.com
>>=20
>=20
>=20
> --
> Eric Sammer
> eric@lifless.net
> http://esammer.blogspot.com
>=20


From general-return-586-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 06 03:16:11 2009
Return-Path: <general-return-586-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50131 invoked from network); 6 Oct 2009 03:16:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Oct 2009 03:16:11 -0000
Received: (qmail 49233 invoked by uid 500); 6 Oct 2009 03:16:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49165 invoked by uid 500); 6 Oct 2009 03:16:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49155 invoked by uid 99); 6 Oct 2009 03:16:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Oct 2009 03:16:10 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [166.84.7.136] (HELO vc136.vc.panix.com) (166.84.7.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Oct 2009 03:16:00 +0000
Received: from eric-sammers-macbook-pro.local (cpe-66-108-32-184.nyc.res.rr.com [66.108.32.184])
	by vc136.vc.panix.com (Postfix) with ESMTP id 6B438DC37C
	for <general@hadoop.apache.org>; Mon,  5 Oct 2009 23:15:09 -0400 (EDT)
Message-ID: <4ACAB63C.4010803@lifeless.net>
Date: Mon, 05 Oct 2009 23:15:08 -0400
From: Eric Sammer <eric@lifeless.net>
User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: HTTP transport?
References: <C6F0009C.1380D%scott@richrelevance.com>
In-Reply-To: <C6F0009C.1380D%scott@richrelevance.com>
X-Enigmail-Version: 0.96.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Scott Carey wrote:
> Even in the beacon case, if the browser is likely to send another request
> shortly, it cuts the effective network latency in half.

Which is generally not the case in the beacon / ad server use case. That
was the only point I was making. That's besides the point, though. I
think we both agree that KA for something like Avro transport is
probably good.

> Establishing a TCP
> connection is at minimum one full round trip -- before the request.  If
> latency is important KeepAlive is useful as long as a second request is
> expected in a short enough time.
> 
> As long as the server is not process or thread per connection, one can scale
> up connection count rather high (20k) if necessary.
> 
> With respect to Avro/Hadoop, I suspect requests from clients to be time
> clustered.

That was my thought as well. The thing that gets me is that in the case
of Hadoop (and the related subprojects) the clients utilizing this
particular HTTP connection are probably going to be pretty small (maybe
low thousands?). This is even better for keep alive as there's a solid
chance you're going to have a high reuse rate. Of course, I'm assuming
we're talking about things like name node to data node, hbase client to
region servers, and those types of communications. Even if you just used
Jetty (or any other thin HTTP 1.1 container that supports KA), one
should easily be able to see good performance.

Regards.
-- 
Eric Sammer
eric@lifless.net
http://esammer.blogspot.com

From general-return-587-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 06 03:20:39 2009
Return-Path: <general-return-587-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 50805 invoked from network); 6 Oct 2009 03:20:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Oct 2009 03:20:38 -0000
Received: (qmail 51862 invoked by uid 500); 6 Oct 2009 03:20:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51801 invoked by uid 500); 6 Oct 2009 03:20:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51791 invoked by uid 99); 6 Oct 2009 03:20:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Oct 2009 03:20:37 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [64.78.17.16] (HELO EXHUB018-1.exch018.msoutlookonline.net) (64.78.17.16)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Oct 2009 03:20:27 +0000
Received: from EXVMBX018-1.exch018.msoutlookonline.net ([64.78.17.47]) by
 EXHUB018-1.exch018.msoutlookonline.net ([64.78.17.16]) with mapi; Mon, 5 Oct
 2009 20:19:45 -0700
From: Scott Carey <scott@richrelevance.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 5 Oct 2009 20:19:34 -0700
Subject: Re: HTTP transport?
Thread-Topic: HTTP transport?
Thread-Index: AcpF/RNaX39mo3/YRk6Eiof+R4/2AwANsAyI
Message-ID: <C6F00556.1381B%scott@richrelevance.com>
In-Reply-To: <78568af10910051347q7ea4c92bj4e27d372fea9588@mail.gmail.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org




On 10/5/09 1:47 PM, "Ryan Rawson" <ryanobjc@gmail.com> wrote:

> I have a question about these headers... will they impact the ability to =
do
> many, but small, rpcs? Imagine you'd need to support 5,000 to 50,000
> rpcs/second. Would this help or hinder?
>=20

As long as the HTTP response and request fit in one network packet
(pessimistic - 1KB or so) there is not much overhead.

50k rpcs/sec with gigabit ethernet saturated (~100MB/sec) is ~2KB per
request.

So, on faster networks an extra 100 to 200 bytes or so won't matter.

On the WAN, it will have more of an effect if the bandwidth is low and the
latency also very low if the RPC is very 'chatty' and not 'chunky' enough.

However, on most WAN links network latency is going to kill you far, far
more than an extra 200 bytes.
For example, imagine a 20ms latency link.  The max RPC throughput to a
single client is then 50/sec (one per 20ms).  With a 1k payload per request=
,
that's 50k per sec max data transfer.  HTTP pipelining could help here --
but isn't as well supported as one would like.

If WAN level RPC is a goal, the main challenges there will be latency
related first, and packet size related second.
On a fast local network (gigabit) I suspect throughput problems of other
sorts to be the issue before bandwidth from slightly larger packets.

Furthermore, its not like a TCP packet is 0 bytes on its own.  HTTP adds
some overhead, but it can be kept relatively trim.=20


From general-return-588-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 06 03:31:59 2009
Return-Path: <general-return-588-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 52659 invoked from network); 6 Oct 2009 03:31:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Oct 2009 03:31:58 -0000
Received: (qmail 57026 invoked by uid 500); 6 Oct 2009 03:31:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56901 invoked by uid 500); 6 Oct 2009 03:31:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56718 invoked by uid 99); 6 Oct 2009 03:31:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Oct 2009 03:31:57 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [64.78.17.17] (HELO EXHUB018-2.exch018.msoutlookonline.net) (64.78.17.17)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Oct 2009 03:31:46 +0000
Received: from EXVMBX018-1.exch018.msoutlookonline.net ([64.78.17.47]) by
 EXHUB018-2.exch018.msoutlookonline.net ([64.78.17.17]) with mapi; Mon, 5 Oct
 2009 20:31:04 -0700
From: Scott Carey <scott@richrelevance.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 5 Oct 2009 20:30:50 -0700
Subject: Re: HTTP transport?
Thread-Topic: HTTP transport?
Thread-Index: AcpGM1zaG6gyouT6S/iGbWfT3OWuyAAAgmjo
Message-ID: <C6F007FA.1381D%scott@richrelevance.com>
In-Reply-To: <4ACAB63C.4010803@lifeless.net>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


>> With respect to Avro/Hadoop, I suspect requests from clients to be time
>> clustered.
>=20
> That was my thought as well. The thing that gets me is that in the case
> of Hadoop (and the related subprojects) the clients utilizing this
> particular HTTP connection are probably going to be pretty small (maybe
> low thousands?). This is even better for keep alive as there's a solid
> chance you're going to have a high reuse rate. Of course, I'm assuming
> we're talking about things like name node to data node, hbase client to
> region servers, and those types of communications. Even if you just used
> Jetty (or any other thin HTTP 1.1 container that supports KA), one
> should easily be able to see good performance.
>=20

Absolutely. =20
Additionally, I realized one more thing -- when a client knows they aren't
likely to send another request soon, they can send a request with
Connection: close.

Well behaved clients can help maximize the benefit.




> Regards.
> --
> Eric Sammer
> eric@lifless.net
> http://esammer.blogspot.com
>=20


From general-return-589-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 08 22:11:33 2009
Return-Path: <general-return-589-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33086 invoked from network); 8 Oct 2009 22:11:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Oct 2009 22:11:33 -0000
Received: (qmail 74127 invoked by uid 500); 8 Oct 2009 22:11:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74044 invoked by uid 500); 8 Oct 2009 22:11:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74034 invoked by uid 99); 8 Oct 2009 22:11:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Oct 2009 22:11:32 +0000
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.59.227] (HELO QMTA12.westchester.pa.mail.comcast.net) (76.96.59.227)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Oct 2009 22:11:21 +0000
Received: from OMTA22.westchester.pa.mail.comcast.net ([76.96.62.73])
	by QMTA12.westchester.pa.mail.comcast.net with comcast
	id qAsF1c0051ap0As5CNB1QS; Thu, 08 Oct 2009 22:11:01 +0000
Received: from tryarticle-lm.corp.yahoo.com ([209.131.62.113])
	by OMTA22.westchester.pa.mail.comcast.net with comcast
	id qNFQ1c0052SbwD53iNFTt7; Thu, 08 Oct 2009 22:15:31 +0000
Message-Id: <6A69E66F-E8D3-49D4-8068-AF84D651EC8A@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <C6F007FA.1381D%scott@richrelevance.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: HTTP transport?
Date: Thu, 8 Oct 2009 15:10:50 -0700
References: <C6F007FA.1381D%scott@richrelevance.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

I still don't see how to make this play well with security. Security  
needs to go under the transport layer so that it is easy to add  
encryption on the wire. If you go with HTTP, the only way that is  
portable at all is to use HTTP over SSL. SSL is for when there aren't  
shared keys and Kerberos provides those shared keys. SPNEGO is the  
standard method of using Kerberos with HTTP and we are planning to use  
that for the web UI's. But SPNEGO is very much the least painful of  
the alternatives and I'd rather not force our RPC services into that  
corner.

I also have serious doubts about performance, but that is hard to  
answer until we have code to test. It is an interesting question how  
much we depend on being able to answer queries out of order. There are  
some parts of the code where overlapping requests from the same client  
matter. In particular, the terasort scheduler uses threads to access  
the namenode. That would stop providing any pipelining, which I  
believe would be significant.

In short, I think that an HTTP transport is great for playing with,  
but I don't think you can assume it will work as the primary transport.

-- Owen


From general-return-590-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 09 17:50:31 2009
Return-Path: <general-return-590-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98140 invoked from network); 9 Oct 2009 17:50:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Oct 2009 17:50:31 -0000
Received: (qmail 88959 invoked by uid 500); 9 Oct 2009 17:50:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88896 invoked by uid 500); 9 Oct 2009 17:50:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88886 invoked by uid 99); 9 Oct 2009 17:50:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Oct 2009 17:50:29 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 09 Oct 2009 17:50:18 +0000
Received: (qmail 98047 invoked by uid 99); 9 Oct 2009 17:49:56 -0000
Received: from localhost.apache.org (HELO [192.168.168.109]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Oct 2009 17:49:56 +0000
Message-ID: <4ACF77C3.6020809@apache.org>
Date: Fri, 09 Oct 2009 10:49:55 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: HTTP transport?
References: <C6F007FA.1381D%scott@richrelevance.com> <6A69E66F-E8D3-49D4-8068-AF84D651EC8A@apache.org>
In-Reply-To: <6A69E66F-E8D3-49D4-8068-AF84D651EC8A@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Owen O'Malley wrote:
> SPNEGO is the 
> standard method of using Kerberos with HTTP and we are planning to use 
> that for the web UI's.

Java 6 also supports using SPNEGO for RPC over HTTP out of the box:

http://java.sun.com/javase/6/docs/technotes/guides/net/http-auth.html

> I also have serious doubts about performance, but that is hard to answer 
> until we have code to test.

The good news is that, since the HTTP stuff is already implemented, we 
can test its performance easily.  Performance of insecure access over 
HTTP looks good so far.  It's an open question are how much HTTP-based 
security will slow things versus non-HTTP-based security.

> It is an interesting question how much we 
> depend on being able to answer queries out of order. There are some 
> parts of the code where overlapping requests from the same client 
> matter. In particular, the terasort scheduler uses threads to access the 
> namenode. That would stop providing any pipelining, which I believe 
> would be significant.

No, we wouldn't stop any pipelining, we'd just use more connections to 
implement it.  With HttpClient one can limit the number of pooled 
connnections per host:

http://hc.apache.org/httpclient-3.x/apidocs/org/apache/commons/httpclient/MultiThreadedHttpConnectionManager.html#setMaxConnectionsPerHost%28int%29

Connections are not free of course, but Jetty has been benchmarked at 
20,000 concurrent connections:

http://cometdaily.com/2008/01/07/20000-reasons-that-comet-scales/

> In short, I think that an HTTP transport is great for playing with, but 
> I don't think you can assume it will work as the primary transport.

I agree, we cannot assume it.  But it's easy to try it and see how it 
fares.  Any investment in getting it working is perhaps not wasted, 
since, besides providing a performance baseline, it also may be useful 
to provide HTTP-based access to services even if a higher-performance 
option is implemented.

Doug

From general-return-591-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 09 18:14:48 2009
Return-Path: <general-return-591-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5551 invoked from network); 9 Oct 2009 18:14:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Oct 2009 18:14:48 -0000
Received: (qmail 25988 invoked by uid 500); 9 Oct 2009 18:14:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25930 invoked by uid 500); 9 Oct 2009 18:14:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25920 invoked by uid 99); 9 Oct 2009 18:14:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Oct 2009 18:14:47 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Oct 2009 18:14:34 +0000
Received: from SNV-EXBH01.ds.corp.yahoo.com (snv-exbh01.ds.corp.yahoo.com [207.126.227.249])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n99IDBoa024647
	for <general@hadoop.apache.org>; Fri, 9 Oct 2009 11:13:11 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:x-originalarrivaltime;
	b=G941LSnIbISnmt8yOlL6Vhi/teNdTEPxs4ICJOfJVIgFMWo4N0TbYmtrP/4UpiSb
Received: from SNV-EXVS05.ds.corp.yahoo.com ([207.126.227.225]) by SNV-EXBH01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Fri, 9 Oct 2009 11:13:10 -0700
Received: from 10.72.244.163 ([10.72.244.163]) by SNV-EXVS05.ds.corp.yahoo.com ([207.126.227.45]) with Microsoft Exchange Server HTTP-DAV ;
 Fri,  9 Oct 2009 18:13:10 +0000
User-Agent: Microsoft-Entourage/12.20.0.090605
Date: Fri, 09 Oct 2009 11:13:09 -0700
Subject: Re: HTTP transport?
From: Sanjay Radia <sradia@yahoo-inc.com>
To: <general@hadoop.apache.org>
Message-ID: <C6F4CB45.2FBD2%sradia@yahoo-inc.com>
Thread-Topic: HTTP transport?
Thread-Index: AcpJDCfStlBLAXfKQUu0VVDx7iJBgA==
In-Reply-To: <4ACF77C3.6020809@apache.org>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-OriginalArrivalTime: 09 Oct 2009 18:13:10.0748 (UTC) FILETIME=[28DCE1C0:01CA490C]
X-Virus-Checked: Checked by ClamAV on apache.org




On 10/9/09 10:49 AM, "Doug Cutting" <cutting@apache.org> wrote:

> Owen O'Malley wrote:
>> SPNEGO is the 
>> standard method of using Kerberos with HTTP and we are planning to use
>> that for the web UI's.
> 
> Java 6 also supports using SPNEGO for RPC over HTTP out of the box:
> 
> http://java.sun.com/javase/6/docs/technotes/guides/net/http-auth.html
> 
>> I also have serious doubts about performance, but that is hard to answer
>> until we have code to test.
> 
> The good news is that, since the HTTP stuff is already implemented, we
> can test its performance easily.  Performance of insecure access over
> HTTP looks good so far.  It's an open question are how much HTTP-based
> security will slow things versus non-HTTP-based security.
> 
>> It is an interesting question how much we
>> depend on being able to answer queries out of order. There are some
>> parts of the code where overlapping requests from the same client
>> matter. In particular, the terasort scheduler uses threads to access the
>> namenode. That would stop providing any pipelining, which I believe
>> would be significant.
> 
> No, we wouldn't stop any pipelining, we'd just use more connections to
> implement it.  With HttpClient one can limit the number of pooled
> connnections per host:
> 
> http://hc.apache.org/httpclient-3.x/apidocs/org/apache/commons/httpclient/Mult
> iThreadedHttpConnectionManager.html#setMaxConnectionsPerHost%28int%29
> 
> Connections are not free of course, but Jetty has been benchmarked at
> 20,000 concurrent connections:
> 
> http://cometdaily.com/2008/01/07/20000-reasons-that-comet-scales/
> 
>> In short, I think that an HTTP transport is great for playing with, but
>> I don't think you can assume it will work as the primary transport.
> 
> I agree, we cannot assume it.  But it's easy to try it and see how it
> fares.  Any investment in getting it working is perhaps not wasted,
> since, besides providing a performance baseline, it also may be useful
> to provide HTTP-based access to services even if a higher-performance
> option is implemented.

Will the RPC over HTTP be transparent so that that we can replace with a
different layer if needed?
My worry was the separation of data and checksums; someone had mentioned
that one could do this over 2 RPCs - that is not transparent.

Also the other issue is porting from data transfer socket streams to RPC -
that port will not be transparent. We cannot afford to loose performance
over that change. Further,  moving from streaming sockets to RPC is a very
significant code change to the dfs-client and data nodes. I assume that we
going to create a branch that moves the data transfer protocols to RPC and
test the performance and if it is good then we commit and move to RPC?
I am worried about this part - I am surprised that you two are not. Am I
missing something here?



sanjay



> 
> Doug


From general-return-592-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 09 19:56:57 2009
Return-Path: <general-return-592-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42774 invoked from network); 9 Oct 2009 19:56:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Oct 2009 19:56:57 -0000
Received: (qmail 83284 invoked by uid 500); 9 Oct 2009 19:56:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83219 invoked by uid 500); 9 Oct 2009 19:56:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83209 invoked by uid 99); 9 Oct 2009 19:56:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Oct 2009 19:56:56 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 09 Oct 2009 19:56:53 +0000
Received: (qmail 41559 invoked by uid 99); 9 Oct 2009 19:56:32 -0000
Received: from localhost.apache.org (HELO [192.168.168.109]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Oct 2009 19:56:32 +0000
Message-ID: <4ACF956E.1000201@apache.org>
Date: Fri, 09 Oct 2009 12:56:30 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: HTTP transport?
References: <C6F4CB45.2FBD2%sradia@yahoo-inc.com>
In-Reply-To: <C6F4CB45.2FBD2%sradia@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Sanjay Radia wrote:
> Will the RPC over HTTP be transparent so that that we can replace with a
> different layer if needed?

Yes.

> My worry was the separation of data and checksums; someone had mentioned
> that one could do this over 2 RPCs - that is not transparent.

That was suggested as a possibility if we did not want to use RPC for 
data, but rather raw HTTP, e.g., with a separate URL per block.  The 
zerocopy support built into most HTTP servers only supports entire 
responses from a single file, so if we wanted to take advantage of these 
zerocopy implementations we'd not use RPC for block access, but could 
use HTTP and hence share security, etc.  Using raw HTTP for block access 
might also perform better, since it can use TCP flow control, rather 
than RPC call/response.  In my microbenchmarks, RPC call/response was 
fast enough to easily saturate disks and networks, so that might be 
moot, although RPC call/response for file data may use more CPU than 
we'd like.  With our own transport implementation we could get RPC 
call/response to use zerocopy for file data.

> I assume that we
> going to create a branch that moves the data transfer protocols to RPC and
> test the performance and if it is good then we commit and move to RPC?

Yes.  We obviously cannot change the file data transfer protocol without 
benchmarking.  Ideally file data transfer can share as much as possible 
with other protocols.  The most optimistic approach would be to use 
HTTP-based RPC call/response, so we ought to benchmark that.  This was 
the purpose of my recently-reported microbenchmarks.

We also need to determine whether both TCP flow-control and zerocopy are 
critical to data file performance.  If both are indeed critical, and 
HTTP proves sufficient for everything else, then we should consider 
using non-RPC HTTP for file data transfer, since it supports both 
zerocopy and TCP-based flow control, and the implementation of security, 
etc. could be shared.  But, on the other hand, if HTTP is deemed 
inappropriate for security and we develop our own RPC transport that 
permits zerocopy, and TCP flow-control over entire blocks is not 
required, then we might use RPC for file data.  What I'm hoping we can 
avoid is, as today, using different transports for different protocols, 
re-implementing security, connection pooling, async request processing, 
etc. for each, requiring separate configuration and ports for each, etc. 
  But even that might be required.  We don't know yet.

I think starting with HTTP as a hypothesis permits us to make progress 
without a lot of up-front investment.

Doug

From general-return-593-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 10 17:49:06 2009
Return-Path: <general-return-593-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 11567 invoked from network); 10 Oct 2009 17:49:06 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Oct 2009 17:49:06 -0000
Received: (qmail 91753 invoked by uid 500); 10 Oct 2009 17:49:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91680 invoked by uid 500); 10 Oct 2009 17:49:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91670 invoked by uid 99); 10 Oct 2009 17:49:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Oct 2009 17:49:05 +0000
X-ASF-Spam-Status: No, hits=4.0 required=10.0
	tests=HTML_MESSAGE,SPF_PASS,SUBJ_ALL_CAPS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of cabiwan@gmail.com designates 209.85.218.222 as permitted sender)
Received: from [209.85.218.222] (HELO mail-bw0-f222.google.com) (209.85.218.222)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Oct 2009 17:48:56 +0000
Received: by bwz22 with SMTP id 22so6695800bwz.29
        for <general@hadoop.apache.org>; Sat, 10 Oct 2009 10:48:35 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=kbBE40dJLS3Lf5TsUkoNaZhlukJshDD7YygHaKVRjyI=;
        b=l2zV3ERh3ehMvAmPUuQ51wEFxSLlPsKv4iE702jkam1ZsyDdGUTyioxcghbodDjxR3
         h14EkI3cX+KI+b8Bb0f04x223NTcmgRR8K12WVE2Kge55D22NgLl37P7vZmTzEn6m+y9
         QamoV21icgAgOC2QQfuRqlSqo0D0ucW+BSHpQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=qBN+mY8P8BVa+HtcpycHn2xaJ/+psCnu6FATROCOqKu8fW2Z+A9DaCBLUkUt3bf+Ij
         8kO+Ma0jwY3jt05wdaSytNlt4epf20hxwqnJ8yrd7cHoj7D1fnN24NmvONvsxVG4iKsj
         VSrpvKDJJqdnqdPp5+WMa1y+xNugH6yDxXpGA=
MIME-Version: 1.0
Received: by 10.103.81.2 with SMTP id i2mr1675468mul.81.1255196915502; Sat, 10 
	Oct 2009 10:48:35 -0700 (PDT)
Date: Sat, 10 Oct 2009 19:48:35 +0200
Message-ID: <309f76d00910101048x4028765fg477f1b249e223c8b@mail.gmail.com>
Subject: SIMPLE QUESTIONS ABOUT DISTRIBUTEDCACHE...
From: Alberto Luengo Cabanillas <cabiwan@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6482b5c45912b0475984e67
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6482b5c45912b0475984e67
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi everyone! I=B4m trying to distribute some common data between my nodes, =
but
I=B4m unable to do it. I have some questions about this:

1.- I=B4m working in PseudoDistributed mode in Windows (under Cygwin) and m=
y
DFS server is running on a virtual machine (Ubuntu server). May I use
DistributedCache normally with this configuration?
2.-In the case of the Mapper nodes, in the Driver class I add these two
lines:

URI uriMapper =3Dnew URI("/user/hadoop-user/data/mapper_configuration.dat")=
;
(I also verify this file already exists in the DFS).
DistributedCache.addCacheFile(uriMapper,conf);

3.- In the Mapper class, I have this:

@Override
    protected void setup(Context cont) {
       ...
       String configureCacheName =3D new
Path("/user/hadoop-user/data/mapper_configuration.dat").getName();
       Path [] cacheFiles =3D
DistributedCache.getLocalCacheFiles(cont.getConfiguration());
       System.out.println("The file is: "+cacheFiles.toString());
     ...
    }

Does anyone have an idea about what I=B4m doing wrong? (I=B4m going crazy,
actually...)

Thanks a lot in advance.

--=20
Alberto

--0016e6482b5c45912b0475984e67--

From general-return-594-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 10 18:39:21 2009
Return-Path: <general-return-594-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 26838 invoked from network); 10 Oct 2009 18:39:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Oct 2009 18:39:21 -0000
Received: (qmail 11738 invoked by uid 500); 10 Oct 2009 18:39:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11654 invoked by uid 500); 10 Oct 2009 18:39:20 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11644 invoked by uid 99); 10 Oct 2009 18:39:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Oct 2009 18:39:20 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jason.hadoop@gmail.com designates 209.85.212.186 as permitted sender)
Received: from [209.85.212.186] (HELO mail-vw0-f186.google.com) (209.85.212.186)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Oct 2009 18:39:09 +0000
Received: by vws16 with SMTP id 16so5548525vws.2
        for <general@hadoop.apache.org>; Sat, 10 Oct 2009 11:38:48 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=u+Ygi/Rf11Rtk6v4XIWQuAWYQXsteOa1BKxjs3ydNqo=;
        b=t9KFPc+gTPR/JBDghvoFX0dv2FgvPWQ0cULVrdrB9ki2d5Me2IkzUwv118srH7bL3W
         dmXVPx7mwgaKnJ725xKRO2C/AgtQEb1JydtDjLGIsTFzoIvzJAaVl8VgUMigrt4tcDQ1
         U0S0sxDcRviqviccMbgwNqVakCXOwaDVQoVMQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=MPau9kcokw/tM+Gmi/LQCppnDXGmYQZ2IVu1X4ZzEnu55dcxM7aOcrrGaVWL9D1iaA
         oRxuef9qe4sE6KXHgWWtBFMsz6hl90KdrYz8AtG1AJvi6himNWmGWU9hjP0cYuTwQI5W
         FRpxXdJAJWlJZPNkLU6HpW5RhqfMjSml2pviY=
MIME-Version: 1.0
Received: by 10.220.88.209 with SMTP id b17mr5993402vcm.98.1255199928470; Sat, 
	10 Oct 2009 11:38:48 -0700 (PDT)
In-Reply-To: <309f76d00910101048x4028765fg477f1b249e223c8b@mail.gmail.com>
References: <309f76d00910101048x4028765fg477f1b249e223c8b@mail.gmail.com>
Date: Sat, 10 Oct 2009 11:38:48 -0700
Message-ID: <314098690910101138x4586a05bj9ba03ad483040e1@mail.gmail.com>
Subject: Re: SIMPLE QUESTIONS ABOUT DISTRIBUTEDCACHE...
From: Jason Venner <jason.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636284e58dbcc92047599011d
X-Virus-Checked: Checked by ClamAV on apache.org

--001636284e58dbcc92047599011d
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

The distributed cache does not behave as expected in PsuedoDistributed mode=
.

Since all of the tasks run within the same jvm, there is no easy way to
alter the class path to add distributed cache items, in addition there is n=
o
way to change the working directory of the jvm to the 'per task directory'
that the normal task runner creates, where the distributed cache items made
available.

On Sat, Oct 10, 2009 at 10:48 AM, Alberto Luengo Cabanillas <
cabiwan@gmail.com> wrote:

> Hi everyone! I=B4m trying to distribute some common data between my nodes=
,
> but
> I=B4m unable to do it. I have some questions about this:
>
> 1.- I=B4m working in PseudoDistributed mode in Windows (under Cygwin) and=
 my
> DFS server is running on a virtual machine (Ubuntu server). May I use
> DistributedCache normally with this configuration?
> 2.-In the case of the Mapper nodes, in the Driver class I add these two
> lines:
>
> URI uriMapper =3Dnew URI("/user/hadoop-user/data/mapper_configuration.dat=
");
> (I also verify this file already exists in the DFS).
> DistributedCache.addCacheFile(uriMapper,conf);
>
> 3.- In the Mapper class, I have this:
>
> @Override
>    protected void setup(Context cont) {
>       ...
>       String configureCacheName =3D new
> Path("/user/hadoop-user/data/mapper_configuration.dat").getName();
>       Path [] cacheFiles =3D
> DistributedCache.getLocalCacheFiles(cont.getConfiguration());
>       System.out.println("The file is: "+cacheFiles.toString());
>     ...
>    }
>
> Does anyone have an idea about what I=B4m doing wrong? (I=B4m going crazy=
,
> actually...)
>
> Thanks a lot in advance.
>
> --
> Alberto
>



--=20
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=3Djewlerymall
www.prohadoopbook.com a community for Hadoop Professionals

--001636284e58dbcc92047599011d--

From general-return-595-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Oct 11 01:12:32 2009
Return-Path: <general-return-595-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 833 invoked from network); 11 Oct 2009 01:12:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Oct 2009 01:12:32 -0000
Received: (qmail 49304 invoked by uid 500); 11 Oct 2009 01:12:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49252 invoked by uid 500); 11 Oct 2009 01:12:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49242 invoked by uid 99); 11 Oct 2009 01:12:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 11 Oct 2009 01:12:31 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [64.78.17.17] (HELO EXHUB018-2.exch018.msoutlookonline.net) (64.78.17.17)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 11 Oct 2009 01:12:20 +0000
Received: from EXVMBX018-1.exch018.msoutlookonline.net ([64.78.17.47]) by
 EXHUB018-2.exch018.msoutlookonline.net ([64.78.17.17]) with mapi; Sat, 10 Oct
 2009 18:11:38 -0700
From: Scott Carey <scott@richrelevance.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Sat, 10 Oct 2009 18:11:20 -0700
Subject: Re: HTTP transport?
Thread-Topic: HTTP transport?
Thread-Index: AcpJCQCEHXFVafpVQXaJtrNlnq7atABBr0cn
Message-ID: <C6F67EC8.13E59%scott@richrelevance.com>
In-Reply-To: <4ACF77C3.6020809@apache.org>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


On 10/9/09 10:49 AM, "Doug Cutting" <cutting@apache.org> wrote:

>=20
>> It is an interesting question how much we
>> depend on being able to answer queries out of order. There are some
>> parts of the code where overlapping requests from the same client
>> matter. In particular, the terasort scheduler uses threads to access the
>> namenode. That would stop providing any pipelining, which I believe
>> would be significant.
>=20
> No, we wouldn't stop any pipelining, we'd just use more connections to
> implement it.  With HttpClient one can limit the number of pooled
> connnections per host:
>=20

Also since HTTP supports in-order pipelining out of the box, its only
out-of-order stuff that would require additional connections.

>=20
> Doug
>=20

Requirements may end up ruling out HTTP, but I doubt that performance (in
the insecure case) will be the cause since there are so many high
performance client and server implementations available.
Consider something lower level than the Servlet API for the server side --
it is baggage-laden and does not allow access to all data in unconverted
form or any asynchronous i/o.

In this respect, jetty has lower level, light-weight API access points.
http://docs.codehaus.org/display/JETTY/Architecture

If HTTP is not used, I suggest a strong look at apache MINA for constructin=
g
high performance NIO clients and servers with Java http://mina.apache.org/


From general-return-596-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 12 01:53:25 2009
Return-Path: <general-return-596-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33198 invoked from network); 12 Oct 2009 01:53:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Oct 2009 01:53:25 -0000
Received: (qmail 65404 invoked by uid 500); 12 Oct 2009 01:53:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65318 invoked by uid 500); 12 Oct 2009 01:53:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65308 invoked by uid 99); 12 Oct 2009 01:53:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Oct 2009 01:53:24 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [128.200.80.34] (HELO smtp1.es.uci.edu) (128.200.80.34)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Oct 2009 01:53:22 +0000
Received: from [192.168.1.104] (ip72-211-211-194.oc.oc.cox.net [72.211.211.194])
	(authenticated bits=0)
	by smtp1.es.uci.edu (8.13.1/8.13.1) with ESMTP id n9C1qxYL022343
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT)
	for <general@hadoop.apache.org>; Sun, 11 Oct 2009 18:53:00 -0700
X-UCInetID: vayyalas
Message-ID: <4AD26FD1.8080905@uci.edu>
Date: Sun, 11 Oct 2009 18:52:49 -0500
From: Vandana Ayyalasomayajula <vayyalas@uci.edu>
User-Agent: Thunderbird 2.0.0.23 (Windows/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Simulators for Hadoop ??
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi All,

I am new to Hadoop and planning to do a project which helps me learn 
technologies like Pig and Hive. The problem is I do not have access to 
any Hadoop like cluster. I was wondering if there are any simulators in 
which I can specify the node configurations and run it on my machine. 
Please help me solve this issue.

Thanks
Vandana

From general-return-597-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 12 04:14:19 2009
Return-Path: <general-return-597-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 62329 invoked from network); 12 Oct 2009 04:14:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Oct 2009 04:14:19 -0000
Received: (qmail 7812 invoked by uid 500); 12 Oct 2009 04:14:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7620 invoked by uid 500); 12 Oct 2009 04:14:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7610 invoked by uid 99); 12 Oct 2009 04:14:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Oct 2009 04:14:18 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [203.99.254.144] (HELO rsmtp2.corp.hki.yahoo.com) (203.99.254.144)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Oct 2009 04:14:07 +0000
Received: from builtvalue-dr.eglbp.corp.yahoo.com (builtvalue-dr.eglbp.corp.yahoo.com [10.66.77.165])
	by rsmtp2.corp.hki.yahoo.com (8.13.8/8.13.8/y.rout) with ESMTP id n9C4Dcmc063370
	for <general@hadoop.apache.org>; Sun, 11 Oct 2009 21:13:38 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:date:from:user-agent:mime-version:to:subject:
	references:in-reply-to:content-type:content-transfer-encoding;
	b=Z7fOUbMxA2wuirxa7UTkBIYVdrPegiWMEUc7Ut528OgERPIBy8fbXUJttlfdyeid
Message-ID: <4AD2ACF2.6000309@yahoo-inc.com>
Date: Mon, 12 Oct 2009 09:43:38 +0530
From: Sharad Agarwal <sharadag@yahoo-inc.com>
User-Agent: Thunderbird 2.0.0.23 (X11/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Simulators for Hadoop ??
References: <4AD26FD1.8080905@uci.edu>
In-Reply-To: <4AD26FD1.8080905@uci.edu>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hadoop can be run on single node. See 
http://hadoop.apache.org/common/docs/current/quickstart.html#PseudoDistributed

Sharad

From general-return-598-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 12 09:16:16 2009
Return-Path: <general-return-598-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30087 invoked from network); 12 Oct 2009 09:16:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Oct 2009 09:16:16 -0000
Received: (qmail 48758 invoked by uid 500); 12 Oct 2009 09:16:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48678 invoked by uid 500); 12 Oct 2009 09:16:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48668 invoked by uid 99); 12 Oct 2009 09:16:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Oct 2009 09:16:15 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.85.217.226] (HELO mail-gx0-f226.google.com) (209.85.217.226)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Oct 2009 09:16:13 +0000
Received: by gxk26 with SMTP id 26so2556089gxk.11
        for <general@hadoop.apache.org>; Mon, 12 Oct 2009 02:15:52 -0700 (PDT)
Received: by 10.101.34.10 with SMTP id m10mr5206505anj.32.1255338951945;
        Mon, 12 Oct 2009 02:15:51 -0700 (PDT)
Received: from ?192.168.0.100? (c-24-23-193-72.hsd1.ca.comcast.net [24.23.193.72])
        by mx.google.com with ESMTPS id 21sm2213017yxe.1.2009.10.12.02.15.49
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Mon, 12 Oct 2009 02:15:50 -0700 (PDT)
Message-ID: <4AD2F3C5.5010808@cloudera.com>
Date: Mon, 12 Oct 2009 02:15:49 -0700
From: Amr Awadallah <aaa@cloudera.com>
User-Agent: Thunderbird 2.0.0.23 (Windows/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Simulators for Hadoop ??
References: <4AD26FD1.8080905@uci.edu>
In-Reply-To: <4AD26FD1.8080905@uci.edu>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Vadana,

  This VMware virtual machine has a pre-installed hadoop in local mode 
with both pig and hive:

http://www.cloudera.com/hadoop-training-virtual-machine

Good luck with your project,

-- amr

> Hi All,
>
> I am new to Hadoop and planning to do a project which helps me learn 
> technologies like Pig and Hive. The problem is I do not have access to 
> any Hadoop like cluster. I was wondering if there are any simulators 
> in which I can specify the node configurations and run it on my 
> machine. Please help me solve this issue.
>
> Thanks
> Vandana


From general-return-599-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 12 11:35:37 2009
Return-Path: <general-return-599-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59487 invoked from network); 12 Oct 2009 11:35:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Oct 2009 11:35:37 -0000
Received: (qmail 4343 invoked by uid 500); 12 Oct 2009 11:35:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4263 invoked by uid 500); 12 Oct 2009 11:35:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4253 invoked by uid 99); 12 Oct 2009 11:35:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Oct 2009 11:35:36 +0000
X-ASF-Spam-Status: No, hits=0.0 required=5.0
	tests=BAYES_50
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of sawerset@i.ua designates 91.198.36.2 as permitted sender)
Received: from [91.198.36.2] (HELO web01.mi6.kiev.ua) (91.198.36.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Oct 2009 11:35:29 +0000
Received: from st05.mi6 ([10.0.0.29] helo=st05.mi6.kiev.ua)
	by web01.mi6.kiev.ua with esmtp (Exim 4.69)
	(envelope-from <sawerset@i.ua>)
	id 1MxJB9-0007R2-SJ
	for general@hadoop.apache.org; Mon, 12 Oct 2009 14:35:05 +0300
Received: from web by st05.mi6.kiev.ua with local (Exim 4.69)
	(envelope-from <sawerset@i.ua>)
	id 1MxJB1-0005FT-F2
	for general@hadoop.apache.org; Mon, 12 Oct 2009 14:34:55 +0300
To: general@hadoop.apache.org
Subject: =?windows-1251?B?SG93IHRvIHJlYWQgc2V0dGluZ3MgaGFkb29wIGNsdXN0ZXIgZnJvbSBKYXZhIHByb2dyYW0/?=
Date: Mon, 12 Oct 2009 14:34:55 +0300
From: =?windows-1251?B?yOLg7SDC4PHo6/zl4g==?= <sawerset@i.ua>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; CHARSET=windows-1251
Content-Transfer-Encoding: QUOTED-PRINTABLE
X-User-Agent: Mozilla/5.0 (X11; U; Linux i686; ru; rv:1.9.0.14) Gecko/2009090216 Ubuntu/9.04 (jaunty) Firefox/3.0.14
X-Sender-IP: 89.28.203.82 (192.168.242.135)
X-Mailer: I.UA Mail System
Message-Id: <E1MxJB1-0005FT-F2@st05.mi6.kiev.ua>

How to read settings hadoop cluster from Java program?
Is there any sdk for such things?

-- =F0=E5=EA=EB=E0=EC=E0 --------------------------------------------------=
---------
=CB=F3=F7=F8=E8=E9 =F5=EE=F1=F2=E8=ED=E3 =EE=F2 $3.45
=C4=EE=EC=E5=ED =E2 =EF=EE=E4=E0=F0=EE=EA - www.hostpro.ua


From general-return-600-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 12 15:43:38 2009
Return-Path: <general-return-600-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54840 invoked from network); 12 Oct 2009 15:43:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Oct 2009 15:43:38 -0000
Received: (qmail 48746 invoked by uid 500); 12 Oct 2009 15:43:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48664 invoked by uid 500); 12 Oct 2009 15:43:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48654 invoked by uid 99); 12 Oct 2009 15:43:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Oct 2009 15:43:36 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jason.hadoop@gmail.com designates 209.85.212.186 as permitted sender)
Received: from [209.85.212.186] (HELO mail-vw0-f186.google.com) (209.85.212.186)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Oct 2009 15:43:26 +0000
Received: by vws16 with SMTP id 16so6096154vws.2
        for <general@hadoop.apache.org>; Mon, 12 Oct 2009 08:43:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=x37C4FcUjksUBDWSArTCnZuLUbmA9u2P4hiDOWtKXn8=;
        b=OV8PcTMfKBz7EIt5Eli9VFQiH3cZ41qOqmMvKIkt/SBwkhZfj4fy4p7HuNhPJICSHD
         yeC1IhH3qgQgctaoQ7yrAPd4aEGdVgQaz7UwUcvbe0SGqzQPafF528Is67I7cr30U6uL
         S4/bZus0e3OdrZi8XbPxKQrvzyjTszHiqHcjU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Zx5fkKdBCnwGWxuctJGtfxIsRe5zfefhZL6dgoAy2vnT8usz7y/Camdkyy5KWazHn1
         Yd1B1Qu43gNd/6d4yoAdssX7fjkbU2d6Gq061mfe5bqTzhAoNswCYQtTpCGOrUs5jIty
         xpHpZHIF24qHx2e5lzSFeSKfr95d6nRC1RvRU=
MIME-Version: 1.0
Received: by 10.220.15.68 with SMTP id j4mr8600183vca.18.1255362185310; Mon, 
	12 Oct 2009 08:43:05 -0700 (PDT)
In-Reply-To: <E1MxJB1-0005FT-F2@st05.mi6.kiev.ua>
References: <E1MxJB1-0005FT-F2@st05.mi6.kiev.ua>
Date: Mon, 12 Oct 2009 08:43:04 -0700
Message-ID: <314098690910120843j36dc05b7kb13fb42a1e633211@mail.gmail.com>
Subject: Re: How to read settings hadoop cluster from Java program?
From: Jason Venner <jason.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485e76f121edc5b0475bec96e
X-Virus-Checked: Checked by ClamAV on apache.org

--001485e76f121edc5b0475bec96e
Content-Type: text/plain; charset=KOI8-R
Content-Transfer-Encoding: quoted-printable

You may use the getClusterStatus method of the JobClient object.

2009/10/12 =E9=D7=C1=CE =F7=C1=D3=C9=CC=D8=C5=D7 <sawerset@i.ua>

> How to read settings hadoop cluster from Java program?
> Is there any sdk for such things?
>
> -- =D2=C5=CB=CC=C1=CD=C1 ------------------------------------------------=
-----------
> =EC=D5=DE=DB=C9=CA =C8=CF=D3=D4=C9=CE=C7 =CF=D4 $3.45
> =E4=CF=CD=C5=CE =D7 =D0=CF=C4=C1=D2=CF=CB - www.hostpro.ua
>
>


--=20
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=3Djewlerymall
www.prohadoopbook.com a community for Hadoop Professionals

--001485e76f121edc5b0475bec96e--

From general-return-601-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 12 23:51:17 2009
Return-Path: <general-return-601-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 10684 invoked from network); 12 Oct 2009 23:51:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Oct 2009 23:51:17 -0000
Received: (qmail 74604 invoked by uid 500); 12 Oct 2009 23:51:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74527 invoked by uid 500); 12 Oct 2009 23:51:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74517 invoked by uid 99); 12 Oct 2009 23:51:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Oct 2009 23:51:15 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 74.125.92.24 as permitted sender)
Received: from [74.125.92.24] (HELO qw-out-2122.google.com) (74.125.92.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Oct 2009 23:51:03 +0000
Received: by qw-out-2122.google.com with SMTP id 9so652861qwb.35
        for <general@hadoop.apache.org>; Mon, 12 Oct 2009 16:50:43 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:message-id:subject:from:to
         :content-type;
        bh=cZzOVwsoYICGTeXfgOZbYpV/uMC3OvmkVXyRm6hretw=;
        b=Fp0HqzIjqAX7P965YSB3R7s9m6wktifDlg7Sha2x+cNET/OdGonKDx0xgG3DyWlZiF
         TLBLReh0+UaQSDtCw7l+qyvpavzKT7tQ91vFmB2Jyls6U6TyY0HbE0h9iE5OBkkjET1G
         JwI3KwuZ1xaJ0ufQpBfPe5+uOMqma6pYvMQFU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=ExJUP9ZuwjtPX9ugXOypbbbLt6rHKTo7viJqngUyDIo9EgkS7gILlu1ueCFB2NKcEw
         2+l+vLSVGWzDmQ4zUm/T8YqAghTBlGo46WUpugrGH3AplNIuwcZM9tOp25nurKjaHD/k
         Kp/csu0QrGCa1sn5B+9IVBEfNqTM3hlF9/MnM=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
Received: by 10.229.51.219 with SMTP id e27mr2548695qcg.7.1255391442942; Mon, 
	12 Oct 2009 16:50:42 -0700 (PDT)
In-Reply-To: <7c962aed0910121650g7c6f5c11ja60ec614136d7f78@mail.gmail.com>
References: <7c962aed0910121650g7c6f5c11ja60ec614136d7f78@mail.gmail.com>
Date: Mon, 12 Oct 2009 16:50:42 -0700
X-Google-Sender-Auth: 08831a0c14ffa4e1
Message-ID: <7c962aed0910121650r255d2e92m41919d6bc4a37c88@mail.gmail.com>
Subject: ANN HBase 0.20.1 is available for download
From: stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016367d595a02bda30475c5991d
X-Virus-Checked: Checked by ClamAV on apache.org

--0016367d595a02bda30475c5991d
Content-Type: text/plain; charset=ISO-8859-1

HBase 0.20.1 is available for download: http://hbase.org/releases.html.
Sixty issues have been addressed since hbase 0.20.0.  The release notes are
available here: http://su.pr/1ypRmy.  Please upgrade at your convenience.

Thanks to all who contributed to this release,

Yours,
The HBase Team

--0016367d595a02bda30475c5991d--

From general-return-602-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 13 12:57:18 2009
Return-Path: <general-return-602-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78662 invoked from network); 13 Oct 2009 12:57:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Oct 2009 12:57:18 -0000
Received: (qmail 44342 invoked by uid 500); 13 Oct 2009 12:57:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44275 invoked by uid 500); 13 Oct 2009 12:57:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44265 invoked by uid 99); 13 Oct 2009 12:57:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Oct 2009 12:57:14 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [72.14.220.157] (HELO fg-out-1718.google.com) (72.14.220.157)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Oct 2009 12:57:11 +0000
Received: by fg-out-1718.google.com with SMTP id d23so921688fga.11
        for <general@hadoop.apache.org>; Tue, 13 Oct 2009 05:56:47 -0700 (PDT)
Received: by 10.86.170.22 with SMTP id s22mr826640fge.37.1255438606455;
        Tue, 13 Oct 2009 05:56:46 -0700 (PDT)
Received: from nsf264fa356e37 ([202.51.92.14])
        by mx.google.com with ESMTPS id e20sm2152965fga.15.2009.10.13.05.56.43
        (version=SSLv3 cipher=RC4-MD5);
        Tue, 13 Oct 2009 05:56:45 -0700 (PDT)
From: "Chandan Tamrakar" <chandan.tamrakar@nepasoft.com>
To: <general@hadoop.apache.org>
References: <E1MxJB1-0005FT-F2@st05.mi6.kiev.ua> <314098690910120843j36dc05b7kb13fb42a1e633211@mail.gmail.com>
In-Reply-To: <314098690910120843j36dc05b7kb13fb42a1e633211@mail.gmail.com>
Subject: Reading files from local file system
Date: Tue, 13 Oct 2009 18:41:42 +0545
Message-ID: <4ad4790d.1438560a.1db9.3612@mx.google.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Mailer: Microsoft Office Outlook 12.0
Thread-Index: AcpLUsSGQFOTfocFSCK/dfxCPVUfeAAiAbkA
Content-Language: en-us
x-cr-puzzleid: {2767A896-8A0C-4EF9-A45A-E2007A1BA218}
x-cr-hashedpuzzle: BYc6 Btlb CAvR Ch7a C/cn DF3b Dyji D7lT EWfM Epol EwX8 ExS/ GxvE HRaa I4qY Jkd0;1;ZwBlAG4AZQByAGEAbABAAGgAYQBkAG8AbwBwAC4AYQBwAGEAYwBoAGUALgBvAHIAZwA=;Sosha1_v1;7;{2767A896-8A0C-4EF9-A45A-E2007A1BA218};YwBoAGEAbgBkAGEAbgAuAHQAYQBtAHIAYQBrAGEAcgBAAG4AZQBwAGEAcwBvAGYAdAAuAGMAbwBtAA==;Tue, 13 Oct 2009 12:56:39 GMT;UgBlAGEAZABpAG4AZwAgAGYAaQBsAGUAcwAgAGYAcgBvAG0AIABsAG8AYwBhAGwAIABmAGkAbABlACAAcwB5AHMAdABlAG0A



We are trying to read files from local file system. But when running the map
reduce it is not able to read files from the input location (the input
location is also local file system location).

For this we changed the configuration of the hadoop-site.xml as shown below:

/etc/conf/hadoop/hadoop-site.xml

<property>
    <name>fs.default.name</name>
    <value>file:///</value>
  </property>


 [admin@localhost ~]$ hadoop jar Test.jar /home/admin/input/test.txt output1

Suppose Test.txt is pain text file that contains
Test1
Test2
Test3


While running simple MapReduce job we get following exception  "File not
found exception " , we are using TextInputFormat in our Job configuration 


09/10/13 17:26:35 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
09/10/13 17:26:35 INFO mapred.FileInputFormat: Total input paths to process
: 1
09/10/13 17:26:35 INFO mapred.FileInputFormat: Total input paths to process
: 1
09/10/13 17:26:37 INFO mapred.JobClient: Running job: job_200910131447_0033
09/10/13 17:26:38 INFO mapred.JobClient:  map 0% reduce 0%
09/10/13 17:27:00 INFO mapred.JobClient: Task Id :
attempt_200910131447_0033_m_000000_0, Status : FAILED
java.io.FileNotFoundException: File file:/home/admin/Desktop/input/test.txt
does not exist.
	at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.jav
a:420)
	at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:25
9)
	at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(Checks
umFileSystem.java:117)
	at
org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:275)
	at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:364)
	at
org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:206)
	at
org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.jav
a:50)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:219)
	at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2210)

However, running in the code as a separate Main method does work well.

public static void main (String [] args) throws IOException {

     Configuration conf = new Configuration();
     FileSystem fs = FileSystem.get(conf);

     Path filenamePath = new Path(theFilename);
     FSDataOutputStream out = fs.create(new Path("abc.txt"));
     out.writeUTF("abc");
     out.close();

}

The above code works fine when running it as a jar in hadoop. The above code
successfully creates file in /home/admin/abc.txt when running from admin
user.


From general-return-603-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 13 13:50:00 2009
Return-Path: <general-return-603-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96595 invoked from network); 13 Oct 2009 13:50:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Oct 2009 13:50:00 -0000
Received: (qmail 58129 invoked by uid 500); 13 Oct 2009 13:49:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58088 invoked by uid 500); 13 Oct 2009 13:49:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58078 invoked by uid 99); 13 Oct 2009 13:49:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Oct 2009 13:49:59 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of zjffdu@gmail.com designates 209.85.222.177 as permitted sender)
Received: from [209.85.222.177] (HELO mail-pz0-f177.google.com) (209.85.222.177)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Oct 2009 13:49:56 +0000
Received: by pzk7 with SMTP id 7so1301976pzk.30
        for <general@hadoop.apache.org>; Tue, 13 Oct 2009 06:49:36 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=WC0hhpP1DSe1OtIveAJXT6U9TO1AzmveLadEn/I22C8=;
        b=IkB3PU/bt2Wjl+o4cAlssA4BkfeGmbm6q4+w/h234TDQ4h65+qpTtjPuHHRaZe6lVl
         luycin7MZLtgcS+rLVXSNK4a0D6vvBCIb52sSE+/IYHwhYeGAHl5VS4NXWO9XC7e3j1Z
         qiPJGYln16zGvQInI2GD+uUBB/M/3Cyz+VIfg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=sJ7DCupd7oox9sL58vJH07uy9zPGiixy6Fai1ti0op5x+LfxGrDO0mGzUvVJ4Sg77A
         eSgr5K6lre8li0ZSbEddIk/iyoHGYLi7Ae9lpwPq1IaEOvlpnvRaGXmhZdHMMBPvJTnL
         3wg8H2/n/yhfN05oswHIILZKgasXyhMjWo1bA=
MIME-Version: 1.0
Received: by 10.142.55.14 with SMTP id d14mr76107wfa.270.1255441776107; Tue, 
	13 Oct 2009 06:49:36 -0700 (PDT)
In-Reply-To: <4ad4790d.1438560a.1db9.3612@mx.google.com>
References: <E1MxJB1-0005FT-F2@st05.mi6.kiev.ua>
	 <314098690910120843j36dc05b7kb13fb42a1e633211@mail.gmail.com>
	 <4ad4790d.1438560a.1db9.3612@mx.google.com>
Date: Tue, 13 Oct 2009 06:49:36 -0700
Message-ID: <8211a1320910130649y14b06a67s13b268f31f6f786e@mail.gmail.com>
Subject: Re: Reading files from local file system
From: Jeff Zhang <zjffdu@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636b2bc9919e6da0475d15176

--001636b2bc9919e6da0475d15176
Content-Type: text/plain; charset=UTF-8

Maybe you could debug your mapreduce job in eclipse, since you run it in
local mode.



On Tue, Oct 13, 2009 at 5:56 AM, Chandan Tamrakar <
chandan.tamrakar@nepasoft.com> wrote:

>
>
> We are trying to read files from local file system. But when running the
> map
> reduce it is not able to read files from the input location (the input
> location is also local file system location).
>
> For this we changed the configuration of the hadoop-site.xml as shown
> below:
>
> /etc/conf/hadoop/hadoop-site.xml
>
> <property>
>    <name>fs.default.name</name>
>    <value>file:///</value>
>  </property>
>
>
>  [admin@localhost ~]$ hadoop jar Test.jar /home/admin/input/test.txt
> output1
>
> Suppose Test.txt is pain text file that contains
> Test1
> Test2
> Test3
>
>
> While running simple MapReduce job we get following exception  "File not
> found exception " , we are using TextInputFormat in our Job configuration
>
>
> 09/10/13 17:26:35 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
> 09/10/13 17:26:35 INFO mapred.FileInputFormat: Total input paths to process
> : 1
> 09/10/13 17:26:35 INFO mapred.FileInputFormat: Total input paths to process
> : 1
> 09/10/13 17:26:37 INFO mapred.JobClient: Running job: job_200910131447_0033
> 09/10/13 17:26:38 INFO mapred.JobClient:  map 0% reduce 0%
> 09/10/13 17:27:00 INFO mapred.JobClient: Task Id :
> attempt_200910131447_0033_m_000000_0, Status : FAILED
> java.io.FileNotFoundException: File file:/home/admin/Desktop/input/test.txt
> does not exist.
>        at
>
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.jav
> a:420)
>        at
>
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:25
> 9)
>        at
>
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(Checks
> umFileSystem.java:117)
>        at
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:275)
>        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:364)
>        at
> org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:206)
>        at
>
> org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.jav
> a:50)
>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:219)
>        at
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2210)
>
> However, running in the code as a separate Main method does work well.
>
> public static void main (String [] args) throws IOException {
>
>     Configuration conf = new Configuration();
>     FileSystem fs = FileSystem.get(conf);
>
>     Path filenamePath = new Path(theFilename);
>     FSDataOutputStream out = fs.create(new Path("abc.txt"));
>     out.writeUTF("abc");
>     out.close();
>
> }
>
> The above code works fine when running it as a jar in hadoop. The above
> code
> successfully creates file in /home/admin/abc.txt when running from admin
> user.
>
>

--001636b2bc9919e6da0475d15176--

From general-return-604-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 13 14:17:43 2009
Return-Path: <general-return-604-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5579 invoked from network); 13 Oct 2009 14:17:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Oct 2009 14:17:42 -0000
Received: (qmail 19131 invoked by uid 500); 13 Oct 2009 14:17:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19044 invoked by uid 500); 13 Oct 2009 14:17:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19024 invoked by uid 99); 13 Oct 2009 14:17:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Oct 2009 14:17:39 +0000
X-ASF-Spam-Status: No, hits=-4.6 required=5.0
	tests=AWL,BAYES_00,RCVD_IN_DNSWL_MED
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of tej2@umbc.edu designates 130.85.25.80 as permitted sender)
Received: from [130.85.25.80] (HELO mx5.umbc.edu) (130.85.25.80)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Oct 2009 14:17:36 +0000
Received: from smtp.umbc.edu (localhost [127.0.0.1])
	by umbc.edu (mx5.umbc.edu) with ESMTP id n9DEHFgD022078;
	Tue, 13 Oct 2009 10:17:15 -0400 (EDT)
Received: from [192.168.1.45] (pool-71-179-71-29.bltmmd.east.verizon.net [71.179.71.29])
	(authenticated bits=0)
	by smtp.umbc.edu (mx5-relay.umbc.edu) with ESMTP id n9DEHEJZ022075
	(version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO);
	Tue, 13 Oct 2009 10:17:14 -0400 (EDT)
From: Tejas Lagvankar <tej2@umbc.edu>
Content-Type: multipart/mixed; boundary=Apple-Mail-5-110843886
Subject: 0.20.1 Cluster Setup Problem
Date: Tue, 13 Oct 2009 10:17:14 -0400
Message-Id: <8E86E890-7D86-4CB0-A525-5CF056265717@umbc.edu>
To: general@hadoop.apache.org, common-user@hadoop.apache.org
Mime-Version: 1.0 (Apple Message framework v1076)
X-Mailer: Apple Mail (2.1076)
X-Milter-Key: 1255573035:2b40ae8e7c9d421eeaa39f89888cdbaf
X-ClamAV: OK

--Apple-Mail-5-110843886
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
	charset=us-ascii;
	format=flowed;
	delsp=yes

Hi,


We are trying to set up a cluster (starting with 2 machines) using the  
new 0.20.1 version.

On the master machine, just after the server starts, the name node  
dies off with the following exception:

2009-10-13 01:22:24,740 ERROR  
org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException:  
Incomplete HDFS URI, no host: hdfs://master_hadoop
         at org.apache.hadoop.hdfs.DistributedFileSystem.initialize 
(DistributedFileSystem.java:78)
         at org.apache.hadoop.fs.FileSystem.createFileSystem 
(FileSystem.java:1373)
         at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java: 
66)
         at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java: 
1385)
         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
         at org.apache.hadoop.fs.Trash.<init>(Trash.java:62)
         at  
org.apache.hadoop.hdfs.server.namenode.NameNode.startTrashEmptier 
(NameNode.java:208)
         at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize 
(NameNode.java:204)
         at org.apache.hadoop.hdfs.server.namenode.NameNode.<init> 
(NameNode.java:279)
         at  
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode 
(NameNode.java:956)
         at org.apache.hadoop.hdfs.server.namenode.NameNode.main 
(NameNode.java:965)

Can anyone help ?  Also can anyone send across example configuration  
files for 0.20.1 if they are different than we are using ?

The detail log file is attached along with.


--Apple-Mail-5-110843886
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
	charset=us-ascii;
	format=flowed



The configuration files are as follows:

MASTER CONFIG
------ conf/masters -------
master_hadoop

------ conf/slaves -------
master_hadoop
slave_hadoop

------ core-site.xml -------
<configuration>

<property>
   <name>fs.default.name</name>
   <value>hdfs://master_hadoop</value>
</property>

<property>
   <name>hadoop.tmp.dir</name>
   <value>/opt/hadoop-0.20.1/tmp</value>
</property>

------ hdfs-site.xml -------
<property>
   <name>dfs.replication</name>
   <value>2</value>
</property>


------ mapred-site.xml -------
<property>
   <name>mapred.job.tracker</name>
   <value>tejas_hadoop:9001</value>
</property>





SLAVE CONFIG
------ core-site.xml -------
<property>
   <name>hadoop.tmp.dir</name>
   <value>/opt/hadoop-0.20.1/tmp/</value>
</property>


<property>
   <name>fs.default.name</name>
   <value>hdfs://master_hadoop</value>
</property>


------ hdfs-site.xml -------
<property>
   <name>dfs.replication</name>
   <value>2</value>
</property>

------ mapred-site.xml -------
<property>
   <name>mapred.job.tracker</name>
   <value>tejas_hadoop:9001</value>
</property>



Regards,

Tejas Lagvankar
meettejas@umbc.edu
www.umbc.edu/~tej2




--Apple-Mail-5-110843886--

From general-return-605-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 13 15:44:32 2009
Return-Path: <general-return-605-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39267 invoked from network); 13 Oct 2009 15:44:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Oct 2009 15:44:32 -0000
Received: (qmail 6627 invoked by uid 500); 13 Oct 2009 15:44:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6043 invoked by uid 500); 13 Oct 2009 15:44:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5919 invoked by uid 99); 13 Oct 2009 15:44:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Oct 2009 15:44:28 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [209.85.223.171] (HELO mail-iw0-f171.google.com) (209.85.223.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Oct 2009 15:44:17 +0000
Received: by iwn1 with SMTP id 1so5762210iwn.2
        for <multiple recipients>; Tue, 13 Oct 2009 08:43:52 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.122.103 with SMTP id k39mr2711006ibr.10.1255448631196; 
	Tue, 13 Oct 2009 08:43:51 -0700 (PDT)
In-Reply-To: <8E86E890-7D86-4CB0-A525-5CF056265717@umbc.edu>
References: <8E86E890-7D86-4CB0-A525-5CF056265717@umbc.edu>
Date: Tue, 13 Oct 2009 15:43:51 +0000
Message-ID: <6e8dca540910130843k1743c85fmeb975a4670dfc6fc@mail.gmail.com>
Subject: Re: 0.20.1 Cluster Setup Problem
From: Kevin Sweeney <kevin@yieldex.com>
To: general@hadoop.apache.org
Cc: common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6460848b23c470475d2e994
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6460848b23c470475d2e994
Content-Type: text/plain; charset=ISO-8859-1

Hi Tejas,
I just upgraded to 20.1 as well and you config all looks the same as mine
except in the core-site.xml I have:

<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>

Maybe you need to add the port on yours. I haven't seen that error before,
but it seems to be suggesting it can't resolve the host. I'd say
double-check your names and that they resolve.

Hope that helps,
Kevin

On Tue, Oct 13, 2009 at 2:17 PM, Tejas Lagvankar <tej2@umbc.edu> wrote:

> Hi,
>
>
> We are trying to set up a cluster (starting with 2 machines) using the new
> 0.20.1 version.
>
> On the master machine, just after the server starts, the name node dies off
> with the following exception:
>
> 2009-10-13 01:22:24,740 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException:
> Incomplete HDFS URI, no host: hdfs://master_hadoop
>        at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:78)
>        at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1373)
>        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
>        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1385)
>        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
>        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
>        at org.apache.hadoop.fs.Trash.<init>(Trash.java:62)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startTrashEmptier(NameNode.java:208)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:204)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
>
> Can anyone help ?  Also can anyone send across example configuration files
> for 0.20.1 if they are different than we are using ?
>
> The detail log file is attached along with.
>
>
>
>
> The configuration files are as follows:
>
> MASTER CONFIG
> ------ conf/masters -------
> master_hadoop
>
> ------ conf/slaves -------
> master_hadoop
> slave_hadoop
>
> ------ core-site.xml -------
> <configuration>
>
> <property>
>  <name>fs.default.name</name>
>  <value>hdfs://master_hadoop</value>
> </property>
>
> <property>
>  <name>hadoop.tmp.dir</name>
>  <value>/opt/hadoop-0.20.1/tmp</value>
> </property>
>
> ------ hdfs-site.xml -------
> <property>
>  <name>dfs.replication</name>
>  <value>2</value>
> </property>
>
>
> ------ mapred-site.xml -------
> <property>
>  <name>mapred.job.tracker</name>
>  <value>tejas_hadoop:9001</value>
> </property>
>
>
>
>
>
> SLAVE CONFIG
> ------ core-site.xml -------
> <property>
>  <name>hadoop.tmp.dir</name>
>  <value>/opt/hadoop-0.20.1/tmp/</value>
> </property>
>
>
> <property>
>  <name>fs.default.name</name>
>  <value>hdfs://master_hadoop</value>
> </property>
>
>
> ------ hdfs-site.xml -------
> <property>
>  <name>dfs.replication</name>
>  <value>2</value>
> </property>
>
> ------ mapred-site.xml -------
> <property>
>  <name>mapred.job.tracker</name>
>  <value>tejas_hadoop:9001</value>
> </property>
>
>
>
> Regards,
>
> Tejas Lagvankar
> meettejas@umbc.edu
> www.umbc.edu/~tej2
>
>
>
>
>

--0016e6460848b23c470475d2e994--

From general-return-606-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 13 16:06:26 2009
Return-Path: <general-return-606-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46418 invoked from network); 13 Oct 2009 16:06:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Oct 2009 16:06:26 -0000
Received: (qmail 50810 invoked by uid 500); 13 Oct 2009 16:06:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50741 invoked by uid 500); 13 Oct 2009 16:06:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50731 invoked by uid 99); 13 Oct 2009 16:06:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Oct 2009 16:06:25 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.211.187] (HELO mail-yw0-f187.google.com) (209.85.211.187)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Oct 2009 16:06:22 +0000
Received: by ywh17 with SMTP id 17so6516455ywh.2
        for <general@hadoop.apache.org>; Tue, 13 Oct 2009 09:05:59 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.90.42.27 with SMTP id p27mr3920380agp.9.1255449958131; Tue, 13 
	Oct 2009 09:05:58 -0700 (PDT)
In-Reply-To: <8211a1320910130649y14b06a67s13b268f31f6f786e@mail.gmail.com>
References: <E1MxJB1-0005FT-F2@st05.mi6.kiev.ua> <314098690910120843j36dc05b7kb13fb42a1e633211@mail.gmail.com> 
	<4ad4790d.1438560a.1db9.3612@mx.google.com> <8211a1320910130649y14b06a67s13b268f31f6f786e@mail.gmail.com>
From: Chandan Tamrakar <chandan.tamrakar@nepasoft.com>
Date: Tue, 13 Oct 2009 09:05:38 -0700
Message-ID: <95f8b1a90910130905h38357b3fmd53c5a4b1c9ee5f8@mail.gmail.com>
Subject: Re: Reading files from local file system
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636163e07c9a9390475d3389e

--001636163e07c9a9390475d3389e
Content-Type: text/plain; charset=ISO-8859-1

Do I need to change any configuration beside changing the default file
system to "local file system' ?
I am trying to input for example  input.txt to map job

input.txt will contain file location as following

file://path/abc1.doc
file://path/abc2.doc
..
...

map program will read each line from input.txt and process them

Do i need to change any configuration ? This is similar to how Nutch crawls
.

any feedbacks would be appreciated

thanks



On Tue, Oct 13, 2009 at 6:49 AM, Jeff Zhang <zjffdu@gmail.com> wrote:

> Maybe you could debug your mapreduce job in eclipse, since you run it in
> local mode.
>
>
>
> On Tue, Oct 13, 2009 at 5:56 AM, Chandan Tamrakar <
> chandan.tamrakar@nepasoft.com> wrote:
>
> >
> >
> > We are trying to read files from local file system. But when running the
> > map
> > reduce it is not able to read files from the input location (the input
> > location is also local file system location).
> >
> > For this we changed the configuration of the hadoop-site.xml as shown
> > below:
> >
> > /etc/conf/hadoop/hadoop-site.xml
> >
> > <property>
> >    <name>fs.default.name</name>
> >    <value>file:///</value>
> >  </property>
> >
> >
> >  [admin@localhost ~]$ hadoop jar Test.jar /home/admin/input/test.txt
> > output1
> >
> > Suppose Test.txt is pain text file that contains
> > Test1
> > Test2
> > Test3
> >
> >
> > While running simple MapReduce job we get following exception  "File not
> > found exception " , we are using TextInputFormat in our Job configuration
> >
> >
> > 09/10/13 17:26:35 WARN mapred.JobClient: Use GenericOptionsParser for
> > parsing the arguments. Applications should implement Tool for the same.
> > 09/10/13 17:26:35 INFO mapred.FileInputFormat: Total input paths to
> process
> > : 1
> > 09/10/13 17:26:35 INFO mapred.FileInputFormat: Total input paths to
> process
> > : 1
> > 09/10/13 17:26:37 INFO mapred.JobClient: Running job:
> job_200910131447_0033
> > 09/10/13 17:26:38 INFO mapred.JobClient:  map 0% reduce 0%
> > 09/10/13 17:27:00 INFO mapred.JobClient: Task Id :
> > attempt_200910131447_0033_m_000000_0, Status : FAILED
> > java.io.FileNotFoundException: File
> file:/home/admin/Desktop/input/test.txt
> > does not exist.
> >        at
> >
> >
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.jav
> > a:420)
> >        at
> >
> >
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:25
> > 9)
> >        at
> >
> >
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(Checks
> > umFileSystem.java:117)
> >        at
> > org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:275)
> >        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:364)
> >        at
> >
> org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:206)
> >        at
> >
> >
> org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.jav
> > a:50)
> >        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:219)
> >        at
> > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2210)
> >
> > However, running in the code as a separate Main method does work well.
> >
> > public static void main (String [] args) throws IOException {
> >
> >     Configuration conf = new Configuration();
> >     FileSystem fs = FileSystem.get(conf);
> >
> >     Path filenamePath = new Path(theFilename);
> >     FSDataOutputStream out = fs.create(new Path("abc.txt"));
> >     out.writeUTF("abc");
> >     out.close();
> >
> > }
> >
> > The above code works fine when running it as a jar in hadoop. The above
> > code
> > successfully creates file in /home/admin/abc.txt when running from admin
> > user.
> >
> >
>



-- 
Chandan Tamrakar

--001636163e07c9a9390475d3389e--

From general-return-607-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 13 16:12:57 2009
Return-Path: <general-return-607-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 48641 invoked from network); 13 Oct 2009 16:12:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Oct 2009 16:12:57 -0000
Received: (qmail 59644 invoked by uid 500); 13 Oct 2009 16:12:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59561 invoked by uid 500); 13 Oct 2009 16:12:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59541 invoked by uid 99); 13 Oct 2009 16:12:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Oct 2009 16:12:56 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.211.187] (HELO mail-yw0-f187.google.com) (209.85.211.187)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Oct 2009 16:12:44 +0000
Received: by ywh17 with SMTP id 17so6523436ywh.2
        for <multiple recipients>; Tue, 13 Oct 2009 09:12:22 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.90.10.1 with SMTP id 1mr4428922agj.62.1255450340157; Tue, 13 
	Oct 2009 09:12:20 -0700 (PDT)
In-Reply-To: <8E86E890-7D86-4CB0-A525-5CF056265717@umbc.edu>
References: <8E86E890-7D86-4CB0-A525-5CF056265717@umbc.edu>
From: Chandan Tamrakar <chandan.tamrakar@nepasoft.com>
Date: Tue, 13 Oct 2009 09:12:00 -0700
Message-ID: <95f8b1a90910130912s2887e513ne68e5728e152dadb@mail.gmail.com>
Subject: Re: 0.20.1 Cluster Setup Problem
To: general@hadoop.apache.org
Cc: common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016362835168eea820475d34f3b
X-Virus-Checked: Checked by ClamAV on apache.org

--0016362835168eea820475d34f3b
Content-Type: text/plain; charset=ISO-8859-1

I think you need to specify the port as well for following port

<property>
 <name>fs.default.name</name>
 <value>hdfs://master_hadoop</value>
</property>


On Tue, Oct 13, 2009 at 7:17 AM, Tejas Lagvankar <tej2@umbc.edu> wrote:

> Hi,
>
>
> We are trying to set up a cluster (starting with 2 machines) using the new
> 0.20.1 version.
>
> On the master machine, just after the server starts, the name node dies off
> with the following exception:
>
> 2009-10-13 01:22:24,740 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException:
> Incomplete HDFS URI, no host: hdfs://master_hadoop
>        at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:78)
>        at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1373)
>        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
>        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1385)
>        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
>        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
>        at org.apache.hadoop.fs.Trash.<init>(Trash.java:62)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startTrashEmptier(NameNode.java:208)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:204)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
>
> Can anyone help ?  Also can anyone send across example configuration files
> for 0.20.1 if they are different than we are using ?
>
> The detail log file is attached along with.
>
>
>
>
> The configuration files are as follows:
>
> MASTER CONFIG
> ------ conf/masters -------
> master_hadoop
>
> ------ conf/slaves -------
> master_hadoop
> slave_hadoop
>
> ------ core-site.xml -------
> <configuration>
>
> <property>
>  <name>fs.default.name</name>
>  <value>hdfs://master_hadoop</value>
> </property>
>
> <property>
>  <name>hadoop.tmp.dir</name>
>  <value>/opt/hadoop-0.20.1/tmp</value>
> </property>
>
> ------ hdfs-site.xml -------
> <property>
>  <name>dfs.replication</name>
>  <value>2</value>
> </property>
>
>
> ------ mapred-site.xml -------
> <property>
>  <name>mapred.job.tracker</name>
>  <value>tejas_hadoop:9001</value>
> </property>
>
>
>
>
>
> SLAVE CONFIG
> ------ core-site.xml -------
> <property>
>  <name>hadoop.tmp.dir</name>
>  <value>/opt/hadoop-0.20.1/tmp/</value>
> </property>
>
>
> <property>
>  <name>fs.default.name</name>
>  <value>hdfs://master_hadoop</value>
> </property>
>
>
> ------ hdfs-site.xml -------
> <property>
>  <name>dfs.replication</name>
>  <value>2</value>
> </property>
>
> ------ mapred-site.xml -------
> <property>
>  <name>mapred.job.tracker</name>
>  <value>tejas_hadoop:9001</value>
> </property>
>
>
>
> Regards,
>
> Tejas Lagvankar
> meettejas@umbc.edu
> www.umbc.edu/~tej2 <http://www.umbc.edu/%7Etej2>
>
>
>
>
>


-- 
Chandan Tamrakar

--0016362835168eea820475d34f3b--

From general-return-608-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 13 16:34:47 2009
Return-Path: <general-return-608-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56180 invoked from network); 13 Oct 2009 16:34:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Oct 2009 16:34:46 -0000
Received: (qmail 6994 invoked by uid 500); 13 Oct 2009 16:34:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6904 invoked by uid 500); 13 Oct 2009 16:34:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6880 invoked by uid 99); 13 Oct 2009 16:34:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Oct 2009 16:34:43 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of tej2@umbc.edu designates 130.85.25.79 as permitted sender)
Received: from [130.85.25.79] (HELO mx4.umbc.edu) (130.85.25.79)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Oct 2009 16:34:31 +0000
Received: from smtp.umbc.edu (localhost [127.0.0.1])
	by umbc.edu (mx4.umbc.edu) with ESMTP id n9DGY8L5004545;
	Tue, 13 Oct 2009 12:34:08 -0400 (EDT)
Received: from [192.168.1.45] (pool-71-179-71-29.bltmmd.east.verizon.net [71.179.71.29])
	(authenticated bits=0)
	by smtp.umbc.edu (mx4-relay.umbc.edu) with ESMTP id n9DGY8D9004540
	(version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO);
	Tue, 13 Oct 2009 12:34:08 -0400 (EDT)
Subject: Re: 0.20.1 Cluster Setup Problem
Mime-Version: 1.0 (Apple Message framework v1076)
Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes
From: Tejas Lagvankar <tej2@umbc.edu>
In-Reply-To: <95f8b1a90910130912s2887e513ne68e5728e152dadb@mail.gmail.com>
Date: Tue, 13 Oct 2009 12:34:07 -0400
Cc: general@hadoop.apache.org,
        Chandan Tamrakar <chandan.tamrakar@nepasoft.com>,
        jun hu <jhk312@gmail.com>, Kevin Sweeney <kevin@yieldex.com>
Content-Transfer-Encoding: 7bit
Message-Id: <4BE1D299-6474-4159-9532-82B523024629@umbc.edu>
References: <8E86E890-7D86-4CB0-A525-5CF056265717@umbc.edu> <95f8b1a90910130912s2887e513ne68e5728e152dadb@mail.gmail.com>
To: common-user@hadoop.apache.org
X-Mailer: Apple Mail (2.1076)
X-Milter-Key: 1255581249:a08eca2decbdace0b3112e562de4cfbd
X-ClamAV: OK
X-Virus-Checked: Checked by ClamAV on apache.org


I get the same error even if i specify the port number. I have tried  
with port numbers 54310 as well as 9000.


Regards,
Tejas

On Oct 13, 2009, at 12:12 PM, Chandan Tamrakar wrote:

> I think you need to specify the port as well for following port
>
> <property>
> <name>fs.default.name</name>
> <value>hdfs://master_hadoop</value>
> </property>
>
>
> On Tue, Oct 13, 2009 at 7:17 AM, Tejas Lagvankar <tej2@umbc.edu>  
> wrote:
>
>> Hi,
>>
>>
>> We are trying to set up a cluster (starting with 2 machines) using  
>> the new
>> 0.20.1 version.
>>
>> On the master machine, just after the server starts, the name node  
>> dies off
>> with the following exception:
>>
>> 2009-10-13 01:22:24,740 ERROR
>> org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException:
>> Incomplete HDFS URI, no host: hdfs://master_hadoop
>>       at
>> org.apache.hadoop.hdfs.DistributedFileSystem.initialize 
>> (DistributedFileSystem.java:78)
>>       at
>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java: 
>> 1373)
>>       at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java: 
>> 66)
>>       at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java: 
>> 1385)
>>       at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
>>       at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
>>       at org.apache.hadoop.fs.Trash.<init>(Trash.java:62)
>>       at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.startTrashEmptier 
>> (NameNode.java:208)
>>       at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize 
>> (NameNode.java:204)
>>       at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init> 
>> (NameNode.java:279)
>>       at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode 
>> (NameNode.java:956)
>>       at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java: 
>> 965)
>>
>> Can anyone help ?  Also can anyone send across example  
>> configuration files
>> for 0.20.1 if they are different than we are using ?
>>
>> The detail log file is attached along with.
>>
>>
>>
>>
>> The configuration files are as follows:
>>
>> MASTER CONFIG
>> ------ conf/masters -------
>> master_hadoop
>>
>> ------ conf/slaves -------
>> master_hadoop
>> slave_hadoop
>>
>> ------ core-site.xml -------
>> <configuration>
>>
>> <property>
>> <name>fs.default.name</name>
>> <value>hdfs://master_hadoop</value>
>> </property>
>>
>> <property>
>> <name>hadoop.tmp.dir</name>
>> <value>/opt/hadoop-0.20.1/tmp</value>
>> </property>
>>
>> ------ hdfs-site.xml -------
>> <property>
>> <name>dfs.replication</name>
>> <value>2</value>
>> </property>
>>
>>
>> ------ mapred-site.xml -------
>> <property>
>> <name>mapred.job.tracker</name>
>> <value>tejas_hadoop:9001</value>
>> </property>
>>
>>
>>
>>
>>
>> SLAVE CONFIG
>> ------ core-site.xml -------
>> <property>
>> <name>hadoop.tmp.dir</name>
>> <value>/opt/hadoop-0.20.1/tmp/</value>
>> </property>
>>
>>
>> <property>
>> <name>fs.default.name</name>
>> <value>hdfs://master_hadoop</value>
>> </property>
>>
>>
>> ------ hdfs-site.xml -------
>> <property>
>> <name>dfs.replication</name>
>> <value>2</value>
>> </property>
>>
>> ------ mapred-site.xml -------
>> <property>
>> <name>mapred.job.tracker</name>
>> <value>tejas_hadoop:9001</value>
>> </property>
>>
>>
>>
>> Regards,
>>
>> Tejas Lagvankar
>> meettejas@umbc.edu
>> www.umbc.edu/~tej2 <http://www.umbc.edu/%7Etej2>
>>
>>
>>
>>
>>
>
>
> -- 
> Chandan Tamrakar

Tejas Lagvankar
meettejas@umbc.edu
www.umbc.edu/~tej2




From general-return-609-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 13 16:37:37 2009
Return-Path: <general-return-609-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56994 invoked from network); 13 Oct 2009 16:37:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Oct 2009 16:37:37 -0000
Received: (qmail 12063 invoked by uid 500); 13 Oct 2009 16:37:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11982 invoked by uid 500); 13 Oct 2009 16:37:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11962 invoked by uid 99); 13 Oct 2009 16:37:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Oct 2009 16:37:34 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.223.171] (HELO mail-iw0-f171.google.com) (209.85.223.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Oct 2009 16:37:22 +0000
Received: by iwn1 with SMTP id 1so5796741iwn.2
        for <multiple recipients>; Tue, 13 Oct 2009 09:37:01 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.120.90 with SMTP id c26mr435381ibr.1.1255451820502; Tue, 
	13 Oct 2009 09:37:00 -0700 (PDT)
In-Reply-To: <4BE1D299-6474-4159-9532-82B523024629@umbc.edu>
References: <8E86E890-7D86-4CB0-A525-5CF056265717@umbc.edu>
	 <95f8b1a90910130912s2887e513ne68e5728e152dadb@mail.gmail.com>
	 <4BE1D299-6474-4159-9532-82B523024629@umbc.edu>
Date: Tue, 13 Oct 2009 16:37:00 +0000
Message-ID: <6e8dca540910130937r27557455h42feefcad01376c5@mail.gmail.com>
Subject: Re: 0.20.1 Cluster Setup Problem
From: Kevin Sweeney <kevin@yieldex.com>
To: Tejas Lagvankar <tej2@umbc.edu>
Cc: common-user@hadoop.apache.org, general@hadoop.apache.org, 
	Chandan Tamrakar <chandan.tamrakar@nepasoft.com>, jun hu <jhk312@gmail.com>
Content-Type: multipart/alternative; boundary=0016e6480e08cb2faa0475d3a745
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6480e08cb2faa0475d3a745
Content-Type: text/plain; charset=ISO-8859-1

did you verify the name resolution?

On Tue, Oct 13, 2009 at 4:34 PM, Tejas Lagvankar <tej2@umbc.edu> wrote:

>
> I get the same error even if i specify the port number. I have tried with
> port numbers 54310 as well as 9000.
>
>
> Regards,
> Tejas
>
>
> On Oct 13, 2009, at 12:12 PM, Chandan Tamrakar wrote:
>
>  I think you need to specify the port as well for following port
>>
>> <property>
>> <name>fs.default.name</name>
>> <value>hdfs://master_hadoop</value>
>> </property>
>>
>>
>> On Tue, Oct 13, 2009 at 7:17 AM, Tejas Lagvankar <tej2@umbc.edu> wrote:
>>
>>  Hi,
>>>
>>>
>>> We are trying to set up a cluster (starting with 2 machines) using the
>>> new
>>> 0.20.1 version.
>>>
>>> On the master machine, just after the server starts, the name node dies
>>> off
>>> with the following exception:
>>>
>>> 2009-10-13 01:22:24,740 ERROR
>>> org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException:
>>> Incomplete HDFS URI, no host: hdfs://master_hadoop
>>>      at
>>>
>>> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:78)
>>>      at
>>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1373)
>>>      at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
>>>      at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1385)
>>>      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
>>>      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
>>>      at org.apache.hadoop.fs.Trash.<init>(Trash.java:62)
>>>      at
>>>
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.startTrashEmptier(NameNode.java:208)
>>>      at
>>>
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:204)
>>>      at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279)
>>>      at
>>>
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
>>>      at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
>>>
>>> Can anyone help ?  Also can anyone send across example configuration
>>> files
>>> for 0.20.1 if they are different than we are using ?
>>>
>>> The detail log file is attached along with.
>>>
>>>
>>>
>>>
>>> The configuration files are as follows:
>>>
>>> MASTER CONFIG
>>> ------ conf/masters -------
>>> master_hadoop
>>>
>>> ------ conf/slaves -------
>>> master_hadoop
>>> slave_hadoop
>>>
>>> ------ core-site.xml -------
>>> <configuration>
>>>
>>> <property>
>>> <name>fs.default.name</name>
>>> <value>hdfs://master_hadoop</value>
>>> </property>
>>>
>>> <property>
>>> <name>hadoop.tmp.dir</name>
>>> <value>/opt/hadoop-0.20.1/tmp</value>
>>> </property>
>>>
>>> ------ hdfs-site.xml -------
>>> <property>
>>> <name>dfs.replication</name>
>>> <value>2</value>
>>> </property>
>>>
>>>
>>> ------ mapred-site.xml -------
>>> <property>
>>> <name>mapred.job.tracker</name>
>>> <value>tejas_hadoop:9001</value>
>>> </property>
>>>
>>>
>>>
>>>
>>>
>>> SLAVE CONFIG
>>> ------ core-site.xml -------
>>> <property>
>>> <name>hadoop.tmp.dir</name>
>>> <value>/opt/hadoop-0.20.1/tmp/</value>
>>> </property>
>>>
>>>
>>> <property>
>>> <name>fs.default.name</name>
>>> <value>hdfs://master_hadoop</value>
>>> </property>
>>>
>>>
>>> ------ hdfs-site.xml -------
>>> <property>
>>> <name>dfs.replication</name>
>>> <value>2</value>
>>> </property>
>>>
>>> ------ mapred-site.xml -------
>>> <property>
>>> <name>mapred.job.tracker</name>
>>> <value>tejas_hadoop:9001</value>
>>> </property>
>>>
>>>
>>>
>>> Regards,
>>>
>>> Tejas Lagvankar
>>> meettejas@umbc.edu
>>> www.umbc.edu/~tej2 <http://www.umbc.edu/%7Etej2>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>> --
>> Chandan Tamrakar
>>
>
> Tejas Lagvankar
> meettejas@umbc.edu
> www.umbc.edu/~tej2
>
>
>
>

--0016e6480e08cb2faa0475d3a745--

From general-return-610-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 13 16:41:46 2009
Return-Path: <general-return-610-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57710 invoked from network); 13 Oct 2009 16:41:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Oct 2009 16:41:46 -0000
Received: (qmail 18260 invoked by uid 500); 13 Oct 2009 16:41:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18175 invoked by uid 500); 13 Oct 2009 16:41:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18155 invoked by uid 99); 13 Oct 2009 16:41:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Oct 2009 16:41:43 +0000
X-ASF-Spam-Status: No, hits=-1.8 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of tej2@umbc.edu designates 130.85.25.77 as permitted sender)
Received: from [130.85.25.77] (HELO mx2.umbc.edu) (130.85.25.77)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Oct 2009 16:41:31 +0000
Received: from smtp.umbc.edu (localhost [127.0.0.1])
	by umbc.edu (mx2.umbc.edu) with ESMTP id n9DGf9Hb007267;
	Tue, 13 Oct 2009 12:41:09 -0400 (EDT)
Received: from [192.168.1.45] (pool-71-179-71-29.bltmmd.east.verizon.net [71.179.71.29])
	(authenticated bits=0)
	by smtp.umbc.edu (mx2-relay.umbc.edu) with ESMTP id n9DGf7uv007253
	(version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO);
	Tue, 13 Oct 2009 12:41:08 -0400 (EDT)
Subject: Re: 0.20.1 Cluster Setup Problem
Mime-Version: 1.0 (Apple Message framework v1076)
Content-Type: multipart/alternative; boundary=Apple-Mail-6-119477132
From: Tejas Lagvankar <tej2@umbc.edu>
In-Reply-To: <6e8dca540910130937r27557455h42feefcad01376c5@mail.gmail.com>
Date: Tue, 13 Oct 2009 12:41:07 -0400
Cc: common-user@hadoop.apache.org, general@hadoop.apache.org
Message-Id: <E3347394-0567-4BC6-A613-FF835A44F861@umbc.edu>
References: <8E86E890-7D86-4CB0-A525-5CF056265717@umbc.edu> <95f8b1a90910130912s2887e513ne68e5728e152dadb@mail.gmail.com> <4BE1D299-6474-4159-9532-82B523024629@umbc.edu> <6e8dca540910130937r27557455h42feefcad01376c5@mail.gmail.com>
To: Kevin Sweeney <kevin@yieldex.com>
X-Mailer: Apple Mail (2.1076)
X-Milter-Key: 1255581670:3bb99aea61c9f3150b690e32e6d9b754
X-ClamAV: OK
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-6-119477132
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
	charset=us-ascii;
	format=flowed;
	delsp=yes

By name resolution, I assume that you mean the name mentioned in /etc/ 
hosts.  Yes, in the logs, the IP address appears in the beginning.
Correct me if I'm wrong
I will also try with using just the IP's instead of the aliases.

On Oct 13, 2009, at 12:37 PM, Kevin Sweeney wrote:

> did you verify the name resolution?
>
> On Tue, Oct 13, 2009 at 4:34 PM, Tejas Lagvankar <tej2@umbc.edu>  
> wrote:
>
> I get the same error even if i specify the port number. I have tried  
> with port numbers 54310 as well as 9000.
>
>
> Regards,
> Tejas
>
>
> On Oct 13, 2009, at 12:12 PM, Chandan Tamrakar wrote:
>
> I think you need to specify the port as well for following port
>
> <property>
> <name>fs.default.name</name>
> <value>hdfs://master_hadoop</value>
> </property>
>
>
> On Tue, Oct 13, 2009 at 7:17 AM, Tejas Lagvankar <tej2@umbc.edu>  
> wrote:
>
> Hi,
>
>
> We are trying to set up a cluster (starting with 2 machines) using  
> the new
> 0.20.1 version.
>
> On the master machine, just after the server starts, the name node  
> dies off
> with the following exception:
>
> 2009-10-13 01:22:24,740 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException:
> Incomplete HDFS URI, no host: hdfs://master_hadoop
>      at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize 
> (DistributedFileSystem.java:78)
>      at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1373)
>      at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
>      at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java: 
> 1385)
>      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
>      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
>      at org.apache.hadoop.fs.Trash.<init>(Trash.java:62)
>      at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startTrashEmptier 
> (NameNode.java:208)
>      at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize 
> (NameNode.java:204)
>      at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java: 
> 279)
>      at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode 
> (NameNode.java:956)
>      at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java: 
> 965)
>
> Can anyone help ?  Also can anyone send across example configuration  
> files
> for 0.20.1 if they are different than we are using ?
>
> The detail log file is attached along with.
>
>
>
>
> The configuration files are as follows:
>
> MASTER CONFIG
> ------ conf/masters -------
> master_hadoop
>
> ------ conf/slaves -------
> master_hadoop
> slave_hadoop
>
> ------ core-site.xml -------
> <configuration>
>
> <property>
> <name>fs.default.name</name>
> <value>hdfs://master_hadoop</value>
> </property>
>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/opt/hadoop-0.20.1/tmp</value>
> </property>
>
> ------ hdfs-site.xml -------
> <property>
> <name>dfs.replication</name>
> <value>2</value>
> </property>
>
>
> ------ mapred-site.xml -------
> <property>
> <name>mapred.job.tracker</name>
> <value>tejas_hadoop:9001</value>
> </property>
>
>
>
>
>
> SLAVE CONFIG
> ------ core-site.xml -------
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/opt/hadoop-0.20.1/tmp/</value>
> </property>
>
>
> <property>
> <name>fs.default.name</name>
> <value>hdfs://master_hadoop</value>
> </property>
>
>
> ------ hdfs-site.xml -------
> <property>
> <name>dfs.replication</name>
> <value>2</value>
> </property>
>
> ------ mapred-site.xml -------
> <property>
> <name>mapred.job.tracker</name>
> <value>tejas_hadoop:9001</value>
> </property>
>
>
>
> Regards,
>
> Tejas Lagvankar
> meettejas@umbc.edu
> www.umbc.edu/~tej2 <http://www.umbc.edu/%7Etej2>
>
>
>
>
>
>
>
> -- 
> Chandan Tamrakar
>
> Tejas Lagvankar
> meettejas@umbc.edu
> www.umbc.edu/~tej2
>
>
>
>
>
>
>

Tejas Lagvankar
meettejas@umbc.edu
www.umbc.edu/~tej2




--Apple-Mail-6-119477132--

From general-return-611-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 13 17:02:07 2009
Return-Path: <general-return-611-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66264 invoked from network); 13 Oct 2009 17:02:06 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Oct 2009 17:02:06 -0000
Received: (qmail 68070 invoked by uid 500); 13 Oct 2009 17:02:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67986 invoked by uid 500); 13 Oct 2009 17:02:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67967 invoked by uid 99); 13 Oct 2009 17:02:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Oct 2009 17:02:02 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of tej2@umbc.edu designates 130.85.25.79 as permitted sender)
Received: from [130.85.25.79] (HELO mx4.umbc.edu) (130.85.25.79)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Oct 2009 17:01:51 +0000
Received: from smtp.umbc.edu (localhost [127.0.0.1])
	by umbc.edu (mx4.umbc.edu) with ESMTP id n9DH1UcI017313;
	Tue, 13 Oct 2009 13:01:30 -0400 (EDT)
Received: from [192.168.1.45] (pool-71-179-71-29.bltmmd.east.verizon.net [71.179.71.29])
	(authenticated bits=0)
	by smtp.umbc.edu (mx4-relay.umbc.edu) with ESMTP id n9DH1Tqb017300
	(version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO);
	Tue, 13 Oct 2009 13:01:29 -0400 (EDT)
Subject: Re: 0.20.1 Cluster Setup Problem
Mime-Version: 1.0 (Apple Message framework v1076)
Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes
From: Tejas Lagvankar <tej2@umbc.edu>
In-Reply-To: <E3347394-0567-4BC6-A613-FF835A44F861@umbc.edu>
Date: Tue, 13 Oct 2009 13:01:29 -0400
Cc: Kevin Sweeney <kevin@yieldex.com>, common-user@hadoop.apache.org
Content-Transfer-Encoding: 7bit
Message-Id: <962B880C-EC03-40D7-AEFE-C0A6619D5BE8@umbc.edu>
References: <8E86E890-7D86-4CB0-A525-5CF056265717@umbc.edu> <95f8b1a90910130912s2887e513ne68e5728e152dadb@mail.gmail.com> <4BE1D299-6474-4159-9532-82B523024629@umbc.edu> <6e8dca540910130937r27557455h42feefcad01376c5@mail.gmail.com> <E3347394-0567-4BC6-A613-FF835A44F861@umbc.edu>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1076)
X-Milter-Key: 1255582890:faa1ceb86d3541179c132c8da7b5a473
X-ClamAV: OK
X-Virus-Checked: Checked by ClamAV on apache.org

Hey Kevin,

You were right...
I changed all my aliases to IP addresses. It worked !

Thank you all again :)

Regards,
Tejas

On Oct 13, 2009, at 12:41 PM, Tejas Lagvankar wrote:

> By name resolution, I assume that you mean the name mentioned in / 
> etc/hosts.  Yes, in the logs, the IP address appears in the beginning.
> Correct me if I'm wrong
> I will also try with using just the IP's instead of the aliases.
>
> On Oct 13, 2009, at 12:37 PM, Kevin Sweeney wrote:
>
>> did you verify the name resolution?
>>
>> On Tue, Oct 13, 2009 at 4:34 PM, Tejas Lagvankar <tej2@umbc.edu>  
>> wrote:
>>
>> I get the same error even if i specify the port number. I have  
>> tried with port numbers 54310 as well as 9000.
>>
>>
>> Regards,
>> Tejas
>>
>>
>> On Oct 13, 2009, at 12:12 PM, Chandan Tamrakar wrote:
>>
>> I think you need to specify the port as well for following port
>>
>> <property>
>> <name>fs.default.name</name>
>> <value>hdfs://master_hadoop</value>
>> </property>
>>
>>
>> On Tue, Oct 13, 2009 at 7:17 AM, Tejas Lagvankar <tej2@umbc.edu>  
>> wrote:
>>
>> Hi,
>>
>>
>> We are trying to set up a cluster (starting with 2 machines) using  
>> the new
>> 0.20.1 version.
>>
>> On the master machine, just after the server starts, the name node  
>> dies off
>> with the following exception:
>>
>> 2009-10-13 01:22:24,740 ERROR
>> org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException:
>> Incomplete HDFS URI, no host: hdfs://master_hadoop
>>     at
>> org.apache.hadoop.hdfs.DistributedFileSystem.initialize 
>> (DistributedFileSystem.java:78)
>>     at
>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java: 
>> 1373)
>>     at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
>>     at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java: 
>> 1385)
>>     at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
>>     at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
>>     at org.apache.hadoop.fs.Trash.<init>(Trash.java:62)
>>     at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.startTrashEmptier 
>> (NameNode.java:208)
>>     at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize 
>> (NameNode.java:204)
>>     at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init> 
>> (NameNode.java:279)
>>     at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode 
>> (NameNode.java:956)
>>     at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java: 
>> 965)
>>
>> Can anyone help ?  Also can anyone send across example  
>> configuration files
>> for 0.20.1 if they are different than we are using ?
>>
>> The detail log file is attached along with.
>>
>>
>>
>>
>> The configuration files are as follows:
>>
>> MASTER CONFIG
>> ------ conf/masters -------
>> master_hadoop
>>
>> ------ conf/slaves -------
>> master_hadoop
>> slave_hadoop
>>
>> ------ core-site.xml -------
>> <configuration>
>>
>> <property>
>> <name>fs.default.name</name>
>> <value>hdfs://master_hadoop</value>
>> </property>
>>
>> <property>
>> <name>hadoop.tmp.dir</name>
>> <value>/opt/hadoop-0.20.1/tmp</value>
>> </property>
>>
>> ------ hdfs-site.xml -------
>> <property>
>> <name>dfs.replication</name>
>> <value>2</value>
>> </property>
>>
>>
>> ------ mapred-site.xml -------
>> <property>
>> <name>mapred.job.tracker</name>
>> <value>tejas_hadoop:9001</value>
>> </property>
>>
>>
>>
>>
>>
>> SLAVE CONFIG
>> ------ core-site.xml -------
>> <property>
>> <name>hadoop.tmp.dir</name>
>> <value>/opt/hadoop-0.20.1/tmp/</value>
>> </property>
>>
>>
>> <property>
>> <name>fs.default.name</name>
>> <value>hdfs://master_hadoop</value>
>> </property>
>>
>>
>> ------ hdfs-site.xml -------
>> <property>
>> <name>dfs.replication</name>
>> <value>2</value>
>> </property>
>>
>> ------ mapred-site.xml -------
>> <property>
>> <name>mapred.job.tracker</name>
>> <value>tejas_hadoop:9001</value>
>> </property>
>>
>>
>>
>> Regards,
>>
>> Tejas Lagvankar
>> meettejas@umbc.edu
>> www.umbc.edu/~tej2 <http://www.umbc.edu/%7Etej2>
>>
>>
>>
>>
>>
>>
>>
>> -- 
>> Chandan Tamrakar
>>
>> Tejas Lagvankar
>> meettejas@umbc.edu
>> www.umbc.edu/~tej2
>>
>>
>>
>>
>>
>>
>>
>
> Tejas Lagvankar
> meettejas@umbc.edu
> www.umbc.edu/~tej2
>
>
>

Tejas Lagvankar
meettejas@umbc.edu
www.umbc.edu/~tej2




From general-return-612-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 13 18:30:23 2009
Return-Path: <general-return-612-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 92106 invoked from network); 13 Oct 2009 18:30:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Oct 2009 18:30:23 -0000
Received: (qmail 50127 invoked by uid 500); 13 Oct 2009 18:30:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49589 invoked by uid 500); 13 Oct 2009 18:30:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49475 invoked by uid 99); 13 Oct 2009 18:30:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Oct 2009 18:30:19 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of tej2@umbc.edu designates 130.85.25.77 as permitted sender)
Received: from [130.85.25.77] (HELO mx2.umbc.edu) (130.85.25.77)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Oct 2009 18:30:07 +0000
Received: from smtp.umbc.edu (localhost [127.0.0.1])
	by umbc.edu (mx2.umbc.edu) with ESMTP id n9DITjdj008256;
	Tue, 13 Oct 2009 14:29:45 -0400 (EDT)
Received: from wireless-217-247.wireless.umbc.edu (wireless-217-247.wireless.UMBC.EDU [130.85.217.247])
	(authenticated bits=0)
	by smtp.umbc.edu (mx2-relay.umbc.edu) with ESMTP id n9DITi0J008253
	(version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO);
	Tue, 13 Oct 2009 14:29:44 -0400 (EDT)
Subject: Re: 0.20.1 Cluster Setup Problem
Mime-Version: 1.0 (Apple Message framework v1076)
Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes
From: Tejas Lagvankar <tej2@umbc.edu>
In-Reply-To: <45f85f70910131050p50f11857o2984fc2d4cffd8b5@mail.gmail.com>
Date: Tue, 13 Oct 2009 14:29:44 -0400
Cc: general@hadoop.apache.org, Todd Lipcon <todd@cloudera.com>
Content-Transfer-Encoding: 7bit
Message-Id: <03571714-E395-4189-90A4-40C943457CF0@umbc.edu>
References: <8E86E890-7D86-4CB0-A525-5CF056265717@umbc.edu>  <95f8b1a90910130912s2887e513ne68e5728e152dadb@mail.gmail.com>  <4BE1D299-6474-4159-9532-82B523024629@umbc.edu> <6e8dca540910130937r27557455h42feefcad01376c5@mail.gmail.com>  <E3347394-0567-4BC6-A613-FF835A44F861@umbc.edu> <962B880C-EC03-40D7-AEFE-C0A6619D5BE8@umbc.edu> <45f85f70910131050p50f11857o2984fc2d4cffd8b5@mail.gmail.com>
To: common-user@hadoop.apache.org
X-Mailer: Apple Mail (2.1076)
X-Milter-Key: 1255588185:01d2f63ce28102aafde70949b388b578
X-ClamAV: OK
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks Todd,

I never thought of that !!

Regards,
Tejas
On Oct 13, 2009, at 1:50 PM, Todd Lipcon wrote:

> Your issue was probably that slave_hadoop and master_hadoop are not  
> valid
> host names:
>
> RFCs <http://en.wikipedia.org/wiki/Request_for_Comments> mandate  
> that a
> hostname's labels may contain only the
> ASCII<http://en.wikipedia.org/wiki/ASCII>letters 'a' through 'z'
> (case-insensitive), the digits '0' through '9', and
> the hyphen. Hostname labels cannot begin or end with a hyphen. No  
> other
> symbols, punctuation characters, or blank spaces are permitted.
>
> from http://en.wikipedia.org/wiki/Hostname
>
> -Todd
>
> On Tue, Oct 13, 2009 at 10:01 AM, Tejas Lagvankar <tej2@umbc.edu>  
> wrote:
>
>> Hey Kevin,
>>
>> You were right...
>> I changed all my aliases to IP addresses. It worked !
>>
>> Thank you all again :)
>>
>> Regards,
>> Tejas
>>
>>
>> On Oct 13, 2009, at 12:41 PM, Tejas Lagvankar wrote:
>>
>> By name resolution, I assume that you mean the name mentioned in
>>> /etc/hosts.  Yes, in the logs, the IP address appears in the  
>>> beginning.
>>> Correct me if I'm wrong
>>> I will also try with using just the IP's instead of the aliases.
>>>
>>> On Oct 13, 2009, at 12:37 PM, Kevin Sweeney wrote:
>>>
>>> did you verify the name resolution?
>>>>
>>>> On Tue, Oct 13, 2009 at 4:34 PM, Tejas Lagvankar <tej2@umbc.edu>  
>>>> wrote:
>>>>
>>>> I get the same error even if i specify the port number. I have  
>>>> tried with
>>>> port numbers 54310 as well as 9000.
>>>>
>>>>
>>>> Regards,
>>>> Tejas
>>>>
>>>>
>>>> On Oct 13, 2009, at 12:12 PM, Chandan Tamrakar wrote:
>>>>
>>>> I think you need to specify the port as well for following port
>>>>
>>>> <property>
>>>> <name>fs.default.name</name>
>>>> <value>hdfs://master_hadoop</value>
>>>> </property>
>>>>
>>>>
>>>> On Tue, Oct 13, 2009 at 7:17 AM, Tejas Lagvankar <tej2@umbc.edu>  
>>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>>
>>>> We are trying to set up a cluster (starting with 2 machines)  
>>>> using the
>>>> new
>>>> 0.20.1 version.
>>>>
>>>> On the master machine, just after the server starts, the name  
>>>> node dies
>>>> off
>>>> with the following exception:
>>>>
>>>> 2009-10-13 01:22:24,740 ERROR
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode:  
>>>> java.io.IOException:
>>>> Incomplete HDFS URI, no host: hdfs://master_hadoop
>>>>   at
>>>>
>>>> org.apache.hadoop.hdfs.DistributedFileSystem.initialize 
>>>> (DistributedFileSystem.java:78)
>>>>   at
>>>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java: 
>>>> 1373)
>>>>   at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
>>>>   at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java: 
>>>> 1385)
>>>>   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
>>>>   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
>>>>   at org.apache.hadoop.fs.Trash.<init>(Trash.java:62)
>>>>   at
>>>>
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.startTrashEmptier 
>>>> (NameNode.java:208)
>>>>   at
>>>>
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize 
>>>> (NameNode.java:204)
>>>>   at
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init> 
>>>> (NameNode.java:279)
>>>>   at
>>>>
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode 
>>>> (NameNode.java:956)
>>>>   at
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main 
>>>> (NameNode.java:965)
>>>>
>>>> Can anyone help ?  Also can anyone send across example  
>>>> configuration
>>>> files
>>>> for 0.20.1 if they are different than we are using ?
>>>>
>>>> The detail log file is attached along with.
>>>>
>>>>
>>>>
>>>>
>>>> The configuration files are as follows:
>>>>
>>>> MASTER CONFIG
>>>> ------ conf/masters -------
>>>> master_hadoop
>>>>
>>>> ------ conf/slaves -------
>>>> master_hadoop
>>>> slave_hadoop
>>>>
>>>> ------ core-site.xml -------
>>>> <configuration>
>>>>
>>>> <property>
>>>> <name>fs.default.name</name>
>>>> <value>hdfs://master_hadoop</value>
>>>> </property>
>>>>
>>>> <property>
>>>> <name>hadoop.tmp.dir</name>
>>>> <value>/opt/hadoop-0.20.1/tmp</value>
>>>> </property>
>>>>
>>>> ------ hdfs-site.xml -------
>>>> <property>
>>>> <name>dfs.replication</name>
>>>> <value>2</value>
>>>> </property>
>>>>
>>>>
>>>> ------ mapred-site.xml -------
>>>> <property>
>>>> <name>mapred.job.tracker</name>
>>>> <value>tejas_hadoop:9001</value>
>>>> </property>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> SLAVE CONFIG
>>>> ------ core-site.xml -------
>>>> <property>
>>>> <name>hadoop.tmp.dir</name>
>>>> <value>/opt/hadoop-0.20.1/tmp/</value>
>>>> </property>
>>>>
>>>>
>>>> <property>
>>>> <name>fs.default.name</name>
>>>> <value>hdfs://master_hadoop</value>
>>>> </property>
>>>>
>>>>
>>>> ------ hdfs-site.xml -------
>>>> <property>
>>>> <name>dfs.replication</name>
>>>> <value>2</value>
>>>> </property>
>>>>
>>>> ------ mapred-site.xml -------
>>>> <property>
>>>> <name>mapred.job.tracker</name>
>>>> <value>tejas_hadoop:9001</value>
>>>> </property>
>>>>
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Tejas Lagvankar
>>>> meettejas@umbc.edu
>>>> www.umbc.edu/~tej2 <http://www.umbc.edu/%7Etej2> <
>>>> http://www.umbc.edu/%7Etej2>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Chandan Tamrakar
>>>>
>>>> Tejas Lagvankar
>>>> meettejas@umbc.edu
>>>> www.umbc.edu/~tej2 <http://www.umbc.edu/%7Etej2>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>> Tejas Lagvankar
>>> meettejas@umbc.edu
>>> www.umbc.edu/~tej2 <http://www.umbc.edu/%7Etej2>
>>>
>>>
>>>
>>>
>> Tejas Lagvankar
>> meettejas@umbc.edu
>> www.umbc.edu/~tej2 <http://www.umbc.edu/%7Etej2>
>>
>>
>>
>>

Tejas Lagvankar
meettejas@umbc.edu
www.umbc.edu/~tej2




From general-return-613-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 14 02:01:06 2009
Return-Path: <general-return-613-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60605 invoked from network); 14 Oct 2009 02:01:06 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Oct 2009 02:01:06 -0000
Received: (qmail 20085 invoked by uid 500); 14 Oct 2009 02:01:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20000 invoked by uid 500); 14 Oct 2009 02:01:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19990 invoked by uid 99); 14 Oct 2009 02:01:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Oct 2009 02:01:05 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Oct 2009 02:00:53 +0000
Received: from SNV-EXPF01.ds.corp.yahoo.com (snv-expf01.ds.corp.yahoo.com [207.126.227.250])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n9E1xANG053276
	for <general@hadoop.apache.org>; Tue, 13 Oct 2009 18:59:10 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:x-originalarrivaltime;
	b=PjvMXG9trwso9zoxFbVaZvyJQdb5v21U9jx/YRfXQGgLVCy6YrjCIybACjYUS8Io
Received: from SNV-EXVS01.ds.corp.yahoo.com ([216.145.51.185]) by SNV-EXPF01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Tue, 13 Oct 2009 18:59:09 -0700
Received: from 10.72.106.176 ([10.72.106.176]) by SNV-EXVS01.ds.corp.yahoo.com ([216.145.51.166]) via Exchange Front-End Server snv-webmail.corp.yahoo.com ([207.126.227.248]) with Microsoft Exchange Server HTTP-DAV ;
 Wed, 14 Oct 2009 01:59:10 +0000
User-Agent: Microsoft-Entourage/12.20.0.090605
Date: Tue, 13 Oct 2009 18:59:08 -0700
Subject: Re: HTTP transport?
From: Kan Zhang <kan@yahoo-inc.com>
To: <general@hadoop.apache.org>
Message-ID: <C6FA7E7C.1906D%kan@yahoo-inc.com>
Thread-Topic: HTTP transport?
Thread-Index: AcpMcepVWrv6ZArZ20GR5gnwIvfXLw==
In-Reply-To: <4ACF956E.1000201@apache.org>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-OriginalArrivalTime: 14 Oct 2009 01:59:09.0693 (UTC) FILETIME=[EB5852D0:01CA4C71]
X-Virus-Checked: Checked by ClamAV on apache.org




On 10/9/09 12:56 PM, "Doug Cutting" <cutting@apache.org> wrote:

> Sanjay Radia wrote:
>> Will the RPC over HTTP be transparent so that that we can replace with a
>> different layer if needed?
> 
> Yes.
> 
>> My worry was the separation of data and checksums; someone had mentioned
>> that one could do this over 2 RPCs - that is not transparent.
> 
> That was suggested as a possibility if we did not want to use RPC for
> data, but rather raw HTTP, e.g., with a separate URL per block.  The
> zerocopy support built into most HTTP servers only supports entire
> responses from a single file, so if we wanted to take advantage of these
> zerocopy implementations we'd not use RPC for block access, but could
> use HTTP and hence share security, etc.  Using raw HTTP for block access
> might also perform better, since it can use TCP flow control, rather
> than RPC call/response.  In my microbenchmarks, RPC call/response was
> fast enough to easily saturate disks and networks, so that might be
> moot, although RPC call/response for file data may use more CPU than
> we'd like.  With our own transport implementation we could get RPC
> call/response to use zerocopy for file data.
> 

One problem I see with using HTTP is that it's expensive to provide data
encryption. We're currently adding 2 authentication mechanisms (Kerberos and
DIGEST-MD5) to our existing RPC. Both of them can provide data encryption
for subsequent communication over the authenticated channel. However, when
similar authentication mechanisms are specified for HTTP (SPNEGO and HTTP
DIGEST, respectively), they don't provide data encryption (correct me if I'm
wrong). For data encryption over HTTP, one has to use SSL, which is
expensive.

Kan


From general-return-614-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 14 06:05:20 2009
Return-Path: <general-return-614-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34184 invoked from network); 14 Oct 2009 06:05:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Oct 2009 06:05:11 -0000
Received: (qmail 85933 invoked by uid 500); 14 Oct 2009 06:05:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85853 invoked by uid 500); 14 Oct 2009 06:05:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85836 invoked by uid 99); 14 Oct 2009 06:05:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Oct 2009 06:05:10 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE,URIBL_BLACK
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jason.hadoop@gmail.com designates 209.85.222.195 as permitted sender)
Received: from [209.85.222.195] (HELO mail-pz0-f195.google.com) (209.85.222.195)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Oct 2009 06:05:06 +0000
Received: by pzk33 with SMTP id 33so7994180pzk.2
        for <general@hadoop.apache.org>; Tue, 13 Oct 2009 23:04:46 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=f5XxYZmHeJKkUaYS5zwAoBPLW/CW6UtMSkmGalSgYsk=;
        b=HQcgPdKM5yLdqZj1kGUfe4wy6VYFn4qhhDZGrTbZfKyvTfBCahSOQj8QVxdQPesF8V
         UiSqwORkzrzsd444kD4Bll1/wGjg1YJHjgORmsbLSsQIMV+c6XAY4XC67kYwBMn9EBx9
         KWTa8PdGIExPmc4EKm4moh+f4UnG1cbKf7d4M=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=gCbNKTX1Dq/oOg+3tGwtEX6d+2hf0K/HuY5pr+XUjYPRhGqGuYNOWItJGklrhZ3CGa
         UvIFE4PHsHg1/7Xu1IZPkJJ/D1TrnOidtp8jlpdFD7LPcG/9K26HLDeTYG+N8qRtM67+
         3i/SVA9Kwa3wwH2mmAyYa8qwwoVWyxB1B2HeQ=
MIME-Version: 1.0
Received: by 10.140.136.19 with SMTP id j19mr984370rvd.187.1255500285582; Tue, 
	13 Oct 2009 23:04:45 -0700 (PDT)
In-Reply-To: <95f8b1a90910130905h38357b3fmd53c5a4b1c9ee5f8@mail.gmail.com>
References: <E1MxJB1-0005FT-F2@st05.mi6.kiev.ua>
	 <314098690910120843j36dc05b7kb13fb42a1e633211@mail.gmail.com>
	 <4ad4790d.1438560a.1db9.3612@mx.google.com>
	 <8211a1320910130649y14b06a67s13b268f31f6f786e@mail.gmail.com>
	 <95f8b1a90910130905h38357b3fmd53c5a4b1c9ee5f8@mail.gmail.com>
Date: Tue, 13 Oct 2009 23:04:45 -0700
Message-ID: <314098690910132304s2619b2b4hba0c4a83b76a65e8@mail.gmail.com>
Subject: Re: Reading files from local file system
From: Jason Venner <jason.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd2531a899e170475def0d6

--000e0cd2531a899e170475def0d6
Content-Type: text/plain; charset=ISO-8859-1

If you want to open a local file in hadoop you have 3 simple ways

1: use file:///path
2: get a LocalFileSystem object from the FileSystem
/**
   * Get the local file syste
   * @param conf the configuration to configure the file system with
   * @return a LocalFileSystem
   */
  public static LocalFileSystem getLocal(Configuration conf)
    throws IOException {
    return (LocalFileSystem)get(LocalFileSystem.NAME, conf);
  }

3: use the java.io File* classes.

On Tue, Oct 13, 2009 at 9:05 AM, Chandan Tamrakar <
chandan.tamrakar@nepasoft.com> wrote:

> Do I need to change any configuration beside changing the default file
> system to "local file system' ?
> I am trying to input for example  input.txt to map job
>
> input.txt will contain file location as following
>
> file://path/abc1.doc
> file://path/abc2.doc
> ..
> ...
>
> map program will read each line from input.txt and process them
>
> Do i need to change any configuration ? This is similar to how Nutch crawls
> .
>
> any feedbacks would be appreciated
>
> thanks
>
>
>
> On Tue, Oct 13, 2009 at 6:49 AM, Jeff Zhang <zjffdu@gmail.com> wrote:
>
> > Maybe you could debug your mapreduce job in eclipse, since you run it in
> > local mode.
> >
> >
> >
> > On Tue, Oct 13, 2009 at 5:56 AM, Chandan Tamrakar <
> > chandan.tamrakar@nepasoft.com> wrote:
> >
> > >
> > >
> > > We are trying to read files from local file system. But when running
> the
> > > map
> > > reduce it is not able to read files from the input location (the input
> > > location is also local file system location).
> > >
> > > For this we changed the configuration of the hadoop-site.xml as shown
> > > below:
> > >
> > > /etc/conf/hadoop/hadoop-site.xml
> > >
> > > <property>
> > >    <name>fs.default.name</name>
> > >    <value>file:///</value>
> > >  </property>
> > >
> > >
> > >  [admin@localhost ~]$ hadoop jar Test.jar /home/admin/input/test.txt
> > > output1
> > >
> > > Suppose Test.txt is pain text file that contains
> > > Test1
> > > Test2
> > > Test3
> > >
> > >
> > > While running simple MapReduce job we get following exception  "File
> not
> > > found exception " , we are using TextInputFormat in our Job
> configuration
> > >
> > >
> > > 09/10/13 17:26:35 WARN mapred.JobClient: Use GenericOptionsParser for
> > > parsing the arguments. Applications should implement Tool for the same.
> > > 09/10/13 17:26:35 INFO mapred.FileInputFormat: Total input paths to
> > process
> > > : 1
> > > 09/10/13 17:26:35 INFO mapred.FileInputFormat: Total input paths to
> > process
> > > : 1
> > > 09/10/13 17:26:37 INFO mapred.JobClient: Running job:
> > job_200910131447_0033
> > > 09/10/13 17:26:38 INFO mapred.JobClient:  map 0% reduce 0%
> > > 09/10/13 17:27:00 INFO mapred.JobClient: Task Id :
> > > attempt_200910131447_0033_m_000000_0, Status : FAILED
> > > java.io.FileNotFoundException: File
> > file:/home/admin/Desktop/input/test.txt
> > > does not exist.
> > >        at
> > >
> > >
> >
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.jav
> > > a:420)
> > >        at
> > >
> > >
> >
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:25
> > > 9)
> > >        at
> > >
> > >
> >
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(Checks
> > > umFileSystem.java:117)
> > >        at
> > >
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:275)
> > >        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:364)
> > >        at
> > >
> >
> org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:206)
> > >        at
> > >
> > >
> >
> org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.jav
> > > a:50)
> > >        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:219)
> > >        at
> > > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2210)
> > >
> > > However, running in the code as a separate Main method does work well.
> > >
> > > public static void main (String [] args) throws IOException {
> > >
> > >     Configuration conf = new Configuration();
> > >     FileSystem fs = FileSystem.get(conf);
> > >
> > >     Path filenamePath = new Path(theFilename);
> > >     FSDataOutputStream out = fs.create(new Path("abc.txt"));
> > >     out.writeUTF("abc");
> > >     out.close();
> > >
> > > }
> > >
> > > The above code works fine when running it as a jar in hadoop. The above
> > > code
> > > successfully creates file in /home/admin/abc.txt when running from
> admin
> > > user.
> > >
> > >
> >
>
>
>
> --
> Chandan Tamrakar
>



-- 
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals

--000e0cd2531a899e170475def0d6--

From general-return-615-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 14 06:19:26 2009
Return-Path: <general-return-615-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38664 invoked from network); 14 Oct 2009 06:19:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Oct 2009 06:19:26 -0000
Received: (qmail 10764 invoked by uid 500); 14 Oct 2009 06:19:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 10675 invoked by uid 500); 14 Oct 2009 06:19:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 10665 invoked by uid 99); 14 Oct 2009 06:19:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Oct 2009 06:19:25 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE,URIBL_BLACK
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.85.210.197] (HELO mail-yx0-f197.google.com) (209.85.210.197)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Oct 2009 06:19:21 +0000
Received: by yxe35 with SMTP id 35so8006163yxe.2
        for <general@hadoop.apache.org>; Tue, 13 Oct 2009 23:18:00 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.101.73.18 with SMTP id a18mr6888796anl.66.1255501079341; Tue, 
	13 Oct 2009 23:17:59 -0700 (PDT)
In-Reply-To: <314098690910132304s2619b2b4hba0c4a83b76a65e8@mail.gmail.com>
References: <E1MxJB1-0005FT-F2@st05.mi6.kiev.ua> <314098690910120843j36dc05b7kb13fb42a1e633211@mail.gmail.com> 
	<4ad4790d.1438560a.1db9.3612@mx.google.com> <8211a1320910130649y14b06a67s13b268f31f6f786e@mail.gmail.com> 
	<95f8b1a90910130905h38357b3fmd53c5a4b1c9ee5f8@mail.gmail.com> 
	<314098690910132304s2619b2b4hba0c4a83b76a65e8@mail.gmail.com>
From: Chandan Tamrakar <chandan.tamrakar@nepasoft.com>
Date: Wed, 14 Oct 2009 12:02:39 +0545
Message-ID: <95f8b1a90910132317h87df034nf59b8a3c69cc0b9c@mail.gmail.com>
Subject: Re: Reading files from local file system
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016368e1d58d96cad0475df1f4e

--0016368e1d58d96cad0475df1f4e
Content-Type: text/plain; charset=ISO-8859-1

Joson , i think it worked after changing these two paramters in
hadoop-site.xml

*  1. fs.default.name*
      used local file system as default file:///
  *2. **mapred.job.tracker
*      previously were using hdfs location and removed now*

   *Thanks
chandan
*
*
On Wed, Oct 14, 2009 at 11:49 AM, Jason Venner <jason.hadoop@gmail.com>wrote:

> If you want to open a local file in hadoop you have 3 simple ways
>
> 1: use file:///path
> 2: get a LocalFileSystem object from the FileSystem
> /**
>   * Get the local file syste
>   * @param conf the configuration to configure the file system with
>   * @return a LocalFileSystem
>   */
>  public static LocalFileSystem getLocal(Configuration conf)
>    throws IOException {
>    return (LocalFileSystem)get(LocalFileSystem.NAME, conf);
>  }
>
> 3: use the java.io File* classes.
>
> On Tue, Oct 13, 2009 at 9:05 AM, Chandan Tamrakar <
> chandan.tamrakar@nepasoft.com> wrote:
>
> > Do I need to change any configuration beside changing the default file
> > system to "local file system' ?
> > I am trying to input for example  input.txt to map job
> >
> > input.txt will contain file location as following
> >
> > file://path/abc1.doc
> > file://path/abc2.doc
> > ..
> > ...
> >
> > map program will read each line from input.txt and process them
> >
> > Do i need to change any configuration ? This is similar to how Nutch
> crawls
> > .
> >
> > any feedbacks would be appreciated
> >
> > thanks
> >
> >
> >
> > On Tue, Oct 13, 2009 at 6:49 AM, Jeff Zhang <zjffdu@gmail.com> wrote:
> >
> > > Maybe you could debug your mapreduce job in eclipse, since you run it
> in
> > > local mode.
> > >
> > >
> > >
> > > On Tue, Oct 13, 2009 at 5:56 AM, Chandan Tamrakar <
> > > chandan.tamrakar@nepasoft.com> wrote:
> > >
> > > >
> > > >
> > > > We are trying to read files from local file system. But when running
> > the
> > > > map
> > > > reduce it is not able to read files from the input location (the
> input
> > > > location is also local file system location).
> > > >
> > > > For this we changed the configuration of the hadoop-site.xml as shown
> > > > below:
> > > >
> > > > /etc/conf/hadoop/hadoop-site.xml
> > > >
> > > > <property>
> > > >    <name>fs.default.name</name>
> > > >    <value>file:///</value>
> > > >  </property>
> > > >
> > > >
> > > >  [admin@localhost ~]$ hadoop jar Test.jar /home/admin/input/test.txt
> > > > output1
> > > >
> > > > Suppose Test.txt is pain text file that contains
> > > > Test1
> > > > Test2
> > > > Test3
> > > >
> > > >
> > > > While running simple MapReduce job we get following exception  "File
> > not
> > > > found exception " , we are using TextInputFormat in our Job
> > configuration
> > > >
> > > >
> > > > 09/10/13 17:26:35 WARN mapred.JobClient: Use GenericOptionsParser for
> > > > parsing the arguments. Applications should implement Tool for the
> same.
> > > > 09/10/13 17:26:35 INFO mapred.FileInputFormat: Total input paths to
> > > process
> > > > : 1
> > > > 09/10/13 17:26:35 INFO mapred.FileInputFormat: Total input paths to
> > > process
> > > > : 1
> > > > 09/10/13 17:26:37 INFO mapred.JobClient: Running job:
> > > job_200910131447_0033
> > > > 09/10/13 17:26:38 INFO mapred.JobClient:  map 0% reduce 0%
> > > > 09/10/13 17:27:00 INFO mapred.JobClient: Task Id :
> > > > attempt_200910131447_0033_m_000000_0, Status : FAILED
> > > > java.io.FileNotFoundException: File
> > > file:/home/admin/Desktop/input/test.txt
> > > > does not exist.
> > > >        at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.jav
> > > > a:420)
> > > >        at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:25
> > > > 9)
> > > >        at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(Checks
> > > > umFileSystem.java:117)
> > > >        at
> > > >
> > org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:275)
> > > >        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:364)
> > > >        at
> > > >
> > >
> >
> org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:206)
> > > >        at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.jav
> > > > a:50)
> > > >        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:219)
> > > >        at
> > > >
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2210)
> > > >
> > > > However, running in the code as a separate Main method does work
> well.
> > > >
> > > > public static void main (String [] args) throws IOException {
> > > >
> > > >     Configuration conf = new Configuration();
> > > >     FileSystem fs = FileSystem.get(conf);
> > > >
> > > >     Path filenamePath = new Path(theFilename);
> > > >     FSDataOutputStream out = fs.create(new Path("abc.txt"));
> > > >     out.writeUTF("abc");
> > > >     out.close();
> > > >
> > > > }
> > > >
> > > > The above code works fine when running it as a jar in hadoop. The
> above
> > > > code
> > > > successfully creates file in /home/admin/abc.txt when running from
> > admin
> > > > user.
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Chandan Tamrakar
> >
>
>
>
> --
> Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> http://www.amazon.com/dp/1430219424?tag=jewlerymall
> www.prohadoopbook.com a community for Hadoop Professionals
>



-- 
Chandan Tamrakar

--0016368e1d58d96cad0475df1f4e--

From general-return-616-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 14 16:38:00 2009
Return-Path: <general-return-616-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27409 invoked from network); 14 Oct 2009 16:38:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Oct 2009 16:38:00 -0000
Received: (qmail 83290 invoked by uid 500); 14 Oct 2009 16:37:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83210 invoked by uid 500); 14 Oct 2009 16:37:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83199 invoked by uid 99); 14 Oct 2009 16:37:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Oct 2009 16:37:58 +0000
X-ASF-Spam-Status: No, hits=-10.6 required=5.0
	tests=BAYES_00,RCVD_IN_DNSWL_HI
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 14 Oct 2009 16:37:55 +0000
Received: (qmail 27227 invoked by uid 99); 14 Oct 2009 16:37:35 -0000
Received: from localhost.apache.org (HELO [192.168.168.112]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Oct 2009 16:37:35 +0000
Message-ID: <4AD5FE4E.4040100@apache.org>
Date: Wed, 14 Oct 2009 09:37:34 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: HTTP transport?
References: <C6FA7E7C.1906D%kan@yahoo-inc.com>
In-Reply-To: <C6FA7E7C.1906D%kan@yahoo-inc.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Kan Zhang wrote:
> One problem I see with using HTTP is that it's expensive to provide data
> encryption. We're currently adding 2 authentication mechanisms (Kerberos and
> DIGEST-MD5) to our existing RPC. Both of them can provide data encryption
> for subsequent communication over the authenticated channel. However, when
> similar authentication mechanisms are specified for HTTP (SPNEGO and HTTP
> DIGEST, respectively), they don't provide data encryption (correct me if I'm
> wrong). For data encryption over HTTP, one has to use SSL, which is
> expensive.

Java supports using Kerberos-based encryption for TLS (nee SSL):

http://java.sun.com/j2se/1.5.0/docs/guide/security/jsse/JSSERefGuide.html#KRB

http://tools.ietf.org/html/rfc2712

There's also a standard way to use tickets over TLS:

http://tools.ietf.org/html/rfc4507

Doug


From general-return-617-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 14 16:50:49 2009
Return-Path: <general-return-617-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30034 invoked from network); 14 Oct 2009 16:50:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Oct 2009 16:50:49 -0000
Received: (qmail 14662 invoked by uid 500); 14 Oct 2009 16:50:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14593 invoked by uid 500); 14 Oct 2009 16:50:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14581 invoked by uid 99); 14 Oct 2009 16:50:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Oct 2009 16:50:48 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00,NO_RDNS_DOTCOM_HELO
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Oct 2009 16:50:46 +0000
Received: from SP1-EX07CAS02.ds.corp.yahoo.com (sp1-ex07cas02.ds.corp.yahoo.com [216.252.116.138])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n9EGoMqV052261;
	Wed, 14 Oct 2009 09:50:22 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:references:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:
	content-transfer-encoding:mime-version;
	b=S5CpKy2eZ9eYCy/9aXvDkZ5Cj0hleRgs1FAxPac0S+dso2sbTnHJFahwgm+DQzlQ
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 SP1-EX07CAS02.ds.corp.yahoo.com ([216.252.116.167]) with mapi; Wed, 14 Oct
 2009 09:50:22 -0700
From: Dekel Tankel <dekel@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>,
        "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Date: Wed, 14 Oct 2009 09:50:21 -0700
Subject: RE: Hadoop User Group (Bay Area) - next Wednesday (Oct 21st) at
 Yahoo!
Thread-Topic: Hadoop User Group (Bay Area) - next Wednesday (Oct 21st) at
 Yahoo!
Thread-Index: AcoSMRRU8X5auVLKR8ufbpFTQi1hLgQXaiyQCpfO4pA=
Message-ID: <46E2672895E8744A97D7577A6110858B4FD39F6EDF@SP1-EX07VS01.ds.corp.yahoo.com>
References: <46E2672895E8744A97D7577A6110858B4E1AC69A91@SP1-EX07VS01.ds.corp.yahoo.com> 
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0


Hi all,

RSVP is still open for the next monthly Bay Area Hadoop user group at the Y=
ahoo! Sunnyvale Campus, next Wednesday (Oct 21st), 6PM

Registration and Agenda are available here
http://www.meetup.com/hadoop/calendar/11532125/

Looking forward to seeing you next week!

Dekel


From general-return-618-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 14 17:13:13 2009
Return-Path: <general-return-618-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37716 invoked from network); 14 Oct 2009 17:13:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Oct 2009 17:13:13 -0000
Received: (qmail 78109 invoked by uid 500); 14 Oct 2009 17:13:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78032 invoked by uid 500); 14 Oct 2009 17:13:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78022 invoked by uid 99); 14 Oct 2009 17:13:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Oct 2009 17:13:12 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 14 Oct 2009 17:13:10 +0000
Received: (qmail 37565 invoked by uid 99); 14 Oct 2009 17:12:48 -0000
Received: from localhost.apache.org (HELO [192.168.168.112]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Oct 2009 17:12:48 +0000
Message-ID: <4AD6068E.5030902@apache.org>
Date: Wed, 14 Oct 2009 10:12:46 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: avro-dev@hadoop.apache.org
CC: general@hadoop.apache.org
Subject: new Avro Committer: Thiruvalluvan M. G.
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

The Hadoop PMC has voted to make Thiruvalluvan M. G. an Avro committer.

Congratulations, Thiru!

Doug

From general-return-619-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 14 17:39:11 2009
Return-Path: <general-return-619-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45745 invoked from network); 14 Oct 2009 17:39:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Oct 2009 17:39:11 -0000
Received: (qmail 26756 invoked by uid 500); 14 Oct 2009 17:39:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26652 invoked by uid 500); 14 Oct 2009 17:39:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26629 invoked by uid 99); 14 Oct 2009 17:39:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Oct 2009 17:39:10 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Oct 2009 17:39:02 +0000
Received: from [10.73.152.71] (snvvpn2-10-73-152-c71.hq.corp.yahoo.com [10.73.152.71])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n9EHcRlf078911;
	Wed, 14 Oct 2009 10:38:28 -0700 (PDT)
Message-ID: <4AD60C93.9080005@apache.org>
Date: Wed, 14 Oct 2009 10:38:27 -0700
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: avro-dev@hadoop.apache.org
CC: general@hadoop.apache.org
Subject: Re: new Avro Committer: Thiruvalluvan M. G.
References: <4AD6068E.5030902@apache.org>
In-Reply-To: <4AD6068E.5030902@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Congrats Thiru!

Doug Cutting wrote:
> The Hadoop PMC has voted to make Thiruvalluvan M. G. an Avro committer.
> 
> Congratulations, Thiru!
> 
> Doug

From general-return-620-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 14 17:46:38 2009
Return-Path: <general-return-620-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47637 invoked from network); 14 Oct 2009 17:46:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Oct 2009 17:46:38 -0000
Received: (qmail 40188 invoked by uid 500); 14 Oct 2009 17:46:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40096 invoked by uid 500); 14 Oct 2009 17:46:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40079 invoked by uid 99); 14 Oct 2009 17:46:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Oct 2009 17:46:37 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Oct 2009 17:46:26 +0000
Received: from SNV-EXPF01.ds.corp.yahoo.com (snv-expf01.ds.corp.yahoo.com [207.126.227.250])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n9EHk21R083215
	for <general@hadoop.apache.org>; Wed, 14 Oct 2009 10:46:02 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:date:subject:from:to:message-id:
	thread-topic:thread-index:in-reply-to:mime-version:content-type:
	content-transfer-encoding:x-originalarrivaltime;
	b=IqBBOp+6ea+2J/84S0qYV7Mukhnij/mQLQwfjPA41ikqohFNVceWFL5sly66oZ4q
Received: from SNV-EXVS01.ds.corp.yahoo.com ([216.145.51.185]) by SNV-EXPF01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Wed, 14 Oct 2009 10:46:02 -0700
Received: from 10.72.106.176 ([10.72.106.176]) by SNV-EXVS01.ds.corp.yahoo.com ([216.145.51.166]) via Exchange Front-End Server snv-webmail.corp.yahoo.com ([207.126.227.248]) with Microsoft Exchange Server HTTP-DAV ;
 Wed, 14 Oct 2009 17:46:02 +0000
User-Agent: Microsoft-Entourage/12.20.0.090605
Date: Wed, 14 Oct 2009 10:45:57 -0700
Subject: Re: HTTP transport?
From: Kan Zhang <kan@yahoo-inc.com>
To: <general@hadoop.apache.org>
Message-ID: <C6FB5C65.19125%kan@yahoo-inc.com>
Thread-Topic: HTTP transport?
Thread-Index: AcpM9i8jd/kiWS0OlkWWGuv5+qimEw==
In-Reply-To: <4AD5FE4E.4040100@apache.org>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-OriginalArrivalTime: 14 Oct 2009 17:46:02.0311 (UTC) FILETIME=[324DDD70:01CA4CF6]
X-Virus-Checked: Checked by ClamAV on apache.org




On 10/14/09 9:37 AM, "Doug Cutting" <cutting@apache.org> wrote:

> Kan Zhang wrote:
>> One problem I see with using HTTP is that it's expensive to provide data
>> encryption. We're currently adding 2 authentication mechanisms (Kerberos and
>> DIGEST-MD5) to our existing RPC. Both of them can provide data encryption
>> for subsequent communication over the authenticated channel. However, when
>> similar authentication mechanisms are specified for HTTP (SPNEGO and HTTP
>> DIGEST, respectively), they don't provide data encryption (correct me if I'm
>> wrong). For data encryption over HTTP, one has to use SSL, which is
>> expensive.
> 
> Java supports using Kerberos-based encryption for TLS (nee SSL):
> 
> http://java.sun.com/j2se/1.5.0/docs/guide/security/jsse/JSSERefGuide.html#KRB
> 

This addresses part of my concern (the Kerberos part). I wasn't aware Java
already supports it. Thanks for pointing it out.

Kan


From general-return-621-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 15 05:19:26 2009
Return-Path: <general-return-621-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78231 invoked from network); 15 Oct 2009 05:19:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Oct 2009 05:19:26 -0000
Received: (qmail 93331 invoked by uid 500); 15 Oct 2009 05:19:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93244 invoked by uid 500); 15 Oct 2009 05:19:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93234 invoked by uid 99); 15 Oct 2009 05:19:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Oct 2009 05:19:24 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [67.195.15.180] (HELO web111409.mail.gq1.yahoo.com) (67.195.15.180)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 15 Oct 2009 05:19:20 +0000
Received: (qmail 53400 invoked by uid 60001); 15 Oct 2009 05:18:58 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1255583937; bh=RqskSXy4Vh7zh5cJRY+Lhsd0z0yyM//t38vtYu0h+lk=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=0wFPKRV336vievS0g/I6C+uDpdmw6gxD6ROZGl7hKfXnkXuPzTe6nKk5vsnCClJqrgzwg0MGLBopRTjDelcuoF+73bmB9aoiSra921yn9Su075btNmcYEZYrnh5fPbV8hlqIu4P7vCPoP3hpFpBFt6NUeG99Esx6Vtb8v3Ay43k=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=ZEq2nuwBx2s7r8DGmR0qrvChIVAHrqZZjPDuMAKoOBtz4YjUTrs8iQg03i4ga4EWanAHiaGbG0Vv/JQjc6NFXJNTNUsOUdBZDBPJd91SyClTjp+p6X4S8bJYubRDWZBaIwPOxTVKMARkaZ4tq1M5bz5RYarfS8n1N+dzwZRykqY=;
Message-ID: <967240.52565.qm@web111409.mail.gq1.yahoo.com>
X-YMail-OSG: KGvXRfwVM1mM3IkZGXwMPqZNLrHFUHpe_DrdD_pG3GgJNIfkLOHjm0tdurp2UdIHSj6TTvRoEv9tesSEYbOXXENnd5jFpzDa_kzv7swj6yKnPXHB9xiB6xNaSXRzdmUpFMtHIilF9Ls36V0O.WX5TDiy8Xhu5Ghvzo.o4pedM4IPkctDsIQjTqQTpc_1wsRUNZi35XEdnnTjZTGSjQSGVVJ0XSHS2YylXTz2Sh.VEGFWBPg_kxO6vy0oE4wTnB8.9UGhCMdSSERyYHua1GMFWZfeuGCN5GXlGTK0QBFY5VszksRNyf7oSfrrM669atBQd0RL404rC5vAYroAFBToSaw-
Received: from [24.130.194.117] by web111409.mail.gq1.yahoo.com via HTTP; Wed, 14 Oct 2009 22:18:57 PDT
X-Mailer: YahooMailClassic/7.0.14 YahooMailWebService/0.7.361.3
Date: Wed, 14 Oct 2009 22:18:57 -0700 (PDT)
From: Something Something <luckyguy2050@yahoo.com>
Subject: Question about MapReduce
To: general@hadoop.apache.org, hbase-user@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-977925582-1255583937=:52565"

--0-977925582-1255583937=:52565
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

I would like to start a Map-Reduce job that does not read data from an inpu=
t file or from a database. =A0I would like to pass 3 arguments to the Mappe=
r & Reducer to work on. =A0Basically, these arguments are keys on the 3 dif=
ferent tables on HBase.

In other words, I don't want to use FileInputFormat or DbInputFormat becaus=
e everything I need is already on HBase.=A0

How can I do this? =A0Please let me know. =A0Thanks.

=0A=0A=0A      
--0-977925582-1255583937=:52565--

From general-return-622-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 15 05:37:12 2009
Return-Path: <general-return-622-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82155 invoked from network); 15 Oct 2009 05:37:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Oct 2009 05:37:12 -0000
Received: (qmail 22081 invoked by uid 500); 15 Oct 2009 05:37:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22029 invoked by uid 500); 15 Oct 2009 05:37:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22019 invoked by uid 99); 15 Oct 2009 05:37:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Oct 2009 05:37:11 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [67.195.15.174] (HELO web111408.mail.gq1.yahoo.com) (67.195.15.174)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 15 Oct 2009 05:37:09 +0000
Received: (qmail 37746 invoked by uid 60001); 15 Oct 2009 05:36:47 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1255585007; bh=nABdlWToyFLJnK4GLib7Or57U27PWEckyNU3myQQdnE=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=L0USam2fo9CbNpl2SLRxCQWhjkWwCHq6SSXfNmaCoKBP6+UaD52ZBuCWzqdH5hc6IgEtAg8gG52P2zft/WzGeVuTlKacAi1PbpOkZNVGj/ykcjunuKzyl2vW3REWhrbcxr++wJc/65n2w0Jm09OvNHdiA7pOos43RcxzWHaFJuA=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=ZROh+vvhJcDAEaPp+32zQgTToKyKVnWNx2MVTTAtGCiYp+sJYqPeXMU/TY+w5ZAX6oVlzFiGZqKNMEzLkA4L772LK2+1V6zC0+XhxlKe2JkG0gdkSn2hU1mx/JoZI2J151KmF5Gu5RvZTelPR8FQfriXwGfQAQbEkX8mUtjrLcY=;
Message-ID: <805542.37510.qm@web111408.mail.gq1.yahoo.com>
X-YMail-OSG: NJEVop4VM1nKka.Sy0l5bkdOJRAGbHr4d.I8jLK0MP25OKFcrltwgkgDFUb94YhB9yyY9kQUVgjn7sggtNVlLp5GPpDn5jkH5fKd0ZYUbLNBHZaHY4bPtmCIsmgw0JVJBoM7H_z5gVPbCn_gaxsSxviOS5wZZM7Y2ozMYRIEpr8Y_OTIq0.a.lkG1Xtr4UmcqJryzfD6z2CfsOe4aX9USWDOKmDxmOwRs9UsJnuhpiXLnXFh7HYHkBcRlFRKU.s4_88p.wf0C2ngcTSLKfzUbzAQRDqY0ThXFbjnZWT00kfi8aFaAtoPTY_vj24HM_n8.hptyvApApqc5Wt3UxIn9s_9MT69wND7snGmOHp3Msc7EEEu4wQO.Iw-
Received: from [24.130.194.117] by web111408.mail.gq1.yahoo.com via HTTP; Wed, 14 Oct 2009 22:36:47 PDT
X-Mailer: YahooMailClassic/7.0.14 YahooMailWebService/0.7.361.3
Date: Wed, 14 Oct 2009 22:36:47 -0700 (PDT)
From: Something Something <luckyguy2050@yahoo.com>
Subject: Re: Question about MapReduce
To: general@hadoop.apache.org, hbase-user@hadoop.apache.org
In-Reply-To: <967240.52565.qm@web111409.mail.gq1.yahoo.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-6914967-1255585007=:37510"

--0-6914967-1255585007=:37510
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

If the answer is...

TableMapReduceUtil.initTableMapperJob

I apologize for the spam. =A0If this isn't the right way, please let me kno=
w. =A0Thanks.


--- On Wed, 10/14/09, Something Something <luckyguy2050@yahoo.com> wrote:

From: Something Something <luckyguy2050@yahoo.com>
Subject: Question about MapReduce
To: general@hadoop.apache.org, hbase-user@hadoop.apache.org
Date: Wednesday, October 14, 2009, 10:18 PM

I would like to start a Map-Reduce job that does not read data from an inpu=
t file or from a database. =A0I would like to pass 3 arguments to the Mappe=
r & Reducer to work on. =A0Basically, these arguments are keys on the 3 dif=
ferent tables on HBase.

In other words, I don't want to use FileInputFormat or DbInputFormat becaus=
e everything I need is already on HBase.=A0

How can I do this? =A0Please let me know. =A0Thanks.




=A0 =A0 =A0 =0A=0A=0A      
--0-6914967-1255585007=:37510--

From general-return-623-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 15 06:12:12 2009
Return-Path: <general-return-623-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 92481 invoked from network); 15 Oct 2009 06:12:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Oct 2009 06:12:12 -0000
Received: (qmail 50326 invoked by uid 500); 15 Oct 2009 06:12:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50252 invoked by uid 500); 15 Oct 2009 06:12:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50242 invoked by uid 99); 15 Oct 2009 06:12:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Oct 2009 06:12:11 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jason.hadoop@gmail.com designates 209.85.222.195 as permitted sender)
Received: from [209.85.222.195] (HELO mail-pz0-f195.google.com) (209.85.222.195)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Oct 2009 06:12:01 +0000
Received: by pzk33 with SMTP id 33so532043pzk.2
        for <general@hadoop.apache.org>; Wed, 14 Oct 2009 23:11:40 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=feQDrzGstUGR/hxKcMOv4TlhaUJVRv3/bJteAI6Myi4=;
        b=S1cXcBHeJbfbbC0X5tXmDMY5sEo0+1S4z2fWkYm7KXqyMXjlUZ50veX/HqtasREC3d
         ph/o3aVSVsvBZVo+B3mHhdBOctGKVuFDkC6ypzbTKfX/vmC7Ex9zvbtK1Vn19XljWmTO
         a55ITOYsw1qUqBDMzJxh1wtDtPzq9Q3AZ5kzM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=U7bg3K2/IBliTgWPQbeNUpt0GSE/FtY9m93Dme8yXUFi5RaobLdNG+mdyPPrneqz6j
         x3DgDFu3SLi2Pu86vN9QFjOKaCMlEIoceKPheFLgS4z08Cyi8vFnphgasZU6NZpDMkkx
         o4tAn5OPaMeY3ZK7VNzBFppPSjf12mR/iy4MQ=
MIME-Version: 1.0
Received: by 10.140.133.7 with SMTP id g7mr1060602rvd.169.1255587100330; Wed, 
	14 Oct 2009 23:11:40 -0700 (PDT)
In-Reply-To: <805542.37510.qm@web111408.mail.gq1.yahoo.com>
References: <967240.52565.qm@web111409.mail.gq1.yahoo.com>
	 <805542.37510.qm@web111408.mail.gq1.yahoo.com>
Date: Wed, 14 Oct 2009 23:11:40 -0700
Message-ID: <314098690910142311j1cb65dap4c048e91d7f0c58b@mail.gmail.com>
Subject: Re: Question about MapReduce
From: Jason Venner <jason.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd25310198cb40475f3271f
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd25310198cb40475f3271f
Content-Type: text/plain; charset=ISO-8859-1

I am doing something similar where I have a million input files to process,
I provide a single text file as input and use NLineInputFormat, specifying
the number of lines that are sent to each map task.

In your case you would set it so that 1 line was one input split, which is
the default.

In pre 20, the parameter is mapred.line.input.format.linespermap.

On Wed, Oct 14, 2009 at 10:36 PM, Something Something <
luckyguy2050@yahoo.com> wrote:

> If the answer is...
>
> TableMapReduceUtil.initTableMapperJob
>
> I apologize for the spam.  If this isn't the right way, please let me know.
>  Thanks.
>
>
> --- On Wed, 10/14/09, Something Something <luckyguy2050@yahoo.com> wrote:
>
> From: Something Something <luckyguy2050@yahoo.com>
> Subject: Question about MapReduce
> To: general@hadoop.apache.org, hbase-user@hadoop.apache.org
> Date: Wednesday, October 14, 2009, 10:18 PM
>
> I would like to start a Map-Reduce job that does not read data from an
> input file or from a database.  I would like to pass 3 arguments to the
> Mapper & Reducer to work on.  Basically, these arguments are keys on the 3
> different tables on HBase.
>
> In other words, I don't want to use FileInputFormat or DbInputFormat
> because everything I need is already on HBase.
>
> How can I do this?  Please let me know.  Thanks.
>
>
>
>
>
>
>
>
>



-- 
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals

--000e0cd25310198cb40475f3271f--

From general-return-624-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 15 18:31:32 2009
Return-Path: <general-return-624-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4669 invoked from network); 15 Oct 2009 18:31:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Oct 2009 18:31:32 -0000
Received: (qmail 61378 invoked by uid 500); 15 Oct 2009 18:31:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61308 invoked by uid 500); 15 Oct 2009 18:31:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61288 invoked by uid 99); 15 Oct 2009 18:31:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Oct 2009 18:31:31 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [67.195.15.204] (HELO web111413.mail.gq1.yahoo.com) (67.195.15.204)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 15 Oct 2009 18:31:21 +0000
Received: (qmail 50946 invoked by uid 60001); 15 Oct 2009 18:30:59 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1255631459; bh=u7/RQRPxKRrAAe86bLiqX8r7GaKMFA2R0YRw/nJCPAE=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=hojEtSmjKIxaMcXbUBh6tolMScU/TU0K1CcrNLhEkfgTubhFxGo5TDsRLH9QDh+nqIfRcmly9p5kceCInAm5zxesi48yP6rqhFH1it8xXPDA10RkPNorIsslkjnz7tG7kMwLfRWlu5WzVL0xkwjIn2g/PF261l2Go8A1j3oIGIg=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=s+jfusG8PPmNBFYvOyFIB5Kd/7HERF1yJ2R5eoaoUqR45dpWik4oH7wKprl4vm3MBLaS5IsHBZHR1b95v0E8Osi9fHsXn0S/mo414Fk5OuDjCX5diti82mcwBvhj5Y3bk7RqZO1zZTzcbLn0cvJ+dUmCrPhXeDIVnViw5KtF+0w=;
Message-ID: <297355.50875.qm@web111413.mail.gq1.yahoo.com>
X-YMail-OSG: _8rmYAEVM1lDfFcpUR5.PRBbgA3x2P6hezC_puh2wbaczjvjfg.Je2BqLoUN6B2o4gzxS1GX19RoaCf13mtTeQbIH1Xc4811XTXYSKvJzJrNhQ9xENLVmwHn_6ceJx4znpNdoqegUXcOHujBGQjyR_tO3LSSrzSUtXG7KpCrHp_qBO._73.NLB7T8DGyO9_ykCh7i_zE9MF6mNEoVq7MnX5lGH_6L9ZvhgssfAwOOCeQIm7S.qsZYFSb3lZHtU_7aymUrR8IHxCUvUUYoHcCu059kj4BpVdZf6oZQaX11NTGoA7ZiO8PmoRqE4I._9bOMkH3B9h2E2kmC1PxP3lrSxSRQGx8iwBqBWBI8X_r5c_UjwMRQqTyg5eZrsa_oq4-
Received: from [216.239.45.4] by web111413.mail.gq1.yahoo.com via HTTP; Thu, 15 Oct 2009 11:30:59 PDT
X-Mailer: YahooMailRC/182.10 YahooMailWebService/0.7.361.3
References: <967240.52565.qm@web111409.mail.gq1.yahoo.com> <805542.37510.qm@web111408.mail.gq1.yahoo.com> <5D66A842901F8E41815AF6D27A28EC490A7D6E07EF@Mail-Ab02.rmg-ny.com>
Date: Thu, 15 Oct 2009 11:30:59 -0700 (PDT)
From: Something Something <luckyguy2050@yahoo.com>
Subject: Re: Question about MapReduce
To: hbase-user@hadoop.apache.org, Hadoop <general@hadoop.apache.org>
In-Reply-To: <5D66A842901F8E41815AF6D27A28EC490A7D6E07EF@Mail-Ab02.rmg-ny.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-593399548-1255631459=:50875"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-593399548-1255631459=:50875
Content-Type: text/plain; charset=us-ascii

1) I don't think TableInputFormat is useful in this case.  Looks like it's used for scanning columns from a single HTable.
2) TableMapReduceUtil - same problem.  Seems like this works with just one table.
3) JV recommended NLineInputFormat, but my parameters are not in a file.  They come from multiple files and are in memory.

I guess what I am looking for is something like... InMemoryInputFormat... similar to FileInputFormat & DbInputFormat.  There's no such class right now.

Worse comes to worst, I can write the parameters into a flat file, and use FileInputFormat - but that will slow down this process considerably.  Is there no other way?





________________________________
From: Mark Vigeant <mark.vigeant@riskmetrics.com>
To: "hbase-user@hadoop.apache.org" <hbase-user@hadoop.apache.org>
Sent: Thu, October 15, 2009 7:21:40 AM
Subject: RE: Question about MapReduce

There is a tableInputFormat class in org.apache.hadoop.hbase.mapreduce.TableInputFormat

Also, if you want to use TableMapReduceUtil you probably want to have your mapper function extend TableMapper.

Check out the javadocs for more info: http://hadoop.apache.org/hbase/docs/current/api/index.html



-----Original Message-----
From: Something Something [mailto:luckyguy2050@yahoo.com] 
Sent: Thursday, October 15, 2009 1:37 AM
To: general@hadoop.apache.org; hbase-user@hadoop.apache.org
Subject: Re: Question about MapReduce

If the answer is...

TableMapReduceUtil.initTableMapperJob

I apologize for the spam.  If this isn't the right way, please let me know.  Thanks.


--- On Wed, 10/14/09, Something Something <luckyguy2050@yahoo.com> wrote:

From: Something Something <luckyguy2050@yahoo.com>
Subject: Question about MapReduce
To: general@hadoop.apache.org, hbase-user@hadoop.apache.org
Date: Wednesday, October 14, 2009, 10:18 PM

I would like to start a Map-Reduce job that does not read data from an input file or from a database.  I would like to pass 3 arguments to the Mapper & Reducer to work on.  Basically, these arguments are keys on the 3 different tables on HBase.

In other words, I don't want to use FileInputFormat or DbInputFormat because everything I need is already on HBase. 

How can I do this?  Please let me know.  Thanks.


      
--0-593399548-1255631459=:50875--

From general-return-625-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 15 21:21:10 2009
Return-Path: <general-return-625-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68487 invoked from network); 15 Oct 2009 21:21:10 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Oct 2009 21:21:10 -0000
Received: (qmail 77338 invoked by uid 500); 15 Oct 2009 21:21:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77256 invoked by uid 500); 15 Oct 2009 21:21:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77246 invoked by uid 99); 15 Oct 2009 21:21:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Oct 2009 21:21:09 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [67.195.15.144] (HELO web111403.mail.gq1.yahoo.com) (67.195.15.144)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 15 Oct 2009 21:21:07 +0000
Received: (qmail 41727 invoked by uid 60001); 15 Oct 2009 21:20:46 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1255641645; bh=ILc7oVm8JyvwmYA/ST32F8QjslGjZVoKk+tKrshz6DA=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=cAv1scPVEViW5DBJThuGBaNqpEohUfKX9QFvrM00zPCfdKNajPppHHaPG8PCANIY0Qq/yMivx7wB4Cb7b1zUn0+COoaSXYMpZy9yNG461+1H14Bcqs5qNOanjaFtNVrgD2MNV5EtILROVhy7NqzCmiiNvc8lWIAqlPWUjO1cWTQ=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=fJHtMwBFBsjZmn3MWM+F+TPObvld5oGAyBxa0sEBwuZ0Anp4kbtzQe7yIox9N5n97aP/NP3nKzw+0/La125FawwyLzZxa78zTpZBZszK6xflVl06/ub3OS/YhCbYcW8nj3kT7rgEvPaU4XqSFRojIhZANkFiUw5n+NYGmL1Pzkg=;
Message-ID: <984481.41505.qm@web111403.mail.gq1.yahoo.com>
X-YMail-OSG: Kbxl1WEVM1nyjdCA4aseBPzw83dMb02luWdEXfma9coitLAXlic8PIADcUBp8c7Jy24E383lzS634_4kSL26MdDK2k6scvYKDZRk2yoNffmzRgJn9i1gv24XMRDO9hwB37a4KJ3rejzVP5SOYXOp.uuIkR06BW4WVCIYjtfV7EZem0X_7SeCuBIkKozPBatzGAlLhDDPGvYwyzshC1dbN9i8QeXcPFfIM96oF.zVM5Gzbhx9gsZmw0mhGbwRh1TaFynzJZjIyXjIc2B4x2P6bGrJ1wvV39aC0fWoqcLwXw4U9pG7f1oJi4mzA.AjyRJ5d0SD6CU_z497fVNDHuJqjcSGSbiao6gfyWO4QLI_B3csvugGvlGzGzY_
Received: from [216.239.45.4] by web111403.mail.gq1.yahoo.com via HTTP; Thu, 15 Oct 2009 14:20:45 PDT
X-Mailer: YahooMailRC/182.10 YahooMailWebService/0.7.361.3
References: <967240.52565.qm@web111409.mail.gq1.yahoo.com>  <805542.37510.qm@web111408.mail.gq1.yahoo.com>  <5D66A842901F8E41815AF6D27A28EC490A7D6E07EF@Mail-Ab02.rmg-ny.com>  <297355.50875.qm@web111413.mail.gq1.yahoo.com> <f27b3fc0910151139v1a2b9b34m423e879464dddc51@mail.gmail.com>
Date: Thu, 15 Oct 2009 14:20:45 -0700 (PDT)
From: Something Something <luckyguy2050@yahoo.com>
Subject: Re: Question about MapReduce
To: hbase-user@hadoop.apache.org, general@hadoop.apache.org
In-Reply-To: <f27b3fc0910151139v1a2b9b34m423e879464dddc51@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1820193207-1255641645=:41505"

--0-1820193207-1255641645=:41505
Content-Type: text/plain; charset=us-ascii

I have 3 HTables.... Table1, Table2 & Table3.
I have 3 different flat files.  One contains keys for Table1, 2nd contains keys for Table2 & 3rd contains keys for Table3.

Use case:  For every combination of these 3 keys, I need to perform some complex calculation and save the result in another HTable.  In other words, I need to calculate values for the following combos:

(1,1,1) (1,1,2).......   (1,1,N) (1,2,1) (1,3,1) & so on....

So I figured the best way to do this is to start a MapReduce Job for each of these combinations.  The MapReduce will get (Key1, Key2, Key3) as input, then read Table1, Table2 & Table3 with these keys and perform the calculations.  Is this the correct approach?  If it is, I need to pass Key1, Key2 & Key3 to the Mapper & Reducer.  What's the best way to do this?

At this time, I don't need to join these tables in MapReduce, but in future I might have to.

Thanks.




________________________________
From: Kevin Peterson <kpeterson@biz360.com>
To: hbase-user@hadoop.apache.org
Sent: Thu, October 15, 2009 11:39:22 AM
Subject: Re: Question about MapReduce

On Thu, Oct 15, 2009 at 11:30 AM, Something Something <
luckyguy2050@yahoo.com> wrote:

> 1) I don't think TableInputFormat is useful in this case.  Looks like it's
> used for scanning columns from a single HTable.
> 2) TableMapReduceUtil - same problem.  Seems like this works with just one
> table.
> 3) JV recommended NLineInputFormat, but my parameters are not in a file.
>  They come from multiple files and are in memory.
>
> I guess what I am looking for is something like... InMemoryInputFormat...
> similar to FileInputFormat & DbInputFormat.  There's no such class right
> now.
>
> Worse comes to worst, I can write the parameters into a flat file, and use
> FileInputFormat - but that will slow down this process considerably.  Is
> there no other way?
>
> So you need to pull input from multiple tables at once? Are you expecting
to do a join on these tables? If you explain what the data looks like, we'd
understand better. What are your tables, and what would you like to treat as
a single input record?



      
--0-1820193207-1255641645=:41505--

From general-return-626-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 15 21:45:22 2009
Return-Path: <general-return-626-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78159 invoked from network); 15 Oct 2009 21:45:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Oct 2009 21:45:21 -0000
Received: (qmail 18251 invoked by uid 500); 15 Oct 2009 21:45:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18172 invoked by uid 500); 15 Oct 2009 21:45:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18162 invoked by uid 99); 15 Oct 2009 21:45:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Oct 2009 21:45:21 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 209.85.222.195 is neither permitted nor denied by domain of kpeterson@biz360.com)
Received: from [209.85.222.195] (HELO mail-pz0-f195.google.com) (209.85.222.195)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Oct 2009 21:45:18 +0000
Received: by pzk33 with SMTP id 33so1190724pzk.2
        for <general@hadoop.apache.org>; Thu, 15 Oct 2009 14:44:58 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.142.4.41 with SMTP id 41mr63053wfd.123.1255643098395; Thu, 15 
	Oct 2009 14:44:58 -0700 (PDT)
In-Reply-To: <984481.41505.qm@web111403.mail.gq1.yahoo.com>
References: <967240.52565.qm@web111409.mail.gq1.yahoo.com>
	 <805542.37510.qm@web111408.mail.gq1.yahoo.com>
	 <5D66A842901F8E41815AF6D27A28EC490A7D6E07EF@Mail-Ab02.rmg-ny.com>
	 <297355.50875.qm@web111413.mail.gq1.yahoo.com>
	 <f27b3fc0910151139v1a2b9b34m423e879464dddc51@mail.gmail.com>
	 <984481.41505.qm@web111403.mail.gq1.yahoo.com>
Date: Thu, 15 Oct 2009 14:44:58 -0700
Message-ID: <f27b3fc0910151444g25618089ob1939cad74b65e3a@mail.gmail.com>
Subject: Re: Question about MapReduce
From: Kevin Peterson <kpeterson@biz360.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502b443d835f604760030ee

--00504502b443d835f604760030ee
Content-Type: text/plain; charset=ISO-8859-1

On Thu, Oct 15, 2009 at 2:20 PM, Something Something <luckyguy2050@yahoo.com
> wrote:

> I have 3 HTables.... Table1, Table2 & Table3.
> I have 3 different flat files.  One contains keys for Table1, 2nd contains
> keys for Table2 & 3rd contains keys for Table3.
>
> Use case:  For every combination of these 3 keys, I need to perform some
> complex calculation and save the result in another HTable.  In other words,
> I need to calculate values for the following combos:
>
> (1,1,1) (1,1,2).......   (1,1,N) (1,2,1) (1,3,1) & so on....
>
> So I figured the best way to do this is to start a MapReduce Job for each
> of these combinations.  The MapReduce will get (Key1, Key2, Key3) as input,
> then read Table1, Table2 & Table3 with these keys and perform the
> calculations.  Is this the correct approach?  If it is, I need to pass Key1,
> Key2 & Key3 to the Mapper & Reducer.  What's the best way to do this?
>
> So you need the Cartesian product of all these files. My recommendation:

Run three jobs which each read one of these files and set a flag in the row
of the appropriate table. This way, you don't need the files at all, you
just read some "flag:active" column in the tables.

Next, pick one of the tables. It doesn't really matter which one from a
logical standpoint, you could say table1, you could pick the one with the
most data in it, or you may pick the one iwth the most individual entries
flagged. Use it as input to tableinputformat, with a filter that only passes
through those rows that are flagged.

In the mapper, create a scanner over each of the other two columns using the
same filter. You have two nested loops inside your map. In the innermost
loop, be sure to updated a counter or call progress() to avoid the
jobtracker timing out.

Use tableoutputformat from that job to write to your output table.

Depending on what exactly it means when you get a row key in your original
input files, the next time through you will likely need to go through and
clear all the flags before starting the process again.

You definitely will not be starting multiple map reduce jobs. You will have
one map reduce job that iterates through all the possible combinations, and
your goal needs to be to make sure that the task can be split up enough that
it can be parallelized.

--00504502b443d835f604760030ee--

From general-return-627-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 15 23:14:05 2009
Return-Path: <general-return-627-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 3113 invoked from network); 15 Oct 2009 23:14:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Oct 2009 23:14:05 -0000
Received: (qmail 28576 invoked by uid 500); 15 Oct 2009 23:14:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28507 invoked by uid 500); 15 Oct 2009 23:14:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28497 invoked by uid 99); 15 Oct 2009 23:14:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Oct 2009 23:14:04 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ryanobjc@gmail.com designates 209.85.211.185 as permitted sender)
Received: from [209.85.211.185] (HELO mail-yw0-f185.google.com) (209.85.211.185)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Oct 2009 23:13:56 +0000
Received: by ywh15 with SMTP id 15so1339867ywh.5
        for <general@hadoop.apache.org>; Thu, 15 Oct 2009 16:13:35 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=Zwd554yg3ysETcD0cSddit0Hglhk8eVK4vvMSORcO2U=;
        b=CJDa98CikV0abOWrsvnYtSTObDDfs7KhgWW7ON+5c6ITEdrRPy8RezRnnNfHzgSITY
         dF+p2umg+1jG/ftX08GeeQWAo00MycilQiYBvSlsGwEvHSxDs0yPHxEZZuyvi/casW9a
         K09KCk6gNYFdqhdGMHD/X0d8scLc2Ihi/Dm4g=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=j1AAE2fApF8GxreiDapnzCtEnDzWEUwdq7/dUlhQgracQGhvQ94wuyCXASnitqVhze
         GO+I8tpwZrON7MNif6QFe6n66FKMx1lewGlVKyS3sFYTL8OHjcUVugt7tCMFgG66C3TG
         zBSD73a0R27QGtwNp7eCJye07SPORCMdEB/gs=
MIME-Version: 1.0
Received: by 10.150.239.15 with SMTP id m15mr1446152ybh.109.1255648415603; 
	Thu, 15 Oct 2009 16:13:35 -0700 (PDT)
Date: Thu, 15 Oct 2009 16:13:35 -0700
Message-ID: <78568af10910151613k4742fe1dpb0a9028ac6229c2c@mail.gmail.com>
Subject: Hadoop 3 projects - how to reconcile the duplicated files?
From: Ryan Rawson <ryanobjc@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

While trying to bring up hadoop 0.21 from source, I noted that there
is a bunch of duplicate files between the projects.  The config
directory for example. Also hadoop-mapreduce seems to build
webapps/datanode for some reason (?!).

Someone on irc mentioned using 'ant tar', but it seems to be currently
broken, since it (a) requires forrest and (b) forrest fails to build
the docs on OSX 10.6.1, complaining about some datatype nonsense.

What is the new standard for building a distribution?

From general-return-628-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 16 00:32:19 2009
Return-Path: <general-return-628-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21039 invoked from network); 16 Oct 2009 00:32:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Oct 2009 00:32:19 -0000
Received: (qmail 1369 invoked by uid 500); 16 Oct 2009 00:32:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1309 invoked by uid 500); 16 Oct 2009 00:32:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1299 invoked by uid 99); 16 Oct 2009 00:32:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Oct 2009 00:32:18 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [67.195.15.192] (HELO web111411.mail.gq1.yahoo.com) (67.195.15.192)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 16 Oct 2009 00:32:09 +0000
Received: (qmail 52077 invoked by uid 60001); 16 Oct 2009 00:31:46 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1255653106; bh=ckpuDYO/kRdoyqYAoDvM85sQeEJV6/GxS4D2jIORqzQ=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=peHOJYdkXmQ8poLurZgte13sIUW+rtQshn796/b5QXhKKsEKPWoRPOdlwwuGg3Tpx2LW3F5JLGquPVZLNN1SMB4cHSH2UcrLjortXuD8bhpKr7QoArtPYqqWgHGg4/QTQtzsFJn25XfZ1gqx66lq4NtlAAcQjuAzRhwczYzFuFU=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=2kblX+DAgT7rut0CRYdfm+hILn94zH2vE3USDs0hnrVbNFTkGSnqvl4fJBnZVoagm/LBeK7L7pofWgVT6YUZ8WCj5D3qdj8He8m6WzobDnMWTLIA7Av06viZAo6WgUiJacmdctizIrJXCyXWzjvjbwrUILFvv5RJ0VQ9K8sjPgY=;
Message-ID: <515249.51532.qm@web111411.mail.gq1.yahoo.com>
X-YMail-OSG: X_ovcKEVM1nkxwQeVueK_cWXTYPcOqoT_JTY691O5DyhrCmcofmcyeO4Asu7UT6uITGMo__43KllgqJMMEP3CnS0YwDE9m37tD0mRmYuTqIm5.Mj2SqX6xPnS3thkD.9aEprWcgSFpry.935ISrgb1TTTAT8G.l.hHgo.aj40SD.YZI9D5Y18Wvi_tUaH355BA61fXibbtEa4rhPS36.5RDs64XeTr2ridOSfdh4o98sCbqulmLFEU0o8e9.zzkJfVc4EDtHmiDF1MjsmvDAWaY5KMPiqIxaOOZRP6WsHnSA8UAAFsP2MwmgNpG4qsHKxXNvFB36k_G1iHK4DWssSDyT.vprTCY6uIh4_VE7UzxGHmZdk_ATWtXc
Received: from [67.218.102.207] by web111411.mail.gq1.yahoo.com via HTTP; Thu, 15 Oct 2009 17:31:46 PDT
X-Mailer: YahooMailRC/182.10 YahooMailWebService/0.7.361.3
References: <967240.52565.qm@web111409.mail.gq1.yahoo.com>  <805542.37510.qm@web111408.mail.gq1.yahoo.com>  <5D66A842901F8E41815AF6D27A28EC490A7D6E07EF@Mail-Ab02.rmg-ny.com>  <297355.50875.qm@web111413.mail.gq1.yahoo.com>  <f27b3fc0910151139v1a2b9b34m423e879464dddc51@mail.gmail.com>  <984481.41505.qm@web111403.mail.gq1.yahoo.com> <f27b3fc0910151444g25618089ob1939cad74b65e3a@mail.gmail.com>
Date: Thu, 15 Oct 2009 17:31:46 -0700 (PDT)
From: Something Something <luckyguy2050@yahoo.com>
Subject: Re: Question about MapReduce
To: general@hadoop.apache.org, hbase-user@hadoop.apache.org
In-Reply-To: <f27b3fc0910151444g25618089ob1939cad74b65e3a@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-593719154-1255653106=:51532"
X-Virus-Checked: Checked by ClamAV on apache.org

--0-593719154-1255653106=:51532
Content-Type: text/plain; charset=us-ascii

Kevin - Interesting way to solve the problem, but I don't think this solution is bullet-proof.  While the MapReduce is running, someone may modify the "flag" and that will completely change the outcome - unless of course there's some way in HBase to lock this column.

Amandeep - I was hoping to do this without writing to a flat file since the data I need is already in memory.  Also, I am not sure what you mean by "Why are you using HBase?"  I am using it because that's where the data I need for calculations is.





________________________________
From: Kevin Peterson <kpeterson@biz360.com>
To: general@hadoop.apache.org
Sent: Thu, October 15, 2009 2:44:58 PM
Subject: Re: Question about MapReduce

On Thu, Oct 15, 2009 at 2:20 PM, Something Something <luckyguy2050@yahoo.com
> wrote:

> I have 3 HTables.... Table1, Table2 & Table3.
> I have 3 different flat files.  One contains keys for Table1, 2nd contains
> keys for Table2 & 3rd contains keys for Table3.
>
> Use case:  For every combination of these 3 keys, I need to perform some
> complex calculation and save the result in another HTable.  In other words,
> I need to calculate values for the following combos:
>
> (1,1,1) (1,1,2).......   (1,1,N) (1,2,1) (1,3,1) & so on....
>
> So I figured the best way to do this is to start a MapReduce Job for each
> of these combinations.  The MapReduce will get (Key1, Key2, Key3) as input,
> then read Table1, Table2 & Table3 with these keys and perform the
> calculations.  Is this the correct approach?  If it is, I need to pass Key1,
> Key2 & Key3 to the Mapper & Reducer.  What's the best way to do this?
>
> So you need the Cartesian product of all these files. My recommendation:

Run three jobs which each read one of these files and set a flag in the row
of the appropriate table. This way, you don't need the files at all, you
just read some "flag:active" column in the tables.

Next, pick one of the tables. It doesn't really matter which one from a
logical standpoint, you could say table1, you could pick the one with the
most data in it, or you may pick the one iwth the most individual entries
flagged. Use it as input to tableinputformat, with a filter that only passes
through those rows that are flagged.

In the mapper, create a scanner over each of the other two columns using the
same filter. You have two nested loops inside your map. In the innermost
loop, be sure to updated a counter or call progress() to avoid the
jobtracker timing out.

Use tableoutputformat from that job to write to your output table.

Depending on what exactly it means when you get a row key in your original
input files, the next time through you will likely need to go through and
clear all the flags before starting the process again.

You definitely will not be starting multiple map reduce jobs. You will have
one map reduce job that iterates through all the possible combinations, and
your goal needs to be to make sure that the task can be split up enough that
it can be parallelized.



      
--0-593719154-1255653106=:51532--

From general-return-629-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 16 13:26:35 2009
Return-Path: <general-return-629-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 457 invoked from network); 16 Oct 2009 13:26:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Oct 2009 13:26:35 -0000
Received: (qmail 59490 invoked by uid 500); 16 Oct 2009 13:26:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59395 invoked by uid 500); 16 Oct 2009 13:26:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59375 invoked by uid 99); 16 Oct 2009 13:26:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Oct 2009 13:26:34 +0000
X-ASF-Spam-Status: No, hits=-0.6 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [64.18.2.179] (HELO exprod7og113.obsmtp.com) (64.18.2.179)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Oct 2009 13:26:25 +0000
Received: from source ([209.85.223.188]) by exprod7ob113.postini.com ([64.18.6.12]) with SMTP
	ID DSNKSth0a4+flmWmlK6SJuVJO9Bg0q2HAOr2@postini.com; Fri, 16 Oct 2009 06:26:04 PDT
Received: by iwn26 with SMTP id 26so1221951iwn.5
        for <general@hadoop.apache.org>; Fri, 16 Oct 2009 06:26:03 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.42.150 with SMTP id s22mr2947477ibe.22.1255699563231; Fri, 
	16 Oct 2009 06:26:03 -0700 (PDT)
Date: Fri, 16 Oct 2009 16:26:03 +0300
Message-ID: <6eeff0ad0910160626u33505253xfb3a91c13267cc9@mail.gmail.com>
Subject: Hadoop and org.apache.hadoop.conf.Configuration class.
From: Vitaliy Avdeev <vavdeev@xasax.net>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001517741a14693b1004760d56c9
X-Virus-Checked: Checked by ClamAV on apache.org

--001517741a14693b1004760d56c9
Content-Type: text/plain; charset=ISO-8859-1

Hello. I have such situation. I neet to write gui on Java for editing
core-site.xml.
For getting an editing data I am using org.apache.hadoop.conf.Configuration
class.
But when i saving data from it such with writeXML() method atributes such as
<finaly>true</finaly> are lost.
Is org.apache.hadoop.conf.Configuration supports such atributes?

--001517741a14693b1004760d56c9--

From general-return-630-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 16 16:11:36 2009
Return-Path: <general-return-630-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 65867 invoked from network); 16 Oct 2009 16:11:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Oct 2009 16:11:35 -0000
Received: (qmail 1387 invoked by uid 500); 16 Oct 2009 16:11:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1315 invoked by uid 500); 16 Oct 2009 16:11:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1305 invoked by uid 99); 16 Oct 2009 16:11:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Oct 2009 16:11:34 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [209.85.222.195] (HELO mail-pz0-f195.google.com) (209.85.222.195)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Oct 2009 16:11:26 +0000
Received: by pzk33 with SMTP id 33so1855130pzk.2
        for <general@hadoop.apache.org>; Fri, 16 Oct 2009 09:11:05 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.114.2.12 with SMTP id 12mr1665097wab.52.1255709465120; Fri, 16 
	Oct 2009 09:11:05 -0700 (PDT)
In-Reply-To: <6eeff0ad0910160626u33505253xfb3a91c13267cc9@mail.gmail.com>
References: <6eeff0ad0910160626u33505253xfb3a91c13267cc9@mail.gmail.com>
From: Philip Zeyliger <philip@cloudera.com>
Date: Fri, 16 Oct 2009 09:10:45 -0700
Message-ID: <15da8a100910160910j68f2c485nbdda1ff8ebdce05d@mail.gmail.com>
Subject: Re: Hadoop and org.apache.hadoop.conf.Configuration class.
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

I think you meant to use "<final>", not "finaly".  See the "Final
Parameters" section of
http://hadoop.apache.org/common/docs/r0.20.1/api/org/apache/hadoop/conf/Configuration.html
.

On Fri, Oct 16, 2009 at 6:26 AM, Vitaliy Avdeev <vavdeev@xasax.net> wrote:
> Hello. I have such situation. I neet to write gui on Java for editing
> core-site.xml.
> For getting an editing data I am using org.apache.hadoop.conf.Configuration
> class.
> But when i saving data from it such with writeXML() method atributes such as
> <finaly>true</finaly> are lost.
> Is org.apache.hadoop.conf.Configuration supports such atributes?
>

From general-return-631-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 16 16:20:43 2009
Return-Path: <general-return-631-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70995 invoked from network); 16 Oct 2009 16:20:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Oct 2009 16:20:43 -0000
Received: (qmail 22136 invoked by uid 500); 16 Oct 2009 16:20:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22062 invoked by uid 500); 16 Oct 2009 16:20:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22052 invoked by uid 99); 16 Oct 2009 16:20:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Oct 2009 16:20:42 +0000
X-ASF-Spam-Status: No, hits=-5.4 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE,RCVD_IN_DNSWL_MED
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [64.18.2.179] (HELO exprod7og113.obsmtp.com) (64.18.2.179)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Oct 2009 16:20:40 +0000
Received: from source ([209.85.223.197]) by exprod7ob113.postini.com ([64.18.6.12]) with SMTP
	ID DSNKStidQ77TBDYU3q7UeUBES85LRK1TF9SI@postini.com; Fri, 16 Oct 2009 09:20:20 PDT
Received: by iwn35 with SMTP id 35so1254055iwn.26
        for <general@hadoop.apache.org>; Fri, 16 Oct 2009 09:20:19 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.117.215 with SMTP id s23mr5383277ibq.41.1255710018996; 
	Fri, 16 Oct 2009 09:20:18 -0700 (PDT)
In-Reply-To: <15da8a100910160910j68f2c485nbdda1ff8ebdce05d@mail.gmail.com>
References: <6eeff0ad0910160626u33505253xfb3a91c13267cc9@mail.gmail.com>
	 <15da8a100910160910j68f2c485nbdda1ff8ebdce05d@mail.gmail.com>
Date: Fri, 16 Oct 2009 19:20:18 +0300
Message-ID: <6eeff0ad0910160920y39ca3ea8k6e2e97f58e2e9ba3@mail.gmail.com>
Subject: Re: Hadoop and org.apache.hadoop.conf.Configuration class.
From: Vitaliy Avdeev <vavdeev@xasax.net>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636920ca99f8a4c04760fc589

--001636920ca99f8a4c04760fc589
Content-Type: text/plain; charset=ISO-8859-1

Yes I meant final.
But why when I am using  Configuration.writeXML(out) there is no  <final>
parameters in saved file?

On Fri, Oct 16, 2009 at 7:10 PM, Philip Zeyliger <philip@cloudera.com>wrote:

> I think you meant to use "<final>", not "finaly".  See the "Final
> Parameters" section of
>
> http://hadoop.apache.org/common/docs/r0.20.1/api/org/apache/hadoop/conf/Configuration.html
> .
>
> On Fri, Oct 16, 2009 at 6:26 AM, Vitaliy Avdeev <vavdeev@xasax.net> wrote:
> > Hello. I have such situation. I neet to write gui on Java for editing
> > core-site.xml.
> > For getting an editing data I am using
> org.apache.hadoop.conf.Configuration
> > class.
> > But when i saving data from it such with writeXML() method atributes such
> as
> > <finaly>true</finaly> are lost.
> > Is org.apache.hadoop.conf.Configuration supports such atributes?
> >
>

--001636920ca99f8a4c04760fc589--

From general-return-632-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 16 16:46:46 2009
Return-Path: <general-return-632-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 79394 invoked from network); 16 Oct 2009 16:46:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Oct 2009 16:46:46 -0000
Received: (qmail 54706 invoked by uid 500); 16 Oct 2009 16:46:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54618 invoked by uid 500); 16 Oct 2009 16:46:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54601 invoked by uid 99); 16 Oct 2009 16:46:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Oct 2009 16:46:45 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.216.204] (HELO mail-px0-f204.google.com) (209.85.216.204)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Oct 2009 16:46:42 +0000
Received: by pxi42 with SMTP id 42so1788893pxi.5
        for <general@hadoop.apache.org>; Fri, 16 Oct 2009 09:46:22 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.114.9.3 with SMTP id 3mr1691658wai.1.1255711581941; Fri, 16 
	Oct 2009 09:46:21 -0700 (PDT)
In-Reply-To: <6eeff0ad0910160920y39ca3ea8k6e2e97f58e2e9ba3@mail.gmail.com>
References: <6eeff0ad0910160626u33505253xfb3a91c13267cc9@mail.gmail.com> 
	<15da8a100910160910j68f2c485nbdda1ff8ebdce05d@mail.gmail.com> 
	<6eeff0ad0910160920y39ca3ea8k6e2e97f58e2e9ba3@mail.gmail.com>
From: Philip Zeyliger <philip@cloudera.com>
Date: Fri, 16 Oct 2009 09:44:33 -0700
Message-ID: <15da8a100910160944w6b0b4ee6g5b467eca6a819f6@mail.gmail.com>
Subject: Re: Hadoop and org.apache.hadoop.conf.Configuration class.
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I looked at the code: you're right in what you're
seeing---Configuration.writeXml doesn't write the final attribute.
It's mostly used for serialization a job's configuration when
necessary, and it doesn't need to preserve final in that context.

-- Philip

On Fri, Oct 16, 2009 at 9:20 AM, Vitaliy Avdeev <vavdeev@xasax.net> wrote:
> Yes I meant final.
> But why when I am using =A0Configuration.writeXML(out) there is no =A0<fi=
nal>
> parameters in saved file?
>
> On Fri, Oct 16, 2009 at 7:10 PM, Philip Zeyliger <philip@cloudera.com>wro=
te:
>
>> I think you meant to use "<final>", not "finaly". =A0See the "Final
>> Parameters" section of
>>
>> http://hadoop.apache.org/common/docs/r0.20.1/api/org/apache/hadoop/conf/=
Configuration.html
>> .
>>
>> On Fri, Oct 16, 2009 at 6:26 AM, Vitaliy Avdeev <vavdeev@xasax.net> wrot=
e:
>> > Hello. I have such situation. I neet to write gui on Java for editing
>> > core-site.xml.
>> > For getting an editing data I am using
>> org.apache.hadoop.conf.Configuration
>> > class.
>> > But when i saving data from it such with writeXML() method atributes s=
uch
>> as
>> > <finaly>true</finaly> are lost.
>> > Is org.apache.hadoop.conf.Configuration supports such atributes?
>> >
>>
>

From general-return-633-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 16 16:57:36 2009
Return-Path: <general-return-633-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82997 invoked from network); 16 Oct 2009 16:57:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Oct 2009 16:57:35 -0000
Received: (qmail 78967 invoked by uid 500); 16 Oct 2009 16:57:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78896 invoked by uid 500); 16 Oct 2009 16:57:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78886 invoked by uid 99); 16 Oct 2009 16:57:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Oct 2009 16:57:35 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Oct 2009 16:57:24 +0000
Received: from [192.168.1.71] (snvvpn3-10-72-76-c90.hq.corp.yahoo.com [10.72.76.90])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n9GGuYrq090825
	for <general@hadoop.apache.org>; Fri, 16 Oct 2009 09:56:34 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=mzmCL/vFQWFh9AYzYoakThAyebR8Tk3cFyVkRFgjmtz8riqRpC4V2uvdu4d4o1D4
Message-Id: <7AEF9819-E5D3-40EA-96BE-7724EC942C45@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <6eeff0ad0910160920y39ca3ea8k6e2e97f58e2e9ba3@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Hadoop and org.apache.hadoop.conf.Configuration class.
Date: Fri, 16 Oct 2009 09:56:34 -0700
References: <6eeff0ad0910160626u33505253xfb3a91c13267cc9@mail.gmail.com> <15da8a100910160910j68f2c485nbdda1ff8ebdce05d@mail.gmail.com> <6eeff0ad0910160920y39ca3ea8k6e2e97f58e2e9ba3@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


On Oct 16, 2009, at 9:20 AM, Vitaliy Avdeev wrote:

> Yes I meant final.
> But why when I am using  Configuration.writeXML(out) there is no   
> <final>
> parameters in saved file?

This was a conscious decision we made in HADOOP-785.

The idea is that a Configuration is serialized from the 'client  
context'. The client configuration is always loaded on 'top' of the  
configuration stack at runtime on the server (either at the JobTracker  
on job-submission or at the TaskTracker during task-execution) and  
thus the notion of 'final' parameters is moot.

Having said that you do have a valid cause for complaint; I've  
recently been involved in some discussions with the Pig team where we  
concluded that this feature could be useful during task localization  
on the TaskTracker i.e. the TaskTracker should set the 'localized'  
parameters of the task to be 'final'. The short end of the stick is  
I'd request you to open a jira asking for this enhancement... please?

thanks,
Arun


From general-return-634-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 16 17:00:17 2009
Return-Path: <general-return-634-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 84876 invoked from network); 16 Oct 2009 17:00:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Oct 2009 17:00:16 -0000
Received: (qmail 87512 invoked by uid 500); 16 Oct 2009 17:00:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87444 invoked by uid 500); 16 Oct 2009 17:00:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87434 invoked by uid 99); 16 Oct 2009 17:00:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Oct 2009 17:00:15 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Oct 2009 17:00:05 +0000
Received: from [192.168.1.71] (snvvpn3-10-72-76-c90.hq.corp.yahoo.com [10.72.76.90])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n9GGxXRN092378
	for <general@hadoop.apache.org>; Fri, 16 Oct 2009 09:59:33 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=ImhhgLOqnejqvkQxzGNWqUFbLEBy5KE1MrKjVXIF6odnjWm5vAszcONC37TS5GKV
Message-Id: <B4345659-2CB8-4D64-97FD-B99479EA433D@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <78568af10910151613k4742fe1dpb0a9028ac6229c2c@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Hadoop 3 projects - how to reconcile the duplicated files?
Date: Fri, 16 Oct 2009 09:59:34 -0700
References: <78568af10910151613k4742fe1dpb0a9028ac6229c2c@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

Ryan,

On Oct 15, 2009, at 4:13 PM, Ryan Rawson wrote:

> While trying to bring up hadoop 0.21 from source, I noted that there
> is a bunch of duplicate files between the projects.  The config
> directory for example. Also hadoop-mapreduce seems to build
> webapps/datanode for some reason (?!)

  I agree that there are still some rough edges with the project- 
split, something we are working to solve.

> Someone on irc mentioned using 'ant tar', but it seems to be currently
> broken, since it (a) requires forrest and (b) forrest fails to build
> the docs on OSX 10.6.1, complaining about some datatype nonsense.
>
> What is the new standard for building a distribution?

AFAIK the only issue with the 'ant docs' target (which 'ant target'  
relies on) is that you need java1.5 - if you can confirm this isn't  
the case would you mind opening a jira?

thanks,
Arun

From general-return-635-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 16 17:15:34 2009
Return-Path: <general-return-635-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 90170 invoked from network); 16 Oct 2009 17:15:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Oct 2009 17:15:33 -0000
Received: (qmail 13740 invoked by uid 500); 16 Oct 2009 17:15:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13653 invoked by uid 500); 16 Oct 2009 17:15:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13643 invoked by uid 99); 16 Oct 2009 17:15:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Oct 2009 17:15:32 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.85.222.195] (HELO mail-pz0-f195.google.com) (209.85.222.195)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Oct 2009 17:15:30 +0000
Received: by pzk33 with SMTP id 33so1902051pzk.2
        for <general@hadoop.apache.org>; Fri, 16 Oct 2009 10:15:09 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.115.80.6 with SMTP id h6mr1700294wal.108.1255713309838; Fri, 
	16 Oct 2009 10:15:09 -0700 (PDT)
In-Reply-To: <B4345659-2CB8-4D64-97FD-B99479EA433D@yahoo-inc.com>
References: <78568af10910151613k4742fe1dpb0a9028ac6229c2c@mail.gmail.com> 
	<B4345659-2CB8-4D64-97FD-B99479EA433D@yahoo-inc.com>
From: Philip Zeyliger <philip@cloudera.com>
Date: Fri, 16 Oct 2009 10:14:49 -0700
Message-ID: <15da8a100910161014vdd50480ub20e7f7ba14d9b19@mail.gmail.com>
Subject: Re: Hadoop 3 projects - how to reconcile the duplicated files?
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

> AFAIK the only issue with the 'ant docs' target (which 'ant target' relies
> on) is that you need java1.5 - if you can confirm this isn't the case would
> you mind opening a jira?

The magic you want is to add something like the following to your ant
command line.

-Dforrest.home=~/pub/apache-forrest-0.8/bin/forrest
-Dfindbugs.home=$HOME/pub/findbugs-1.3.8
-Djava5.home=/System/Library/Frameworks/JavaVM.framework/Versions/1.5/Home

and obviously you need to download findbugs and forrest themselves.
(Avro manages to download them via ivy; that might be a sensible
improvement.)

-- Philip

From general-return-636-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 16 17:37:17 2009
Return-Path: <general-return-636-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98300 invoked from network); 16 Oct 2009 17:37:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Oct 2009 17:37:16 -0000
Received: (qmail 54839 invoked by uid 500); 16 Oct 2009 17:37:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54793 invoked by uid 500); 16 Oct 2009 17:37:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 50602 invoked by uid 99); 13 Oct 2009 17:51:00 -0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <962B880C-EC03-40D7-AEFE-C0A6619D5BE8@umbc.edu>
References: <8E86E890-7D86-4CB0-A525-5CF056265717@umbc.edu> 
	<95f8b1a90910130912s2887e513ne68e5728e152dadb@mail.gmail.com> 
	<4BE1D299-6474-4159-9532-82B523024629@umbc.edu> <6e8dca540910130937r27557455h42feefcad01376c5@mail.gmail.com> 
	<E3347394-0567-4BC6-A613-FF835A44F861@umbc.edu> <962B880C-EC03-40D7-AEFE-C0A6619D5BE8@umbc.edu>
From: Todd Lipcon <todd@cloudera.com>
Date: Tue, 13 Oct 2009 10:50:13 -0700
Message-ID: <45f85f70910131050p50f11857o2984fc2d4cffd8b5@mail.gmail.com>
Subject: Re: 0.20.1 Cluster Setup Problem
To: common-user@hadoop.apache.org
Cc: general@hadoop.apache.org, Kevin Sweeney <kevin@yieldex.com>
Content-Type: multipart/alternative; boundary=0016e64714a4ce239f0475d4ae67

--0016e64714a4ce239f0475d4ae67
Content-Type: text/plain; charset=ISO-8859-1

Your issue was probably that slave_hadoop and master_hadoop are not valid
host names:

RFCs <http://en.wikipedia.org/wiki/Request_for_Comments> mandate that a
hostname's labels may contain only the
ASCII<http://en.wikipedia.org/wiki/ASCII>letters 'a' through 'z'
(case-insensitive), the digits '0' through '9', and
the hyphen. Hostname labels cannot begin or end with a hyphen. No other
symbols, punctuation characters, or blank spaces are permitted.

from http://en.wikipedia.org/wiki/Hostname

-Todd

On Tue, Oct 13, 2009 at 10:01 AM, Tejas Lagvankar <tej2@umbc.edu> wrote:

> Hey Kevin,
>
> You were right...
> I changed all my aliases to IP addresses. It worked !
>
> Thank you all again :)
>
> Regards,
> Tejas
>
>
> On Oct 13, 2009, at 12:41 PM, Tejas Lagvankar wrote:
>
>  By name resolution, I assume that you mean the name mentioned in
>> /etc/hosts.  Yes, in the logs, the IP address appears in the beginning.
>> Correct me if I'm wrong
>> I will also try with using just the IP's instead of the aliases.
>>
>> On Oct 13, 2009, at 12:37 PM, Kevin Sweeney wrote:
>>
>>  did you verify the name resolution?
>>>
>>> On Tue, Oct 13, 2009 at 4:34 PM, Tejas Lagvankar <tej2@umbc.edu> wrote:
>>>
>>> I get the same error even if i specify the port number. I have tried with
>>> port numbers 54310 as well as 9000.
>>>
>>>
>>> Regards,
>>> Tejas
>>>
>>>
>>> On Oct 13, 2009, at 12:12 PM, Chandan Tamrakar wrote:
>>>
>>> I think you need to specify the port as well for following port
>>>
>>> <property>
>>> <name>fs.default.name</name>
>>> <value>hdfs://master_hadoop</value>
>>> </property>
>>>
>>>
>>> On Tue, Oct 13, 2009 at 7:17 AM, Tejas Lagvankar <tej2@umbc.edu> wrote:
>>>
>>> Hi,
>>>
>>>
>>> We are trying to set up a cluster (starting with 2 machines) using the
>>> new
>>> 0.20.1 version.
>>>
>>> On the master machine, just after the server starts, the name node dies
>>> off
>>> with the following exception:
>>>
>>> 2009-10-13 01:22:24,740 ERROR
>>> org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException:
>>> Incomplete HDFS URI, no host: hdfs://master_hadoop
>>>    at
>>>
>>> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:78)
>>>    at
>>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1373)
>>>    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
>>>    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1385)
>>>    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
>>>    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
>>>    at org.apache.hadoop.fs.Trash.<init>(Trash.java:62)
>>>    at
>>>
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.startTrashEmptier(NameNode.java:208)
>>>    at
>>>
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:204)
>>>    at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279)
>>>    at
>>>
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
>>>    at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
>>>
>>> Can anyone help ?  Also can anyone send across example configuration
>>> files
>>> for 0.20.1 if they are different than we are using ?
>>>
>>> The detail log file is attached along with.
>>>
>>>
>>>
>>>
>>> The configuration files are as follows:
>>>
>>> MASTER CONFIG
>>> ------ conf/masters -------
>>> master_hadoop
>>>
>>> ------ conf/slaves -------
>>> master_hadoop
>>> slave_hadoop
>>>
>>> ------ core-site.xml -------
>>> <configuration>
>>>
>>> <property>
>>> <name>fs.default.name</name>
>>> <value>hdfs://master_hadoop</value>
>>> </property>
>>>
>>> <property>
>>> <name>hadoop.tmp.dir</name>
>>> <value>/opt/hadoop-0.20.1/tmp</value>
>>> </property>
>>>
>>> ------ hdfs-site.xml -------
>>> <property>
>>> <name>dfs.replication</name>
>>> <value>2</value>
>>> </property>
>>>
>>>
>>> ------ mapred-site.xml -------
>>> <property>
>>> <name>mapred.job.tracker</name>
>>> <value>tejas_hadoop:9001</value>
>>> </property>
>>>
>>>
>>>
>>>
>>>
>>> SLAVE CONFIG
>>> ------ core-site.xml -------
>>> <property>
>>> <name>hadoop.tmp.dir</name>
>>> <value>/opt/hadoop-0.20.1/tmp/</value>
>>> </property>
>>>
>>>
>>> <property>
>>> <name>fs.default.name</name>
>>> <value>hdfs://master_hadoop</value>
>>> </property>
>>>
>>>
>>> ------ hdfs-site.xml -------
>>> <property>
>>> <name>dfs.replication</name>
>>> <value>2</value>
>>> </property>
>>>
>>> ------ mapred-site.xml -------
>>> <property>
>>> <name>mapred.job.tracker</name>
>>> <value>tejas_hadoop:9001</value>
>>> </property>
>>>
>>>
>>>
>>> Regards,
>>>
>>> Tejas Lagvankar
>>> meettejas@umbc.edu
>>> www.umbc.edu/~tej2 <http://www.umbc.edu/%7Etej2> <
>>> http://www.umbc.edu/%7Etej2>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Chandan Tamrakar
>>>
>>> Tejas Lagvankar
>>> meettejas@umbc.edu
>>> www.umbc.edu/~tej2 <http://www.umbc.edu/%7Etej2>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>> Tejas Lagvankar
>> meettejas@umbc.edu
>> www.umbc.edu/~tej2 <http://www.umbc.edu/%7Etej2>
>>
>>
>>
>>
> Tejas Lagvankar
> meettejas@umbc.edu
> www.umbc.edu/~tej2 <http://www.umbc.edu/%7Etej2>
>
>
>
>

--0016e64714a4ce239f0475d4ae67--

From general-return-637-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 16 17:37:38 2009
Return-Path: <general-return-637-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98491 invoked from network); 16 Oct 2009 17:37:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Oct 2009 17:37:38 -0000
Received: (qmail 55834 invoked by uid 500); 16 Oct 2009 17:37:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55765 invoked by uid 500); 16 Oct 2009 17:37:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 13035 invoked by uid 99); 14 Oct 2009 17:32:07 -0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <4AD6068E.5030902@apache.org>
References: <4AD6068E.5030902@apache.org>
From: Matt Massie <matt@cloudera.com>
Date: Wed, 14 Oct 2009 10:31:25 -0700
Message-ID: <35538fbe0910141031h6b6d6fd3ycaa5e789da67659f@mail.gmail.com>
Subject: Re: new Avro Committer: Thiruvalluvan M. G.
To: avro-dev@hadoop.apache.org
Cc: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502b0ec69d73c0475e889c7

--00504502b0ec69d73c0475e889c7
Content-Type: text/plain; charset=ISO-8859-1

Congratulations, Thiru!

-Matt

On Wed, Oct 14, 2009 at 10:12 AM, Doug Cutting <cutting@apache.org> wrote:

> The Hadoop PMC has voted to make Thiruvalluvan M. G. an Avro committer.
>
> Congratulations, Thiru!
>
> Doug
>

--00504502b0ec69d73c0475e889c7--

From general-return-638-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 16 17:57:16 2009
Return-Path: <general-return-638-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6710 invoked from network); 16 Oct 2009 17:57:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Oct 2009 17:57:15 -0000
Received: (qmail 6707 invoked by uid 500); 16 Oct 2009 17:57:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6659 invoked by uid 500); 16 Oct 2009 17:57:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6647 invoked by uid 99); 16 Oct 2009 17:57:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Oct 2009 17:57:15 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Oct 2009 17:57:04 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n9GHufnt022246
	for <general@hadoop.apache.org>; Fri, 16 Oct 2009 10:56:41 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=Sl0w/q29rrbAwqow1etAiuaZ4bSOdzcX3UFCcaF7PMOLa3l9roVSbx2qunu5VolY
Message-Id: <01BE13FF-2ED2-45FD-9AE8-C45998B91077@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <7AEF9819-E5D3-40EA-96BE-7724EC942C45@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Hadoop and org.apache.hadoop.conf.Configuration class.
Date: Fri, 16 Oct 2009 10:56:40 -0700
References: <6eeff0ad0910160626u33505253xfb3a91c13267cc9@mail.gmail.com> <15da8a100910160910j68f2c485nbdda1ff8ebdce05d@mail.gmail.com> <6eeff0ad0910160920y39ca3ea8k6e2e97f58e2e9ba3@mail.gmail.com> <7AEF9819-E5D3-40EA-96BE-7724EC942C45@yahoo-inc.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

https://issues.apache.org/jira/browse/HADOOP-6317

Arun

On Oct 16, 2009, at 9:56 AM, Arun C Murthy wrote:

>
> On Oct 16, 2009, at 9:20 AM, Vitaliy Avdeev wrote:
>
>> Yes I meant final.
>> But why when I am using  Configuration.writeXML(out) there is no   
>> <final>
>> parameters in saved file?
>
> This was a conscious decision we made in HADOOP-785.
>
> The idea is that a Configuration is serialized from the 'client  
> context'. The client configuration is always loaded on 'top' of the  
> configuration stack at runtime on the server (either at the  
> JobTracker on job-submission or at the TaskTracker during task- 
> execution) and thus the notion of 'final' parameters is moot.
>
> Having said that you do have a valid cause for complaint; I've  
> recently been involved in some discussions with the Pig team where  
> we concluded that this feature could be useful during task  
> localization on the TaskTracker i.e. the TaskTracker should set the  
> 'localized' parameters of the task to be 'final'. The short end of  
> the stick is I'd request you to open a jira asking for this  
> enhancement... please?
>
> thanks,
> Arun
>


From general-return-639-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 16 20:19:29 2009
Return-Path: <general-return-639-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66860 invoked from network); 16 Oct 2009 20:19:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Oct 2009 20:19:29 -0000
Received: (qmail 37815 invoked by uid 500); 16 Oct 2009 20:19:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37737 invoked by uid 500); 16 Oct 2009 20:19:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37725 invoked by uid 99); 16 Oct 2009 20:19:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Oct 2009 20:19:28 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ryanobjc@gmail.com designates 209.85.217.228 as permitted sender)
Received: from [209.85.217.228] (HELO mail-gx0-f228.google.com) (209.85.217.228)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Oct 2009 20:19:16 +0000
Received: by gxk28 with SMTP id 28so2130551gxk.9
        for <general@hadoop.apache.org>; Fri, 16 Oct 2009 13:18:55 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=ldy5Ukhs6VlQz5NzDpZNX8YvTdPjLkBnd72vua//4rc=;
        b=axdXWacO6jCAWbklfZs4dnpgjrZumHKTRTxUNoxIOZlnfHltspkOOS7ecIhlYY8cMh
         91uYVE2u7EGLs+7Bcbb1ExIt5TDph37TPTYNKR/0s2kXzqx4Y7i/fzCdslpPkGq9jU25
         W07kl68+BEOirc8xRBijEASUrYSyrmT3fWpIM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=VOx07ZxVgm2Fi2WLl+nQgxmNWm39VwYYLdBev6dY39oXdjJMBrlJWCSLkC0gRpOzd6
         vDm+Kkwseh9eCHNiDsMDV5TSyk/2Hft5sXNMq1u3uQeOPYziSc8fiij475HrzUAXbkIb
         0zwUoDd6JUas17pkRKY+Hk9muJvLr6EsWBqjw=
MIME-Version: 1.0
Received: by 10.150.113.19 with SMTP id l19mr3529503ybc.239.1255724335603; 
	Fri, 16 Oct 2009 13:18:55 -0700 (PDT)
In-Reply-To: <15da8a100910161014vdd50480ub20e7f7ba14d9b19@mail.gmail.com>
References: <78568af10910151613k4742fe1dpb0a9028ac6229c2c@mail.gmail.com>
	 <B4345659-2CB8-4D64-97FD-B99479EA433D@yahoo-inc.com>
	 <15da8a100910161014vdd50480ub20e7f7ba14d9b19@mail.gmail.com>
Date: Fri, 16 Oct 2009 13:18:55 -0700
Message-ID: <78568af10910161318k35a17170v91776a8e06c822dd@mail.gmail.com>
Subject: Re: Hadoop 3 projects - how to reconcile the duplicated files?
From: Ryan Rawson <ryanobjc@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks for the reply,

I'm currently using git://github.com/toddlipcon/hadoop-meta.git to
help me build things... it's tricky because there are 3 projects and
you seem to have to manually carry the dependent jars around between
projects.

There is still the problem that the docs target wont build on OSX
10.6.1, but I'm sure you guys know about that one.

-ryan

On Fri, Oct 16, 2009 at 10:14 AM, Philip Zeyliger <philip@cloudera.com> wrote:
>> AFAIK the only issue with the 'ant docs' target (which 'ant target' relies
>> on) is that you need java1.5 - if you can confirm this isn't the case would
>> you mind opening a jira?
>
> The magic you want is to add something like the following to your ant
> command line.
>
> -Dforrest.home=~/pub/apache-forrest-0.8/bin/forrest
> -Dfindbugs.home=$HOME/pub/findbugs-1.3.8
> -Djava5.home=/System/Library/Frameworks/JavaVM.framework/Versions/1.5/Home
>
> and obviously you need to download findbugs and forrest themselves.
> (Avro manages to download them via ivy; that might be a sensible
> improvement.)
>
> -- Philip
>

From general-return-640-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 19 11:14:49 2009
Return-Path: <general-return-640-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14635 invoked from network); 19 Oct 2009 11:14:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Oct 2009 11:14:48 -0000
Received: (qmail 23289 invoked by uid 500); 19 Oct 2009 11:14:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23201 invoked by uid 500); 19 Oct 2009 11:14:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23191 invoked by uid 99); 19 Oct 2009 11:14:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Oct 2009 11:14:47 +0000
X-ASF-Spam-Status: No, hits=-1.7 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of luis.junges@gmail.com designates 209.85.211.185 as permitted sender)
Received: from [209.85.211.185] (HELO mail-yw0-f185.google.com) (209.85.211.185)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Oct 2009 11:14:45 +0000
Received: by ywh15 with SMTP id 15so3743731ywh.5
        for <general@hadoop.apache.org>; Mon, 19 Oct 2009 04:14:24 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=/VEoxe6YU16+w8vxJQUAdvC7lSku4N+WBeyGwRExq68=;
        b=YgfZ4hD4v/cnY31Dq/6xlEQrTWkhtYxbaEv4KAPo/dOUk5SB8Rwo2I7+4c0kddKsp4
         8dmioJKZJ6r02g8SRif+huFqhM+QSjaQMicpNLeDUmn1xEx+o+42Sc13XmOlb41WpjLT
         SoQzy/tRLXSmbF4tiKw9Oif+o33RDvszfAxxI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=VkYQhXNAvo1yVQaMcGATviusjVSrcqAu7vDlaLTV40KAlcf+E9nhI9nDfDUOFSZZyU
         DKai/WFoC+gMYboayHzrZqsJw+dB6LhKqG2x5L6Ay7GXL11Me5hAJ9n7oe0/HLcl9qFR
         O/ax1nRzsq+qsZXCuQWql8klghiaVV4UGki4E=
MIME-Version: 1.0
Received: by 10.101.57.14 with SMTP id j14mr3086101ank.105.1255950864621; Mon, 
	19 Oct 2009 04:14:24 -0700 (PDT)
Date: Mon, 19 Oct 2009 09:14:24 -0200
Message-ID: <752a037e0910190414j5f36de24g48e8f92019504a53@mail.gmail.com>
Subject: Store Large files/images HBase
From: Luis Carlos Junges <luis.junges@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636d34f702420c5047647d9ae

--001636d34f702420c5047647d9ae
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi,

I am currently doing some research on distributed database that can be
scaled easily in terms of storage capacity.

The reason is to use it on the brazilian federal project called
"portal do aluno" wich will have around 10 million kids accessing it
monthly. The idea is to build a portal similar to facebook/orkut with
the main objective to spread knowledge amoung kids (6 -13 years old).

well, now the problem:

Those kids will generate a lot of data which include photos, videos,
presentations, school tasks among others. In order to have a 100%
available system and also to scale this amount of data (initial
estimative is 10  TB at the full use of the portal), a distributed
storage engine seems to be the solution.

On the avialable solutions, i liked voldemort because it seems not to have =
a
SPOF (single point of
failure) when compared to HBase. However HBase seems to integrate with more
tools and sub-projects.

my question is concerned to the fact of storing such big items (2 MB
photo for example) with HBase. I read on on blogs that HBase has a high
latency which leads it to
be inappropriate to serve dynamic pages. Will the performance of HBase
decrease even more if large binary objects are stored on it?

Other question i have is related to the fact of modelling the data
using key/value pattern. With relational database it is just follow cake
recipe and it=B4s done. Do we have such recipe for key/value? Currently
a lot of code was done with relational database postgreSQL using
hibernate to mapping the objects.


i will appreciate any comments




--=20
"A realidade de cada lugar e de cada =E9poca =E9 uma alucina=E7=E3o coletiv=
a."


Bloom, Howard

--001636d34f702420c5047647d9ae--

From general-return-641-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 19 14:34:25 2009
Return-Path: <general-return-641-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 74190 invoked from network); 19 Oct 2009 14:34:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Oct 2009 14:34:25 -0000
Received: (qmail 38710 invoked by uid 500); 19 Oct 2009 14:34:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38601 invoked by uid 500); 19 Oct 2009 14:34:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 38554 invoked by uid 99); 19 Oct 2009 14:34:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Oct 2009 14:34:23 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.18.222.49] (HELO smtp3.4emm.com) (69.18.222.49)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Oct 2009 14:34:15 +0000
Received: from EX2K7VS03.4emm.local ([192.168.160.203]) by HUB03.4emm.local
 ([192.168.161.134]) with mapi; Mon, 19 Oct 2009 10:32:50 -0400
From: Doug Meil <doug.meil@explorysmedical.com>
To: "hbase-user@hadoop.apache.org" <hbase-user@hadoop.apache.org>,
	"general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 19 Oct 2009 10:34:03 -0400
Subject: RE: Question about MapReduce
Thread-Topic: Question about MapReduce
Thread-Index: AcpNxZGJRYqvqRyNRc+UbNEJB+LiywDAxldA
Message-ID: <67680900F79B1D4F99C844EE386FC5950196E34A15@EX2K7VS03.4emm.local>
References: <967240.52565.qm@web111409.mail.gq1.yahoo.com>
 <805542.37510.qm@web111408.mail.gq1.yahoo.com>
 <5D66A842901F8E41815AF6D27A28EC490A7D6E07EF@Mail-Ab02.rmg-ny.com>
 <297355.50875.qm@web111413.mail.gq1.yahoo.com>
In-Reply-To: <297355.50875.qm@web111413.mail.gq1.yahoo.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Hi there-

I didn't see the option in the thread yet which seems pretty straightforwar=
d:

When setting up the job:

    Job job =3D new Job(conf, "my job");

...
	   =20
    conf.setStrings("param", "param1");


And then in the map method:

      String paramVal =3D context.getConfiguration().get("param");


This is using Hadoop .20 syntax, the previous verison had a 'configure' met=
hod you had to implement.



-----Original Message-----
From: Something Something [mailto:luckyguy2050@yahoo.com]=20
Sent: Thursday, October 15, 2009 2:31 PM
To: hbase-user@hadoop.apache.org; Hadoop
Subject: Re: Question about MapReduce

1) I don't think TableInputFormat is useful in this case.  Looks like it's =
used for scanning columns from a single HTable.
2) TableMapReduceUtil - same problem.  Seems like this works with just one =
table.
3) JV recommended NLineInputFormat, but my parameters are not in a file.  T=
hey come from multiple files and are in memory.

I guess what I am looking for is something like... InMemoryInputFormat... s=
imilar to FileInputFormat & DbInputFormat.  There's no such class right now=
.

Worse comes to worst, I can write the parameters into a flat file, and use =
FileInputFormat - but that will slow down this process considerably.  Is th=
ere no other way?





________________________________
From: Mark Vigeant <mark.vigeant@riskmetrics.com>
To: "hbase-user@hadoop.apache.org" <hbase-user@hadoop.apache.org>
Sent: Thu, October 15, 2009 7:21:40 AM
Subject: RE: Question about MapReduce

There is a tableInputFormat class in org.apache.hadoop.hbase.mapreduce.Tabl=
eInputFormat

Also, if you want to use TableMapReduceUtil you probably want to have your =
mapper function extend TableMapper.

Check out the javadocs for more info: http://hadoop.apache.org/hbase/docs/c=
urrent/api/index.html



-----Original Message-----
From: Something Something [mailto:luckyguy2050@yahoo.com]=20
Sent: Thursday, October 15, 2009 1:37 AM
To: general@hadoop.apache.org; hbase-user@hadoop.apache.org
Subject: Re: Question about MapReduce

If the answer is...

TableMapReduceUtil.initTableMapperJob

I apologize for the spam.  If this isn't the right way, please let me know.=
  Thanks.


--- On Wed, 10/14/09, Something Something <luckyguy2050@yahoo.com> wrote:

From: Something Something <luckyguy2050@yahoo.com>
Subject: Question about MapReduce
To: general@hadoop.apache.org, hbase-user@hadoop.apache.org
Date: Wednesday, October 14, 2009, 10:18 PM

I would like to start a Map-Reduce job that does not read data from an inpu=
t file or from a database.  I would like to pass 3 arguments to the Mapper =
& Reducer to work on.  Basically, these arguments are keys on the 3 differe=
nt tables on HBase.

In other words, I don't want to use FileInputFormat or DbInputFormat becaus=
e everything I need is already on HBase.=20

How can I do this?  Please let me know.  Thanks.


     =20

From general-return-642-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 19 15:23:54 2009
Return-Path: <general-return-642-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97374 invoked from network); 19 Oct 2009 15:23:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Oct 2009 15:23:54 -0000
Received: (qmail 48366 invoked by uid 500); 19 Oct 2009 15:23:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48294 invoked by uid 500); 19 Oct 2009 15:23:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48284 invoked by uid 99); 19 Oct 2009 15:23:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Oct 2009 15:23:53 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [67.195.15.168] (HELO web111407.mail.gq1.yahoo.com) (67.195.15.168)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 19 Oct 2009 15:23:51 +0000
Received: (qmail 66653 invoked by uid 60001); 19 Oct 2009 15:23:29 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1255965809; bh=c0EIVqG5V1jforZsnaYI2tASi401jREGjH6zYFcPK9I=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=myzqWxy0Mz+o7xFQv0PUPzbuCROjutnxtsyKVs6J+hunXOIPGMPPq8VfKA4L9WlpHddkxLIUW3qXx0Db2w6VbbplFfKjc/wqE/kEAB7oi7URzSpcCGsWrqIznBs11Rr3HbHRgozxoghGld9hHkzqTFwVTmrzE4/Te1kgSOrdZhg=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=3sHBXuNEthTUQnqCjJg5Mda7VhzFWXoRJfb8T3wY5/pQckVspieii+n9VdvAYTAOdPr4A1iwSmbBtWXBrgmisj/lhdTDMPkXb2OSsw7uhWPladVohA+76/uyFY9ITHlKNmPYAn0M9jUX6u0RpL1QkzpZrgxnKCPYqkkPSiMpaNw=;
Message-ID: <797590.66623.qm@web111407.mail.gq1.yahoo.com>
X-YMail-OSG: kCv3R.EVM1m_bBwDvhY7hJFZJZ63BCl2a_miwLF4QF3syGNE2QdTkXmNx8H1yjmEhGyi6ICO15VVPVwtFVTVlVmaGGtIMl4swiQf2GSjVWykGANbzG1Cx6UkmPtUzUJPgLwYmkk9ezITohgkny0ldhoLz0FUjM6tHK3yIEIB2JZiRongTkyhV12z8KtVs.qay2IVyz_PNdbdbrEJjwO8AfPxFLB6ywjwDIY8CTYTCa9pRnVrWQdZfyvYHKYFwzwLaoV.7Z0FaR3ZeAKIhOY.JVV_CJLyxVNZ5aZ8HBx0igPoEn1YQNEm0323uQj1_6ZEGp2DaKMB4iD9UyaJc3Cx7gUIDFnnjzwB5DNIEhLkTSYleh66gs5Dq3xRc46qWuM-
Received: from [67.218.106.224] by web111407.mail.gq1.yahoo.com via HTTP; Mon, 19 Oct 2009 08:23:29 PDT
X-Mailer: YahooMailRC/182.10 YahooMailWebService/0.7.361.3
References: <967240.52565.qm@web111409.mail.gq1.yahoo.com> <805542.37510.qm@web111408.mail.gq1.yahoo.com> <5D66A842901F8E41815AF6D27A28EC490A7D6E07EF@Mail-Ab02.rmg-ny.com> <297355.50875.qm@web111413.mail.gq1.yahoo.com> <67680900F79B1D4F99C844EE386FC5950196E34A15@EX2K7VS03.4emm.local>
Date: Mon, 19 Oct 2009 08:23:29 -0700 (PDT)
From: Something Something <luckyguy2050@yahoo.com>
Subject: Re: Question about MapReduce
To: general@hadoop.apache.org,
  "hbase-user@hadoop.apache.org" <hbase-user@hadoop.apache.org>
In-Reply-To: <67680900F79B1D4F99C844EE386FC5950196E34A15@EX2K7VS03.4emm.local>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1467626233-1255965809=:66623"

--0-1467626233-1255965809=:66623
Content-Type: text/plain; charset=us-ascii

Interesting... I haven't tried this yet.. but in this case what would you specify as the 'InputPath'?  I was under the assumption that a Job needs some kind of InputPath.. no?  I don't see a NullInputPath.  Is there something equivalent?



________________________________
From: Doug Meil <doug.meil@explorysmedical.com>
To: "hbase-user@hadoop.apache.org" <hbase-user@hadoop.apache.org>; "general@hadoop.apache.org" <general@hadoop.apache.org>
Sent: Mon, October 19, 2009 7:34:03 AM
Subject: RE: Question about MapReduce

Hi there-

I didn't see the option in the thread yet which seems pretty straightforward:

When setting up the job:

    Job job = new Job(conf, "my job");

...
        
    conf.setStrings("param", "param1");


And then in the map method:

      String paramVal = context.getConfiguration().get("param");


This is using Hadoop .20 syntax, the previous verison had a 'configure' method you had to implement.



-----Original Message-----
From: Something Something [mailto:luckyguy2050@yahoo.com] 
Sent: Thursday, October 15, 2009 2:31 PM
To: hbase-user@hadoop.apache.org; Hadoop
Subject: Re: Question about MapReduce

1) I don't think TableInputFormat is useful in this case.  Looks like it's used for scanning columns from a single HTable.
2) TableMapReduceUtil - same problem.  Seems like this works with just one table.
3) JV recommended NLineInputFormat, but my parameters are not in a file.  They come from multiple files and are in memory.

I guess what I am looking for is something like... InMemoryInputFormat... similar to FileInputFormat & DbInputFormat.  There's no such class right now.

Worse comes to worst, I can write the parameters into a flat file, and use FileInputFormat - but that will slow down this process considerably.  Is there no other way?





________________________________
From: Mark Vigeant <mark.vigeant@riskmetrics.com>
To: "hbase-user@hadoop.apache.org" <hbase-user@hadoop.apache.org>
Sent: Thu, October 15, 2009 7:21:40 AM
Subject: RE: Question about MapReduce

There is a tableInputFormat class in org.apache.hadoop.hbase.mapreduce.TableInputFormat

Also, if you want to use TableMapReduceUtil you probably want to have your mapper function extend TableMapper.

Check out the javadocs for more info: http://hadoop.apache.org/hbase/docs/current/api/index.html



-----Original Message-----
From: Something Something [mailto:luckyguy2050@yahoo.com] 
Sent: Thursday, October 15, 2009 1:37 AM
To: general@hadoop.apache.org; hbase-user@hadoop.apache.org
Subject: Re: Question about MapReduce

If the answer is...

TableMapReduceUtil.initTableMapperJob

I apologize for the spam.  If this isn't the right way, please let me know.  Thanks.


--- On Wed, 10/14/09, Something Something <luckyguy2050@yahoo.com> wrote:

From: Something Something <luckyguy2050@yahoo.com>
Subject: Question about MapReduce
To: general@hadoop.apache.org, hbase-user@hadoop.apache.org
Date: Wednesday, October 14, 2009, 10:18 PM

I would like to start a Map-Reduce job that does not read data from an input file or from a database.  I would like to pass 3 arguments to the Mapper & Reducer to work on.  Basically, these arguments are keys on the 3 different tables on HBase.

In other words, I don't want to use FileInputFormat or DbInputFormat because everything I need is already on HBase. 

How can I do this?  Please let me know.  Thanks.


      
--0-1467626233-1255965809=:66623--

From general-return-643-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 19 16:52:01 2009
Return-Path: <general-return-643-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 32427 invoked from network); 19 Oct 2009 16:52:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Oct 2009 16:52:00 -0000
Received: (qmail 89951 invoked by uid 500); 19 Oct 2009 16:51:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89841 invoked by uid 500); 19 Oct 2009 16:51:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89822 invoked by uid 99); 19 Oct 2009 16:51:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Oct 2009 16:51:59 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.18.222.49] (HELO smtp3.4emm.com) (69.18.222.49)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Oct 2009 16:51:56 +0000
Received: from EX2K7VS03.4emm.local ([192.168.160.203]) by HUB03.4emm.local
 ([192.168.161.134]) with mapi; Mon, 19 Oct 2009 12:50:31 -0400
From: Doug Meil <doug.meil@explorysmedical.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>,
	"hbase-user@hadoop.apache.org" <hbase-user@hadoop.apache.org>
Date: Mon, 19 Oct 2009 12:51:46 -0400
Subject: RE: Question about MapReduce
Thread-Topic: Question about MapReduce
Thread-Index: AcpQ0AdV0WWuSZx4SUOjxlRvRkko8AAC8n3Q
Message-ID: <67680900F79B1D4F99C844EE386FC5950196E34A95@EX2K7VS03.4emm.local>
References: <967240.52565.qm@web111409.mail.gq1.yahoo.com>
 <805542.37510.qm@web111408.mail.gq1.yahoo.com>
 <5D66A842901F8E41815AF6D27A28EC490A7D6E07EF@Mail-Ab02.rmg-ny.com>
 <297355.50875.qm@web111413.mail.gq1.yahoo.com>
 <67680900F79B1D4F99C844EE386FC5950196E34A15@EX2K7VS03.4emm.local>
 <797590.66623.qm@web111407.mail.gq1.yahoo.com>
In-Reply-To: <797590.66623.qm@web111407.mail.gq1.yahoo.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

The job still needs everything else (input path, output path, mapper-class,=
 etc.).  This is trying to address the parameter-passing question of how pa=
rameters can be passed to mappers/reducers.

Correction:  there is a bug in my example...  Since I'm reading the paramet=
er with a 'get() ' I should have called 'set' instead of 'setStrings'.  But=
 that's the general idea.



-----Original Message-----
From: Something Something [mailto:luckyguy2050@yahoo.com]=20
Sent: Monday, October 19, 2009 11:23 AM
To: general@hadoop.apache.org; hbase-user@hadoop.apache.org
Subject: Re: Question about MapReduce

Interesting... I haven't tried this yet.. but in this case what would you s=
pecify as the 'InputPath'?  I was under the assumption that a Job needs som=
e kind of InputPath.. no?  I don't see a NullInputPath.  Is there something=
 equivalent?



________________________________
From: Doug Meil <doug.meil@explorysmedical.com>
To: "hbase-user@hadoop.apache.org" <hbase-user@hadoop.apache.org>; "general=
@hadoop.apache.org" <general@hadoop.apache.org>
Sent: Mon, October 19, 2009 7:34:03 AM
Subject: RE: Question about MapReduce

Hi there-

I didn't see the option in the thread yet which seems pretty straightforwar=
d:

When setting up the job:

    Job job =3D new Job(conf, "my job");

...
       =20
    conf.setStrings("param", "param1");


And then in the map method:

      String paramVal =3D context.getConfiguration().get("param");


This is using Hadoop .20 syntax, the previous verison had a 'configure' met=
hod you had to implement.



-----Original Message-----
From: Something Something [mailto:luckyguy2050@yahoo.com]=20
Sent: Thursday, October 15, 2009 2:31 PM
To: hbase-user@hadoop.apache.org; Hadoop
Subject: Re: Question about MapReduce

1) I don't think TableInputFormat is useful in this case.  Looks like it's =
used for scanning columns from a single HTable.
2) TableMapReduceUtil - same problem.  Seems like this works with just one =
table.
3) JV recommended NLineInputFormat, but my parameters are not in a file.  T=
hey come from multiple files and are in memory.

I guess what I am looking for is something like... InMemoryInputFormat... s=
imilar to FileInputFormat & DbInputFormat.  There's no such class right now=
.

Worse comes to worst, I can write the parameters into a flat file, and use =
FileInputFormat - but that will slow down this process considerably.  Is th=
ere no other way?





________________________________
From: Mark Vigeant <mark.vigeant@riskmetrics.com>
To: "hbase-user@hadoop.apache.org" <hbase-user@hadoop.apache.org>
Sent: Thu, October 15, 2009 7:21:40 AM
Subject: RE: Question about MapReduce

There is a tableInputFormat class in org.apache.hadoop.hbase.mapreduce.Tabl=
eInputFormat

Also, if you want to use TableMapReduceUtil you probably want to have your =
mapper function extend TableMapper.

Check out the javadocs for more info: http://hadoop.apache.org/hbase/docs/c=
urrent/api/index.html



-----Original Message-----
From: Something Something [mailto:luckyguy2050@yahoo.com]=20
Sent: Thursday, October 15, 2009 1:37 AM
To: general@hadoop.apache.org; hbase-user@hadoop.apache.org
Subject: Re: Question about MapReduce

If the answer is...

TableMapReduceUtil.initTableMapperJob

I apologize for the spam.  If this isn't the right way, please let me know.=
  Thanks.


--- On Wed, 10/14/09, Something Something <luckyguy2050@yahoo.com> wrote:

From: Something Something <luckyguy2050@yahoo.com>
Subject: Question about MapReduce
To: general@hadoop.apache.org, hbase-user@hadoop.apache.org
Date: Wednesday, October 14, 2009, 10:18 PM

I would like to start a Map-Reduce job that does not read data from an inpu=
t file or from a database.  I would like to pass 3 arguments to the Mapper =
& Reducer to work on.  Basically, these arguments are keys on the 3 differe=
nt tables on HBase.

In other words, I don't want to use FileInputFormat or DbInputFormat becaus=
e everything I need is already on HBase.=20

How can I do this?  Please let me know.  Thanks.


     =20

From general-return-644-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 20 02:38:06 2009
Return-Path: <general-return-644-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59747 invoked from network); 20 Oct 2009 02:38:06 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Oct 2009 02:38:06 -0000
Received: (qmail 6916 invoked by uid 500); 20 Oct 2009 02:38:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6831 invoked by uid 500); 20 Oct 2009 02:38:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6817 invoked by uid 99); 20 Oct 2009 02:38:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Oct 2009 02:38:05 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of hkumar.arora@gmail.com designates 209.85.223.188 as permitted sender)
Received: from [209.85.223.188] (HELO mail-iw0-f188.google.com) (209.85.223.188)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Oct 2009 02:37:54 +0000
Received: by iwn26 with SMTP id 26so2827910iwn.5
        for <general@hadoop.apache.org>; Mon, 19 Oct 2009 19:37:33 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=F2mQe0j5Kqxr8sVFq1/6hhavnp367if2ZU5+OK/5kTw=;
        b=IotUgrFNNzDoX0Y/egPwYl7zMjuMcfZNx9MJGPrUS8qAJhotYG0+n4h2ncEb8D7Ajs
         OSjn3rLMq4cH/xXFiUfkneOtJguvqNKn9EUWP7pO11x2CBFLF7ccMguMSpB8vP9YpSna
         EhCF8/vYIEwVQGZMKtcD5LjxWAavv4NuGjmZI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=YqvQyVH+0X/adprZHC+geIklQiDOIaLNW9fRH7SYilioBo4U4kj/c748+XMLe61kGD
         lYat3meVdo9mXSV8YnWk52ZBPJXoVMJgHrvxXD90Oz/qdgrbcTMrvAaEsPeE0tpO1J8k
         LnDkeTZVvZrNL0oCHeGbM3jO+zb/xrmdl84/g=
MIME-Version: 1.0
Received: by 10.231.25.29 with SMTP id x29mr3683957ibb.31.1256006253138; Mon, 
	19 Oct 2009 19:37:33 -0700 (PDT)
Date: Tue, 20 Oct 2009 11:37:33 +0900
Message-ID: <e497591c0910191937p1c8e2b1dh5df63f4ae9d5b057@mail.gmail.com>
Subject: org.apache.hadoop.util.DiskChecker$DiskErrorException
From: Harshit Kumar <hkumar.arora@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00151773e7308dd307047654be18
X-Virus-Checked: Checked by ClamAV on apache.org

--00151773e7308dd307047654be18
Content-Type: text/plain; charset=UTF-8

Hi
I get the following error on executing a map-reduce application on EC2.
Can any one please pass suggest some pointers what is going wrong?

I tried searching the mailing list, but couldnt find any help regarding this
type of error.

*[root@ip-10-243-47-69 hadoop-0.19.0]# bin/hadoop jar lubm.jar
bike.lubm.SubPredObj In*
*09/10/19 22:26:41 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=*
*09/10/19 22:26:41 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.*
*09/10/19 22:26:42 INFO mapred.FileInputFormat: Total input paths to process
: 1*
*09/10/19 22:26:42 INFO mapred.JobClient: Running job: job_local_0001*
*09/10/19 22:26:42 INFO mapred.FileInputFormat: Total input paths to process
: 1*
*09/10/19 22:26:43 INFO mapred.MapTask: numReduceTasks: 1*
*09/10/19 22:26:43 INFO mapred.MapTask: io.sort.mb = 100*
*09/10/19 22:26:43 INFO mapred.MapTask: data buffer = 79691776/99614720*
*09/10/19 22:26:43 INFO mapred.MapTask: record buffer = 262144/327680*
*09/10/19 22:26:43 INFO mapred.JobClient:  map 0% reduce 0%*
*09/10/19 22:26:44 WARN impl.RDFDefaultErrorHandler: unknown-source: {W136}
Relative URIs are not permitted in RDF: specifically <RDF/XML>*
*09/10/19 22:26:44 ERROR impl.RDFDefaultErrorHandler:
file:///usr/local/hadoop-0.19.0/RDF/XML(line 9 column 28): {E215} Resolving
against relative URI <RDF/XML>: <>*
*09/10/19 22:26:44 WARN impl.RDFDefaultErrorHandler:
file:///usr/local/hadoop-0.19.0/RDF/XML(line 9 column 28): {W136} Relative
URIs are not permitted in RDF: specifically <RDF/XML>*
*09/10/19 22:26:49 INFO mapred.MapTask: Starting flush of map output*
*09/10/19 22:26:49 WARN mapred.LocalJobRunner: job_local_0001*
*org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any
valid local directory for
taskTracker/jobcache/job_local_0001/attempt_local_0001_m_000000_0/output/spill0.out
*
*        at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:335)
*
*        at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
*
*        at
org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:107)
*
*        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:920)
*
*        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:832)*
*        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)*
*        at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:138)*
*java.io.IOException: Job failed!*
*        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1217)*
*        at bike.lubm.SubPredObj.getSPOFile(SubPredObj.java:63)*
*        at bike.lubm.SubPredObj.main(SubPredObj.java:12)*
*        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)*
*        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
*
*        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
*
*        at java.lang.reflect.Method.invoke(Method.java:597)*
*        at org.apache.hadoop.util.RunJar.main(RunJar.java:165)*
*        at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)*
*        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)*
*        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)*
*        at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)*

Thanks
H. Kumar
skype: harshit900
Blog: http://harshitkumar.wordpress.com
Website: http:/kumarharmuscat.tripod.com

--00151773e7308dd307047654be18--

From general-return-645-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 20 12:49:28 2009
Return-Path: <general-return-645-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17299 invoked from network); 20 Oct 2009 12:49:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Oct 2009 12:49:27 -0000
Received: (qmail 76332 invoked by uid 500); 20 Oct 2009 12:49:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76278 invoked by uid 500); 20 Oct 2009 12:49:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76141 invoked by uid 99); 20 Oct 2009 12:49:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Oct 2009 12:49:26 +0000
X-ASF-Spam-Status: No, hits=-0.6 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [64.18.2.179] (HELO exprod7og113.obsmtp.com) (64.18.2.179)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 20 Oct 2009 12:49:16 +0000
Received: from source ([209.85.223.199]) by exprod7ob113.postini.com ([64.18.6.12]) with SMTP
	ID DSNKSt2xtTL5BtSaOnLyfJoCpJPn3t3wyjfD@postini.com; Tue, 20 Oct 2009 05:48:55 PDT
Received: by iwn37 with SMTP id 37so2726073iwn.16
        for <general@hadoop.apache.org>; Tue, 20 Oct 2009 05:48:53 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.121.93 with SMTP id g29mr14711053ibr.13.1256042933545; 
	Tue, 20 Oct 2009 05:48:53 -0700 (PDT)
Date: Tue, 20 Oct 2009 15:48:53 +0300
Message-ID: <6eeff0ad0910200548q245deeebq2a8afebcf18807c@mail.gmail.com>
Subject: Lunching webapps with hadoop
From: Vitaliy Avdeev <vavdeev@xasax.net>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e646926ae06b1504765d4848
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e646926ae06b1504765d4848
Content-Type: text/plain; charset=ISO-8859-1

Hello.
I need to lunch my web aplication with hadoop.
It is writed with jsp.
What i need to do to lunch it witch hadoop?

--0016e646926ae06b1504765d4848--

From general-return-646-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 20 15:06:08 2009
Return-Path: <general-return-646-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85760 invoked from network); 20 Oct 2009 15:06:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Oct 2009 15:06:05 -0000
Received: (qmail 62857 invoked by uid 500); 20 Oct 2009 15:06:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62798 invoked by uid 500); 20 Oct 2009 15:06:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62343 invoked by uid 99); 20 Oct 2009 15:05:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Oct 2009 15:05:59 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bogdan.maryniuk@gmail.com designates 209.85.219.209 as permitted sender)
Received: from [209.85.219.209] (HELO mail-ew0-f209.google.com) (209.85.219.209)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Oct 2009 15:05:56 +0000
Received: by ewy5 with SMTP id 5so2214536ewy.36
        for <general@hadoop.apache.org>; Tue, 20 Oct 2009 08:05:35 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=Okv/xg0iHU4cwleonZbxNj53hizXHsjaQKEtI21tY9o=;
        b=GOySWvYCM5BgNuzTsN00UilVdV7iooDTTGkR03uV1DclkWproJZbcBf/9kotMvMLuK
         4Yj/O5VlGT2+AlQpJiWzAZ21jgzUuXsQamqJlJmuN7y3jzxP+2N0B89R/gxiHdOtJyGi
         1m49tlsiz47MwpuKn1nBp7El7G+xGQOi4mWJ8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=XkyyAuXBB6zQ3UCm/HQX8dBw73BXuxqg7eqrW9cj1KzvbJQn6VRIpFLpumgbYMRUJR
         +hHFOKbfxGiaKGLWPMNs70Q+A3LzeH5YybdzsmaHNl4GrW3UPAoYAu3VBn0A1TUnSPJH
         cUTeqn9RqyBM13sEX+jHFKelqiM4dLBgEKeco=
MIME-Version: 1.0
Received: by 10.211.157.7 with SMTP id j7mr7533818ebo.2.1256051135363; Tue, 20 
	Oct 2009 08:05:35 -0700 (PDT)
In-Reply-To: <6eeff0ad0910200548q245deeebq2a8afebcf18807c@mail.gmail.com>
References: <6eeff0ad0910200548q245deeebq2a8afebcf18807c@mail.gmail.com>
Date: Wed, 21 Oct 2009 00:05:35 +0900
Message-ID: <fa561940910200805n4165e52ay8e1a700b3e45b98b@mail.gmail.com>
Subject: Re: Lunching webapps with hadoop
From: "Bogdan M. Maryniuk" <bogdan.maryniuk@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On Tue, Oct 20, 2009 at 9:48 PM, Vitaliy Avdeev <vavdeev@xasax.net> wrote:
> I need to lunch my web aplication with hadoop.
Probably "launch" instead, since you're not gonna eat it... :-)
Anyway, what do you mean when saying "launch web application with
hadoop"? What is the design and what you're gonna do?

> What i need to do to lunch it witch hadoop?
Well... first step is probably install Hadoop itself =E2=80=94 that's for
sure!... :-) The next step is probably to explain us your app design
and what you want to achieve.

--=20
Kind regards, BM

Things, that are stupid at the beginning, rarely ends up wisely.

From general-return-647-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 21 17:51:17 2009
Return-Path: <general-return-647-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38142 invoked from network); 21 Oct 2009 17:51:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Oct 2009 17:51:17 -0000
Received: (qmail 48686 invoked by uid 500); 21 Oct 2009 16:15:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48572 invoked by uid 500); 21 Oct 2009 16:15:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48485 invoked by uid 99); 21 Oct 2009 16:15:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Oct 2009 16:15:32 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jonathan.seidman@gmail.com designates 209.85.210.194 as permitted sender)
Received: from [209.85.210.194] (HELO mail-yx0-f194.google.com) (209.85.210.194)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Oct 2009 16:15:25 +0000
Received: by yxe32 with SMTP id 32so6189054yxe.5
        for <general@hadoop.apache.org>; Wed, 21 Oct 2009 09:15:04 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=BF8q0oxqECiM3ZbXOxcjSW33gTVQbnj3AVb/pOsmQnU=;
        b=XmYt1Pg9NRilJYbTZY655xjEEWxQZuVmkEsiYC3zxmAPVU+JQH+5XIUvF60bplaNXz
         xOPpHuobychrBS7uzR/asYgLOj3B5K9ER8rsXOefA1dGS9uetbT7sN5GXW3B+gc2hXoR
         VFtzUTLh3LNeOrqIXA4dZvrR0IV9t5zvTWLuk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=ACEPGeWRD5X9fPh4G3icluvpu5Vv6sHSoA/cC2VPucmmwF11KZqts35z5Vl4hiBw8O
         h5L8WKdSFS7Ak2yYTtoZBL6CAI8briVqVTYgh7WSQN32W66bY+hSD1iXaueQq6X8bxOI
         j8+I5PptBa7qceVkeZJhMj7vceWnvn0YxyXTo=
MIME-Version: 1.0
Received: by 10.101.208.26 with SMTP id k26mr5064849anq.6.1256141703934; Wed, 
	21 Oct 2009 09:15:03 -0700 (PDT)
Date: Wed, 21 Oct 2009 11:15:03 -0500
Message-ID: <7ef0f1a90910210915p6e404c1eq7979c0a969a504a3@mail.gmail.com>
Subject: Namenode with External Storage?
From: Jonathan Seidman <jonathan.seidman@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d3cf2f0cf40b0476744858
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d3cf2f0cf40b0476744858
Content-Type: text/plain; charset=ISO-8859-1

Apologies if this has been answered previously, but I'm unable to find
anything that seems to cover this.

It's clear that datanodes require local storage for Hadoop to function
efficiently, but is there any significant disadvantage to using external
storage for namenodes? We're exploring the possibility of using a different
class of hardware for our namenodes with attached storage and little or no
internal storage. Some of the benefits this would provide us are: 1)
allowing our sysadmins to deploy hardware that they're familiar with and
already have considerable experience keeping up in a production environment.
2) no namenode downtime to replace a failed disk.

We don't anticipate that this approach would cause any significant
degradation to performance, but let me know if there's something we're not
considering.

Thanks.

Jonathan

--0016e6d3cf2f0cf40b0476744858--

From general-return-648-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 21 18:28:46 2009
Return-Path: <general-return-648-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 59366 invoked from network); 21 Oct 2009 18:28:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Oct 2009 18:28:24 -0000
Received: (qmail 96479 invoked by uid 500); 21 Oct 2009 18:27:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95714 invoked by uid 500); 21 Oct 2009 18:27:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95403 invoked by uid 99); 21 Oct 2009 18:27:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Oct 2009 18:27:46 +0000
X-ASF-Spam-Status: No, hits=-1.9 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 209.85.211.185 as permitted sender)
Received: from [209.85.211.185] (HELO mail-yw0-f185.google.com) (209.85.211.185)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Oct 2009 18:22:24 +0000
Received: by ywh15 with SMTP id 15so6322260ywh.5
        for <general@hadoop.apache.org>; Wed, 21 Oct 2009 11:22:03 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=UqpKzbI7ZhgewU/C2MGkCefDqxAlLMOmD660ePuwn4o=;
        b=QRa06Qr1YIk0nYyRqWmfCQiGdyouI2pwwIXldjNJkP+CE/gzZwR6X/KwABugXF/S+P
         mj9rda7ROmXVNcdLrkOBLUe+3G83U3t+fmUD/XsBFEMiOhI9mPpmmp1oUjtQn/dPAmnM
         uM6Q2EbaOY5ztAIOWKC5GDTtYQJ4cCAymUxgU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=c+i2Xum3RFHBKMkLYEdSiOa5dTu00fbSnUk7hVTo6JEzoqHMZDVt12WjEUmR8U4WPu
         S9+I7hSrClmzteUjOypl0bRaHknBszMWyluweqQIZp2EsQzDPY0CPuWT+JcAKAql0EgR
         y6C4u1mSF/EWougVw+bocFSRedyLq+ZfbD2ak=
MIME-Version: 1.0
Received: by 10.150.237.9 with SMTP id k9mr13743758ybh.108.1256149323739; Wed, 
	21 Oct 2009 11:22:03 -0700 (PDT)
In-Reply-To: <7ef0f1a90910210915p6e404c1eq7979c0a969a504a3@mail.gmail.com>
References: <7ef0f1a90910210915p6e404c1eq7979c0a969a504a3@mail.gmail.com>
Date: Wed, 21 Oct 2009 11:22:03 -0700
Message-ID: <4aa34eb70910211122k389f5c6al26e860054b2a45f4@mail.gmail.com>
Subject: Re: Namenode with External Storage?
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd23b2039f2060476760e8b

--000e0cd23b2039f2060476760e8b
Content-Type: text/plain; charset=ISO-8859-1

The namenode uses storage to write file system transaction logs as well as
to write server-debug-log-messages. Both of these could be stored in a NFS
mounted file system, should work well.

thanks,
dhruba


On Wed, Oct 21, 2009 at 9:15 AM, Jonathan Seidman <
jonathan.seidman@gmail.com> wrote:

> Apologies if this has been answered previously, but I'm unable to find
> anything that seems to cover this.
>
> It's clear that datanodes require local storage for Hadoop to function
> efficiently, but is there any significant disadvantage to using external
> storage for namenodes? We're exploring the possibility of using a different
> class of hardware for our namenodes with attached storage and little or no
> internal storage. Some of the benefits this would provide us are: 1)
> allowing our sysadmins to deploy hardware that they're familiar with and
> already have considerable experience keeping up in a production
> environment.
> 2) no namenode downtime to replace a failed disk.
>
> We don't anticipate that this approach would cause any significant
> degradation to performance, but let me know if there's something we're not
> considering.
>
> Thanks.
>
> Jonathan
>



-- 
Connect to me at http://www.facebook.com/dhruba

--000e0cd23b2039f2060476760e8b--

From general-return-649-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 22 02:59:20 2009
Return-Path: <general-return-649-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 28776 invoked from network); 22 Oct 2009 02:59:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Oct 2009 02:59:19 -0000
Received: (qmail 34451 invoked by uid 500); 22 Oct 2009 02:59:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34358 invoked by uid 500); 22 Oct 2009 02:59:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34348 invoked by uid 99); 22 Oct 2009 02:59:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Oct 2009 02:59:18 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [128.200.80.34] (HELO smtp1.es.uci.edu) (128.200.80.34)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Oct 2009 02:59:09 +0000
Received: from [192.168.1.104] (ip72-211-211-194.oc.oc.cox.net [72.211.211.194])
	(authenticated bits=0)
	by smtp1.es.uci.edu (8.13.1/8.13.1) with ESMTP id n9M2wkc0027159
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT)
	for <general@hadoop.apache.org>; Wed, 21 Oct 2009 19:58:47 -0700
X-UCInetID: vayyalas
Message-ID: <4ADFCA54.9000400@uci.edu>
Date: Wed, 21 Oct 2009 19:58:28 -0700
From: Vandana Ayyalasomayajula <vayyalas@uci.edu>
User-Agent: Thunderbird 2.0.0.23 (Windows/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Eclipse plugin for Hadoop
Content-Type: multipart/alternative;
 boundary="------------070305070904080608030100"
X-Virus-Checked: Checked by ClamAV on apache.org

--------------070305070904080608030100
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi All,

I was configuring eclipse o run map- reduce jobs. I am following the 
tutorial on 
http://developer.yahoo.com/hadoop/tutorial/module3.html#plugin-conf . I 
am not able to find the plug in mentioned in the site. I downloaded one 
from http://code.google.com/edu/parallel/tools/hadoopvm/index.html. 
<http://code.google.com/edu/parallel/tools/hadoopvm/index.html> But when 
I try to configure the hadoop server, I am getting an error message 
saying: "No hadoop found in this location". I think I am giving an 
incorrect location for installation directory. What should be the 
correct value ?

If anyone has used the tutorial provided by Yahoo! , please help me out 
with this problem.
Thanks in advance.
Vandana

--------------070305070904080608030100--

From general-return-650-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 22 03:16:42 2009
Return-Path: <general-return-650-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31911 invoked from network); 22 Oct 2009 03:16:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Oct 2009 03:16:42 -0000
Received: (qmail 40640 invoked by uid 500); 22 Oct 2009 03:16:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40557 invoked by uid 500); 22 Oct 2009 03:16:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40547 invoked by uid 99); 22 Oct 2009 03:16:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Oct 2009 03:16:41 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of hkumar.arora@gmail.com designates 209.85.223.188 as permitted sender)
Received: from [209.85.223.188] (HELO mail-iw0-f188.google.com) (209.85.223.188)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Oct 2009 03:16:32 +0000
Received: by iwn26 with SMTP id 26so4228407iwn.5
        for <general@hadoop.apache.org>; Wed, 21 Oct 2009 20:16:11 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=/D7sagvqVX1Xdv3syag0XM+1AxZRwnOPBjEC/umFMls=;
        b=ZWDz5hHeUUPb1TTWPbfi9hGzni9A3RZV+eKzTxSIiJafeTIsNCSmAzxkC45PWFPoHz
         WvYf8OmOE1gV4lnTF8gINzDzGIlF2E0jzJY6kED/O+dMhy+9PDMBCJcZduuLyxCOTOq2
         MIhL1eV+8jdBF8eZAzhIdYWd7aT3FbANmrR4I=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=UsDhwNcnSC1hAnWn5DU6nEV8OuAsR7tQJPjzFV54H7rYl4WMxOodIBDdghZDlu4cSg
         PdWfq88Q/2OCn9Fvcubp2AsJ5uRG7sQqYlVVbd7GwDyNOPOVWWw5Ug35U11LzpFGwhd6
         uR7f1JMq4reIMCALP627AJ8k+vx4Tczrijrsc=
MIME-Version: 1.0
Received: by 10.231.123.41 with SMTP id n41mr5414170ibr.46.1256181371634; Wed, 
	21 Oct 2009 20:16:11 -0700 (PDT)
In-Reply-To: <4ADFCA54.9000400@uci.edu>
References: <4ADFCA54.9000400@uci.edu>
Date: Thu, 22 Oct 2009 12:16:11 +0900
Message-ID: <e497591c0910212016y4fdcb3dcxb2c9f8b263bded07@mail.gmail.com>
Subject: Re: Eclipse plugin for Hadoop
From: Harshit Kumar <hkumar.arora@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e647195a6e080d04767d8416
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e647195a6e080d04767d8416
Content-Type: text/plain; charset=UTF-8

Dont know why do u need to download the plug-in from some site. The plug-in
is included in the downloaded hadoop tar file. You can find plugin jar file
inside /your-directory-structure/hadoop-version/contrib/eclipse-plugin/ -


H. Kumar


2009/10/22 Vandana Ayyalasomayajula <vayyalas@uci.edu>

> Hi All,
>
> I was configuring eclipse o run map- reduce jobs. I am following the
> tutorial on
> http://developer.yahoo.com/hadoop/tutorial/module3.html#plugin-conf . I am
> not able to find the plug in mentioned in the site. I downloaded one from
> http://code.google.com/edu/parallel/tools/hadoopvm/index.html. <
> http://code.google.com/edu/parallel/tools/hadoopvm/index.html> But when I
> try to configure the hadoop server, I am getting an error message saying:
> "No hadoop found in this location". I think I am giving an incorrect
> location for installation directory. What should be the correct value ?
>
> If anyone has used the tutorial provided by Yahoo! , please help me out
> with this problem.
> Thanks in advance.
> Vandana
>

--0016e647195a6e080d04767d8416--

From general-return-651-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 22 04:53:33 2009
Return-Path: <general-return-651-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60524 invoked from network); 22 Oct 2009 04:53:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Oct 2009 04:53:33 -0000
Received: (qmail 13418 invoked by uid 500); 22 Oct 2009 04:53:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13343 invoked by uid 500); 22 Oct 2009 04:53:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13333 invoked by uid 99); 22 Oct 2009 04:53:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Oct 2009 04:53:31 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [128.200.80.28] (HELO smtp2.es.uci.edu) (128.200.80.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Oct 2009 04:53:21 +0000
Received: from [192.168.1.104] (ip72-211-211-194.oc.oc.cox.net [72.211.211.194])
	(authenticated bits=0)
	by smtp2.es.uci.edu (8.13.1/8.13.1) with ESMTP id n9M4qw6I002950
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT)
	for <general@hadoop.apache.org>; Wed, 21 Oct 2009 21:52:59 -0700
X-UCInetID: vayyalas
Message-ID: <4ADFE517.9000703@uci.edu>
Date: Wed, 21 Oct 2009 21:52:39 -0700
From: Vandana Ayyalasomayajula <vayyalas@uci.edu>
User-Agent: Thunderbird 2.0.0.23 (Windows/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Eclipse plugin for Hadoop
References: <4ADFCA54.9000400@uci.edu> <e497591c0910212016y4fdcb3dcxb2c9f8b263bded07@mail.gmail.com>
In-Reply-To: <e497591c0910212016y4fdcb3dcxb2c9f8b263bded07@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks for the information. I just overlooked the contents of the folder.

Vandana
Harshit Kumar wrote:
> Dont know why do u need to download the plug-in from some site. The plug-in
> is included in the downloaded hadoop tar file. You can find plugin jar file
> inside /your-directory-structure/hadoop-version/contrib/eclipse-plugin/ -
>
>
> H. Kumar
>
>
> 2009/10/22 Vandana Ayyalasomayajula <vayyalas@uci.edu>
>
>   
>> Hi All,
>>
>> I was configuring eclipse o run map- reduce jobs. I am following the
>> tutorial on
>> http://developer.yahoo.com/hadoop/tutorial/module3.html#plugin-conf . I am
>> not able to find the plug in mentioned in the site. I downloaded one from
>> http://code.google.com/edu/parallel/tools/hadoopvm/index.html. <
>> http://code.google.com/edu/parallel/tools/hadoopvm/index.html> But when I
>> try to configure the hadoop server, I am getting an error message saying:
>> "No hadoop found in this location". I think I am giving an incorrect
>> location for installation directory. What should be the correct value ?
>>
>> If anyone has used the tutorial provided by Yahoo! , please help me out
>> with this problem.
>> Thanks in advance.
>> Vandana
>>
>>     
>
>   


From general-return-652-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 22 16:37:52 2009
Return-Path: <general-return-652-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82846 invoked from network); 22 Oct 2009 16:37:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Oct 2009 16:37:51 -0000
Received: (qmail 5434 invoked by uid 500); 22 Oct 2009 16:37:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5353 invoked by uid 500); 22 Oct 2009 16:37:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5343 invoked by uid 99); 22 Oct 2009 16:37:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Oct 2009 16:37:50 +0000
X-ASF-Spam-Status: No, hits=-6.6 required=5.0
	tests=BAYES_00,RCVD_IN_DNSWL_MED
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of gmackey@cs.ucf.edu designates 132.170.108.1 as permitted sender)
Received: from [132.170.108.1] (HELO longwood.cs.ucf.edu) (132.170.108.1)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Oct 2009 16:37:47 +0000
Received: from longwood.cs.ucf.edu (localhost [127.0.0.1])
	by longwood.cs.ucf.edu (8.14.1/8.14.1) with ESMTP id n9MGbMw8009756
	for <general@hadoop.apache.org>; Thu, 22 Oct 2009 12:37:22 -0400 (EDT)
Received: (from prayer@localhost)
	by longwood.cs.ucf.edu (8.14.1/8.14.1/Submit) id n9MGbMW2009755
	for general@hadoop.apache.org; Thu, 22 Oct 2009 12:37:22 -0400 (EDT)
X-Authentication-Warning: longwood.cs.ucf.edu: prayer set sender to gmackey@cs.ucf.edu using -f
From: gmackey@cs.ucf.edu
To: general@hadoop.apache.org
Subject: Re: Namenode with External Storage?
Date: 22 Oct 2009 12:37:21 -0400
X-Mailer: Prayer v1.0.16
X-Originating-IP: [10.173.214.212]
In-Reply-To: <7ef0f1a90910210915p6e404c1eq7979c0a969a504a3@mail.gmail.com>
Message-ID: <Prayer.1.0.16.0910221237210.7700@longwood.cs.ucf.edu>
References: <7ef0f1a90910210915p6e404c1eq7979c0a969a504a3@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; format=flowed; charset=ISO-8859-1
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on gondor.cs.ucf.edu
X-Virus-Scanned: clamav-milter 0.95.2 at longwood
X-Virus-Status: Clean
X-Old-Spam-Status: No, score=-1.4 required=6.0 tests=ALL_TRUSTED autolearn=failed
	version=3.2.5

As with Dhruba's comment, so long as it is just the namenode that is 
running on a networked file system everything should be chill. The namenode 
keeps all of its working metadata in main mem, and it only occasionally 
pushes a log file out to hard storage (and if I remember correctly you can 
adjust this time window in one of the site files).

However, you are going to run into huge performance issues running 
datanodes over a networked storage system. Having to push that many file 
requests over a network for a respectable mapreduce job is going to kill 
your equipment.

 - Grant

On Oct 21 2009, Jonathan Seidman wrote:

>Apologies if this has been answered previously, but I'm unable to find
>anything that seems to cover this.
>
> It's clear that datanodes require local storage for Hadoop to function 
> efficiently, but is there any significant disadvantage to using external 
> storage for namenodes? We're exploring the possibility of using a 
> different class of hardware for our namenodes with attached storage and 
> little or no internal storage. Some of the benefits this would provide us 
> are: 1) allowing our sysadmins to deploy hardware that they're familiar 
> with and already have considerable experience keeping up in a production 
> environment. 2) no namenode downtime to replace a failed disk.
>
>We don't anticipate that this approach would cause any significant
>degradation to performance, but let me know if there's something we're not
>considering.
>
>Thanks.
>
>Jonathan
>

-- 
--
Grant Mackey
PhD student Computer Engineering
University of Central Florida
Rm 231 cube 5 (321) 960-8851


From general-return-653-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 22 17:01:30 2009
Return-Path: <general-return-653-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 90044 invoked from network); 22 Oct 2009 17:01:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Oct 2009 17:01:30 -0000
Received: (qmail 37249 invoked by uid 500); 22 Oct 2009 17:01:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37164 invoked by uid 500); 22 Oct 2009 17:01:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37154 invoked by uid 99); 22 Oct 2009 17:01:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Oct 2009 17:01:29 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Oct 2009 17:01:26 +0000
Received: from oceanfarearth-lm.corp.yahoo.com (oceanfarearth-lm.corp.yahoo.com [10.72.113.156])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n9MH0Bvq026028
	for <general@hadoop.apache.org>; Thu, 22 Oct 2009 10:00:12 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:mime-version:
	subject:date:references:x-mailer;
	b=o0oHSJipC3lmD0VUr1i/nkbl5UL5JPt/Xy6Sf3AKbYo9vsxpMw0EdIIvvOcelEhh
Message-Id: <5480928D-30E6-4B28-AC23-A77F31EF3164@yahoo-inc.com>
From: Sanjay Radia <sradia@yahoo-inc.com>
To: <general@hadoop.apache.org>
In-Reply-To: <Prayer.1.0.16.0910221237210.7700@longwood.cs.ucf.edu>
Content-Type: multipart/alternative; boundary=Apple-Mail-4-898220983
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Namenode with External Storage?
Date: Thu, 22 Oct 2009 10:00:11 -0700
References: <7ef0f1a90910210915p6e404c1eq7979c0a969a504a3@mail.gmail.com> <Prayer.1.0.16.0910221237210.7700@longwood.cs.ucf.edu>
X-Mailer: Apple Mail (2.936)

--Apple-Mail-4-898220983
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


On Oct 22, 2009, at 9:37 AM, <gmackey@cs.ucf.edu> wrote:

> As with Dhruba's comment, so long as it is just the namenode that is
> running on a networked file system everything should be chill. The  
> namenode
> keeps all of its working metadata in main mem, and it only  
> occasionally
> pushes a log file out to hard storage (and if I remember correctly  
> you can
> adjust this time window in one of the site files).
>

Actually it pushes out the update logs on each and every update  
synchronously.
The checkpoint however is pushed out periodically.

Also, at yahoo, we push out NN state to multiple disks and one of the  
"disks" is a nfs filer. This is configurable.

sanjay
>
> However, you are going to run into huge performance issues running
> datanodes over a networked storage system. Having to push that many  
> file
> requests over a network for a respectable mapreduce job is going to  
> kill
> your equipment.
>
>  - Grant
>
> On Oct 21 2009, Jonathan Seidman wrote:
>
> >Apologies if this has been answered previously, but I'm unable to  
> find
> >anything that seems to cover this.
> >
> > It's clear that datanodes require local storage for Hadoop to  
> function
> > efficiently, but is there any significant disadvantage to using  
> external
> > storage for namenodes? We're exploring the possibility of using a
> > different class of hardware for our namenodes with attached  
> storage and
> > little or no internal storage. Some of the benefits this would  
> provide us
> > are: 1) allowing our sysadmins to deploy hardware that they're  
> familiar
> > with and already have considerable experience keeping up in a  
> production
> > environment. 2) no namenode downtime to replace a failed disk.
> >
> >We don't anticipate that this approach would cause any significant
> >degradation to performance, but let me know if there's something  
> we're not
> >considering.
> >
> >Thanks.
> >
> >Jonathan
> >
>
> --
> --
> Grant Mackey
> PhD student Computer Engineering
> University of Central Florida
> Rm 231 cube 5 (321) 960-8851
>
>


--Apple-Mail-4-898220983--

From general-return-654-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 22 19:16:56 2009
Return-Path: <general-return-654-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 48029 invoked from network); 22 Oct 2009 19:16:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Oct 2009 19:16:55 -0000
Received: (qmail 32501 invoked by uid 500); 22 Oct 2009 19:16:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32410 invoked by uid 500); 22 Oct 2009 19:16:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32399 invoked by uid 99); 22 Oct 2009 19:16:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Oct 2009 19:16:52 +0000
X-ASF-Spam-Status: No, hits=-6.6 required=5.0
	tests=BAYES_00,RCVD_IN_DNSWL_MED
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of gmackey@cs.ucf.edu designates 132.170.108.1 as permitted sender)
Received: from [132.170.108.1] (HELO longwood.cs.ucf.edu) (132.170.108.1)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Oct 2009 19:16:49 +0000
Received: from longwood.cs.ucf.edu (localhost [127.0.0.1])
	by longwood.cs.ucf.edu (8.14.1/8.14.1) with ESMTP id n9MJGRbF002710
	for <general@hadoop.apache.org>; Thu, 22 Oct 2009 15:16:27 -0400 (EDT)
Received: (from apache@localhost)
	by longwood.cs.ucf.edu (8.14.1/8.14.1/Submit) id n9MJGO55002705
	for general@hadoop.apache.org; Thu, 22 Oct 2009 15:16:24 -0400 (EDT)
X-Authentication-Warning: longwood.cs.ucf.edu: apache set sender to gmackey@cs.ucf.edu using -f
Received: from 10.173.214.212 ([10.173.214.212]) by mail.cs.ucf.edu (Horde
 Framework) with HTTP; Thu, 22 Oct 2009 15:16:24 -0400
Message-ID: <20091022151624.30264yzwbswk81wk@mail.cs.ucf.edu>
Date: Thu, 22 Oct 2009 15:16:24 -0400
From: Grant Mackey <gmackey@cs.ucf.edu>
To: general@hadoop.apache.org
Subject: Re: Namenode with External Storage?
References: <7ef0f1a90910210915p6e404c1eq7979c0a969a504a3@mail.gmail.com>
 <Prayer.1.0.16.0910221237210.7700@longwood.cs.ucf.edu>
 <5480928D-30E6-4B28-AC23-A77F31EF3164@yahoo-inc.com>
In-Reply-To: <5480928D-30E6-4B28-AC23-A77F31EF3164@yahoo-inc.com>
MIME-Version: 1.0
Content-Type: text/plain;
 charset=ISO-8859-1;
 DelSp="Yes";
 format="flowed"
Content-Disposition: inline
Content-Transfer-Encoding: 7bit
User-Agent: Internet Messaging Program (IMP) 4.3.3
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on gondor.cs.ucf.edu
X-Virus-Scanned: clamav-milter 0.95.2 at longwood
X-Virus-Status: Clean
X-Old-Spam-Status: No, score=-1.4 required=6.0 tests=ALL_TRUSTED autolearn=failed
	version=3.2.5

oops, that is correct. My mistake

Quoting Sanjay Radia <sradia@yahoo-inc.com>:

>
> On Oct 22, 2009, at 9:37 AM, <gmackey@cs.ucf.edu> wrote:
>
>> As with Dhruba's comment, so long as it is just the namenode that is
>> running on a networked file system everything should be chill. The  namenode
>> keeps all of its working metadata in main mem, and it only  occasionally
>> pushes a log file out to hard storage (and if I remember correctly  you can
>> adjust this time window in one of the site files).
>>
>
> Actually it pushes out the update logs on each and every update   
> synchronously.
> The checkpoint however is pushed out periodically.
>
> Also, at yahoo, we push out NN state to multiple disks and one of  
> the  "disks" is a nfs filer. This is configurable.
>
> sanjay
>>
>> However, you are going to run into huge performance issues running
>> datanodes over a networked storage system. Having to push that many  file
>> requests over a network for a respectable mapreduce job is going to  kill
>> your equipment.
>>
>> - Grant
>>
>> On Oct 21 2009, Jonathan Seidman wrote:
>>
>>> Apologies if this has been answered previously, but I'm unable to  find
>>> anything that seems to cover this.
>>>
>>> It's clear that datanodes require local storage for Hadoop to  function
>>> efficiently, but is there any significant disadvantage to using  external
>>> storage for namenodes? We're exploring the possibility of using a
>>> different class of hardware for our namenodes with attached  storage and
>>> little or no internal storage. Some of the benefits this would  provide us
>>> are: 1) allowing our sysadmins to deploy hardware that they're  familiar
>>> with and already have considerable experience keeping up in a  production
>>> environment. 2) no namenode downtime to replace a failed disk.
>>>
>>> We don't anticipate that this approach would cause any significant
>>> degradation to performance, but let me know if there's something  we're not
>>> considering.
>>>
>>> Thanks.
>>>
>>> Jonathan
>>>
>>
>> --
>> --
>> Grant Mackey
>> PhD student Computer Engineering
>> University of Central Florida
>> Rm 231 cube 5 (321) 960-8851
>>
>>
>
>



Grant Mackey
UCF Research Assistant
Engineering III
Rm 238 Cubicle 1

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.


From general-return-655-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 23 19:52:44 2009
Return-Path: <general-return-655-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76534 invoked from network); 23 Oct 2009 19:52:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Oct 2009 19:52:44 -0000
Received: (qmail 83309 invoked by uid 500); 23 Oct 2009 19:52:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 83230 invoked by uid 500); 23 Oct 2009 19:52:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 83220 invoked by uid 99); 23 Oct 2009 19:52:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Oct 2009 19:52:43 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [68.142.207.135] (HELO web32204.mail.mud.yahoo.com) (68.142.207.135)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 23 Oct 2009 19:52:40 +0000
Received: (qmail 11558 invoked by uid 60001); 23 Oct 2009 19:52:19 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1256327539; bh=5y3IyiEjZ9xNVTi9YZfPjirH1PnlQDLDQa1mUqC57k8=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=dOuVp4AcrJxBWd0YDn2UdQpq4IS0GlFK/FrbBI6HP2ZBgCOaUPe/PhYbVru9OjlWVSLpPFQlytFQaOb08eP7MtDICOZS73C+GKBWfiWE2w8t6u6960elKthzVS7vW3pr6SVKJKeZF0fQuEaWG6nEX2quGTlhziujG+/bZyIjRxo=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=mHJQIIdxgAYacF43B5EQwumeE/oUqIc7nsicyW88akvnOghzk2qpJlMZ4yvusCUuwQJus3jpJOuNHzRNBLPMSYxvpLRG9dKCIDwc2/CYm+8YmST7qZmlH8np/zB3iJPRZzN9iY6n4EY40tjoZF87i7WScjia6JJ7tntQ/156WjY=;
Message-ID: <318566.10596.qm@web32204.mail.mud.yahoo.com>
X-YMail-OSG: fs_TluAVM1kYs8pDPmkmcOzAQQIiFEtsZK3dFqhgBZVmvGkcso1Ie0GKPRfLPUDmKrwRJ1UWYQl.a370nYUWg147KaaLTF3_DhI1m0LOIOO6KFp_UI8cJxLPgIMT7nyPOzDtdsTpZYu4iwJj3IouOVIG0LMs1CJV41pH6PYN1ZoqFQNQXdn5isjVA7dBBw_z9_qg68cXGqCgRhsBaPXIzyy1gzhtHF44QWOWeSX5SXm2phFqv63Nrc..mVxk6vcmKg0Ctnr.x0XWfsbm_Nj7yZQ.Mow2IPuX64Aj66y.fjF1i0LnVnrH61QVZpu1pWmY8Bce0N97YYZcELUS6Y5FvAspx7hy8TjUxJVZ8EN_EsurOHsuubMtIvdzcz1D8ayHLhrQfC3Oqx75.HihU3Uitw--
Received: from [208.7.132.130] by web32204.mail.mud.yahoo.com via HTTP; Fri, 23 Oct 2009 12:52:19 PDT
X-Mailer: YahooMailClassic/7.0.14 YahooMailWebService/0.7.347.3
Date: Fri, 23 Oct 2009 12:52:19 -0700 (PDT)
From: Rajiv Maheshwari <rajivm01@yahoo.com>
Subject: CombineFileInput Format vs. Large SequenceFile
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1902091781-1256327539=:10596"

--0-1902091781-1256327539=:10596
Content-Type: text/plain; charset=us-ascii

Hi everyone,

I have a need to process a large number (millions) of relatively small XML files (I would imagine mostly 1K to 1 MB). I guess sending each file to a map-reduce task will cause too much overhead in setup and teardown of the tasks. So, I am considering 2 alternatives:

1) Generate 1 large SequenceFile with <K,V> = <Filename/URI, File XML content> for all the files. This SequenceFile would be huge. (I wonder is there any max record length / max size limit on HDFS file?)

2) Use CombineFileInputFormat

I would appreciate any comments on performance considerations and other pros and cons.

One more question: What if I have one large XML file composed of (a series of) each file's XML content, could it be possible to use StreamXMLReader while combining component files before sending to map-reduce task?

Thanks,
Rajiv


      
--0-1902091781-1256327539=:10596--

From general-return-656-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 24 09:06:59 2009
Return-Path: <general-return-656-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 32261 invoked from network); 24 Oct 2009 09:06:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Oct 2009 09:06:59 -0000
Received: (qmail 5262 invoked by uid 500); 24 Oct 2009 09:06:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5175 invoked by uid 500); 24 Oct 2009 09:06:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5165 invoked by uid 99); 24 Oct 2009 09:06:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Oct 2009 09:06:57 +0000
X-ASF-Spam-Status: No, hits=-2.0 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 209.85.217.228 as permitted sender)
Received: from [209.85.217.228] (HELO mail-gx0-f228.google.com) (209.85.217.228)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Oct 2009 09:06:55 +0000
Received: by gxk28 with SMTP id 28so8696342gxk.9
        for <general@hadoop.apache.org>; Sat, 24 Oct 2009 02:06:34 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=x9KXdtd0d2VJVtG4tqkPmKWEENI63yHHL1O+gz/FAWk=;
        b=j0ixdIczdH18McXpV27iy3JR7b6IMhxQ+6ldMsrIwx+0HCp3giQ7+m9BObtX7PCDlJ
         1eWHpR5gN0EEUKUaBSbKLleFXZgE23Kfrx6X7XRRfREUpaFiBnNUy507sY3YuaUPLJYf
         ck5s1kV8Te5CrAeHk9ZPlJgsoj23BHk3bRUz0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=J+yLv13YFzM/Y74KsN+dP8uig5spCwfpJMZkkkZqarSeN9eL9LPn/9DiWEynmA3/dw
         alPgUumHoUGBchQHFZMkCpwRMXSfmLqk/wFsTgV7jhAAg3Fia0nsw9BN7cydSXZCoczV
         7FLH7OCYOPj+vBuAPLTsHtPVm+aQ/fKPG2zeE=
MIME-Version: 1.0
Received: by 10.151.86.11 with SMTP id o11mr10595493ybl.134.1256375194327; 
	Sat, 24 Oct 2009 02:06:34 -0700 (PDT)
In-Reply-To: <318566.10596.qm@web32204.mail.mud.yahoo.com>
References: <318566.10596.qm@web32204.mail.mud.yahoo.com>
Date: Sat, 24 Oct 2009 02:06:34 -0700
Message-ID: <4aa34eb70910240206j5a1a46datc0f927b00c13d2ed@mail.gmail.com>
Subject: Re: CombineFileInput Format vs. Large SequenceFile
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd293f8299b790476aaa5a0

--000e0cd293f8299b790476aaa5a0
Content-Type: text/plain; charset=ISO-8859-1

you could use both. if you query is to be done once, then either way is ok.
If the query is to be done multiple times, it is better to create a large
sequencefile as the first step and then continue to query that large file
multiple times, otherwise you might land up  transferring all the data
across the network multiple times, one for every run of the query.

thanks
dhruba

On Fri, Oct 23, 2009 at 12:52 PM, Rajiv Maheshwari <rajivm01@yahoo.com>wrote:

> Hi everyone,
>
> I have a need to process a large number (millions) of relatively small XML
> files (I would imagine mostly 1K to 1 MB). I guess sending each file to a
> map-reduce task will cause too much overhead in setup and teardown of the
> tasks. So, I am considering 2 alternatives:
>
> 1) Generate 1 large SequenceFile with <K,V> = <Filename/URI, File XML
> content> for all the files. This SequenceFile would be huge. (I wonder is
> there any max record length / max size limit on HDFS file?)
>
> 2) Use CombineFileInputFormat
>
> I would appreciate any comments on performance considerations and other
> pros and cons.
>
> One more question: What if I have one large XML file composed of (a series
> of) each file's XML content, could it be possible to use StreamXMLReader
> while combining component files before sending to map-reduce task?
>
> Thanks,
> Rajiv
>
>
>




-- 
Connect to me at http://www.facebook.com/dhruba

--000e0cd293f8299b790476aaa5a0--

From general-return-657-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 26 04:19:42 2009
Return-Path: <general-return-657-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 274 invoked from network); 26 Oct 2009 04:19:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Oct 2009 04:19:41 -0000
Received: (qmail 61263 invoked by uid 500); 26 Oct 2009 04:19:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61110 invoked by uid 500); 26 Oct 2009 04:19:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61100 invoked by uid 99); 26 Oct 2009 04:19:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Oct 2009 04:19:40 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of hkumar.arora@gmail.com designates 209.85.223.171 as permitted sender)
Received: from [209.85.223.171] (HELO mail-iw0-f171.google.com) (209.85.223.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Oct 2009 04:19:37 +0000
Received: by iwn1 with SMTP id 1so6147843iwn.2
        for <general@hadoop.apache.org>; Sun, 25 Oct 2009 21:19:17 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=D6TxFbS+mJ5/qLUh0/+w3RyMKr30NniGnD9jRTxFCiI=;
        b=ANWnjYHFQJIr/sRDhMM62vQ6TL5QXywgGzAjbFUrJw0kP/Fz8fDBJkOg1P0IqlxBQ/
         HR+n69Em1N/YHQeWsrMQCwdawZZFUF1JTHMVWbh1jDmorxnCBats0acSlcGn4nonpThA
         EzRL68zx+d/UulTj8rCtEh//mZPqueK0UfBb8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=aPk68jLkL4ABf13BkV7EMQf5Z8ktIixPLgQfxyUD0DoKeW8ZuiICksKIOlevtiH/rf
         LhQGF+cdDE4TvMQ+kPFSSIsyKtKIpLPIPX9/08CNqK6EDS15EPJF8prBcSABrku22er5
         zBpHa7zSjReRNix/PtHLUlN2+kDEv0ICqd1rA=
MIME-Version: 1.0
Received: by 10.231.5.90 with SMTP id 26mr4979340ibu.42.1256530757171; Sun, 25 
	Oct 2009 21:19:17 -0700 (PDT)
Date: Mon, 26 Oct 2009 13:19:17 +0900
Message-ID: <e497591c0910252119v1483b0bt3c18cb64633e75c7@mail.gmail.com>
Subject: S3 Exception
From: Harshit Kumar <hkumar.arora@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015177409886e39aa0476cedd87

--0015177409886e39aa0476cedd87
Content-Type: text/plain; charset=UTF-8

Hi

There is 1 GB of rdf/owl files that I am executing on EC2. Execution throws
the following exception
-------------------

08/11/19 16:08:27 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the
same.
org.apache.hadoop.fs.s3.S3Exception:
org.jets3t.service.S3ServiceException: S3 PUT failed for '/' XML Error
Message: <?xml version="1.0"
encoding="UTF-8"?><Error><Code>OperationAborted</Code><Message>*A
conflicting conditional operation is currently in progress against
**this** resource*. Please try
again.</Message><RequestId>324E696A4BCA8731</RequestId><HostId>{REMOVED}</HostId></Error>
	at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.createBucket(Jets3tNativeFileSystemStore.java:74)
	at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.initialize(Jets3tNativeFileSystemStore.java:63)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
	at org.apache.hadoop.fs.s3native.$Proxy2.initialize(Unknown Source)
	at org.apache.hadoop.fs.s3native.NativeS3FileSystem.initialize(NativeS3FileSystem.java:215)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1339)
	at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1351)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:213)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:175)
	at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:158)
	at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:210)
	at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
	at org.apache.hadoop.streaming.StreamJob.submitAndMonitorJob(StreamJob.java:925)
	at org.apache.hadoop.streaming.StreamJob.go(StreamJob.java:115)
	at org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreaming.java:33)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
	at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
	at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
Caused by: org.jets3t.service.S3ServiceException: S3 PUT failed for
'/' XML Error Message: <?xml version="1.0"
encoding="UTF-8"?><Error><Code>OperationAborted</Code><Message>A
conflicting conditional operation is currently in progress against
this resource. Please try
again.</Message><RequestId>{REMOVED}</RequestId><HostId>{REMOVED}</HostId></Error>
	at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:424)
	at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestPut(RestS3Service.java:734)
	at org.jets3t.service.impl.rest.httpclient.RestS3Service.createObjectImpl(RestS3Service.java:1357)
	at org.jets3t.service.impl.rest.httpclient.RestS3Service.createBucketImpl(RestS3Service.java:1234)
	at org.jets3t.service.S3Service.createBucket(S3Service.java:1390)
	at org.jets3t.service.S3Service.createBucket(S3Service.java:1158)
	at org.jets3t.service.S3Service.createBucket(S3Service.java:1177)
	at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.createBucket(Jets3tNativeFileSystemStore.java:69)
	... 29 more

---------------------
There were like 50 failed map tasks out of total of 1141 maps. Though hadoop
job tracker successfully executed failed tasks again, but if a task fails,
and then restarting it again, takes execution time. If somehow, this
exception can be avoided altogether, program execution will be faster.

Came across a JIRA post that says, the exception is because s3 tries to
create a bucket which it should not create, and a patch is also posted for
the same. Applied the patch, but still the same error.

IMO, this error is because a operation is trying to access a part of file
which is already been accessed by another map job.

Please help resolve this issue or pass on any pointers that can suggest why
is this error happening?

Thanks and Regards

H. Kumar
skype: harshit900
Blog: http://harshitkumar.wordpress.com
Website: http:/kumarharmuscat.tripod.com

--0015177409886e39aa0476cedd87--

From general-return-658-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 27 17:06:31 2009
Return-Path: <general-return-658-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83858 invoked from network); 27 Oct 2009 17:06:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Oct 2009 17:06:31 -0000
Received: (qmail 37675 invoked by uid 500); 27 Oct 2009 17:06:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37552 invoked by uid 500); 27 Oct 2009 17:06:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37534 invoked by uid 99); 27 Oct 2009 17:06:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Oct 2009 17:06:27 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00,NO_RDNS_DOTCOM_HELO
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Oct 2009 17:06:24 +0000
Received: from SP1-EX07CAS02.ds.corp.yahoo.com (sp1-ex07cas02.ds.corp.yahoo.com [216.252.116.138])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n9RH4S1V003735;
	Tue, 27 Oct 2009 10:05:47 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:references:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:
	content-transfer-encoding:mime-version;
	b=KGR5swrKFssuMxtnaoDRg4ZeZTZTUMAm5WhJLS4xaXcpx5xJ+8juGoSezXnsi9ox
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 SP1-EX07CAS02.ds.corp.yahoo.com ([216.252.116.167]) with mapi; Tue, 27 Oct
 2009 10:05:20 -0700
From: Dekel Tankel <dekel@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>,
        "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Date: Tue, 27 Oct 2009 10:05:18 -0700
Subject: RE: Hadoop User Group (Bay Area) - Nov 18th at Yahoo!
Thread-Topic: Hadoop User Group (Bay Area) - Nov 18th at Yahoo!
Thread-Index: AcoSMRRU8X5auVLKR8ufbpFTQi1hLgQXaiyQDSYUL9A=
Message-ID: <46E2672895E8744A97D7577A6110858B4FD5DE68ED@SP1-EX07VS01.ds.corp.yahoo.com>
References: <46E2672895E8744A97D7577A6110858B4E1AC69A91@SP1-EX07VS01.ds.corp.yahoo.com> 
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi all,

Thank you for those who attended the meeting last week, I hope you found it=
 interesting and helpful. Special thanks to all the presenters!

Presentations are available here:
http://developer.yahoo.net/blogs/hadoop/2009/10/hadoop_user_group_hug_oct_2=
1st.html


RSVP is now open for the next monthly Bay Area Hadoop user group at the Yah=
oo! Sunnyvale Campus, planed for Nov 18th.

Registration is available here
http://www.meetup.com/hadoop/calendar/11724002/

We would like to include a session describing an application using Hadoop a=
nd/or related technologies. If you are interested to present please send me=
 an email with a short description.

Looking forward to seeing you and enjoy ApacheCon!

Dekel


From general-return-659-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 28 08:14:43 2009
Return-Path: <general-return-659-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25357 invoked from network); 28 Oct 2009 08:14:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Oct 2009 08:14:43 -0000
Received: (qmail 39598 invoked by uid 500); 28 Oct 2009 08:14:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39510 invoked by uid 500); 28 Oct 2009 08:14:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39496 invoked by uid 99); 28 Oct 2009 08:14:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Oct 2009 08:14:41 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of hadoopmaillist@gmail.com designates 209.85.211.185 as permitted sender)
Received: from [209.85.211.185] (HELO mail-yw0-f185.google.com) (209.85.211.185)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Oct 2009 08:14:39 +0000
Received: by ywh15 with SMTP id 15so540072ywh.5
        for <general@hadoop.apache.org>; Wed, 28 Oct 2009 01:14:18 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=AulgP5PsgWJyN1MG+tvDwYkunHhMjeks+PYE8tZ8zdQ=;
        b=cDk4PRos8oOuu9Sfed1YHIwsT5lXTb6KZfSMDjS+VIgrEYABvy12Fxw3+yH4bMOeE2
         Li/PPIvEYvtFNygvX1+n+2/YXPCr4KGZlNSjR7z3cyuYPxG7PGIz3fqmZFXPxDCbOJQA
         guRjHf8+B9zCmrLwErJyZ+KkwnW6S6B4BSfqQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=qAJnIMiyNa/k9hkFCHhhtjxqhEynBuOGW7nLyLGS5hF6yYGnpdiZeXAY2hutNYREm+
         0dYS1QQ6Wjd/npn7Mve2V8741J1aFRNjfsfEFOvx5YQLSqnSAFbrfov/jYOGyPvk9uuG
         KtPkVT2AFsVb8BD6T6+pjEyS5OmdpBgwQF6c4=
MIME-Version: 1.0
Received: by 10.90.23.21 with SMTP id 21mr13787659agw.59.1256717658144; Wed, 
	28 Oct 2009 01:14:18 -0700 (PDT)
Date: Wed, 28 Oct 2009 16:14:18 +0800
Message-ID: <5c86176b0910280114i79370febrfb4bbdd833c64811@mail.gmail.com>
Subject: Need Help: The problem with text key of MapFile
From: lei wang <hadoopmaillist@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00148534bcba98c1af0476fa6123

--00148534bcba98c1af0476fa6123
Content-Type: text/plain; charset=ISO-8859-1

Hi, friends
I need store the web pages(a huge one) in the MapFile of the hadoop, So i
did use the url as the key, and its type is "text", When  writring the
records into the mapfile, it give an error as "out of order", which type
should I choose to represent  the key "url", can anyone give me some detail
answer, thanks for you help.

--00148534bcba98c1af0476fa6123--

From general-return-660-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 28 08:26:39 2009
Return-Path: <general-return-660-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 28415 invoked from network); 28 Oct 2009 08:26:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Oct 2009 08:26:39 -0000
Received: (qmail 54265 invoked by uid 500); 28 Oct 2009 08:26:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54186 invoked by uid 500); 28 Oct 2009 08:26:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54176 invoked by uid 99); 28 Oct 2009 08:26:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Oct 2009 08:26:38 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of zjffdu@gmail.com designates 209.85.160.50 as permitted sender)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Oct 2009 08:26:30 +0000
Received: by pwi4 with SMTP id 4so723467pwi.29
        for <general@hadoop.apache.org>; Wed, 28 Oct 2009 01:26:09 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=EBh6ycktuSXMJGnNzVMwcZ9Nr23qlp3PBOD0vYZeU1w=;
        b=o5Ny7MU4KHaEeISwhyZu6cHrVJV8aT/8s7A+nOHgWcMe85EN7gXjIuHzRfTKXQyXMH
         ynlH+pRvVeXZvWHcyWIA+RoLPVZZombvkLuvFaK1Fr8g+roI8hW+ganeJfmmhl1mJz9n
         GejPK4zJJK8u8MR97lbEqU+I/wsaUgcxP1QJ4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=kRFxHzJDRHFEc4B2oDqCHo4InVZxRhoK8ArWO9h2chHVBIC00NtqoQWxNxwPOSVRDh
         N99a3mVHA5HOAhtbpWE9PeAhc6d8Jo6l5WKgauDp/+UrVjJT0ay7ibtn3DZf2ut17OIR
         xmnND3wEX+fEABatCpWeWev85wFt3d4vOpSL8=
MIME-Version: 1.0
Received: by 10.142.120.1 with SMTP id s1mr871117wfc.245.1256718369431; Wed, 
	28 Oct 2009 01:26:09 -0700 (PDT)
In-Reply-To: <5c86176b0910280114i79370febrfb4bbdd833c64811@mail.gmail.com>
References: <5c86176b0910280114i79370febrfb4bbdd833c64811@mail.gmail.com>
Date: Wed, 28 Oct 2009 16:26:09 +0800
Message-ID: <8211a1320910280126t62f0101fg547acf8fc0f702d7@mail.gmail.com>
Subject: Re: Need Help: The problem with text key of MapFile
From: Jeff Zhang <zjffdu@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636e0b9c6fe2cb30476fa8bf9
X-Virus-Checked: Checked by ClamAV on apache.org

--001636e0b9c6fe2cb30476fa8bf9
Content-Type: text/plain; charset=UTF-8

Hi Wang,

The keys of MapFile should be in order, so when you add records into
MapFile, you should make sure you insert them in order

Best Regards,

Jeff Zhang


On Wed, Oct 28, 2009 at 4:14 PM, lei wang <hadoopmaillist@gmail.com> wrote:

> Hi, friends
> I need store the web pages(a huge one) in the MapFile of the hadoop, So i
> did use the url as the key, and its type is "text", When  writring the
> records into the mapfile, it give an error as "out of order", which type
> should I choose to represent  the key "url", can anyone give me some detail
> answer, thanks for you help.
>

--001636e0b9c6fe2cb30476fa8bf9--

From general-return-661-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 28 08:48:16 2009
Return-Path: <general-return-661-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31934 invoked from network); 28 Oct 2009 08:48:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Oct 2009 08:48:16 -0000
Received: (qmail 78408 invoked by uid 500); 28 Oct 2009 08:48:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78336 invoked by uid 500); 28 Oct 2009 08:48:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78324 invoked by uid 99); 28 Oct 2009 08:48:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Oct 2009 08:48:15 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of hadoopmaillist@gmail.com designates 209.85.210.194 as permitted sender)
Received: from [209.85.210.194] (HELO mail-yx0-f194.google.com) (209.85.210.194)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Oct 2009 08:48:12 +0000
Received: by yxe32 with SMTP id 32so570096yxe.5
        for <general@hadoop.apache.org>; Wed, 28 Oct 2009 01:47:52 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=xrYAcS4IY2jjzpJ7MglaweJaXDhYr9qHU+U/BxarTvY=;
        b=ZWR7AO+E4yQrTFuRy8hA2jm5MLfjqhzu/L7a+PkekbQNi1VyNZvXGJ/HfyO4fOC+nu
         xm9H+Fv/KxC3gtkM00Whj1QE74mRq6ijsC/MgtwQVks4HUuLdoo/cPMGdsN0liqWYXhN
         vVA02byK2Z1fWqeyopZaDCwPNc+juUJFto3ws=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=dvJEiZOOhVjbTDUzRA9ejLAf++G2tbUylyscgKV8U5hRbKwcg+r+HCEfoCnDYUA9Nt
         ertuRyGYOazBlD+LDYAt/bWdCmwxt8uksAhCND8wb6RshRX8XZvo/pLRMjLsSRoOE7Oa
         qE8AhOY/l1bpr9IVde6dMygeZrlzEGVSwHjwI=
MIME-Version: 1.0
Received: by 10.90.62.22 with SMTP id k22mr9955264aga.48.1256719671784; Wed, 
	28 Oct 2009 01:47:51 -0700 (PDT)
In-Reply-To: <8211a1320910280126t62f0101fg547acf8fc0f702d7@mail.gmail.com>
References: <5c86176b0910280114i79370febrfb4bbdd833c64811@mail.gmail.com>
	 <8211a1320910280126t62f0101fg547acf8fc0f702d7@mail.gmail.com>
Date: Wed, 28 Oct 2009 16:47:51 +0800
Message-ID: <5c86176b0910280147s593aac15lc08e08b714135aea@mail.gmail.com>
Subject: Re: Need Help: The problem with text key of MapFile
From: lei wang <hadoopmaillist@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016362839e69e764f0476fad915

--0016362839e69e764f0476fad915
Content-Type: text/plain; charset=ISO-8859-1

but now, "url" is not in order,  must the key be intwritable ? should it  be
comparable ?
How to make sure them in order?sort it first?
I just want to insert the pages for  random acess by "url ".

On Wed, Oct 28, 2009 at 4:26 PM, Jeff Zhang <zjffdu@gmail.com> wrote:

> Hi Wang,
>
> The keys of MapFile should be in order, so when you add records into
> MapFile, you should make sure you insert them in order
>
> Best Regards,
>
> Jeff Zhang
>
>
> On Wed, Oct 28, 2009 at 4:14 PM, lei wang <hadoopmaillist@gmail.com>
> wrote:
>
> > Hi, friends
> > I need store the web pages(a huge one) in the MapFile of the hadoop, So i
> > did use the url as the key, and its type is "text", When  writring the
> > records into the mapfile, it give an error as "out of order", which type
> > should I choose to represent  the key "url", can anyone give me some
> detail
> > answer, thanks for you help.
> >
>

--0016362839e69e764f0476fad915--

From general-return-662-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 28 09:12:27 2009
Return-Path: <general-return-662-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38298 invoked from network); 28 Oct 2009 09:12:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Oct 2009 09:12:27 -0000
Received: (qmail 97509 invoked by uid 500); 28 Oct 2009 09:12:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97454 invoked by uid 500); 28 Oct 2009 09:12:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97444 invoked by uid 99); 28 Oct 2009 09:12:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Oct 2009 09:12:26 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of zjffdu@gmail.com designates 209.85.160.50 as permitted sender)
Received: from [209.85.160.50] (HELO mail-pw0-f50.google.com) (209.85.160.50)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Oct 2009 09:12:18 +0000
Received: by pwi4 with SMTP id 4so740612pwi.29
        for <general@hadoop.apache.org>; Wed, 28 Oct 2009 02:11:56 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=BAioKTFm/huReGXArKGLPbf0DI6a9azitGsz4GMPRCc=;
        b=sO7SeaVj4VyDSPCqJG92E69pqgRtVXzpl/4bsnelAtP45v3LRvHHCKic9mxicA5zD8
         rPVe2Kttwu4DZtezukTDiV5mHJMBiGN6WeRJOfXONC47ORq4ndgqvXPQ8n7/8Bc1Ltcr
         pBnaWyQVUpCL1/x/E0k8FmpaauwThHxUy25VI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=NapSjlX6RhiXY4FbRHevH6A3TE19N6opEwZ+WgDJXEE77rzfVsYAvb2bTIKHnJkYjY
         QnDt7DCKeTIxgXv+UDTiv8Z0M+fEsmje/7hzCzWyQAYUaoDrbNbp2JfzZoAU2JqD9tNr
         GKOvaNxvp/5/zx5a01n4ZSgEGhANLikhbW/1A=
MIME-Version: 1.0
Received: by 10.143.21.30 with SMTP id y30mr1444970wfi.229.1256721116723; Wed, 
	28 Oct 2009 02:11:56 -0700 (PDT)
In-Reply-To: <5c86176b0910280147s593aac15lc08e08b714135aea@mail.gmail.com>
References: <5c86176b0910280114i79370febrfb4bbdd833c64811@mail.gmail.com>
	 <8211a1320910280126t62f0101fg547acf8fc0f702d7@mail.gmail.com>
	 <5c86176b0910280147s593aac15lc08e08b714135aea@mail.gmail.com>
Date: Wed, 28 Oct 2009 17:11:56 +0800
Message-ID: <8211a1320910280211tc4f19fdk88c695939b629b91@mail.gmail.com>
Subject: Re: Need Help: The problem with text key of MapFile
From: Jeff Zhang <zjffdu@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502ce03be7ca70476fb2f44
X-Virus-Checked: Checked by ClamAV on apache.org

--00504502ce03be7ca70476fb2f44
Content-Type: text/plain; charset=UTF-8

I do not know why you need use MapFile, could you use SequenceFile instead ?

The MapFile's advantage is its read performance, because it build index on
its keys. So its keys must be in order.

If you really want to use MapFile, you can first write your data to
SequenceFile and then covert it to MapFile.

About  how to convert SequenceFile to MapFile:
1. Sort the SequenceFile using sort in examples of hadoop
2. create index for the output of the above step. then you get both of the
data file and index file


You an refer Tom Whilte's book "Hadoop definitive guide" for details about
how to convert SequenceFile into MapFile

Jeff Zhang



On Wed, Oct 28, 2009 at 4:47 PM, lei wang <hadoopmaillist@gmail.com> wrote:

> but now, "url" is not in order,  must the key be intwritable ? should it
>  be
> comparable ?
> How to make sure them in order?sort it first?
> I just want to insert the pages for  random acess by "url ".
>
> On Wed, Oct 28, 2009 at 4:26 PM, Jeff Zhang <zjffdu@gmail.com> wrote:
>
> > Hi Wang,
> >
> > The keys of MapFile should be in order, so when you add records into
> > MapFile, you should make sure you insert them in order
> >
> > Best Regards,
> >
> > Jeff Zhang
> >
> >
> > On Wed, Oct 28, 2009 at 4:14 PM, lei wang <hadoopmaillist@gmail.com>
> > wrote:
> >
> > > Hi, friends
> > > I need store the web pages(a huge one) in the MapFile of the hadoop, So
> i
> > > did use the url as the key, and its type is "text", When  writring the
> > > records into the mapfile, it give an error as "out of order", which
> type
> > > should I choose to represent  the key "url", can anyone give me some
> > detail
> > > answer, thanks for you help.
> > >
> >
>

--00504502ce03be7ca70476fb2f44--

From general-return-663-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 28 22:54:37 2009
Return-Path: <general-return-663-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35064 invoked from network); 28 Oct 2009 22:54:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Oct 2009 22:54:36 -0000
Received: (qmail 29454 invoked by uid 500); 28 Oct 2009 22:54:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29359 invoked by uid 500); 28 Oct 2009 22:54:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29346 invoked by uid 99); 28 Oct 2009 22:54:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Oct 2009 22:54:35 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of slimtbourbi@gmail.com designates 209.85.219.226 as permitted sender)
Received: from [209.85.219.226] (HELO mail-ew0-f226.google.com) (209.85.219.226)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Oct 2009 22:54:27 +0000
Received: by ewy26 with SMTP id 26so1311423ewy.29
        for <general@hadoop.apache.org>; Wed, 28 Oct 2009 15:54:07 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=Kg79npvJ8QKg5hWbIhirpfjYY8PP7odBatHI4Xi/glY=;
        b=nhnbhGKYm4FzXu3e1nQgDkCRyfF+HUJ0S/HmnhxUDxnhV4b9PgVFTpN2aD6Ad/VWup
         Tzu4WWJkhc4fG3CY5CoMVMzPpcUqkncYPNaMCh9b34d4r1UZmfeLq7wlyLrViOXp8jkd
         ge3vWlHMZqpf713edPPiFI3Q+xXyP8Fsexexo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=od4CDb+dUxENn23RI47VhxIFCDDbTL71iVzCWcCWljmrpcSuHKuQmiBjSmz9B/Ezxo
         tlhG8irgkMlaKngNvDXD8zsSFEvonKs16HzKpPIP8EcNfaGMN0YZnQex5gGSfufXxy/k
         /UBQKtZMzyrk2UmXlJzhmtQVC3QEondUzB2Og=
MIME-Version: 1.0
Received: by 10.216.86.131 with SMTP id w3mr242104wee.156.1256770447736; Wed, 
	28 Oct 2009 15:54:07 -0700 (PDT)
Date: Wed, 28 Oct 2009 23:54:07 +0100
Message-ID: <78aa0af90910281554v441325faxbcd52ca8144457cb@mail.gmail.com>
Subject: try hadoop on amazon
From: slim tebourbi <slimtbourbi@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d7787f19ff98047706acc2
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6d7787f19ff98047706acc2
Content-Type: text/plain; charset=ISO-8859-1

Hello guys,
Is there any way to use the amazon's services for free just to make tests?
Or any other services that allow this?
thxs.

--0016e6d7787f19ff98047706acc2--

From general-return-664-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 29 00:37:18 2009
Return-Path: <general-return-664-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66712 invoked from network); 29 Oct 2009 00:37:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Oct 2009 00:37:18 -0000
Received: (qmail 14501 invoked by uid 500); 29 Oct 2009 00:37:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14402 invoked by uid 500); 29 Oct 2009 00:37:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14391 invoked by uid 99); 29 Oct 2009 00:37:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Oct 2009 00:37:16 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of matei@eecs.berkeley.edu designates 169.229.60.87 as permitted sender)
Received: from [169.229.60.87] (HELO gateway0.EECS.Berkeley.EDU) (169.229.60.87)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Oct 2009 00:37:07 +0000
Received: from dhcp-44-233.eecs.berkeley.edu (dhcp-44-233.EECS.Berkeley.EDU [128.32.44.233])
	(authenticated bits=0)
	by gateway0.EECS.Berkeley.EDU (8.14.3/8.13.5) with ESMTP id n9T0aiIj001476
	(version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT)
	for <general@hadoop.apache.org>; Wed, 28 Oct 2009 17:36:45 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes
Mime-Version: 1.0 (Apple Message framework v1076)
Subject: Re: try hadoop on amazon
From: Matei Zaharia <matei@eecs.berkeley.edu>
In-Reply-To: <78aa0af90910281554v441325faxbcd52ca8144457cb@mail.gmail.com>
Date: Wed, 28 Oct 2009 17:36:43 -0700
Content-Transfer-Encoding: 7bit
Message-Id: <35ECE16E-82D6-4BE8-ABBE-0B765C298E3D@eecs.berkeley.edu>
References: <78aa0af90910281554v441325faxbcd52ca8144457cb@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1076)
X-Virus-Checked: Checked by ClamAV on apache.org

If you just want to develop and test Hadoop applications, you can run  
Hadoop on your own machine by following the Quick Start instructions  
linked at http://hadoop.apache.org/common/docs/r0.20.0/. If you want  
to run on a cluster, Amazon is very cheap for small computations  
(several dollars per hour).

On Oct 28, 2009, at 3:54 PM, slim tebourbi wrote:

> Hello guys,
> Is there any way to use the amazon's services for free just to make  
> tests?
> Or any other services that allow this?
> thxs.


From general-return-665-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 29 00:53:41 2009
Return-Path: <general-return-665-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76819 invoked from network); 29 Oct 2009 00:53:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Oct 2009 00:53:41 -0000
Received: (qmail 22703 invoked by uid 500); 29 Oct 2009 00:53:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22630 invoked by uid 500); 29 Oct 2009 00:53:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22620 invoked by uid 99); 29 Oct 2009 00:53:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Oct 2009 00:53:40 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of hadoopmaillist@gmail.com designates 209.85.211.185 as permitted sender)
Received: from [209.85.211.185] (HELO mail-yw0-f185.google.com) (209.85.211.185)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Oct 2009 00:53:38 +0000
Received: by ywh15 with SMTP id 15so1531308ywh.5
        for <general@hadoop.apache.org>; Wed, 28 Oct 2009 17:53:17 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=ExACVlwQe4/LC0yJmccT3F7XKaj4a3CqTCm9uZZnBQc=;
        b=fSkXJ5kT4qBW1ud+w2MOdRX9+yoCtuZatfrHMxJpMhYe7XF/SbwhMIDFDMlR8Z246l
         feOkWK1//diOVIcILLgjn9Uyq1jkKqr3LUxEMfrRnqmQJIEqYRsx579yNwih7glrDYOk
         drHLZzG909cQNHupvdMkMUj06d4mIR7jdSlQ4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=epOq2yQSsoIWilZkXUdCmi0obnGo4soTA06rxYesycsrceRGamkWAvIYQ5YcfZ6RNl
         2buEzY9Osund4KXK6R5c4KwaKtCqbx1ojpM4ZbSIyiIlC4DRLnNKQ7vQC/aryjDota+d
         8Oc9dA0gJoDCRXRq85ho+vpuLv9oMqIk2aScc=
MIME-Version: 1.0
Received: by 10.101.105.1 with SMTP id h1mr730907anm.126.1256777596954; Wed, 
	28 Oct 2009 17:53:16 -0700 (PDT)
In-Reply-To: <8211a1320910280211tc4f19fdk88c695939b629b91@mail.gmail.com>
References: <5c86176b0910280114i79370febrfb4bbdd833c64811@mail.gmail.com>
	 <8211a1320910280126t62f0101fg547acf8fc0f702d7@mail.gmail.com>
	 <5c86176b0910280147s593aac15lc08e08b714135aea@mail.gmail.com>
	 <8211a1320910280211tc4f19fdk88c695939b629b91@mail.gmail.com>
Date: Thu, 29 Oct 2009 08:53:16 +0800
Message-ID: <5c86176b0910281753y5dffb328m3de2f48cebd3bca6@mail.gmail.com>
Subject: Re: Need Help: The problem with text key of MapFile
From: lei wang <hadoopmaillist@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

hi,juff, thanks for your comments.
   I did read this book early, I use MapFile to store my web pages for
random access.
First I think the SquenceFile conversion as a solution, howerve, the
problem is that I need append the new pages to the MapFile by minute
or second, so I didn't think SquenceFile conversion can manage this.
Would you give me some suggestion? Think your very much!

Best wishes.

On 10/28/09, Jeff Zhang <zjffdu@gmail.com> wrote:
> I do not know why you need use MapFile, could you use SequenceFile instead ?
>
> The MapFile's advantage is its read performance, because it build index on
> its keys. So its keys must be in order.
>
> If you really want to use MapFile, you can first write your data to
> SequenceFile and then covert it to MapFile.
>
> About  how to convert SequenceFile to MapFile:
> 1. Sort the SequenceFile using sort in examples of hadoop
> 2. create index for the output of the above step. then you get both of the
> data file and index file
>
>
> You an refer Tom Whilte's book "Hadoop definitive guide" for details about
> how to convert SequenceFile into MapFile
>
> Jeff Zhang
>
>
>
> On Wed, Oct 28, 2009 at 4:47 PM, lei wang <hadoopmaillist@gmail.com> wrote:
>
>> but now, "url" is not in order,  must the key be intwritable ? should it
>>  be
>> comparable ?
>> How to make sure them in order?sort it first?
>> I just want to insert the pages for  random acess by "url ".
>>
>> On Wed, Oct 28, 2009 at 4:26 PM, Jeff Zhang <zjffdu@gmail.com> wrote:
>>
>> > Hi Wang,
>> >
>> > The keys of MapFile should be in order, so when you add records into
>> > MapFile, you should make sure you insert them in order
>> >
>> > Best Regards,
>> >
>> > Jeff Zhang
>> >
>> >
>> > On Wed, Oct 28, 2009 at 4:14 PM, lei wang <hadoopmaillist@gmail.com>
>> > wrote:
>> >
>> > > Hi, friends
>> > > I need store the web pages(a huge one) in the MapFile of the hadoop,
>> > > So
>> i
>> > > did use the url as the key, and its type is "text", When  writring the
>> > > records into the mapfile, it give an error as "out of order", which
>> type
>> > > should I choose to represent  the key "url", can anyone give me some
>> > detail
>> > > answer, thanks for you help.
>> > >
>> >
>>
>

From general-return-666-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 29 02:17:19 2009
Return-Path: <general-return-666-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95508 invoked from network); 29 Oct 2009 02:17:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Oct 2009 02:17:19 -0000
Received: (qmail 81028 invoked by uid 500); 29 Oct 2009 02:17:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80959 invoked by uid 500); 29 Oct 2009 02:17:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80949 invoked by uid 99); 29 Oct 2009 02:17:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Oct 2009 02:17:18 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of zjffdu@gmail.com designates 209.85.216.204 as permitted sender)
Received: from [209.85.216.204] (HELO mail-px0-f204.google.com) (209.85.216.204)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Oct 2009 02:17:10 +0000
Received: by pxi42 with SMTP id 42so971891pxi.5
        for <general@hadoop.apache.org>; Wed, 28 Oct 2009 19:16:48 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=Ff1oz9ZLaJ16D0RA08b5/Fhes8xdRz+IokgpZv5hOtY=;
        b=pRa9Anp2GEY2hpInN5VLVVUMwT/W8vSDMaIXOhux0F+864WPqcwz7uHtjfhmj2wZg6
         2Y/w+qECI+Irc9I733vDMN4GAAEJUKFDq4MEzo8gLosMz2IBva8vIyqr/4WWsku6KIjh
         7SXc0/RCYw9CgA9q3RSgcSDkYtKNjgVLb5BGg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=fz5YroDPnQjtjQgotVpT1WUFIfAnPJVu7OeJA3RxRl3izD5yXtbQe2qOm/VudHhhCk
         yGMEpPirn/Vln32x9sliArlRQSm42Z0RHRL6U7J00f+tLxE+jbDLUf/CuoC892qF12eC
         PdYsOuAxqDAn4+QNxGeYz4CQRSKKJVDPxwGuM=
MIME-Version: 1.0
Received: by 10.142.248.41 with SMTP id v41mr1554084wfh.284.1256782608830; 
	Wed, 28 Oct 2009 19:16:48 -0700 (PDT)
In-Reply-To: <5c86176b0910281753y5dffb328m3de2f48cebd3bca6@mail.gmail.com>
References: <5c86176b0910280114i79370febrfb4bbdd833c64811@mail.gmail.com>
	 <8211a1320910280126t62f0101fg547acf8fc0f702d7@mail.gmail.com>
	 <5c86176b0910280147s593aac15lc08e08b714135aea@mail.gmail.com>
	 <8211a1320910280211tc4f19fdk88c695939b629b91@mail.gmail.com>
	 <5c86176b0910281753y5dffb328m3de2f48cebd3bca6@mail.gmail.com>
Date: Thu, 29 Oct 2009 10:16:48 +0800
Message-ID: <8211a1320910281916v3ea4f5c1vc4b23fa0260c753@mail.gmail.com>
Subject: Re: Need Help: The problem with text key of MapFile
From: Jeff Zhang <zjffdu@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502cb28f58ff7047709801f
X-Virus-Checked: Checked by ClamAV on apache.org

--00504502cb28f58ff7047709801f
Content-Type: text/plain; charset=UTF-8

I guess maybe HBase will be fit for you.   HBase is a distributed database
built upon Hadoop.
You can use the url as the row key and put other fields into columns.

then you can retrieve the web page through HBase Client API and insert new
web page into it. The performance of HBase 0.20 is good enough for you.

Best Regards,
Jeff zhang


On Thu, Oct 29, 2009 at 8:53 AM, lei wang <hadoopmaillist@gmail.com> wrote:

> hi,juff, thanks for your comments.
>   I did read this book early, I use MapFile to store my web pages for
> random access.
> First I think the SquenceFile conversion as a solution, howerve, the
> problem is that I need append the new pages to the MapFile by minute
> or second, so I didn't think SquenceFile conversion can manage this.
> Would you give me some suggestion? Think your very much!
>
> Best wishes.
>
> On 10/28/09, Jeff Zhang <zjffdu@gmail.com> wrote:
> > I do not know why you need use MapFile, could you use SequenceFile
> instead ?
> >
> > The MapFile's advantage is its read performance, because it build index
> on
> > its keys. So its keys must be in order.
> >
> > If you really want to use MapFile, you can first write your data to
> > SequenceFile and then covert it to MapFile.
> >
> > About  how to convert SequenceFile to MapFile:
> > 1. Sort the SequenceFile using sort in examples of hadoop
> > 2. create index for the output of the above step. then you get both of
> the
> > data file and index file
> >
> >
> > You an refer Tom Whilte's book "Hadoop definitive guide" for details
> about
> > how to convert SequenceFile into MapFile
> >
> > Jeff Zhang
> >
> >
> >
> > On Wed, Oct 28, 2009 at 4:47 PM, lei wang <hadoopmaillist@gmail.com>
> wrote:
> >
> >> but now, "url" is not in order,  must the key be intwritable ? should it
> >>  be
> >> comparable ?
> >> How to make sure them in order?sort it first?
> >> I just want to insert the pages for  random acess by "url ".
> >>
> >> On Wed, Oct 28, 2009 at 4:26 PM, Jeff Zhang <zjffdu@gmail.com> wrote:
> >>
> >> > Hi Wang,
> >> >
> >> > The keys of MapFile should be in order, so when you add records into
> >> > MapFile, you should make sure you insert them in order
> >> >
> >> > Best Regards,
> >> >
> >> > Jeff Zhang
> >> >
> >> >
> >> > On Wed, Oct 28, 2009 at 4:14 PM, lei wang <hadoopmaillist@gmail.com>
> >> > wrote:
> >> >
> >> > > Hi, friends
> >> > > I need store the web pages(a huge one) in the MapFile of the hadoop,
> >> > > So
> >> i
> >> > > did use the url as the key, and its type is "text", When  writring
> the
> >> > > records into the mapfile, it give an error as "out of order", which
> >> type
> >> > > should I choose to represent  the key "url", can anyone give me some
> >> > detail
> >> > > answer, thanks for you help.
> >> > >
> >> >
> >>
> >
>

--00504502cb28f58ff7047709801f--

From general-return-667-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 29 02:22:15 2009
Return-Path: <general-return-667-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96366 invoked from network); 29 Oct 2009 02:22:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Oct 2009 02:22:15 -0000
Received: (qmail 86083 invoked by uid 500); 29 Oct 2009 02:22:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85998 invoked by uid 500); 29 Oct 2009 02:22:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85988 invoked by uid 99); 29 Oct 2009 02:22:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Oct 2009 02:22:14 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=AWL,BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of hadoopmaillist@gmail.com designates 209.85.211.185 as permitted sender)
Received: from [209.85.211.185] (HELO mail-yw0-f185.google.com) (209.85.211.185)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Oct 2009 02:22:12 +0000
Received: by ywh15 with SMTP id 15so1601243ywh.5
        for <general@hadoop.apache.org>; Wed, 28 Oct 2009 19:21:51 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=ek75P3Fv4EMf3UxvOXdmflp4fTdQdwpl6FLTNKAaP0c=;
        b=H+q6bqsb85THFoJO7Ll/MDkX1ofZKULCl2egLfSoXw2bpy6iHk/lIaw4qkxP9empD6
         a15v1rtplxvbtfQ4Yma4GZdWODgT0sYtGGo2XVxbEfzz/yntMVgSTe9sZj5Q/HbmxYzK
         nWuQJBqoYpzO3hCdx53cg7lqLWNCm9qA4Z7BQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=POnfzUwhyOVHudimFLL/HvZQAqJnUJgzfMnbnoBxha9ORqFZTW8r4VJLAj4dbniLG6
         s8Dr3PseCe5el/PBa4BA4qhpK00QwTPfIzhsuKvcO2ajsGEgEcKGvvkQpGVbfE5W8mLH
         El5FboR6sSPXlJATwUW8R0sy4F3sRKhIri0CQ=
MIME-Version: 1.0
Received: by 10.100.224.3 with SMTP id w3mr718520ang.108.1256782911049; Wed, 
	28 Oct 2009 19:21:51 -0700 (PDT)
In-Reply-To: <8211a1320910281916v3ea4f5c1vc4b23fa0260c753@mail.gmail.com>
References: <5c86176b0910280114i79370febrfb4bbdd833c64811@mail.gmail.com>
	 <8211a1320910280126t62f0101fg547acf8fc0f702d7@mail.gmail.com>
	 <5c86176b0910280147s593aac15lc08e08b714135aea@mail.gmail.com>
	 <8211a1320910280211tc4f19fdk88c695939b629b91@mail.gmail.com>
	 <5c86176b0910281753y5dffb328m3de2f48cebd3bca6@mail.gmail.com>
	 <8211a1320910281916v3ea4f5c1vc4b23fa0260c753@mail.gmail.com>
Date: Thu, 29 Oct 2009 10:21:50 +0800
Message-ID: <5c86176b0910281921x4d19970aqb5ddd0aca435bb80@mail.gmail.com>
Subject: Re: Need Help: The problem with text key of MapFile
From: lei wang <hadoopmaillist@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163623b154f90f9d04770992ce

--00163623b154f90f9d04770992ce
Content-Type: text/plain; charset=ISO-8859-1

Oh, I  have tried hbase in the early.
But I think HDFS may give me a choice.
Thanks.

On Thu, Oct 29, 2009 at 10:16 AM, Jeff Zhang <zjffdu@gmail.com> wrote:

> I guess maybe HBase will be fit for you.   HBase is a distributed database
> built upon Hadoop.
> You can use the url as the row key and put other fields into columns.
>
> then you can retrieve the web page through HBase Client API and insert new
> web page into it. The performance of HBase 0.20 is good enough for you.
>
> Best Regards,
> Jeff zhang
>
>
> On Thu, Oct 29, 2009 at 8:53 AM, lei wang <hadoopmaillist@gmail.com>
> wrote:
>
> > hi,juff, thanks for your comments.
> >   I did read this book early, I use MapFile to store my web pages for
> > random access.
> > First I think the SquenceFile conversion as a solution, howerve, the
> > problem is that I need append the new pages to the MapFile by minute
> > or second, so I didn't think SquenceFile conversion can manage this.
> > Would you give me some suggestion? Think your very much!
> >
> > Best wishes.
> >
> > On 10/28/09, Jeff Zhang <zjffdu@gmail.com> wrote:
> > > I do not know why you need use MapFile, could you use SequenceFile
> > instead ?
> > >
> > > The MapFile's advantage is its read performance, because it build index
> > on
> > > its keys. So its keys must be in order.
> > >
> > > If you really want to use MapFile, you can first write your data to
> > > SequenceFile and then covert it to MapFile.
> > >
> > > About  how to convert SequenceFile to MapFile:
> > > 1. Sort the SequenceFile using sort in examples of hadoop
> > > 2. create index for the output of the above step. then you get both of
> > the
> > > data file and index file
> > >
> > >
> > > You an refer Tom Whilte's book "Hadoop definitive guide" for details
> > about
> > > how to convert SequenceFile into MapFile
> > >
> > > Jeff Zhang
> > >
> > >
> > >
> > > On Wed, Oct 28, 2009 at 4:47 PM, lei wang <hadoopmaillist@gmail.com>
> > wrote:
> > >
> > >> but now, "url" is not in order,  must the key be intwritable ? should
> it
> > >>  be
> > >> comparable ?
> > >> How to make sure them in order?sort it first?
> > >> I just want to insert the pages for  random acess by "url ".
> > >>
> > >> On Wed, Oct 28, 2009 at 4:26 PM, Jeff Zhang <zjffdu@gmail.com> wrote:
> > >>
> > >> > Hi Wang,
> > >> >
> > >> > The keys of MapFile should be in order, so when you add records into
> > >> > MapFile, you should make sure you insert them in order
> > >> >
> > >> > Best Regards,
> > >> >
> > >> > Jeff Zhang
> > >> >
> > >> >
> > >> > On Wed, Oct 28, 2009 at 4:14 PM, lei wang <hadoopmaillist@gmail.com
> >
> > >> > wrote:
> > >> >
> > >> > > Hi, friends
> > >> > > I need store the web pages(a huge one) in the MapFile of the
> hadoop,
> > >> > > So
> > >> i
> > >> > > did use the url as the key, and its type is "text", When  writring
> > the
> > >> > > records into the mapfile, it give an error as "out of order",
> which
> > >> type
> > >> > > should I choose to represent  the key "url", can anyone give me
> some
> > >> > detail
> > >> > > answer, thanks for you help.
> > >> > >
> > >> >
> > >>
> > >
> >
>

--00163623b154f90f9d04770992ce--

From general-return-668-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 29 02:28:00 2009
Return-Path: <general-return-668-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97197 invoked from network); 29 Oct 2009 02:28:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Oct 2009 02:28:00 -0000
Received: (qmail 89481 invoked by uid 500); 29 Oct 2009 02:27:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 89406 invoked by uid 500); 29 Oct 2009 02:27:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 89396 invoked by uid 99); 29 Oct 2009 02:27:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Oct 2009 02:27:59 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [67.59.59.114] (HELO trironport1.altair.com) (67.59.59.114)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Oct 2009 02:27:51 +0000
X-IronPort-AV: E=Sophos;i="4.44,643,1249272000"; 
   d="scan'208";a="9275828"
Received: from unknown (HELO TR-EXCH07.prog.altair.com) ([204.235.24.175])
  by trironport1.altair.com with ESMTP; 28 Oct 2009 22:27:30 -0400
Received: from TR-EXCH07.prog.altair.com ([204.235.24.175]) by
 TR-EXCH07.prog.altair.com ([204.235.24.175]) with mapi; Wed, 28 Oct 2009
 22:28:35 -0400
From: Lori Ann Martin <lmartin@altair.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 28 Oct 2009 22:27:27 -0400
Subject: RE: Need Help: The problem with text key of MapFile
Thread-Topic: Need Help: The problem with text key of MapFile
Thread-Index: AcpYPs0r5gtR30s0RTOc6Flt94ng1QAAIaqA
Message-ID: <7232A3A7C53F634B8E91E65BC7833C5112C5714B34@TR-EXCH07.prog.altair.com>
References: <5c86176b0910280114i79370febrfb4bbdd833c64811@mail.gmail.com>
	 <8211a1320910280126t62f0101fg547acf8fc0f702d7@mail.gmail.com>
	 <5c86176b0910280147s593aac15lc08e08b714135aea@mail.gmail.com>
	 <8211a1320910280211tc4f19fdk88c695939b629b91@mail.gmail.com>
	 <5c86176b0910281753y5dffb328m3de2f48cebd3bca6@mail.gmail.com>
	 <8211a1320910281916v3ea4f5c1vc4b23fa0260c753@mail.gmail.com>
 <5c86176b0910281921x4d19970aqb5ddd0aca435bb80@mail.gmail.com>
In-Reply-To: <5c86176b0910281921x4d19970aqb5ddd0aca435bb80@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

heck out www.HiQube.com or www.pbsgridworks.com

-----Original Message-----
From: lei wang [mailto:hadoopmaillist@gmail.com]=20
Sent: Wednesday, October 28, 2009 7:22 PM
To: general@hadoop.apache.org
Subject: Re: Need Help: The problem with text key of MapFile

Oh, I  have tried hbase in the early.
But I think HDFS may give me a choice.
Thanks.

On Thu, Oct 29, 2009 at 10:16 AM, Jeff Zhang <zjffdu@gmail.com> wrote:

> I guess maybe HBase will be fit for you.   HBase is a distributed databas=
e
> built upon Hadoop.
> You can use the url as the row key and put other fields into columns.
>
> then you can retrieve the web page through HBase Client API and insert ne=
w
> web page into it. The performance of HBase 0.20 is good enough for you.
>
> Best Regards,
> Jeff zhang
>
>
> On Thu, Oct 29, 2009 at 8:53 AM, lei wang <hadoopmaillist@gmail.com>
> wrote:
>
> > hi,juff, thanks for your comments.
> >   I did read this book early, I use MapFile to store my web pages for
> > random access.
> > First I think the SquenceFile conversion as a solution, howerve, the
> > problem is that I need append the new pages to the MapFile by minute
> > or second, so I didn't think SquenceFile conversion can manage this.
> > Would you give me some suggestion? Think your very much!
> >
> > Best wishes.
> >
> > On 10/28/09, Jeff Zhang <zjffdu@gmail.com> wrote:
> > > I do not know why you need use MapFile, could you use SequenceFile
> > instead ?
> > >
> > > The MapFile's advantage is its read performance, because it build ind=
ex
> > on
> > > its keys. So its keys must be in order.
> > >
> > > If you really want to use MapFile, you can first write your data to
> > > SequenceFile and then covert it to MapFile.
> > >
> > > About  how to convert SequenceFile to MapFile:
> > > 1. Sort the SequenceFile using sort in examples of hadoop
> > > 2. create index for the output of the above step. then you get both o=
f
> > the
> > > data file and index file
> > >
> > >
> > > You an refer Tom Whilte's book "Hadoop definitive guide" for details
> > about
> > > how to convert SequenceFile into MapFile
> > >
> > > Jeff Zhang
> > >
> > >
> > >
> > > On Wed, Oct 28, 2009 at 4:47 PM, lei wang <hadoopmaillist@gmail.com>
> > wrote:
> > >
> > >> but now, "url" is not in order,  must the key be intwritable ? shoul=
d
> it
> > >>  be
> > >> comparable ?
> > >> How to make sure them in order?sort it first?
> > >> I just want to insert the pages for  random acess by "url ".
> > >>
> > >> On Wed, Oct 28, 2009 at 4:26 PM, Jeff Zhang <zjffdu@gmail.com> wrote=
:
> > >>
> > >> > Hi Wang,
> > >> >
> > >> > The keys of MapFile should be in order, so when you add records in=
to
> > >> > MapFile, you should make sure you insert them in order
> > >> >
> > >> > Best Regards,
> > >> >
> > >> > Jeff Zhang
> > >> >
> > >> >
> > >> > On Wed, Oct 28, 2009 at 4:14 PM, lei wang <hadoopmaillist@gmail.co=
m
> >
> > >> > wrote:
> > >> >
> > >> > > Hi, friends
> > >> > > I need store the web pages(a huge one) in the MapFile of the
> hadoop,
> > >> > > So
> > >> i
> > >> > > did use the url as the key, and its type is "text", When  writri=
ng
> > the
> > >> > > records into the mapfile, it give an error as "out of order",
> which
> > >> type
> > >> > > should I choose to represent  the key "url", can anyone give me
> some
> > >> > detail
> > >> > > answer, thanks for you help.
> > >> > >
> > >> >
> > >>
> > >
> >
>

From general-return-669-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 29 04:30:50 2009
Return-Path: <general-return-669-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 18148 invoked from network); 29 Oct 2009 04:30:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Oct 2009 04:30:49 -0000
Received: (qmail 65253 invoked by uid 500); 29 Oct 2009 04:30:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 65112 invoked by uid 500); 29 Oct 2009 04:30:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 65094 invoked by uid 99); 29 Oct 2009 04:30:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Oct 2009 04:30:48 +0000
X-ASF-Spam-Status: No, hits=-2.6 required=5.0
	tests=BAYES_00,HTML_MESSAGE
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of hkumar.arora@gmail.com designates 209.85.223.171 as permitted sender)
Received: from [209.85.223.171] (HELO mail-iw0-f171.google.com) (209.85.223.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Oct 2009 04:30:45 +0000
Received: by iwn1 with SMTP id 1so1183968iwn.2
        for <multiple recipients>; Wed, 28 Oct 2009 21:30:24 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:cc:content-type;
        bh=+P4ps4m4tFgYn4FItNPq5QoE0G0D/NtxzK96/jGUXbk=;
        b=lInWYBrZAidvuCUUz5lKl7qPm+hEoHfFA9/G1IG8gr6LU313/NB02cFILlao6uTb8H
         jAGwtard2HJ8Udhfn1zBPYAOXfWoEbtRJsIPjhEvbtRj+o0t7IATt2uzeQsxq8ClqJ1c
         KcRCV+pzogWtQhms6xU6kTSZnQe2pVRFt20oE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:cc:content-type;
        b=xzoHYTyKZYeIW07zjNWRvW8PiTuJxOp6fhhPljD223Dn58zhMp2sf9LARXlFPQo3Hy
         M/aQ2YEUkU+D49/tlg3udEzjwb7cq0ganrtkNLq268iSR0WQBpsE+YuKmlKEPitZmA74
         edYcXR/QVYbgDyouJzM6nwAB4+bL712bEFfT8=
MIME-Version: 1.0
Received: by 10.231.10.16 with SMTP id n16mr2133517ibn.24.1256790624609; Wed, 
	28 Oct 2009 21:30:24 -0700 (PDT)
Date: Thu, 29 Oct 2009 13:30:24 +0900
Message-ID: <e497591c0910282130s160678f3m561b83bc45dbfa25@mail.gmail.com>
Subject: S3 Exception for a Map Reduce job on EC2
From: Harshit Kumar <hkumar.arora@gmail.com>
To: general@hadoop.apache.org
Cc: common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000325575d12bca8c204770b5ece

--000325575d12bca8c204770b5ece
Content-Type: text/plain; charset=UTF-8

Hi

There is 1 GB of rdf/owl files that I am executing on EC2. Execution throws
the following exception
-------------------

08/11/19 16:08:27 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the
same.
org.apache.hadoop.fs.s3.S3Exception:
org.jets3t.service.S3ServiceException: S3 PUT failed for '/' XML Error
Message: <?xml version="1.0"
encoding="UTF-8"?><Error><Code>OperationAborted</Code><Message>*A
conflicting conditional operation is currently in progress against
**this** resource*. Please try
again.</Message><RequestId>324E696A4BCA8731</RequestId><HostId>{REMOVED}</HostId></Error>
	at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.createBucket(Jets3tNativeFileSystemStore.java:74)
	at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.initialize(Jets3tNativeFileSystemStore.java:63)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
	at org.apache.hadoop.fs.s3native.$Proxy2.initialize(Unknown Source)
	at org.apache.hadoop.fs.s3native.NativeS3FileSystem.initialize(NativeS3FileSystem.java:215)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1339)
	at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1351)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:213)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:175)
	at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:158)
	at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:210)
	at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
	at org.apache.hadoop.streaming.StreamJob.submitAndMonitorJob(StreamJob.java:925)
	at org.apache.hadoop.streaming.StreamJob.go(StreamJob.java:115)
	at org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreaming.java:33)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
	at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
	at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
Caused by: org.jets3t.service.S3ServiceException: S3 PUT failed for
'/' XML Error Message: <?xml version="1.0"
encoding="UTF-8"?><Error><Code>OperationAborted</Code><Message>A
conflicting conditional operation is currently in progress against
this resource. Please try
again.</Message><RequestId>{REMOVED}</RequestId><HostId>{REMOVED}</HostId></Error>
	at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:424)
	at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestPut(RestS3Service.java:734)
	at org.jets3t.service.impl.rest.httpclient.RestS3Service.createObjectImpl(RestS3Service.java:1357)
	at org.jets3t.service.impl.rest.httpclient.RestS3Service.createBucketImpl(RestS3Service.java:1234)
	at org.jets3t.service.S3Service.createBucket(S3Service.java:1390)
	at org.jets3t.service.S3Service.createBucket(S3Service.java:1158)
	at org.jets3t.service.S3Service.createBucket(S3Service.java:1177)
	at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.createBucket(Jets3tNativeFileSystemStore.java:69)
	... 29 more

---------------------
There were like 50 failed map tasks out of total of 1141 maps. Though hadoop
job tracker successfully executed failed tasks again, but if a task fails,
and then restarting it again, takes execution time. If somehow, this
exception can be avoided altogether, program execution will be faster.

Came across a JIRA post that says, the exception is because s3 tries to
create a bucket which it should not create, and a patch is also posted for
the same. Applied the patch, but still the same error.

IMO, this error is because a operation is trying to access a part of file
which is already been accessed by another map job.

Please help resolve this issue or pass on any pointers that can suggest what
is the source of error?

Thanks and Regards

H. Kumar
skype: harshit900
Blog: http://harshitkumar.wordpress.com
Website: http:/kumarharmuscat.tripod.comg
H. Kumar
Phone(Mobile): +82-10-2892-9663
Phone(Office): +82-31-
skype: harshit900
Blog: http://harshitkumar.wordpress.com
Website: http:/kumarharmuscat.tripod.com

--000325575d12bca8c204770b5ece--

From general-return-670-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 29 04:36:09 2009
Return-Path: <general-return-670-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19097 invoked from network); 29 Oct 2009 04:36:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Oct 2009 04:36:09 -0000
Received: (qmail 72129 invoked by uid 500); 29 Oct 2009 04:36:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72033 invoked by uid 500); 29 Oct 2009 04:36:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72014 invoked by uid 99); 29 Oct 2009 04:36:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Oct 2009 04:36:08 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of hkumar.arora@gmail.com designates 209.85.223.171 as permitted sender)
Received: from [209.85.223.171] (HELO mail-iw0-f171.google.com) (209.85.223.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Oct 2009 04:35:56 +0000
Received: by iwn1 with SMTP id 1so1186026iwn.2
        for <multiple recipients>; Wed, 28 Oct 2009 21:35:35 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:cc:content-type;
        bh=/Ax7fpwjTtFK4gH3CZ7EFjTIpt2VF+GJAnhdDmvIQXc=;
        b=h1DO3CJIhH30raqFO9qdg4/NTpPMNAwrzPfbhTejLZLZY+hlTCp9Ya5t6GLVnFNU67
         W0R5ZRTcRo6G4UcoamiU/r21iOg9pgfjrSIQemOZb8aMxhhaTkaJf8m7fAop0Gi5proJ
         caWdXU+MRkjd84/XpU3DMysqAz0efy8NymgNo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=aUa8cua+r9RNLycgn+zPiaiHvX2L5o4UhBzvrgW4uY7tbxWa7mqW9VPJLsxv3+mMMo
         ZBXhTs0reOk4FFZBe2+kLiJd7oGysDBcv7Wn16wfDM3MX0vYdD8B5Dvw2djjjcqRWCC4
         baT26yGcv8L98gIMpuFIPwYedZbXkyxIqP68Y=
MIME-Version: 1.0
Received: by 10.231.122.103 with SMTP id k39mr4570229ibr.10.1256790933361; 
	Wed, 28 Oct 2009 21:35:33 -0700 (PDT)
In-Reply-To: <e497591c0910282130s160678f3m561b83bc45dbfa25@mail.gmail.com>
References: <e497591c0910282130s160678f3m561b83bc45dbfa25@mail.gmail.com>
Date: Thu, 29 Oct 2009 13:35:33 +0900
Message-ID: <e497591c0910282135t455bef38jfa4039de8b50d067@mail.gmail.com>
Subject: S3 Exception for a Map Reduce job on EC2
From: Harshit Kumar <hkumar.arora@gmail.com>
To: common-user@hadoop.apache.org
Cc: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e646084823d2ea04770b719c
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e646084823d2ea04770b719c
Content-Type: text/plain; charset=UTF-8

Hi

There is 1 GB of rdf/owl files that I am executing on EC2. Execution throws
the following exception
-------------------

08/11/19 16:08:27 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the
same.
org.apache.hadoop.fs.s3.S3Exception:
org.jets3t.service.S3ServiceException: S3 PUT failed for '/' XML Error
Message: <?xml version="1.0"
encoding="UTF-8"?><Error><Code>OperationAborted</Code><Message>*A
conflicting conditional operation is currently in progress against
**this** resource*. Please try
again.</Message><RequestId>324E696A4BCA8731</RequestId><HostId>{REMOVED}</HostId></Error>
	at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.createBucket(Jets3tNativeFileSystemStore.java:74)
	at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.initialize(Jets3tNativeFileSystemStore.java:63)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
	at org.apache.hadoop.fs.s3native.$Proxy2.initialize(Unknown Source)
	at org.apache.hadoop.fs.s3native.NativeS3FileSystem.initialize(NativeS3FileSystem.java:215)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1339)
	at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1351)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:213)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:175)
	at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:158)
	at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:210)
	at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
	at org.apache.hadoop.streaming.StreamJob.submitAndMonitorJob(StreamJob.java:925)
	at org.apache.hadoop.streaming.StreamJob.go(StreamJob.java:115)
	at org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreaming.java:33)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
	at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
	at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
Caused by: org.jets3t.service.S3ServiceException: S3 PUT failed for
'/' XML Error Message: <?xml version="1.0"
encoding="UTF-8"?><Error><Code>OperationAborted</Code><Message>A
conflicting conditional operation is currently in progress against
this resource. Please try
again.</Message><RequestId>{REMOVED}</RequestId><HostId>{REMOVED}</HostId></Error>
	at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:424)
	at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestPut(RestS3Service.java:734)
	at org.jets3t.service.impl.rest.httpclient.RestS3Service.createObjectImpl(RestS3Service.java:1357)
	at org.jets3t.service.impl.rest.httpclient.RestS3Service.createBucketImpl(RestS3Service.java:1234)
	at org.jets3t.service.S3Service.createBucket(S3Service.java:1390)
	at org.jets3t.service.S3Service.createBucket(S3Service.java:1158)
	at org.jets3t.service.S3Service.createBucket(S3Service.java:1177)
	at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.createBucket(Jets3tNativeFileSystemStore.java:69)
	... 29 more

---------------------
There were like 50 failed map tasks out of total of 1141 maps. Though hadoop
job tracker successfully executed failed tasks again, but if a task fails,
and then restarting it again, takes execution time. If somehow, this
exception can be avoided altogether, program execution will be faster.

Came across a JIRA post that says, the exception is because s3 tries to
create a bucket which it should not create, and a patch is also posted for
the same. Applied the patch, but still the same error.

IMO, this error is because a operation is trying to access a part of file
which is already been accessed by another map job.

Please help resolve this issue or pass on any pointers that can suggest what
is the source of error?

Thanks and Regards

H. Kumar
skype: harshit900
Blog: http://harshitkumar.wordpress.com
Website: http:/kumarharmuscat.tripod.comg

--0016e646084823d2ea04770b719c--

From general-return-671-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 29 04:49:44 2009
Return-Path: <general-return-671-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22010 invoked from network); 29 Oct 2009 04:49:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Oct 2009 04:49:43 -0000
Received: (qmail 82030 invoked by uid 500); 29 Oct 2009 04:49:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81872 invoked by uid 500); 29 Oct 2009 04:49:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81862 invoked by uid 99); 29 Oct 2009 04:49:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Oct 2009 04:49:42 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of hadoopmaillist@gmail.com designates 209.85.210.194 as permitted sender)
Received: from [209.85.210.194] (HELO mail-yx0-f194.google.com) (209.85.210.194)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Oct 2009 04:49:32 +0000
Received: by yxe32 with SMTP id 32so1715885yxe.5
        for <general@hadoop.apache.org>; Wed, 28 Oct 2009 21:49:11 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=Wf/eS6slndxEMZdDbYOH9tLQ6zW1WFUnicTeKIt2WRc=;
        b=w5cuyFs2iHsTIjaWsjyWYtS65l6W1uBqycq7ZvVw7k7YYyAsQtpepLy8StStNFE4AE
         AB+u2c72cZNQrmFkoC7dVUSjKDoz1UsXMDlsAK1LFrn8JwhWjv0WqZVKot2ZH6VVzHpl
         1wpAkGQqnhVLAdnoBkV5U76Ptx2ZveEB1hFiM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=dUP0W0a24FnB2T6lNs97JtCefztHNBC33BYXXPuHMFpPlumzSneH7n5wEC9dUSDulA
         Jozg6C4r+2Q34HPptTeffL5w08stpZSF5HPfZs5cWzj5Ryqviq0db2xi+bAZbjnj8uIB
         sitTfukWmJpw+I+xoaocnfkh1AExfSQ6xlyak=
MIME-Version: 1.0
Received: by 10.101.165.36 with SMTP id s36mr756279ano.95.1256791751711; Wed, 
	28 Oct 2009 21:49:11 -0700 (PDT)
In-Reply-To: <7232A3A7C53F634B8E91E65BC7833C5112C5714B34@TR-EXCH07.prog.altair.com>
References: <5c86176b0910280114i79370febrfb4bbdd833c64811@mail.gmail.com>
	 <8211a1320910280126t62f0101fg547acf8fc0f702d7@mail.gmail.com>
	 <5c86176b0910280147s593aac15lc08e08b714135aea@mail.gmail.com>
	 <8211a1320910280211tc4f19fdk88c695939b629b91@mail.gmail.com>
	 <5c86176b0910281753y5dffb328m3de2f48cebd3bca6@mail.gmail.com>
	 <8211a1320910281916v3ea4f5c1vc4b23fa0260c753@mail.gmail.com>
	 <5c86176b0910281921x4d19970aqb5ddd0aca435bb80@mail.gmail.com>
	 <7232A3A7C53F634B8E91E65BC7833C5112C5714B34@TR-EXCH07.prog.altair.com>
Date: Thu, 29 Oct 2009 12:49:11 +0800
Message-ID: <5c86176b0910282149g4e7b800aleae35af5eae19ce@mail.gmail.com>
Subject: Re: Need Help: The problem with text key of MapFile
From: lei wang <hadoopmaillist@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c5bac7eadb5204770ba17f
X-Virus-Checked: Checked by ClamAV on apache.org

--001636c5bac7eadb5204770ba17f
Content-Type: text/plain; charset=ISO-8859-1

I cann't understand why you give me two web sites.

On Thu, Oct 29, 2009 at 10:27 AM, Lori Ann Martin <lmartin@altair.com>wrote:

> heck out www.HiQube.com or www.pbsgridworks.com
>
> -----Original Message-----
> From: lei wang [mailto:hadoopmaillist@gmail.com]
> Sent: Wednesday, October 28, 2009 7:22 PM
> To: general@hadoop.apache.org
> Subject: Re: Need Help: The problem with text key of MapFile
>
> Oh, I  have tried hbase in the early.
> But I think HDFS may give me a choice.
> Thanks.
>
> On Thu, Oct 29, 2009 at 10:16 AM, Jeff Zhang <zjffdu@gmail.com> wrote:
>
> > I guess maybe HBase will be fit for you.   HBase is a distributed
> database
> > built upon Hadoop.
> > You can use the url as the row key and put other fields into columns.
> >
> > then you can retrieve the web page through HBase Client API and insert
> new
> > web page into it. The performance of HBase 0.20 is good enough for you.
> >
> > Best Regards,
> > Jeff zhang
> >
> >
> > On Thu, Oct 29, 2009 at 8:53 AM, lei wang <hadoopmaillist@gmail.com>
> > wrote:
> >
> > > hi,juff, thanks for your comments.
> > >   I did read this book early, I use MapFile to store my web pages for
> > > random access.
> > > First I think the SquenceFile conversion as a solution, howerve, the
> > > problem is that I need append the new pages to the MapFile by minute
> > > or second, so I didn't think SquenceFile conversion can manage this.
> > > Would you give me some suggestion? Think your very much!
> > >
> > > Best wishes.
> > >
> > > On 10/28/09, Jeff Zhang <zjffdu@gmail.com> wrote:
> > > > I do not know why you need use MapFile, could you use SequenceFile
> > > instead ?
> > > >
> > > > The MapFile's advantage is its read performance, because it build
> index
> > > on
> > > > its keys. So its keys must be in order.
> > > >
> > > > If you really want to use MapFile, you can first write your data to
> > > > SequenceFile and then covert it to MapFile.
> > > >
> > > > About  how to convert SequenceFile to MapFile:
> > > > 1. Sort the SequenceFile using sort in examples of hadoop
> > > > 2. create index for the output of the above step. then you get both
> of
> > > the
> > > > data file and index file
> > > >
> > > >
> > > > You an refer Tom Whilte's book "Hadoop definitive guide" for details
> > > about
> > > > how to convert SequenceFile into MapFile
> > > >
> > > > Jeff Zhang
> > > >
> > > >
> > > >
> > > > On Wed, Oct 28, 2009 at 4:47 PM, lei wang <hadoopmaillist@gmail.com>
> > > wrote:
> > > >
> > > >> but now, "url" is not in order,  must the key be intwritable ?
> should
> > it
> > > >>  be
> > > >> comparable ?
> > > >> How to make sure them in order?sort it first?
> > > >> I just want to insert the pages for  random acess by "url ".
> > > >>
> > > >> On Wed, Oct 28, 2009 at 4:26 PM, Jeff Zhang <zjffdu@gmail.com>
> wrote:
> > > >>
> > > >> > Hi Wang,
> > > >> >
> > > >> > The keys of MapFile should be in order, so when you add records
> into
> > > >> > MapFile, you should make sure you insert them in order
> > > >> >
> > > >> > Best Regards,
> > > >> >
> > > >> > Jeff Zhang
> > > >> >
> > > >> >
> > > >> > On Wed, Oct 28, 2009 at 4:14 PM, lei wang <
> hadoopmaillist@gmail.com
> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Hi, friends
> > > >> > > I need store the web pages(a huge one) in the MapFile of the
> > hadoop,
> > > >> > > So
> > > >> i
> > > >> > > did use the url as the key, and its type is "text", When
>  writring
> > > the
> > > >> > > records into the mapfile, it give an error as "out of order",
> > which
> > > >> type
> > > >> > > should I choose to represent  the key "url", can anyone give me
> > some
> > > >> > detail
> > > >> > > answer, thanks for you help.
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>

--001636c5bac7eadb5204770ba17f--

From general-return-1479-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat May 01 16:42:11 2010
Return-Path: <general-return-1479-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 62175 invoked from network); 1 May 2010 16:42:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 May 2010 16:42:11 -0000
Received: (qmail 50840 invoked by uid 500); 1 May 2010 16:42:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50771 invoked by uid 500); 1 May 2010 16:42:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50763 invoked by uid 99); 1 May 2010 16:42:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 01 May 2010 16:42:09 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_FRT_CONTACT,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of lars.george@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 01 May 2010 16:42:03 +0000
Received: by wwb28 with SMTP id 28so853981wwb.35
        for <general@hadoop.apache.org>; Sat, 01 May 2010 09:41:43 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type:content-transfer-encoding;
        bh=PITVrkVbsPTbZROAV0572EVPsYL+YJ85Iid99+fiICI=;
        b=iYYLzyrG6FW99oScAAegxjsnxP0bhms07rHKKulyoKsbRfEHAImNWj0UAG26ePK2O5
         IVZ/hgnMIqO7IwYzbMG951hchtw9WL3RavzWbijXz/iKE/++uN8a6veRJVuGfLnN5Xrv
         FK59nHkwItXph6cKI+/N+1F/0Ts6MExEBwmk8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=wLDuu35nku9fYlZw2tnWgoozPlUxHgY7x5F0MjZ9C80Ae2q1C3CfT23idMeFUyEl2l
         rb2WMr4vdJ0/KAo/SYjT+dA4ernuAFjvoEZs1KyEiH8N9y7VKAsOyyHyzazoSj4MFZDk
         6XAF8f4gd8RY+SnUR/3OJnXpDx/FnkpPpjY58=
MIME-Version: 1.0
Received: by 10.216.93.2 with SMTP id k2mr2038221wef.56.1272732103417; Sat, 01 
	May 2010 09:41:43 -0700 (PDT)
Received: by 10.216.180.72 with HTTP; Sat, 1 May 2010 09:41:43 -0700 (PDT)
Date: Sat, 1 May 2010 18:41:43 +0200
Message-ID: <q2o61770b881005010941reabac4e0x819747fde4b5fac4@mail.gmail.com>
Subject: 3rd Munich OpenHUG Meeting
From: Lars George <lars.george@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

3rd Munich OpenHUG Meeting

I am pleased to invite you to our third Munich Open Hadoop User Group Meeti=
ng!

Like always we are looking forward to see everyone again and are
welcoming new attendees to join our group. We are enthusiast about all
things related to scalable, distributed storage system. We are not
limiting us to a particular system but appreciate anyone who would
like to share about their experiences.

When: Thursday May 6th, 2010 at 6pm (open end)
Where: eCircle AG, Nymphenburger Stra=DFe 86, 80636 M=FCnchen ["Bruckmann"
Building, "U1 Mailinger Str", map
http://www.ecircle.com/de/kontakt/anfahrt.html (in German) and look
for the signs]

Thanks again to Bob Schulze from eCircle for providing the infrastructure.

We have a talk scheduled by Stefan Seelmann who is a member of the
project committee for the Apache Directory project. This is followed
by an open discussion.

Please RSVP here:

Xing: https://www.xing.com/events/3rd-munich-openhug-meeting-506082

Yahoo Upcoming:
http://upcoming.yahoo.com/event/5771044/BY/Mnchen/3rd-Munich-OpenHUG-Meetin=
g/eCircle-AG

Looking forward to seeing you there!

Best regards,
Lars

From general-return-1480-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 03 04:57:47 2010
Return-Path: <general-return-1480-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 65037 invoked from network); 3 May 2010 04:57:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 May 2010 04:57:47 -0000
Received: (qmail 58649 invoked by uid 500); 3 May 2010 04:57:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57257 invoked by uid 500); 3 May 2010 04:57:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57249 invoked by uid 99); 3 May 2010 04:57:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 May 2010 04:57:42 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [128.200.80.34] (HELO smtp1.es.uci.edu) (128.200.80.34)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 May 2010 04:57:33 +0000
Received: from [169.234.248.13] (vcv248013.vpn.uci.edu [169.234.248.13])
	(authenticated bits=0)
	by smtp1.es.uci.edu (8.13.1/8.13.1) with ESMTP id o434vAtf013048
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT)
	for <general@hadoop.apache.org>; Sun, 2 May 2010 21:57:11 -0700
X-UCInetID: vayyalas
Message-ID: <4BDE57AA.70502@uci.edu>
Date: Sun, 02 May 2010 21:57:14 -0700
From: Vandana Ayyalasomayajula <vayyalas@uci.edu>
User-Agent: Thunderbird 2.0.0.24 (Windows/20100228)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Problem with configuring Fair Schduler
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hi All,

I am trying to configure fair scheduler in hadoop. I have followed all 
the instructions given in hadoop common docs ( 
http://hadoop.apache.org/common/docs/current/fair_scheduler.html#Configuring+the+Fair+scheduler 
). But when I try to click on the job queue in the job tracker page , I 
see the following exception:

java.lang.NullPointerException
	at org.apache.hadoop.mapred.FairScheduler.getJobs(FairScheduler.java:765)
	at org.apache.hadoop.mapred.jobqueue_005fdetails_jsp._jspService(jobqueue_005fdetails_jsp.java:64)
	at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
	at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:324)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)

If anybody has come across this problem, please do let me know where I 
am going wrong.

Thanks for the help.
Vandana

From general-return-1481-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 03 05:56:37 2010
Return-Path: <general-return-1481-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 71118 invoked from network); 3 May 2010 05:56:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 May 2010 05:56:37 -0000
Received: (qmail 81937 invoked by uid 500); 3 May 2010 05:56:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81630 invoked by uid 500); 3 May 2010 05:56:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 66410 invoked by uid 99); 3 May 2010 05:17:08 -0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
Sender: kumar@bike.snu.ac.kr
In-Reply-To: <4BDE57AA.70502@uci.edu>
References: <4BDE57AA.70502@uci.edu>
Date: Mon, 3 May 2010 14:16:41 +0900
X-Google-Sender-Auth: f3e95d672fe947db
Message-ID: <r2udfdaa57e1005022216k3ee8b0f3h4c7aa2a29e357466@mail.gmail.com>
Subject: Re: Problem with configuring Fair Schduler
From: Kumar Harshit <hkumar.arora@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636ed6d74b55a340485a9b274

--001636ed6d74b55a340485a9b274
Content-Type: text/plain; charset=ISO-8859-1

Hi

Can I have a look at your project report on Pig and Hive query execution?

I stumbled upon it but couldnt download it.

Thanks
Kumar

On Mon, May 3, 2010 at 1:57 PM, Vandana Ayyalasomayajula
<vayyalas@uci.edu>wrote:

> Hi All,
>
> I am trying to configure fair scheduler in hadoop. I have followed all the
> instructions given in hadoop common docs (
> http://hadoop.apache.org/common/docs/current/fair_scheduler.html#Configuring+the+Fair+scheduler). But when I try to click on the job queue in the job tracker page , I see
> the following exception:
>
> java.lang.NullPointerException
>        at
> org.apache.hadoop.mapred.FairScheduler.getJobs(FairScheduler.java:765)
>        at
> org.apache.hadoop.mapred.jobqueue_005fdetails_jsp._jspService(jobqueue_005fdetails_jsp.java:64)
>        at
> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>        at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
>        at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
>        at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>        at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>        at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>        at
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>        at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>        at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>        at org.mortbay.jetty.Server.handle(Server.java:324)
>        at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>        at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>        at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>        at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
> If anybody has come across this problem, please do let me know where I am
> going wrong.
>
> Thanks for the help.
> Vandana
>

--001636ed6d74b55a340485a9b274--

From general-return-1482-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 04 18:32:25 2010
Return-Path: <general-return-1482-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 95040 invoked from network); 4 May 2010 18:32:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 May 2010 18:32:24 -0000
Received: (qmail 55251 invoked by uid 500); 4 May 2010 18:32:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54500 invoked by uid 500); 4 May 2010 18:32:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54407 invoked by uid 99); 4 May 2010 18:32:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 May 2010 18:32:19 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=AWL,HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 May 2010 18:32:14 +0000
Received: from sp1-ex07cas01.ds.corp.yahoo.com (sp1-ex07cas01.ds.corp.yahoo.com [216.252.116.137])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o44IUsbm069300;
	Tue, 4 May 2010 11:31:00 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:accept-language:content-language:x-ms-has-attach:
	x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=OZzWuLYencBol07dukoBvrZWU08sesiS1inx8L38hUIAJIvGq98MyaCPL9KWMHzs
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas01.ds.corp.yahoo.com ([216.252.116.137]) with mapi; Tue, 4 May
 2010 11:29:51 -0700
From: Dekel Tankel <dekel@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>,
        "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Date: Tue, 4 May 2010 11:29:50 -0700
Subject: RE: Hadoop User Group - May 19th at Yahoo!
Thread-Topic: RE: Hadoop User Group - May 19th at Yahoo!
Thread-Index: Acrrt8hyTLFBKu7ESwWjdinJLK+osw==
Message-ID: <46E2672895E8744A97D7577A6110858B4FEA3857EB@SP1-EX07VS01.ds.corp.yahoo.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_46E2672895E8744A97D7577A6110858B4FEA3857EBSP1EX07VS01ds_"
MIME-Version: 1.0

--_000_46E2672895E8744A97D7577A6110858B4FEA3857EBSP1EX07VS01ds_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi

Agenda is available for the upcoming HUG.

Hope to see you all there.

http://www.meetup.com/hadoop/calendar/13048582/

thanks

            Dekel






Register today for Hadoop Summit 2010 June 29th, Hyatt, Santa Clara, CA
http://hadoopsummit2010.eventbrite.com/

Presentation submission deadline extended until May 10th
http://developer.yahoo.com/events/hadoopsummit2010/presentationguidelines.h=
tml






--_000_46E2672895E8744A97D7577A6110858B4FEA3857EBSP1EX07VS01ds_--

From general-return-1483-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 04 20:31:37 2010
Return-Path: <general-return-1483-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24675 invoked from network); 4 May 2010 20:31:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 May 2010 20:31:37 -0000
Received: (qmail 30074 invoked by uid 500); 4 May 2010 20:31:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29875 invoked by uid 500); 4 May 2010 20:31:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29867 invoked by uid 99); 4 May 2010 20:31:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 May 2010 20:31:35 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 04 May 2010 20:31:33 +0000
Received: (qmail 24571 invoked by uid 99); 4 May 2010 20:31:11 -0000
Received: from localhost.apache.org (HELO mail-gw0-f48.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 May 2010 20:31:11 +0000
Received: by gwj15 with SMTP id 15so274793gwj.35
        for <multiple recipients>; Tue, 04 May 2010 13:31:10 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.166.8 with SMTP id k8mr811110iby.93.1273005063761; Tue, 04 
	May 2010 13:31:03 -0700 (PDT)
Received: by 10.231.149.73 with HTTP; Tue, 4 May 2010 13:31:03 -0700 (PDT)
Date: Tue, 4 May 2010 13:31:03 -0700
Message-ID: <h2q1267dd3b1005041331r7d8f696di370a279ff605832f@mail.gmail.com>
Subject: [DISCUSSION] Adopt the Release Manager role for Hadoop
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org, private@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

After 0.20 was branched in 2008, there have been no releases of Hadoop
from trunk despite active development of the mainline. Tom White is
working to cut a new release, but given its record in the last year,
Hadoop's current release process is in need of justification and/or
revision.

As Owen suggested earlier, Apache HTTP Server project rules have some
compelling features. In particular, the Release Manager seems like a
useful role. If Hadoop adopted these rules, it could enable more
experimentation in alpha, more frequent releases, and it could still
deliver stable versions. That said, moving from our current model
immediately to odd/even releases, concurrent releases of minor
versions, expressing stability/confidence in versioning, etc. would be
a dramatic and likely disruptive shift. Even discussion on adopting
these rules would unnecessarily delay the next release.

For now, I'd like to propose we adopt only the Release Manager (RM)
role described here:

http://httpd.apache.org/dev/release.html

with the following procedures and caveats:

0) The RM for the next release is confirmed by majority approval of
the PMC (at least 3 +1 votes, more +1 than -1 votes).
1) The RM may be replaced by majority approval of the PMC. Similarly,
if one wants to cut a release from trunk before the next, scheduled
release manager does, a single vote may appoint an intermediate RM
(e.g. a vote could add a RM for a new minor release between releases
run by Tom and Owen)
2) Until the projects can be released separately, one must have commit
privileges on Common, HDFS, and MapReduce to be a RM
3) As in the httpd model, the RM decides how to manage the next
release ({date,feature}-based branching, patches included or elided,
etc.). However:
3.i) All bug fixes/features/etc. in the release must also be in trunk
3.ii) Only blocker bugs, documentation, etc. should be fixed in a patch release
3.iii) Exceptions to (3.ii) are passed per majority consensus by the
committers of the affected projects
4) The product of this process will still become an official release
by majority approval in the PMC

(3.*) and (4) should be familiar.

In the short to medium term, all releases would come off of trunk. If
a self-nominated RM for 0.x does not release before 0.y is ready, then
0.y is released as 0.x, the 0.x branch is retired, and the RM for 0.x
needs to be reelected to release 0.z (where x < y < z). Does this make
sense?

If possible, I would like to avoid related, contentious issues on
feature branching and versioning and restrict this discussion to
whether adopting this role would be useful to producing more frequent
Hadoop releases. Tom has volunteered to be the RM for 0.21 and Owen
has volunteered for 0.22; each would be confirmed in a vote apart from
the vote adopting this modification, which will open once this thread
appears to reach consensus. -C

From general-return-1484-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 04 21:35:46 2010
Return-Path: <general-return-1484-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38334 invoked from network); 4 May 2010 21:35:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 May 2010 21:35:45 -0000
Received: (qmail 16036 invoked by uid 500); 4 May 2010 21:35:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15902 invoked by uid 500); 4 May 2010 21:35:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15894 invoked by uid 99); 4 May 2010 21:35:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 May 2010 21:35:44 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of apache@jacobrideout.net designates 74.50.50.180 as permitted sender)
Received: from [74.50.50.180] (HELO jacobrideout.net) (74.50.50.180)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 May 2010 21:35:39 +0000
Received: from mail-wy0-f176.google.com (mail-wy0-f176.google.com [74.125.82.176])
	(using TLSv1 with cipher RC4-MD5 (128/128 bits))
	(No client certificate requested)
	by jacobrideout.net (Postfix) with ESMTPSA id 3F6E1400F8
	for <general@hadoop.apache.org>; Tue,  4 May 2010 15:35:17 -0600 (MDT)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=jacobrideout.net;
	s=2009a; t=1273008918; bh=JoL73HeHsPH/RfTGunBD04jn8EF1lC1Icbf1af4+R
	FQ=; h=MIME-Version:From:Date:Message-ID:Subject:To:Content-Type;
	b=e1MVLoH96nLBscV6953GetsjqHEJFQ7i+Pvoyf7LUd6l69bG3Eb87AtHRFEU2fFR/
	lNnkxyFiEnnq8/bOQp/3Hvq34f19WQ+ssEvvdM3NoAXYm2nnp9bE+rXVMsvdBc4ucrJ
	Y9lV/3Sf4ljoVSb7bAJVfvrreoRAC1QzH37bCvk=
Received: by wyb34 with SMTP id 34so69431wyb.35
        for <general@hadoop.apache.org>; Tue, 04 May 2010 14:35:16 -0700 (PDT)
Received: by 10.216.155.143 with SMTP id j15mr2699344wek.46.1273008916270; 
	Tue, 04 May 2010 14:35:16 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.16.33 with HTTP; Tue, 4 May 2010 14:34:56 -0700 (PDT)
From: Jacob R Rideout <apache@jacobrideout.net>
Date: Tue, 4 May 2010 15:34:56 -0600
Message-ID: <x2id37f1a781005041434w86e2c116s5fac0f8951e69a2e@mail.gmail.com>
Subject: Boulder/Denver Colorado Hadoop Meetup
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8

I am happy to announce we have started a Hadoop users group in Boulder
and Denver Colorado area. We'll be having our next meeting on May 12th
at 7pm. I would like to invite anyone in the area to attend. Cold
Creek Solutions (a hardware VAR) will be sponsoring with
pizza/beer/soda.

Details:
http://www.meetup.com/Boulder-Denver-Hadoop/

Time:
7pm MDT on 5/12/2010

Location:
The offices of Gnip
1610 Pearl Street
Boulder, CO 80302

If anyone would like to give a talk please state your topic on the meetup site.

Thanks,

I hope to see you few of you there!

Jacob Rideout

From general-return-1485-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed May 05 04:57:48 2010
Return-Path: <general-return-1485-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 13603 invoked from network); 5 May 2010 04:57:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 May 2010 04:57:48 -0000
Received: (qmail 22356 invoked by uid 500); 5 May 2010 04:57:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22294 invoked by uid 500); 5 May 2010 04:57:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22286 invoked by uid 99); 5 May 2010 04:57:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 May 2010 04:57:44 +0000
X-ASF-Spam-Status: No, hits=2.7 required=10.0
	tests=AWL,FREEMAIL_FROM,FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com designates 209.85.221.171 as permitted sender)
Received: from [209.85.221.171] (HELO mail-qy0-f171.google.com) (209.85.221.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 May 2010 04:57:39 +0000
Received: by qyk1 with SMTP id 1so138255qyk.5
        for <general@hadoop.apache.org>; Tue, 04 May 2010 21:57:18 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=+1vk/Ev+uhE4RnuLLR5SYOJOFslA2CPf2ip661hEkho=;
        b=YP3iNrcOkAhdQ8YAgPl/YwRFD4IZFxxyctxbC2+zeTiI/wKP0HEJvgrLiGTU+t9+fW
         BUDZ1pdldCqPgmDsE42kE63dRe25S2HWCeXu1h9u1VzClM9qFTnFpBwyOIt46BtUNYl4
         xWmgLOvOb/CvzgNmFHWsPNFlRWttuJu2MylM8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=hVqHq/YPvlzDWh5Fchy2DecPmWhgeRecfpl++IRCTMwId/DvAS41xxunfvXQSViV/G
         3f5KyHY6mXBMXfWxjF40bjYFkoAW+Ad2UPJkAoCcBPa+HLqvHbE5CThiAHddBiCWF1Os
         O2lAUkXMZ7HqPI2w3MUm0Ay9vJl4PSIbne5l4=
MIME-Version: 1.0
Received: by 10.229.96.80 with SMTP id g16mr3856358qcn.85.1273035438053; Tue, 
	04 May 2010 21:57:18 -0700 (PDT)
Received: by 10.229.2.13 with HTTP; Tue, 4 May 2010 21:57:18 -0700 (PDT)
In-Reply-To: <4061df20909100631g5b6b52eeh90a3af5d3b9549ef@mail.gmail.com>
References: <14015.97592.qm@web38406.mail.mud.yahoo.com>
	 <93d501de0909091936r5e234e69k80f849fb6a5ed345@mail.gmail.com>
	 <4061df20909100631g5b6b52eeh90a3af5d3b9549ef@mail.gmail.com>
Date: Tue, 4 May 2010 21:57:18 -0700
Message-ID: <o2m17e273101005042157v782e2765r90aba79fdc5f5f1e@mail.gmail.com>
Subject: Re: multicore node clusters
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364edc141259360485d1a991

--0016364edc141259360485d1a991
Content-Type: text/plain; charset=ISO-8859-1

Hi,
I am looking for config parameter similar to the following which would allow
us to limit the total number of mapper and reducer tasks:

mapred.tasktracker.tasks.maximum

Please advise.

On Thu, Sep 10, 2009 at 6:31 AM, Chandraprakash Bhagtani <
cpbhagtani@gmail.com> wrote:

> Hi,
>
> You should definitely change mapred.tasktracker.map/reduce.tasks.maximum.
> If
> your tasks are more CPU bound then you should run the tasks equal to the
> number of CPU cores otherwise you can run more tasks than cores. You can
> determine CPU and memory usage by running "top" command on datanodes. You
> should also take care of following configuration parameters to achieve best
> performance
>
> *mapred.compress.map.output:* Faster data transfer (from mapper to
> reducers), saves disk space, faster disk writing. Extra time in compression
> and decompression
>
> *io.sort.mb: *If you have idle physical memory after running all tasks you
> can increase this value. But swap space should not be used since it makes
> it
> slow.*
>
> **io.sort.factor: *If your map tasks have large number of spills* *then you
> should increase this value.It also helps in merging at reducers.
>
> *mapred.job.reuse.jvm.num.tasks: *The overhead of JVM creation for each
> task
> is around 1 second. So for the tasks which live for seconds or a few
> minutes
> and have lengthy initialization, this value can be increased to gain
> performance.
>
> *mapred.reduce.parallel.copies: *For Large jobs (the jobs in which map
> output is very large), value of this property can be increased keeping in
> mind that it will increase the total CPU usage.*
>
> **mapred.map/reduce.tasks.speculative.execution: *set to false to gain high
> throughput.
>
> *dfs.block.size* or *mapred.min.split.size* or *mapred.max.split.size* : to
> control the number of maps
>
> On Thu, Sep 10, 2009 at 8:06 AM, Mat Kelcey <matthew.kelcey@gmail.com
> >wrote:
>
> > > I've a cluster where every node is a multicore. From doing internet
> > searches I've figured out that I definitely need to change
> > mapred.tasktracker.tasks.maximum according to the number of clusters. But
> > there are definitely other things that I would like to change for example
> > mapred.map.tasks. Can someone point me out the list of things I should
> > change to get the best performance out of my cluster ?
> >
> > nothing will give you better results than benchmarking with some jobs
> > indicative to your domain!
> >
>
>
>
> --
> Thanks & Regards,
> Chandra Prakash Bhagtani,
>

--0016364edc141259360485d1a991--

From general-return-1486-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed May 05 08:11:53 2010
Return-Path: <general-return-1486-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57260 invoked from network); 5 May 2010 08:11:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 May 2010 08:11:53 -0000
Received: (qmail 12575 invoked by uid 500); 5 May 2010 08:11:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12252 invoked by uid 500); 5 May 2010 08:11:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12244 invoked by uid 99); 5 May 2010 08:11:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 May 2010 08:11:49 +0000
X-ASF-Spam-Status: No, hits=2.4 required=10.0
	tests=AWL,HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [83.166.169.248] (HELO imap.packtpub.com) (83.166.169.248)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 May 2010 08:11:44 +0000
Received: by imap.packtpub.com (Postfix, from userid 1000)
	id 0CC065700035; Wed,  5 May 2010 09:11:22 +0100 (BST)
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on imap.pactpub.com
X-Spam-Level: 
Received: from sonyPC (unknown [122.169.12.134])
	by imap.packtpub.com (Postfix) with ESMTP id 7F30D570002E
	for <general@hadoop.apache.org>; Wed,  5 May 2010 09:11:20 +0100 (BST)
Message-ID: <851BF33D91AB4949A95E30A285433C67@sonyPC>
From: "Kshipra Singh" <kshipras@packtpub.com>
To: <general@hadoop.apache.org>
Subject: Author Hadoop Books- Packt Publishing. 
Date: Wed, 5 May 2010 13:41:18 +0530
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_NextPart_000_02BB_01CAEC58.A3EE2190"
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Windows Mail 6.0.6001.18000
X-MimeOLE: Produced By Microsoft MimeOLE V6.0.6001.18049
X-Old-Spam-Status: No, score=-95.8 required=5.0 tests=AWL,HTML_MESSAGE,RCVD_IN_PBL,
	RCVD_IN_SORBS_DUL,RCVD_IN_XBL,RDNS_NONE,USER_IN_WHITELIST autolearn=no
	version=3.2.5

------=_NextPart_000_02BB_01CAEC58.A3EE2190
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Hi All,=20

I am writing to you for Packt Publishing, the publishers of computer =
related books.=20

We are planning to expand the catalogue of our books on databases and =
are looking forward to publish some books on NoSQL.=20
Currently we are inviting people interested in writing NoSQL books for =
Packt. This doesn't need any past writing experience. All that we need =
is an expert subject knowledge, a passion to share it with others and an =
ability to communicate clearly in English.

So, if you love Hadoop and fancy writing a book, send your book ideas to =
us at author@packtpub.com. Even if you don't have a book idea and are =
simply interested in authoring a book, we are still keen to hear from =
you.=20

More details about the opportunity are available at: =
http://authors.packtpub.com/content/inviting-open-source-nosql-databases-=
fanatics-write-packt

Thanks
Kshipra Singh
Author Relationship Manager
Packt Publishing
www.PacktPub.com
=20
Skype: kshiprasingh15
Twitter: http://twitter.com/packtauthors
=20
Interested in becoming an author? Visit http://authors.packtpub.com for =
all the information you need about writing for Packt.

------=_NextPart_000_02BB_01CAEC58.A3EE2190--



From general-return-1487-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed May 05 15:30:15 2010
Return-Path: <general-return-1487-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60218 invoked from network); 5 May 2010 15:30:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 May 2010 15:30:15 -0000
Received: (qmail 69430 invoked by uid 500); 5 May 2010 15:30:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69285 invoked by uid 500); 5 May 2010 15:30:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69082 invoked by uid 99); 5 May 2010 15:30:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 May 2010 15:30:12 +0000
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of renatoj.marroquin@gmail.com designates 209.85.210.184 as permitted sender)
Received: from [209.85.210.184] (HELO mail-yx0-f184.google.com) (209.85.210.184)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 May 2010 15:30:06 +0000
Received: by yxe14 with SMTP id 14so1684221yxe.5
        for <general@hadoop.apache.org>; Wed, 05 May 2010 08:29:45 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=uj3Bov/3PleLzkX7mTgaZTLaCQrIwiuf5EgvuHhRla4=;
        b=H0zJhjAe/3ISvzPsV5ZAwR6vYHQNNqE0CyLKvSPLku+n25VoiP0/YqWGRS09/K5CQQ
         98euopMbclvd/iAA5QslCSvGPdCo1FGrSMTSQq0O88xkFlSutDBpy1oo8nTO/vNLQHc3
         u9M7/0opq6OOswUcJxQVqy3zIS6eWp23/OfsU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=Aj0AxLsYWGJYwJrH+RaMZBEz2Pusucq0yPEQtgf5bVxBjr7c/4XW59yh7Vy67wN13B
         k0r3Nag2n6A+SRYcUM6n/Lt74hlB9Z4HVto9KZ0HTCY10vw9nguEUHaD22Na/hXsJ2dO
         Xdwf9uRwHJksX6r8rAGpo7Ebm8thPesx77BJQ=
MIME-Version: 1.0
Received: by 10.231.147.199 with SMTP id m7mr1636594ibv.87.1273073383563; Wed, 
	05 May 2010 08:29:43 -0700 (PDT)
Received: by 10.231.32.137 with HTTP; Wed, 5 May 2010 08:29:43 -0700 (PDT)
Date: Wed, 5 May 2010 12:29:43 -0300
Message-ID: <y2p44be54d01005050829gf270a41du98eaf93b0fe2f070@mail.gmail.com>
Subject: Hadoop Data Sharing
From: =?ISO-8859-1?Q?Renato_Marroqu=EDn_Mogrovejo?= <renatoj.marroquin@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e647ecc4cdc9a00485da7e71

--0016e647ecc4cdc9a00485da7e71
Content-Type: text/plain; charset=ISO-8859-1

Hi everyone, I have recently started to play around with hadoop, but I am
getting some into some "design" problems.
I need to make a loop to execute the same job several times, and in each
iteration get the processed values (not using a file because I would need to
read it). I was using an static vector in my main class (the one that
iterates and executes the job in each iteration) to retrieve those values,
and it did work while I was using a standalone mode. Now I tried to test it
on a pseudo-distributed manner and obviously is not working.
Any suggestions, please???

Thanks in advance,


Renato M.

--0016e647ecc4cdc9a00485da7e71--

From general-return-1488-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed May 05 16:46:04 2010
Return-Path: <general-return-1488-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 80343 invoked from network); 5 May 2010 16:46:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 May 2010 16:46:03 -0000
Received: (qmail 16186 invoked by uid 500); 5 May 2010 16:46:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15992 invoked by uid 500); 5 May 2010 16:46:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15978 invoked by uid 99); 5 May 2010 16:46:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 May 2010 16:46:01 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.218.211] (HELO mail-bw0-f211.google.com) (209.85.218.211)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 May 2010 16:45:53 +0000
Received: by bwz3 with SMTP id 3so2862587bwz.11
        for <multiple recipients>; Wed, 05 May 2010 09:45:31 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.204.6.203 with SMTP id a11mr4649539bka.8.1273077931312; Wed, 
	05 May 2010 09:45:31 -0700 (PDT)
Received: by 10.204.61.84 with HTTP; Wed, 5 May 2010 09:45:31 -0700 (PDT)
In-Reply-To: <h2q1267dd3b1005041331r7d8f696di370a279ff605832f@mail.gmail.com>
References: <h2q1267dd3b1005041331r7d8f696di370a279ff605832f@mail.gmail.com>
Date: Wed, 5 May 2010 09:45:31 -0700
Message-ID: <g2iac79ea401005050945x400320cdo99696a198d8bc477@mail.gmail.com>
Subject: Re: [DISCUSSION] Adopt the Release Manager role for Hadoop
From: Tom White <tom@cloudera.com>
To: general@hadoop.apache.org
Cc: private@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Chris,

Overall this looks good to me. Thanks for drafting it.

Should we create a document for the website that we vote on? I think
you've captured most of what we need now in this email, but it would
be a good start for the bylaws when we get back to them.

I've included a couple more comments inline below.

Cheers,
Tom

On Tue, May 4, 2010 at 1:31 PM, Chris Douglas <cdouglas@apache.org> wrote:
> After 0.20 was branched in 2008, there have been no releases of Hadoop
> from trunk despite active development of the mainline. Tom White is
> working to cut a new release, but given its record in the last year,
> Hadoop's current release process is in need of justification and/or
> revision.
>
> As Owen suggested earlier, Apache HTTP Server project rules have some
> compelling features. In particular, the Release Manager seems like a
> useful role. If Hadoop adopted these rules, it could enable more
> experimentation in alpha, more frequent releases, and it could still
> deliver stable versions. That said, moving from our current model
> immediately to odd/even releases, concurrent releases of minor
> versions, expressing stability/confidence in versioning, etc. would be
> a dramatic and likely disruptive shift. Even discussion on adopting
> these rules would unnecessarily delay the next release.
>
> For now, I'd like to propose we adopt only the Release Manager (RM)
> role described here:
>
> http://httpd.apache.org/dev/release.html
>
> with the following procedures and caveats:
>
> 0) The RM for the next release is confirmed by majority approval of
> the PMC (at least 3 +1 votes, more +1 than -1 votes).
> 1) The RM may be replaced by majority approval of the PMC. Similarly,
> if one wants to cut a release from trunk before the next, scheduled
> release manager does, a single vote may appoint an intermediate RM
> (e.g. a vote could add a RM for a new minor release between releases
> run by Tom and Owen)
> 2) Until the projects can be released separately, one must have commit
> privileges on Common, HDFS, and MapReduce to be a RM
> 3) As in the httpd model, the RM decides how to manage the next
> release ({date,feature}-based branching, patches included or elided,
> etc.). However:
> 3.i) All bug fixes/features/etc. in the release must also be in trunk
> 3.ii) Only blocker bugs, documentation, etc. should be fixed in a patch release
> 3.iii) Exceptions to (3.ii) are passed per majority consensus by the
> committers of the affected projects
> 4) The product of this process will still become an official release
> by majority approval in the PMC
>
> (3.*) and (4) should be familiar.
>
> In the short to medium term, all releases would come off of trunk.

Wouldn't this prohibit a point 0.21 release? Do we want to do this?

> If
> a self-nominated RM for 0.x does not release before 0.y is ready, then
> 0.y is released as 0.x, the 0.x branch is retired, and the RM for 0.x
> needs to be reelected to release 0.z (where x < y < z). Does this make
> sense?

I agree with the sentiment, but in practice it might be confusing to
release 0.y as 0.x - I worry about getting JIRA issues marked as being
fixed in 0.x not 0.y, and in general updating all the references to
0.y (including branches). How about just skipping the 0.x version?

>
> If possible, I would like to avoid related, contentious issues on
> feature branching and versioning and restrict this discussion to
> whether adopting this role would be useful to producing more frequent
> Hadoop releases. Tom has volunteered to be the RM for 0.21 and Owen
> has volunteered for 0.22; each would be confirmed in a vote apart from
> the vote adopting this modification, which will open once this thread
> appears to reach consensus. -C
>

From general-return-1489-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed May 05 19:01:32 2010
Return-Path: <general-return-1489-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17104 invoked from network); 5 May 2010 19:01:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 May 2010 19:01:32 -0000
Received: (qmail 1386 invoked by uid 500); 5 May 2010 19:01:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1242 invoked by uid 500); 5 May 2010 19:01:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1233 invoked by uid 99); 5 May 2010 19:01:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 May 2010 19:01:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 05 May 2010 19:01:28 +0000
Received: (qmail 16967 invoked by uid 99); 5 May 2010 19:01:06 -0000
Received: from localhost.apache.org (HELO mail-gw0-f48.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 May 2010 19:01:06 +0000
Received: by gwj15 with SMTP id 15so992471gwj.35
        for <multiple recipients>; Wed, 05 May 2010 12:01:05 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.190.204 with SMTP id dj12mr1215882ibb.9.1273086065075; 
	Wed, 05 May 2010 12:01:05 -0700 (PDT)
Received: by 10.231.149.73 with HTTP; Wed, 5 May 2010 12:01:04 -0700 (PDT)
In-Reply-To: <g2iac79ea401005050945x400320cdo99696a198d8bc477@mail.gmail.com>
References: <h2q1267dd3b1005041331r7d8f696di370a279ff605832f@mail.gmail.com>
	 <g2iac79ea401005050945x400320cdo99696a198d8bc477@mail.gmail.com>
Date: Wed, 5 May 2010 12:01:04 -0700
Message-ID: <t2y1267dd3b1005051201h7116e4caud75673ac9d5128d6@mail.gmail.com>
Subject: Re: [DISCUSSION] Adopt the Release Manager role for Hadoop
From: Chris Douglas <cdouglas@apache.org>
To: private@hadoop.apache.org
Cc: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

> Should we create a document for the website that we vote on? I think
> you've captured most of what we need now in this email, but it would
> be a good start for the bylaws when we get back to them.

I was following the sequence Doug suggested in the last thread on the
release process. I can draft a vote if no discussion of this is
necessary.

>> In the short to medium term, all releases would come off of trunk.
>
> Wouldn't this prohibit a point 0.21 release? Do we want to do this?

Sorry, I meant minor releases. I assumed patch releases per (3.ii) and
(3.iii). The rules seem to allow an RM to cut releases from branches
as well as trunk, so I was trying to avoid the confusing prospect of
releasing, for example, 0.22 from the 0.21 branch instead of from
trunk (unless the odd/even release convention were adopted).

>> If
>> a self-nominated RM for 0.x does not release before 0.y is ready, then
>> 0.y is released as 0.x, the 0.x branch is retired, and the RM for 0.x
>> needs to be reelected to release 0.z (where x < y < z). Does this make
>> sense?
>
> I agree with the sentiment, but in practice it might be confusing to
> release 0.y as 0.x - I worry about getting JIRA issues marked as being
> fixed in 0.x not 0.y, and in general updating all the references to
> 0.y (including branches). How about just skipping the 0.x version?

I meant to describe what's happening with 0.21, more or less. Given
requirement (3.i), nothing should be in (unreleased) 0.x that isn't
also in trunk. If 0.x has a complex/buggy feature that's preventing it
from being released, allowing someone to RM a release without it and
call that 0.x, pushing the feature to 0.y is messy, but not as
confusing (given Hadoop's deprecation and interface stability
guarantees). It's also a merciful way to kill a stuck release without
indicting the struggling RM. -C

From general-return-1490-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed May 05 21:26:34 2010
Return-Path: <general-return-1490-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47118 invoked from network); 5 May 2010 21:26:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 May 2010 21:26:34 -0000
Received: (qmail 95462 invoked by uid 500); 5 May 2010 21:26:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95399 invoked by uid 500); 5 May 2010 21:26:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95253 invoked by uid 99); 5 May 2010 21:26:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 May 2010 21:26:30 +0000
X-ASF-Spam-Status: No, hits=-0.4 required=10.0
	tests=AWL,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 209.85.221.187 as permitted sender)
Received: from [209.85.221.187] (HELO mail-qy0-f187.google.com) (209.85.221.187)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 May 2010 21:26:23 +0000
Received: by qyk17 with SMTP id 17so456368qyk.12
        for <multiple recipients>; Wed, 05 May 2010 14:26:02 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        bh=MyyAiPlYbST+hAX16jE3xLMlcCv6xED8JA3CGxfdY2I=;
        b=qV8B65IXlEpLKKAce1kACylulKSXv85bAm3JTdhg2/YItHIShE8jOoU0l6EKw0DsAW
         0LJH+DBlEQFBdLTU5cgJRDHH4Ye+9M7XjHTe4ilPlsYuI81zkMAge5U3enLH/tw370r5
         hhQY+2i67UAdI4Aw7JoG5pk7/HjCtW+ZtcmpM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        b=Tg7d2K52HaU5CIf0P3MKWQcJlPJRDuLDVExhuE7Ukrc9NlrxzSYRLasNSTdbiKmpYG
         ffdLudXFpoCcmSAK2dIlP81kogTjsnGS3xyPsJTITldMlanrc9xwUrtCMsWsfBs1VxeK
         6HUWnV1ZB9oKONGLWsRm3jvggJgBp3ihLd2Rs=
MIME-Version: 1.0
Received: by 10.229.221.84 with SMTP id ib20mr4700371qcb.93.1273094762478; 
	Wed, 05 May 2010 14:26:02 -0700 (PDT)
Sender: saint.ack@gmail.com
Received: by 10.229.220.66 with HTTP; Wed, 5 May 2010 14:26:02 -0700 (PDT)
Date: Wed, 5 May 2010 14:26:02 -0700
X-Google-Sender-Auth: 795518b9665c0cf1
Message-ID: <q2g7c962aed1005051426j430eac66h561a1d81414b9767@mail.gmail.com>
Subject: [ANN] HBase-0.20.4 released
From: Stack <stack@duboce.net>
To: HBase Dev List <hbase-dev@hadoop.apache.org>, hbase-user@hadoop.apache.org, 
	general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

HBase 0.20.0 is available for download:

   http://hadoop.apache.org/hbase/releases.html

The Release Notes are available here [1].  This relesae includes
critical fixes, some improvements  and performance improvements.  We
recommend that all upgrade to this latest release.

Thanks to all who contributed to this release

Yours,
The HBase Team

From general-return-1491-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu May 06 01:04:56 2010
Return-Path: <general-return-1491-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96023 invoked from network); 6 May 2010 01:04:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 May 2010 01:04:56 -0000
Received: (qmail 44520 invoked by uid 500); 6 May 2010 01:04:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44477 invoked by uid 500); 6 May 2010 01:04:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44469 invoked by uid 99); 6 May 2010 01:04:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 May 2010 01:04:55 +0000
X-ASF-Spam-Status: No, hits=0.6 required=10.0
	tests=AWL,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.176] (HELO mail-px0-f176.google.com) (209.85.212.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 May 2010 01:04:48 +0000
Received: by pxi10 with SMTP id 10so2958287pxi.35
        for <general@hadoop.apache.org>; Wed, 05 May 2010 18:04:28 -0700 (PDT)
Received: by 10.141.90.17 with SMTP id s17mr6407317rvl.207.1273107868132; Wed, 
	05 May 2010 18:04:28 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.140.169.13 with HTTP; Wed, 5 May 2010 18:04:08 -0700 (PDT)
In-Reply-To: <y2p44be54d01005050829gf270a41du98eaf93b0fe2f070@mail.gmail.com>
References: <y2p44be54d01005050829gf270a41du98eaf93b0fe2f070@mail.gmail.com>
From: Aaron Kimball <aaron@cloudera.com>
Date: Wed, 5 May 2010 18:04:08 -0700
Message-ID: <j2hd6d7c4411005051804p40b2b598ud593f984c09d779@mail.gmail.com>
Subject: Re: Hadoop Data Sharing
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd118263da5870485e28698

--000e0cd118263da5870485e28698
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Renato,

In general if you need to perform a multi-pass MapReduce workflow, each pas=
s
materializes its output to files. The subsequent pass then reads those same
files back in as input. This allows the workflow to start at the last
"checkpoint" if it gets interrupted. There is no persistent in-memory
distributed storage feature in Hadoop that would allow a MapReduce job to
post results to memory for consumption by a subsequent job.

So you would just read your initial data from /input, and write your interi=
m
results to /iteration0. Then the next pass reads from /iteration0 and write=
s
to /iteration1, etc..

If your data is reasonably small and you think it could fit in memory
somewhere, then you could experiment with using other distributed key-value
stores (memcached[b], hbase, cassandra, etc..) to hold intermediate results=
.
But this will require some integration work on your part.
- Aaron

On Wed, May 5, 2010 at 8:29 AM, Renato Marroqu=EDn Mogrovejo <
renatoj.marroquin@gmail.com> wrote:

> Hi everyone, I have recently started to play around with hadoop, but I am
> getting some into some "design" problems.
> I need to make a loop to execute the same job several times, and in each
> iteration get the processed values (not using a file because I would need
> to
> read it). I was using an static vector in my main class (the one that
> iterates and executes the job in each iteration) to retrieve those values=
,
> and it did work while I was using a standalone mode. Now I tried to test =
it
> on a pseudo-distributed manner and obviously is not working.
> Any suggestions, please???
>
> Thanks in advance,
>
>
> Renato M.
>

--000e0cd118263da5870485e28698--

From general-return-1492-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri May 07 09:53:28 2010
Return-Path: <general-return-1492-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1727 invoked from network); 7 May 2010 09:53:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 May 2010 09:53:27 -0000
Received: (qmail 19671 invoked by uid 500); 7 May 2010 09:53:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19518 invoked by uid 500); 7 May 2010 09:53:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19506 invoked by uid 99); 7 May 2010 09:53:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 May 2010 09:53:24 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of pierreact@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 May 2010 09:53:18 +0000
Received: by wye20 with SMTP id 20so676765wye.35
        for <general@hadoop.apache.org>; Fri, 07 May 2010 02:52:54 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=ExRSK3aHq/L3U3S2dwH9rbJGkmCoXb3gkk0ZcFjCFaE=;
        b=DaBAm2qEnSo7Hw/G72gkBjF4+riFwgSWSFTU1uBp4ORmUhW72r7gInmwczP6kd4N5S
         f4T3KrT94910ChHUjsYEk2wGvPPXj2QqX4fzHT07Ivf688poyLRXqj+xEmmb6eRi83gI
         YtTiKJDRyXa2AWxyetQu9ZLItJt21do9wcnBc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=esLz69hkghfoJuHjLdwP4BB3P2ytgWpr7os4YdrGFLHEFU66Jett3TfuYy3E0hdSe6
         3S5dSf6Yfljg1T5l/1uk7W3jeyjwmBcAGPfvjoBLv7tulNwATDUdC0ZAYK1X7IQ3nFR1
         ae6Ze7/8kBN/Txa4jqAdWgXx44aNAlaZnSdZg=
MIME-Version: 1.0
Received: by 10.227.137.202 with SMTP id x10mr5543102wbt.26.1273225974771; 
	Fri, 07 May 2010 02:52:54 -0700 (PDT)
Received: by 10.216.154.75 with HTTP; Fri, 7 May 2010 02:52:54 -0700 (PDT)
Date: Fri, 7 May 2010 11:52:54 +0200
Message-ID: <v2x8abfdb8a1005070252sccae6f4an2c15487960279999@mail.gmail.com>
Subject: Could not obtain block blk_
From: Pierre ANCELOT <pierreact@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016367b66c2f1f4000485fe057f

--0016367b66c2f1f4000485fe057f
Content-Type: text/plain; charset=ISO-8859-1

Hello, we're having some issues with this...
Any idea please?

2010-05-07 05:59:46,998 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Cannot initialize JVM Metrics with processName=MAP, sessionId= - already
initialized
2010-05-07 05:59:54,846 INFO org.apache.hadoop.hdfs.DFSClient: Could not
obtain block blk_6180178384047700588_1429320 from any node:
java.io.IOException: No live nodes contain current block
2010-05-07 06:00:05,493 INFO org.apache.hadoop.hdfs.DFSClient: Could not
obtain block blk_6180178384047700588_1429320 from any node:
java.io.IOException: No live nodes contain current block
2010-05-07 06:00:10,358 INFO org.apache.hadoop.hdfs.DFSClient: Could not
obtain block blk_6180178384047700588_1429320 from any node:
java.io.IOException: No live nodes contain current block
2010-05-07 06:00:14,945 WARN org.apache.hadoop.hdfs.DFSClient: DFS Read:
java.io.IOException: Could not obtain block: blk_6180178384047700588_1429320
file=/zzz/Dispatch/input/zzz_24ED346D41543586AE7342997C07F5C8_2010_5_6_mr_24ED346D41543586AE7342997C07F5C8_201056.txt_128
    at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1812)
    at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1638)
    at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1767)
    at java.io.DataInputStream.read(DataInputStream.java:83)
    at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
    at
org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:97)
    at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:423)
    at
org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)

--0016367b66c2f1f4000485fe057f--

From general-return-1493-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri May 07 10:07:38 2010
Return-Path: <general-return-1493-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4881 invoked from network); 7 May 2010 10:07:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 May 2010 10:07:38 -0000
Received: (qmail 37494 invoked by uid 500); 7 May 2010 10:07:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37259 invoked by uid 500); 7 May 2010 10:07:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37249 invoked by uid 99); 7 May 2010 10:07:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 May 2010 10:07:35 +0000
X-ASF-Spam-Status: No, hits=-2.1 required=10.0
	tests=AWL,RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 May 2010 10:07:27 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 9D442B7DAC
	for <general@hadoop.apache.org>; Fri,  7 May 2010 11:07:05 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id aW98sMAbUXgV for <general@hadoop.apache.org>;
	Fri,  7 May 2010 11:06:54 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id B618EB7D77
	for <general@hadoop.apache.org>; Fri,  7 May 2010 11:06:50 +0100 (BST)
MailScanner-NULL-Check: 1273831599.04954@Bew4QoieBv1OTF9xC1Yjog
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o47A6YAP018068
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Fri, 7 May 2010 11:06:35 +0100 (BST)
Message-ID: <4BE3E6DD.5050205@apache.org>
Date: Fri, 07 May 2010 11:09:33 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100228)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Could not obtain block blk_
References: <v2x8abfdb8a1005070252sccae6f4an2c15487960279999@mail.gmail.com>
In-Reply-To: <v2x8abfdb8a1005070252sccae6f4an2c15487960279999@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o47A6YAP018068
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

Pierre ANCELOT wrote:
> Hello, we're having some issues with this...
> Any idea please?
> 
> 2010-05-07 05:59:46,998 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Cannot initialize JVM Metrics with processName=MAP, sessionId= - already
> initialized
> 2010-05-07 05:59:54,846 INFO org.apache.hadoop.hdfs.DFSClient: Could not
> obtain block blk_6180178384047700588_1429320 from any node:
> java.io.IOException: No live nodes contain current block
> 2010-05-07 06:00:05,493 INFO org.apache.hadoop.hdfs.DFSClient: Could not
> obtain block blk_6180178384047700588_1429320 from any node:
> java.io.IOException: No live nodes contain current block
> 2010-05-07 06:00:10,358 INFO org.apache.hadoop.hdfs.DFSClient: Could not
> obtain block blk_6180178384047700588_1429320 from any node:
> java.io.IOException: No live nodes contain current block
> 2010-05-07 06:00:14,945 WARN org.apache.hadoop.hdfs.DFSClient: DFS Read:

As it says, "No live nodes contain current block".
This means: no live datanodes have a copy of the block of that file. 
Either some nodes have gone offline, a disk failed or it wasn't replicated.


> java.io.IOException: Could not obtain block: blk_6180178384047700588_1429320
> file=/zzz/Dispatch/input/zzz_24ED346D41543586AE7342997C07F5C8_2010_5_6_mr_24ED346D41543586AE7342997C07F5C8_201056.txt_128
>     at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1812)
>     at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1638)
>     at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1767)
>     at java.io.DataInputStream.read(DataInputStream.java:83)
>     at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
>     at
> org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:97)
>     at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:423)
>     at
> org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
>     at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
>     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
>     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>     at org.apache.hadoop.mapred.Child.main(Child.java:170)
> 


From general-return-1494-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri May 07 10:09:43 2010
Return-Path: <general-return-1494-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5166 invoked from network); 7 May 2010 10:09:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 May 2010 10:09:43 -0000
Received: (qmail 41457 invoked by uid 500); 7 May 2010 10:09:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 41386 invoked by uid 500); 7 May 2010 10:09:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 41378 invoked by uid 99); 7 May 2010 10:09:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 May 2010 10:09:40 +0000
X-ASF-Spam-Status: No, hits=1.7 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of pierreact@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 May 2010 10:09:35 +0000
Received: by wwb28 with SMTP id 28so203288wwb.35
        for <general@hadoop.apache.org>; Fri, 07 May 2010 03:09:13 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=qVYTOmIiM528xDugRknJhjEC3PfCi2a3YNRBXQOG3Xw=;
        b=aiFFQcM0WEw6mNq3lPPJaBu6aWqIMZf4WvPSJ9XJ0umeGpFwA3lrYctmhiwFXHoNDI
         xQ4G5DyD/cPunzxVqktmHATHQwODEk5cEo5TzDxeQpkGTdDc7QSEQtvKuJ6ejhkF0mpr
         OS0WfVE3hzonZGW008Wk7LK7aP/I+0SLRG2Y0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=L8rBwwbsYLhascw7lddF6BiLij+wOWZBCa7cEymMWVLwnvVqFacTB1tw1GJ5LxPnc/
         OGottjwPdk4N5yB3QDWbTf/OU+X0mXzAClR4SRd9blQ5p1/7SInDuyeOMR+t9DpOY6nb
         sMCRHWNiX8hlTG3OQRLCrnoTaEe2x3/2JpeK8=
MIME-Version: 1.0
Received: by 10.216.93.21 with SMTP id k21mr467135wef.68.1273226953441; Fri, 
	07 May 2010 03:09:13 -0700 (PDT)
Received: by 10.216.154.75 with HTTP; Fri, 7 May 2010 03:09:13 -0700 (PDT)
In-Reply-To: <4BE3E6DD.5050205@apache.org>
References: <v2x8abfdb8a1005070252sccae6f4an2c15487960279999@mail.gmail.com>
	 <4BE3E6DD.5050205@apache.org>
Date: Fri, 7 May 2010 12:09:13 +0200
Message-ID: <k2p8abfdb8a1005070309p38d26f7dqae30854020ec8a7e@mail.gmail.com>
Subject: Re: Could not obtain block blk_
From: Pierre ANCELOT <pierreact@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d7e9a347414e0485fe408f

--0016e6d7e9a347414e0485fe408f
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

All my nodes are up, I can get the file through the DFS client...
0.20.2

On Fri, May 7, 2010 at 12:09 PM, Steve Loughran <stevel@apache.org> wrote:

> Pierre ANCELOT wrote:
>
>> Hello, we're having some issues with this...
>> Any idea please?
>>
>> 2010-05-07 05:59:46,998 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
>> Cannot initialize JVM Metrics with processName=3DMAP, sessionId=3D - alr=
eady
>> initialized
>> 2010-05-07 05:59:54,846 INFO org.apache.hadoop.hdfs.DFSClient: Could not
>> obtain block blk_6180178384047700588_1429320 from any node:
>> java.io.IOException: No live nodes contain current block
>> 2010-05-07 06:00:05,493 INFO org.apache.hadoop.hdfs.DFSClient: Could not
>> obtain block blk_6180178384047700588_1429320 from any node:
>> java.io.IOException: No live nodes contain current block
>> 2010-05-07 06:00:10,358 INFO org.apache.hadoop.hdfs.DFSClient: Could not
>> obtain block blk_6180178384047700588_1429320 from any node:
>> java.io.IOException: No live nodes contain current block
>> 2010-05-07 06:00:14,945 WARN org.apache.hadoop.hdfs.DFSClient: DFS Read:
>>
>
> As it says, "No live nodes contain current block".
> This means: no live datanodes have a copy of the block of that file. Eith=
er
> some nodes have gone offline, a disk failed or it wasn't replicated.
>
>
>
>  java.io.IOException: Could not obtain block:
>> blk_6180178384047700588_1429320
>>
>> file=3D/zzz/Dispatch/input/zzz_24ED346D41543586AE7342997C07F5C8_2010_5_6=
_mr_24ED346D41543586AE7342997C07F5C8_201056.txt_128
>>    at
>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient=
.java:1812)
>>    at
>>
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.ja=
va:1638)
>>    at
>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1767=
)
>>    at java.io.DataInputStream.read(DataInputStream.java:83)
>>    at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
>>    at
>>
>> org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(Line=
RecordReader.java:97)
>>    at
>>
>> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(Ma=
pTask.java:423)
>>    at
>> org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
>>    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
>>    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
>>    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>>    at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>
>>
>


--=20
http://www.neko-consulting.com
Ego sum quis ego servo
"Je suis ce que je prot=E8ge"
"I am what I protect"

--0016e6d7e9a347414e0485fe408f--

From general-return-1495-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri May 07 10:30:55 2010
Return-Path: <general-return-1495-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 7191 invoked from network); 7 May 2010 10:30:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 May 2010 10:30:54 -0000
Received: (qmail 58389 invoked by uid 500); 7 May 2010 10:30:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58217 invoked by uid 500); 7 May 2010 10:30:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58208 invoked by uid 99); 7 May 2010 10:30:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 May 2010 10:30:50 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 May 2010 10:30:40 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id C2AEEB7E5E
	for <general@hadoop.apache.org>; Fri,  7 May 2010 11:30:19 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id ephNC7dR1yps for <general@hadoop.apache.org>;
	Fri,  7 May 2010 11:30:08 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id B2055B7E2F
	for <general@hadoop.apache.org>; Fri,  7 May 2010 11:30:08 +0100 (BST)
MailScanner-NULL-Check: 1273832996.95615@2qfPaLs/q5CoN6fhr+kL6g
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o47ATq9M018589
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Fri, 7 May 2010 11:29:53 +0100 (BST)
Message-ID: <4BE3EC53.7030300@apache.org>
Date: Fri, 07 May 2010 11:32:51 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100228)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Could not obtain block blk_
References: <v2x8abfdb8a1005070252sccae6f4an2c15487960279999@mail.gmail.com>	 <4BE3E6DD.5050205@apache.org> <k2p8abfdb8a1005070309p38d26f7dqae30854020ec8a7e@mail.gmail.com>
In-Reply-To: <k2p8abfdb8a1005070309p38d26f7dqae30854020ec8a7e@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o47ATq9M018589
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

Pierre ANCELOT wrote:
> All my nodes are up, I can get the file through the DFS client...
> 0.20.2

OK, that's interesting. Try looking at the replication count and upping 
it, or copying the file to make sure its all there.

From general-return-1496-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri May 07 15:15:43 2010
Return-Path: <general-return-1496-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67390 invoked from network); 7 May 2010 15:15:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 May 2010 15:15:43 -0000
Received: (qmail 76847 invoked by uid 500); 7 May 2010 15:15:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76684 invoked by uid 500); 7 May 2010 15:15:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76676 invoked by uid 99); 7 May 2010 15:15:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 May 2010 15:15:41 +0000
X-ASF-Spam-Status: No, hits=-2.7 required=10.0
	tests=AWL,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 May 2010 15:15:36 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:Received:Received:From:To:Subject:
   Thread-Topic:Thread-Index:Date:Message-ID:References:
   In-Reply-To:Accept-Language:Content-Language:
   X-MS-Has-Attach:X-MS-TNEF-Correlator:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version:
   Return-Path:X-OriginalArrivalTime;
  b=MNvbvA2w4aqdpxgqz7JyccePeZYycQCAeb1tFCKgYumjp/zDXUbsk3Xj
   XP/6T6E52fP9y1Kn+LrOiAXfu4WBBzqY7XD4uNFOtkV7bfSaCrJsxB3dU
   7v3mqQMMaW8YoNl;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1273245336; x=1304781336;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20Could=20not=20obtain=20block=20blk_
   |Date:=20Fri,=207=20May=202010=2015:15:14=20+0000
   |Message-ID:=20<9321E246-3120-421A-B0C6-7575EBF245F6@link
   edin.com>|To:=20"<general@hadoop.apache.org>"=20<general@
   hadoop.apache.org>|MIME-Version:=201.0
   |Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<4ff0f662-31f5-4ba5-a4aa-212dc7e8cb43>
   |In-Reply-To:=20<4BE3EC53.7030300@apache.org>|References:
   =20<v2x8abfdb8a1005070252sccae6f4an2c15487960279999@mail.
   gmail.com>=0D=0A=09=20<4BE3E6DD.5050205@apache.org>=0D=0A
   =20<k2p8abfdb8a1005070309p38d26f7dqae30854020ec8a7e@mail.
   gmail.com>=0D=0A=20<4BE3EC53.7030300@apache.org>;
  bh=HdLlnmGfb/g2xGJOnO4nHr7xS5CkdbyG6IsnvaeoIo4=;
  b=tUwhrbKUzm+n7fBi2GPUzareoJm6qulmNh31ft5JoqlxrwOhQ7nj+41W
   NynN5AY9TNn/OL7fXAfcrMj+PznS0QP5nCjHhDBqD+/sko9T2b7wkq83S
   15tDJZHz3PzO/um;
X-IronPort-AV: E=Sophos;i="4.52,349,1270450800"; 
   d="scan'208";a="12754253"
Received: from esv4-exctest.linkedin.biz ([172.18.46.60]) by CORP-MAIL.linkedin.biz with Microsoft SMTPSVC(6.0.3790.3959);
	 Fri, 7 May 2010 08:15:16 -0700
Received: from ESV4-CAS02.linkedin.biz (172.18.46.142) by
 esv4-exctest.linkedin.biz (172.18.46.60) with Microsoft SMTP Server (TLS) id
 14.0.682.1; Fri, 7 May 2010 08:15:16 -0700
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi; Fri, 7 May 2010 08:15:15
 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: Could not obtain block blk_
Thread-Topic: Could not obtain block blk_
Thread-Index: AQHK7cslvRqArxoxHkGfS1sgsrKckJJGNImA////6ICAAAabgIAATuUA
Date: Fri, 7 May 2010 15:15:14 +0000
Message-ID: <9321E246-3120-421A-B0C6-7575EBF245F6@linkedin.com>
References: <v2x8abfdb8a1005070252sccae6f4an2c15487960279999@mail.gmail.com>
	 <4BE3E6DD.5050205@apache.org>
 <k2p8abfdb8a1005070309p38d26f7dqae30854020ec8a7e@mail.gmail.com>
 <4BE3EC53.7030300@apache.org>
In-Reply-To: <4BE3EC53.7030300@apache.org>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Content-Type: text/plain; charset="us-ascii"
Content-ID: <4ff0f662-31f5-4ba5-a4aa-212dc7e8cb43>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginalArrivalTime: 07 May 2010 15:15:16.0780 (UTC) FILETIME=[196E1EC0:01CAEDF8]


On May 7, 2010, at 3:32 AM, Steve Loughran wrote:

> Pierre ANCELOT wrote:
>> All my nodes are up, I can get the file through the DFS client...
>> 0.20.2
>=20
> OK, that's interesting. Try looking at the replication count and upping i=
t, or copying the file to make sure its all there.

Or running fsck against that file directly.=

From general-return-1497-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri May 07 17:35:03 2010
Return-Path: <general-return-1497-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98277 invoked from network); 7 May 2010 17:35:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 May 2010 17:35:03 -0000
Received: (qmail 91845 invoked by uid 500); 7 May 2010 17:35:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91738 invoked by uid 500); 7 May 2010 17:35:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91730 invoked by uid 99); 7 May 2010 17:35:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 May 2010 17:35:01 +0000
X-ASF-Spam-Status: No, hits=2.4 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 74.125.83.48 as permitted sender)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 May 2010 17:34:56 +0000
Received: by gwj15 with SMTP id 15so917198gwj.35
        for <general@hadoop.apache.org>; Fri, 07 May 2010 10:34:35 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=P6FTkANRgv2R2YuA4o5jpaAN9rkZGnvsDoy/tUap828=;
        b=WVwRyZ44xij910nzKvcgGCgN/xLuIgmHzRnsNwGpqoUiaJI9BGvgG9RYra+HQ3UBSv
         aOEdayF6hNlKR9hGWAE8ZY2lq7mVDYXXpOYo1CshecsxmhSHCupKWz+HUd96KmyePg7M
         1hXjgwP7UNCS8XaPpmfy9VOcQNazvRBTzrvs0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=O45//0hGlP0fRNu1LPSpSzjM787aRsswNdT5JzdVw/2pmghuQ0hyKX5nXcnm9spQuB
         xMLhX5e2N9o0Qewp9dUE04ql1iZ8nD6hHzkFSFbAHn5SPktz2Kcf9LKV2K9xgTd2jJI2
         iskQAelZ9+PWpkpZCXRC31v7cL0+TNjnuwAxU=
MIME-Version: 1.0
Received: by 10.150.254.7 with SMTP id b7mr3700149ybi.293.1273253675577; Fri, 
	07 May 2010 10:34:35 -0700 (PDT)
Received: by 10.151.51.3 with HTTP; Fri, 7 May 2010 10:34:35 -0700 (PDT)
Date: Fri, 7 May 2010 10:34:35 -0700
Message-ID: <h2m4aa34eb71005071034ja97e2f4ckca2a016efedee897@mail.gmail.com>
Subject: Hadoop support for hbase
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd34dfa0ab4c204860479f9

--000e0cd34dfa0ab4c204860479f9
Content-Type: text/plain; charset=ISO-8859-1

Hi folks,

I would like to open a discussion on how we can make HBase work well with a
supported/released version of Hadoop. HBase currently ships with a hadoop
jar and that hadoop jar is from hadoop 0.20 + a set of ten/twenty patches.
Most of these patches are focussed on HDFS append support in hadoop 0.20.
These cannot be ported back to the 0.20 branch without affecting stability
of the hadoop 0.20 branch. On the other hand, it is premature for hbase
deployments to use hadoop 0.21 because hadoop 0.21 is still under testing
and will take some time to stabilize.

My proposal is to create a new branch off the hadoop 0.20 branch and name
it branch-0.20-hbase. It will have support for append/sync and will be API
compatible with the hadoop 0.20 branch. However, this branch will be marked
"experimental" and API compatibility is subject to change. This branch will
contain all of hdfs/mapreduce/core.

If the community likes this idea, I will volunteer myself to be the release
manager for this new branch and will propose a formal vote.

comments/feedback/questions are most welcome.
dhruba


-- 
Connect to me at http://www.facebook.com/dhruba

--000e0cd34dfa0ab4c204860479f9--

From general-return-1498-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri May 07 17:58:38 2010
Return-Path: <general-return-1498-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 3254 invoked from network); 7 May 2010 17:58:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 May 2010 17:58:38 -0000
Received: (qmail 26056 invoked by uid 500); 7 May 2010 17:58:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26008 invoked by uid 500); 7 May 2010 17:58:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26000 invoked by uid 99); 7 May 2010 17:58:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 May 2010 17:58:37 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 May 2010 17:58:30 +0000
Received: by gyf1 with SMTP id 1so933352gyf.35
        for <general@hadoop.apache.org>; Fri, 07 May 2010 10:58:09 -0700 (PDT)
Received: by 10.231.190.5 with SMTP id dg5mr155711ibb.44.1273255089593; Fri, 
	07 May 2010 10:58:09 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.160.70 with HTTP; Fri, 7 May 2010 10:57:48 -0700 (PDT)
In-Reply-To: <h2m4aa34eb71005071034ja97e2f4ckca2a016efedee897@mail.gmail.com>
References: <h2m4aa34eb71005071034ja97e2f4ckca2a016efedee897@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Fri, 7 May 2010 10:57:48 -0700
Message-ID: <z2i45f85f71005071057u302f5d7by2f6857d4295f248d@mail.gmail.com>
Subject: Re: Hadoop support for hbase
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016367b6be852dc9b048604cda4
X-Virus-Checked: Checked by ClamAV on apache.org

--0016367b6be852dc9b048604cda4
Content-Type: text/plain; charset=ISO-8859-1

I have a few questions about this proposal:

1) Will we open new JIRAs separately for each change we want to commit, and
go through the normal review process? Currently the 20-append work has been
mostly going on under HDFS-142 for whatever reason, with ancillary issues
only for bugs that also exist in trunk.
2) Do we plan to do a release off the branch, or is it meant only as a
repository for sharing patches and a tree?
3) If we do a release, what version number would we give it and how would it
be presented on the download/release pages?

I'm certainly not against the idea, just would like to open discussion on
the above points. The other alternative as I see it is to have those working
on this branch do so somewhere like github - the advantages to that would be
(a) it provides a more open way for non-committers to contribute, which is
important since we're working closely with the HBase team on this, and (b)
it doesn't add confusion to the main Hadoop jira and download pages. The
disadvantage of course is that it fragments the code repository and we can't
really do a release as easily.

Thanks
-Todd

On Fri, May 7, 2010 at 10:34 AM, Dhruba Borthakur <dhruba@gmail.com> wrote:

> Hi folks,
>
> I would like to open a discussion on how we can make HBase work well with a
> supported/released version of Hadoop. HBase currently ships with a hadoop
> jar and that hadoop jar is from hadoop 0.20 + a set of ten/twenty patches.
> Most of these patches are focussed on HDFS append support in hadoop 0.20.
> These cannot be ported back to the 0.20 branch without affecting stability
> of the hadoop 0.20 branch. On the other hand, it is premature for hbase
> deployments to use hadoop 0.21 because hadoop 0.21 is still under testing
> and will take some time to stabilize.
>
> My proposal is to create a new branch off the hadoop 0.20 branch and name
> it branch-0.20-hbase. It will have support for append/sync and will be API
> compatible with the hadoop 0.20 branch. However, this branch will be marked
> "experimental" and API compatibility is subject to change. This branch will
> contain all of hdfs/mapreduce/core.
>
> If the community likes this idea, I will volunteer myself to be the release
> manager for this new branch and will propose a formal vote.
>
> comments/feedback/questions are most welcome.
> dhruba
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--0016367b6be852dc9b048604cda4--

From general-return-1499-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri May 07 19:26:31 2010
Return-Path: <general-return-1499-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31313 invoked from network); 7 May 2010 19:26:31 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 May 2010 19:26:31 -0000
Received: (qmail 48797 invoked by uid 500); 7 May 2010 19:26:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48628 invoked by uid 500); 7 May 2010 19:26:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48620 invoked by uid 99); 7 May 2010 19:26:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 May 2010 19:26:30 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 07 May 2010 19:26:27 +0000
Received: (qmail 31282 invoked by uid 99); 7 May 2010 19:26:06 -0000
Received: from localhost.apache.org (HELO [192.168.168.114]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 May 2010 19:26:06 +0000
Message-ID: <4BE4694D.4030808@apache.org>
Date: Fri, 07 May 2010 12:26:05 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.9) Gecko/20100423 Thunderbird/3.0.4
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Hadoop support for hbase
References: <h2m4aa34eb71005071034ja97e2f4ckca2a016efedee897@mail.gmail.com> <z2i45f85f71005071057u302f5d7by2f6857d4295f248d@mail.gmail.com>
In-Reply-To: <z2i45f85f71005071057u302f5d7by2f6857d4295f248d@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 05/07/2010 10:57 AM, Todd Lipcon wrote:
> 1) Will we open new JIRAs separately for each change we want to commit, and
> go through the normal review process? Currently the 20-append work has been
> mostly going on under HDFS-142 for whatever reason, with ancillary issues
> only for bugs that also exist in trunk.

I think the proposal discussed yesterday was that the release manager 
has the power to determine which patches are merged from trunk to his 
branch, to cherry pick.  But for patches that are never committed to 
trunk but only to his branch, I think we should use normal rules.  Note 
that, under normal rules, the release manager still has veto power over 
his branch.

> The other alternative as I see it is to have those working
> on this branch do so somewhere like github - the advantages to that would be
> (a) it provides a more open way for non-committers to contribute, which is
> important since we're working closely with the HBase team on this, and (b)
> it doesn't add confusion to the main Hadoop jira and download pages. The
> disadvantage of course is that it fragments the code repository and we can't
> really do a release as easily.

If the purpose of the branch is for diverse Apache contributors to share 
code and make Apache releases, we should keep it at Apache rather than 
at github.

If the branch really is only intended to be used by HBase, then perhaps 
the branch could live under hbase, e.g., hbase/branches/hdfs-0.20

If we think it might be used by others beyond HBase, or by clusters that 
are used for more than just HBase, then we might keep it under HDFS.  We 
could label it according to the feature it concerns, e.g., 
hdfs/branches/0.20-append, or, if we think it's a set of features of 
which HBase is the only consumer, then hdfs/branches/0.20-hbase might 
make sense.

Another alternative might be to name the branch after the release 
manager.  So this could be named hdfs/branches/0.20-dhruba.  Then, when 
it comes time to make a release from this branch, we might name the 
release something different, like or 0.20.3 or somesuch.

This has the potential to create a lot of fragmentation.  Ultimately we 
depend on the PMC to control this by not voting for releases from a 
multitude of branches.  But, prior to that, we might discourage folks 
from creating such branches if we think they'll unduly fragment things.

The current proposal is to fragment 0.20 into two variants, hbase and 
vanilla.  It's probable that someone might also propose a secure 
fragment.  Three variants might be confusing.  Is there any chance we 
could combine some of these?  For example, could vanilla become secure, 
or could hbase become secure?  Do folks think three fragments would be 
acceptable?  If not, which would you drop or merge?

We can delay final answers to such questions until a release vote, but 
it'd be good to have at least a general direction earlier.

Doug

From general-return-1500-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat May 08 08:58:57 2010
Return-Path: <general-return-1500-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 61778 invoked from network); 8 May 2010 08:58:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 May 2010 08:58:56 -0000
Received: (qmail 70627 invoked by uid 500); 8 May 2010 08:58:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70382 invoked by uid 500); 8 May 2010 08:58:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70374 invoked by uid 99); 8 May 2010 08:58:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 08 May 2010 08:58:54 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of panxiaoliang@baixing.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 08 May 2010 08:58:47 +0000
Received: by pwi2 with SMTP id 2so954240pwi.35
        for <general@hadoop.apache.org>; Sat, 08 May 2010 01:58:26 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.141.188.37 with SMTP id q37mr688086rvp.212.1273309106239; Sat, 
	08 May 2010 01:58:26 -0700 (PDT)
Received: by 10.140.142.1 with HTTP; Sat, 8 May 2010 01:58:26 -0700 (PDT)
Date: Sat, 8 May 2010 16:58:26 +0800
Message-ID: <s2w4ac67a081005080158jfb49ff5bg7ab6226d052072f4@mail.gmail.com>
Subject: baixing.com is looking for data warehouse talents
From: =?UTF-8?B?5r2Y5pmT6Imv?= <panxiaoliang@baixing.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd1b4f6f77b1804861160b2
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd1b4f6f77b1804861160b2
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Sorry for bothering you. This is an advertisement, about one job opportunit=
y
in Shanghai, China. If you don't like it, please ignore it.

Baixing.com is one start-up company invested by eBay, and it's in eBay
Classified Group.

We started in March 1st, 2005, and now we have a big C2C classified site
with one billion page view every month, and 300 million ads. If you want to
learn more, please try using it: http://www.baixing.com

Why work in baixing.com?

1. We are a small team with talents
a. Keep strong growth with only 1/10 of employees of competitors
b. 1 million RMB revenue per employee in last year
c. High pressure, Fast growth and High return

2. We also have big global company's benefit=EF=BC=89
a. We are sharing with classified experience globally, half of our employee=
s
have business trips to U.S. or Europe
b. We have leveling and performance reviewing system, good work, good pay
c. We are in Xujiahui CBD with Convenient traffic, and metrical lines 1#,
9#, 10# are nearby
d. For employees who need relocate, we will pay two month's house rent fee,
and after that we have allowance for house renting too
e. We have free breakfast, fruit every day,  and we have many kinds of
sports like basketball, badminton, travel etc.

Here is jobs opening:

   - Software Engineer=EF=BC=88Data
Warehouse=EF=BC=89<http://shanghai.baixing.com/chengxuyuan/a49073017.html>



--=20
=E6=99=93=E8=89=AF

--000e0cd1b4f6f77b1804861160b2--

From general-return-1501-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat May 08 16:59:45 2010
Return-Path: <general-return-1501-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 31979 invoked from network); 8 May 2010 16:59:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 May 2010 16:59:44 -0000
Received: (qmail 86418 invoked by uid 500); 8 May 2010 16:59:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86348 invoked by uid 500); 8 May 2010 16:59:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86333 invoked by uid 99); 8 May 2010 16:59:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 08 May 2010 16:59:43 +0000
X-ASF-Spam-Status: No, hits=-0.3 required=10.0
	tests=AWL,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [88.198.2.104] (HELO koch.ro) (88.198.2.104)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 08 May 2010 16:59:36 +0000
Received: from 224-162.76-83.cust.bluewin.ch ([83.76.162.224] helo=jona.localnet)
	by koch.ro with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
	(Exim 4.69)
	(envelope-from <thomas@koch.ro>)
	id 1OAnGX-0005sp-Hj; Sat, 08 May 2010 18:52:37 +0200
From: Thomas Koch <thomas@koch.ro>
Reply-To: thomas@koch.ro
To: general@hadoop.apache.org
Subject: Re: Hadoop support for hbase
Date: Sat, 8 May 2010 18:59:07 +0200
User-Agent: KMail/1.13.3 (Linux/2.6.32-4-amd64; KDE/4.4.3; x86_64; ; )
Cc: Doug Cutting <cutting@apache.org>
References: <h2m4aa34eb71005071034ja97e2f4ckca2a016efedee897@mail.gmail.com> <z2i45f85f71005071057u302f5d7by2f6857d4295f248d@mail.gmail.com> <4BE4694D.4030808@apache.org>
In-Reply-To: <4BE4694D.4030808@apache.org>
MIME-Version: 1.0
Content-Type: Text/Plain;
  charset="us-ascii"
Content-Transfer-Encoding: 7bit
Message-Id: <201005081859.08029.thomas@koch.ro>

I'm a little confused and concerned now that I learn that hbase uses a patches 
hadoop. For Debian I use plain hadoop under hbase and it seems to work in 
testing environments.

- Are these patches necessary to run HBase?
- Where can I find these patches?
- Why aren't these patches included in hadoop? Are they too unstable?
- If they're unstable, does this mean, HBase is unstable?

Should I worry at all about these patches for the Debian packages?

regards,

Thomas Koch, http://www.koch.ro

From general-return-1502-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat May 08 17:10:48 2010
Return-Path: <general-return-1502-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33826 invoked from network); 8 May 2010 17:10:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 May 2010 17:10:48 -0000
Received: (qmail 93117 invoked by uid 500); 8 May 2010 17:10:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93071 invoked by uid 500); 8 May 2010 17:10:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93057 invoked by uid 99); 8 May 2010 17:10:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 08 May 2010 17:10:47 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=AWL,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 08 May 2010 17:10:41 +0000
Received: by gyf1 with SMTP id 1so1540029gyf.35
        for <general@hadoop.apache.org>; Sat, 08 May 2010 10:10:20 -0700 (PDT)
Received: by 10.231.156.75 with SMTP id v11mr1077230ibw.25.1273338620591; Sat, 
	08 May 2010 10:10:20 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.160.70 with HTTP; Sat, 8 May 2010 10:10:00 -0700 (PDT)
In-Reply-To: <201005081859.08029.thomas@koch.ro>
References: <h2m4aa34eb71005071034ja97e2f4ckca2a016efedee897@mail.gmail.com> 
	<z2i45f85f71005071057u302f5d7by2f6857d4295f248d@mail.gmail.com> 
	<4BE4694D.4030808@apache.org> <201005081859.08029.thomas@koch.ro>
From: Todd Lipcon <todd@cloudera.com>
Date: Sat, 8 May 2010 10:10:00 -0700
Message-ID: <i2j45f85f71005081010nf6a5b6f0k24d989d15361840f@mail.gmail.com>
Subject: Re: Hadoop support for hbase
To: general@hadoop.apache.org, thomas@koch.ro
Cc: Doug Cutting <cutting@apache.org>
Content-Type: multipart/alternative; boundary=0050450156d628bd750486184085

--0050450156d628bd750486184085
Content-Type: text/plain; charset=ISO-8859-1

On Sat, May 8, 2010 at 9:59 AM, Thomas Koch <thomas@koch.ro> wrote:

> I'm a little confused and concerned now that I learn that hbase uses a
> patches
> hadoop. For Debian I use plain hadoop under hbase and it seems to work in
> testing environments.


> - Are these patches necessary to run HBase?
>

It will work unless you have failures, in which case it will lose edits.
HBase relies on the "hflush" API (called "sync" in 0.20) which does not work
properly in 0.20 without significant patching.

Without this patch series, HBase will certainly run, but I could never
recommend running it in a production environment where data loss is a
show-stopper.


> - Where can I find these patches?
>

Currently they're in various places on the JIRA - HDFS-200, HDFS-142,
HDFS-826, HDFS-561, etc. I have a github branch up which contains them all
applied, but I haven't tested it beyond unit tests - my testing is all
happening in our CDH3 tree, and afaik Dhruba's testing is on their FB
internal tree.



> - Why aren't these patches included in hadoop? Are they too unstable?
>

Yes, the policy is not to make such significant changes in patch releases,
so they would need to be voted into the 0.20 series. It's not that they're
entirely unstable, it's just that the code is very tricky and still under
development. The upcoming 0.21 release has a *different* implementation of
Append which also hasn't been tested significantly in real life failure
scenarios, but it's important that we keep the stable release stable.


> - If they're unstable, does this mean, HBase is unstable?
>
>
Again I would not say it's terribly unstable - but it's nowhere near the
level of stability that Hadoop is

>  Should I worry at all about these patches for the Debian packages?

If you expect that people might actually want to run a production HBase,
they should have the patches. If you expect people to just be playing around
on single node "clusters" where failures aren't an issue, best to skip. Of
course, even for production usage, I wouldn't recommend running what we've
got now - wait a month or two and it should be one more notch up the
stability/testing scale.

-Todd

-- 
Todd Lipcon
Software Engineer, Cloudera

--0050450156d628bd750486184085--

From general-return-1503-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun May 09 05:42:53 2010
Return-Path: <general-return-1503-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 18313 invoked from network); 9 May 2010 05:42:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 May 2010 05:42:53 -0000
Received: (qmail 31775 invoked by uid 500); 9 May 2010 05:42:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31598 invoked by uid 500); 9 May 2010 05:42:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31590 invoked by uid 99); 9 May 2010 05:42:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 09 May 2010 05:42:51 +0000
X-ASF-Spam-Status: No, hits=0.8 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of raovijay@gmail.com designates 209.85.221.172 as permitted sender)
Received: from [209.85.221.172] (HELO mail-qy0-f172.google.com) (209.85.221.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 09 May 2010 05:42:45 +0000
Received: by qyk2 with SMTP id 2so4553534qyk.20
        for <general@hadoop.apache.org>; Sat, 08 May 2010 22:42:24 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=gOEgX6V/9uZ1J4aGEOCdhf3eRRxUXRrXx3Ny/epu4ss=;
        b=aW580zvayJnRGkjJW406yGWyeJmYYSnRbmfFba8QS0MfCXFcqzgxSoE48qJWsDt1tO
         FEqw/33zGgE+KLynGnYB5HsOsHHfPHKBmRAuG7lHJ939AQrdZKa7pH5tewTtOekysHRr
         0G2cNysPU7CsI/dCYrMUNVVncSP0laIWvCgRk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=Tuc3501cMNiZSZBTvZoDDgzi9Dw2JDUeU6yelHjT/xrTNlnGBrlO8pSNhlCdPIzOcL
         GiVRpyjgRmGIn3JriAL+vbz3v0anvtqog6hkcxWaeZslFF+x3GWkFIdZdF81T0ND6F+J
         3ud+CmvKrA849VLiUsBWToJ6hp4LJkIIsz5D0=
MIME-Version: 1.0
Received: by 10.229.188.72 with SMTP id cz8mr1704762qcb.75.1273383744440; Sat, 
	08 May 2010 22:42:24 -0700 (PDT)
Received: by 10.229.96.209 with HTTP; Sat, 8 May 2010 22:42:24 -0700 (PDT)
Date: Sat, 8 May 2010 22:42:24 -0700
Message-ID: <y2zbc8d35a01005082242ib9707246m4a7f79178c58d774@mail.gmail.com>
Subject: Fundamental question
From: Vijay Rao <raovijay@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016364ecce8c00a39048622c177

--0016364ecce8c00a39048622c177
Content-Type: text/plain; charset=ISO-8859-1

Hello,

I am just reading and understanding Hadoop and all the other components.
However I have a fundamental question for which I am not getting answers in
any of the online material that is out there.

1) If hadoop is used then all the slaves and other machines in the cluster
need to be formatted to have HDFS file system. If so what happens to the
tera bytes of data that need to be crunched? Or is the data on a different
machine?

2) Everywhere it is mentioned that the main advantage of map/reduce and
hadoop is that it runs on data that is available locally. So does this mean
that once the file system is formatted then I have to move my terabytes of
data and split them across the cluster?

Thanks
VJ

--0016364ecce8c00a39048622c177--

From general-return-1504-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun May 09 06:45:08 2010
Return-Path: <general-return-1504-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33397 invoked from network); 9 May 2010 06:45:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 May 2010 06:45:07 -0000
Received: (qmail 57517 invoked by uid 500); 9 May 2010 06:45:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57411 invoked by uid 500); 9 May 2010 06:45:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57403 invoked by uid 99); 9 May 2010 06:45:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 09 May 2010 06:45:06 +0000
X-ASF-Spam-Status: No, hits=1.4 required=10.0
	tests=AWL,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of timrobertson100@gmail.com designates 209.85.210.184 as permitted sender)
Received: from [209.85.210.184] (HELO mail-yx0-f184.google.com) (209.85.210.184)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 09 May 2010 06:45:00 +0000
Received: by yxe14 with SMTP id 14so2756yxe.5
        for <general@hadoop.apache.org>; Sat, 08 May 2010 23:44:39 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=C3+E2Rp1MDwC0JH0lpONb6Zjn8t0zrBfax6KdDhm9Eg=;
        b=ED5MoNGQ/j2+ksAtyIc4c8SmAPvMpCoEEEjDf5NPtDfqHPwjsURzUIjNRjQurL0S/Z
         vsvFGMpLshWNYQtiazZbfdIEeimTpooTLI37hSz0lAvT5lXpAgMgZuNi0Z0arZQ3Vb3k
         bhtTjur1CACUOLfRLYkh1TS8FvgbOUlN6geDk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=X/+BX+ezPh0xaYQpYcuOulQxTMi/7mqJYDmJTfV/eup/EBiwjY05k1TgDtyU+ZgCQZ
         w3cA5xy52lNBOrsetmHbf3vgvSv5gxTjVdcjlFTlgop1t3BvSAr5AcGhjOgDaTYQzcEM
         EY5VU9SZ79MKMQpwtAntTXV+QuNSu0d9flQUk=
MIME-Version: 1.0
Received: by 10.150.239.1 with SMTP id m1mr6509505ybh.311.1273387478756; Sat, 
	08 May 2010 23:44:38 -0700 (PDT)
Received: by 10.150.96.7 with HTTP; Sat, 8 May 2010 23:44:38 -0700 (PDT)
In-Reply-To: <y2zbc8d35a01005082242ib9707246m4a7f79178c58d774@mail.gmail.com>
References: <y2zbc8d35a01005082242ib9707246m4a7f79178c58d774@mail.gmail.com>
Date: Sun, 9 May 2010 08:44:38 +0200
Message-ID: <s2x32120a6a1005082344za34e4ea0i3ac23f02eb0c41cd@mail.gmail.com>
Subject: Re: Fundamental question
From: Tim Robertson <timrobertson100@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd25880552fca048623a069

--000e0cd25880552fca048623a069
Content-Type: text/plain; charset=ISO-8859-1

Hi VJ

1) If hadoop is used then all the slaves and other machines in the cluster
> need to be formatted to have HDFS file system. If so what happens to the
> tera bytes of data that need to be crunched? Or is the data on a different
> machine?
>

You actually assign directories on the machines to be the directories Hadoop
uses in the DFS.  Therefore the machines can also have other data as you
don't format the whole drives.  So for example I have /mnt/disk1/hadoop and
/mnt/disk2/hadoop on each of my Data Nodes for HDFS to use.  My machines are
dedicated to Hadoop so don't store other data in addition.

2) Everywhere it is mentioned that the main advantage of map/reduce and
> hadoop is that it runs on data that is available locally. So does this mean
> that once the file system is formatted then I have to move my terabytes of
> data and split them across the cluster?
>

Once you copy data into HDFS which you then *might* consider removing from
the local drives.  I think it is more common to dedicate a cluster to Hadoop
and then copy data into the DFS from external locations (e.g. it doesn't
also sit on local drives in the Hadoop cluster).  This is how we use it
anyway.  When you launch a MR job, it knows where the data chunks are
located and then runs the processing on the machine in the cluster with
those chunks.  Remember HDFS will store redundant copies, so you might copy
in a 200GB file, it gets split up and copied around the cluster and perhaps
each chunk saved 3 times.  Then when it needs to process, there are 3
machines with any given chunk locally stored - Hadoop will try and schedule
the tasks needed to complete the job to minimise copying around and run it
on a machine with the data already.

Since you seem interested in the best set up for MapReduce you might get
better responses on the mapreduce-user mailing list.

Hope this helps,
Tim


> Thanks
> VJ
>

--000e0cd25880552fca048623a069--

From general-return-1505-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun May 09 21:09:07 2010
Return-Path: <general-return-1505-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68645 invoked from network); 9 May 2010 21:09:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 May 2010 21:09:06 -0000
Received: (qmail 46524 invoked by uid 500); 9 May 2010 21:09:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46446 invoked by uid 500); 9 May 2010 21:09:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46438 invoked by uid 99); 9 May 2010 21:09:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 09 May 2010 21:09:04 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=AWL,HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 74.125.82.48 is neither permitted nor denied by domain of oded@legolas-media.com)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 09 May 2010 21:08:59 +0000
Received: by wwb28 with SMTP id 28so1770484wwb.35
        for <general@hadoop.apache.org>; Sun, 09 May 2010 14:08:37 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.154.69 with SMTP id g47mr1790831wek.82.1273439317216; Sun, 
	09 May 2010 14:08:37 -0700 (PDT)
Received: by 10.216.139.233 with HTTP; Sun, 9 May 2010 14:08:37 -0700 (PDT)
Date: Mon, 10 May 2010 00:08:37 +0300
Message-ID: <AANLkTilXGm2IwnoVEP8hHbtXd6wuzU8Ln3f7xS-27KwT@mail.gmail.com>
Subject: MultipleInputs in 0.20
From: Oded Rosen <oded@legolas-media.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e65ae518254ff904862fb27e

--0016e65ae518254ff904862fb27e
Content-Type: text/plain; charset=ISO-8859-1

By what I've learned from different sites around the web (hadoop wiki,
cloudera<http://www.cloudera.com/blog/2009/05/what%E2%80%99s-new-in-hadoop-core-020/>,
mail archive, etc),
the MultipleInputs class that was available in 0.18-0.19 versions of hadoop,
was not moved to the 0.20 new API.
(so does MultipleOutputs, but that's another story)

I wanted to know if there is a way around this - to use two different paths
with two different input format (sequence file, text file) as sources to the
same job,
with a special mapper for each input type - using hadoop 0.20 API. I think
that writing a new job using 0.19 API only means more trouble later, when
it's officially deprecated.

I saw there is a jira <goog_292716485>
(MAPREDUCE-1170)<https://issues.apache.org/jira/browse/MAPREDUCE-1170>open
for this issue, with a patch marked as "Won't fix".
If someone out there can help me with this, I will be most thankful.

Cheers,
-- 
Oded

--0016e65ae518254ff904862fb27e--

From general-return-1506-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun May 09 22:06:46 2010
Return-Path: <general-return-1506-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76230 invoked from network); 9 May 2010 22:06:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 May 2010 22:06:46 -0000
Received: (qmail 72598 invoked by uid 500); 9 May 2010 22:06:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72522 invoked by uid 500); 9 May 2010 22:06:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72514 invoked by uid 99); 9 May 2010 22:06:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 09 May 2010 22:06:45 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 209.85.221.172 as permitted sender)
Received: from [209.85.221.172] (HELO mail-qy0-f172.google.com) (209.85.221.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 09 May 2010 22:06:37 +0000
Received: by qyk2 with SMTP id 2so5517009qyk.20
        for <general@hadoop.apache.org>; Sun, 09 May 2010 15:06:17 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        bh=ayxvp4+g46sET8Ws7AaMdfq1L4sePdXs4Btw48oRHjU=;
        b=h/D1t1EI302QzIwYs79p8MxLDufuYGHJzCMG92BCWBAszppQ1SFoVO94kAv+mu7PXZ
         S6bNvTkgLKp5wPULryYCYq1ho5IOd83t+/6V85zKRc1fs3ULIGcC7mfhbGECDwH1ol6D
         MlEB1oiKryuuJKWwqhf9cS88NFRZbuRQFt7VY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=l4dsDXbQpLoJtFwCgLAnxRyC0g7OV/7VGvxybKuPKU7pQAu0UB8yDi+F5A+0ASyowP
         ofAoLR39UhDCBwgcpxlqDmaW9avusbOGuJ49aSDMMAZMwkqdgeKxCj+wfnJHWa9Ozgdd
         lcCcnAwhKdgahJo1YPp60vjCToTEM42FEEN2g=
MIME-Version: 1.0
Received: by 10.224.104.93 with SMTP id n29mr1987181qao.193.1273442776234; 
	Sun, 09 May 2010 15:06:16 -0700 (PDT)
Sender: saint.ack@gmail.com
Received: by 10.229.220.66 with HTTP; Sun, 9 May 2010 15:06:16 -0700 (PDT)
In-Reply-To: <h2m4aa34eb71005071034ja97e2f4ckca2a016efedee897@mail.gmail.com>
References: <h2m4aa34eb71005071034ja97e2f4ckca2a016efedee897@mail.gmail.com>
Date: Sun, 9 May 2010 15:06:16 -0700
X-Google-Sender-Auth: 4940e7b41469aa5b
Message-ID: <j2p7c962aed1005091506hab074e8ew505475e2d2401ccb@mail.gmail.com>
Subject: Re: Hadoop support for hbase
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Fri, May 7, 2010 at 10:34 AM, Dhruba Borthakur <dhruba@gmail.com> wrote:
> My proposal is to create a new branch off the hadoop 0.20 branch and name
> it branch-0.20-hbase. It will have support for append/sync and will be API
> compatible with the hadoop 0.20 branch. However, this branch will be marked
> "experimental" and API compatibility is subject to change. This branch will
> contain all of hdfs/mapreduce/core.
>
> If the community likes this idea...
>
I like this idea.  It can be a little tough keeping up with latest
state of the 0.20 "append" patch set.  Having the set all applied in
the one well-known place will make collaboration and discussion of the
current state of 0.20 "append" the easier.

Would hbase be the only party interested in such a branch?  (Scribe?)

St.Ack

From general-return-1507-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun May 09 22:15:56 2010
Return-Path: <general-return-1507-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77080 invoked from network); 9 May 2010 22:15:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 May 2010 22:15:56 -0000
Received: (qmail 74986 invoked by uid 500); 9 May 2010 22:15:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74943 invoked by uid 500); 9 May 2010 22:15:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74935 invoked by uid 99); 9 May 2010 22:15:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 09 May 2010 22:15:55 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 209.85.221.172 as permitted sender)
Received: from [209.85.221.172] (HELO mail-qy0-f172.google.com) (209.85.221.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 09 May 2010 22:15:47 +0000
Received: by qyk2 with SMTP id 2so5527153qyk.20
        for <general@hadoop.apache.org>; Sun, 09 May 2010 15:15:26 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:content-type;
        bh=yfGpb7vU/qxmZKZyKvtZSXkRlJ5OIfN9iJdFetfg4sw=;
        b=PpLwB5YWnkXhOUqhfknWsOZ4AB8plsES1zE9jrZXu8+E57LiRktDiIN0DXPeHtHS4S
         wMXuuEy6rjTtz/4mQ35aHXoZo4JCODJ4DE2Wl8tpni9YNjO5Iy8QKb4jltcg1y5nRz7F
         iBeVmwSqXnONdyFKZClX2iqnmFJkU4AnfABRI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=eXST14Yx81VzOpfolHP5aaJRZya4O//QAVj7JNHCyVWrYG3iEr4ElC/KdxQWvtaQPv
         gB74noMyTTbqPlcknLK2w1WxgUgUBP3haW/yp6KgiDsxrOny/2kZl97olOa9gg/CeZGP
         ERyEy4B06vw0xDo+1tmaRk0nqtc+TOYA5fXsM=
MIME-Version: 1.0
Received: by 10.229.225.72 with SMTP id ir8mr2002935qcb.73.1273443326232; Sun, 
	09 May 2010 15:15:26 -0700 (PDT)
Sender: saint.ack@gmail.com
Received: by 10.229.220.66 with HTTP; Sun, 9 May 2010 15:15:26 -0700 (PDT)
In-Reply-To: <h2m4aa34eb71005071034ja97e2f4ckca2a016efedee897@mail.gmail.com>
References: <h2m4aa34eb71005071034ja97e2f4ckca2a016efedee897@mail.gmail.com>
Date: Sun, 9 May 2010 15:15:26 -0700
X-Google-Sender-Auth: 7569681478dc4e99
Message-ID: <w2q7c962aed1005091515y4965fb9dpa345178a39fc9c45@mail.gmail.com>
Subject: Re: Hadoop support for hbase
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Fri, May 7, 2010 at 10:34 AM, Dhruba Borthakur <dhruba@gmail.com> wrote:
>This branch will contain all of hdfs/mapreduce/core.
>
Should it be all of hadoop?  Could it be hdfs only?
St.Ack

From general-return-1508-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun May 09 22:26:51 2010
Return-Path: <general-return-1508-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 80080 invoked from network); 9 May 2010 22:26:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 May 2010 22:26:51 -0000
Received: (qmail 5162 invoked by uid 500); 9 May 2010 22:26:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4918 invoked by uid 500); 9 May 2010 22:26:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4909 invoked by uid 99); 9 May 2010 22:26:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 09 May 2010 22:26:50 +0000
X-ASF-Spam-Status: No, hits=-0.4 required=10.0
	tests=AWL,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 209.85.221.172 as permitted sender)
Received: from [209.85.221.172] (HELO mail-qy0-f172.google.com) (209.85.221.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 09 May 2010 22:26:44 +0000
Received: by qyk2 with SMTP id 2so5539323qyk.20
        for <general@hadoop.apache.org>; Sun, 09 May 2010 15:26:23 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:content-type:content-transfer-encoding;
        bh=6cmyvfMC5OsVaZmHLoh36wy5j1SynxpF2ceKaRcR1ns=;
        b=xLx40RSgJ/IIra9FXhgU09yTOU1Vj5UTZHsFrdvnEcBXNSonjCvm86RmJYDXswCclk
         AYvge1qgrA7vkvJtSgc+iO/kwFbaXlHb7YLpJ9W045EMWrPqSw7+C/ly/r5Kd1mr9azr
         rUmRauk0T+s80x0RO5QoaPkwOuddeUZNbfTuI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=k1DLigdabJTT911uVpzDEP4R8gt6Qoodz4joe1VIPdsApSKhLfCskk17saHKtzGCXI
         KMp+vOfst57HwT41eZgLOC5Mi5nUpuAzv4cnKm3wQTfpsqTt+QGNmXSeexym/xncqMHn
         nCRPpVOBWKzrJddioTqJ7adSFMxlLQ+EG5UW8=
MIME-Version: 1.0
Received: by 10.229.227.10 with SMTP id iy10mr2478716qcb.48.1273443983808; 
	Sun, 09 May 2010 15:26:23 -0700 (PDT)
Sender: saint.ack@gmail.com
Received: by 10.229.220.66 with HTTP; Sun, 9 May 2010 15:26:23 -0700 (PDT)
In-Reply-To: <w2q7c962aed1005091515y4965fb9dpa345178a39fc9c45@mail.gmail.com>
References: <h2m4aa34eb71005071034ja97e2f4ckca2a016efedee897@mail.gmail.com>
	 <w2q7c962aed1005091515y4965fb9dpa345178a39fc9c45@mail.gmail.com>
Date: Sun, 9 May 2010 15:26:23 -0700
X-Google-Sender-Auth: c085420475694e1e
Message-ID: <w2p7c962aed1005091526ne7a51896u7165b7019322a0a6@mail.gmail.com>
Subject: Re: Hadoop support for hbase
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Sun, May 9, 2010 at 3:15 PM, Stack <stack@duboce.net> wrote:
> Should it be all of hadoop? =A0Could it be hdfs only?
>

Please ignore the above question (I just took a look at 0.20 repo).
St.Ack

From general-return-1509-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun May 09 23:50:40 2010
Return-Path: <general-return-1509-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91539 invoked from network); 9 May 2010 23:50:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 May 2010 23:50:40 -0000
Received: (qmail 61829 invoked by uid 500); 9 May 2010 23:50:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61783 invoked by uid 500); 9 May 2010 23:50:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61775 invoked by uid 99); 9 May 2010 23:50:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 09 May 2010 23:50:39 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates 209.85.221.172 as permitted sender)
Received: from [209.85.221.172] (HELO mail-qy0-f172.google.com) (209.85.221.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 09 May 2010 23:50:33 +0000
Received: by qyk2 with SMTP id 2so5639003qyk.20
        for <general@hadoop.apache.org>; Sun, 09 May 2010 16:50:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=ca9GhK33KSfNF/kUCMkkiDxdjedvYL5+44dP5o2wms8=;
        b=sz8EO4R5+FOC5mPq7fzd6BfFLI5t7Z7uYyPDu2YPZraF0Ue28t5ArY9ab/sMl09M1F
         7+Exsl1hKNfL78tAwN1/N5izEltrgr7fD4aKIzBGfWUruI6QoihOSgmhqQbUuNfg/YjC
         sC1R9qfxMpojvq4drxLlYO1GO2iCH67+npZjU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=btKHcI8zC9vCn4vlUg+uYGkMNYC15nRTocqUES7uD6slsVKEtH+OUV3CHMjTTAHQpb
         q2He8TAOkix7Lyr2pBkRKF3H6tiq4aayVNxP9k6Jx+XwNGivgy0nPdOB4LUbMATOHkCf
         Hn7qVnAAD/h2NrkNwiI1/y1PTJ59MDXGs6+ds=
MIME-Version: 1.0
Received: by 10.224.43.197 with SMTP id x5mr2042732qae.127.1273449011987; Sun, 
	09 May 2010 16:50:11 -0700 (PDT)
Received: by 10.229.2.13 with HTTP; Sun, 9 May 2010 16:50:11 -0700 (PDT)
In-Reply-To: <AANLkTilXGm2IwnoVEP8hHbtXd6wuzU8Ln3f7xS-27KwT@mail.gmail.com>
References: <AANLkTilXGm2IwnoVEP8hHbtXd6wuzU8Ln3f7xS-27KwT@mail.gmail.com>
Date: Sun, 9 May 2010 16:50:11 -0700
Message-ID: <z2q17e273101005091650ue384615cic72eb2629541d7cc@mail.gmail.com>
Subject: Re: MultipleInputs in 0.20
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00c09f88d3c2ffd4b0048631f378
X-Virus-Checked: Checked by ClamAV on apache.org

--00c09f88d3c2ffd4b0048631f378
Content-Type: text/plain; charset=ISO-8859-1

Please refer to MAPREDUCE-1743.

Other option is to duplicate MultipleInputs, DelegatingInputFormat classes
and slightly modify TaggedInputSplit (as I suggested earlier).
This way you use your own (functional) version :-)

On Sun, May 9, 2010 at 2:08 PM, Oded Rosen <oded@legolas-media.com> wrote:

> By what I've learned from different sites around the web (hadoop wiki,
> cloudera<
> http://www.cloudera.com/blog/2009/05/what%E2%80%99s-new-in-hadoop-core-020/
> >,
> mail archive, etc),
> the MultipleInputs class that was available in 0.18-0.19 versions of
> hadoop,
> was not moved to the 0.20 new API.
> (so does MultipleOutputs, but that's another story)
>
> I wanted to know if there is a way around this - to use two different paths
> with two different input format (sequence file, text file) as sources to
> the
> same job,
> with a special mapper for each input type - using hadoop 0.20 API. I think
> that writing a new job using 0.19 API only means more trouble later, when
> it's officially deprecated.
>
> I saw there is a jira <goog_292716485>
> (MAPREDUCE-1170)<https://issues.apache.org/jira/browse/MAPREDUCE-1170>open
> for this issue, with a patch marked as "Won't fix".
> If someone out there can help me with this, I will be most thankful.
>
> Cheers,
> --
> Oded
>

--00c09f88d3c2ffd4b0048631f378--

From general-return-1510-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 10 04:39:59 2010
Return-Path: <general-return-1510-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 20598 invoked from network); 10 May 2010 04:39:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 May 2010 04:39:59 -0000
Received: (qmail 64246 invoked by uid 500); 10 May 2010 04:39:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63899 invoked by uid 500); 10 May 2010 04:39:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63891 invoked by uid 99); 10 May 2010 04:39:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 May 2010 04:39:55 +0000
X-ASF-Spam-Status: No, hits=1.9 required=10.0
	tests=AWL,HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 May 2010 04:39:49 +0000
Received: from EGL-EX07CAS03.ds.corp.yahoo.com (egl-ex07cas03.eglbp.corp.yahoo.com [203.83.248.219])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o4A4cnZE005249
	for <general@hadoop.apache.org>; Sun, 9 May 2010 21:39:06 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:in-reply-to:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=Igm9HqaNHHsLLBEEIFtnvsgKDZxoXfSmk6LnrQ1nek1jZrLzb7ubqWb07OQn6ebm
Received: from EGL-EX07VS01.ds.corp.yahoo.com ([203.83.248.205]) by
 EGL-EX07CAS03.ds.corp.yahoo.com ([203.83.248.219]) with mapi; Mon, 10 May
 2010 10:08:59 +0530
From: Amareshwari Sri Ramadasu <amarsri@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 10 May 2010 10:09:04 +0530
Subject: Re: MultipleInputs in 0.20
Thread-Topic: MultipleInputs in 0.20
Thread-Index: Acrvu+0qqD52JvunR/CsV+32kgAZZQAPsrBZ
Message-ID: <C80D8BC0.160C5%amarsri@yahoo-inc.com>
In-Reply-To: <AANLkTilXGm2IwnoVEP8hHbtXd6wuzU8Ln3f7xS-27KwT@mail.gmail.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C80D8BC0160C5amarsriyahooinccom_"
MIME-Version: 1.0

--_000_C80D8BC0160C5amarsriyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

MultipleInputs is ported new api in branch 0.21 through https://issues.apac=
he.org/jira/browse/MAPREDUCE-369

-Amareshwari

On 5/10/10 2:38 AM, "Oded Rosen" <oded@legolas-media.com> wrote:

By what I've learned from different sites around the web (hadoop wiki,
cloudera<http://www.cloudera.com/blog/2009/05/what%E2%80%99s-new-in-hadoop-=
core-020/>,
mail archive, etc),
the MultipleInputs class that was available in 0.18-0.19 versions of hadoop=
,
was not moved to the 0.20 new API.
(so does MultipleOutputs, but that's another story)

I wanted to know if there is a way around this - to use two different paths
with two different input format (sequence file, text file) as sources to th=
e
same job,
with a special mapper for each input type - using hadoop 0.20 API. I think
that writing a new job using 0.19 API only means more trouble later, when
it's officially deprecated.

I saw there is a jira <goog_292716485>
(MAPREDUCE-1170)<https://issues.apache.org/jira/browse/MAPREDUCE-1170>open
for this issue, with a patch marked as "Won't fix".
If someone out there can help me with this, I will be most thankful.

Cheers,
--
Oded


--_000_C80D8BC0160C5amarsriyahooinccom_--

From general-return-1511-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 10 04:51:10 2010
Return-Path: <general-return-1511-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21428 invoked from network); 10 May 2010 04:51:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 May 2010 04:51:09 -0000
Received: (qmail 66701 invoked by uid 500); 10 May 2010 04:51:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66630 invoked by uid 500); 10 May 2010 04:51:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66622 invoked by uid 99); 10 May 2010 04:51:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 May 2010 04:51:07 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of dhruba@gmail.com designates 209.85.211.199 as permitted sender)
Received: from [209.85.211.199] (HELO mail-yw0-f199.google.com) (209.85.211.199)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 May 2010 04:51:02 +0000
Received: by ywh37 with SMTP id 37so2279327ywh.2
        for <general@hadoop.apache.org>; Sun, 09 May 2010 21:50:41 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=w7Tqd/W3UEvIlet4XF7zMSjxYD+Knj+AOiqWtjD5TCQ=;
        b=q6o2/wH9+kiUZZwuMgor0CEWniUJxe79T0/Ox+6+7jSzizI92gWjELVaSuFOA9L2Ao
         qMjP8gUXc+Tnmwid0lqu5rYAcjYdi54r3pahylNERcD6lQ8iQAmVtUWzBN5BuAIj/BDG
         d92s+Rrad9FB/QATPtwi4yeXVn1O+OLioiV+I=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=X0Umr0uXIdp4bsEKs9XLYP2igSBEGuEFlNunkwzntCrdKW+ADbCPJ2qChCX1SuSh1g
         LzgssMpoxrM6aVxJXMcYdnf05Cr7XnE8NHEtNGWvbeh+Wyko9bHuzooYBSzI1GfCUpe+
         FvzL5y7WeyuaZCGo+AtqXf62bWdSKQZ/I/zhY=
MIME-Version: 1.0
Received: by 10.150.118.26 with SMTP id q26mr7703623ybc.325.1273467041731; 
	Sun, 09 May 2010 21:50:41 -0700 (PDT)
Received: by 10.151.51.3 with HTTP; Sun, 9 May 2010 21:50:41 -0700 (PDT)
In-Reply-To: <w2p7c962aed1005091526ne7a51896u7165b7019322a0a6@mail.gmail.com>
References: <h2m4aa34eb71005071034ja97e2f4ckca2a016efedee897@mail.gmail.com>
	 <w2q7c962aed1005091515y4965fb9dpa345178a39fc9c45@mail.gmail.com>
	 <w2p7c962aed1005091526ne7a51896u7165b7019322a0a6@mail.gmail.com>
Date: Sun, 9 May 2010 21:50:41 -0700
Message-ID: <k2i4aa34eb71005092150sdcc4ab8fnac9b152b2788ce38@mail.gmail.com>
Subject: Re: Hadoop support for hbase
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd70ecea7d4c3048636267b

--000e0cd70ecea7d4c3048636267b
Content-Type: text/plain; charset=ISO-8859-1

I will try to explain my opinion on some of the questions being raised so
far. Of course all these are open for discussion and nothing is final at
this moment.

1. This code base is primarily targeted for usage by HBase and Scribe. Both
of these are Apache open source projects.
2. Accordingly, my proposal is to name the branch hadoop-0.20-append. This
name is more generic than the earlier one I suggested.
3. The proposal to pull patches into this branch will follow normal
conventions that we follow in Hadoop. However, the release manager would
have an option to veto a patch from being pulled into the branch.
4. if this type of branching causes too many forks of core Hadoop, then we
can aim to merge some of them after some time. The time duration depends on
the stability of the code in that particular branch and is difficult to
predict. The PMC, of course, has to approve of any new branch; so, in effect
can prevent undesired multiple forks if that becomes a problem in the
future.
5. code changes to this new branch will go through the normal process via
JIRAs, code reviews, unit tests, etc.
6. the goal is to have a standard hadoop release from  this branch at some
future point. if course, such a release has to be approved by the PMC. The
release could be marked as "experimental" or some such thing if deemed
appropriate by the PMC.

thanks,
dhruba


On Sun, May 9, 2010 at 3:26 PM, Stack <stack@duboce.net> wrote:

> On Sun, May 9, 2010 at 3:15 PM, Stack <stack@duboce.net> wrote:
> > Should it be all of hadoop?  Could it be hdfs only?
> >
>
> Please ignore the above question (I just took a look at 0.20 repo).
> St.Ack
>



-- 
Connect to me at http://www.facebook.com/dhruba

--000e0cd70ecea7d4c3048636267b--

From general-return-1512-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 10 14:07:59 2010
Return-Path: <general-return-1512-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49651 invoked from network); 10 May 2010 14:07:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 May 2010 14:07:58 -0000
Received: (qmail 58364 invoked by uid 500); 10 May 2010 14:07:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58305 invoked by uid 500); 10 May 2010 14:07:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58297 invoked by uid 99); 10 May 2010 14:07:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 May 2010 14:07:56 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of corneliutudor.vlad@ens-lyon.fr designates 140.77.51.2 as permitted sender)
Received: from [140.77.51.2] (HELO jabiru.ens-lyon.fr) (140.77.51.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 May 2010 14:07:48 +0000
Received: from localhost (localhost [127.0.0.1])
	by jabiru.ens-lyon.fr (Postfix) with ESMTP id A27481EB1B2
	for <general@hadoop.apache.org>; Mon, 10 May 2010 16:07:26 +0200 (CEST)
X-Virus-Scanned: by amavisd-new-2.6.1 (20080629) (Debian) at ens-lyon.fr
Received: from jabiru.ens-lyon.fr ([127.0.0.1])
	by localhost (jabiru.ens-lyon.fr [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id pzowde50vedi for <general@hadoop.apache.org>;
	Mon, 10 May 2010 16:07:24 +0200 (CEST)
Received: from agrobate-domu-webmail.ens-lyon.fr (agrobate-domu-webmail.ens-lyon.fr [140.77.51.15])
	by jabiru.ens-lyon.fr (Postfix) with ESMTP id D39311EB1B3
	for <general@hadoop.apache.org>; Mon, 10 May 2010 16:07:24 +0200 (CEST)
Received: by agrobate-domu-webmail.ens-lyon.fr (Postfix, from userid 33)
	id D76DF58209; Mon, 10 May 2010 16:07:24 +0200 (CEST)
Received: from dhcp-13-143.lip.ens-lyon.fr (dhcp-13-143.lip.ens-lyon.fr
 [140.77.13.143]) by webmail.ens-lyon.fr (Horde Framework) with HTTP; Mon,
 10 May 2010 16:07:24 +0200
Message-ID: <20100510160724.137264ztyvvvcie4@webmail.ens-lyon.fr>
Date: Mon, 10 May 2010 16:07:24 +0200
From: Corneliu-Tudor Vlad <corneliutudor.vlad@ens-lyon.fr>
To: general@hadoop.apache.org
Subject: Problem with Hadoop Streaming and -D
 mapred.tasktracker.map.tasks.maximum option
MIME-Version: 1.0
Content-Type: text/plain;
 charset=ISO-8859-1;
 DelSp="Yes";
 format="flowed"
Content-Disposition: inline
Content-Transfer-Encoding: 7bit
User-Agent: Dynamic Internet Messaging Program (DIMP) H3 (1.1.4)


Hello

I am a new user of Hadoop and I have some trouble using Hadoop  
Streaming and the "-D mapred.tasktracker.map.tasks.maximum" option.

I'm experimenting with an unmanaged application (C++) which I want to  
run over several nodes in 2 scenarious
1) the number of maps (input splits) is equal to the number of nodes
2) the number of maps is a multiple of the number of nodes (5, 10, 20, ...

Initially, when running the tests in scenario 1 I would sometimes get  
2 process/node on half the nodes. However I fixed this by adding the  
directive -D mapred.tasktracker.map.tasks.maximum=1, so everything  
works fine.

In the case of scenario 2 (more maps than nodes) this directive no  
longer works, always obtaining 2 processes/node. I tested the even  
with putting maximum=5 and I still get 2 processes/node.

The entire command I use is:

/usr/bin/time --format="-duration:\t%e |\t-MFaults:\t%F  
|\t-ContxtSwitch:\t%w" \
  /opt/hadoop/bin/hadoop jar  
/opt/hadoop/contrib/streaming/hadoop-0.20.2-streaming.jar \
  -D mapred.tasktracker.map.tasks.maximum=1 \
  -D mapred.map.tasks=30 \
  -D mapred.reduce.tasks=0 \
  -D io.file.buffer.size=5242880 \
  -libjars "/opt/hadoop/contrib/streaming/hadoop-7debug.jar" \
  -input input/test553short \
  -output out1 \
  -mapper "/opt/jobdata/script_1k" \
  -inputformat "me.MyInputFormat"

I'm using is Debian Lenny x64, and Hadoop 0.20.2.

My question is: why is this happening and how can I make it work  
properly (i.e. be able to limit exactly how many mappers I can have at  
1 time per node)

Thank you in advance,
T


From general-return-1513-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 10 16:50:18 2010
Return-Path: <general-return-1513-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22545 invoked from network); 10 May 2010 16:50:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 May 2010 16:50:17 -0000
Received: (qmail 2644 invoked by uid 500); 10 May 2010 16:50:16 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2576 invoked by uid 500); 10 May 2010 16:50:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2568 invoked by uid 99); 10 May 2010 16:50:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 May 2010 16:50:16 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 May 2010 16:50:10 +0000
Received: by gyf1 with SMTP id 1so2655587gyf.35
        for <general@hadoop.apache.org>; Mon, 10 May 2010 09:49:49 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:content-type:content-transfer-encoding;
        bh=Aws75sL7oqXltToBWXs8ma785TXlD+2agQgndLjr9iU=;
        b=YwnLMgqKzxfYqpqeQPScJr1NO2Txs8yrjCBef+IKAYG7nuwAJ+hj4/iQAY/lZobzJR
         kNwe2W0KfgSSKj36jxzBrhkq5gqxN1zhKscRbcA+J4rHKYvx7/kGqydj4SrjT6fPAhaI
         q/svDFFtf/WME+yGzX4qb14+nHqor2H+zaS1U=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=CemoopX9m5YMiYRpHJdIfAj37jpERE2/sj5xoUg6vO4szRtctTjIPf8JYdFR5VmK9Z
         3TcnyVwwLKMiVT7rPABlbZ+GpvK+yfJKDg7OjHKLIUeOQTILXS2g/8xeakK3x1tKIbiX
         wSZnXIzWIYqEELHXffxuRreV6PgLzPKxpIysU=
MIME-Version: 1.0
Received: by 10.229.187.208 with SMTP id cx16mr3641776qcb.66.1273510189035; 
	Mon, 10 May 2010 09:49:49 -0700 (PDT)
Sender: saint.ack@gmail.com
Received: by 10.229.186.193 with HTTP; Mon, 10 May 2010 09:49:48 -0700 (PDT)
In-Reply-To: <k2i4aa34eb71005092150sdcc4ab8fnac9b152b2788ce38@mail.gmail.com>
References: <h2m4aa34eb71005071034ja97e2f4ckca2a016efedee897@mail.gmail.com>
	 <w2q7c962aed1005091515y4965fb9dpa345178a39fc9c45@mail.gmail.com>
	 <w2p7c962aed1005091526ne7a51896u7165b7019322a0a6@mail.gmail.com>
	 <k2i4aa34eb71005092150sdcc4ab8fnac9b152b2788ce38@mail.gmail.com>
Date: Mon, 10 May 2010 09:49:48 -0700
X-Google-Sender-Auth: rKqBTOXjKaD_dR_fATp7vDiE9rY
Message-ID: <AANLkTilcRuUbUzvGGe_lvuxauWCUJOwaM4U0pciKV_Ph@mail.gmail.com>
Subject: Re: Hadoop support for hbase
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

+1 on the below.
St.Ack

On Sun, May 9, 2010 at 9:50 PM, Dhruba Borthakur <dhruba@gmail.com> wrote:
> I will try to explain my opinion on some of the questions being raised so
> far. Of course all these are open for discussion and nothing is final at
> this moment.
>
> 1. This code base is primarily targeted for usage by HBase and Scribe. Bo=
th
> of these are Apache open source projects.
> 2. Accordingly, my proposal is to name the branch hadoop-0.20-append. Thi=
s
> name is more generic than the earlier one I suggested.
> 3. The proposal to pull patches into this branch will follow normal
> conventions that we follow in Hadoop. However, the release manager would
> have an option to veto a patch from being pulled into the branch.
> 4. if this type of branching causes too many forks of core Hadoop, then w=
e
> can aim to merge some of them after some time. The time duration depends =
on
> the stability of the code in that particular branch and is difficult to
> predict. The PMC, of course, has to approve of any new branch; so, in eff=
ect
> can prevent undesired multiple forks if that becomes a problem in the
> future.
> 5. code changes to this new branch will go through the normal process via
> JIRAs, code reviews, unit tests, etc.
> 6. the goal is to have a standard hadoop release from =A0this branch at s=
ome
> future point. if course, such a release has to be approved by the PMC. Th=
e
> release could be marked as "experimental" or some such thing if deemed
> appropriate by the PMC.
>
> thanks,
> dhruba
>
>
> On Sun, May 9, 2010 at 3:26 PM, Stack <stack@duboce.net> wrote:
>
>> On Sun, May 9, 2010 at 3:15 PM, Stack <stack@duboce.net> wrote:
>> > Should it be all of hadoop? =A0Could it be hdfs only?
>> >
>>
>> Please ignore the above question (I just took a look at 0.20 repo).
>> St.Ack
>>
>
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
>

From general-return-1514-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 10 16:59:21 2010
Return-Path: <general-return-1514-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24267 invoked from network); 10 May 2010 16:59:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 May 2010 16:59:20 -0000
Received: (qmail 16235 invoked by uid 500); 10 May 2010 16:59:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16194 invoked by uid 500); 10 May 2010 16:59:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16186 invoked by uid 99); 10 May 2010 16:59:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 May 2010 16:59:19 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 May 2010 16:59:13 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:Received:Received:From:To:Subject:
   Thread-Topic:Thread-Index:Date:Message-ID:References:
   In-Reply-To:Accept-Language:Content-Language:
   X-MS-Has-Attach:X-MS-TNEF-Correlator:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version:
   Return-Path:X-OriginalArrivalTime;
  b=GjFjGrUQa89cyrKZ8opbgoJzWoBQhOOJ5cymPKeNwFH9XyTnsaSoY+3p
   87tY1wObvGUhNEloMwQL4EjTQpwUIthGvlm4y+SNqyVhImutKEb3C9RMu
   EJou4DzRMSx4Vbd;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1273510753; x=1305046753;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20Hadoop=20support=20for=20hbase|Date:=20
   Mon,=2010=20May=202010=2016:58:50=20+0000|Message-ID:=20<
   EA6A5E07-30C9-4209-945D-413DC9F45090@linkedin.com>|To:=20
   "<general@hadoop.apache.org>"=20<general@hadoop.apache.or
   g>|MIME-Version:=201.0|Content-Transfer-Encoding:=20quote
   d-printable|Content-ID:=20<87719bea-0a40-4454-a67e-be69b2
   7e0bec>|In-Reply-To:=20<k2i4aa34eb71005092150sdcc4ab8fnac
   9b152b2788ce38@mail.gmail.com>|References:=20<h2m4aa34eb7
   1005071034ja97e2f4ckca2a016efedee897@mail.gmail.com>=0D
   =0A=20<w2q7c962aed1005091515y4965fb9dpa345178a39fc9c45@ma
   il.gmail.com>=0D=0A=20<w2p7c962aed1005091526ne7a51896u716
   5b7019322a0a6@mail.gmail.com>=0D=0A=20<k2i4aa34eb71005092
   150sdcc4ab8fnac9b152b2788ce38@mail.gmail.com>;
  bh=9tSnw2nKNtWTtAY65kJkiFmSjAiEq17AdpvH+G8Sjeo=;
  b=FVLp8C+Jrfo5vqcIa24qLYn7gHOuXNnXXxX6lqn1GDXTmDSqpRhpiuca
   iREdx6LtjoZ8xuDYxxLk2tyfQrICpb0KRkgX1/PBHY7brO7uvSLL/96FC
   UUNdjexWn17S/5i;
X-IronPort-AV: E=Sophos;i="4.52,363,1270450800"; 
   d="scan'208";a="12796618"
Received: from esv4-exctest.linkedin.biz ([172.18.46.60]) by CORP-MAIL.linkedin.biz with Microsoft SMTPSVC(6.0.3790.3959);
	 Mon, 10 May 2010 09:58:52 -0700
Received: from ESV4-CAS01.linkedin.biz (172.18.46.140) by
 esv4-exctest.linkedin.biz (172.18.46.60) with Microsoft SMTP Server (TLS) id
 14.0.682.1; Mon, 10 May 2010 09:58:51 -0700
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi; Mon, 10 May 2010
 09:58:51 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: Hadoop support for hbase
Thread-Topic: Hadoop support for hbase
Thread-Index: AQHK7gugwCsp8tpMpUe2iaMIChehi5JKI4EAgAADD4CAAGtggIAAy3EA
Date: Mon, 10 May 2010 16:58:50 +0000
Message-ID: <EA6A5E07-30C9-4209-945D-413DC9F45090@linkedin.com>
References: <h2m4aa34eb71005071034ja97e2f4ckca2a016efedee897@mail.gmail.com>
 <w2q7c962aed1005091515y4965fb9dpa345178a39fc9c45@mail.gmail.com>
 <w2p7c962aed1005091526ne7a51896u7165b7019322a0a6@mail.gmail.com>
 <k2i4aa34eb71005092150sdcc4ab8fnac9b152b2788ce38@mail.gmail.com>
In-Reply-To: <k2i4aa34eb71005092150sdcc4ab8fnac9b152b2788ce38@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Content-Type: text/plain; charset="us-ascii"
Content-ID: <87719bea-0a40-4454-a67e-be69b27e0bec>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginalArrivalTime: 10 May 2010 16:58:52.0413 (UTC) FILETIME=[1179B2D0:01CAF062]
X-Virus-Checked: Checked by ClamAV on apache.org


Let me understand this:

a) the hbase folks have been required to patch hadoop due to bugs
b) they have been doing this for X months now
c) we finally have momentum on getting 0.21 out the door
d) hey, let's make their life easier and take resources out of 0.21 by crea=
ting a branch

Are we willing to create a branch for everyone that asks? What happens if m=
ahout/lucene/solr/joes-random-application asks?  Is it a good idea to creat=
e a branch that hopefully has a very short life time?   Is it the latest fa=
shion amongst the PMC to request random branches? =20

trunk
0.21
0.20
0.20-append
0.20-security-sure-hope-this-turns-into-1.0
0.20-lifecycle-management-stuff-that-seems-to-never-get-committed
...

I'm not PMC, but can I make requests too, since I have to patch what we use=
?

0.20-capacity-scheduler-is-broken-but-we-shipped-it-anyway
0.20-stop-using-bsd-and-gnu-only-extensions-during-forks
0.20-hadoop-devs-cant-spell

I'd rather have the HBase folks put pressure on the Hadoop PMC to get 0.21 =
out the door than create a custom-to-be-thrown-away-in-less-than-a-year bra=
nch.  This way we *all* benefit rather than a select, but vocal group.=

From general-return-1515-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 10 17:18:54 2010
Return-Path: <general-return-1515-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 28266 invoked from network); 10 May 2010 17:18:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 May 2010 17:18:53 -0000
Received: (qmail 41160 invoked by uid 500); 10 May 2010 17:18:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 41121 invoked by uid 500); 10 May 2010 17:18:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 41113 invoked by uid 99); 10 May 2010 17:18:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 May 2010 17:18:52 +0000
X-ASF-Spam-Status: No, hits=-0.4 required=10.0
	tests=AWL,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 May 2010 17:18:48 +0000
Received: by vws12 with SMTP id 12so134174vws.35
        for <general@hadoop.apache.org>; Mon, 10 May 2010 10:18:27 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:sender:received
         :in-reply-to:references:date:x-google-sender-auth:message-id:subject
         :from:to:content-type:content-transfer-encoding;
        bh=dWqCKwCwLYrqa8RznWn4MHeycft6i7LcPc79IS9iJrM=;
        b=PRbErV0ixlN1ZRg/YUUgJlYItHGyhZcqt5zrwDSzf5IQPq7hwodE5LVQEUnF195fHu
         PKx5qzNMo5MhB+qiYCq3ahjmMW3BaooM20DMMo8U9eW1PLWr0SVsQ7yOUep2ekg3N1nk
         jMNg/wYMwKd+w9v48hWfqyPt6KrWUtO2hDC/w=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=RsTmwQIe3eiXA/MLYM3RRaI2aigDRGwWJ+unsXsgnXSpKFQSDNv/NojWnCbBGXMMLR
         11y+DTy9WKImQGheIKktjt2Xo7oFxPrpeKs1akGagtF+o+Z23nbuPPt3EuClpjjXJNTX
         c4JgVO7KhHk9brQ/dhUiFXg9FXwcfQereBAZY=
MIME-Version: 1.0
Received: by 10.229.237.142 with SMTP id ko14mr3738845qcb.11.1273511906896; 
	Mon, 10 May 2010 10:18:26 -0700 (PDT)
Sender: saint.ack@gmail.com
Received: by 10.229.186.193 with HTTP; Mon, 10 May 2010 10:18:26 -0700 (PDT)
In-Reply-To: <EA6A5E07-30C9-4209-945D-413DC9F45090@linkedin.com>
References: <h2m4aa34eb71005071034ja97e2f4ckca2a016efedee897@mail.gmail.com>
	 <w2q7c962aed1005091515y4965fb9dpa345178a39fc9c45@mail.gmail.com>
	 <w2p7c962aed1005091526ne7a51896u7165b7019322a0a6@mail.gmail.com>
	 <k2i4aa34eb71005092150sdcc4ab8fnac9b152b2788ce38@mail.gmail.com>
	 <EA6A5E07-30C9-4209-945D-413DC9F45090@linkedin.com>
Date: Mon, 10 May 2010 10:18:26 -0700
X-Google-Sender-Auth: MmTMxcw96oOXc7an_5W4DelnlYA
Message-ID: <AANLkTin5amvNCn9apT64DwJNAEmBZWm8v6k1X-6x9_qz@mail.gmail.com>
Subject: Re: Hadoop support for hbase
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Mon, May 10, 2010 at 9:58 AM, Allen Wittenauer
<awittenauer@linkedin.com> wrote:
>
> Let me understand this:
>
> a) the hbase folks have been required to patch hadoop due to bugs

The branch is to work on adding a feature to 0.20, not for fixing bugs.

> c) we finally have momentum on getting 0.21 out the door
> d) hey, let's make their life easier and take resources out of 0.21 by cr=
eating a branch
>

The above is a fallacious setup.  How does a branch in 0.20 detract
from the 0.21 momentum (The append feature that we'd work on in 0.20
branch has little relation to how append works in 0.21).


> Are we willing to create a branch for everyone that asks? What happens if=
 mahout/lucene/solr/joes-random-application asks? =A0Is it a good idea to c=
reate a branch that hopefully has a very short life time? =A0 Is it the lat=
est fashion amongst the PMC to request random branches?
>
> trunk
> 0.21
> 0.20
> 0.20-append
> 0.20-security-sure-hope-this-turns-into-1.0
> 0.20-lifecycle-management-stuff-that-seems-to-never-get-committed
> ...
>
> I'm not PMC, but can I make requests too, since I have to patch what we u=
se?
>

Of course you can.

> I'd rather have the HBase folks put pressure on the Hadoop PMC to get 0.2=
1 out the door than create a custom-to-be-thrown-away-in-less-than-a-year b=
ranch. =A0This way we *all* benefit rather than a select, but vocal group.

We're all behind 0.21.  Its the future.

St.Ack

From general-return-1516-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 10 17:25:03 2010
Return-Path: <general-return-1516-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29343 invoked from network); 10 May 2010 17:25:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 May 2010 17:25:03 -0000
Received: (qmail 47548 invoked by uid 500); 10 May 2010 17:25:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47507 invoked by uid 500); 10 May 2010 17:25:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47499 invoked by uid 99); 10 May 2010 17:25:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 May 2010 17:25:02 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 May 2010 17:24:56 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:Received:Received:From:To:Subject:
   Thread-Topic:Thread-Index:Date:Message-ID:References:
   In-Reply-To:Accept-Language:Content-Language:
   X-MS-Has-Attach:X-MS-TNEF-Correlator:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version:
   Return-Path:X-OriginalArrivalTime;
  b=G80RwZi35dPlgi67DxFpRq62ekFRKbWP/TkF8FgMbj0vwWwKEihIiVFh
   udxcs64RghsSORg1QkNoHuy25xjLSVNb4YwFxD6kJ/gsPTanFvnvSR18j
   0vfr8z+5WB5abtJ;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1273512296; x=1305048296;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20Hadoop=20support=20for=20hbase|Date:=20
   Mon,=2010=20May=202010=2017:24:32=20+0000|Message-ID:=20<
   7638AF7B-1C16-45C1-A9CE-025572A7E067@linkedin.com>|To:=20
   "<general@hadoop.apache.org>"=20<general@hadoop.apache.or
   g>|MIME-Version:=201.0|Content-Transfer-Encoding:=20quote
   d-printable|Content-ID:=20<70802053-c141-4ee0-a4d1-e89e8a
   0f8e32>|In-Reply-To:=20<AANLkTin5amvNCn9apT64DwJNAEmBZWm8
   v6k1X-6x9_qz@mail.gmail.com>|References:=20<h2m4aa34eb710
   05071034ja97e2f4ckca2a016efedee897@mail.gmail.com>=0D=0A
   =20<w2q7c962aed1005091515y4965fb9dpa345178a39fc9c45@mail.
   gmail.com>=0D=0A=20<w2p7c962aed1005091526ne7a51896u7165b7
   019322a0a6@mail.gmail.com>=0D=0A=20<k2i4aa34eb71005092150
   sdcc4ab8fnac9b152b2788ce38@mail.gmail.com>=0D=0A=20<EA6A5
   E07-30C9-4209-945D-413DC9F45090@linkedin.com>=0D=0A=20<AA
   NLkTin5amvNCn9apT64DwJNAEmBZWm8v6k1X-6x9_qz@mail.gmail.co
   m>;
  bh=zhREF71k/NUzgjEa467g/g0zbB0RZITHd7ORHO4+jlc=;
  b=OUeCBVWjD+yc2QyYiZJOMNv7+V5UgvXU9iHcruKDYagCqdtqyaBd5pmb
   4OdkQMsYd3d7LgSimePJr508Qa6Ng07TxA/Cn05yVUsPow8Zq6nRsCs9P
   MQlYUgeFmYI1MNi;
X-IronPort-AV: E=Sophos;i="4.52,363,1270450800"; 
   d="scan'208";a="12797401"
Received: from esv4-exctest.linkedin.biz ([172.18.46.60]) by CORP-MAIL.linkedin.biz with Microsoft SMTPSVC(6.0.3790.3959);
	 Mon, 10 May 2010 10:24:35 -0700
Received: from ESV4-CAS01.linkedin.biz (172.18.46.140) by
 esv4-exctest.linkedin.biz (172.18.46.60) with Microsoft SMTP Server (TLS) id
 14.0.682.1; Mon, 10 May 2010 10:24:34 -0700
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi; Mon, 10 May 2010
 10:24:34 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: Hadoop support for hbase
Thread-Topic: Hadoop support for hbase
Thread-Index: AQHK7gugwCsp8tpMpUe2iaMIChehi5JKI4EAgAADD4CAAGtggIAAy3EAgAAFegCAAAG0AA==
Date: Mon, 10 May 2010 17:24:32 +0000
Message-ID: <7638AF7B-1C16-45C1-A9CE-025572A7E067@linkedin.com>
References: <h2m4aa34eb71005071034ja97e2f4ckca2a016efedee897@mail.gmail.com>
 <w2q7c962aed1005091515y4965fb9dpa345178a39fc9c45@mail.gmail.com>
 <w2p7c962aed1005091526ne7a51896u7165b7019322a0a6@mail.gmail.com>
 <k2i4aa34eb71005092150sdcc4ab8fnac9b152b2788ce38@mail.gmail.com>
 <EA6A5E07-30C9-4209-945D-413DC9F45090@linkedin.com>
 <AANLkTin5amvNCn9apT64DwJNAEmBZWm8v6k1X-6x9_qz@mail.gmail.com>
In-Reply-To: <AANLkTin5amvNCn9apT64DwJNAEmBZWm8v6k1X-6x9_qz@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Content-Type: text/plain; charset="us-ascii"
Content-ID: <70802053-c141-4ee0-a4d1-e89e8a0f8e32>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginalArrivalTime: 10 May 2010 17:24:35.0000 (UTC) FILETIME=[A8EDCB80:01CAF065]
X-Virus-Checked: Checked by ClamAV on apache.org


On May 10, 2010, at 10:18 AM, Stack wrote:
> The above is a fallacious setup.  How does a branch in 0.20 detract
> from the 0.21 momentum (The append feature that we'd work on in 0.20
> branch has little relation to how append works in 0.21).

There are X amount of hours that people can put into Hadoop.  If Y amount g=
oes into 0.20-append, then 0.21 is now left with X-Y.   X-Y < X.

>> I'd rather have the HBase folks put pressure on the Hadoop PMC to get 0.=
21 out the door than create a custom-to-be-thrown-away-in-less-than-a-year =
branch.  This way we *all* benefit rather than a select, but vocal group.
>=20
> We're all behind 0.21.  Its the future.

Great.  Start using the time you'd put into 0.20-append into 0.21 and then =
we'll all be in a better place.




From general-return-1517-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 10 17:46:03 2010
Return-Path: <general-return-1517-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37564 invoked from network); 10 May 2010 17:46:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 May 2010 17:46:03 -0000
Received: (qmail 84424 invoked by uid 500); 10 May 2010 17:46:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 84381 invoked by uid 500); 10 May 2010 17:46:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 84373 invoked by uid 99); 10 May 2010 17:46:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 May 2010 17:46:01 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 May 2010 17:45:55 +0000
Received: by gwj20 with SMTP id 20so1097890gwj.35
        for <general@hadoop.apache.org>; Mon, 10 May 2010 10:45:34 -0700 (PDT)
Received: by 10.231.79.201 with SMTP id q9mr232377ibk.9.1273513534383; Mon, 10 
	May 2010 10:45:34 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.160.70 with HTTP; Mon, 10 May 2010 10:45:13 -0700 (PDT)
In-Reply-To: <AANLkTin5amvNCn9apT64DwJNAEmBZWm8v6k1X-6x9_qz@mail.gmail.com>
References: <h2m4aa34eb71005071034ja97e2f4ckca2a016efedee897@mail.gmail.com> 
	<w2q7c962aed1005091515y4965fb9dpa345178a39fc9c45@mail.gmail.com> 
	<w2p7c962aed1005091526ne7a51896u7165b7019322a0a6@mail.gmail.com> 
	<k2i4aa34eb71005092150sdcc4ab8fnac9b152b2788ce38@mail.gmail.com> 
	<EA6A5E07-30C9-4209-945D-413DC9F45090@linkedin.com> <AANLkTin5amvNCn9apT64DwJNAEmBZWm8v6k1X-6x9_qz@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 10 May 2010 10:45:13 -0700
Message-ID: <AANLkTil4M5IetREXm0Paj13gxC4zS6U_mXbxHNf_QjJF@mail.gmail.com>
Subject: Re: Hadoop support for hbase
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Mon, May 10, 2010 at 10:18 AM, Stack <stack@duboce.net> wrote:
> On Mon, May 10, 2010 at 9:58 AM, Allen Wittenauer
> <awittenauer@linkedin.com> wrote:
>>
>> Let me understand this:
>>
>> a) the hbase folks have been required to patch hadoop due to bugs
>
> The branch is to work on adding a feature to 0.20, not for fixing bugs.
>
>> c) we finally have momentum on getting 0.21 out the door
>> d) hey, let's make their life easier and take resources out of 0.21 by c=
reating a branch
>>
>
> The above is a fallacious setup. =A0How does a branch in 0.20 detract
> from the 0.21 momentum (The append feature that we'd work on in 0.20
> branch has little relation to how append works in 0.21).

For what it's worth, though, the majority of the size of the 0.20
append patch is made up of additional unit tests. I have started
forward-porting these new tests to the trunk append and it's already
exposed a number of bugs. So while it's tempting to say that the 0.20
append is "wasted effort", it really is benefiting the entire
community and the 0.21 release as well.

-Todd

>
>
>> Are we willing to create a branch for everyone that asks? What happens i=
f mahout/lucene/solr/joes-random-application asks? =A0Is it a good idea to =
create a branch that hopefully has a very short life time? =A0 Is it the la=
test fashion amongst the PMC to request random branches?
>>
>> trunk
>> 0.21
>> 0.20
>> 0.20-append
>> 0.20-security-sure-hope-this-turns-into-1.0
>> 0.20-lifecycle-management-stuff-that-seems-to-never-get-committed
>> ...
>>
>> I'm not PMC, but can I make requests too, since I have to patch what we =
use?
>>
>
> Of course you can.
>
>> I'd rather have the HBase folks put pressure on the Hadoop PMC to get 0.=
21 out the door than create a custom-to-be-thrown-away-in-less-than-a-year =
branch. =A0This way we *all* benefit rather than a select, but vocal group.
>
> We're all behind 0.21. =A0Its the future.
>
> St.Ack
>



--=20
Todd Lipcon
Software Engineer, Cloudera

From general-return-1518-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 10 18:05:53 2010
Return-Path: <general-return-1518-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44107 invoked from network); 10 May 2010 18:05:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 May 2010 18:05:52 -0000
Received: (qmail 8845 invoked by uid 500); 10 May 2010 18:05:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8802 invoked by uid 500); 10 May 2010 18:05:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8794 invoked by uid 99); 10 May 2010 18:05:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 May 2010 18:05:51 +0000
X-ASF-Spam-Status: No, hits=-0.4 required=10.0
	tests=AWL,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ryanobjc@gmail.com designates 209.85.212.176 as permitted sender)
Received: from [209.85.212.176] (HELO mail-px0-f176.google.com) (209.85.212.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 May 2010 18:05:47 +0000
Received: by pxi10 with SMTP id 10so1898234pxi.35
        for <general@hadoop.apache.org>; Mon, 10 May 2010 11:05:24 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=xbgw2d34J+ek4e4DtN8DlVOTH5I3LDn2KJ7Rjhkrgkg=;
        b=utagEOIsepJtROAiowhqFlOalRDm9g9q7SeEdGOzlmxZongGLHFpPqCWFLsZUEassS
         P+EcVLai0bHtHQVk2+vLYjbY4mOyEQQZ7KE40zo0FtTuQwkbTjwEJf1h8WbD3mU1Q2X6
         o7xkvisXmq4aUB0gMURMNXU7vgVYHVgyg6qcc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=K3lsc9BvXVzSp4gS4olkYBwY0zPBH5ydyuNuenY/OYv9KJX9OHL9KFHlEX6NrpNWev
         9G1CaMQT4qOF4PQfgHsZjJrRkxKdf4MyqJyR2zImV3QiqjS0K5//GE9Ip/k8X/T2sgph
         6YNsTMapuQefB9XcXTlOlC0238yBcGH5QsF1E=
MIME-Version: 1.0
Received: by 10.140.57.8 with SMTP id f8mr2936381rva.102.1273514724003; Mon, 
	10 May 2010 11:05:24 -0700 (PDT)
Received: by 10.140.147.19 with HTTP; Mon, 10 May 2010 11:05:23 -0700 (PDT)
In-Reply-To: <7638AF7B-1C16-45C1-A9CE-025572A7E067@linkedin.com>
References: <h2m4aa34eb71005071034ja97e2f4ckca2a016efedee897@mail.gmail.com>
	 <w2q7c962aed1005091515y4965fb9dpa345178a39fc9c45@mail.gmail.com>
	 <w2p7c962aed1005091526ne7a51896u7165b7019322a0a6@mail.gmail.com>
	 <k2i4aa34eb71005092150sdcc4ab8fnac9b152b2788ce38@mail.gmail.com>
	 <EA6A5E07-30C9-4209-945D-413DC9F45090@linkedin.com>
	 <AANLkTin5amvNCn9apT64DwJNAEmBZWm8v6k1X-6x9_qz@mail.gmail.com>
	 <7638AF7B-1C16-45C1-A9CE-025572A7E067@linkedin.com>
Date: Mon, 10 May 2010 11:05:23 -0700
Message-ID: <i2x78568af11005101105l33eccf67p613d62a45805234b@mail.gmail.com>
Subject: Re: Hadoop support for hbase
From: Ryan Rawson <ryanobjc@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Mon, May 10, 2010 at 10:24 AM, Allen Wittenauer
<awittenauer@linkedin.com> wrote:
>
> On May 10, 2010, at 10:18 AM, Stack wrote:
>> The above is a fallacious setup. =A0How does a branch in 0.20 detract
>> from the 0.21 momentum (The append feature that we'd work on in 0.20
>> branch has little relation to how append works in 0.21).
>
> There are X amount of hours that people can put into Hadoop. =A0If Y amou=
nt goes into 0.20-append, then 0.21 is now left with X-Y. =A0 X-Y < X.
>
>>> I'd rather have the HBase folks put pressure on the Hadoop PMC to get 0=
.21 out the door than create a custom-to-be-thrown-away-in-less-than-a-year=
 branch. =A0This way we *all* benefit rather than a select, but vocal group=
.
>>
>> We're all behind 0.21. =A0Its the future.
>
> Great. =A0Start using the time you'd put into 0.20-append into 0.21 and t=
hen we'll all be in a better place.

That's not how it works though - people have adopted and use Hadoop
0.20 because of the fact that people like Yahoo, Facebook, etc run it
on multi-thousand node clusters and have done so for months (or soon
to be years now).

From general-return-1519-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 10 18:14:38 2010
Return-Path: <general-return-1519-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45857 invoked from network); 10 May 2010 18:14:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 May 2010 18:14:38 -0000
Received: (qmail 18738 invoked by uid 500); 10 May 2010 18:14:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18698 invoked by uid 500); 10 May 2010 18:14:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18690 invoked by uid 99); 10 May 2010 18:14:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 May 2010 18:14:37 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 May 2010 18:14:31 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:Received:Received:From:To:Subject:
   Thread-Topic:Thread-Index:Date:Message-ID:References:
   In-Reply-To:Accept-Language:Content-Language:
   X-MS-Has-Attach:X-MS-TNEF-Correlator:Content-Type:
   Content-ID:Content-Transfer-Encoding:MIME-Version:
   Return-Path:X-OriginalArrivalTime;
  b=UA57iSNejdmfOz+eAH1i4a32enh1HpBEcQDyGsp0PPGsRKtD6ODnwBzM
   a/x3aaKJ4NpD2tF6RT7eFT9b5mhuisUCsl5QRAKkX25gxrstpRYzuGC7H
   sdbGJn4ruAI0JJ/;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1273515271; x=1305051271;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20Hadoop=20support=20for=20hbase|Date:=20
   Mon,=2010=20May=202010=2018:14:07=20+0000|Message-ID:=20<
   5DED5EE4-6B01-4798-8F78-16F54E2AAD51@linkedin.com>|To:=20
   "<general@hadoop.apache.org>"=20<general@hadoop.apache.or
   g>|MIME-Version:=201.0|Content-Transfer-Encoding:=20quote
   d-printable|Content-ID:=20<5715a309-14d5-42da-b317-e598b3
   5c11ab>|In-Reply-To:=20<i2x78568af11005101105l33eccf67p61
   3d62a45805234b@mail.gmail.com>|References:=20<h2m4aa34eb7
   1005071034ja97e2f4ckca2a016efedee897@mail.gmail.com>=0D
   =0A=20<w2q7c962aed1005091515y4965fb9dpa345178a39fc9c45@ma
   il.gmail.com>=0D=0A=20<w2p7c962aed1005091526ne7a51896u716
   5b7019322a0a6@mail.gmail.com>=0D=0A=20<k2i4aa34eb71005092
   150sdcc4ab8fnac9b152b2788ce38@mail.gmail.com>=0D=0A=20<EA
   6A5E07-30C9-4209-945D-413DC9F45090@linkedin.com>=0D=0A=20
   <AANLkTin5amvNCn9apT64DwJNAEmBZWm8v6k1X-6x9_qz@mail.gmail
   .com>=0D=0A=20<7638AF7B-1C16-45C1-A9CE-025572A7E067@linke
   din.com>=0D=0A=20<i2x78568af11005101105l33eccf67p613d62a4
   5805234b@mail.gmail.com>;
  bh=fWDi7+QThzQ6wTPhWAAXQSLzHH1lSYHvd7X2wUhQzTI=;
  b=dDe3c83RNpoKH4+ignbLHGL6+hosZ5H/o1slmWKRKrw14hpIS10HxfHz
   +dUMZZrbfoXYg/385lRMnmiFFV3OJzx5M12yE9WI4pmroCMIFcvHnFHUL
   +ayTXyVbT8Y2Bmj;
X-IronPort-AV: E=Sophos;i="4.52,363,1270450800"; 
   d="scan'208";a="12798792"
Received: from esv4-exctest.linkedin.biz ([172.18.46.60]) by CORP-MAIL.linkedin.biz with Microsoft SMTPSVC(6.0.3790.3959);
	 Mon, 10 May 2010 11:14:09 -0700
Received: from ESV4-CAS02.linkedin.biz (172.18.46.142) by
 esv4-exctest.linkedin.biz (172.18.46.60) with Microsoft SMTP Server (TLS) id
 14.0.682.1; Mon, 10 May 2010 11:14:09 -0700
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi; Mon, 10 May 2010
 11:14:09 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: Hadoop support for hbase
Thread-Topic: Hadoop support for hbase
Thread-Index: AQHK7gugwCsp8tpMpUe2iaMIChehi5JKI4EAgAADD4CAAGtggIAAy3EAgAAFegCAAAG0AIAAC2qAgAACcYA=
Date: Mon, 10 May 2010 18:14:07 +0000
Message-ID: <5DED5EE4-6B01-4798-8F78-16F54E2AAD51@linkedin.com>
References: <h2m4aa34eb71005071034ja97e2f4ckca2a016efedee897@mail.gmail.com>
 <w2q7c962aed1005091515y4965fb9dpa345178a39fc9c45@mail.gmail.com>
 <w2p7c962aed1005091526ne7a51896u7165b7019322a0a6@mail.gmail.com>
 <k2i4aa34eb71005092150sdcc4ab8fnac9b152b2788ce38@mail.gmail.com>
 <EA6A5E07-30C9-4209-945D-413DC9F45090@linkedin.com>
 <AANLkTin5amvNCn9apT64DwJNAEmBZWm8v6k1X-6x9_qz@mail.gmail.com>
 <7638AF7B-1C16-45C1-A9CE-025572A7E067@linkedin.com>
 <i2x78568af11005101105l33eccf67p613d62a45805234b@mail.gmail.com>
In-Reply-To: <i2x78568af11005101105l33eccf67p613d62a45805234b@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Content-Type: text/plain; charset="us-ascii"
Content-ID: <5715a309-14d5-42da-b317-e598b35c11ab>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginalArrivalTime: 10 May 2010 18:14:09.0995 (UTC) FILETIME=[962A01B0:01CAF06C]
X-Virus-Checked: Checked by ClamAV on apache.org


On May 10, 2010, at 11:05 AM, Ryan Rawson wrote:
> That's not how it works though - people have adopted and use Hadoop
> 0.20 because of the fact that people like Yahoo, Facebook, etc run it
> on multi-thousand node clusters and have done so for months (or soon
> to be years now).


If you look through Yahoo!'s changelog, it is hard to call that 0.20.

From general-return-1520-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 10 20:21:57 2010
Return-Path: <general-return-1520-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82555 invoked from network); 10 May 2010 20:21:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 May 2010 20:21:56 -0000
Received: (qmail 77329 invoked by uid 500); 10 May 2010 20:21:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77281 invoked by uid 500); 10 May 2010 20:21:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77273 invoked by uid 99); 10 May 2010 20:21:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 May 2010 20:21:55 +0000
X-ASF-Spam-Status: No, hits=2.3 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of michael_segel@hotmail.com designates 65.55.34.19 as permitted sender)
Received: from [65.55.34.19] (HELO col0-omc1-s9.col0.hotmail.com) (65.55.34.19)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 May 2010 20:21:49 +0000
Received: from COL117-W64 ([65.55.34.9]) by col0-omc1-s9.col0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);
	 Mon, 10 May 2010 13:21:28 -0700
Message-ID: <COL117-W64E02417250AA7DC1C1E828FF90@phx.gbl>
Content-Type: multipart/alternative;
	boundary="_cae9d3bd-57c3-4ac6-b7c8-c3470d634f8e_"
X-Originating-IP: [65.167.11.254]
From: Michael Segel <michael_segel@hotmail.com>
To: <general@hadoop.apache.org>
Subject: RE: Hadoop support for hbase
Date: Mon, 10 May 2010 15:21:28 -0500
Importance: Normal
In-Reply-To: <AANLkTil4M5IetREXm0Paj13gxC4zS6U_mXbxHNf_QjJF@mail.gmail.com>
References:
 <h2m4aa34eb71005071034ja97e2f4ckca2a016efedee897@mail.gmail.com>,<w2q7c962aed1005091515y4965fb9dpa345178a39fc9c45@mail.gmail.com>,<w2p7c962aed1005091526ne7a51896u7165b7019322a0a6@mail.gmail.com>,<k2i4aa34eb71005092150sdcc4ab8fnac9b152b2788ce38@mail.gmail.com>,<EA6A5E07-30C9-4209-945D-413DC9F45090@linkedin.com>
 <AANLkTin5amvNCn9apT64DwJNAEmBZWm8v6k1X-6x9_qz@mail.gmail.com>,<AANLkTil4M5IetREXm0Paj13gxC4zS6U_mXbxHNf_QjJF@mail.gmail.com>
MIME-Version: 1.0
X-OriginalArrivalTime: 10 May 2010 20:21:28.0491 (UTC) FILETIME=[5F0FFFB0:01CAF07E]

--_cae9d3bd-57c3-4ac6-b7c8-c3470d634f8e_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable




> From: todd@cloudera.com
> Date: Mon=2C 10 May 2010 10:45:13 -0700
> Subject: Re: Hadoop support for hbase
> To: general@hadoop.apache.org
>=20

> > The above is a fallacious setup.  How does a branch in 0.20 detract
> > from the 0.21 momentum (The append feature that we'd work on in 0.20
> > branch has little relation to how append works in 0.21).
>=20
> For what it's worth=2C though=2C the majority of the size of the 0.20
> append patch is made up of additional unit tests. I have started
> forward-porting these new tests to the trunk append and it's already
> exposed a number of bugs. So while it's tempting to say that the 0.20
> append is "wasted effort"=2C it really is benefiting the entire
> community and the 0.21 release as well.
>=20
> -Todd
>=20

Sometimes you have to slow down to go faster.


 		 	   		 =20
_________________________________________________________________
Hotmail is redefining busy with tools for the New Busy. Get more from your =
inbox.
http://www.windowslive.com/campaign/thenewbusy?ocid=3DPID28326::T:WLMTAGL:O=
N:WL:en-US:WM_HMP:042010_2=

--_cae9d3bd-57c3-4ac6-b7c8-c3470d634f8e_--

From general-return-1521-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 10 21:06:36 2010
Return-Path: <general-return-1521-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12211 invoked from network); 10 May 2010 21:06:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 May 2010 21:06:36 -0000
Received: (qmail 33145 invoked by uid 500); 10 May 2010 21:06:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33103 invoked by uid 500); 10 May 2010 21:06:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33095 invoked by uid 99); 10 May 2010 21:06:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 May 2010 21:06:34 +0000
X-ASF-Spam-Status: No, hits=0.6 required=10.0
	tests=AWL,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jaybooth@gmail.com designates 209.85.217.225 as permitted sender)
Received: from [209.85.217.225] (HELO mail-gx0-f225.google.com) (209.85.217.225)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 May 2010 21:06:30 +0000
Received: by gxk25 with SMTP id 25so1636133gxk.11
        for <general@hadoop.apache.org>; Mon, 10 May 2010 14:06:09 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=ZQdwlNcrVL6kWMRgdOrPJTx1JEWoRdCn+WVVS7PsEyY=;
        b=agJ9jqGPuq9wChZtCwtD6S6vtYqT3/tfqX/TE/vvDqyYTxhfL7AlHv2T9pD9xuoWp5
         CcTXdt7dIsfKxO9M3im/oM+s28b7BpooIHW4NFDQ31rJ4eM+TIH61wCFb54JFu6BRCoX
         mDbEMzPXMvn571tZ7AezK5qvTfQKKV5M2KjpQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=l5SO6zJLaC3wQqXDnv+OWTssrM6NNnsxozKbxdpuWDdWdB+7khywshcj0I6uUo5Mgw
         ysbpftqawnSWwYL4SLBo+h3kJJsgcQlR7T+TktbefV9daFGJRuSgFOwss2AW5qMsdgHO
         tjB6yGGAJ5gjyuBObomkrj3yYEQYXGZzyITtc=
MIME-Version: 1.0
Received: by 10.100.245.40 with SMTP id s40mr735211anh.137.1273525569422; Mon, 
	10 May 2010 14:06:09 -0700 (PDT)
Received: by 10.100.141.16 with HTTP; Mon, 10 May 2010 14:06:09 -0700 (PDT)
In-Reply-To: <COL117-W64E02417250AA7DC1C1E828FF90@phx.gbl>
References: <h2m4aa34eb71005071034ja97e2f4ckca2a016efedee897@mail.gmail.com>
	 <w2q7c962aed1005091515y4965fb9dpa345178a39fc9c45@mail.gmail.com>
	 <w2p7c962aed1005091526ne7a51896u7165b7019322a0a6@mail.gmail.com>
	 <k2i4aa34eb71005092150sdcc4ab8fnac9b152b2788ce38@mail.gmail.com>
	 <EA6A5E07-30C9-4209-945D-413DC9F45090@linkedin.com>
	 <AANLkTin5amvNCn9apT64DwJNAEmBZWm8v6k1X-6x9_qz@mail.gmail.com>
	 <AANLkTil4M5IetREXm0Paj13gxC4zS6U_mXbxHNf_QjJF@mail.gmail.com>
	 <COL117-W64E02417250AA7DC1C1E828FF90@phx.gbl>
Date: Mon, 10 May 2010 17:06:09 -0400
Message-ID: <AANLkTimgAcxJr8xOkx_Na6vtOvb7APW1tGPTKNzStEX2@mail.gmail.com>
Subject: Re: Hadoop support for hbase
From: Jay Booth <jaybooth@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Given that the 0.20-append branch pretty much already exists
unofficially, via IRC, IM and email forwarded patchsets, it seems like
giving it an official home is just recognizing the status quo.
Especially since 0.21 probably won't be getting rolled out into
production everywhere the first day it's officially released.  If the
work's going on anyways, I don't see how giving people a shared home
hurts matters, if anything it gives them a better shared touchpoint
for forward-porting bugfixes to 0.21.

A case could be made that by making it more painful to run
0.20-append, more momentum is created towards 0.21 but since Tom is
already on top of 21 and seemingly doing an excellent job, and since
the HBase community will probably be some of the first people to move
to 0.21 anyways, I don't see why having 0.20-append will damage 0.21's
momentum at this point.



On Mon, May 10, 2010 at 4:21 PM, Michael Segel
<michael_segel@hotmail.com> wrote:
>
>
>
>> From: todd@cloudera.com
>> Date: Mon, 10 May 2010 10:45:13 -0700
>> Subject: Re: Hadoop support for hbase
>> To: general@hadoop.apache.org
>>
>
>> > The above is a fallacious setup. =A0How does a branch in 0.20 detract
>> > from the 0.21 momentum (The append feature that we'd work on in 0.20
>> > branch has little relation to how append works in 0.21).
>>
>> For what it's worth, though, the majority of the size of the 0.20
>> append patch is made up of additional unit tests. I have started
>> forward-porting these new tests to the trunk append and it's already
>> exposed a number of bugs. So while it's tempting to say that the 0.20
>> append is "wasted effort", it really is benefiting the entire
>> community and the 0.21 release as well.
>>
>> -Todd
>>
>
> Sometimes you have to slow down to go faster.
>
>
>
> _________________________________________________________________
> Hotmail is redefining busy with tools for the New Busy. Get more from you=
r inbox.
> http://www.windowslive.com/campaign/thenewbusy?ocid=3DPID28326::T:WLMTAGL=
:ON:WL:en-US:WM_HMP:042010_2

From general-return-1522-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 11 06:35:40 2010
Return-Path: <general-return-1522-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33775 invoked from network); 11 May 2010 06:35:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 May 2010 06:35:40 -0000
Received: (qmail 52668 invoked by uid 500); 11 May 2010 06:35:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52397 invoked by uid 500); 11 May 2010 06:35:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52389 invoked by uid 99); 11 May 2010 06:35:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 May 2010 06:35:38 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of dhruba@gmail.com designates 209.85.217.216 as permitted sender)
Received: from [209.85.217.216] (HELO mail-gx0-f216.google.com) (209.85.217.216)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 May 2010 06:35:32 +0000
Received: by gxk8 with SMTP id 8so588966gxk.9
        for <general@hadoop.apache.org>; Mon, 10 May 2010 23:35:10 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=E9R8o5NZ/mYDPeqiSasFANDHmCjQb9wHsEHHsANEDRE=;
        b=uuBnfJXZwSnAeiIhcs1aKykCGj2FyvMUT8iF8OAzEKVsVCDInjDZbTA8W5JnpafbWz
         4ZwjykOR9QAzxVfYxmO2FmzkcBPMlSI+jUR3JKYNGXD0elz04VPpP+9fbI7wwxrBTxcX
         bJBDI4nBbKL8mJz3XPIQKUEBum/NUqqUm83XA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=TqIYoyv5eefbRdjDCqWTssCPKeYSf/aWZczsgsaDF81o6qukQ1a+jhySViQ64ng4J2
         hH/ue+4FoISShdConkOmuoMfOLu3lpMWVAvqp6/uKGW334S/ef85R3dITHLNM8fJbypW
         xT7NDZO4lbZU1tI/Jw1tfA4YPJA4GlFnlgOFo=
MIME-Version: 1.0
Received: by 10.150.243.17 with SMTP id q17mr11630662ybh.103.1273559710659; 
	Mon, 10 May 2010 23:35:10 -0700 (PDT)
Received: by 10.151.51.3 with HTTP; Mon, 10 May 2010 23:35:10 -0700 (PDT)
In-Reply-To: <AANLkTimgAcxJr8xOkx_Na6vtOvb7APW1tGPTKNzStEX2@mail.gmail.com>
References: <h2m4aa34eb71005071034ja97e2f4ckca2a016efedee897@mail.gmail.com>
	 <w2q7c962aed1005091515y4965fb9dpa345178a39fc9c45@mail.gmail.com>
	 <w2p7c962aed1005091526ne7a51896u7165b7019322a0a6@mail.gmail.com>
	 <k2i4aa34eb71005092150sdcc4ab8fnac9b152b2788ce38@mail.gmail.com>
	 <EA6A5E07-30C9-4209-945D-413DC9F45090@linkedin.com>
	 <AANLkTin5amvNCn9apT64DwJNAEmBZWm8v6k1X-6x9_qz@mail.gmail.com>
	 <AANLkTil4M5IetREXm0Paj13gxC4zS6U_mXbxHNf_QjJF@mail.gmail.com>
	 <COL117-W64E02417250AA7DC1C1E828FF90@phx.gbl>
	 <AANLkTimgAcxJr8xOkx_Na6vtOvb7APW1tGPTKNzStEX2@mail.gmail.com>
Date: Mon, 10 May 2010 23:35:10 -0700
Message-ID: <AANLkTinB9Sp7FAQqpD5WBu206G3W-KVQ7KP3gAbf7Kk4@mail.gmail.com>
Subject: Re: Hadoop support for hbase
From: Dhruba Borthakur <dhruba@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd298602774d104864bbad9
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd298602774d104864bbad9
Content-Type: text/plain; charset=ISO-8859-1

@Allen: we are definitely behind 0.21 release. Tom White is guiding that
release and most developers are committed to removing blockers for that
release. Todd rightly mentions that the work being done for 0.20 benefits
0.21 as well.

@Jay: Thanks for summing it up so well. I completely agree with your
viewpoint.

thanks
dhruba


On Mon, May 10, 2010 at 2:06 PM, Jay Booth <jaybooth@gmail.com> wrote:

> Given that the 0.20-append branch pretty much already exists
> unofficially, via IRC, IM and email forwarded patchsets, it seems like
> giving it an official home is just recognizing the status quo.
> Especially since 0.21 probably won't be getting rolled out into
> production everywhere the first day it's officially released.  If the
> work's going on anyways, I don't see how giving people a shared home
> hurts matters, if anything it gives them a better shared touchpoint
> for forward-porting bugfixes to 0.21.
>
> A case could be made that by making it more painful to run
> 0.20-append, more momentum is created towards 0.21 but since Tom is
> already on top of 21 and seemingly doing an excellent job, and since
> the HBase community will probably be some of the first people to move
> to 0.21 anyways, I don't see why having 0.20-append will damage 0.21's
> momentum at this point.
>
>
>
> On Mon, May 10, 2010 at 4:21 PM, Michael Segel
> <michael_segel@hotmail.com> wrote:
> >
> >
> >
> >> From: todd@cloudera.com
> >> Date: Mon, 10 May 2010 10:45:13 -0700
> >> Subject: Re: Hadoop support for hbase
> >> To: general@hadoop.apache.org
> >>
> >
> >> > The above is a fallacious setup.  How does a branch in 0.20 detract
> >> > from the 0.21 momentum (The append feature that we'd work on in 0.20
> >> > branch has little relation to how append works in 0.21).
> >>
> >> For what it's worth, though, the majority of the size of the 0.20
> >> append patch is made up of additional unit tests. I have started
> >> forward-porting these new tests to the trunk append and it's already
> >> exposed a number of bugs. So while it's tempting to say that the 0.20
> >> append is "wasted effort", it really is benefiting the entire
> >> community and the 0.21 release as well.
> >>
> >> -Todd
> >>
> >
> > Sometimes you have to slow down to go faster.
> >
> >
> >
> > _________________________________________________________________
> > Hotmail is redefining busy with tools for the New Busy. Get more from
> your inbox.
> >
> http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_2
>



-- 
Connect to me at http://www.facebook.com/dhruba

--000e0cd298602774d104864bbad9--

From general-return-1523-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 11 13:38:50 2010
Return-Path: <general-return-1523-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 20823 invoked from network); 11 May 2010 13:38:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 May 2010 13:38:50 -0000
Received: (qmail 80198 invoked by uid 500); 11 May 2010 13:38:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79793 invoked by uid 500); 11 May 2010 13:38:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 79785 invoked by uid 99); 11 May 2010 13:38:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 May 2010 13:38:47 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of renatoj.marroquin@gmail.com designates 209.85.210.172 as permitted sender)
Received: from [209.85.210.172] (HELO mail-yx0-f172.google.com) (209.85.210.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 May 2010 13:38:40 +0000
Received: by yxe2 with SMTP id 2so2754529yxe.2
        for <general@hadoop.apache.org>; Tue, 11 May 2010 06:38:19 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=wF/zkjbgY+ESwNvfH+faradyAglz34t54TYam/8rDec=;
        b=IUvi9l0MjSkPmM47GJimyA7doZ9GgjuZEiFqzvIKKN9DKKNB30DYegeXwHdEdxKb9x
         9EP3Tnf9e1/Aw6Ei21e4/cPLT3eeo7Ovgk8zp5zkUX1kd3Raj1/4BjreDe5L8e4OrzrL
         i6JPHLNresWb/uwyty+hZpeQfldtUTyI9u72Y=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=wksEz6OCpgZfio6M+oa2bp93KPrz9FWxWKxi4HhxLoSHxLsoJPEPvkzJSvy7LashWS
         VwjE8Gs5kve5nqWbjkvqa3a7o9StEnHYjdYvp+n9AWPJGkuuCz8g2U+LR2UC/m7v6bu4
         Sshaq6zLuwerVTtyXrdHykzIGPAmCla+D5OmM=
MIME-Version: 1.0
Received: by 10.101.147.32 with SMTP id z32mr2137078ann.227.1273585098892; 
	Tue, 11 May 2010 06:38:18 -0700 (PDT)
Received: by 10.100.231.13 with HTTP; Tue, 11 May 2010 06:38:18 -0700 (PDT)
In-Reply-To: <j2hd6d7c4411005051804p40b2b598ud593f984c09d779@mail.gmail.com>
References: <y2p44be54d01005050829gf270a41du98eaf93b0fe2f070@mail.gmail.com>
	 <j2hd6d7c4411005051804p40b2b598ud593f984c09d779@mail.gmail.com>
Date: Tue, 11 May 2010 10:38:18 -0300
Message-ID: <AANLkTinYM84xSEs30UT5RU1K7UjguNuztpy-yQjr2pXZ@mail.gmail.com>
Subject: Re: Hadoop Data Sharing
From: =?ISO-8859-1?Q?Renato_Marroqu=EDn_Mogrovejo?= <renatoj.marroquin@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e68deb376924bd048651a3e9
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e68deb376924bd048651a3e9
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Thanks Aaron! I was thinking the same after doing some reading.
Man what about serialize the objects? Would you think that is a good idea?
Thanks again.

Renato M.


2010/5/5 Aaron Kimball <aaron@cloudera.com>

> Renato,
>
> In general if you need to perform a multi-pass MapReduce workflow, each
> pass
> materializes its output to files. The subsequent pass then reads those sa=
me
> files back in as input. This allows the workflow to start at the last
> "checkpoint" if it gets interrupted. There is no persistent in-memory
> distributed storage feature in Hadoop that would allow a MapReduce job to
> post results to memory for consumption by a subsequent job.
>
> So you would just read your initial data from /input, and write your
> interim
> results to /iteration0. Then the next pass reads from /iteration0 and
> writes
> to /iteration1, etc..
>
> If your data is reasonably small and you think it could fit in memory
> somewhere, then you could experiment with using other distributed key-val=
ue
> stores (memcached[b], hbase, cassandra, etc..) to hold intermediate
> results.
> But this will require some integration work on your part.
> - Aaron
>
> On Wed, May 5, 2010 at 8:29 AM, Renato Marroqu=EDn Mogrovejo <
> renatoj.marroquin@gmail.com> wrote:
>
> > Hi everyone, I have recently started to play around with hadoop, but I =
am
> > getting some into some "design" problems.
> > I need to make a loop to execute the same job several times, and in eac=
h
> > iteration get the processed values (not using a file because I would ne=
ed
> > to
> > read it). I was using an static vector in my main class (the one that
> > iterates and executes the job in each iteration) to retrieve those
> values,
> > and it did work while I was using a standalone mode. Now I tried to tes=
t
> it
> > on a pseudo-distributed manner and obviously is not working.
> > Any suggestions, please???
> >
> > Thanks in advance,
> >
> >
> > Renato M.
> >
>

--0016e68deb376924bd048651a3e9--

From general-return-1524-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 11 13:59:39 2010
Return-Path: <general-return-1524-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24083 invoked from network); 11 May 2010 13:59:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 May 2010 13:59:39 -0000
Received: (qmail 7649 invoked by uid 500); 11 May 2010 13:59:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7586 invoked by uid 500); 11 May 2010 13:59:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7578 invoked by uid 99); 11 May 2010 13:59:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 May 2010 13:59:38 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [202.144.10.51] (HELO orcon.sify.net) (202.144.10.51)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 May 2010 13:59:29 +0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by orcon.sify.net (Postfix) with ESMTP id EA33632A2B7
	for <general@hadoop.apache.org>; Tue, 11 May 2010 19:29:06 +0530 (IST)
X-Virus-Scanned: amavisd-new at example.com
Received: from orcon.sify.net ([127.0.0.1])
	by localhost (orcon.sify.net [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id qVhfru64CV9e for <general@hadoop.apache.org>;
	Tue, 11 May 2010 19:29:06 +0530 (IST)
Received: from mail1.mkhoj.com (mx.mkhoj.com [210.210.41.205])
	by orcon.sify.net (Postfix) with ESMTP id 3FA3732843D
	for <general@hadoop.apache.org>; Tue, 11 May 2010 19:29:06 +0530 (IST)
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: ArYFACv96EsKDmQH/2dsb2JhbACRZI04uziFEASDPg
X-IronPort-AV: E=Sophos;i="4.53,207,1272825000"; 
   d="scan'208";a="668326"
Received: from mk-exch-1.mkhoj.com ([10.14.100.7])
  by mail1.mkhoj.com with ESMTP; 11 May 2010 19:29:04 +0530
Received: from [10.14.100.79] (10.14.100.79) by MK-EXCH-1.MKHOJ.COM
 (10.14.100.7) with Microsoft SMTP Server id 8.1.340.0; Tue, 11 May 2010
 19:22:49 +0530
Message-ID: <4BE96263.8040102@inmobi.com>
Date: Tue, 11 May 2010 19:27:55 +0530
From: Rohan Rai <rohan.rai@inmobi.com>
User-Agent: Thunderbird 2.0.0.24 (X11/20100411)
MIME-Version: 1.0
To: <general@hadoop.apache.org>
Subject: JVM Reuse for parallel execution
Content-Type: text/plain; charset="ISO-8859-1"; format=flowed
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Hi

Is there a way of executing multiple tasks in parallel in a single JVM.

For a particular job we see a child process being spawned for each task,
despite
mapred.job.reuse.jvm.num.tasks being set to -1

In order to avoid the overhead of starting a JVM for each task and yet
achieve
parallel execution of tasks, wanted to know sharing the JVM is possible, fo=
r
parallel task execution.

Regards
Rohan

The information contained in this communication is intended solely for the =
use of the individual or entity to whom it is addressed and others authoriz=
ed to receive it. It may contain confidential or legally privileged informa=
tion. If you are not the intended recipient you are hereby notified that an=
y disclosure, copying, distribution or taking any action in reliance on the=
 contents of this information is strictly prohibited and may be unlawful. I=
f you have received this communication in error, please notify us immediate=
ly by responding to this email and then delete it from your system. The fir=
m is neither liable for the proper and complete transmission of the informa=
tion contained in this communication nor for any delay in its receipt.

From general-return-1525-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 11 14:04:32 2010
Return-Path: <general-return-1525-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25693 invoked from network); 11 May 2010 14:04:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 May 2010 14:04:32 -0000
Received: (qmail 11262 invoked by uid 500); 11 May 2010 14:04:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11027 invoked by uid 500); 11 May 2010 14:04:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11018 invoked by uid 99); 11 May 2010 14:04:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 May 2010 14:04:31 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [202.144.10.51] (HELO orcon.sify.net) (202.144.10.51)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 May 2010 14:04:22 +0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by orcon.sify.net (Postfix) with ESMTP id 310BA32A2F2
	for <general@hadoop.apache.org>; Tue, 11 May 2010 19:34:01 +0530 (IST)
X-Virus-Scanned: amavisd-new at example.com
Received: from orcon.sify.net ([127.0.0.1])
	by localhost (orcon.sify.net [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id iLFMxxhylBol for <general@hadoop.apache.org>;
	Tue, 11 May 2010 19:34:01 +0530 (IST)
Received: from mail1.mkhoj.com (mx.mkhoj.com [210.210.41.205])
	by orcon.sify.net (Postfix) with ESMTP id D722032A2AE
	for <general@hadoop.apache.org>; Tue, 11 May 2010 19:34:00 +0530 (IST)
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: ApsEACv96EsKDmQH/2dsb2JhbACfHLs4hRAEgz4
X-IronPort-AV: E=Sophos;i="4.53,207,1272825000"; 
   d="scan'208";a="668345"
Received: from mk-exch-1.mkhoj.com ([10.14.100.7])
  by mail1.mkhoj.com with ESMTP; 11 May 2010 19:33:59 +0530
Received: from [10.14.100.79] (10.14.100.79) by MK-EXCH-1.MKHOJ.COM
 (10.14.100.7) with Microsoft SMTP Server id 8.1.340.0; Tue, 11 May 2010
 19:27:44 +0530
Message-ID: <4BE9638A.5070409@inmobi.com>
Date: Tue, 11 May 2010 19:32:50 +0530
From: Rohan Rai <rohan.rai@inmobi.com>
User-Agent: Thunderbird 2.0.0.24 (X11/20100411)
MIME-Version: 1.0
To: Rohan Rai <rohan.rai@inmobi.com>
CC: <general@hadoop.apache.org>
Subject: Re: JVM Reuse for parallel execution
References: <4BE96263.8040102@inmobi.com>
In-Reply-To: <4BE96263.8040102@inmobi.com>
Content-Type: text/plain; charset="ISO-8859-1"; format=flowed
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Forgot to add, we are using the following configuration.

mapred.tasktracker.map.tasks.maximum as 4
and mapred.tasktracker.reduce.tasks.maximum as 3.

Regards
Rohan


Rohan Rai wrote:
> Hi
>
> Is there a way of executing multiple tasks in parallel in a single JVM.
>
> For a particular job we see a child process being spawned for each
> task, despite
> mapred.job.reuse.jvm.num.tasks being set to -1
>
> In order to avoid the overhead of starting a JVM for each task and yet
> achieve
> parallel execution of tasks, wanted to know sharing the JVM is
> possible, for
> parallel task execution.
>
> Regards
> Rohan
>


The information contained in this communication is intended solely for the =
use of the individual or entity to whom it is addressed and others authoriz=
ed to receive it. It may contain confidential or legally privileged informa=
tion. If you are not the intended recipient you are hereby notified that an=
y disclosure, copying, distribution or taking any action in reliance on the=
 contents of this information is strictly prohibited and may be unlawful. I=
f you have received this communication in error, please notify us immediate=
ly by responding to this email and then delete it from your system. The fir=
m is neither liable for the proper and complete transmission of the informa=
tion contained in this communication nor for any delay in its receipt.

From general-return-1526-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 11 14:23:42 2010
Return-Path: <general-return-1526-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30781 invoked from network); 11 May 2010 14:23:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 May 2010 14:23:42 -0000
Received: (qmail 45733 invoked by uid 500); 11 May 2010 14:23:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45695 invoked by uid 500); 11 May 2010 14:23:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45687 invoked by uid 99); 11 May 2010 14:23:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 May 2010 14:23:41 +0000
X-ASF-Spam-Status: No, hits=1.4 required=10.0
	tests=AWL,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of timrobertson100@gmail.com designates 209.85.211.199 as permitted sender)
Received: from [209.85.211.199] (HELO mail-yw0-f199.google.com) (209.85.211.199)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 May 2010 14:23:36 +0000
Received: by ywh37 with SMTP id 37so3604299ywh.2
        for <general@hadoop.apache.org>; Tue, 11 May 2010 07:23:15 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=QujxtGR10k6nuJXY9GXywoa4hwXMIhJM7jZVLV8fSlQ=;
        b=gXCcfZWIRCAES6ss6p4req7pRQDHn3/JiTLWagHXkV5nC4kelx/5zAYw2jWKeUs3b3
         7vUz/vJ62RakxBdLy1omnt46e8JZ+w+qZp/+7hMKSvJiKoTCMaA+dCGo9xbp0HtyYWBC
         wPFsEgT9I9aXsLGb4f1Vb4eDZktC9qR5XaFt8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=m3JZN+dwQi7RIvAprLN4eW4vH+QOM32ixmtKI5aJezriRO7x4v8jZlsy1hoeuFZAwy
         cCKGpIRRinBwT2LL7/24TP/c0/wVUvAef0uH3rJqeXGmwCxBv/FC9L/zmuqEGpVg0zjb
         f9N4s9i3r94Bwd2CtdW35oZKsC0ABQ+dKIntk=
MIME-Version: 1.0
Received: by 10.151.29.9 with SMTP id g9mr10858792ybj.46.1273587794978; Tue, 
	11 May 2010 07:23:14 -0700 (PDT)
Received: by 10.150.96.7 with HTTP; Tue, 11 May 2010 07:23:14 -0700 (PDT)
In-Reply-To: <4BE9638A.5070409@inmobi.com>
References: <4BE96263.8040102@inmobi.com> <4BE9638A.5070409@inmobi.com>
Date: Tue, 11 May 2010 16:23:14 +0200
Message-ID: <AANLkTilVFEj8MxBkUrCD3axHtFXdYVMNB_IdOKIHY8NR@mail.gmail.com>
Subject: Re: JVM Reuse for parallel execution
From: Tim Robertson <timrobertson100@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd25d101c3df0048652445f

--000e0cd25d101c3df0048652445f
Content-Type: text/plain; charset=ISO-8859-1

http://hadoop.apache.org/common/docs/current/mapred_tutorial.html#Task+JVM+Reuse?

(Just so you know - there is a mapreduce users mailing list for MR specific
questions)

Hope this helps,
Tim

On Tue, May 11, 2010 at 4:02 PM, Rohan Rai <rohan.rai@inmobi.com> wrote:

> Forgot to add, we are using the following configuration.
>
> mapred.tasktracker.map.tasks.maximum as 4
> and mapred.tasktracker.reduce.tasks.maximum as 3.
>
> Regards
> Rohan
>
>
>
> Rohan Rai wrote:
>
>> Hi
>>
>> Is there a way of executing multiple tasks in parallel in a single JVM.
>>
>> For a particular job we see a child process being spawned for each
>> task, despite
>> mapred.job.reuse.jvm.num.tasks being set to -1
>>
>> In order to avoid the overhead of starting a JVM for each task and yet
>> achieve
>> parallel execution of tasks, wanted to know sharing the JVM is
>> possible, for
>> parallel task execution.
>>
>> Regards
>> Rohan
>>
>>
>
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify us
> immediately by responding to this email and then delete it from your system.
> The firm is neither liable for the proper and complete transmission of the
> information contained in this communication nor for any delay in its
> receipt.
>

--000e0cd25d101c3df0048652445f--

From general-return-1527-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 11 17:00:28 2010
Return-Path: <general-return-1527-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76249 invoked from network); 11 May 2010 17:00:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 May 2010 17:00:27 -0000
Received: (qmail 43526 invoked by uid 500); 11 May 2010 17:00:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 43482 invoked by uid 500); 11 May 2010 17:00:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 43474 invoked by uid 99); 11 May 2010 17:00:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 May 2010 17:00:26 +0000
X-ASF-Spam-Status: No, hits=0.5 required=10.0
	tests=AWL,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.51] (HELO mail-ww0-f51.google.com) (74.125.82.51)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 May 2010 17:00:21 +0000
Received: by wwb24 with SMTP id 24so31294wwb.38
        for <general@hadoop.apache.org>; Tue, 11 May 2010 09:59:59 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.168.71 with SMTP id j49mr3139915wel.175.1273597198903; 
	Tue, 11 May 2010 09:59:58 -0700 (PDT)
Received: by 10.216.181.82 with HTTP; Tue, 11 May 2010 09:59:58 -0700 (PDT)
In-Reply-To: <20100510160724.137264ztyvvvcie4@webmail.ens-lyon.fr>
References: <20100510160724.137264ztyvvvcie4@webmail.ens-lyon.fr>
Date: Tue, 11 May 2010 12:59:58 -0400
Message-ID: <AANLkTikNpb5roUQUw7TpAJbh4gntU3Sdmsyt-B04Qq3J@mail.gmail.com>
Subject: Re: Problem with Hadoop Streaming and -D mapred.tasktracker.map.tasks.maximum 
	option
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

The short answer is that with Hadoop, you generally do not decide the
exact number of map tasks that are spawned. The number of map tasks
spawned is usually a function of the number of blocks in the input
data set. Task trackers are configured with a number of slots for map
and reduce tasks. Tasks are assigned to slots on task trackers. By
default, task trackers have 2 map slots and 2 reduce slots per task
tracker.

The manner with which Hadoop assigns tasks to task trackers is based
on a number of factors.

You can attempt to control parallelization at a micro level (as you're
doing) but it's generally a bad idea. Not only are you not taking full
advantage of your cluster, but you are not taking advantage of what
Hadoop is actually good at. In fact, it may not be possible to control
it exactly as you wish. Is there a reason why you need to control
things so strictly? Do you need exactly a multiple of the number of
nodes, or an approximation thereof? What is the rationale for wanting
to run only one task per node?

On Mon, May 10, 2010 at 10:07 AM, Corneliu-Tudor Vlad
<corneliutudor.vlad@ens-lyon.fr> wrote:
>
> Hello
>
> I am a new user of Hadoop and I have some trouble using Hadoop Streaming =
and
> the "-D mapred.tasktracker.map.tasks.maximum" option.
>
> I'm experimenting with an unmanaged application (C++) which I want to run
> over several nodes in 2 scenarious
> 1) the number of maps (input splits) is equal to the number of nodes
> 2) the number of maps is a multiple of the number of nodes (5, 10, 20, ..=
.
>
> Initially, when running the tests in scenario 1 I would sometimes get 2
> process/node on half the nodes. However I fixed this by adding the direct=
ive
> -D mapred.tasktracker.map.tasks.maximum=3D1, so everything works fine.
>
> In the case of scenario 2 (more maps than nodes) this directive no longer
> works, always obtaining 2 processes/node. I tested the even with putting
> maximum=3D5 and I still get 2 processes/node.
>
> The entire command I use is:
>
> /usr/bin/time --format=3D"-duration:\t%e |\t-MFaults:\t%F
> |\t-ContxtSwitch:\t%w" \
> =A0/opt/hadoop/bin/hadoop jar
> /opt/hadoop/contrib/streaming/hadoop-0.20.2-streaming.jar \
> =A0-D mapred.tasktracker.map.tasks.maximum=3D1 \
> =A0-D mapred.map.tasks=3D30 \
> =A0-D mapred.reduce.tasks=3D0 \
> =A0-D io.file.buffer.size=3D5242880 \
> =A0-libjars "/opt/hadoop/contrib/streaming/hadoop-7debug.jar" \
> =A0-input input/test553short \
> =A0-output out1 \
> =A0-mapper "/opt/jobdata/script_1k" \
> =A0-inputformat "me.MyInputFormat"
>
> I'm using is Debian Lenny x64, and Hadoop 0.20.2.
>
> My question is: why is this happening and how can I make it work properly
> (i.e. be able to limit exactly how many mappers I can have at 1 time per
> node)
>
> Thank you in advance,
> T
>
>



--=20
Eric Sammer
phone: +1-917-287-2675
twitter: esammer
data: www.cloudera.com

From general-return-1528-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 11 17:32:03 2010
Return-Path: <general-return-1528-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85596 invoked from network); 11 May 2010 17:32:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 May 2010 17:32:03 -0000
Received: (qmail 93485 invoked by uid 500); 11 May 2010 17:32:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93222 invoked by uid 500); 11 May 2010 17:32:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93214 invoked by uid 99); 11 May 2010 17:32:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 May 2010 17:32:02 +0000
X-ASF-Spam-Status: No, hits=0.6 required=10.0
	tests=AWL,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 May 2010 17:31:56 +0000
Received: by pwi6 with SMTP id 6so122641pwi.35
        for <general@hadoop.apache.org>; Tue, 11 May 2010 10:31:36 -0700 (PDT)
Received: by 10.141.22.19 with SMTP id z19mr3963708rvi.250.1273599096185; Tue, 
	11 May 2010 10:31:36 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.140.169.13 with HTTP; Tue, 11 May 2010 10:31:16 -0700 (PDT)
In-Reply-To: <AANLkTinYM84xSEs30UT5RU1K7UjguNuztpy-yQjr2pXZ@mail.gmail.com>
References: <y2p44be54d01005050829gf270a41du98eaf93b0fe2f070@mail.gmail.com> 
	<j2hd6d7c4411005051804p40b2b598ud593f984c09d779@mail.gmail.com> 
	<AANLkTinYM84xSEs30UT5RU1K7UjguNuztpy-yQjr2pXZ@mail.gmail.com>
From: Aaron Kimball <aaron@cloudera.com>
Date: Tue, 11 May 2010 10:31:16 -0700
Message-ID: <AANLkTinqlIpg7UNX8DaWCfqoZghXwgh5dHcwHpYM6asl@mail.gmail.com>
Subject: Re: Hadoop Data Sharing
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd1821cb6e5cd048654e526

--000e0cd1821cb6e5cd048654e526
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

What objects are you referring to? I'm not sure I understand your question.
- Aaron

On Tue, May 11, 2010 at 6:38 AM, Renato Marroqu=EDn Mogrovejo <
renatoj.marroquin@gmail.com> wrote:

> Thanks Aaron! I was thinking the same after doing some reading.
> Man what about serialize the objects? Would you think that is a good idea=
?
> Thanks again.
>
> Renato M.
>
>
> 2010/5/5 Aaron Kimball <aaron@cloudera.com>
>
> > Renato,
> >
> > In general if you need to perform a multi-pass MapReduce workflow, each
> > pass
> > materializes its output to files. The subsequent pass then reads those
> same
> > files back in as input. This allows the workflow to start at the last
> > "checkpoint" if it gets interrupted. There is no persistent in-memory
> > distributed storage feature in Hadoop that would allow a MapReduce job =
to
> > post results to memory for consumption by a subsequent job.
> >
> > So you would just read your initial data from /input, and write your
> > interim
> > results to /iteration0. Then the next pass reads from /iteration0 and
> > writes
> > to /iteration1, etc..
> >
> > If your data is reasonably small and you think it could fit in memory
> > somewhere, then you could experiment with using other distributed
> key-value
> > stores (memcached[b], hbase, cassandra, etc..) to hold intermediate
> > results.
> > But this will require some integration work on your part.
> > - Aaron
> >
> > On Wed, May 5, 2010 at 8:29 AM, Renato Marroqu=EDn Mogrovejo <
> > renatoj.marroquin@gmail.com> wrote:
> >
> > > Hi everyone, I have recently started to play around with hadoop, but =
I
> am
> > > getting some into some "design" problems.
> > > I need to make a loop to execute the same job several times, and in
> each
> > > iteration get the processed values (not using a file because I would
> need
> > > to
> > > read it). I was using an static vector in my main class (the one that
> > > iterates and executes the job in each iteration) to retrieve those
> > values,
> > > and it did work while I was using a standalone mode. Now I tried to
> test
> > it
> > > on a pseudo-distributed manner and obviously is not working.
> > > Any suggestions, please???
> > >
> > > Thanks in advance,
> > >
> > >
> > > Renato M.
> > >
> >
>

--000e0cd1821cb6e5cd048654e526--

From general-return-1529-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 11 17:43:31 2010
Return-Path: <general-return-1529-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88389 invoked from network); 11 May 2010 17:43:31 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 May 2010 17:43:31 -0000
Received: (qmail 6964 invoked by uid 500); 11 May 2010 17:43:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6922 invoked by uid 500); 11 May 2010 17:43:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6914 invoked by uid 99); 11 May 2010 17:43:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 May 2010 17:43:29 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 May 2010 17:43:22 +0000
Received: by pvg13 with SMTP id 13so597826pvg.35
        for <general@hadoop.apache.org>; Tue, 11 May 2010 10:43:01 -0700 (PDT)
Received: by 10.141.89.1 with SMTP id r1mr4010857rvl.290.1273599780888; Tue, 
	11 May 2010 10:43:00 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.140.169.13 with HTTP; Tue, 11 May 2010 10:34:49 -0700 (PDT)
In-Reply-To: <AANLkTinqlIpg7UNX8DaWCfqoZghXwgh5dHcwHpYM6asl@mail.gmail.com>
References: <y2p44be54d01005050829gf270a41du98eaf93b0fe2f070@mail.gmail.com> 
	<j2hd6d7c4411005051804p40b2b598ud593f984c09d779@mail.gmail.com> 
	<AANLkTinYM84xSEs30UT5RU1K7UjguNuztpy-yQjr2pXZ@mail.gmail.com> 
	<AANLkTinqlIpg7UNX8DaWCfqoZghXwgh5dHcwHpYM6asl@mail.gmail.com>
From: Aaron Kimball <aaron@cloudera.com>
Date: Tue, 11 May 2010 10:34:49 -0700
Message-ID: <AANLkTimMjEYBtO5dwkHJ5oPW49P2eVngVRBdLFyxCc19@mail.gmail.com>
Subject: Re: Hadoop Data Sharing
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd13a38869f9d0486550e4b
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd13a38869f9d0486550e4b
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Perhaps this is guidance in the area you were hoping for: If your data is i=
n
objects that implement the interface 'Writable', then you can use the
SequenceFileOutputFormat and SequenceFileInputFormat to store your
intermediate data in binary form in disk-backed files called SequenceFiles.
The serialization will happen through the write() and readFields() methods
of your objects, which will automatically be called by the
OutputFormat/InputFormat as they move through the system. So your subsequen=
t
MR pass will receive objects back in the same form as they were emitted.
This is a considerably better idea (from both a throughput and a sanity
perspective) in a chained MapReduce job.

- Aaron

On Tue, May 11, 2010 at 10:31 AM, Aaron Kimball <aaron@cloudera.com> wrote:

> What objects are you referring to? I'm not sure I understand your questio=
n.
> - Aaron
>
>
> On Tue, May 11, 2010 at 6:38 AM, Renato Marroqu=EDn Mogrovejo <
> renatoj.marroquin@gmail.com> wrote:
>
>> Thanks Aaron! I was thinking the same after doing some reading.
>> Man what about serialize the objects? Would you think that is a good ide=
a?
>> Thanks again.
>>
>> Renato M.
>>
>>
>> 2010/5/5 Aaron Kimball <aaron@cloudera.com>
>>
>> > Renato,
>> >
>> > In general if you need to perform a multi-pass MapReduce workflow, eac=
h
>> > pass
>> > materializes its output to files. The subsequent pass then reads those
>> same
>> > files back in as input. This allows the workflow to start at the last
>> > "checkpoint" if it gets interrupted. There is no persistent in-memory
>> > distributed storage feature in Hadoop that would allow a MapReduce job
>> to
>> > post results to memory for consumption by a subsequent job.
>> >
>> > So you would just read your initial data from /input, and write your
>> > interim
>> > results to /iteration0. Then the next pass reads from /iteration0 and
>> > writes
>> > to /iteration1, etc..
>> >
>> > If your data is reasonably small and you think it could fit in memory
>> > somewhere, then you could experiment with using other distributed
>> key-value
>> > stores (memcached[b], hbase, cassandra, etc..) to hold intermediate
>> > results.
>> > But this will require some integration work on your part.
>> > - Aaron
>> >
>> > On Wed, May 5, 2010 at 8:29 AM, Renato Marroqu=EDn Mogrovejo <
>> > renatoj.marroquin@gmail.com> wrote:
>> >
>> > > Hi everyone, I have recently started to play around with hadoop, but=
 I
>> am
>> > > getting some into some "design" problems.
>> > > I need to make a loop to execute the same job several times, and in
>> each
>> > > iteration get the processed values (not using a file because I would
>> need
>> > > to
>> > > read it). I was using an static vector in my main class (the one tha=
t
>> > > iterates and executes the job in each iteration) to retrieve those
>> > values,
>> > > and it did work while I was using a standalone mode. Now I tried to
>> test
>> > it
>> > > on a pseudo-distributed manner and obviously is not working.
>> > > Any suggestions, please???
>> > >
>> > > Thanks in advance,
>> > >
>> > >
>> > > Renato M.
>> > >
>> >
>>
>
>

--000e0cd13a38869f9d0486550e4b--

From general-return-1530-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 11 20:09:38 2010
Return-Path: <general-return-1530-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 32140 invoked from network); 11 May 2010 20:09:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 May 2010 20:09:38 -0000
Received: (qmail 30198 invoked by uid 500); 11 May 2010 20:09:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29999 invoked by uid 500); 11 May 2010 20:09:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 47474 invoked by uid 99); 11 May 2010 19:15:34 -0000
X-ASF-Spam-Status: No, hits=3.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_SOFTFAIL
X-Spam-Check-By: apache.org
Received-SPF: softfail (nike.apache.org: transitioning domain of kiko@meebo-inc.com does not designate 74.125.83.176 as permitted sender)
MIME-Version: 1.0
Date: Tue, 11 May 2010 12:15:05 -0700
Message-ID: <AANLkTikWHRrjnejkR6K4YkW6aI3bZnYrBlQnfPyM6R67@mail.gmail.com>
Subject: @ Meebo - Data Engineer (Analytics)
From: Kiko Griffin <kiko@meebo-inc.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636e0ae4dd7e1a604865657e8
X-Virus-Checked: Checked by ClamAV on apache.org

--001636e0ae4dd7e1a604865657e8
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Hello!

We have an immediate need here at Meebo and I'm positive someone from the
community would be interested.  This is a permanent opportunity located in
Mountain View, CA.  We are looking for a Data Engineer (Hadoop, MapReduce)
to join the Meebo ranks.  If at all interested OR know someone who might be
interested please reply directly to Kiko@meebo-inc.com.

data engineer, analytics

Position located in Mountain View

Meebo is seeking an empirically-minded groundbreaker to join its ranks. We
love instrumentation, A/B testing, and take delight in uncovering the
unexpected. We=92re seeking our next team member who will build a world-cla=
ss
product-oriented data system that will serve as the compass for nearly all
strategic company decisions. This individual should have a deep desire to
work with very large data sets to guide product decisions and to strengthen
the end-user voice within Meebo.

*In this role, your responsibilities will include:*

   - Design and deploy Meebo=92s core product data analysis framework that
   includes A/B testing, real-time log processing, segmentation capabilitie=
s,
   and surfacing moving metrics to the most relevant teams
   - Create tools, web interfaces, and visualizations to allow product team=
s
   and partners to access real-time usage statistics
   - Instrument JavaScript and C/C++ codebase to collect usage behavior dat=
a
   - Proactively recommend metrics for understanding product health and use=
r
   satisfaction
   - Collaborate with the Operations team to design and deploy Meebo=92s ne=
xt
   generation distributed data system

*To be most effective, you will have:*

   - BS or MS in Computer Science with an emphasis in Information Analytics=
,
   Data Mining, Machine Learning, Information Retrieval, Artificial
   Intelligence, or related field
   - Expertise in MapReduce programming models such as Hadoop and other
   parallel computing implementations for processing and generating large d=
ata
   sets
   - Scripting skills including HTML/CSS/JavaScript, Python, and SQL
   - The product and business savvy mindset to pinpoint what metric matters
   most and the communication skills to convey those thoughts to others

*You will be measured by your ability to:*

Work with a diverse team to cultivate ideas, foster user-centric thinking,
and positively influence product direction Listen deeply to team product an=
d
business issues and make data and research suggestions that will best infor=
m
strategic-thinking Build a world-class stats infrastructure that is
accessible company-wide that will serve as the groundwork for all future
stats projects at Meebo

--001636e0ae4dd7e1a604865657e8--

From general-return-1531-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 11 20:20:26 2010
Return-Path: <general-return-1531-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34319 invoked from network); 11 May 2010 20:20:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 May 2010 20:20:25 -0000
Received: (qmail 40187 invoked by uid 500); 11 May 2010 20:20:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40108 invoked by uid 500); 11 May 2010 20:20:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40100 invoked by uid 99); 11 May 2010 20:20:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 May 2010 20:20:24 +0000
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of renatoj.marroquin@gmail.com designates 209.85.211.199 as permitted sender)
Received: from [209.85.211.199] (HELO mail-yw0-f199.google.com) (209.85.211.199)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 May 2010 20:20:19 +0000
Received: by ywh37 with SMTP id 37so3915725ywh.2
        for <general@hadoop.apache.org>; Tue, 11 May 2010 13:19:58 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=VI4+wZxqx3q/Lbxa2tmu6yG/LZU8hKJvfIdEzl9b3Gw=;
        b=Zbc0XKu4KIkYBrJfkUH41dm67tMKIj/aQR8Kil6In+uiS76ItmXTQ2Id0FNOU0twQs
         QBZkU88oTP72299dJeMH10iZi66QQIo2QCL/GNEDJeWMZBe8lMM4tasAZ1GqGaoA44v5
         1gSckA+dRVBGZchlAg0Q9udypgARTbGn3yTCc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=uMn6+D08Ypkufmk7CTRU3G/IBBxFoe3ugTik/nvsSkfL/vZWFCPIH/epBOWjZHr5Jw
         XKqhujaiFUmBHCyakp7oVIg5SxrVVnTJClBb+LkU7oUMQ74kbSckATQ5rG0akHODm++g
         3ewt+dV4PWGWHYXHL2QY31Ks37b89vNkb0YpU=
MIME-Version: 1.0
Received: by 10.101.2.13 with SMTP id e13mr2969071ani.107.1273609196882; Tue, 
	11 May 2010 13:19:56 -0700 (PDT)
Received: by 10.100.231.13 with HTTP; Tue, 11 May 2010 13:19:56 -0700 (PDT)
In-Reply-To: <AANLkTimMjEYBtO5dwkHJ5oPW49P2eVngVRBdLFyxCc19@mail.gmail.com>
References: <y2p44be54d01005050829gf270a41du98eaf93b0fe2f070@mail.gmail.com>
	 <j2hd6d7c4411005051804p40b2b598ud593f984c09d779@mail.gmail.com>
	 <AANLkTinYM84xSEs30UT5RU1K7UjguNuztpy-yQjr2pXZ@mail.gmail.com>
	 <AANLkTinqlIpg7UNX8DaWCfqoZghXwgh5dHcwHpYM6asl@mail.gmail.com>
	 <AANLkTimMjEYBtO5dwkHJ5oPW49P2eVngVRBdLFyxCc19@mail.gmail.com>
Date: Tue, 11 May 2010 17:19:56 -0300
Message-ID: <AANLkTikKkO_gpjZ-Z6hu0TZjb3pAY0qVJDQ1eHmlEO8E@mail.gmail.com>
Subject: Re: Hadoop Data Sharing
From: =?ISO-8859-1?Q?Renato_Marroqu=EDn_Mogrovejo?= <renatoj.marroquin@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016368e357cc376080486573ff5

--0016368e357cc376080486573ff5
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Aaron,

The thing is that I had a data structure that is saved into a vector, and
this vector needs to be available for my MapReduce jobs while iterating. So
would you think it would a good and easy way to serialize this objects? It'=
s
a vector that each node contains another user define data structure. Maybe =
I
will try to do it first just using files, and see how the throughput goes.
Hey do you know where I can find some examples of serializing objects for
Hadoop to save them into SequenceFiles?
Thanks in advance.

Renato M.


2010/5/11 Aaron Kimball <aaron@cloudera.com>

> Perhaps this is guidance in the area you were hoping for: If your data is
> in
> objects that implement the interface 'Writable', then you can use the
> SequenceFileOutputFormat and SequenceFileInputFormat to store your
> intermediate data in binary form in disk-backed files called SequenceFile=
s.
> The serialization will happen through the write() and readFields() method=
s
> of your objects, which will automatically be called by the
> OutputFormat/InputFormat as they move through the system. So your
> subsequent
> MR pass will receive objects back in the same form as they were emitted.
> This is a considerably better idea (from both a throughput and a sanity
> perspective) in a chained MapReduce job.
>
> - Aaron
>
> On Tue, May 11, 2010 at 10:31 AM, Aaron Kimball <aaron@cloudera.com>
> wrote:
>
> > What objects are you referring to? I'm not sure I understand your
> question.
> > - Aaron
> >
> >
> > On Tue, May 11, 2010 at 6:38 AM, Renato Marroqu=EDn Mogrovejo <
> > renatoj.marroquin@gmail.com> wrote:
> >
> >> Thanks Aaron! I was thinking the same after doing some reading.
> >> Man what about serialize the objects? Would you think that is a good
> idea?
> >> Thanks again.
> >>
> >> Renato M.
> >>
> >>
> >> 2010/5/5 Aaron Kimball <aaron@cloudera.com>
> >>
> >> > Renato,
> >> >
> >> > In general if you need to perform a multi-pass MapReduce workflow,
> each
> >> > pass
> >> > materializes its output to files. The subsequent pass then reads tho=
se
> >> same
> >> > files back in as input. This allows the workflow to start at the las=
t
> >> > "checkpoint" if it gets interrupted. There is no persistent in-memor=
y
> >> > distributed storage feature in Hadoop that would allow a MapReduce j=
ob
> >> to
> >> > post results to memory for consumption by a subsequent job.
> >> >
> >> > So you would just read your initial data from /input, and write your
> >> > interim
> >> > results to /iteration0. Then the next pass reads from /iteration0 an=
d
> >> > writes
> >> > to /iteration1, etc..
> >> >
> >> > If your data is reasonably small and you think it could fit in memor=
y
> >> > somewhere, then you could experiment with using other distributed
> >> key-value
> >> > stores (memcached[b], hbase, cassandra, etc..) to hold intermediate
> >> > results.
> >> > But this will require some integration work on your part.
> >> > - Aaron
> >> >
> >> > On Wed, May 5, 2010 at 8:29 AM, Renato Marroqu=EDn Mogrovejo <
> >> > renatoj.marroquin@gmail.com> wrote:
> >> >
> >> > > Hi everyone, I have recently started to play around with hadoop, b=
ut
> I
> >> am
> >> > > getting some into some "design" problems.
> >> > > I need to make a loop to execute the same job several times, and i=
n
> >> each
> >> > > iteration get the processed values (not using a file because I wou=
ld
> >> need
> >> > > to
> >> > > read it). I was using an static vector in my main class (the one
> that
> >> > > iterates and executes the job in each iteration) to retrieve those
> >> > values,
> >> > > and it did work while I was using a standalone mode. Now I tried t=
o
> >> test
> >> > it
> >> > > on a pseudo-distributed manner and obviously is not working.
> >> > > Any suggestions, please???
> >> > >
> >> > > Thanks in advance,
> >> > >
> >> > >
> >> > > Renato M.
> >> > >
> >> >
> >>
> >
> >
>

--0016368e357cc376080486573ff5--

From general-return-1532-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 11 20:27:22 2010
Return-Path: <general-return-1532-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35682 invoked from network); 11 May 2010 20:27:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 May 2010 20:27:22 -0000
Received: (qmail 50236 invoked by uid 500); 11 May 2010 20:27:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50193 invoked by uid 500); 11 May 2010 20:27:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50185 invoked by uid 99); 11 May 2010 20:27:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 May 2010 20:27:21 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=AWL,MIME_QP_LONG_LINE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of corneliutudor.vlad@ens-lyon.fr designates 140.77.51.2 as permitted sender)
Received: from [140.77.51.2] (HELO jabiru.ens-lyon.fr) (140.77.51.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 May 2010 20:27:14 +0000
Received: from localhost (localhost [127.0.0.1])
	by jabiru.ens-lyon.fr (Postfix) with ESMTP id DBE691EB199
	for <general@hadoop.apache.org>; Tue, 11 May 2010 22:26:52 +0200 (CEST)
X-Virus-Scanned: by amavisd-new-2.6.1 (20080629) (Debian) at ens-lyon.fr
Received: from jabiru.ens-lyon.fr ([127.0.0.1])
	by localhost (jabiru.ens-lyon.fr [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id OmpTj-Z-YYTq for <general@hadoop.apache.org>;
	Tue, 11 May 2010 22:26:50 +0200 (CEST)
Received: from agrobate-domu-webmail.ens-lyon.fr (agrobate-domu-webmail.ens-lyon.fr [140.77.51.15])
	by jabiru.ens-lyon.fr (Postfix) with ESMTP id C5ADF1EB1A2
	for <general@hadoop.apache.org>; Tue, 11 May 2010 22:26:50 +0200 (CEST)
Received: by agrobate-domu-webmail.ens-lyon.fr (Postfix, from userid 33)
	id BCC9A581D0; Tue, 11 May 2010 22:26:50 +0200 (CEST)
Received: from dhcp-128-245.residence.ens-lyon.fr
 (dhcp-128-245.residence.ens-lyon.fr [140.77.128.245]) by
 webmail.ens-lyon.fr (Horde Framework) with HTTP; Tue, 11 May 2010 22:26:50
 +0200
Message-ID: <20100511222650.10634snj3fsw2vt6@webmail.ens-lyon.fr>
X-Priority: 3 (Normal)
Date: Tue, 11 May 2010 22:26:50 +0200
From: Corneliu-Tudor Vlad <corneliutudor.vlad@ens-lyon.fr>
To: general@hadoop.apache.org
Subject: Re: Problem with Hadoop Streaming and -D
 mapred.tasktracker.map.tasks.maximum option
References: <20100510160724.137264ztyvvvcie4@webmail.ens-lyon.fr>
 <AANLkTikNpb5roUQUw7TpAJbh4gntU3Sdmsyt-B04Qq3J@mail.gmail.com>
In-Reply-To: <AANLkTikNpb5roUQUw7TpAJbh4gntU3Sdmsyt-B04Qq3J@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain;
 charset=ISO-8859-1;
 DelSp="Yes";
 format="flowed"
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
User-Agent: Internet Messaging Program (IMP) 4.3.3

Hello

Thank you for the answer, it is a little clearer now. If you could =20
point me to some additional reading, I will be grateful.

In the meantime I created an issue on Jira [ MAPREDUCE-1781 ] and I =20
both received answer there and provided insight on my objective.

As I understand from both your answer and Hemanth's on Jira, I am =20
using Hadoop in a non-standard way. The reason is that I am performing =20
some test on the feasibility of using Hadoop as a parallelization =20
framework for a highly-CPU & memory bounded application, but only from =20
the point of view of the distributed computed, not multicore. That is =20
why I only want 1 process at a time.

Additionally I will test it on a heterogenous datacenter, possibly =20
with both dual cores and quads, thus even if I use 2 mappers at once, =20
I won't fully use the power of the cluster (from what I understand).

 From what I tested today, my intended approach works with the =20
tasks.maximum option in the config file at startup.

Thank you,
T


Quoting Eric Sammer <esammer@cloudera.com>:

> The short answer is that with Hadoop, you generally do not decide the
> exact number of map tasks that are spawned. The number of map tasks
> spawned is usually a function of the number of blocks in the input
> data set. Task trackers are configured with a number of slots for map
> and reduce tasks. Tasks are assigned to slots on task trackers. By
> default, task trackers have 2 map slots and 2 reduce slots per task
> tracker.
>
> The manner with which Hadoop assigns tasks to task trackers is based
> on a number of factors.
>
> You can attempt to control parallelization at a micro level (as you're
> doing) but it's generally a bad idea. Not only are you not taking full
> advantage of your cluster, but you are not taking advantage of what
> Hadoop is actually good at. In fact, it may not be possible to control
> it exactly as you wish. Is there a reason why you need to control
> things so strictly? Do you need exactly a multiple of the number of
> nodes, or an approximation thereof? What is the rationale for wanting
> to run only one task per node?
>
> On Mon, May 10, 2010 at 10:07 AM, Corneliu-Tudor Vlad
> <corneliutudor.vlad@ens-lyon.fr> wrote:
>>
>> Hello
>>
>> I am a new user of Hadoop and I have some trouble using Hadoop Streaming =
and
>> the "-D mapred.tasktracker.map.tasks.maximum" option.
>>
>> I'm experimenting with an unmanaged application (C++) which I want to run
>> over several nodes in 2 scenarious
>> 1) the number of maps (input splits) is equal to the number of nodes
>> 2) the number of maps is a multiple of the number of nodes (5, 10, 20, ..=
.
>>
>> Initially, when running the tests in scenario 1 I would sometimes get 2
>> process/node on half the nodes. However I fixed this by adding the direct=
ive
>> -D mapred.tasktracker.map.tasks.maximum=3D1, so everything works fine.
>>
>> In the case of scenario 2 (more maps than nodes) this directive no longer
>> works, always obtaining 2 processes/node. I tested the even with putting
>> maximum=3D5 and I still get 2 processes/node.
>>
>> The entire command I use is:
>>
>> /usr/bin/time --format=3D"-duration:\t%e |\t-MFaults:\t%F
>> |\t-ContxtSwitch:\t%w" \
>> =A0/opt/hadoop/bin/hadoop jar
>> /opt/hadoop/contrib/streaming/hadoop-0.20.2-streaming.jar \
>> =A0-D mapred.tasktracker.map.tasks.maximum=3D1 \
>> =A0-D mapred.map.tasks=3D30 \
>> =A0-D mapred.reduce.tasks=3D0 \
>> =A0-D io.file.buffer.size=3D5242880 \
>> =A0-libjars "/opt/hadoop/contrib/streaming/hadoop-7debug.jar" \
>> =A0-input input/test553short \
>> =A0-output out1 \
>> =A0-mapper "/opt/jobdata/script_1k" \
>> =A0-inputformat "me.MyInputFormat"
>>
>> I'm using is Debian Lenny x64, and Hadoop 0.20.2.
>>
>> My question is: why is this happening and how can I make it work properly
>> (i.e. be able to limit exactly how many mappers I can have at 1 time per
>> node)
>>
>> Thank you in advance,
>> T
>>
>>
>
>
>
> --
> Eric Sammer
> phone: +1-917-287-2675
> twitter: esammer
> data: www.cloudera.com
>



From general-return-1533-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 11 20:34:59 2010
Return-Path: <general-return-1533-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37153 invoked from network); 11 May 2010 20:34:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 May 2010 20:34:59 -0000
Received: (qmail 62524 invoked by uid 500); 11 May 2010 20:34:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62457 invoked by uid 500); 11 May 2010 20:34:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62449 invoked by uid 99); 11 May 2010 20:34:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 May 2010 20:34:58 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jaybooth@gmail.com designates 74.125.83.48 as permitted sender)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 May 2010 20:34:51 +0000
Received: by gwj20 with SMTP id 20so2132280gwj.35
        for <general@hadoop.apache.org>; Tue, 11 May 2010 13:34:30 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=eEKOdMJ8X5vN5GF0Nle4fePK+loNo0HaMU8YxC+B5x8=;
        b=nL/t/nDOzyZD13Wr98eS9buH4fPI0rHVI2vUFHc2iBYIiK7IacFyr2SzmpvBQryqle
         pi/aLOUUtVLMM/tCyS50elA2Wwg2vHpa8a276z01edrSsw9NW1BR3GvCxYX/RK1OBkOW
         GOAlvAOrjkF1HAIECRwLpEqMJZs5CcoY0nJ1I=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=CqKQZuJSrps6Ml/fgdbjTiOpBDq4mj2M0TrPWcp5EaaHPZ567ktEHYNELYfqOGP3ZQ
         V0mKR1l8T0QIQQwaiMFrWA9u4TP+Oj0Kpft+eJ2T6g9mOKH+hR8I+BlTvrZncxMUKo7y
         ZO1JdYwhIiwxLTbWfUjCiHsuasp/i26O7kjrc=
MIME-Version: 1.0
Received: by 10.101.172.4 with SMTP id z4mr3095903ano.197.1273610068736; Tue, 
	11 May 2010 13:34:28 -0700 (PDT)
Received: by 10.100.141.16 with HTTP; Tue, 11 May 2010 13:34:28 -0700 (PDT)
In-Reply-To: <AANLkTikKkO_gpjZ-Z6hu0TZjb3pAY0qVJDQ1eHmlEO8E@mail.gmail.com>
References: <y2p44be54d01005050829gf270a41du98eaf93b0fe2f070@mail.gmail.com>
	 <j2hd6d7c4411005051804p40b2b598ud593f984c09d779@mail.gmail.com>
	 <AANLkTinYM84xSEs30UT5RU1K7UjguNuztpy-yQjr2pXZ@mail.gmail.com>
	 <AANLkTinqlIpg7UNX8DaWCfqoZghXwgh5dHcwHpYM6asl@mail.gmail.com>
	 <AANLkTimMjEYBtO5dwkHJ5oPW49P2eVngVRBdLFyxCc19@mail.gmail.com>
	 <AANLkTikKkO_gpjZ-Z6hu0TZjb3pAY0qVJDQ1eHmlEO8E@mail.gmail.com>
Date: Tue, 11 May 2010 16:34:28 -0400
Message-ID: <AANLkTikdR0ri6Y32uPlNZp6z3y6gHwwRBShtvNTYoGiL@mail.gmail.com>
Subject: Re: Hadoop Data Sharing
From: Jay Booth <jaybooth@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Probably the most direct route to get your desired result is to save
the objects to either a SequenceFile or plain text file on DFS.  Then
in the configure() section of your mapreduce jobs, you open the file
on DFS, stream contents into a local variable and refer to it as  you
need to.  Either way, you'll need some sort of serialization via
Writable or plain text.

On Tue, May 11, 2010 at 4:19 PM, Renato Marroqu=EDn Mogrovejo
<renatoj.marroquin@gmail.com> wrote:
> Hi Aaron,
>
> The thing is that I had a data structure that is saved into a vector, and
> this vector needs to be available for my MapReduce jobs while iterating. =
So
> would you think it would a good and easy way to serialize this objects? I=
t's
> a vector that each node contains another user define data structure. Mayb=
e I
> will try to do it first just using files, and see how the throughput goes=
.
> Hey do you know where I can find some examples of serializing objects for
> Hadoop to save them into SequenceFiles?
> Thanks in advance.
>
> Renato M.
>
>
> 2010/5/11 Aaron Kimball <aaron@cloudera.com>
>
>> Perhaps this is guidance in the area you were hoping for: If your data i=
s
>> in
>> objects that implement the interface 'Writable', then you can use the
>> SequenceFileOutputFormat and SequenceFileInputFormat to store your
>> intermediate data in binary form in disk-backed files called SequenceFil=
es.
>> The serialization will happen through the write() and readFields() metho=
ds
>> of your objects, which will automatically be called by the
>> OutputFormat/InputFormat as they move through the system. So your
>> subsequent
>> MR pass will receive objects back in the same form as they were emitted.
>> This is a considerably better idea (from both a throughput and a sanity
>> perspective) in a chained MapReduce job.
>>
>> - Aaron
>>
>> On Tue, May 11, 2010 at 10:31 AM, Aaron Kimball <aaron@cloudera.com>
>> wrote:
>>
>> > What objects are you referring to? I'm not sure I understand your
>> question.
>> > - Aaron
>> >
>> >
>> > On Tue, May 11, 2010 at 6:38 AM, Renato Marroqu=EDn Mogrovejo <
>> > renatoj.marroquin@gmail.com> wrote:
>> >
>> >> Thanks Aaron! I was thinking the same after doing some reading.
>> >> Man what about serialize the objects? Would you think that is a good
>> idea?
>> >> Thanks again.
>> >>
>> >> Renato M.
>> >>
>> >>
>> >> 2010/5/5 Aaron Kimball <aaron@cloudera.com>
>> >>
>> >> > Renato,
>> >> >
>> >> > In general if you need to perform a multi-pass MapReduce workflow,
>> each
>> >> > pass
>> >> > materializes its output to files. The subsequent pass then reads th=
ose
>> >> same
>> >> > files back in as input. This allows the workflow to start at the la=
st
>> >> > "checkpoint" if it gets interrupted. There is no persistent in-memo=
ry
>> >> > distributed storage feature in Hadoop that would allow a MapReduce =
job
>> >> to
>> >> > post results to memory for consumption by a subsequent job.
>> >> >
>> >> > So you would just read your initial data from /input, and write you=
r
>> >> > interim
>> >> > results to /iteration0. Then the next pass reads from /iteration0 a=
nd
>> >> > writes
>> >> > to /iteration1, etc..
>> >> >
>> >> > If your data is reasonably small and you think it could fit in memo=
ry
>> >> > somewhere, then you could experiment with using other distributed
>> >> key-value
>> >> > stores (memcached[b], hbase, cassandra, etc..) to hold intermediate
>> >> > results.
>> >> > But this will require some integration work on your part.
>> >> > - Aaron
>> >> >
>> >> > On Wed, May 5, 2010 at 8:29 AM, Renato Marroqu=EDn Mogrovejo <
>> >> > renatoj.marroquin@gmail.com> wrote:
>> >> >
>> >> > > Hi everyone, I have recently started to play around with hadoop, =
but
>> I
>> >> am
>> >> > > getting some into some "design" problems.
>> >> > > I need to make a loop to execute the same job several times, and =
in
>> >> each
>> >> > > iteration get the processed values (not using a file because I wo=
uld
>> >> need
>> >> > > to
>> >> > > read it). I was using an static vector in my main class (the one
>> that
>> >> > > iterates and executes the job in each iteration) to retrieve thos=
e
>> >> > values,
>> >> > > and it did work while I was using a standalone mode. Now I tried =
to
>> >> test
>> >> > it
>> >> > > on a pseudo-distributed manner and obviously is not working.
>> >> > > Any suggestions, please???
>> >> > >
>> >> > > Thanks in advance,
>> >> > >
>> >> > >
>> >> > > Renato M.
>> >> > >
>> >> >
>> >>
>> >
>> >
>>
>

From general-return-1534-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 11 23:13:44 2010
Return-Path: <general-return-1534-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82639 invoked from network); 11 May 2010 23:13:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 May 2010 23:13:44 -0000
Received: (qmail 16052 invoked by uid 500); 11 May 2010 23:13:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15843 invoked by uid 500); 11 May 2010 23:13:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15807 invoked by uid 99); 11 May 2010 23:13:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 May 2010 23:13:38 +0000
X-ASF-Spam-Status: No, hits=4.2 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL,URIBL_BLACK
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.217.216] (HELO mail-gx0-f216.google.com) (209.85.217.216)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 May 2010 23:13:32 +0000
Received: by gxk8 with SMTP id 8so1151112gxk.9
        for <multiple recipients>; Tue, 11 May 2010 16:13:11 -0700 (PDT)
Received: by 10.91.1.2 with SMTP id d2mr2721841agi.121.1273619588779; Tue, 11 
	May 2010 16:13:08 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.90.97.2 with HTTP; Tue, 11 May 2010 16:12:48 -0700 (PDT)
From: Christophe Bisciglia <christophe@cloudera.com>
Date: Tue, 11 May 2010 16:12:48 -0700
Message-ID: <AANLkTinLQ-7zITzjnPLbrtcJvAnvyp3Jt-MEffe4kxTr@mail.gmail.com>
Subject: Hadoop Training @ Hadoop Summit - Early Bird Discount Expires Soon!
To: general@hadoop.apache.org, core-user@hadoop.apache.org, 
	hive-user@hadoop.apache.org, hbase-user <hbase-user@hadoop.apache.org>
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hadoop Fans, just a quick note about training options at the Hadoop
Summit. There are discounts expiring soon, so if you planned to
attend, or didn't know, we want to make sure you stay in the loop.

We're offering certification courses for developers and admins, as
well as an introduction to Hadoop. We'll also debut courses on Hive
and HBase because you asked for them.

You get the cost of your Summit registration ($100) off any of these
courses just by using the discount code included with your Summit
registration email confirmation, but if you register 45 days in
advance, you save another $100 per day (and Monday's courses are just
47 days out now!).

Intro to Hadoop (Monday): http://www.eventbrite.com/event/621620283/apache0511
Cloudera Desktop SDK (Monday):
http://www.eventbrite.com/event/621677454/apache0511
Hadoop for Developers + Certification (Wednesday-Thursday):
http://www.eventbrite.com/event/621640343/apache0511
Hadoop for Administrators + Certification (Wednesday-Thursday):
http://www.eventbrite.com/event/621643352/apache0511
Hive (Friday): http://www.eventbrite.com/event/621672439/apache0511
HBase (Friday): http://www.eventbrite.com/event/621670433/apache0511

You can see an overview here:
http://www.cloudera.com/hadoop-training/hadoop-summit-2010/

Cheers,
Christophe

-- 
get hadoop: cloudera.com/hadoop
online training: cloudera.com/hadoop-training
blog: cloudera.com/blog
twitter: twitter.com/cloudera

From general-return-1535-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed May 12 14:52:16 2010
Return-Path: <general-return-1535-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 8132 invoked from network); 12 May 2010 14:52:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 May 2010 14:52:15 -0000
Received: (qmail 82028 invoked by uid 500); 12 May 2010 14:52:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81962 invoked by uid 500); 12 May 2010 14:52:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81954 invoked by uid 99); 12 May 2010 14:52:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 May 2010 14:52:14 +0000
X-ASF-Spam-Status: No, hits=2.4 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jonathan.seidman@gmail.com designates 209.85.211.199 as permitted sender)
Received: from [209.85.211.199] (HELO mail-yw0-f199.google.com) (209.85.211.199)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 May 2010 14:52:07 +0000
Received: by ywh37 with SMTP id 37so4586860ywh.2
        for <general@hadoop.apache.org>; Wed, 12 May 2010 07:51:46 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=vC9COFOuVcxbNT7QaPQg6uBiZtJjoVwRBlDui5bJgL8=;
        b=uyS3BOMyw2hzUQf5vxLCYuqu7QOOpyTwz39o/JUgygamrkQsMO8akC7pZXvNT+aofG
         He34TXBYe2N40L2mSQn5XgcO7Ilz2izIGBbWkCnLgEyWscDKglSbqhz/3P0Cx2LSz2j/
         8hwOPATZzzBspF+SDB1lamDDyNJ9Oyimr52vw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=wgrFFwUVYrjKtzpre5VFNJNLX73MQMvBPFtgPS3OltAq+FRSE+7meIPyAzcYi5ENDQ
         zNxJiI9aZX4YlnfYv3W421HF/Y+Z7Ll3x2laUzHYI2ZBqkDNgh0CHC0JFGXxKxo+GWV4
         EnyrcE/5ub3YyR6a2iOmH8xkMe6EPQ0lxJmBs=
MIME-Version: 1.0
Received: by 10.100.245.40 with SMTP id s40mr4147421anh.137.1273675902272; 
	Wed, 12 May 2010 07:51:42 -0700 (PDT)
Received: by 10.100.154.10 with HTTP; Wed, 12 May 2010 07:51:41 -0700 (PDT)
Date: Wed, 12 May 2010 09:51:41 -0500
Message-ID: <AANLkTikQp2CODbZWuy_Cm3VM-beJ5eoS2NtxDVTQYqkR@mail.gmail.com>
Subject: [ANNOUNCE] Chicago Hadoop User Group Kick-off Meeting
From: Jonathan Seidman <jonathan.seidman@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e68dec4cb6ce4e048666c778

--0016e68dec4cb6ce4e048666c778
Content-Type: text/plain; charset=ISO-8859-1

The first meeting of the Chicago area Hadoop User Group will be help on
Wednesday, May 19th from 6:00-8:00PM. The meeting will be held at  Orbitz
Worldwide headquarters located in the Citigroup building at 500 West Madison
in downtown Chicago. Doug Cutting will give a talk by video conference, and
we'll also hear about using R with Amazon's Elastic MapReduce.

If you're planning to attend please send me a message by May 17th. You must
bring a photo ID to present to security. Please check in at the 3rd floor
visitor center as close to 6:00PM as possible and an Orbitz employee will
escort you up to the meeting.

Thanks! Hope to see you there.

Jonathan

--0016e68dec4cb6ce4e048666c778--

From general-return-1536-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed May 12 16:31:50 2010
Return-Path: <general-return-1536-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34617 invoked from network); 12 May 2010 16:31:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 May 2010 16:31:49 -0000
Received: (qmail 43079 invoked by uid 500); 12 May 2010 16:31:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 43009 invoked by uid 500); 12 May 2010 16:31:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 43001 invoked by uid 99); 12 May 2010 16:31:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 May 2010 16:31:48 +0000
X-ASF-Spam-Status: No, hits=0.1 required=10.0
	tests=AWL,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of chris.cooper@navteq.com designates 204.120.70.37 as permitted sender)
Received: from [204.120.70.37] (HELO xmailfargo.navteq.com) (204.120.70.37)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 May 2010 16:31:43 +0000
Received: from fargo-spamfw.navteq.com (navteq-spamfw.navteq.com [10.6.10.131])
	by xmailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id o4CGDHFC013903
	for <general@hadoop.apache.org>; Wed, 12 May 2010 11:13:17 -0500
Received: from imailfargo.navteq.com (localhost [127.0.0.1])
	by fargo-spamfw.navteq.com (Spam & Virus Firewall) with ESMTP id AC152491B125
	for <general@hadoop.apache.org>; Wed, 12 May 2010 11:31:21 -0500 (CDT)
Received: from imailfargo.navteq.com (imailfargo.navteq.com [10.6.10.130]) by fargo-spamfw.navteq.com with ESMTP id AteYSEEPdWBJInrd for <general@hadoop.apache.org>; Wed, 12 May 2010 11:31:21 -0500 (CDT)
Received: from hq-ex-ht02.ad.navteq.com ([10.8.222.172])
	by imailfargo.navteq.com (8.13.6/8.13.6) with ESMTP id o4CGVLFp018002
	for <general@hadoop.apache.org>; Wed, 12 May 2010 11:31:21 -0500
Received: from hq-ex-mb01.ad.navteq.com ([fe80::1dd0:ea36:7d35:db66]) by
 hq-ex-ht02.ad.navteq.com ([fe80::24f6:ca50:734d:df6e%14]) with mapi; Wed, 12
 May 2010 11:31:21 -0500
From: "Cooper, Chris" <chris.cooper@navteq.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 12 May 2010 11:31:20 -0500
Subject: RE: [ANNOUNCE] Chicago Hadoop User Group Kick-off Meeting
Thread-Topic: [ANNOUNCE] Chicago Hadoop User Group Kick-off Meeting
Thread-Index: Acrx4rcvDX3MqTfcRBSSIlUB/zCxvwADc9Ng
Message-ID: <76A09B97FD0B434ABCD9A75F688CF7E5038C8EB8@hq-ex-mb01.ad.navteq.com>
References: <AANLkTikQp2CODbZWuy_Cm3VM-beJ5eoS2NtxDVTQYqkR@mail.gmail.com>
In-Reply-To: <AANLkTikQp2CODbZWuy_Cm3VM-beJ5eoS2NtxDVTQYqkR@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-NAIMIME-Disclaimer: 1
X-NAIMIME-Modified: 1

SSBwbGFuIHRvIGF0dGVuZC4NCg0KLUNDDQoNCkNocmlzIENvb3Blcg0KQXJjaGl0ZWN0LKAgUiZE
DQpOQVZURVENCjQyNSBXZXN0IFJhbmRvbHBoIFN0cmVldA0KQ2hpY2FnbywgSUwgNjA2MDYNCihU
KSCgKzEgMzEyLTc4MC04OTkzDQooQymgICsxIDYzMC03NzYtNzg5OQ0Kd3d3Lm5hdnRlcS5jb20N
Cg0KDQotLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KRnJvbTogSm9uYXRoYW4gU2VpZG1hbiBb
bWFpbHRvOmpvbmF0aGFuLnNlaWRtYW5AZ21haWwuY29tXSANClNlbnQ6IFdlZG5lc2RheSwgTWF5
IDEyLCAyMDEwIDk6NTIgQU0NClRvOiBnZW5lcmFsQGhhZG9vcC5hcGFjaGUub3JnDQpTdWJqZWN0
OiBbQU5OT1VOQ0VdIENoaWNhZ28gSGFkb29wIFVzZXIgR3JvdXAgS2ljay1vZmYgTWVldGluZw0K
DQpUaGUgZmlyc3QgbWVldGluZyBvZiB0aGUgQ2hpY2FnbyBhcmVhIEhhZG9vcCBVc2VyIEdyb3Vw
IHdpbGwgYmUgaGVscCBvbg0KV2VkbmVzZGF5LCBNYXkgMTl0aCBmcm9tIDY6MDAtODowMFBNLiBU
aGUgbWVldGluZyB3aWxsIGJlIGhlbGQgYXQgIE9yYml0eg0KV29ybGR3aWRlIGhlYWRxdWFydGVy
cyBsb2NhdGVkIGluIHRoZSBDaXRpZ3JvdXAgYnVpbGRpbmcgYXQgNTAwIFdlc3QgTWFkaXNvbg0K
aW4gZG93bnRvd24gQ2hpY2Fnby4gRG91ZyBDdXR0aW5nIHdpbGwgZ2l2ZSBhIHRhbGsgYnkgdmlk
ZW8gY29uZmVyZW5jZSwgYW5kDQp3ZSdsbCBhbHNvIGhlYXIgYWJvdXQgdXNpbmcgUiB3aXRoIEFt
YXpvbidzIEVsYXN0aWMgTWFwUmVkdWNlLg0KDQpJZiB5b3UncmUgcGxhbm5pbmcgdG8gYXR0ZW5k
IHBsZWFzZSBzZW5kIG1lIGEgbWVzc2FnZSBieSBNYXkgMTd0aC4gWW91IG11c3QNCmJyaW5nIGEg
cGhvdG8gSUQgdG8gcHJlc2VudCB0byBzZWN1cml0eS4gUGxlYXNlIGNoZWNrIGluIGF0IHRoZSAz
cmQgZmxvb3INCnZpc2l0b3IgY2VudGVyIGFzIGNsb3NlIHRvIDY6MDBQTSBhcyBwb3NzaWJsZSBh
bmQgYW4gT3JiaXR6IGVtcGxveWVlIHdpbGwNCmVzY29ydCB5b3UgdXAgdG8gdGhlIG1lZXRpbmcu
DQoNClRoYW5rcyEgSG9wZSB0byBzZWUgeW91IHRoZXJlLg0KDQpKb25hdGhhbg0KDQoNClRoZSBp
bmZvcm1hdGlvbiBjb250YWluZWQgaW4gdGhpcyBjb21tdW5pY2F0aW9uIG1heSBiZSBDT05GSURF
TlRJQUwgYW5kIGlzIGludGVuZGVkIG9ubHkgZm9yIHRoZSB1c2Ugb2YgdGhlIHJlY2lwaWVudChz
KSBuYW1lZCBhYm92ZS4gIElmIHlvdSBhcmUgbm90IHRoZSBpbnRlbmRlZCByZWNpcGllbnQsIHlv
dSBhcmUgaGVyZWJ5IG5vdGlmaWVkIHRoYXQgYW55IGRpc3NlbWluYXRpb24sIGRpc3RyaWJ1dGlv
biwgb3IgY29weWluZyBvZiB0aGlzIGNvbW11bmljYXRpb24sIG9yIGFueSBvZiBpdHMgY29udGVu
dHMsIGlzIHN0cmljdGx5IHByb2hpYml0ZWQuICBJZiB5b3UgaGF2ZSByZWNlaXZlZCB0aGlzIGNv
bW11bmljYXRpb24gaW4gZXJyb3IsIHBsZWFzZSBub3RpZnkgdGhlIHNlbmRlciBhbmQgZGVsZXRl
L2Rlc3Ryb3kgdGhlIG9yaWdpbmFsIG1lc3NhZ2UgYW5kIGFueSBjb3B5IG9mIGl0IGZyb20geW91
ciBjb21wdXRlciBvciBwYXBlciBmaWxlcy4NCg==

From general-return-1537-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed May 12 18:38:14 2010
Return-Path: <general-return-1537-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78332 invoked from network); 12 May 2010 18:38:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 May 2010 18:38:13 -0000
Received: (qmail 34851 invoked by uid 500); 12 May 2010 18:38:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34809 invoked by uid 500); 12 May 2010 18:38:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34801 invoked by uid 99); 12 May 2010 18:38:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 May 2010 18:38:12 +0000
X-ASF-Spam-Status: No, hits=2.3 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jonathan.seidman@gmail.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 May 2010 18:38:07 +0000
Received: by gyf1 with SMTP id 1so175929gyf.35
        for <general@hadoop.apache.org>; Wed, 12 May 2010 11:37:46 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=10JNZ62/SJf6EuhHgMTIjd5D2Jjo6NewhuPdaFGBQyM=;
        b=Lb6Y1MKuibp5gzWPq2bbr5KtNuz/eW3jwWGRV/sc4z9YEwncaPakh6RJbL04OhDOc3
         Rz1M1JApw0HIBCX3a0wCaFwGkFto7Ys5AxyjNTPZBYMPd6rTMui+BTLEr2/Gqbazoc7d
         ntG7EvPL1IWO4i6UtXDR4GtQiwmUW2OWkJdpE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=rDkcOQog7UCJUfRyFjsOfr/gVdtg44QoJVRVtsq8rPKptsUMclKQbD58G5f209ya6z
         4aaCOq1h8wSfPogCYHW8Aww73+RR44ntnLuNXPCjlx6vVk8P2gGLMwoUnRLsQSQRxt43
         87BhGU4+4i5JM14zpBH3e22Tfr8Nh0yw0jHo4=
MIME-Version: 1.0
Received: by 10.101.28.24 with SMTP id f24mr4869046anj.253.1273689460275; Wed, 
	12 May 2010 11:37:40 -0700 (PDT)
Received: by 10.100.154.10 with HTTP; Wed, 12 May 2010 11:37:40 -0700 (PDT)
In-Reply-To: <AANLkTikQp2CODbZWuy_Cm3VM-beJ5eoS2NtxDVTQYqkR@mail.gmail.com>
References: <AANLkTikQp2CODbZWuy_Cm3VM-beJ5eoS2NtxDVTQYqkR@mail.gmail.com>
Date: Wed, 12 May 2010 13:37:40 -0500
Message-ID: <AANLkTimk_zO1LUiHZcaAh1V3lryBvI3NiHQF4pKIzHbk@mail.gmail.com>
Subject: Re: [ANNOUNCE] Chicago Hadoop User Group Kick-off Meeting
From: Jonathan Seidman <jonathan.seidman@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636b2aef3d5804a048669ef26

--001636b2aef3d5804a048669ef26
Content-Type: text/plain; charset=ISO-8859-1

Just want to add a couple of thanks:

To Cloudera for providing sponsorship
Orbitz Worldwide for hosting
and Mike Segel of MSCC for providing the meetup group as well as
organizational help.

The meetup group can be found at:

http://www.meetup.com/Chicago-area-Hadoop-User-Group-CHUG/

On Wed, May 12, 2010 at 9:51 AM, Jonathan Seidman <
jonathan.seidman@gmail.com> wrote:

> The first meeting of the Chicago area Hadoop User Group will be help on
> Wednesday, May 19th from 6:00-8:00PM. The meeting will be held at  Orbitz
> Worldwide headquarters located in the Citigroup building at 500 West Madison
> in downtown Chicago. Doug Cutting will give a talk by video conference, and
> we'll also hear about using R with Amazon's Elastic MapReduce.
>
> If you're planning to attend please send me a message by May 17th. You must
> bring a photo ID to present to security. Please check in at the 3rd floor
> visitor center as close to 6:00PM as possible and an Orbitz employee will
> escort you up to the meeting.
>
> Thanks! Hope to see you there.
>
> Jonathan
>

--001636b2aef3d5804a048669ef26--

From general-return-1538-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun May 16 02:05:38 2010
Return-Path: <general-return-1538-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16211 invoked from network); 16 May 2010 02:05:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 May 2010 02:05:38 -0000
Received: (qmail 51849 invoked by uid 500); 16 May 2010 02:05:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51779 invoked by uid 500); 16 May 2010 02:05:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51771 invoked by uid 99); 16 May 2010 02:05:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 16 May 2010 02:05:36 +0000
X-ASF-Spam-Status: No, hits=2.1 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of renatoj.marroquin@gmail.com designates 209.85.211.199 as permitted sender)
Received: from [209.85.211.199] (HELO mail-yw0-f199.google.com) (209.85.211.199)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 16 May 2010 02:05:31 +0000
Received: by ywh37 with SMTP id 37so2748420ywh.2
        for <general@hadoop.apache.org>; Sat, 15 May 2010 19:05:10 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=wtaHIz0ZKJt61EPNNdsTQkzSRfGJHVMi8byOWHhv7Zo=;
        b=Awwm5VnZYol+gJ2oLzK2buEaY5LvNtaFfs0ZaxI+Kf7C8UU2I/rtkMxuKNzIfiSD85
         oigHvkVDDe286S1iS9M9exzPSBAoolcoa6yHGqY8Ci1/qi/VIOl6uTeT3ZGji4aO03QV
         ihd7qDFhgz5eHwyMHFRlAexPqxc54VXASOy9Q=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=GgRvavQRdb5x+zshzZ3MnKwu9VK9dmkhYDd8f+tQYlV4NrtlQWt/0vroF5Hir/mPCS
         aHMjRPxf/An8ZX4HboToa6NgEkpyM4yEnswDf9LUMUe0XvVAEBkIRvX60VJFCXI571XT
         nXNVFSLuVMOTBVJBKK5O6IoJO5DLhepHM+md4=
MIME-Version: 1.0
Received: by 10.101.193.40 with SMTP id v40mr4053970anp.204.1273975510479; 
	Sat, 15 May 2010 19:05:10 -0700 (PDT)
Received: by 10.100.91.7 with HTTP; Sat, 15 May 2010 19:05:10 -0700 (PDT)
In-Reply-To: <AANLkTikdR0ri6Y32uPlNZp6z3y6gHwwRBShtvNTYoGiL@mail.gmail.com>
References: <y2p44be54d01005050829gf270a41du98eaf93b0fe2f070@mail.gmail.com>
	 <j2hd6d7c4411005051804p40b2b598ud593f984c09d779@mail.gmail.com>
	 <AANLkTinYM84xSEs30UT5RU1K7UjguNuztpy-yQjr2pXZ@mail.gmail.com>
	 <AANLkTinqlIpg7UNX8DaWCfqoZghXwgh5dHcwHpYM6asl@mail.gmail.com>
	 <AANLkTimMjEYBtO5dwkHJ5oPW49P2eVngVRBdLFyxCc19@mail.gmail.com>
	 <AANLkTikKkO_gpjZ-Z6hu0TZjb3pAY0qVJDQ1eHmlEO8E@mail.gmail.com>
	 <AANLkTikdR0ri6Y32uPlNZp6z3y6gHwwRBShtvNTYoGiL@mail.gmail.com>
Date: Sat, 15 May 2010 21:05:10 -0500
Message-ID: <AANLkTinN68Da-cSO0e54nHbcHMXKC9LqdiNVkTCLTsPI@mail.gmail.com>
Subject: Re: Hadoop Data Sharing
From: =?ISO-8859-1?Q?Renato_Marroqu=EDn_Mogrovejo?= <renatoj.marroquin@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636c92a8ac136dc0486ac8924

--001636c92a8ac136dc0486ac8924
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Thanks for your replies. Yeah I have had to restructure a part of my code
but it is all good now.
Thanks again for your suggestions.

Renato M.

2010/5/11 Jay Booth <jaybooth@gmail.com>

> Probably the most direct route to get your desired result is to save
> the objects to either a SequenceFile or plain text file on DFS.  Then
> in the configure() section of your mapreduce jobs, you open the file
> on DFS, stream contents into a local variable and refer to it as  you
> need to.  Either way, you'll need some sort of serialization via
> Writable or plain text.
>
> On Tue, May 11, 2010 at 4:19 PM, Renato Marroqu=EDn Mogrovejo
> <renatoj.marroquin@gmail.com> wrote:
> > Hi Aaron,
> >
> > The thing is that I had a data structure that is saved into a vector, a=
nd
> > this vector needs to be available for my MapReduce jobs while iterating=
.
> So
> > would you think it would a good and easy way to serialize this objects?
> It's
> > a vector that each node contains another user define data structure.
> Maybe I
> > will try to do it first just using files, and see how the throughput
> goes.
> > Hey do you know where I can find some examples of serializing objects f=
or
> > Hadoop to save them into SequenceFiles?
> > Thanks in advance.
> >
> > Renato M.
> >
> >
> > 2010/5/11 Aaron Kimball <aaron@cloudera.com>
> >
> >> Perhaps this is guidance in the area you were hoping for: If your data
> is
> >> in
> >> objects that implement the interface 'Writable', then you can use the
> >> SequenceFileOutputFormat and SequenceFileInputFormat to store your
> >> intermediate data in binary form in disk-backed files called
> SequenceFiles.
> >> The serialization will happen through the write() and readFields()
> methods
> >> of your objects, which will automatically be called by the
> >> OutputFormat/InputFormat as they move through the system. So your
> >> subsequent
> >> MR pass will receive objects back in the same form as they were emitte=
d.
> >> This is a considerably better idea (from both a throughput and a sanit=
y
> >> perspective) in a chained MapReduce job.
> >>
> >> - Aaron
> >>
> >> On Tue, May 11, 2010 at 10:31 AM, Aaron Kimball <aaron@cloudera.com>
> >> wrote:
> >>
> >> > What objects are you referring to? I'm not sure I understand your
> >> question.
> >> > - Aaron
> >> >
> >> >
> >> > On Tue, May 11, 2010 at 6:38 AM, Renato Marroqu=EDn Mogrovejo <
> >> > renatoj.marroquin@gmail.com> wrote:
> >> >
> >> >> Thanks Aaron! I was thinking the same after doing some reading.
> >> >> Man what about serialize the objects? Would you think that is a goo=
d
> >> idea?
> >> >> Thanks again.
> >> >>
> >> >> Renato M.
> >> >>
> >> >>
> >> >> 2010/5/5 Aaron Kimball <aaron@cloudera.com>
> >> >>
> >> >> > Renato,
> >> >> >
> >> >> > In general if you need to perform a multi-pass MapReduce workflow=
,
> >> each
> >> >> > pass
> >> >> > materializes its output to files. The subsequent pass then reads
> those
> >> >> same
> >> >> > files back in as input. This allows the workflow to start at the
> last
> >> >> > "checkpoint" if it gets interrupted. There is no persistent
> in-memory
> >> >> > distributed storage feature in Hadoop that would allow a MapReduc=
e
> job
> >> >> to
> >> >> > post results to memory for consumption by a subsequent job.
> >> >> >
> >> >> > So you would just read your initial data from /input, and write
> your
> >> >> > interim
> >> >> > results to /iteration0. Then the next pass reads from /iteration0
> and
> >> >> > writes
> >> >> > to /iteration1, etc..
> >> >> >
> >> >> > If your data is reasonably small and you think it could fit in
> memory
> >> >> > somewhere, then you could experiment with using other distributed
> >> >> key-value
> >> >> > stores (memcached[b], hbase, cassandra, etc..) to hold intermedia=
te
> >> >> > results.
> >> >> > But this will require some integration work on your part.
> >> >> > - Aaron
> >> >> >
> >> >> > On Wed, May 5, 2010 at 8:29 AM, Renato Marroqu=EDn Mogrovejo <
> >> >> > renatoj.marroquin@gmail.com> wrote:
> >> >> >
> >> >> > > Hi everyone, I have recently started to play around with hadoop=
,
> but
> >> I
> >> >> am
> >> >> > > getting some into some "design" problems.
> >> >> > > I need to make a loop to execute the same job several times, an=
d
> in
> >> >> each
> >> >> > > iteration get the processed values (not using a file because I
> would
> >> >> need
> >> >> > > to
> >> >> > > read it). I was using an static vector in my main class (the on=
e
> >> that
> >> >> > > iterates and executes the job in each iteration) to retrieve
> those
> >> >> > values,
> >> >> > > and it did work while I was using a standalone mode. Now I trie=
d
> to
> >> >> test
> >> >> > it
> >> >> > > on a pseudo-distributed manner and obviously is not working.
> >> >> > > Any suggestions, please???
> >> >> > >
> >> >> > > Thanks in advance,
> >> >> > >
> >> >> > >
> >> >> > > Renato M.
> >> >> > >
> >> >> >
> >> >>
> >> >
> >> >
> >>
> >
>

--001636c92a8ac136dc0486ac8924--

From general-return-1539-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 17 16:21:03 2010
Return-Path: <general-return-1539-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9231 invoked from network); 17 May 2010 16:21:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 May 2010 16:21:03 -0000
Received: (qmail 25955 invoked by uid 500); 17 May 2010 16:21:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25489 invoked by uid 500); 17 May 2010 16:21:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25469 invoked by uid 99); 17 May 2010 16:21:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 May 2010 16:21:00 +0000
X-ASF-Spam-Status: No, hits=-283.6 required=10.0
	tests=AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 17 May 2010 16:20:59 +0000
Received: (qmail 8916 invoked by uid 99); 17 May 2010 16:20:38 -0000
Received: from localhost.apache.org (HELO [10.0.0.119]) (127.0.0.1)
  (smtp-auth username phunt, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 May 2010 16:20:38 +0000
Message-ID: <4BF16CD9.4020205@apache.org>
Date: Mon, 17 May 2010 09:20:41 -0700
From: Patrick Hunt <phunt@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4
MIME-Version: 1.0
To: announce@apache.org, 
 "zookeeper-user@hadoop.apache.org" <zookeeper-user@hadoop.apache.org>,
 "zookeeper-dev@hadoop.apache.org" <zookeeper-dev@hadoop.apache.org>, 
 general@hadoop.apache.org, private@hadoop.apache.org
Subject: [ANNOUNCE] Apache ZooKeeper 3.3.1
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

The Apache ZooKeeper team is proud to announce Apache ZooKeeper version 
3.3.1

ZooKeeper is a high-performance coordination service for distributed 
applications. It exposes common services - such as naming, configuration 
management, synchronization, and group services - in a simple interface 
so you don't have to write them from scratch. You can use it 
off-the-shelf to implement consensus, group management, leader election, 
and presence protocols. And you can build on it for your own, specific 
needs.

For ZooKeeper release details and downloads, visit:
http://hadoop.apache.org/zookeeper/releases.html

ZooKeeper 3.3.1 Release Notes are at:
http://hadoop.apache.org/zookeeper/docs/r3.3.1/releasenotes.html

Regards,

The ZooKeeper Team




From general-return-1540-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 17 16:58:14 2010
Return-Path: <general-return-1540-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16984 invoked from network); 17 May 2010 16:58:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 May 2010 16:58:14 -0000
Received: (qmail 16458 invoked by uid 500); 17 May 2010 16:58:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16352 invoked by uid 500); 17 May 2010 16:58:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16344 invoked by uid 99); 17 May 2010 16:58:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 May 2010 16:58:12 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com designates 209.85.221.199 as permitted sender)
Received: from [209.85.221.199] (HELO mail-qy0-f199.google.com) (209.85.221.199)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 May 2010 16:58:06 +0000
Received: by qyk37 with SMTP id 37so8190840qyk.22
        for <general@hadoop.apache.org>; Mon, 17 May 2010 09:57:44 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=SV9RD1zo7PfE9uZREp2fBcR15WJBmVpq5iW/E2K3BE8=;
        b=H9ZYI/sbpl7LQahGXKsf17owbK3w+ksepiXOk8+CWJiEIM5l9A2b3HxgQgiZWLBGl6
         IqiNoXBwJXB4a86eMMQ6WKpongtL/mvkhayViy66bWbZpfO1clvV1BVR42O9TMjiKOHZ
         Xc1dwUb5983vRaJm6Q9ku4SKmRBIU6G9wRlWM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=GQ9sCCWWLCbDzBbxCSMQ1NX0HmMhL8WMjrKAzjo0I6ovPjl0r99slh0nTqsGs1itB2
         2OCS6Jlzek0QkMc3uixRjq2Gl13jja+Pa5GEGhk8zQ8lo1jb44dYAqzi+S3YvI8D7zvt
         Y57L4QCBQvV+D/TQQQJndI0znYtdkU9VgC1Qo=
MIME-Version: 1.0
Received: by 10.224.59.100 with SMTP id k36mr2979624qah.139.1274115463476; 
	Mon, 17 May 2010 09:57:43 -0700 (PDT)
Received: by 10.229.2.13 with HTTP; Mon, 17 May 2010 09:57:43 -0700 (PDT)
Date: Mon, 17 May 2010 09:57:43 -0700
Message-ID: <AANLkTinQJfe_9Tjm2J4VXQ4kQMsYMib0ai9mC2Dkbz2O@mail.gmail.com>
Subject: datanode goes down, maybe due to Unexpected problem in creating 
	temporary file
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00c09fa21a7a9a77220486cd1f2b

--00c09fa21a7a9a77220486cd1f2b
Content-Type: text/plain; charset=ISO-8859-1

Hi,
We use CDH2 hadoop-0.20.2+228 which crashed on datanode smsrv10.ciq.com

I found this in datanode log:

2010-05-15 07:37:35,955 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_926027507678171558_3620 src: /10.32.56.170:53378 dest: /
10.32.56.171:50010
2010-05-15 07:37:35,956 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
blk_926027507678171558_3620 received exception java.io.IOException:
Unexpected problem in creating temporary file for
blk_926027507678171558_3620.  File
/home/hadoop/m2m_3.0.x/3.0.trunk.39-270238/data/hadoop-data/dfs/data/tmp/blk_926027507678171558
should not be present, but is.
2010-05-15 07:37:35,956 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
10.32.56.171:50010,
storageID=DS-1723593983-10.32.56.171-50010-1273792791835, infoPort=50075,
ipcPort=50020):DataXceiver
java.io.IOException: Unexpected problem in creating temporary file for
blk_926027507678171558_3620.  File
/home/hadoop/m2m_3.0.x/3.0.trunk.39-270238/data/hadoop-data/dfs/data/tmp/blk_926027507678171558
should not be present, but is.
        at
org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:398)
        at
org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:376)
        at
org.apache.hadoop.hdfs.server.datanode.FSDataset.createTmpFile(FSDataset.java:1133)
        at
org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:1022)
        at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:98)
        at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:259)
        at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
        at java.lang.Thread.run(Thread.java:619)

Can someone provide some clue ?

Thanks

--00c09fa21a7a9a77220486cd1f2b--

From general-return-1541-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 17 18:44:02 2010
Return-Path: <general-return-1541-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56986 invoked from network); 17 May 2010 18:44:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 May 2010 18:44:02 -0000
Received: (qmail 16734 invoked by uid 500); 17 May 2010 18:44:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16685 invoked by uid 500); 17 May 2010 18:44:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16677 invoked by uid 99); 17 May 2010 18:44:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 May 2010 18:44:01 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_NEUTRAL,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 May 2010 18:43:54 +0000
Received: by iwn3 with SMTP id 3so636379iwn.35
        for <general@hadoop.apache.org>; Mon, 17 May 2010 11:43:32 -0700 (PDT)
Received: by 10.231.178.132 with SMTP id bm4mr621415ibb.62.1274121812596; Mon, 
	17 May 2010 11:43:32 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.161.70 with HTTP; Mon, 17 May 2010 11:43:12 -0700 (PDT)
In-Reply-To: <AANLkTinQJfe_9Tjm2J4VXQ4kQMsYMib0ai9mC2Dkbz2O@mail.gmail.com>
References: <AANLkTinQJfe_9Tjm2J4VXQ4kQMsYMib0ai9mC2Dkbz2O@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 17 May 2010 11:43:12 -0700
Message-ID: <AANLkTikgaltFN1-usbM8_9AT3dKmN5EroOkP2YrZz761@mail.gmail.com>
Subject: Re: datanode goes down, maybe due to Unexpected problem in creating 
	temporary file
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502bf630a5b130486ce9a50
X-Virus-Checked: Checked by ClamAV on apache.org

--00504502bf630a5b130486ce9a50
Content-Type: text/plain; charset=ISO-8859-1

Hi Ted,

Can you please grep your NN and DN logs for blk_926027507678171558 and
pastebin the results?

-Todd

On Mon, May 17, 2010 at 9:57 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> Hi,
> We use CDH2 hadoop-0.20.2+228 which crashed on datanode smsrv10.ciq.com
>
> I found this in datanode log:
>
> 2010-05-15 07:37:35,955 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_926027507678171558_3620 src: /10.32.56.170:53378 dest: /
> 10.32.56.171:50010
> 2010-05-15 07:37:35,956 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
> blk_926027507678171558_3620 received exception java.io.IOException:
> Unexpected problem in creating temporary file for
> blk_926027507678171558_3620.  File
>
> /home/hadoop/m2m_3.0.x/3.0.trunk.39-270238/data/hadoop-data/dfs/data/tmp/blk_926027507678171558
> should not be present, but is.
> 2010-05-15 07:37:35,956 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
> 10.32.56.171:50010,
> storageID=DS-1723593983-10.32.56.171-50010-1273792791835, infoPort=50075,
> ipcPort=50020):DataXceiver
> java.io.IOException: Unexpected problem in creating temporary file for
> blk_926027507678171558_3620.  File
>
> /home/hadoop/m2m_3.0.x/3.0.trunk.39-270238/data/hadoop-data/dfs/data/tmp/blk_926027507678171558
> should not be present, but is.
>        at
>
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:398)
>        at
>
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:376)
>        at
>
> org.apache.hadoop.hdfs.server.datanode.FSDataset.createTmpFile(FSDataset.java:1133)
>        at
>
> org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:1022)
>        at
>
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:98)
>        at
>
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:259)
>        at
>
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
>        at java.lang.Thread.run(Thread.java:619)
>
> Can someone provide some clue ?
>
> Thanks
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--00504502bf630a5b130486ce9a50--

From general-return-1542-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 17 19:40:33 2010
Return-Path: <general-return-1542-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6750 invoked from network); 17 May 2010 19:40:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 May 2010 19:40:33 -0000
Received: (qmail 26211 invoked by uid 500); 17 May 2010 19:40:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26140 invoked by uid 500); 17 May 2010 19:40:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26132 invoked by uid 99); 17 May 2010 19:40:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 May 2010 19:40:31 +0000
X-ASF-Spam-Status: No, hits=1.6 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com designates 209.85.221.199 as permitted sender)
Received: from [209.85.221.199] (HELO mail-qy0-f199.google.com) (209.85.221.199)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 May 2010 19:40:26 +0000
Received: by qyk37 with SMTP id 37so8415535qyk.22
        for <general@hadoop.apache.org>; Mon, 17 May 2010 12:40:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=npZAw9KQBSirYJlCXlqmUU68xvT5MlEEDL6cAg10UbU=;
        b=f7m1LJGuqvxWv1t1ogSvo2yeAWBp3pQcGwocuVSB46GhJKcKdtc+YnF3lRUnesWioW
         bNnqZQRXCodjYL+80t6U7VEo0/2p5zQUxj1sYyEaEuQWlECtG0treKQqZLDUrGTqrBwy
         ZF9emQC/DW1itVhFDU2ziGhtA6drkrc14s6bA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Qni+lu+y+y1v8zAcpmIaj1f2DBU+xSjYyyN0ZYI4aorOtiHwpnuAknM/Yigak4tknD
         0kOwY5sDQxVx8MBUSTF1Oo8RyPW0SaWX1avAI1Vlj5UbZ+s4E3f8yNYBd5VOVQJK74uU
         WH0a9pjiiFA02olkEwwflOE8rFw1hHQ8cgaVw=
MIME-Version: 1.0
Received: by 10.224.65.158 with SMTP id j30mr3080071qai.390.1274125205035; 
	Mon, 17 May 2010 12:40:05 -0700 (PDT)
Received: by 10.229.57.80 with HTTP; Mon, 17 May 2010 12:40:04 -0700 (PDT)
In-Reply-To: <AANLkTikgaltFN1-usbM8_9AT3dKmN5EroOkP2YrZz761@mail.gmail.com>
References: <AANLkTinQJfe_9Tjm2J4VXQ4kQMsYMib0ai9mC2Dkbz2O@mail.gmail.com>
	 <AANLkTikgaltFN1-usbM8_9AT3dKmN5EroOkP2YrZz761@mail.gmail.com>
Date: Mon, 17 May 2010 12:40:04 -0700
Message-ID: <AANLkTimpAlbaWCxrte0cRfDCWjD-A0KwdxVOaKwybHh0@mail.gmail.com>
Subject: Re: datanode goes down, maybe due to Unexpected problem in creating 
	temporary file
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00c09f9721e23edf3c0486cf64b8

--00c09f9721e23edf3c0486cf64b8
Content-Type: text/plain; charset=ISO-8859-1

That blk doesn't appear in NameNode log.

For datanode,
2010-05-15 00:09:31,023 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_926027507678171558_3620 src: /10.32.56.170:49172 dest: /
10.32.56.171:50010
2010-05-15 00:09:31,024 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
blk_926027507678171558_3620 received exception java.io.IOException:
Unexpected problem in creating temporary file for
blk_926027507678171558_3620.  File
/home/hadoop/m2m_3.0.x/3.0.trunk.39-270238/data/hadoop-data/dfs/data/tmp/blk_926027507678171558
should not be present, but is.
2010-05-15 00:09:31,024 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
blk_-5814095875968936685_2910 received exception java.io.IOException:
Unexpected problem in creating temporary file for
blk_-5814095875968936685_2910.  File
/home/hadoop/m2m_3.0.x/3.0.trunk.39-270238/data/hadoop-data/dfs/data/tmp/blk_-5814095875968936685
should not be present, but is.
2010-05-15 00:09:31,025 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
10.32.56.171:50010,
storageID=DS-1723593983-10.32.56.171-50010-1273792791835, infoPort=50075,
ipcPort=50020):DataXceiver
java.io.IOException: Unexpected problem in creating temporary file for
blk_926027507678171558_3620.  File
/home/hadoop/m2m_3.0.x/3.0.trunk.39-270238/data/hadoop-data/dfs/data/tmp/blk_926027507678171558
should not be present, but is.
        at
org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:398)
        at
org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:376)
        at
org.apache.hadoop.hdfs.server.datanode.FSDataset.createTmpFile(FSDataset.java:1133)
        at
org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:1022)
        at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:98)
        at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:259)
        at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
        at java.lang.Thread.run(Thread.java:619)
2010-05-15 00:09:31,025 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
10.32.56.171:50010,
storageID=DS-1723593983-10.32.56.171-50010-1273792791835, infoPort=50075,
ipcPort=50020):DataXceiver

2010-05-15 00:19:28,334 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_926027507678171558_3620 src: /10.32.56.170:36887 dest: /
10.32.56.171:50010
2010-05-15 00:19:28,334 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
blk_926027507678171558_3620 received exception java.io.IOException:
Unexpected problem in creating temporary file for
blk_926027507678171558_3620.  File
/home/hadoop/m2m_3.0.x/3.0.trunk.39-270238/data/hadoop-data/dfs/data/tmp/blk_926027507678171558
should not be present, but is.
2010-05-15 00:19:28,334 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
10.32.56.171:50010,
storageID=DS-1723593983-10.32.56.171-50010-1273792791835, infoPort=50075,
ipcPort=50020):DataXceiver
java.io.IOException: Unexpected problem in creating temporary file for
blk_926027507678171558_3620.  File
/home/hadoop/m2m_3.0.x/3.0.trunk.39-270238/data/hadoop-data/dfs/data/tmp/blk_926027507678171558
should not be present, but is.
        at
org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:398)
        at
org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:376)
        at
org.apache.hadoop.hdfs.server.datanode.FSDataset.createTmpFile(FSDataset.java:1133)
        at
org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:1022)
        at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:98)
        at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:259)
        at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
        at java.lang.Thread.run(Thread.java:619)
2010-05-15 00:29:25,635 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_926027507678171558_3620 src: /10.32.56.170:34823 dest: /
10.32.56.171:50010

On Mon, May 17, 2010 at 11:43 AM, Todd Lipcon <todd@cloudera.com> wrote:

> Hi Ted,
>
> Can you please grep your NN and DN logs for blk_926027507678171558 and
> pastebin the results?
>
> -Todd
>
> On Mon, May 17, 2010 at 9:57 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > Hi,
> > We use CDH2 hadoop-0.20.2+228 which crashed on datanode smsrv10.ciq.com
> >
> > I found this in datanode log:
> >
> > 2010-05-15 07:37:35,955 INFO
> > org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> > blk_926027507678171558_3620 src: /10.32.56.170:53378 dest: /
> > 10.32.56.171:50010
> > 2010-05-15 07:37:35,956 INFO
> > org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
> > blk_926027507678171558_3620 received exception java.io.IOException:
> > Unexpected problem in creating temporary file for
> > blk_926027507678171558_3620.  File
> >
> >
> /home/hadoop/m2m_3.0.x/3.0.trunk.39-270238/data/hadoop-data/dfs/data/tmp/blk_926027507678171558
> > should not be present, but is.
> > 2010-05-15 07:37:35,956 ERROR
> > org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
> > 10.32.56.171:50010,
> > storageID=DS-1723593983-10.32.56.171-50010-1273792791835, infoPort=50075,
> > ipcPort=50020):DataXceiver
> > java.io.IOException: Unexpected problem in creating temporary file for
> > blk_926027507678171558_3620.  File
> >
> >
> /home/hadoop/m2m_3.0.x/3.0.trunk.39-270238/data/hadoop-data/dfs/data/tmp/blk_926027507678171558
> > should not be present, but is.
> >        at
> >
> >
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:398)
> >        at
> >
> >
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:376)
> >        at
> >
> >
> org.apache.hadoop.hdfs.server.datanode.FSDataset.createTmpFile(FSDataset.java:1133)
> >        at
> >
> >
> org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:1022)
> >        at
> >
> >
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:98)
> >        at
> >
> >
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:259)
> >        at
> >
> >
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
> >        at java.lang.Thread.run(Thread.java:619)
> >
> > Can someone provide some clue ?
> >
> > Thanks
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

--00c09f9721e23edf3c0486cf64b8--

From general-return-1543-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 17 19:44:35 2010
Return-Path: <general-return-1543-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9467 invoked from network); 17 May 2010 19:44:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 May 2010 19:44:34 -0000
Received: (qmail 32654 invoked by uid 500); 17 May 2010 19:44:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32593 invoked by uid 500); 17 May 2010 19:44:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32585 invoked by uid 99); 17 May 2010 19:44:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 May 2010 19:44:33 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_NEUTRAL,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.214.176] (HELO mail-iw0-f176.google.com) (209.85.214.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 May 2010 19:44:27 +0000
Received: by iwn3 with SMTP id 3so682918iwn.35
        for <general@hadoop.apache.org>; Mon, 17 May 2010 12:44:05 -0700 (PDT)
Received: by 10.231.125.129 with SMTP id y1mr659460ibr.96.1274125445264; Mon, 
	17 May 2010 12:44:05 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.161.70 with HTTP; Mon, 17 May 2010 12:43:45 -0700 (PDT)
In-Reply-To: <AANLkTimpAlbaWCxrte0cRfDCWjD-A0KwdxVOaKwybHh0@mail.gmail.com>
References: <AANLkTinQJfe_9Tjm2J4VXQ4kQMsYMib0ai9mC2Dkbz2O@mail.gmail.com> 
	<AANLkTikgaltFN1-usbM8_9AT3dKmN5EroOkP2YrZz761@mail.gmail.com> 
	<AANLkTimpAlbaWCxrte0cRfDCWjD-A0KwdxVOaKwybHh0@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Mon, 17 May 2010 12:43:45 -0700
Message-ID: <AANLkTimcY9jHfSV-AkRbD-lZjNaLko4fdrK1kXYaJ9sT@mail.gmail.com>
Subject: Re: datanode goes down, maybe due to Unexpected problem in creating 
	temporary file
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64653be9076280486cf7291
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e64653be9076280486cf7291
Content-Type: text/plain; charset=ISO-8859-1

Have you disabled the statechange log on the NN? This block has to be in
there.

Also, are you by any chance running with append enabled on unpatched 0.20?

-Todd

On Mon, May 17, 2010 at 12:40 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> That blk doesn't appear in NameNode log.
>
> For datanode,
> 2010-05-15 00:09:31,023 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_926027507678171558_3620 src: /10.32.56.170:49172 dest: /
> 10.32.56.171:50010
> 2010-05-15 00:09:31,024 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
> blk_926027507678171558_3620 received exception java.io.IOException:
> Unexpected problem in creating temporary file for
> blk_926027507678171558_3620.  File
>
> /home/hadoop/m2m_3.0.x/3.0.trunk.39-270238/data/hadoop-data/dfs/data/tmp/blk_926027507678171558
> should not be present, but is.
> 2010-05-15 00:09:31,024 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
> blk_-5814095875968936685_2910 received exception java.io.IOException:
> Unexpected problem in creating temporary file for
> blk_-5814095875968936685_2910.  File
>
> /home/hadoop/m2m_3.0.x/3.0.trunk.39-270238/data/hadoop-data/dfs/data/tmp/blk_-5814095875968936685
> should not be present, but is.
> 2010-05-15 00:09:31,025 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
> 10.32.56.171:50010,
> storageID=DS-1723593983-10.32.56.171-50010-1273792791835, infoPort=50075,
> ipcPort=50020):DataXceiver
> java.io.IOException: Unexpected problem in creating temporary file for
> blk_926027507678171558_3620.  File
>
> /home/hadoop/m2m_3.0.x/3.0.trunk.39-270238/data/hadoop-data/dfs/data/tmp/blk_926027507678171558
> should not be present, but is.
>        at
>
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:398)
>        at
>
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:376)
>        at
>
> org.apache.hadoop.hdfs.server.datanode.FSDataset.createTmpFile(FSDataset.java:1133)
>        at
>
> org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:1022)
>        at
>
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:98)
>        at
>
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:259)
>        at
>
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
>        at java.lang.Thread.run(Thread.java:619)
> 2010-05-15 00:09:31,025 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
> 10.32.56.171:50010,
> storageID=DS-1723593983-10.32.56.171-50010-1273792791835, infoPort=50075,
> ipcPort=50020):DataXceiver
>
> 2010-05-15 00:19:28,334 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_926027507678171558_3620 src: /10.32.56.170:36887 dest: /
> 10.32.56.171:50010
> 2010-05-15 00:19:28,334 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
> blk_926027507678171558_3620 received exception java.io.IOException:
> Unexpected problem in creating temporary file for
> blk_926027507678171558_3620.  File
>
> /home/hadoop/m2m_3.0.x/3.0.trunk.39-270238/data/hadoop-data/dfs/data/tmp/blk_926027507678171558
> should not be present, but is.
> 2010-05-15 00:19:28,334 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
> 10.32.56.171:50010,
> storageID=DS-1723593983-10.32.56.171-50010-1273792791835, infoPort=50075,
> ipcPort=50020):DataXceiver
> java.io.IOException: Unexpected problem in creating temporary file for
> blk_926027507678171558_3620.  File
>
> /home/hadoop/m2m_3.0.x/3.0.trunk.39-270238/data/hadoop-data/dfs/data/tmp/blk_926027507678171558
> should not be present, but is.
>        at
>
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:398)
>        at
>
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:376)
>        at
>
> org.apache.hadoop.hdfs.server.datanode.FSDataset.createTmpFile(FSDataset.java:1133)
>        at
>
> org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:1022)
>        at
>
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:98)
>        at
>
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:259)
>        at
>
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
>        at java.lang.Thread.run(Thread.java:619)
> 2010-05-15 00:29:25,635 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_926027507678171558_3620 src: /10.32.56.170:34823 dest: /
> 10.32.56.171:50010
>
> On Mon, May 17, 2010 at 11:43 AM, Todd Lipcon <todd@cloudera.com> wrote:
>
> > Hi Ted,
> >
> > Can you please grep your NN and DN logs for blk_926027507678171558 and
> > pastebin the results?
> >
> > -Todd
> >
> > On Mon, May 17, 2010 at 9:57 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> >
> > > Hi,
> > > We use CDH2 hadoop-0.20.2+228 which crashed on datanode
> smsrv10.ciq.com
> > >
> > > I found this in datanode log:
> > >
> > > 2010-05-15 07:37:35,955 INFO
> > > org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> > > blk_926027507678171558_3620 src: /10.32.56.170:53378 dest: /
> > > 10.32.56.171:50010
> > > 2010-05-15 07:37:35,956 INFO
> > > org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
> > > blk_926027507678171558_3620 received exception java.io.IOException:
> > > Unexpected problem in creating temporary file for
> > > blk_926027507678171558_3620.  File
> > >
> > >
> >
> /home/hadoop/m2m_3.0.x/3.0.trunk.39-270238/data/hadoop-data/dfs/data/tmp/blk_926027507678171558
> > > should not be present, but is.
> > > 2010-05-15 07:37:35,956 ERROR
> > > org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
> > > 10.32.56.171:50010,
> > > storageID=DS-1723593983-10.32.56.171-50010-1273792791835,
> infoPort=50075,
> > > ipcPort=50020):DataXceiver
> > > java.io.IOException: Unexpected problem in creating temporary file for
> > > blk_926027507678171558_3620.  File
> > >
> > >
> >
> /home/hadoop/m2m_3.0.x/3.0.trunk.39-270238/data/hadoop-data/dfs/data/tmp/blk_926027507678171558
> > > should not be present, but is.
> > >        at
> > >
> > >
> >
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:398)
> > >        at
> > >
> > >
> >
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:376)
> > >        at
> > >
> > >
> >
> org.apache.hadoop.hdfs.server.datanode.FSDataset.createTmpFile(FSDataset.java:1133)
> > >        at
> > >
> > >
> >
> org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:1022)
> > >        at
> > >
> > >
> >
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:98)
> > >        at
> > >
> > >
> >
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:259)
> > >        at
> > >
> > >
> >
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
> > >        at java.lang.Thread.run(Thread.java:619)
> > >
> > > Can someone provide some clue ?
> > >
> > > Thanks
> > >
> >
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--0016e64653be9076280486cf7291--

From general-return-1544-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 18 01:20:01 2010
Return-Path: <general-return-1544-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70899 invoked from network); 18 May 2010 01:20:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 May 2010 01:20:01 -0000
Received: (qmail 19352 invoked by uid 500); 18 May 2010 01:20:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19306 invoked by uid 500); 18 May 2010 01:20:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19297 invoked by uid 99); 18 May 2010 01:20:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 May 2010 01:20:00 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates 209.85.221.199 as permitted sender)
Received: from [209.85.221.199] (HELO mail-qy0-f199.google.com) (209.85.221.199)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 May 2010 01:19:53 +0000
Received: by qyk37 with SMTP id 37so8876681qyk.22
        for <general@hadoop.apache.org>; Mon, 17 May 2010 18:19:32 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=NskYMV/SCZwljOBocb+eUlCd30ENmKgWYqz7ijFajAQ=;
        b=qSqlLNS4YqGE0QaHM063hoMUQ7DkV4L8ra/5rnEkiGHG8IQz17olzPXlbdV20cCboV
         8rpeqiERFoauDuVNI4sm99WE8d3LY2dSG+ctU6LLdwqxPJhlpL2F3CiXUla2XV7nmFnB
         PIJmh05100AfxJ/2sq7QND+otIaahh/fl5hCM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=PM1V4/hs13L9gEdRObVqgfXWo+c51U9lqcIP3VWD2UUj39+DZ+nVidHaJRUWi/a5Hf
         KswNwHKlzxRCGh8NQF7K8qsD5sByb0StYiQG1JpIQGJzSlUa5se7sSBeLtwPFmcwx0yy
         INEew08zYrx3SrS55sXrBizRcRjtbMriJAM3I=
MIME-Version: 1.0
Received: by 10.224.86.207 with SMTP id t15mr3312321qal.227.1274145570737; 
	Mon, 17 May 2010 18:19:30 -0700 (PDT)
Received: by 10.229.57.80 with HTTP; Mon, 17 May 2010 18:19:30 -0700 (PDT)
In-Reply-To: <AANLkTimpAlbaWCxrte0cRfDCWjD-A0KwdxVOaKwybHh0@mail.gmail.com>
References: <AANLkTinQJfe_9Tjm2J4VXQ4kQMsYMib0ai9mC2Dkbz2O@mail.gmail.com>
	 <AANLkTikgaltFN1-usbM8_9AT3dKmN5EroOkP2YrZz761@mail.gmail.com>
	 <AANLkTimpAlbaWCxrte0cRfDCWjD-A0KwdxVOaKwybHh0@mail.gmail.com>
Date: Mon, 17 May 2010 18:19:30 -0700
Message-ID: <AANLkTimW2a6S5v9jiHOaidJzQfVtHiaSrNQlN-uscGcA@mail.gmail.com>
Subject: Re: datanode goes down, maybe due to Unexpected problem in creating 
	temporary file
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00c09f9db1be22cfe80486d422c7
X-Virus-Checked: Checked by ClamAV on apache.org

--00c09f9db1be22cfe80486d422c7
Content-Type: text/plain; charset=ISO-8859-1

[sjc1-qa-certiq1.sjc1:hadoop 25763]df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                     303346048 281730720   6206176  98% /

In hadoop-hadoop-jobtracker- log repeated for over 50 minutes:
2010-05-18 00:57:38,451 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
up the system directory
2010-05-18 00:57:38,452 INFO org.apache.hadoop.mapred.JobTracker: problem
cleaning system directory: hdfs://
sjc1-qa-certiq1.sjc1.carrieriq.com:9000/tmp/hadoop/mapred/system
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
/tmp/hadoop/mapred/system. Name node is in safe mode.
The reported blocks 793 needs additional 2 blocks to reach the threshold
0.9990 of total blocks 796. Safe mode will be turned off automatically.
    at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1711)
    at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1691)
    at
org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:565)
    at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:966)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:962)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:960)

    at org.apache.hadoop.ipc.Client.call(Client.java:740)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
    at $Proxy4.delete(Unknown Source)
    at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
    at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
    at $Proxy4.delete(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:586)
    at
org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1710)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:188)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:180)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3735)

In datanode log two entries were written after 793 blocks getting processed:
2010-05-17 16:56:22,135 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_5132109619501575576_2882
2010-05-17 17:06:22,245 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_-4733675789688469676_1685
2010-05-17 17:16:22,383 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_5502091787302369348_2833
2010-05-17 17:26:22,507 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_-6935269688635441525_1778
2010-05-17 17:36:22,709 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_8901766802925964345_1804
2010-05-17 17:41:43,929 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: *BlockReport of 793 blocks
got processed in 107 msecs*
2010-05-17 17:46:23,065 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_-8536166246203803199_2384
2010-05-17 17:56:22,445 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_3069643184898592926_1667

I wonder what prevented from the last two blocks from being recognized by
Namenode ?

Here is the log from the other datanode:

2010-05-17 14:38:10,838 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
10.32.56.170:50010,
storageID=DS-1307210060-10.32.56.170-50010-1274132290796, infoPort=50075,
ipcPort=50020)In DataNode.run, data =
FSDataset{dirpath='/home/hadoop/m2m_3.0.x/3.0.trunk.39-270238/data/hadoop-data/dfs/data/current'}
2010-05-17 14:38:10,839 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: using BLOCKREPORT_INTERVAL
of 3600000msec Initial delay: 0msec
2010-05-17 14:38:10,887 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 0 blocks got
processed in 5 msecs
2010-05-17 14:38:10,888 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Starting Periodic block
scanner.
2010-05-17 15:15:51,037 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 0 blocks got
processed in 3 msecs

Currently our QA testing is blocked.

Can someone provide help ?

Thanks

On Mon, May 17, 2010 at 12:40 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> That blk doesn't appear in NameNode log.
>
> For datanode,
> 2010-05-15 00:09:31,023 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_926027507678171558_3620 src: /10.32.56.170:49172 dest: /
> 10.32.56.171:50010
> 2010-05-15 00:09:31,024 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
> blk_926027507678171558_3620 received exception java.io.IOException:
> Unexpected problem in creating temporary file for
> blk_926027507678171558_3620.  File
> /home/hadoop/m2m_3.0.x/3.0.trunk.39-270238/data/hadoop-data/dfs/data/tmp/blk_926027507678171558
> should not be present, but is.
> 2010-05-15 00:09:31,024 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
> blk_-5814095875968936685_2910 received exception java.io.IOException:
> Unexpected problem in creating temporary file for
> blk_-5814095875968936685_2910.  File
> /home/hadoop/m2m_3.0.x/3.0.trunk.39-270238/data/hadoop-data/dfs/data/tmp/blk_-5814095875968936685
> should not be present, but is.
> 2010-05-15 00:09:31,025 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
> 10.32.56.171:50010,
> storageID=DS-1723593983-10.32.56.171-50010-1273792791835, infoPort=50075,
> ipcPort=50020):DataXceiver
>
> java.io.IOException: Unexpected problem in creating temporary file for
> blk_926027507678171558_3620.  File
> /home/hadoop/m2m_3.0.x/3.0.trunk.39-270238/data/hadoop-data/dfs/data/tmp/blk_926027507678171558
> should not be present, but is.
>         at
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:398)
>         at
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:376)
>         at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.createTmpFile(FSDataset.java:1133)
>         at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:1022)
>         at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:98)
>         at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:259)
>         at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
>         at java.lang.Thread.run(Thread.java:619)
> 2010-05-15 00:09:31,025 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
> 10.32.56.171:50010,
> storageID=DS-1723593983-10.32.56.171-50010-1273792791835, infoPort=50075,
> ipcPort=50020):DataXceiver
>
> 2010-05-15 00:19:28,334 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_926027507678171558_3620 src: /10.32.56.170:36887 dest: /
> 10.32.56.171:50010
> 2010-05-15 00:19:28,334 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
> blk_926027507678171558_3620 received exception java.io.IOException:
> Unexpected problem in creating temporary file for
> blk_926027507678171558_3620.  File
> /home/hadoop/m2m_3.0.x/3.0.trunk.39-270238/data/hadoop-data/dfs/data/tmp/blk_926027507678171558
> should not be present, but is.
> 2010-05-15 00:19:28,334 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
> 10.32.56.171:50010,
> storageID=DS-1723593983-10.32.56.171-50010-1273792791835, infoPort=50075,
> ipcPort=50020):DataXceiver
>
> java.io.IOException: Unexpected problem in creating temporary file for
> blk_926027507678171558_3620.  File
> /home/hadoop/m2m_3.0.x/3.0.trunk.39-270238/data/hadoop-data/dfs/data/tmp/blk_926027507678171558
> should not be present, but is.
>         at
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:398)
>         at
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:376)
>         at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.createTmpFile(FSDataset.java:1133)
>         at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:1022)
>         at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:98)
>         at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:259)
>         at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
>         at java.lang.Thread.run(Thread.java:619)
> 2010-05-15 00:29:25,635 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_926027507678171558_3620 src: /10.32.56.170:34823 dest: /
> 10.32.56.171:50010
>
>
> On Mon, May 17, 2010 at 11:43 AM, Todd Lipcon <todd@cloudera.com> wrote:
>
>> Hi Ted,
>>
>> Can you please grep your NN and DN logs for blk_926027507678171558 and
>> pastebin the results?
>>
>> -Todd
>>
>> On Mon, May 17, 2010 at 9:57 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>>
>> > Hi,
>> > We use CDH2 hadoop-0.20.2+228 which crashed on datanode smsrv10.ciq.com
>> >
>> > I found this in datanode log:
>> >
>> > 2010-05-15 07:37:35,955 INFO
>> > org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
>> > blk_926027507678171558_3620 src: /10.32.56.170:53378 dest: /
>> > 10.32.56.171:50010
>> > 2010-05-15 07:37:35,956 INFO
>> > org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock
>> > blk_926027507678171558_3620 received exception java.io.IOException:
>> > Unexpected problem in creating temporary file for
>> > blk_926027507678171558_3620.  File
>> >
>> >
>> /home/hadoop/m2m_3.0.x/3.0.trunk.39-270238/data/hadoop-data/dfs/data/tmp/blk_926027507678171558
>> > should not be present, but is.
>> > 2010-05-15 07:37:35,956 ERROR
>> > org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
>> > 10.32.56.171:50010,
>> > storageID=DS-1723593983-10.32.56.171-50010-1273792791835,
>> infoPort=50075,
>> > ipcPort=50020):DataXceiver
>> > java.io.IOException: Unexpected problem in creating temporary file for
>> > blk_926027507678171558_3620.  File
>> >
>> >
>> /home/hadoop/m2m_3.0.x/3.0.trunk.39-270238/data/hadoop-data/dfs/data/tmp/blk_926027507678171558
>> > should not be present, but is.
>> >        at
>> >
>> >
>> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:398)
>> >        at
>> >
>> >
>> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:376)
>> >        at
>> >
>> >
>> org.apache.hadoop.hdfs.server.datanode.FSDataset.createTmpFile(FSDataset.java:1133)
>> >        at
>> >
>> >
>> org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:1022)
>> >        at
>> >
>> >
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:98)
>> >        at
>> >
>> >
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:259)
>> >        at
>> >
>> >
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
>> >        at java.lang.Thread.run(Thread.java:619)
>> >
>> > Can someone provide some clue ?
>> >
>> > Thanks
>> >
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>
>

--00c09f9db1be22cfe80486d422c7--

From general-return-1545-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 18 23:17:30 2010
Return-Path: <general-return-1545-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6031 invoked from network); 18 May 2010 23:17:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 May 2010 23:17:30 -0000
Received: (qmail 96130 invoked by uid 500); 18 May 2010 23:17:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96042 invoked by uid 500); 18 May 2010 23:17:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96034 invoked by uid 99); 18 May 2010 23:17:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 May 2010 23:17:28 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=AWL,HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of Deepika.Khera@avg.com designates 193.85.188.248 as permitted sender)
Received: from [193.85.188.248] (HELO ms.grisoft.cz) (193.85.188.248)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 May 2010 23:17:21 +0000
Received: from phobos.cz.avg.com (unknown [192.168.200.160])
	(using TLSv1 with cipher AES128-SHA (128/128 bits))
	(No client certificate requested)
	by ms.grisoft.cz (Postfix) with ESMTP id F3F453815D
	for <general@hadoop.apache.org>; Wed, 19 May 2010 01:16:58 +0200 (CEST)
Received: from miranda.us.avg.com (10.32.34.13) by phobos.cz.avg.com
 (192.168.200.160) with Microsoft SMTP Server (TLS) id 8.2.254.0; Wed, 19 May
 2010 01:16:58 +0200
Received: from miranda.us.avg.com ([10.32.34.13]) by miranda ([10.32.34.13])
 with mapi; Tue, 18 May 2010 19:16:57 -0400
From: Deepika Khera <Deepika.Khera@avg.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 18 May 2010 19:13:34 -0400
Subject: Splitting support for bzip2 format
Thread-Topic: Splitting support for bzip2 format
Thread-Index: Acr2371iGU5aoj1YSjuGvD/mHAEcIw==
Message-ID: <DA852C02397DBC4EBFAA1B7192FBD15061DC83F6EA@miranda>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-cr-puzzleid: {23E6B7C5-44E7-4511-B2C8-F8255A36B9D1}
x-cr-hashedpuzzle: ZAY= AZIS BEt9 BOfv COZo CeSs Di2I EP54 FwNk Gxff HBkB
 H9UO IR7Q I/fG LQqE
 M6OP;1;ZwBlAG4AZQByAGEAbABAAGgAYQBkAG8AbwBwAC4AYQBwAGEAYwBoAGUALgBvAHIAZwA=;Sosha1_v1;7;{23E6B7C5-44E7-4511-B2C8-F8255A36B9D1};ZABlAGUAcABpAGsAYQAuAGsAaABlAHIAYQBAAGEAdgBnAC4AYwBvAG0A;Tue,
 18 May 2010 23:13:34
 GMT;UwBwAGwAaQB0AHQAaQBuAGcAIABzAHUAcABwAG8AcgB0ACAAZgBvAHIAIABiAHoAaQBwADIAIABmAG8AcgBtAGEAdAA=
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_DA852C02397DBC4EBFAA1B7192FBD15061DC83F6EAmiranda_"
MIME-Version: 1.0

--_000_DA852C02397DBC4EBFAA1B7192FBD15061DC83F6EAmiranda_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi,

Do we have a patch to support splitting with the bzip2 format for the curre=
nt stable version 0.20.2 ? Refer to JIRA below :

https://issues.apache.org/jira/browse/HADOOP-4012

Thanks,
Deepika

--_000_DA852C02397DBC4EBFAA1B7192FBD15061DC83F6EAmiranda_--

From general-return-1546-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu May 20 17:35:00 2010
Return-Path: <general-return-1546-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 13745 invoked from network); 20 May 2010 17:35:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 May 2010 17:35:00 -0000
Received: (qmail 86117 invoked by uid 500); 20 May 2010 17:34:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86039 invoked by uid 500); 20 May 2010 17:34:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86031 invoked by uid 99); 20 May 2010 17:34:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 May 2010 17:34:58 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=TO_NO_BRKTS_DIRECT
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 20 May 2010 17:34:55 +0000
Received: (qmail 13677 invoked by uid 99); 20 May 2010 17:34:33 -0000
Received: from localhost.apache.org (HELO [10.0.0.113]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 May 2010 17:34:33 +0000
Message-ID: <4BF572A9.80108@apache.org>
Date: Thu, 20 May 2010 10:34:33 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.9) Gecko/20100423 Thunderbird/3.0.4
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Hadoop takes Big Data beyond Java =?windows-1252?Q?=95_The_R?=
 =?windows-1252?Q?egister?=
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

I'd like folks to know that I am misquoted in this article:

http://www.theregister.co.uk/2010/05/17/doug_cutting_hadoop/

First, I don't know what interview with me they're referring to.  I've 
done no recent interviews with The Register.

They assert that I said that Yahoo!'s work on security has delayed a 1.0 
release.  I have never said that and it is not something I believe.  I 
did answer a question about 1.0 at yesterday's Chicago Hadoop Users 
Group meeting.  I said that I personally felt we could call 0.20 1.0, 
but that the majority of contributors felt we should hold off, and that 
individuals don't make such decisions at Apache.  I did not relate this 
to security or Y! in any way.  That part seems to either be the 
misunderstanding or fabrication of El Reg.

The spin of the article seems to be created by the author rather than 
anything I said.

Doug

From general-return-1547-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu May 20 20:56:24 2010
Return-Path: <general-return-1547-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47031 invoked from network); 20 May 2010 20:56:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 May 2010 20:56:24 -0000
Received: (qmail 1609 invoked by uid 500); 20 May 2010 20:56:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1517 invoked by uid 500); 20 May 2010 20:56:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1509 invoked by uid 99); 20 May 2010 20:56:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 May 2010 20:56:22 +0000
X-ASF-Spam-Status: No, hits=0.1 required=10.0
	tests=AWL,HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of Deepika.Khera@avg.com designates 193.85.188.248 as permitted sender)
Received: from [193.85.188.248] (HELO ms.grisoft.cz) (193.85.188.248)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 May 2010 20:56:16 +0000
Received: from deimos.cz.avg.com (unknown [192.168.200.161])
	(using TLSv1 with cipher AES128-SHA (128/128 bits))
	(No client certificate requested)
	by ms.grisoft.cz (Postfix) with ESMTP id E806A38189
	for <general@hadoop.apache.org>; Thu, 20 May 2010 22:55:53 +0200 (CEST)
Received: from miranda.us.avg.com (10.32.34.13) by deimos.cz.avg.com
 (192.168.200.161) with Microsoft SMTP Server (TLS) id 8.2.254.0; Thu, 20 May
 2010 22:55:53 +0200
Received: from miranda.us.avg.com ([10.32.34.13]) by miranda ([10.32.34.13])
 with mapi; Thu, 20 May 2010 16:55:50 -0400
From: Deepika Khera <Deepika.Khera@avg.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 20 May 2010 16:52:26 -0400
Subject: Splitting support for bz2 format
Thread-Topic: Splitting support for bz2 format
Thread-Index: Acr4Xlq9EG5Aw3igQYSe/wj1m7nFhQ==
Message-ID: <DA852C02397DBC4EBFAA1B7192FBD15061DC83F7D4@miranda>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-cr-puzzleid: {9401D4F1-E630-48BE-AAF5-B16D1D4F454C}
x-cr-hashedpuzzle: AWEd AZsc BJTk B8Gw CF6a COjQ C2Bg DwCg EUld EbUQ Etnj
 Ezoe F0h3 Gy6Q Hxb2
 IdpV;1;ZwBlAG4AZQByAGEAbABAAGgAYQBkAG8AbwBwAC4AYQBwAGEAYwBoAGUALgBvAHIAZwA=;Sosha1_v1;7;{9401D4F1-E630-48BE-AAF5-B16D1D4F454C};ZABlAGUAcABpAGsAYQAuAGsAaABlAHIAYQBAAGEAdgBnAC4AYwBvAG0A;Thu,
 20 May 2010 20:52:26
 GMT;UwBwAGwAaQB0AHQAaQBuAGcAIABzAHUAcABwAG8AcgB0ACAAZgBvAHIAIABiAHoAMgAgAGYAbwByAG0AYQB0AA==
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_DA852C02397DBC4EBFAA1B7192FBD15061DC83F7D4miranda_"
MIME-Version: 1.0

--_000_DA852C02397DBC4EBFAA1B7192FBD15061DC83F7D4miranda_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi,



Do we have a patch to support splitting with the bzip2 format for the curre=
nt stable version 0.20.2 ? Please refer to JIRA below :



https://issues.apache.org/jira/browse/HADOOP-4012



Thanks,

Deepika


--_000_DA852C02397DBC4EBFAA1B7192FBD15061DC83F7D4miranda_--

From general-return-1548-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri May 21 20:42:59 2010
Return-Path: <general-return-1548-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 15019 invoked from network); 21 May 2010 20:42:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 May 2010 20:42:58 -0000
Received: (qmail 36852 invoked by uid 500); 21 May 2010 20:42:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36748 invoked by uid 500); 21 May 2010 20:42:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36740 invoked by uid 99); 21 May 2010 20:42:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 May 2010 20:42:56 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.212.176] (HELO mail-px0-f176.google.com) (209.85.212.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 May 2010 20:42:50 +0000
Received: by pxi10 with SMTP id 10so850589pxi.35
        for <general@hadoop.apache.org>; Fri, 21 May 2010 13:42:28 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.142.120.26 with SMTP id s26mr1458787wfc.141.1274474548306; 
	Fri, 21 May 2010 13:42:28 -0700 (PDT)
Received: by 10.143.16.21 with HTTP; Fri, 21 May 2010 13:42:28 -0700 (PDT)
Date: Fri, 21 May 2010 13:42:28 -0700
Message-ID: <AANLkTimLB-YNWnMwa93YL8Lpg9hJLxrCKbYL3YUr6de8@mail.gmail.com>
Subject: [DISCUSSION] Proposal for making core Hadoop changes
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

As HDFS and MapReduce have matured the cost and complexity of
introducing features has grown. Each new feature has to consider
interactions with a growing set of existing features, a growing user
base (upgrades, backwards compatibility) and additional use cases
(more and more projects now build on them). At the same time we don't
want the high bar for contribution to unnecessarily hinder new
development and releases.

Many projects at a similar stage address this by adopting a more
formal way to describe, socialize and shepherd enhancements to their
platforms. Today, new features are often discussed via an umbrella
jira, which may have an attached design document. There are a number
of issues with this approach. The design documents vary in format and
quality, and are often reviewed by a limited audience. They aren't
version controlled. Sometimes the proposal is only partially
specified. Jiras are often ignored. Understanding a proposal and it's
implications through a series of threads in the jira comments is
difficult. It's hard for contributors and users to find these
top-level jiras and follow their status.

I'd like to propose that core Hadoop adopts something similar to
Python's PEP (Python Enhancement Proposal) [1]. A "HEP" would be a
single primary mechanism for proposing new features, incorporating
community feedback, and recording decisions. The author of the HEP
would be responsible for building consensus and moving the feature
forward. Similarly, some subset of the community would be responsible
for reviewing HEPs in a timely manner and identifying missing pieces
in the proposal. Discussion would occur before patches showed up on
jira. People interested in the core Hadoop roadmap could keep an eye
on the HEPs without the overhead of following jira traffic.

Why base this on the PEP? The format has proven useful to a
substantial existing project, and I think the workflow is not too
heavy-weight, and well-suited to a community such as ours. That being
said, we could discuss other models (eg Java's JSR).

Before we get into specifics, is this something the community would
like to adopt in some form? Does adapting the PEP and its workflow to
our projects, community and bylaws seem reasonable?

Thanks,
Eli

1. http://www.python.org/dev/peps/pep-0001

From general-return-1549-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat May 22 02:26:59 2010
Return-Path: <general-return-1549-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 3366 invoked from network); 22 May 2010 02:26:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 May 2010 02:26:58 -0000
Received: (qmail 90600 invoked by uid 500); 22 May 2010 02:26:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90541 invoked by uid 500); 22 May 2010 02:26:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90533 invoked by uid 99); 22 May 2010 02:26:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 May 2010 02:26:57 +0000
X-ASF-Spam-Status: No, hits=0.5 required=10.0
	tests=AWL,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 May 2010 02:26:50 +0000
Received: by pwi4 with SMTP id 4so980527pwi.35
        for <general@hadoop.apache.org>; Fri, 21 May 2010 19:26:30 -0700 (PDT)
Received: by 10.140.248.7 with SMTP id v7mr1766272rvh.252.1274495190015;
        Fri, 21 May 2010 19:26:30 -0700 (PDT)
Received: from [192.168.42.239] ([38.102.147.105])
        by mx.google.com with ESMTPS id k17sm1433618rvh.17.2010.05.21.19.26.28
        (version=SSLv3 cipher=RC4-MD5);
        Fri, 21 May 2010 19:26:28 -0700 (PDT)
Message-ID: <4BF740CF.4080905@cloudera.com>
Date: Fri, 21 May 2010 19:26:23 -0700
From: Amr Awadallah <aaa@cloudera.com>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.9) Gecko/20100317 Lightning/1.0b1 Thunderbird/3.0.4 ThunderBrowse/3.2.8.1
MIME-Version: 1.0
To: general@hadoop.apache.org
CC: Eli Collins <eli@cloudera.com>
Subject: Re: [DISCUSSION] Proposal for making core Hadoop changes
References: <AANLkTimLB-YNWnMwa93YL8Lpg9hJLxrCKbYL3YUr6de8@mail.gmail.com>
In-Reply-To: <AANLkTimLB-YNWnMwa93YL8Lpg9hJLxrCKbYL3YUr6de8@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

 > Does adapting the PEP and its workflow to our projects, community and 
bylaws seem reasonable?

+1

On 5/21/2010 1:42 PM, Eli Collins wrote:
> As HDFS and MapReduce have matured the cost and complexity of
> introducing features has grown. Each new feature has to consider
> interactions with a growing set of existing features, a growing user
> base (upgrades, backwards compatibility) and additional use cases
> (more and more projects now build on them). At the same time we don't
> want the high bar for contribution to unnecessarily hinder new
> development and releases.
>
> Many projects at a similar stage address this by adopting a more
> formal way to describe, socialize and shepherd enhancements to their
> platforms. Today, new features are often discussed via an umbrella
> jira, which may have an attached design document. There are a number
> of issues with this approach. The design documents vary in format and
> quality, and are often reviewed by a limited audience. They aren't
> version controlled. Sometimes the proposal is only partially
> specified. Jiras are often ignored. Understanding a proposal and it's
> implications through a series of threads in the jira comments is
> difficult. It's hard for contributors and users to find these
> top-level jiras and follow their status.
>
> I'd like to propose that core Hadoop adopts something similar to
> Python's PEP (Python Enhancement Proposal) [1]. A "HEP" would be a
> single primary mechanism for proposing new features, incorporating
> community feedback, and recording decisions. The author of the HEP
> would be responsible for building consensus and moving the feature
> forward. Similarly, some subset of the community would be responsible
> for reviewing HEPs in a timely manner and identifying missing pieces
> in the proposal. Discussion would occur before patches showed up on
> jira. People interested in the core Hadoop roadmap could keep an eye
> on the HEPs without the overhead of following jira traffic.
>
> Why base this on the PEP? The format has proven useful to a
> substantial existing project, and I think the workflow is not too
> heavy-weight, and well-suited to a community such as ours. That being
> said, we could discuss other models (eg Java's JSR).
>
> Before we get into specifics, is this something the community would
> like to adopt in some form? Does adapting the PEP and its workflow to
> our projects, community and bylaws seem reasonable?
>
> Thanks,
> Eli
>
> 1. http://www.python.org/dev/peps/pep-0001
>    

From general-return-1550-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat May 22 02:38:42 2010
Return-Path: <general-return-1550-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 5001 invoked from network); 22 May 2010 02:38:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 May 2010 02:38:42 -0000
Received: (qmail 93601 invoked by uid 500); 22 May 2010 02:38:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93505 invoked by uid 500); 22 May 2010 02:38:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93497 invoked by uid 99); 22 May 2010 02:38:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 May 2010 02:38:40 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates 209.85.221.184 as permitted sender)
Received: from [209.85.221.184] (HELO mail-qy0-f184.google.com) (209.85.221.184)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 May 2010 02:38:33 +0000
Received: by qyk14 with SMTP id 14so219607qyk.21
        for <general@hadoop.apache.org>; Fri, 21 May 2010 19:38:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=3d44V9v1VJoGxWuTVfQ/NR5wKKpoRjfnsAnyCnZavtE=;
        b=jkxfZ8k1Db7wTm/kEfgaZ8dN/mv7PUM5o3NN03RSawY649IGw3aicYDPNb4nV9BtY8
         bOdb3T+KiMZf9DXeygbbSY702KV1yfb9Dso++EdMWnDDCuTsQD6CPPgoVIVq6JBIGIwe
         mVAnf5OkjSe/2HvwI4qg1aT2Jjq/84iqqkCo4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=vE5wLq2ZEODbY94bUIIopRF5tg2rrhIT2luR5MIQI0hLqB61NdbIMEgHtwqiRQIUtQ
         FarYgMecllTV8DyThdfhepk7RMsvA+cTvo017Trk9CMfW3EDdcH8l51TBjqUT5Sd8Lqv
         /nXLrSypBfSNgDHSney1CY2bEl8VXB74sGTi4=
MIME-Version: 1.0
Received: by 10.229.193.9 with SMTP id ds9mr609690qcb.124.1274495892396; Fri, 
	21 May 2010 19:38:12 -0700 (PDT)
Received: by 10.229.57.80 with HTTP; Fri, 21 May 2010 19:38:12 -0700 (PDT)
Date: Fri, 21 May 2010 19:38:12 -0700
Message-ID: <AANLkTin2-c9Tnhw1iwAukdlF7JdLnJI5eFUWbsqh5etx@mail.gmail.com>
Subject: tasktracker JMX query
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016363b8240ef1c87048725b291
X-Virus-Checked: Checked by ClamAV on apache.org

--0016363b8240ef1c87048725b291
Content-Type: text/plain; charset=ISO-8859-1

Hi,
We monitor number of open connections to task tracker:
    public static IMonitorQuery RPC_NUM_OPEN_CONNECTIONS = new JmxQuery("",
HADOOP_TASKTRACKER_WILDCARD(), "NumOpenConnections", ReportType.PERIODIC,
RPC_STATISTICS_QUERY);

The above worked with 0.19.1 hadoop but when it doesn't work with 0.20.1:

 INFO [2010-05-21 19:26:06] (ExecUtil.java:232) - Exception in thread
"agentQueryingTimer" java.lang.IllegalStateException: no object name for
wildcard name 'hadoop.dfs:service=TaskTracker,*' found at
service:jmx:rmi:///jndi/rmi://sjc9-flash-grid01.ciq.com:11004/jmxrmi

I am wondering how we should query against 0.20.1

Thanks

--0016363b8240ef1c87048725b291--

From general-return-1551-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat May 22 20:09:16 2010
Return-Path: <general-return-1551-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 14681 invoked from network); 22 May 2010 20:09:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 May 2010 20:09:16 -0000
Received: (qmail 74787 invoked by uid 500); 22 May 2010 20:09:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74702 invoked by uid 500); 22 May 2010 20:09:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74694 invoked by uid 99); 22 May 2010 20:09:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 May 2010 20:09:14 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ndaley@mac.com designates 17.148.16.104 as permitted sender)
Received: from [17.148.16.104] (HELO asmtpout029.mac.com) (17.148.16.104)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 May 2010 20:09:07 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; charset=us-ascii; format=flowed; delsp=yes
Received: from [10.45.223.96]
 (166-205-138-060.mobile.mymmode.com [166.205.138.60])
 by asmtp029.mac.com (Sun Java(tm) System Messaging Server 6.3-8.01 (built Dec
 16 2008; 32bit)) with ESMTPSA id <0L2U00FNW7YE5V90@asmtp029.mac.com> for
 general@hadoop.apache.org; Sat, 22 May 2010 13:08:45 -0700 (PDT)
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0
 reason=mlx engine=6.0.2-1004200000 definitions=main-1005220152
X-Proofpoint-Virus-Version: vendor=fsecure
 engine=1.12.8161:2.4.5,1.2.40,4.0.166
 definitions=2010-05-21_02:2010-02-06,2010-05-21,2010-05-22 signatures=0
Message-id: <66BC29ED-35C1-4916-BDFE-C768810C6EA6@mac.com>
From: Nigel Daley <ndaley@mac.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-reply-to: <AANLkTimLB-YNWnMwa93YL8Lpg9hJLxrCKbYL3YUr6de8@mail.gmail.com>
X-Mailer: iPhone Mail (7E18)
Subject: Re: [DISCUSSION] Proposal for making core Hadoop changes
Date: Sat, 22 May 2010 13:08:32 -0700
References: <AANLkTimLB-YNWnMwa93YL8Lpg9hJLxrCKbYL3YUr6de8@mail.gmail.com>
X-Virus-Checked: Checked by ClamAV on apache.org

+1 to better process around feature enhancements. I like that PEP also  
includes process enhancements too.

For comparison, anyone have a references to similar processes?

Cheers,
Nige

On May 21, 2010, at 1:42 PM, Eli Collins <eli@cloudera.com> wrote:

> As HDFS and MapReduce have matured the cost and complexity of
> introducing features has grown. Each new feature has to consider
> interactions with a growing set of existing features, a growing user
> base (upgrades, backwards compatibility) and additional use cases
> (more and more projects now build on them). At the same time we don't
> want the high bar for contribution to unnecessarily hinder new
> development and releases.
>
> Many projects at a similar stage address this by adopting a more
> formal way to describe, socialize and shepherd enhancements to their
> platforms. Today, new features are often discussed via an umbrella
> jira, which may have an attached design document. There are a number
> of issues with this approach. The design documents vary in format and
> quality, and are often reviewed by a limited audience. They aren't
> version controlled. Sometimes the proposal is only partially
> specified. Jiras are often ignored. Understanding a proposal and it's
> implications through a series of threads in the jira comments is
> difficult. It's hard for contributors and users to find these
> top-level jiras and follow their status.
>
> I'd like to propose that core Hadoop adopts something similar to
> Python's PEP (Python Enhancement Proposal) [1]. A "HEP" would be a
> single primary mechanism for proposing new features, incorporating
> community feedback, and recording decisions. The author of the HEP
> would be responsible for building consensus and moving the feature
> forward. Similarly, some subset of the community would be responsible
> for reviewing HEPs in a timely manner and identifying missing pieces
> in the proposal. Discussion would occur before patches showed up on
> jira. People interested in the core Hadoop roadmap could keep an eye
> on the HEPs without the overhead of following jira traffic.
>
> Why base this on the PEP? The format has proven useful to a
> substantial existing project, and I think the workflow is not too
> heavy-weight, and well-suited to a community such as ours. That being
> said, we could discuss other models (eg Java's JSR).
>
> Before we get into specifics, is this something the community would
> like to adopt in some form? Does adapting the PEP and its workflow to
> our projects, community and bylaws seem reasonable?
>
> Thanks,
> Eli
>
> 1. http://www.python.org/dev/peps/pep-0001

From general-return-1552-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat May 22 21:10:03 2010
Return-Path: <general-return-1552-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25089 invoked from network); 22 May 2010 21:10:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 May 2010 21:10:03 -0000
Received: (qmail 96933 invoked by uid 500); 22 May 2010 21:10:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96893 invoked by uid 500); 22 May 2010 21:10:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96885 invoked by uid 99); 22 May 2010 21:10:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 May 2010 21:10:02 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 74.125.82.48 is neither permitted nor denied by domain of oded@legolas-media.com)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 May 2010 21:09:54 +0000
Received: by wwb18 with SMTP id 18so1411070wwb.35
        for <general@hadoop.apache.org>; Sat, 22 May 2010 14:09:32 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.157.134 with SMTP id o6mr1912507wek.201.1274562572733; 
	Sat, 22 May 2010 14:09:32 -0700 (PDT)
Received: by 10.216.172.11 with HTTP; Sat, 22 May 2010 14:09:32 -0700 (PDT)
Date: Sun, 23 May 2010 00:09:32 +0300
Message-ID: <AANLkTinzP8-nnGg8Q5aaJ8gXCCg6Som7e8Xarc_2PGDD@mail.gmail.com>
Subject: How to write a complex Writable
From: Oded Rosen <oded@legolas-media.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f44ff8644e650487353971
X-Virus-Checked: Checked by ClamAV on apache.org

--001485f44ff8644e650487353971
Content-Type: text/plain; charset=ISO-8859-1

Since there are few sources on how to write a good Writable,
I want to share some tips I've learned over the last few days, while writing
a job that demanded some complex Writable object.
I will be glad to hear more tips, corrections, etc.


Wrtiables are a way to transfer complex data types from the mapper to the
reducer (/ combiner), or as a flexible output format for mapreduce jobs.
However, Hadoop does not always use them the way you think it does:
1. Hadoop reuses writable objects (at least the old API does) - so strange
data miraculously appears in your writables, if they are not cleaned right
before being used.
2. Hadoop compares Writables and hashes them a lot - so writing good
"hashCode" and "equals" functions is a necessity.
3. Hadoop needs an empty constructor for writables - so if you write another
constructor, be sure to also implement the empty one.

Any complex writable object you write (complex = more then just a couple of
fields) should:

*. Override Object's "*equals*": compare all available fields (deep
compare), check unique fields first, to avoid checking the rest.

*. Override Object's "*hashCode*": the simplest way is XORing (^) the hash
codes of the most important fields.

*. Create an *empty constructor* - even if you don't need one. Implementing
a different constructor is ok, as long as the empty is also available.

*. Implement (the mandatory) Writable's *readFields()* and *write()*. Use
versioning to allow scalability over time.
In the very begining of readFields(), clear all available fields (lists,
primitives, etc).
The best way to to do that is to create a clearFields() function, that will
be called both from "readFields()" and from the empty constructor.
Remember Hadoop reuses writables (again, at least the old API - "mapred" -
does), so this is not just a good habit, but clearly a must.

*. implement "*read()*" - this isn't mandatory but it's simple and helpful:

public static UserWritable read(DataInput in) throws IOException {
   UserWritable u = new UserWritable();
   u.readFields(in);
   return u;
}


More golden tips are welcomed. So does remarks.

-- 
Oded

--001485f44ff8644e650487353971--

From general-return-1553-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat May 22 21:51:35 2010
Return-Path: <general-return-1553-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29350 invoked from network); 22 May 2010 21:51:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 May 2010 21:51:34 -0000
Received: (qmail 12079 invoked by uid 500); 22 May 2010 21:51:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12033 invoked by uid 500); 22 May 2010 21:51:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12025 invoked by uid 99); 22 May 2010 21:51:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 May 2010 21:51:33 +0000
X-ASF-Spam-Status: No, hits=1.6 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com designates 209.85.221.172 as permitted sender)
Received: from [209.85.221.172] (HELO mail-qy0-f172.google.com) (209.85.221.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 May 2010 21:51:28 +0000
Received: by qyk2 with SMTP id 2so3604083qyk.20
        for <general@hadoop.apache.org>; Sat, 22 May 2010 14:51:07 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=EWOX3nb/B8EMK7nyAx2ns1WozjIe8K6Y/SnE4FcqTqo=;
        b=M5MrB/B/HqXaTuwRIAEZj02Q8F1qndld2mtqgNmuxuiVSpmMO72+wRserzoP0wXLo2
         eUw7s3BfLBbikJXYJUAdUWZektBpYRN39KpFnSX0kMNUVIjwVFttAFroSvCfRPaeOb+g
         qNxpq1bpmIj/eSuhmVgwZBn7Kvem+/f84+UTo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=opmlN+6QULxWiMCawJtP0lEVZy7UF9Mz7xTiHePjWu2l0FGQnpvzoljt/Oto+SrO6l
         0rbusUtVTx4rm+hzJRlTKDgl6EqmR1k8Bui/FTh+q8SIHBddvG1fuGyq5bgtW13dHobc
         1Fm2IAa1TQ6ZHsn+W642IjtlpaJwZ9SrojEjI=
MIME-Version: 1.0
Received: by 10.224.87.149 with SMTP id w21mr2229572qal.18.1274565067587; Sat, 
	22 May 2010 14:51:07 -0700 (PDT)
Received: by 10.229.57.80 with HTTP; Sat, 22 May 2010 14:51:07 -0700 (PDT)
In-Reply-To: <AANLkTinzP8-nnGg8Q5aaJ8gXCCg6Som7e8Xarc_2PGDD@mail.gmail.com>
References: <AANLkTinzP8-nnGg8Q5aaJ8gXCCg6Som7e8Xarc_2PGDD@mail.gmail.com>
Date: Sat, 22 May 2010 14:51:07 -0700
Message-ID: <AANLkTilTalfF-CRpFeyqcVo4BiH2D6R7RvjzAw8PodlK@mail.gmail.com>
Subject: Re: How to write a complex Writable
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00c09f99e5b018c1f0048735ce33

--00c09f99e5b018c1f0048735ce33
Content-Type: text/plain; charset=ISO-8859-1

Thanks for sharing your precious experience.

Hadoop achieves higher efficiency with Writables than with Serializables
through object resuse. That's why clearFields() function is needed.

In my job, I found it useful to persist the size of one record into
DataInput before the actual record if your Writable instance has more than
one record in it. This is for preparation of corrupt record (which we faced
in our trial) so that you can skip to the next record and reduce data loss.

Also DataOutput.writeUTF() may surprise you. At least it gave us a little
surprise.

Versioning is an interesting subject. I hope to see more suggestions on this
topic.

Cheers

On Sat, May 22, 2010 at 2:09 PM, Oded Rosen <oded@legolas-media.com> wrote:

> Since there are few sources on how to write a good Writable,
> I want to share some tips I've learned over the last few days, while
> writing
> a job that demanded some complex Writable object.
> I will be glad to hear more tips, corrections, etc.
>
>
> Wrtiables are a way to transfer complex data types from the mapper to the
> reducer (/ combiner), or as a flexible output format for mapreduce jobs.
> However, Hadoop does not always use them the way you think it does:
> 1. Hadoop reuses writable objects (at least the old API does) - so strange
> data miraculously appears in your writables, if they are not cleaned right
> before being used.
> 2. Hadoop compares Writables and hashes them a lot - so writing good
> "hashCode" and "equals" functions is a necessity.
> 3. Hadoop needs an empty constructor for writables - so if you write
> another
> constructor, be sure to also implement the empty one.
>
> Any complex writable object you write (complex = more then just a couple of
> fields) should:
>
> *. Override Object's "*equals*": compare all available fields (deep
> compare), check unique fields first, to avoid checking the rest.
>
> *. Override Object's "*hashCode*": the simplest way is XORing (^) the hash
> codes of the most important fields.
>
> *. Create an *empty constructor* - even if you don't need one. Implementing
> a different constructor is ok, as long as the empty is also available.
>
> *. Implement (the mandatory) Writable's *readFields()* and *write()*. Use
> versioning to allow scalability over time.
> In the very begining of readFields(), clear all available fields (lists,
> primitives, etc).
> The best way to to do that is to create a clearFields() function, that will
> be called both from "readFields()" and from the empty constructor.
> Remember Hadoop reuses writables (again, at least the old API - "mapred" -
> does), so this is not just a good habit, but clearly a must.
>
> *. implement "*read()*" - this isn't mandatory but it's simple and helpful:
>
> public static UserWritable read(DataInput in) throws IOException {
>   UserWritable u = new UserWritable();
>   u.readFields(in);
>   return u;
> }
>
>
> More golden tips are welcomed. So does remarks.
>
> --
> Oded
>

--00c09f99e5b018c1f0048735ce33--

From general-return-1554-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat May 22 22:11:50 2010
Return-Path: <general-return-1554-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35742 invoked from network); 22 May 2010 22:11:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 May 2010 22:11:50 -0000
Received: (qmail 18032 invoked by uid 500); 22 May 2010 22:11:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17986 invoked by uid 500); 22 May 2010 22:11:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17978 invoked by uid 99); 22 May 2010 22:11:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 May 2010 22:11:49 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 74.125.82.176 is neither permitted nor denied by domain of oded@legolas-media.com)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 May 2010 22:11:42 +0000
Received: by wyb32 with SMTP id 32so955578wyb.35
        for <general@hadoop.apache.org>; Sat, 22 May 2010 15:11:21 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.157.134 with SMTP id o6mr1944158wek.201.1274566281255; 
	Sat, 22 May 2010 15:11:21 -0700 (PDT)
Received: by 10.216.172.11 with HTTP; Sat, 22 May 2010 15:11:21 -0700 (PDT)
In-Reply-To: <AANLkTilTalfF-CRpFeyqcVo4BiH2D6R7RvjzAw8PodlK@mail.gmail.com>
References: <AANLkTinzP8-nnGg8Q5aaJ8gXCCg6Som7e8Xarc_2PGDD@mail.gmail.com>
	 <AANLkTilTalfF-CRpFeyqcVo4BiH2D6R7RvjzAw8PodlK@mail.gmail.com>
Date: Sun, 23 May 2010 01:11:21 +0300
Message-ID: <AANLkTim9wjucIcvZ3cpJKQTsRHIzxIJ6a5tQErKVb8xz@mail.gmail.com>
Subject: Re: How to write a complex Writable
From: Oded Rosen <oded@legolas-media.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f44ff86fda220487361619
X-Virus-Checked: Checked by ClamAV on apache.org

--001485f44ff86fda220487361619
Content-Type: text/plain; charset=ISO-8859-1

Thanks,
The "record size" tip looks helpful,
But I do not like surprises (at least not while programming):
what do you mean by "DataOutput.writeUTF() may surprise you"?

On Sun, May 23, 2010 at 12:51 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> Thanks for sharing your precious experience.
>
> Hadoop achieves higher efficiency with Writables than with Serializables
> through object resuse. That's why clearFields() function is needed.
>
> In my job, I found it useful to persist the size of one record into
> DataInput before the actual record if your Writable instance has more than
> one record in it. This is for preparation of corrupt record (which we faced
> in our trial) so that you can skip to the next record and reduce data loss.
>
> Also DataOutput.writeUTF() may surprise you. At least it gave us a little
> surprise.
>
> Versioning is an interesting subject. I hope to see more suggestions on
> this
> topic.
>
> Cheers
>
> On Sat, May 22, 2010 at 2:09 PM, Oded Rosen <oded@legolas-media.com>
> wrote:
>
> > Since there are few sources on how to write a good Writable,
> > I want to share some tips I've learned over the last few days, while
> > writing
> > a job that demanded some complex Writable object.
> > I will be glad to hear more tips, corrections, etc.
> >
> >
> > Wrtiables are a way to transfer complex data types from the mapper to the
> > reducer (/ combiner), or as a flexible output format for mapreduce jobs.
> > However, Hadoop does not always use them the way you think it does:
> > 1. Hadoop reuses writable objects (at least the old API does) - so
> strange
> > data miraculously appears in your writables, if they are not cleaned
> right
> > before being used.
> > 2. Hadoop compares Writables and hashes them a lot - so writing good
> > "hashCode" and "equals" functions is a necessity.
> > 3. Hadoop needs an empty constructor for writables - so if you write
> > another
> > constructor, be sure to also implement the empty one.
> >
> > Any complex writable object you write (complex = more then just a couple
> of
> > fields) should:
> >
> > *. Override Object's "*equals*": compare all available fields (deep
> > compare), check unique fields first, to avoid checking the rest.
> >
> > *. Override Object's "*hashCode*": the simplest way is XORing (^) the
> hash
> > codes of the most important fields.
> >
> > *. Create an *empty constructor* - even if you don't need one.
> Implementing
> > a different constructor is ok, as long as the empty is also available.
> >
> > *. Implement (the mandatory) Writable's *readFields()* and *write()*. Use
> > versioning to allow scalability over time.
> > In the very begining of readFields(), clear all available fields (lists,
> > primitives, etc).
> > The best way to to do that is to create a clearFields() function, that
> will
> > be called both from "readFields()" and from the empty constructor.
> > Remember Hadoop reuses writables (again, at least the old API - "mapred"
> -
> > does), so this is not just a good habit, but clearly a must.
> >
> > *. implement "*read()*" - this isn't mandatory but it's simple and
> helpful:
> >
> > public static UserWritable read(DataInput in) throws IOException {
> >   UserWritable u = new UserWritable();
> >   u.readFields(in);
> >   return u;
> > }
> >
> >
> > More golden tips are welcomed. So does remarks.
> >
> > --
> > Oded
> >
>



-- 
Oded

--001485f44ff86fda220487361619--

From general-return-1555-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat May 22 22:13:14 2010
Return-Path: <general-return-1555-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 35868 invoked from network); 22 May 2010 22:13:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 May 2010 22:13:13 -0000
Received: (qmail 19544 invoked by uid 500); 22 May 2010 22:13:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19486 invoked by uid 500); 22 May 2010 22:13:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19475 invoked by uid 99); 22 May 2010 22:13:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 May 2010 22:13:12 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.62.16] (HELO qmta01.westchester.pa.mail.comcast.net) (76.96.62.16)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 May 2010 22:13:03 +0000
Received: from omta03.westchester.pa.mail.comcast.net ([76.96.62.27])
	by qmta01.westchester.pa.mail.comcast.net with comcast
	id Lm8e1e0030bG4ec51mCicV; Sat, 22 May 2010 22:12:42 +0000
Received: from [192.168.1.68] ([209.131.62.115])
	by omta03.westchester.pa.mail.comcast.net with comcast
	id LmCY1e0072VBGtd3PmCb8u; Sat, 22 May 2010 22:12:40 +0000
Message-Id: <AAB436DF-AA6F-4B4A-A299-754207144F0B@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <AANLkTinzP8-nnGg8Q5aaJ8gXCCg6Som7e8Xarc_2PGDD@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: How to write a complex Writable
Date: Sat, 22 May 2010 15:12:31 -0700
References: <AANLkTinzP8-nnGg8Q5aaJ8gXCCg6Som7e8Xarc_2PGDD@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org


On May 22, 2010, at 2:09 PM, Oded Rosen wrote:

> *. Override Object's "*equals*": compare all available fields (deep
> compare), check unique fields first, to avoid checking the rest.

You should implement equals, but more important for keys is to define  
a RawComparator for your type. If you don't define one, the framework  
will de-serialize your type a *lot*. Look at one of the writables like  
IntWritable or Text for how to register your comparator.

-- Owen

From general-return-1556-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 24 16:16:45 2010
Return-Path: <general-return-1556-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37128 invoked from network); 24 May 2010 16:16:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 May 2010 16:16:44 -0000
Received: (qmail 12120 invoked by uid 500); 24 May 2010 16:16:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12059 invoked by uid 500); 24 May 2010 16:16:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 12051 invoked by uid 99); 24 May 2010 16:16:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 May 2010 16:16:42 +0000
X-ASF-Spam-Status: No, hits=1.6 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com designates 209.85.221.172 as permitted sender)
Received: from [209.85.221.172] (HELO mail-qy0-f172.google.com) (209.85.221.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 May 2010 16:16:36 +0000
Received: by qyk2 with SMTP id 2so6312275qyk.20
        for <general@hadoop.apache.org>; Mon, 24 May 2010 09:16:15 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:date:message-id
         :subject:from:to:content-type;
        bh=Ijc7NdwZYmuEN4mbFbgIOGIV0PS4RU2nDZsDSxYJG6Q=;
        b=byIhSRul3DMaYu1rt3nbspqg1tvRnmhuXPPNe9hYkqJnxO0QbFWcrxjJ+jtRvQDQtN
         TeSRrN+EmPyp8gGPRUF6/PpG7Av/Kv7dugNZaGEf8LIOH9f6jwWqUpCLN4pQSsw0+G/1
         o+eH2QfYxpcUw0B1TM4y6iV+vWxDt7s/HMALA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=jnpGYVbFbyczXGqisU6+x7OurQM0HhbOxrff+avJ8NaDYg9DpqvIomEjc/4ZDDrXu+
         ++WxfNTCFu2Hx57czBi8qW+v7XinEiKruthySSXhjh7oSPWnFabxfq9aJJhGdFPau2HV
         6eANGh+NbeFTZpLiTHIVaj5+X2hG5FmxlcC9o=
MIME-Version: 1.0
Received: by 10.224.88.220 with SMTP id b28mr3140404qam.91.1274717775312; Mon, 
	24 May 2010 09:16:15 -0700 (PDT)
Received: by 10.229.57.80 with HTTP; Mon, 24 May 2010 09:16:15 -0700 (PDT)
Date: Mon, 24 May 2010 09:16:15 -0700
Message-ID: <AANLkTikF-1z9VHldIoiPzMvCE3DMLYJHtcyjTWM0PNnN@mail.gmail.com>
Subject: checksum error
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00c09fa21a452fb8ec0487595cfe

--00c09fa21a452fb8ec0487595cfe
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi,
We currently use hadoop 0.20.1

I found the following trace in our log:

 2010-05-17 16:19:55,389 INFO  [FSInputChecker] Found checksum error: b[0,
512]=3D53455106246f72672e6170616368652e6861646f6f702e696f2e426f6f6c65616e57=
72697461626c652b636f6d2e6361727269657269712e6d326d2e636f6d6d6f6e732e4f74615=
5706c6f61645772697461626c6501002a6f72672e6170616368652e6861646f6f702e696f2e=
636f6d70726573732e44656661756c74436f64656300000000aa555ae318451e8e76c0230f5=
6f74d1f000014660000000101789cdddb09785355da00e0ef26699bae49ba2f94de2ed43685=
7ab3364194b414294a355296e29aa449079025968ae82f78cb22051c088202e2125040d9acb=
88bc36410441d9dbf50dc1e06c57143749c8e0e0c7bff73ee92a6e9a7f23f33cef33f7f1f08=
39eff9ee3df77ef7dcedeb03130facc763b437197d76abd96237983c6e8399731b3d564395d=
9e2b6797c6e60b9268fdb5d55d56c31180c6693c56bb21bc9327683d5e036dadd162fb056af=
c76ae17cd62a8bd1e7717b0d3e6f95d5e2357b4c55be260fd76406d6e76b32992c9e2aafdb6=
2b659bccd9cdb63b134f92c4d4d1e5b95d5e401d66430d8dc06b7d76e68f61a9acd469bdb6a=
37703ea3c9e33556594c1c19a5b9c964f671568bc96b34b83d55551eb2850693db6d767306b=
3d10a6c1367367bb9660f89b47a7d469fd96771db7d06aec9cbd9c9c69351ec5683917ce38c=
16bbcd62b17bcc4dcd96263347f6c4d2ecf536937d31588c16a3bdb9
org.apache.hadoop.fs.ChecksumException: Checksum error:
file:/incoming_ub/ciq_dca_uploads/2010/03/26/100813/17_00.ub at 0
        at
org.apache.hadoop.fs.FSInputChecker.verifySum(FSInputChecker.java:277)
        at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:2=
41)
        at org.apache.hadoop.fs.FSInputChecker.fill(FSInputChecker.java:176=
)
        at
org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:193)
        at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158=
)
        at java.io.DataInputStream.readFully(DataInputStream.java:178)
        at java.io.DataInputStream.readFully(DataInputStream.java:152)
        at
org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1450)
        at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1428)
        at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417)
        at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412)
        at
org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecord=
Reader.java:43)
        at
com.ciq.m2m.platform.util.hadoop.ReportingSequenceFileRecordReader.<init>(R=
eportingSequenceFileRecordReader.java:54)
        at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown
Source)
        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstru=
ctorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at
com.ciq.m2m.platform.util.hadoop.MultiFileRecordReader.assureReader(MultiFi=
leRecordReader.java:128)
        at
com.ciq.m2m.platform.util.hadoop.MultiFileRecordReader.readNextFromAnyReade=
r(MultiFileRecordReader.java:146)
        at
com.ciq.m2m.platform.util.hadoop.MultiFileRecordReader.readNextFromAnyReade=
r(MultiFileRecordReader.java:181)
        at
com.ciq.m2m.platform.util.hadoop.MultiFileRecordReader.next(MultiFileRecord=
Reader.java:122)
        at
com.ciq.m2m.platform.util.hadoop.MultiFileRecordReader.next(MultiFileRecord=
Reader.java:20)
        at
com.ciq.m2m.platform.mmp2.input.BaseOtaUploadRecordReader.readNextUpload(Ba=
seOtaUploadRecordReader.java:125)
        at
com.ciq.m2m.platform.mmp2.input.OtaUploadToBinaryPackageRecordReader.next(O=
taUploadToBinaryPackageRecordReader.java:176)
        at
com.ciq.m2m.platform.mmp2.input.OtaUploadToBinaryPackageRecordReader.next(O=
taUploadToBinaryPackageRecordReader.java:56)
        at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.jav=
a:192)
        at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:176)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
        at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:176)

If you saw similar trace before, please share your comment.

Thanks

--00c09fa21a452fb8ec0487595cfe--

From general-return-1557-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 24 21:39:13 2010
Return-Path: <general-return-1557-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69062 invoked from network); 24 May 2010 21:39:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 May 2010 21:39:12 -0000
Received: (qmail 97424 invoked by uid 500); 24 May 2010 21:39:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 97349 invoked by uid 500); 24 May 2010 21:39:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 97341 invoked by uid 99); 24 May 2010 21:39:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 May 2010 21:39:11 +0000
X-ASF-Spam-Status: No, hits=-0.1 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of Deepika.Khera@avg.com designates 193.85.188.248 as permitted sender)
Received: from [193.85.188.248] (HELO ms.grisoft.cz) (193.85.188.248)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 May 2010 21:39:03 +0000
Received: from deimos.cz.avg.com (unknown [192.168.200.161])
	(using TLSv1 with cipher AES128-SHA (128/128 bits))
	(No client certificate requested)
	by ms.grisoft.cz (Postfix) with ESMTP id 8404A3818A
	for <general@hadoop.apache.org>; Mon, 24 May 2010 23:38:42 +0200 (CEST)
Received: from miranda.us.avg.com (10.32.34.13) by deimos.cz.avg.com
 (192.168.200.161) with Microsoft SMTP Server (TLS) id 8.2.254.0; Mon, 24 May
 2010 23:38:42 +0200
Received: from miranda.us.avg.com ([10.32.34.13]) by miranda ([10.32.34.13])
 with mapi; Mon, 24 May 2010 17:38:40 -0400
From: Deepika Khera <Deepika.Khera@avg.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 24 May 2010 17:35:10 -0400
Subject: Hash Partitioner
Thread-Topic: Hash Partitioner
Thread-Index: Acr7iPzXnlegAwaWSIWvp8bSMNYjlw==
Message-ID: <DA852C02397DBC4EBFAA1B7192FBD15061DC83F92C@miranda>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_DA852C02397DBC4EBFAA1B7192FBD15061DC83F92Cmiranda_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_DA852C02397DBC4EBFAA1B7192FBD15061DC83F92Cmiranda_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi,

I am using a HashPartitioner on my key for a map reducer job.  I am wonderi=
ng how sometimes 2 reducers end up getting the same key ? I have the hashCo=
de method defined for my key.

Also, I have speculative execution turned off for my jobs..

Would appreciate any help.

Thanks,
Deepika

--_000_DA852C02397DBC4EBFAA1B7192FBD15061DC83F92Cmiranda_--

From general-return-1558-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 24 22:07:26 2010
Return-Path: <general-return-1558-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82826 invoked from network); 24 May 2010 22:07:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 May 2010 22:07:25 -0000
Received: (qmail 41736 invoked by uid 500); 24 May 2010 22:07:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 41658 invoked by uid 500); 24 May 2010 22:07:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 41650 invoked by uid 99); 24 May 2010 22:07:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 May 2010 22:07:23 +0000
X-ASF-Spam-Status: No, hits=0.5 required=10.0
	tests=AWL,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 May 2010 22:07:19 +0000
Received: by wyb32 with SMTP id 32so335835wyb.35
        for <general@hadoop.apache.org>; Mon, 24 May 2010 15:06:57 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.171.147 with SMTP id r19mr4016428wel.70.1274738817329; 
	Mon, 24 May 2010 15:06:57 -0700 (PDT)
Received: by 10.216.80.14 with HTTP; Mon, 24 May 2010 15:06:57 -0700 (PDT)
In-Reply-To: <DA852C02397DBC4EBFAA1B7192FBD15061DC83F92C@miranda>
References: <DA852C02397DBC4EBFAA1B7192FBD15061DC83F92C@miranda>
Date: Mon, 24 May 2010 18:06:57 -0400
Message-ID: <AANLkTilDlXliVirB601lvz9fLFf4LO_S6YkDI83VlNz-@mail.gmail.com>
Subject: Re: Hash Partitioner
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Deepika:

That sounds very strange. Can you let us know what version of Hadoop
(e.g. Apache 0.20.x, CDH2, etc.) you're running and a bit more about
your hashCode() implementation? When this happens, do you see the same
values for the duplicate key? Did you also implement a grouping
comparator?

The hash partitioner is extremely simple. It basically does
key.hashCode() % numberOfReduces =3D partition number to which a key is
assigned. If one incorrectly implements a grouping comparator, it's
possible you could see odd behavior, though.

On Mon, May 24, 2010 at 5:35 PM, Deepika Khera <Deepika.Khera@avg.com> wrot=
e:
> Hi,
>
> I am using a HashPartitioner on my key for a map reducer job. =A0I am won=
dering how sometimes 2 reducers end up getting the same key ? I have the ha=
shCode method defined for my key.
>
> Also, I have speculative execution turned off for my jobs..
>
> Would appreciate any help.
>
> Thanks,
> Deepika
>



--=20
Eric Sammer
phone: +1-917-287-2675
twitter: esammer
data: www.cloudera.com

From general-return-1559-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 24 22:33:24 2010
Return-Path: <general-return-1559-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91804 invoked from network); 24 May 2010 22:33:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 May 2010 22:33:23 -0000
Received: (qmail 69571 invoked by uid 500); 24 May 2010 22:33:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69523 invoked by uid 500); 24 May 2010 22:33:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69515 invoked by uid 99); 24 May 2010 22:33:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 May 2010 22:33:22 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of Deepika.Khera@avg.com designates 193.85.188.248 as permitted sender)
Received: from [193.85.188.248] (HELO ms.grisoft.cz) (193.85.188.248)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 May 2010 22:33:17 +0000
Received: from phobos.cz.avg.com (unknown [192.168.200.160])
	(using TLSv1 with cipher AES128-SHA (128/128 bits))
	(No client certificate requested)
	by ms.grisoft.cz (Postfix) with ESMTP id 988BA38161
	for <general@hadoop.apache.org>; Tue, 25 May 2010 00:32:56 +0200 (CEST)
Received: from miranda.us.avg.com (10.32.34.13) by phobos.cz.avg.com
 (192.168.200.160) with Microsoft SMTP Server (TLS) id 8.2.254.0; Tue, 25 May
 2010 00:32:56 +0200
Received: from miranda.us.avg.com ([10.32.34.13]) by miranda ([10.32.34.13])
 with mapi; Mon, 24 May 2010 18:32:54 -0400
From: Deepika Khera <Deepika.Khera@avg.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 24 May 2010 18:32:52 -0400
Subject: RE: Re: Hash Partitioner
Thread-Topic: Re: Hash Partitioner
Thread-Index: Acr7jYB8UAeH7K8uQ3q/DfMQjjTWrwAAhQPw
Message-ID: <DA852C02397DBC4EBFAA1B7192FBD15061DC83F931@miranda>
References: <DA852C02397DBC4EBFAA1B7192FBD15061DC83F92C@miranda>
 <AANLkTilDlXliVirB601lvz9fLFf4LO_S6YkDI83VlNz-@mail.gmail.com>
In-Reply-To: <AANLkTilDlXliVirB601lvz9fLFf4LO_S6YkDI83VlNz-@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks for your response Eric.

I am using hadoop 0.20.2.

Here is what the hashCode() implementation looks like (I actually had the I=
DE generate it for me)

Main key (for mapper & reducer):

public int hashCode() {
        int result =3D kVersion;
        result =3D 31 * result + (aKey !=3D null ? aKey.hashCode() : 0);
        result =3D 31 * result + (gKey !=3D null ? gKey.hashCode() : 0);
        result =3D 31 * result + (int) (date ^ (date >>> 32));
        result =3D 31 * result + (ma !=3D null ? ma.hashCode() : 0);
        result =3D 31 * result + (cl !=3D null ? cl.hashCode() : 0);
        return result;
    }
   =20
   =20
aKey : AKey class

   =20
    public int hashCode() {
        int result =3D kVersion;
        result =3D 31 * result + (v !=3D null ? v.hashCode() : 0);
        result =3D 31 * result + (s !=3D null ? s.hashCode() : 0);
        result =3D 31 * result + (o !=3D null ? o.hashCode() : 0);
        result =3D 31 * result + (l !=3D null ? l.hashCode() : 0);
        result =3D 31 * result + (e ? 1 : 0); //boolean
        result =3D 31 * result + (li ? 1 : 0); //boolean
        result =3D 31 * result + (aut ? 1 : 0); //boolean
        return result;
    }

When this happens, I do see the same values for the key. Also I am not usin=
g a grouping comparator.

I was wondering since the call to HashPartitioner.getPartition() is done fr=
om a map task, several of which are running on different machines, is it po=
ssible that they get a different hashcode and hence get different reducers =
assigned even when the key is the same.

Thanks,
Deepika



-----Original Message-----
From: Eric Sammer [mailto:esammer@cloudera.com]=20
Sent: Monday, May 24, 2010 3:07 PM
To: general@hadoop.apache.org
Subject: Re: Hash Partitioner

Deepika:

That sounds very strange. Can you let us know what version of Hadoop
(e.g. Apache 0.20.x, CDH2, etc.) you're running and a bit more about
your hashCode() implementation? When this happens, do you see the same
values for the duplicate key? Did you also implement a grouping
comparator?

The hash partitioner is extremely simple. It basically does
key.hashCode() % numberOfReduces =3D partition number to which a key is
assigned. If one incorrectly implements a grouping comparator, it's
possible you could see odd behavior, though.

On Mon, May 24, 2010 at 5:35 PM, Deepika Khera <Deepika.Khera@avg.com> wrot=
e:
> Hi,
>
> I am using a HashPartitioner on my key for a map reducer job. =A0I am won=
dering how sometimes 2 reducers end up getting the same key ? I have the ha=
shCode method defined for my key.
>
> Also, I have speculative execution turned off for my jobs..
>
> Would appreciate any help.
>
> Thanks,
> Deepika
>



--=20
Eric Sammer
phone: +1-917-287-2675
twitter: esammer
data: www.cloudera.com

From general-return-1560-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 24 23:46:56 2010
Return-Path: <general-return-1560-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 11468 invoked from network); 24 May 2010 23:46:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 May 2010 23:46:56 -0000
Received: (qmail 36461 invoked by uid 500); 24 May 2010 23:46:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 36415 invoked by uid 500); 24 May 2010 23:46:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 36407 invoked by uid 99); 24 May 2010 23:46:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 May 2010 23:46:54 +0000
X-ASF-Spam-Status: No, hits=2.4 required=10.0
	tests=AWL,EXTRA_MPART_TYPE,HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [117.120.16.147] (HELO mail62.messagelabs.com) (117.120.16.147)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 May 2010 23:46:48 +0000
X-VirusChecked: Checked
X-Env-Sender: Anthony.Ikeda@cardlink.com.au
X-Msg-Ref: server-14.tower-62.messagelabs.com!1274744785!15816406!1
X-StarScan-Version: 6.2.4; banners=cardlink.com.au,-,-
X-Originating-IP: [203.36.58.163]
Received: (qmail 30288 invoked from network); 24 May 2010 23:46:25 -0000
Received: from unknown (HELO rmail.5f55.cardlink.local) (203.36.58.163)
  by server-14.tower-62.messagelabs.com with DHE-RSA-AES256-SHA encrypted SMTP; 24 May 2010 23:46:25 -0000
Received: from r-exchange.cardlink.local (r-exchange.cardlink.local [10.3.3.3])
	by rmail.5f55.cardlink.local (8.13.5.20060308/8.13.4) with ESMTP id o4ONkPN3030259
	for <general@hadoop.apache.org>; Tue, 25 May 2010 09:46:25 +1000 (EST)
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/related;
	boundary="----_=_NextPart_001_01CAFB9B.51D6EEFB";
	type="multipart/alternative"
Subject: Active-Active Performance
Date: Tue, 25 May 2010 09:46:24 +1000
Message-ID: <590CDE7A083C4142AF05E96F2FB543BC9733D2@r-exchange.cardlink.local>
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator: 
Thread-Topic: Active-Active Performance
Thread-Index: Acr7m1GOZqwdJb/wQWutWjrPXspfRA==
From: "Anthony Ikeda" <Anthony.Ikeda@cardlink.com.au>
To: <general@hadoop.apache.org>

------_=_NextPart_001_01CAFB9B.51D6EEFB
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_002_01CAFB9B.51D6EEFB"


------_=_NextPart_002_01CAFB9B.51D6EEFB
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

I'm new to Hadoop and I've been given the task to see how we might
utilise Hadoop and HBase to implement an Active-Active site layer for
sharing information across a distributed application.

=20

I've been able to:

*         Install and get Hadoop running on a single node and am in the
process of configure a 2 node setup.

*         Install HBase on a single node and create a table and mapping
as well as insert data into the system

=20

Once I've got the mutli-node configured I hope to run some tests as
well.

=20

I've noticed that trying to start Hadoop in distributed mode, the slave
will ssh to the master to start it as well (bin/start-all.sh) provided
the same path is setup on the remote machine.

=20

Questions:

Can I configure the system IF the Hadoop installation is not in the same
location per machine?

If the master node goes down (say due to electrical fault or system
fault) how do the slave nodes react? Will they continue to run? Will the
nodes be back in sync once the master starts again?

Would I require a particular configuration to ensure that both our sites
can operate within the cluster as well as in a detached fashion (due to
maintenance or network issues)?

=20

We want to ensure that data is added to HBase on each site with the data
synced across both sites. If one site goes down then recovery of data is
imperative.

=20

=20

=20

Anthony Ikeda

Java Analyst/Programmer

Cardlink Services Limited

Level 4, 3 Rider Boulevard

Rhodes NSW 2138

=20

Web: www.cardlink.com.au | Tel: + 61 2 9646 9221 | Fax: + 61 2 9646 9283

=20

=20


**********************************************************************
This e-mail message and any attachments are intended only for the use of t=
he addressee(s) named above and may contain information that is privileged=
 and confidential. If you are not the intended recipient, any display, dis=
semination, distribution, or copying is strictly prohibited.   If you beli=
eve you have received this e-mail message in error, please immediately not=
ify the sender by replying to this e-mail message or by telephone to (02) =
9646 9222. Please delete the email and any attachments and do not retain t=
he email or any attachments in any form.
**********************************************************************
------_=_NextPart_002_01CAFB9B.51D6EEFB
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-mic=
rosoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word=
" xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"ht=
tp://www.w3.org/TR/REC-html40">

<head>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; charset=3Dus-ascii=
">
<meta name=3DGenerator content=3D"Microsoft Word 12 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<style>
<!--
 /* Font Definitions */
 @font-face
=09{font-family:"Cambria Math";
=09panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
=09{font-family:Calibri;
=09panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
=09{font-family:Tahoma;
=09panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
=09{font-family:Consolas;
=09panose-1:2 11 6 9 2 2 4 3 2 4;}
 /* Style Definitions */
 p.MsoNormal, li.MsoNormal, div.MsoNormal
=09{margin:0cm;
=09margin-bottom:.0001pt;
=09font-size:11.0pt;
=09font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
=09{mso-style-priority:99;
=09color:blue;
=09text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
=09{mso-style-priority:99;
=09color:purple;
=09text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
=09{mso-style-priority:99;
=09mso-style-link:"Balloon Text Char";
=09margin:0cm;
=09margin-bottom:.0001pt;
=09font-size:8.0pt;
=09font-family:"Tahoma","sans-serif";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
=09{mso-style-priority:34;
=09margin-top:0cm;
=09margin-right:0cm;
=09margin-bottom:0cm;
=09margin-left:36.0pt;
=09margin-bottom:.0001pt;
=09font-size:11.0pt;
=09font-family:"Calibri","sans-serif";}
span.EmailStyle17
=09{mso-style-type:personal-compose;
=09font-family:"Calibri","sans-serif";
=09color:windowtext;}
span.BalloonTextChar
=09{mso-style-name:"Balloon Text Char";
=09mso-style-priority:99;
=09mso-style-link:"Balloon Text";
=09font-family:"Tahoma","sans-serif";}
.MsoChpDefault
=09{mso-style-type:export-only;}
@page Section1
=09{size:612.0pt 792.0pt;
=09margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.Section1
=09{page:Section1;}
 /* List Definitions */
 @list l0
=09{mso-list-id:616915372;
=09mso-list-type:hybrid;
=09mso-list-template-ids:-1070316046 1735979434 201916419 201916421 201916=
417 201916419 201916421 201916417 201916419 201916421;}
@list l0:level1
=09{mso-level-start-at:0;
=09mso-level-number-format:bullet;
=09mso-level-text:\F0B7;
=09mso-level-tab-stop:none;
=09mso-level-number-position:left;
=09text-indent:-18.0pt;
=09font-family:Symbol;
=09mso-fareast-font-family:Calibri;
=09mso-bidi-font-family:"Times New Roman";}
@list l0:level2
=09{mso-level-tab-stop:72.0pt;
=09mso-level-number-position:left;
=09text-indent:-18.0pt;}
@list l0:level3
=09{mso-level-tab-stop:108.0pt;
=09mso-level-number-position:left;
=09text-indent:-18.0pt;}
@list l0:level4
=09{mso-level-tab-stop:144.0pt;
=09mso-level-number-position:left;
=09text-indent:-18.0pt;}
@list l0:level5
=09{mso-level-tab-stop:180.0pt;
=09mso-level-number-position:left;
=09text-indent:-18.0pt;}
@list l0:level6
=09{mso-level-tab-stop:216.0pt;
=09mso-level-number-position:left;
=09text-indent:-18.0pt;}
@list l0:level7
=09{mso-level-tab-stop:252.0pt;
=09mso-level-number-position:left;
=09text-indent:-18.0pt;}
@list l0:level8
=09{mso-level-tab-stop:288.0pt;
=09mso-level-number-position:left;
=09text-indent:-18.0pt;}
@list l0:level9
=09{mso-level-tab-stop:324.0pt;
=09mso-level-number-position:left;
=09text-indent:-18.0pt;}
ol
=09{margin-bottom:0cm;}
ul
=09{margin-bottom:0cm;}
-->
</style>
<!--[if gte mso 9]><xml>
 <o:shapedefaults v:ext=3D"edit" spidmax=3D"2050" />
</xml><![endif]--><!--[if gte mso 9]><xml>
 <o:shapelayout v:ext=3D"edit">
  <o:idmap v:ext=3D"edit" data=3D"1" />
 </o:shapelayout></xml><![endif]-->
</head>

<body lang=3DEN-AU link=3Dblue vlink=3Dpurple>

<div class=3DSection1>

<p class=3DMsoNormal>I&#8217;m new to Hadoop and I&#8217;ve been given the=
 task
to see how we might utilise Hadoop and HBase to implement an Active-Active=
 site
layer for sharing information across a distributed application.<o:p></o:p>=
</p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>I&#8217;ve been able to:<o:p></o:p></p>

<p class=3DMsoListParagraph style=3D'text-indent:-18.0pt;mso-list:l0 level=
1 lfo1'><![if !supportLists]><span
style=3D'font-family:Symbol'><span style=3D'mso-list:Ignore'>&middot;<span=

style=3D'font:7.0pt "Times New Roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp;&nbsp;
</span></span></span><![endif]>Install and get Hadoop running on a single =
node
and am in the process of configure a 2 node setup.<o:p></o:p></p>

<p class=3DMsoListParagraph style=3D'text-indent:-18.0pt;mso-list:l0 level=
1 lfo1'><![if !supportLists]><span
style=3D'font-family:Symbol'><span style=3D'mso-list:Ignore'>&middot;<span=

style=3D'font:7.0pt "Times New Roman"'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp;&nbsp;
</span></span></span><![endif]>Install HBase on a single node and create a=

table and mapping as well as insert data into the system<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>Once I&#8217;ve got the mutli-node configured I hope =
to run
some tests as well.<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>I&#8217;ve noticed that trying to start Hadoop in
distributed mode, the slave will ssh to the master to start it as well
(bin/start-all.sh) provided the same path is setup on the remote machine.<=
o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>Questions:<o:p></o:p></p>

<p class=3DMsoNormal>Can I configure the system IF the Hadoop installation=
 is not
in the same location per machine?<o:p></o:p></p>

<p class=3DMsoNormal>If the master node goes down (say due to electrical f=
ault or
system fault) how do the slave nodes react? Will they continue to run? Wil=
l the
nodes be back in sync once the master starts again?<o:p></o:p></p>

<p class=3DMsoNormal>Would I require a particular configuration to ensure =
that
both our sites can operate within the cluster as well as in a detached fas=
hion
(due to maintenance or network issues)?<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>We want to ensure that data is added to HBase on each=
 site
with the=20data synced across both sites. If one site goes down then recov=
ery of
data is imperative.<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal><span lang=3DEN-US>Anthony Ikeda<o:p></o:p></span></p=
>

<p class=3DMsoNormal><span lang=3DEN-US>Java Analyst/Programmer<o:p></o:p>=
</span></p>

<p class=3DMsoNormal><span lang=3DEN-US>Cardlink Services Limited<o:p></o:=
p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US style=3D'font-size:10.5pt;font-fam=
ily:Consolas'>Level
4, 3 Rider Boulevard<o:p></o:p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US style=3D'font-size:10.5pt;font-fam=
ily:Consolas'>Rhodes
NSW 2138<o:p></o:p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US style=3D'font-size:10.5pt;font-fam=
ily:Consolas'>&nbsp;<o:p></o:p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US style=3D'font-size:10.5pt;font-fam=
ily:Consolas'>Web:
<a href=3D"http://www.cardlink.com.au">www.cardlink.com.au</a> | Tel: + 61=
 2 9646
9221 | Fax: + 61 2 9646 9283<o:p></o:p></span></p>

<p class=3DMsoNormal><img border=3D0 width=3D155 height=3D108 id=3D"Pictur=
e_x0020_1"
src=3D"cid:image001.gif@01CAFBEF.233A76B0" alt=3D"logo_cardlink1"><span la=
ng=3DEN-US><o:p></o:p></span></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

</div>

<BR>
**********************************************************************<BR>=

This e-mail message and any attachments are intended only for the use of t=
he addressee(s) named above and may contain information that is privileged=
 and confidential. If you are not the intended recipient, any display, dis=
semination, distribution, or copying is strictly prohibited.   If you beli=
eve you have received this e-mail message in error, please immediately not=
ify the sender by replying to this e-mail message or by telephone to (02) =
9646 9222. Please delete the email and any attachments and do not retain t=
he email or any attachments in any form.<BR>
**********************************************************************<BR>=

</body>

</html>

------_=_NextPart_002_01CAFB9B.51D6EEFB--

------_=_NextPart_001_01CAFB9B.51D6EEFB--

From general-return-1561-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 25 02:08:49 2010
Return-Path: <general-return-1561-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60539 invoked from network); 25 May 2010 02:08:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 May 2010 02:08:48 -0000
Received: (qmail 23847 invoked by uid 500); 25 May 2010 02:08:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23792 invoked by uid 500); 25 May 2010 02:08:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23784 invoked by uid 99); 25 May 2010 02:08:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 02:08:47 +0000
X-ASF-Spam-Status: No, hits=0.8 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yhemanth@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 02:08:41 +0000
Received: by pwi9 with SMTP id 9so1261692pwi.35
        for <general@hadoop.apache.org>; Mon, 24 May 2010 19:08:21 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=aLL4fV89N1BhJwSSwaSvpUKiZPYBefzzRopDg01nKY0=;
        b=t5UZbY98p/+mqyK3Q5D6/G/bCz/n5MjYdv480cPSqwf4tnnBsIa6uqa09yO9IWX6Q9
         kY6kyBuP6ogs6ZjRNxfNi43UXEyka89lMMkCCpq2Dei0ywvd4REaS7/I0rGoYqd+QGN8
         +LmeajtGEUeueHX/rYvEJnHVaTlFRVMhc7EcI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=nllhFku5gLwjdc02nn+bVRCZi2i9zikrXGXwZxKEzDbQkmUMC2wAHrX5URo1Gt1yuB
         6GCLOwGkXO5OpNw7tJjjpqU96A4+98vcZUrubA7kCrjJFZMSo1y35SZWJkXkuEQ5BkNG
         /dToFUZN4KDeZrLNt2GOr4PJwtSSIIEQuAIaI=
MIME-Version: 1.0
Received: by 10.143.27.12 with SMTP id e12mr4541520wfj.87.1274753300827; Mon, 
	24 May 2010 19:08:20 -0700 (PDT)
Received: by 10.142.203.16 with HTTP; Mon, 24 May 2010 19:08:20 -0700 (PDT)
In-Reply-To: <590CDE7A083C4142AF05E96F2FB543BC9733D2@r-exchange.cardlink.local>
References: <590CDE7A083C4142AF05E96F2FB543BC9733D2@r-exchange.cardlink.local>
Date: Tue, 25 May 2010 07:38:20 +0530
Message-ID: <AANLkTil1XC1QF5XeGhGK_ZsQzXHaG5tNcRrNt9S2okSW@mail.gmail.com>
Subject: Re: Active-Active Performance
From: Hemanth Yamijala <yhemanth@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502cc73ac0e3f048761a1ba

--00504502cc73ac0e3f048761a1ba
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Anthony,

I=92m new to Hadoop and I=92ve been given the task to see how we might util=
ise
> Hadoop and HBase to implement an Active-Active site layer for sharing
> information across a distributed application.
>
>
>
> I=92ve been able to:
>
> =B7         Install and get Hadoop running on a single node and am in the
> process of configure a 2 node setup.
>
> =B7         Install HBase on a single node and create a table and mapping=
 as
> well as insert data into the system
>
>
>
> Once I=92ve got the mutli-node configured I hope to run some tests as wel=
l.
>
>
>
> I=92ve noticed that trying to start Hadoop in distributed mode, the slave
> will ssh to the master to start it as well (bin/start-all.sh) provided th=
e
> same path is setup on the remote machine.
>
>
>
> Questions:
>
> Can I configure the system IF the Hadoop installation is not in the same
> location per machine?
>

I would think configuring and managing such a system would get very complex
- for e.g. if you'll want to add nodes to expand in future. You would also
not be able to take advantage of the very helpful scripts that come with
Hadoop. Is there a reason why you want to do this ?

> If the master node goes down (say due to electrical fault or system fault=
)
> how do the slave nodes react? Will they continue to run? Will the nodes b=
e
> back in sync once the master starts again?
>

Hadoop slaves will continue. They will enter a retry loop trying to connect
to the master until it comes up. In doing so, they could fill up log files
very fast though. If the master starts with the same configuration, (same
host, ports), they should be able to connect and resume.

> Would I require a particular configuration to ensure that both our sites
> can operate within the cluster as well as in a detached fashion (due to
> maintenance or network issues)?
>
>
>
I did not quite follow this. Can you explain a little more about how you
want to setup your system ?

Thanks
Hemanth

--00504502cc73ac0e3f048761a1ba--

From general-return-1562-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 25 02:13:14 2010
Return-Path: <general-return-1562-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 61258 invoked from network); 25 May 2010 02:13:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 May 2010 02:13:14 -0000
Received: (qmail 26555 invoked by uid 500); 25 May 2010 02:13:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26512 invoked by uid 500); 25 May 2010 02:13:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26502 invoked by uid 99); 25 May 2010 02:13:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 02:13:12 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=AWL,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [216.82.242.195] (HELO mail16.messagelabs.com) (216.82.242.195)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 02:13:07 +0000
X-VirusChecked: Checked
X-Env-Sender: Anthony.Ikeda@cardlink.com.au
X-Msg-Ref: server-12.tower-16.messagelabs.com!1274753561!5226447!1
X-StarScan-Version: 6.2.4; banners=cardlink.com.au,-,-
X-Originating-IP: [203.36.58.163]
Received: (qmail 11544 invoked from network); 25 May 2010 02:12:43 -0000
Received: from unknown (HELO rmail.5f55.cardlink.local) (203.36.58.163)
  by server-12.tower-16.messagelabs.com with DHE-RSA-AES256-SHA encrypted SMTP; 25 May 2010 02:12:43 -0000
Received: from r-exchange.cardlink.local (r-exchange.cardlink.local [10.3.3.3])
	by rmail.5f55.cardlink.local (8.13.5.20060308/8.13.4) with ESMTP id o4P2Ceb5017651
	for <general@hadoop.apache.org>; Tue, 25 May 2010 12:12:41 +1000 (EST)
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: RE: Active-Active Performance
Date: Tue, 25 May 2010 12:12:37 +1000
Message-ID: <590CDE7A083C4142AF05E96F2FB543BC9733F9@r-exchange.cardlink.local>
In-Reply-To: <AANLkTil1XC1QF5XeGhGK_ZsQzXHaG5tNcRrNt9S2okSW@mail.gmail.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Active-Active Performance
Thread-Index: Acr7rzlKBO1sxD+mSPq4kslWPwyJNAAACHJg
References: <590CDE7A083C4142AF05E96F2FB543BC9733D2@r-exchange.cardlink.local> <AANLkTil1XC1QF5XeGhGK_ZsQzXHaG5tNcRrNt9S2okSW@mail.gmail.com>
From: "Anthony Ikeda" <Anthony.Ikeda@cardlink.com.au>
To: <general@hadoop.apache.org>

Thanks Hemanth,

In regards to different locations of the HADOOP home this is low
priority more for testing not production. I was trying to install HADOOP
for testing over 2 machines with only a Windows XP machine running
Cygwin and a Mac running Darwin. Not a priority.

In regards to my last question about operating in a detached fashion, we
are trying to factor in what happens when the link between both sites is
cut. Will both sites operate independently until the connection is
re-established? Is there any particular setup required to ensure we can
cover this scenario or is it an out-of-the-box feature?

Anthony


-----Original Message-----
From: Hemanth Yamijala [mailto:yhemanth@gmail.com]=20
Sent: Tuesday, 25 May 2010 12:08 PM
To: general@hadoop.apache.org
Subject: Re: Active-Active Performance

Anthony,

I'm new to Hadoop and I've been given the task to see how we might
utilise
> Hadoop and HBase to implement an Active-Active site layer for sharing
> information across a distributed application.
>
>
>
> I've been able to:
>
> *         Install and get Hadoop running on a single node and am in
the
> process of configure a 2 node setup.
>
> *         Install HBase on a single node and create a table and
mapping as
> well as insert data into the system
>
>
>
> Once I've got the mutli-node configured I hope to run some tests as
well.
>
>
>
> I've noticed that trying to start Hadoop in distributed mode, the
slave
> will ssh to the master to start it as well (bin/start-all.sh) provided
the
> same path is setup on the remote machine.
>
>
>
> Questions:
>
> Can I configure the system IF the Hadoop installation is not in the
same
> location per machine?
>

I would think configuring and managing such a system would get very
complex
- for e.g. if you'll want to add nodes to expand in future. You would
also
not be able to take advantage of the very helpful scripts that come with
Hadoop. Is there a reason why you want to do this ?

> If the master node goes down (say due to electrical fault or system
fault)
> how do the=20slave nodes react? Will they continue to run? Will the
nodes be
> back in sync once the master starts again?
>

Hadoop slaves will continue. They will enter a retry loop trying to
connect
to the master until it comes up. In doing so, they could fill up log
files
very fast though. If the master starts with the same configuration,
(same
host, ports), they should be able to connect and resume.

> Would I require a particular configuration to ensure that both our
sites
> can operate within the cluster as well as in a detached fashion (due
to
> maintenance or network issues)?
>
>
>
I did not quite follow this. Can you explain a little more about how you
want to setup your system ?

Thanks
Hemanth

_____________________________________________________________________=20
This e-mail has been scanned for viruses by MCI's Internet Managed=20
Scanning Services - powered by MessageLabs. For further information=20
visit http://www.mci.com

**********************************************************************
This e-mail message and any attachments are intended only for the use of t=
he addressee(s) named above and may contain information that is privileged=
 and confidential. If you are not the intended recipient, any display, dis=
semination, distribution, or copying is strictly prohibited.   If you beli=
eve you have received this e-mail message in error, please immediately not=
ify the sender by replying to this e-mail message or by telephone to (02) =
9646 9222. Please delete the email and any attachments and do not retain t=
he email or any attachments in any form.
**********************************************************************

From general-return-1563-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 25 02:19:27 2010
Return-Path: <general-return-1563-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 62223 invoked from network); 25 May 2010 02:19:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 May 2010 02:19:26 -0000
Received: (qmail 30645 invoked by uid 500); 25 May 2010 02:19:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 30568 invoked by uid 500); 25 May 2010 02:19:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 30560 invoked by uid 99); 25 May 2010 02:19:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 02:19:25 +0000
X-ASF-Spam-Status: No, hits=-0.2 required=10.0
	tests=AWL,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yhemanth@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 02:19:20 +0000
Received: by pwi9 with SMTP id 9so1267027pwi.35
        for <general@hadoop.apache.org>; Mon, 24 May 2010 19:19:00 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=zj/3mW0+RjCkGi6AV8JaCqNkYB40t08bLaWG1dPKWF4=;
        b=akwqQVaYD3vHn3eZieMqwg3jnJi7TCAR1nB+jz3ZK1Bi9M1EFB1+0eWs2WnW+uE5RG
         SU684pYrV8JMEZN0t69AMzU99k6L7dcnrOp2SOkRQEL07jsKhh5ckNsMx6DXQGDj12Cq
         Nq6ipXgt5KJOrHv4z6AkBYBjYUBHDkpb7xcBE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=awDRufWeT9EtNQ+vfG7kc1Q28m8grmrrvZs++nMDPkki7YZ6VHWaTCVtltz21uScm6
         LJUUOFguhK91gesEoNeHlRRuMcR+bZaMBNEbdQgPQLr81JUl84G14vtDnIEX2qPMTwuW
         0Pe/9/sxj8+5JV6FqE4/32DD4mbxjFL+zGh8E=
MIME-Version: 1.0
Received: by 10.142.2.2 with SMTP id 2mr4485602wfb.75.1274753940674; Mon, 24 
	May 2010 19:19:00 -0700 (PDT)
Received: by 10.142.203.16 with HTTP; Mon, 24 May 2010 19:19:00 -0700 (PDT)
In-Reply-To: <590CDE7A083C4142AF05E96F2FB543BC9733F9@r-exchange.cardlink.local>
References: <590CDE7A083C4142AF05E96F2FB543BC9733D2@r-exchange.cardlink.local>
	 <AANLkTil1XC1QF5XeGhGK_ZsQzXHaG5tNcRrNt9S2okSW@mail.gmail.com>
	 <590CDE7A083C4142AF05E96F2FB543BC9733F9@r-exchange.cardlink.local>
Date: Tue, 25 May 2010 07:49:00 +0530
Message-ID: <AANLkTilw_qnYDS1UMR-3G3zg2c6sN29QQqOl8VF5kC-W@mail.gmail.com>
Subject: Re: Active-Active Performance
From: Hemanth Yamijala <yhemanth@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Anthony,

> In regards to different locations of the HADOOP home this is low
> priority more for testing not production. I was trying to install HADOOP
> for testing over 2 machines with only a Windows XP machine running
> Cygwin and a Mac running Darwin. Not a priority.
>
> In regards to my last question about operating in a detached fashion, we
> are trying to factor in what happens when the link between both sites is
> cut. Will both sites operate independently until the connection is
> re-established? Is there any particular setup required to ensure we can
> cover this scenario or is it an out-of-the-box feature?

When you say 'sites', do you mean two different Hadoop installations ?
In general, each site is independent. So, I am unable to understand
where the link comes in.


>
> Anthony
>
>
> -----Original Message-----
> From: Hemanth Yamijala [mailto:yhemanth@gmail.com]
> Sent: Tuesday, 25 May 2010 12:08 PM
> To: general@hadoop.apache.org
> Subject: Re: Active-Active Performance
>
> Anthony,
>
> I'm new to Hadoop and I've been given the task to see how we might
> utilise
>> Hadoop and HBase to implement an Active-Active site layer for sharing
>> information across a distributed application.
>>
>>
>>
>> I've been able to:
>>
>> * =A0 =A0 =A0 =A0 Install and get Hadoop running on a single node and am=
 in
> the
>> process of configure a 2 node setup.
>>
>> * =A0 =A0 =A0 =A0 Install HBase on a single node and create a table and
> mapping as
>> well as insert data into the system
>>
>>
>>
>> Once I've got the mutli-node configured I hope to run some tests as
> well.
>>
>>
>>
>> I've noticed that trying to start Hadoop in distributed mode, the
> slave
>> will ssh to the master to start it as well (bin/start-all.sh) provided
> the
>> same path is setup on the remote machine.
>>
>>
>>
>> Questions:
>>
>> Can I configure the system IF the Hadoop installation is not in the
> same
>> location per machine?
>>
>
> I would think configuring and managing such a system would get very
> complex
> - for e.g. if you'll want to add nodes to expand in future. You would
> also
> not be able to take advantage of the very helpful scripts that come with
> Hadoop. Is there a reason why you want to do this ?
>
>> If the master node goes down (say due to electrical fault or system
> fault)
>> how do the slave nodes react? Will they continue to run? Will the
> nodes be
>> back in sync once the master starts again?
>>
>
> Hadoop slaves will continue. They will enter a retry loop trying to
> connect
> to the master until it comes up. In doing so, they could fill up log
> files
> very fast though. If the master starts with the same configuration,
> (same
> host, ports), they should be able to connect and resume.
>
>> Would I require a particular configuration to ensure that both our
> sites
>> can operate within the cluster as well as in a detached fashion (due
> to
>> maintenance or network issues)?
>>
>>
>>
> I did not quite follow this. Can you explain a little more about how you
> want to setup your system ?
>
> Thanks
> Hemanth
>
> _____________________________________________________________________
> This e-mail has been scanned for viruses by MCI's Internet Managed
> Scanning Services - powered by MessageLabs. For further information
> visit http://www.mci.com
>
> **********************************************************************
> This e-mail message and any attachments are intended only for the use of =
the addressee(s) named above and may contain information that is privileged=
 and confidential. If you are not the intended recipient, any display, diss=
emination, distribution, or copying is strictly prohibited. =A0 If you beli=
eve you have received this e-mail message in error, please immediately noti=
fy the sender by replying to this e-mail message or by telephone to (02) 96=
46 9222. Please delete the email and any attachments and do not retain the =
email or any attachments in any form.
> **********************************************************************
>

From general-return-1564-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 25 02:29:35 2010
Return-Path: <general-return-1564-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 64817 invoked from network); 25 May 2010 02:29:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 May 2010 02:29:34 -0000
Received: (qmail 39304 invoked by uid 500); 25 May 2010 02:29:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39190 invoked by uid 500); 25 May 2010 02:29:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39182 invoked by uid 99); 25 May 2010 02:29:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 02:29:33 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [117.120.20.163] (HELO mail118.messagelabs.com) (117.120.20.163)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 02:29:26 +0000
X-VirusChecked: Checked
X-Env-Sender: Anthony.Ikeda@cardlink.com.au
X-Msg-Ref: server-2.tower-118.messagelabs.com!1274754537!43481323!1
X-StarScan-Version: 6.2.4; banners=cardlink.com.au,-,-
X-Originating-IP: [203.36.58.163]
Received: (qmail 23195 invoked from network); 25 May 2010 02:28:58 -0000
Received: from unknown (HELO rmail.5f55.cardlink.local) (203.36.58.163)
  by server-2.tower-118.messagelabs.com with DHE-RSA-AES256-SHA encrypted SMTP; 25 May 2010 02:28:58 -0000
Received: from r-exchange.cardlink.local (r-exchange.cardlink.local [10.3.3.3])
	by rmail.5f55.cardlink.local (8.13.5.20060308/8.13.4) with ESMTP id o4P2SuNF004760
	for <general@hadoop.apache.org>; Tue, 25 May 2010 12:28:56 +1000 (EST)
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Subject: RE: Active-Active Performance
Date: Tue, 25 May 2010 12:28:56 +1000
Message-ID: <590CDE7A083C4142AF05E96F2FB543BC9733FE@r-exchange.cardlink.local>
In-Reply-To: <AANLkTilw_qnYDS1UMR-3G3zg2c6sN29QQqOl8VF5kC-W@mail.gmail.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Active-Active Performance
Thread-Index: Acr7sLZJABXNjv+kRxyednu9Bjl7VgAAE1Eg
References: <590CDE7A083C4142AF05E96F2FB543BC9733D2@r-exchange.cardlink.local> <AANLkTil1XC1QF5XeGhGK_ZsQzXHaG5tNcRrNt9S2okSW@mail.gmail.com> <590CDE7A083C4142AF05E96F2FB543BC9733F9@r-exchange.cardlink.local> <AANLkTilw_qnYDS1UMR-3G3zg2c6sN29QQqOl8VF5kC-W@mail.gmail.com>
From: "Anthony Ikeda" <Anthony.Ikeda@cardlink.com.au>
To: <general@hadoop.apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

Sorry Hemantha, internal terminology :)

We will most likely have 2 data centres (Site A and Site B) and we need da=
ta to be available between the 2 sites, i.e., if Site A stores data, Site =
B must be able to read it pretty much straight away.

We are thinking of installing Hadoop and HBase on each Site to ensure data=
 stored at Site A is readily available for Site B. We are not sure how man=
y installations are required per site yet but I'm guessing one site will h=
ouse the Master the other the Slave.

Because this is an Active-Active configuration, we need to ensure that if =
the link between each site goes down, they can still operate to a degree a=
nd once the link is recovered, both sites will sync up once more.

Also from a maintenance point of view, we may wish to add more instances o=
f Hadoop/HBase slaves on new machines in each site without disturbing the =
operation of the application.

Anthony

-----Original Message-----
From: Hemanth Yamijala [mailto:yhemanth@gmail.com]=20
Sent: Tuesday, 25 May 2010 12:19 PM
To: general@hadoop.apache.org
Subject: Re: Active-Active Performance

Anthony,

> In regards to different locations of the HADOOP home this is low
> priority more for testing not production. I was trying to install HADOOP=

> for testing over 2 machines with only a Windows XP machine running
> Cygwin and a Mac running Darwin. Not a priority.
>
> In regards to my last question about operating in a detached fashion, we=

> are trying to factor in what happens when the link between both sites is=

> cut. Will both sites operate independently until the connection is
> re-established? Is there any particular setup required to ensure we can
> cover this scenario or is it an out-of-the-box feature?

When you say 'sites', do you mean two different Hadoop installations ?
In general, each site is independent. So, I am unable to understand
where the link comes in.


>
> Anthony
>
>
> -----Original Message-----
> From: Hemanth Yamijala [mailto:yhemanth@gmail.com]
> Sent: Tuesday, 25 May 2010 12:08 PM
> To: general@hadoop.apache.org
> Subject: Re: Active-Active Performance
>
> Anthony,
>
> I'm new to Hadoop and I've been given the task to see how we might
> utilise
>> Hadoop and HBase to implement an Active-Active site layer for sharing
>> information across a distributed application.
>>
>>
>>
>> I've been able to:
>>
>> * =A0 =A0 =A0 =A0 Install and get Hadoop running on a single node and a=
m in
> the
>> process of configure a 2 node setup.
>>
>> * =A0 =A0 =A0 =A0 Install HBase on a single node and create a table and=

> mapping as
>> well as insert data into the system
>>
>>
>>
>> Once I've got the mutli-node configured I hope to run some tests as
> well.
>>
>>
>>
>> I've noticed that trying to start Hadoop in distributed mode, the
> slave
>> will ssh to the master to start it as well (bin/start-all.sh) provided
> the
>> same path is setup on the remote machine.
>>
>>
>>
>> Questions:
>>
>> Can I configure the system IF the Hadoop installation is not in the
> same
>> location per machine?
>>
>
> I would think configuring and managing such a system=20would get very
> complex
> - for e.g. if you'll want to add nodes to expand in future. You would
> also
> not be able to take advantage of the very helpful scripts that come with=

> Hadoop. Is there a reason why you want to do this ?
>
>> If the master node goes down (say due to electrical fault or system
> fault)
>> how do the slave nodes react? Will they continue to run? Will the
> nodes be
>> back in sync once the master starts again?
>>
>
> Hadoop slaves will continue. They will enter a retry loop trying to
> connect
> to the master until it comes up. In doing so, they could fill up log
> files
> very fast though. If the master starts with the same configuration,
> (same
> host, ports), they should be able to connect and resume.
>
>> Would I require a particular configuration to ensure that both our
> sites
>> can operate within the cluster as well as in a detached fashion (due
> to
>> maintenance or network issues)?
>>
>>
>>
> I did not quite follow this. Can you explain a little more about how you=

> want to setup your system ?
>
> Thanks
> Hemanth
>
> _____________________________________________________________________
> This e-mail has been scanned for viruses by MCI's Internet Managed
> Scanning Services - powered by MessageLabs. For further information
> visit http://www.mci.com
>
> **********************************************************************
> This e-mail message and any attachments are intended only for the use of=
 the addressee(s) named above and may contain information that is privileg=
ed and confidential. If you are not the intended recipient, any display, d=
issemination, distribution, or copying is strictly prohibited. =A0 If you =
believe you have received this e-mail message in error, please immediately=
 notify the sender by replying to this e-mail message or by telephone to (=
02) 9646 9222. Please delete the email and any attachments and do not reta=
in the email or any attachments in any form.
> **********************************************************************
>

_____________________________________________________________________=20
This e-mail has been scanned for viruses by MCI's Internet Managed=20
Scanning Services - powered by MessageLabs. For further information=20
visit http://www.mci.com

**********************************************************************
This e-mail message and any attachments are intended only for the use of t=
he addressee(s) named above and may contain information that is privileged=
 and confidential. If you are not the intended recipient, any display, dis=
semination, distribution, or copying is strictly prohibited.   If you beli=
eve you have received this e-mail message in error, please immediately not=
ify the sender by replying to this e-mail message or by telephone to (02) =
9646 9222. Please delete the email and any attachments and do not retain t=
he email or any attachments in any form.
**********************************************************************

From general-return-1565-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 25 08:10:20 2010
Return-Path: <general-return-1565-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81287 invoked from network); 25 May 2010 08:10:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 May 2010 08:10:19 -0000
Received: (qmail 90979 invoked by uid 500); 25 May 2010 08:10:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90708 invoked by uid 500); 25 May 2010 08:10:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90700 invoked by uid 99); 25 May 2010 08:10:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 08:10:15 +0000
X-ASF-Spam-Status: No, hits=1.4 required=10.0
	tests=AWL,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.221.172] (HELO mail-qy0-f172.google.com) (209.85.221.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 08:10:10 +0000
Received: by qyk2 with SMTP id 2so89965qyk.20
        for <general@hadoop.apache.org>; Tue, 25 May 2010 01:09:48 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.224.95.152 with SMTP id d24mr3740784qan.384.1274774988558; 
	Tue, 25 May 2010 01:09:48 -0700 (PDT)
Received: by 10.229.94.203 with HTTP; Tue, 25 May 2010 01:09:48 -0700 (PDT)
In-Reply-To: <66BC29ED-35C1-4916-BDFE-C768810C6EA6@mac.com>
References: <AANLkTimLB-YNWnMwa93YL8Lpg9hJLxrCKbYL3YUr6de8@mail.gmail.com>
	 <66BC29ED-35C1-4916-BDFE-C768810C6EA6@mac.com>
Date: Tue, 25 May 2010 01:09:48 -0700
Message-ID: <AANLkTimCkqddJ521jIPp0dVHC__vZ2PcavAIrNl334L0@mail.gmail.com>
Subject: Re: [DISCUSSION] Proposal for making core Hadoop changes
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00c09f8993f95c91f7048766ae55

--00c09f8993f95c91f7048766ae55
Content-Type: text/plain; charset=ISO-8859-1

> For comparison, anyone have a references to similar processes?
>

Java has the Java Community Process: http://jcp.org/en/home/index

--00c09f8993f95c91f7048766ae55--

From general-return-1566-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 25 10:04:37 2010
Return-Path: <general-return-1566-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 15479 invoked from network); 25 May 2010 10:04:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 May 2010 10:04:37 -0000
Received: (qmail 18307 invoked by uid 500); 25 May 2010 10:04:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18037 invoked by uid 500); 25 May 2010 10:04:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18029 invoked by uid 99); 25 May 2010 10:04:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 10:04:32 +0000
X-ASF-Spam-Status: No, hits=-2.0 required=10.0
	tests=AWL,RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 10:04:24 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id CE000B7E48
	for <general@hadoop.apache.org>; Tue, 25 May 2010 11:04:01 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id rQpt+rX-voBq for <general@hadoop.apache.org>;
	Tue, 25 May 2010 11:03:51 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id D986FB7C6D
	for <general@hadoop.apache.org>; Tue, 25 May 2010 11:03:50 +0100 (BST)
MailScanner-NULL-Check: 1275386617.53637@tfPlj9dxNcrkHzeTvUjaDQ
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o4PA3a1o002163
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Tue, 25 May 2010 11:03:37 +0100 (BST)
Message-ID: <4BFBA078.5000505@apache.org>
Date: Tue, 25 May 2010 11:03:36 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100228)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Active-Active Performance
References: <590CDE7A083C4142AF05E96F2FB543BC9733D2@r-exchange.cardlink.local> <AANLkTil1XC1QF5XeGhGK_ZsQzXHaG5tNcRrNt9S2okSW@mail.gmail.com> <590CDE7A083C4142AF05E96F2FB543BC9733F9@r-exchange.cardlink.local>
In-Reply-To: <590CDE7A083C4142AF05E96F2FB543BC9733F9@r-exchange.cardlink.local>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o4PA3a1o002163
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

Anthony Ikeda wrote:
> Thanks Hemanth,
> 
> In regards to different locations of the HADOOP home this is low
> priority more for testing not production. I was trying to install HADOOP
> for testing over 2 machines with only a Windows XP machine running
> Cygwin and a Mac running Darwin. Not a priority.

Things are much easier if
  -all your machines have the same OS, disk structure
  -you are running on linux
  -you use some CM tool to automate setup/deploy, pushing out of config 
files

Start now, start with VMWare or virtualbox images now, so you learn 
about management sooner rather than later

> In regards to my last question about operating in a detached fashion, we
> are trying to factor in what happens when the link between both sites is
> cut. Will both sites operate independently until the connection is
> re-established? Is there any particular setup required to ensure we can
> cover this scenario or is it an out-of-the-box feature?

HDFS and the MapReduce engine is designed to run on a single datacentre 
with high bandwidth, high reliability links, current releases assume the 
facility is secure and all users are trusted. The key SPOF, the 
Namenode, doesn't do failover, so when it goes down or the network 
partitions, all machines that cannot see the NN poll and spin until it 
comes back -which can take a while, unless you have a secondary namenode 
to keep the persistent files up to date.  the workers all assume that 
the hostname and IPAddr of the namenode doesn't change, and never reread 
their config. You could use DNS to do failover, but you have to tune the 
JVMs to not cache IP addresses for very long.

To do cross site stuff you'd need a separate HDFS filesystem per site, 
synchronisation of data now becomes a task for the higher level apps. I 
don't know what HBase, Cassandra or other column DB tools do here.


-steve



From general-return-1567-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 25 15:10:21 2010
Return-Path: <general-return-1567-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 49859 invoked from network); 25 May 2010 15:10:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 May 2010 15:10:21 -0000
Received: (qmail 15778 invoked by uid 500); 25 May 2010 15:10:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15690 invoked by uid 500); 25 May 2010 15:10:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15682 invoked by uid 99); 25 May 2010 15:10:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 15:10:19 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 15:10:13 +0000
Received: by wyb32 with SMTP id 32so936019wyb.35
        for <general@hadoop.apache.org>; Tue, 25 May 2010 08:09:53 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.216.85.194 with SMTP id u44mr4626652wee.10.1274800192637; Tue, 
	25 May 2010 08:09:52 -0700 (PDT)
Received: by 10.216.80.14 with HTTP; Tue, 25 May 2010 08:09:52 -0700 (PDT)
In-Reply-To: <DA852C02397DBC4EBFAA1B7192FBD15061DC83F931@miranda>
References: <DA852C02397DBC4EBFAA1B7192FBD15061DC83F92C@miranda>
	 <AANLkTilDlXliVirB601lvz9fLFf4LO_S6YkDI83VlNz-@mail.gmail.com>
	 <DA852C02397DBC4EBFAA1B7192FBD15061DC83F931@miranda>
Date: Tue, 25 May 2010 11:09:52 -0400
Message-ID: <AANLkTilGCNVJpT7fUXNBFySFvOO-FaDz5p7tu4yTjsGr@mail.gmail.com>
Subject: Re: Re: Hash Partitioner
From: Eric Sammer <esammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

On Mon, May 24, 2010 at 6:32 PM, Deepika Khera <Deepika.Khera@avg.com> wrot=
e:
> Thanks for your response Eric.
>
> I am using hadoop 0.20.2.
>
> Here is what the hashCode() implementation looks like (I actually had the=
 IDE generate it for me)
>
> Main key (for mapper & reducer):
>
> public int hashCode() {
> =A0 =A0 =A0 =A0int result =3D kVersion;
> =A0 =A0 =A0 =A0result =3D 31 * result + (aKey !=3D null ? aKey.hashCode()=
 : 0);
> =A0 =A0 =A0 =A0result =3D 31 * result + (gKey !=3D null ? gKey.hashCode()=
 : 0);
> =A0 =A0 =A0 =A0result =3D 31 * result + (int) (date ^ (date >>> 32));
> =A0 =A0 =A0 =A0result =3D 31 * result + (ma !=3D null ? ma.hashCode() : 0=
);
> =A0 =A0 =A0 =A0result =3D 31 * result + (cl !=3D null ? cl.hashCode() : 0=
);
> =A0 =A0 =A0 =A0return result;
> =A0 =A0}
>
>
> aKey : AKey class
>
>
> =A0 =A0public int hashCode() {
> =A0 =A0 =A0 =A0int result =3D kVersion;
> =A0 =A0 =A0 =A0result =3D 31 * result + (v !=3D null ? v.hashCode() : 0);
> =A0 =A0 =A0 =A0result =3D 31 * result + (s !=3D null ? s.hashCode() : 0);
> =A0 =A0 =A0 =A0result =3D 31 * result + (o !=3D null ? o.hashCode() : 0);
> =A0 =A0 =A0 =A0result =3D 31 * result + (l !=3D null ? l.hashCode() : 0);
> =A0 =A0 =A0 =A0result =3D 31 * result + (e ? 1 : 0); //boolean
> =A0 =A0 =A0 =A0result =3D 31 * result + (li ? 1 : 0); //boolean
> =A0 =A0 =A0 =A0result =3D 31 * result + (aut ? 1 : 0); //boolean
> =A0 =A0 =A0 =A0return result;
> =A0 =A0}
>

Both of these look fine, assuming all the other hashCode()s return the
same value every time.

> When this happens, I do see the same values for the key. Also I am not us=
ing a grouping comparator.

So you see two reduce methods getting the same key with the same
values? That's extremely odd. If this is the case, there's a bug in
Hadoop. Can you find the relevant logs from the reducers where Hadoop
fetches the map output? Does it look like its fetching the same output
twice? Do the two tasks where you see the duplicates have the same
task ID? Can you confirm the reduce tasks are from the same job ID for
us?

> I was wondering since the call to HashPartitioner.getPartition() is done =
from a map task, several of which are running on different machines, is it =
possible that they get a different hashcode and hence get different reducer=
s assigned even when the key is the same.

The hashCode() result should *always* be the same given the same
internal state. In other words, it should be consistent and stable. If
I have a string new String("hello world") it will always have the
exact same hashCode(). If this isn't true, you will get wildly
unpredictable results not just with Hadoop but with Java's
comparators, collections, etc.

--=20
Eric Sammer
phone: +1-917-287-2675
twitter: esammer
data: www.cloudera.com

From general-return-1568-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 25 16:29:47 2010
Return-Path: <general-return-1568-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82803 invoked from network); 25 May 2010 16:29:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 May 2010 16:29:47 -0000
Received: (qmail 52869 invoked by uid 500); 25 May 2010 16:29:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52822 invoked by uid 500); 25 May 2010 16:29:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52814 invoked by uid 99); 25 May 2010 16:29:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 16:29:45 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 16:29:38 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id C2259B7DEA
	for <general@hadoop.apache.org>; Tue, 25 May 2010 17:29:16 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id e7-GRQ3OK0Ay for <general@hadoop.apache.org>;
	Tue, 25 May 2010 17:29:06 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id D00D2B7E4E
	for <general@hadoop.apache.org>; Tue, 25 May 2010 17:29:05 +0100 (BST)
MailScanner-NULL-Check: 1275409733.08707@/hsN5ieyxyt4QFkzPxJATw
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o4PGSpmt012881
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Tue, 25 May 2010 17:28:52 +0100 (BST)
Message-ID: <4BFBFAC3.2040807@apache.org>
Date: Tue, 25 May 2010 17:28:51 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100228)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSSION] Proposal for making core Hadoop changes
References: <AANLkTimLB-YNWnMwa93YL8Lpg9hJLxrCKbYL3YUr6de8@mail.gmail.com>	 <66BC29ED-35C1-4916-BDFE-C768810C6EA6@mac.com> <AANLkTimCkqddJ521jIPp0dVHC__vZ2PcavAIrNl334L0@mail.gmail.com>
In-Reply-To: <AANLkTimCkqddJ521jIPp0dVHC__vZ2PcavAIrNl334L0@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o4PGSpmt012881
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

Jeff Hammerbacher wrote:
>> For comparison, anyone have a references to similar processes?
>>
> 
> Java has the Java Community Process: http://jcp.org/en/home/index
> 

a process that nobody liked, such as this comment by GregW of the jetty 
team on JSP 3
http://blogs.webtide.com/gregw/entry/servlet_3_0_public_review

JCP has some advantage over standards bodies I've been in
  * they recognise the value of tests.
  * better remote collaboration
  * more open to interested third parties
But that's it. Very vendor-managed, Sun was usually in charge, you'd be 
hard pressed to find anyone on the Apache jcp-open list (yes, we have 
one!) who is happy.

I haven't looked at Elliot's proposal in enough detail to comment, here 
are my thoughts from working on the lifecycle stuff, and on other ASF 
projects

* evolution in the codebase is a good way of getting stuff to meet 
people's needs. If you have to have big branches until things are 
perfect you have the cost of maintaining branches, its harder for people 
to experiment with your stuff.
* If the cost of adding features is high -and maintaining branches, 
merging, identifying test failures is high- the barrier to participation 
is pretty steep. you need a team of engineers to work on every feature

-steve

From general-return-1569-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 25 18:22:28 2010
Return-Path: <general-return-1569-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27214 invoked from network); 25 May 2010 18:22:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 May 2010 18:22:27 -0000
Received: (qmail 2828 invoked by uid 500); 25 May 2010 18:22:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2652 invoked by uid 500); 25 May 2010 18:22:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2642 invoked by uid 99); 25 May 2010 18:22:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 18:22:26 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 18:22:19 +0000
Received: from EGL-EX07CAS01.ds.corp.yahoo.com (egl-ex07cas01.eglbp.corp.yahoo.com [203.83.248.208])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o4PILQZn083703
	for <general@hadoop.apache.org>; Tue, 25 May 2010 11:21:27 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:in-reply-to:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=qiMv9/d1Ydw2qVt/XX+Y5Rfkb4FtdLwt5scg6bDWnSV7DKkhdNTZewUZYL+NeNQD
Received: from EGL-EX07VS01.ds.corp.yahoo.com ([203.83.248.205]) by
 EGL-EX07CAS01.ds.corp.yahoo.com ([203.83.248.215]) with mapi; Tue, 25 May
 2010 23:51:26 +0530
From: "Ankur C. Goel" <gankur@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 25 May 2010 23:51:24 +0530
Subject: Re: Hash Partitioner
Thread-Topic: Hash Partitioner
Thread-Index: Acr8HKEVbFionPTnTc+wSk5HMY3lnAAGnP0T
Message-ID: <C82212FC.9C8B%gankur@yahoo-inc.com>
In-Reply-To: <AANLkTilGCNVJpT7fUXNBFySFvOO-FaDz5p7tu4yTjsGr@mail.gmail.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C82212FC9C8Bgankuryahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C82212FC9C8Bgankuryahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Another thing I would look at critically is to see if there are any bugs in=
 the write() and readFields() method of my key class as there could hard to=
 identify serialization / de-serialization issues.

-@nkur


On 5/25/10 8:39 PM, "Eric Sammer" <esammer@cloudera.com> wrote:

On Mon, May 24, 2010 at 6:32 PM, Deepika Khera <Deepika.Khera@avg.com> wrot=
e:
> Thanks for your response Eric.
>
> I am using hadoop 0.20.2.
>
> Here is what the hashCode() implementation looks like (I actually had the=
 IDE generate it for me)
>
> Main key (for mapper & reducer):
>
> public int hashCode() {
>        int result =3D kVersion;
>        result =3D 31 * result + (aKey !=3D null ? aKey.hashCode() : 0);
>        result =3D 31 * result + (gKey !=3D null ? gKey.hashCode() : 0);
>        result =3D 31 * result + (int) (date ^ (date >>> 32));
>        result =3D 31 * result + (ma !=3D null ? ma.hashCode() : 0);
>        result =3D 31 * result + (cl !=3D null ? cl.hashCode() : 0);
>        return result;
>    }
>
>
> aKey : AKey class
>
>
>    public int hashCode() {
>        int result =3D kVersion;
>        result =3D 31 * result + (v !=3D null ? v.hashCode() : 0);
>        result =3D 31 * result + (s !=3D null ? s.hashCode() : 0);
>        result =3D 31 * result + (o !=3D null ? o.hashCode() : 0);
>        result =3D 31 * result + (l !=3D null ? l.hashCode() : 0);
>        result =3D 31 * result + (e ? 1 : 0); //boolean
>        result =3D 31 * result + (li ? 1 : 0); //boolean
>        result =3D 31 * result + (aut ? 1 : 0); //boolean
>        return result;
>    }
>

Both of these look fine, assuming all the other hashCode()s return the
same value every time.

> When this happens, I do see the same values for the key. Also I am not us=
ing a grouping comparator.

So you see two reduce methods getting the same key with the same
values? That's extremely odd. If this is the case, there's a bug in
Hadoop. Can you find the relevant logs from the reducers where Hadoop
fetches the map output? Does it look like its fetching the same output
twice? Do the two tasks where you see the duplicates have the same
task ID? Can you confirm the reduce tasks are from the same job ID for
us?

> I was wondering since the call to HashPartitioner.getPartition() is done =
from a map task, several of which are running on different machines, is it =
possible that they get a different hashcode and hence get different reducer=
s assigned even when the key is the same.

The hashCode() result should *always* be the same given the same
internal state. In other words, it should be consistent and stable. If
I have a string new String("hello world") it will always have the
exact same hashCode(). If this isn't true, you will get wildly
unpredictable results not just with Hadoop but with Java's
comparators, collections, etc.

--
Eric Sammer
phone: +1-917-287-2675
twitter: esammer
data: www.cloudera.com


--_000_C82212FC9C8Bgankuryahooinccom_--

From general-return-1570-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 25 19:42:33 2010
Return-Path: <general-return-1570-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85053 invoked from network); 25 May 2010 19:42:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 May 2010 19:42:32 -0000
Received: (qmail 25871 invoked by uid 500); 25 May 2010 19:42:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25836 invoked by uid 500); 25 May 2010 19:42:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25828 invoked by uid 99); 25 May 2010 19:42:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 19:42:31 +0000
X-ASF-Spam-Status: No, hits=-0.5 required=10.0
	tests=AWL,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 19:42:27 +0000
Received: by pwi9 with SMTP id 9so1870211pwi.35
        for <general@hadoop.apache.org>; Tue, 25 May 2010 12:42:06 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.142.209.15 with SMTP id h15mr4935829wfg.150.1274816526639; 
	Tue, 25 May 2010 12:42:06 -0700 (PDT)
Received: by 10.143.16.21 with HTTP; Tue, 25 May 2010 12:42:06 -0700 (PDT)
In-Reply-To: <4BFBFAC3.2040807@apache.org>
References: <AANLkTimLB-YNWnMwa93YL8Lpg9hJLxrCKbYL3YUr6de8@mail.gmail.com>
	 <66BC29ED-35C1-4916-BDFE-C768810C6EA6@mac.com>
	 <AANLkTimCkqddJ521jIPp0dVHC__vZ2PcavAIrNl334L0@mail.gmail.com>
	 <4BFBFAC3.2040807@apache.org>
Date: Tue, 25 May 2010 12:42:06 -0700
Message-ID: <AANLkTilhcUYlFRwc1vc7-sFybjUw5c_uPosptIGb-Gn1@mail.gmail.com>
Subject: Re: [DISCUSSION] Proposal for making core Hadoop changes
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Tue, May 25, 2010 at 9:28 AM, Steve Loughran <stevel@apache.org> wrote:
> Jeff Hammerbacher wrote:
>>>
>>> For comparison, anyone have a references to similar processes?
>>>
>>
>> Java has the Java Community Process: http://jcp.org/en/home/index
>>
>
> a process that nobody liked, such as this comment by GregW of the jetty t=
eam
> on JSP 3
> http://blogs.webtide.com/gregw/entry/servlet_3_0_public_review
>
> JCP has some advantage over standards bodies I've been in
> =A0* they recognise the value of tests.
> =A0* better remote collaboration
> =A0* more open to interested third parties
> But that's it. Very vendor-managed, Sun was usually in charge, you'd be h=
ard
> pressed to find anyone on the Apache jcp-open list (yes, we have one!) wh=
o
> is happy.

The JCP seems heavy weight, we'll want to make sure the pendulum
doesn't swing too far in the opposite direction.
Would be interesting if there are other good light weight alternatives
to the PEP, I looked and didn't turn up many.

> * evolution in the codebase is a good way of getting stuff to meet people=
's
> needs. If you have to have big branches until things are perfect you have
> the cost of maintaining branches, its harder for people to experiment wit=
h
> your stuff.
> * If the cost of adding features is high -and maintaining branches, mergi=
ng,
> identifying test failures is high- the barrier to participation is pretty
> steep. you need a team of engineers to work on every feature

The cost of adding features has gotten high anyway (even without
branching). It's a classic trade-off -- merge overhead vs moving
faster without burdening others -- as the overhead imposed on others
increases, and tools (git) make it easier to live and collaborate on
branches it makes more sense (you don't need a team of engineers or
dedicated merge engineer to maintain the branch). Might find the
following interesting:
http://incubator.apache.org/learn/rules-for-revolutionaries.html

Thanks,
Eli

From general-return-1571-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 25 20:08:16 2010
Return-Path: <general-return-1571-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 37408 invoked from network); 25 May 2010 20:08:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 May 2010 20:08:16 -0000
Received: (qmail 67250 invoked by uid 500); 25 May 2010 20:08:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67202 invoked by uid 500); 25 May 2010 20:08:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67192 invoked by uid 99); 25 May 2010 20:08:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 20:08:15 +0000
X-ASF-Spam-Status: No, hits=-1.7 required=10.0
	tests=AWL,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of abhig@princeton.edu designates 128.112.128.213 as permitted sender)
Received: from [128.112.128.213] (HELO ppa01.Princeton.EDU) (128.112.128.213)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 20:08:07 +0000
Received: from smtpserver2.Princeton.EDU (smtpserver2.Princeton.EDU [128.112.129.148])
	by ppa01.Princeton.EDU (8.14.3/8.14.3) with ESMTP id o4PK7j7c009549
	for <general@hadoop.apache.org>; Tue, 25 May 2010 16:07:46 -0400
Received: from phy-abhig.Princeton.EDU (phy-abhig.Princeton.EDU [128.112.85.133])
	(authenticated bits=0)
	by smtpserver2.Princeton.EDU (8.12.9/8.12.9) with ESMTP id o4PK7igm026083
	(version=TLSv1/SSLv3 cipher=RC4-MD5 bits=128 verify=NOT);
	Tue, 25 May 2010 16:07:44 -0400 (EDT)
Message-ID: <4BFC2E10.3030206@princeton.edu>
Date: Tue, 25 May 2010 16:07:44 -0400
From: Abhishek Gupta <abhig@princeton.edu>
User-Agent: Thunderbird 2.0.0.19 (X11/20090107)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: HDFS documentation
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi,
I am a new user for HDFS. I tried to understand the documentation given 
on the website but I am not able to get most of the things.
Is there any complete or more elaborated documentation for HDFS 
configuration?
If anyone has their configuration notes that can be shared, it would be 
great.
Thanks,
Abhi.

From general-return-1572-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 25 20:17:13 2010
Return-Path: <general-return-1572-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39243 invoked from network); 25 May 2010 20:17:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 May 2010 20:17:12 -0000
Received: (qmail 80148 invoked by uid 500); 25 May 2010 20:17:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80066 invoked by uid 500); 25 May 2010 20:17:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80058 invoked by uid 99); 25 May 2010 20:17:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 20:17:11 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 20:17:06 +0000
Received: by pvc21 with SMTP id 21so135681pvc.35
        for <general@hadoop.apache.org>; Tue, 25 May 2010 13:16:44 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.142.67.35 with SMTP id p35mr5466399wfa.203.1274818604084; Tue, 
	25 May 2010 13:16:44 -0700 (PDT)
Received: by 10.143.16.21 with HTTP; Tue, 25 May 2010 13:16:44 -0700 (PDT)
In-Reply-To: <4BFC2E10.3030206@princeton.edu>
References: <4BFC2E10.3030206@princeton.edu>
Date: Tue, 25 May 2010 13:16:44 -0700
Message-ID: <AANLkTimaaJTHh3SvB4lPDv6Pifi55ieSujr1Yxiq5IOe@mail.gmail.com>
Subject: Re: HDFS documentation
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hey Abhi,

Two good pages on HDFS configuration:

* http://hadoop.apache.org/common/docs/current/hdfs-default.html
* http://developer.yahoo.com/hadoop/tutorial/module2.html

There's also an excellent chapter on HDFS configuration in Hadoop: the
definitive guide.

http://oreilly.com/catalog/9780596521981

Thanks,
Eli



On Tue, May 25, 2010 at 1:07 PM, Abhishek Gupta <abhig@princeton.edu> wrote:
> Hi,
> I am a new user for HDFS. I tried to understand the documentation given on
> the website but I am not able to get most of the things.
> Is there any complete or more elaborated documentation for HDFS
> configuration?
> If anyone has their configuration notes that can be shared, it would be
> great.
> Thanks,
> Abhi.
>

From general-return-1573-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 25 20:18:46 2010
Return-Path: <general-return-1573-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39698 invoked from network); 25 May 2010 20:18:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 May 2010 20:18:46 -0000
Received: (qmail 83060 invoked by uid 500); 25 May 2010 20:18:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82927 invoked by uid 500); 25 May 2010 20:18:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82917 invoked by uid 99); 25 May 2010 20:18:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 20:18:45 +0000
X-ASF-Spam-Status: No, hits=-0.5 required=10.0
	tests=AWL,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 20:18:41 +0000
Received: by pwi9 with SMTP id 9so1892501pwi.35
        for <general@hadoop.apache.org>; Tue, 25 May 2010 13:18:20 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.143.178.6 with SMTP id f6mr3734715wfp.342.1274818698228; Tue, 
	25 May 2010 13:18:18 -0700 (PDT)
Received: by 10.143.16.21 with HTTP; Tue, 25 May 2010 13:18:18 -0700 (PDT)
In-Reply-To: <AANLkTimaaJTHh3SvB4lPDv6Pifi55ieSujr1Yxiq5IOe@mail.gmail.com>
References: <4BFC2E10.3030206@princeton.edu>
	 <AANLkTimaaJTHh3SvB4lPDv6Pifi55ieSujr1Yxiq5IOe@mail.gmail.com>
Date: Tue, 25 May 2010 13:18:18 -0700
Message-ID: <AANLkTinBgZ1Ics2_kWctU8VJFLKgVTizka6Qy-xdU8gG@mail.gmail.com>
Subject: Re: HDFS documentation
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Forgot to mention, the following also has some good tips on
HDFS-related configuration.

http://www.cloudera.com/blog/2009/03/configuration-parameters-what-can-you-just-ignore

Thanks,
Eli

On Tue, May 25, 2010 at 1:16 PM, Eli Collins <eli@cloudera.com> wrote:
> Hey Abhi,
>
> Two good pages on HDFS configuration:
>
> * http://hadoop.apache.org/common/docs/current/hdfs-default.html
> * http://developer.yahoo.com/hadoop/tutorial/module2.html
>
> There's also an excellent chapter on HDFS configuration in Hadoop: the
> definitive guide.
>
> http://oreilly.com/catalog/9780596521981
>
> Thanks,
> Eli
>
>
>
> On Tue, May 25, 2010 at 1:07 PM, Abhishek Gupta <abhig@princeton.edu> wrote:
>> Hi,
>> I am a new user for HDFS. I tried to understand the documentation given on
>> the website but I am not able to get most of the things.
>> Is there any complete or more elaborated documentation for HDFS
>> configuration?
>> If anyone has their configuration notes that can be shared, it would be
>> great.
>> Thanks,
>> Abhi.
>>
>

From general-return-1574-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 25 20:33:43 2010
Return-Path: <general-return-1574-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 42418 invoked from network); 25 May 2010 20:33:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 May 2010 20:33:42 -0000
Received: (qmail 96679 invoked by uid 500); 25 May 2010 20:33:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96631 invoked by uid 500); 25 May 2010 20:33:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96623 invoked by uid 99); 25 May 2010 20:33:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 20:33:41 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of abhig@princeton.edu designates 128.112.128.214 as permitted sender)
Received: from [128.112.128.214] (HELO ppa03.Princeton.EDU) (128.112.128.214)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 20:33:34 +0000
Received: from smtpserver1.Princeton.EDU (smtpserver1.Princeton.EDU [128.112.129.65])
	by ppa03.Princeton.EDU (8.14.3/8.14.3) with ESMTP id o4PKXD2V029278
	for <general@hadoop.apache.org>; Tue, 25 May 2010 16:33:13 -0400
Received: from phy-abhig.Princeton.EDU (phy-abhig.Princeton.EDU [128.112.85.133])
	(authenticated bits=0)
	by smtpserver1.Princeton.EDU (8.12.9/8.12.9) with ESMTP id o4PKXC7h001997
	(version=TLSv1/SSLv3 cipher=RC4-MD5 bits=128 verify=NOT);
	Tue, 25 May 2010 16:33:12 -0400 (EDT)
Message-ID: <4BFC3407.7020908@princeton.edu>
Date: Tue, 25 May 2010 16:33:11 -0400
From: Abhishek Gupta <abhig@princeton.edu>
User-Agent: Thunderbird 2.0.0.19 (X11/20090107)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: HDFS documentation
References: <4BFC2E10.3030206@princeton.edu>	 <AANLkTimaaJTHh3SvB4lPDv6Pifi55ieSujr1Yxiq5IOe@mail.gmail.com> <AANLkTinBgZ1Ics2_kWctU8VJFLKgVTizka6Qy-xdU8gG@mail.gmail.com>
In-Reply-To: <AANLkTinBgZ1Ics2_kWctU8VJFLKgVTizka6Qy-xdU8gG@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks Eli,
Abhi.

Eli Collins wrote:
> Forgot to mention, the following also has some good tips on
> HDFS-related configuration.
>
> http://www.cloudera.com/blog/2009/03/configuration-parameters-what-can-you-just-ignore
>
> Thanks,
> Eli
>
> On Tue, May 25, 2010 at 1:16 PM, Eli Collins <eli@cloudera.com> wrote:
>   
>> Hey Abhi,
>>
>> Two good pages on HDFS configuration:
>>
>> * http://hadoop.apache.org/common/docs/current/hdfs-default.html
>> * http://developer.yahoo.com/hadoop/tutorial/module2.html
>>
>> There's also an excellent chapter on HDFS configuration in Hadoop: the
>> definitive guide.
>>
>> http://oreilly.com/catalog/9780596521981
>>
>> Thanks,
>> Eli
>>
>>
>>
>> On Tue, May 25, 2010 at 1:07 PM, Abhishek Gupta <abhig@princeton.edu> wrote:
>>     
>>> Hi,
>>> I am a new user for HDFS. I tried to understand the documentation given on
>>> the website but I am not able to get most of the things.
>>> Is there any complete or more elaborated documentation for HDFS
>>> configuration?
>>> If anyone has their configuration notes that can be shared, it would be
>>> great.
>>> Thanks,
>>> Abhi.
>>>
>>>       

From general-return-1575-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue May 25 21:03:32 2010
Return-Path: <general-return-1575-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47671 invoked from network); 25 May 2010 21:03:31 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 May 2010 21:03:31 -0000
Received: (qmail 32251 invoked by uid 500); 25 May 2010 21:03:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32212 invoked by uid 500); 25 May 2010 21:03:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32204 invoked by uid 99); 25 May 2010 21:03:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 21:03:30 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of Deepika.Khera@avg.com designates 193.85.188.248 as permitted sender)
Received: from [193.85.188.248] (HELO ms.grisoft.cz) (193.85.188.248)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 21:03:24 +0000
Received: from phobos.cz.avg.com (unknown [192.168.200.160])
	(using TLSv1 with cipher AES128-SHA (128/128 bits))
	(No client certificate requested)
	by ms.grisoft.cz (Postfix) with ESMTP id 7F937384C5
	for <general@hadoop.apache.org>; Tue, 25 May 2010 23:03:03 +0200 (CEST)
Received: from miranda.us.avg.com (10.32.34.13) by phobos.cz.avg.com
 (192.168.200.160) with Microsoft SMTP Server (TLS) id 8.2.254.0; Tue, 25 May
 2010 23:03:03 +0200
Received: from miranda.us.avg.com ([10.32.34.13]) by miranda ([10.32.34.13])
 with mapi; Tue, 25 May 2010 17:03:01 -0400
From: Deepika Khera <Deepika.Khera@avg.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 25 May 2010 17:03:00 -0400
Subject: RE: Re: Re: Hash Partitioner
Thread-Topic: Re: Re: Hash Partitioner
Thread-Index: Acr8HGcv7YU8+mRFRRShjsHCpuhKbQALyO+Q
Message-ID: <DA852C02397DBC4EBFAA1B7192FBD15061DC83F9C5@miranda>
References: <DA852C02397DBC4EBFAA1B7192FBD15061DC83F92C@miranda>
	 <AANLkTilDlXliVirB601lvz9fLFf4LO_S6YkDI83VlNz-@mail.gmail.com>
	 <DA852C02397DBC4EBFAA1B7192FBD15061DC83F931@miranda>
 <AANLkTilGCNVJpT7fUXNBFySFvOO-FaDz5p7tu4yTjsGr@mail.gmail.com>
In-Reply-To: <AANLkTilGCNVJpT7fUXNBFySFvOO-FaDz5p7tu4yTjsGr@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

So I ran my process again with some more logging and here is what I see.=20

I used my own HashPartitioner(basically copied hadoop's partitioner and add=
ed some logging for analysis).I printed here the Key and the reducer that i=
s assigned to the key (based on the hash code).=20

My process triggered off 2 mappers (running on 2 different hadoop machines)=
, hence both of these try to find reducers for the split file assigned to t=
hem. I see that for a same object key assigned to both these mappers, I am =
getting 2 different reducers allocated by the Partitioner.

In the reducers I see -

1) 2 different reducers (the ones that the partitioner assigned the key to)=
 printing out the same Key (I did not print out the value as I thought that=
 wouldn't matter)
2) Here are the logs from where reducers copies data from the mapper -
=09
Reducer1:

2010-05-25 11:34:49,810 INFO org.apache.hadoop.mapred.ReduceTask: Read 1002=
612 bytes from map-output for attempt_201005251129_0001_m_000001_0
2010-05-25 11:34:49,831 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 fr=
om attempt_201005251129_0001_m_000001_0 -> (127, 36) from hadoop-49.c.a.com
2010-05-25 11:34:50,797 INFO org.apache.hadoop.mapred.ReduceTask: attempt_2=
01005251129_0001_r_000006_0: Got 1 new map-outputs
2010-05-25 11:34:54,835 INFO org.apache.hadoop.mapred.ReduceTask: attempt_2=
01005251129_0001_r_000006_0 Scheduled 1 outputs (0 slow hosts and0 dup host=
s)
2010-05-25 11:34:54,841 INFO org.apache.hadoop.mapred.ReduceTask: header: a=
ttempt_201005251129_0001_m_000000_0, compressed len: 1553902, decompressed =
len: 1553898
2010-05-25 11:34:54,841 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling=
 1553898 bytes (1553902 raw bytes) into RAM from attempt_201005251129_0001_=
m_000000_0
2010-05-25 11:34:54,924 INFO org.apache.hadoop.mapred.ReduceTask: Read 1553=
898 bytes from map-output for attempt_201005251129_0001_m_000000_0
2010-05-25 11:34:54,944 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 fr=
om attempt_201005251129_0001_m_000000_0 -> (143, 36) from hadoop-25.c.a.com


Reducer2:=20
=20
2010-05-25 11:34:49,822 INFO org.apache.hadoop.mapred.ReduceTask: Read 6376=
57 bytes from map-output for attempt_201005251129_0001_m_000001_0
2010-05-25 11:34:49,911 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 fr=
om attempt_201005251129_0001_m_000001_0 -> (125, 36) from hadoop-49.c.a.com
2010-05-25 11:34:50,806 INFO org.apache.hadoop.mapred.ReduceTask: attempt_2=
01005251129_0001_r_000008_0: Got 1 new map-outputs
2010-05-25 11:34:54,915 INFO org.apache.hadoop.mapred.ReduceTask: attempt_2=
01005251129_0001_r_000008_0 Scheduled 1 outputs (0 slow hosts and0 dup host=
s)
2010-05-25 11:34:54,920 INFO org.apache.hadoop.mapred.ReduceTask: header: a=
ttempt_201005251129_0001_m_000000_0, compressed len: 1462335, decompressed =
len: 1462331
2010-05-25 11:34:54,920 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling=
 1462331 bytes (1462335 raw bytes) into RAM from attempt_201005251129_0001_=
m_000000_0
2010-05-25 11:34:54,937 INFO org.apache.hadoop.mapred.ReduceTask: Read 1462=
331 bytes from map-output for attempt_201005251129_0001_m_000000_0
2010-05-25 11:34:54,937 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 fr=
om attempt_201005251129_0001_m_000000_0 -> (147, 36) from hadoop-25.c.a.com


The 2 reduce tasks have different task ids and belong to the same job.

Thanks,
Deepika

-----Original Message-----
From: Eric Sammer [mailto:esammer@cloudera.com]=20
Sent: Tuesday, May 25, 2010 8:10 AM
To: general@hadoop.apache.org
Subject: Re: Re: Hash Partitioner

On Mon, May 24, 2010 at 6:32 PM, Deepika Khera <Deepika.Khera@avg.com> wrot=
e:
> Thanks for your response Eric.
>
> I am using hadoop 0.20.2.
>
> Here is what the hashCode() implementation looks like (I actually had the=
 IDE generate it for me)
>
> Main key (for mapper & reducer):
>
> public int hashCode() {
> =A0 =A0 =A0 =A0int result =3D kVersion;
> =A0 =A0 =A0 =A0result =3D 31 * result + (aKey !=3D null ? aKey.hashCode()=
 : 0);
> =A0 =A0 =A0 =A0result =3D 31 * result + (gKey !=3D null ? gKey.hashCode()=
 : 0);
> =A0 =A0 =A0 =A0result =3D 31 * result + (int) (date ^ (date >>> 32));
> =A0 =A0 =A0 =A0result =3D 31 * result + (ma !=3D null ? ma.hashCode() : 0=
);
> =A0 =A0 =A0 =A0result =3D 31 * result + (cl !=3D null ? cl.hashCode() : 0=
);
> =A0 =A0 =A0 =A0return result;
> =A0 =A0}
>
>
> aKey : AKey class
>
>
> =A0 =A0public int hashCode() {
> =A0 =A0 =A0 =A0int result =3D kVersion;
> =A0 =A0 =A0 =A0result =3D 31 * result + (v !=3D null ? v.hashCode() : 0);
> =A0 =A0 =A0 =A0result =3D 31 * result + (s !=3D null ? s.hashCode() : 0);
> =A0 =A0 =A0 =A0result =3D 31 * result + (o !=3D null ? o.hashCode() : 0);
> =A0 =A0 =A0 =A0result =3D 31 * result + (l !=3D null ? l.hashCode() : 0);
> =A0 =A0 =A0 =A0result =3D 31 * result + (e ? 1 : 0); //boolean
> =A0 =A0 =A0 =A0result =3D 31 * result + (li ? 1 : 0); //boolean
> =A0 =A0 =A0 =A0result =3D 31 * result + (aut ? 1 : 0); //boolean
> =A0 =A0 =A0 =A0return result;
> =A0 =A0}
>

Both of these look fine, assuming all the other hashCode()s return the
same value every time.

> When this happens, I do see the same values for the key. Also I am not us=
ing a grouping comparator.

So you see two reduce methods getting the same key with the same
values? That's extremely odd. If this is the case, there's a bug in
Hadoop. Can you find the relevant logs from the reducers where Hadoop
fetches the map output? Does it look like its fetching the same output
twice? Do the two tasks where you see the duplicates have the same
task ID? Can you confirm the reduce tasks are from the same job ID for
us?

> I was wondering since the call to HashPartitioner.getPartition() is done =
from a map task, several of which are running on different machines, is it =
possible that they get a different hashcode and hence get different reducer=
s assigned even when the key is the same.

The hashCode() result should *always* be the same given the same
internal state. In other words, it should be consistent and stable. If
I have a string new String("hello world") it will always have the
exact same hashCode(). If this isn't true, you will get wildly
unpredictable results not just with Hadoop but with Java's
comparators, collections, etc.

--=20
Eric Sammer
phone: +1-917-287-2675
twitter: esammer
data: www.cloudera.com

From general-return-1576-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed May 26 03:08:25 2010
Return-Path: <general-return-1576-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55643 invoked from network); 26 May 2010 03:08:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 May 2010 03:08:24 -0000
Received: (qmail 39527 invoked by uid 500); 26 May 2010 03:08:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39194 invoked by uid 500); 26 May 2010 03:08:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 89515 invoked by uid 99); 26 May 2010 01:45:08 -0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of sysolve@gmail.com designates 209.85.222.200 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:received:in-reply-to
         :references:date:message-id:subject:from:to:content-type;
        bh=zayNBjHIPbwZuBvCh3eJEEgAJqm6+VblqFU/cwJB5sI=;
        b=a+k4j5yUTYtvQgWtuXSvf2ijN9HfTnNlWBw6mw5X3SknuJMTBaEXjkndsgwiDvao/0
         aDyN2UeqJAuOYRj04dI+72G6XiFyMxYp5cxPoAZJpQ7emZLAuShcs8xlFb4+FrrHFXwP
         59gcF8ZmijFqCCwrmNEkOicq4zXhVWt8LdyYg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=jHJRp1mq6PY2RrN6Z9oNvkpY27KAXg6sCFDdz7q9G5V98A2dINepG5sbKZ/lRqOoy1
         BM+1skzfS0qUj3X1koRwsXxCo8J4Huld0PP6+5lVbqSwpoA1AvcnD6Bk0OEeXty6eYec
         NpaDDqdFVYy3N42geseKfUPcDP//GXJvcsGiM=
MIME-Version: 1.0
In-Reply-To: <AANLkTinWLT7IdmnIcrBPuE2OhjFK780twtnTC7T9UTKA@mail.gmail.com>
References: <AANLkTinWLT7IdmnIcrBPuE2OhjFK780twtnTC7T9UTKA@mail.gmail.com>
Date: Wed, 26 May 2010 09:44:42 +0800
Message-ID: <AANLkTimkwpAokI5XyRiYNRfmmtdsFxoucl-5jtG3nfuU@mail.gmail.com>
Subject: why do data-local maps still need to do remote reads?
From: Grace <sysolve@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502cc39f3d52f0487756a3d

--00504502cc39f3d52f0487756a3d
Content-Type: text/plain; charset=ISO-8859-1

Hi all,

According to the map task scheduling rules, it prefers a task with data
local. And seeing the Data-local map counter(in the job report), it does
have a very high locality for all the map tasks. However, when observing the
metrics of DataNode( read_from_local and read_from_remote) , there is higher
remote read rate than the job reported. It seems the Data-local Map counter
is not so accurate as we expected. I wonder when or why it will trigger the
HDFS remote read while already assigning a data-local map task.

Thanks for your time.

Best Regards,
Grace

--00504502cc39f3d52f0487756a3d--

From general-return-1577-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed May 26 04:49:52 2010
Return-Path: <general-return-1577-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76874 invoked from network); 26 May 2010 04:49:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 May 2010 04:49:51 -0000
Received: (qmail 951 invoked by uid 500); 26 May 2010 04:49:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 567 invoked by uid 500); 26 May 2010 04:49:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 558 invoked by uid 99); 26 May 2010 04:49:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 May 2010 04:49:49 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=EXTRA_MPART_TYPE,HTML_MESSAGE,MIME_QP_LONG_LINE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [117.120.16.163] (HELO mail88.messagelabs.com) (117.120.16.163)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 May 2010 04:49:40 +0000
X-VirusChecked: Checked
X-Env-Sender: Anthony.Ikeda@cardlink.com.au
X-Msg-Ref: server-2.tower-88.messagelabs.com!1274849354!30312359!1
X-StarScan-Version: 6.2.4; banners=cardlink.com.au,-,-
X-Originating-IP: [203.36.58.163]
Received: (qmail 27573 invoked from network); 26 May 2010 04:49:15 -0000
Received: from unknown (HELO rmail.5f55.cardlink.local) (203.36.58.163)
  by server-2.tower-88.messagelabs.com with DHE-RSA-AES256-SHA encrypted SMTP; 26 May 2010 04:49:15 -0000
Received: from r-exchange.cardlink.local (r-exchange.cardlink.local [10.3.3.3])
	by rmail.5f55.cardlink.local (8.13.5.20060308/8.13.4) with ESMTP id o4Q4n3S1032074
	for <general@hadoop.apache.org>; Wed, 26 May 2010 14:49:14 +1000 (EST)
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/related;
	boundary="----_=_NextPart_001_01CAFC8E.C9F41A55";
	type="multipart/alternative"
Subject: HBase and Stargate
Date: Wed, 26 May 2010 14:49:12 +1000
Message-ID: <590CDE7A083C4142AF05E96F2FB543BC9734BE@r-exchange.cardlink.local>
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator: 
Thread-Topic: HBase and Stargate
Thread-Index: Acr8jskK0LuaDwLUQLuIMnKZhn8k7Q==
From: "Anthony Ikeda" <Anthony.Ikeda@cardlink.com.au>
To: <general@hadoop.apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

------_=_NextPart_001_01CAFC8E.C9F41A55
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_002_01CAFC8E.C9F41A55"


------_=_NextPart_002_01CAFC8E.C9F41A55
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

I've created a client to connect to the Stargate REST interface which
appears to be working. However, when I tried to create my initial table
it appears that there is no data in the response. Is this expected?

=20

I can't seem to locate any documentation on the Input/Output of calls to
Stargate. The package-summary of the HBase API uses curl as an example
with data being returned at the command line, but each call I make via
JAX-WS the response is empty. When I use curl (on my Mac for example)
nothing is returned to the command line.

=20

Would I be correct in Assuming:

PUT =3D void

POST =3D void

GET =3D String

DELETE =3D void   ?

=20

Anthony Ikeda

Java Analyst/Programmer

Cardlink Services Limited

Level 4, 3 Rider Boulevard

Rhodes NSW 2138

=20

Web: www.cardlink.com.au | Tel: + 61 2 9646 9221 | Fax: + 61 2 9646 9283

=20

=20


**********************************************************************
This e-mail message and any attachments are intended only for the use of t=
he addressee(s) named above and may contain=20information that is privileg=
ed and confidential. If you are not the intended recipient, any display, d=
issemination, distribution, or copying is strictly prohibited.   If you be=
lieve you have received this e-mail message in error, please immediately n=
otify the sender by replying to this e-mail message or by telephone to (02=
) 9646 9222. Please delete the email and any attachments and do not retain=
 the email or any attachments in any form.
**********************************************************************
------_=_NextPart_002_01CAFC8E.C9F41A55
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-mic=
rosoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word=
" xmlns:x=3D"urn:schemas-microsoft-com:office:excel" xmlns:p=3D"urn:schema=
s-microsoft-com:office:powerpoint" xmlns:a=3D"urn:schemas-microsoft-com:of=
fice:access" xmlns:dt=3D"uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:=
s=3D"uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs=3D"urn:schemas-mi=
crosoft-com:rowset" xmlns:z=3D"#RowsetSchema" xmlns:b=3D"urn:schemas-micro=
soft-com:office:publisher" xmlns:ss=3D"urn:schemas-microsoft-com:office:sp=
readsheet" xmlns:c=3D"urn:schemas-microsoft-com:office:component:spreadshe=
et" xmlns:odc=3D"urn:schemas-microsoft-com:office:odc" xmlns:oa=3D"urn:sch=
emas-microsoft-com:office:activation" xmlns:html=3D"http://www.w3.org/TR/R=
EC-html40" xmlns:q=3D"http://schemas.xmlsoap.org/soap/envelope/" xmlns:rtc=
=3D"http://microsoft.com/officenet/conferencing" xmlns:D=3D"DAV:" xmlns:Re=
pl=3D"http://schemas.microsoft.com/repl/" xmlns:mt=3D"http://schemas.micro=
soft.com/sharepoint/soap/meetings/" xmlns:x2=3D"http://schemas.microsoft.c=
om/office/excel/2003/xml" xmlns:ppda=3D"http://www.passport.com/NameSpace.=
xsd" xmlns:ois=3D"http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns=
:dir=3D"http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds=3D=
"http://www.w3.org/2000/09/xmldsig#" xmlns:dsp=3D"http://schemas.microsoft=
.com/sharepoint/dsp" xmlns:udc=3D"http://schemas.microsoft.com/data/udc" x=
mlns:xsd=3D"http://www.w3.org/2001/XMLSchema" xmlns:sub=3D"http://schemas.=
microsoft.com/sharepoint/soap/2002/1/alerts/" xmlns:ec=3D"http://www.w3.or=
g/2001/04/xmlenc#" xmlns:sp=3D"http://schemas.microsoft.com/sharepoint/" x=
mlns:sps=3D"http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsi=3D"ht=
tp://www.w3.org/2001/XMLSchema-instance" xmlns:udcs=3D"http://schemas.micr=
osoft.com/data/udc/soap" xmlns:udcxf=3D"http://schemas.microsoft.com/data/=
udc/xmlfile" xmlns:udcp2p=3D"http://schemas.microsoft.com/data/udc/parttop=
art" xmlns:wf=3D"http://schemas.microsoft.com/sharepoint/soap/workflow/" x=
mlns:dsss=3D"http://schemas.microsoft.com/office/2006/digsig-setup" xmlns:=
dssi=3D"http://schemas.microsoft.com/office/2006/digsig" xmlns:mdssi=3D"ht=
tp://schemas.openxmlformats.org/package/2006/digital-signature" xmlns:mver=
=3D"http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:m=3D=
"http://schemas.microsoft.com/office/2004/12/omml" xmlns:mrels=3D"http://s=
chemas.openxmlformats.org/package/2006/relationships" xmlns:spwp=3D"http:/=
/microsoft.com/sharepoint/webpartpages" xmlns:ex12t=3D"http://schemas.micr=
osoft.com/exchange/services/2006/types" xmlns:ex12m=3D"http://schemas.micr=
osoft.com/exchange/services/2006/messages" xmlns:pptsl=3D"http://schemas.m=
icrosoft.com/sharepoint/soap/SlideLibrary/" xmlns:spsl=3D"http://microsoft=
.com/webservices/SharePointPortalServer/PublishedLinksService" xmlns:Z=3D"=
urn:schemas-microsoft-com:" xmlns:st=3D"&#1;" xmlns=3D"http://www.w3.org/T=
R/REC-html40">

<head>
<meta http-equiv=3DContent-Type content=3D"text/html; charset=3Dus-ascii">=

<meta name=3DGenerator content=3D"Microsoft Word 12 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<style>
<!--
 /* Font Definitions */
 @font-face
=09{font-family:"Cambria Math";
=09panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
=09{font-family:Calibri;
=09panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
=09{font-family:Tahoma;
=09panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
=09{font-family:Consolas;
=09panose-1:2 11 6 9 2 2 4 3 2 4;}
 /* Style Definitions */
 p.MsoNormal, li.MsoNormal, div.MsoNormal
=09{margin:0cm;
=09margin-bottom:.0001pt;
=09font-size:11.0pt;
=09font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
=09{mso-style-priority:99;
=09color:blue;
=09text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
=09{mso-style-priority:99;
=09color:purple;
=09text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
=09{mso-style-priority:99;
=09mso-style-link:"Balloon Text Char";
=09margin:0cm;
=09margin-bottom:.0001pt;
=09font-size:8.0pt;
=09font-family:"Tahoma","sans-serif";}
span.EmailStyle17
=09{mso-style-type:personal-compose;
=09font-family:"Calibri","sans-serif";
=09color:windowtext;}
span.BalloonTextChar
=09{mso-style-name:"Balloon Text Char";
=09mso-style-priority:99;
=09mso-style-link:"Balloon Text";
=09font-family:"Tahoma","sans-serif";}
.MsoChpDefault
=09{mso-style-type:export-only;}
@page Section1
=09{size:612.0pt 792.0pt;
=09margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.Section1
=09{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
 <o:shapedefaults v:ext=3D"edit" spidmax=3D"2050" />
</xml><![endif]--><!--[if gte mso 9]><xml>
 <o:shapelayout v:ext=3D"edit">
  <o:idmap v:ext=3D"edit" data=3D"1" />
 </o:shapelayout></xml><![endif]-->
</head>

<body lang=3DEN-AU link=3Dblue vlink=3Dpurple>

<div class=3DSection1>

<p class=3DMsoNormal>I&#8217;ve created a client to connect to the Stargat=
e REST
interface which appears to be working. However, when I tried to create my
initial table it appears that there is no data in the response. Is this
expected?<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>I can&#8217;t seem to locate any documentation on the=

Input/Output of calls to Stargate. The package-summary of the HBase API us=
es
curl as an example with data being returned at the command line, but each =
call
I make via JAX-WS the response is empty. When I use curl (on my Mac for
example) nothing is returned to the command line.<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>Would I be correct in Assuming:<o:p></o:p></p>

<p class=3DMsoNormal>PUT =3D void<o:p></o:p></p>

<p class=3DMsoNormal>POST =3D void<o:p></o:p></p>

<p class=3DMsoNormal>GET =3D String<o:p></o:p></p>

<p class=3DMsoNormal>DELETE =3D void&nbsp;&nbsp; ?<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal><span lang=3DEN-US>Anthony Ikeda<o:p></o:p></span></p=
>

<p class=3DMsoNormal><span lang=3DEN-US>Java Analyst/Programmer<o:p></o:p>=
</span></p>

<p class=3DMsoNormal><span lang=3DEN-US>Cardlink Services Limited<o:p></o:=
p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US style=3D'font-size:10.5pt;font-fam=
ily:Consolas'>Level
4, 3 Rider Boulevard<o:p></o:p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US style=3D'font-size:10.5pt;font-fam=
ily:Consolas'>Rhodes
NSW 2138<o:p></o:p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US style=3D'font-size:10.5pt;font-fam=
ily:Consolas'>&nbsp;<o:p></o:p></span></p>

<p class=3DMsoNormal><span lang=3DEN-US style=3D'font-size:10.5pt;font-fam=
ily:Consolas'>Web:
<a href=3D"http://www.cardlink.com.au">www.cardlink.com.au</a> | Tel: + 61=
 2 9646
9221 | Fax: + 61 2 9646 9283<o:p></o:p></span></p>

<p class=3DMsoNormal><img border=3D0 width=3D155 height=3D108 id=3D"Pictur=
e_x0020_1"
src=3D"cid:image001.gif@01CAFCE2.024A9090" alt=3D"logo_cardlink1"><span la=
ng=3DEN-US><o:p></o:p></span></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

</div>

<BR>
**********************************************************************<BR>=

This e-mail message and any attachments are intended only for the use of t=
he addressee(s) named above and may contain information that is privileged=
 and confidential. If you are not the intended recipient, any display, dis=
semination, distribution, or copying is strictly prohibited.   If you beli=
eve you have received this e-mail message in error, please immediately not=
ify the sender by replying to this e-mail message or by telephone to (02) =
9646 9222. Please delete the email and any attachments and do not retain t=
he email or any attachments in any form.<BR>
**********************************************************************<BR>=

</body>

</html>

------_=_NextPart_002_01CAFC8E.C9F41A55--

------_=_NextPart_001_01CAFC8E.C9F41A55--

From general-return-1578-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed May 26 04:52:45 2010
Return-Path: <general-return-1578-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77162 invoked from network); 26 May 2010 04:52:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 May 2010 04:52:45 -0000
Received: (qmail 2521 invoked by uid 500); 26 May 2010 04:52:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2386 invoked by uid 500); 26 May 2010 04:52:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2378 invoked by uid 99); 26 May 2010 04:52:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 May 2010 04:52:43 +0000
X-ASF-Spam-Status: No, hits=-0.1 required=10.0
	tests=HTML_MESSAGE,MIME_QP_LONG_LINE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [117.120.16.147] (HELO mail62.messagelabs.com) (117.120.16.147)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 May 2010 04:52:36 +0000
X-VirusChecked: Checked
X-Env-Sender: Anthony.Ikeda@cardlink.com.au
X-Msg-Ref: server-11.tower-62.messagelabs.com!1274849530!14692121!1
X-StarScan-Version: 6.2.4; banners=cardlink.com.au,-,-
X-Originating-IP: [203.36.58.163]
Received: (qmail 6112 invoked from network); 26 May 2010 04:52:10 -0000
Received: from unknown (HELO rmail.5f55.cardlink.local) (203.36.58.163)
  by server-11.tower-62.messagelabs.com with DHE-RSA-AES256-SHA encrypted SMTP; 26 May 2010 04:52:10 -0000
Received: from r-exchange.cardlink.local (r-exchange.cardlink.local [10.3.3.3])
	by rmail.5f55.cardlink.local (8.13.5.20060308/8.13.4) with ESMTP id o4Q4qAfI017831
	for <general@hadoop.apache.org>; Wed, 26 May 2010 14:52:10 +1000 (EST)
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01CAFC8F.33082EB7"
Subject: RE: HBase and Stargate
Date: Wed, 26 May 2010 14:52:09 +1000
Message-ID: <590CDE7A083C4142AF05E96F2FB543BC9734C1@r-exchange.cardlink.local>
In-Reply-To: <590CDE7A083C4142AF05E96F2FB543BC9734BE@r-exchange.cardlink.local>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: HBase and Stargate
Thread-Index: Acr8jskK0LuaDwLUQLuIMnKZhn8k7QAAFw8g
References: <590CDE7A083C4142AF05E96F2FB543BC9734BE@r-exchange.cardlink.local>
From: "Anthony Ikeda" <Anthony.Ikeda@cardlink.com.au>
To: <general@hadoop.apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

------_=_NextPart_001_01CAFC8F.33082EB7
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

I've also noticed that creating a scanner is not returning me the
scanner URI.

=20

From: Anthony Ikeda [mailto:Anthony.Ikeda@cardlink.com.au]=20
Sent: Wednesday, 26 May 2010 2:49 PM
To: general@hadoop.apache.org
Subject: HBase and Stargate

=20

I've created a client to connect to the Stargate REST interface which
appears to be working. However, when I tried to create my initial table
it appears that there is no data in the response. Is this expected?

=20

I can't seem to locate any documentation on the Input/Output of calls to
Stargate. The package-summary of the HBase API uses curl as an example
with data being returned at the command line, but each call I make via
JAX-WS the response is empty. When I use curl (on my Mac for example)
nothing is returned to the command line.

=20

Would I be correct in Assuming:

PUT =3D void

POST =3D void

GET =3D String

DELETE =3D void   ?

=20

Anthony Ikeda

Java Analyst/Programmer

Cardlink Services Limited

Level 4, 3 Rider Boulevard

Rhodes NSW 2138

=20

Web: www.cardlink.com.au | Tel: + 61 2 9646 9221 | Fax: + 61 2 9646 9283

logo_cardlink1

=20


**********************************************************************
This e-mail message and any attachments are intended only for the use of
the addressee(s) named above and may contain information that is
privileged and confidential. If you are not the intended recipient, any
display, dissemination, distribution, or copying is strictly prohibited.
If you believe you have received this e-mail message in error, please
immediately notify the sender by replying to this e-mail message or by
telephone to (02) 9646 9222. Please delete the email and any attachments
and do not retain the email or any attachments in any form.
**********************************************************************

_____________________________________________________________________=20
This e-mail has been scanned for viruses by MCI's Internet Managed=20
Scanning Services - powered by MessageLabs. For further information=20
visit http://www.mci.com


**********************************************************************
This e-mail message and any attachments are intended only for the use of t=
he addressee(s) named above and may contain information that is privileged=
 and confidential. If you are not the intended recipient, any display, dis=
semination, distribution, or copying is strictly prohibited.   If you beli=
eve you have received this e-mail message in error, please immediately not=
ify the sender by replying to this e-mail message or by telephone to (02) =
9646 9222. Please delete the email and any attachments and do not retain t=
he email or any attachments in any form.
**********************************************************************
------_=_NextPart_001_01CAFC8F.33082EB7--

From general-return-1579-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed May 26 06:42:26 2010
Return-Path: <general-return-1579-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12171 invoked from network); 26 May 2010 06:42:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 May 2010 06:42:25 -0000
Received: (qmail 82026 invoked by uid 500); 26 May 2010 06:42:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81698 invoked by uid 500); 26 May 2010 06:42:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81683 invoked by uid 99); 26 May 2010 06:42:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 May 2010 06:42:23 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jgray@facebook.com designates 69.63.179.25 as permitted sender)
Received: from [69.63.179.25] (HELO mailout-sf2p.facebook.com) (69.63.179.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 May 2010 06:42:16 +0000
Received: from mail.thefacebook.com ([192.168.18.105])
	by pp02.snc1.tfbnw.net (8.14.3/8.14.3) with ESMTP id o4Q6fAmE020755
	(version=TLSv1/SSLv3 cipher=RC4-MD5 bits=128 verify=NOT);
	Tue, 25 May 2010 23:41:10 -0700
Received: from SC-MBXC1.TheFacebook.com ([192.168.18.102]) by
 sc-hub02.TheFacebook.com ([192.168.18.105]) with mapi; Tue, 25 May 2010
 23:41:48 -0700
From: Jonathan Gray <jgray@facebook.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>,
        "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "Anthony.Ikeda@cardlink.com.au" <Anthony.Ikeda@cardlink.com.au>
Date: Tue, 25 May 2010 23:41:44 -0700
Subject: RE: HBase and Stargate
Thread-Topic: HBase and Stargate
Thread-Index: Acr8jskK0LuaDwLUQLuIMnKZhn8k7QAD4eMA
Message-ID: <8D66B74984F9564BBB25C3C67D630F2D68C7CEAC@SC-MBXC1.TheFacebook.com>
References: <590CDE7A083C4142AF05E96F2FB543BC9734BE@r-exchange.cardlink.local>
In-Reply-To: <590CDE7A083C4142AF05E96F2FB543BC9734BE@r-exchange.cardlink.local>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_8D66B74984F9564BBB25C3C67D630F2D68C7CEACSCMBXC1TheFaceb_"
MIME-Version: 1.0
X-Proofpoint-Virus-Version: vendor=fsecure engine=1.12.8161:2.4.5,1.2.40,4.0.166
 definitions=2010-05-26_01:2010-02-06,2010-05-25,2010-05-25 signatures=0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_8D66B74984F9564BBB25C3C67D630F2D68C7CEACSCMBXC1TheFaceb_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Redirecting to user@hbase mailing list where you might get an answer.

From: Anthony Ikeda [mailto:Anthony.Ikeda@cardlink.com.au]
Sent: Tuesday, May 25, 2010 9:49 PM
To: general@hadoop.apache.org
Subject: HBase and Stargate

I've created a client to connect to the Stargate REST interface which appea=
rs to be working. However, when I tried to create my initial table it appea=
rs that there is no data in the response. Is this expected?

I can't seem to locate any documentation on the Input/Output of calls to St=
argate. The package-summary of the HBase API uses curl as an example with d=
ata being returned at the command line, but each call I make via JAX-WS the=
 response is empty. When I use curl (on my Mac for example) nothing is retu=
rned to the command line.

Would I be correct in Assuming:
PUT =3D void
POST =3D void
GET =3D String
DELETE =3D void   ?

Anthony Ikeda
Java Analyst/Programmer
Cardlink Services Limited
Level 4, 3 Rider Boulevard
Rhodes NSW 2138

Web: www.cardlink.com.au<http://www.cardlink.com.au> | Tel: + 61 2 9646 922=
1 | Fax: + 61 2 9646 9283
[cid:image001.gif@01CAFCE2.024A9090]


**********************************************************************
This e-mail message and any attachments are intended only for the use of th=
e addressee(s) named above and may contain information that is privileged a=
nd confidential. If you are not the intended recipient, any display, dissem=
ination, distribution, or copying is strictly prohibited. If you believe yo=
u have received this e-mail message in error, please immediately notify the=
 sender by replying to this e-mail message or by telephone to (02) 9646 922=
2. Please delete the email and any attachments and do not retain the email =
or any attachments in any form.
**********************************************************************

--_000_8D66B74984F9564BBB25C3C67D630F2D68C7CEACSCMBXC1TheFaceb_--

From general-return-1580-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed May 26 10:25:12 2010
Return-Path: <general-return-1580-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78209 invoked from network); 26 May 2010 10:25:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 May 2010 10:25:12 -0000
Received: (qmail 34551 invoked by uid 500); 26 May 2010 10:25:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34319 invoked by uid 500); 26 May 2010 10:25:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34311 invoked by uid 99); 26 May 2010 10:25:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 May 2010 10:25:08 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 May 2010 10:25:00 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id B9034B7EBB
	for <general@hadoop.apache.org>; Wed, 26 May 2010 11:24:39 +0100 (BST)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id gDZ6Mp8VwFLF for <general@hadoop.apache.org>;
	Wed, 26 May 2010 11:24:28 +0100 (BST)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 979F4B7EB4
	for <general@hadoop.apache.org>; Wed, 26 May 2010 11:24:28 +0100 (BST)
MailScanner-NULL-Check: 1275474255.89919@i0laM/zKvvGP4s+Uc+iHtg
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o4QAOEnQ005323
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Wed, 26 May 2010 11:24:15 +0100 (BST)
Message-ID: <4BFCF6CF.2000605@apache.org>
Date: Wed, 26 May 2010 11:24:15 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100228)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: [DISCUSSION] Proposal for making core Hadoop changes
References: <AANLkTimLB-YNWnMwa93YL8Lpg9hJLxrCKbYL3YUr6de8@mail.gmail.com>	 <66BC29ED-35C1-4916-BDFE-C768810C6EA6@mac.com>	 <AANLkTimCkqddJ521jIPp0dVHC__vZ2PcavAIrNl334L0@mail.gmail.com>	 <4BFBFAC3.2040807@apache.org> <AANLkTilhcUYlFRwc1vc7-sFybjUw5c_uPosptIGb-Gn1@mail.gmail.com>
In-Reply-To: <AANLkTilhcUYlFRwc1vc7-sFybjUw5c_uPosptIGb-Gn1@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o4QAOEnQ005323
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

Eli Collins wrote:

> The cost of adding features has gotten high anyway (even without
> branching). It's a classic trade-off -- merge overhead vs moving
> faster without burdening others -- as the overhead imposed on others
> increases, and tools (git) make it easier to live and collaborate on
> branches it makes more sense 

maybe, but if you are trying to keep >1 branch in sync, all the low cost 
refactorings become expensive to perform
  -renaming variables
  -hitting the reformat-code button to align the code with the project 
layout rules
  -moving methods around
Life is simplest if you own the entire codebase and can move stuff 
around without any discussion. Closed source projects can do that, but 
even then it annoys other team members. In any OSS project, keeping 
stuff more stable makes it easier to take in third party patches, and 
ensures that stack traces from various versions all point to roughly the 
same code, always handy. Once you try to keep multiple branches alive, 
it becomes very hard to do big changes in trunk.

>(you don't need a team of engineers or
> dedicated merge engineer to maintain the branch).

No, but I'd estimate the cost of merging at 1-2 days work a week just to 
pull in the code *and identify why the tests are failing*. Git may be 
better at merging in changes, but if Hadoop doesn't work on my machine 
after the merge, I need to identify whether its my code, the merged 
code, some machine quirk, etc. It's the testing that is the problem for 
me, not the
merge effort. That's the Hadoop own tests any my own functional test 
suites, the ones that bring up clusters and push work through. Those are 
the troublespots, as they do things that hadoop's own tests don't do, 
like as for all the JSP pages.

> Might find the
> following interesting:
> http://incubator.apache.org/learn/rules-for-revolutionaries.html

There's a long story behind JDD's paper, I'm glad you have read it, it 
does lay out what is effectively the ASF process for effecting 
significant change -but it doesn't imply that's the only process for 
having changes.

One of the big issues that in any successful project it becomes hard to 
do a big rewrite, and you end up with what was done early on, despite 
known issues. The "Some Thoughts on Ant 1.3 and 2.0" discussion is 
related to this  we -and I wasn't a committer at this time, just a user- 
weren't able to do the big rework so we are left today with the design 
errors of the past (like the way undefined properties just get retained 
as ${undefined.property} instead of some kind of error appearing):
http://www.mail-archive.com/ant-dev@jakarta.apache.org/msg05984.html

I think gradual evolution in trunk is good, it lets people play with 
what's coming in. Having lots of separate branches and everyone's 
private release being a merge of many patches that you choose is bad. 
Because it means my version != your version != anyone else's, which 
implies that your tests mean nothing to me unless I also test at scale. 
Which I can do, but with different hardware and network configs from 
other people, it's still tricky to assign blame. Is it my merge that 
isn't working, is it some quirk of virtualisation underneath, or is it 
just this week's trunk playing up?






From general-return-1581-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed May 26 16:14:02 2010
Return-Path: <general-return-1581-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19291 invoked from network); 26 May 2010 16:14:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 May 2010 16:14:02 -0000
Received: (qmail 22127 invoked by uid 500); 26 May 2010 16:14:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22056 invoked by uid 500); 26 May 2010 16:14:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22048 invoked by uid 99); 26 May 2010 16:14:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 May 2010 16:14:00 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.212.176] (HELO mail-px0-f176.google.com) (209.85.212.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 May 2010 16:13:52 +0000
Received: by pxi10 with SMTP id 10so4000088pxi.35
        for <general@hadoop.apache.org>; Wed, 26 May 2010 09:13:31 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.143.169.5 with SMTP id w5mr5712233wfo.222.1274890411112; Wed, 
	26 May 2010 09:13:31 -0700 (PDT)
Received: by 10.143.16.21 with HTTP; Wed, 26 May 2010 09:13:31 -0700 (PDT)
In-Reply-To: <4BFCF6CF.2000605@apache.org>
References: <AANLkTimLB-YNWnMwa93YL8Lpg9hJLxrCKbYL3YUr6de8@mail.gmail.com>
	 <66BC29ED-35C1-4916-BDFE-C768810C6EA6@mac.com>
	 <AANLkTimCkqddJ521jIPp0dVHC__vZ2PcavAIrNl334L0@mail.gmail.com>
	 <4BFBFAC3.2040807@apache.org>
	 <AANLkTilhcUYlFRwc1vc7-sFybjUw5c_uPosptIGb-Gn1@mail.gmail.com>
	 <4BFCF6CF.2000605@apache.org>
Date: Wed, 26 May 2010 09:13:31 -0700
Message-ID: <AANLkTilLOmfMwuIFc2kzsXMQ7yhbXtrA0jhb3pigTEht@mail.gmail.com>
Subject: Re: [DISCUSSION] Proposal for making core Hadoop changes
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

> No, but I'd estimate the cost of merging at 1-2 days work a week just to
> pull in the code *and identify why the tests are failing*. Git may be better
> at merging in changes, but if Hadoop doesn't work on my machine after the
> merge, I need to identify whether its my code, the merged code, some machine
> quirk, etc. It's the testing that is the problem for me, not the
> merge effort. That's the Hadoop own tests any my own functional test suites,
> the ones that bring up clusters and push work through. Those are the
> troublespots, as they do things that hadoop's own tests don't do, like as
> for all the JSP pages.

I've lived off a git branch of common/hdfs for half a year with a big
uncommitted patch, it's no where near 1-2 days of effort per week to
merge in changes from trunk. If the tests are passing on trunk, and
they fail after your merge then those are real test failures due to
your change (and therefore should require effort). The issues with
your internal tests failing due to changes on trunk is the same
whether you merge or you just do an update - you have to update before
checking in the patch anyway - so that issue is about the state of
trunk when you merge or update, rather than about being on a branch.

>
>> Might find the
>> following interesting:
>> http://incubator.apache.org/learn/rules-for-revolutionaries.html
>
> There's a long story behind JDD's paper, I'm glad you have read it, it does
> lay out what is effectively the ASF process for effecting significant change
> -but it doesn't imply that's the only process for having changes.
>

Just to be clear I don't mean imply that branches are the only process
for making changes. Interesting that this is considered the effective
ASF process, it hasn't seemed to me that recent big features on hadoop
have used it, only one I'm aware of that was done on a branch was
append.

> I think gradual evolution in trunk is good, it lets people play with what's
> coming in. Having lots of separate branches and everyone's private release
> being a merge of many patches that you choose is bad.

Agreed.  Personally I don't think people should release from branches.
And in practice I don't think you'll see lots of branches, people can
and would still develop on trunk. Getting changes merged from a branch
back to trunk before the whole branch is merged is a good thing, the
whole branch may never be merged and that's OK too. Branches are a
mechanism, releases are policy.

Thanks,
Eli

From general-return-1582-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu May 27 05:46:44 2010
Return-Path: <general-return-1582-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23980 invoked from network); 27 May 2010 05:46:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 May 2010 05:46:43 -0000
Received: (qmail 49777 invoked by uid 500); 27 May 2010 05:46:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49551 invoked by uid 500); 27 May 2010 05:46:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49543 invoked by uid 99); 27 May 2010 05:46:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 May 2010 05:46:38 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of vidur@students.iiit.ac.in designates 121.242.23.201 as permitted sender)
Received: from [121.242.23.201] (HELO students.iiit.ac.in) (121.242.23.201)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 May 2010 05:46:31 +0000
Received: from students.iiit.ac.in (localhost.localdomain [127.0.0.1])
	by students.iiit.ac.in (8.13.8/8.13.8) with ESMTP id o4R5k9hv006692
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Thu, 27 May 2010 11:16:09 +0530
Received: (from apache@localhost)
	by students.iiit.ac.in (8.13.8/8.14.1/Submit) id o4R5k8Z6006690;
	Thu, 27 May 2010 11:16:08 +0530
X-Authentication-Warning: students.iiit.ac.in: apache set sender to vidur@localhost using -f
Received: from 125.16.17.151 (proxying for unknown)
        (SquirrelMail authenticated user vidur)
        by students.iiit.ac.in with HTTP;
        Thu, 27 May 2010 11:16:08 +0530 (IST)
Message-ID: <60655.125.16.17.151.1274939168.squirrel@students.iiit.ac.in>
Date: Thu, 27 May 2010 11:16:08 +0530 (IST)
Subject: HDFS API's
From: "Vidur Goyal" <vidur@students.iiit.ac.in>
To: general@hadoop.apache.org
User-Agent: SquirrelMail/1.4.8-4.el5
MIME-Version: 1.0
Content-Type: text/plain;charset=iso-8859-1
Content-Transfer-Encoding: 8bit
X-Priority: 3 (Normal)
Importance: Normal
X-Spam-Level: *
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on students.iiit.ac.in
X-Virus-Checked: Checked by ClamAV on apache.org
X-Old-Spam-Status: No, score=1.9 required=5.0 tests=ALL_TRUSTED,FH_DATE_PAST_20XX
	autolearn=no version=3.2.5

I was trying to run hdfs_test.c present in $HADOOP_HOME/src/c++/libhdfs
directory of hadoop. I am facing problems running it properly. Can
somebody guide me how to run it.

vidur



From general-return-1583-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu May 27 08:34:41 2010
Return-Path: <general-return-1583-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70034 invoked from network); 27 May 2010 08:34:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 May 2010 08:34:41 -0000
Received: (qmail 25430 invoked by uid 500); 27 May 2010 08:34:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25126 invoked by uid 500); 27 May 2010 08:34:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25118 invoked by uid 99); 27 May 2010 08:34:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 May 2010 08:34:37 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=SPF_PASS,TO_NO_BRKTS_DIRECT
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [193.252.149.37] (HELO smtpout01s1.x-echo.com) (193.252.149.37)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 May 2010 08:34:29 +0000
X-Spam-Level: 
Message-ID: <4BFE3004.5000603@echo.fr>
Date: Thu, 27 May 2010 10:40:36 +0200
From: Varene Olivier <varene@echo.fr>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Mapper Reducer : Unit Test and mocking with static variables
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-AV-Checked: ClamAV using ClamSMTP
X-Virus-Checked: Checked by ClamAV on apache.org
X-Old-Spam-Status: No, score=-4.3 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00
	autolearn=ham version=3.2.3

Hello to all,

first of all many thanks for this great piece of software you are all
contributing to :)

I am actually creating a program in Hadoop MapReduce, but I intend to
massively use JUnit Tests to validate my code.

To test a Mapper, I mock a Mapper.Context object and run the Mapper.map
function with it, but inside my Mapper.map function, I am using Counters 
to maintain internal statistics. The update of those counters make the 
test fails with a null pointer (JavaNullPointerException).
There is an issue "Counters" of variable visibility from my test ...

Do you have any idea how to mock/validate/overcome this issue ?



Thanks a lots
Best Regards
Olivier Varene


: Code :
--------

private static enum Counters {NBLINES};
private static IntWritable one = new IntWritable(1);
private static LongWritable oneL = new LongWritable(1);


// Mapper
/**
  * outputs 1 for each line encountered
  */
public static class LCMapper extends
	Mapper<LongWritable, Text, IntWritable, LongWritable>
{
   private final static IntWritable one = new IntWritable(1);
   private final static LongWritable oneL = new LongWritable(1);
   /**
   * for each line encountered outputs 1 for the key 1
   */
   public void map(LongWritable p_key, Text p_inputs, Context p_context)
	throws IOException, InterruptedException
   {
     p_context.getCounter(Counters.NBLINES).increment(1);
     p_context.write(one,oneL);	
   } // end map
} // end Mapper

: Test :
--------

@Test
public void testLCMapper() throws IOException, InterruptedException
{	
   LCMapper mapper = new LCMapper();
   Text value = new Text("ligneA\nlignB\n");
   // mock the map job execution context
   Context context = mock(Context.class);
   try {
     mapper.map(null, value, context);
   } catch (NullPointerException e)
   {
     // exception raised by context.getCounter(...) null pointer
     // HOW TO MOCK ???
     // Counter is a static enum from the englobing class ...
     // issue with variable visibility (from map function) in this test
   }
   // verify that the Mapper issued (one,oneL) pair
   verify(context).write(one,oneL);
		
} // end testLCMapper



From general-return-1584-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu May 27 16:07:04 2010
Return-Path: <general-return-1584-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 28872 invoked from network); 27 May 2010 16:07:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 May 2010 16:07:04 -0000
Received: (qmail 85701 invoked by uid 500); 27 May 2010 16:07:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85637 invoked by uid 500); 27 May 2010 16:07:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85629 invoked by uid 99); 27 May 2010 16:07:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 May 2010 16:07:01 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=SPF_PASS,TO_NO_BRKTS_DIRECT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [193.252.149.37] (HELO smtpout01s1.x-echo.com) (193.252.149.37)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 May 2010 16:06:55 +0000
X-Spam-Level: 
Message-ID: <4BFE9A0F.9010606@echo.fr>
Date: Thu, 27 May 2010 18:13:03 +0200
From: Varene Olivier <varene@echo.fr>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Mapper Reducer : Unit Test and mocking with static variables
References: <4BFE3004.5000603@echo.fr>
In-Reply-To: <4BFE3004.5000603@echo.fr>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit
X-AV-Checked: ClamAV using ClamSMTP
X-Old-Spam-Status: No, score=-4.3 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00
	autolearn=ham version=3.2.3

FYI
I am on Hadoop 0.20.2, JUnit 4


Varene Olivier a écrit :
> Hello to all,
> 
> first of all many thanks for this great piece of software you are all
> contributing to :)
> 
> I am actually creating a program in Hadoop MapReduce, but I intend to
> massively use JUnit Tests to validate my code.
> 
> To test a Mapper, I mock a Mapper.Context object and run the Mapper.map
> function with it, but inside my Mapper.map function, I am using Counters 
> to maintain internal statistics. The update of those counters make the 
> test fails with a null pointer (JavaNullPointerException).
> There is an issue "Counters" of variable visibility from my test ...
> 
> Do you have any idea how to mock/validate/overcome this issue ?
> 
> 
> 
> Thanks a lots
> Best Regards
> Olivier Varene
> 
> 
> : Code :
> --------
> 
> private static enum Counters {NBLINES};
> private static IntWritable one = new IntWritable(1);
> private static LongWritable oneL = new LongWritable(1);
> 
> 
> // Mapper
> /**
>  * outputs 1 for each line encountered
>  */
> public static class LCMapper extends
>     Mapper<LongWritable, Text, IntWritable, LongWritable>
> {
>   private final static IntWritable one = new IntWritable(1);
>   private final static LongWritable oneL = new LongWritable(1);
>   /**
>   * for each line encountered outputs 1 for the key 1
>   */
>   public void map(LongWritable p_key, Text p_inputs, Context p_context)
>     throws IOException, InterruptedException
>   {
>     p_context.getCounter(Counters.NBLINES).increment(1);
>     p_context.write(one,oneL);   
>   } // end map
> } // end Mapper
> 
> : Test :
> --------
> 
> @Test
> public void testLCMapper() throws IOException, InterruptedException
> {   
>   LCMapper mapper = new LCMapper();
>   Text value = new Text("ligneA\nlignB\n");
>   // mock the map job execution context
>   Context context = mock(Context.class);
>   try {
>     mapper.map(null, value, context);
>   } catch (NullPointerException e)
>   {
>     // exception raised by context.getCounter(...) null pointer
>     // HOW TO MOCK ???
>     // Counter is a static enum from the englobing class ...
>     // issue with variable visibility (from map function) in this test
>   }
>   // verify that the Mapper issued (one,oneL) pair
>   verify(context).write(one,oneL);
>        
> } // end testLCMapper
> 
> 

From general-return-1585-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri May 28 00:13:43 2010
Return-Path: <general-return-1585-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93933 invoked from network); 28 May 2010 00:13:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 May 2010 00:13:42 -0000
Received: (qmail 56141 invoked by uid 500); 28 May 2010 00:13:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56027 invoked by uid 500); 28 May 2010 00:13:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56019 invoked by uid 99); 28 May 2010 00:13:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 May 2010 00:13:41 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 May 2010 00:13:34 +0000
Received: by pvf33 with SMTP id 33so382884pvf.35
        for <general@hadoop.apache.org>; Thu, 27 May 2010 17:13:13 -0700 (PDT)
Received: by 10.141.188.30 with SMTP id q30mr8566067rvp.212.1275005593054; 
	Thu, 27 May 2010 17:13:13 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.140.158.21 with HTTP; Thu, 27 May 2010 17:12:53 -0700 (PDT)
In-Reply-To: <4BFE9A0F.9010606@echo.fr>
References: <4BFE3004.5000603@echo.fr> <4BFE9A0F.9010606@echo.fr>
From: Aaron Kimball <aaron@cloudera.com>
Date: Thu, 27 May 2010 17:12:53 -0700
Message-ID: <AANLkTimUvNhuga7Zb2dflF6TLf4l1mgq7NVvzTBi2lg6@mail.gmail.com>
Subject: Re: Mapper Reducer : Unit Test and mocking with static variables
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd1720075ef4904879c5f9e
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd1720075ef4904879c5f9e
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Varene,

You might want to check out MRUnit. It's a unit test harness that contains
mock objects for the context & other associated classes, and works with
JUnit.

It's included in the (unreleased) Hadoop 0.21, as well as Cloudera's
Distribution for Hadoop. See
http://archive.cloudera.com/docs/mrunit/index.html for MRUnit documentation=
.

Cheers,
- Aaron

On Thu, May 27, 2010 at 9:13 AM, Varene Olivier <varene@echo.fr> wrote:

> FYI
> I am on Hadoop 0.20.2, JUnit 4
>
>
> Varene Olivier a =E9crit :
>
>  Hello to all,
>>
>> first of all many thanks for this great piece of software you are all
>> contributing to :)
>>
>> I am actually creating a program in Hadoop MapReduce, but I intend to
>> massively use JUnit Tests to validate my code.
>>
>> To test a Mapper, I mock a Mapper.Context object and run the Mapper.map
>> function with it, but inside my Mapper.map function, I am using Counters
>> to maintain internal statistics. The update of those counters make the t=
est
>> fails with a null pointer (JavaNullPointerException).
>> There is an issue "Counters" of variable visibility from my test ...
>>
>> Do you have any idea how to mock/validate/overcome this issue ?
>>
>>
>>
>> Thanks a lots
>> Best Regards
>> Olivier Varene
>>
>>
>> : Code :
>> --------
>>
>> private static enum Counters {NBLINES};
>> private static IntWritable one =3D new IntWritable(1);
>> private static LongWritable oneL =3D new LongWritable(1);
>>
>>
>> // Mapper
>> /**
>>  * outputs 1 for each line encountered
>>  */
>> public static class LCMapper extends
>>    Mapper<LongWritable, Text, IntWritable, LongWritable>
>> {
>>  private final static IntWritable one =3D new IntWritable(1);
>>  private final static LongWritable oneL =3D new LongWritable(1);
>>  /**
>>  * for each line encountered outputs 1 for the key 1
>>  */
>>  public void map(LongWritable p_key, Text p_inputs, Context p_context)
>>    throws IOException, InterruptedException
>>  {
>>    p_context.getCounter(Counters.NBLINES).increment(1);
>>    p_context.write(one,oneL);     } // end map
>> } // end Mapper
>>
>> : Test :
>> --------
>>
>> @Test
>> public void testLCMapper() throws IOException, InterruptedException
>> {     LCMapper mapper =3D new LCMapper();
>>  Text value =3D new Text("ligneA\nlignB\n");
>>  // mock the map job execution context
>>  Context context =3D mock(Context.class);
>>  try {
>>    mapper.map(null, value, context);
>>  } catch (NullPointerException e)
>>  {
>>    // exception raised by context.getCounter(...) null pointer
>>    // HOW TO MOCK ???
>>    // Counter is a static enum from the englobing class ...
>>    // issue with variable visibility (from map function) in this test
>>  }
>>  // verify that the Mapper issued (one,oneL) pair
>>  verify(context).write(one,oneL);
>>       } // end testLCMapper
>>
>>
>>

--000e0cd1720075ef4904879c5f9e--

From general-return-1586-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri May 28 00:59:33 2010
Return-Path: <general-return-1586-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 6776 invoked from network); 28 May 2010 00:59:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 May 2010 00:59:33 -0000
Received: (qmail 93312 invoked by uid 500); 28 May 2010 00:59:32 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93242 invoked by uid 500); 28 May 2010 00:59:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93234 invoked by uid 99); 28 May 2010 00:59:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 May 2010 00:59:32 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of Deepika.Khera@avg.com designates 193.85.188.248 as permitted sender)
Received: from [193.85.188.248] (HELO ms.grisoft.cz) (193.85.188.248)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 May 2010 00:59:26 +0000
Received: from phobos.cz.avg.com (unknown [192.168.200.160])
	(using TLSv1 with cipher AES128-SHA (128/128 bits))
	(No client certificate requested)
	by ms.grisoft.cz (Postfix) with ESMTP id 8F5A6380D8
	for <general@hadoop.apache.org>; Fri, 28 May 2010 02:59:05 +0200 (CEST)
Received: from miranda.us.avg.com (10.32.34.13) by phobos.cz.avg.com
 (192.168.200.160) with Microsoft SMTP Server (TLS) id 8.2.254.0; Fri, 28 May
 2010 02:59:05 +0200
Received: from miranda.us.avg.com ([10.32.34.13]) by miranda ([10.32.34.13])
 with mapi; Thu, 27 May 2010 20:59:01 -0400
From: Deepika Khera <Deepika.Khera@avg.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 27 May 2010 20:57:26 -0400
Subject: RE: RE: Re: Re: Hash Partitioner
Thread-Topic: RE: Re: Re: Hash Partitioner
Thread-Index: Acr8HGcv7YU8+mRFRRShjsHCpuhKbQALyO+QAG0jQRA=
Message-ID: <DA852C02397DBC4EBFAA1B7192FBD15061DC83FB06@miranda>
References: <DA852C02397DBC4EBFAA1B7192FBD15061DC83F92C@miranda>
	 <AANLkTilDlXliVirB601lvz9fLFf4LO_S6YkDI83VlNz-@mail.gmail.com>
	 <DA852C02397DBC4EBFAA1B7192FBD15061DC83F931@miranda>
 <AANLkTilGCNVJpT7fUXNBFySFvOO-FaDz5p7tu4yTjsGr@mail.gmail.com>
 <DA852C02397DBC4EBFAA1B7192FBD15061DC83F9C5@miranda>
In-Reply-To: <DA852C02397DBC4EBFAA1B7192FBD15061DC83F9C5@miranda>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

This is just to close this one. I finally resolved my issue. Problem was th=
at I had some enums in my key, the hash code of which is not constant over =
JVMs. So instead of doing myEnum.hashCode(), I should have first converted =
it to a string and then taken the hashcode (like myEnum.name().hashCode()).=
 I was relying on a correct hashCode() method, since the IDE had generated =
it for me. I just had to be careful for my case.

Thanks for all the help!

Deepika



-----Original Message-----
From: Deepika Khera [mailto:Deepika.Khera@avg.com]=20
Sent: Tuesday, May 25, 2010 2:03 PM
To: general@hadoop.apache.org
Subject: RE: Re: Re: Hash Partitioner

So I ran my process again with some more logging and here is what I see.=20

I used my own HashPartitioner(basically copied hadoop's partitioner and add=
ed some logging for analysis).I printed here the Key and the reducer that i=
s assigned to the key (based on the hash code).=20

My process triggered off 2 mappers (running on 2 different hadoop machines)=
, hence both of these try to find reducers for the split file assigned to t=
hem. I see that for a same object key assigned to both these mappers, I am =
getting 2 different reducers allocated by the Partitioner.

In the reducers I see -

1) 2 different reducers (the ones that the partitioner assigned the key to)=
 printing out the same Key (I did not print out the value as I thought that=
 wouldn't matter)
2) Here are the logs from where reducers copies data from the mapper -
=09
Reducer1:

2010-05-25 11:34:49,810 INFO org.apache.hadoop.mapred.ReduceTask: Read 1002=
612 bytes from map-output for attempt_201005251129_0001_m_000001_0
2010-05-25 11:34:49,831 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 fr=
om attempt_201005251129_0001_m_000001_0 -> (127, 36) from hadoop-49.c.a.com
2010-05-25 11:34:50,797 INFO org.apache.hadoop.mapred.ReduceTask: attempt_2=
01005251129_0001_r_000006_0: Got 1 new map-outputs
2010-05-25 11:34:54,835 INFO org.apache.hadoop.mapred.ReduceTask: attempt_2=
01005251129_0001_r_000006_0 Scheduled 1 outputs (0 slow hosts and0 dup host=
s)
2010-05-25 11:34:54,841 INFO org.apache.hadoop.mapred.ReduceTask: header: a=
ttempt_201005251129_0001_m_000000_0, compressed len: 1553902, decompressed =
len: 1553898
2010-05-25 11:34:54,841 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling=
 1553898 bytes (1553902 raw bytes) into RAM from attempt_201005251129_0001_=
m_000000_0
2010-05-25 11:34:54,924 INFO org.apache.hadoop.mapred.ReduceTask: Read 1553=
898 bytes from map-output for attempt_201005251129_0001_m_000000_0
2010-05-25 11:34:54,944 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 fr=
om attempt_201005251129_0001_m_000000_0 -> (143, 36) from hadoop-25.c.a.com


Reducer2:=20
=20
2010-05-25 11:34:49,822 INFO org.apache.hadoop.mapred.ReduceTask: Read 6376=
57 bytes from map-output for attempt_201005251129_0001_m_000001_0
2010-05-25 11:34:49,911 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 fr=
om attempt_201005251129_0001_m_000001_0 -> (125, 36) from hadoop-49.c.a.com
2010-05-25 11:34:50,806 INFO org.apache.hadoop.mapred.ReduceTask: attempt_2=
01005251129_0001_r_000008_0: Got 1 new map-outputs
2010-05-25 11:34:54,915 INFO org.apache.hadoop.mapred.ReduceTask: attempt_2=
01005251129_0001_r_000008_0 Scheduled 1 outputs (0 slow hosts and0 dup host=
s)
2010-05-25 11:34:54,920 INFO org.apache.hadoop.mapred.ReduceTask: header: a=
ttempt_201005251129_0001_m_000000_0, compressed len: 1462335, decompressed =
len: 1462331
2010-05-25 11:34:54,920 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling=
 1462331 bytes (1462335 raw bytes) into RAM from attempt_201005251129_0001_=
m_000000_0
2010-05-25 11:34:54,937 INFO org.apache.hadoop.mapred.ReduceTask: Read 1462=
331 bytes from map-output for attempt_201005251129_0001_m_000000_0
2010-05-25 11:34:54,937 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 fr=
om attempt_201005251129_0001_m_000000_0 -> (147, 36) from hadoop-25.c.a.com


The 2 reduce tasks have different task ids and belong to the same job.

Thanks,
Deepika

-----Original Message-----
From: Eric Sammer [mailto:esammer@cloudera.com]=20
Sent: Tuesday, May 25, 2010 8:10 AM
To: general@hadoop.apache.org
Subject: Re: Re: Hash Partitioner

On Mon, May 24, 2010 at 6:32 PM, Deepika Khera <Deepika.Khera@avg.com> wrot=
e:
> Thanks for your response Eric.
>
> I am using hadoop 0.20.2.
>
> Here is what the hashCode() implementation looks like (I actually had the=
 IDE generate it for me)
>
> Main key (for mapper & reducer):
>
> public int hashCode() {
> =A0 =A0 =A0 =A0int result =3D kVersion;
> =A0 =A0 =A0 =A0result =3D 31 * result + (aKey !=3D null ? aKey.hashCode()=
 : 0);
> =A0 =A0 =A0 =A0result =3D 31 * result + (gKey !=3D null ? gKey.hashCode()=
 : 0);
> =A0 =A0 =A0 =A0result =3D 31 * result + (int) (date ^ (date >>> 32));
> =A0 =A0 =A0 =A0result =3D 31 * result + (ma !=3D null ? ma.hashCode() : 0=
);
> =A0 =A0 =A0 =A0result =3D 31 * result + (cl !=3D null ? cl.hashCode() : 0=
);
> =A0 =A0 =A0 =A0return result;
> =A0 =A0}
>
>
> aKey : AKey class
>
>
> =A0 =A0public int hashCode() {
> =A0 =A0 =A0 =A0int result =3D kVersion;
> =A0 =A0 =A0 =A0result =3D 31 * result + (v !=3D null ? v.hashCode() : 0);
> =A0 =A0 =A0 =A0result =3D 31 * result + (s !=3D null ? s.hashCode() : 0);
> =A0 =A0 =A0 =A0result =3D 31 * result + (o !=3D null ? o.hashCode() : 0);
> =A0 =A0 =A0 =A0result =3D 31 * result + (l !=3D null ? l.hashCode() : 0);
> =A0 =A0 =A0 =A0result =3D 31 * result + (e ? 1 : 0); //boolean
> =A0 =A0 =A0 =A0result =3D 31 * result + (li ? 1 : 0); //boolean
> =A0 =A0 =A0 =A0result =3D 31 * result + (aut ? 1 : 0); //boolean
> =A0 =A0 =A0 =A0return result;
> =A0 =A0}
>

Both of these look fine, assuming all the other hashCode()s return the
same value every time.

> When this happens, I do see the same values for the key. Also I am not us=
ing a grouping comparator.

So you see two reduce methods getting the same key with the same
values? That's extremely odd. If this is the case, there's a bug in
Hadoop. Can you find the relevant logs from the reducers where Hadoop
fetches the map output? Does it look like its fetching the same output
twice? Do the two tasks where you see the duplicates have the same
task ID? Can you confirm the reduce tasks are from the same job ID for
us?

> I was wondering since the call to HashPartitioner.getPartition() is done =
from a map task, several of which are running on different machines, is it =
possible that they get a different hashcode and hence get different reducer=
s assigned even when the key is the same.

The hashCode() result should *always* be the same given the same
internal state. In other words, it should be consistent and stable. If
I have a string new String("hello world") it will always have the
exact same hashCode(). If this isn't true, you will get wildly
unpredictable results not just with Hadoop but with Java's
comparators, collections, etc.

--=20
Eric Sammer
phone: +1-917-287-2675
twitter: esammer
data: www.cloudera.com

From general-return-1587-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri May 28 06:32:15 2010
Return-Path: <general-return-1587-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1034 invoked from network); 28 May 2010 06:32:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 May 2010 06:32:13 -0000
Received: (qmail 12166 invoked by uid 500); 28 May 2010 06:32:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11841 invoked by uid 500); 28 May 2010 06:32:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 11832 invoked by uid 99); 28 May 2010 06:32:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 May 2010 06:32:09 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=SPF_PASS,TO_NO_BRKTS_DIRECT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [193.252.149.37] (HELO smtpout01s1.x-echo.com) (193.252.149.37)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 May 2010 06:32:05 +0000
X-Spam-Level: 
Message-ID: <4BFF64D6.6010409@echo.fr>
Date: Fri, 28 May 2010 08:38:14 +0200
From: Varene Olivier <varene@echo.fr>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Mapper Reducer : Unit Test and mocking with static variables
References: <4BFE3004.5000603@echo.fr> <4BFE9A0F.9010606@echo.fr> <AANLkTimUvNhuga7Zb2dflF6TLf4l1mgq7NVvzTBi2lg6@mail.gmail.com>
In-Reply-To: <AANLkTimUvNhuga7Zb2dflF6TLf4l1mgq7NVvzTBi2lg6@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit
X-AV-Checked: ClamAV using ClamSMTP
X-Old-Spam-Status: No, score=-4.3 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00
	autolearn=ham version=3.2.3

Thanks a lot Aaron,

I was looking into it, I just hope, it will be compliant with 0.20.2 ...

Otherwise, I am stuck :(

Cheers
Olivier

Aaron Kimball a écrit :
> Varene,
> 
> You might want to check out MRUnit. It's a unit test harness that contains
> mock objects for the context & other associated classes, and works with
> JUnit.
> 
> It's included in the (unreleased) Hadoop 0.21, as well as Cloudera's
> Distribution for Hadoop. See
> http://archive.cloudera.com/docs/mrunit/index.html for MRUnit documentation.
> 
> Cheers,
> - Aaron
> 
> On Thu, May 27, 2010 at 9:13 AM, Varene Olivier <varene@echo.fr> wrote:
> 
>> FYI
>> I am on Hadoop 0.20.2, JUnit 4
>>
>>
>> Varene Olivier a écrit :
>>
>>  Hello to all,
>>> first of all many thanks for this great piece of software you are all
>>> contributing to :)
>>>
>>> I am actually creating a program in Hadoop MapReduce, but I intend to
>>> massively use JUnit Tests to validate my code.
>>>
>>> To test a Mapper, I mock a Mapper.Context object and run the Mapper.map
>>> function with it, but inside my Mapper.map function, I am using Counters
>>> to maintain internal statistics. The update of those counters make the test
>>> fails with a null pointer (JavaNullPointerException).
>>> There is an issue "Counters" of variable visibility from my test ...
>>>
>>> Do you have any idea how to mock/validate/overcome this issue ?
>>>
>>>
>>>
>>> Thanks a lots
>>> Best Regards
>>> Olivier Varene
>>>
>>>
>>> : Code :
>>> --------
>>>
>>> private static enum Counters {NBLINES};
>>> private static IntWritable one = new IntWritable(1);
>>> private static LongWritable oneL = new LongWritable(1);
>>>
>>>
>>> // Mapper
>>> /**
>>>  * outputs 1 for each line encountered
>>>  */
>>> public static class LCMapper extends
>>>    Mapper<LongWritable, Text, IntWritable, LongWritable>
>>> {
>>>  private final static IntWritable one = new IntWritable(1);
>>>  private final static LongWritable oneL = new LongWritable(1);
>>>  /**
>>>  * for each line encountered outputs 1 for the key 1
>>>  */
>>>  public void map(LongWritable p_key, Text p_inputs, Context p_context)
>>>    throws IOException, InterruptedException
>>>  {
>>>    p_context.getCounter(Counters.NBLINES).increment(1);
>>>    p_context.write(one,oneL);     } // end map
>>> } // end Mapper
>>>
>>> : Test :
>>> --------
>>>
>>> @Test
>>> public void testLCMapper() throws IOException, InterruptedException
>>> {     LCMapper mapper = new LCMapper();
>>>  Text value = new Text("ligneA\nlignB\n");
>>>  // mock the map job execution context
>>>  Context context = mock(Context.class);
>>>  try {
>>>    mapper.map(null, value, context);
>>>  } catch (NullPointerException e)
>>>  {
>>>    // exception raised by context.getCounter(...) null pointer
>>>    // HOW TO MOCK ???
>>>    // Counter is a static enum from the englobing class ...
>>>    // issue with variable visibility (from map function) in this test
>>>  }
>>>  // verify that the Mapper issued (one,oneL) pair
>>>  verify(context).write(one,oneL);
>>>       } // end testLCMapper
>>>
>>>
>>>
> 

From general-return-1588-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri May 28 12:48:58 2010
Return-Path: <general-return-1588-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94073 invoked from network); 28 May 2010 12:48:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 May 2010 12:48:58 -0000
Received: (qmail 80458 invoked by uid 500); 28 May 2010 12:48:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80344 invoked by uid 500); 28 May 2010 12:48:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80336 invoked by uid 99); 28 May 2010 12:48:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 May 2010 12:48:56 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=SPF_PASS,TO_NO_BRKTS_DIRECT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [193.252.149.37] (HELO smtpout01s1.x-echo.com) (193.252.149.37)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 May 2010 12:48:50 +0000
X-Spam-Level: 
Message-ID: <4BFFBD26.10700@echo.fr>
Date: Fri, 28 May 2010 14:55:02 +0200
From: Varene Olivier <varene@echo.fr>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Mapper Reducer : Unit Test and mocking with static variables
References: <4BFE3004.5000603@echo.fr> <4BFE9A0F.9010606@echo.fr> <AANLkTimUvNhuga7Zb2dflF6TLf4l1mgq7NVvzTBi2lg6@mail.gmail.com> <4BFF64D6.6010409@echo.fr>
In-Reply-To: <4BFF64D6.6010409@echo.fr>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit
X-AV-Checked: ClamAV using ClamSMTP
X-Old-Spam-Status: No, score=-4.3 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00
	autolearn=ham version=3.2.3

FYI

MRUnit is the way to go and it is compatible (as far as I am aware of) 
with the 0.20.2

Cheers
Olivier

Varene Olivier a écrit :
> Thanks a lot Aaron,
> 
> I was looking into it, I just hope, it will be compliant with 0.20.2 ...
> 
> Otherwise, I am stuck :(
> 
> Cheers
> Olivier
> 
> Aaron Kimball a écrit :
>> Varene,
>>
>> You might want to check out MRUnit. It's a unit test harness that 
>> contains
>> mock objects for the context & other associated classes, and works with
>> JUnit.
>>
>> It's included in the (unreleased) Hadoop 0.21, as well as Cloudera's
>> Distribution for Hadoop. See
>> http://archive.cloudera.com/docs/mrunit/index.html for MRUnit 
>> documentation.
>>
>> Cheers,
>> - Aaron
>>
>> On Thu, May 27, 2010 at 9:13 AM, Varene Olivier <varene@echo.fr> wrote:
>>
>>> FYI
>>> I am on Hadoop 0.20.2, JUnit 4
>>>
>>>
>>> Varene Olivier a écrit :
>>>
>>>  Hello to all,
>>>> first of all many thanks for this great piece of software you are all
>>>> contributing to :)
>>>>
>>>> I am actually creating a program in Hadoop MapReduce, but I intend to
>>>> massively use JUnit Tests to validate my code.
>>>>
>>>> To test a Mapper, I mock a Mapper.Context object and run the Mapper.map
>>>> function with it, but inside my Mapper.map function, I am using 
>>>> Counters
>>>> to maintain internal statistics. The update of those counters make 
>>>> the test
>>>> fails with a null pointer (JavaNullPointerException).
>>>> There is an issue "Counters" of variable visibility from my test ...
>>>>
>>>> Do you have any idea how to mock/validate/overcome this issue ?
>>>>
>>>>
>>>>
>>>> Thanks a lots
>>>> Best Regards
>>>> Olivier Varene
>>>>
>>>>
>>>> : Code :
>>>> --------
>>>>
>>>> private static enum Counters {NBLINES};
>>>> private static IntWritable one = new IntWritable(1);
>>>> private static LongWritable oneL = new LongWritable(1);
>>>>
>>>>
>>>> // Mapper
>>>> /**
>>>>  * outputs 1 for each line encountered
>>>>  */
>>>> public static class LCMapper extends
>>>>    Mapper<LongWritable, Text, IntWritable, LongWritable>
>>>> {
>>>>  private final static IntWritable one = new IntWritable(1);
>>>>  private final static LongWritable oneL = new LongWritable(1);
>>>>  /**
>>>>  * for each line encountered outputs 1 for the key 1
>>>>  */
>>>>  public void map(LongWritable p_key, Text p_inputs, Context p_context)
>>>>    throws IOException, InterruptedException
>>>>  {
>>>>    p_context.getCounter(Counters.NBLINES).increment(1);
>>>>    p_context.write(one,oneL);     } // end map
>>>> } // end Mapper
>>>>
>>>> : Test :
>>>> --------
>>>>
>>>> @Test
>>>> public void testLCMapper() throws IOException, InterruptedException
>>>> {     LCMapper mapper = new LCMapper();
>>>>  Text value = new Text("ligneA\nlignB\n");
>>>>  // mock the map job execution context
>>>>  Context context = mock(Context.class);
>>>>  try {
>>>>    mapper.map(null, value, context);
>>>>  } catch (NullPointerException e)
>>>>  {
>>>>    // exception raised by context.getCounter(...) null pointer
>>>>    // HOW TO MOCK ???
>>>>    // Counter is a static enum from the englobing class ...
>>>>    // issue with variable visibility (from map function) in this test
>>>>  }
>>>>  // verify that the Mapper issued (one,oneL) pair
>>>>  verify(context).write(one,oneL);
>>>>       } // end testLCMapper
>>>>
>>>>
>>>>
>>

From general-return-1589-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri May 28 21:30:00 2010
Return-Path: <general-return-1589-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54071 invoked from network); 28 May 2010 21:29:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 May 2010 21:29:59 -0000
Received: (qmail 11679 invoked by uid 500); 28 May 2010 21:29:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 11436 invoked by uid 500); 28 May 2010 21:29:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 34952 invoked by uid 99); 26 May 2010 23:51:45 -0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of robotics@students.iiit.ac.in designates 121.242.23.201 as permitted sender)
Message-Id: <201005262351.o4QNpHtY002366@students.iiit.ac.in>
Content-Type: multipart/mixed; boundary="===============1646007950=="
MIME-Version: 1.0
From: IIIT-H Robotics<robotics@students.iiit.ac.in>
To: general@hadoop.apache.org
Date: Thu, 27 May 2010 04:48:01 +0530
Subject: Reg: Embedded Systems Workshop for Robotic Applications and System
	Engineering by IIIT- Hyderabad on 28th/ 29th/30th May
X-Spam-Level: *
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on students.iiit.ac.in
X-Old-Spam-Status: No, score=1.9 required=5.0 tests=ALL_TRUSTED,FH_DATE_PAST_20XX,
	MISSING_MID autolearn=no version=3.2.5

--===============1646007950==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit

Hello,

Greetings from IIIT, Hyderabad,

We are delighted to inform you that IIIT-H is proudly conducting the first
ever nationwide Workshop on Embedded Systems and its application in
Robotics and System Engineering. This event begins from May 28th in
IIIT-Hyderabad.


The workshop would cover a wide range of topics right from the beginners
level to an advanced stage. The topics include but are not limited to
FPGAs, Microcontrollers and Programmable Logic Controllers which can be
used to design various System Engineering Aspects and robotic applications
. We will be covering the tools and system concepts of ATMEL AVR
Microcontrollers and Xilinx, Altera FPGAs.


This one day event is tuned to industry requirements and aims to provide
you with knowledge which would be valuable in the practical sphere for a
corporate job in the Semiconductor and IT Industry.



Registration Process

The workshop will be conducted in IIIT-H Campus on 28th , 29th and 30th of
May. Students can register on our webpage :
http://web.iiit.ac.in/~robotics/EW2010/register.html  for a preferred date
or come on the day of the Event and register onsite.

Registration Fees :  Rs 399 /-


Program Schedule on the Day of Event

10 AM - 11 AM :  On-campus Registration and Fees Collection

11 AM - 12:30 PM : Lecture on FPGAs and Using Custom Toolkits with
Demonstrations.

30 min Break

1 PM - 2:30 PM : Lecture on Micro controllers and Demos in robotic
applications.

2:30 PM - 3:30 PM : Brain-Storming Session

3:30 PM - 4:00 PM : Distribution of Participation Certificates


Who to Contact

Please visit our page http://web.iiit.ac.in/~robotics/EW2010 for more
details or please contact Mr. Jairaj Bhattacharya at +91 9989020007 or Mr.
Shashank Pandey at +91 9000405901.


We look forward to meeting you and your students at the EW2010.


Best Regards,

Embedded Robotics Team



IIIT-H has been ranked among the Best 7 Universities in South Asia, in
terms of .scholarly papers. on the Internet, in the latest ranking, by
Cybermetric Lab, Spain.s largest public research institution. (Oct 2009)


--===============1646007950==--

From general-return-1590-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri May 28 21:41:06 2010
Return-Path: <general-return-1590-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55910 invoked from network); 28 May 2010 21:41:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 May 2010 21:41:05 -0000
Received: (qmail 20650 invoked by uid 500); 28 May 2010 21:41:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20594 invoked by uid 500); 28 May 2010 21:41:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 20586 invoked by uid 99); 28 May 2010 21:41:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 May 2010 21:41:04 +0000
X-ASF-Spam-Status: No, hits=1.4 required=10.0
	tests=AWL,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.62.80] (HELO qmta08.westchester.pa.mail.comcast.net) (76.96.62.80)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 May 2010 21:40:56 +0000
Received: from omta13.westchester.pa.mail.comcast.net ([76.96.62.52])
	by qmta08.westchester.pa.mail.comcast.net with comcast
	id P7ay1e00517dt5G589gcm5; Fri, 28 May 2010 21:40:36 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta13.westchester.pa.mail.comcast.net with comcast
	id P9gS1e00J2SbwD53Z9gVN0; Fri, 28 May 2010 21:40:34 +0000
Message-Id: <9DD2220F-023D-4E6B-88C5-F08B184B6102@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=Apple-Mail-4-422882174
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Last call: ApacheCon TechTalks CFP closes today
Date: Fri, 28 May 2010 14:40:25 -0700
X-Mailer: Apple Mail (2.936)

--Apple-Mail-4-422882174
Content-Type: text/plain;
	charset=WINDOWS-1252;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: quoted-printable

All,
    As a reminder the CFP for proposals closes today.  The CFP is at =
http://blogs.apache.org/conferences/date/20100428

-- Owen

ApacheCon North America 2010
1-5 November 2010 -- Westin Peachtree in Atlanta

Technical Tracks: Call For Participation
All submissions must be received by Friday, 28 May 2010 at midnight =20
Pacific Time.
The official conference, trainings, and expo of The Apache Software =20
Foundation (ASF) returns to Atlanta this November, with dozens of =20
technical, business, and community-focused sessions at the beginner, =20
intermediate, and advanced levels.

Over the past decade, the ASF has gone from strength to strength, =20
developing and shepherding nearly 150 Top-Level Projects and new =20
initiatives in the Apache Incubator and Labs. This year's ApacheCon =20
celebrates how Apache technologies have sparked creativity, challenged =20=

processes, streamlined development, improved collaboration, launched =20
businesses, bolstered economies, and improved lives.

We are proud of our achievements and recognize that the global Apache =20=

community --both developers and users-- are responsible for the =20
success and popularity of our products.

The ApacheCon Planning Team are soliciting 50-minute technical =20
presentations for the next conference, which will focus on the theme =20
=93Servers, the Cloud, and Innovation=94.

We are particularly interested in highly-relevant, professionally-=20
directed presentations that demonstrate specific probrlems and real-=20
world solutions. Part of the technical program has already been =20
planned; we welcome proposals based on the following Apache Projects =20
and related technical areas:

- Cassandra/NoSQL
- Content Technologies
- (Java) Enterprise Development
- Felix/OSGi
- Geronimo
- Hadoop + friends/Cloud Computing
- Lucene, Mahout + friends/Search
- Tomcat
- Tuscany
Submissions are open to anyone with relevant expertise: ASF =20
affiliation is not required to present at, attend, or otherwise =20
participate in ApacheCon.

Please keep in mind that whilst we encourage submissions that the =20
highlight the use of specific Apache solutions, we are unable to =20
accept marketing/commercially-oriented presentations.

Other proposals, such as panels, or those longer than 50 minutes in =20
duration have been considered in the past. You are welcome to submit =20
an alternate presentation, however, such sessions are accepted under =20
exceptional circumstances. Please be as descriptive as possible, =20
including names/bios of proposed panelists and any related details.

All accepted speakers (not co-presenters) qualify for general =20
conference admission and a minimum of two nights lodging at the =20
conference hotel. Additional hotel nights and travel assistance are =20
possible, depending on the number of presentations given and type of =20
assistance needed.

To submit a presentation proposal, please send an email to submissions =20=

AT apachecon DOT com containing the following information in plaintext =20=

(no attachments, please):

1. Your full name, title, and organization

2. Contact information, including your address

3. The name of your proposed session (keep your title simple and =20
relevant to the topic)

4. The technical category of the intended presentation (Cassandra/=20
NoSQL; Content Technologies; (Java) Enterprise Development; Felix/=20
OSGi; Geronimo; Hadoop + friends/Cloud Computing; Lucene, Mahout + =20
friends/Search; Tomcat; or Tuscany)

5. The classification for each presentation (Servers, Cloud, or =20
Innovation) =96 some presentations may have more than one theme (e.g., a =
=20
next-generation server can be classified both as "Servers" and =20
"Innovation"

6. The intended audience level (beginner, intermediate, advanced)

7. A 75-200 word overview of your presentation

8. A 100-200-word speaker bio that includes prior conference speaking =20=

or related experience

9. Feedback or references (with contact information) on presentations =20=

given within the last three years

To be considered, proposals must be received by Friday, 28 May 2010 at =20=

midnight Pacific Time. Please email any questions regarding proposal =20
submissions to cfp AT apachecon DOT com.

Technical Tracks Key Dates

23 April 2010: Call For Participation Open
28 May 2010: Call For Participation Closes
11 June 2010: Speaker Acceptance/Rejection Notification
1-5 November 2010: ApacheCon NA 2010
We look forward to seeing you in Atlanta!

For the ApacheCon Planning team,
Sally Khudairi, Program Lead

--Apple-Mail-4-422882174--

From general-return-1591-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat May 29 00:37:34 2010
Return-Path: <general-return-1591-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 97539 invoked from network); 29 May 2010 00:37:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 May 2010 00:37:34 -0000
Received: (qmail 37884 invoked by uid 500); 29 May 2010 00:37:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37813 invoked by uid 500); 29 May 2010 00:37:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37805 invoked by uid 99); 29 May 2010 00:37:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 29 May 2010 00:37:32 +0000
X-ASF-Spam-Status: No, hits=-1433.0 required=10.0
	tests=ALL_TRUSTED,AWL,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Sat, 29 May 2010 00:37:32 +0000
Received: (qmail 97425 invoked by uid 99); 29 May 2010 00:37:09 -0000
Received: from localhost.apache.org (HELO mail-gw0-f48.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Sat, 29 May 2010 00:37:09 +0000
Received: by gwj15 with SMTP id 15so1791963gwj.35
        for <multiple recipients>; Fri, 28 May 2010 17:37:06 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.148.143 with SMTP id p15mr1325614ibv.15.1275093426440; 
	Fri, 28 May 2010 17:37:06 -0700 (PDT)
Received: by 10.231.148.65 with HTTP; Fri, 28 May 2010 17:37:06 -0700 (PDT)
Date: Fri, 28 May 2010 17:37:06 -0700
Message-ID: <AANLkTimeqo7RDWvCIWDqiUM4M7XKoeeXlCDPw8TY1xix@mail.gmail.com>
Subject: Contributor Meeting Minutes 05/28/2010
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org, mapreduce-dev@hadoop.apache.org, 
	hdfs-dev@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

This month, the MapReduce + HDFS contributor meeting was held at
Cloudera Headquarters.

Announcements for contributor meetings are here:
http://www.meetup.com/Hadoop-Contributors/

Minutes follow. No decisions were made at this meeting, but the
following issues were discussed and may presage future discussion and
decisions on these lists.

Eli, I think you have all the slides. Would you mind sending them out? -C

== 0.21 release update ==
* Continuing to close blockers, ping people for updates and suggestions
* About 20 open blockers. Many are MapReduce documentation that may be
pushed. Speak up if 0.21 is missing anything substantive.
* Common/HDFS visibility and annotations are close to consensus;
MapReduce annotations are committed to trunk and the 0.21 branch

== HEP proposal ==
(what follows is the sketch presented at the meeting. A full proposal
with concrete details will be circulated on the list)

* Based on- and very similar to- the PEP (Python Enhancement Proposal) Process
* Audience is HDFS and MapReduce; not necessarily adopted by other subprojects
  - Addresses the perception that there is friction between
innovation/experimentation and stability
* Not for small enhancements, features, and bug fixes. This should not
slow down typical development or impede casual contribution to Hadoop
* Primary mechanism for new features, collecting input, documenting
design decisions
* JIRA is good for details, but not for deciding on wide shifts in direction
* Purpose is for author to build consensus and gather dissenting opinions.
  - All may comment, but Editors will review incoming HEP material
  - Editors determine only whether the HEP is complete, not whether
they believe it is a sound idea
  - Editors are appointed by the PMC
  - Mechanism for appointing Editors and term of service TBD
    - Apache Board appoints Shepherds for projects somewhat randomly,
to projects. A similar mechanism could work for incoming HEPs
  - Proposal *may* come with code, but not necessarily.
Drafting/baking of the HEP occurs in public on a list dedicated to
that particular proposal. Once Editors certify the HEP as complete, it
is sent to general@ for wider discussion.
    - The discussion phase begins on general@. The mailing list exists
to ensure the HEP is complete enough to present to the community.
  - Some discussion on the difference between posting to general@ and
posting to the HEP list. Completeness is, of course, subjective. If
the Editor and Author disagree whether the proposal affects an aspect
of the framework enough to merit special consideration, it is not
entirely clear how to resolve the disagreement.
    - In general, the role of the Editor in the community-driven
process of Hadoop is not entirely clear. It may be possible to
optimize it out.
  - Once discussion ends, the HEP is passed (or fails to pass) by a
vote of the PMC (mechanics undefined). In Python, the result is
committed to the repository. A similar practice would make sense in
Hadoop.
* Which issues require HEPs?
  - Discussion ranged. Append, backup namenode, edit log rewrite, et
al. were examples of features substantial enough to merit a HEP. Pure
Java CRC is an example of an enhancement that would not. Whether an
explicit process must be in place to determine whether an issue
requires a HEP is not clear.
  - Viewing HEPs as a way of soliciting consensus for an approach
might be more accurate. Going through the HEP process should always
improve the chances of a successful proposal

* Evaluation
  - The proposal may be rejected if it is redundant with existing
functionality, technically unsound, insufficiently motivated, no
backwards compatibility story, etc.
  - Implementation is not necessary, and is lightly discouraged.
Feedback is less welcome once code is in hand.
  - Purpose is to be clear about the acceptance criteria for that
issue, e.g. concerns that the proposal may not scale or may harm
performance
  - Dissenting opinions must be recorded accurately. Quoting would be
a safe practice for the Author to encourage HEP reviewers not to block
the product of the proposal.

* The testing burden and completion strategy may be ambiguous
  - Whether the proposal affects scalability may not be testable by
the implementer. Completing the proposal to address all use cases may
require considerably more work than the Author is willing or motivated
to invest.
  - The HEP discussion on general@ should explore whether such
objections are merited and reasonable. For example, a particularly
obscure/esoteric use case could be included as a condition for
acceptance if the dissenter is willing to invest the resources to
test/validate it. The process is flexible in this regard.
    - But it is not infinitely flexible. Backwards compatibility,
performance regression, availability, and other considerations need
not be called out in every HEP.
    - Traditional concerns need to be documented. Acceptance criteria
should ideally be automated and reproducible in different
organizations

== Branching ==
* A patch and a branch are isomorphic from a policy perspective. Of
course, they are functionally distinct: branches are easier to
collaborate on and are, generally, longer-lived than are patches. But
special policies need not be derived to account for these differences,
which concern the production of the code, not its review and
acceptance.
* Some developers find branches to be easier to review than very large
patches and easier to merge, given a toolchain that supports this.
  - Subversion currently is difficult to adapt to this model
  - Could be done on a HEP-by-HEP basis, as a condition for acceptance
* Eclipse Labs
  - Branded version of Google Code (same functionality, w/ Eclipse brand)
  - Not official Eclipse projects, but associated with Eclipse
  - Apache/Hadoop may consider a similar strategy
  - Distinct from Apache Labs, as one need not be a committer, follow
its rules for releases, etc.

== Contrib ==
* Modules (such as fuse-dfs) are not actively maintained in the main
repository and would benefit from a release schedule decoupled from
the rest of Hadoop
* With few exceptions, the contrib modules have smaller, often
discrete groups of maintainers. It may be worth exploring whether
these projects could live elsewhere

From general-return-1592-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat May 29 03:00:09 2010
Return-Path: <general-return-1592-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 26777 invoked from network); 29 May 2010 03:00:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 May 2010 03:00:08 -0000
Received: (qmail 14507 invoked by uid 500); 29 May 2010 03:00:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14274 invoked by uid 500); 29 May 2010 03:00:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14253 invoked by uid 99); 29 May 2010 03:00:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 29 May 2010 03:00:06 +0000
X-ASF-Spam-Status: No, hits=-0.8 required=10.0
	tests=AWL,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.85.222.189] (HELO mail-pz0-f189.google.com) (209.85.222.189)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 29 May 2010 03:00:01 +0000
Received: by pzk27 with SMTP id 27so1282011pzk.2
        for <multiple recipients>; Fri, 28 May 2010 19:59:41 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.143.25.27 with SMTP id c27mr832661wfj.65.1275101980955; Fri, 
	28 May 2010 19:59:40 -0700 (PDT)
Received: by 10.143.16.21 with HTTP; Fri, 28 May 2010 19:59:40 -0700 (PDT)
In-Reply-To: <AANLkTimeqo7RDWvCIWDqiUM4M7XKoeeXlCDPw8TY1xix@mail.gmail.com>
References: <AANLkTimeqo7RDWvCIWDqiUM4M7XKoeeXlCDPw8TY1xix@mail.gmail.com>
Date: Fri, 28 May 2010 19:59:40 -0700
Message-ID: <AANLkTinbgBYlWwaedkPTwfk8qpvnUgczFojTnoQJr0i0@mail.gmail.com>
Subject: Re: Contributor Meeting Minutes 05/28/2010
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Cc: mapreduce-dev@hadoop.apache.org, hdfs-dev@hadoop.apache.org
Content-Type: multipart/mixed; boundary=001636e1f7aca0939c0487b2d0e3

--001636e1f7aca0939c0487b2d0e3
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Slides attached.  Thanks for taking notes Chris!


On Fri, May 28, 2010 at 5:37 PM, Chris Douglas <cdouglas@apache.org> wrote:
> This month, the MapReduce + HDFS contributor meeting was held at
> Cloudera Headquarters.
>
> Announcements for contributor meetings are here:
> http://www.meetup.com/Hadoop-Contributors/
>
> Minutes follow. No decisions were made at this meeting, but the
> following issues were discussed and may presage future discussion and
> decisions on these lists.
>
> Eli, I think you have all the slides. Would you mind sending them out? -C
>
> =3D=3D 0.21 release update =3D=3D
> * Continuing to close blockers, ping people for updates and suggestions
> * About 20 open blockers. Many are MapReduce documentation that may be
> pushed. Speak up if 0.21 is missing anything substantive.
> * Common/HDFS visibility and annotations are close to consensus;
> MapReduce annotations are committed to trunk and the 0.21 branch
>
> =3D=3D HEP proposal =3D=3D
> (what follows is the sketch presented at the meeting. A full proposal
> with concrete details will be circulated on the list)
>
> * Based on- and very similar to- the PEP (Python Enhancement Proposal) Pr=
ocess
> * Audience is HDFS and MapReduce; not necessarily adopted by other subpro=
jects
> =A0- Addresses the perception that there is friction between
> innovation/experimentation and stability
> * Not for small enhancements, features, and bug fixes. This should not
> slow down typical development or impede casual contribution to Hadoop
> * Primary mechanism for new features, collecting input, documenting
> design decisions
> * JIRA is good for details, but not for deciding on wide shifts in direct=
ion
> * Purpose is for author to build consensus and gather dissenting opinions=
.
> =A0- All may comment, but Editors will review incoming HEP material
> =A0- Editors determine only whether the HEP is complete, not whether
> they believe it is a sound idea
> =A0- Editors are appointed by the PMC
> =A0- Mechanism for appointing Editors and term of service TBD
> =A0 =A0- Apache Board appoints Shepherds for projects somewhat randomly,
> to projects. A similar mechanism could work for incoming HEPs
> =A0- Proposal *may* come with code, but not necessarily.
> Drafting/baking of the HEP occurs in public on a list dedicated to
> that particular proposal. Once Editors certify the HEP as complete, it
> is sent to general@ for wider discussion.
> =A0 =A0- The discussion phase begins on general@. The mailing list exists
> to ensure the HEP is complete enough to present to the community.
> =A0- Some discussion on the difference between posting to general@ and
> posting to the HEP list. Completeness is, of course, subjective. If
> the Editor and Author disagree whether the proposal affects an aspect
> of the framework enough to merit special consideration, it is not
> entirely clear how to resolve the disagreement.
> =A0 =A0- In general, the role of the Editor in the community-driven
> process of Hadoop is not entirely clear. It may be possible to
> optimize it out.
> =A0- Once discussion ends, the HEP is passed (or fails to pass) by a
> vote of the PMC (mechanics undefined). In Python, the result is
> committed to the repository. A similar practice would make sense in
> Hadoop.
> * Which issues require HEPs?
> =A0- Discussion ranged. Append, backup namenode, edit log rewrite, et
> al. were examples of features substantial enough to merit a HEP. Pure
> Java CRC is an example of an enhancement that would not. Whether an
> explicit process must be in place to determine whether an issue
> requires a HEP is not clear.
> =A0- Viewing HEPs as a way of soliciting consensus for an approach
> might be more accurate. Going through the HEP process should always
> improve the chances of a successful proposal
>
> * Evaluation
> =A0- The proposal may be rejected if it is redundant with existing
> functionality, technically unsound, insufficiently motivated, no
> backwards compatibility story, etc.
> =A0- Implementation is not necessary, and is lightly discouraged.
> Feedback is less welcome once code is in hand.
> =A0- Purpose is to be clear about the acceptance criteria for that
> issue, e.g. concerns that the proposal may not scale or may harm
> performance
> =A0- Dissenting opinions must be recorded accurately. Quoting would be
> a safe practice for the Author to encourage HEP reviewers not to block
> the product of the proposal.
>
> * The testing burden and completion strategy may be ambiguous
> =A0- Whether the proposal affects scalability may not be testable by
> the implementer. Completing the proposal to address all use cases may
> require considerably more work than the Author is willing or motivated
> to invest.
> =A0- The HEP discussion on general@ should explore whether such
> objections are merited and reasonable. For example, a particularly
> obscure/esoteric use case could be included as a condition for
> acceptance if the dissenter is willing to invest the resources to
> test/validate it. The process is flexible in this regard.
> =A0 =A0- But it is not infinitely flexible. Backwards compatibility,
> performance regression, availability, and other considerations need
> not be called out in every HEP.
> =A0 =A0- Traditional concerns need to be documented. Acceptance criteria
> should ideally be automated and reproducible in different
> organizations
>
> =3D=3D Branching =3D=3D
> * A patch and a branch are isomorphic from a policy perspective. Of
> course, they are functionally distinct: branches are easier to
> collaborate on and are, generally, longer-lived than are patches. But
> special policies need not be derived to account for these differences,
> which concern the production of the code, not its review and
> acceptance.
> * Some developers find branches to be easier to review than very large
> patches and easier to merge, given a toolchain that supports this.
> =A0- Subversion currently is difficult to adapt to this model
> =A0- Could be done on a HEP-by-HEP basis, as a condition for acceptance
> * Eclipse Labs
> =A0- Branded version of Google Code (same functionality, w/ Eclipse brand=
)
> =A0- Not official Eclipse projects, but associated with Eclipse
> =A0- Apache/Hadoop may consider a similar strategy
> =A0- Distinct from Apache Labs, as one need not be a committer, follow
> its rules for releases, etc.
>
> =3D=3D Contrib =3D=3D
> * Modules (such as fuse-dfs) are not actively maintained in the main
> repository and would benefit from a release schedule decoupled from
> the rest of Hadoop
> * With few exceptions, the contrib modules have smaller, often
> discrete groups of maintainers. It may be worth exploring whether
> these projects could live elsewhere
>

--001636e1f7aca0939c0487b2d0e3--

From general-return-1593-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat May 29 06:20:10 2010
Return-Path: <general-return-1593-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 64403 invoked from network); 29 May 2010 06:20:10 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 May 2010 06:20:10 -0000
Received: (qmail 62851 invoked by uid 500); 29 May 2010 06:20:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62503 invoked by uid 500); 29 May 2010 06:20:08 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62482 invoked by uid 99); 29 May 2010 06:20:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 29 May 2010 06:20:07 +0000
X-ASF-Spam-Status: No, hits=-0.4 required=10.0
	tests=AWL,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 29 May 2010 06:20:02 +0000
Received: by pwj10 with SMTP id 10so1314042pwj.35
        for <multiple recipients>; Fri, 28 May 2010 23:19:41 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.142.210.17 with SMTP id i17mr972778wfg.83.1275113981744; Fri, 
	28 May 2010 23:19:41 -0700 (PDT)
Received: by 10.143.16.21 with HTTP; Fri, 28 May 2010 23:19:41 -0700 (PDT)
In-Reply-To: <AANLkTinbgBYlWwaedkPTwfk8qpvnUgczFojTnoQJr0i0@mail.gmail.com>
References: <AANLkTimeqo7RDWvCIWDqiUM4M7XKoeeXlCDPw8TY1xix@mail.gmail.com>
	<AANLkTinbgBYlWwaedkPTwfk8qpvnUgczFojTnoQJr0i0@mail.gmail.com>
Date: Fri, 28 May 2010 23:19:41 -0700
Message-ID: <AANLkTinTvuQuSwbgzgqXxtEwucypNEksAb8VkBa83yeP@mail.gmail.com>
Subject: Re: Contributor Meeting Minutes 05/28/2010
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Cc: mapreduce-dev@hadoop.apache.org, hdfs-dev@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

The list stripped my slides. Posted notes to the wiki, which doesn't
seem to allow attachments so not sure where to put slides.

http://wiki.apache.org/hadoop/HadoopContributorsMeeting20100528


On Fri, May 28, 2010 at 7:59 PM, Eli Collins <eli@cloudera.com> wrote:
> Slides attached. =A0Thanks for taking notes Chris!
>
>
> On Fri, May 28, 2010 at 5:37 PM, Chris Douglas <cdouglas@apache.org> wrot=
e:
>> This month, the MapReduce + HDFS contributor meeting was held at
>> Cloudera Headquarters.
>>
>> Announcements for contributor meetings are here:
>> http://www.meetup.com/Hadoop-Contributors/
>>
>> Minutes follow. No decisions were made at this meeting, but the
>> following issues were discussed and may presage future discussion and
>> decisions on these lists.
>>
>> Eli, I think you have all the slides. Would you mind sending them out? -=
C
>>
>> =3D=3D 0.21 release update =3D=3D
>> * Continuing to close blockers, ping people for updates and suggestions
>> * About 20 open blockers. Many are MapReduce documentation that may be
>> pushed. Speak up if 0.21 is missing anything substantive.
>> * Common/HDFS visibility and annotations are close to consensus;
>> MapReduce annotations are committed to trunk and the 0.21 branch
>>
>> =3D=3D HEP proposal =3D=3D
>> (what follows is the sketch presented at the meeting. A full proposal
>> with concrete details will be circulated on the list)
>>
>> * Based on- and very similar to- the PEP (Python Enhancement Proposal) P=
rocess
>> * Audience is HDFS and MapReduce; not necessarily adopted by other subpr=
ojects
>> =A0- Addresses the perception that there is friction between
>> innovation/experimentation and stability
>> * Not for small enhancements, features, and bug fixes. This should not
>> slow down typical development or impede casual contribution to Hadoop
>> * Primary mechanism for new features, collecting input, documenting
>> design decisions
>> * JIRA is good for details, but not for deciding on wide shifts in direc=
tion
>> * Purpose is for author to build consensus and gather dissenting opinion=
s.
>> =A0- All may comment, but Editors will review incoming HEP material
>> =A0- Editors determine only whether the HEP is complete, not whether
>> they believe it is a sound idea
>> =A0- Editors are appointed by the PMC
>> =A0- Mechanism for appointing Editors and term of service TBD
>> =A0 =A0- Apache Board appoints Shepherds for projects somewhat randomly,
>> to projects. A similar mechanism could work for incoming HEPs
>> =A0- Proposal *may* come with code, but not necessarily.
>> Drafting/baking of the HEP occurs in public on a list dedicated to
>> that particular proposal. Once Editors certify the HEP as complete, it
>> is sent to general@ for wider discussion.
>> =A0 =A0- The discussion phase begins on general@. The mailing list exist=
s
>> to ensure the HEP is complete enough to present to the community.
>> =A0- Some discussion on the difference between posting to general@ and
>> posting to the HEP list. Completeness is, of course, subjective. If
>> the Editor and Author disagree whether the proposal affects an aspect
>> of the framework enough to merit special consideration, it is not
>> entirely clear how to resolve the disagreement.
>> =A0 =A0- In general, the role of the Editor in the community-driven
>> process of Hadoop is not entirely clear. It may be possible to
>> optimize it out.
>> =A0- Once discussion ends, the HEP is passed (or fails to pass) by a
>> vote of the PMC (mechanics undefined). In Python, the result is
>> committed to the repository. A similar practice would make sense in
>> Hadoop.
>> * Which issues require HEPs?
>> =A0- Discussion ranged. Append, backup namenode, edit log rewrite, et
>> al. were examples of features substantial enough to merit a HEP. Pure
>> Java CRC is an example of an enhancement that would not. Whether an
>> explicit process must be in place to determine whether an issue
>> requires a HEP is not clear.
>> =A0- Viewing HEPs as a way of soliciting consensus for an approach
>> might be more accurate. Going through the HEP process should always
>> improve the chances of a successful proposal
>>
>> * Evaluation
>> =A0- The proposal may be rejected if it is redundant with existing
>> functionality, technically unsound, insufficiently motivated, no
>> backwards compatibility story, etc.
>> =A0- Implementation is not necessary, and is lightly discouraged.
>> Feedback is less welcome once code is in hand.
>> =A0- Purpose is to be clear about the acceptance criteria for that
>> issue, e.g. concerns that the proposal may not scale or may harm
>> performance
>> =A0- Dissenting opinions must be recorded accurately. Quoting would be
>> a safe practice for the Author to encourage HEP reviewers not to block
>> the product of the proposal.
>>
>> * The testing burden and completion strategy may be ambiguous
>> =A0- Whether the proposal affects scalability may not be testable by
>> the implementer. Completing the proposal to address all use cases may
>> require considerably more work than the Author is willing or motivated
>> to invest.
>> =A0- The HEP discussion on general@ should explore whether such
>> objections are merited and reasonable. For example, a particularly
>> obscure/esoteric use case could be included as a condition for
>> acceptance if the dissenter is willing to invest the resources to
>> test/validate it. The process is flexible in this regard.
>> =A0 =A0- But it is not infinitely flexible. Backwards compatibility,
>> performance regression, availability, and other considerations need
>> not be called out in every HEP.
>> =A0 =A0- Traditional concerns need to be documented. Acceptance criteria
>> should ideally be automated and reproducible in different
>> organizations
>>
>> =3D=3D Branching =3D=3D
>> * A patch and a branch are isomorphic from a policy perspective. Of
>> course, they are functionally distinct: branches are easier to
>> collaborate on and are, generally, longer-lived than are patches. But
>> special policies need not be derived to account for these differences,
>> which concern the production of the code, not its review and
>> acceptance.
>> * Some developers find branches to be easier to review than very large
>> patches and easier to merge, given a toolchain that supports this.
>> =A0- Subversion currently is difficult to adapt to this model
>> =A0- Could be done on a HEP-by-HEP basis, as a condition for acceptance
>> * Eclipse Labs
>> =A0- Branded version of Google Code (same functionality, w/ Eclipse bran=
d)
>> =A0- Not official Eclipse projects, but associated with Eclipse
>> =A0- Apache/Hadoop may consider a similar strategy
>> =A0- Distinct from Apache Labs, as one need not be a committer, follow
>> its rules for releases, etc.
>>
>> =3D=3D Contrib =3D=3D
>> * Modules (such as fuse-dfs) are not actively maintained in the main
>> repository and would benefit from a release schedule decoupled from
>> the rest of Hadoop
>> * With few exceptions, the contrib modules have smaller, often
>> discrete groups of maintainers. It may be worth exploring whether
>> these projects could live elsewhere
>>
>

From general-return-1594-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat May 29 06:43:41 2010
Return-Path: <general-return-1594-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72700 invoked from network); 29 May 2010 06:43:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 May 2010 06:43:41 -0000
Received: (qmail 70848 invoked by uid 500); 29 May 2010 06:43:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70735 invoked by uid 500); 29 May 2010 06:43:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 17582 invoked by uid 99); 29 May 2010 03:04:24 -0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of deepakmca05@gmail.com designates 209.85.212.48 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:mime-version:received:in-reply-to
         :references:from:date:message-id:subject:to:cc:content-type;
        bh=XsvQ4UbJ+j5ivZnXoQ+IYGkTj3z92eLAyE9T8OfaQy4=;
        b=GJJIxVmEWhlMibFT+gapMCpngGbcfsWtMS7czUIFLyj+Nbh4hOK5VP45yqzkqFczeD
         NlNKx7B8MqIcE85ZHfhDaMG2sy95AgMpy3LWAab4zMZWKXLLkxtNaNaVO/JhhAe7Y/dD
         7nIdxShvvoci+dZl5Arh8/xxfVg16DdEflsW8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :cc:content-type;
        b=omqfn+ij5lmo9XtxKljaS+uCta0+/2WlYDI9ZdopDYYz+CCuTBvw/NOIFMo2nFts+I
         OiWyjOnhAFZCbAzSWHWYRbkSvBtxk13/JplZ1BtEKsA7qAyrGLODAUGXM/Cf8sfUL8Wp
         Q96wJhbjbwNQ/OfwAYEzOSFfipkf08v3ctX5c=
MIME-Version: 1.0
In-Reply-To: <AANLkTimeqo7RDWvCIWDqiUM4M7XKoeeXlCDPw8TY1xix@mail.gmail.com>
References: <AANLkTimeqo7RDWvCIWDqiUM4M7XKoeeXlCDPw8TY1xix@mail.gmail.com>
From: Deepak Sharma <deepakmca05@gmail.com>
Date: Sat, 29 May 2010 08:33:35 +0530
Message-ID: <AANLkTik9pAdcn3AAGL7HrKDjWlt4RRw-Hg8kM5vV2xln@mail.gmail.com>
Subject: Re: Contributor Meeting Minutes 05/28/2010
To: mapreduce-dev@hadoop.apache.org
Cc: general@hadoop.apache.org, hdfs-dev@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016363b9360c84e310487b2df52

--0016363b9360c84e310487b2df52
Content-Type: text/plain; charset=ISO-8859-1

Hi All,
I am from India and involved with Hadoop for last 1-2 month.
I am planning to start Hadoop tutorial in India and would need help here.
Please let me know some really good tutorials on Hadoop and also if you all
can suggest what all can be included in the course content , such that this
becomes a job oriented course.

Looking forward to your reply.

Thanks,
Deepak

On Sat, May 29, 2010 at 6:07 AM, Chris Douglas <cdouglas@apache.org> wrote:

> This month, the MapReduce + HDFS contributor meeting was held at
> Cloudera Headquarters.
>
> Announcements for contributor meetings are here:
> http://www.meetup.com/Hadoop-Contributors/
>
> Minutes follow. No decisions were made at this meeting, but the
> following issues were discussed and may presage future discussion and
> decisions on these lists.
>
> Eli, I think you have all the slides. Would you mind sending them out? -C
>
> == 0.21 release update ==
> * Continuing to close blockers, ping people for updates and suggestions
> * About 20 open blockers. Many are MapReduce documentation that may be
> pushed. Speak up if 0.21 is missing anything substantive.
> * Common/HDFS visibility and annotations are close to consensus;
> MapReduce annotations are committed to trunk and the 0.21 branch
>
> == HEP proposal ==
> (what follows is the sketch presented at the meeting. A full proposal
> with concrete details will be circulated on the list)
>
> * Based on- and very similar to- the PEP (Python Enhancement Proposal)
> Process
> * Audience is HDFS and MapReduce; not necessarily adopted by other
> subprojects
>  - Addresses the perception that there is friction between
> innovation/experimentation and stability
> * Not for small enhancements, features, and bug fixes. This should not
> slow down typical development or impede casual contribution to Hadoop
> * Primary mechanism for new features, collecting input, documenting
> design decisions
> * JIRA is good for details, but not for deciding on wide shifts in
> direction
> * Purpose is for author to build consensus and gather dissenting opinions.
>  - All may comment, but Editors will review incoming HEP material
>  - Editors determine only whether the HEP is complete, not whether
> they believe it is a sound idea
>  - Editors are appointed by the PMC
>  - Mechanism for appointing Editors and term of service TBD
>    - Apache Board appoints Shepherds for projects somewhat randomly,
> to projects. A similar mechanism could work for incoming HEPs
>  - Proposal *may* come with code, but not necessarily.
> Drafting/baking of the HEP occurs in public on a list dedicated to
> that particular proposal. Once Editors certify the HEP as complete, it
> is sent to general@ for wider discussion.
>    - The discussion phase begins on general@. The mailing list exists
> to ensure the HEP is complete enough to present to the community.
>  - Some discussion on the difference between posting to general@ and
> posting to the HEP list. Completeness is, of course, subjective. If
> the Editor and Author disagree whether the proposal affects an aspect
> of the framework enough to merit special consideration, it is not
> entirely clear how to resolve the disagreement.
>    - In general, the role of the Editor in the community-driven
> process of Hadoop is not entirely clear. It may be possible to
> optimize it out.
>  - Once discussion ends, the HEP is passed (or fails to pass) by a
> vote of the PMC (mechanics undefined). In Python, the result is
> committed to the repository. A similar practice would make sense in
> Hadoop.
> * Which issues require HEPs?
>  - Discussion ranged. Append, backup namenode, edit log rewrite, et
> al. were examples of features substantial enough to merit a HEP. Pure
> Java CRC is an example of an enhancement that would not. Whether an
> explicit process must be in place to determine whether an issue
> requires a HEP is not clear.
>  - Viewing HEPs as a way of soliciting consensus for an approach
> might be more accurate. Going through the HEP process should always
> improve the chances of a successful proposal
>
> * Evaluation
>  - The proposal may be rejected if it is redundant with existing
> functionality, technically unsound, insufficiently motivated, no
> backwards compatibility story, etc.
>  - Implementation is not necessary, and is lightly discouraged.
> Feedback is less welcome once code is in hand.
>  - Purpose is to be clear about the acceptance criteria for that
> issue, e.g. concerns that the proposal may not scale or may harm
> performance
>  - Dissenting opinions must be recorded accurately. Quoting would be
> a safe practice for the Author to encourage HEP reviewers not to block
> the product of the proposal.
>
> * The testing burden and completion strategy may be ambiguous
>  - Whether the proposal affects scalability may not be testable by
> the implementer. Completing the proposal to address all use cases may
> require considerably more work than the Author is willing or motivated
> to invest.
>  - The HEP discussion on general@ should explore whether such
> objections are merited and reasonable. For example, a particularly
> obscure/esoteric use case could be included as a condition for
> acceptance if the dissenter is willing to invest the resources to
> test/validate it. The process is flexible in this regard.
>    - But it is not infinitely flexible. Backwards compatibility,
> performance regression, availability, and other considerations need
> not be called out in every HEP.
>    - Traditional concerns need to be documented. Acceptance criteria
> should ideally be automated and reproducible in different
> organizations
>
> == Branching ==
> * A patch and a branch are isomorphic from a policy perspective. Of
> course, they are functionally distinct: branches are easier to
> collaborate on and are, generally, longer-lived than are patches. But
> special policies need not be derived to account for these differences,
> which concern the production of the code, not its review and
> acceptance.
> * Some developers find branches to be easier to review than very large
> patches and easier to merge, given a toolchain that supports this.
>  - Subversion currently is difficult to adapt to this model
>  - Could be done on a HEP-by-HEP basis, as a condition for acceptance
> * Eclipse Labs
>  - Branded version of Google Code (same functionality, w/ Eclipse brand)
>  - Not official Eclipse projects, but associated with Eclipse
>  - Apache/Hadoop may consider a similar strategy
>  - Distinct from Apache Labs, as one need not be a committer, follow
> its rules for releases, etc.
>
> == Contrib ==
> * Modules (such as fuse-dfs) are not actively maintained in the main
> repository and would benefit from a release schedule decoupled from
> the rest of Hadoop
> * With few exceptions, the contrib modules have smaller, often
> discrete groups of maintainers. It may be worth exploring whether
> these projects could live elsewhere
>



-- 
Deepak Sharma
http://www.linkedin.com/in/rikindia

--0016363b9360c84e310487b2df52--

From general-return-1595-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 31 14:40:27 2010
Return-Path: <general-return-1595-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93891 invoked from network); 31 May 2010 14:40:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 May 2010 14:40:27 -0000
Received: (qmail 12652 invoked by uid 500); 31 May 2010 14:40:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 12532 invoked by uid 500); 31 May 2010 14:40:25 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 92778 invoked by uid 99); 31 May 2010 12:44:57 -0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of g.maxia@gmail.com designates 74.125.82.48 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:reply-to
         :user-agent:mime-version:to:subject:content-type
         :content-transfer-encoding;
        bh=nit0gBQ+/N5d//kJ6qKgeJQW54dsMMo1Up+9zEPTTA4=;
        b=gx0tM3EjkouBWoAS4TCzxSIwPThjmMIxFanALV54/N3D21tQSs5D1jK7YKqk4wqxwN
         NDeUGuxDBDPSjlPURiC/RXeQ5uOoG/KdsquvmcVRDCMZh9OMnZYAJo7qla+eEtH+fV7Z
         UA7XAe9R4BGyr9qqVtUhbQtE4dirs9UlEPsoA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:reply-to:user-agent:mime-version:to:subject
         :content-type:content-transfer-encoding;
        b=MTEP/oY3vmq+khjKIVMZL+bEmkf/vpVCGZEEg8BJf4GZG19B3PwSR2E214XMrj9ExB
         z2OSWsuGPmn/LbWyeuRpmqq0cnEeORDsi0VCSJMyj1ob/1ysiGTiAINTS4RfGq75dH1C
         e3RTWS9bbHJo97BI35bKxnSR4dx8SQTEr431Y=
Message-ID: <4C03AF18.40409@gmail.com>
Date: Mon, 31 May 2010 14:44:08 +0200
From: Giuseppe Maxia <g.maxia@gmail.com>
Reply-To: opensqlcamp <opensqlcamp@googlegroups.com>
User-Agent: Thunderbird 2.0.0.24 (Macintosh/20100228)
MIME-Version: 1.0
To: databases-discuss@opensolaris.org, derby-user@db.apache.org, 
 firebird-devel@lists.sourceforge.net, general@hadoop.apache.org, 
 hbase-user@hadoop.apache.org, hsqldb-user@lists.sourceforge.net, 
 mongodb-user@googlegroups.com, pbxt-discuss@lists.launchpad.net, 
 pgsql-advocacy@postgresql.org, sqlite-users@sqlite.org, 
 blackray-dev@lists.softmethod.de, maatkit-discuss@googlegroups.com, 
 mmm-devel@googlegroups.com, rethinkdb@googlegroups.com
Subject: OpenSQLCamp EU 2010 - Call for participation is now open
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

OpenSQL Camp is a free conference of, by, and for the open-source database
community of users and developers. The OpenSQLCamp 2010, European Edition
(http://opensqlcamp.org/) will take part in parallel to the Free and Open
Source Conference 2010 (http://froscon.org/) on Saturday 21st and Sunday
22nd August in St. Augustin, Germany, which is located close to Bonn and
Cologne.

The goal of this event is to spread the word about the vibrant communities
and large ecosystems that exist around Open Source Databases and to educate
the attendees about possible alternatives to commercial databases. It is a
place where people come to learn, to participate and to contribute.

We would like to invite your project to participate in this event.

We've set up a call for participation
(http://opensqlcamp.org/Events/2010/Call_for_Participation) - the deadline
for submitting your proposal is July 11th.

We are seeking talks related to Open Source Databases of all kind, not just
relational databases! Submission about tools and technologies related to
OSS databases (e.g. connectors/APIs) are also welcome.

Some ideas and for submissions:

* A how to presentation, showing how to solve a common problem in the
database field.
* An introduction/overview about a certain database project/product or
  related tool
* Providing "best practices" information for administrators
* A deeply technical and developer-centric session about some project's
  internals or an API used to connect to a database

We look forward to your contribution! Please don't hesitate to contact us
via IRC (#opensqlcamp on FreeNode) or our Discussion Group
(http://groups.google.com/group/opensqlcamp)

From general-return-1596-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon May 31 17:17:18 2010
Return-Path: <general-return-1596-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34172 invoked from network); 31 May 2010 17:17:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 May 2010 17:17:18 -0000
Received: (qmail 92923 invoked by uid 500); 31 May 2010 17:17:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 92861 invoked by uid 500); 31 May 2010 17:17:16 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 92853 invoked by uid 99); 31 May 2010 17:17:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 May 2010 17:17:16 +0000
X-ASF-Spam-Status: No, hits=0.9 required=10.0
	tests=AWL,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 May 2010 17:17:11 +0000
Received: by gwj15 with SMTP id 15so3568130gwj.35
        for <general@hadoop.apache.org>; Mon, 31 May 2010 10:16:51 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.187.82 with SMTP id cv18mr735806qcb.83.1275326210709; Mon, 
	31 May 2010 10:16:50 -0700 (PDT)
Received: by 10.229.94.203 with HTTP; Mon, 31 May 2010 10:16:50 -0700 (PDT)
In-Reply-To: <AANLkTilLOmfMwuIFc2kzsXMQ7yhbXtrA0jhb3pigTEht@mail.gmail.com>
References: <AANLkTimLB-YNWnMwa93YL8Lpg9hJLxrCKbYL3YUr6de8@mail.gmail.com>
	<66BC29ED-35C1-4916-BDFE-C768810C6EA6@mac.com>
	<AANLkTimCkqddJ521jIPp0dVHC__vZ2PcavAIrNl334L0@mail.gmail.com>
	<4BFBFAC3.2040807@apache.org>
	<AANLkTilhcUYlFRwc1vc7-sFybjUw5c_uPosptIGb-Gn1@mail.gmail.com>
	<4BFCF6CF.2000605@apache.org>
	<AANLkTilLOmfMwuIFc2kzsXMQ7yhbXtrA0jhb3pigTEht@mail.gmail.com>
Date: Mon, 31 May 2010 10:16:50 -0700
Message-ID: <AANLkTikuyUulssueBs8EQJNeJJN3_NFa7GfflAaeAFgH@mail.gmail.com>
Subject: Re: [DISCUSSION] Proposal for making core Hadoop changes
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e65bba76c319180487e705a4

--0016e65bba76c319180487e705a4
Content-Type: text/plain; charset=ISO-8859-1

A far more lightweight example of multi-issue feature planning in an open
source project comes from Drizzle and their "blueprints":
https://blueprints.launchpad.net/drizzle.

Each "spec" has a drafter, an approver, and an assignee; declares the other
specs on which it depends; points to the relevant branches in the source
tree and issues in the issue tracker; and has a priority, definition state,
and implementation state.

I don't know how it's working out for them in practice, but on paper it
looks quite nice.

On Wed, May 26, 2010 at 9:13 AM, Eli Collins <eli@cloudera.com> wrote:

> > No, but I'd estimate the cost of merging at 1-2 days work a week just to
> > pull in the code *and identify why the tests are failing*. Git may be
> better
> > at merging in changes, but if Hadoop doesn't work on my machine after the
> > merge, I need to identify whether its my code, the merged code, some
> machine
> > quirk, etc. It's the testing that is the problem for me, not the
> > merge effort. That's the Hadoop own tests any my own functional test
> suites,
> > the ones that bring up clusters and push work through. Those are the
> > troublespots, as they do things that hadoop's own tests don't do, like as
> > for all the JSP pages.
>
> I've lived off a git branch of common/hdfs for half a year with a big
> uncommitted patch, it's no where near 1-2 days of effort per week to
> merge in changes from trunk. If the tests are passing on trunk, and
> they fail after your merge then those are real test failures due to
> your change (and therefore should require effort). The issues with
> your internal tests failing due to changes on trunk is the same
> whether you merge or you just do an update - you have to update before
> checking in the patch anyway - so that issue is about the state of
> trunk when you merge or update, rather than about being on a branch.
>
> >
> >> Might find the
> >> following interesting:
> >> http://incubator.apache.org/learn/rules-for-revolutionaries.html
> >
> > There's a long story behind JDD's paper, I'm glad you have read it, it
> does
> > lay out what is effectively the ASF process for effecting significant
> change
> > -but it doesn't imply that's the only process for having changes.
> >
>
> Just to be clear I don't mean imply that branches are the only process
> for making changes. Interesting that this is considered the effective
> ASF process, it hasn't seemed to me that recent big features on hadoop
> have used it, only one I'm aware of that was done on a branch was
> append.
>
> > I think gradual evolution in trunk is good, it lets people play with
> what's
> > coming in. Having lots of separate branches and everyone's private
> release
> > being a merge of many patches that you choose is bad.
>
> Agreed.  Personally I don't think people should release from branches.
> And in practice I don't think you'll see lots of branches, people can
> and would still develop on trunk. Getting changes merged from a branch
> back to trunk before the whole branch is merged is a good thing, the
> whole branch may never be merged and that's OK too. Branches are a
> mechanism, releases are policy.
>
> Thanks,
> Eli
>

--0016e65bba76c319180487e705a4--

From general-return-1120-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 01 07:03:17 2010
Return-Path: <general-return-1120-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83311 invoked from network); 1 Mar 2010 07:03:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Mar 2010 07:03:16 -0000
Received: (qmail 87041 invoked by uid 500); 1 Mar 2010 04:13:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87001 invoked by uid 500); 1 Mar 2010 04:13:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86993 invoked by uid 99); 1 Mar 2010 04:13:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Mar 2010 04:13:15 +0000
X-ASF-Spam-Status: No, hits=4.6 required=10.0
	tests=FS_REPLICA,HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 209.85.222.181 is neither permitted nor denied by domain of peter@motally.com)
Received: from [209.85.222.181] (HELO mail-pz0-f181.google.com) (209.85.222.181)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Mar 2010 04:13:08 +0000
Received: by pzk11 with SMTP id 11so1644579pzk.9
        for <general@hadoop.apache.org>; Sun, 28 Feb 2010 20:12:48 -0800 (PST)
MIME-Version: 1.0
Received: by 10.114.215.40 with SMTP id n40mr2201534wag.209.1267416768157; 
	Sun, 28 Feb 2010 20:12:48 -0800 (PST)
In-Reply-To: <1cbd6f831002281657y7a0eeedbq57b78601c2e11f91@mail.gmail.com>
References: <1cbd6f831002281657y7a0eeedbq57b78601c2e11f91@mail.gmail.com>
Date: Sun, 28 Feb 2010 20:12:48 -0800
Message-ID: <2c44f6691002282012n740a4d93x8b4be25028bfd0d2@mail.gmail.com>
Subject: Re: testing if replication working
From: Peter Sankauskas <peter@motally.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e64ce60c3f97ac0480b576e9

--0016e64ce60c3f97ac0480b576e9
Content-Type: text/plain; charset=UTF-8

Using the web interface, you can browse the data on individual data nodes,
so that is one (manual) way of doing it without powering off any machines.

Kind regards,
Peter Sankauskas



On Sun, Feb 28, 2010 at 4:57 PM, Mag Gam <magawake@gmail.com> wrote:

> I just setup my first hadoop cluster with 5 nodes. What is the best
> way to check if replication is really working? I assume the best way
> is to power down 2 nodes and see if I can still reach my data?
>
> Or are there any others ways?
>
> TIA
>

--0016e64ce60c3f97ac0480b576e9--

From general-return-1115-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 01 07:13:48 2010
Return-Path: <general-return-1115-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19113 invoked from network); 1 Mar 2010 07:13:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Mar 2010 07:13:48 -0000
Received: (qmail 55015 invoked by uid 500); 28 Feb 2010 22:07:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54975 invoked by uid 500); 28 Feb 2010 22:07:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54964 invoked by uid 99); 28 Feb 2010 22:07:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 28 Feb 2010 22:07:46 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mvgfr1@gmail.com designates 74.125.83.48 as permitted sender)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 28 Feb 2010 22:07:38 +0000
Received: by gwaa18 with SMTP id a18so836540gwa.35
        for <general@hadoop.apache.org>; Sun, 28 Feb 2010 14:07:17 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=dpN6DioIcjDsq6P4EhmbaWhA/WrA/x9m16JAl0ugtck=;
        b=IpocQ/5K2Us7k97T/ST2UXDNcrWcnQEgq02xMLdHxsWgJQeaSlva0chGVBH36wYbxi
         R9FODnoAv2GmO1xLmd9RjshNpi0fLTh9lPsCL+g/TV2Kk23UrzRSIsi92BR0yvzkMB/J
         Ne8vBodtjpHQvErUKmHVbW+BKGawgJG0i3sOQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=mwNChpOAZfgoDFcFnNuQ+CPJshv+FrNycSqwCED28loDunaICPLmXCDatyoO7ATmqS
         FFGxuhj47bKUTDXZVjzG46e3mBArgBCWlPJ5aY2vlKO8T1ftSIjuEXhmCvt+qDIuW8IO
         sLDNbUaG+BtobTfaTBtH+q0vhrEjB6oGA20+A=
MIME-Version: 1.0
Received: by 10.150.55.25 with SMTP id d25mr4791000yba.330.1267394837376; Sun, 
	28 Feb 2010 14:07:17 -0800 (PST)
In-Reply-To: <1703587b1002281345k118d980al3ec458cd0d74d343@mail.gmail.com>
References: <1703587b1002281345k118d980al3ec458cd0d74d343@mail.gmail.com>
Date: Sun, 28 Feb 2010 17:07:17 -0500
Message-ID: <4c7b1781002281407l6eaf4acau43e17d958930e73d@mail.gmail.com>
Subject: Re: Adding hard-disks to an existing HDFS cluster
From: Marc Farnum Rendino <mvgfr1@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd705be126fe20480b05b3e
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd705be126fe20480b05b3e
Content-Type: text/plain; charset=ISO-8859-1

That's on my list; is it perhaps as simple as adjusting the config file (ex:
/etc/hadoop/conf/core-site.xml) by adding another value to the dfs.data.dir
property?

(Ref <http://developer.yahoo.com/hadoop/tutorial/module7.html>)

If so, afterward, it may or may not be necessary to rebalance; it can
apparently often just take care of itself, in the normal course of
operation.

On Sun, Feb 28, 2010 at 4:45 PM, Oded Rosen <oded@legolas-media.com> wrote:

> We have an existing HDFS cluster with several datanodes, and we want to add
> each of the datanodes another hard-disk...

--000e0cd705be126fe20480b05b3e--

From general-return-1116-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 01 07:15:06 2010
Return-Path: <general-return-1116-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21894 invoked from network); 1 Mar 2010 07:15:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Mar 2010 07:15:05 -0000
Received: (qmail 59470 invoked by uid 500); 28 Feb 2010 22:15:05 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59428 invoked by uid 500); 28 Feb 2010 22:15:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59420 invoked by uid 99); 28 Feb 2010 22:15:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 28 Feb 2010 22:15:04 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.85.222.195] (HELO mail-pz0-f195.google.com) (209.85.222.195)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 28 Feb 2010 22:14:58 +0000
Received: by pzk33 with SMTP id 33so1577342pzk.5
        for <general@hadoop.apache.org>; Sun, 28 Feb 2010 14:14:37 -0800 (PST)
MIME-Version: 1.0
Received: by 10.142.75.21 with SMTP id x21mr2040043wfa.212.1267395277583; Sun, 
	28 Feb 2010 14:14:37 -0800 (PST)
In-Reply-To: <1703587b1002281345k118d980al3ec458cd0d74d343@mail.gmail.com>
References: <1703587b1002281345k118d980al3ec458cd0d74d343@mail.gmail.com>
Date: Sun, 28 Feb 2010 14:14:37 -0800
Message-ID: <dfe484f01002281414h737fddc3s9dd5467324fd0bf6@mail.gmail.com>
Subject: Re: Adding hard-disks to an existing HDFS cluster
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hey Oded,

You don't need to format to to add disks to DNs. Just format them and
add the directories to dfs.data.dir in the config file, and restart
the DN. The data in an individual DN won't be automatically balanced
across disks when you restart. Rebalancing is not necessary as the DN
will round robin blocks over all disks and stop writing to a disk when
it fills. If you want the disks to be balanced you can do that
manually by copying the block files from the existing data directories
to the new ones--HDFS just checks for the blocks at startup, it
doesn't keep track of which directory they were stored in.

Thanks,
Eli

On Sun, Feb 28, 2010 at 1:45 PM, Oded Rosen <oded@legolas-media.com> wrote:
> We have an existing HDFS cluster with several datanodes, and we want to add
> each of the datanodes another hard-disk, as an addition to the existing
> ones.
> Is there a way of doing this without formatting the cluster? Our aim is to
> save all the data where it is, add and configure the new disks, perform a
> balance - with no format whatsoever.
> Is it possible? if so, how?
>
> Any kind of help will be welcomed.
> Thanks,
>
> --
> Oded
>

From general-return-1117-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 01 07:24:34 2010
Return-Path: <general-return-1117-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 39402 invoked from network); 1 Mar 2010 07:24:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Mar 2010 07:24:34 -0000
Received: (qmail 64841 invoked by uid 500); 28 Feb 2010 22:18:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 64811 invoked by uid 500); 28 Feb 2010 22:18:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 64803 invoked by uid 99); 28 Feb 2010 22:18:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 28 Feb 2010 22:18:33 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mvgfr1@gmail.com designates 209.85.211.199 as permitted sender)
Received: from [209.85.211.199] (HELO mail-yw0-f199.google.com) (209.85.211.199)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 28 Feb 2010 22:18:25 +0000
Received: by ywh37 with SMTP id 37so1088934ywh.2
        for <general@hadoop.apache.org>; Sun, 28 Feb 2010 14:18:05 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=TDFp9pdK/Nayv/Ph7RsRiTi71g25iEKnYsv1e3YOH08=;
        b=YkpvWnAPKp9YN7dPA080SBClWl8qtOD9KNYHI5XyXHGFBkc3YI9UvSutEAVrGivGoO
         Q7hXp2hlXl6oV14K6D356fN/+GXVNVZCNSn04XYRNESpDyvOhvkQWHlI8BRMWP79f8ox
         OuBCihvKAikWf16xeu0j9YVS8q+fPgzVYSmUg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=KoL6uIJvVVK0B4zArs9h2uu2YpHNi8Jdb822r2MqVEBXtaE6QMA+LvSfrTtN7uCh64
         23dmwqo62+7MDeH3ijOBrpLyBM5540Itc1tnyBEjaSEO2cp5iw+Z7DxaSIaKW5m66aiK
         tPSJZeImq4W5MautyopfhPdWB3myZY+kXaYao=
MIME-Version: 1.0
Received: by 10.150.55.25 with SMTP id d25mr4804466yba.330.1267395484957; Sun, 
	28 Feb 2010 14:18:04 -0800 (PST)
In-Reply-To: <dfe484f01002281414h737fddc3s9dd5467324fd0bf6@mail.gmail.com>
References: <1703587b1002281345k118d980al3ec458cd0d74d343@mail.gmail.com>
	 <dfe484f01002281414h737fddc3s9dd5467324fd0bf6@mail.gmail.com>
Date: Sun, 28 Feb 2010 17:18:04 -0500
Message-ID: <4c7b1781002281418k578fabb4n81778bb0890e5c90@mail.gmail.com>
Subject: Re: Adding hard-disks to an existing HDFS cluster
From: Marc Farnum Rendino <mvgfr1@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd705beabbb9c0480b081e0
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd705beabbb9c0480b081e0
Content-Type: text/plain; charset=ISO-8859-1

On Sun, Feb 28, 2010 at 5:14 PM, Eli Collins <eli@cloudera.com> wrote:

> You don't need to format to to add disks to DNs. Just format them and
> add the directories to dfs.data.dir in the config file, and restart
> the DN


Nice; I was hoping it was that simple! :)

I presume it makes no sense to try to spread the NameNode across multiple
disks?

Thanks,

Marc

--000e0cd705beabbb9c0480b081e0--

From general-return-1118-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 01 07:33:52 2010
Return-Path: <general-return-1118-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56913 invoked from network); 1 Mar 2010 07:33:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Mar 2010 07:33:52 -0000
Received: (qmail 72544 invoked by uid 500); 28 Feb 2010 22:27:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72512 invoked by uid 500); 28 Feb 2010 22:27:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72504 invoked by uid 99); 28 Feb 2010 22:27:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 28 Feb 2010 22:27:51 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.85.222.195] (HELO mail-pz0-f195.google.com) (209.85.222.195)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 28 Feb 2010 22:27:45 +0000
Received: by pzk33 with SMTP id 33so1582429pzk.5
        for <general@hadoop.apache.org>; Sun, 28 Feb 2010 14:27:24 -0800 (PST)
MIME-Version: 1.0
Received: by 10.142.249.19 with SMTP id w19mr2038840wfh.194.1267396044692; 
	Sun, 28 Feb 2010 14:27:24 -0800 (PST)
In-Reply-To: <4c7b1781002281418k578fabb4n81778bb0890e5c90@mail.gmail.com>
References: <1703587b1002281345k118d980al3ec458cd0d74d343@mail.gmail.com>
	 <dfe484f01002281414h737fddc3s9dd5467324fd0bf6@mail.gmail.com>
	 <4c7b1781002281418k578fabb4n81778bb0890e5c90@mail.gmail.com>
Date: Sun, 28 Feb 2010 14:27:24 -0800
Message-ID: <dfe484f01002281427o2d4b546ax6ecf59d3c2bce751@mail.gmail.com>
Subject: Re: Adding hard-disks to an existing HDFS cluster
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

> I presume it makes no sense to try to spread the NameNode across multiple
> disks?

Not quite sure what you mean here, but dfs.name.dir (where the NN
stores its metadata) should have multiple directories on different
disks to guard against the failure of any single disk. Many people
also use RAIDed disks and include an NFS mount in dfs.name.dir to have
additional, reliable copies of this data.

Thanks,
Eli

From general-return-1121-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 01 07:49:29 2010
Return-Path: <general-return-1121-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83217 invoked from network); 1 Mar 2010 07:49:29 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Mar 2010 07:49:29 -0000
Received: (qmail 63637 invoked by uid 500); 1 Mar 2010 07:49:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63599 invoked by uid 500); 1 Mar 2010 07:49:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63591 invoked by uid 99); 1 Mar 2010 07:49:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Mar 2010 07:49:28 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=FS_REPLICA,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [132.239.27.72] (HELO tayla.daedalus.ca) (132.239.27.72)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Mar 2010 07:49:22 +0000
Received: from terrence-martins-macbook-pro.local (unknown [192.168.1.253])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	(Authenticated sender: tmartin)
	by tayla.daedalus.ca (Postfix) with ESMTP id 417A6D40016
	for <general@hadoop.apache.org>; Sun, 28 Feb 2010 23:49:00 -0800 (PST)
Message-ID: <4B8B7169.8010805@physics.ucsd.edu>
Date: Sun, 28 Feb 2010 23:48:57 -0800
From: Terrence Martin <tmartin@physics.ucsd.edu>
User-Agent: Thunderbird 2.0.0.23 (Macintosh/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: testing if replication working
References: <1cbd6f831002281657y7a0eeedbq57b78601c2e11f91@mail.gmail.com>
In-Reply-To: <1cbd6f831002281657y7a0eeedbq57b78601c2e11f91@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

Mag Gam wrote:
> I just setup my first hadoop cluster with 5 nodes. What is the best
> way to check if replication is really working? I assume the best way
> is to power down 2 nodes and see if I can still reach my data?
>
> Or are there any others ways?
>
> TIA
>   
Well if you run

hadoop fsck /

it will give you a report of the number of replicas, if a file has less 
than your configured replicas it will tell you that you are under 
replicated.

As for powering off 2 nodes, well that will likely result in a few 
missing blocks unless you have more than 2 replicas of each block. Now 
if you turn off one data node after a while Hadoop will replicate the 
blocks to the remaining 4 nodes, when you turn the 5th back on you will 
have over replicated blocks.

Terrence


From general-return-1119-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 01 08:04:42 2010
Return-Path: <general-return-1119-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 23748 invoked from network); 1 Mar 2010 08:04:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Mar 2010 08:04:41 -0000
Received: (qmail 3213 invoked by uid 500); 1 Mar 2010 00:58:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3148 invoked by uid 500); 1 Mar 2010 00:58:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3138 invoked by uid 99); 1 Mar 2010 00:58:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Mar 2010 00:58:01 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=FS_REPLICA,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of magawake@gmail.com designates 209.85.223.172 as permitted sender)
Received: from [209.85.223.172] (HELO mail-iw0-f172.google.com) (209.85.223.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Mar 2010 00:57:55 +0000
Received: by iwn2 with SMTP id 2so2091515iwn.29
        for <general@hadoop.apache.org>; Sun, 28 Feb 2010 16:57:34 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=XgeAwZxk6joF0JIYq3trbKJn2W/LCbC7mspnktojUK8=;
        b=URHd44xQsjQ5GZhRbazOkKzSVMgRNovVdBKUMUYEjNqzDxjuS+34nBblC1YyQRSfTE
         4fCfu5IVcqG7q33ZhOo0R48PYGzOGM7Oxm7lzwuHquF6VZyu5BOx0eLzeFP6a8ewcP+l
         NquCtS6SDCHbJ4Szo9Dr7L7wM81sv+z1Zrs/4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=YpnUr8thhtCHOcbz5m1DUtCUXR7McPTlt6SeDs+yQsK5Y60O8cY3tiXl/aKbzv/ScA
         OX4jKXK1mo8Fpdgiy6IDSQ7cSa5zQ1egst/eNd0b3vkc2gmTtXEM3sLBQYpHkR4sTwv/
         1eKhGNV2jnVvoejtYsHb5SpYNBqYXZGhhWInA=
MIME-Version: 1.0
Received: by 10.231.154.197 with SMTP id p5mr288458ibw.28.1267405054674; Sun, 
	28 Feb 2010 16:57:34 -0800 (PST)
Date: Sun, 28 Feb 2010 19:57:34 -0500
Message-ID: <1cbd6f831002281657y7a0eeedbq57b78601c2e11f91@mail.gmail.com>
Subject: testing if replication working
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8

I just setup my first hadoop cluster with 5 nodes. What is the best
way to check if replication is really working? I assume the best way
is to power down 2 nodes and see if I can still reach my data?

Or are there any others ways?

TIA

From general-return-1114-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 01 08:59:13 2010
Return-Path: <general-return-1114-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 28353 invoked from network); 1 Mar 2010 08:59:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Mar 2010 08:59:12 -0000
Received: (qmail 29473 invoked by uid 500); 28 Feb 2010 21:45:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 29418 invoked by uid 500); 28 Feb 2010 21:45:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 29410 invoked by uid 99); 28 Feb 2010 21:45:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 28 Feb 2010 21:45:51 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 209.85.223.172 is neither permitted nor denied by domain of oded@legolas-media.com)
Received: from [209.85.223.172] (HELO mail-iw0-f172.google.com) (209.85.223.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 28 Feb 2010 21:45:44 +0000
Received: by iwn2 with SMTP id 2so1997695iwn.29
        for <general@hadoop.apache.org>; Sun, 28 Feb 2010 13:45:23 -0800 (PST)
MIME-Version: 1.0
Received: by 10.231.153.205 with SMTP id l13mr142180ibw.64.1267393523341; Sun, 
	28 Feb 2010 13:45:23 -0800 (PST)
Date: Sun, 28 Feb 2010 23:45:22 +0200
Message-ID: <1703587b1002281345k118d980al3ec458cd0d74d343@mail.gmail.com>
Subject: Adding hard-disks to an existing HDFS cluster
From: Oded Rosen <oded@legolas-media.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=005045013bffbfe0e80480b00c06

--005045013bffbfe0e80480b00c06
Content-Type: text/plain; charset=ISO-8859-1

We have an existing HDFS cluster with several datanodes, and we want to add
each of the datanodes another hard-disk, as an addition to the existing
ones.
Is there a way of doing this without formatting the cluster? Our aim is to
save all the data where it is, add and configure the new disks, perform a
balance - with no format whatsoever.
Is it possible? if so, how?

Any kind of help will be welcomed.
Thanks,

-- 
Oded

--005045013bffbfe0e80480b00c06--

From general-return-1122-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 01 10:46:38 2010
Return-Path: <general-return-1122-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33323 invoked from network); 1 Mar 2010 10:46:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Mar 2010 10:46:38 -0000
Received: (qmail 88438 invoked by uid 500); 1 Mar 2010 10:46:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 88411 invoked by uid 500); 1 Mar 2010 10:46:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 88403 invoked by uid 99); 1 Mar 2010 10:46:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Mar 2010 10:46:37 +0000
X-ASF-Spam-Status: No, hits=-0.8 required=10.0
	tests=ASF_LIST_OPS,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [62.99.145.3] (HELO mx.inode.at) (62.99.145.3)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Mar 2010 10:46:28 +0000
Received: from [213.47.112.79] (port=13228 helo=localhost)
	by smartmx-03.inode.at with esmtpa (Exim 4.69)
	(envelope-from <rho@devc.at>)
	id 1Nm38Z-0000Eb-Ml
	for general@hadoop.apache.org; Mon, 01 Mar 2010 11:46:07 +0100
Resent-From: rho@devc.at
Resent-Date: Mon, 1 Mar 2010 11:46:06 +0100
Resent-Message-ID: <20100301104606.GA9844@odman.int.devc.at>
Resent-To: general@hadoop.apache.org
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on admon.int.devc.at
X-Spam-Level: 
Received: from localhost ([127.0.0.1] helo=admon.int.devc.at)
	by admon.int.devc.at with esmtp (Exim 4.69)
	(envelope-from <rho@devc.at>)
	id 1Nm2t4-0001ab-DB
	for rho@localhost; Mon, 01 Mar 2010 11:30:06 +0100
Received: from mx.inode.at [213.229.60.100]
	by admon.int.devc.at with POP3 (fetchmail-6.3.9-rc2)
	for <rho@localhost> (single-drop); Mon, 01 Mar 2010 11:30:06 +0100 (CET)
Received: from [213.47.112.79] (port=10695 helo=localhost)
	by smartmx-14.inode.at with esmtpa (Exim 4.69)
	(envelope-from <rho@devc.at>)
	id 1Nm2sl-0007eF-W4; Mon, 01 Mar 2010 11:29:50 +0100
Date: Mon, 1 Mar 2010 11:29:28 +0100
From: Robert Barta <rho@devc.at>
To: general-help@hadoop.apache.org
Subject: MapReduce/Hadoop semantic maps
Message-ID: <20100301102927.GI23015@odman.int.devc.at>
Reply-To: rho@devc.at
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Sender: Robert Barta <rho@devc.at>
User-Agent: Mutt/1.5.18 (2008-05-17)
X-Virus-Checked: Checked by ClamAV on apache.org
X-Old-Spam-Status: No, score=-2.4 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00,
	FH_DATE_PAST_20XX autolearn=no version=3.2.5

Hi folks,

Maybe a bit off topic, but then not: I have been playing for a while
with a semantic network covering the Hadoop ecosphere, MapReduce in
general, a bit of CC technologies. My real objective though is on
visualizing that into a "topic map":

   http://kill.devc.at/node/313

I use the technology for semantic text mining, and then for teaching,
uhm, I mean of course knowledge transfer ;-) Clearly, this is work
in progress.

If you care to take a look, I would be interested to hear how _your_
intuitive picture of the landscape differs. I am also happy to add
more specific material. I am a lousy librarian...

\rho


From general-return-1123-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 01 10:49:17 2010
Return-Path: <general-return-1123-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 34580 invoked from network); 1 Mar 2010 10:49:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Mar 2010 10:49:16 -0000
Received: (qmail 95089 invoked by uid 500); 1 Mar 2010 10:49:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95053 invoked by uid 500); 1 Mar 2010 10:49:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95045 invoked by uid 99); 1 Mar 2010 10:49:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Mar 2010 10:49:15 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Mar 2010 10:49:05 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id E4F1D1BA46C
	for <general@hadoop.apache.org>; Mon,  1 Mar 2010 10:48:44 +0000 (GMT)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id UOWPqytNUC4B for <general@hadoop.apache.org>;
	Mon,  1 Mar 2010 10:48:44 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by colossus.hpl.hp.com (Postfix) with ESMTPS id 884BF1BA464
	for <general@hadoop.apache.org>; Mon,  1 Mar 2010 10:48:44 +0000 (GMT)
MailScanner-NULL-Check: 1268045314.08303@GiGpo+Ol7SbCeLQDBqez0Q
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o21AmW93009824
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Mon, 1 Mar 2010 10:48:33 GMT
Message-ID: <4B8B9B80.3070509@apache.org>
Date: Mon, 01 Mar 2010 10:48:32 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Adding hard-disks to an existing HDFS cluster
References: <1703587b1002281345k118d980al3ec458cd0d74d343@mail.gmail.com>	 <dfe484f01002281414h737fddc3s9dd5467324fd0bf6@mail.gmail.com>	 <4c7b1781002281418k578fabb4n81778bb0890e5c90@mail.gmail.com> <dfe484f01002281427o2d4b546ax6ecf59d3c2bce751@mail.gmail.com>
In-Reply-To: <dfe484f01002281427o2d4b546ax6ecf59d3c2bce751@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o21AmW93009824
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

Eli Collins wrote:
>> I presume it makes no sense to try to spread the NameNode across multiple
>> disks?
> 
> Not quite sure what you mean here, but dfs.name.dir (where the NN
> stores its metadata) should have multiple directories on different
> disks to guard against the failure of any single disk. Many people
> also use RAIDed disks and include an NFS mount in dfs.name.dir to have
> additional, reliable copies of this data.

Best of all: a secondary namenode to get the streamed event log, as that 
will mean your cluster restarts faster. You do not want to lose your NN 
data.


We've discussed making it easier to hot-swap a disk drive on a live 
datanode, that could be simplified by having the DN take all data off 
that disks onto its other disks (if there is room), and then copying it 
back in later. Nobody has done any work on this yet:
http://issues.apache.org/jira/browse/HDFS-664

From general-return-1124-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 01 12:38:59 2010
Return-Path: <general-return-1124-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81957 invoked from network); 1 Mar 2010 12:38:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Mar 2010 12:38:58 -0000
Received: (qmail 35421 invoked by uid 500); 1 Mar 2010 12:38:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 35374 invoked by uid 500); 1 Mar 2010 12:38:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 35360 invoked by uid 99); 1 Mar 2010 12:38:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Mar 2010 12:38:57 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=FS_REPLICA,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of magawake@gmail.com designates 209.85.223.172 as permitted sender)
Received: from [209.85.223.172] (HELO mail-iw0-f172.google.com) (209.85.223.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Mar 2010 12:38:49 +0000
Received: by iwn2 with SMTP id 2so2477801iwn.29
        for <general@hadoop.apache.org>; Mon, 01 Mar 2010 04:38:28 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=XHMS6GW9RC4k2+Lv9ZzZGvdAvcZ2JSEE9kvdpbweVa4=;
        b=YrvNehfy8tnreNyOGfZxYxig8S0xRfU5tgCNrKbIumzojwIxTiAPeQoDY0ZZdLTh38
         hEaFUWLYVJ35ADEFjsQqCQmHp9M2vRVHq3pZPtTb9RBaee3q28Xp/4iK7oZB98BrtUFj
         u6tAl3bJrzgd8vSjQJ2pFR6PCItEecq/7a0mM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=mguNZGda5alophWFJsh5YGSKZXcpgRTzBxCEtCZ9ZKLNzMbYT3IIT2dY+0efqQRliO
         VOU88AzMaLMw0BiFQuptg6oCrKWU10s9/wB+mpfbHVRDAxWSg1rSL/F+p7AQV05bxmvo
         mW5Mauafi2pFgTd0o7W37Y37vsO9UbFweEt7o=
MIME-Version: 1.0
Received: by 10.231.150.74 with SMTP id x10mr331574ibv.97.1267447107117; Mon, 
	01 Mar 2010 04:38:27 -0800 (PST)
In-Reply-To: <4B8B7169.8010805@physics.ucsd.edu>
References: <1cbd6f831002281657y7a0eeedbq57b78601c2e11f91@mail.gmail.com>
	 <4B8B7169.8010805@physics.ucsd.edu>
Date: Mon, 1 Mar 2010 07:38:27 -0500
Message-ID: <1cbd6f831003010438v5f52a570p527764ce65288ad3@mail.gmail.com>
Subject: Re: testing if replication working
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks for your responses.





On Mon, Mar 1, 2010 at 2:48 AM, Terrence Martin
<tmartin@physics.ucsd.edu> wrote:
> Mag Gam wrote:
>>
>> I just setup my first hadoop cluster with 5 nodes. What is the best
>> way to check if replication is really working? I assume the best way
>> is to power down 2 nodes and see if I can still reach my data?
>>
>> Or are there any others ways?
>>
>> TIA
>>
>
> Well if you run
>
> hadoop fsck /
>
> it will give you a report of the number of replicas, if a file has less than
> your configured replicas it will tell you that you are under replicated.
>
> As for powering off 2 nodes, well that will likely result in a few missing
> blocks unless you have more than 2 replicas of each block. Now if you turn
> off one data node after a while Hadoop will replicate the blocks to the
> remaining 4 nodes, when you turn the 5th back on you will have over
> replicated blocks.
>
> Terrence
>
>

From general-return-1125-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 01 16:19:48 2010
Return-Path: <general-return-1125-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 81255 invoked from network); 1 Mar 2010 16:19:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Mar 2010 16:19:48 -0000
Received: (qmail 13619 invoked by uid 500); 1 Mar 2010 16:19:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13579 invoked by uid 500); 1 Mar 2010 16:19:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13571 invoked by uid 99); 1 Mar 2010 16:19:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Mar 2010 16:19:46 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mvgfr1@gmail.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Mar 2010 16:19:38 +0000
Received: by gyb13 with SMTP id 13so1154742gyb.35
        for <general@hadoop.apache.org>; Mon, 01 Mar 2010 08:19:17 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=Yg/yaYt91iCgaPYoM8FvrGEllUPdWlUaOk+U1C60Acw=;
        b=AdBQsYwsD/k1LAvdhpDrYfK2QI51YNll6NbtZVqRUATAU4rEPI25QaZ5a5k7ZV0vzU
         xngLEdPiWDTFUDLbZnolB9eZyoFODGk8ij5Mb+pogTCwMG9ko+bZrNvbCMNrYTkfbi5T
         ZmAGbV3hDU4iSXq6UrQw4fiUCh43dut/lDMrc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Ppe7HpQr52WgnoVO1L3x6Lg6HnbsQKZtm8lHuUWUZ17/Do0WVmZwmBt3w+8UgF8eYh
         4/PSs9Twu/BSo6MCI2ovIsrLehJrtRWimFDLtyJWSeycd+RIcBdSRundKmj6M2DbxhcG
         7noxslN6H+K3iTIKt775Vq1QYTG7JXpGq2GfI=
MIME-Version: 1.0
Received: by 10.150.65.18 with SMTP id n18mr1302479yba.136.1267460357681; Mon, 
	01 Mar 2010 08:19:17 -0800 (PST)
In-Reply-To: <dfe484f01002281427o2d4b546ax6ecf59d3c2bce751@mail.gmail.com>
References: <1703587b1002281345k118d980al3ec458cd0d74d343@mail.gmail.com>
	 <dfe484f01002281414h737fddc3s9dd5467324fd0bf6@mail.gmail.com>
	 <4c7b1781002281418k578fabb4n81778bb0890e5c90@mail.gmail.com>
	 <dfe484f01002281427o2d4b546ax6ecf59d3c2bce751@mail.gmail.com>
Date: Mon, 1 Mar 2010 11:19:17 -0500
Message-ID: <4c7b1781003010819v61f9a792t2ac7c8bba2ec4964@mail.gmail.com>
Subject: Re: Adding hard-disks to an existing HDFS cluster
From: Marc Farnum Rendino <mvgfr1@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd5c6fa62f47c0480bf9cfe
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd5c6fa62f47c0480bf9cfe
Content-Type: text/plain; charset=ISO-8859-1

On Sun, Feb 28, 2010 at 5:27 PM, Eli Collins <eli@cloudera.com> wrote:

> dfs.name.dir (where the NN
> stores its metadata) should have multiple directories on different
> disks to guard against the failure of any single disk. Many people
> also use RAIDed disks and include an NFS mount in dfs.name.dir to have
> additional, reliable copies of this data.


Cool; so just to make sure I understand:

The OS presents a single directory to the NameNode; that single directory
can have other measures under it, like RAID.

Right?

Thanks,

Marc

--000e0cd5c6fa62f47c0480bf9cfe--

From general-return-1126-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 01 16:22:13 2010
Return-Path: <general-return-1126-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 82873 invoked from network); 1 Mar 2010 16:22:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Mar 2010 16:22:13 -0000
Received: (qmail 18751 invoked by uid 500); 1 Mar 2010 16:22:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18703 invoked by uid 500); 1 Mar 2010 16:22:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18695 invoked by uid 99); 1 Mar 2010 16:22:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Mar 2010 16:22:11 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mvgfr1@gmail.com designates 74.125.83.48 as permitted sender)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Mar 2010 16:22:05 +0000
Received: by gwaa18 with SMTP id a18so1156979gwa.35
        for <general@hadoop.apache.org>; Mon, 01 Mar 2010 08:21:44 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=R4PcAxAj4pS7v5fTmn8v88v99/Bq8gUOLpGdJCj+4hM=;
        b=T7w/YdjT5Acs7rBkqXT2jqQTFQklOqd/8/N1kzS0lYROyaPnUM6G53rXdc+3aUJihw
         gvdyWu6A4Egh/pgsc9L7il9qXlhT1pXySv/nmhm8oJUFPwfAp22ygwHH7Hc7c/hu/xmp
         rNehec0AbREriOniTlquGirJ1BDYElMX3ws0Q=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=wGYmymC8bcu/d5tiS/NmDlbtXFZdUgdmG+ZHxLQhhnwrWAr817Of+PYYlJxyS5y7xN
         LIaUQxTyb1YCkmBNF/e18Eut51oRtlNgiPp5kdp03V2wSu/ZDMBbRVWxOFjrU46NA5XC
         C/URAmtDW/7qdjeKq9IIkMSeqkSskiqV78d+k=
MIME-Version: 1.0
Received: by 10.150.74.17 with SMTP id w17mr6515413yba.313.1267460504165; Mon, 
	01 Mar 2010 08:21:44 -0800 (PST)
In-Reply-To: <4B8B9B80.3070509@apache.org>
References: <1703587b1002281345k118d980al3ec458cd0d74d343@mail.gmail.com>
	 <dfe484f01002281414h737fddc3s9dd5467324fd0bf6@mail.gmail.com>
	 <4c7b1781002281418k578fabb4n81778bb0890e5c90@mail.gmail.com>
	 <dfe484f01002281427o2d4b546ax6ecf59d3c2bce751@mail.gmail.com>
	 <4B8B9B80.3070509@apache.org>
Date: Mon, 1 Mar 2010 11:21:44 -0500
Message-ID: <4c7b1781003010821w68336d83x3ebcfd1b2939b940@mail.gmail.com>
Subject: Re: Adding hard-disks to an existing HDFS cluster
From: Marc Farnum Rendino <mvgfr1@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd48df61e1f4c0480bfa5db

--000e0cd48df61e1f4c0480bfa5db
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Mar 1, 2010 at 5:48 AM, Steve Loughran <stevel@apache.org> wrote:

> Best of all: a secondary namenode to get the streamed event log, as that
> will mean your cluster restarts faster. You do not want to lose your NN
> data.
>

If the NN data is lost, all the HDFS data is functionally lost, right?

- Marc

--000e0cd48df61e1f4c0480bfa5db--

From general-return-1127-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 01 18:02:57 2010
Return-Path: <general-return-1127-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43831 invoked from network); 1 Mar 2010 18:02:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Mar 2010 18:02:57 -0000
Received: (qmail 50237 invoked by uid 500); 1 Mar 2010 18:02:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50056 invoked by uid 500); 1 Mar 2010 18:02:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50048 invoked by uid 99); 1 Mar 2010 18:02:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Mar 2010 18:02:55 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=FS_REPLICA,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Mar 2010 18:02:45 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:User-Agent:Date:Subject:From:To:
   Message-ID:Thread-Topic:Thread-Index:In-Reply-To:
   Mime-version:Content-type:Content-transfer-encoding;
  b=SvmGqgFBOAxDeyyTxNulRMfCNon/f/sLXt7OT+PbQwIKtwdAxevnu6FM
   XPmNJbu78EBkbyXHSJcixGPpKZpqBWOesZKUNj2I0BWgRvlmmAM04m4wC
   +gkoq+qC8JwLC9u;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1267466565; x=1299002565;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20testing=20if=20replication=20working
   |Date:=20Mon,=2001=20Mar=202010=2010:02:24=20-0800
   |Message-ID:=20<C7B14130.3551%awittenauer@linkedin.com>
   |To:=20<general@hadoop.apache.org>|Mime-version:=201.0
   |Content-transfer-encoding:=207bit|In-Reply-To:=20<4B8B71
   69.8010805@physics.ucsd.edu>;
  bh=QFfXwj7dulTESHkQKy4HppHhOKmURo+YkEmsIA3yP8I=;
  b=sRnnh2UhDPhHGFIO55L3QuM1/pD0T+rQEEqiixhqgOEZsc9AyI1pKDJO
   hmm7sXPgLM7gq8ZRLD7OQpShkpY3VN+wtK1DJhX5keXiJEXu7DliFbMqk
   87tsXphXFelxGri;
X-IronPort-AV: E=Sophos;i="4.49,561,1262592000"; 
   d="scan'208";a="11464825"
Received: from 172.18.36.164 ([172.18.36.164]) by CORP-MAIL.linkedin.biz ([172.18.46.135]) via Exchange Front-End Server mail-access.linkedin.biz ([172.18.46.133]) with Microsoft Exchange Server HTTP-DAV ;
 Mon,  1 Mar 2010 18:02:25 +0000
User-Agent: Microsoft-Entourage/12.10.0.080409
Date: Mon, 01 Mar 2010 10:02:24 -0800
Subject: Re: testing if replication working
From: Allen Wittenauer <awittenauer@linkedin.com>
To: <general@hadoop.apache.org>
Message-ID: <C7B14130.3551%awittenauer@linkedin.com>
Thread-Topic: testing if replication working
Thread-Index: Acq5aVhxQVXP1nv2hku78Ll7JyoyKw==
In-Reply-To: <4B8B7169.8010805@physics.ucsd.edu>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit




On 2/28/10 11:48 PM, "Terrence Martin" <tmartin@physics.ucsd.edu> wrote:

> Mag Gam wrote:
>> I just setup my first hadoop cluster with 5 nodes. What is the best
>> way to check if replication is really working? I assume the best way
>> is to power down 2 nodes and see if I can still reach my data?

> Well if you run
> 
> hadoop fsck /
> 
> it will give you a report of the number of replicas, if a file has less
> than your configured replicas it will tell you that you are under
> replicated.

You can also tell fsck to show you which blocks are on which datanodes ...

hadoop fsck / -files -blocks -locations

... to do a manual inspection.


From general-return-1128-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 01 19:00:51 2010
Return-Path: <general-return-1128-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67186 invoked from network); 1 Mar 2010 19:00:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Mar 2010 19:00:51 -0000
Received: (qmail 54440 invoked by uid 500); 1 Mar 2010 19:00:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 54402 invoked by uid 500); 1 Mar 2010 19:00:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 54335 invoked by uid 99); 1 Mar 2010 19:00:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Mar 2010 19:00:48 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.177] (HELO mail-px0-f177.google.com) (209.85.216.177)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Mar 2010 19:00:40 +0000
Received: by pxi7 with SMTP id 7so961277pxi.30
        for <general@hadoop.apache.org>; Mon, 01 Mar 2010 11:00:19 -0800 (PST)
MIME-Version: 1.0
Received: by 10.142.62.15 with SMTP id k15mr2782427wfa.139.1267470019137; Mon, 
	01 Mar 2010 11:00:19 -0800 (PST)
In-Reply-To: <4c7b1781003010819v61f9a792t2ac7c8bba2ec4964@mail.gmail.com>
References: <1703587b1002281345k118d980al3ec458cd0d74d343@mail.gmail.com>
	 <dfe484f01002281414h737fddc3s9dd5467324fd0bf6@mail.gmail.com>
	 <4c7b1781002281418k578fabb4n81778bb0890e5c90@mail.gmail.com>
	 <dfe484f01002281427o2d4b546ax6ecf59d3c2bce751@mail.gmail.com>
	 <4c7b1781003010819v61f9a792t2ac7c8bba2ec4964@mail.gmail.com>
Date: Mon, 1 Mar 2010 11:00:19 -0800
Message-ID: <dfe484f01003011100m1e36a553pe55232b5b9cf993b@mail.gmail.com>
Subject: Re: Adding hard-disks to an existing HDFS cluster
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Mon, Mar 1, 2010 at 8:19 AM, Marc Farnum Rendino <mvgfr1@gmail.com> wrote:
> On Sun, Feb 28, 2010 at 5:27 PM, Eli Collins <eli@cloudera.com> wrote:
>
>> dfs.name.dir (where the NN
>> stores its metadata) should have multiple directories on different
>> disks to guard against the failure of any single disk. Many people
>> also use RAIDed disks and include an NFS mount in dfs.name.dir to have
>> additional, reliable copies of this data.
>
>
> Cool; so just to make sure I understand:
>
> The OS presents a single directory to the NameNode; that single directory
> can have other measures under it, like RAID.
>
> Right?

Yes, it's good to have multiple directories as well as may each or at
least some of the directories reliable, eg below
/data/<N>/dfs/namenode are local disks and /mnt/filer-hdfs is a
reliable NFS filer.

<name>dfs.name.dir</name>
<value>/data/1/dfs/namenode,/data/2/dfs/namenode,/mnt/filer-hdfs/dfs/namenode</value>

You also of course want to run a secondary namenode (2NN), though
that's more to keep the edits log size (and restart time) manageable
rather than for reliability purposes. Having a 2NN on a separate host
does act as a backup of your NN metadata, though it will always be out
of date. There's work on trunk to add a backup name node which gets a
stream of edits from the NN so it has an update copy of the metadata.

 Thanks,
Eli


>
> Thanks,
>
> Marc
>

From general-return-1129-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 01 19:03:25 2010
Return-Path: <general-return-1129-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67637 invoked from network); 1 Mar 2010 19:03:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Mar 2010 19:03:25 -0000
Received: (qmail 56218 invoked by uid 500); 1 Mar 2010 19:03:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56170 invoked by uid 500); 1 Mar 2010 19:03:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56162 invoked by uid 99); 1 Mar 2010 19:03:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Mar 2010 19:03:23 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.85.222.181] (HELO mail-pz0-f181.google.com) (209.85.222.181)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Mar 2010 19:03:16 +0000
Received: by pzk11 with SMTP id 11so2213186pzk.9
        for <general@hadoop.apache.org>; Mon, 01 Mar 2010 11:02:56 -0800 (PST)
MIME-Version: 1.0
Received: by 10.142.66.10 with SMTP id o10mr2767129wfa.321.1267470176102; Mon, 
	01 Mar 2010 11:02:56 -0800 (PST)
In-Reply-To: <4c7b1781003010821w68336d83x3ebcfd1b2939b940@mail.gmail.com>
References: <1703587b1002281345k118d980al3ec458cd0d74d343@mail.gmail.com>
	 <dfe484f01002281414h737fddc3s9dd5467324fd0bf6@mail.gmail.com>
	 <4c7b1781002281418k578fabb4n81778bb0890e5c90@mail.gmail.com>
	 <dfe484f01002281427o2d4b546ax6ecf59d3c2bce751@mail.gmail.com>
	 <4B8B9B80.3070509@apache.org>
	 <4c7b1781003010821w68336d83x3ebcfd1b2939b940@mail.gmail.com>
Date: Mon, 1 Mar 2010 11:02:55 -0800
Message-ID: <dfe484f01003011102n1af6d76cn1a814eaf2a817d39@mail.gmail.com>
Subject: Re: Adding hard-disks to an existing HDFS cluster
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Mar 1, 2010 at 8:21 AM, Marc Farnum Rendino <mvgfr1@gmail.com> wrote:
> On Mon, Mar 1, 2010 at 5:48 AM, Steve Loughran <stevel@apache.org> wrote:
>
>> Best of all: a secondary namenode to get the streamed event log, as that
>> will mean your cluster restarts faster. You do not want to lose your NN
>> data.
>>
>
> If the NN data is lost, all the HDFS data is functionally lost, right?
>

Yes, it is important that the NN metadata (the fsimage and edits log)
are stored reliably.  There's a great chapter on HDFS administration
in the book Hadoop: The Definitive Guide.

Thanks,
Eli

From general-return-1130-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 01 19:58:18 2010
Return-Path: <general-return-1130-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87695 invoked from network); 1 Mar 2010 19:58:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Mar 2010 19:58:18 -0000
Received: (qmail 50241 invoked by uid 500); 1 Mar 2010 19:58:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 50184 invoked by uid 500); 1 Mar 2010 19:58:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 50148 invoked by uid 99); 1 Mar 2010 19:58:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Mar 2010 19:58:15 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of zshao9@gmail.com designates 209.85.223.182 as permitted sender)
Received: from [209.85.223.182] (HELO mail-iw0-f182.google.com) (209.85.223.182)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Mar 2010 19:58:04 +0000
Received: by iwn12 with SMTP id 12so2727837iwn.21
        for <multiple recipients>; Mon, 01 Mar 2010 11:57:43 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=idsR/Ju4HRWQWrF9EROekZy0oKSVfwC2doDy+RmH9uA=;
        b=iM4MVzupxCaQQWuaXm3wgthLPWz3KKH+YxPTY8RPZLzYFBZZ4zzjxGp4d9zeB+nKcd
         BEeCT85UwJxBScve/sn5jpbzWmGDL7ZV0kZUHe193P34/btOlARe4queqtBGWW8NLPaq
         FwbRKCWo1l/BkHKLaLl5vJSze/Bw4hoRG+2No=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=unaHqPpopjHsO3r4rpp7lnFR16OyArIW0ctVaS67uEbDj2llM553g+uWWevvvrODrG
         /EaIIJs64TEIX6lwWKp/OLSSY2Cb6DJnrzF5tCKIikKgVT3xQma9j8vLDxqqmRDesLFr
         ulY/QPIVGI2dg8hmNEYNUvsTsudKxiqgZs4Ho=
MIME-Version: 1.0
Received: by 10.231.156.80 with SMTP id v16mr101743ibw.79.1267473463512; Mon, 
	01 Mar 2010 11:57:43 -0800 (PST)
In-Reply-To: <34fd060d1002261356u1f37e824x8f879d20ba1ebdb6@mail.gmail.com>
References: <34fd060d1002261356u1f37e824x8f879d20ba1ebdb6@mail.gmail.com>
Date: Mon, 1 Mar 2010 11:57:43 -0800
Message-ID: <34fd060d1003011157k7453df1nba040378bb121a75@mail.gmail.com>
Subject: Re: Hive User Group Meeting 3/18/2010 7pm at Facebook
From: Zheng Shao <zshao9@gmail.com>
To: hive-user@hadoop.apache.org, hive-dev@hadoop.apache.org, 
	core-user@hadoop.apache.org, general@hadoop.apache.org, 
	common-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

We also created a Meetup group in case you prefer to register on meetup.com

http://www.meetup.com/Hive-User-Group-Meeting/calendar/12741356/

We are hosting a Hive User Group Meeting, open to all current and
potential hadoop/hive users.

Agenda:
* Hive Tutorial (Carl Steinbach, cloudera): 20 min
* Hive User Case Study (Eva Tse, netflix): 20 min
* New Features and API (Hive team, Facebook): 25 min
JDBC/ODBC and CTAS(Create Table As Select)
UDF/UDAF/UDTF (User-defined Functions)
Create View/HBaseInputFormat (Hive and HBase integration)
Hive Join Strategy (How Hive does the join)
SerDe (Hive's serialization/deserialization framework)


Hive is a scalable data warehouse infrastructure built on top of
Hadoop. It provides tools to enable easy data ETL, a mechanism to put
structures on the data, and the capability to querying and analysis of
large data sets stored in Hadoop files. Hive defines a simple SQL-like
query language, called HiveQL, that enables users familiar with SQL to
query the data. At the same time, this language also allows
programmers who are familiar with MapReduce to be able to plug in
their custom mappers and reducers to perform more sophisticated
analysis.

The current largest deployment of Hive is the silver cluster at
Facebook, which consists of 1100 nodes with 8 CPU-cores and 12
1TB-disk each. The total capacity is 8800 CPU-cores with 13 PB of raw
storage space. More than 4 TB of compressed data (20+ TB uncompressed)
are loaded into Hive every day.


If you'd like to network with fellow Hive/Hadoop users online, feel
free to find them here:
http://www.facebook.com/event.php?eid=3D319237846974



Zheng

On Fri, Feb 26, 2010 at 1:56 PM, Zheng Shao <zshao9@gmail.com> wrote:
> Hi all,
>
> We are going to hold the second Hive User Group Meeting at 7PM on
> 3/18/2010 Thursday.
>
> The agenda will be:
>
> * Hive Tutorial: 20 min
> * Hive User Case Study: 20 min
> * New Features and API: 25 min
> =A0JDBC/ODBC and CTAS
> =A0UDF/UDAF/UDTF
> =A0Create View/HBaseInputFormat
> =A0Hive Join Strategy
> =A0SerDe
>
> The audience is beginner to intermediate Hive users/developers.
>
> *** The details are here: http://www.facebook.com/event.php?eid=3D3192378=
46974 ***
> *** Please RSVP so we can schedule logistics accordingly. ***
>
> --
> Yours,
> Zheng
>



--=20
Yours,
Zheng

From general-return-1131-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 01 21:14:40 2010
Return-Path: <general-return-1131-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 21173 invoked from network); 1 Mar 2010 21:14:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Mar 2010 21:14:40 -0000
Received: (qmail 72719 invoked by uid 500); 1 Mar 2010 21:14:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 72653 invoked by uid 500); 1 Mar 2010 21:14:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 72645 invoked by uid 99); 1 Mar 2010 21:14:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Mar 2010 21:14:37 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 01 Mar 2010 21:14:37 +0000
Received: (qmail 21038 invoked by uid 99); 1 Mar 2010 21:14:18 -0000
Received: from localhost.apache.org (HELO [192.168.168.108]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Mar 2010 21:14:18 +0000
Message-ID: <4B8C2E28.6090402@apache.org>
Date: Mon, 01 Mar 2010 13:14:16 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
CC: avro-dev@hadoop.apache.org
Subject: Avro: 1.3.0 available
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Avro 1.3.0 is now available.

In this release:
  - the Avro file format has been revised and simplified;
  - the source tree and release artifacts have been restructured;
  - the Java port has been significantly optimized;
  - the Python port has been largely rewritten;
  - the C port is now considerably more complete, supporting data files;
  - a Ruby port has been added, supporting data files and RPC.

Release artifacts may be downloaded from a mirror at:

   http://www.apache.org/dyn/closer.cgi/hadoop/avro/avro-1.3.0/

The Java release is available through Maven.

The full change list is at:

   http://tinyurl.com/avro130relnotes

Doug

From general-return-1132-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 01 21:32:17 2010
Return-Path: <general-return-1132-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30063 invoked from network); 1 Mar 2010 21:32:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Mar 2010 21:32:17 -0000
Received: (qmail 5425 invoked by uid 500); 1 Mar 2010 21:32:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5392 invoked by uid 500); 1 Mar 2010 21:32:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5384 invoked by uid 99); 1 Mar 2010 21:32:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Mar 2010 21:32:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 01 Mar 2010 21:32:05 +0000
Received: (qmail 29796 invoked by uid 99); 1 Mar 2010 21:31:45 -0000
Received: from localhost.apache.org (HELO mail-bw0-f213.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Mar 2010 21:31:45 +0000
Received: by bwz5 with SMTP id 5so2061390bwz.12
        for <multiple recipients>; Mon, 01 Mar 2010 13:31:41 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.152.153 with SMTP id g25mr3504069bkw.158.1267479101017; 
	Mon, 01 Mar 2010 13:31:41 -0800 (PST)
Date: Mon, 1 Mar 2010 13:31:40 -0800
Message-ID: <1267dd3b1003011331x13ee1070gf081ce0c58f549e8@mail.gmail.com>
Subject: Hadoop 0.20.2 is released
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org, private@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hadoop 0.20.2 is released and propagating to mirrors. The list of
changes may be found in the release notes:

http://bit.ly/dd6rwL

And in JIRA:

COMMON: http://bit.ly/9fDo2f
MAPREDUCE: http://bit.ly/aD0BM5
HDFS: http://bit.ly/ckvMfO

Thanks to everyone who worked on this release. -C

From general-return-1133-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 02 00:09:03 2010
Return-Path: <general-return-1133-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73829 invoked from network); 2 Mar 2010 00:09:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Mar 2010 00:09:02 -0000
Received: (qmail 26330 invoked by uid 500); 2 Mar 2010 00:09:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26298 invoked by uid 500); 2 Mar 2010 00:09:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26290 invoked by uid 99); 2 Mar 2010 00:08:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Mar 2010 00:08:59 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mvgfr1@gmail.com designates 209.85.211.199 as permitted sender)
Received: from [209.85.211.199] (HELO mail-yw0-f199.google.com) (209.85.211.199)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Mar 2010 00:08:51 +0000
Received: by ywh37 with SMTP id 37so1757932ywh.2
        for <general@hadoop.apache.org>; Mon, 01 Mar 2010 16:08:30 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=gE61MftzCaWHS1bhGkGtjcC8Lj7Ebe6/RK8MxVhnBwM=;
        b=knCI8h1NJJXazkZ3VRdd7MPvj6pL4r8sB83fYSXWSeip+YG3XcCEek1uHhuDpRDj31
         W4Ydx3LCjDND015mIuWlpRvYngGTFdvn3fQGNq9wSNe0FE3tjdA1kwjOP+aSO5dNWZfn
         ycC7K/iteRPtmgxSSF5jilNDdoBn8R5D+xl8g=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=SX45OnBJ4WHNrpkYMop5h4+MbyRln76pYMb6nNEoBtUSdLFNaoTd1qsIcwIQsHoKNI
         /R0U/2F2863bdZbJGLwyrOxD4CKCtkj8J0s83WJ11OZDx9qRhydznJSYrbihNa/TD1GJ
         /5xaTs8dca6eMrS4Q38abkMFKsdyGtpLoA1UU=
MIME-Version: 1.0
Received: by 10.151.24.21 with SMTP id b21mr7471855ybj.201.1267488510773; Mon, 
	01 Mar 2010 16:08:30 -0800 (PST)
In-Reply-To: <dfe484f01003011100m1e36a553pe55232b5b9cf993b@mail.gmail.com>
References: <1703587b1002281345k118d980al3ec458cd0d74d343@mail.gmail.com>
	 <dfe484f01002281414h737fddc3s9dd5467324fd0bf6@mail.gmail.com>
	 <4c7b1781002281418k578fabb4n81778bb0890e5c90@mail.gmail.com>
	 <dfe484f01002281427o2d4b546ax6ecf59d3c2bce751@mail.gmail.com>
	 <4c7b1781003010819v61f9a792t2ac7c8bba2ec4964@mail.gmail.com>
	 <dfe484f01003011100m1e36a553pe55232b5b9cf993b@mail.gmail.com>
Date: Mon, 1 Mar 2010 19:08:30 -0500
Message-ID: <4c7b1781003011608r40cb6a91vaf6c88e852508032@mail.gmail.com>
Subject: Re: Adding hard-disks to an existing HDFS cluster
From: Marc Farnum Rendino <mvgfr1@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd297a8710aec0480c62a8b
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd297a8710aec0480c62a8b
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Mar 1, 2010 at 2:00 PM, Eli Collins <eli@cloudera.com> wrote:

> Yes, it's good to have multiple directories as well as may each or at
> least some of the directories reliable, eg below
> /data/<N>/dfs/namenode are local disks and /mnt/filer-hdfs is a
> reliable NFS filer.
>
> <name>dfs.name.dir</name>
>
> <value>/data/1/dfs/namenode,/data/2/dfs/namenode,/mnt/filer-hdfs/dfs/namenode</value>
>

Ah; nice - preciate the clarification.


> You also of course want to run a secondary namenode (2NN), though
> that's more to keep the edits log size (and restart time) manageable
> rather than for reliability purposes. Having a 2NN on a separate host
> does act as a backup of your NN metadata, though it will always be out
> of date.


But that's one of those "better than the alternative" situations. :)


> There's work on trunk to add a backup name node which gets a
> stream of edits from the NN so it has an update copy of the metadata.
>

Ah; I think this is alluded to in the conf file, right?

- Marc

--000e0cd297a8710aec0480c62a8b--

From general-return-1134-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 02 00:21:31 2010
Return-Path: <general-return-1134-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76772 invoked from network); 2 Mar 2010 00:21:31 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Mar 2010 00:21:31 -0000
Received: (qmail 45798 invoked by uid 500); 2 Mar 2010 00:21:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45759 invoked by uid 500); 2 Mar 2010 00:21:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45751 invoked by uid 99); 2 Mar 2010 00:21:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Mar 2010 00:21:28 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Mar 2010 00:21:17 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:User-Agent:Date:Subject:From:To:
   Message-ID:Thread-Topic:Thread-Index:In-Reply-To:
   Mime-version:Content-type:Content-transfer-encoding;
  b=SWtaP7Xo17OuSa8yXOaKDHAEgC8Y2D7BGqsZEhfXdI+jyA710L1VjsTI
   if4UZPZoHXS7aEbuvNGyQgvO31zyToY7Fs3ikR6x5sVCdDRe3/3rwYe3j
   yVHxN/5Vs7nPFf5;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1267489277; x=1299025277;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20Adding=20hard-disks=20to=20an=20existin
   g=20HDFS=20cluster|Date:=20Mon,=2001=20Mar=202010=2016:20
   :24=20-0800|Message-ID:=20<C7B199C8.358D%awittenauer@link
   edin.com>|To:=20<general@hadoop.apache.org>|Mime-version:
   =201.0|Content-transfer-encoding:=207bit|In-Reply-To:=20<
   4c7b1781003011608r40cb6a91vaf6c88e852508032@mail.gmail.co
   m>;
  bh=YanaOkIg8vm1iQfzEv1IkRc0eWT0BkiT3xJkw5YAwuY=;
  b=PGG4/2SMN5KJPCcllmS5IVCQHCM70DxYzyalXnCuOfMY2UzblU6bag+V
   ZeImHCi2ObWGda4ahgj4fLpEusIen4ASFrCBpglPZcx4pIZV1VUlxcCyb
   KA9egqkv88luWGE;
X-IronPort-AV: E=Sophos;i="4.49,563,1262592000"; 
   d="scan'208";a="11473749"
Received: from 172.16.19.141 ([172.16.19.141]) by CORP-MAIL.linkedin.biz ([172.18.46.135]) via Exchange Front-End Server mail-access.linkedin.biz ([172.18.46.133]) with Microsoft Exchange Server HTTP-DAV ;
 Tue,  2 Mar 2010 00:20:27 +0000
User-Agent: Microsoft-Entourage/12.10.0.080409
Date: Mon, 01 Mar 2010 16:20:24 -0800
Subject: Re: Adding hard-disks to an existing HDFS cluster
From: Allen Wittenauer <awittenauer@linkedin.com>
To: <general@hadoop.apache.org>
Message-ID: <C7B199C8.358D%awittenauer@linkedin.com>
Thread-Topic: Adding hard-disks to an existing HDFS cluster
Thread-Index: Acq5nibGx8/GAOWDI0qlJceFJVNL0A==
In-Reply-To: <4c7b1781003011608r40cb6a91vaf6c88e852508032@mail.gmail.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org



Marc,

You might find my preso I did on Hadoop at Apachecon EU last year handy:

http://wiki.apache.org/hadoop/HadoopPresentations?action=AttachFile&do=view&
target=aw-apachecon-eu-2009.pdf

aka

http://bit.ly/d3UU4A

It talks a bit about the care and feeding of your Hadoop grid, including how
to configure multiple dirs for the NN.


From general-return-1135-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 02 00:22:37 2010
Return-Path: <general-return-1135-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77198 invoked from network); 2 Mar 2010 00:22:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Mar 2010 00:22:36 -0000
Received: (qmail 48995 invoked by uid 500); 2 Mar 2010 00:22:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 48938 invoked by uid 500); 2 Mar 2010 00:22:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 48930 invoked by uid 99); 2 Mar 2010 00:22:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Mar 2010 00:22:33 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mvgfr1@gmail.com designates 209.85.211.199 as permitted sender)
Received: from [209.85.211.199] (HELO mail-yw0-f199.google.com) (209.85.211.199)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Mar 2010 00:22:25 +0000
Received: by ywh37 with SMTP id 37so1763712ywh.2
        for <general@hadoop.apache.org>; Mon, 01 Mar 2010 16:22:05 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=H7sqWC9B+B8OkEf94AW5/ezrmOwaxh+v307VzOmRJt8=;
        b=IHp5lpOXh2RNPwRoVc5sCyQKx+b/aG1IyEyU6Q+x0GmBXF+5tqna3GtvupZkLtb3LE
         8/mMtn4D7LXOgX6Yi6IYKRIttA4ASp6tbRIeZEyKI+SR9TxahwvL5tApDXCG3Govg0ja
         DfbZzojr1oRhL4f6VclaMpmjq/q0MdcwqJXs0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=u6da6MJGE/ynLgr4VQkvxycGdcxtZCpIA7z9nc7kClIFO7j+WN9Ke4Po3bWkcghmJD
         XtqxbUvRrWykcY+gg87tb7llW1s3yxaWptA91FlLPcEgaFWZcjidSvI2jcMZI5GuKchp
         9t0ws++Q977M911iknKd17jTAh9PhV4vtik+8=
MIME-Version: 1.0
Received: by 10.151.129.4 with SMTP id g4mr5632961ybn.1.1267488909441; Mon, 01 
	Mar 2010 16:15:09 -0800 (PST)
In-Reply-To: <dfe484f01003011102n1af6d76cn1a814eaf2a817d39@mail.gmail.com>
References: <1703587b1002281345k118d980al3ec458cd0d74d343@mail.gmail.com>
	 <dfe484f01002281414h737fddc3s9dd5467324fd0bf6@mail.gmail.com>
	 <4c7b1781002281418k578fabb4n81778bb0890e5c90@mail.gmail.com>
	 <dfe484f01002281427o2d4b546ax6ecf59d3c2bce751@mail.gmail.com>
	 <4B8B9B80.3070509@apache.org>
	 <4c7b1781003010821w68336d83x3ebcfd1b2939b940@mail.gmail.com>
	 <dfe484f01003011102n1af6d76cn1a814eaf2a817d39@mail.gmail.com>
Date: Mon, 1 Mar 2010 19:15:09 -0500
Message-ID: <4c7b1781003011615s18f27f1ds252552ad8a3b37e6@mail.gmail.com>
Subject: Re: Adding hard-disks to an existing HDFS cluster
From: Marc Farnum Rendino <mvgfr1@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001e680f0fa8343b420480c64233
X-Virus-Checked: Checked by ClamAV on apache.org

--001e680f0fa8343b420480c64233
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Mar 1, 2010 at 2:02 PM, Eli Collins <eli@cloudera.com> wrote:

> Yes, it is important that the NN metadata (the fsimage and edits log)
> are stored reliably.  There's a great chapter on HDFS administration
> in the book Hadoop: The Definitive Guide.


Ah; I'd discounted it, because the hadoop ecosystem seems to be evolving so
fast, it seemed better to focus on online and recently-updated info.

Maybe it's not quite that divergent yet. :)

Thanks,

Marc

--001e680f0fa8343b420480c64233--

From general-return-1136-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 02 00:23:26 2010
Return-Path: <general-return-1136-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77707 invoked from network); 2 Mar 2010 00:23:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Mar 2010 00:23:26 -0000
Received: (qmail 52026 invoked by uid 500); 2 Mar 2010 00:23:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51971 invoked by uid 500); 2 Mar 2010 00:23:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51961 invoked by uid 99); 2 Mar 2010 00:23:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Mar 2010 00:23:23 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mvgfr1@gmail.com designates 209.85.217.225 as permitted sender)
Received: from [209.85.217.225] (HELO mail-gx0-f225.google.com) (209.85.217.225)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Mar 2010 00:23:16 +0000
Received: by gxk25 with SMTP id 25so730866gxk.11
        for <general@hadoop.apache.org>; Mon, 01 Mar 2010 16:22:55 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=gh6oAHAxqda5mjNsChsVRZ9GiH/jivhIIZ/1IdNQuDQ=;
        b=ufItpHyVX5oeUcejis5/bbXznEmYsAVhhtT4YVBvq22wZrPy8Q8Oy/+SviGQwz/LhE
         2Ua168zg995BfP4Eys5mXth8POQBo569b8pJKpDhXEmxOlpw+99NN8isUlxOq7Ih/4SV
         aX+4qGj7dbkqo3/H/HRRjy/aiYCKYVLfDlEII=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=V/9oNcn6YnBTthnKXcsCHxFak2EJcaUvGNFWga6aqW4DlZsF067kMm/wou7I+AFpqE
         OmyGHbkxFMfH8Hso4Y06hYlpBbI/cqQoj9c72imMhWkUxhVP+LyjOrx8lxyQ8NRH8mhX
         g+V8hP0BJ/KI1xS5U13kTeSQ8DqeXpsDV8vf8=
MIME-Version: 1.0
Received: by 10.150.166.14 with SMTP id o14mr4839637ybe.219.1267489375550; 
	Mon, 01 Mar 2010 16:22:55 -0800 (PST)
In-Reply-To: <C7B199C8.358D%awittenauer@linkedin.com>
References: <4c7b1781003011608r40cb6a91vaf6c88e852508032@mail.gmail.com>
	 <C7B199C8.358D%awittenauer@linkedin.com>
Date: Mon, 1 Mar 2010 19:22:55 -0500
Message-ID: <4c7b1781003011622r36f54030oebe38324823d02e4@mail.gmail.com>
Subject: Re: Adding hard-disks to an existing HDFS cluster
From: Marc Farnum Rendino <mvgfr1@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd4a76cfc7d740480c65de7

--000e0cd4a76cfc7d740480c65de7
Content-Type: text/plain; charset=ISO-8859-1

On Mon, Mar 1, 2010 at 7:20 PM, Allen Wittenauer
<awittenauer@linkedin.com>wrote:

> You might find my preso I did on Hadoop at Apachecon EU last year handy...


Terrific; thanks!

- Marc

--000e0cd4a76cfc7d740480c65de7--

From general-return-1137-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 02 00:29:58 2010
Return-Path: <general-return-1137-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 79380 invoked from network); 2 Mar 2010 00:29:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Mar 2010 00:29:58 -0000
Received: (qmail 56713 invoked by uid 500); 2 Mar 2010 00:29:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56679 invoked by uid 500); 2 Mar 2010 00:29:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56671 invoked by uid 99); 2 Mar 2010 00:29:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Mar 2010 00:29:55 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.83.176] (HELO mail-pv0-f176.google.com) (74.125.83.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Mar 2010 00:29:49 +0000
Received: by pvc7 with SMTP id 7so98580pvc.35
        for <general@hadoop.apache.org>; Mon, 01 Mar 2010 16:29:28 -0800 (PST)
MIME-Version: 1.0
Received: by 10.142.56.16 with SMTP id e16mr3020812wfa.109.1267489768532; Mon, 
	01 Mar 2010 16:29:28 -0800 (PST)
In-Reply-To: <4c7b1781003011608r40cb6a91vaf6c88e852508032@mail.gmail.com>
References: <1703587b1002281345k118d980al3ec458cd0d74d343@mail.gmail.com>
	 <dfe484f01002281414h737fddc3s9dd5467324fd0bf6@mail.gmail.com>
	 <4c7b1781002281418k578fabb4n81778bb0890e5c90@mail.gmail.com>
	 <dfe484f01002281427o2d4b546ax6ecf59d3c2bce751@mail.gmail.com>
	 <4c7b1781003010819v61f9a792t2ac7c8bba2ec4964@mail.gmail.com>
	 <dfe484f01003011100m1e36a553pe55232b5b9cf993b@mail.gmail.com>
	 <4c7b1781003011608r40cb6a91vaf6c88e852508032@mail.gmail.com>
Date: Mon, 1 Mar 2010 16:29:28 -0800
Message-ID: <dfe484f01003011629m5180bf00t2fa3bdf090bde161@mail.gmail.com>
Subject: Re: Adding hard-disks to an existing HDFS cluster
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

>> There's work on trunk to add a backup name node which gets a
>> stream of edits from the NN so it has an update copy of the metadata.
>>
>
> Ah; I think this is alluded to in the conf file, right?

It's  dfs.namenode.backup.* in conf files but you probably haven't
bumped into them unless you're using trunk, they're not in 20.

> Ah; I'd discounted it, because the hadoop ecosystem seems to be evolving so
> fast, it seemed better to focus on online and recently-updated info.
>
> Maybe it's not quite that divergent yet. :)

The user-facing stuff and discussion are still very pertinent.

Thanks,
Eli

From general-return-1138-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 02 03:53:47 2010
Return-Path: <general-return-1138-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 28299 invoked from network); 2 Mar 2010 03:53:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Mar 2010 03:53:47 -0000
Received: (qmail 16410 invoked by uid 500); 2 Mar 2010 03:53:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16162 invoked by uid 500); 2 Mar 2010 03:53:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16152 invoked by uid 99); 2 Mar 2010 03:53:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Mar 2010 03:53:43 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Mar 2010 03:53:34 +0000
Received: from [192.168.0.198] (snvvpn3-10-72-77-c6.hq.corp.yahoo.com [10.72.77.6])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o223qNt3079298;
	Mon, 1 Mar 2010 19:52:23 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:content-type:
	content-transfer-encoding:mime-version:subject:date:cc:x-mailer;
	b=nXiI3bcCjtG5R+JdhQqeFh3EGbJwDYP5S75PKJXf1sMcrcd/wf/sn9NP21JtMNgw
Message-Id: <A28689CA-DD27-45E0-A48E-2889BDDD3800@yahoo-inc.com>
From: Alan Gates <gates@yahoo-inc.com>
To: pig-user@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Pig 0.6.0 released
Date: Mon, 1 Mar 2010 19:52:22 -0800
Cc: general@hadoop.apache.org
X-Mailer: Apple Mail (2.936)

Pig 0.6.0 is released.  This release includes performance and memory  
usage improvements, a new Accumulator interface for UDFs, and many bug  
fixes.  You can see the details of the release at http://hadoop.apache.org/pig/releases.html

Alan.

From general-return-1139-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 02 12:22:21 2010
Return-Path: <general-return-1139-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 75583 invoked from network); 2 Mar 2010 12:22:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Mar 2010 12:22:21 -0000
Received: (qmail 44633 invoked by uid 500); 2 Mar 2010 12:22:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44478 invoked by uid 500); 2 Mar 2010 12:22:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44470 invoked by uid 99); 2 Mar 2010 12:22:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Mar 2010 12:22:17 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of drew.farris@gmail.com designates 209.85.223.186 as permitted sender)
Received: from [209.85.223.186] (HELO mail-iw0-f186.google.com) (209.85.223.186)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Mar 2010 12:22:08 +0000
Received: by iwn16 with SMTP id 16so119527iwn.20
        for <general@hadoop.apache.org>; Tue, 02 Mar 2010 04:21:48 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=WIE9pqsHXlpWAwANR2B08m9lHDNh6yzdfeLcIWJ2CzM=;
        b=wQVoHQaQjDOAnbAkxrms5Yf8bpxI+tQplABGa72wBeAlmKeK9Fa07ok8Jd6OQCBHin
         MT9YPoQcfqwADblS1/Nd2RLi+G3tPKbJdnDYEeyFq9O/YeKmu7D0tGPXZ/Bl2pJJEmDF
         zdauPOdVX5eXauZwlbHvUa4hQhtFhEDkM9BNY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=NtMwrxP5bqLSdViRBoo5kIwo6OyBz8P+Wwz6SdFGN0HqkXtedlRwNvJp4/hmIcxFOA
         lVOa2J5fp3ftEbKUBXtAvc1LHp8y8h4kUPSjAxhzWzCzd0rKodUUj0Rl3DYn6Tog2O5z
         xA3+ne2TDRAHMVoLIpoKrRWF5rGVuD9LcU8xE=
MIME-Version: 1.0
Received: by 10.231.146.130 with SMTP id h2mr3924ibv.43.1267532304115; Tue, 02 
	Mar 2010 04:18:24 -0800 (PST)
In-Reply-To: <1267dd3b1003011331x13ee1070gf081ce0c58f549e8@mail.gmail.com>
References: <1267dd3b1003011331x13ee1070gf081ce0c58f549e8@mail.gmail.com>
Date: Tue, 2 Mar 2010 07:18:24 -0500
Message-ID: <8f8e14c41003020418s1753b966x9279bc1d457b53b@mail.gmail.com>
Subject: Re: Hadoop 0.20.2 is released
From: Drew Farris <drew.farris@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Congratulations, and thanks for the 0.20.2 release.

I can see the release on the mirrors, but can someone also push it to
the repo at repository.apache.org?

Thanks,
Drew

On Mon, Mar 1, 2010 at 4:31 PM, Chris Douglas <cdouglas@apache.org> wrote:
> Hadoop 0.20.2 is released and propagating to mirrors. The list of
> changes may be found in the release notes:
>
> http://bit.ly/dd6rwL
>
> And in JIRA:
>
> COMMON: http://bit.ly/9fDo2f
> MAPREDUCE: http://bit.ly/aD0BM5
> HDFS: http://bit.ly/ckvMfO
>
> Thanks to everyone who worked on this release. -C
>

From general-return-1140-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 03 03:43:57 2010
Return-Path: <general-return-1140-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29531 invoked from network); 3 Mar 2010 03:43:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Mar 2010 03:43:57 -0000
Received: (qmail 31703 invoked by uid 500); 3 Mar 2010 03:43:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31670 invoked by uid 500); 3 Mar 2010 03:43:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31662 invoked by uid 99); 3 Mar 2010 03:43:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Mar 2010 03:43:50 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of magawake@gmail.com designates 209.85.223.186 as permitted sender)
Received: from [209.85.223.186] (HELO mail-iw0-f186.google.com) (209.85.223.186)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Mar 2010 03:43:43 +0000
Received: by iwn16 with SMTP id 16so974234iwn.20
        for <general@hadoop.apache.org>; Tue, 02 Mar 2010 19:43:23 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=MsBD1j/N9OjhWGL+I4I68R02vM86g+vAG7mYQvnOVo0=;
        b=hkKtF8gYe57+2dcLUTiE4ujOHOk4qmVlmpxpgUuvvElU1d0t59Dq6rMG9PIs0c7jYh
         2+X4T4XaXgsL5AdwvDN9a8nq7M7DpRzBLo4KlkVOw3yToUnl589cQs9GINpYDb4dMNK4
         hHyE0w9z3AubGfM/94QYV5qmW9gd6SroXpviY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=O0yL4HIQeraGLwNo5PCwT0/cOQrAqojBIM6LdEeYjeqmms8ZVmAul+JrtwhviaMn75
         3MG17N4g+WtHGUXqd04nbbdiEpGqSGNZ1LG1CC33MF36SY79fpjqPXmPJSV9sqMcAVsv
         6MvMyV52DvOgtAWY0skY0Z8IyeBGuFGWI+Pd0=
MIME-Version: 1.0
Received: by 10.231.162.13 with SMTP id t13mr116816ibx.3.1267587802743; Tue, 
	02 Mar 2010 19:43:22 -0800 (PST)
Date: Tue, 2 Mar 2010 22:43:22 -0500
Message-ID: <1cbd6f831003021943j6438bd26i72c9c162d7e3e5a3@mail.gmail.com>
Subject: rack awareness help
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8

I have a 5 slave servers and I would like to be rackaware meaning each
server represents 1 rack resulting in 5 racks. I have looked around
for examples online but could not find anything concrete. Can someone
please show me an example on how to set this up?

TIA

From general-return-1141-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 03 06:46:25 2010
Return-Path: <general-return-1141-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44836 invoked from network); 3 Mar 2010 05:38:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Mar 2010 05:38:01 -0000
Received: (qmail 40010 invoked by uid 500); 3 Mar 2010 05:37:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39793 invoked by uid 500); 3 Mar 2010 05:37:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39785 invoked by uid 99); 3 Mar 2010 05:37:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Mar 2010 05:37:53 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.221.177] (HELO mail-qy0-f177.google.com) (209.85.221.177)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Mar 2010 05:37:45 +0000
Received: by qyk7 with SMTP id 7so734383qyk.21
        for <general@hadoop.apache.org>; Tue, 02 Mar 2010 21:37:24 -0800 (PST)
MIME-Version: 1.0
Received: by 10.229.91.68 with SMTP id l4mr73267qcm.68.1267594644584; Tue, 02 
	Mar 2010 21:37:24 -0800 (PST)
In-Reply-To: <1cbd6f831003021943j6438bd26i72c9c162d7e3e5a3@mail.gmail.com>
References: <1cbd6f831003021943j6438bd26i72c9c162d7e3e5a3@mail.gmail.com>
Date: Tue, 2 Mar 2010 21:37:24 -0800
Message-ID: <c9118d4c1003022137r79ff4babifc345be76dba547@mail.gmail.com>
Subject: Re: rack awareness help
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016363b79a8827a520480dee009
X-Virus-Checked: Checked by ClamAV on apache.org

--0016363b79a8827a520480dee009
Content-Type: text/plain; charset=ISO-8859-1

Hey,

Have you looked at
http://hadoop.apache.org/common/docs/r0.20.0/cluster_setup.html#Hadoop+Rack+Awareness?
If there's something lacking in that documentation, perhaps you can submit a
patch with a more useful explanation.

Regards,
Jeff

On Tue, Mar 2, 2010 at 7:43 PM, Mag Gam <magawake@gmail.com> wrote:

> I have a 5 slave servers and I would like to be rackaware meaning each
> server represents 1 rack resulting in 5 racks. I have looked around
> for examples online but could not find anything concrete. Can someone
> please show me an example on how to set this up?
>
> TIA
>

--0016363b79a8827a520480dee009--

From general-return-1142-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 03 12:12:25 2010
Return-Path: <general-return-1142-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85730 invoked from network); 3 Mar 2010 12:12:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Mar 2010 12:12:25 -0000
Received: (qmail 17799 invoked by uid 500); 3 Mar 2010 12:12:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17756 invoked by uid 500); 3 Mar 2010 12:12:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17748 invoked by uid 99); 3 Mar 2010 12:12:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Mar 2010 12:12:17 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of magawake@gmail.com designates 209.85.223.186 as permitted sender)
Received: from [209.85.223.186] (HELO mail-iw0-f186.google.com) (209.85.223.186)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Mar 2010 12:12:09 +0000
Received: by iwn16 with SMTP id 16so1268921iwn.20
        for <general@hadoop.apache.org>; Wed, 03 Mar 2010 04:11:49 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=JmTkc7A2vKlXnbiGp3Sxe9M9f8xyb/VqZzA7v47Ayis=;
        b=NPlYs8QgTXg22MBuuS8f5HfrVVrTRhPZbGyk7XskE1HvYorT2a6b2NEAdrTlvrRdsc
         zbd0iqjwxdtDMkdKYd5jJqPeDuk79h62ciazrtO3kuLlColMxvs+FHqph2goXIwjMey1
         TA+84a+uIyaiVjTNZG3tfxNYSSjpDYltKBxfw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=usIT3f3HYeJFxbECYPcojPuBQb6+rvNiMh4mJRve13jK5v5EfHhqcu3Y/dQB+OE4O9
         TBPR0o1KJd9nZgQ34rOblXYck6ByryB5oPUCA+hhZTW7aOCviIpKmjp2PM54+rRQkIrL
         R2vI24xK+SFoJOnXomlElUGO7JyiHlag1+Nio=
MIME-Version: 1.0
Received: by 10.231.170.14 with SMTP id b14mr12242ibz.26.1267618309276; Wed, 
	03 Mar 2010 04:11:49 -0800 (PST)
In-Reply-To: <c9118d4c1003022137r79ff4babifc345be76dba547@mail.gmail.com>
References: <1cbd6f831003021943j6438bd26i72c9c162d7e3e5a3@mail.gmail.com>
	 <c9118d4c1003022137r79ff4babifc345be76dba547@mail.gmail.com>
Date: Wed, 3 Mar 2010 07:11:49 -0500
Message-ID: <1cbd6f831003030411i161b11e1o7b123d5dd4ccfed9@mail.gmail.com>
Subject: Re: rack awareness help
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8

hello jeff:

yes, I have looked at this page many times.

I guess I am a bit confused with this line, "The default
implementation of the same runs a script/command configured using
topology.script.file.name. If topology.script.file.name is not set,
the rack id /default-rack is returned for any passed IP address."

Do I have to write a script which will convert my IP address into a
unique rack? Once we get the unique id  hdfs will write according to
the id?

An example would be very helpful. There is only 1 paragraph about this
but its far too important not to have an example or two.





On Wed, Mar 3, 2010 at 12:37 AM, Jeff Hammerbacher <hammer@cloudera.com> wrote:
> Hey,
>
> Have you looked at
> http://hadoop.apache.org/common/docs/r0.20.0/cluster_setup.html#Hadoop+Rack+Awareness?
> If there's something lacking in that documentation, perhaps you can submit a
> patch with a more useful explanation.
>
> Regards,
> Jeff
>
> On Tue, Mar 2, 2010 at 7:43 PM, Mag Gam <magawake@gmail.com> wrote:
>
>> I have a 5 slave servers and I would like to be rackaware meaning each
>> server represents 1 rack resulting in 5 racks. I have looked around
>> for examples online but could not find anything concrete. Can someone
>> please show me an example on how to set this up?
>>
>> TIA
>>
>

From general-return-1143-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 03 12:26:39 2010
Return-Path: <general-return-1143-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 90529 invoked from network); 3 Mar 2010 12:26:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Mar 2010 12:26:39 -0000
Received: (qmail 38547 invoked by uid 500); 3 Mar 2010 12:26:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 38519 invoked by uid 500); 3 Mar 2010 12:26:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 38511 invoked by uid 99); 3 Mar 2010 12:26:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Mar 2010 12:26:31 +0000
X-ASF-Spam-Status: No, hits=-1.0 required=10.0
	tests=RCVD_IN_DNSWL_LOW
X-Spam-Check-By: apache.org
Received-SPF: unknown (nike.apache.org: error in processing during lookup of dkumar@qasource.com)
Received: from [161.58.59.72] (HELO qasource.com) (161.58.59.72)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Mar 2010 12:26:22 +0000
Received: from [127.0.0.1] ([116.193.163.2])
	(authenticated bits=0)
	by qasource.com (8.13.6.20060614/8.13.6) with ESMTP id o23CPsDA089778
	for <general@hadoop.apache.org>; Wed, 3 Mar 2010 12:25:59 GMT
Message-ID: <4B8E55A7.80906@qasource.com>
Date: Wed, 03 Mar 2010 17:57:19 +0530
From: Devender <dkumar@qasource.com>
User-Agent: Thunderbird 2.0.0.23 (Windows/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Performance tools for Hadoop
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Team,

Is there anyone who can suggest any performance tools that can be used 
on the hadoop environment?

Regards,
Dev


From general-return-1144-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 03 12:29:23 2010
Return-Path: <general-return-1144-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91421 invoked from network); 3 Mar 2010 12:29:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Mar 2010 12:29:23 -0000
Received: (qmail 43248 invoked by uid 500); 3 Mar 2010 12:29:15 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 43198 invoked by uid 500); 3 Mar 2010 12:29:15 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 43190 invoked by uid 99); 3 Mar 2010 12:29:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Mar 2010 12:29:15 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yangkuntime@gmail.com designates 74.125.92.27 as permitted sender)
Received: from [74.125.92.27] (HELO qw-out-2122.google.com) (74.125.92.27)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Mar 2010 12:29:09 +0000
Received: by qw-out-2122.google.com with SMTP id 9so367325qwb.35
        for <general@hadoop.apache.org>; Wed, 03 Mar 2010 04:28:48 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=r1Z3gRa2SqwjNf2J4ApwX4xkW8Ublfm2g70SPw4uRbo=;
        b=tS5m2d+7nJLQRDbVv6aebtHBXD85fPy8Kzgw95u1VDnLTSGQwlHC2nvV8OjjmaYw7K
         LNln8rggCJgJQYLXCBtYY1xYAqo5V06i5FmY12cbK3q+F09uz5sxftFDA/AzGwMI4Kcu
         NQOPT0dJ7zLf7FV0qm9J/kPTvhhUDhajoeKTQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=vq6dxTpfMpTv4E0/vBdf6j3wm95HoQvtlYtpKur9M9ohqF21fpCI97qlDdCg3kvPZM
         BerwaL0vlul1ZIfvgLWtCLlPnjdRWAkJh3uLoN6WxbUtryli49qZu4LPDTYqJ5YhtxxX
         SArcSCr+tqaW+d3rIoCij5KOSdxlq4EVwVTIM=
MIME-Version: 1.0
Received: by 10.224.81.148 with SMTP id x20mr4157904qak.311.1267619327598; 
	Wed, 03 Mar 2010 04:28:47 -0800 (PST)
In-Reply-To: <4B8E55A7.80906@qasource.com>
References: <4B8E55A7.80906@qasource.com>
Date: Wed, 3 Mar 2010 20:28:47 +0800
Message-ID: <83c9bd211003030428j6c729864ra1f64cd18bb3cc4e@mail.gmail.com>
Subject: Re: Performance tools for Hadoop
From: Kun Yang <yangkuntime@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000feaed62a1bbdab90480e49ff4

--000feaed62a1bbdab90480e49ff4
Content-Type: text/plain; charset=ISO-8859-1

up

On Wed, Mar 3, 2010 at 8:27 PM, Devender <dkumar@qasource.com> wrote:

> Hi Team,
>
> Is there anyone who can suggest any performance tools that can be used on
> the hadoop environment?
>
> Regards,
> Dev
>
>

--000feaed62a1bbdab90480e49ff4--

From general-return-1145-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 03 13:04:42 2010
Return-Path: <general-return-1145-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 7130 invoked from network); 3 Mar 2010 13:04:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Mar 2010 13:04:42 -0000
Received: (qmail 8296 invoked by uid 500); 3 Mar 2010 13:04:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8209 invoked by uid 500); 3 Mar 2010 13:04:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8035 invoked by uid 99); 3 Mar 2010 13:04:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Mar 2010 13:04:33 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of cpbhagtani@gmail.com designates 74.125.92.25 as permitted sender)
Received: from [74.125.92.25] (HELO qw-out-2122.google.com) (74.125.92.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Mar 2010 13:04:27 +0000
Received: by qw-out-2122.google.com with SMTP id 9so375518qwb.35
        for <general@hadoop.apache.org>; Wed, 03 Mar 2010 05:04:06 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=aVX2vF85kSE03mVjOG6EbUXTzUam041PvaUb5dyDfqE=;
        b=iuENCL+dxPmPcsUQIEHEM3hE+P7PvxqSrZQvk61Mibdkc7rv2LVi5HhN2HeGzlpSI5
         1x7soBfnGsi0kpv+yxVyO6F0wzonvCCNBSq1rbBBw//hvQi56SWGdJ7vJ8+nxLO0gb4l
         GLiXgxRUO/srVAOqCedQwJhbGp9Xx4TcArXq4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=MTlyKwQIqId8cHf4oLdt1ZlnJDfKsYTFvNM+uSo4AXvbjWZAJXBhX/Biu9wBo4v6kC
         JhW8nq+n/iJs1Zg0EZn3WL490q0AhDCnrM/5kFOB5qbc6kVO6dDMJWxXKgEEkJIFfAcx
         /zEjyMgp+Wf7pJF302Fkdmsiuu6i92JScyPs4=
MIME-Version: 1.0
Received: by 10.229.38.74 with SMTP id a10mr2204999qce.103.1267621446135; Wed, 
	03 Mar 2010 05:04:06 -0800 (PST)
In-Reply-To: <4B8E55A7.80906@qasource.com>
References: <4B8E55A7.80906@qasource.com>
Date: Wed, 3 Mar 2010 18:34:06 +0530
Message-ID: <4061df21003030504k6f3af613y9a0977704fae4843@mail.gmail.com>
Subject: Re: Performance tools for Hadoop
From: Chandraprakash Bhagtani <cpbhagtani@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016363b826801b0010480e51e05

--0016363b826801b0010480e51e05
Content-Type: text/plain; charset=ISO-8859-1

Hi Devender,

Currently there are 2 ways to analyze performance of hadoop cluster and jobs

 1. Hadoop Vaidya: is a performance diagnostic tool for hadoop jobs which
executes a set of rules against the job counters and gives a report of
performance improvement areas as a result. But Hadoop Vaidya is a separate
contrib module and has a limited set of rules as of now.

 2. Hadoop Metrics(used with Ganglia or Nagios): All  Hadoop daemons expose
runtime metrics which are then analyzed by some cluster monitoring system
like Ganglia or Nagios. Hadoop Metrics has a lot of dependencies on third
party libraries and limited number of metrics as of now.

We are enhancing Hadoop job analysis web UI which will include some features
of Vaidya and Metrics as a feature in Hadoop itself.
It will have no dependencies on any third party libraries. You will have
complete analytic and diagnostic report of your job by just a single click.

We are planning to contribute a patch soon.



On Wed, Mar 3, 2010 at 5:57 PM, Devender <dkumar@qasource.com> wrote:

> Hi Team,
>
> Is there anyone who can suggest any performance tools that can be used on
> the hadoop environment?
>
> Regards,
> Dev
>
>


-- 
Thanks & Regards,
Chandra Prakash Bhagtani,
Impetus Infotech (india) Pvt Ltd.

--0016363b826801b0010480e51e05--

From general-return-1146-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 03 16:57:20 2010
Return-Path: <general-return-1146-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87244 invoked from network); 3 Mar 2010 16:57:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Mar 2010 16:57:20 -0000
Received: (qmail 80241 invoked by uid 500); 3 Mar 2010 16:57:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 80209 invoked by uid 500); 3 Mar 2010 16:57:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 80201 invoked by uid 99); 3 Mar 2010 16:57:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Mar 2010 16:57:11 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Mar 2010 16:57:02 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:User-Agent:Date:Subject:From:To:
   Message-ID:Thread-Topic:Thread-Index:In-Reply-To:
   Mime-version:Content-type:Content-transfer-encoding;
  b=YrGsyH3qtMOeuI6WjzhVKo+3Fk1jTMBC6AIakqqyG7B1r6iDtw71Rj+3
   fwjOHkuGohQ1XBTMM6VNjUiUYwmQgXYwKMTLwKBts01HQpydu1DeRVXDR
   fkXoZE8hdIxmQ8B;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1267635422; x=1299171422;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20rack=20awareness=20help|Date:=20Wed,=20
   03=20Mar=202010=2008:55:49=20-0800|Message-ID:=20<C7B3D49
   5.366B%awittenauer@linkedin.com>|To:=20<general@hadoop.ap
   ache.org>|Mime-version:=201.0|Content-transfer-encoding:
   =207bit|In-Reply-To:=20<1cbd6f831003021943j6438bd26i72c9c
   162d7e3e5a3@mail.gmail.com>;
  bh=0WG6RVmx23UeQDl6+9ITqNLhS8h8lADiTraD/4Mli0Q=;
  b=xIi34lBI0heOstdLo+sriOw9ObShuO8bOqggGIEfkHTMqu+Kn7ShJTLy
   OM92S0dqncpesjIZKRgMKvR6Gz4hJB9EUJJ1YycVHVfc2d3M0uV3DSUvD
   m0NDO4ZmFyAZEDg;
X-IronPort-AV: E=Sophos;i="4.49,574,1262592000"; 
   d="scan'208";a="11513357"
Received: from 172.16.19.141 ([172.16.19.141]) by CORP-MAIL.linkedin.biz ([172.18.46.135]) via Exchange Front-End Server mail-access.linkedin.biz ([172.18.46.133]) with Microsoft Exchange Server HTTP-DAV ;
 Wed,  3 Mar 2010 16:55:51 +0000
User-Agent: Microsoft-Entourage/12.10.0.080409
Date: Wed, 03 Mar 2010 08:55:49 -0800
Subject: Re: rack awareness help
From: Allen Wittenauer <awittenauer@linkedin.com>
To: <general@hadoop.apache.org>
Message-ID: <C7B3D495.366B%awittenauer@linkedin.com>
Thread-Topic: rack awareness help
Thread-Index: Acq68mAQg6WkS1WE3kCWO1cOOqmtuQ==
In-Reply-To: <1cbd6f831003021943j6438bd26i72c9c162d7e3e5a3@mail.gmail.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit




On 3/2/10 7:43 PM, "Mag Gam" <magawake@gmail.com> wrote:

> I have a 5 slave servers and I would like to be rackaware meaning each
> server represents 1 rack resulting in 5 racks. I have looked around
> for examples online but could not find anything concrete. Can someone
> please show me an example on how to set this up?


You're in luck.

If you don't provide a script for rack awareness, it treats every node as if
it was its own rack.



From general-return-1147-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 03 16:58:20 2010
Return-Path: <general-return-1147-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87801 invoked from network); 3 Mar 2010 16:58:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Mar 2010 16:58:20 -0000
Received: (qmail 83048 invoked by uid 500); 3 Mar 2010 16:58:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82984 invoked by uid 500); 3 Mar 2010 16:58:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82976 invoked by uid 99); 3 Mar 2010 16:58:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Mar 2010 16:58:12 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Mar 2010 16:58:02 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:User-Agent:Date:Subject:From:To:
   Message-ID:Thread-Topic:Thread-Index:In-Reply-To:
   Mime-version:Content-type:Content-transfer-encoding;
  b=ADyGxNy+rKBdkEMHJcMaq+3SvrHICcEG9fMHBwv4z2DZunnjW7ictp0B
   2JzXFT9UcxxHc5Fr91GZmPQrbKKLTwy9uWB/zG+qym0U6A1S5ZfXmcEeR
   3GwMRbGCx3FGDwT;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1267635482; x=1299171482;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20rack=20awareness=20help|Date:=20Wed,=20
   03=20Mar=202010=2008:57:33=20-0800|Message-ID:=20<C7B3D4F
   D.366F%awittenauer@linkedin.com>|To:=20<general@hadoop.ap
   ache.org>|Mime-version:=201.0|Content-transfer-encoding:
   =207bit|In-Reply-To:=20<1cbd6f831003030411i161b11e1o7b123
   d5dd4ccfed9@mail.gmail.com>;
  bh=w6tOj+7TOrT+h3+4CR7ORdUftYyNutQoOtYLHuOTmJU=;
  b=kOFAeCr30MixQGMtDKA8kYaLDACaGKPMGQoByAV34gIMVS9HqGKd7KZf
   9iJdm0Ceb8un9wm9Iv/cPYhFiX4KPGksjR/YUv6sk9FKT8tcH6YRHUazm
   dI+hDmVZIjPHLam;
X-IronPort-AV: E=Sophos;i="4.49,574,1262592000"; 
   d="scan'208";a="11513387"
Received: from 172.16.19.141 ([172.16.19.141]) by CORP-MAIL.linkedin.biz ([172.18.46.135]) via Exchange Front-End Server mail-access.linkedin.biz ([172.18.46.133]) with Microsoft Exchange Server HTTP-DAV ;
 Wed,  3 Mar 2010 16:57:34 +0000
User-Agent: Microsoft-Entourage/12.10.0.080409
Date: Wed, 03 Mar 2010 08:57:33 -0800
Subject: Re: rack awareness help
From: Allen Wittenauer <awittenauer@linkedin.com>
To: <general@hadoop.apache.org>
Message-ID: <C7B3D4FD.366F%awittenauer@linkedin.com>
Thread-Topic: rack awareness help
Thread-Index: Acq68p4NhLesKoA2rkCasDJ2E0C1ng==
In-Reply-To: <1cbd6f831003030411i161b11e1o7b123d5dd4ccfed9@mail.gmail.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit




On 3/3/10 4:11 AM, "Mag Gam" <magawake@gmail.com> wrote:
> An example would be very helpful. There is only 1 paragraph about this
> but its far too important not to have an example or two.

I covered this in my preso to apachecon last year:

http://wiki.apache.org/hadoop/HadoopPresentations?action=AttachFile&do=view&
target=aw-apachecon-eu-2009.pdf

aka

http://bit.ly/d3UU4A

You might find the example in/out helpful.  No code, but it is (seriously)
trivial to write.



From general-return-1148-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 03 18:08:27 2010
Return-Path: <general-return-1148-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 10454 invoked from network); 3 Mar 2010 18:08:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Mar 2010 18:08:26 -0000
Received: (qmail 42735 invoked by uid 500); 3 Mar 2010 18:08:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42701 invoked by uid 500); 3 Mar 2010 18:08:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42685 invoked by uid 99); 3 Mar 2010 18:08:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Mar 2010 18:08:17 +0000
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.20] (HELO mrout1-b.corp.re1.yahoo.com) (69.147.107.20)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Mar 2010 18:08:06 +0000
Received: from sp1-ex07cas03.ds.corp.yahoo.com (sp1-ex07cas03.ds.corp.yahoo.com [216.252.116.151])
	by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o23I6QML050697;
	Wed, 3 Mar 2010 10:07:17 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:references:in-reply-to:accept-language:
	content-language:x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:
	content-type:content-transfer-encoding:mime-version;
	b=vxhqrrk1zLzJ3hfnJGE9Ql5Lnm961fzlFjvMNnIWf/1ZVjZRDb9h3PkukomBr/2u
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas03.ds.corp.yahoo.com ([216.252.116.151]) with mapi; Wed, 3 Mar
 2010 10:06:51 -0800
From: Dekel Tankel <dekel@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>,
        "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Date: Wed, 3 Mar 2010 10:06:50 -0800
Subject: Hadoop User Group (Bay Area) - March 24th at Yahoo!
Thread-Topic: Hadoop User Group (Bay Area) - March 24th at Yahoo!
Thread-Index: AcoSMRRU8X5auVLKR8ufbpFTQi1hLgQXaiyQCpfO4pAWCVl24AV6I0+g
Message-ID: <46E2672895E8744A97D7577A6110858B4FDE3FBF3A@SP1-EX07VS01.ds.corp.yahoo.com>
References: <46E2672895E8744A97D7577A6110858B4E1AC69A91@SP1-EX07VS01.ds.corp.yahoo.com>
  <46E2672895E8744A97D7577A6110858B4FD39F6EDF@SP1-EX07VS01.ds.corp.yahoo.com>
 <46E2672895E8744A97D7577A6110858B4FDBF8B10A@SP1-EX07VS01.ds.corp.yahoo.com>
In-Reply-To: <46E2672895E8744A97D7577A6110858B4FDBF8B10A@SP1-EX07VS01.ds.corp.yahoo.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

Hi all,

RSVP is open for the next monthly Bay Area Hadoop user group at the Yahoo! =
Sunnyvale Campus, Wednesday, March 24th, 6PM

Please note that due to the growing demand we are moving the meeting locati=
on to a larger facility *Building C, Second Floor, Classroom 5*=20
(It's in the same campus, just cross the street and walk pass building D to=
 Building C)

Registration and Agenda are available here
http://www.meetup.com/hadoop/calendar/12710846/

Looking forward to seeing you there!

Dekel


From general-return-1149-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 03 19:09:52 2010
Return-Path: <general-return-1149-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43752 invoked from network); 3 Mar 2010 19:09:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Mar 2010 19:09:52 -0000
Received: (qmail 63289 invoked by uid 500); 3 Mar 2010 19:09:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63260 invoked by uid 500); 3 Mar 2010 19:09:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63252 invoked by uid 99); 3 Mar 2010 19:09:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Mar 2010 19:09:43 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of neilbliss@gmail.com designates 209.85.222.189 as permitted sender)
Received: from [209.85.222.189] (HELO mail-pz0-f189.google.com) (209.85.222.189)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Mar 2010 19:09:33 +0000
Received: by pzk27 with SMTP id 27so1899906pzk.2
        for <general@hadoop.apache.org>; Wed, 03 Mar 2010 11:09:12 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=zFdk6uXT6EpfIhglOflV1Xt6wA3xWP7ZuZ2Q3+Cbjj8=;
        b=i765aMPyDY8i2zJF0qnJJ1JPkc/AbEQlFol7SssMprFz6xWCb/GHc1UEffLpLfgli6
         WnWAmPgcb/JNomXTYM92KrwPjOc5WvmKFwphn9G20Hwfyj0B1s1o80gskP9T8JXHiJan
         REQrKgaXunPN8LLmnhKgWF2KLU/VHFrQbnNL4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=uOjlTuFItGwB82xEIf8PnihzACYIg0RaUKqkcaiLbo/NB57OXMzY3z8yuYWwvnQQYg
         QMOlas+PUtlSxEOled/zW9P5uJOeCkK37qp3ZNMLacRreojTyFpLZ7EVGIMXsyKQx0if
         p+tJoHO5qFODjHTuQoH0+H7rRb5HWqDRDIL6Y=
MIME-Version: 1.0
Received: by 10.115.98.13 with SMTP id a13mr4663199wam.88.1267643351907; Wed, 
	03 Mar 2010 11:09:11 -0800 (PST)
In-Reply-To: <C7B3D4FD.366F%awittenauer@linkedin.com>
References: <1cbd6f831003030411i161b11e1o7b123d5dd4ccfed9@mail.gmail.com>
	 <C7B3D4FD.366F%awittenauer@linkedin.com>
Date: Wed, 3 Mar 2010 11:09:11 -0800
Message-ID: <c0078ad61003031109k17d2c915h7f5de8432fb00afd@mail.gmail.com>
Subject: Re: rack awareness help
From: Neil Bliss <neilbliss@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e649d552b1403a0480ea379e
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e649d552b1403a0480ea379e
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Mar 3, 2010 at 8:57 AM, Allen Wittenauer
<awittenauer@linkedin.com>wrote:

> On 3/3/10 4:11 AM, "Mag Gam" <magawake@gmail.com> wrote:
> > An example would be very helpful. There is only 1 paragraph about this
> > but its far too important not to have an example or two.
>
> I covered this in my preso to apachecon last year:
>
>
> http://wiki.apache.org/hadoop/HadoopPresentations?action=AttachFile&do=view&
> target=aw-apachecon-eu-2009.pdf
>
> aka
>
> http://bit.ly/d3UU4A
>
> You might find the example in/out helpful.  No code, but it is (seriously)
> trivial to write.
>

So in the "rack awareness" output, are there meaningful distance metrics
that should be provided, or is it simply a "local rack vs. non-local rack"
determination?

thanks,

Neil

--0016e649d552b1403a0480ea379e--

From general-return-1150-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 03 20:54:30 2010
Return-Path: <general-return-1150-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 69065 invoked from network); 3 Mar 2010 20:54:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Mar 2010 20:54:30 -0000
Received: (qmail 87831 invoked by uid 500); 3 Mar 2010 20:54:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87792 invoked by uid 500); 3 Mar 2010 20:54:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87784 invoked by uid 99); 3 Mar 2010 20:54:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Mar 2010 20:54:21 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [145.100.8.32] (HELO planck.ka.sara.nl) (145.100.8.32)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Mar 2010 20:54:13 +0000
Received: from planck.ka.sara.nl ([145.100.8.32]) by planck.ka.sara.nl
 ([145.100.8.32]) with mapi; Wed, 3 Mar 2010 21:53:50 +0100
From: Evert Lammerts <Evert.Lammerts@sara.nl>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 3 Mar 2010 21:53:49 +0100
Subject: Dutch Hadoop scientific user group?
Thread-Topic: Dutch Hadoop scientific user group?
Thread-Index: AQHKuxOfN0xMa7AJaEWY7HcfJO4pCA==
Message-ID: <ADF94D8555C7A246B86A633685E0178AB867BBD228@planck.ka.sara.nl>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi all,

As the Dutch Center for High Performance Computing & Networking we (SARA - =
http://www.sara.nl) are experimenting with a number of new technologies and=
 paradigms, among which Hadoop. At the moment our experiments are limited t=
o an (also experimental and OpenNebula based) HPC cloud infrastructure, whi=
ch is obviously sub-optimal, and we are considering our options.

We are interested to know whether Hadoop is being used by scientists based =
in The Netherlands, and if so, how and for what it is used. If people are i=
nterested and we can reach a certain number of users, we will be able to or=
ganize and host a Dutch Hadoop scientific user group.

I can be reached on this email address and by both telephone numbers listed=
 below.

Best regards,

Evert Lammerts

Adviseur
SARA Computing & Network Services
High Performance Computing & Visualization
eScience Support Group

Office phone: +31 20 888 4101
Mobile phone: +31 61 055 8303=

From general-return-1151-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 04 01:02:22 2010
Return-Path: <general-return-1151-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 51415 invoked from network); 4 Mar 2010 01:02:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Mar 2010 01:02:22 -0000
Received: (qmail 2719 invoked by uid 500); 4 Mar 2010 01:02:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2682 invoked by uid 500); 4 Mar 2010 01:02:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2674 invoked by uid 99); 4 Mar 2010 01:02:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Mar 2010 01:02:12 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of magawake@gmail.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Mar 2010 01:02:03 +0000
Received: by gyb13 with SMTP id 13so894103gyb.35
        for <general@hadoop.apache.org>; Wed, 03 Mar 2010 17:01:43 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=VrTOF4tTY/nDTktT04qNBBXYiSxY53dhuXYcgfAhIK4=;
        b=aZyLcjuGIKVoK7ecmqfGDbak5Gq3PfOFaTLEaXG6Cqn3e0gn9LLq2fX2yRyINcng1Z
         1nE1LXnlm2pejfP82A0pmOyq6M2DR9XbFK7dvYxlrfBh9L9iOFxe0IeLmFCxn/RJRHL3
         fhOwk2S0RvHbJ6mv6ZyUt4zynWfm1wOkSGkxg=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=TvIO3l4ruuU8qHBKIw6+Togk8dDpdc7XN4rZ2i9KQpcguYbdCvx9+T3nEtVFr76aom
         qqjDjD3F2HK3rSG/YAQuNn6cI7gPNfejXA2VVXh3lwfCImOgYhqHyaydO1fN7nqtmwew
         XeUP+fknjgRQjOZynd9bidgMtJaPSYYVlKrXc=
MIME-Version: 1.0
Received: by 10.150.127.16 with SMTP id z16mr1732045ybc.152.1267664500879; 
	Wed, 03 Mar 2010 17:01:40 -0800 (PST)
In-Reply-To: <C7B3D4FD.366F%awittenauer@linkedin.com>
References: <1cbd6f831003030411i161b11e1o7b123d5dd4ccfed9@mail.gmail.com>
	 <C7B3D4FD.366F%awittenauer@linkedin.com>
Date: Wed, 3 Mar 2010 20:01:40 -0500
Message-ID: <1cbd6f831003031701x5f218995t9a428d7777add9a@mail.gmail.com>
Subject: Re: rack awareness help
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Thanks Alan! Your presentation is very nice!

"If you don't provide a script for rack awareness, it treats every
node as if it was its own rack". I am using the default settings and
the report still says only 1 rack.


Do you mind sharing a script with us on how you determine a rack? and
a sample <configuration> </configuration> syntax?

TIA




On Wed, Mar 3, 2010 at 11:57 AM, Allen Wittenauer
<awittenauer@linkedin.com> wrote:
>
>
>
> On 3/3/10 4:11 AM, "Mag Gam" <magawake@gmail.com> wrote:
>> An example would be very helpful. There is only 1 paragraph about this
>> but its far too important not to have an example or two.
>
> I covered this in my preso to apachecon last year:
>
> http://wiki.apache.org/hadoop/HadoopPresentations?action=3DAttachFile&do=
=3Dview&
> target=3Daw-apachecon-eu-2009.pdf
>
> aka
>
> http://bit.ly/d3UU4A
>
> You might find the example in/out helpful. =C2=A0No code, but it is (seri=
ously)
> trivial to write.
>
>
>

From general-return-1152-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 04 01:09:41 2010
Return-Path: <general-return-1152-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55285 invoked from network); 4 Mar 2010 01:09:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Mar 2010 01:09:40 -0000
Received: (qmail 10915 invoked by uid 500); 4 Mar 2010 01:09:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 10880 invoked by uid 500); 4 Mar 2010 01:09:31 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 10872 invoked by uid 99); 4 Mar 2010 01:09:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Mar 2010 01:09:31 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_HELO_PASS,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [64.81.36.244] (HELO foobar.kobold.org) (64.81.36.244)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Mar 2010 01:09:22 +0000
Received: from localhost.localdomain ([131.215.207.75])
	(authenticated bits=0)
	by foobar.kobold.org (8.14.1/8.14.1) with ESMTP id o23NwVax000437
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Wed, 3 Mar 2010 15:58:32 -0800
Message-ID: <4B8F082C.3070804@hep.caltech.edu>
Date: Wed, 03 Mar 2010 17:09:00 -0800
From: Michael Thomas <thomas@hep.caltech.edu>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.8) Gecko/20100225 Fedora/3.0.2-1.fc12 Thunderbird/3.0.2
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: rack awareness help
References: <1cbd6f831003030411i161b11e1o7b123d5dd4ccfed9@mail.gmail.com>	 <C7B3D4FD.366F%awittenauer@linkedin.com> <1cbd6f831003031701x5f218995t9a428d7777add9a@mail.gmail.com>
In-Reply-To: <1cbd6f831003031701x5f218995t9a428d7777add9a@mail.gmail.com>
X-Enigmail-Version: 1.0.1
Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms050107030105040900080405"

--------------ms050107030105040900080405
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi,

=46rom what I've observed, if you don't provide a script for rack
awareness, hadoop treats all nodes as if they belong the the same rack.

Here is the section in hadoop-site xml where we define our rack
awareness script:

<property>
  <name>topology.script.file.name</name>
  <value>/usr/bin/ip-to-rack.sh</value>
</property>


This is the ip-to-rack.sh script that we use on our cluster.  It assumes
that the reverse lookup of the IP returns a hostname of the form
'compute-x-y', where 'x' is the rack id.  This type of naming is common
for Rocks-managed clusters:

#!/bin/sh

# The default rule assumes that the nodes are connected to the PDU and
switch
# located in the same rack.  Only the exceptions need to be
# explicitly listed here.
for ip in $@ ; do
    hostname=3D`nslookup $ip | grep "name =3D" | awk '{print $4}' | sed -=
e
's/\.local\.$//' `
    case $hostname in
        compute-14-3) rack=3D"/Rack15" ;;
        *)
            rack=3D`echo $hostname | sed -e
's/^[a-z]*-\([0-9]*\)-[0-9]*.*/\/Rack\1/'`
            ;;
        esac
        echo $rack
done

Hope this helps,

--Mike

On 03/03/2010 05:01 PM, Mag Gam wrote:
> Thanks Alan! Your presentation is very nice!
>=20
> "If you don't provide a script for rack awareness, it treats every
> node as if it was its own rack". I am using the default settings and
> the report still says only 1 rack.
>=20
>=20
> Do you mind sharing a script with us on how you determine a rack? and
> a sample <configuration> </configuration> syntax?
>=20
> TIA
>=20
>=20
>=20
>=20
> On Wed, Mar 3, 2010 at 11:57 AM, Allen Wittenauer
> <awittenauer@linkedin.com> wrote:
>>
>>
>>
>> On 3/3/10 4:11 AM, "Mag Gam" <magawake@gmail.com> wrote:
>>> An example would be very helpful. There is only 1 paragraph about thi=
s
>>> but its far too important not to have an example or two.
>>
>> I covered this in my preso to apachecon last year:
>>
>> http://wiki.apache.org/hadoop/HadoopPresentations?action=3DAttachFile&=
do=3Dview&
>> target=3Daw-apachecon-eu-2009.pdf
>>
>> aka
>>
>> http://bit.ly/d3UU4A
>>
>> You might find the example in/out helpful.  No code, but it is (seriou=
sly)
>> trivial to write.
>>
>>
>>



--------------ms050107030105040900080405
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIMVjCC
A/gwggLgoAMCAQICASkwDQYJKoZIhvcNAQEFBQAwdTETMBEGCgmSJomT8ixkARkWA25ldDES
MBAGCgmSJomT8ixkARkWAkVTMQ4wDAYDVQQKEwVFU25ldDEgMB4GA1UECxMXQ2VydGlmaWNh
dGUgQXV0aG9yaXRpZXMxGDAWBgNVBAMTD0VTbmV0IFJvb3QgQ0EgMTAeFw0wMjEyMDUwODAw
MDBaFw0xMzAxMjUwODAwMDBaMGkxEzARBgoJkiaJk/IsZAEZFgNvcmcxGDAWBgoJkiaJk/Is
ZAEZFghET0VHcmlkczEgMB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9yaXRpZXMxFjAUBgNV
BAMTDURPRUdyaWRzIENBIDEwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC09dYj
YaPbCD5mtbiQb7Ka3y1qAm0ZcqKCFciWcfe8Kwcuy9tjHuIsLf9ZItdkDW4xy8sua9nJlx3K
lwjtumTMtOtg35KZCknUd8KM4VGTSFdLVG9AbNayef76caVCGM1+jyF0Lq03kauGOPTcNfZe
1TZa3e1c9rc8ljV5OSWa/mfsCACyS5zFIWu0yIDNyJdf+n0hwaPN53wllpJ30taD+JBjQ7h2
k4xRWzeaznLOb9OztZVRA/1sVze+iczFh2xwa4VdGy0eIIPw1pfvYwxO36rm0S109qvbsNla
roPRbxerPKakQLpKe034Xcx7gBPqUk/FxoRRWin5EWN3rz9LAgMBAAGjgZ4wgZswDgYDVR0P
AQH/BAQDAgGGMBEGCWCGSAGG+EIBAQQEAwIAhzAdBgNVHQ4EFgQUyhkdEo5upDhdQtQxDgjb
2Y0XDV0wHwYDVR0jBBgwFoAUvF1NSC/4NZRZq1yJSz7RsjoUAeowDwYDVR0TAQH/BAUwAwEB
/zAlBgNVHREEHjAcgRpET0VHcmlkcy1DQS0xQGRvZWdyaWRzLm9yZzANBgkqhkiG9w0BAQUF
AAOCAQEAZNVrIDLqe39CEOiJt7Q7EpBPhAihMvDTSf/42u0SMbUmChww4mLmph5DBghZUVF8
Yn59kRZMn1QLOtO1HzLqvAvPITacZVPlJgG2IXzlR636YghZFAycbIUEOJDBHR4vtQO1KDxg
ZwvAbtmKIoxvhUCq2xsfFt9kCBBn+JYtQ6O5LsBJq3PmuubeMcc7mbQAfJZ7h/3QghgkFIhm
E1+LBXPJbkuP8vgfg6h2BKoAf5TFfZECgGZKimfN110tBvfedGZwYYd3/GsJc83B0JN1gny0
gqNVPm392UchXGeBRrHnm2gkhIkr48Oq6EmNGV9/a6XfbplQW/JWbtPVPWkaizCCBCkwggMR
oAMCAQICAwChZzANBgkqhkiG9w0BAQUFADBpMRMwEQYKCZImiZPyLGQBGRYDb3JnMRgwFgYK
CZImiZPyLGQBGRYIRE9FR3JpZHMxIDAeBgNVBAsTF0NlcnRpZmljYXRlIEF1dGhvcml0aWVz
MRYwFAYDVQQDEw1ET0VHcmlkcyBDQSAxMB4XDTEwMDEyNTE4MzEyMFoXDTExMDEyNTE4MzEy
MFowYDETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCGRvZWdyaWRzMQ8w
DQYDVQQLEwZQZW9wbGUxHjAcBgNVBAMTFU1pY2hhZWwgVGhvbWFzIDcyNTM3NDCCASIwDQYJ
KoZIhvcNAQEBBQADggEPADCCAQoCggEBALchCLXStiJKw5u2+nFW0020jqo1X8/kRB9kus8a
6Op5ds3j5O3IZkyZ41tQwE9Rc0yRDiDGpT1jL/n2LK1HPXvZbasmPCxxTMY3kNTm4fWAnRS7
UzLo0Adaz81bcT8Buo7Il1FCS684LhsK19JChqU2/iY8G7H0uBImj6QDLR0+w5QsLfD7aOXx
M8Njhse1ZqMyjwjhw5FqWyWNhlKi1oW42eXtYhbsqyAQZSXLoz2SwqBnLTRqcRMn4GgvB83Z
1FObyb79HFiuqlbaMlXWy0s8ywajJnMG8LsgwkNJZQLJsPalz5rwe/pOG1abasLMBVpJGGSB
AQktY8s5M+U9RlMCAwEAAaOB4jCB3zARBglghkgBhvhCAQEEBAMCBeAwDgYDVR0PAQH/BAQD
AgTwMDYGA1UdIAQvMC0wDQYLKoZIhvdMAwcBAwEwDAYKKoZIhvdMBQICATAOBgwqhkiG90wF
AgMCAQEwPgYDVR0fBDcwNTAzoDGgL4YtaHR0cDovL2NybC5kb2Vncmlkcy5vcmcvMWMzZjJj
YTgvMWMzZjJjYTguY3JsMCEGA1UdEQQaMBiBFnRob21hc0BoZXAuY2FsdGVjaC5lZHUwHwYD
VR0jBBgwFoAUyhkdEo5upDhdQtQxDgjb2Y0XDV0wDQYJKoZIhvcNAQEFBQADggEBAGcleFpm
d9Ho37gHZrrF0zulSQ3Aemrbx2O4EVJotQbtAs2v4nnH0eAa0F37BXRA6DAvgaC4YODA6z5M
HHE2mnsIebv0nS8DLj1yN7C1nqDSUXlaIzExEGQ1vXG2XlVo6pMZgJd2ML9+dyUUOcnNj0kQ
GeMysWBHbDChdfVG8y2PuGzoFY/ikNtWWhTNcPVZHIL5I2JCYqV5lyLQhlkC0Q5eqbqYKAq1
GuQmLO8hPfzkb6qbAkzxB8/FFqjlPgBgFcd9Nf2fAI14kI77iRbEuu8jSTPx13p6uYvcX4hy
1cW4Y7lnE8bBkAFWrUuNGc+aw7Ang6rJANWXtF/tN+UqUOcwggQpMIIDEaADAgECAgMAoWcw
DQYJKoZIhvcNAQEFBQAwaTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkW
CERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UEAxMN
RE9FR3JpZHMgQ0EgMTAeFw0xMDAxMjUxODMxMjBaFw0xMTAxMjUxODMxMjBaMGAxEzARBgoJ
kiaJk/IsZAEZFgNvcmcxGDAWBgoJkiaJk/IsZAEZFghkb2VncmlkczEPMA0GA1UECxMGUGVv
cGxlMR4wHAYDVQQDExVNaWNoYWVsIFRob21hcyA3MjUzNzQwggEiMA0GCSqGSIb3DQEBAQUA
A4IBDwAwggEKAoIBAQC3IQi10rYiSsObtvpxVtNNtI6qNV/P5EQfZLrPGujqeXbN4+TtyGZM
meNbUMBPUXNMkQ4gxqU9Yy/59iytRz172W2rJjwscUzGN5DU5uH1gJ0Uu1My6NAHWs/NW3E/
AbqOyJdRQkuvOC4bCtfSQoalNv4mPBux9LgSJo+kAy0dPsOULC3w+2jl8TPDY4bHtWajMo8I
4cORalsljYZSotaFuNnl7WIW7KsgEGUly6M9ksKgZy00anETJ+BoLwfN2dRTm8m+/RxYrqpW
2jJV1stLPMsGoyZzBvC7IMJDSWUCybD2pc+a8Hv6ThtWm2rCzAVaSRhkgQEJLWPLOTPlPUZT
AgMBAAGjgeIwgd8wEQYJYIZIAYb4QgEBBAQDAgXgMA4GA1UdDwEB/wQEAwIE8DA2BgNVHSAE
LzAtMA0GCyqGSIb3TAMHAQMBMAwGCiqGSIb3TAUCAgEwDgYMKoZIhvdMBQIDAgEBMD4GA1Ud
HwQ3MDUwM6AxoC+GLWh0dHA6Ly9jcmwuZG9lZ3JpZHMub3JnLzFjM2YyY2E4LzFjM2YyY2E4
LmNybDAhBgNVHREEGjAYgRZ0aG9tYXNAaGVwLmNhbHRlY2guZWR1MB8GA1UdIwQYMBaAFMoZ
HRKObqQ4XULUMQ4I29mNFw1dMA0GCSqGSIb3DQEBBQUAA4IBAQBnJXhaZnfR6N+4B2a6xdM7
pUkNwHpq28djuBFSaLUG7QLNr+J5x9HgGtBd+wV0QOgwL4GguGDgwOs+TBxxNpp7CHm79J0v
Ay49cjewtZ6g0lF5WiMxMRBkNb1xtl5VaOqTGYCXdjC/fnclFDnJzY9JEBnjMrFgR2wwoXX1
RvMtj7hs6BWP4pDbVloUzXD1WRyC+SNiQmKleZci0IZZAtEOXqm6mCgKtRrkJizvIT385G+q
mwJM8QfPxRao5T4AYBXHfTX9nwCNeJCO+4kWxLrvI0kz8dd6ermL3F+IctXFuGO5ZxPGwZAB
Vq1LjRnPmsOwJ4OqyQDVl7Rf7TflKlDnMYIDXjCCA1oCAQEwcDBpMRMwEQYKCZImiZPyLGQB
GRYDb3JnMRgwFgYKCZImiZPyLGQBGRYIRE9FR3JpZHMxIDAeBgNVBAsTF0NlcnRpZmljYXRl
IEF1dGhvcml0aWVzMRYwFAYDVQQDEw1ET0VHcmlkcyBDQSAxAgMAoWcwCQYFKw4DAhoFAKCC
AcMwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTAwMzA0MDEw
OTAwWjAjBgkqhkiG9w0BCQQxFgQUi9SpjOgoVv8dQC3p6Y0y3iHMNnYwXwYJKoZIhvcNAQkP
MVIwUDALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3
DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMH8GCSsGAQQBgjcQBDFyMHAwaTETMBEG
CgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERPRUdyaWRzMSAwHgYDVQQLExdD
ZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3JpZHMgQ0EgMQIDAKFnMIGB
BgsqhkiG9w0BCRACCzFyoHAwaTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixk
ARkWCERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UE
AxMNRE9FR3JpZHMgQ0EgMQIDAKFnMA0GCSqGSIb3DQEBAQUABIIBADb66A4KCGTz/QVv3JMn
2IOdQdLFViDAS5TF6wI9dJcHQKULRNOXhm0cVqf+NAUru29M6mKoipwg+5RczbxcRc4qzDJD
cAI2gYyiGdD11KM/6Ux1bYXap9oVgDBvS9QeYbpY9Ur+ltH0dJtFJ8C2ZZ+xPsWK0foE3NgH
Bz1uiFjvtHhB5NGHawiAb9DGoReUecWAD+8xaox6T+N6S2ml8qcZd1ZbtUCeCvVt9zTK7HVF
EGgUPxvn7nGYI1/Cq+OU2jog8pb/IEpZcvcgkMEueZ7DCAa81u/BAeJPvdtXWzvcarpRLt/4
RKw+aALxTWXXtUwBtMSZeAW/rPZJTfmyAd0AAAAAAAA=
--------------ms050107030105040900080405--

From general-return-1153-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 04 03:54:24 2010
Return-Path: <general-return-1153-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 86059 invoked from network); 4 Mar 2010 03:54:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Mar 2010 03:54:24 -0000
Received: (qmail 24765 invoked by uid 500); 4 Mar 2010 03:54:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24673 invoked by uid 500); 4 Mar 2010 03:54:14 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24665 invoked by uid 99); 4 Mar 2010 03:54:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Mar 2010 03:54:14 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of magawake@gmail.com designates 209.85.223.192 as permitted sender)
Received: from [209.85.223.192] (HELO mail-iw0-f192.google.com) (209.85.223.192)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Mar 2010 03:54:06 +0000
Received: by iwn30 with SMTP id 30so1795872iwn.32
        for <general@hadoop.apache.org>; Wed, 03 Mar 2010 19:53:45 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=I23orDpkLe+Bed2a6fhLCBUysBQwUaIkQ8rW0aPeyj0=;
        b=oUBGW9se3rpXSSFuHiT2x6qr6RS1kF/Lp5HzwwuIkFZETngnSMk2n6Keb+a7x0BmuU
         0lRwZvehj8wtL5LtUJZ9eGO1ZZmzBJaS+Tr+FynW8IVDBDIslr0jdYqLmH6InJvuK76s
         P6428H8yppHNdtYMzJTBDen+ig0fY5j+csluk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=G3pplpg1qJ0amiXNEhXs00n5vPik70GUItsaLBUfX7/RAAGUWuWnHkw/qJKovtnaGI
         5gYVGMuNfttlMTJ9GnHskNNoZCKuqTug0VCVnFNohVXzDIBEDATHF9gmNovwjxJ+2LJv
         IJsCB1b9rDmz3Y2tpH1zRzeYn3xtcrj0K64Aw=
MIME-Version: 1.0
Received: by 10.231.146.203 with SMTP id i11mr200113ibv.19.1267674825082; Wed, 
	03 Mar 2010 19:53:45 -0800 (PST)
Date: Wed, 3 Mar 2010 22:53:45 -0500
Message-ID: <1cbd6f831003031953p7462a54bq1cd5194ba00bf22a@mail.gmail.com>
Subject: setting a proper threshold
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
X-Virus-Checked: Checked by ClamAV on apache.org

I plan to setup HDFS on 20 servers. I plan on using /fs for my data. I
want /fs to have atleast 20% free therefore I don't want this
filesystem to be filled up with hdfs data. Is there a way to restrict
hadoop so it does not write here if the filesystem is  over 80%?

TIA

From general-return-1154-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 04 05:22:43 2010
Return-Path: <general-return-1154-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 2581 invoked from network); 4 Mar 2010 05:22:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Mar 2010 05:22:42 -0000
Received: (qmail 75675 invoked by uid 500); 4 Mar 2010 05:22:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 75370 invoked by uid 500); 4 Mar 2010 05:22:32 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 75362 invoked by uid 99); 4 Mar 2010 05:22:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Mar 2010 05:22:32 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Mar 2010 05:22:23 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:User-Agent:Date:Subject:From:To:
   Message-ID:Thread-Topic:Thread-Index:In-Reply-To:
   Mime-version:Content-type:Content-transfer-encoding;
  b=YU9PHbmwQzjqUbDEpmbMshBaN/4O1mAnbLRRaF55wcQwnpm77oYQY69N
   OBhKiIJlY2CgHTPT1OVMKWkDfTX1cwmxtV3ZwxuT625HzjDZHwjpms3AU
   1ydmsPgjCAXlD+t;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1267680143; x=1299216143;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20rack=20awareness=20help|Date:=20Wed,=20
   03=20Mar=202010=2021:21:21=20-0800|Message-ID:=20<C7B4835
   1.36DF%awittenauer@linkedin.com>|To:=20<general@hadoop.ap
   ache.org>|Mime-version:=201.0|Content-transfer-encoding:
   =207bit|In-Reply-To:=20<1cbd6f831003031701x5f218995t9a428
   d7777add9a@mail.gmail.com>;
  bh=FDZkDeIYx6WU97WSvQjnholKSOtjKRScWAtV6Y5Mgt4=;
  b=c/zIVga1iWfOGumHR+fkiIzEEAxmdzMGkFI7CTHP4nWWS999PR8/hBkt
   F4qAjMhg3BtX1oW6cpqFqHa3oHc3n5oxFr7Uub75mtHMBblkjm0JmuAy+
   l4vE1w6P0Tj6AIk;
X-IronPort-AV: E=Sophos;i="4.49,579,1262592000"; 
   d="scan'208";a="11527244"
Received: from 172.18.36.181 ([172.18.36.181]) by CORP-MAIL.linkedin.biz ([172.18.46.135]) via Exchange Front-End Server mail-access.linkedin.biz ([172.18.46.133]) with Microsoft Exchange Server HTTP-DAV ;
 Thu,  4 Mar 2010 05:21:22 +0000
User-Agent: Microsoft-Entourage/12.10.0.080409
Date: Wed, 03 Mar 2010 21:21:21 -0800
Subject: Re: rack awareness help
From: Allen Wittenauer <awittenauer@linkedin.com>
To: <general@hadoop.apache.org>
Message-ID: <C7B48351.36DF%awittenauer@linkedin.com>
Thread-Topic: rack awareness help
Thread-Index: Acq7WoZpLtlJdHl8LEG2THkbEpxcIw==
In-Reply-To: <1cbd6f831003031701x5f218995t9a428d7777add9a@mail.gmail.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit

On 3/3/10 5:01 PM, "Mag Gam" <magawake@gmail.com> wrote:

> Thanks Alan! Your presentation is very nice!

Thanks. :)

> "If you don't provide a script for rack awareness, it treats every
> node as if it was its own rack". I am using the default settings and
> the report still says only 1 rack.

Let's take a different approach to convince you. :)

Think about the question:  Is there a difference between all nodes in one
rack vs. every node acting as a lone rack?

The answer is no, there isn't any difference.  In both cases, all copies of
the blocks can go to pretty much any node. When a MR job runs, every node
would either be considered 'off rack' or 'rack-local'.

So there is no difference.


> Do you mind sharing a script with us on how you determine a rack? and
> a sample <configuration> </configuration> syntax?

Michael has already posted his, so I'll skip this one. :)


From general-return-1155-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 04 08:07:40 2010
Return-Path: <general-return-1155-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 41082 invoked from network); 4 Mar 2010 08:07:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Mar 2010 08:07:40 -0000
Received: (qmail 19557 invoked by uid 500); 4 Mar 2010 08:07:29 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19466 invoked by uid 500); 4 Mar 2010 08:07:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19455 invoked by uid 99); 4 Mar 2010 08:07:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Mar 2010 08:07:29 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ctubbsii@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Mar 2010 08:07:22 +0000
Received: by wwb29 with SMTP id 29so1153006wwb.35
        for <general@hadoop.apache.org>; Thu, 04 Mar 2010 00:07:01 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=P0RPT5Gk/QA96ca4c2YKBYdYdDaELRGzu+IseQ5WFOw=;
        b=fWP5M8MpyE8p0EP8t/kSqC4+t2UVsUJmN3pHuHmoRrtM+8UNC6/f3Xs3/BSsffCbQf
         Yvf5hE2qTxIPk1FA68gTiWyeHnPzQcpP9Bo6eP9Fun/kuHq+s2k9wwPypl+z9t1livSv
         f+P95OtCrrH+SM0/jSUgyC/Us1FH8Pbw4M/oI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=oa80oPYWgWh+jNW75lwVu1UWX/AigtRAAcjyI6Q5uHb3FaETLPnm7j/AbkY1/p6z/U
         WDz2kHU20PbLOHD25TP5oNU7TyotWkg+1pgADE6aDe5J6odp1Fum1rBHjU9lrVGkShxg
         yFo6hTXgQNtGB3JUYfJmlzsW3iGttFI5pAyL4=
MIME-Version: 1.0
Received: by 10.216.88.79 with SMTP id z57mr1549817wee.22.1267690020642; Thu, 
	04 Mar 2010 00:07:00 -0800 (PST)
In-Reply-To: <1cbd6f831003031953p7462a54bq1cd5194ba00bf22a@mail.gmail.com>
References: <1cbd6f831003031953p7462a54bq1cd5194ba00bf22a@mail.gmail.com>
Date: Thu, 4 Mar 2010 03:07:00 -0500
Message-ID: <9db00c201003040007j4971215w4f0e619cf36b1ebb@mail.gmail.com>
Subject: Re: setting a proper threshold
From: Christopher Tubbs <ctubbsii@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Can't you just set quotas for the user you run HDFS as?

On Wed, Mar 3, 2010 at 10:53 PM, Mag Gam <magawake@gmail.com> wrote:
> I plan to setup HDFS on 20 servers. I plan on using /fs for my data. I
> want /fs to have atleast 20% free therefore I don't want this
> filesystem to be filled up with hdfs data. Is there a way to restrict
> hadoop so it does not write here if the filesystem is =A0over 80%?
>
> TIA
>

From general-return-1156-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 04 12:03:45 2010
Return-Path: <general-return-1156-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 8440 invoked from network); 4 Mar 2010 12:03:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Mar 2010 12:03:44 -0000
Received: (qmail 3300 invoked by uid 500); 4 Mar 2010 12:03:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3149 invoked by uid 500); 4 Mar 2010 12:03:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3141 invoked by uid 99); 4 Mar 2010 12:03:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Mar 2010 12:03:33 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of magawake@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Mar 2010 12:03:27 +0000
Received: by vws8 with SMTP id 8so882915vws.35
        for <general@hadoop.apache.org>; Thu, 04 Mar 2010 04:03:06 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=5wXhoAnqi2D7Xj44UECp7oF+99aEbOCuJutBvbJ+4o0=;
        b=p2P2JAFocuL0W846Ggv+3WLG8WsqcHkp+BSDCPzLCAafLw4kGkVTAIJh2THSrrr3rM
         lAAFgA7Gyyq50H2qJ1xSRg5sYRWFYxc2Ia8IFDlQNXJbY7C5dpN621j5Q7CGNq8cXZuU
         czYZdgZybFnIaU44c9OkwRPRmLQ4x0oMvljJU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=t2j9bHKJJM5Sb5Rvj6a+URTSktUrhSQAbszoxPNSJKTzEtv2+4pYWWn7hru7frZorS
         DqIK5hgkd7t39PbG5BD0G+QkSAbCDYE1HtHjAynfH2As7keaIIITV9A63J8B8pkqG5Sv
         tceFLj4TK1ERAEK9jXBSH/EIaLqhRzuBZqSTI=
MIME-Version: 1.0
Received: by 10.220.122.169 with SMTP id l41mr888487vcr.175.1267704186114; 
	Thu, 04 Mar 2010 04:03:06 -0800 (PST)
In-Reply-To: <9db00c201003040007j4971215w4f0e619cf36b1ebb@mail.gmail.com>
References: <1cbd6f831003031953p7462a54bq1cd5194ba00bf22a@mail.gmail.com>
	 <9db00c201003040007j4971215w4f0e619cf36b1ebb@mail.gmail.com>
Date: Thu, 4 Mar 2010 07:03:06 -0500
Message-ID: <1cbd6f831003040403k6f3320bk41eafd12fa6f6802@mail.gmail.com>
Subject: Re: setting a proper threshold
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

If you mean filesystem quotas I don't have root access

On Thu, Mar 4, 2010 at 3:07 AM, Christopher Tubbs <ctubbsii@gmail.com> wrot=
e:
> Can't you just set quotas for the user you run HDFS as?
>
> On Wed, Mar 3, 2010 at 10:53 PM, Mag Gam <magawake@gmail.com> wrote:
>> I plan to setup HDFS on 20 servers. I plan on using /fs for my data. I
>> want /fs to have atleast 20% free therefore I don't want this
>> filesystem to be filled up with hdfs data. Is there a way to restrict
>> hadoop so it does not write here if the filesystem is =C2=A0over 80%?
>>
>> TIA
>>
>

From general-return-1157-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 04 14:21:57 2010
Return-Path: <general-return-1157-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 54796 invoked from network); 4 Mar 2010 14:21:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Mar 2010 14:21:57 -0000
Received: (qmail 62756 invoked by uid 500); 4 Mar 2010 14:21:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 62719 invoked by uid 500); 4 Mar 2010 14:21:46 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 62711 invoked by uid 99); 4 Mar 2010 14:21:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Mar 2010 14:21:45 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Mar 2010 14:21:35 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 4A621B7F15
	for <general@hadoop.apache.org>; Thu,  4 Mar 2010 14:21:14 +0000 (GMT)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id E361Bf3lYEzq for <general@hadoop.apache.org>;
	Thu,  4 Mar 2010 14:21:12 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id B11CFB7F14
	for <general@hadoop.apache.org>; Thu,  4 Mar 2010 14:21:12 +0000 (GMT)
MailScanner-NULL-Check: 1268317259.02421@Lib4ERslxFTzZdEirHewKQ
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o24EKvDT026524
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Thu, 4 Mar 2010 14:20:58 GMT
Message-ID: <4B8FC1CC.2090108@apache.org>
Date: Thu, 04 Mar 2010 14:21:00 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: rack awareness help
References: <C7B48351.36DF%awittenauer@linkedin.com>
In-Reply-To: <C7B48351.36DF%awittenauer@linkedin.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o24EKvDT026524
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org
X-Virus-Checked: Checked by ClamAV on apache.org

Allen Wittenauer wrote:
> On 3/3/10 5:01 PM, "Mag Gam" <magawake@gmail.com> wrote:
> 
>> Thanks Alan! Your presentation is very nice!
> 
> Thanks. :)
> 
>> "If you don't provide a script for rack awareness, it treats every
>> node as if it was its own rack". I am using the default settings and
>> the report still says only 1 rack.
> 
> Let's take a different approach to convince you. :)
> 
> Think about the question:  Is there a difference between all nodes in one
> rack vs. every node acting as a lone rack?
> 
> The answer is no, there isn't any difference.  In both cases, all copies of
> the blocks can go to pretty much any node. When a MR job runs, every node
> would either be considered 'off rack' or 'rack-local'.
> 
> So there is no difference.
> 
> 
>> Do you mind sharing a script with us on how you determine a rack? and
>> a sample <configuration> </configuration> syntax?
> 
> Michael has already posted his, so I'll skip this one. :)
> 

Think Mag probably wanted a shell script.

Mag, give your machines IPv4 addresses that map to rack number. 10.1.1.* 
for rack one, 10.1.2.* for rack 2, etc. Then just filter out the IP 
address by the top bytes, returning "10.1.1" for everything in rack one, 
"10.1.2" for rack 2; Hadoop will be happy

From general-return-1158-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 04 15:25:46 2010
Return-Path: <general-return-1158-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 76743 invoked from network); 4 Mar 2010 15:25:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Mar 2010 15:25:46 -0000
Received: (qmail 4468 invoked by uid 500); 4 Mar 2010 15:25:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4428 invoked by uid 500); 4 Mar 2010 15:25:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4420 invoked by uid 99); 4 Mar 2010 15:25:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Mar 2010 15:25:34 +0000
X-ASF-Spam-Status: No, hits=-1.8 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of swatt@us.ibm.com designates 32.97.110.149 as permitted sender)
Received: from [32.97.110.149] (HELO e31.co.us.ibm.com) (32.97.110.149)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Mar 2010 15:25:23 +0000
Received: from d03relay05.boulder.ibm.com (d03relay05.boulder.ibm.com [9.17.195.107])
	by e31.co.us.ibm.com (8.14.3/8.13.1) with ESMTP id o24FGEGR021108
	for <general@hadoop.apache.org>; Thu, 4 Mar 2010 08:16:14 -0700
Received: from d03av03.boulder.ibm.com (d03av03.boulder.ibm.com [9.17.195.169])
	by d03relay05.boulder.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id o24FObWo106140
	for <general@hadoop.apache.org>; Thu, 4 Mar 2010 08:24:42 -0700
Received: from d03av03.boulder.ibm.com (loopback [127.0.0.1])
	by d03av03.boulder.ibm.com (8.14.3/8.13.1/NCO v10.0 AVout) with ESMTP id o248Oack031493
	for <general@hadoop.apache.org>; Thu, 4 Mar 2010 01:24:36 -0700
Received: from d03nm123.boulder.ibm.com (d03nm123.boulder.ibm.com [9.17.195.149])
	by d03av03.boulder.ibm.com (8.14.3/8.13.1/NCO v10.0 AVin) with ESMTP id o248OafZ031490
	for <general@hadoop.apache.org>; Thu, 4 Mar 2010 01:24:36 -0700
In-Reply-To: <4B8FC1CC.2090108@apache.org>
References: <C7B48351.36DF%awittenauer@linkedin.com> <4B8FC1CC.2090108@apache.org>
To: general@hadoop.apache.org
MIME-Version: 1.0
Subject: Hadoop 0.20.2 - New Mockito jar in lib ? - And discovering the reason for
 new jars in general
X-KeepSent: 3AFEBF5F:5933463E-862576DC:0053C094;
 type=4; name=$KeepSent
X-Mailer: Lotus Notes Release 8.5 December 05, 2008
Message-ID: <OF3AFEBF5F.5933463E-ON862576DC.0053C094-862576DC.0054A5B0@us.ibm.com>
From: Stephen Watt <swatt@us.ibm.com>
Date: Thu, 4 Mar 2010 09:24:35 -0600
X-MIMETrack: Serialize by Router on D03NM123/03/M/IBM(Release 8.5.1HF41 | October 22, 2009) at
 03/04/2010 08:24:36,
	Serialize complete at 03/04/2010 08:24:36
Content-Type: multipart/alternative; boundary="=_alternative 0054A5AF862576DC_="
X-Virus-Checked: Checked by ClamAV on apache.org

--=_alternative 0054A5AF862576DC_=
Content-Type: text/plain; charset="US-ASCII"

Hi Folks

I notice in 0.20.2 we now have a new jar in the lib called 
mockito-all-1.8.0 (mockito.org states it is a Java Test mock-up 
framework). In the interest of teaching this man to fish, in cases such as 
this, how can I determine the reason for the addition of this jar to the 
release ? I've taken a look at the JIRA tickets in the bundled 
CHANGES.txt, but its not immediately obvious without opening up every 
single patch within every single mentioned ticket. Is there another, more 
efficient means ?

Kind regards
Steve Watt
--=_alternative 0054A5AF862576DC_=--

From general-return-1159-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 04 16:02:17 2010
Return-Path: <general-return-1159-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 89237 invoked from network); 4 Mar 2010 16:02:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Mar 2010 16:02:17 -0000
Received: (qmail 86549 invoked by uid 500); 4 Mar 2010 16:02:06 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 86518 invoked by uid 500); 4 Mar 2010 16:02:06 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 86509 invoked by uid 99); 4 Mar 2010 16:02:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Mar 2010 16:02:06 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [209.85.222.189] (HELO mail-pz0-f189.google.com) (209.85.222.189)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Mar 2010 16:01:56 +0000
Received: by pzk27 with SMTP id 27so2652249pzk.2
        for <general@hadoop.apache.org>; Thu, 04 Mar 2010 08:01:35 -0800 (PST)
MIME-Version: 1.0
Received: by 10.141.14.6 with SMTP id r6mr5346275rvi.280.1267718495092; Thu, 
	04 Mar 2010 08:01:35 -0800 (PST)
In-Reply-To: <OF3AFEBF5F.5933463E-ON862576DC.0053C094-862576DC.0054A5B0@us.ibm.com>
References: <C7B48351.36DF%awittenauer@linkedin.com> <4B8FC1CC.2090108@apache.org> 
	<OF3AFEBF5F.5933463E-ON862576DC.0053C094-862576DC.0054A5B0@us.ibm.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Thu, 4 Mar 2010 08:01:15 -0800
Message-ID: <45f85f71003040801o2abb50cbt76027cc6ab9a410e@mail.gmail.com>
Subject: Re: Hadoop 0.20.2 - New Mockito jar in lib ? - And discovering the 
	reason for new jars in general
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd112529341e60480fbb6a0
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd112529341e60480fbb6a0
Content-Type: text/plain; charset=ISO-8859-1

Hi Stephen,

You can consult the svn logs for ivy.xml. The dependency on mockito was
backported with HDFS-927 to enable those tests.

-Todd

On Thu, Mar 4, 2010 at 7:24 AM, Stephen Watt <swatt@us.ibm.com> wrote:

> Hi Folks
>
> I notice in 0.20.2 we now have a new jar in the lib called
> mockito-all-1.8.0 (mockito.org states it is a Java Test mock-up
> framework). In the interest of teaching this man to fish, in cases such as
> this, how can I determine the reason for the addition of this jar to the
> release ? I've taken a look at the JIRA tickets in the bundled
> CHANGES.txt, but its not immediately obvious without opening up every
> single patch within every single mentioned ticket. Is there another, more
> efficient means ?
>
> Kind regards
> Steve Watt

--000e0cd112529341e60480fbb6a0--

From general-return-1160-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 04 17:42:13 2010
Return-Path: <general-return-1160-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30104 invoked from network); 4 Mar 2010 17:42:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Mar 2010 17:42:13 -0000
Received: (qmail 98702 invoked by uid 500); 4 Mar 2010 17:42:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98670 invoked by uid 500); 4 Mar 2010 17:42:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98662 invoked by uid 99); 4 Mar 2010 17:42:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Mar 2010 17:42:01 +0000
X-ASF-Spam-Status: No, hits=-4.0 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Mar 2010 17:41:52 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:User-Agent:Date:Subject:From:To:
   Message-ID:Thread-Topic:Thread-Index:In-Reply-To:
   Mime-version:Content-type:Content-transfer-encoding;
  b=L8MHzaJINiVTF6Bnkfh/jcfv+ENNYUFH1wO4xmGqZa52zNrjRrJ2qcOB
   MsfjCqdWyQAW8ptCUXJBp5IVQTzm/VcSYRhv8MrpXKP4BuJp9lVkIweRr
   ERZypvsqV+8Cqxs;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1267724512; x=1299260512;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20Hadoop=200.20.2=20-=20New=20Mockito=20j
   ar=20in=20lib=20?=20-=20And=20discovering=20the=0D=0A=20r
   eason=20for=20new=20jars=20in=20general|Date:=20Thu,=2004
   =20Mar=202010=2009:41:33=20-0800|Message-ID:=20<C7B530CD.
   370F%awittenauer@linkedin.com>|To:=20<general@hadoop.apac
   he.org>|Mime-version:=201.0|Content-transfer-encoding:=20
   7bit|In-Reply-To:=20<OF3AFEBF5F.5933463E-ON862576DC.0053C
   094-862576DC.0054A5B0@us.ibm.com>;
  bh=klpKd7+Qssf4aAaP42Dp2b5fFfxlTFS6tNt7yOphQVQ=;
  b=JkZmJ12aGSfzlIUCjbJRjPYIcWPaLN7o/6UgWYdRnR0l3PKNFJoEw7P2
   u0ecw5sBLE5U76nUGIjw5wIjsYpi4/OdPmlCcMxWoF2p5R44FTXFKaMIH
   PjaTR+PM7ogv3fO;
X-IronPort-AV: E=Sophos;i="4.49,582,1262592000"; 
   d="scan'208";a="11536346"
Received: from 172.18.36.184 ([172.18.36.184]) by CORP-MAIL.linkedin.biz ([172.18.46.135]) via Exchange Front-End Server mail-access.linkedin.biz ([172.18.46.133]) with Microsoft Exchange Server HTTP-DAV ;
 Thu,  4 Mar 2010 17:41:32 +0000
User-Agent: Microsoft-Entourage/12.10.0.080409
Date: Thu, 04 Mar 2010 09:41:33 -0800
Subject: Re: Hadoop 0.20.2 - New Mockito jar in lib ? - And discovering the
 reason for new jars in general
From: Allen Wittenauer <awittenauer@linkedin.com>
To: <general@hadoop.apache.org>
Message-ID: <C7B530CD.370F%awittenauer@linkedin.com>
Thread-Topic: Hadoop 0.20.2 - New Mockito jar in lib ? - And discovering the
 reason for new jars in general
Thread-Index: Acq7we4HHtXAEzstq0aFR9xOjocAEQ==
In-Reply-To: <OF3AFEBF5F.5933463E-ON862576DC.0053C094-862576DC.0054A5B0@us.ibm.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit




On 3/4/10 7:24 AM, "Stephen Watt" <swatt@us.ibm.com> wrote:
> I notice in 0.20.2 we now have a new jar in the lib called
> mockito-all-1.8.0 (mockito.org states it is a Java Test mock-up
> framework). In the interest of teaching this man to fish, in cases such as
> this, how can I determine the reason for the addition of this jar to the
> release ? I've taken a look at the JIRA tickets in the bundled
> CHANGES.txt, but its not immediately obvious without opening up every
> single patch within every single mentioned ticket. Is there another, more
> efficient means ?

You'll eventually get used to things that are in the release notes being
incomplete. :/

That said, they are still worlds ahead where we were in the early days when
we didn't have any at all. :)


From general-return-1161-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 04 23:39:52 2010
Return-Path: <general-return-1161-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38099 invoked from network); 4 Mar 2010 23:39:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Mar 2010 23:39:52 -0000
Received: (qmail 3454 invoked by uid 500); 4 Mar 2010 23:39:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3394 invoked by uid 500); 4 Mar 2010 23:39:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3386 invoked by uid 99); 4 Mar 2010 23:39:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Mar 2010 23:39:39 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of magawake@gmail.com designates 209.85.223.192 as permitted sender)
Received: from [209.85.223.192] (HELO mail-iw0-f192.google.com) (209.85.223.192)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Mar 2010 23:39:31 +0000
Received: by iwn30 with SMTP id 30so2297124iwn.32
        for <general@hadoop.apache.org>; Thu, 04 Mar 2010 15:39:10 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=6HXzDADlXNcC3/xVzRviNkJb6FrdJQFGyZFXjeOwf3o=;
        b=Hb/y6TJrBI0AXQTq1X7k7G9NEcCnjvl9KI8iP53VyxDnY09/Um562/9MnHsWzBnkzS
         usNv+hTFxb3pmU2yvGyQ5ri37wVPHXmNtyQWEPSePShvTbZD60T68Dn2LV74tjGp7CdT
         cuf5krozO6OUC5LvocomEMq68c3uEYIFGD4v0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=gVvJcZV4pnYuH9BeejsOExFjymFiYKerSTkg3OlaldyBa2/w0mydBKAzAEHcWIRtXP
         zGSp8HVZhZZdBw1JXkNfpfaweUHO+DWl/FtVC2eBA8vF7OYMJ7C9+YNcuuW7eZDaz4QO
         L/rVDcHDVwwhE51thNo4jOxNZWEYFqUbxd4pY=
MIME-Version: 1.0
Received: by 10.231.170.136 with SMTP id d8mr946711ibz.17.1267745950683; Thu, 
	04 Mar 2010 15:39:10 -0800 (PST)
In-Reply-To: <4B8FC1CC.2090108@apache.org>
References: <C7B48351.36DF%awittenauer@linkedin.com>
	 <4B8FC1CC.2090108@apache.org>
Date: Thu, 4 Mar 2010 18:39:10 -0500
Message-ID: <1cbd6f831003041539r75b360ecrb8e604d4dce1350f@mail.gmail.com>
Subject: Re: rack awareness help
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks everyone for explaining this to me instead of giving me RTFM!

I will play around with it and see how far I get.



On Thu, Mar 4, 2010 at 9:21 AM, Steve Loughran <stevel@apache.org> wrote:
> Allen Wittenauer wrote:
>>
>> On 3/3/10 5:01 PM, "Mag Gam" <magawake@gmail.com> wrote:
>>
>>> Thanks Alan! Your presentation is very nice!
>>
>> Thanks. :)
>>
>>> "If you don't provide a script for rack awareness, it treats every
>>> node as if it was its own rack". I am using the default settings and
>>> the report still says only 1 rack.
>>
>> Let's take a different approach to convince you. :)
>>
>> Think about the question: =C2=A0Is there a difference between all nodes =
in one
>> rack vs. every node acting as a lone rack?
>>
>> The answer is no, there isn't any difference. =C2=A0In both cases, all c=
opies
>> of
>> the blocks can go to pretty much any node. When a MR job runs, every nod=
e
>> would either be considered 'off rack' or 'rack-local'.
>>
>> So there is no difference.
>>
>>
>>> Do you mind sharing a script with us on how you determine a rack? and
>>> a sample <configuration> </configuration> syntax?
>>
>> Michael has already posted his, so I'll skip this one. :)
>>
>
> Think Mag probably wanted a shell script.
>
> Mag, give your machines IPv4 addresses that map to rack number. 10.1.1.* =
for
> rack one, 10.1.2.* for rack 2, etc. Then just filter out the IP address b=
y
> the top bytes, returning "10.1.1" for everything in rack one, "10.1.2" fo=
r
> rack 2; Hadoop will be happy
>

From general-return-1162-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 05 02:30:56 2010
Return-Path: <general-return-1162-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91387 invoked from network); 5 Mar 2010 02:30:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Mar 2010 02:30:56 -0000
Received: (qmail 60455 invoked by uid 500); 5 Mar 2010 02:30:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60416 invoked by uid 500); 5 Mar 2010 02:30:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 60408 invoked by uid 99); 5 Mar 2010 02:30:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Mar 2010 02:30:43 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.221.204] (HELO mail-qy0-f204.google.com) (209.85.221.204)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Mar 2010 02:30:35 +0000
Received: by qyk42 with SMTP id 42so730953qyk.29
        for <general@hadoop.apache.org>; Thu, 04 Mar 2010 18:30:13 -0800 (PST)
MIME-Version: 1.0
Received: by 10.229.230.84 with SMTP id jl20mr39438qcb.88.1267756213488; Thu, 
	04 Mar 2010 18:30:13 -0800 (PST)
In-Reply-To: <1cbd6f831003041539r75b360ecrb8e604d4dce1350f@mail.gmail.com>
References: <C7B48351.36DF%awittenauer@linkedin.com>
	 <4B8FC1CC.2090108@apache.org>
	 <1cbd6f831003041539r75b360ecrb8e604d4dce1350f@mail.gmail.com>
Date: Thu, 4 Mar 2010 18:30:13 -0800
Message-ID: <c9118d4c1003041830h47961fc4p323c2ebdd0df2ffc@mail.gmail.com>
Subject: Re: rack awareness help
From: Jeff Hammerbacher <hammer@cloudera.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163630ff21c44edc0481047e4e
X-Virus-Checked: Checked by ClamAV on apache.org

--00163630ff21c44edc0481047e4e
Content-Type: text/plain; charset=ISO-8859-1

Hey Mag,

Glad you have solved the problem. I've created a JIRA ticket to improve the
existing documentation: https://issues.apache.org/jira/browse/HADOOP-6616.
If you have some time, it would be useful to hear what could be added to the
existing documentation that would have helped you figure this out sooner.

Thanks,
Jeff

On Thu, Mar 4, 2010 at 3:39 PM, Mag Gam <magawake@gmail.com> wrote:

> Thanks everyone for explaining this to me instead of giving me RTFM!
>
> I will play around with it and see how far I get.
>
>
>
> On Thu, Mar 4, 2010 at 9:21 AM, Steve Loughran <stevel@apache.org> wrote:
> > Allen Wittenauer wrote:
> >>
> >> On 3/3/10 5:01 PM, "Mag Gam" <magawake@gmail.com> wrote:
> >>
> >>> Thanks Alan! Your presentation is very nice!
> >>
> >> Thanks. :)
> >>
> >>> "If you don't provide a script for rack awareness, it treats every
> >>> node as if it was its own rack". I am using the default settings and
> >>> the report still says only 1 rack.
> >>
> >> Let's take a different approach to convince you. :)
> >>
> >> Think about the question:  Is there a difference between all nodes in
> one
> >> rack vs. every node acting as a lone rack?
> >>
> >> The answer is no, there isn't any difference.  In both cases, all copies
> >> of
> >> the blocks can go to pretty much any node. When a MR job runs, every
> node
> >> would either be considered 'off rack' or 'rack-local'.
> >>
> >> So there is no difference.
> >>
> >>
> >>> Do you mind sharing a script with us on how you determine a rack? and
> >>> a sample <configuration> </configuration> syntax?
> >>
> >> Michael has already posted his, so I'll skip this one. :)
> >>
> >
> > Think Mag probably wanted a shell script.
> >
> > Mag, give your machines IPv4 addresses that map to rack number. 10.1.1.*
> for
> > rack one, 10.1.2.* for rack 2, etc. Then just filter out the IP address
> by
> > the top bytes, returning "10.1.1" for everything in rack one, "10.1.2"
> for
> > rack 2; Hadoop will be happy
> >
>

--00163630ff21c44edc0481047e4e--

From general-return-1163-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 05 19:34:03 2010
Return-Path: <general-return-1163-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 44536 invoked from network); 5 Mar 2010 19:34:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Mar 2010 19:34:03 -0000
Received: (qmail 94523 invoked by uid 500); 5 Mar 2010 19:33:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94466 invoked by uid 500); 5 Mar 2010 19:33:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94458 invoked by uid 99); 5 Mar 2010 19:33:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Mar 2010 19:33:46 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.191.91.165] (HELO web37903.mail.mud.yahoo.com) (209.191.91.165)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 05 Mar 2010 19:33:45 +0000
Received: (qmail 77651 invoked by uid 60001); 5 Mar 2010 19:33:23 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1267817603; bh=x91+DKEGE6nuNFUQnqAhvW5Sp9arm0Mc7M95sMHoLPo=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=drRu3ylIRFiQIaewpo3oDKsF8HAj6f4/3zVW5FzueKYz9r8sOLqfmjmevXUa3PBDRfqVJhrLlK1M412JGDgja0L/NqjzK7srN+8oPdTYt76YQXPZHngwlcXO8JzMzhRjJuEXkrgKycwm1+O0h40oyFV0TscqNPbpOrdy/9oXR4c=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=ZF9q7O8JyX0GUmGDIIvt2jbE1atHJ8J5PfLNxNsvwNYruNyTRG+vnFtHB4j42F7ucn+Kvd/G41Q3WrLmTaePisa6lVMXvPZm/8gD292Q8EpnAN6c0am2gu3ZPyniZi5SCBdbD9Uxa5d/tDeRXhzjRiLrKggMt1MDRkRczxgLvbA=;
Message-ID: <555270.77646.qm@web37903.mail.mud.yahoo.com>
X-YMail-OSG: QfUuGzQVM1k6k3mgAMPYERJ211zQwjHDqjtxc2ysjMffwmQ
 KXxs79vppdsGbfZ0Ba16ZrsHm7JbqWN1gkt4yUJ.3TktWr27Ml59LZThAeAN
 ..DRHD0qI8qny7UelJ7CFaEj_.idmSxT9LEzoXWfNI90zRAUTcASdAfJdkJj
 xELAoRLAoHFbOS1wPthjpl6cRLMfH.tW.HSey3h3LlRenYL3TmFWatVLJOIJ
 PuBFAh_WpiFWo_hwHbFyNSKsmoXTS6HwYSIe94EEsFKzwki6oD6qSDLc6tvD
 fmnRifdmA3BvDzJ79ybHs.AlV4cRVCJdGTu6HAxvcqGRx0FTIfeSqdQcLMk1
 VReWZSppYEYeQV7i6xPadJnnD4pcX
Received: from [63.147.59.14] by web37903.mail.mud.yahoo.com via HTTP; Fri, 05 Mar 2010 11:33:23 PST
X-Mailer: YahooMailClassic/9.2.12 YahooMailWebService/0.8.100.260964
Date: Fri, 5 Mar 2010 11:33:23 -0800 (PST)
From: Gary Yang <garyyang6@yahoo.com>
Subject: How to download the source code of hadoop-0.20.2?
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii

Hi,

I would like to download the source code of hadoop-0.20.2. I am at http://mirrors.devlib.org/apache/hadoop/core/hadoop-0.20.2/hadoop-0.20.2. It shows me the "src" directory. However, I am not able to download the source code there after I clicked the src. Can someone direct me where I can get the entire source code of hadoop-0.20.2?


Thanks,


Gary



      

From general-return-1164-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 05 19:38:03 2010
Return-Path: <general-return-1164-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45598 invoked from network); 5 Mar 2010 19:38:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Mar 2010 19:38:03 -0000
Received: (qmail 2865 invoked by uid 500); 5 Mar 2010 19:37:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2831 invoked by uid 500); 5 Mar 2010 19:37:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2823 invoked by uid 99); 5 Mar 2010 19:37:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Mar 2010 19:37:47 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Mar 2010 19:37:41 +0000
Received: by pwi6 with SMTP id 6so2397497pwi.35
        for <general@hadoop.apache.org>; Fri, 05 Mar 2010 11:37:20 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=UscuIxMNFVSt+tCbOHJJDfD7DiEbeKIR78al+3XSSZY=;
        b=pEohmIpjWlAUANPpJK12KHBaaIDd6MV/NVqv+nEjutyppAJZH9D4cZeVkyyIcwrBaC
         sjX/ae8jG9juJ68Z+4VtHFaW0n6LoqNIHRza7pXjjHe9dfAYuOBG4cH+8x12Dc00QVq5
         ko0dxTf4lhQ8A9le9Ey4LzDPZ17sZwJcDfRnc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=mTLcPflbdAc5k+mHJcji2Q/8qfZ7L8pIRWseGDcMSxLTnKP31MWdm0RixJCoPcqnMx
         oBy5LCSGdPi7p9R5ehmflvRH42mcIZesO4Qh/nNcwPoH1Umfy7zP9ohdqb/5yc5m0dO0
         DSRkECpUbrU2WCwbiJUiSL0c5H+9CAS82Hxk8=
MIME-Version: 1.0
Received: by 10.142.122.6 with SMTP id u6mr940839wfc.8.1267817840433; Fri, 05 
	Mar 2010 11:37:20 -0800 (PST)
In-Reply-To: <555270.77646.qm@web37903.mail.mud.yahoo.com>
References: <555270.77646.qm@web37903.mail.mud.yahoo.com>
Date: Fri, 5 Mar 2010 11:37:20 -0800
Message-ID: <17e273101003051137q45aac6d7hc0df8284910da205@mail.gmail.com>
Subject: Re: How to download the source code of hadoop-0.20.2?
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636e0ba0304d96a048112d824
X-Virus-Checked: Checked by ClamAV on apache.org

--001636e0ba0304d96a048112d824
Content-Type: text/plain; charset=ISO-8859-1

Download hadoop-0.20.2.tar.gz and expand it.

On Fri, Mar 5, 2010 at 11:33 AM, Gary Yang <garyyang6@yahoo.com> wrote:

> Hi,
>
> I would like to download the source code of hadoop-0.20.2. I am at
> http://mirrors.devlib.org/apache/hadoop/core/hadoop-0.20.2/hadoop-0.20.2.
> It shows me the "src" directory. However, I am not able to download the
> source code there after I clicked the src. Can someone direct me where I can
> get the entire source code of hadoop-0.20.2?
>
>
> Thanks,
>
>
> Gary
>
>
>
>
>

--001636e0ba0304d96a048112d824--

From general-return-1165-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 05 19:39:03 2010
Return-Path: <general-return-1165-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45774 invoked from network); 5 Mar 2010 19:39:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Mar 2010 19:39:03 -0000
Received: (qmail 4090 invoked by uid 500); 5 Mar 2010 19:38:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4058 invoked by uid 500); 5 Mar 2010 19:38:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3967 invoked by uid 99); 5 Mar 2010 19:38:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Mar 2010 19:38:47 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Mar 2010 19:38:43 +0000
Received: by gyb13 with SMTP id 13so1792763gyb.35
        for <general@hadoop.apache.org>; Fri, 05 Mar 2010 11:38:22 -0800 (PST)
MIME-Version: 1.0
Received: by 10.101.137.5 with SMTP id p5mr2042198ann.198.1267817902252; Fri, 
	05 Mar 2010 11:38:22 -0800 (PST)
In-Reply-To: <555270.77646.qm@web37903.mail.mud.yahoo.com>
References: <555270.77646.qm@web37903.mail.mud.yahoo.com>
From: Philip Zeyliger <philip@cloudera.com>
Date: Fri, 5 Mar 2010 11:38:02 -0800
Message-ID: <15da8a101003051138k60b1fb50rb148b0f63029b40e@mail.gmail.com>
Subject: Re: How to download the source code of hadoop-0.20.2?
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e680b372b42131048112db90

--0016e680b372b42131048112db90
Content-Type: text/plain; charset=ISO-8859-1

The source code is in the "regular" tarball.

For example,
http://mirror.cloudera.com/apache/hadoop/core/hadoop-0.20.2/hadoop-0.20.2.tar.gz

On Fri, Mar 5, 2010 at 11:33 AM, Gary Yang <garyyang6@yahoo.com> wrote:

> Hi,
>
> I would like to download the source code of hadoop-0.20.2. I am at
> http://mirrors.devlib.org/apache/hadoop/core/hadoop-0.20.2/hadoop-0.20.2.
> It shows me the "src" directory. However, I am not able to download the
> source code there after I clicked the src. Can someone direct me where I can
> get the entire source code of hadoop-0.20.2?
>
>
> Thanks,
>
>
> Gary
>
>
>
>
>

--0016e680b372b42131048112db90--

From general-return-1166-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 05 23:48:38 2010
Return-Path: <general-return-1166-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 9838 invoked from network); 5 Mar 2010 23:48:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Mar 2010 23:48:38 -0000
Received: (qmail 21038 invoked by uid 500); 5 Mar 2010 23:48:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20990 invoked by uid 500); 5 Mar 2010 23:48:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 20982 invoked by uid 99); 5 Mar 2010 23:48:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Mar 2010 23:48:22 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.191.91.170] (HELO web37908.mail.mud.yahoo.com) (209.191.91.170)
    by apache.org (qpsmtpd/0.29) with SMTP; Fri, 05 Mar 2010 23:48:20 +0000
Received: (qmail 78244 invoked by uid 60001); 5 Mar 2010 23:47:58 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1267832878; bh=769AXqYTtvWB4vB8J+czAVJdw8GgBbr394jCsBkyZEY=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=R4zpZVJB++E3dJ/oBM8hhnkZvkVgK0HJLywJxLKa+wVvYKBBEr9E9F/RajlUtRjYptNx8GUzvHCiRnJ0eajtaafRxtSKf9N2Galm8nNOouz3j/ASsfcIs39KXRKshPmyXM+mLUv5RQRIBF1Rmpv2itS+kmG3yIZO6zxz0Dd9V1k=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=GtNR3YaZvVs81sdpWnfk0EJuJ40P/4F/891l7K0EZc/ErxWglR6igMzamow1fZOZ88RhxaTOoUFmbO3KQBaB83PpOb2G9SFPpGBF1DFVROMevCWr9NzegSH8FS8/QDtfQhQYSnQ0QQt40/n0NK/HmGYbbgtGul91i7kMBRqXm8c=;
Message-ID: <622128.78209.qm@web37908.mail.mud.yahoo.com>
X-YMail-OSG: 3iaU97sVM1lMe94dQP7nB0Wds2qB8kUP3uQ21VDuJYygPqY
 uWg1NFAJ_Ntf94SKHEyzpdrtDVOJqdx3YRP_fIr9.wbwbgXUKgTBuYDhsjFu
 A2t07k_rfgPwZSedShtgcSvs2hO2FwtT_6EeKilxzN.utl47r6LUWBvMnCOR
 VYy83Bq9.R7drxTvRNsSwYCVw4.cz8ejARJ0JmyVoVeip2ZLzHp6rPPOUwac
 qvINfLCDRusx3aKZMHgGxCIF_PMzilhHjs.DvUEmFl_j8TpjPqQAI9j3NJH2
 Wc7ogrxcvIGTN7DBK7jmhH5_lANLV4zFalWs6uaZLZ5OPwlUV0kS0Aao-
Received: from [63.147.59.14] by web37908.mail.mud.yahoo.com via HTTP; Fri, 05 Mar 2010 15:47:58 PST
X-Mailer: YahooMailClassic/9.2.12 YahooMailWebService/0.8.100.260964
Date: Fri, 5 Mar 2010 15:47:58 -0800 (PST)
From: Gary Yang <garyyang6@yahoo.com>
Subject: Compilation failed when compile hadoop common release-0.20.2
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii

Hi,

I try to compile hadoop common of the release 0.20.2. Below are the error messages and java and ant versions I am using. Please tell me what I missed.

......

  [javadoc] Standard Doclet version 1.6.0_18
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...

java5.check:

BUILD FAILED
/hadoop_src/common/build.xml:908: 'java5.home' is not defined.  Forrest requires Java 5.  Please pass -Djava5.home=<base of Java 5 distribution> to Ant on the command-line.


echo $JAVA_HOME
/usr/java/latest

which java
/usr/java/latest/bin/java

/usr/java/latest/bin/java -version
java version "1.6.0_18"
Java(TM) SE Runtime Environment (build 1.6.0_18-b07)
Java HotSpot(TM) Server VM (build 16.0-b13, mixed mode)

ant -version
Apache Ant version 1.7.1 compiled on June 27 2008


Thanks,


Gary





      

From general-return-1167-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 06 00:17:57 2010
Return-Path: <general-return-1167-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17163 invoked from network); 6 Mar 2010 00:17:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Mar 2010 00:17:57 -0000
Received: (qmail 42724 invoked by uid 500); 6 Mar 2010 00:17:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 42696 invoked by uid 500); 6 Mar 2010 00:17:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 42688 invoked by uid 99); 6 Mar 2010 00:17:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 06 Mar 2010 00:17:41 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com designates 209.85.216.201 as permitted sender)
Received: from [209.85.216.201] (HELO mail-px0-f201.google.com) (209.85.216.201)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 06 Mar 2010 00:17:40 +0000
Received: by pxi39 with SMTP id 39so1439552pxi.29
        for <general@hadoop.apache.org>; Fri, 05 Mar 2010 16:17:20 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=L+Z/1dyHnvSuaEzCH+I4NGUoTp63OcI5+V81GpOA4ds=;
        b=vpOwddTqh4AW8/9Ra3pjlOtVOgjN/v8sfotT/TVB31i4kedQd6adzKRo0nJVD/twIw
         oqNIb7Ozy7kaXm2s7FroJugUipLoWacEtZfPe6YMM6QMDI0V0k2AwpxJFLlxLPiJyIvP
         OZ++QtBtv6p1sGUqXDMs4JBJBqT8g15YmDvgc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=htlQax7XiOgHkq5d1/YEZ74cAZqsuHrwmK16iETFv03b7aTfkimkG3iDRmBl+Bet7r
         9TTok6j+fsRa6+INmatqPeC8LKCzqxe/JV6X8btlAP3zUElqdj9C3RPCiNwo+iohNp89
         MYPjhMaLbFjWyeZCcDMG62t+WP2nE/u37obdA=
MIME-Version: 1.0
Received: by 10.142.121.2 with SMTP id t2mr1116347wfc.100.1267834640069; Fri, 
	05 Mar 2010 16:17:20 -0800 (PST)
In-Reply-To: <622128.78209.qm@web37908.mail.mud.yahoo.com>
References: <622128.78209.qm@web37908.mail.mud.yahoo.com>
Date: Fri, 5 Mar 2010 16:17:19 -0800
Message-ID: <17e273101003051617j3348d557r2ff23208ac1d8d9f@mail.gmail.com>
Subject: Re: Compilation failed when compile hadoop common release-0.20.2
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636e0ab1a5af4b4048116c1e6

--001636e0ab1a5af4b4048116c1e6
Content-Type: text/plain; charset=ISO-8859-1

Did you first try 'ant jar' from under /hadoop_src ?

On Fri, Mar 5, 2010 at 3:47 PM, Gary Yang <garyyang6@yahoo.com> wrote:

> Hi,
>
> I try to compile hadoop common of the release 0.20.2. Below are the error
> messages and java and ant versions I am using. Please tell me what I missed.
>
> ......
>
>  [javadoc] Standard Doclet version 1.6.0_18
>  [javadoc] Building tree for all the packages and classes...
>  [javadoc] Building index for all the packages and classes...
>  [javadoc] Building index for all classes...
>
> java5.check:
>
> BUILD FAILED
> /hadoop_src/common/build.xml:908: 'java5.home' is not defined.  Forrest
> requires Java 5.  Please pass -Djava5.home=<base of Java 5 distribution> to
> Ant on the command-line.
>
>
> echo $JAVA_HOME
> /usr/java/latest
>
> which java
> /usr/java/latest/bin/java
>
> /usr/java/latest/bin/java -version
> java version "1.6.0_18"
> Java(TM) SE Runtime Environment (build 1.6.0_18-b07)
> Java HotSpot(TM) Server VM (build 16.0-b13, mixed mode)
>
> ant -version
> Apache Ant version 1.7.1 compiled on June 27 2008
>
>
> Thanks,
>
>
> Gary
>
>
>
>
>
>
>

--001636e0ab1a5af4b4048116c1e6--

From general-return-1168-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 06 00:33:11 2010
Return-Path: <general-return-1168-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 20803 invoked from network); 6 Mar 2010 00:33:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Mar 2010 00:33:11 -0000
Received: (qmail 55936 invoked by uid 500); 6 Mar 2010 00:32:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 55905 invoked by uid 500); 6 Mar 2010 00:32:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 55897 invoked by uid 99); 6 Mar 2010 00:32:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 06 Mar 2010 00:32:55 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.62.64] (HELO qmta07.westchester.pa.mail.comcast.net) (76.96.62.64)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 06 Mar 2010 00:32:53 +0000
Received: from omta08.westchester.pa.mail.comcast.net ([76.96.62.12])
	by qmta07.westchester.pa.mail.comcast.net with comcast
	id pU0n1d00F0Fqzac57cYa1X; Sat, 06 Mar 2010 00:32:34 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta08.westchester.pa.mail.comcast.net with comcast
	id pcYQ1d00E2SbwD53UcYTSb; Sat, 06 Mar 2010 00:32:32 +0000
Message-Id: <B8873D51-C95D-4509-A292-6836B3493BF8@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
In-Reply-To: <622128.78209.qm@web37908.mail.mud.yahoo.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Compilation failed when compile hadoop common release-0.20.2
Date: Fri, 5 Mar 2010 16:32:23 -0800
References: <622128.78209.qm@web37908.mail.mud.yahoo.com>
X-Mailer: Apple Mail (2.936)


On Mar 5, 2010, at 3:47 PM, Gary Yang wrote:

> Hi,
>
> I try to compile hadoop common of the release 0.20.2. Below are the  
> error messages and java and ant versions I am using. Please tell me  
> what I missed.

Forrest, which we use to generate the documentation, requires java 5.  
Therefore, run:

ant -Djava5.home=/some/path/to/java5 tar

There are several more you need. For a more complete list, I'd look at  
the how to release page:

http://wiki.apache.org/hadoop/HowToRelease

-- Owen

From general-return-1169-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 06 03:51:10 2010
Return-Path: <general-return-1169-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 56121 invoked from network); 6 Mar 2010 03:51:10 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Mar 2010 03:51:10 -0000
Received: (qmail 52214 invoked by uid 500); 6 Mar 2010 03:50:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51904 invoked by uid 500); 6 Mar 2010 03:50:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51896 invoked by uid 99); 6 Mar 2010 03:50:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 06 Mar 2010 03:50:53 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of zhangyf14@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 06 Mar 2010 03:50:48 +0000
Received: by pwi6 with SMTP id 6so2581988pwi.35
        for <general@hadoop.apache.org>; Fri, 05 Mar 2010 19:50:27 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=j5k6xDay9djzaR/gXzuU2aMVekP12Yy5l+6Qp2IVFQg=;
        b=D7ks5SyUFLUFmzUtJGWhr5yegTTI21uz2NSDgwLPcHfn4AohxFYeZFeQaLXwsV+ft4
         t8NWDB8VgVbeQtNvNSNxh6hhBlgWPlM71tZnmvoUYmF/6OfgONs0zcQUTqJjcio0vPO3
         zJsoem6tfNbRdVUSvZTIJYGD8+87E+a4G+crY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=MiTcq8fYjafAbHrrl3yK8/8qW2EpwgIu5yReLs6TF+7OlqwbUKTEdLUhsT6yqKiqFa
         +pvlD7q9Fe+pctF6xFIlHaod9+vc1FWSJE2VT9fWOoLpIVkchlSCRidpMlG7RAQVJpIG
         avxynLIqcQUtnfjb2UXVFcqg74JtxebQ5yZ9E=
MIME-Version: 1.0
Received: by 10.114.242.20 with SMTP id p20mr1280653wah.135.1267847427023; 
	Fri, 05 Mar 2010 19:50:27 -0800 (PST)
Date: Fri, 5 Mar 2010 22:50:26 -0500
Message-ID: <bd7d2ecd1003051950jc21e4d0pe682b29866dce7b4@mail.gmail.com>
Subject: How to send KV pair to a reduce task on a particular machine?
From: Yanfeng Zhang <zhangyf14@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00163646c9d8846f35048119bb96

--00163646c9d8846f35048119bb96
Content-Type: text/plain; charset=ISO-8859-1

Hi, all

The KV pairs (kv1, kv2, kv3 kv4) out from mapper would be partitioned into R
parts (e.g. R=2) by a partitioner. For example, kv1 and kv2 are in
partition1, while kv3 and kv4 are in partition2, the reducers will get KV
pairs from these two partitions, reducer1 get KV pairs from partition1 and
reducer2 get KV pairs from partition2.

I want to let machine1 get KV pairs from partition1 and machine2 get KV
pairs from partition2. But reducer1 is not always on machine1, reducer2 is
not always on machine2. Is there any way to make sure kv1 and kv2 are sent
to machine1 and kv3, kv4 are sent to machine2?

Thank you in advance!

Sincerely,
-- 
Yanfeng Zhang

--00163646c9d8846f35048119bb96--

From general-return-1170-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 06 07:05:35 2010
Return-Path: <general-return-1170-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58501 invoked from network); 6 Mar 2010 07:05:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Mar 2010 07:05:35 -0000
Received: (qmail 23769 invoked by uid 500); 6 Mar 2010 07:05:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 23454 invoked by uid 500); 6 Mar 2010 07:05:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 23446 invoked by uid 99); 6 Mar 2010 07:05:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 06 Mar 2010 07:05:17 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of magawake@gmail.com designates 209.85.223.182 as permitted sender)
Received: from [209.85.223.182] (HELO mail-iw0-f182.google.com) (209.85.223.182)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 06 Mar 2010 07:05:15 +0000
Received: by iwn12 with SMTP id 12so3242094iwn.21
        for <general@hadoop.apache.org>; Fri, 05 Mar 2010 23:04:55 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=+oJnW3rCiCW89Mv6E+KjhFACR0Qi/xaKGmIGbSnIdRc=;
        b=X3+GcTag6svRpABDI5gBOe3wLEhsr2E+tkv2dSLGdC+hqnnwBvQRH98GdMpDLmF/uK
         9t69OoKwaZkHlJQOwENRSOKqN5uBp10sjJUBMOb9eEhXsxCFzbc7LlxeJWQR27opXAGz
         nErcf3hMvstE38wEBDx7xY9EN9i09fwT7NuUU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=k2Am1wWECGa0ExFmyFQTlTe7OV9JTDqWn4/TYQBLZYnFdz6RnN7tuRHrV9mq3xl8xR
         8FD/ZYV4EgCn4/cKN7mrGYVO4JulvdvonLMFsDziNJsAFYIssKHz9tQKJULas6meR0rv
         9scyT64wRcRydrawMzRY2I7z1zicZHBSDjEKA=
MIME-Version: 1.0
Received: by 10.231.148.83 with SMTP id o19mr652400ibv.39.1267859095277; Fri, 
	05 Mar 2010 23:04:55 -0800 (PST)
Date: Sat, 6 Mar 2010 02:04:55 -0500
Message-ID: <1cbd6f831003052304i51ce9ef8r84228cb2ef1727b3@mail.gmail.com>
Subject: copyFromLocal problem
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8

I just setup a new Hadoop cluster.

When I do
hadoop dfs -copyFromLocal /tmp/zero zero

the file zero is being copied to the current working directory. Its
not going to the DFS. Is there a way to fix this?

tia

From general-return-1171-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 06 07:51:29 2010
Return-Path: <general-return-1171-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 64182 invoked from network); 6 Mar 2010 07:51:29 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Mar 2010 07:51:29 -0000
Received: (qmail 41399 invoked by uid 500); 6 Mar 2010 07:51:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 41060 invoked by uid 500); 6 Mar 2010 07:51:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 41052 invoked by uid 99); 6 Mar 2010 07:51:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 06 Mar 2010 07:51:11 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 06 Mar 2010 07:51:03 +0000
Received: from sp1-ex07cas01.ds.corp.yahoo.com (sp1-ex07cas01.ds.corp.yahoo.com [216.252.116.137])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o267oCFd037036;
	Fri, 5 Mar 2010 23:50:12 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:cc:date:subject:thread-topic:
	thread-index:message-id:in-reply-to:accept-language:
	content-language:x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:
	content-type:mime-version;
	b=zUJTGqltkVTWUUmxMkithRQwB0lqwhBdkTzeQ/hBPinuneUJiH8aCfu5QYZtbkdr
Received: from SP1-EX07VS02.ds.corp.yahoo.com ([216.252.116.135]) by
 sp1-ex07cas01.ds.corp.yahoo.com ([216.252.116.137]) with mapi; Fri, 5 Mar
 2010 23:50:12 -0800
From: Ravi Phulari <rphulari@yahoo-inc.com>
To: iprafulla <iprafulla@yahoo.com>
CC: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Fri, 5 Mar 2010 23:50:11 -0800
Subject: Re: Fault Tolerancy in core hadoop  and HDFS
Thread-Topic: Fault Tolerancy in core hadoop  and HDFS
Thread-Index: Acq8mBvpMRCk4octR9m+FedhCwLVCgAaYoEc
Message-ID: <C7B74933.17C82%rphulari@yahoo-inc.com>
In-Reply-To: <27798051.post@talk.nabble.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C7B7493317C82rphulariyahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C7B7493317C82rphulariyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

HDFS Architecture  talks more about fault tolerance. - http://hadoop.apache=
.org/common/docs/current/hdfs_design.html


Removing core-dev@ & common-dev@ from list.
Please post general questions to general@ or  comman-user@ .

-
Ravi


On 3/5/10 11:13 AM, "iprafulla" <iprafulla@yahoo.com> wrote:



Hi Everyone,

I am researching fault tolerancy aspect of HDFS and hadoop core system.

I went through slides at hadoop  homepage  and was good starting point and
googled papers that were helpful.

I am familiar with the hadoop system and actually looking for the detail
architecture and implementation of hadoop  system so that I can dig in the
topic.
Any suggestions are most welcome .

and of course except in the code :)

Thank u.

prafulla
--
View this message in context: http://old.nabble.com/Fault-Tolerancy-in-core=
-hadoop--and-HDFS-tp27798051p27798051.html
Sent from the Hadoop core-dev mailing list archive at Nabble.com.



Ravi
--


--_000_C7B7493317C82rphulariyahooinccom_--

From general-return-1172-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 08 02:14:24 2010
Return-Path: <general-return-1172-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 62753 invoked from network); 8 Mar 2010 02:14:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Mar 2010 02:14:24 -0000
Received: (qmail 82072 invoked by uid 500); 8 Mar 2010 02:14:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81940 invoked by uid 500); 8 Mar 2010 02:14:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81932 invoked by uid 99); 8 Mar 2010 02:14:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Mar 2010 02:14:00 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of bbsexclusive@gmail.com designates 74.125.83.48 as permitted sender)
Received: from [74.125.83.48] (HELO mail-gw0-f48.google.com) (74.125.83.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Mar 2010 02:13:52 +0000
Received: by gwaa11 with SMTP id a11so2533586gwa.35
        for <general@hadoop.apache.org>; Sun, 07 Mar 2010 18:13:30 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=8EbIZhWVNMWWEXTVkv3qszZwpb+eZKDDTxm/v9/jiFw=;
        b=OjBrHumZYDImXA7GAVtn6DIgPCAQaxljiJ2z4F9YhwOTxIPzp/HH6i331/LNpjIf1U
         P78lmzZWT2NJedTyxzbZl7mRTcpav0Gl5GK/WDd+3VIeS9DM1ejI6pkVH2HNyBrmm9f8
         QxH9p2AyatM1qMYiiaZFEc8fLT0jMo2m/OxQc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=F6QZgHYKmU/Gv90y1QUTxhMCFmDWOL+qbZUld+jpLVmQLsxAPrW5pDXkWVePKHURay
         hujxd0CgHCUbVCXR4DRo/aZ/GFYh7v3IPIpXLLeNPzF/qGQZcYxzkw0d5P++QzfD8oJP
         gwEXIZ/cBcEQUpl1UlnoSjDXWg6ottBkBGR4w=
MIME-Version: 1.0
Received: by 10.101.58.5 with SMTP id l5mr4884223ank.73.1268014410767; Sun, 07 
	Mar 2010 18:13:30 -0800 (PST)
In-Reply-To: <6241f7fc1003070240j1b71b83qba5c21cfc0ce7842@mail.gmail.com>
References: <6241f7fc1003070240j1b71b83qba5c21cfc0ce7842@mail.gmail.com>
Date: Mon, 8 Mar 2010 10:13:30 +0800
Message-ID: <6241f7fc1003071813p612abde4qd7f2c97c44f1ab44@mail.gmail.com>
Subject: I got java crash error when writing SequenceFile
From: forbbs forbbs <bbsexclusive@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/mixed; boundary=001636eee35d861f2f0481409c75
X-Virus-Checked: Checked by ClamAV on apache.org

--001636eee35d861f2f0481409c75
Content-Type: multipart/alternative; boundary=001636eee35d861f220481409c73

--001636eee35d861f220481409c73
Content-Type: text/plain; charset=ISO-8859-1

*I just ran a simple program, and got the error below:*

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGFPE (0x8) at pc=0x00000030bda07927, pid=22891, tid=1076017504
#
# JRE version: 6.0_18-b07
# Java VM: Java HotSpot(TM) 64-Bit Server VM (16.0-b13 mixed mode
linux-amd64 )
# Problematic frame:
# C  [ld-linux-x86-64.so.2+0x7927]
#
# An error report file with more information is saved as:
# /home1/wwp/amy/SDB/hs_err_pid22891.log
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Aborted


*The code is CreatePartition.java:*

import java.io.IOException;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.io.SequenceFile.Writer;
import org.apache.hadoop.io.SequenceFile;

public class CreatePartition
{
 public static void main(String args[]) throws IOException
 {
   Configuration conf = new Configuration();
   FileSystem fs = FileSystem.get(conf);
   SequenceFile.Writer wr = SequenceFile.createWriter(fs, conf, new
Path("_sortPartitioning"), IntWritable.class, IntWritable.class);
   IntWritable k = new IntWritable(123);
   IntWritable v = new IntWritable(123);
   wr.append(k, v);
   wr.close();
 }
}
*
*
*
*
*
---------------  S Y S T E M  ---------------

OS:Red Hat Enterprise Linux AS release 4 (Nahant Update 6)

uname:Linux 2.6.9-67.ELsmp #1 SMP Wed Nov 7 13:56:44 EST 2007 x86_64
libc:glibc 2.3.4 NPTL 2.3.4
rlimit: STACK 10240k, CORE 0k, NPROC 73728, NOFILE 1024, AS infinity
load average:0.08 0.08 0.04

CPU:total 4 (2 cores per cpu, 1 threads per core) family 15 model 33
stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, mmxext, 3dnow, 3dnowext

Memory: 4k page, physical 8162896k(1127340k free), swap 16386292k(16386052k
free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (16.0-b13) for linux-amd64 JRE
(1.6.0_18-b07), built on Dec 17 2009 13:42:22 by "java_re" with gcc 3.2.2
(SuSE Linux)
*
*
*
*Hadoop 0.19.2*
*
*
*The error log /home1/wwp/amy/SDB/hs_err_pid22891.log is attached with this
mail. *

--001636eee35d861f220481409c73
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div class=3D"gmail_quote"><b><font color=3D"#006600">I just ran a simple p=
rogram, and got the error below:</font></b><br><br>#<br># A fatal error has=
 been detected by the Java Runtime Environment:<br>#<br># =A0SIGFPE (0x8) a=
t pc=3D0x00000030bda07927, pid=3D22891, tid=3D1076017504<br>

#<br># JRE version: 6.0_18-b07<br># Java VM: Java HotSpot(TM) 64-Bit Server=
 VM (16.0-b13 mixed mode linux-amd64 )<br># Problematic frame:<br># C =A0[l=
d-linux-x86-64.so.2+0x7927]<br>#<br># An error report file with more inform=
ation is saved as:<br>

# /home1/wwp/amy/SDB/hs_err_pid22891.log<br>#<br># If you would like to sub=
mit a bug report, please visit:<br># =A0 <a href=3D"http://java.sun.com/web=
apps/bugreport/crash.jsp" target=3D"_blank">http://java.sun.com/webapps/bug=
report/crash.jsp</a><br>

# The crash happened outside the Java Virtual Machine in native code.<br># =
See problematic frame for where to report the bug.<br>#<br>Aborted<br><br><=
br><b><font color=3D"#006600">The code is CreatePartition.java:</font></b><=
div>

<br>import java.io.IOException;<br>import org.apache.hadoop.fs.Path;<br>imp=
ort org.apache.hadoop.io.IntWritable;<br>import org.apache.hadoop.fs.FileSy=
stem;<br>import org.apache.hadoop.conf.Configuration;<br>import org.apache.=
hadoop.io.SequenceFile.Writer;<br>

import org.apache.hadoop.io.SequenceFile;<br><br>public class CreatePartiti=
on<br>{<br> =A0public static void main(String args[]) throws IOException<br=
> =A0{<br> =A0 =A0Configuration conf =3D new Configuration();<br> =A0 =A0Fi=
leSystem fs =3D FileSystem.get(conf);<br>

 =A0 =A0SequenceFile.Writer wr =3D SequenceFile.createWriter(fs, conf, new =
Path(&quot;_sortPartitioning&quot;), IntWritable.class, IntWritable.class);=
<br> =A0 =A0IntWritable k =3D new IntWritable(123);<br> =A0 =A0IntWritable =
v =3D new IntWritable(123);<br>

 =A0 =A0wr.append(k, v);<br> =A0 =A0wr.close();<br> =A0}<br>} </div><div><f=
ont color=3D"#666600"><b><font color=3D"#003300"><br></font></b></font></di=
v><div><font color=3D"#666600"><b><font color=3D"#003300"><span style=3D"co=
lor: rgb(0, 102, 0); font-weight: normal;"><b><br>

</b></span></font></b></font></div><div><font color=3D"#666600"><b><font co=
lor=3D"#003300"><span style=3D"color: rgb(0, 102, 0); font-weight: normal;"=
><b><div>
--------------- =A0S Y S T E M =A0---------------</div><div>=A0</div><div><=
span style=3D"font-weight: normal;"><font color=3D"#000000">OS:Red Hat Ente=
rprise Linux AS release 4 (Nahant Update 6)</font></span></div>
<div><span style=3D"font-weight: normal;"><font color=3D"#000000">=A0</font=
></span></div><div><span style=3D"font-weight: normal;"><font color=3D"#000=
000">uname:Linux 2.6.9-67.ELsmp #1 SMP Wed Nov 7 13:56:44 EST 2007 x86_64</=
font></span></div>

<div><span style=3D"font-weight: normal;"><font color=3D"#000000">libc:glib=
c 2.3.4 NPTL 2.3.4</font></span></div><div><span style=3D"font-weight: norm=
al;"><font color=3D"#000000">rlimit: STACK 10240k, CORE 0k, NPROC 73728, NO=
FILE 1024, AS infinity</font></span></div>

<div><span style=3D"font-weight: normal;"><font color=3D"#000000">load aver=
age:0.08 0.08 0.04</font></span></div><div><span style=3D"font-weight: norm=
al;"><font color=3D"#000000">=A0</font></span></div>
<div><span style=3D"font-weight: normal;"><font color=3D"#000000">CPU:total=
 4 (2 cores per cpu, 1 threads per core) family 15 model 33 stepping 2, cmo=
v, cx8, fxsr, mmx, sse, sse2, sse3, mmxext, 3dnow, 3dnowext</font></span></=
div>

<div><span style=3D"font-weight: normal;"><font color=3D"#000000">=A0</font=
></span></div><div><span style=3D"font-weight: normal;"><font color=3D"#000=
000">Memory: 4k page, physical 8162896k(1127340k free), swap 16386292k(1638=
6052k free)</font></span></div>

<div><span style=3D"font-weight: normal;"><font color=3D"#000000">=A0</font=
></span></div><div><span style=3D"font-weight: normal;"><font color=3D"#000=
000">vm_info: Java HotSpot(TM) 64-Bit Server VM (16.0-b13) for linux-amd64 =
JRE (1.6.0_18-b07), built on Dec 17 2009 13:42:22 by &quot;java_re&quot; wi=
th gcc 3.2.2 (SuSE Linux)</font></span></div>

</b></span></font></b></font></div><div><font color=3D"#666600"><b><font co=
lor=3D"#003300"><span style=3D"color: rgb(0, 102, 0); font-weight: normal;"=
><b><br>
</b></span></font></b></font></div><div><font color=3D"#666600"><b><font co=
lor=3D"#003300"><span style=3D"color: rgb(0, 102, 0); font-weight: normal;"=
><b>Hadoop 0.19.2</b></span></font></b></font></div>
<div><font color=3D"#666600"><b><font color=3D"#003300"><span style=3D"colo=
r: rgb(0, 102, 0); font-weight: normal;"><b><br></b></span></font></b></fon=
t></div>
<div><font color=3D"#666600"><b><font color=3D"#003300"><span style=3D"colo=
r: rgb(0, 102, 0); font-weight: normal;"><b>The error log /home1/wwp/amy/SD=
B/hs_err_pid22891.log is=A0attached with this mail.=A0</b></span></font></b=
></font></div>

</div><br>

--001636eee35d861f220481409c73--
--001636eee35d861f2f0481409c75--

From general-return-1173-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 08 06:23:31 2010
Return-Path: <general-return-1173-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 3647 invoked from network); 8 Mar 2010 06:23:31 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Mar 2010 06:23:31 -0000
Received: (qmail 95531 invoked by uid 500); 8 Mar 2010 06:23:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95084 invoked by uid 500); 8 Mar 2010 06:23:05 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95076 invoked by uid 99); 8 Mar 2010 06:23:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Mar 2010 06:23:04 +0000
X-ASF-Spam-Status: No, hits=1.5 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of zshao@facebook.com designates 69.63.179.25 as permitted sender)
Received: from [69.63.179.25] (HELO mailout-sf2p.facebook.com) (69.63.179.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Mar 2010 06:22:57 +0000
Received: from mail.thefacebook.com ([192.168.18.83])
	by pp02.snc1.tfbnw.net (8.14.3/8.14.3) with ESMTP id o286M6VF013996
	(version=TLSv1/SSLv3 cipher=RC4-MD5 bits=128 verify=NOT)
	for <general@hadoop.apache.org>; Sun, 7 Mar 2010 22:22:06 -0800
Received: from SC-MBXC1.TheFacebook.com ([192.168.18.102]) by
 sc-hub06.TheFacebook.com ([192.168.18.83]) with mapi; Sun, 7 Mar 2010
 22:22:36 -0800
From: Zheng Shao <zshao@facebook.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Sun, 7 Mar 2010 22:21:44 -0800
Subject: RE: I got java crash error when writing SequenceFile
Thread-Topic: I got java crash error when writing SequenceFile
Thread-Index: Acq+ZQW/2oYVKYN8RjKa/28+SSzZAQAIpoqG
Message-ID: <CD2E0A273BDAAD409C70BB3F48EE6A126CE9862B32@SC-MBXC1.TheFacebook.com>
References: <6241f7fc1003070240j1b71b83qba5c21cfc0ce7842@mail.gmail.com>,<6241f7fc1003071813p612abde4qd7f2c97c44f1ab44@mail.gmail.com>
In-Reply-To: <6241f7fc1003071813p612abde4qd7f2c97c44f1ab44@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_CD2E0A273BDAAD409C70BB3F48EE6A126CE9862B32SCMBXC1TheFac_"
MIME-Version: 1.0
X-Proofpoint-Virus-Version: vendor=fsecure engine=1.12.8161:2.4.5,1.2.40,4.0.166
 definitions=2010-03-08_05:2010-02-06,2010-03-08,2010-03-08 signatures=0

--_000_CD2E0A273BDAAD409C70BB3F48EE6A126CE9862B32SCMBXC1TheFac_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

We had a similar problem and it turns out that hadoop native library built =
on CentOS 5 cannot run reliably on Fedora Core 4.
You might want to recompile the hadoop native library.

Zheng

________________________________
From: forbbs forbbs [bbsexclusive@gmail.com]
Sent: Sunday, March 07, 2010 6:13 PM
To: general@hadoop.apache.org
Subject: I got java crash error when writing SequenceFile

I just ran a simple program, and got the error below:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGFPE (0x8) at pc=3D0x00000030bda07927, pid=3D22891, tid=3D1076017504
#
# JRE version: 6.0_18-b07
# Java VM: Java HotSpot(TM) 64-Bit Server VM (16.0-b13 mixed mode linux-amd=
64 )
# Problematic frame:
# C  [ld-linux-x86-64.so.2+0x7927]
#
# An error report file with more information is saved as:
# /home1/wwp/amy/SDB/hs_err_pid22891.log
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Aborted


The code is CreatePartition.java:

import java.io.IOException;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.io.SequenceFile.Writer;
import org.apache.hadoop.io.SequenceFile;

public class CreatePartition
{
 public static void main(String args[]) throws IOException
 {
   Configuration conf =3D new Configuration();
   FileSystem fs =3D FileSystem.get(conf);
   SequenceFile.Writer wr =3D SequenceFile.createWriter(fs, conf, new Path(=
"_sortPartitioning"), IntWritable.class, IntWritable.class);
   IntWritable k =3D new IntWritable(123);
   IntWritable v =3D new IntWritable(123);
   wr.append(k, v);
   wr.close();
 }
}


---------------  S Y S T E M  ---------------

OS:Red Hat Enterprise Linux AS release 4 (Nahant Update 6)

uname:Linux 2.6.9-67.ELsmp #1 SMP Wed Nov 7 13:56:44 EST 2007 x86_64
libc:glibc 2.3.4 NPTL 2.3.4
rlimit: STACK 10240k, CORE 0k, NPROC 73728, NOFILE 1024, AS infinity
load average:0.08 0.08 0.04

CPU:total 4 (2 cores per cpu, 1 threads per core) family 15 model 33 steppi=
ng 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, mmxext, 3dnow, 3dnowext

Memory: 4k page, physical 8162896k(1127340k free), swap 16386292k(16386052k=
 free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (16.0-b13) for linux-amd64 JRE (=
1.6.0_18-b07), built on Dec 17 2009 13:42:22 by "java_re" with gcc 3.2.2 (S=
uSE Linux)

Hadoop 0.19.2

The error log /home1/wwp/amy/SDB/hs_err_pid22891.log is attached with this =
mail.


--_000_CD2E0A273BDAAD409C70BB3F48EE6A126CE9862B32SCMBXC1TheFac_--

From general-return-1174-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 08 18:07:29 2010
Return-Path: <general-return-1174-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 16095 invoked from network); 8 Mar 2010 18:07:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Mar 2010 18:07:28 -0000
Received: (qmail 46492 invoked by uid 500); 8 Mar 2010 18:07:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 46421 invoked by uid 500); 8 Mar 2010 18:07:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 46413 invoked by uid 99); 8 Mar 2010 18:07:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Mar 2010 18:07:03 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [209.191.91.169] (HELO web37907.mail.mud.yahoo.com) (209.191.91.169)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 08 Mar 2010 18:06:54 +0000
Received: (qmail 27526 invoked by uid 60001); 8 Mar 2010 18:06:32 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1268071592; bh=H/M/sKujn9gVoR2Oe6sgUmTp9o9pEW511tEFGzSj1uk=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=izzwGJ0YdSGlWdJKmqW1fElNP4tIwL6iWeZpn69PeIAWtmrIHMnOPGo4U1JztmEO8TWGaFAl/68kZtxF1XARCBoRhe43uod0+qWxfhOsDyoS+4XEfpeskEAHLECaKrv1/gvRe+oiVaT7u3EF+5LviNLShUTlxyjaUcvFu3b51mE=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=Y5nBEWdnYvxfBhFgKSomsGoElMt3UpulLOe6sjoU6dkCA4u4GUBIuLM7D44Ak9/7CrZVQGxg2GZ42wohsjAMZ3nVCOaCZbvoQJVh6PimShp8+aQaaGzdsvdOCiqxdiXBidK/V7cpDz3UEHO2O5yPWWFDaPLBWTtpz80rDeDtSt8=;
Message-ID: <673653.27493.qm@web37907.mail.mud.yahoo.com>
X-YMail-OSG: f6UDwPkVM1lyZRKY8pbbH0BsYjDUWAItEZw6O2CTA.zYTKw
 XQS_n_66NuBR6EiitgqbN6sjLtcXmGm.HCSIy3XhphB2byLPxrKC5H5TLMiL
 GlDAPe0KzKOURz6asqh97V4rAz8hCeaR3G27vzwKBnZIlsxkKg.9fRzGea_N
 .yexbjliRqTxS4BnBAe92ehqBGz5TVLC9eoUZ8V_pdCRn7ZglmryJt0T8XU6
 SAYvk2jmsuo3M3xkCsP0uc4xFyFc9mU_h1TyqYLkxZEvworPdnqmfIHt5FIK
 x.ym4ZDz_gtSI8u0DvEDmwBx7sJx9ZerIli2_n2uNCmUf8FUYJmAZ9qaA_Zx
 JbcD2ryULtXCOaipKKqnYsm2AdRALEg--
Received: from [63.147.59.14] by web37907.mail.mud.yahoo.com via HTTP; Mon, 08 Mar 2010 10:06:32 PST
X-Mailer: YahooMailClassic/9.2.12 YahooMailWebService/0.8.100.260964
Date: Mon, 8 Mar 2010 10:06:32 -0800 (PST)
From: Gary Yang <garyyang6@yahoo.com>
Subject: Re: Compilation failed when compile hadoop common release-0.20.2
To: general@hadoop.apache.org
In-Reply-To: <B8873D51-C95D-4509-A292-6836B3493BF8@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Owen,

Thanks for the reply. From the link you provided, I found the build instruction. I do not understand the option, "-Djava5.home=/usr/local/jdk1.5". Does it mean I have to use JDK 1.5? I read somewhere it suggested to use JDK 1.6. 

Also, the very first line is "export JAVA_HOME=/path/to/32bit/jdk
". Does it mean the JAVA_HOME jdk has to be 1.5?  Please let me know.


export JAVA_HOME=/path/to/32bit/jdk
export CFLAGS=-m32
export CXXFLAGS=-m32
ant -Dversion=X.Y.Z -Dcompile.native=true -Dcompile.c++=true -Dlibhdfs=1 -Dlibrecordio=true -Dxercescroot=/usr/local/xerces-c -Declipse.home=/usr/lib/eclipse -Dforrest.home=/usr/local/forrest -Djava5.home=/usr/local/jdk1.5 clean api-report tar test test-c++-libhdfs
export JAVA_HOME=/path/to/64bit/jdk
export CFLAGS=-m64
export CXXFLAGS=-m64
ant -Dversion=X.Y.Z -Dcompile.native=true -Dcompile.c++=true compile-core-native compile-c++ tar


Thanks,


Gary



--- On Fri, 3/5/10, Owen O'Malley <omalley@apache.org> wrote:

> From: Owen O'Malley <omalley@apache.org>
> Subject: Re: Compilation failed when compile hadoop common release-0.20.2
> To: general@hadoop.apache.org
> Date: Friday, March 5, 2010, 4:32 PM
> 
> On Mar 5, 2010, at 3:47 PM, Gary Yang wrote:
> 
> > Hi,
> > 
> > I try to compile hadoop common of the release 0.20.2.
> Below are the error messages and java and ant versions I am
> using. Please tell me what I missed.
> 
> Forrest, which we use to generate the documentation,
> requires java 5. Therefore, run:
> 
> ant -Djava5.home=/some/path/to/java5 tar
> 
> There are several more you need. For a more complete list,
> I'd look at the how to release page:
> 
> http://wiki.apache.org/hadoop/HowToRelease
> 
> -- Owen
> 


      

From general-return-1175-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 08 18:32:20 2010
Return-Path: <general-return-1175-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29429 invoked from network); 8 Mar 2010 18:32:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Mar 2010 18:32:19 -0000
Received: (qmail 77314 invoked by uid 500); 8 Mar 2010 18:31:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 77249 invoked by uid 500); 8 Mar 2010 18:31:54 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 77241 invoked by uid 99); 8 Mar 2010 18:31:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Mar 2010 18:31:54 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Mar 2010 18:31:52 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:User-Agent:Date:Subject:From:To:
   Message-ID:Thread-Topic:Thread-Index:In-Reply-To:
   Mime-version:Content-type:Content-transfer-encoding;
  b=rpH8xmZzta+ryKQ4yvcCyvsTY6duS9wp8c+RHv90wsOCPQI3qGeWLQLh
   dasKJHGcI+dZoqeBZ5WTl6oY+oIvWTAphHWKQzR8BdTpa2SlUNLCcK7h0
   7JKWPjapfzubKYp;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1268073112; x=1299609112;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20Compilation=20failed=20when=20compile
   =20hadoop=20common=20release-0.20.2|Date:=20Mon,=2008=20M
   ar=202010=2010:30:30=20-0800|Message-ID:=20<C7BA8246.386D
   %awittenauer@linkedin.com>|To:=20<general@hadoop.apache.o
   rg>|Mime-version:=201.0|Content-transfer-encoding:=207bit
   |In-Reply-To:=20<673653.27493.qm@web37907.mail.mud.yahoo.
   com>;
  bh=v0uv4/IM/LKe8GWKcVKRSz42WlTPuf+j3KngmFCu8R8=;
  b=SYVeRTt1J4Oct4Huao9xHi1HQDuBdwP0Z1ExKzrSc0bBz/y7JpU2uyN/
   vXLbpBjOfW5nQHzHtGt/Ve04+U6Je2oIu/N4LYm8HiarXpKXyAuZO2h0r
   n4n0cRsGDLgeqRy;
X-IronPort-AV: E=Sophos;i="4.49,603,1262592000"; 
   d="scan'208";a="11254961"
Received: from 172.16.19.141 ([172.16.19.141]) by CORP-MAIL.linkedin.biz ([172.18.46.135]) via Exchange Front-End Server mail-access.linkedin.biz ([172.18.46.133]) with Microsoft Exchange Server HTTP-DAV ;
 Mon,  8 Mar 2010 18:30:32 +0000
User-Agent: Microsoft-Entourage/12.10.0.080409
Date: Mon, 08 Mar 2010 10:30:30 -0800
Subject: Re: Compilation failed when compile hadoop common release-0.20.2
From: Allen Wittenauer <awittenauer@linkedin.com>
To: <general@hadoop.apache.org>
Message-ID: <C7BA8246.386D%awittenauer@linkedin.com>
Thread-Topic: Compilation failed when compile hadoop common release-0.20.2
Thread-Index: Acq+7W5Ei0yddHnQFky5zH/Mc9+YWg==
In-Reply-To: <673653.27493.qm@web37907.mail.mud.yahoo.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit




On 3/8/10 10:06 AM, "Gary Yang" <garyyang6@yahoo.com> wrote:

> Hi Owen,
> 
> Thanks for the reply. From the link you provided, I found the build
> instruction. I do not understand the option, "-Djava5.home=/usr/local/jdk1.5".
> Does it mean I have to use JDK 1.5? I read somewhere it suggested to use JDK
> 1.6. 

JDK 1.5 is required to build the documentation.  JDK1.6 is used everywhere
else. So it uses the java5.home setting to kick off forrest without
upsetting the JAVA_HOME env var.



From general-return-1176-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 08 20:25:51 2010
Return-Path: <general-return-1176-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58381 invoked from network); 8 Mar 2010 20:25:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Mar 2010 20:25:51 -0000
Received: (qmail 44053 invoked by uid 500); 8 Mar 2010 20:25:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 43998 invoked by uid 500); 8 Mar 2010 20:25:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 43990 invoked by uid 99); 8 Mar 2010 20:25:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Mar 2010 20:25:25 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of swatt@us.ibm.com designates 32.97.110.160 as permitted sender)
Received: from [32.97.110.160] (HELO e39.co.us.ibm.com) (32.97.110.160)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Mar 2010 20:25:15 +0000
Received: from d03relay02.boulder.ibm.com (d03relay02.boulder.ibm.com [9.17.195.227])
	by e39.co.us.ibm.com (8.14.3/8.13.1) with ESMTP id o28KH7TY022579
	for <general@hadoop.apache.org>; Mon, 8 Mar 2010 13:17:07 -0700
Received: from d03av02.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.195.168])
	by d03relay02.boulder.ibm.com (8.13.8/8.13.8/NCO v9.1) with ESMTP id o28KOogL030216
	for <general@hadoop.apache.org>; Mon, 8 Mar 2010 13:24:50 -0700
Received: from d03av02.boulder.ibm.com (loopback [127.0.0.1])
	by d03av02.boulder.ibm.com (8.14.3/8.13.1/NCO v10.0 AVout) with ESMTP id o28KOovK024495
	for <general@hadoop.apache.org>; Mon, 8 Mar 2010 13:24:50 -0700
Received: from d03nm123.boulder.ibm.com (d03nm123.boulder.ibm.com [9.17.195.149])
	by d03av02.boulder.ibm.com (8.14.3/8.13.1/NCO v10.0 AVin) with ESMTP id o28KOoq6024490
	for <general@hadoop.apache.org>; Mon, 8 Mar 2010 13:24:50 -0700
In-Reply-To: <673653.27493.qm@web37907.mail.mud.yahoo.com>
References: <B8873D51-C95D-4509-A292-6836B3493BF8@apache.org> <673653.27493.qm@web37907.mail.mud.yahoo.com>
To: general@hadoop.apache.org
MIME-Version: 1.0
Subject: Re: Compilation failed when compile hadoop common release-0.20.2
X-KeepSent: 781492A6:58013FC4-862576E0:006FC000;
 type=4; name=$KeepSent
X-Mailer: Lotus Notes Release 8.5 December 05, 2008
Message-ID: <OF781492A6.58013FC4-ON862576E0.006FC000-862576E0.0070229E@us.ibm.com>
From: Stephen Watt <swatt@us.ibm.com>
Date: Mon, 8 Mar 2010 14:24:49 -0600
X-MIMETrack: Serialize by Router on D03NM123/03/M/IBM(Release 8.5.1HF41 | October 22, 2009) at
 03/08/2010 13:24:49,
	Serialize complete at 03/08/2010 13:24:49
Content-Type: multipart/alternative; boundary="=_alternative 0070229C862576E0_="
X-Virus-Checked: Checked by ClamAV on apache.org

--=_alternative 0070229C862576E0_=
Content-Type: text/plain; charset="US-ASCII"

Hi Gary

This is a script I put together based on the WikiPage link Owen sent you 
that will build most versions of Hadoop including 0.20.2 .  You are 
welcome to use it. Notice how I have JAVA_HOME point to java 6 and JAVA5 
point to java 5 (for Forrest). In this script its pointing to IBM Java 
installations, but you can use a Sun/Oracle JDK if you wish to as well. 
The Wiki Page should be able to answer other details.

#!/bin/sh
export VERSION=0.20.2
set PATH=$PATH:/home/hadoop/Java-Versions/ibm-java-i386-60/bin/
export HADOOP_INSTALL=/home/hadoop/Hadoop-Versions/hadoop-$VERSION
export FORREST_INSTALL=/home/hadoop/Test-Dependencies/apache-forrest-0.8
export XERCES_INSTALL=/home/hadoop/Test-Dependencies/xerces-c_2_8_0
export ANT_HOME=/home/hadoop/Test-Dependencies/apache-ant-1.7.1
export JAVA_HOME=/home/hadoop/Java-Versions/ibm-java-i386-60
export JAVA5=/home/hadoop/Java-Versions/ibm-java2-i386-50
export CFLAGS=-m32
export CXXFLAGS=-m32
export PATH=$PATH:$ANT_HOME/bin

cd $HADOOP_INSTALL

# For some reason these scripts do not have execute permissions
chmod 777 src/c++/utils/configure
chmod 777 src/examples/pipes/configure
chmod 777 src/native/configure

# Clean, Build and Run the Core (Non-Contrib) Unit Tests
ant -Dversion=$VERSION -Dcompile.native=true -Dcompile.c++=true 
-Dlibhdfs=1 -Dlibrecordio=true -Dxercescroot=$XERCES_INSTALL 
-Dforrest.home=$FORREST_INSTALL -Djava5.home=$JAVA5 clean tar test-core > 
/home/hadoop/Test-Scripts/Hadoop-$VERSION/ibm32build.out

Kind regards
Steve Watt



From:
Gary Yang <garyyang6@yahoo.com>
To:
general@hadoop.apache.org
Date:
03/08/2010 12:07 PM
Subject:
Re: Compilation failed when compile hadoop common release-0.20.2



Hi Owen,

Thanks for the reply. From the link you provided, I found the build 
instruction. I do not understand the option, 
"-Djava5.home=/usr/local/jdk1.5". Does it mean I have to use JDK 1.5? I 
read somewhere it suggested to use JDK 1.6. 

Also, the very first line is "export JAVA_HOME=/path/to/32bit/jdk
". Does it mean the JAVA_HOME jdk has to be 1.5?  Please let me know.


export JAVA_HOME=/path/to/32bit/jdk
export CFLAGS=-m32
export CXXFLAGS=-m32
ant -Dversion=X.Y.Z -Dcompile.native=true -Dcompile.c++=true -Dlibhdfs=1 
-Dlibrecordio=true -Dxercescroot=/usr/local/xerces-c 
-Declipse.home=/usr/lib/eclipse -Dforrest.home=/usr/local/forrest 
-Djava5.home=/usr/local/jdk1.5 clean api-report tar test test-c++-libhdfs
export JAVA_HOME=/path/to/64bit/jdk
export CFLAGS=-m64
export CXXFLAGS=-m64
ant -Dversion=X.Y.Z -Dcompile.native=true -Dcompile.c++=true 
compile-core-native compile-c++ tar


Thanks,


Gary



--- On Fri, 3/5/10, Owen O'Malley <omalley@apache.org> wrote:

> From: Owen O'Malley <omalley@apache.org>
> Subject: Re: Compilation failed when compile hadoop common 
release-0.20.2
> To: general@hadoop.apache.org
> Date: Friday, March 5, 2010, 4:32 PM
> 
> On Mar 5, 2010, at 3:47 PM, Gary Yang wrote:
> 
> > Hi,
> > 
> > I try to compile hadoop common of the release 0.20.2.
> Below are the error messages and java and ant versions I am
> using. Please tell me what I missed.
> 
> Forrest, which we use to generate the documentation,
> requires java 5. Therefore, run:
> 
> ant -Djava5.home=/some/path/to/java5 tar
> 
> There are several more you need. For a more complete list,
> I'd look at the how to release page:
> 
> http://wiki.apache.org/hadoop/HowToRelease
> 
> -- Owen
> 


 



--=_alternative 0070229C862576E0_=--

From general-return-1177-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 08 20:35:04 2010
Return-Path: <general-return-1177-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60328 invoked from network); 8 Mar 2010 20:35:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Mar 2010 20:35:04 -0000
Received: (qmail 66574 invoked by uid 500); 8 Mar 2010 20:34:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 66529 invoked by uid 500); 8 Mar 2010 20:34:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 66521 invoked by uid 99); 8 Mar 2010 20:34:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Mar 2010 20:34:38 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.191.91.167] (HELO web37905.mail.mud.yahoo.com) (209.191.91.167)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 08 Mar 2010 20:34:34 +0000
Received: (qmail 55572 invoked by uid 60001); 8 Mar 2010 20:34:13 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1268080452; bh=htHgxrJVDOEH4/dwFt+jBgjPAJwc7gF1bxsItqlguoI=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=TvVIyxKaA5JkhRZoddOyds6HhRhGEtgfHxuJRWBB76goyPbobeAati0Mw+CxAJKWUwTh8ATui6GLNju8b1Me87x6pciQZZ0bTDa+TZecSZS3C6u1WR8sYCpJxccKWXXoBcQWy9ESNrLaXYtqUMDd5Dq8PhE7Bg+R8D/mrI0lOTg=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding;
  b=cg4TqUfZzDpIvN1OZAgLeCEQ7zgN/f5mrS3nwqBAUnXLYsFbk4YOAT01My1yPvLFh8A/Lp6d99X1DTzp6UUGqatjMbGWmOA/yPp3peoJDxdHu2o+Kfs5We7i6qdtLBacHE/F3b1pKDlhKli185qeSY/+c7XNSRehhyQAdzStiOc=;
Message-ID: <940237.55385.qm@web37905.mail.mud.yahoo.com>
X-YMail-OSG: RnRqtWUVM1k_gX14t7rbJaDWU0vY.ckqw_a9rHpkC2FKBVoWMqNqZjRypGuUXxxUOrj11YJqDFo1x18Htdoe3fsa0SpRH45b4B03W4RkJO3KD6WENlHLRoAjYx3uk_bgkripgvhjjVWHmcRwN.kjQVjFwnGbDqLSWcoFbezrcddsSBjsW1T2NHz7ssUr1LH8IAeibOlUVrY4n6_xDvMO5mw_aWIRHBHZDys7gREpFQTAzKN7kkfWhHnr_qgOIGC55qAMkdQKWLy5MnelxljptlJSh9BAy4ZGQmP21Q4oPQ8mawOli3XG8D9bWjCkVkorV1FN5.pSxP36klORAAjKPrtIZ2CQJbO3wPnPfMhbjxT3i2d8YUQPSGk-
Received: from [63.147.59.14] by web37905.mail.mud.yahoo.com via HTTP; Mon, 08 Mar 2010 12:34:12 PST
X-Mailer: YahooMailClassic/9.2.12 YahooMailWebService/0.8.100.260964
Date: Mon, 8 Mar 2010 12:34:12 -0800 (PST)
From: Gary Yang <garyyang6@yahoo.com>
Subject: Re: Compilation failed when compile hadoop common release-0.20.2
To: general@hadoop.apache.org
In-Reply-To: <C7BA8246.386D%awittenauer@linkedin.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Allen,

I did not find the java 1.5. However, I found jdk 5.01 EE edition. I think =
it is java1.5. I downloaded and installed it. I got another compilation err=
or. See below. Any idea?

java_ee_sdk-5_01-linux.bin  (the java I downloaded)

/tmp/jdk1.5/jdk/bin/java -version
java version "1.5.0_09"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b03)


ls /usr/forrest-0.8
bin/  build/  etc/  index.html  KEYS  lib/  LICENSE.txt  main/  NOTICE.txt =
 plugins/  README.txt  site-author/  tools/  whiteboard/

ls /tmp/eclipse
about_files/  artifacts.xml   dropins/  eclipse.ini   features/  libcairo-s=
wt.so*  p2/       readme/
about.html    configuration/  eclipse*  epl-v10.html  icon.xpm   notice.htm=
l       plugins/



My build scripts:

#!/bin/sh

export JAVA_HOME=3D/tmp/jdk1.5/jdk
export CFLAGS=3D-m32
export CXXFLAGS=3D-m32
ant -Dversion=3D0.20.2 -Dcompile.native=3Dtrue -Dcompile.c++=3Dtrue -Dlibhd=
fs=3D1 -Dlibrecordio=3Dtrue -Dxercescroot=3D/usr/xercesc -Declipse.home=3D/=
tmp/eclipse -Dforrest.home=3D/usr/forrest-0.8 -Djava5.home=3D/tmp/jdk1.5/jd=
k clean api-report tar test test-c++-libhdfs



Error Messages:

    [touch] Creating /tmp/null568498220
   [delete] Deleting: /tmp/null568498220
     [copy] Copying 7 files to /tmp/hadoop-common-0.20.2/build/webapps

record-parser:

compile-rcc-compiler:
    [javac] Compiling 29 source files to /tmp/hadoop-common-0.20.2/build/cl=
asses
    [javac] javac: invalid target release: 1.6
    [javac] Usage: javac <options> <source files>
    [javac] where possible options include:
    [javac]   -g                         Generate all debugging info
    [javac]   -g:none                    Generate no debugging info
    [javac]   -g:{lines,vars,source}     Generate only some debugging info
    [javac]   -nowarn                    Generate no warnings
    [javac]   -verbose                   Output messages about what the com=
piler is doing
    [javac]   -deprecation               Output source locations where depr=
ecated APIs are used
    [javac]   -classpath <path>          Specify where to find user class f=
iles
    [javac]   -cp <path>                 Specify where to find user class f=
iles
    [javac]   -sourcepath <path>         Specify where to find input source=
 files
    [javac]   -bootclasspath <path>      Override location of bootstrap cla=
ss files
    [javac]   -extdirs <dirs>            Override location of installed ext=
ensions
    [javac]   -endorseddirs <dirs>       Override location of endorsed stan=
dards path
    [javac]   -d <directory>             Specify where to place generated c=
lass files
    [javac]   -encoding <encoding>       Specify character encoding used by=
 source files
    [javac]   -source <release>          Provide source compatibility with =
specified release
    [javac]   -target <release>          Generate class files for specific =
VM version
    [javac]   -version                   Version information
    [javac]   -help                      Print a synopsis of standard optio=
ns
    [javac]   -X                         Print a synopsis of nonstandard op=
tions
    [javac]   -J<flag>                   Pass <flag> directly to the runtim=
e system
    [javac]=20

BUILD FAILED
/tmp/hadoop-common-0.20.2/build.xml:316: Compile failed; see the compiler e=
rror output for details.


I deleted my java 1.6. I do not understand why I got the error below.

    [javac] javac: invalid target release: 1.6
    [javac] Usage: javac <options> <source files>


Thanks,


Gary


--- On Mon, 3/8/10, Allen Wittenauer <awittenauer@linkedin.com> wrote:

> From: Allen Wittenauer <awittenauer@linkedin.com>
> Subject: Re: Compilation failed when compile hadoop common release-0.20.2
> To: general@hadoop.apache.org
> Date: Monday, March 8, 2010, 10:30 AM
>=20
>=20
>=20
> On 3/8/10 10:06 AM, "Gary Yang" <garyyang6@yahoo.com>
> wrote:
>=20
> > Hi Owen,
> >=20
> > Thanks for the reply. From the link you provided, I
> found the build
> > instruction. I do not understand the option,
> "-Djava5.home=3D/usr/local/jdk1.5".
> > Does it mean I have to use JDK 1.5? I read somewhere
> it suggested to use JDK
> > 1.6.=20
>=20
> JDK 1.5 is required to build the documentation.=A0
> JDK1.6 is used everywhere
> else. So it uses the java5.home setting to kick off forrest
> without
> upsetting the JAVA_HOME env var.
>=20
>=20
> =0A=0A=0A      

From general-return-1178-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 08 20:38:23 2010
Return-Path: <general-return-1178-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 61393 invoked from network); 8 Mar 2010 20:38:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Mar 2010 20:38:23 -0000
Received: (qmail 70540 invoked by uid 500); 8 Mar 2010 20:37:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70491 invoked by uid 500); 8 Mar 2010 20:37:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70483 invoked by uid 99); 8 Mar 2010 20:37:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Mar 2010 20:37:57 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.191.91.163] (HELO web37901.mail.mud.yahoo.com) (209.191.91.163)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 08 Mar 2010 20:37:53 +0000
Received: (qmail 49392 invoked by uid 60001); 8 Mar 2010 20:37:32 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1268080651; bh=/SAOrRcNxEwRmnv7cQt6SWYQEn8er711Lj7KzlMwW2w=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=d28OBB3xgcnCz+7dlnRngC5lOxz7E1a5Mb6GN5WzmOa1ZQvhqyNFHOcoiaUVDUN307w9ixx2r3CFE602wcZY2Iqi7z4Xx3oq/IujPcS9GTttEH+1QngaXsxF89UK4n7TTbEQqYpIsCjg/eLKU138YqDNIaRPYJVvkFEeT7Oif4c=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding;
  b=LZK/e75P7SYy6+CJuj22AfzD3gRQqEOBWuEYuZCs/aQNq583tWdcpJlYUTPRUcb7Ky5GHRjTYypaK6cYCL0Leb+NmQVlhVoZ0H2uKzFsdYUbVF5Oa18r+MwzU/o5oqxr8jQ4rvAWecJF6LV7MgTMkjPm9apescr5wEp+eANUZ/c=;
Message-ID: <940415.39128.qm@web37901.mail.mud.yahoo.com>
X-YMail-OSG: 9jBqQ1IVM1n8f4M6BCe.t2SJJkmpDHYhUUDWrBa74RyZEsV
 LryzdxLeubWjYJepDu4Dp_L7ykIYrDUbUTRQe9cWKbrxXPYZy6GANlYwZXm5
 YVfR8z936xI7VpV3hlETqZMDpR6ciVTH3MlwbETKaFJNMifqo1wNq23_lADI
 Kyf5P9NQEW4XI1Rc3g9NekSR86DZfW_ny7TFkjLULpnWGsjTlHqTb3Jl1jgK
 09Wf2PBk.16rhumNepAdwImPIm1gyEYFkmsnjU.GW5RlqQHOyd.A7bsBOY8w
 YK1D1S8rxjdUeiA8514JARKePRj0kHcmv.p2zUeZ0Q48qRLRcUKm_RgWNATR
 B
Received: from [63.147.59.14] by web37901.mail.mud.yahoo.com via HTTP; Mon, 08 Mar 2010 12:37:31 PST
X-Mailer: YahooMailClassic/9.2.12 YahooMailWebService/0.8.100.260964
Date: Mon, 8 Mar 2010 12:37:31 -0800 (PST)
From: Gary Yang <garyyang6@yahoo.com>
Subject: Re: Compilation failed when compile hadoop common release-0.20.2
To: general@hadoop.apache.org
In-Reply-To: <OF781492A6.58013FC4-ON862576E0.006FC000-862576E0.0070229E@us.ibm.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Steve,

Thank you very much for this email. I sent an email and reported an compila=
tion error before I got your email. I am going to try your script and let y=
ou know.

Thank you again,


Gary


--- On Mon, 3/8/10, Stephen Watt <swatt@us.ibm.com> wrote:

> From: Stephen Watt <swatt@us.ibm.com>
> Subject: Re: Compilation failed when compile hadoop common release-0.20.2
> To: general@hadoop.apache.org
> Date: Monday, March 8, 2010, 12:24 PM
> Hi Gary
>=20
> This is a script I put together based on the WikiPage link
> Owen sent you=20
> that will build most versions of Hadoop including 0.20.2
> .=A0 You are=20
> welcome to use it. Notice how I have JAVA_HOME point to
> java 6 and JAVA5=20
> point to java 5 (for Forrest). In this script its pointing
> to IBM Java=20
> installations, but you can use a Sun/Oracle JDK if you wish
> to as well.=20
> The Wiki Page should be able to answer other details.
>=20
> #!/bin/sh
> export VERSION=3D0.20.2
> set
> PATH=3D$PATH:/home/hadoop/Java-Versions/ibm-java-i386-60/bin/
> export
> HADOOP_INSTALL=3D/home/hadoop/Hadoop-Versions/hadoop-$VERSION
> export
> FORREST_INSTALL=3D/home/hadoop/Test-Dependencies/apache-forrest-0.8
> export
> XERCES_INSTALL=3D/home/hadoop/Test-Dependencies/xerces-c_2_8_0
> export
> ANT_HOME=3D/home/hadoop/Test-Dependencies/apache-ant-1.7.1
> export
> JAVA_HOME=3D/home/hadoop/Java-Versions/ibm-java-i386-60
> export JAVA5=3D/home/hadoop/Java-Versions/ibm-java2-i386-50
> export CFLAGS=3D-m32
> export CXXFLAGS=3D-m32
> export PATH=3D$PATH:$ANT_HOME/bin
>=20
> cd $HADOOP_INSTALL
>=20
> # For some reason these scripts do not have execute
> permissions
> chmod 777 src/c++/utils/configure
> chmod 777 src/examples/pipes/configure
> chmod 777 src/native/configure
>=20
> # Clean, Build and Run the Core (Non-Contrib) Unit Tests
> ant -Dversion=3D$VERSION -Dcompile.native=3Dtrue
> -Dcompile.c++=3Dtrue=20
> -Dlibhdfs=3D1 -Dlibrecordio=3Dtrue
> -Dxercescroot=3D$XERCES_INSTALL=20
> -Dforrest.home=3D$FORREST_INSTALL -Djava5.home=3D$JAVA5 clean
> tar test-core >=20
> /home/hadoop/Test-Scripts/Hadoop-$VERSION/ibm32build.out
>=20
> Kind regards
> Steve Watt
>=20
>=20
>=20
> From:
> Gary Yang <garyyang6@yahoo.com>
> To:
> general@hadoop.apache.org
> Date:
> 03/08/2010 12:07 PM
> Subject:
> Re: Compilation failed when compile hadoop common
> release-0.20.2
>=20
>=20
>=20
> Hi Owen,
>=20
> Thanks for the reply. From the link you provided, I found
> the build=20
> instruction. I do not understand the option,=20
> "-Djava5.home=3D/usr/local/jdk1.5". Does it mean I have to
> use JDK 1.5? I=20
> read somewhere it suggested to use JDK 1.6.=20
>=20
> Also, the very first line is "export
> JAVA_HOME=3D/path/to/32bit/jdk
> ". Does it mean the JAVA_HOME jdk has to be 1.5?=A0
> Please let me know.
>=20
>=20
> export JAVA_HOME=3D/path/to/32bit/jdk
> export CFLAGS=3D-m32
> export CXXFLAGS=3D-m32
> ant -Dversion=3DX.Y.Z -Dcompile.native=3Dtrue
> -Dcompile.c++=3Dtrue -Dlibhdfs=3D1=20
> -Dlibrecordio=3Dtrue -Dxercescroot=3D/usr/local/xerces-c=20
> -Declipse.home=3D/usr/lib/eclipse
> -Dforrest.home=3D/usr/local/forrest=20
> -Djava5.home=3D/usr/local/jdk1.5 clean api-report tar test
> test-c++-libhdfs
> export JAVA_HOME=3D/path/to/64bit/jdk
> export CFLAGS=3D-m64
> export CXXFLAGS=3D-m64
> ant -Dversion=3DX.Y.Z -Dcompile.native=3Dtrue
> -Dcompile.c++=3Dtrue=20
> compile-core-native compile-c++ tar
>=20
>=20
> Thanks,
>=20
>=20
> Gary
>=20
>=20
>=20
> --- On Fri, 3/5/10, Owen O'Malley <omalley@apache.org>
> wrote:
>=20
> > From: Owen O'Malley <omalley@apache.org>
> > Subject: Re: Compilation failed when compile hadoop
> common=20
> release-0.20.2
> > To: general@hadoop.apache.org
> > Date: Friday, March 5, 2010, 4:32 PM
> >=20
> > On Mar 5, 2010, at 3:47 PM, Gary Yang wrote:
> >=20
> > > Hi,
> > >=20
> > > I try to compile hadoop common of the release
> 0.20.2.
> > Below are the error messages and java and ant versions
> I am
> > using. Please tell me what I missed.
> >=20
> > Forrest, which we use to generate the documentation,
> > requires java 5. Therefore, run:
> >=20
> > ant -Djava5.home=3D/some/path/to/java5 tar
> >=20
> > There are several more you need. For a more complete
> list,
> > I'd look at the how to release page:
> >=20
> > http://wiki.apache.org/hadoop/HowToRelease
> >=20
> > -- Owen
> >=20
>=20
>=20
> =20
>=20
>=20
> =0A=0A=0A      

From general-return-1179-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 08 22:21:28 2010
Return-Path: <general-return-1179-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 88590 invoked from network); 8 Mar 2010 22:21:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Mar 2010 22:21:28 -0000
Received: (qmail 31143 invoked by uid 500); 8 Mar 2010 22:21:02 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 31108 invoked by uid 500); 8 Mar 2010 22:21:02 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 31096 invoked by uid 99); 8 Mar 2010 22:21:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 08 Mar 2010 22:21:02 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.191.91.164] (HELO web37902.mail.mud.yahoo.com) (209.191.91.164)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 08 Mar 2010 22:20:58 +0000
Received: (qmail 24463 invoked by uid 60001); 8 Mar 2010 22:20:36 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1268086836; bh=rCEsivLLrofsCz6G38BwFKzpjmmN4sGM51tAAp2z+eM=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=qWgxzWHCSVIfyxJSGLASmMUwP5Uwk8GKK5HWKiR1uRO5V4LuQFlnCLMqa0TUBX/UlAC4GlwrgTJQN0f8K07u9n6/8HkeJTuHyIiJ8IQ/od5+Kv7V/m0nGRe9QcMPNRmFPXTtjvNKhhmjp6/64sHBkAM5Le3qCFnqiQyoUC5W5Gs=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding;
  b=aTNh35B167Ejq6Vat0seTj4ooQX8TVGAI89ibxi41vgtWYD05eMS8lrQldiYxHqJHWV+eBLV1PiK2as/IMBy1kdkQkEDTvXkfEEw3YU1UkDQ4U7ruL/pQaGYzhJPMR3D7GFzKBhZa3uzNJsXHJH9kG8n1+pptTUhQfYbGHBNLXc=;
Message-ID: <813950.21244.qm@web37902.mail.mud.yahoo.com>
X-YMail-OSG: sy5EyuYVM1kCR_ocL4PUlGmZNYl5u4wrRVw8kxyGtxhH98ynXZPfkYZm7r9G4MdderPBxraE8k3Sf6tJvhbRvCl4Mz2a5XA4vQhdiZ1mfJhVpTNi8b5Rj6mjYbEhZC8euZEjkVLlli8hcKt9WnHCpop7eWZqNqdnXxHIZOc5fWAw3GfR8r4bGOv5wGx_c5DPGq6BEETYoRDLJ97zeS876c1QzW6SE.rda7K9eZ8jn9oeJ9bykb42r9x3BuTDlISK.2KNdrP_BX9lT_yvJXSyRKdwMvk6q0YewH0MAGyooHE8BwxjtdjQKxsSyHJXOqGFr.Hr0zaW8vm4fPS5pUO9oALk8Dgs
Received: from [63.147.59.14] by web37902.mail.mud.yahoo.com via HTTP; Mon, 08 Mar 2010 14:20:36 PST
X-Mailer: YahooMailClassic/9.2.12 YahooMailWebService/0.8.100.260964
Date: Mon, 8 Mar 2010 14:20:36 -0800 (PST)
From: Gary Yang <garyyang6@yahoo.com>
Subject: Re: Compilation failed when compile hadoop common release-0.20.2
To: general@hadoop.apache.org
In-Reply-To: <OF781492A6.58013FC4-ON862576E0.006FC000-862576E0.0070229E@us.ibm.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Steve,

I used your script. However, I got errors related to xerces-c-3.1.0.

Below is how I compiled and installed xerces.

./configure --prefix=3D/path/tools/xerces-c-3.1.0
make
make install



Below are errors related to xerces-c-3.1.0. Do you have any idea?

jar-test:
      [jar] Building jar: /tmp/hadoop-common-0.20.2/build/hadoop-0.20.2-tes=
t.jar

ant-tasks:
     [copy] Copying 1 file to /tmp/hadoop-common-0.20.2/build/ant/org/apach=
e/hadoop/ant
      [jar] Building jar: /tmp/hadoop-common-0.20.2/build/hadoop-0.20.2-ant=
.jar

compile-librecordio:
    [mkdir] Created dir: /tmp/hadoop-common-0.20.2/build/librecordio
     [exec] g++ -g3 -O0 -Wall -c -I/path/tools/xerces-c-3.1.0/include -o /t=
mp/hadoop-common-0.20.2/build/librecordio/recordio.o recordio.cc
     [exec] In file included from recordio.cc:22:
     [exec] xmlarchive.hh:77: error: conflicting return type specified for =
`virtual unsigned int hadoop::MyBinInputStream::curPos() const'
     [exec] /path/tools/xerces-c-3.1.0/include/xercesc/util/BinInputStream.=
hpp:41: error:   overriding `virtual XMLFilePos xercesc_3_1::BinInputStream=
::curPos() const'
     [exec] xmlarchive.hh: In member function `virtual xercesc_3_1::BinInpu=
tStream* hadoop::MyInputSource::makeStream() const':
     [exec] xmlarchive.hh:97: error: cannot allocate an object of type `had=
oop::MyBinInputStream'
     [exec] xmlarchive.hh:97: error:   because the following virtual functi=
ons are abstract:
     [exec] /path/tools/xerces-c-3.1.0/include/xercesc/util/BinInputStream.=
hpp:67: error:  virtual const XMLCh* xercesc_3_1::BinInputStream::getConten=
tType() const
     [exec] make: *** [/tmp/hadoop-common-0.20.2/build/librecordio/recordio=
.o] Error 1

BUILD FAILED
/tmp/hadoop-common-0.20.2/build.xml:1316: exec returned: 2



Thanks,


Gary



--- On Mon, 3/8/10, Stephen Watt <swatt@us.ibm.com> wrote:

> From: Stephen Watt <swatt@us.ibm.com>
> Subject: Re: Compilation failed when compile hadoop common release-0.20.2
> To: general@hadoop.apache.org
> Date: Monday, March 8, 2010, 12:24 PM
> Hi Gary
>=20
> This is a script I put together based on the WikiPage link
> Owen sent you=20
> that will build most versions of Hadoop including 0.20.2
> .=A0 You are=20
> welcome to use it. Notice how I have JAVA_HOME point to
> java 6 and JAVA5=20
> point to java 5 (for Forrest). In this script its pointing
> to IBM Java=20
> installations, but you can use a Sun/Oracle JDK if you wish
> to as well.=20
> The Wiki Page should be able to answer other details.
>=20
> #!/bin/sh
> export VERSION=3D0.20.2
> set
> PATH=3D$PATH:/home/hadoop/Java-Versions/ibm-java-i386-60/bin/
> export
> HADOOP_INSTALL=3D/home/hadoop/Hadoop-Versions/hadoop-$VERSION
> export
> FORREST_INSTALL=3D/home/hadoop/Test-Dependencies/apache-forrest-0.8
> export
> XERCES_INSTALL=3D/home/hadoop/Test-Dependencies/xerces-c_2_8_0
> export
> ANT_HOME=3D/home/hadoop/Test-Dependencies/apache-ant-1.7.1
> export
> JAVA_HOME=3D/home/hadoop/Java-Versions/ibm-java-i386-60
> export JAVA5=3D/home/hadoop/Java-Versions/ibm-java2-i386-50
> export CFLAGS=3D-m32
> export CXXFLAGS=3D-m32
> export PATH=3D$PATH:$ANT_HOME/bin
>=20
> cd $HADOOP_INSTALL
>=20
> # For some reason these scripts do not have execute
> permissions
> chmod 777 src/c++/utils/configure
> chmod 777 src/examples/pipes/configure
> chmod 777 src/native/configure
>=20
> # Clean, Build and Run the Core (Non-Contrib) Unit Tests
> ant -Dversion=3D$VERSION -Dcompile.native=3Dtrue
> -Dcompile.c++=3Dtrue=20
> -Dlibhdfs=3D1 -Dlibrecordio=3Dtrue
> -Dxercescroot=3D$XERCES_INSTALL=20
> -Dforrest.home=3D$FORREST_INSTALL -Djava5.home=3D$JAVA5 clean
> tar test-core >=20
> /home/hadoop/Test-Scripts/Hadoop-$VERSION/ibm32build.out
>=20
> Kind regards
> Steve Watt
>=20
>=20
>=20
> From:
> Gary Yang <garyyang6@yahoo.com>
> To:
> general@hadoop.apache.org
> Date:
> 03/08/2010 12:07 PM
> Subject:
> Re: Compilation failed when compile hadoop common
> release-0.20.2
>=20
>=20
>=20
> Hi Owen,
>=20
> Thanks for the reply. From the link you provided, I found
> the build=20
> instruction. I do not understand the option,=20
> "-Djava5.home=3D/usr/local/jdk1.5". Does it mean I have to
> use JDK 1.5? I=20
> read somewhere it suggested to use JDK 1.6.=20
>=20
> Also, the very first line is "export
> JAVA_HOME=3D/path/to/32bit/jdk
> ". Does it mean the JAVA_HOME jdk has to be 1.5?=A0
> Please let me know.
>=20
>=20
> export JAVA_HOME=3D/path/to/32bit/jdk
> export CFLAGS=3D-m32
> export CXXFLAGS=3D-m32
> ant -Dversion=3DX.Y.Z -Dcompile.native=3Dtrue
> -Dcompile.c++=3Dtrue -Dlibhdfs=3D1=20
> -Dlibrecordio=3Dtrue -Dxercescroot=3D/usr/local/xerces-c=20
> -Declipse.home=3D/usr/lib/eclipse
> -Dforrest.home=3D/usr/local/forrest=20
> -Djava5.home=3D/usr/local/jdk1.5 clean api-report tar test
> test-c++-libhdfs
> export JAVA_HOME=3D/path/to/64bit/jdk
> export CFLAGS=3D-m64
> export CXXFLAGS=3D-m64
> ant -Dversion=3DX.Y.Z -Dcompile.native=3Dtrue
> -Dcompile.c++=3Dtrue=20
> compile-core-native compile-c++ tar
>=20
>=20
> Thanks,
>=20
>=20
> Gary
>=20
>=20
>=20
> --- On Fri, 3/5/10, Owen O'Malley <omalley@apache.org>
> wrote:
>=20
> > From: Owen O'Malley <omalley@apache.org>
> > Subject: Re: Compilation failed when compile hadoop
> common=20
> release-0.20.2
> > To: general@hadoop.apache.org
> > Date: Friday, March 5, 2010, 4:32 PM
> >=20
> > On Mar 5, 2010, at 3:47 PM, Gary Yang wrote:
> >=20
> > > Hi,
> > >=20
> > > I try to compile hadoop common of the release
> 0.20.2.
> > Below are the error messages and java and ant versions
> I am
> > using. Please tell me what I missed.
> >=20
> > Forrest, which we use to generate the documentation,
> > requires java 5. Therefore, run:
> >=20
> > ant -Djava5.home=3D/some/path/to/java5 tar
> >=20
> > There are several more you need. For a more complete
> list,
> > I'd look at the how to release page:
> >=20
> > http://wiki.apache.org/hadoop/HowToRelease
> >=20
> > -- Owen
> >=20
>=20
>=20
> =20
>=20
>=20
> =0A=0A=0A      

From general-return-1180-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 09 04:20:49 2010
Return-Path: <general-return-1180-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83517 invoked from network); 9 Mar 2010 04:20:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Mar 2010 04:20:48 -0000
Received: (qmail 4451 invoked by uid 500); 9 Mar 2010 04:20:22 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4168 invoked by uid 500); 9 Mar 2010 04:20:22 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4160 invoked by uid 99); 9 Mar 2010 04:20:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Mar 2010 04:20:21 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of zookog@gmail.com designates 209.85.223.197 as permitted sender)
Received: from [209.85.223.197] (HELO mail-iw0-f197.google.com) (209.85.223.197)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Mar 2010 04:20:17 +0000
Received: by iwn35 with SMTP id 35so1647605iwn.2
        for <general@hadoop.apache.org>; Mon, 08 Mar 2010 20:19:56 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=NUpKd86HcC+IV9m7q9qNVrr4sf9tWWYXSrPp0i3Koeg=;
        b=cI9o8k0ytuslni8FpiScAbLg9UXj9miIvv2EZwcx6wJgDPSsLWkmt0+XzBS7WO7V9g
         KuoQJ9rEigzDMTPVqzwL9Dvg9j0Gemmmz+dEte6sas13su5areLNpNLrbOdl3R47OWzo
         L30v+gUX2EWJeg7uOlEK5oqaYoi7V4/04EZzE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=atQrTN8S3BaQQH8uPPFF0tzJj/KnyiCmSfKPiZKTZeIOdFrE2KSJlDQL0uWQuR3UrF
         FykojmwW2HOhoEOO/8AdqEGi64C4xTfrDq4lW4TxF/jA8D4PbCq7tq0pZsULBHXzHOjx
         S5snPmFq1cmVIcaru/gXm/Rw42P5KbH0OMIi4=
MIME-Version: 1.0
Received: by 10.231.79.193 with SMTP id q1mr71177ibk.59.1268108396045; Mon, 08 
	Mar 2010 20:19:56 -0800 (PST)
In-Reply-To: <C7B74933.17C82%rphulari@yahoo-inc.com>
References: <27798051.post@talk.nabble.com>
	 <C7B74933.17C82%rphulari@yahoo-inc.com>
Date: Mon, 8 Mar 2010 21:19:55 -0700
Message-ID: <cd6401a1003082019y2032dc73xf03c45a7c1fbf487@mail.gmail.com>
Subject: Re: Fault Tolerancy in core hadoop and HDFS
From: "Zooko O'Whielacronx" <zookog@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8

You might also be interested in hadoop-lafs
(http://code.google.com/p/hadoop-lafs ) which lets Hadoop use
Tahoe-LAFS (http://tahoe-lafs.org ) instead of HDFS. Tahoe-LAFS is a
fault-tolerant distributed data store which uses erasure coding (c.f.
https://issues.apache.org/jira/browse/HDFS-503 ) and SHA-256-based
integrity checks.

Regards,

Zooko

From general-return-1181-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 09 19:23:22 2010
Return-Path: <general-return-1181-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72547 invoked from network); 9 Mar 2010 19:23:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Mar 2010 19:23:21 -0000
Received: (qmail 7292 invoked by uid 500); 9 Mar 2010 19:22:53 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 7234 invoked by uid 500); 9 Mar 2010 19:22:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 7226 invoked by uid 99); 9 Mar 2010 19:22:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Mar 2010 19:22:52 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 09 Mar 2010 19:22:50 +0000
Received: (qmail 72424 invoked by uid 99); 9 Mar 2010 19:22:55 -0000
Received: from localhost.apache.org (HELO [192.168.168.110]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Mar 2010 19:22:55 +0000
Message-ID: <4B969FF3.7070607@apache.org>
Date: Tue, 09 Mar 2010 11:22:27 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Hadoop 0.20.2 is released
References: <1267dd3b1003011331x13ee1070gf081ce0c58f549e8@mail.gmail.com>
In-Reply-To: <1267dd3b1003011331x13ee1070gf081ce0c58f549e8@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Chris, do you intend to publish the jars to the Maven repository, or 
should someone else volunteer to do this?  It would be best if the 
Hadoop project did this officially in org.apache.hadoop, otherwise 
someone else will publish these jars elsewhere, which creates confusion.

https://issues.apache.org/jira/browse/HADOOP-6617

Instructions for doing this have not yet been added to 
http://wiki.apache.org/hadoop/HowToRelease, but in Avro I've just used 
scp to publish jars to Maven.

http://wiki.apache.org/hadoop/Avro/HowToRelease

Cheers,

Doug

Chris Douglas wrote:
> Hadoop 0.20.2 is released and propagating to mirrors. The list of
> changes may be found in the release notes:
> 
> http://bit.ly/dd6rwL
> 
> And in JIRA:
> 
> COMMON: http://bit.ly/9fDo2f
> MAPREDUCE: http://bit.ly/aD0BM5
> HDFS: http://bit.ly/ckvMfO
> 
> Thanks to everyone who worked on this release. -C

From general-return-1182-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 09 20:19:57 2010
Return-Path: <general-return-1182-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 87687 invoked from network); 9 Mar 2010 20:19:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Mar 2010 20:19:57 -0000
Received: (qmail 85687 invoked by uid 500); 9 Mar 2010 20:19:28 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85652 invoked by uid 500); 9 Mar 2010 20:19:28 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85644 invoked by uid 99); 9 Mar 2010 20:19:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Mar 2010 20:19:28 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Mar 2010 20:19:24 +0000
Received: from wlanvpn-mc2e-246-12.corp.yahoo.com (wlanvpn-mc2e-246-12.corp.yahoo.com [172.21.148.12])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o29KIIFZ087307
	for <general@hadoop.apache.org>; Tue, 9 Mar 2010 12:18:19 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=s7NgM5N1lMHE2HpoyOWbwVRrCkvvoDdLcwZI5X07ew4zjMgi+e1OCXWxurzJXMjP
Message-Id: <52E5C1DD-8859-4F92-8886-DD4725FD475B@yahoo-inc.com>
From: Lee Tucker <ltucker@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
In-Reply-To: <4B969FF3.7070607@apache.org>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Hadoop 0.20.2 is released
Date: Tue, 9 Mar 2010 12:18:18 -0800
References: <1267dd3b1003011331x13ee1070gf081ce0c58f549e8@mail.gmail.com> <4B969FF3.7070607@apache.org>
X-Mailer: Apple Mail (2.936)

HADOOP-6382 exists to back port enough of the maven-in-ant  
infrastructure into HADOOP 20 to allow these jars to be published.

Lee Tucker
ltucker@yahoo-inc.com



On Mar 9, 2010, at 11:22 AM, Doug Cutting wrote:

> Chris, do you intend to publish the jars to the Maven repository, or
> should someone else volunteer to do this?  It would be best if the
> Hadoop project did this officially in org.apache.hadoop, otherwise
> someone else will publish these jars elsewhere, which creates  
> confusion.
>
> https://issues.apache.org/jira/browse/HADOOP-6617
>
> Instructions for doing this have not yet been added to
> http://wiki.apache.org/hadoop/HowToRelease, but in Avro I've just used
> scp to publish jars to Maven.
>
> http://wiki.apache.org/hadoop/Avro/HowToRelease
>
> Cheers,
>
> Doug
>
> Chris Douglas wrote:
>> Hadoop 0.20.2 is released and propagating to mirrors. The list of
>> changes may be found in the release notes:
>>
>> http://bit.ly/dd6rwL
>>
>> And in JIRA:
>>
>> COMMON: http://bit.ly/9fDo2f
>> MAPREDUCE: http://bit.ly/aD0BM5
>> HDFS: http://bit.ly/ckvMfO
>>
>> Thanks to everyone who worked on this release. -C


From general-return-1183-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 09 21:01:41 2010
Return-Path: <general-return-1183-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 99818 invoked from network); 9 Mar 2010 21:01:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Mar 2010 21:01:41 -0000
Received: (qmail 69116 invoked by uid 500); 9 Mar 2010 21:01:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 69060 invoked by uid 500); 9 Mar 2010 21:01:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 69051 invoked by uid 99); 9 Mar 2010 21:01:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Mar 2010 21:01:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 09 Mar 2010 21:01:07 +0000
Received: (qmail 99761 invoked by uid 99); 9 Mar 2010 21:01:15 -0000
Received: from localhost.apache.org (HELO mail-bw0-f211.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Mar 2010 21:01:15 +0000
Received: by bwz3 with SMTP id 3so1332877bwz.29
        for <general@hadoop.apache.org>; Tue, 09 Mar 2010 13:00:45 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.144.153 with SMTP id z25mr376571bku.198.1268168445451; 
	Tue, 09 Mar 2010 13:00:45 -0800 (PST)
In-Reply-To: <52E5C1DD-8859-4F92-8886-DD4725FD475B@yahoo-inc.com>
References: <1267dd3b1003011331x13ee1070gf081ce0c58f549e8@mail.gmail.com>
	 <4B969FF3.7070607@apache.org>
	 <52E5C1DD-8859-4F92-8886-DD4725FD475B@yahoo-inc.com>
Date: Tue, 9 Mar 2010 13:00:45 -0800
Message-ID: <1267dd3b1003091300l33c73746r38a2eeeebdf2e694@mail.gmail.com>
Subject: Re: Hadoop 0.20.2 is released
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Lee is right. I'll work with Lee and Giri to publish 0.20.2, but
someone more familiar with maven should work on getting HADOOP-6382
into 0.20.3. -C

On Tue, Mar 9, 2010 at 12:18 PM, Lee Tucker <ltucker@yahoo-inc.com> wrote:
> HADOOP-6382 exists to back port enough of the maven-in-ant infrastructure
> into HADOOP 20 to allow these jars to be published.
>
> Lee Tucker
> ltucker@yahoo-inc.com
>
>
>
> On Mar 9, 2010, at 11:22 AM, Doug Cutting wrote:
>
>> Chris, do you intend to publish the jars to the Maven repository, or
>> should someone else volunteer to do this? =A0It would be best if the
>> Hadoop project did this officially in org.apache.hadoop, otherwise
>> someone else will publish these jars elsewhere, which creates confusion.
>>
>> https://issues.apache.org/jira/browse/HADOOP-6617
>>
>> Instructions for doing this have not yet been added to
>> http://wiki.apache.org/hadoop/HowToRelease, but in Avro I've just used
>> scp to publish jars to Maven.
>>
>> http://wiki.apache.org/hadoop/Avro/HowToRelease
>>
>> Cheers,
>>
>> Doug
>>
>> Chris Douglas wrote:
>>>
>>> Hadoop 0.20.2 is released and propagating to mirrors. The list of
>>> changes may be found in the release notes:
>>>
>>> http://bit.ly/dd6rwL
>>>
>>> And in JIRA:
>>>
>>> COMMON: http://bit.ly/9fDo2f
>>> MAPREDUCE: http://bit.ly/aD0BM5
>>> HDFS: http://bit.ly/ckvMfO
>>>
>>> Thanks to everyone who worked on this release. -C
>
>

From general-return-1184-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 10 04:46:00 2010
Return-Path: <general-return-1184-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 15910 invoked from network); 10 Mar 2010 04:45:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Mar 2010 04:45:59 -0000
Received: (qmail 51368 invoked by uid 500); 10 Mar 2010 04:45:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51036 invoked by uid 500); 10 Mar 2010 04:45:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51028 invoked by uid 99); 10 Mar 2010 04:45:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Mar 2010 04:45:28 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ctubbsii@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Mar 2010 04:45:23 +0000
Received: by wyb38 with SMTP id 38so3472853wyb.35
        for <general@hadoop.apache.org>; Tue, 09 Mar 2010 20:45:02 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=Ac4GJrRcSQxkG2FSRNv4G/sszubVrCgREhP7hXoVyLw=;
        b=YEZQ3ZwMuseQH29hmzOLAZ3tIZuI2bfPoNCLiNGt1H0+KymgLkKwJGe8OxwWx2c0Pe
         eXvGncB1A3i3gisKiOFUMySUMOrn2TCW9YfWnh+m3ICxRoAl5/RrOg4UiaGP2hS+2IBf
         2X8a926M1I0EmZP2EfBOZMpElgEdkfH6QRiEk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=C7wkPSrzROwxnJ7nn900RFveivo2BX3ysdOocH/lQ+dpejbxj+tjkFJkMiioMrmR0a
         k4X07sYRNHJtjRyWhmXH1swRQsAJK+plq9bScsSOB23wWqrt05mnBs17VHTeSP2VLdJk
         RX3WuoxULafoF1eD6ourrbLcyWzBbZHJbA9oA=
MIME-Version: 1.0
Received: by 10.216.85.9 with SMTP id t9mr146028wee.79.1268196302484; Tue, 09 
	Mar 2010 20:45:02 -0800 (PST)
Date: Tue, 9 Mar 2010 23:45:02 -0500
Message-ID: <9db00c201003092045k31990da2j587a9e92235cc7c4@mail.gmail.com>
Subject: Native libraries on Mac
From: Christopher Tubbs <ctubbsii@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

I'm trying to build the native libraries for hadoop on Mac OS X
Leopard (10.5.8) for Hadoop 0.20.1

I tried the "ant compile-native" method, but I don't have access to
the internet. I do have a local repository though with all the
dependencies (not to mention they are in the $HADOOP_HOME/lib). I was
able to edit the build.xml file to point the repository to my local
one so I could download ivy, but now ivy can't find the dependencies,
and I can't figure out how to point ivy to my maven repository.

Since the first method didn't work, I tried applying the patch in
HADOOP-3659 (has that been included in 0.20.2?), and got things
working with a little messing with aclocal and friends, but when I run
make, I get a ZLibCompressor.lo, missing argument for "-m" error in
the build output before it quits.

Any help on building the native libs by either method would be greatly
appreciated (preferably the second method, as ivy seems way too
complicated and frustrating to deal with just for this little issue...
unless it's something simple).

Christopher

From general-return-1185-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 10 23:44:06 2010
Return-Path: <general-return-1185-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 61353 invoked from network); 10 Mar 2010 23:44:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Mar 2010 23:44:06 -0000
Received: (qmail 3638 invoked by uid 500); 10 Mar 2010 23:43:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3611 invoked by uid 500); 10 Mar 2010 23:43:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3603 invoked by uid 99); 10 Mar 2010 23:43:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Mar 2010 23:43:33 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ctubbsii@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Mar 2010 23:43:31 +0000
Received: by wyi11 with SMTP id 11so435700wyi.35
        for <general@hadoop.apache.org>; Wed, 10 Mar 2010 15:43:10 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=u2b6ee7z6D1eVDDXf5OnUetCiF1wfoZVYHhaEw2Qc1I=;
        b=kjfWdMuphvBW29kybr+miyME71qX13CaxuaLJM6KifAIGxtFSwYsWtqW3/o36MDj6O
         Q+Ju9pWi0aP85kltQlnHvNJB6QCN2xRdROENg6oZ/WNVmFQKF8GbIRKBdRpKCT6gkJAA
         E//Xbdj55hLEhCUW3y34lqAgD7gv9g1O5lvds=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=sloj3AnFZ5H4K3Y9Z0dSNHbSNJlGqtr6DTnnhVlHVPcRUBFcOUBhJOGymQGIoU8EQg
         F3jZpAQiTrs2SysDX0BHBkAkc5rNRwqgSscG1JdJ3x2fpLA0QYwzqXzab8bXPQgvY1hN
         8B+t6+qrRkJLCYJOpuGxDWZA6+zdMo3yFyHV8=
MIME-Version: 1.0
Received: by 10.216.86.67 with SMTP id v45mr1238516wee.70.1268264590431; Wed, 
	10 Mar 2010 15:43:10 -0800 (PST)
In-Reply-To: <9db00c201003092045k31990da2j587a9e92235cc7c4@mail.gmail.com>
References: <9db00c201003092045k31990da2j587a9e92235cc7c4@mail.gmail.com>
Date: Wed, 10 Mar 2010 18:43:10 -0500
Message-ID: <9db00c201003101543y45dcce4fh6076f4838d6250fb@mail.gmail.com>
Subject: Re: Native libraries on Mac
From: Christopher Tubbs <ctubbsii@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

A follow up:

I've successfully figured out how to point ivy to my maven repository
by editing the ivy/ivysettings.xml file, and have successfully built
the binaries. However, now I can't figure out how to load the
libraries.

I try to run a simple java program that calls
'System.loadLibrary("hadoop");' from the command line with the library
path set:
java -Djava.library.path=$HADOOP_HOME/lib/native/Mac_OS_X-x86_64-64 TestClass

But I get a stack trace saying it can't find the library!

Perhaps this is related to a bug reported on the common-user mailing list:
http://www.mail-archive.com/common-user@hadoop.apache.org/msg04768.html

If anybody has any ideas or a patch to fix the 64 bit build on Mac or
whatever, I'd greatly appreciate it.

Thanks,

Christopher


On Tue, Mar 9, 2010 at 11:45 PM, Christopher Tubbs <ctubbsii@gmail.com> wrote:
> I'm trying to build the native libraries for hadoop on Mac OS X
> Leopard (10.5.8) for Hadoop 0.20.1
>
> I tried the "ant compile-native" method, but I don't have access to
> the internet. I do have a local repository though with all the
> dependencies (not to mention they are in the $HADOOP_HOME/lib). I was
> able to edit the build.xml file to point the repository to my local
> one so I could download ivy, but now ivy can't find the dependencies,
> and I can't figure out how to point ivy to my maven repository.
>
> Since the first method didn't work, I tried applying the patch in
> HADOOP-3659 (has that been included in 0.20.2?), and got things
> working with a little messing with aclocal and friends, but when I run
> make, I get a ZLibCompressor.lo, missing argument for "-m" error in
> the build output before it quits.
>
> Any help on building the native libs by either method would be greatly
> appreciated (preferably the second method, as ivy seems way too
> complicated and frustrating to deal with just for this little issue...
> unless it's something simple).
>
> Christopher
>

From general-return-1186-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 11 02:13:20 2010
Return-Path: <general-return-1186-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1614 invoked from network); 11 Mar 2010 02:13:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Mar 2010 02:13:20 -0000
Received: (qmail 14391 invoked by uid 500); 11 Mar 2010 02:12:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14349 invoked by uid 500); 11 Mar 2010 02:12:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14341 invoked by uid 99); 11 Mar 2010 02:12:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Mar 2010 02:12:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 11 Mar 2010 02:12:45 +0000
Received: (qmail 1436 invoked by uid 99); 11 Mar 2010 02:12:55 -0000
Received: from localhost.apache.org (HELO mail-bw0-f211.google.com) (127.0.0.1)
  (smtp-auth username cdouglas, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Mar 2010 02:12:55 +0000
Received: by bwz3 with SMTP id 3so2662977bwz.29
        for <general@hadoop.apache.org>; Wed, 10 Mar 2010 18:12:21 -0800 (PST)
MIME-Version: 1.0
Received: by 10.204.35.11 with SMTP id n11mr1302909bkd.4.1268273541519; Wed, 
	10 Mar 2010 18:12:21 -0800 (PST)
In-Reply-To: <8f8e14c41003020418s1753b966x9279bc1d457b53b@mail.gmail.com>
References: <1267dd3b1003011331x13ee1070gf081ce0c58f549e8@mail.gmail.com>
	 <8f8e14c41003020418s1753b966x9279bc1d457b53b@mail.gmail.com>
Date: Wed, 10 Mar 2010 18:12:21 -0800
Message-ID: <1267dd3b1003101812s60ded730h5e6627e217281a3@mail.gmail.com>
Subject: Re: Hadoop 0.20.2 is released
From: Chris Douglas <cdouglas@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hadoop 0.20.2 has been pushed to repository.apache.org and members of
the Mahout community report that it is working.

***NOTE***
Users of the repository should be aware that the naming convention
enforced by Maven requires that the .jar names *differ* from those
produced by the 0.20.2 build. For example, instead of
hadoop-0.20.2-core.jar, one receives hadoop-core-0.20.2.jar from the
repo. The Maven naming convention will be used for all artifacts in
0.20.3 and later. To avoid confusion in downstream projects, I would
like to roll a 0.20.3 very soon, in the next few weeks if possible.

HADOOP-6382 will be applied to 0.20.3 presently. Direct all thanks to
Giri for his excellent work on this issue. -C

On Tue, Mar 2, 2010 at 4:18 AM, Drew Farris <drew.farris@gmail.com> wrote:
> Congratulations, and thanks for the 0.20.2 release.
>
> I can see the release on the mirrors, but can someone also push it to
> the repo at repository.apache.org?
>
> Thanks,
> Drew
>
> On Mon, Mar 1, 2010 at 4:31 PM, Chris Douglas <cdouglas@apache.org> wrote:
>> Hadoop 0.20.2 is released and propagating to mirrors. The list of
>> changes may be found in the release notes:
>>
>> http://bit.ly/dd6rwL
>>
>> And in JIRA:
>>
>> COMMON: http://bit.ly/9fDo2f
>> MAPREDUCE: http://bit.ly/aD0BM5
>> HDFS: http://bit.ly/ckvMfO
>>
>> Thanks to everyone who worked on this release. -C
>>
>

From general-return-1187-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 11 14:10:24 2010
Return-Path: <general-return-1187-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 7050 invoked from network); 11 Mar 2010 14:10:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Mar 2010 14:10:24 -0000
Received: (qmail 98207 invoked by uid 500); 11 Mar 2010 14:09:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98152 invoked by uid 500); 11 Mar 2010 14:09:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98141 invoked by uid 99); 11 Mar 2010 14:09:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Mar 2010 14:09:49 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [145.100.8.32] (HELO planck.ka.sara.nl) (145.100.8.32)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Mar 2010 14:09:46 +0000
Received: from planck.ka.sara.nl ([145.100.8.32]) by planck.ka.sara.nl
 ([145.100.8.32]) with mapi; Thu, 11 Mar 2010 15:09:23 +0100
From: Evert Lammerts <Evert.Lammerts@sara.nl>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 11 Mar 2010 15:09:22 +0100
Subject: Compile error
Thread-Topic: Compile error
Thread-Index: AcrBJHKgmIJXALbBSKqwImz/5TiqAg==
Message-ID: <ADF94D8555C7A246B86A633685E0178AB86F2602D6@planck.ka.sara.nl>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/signed; protocol="application/x-pkcs7-signature";
	micalg=SHA1; boundary="----=_NextPart_000_018C_01CAC12C.D46C7580"
MIME-Version: 1.0

------=_NextPart_000_018C_01CAC12C.D46C7580
Content-Type: multipart/alternative;
	boundary="----=_NextPart_001_018D_01CAC12C.D46C7580"


------=_NextPart_001_018D_01CAC12C.D46C7580
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit

Hi list,

 

I remember seeing a mail about this already, but I can't find it back in the
archives - maybe I remembered wrong.

 

When I try to compile hadoop-core 0.20.2 I'm getting a SocketException while
ant is trying to download Ivy (see below). Apparently it thinks there's no
connection, but a ping and wget proves otherwise. What am I / is it doing
wrong here?

 

Best regards,

Evert Lammerts

 

Buildfile: /home/hadoop/opt/code/hadoop/hadoop-0.20.2/build.xml

 

ivy-download:

      [get] Getting:
http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.0.0-rc2/ivy-2.0.0-rc2.jar

      [get] To:
/home/hadoop/opt/code/hadoop/hadoop-0.20.2/ivy/ivy-2.0.0-rc2.jar

      [get] Error getting
http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.0.0-rc2/ivy-2.0.0-rc2.jar
to /home/hadoop/opt/code/hadoop/hadoop-0.20.2/ivy/ivy-2.0.0-rc2.jar

 

BUILD FAILED

/home/hadoop/opt/code/hadoop/hadoop-0.20.2/build.xml:1630:
java.net.SocketException: Network is unreachable

        at java.net.PlainSocketImpl.socketConnect(Native Method)

        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)

        at
java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)

        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)

        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)

        at java.net.Socket.connect(Socket.java:525)

        at java.net.Socket.connect(Socket.java:475)

        at sun.net.NetworkClient.doConnect(NetworkClient.java:163)

        at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)

        at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)

        at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)

        at sun.net.www.http.HttpClient.New(HttpClient.java:306)

        at sun.net.www.http.HttpClient.New(HttpClient.java:323)

        at
sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnecti
on.java:860)

        at
sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.j
ava:801)

        at
sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:7
26)

        at
org.apache.tools.ant.taskdefs.Get$GetThread.openConnection(Get.java:661)

        at org.apache.tools.ant.taskdefs.Get$GetThread.get(Get.java:581)

        at org.apache.tools.ant.taskdefs.Get$GetThread.run(Get.java:571)

 

Evert Lammerts

Adviseur

SARA Computing & Network Services 

High Performance Computing & Visualization

eScience Support Group

 

Phone: +31 20 888 4101

Email: evert.lammerts@sara.nl

 


------=_NextPart_001_018D_01CAC12C.D46C7580
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" =
xmlns=3D"http://www.w3.org/TR/REC-html40">

<head>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<meta name=3DGenerator content=3D"Microsoft Word 12 (filtered medium)">
<style>
<!--
 /* Font Definitions */
 @font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
	{font-family:"Malgun Gothic";
	panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
	{font-family:"\@Malgun Gothic";}
 /* Style Definitions */
 p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0cm;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:purple;
	text-decoration:underline;}
span.EmailStyle17
	{mso-style-type:personal-compose;
	font-family:"Calibri","sans-serif";
	color:windowtext;}
.MsoChpDefault
	{mso-style-type:export-only;}
@page Section1
	{size:612.0pt 792.0pt;
	margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.Section1
	{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
 <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
 <o:shapelayout v:ext=3D"edit">
  <o:idmap v:ext=3D"edit" data=3D"1" />
 </o:shapelayout></xml><![endif]-->
</head>

<body lang=3DEN-US link=3Dblue vlink=3Dpurple>

<div class=3DSection1>

<p class=3DMsoNormal>Hi list,<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>I remember seeing a mail about this already, but I =
can&#8217;t
find it back in the archives &#8211; maybe I remembered =
wrong.<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>When I try to compile hadoop-core 0.20.2 I&#8217;m =
getting a
SocketException while ant is trying to download Ivy (see below). =
Apparently it
thinks there&#8217;s no connection, but a ping and wget proves =
otherwise. What
am I / is it doing wrong here?<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>Best regards,<o:p></o:p></p>

<p class=3DMsoNormal>Evert Lammerts<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>Buildfile:
/home/hadoop/opt/code/hadoop/hadoop-0.20.2/build.xml<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>ivy-download:<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; [get] Getting:
http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.0.0-rc2/ivy-2.0.0-rc2.=
jar<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; [get] To:
/home/hadoop/opt/code/hadoop/hadoop-0.20.2/ivy/ivy-2.0.0-rc2.jar<o:p></o:=
p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; [get] Error getting
http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.0.0-rc2/ivy-2.0.0-rc2.=
jar to
/home/hadoop/opt/code/hadoop/hadoop-0.20.2/ivy/ivy-2.0.0-rc2.jar<o:p></o:=
p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>BUILD FAILED<o:p></o:p></p>

<p =
class=3DMsoNormal>/home/hadoop/opt/code/hadoop/hadoop-0.20.2/build.xml:16=
30:
java.net.SocketException: Network is unreachable<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at =
java.net.PlainSocketImpl.socketConnect(Native
Method)<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)<o:p></o:p></=
p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)<o:p><=
/o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)<o:p></o:p></p>=


<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)<o:p></o:p></p>=


<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at =
java.net.Socket.connect(Socket.java:525)<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at =
java.net.Socket.connect(Socket.java:475)<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at =
sun.net.NetworkClient.doConnect(NetworkClient.java:163)<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
sun.net.www.http.HttpClient.openServer(HttpClient.java:394)<o:p></o:p></p=
>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
sun.net.www.http.HttpClient.openServer(HttpClient.java:529)<o:p></o:p></p=
>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
sun.net.www.http.HttpClient.&lt;init&gt;(HttpClient.java:233)<o:p></o:p><=
/p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
sun.net.www.http.HttpClient.New(HttpClient.java:306)<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
sun.net.www.http.HttpClient.New(HttpClient.java:323)<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConne=
ction.java:860)<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at =
sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnectio=
n.java:801)<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.jav=
a:726)<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
org.apache.tools.ant.taskdefs.Get$GetThread.openConnection(Get.java:661)<=
o:p></o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;at
org.apache.tools.ant.taskdefs.Get$GetThread.get(Get.java:581)<o:p></o:p><=
/p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
org.apache.tools.ant.taskdefs.Get$GetThread.run(Get.java:571)<o:p></o:p><=
/p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal><span =
style=3D'font-size:10.0pt;font-family:"Courier New"'>Evert
Lammerts<o:p></o:p></span></p>

<p class=3DMsoNormal><span =
style=3D'font-size:10.0pt;font-family:"Courier =
New"'>Adviseur<o:p></o:p></span></p>

<p class=3DMsoNormal><span =
style=3D'font-size:10.0pt;font-family:"Courier New"'>SARA
Computing &amp; Network Services <o:p></o:p></span></p>

<p class=3DMsoNormal><span =
style=3D'font-size:10.0pt;font-family:"Courier New"'>High
Performance Computing &amp; Visualization<o:p></o:p></span></p>

<p class=3DMsoNormal><span =
style=3D'font-size:10.0pt;font-family:"Courier New"'>eScience
Support Group<o:p></o:p></span></p>

<p class=3DMsoNormal><span =
style=3D'font-size:10.0pt;font-family:"Courier =
New"'><o:p>&nbsp;</o:p></span></p>

<p class=3DMsoNormal><span =
style=3D'font-size:10.0pt;font-family:"Courier New"'>Phone:
+31 20 888 4101<o:p></o:p></span></p>

<p class=3DMsoNormal><span =
style=3D'font-size:10.0pt;font-family:"Courier New"'>Email:
evert.lammerts@sara.nl<o:p></o:p></span></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

</div>

</body>

</html>

------=_NextPart_001_018D_01CAC12C.D46C7580--

------=_NextPart_000_018C_01CAC12C.D46C7580
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIPUDCCBDIw
ggMaoAMCAQICAQEwDQYJKoZIhvcNAQEFBQAwezELMAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0
ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0
ZWQxITAfBgNVBAMMGEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczAeFw0wNDAxMDEwMDAwMDBaFw0y
ODEyMzEyMzU5NTlaMHsxCzAJBgNVBAYTAkdCMRswGQYDVQQIDBJHcmVhdGVyIE1hbmNoZXN0ZXIx
EDAOBgNVBAcMB1NhbGZvcmQxGjAYBgNVBAoMEUNvbW9kbyBDQSBMaW1pdGVkMSEwHwYDVQQDDBhB
QUEgQ2VydGlmaWNhdGUgU2VydmljZXMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC+
QJ30buHqdoccTUVEjr5GyIMGncEq/hgfjuQC+vOrXVCKFjELmgbQxXAizUktVGPMtm5oRgtT6stM
JMC8ck7q8RWu9FSaEgrDerIzYOLaiVXzIljz3tzP74OGooyUT59o8piQRoQnx3a/48w1LIteB2Rl
gsBIsKiR+WGfdiBQqJHHZrXreGIDVvCKGhPqMaMeoJn9OPb2JzJYbwf1a7j7FCuvt6rM1mNfc4za
BZmoOKjLF3g2UazpnvR4Oo3PD9lC4pgMqy+fDgHe75+ZSfEt36x0TRuYtUfF5SnR+ZAYx2KcvoPH
Jns+iiXHwN2d5jVoECCdj9je0sOEnA1e6C/JAgMBAAGjgcAwgb0wHQYDVR0OBBYEFKARCiM+lvEH
7OKvKe+CpX/QMKS0MA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MHsGA1UdHwR0MHIw
OKA2oDSGMmh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0FBQUNlcnRpZmljYXRlU2VydmljZXMuY3Js
MDagNKAyhjBodHRwOi8vY3JsLmNvbW9kby5uZXQvQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmww
DQYJKoZIhvcNAQEFBQADggEBAAhW/ALwm+j/pPrWe8ZEgM5PxMX2AFjMpra8FEloBHbo5u5d7AIP
YNaNUBhPJk4B4+awpe6/vHRUQb/9/BK4x09a9IlgBX9gtwVK8/bxwr/EuXSGti19a8zS80bdL8bg
asPDNAMsfZbdWsIOpwqZwQWLqwwv81w6z2w3VQmH3lNAbFjv/LarZW4E9hvcPOBaFcae2fFZSDAh
ZQNs7Okhc+ybA6HgN62gFRiP+roCzqcsqRATLNTlCCarIpdg+JBedNSimlO98qlo4KJuwtdssaMP
nr/raOdW8q7y4ys4OgmBtWuF174t7T8at7Jj4vViLILUagBBUPE5g5+V6TaWmG4wggTdMIIDxaAD
AgECAhBxkvvmGV+sTRKFdHE0ohinMA0GCSqGSIb3DQEBBQUAMHsxCzAJBgNVBAYTAkdCMRswGQYD
VQQIDBJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcMB1NhbGZvcmQxGjAYBgNVBAoMEUNvbW9k
byBDQSBMaW1pdGVkMSEwHwYDVQQDDBhBQUEgQ2VydGlmaWNhdGUgU2VydmljZXMwHhcNMDQwMTAx
MDAwMDAwWhcNMjgxMjMxMjM1OTU5WjCBrjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYD
VQQHEw5TYWx0IExha2UgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYD
VQQLExhodHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1VU0VSRmlyc3QtQ2xp
ZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBFbWFpbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBALI5haTyfatBO2JGN67NwWB1vDll+UoaR6K5zEjMapjVTTUZuaRC5c5J4oovHnzSMQfHTrSD
ZJ0uKdWiZMSFvYVRNXmkTmiQexx6pJKoF/KYFfKTzMmkMpW7DE8wvZigC4vlbhuiRvp4vKJvq1le
pS/Pytptqi/rrKGzaqq3Lmc1i3nhHmmI4uZGzaCl6r4LznY6eg6b6vzaJ1s9cx8i5khhxkzzabGo
Lhu21DEgLLyCio6kDqXXiUP8FlqvHXHXEVnauocNr/rz4cLwpMVnjNbWVDreCqS6A3ezZcj9HtN0
YqoYymiTHqGFfvVHZcv4TVcodNI0/zC27vZiMBSMLOsCAwEAAaOCAScwggEjMB8GA1UdIwQYMBaA
FKARCiM+lvEH7OKvKe+CpX/QMKS0MB0GA1UdDgQWBBSJgmd9xJ0mcABLtFBIfN49rgRufTAOBgNV
HQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUH
AwQwEQYDVR0gBAowCDAGBgRVHSAAMHsGA1UdHwR0MHIwOKA2oDSGMmh0dHA6Ly9jcmwuY29tb2Rv
Y2EuY29tL0FBQUNlcnRpZmljYXRlU2VydmljZXMuY3JsMDagNKAyhjBodHRwOi8vY3JsLmNvbW9k
by5uZXQvQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwEQYJYIZIAYb4QgEBBAQDAgEGMA0GCSqG
SIb3DQEBBQUAA4IBAQCdlcs8uH6lCcQevwvCx3aOOTyUxhCqTwzJ4KuEXYlU4GU7820cfDcsJVRf
liH8N4SRnRXcFE+Bz1Qda2xFYMct+ZdRTPlmyjyggoymyPDi6dRK+ew/VsnddozDggFPbADzHhph
dARHA6nGQFeRvGUixSdnT1fbZFrZjR+6hi/0Bq6cae3p9M8pF9jgSp8aIC+XTFG7RgfEijdOIOMJ
MWjHnsSLneh+EbwyaBCWEZhE2CpRYE2I63Q630MGMsg5Vow6EVLTQaRDA/Tt7zMn2zngFE4mydj1
OeKJuJNdtykmQeqzm66D/Hd1yujKtf7iZUpjPkTE0MNeh3OpmByvfxV/MIIGNTCCBR2gAwIBAgIR
APUHLR99vIaqc6YYFoh++L8wDQYJKoZIhvcNAQEFBQAwga4xCzAJBgNVBAYTAlVTMQswCQYDVQQI
EwJVVDEXMBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNVBAoTFVRoZSBVU0VSVFJVU1QgTmV0
d29yazEhMB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3QuY29tMTYwNAYDVQQDEy1VVE4tVVNF
UkZpcnN0LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1haWwwHhcNMTAwMjA4MDAwMDAwWhcN
MTEwMjA4MjM1OTU5WjCB4DE1MDMGA1UECxMsQ29tb2RvIFRydXN0IE5ldHdvcmsgLSBQRVJTT05B
IE5PVCBWQUxJREFURUQxRjBEBgNVBAsTPVRlcm1zIGFuZCBDb25kaXRpb25zIG9mIHVzZTogaHR0
cDovL3d3dy5jb21vZG8ubmV0L3JlcG9zaXRvcnkxHzAdBgNVBAsTFihjKTIwMDMgQ29tb2RvIExp
bWl0ZWQxFzAVBgNVBAMTDkV2ZXJ0IExhbW1lcnRzMSUwIwYJKoZIhvcNAQkBFhZldmVydC5sYW1t
ZXJ0c0BzYXJhLm5sMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtYAj4sXuBqEwPxYY
BMkYLPGrnqT35USIERoWs26xtSY+dLVc7HqaHDZ1lU5m9wFx/Jqph1urlRuAcCj03kGSS23Ta6kt
HTRMYRCHI3nOLfEEbH4POT+wDytORUHNeEKp/PrBn1kmKk+dk+bJI7MNdy2XkiEeO+OqKmLiBGkT
glIxngOeqgqR2IvmVv2hLAyzq8Ax20inwveZsDMa7QdQLSVUaX1kSRSihwz4Jw9X/K+6oA3ZLGp9
pZYnFHBNTHM2DR/C7dNwQgbZS7TY/Jb1kObYNhwD2JzzJlkoe0blR17MeJkF7/+Y414j0AdPFB7I
ggZ4Tm7Maheer9cIQSsvIQIDAQABo4ICGDCCAhQwHwYDVR0jBBgwFoAUiYJnfcSdJnAAS7RQSHze
Pa4Ebn0wHQYDVR0OBBYEFNTyUAqBNg9JXuiPHa9jfLvRM3IyMA4GA1UdDwEB/wQEAwIFoDAMBgNV
HRMBAf8EAjAAMCAGA1UdJQQZMBcGCCsGAQUFBwMEBgsrBgEEAbIxAQMFAjARBglghkgBhvhCAQEE
BAMCBSAwRgYDVR0gBD8wPTA7BgwrBgEEAbIxAQIBAQEwKzApBggrBgEFBQcCARYdaHR0cHM6Ly9z
ZWN1cmUuY29tb2RvLm5ldC9DUFMwgaUGA1UdHwSBnTCBmjBMoEqgSIZGaHR0cDovL2NybC5jb21v
ZG9jYS5jb20vVVROLVVTRVJGaXJzdC1DbGllbnRBdXRoZW50aWNhdGlvbmFuZEVtYWlsLmNybDBK
oEigRoZEaHR0cDovL2NybC5jb21vZG8ubmV0L1VUTi1VU0VSRmlyc3QtQ2xpZW50QXV0aGVudGlj
YXRpb25hbmRFbWFpbC5jcmwwbAYIKwYBBQUHAQEEYDBeMDYGCCsGAQUFBzAChipodHRwOi8vY3J0
LmNvbW9kb2NhLmNvbS9VVE5BQUFDbGllbnRDQS5jcnQwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3Nw
LmNvbW9kb2NhLmNvbTAhBgNVHREEGjAYgRZldmVydC5sYW1tZXJ0c0BzYXJhLm5sMA0GCSqGSIb3
DQEBBQUAA4IBAQBaJvuiBPgqdq9MEYH9dj2vW8hprAIBEnOOvfXdoP604kBhSbj3s/l0ZQyQY35W
YVAltuJQ365uJKigPQHxS+slpUOAJPEVNmwPUIW0GAjxCPxgLdJgAc4jOV8f7oF3pnfI4ukCmjPN
LpaylZzMhFt+t6zDWiMtUqyxK/4on62LoLp2D88lxwxI3q3i00bPlV0wZ+mjzS7vrQGcUgDPWEBL
FD0I3pR+Z1EH1qjelBmBJV9paVzfzmoU93LIDJA7/MEUXd3JiMZGEoVd2qE5qIZO4fNJ+Cyo6tu1
ECSQxoL6qN+sBwekiFA9Q/zlUp0HfU6Jt8WmtZ44SOh67LcmCL4ZMYIEaDCCBGQCAQEwgcQwga4x
CzAJBgNVBAYTAlVTMQswCQYDVQQIEwJVVDEXMBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNV
BAoTFVRoZSBVU0VSVFJVU1QgTmV0d29yazEhMB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3Qu
Y29tMTYwNAYDVQQDEy1VVE4tVVNFUkZpcnN0LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1h
aWwCEQD1By0ffbyGqnOmGBaIfvi/MAkGBSsOAwIaBQCgggJ4MBgGCSqGSIb3DQEJAzELBgkqhkiG
9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTEwMDMxMTE0MDkyMlowIwYJKoZIhvcNAQkEMRYEFHLfpp/3
wn8XtdNvaJmircMEgFqrMGcGCSqGSIb3DQEJDzFaMFgwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwIC
AgCAMA0GCCqGSIb3DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMAcGBSsOAwIaMAoGCCqG
SIb3DQIFMIHVBgkrBgEEAYI3EAQxgccwgcQwga4xCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJVVDEX
MBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNVBAoTFVRoZSBVU0VSVFJVU1QgTmV0d29yazEh
MB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3QuY29tMTYwNAYDVQQDEy1VVE4tVVNFUkZpcnN0
LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1haWwCEQD1By0ffbyGqnOmGBaIfvi/MIHXBgsq
hkiG9w0BCRACCzGBx6CBxDCBrjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5T
YWx0IExha2UgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYDVQQLExho
dHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1VU0VSRmlyc3QtQ2xpZW50IEF1
dGhlbnRpY2F0aW9uIGFuZCBFbWFpbAIRAPUHLR99vIaqc6YYFoh++L8wDQYJKoZIhvcNAQEBBQAE
ggEAsHB8zVgFVT5qvfntuK+uosjsbFtzcNr+hfyQR9ywxy9NoAqCXPfCiI/aDPXOHBKFMED9SrYB
woZEC03je/YyG8Mx6+LmXakBXnXoCALlhr0c4DKBv74yeF9s0sszUAnxIj1QCGaMujp25Rn5viMr
5Jee6GehjZMCsFSgHAlY4OGozWFYANsndrq9TPrnQkqXos4MNeBExXOT8pZKE2DGe+nFPw5QbeYo
7FROUMO/gPJNUWGK9uhKQlZBoVGesLQZlfIygULd6BO6/j0v5nFwOeFMsjIRjsZZhVVJT7DNQwXC
e1GeDuXYmVs5P852+KGif2xuINYPOTZAgIllNlmXqAAAAAAAAA==

------=_NextPart_000_018C_01CAC12C.D46C7580--

From general-return-1188-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 11 16:25:29 2010
Return-Path: <general-return-1188-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 60590 invoked from network); 11 Mar 2010 16:25:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Mar 2010 16:25:28 -0000
Received: (qmail 70212 invoked by uid 500); 11 Mar 2010 16:24:54 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 70158 invoked by uid 500); 11 Mar 2010 16:24:53 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 70150 invoked by uid 99); 11 Mar 2010 16:24:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Mar 2010 16:24:53 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [145.100.8.32] (HELO planck.ka.sara.nl) (145.100.8.32)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Mar 2010 16:24:46 +0000
Received: from planck.ka.sara.nl ([145.100.8.32]) by planck.ka.sara.nl
 ([145.100.8.32]) with mapi; Thu, 11 Mar 2010 17:24:25 +0100
From: Evert Lammerts <Evert.Lammerts@sara.nl>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Thu, 11 Mar 2010 17:24:25 +0100
Subject: RE: Compile error
Thread-Topic: Compile error
Thread-Index: AcrBJHKgmIJXALbBSKqwImz/5TiqAgAEpxeA
Message-ID: <ADF94D8555C7A246B86A633685E0178AB86F2602DC@planck.ka.sara.nl>
References: <ADF94D8555C7A246B86A633685E0178AB86F2602D6@planck.ka.sara.nl>
In-Reply-To: <ADF94D8555C7A246B86A633685E0178AB86F2602D6@planck.ka.sara.nl>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/signed; protocol="application/x-pkcs7-signature";
	micalg=SHA1; boundary="----=_NextPart_000_01E4_01CAC13F.B20F7BA0"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

------=_NextPart_000_01E4_01CAC13F.B20F7BA0
Content-Type: multipart/alternative;
	boundary="----=_NextPart_001_01E5_01CAC13F.B20F7BA0"


------=_NextPart_001_01E5_01CAC13F.B20F7BA0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit

Hi again, my problem is solved. Setting bindv6only to 0 in
/etc/sysctl.d/bindv6only.conf on my debian squeeze installation seems to
have fixed the problem. Sorry for littering the list.

 

Evert Lammerts

 

From: Evert Lammerts [mailto:Evert.Lammerts@sara.nl] 
Sent: donderdag 11 maart 2010 15:09
To: general@hadoop.apache.org
Subject: Compile error

 

Hi list,

 

I remember seeing a mail about this already, but I can't find it back in the
archives - maybe I remembered wrong.

 

When I try to compile hadoop-core 0.20.2 I'm getting a SocketException while
ant is trying to download Ivy (see below). Apparently it thinks there's no
connection, but a ping and wget proves otherwise. What am I / is it doing
wrong here?

 

Best regards,

Evert Lammerts

 

Buildfile: /home/hadoop/opt/code/hadoop/hadoop-0.20.2/build.xml

 

ivy-download:

      [get] Getting:
http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.0.0-rc2/ivy-2.0.0-rc2.jar

      [get] To:
/home/hadoop/opt/code/hadoop/hadoop-0.20.2/ivy/ivy-2.0.0-rc2.jar

      [get] Error getting
http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.0.0-rc2/ivy-2.0.0-rc2.jar
to /home/hadoop/opt/code/hadoop/hadoop-0.20.2/ivy/ivy-2.0.0-rc2.jar

 

BUILD FAILED

/home/hadoop/opt/code/hadoop/hadoop-0.20.2/build.xml:1630:
java.net.SocketException: Network is unreachable

        at java.net.PlainSocketImpl.socketConnect(Native Method)

        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)

        at
java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)

        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)

        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)

        at java.net.Socket.connect(Socket.java:525)

        at java.net.Socket.connect(Socket.java:475)

        at sun.net.NetworkClient.doConnect(NetworkClient.java:163)

        at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)

        at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)

        at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)

        at sun.net.www.http.HttpClient.New(HttpClient.java:306)

        at sun.net.www.http.HttpClient.New(HttpClient.java:323)

        at
sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnecti
on.java:860)

        at
sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.j
ava:801)

        at
sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:7
26)

        at
org.apache.tools.ant.taskdefs.Get$GetThread.openConnection(Get.java:661)

        at org.apache.tools.ant.taskdefs.Get$GetThread.get(Get.java:581)

        at org.apache.tools.ant.taskdefs.Get$GetThread.run(Get.java:571)

 

Evert Lammerts

Adviseur

SARA Computing & Network Services 

High Performance Computing & Visualization

eScience Support Group

 

Phone: +31 20 888 4101

Email: evert.lammerts@sara.nl

 


------=_NextPart_001_01E5_01CAC13F.B20F7BA0
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" =
xmlns=3D"http://www.w3.org/TR/REC-html40">

<head>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<meta name=3DGenerator content=3D"Microsoft Word 12 (filtered medium)">
<style>
<!--
 /* Font Definitions */
 @font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
	{font-family:Tahoma;
	panose-1:2 11 6 4 3 5 4 4 2 4;}
 /* Style Definitions */
 p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0cm;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:purple;
	text-decoration:underline;}
span.EmailStyle17
	{mso-style-type:personal;
	font-family:"Calibri","sans-serif";
	color:windowtext;}
span.EmailStyle18
	{mso-style-type:personal-reply;
	font-family:"Calibri","sans-serif";
	color:#1F497D;}
.MsoChpDefault
	{mso-style-type:export-only;
	font-size:10.0pt;}
@page Section1
	{size:612.0pt 792.0pt;
	margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.Section1
	{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
 <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
 <o:shapelayout v:ext=3D"edit">
  <o:idmap v:ext=3D"edit" data=3D"1" />
 </o:shapelayout></xml><![endif]-->
</head>

<body lang=3DEN-US link=3Dblue vlink=3Dpurple>

<div class=3DSection1>

<p class=3DMsoNormal><span style=3D'color:#1F497D'>Hi again, my problem =
is solved. Setting
bindv6only to 0 in /etc/sysctl.d/bindv6only.conf on my debian squeeze
installation seems to have fixed the problem. Sorry for littering the =
list.<o:p></o:p></span></p>

<p class=3DMsoNormal><span =
style=3D'color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<p class=3DMsoNormal><span style=3D'color:#1F497D'>Evert =
Lammerts<o:p></o:p></span></p>

<p class=3DMsoNormal><span =
style=3D'color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<div style=3D'border:none;border-left:solid blue 1.5pt;padding:0cm 0cm =
0cm 4.0pt'>

<div>

<div style=3D'border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt =
0cm 0cm 0cm'>

<p class=3DMsoNormal><b><span =
style=3D'font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span>=
</b><span
style=3D'font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Evert =
Lammerts
[mailto:Evert.Lammerts@sara.nl] <br>
<b>Sent:</b> donderdag 11 maart 2010 15:09<br>
<b>To:</b> general@hadoop.apache.org<br>
<b>Subject:</b> Compile error<o:p></o:p></span></p>

</div>

</div>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>Hi list,<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>I remember seeing a mail about this already, but I =
can&#8217;t
find it back in the archives &#8211; maybe I remembered =
wrong.<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>When I try to compile hadoop-core 0.20.2 I&#8217;m =
getting a
SocketException while ant is trying to download Ivy (see below). =
Apparently it
thinks there&#8217;s no connection, but a ping and wget proves =
otherwise. What am I /
is it doing wrong here?<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>Best regards,<o:p></o:p></p>

<p class=3DMsoNormal>Evert Lammerts<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>Buildfile:
/home/hadoop/opt/code/hadoop/hadoop-0.20.2/build.xml<o:p></o:p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>ivy-download:<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; [get] Getting:
http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.0.0-rc2/ivy-2.0.0-rc2.=
jar<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; [get] To:
/home/hadoop/opt/code/hadoop/hadoop-0.20.2/ivy/ivy-2.0.0-rc2.jar<o:p></o:=
p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; [get] Error getting
http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.0.0-rc2/ivy-2.0.0-rc2.=
jar to
/home/hadoop/opt/code/hadoop/hadoop-0.20.2/ivy/ivy-2.0.0-rc2.jar<o:p></o:=
p></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal>BUILD FAILED<o:p></o:p></p>

<p =
class=3DMsoNormal>/home/hadoop/opt/code/hadoop/hadoop-0.20.2/build.xml:16=
30:
java.net.SocketException: Network is unreachable<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
java.net.PlainSocketImpl.socketConnect(Native Method)<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)<o:p></o:p></=
p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)<o:p><=
/o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)<o:p></o:p></p>=


<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)<o:p></o:p></p>=


<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
java.net.Socket.connect(Socket.java:525)<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
java.net.Socket.connect(Socket.java:475)<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
sun.net.NetworkClient.doConnect(NetworkClient.java:163)<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
sun.net.www.http.HttpClient.openServer(HttpClient.java:394)<o:p></o:p></p=
>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
sun.net.www.http.HttpClient.openServer(HttpClient.java:529)<o:p></o:p></p=
>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
sun.net.www.http.HttpClient.&lt;init&gt;(HttpClient.java:233)<o:p></o:p><=
/p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
sun.net.www.http.HttpClient.New(HttpClient.java:306)<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
sun.net.www.http.HttpClient.New(HttpClient.java:323)<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConne=
ction.java:860)<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnectio=
n.java:801)<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at =
sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.jav=
a:726)<o:p></o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at
org.apache.tools.ant.taskdefs.Get$GetThread.openConnection(Get.java:661)<=
o:p></o:p></p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;at
org.apache.tools.ant.taskdefs.Get$GetThread.get(Get.java:581)<o:p></o:p><=
/p>

<p class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at =
org.apache.tools.ant.taskdefs.Get$GetThread.run(Get.java:571)<o:p></o:p><=
/p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal><span =
style=3D'font-size:10.0pt;font-family:"Courier New"'>Evert
Lammerts<o:p></o:p></span></p>

<p class=3DMsoNormal><span =
style=3D'font-size:10.0pt;font-family:"Courier =
New"'>Adviseur<o:p></o:p></span></p>

<p class=3DMsoNormal><span =
style=3D'font-size:10.0pt;font-family:"Courier New"'>SARA
Computing &amp; Network Services <o:p></o:p></span></p>

<p class=3DMsoNormal><span =
style=3D'font-size:10.0pt;font-family:"Courier New"'>High
Performance Computing &amp; Visualization<o:p></o:p></span></p>

<p class=3DMsoNormal><span =
style=3D'font-size:10.0pt;font-family:"Courier New"'>eScience
Support Group<o:p></o:p></span></p>

<p class=3DMsoNormal><span =
style=3D'font-size:10.0pt;font-family:"Courier =
New"'><o:p>&nbsp;</o:p></span></p>

<p class=3DMsoNormal><span =
style=3D'font-size:10.0pt;font-family:"Courier New"'>Phone:
+31 20 888 4101<o:p></o:p></span></p>

<p class=3DMsoNormal><span =
style=3D'font-size:10.0pt;font-family:"Courier New"'>Email:
evert.lammerts@sara.nl<o:p></o:p></span></p>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

</div>

</div>

</body>

</html>

------=_NextPart_001_01E5_01CAC13F.B20F7BA0--

------=_NextPart_000_01E4_01CAC13F.B20F7BA0
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIPUDCCBDIw
ggMaoAMCAQICAQEwDQYJKoZIhvcNAQEFBQAwezELMAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0
ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0
ZWQxITAfBgNVBAMMGEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczAeFw0wNDAxMDEwMDAwMDBaFw0y
ODEyMzEyMzU5NTlaMHsxCzAJBgNVBAYTAkdCMRswGQYDVQQIDBJHcmVhdGVyIE1hbmNoZXN0ZXIx
EDAOBgNVBAcMB1NhbGZvcmQxGjAYBgNVBAoMEUNvbW9kbyBDQSBMaW1pdGVkMSEwHwYDVQQDDBhB
QUEgQ2VydGlmaWNhdGUgU2VydmljZXMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC+
QJ30buHqdoccTUVEjr5GyIMGncEq/hgfjuQC+vOrXVCKFjELmgbQxXAizUktVGPMtm5oRgtT6stM
JMC8ck7q8RWu9FSaEgrDerIzYOLaiVXzIljz3tzP74OGooyUT59o8piQRoQnx3a/48w1LIteB2Rl
gsBIsKiR+WGfdiBQqJHHZrXreGIDVvCKGhPqMaMeoJn9OPb2JzJYbwf1a7j7FCuvt6rM1mNfc4za
BZmoOKjLF3g2UazpnvR4Oo3PD9lC4pgMqy+fDgHe75+ZSfEt36x0TRuYtUfF5SnR+ZAYx2KcvoPH
Jns+iiXHwN2d5jVoECCdj9je0sOEnA1e6C/JAgMBAAGjgcAwgb0wHQYDVR0OBBYEFKARCiM+lvEH
7OKvKe+CpX/QMKS0MA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MHsGA1UdHwR0MHIw
OKA2oDSGMmh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0FBQUNlcnRpZmljYXRlU2VydmljZXMuY3Js
MDagNKAyhjBodHRwOi8vY3JsLmNvbW9kby5uZXQvQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmww
DQYJKoZIhvcNAQEFBQADggEBAAhW/ALwm+j/pPrWe8ZEgM5PxMX2AFjMpra8FEloBHbo5u5d7AIP
YNaNUBhPJk4B4+awpe6/vHRUQb/9/BK4x09a9IlgBX9gtwVK8/bxwr/EuXSGti19a8zS80bdL8bg
asPDNAMsfZbdWsIOpwqZwQWLqwwv81w6z2w3VQmH3lNAbFjv/LarZW4E9hvcPOBaFcae2fFZSDAh
ZQNs7Okhc+ybA6HgN62gFRiP+roCzqcsqRATLNTlCCarIpdg+JBedNSimlO98qlo4KJuwtdssaMP
nr/raOdW8q7y4ys4OgmBtWuF174t7T8at7Jj4vViLILUagBBUPE5g5+V6TaWmG4wggTdMIIDxaAD
AgECAhBxkvvmGV+sTRKFdHE0ohinMA0GCSqGSIb3DQEBBQUAMHsxCzAJBgNVBAYTAkdCMRswGQYD
VQQIDBJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcMB1NhbGZvcmQxGjAYBgNVBAoMEUNvbW9k
byBDQSBMaW1pdGVkMSEwHwYDVQQDDBhBQUEgQ2VydGlmaWNhdGUgU2VydmljZXMwHhcNMDQwMTAx
MDAwMDAwWhcNMjgxMjMxMjM1OTU5WjCBrjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYD
VQQHEw5TYWx0IExha2UgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYD
VQQLExhodHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1VU0VSRmlyc3QtQ2xp
ZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBFbWFpbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBALI5haTyfatBO2JGN67NwWB1vDll+UoaR6K5zEjMapjVTTUZuaRC5c5J4oovHnzSMQfHTrSD
ZJ0uKdWiZMSFvYVRNXmkTmiQexx6pJKoF/KYFfKTzMmkMpW7DE8wvZigC4vlbhuiRvp4vKJvq1le
pS/Pytptqi/rrKGzaqq3Lmc1i3nhHmmI4uZGzaCl6r4LznY6eg6b6vzaJ1s9cx8i5khhxkzzabGo
Lhu21DEgLLyCio6kDqXXiUP8FlqvHXHXEVnauocNr/rz4cLwpMVnjNbWVDreCqS6A3ezZcj9HtN0
YqoYymiTHqGFfvVHZcv4TVcodNI0/zC27vZiMBSMLOsCAwEAAaOCAScwggEjMB8GA1UdIwQYMBaA
FKARCiM+lvEH7OKvKe+CpX/QMKS0MB0GA1UdDgQWBBSJgmd9xJ0mcABLtFBIfN49rgRufTAOBgNV
HQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUH
AwQwEQYDVR0gBAowCDAGBgRVHSAAMHsGA1UdHwR0MHIwOKA2oDSGMmh0dHA6Ly9jcmwuY29tb2Rv
Y2EuY29tL0FBQUNlcnRpZmljYXRlU2VydmljZXMuY3JsMDagNKAyhjBodHRwOi8vY3JsLmNvbW9k
by5uZXQvQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwEQYJYIZIAYb4QgEBBAQDAgEGMA0GCSqG
SIb3DQEBBQUAA4IBAQCdlcs8uH6lCcQevwvCx3aOOTyUxhCqTwzJ4KuEXYlU4GU7820cfDcsJVRf
liH8N4SRnRXcFE+Bz1Qda2xFYMct+ZdRTPlmyjyggoymyPDi6dRK+ew/VsnddozDggFPbADzHhph
dARHA6nGQFeRvGUixSdnT1fbZFrZjR+6hi/0Bq6cae3p9M8pF9jgSp8aIC+XTFG7RgfEijdOIOMJ
MWjHnsSLneh+EbwyaBCWEZhE2CpRYE2I63Q630MGMsg5Vow6EVLTQaRDA/Tt7zMn2zngFE4mydj1
OeKJuJNdtykmQeqzm66D/Hd1yujKtf7iZUpjPkTE0MNeh3OpmByvfxV/MIIGNTCCBR2gAwIBAgIR
APUHLR99vIaqc6YYFoh++L8wDQYJKoZIhvcNAQEFBQAwga4xCzAJBgNVBAYTAlVTMQswCQYDVQQI
EwJVVDEXMBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNVBAoTFVRoZSBVU0VSVFJVU1QgTmV0
d29yazEhMB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3QuY29tMTYwNAYDVQQDEy1VVE4tVVNF
UkZpcnN0LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1haWwwHhcNMTAwMjA4MDAwMDAwWhcN
MTEwMjA4MjM1OTU5WjCB4DE1MDMGA1UECxMsQ29tb2RvIFRydXN0IE5ldHdvcmsgLSBQRVJTT05B
IE5PVCBWQUxJREFURUQxRjBEBgNVBAsTPVRlcm1zIGFuZCBDb25kaXRpb25zIG9mIHVzZTogaHR0
cDovL3d3dy5jb21vZG8ubmV0L3JlcG9zaXRvcnkxHzAdBgNVBAsTFihjKTIwMDMgQ29tb2RvIExp
bWl0ZWQxFzAVBgNVBAMTDkV2ZXJ0IExhbW1lcnRzMSUwIwYJKoZIhvcNAQkBFhZldmVydC5sYW1t
ZXJ0c0BzYXJhLm5sMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtYAj4sXuBqEwPxYY
BMkYLPGrnqT35USIERoWs26xtSY+dLVc7HqaHDZ1lU5m9wFx/Jqph1urlRuAcCj03kGSS23Ta6kt
HTRMYRCHI3nOLfEEbH4POT+wDytORUHNeEKp/PrBn1kmKk+dk+bJI7MNdy2XkiEeO+OqKmLiBGkT
glIxngOeqgqR2IvmVv2hLAyzq8Ax20inwveZsDMa7QdQLSVUaX1kSRSihwz4Jw9X/K+6oA3ZLGp9
pZYnFHBNTHM2DR/C7dNwQgbZS7TY/Jb1kObYNhwD2JzzJlkoe0blR17MeJkF7/+Y414j0AdPFB7I
ggZ4Tm7Maheer9cIQSsvIQIDAQABo4ICGDCCAhQwHwYDVR0jBBgwFoAUiYJnfcSdJnAAS7RQSHze
Pa4Ebn0wHQYDVR0OBBYEFNTyUAqBNg9JXuiPHa9jfLvRM3IyMA4GA1UdDwEB/wQEAwIFoDAMBgNV
HRMBAf8EAjAAMCAGA1UdJQQZMBcGCCsGAQUFBwMEBgsrBgEEAbIxAQMFAjARBglghkgBhvhCAQEE
BAMCBSAwRgYDVR0gBD8wPTA7BgwrBgEEAbIxAQIBAQEwKzApBggrBgEFBQcCARYdaHR0cHM6Ly9z
ZWN1cmUuY29tb2RvLm5ldC9DUFMwgaUGA1UdHwSBnTCBmjBMoEqgSIZGaHR0cDovL2NybC5jb21v
ZG9jYS5jb20vVVROLVVTRVJGaXJzdC1DbGllbnRBdXRoZW50aWNhdGlvbmFuZEVtYWlsLmNybDBK
oEigRoZEaHR0cDovL2NybC5jb21vZG8ubmV0L1VUTi1VU0VSRmlyc3QtQ2xpZW50QXV0aGVudGlj
YXRpb25hbmRFbWFpbC5jcmwwbAYIKwYBBQUHAQEEYDBeMDYGCCsGAQUFBzAChipodHRwOi8vY3J0
LmNvbW9kb2NhLmNvbS9VVE5BQUFDbGllbnRDQS5jcnQwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3Nw
LmNvbW9kb2NhLmNvbTAhBgNVHREEGjAYgRZldmVydC5sYW1tZXJ0c0BzYXJhLm5sMA0GCSqGSIb3
DQEBBQUAA4IBAQBaJvuiBPgqdq9MEYH9dj2vW8hprAIBEnOOvfXdoP604kBhSbj3s/l0ZQyQY35W
YVAltuJQ365uJKigPQHxS+slpUOAJPEVNmwPUIW0GAjxCPxgLdJgAc4jOV8f7oF3pnfI4ukCmjPN
LpaylZzMhFt+t6zDWiMtUqyxK/4on62LoLp2D88lxwxI3q3i00bPlV0wZ+mjzS7vrQGcUgDPWEBL
FD0I3pR+Z1EH1qjelBmBJV9paVzfzmoU93LIDJA7/MEUXd3JiMZGEoVd2qE5qIZO4fNJ+Cyo6tu1
ECSQxoL6qN+sBwekiFA9Q/zlUp0HfU6Jt8WmtZ44SOh67LcmCL4ZMYIEaDCCBGQCAQEwgcQwga4x
CzAJBgNVBAYTAlVTMQswCQYDVQQIEwJVVDEXMBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNV
BAoTFVRoZSBVU0VSVFJVU1QgTmV0d29yazEhMB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3Qu
Y29tMTYwNAYDVQQDEy1VVE4tVVNFUkZpcnN0LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1h
aWwCEQD1By0ffbyGqnOmGBaIfvi/MAkGBSsOAwIaBQCgggJ4MBgGCSqGSIb3DQEJAzELBgkqhkiG
9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTEwMDMxMTE2MjQyNFowIwYJKoZIhvcNAQkEMRYEFNzhseZG
edp4Su4e0bvqgAa5NV0aMGcGCSqGSIb3DQEJDzFaMFgwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwIC
AgCAMA0GCCqGSIb3DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMAcGBSsOAwIaMAoGCCqG
SIb3DQIFMIHVBgkrBgEEAYI3EAQxgccwgcQwga4xCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJVVDEX
MBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNVBAoTFVRoZSBVU0VSVFJVU1QgTmV0d29yazEh
MB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3QuY29tMTYwNAYDVQQDEy1VVE4tVVNFUkZpcnN0
LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1haWwCEQD1By0ffbyGqnOmGBaIfvi/MIHXBgsq
hkiG9w0BCRACCzGBx6CBxDCBrjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5T
YWx0IExha2UgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYDVQQLExho
dHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1VU0VSRmlyc3QtQ2xpZW50IEF1
dGhlbnRpY2F0aW9uIGFuZCBFbWFpbAIRAPUHLR99vIaqc6YYFoh++L8wDQYJKoZIhvcNAQEBBQAE
ggEAJiWcunPtmcQ/JP4twaAhLNeP2PuSbIjckruo7UawOGXLhLIINzs9pDxgXJKW6innrWXO+De6
H7G1fVSuRQILOekOxufzDcz3o0s+9udLwMnMTnMC/pNvzap7LwAhTs615ovA0Vy4vBjnvhusJk7t
yDIt5tWe16Xeb4zzOZY3RWj4/INAXpPdj6VUpCnvzgWM0+XAlfg7cQMvNqhTuGJijsDGh7fBU1sz
SgtjFoUeKjGnGjBrHQ+mlgNvJRZMfklAv2quVr/uY64kZpjeTDtGv2pcKQVyyABLvfHtc31biXya
dPPvDSCkboMOkIw6KtOXpho+5OWsVzaZ0EgfITpPHQAAAAAAAA==

------=_NextPart_000_01E4_01CAC13F.B20F7BA0--

From general-return-1189-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 11 18:12:56 2010
Return-Path: <general-return-1189-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67361 invoked from network); 11 Mar 2010 18:12:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Mar 2010 18:12:56 -0000
Received: (qmail 81332 invoked by uid 500); 11 Mar 2010 18:12:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81239 invoked by uid 500); 11 Mar 2010 18:12:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81231 invoked by uid 99); 11 Mar 2010 18:12:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Mar 2010 18:12:21 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ctubbsii@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Mar 2010 18:12:13 +0000
Received: by wyi11 with SMTP id 11so152183wyi.35
        for <general@hadoop.apache.org>; Thu, 11 Mar 2010 10:11:53 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=QSXAS7OKLaGcJxZR8u9yesVZLKng4tGGcOVy0ekyJWM=;
        b=NffTxnqjfiVKCZHHAdlReuejCod/swo4B2gbMYcM2NSl3IqSaxAdjMJDyxSgldPfAK
         rrg4ltdmv5VwfLPdOdumTW63VmL7khMF+OYdAqDF4w8NfNbm79GRPeiekOZSk6/T6azc
         jArsv65/RIxwWmPhZM8oRPPbSlstvUmxb7zq4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=hz88khq5kn8UZH9jGZi6nUoOrvHzM7+ceh/mpuXss9qJc9CRc9qK4ToUPuAIJQVvy4
         kcM0JPhqug8Lwy99WPah7awj7zanCPc7gi/MZLM2JMoLDrE3VASy3uBmXYWnYCrKYndu
         ZmZVr2U37hvi8SlwgM4aFxkalQKlK9tiBE+vk=
MIME-Version: 1.0
Received: by 10.216.182.78 with SMTP id n56mr1165052wem.148.1268331113314; 
	Thu, 11 Mar 2010 10:11:53 -0800 (PST)
In-Reply-To: <9db00c201003092045k31990da2j587a9e92235cc7c4@mail.gmail.com>
References: <9db00c201003092045k31990da2j587a9e92235cc7c4@mail.gmail.com>
Date: Thu, 11 Mar 2010 13:11:53 -0500
Message-ID: <9db00c201003111011r57c1d2e3pdb316522d564f5e7@mail.gmail.com>
Subject: Re: Native libraries on Mac
From: Christopher Tubbs <ctubbsii@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Well, I finally figured it out.

Apparently, the library being generated WAS i386 and not x86_64.
So, you have to set CFLAGS="-arch x86_64" prior to running "ant compile-native"

Think we can get patches into 0.20.3 to fix building on a Mac?

Also, it would be nice if ivy could be preconfigured for grabbing
dependencies from the lib directory of the tarball, instead of
requiring them to be redownloaded.

Please and thanks.

On Tue, Mar 9, 2010 at 11:45 PM, Christopher Tubbs <ctubbsii@gmail.com> wrote:
> I'm trying to build the native libraries for hadoop on Mac OS X
> Leopard (10.5.8) for Hadoop 0.20.1
>
> I tried the "ant compile-native" method, but I don't have access to
> the internet. I do have a local repository though with all the
> dependencies (not to mention they are in the $HADOOP_HOME/lib). I was
> able to edit the build.xml file to point the repository to my local
> one so I could download ivy, but now ivy can't find the dependencies,
> and I can't figure out how to point ivy to my maven repository.
>
> Since the first method didn't work, I tried applying the patch in
> HADOOP-3659 (has that been included in 0.20.2?), and got things
> working with a little messing with aclocal and friends, but when I run
> make, I get a ZLibCompressor.lo, missing argument for "-m" error in
> the build output before it quits.
>
> Any help on building the native libs by either method would be greatly
> appreciated (preferably the second method, as ivy seems way too
> complicated and frustrating to deal with just for this little issue...
> unless it's something simple).
>
> Christopher
>

From general-return-1190-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 11 18:24:30 2010
Return-Path: <general-return-1190-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77522 invoked from network); 11 Mar 2010 18:24:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Mar 2010 18:24:30 -0000
Received: (qmail 8578 invoked by uid 500); 11 Mar 2010 18:23:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8542 invoked by uid 500); 11 Mar 2010 18:23:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8534 invoked by uid 99); 11 Mar 2010 18:23:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Mar 2010 18:23:55 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Mar 2010 18:23:49 +0000
Received: from walkduty-lm.corp.yahoo.com (walkduty-lm.corp.yahoo.com [10.72.104.13])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o2BIN9cB041654
	for <general@hadoop.apache.org>; Thu, 11 Mar 2010 10:23:09 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:
	content-transfer-encoding:mime-version:subject:date:references:x-mailer;
	b=0fndibBElurqkcz7Ek86uUEQ5xeHpAuTtuW9RZ4eBvwBGq3oMsc7KV6Q9OPhxHLJ
Message-Id: <0373489A-94A8-4CAE-AC98-1DF10ECA9D95@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: general@hadoop.apache.org
In-Reply-To: <9db00c201003111011r57c1d2e3pdb316522d564f5e7@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: Native libraries on Mac
Date: Thu, 11 Mar 2010 10:23:09 -0800
References: <9db00c201003092045k31990da2j587a9e92235cc7c4@mail.gmail.com> <9db00c201003111011r57c1d2e3pdb316522d564f5e7@mail.gmail.com>
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

Of course, please open a jira and attach the patch. Thanks!

On Mar 11, 2010, at 10:11 AM, Christopher Tubbs wrote:

> Well, I finally figured it out.
>
> Apparently, the library being generated WAS i386 and not x86_64.
> So, you have to set CFLAGS="-arch x86_64" prior to running "ant  
> compile-native"
>
> Think we can get patches into 0.20.3 to fix building on a Mac?
>
> Also, it would be nice if ivy could be preconfigured for grabbing
> dependencies from the lib directory of the tarball, instead of
> requiring them to be redownloaded.
>
> Please and thanks.
>
> On Tue, Mar 9, 2010 at 11:45 PM, Christopher Tubbs  
> <ctubbsii@gmail.com> wrote:
>> I'm trying to build the native libraries for hadoop on Mac OS X
>> Leopard (10.5.8) for Hadoop 0.20.1
>>
>> I tried the "ant compile-native" method, but I don't have access to
>> the internet. I do have a local repository though with all the
>> dependencies (not to mention they are in the $HADOOP_HOME/lib). I was
>> able to edit the build.xml file to point the repository to my local
>> one so I could download ivy, but now ivy can't find the dependencies,
>> and I can't figure out how to point ivy to my maven repository.
>>
>> Since the first method didn't work, I tried applying the patch in
>> HADOOP-3659 (has that been included in 0.20.2?), and got things
>> working with a little messing with aclocal and friends, but when I  
>> run
>> make, I get a ZLibCompressor.lo, missing argument for "-m" error in
>> the build output before it quits.
>>
>> Any help on building the native libs by either method would be  
>> greatly
>> appreciated (preferably the second method, as ivy seems way too
>> complicated and frustrating to deal with just for this little  
>> issue...
>> unless it's something simple).
>>
>> Christopher
>>


From general-return-1191-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 11 18:31:05 2010
Return-Path: <general-return-1191-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 79617 invoked from network); 11 Mar 2010 18:31:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Mar 2010 18:31:05 -0000
Received: (qmail 20336 invoked by uid 500); 11 Mar 2010 18:30:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20300 invoked by uid 500); 11 Mar 2010 18:30:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 20292 invoked by uid 99); 11 Mar 2010 18:30:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Mar 2010 18:30:30 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Mar 2010 18:30:26 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:User-Agent:Date:Subject:From:To:
   Message-ID:Thread-Topic:Thread-Index:In-Reply-To:
   Mime-version:Content-type:Content-transfer-encoding;
  b=cD6aQi6O6cETB3utdCb81N5gVpIFDZAZ02sTTL98AuPtZc9QMZhQPR3N
   5/FOJTVfC3N1G4SehFg9P0V8lRP+SNxdAltpcvCJFg9JJnv8CAVDrYhRT
   yiPY8SSJnkcVHAi;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1268332226; x=1299868226;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20Native=20libraries=20on=20Mac|Date:=20T
   hu,=2011=20Mar=202010=2010:29:15=20-0800|Message-ID:=20<C
   7BE767B.3A36%awittenauer@linkedin.com>|To:=20<general@had
   oop.apache.org>|Mime-version:=201.0
   |Content-transfer-encoding:=207bit|In-Reply-To:=20<9db00c
   201003111011r57c1d2e3pdb316522d564f5e7@mail.gmail.com>;
  bh=o7NAVsWC+2o9PHTuz4bPBFS59H0aO5Ktf5GWzmT/8Ns=;
  b=eiAiSuymvhOMD0c/7fdvzMTgEpkr8+6POZ+h36WK6hAkM/TTdRebtCTg
   W32wSBiqIcZcfZSUPAoV2uehSU+Xtf5NvXjooXTi746KQVIujDERnmxUK
   /1McpeBl3iSJTZH;
X-IronPort-AV: E=Sophos;i="4.49,621,1262592000"; 
   d="scan'208";a="11320157"
Received: from 172.16.19.141 ([172.16.19.141]) by CORP-MAIL.linkedin.biz ([172.18.46.135]) via Exchange Front-End Server mail-access.linkedin.biz ([172.18.46.133]) with Microsoft Exchange Server HTTP-DAV ;
 Thu, 11 Mar 2010 18:29:16 +0000
User-Agent: Microsoft-Entourage/12.10.0.080409
Date: Thu, 11 Mar 2010 10:29:15 -0800
Subject: Re: Native libraries on Mac
From: Allen Wittenauer <awittenauer@linkedin.com>
To: <general@hadoop.apache.org>
Message-ID: <C7BE767B.3A36%awittenauer@linkedin.com>
Thread-Topic: Native libraries on Mac
Thread-Index: AcrBSMDNQwOsTSADKkGPYlTllEJ4Dg==
In-Reply-To: <9db00c201003111011r57c1d2e3pdb316522d564f5e7@mail.gmail.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit


*nods*

There is a bug with the bitness detection in general (it breaks horribly on
Solaris due to the java executable supporting both). I've started working on
a general patch (it a] asks Java what bitness it is and b] asks java which
JVM lib to link against), but got sidetracked by switching our scheduler
over from fair share to capacity.

I wouldn't be surprised if the issue you are seeing doesn't have the same
root cause.


On 3/11/10 10:11 AM, "Christopher Tubbs" <ctubbsii@gmail.com> wrote:

> Well, I finally figured it out.
> 
> Apparently, the library being generated WAS i386 and not x86_64.
> So, you have to set CFLAGS="-arch x86_64" prior to running "ant
> compile-native"
> 
> Think we can get patches into 0.20.3 to fix building on a Mac?
> 
> Also, it would be nice if ivy could be preconfigured for grabbing
> dependencies from the lib directory of the tarball, instead of
> requiring them to be redownloaded.
> 
> Please and thanks.
> 
> On Tue, Mar 9, 2010 at 11:45 PM, Christopher Tubbs <ctubbsii@gmail.com> wrote:
>> I'm trying to build the native libraries for hadoop on Mac OS X
>> Leopard (10.5.8) for Hadoop 0.20.1
>> 
>> I tried the "ant compile-native" method, but I don't have access to
>> the internet. I do have a local repository though with all the
>> dependencies (not to mention they are in the $HADOOP_HOME/lib). I was
>> able to edit the build.xml file to point the repository to my local
>> one so I could download ivy, but now ivy can't find the dependencies,
>> and I can't figure out how to point ivy to my maven repository.
>> 
>> Since the first method didn't work, I tried applying the patch in
>> HADOOP-3659 (has that been included in 0.20.2?), and got things
>> working with a little messing with aclocal and friends, but when I run
>> make, I get a ZLibCompressor.lo, missing argument for "-m" error in
>> the build output before it quits.
>> 
>> Any help on building the native libs by either method would be greatly
>> appreciated (preferably the second method, as ivy seems way too
>> complicated and frustrating to deal with just for this little issue...
>> unless it's something simple).
>> 
>> Christopher
>> 


From general-return-1192-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 11 18:48:17 2010
Return-Path: <general-return-1192-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 85106 invoked from network); 11 Mar 2010 18:48:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Mar 2010 18:48:17 -0000
Received: (qmail 52734 invoked by uid 500); 11 Mar 2010 18:47:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52707 invoked by uid 500); 11 Mar 2010 18:47:42 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52699 invoked by uid 99); 11 Mar 2010 18:47:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Mar 2010 18:47:42 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Mar 2010 18:47:36 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:User-Agent:Date:Subject:From:To:
   Message-ID:Thread-Topic:Thread-Index:Mime-version:
   Content-type:Content-transfer-encoding;
  b=iPqmA09xDVn1vbHWVXDKjW8v/wZJa/eMbxFyb2YmgbeiFMtrEYg13/rr
   xwcyk2ctJeaiCWhIq7qpJTlhH/IoetMC6GLu1C7j7X9FEZANej+pMtgvg
   wXre7bKAmNSIgGl;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1268333256; x=1299869256;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Commit=20Log=20Hacks=20(=20https://issues.apa
   che.org/jira/browse/HADOOP-6628=0D=0A=20)|Date:=20Thu,=20
   11=20Mar=202010=2010:46:20=20-0800|Message-ID:=20<C7BE7A7
   C.3A42%awittenauer@linkedin.com>|To:=20"general@hadoop.ap
   ache.org"=20<general@hadoop.apache.org>|Mime-version:=201
   .0|Content-transfer-encoding:=207bit;
  bh=tvj4y8Ikh2NFYQMZfXWih8/G8xspA5mM5579sFEEPqQ=;
  b=NpYCG3A0uQe80+BVWTsMZH6RXiX05u0EyhqP/GSAObzPuFsSldVbbINY
   Dv/IRbn5xvoJWzk4vvGNfVqDpbGeV9zX5dn7sfjg8ZD9UrFdGsqZpkPVh
   AMQqWR15UrS/Ktu;
X-IronPort-AV: E=Sophos;i="4.49,621,1262592000"; 
   d="scan'208";a="11320558"
Received: from 172.16.19.141 ([172.16.19.141]) by CORP-MAIL.linkedin.biz ([172.18.46.135]) via Exchange Front-End Server mail-access.linkedin.biz ([172.18.46.133]) with Microsoft Exchange Server HTTP-DAV ;
 Thu, 11 Mar 2010 18:46:20 +0000
User-Agent: Microsoft-Entourage/12.10.0.080409
Date: Thu, 11 Mar 2010 10:46:20 -0800
Subject: Commit Log Hacks ( https://issues.apache.org/jira/browse/HADOOP-6628
 )
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Message-ID: <C7BE7A7C.3A42%awittenauer@linkedin.com>
Thread-Topic: Commit Log Hacks (
 https://issues.apache.org/jira/browse/HADOOP-6628 )
Thread-Index: AcrBSyPAGA1g8gWZ3EqGMBYXauwuNA==
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org


Seeing as how our two non-mainline distributions don't have real release
notes published, I've built two quick and dirty perl scripts (yes perl! Fear
the ops people!) to take the commit log from CDH and the Yahoo! changes
files to spit out Confluence-style wiki output.  Changing it to other wiki
output shouldn't be too difficult.

99% of the time, it gets it correct, but sometimes weirdness happens and it
will take some manual cleaning to get it perfect.

Anyway, I thought these might be handy for other folks to help compare and
contrast the differences between the two.  Now it is pretty easy to see how
divergent a given distribution is from mainline, how hard it will be for
them to rebase onto mainline, etc.  Another interesting data point is
comparing the official bug description and the description in the commit
log....

Have fun.

[Files are attached to https://issues.apache.org/jira/browse/HADOOP-6628.  I
might improve them as we go along, but who knows for sure.  ]


From general-return-1193-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 11 19:05:02 2010
Return-Path: <general-return-1193-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 89245 invoked from network); 11 Mar 2010 19:05:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Mar 2010 19:05:02 -0000
Received: (qmail 78831 invoked by uid 500); 11 Mar 2010 19:04:27 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78790 invoked by uid 500); 11 Mar 2010 19:04:27 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78782 invoked by uid 99); 11 Mar 2010 19:04:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Mar 2010 19:04:27 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ctubbsii@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Mar 2010 19:04:24 +0000
Received: by wyi11 with SMTP id 11so177164wyi.35
        for <general@hadoop.apache.org>; Thu, 11 Mar 2010 11:04:03 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=w/W8MphN4mJb+bpENtrx9pqKpO0Y5YQLBZr92sHK3PY=;
        b=YB+Am2GW8HitA18ICYPnr1fiBFkLTeuCxQtHAVUGDF2bodAvkwj9nwKBFh0EgbsTOZ
         C3HVGjFd/csFIIoli8c/u4FPJ1PkadV+SRMbbub4uzsuhl0jWfcGAZ5ua3k04qW33zyi
         PuCSsfXh+whyCtXK7lDLcIO9GYiFhxK+RTEK0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=fHnUYBB/dyAwxOQNKYiyIeTzNTtBkgEG279luYVF8qJhWiLKcjX7Aw8VLKX2GXK71J
         agOPOhBqyi2/0bhxk/WTOZQz9qpSOo0uxUzJh8xcvVtYq6Ja2U81d9iA9MhkWJO88cYB
         7kIxRG13piQO9MELqjbqU/ro1q56hr85L9M+I=
MIME-Version: 1.0
Received: by 10.216.176.212 with SMTP id b62mr995909wem.179.1268334243319; 
	Thu, 11 Mar 2010 11:04:03 -0800 (PST)
In-Reply-To: <C7BE767B.3A36%awittenauer@linkedin.com>
References: <9db00c201003111011r57c1d2e3pdb316522d564f5e7@mail.gmail.com>
	 <C7BE767B.3A36%awittenauer@linkedin.com>
Date: Thu, 11 Mar 2010 14:04:03 -0500
Message-ID: <9db00c201003111104x26d0dc32o7d07014b012b6c65@mail.gmail.com>
Subject: Re: Native libraries on Mac
From: Christopher Tubbs <ctubbsii@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

For now, I think it would be sufficient to apply HADOOP-3659 and
document the rest.
Later, the build.xml can be patched to set the environment correctly.
I'm not familiar with ivy to determine how to point to the local lib
directory, but I'm sure somebody savvy with ivy could do it
easily....or allow it to read ~/.m2/settings.xml to find the servers.

On Thu, Mar 11, 2010 at 1:29 PM, Allen Wittenauer
<awittenauer@linkedin.com> wrote:
>
> *nods*
>
> There is a bug with the bitness detection in general (it breaks horribly on
> Solaris due to the java executable supporting both). I've started working on
> a general patch (it a] asks Java what bitness it is and b] asks java which
> JVM lib to link against), but got sidetracked by switching our scheduler
> over from fair share to capacity.
>
> I wouldn't be surprised if the issue you are seeing doesn't have the same
> root cause.
>
>
> On 3/11/10 10:11 AM, "Christopher Tubbs" <ctubbsii@gmail.com> wrote:
>
>> Well, I finally figured it out.
>>
>> Apparently, the library being generated WAS i386 and not x86_64.
>> So, you have to set CFLAGS="-arch x86_64" prior to running "ant
>> compile-native"
>>
>> Think we can get patches into 0.20.3 to fix building on a Mac?
>>
>> Also, it would be nice if ivy could be preconfigured for grabbing
>> dependencies from the lib directory of the tarball, instead of
>> requiring them to be redownloaded.
>>
>> Please and thanks.
>>
>> On Tue, Mar 9, 2010 at 11:45 PM, Christopher Tubbs <ctubbsii@gmail.com> wrote:
>>> I'm trying to build the native libraries for hadoop on Mac OS X
>>> Leopard (10.5.8) for Hadoop 0.20.1
>>>
>>> I tried the "ant compile-native" method, but I don't have access to
>>> the internet. I do have a local repository though with all the
>>> dependencies (not to mention they are in the $HADOOP_HOME/lib). I was
>>> able to edit the build.xml file to point the repository to my local
>>> one so I could download ivy, but now ivy can't find the dependencies,
>>> and I can't figure out how to point ivy to my maven repository.
>>>
>>> Since the first method didn't work, I tried applying the patch in
>>> HADOOP-3659 (has that been included in 0.20.2?), and got things
>>> working with a little messing with aclocal and friends, but when I run
>>> make, I get a ZLibCompressor.lo, missing argument for "-m" error in
>>> the build output before it quits.
>>>
>>> Any help on building the native libs by either method would be greatly
>>> appreciated (preferably the second method, as ivy seems way too
>>> complicated and frustrating to deal with just for this little issue...
>>> unless it's something simple).
>>>
>>> Christopher
>>>
>
>

From general-return-1194-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 11 19:48:09 2010
Return-Path: <general-return-1194-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98932 invoked from network); 11 Mar 2010 19:48:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Mar 2010 19:48:09 -0000
Received: (qmail 39961 invoked by uid 500); 11 Mar 2010 19:47:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39917 invoked by uid 500); 11 Mar 2010 19:47:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39909 invoked by uid 99); 11 Mar 2010 19:47:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Mar 2010 19:47:34 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Mar 2010 19:47:31 +0000
Received: from SNV-EXPF01.ds.corp.yahoo.com (snv-expf01.ds.corp.yahoo.com [207.126.227.250])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o2BJkqKW075397
	for <general@hadoop.apache.org>; Thu, 11 Mar 2010 11:46:53 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:user-agent:mime-version:content-type:
	content-class:x-mimeole:subject:date:message-id:in-reply-to:
	x-ms-has-attach:x-ms-tnef-correlator:thread-topic:thread-index:from:to:x-originalarrivaltime;
	b=ggR7MZU5AUZZJAntvY7Z9EPPtWIp6ykqLFhfXPD3M6Ib+2ZHdjLCoBfbfvuvnrUH
Received: from SNV-EXVS06.ds.corp.yahoo.com ([207.126.227.234]) by SNV-EXPF01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Thu, 11 Mar 2010 11:46:52 -0800
Received:  from 10.72.187.241 ([10.72.187.241]) by SNV-EXVS06.ds.corp.yahoo.com ([207.126.227.82]) via Exchange Front-End Server SNV-WEBMAIL.CORP.YAHOO.COM ([207.126.227.60]) with Microsoft Exchange Server HTTP-DAV ; Thu, 11 Mar 2010 19:43:12 +0000
User-Agent: Microsoft-Entourage/12.20.0.090605
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01CAC153.1599624D"
Content-class: urn:content-classes:message
X-MimeOLE: Produced By Microsoft Exchange V6.5
Subject: Re: Native libraries on Mac
Date: Thu, 11 Mar 2010 11:43:12 -0800
Message-ID: <C7BE8367.2EEEE%htang@yahoo-inc.com>
In-Reply-To: <0373489A-94A8-4CAE-AC98-1DF10ECA9D95@yahoo-inc.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Native libraries on Mac
Thread-Index: AcrBUHSGWY0BewTnP0uzk57W1E/GVw==
From: "Hong Tang" <htang@yahoo-inc.com>
To: <general@hadoop.apache.org>
X-OriginalArrivalTime: 11 Mar 2010 19:46:52.0495 (UTC) FILETIME=[98E325F0:01CAC153]

------_=_NextPart_001_01CAC153.1599624D
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Right, this is similar to the instructions on hadoop-gpl-compression FAQ
page: http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ (the last
question).


On 3/11/10 10:23 AM, "Arun C Murthy" <acm@yahoo-inc.com> wrote:

> Of course, please open a jira and attach the patch. Thanks!
>=20
> On Mar 11, 2010, at 10:11 AM, Christopher Tubbs wrote:
>=20
>> Well, I finally figured it out.
>>=20
>> Apparently, the library being generated WAS i386 and not x86_64.
>> So, you have to set CFLAGS=3D"-arch x86_64" prior to running "ant
>> compile-native"
>>=20
>> Think we can get patches into 0.20.3 to fix building on a Mac?
>>=20
>> Also, it would be nice if ivy could be preconfigured for grabbing
>> dependencies from the lib directory of the tarball, instead of
>> requiring them to be redownloaded.
>>=20
>> Please and thanks.
>>=20
>> On Tue, Mar 9, 2010 at 11:45 PM, Christopher Tubbs
>> <ctubbsii@gmail.com> wrote:
>>> I'm trying to build the native libraries for hadoop on Mac OS X
>>> Leopard (10.5.8) for Hadoop 0.20.1
>>>=20
>>> I tried the "ant compile-native" method, but I don't have access to
>>> the internet. I do have a local repository though with all the
>>> dependencies (not to mention they are in the $HADOOP_HOME/lib). I =
was
>>> able to edit the build.xml file to point the repository to my local
>>> one so I could download ivy, but now ivy can't find the =
dependencies,
>>> and I can't figure out how to point ivy to my maven repository.
>>>=20
>>> Since the first method didn't work, I tried applying the patch in
>>> HADOOP-3659 (has that been included in 0.20.2?), and got things
>>> working with a little messing with aclocal and friends, but when I
>>> run
>>> make, I get a ZLibCompressor.lo, missing argument for "-m" error in
>>> the build output before it quits.
>>>=20
>>> Any help on building the native libs by either method would be
>>> greatly
>>> appreciated (preferably the second method, as ivy seems way too
>>> complicated and frustrating to deal with just for this little
>>> issue...
>>> unless it's something simple).
>>>=20
>>> Christopher
>>>=20
>=20


------_=_NextPart_001_01CAC153.1599624D--

From general-return-1195-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 12 00:29:23 2010
Return-Path: <general-return-1195-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 77817 invoked from network); 12 Mar 2010 00:29:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Mar 2010 00:29:23 -0000
Received: (qmail 99178 invoked by uid 500); 12 Mar 2010 00:28:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 99085 invoked by uid 500); 12 Mar 2010 00:28:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 81532 invoked by uid 99); 12 Mar 2010 00:07:33 -0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Date: Thu, 11 Mar 2010 19:07:10 -0500
From: Mohammed A Hassan <mhassanb@gmu.edu>
Subject: hadoop 0.20.2 and Ubuntu 9.04 and 9.10 problem.
To: general@hadoop.apache.org
Message-id: <f688e45721929.4b993f5e@gmu.edu>
MIME-version: 1.0
X-Mailer: Sun Java(tm) System Messenger Express 6.2-2.05 (built Apr 28 2005)
Content-type: text/plain; charset=us-ascii
Content-language: en
Content-transfer-encoding: 7BIT
Content-disposition: inline
X-Accept-Language: en
Priority: normal

Hi,

I am working on Ubuntu 9.04/9.10 and hadoop 0.20.2. Here the problem is that for Pseudo Distributed and Distributed clusters, it takes undefined time (1 minute to several minutes, sometimes more than 10 minutes without any result) to up the live nodes. 
It shows that the number of live nodes is zero in http://localhost:50070/dfshealth.jsp for an uncertain amount of time.

Regards
M. Hassan 

From general-return-1196-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 12 00:36:25 2010
Return-Path: <general-return-1196-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 79966 invoked from network); 12 Mar 2010 00:36:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Mar 2010 00:36:25 -0000
Received: (qmail 3342 invoked by uid 500); 12 Mar 2010 00:35:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3309 invoked by uid 500); 12 Mar 2010 00:35:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 3301 invoked by uid 99); 12 Mar 2010 00:35:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Mar 2010 00:35:49 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.85.222.189] (HELO mail-pz0-f189.google.com) (209.85.222.189)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Mar 2010 00:35:47 +0000
Received: by pzk27 with SMTP id 27so467767pzk.2
        for <general@hadoop.apache.org>; Thu, 11 Mar 2010 16:35:27 -0800 (PST)
MIME-Version: 1.0
Received: by 10.141.107.10 with SMTP id j10mr2239130rvm.282.1268354127149; 
	Thu, 11 Mar 2010 16:35:27 -0800 (PST)
In-Reply-To: <f688e45721929.4b993f5e@gmu.edu>
References: <f688e45721929.4b993f5e@gmu.edu>
From: Todd Lipcon <todd@cloudera.com>
Date: Thu, 11 Mar 2010 16:35:07 -0800
Message-ID: <45f85f71003111635j2940bf83g1c26e0d9d7410d98@mail.gmail.com>
Subject: Re: hadoop 0.20.2 and Ubuntu 9.04 and 9.10 problem.
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd13aac32bc0104818fb5b2

--000e0cd13aac32bc0104818fb5b2
Content-Type: text/plain; charset=ISO-8859-1

Hi,

My guess is that the hosts in your cluster are low on entropy.

If you jstack the daemons as they are starting up, do you see them stuck in
a stace trace inside SecureRandom?

-Todd

On Thu, Mar 11, 2010 at 4:07 PM, Mohammed A Hassan <mhassanb@gmu.edu> wrote:

> Hi,
>
> I am working on Ubuntu 9.04/9.10 and hadoop 0.20.2. Here the problem is
> that for Pseudo Distributed and Distributed clusters, it takes undefined
> time (1 minute to several minutes, sometimes more than 10 minutes without
> any result) to up the live nodes.
> It shows that the number of live nodes is zero in
> http://localhost:50070/dfshealth.jsp for an uncertain amount of time.
>
> Regards
> M. Hassan
>



-- 
Todd Lipcon
Software Engineer, Cloudera

--000e0cd13aac32bc0104818fb5b2--

From general-return-1197-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 12 02:29:15 2010
Return-Path: <general-return-1197-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17593 invoked from network); 12 Mar 2010 02:29:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Mar 2010 02:29:14 -0000
Received: (qmail 87136 invoked by uid 500); 12 Mar 2010 02:28:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 87084 invoked by uid 500); 12 Mar 2010 02:28:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 87076 invoked by uid 99); 12 Mar 2010 02:28:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Mar 2010 02:28:38 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jason.hadoop@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Mar 2010 02:28:32 +0000
Received: by pwi6 with SMTP id 6so397582pwi.35
        for <general@hadoop.apache.org>; Thu, 11 Mar 2010 18:28:10 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=VaAMvDHOm5tgoIt37v4GKcwo38sRqnWa/hQaSwH54b0=;
        b=xgGa3irBaqzsZ3xsuBBN92D1i4g/4BG+68vBdS4ikEA9sABw7qtFXELTC5Xaoff0T7
         1UQ0G3Ws1Sn5M6sCLQdUqsYUvWkVAbRj8/Rx6vJyDaqis69Cr0DVdcPnmt2UMGVQ6IgE
         TjvaFnQFL4YgDuvsbrZmhRlNQRUwcDbwEaTWs=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=wgtJCRpLFEsgEvElcSOn1cLuZSvgFUndiqAG5duaglmVW14C6fe9K0JULBrzd5xM0U
         Ck1vIMuUExd0fB39cfhSL5Si0Ji5/x1hHEYDG+K700yLx6eCFuC0AkxsnAwAho7C/Gfc
         KPs9DCU2o3aisjMfXt7uN5c7a+MZLhS/abf1s=
MIME-Version: 1.0
Received: by 10.115.100.37 with SMTP id c37mr1944883wam.126.1268360890497; 
	Thu, 11 Mar 2010 18:28:10 -0800 (PST)
In-Reply-To: <C7BE8367.2EEEE%htang@yahoo-inc.com>
References: <0373489A-94A8-4CAE-AC98-1DF10ECA9D95@yahoo-inc.com>
	 <C7BE8367.2EEEE%htang@yahoo-inc.com>
Date: Thu, 11 Mar 2010 18:28:10 -0800
Message-ID: <314098691003111828w3fc3e24fx5e89cef26d532fb4@mail.gmail.com>
Subject: Re: Native libraries on Mac
From: Jason Venner <jason.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

It is funny, I just put this together myself, my next step is figuring
out why the libjvm.dynlib doesn't actually define the get a jvm
instance symbol, for libhdfs. Once that is sorted out, I will have
fuse_dfs going on the mac.
The build stuff for solaris and macos is ugly, I have ended up hand
hacking things to make the paths work, and because they are horrible
hacks I don't submit them back.

On Thu, Mar 11, 2010 at 11:43 AM, Hong Tang <htang@yahoo-inc.com> wrote:
> Right, this is similar to the instructions on hadoop-gpl-compression FAQ
> page: http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ (the last
> question).
>
>
> On 3/11/10 10:23 AM, "Arun C Murthy" <acm@yahoo-inc.com> wrote:
>
>> Of course, please open a jira and attach the patch. Thanks!
>>
>> On Mar 11, 2010, at 10:11 AM, Christopher Tubbs wrote:
>>
>>> Well, I finally figured it out.
>>>
>>> Apparently, the library being generated WAS i386 and not x86_64.
>>> So, you have to set CFLAGS="-arch x86_64" prior to running "ant
>>> compile-native"
>>>
>>> Think we can get patches into 0.20.3 to fix building on a Mac?
>>>
>>> Also, it would be nice if ivy could be preconfigured for grabbing
>>> dependencies from the lib directory of the tarball, instead of
>>> requiring them to be redownloaded.
>>>
>>> Please and thanks.
>>>
>>> On Tue, Mar 9, 2010 at 11:45 PM, Christopher Tubbs
>>> <ctubbsii@gmail.com> wrote:
>>>> I'm trying to build the native libraries for hadoop on Mac OS X
>>>> Leopard (10.5.8) for Hadoop 0.20.1
>>>>
>>>> I tried the "ant compile-native" method, but I don't have access to
>>>> the internet. I do have a local repository though with all the
>>>> dependencies (not to mention they are in the $HADOOP_HOME/lib). I was
>>>> able to edit the build.xml file to point the repository to my local
>>>> one so I could download ivy, but now ivy can't find the dependencies,
>>>> and I can't figure out how to point ivy to my maven repository.
>>>>
>>>> Since the first method didn't work, I tried applying the patch in
>>>> HADOOP-3659 (has that been included in 0.20.2?), and got things
>>>> working with a little messing with aclocal and friends, but when I
>>>> run
>>>> make, I get a ZLibCompressor.lo, missing argument for "-m" error in
>>>> the build output before it quits.
>>>>
>>>> Any help on building the native libs by either method would be
>>>> greatly
>>>> appreciated (preferably the second method, as ivy seems way too
>>>> complicated and frustrating to deal with just for this little
>>>> issue...
>>>> unless it's something simple).
>>>>
>>>> Christopher
>>>>
>>
>
>



-- 
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals

From general-return-1198-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 12 04:00:01 2010
Return-Path: <general-return-1198-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 36613 invoked from network); 12 Mar 2010 04:00:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Mar 2010 04:00:01 -0000
Received: (qmail 41895 invoked by uid 500); 12 Mar 2010 03:59:25 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 41651 invoked by uid 500); 12 Mar 2010 03:59:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 41643 invoked by uid 99); 12 Mar 2010 03:59:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Mar 2010 03:59:24 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Mar 2010 03:59:21 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:User-Agent:Date:Subject:From:To:
   Message-ID:Thread-Topic:Thread-Index:In-Reply-To:
   Mime-version:Content-type:Content-transfer-encoding;
  b=W8UD6Cu6FReJpPdV4UGKQseejrm743KrFoyCwBC7s8xONt2Nu69ZMXaZ
   8If/g3QauFJb3NrvIKJsXR4n+ve66pAlqjZjCXA/Oku6vJWmQhDhvm7F6
   lw+yrx5OhE3zRjM;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1268366361; x=1299902361;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20Native=20libraries=20on=20Mac|Date:=20T
   hu,=2011=20Mar=202010=2019:58:55=20-0800|Message-ID:=20<C
   7BEFBFF.3A85%awittenauer@linkedin.com>|To:=20<general@had
   oop.apache.org>|Mime-version:=201.0
   |Content-transfer-encoding:=207bit|In-Reply-To:=20<314098
   691003111828w3fc3e24fx5e89cef26d532fb4@mail.gmail.com>;
  bh=nBTu6xmpcvj8X2rZJ8isQcZtwkBm3sTbzW6QKzIyPdM=;
  b=SoHRFKU1minj4JI/Ik7TYIZSudF2mgwdleSQkArYH9Syc/obiQgUlas7
   s/v+XSGq1Pv5kQLLqzLHAmf32IMxa46I9lRo6/i9ayJMzAxUuMBQMX9jz
   KNOGlc3fJsgOTn4;
X-IronPort-AV: E=Sophos;i="4.49,624,1262592000"; 
   d="scan'208";a="11672869"
Received: from 172.18.36.42 ([172.18.36.42]) by CORP-MAIL.linkedin.biz ([172.18.46.135]) via Exchange Front-End Server mail-access.linkedin.biz ([172.18.46.133]) with Microsoft Exchange Server HTTP-DAV ;
 Fri, 12 Mar 2010 03:58:56 +0000
User-Agent: Microsoft-Entourage/12.10.0.080409
Date: Thu, 11 Mar 2010 19:58:55 -0800
Subject: Re: Native libraries on Mac
From: Allen Wittenauer <awittenauer@linkedin.com>
To: <general@hadoop.apache.org>
Message-ID: <C7BEFBFF.3A85%awittenauer@linkedin.com>
Thread-Topic: Native libraries on Mac
Thread-Index: AcrBmFWr2zqFhPSEz0uE65o/4tafew==
In-Reply-To: <314098691003111828w3fc3e24fx5e89cef26d532fb4@mail.gmail.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit


If you want, I can send you my patches and see if that fixes your issues.

Maybe I'll work on this tomorrow and actually get a JIRA posted, etc. lol



On 3/11/10 6:28 PM, "Jason Venner" <jason.hadoop@gmail.com> wrote:

> It is funny, I just put this together myself, my next step is figuring
> out why the libjvm.dynlib doesn't actually define the get a jvm
> instance symbol, for libhdfs. Once that is sorted out, I will have
> fuse_dfs going on the mac.
> The build stuff for solaris and macos is ugly, I have ended up hand
> hacking things to make the paths work, and because they are horrible
> hacks I don't submit them back.
> 
> On Thu, Mar 11, 2010 at 11:43 AM, Hong Tang <htang@yahoo-inc.com> wrote:
>> Right, this is similar to the instructions on hadoop-gpl-compression FAQ
>> page: http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ (the last
>> question).
>> 
>> 
>> On 3/11/10 10:23 AM, "Arun C Murthy" <acm@yahoo-inc.com> wrote:
>> 
>>> Of course, please open a jira and attach the patch. Thanks!
>>> 
>>> On Mar 11, 2010, at 10:11 AM, Christopher Tubbs wrote:
>>> 
>>>> Well, I finally figured it out.
>>>> 
>>>> Apparently, the library being generated WAS i386 and not x86_64.
>>>> So, you have to set CFLAGS="-arch x86_64" prior to running "ant
>>>> compile-native"
>>>> 
>>>> Think we can get patches into 0.20.3 to fix building on a Mac?
>>>> 
>>>> Also, it would be nice if ivy could be preconfigured for grabbing
>>>> dependencies from the lib directory of the tarball, instead of
>>>> requiring them to be redownloaded.
>>>> 
>>>> Please and thanks.
>>>> 
>>>> On Tue, Mar 9, 2010 at 11:45 PM, Christopher Tubbs
>>>> <ctubbsii@gmail.com> wrote:
>>>>> I'm trying to build the native libraries for hadoop on Mac OS X
>>>>> Leopard (10.5.8) for Hadoop 0.20.1
>>>>> 
>>>>> I tried the "ant compile-native" method, but I don't have access to
>>>>> the internet. I do have a local repository though with all the
>>>>> dependencies (not to mention they are in the $HADOOP_HOME/lib). I was
>>>>> able to edit the build.xml file to point the repository to my local
>>>>> one so I could download ivy, but now ivy can't find the dependencies,
>>>>> and I can't figure out how to point ivy to my maven repository.
>>>>> 
>>>>> Since the first method didn't work, I tried applying the patch in
>>>>> HADOOP-3659 (has that been included in 0.20.2?), and got things
>>>>> working with a little messing with aclocal and friends, but when I
>>>>> run
>>>>> make, I get a ZLibCompressor.lo, missing argument for "-m" error in
>>>>> the build output before it quits.
>>>>> 
>>>>> Any help on building the native libs by either method would be
>>>>> greatly
>>>>> appreciated (preferably the second method, as ivy seems way too
>>>>> complicated and frustrating to deal with just for this little
>>>>> issue...
>>>>> unless it's something simple).
>>>>> 
>>>>> Christopher
>>>>> 
>>> 
>> 
>> 
> 
> 


From general-return-1199-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 12 06:11:25 2010
Return-Path: <general-return-1199-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66499 invoked from network); 12 Mar 2010 06:11:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Mar 2010 06:11:24 -0000
Received: (qmail 32585 invoked by uid 500); 12 Mar 2010 06:10:47 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 32193 invoked by uid 500); 12 Mar 2010 06:10:47 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 32179 invoked by uid 99); 12 Mar 2010 06:10:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Mar 2010 06:10:46 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jason.hadoop@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Mar 2010 06:10:41 +0000
Received: by pwi6 with SMTP id 6so486439pwi.35
        for <general@hadoop.apache.org>; Thu, 11 Mar 2010 22:10:21 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=1FyNr6AZpA3jnj7jQKB/cnguiLc08GaMCKU0mByzroI=;
        b=svrmFaIHfQ40Mo0wKFFwDHSE2aqWv4bp1vIB10ycYk5kcxqm5BQq9OgzsOSnyMk7eo
         xNYtYlpVwddN/Hq7DT0A3Y9QNWqwVx7SdWBJGybykhmOBAplnuKS4SE3u7GGoZ1FYJGt
         zZX7pc3zjOwr9W3QUk6iTDSOVb+bhlwpVM0OE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=MPzjdh625yjLKLMWV1elLTqXJgh4XnaN3qdZsH7XXVqFVMq+aQfQ9qNAbtvRtM6iZz
         JUxSQp84+1Oa5lAGtqT/11YtLY1YloXDFv0ZuHkVkLAVBa+izFsiAuyextsHiwnBf5Fj
         SL55nTuo+rsjKpINjxwhffl9EBmZH5jWGm+ms=
MIME-Version: 1.0
Received: by 10.115.25.5 with SMTP id c5mr1106602waj.171.1268374221497; Thu, 
	11 Mar 2010 22:10:21 -0800 (PST)
In-Reply-To: <C7BEFBFF.3A85%awittenauer@linkedin.com>
References: <314098691003111828w3fc3e24fx5e89cef26d532fb4@mail.gmail.com>
	 <C7BEFBFF.3A85%awittenauer@linkedin.com>
Date: Thu, 11 Mar 2010 22:10:21 -0800
Message-ID: <314098691003112210w3b73bc62n974a59a62b62e301@mail.gmail.com>
Subject: Re: Native libraries on Mac
From: Jason Venner <jason.hadoop@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Thank you Allen.


On Thu, Mar 11, 2010 at 7:58 PM, Allen Wittenauer
<awittenauer@linkedin.com> wrote:
>
> If you want, I can send you my patches and see if that fixes your issues.
>
> Maybe I'll work on this tomorrow and actually get a JIRA posted, etc. lol
>
>
>
> On 3/11/10 6:28 PM, "Jason Venner" <jason.hadoop@gmail.com> wrote:
>
>> It is funny, I just put this together myself, my next step is figuring
>> out why the libjvm.dynlib doesn't actually define the get a jvm
>> instance symbol, for libhdfs. Once that is sorted out, I will have
>> fuse_dfs going on the mac.
>> The build stuff for solaris and macos is ugly, I have ended up hand
>> hacking things to make the paths work, and because they are horrible
>> hacks I don't submit them back.
>>
>> On Thu, Mar 11, 2010 at 11:43 AM, Hong Tang <htang@yahoo-inc.com> wrote:
>>> Right, this is similar to the instructions on hadoop-gpl-compression FAQ
>>> page: http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ (the last
>>> question).
>>>
>>>
>>> On 3/11/10 10:23 AM, "Arun C Murthy" <acm@yahoo-inc.com> wrote:
>>>
>>>> Of course, please open a jira and attach the patch. Thanks!
>>>>
>>>> On Mar 11, 2010, at 10:11 AM, Christopher Tubbs wrote:
>>>>
>>>>> Well, I finally figured it out.
>>>>>
>>>>> Apparently, the library being generated WAS i386 and not x86_64.
>>>>> So, you have to set CFLAGS="-arch x86_64" prior to running "ant
>>>>> compile-native"
>>>>>
>>>>> Think we can get patches into 0.20.3 to fix building on a Mac?
>>>>>
>>>>> Also, it would be nice if ivy could be preconfigured for grabbing
>>>>> dependencies from the lib directory of the tarball, instead of
>>>>> requiring them to be redownloaded.
>>>>>
>>>>> Please and thanks.
>>>>>
>>>>> On Tue, Mar 9, 2010 at 11:45 PM, Christopher Tubbs
>>>>> <ctubbsii@gmail.com> wrote:
>>>>>> I'm trying to build the native libraries for hadoop on Mac OS X
>>>>>> Leopard (10.5.8) for Hadoop 0.20.1
>>>>>>
>>>>>> I tried the "ant compile-native" method, but I don't have access to
>>>>>> the internet. I do have a local repository though with all the
>>>>>> dependencies (not to mention they are in the $HADOOP_HOME/lib). I was
>>>>>> able to edit the build.xml file to point the repository to my local
>>>>>> one so I could download ivy, but now ivy can't find the dependencies,
>>>>>> and I can't figure out how to point ivy to my maven repository.
>>>>>>
>>>>>> Since the first method didn't work, I tried applying the patch in
>>>>>> HADOOP-3659 (has that been included in 0.20.2?), and got things
>>>>>> working with a little messing with aclocal and friends, but when I
>>>>>> run
>>>>>> make, I get a ZLibCompressor.lo, missing argument for "-m" error in
>>>>>> the build output before it quits.
>>>>>>
>>>>>> Any help on building the native libs by either method would be
>>>>>> greatly
>>>>>> appreciated (preferably the second method, as ivy seems way too
>>>>>> complicated and frustrating to deal with just for this little
>>>>>> issue...
>>>>>> unless it's something simple).
>>>>>>
>>>>>> Christopher
>>>>>>
>>>>
>>>
>>>
>>
>>
>
>



-- 
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals

From general-return-1200-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 12 10:13:11 2010
Return-Path: <general-return-1200-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24133 invoked from network); 12 Mar 2010 10:13:10 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Mar 2010 10:13:10 -0000
Received: (qmail 6956 invoked by uid 500); 12 Mar 2010 10:12:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6639 invoked by uid 500); 12 Mar 2010 10:12:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6622 invoked by uid 99); 12 Mar 2010 10:12:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Mar 2010 10:12:32 +0000
X-ASF-Spam-Status: No, hits=-1.6 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Mar 2010 10:12:30 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 6ED0BB7DAC
	for <general@hadoop.apache.org>; Fri, 12 Mar 2010 10:12:08 +0000 (GMT)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id sdNWi5L+hrfL for <general@hadoop.apache.org>;
	Fri, 12 Mar 2010 10:12:07 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id 9CF21B7DE6
	for <general@hadoop.apache.org>; Fri, 12 Mar 2010 10:12:07 +0000 (GMT)
MailScanner-NULL-Check: 1268993515.87407@gzYC4gjtbaPa+CCgbZIbzA
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o2CABsEJ002273
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Fri, 12 Mar 2010 10:11:55 GMT
Message-ID: <4B9A1373.40404@apache.org>
Date: Fri, 12 Mar 2010 10:12:03 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090812)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: hadoop 0.20.2 and Ubuntu 9.04 and 9.10 problem.
References: <f688e45721929.4b993f5e@gmu.edu> <45f85f71003111635j2940bf83g1c26e0d9d7410d98@mail.gmail.com>
In-Reply-To: <45f85f71003111635j2940bf83g1c26e0d9d7410d98@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o2CABsEJ002273
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

Todd Lipcon wrote:
> Hi,
> 
> My guess is that the hosts in your cluster are low on entropy.
> 
> If you jstack the daemons as they are starting up, do you see them stuck in
> a stace trace inside SecureRandom?
> 
> -Todd


Or the machines don't know who they are and are stuck in DNS operations. 
Again a stack trace will show you where things lie

> 
> On Thu, Mar 11, 2010 at 4:07 PM, Mohammed A Hassan <mhassanb@gmu.edu> wrote:
> 
>> Hi,
>>
>> I am working on Ubuntu 9.04/9.10 and hadoop 0.20.2. Here the problem is
>> that for Pseudo Distributed and Distributed clusters, it takes undefined
>> time (1 minute to several minutes, sometimes more than 10 minutes without
>> any result) to up the live nodes.
>> It shows that the number of live nodes is zero in
>> http://localhost:50070/dfshealth.jsp for an uncertain amount of time.
>>
>> Regards
>> M. Hassan
>>
> 
> 
> 


From general-return-1201-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 12 23:24:52 2010
Return-Path: <general-return-1201-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46166 invoked from network); 12 Mar 2010 23:24:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Mar 2010 23:24:51 -0000
Received: (qmail 37843 invoked by uid 500); 12 Mar 2010 23:24:13 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 37651 invoked by uid 500); 12 Mar 2010 23:24:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 37643 invoked by uid 99); 12 Mar 2010 23:24:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Mar 2010 23:24:12 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.178] (HELO mail-px0-f178.google.com) (209.85.216.178)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Mar 2010 23:24:07 +0000
Received: by pxi8 with SMTP id 8so1026612pxi.29
        for <general@hadoop.apache.org>; Fri, 12 Mar 2010 15:23:45 -0800 (PST)
MIME-Version: 1.0
Received: by 10.143.136.2 with SMTP id o2mr3123652wfn.189.1268436225258; Fri, 
	12 Mar 2010 15:23:45 -0800 (PST)
In-Reply-To: <C7BE7A7C.3A42%awittenauer@linkedin.com>
References: <C7BE7A7C.3A42%awittenauer@linkedin.com>
Date: Fri, 12 Mar 2010 15:23:45 -0800
Message-ID: <dfe484f01003121523i7a2d1511r6a56c8c409b911a2@mail.gmail.com>
Subject: Re: Commit Log Hacks ( https://issues.apache.org/jira/browse/HADOOP-6628 
	)
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Hey Allen,

CDH now includes Apache style release notes generated from the change
log. Hope this helps.

http://archive.cloudera.com/cdh/testing/hadoop-0.20.1+169.56.releasenotes.h=
tml

Thanks,
Eli

On Thu, Mar 11, 2010 at 10:46 AM, Allen Wittenauer
<awittenauer@linkedin.com> wrote:
>
> Seeing as how our two non-mainline distributions don't have real release
> notes published, I've built two quick and dirty perl scripts (yes perl! F=
ear
> the ops people!) to take the commit log from CDH and the Yahoo! changes
> files to spit out Confluence-style wiki output. =A0Changing it to other w=
iki
> output shouldn't be too difficult.
>
> 99% of the time, it gets it correct, but sometimes weirdness happens and =
it
> will take some manual cleaning to get it perfect.
>
> Anyway, I thought these might be handy for other folks to help compare an=
d
> contrast the differences between the two. =A0Now it is pretty easy to see=
 how
> divergent a given distribution is from mainline, how hard it will be for
> them to rebase onto mainline, etc. =A0Another interesting data point is
> comparing the official bug description and the description in the commit
> log....
>
> Have fun.
>
> [Files are attached to https://issues.apache.org/jira/browse/HADOOP-6628.=
 =A0I
> might improve them as we go along, but who knows for sure. =A0]
>
>

From general-return-1202-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 13 13:32:00 2010
Return-Path: <general-return-1202-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4851 invoked from network); 13 Mar 2010 13:32:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Mar 2010 13:32:00 -0000
Received: (qmail 25554 invoked by uid 500); 13 Mar 2010 13:31:20 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 25504 invoked by uid 500); 13 Mar 2010 13:31:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 25496 invoked by uid 99); 13 Mar 2010 13:31:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 13 Mar 2010 13:31:19 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_NEUTRAL,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 74.125.82.176 is neither permitted nor denied by domain of oded@legolas-media.com)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 13 Mar 2010 13:31:11 +0000
Received: by wyf19 with SMTP id 19so512308wyf.35
        for <general@hadoop.apache.org>; Sat, 13 Mar 2010 05:30:49 -0800 (PST)
MIME-Version: 1.0
Received: by 10.216.172.137 with SMTP id t9mr514933wel.169.1268487049569; Sat, 
	13 Mar 2010 05:30:49 -0800 (PST)
Date: Sat, 13 Mar 2010 15:30:49 +0200
Message-ID: <1703587b1003130530m3e54d1d6r743a39486cf74f1a@mail.gmail.com>
Subject: Problem with multiple output dirs
From: Oded Rosen <oded@legolas-media.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6587a02fde6b20481aea71a
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e6587a02fde6b20481aea71a
Content-Type: text/plain; charset=ISO-8859-1

Hi,

We get errors when we use MultipleTextOutputFormat to split reduce results
into different dirs on the hdfs.
The errors will occur only when there are more then ~10 different output
dirs. We get no error when using fewer output dirs.
I imagine this may be solvable by changing a configuration parameter.

Notice this class (MultipleTextOutputFormat) is now deprecated (our servers
run hadoop-0.20 but our code is still in 0.18 classes).
Will we get the same errors if we work with 0.20 syntax?

Here's what we get:

org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/user/hadoop/etl_2010-03-13_03-12-03/_temporary/_attempt_201003050309_0154_r_000000_0/20100111/part-00000
could only be replicated to 0 nodes, instead of 1
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1282)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:469)
	at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:966)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:962)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:960)

	at org.apache.hadoop.ipc.Client.call(Client.java:740)
	at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
	at $Proxy1.addBlock(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
	at $Proxy1.addBlock(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2932)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2807)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2087)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2274)


And also:

java.io.IOException: All datanodes 10.21.204.136:50010 are bad. Aborting...
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2542)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2087)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2251)



Thanks,
-- 
Oded

--0016e6587a02fde6b20481aea71a--

From general-return-1203-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 13 16:46:27 2010
Return-Path: <general-return-1203-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38314 invoked from network); 13 Mar 2010 16:46:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Mar 2010 16:46:27 -0000
Received: (qmail 57171 invoked by uid 500); 13 Mar 2010 16:45:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 57117 invoked by uid 500); 13 Mar 2010 16:45:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 57109 invoked by uid 99); 13 Mar 2010 16:45:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 13 Mar 2010 16:45:45 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of magawake@gmail.com designates 209.85.223.197 as permitted sender)
Received: from [209.85.223.197] (HELO mail-iw0-f197.google.com) (209.85.223.197)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 13 Mar 2010 16:45:37 +0000
Received: by iwn35 with SMTP id 35so2338431iwn.2
        for <general@hadoop.apache.org>; Sat, 13 Mar 2010 08:45:16 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=d507N5++er9WWUmDf4QQynne5UeCVsSdH6j7rdA/w8o=;
        b=cCER12MNBDS/AAG6sxmzz4cZnnmkpWlcSSA0YxdgNkQGY7BrerkXB0NAOKuogahWj8
         WPlwPmp0XCBaAB0o8n3Ih+awNmFCYdKE+NV8OUBSneZxfzP58CA8kbrP5O6l1BArPbJY
         2bUloHiz6/3pUajDiohXh1N0Hv4O0w1XO+1K8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=Zp3vwQ9IVOnHMA2fdpfP+Kfi625IM2uDmdRXz177nsz8aQ7ta90OCfCRwibKs8J2x0
         W3aioIkmqZVEfynajLjWgLpdx6St6yJe3pv/mzAlEAu13ra6LMGy1Q5GdPwMux/NVCTY
         nlTAVwETkKslOf4NYNPUx6LXpf415FMGVNl8Q=
MIME-Version: 1.0
Received: by 10.231.174.19 with SMTP id r19mr1066013ibz.15.1268498716682; Sat, 
	13 Mar 2010 08:45:16 -0800 (PST)
Date: Sat, 13 Mar 2010 11:45:16 -0500
Message-ID: <1cbd6f831003130845h243bec01o8aa6eb38678f1828@mail.gmail.com>
Subject: configuration question
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
X-Virus-Checked: Checked by ClamAV on apache.org

I have 4 hosts, hostA, hostB, hostC, hostD.


I plan on hosting my data directory as follows:

hostA:
/data

hostB:
/somedata

hostC:
/data

hostD:
/blahBalh


Is it possible to give a "macro" for each host in the core-site.xml to say:

if (host=hostA):
  use /data as dfs.data.dir

if (host=hostB):
  use /somedata

if (host=hostC):
  use /data

if (host=hostD):
  use /blahBalh


I want to do this because I plan to share the same core-site-xml file

From general-return-1204-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 13 18:54:20 2010
Return-Path: <general-return-1204-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 65220 invoked from network); 13 Mar 2010 18:54:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Mar 2010 18:54:19 -0000
Received: (qmail 20424 invoked by uid 500); 13 Mar 2010 18:53:38 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 20361 invoked by uid 500); 13 Mar 2010 18:53:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 20353 invoked by uid 99); 13 Mar 2010 18:53:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 13 Mar 2010 18:53:37 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [166.84.7.136] (HELO vc136.vc.panix.com) (166.84.7.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 13 Mar 2010 18:53:31 +0000
Received: from localhost (localhost [127.0.0.1])
	by vc136.vc.panix.com (Postfix) with ESMTP id 5FD19DC3B2
	for <general@hadoop.apache.org>; Sat, 13 Mar 2010 13:53:09 -0500 (EST)
Received: from vc136.vc.panix.com ([127.0.0.1])
	by localhost (vc136.vc.panix.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id oJURE-OFIjjZ for <general@hadoop.apache.org>;
	Sat, 13 Mar 2010 13:53:01 -0500 (EST)
Received: from eric-sammers-macbook-pro.local (cpe-66-108-36-45.nyc.res.rr.com [66.108.36.45])
	by vc136.vc.panix.com (Postfix) with ESMTP id D0BA5DC267
	for <general@hadoop.apache.org>; Sat, 13 Mar 2010 13:53:00 -0500 (EST)
Message-ID: <4B9BDF0D.6090101@lifeless.net>
Date: Sat, 13 Mar 2010 13:53:01 -0500
From: Eric Sammer <eric@lifeless.net>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: configuration question
References: <1cbd6f831003130845h243bec01o8aa6eb38678f1828@mail.gmail.com>
In-Reply-To: <1cbd6f831003130845h243bec01o8aa6eb38678f1828@mail.gmail.com>
X-Enigmail-Version: 1.0.1
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

On 3/13/10 11:45 AM, Mag Gam wrote:
> Is it possible to give a "macro" for each host in the core-site.xml to say:
> 
> if (host=hostA):
>   use /data as dfs.data.dir
> 
> if (host=hostB):
>   use /somedata
> 
> if (host=hostC):
>   use /data
> 
> if (host=hostD):
>   use /blahBalh
> 
> 
> I want to do this because I plan to share the same core-site-xml file

Usually what people do is use something like Puppet or cfengine and
build and deploy configurations from a template.

Check out:
http://reductivelabs.com/trac/puppet/wiki/BigPicture

-- 
Eric Sammer
eric@lifeless.net
http://esammer.blogspot.com

From general-return-1205-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 15 19:02:46 2010
Return-Path: <general-return-1205-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 20224 invoked from network); 15 Mar 2010 19:02:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Mar 2010 19:02:46 -0000
Received: (qmail 47686 invoked by uid 500); 15 Mar 2010 19:01:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47631 invoked by uid 500); 15 Mar 2010 19:01:57 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47623 invoked by uid 99); 15 Mar 2010 19:01:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Mar 2010 19:01:57 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Mon, 15 Mar 2010 19:01:55 +0000
Received: (qmail 19469 invoked by uid 99); 15 Mar 2010 19:02:20 -0000
Received: from localhost.apache.org (HELO [192.168.1.110]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Mar 2010 19:02:20 +0000
Message-ID: <4B9E840C.8010508@apache.org>
Date: Mon, 15 Mar 2010 12:01:32 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: [Fwd: [VOTE] Avro release 1.3.1 (rc 0)]
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

FYI, an Avro release vote is in progress on avro-dev@h.a.o.

Doug

-------- Original Message --------
Subject: [VOTE] Avro release 1.3.1 (rc 0)
Date: Fri, 12 Mar 2010 13:43:12 -0800
From: Doug Cutting <cutting@apache.org>
To: avro-dev@hadoop.apache.org

I have created a candidate build for Avro release 1.3.1.

Changes are listed at:

   http://tinyurl.com/avro131

Please download, test, and vote by 16 March.

   http://people.apache.org/~cutting/avro-1.3.1-rc0/

Thanks,

Doug

From general-return-1206-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 15 21:00:13 2010
Return-Path: <general-return-1206-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 65678 invoked from network); 15 Mar 2010 21:00:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Mar 2010 21:00:13 -0000
Received: (qmail 15963 invoked by uid 500); 15 Mar 2010 20:59:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15828 invoked by uid 500); 15 Mar 2010 20:59:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15725 invoked by uid 99); 15 Mar 2010 20:59:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Mar 2010 20:59:21 +0000
X-ASF-Spam-Status: No, hits=-0.2 required=10.0
	tests=AWL,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of zshao9@gmail.com designates 209.85.223.197 as permitted sender)
Received: from [209.85.223.197] (HELO mail-iw0-f197.google.com) (209.85.223.197)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Mar 2010 20:59:16 +0000
Received: by iwn35 with SMTP id 35so176186iwn.2
        for <multiple recipients>; Mon, 15 Mar 2010 13:58:55 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=b/W+jTKB/9dmDKTD7tGaz3esfp+Oqs8bv6VQZuv2GBU=;
        b=de64Z5VIMFbdjiLJ/mavcV6pyVpj7UxmTsh9S512ztYJQ04JfsvxPz9Hihn2l/1Ch6
         RkthL8Kbt4mdqC6Ch56Hil4UHFInJ83fDanwekCnfrEQOuGjdOU75dZ8tLdK8csg/bwS
         FsoONwYADUzQPUod4Ea/xLZ5C7ShYNYy7g0Vs=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=Pm+KV6mq9cfKRBh8aEdQY3iR62jNU4kDRSAN4gAVdFc/tDijwEW6n8DSqu/bxmPl8s
         s2FYz0EAI6rnCzK36WNM9whuZL8hBovAZLOuXWA8PZyY9pNCScsJdJGxjlvGdCn+6kUD
         6quuV6Y1rFkRrDri5d7eO1I7ZgSf4iFebfy4U=
MIME-Version: 1.0
Received: by 10.231.192.131 with SMTP id dq3mr56571ibb.61.1268686734740; Mon, 
	15 Mar 2010 13:58:54 -0700 (PDT)
In-Reply-To: <34fd060d1003011157k7453df1nba040378bb121a75@mail.gmail.com>
References: <34fd060d1002261356u1f37e824x8f879d20ba1ebdb6@mail.gmail.com>
	 <34fd060d1003011157k7453df1nba040378bb121a75@mail.gmail.com>
Date: Mon, 15 Mar 2010 13:58:54 -0700
Message-ID: <34fd060d1003151358o59cd58e4la405349e73a0b810@mail.gmail.com>
Subject: Re: Hive User Group Meeting 3/18/2010 7pm at Facebook
From: Zheng Shao <zshao9@gmail.com>
To: hive-user@hadoop.apache.org, hive-dev@hadoop.apache.org, 
	core-user@hadoop.apache.org, general@hadoop.apache.org, 
	common-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Just a reminder that we have Hive User Group Meeting this Thursday at Faceb=
ook.

Please register on
http://www.meetup.com/Hive-User-Group-Meeting/calendar/12741356/ if
you plan to come.

Zheng

On Mon, Mar 1, 2010 at 12:57 PM, Zheng Shao <zshao9@gmail.com> wrote:
> We also created a Meetup group in case you prefer to register on meetup.c=
om
>
> http://www.meetup.com/Hive-User-Group-Meeting/calendar/12741356/
>
> We are hosting a Hive User Group Meeting, open to all current and
> potential hadoop/hive users.
>
> Agenda:
> * Hive Tutorial (Carl Steinbach, cloudera): 20 min
> * Hive User Case Study (Eva Tse, netflix): 20 min
> * New Features and API (Hive team, Facebook): 25 min
> JDBC/ODBC and CTAS(Create Table As Select)
> UDF/UDAF/UDTF (User-defined Functions)
> Create View/HBaseInputFormat (Hive and HBase integration)
> Hive Join Strategy (How Hive does the join)
> SerDe (Hive's serialization/deserialization framework)
>
>
> Hive is a scalable data warehouse infrastructure built on top of
> Hadoop. It provides tools to enable easy data ETL, a mechanism to put
> structures on the data, and the capability to querying and analysis of
> large data sets stored in Hadoop files. Hive defines a simple SQL-like
> query language, called HiveQL, that enables users familiar with SQL to
> query the data. At the same time, this language also allows
> programmers who are familiar with MapReduce to be able to plug in
> their custom mappers and reducers to perform more sophisticated
> analysis.
>
> The current largest deployment of Hive is the silver cluster at
> Facebook, which consists of 1100 nodes with 8 CPU-cores and 12
> 1TB-disk each. The total capacity is 8800 CPU-cores with 13 PB of raw
> storage space. More than 4 TB of compressed data (20+ TB uncompressed)
> are loaded into Hive every day.
>
>
> If you'd like to network with fellow Hive/Hadoop users online, feel
> free to find them here:
> http://www.facebook.com/event.php?eid=3D319237846974
>
>
>
> Zheng
>
> On Fri, Feb 26, 2010 at 1:56 PM, Zheng Shao <zshao9@gmail.com> wrote:
>> Hi all,
>>
>> We are going to hold the second Hive User Group Meeting at 7PM on
>> 3/18/2010 Thursday.
>>
>> The agenda will be:
>>
>> * Hive Tutorial: 20 min
>> * Hive User Case Study: 20 min
>> * New Features and API: 25 min
>> =A0JDBC/ODBC and CTAS
>> =A0UDF/UDAF/UDTF
>> =A0Create View/HBaseInputFormat
>> =A0Hive Join Strategy
>> =A0SerDe
>>
>> The audience is beginner to intermediate Hive users/developers.
>>
>> *** The details are here: http://www.facebook.com/event.php?eid=3D319237=
846974 ***
>> *** Please RSVP so we can schedule logistics accordingly. ***
>>
>> --
>> Yours,
>> Zheng
>>
>
>
>
> --
> Yours,
> Zheng
>



--=20
Yours,
Zheng

From general-return-1207-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 17 05:30:00 2010
Return-Path: <general-return-1207-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 25608 invoked from network); 17 Mar 2010 05:30:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Mar 2010 05:30:00 -0000
Received: (qmail 6228 invoked by uid 500); 17 Mar 2010 05:29:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6088 invoked by uid 500); 17 Mar 2010 05:29:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6080 invoked by uid 99); 17 Mar 2010 05:29:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Mar 2010 05:29:57 +0000
X-ASF-Spam-Status: No, hits=4.4 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mailinglists19@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Mar 2010 05:29:49 +0000
Received: by wwc33 with SMTP id 33so473104wwc.35
        for <general@hadoop.apache.org>; Tue, 16 Mar 2010 22:29:29 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=h6ue2V/TWpZ11nM+TIfpVEYG+Xq73xgAPakizqTWHlA=;
        b=ZjZSDtWVmHBvIWgu2G0boIMaiztG3Z+kWPZN/v+n2M2/n8c0+H8f2afn1OTQPfFHz5
         3SHyhl7bvxUWiDgtvXb6++0WF5E0wVmUg4PbEvGm2Ezh2ZkcjbmsBBYxxC1AK0e39p+1
         b0ZFnPcJ3b+ailfn8lb2DY2CXEWCfdUH3w3c0=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=n2ZTBZGlEZWimMTYOfjaKKi2g6u4/zeyPr77gg/Fl9iERoe/0+JqQZmNttI6Axxb3y
         kbh5542qOuJz02LkpFzwwE8N0caZ8CP/3KbIhEBGYKoCkVpDOtdG0Ch4H4fpFQySOnzz
         pNl9b65LET9Mt9mDT+Z/7rJeCt3ppHEg+DjJE=
MIME-Version: 1.0
Received: by 10.216.187.148 with SMTP id y20mr172834wem.88.1268803769739; Tue, 
	16 Mar 2010 22:29:29 -0700 (PDT)
Date: Tue, 16 Mar 2010 22:29:29 -0700
Message-ID: <1eabbac31003162229t59389312yf818294187a89d55@mail.gmail.com>
Subject: Retrieving job information from JobTracker
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636833958fc2f380481f865bf
X-Virus-Checked: Checked by ClamAV on apache.org

--001636833958fc2f380481f865bf
Content-Type: text/plain; charset=ISO-8859-1

Is there an easy way to get information about jobs (such as Job status etc)
programatically?  Basically retrieving all the info that's usually available
at:  http://localhost:50030/jobtracker.jsp

Seems like there was a JIRA issue:
http://mail-archives.apache.org/mod_mbox/hadoop-mapreduce-issues/200908.mbox/%3C1692689010.1251507153184.JavaMail.jira@brutus%3E

This one talks about 'XML-based metrics as JSP servlet for JobTracker', but
not sure if anything similar (such as REST API) was implemented.  Can
someone please let me know.  Thanks.

--001636833958fc2f380481f865bf--

From general-return-1208-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 17 06:44:42 2010
Return-Path: <general-return-1208-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 45630 invoked from network); 17 Mar 2010 06:44:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Mar 2010 06:44:42 -0000
Received: (qmail 50411 invoked by uid 500); 17 Mar 2010 06:44:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 49933 invoked by uid 500); 17 Mar 2010 06:44:38 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 49908 invoked by uid 99); 17 Mar 2010 06:44:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Mar 2010 06:44:37 +0000
X-ASF-Spam-Status: No, hits=1.6 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of weliam.cloud@gmail.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Mar 2010 06:44:31 +0000
Received: by gyd8 with SMTP id 8so353772gyd.35
        for <multiple recipients>; Tue, 16 Mar 2010 23:44:10 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:from:date:message-id
         :subject:to:content-type;
        bh=SkFBjhNlTfRDeUrruS354D32jdcJ70xx6Csmld6czi0=;
        b=c9pFeGZc6sxaRYDOGHOrfGXeqG2zY9yoDcKICV5xn5Arj3EguBmJIwAb7gpCxyhEga
         lvcmCFrrSabfDXZZQirUSP/VfghrACM/5JACC2+GR4RuXHMZa7ATQoQIdp2KzHWMWJcq
         6n+nqzmoTKuXs+Mv9Lj2SDxMMEibcxibDKF0I=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:from:date:message-id:subject:to:content-type;
        b=uNCCrkLxlmJipnwTyPCCml6cvhZcfUCFRAFof8jsZWyCASEO/gzj1QJF4ybeG9t1HM
         wwraST27sX0TTNOspYHIJkKMk02MT+2A2PyaB/zIxZWg3GU+1q7Qid9QE3yL77A4v5QS
         WYECrT5GspXr85c3ZD65xk2aDHpApyfMP/MEk=
MIME-Version: 1.0
Received: by 10.90.7.13 with SMTP id 13mr297113agg.75.1268808250120; Tue, 16 
	Mar 2010 23:44:10 -0700 (PDT)
From: William Kang <weliam.cloud@gmail.com>
Date: Wed, 17 Mar 2010 02:43:50 -0400
Message-ID: <ca73442f1003162343x350fe04cj1c3e35efbb773d40@mail.gmail.com>
Subject: Distributed hadoop setup 0 live datanode problem in cluster
To: general@hadoop.apache.org, common-user <common-user@hadoop.apache.org>, 
	core-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016361e7d34095bfe0481f971d8

--0016361e7d34095bfe0481f971d8
Content-Type: text/plain; charset=ISO-8859-1

Hi,
I just moved from pseudo distributed hadoop to a four machine full
distributed hadoop setup.

But, after I start the dfs, there is no live node showing up. If I make
master a slave too, then the datanode in master machine will show up.

I looked up all logs and found no errors. The only thing
looks suspicious  is the log in the datanode:


************************************
2010-03-17 02:39:04,003 INFO org.apache.hadoop.ipc.RPC: Server at
/xx.xx.xx.xx:9000 not available yet, Zzzzz...
2010-03-17 02:39:06,064 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: /xx.xx.xx.xx:9000. Already tried 0 time(s).
2010-03-17 02:39:07,076 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: /xx.xx.xx.xx:9000. Already tried 1 time(s).
2010-03-17 02:39:08,081 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: /xx.xx.xx.xx:9000. Already tried 2 time(s).
2010-03-17 02:39:09,098 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: /xx.xx.xx.xx6:9000. Already tried 3 time(s).
2010-03-17 02:39:10,159 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: /xx.xx.xx.xx:9000. Already tried 4 time(s).
2010-03-17 02:39:11,179 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: /xx.xx.xx.xx:9000. Already tried 5 time(s).
2010-03-17 02:39:12,221 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: /xx.xx.xx.xx:9000. Already tried 6 time(s).
2010-03-17 02:39:13,372 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: /xx.xx.xx.xx:9000. Already tried 7 time(s).
2010-03-17 02:39:14,545 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: /xx.xx.xx.xx:9000. Already tried 8 time(s).
2010-03-17 02:39:15,558 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: /xx.xx.xx.xx:9000. Already tried 9 time(s).
*************************************

Does anybody know what might cause this problem?

ssh among these machines are fine without password. The owner of hadoop
folder has been changed to the same hadoop user.

Thanks!


William

--0016361e7d34095bfe0481f971d8--

From general-return-1209-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 17 17:12:42 2010
Return-Path: <general-return-1209-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 13304 invoked from network); 17 Mar 2010 17:12:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Mar 2010 17:12:42 -0000
Received: (qmail 67353 invoked by uid 500); 17 Mar 2010 17:12:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67170 invoked by uid 500); 17 Mar 2010 17:12:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67136 invoked by uid 99); 17 Mar 2010 17:12:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Mar 2010 17:12:38 +0000
X-ASF-Spam-Status: No, hits=-2.2 required=10.0
	tests=AWL,HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of swatt@us.ibm.com designates 32.97.110.149 as permitted sender)
Received: from [32.97.110.149] (HELO e31.co.us.ibm.com) (32.97.110.149)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Mar 2010 17:12:30 +0000
Received: from d03relay02.boulder.ibm.com (d03relay02.boulder.ibm.com [9.17.195.227])
	by e31.co.us.ibm.com (8.14.3/8.13.1) with ESMTP id o2HH3BQd021553;
	Wed, 17 Mar 2010 11:03:11 -0600
Received: from d03av05.boulder.ibm.com (d03av05.boulder.ibm.com [9.17.195.85])
	by d03relay02.boulder.ibm.com (8.13.8/8.13.8/NCO v9.1) with ESMTP id o2HHC7WF246476;
	Wed, 17 Mar 2010 11:12:07 -0600
Received: from d03av05.boulder.ibm.com (loopback [127.0.0.1])
	by d03av05.boulder.ibm.com (8.14.3/8.13.1/NCO v10.0 AVout) with ESMTP id o2HHC49j013457;
	Wed, 17 Mar 2010 11:12:04 -0600
Received: from d03nm123.boulder.ibm.com (d03nm123.boulder.ibm.com [9.17.195.149])
	by d03av05.boulder.ibm.com (8.14.3/8.13.1/NCO v10.0 AVin) with ESMTP id o2HHC4He013450;
	Wed, 17 Mar 2010 11:12:04 -0600
In-Reply-To: <ca73442f1003162343x350fe04cj1c3e35efbb773d40@mail.gmail.com>
References: <ca73442f1003162343x350fe04cj1c3e35efbb773d40@mail.gmail.com>
To: general@hadoop.apache.org, common-user@hadoop.apache.org
MIME-Version: 1.0
Subject: Austin Hadoop Users Group - Tomorrow Evening (Thursday)
X-KeepSent: 0782D234:D418EB9A-862576E9:005E3C5C;
 type=4; name=$KeepSent
X-Mailer: Lotus Notes Release 8.5 December 05, 2008
Message-ID: <OF0782D234.D418EB9A-ON862576E9.005E3C5C-862576E9.005E7CAD@us.ibm.com>
From: Stephen Watt <swatt@us.ibm.com>
Date: Wed, 17 Mar 2010 12:12:03 -0500
X-MIMETrack: Serialize by Router on D03NM123/03/M/IBM(Release 8.5.1HF41 | October 22, 2009) at
 03/17/2010 11:12:03,
	Serialize complete at 03/17/2010 11:12:03
Content-Type: multipart/alternative; boundary="=_alternative 005E7CAC862576E9_="

--=_alternative 005E7CAC862576E9_=
Content-Type: text/plain; charset="US-ASCII"

Hi Folks

The Austin HUG is meeting tomorrow night. I hope to see you there. We have 
speakers from Rackspace (Stu Hood on Cassandra) and IBM (Gino Bustelo on 
BigSheets).

Detailed Information is available at http://austinhug.blogspot.com/

Kind regards
Steve Watt
--=_alternative 005E7CAC862576E9_=--

From general-return-1210-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 18 03:35:09 2010
Return-Path: <general-return-1210-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 662 invoked from network); 18 Mar 2010 03:35:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Mar 2010 03:35:08 -0000
Received: (qmail 68970 invoked by uid 500); 18 Mar 2010 03:35:08 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 68796 invoked by uid 500); 18 Mar 2010 03:35:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 68788 invoked by uid 99); 18 Mar 2010 03:35:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Mar 2010 03:35:06 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of magawake@gmail.com designates 209.85.223.198 as permitted sender)
Received: from [209.85.223.198] (HELO mail-iw0-f198.google.com) (209.85.223.198)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Mar 2010 03:35:00 +0000
Received: by iwn36 with SMTP id 36so1669951iwn.29
        for <general@hadoop.apache.org>; Wed, 17 Mar 2010 20:34:39 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=eiCJThR5X9BUasl48ludyr/BV23Duo1ErZlxvkzKGkY=;
        b=XC9bsykX34qfRnAqicinkdv5UmT7JPn7lsHFn1GKZ3R18zaVYFRJtcC9GxHbL4iAMu
         LokC0Bgp50EtoJpLfj/qmz0pdQmzD2gCA8rS77YF1fYveSgy2E82iecuowU1acYrt1mW
         bhgM/rI29doHthdmSqDYL7begS3HWLmb9LQHQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=jlVbQd/0mHdaqpPaNlJGAdsaqOQ/PzgrdcbJB+0/JgWgmoVVeL4nz61AvDjVb+d5rV
         RLMmOvV18nptm6KMU1PL9lapctx1xtj2Acv3itnMEQsuf2xyVuVhjs22rGswExlgTgu6
         5bqaJRttcG8VVY0uJ5ECoioNfAVhcfNJmmxV8=
MIME-Version: 1.0
Received: by 10.231.170.14 with SMTP id b14mr738499ibz.26.1268883279544; Wed, 
	17 Mar 2010 20:34:39 -0700 (PDT)
In-Reply-To: <c9118d4c1003041830h47961fc4p323c2ebdd0df2ffc@mail.gmail.com>
References: <C7B48351.36DF%awittenauer@linkedin.com>
	 <4B8FC1CC.2090108@apache.org>
	 <1cbd6f831003041539r75b360ecrb8e604d4dce1350f@mail.gmail.com>
	 <c9118d4c1003041830h47961fc4p323c2ebdd0df2ffc@mail.gmail.com>
Date: Wed, 17 Mar 2010 23:34:39 -0400
Message-ID: <1cbd6f831003172034r3d699283hd4e41f5fc0f37231@mail.gmail.com>
Subject: Re: rack awareness help
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Well,  I didn't really solve the problem. Now I have even more questions.

I came across this script,
http://wiki.apache.org/hadoop/topology_rack_awareness_scripts

but it makes no sense to me! Can someone please try to explain what
its trying to do?


MikeThomas:

Your script isn't working for me. I think there are some syntax
errors. Is this how its supposed to look: http://pastebin.ca/1844287

thanks



On Thu, Mar 4, 2010 at 10:30 PM, Jeff Hammerbacher <hammer@cloudera.com> wr=
ote:
> Hey Mag,
>
> Glad you have solved the problem. I've created a JIRA ticket to improve t=
he
> existing documentation: https://issues.apache.org/jira/browse/HADOOP-6616=
.
> If you have some time, it would be useful to hear what could be added to =
the
> existing documentation that would have helped you figure this out sooner.
>
> Thanks,
> Jeff
>
> On Thu, Mar 4, 2010 at 3:39 PM, Mag Gam <magawake@gmail.com> wrote:
>
>> Thanks everyone for explaining this to me instead of giving me RTFM!
>>
>> I will play around with it and see how far I get.
>>
>>
>>
>> On Thu, Mar 4, 2010 at 9:21 AM, Steve Loughran <stevel@apache.org> wrote=
:
>> > Allen Wittenauer wrote:
>> >>
>> >> On 3/3/10 5:01 PM, "Mag Gam" <magawake@gmail.com> wrote:
>> >>
>> >>> Thanks Alan! Your presentation is very nice!
>> >>
>> >> Thanks. :)
>> >>
>> >>> "If you don't provide a script for rack awareness, it treats every
>> >>> node as if it was its own rack". I am using the default settings and
>> >>> the report still says only 1 rack.
>> >>
>> >> Let's take a different approach to convince you. :)
>> >>
>> >> Think about the question: =C2=A0Is there a difference between all nod=
es in
>> one
>> >> rack vs. every node acting as a lone rack?
>> >>
>> >> The answer is no, there isn't any difference. =C2=A0In both cases, al=
l copies
>> >> of
>> >> the blocks can go to pretty much any node. When a MR job runs, every
>> node
>> >> would either be considered 'off rack' or 'rack-local'.
>> >>
>> >> So there is no difference.
>> >>
>> >>
>> >>> Do you mind sharing a script with us on how you determine a rack? an=
d
>> >>> a sample <configuration> </configuration> syntax?
>> >>
>> >> Michael has already posted his, so I'll skip this one. :)
>> >>
>> >
>> > Think Mag probably wanted a shell script.
>> >
>> > Mag, give your machines IPv4 addresses that map to rack number. 10.1.1=
.*
>> for
>> > rack one, 10.1.2.* for rack 2, etc. Then just filter out the IP addres=
s
>> by
>> > the top bytes, returning "10.1.1" for everything in rack one, "10.1.2"
>> for
>> > rack 2; Hadoop will be happy
>> >
>>
>

From general-return-1211-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 18 05:00:42 2010
Return-Path: <general-return-1211-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 10632 invoked from network); 18 Mar 2010 05:00:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Mar 2010 05:00:42 -0000
Received: (qmail 98457 invoked by uid 500); 18 Mar 2010 05:00:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 98262 invoked by uid 500); 18 Mar 2010 05:00:40 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 98254 invoked by uid 99); 18 Mar 2010 05:00:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Mar 2010 05:00:39 +0000
X-ASF-Spam-Status: No, hits=0.9 required=10.0
	tests=AWL,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mailinglists19@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Mar 2010 05:00:33 +0000
Received: by wyf19 with SMTP id 19so838603wyf.35
        for <general@hadoop.apache.org>; Wed, 17 Mar 2010 22:00:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=JF/uvriwYgn2BYR0fHyTcffkOuH0StB667S1xZ9Ukio=;
        b=r7YvnVPaRJv9YK9sdwOICriv0/sHCBvwiROGZtREnofaq1k4mhE7pWVoKu6rnvrXSa
         cdkzbN8GCiBJKWUymmdhmYRyUQXpUOg1SaxhlTu2lDra1m5jUBaREBUb+LfEauNg2d0s
         xofeG00v+Cn+u9dNFRpUI4e1OxnNPf7Xo0mek=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=RcK6qXV46KpdcHjI/dGwMkUOXWpUcGvwWVCBu2WJFzKixnsIAGz/76l+5KBzYFCbGR
         DVftfytrEtAjDy1ChYOv7A9Tf7b4FBXWEm/1ZfPbANA+PwDoVIH+Y17D567DSbObCPzy
         H8BeQbBbJLEZkVkAQJv4sRre+NHTg7brk/JQo=
MIME-Version: 1.0
Received: by 10.216.88.16 with SMTP id z16mr1107123wee.126.1268888411970; Wed, 
	17 Mar 2010 22:00:11 -0700 (PDT)
Date: Wed, 17 Mar 2010 22:00:11 -0700
Message-ID: <1eabbac31003172200p59f57bd4q3979ce9b51a86095@mail.gmail.com>
Subject: Why is JobInProgress class not public
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d63fe00e1cde04820c1b01

--0016e6d63fe00e1cde04820c1b01
Content-Type: text/plain; charset=ISO-8859-1

JobTracker has methods such as getRunningJobs, completedJobs, failedJobs
that return a collection of JobInProgress.  All these methods are 'public',
but the type of elements that they return (JobInProgress) is NOT public.
Why?

--0016e6d63fe00e1cde04820c1b01--

From general-return-1212-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 18 07:15:43 2010
Return-Path: <general-return-1212-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 43877 invoked from network); 18 Mar 2010 07:15:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Mar 2010 07:15:42 -0000
Received: (qmail 79494 invoked by uid 500); 18 Mar 2010 07:15:42 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 79313 invoked by uid 500); 18 Mar 2010 07:15:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 79305 invoked by uid 99); 18 Mar 2010 07:15:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Mar 2010 07:15:40 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of ctubbsii@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Mar 2010 07:15:34 +0000
Received: by wwc33 with SMTP id 33so848498wwc.35
        for <general@hadoop.apache.org>; Thu, 18 Mar 2010 00:15:14 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=9wJY87RCJ0D5BOtHNSKWZqCG7HlzgorYrIph3x2nbMM=;
        b=Ob9olCwTtIXAUKLwsgIsucxRS0FiBnoarc7f+frsTgQ+o5M1PF0H3aNDFwGx88vdS0
         /1QOZ4OynC/SdZL8I0BFDm231xfYSShIHBRfWGPOiWakEmwv63Wx8YR/psrxcXtpczN3
         HEhOzYnwVw/Vzd73o8IfD/elxzx1kGmSpH4jE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=Ysn4lJckKaIInkrZgGRNz5oBWorR+J9kSev7KMbTiByZm7y2Emo4xM9z53ivKzANme
         TQdwZ9XUVHusiUkN6y117eVRmbbPTNfkeJh2fM/q71lQR3EM7pyLbxlpC4zzjhIcdkGZ
         DRtMxdfeg9K1Ij4LhMufia0b+JtbNZZcfsuPI=
MIME-Version: 1.0
Received: by 10.216.156.203 with SMTP id m53mr1127265wek.209.1268896513988; 
	Thu, 18 Mar 2010 00:15:13 -0700 (PDT)
In-Reply-To: <1cbd6f831003172034r3d699283hd4e41f5fc0f37231@mail.gmail.com>
References: <C7B48351.36DF%awittenauer@linkedin.com>
	 <4B8FC1CC.2090108@apache.org>
	 <1cbd6f831003041539r75b360ecrb8e604d4dce1350f@mail.gmail.com>
	 <c9118d4c1003041830h47961fc4p323c2ebdd0df2ffc@mail.gmail.com>
	 <1cbd6f831003172034r3d699283hd4e41f5fc0f37231@mail.gmail.com>
Date: Thu, 18 Mar 2010 03:15:13 -0400
Message-ID: <9db00c201003180015n1e4e8941i2c53296edd21258f@mail.gmail.com>
Subject: Re: rack awareness help
From: Christopher Tubbs <ctubbsii@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Hadoop will identify data nodes in your cluster by name and execute
your script with the data node as an argument. The expected output of
your script is the name of the rack on which it is located.

The script you referenced takes the node name as an argument ($1), and
crawls through a separate file looking up that node in the left
column, and printing the value in the second column if it finds it.

If you were to use this script, you would just create the topology
file that lists all your nodes by name/ip on the left and the rack
they are in on the right.

On Wed, Mar 17, 2010 at 11:34 PM, Mag Gam <magawake@gmail.com> wrote:
> Well, =A0I didn't really solve the problem. Now I have even more question=
s.
>
> I came across this script,
> http://wiki.apache.org/hadoop/topology_rack_awareness_scripts
>
> but it makes no sense to me! Can someone please try to explain what
> its trying to do?
>
>
> MikeThomas:
>
> Your script isn't working for me. I think there are some syntax
> errors. Is this how its supposed to look: http://pastebin.ca/1844287
>
> thanks
>
>
>
> On Thu, Mar 4, 2010 at 10:30 PM, Jeff Hammerbacher <hammer@cloudera.com> =
wrote:
>> Hey Mag,
>>
>> Glad you have solved the problem. I've created a JIRA ticket to improve =
the
>> existing documentation: https://issues.apache.org/jira/browse/HADOOP-661=
6.
>> If you have some time, it would be useful to hear what could be added to=
 the
>> existing documentation that would have helped you figure this out sooner=
.
>>
>> Thanks,
>> Jeff
>>
>> On Thu, Mar 4, 2010 at 3:39 PM, Mag Gam <magawake@gmail.com> wrote:
>>
>>> Thanks everyone for explaining this to me instead of giving me RTFM!
>>>
>>> I will play around with it and see how far I get.
>>>
>>>
>>>
>>> On Thu, Mar 4, 2010 at 9:21 AM, Steve Loughran <stevel@apache.org> wrot=
e:
>>> > Allen Wittenauer wrote:
>>> >>
>>> >> On 3/3/10 5:01 PM, "Mag Gam" <magawake@gmail.com> wrote:
>>> >>
>>> >>> Thanks Alan! Your presentation is very nice!
>>> >>
>>> >> Thanks. :)
>>> >>
>>> >>> "If you don't provide a script for rack awareness, it treats every
>>> >>> node as if it was its own rack". I am using the default settings an=
d
>>> >>> the report still says only 1 rack.
>>> >>
>>> >> Let's take a different approach to convince you. :)
>>> >>
>>> >> Think about the question: =A0Is there a difference between all nodes=
 in
>>> one
>>> >> rack vs. every node acting as a lone rack?
>>> >>
>>> >> The answer is no, there isn't any difference. =A0In both cases, all =
copies
>>> >> of
>>> >> the blocks can go to pretty much any node. When a MR job runs, every
>>> node
>>> >> would either be considered 'off rack' or 'rack-local'.
>>> >>
>>> >> So there is no difference.
>>> >>
>>> >>
>>> >>> Do you mind sharing a script with us on how you determine a rack? a=
nd
>>> >>> a sample <configuration> </configuration> syntax?
>>> >>
>>> >> Michael has already posted his, so I'll skip this one. :)
>>> >>
>>> >
>>> > Think Mag probably wanted a shell script.
>>> >
>>> > Mag, give your machines IPv4 addresses that map to rack number. 10.1.=
1.*
>>> for
>>> > rack one, 10.1.2.* for rack 2, etc. Then just filter out the IP addre=
ss
>>> by
>>> > the top bytes, returning "10.1.1" for everything in rack one, "10.1.2=
"
>>> for
>>> > rack 2; Hadoop will be happy
>>> >
>>>
>>
>

From general-return-1213-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 18 20:48:45 2010
Return-Path: <general-return-1213-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96285 invoked from network); 18 Mar 2010 20:48:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Mar 2010 20:48:45 -0000
Received: (qmail 75004 invoked by uid 500); 18 Mar 2010 20:48:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74945 invoked by uid 500); 18 Mar 2010 20:48:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74937 invoked by uid 99); 18 Mar 2010 20:48:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Mar 2010 20:48:44 +0000
X-ASF-Spam-Status: No, hits=1.1 required=10.0
	tests=AWL,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.62.64] (HELO qmta07.westchester.pa.mail.comcast.net) (76.96.62.64)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Mar 2010 20:48:35 +0000
Received: from omta09.westchester.pa.mail.comcast.net ([76.96.62.20])
	by qmta07.westchester.pa.mail.comcast.net with comcast
	id ud5w1d0120SCNGk57koF9u; Thu, 18 Mar 2010 20:48:15 +0000
Received: from [10.72.120.91] ([209.131.62.113])
	by omta09.westchester.pa.mail.comcast.net with comcast
	id uko51d00F2SbwD53Vko8tF; Thu, 18 Mar 2010 20:48:13 +0000
Message-Id: <0A7BE353-65C7-47BC-9ACD-158BE7ADFD71@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=Apple-Mail-5-727791775
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Hadoop sub-project spin outs...
Date: Thu, 18 Mar 2010 13:48:04 -0700
X-Mailer: Apple Mail (2.936)

--Apple-Mail-5-727791775
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit

As you may have seen on the various dev lists, some of the Hadoop sub- 
projects such as HBase and Avro have started discussions on their dev  
lists about becoming top level Apache projects. This is largely  
motivated by the Apache board's continued warnings to Hadoop and  
Lucene against becoming an umbrella project. (Umbrella projects are  
projects that are meta-projects where the development communities and  
releases of the sub-projects are largely disjoint.) I've heard some  
questions about whether if sub-project X becomes a TLP it will be  
included in events like Hadoop Summit, Hadoop World, etc. For Hadoop  
Summit and the Bay Area Hadoop User Group meetings, which Yahoo  
organizes, a change in to a TLP won't change anything. I don't have  
any reason to believe the other meetings will behave differently.

Thanks,
    Owen
--Apple-Mail-5-727791775--

From general-return-1214-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 18 23:30:46 2010
Return-Path: <general-return-1214-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 47230 invoked from network); 18 Mar 2010 23:30:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Mar 2010 23:30:45 -0000
Received: (qmail 28307 invoked by uid 500); 18 Mar 2010 23:30:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 28245 invoked by uid 500); 18 Mar 2010 23:30:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 28237 invoked by uid 99); 18 Mar 2010 23:30:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Mar 2010 23:30:44 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Thu, 18 Mar 2010 23:30:42 +0000
Received: (qmail 47122 invoked by uid 99); 18 Mar 2010 23:30:16 -0000
Received: from localhost.apache.org (HELO [192.168.168.105]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Thu, 18 Mar 2010 23:30:16 +0000
Message-ID: <4BA2B786.4030005@apache.org>
Date: Thu, 18 Mar 2010 16:30:14 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: avro-dev@hadoop.apache.org, general@hadoop.apache.org
Subject: Re: [VOTE] Avro release 1.3.1 (rc 0)
References: <4B9AB570.6020402@apache.org> <ac79ea401003181611r649deb91v3a5316914648f1db@mail.gmail.com>
In-Reply-To: <ac79ea401003181611r649deb91v3a5316914648f1db@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks, Tom!

We just need one more vote from a Hadoop PMC member to release this.

Doug

Tom White wrote:
> +1
> 
> Based on checking checksums and signatures, and running tests.
> 
> Tom
> 
> On Fri, Mar 12, 2010 at 2:43 PM, Doug Cutting <cutting@apache.org> wrote:
>> I have created a candidate build for Avro release 1.3.1.
>>
>> Changes are listed at:
>>
>>  http://tinyurl.com/avro131
>>
>> Please download, test, and vote by 16 March.
>>
>>  http://people.apache.org/~cutting/avro-1.3.1-rc0/
>>
>> Thanks,
>>
>> Doug
>>

From general-return-1215-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 19 00:42:22 2010
Return-Path: <general-return-1215-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67169 invoked from network); 19 Mar 2010 00:42:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Mar 2010 00:42:21 -0000
Received: (qmail 76634 invoked by uid 500); 19 Mar 2010 00:42:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76589 invoked by uid 500); 19 Mar 2010 00:42:21 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76579 invoked by uid 99); 19 Mar 2010 00:42:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 00:42:21 +0000
X-ASF-Spam-Status: No, hits=-2.9 required=10.0
	tests=AWL,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of magawake@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 00:42:16 +0000
Received: by vws6 with SMTP id 6so770206vws.35
        for <general@hadoop.apache.org>; Thu, 18 Mar 2010 17:41:55 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=B7sGE1q+UrIyp5KFJexcsLfxeIzhC9mS/iEFOsdYqG4=;
        b=gg0gk38u+Zn/jz12pG9afaDQM2oMUwhEYYFCFPIlYFdwzPDiRTK6TBjoZWO+ZCwuoV
         pXGOVW+mgWLaPuJ0w9rgS/jHxU2IwVf3KqIepotwtCSjz9Beahkudtxgva+/iA+ienOz
         ZAorXkcU+L/eiNTdqkWz6pjwKxIPAb2JQF4P4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=QFjt2fPRrJwrobvN6NEQMiX/AZflNQ89bv1fofq8jZrD2Z0YGZFDTua825NaLlsI8Z
         1bdA93atCE7TtZBnEEdrgIB98QV1o9F86uWdAE7q+pFg7DEAvgfbnqjK8Yi3VoRWnTkJ
         +9iE02kAQCL7Kci/FdbgRUPBl2QCAKLCb2Wj4=
MIME-Version: 1.0
Received: by 10.220.126.166 with SMTP id c38mr861378vcs.169.1268959315129; 
	Thu, 18 Mar 2010 17:41:55 -0700 (PDT)
In-Reply-To: <9db00c201003180015n1e4e8941i2c53296edd21258f@mail.gmail.com>
References: <C7B48351.36DF%awittenauer@linkedin.com>
	 <4B8FC1CC.2090108@apache.org>
	 <1cbd6f831003041539r75b360ecrb8e604d4dce1350f@mail.gmail.com>
	 <c9118d4c1003041830h47961fc4p323c2ebdd0df2ffc@mail.gmail.com>
	 <1cbd6f831003172034r3d699283hd4e41f5fc0f37231@mail.gmail.com>
	 <9db00c201003180015n1e4e8941i2c53296edd21258f@mail.gmail.com>
Date: Thu, 18 Mar 2010 20:41:55 -0400
Message-ID: <1cbd6f831003181741t2f538eb8xc872cba6dfe00761@mail.gmail.com>
Subject: Re: rack awareness help
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Chris:

This clears up my questions a lot! Thankyou.

So, if I have 4 data servers and I want 2 racks. I can do this

#!/bin/bash
#rack1.sh
echo rack1

#bin/bash
#rack2.sh
echo rack2


So, I can do this for 2 servers


<property>
 <name>topology.script.file.name</name>
 <value>rack1.sh</value>
</property>

And for the other 2 servers, I can do this:


<property>
 <name>topology.script.file.name</name>
 <value>rack2.sh</value>
</property>


correct?


On Thu, Mar 18, 2010 at 3:15 AM, Christopher Tubbs <ctubbsii@gmail.com> wro=
te:
> Hadoop will identify data nodes in your cluster by name and execute
> your script with the data node as an argument. The expected output of
> your script is the name of the rack on which it is located.
>
> The script you referenced takes the node name as an argument ($1), and
> crawls through a separate file looking up that node in the left
> column, and printing the value in the second column if it finds it.
>
> If you were to use this script, you would just create the topology
> file that lists all your nodes by name/ip on the left and the rack
> they are in on the right.
>
> On Wed, Mar 17, 2010 at 11:34 PM, Mag Gam <magawake@gmail.com> wrote:
>> Well, =C2=A0I didn't really solve the problem. Now I have even more ques=
tions.
>>
>> I came across this script,
>> http://wiki.apache.org/hadoop/topology_rack_awareness_scripts
>>
>> but it makes no sense to me! Can someone please try to explain what
>> its trying to do?
>>
>>
>> MikeThomas:
>>
>> Your script isn't working for me. I think there are some syntax
>> errors. Is this how its supposed to look: http://pastebin.ca/1844287
>>
>> thanks
>>
>>
>>
>> On Thu, Mar 4, 2010 at 10:30 PM, Jeff Hammerbacher <hammer@cloudera.com>=
 wrote:
>>> Hey Mag,
>>>
>>> Glad you have solved the problem. I've created a JIRA ticket to improve=
 the
>>> existing documentation: https://issues.apache.org/jira/browse/HADOOP-66=
16.
>>> If you have some time, it would be useful to hear what could be added t=
o the
>>> existing documentation that would have helped you figure this out soone=
r.
>>>
>>> Thanks,
>>> Jeff
>>>
>>> On Thu, Mar 4, 2010 at 3:39 PM, Mag Gam <magawake@gmail.com> wrote:
>>>
>>>> Thanks everyone for explaining this to me instead of giving me RTFM!
>>>>
>>>> I will play around with it and see how far I get.
>>>>
>>>>
>>>>
>>>> On Thu, Mar 4, 2010 at 9:21 AM, Steve Loughran <stevel@apache.org> wro=
te:
>>>> > Allen Wittenauer wrote:
>>>> >>
>>>> >> On 3/3/10 5:01 PM, "Mag Gam" <magawake@gmail.com> wrote:
>>>> >>
>>>> >>> Thanks Alan! Your presentation is very nice!
>>>> >>
>>>> >> Thanks. :)
>>>> >>
>>>> >>> "If you don't provide a script for rack awareness, it treats every
>>>> >>> node as if it was its own rack". I am using the default settings a=
nd
>>>> >>> the report still says only 1 rack.
>>>> >>
>>>> >> Let's take a different approach to convince you. :)
>>>> >>
>>>> >> Think about the question: =C2=A0Is there a difference between all n=
odes in
>>>> one
>>>> >> rack vs. every node acting as a lone rack?
>>>> >>
>>>> >> The answer is no, there isn't any difference. =C2=A0In both cases, =
all copies
>>>> >> of
>>>> >> the blocks can go to pretty much any node. When a MR job runs, ever=
y
>>>> node
>>>> >> would either be considered 'off rack' or 'rack-local'.
>>>> >>
>>>> >> So there is no difference.
>>>> >>
>>>> >>
>>>> >>> Do you mind sharing a script with us on how you determine a rack? =
and
>>>> >>> a sample <configuration> </configuration> syntax?
>>>> >>
>>>> >> Michael has already posted his, so I'll skip this one. :)
>>>> >>
>>>> >
>>>> > Think Mag probably wanted a shell script.
>>>> >
>>>> > Mag, give your machines IPv4 addresses that map to rack number. 10.1=
.1.*
>>>> for
>>>> > rack one, 10.1.2.* for rack 2, etc. Then just filter out the IP addr=
ess
>>>> by
>>>> > the top bytes, returning "10.1.1" for everything in rack one, "10.1.=
2"
>>>> for
>>>> > rack 2; Hadoop will be happy
>>>> >
>>>>
>>>
>>
>

From general-return-1216-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 19 01:11:51 2010
Return-Path: <general-return-1216-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 71884 invoked from network); 19 Mar 2010 01:11:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Mar 2010 01:11:51 -0000
Received: (qmail 882 invoked by uid 500); 19 Mar 2010 01:11:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 838 invoked by uid 500); 19 Mar 2010 01:11:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 829 invoked by uid 99); 19 Mar 2010 01:11:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 01:11:50 +0000
X-ASF-Spam-Status: No, hits=-1.0 required=10.0
	tests=AWL,SPF_HELO_PASS,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [64.81.36.244] (HELO foobar.kobold.org) (64.81.36.244)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 01:11:45 +0000
Received: from owl.kobold.org (owl.kobold.org [192.168.1.7])
	(authenticated bits=0)
	by foobar.kobold.org (8.14.1/8.14.1) with ESMTP id o2INvO3q008079
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Thu, 18 Mar 2010 16:57:30 -0700
Message-ID: <4BA2CF33.1000504@hep.caltech.edu>
Date: Thu, 18 Mar 2010 18:11:15 -0700
From: Michael Thomas <thomas@hep.caltech.edu>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.7) Gecko/20100120 Fedora/3.0.1-1.fc12 Thunderbird/3.0.1
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: rack awareness help
References: <C7B48351.36DF%awittenauer@linkedin.com>	 <4B8FC1CC.2090108@apache.org>	 <1cbd6f831003041539r75b360ecrb8e604d4dce1350f@mail.gmail.com>	 <c9118d4c1003041830h47961fc4p323c2ebdd0df2ffc@mail.gmail.com>	 <1cbd6f831003172034r3d699283hd4e41f5fc0f37231@mail.gmail.com>	 <9db00c201003180015n1e4e8941i2c53296edd21258f@mail.gmail.com> <1cbd6f831003181741t2f538eb8xc872cba6dfe00761@mail.gmail.com>
In-Reply-To: <1cbd6f831003181741t2f538eb8xc872cba6dfe00761@mail.gmail.com>
Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms070408020806080508010302"

--------------ms070408020806080508010302
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: quoted-printable

On 03/18/2010 05:41 PM, Mag Gam wrote:
> Chris:
>
> This clears up my questions a lot! Thankyou.
>
> So, if I have 4 data servers and I want 2 racks. I can do this
>
> #!/bin/bash
> #rack1.sh
> echo rack1
>
> #bin/bash
> #rack2.sh
> echo rack2
>
>
> So, I can do this for 2 servers
>
>
> <property>
>   <name>topology.script.file.name</name>
>   <value>rack1.sh</value>
> </property>
>
> And for the other 2 servers, I can do this:
>
>
> <property>
>   <name>topology.script.file.name</name>
>   <value>rack2.sh</value>
> </property>
>
>
> correct?

Incorrect.  You only specify a single topology.script.file.name on the=20
namenode.  This attribute is ignored on the datanodes.

Also be aware that the namenode invokes the script with the _ip address_ =

of each datanode, not the hostname of each datanode.  If you need to=20
convert the ip address to a hostname in order to determine which rack=20
the datanode is on, then you have to put that conversion logic in your=20
topology.script.file.name.

--Mike

> On Thu, Mar 18, 2010 at 3:15 AM, Christopher Tubbs<ctubbsii@gmail.com> =
 wrote:
>> Hadoop will identify data nodes in your cluster by name and execute
>> your script with the data node as an argument. The expected output of
>> your script is the name of the rack on which it is located.
>>
>> The script you referenced takes the node name as an argument ($1), and=

>> crawls through a separate file looking up that node in the left
>> column, and printing the value in the second column if it finds it.
>>
>> If you were to use this script, you would just create the topology
>> file that lists all your nodes by name/ip on the left and the rack
>> they are in on the right.
>>
>> On Wed, Mar 17, 2010 at 11:34 PM, Mag Gam<magawake@gmail.com>  wrote:
>>> Well,  I didn't really solve the problem. Now I have even more questi=
ons.
>>>
>>> I came across this script,
>>> http://wiki.apache.org/hadoop/topology_rack_awareness_scripts
>>>
>>> but it makes no sense to me! Can someone please try to explain what
>>> its trying to do?
>>>
>>>
>>> MikeThomas:
>>>
>>> Your script isn't working for me. I think there are some syntax
>>> errors. Is this how its supposed to look: http://pastebin.ca/1844287
>>>
>>> thanks
>>>
>>>
>>>
>>> On Thu, Mar 4, 2010 at 10:30 PM, Jeff Hammerbacher<hammer@cloudera.co=
m>  wrote:
>>>> Hey Mag,
>>>>
>>>> Glad you have solved the problem. I've created a JIRA ticket to impr=
ove the
>>>> existing documentation: https://issues.apache.org/jira/browse/HADOOP=
-6616.
>>>> If you have some time, it would be useful to hear what could be adde=
d to the
>>>> existing documentation that would have helped you figure this out so=
oner.
>>>>
>>>> Thanks,
>>>> Jeff
>>>>
>>>> On Thu, Mar 4, 2010 at 3:39 PM, Mag Gam<magawake@gmail.com>  wrote:
>>>>
>>>>> Thanks everyone for explaining this to me instead of giving me RTFM=
!
>>>>>
>>>>> I will play around with it and see how far I get.
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Mar 4, 2010 at 9:21 AM, Steve Loughran<stevel@apache.org>  =
wrote:
>>>>>> Allen Wittenauer wrote:
>>>>>>>
>>>>>>> On 3/3/10 5:01 PM, "Mag Gam"<magawake@gmail.com>  wrote:
>>>>>>>
>>>>>>>> Thanks Alan! Your presentation is very nice!
>>>>>>>
>>>>>>> Thanks. :)
>>>>>>>
>>>>>>>> "If you don't provide a script for rack awareness, it treats eve=
ry
>>>>>>>> node as if it was its own rack". I am using the default settings=
 and
>>>>>>>> the report still says only 1 rack.
>>>>>>>
>>>>>>> Let's take a different approach to convince you. :)
>>>>>>>
>>>>>>> Think about the question:  Is there a difference between all node=
s in
>>>>> one
>>>>>>> rack vs. every node acting as a lone rack?
>>>>>>>
>>>>>>> The answer is no, there isn't any difference.  In both cases, all=
 copies
>>>>>>> of
>>>>>>> the blocks can go to pretty much any node. When a MR job runs, ev=
ery
>>>>> node
>>>>>>> would either be considered 'off rack' or 'rack-local'.
>>>>>>>
>>>>>>> So there is no difference.
>>>>>>>
>>>>>>>
>>>>>>>> Do you mind sharing a script with us on how you determine a rack=
? and
>>>>>>>> a sample<configuration>  </configuration>  syntax?
>>>>>>>
>>>>>>> Michael has already posted his, so I'll skip this one. :)
>>>>>>>
>>>>>>
>>>>>> Think Mag probably wanted a shell script.
>>>>>>
>>>>>> Mag, give your machines IPv4 addresses that map to rack number. 10=
=2E1.1.*
>>>>> for
>>>>>> rack one, 10.1.2.* for rack 2, etc. Then just filter out the IP ad=
dress
>>>>> by
>>>>>> the top bytes, returning "10.1.1" for everything in rack one, "10.=
1.2"
>>>>> for
>>>>>> rack 2; Hadoop will be happy
>>>>>>
>>>>>
>>>>
>>>
>>



--------------ms070408020806080508010302
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIMVjCC
A/gwggLgoAMCAQICASkwDQYJKoZIhvcNAQEFBQAwdTETMBEGCgmSJomT8ixkARkWA25ldDES
MBAGCgmSJomT8ixkARkWAkVTMQ4wDAYDVQQKEwVFU25ldDEgMB4GA1UECxMXQ2VydGlmaWNh
dGUgQXV0aG9yaXRpZXMxGDAWBgNVBAMTD0VTbmV0IFJvb3QgQ0EgMTAeFw0wMjEyMDUwODAw
MDBaFw0xMzAxMjUwODAwMDBaMGkxEzARBgoJkiaJk/IsZAEZFgNvcmcxGDAWBgoJkiaJk/Is
ZAEZFghET0VHcmlkczEgMB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9yaXRpZXMxFjAUBgNV
BAMTDURPRUdyaWRzIENBIDEwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC09dYj
YaPbCD5mtbiQb7Ka3y1qAm0ZcqKCFciWcfe8Kwcuy9tjHuIsLf9ZItdkDW4xy8sua9nJlx3K
lwjtumTMtOtg35KZCknUd8KM4VGTSFdLVG9AbNayef76caVCGM1+jyF0Lq03kauGOPTcNfZe
1TZa3e1c9rc8ljV5OSWa/mfsCACyS5zFIWu0yIDNyJdf+n0hwaPN53wllpJ30taD+JBjQ7h2
k4xRWzeaznLOb9OztZVRA/1sVze+iczFh2xwa4VdGy0eIIPw1pfvYwxO36rm0S109qvbsNla
roPRbxerPKakQLpKe034Xcx7gBPqUk/FxoRRWin5EWN3rz9LAgMBAAGjgZ4wgZswDgYDVR0P
AQH/BAQDAgGGMBEGCWCGSAGG+EIBAQQEAwIAhzAdBgNVHQ4EFgQUyhkdEo5upDhdQtQxDgjb
2Y0XDV0wHwYDVR0jBBgwFoAUvF1NSC/4NZRZq1yJSz7RsjoUAeowDwYDVR0TAQH/BAUwAwEB
/zAlBgNVHREEHjAcgRpET0VHcmlkcy1DQS0xQGRvZWdyaWRzLm9yZzANBgkqhkiG9w0BAQUF
AAOCAQEAZNVrIDLqe39CEOiJt7Q7EpBPhAihMvDTSf/42u0SMbUmChww4mLmph5DBghZUVF8
Yn59kRZMn1QLOtO1HzLqvAvPITacZVPlJgG2IXzlR636YghZFAycbIUEOJDBHR4vtQO1KDxg
ZwvAbtmKIoxvhUCq2xsfFt9kCBBn+JYtQ6O5LsBJq3PmuubeMcc7mbQAfJZ7h/3QghgkFIhm
E1+LBXPJbkuP8vgfg6h2BKoAf5TFfZECgGZKimfN110tBvfedGZwYYd3/GsJc83B0JN1gny0
gqNVPm392UchXGeBRrHnm2gkhIkr48Oq6EmNGV9/a6XfbplQW/JWbtPVPWkaizCCBCkwggMR
oAMCAQICAwChZzANBgkqhkiG9w0BAQUFADBpMRMwEQYKCZImiZPyLGQBGRYDb3JnMRgwFgYK
CZImiZPyLGQBGRYIRE9FR3JpZHMxIDAeBgNVBAsTF0NlcnRpZmljYXRlIEF1dGhvcml0aWVz
MRYwFAYDVQQDEw1ET0VHcmlkcyBDQSAxMB4XDTEwMDEyNTE4MzEyMFoXDTExMDEyNTE4MzEy
MFowYDETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCGRvZWdyaWRzMQ8w
DQYDVQQLEwZQZW9wbGUxHjAcBgNVBAMTFU1pY2hhZWwgVGhvbWFzIDcyNTM3NDCCASIwDQYJ
KoZIhvcNAQEBBQADggEPADCCAQoCggEBALchCLXStiJKw5u2+nFW0020jqo1X8/kRB9kus8a
6Op5ds3j5O3IZkyZ41tQwE9Rc0yRDiDGpT1jL/n2LK1HPXvZbasmPCxxTMY3kNTm4fWAnRS7
UzLo0Adaz81bcT8Buo7Il1FCS684LhsK19JChqU2/iY8G7H0uBImj6QDLR0+w5QsLfD7aOXx
M8Njhse1ZqMyjwjhw5FqWyWNhlKi1oW42eXtYhbsqyAQZSXLoz2SwqBnLTRqcRMn4GgvB83Z
1FObyb79HFiuqlbaMlXWy0s8ywajJnMG8LsgwkNJZQLJsPalz5rwe/pOG1abasLMBVpJGGSB
AQktY8s5M+U9RlMCAwEAAaOB4jCB3zARBglghkgBhvhCAQEEBAMCBeAwDgYDVR0PAQH/BAQD
AgTwMDYGA1UdIAQvMC0wDQYLKoZIhvdMAwcBAwEwDAYKKoZIhvdMBQICATAOBgwqhkiG90wF
AgMCAQEwPgYDVR0fBDcwNTAzoDGgL4YtaHR0cDovL2NybC5kb2Vncmlkcy5vcmcvMWMzZjJj
YTgvMWMzZjJjYTguY3JsMCEGA1UdEQQaMBiBFnRob21hc0BoZXAuY2FsdGVjaC5lZHUwHwYD
VR0jBBgwFoAUyhkdEo5upDhdQtQxDgjb2Y0XDV0wDQYJKoZIhvcNAQEFBQADggEBAGcleFpm
d9Ho37gHZrrF0zulSQ3Aemrbx2O4EVJotQbtAs2v4nnH0eAa0F37BXRA6DAvgaC4YODA6z5M
HHE2mnsIebv0nS8DLj1yN7C1nqDSUXlaIzExEGQ1vXG2XlVo6pMZgJd2ML9+dyUUOcnNj0kQ
GeMysWBHbDChdfVG8y2PuGzoFY/ikNtWWhTNcPVZHIL5I2JCYqV5lyLQhlkC0Q5eqbqYKAq1
GuQmLO8hPfzkb6qbAkzxB8/FFqjlPgBgFcd9Nf2fAI14kI77iRbEuu8jSTPx13p6uYvcX4hy
1cW4Y7lnE8bBkAFWrUuNGc+aw7Ang6rJANWXtF/tN+UqUOcwggQpMIIDEaADAgECAgMAoWcw
DQYJKoZIhvcNAQEFBQAwaTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkW
CERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UEAxMN
RE9FR3JpZHMgQ0EgMTAeFw0xMDAxMjUxODMxMjBaFw0xMTAxMjUxODMxMjBaMGAxEzARBgoJ
kiaJk/IsZAEZFgNvcmcxGDAWBgoJkiaJk/IsZAEZFghkb2VncmlkczEPMA0GA1UECxMGUGVv
cGxlMR4wHAYDVQQDExVNaWNoYWVsIFRob21hcyA3MjUzNzQwggEiMA0GCSqGSIb3DQEBAQUA
A4IBDwAwggEKAoIBAQC3IQi10rYiSsObtvpxVtNNtI6qNV/P5EQfZLrPGujqeXbN4+TtyGZM
meNbUMBPUXNMkQ4gxqU9Yy/59iytRz172W2rJjwscUzGN5DU5uH1gJ0Uu1My6NAHWs/NW3E/
AbqOyJdRQkuvOC4bCtfSQoalNv4mPBux9LgSJo+kAy0dPsOULC3w+2jl8TPDY4bHtWajMo8I
4cORalsljYZSotaFuNnl7WIW7KsgEGUly6M9ksKgZy00anETJ+BoLwfN2dRTm8m+/RxYrqpW
2jJV1stLPMsGoyZzBvC7IMJDSWUCybD2pc+a8Hv6ThtWm2rCzAVaSRhkgQEJLWPLOTPlPUZT
AgMBAAGjgeIwgd8wEQYJYIZIAYb4QgEBBAQDAgXgMA4GA1UdDwEB/wQEAwIE8DA2BgNVHSAE
LzAtMA0GCyqGSIb3TAMHAQMBMAwGCiqGSIb3TAUCAgEwDgYMKoZIhvdMBQIDAgEBMD4GA1Ud
HwQ3MDUwM6AxoC+GLWh0dHA6Ly9jcmwuZG9lZ3JpZHMub3JnLzFjM2YyY2E4LzFjM2YyY2E4
LmNybDAhBgNVHREEGjAYgRZ0aG9tYXNAaGVwLmNhbHRlY2guZWR1MB8GA1UdIwQYMBaAFMoZ
HRKObqQ4XULUMQ4I29mNFw1dMA0GCSqGSIb3DQEBBQUAA4IBAQBnJXhaZnfR6N+4B2a6xdM7
pUkNwHpq28djuBFSaLUG7QLNr+J5x9HgGtBd+wV0QOgwL4GguGDgwOs+TBxxNpp7CHm79J0v
Ay49cjewtZ6g0lF5WiMxMRBkNb1xtl5VaOqTGYCXdjC/fnclFDnJzY9JEBnjMrFgR2wwoXX1
RvMtj7hs6BWP4pDbVloUzXD1WRyC+SNiQmKleZci0IZZAtEOXqm6mCgKtRrkJizvIT385G+q
mwJM8QfPxRao5T4AYBXHfTX9nwCNeJCO+4kWxLrvI0kz8dd6ermL3F+IctXFuGO5ZxPGwZAB
Vq1LjRnPmsOwJ4OqyQDVl7Rf7TflKlDnMYIDXjCCA1oCAQEwcDBpMRMwEQYKCZImiZPyLGQB
GRYDb3JnMRgwFgYKCZImiZPyLGQBGRYIRE9FR3JpZHMxIDAeBgNVBAsTF0NlcnRpZmljYXRl
IEF1dGhvcml0aWVzMRYwFAYDVQQDEw1ET0VHcmlkcyBDQSAxAgMAoWcwCQYFKw4DAhoFAKCC
AcMwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTAwMzE5MDEx
MTE1WjAjBgkqhkiG9w0BCQQxFgQUDWVt/RDszyoVr1CGTYonVGZ4M5cwXwYJKoZIhvcNAQkP
MVIwUDALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3
DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMH8GCSsGAQQBgjcQBDFyMHAwaTETMBEG
CgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERPRUdyaWRzMSAwHgYDVQQLExdD
ZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3JpZHMgQ0EgMQIDAKFnMIGB
BgsqhkiG9w0BCRACCzFyoHAwaTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixk
ARkWCERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UE
AxMNRE9FR3JpZHMgQ0EgMQIDAKFnMA0GCSqGSIb3DQEBAQUABIIBAIsgUf2uvuEeyHHL5IYo
4AUkMk2q87mWy9z8wtjg3bQ1o43XbRefwtUzWJJeRCIB67HTyhk7PzLwyeu4DPKkoBV6GgSQ
6yCYOANf9bWRMeavCAcFkZIDnOO2YSKrrLZUrGpdx5qI+DVg9H1MXHcotQ4109gZxEpKnCM0
/qECpLMQTCZtoieYAXr5xN0V4IdqyMm65AOSIEdm7pRqeuK5aqa27RCZMV2D2zkGfXfTyuU3
qZI7SSH1LsGDcPuV2Gz4KKbRvdPHkAypfevHfMrvwafcaYqoq3NNW0f8fO08D7KxBUTebNZl
eheDQDnJdyaTFkwevdp9HGNHrkAGnT0lef8AAAAAAAA=
--------------ms070408020806080508010302--

From general-return-1217-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 19 01:21:38 2010
Return-Path: <general-return-1217-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73488 invoked from network); 19 Mar 2010 01:21:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Mar 2010 01:21:38 -0000
Received: (qmail 8342 invoked by uid 500); 19 Mar 2010 01:21:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 8305 invoked by uid 500); 19 Mar 2010 01:21:37 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 8297 invoked by uid 99); 19 Mar 2010 01:21:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 01:21:37 +0000
X-ASF-Spam-Status: No, hits=-0.2 required=10.0
	tests=AWL,SPF_HELO_PASS,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [64.81.36.244] (HELO foobar.kobold.org) (64.81.36.244)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 01:21:32 +0000
Received: from owl.kobold.org (owl.kobold.org [192.168.1.7])
	(authenticated bits=0)
	by foobar.kobold.org (8.14.1/8.14.1) with ESMTP id o2J07JpR008123
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Thu, 18 Mar 2010 17:07:20 -0700
Message-ID: <4BA2D187.2080102@hep.caltech.edu>
Date: Thu, 18 Mar 2010 18:21:11 -0700
From: Michael Thomas <thomas@hep.caltech.edu>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.7) Gecko/20100120 Fedora/3.0.1-1.fc12 Thunderbird/3.0.1
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: rack awareness help
References: <C7B48351.36DF%awittenauer@linkedin.com>	 <4B8FC1CC.2090108@apache.org>	 <1cbd6f831003041539r75b360ecrb8e604d4dce1350f@mail.gmail.com>	 <c9118d4c1003041830h47961fc4p323c2ebdd0df2ffc@mail.gmail.com> <1cbd6f831003172034r3d699283hd4e41f5fc0f37231@mail.gmail.com>
In-Reply-To: <1cbd6f831003172034r3d699283hd4e41f5fc0f37231@mail.gmail.com>
Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms040604060707030505030707"

--------------ms040604060707030505030707
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: quoted-printable

On 03/17/2010 08:34 PM, Mag Gam wrote:
> Well,  I didn't really solve the problem. Now I have even more question=
s.
>
> I came across this script,
> http://wiki.apache.org/hadoop/topology_rack_awareness_scripts
>
> but it makes no sense to me! Can someone please try to explain what
> its trying to do?
>
>
> MikeThomas:
>
> Your script isn't working for me. I think there are some syntax
> errors. Is this how its supposed to look: http://pastebin.ca/1844287

Not quite.  A couple of lines got incorrectly wrapped.  It should look=20
like this:

http://pastebin.ca/1845286

--Mike

> On Thu, Mar 4, 2010 at 10:30 PM, Jeff Hammerbacher<hammer@cloudera.com>=
  wrote:
>> Hey Mag,
>>
>> Glad you have solved the problem. I've created a JIRA ticket to improv=
e the
>> existing documentation: https://issues.apache.org/jira/browse/HADOOP-6=
616.
>> If you have some time, it would be useful to hear what could be added =
to the
>> existing documentation that would have helped you figure this out soon=
er.
>>
>> Thanks,
>> Jeff
>>
>> On Thu, Mar 4, 2010 at 3:39 PM, Mag Gam<magawake@gmail.com>  wrote:
>>
>>> Thanks everyone for explaining this to me instead of giving me RTFM!
>>>
>>> I will play around with it and see how far I get.
>>>
>>>
>>>
>>> On Thu, Mar 4, 2010 at 9:21 AM, Steve Loughran<stevel@apache.org>  wr=
ote:
>>>> Allen Wittenauer wrote:
>>>>>
>>>>> On 3/3/10 5:01 PM, "Mag Gam"<magawake@gmail.com>  wrote:
>>>>>
>>>>>> Thanks Alan! Your presentation is very nice!
>>>>>
>>>>> Thanks. :)
>>>>>
>>>>>> "If you don't provide a script for rack awareness, it treats every=

>>>>>> node as if it was its own rack". I am using the default settings a=
nd
>>>>>> the report still says only 1 rack.
>>>>>
>>>>> Let's take a different approach to convince you. :)
>>>>>
>>>>> Think about the question:  Is there a difference between all nodes =
in
>>> one
>>>>> rack vs. every node acting as a lone rack?
>>>>>
>>>>> The answer is no, there isn't any difference.  In both cases, all c=
opies
>>>>> of
>>>>> the blocks can go to pretty much any node. When a MR job runs, ever=
y
>>> node
>>>>> would either be considered 'off rack' or 'rack-local'.
>>>>>
>>>>> So there is no difference.
>>>>>
>>>>>
>>>>>> Do you mind sharing a script with us on how you determine a rack? =
and
>>>>>> a sample<configuration>  </configuration>  syntax?
>>>>>
>>>>> Michael has already posted his, so I'll skip this one. :)
>>>>>
>>>>
>>>> Think Mag probably wanted a shell script.
>>>>
>>>> Mag, give your machines IPv4 addresses that map to rack number. 10.1=
=2E1.*
>>> for
>>>> rack one, 10.1.2.* for rack 2, etc. Then just filter out the IP addr=
ess
>>> by
>>>> the top bytes, returning "10.1.1" for everything in rack one, "10.1.=
2"
>>> for
>>>> rack 2; Hadoop will be happy
>>>>
>>>
>>



--------------ms040604060707030505030707
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIMVjCC
A/gwggLgoAMCAQICASkwDQYJKoZIhvcNAQEFBQAwdTETMBEGCgmSJomT8ixkARkWA25ldDES
MBAGCgmSJomT8ixkARkWAkVTMQ4wDAYDVQQKEwVFU25ldDEgMB4GA1UECxMXQ2VydGlmaWNh
dGUgQXV0aG9yaXRpZXMxGDAWBgNVBAMTD0VTbmV0IFJvb3QgQ0EgMTAeFw0wMjEyMDUwODAw
MDBaFw0xMzAxMjUwODAwMDBaMGkxEzARBgoJkiaJk/IsZAEZFgNvcmcxGDAWBgoJkiaJk/Is
ZAEZFghET0VHcmlkczEgMB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9yaXRpZXMxFjAUBgNV
BAMTDURPRUdyaWRzIENBIDEwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC09dYj
YaPbCD5mtbiQb7Ka3y1qAm0ZcqKCFciWcfe8Kwcuy9tjHuIsLf9ZItdkDW4xy8sua9nJlx3K
lwjtumTMtOtg35KZCknUd8KM4VGTSFdLVG9AbNayef76caVCGM1+jyF0Lq03kauGOPTcNfZe
1TZa3e1c9rc8ljV5OSWa/mfsCACyS5zFIWu0yIDNyJdf+n0hwaPN53wllpJ30taD+JBjQ7h2
k4xRWzeaznLOb9OztZVRA/1sVze+iczFh2xwa4VdGy0eIIPw1pfvYwxO36rm0S109qvbsNla
roPRbxerPKakQLpKe034Xcx7gBPqUk/FxoRRWin5EWN3rz9LAgMBAAGjgZ4wgZswDgYDVR0P
AQH/BAQDAgGGMBEGCWCGSAGG+EIBAQQEAwIAhzAdBgNVHQ4EFgQUyhkdEo5upDhdQtQxDgjb
2Y0XDV0wHwYDVR0jBBgwFoAUvF1NSC/4NZRZq1yJSz7RsjoUAeowDwYDVR0TAQH/BAUwAwEB
/zAlBgNVHREEHjAcgRpET0VHcmlkcy1DQS0xQGRvZWdyaWRzLm9yZzANBgkqhkiG9w0BAQUF
AAOCAQEAZNVrIDLqe39CEOiJt7Q7EpBPhAihMvDTSf/42u0SMbUmChww4mLmph5DBghZUVF8
Yn59kRZMn1QLOtO1HzLqvAvPITacZVPlJgG2IXzlR636YghZFAycbIUEOJDBHR4vtQO1KDxg
ZwvAbtmKIoxvhUCq2xsfFt9kCBBn+JYtQ6O5LsBJq3PmuubeMcc7mbQAfJZ7h/3QghgkFIhm
E1+LBXPJbkuP8vgfg6h2BKoAf5TFfZECgGZKimfN110tBvfedGZwYYd3/GsJc83B0JN1gny0
gqNVPm392UchXGeBRrHnm2gkhIkr48Oq6EmNGV9/a6XfbplQW/JWbtPVPWkaizCCBCkwggMR
oAMCAQICAwChZzANBgkqhkiG9w0BAQUFADBpMRMwEQYKCZImiZPyLGQBGRYDb3JnMRgwFgYK
CZImiZPyLGQBGRYIRE9FR3JpZHMxIDAeBgNVBAsTF0NlcnRpZmljYXRlIEF1dGhvcml0aWVz
MRYwFAYDVQQDEw1ET0VHcmlkcyBDQSAxMB4XDTEwMDEyNTE4MzEyMFoXDTExMDEyNTE4MzEy
MFowYDETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCGRvZWdyaWRzMQ8w
DQYDVQQLEwZQZW9wbGUxHjAcBgNVBAMTFU1pY2hhZWwgVGhvbWFzIDcyNTM3NDCCASIwDQYJ
KoZIhvcNAQEBBQADggEPADCCAQoCggEBALchCLXStiJKw5u2+nFW0020jqo1X8/kRB9kus8a
6Op5ds3j5O3IZkyZ41tQwE9Rc0yRDiDGpT1jL/n2LK1HPXvZbasmPCxxTMY3kNTm4fWAnRS7
UzLo0Adaz81bcT8Buo7Il1FCS684LhsK19JChqU2/iY8G7H0uBImj6QDLR0+w5QsLfD7aOXx
M8Njhse1ZqMyjwjhw5FqWyWNhlKi1oW42eXtYhbsqyAQZSXLoz2SwqBnLTRqcRMn4GgvB83Z
1FObyb79HFiuqlbaMlXWy0s8ywajJnMG8LsgwkNJZQLJsPalz5rwe/pOG1abasLMBVpJGGSB
AQktY8s5M+U9RlMCAwEAAaOB4jCB3zARBglghkgBhvhCAQEEBAMCBeAwDgYDVR0PAQH/BAQD
AgTwMDYGA1UdIAQvMC0wDQYLKoZIhvdMAwcBAwEwDAYKKoZIhvdMBQICATAOBgwqhkiG90wF
AgMCAQEwPgYDVR0fBDcwNTAzoDGgL4YtaHR0cDovL2NybC5kb2Vncmlkcy5vcmcvMWMzZjJj
YTgvMWMzZjJjYTguY3JsMCEGA1UdEQQaMBiBFnRob21hc0BoZXAuY2FsdGVjaC5lZHUwHwYD
VR0jBBgwFoAUyhkdEo5upDhdQtQxDgjb2Y0XDV0wDQYJKoZIhvcNAQEFBQADggEBAGcleFpm
d9Ho37gHZrrF0zulSQ3Aemrbx2O4EVJotQbtAs2v4nnH0eAa0F37BXRA6DAvgaC4YODA6z5M
HHE2mnsIebv0nS8DLj1yN7C1nqDSUXlaIzExEGQ1vXG2XlVo6pMZgJd2ML9+dyUUOcnNj0kQ
GeMysWBHbDChdfVG8y2PuGzoFY/ikNtWWhTNcPVZHIL5I2JCYqV5lyLQhlkC0Q5eqbqYKAq1
GuQmLO8hPfzkb6qbAkzxB8/FFqjlPgBgFcd9Nf2fAI14kI77iRbEuu8jSTPx13p6uYvcX4hy
1cW4Y7lnE8bBkAFWrUuNGc+aw7Ang6rJANWXtF/tN+UqUOcwggQpMIIDEaADAgECAgMAoWcw
DQYJKoZIhvcNAQEFBQAwaTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkW
CERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UEAxMN
RE9FR3JpZHMgQ0EgMTAeFw0xMDAxMjUxODMxMjBaFw0xMTAxMjUxODMxMjBaMGAxEzARBgoJ
kiaJk/IsZAEZFgNvcmcxGDAWBgoJkiaJk/IsZAEZFghkb2VncmlkczEPMA0GA1UECxMGUGVv
cGxlMR4wHAYDVQQDExVNaWNoYWVsIFRob21hcyA3MjUzNzQwggEiMA0GCSqGSIb3DQEBAQUA
A4IBDwAwggEKAoIBAQC3IQi10rYiSsObtvpxVtNNtI6qNV/P5EQfZLrPGujqeXbN4+TtyGZM
meNbUMBPUXNMkQ4gxqU9Yy/59iytRz172W2rJjwscUzGN5DU5uH1gJ0Uu1My6NAHWs/NW3E/
AbqOyJdRQkuvOC4bCtfSQoalNv4mPBux9LgSJo+kAy0dPsOULC3w+2jl8TPDY4bHtWajMo8I
4cORalsljYZSotaFuNnl7WIW7KsgEGUly6M9ksKgZy00anETJ+BoLwfN2dRTm8m+/RxYrqpW
2jJV1stLPMsGoyZzBvC7IMJDSWUCybD2pc+a8Hv6ThtWm2rCzAVaSRhkgQEJLWPLOTPlPUZT
AgMBAAGjgeIwgd8wEQYJYIZIAYb4QgEBBAQDAgXgMA4GA1UdDwEB/wQEAwIE8DA2BgNVHSAE
LzAtMA0GCyqGSIb3TAMHAQMBMAwGCiqGSIb3TAUCAgEwDgYMKoZIhvdMBQIDAgEBMD4GA1Ud
HwQ3MDUwM6AxoC+GLWh0dHA6Ly9jcmwuZG9lZ3JpZHMub3JnLzFjM2YyY2E4LzFjM2YyY2E4
LmNybDAhBgNVHREEGjAYgRZ0aG9tYXNAaGVwLmNhbHRlY2guZWR1MB8GA1UdIwQYMBaAFMoZ
HRKObqQ4XULUMQ4I29mNFw1dMA0GCSqGSIb3DQEBBQUAA4IBAQBnJXhaZnfR6N+4B2a6xdM7
pUkNwHpq28djuBFSaLUG7QLNr+J5x9HgGtBd+wV0QOgwL4GguGDgwOs+TBxxNpp7CHm79J0v
Ay49cjewtZ6g0lF5WiMxMRBkNb1xtl5VaOqTGYCXdjC/fnclFDnJzY9JEBnjMrFgR2wwoXX1
RvMtj7hs6BWP4pDbVloUzXD1WRyC+SNiQmKleZci0IZZAtEOXqm6mCgKtRrkJizvIT385G+q
mwJM8QfPxRao5T4AYBXHfTX9nwCNeJCO+4kWxLrvI0kz8dd6ermL3F+IctXFuGO5ZxPGwZAB
Vq1LjRnPmsOwJ4OqyQDVl7Rf7TflKlDnMYIDXjCCA1oCAQEwcDBpMRMwEQYKCZImiZPyLGQB
GRYDb3JnMRgwFgYKCZImiZPyLGQBGRYIRE9FR3JpZHMxIDAeBgNVBAsTF0NlcnRpZmljYXRl
IEF1dGhvcml0aWVzMRYwFAYDVQQDEw1ET0VHcmlkcyBDQSAxAgMAoWcwCQYFKw4DAhoFAKCC
AcMwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTAwMzE5MDEy
MTExWjAjBgkqhkiG9w0BCQQxFgQUwl3m28WplQ6TW+zdQty5aQUqNaswXwYJKoZIhvcNAQkP
MVIwUDALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3
DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMH8GCSsGAQQBgjcQBDFyMHAwaTETMBEG
CgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERPRUdyaWRzMSAwHgYDVQQLExdD
ZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3JpZHMgQ0EgMQIDAKFnMIGB
BgsqhkiG9w0BCRACCzFyoHAwaTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixk
ARkWCERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UE
AxMNRE9FR3JpZHMgQ0EgMQIDAKFnMA0GCSqGSIb3DQEBAQUABIIBALTR3JNn2V0UZnPxSeCD
itN+DYSX88fpuGGSSJlgUK8aQMv0lXRFLl408TuYMLbkBZ4hmL9ESuqeSv8QGUEazaRCXNxD
lpHBwSekbjwt/r0+vBbhY1Kul1STsmmvhSGgL/NbfyBnthtf5C54naTkWxII1DdykctogyqX
WLf8uShitfu8LJSLdWMtDEf8eiRaNGeZNxH/I8C9QwRBCIbcApyir5s7ZK51iXz98X6np78e
15IM8EX7mjAUTSaxp/+9/IhApI+lq/qVB0ZktxDgvZfotlVkjC6PHrLhV2PXug4rm7Ykidpz
NadzAwUvvG9DS3dxJJat6UOOjo/Pzh7Emr4AAAAAAAA=
--------------ms040604060707030505030707--

From general-return-1218-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 19 01:27:18 2010
Return-Path: <general-return-1218-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 86910 invoked from network); 19 Mar 2010 01:27:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Mar 2010 01:27:18 -0000
Received: (qmail 18901 invoked by uid 500); 19 Mar 2010 01:27:17 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18869 invoked by uid 500); 19 Mar 2010 01:27:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18861 invoked by uid 99); 19 Mar 2010 01:27:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 01:27:17 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_HELO_PASS,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [64.81.36.244] (HELO foobar.kobold.org) (64.81.36.244)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 01:27:10 +0000
Received: from owl.kobold.org (owl.kobold.org [192.168.1.7])
	(authenticated bits=0)
	by foobar.kobold.org (8.14.1/8.14.1) with ESMTP id o2J0CtWx008134
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Thu, 18 Mar 2010 17:12:55 -0700
Message-ID: <4BA2D2D7.9060008@hep.caltech.edu>
Date: Thu, 18 Mar 2010 18:26:47 -0700
From: Michael Thomas <thomas@hep.caltech.edu>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.7) Gecko/20100120 Fedora/3.0.1-1.fc12 Thunderbird/3.0.1
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: rack awareness help
References: <C7B48351.36DF%awittenauer@linkedin.com>	 <4B8FC1CC.2090108@apache.org>	 <1cbd6f831003041539r75b360ecrb8e604d4dce1350f@mail.gmail.com>	 <c9118d4c1003041830h47961fc4p323c2ebdd0df2ffc@mail.gmail.com> <1cbd6f831003172034r3d699283hd4e41f5fc0f37231@mail.gmail.com> <4BA2D187.2080102@hep.caltech.edu>
In-Reply-To: <4BA2D187.2080102@hep.caltech.edu>
Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms030209070508020206040200"
X-Virus-Checked: Checked by ClamAV on apache.org

--------------ms030209070508020206040200
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: quoted-printable

On 03/18/2010 06:21 PM, Michael Thomas wrote:
> On 03/17/2010 08:34 PM, Mag Gam wrote:
>> Well, I didn't really solve the problem. Now I have even more question=
s.
>>
>> I came across this script,
>> http://wiki.apache.org/hadoop/topology_rack_awareness_scripts
>>
>> but it makes no sense to me! Can someone please try to explain what
>> its trying to do?
>>
>>
>> MikeThomas:
>>
>> Your script isn't working for me. I think there are some syntax
>> errors. Is this how its supposed to look: http://pastebin.ca/1844287
>
> Not quite. A couple of lines got incorrectly wrapped. It should look
> like this:
>
> http://pastebin.ca/1845286

One more miswrapped line.  I hate auto-wrapping...

http://pastebin.ca/1845290

--Mike


> --Mike
>
>> On Thu, Mar 4, 2010 at 10:30 PM, Jeff
>> Hammerbacher<hammer@cloudera.com> wrote:
>>> Hey Mag,
>>>
>>> Glad you have solved the problem. I've created a JIRA ticket to
>>> improve the
>>> existing documentation:
>>> https://issues.apache.org/jira/browse/HADOOP-6616.
>>> If you have some time, it would be useful to hear what could be added=

>>> to the
>>> existing documentation that would have helped you figure this out
>>> sooner.
>>>
>>> Thanks,
>>> Jeff
>>>
>>> On Thu, Mar 4, 2010 at 3:39 PM, Mag Gam<magawake@gmail.com> wrote:
>>>
>>>> Thanks everyone for explaining this to me instead of giving me RTFM!=

>>>>
>>>> I will play around with it and see how far I get.
>>>>
>>>>
>>>>
>>>> On Thu, Mar 4, 2010 at 9:21 AM, Steve Loughran<stevel@apache.org>
>>>> wrote:
>>>>> Allen Wittenauer wrote:
>>>>>>
>>>>>> On 3/3/10 5:01 PM, "Mag Gam"<magawake@gmail.com> wrote:
>>>>>>
>>>>>>> Thanks Alan! Your presentation is very nice!
>>>>>>
>>>>>> Thanks. :)
>>>>>>
>>>>>>> "If you don't provide a script for rack awareness, it treats ever=
y
>>>>>>> node as if it was its own rack". I am using the default settings =
and
>>>>>>> the report still says only 1 rack.
>>>>>>
>>>>>> Let's take a different approach to convince you. :)
>>>>>>
>>>>>> Think about the question: Is there a difference between all nodes =
in
>>>> one
>>>>>> rack vs. every node acting as a lone rack?
>>>>>>
>>>>>> The answer is no, there isn't any difference. In both cases, all
>>>>>> copies
>>>>>> of
>>>>>> the blocks can go to pretty much any node. When a MR job runs, eve=
ry
>>>> node
>>>>>> would either be considered 'off rack' or 'rack-local'.
>>>>>>
>>>>>> So there is no difference.
>>>>>>
>>>>>>
>>>>>>> Do you mind sharing a script with us on how you determine a rack?=

>>>>>>> and
>>>>>>> a sample<configuration> </configuration> syntax?
>>>>>>
>>>>>> Michael has already posted his, so I'll skip this one. :)
>>>>>>
>>>>>
>>>>> Think Mag probably wanted a shell script.
>>>>>
>>>>> Mag, give your machines IPv4 addresses that map to rack number.
>>>>> 10.1.1.*
>>>> for
>>>>> rack one, 10.1.2.* for rack 2, etc. Then just filter out the IP
>>>>> address
>>>> by
>>>>> the top bytes, returning "10.1.1" for everything in rack one, "10.1=
=2E2"
>>>> for
>>>>> rack 2; Hadoop will be happy
>>>>>
>>>>
>>>
>
>



--------------ms030209070508020206040200
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIMVjCC
A/gwggLgoAMCAQICASkwDQYJKoZIhvcNAQEFBQAwdTETMBEGCgmSJomT8ixkARkWA25ldDES
MBAGCgmSJomT8ixkARkWAkVTMQ4wDAYDVQQKEwVFU25ldDEgMB4GA1UECxMXQ2VydGlmaWNh
dGUgQXV0aG9yaXRpZXMxGDAWBgNVBAMTD0VTbmV0IFJvb3QgQ0EgMTAeFw0wMjEyMDUwODAw
MDBaFw0xMzAxMjUwODAwMDBaMGkxEzARBgoJkiaJk/IsZAEZFgNvcmcxGDAWBgoJkiaJk/Is
ZAEZFghET0VHcmlkczEgMB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9yaXRpZXMxFjAUBgNV
BAMTDURPRUdyaWRzIENBIDEwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC09dYj
YaPbCD5mtbiQb7Ka3y1qAm0ZcqKCFciWcfe8Kwcuy9tjHuIsLf9ZItdkDW4xy8sua9nJlx3K
lwjtumTMtOtg35KZCknUd8KM4VGTSFdLVG9AbNayef76caVCGM1+jyF0Lq03kauGOPTcNfZe
1TZa3e1c9rc8ljV5OSWa/mfsCACyS5zFIWu0yIDNyJdf+n0hwaPN53wllpJ30taD+JBjQ7h2
k4xRWzeaznLOb9OztZVRA/1sVze+iczFh2xwa4VdGy0eIIPw1pfvYwxO36rm0S109qvbsNla
roPRbxerPKakQLpKe034Xcx7gBPqUk/FxoRRWin5EWN3rz9LAgMBAAGjgZ4wgZswDgYDVR0P
AQH/BAQDAgGGMBEGCWCGSAGG+EIBAQQEAwIAhzAdBgNVHQ4EFgQUyhkdEo5upDhdQtQxDgjb
2Y0XDV0wHwYDVR0jBBgwFoAUvF1NSC/4NZRZq1yJSz7RsjoUAeowDwYDVR0TAQH/BAUwAwEB
/zAlBgNVHREEHjAcgRpET0VHcmlkcy1DQS0xQGRvZWdyaWRzLm9yZzANBgkqhkiG9w0BAQUF
AAOCAQEAZNVrIDLqe39CEOiJt7Q7EpBPhAihMvDTSf/42u0SMbUmChww4mLmph5DBghZUVF8
Yn59kRZMn1QLOtO1HzLqvAvPITacZVPlJgG2IXzlR636YghZFAycbIUEOJDBHR4vtQO1KDxg
ZwvAbtmKIoxvhUCq2xsfFt9kCBBn+JYtQ6O5LsBJq3PmuubeMcc7mbQAfJZ7h/3QghgkFIhm
E1+LBXPJbkuP8vgfg6h2BKoAf5TFfZECgGZKimfN110tBvfedGZwYYd3/GsJc83B0JN1gny0
gqNVPm392UchXGeBRrHnm2gkhIkr48Oq6EmNGV9/a6XfbplQW/JWbtPVPWkaizCCBCkwggMR
oAMCAQICAwChZzANBgkqhkiG9w0BAQUFADBpMRMwEQYKCZImiZPyLGQBGRYDb3JnMRgwFgYK
CZImiZPyLGQBGRYIRE9FR3JpZHMxIDAeBgNVBAsTF0NlcnRpZmljYXRlIEF1dGhvcml0aWVz
MRYwFAYDVQQDEw1ET0VHcmlkcyBDQSAxMB4XDTEwMDEyNTE4MzEyMFoXDTExMDEyNTE4MzEy
MFowYDETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCGRvZWdyaWRzMQ8w
DQYDVQQLEwZQZW9wbGUxHjAcBgNVBAMTFU1pY2hhZWwgVGhvbWFzIDcyNTM3NDCCASIwDQYJ
KoZIhvcNAQEBBQADggEPADCCAQoCggEBALchCLXStiJKw5u2+nFW0020jqo1X8/kRB9kus8a
6Op5ds3j5O3IZkyZ41tQwE9Rc0yRDiDGpT1jL/n2LK1HPXvZbasmPCxxTMY3kNTm4fWAnRS7
UzLo0Adaz81bcT8Buo7Il1FCS684LhsK19JChqU2/iY8G7H0uBImj6QDLR0+w5QsLfD7aOXx
M8Njhse1ZqMyjwjhw5FqWyWNhlKi1oW42eXtYhbsqyAQZSXLoz2SwqBnLTRqcRMn4GgvB83Z
1FObyb79HFiuqlbaMlXWy0s8ywajJnMG8LsgwkNJZQLJsPalz5rwe/pOG1abasLMBVpJGGSB
AQktY8s5M+U9RlMCAwEAAaOB4jCB3zARBglghkgBhvhCAQEEBAMCBeAwDgYDVR0PAQH/BAQD
AgTwMDYGA1UdIAQvMC0wDQYLKoZIhvdMAwcBAwEwDAYKKoZIhvdMBQICATAOBgwqhkiG90wF
AgMCAQEwPgYDVR0fBDcwNTAzoDGgL4YtaHR0cDovL2NybC5kb2Vncmlkcy5vcmcvMWMzZjJj
YTgvMWMzZjJjYTguY3JsMCEGA1UdEQQaMBiBFnRob21hc0BoZXAuY2FsdGVjaC5lZHUwHwYD
VR0jBBgwFoAUyhkdEo5upDhdQtQxDgjb2Y0XDV0wDQYJKoZIhvcNAQEFBQADggEBAGcleFpm
d9Ho37gHZrrF0zulSQ3Aemrbx2O4EVJotQbtAs2v4nnH0eAa0F37BXRA6DAvgaC4YODA6z5M
HHE2mnsIebv0nS8DLj1yN7C1nqDSUXlaIzExEGQ1vXG2XlVo6pMZgJd2ML9+dyUUOcnNj0kQ
GeMysWBHbDChdfVG8y2PuGzoFY/ikNtWWhTNcPVZHIL5I2JCYqV5lyLQhlkC0Q5eqbqYKAq1
GuQmLO8hPfzkb6qbAkzxB8/FFqjlPgBgFcd9Nf2fAI14kI77iRbEuu8jSTPx13p6uYvcX4hy
1cW4Y7lnE8bBkAFWrUuNGc+aw7Ang6rJANWXtF/tN+UqUOcwggQpMIIDEaADAgECAgMAoWcw
DQYJKoZIhvcNAQEFBQAwaTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkW
CERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UEAxMN
RE9FR3JpZHMgQ0EgMTAeFw0xMDAxMjUxODMxMjBaFw0xMTAxMjUxODMxMjBaMGAxEzARBgoJ
kiaJk/IsZAEZFgNvcmcxGDAWBgoJkiaJk/IsZAEZFghkb2VncmlkczEPMA0GA1UECxMGUGVv
cGxlMR4wHAYDVQQDExVNaWNoYWVsIFRob21hcyA3MjUzNzQwggEiMA0GCSqGSIb3DQEBAQUA
A4IBDwAwggEKAoIBAQC3IQi10rYiSsObtvpxVtNNtI6qNV/P5EQfZLrPGujqeXbN4+TtyGZM
meNbUMBPUXNMkQ4gxqU9Yy/59iytRz172W2rJjwscUzGN5DU5uH1gJ0Uu1My6NAHWs/NW3E/
AbqOyJdRQkuvOC4bCtfSQoalNv4mPBux9LgSJo+kAy0dPsOULC3w+2jl8TPDY4bHtWajMo8I
4cORalsljYZSotaFuNnl7WIW7KsgEGUly6M9ksKgZy00anETJ+BoLwfN2dRTm8m+/RxYrqpW
2jJV1stLPMsGoyZzBvC7IMJDSWUCybD2pc+a8Hv6ThtWm2rCzAVaSRhkgQEJLWPLOTPlPUZT
AgMBAAGjgeIwgd8wEQYJYIZIAYb4QgEBBAQDAgXgMA4GA1UdDwEB/wQEAwIE8DA2BgNVHSAE
LzAtMA0GCyqGSIb3TAMHAQMBMAwGCiqGSIb3TAUCAgEwDgYMKoZIhvdMBQIDAgEBMD4GA1Ud
HwQ3MDUwM6AxoC+GLWh0dHA6Ly9jcmwuZG9lZ3JpZHMub3JnLzFjM2YyY2E4LzFjM2YyY2E4
LmNybDAhBgNVHREEGjAYgRZ0aG9tYXNAaGVwLmNhbHRlY2guZWR1MB8GA1UdIwQYMBaAFMoZ
HRKObqQ4XULUMQ4I29mNFw1dMA0GCSqGSIb3DQEBBQUAA4IBAQBnJXhaZnfR6N+4B2a6xdM7
pUkNwHpq28djuBFSaLUG7QLNr+J5x9HgGtBd+wV0QOgwL4GguGDgwOs+TBxxNpp7CHm79J0v
Ay49cjewtZ6g0lF5WiMxMRBkNb1xtl5VaOqTGYCXdjC/fnclFDnJzY9JEBnjMrFgR2wwoXX1
RvMtj7hs6BWP4pDbVloUzXD1WRyC+SNiQmKleZci0IZZAtEOXqm6mCgKtRrkJizvIT385G+q
mwJM8QfPxRao5T4AYBXHfTX9nwCNeJCO+4kWxLrvI0kz8dd6ermL3F+IctXFuGO5ZxPGwZAB
Vq1LjRnPmsOwJ4OqyQDVl7Rf7TflKlDnMYIDXjCCA1oCAQEwcDBpMRMwEQYKCZImiZPyLGQB
GRYDb3JnMRgwFgYKCZImiZPyLGQBGRYIRE9FR3JpZHMxIDAeBgNVBAsTF0NlcnRpZmljYXRl
IEF1dGhvcml0aWVzMRYwFAYDVQQDEw1ET0VHcmlkcyBDQSAxAgMAoWcwCQYFKw4DAhoFAKCC
AcMwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTAwMzE5MDEy
NjQ3WjAjBgkqhkiG9w0BCQQxFgQUZ1Oi5s+YmPIlPsP8TBLr2gpd4IgwXwYJKoZIhvcNAQkP
MVIwUDALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3
DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMH8GCSsGAQQBgjcQBDFyMHAwaTETMBEG
CgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERPRUdyaWRzMSAwHgYDVQQLExdD
ZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3JpZHMgQ0EgMQIDAKFnMIGB
BgsqhkiG9w0BCRACCzFyoHAwaTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixk
ARkWCERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UE
AxMNRE9FR3JpZHMgQ0EgMQIDAKFnMA0GCSqGSIb3DQEBAQUABIIBAImElMXK9+1TATh/Bhv9
NI9viWhFP5XqqdTNY6BeMkEw7SSFo6U4VboLrdXOaaqh5EcBbZfwFTBojRalneK+VvYgLz3a
6iRDjF0LlBSpqD47Rv8UeK014vnkeLrarQud38iUtx8Zm/crp3AdIENpbVKliRmr9n/q9dU0
lxPOi8eewSH5Su9r65en7ffkX8HLH8LsQELW9ivjQsGXR/b1qODo1XplUZzTZcJzeNkS5FtS
CRLHFGbt/KcJxoRvRa9uVTYjSUMiNqIzKl2sqwV2OpzwMHlqwyZVi9sqqGEYW3K4B62cjlTM
m2o9RD6mUd4hsHyeJJmKDnk/gQphZaIV88kAAAAAAAA=
--------------ms030209070508020206040200--

From general-return-1219-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 19 05:01:01 2010
Return-Path: <general-return-1219-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 19284 invoked from network); 19 Mar 2010 05:01:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Mar 2010 05:01:01 -0000
Received: (qmail 39487 invoked by uid 500); 19 Mar 2010 05:01:00 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 39349 invoked by uid 500); 19 Mar 2010 05:01:00 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 39341 invoked by uid 99); 19 Mar 2010 05:00:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 05:00:59 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ctubbsii@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 05:00:54 +0000
Received: by wyf19 with SMTP id 19so1308360wyf.35
        for <general@hadoop.apache.org>; Thu, 18 Mar 2010 22:00:33 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=2smDWXzs3A9a5SkoZbwZwrMQsNmIeDpe7Pd8kVxF6ws=;
        b=upOqqL1603dqBWRVUUyGuPonBbC/+5P9U2E6OB69BvyEjrIscqFi1Y/ys6lRfRiGHg
         7V2RE6QoRpyYsBBBrpg9/y3Tl3zUZKQDbRDLycDuyCda7mUQcaOeKzDXn/MlaAyx8AiW
         b+UIsddNPry5JGQrmHhW6II/ZefuscVL/B5Nc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=rUvwAG0ZW98uLCQC2c68c4nxvrQsSpOExZ6TOdsNcyREsl7AVq9F0ib75am4yECmHm
         9k9PbmNXZj6wOQkAIGfykz9ogAsnjg8quJLq+SBECfkk3yWBB4rLro3lTFn0PhdXr1Wr
         6u1JkLNXhiDNn100ftCOuMw7aVsyN9ltRMo80=
MIME-Version: 1.0
Received: by 10.216.87.16 with SMTP id x16mr1461105wee.27.1268974833498; Thu, 
	18 Mar 2010 22:00:33 -0700 (PDT)
In-Reply-To: <1cbd6f831003181741t2f538eb8xc872cba6dfe00761@mail.gmail.com>
References: <C7B48351.36DF%awittenauer@linkedin.com>
	 <4B8FC1CC.2090108@apache.org>
	 <1cbd6f831003041539r75b360ecrb8e604d4dce1350f@mail.gmail.com>
	 <c9118d4c1003041830h47961fc4p323c2ebdd0df2ffc@mail.gmail.com>
	 <1cbd6f831003172034r3d699283hd4e41f5fc0f37231@mail.gmail.com>
	 <9db00c201003180015n1e4e8941i2c53296edd21258f@mail.gmail.com>
	 <1cbd6f831003181741t2f538eb8xc872cba6dfe00761@mail.gmail.com>
Date: Fri, 19 Mar 2010 01:00:33 -0400
Message-ID: <9db00c201003182200y5d572e87xba1e3e8cf3bd539c@mail.gmail.com>
Subject: Re: rack awareness help
From: Christopher Tubbs <ctubbsii@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

You only specify the script on the namenode.
So, you could do something like:

#!/bin/bash
#rack_decider.sh

if [ $1 =3D "server1.mydomain" -o $1 =3D "192.168.0.1" ] ; then
  echo rack1
elif [ $1 =3D "server2.mydomain" -o $1 =3D "192.168.0.2" ] ; then
  echo rack1
elif [ $1 =3D "server3.mydomain" -o $1 =3D "192.168.0.3" ] ; then
  echo rack2
elif [ $1 =3D "server4.mydomain" -o $1 =3D "192.168.0.4" ] ; then
  echo rack2
else
  echo unknown_rack
fi
# EOF

Of course, this is by far the most basic script you could have (I'm
not sure why it wasn't offered as an example instead of a more
complicated one).

On Thu, Mar 18, 2010 at 8:41 PM, Mag Gam <magawake@gmail.com> wrote:
> Chris:
>
> This clears up my questions a lot! Thankyou.
>
> So, if I have 4 data servers and I want 2 racks. I can do this
>
> #!/bin/bash
> #rack1.sh
> echo rack1
>
> #bin/bash
> #rack2.sh
> echo rack2
>
>
> So, I can do this for 2 servers
>
>
> <property>
> =A0<name>topology.script.file.name</name>
> =A0<value>rack1.sh</value>
> </property>
>
> And for the other 2 servers, I can do this:
>
>
> <property>
> =A0<name>topology.script.file.name</name>
> =A0<value>rack2.sh</value>
> </property>
>
>
> correct?
>
>
> On Thu, Mar 18, 2010 at 3:15 AM, Christopher Tubbs <ctubbsii@gmail.com> w=
rote:
>> Hadoop will identify data nodes in your cluster by name and execute
>> your script with the data node as an argument. The expected output of
>> your script is the name of the rack on which it is located.
>>
>> The script you referenced takes the node name as an argument ($1), and
>> crawls through a separate file looking up that node in the left
>> column, and printing the value in the second column if it finds it.
>>
>> If you were to use this script, you would just create the topology
>> file that lists all your nodes by name/ip on the left and the rack
>> they are in on the right.
>>
>> On Wed, Mar 17, 2010 at 11:34 PM, Mag Gam <magawake@gmail.com> wrote:
>>> Well, =A0I didn't really solve the problem. Now I have even more questi=
ons.
>>>
>>> I came across this script,
>>> http://wiki.apache.org/hadoop/topology_rack_awareness_scripts
>>>
>>> but it makes no sense to me! Can someone please try to explain what
>>> its trying to do?
>>>
>>>
>>> MikeThomas:
>>>
>>> Your script isn't working for me. I think there are some syntax
>>> errors. Is this how its supposed to look: http://pastebin.ca/1844287
>>>
>>> thanks
>>>
>>>
>>>
>>> On Thu, Mar 4, 2010 at 10:30 PM, Jeff Hammerbacher <hammer@cloudera.com=
> wrote:
>>>> Hey Mag,
>>>>
>>>> Glad you have solved the problem. I've created a JIRA ticket to improv=
e the
>>>> existing documentation: https://issues.apache.org/jira/browse/HADOOP-6=
616.
>>>> If you have some time, it would be useful to hear what could be added =
to the
>>>> existing documentation that would have helped you figure this out soon=
er.
>>>>
>>>> Thanks,
>>>> Jeff
>>>>
>>>> On Thu, Mar 4, 2010 at 3:39 PM, Mag Gam <magawake@gmail.com> wrote:
>>>>
>>>>> Thanks everyone for explaining this to me instead of giving me RTFM!
>>>>>
>>>>> I will play around with it and see how far I get.
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Mar 4, 2010 at 9:21 AM, Steve Loughran <stevel@apache.org> wr=
ote:
>>>>> > Allen Wittenauer wrote:
>>>>> >>
>>>>> >> On 3/3/10 5:01 PM, "Mag Gam" <magawake@gmail.com> wrote:
>>>>> >>
>>>>> >>> Thanks Alan! Your presentation is very nice!
>>>>> >>
>>>>> >> Thanks. :)
>>>>> >>
>>>>> >>> "If you don't provide a script for rack awareness, it treats ever=
y
>>>>> >>> node as if it was its own rack". I am using the default settings =
and
>>>>> >>> the report still says only 1 rack.
>>>>> >>
>>>>> >> Let's take a different approach to convince you. :)
>>>>> >>
>>>>> >> Think about the question: =A0Is there a difference between all nod=
es in
>>>>> one
>>>>> >> rack vs. every node acting as a lone rack?
>>>>> >>
>>>>> >> The answer is no, there isn't any difference. =A0In both cases, al=
l copies
>>>>> >> of
>>>>> >> the blocks can go to pretty much any node. When a MR job runs, eve=
ry
>>>>> node
>>>>> >> would either be considered 'off rack' or 'rack-local'.
>>>>> >>
>>>>> >> So there is no difference.
>>>>> >>
>>>>> >>
>>>>> >>> Do you mind sharing a script with us on how you determine a rack?=
 and
>>>>> >>> a sample <configuration> </configuration> syntax?
>>>>> >>
>>>>> >> Michael has already posted his, so I'll skip this one. :)
>>>>> >>
>>>>> >
>>>>> > Think Mag probably wanted a shell script.
>>>>> >
>>>>> > Mag, give your machines IPv4 addresses that map to rack number. 10.=
1.1.*
>>>>> for
>>>>> > rack one, 10.1.2.* for rack 2, etc. Then just filter out the IP add=
ress
>>>>> by
>>>>> > the top bytes, returning "10.1.1" for everything in rack one, "10.1=
.2"
>>>>> for
>>>>> > rack 2; Hadoop will be happy
>>>>> >
>>>>>
>>>>
>>>
>>
>

From general-return-1220-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 19 11:32:49 2010
Return-Path: <general-return-1220-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 8662 invoked from network); 19 Mar 2010 11:32:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Mar 2010 11:32:49 -0000
Received: (qmail 61811 invoked by uid 500); 19 Mar 2010 11:32:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 61774 invoked by uid 500); 19 Mar 2010 11:32:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 61765 invoked by uid 99); 19 Mar 2010 11:32:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 11:32:48 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of magawake@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 11:32:41 +0000
Received: by vws6 with SMTP id 6so940723vws.35
        for <general@hadoop.apache.org>; Fri, 19 Mar 2010 04:32:20 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=7PEeAjWgHtgOYYn6rPnaQooKZz//NnaocAmS/SLb9/A=;
        b=kKk3BgxxrKBapO7nap0et753loHVfMVfZYWL+PTOHXwpXX3UMeSp7A+t3hEp/kIdub
         +Apuw3626p0Z6fPDwmjKzFOow/SIGrCVb2nlvA4dhdcQqqucVS756QFkoHHeU/CG1mPV
         0LmObtvFrBLR+y2e6F2DZ4fCMkmH0WdWg0Fn8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=lTvK7SysOGPcPc6RXtYFYt1ePHMQ2DK8Nb1SYqHAsuatOMVkBZTwkiuLrA9sDhPGsr
         nxr5CBhom7Qkvbe5d3tHkyiUrOMtgchl5zlLymldnliHfCWlLzpgu4ATctFaxxfxOFIe
         8e57dA/2nG68L/unP0AWzi+72KvIEbDPSA0oE=
MIME-Version: 1.0
Received: by 10.220.107.17 with SMTP id z17mr1584981vco.75.1268998340404; Fri, 
	19 Mar 2010 04:32:20 -0700 (PDT)
In-Reply-To: <9db00c201003182200y5d572e87xba1e3e8cf3bd539c@mail.gmail.com>
References: <C7B48351.36DF%awittenauer@linkedin.com>
	 <4B8FC1CC.2090108@apache.org>
	 <1cbd6f831003041539r75b360ecrb8e604d4dce1350f@mail.gmail.com>
	 <c9118d4c1003041830h47961fc4p323c2ebdd0df2ffc@mail.gmail.com>
	 <1cbd6f831003172034r3d699283hd4e41f5fc0f37231@mail.gmail.com>
	 <9db00c201003180015n1e4e8941i2c53296edd21258f@mail.gmail.com>
	 <1cbd6f831003181741t2f538eb8xc872cba6dfe00761@mail.gmail.com>
	 <9db00c201003182200y5d572e87xba1e3e8cf3bd539c@mail.gmail.com>
Date: Fri, 19 Mar 2010 07:32:20 -0400
Message-ID: <1cbd6f831003190432l1845d62cl2c786a4a1e35d136@mail.gmail.com>
Subject: Re: rack awareness help
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

Thanks everyone. I think everyone can agree that this part of the
documentation is lacking for hadoop.

Can someone please provide be a use case, for example:

#server 1
Input > script.sh
Output > rack01

#server 2
Input > script.sh
Output > rack02


Is this how its supposed to work? I am bad with bash so I am trying to
understand the logic so I can implement it with another language such
as tcl


On Fri, Mar 19, 2010 at 1:00 AM, Christopher Tubbs <ctubbsii@gmail.com> wro=
te:
> You only specify the script on the namenode.
> So, you could do something like:
>
> #!/bin/bash
> #rack_decider.sh
>
> if [ $1 =3D "server1.mydomain" -o $1 =3D "192.168.0.1" ] ; then
> =C2=A0echo rack1
> elif [ $1 =3D "server2.mydomain" -o $1 =3D "192.168.0.2" ] ; then
> =C2=A0echo rack1
> elif [ $1 =3D "server3.mydomain" -o $1 =3D "192.168.0.3" ] ; then
> =C2=A0echo rack2
> elif [ $1 =3D "server4.mydomain" -o $1 =3D "192.168.0.4" ] ; then
> =C2=A0echo rack2
> else
> =C2=A0echo unknown_rack
> fi
> # EOF
>
> Of course, this is by far the most basic script you could have (I'm
> not sure why it wasn't offered as an example instead of a more
> complicated one).
>
> On Thu, Mar 18, 2010 at 8:41 PM, Mag Gam <magawake@gmail.com> wrote:
>> Chris:
>>
>> This clears up my questions a lot! Thankyou.
>>
>> So, if I have 4 data servers and I want 2 racks. I can do this
>>
>> #!/bin/bash
>> #rack1.sh
>> echo rack1
>>
>> #bin/bash
>> #rack2.sh
>> echo rack2
>>
>>
>> So, I can do this for 2 servers
>>
>>
>> <property>
>> =C2=A0<name>topology.script.file.name</name>
>> =C2=A0<value>rack1.sh</value>
>> </property>
>>
>> And for the other 2 servers, I can do this:
>>
>>
>> <property>
>> =C2=A0<name>topology.script.file.name</name>
>> =C2=A0<value>rack2.sh</value>
>> </property>
>>
>>
>> correct?
>>
>>
>> On Thu, Mar 18, 2010 at 3:15 AM, Christopher Tubbs <ctubbsii@gmail.com> =
wrote:
>>> Hadoop will identify data nodes in your cluster by name and execute
>>> your script with the data node as an argument. The expected output of
>>> your script is the name of the rack on which it is located.
>>>
>>> The script you referenced takes the node name as an argument ($1), and
>>> crawls through a separate file looking up that node in the left
>>> column, and printing the value in the second column if it finds it.
>>>
>>> If you were to use this script, you would just create the topology
>>> file that lists all your nodes by name/ip on the left and the rack
>>> they are in on the right.
>>>
>>> On Wed, Mar 17, 2010 at 11:34 PM, Mag Gam <magawake@gmail.com> wrote:
>>>> Well, =C2=A0I didn't really solve the problem. Now I have even more qu=
estions.
>>>>
>>>> I came across this script,
>>>> http://wiki.apache.org/hadoop/topology_rack_awareness_scripts
>>>>
>>>> but it makes no sense to me! Can someone please try to explain what
>>>> its trying to do?
>>>>
>>>>
>>>> MikeThomas:
>>>>
>>>> Your script isn't working for me. I think there are some syntax
>>>> errors. Is this how its supposed to look: http://pastebin.ca/1844287
>>>>
>>>> thanks
>>>>
>>>>
>>>>
>>>> On Thu, Mar 4, 2010 at 10:30 PM, Jeff Hammerbacher <hammer@cloudera.co=
m> wrote:
>>>>> Hey Mag,
>>>>>
>>>>> Glad you have solved the problem. I've created a JIRA ticket to impro=
ve the
>>>>> existing documentation: https://issues.apache.org/jira/browse/HADOOP-=
6616.
>>>>> If you have some time, it would be useful to hear what could be added=
 to the
>>>>> existing documentation that would have helped you figure this out soo=
ner.
>>>>>
>>>>> Thanks,
>>>>> Jeff
>>>>>
>>>>> On Thu, Mar 4, 2010 at 3:39 PM, Mag Gam <magawake@gmail.com> wrote:
>>>>>
>>>>>> Thanks everyone for explaining this to me instead of giving me RTFM!
>>>>>>
>>>>>> I will play around with it and see how far I get.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 4, 2010 at 9:21 AM, Steve Loughran <stevel@apache.org> w=
rote:
>>>>>> > Allen Wittenauer wrote:
>>>>>> >>
>>>>>> >> On 3/3/10 5:01 PM, "Mag Gam" <magawake@gmail.com> wrote:
>>>>>> >>
>>>>>> >>> Thanks Alan! Your presentation is very nice!
>>>>>> >>
>>>>>> >> Thanks. :)
>>>>>> >>
>>>>>> >>> "If you don't provide a script for rack awareness, it treats eve=
ry
>>>>>> >>> node as if it was its own rack". I am using the default settings=
 and
>>>>>> >>> the report still says only 1 rack.
>>>>>> >>
>>>>>> >> Let's take a different approach to convince you. :)
>>>>>> >>
>>>>>> >> Think about the question: =C2=A0Is there a difference between all=
 nodes in
>>>>>> one
>>>>>> >> rack vs. every node acting as a lone rack?
>>>>>> >>
>>>>>> >> The answer is no, there isn't any difference. =C2=A0In both cases=
, all copies
>>>>>> >> of
>>>>>> >> the blocks can go to pretty much any node. When a MR job runs, ev=
ery
>>>>>> node
>>>>>> >> would either be considered 'off rack' or 'rack-local'.
>>>>>> >>
>>>>>> >> So there is no difference.
>>>>>> >>
>>>>>> >>
>>>>>> >>> Do you mind sharing a script with us on how you determine a rack=
? and
>>>>>> >>> a sample <configuration> </configuration> syntax?
>>>>>> >>
>>>>>> >> Michael has already posted his, so I'll skip this one. :)
>>>>>> >>
>>>>>> >
>>>>>> > Think Mag probably wanted a shell script.
>>>>>> >
>>>>>> > Mag, give your machines IPv4 addresses that map to rack number. 10=
.1.1.*
>>>>>> for
>>>>>> > rack one, 10.1.2.* for rack 2, etc. Then just filter out the IP ad=
dress
>>>>>> by
>>>>>> > the top bytes, returning "10.1.1" for everything in rack one, "10.=
1.2"
>>>>>> for
>>>>>> > rack 2; Hadoop will be happy
>>>>>> >
>>>>>>
>>>>>
>>>>
>>>
>>
>

From general-return-1221-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 19 13:36:37 2010
Return-Path: <general-return-1221-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 52555 invoked from network); 19 Mar 2010 13:36:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Mar 2010 13:36:37 -0000
Received: (qmail 24241 invoked by uid 500); 19 Mar 2010 13:36:36 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 24102 invoked by uid 500); 19 Mar 2010 13:36:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 24094 invoked by uid 99); 19 Mar 2010 13:36:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 13:36:36 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ctubbsii@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 13:36:31 +0000
Received: by wwc33 with SMTP id 33so1721049wwc.35
        for <general@hadoop.apache.org>; Fri, 19 Mar 2010 06:36:10 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=8mQrN6SMgO5Ix6axT+jz4UqJPq8X5U+TBnkhPqW72F8=;
        b=IWPoUlVbomj3RlatpR2UYYrxIkSNEin55vwrcnT1PZm373c/ksGfzrAOtmHHTX0dGP
         gj4QkD2fPi6j138ydFzMMdhVBtrNgDglnFIxCFPBaj1IUkyejVr9Ibi7rQQoA5pGoD/u
         jog7aLqS74J1rF0Liv9t3PmQLG4XiREx/qKCM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=FifN2mxPgjkXUOVmoX2kcNP8mhrpTAtDmJhToZA2V5Yxa+IZGrom0xaFl4h04r+Y0H
         6XwPNxUkK4Q0y9EShbX+vZqmFdqbjkhvN49GNEspyrkrgAGoUyF7WAgGSp1zrXYymjT/
         vOqWi9aH7Oxzp6hITRKuZp5HeWWaD0yFg33rw=
MIME-Version: 1.0
Received: by 10.216.86.211 with SMTP id w61mr1310196wee.50.1269005770276; Fri, 
	19 Mar 2010 06:36:10 -0700 (PDT)
In-Reply-To: <1cbd6f831003190432l1845d62cl2c786a4a1e35d136@mail.gmail.com>
References: <C7B48351.36DF%awittenauer@linkedin.com>
	 <4B8FC1CC.2090108@apache.org>
	 <1cbd6f831003041539r75b360ecrb8e604d4dce1350f@mail.gmail.com>
	 <c9118d4c1003041830h47961fc4p323c2ebdd0df2ffc@mail.gmail.com>
	 <1cbd6f831003172034r3d699283hd4e41f5fc0f37231@mail.gmail.com>
	 <9db00c201003180015n1e4e8941i2c53296edd21258f@mail.gmail.com>
	 <1cbd6f831003181741t2f538eb8xc872cba6dfe00761@mail.gmail.com>
	 <9db00c201003182200y5d572e87xba1e3e8cf3bd539c@mail.gmail.com>
	 <1cbd6f831003190432l1845d62cl2c786a4a1e35d136@mail.gmail.com>
Date: Fri, 19 Mar 2010 09:36:10 -0400
Message-ID: <9db00c201003190636v1e572b1bo215aae0e0afe1526@mail.gmail.com>
Subject: Re: rack awareness help
From: Christopher Tubbs <ctubbsii@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

More like the following (shown with the bash prompt). You could type
this for testing. However, ultimately, hadoop itself will actually be
executing this script and reading its output.

$ ./script.sh server1.mydomain
rack1
$ ./script.sh server2.mydomain
rack1
$ ./script.sh server3.mydomain
rack2
$ ./script.sh server4.mydomain
rack2

On Fri, Mar 19, 2010 at 7:32 AM, Mag Gam <magawake@gmail.com> wrote:
> Thanks everyone. I think everyone can agree that this part of the
> documentation is lacking for hadoop.
>
> Can someone please provide be a use case, for example:
>
> #server 1
> Input > script.sh
> Output > rack01
>
> #server 2
> Input > script.sh
> Output > rack02
>
>
> Is this how its supposed to work? I am bad with bash so I am trying to
> understand the logic so I can implement it with another language such
> as tcl
>
>
> On Fri, Mar 19, 2010 at 1:00 AM, Christopher Tubbs <ctubbsii@gmail.com> w=
rote:
>> You only specify the script on the namenode.
>> So, you could do something like:
>>
>> #!/bin/bash
>> #rack_decider.sh
>>
>> if [ $1 =3D "server1.mydomain" -o $1 =3D "192.168.0.1" ] ; then
>> =A0echo rack1
>> elif [ $1 =3D "server2.mydomain" -o $1 =3D "192.168.0.2" ] ; then
>> =A0echo rack1
>> elif [ $1 =3D "server3.mydomain" -o $1 =3D "192.168.0.3" ] ; then
>> =A0echo rack2
>> elif [ $1 =3D "server4.mydomain" -o $1 =3D "192.168.0.4" ] ; then
>> =A0echo rack2
>> else
>> =A0echo unknown_rack
>> fi
>> # EOF
>>
>> Of course, this is by far the most basic script you could have (I'm
>> not sure why it wasn't offered as an example instead of a more
>> complicated one).
>>
>> On Thu, Mar 18, 2010 at 8:41 PM, Mag Gam <magawake@gmail.com> wrote:
>>> Chris:
>>>
>>> This clears up my questions a lot! Thankyou.
>>>
>>> So, if I have 4 data servers and I want 2 racks. I can do this
>>>
>>> #!/bin/bash
>>> #rack1.sh
>>> echo rack1
>>>
>>> #bin/bash
>>> #rack2.sh
>>> echo rack2
>>>
>>>
>>> So, I can do this for 2 servers
>>>
>>>
>>> <property>
>>> =A0<name>topology.script.file.name</name>
>>> =A0<value>rack1.sh</value>
>>> </property>
>>>
>>> And for the other 2 servers, I can do this:
>>>
>>>
>>> <property>
>>> =A0<name>topology.script.file.name</name>
>>> =A0<value>rack2.sh</value>
>>> </property>
>>>
>>>
>>> correct?
>>>
>>>
>>> On Thu, Mar 18, 2010 at 3:15 AM, Christopher Tubbs <ctubbsii@gmail.com>=
 wrote:
>>>> Hadoop will identify data nodes in your cluster by name and execute
>>>> your script with the data node as an argument. The expected output of
>>>> your script is the name of the rack on which it is located.
>>>>
>>>> The script you referenced takes the node name as an argument ($1), and
>>>> crawls through a separate file looking up that node in the left
>>>> column, and printing the value in the second column if it finds it.
>>>>
>>>> If you were to use this script, you would just create the topology
>>>> file that lists all your nodes by name/ip on the left and the rack
>>>> they are in on the right.
>>>>
>>>> On Wed, Mar 17, 2010 at 11:34 PM, Mag Gam <magawake@gmail.com> wrote:
>>>>> Well, =A0I didn't really solve the problem. Now I have even more ques=
tions.
>>>>>
>>>>> I came across this script,
>>>>> http://wiki.apache.org/hadoop/topology_rack_awareness_scripts
>>>>>
>>>>> but it makes no sense to me! Can someone please try to explain what
>>>>> its trying to do?
>>>>>
>>>>>
>>>>> MikeThomas:
>>>>>
>>>>> Your script isn't working for me. I think there are some syntax
>>>>> errors. Is this how its supposed to look: http://pastebin.ca/1844287
>>>>>
>>>>> thanks
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Mar 4, 2010 at 10:30 PM, Jeff Hammerbacher <hammer@cloudera.c=
om> wrote:
>>>>>> Hey Mag,
>>>>>>
>>>>>> Glad you have solved the problem. I've created a JIRA ticket to impr=
ove the
>>>>>> existing documentation: https://issues.apache.org/jira/browse/HADOOP=
-6616.
>>>>>> If you have some time, it would be useful to hear what could be adde=
d to the
>>>>>> existing documentation that would have helped you figure this out so=
oner.
>>>>>>
>>>>>> Thanks,
>>>>>> Jeff
>>>>>>
>>>>>> On Thu, Mar 4, 2010 at 3:39 PM, Mag Gam <magawake@gmail.com> wrote:
>>>>>>
>>>>>>> Thanks everyone for explaining this to me instead of giving me RTFM=
!
>>>>>>>
>>>>>>> I will play around with it and see how far I get.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Mar 4, 2010 at 9:21 AM, Steve Loughran <stevel@apache.org> =
wrote:
>>>>>>> > Allen Wittenauer wrote:
>>>>>>> >>
>>>>>>> >> On 3/3/10 5:01 PM, "Mag Gam" <magawake@gmail.com> wrote:
>>>>>>> >>
>>>>>>> >>> Thanks Alan! Your presentation is very nice!
>>>>>>> >>
>>>>>>> >> Thanks. :)
>>>>>>> >>
>>>>>>> >>> "If you don't provide a script for rack awareness, it treats ev=
ery
>>>>>>> >>> node as if it was its own rack". I am using the default setting=
s and
>>>>>>> >>> the report still says only 1 rack.
>>>>>>> >>
>>>>>>> >> Let's take a different approach to convince you. :)
>>>>>>> >>
>>>>>>> >> Think about the question: =A0Is there a difference between all n=
odes in
>>>>>>> one
>>>>>>> >> rack vs. every node acting as a lone rack?
>>>>>>> >>
>>>>>>> >> The answer is no, there isn't any difference. =A0In both cases, =
all copies
>>>>>>> >> of
>>>>>>> >> the blocks can go to pretty much any node. When a MR job runs, e=
very
>>>>>>> node
>>>>>>> >> would either be considered 'off rack' or 'rack-local'.
>>>>>>> >>
>>>>>>> >> So there is no difference.
>>>>>>> >>
>>>>>>> >>
>>>>>>> >>> Do you mind sharing a script with us on how you determine a rac=
k? and
>>>>>>> >>> a sample <configuration> </configuration> syntax?
>>>>>>> >>
>>>>>>> >> Michael has already posted his, so I'll skip this one. :)
>>>>>>> >>
>>>>>>> >
>>>>>>> > Think Mag probably wanted a shell script.
>>>>>>> >
>>>>>>> > Mag, give your machines IPv4 addresses that map to rack number. 1=
0.1.1.*
>>>>>>> for
>>>>>>> > rack one, 10.1.2.* for rack 2, etc. Then just filter out the IP a=
ddress
>>>>>>> by
>>>>>>> > the top bytes, returning "10.1.1" for everything in rack one, "10=
.1.2"
>>>>>>> for
>>>>>>> > rack 2; Hadoop will be happy
>>>>>>> >
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

From general-return-1222-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 19 15:13:54 2010
Return-Path: <general-return-1222-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 614 invoked from network); 19 Mar 2010 15:13:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Mar 2010 15:13:54 -0000
Received: (qmail 67253 invoked by uid 500); 19 Mar 2010 15:13:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 67176 invoked by uid 500); 19 Mar 2010 15:13:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 67094 invoked by uid 99); 19 Mar 2010 15:13:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 15:13:51 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.25 as permitted sender)
Received: from [69.28.149.25] (HELO esv4-mav03.corp.linkedin.com) (69.28.149.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 15:13:45 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:User-Agent:Date:Subject:From:To:
   Message-ID:Thread-Topic:Thread-Index:In-Reply-To:
   Mime-version:Content-type:Content-transfer-encoding;
  b=fjjAcrTmxknDFx6Z4UauwQQlFN1Tgr4GlPHruqSGE7jD16jZEi5eNhNT
   06cjhHKQuVRrpF9PvKjcrU5Hk3ttf5UETN5FOenp7VwXeZWNoQR6KSVAA
   AbN0QwI3tU36zFx;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1269011625; x=1300547625;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20rack=20awareness=20help|Date:=20Fri,=20
   19=20Mar=202010=2008:13:21=20-0700|Message-ID:=20<C7C8E2A
   1.3CA3%awittenauer@linkedin.com>|To:=20<general@hadoop.ap
   ache.org>|Mime-version:=201.0|Content-transfer-encoding:
   =207bit|In-Reply-To:=20<1cbd6f831003190432l1845d62cl2c786
   a4a1e35d136@mail.gmail.com>;
  bh=E0CkSxpRGBqlgil53VajpJOatUSYA46EGxKV4qDs6a0=;
  b=Hsq1TLonNSYJ57R6iCqzIlwR2auQsD6zd8q2EqPjqLg5A2WAARcRAWlW
   hG16pTuyj4vzEY1uXlDY+I6F7B7tAG1Eg2ZCiSAzc8kMO/y+sevinGDu2
   o+W+36uMJQLeiUJ;
X-IronPort-AV: E=Sophos;i="4.51,274,1267430400"; 
   d="scan'208";a="11472544"
Received: from 172.18.36.111 ([172.18.36.111]) by CORP-MAIL.linkedin.biz ([172.18.46.135]) via Exchange Front-End Server mail-access.linkedin.biz ([172.18.46.133]) with Microsoft Exchange Server HTTP-DAV ;
 Fri, 19 Mar 2010 15:13:23 +0000
User-Agent: Microsoft-Entourage/12.10.0.080409
Date: Fri, 19 Mar 2010 08:13:21 -0700
Subject: Re: rack awareness help
From: Allen Wittenauer <awittenauer@linkedin.com>
To: <general@hadoop.apache.org>
Message-ID: <C7C8E2A1.3CA3%awittenauer@linkedin.com>
Thread-Topic: rack awareness help
Thread-Index: AcrHdrYte1L2fw0FfUGFfGehMPnVdg==
In-Reply-To: <1cbd6f831003190432l1845d62cl2c786a4a1e35d136@mail.gmail.com>
Mime-version: 1.0
Content-type: text/plain;
	charset="US-ASCII"
Content-transfer-encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org




On 3/19/10 4:32 AM, "Mag Gam" <magawake@gmail.com> wrote:

> Thanks everyone. I think everyone can agree that this part of the
> documentation is lacking for hadoop.
>
> Can someone please provide be a use case, for example:
> 
> #server 1
> Input > script.sh
> Output > rack01
> 
> #server 2
> Input > script.sh
> Output > rack02

I think you have it in your head that the NameNode asks the DataNode what
rack it is.  This is completely backwards.  The DataNode has *no* concept of
what a rack is.  It is purely a storage process.  There isn't much logic in
it at all.

The topology script is *ONLY* run by the NameNode and JobTracker processes.
That's it.  It is not run on the compute nodes.  That setting is completely
*ignored* by the DataNode and TaskTracker processes.

So to rewrite your use case:

# NameNode 
Input > server 1
Output > rack01

# NameNode
Input > server 2
Output > rack02

> Is this how its supposed to work? I am bad with bash so I am trying to
> understand the logic so I can implement it with another language such
> as tcl


The program logic is :

Input -> IP address or Hostname
Output -> /racklocation

That's it.  There is nothing fancy going on here.  


From general-return-1223-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 19 16:43:52 2010
Return-Path: <general-return-1223-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 29393 invoked from network); 19 Mar 2010 16:43:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Mar 2010 16:43:51 -0000
Received: (qmail 59015 invoked by uid 500); 19 Mar 2010 16:43:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58982 invoked by uid 500); 19 Mar 2010 16:43:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58974 invoked by uid 99); 19 Mar 2010 16:43:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 16:43:50 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jerdavis@speakeasy.net designates 69.17.117.6 as permitted sender)
Received: from [69.17.117.6] (HELO mail4.sea5.speakeasy.net) (69.17.117.6)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 16:43:41 +0000
Received: (qmail 13325 invoked from network); 19 Mar 2010 16:43:19 -0000
Received: from c-71-231-206-119.hsd1.wa.comcast.net (HELO [10.0.1.5]) (jerdavis@[71.231.206.119])
          (envelope-sender <jerdavis@speakeasy.net>)
          by mail4.sea5.speakeasy.net (qmail-ldap-1.03) with AES128-SHA encrypted SMTP
          for <general@hadoop.apache.org>; 19 Mar 2010 16:43:19 -0000
Message-Id: <656B2F92-03D7-46AB-B49B-C8CBFE2F8BA9@speakeasy.net>
From: Jeremy Davis <jerdavis@speakeasy.net>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=Apple-Mail-15-799490642
Mime-Version: 1.0 (Apple Message framework v936)
Subject: 0.21 Release
Date: Fri, 19 Mar 2010 09:43:02 -0700
X-Mailer: Apple Mail (2.936)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-15-799490642
Content-Type: text/plain;
	charset=US-ASCII;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: 7bit


Could someone clarify the position on the 0.21 release?
This thread:
http://mail-archives.apache.org/mod_mbox/hadoop-general/201002.mbox/<dfe484f01002161412h8bc953axee2a73d81a234bdf@mail.gmail.com 
 >
kind of died out (it seems) without any resolution.

I saw two suggestions: Rebase 0.21 against trunk and then try to  
release, and I also saw release 0.21 as is with stack as the release  
manager. Was there a final decision on this off list?

Thanks.
--Apple-Mail-15-799490642--

From general-return-1224-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 19 18:52:11 2010
Return-Path: <general-return-1224-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83638 invoked from network); 19 Mar 2010 18:52:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Mar 2010 18:52:11 -0000
Received: (qmail 45438 invoked by uid 500); 19 Mar 2010 18:52:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45406 invoked by uid 500); 19 Mar 2010 18:52:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45398 invoked by uid 99); 19 Mar 2010 18:52:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 18:52:10 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 209.85.221.198 as permitted sender)
Received: from [209.85.221.198] (HELO mail-qy0-f198.google.com) (209.85.221.198)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 18:52:02 +0000
Received: by qyk36 with SMTP id 36so2138097qyk.30
        for <general@hadoop.apache.org>; Fri, 19 Mar 2010 11:51:41 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:message-id:subject:from:to
         :content-type;
        bh=VhJ8885KEV3mH4+D5z7mJqeCH+tbpw2OK/JcDMByz/I=;
        b=M4wmTCEyF3h3agGHJFZ1eZUfyGW9Jge97yvIREGiRVd1CCITthwnv8KcbEh0QkyF6o
         V8lNfYXDtzDhVLk3vltTD7o1WqrZ47R3YuG776mkIbtiG8IWhjS5y5UpoCY2WjS7l2v0
         DzATGCGFcyHayAJIj/cBRdRXyP2pPIIEWTuw8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=QsAE0fgyTggaPicfg3HyTCiVQT+9wHmbjKt9vMNmiNsQuwNCqZo0R0uiK1/IjNQV94
         ZjwaPhekHUkJ3HtwlRPCzqLh/Fbz0kCxTIghmOp01DECOnjzzWH/v70Ax732YXNQpVhC
         3xlxEhgbBQbCkwv63iVRb4qIpHrb7mrRuSs8o=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
Received: by 10.229.81.81 with SMTP id w17mr4481689qck.4.1269024701452; Fri, 
	19 Mar 2010 11:51:41 -0700 (PDT)
In-Reply-To: <656B2F92-03D7-46AB-B49B-C8CBFE2F8BA9@speakeasy.net>
References: <656B2F92-03D7-46AB-B49B-C8CBFE2F8BA9@speakeasy.net>
Date: Fri, 19 Mar 2010 11:51:41 -0700
X-Google-Sender-Auth: 5500f798bf56bb64
Message-ID: <7c962aed1003191151g18b6efc3m5b89385ade5cc29f@mail.gmail.com>
Subject: Re: 0.21 Release
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Fri, Mar 19, 2010 at 9:43 AM, Jeremy Davis <jerdavis@speakeasy.net> wrote:
> ...and I also saw release 0.21 as is with stack as the release manager. Was
> there a final decision on this off list?
>

On the above, I was toying with the idea of being release manager for
releasing as hadoop 0.21.0 what is in current 0.21 hadoop branch but I
subsequently decided against it after chatting with folks and figuring
that the only group that seemed interested in driving a release of the
hadoop 0.21 branch was the hbase crew.  If I were to guess, an hadoop
vouched for by a couple of hbasers with their spotty hdfs and
mapreduce knowledge probably wouldn't have the penetration of a
release backed by, say, a Yahoo.  No one would trust their data to
such a release.  If no data in hadoop 0.21 clusters, hbase wouldn't
have anything to run against.  So I let it go and figured time could
be spent better elsewhere; e.g. helping test the set of patches that
could get us a sync/flush/append on a patched hadoop 0.20 (hdfs-200,
etc.).

Sorry, I should have added a note to cited thread that I'd wandered...

St.Ack

From general-return-1225-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 19 19:18:10 2010
Return-Path: <general-return-1225-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 92980 invoked from network); 19 Mar 2010 19:18:10 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Mar 2010 19:18:10 -0000
Received: (qmail 93142 invoked by uid 500); 19 Mar 2010 19:18:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93100 invoked by uid 500); 19 Mar 2010 19:18:09 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93092 invoked by uid 99); 19 Mar 2010 19:18:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 19:18:09 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jerdavis@speakeasy.net designates 69.17.117.6 as permitted sender)
Received: from [69.17.117.6] (HELO mail4.sea5.speakeasy.net) (69.17.117.6)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 19:18:01 +0000
Received: (qmail 27334 invoked from network); 19 Mar 2010 19:17:41 -0000
Received: from c-71-231-206-119.hsd1.wa.comcast.net (HELO [10.0.1.5]) (jerdavis@[71.231.206.119])
          (envelope-sender <jerdavis@speakeasy.net>)
          by mail4.sea5.speakeasy.net (qmail-ldap-1.03) with AES128-SHA encrypted SMTP
          for <general@hadoop.apache.org>; 19 Mar 2010 19:17:41 -0000
Message-Id: <F0EEBAC7-2C15-4C99-9153-58F20DD27A56@speakeasy.net>
From: Jeremy Davis <jerdavis@speakeasy.net>
To: general@hadoop.apache.org
In-Reply-To: <7c962aed1003191151g18b6efc3m5b89385ade5cc29f@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Re: 0.21 Release
Date: Fri, 19 Mar 2010 12:17:40 -0700
References: <656B2F92-03D7-46AB-B49B-C8CBFE2F8BA9@speakeasy.net> <7c962aed1003191151g18b6efc3m5b89385ade5cc29f@mail.gmail.com>
X-Mailer: Apple Mail (2.936)

Thank you for your reply,

So what would be your (or anyones) advice on getting, HDFS sync/flush/ 
append functionality?
You seem to indicate that 0.21 branch as is might not be the best  
idea, with your preference being a patch set against 0.20.

We definitely have an application for this specific functionality, and  
I need to provide some direction/answers in this area for my colleagues.

For example, I might say: We will install the current 0.21 now, and  
develop against it.. But in X time we will install CDH2 (and get all  
the goodness it brings), and then apply a given patch set against it.  
Is this in line with what you are thinking? If so, could I get your  
perceived level of effort, maybe a time frame? April/May/June/ etc..

I'm sure I'm not the only one that has plans for this feature, as it's  
in  "Hadoop: the Definitive Guide" by O'Reilly published in September  
'09.

Thanks,
-JD




On Mar 19, 2010, at 11:51 AM, Stack wrote:

> On Fri, Mar 19, 2010 at 9:43 AM, Jeremy Davis  
> <jerdavis@speakeasy.net> wrote:
>> ...and I also saw release 0.21 as is with stack as the release  
>> manager. Was
>> there a final decision on this off list?
>>
>
> On the above, I was toying with the idea of being release manager for
> releasing as hadoop 0.21.0 what is in current 0.21 hadoop branch but I
> subsequently decided against it after chatting with folks and figuring
> that the only group that seemed interested in driving a release of the
> hadoop 0.21 branch was the hbase crew.  If I were to guess, an hadoop
> vouched for by a couple of hbasers with their spotty hdfs and
> mapreduce knowledge probably wouldn't have the penetration of a
> release backed by, say, a Yahoo.  No one would trust their data to
> such a release.  If no data in hadoop 0.21 clusters, hbase wouldn't
> have anything to run against.  So I let it go and figured time could
> be spent better elsewhere; e.g. helping test the set of patches that
> could get us a sync/flush/append on a patched hadoop 0.20 (hdfs-200,
> etc.).
>
> Sorry, I should have added a note to cited thread that I'd wandered...
>
> St.Ack


From general-return-1226-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 19 19:32:13 2010
Return-Path: <general-return-1226-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96856 invoked from network); 19 Mar 2010 19:32:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Mar 2010 19:32:13 -0000
Received: (qmail 19844 invoked by uid 500); 19 Mar 2010 19:32:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 19694 invoked by uid 500); 19 Mar 2010 19:32:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 19686 invoked by uid 99); 19 Mar 2010 19:32:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 19:32:12 +0000
X-ASF-Spam-Status: No, hits=0.3 required=10.0
	tests=AWL,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.223.201] (HELO mail-iw0-f201.google.com) (209.85.223.201)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 19:32:06 +0000
Received: by iwn39 with SMTP id 39so1188095iwn.2
        for <general@hadoop.apache.org>; Fri, 19 Mar 2010 12:31:45 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.231.169.144 with SMTP id z16mr1771959iby.25.1269027105368; 
	Fri, 19 Mar 2010 12:31:45 -0700 (PDT)
In-Reply-To: <F0EEBAC7-2C15-4C99-9153-58F20DD27A56@speakeasy.net>
References: <656B2F92-03D7-46AB-B49B-C8CBFE2F8BA9@speakeasy.net> 
	<7c962aed1003191151g18b6efc3m5b89385ade5cc29f@mail.gmail.com> 
	<F0EEBAC7-2C15-4C99-9153-58F20DD27A56@speakeasy.net>
From: Todd Lipcon <todd@cloudera.com>
Date: Fri, 19 Mar 2010 12:31:25 -0700
Message-ID: <45f85f71003191231w23227cb1q7896f8c4e18b5a89@mail.gmail.com>
Subject: Re: 0.21 Release
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d26c5ad36a3704822c65ef

--0016e6d26c5ad36a3704822c65ef
Content-Type: text/plain; charset=ISO-8859-1

On Fri, Mar 19, 2010 at 12:17 PM, Jeremy Davis <jerdavis@speakeasy.net>wrote:

> Thank you for your reply,
>
> So what would be your (or anyones) advice on getting, HDFS
> sync/flush/append functionality?
> You seem to indicate that 0.21 branch as is might not be the best idea,
> with your preference being a patch set against 0.20.
>
> We definitely have an application for this specific functionality, and I
> need to provide some direction/answers in this area for my colleagues.
>
> For example, I might say: We will install the current 0.21 now, and develop
> against it.. But in X time we will install CDH2 (and get all the goodness it
> brings), and then apply a given patch set against it. Is this in line with
> what you are thinking? If so, could I get your perceived level of effort,
> maybe a time frame? April/May/June/ etc..
>
>
FYI the 0.20 sync patches mentioned by Stack will be going into CDH3 at some
point this spring. This is mainly for the benefit of HBase but of course
other applications will benefit as well.

Thanks
-Todd


> I'm sure I'm not the only one that has plans for this feature, as it's in
>  "Hadoop: the Definitive Guide" by O'Reilly published in September '09.
>
> Thanks,
> -JD
>
>
>
>
>
> On Mar 19, 2010, at 11:51 AM, Stack wrote:
>
>  On Fri, Mar 19, 2010 at 9:43 AM, Jeremy Davis <jerdavis@speakeasy.net>
>> wrote:
>>
>>> ...and I also saw release 0.21 as is with stack as the release manager.
>>> Was
>>> there a final decision on this off list?
>>>
>>>
>> On the above, I was toying with the idea of being release manager for
>> releasing as hadoop 0.21.0 what is in current 0.21 hadoop branch but I
>> subsequently decided against it after chatting with folks and figuring
>> that the only group that seemed interested in driving a release of the
>> hadoop 0.21 branch was the hbase crew.  If I were to guess, an hadoop
>> vouched for by a couple of hbasers with their spotty hdfs and
>> mapreduce knowledge probably wouldn't have the penetration of a
>> release backed by, say, a Yahoo.  No one would trust their data to
>> such a release.  If no data in hadoop 0.21 clusters, hbase wouldn't
>> have anything to run against.  So I let it go and figured time could
>> be spent better elsewhere; e.g. helping test the set of patches that
>> could get us a sync/flush/append on a patched hadoop 0.20 (hdfs-200,
>> etc.).
>>
>> Sorry, I should have added a note to cited thread that I'd wandered...
>>
>> St.Ack
>>
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

--0016e6d26c5ad36a3704822c65ef--

From general-return-1227-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 19 19:46:34 2010
Return-Path: <general-return-1227-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 278 invoked from network); 19 Mar 2010 19:46:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Mar 2010 19:46:34 -0000
Received: (qmail 33741 invoked by uid 500); 19 Mar 2010 19:46:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 33691 invoked by uid 500); 19 Mar 2010 19:46:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 33683 invoked by uid 99); 19 Mar 2010 19:46:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 19:46:33 +0000
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com designates 74.125.92.27 as permitted sender)
Received: from [74.125.92.27] (HELO qw-out-2122.google.com) (74.125.92.27)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 19:46:27 +0000
Received: by qw-out-2122.google.com with SMTP id 8so767705qwh.35
        for <general@hadoop.apache.org>; Fri, 19 Mar 2010 12:46:06 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=x6Ciedq0JqBgg+7v6TFww9ki882zScaupQuwEK8xrqg=;
        b=Cq0F+c0H2hWk2F1HuACq5gsNIDqvShKQhYi170Xhk5ELn4b7YJ4hgyqAMaaYVhFIfw
         dUj2+qreVs410Fw8GGJgg0UxTuUbG1Rx1FYydfHTrU3I9pMWR2KE1sNY/8XNKYtJEiCb
         Gqxdt6BwYV1EpmaXFQXVKG+8bD8SIDI0fkMmQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=nsIn1nVk0yOXm7hds4u7dPYYd+uX/fS15bY/qgHHjT98vkVxK9iAPkAEJtSa4kK63a
         CHLADJFfXOT305km0K20IVUZeuR87U+IEXCVHeavFrRTbDNI6hQn0KAG2xT/SrfKgZNh
         YXJPhoBzwmxbLNuLMengQasNQpzU9kX+4/AqE=
MIME-Version: 1.0
Received: by 10.229.251.72 with SMTP id mr8mr1083594qcb.30.1269027966834; Fri, 
	19 Mar 2010 12:46:06 -0700 (PDT)
In-Reply-To: <45f85f71003191231w23227cb1q7896f8c4e18b5a89@mail.gmail.com>
References: <656B2F92-03D7-46AB-B49B-C8CBFE2F8BA9@speakeasy.net>
	 <7c962aed1003191151g18b6efc3m5b89385ade5cc29f@mail.gmail.com>
	 <F0EEBAC7-2C15-4C99-9153-58F20DD27A56@speakeasy.net>
	 <45f85f71003191231w23227cb1q7896f8c4e18b5a89@mail.gmail.com>
Date: Fri, 19 Mar 2010 12:46:06 -0700
Message-ID: <17e273101003191246m4786a9e9m4c7112fd07177814@mail.gmail.com>
Subject: Re: 0.21 Release
From: Ted Yu <yuzhihong@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016363b808c2c5a2904822c99c8

--0016363b808c2c5a2904822c99c8
Content-Type: text/plain; charset=ISO-8859-1

I want to bring up the issue about shuffler from thread:
Shuffle In Memory OutOfMemoryError

I was expecting improved performance from MAPREDUCE-1182

That would be one of the reasons for people to try out 0.21 release

Regards

On Fri, Mar 19, 2010 at 12:31 PM, Todd Lipcon <todd@cloudera.com> wrote:

> On Fri, Mar 19, 2010 at 12:17 PM, Jeremy Davis <jerdavis@speakeasy.net
> >wrote:
>
> > Thank you for your reply,
> >
> > So what would be your (or anyones) advice on getting, HDFS
> > sync/flush/append functionality?
> > You seem to indicate that 0.21 branch as is might not be the best idea,
> > with your preference being a patch set against 0.20.
> >
> > We definitely have an application for this specific functionality, and I
> > need to provide some direction/answers in this area for my colleagues.
> >
> > For example, I might say: We will install the current 0.21 now, and
> develop
> > against it.. But in X time we will install CDH2 (and get all the goodness
> it
> > brings), and then apply a given patch set against it. Is this in line
> with
> > what you are thinking? If so, could I get your perceived level of effort,
> > maybe a time frame? April/May/June/ etc..
> >
> >
> FYI the 0.20 sync patches mentioned by Stack will be going into CDH3 at
> some
> point this spring. This is mainly for the benefit of HBase but of course
> other applications will benefit as well.
>
> Thanks
> -Todd
>
>
> > I'm sure I'm not the only one that has plans for this feature, as it's in
> >  "Hadoop: the Definitive Guide" by O'Reilly published in September '09.
> >
> > Thanks,
> > -JD
> >
> >
> >
> >
> >
> > On Mar 19, 2010, at 11:51 AM, Stack wrote:
> >
> >  On Fri, Mar 19, 2010 at 9:43 AM, Jeremy Davis <jerdavis@speakeasy.net>
> >> wrote:
> >>
> >>> ...and I also saw release 0.21 as is with stack as the release manager.
> >>> Was
> >>> there a final decision on this off list?
> >>>
> >>>
> >> On the above, I was toying with the idea of being release manager for
> >> releasing as hadoop 0.21.0 what is in current 0.21 hadoop branch but I
> >> subsequently decided against it after chatting with folks and figuring
> >> that the only group that seemed interested in driving a release of the
> >> hadoop 0.21 branch was the hbase crew.  If I were to guess, an hadoop
> >> vouched for by a couple of hbasers with their spotty hdfs and
> >> mapreduce knowledge probably wouldn't have the penetration of a
> >> release backed by, say, a Yahoo.  No one would trust their data to
> >> such a release.  If no data in hadoop 0.21 clusters, hbase wouldn't
> >> have anything to run against.  So I let it go and figured time could
> >> be spent better elsewhere; e.g. helping test the set of patches that
> >> could get us a sync/flush/append on a patched hadoop 0.20 (hdfs-200,
> >> etc.).
> >>
> >> Sorry, I should have added a note to cited thread that I'd wandered...
> >>
> >> St.Ack
> >>
> >
> >
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

--0016363b808c2c5a2904822c99c8--

From general-return-1228-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 19 20:01:52 2010
Return-Path: <general-return-1228-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 4413 invoked from network); 19 Mar 2010 20:01:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Mar 2010 20:01:52 -0000
Received: (qmail 51818 invoked by uid 500); 19 Mar 2010 20:01:51 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51764 invoked by uid 500); 19 Mar 2010 20:01:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51755 invoked by uid 99); 19 Mar 2010 20:01:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 20:01:51 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of saint.ack@gmail.com designates 74.125.92.26 as permitted sender)
Received: from [74.125.92.26] (HELO qw-out-2122.google.com) (74.125.92.26)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Mar 2010 20:01:43 +0000
Received: by qw-out-2122.google.com with SMTP id 8so772257qwh.35
        for <general@hadoop.apache.org>; Fri, 19 Mar 2010 13:01:22 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:message-id:subject:from:to
         :content-type;
        bh=5GLkihGxcKEue1Y9k1cud7gEk/VYJyht4nbHgt/YOMc=;
        b=FiH7cCdf1IsMUKwb+wLgskwG+YM1lbcPy0TMhJP20FcVHzdVruC6xC//V14AslWqmZ
         JYYgyGFM4azHnUr/B2k5F/csFkAsxE0UR93xaxJ2C81vm4W0CCrVTv94/d0dSaper+8f
         pqPw0b3jQJYvjs5tlqsd1b+qAs+EMnTVADjBo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type;
        b=dRjiok981DAWwcRVRvbYc/nsjeDOMeVczJaYE9zD1eqXc5nTqiOIlV2rrWhOOKHPE+
         tJ+Qz5HpsLa1ZrBlXwbx4bXTpAjEniAQlVaAKpGR+/EYZ2Sw6ytbjElK5bcfg2esfzfs
         NA3fqWZ0Il2QCr14SV0g+dgcWYY25iAge4Wwo=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
Received: by 10.229.104.195 with SMTP id q3mr1674941qco.56.1269028881045; Fri, 
	19 Mar 2010 13:01:21 -0700 (PDT)
In-Reply-To: <F0EEBAC7-2C15-4C99-9153-58F20DD27A56@speakeasy.net>
References: <656B2F92-03D7-46AB-B49B-C8CBFE2F8BA9@speakeasy.net>
	 <7c962aed1003191151g18b6efc3m5b89385ade5cc29f@mail.gmail.com>
	 <F0EEBAC7-2C15-4C99-9153-58F20DD27A56@speakeasy.net>
Date: Fri, 19 Mar 2010 13:01:20 -0700
X-Google-Sender-Auth: a1a9cb0034593512
Message-ID: <7c962aed1003191301q1f5ceef7ua0e30fec30ed3bb5@mail.gmail.com>
Subject: Re: 0.21 Release
From: Stack <stack@duboce.net>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

On Fri, Mar 19, 2010 at 12:17 PM, Jeremy Davis <jerdavis@speakeasy.net> wrote:
> So what would be your (or anyones) advice on getting, HDFS sync/flush/append
> functionality?
> You seem to indicate that 0.21 branch as is might not be the best idea, with
> your preference being a patch set against 0.20.
>

There'll be a hadoop release that ships with the working sync/append
(In our testing, whats in the 0.21 branch+TRUNK, hdfs-265, works very
nicely).  I'm just not sure when.

St.Ack

From general-return-1229-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 20 18:07:47 2010
Return-Path: <general-return-1229-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94924 invoked from network); 20 Mar 2010 12:55:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Mar 2010 12:55:36 -0000
Received: (qmail 90864 invoked by uid 500); 20 Mar 2010 12:55:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90801 invoked by uid 500); 20 Mar 2010 12:55:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90792 invoked by uid 99); 20 Mar 2010 12:55:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 20 Mar 2010 12:55:34 +0000
X-ASF-Spam-Status: No, hits=-2.8 required=10.0
	tests=AWL,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of magawake@gmail.com designates 209.85.223.193 as permitted sender)
Received: from [209.85.223.193] (HELO mail-iw0-f193.google.com) (209.85.223.193)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 20 Mar 2010 12:55:29 +0000
Received: by iwn31 with SMTP id 31so3556087iwn.30
        for <general@hadoop.apache.org>; Sat, 20 Mar 2010 05:55:09 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=QiHWLRDGzMW/xIY3JfLoTrCUgpe/saye5ihsOpsaotY=;
        b=IZYDH+Scz076oJQEZj6GaHYjjKBmn0bJzBFEHXyXITKUE0VCtAPI/4Uhhl8LJIP77o
         e+R90jF0zmk7rE085QwwdHH1N+CFCBi3bq19VNWdyKwfnUOxB5HmMBeRPfE9c0vK+1vJ
         O/Wq0VAJ67QUcrstLmrnvv13Jvo9oGJCFcByo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=D7d3XCXAglmIvSQZ0p6pE4nH2zKmNRp2PyMmmn+uGfRPzP7rzywaAIeCvHRho8TVo5
         ouzou/JqsZkjZ3jLAvdLqe6Q1bHNUUgd32A/QyMSZpc8GfQykPI2o04szCbu6gLl5+p7
         2Bj4Xn+/9j76AaUqI2RUwSni1186G12bXwNJs=
MIME-Version: 1.0
Received: by 10.231.182.19 with SMTP id ca19mr643048ibb.58.1269089709012; Sat, 
	20 Mar 2010 05:55:09 -0700 (PDT)
In-Reply-To: <C7C8E2A1.3CA3%awittenauer@linkedin.com>
References: <1cbd6f831003190432l1845d62cl2c786a4a1e35d136@mail.gmail.com>
	 <C7C8E2A1.3CA3%awittenauer@linkedin.com>
Date: Sat, 20 Mar 2010 07:55:08 -0500
Message-ID: <1cbd6f831003200555o1f78c8f8p9dcef4a21ef94b8c@mail.gmail.com>
Subject: Re: rack awareness help
From: Mag Gam <magawake@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Thanks!

I managed to write a script which will give me rack01, rack02
depending on the ip address.

Now, how would I check how many racks there are in my cluster? That
not documented anywhere.

My intention is after I get all this working is to submit a bug report
for the documentation team so they can fix this.



On Fri, Mar 19, 2010 at 10:13 AM, Allen Wittenauer
<awittenauer@linkedin.com> wrote:
>
>
>
> On 3/19/10 4:32 AM, "Mag Gam" <magawake@gmail.com> wrote:
>
>> Thanks everyone. I think everyone can agree that this part of the
>> documentation is lacking for hadoop.
>>
>> Can someone please provide be a use case, for example:
>>
>> #server 1
>> Input > script.sh
>> Output > rack01
>>
>> #server 2
>> Input > script.sh
>> Output > rack02
>
> I think you have it in your head that the NameNode asks the DataNode what
> rack it is. =C2=A0This is completely backwards. =C2=A0The DataNode has *n=
o* concept of
> what a rack is. =C2=A0It is purely a storage process. =C2=A0There isn't m=
uch logic in
> it at all.
>
> The topology script is *ONLY* run by the NameNode and JobTracker processe=
s.
> That's it. =C2=A0It is not run on the compute nodes. =C2=A0That setting i=
s completely
> *ignored* by the DataNode and TaskTracker processes.
>
> So to rewrite your use case:
>
> # NameNode
> Input > server 1
> Output > rack01
>
> # NameNode
> Input > server 2
> Output > rack02
>
>> Is this how its supposed to work? I am bad with bash so I am trying to
>> understand the logic so I can implement it with another language such
>> as tcl
>
>
> The program logic is :
>
> Input -> IP address or Hostname
> Output -> /racklocation
>
> That's it. =C2=A0There is nothing fancy going on here.
>
>

From general-return-1230-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 20 18:08:38 2010
Return-Path: <general-return-1230-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 99223 invoked from network); 20 Mar 2010 13:46:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Mar 2010 13:46:51 -0000
Received: (qmail 21960 invoked by uid 500); 20 Mar 2010 13:46:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21910 invoked by uid 500); 20 Mar 2010 13:46:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21902 invoked by uid 99); 20 Mar 2010 13:46:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 20 Mar 2010 13:46:50 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of ctubbsii@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 20 Mar 2010 13:46:45 +0000
Received: by wyf19 with SMTP id 19so1792291wyf.35
        for <general@hadoop.apache.org>; Sat, 20 Mar 2010 06:46:24 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        bh=QQtfGys4jJO+bCer9nD+KgiRDwTpwzGn+YCSGSNzG1A=;
        b=AhthjcJSA4+RRvnFTZQfH62smePix7Ibp2nT4sIPbpWzcqTi1Gqc3P2jUlz+S0vITK
         kfg9nxeTBktwxFcnuDb2FRZy5SHC4y1GTgqLxbT/7ch/rgqkK8KOO0neavwp//wFQWfS
         kRp9I1E1jUx/JuFwQuUpHxcil+oNy4quPH9Jw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=igkKpT4mqEvRhpSPSYMiRaHIEDgusnWzAtdqdAgznv4ICtccXBxLK+wHlBv6Ya0AMN
         xydZrP6HX8ry56n57jn5Jg3SxAuPLA09BIj/Cbw5FF0JT7l4Pd1UUtZl1r2oY4elzPrI
         Muwyqb6XTbuvKBZ4tiO5C/Uv6pCtSt4nvG6SU=
MIME-Version: 1.0
Received: by 10.216.85.78 with SMTP id t56mr1724082wee.223.1269092783760; Sat, 
	20 Mar 2010 06:46:23 -0700 (PDT)
In-Reply-To: <1cbd6f831003200555o1f78c8f8p9dcef4a21ef94b8c@mail.gmail.com>
References: <1cbd6f831003190432l1845d62cl2c786a4a1e35d136@mail.gmail.com>
	 <C7C8E2A1.3CA3%awittenauer@linkedin.com>
	 <1cbd6f831003200555o1f78c8f8p9dcef4a21ef94b8c@mail.gmail.com>
Date: Sat, 20 Mar 2010 09:46:23 -0400
Message-ID: <9db00c201003200646y4f3ca892l6b066b6c9877a81e@mail.gmail.com>
Subject: Re: rack awareness help
From: Christopher Tubbs <ctubbsii@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I think hadoop's fsck will report the number of racks.

On Sat, Mar 20, 2010 at 8:55 AM, Mag Gam <magawake@gmail.com> wrote:
> Thanks!
>
> I managed to write a script which will give me rack01, rack02
> depending on the ip address.
>
> Now, how would I check how many racks there are in my cluster? That
> not documented anywhere.
>
> My intention is after I get all this working is to submit a bug report
> for the documentation team so they can fix this.
>
>
>
> On Fri, Mar 19, 2010 at 10:13 AM, Allen Wittenauer
> <awittenauer@linkedin.com> wrote:
>>
>>
>>
>> On 3/19/10 4:32 AM, "Mag Gam" <magawake@gmail.com> wrote:
>>
>>> Thanks everyone. I think everyone can agree that this part of the
>>> documentation is lacking for hadoop.
>>>
>>> Can someone please provide be a use case, for example:
>>>
>>> #server 1
>>> Input > script.sh
>>> Output > rack01
>>>
>>> #server 2
>>> Input > script.sh
>>> Output > rack02
>>
>> I think you have it in your head that the NameNode asks the DataNode wha=
t
>> rack it is. =A0This is completely backwards. =A0The DataNode has *no* co=
ncept of
>> what a rack is. =A0It is purely a storage process. =A0There isn't much l=
ogic in
>> it at all.
>>
>> The topology script is *ONLY* run by the NameNode and JobTracker process=
es.
>> That's it. =A0It is not run on the compute nodes. =A0That setting is com=
pletely
>> *ignored* by the DataNode and TaskTracker processes.
>>
>> So to rewrite your use case:
>>
>> # NameNode
>> Input > server 1
>> Output > rack01
>>
>> # NameNode
>> Input > server 2
>> Output > rack02
>>
>>> Is this how its supposed to work? I am bad with bash so I am trying to
>>> understand the logic so I can implement it with another language such
>>> as tcl
>>
>>
>> The program logic is :
>>
>> Input -> IP address or Hostname
>> Output -> /racklocation
>>
>> That's it. =A0There is nothing fancy going on here.
>>
>>
>

From general-return-1231-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 20 18:09:40 2010
Return-Path: <general-return-1231-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 12392 invoked from network); 20 Mar 2010 18:05:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Mar 2010 18:05:30 -0000
Received: (qmail 79042 invoked by uid 500); 20 Mar 2010 16:16:07 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 78942 invoked by uid 500); 20 Mar 2010 16:16:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 78926 invoked by uid 99); 20 Mar 2010 16:16:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 20 Mar 2010 16:16:07 +0000
X-ASF-Spam-Status: No, hits=0.1 required=10.0
	tests=AWL,SPF_HELO_PASS,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [64.81.36.244] (HELO foobar.kobold.org) (64.81.36.244)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 20 Mar 2010 16:16:02 +0000
Received: from owl.kobold.org (owl.kobold.org [192.168.1.7])
	(authenticated bits=0)
	by foobar.kobold.org (8.14.1/8.14.1) with ESMTP id o2KF1Qkp021361
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Sat, 20 Mar 2010 08:01:26 -0700
Message-ID: <4BA4F4AB.7000904@hep.caltech.edu>
Date: Sat, 20 Mar 2010 09:15:39 -0700
From: Michael Thomas <thomas@hep.caltech.edu>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.7) Gecko/20100120 Fedora/3.0.1-1.fc12 Thunderbird/3.0.1
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: rack awareness help
References: <1cbd6f831003190432l1845d62cl2c786a4a1e35d136@mail.gmail.com>	 <C7C8E2A1.3CA3%awittenauer@linkedin.com> <1cbd6f831003200555o1f78c8f8p9dcef4a21ef94b8c@mail.gmail.com>
In-Reply-To: <1cbd6f831003200555o1f78c8f8p9dcef4a21ef94b8c@mail.gmail.com>
Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms050308010006070902000402"

--------------ms050308010006070902000402
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: quoted-printable

On 03/20/2010 05:55 AM, Mag Gam wrote:
> Thanks!
>
> I managed to write a script which will give me rack01, rack02
> depending on the ip address.
>
> Now, how would I check how many racks there are in my cluster? That
> not documented anywhere.

'hadoop fsck /' will report the # of racks at the end:

[...]
  Number of data-nodes:          163
  Number of racks:               12

You can verify the IP-to-rack mappings with 'hadoop dfsadmin -report':
[...]
Name: 10.3.255.144:50010
Rack: /Rack12
[...]
Name: 10.3.255.62:50010
Rack: /Rack16

--Mike

> My intention is after I get all this working is to submit a bug report
> for the documentation team so they can fix this.
>
>
>
> On Fri, Mar 19, 2010 at 10:13 AM, Allen Wittenauer
> <awittenauer@linkedin.com>  wrote:
>>
>>
>>
>> On 3/19/10 4:32 AM, "Mag Gam"<magawake@gmail.com>  wrote:
>>
>>> Thanks everyone. I think everyone can agree that this part of the
>>> documentation is lacking for hadoop.
>>>
>>> Can someone please provide be a use case, for example:
>>>
>>> #server 1
>>> Input>  script.sh
>>> Output>  rack01
>>>
>>> #server 2
>>> Input>  script.sh
>>> Output>  rack02
>>
>> I think you have it in your head that the NameNode asks the DataNode w=
hat
>> rack it is.  This is completely backwards.  The DataNode has *no* conc=
ept of
>> what a rack is.  It is purely a storage process.  There isn't much log=
ic in
>> it at all.
>>
>> The topology script is *ONLY* run by the NameNode and JobTracker proce=
sses.
>> That's it.  It is not run on the compute nodes.  That setting is compl=
etely
>> *ignored* by the DataNode and TaskTracker processes.
>>
>> So to rewrite your use case:
>>
>> # NameNode
>> Input>  server 1
>> Output>  rack01
>>
>> # NameNode
>> Input>  server 2
>> Output>  rack02
>>
>>> Is this how its supposed to work? I am bad with bash so I am trying t=
o
>>> understand the logic so I can implement it with another language such=

>>> as tcl
>>
>>
>> The program logic is :
>>
>> Input ->  IP address or Hostname
>> Output ->  /racklocation
>>
>> That's it.  There is nothing fancy going on here.
>>
>>



--------------ms050308010006070902000402
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIMVjCC
A/gwggLgoAMCAQICASkwDQYJKoZIhvcNAQEFBQAwdTETMBEGCgmSJomT8ixkARkWA25ldDES
MBAGCgmSJomT8ixkARkWAkVTMQ4wDAYDVQQKEwVFU25ldDEgMB4GA1UECxMXQ2VydGlmaWNh
dGUgQXV0aG9yaXRpZXMxGDAWBgNVBAMTD0VTbmV0IFJvb3QgQ0EgMTAeFw0wMjEyMDUwODAw
MDBaFw0xMzAxMjUwODAwMDBaMGkxEzARBgoJkiaJk/IsZAEZFgNvcmcxGDAWBgoJkiaJk/Is
ZAEZFghET0VHcmlkczEgMB4GA1UECxMXQ2VydGlmaWNhdGUgQXV0aG9yaXRpZXMxFjAUBgNV
BAMTDURPRUdyaWRzIENBIDEwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC09dYj
YaPbCD5mtbiQb7Ka3y1qAm0ZcqKCFciWcfe8Kwcuy9tjHuIsLf9ZItdkDW4xy8sua9nJlx3K
lwjtumTMtOtg35KZCknUd8KM4VGTSFdLVG9AbNayef76caVCGM1+jyF0Lq03kauGOPTcNfZe
1TZa3e1c9rc8ljV5OSWa/mfsCACyS5zFIWu0yIDNyJdf+n0hwaPN53wllpJ30taD+JBjQ7h2
k4xRWzeaznLOb9OztZVRA/1sVze+iczFh2xwa4VdGy0eIIPw1pfvYwxO36rm0S109qvbsNla
roPRbxerPKakQLpKe034Xcx7gBPqUk/FxoRRWin5EWN3rz9LAgMBAAGjgZ4wgZswDgYDVR0P
AQH/BAQDAgGGMBEGCWCGSAGG+EIBAQQEAwIAhzAdBgNVHQ4EFgQUyhkdEo5upDhdQtQxDgjb
2Y0XDV0wHwYDVR0jBBgwFoAUvF1NSC/4NZRZq1yJSz7RsjoUAeowDwYDVR0TAQH/BAUwAwEB
/zAlBgNVHREEHjAcgRpET0VHcmlkcy1DQS0xQGRvZWdyaWRzLm9yZzANBgkqhkiG9w0BAQUF
AAOCAQEAZNVrIDLqe39CEOiJt7Q7EpBPhAihMvDTSf/42u0SMbUmChww4mLmph5DBghZUVF8
Yn59kRZMn1QLOtO1HzLqvAvPITacZVPlJgG2IXzlR636YghZFAycbIUEOJDBHR4vtQO1KDxg
ZwvAbtmKIoxvhUCq2xsfFt9kCBBn+JYtQ6O5LsBJq3PmuubeMcc7mbQAfJZ7h/3QghgkFIhm
E1+LBXPJbkuP8vgfg6h2BKoAf5TFfZECgGZKimfN110tBvfedGZwYYd3/GsJc83B0JN1gny0
gqNVPm392UchXGeBRrHnm2gkhIkr48Oq6EmNGV9/a6XfbplQW/JWbtPVPWkaizCCBCkwggMR
oAMCAQICAwChZzANBgkqhkiG9w0BAQUFADBpMRMwEQYKCZImiZPyLGQBGRYDb3JnMRgwFgYK
CZImiZPyLGQBGRYIRE9FR3JpZHMxIDAeBgNVBAsTF0NlcnRpZmljYXRlIEF1dGhvcml0aWVz
MRYwFAYDVQQDEw1ET0VHcmlkcyBDQSAxMB4XDTEwMDEyNTE4MzEyMFoXDTExMDEyNTE4MzEy
MFowYDETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCGRvZWdyaWRzMQ8w
DQYDVQQLEwZQZW9wbGUxHjAcBgNVBAMTFU1pY2hhZWwgVGhvbWFzIDcyNTM3NDCCASIwDQYJ
KoZIhvcNAQEBBQADggEPADCCAQoCggEBALchCLXStiJKw5u2+nFW0020jqo1X8/kRB9kus8a
6Op5ds3j5O3IZkyZ41tQwE9Rc0yRDiDGpT1jL/n2LK1HPXvZbasmPCxxTMY3kNTm4fWAnRS7
UzLo0Adaz81bcT8Buo7Il1FCS684LhsK19JChqU2/iY8G7H0uBImj6QDLR0+w5QsLfD7aOXx
M8Njhse1ZqMyjwjhw5FqWyWNhlKi1oW42eXtYhbsqyAQZSXLoz2SwqBnLTRqcRMn4GgvB83Z
1FObyb79HFiuqlbaMlXWy0s8ywajJnMG8LsgwkNJZQLJsPalz5rwe/pOG1abasLMBVpJGGSB
AQktY8s5M+U9RlMCAwEAAaOB4jCB3zARBglghkgBhvhCAQEEBAMCBeAwDgYDVR0PAQH/BAQD
AgTwMDYGA1UdIAQvMC0wDQYLKoZIhvdMAwcBAwEwDAYKKoZIhvdMBQICATAOBgwqhkiG90wF
AgMCAQEwPgYDVR0fBDcwNTAzoDGgL4YtaHR0cDovL2NybC5kb2Vncmlkcy5vcmcvMWMzZjJj
YTgvMWMzZjJjYTguY3JsMCEGA1UdEQQaMBiBFnRob21hc0BoZXAuY2FsdGVjaC5lZHUwHwYD
VR0jBBgwFoAUyhkdEo5upDhdQtQxDgjb2Y0XDV0wDQYJKoZIhvcNAQEFBQADggEBAGcleFpm
d9Ho37gHZrrF0zulSQ3Aemrbx2O4EVJotQbtAs2v4nnH0eAa0F37BXRA6DAvgaC4YODA6z5M
HHE2mnsIebv0nS8DLj1yN7C1nqDSUXlaIzExEGQ1vXG2XlVo6pMZgJd2ML9+dyUUOcnNj0kQ
GeMysWBHbDChdfVG8y2PuGzoFY/ikNtWWhTNcPVZHIL5I2JCYqV5lyLQhlkC0Q5eqbqYKAq1
GuQmLO8hPfzkb6qbAkzxB8/FFqjlPgBgFcd9Nf2fAI14kI77iRbEuu8jSTPx13p6uYvcX4hy
1cW4Y7lnE8bBkAFWrUuNGc+aw7Ang6rJANWXtF/tN+UqUOcwggQpMIIDEaADAgECAgMAoWcw
DQYJKoZIhvcNAQEFBQAwaTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkW
CERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UEAxMN
RE9FR3JpZHMgQ0EgMTAeFw0xMDAxMjUxODMxMjBaFw0xMTAxMjUxODMxMjBaMGAxEzARBgoJ
kiaJk/IsZAEZFgNvcmcxGDAWBgoJkiaJk/IsZAEZFghkb2VncmlkczEPMA0GA1UECxMGUGVv
cGxlMR4wHAYDVQQDExVNaWNoYWVsIFRob21hcyA3MjUzNzQwggEiMA0GCSqGSIb3DQEBAQUA
A4IBDwAwggEKAoIBAQC3IQi10rYiSsObtvpxVtNNtI6qNV/P5EQfZLrPGujqeXbN4+TtyGZM
meNbUMBPUXNMkQ4gxqU9Yy/59iytRz172W2rJjwscUzGN5DU5uH1gJ0Uu1My6NAHWs/NW3E/
AbqOyJdRQkuvOC4bCtfSQoalNv4mPBux9LgSJo+kAy0dPsOULC3w+2jl8TPDY4bHtWajMo8I
4cORalsljYZSotaFuNnl7WIW7KsgEGUly6M9ksKgZy00anETJ+BoLwfN2dRTm8m+/RxYrqpW
2jJV1stLPMsGoyZzBvC7IMJDSWUCybD2pc+a8Hv6ThtWm2rCzAVaSRhkgQEJLWPLOTPlPUZT
AgMBAAGjgeIwgd8wEQYJYIZIAYb4QgEBBAQDAgXgMA4GA1UdDwEB/wQEAwIE8DA2BgNVHSAE
LzAtMA0GCyqGSIb3TAMHAQMBMAwGCiqGSIb3TAUCAgEwDgYMKoZIhvdMBQIDAgEBMD4GA1Ud
HwQ3MDUwM6AxoC+GLWh0dHA6Ly9jcmwuZG9lZ3JpZHMub3JnLzFjM2YyY2E4LzFjM2YyY2E4
LmNybDAhBgNVHREEGjAYgRZ0aG9tYXNAaGVwLmNhbHRlY2guZWR1MB8GA1UdIwQYMBaAFMoZ
HRKObqQ4XULUMQ4I29mNFw1dMA0GCSqGSIb3DQEBBQUAA4IBAQBnJXhaZnfR6N+4B2a6xdM7
pUkNwHpq28djuBFSaLUG7QLNr+J5x9HgGtBd+wV0QOgwL4GguGDgwOs+TBxxNpp7CHm79J0v
Ay49cjewtZ6g0lF5WiMxMRBkNb1xtl5VaOqTGYCXdjC/fnclFDnJzY9JEBnjMrFgR2wwoXX1
RvMtj7hs6BWP4pDbVloUzXD1WRyC+SNiQmKleZci0IZZAtEOXqm6mCgKtRrkJizvIT385G+q
mwJM8QfPxRao5T4AYBXHfTX9nwCNeJCO+4kWxLrvI0kz8dd6ermL3F+IctXFuGO5ZxPGwZAB
Vq1LjRnPmsOwJ4OqyQDVl7Rf7TflKlDnMYIDXjCCA1oCAQEwcDBpMRMwEQYKCZImiZPyLGQB
GRYDb3JnMRgwFgYKCZImiZPyLGQBGRYIRE9FR3JpZHMxIDAeBgNVBAsTF0NlcnRpZmljYXRl
IEF1dGhvcml0aWVzMRYwFAYDVQQDEw1ET0VHcmlkcyBDQSAxAgMAoWcwCQYFKw4DAhoFAKCC
AcMwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTAwMzIwMTYx
NTM5WjAjBgkqhkiG9w0BCQQxFgQUiI+5+lN9+PrFyKK0Q+KZ4E8OGrQwXwYJKoZIhvcNAQkP
MVIwUDALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3
DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMH8GCSsGAQQBgjcQBDFyMHAwaTETMBEG
CgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixkARkWCERPRUdyaWRzMSAwHgYDVQQLExdD
ZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UEAxMNRE9FR3JpZHMgQ0EgMQIDAKFnMIGB
BgsqhkiG9w0BCRACCzFyoHAwaTETMBEGCgmSJomT8ixkARkWA29yZzEYMBYGCgmSJomT8ixk
ARkWCERPRUdyaWRzMSAwHgYDVQQLExdDZXJ0aWZpY2F0ZSBBdXRob3JpdGllczEWMBQGA1UE
AxMNRE9FR3JpZHMgQ0EgMQIDAKFnMA0GCSqGSIb3DQEBAQUABIIBAGhxyOWwtrXa+RWMSAbV
Yo6oQrttFiwTbMDW5NXJXsvhv7WhXgUrV7PV/ecv3hLQWHLbnFsryKfSPXqrRhzbhHqh01ym
PWX6lYCCpoVkPSFSkvEX8XUSB+wN1iAzBDnS/wMNBE83GjIpGE8WFcGX7n6mBNqdyOENEpLw
SADygqkncHHw+LpyWKtrWsjKbfRpOklT1kxk3J8X43FEXulSqQBifxF/AIgvthBixtsbd44W
vRKrIvBdzlxJ2SQx2V8/AkQJt1MdmZL57aD7sdYAFpb/GLzo1xM/vigHlLCMtCon9eswhM8y
34wAROrjh2EQ62GrM+bVAcZP08UDaBk+JB0AAAAAAAA=
--------------ms050308010006070902000402--

From general-return-1232-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 22 04:09:01 2010
Return-Path: <general-return-1232-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 22921 invoked from network); 22 Mar 2010 04:09:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Mar 2010 04:09:01 -0000
Received: (qmail 81849 invoked by uid 500); 22 Mar 2010 03:06:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 81665 invoked by uid 500); 22 Mar 2010 03:06:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 81657 invoked by uid 99); 22 Mar 2010 03:06:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Mar 2010 03:06:58 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.221.198] (HELO mail-qy0-f198.google.com) (209.85.221.198)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Mar 2010 03:06:48 +0000
Received: by qyk36 with SMTP id 36so3167603qyk.30
        for <general@hadoop.apache.org>; Sun, 21 Mar 2010 20:06:27 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.219.143 with SMTP id hu15mr2734223qcb.12.1269227187420; 
	Sun, 21 Mar 2010 20:06:27 -0700 (PDT)
In-Reply-To: <bd7d2ecd1003051950jc21e4d0pe682b29866dce7b4@mail.gmail.com>
References: <bd7d2ecd1003051950jc21e4d0pe682b29866dce7b4@mail.gmail.com>
From: Aaron Kimball <aaron@cloudera.com>
Date: Sun, 21 Mar 2010 20:06:07 -0700
Message-ID: <d6d7c4411003212006i521115a2mdbc260370d411e99@mail.gmail.com>
Subject: Re: How to send KV pair to a reduce task on a particular machine?
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636164079a53f7304825afb91
X-Virus-Checked: Checked by ClamAV on apache.org

--001636164079a53f7304825afb91
Content-Type: text/plain; charset=ISO-8859-1

Yanfeng,

The sort of behavior you want is intentionally omitted from MapReduce's
capabilities. Reduce partitions are kept as abstract notions and your
MapReduce program cannot bind partitions to particular physical nodes. This
is for fault-tolerance purposes. If machine1 crashes, then partition1 can
still be rescheduled onto machine3 and the computation can continue.

Sorry that's frustrating for your use case!
- Aaron

On Fri, Mar 5, 2010 at 8:50 PM, Yanfeng Zhang <zhangyf14@gmail.com> wrote:

> Hi, all
>
> The KV pairs (kv1, kv2, kv3 kv4) out from mapper would be partitioned into
> R
> parts (e.g. R=2) by a partitioner. For example, kv1 and kv2 are in
> partition1, while kv3 and kv4 are in partition2, the reducers will get KV
> pairs from these two partitions, reducer1 get KV pairs from partition1 and
> reducer2 get KV pairs from partition2.
>
> I want to let machine1 get KV pairs from partition1 and machine2 get KV
> pairs from partition2. But reducer1 is not always on machine1, reducer2 is
> not always on machine2. Is there any way to make sure kv1 and kv2 are sent
> to machine1 and kv3, kv4 are sent to machine2?
>
> Thank you in advance!
>
> Sincerely,
> --
> Yanfeng Zhang
>

--001636164079a53f7304825afb91--

From general-return-1233-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 22 13:32:47 2010
Return-Path: <general-return-1233-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 66459 invoked from network); 22 Mar 2010 13:32:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Mar 2010 13:32:46 -0000
Received: (qmail 91221 invoked by uid 500); 22 Mar 2010 13:32:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91153 invoked by uid 500); 22 Mar 2010 13:32:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91145 invoked by uid 99); 22 Mar 2010 13:32:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Mar 2010 13:32:45 +0000
X-ASF-Spam-Status: No, hits=-3.0 required=10.0
	tests=AWL,RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.2] (HELO colossus.hpl.hp.com) (192.6.10.2)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Mar 2010 13:32:37 +0000
Received: from localhost (localhost [127.0.0.1])
	by colossus.hpl.hp.com (Postfix) with ESMTP id 7D5681BA4FA
	for <general@hadoop.apache.org>; Mon, 22 Mar 2010 13:32:15 +0000 (GMT)
X-Virus-Scanned: Debian amavisd-new at hpl.hp.com
Received: from colossus.hpl.hp.com ([127.0.0.1])
	by localhost (colossus.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id YYy6ne1HiOfB for <general@hadoop.apache.org>;
	Mon, 22 Mar 2010 13:32:15 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by colossus.hpl.hp.com (Postfix) with ESMTPS id 04C0A1BA4E4
	for <general@hadoop.apache.org>; Mon, 22 Mar 2010 13:32:14 +0000 (GMT)
MailScanner-NULL-Check: 1269869523.08496@uTyJjD+XQQ8lvCj7gQ7W/g
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o2MDVx4c014475
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Mon, 22 Mar 2010 13:32:00 GMT
Message-ID: <4BA77159.4090902@apache.org>
Date: Mon, 22 Mar 2010 13:32:09 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100228)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: 0.21 Release
References: <656B2F92-03D7-46AB-B49B-C8CBFE2F8BA9@speakeasy.net> <7c962aed1003191151g18b6efc3m5b89385ade5cc29f@mail.gmail.com>
In-Reply-To: <7c962aed1003191151g18b6efc3m5b89385ade5cc29f@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o2MDVx4c014475
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

Stack wrote:
> On Fri, Mar 19, 2010 at 9:43 AM, Jeremy Davis <jerdavis@speakeasy.net> wrote:
>> ...and I also saw release 0.21 as is with stack as the release manager. Was
>> there a final decision on this off list?
>>
> 
> On the above, I was toying with the idea of being release manager for
> releasing as hadoop 0.21.0 what is in current 0.21 hadoop branch but I
> subsequently decided against it after chatting with folks and figuring
> that the only group that seemed interested in driving a release of the
> hadoop 0.21 branch was the hbase crew.  If I were to guess, an hadoop
> vouched for by a couple of hbasers with their spotty hdfs and
> mapreduce knowledge probably wouldn't have the penetration of a
> release backed by, say, a Yahoo.  No one would trust their data to
> such a release.  If no data in hadoop 0.21 clusters, hbase wouldn't
> have anything to run against.  So I let it go and figured time could
> be spent better elsewhere; e.g. helping test the set of patches that
> could get us a sync/flush/append on a patched hadoop 0.20 (hdfs-200,
> etc.).
> 
> Sorry, I should have added a note to cited thread that I'd wandered...
> 
> St.Ack

I'm not going to volunteer to help with this as my changes are still not 
in 0.22, which is what I'm trying to target, but I'd certainly approve 
of cutting a release if only because
  - it ensures the release process itself works well. When we were doing 
releases every fortnight at work, you make sure everything is automated 
that can be automated.
  - with 0.22 adding security, I expect its deployment will be traumatic 
and take longer to roll out than anyone expects
  - there are lots of improvements in 0.21; getting into a world of 
backports is complicated. Better to keep momentum up.

The big concern has to be the filesystem. Who out there is willing/able 
to test 0.21-based filesystems, and what size clusters do they have?

-steve

From general-return-1234-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 22 17:12:00 2010
Return-Path: <general-return-1234-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57357 invoked from network); 22 Mar 2010 17:11:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Mar 2010 17:11:59 -0000
Received: (qmail 5435 invoked by uid 500); 22 Mar 2010 17:11:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5378 invoked by uid 500); 22 Mar 2010 17:11:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5370 invoked by uid 99); 22 Mar 2010 17:11:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Mar 2010 17:11:58 +0000
X-ASF-Spam-Status: No, hits=2.8 required=10.0
	tests=AWL,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,FS_REPLICA,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mailinglists19@gmail.com designates 209.85.216.192 as permitted sender)
Received: from [209.85.216.192] (HELO mail-px0-f192.google.com) (209.85.216.192)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Mar 2010 17:11:52 +0000
Received: by pxi30 with SMTP id 30so3707707pxi.20
        for <general@hadoop.apache.org>; Mon, 22 Mar 2010 10:11:32 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=rSH/0+4L+E0VS9xEEeQ8FQJCxq2op2Ogrho/LSDxvDg=;
        b=XiiiH0HRnzvAXPZr8FInpAMSfyPhiW3eAE5MibmiIXscgMIDxDSf5f7xY5VUXr6suX
         aBEHp7eHuQJ2ESAzrz6ZpEKsl/mE+ryejX5i9pnkrJmD5+U5tIYgAx90F5tSju/VHYY+
         qYrXrQpEjvFoVXnKNXyx9ECSEwobfEfgux3UA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=IOK7FUJIuXeIAI4Y/shVBdTao4miMFkaTLE3KkWoACDnkrdfFfgGpDiJAMdenmjpKM
         uxJjg1ZdpJNVkfqYMAtTquZFkoOVyCytaIKUCvtlj4jsgutTC8KcOxYI/4NllCwnSe8N
         FoZb9+8XR1geAKKPSrGNi3xmaAaegKPdaKmsE=
MIME-Version: 1.0
Received: by 10.140.57.5 with SMTP id f5mr5152697rva.173.1269277891715; Mon, 
	22 Mar 2010 10:11:31 -0700 (PDT)
Date: Mon, 22 Mar 2010 10:11:31 -0700
Message-ID: <1eabbac31003221011x468ebbabw5776550bbaf2d9e5@mail.gmail.com>
Subject: Error: could only be replicated to 0 nodes, instead of 1
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636b2b0e3db66dc048266c95a

--001636b2b0e3db66dc048266c95a
Content-Type: text/plain; charset=ISO-8859-1

After upgrading to hadoop-0.20.2, I started seeing these messages in the
log:


2010-03-22 09:56:28,393 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 1 on 9000, call addBlock(/home/training/hadoop/mapred/system/
jobtracker.info, DFSClient_-1956918169) from 127.0.0.1:41067: error:
java.io.IOException: File /home/training/hadoop/mapred/system/
jobtracker.info could only be replicated to 0 nodes, instead of 1
java.io.IOException: File /home/training/hadoop/mapred/system/
jobtracker.info could only be replicated to 0 nodes, instead of 1
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)


I am running in Psuedo-distributed mode.  Before upgrading I used to see
these messages occasionally, but they were harmless.  Now I see these when I
try to copy files to DFS using -copyFromLocal, and the files no longer get
copied :(

I mean, it's quite possible, this has nothing to do with the Hadoop upgrade
but something changed in my environment, but any help will be appreciated.
 Thanks.

--001636b2b0e3db66dc048266c95a--

From general-return-1235-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 22 17:33:31 2010
Return-Path: <general-return-1235-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 65906 invoked from network); 22 Mar 2010 17:33:31 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Mar 2010 17:33:31 -0000
Received: (qmail 45567 invoked by uid 500); 22 Mar 2010 17:33:30 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45493 invoked by uid 500); 22 Mar 2010 17:33:30 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45485 invoked by uid 99); 22 Mar 2010 17:33:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Mar 2010 17:33:29 +0000
X-ASF-Spam-Status: No, hits=1.6 required=10.0
	tests=AWL,FREEMAIL_FROM,FS_REPLICA,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jaybooth@gmail.com designates 209.85.211.174 as permitted sender)
Received: from [209.85.211.174] (HELO mail-yw0-f174.google.com) (209.85.211.174)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Mar 2010 17:33:23 +0000
Received: by ywh4 with SMTP id 4so760577ywh.5
        for <general@hadoop.apache.org>; Mon, 22 Mar 2010 10:33:00 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=aI9SVQEfJDvr3JojD8Zn255+8pnSE4UGlWdcGEXhRjc=;
        b=GKgIj6kmBGZjacefu7H/E8LLbuOsH2HDWs+R8eGUfiJTNunoiBeiwRrcBMn5bRIzdd
         RhVAOH76Emdz/640MLFWZrVez/Bc9N/ozokH36N6oz9XEt3IyJXyDMcTdVvfrzbQKRIw
         ThpuPypinz1HdfANavJ4IGSmGh3pEs1YLz8YQ=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=awHV4vZ4x1f8pgQlewvk4nWFMD0Qcthxkf6VxOXMsY5ptzr98ZO4VW/pVfaF4I/0LR
         2r/m3UFSCHam37Pr0kiVGvAXtnLmMU/yiosYw6sxHpWQ8z4bj6vbNxb/03qfU3NDP0PQ
         aP9AaYWXVoZw6+SRuVDZtaYIftFmaI9rSv0WI=
MIME-Version: 1.0
Received: by 10.101.137.13 with SMTP id p13mr10619684ann.146.1269279180510; 
	Mon, 22 Mar 2010 10:33:00 -0700 (PDT)
In-Reply-To: <1eabbac31003221011x468ebbabw5776550bbaf2d9e5@mail.gmail.com>
References: <1eabbac31003221011x468ebbabw5776550bbaf2d9e5@mail.gmail.com>
Date: Mon, 22 Mar 2010 13:33:00 -0400
Message-ID: <54dc3c51003221033i5d9a4299iddb239a03ae7a763@mail.gmail.com>
Subject: Re: Error: could only be replicated to 0 nodes, instead of 1
From: Jay Booth <jaybooth@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e68ee28cacd81704826716fc

--0016e68ee28cacd81704826716fc
Content-Type: text/plain; charset=ISO-8859-1

Usually when I see this in pseudo-distributed, it means the datanode isn't
up.  Check logs?

On Mon, Mar 22, 2010 at 1:11 PM, Something Something <
mailinglists19@gmail.com> wrote:

> After upgrading to hadoop-0.20.2, I started seeing these messages in the
> log:
>
>
> 2010-03-22 09:56:28,393 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 9000, call addBlock(/home/training/hadoop/mapred/system/
> jobtracker.info, DFSClient_-1956918169) from 127.0.0.1:41067: error:
> java.io.IOException: File /home/training/hadoop/mapred/system/
> jobtracker.info could only be replicated to 0 nodes, instead of 1
> java.io.IOException: File /home/training/hadoop/mapred/system/
> jobtracker.info could only be replicated to 0 nodes, instead of 1
>        at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at javax.security.auth.Subject.doAs(Subject.java:396)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
>
> I am running in Psuedo-distributed mode.  Before upgrading I used to see
> these messages occasionally, but they were harmless.  Now I see these when
> I
> try to copy files to DFS using -copyFromLocal, and the files no longer get
> copied :(
>
> I mean, it's quite possible, this has nothing to do with the Hadoop upgrade
> but something changed in my environment, but any help will be appreciated.
>  Thanks.
>

--0016e68ee28cacd81704826716fc--

From general-return-1236-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 22 17:39:59 2010
Return-Path: <general-return-1236-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67853 invoked from network); 22 Mar 2010 17:39:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Mar 2010 17:39:59 -0000
Received: (qmail 56859 invoked by uid 500); 22 Mar 2010 17:39:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 56816 invoked by uid 500); 22 Mar 2010 17:39:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 56808 invoked by uid 99); 22 Mar 2010 17:39:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Mar 2010 17:39:58 +0000
X-ASF-Spam-Status: No, hits=3.1 required=10.0
	tests=AWL,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,FS_REPLICA,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mailinglists19@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Mar 2010 17:39:53 +0000
Received: by pwi7 with SMTP id 7so3903295pwi.35
        for <general@hadoop.apache.org>; Mon, 22 Mar 2010 10:39:33 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=QBiW5XBQPvq12Mzft81tZjzXNIDbXoNoDrDJABuLBbc=;
        b=uIXcBVs7bzDCwqkBQV7irt7Xqmz5KKhiA82ql7aFBoyTFlEsJSLw8+zh7ZpVs37cxG
         Hh6zaEQcYlrSQhRd27gNG6kza4AxEy55ItSd/ca+U7YrcbIHquRR74Nf+MvX6WMl0Ot6
         QrUaeQJBv05yAHmQ5+7dWHwKTxootVAiBhoGw=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=eBuQGLymLJMu9MoOdCQ0NztfXgAPqymJ0rs02oPa1eRE0lHPTAL7w9/G/hbFtsiXCe
         jnL6sMzzK53uxLa2AjRfd4sxfRi4f+FZb0dkPtLftW8rW4RBjqyCM9aBdCiSBr3mPsMW
         TpUvCarsuS32gxR8bzN+iYeHPRGFOTmH64iJI=
MIME-Version: 1.0
Received: by 10.141.187.9 with SMTP id o9mr4279792rvp.211.1269279573081; Mon, 
	22 Mar 2010 10:39:33 -0700 (PDT)
In-Reply-To: <54dc3c51003221033i5d9a4299iddb239a03ae7a763@mail.gmail.com>
References: <1eabbac31003221011x468ebbabw5776550bbaf2d9e5@mail.gmail.com>
	 <54dc3c51003221033i5d9a4299iddb239a03ae7a763@mail.gmail.com>
Date: Mon, 22 Mar 2010 10:39:33 -0700
Message-ID: <1eabbac31003221039kda88dd6pcf9e59fcf4d48634@mail.gmail.com>
Subject: Re: Error: could only be replicated to 0 nodes, instead of 1
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd1acf01300dd0482672e8f

--000e0cd1acf01300dd0482672e8f
Content-Type: text/plain; charset=ISO-8859-1

Everything in datanode log looks normal to me.  Here it is..  Let me know if
you see any problem.  Thanks.

2010-03-22 09:55:02,578 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG:   host = training-vm/127.0.0.1
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 0.20.2
STARTUP_MSG:   build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2010-03-22 09:55:18,091 INFO org.apache.hadoop.hdfs.server.common.Storage:
Storage directory /home/training/hadoop/dfs/data is not formatted.
2010-03-22 09:55:18,093 INFO org.apache.hadoop.hdfs.server.common.Storage:
Formatting ...
2010-03-22 09:55:18,527 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Registered
FSDatasetStatusMBean
2010-03-22 09:55:18,533 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Opened info server at 50010
2010-03-22 09:55:18,560 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is
1048576 bytes/s
2010-03-22 09:55:33,986 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2010-03-22 09:55:34,386 INFO org.apache.hadoop.http.HttpServer: Port
returned by webServer.getConnectors()[0].getLocalPort() before open() is -1.
Opening the listener on 50075
2010-03-22 09:55:34,386 INFO org.apache.hadoop.http.HttpServer:
listener.getLocalPort() returned 50075
webServer.getConnectors()[0].getLocalPort() returned 50075
2010-03-22 09:55:34,386 INFO org.apache.hadoop.http.HttpServer: Jetty bound
to port 50075
2010-03-22 09:55:34,387 INFO org.mortbay.log: jetty-6.1.14
2010-03-22 09:57:00,445 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:50075
2010-03-22 09:57:00,480 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=DataNode, sessionId=null
2010-03-22 09:57:15,652 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=DataNode, port=50020
2010-03-22 09:57:15,676 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2010-03-22 09:57:15,713 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 50020: starting
2010-03-22 09:57:15,738 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 50020: starting
2010-03-22 09:57:15,766 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 1 on 50020: starting
2010-03-22 09:57:15,836 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration =
DatanodeRegistration(localhost:50010, storageID=, infoPort=50075,
ipcPort=50020)
2010-03-22 09:57:15,880 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 2 on 50020: starting
2010-03-22 09:57:15,988 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: New storage id
DS-3114065-127.0.0.1-50010-1269277035840 is assigned to data-node
127.0.0.1:50010
2010-03-22 09:57:16,022 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
127.0.0.1:50010, storageID=DS-3114065-127.0.0.1-50010-1269277035840,
infoPort=50075, ipcPort=50020)In DataNode.run, data =
FSDataset{dirpath='/home/training/hadoop/dfs/data/current'}
2010-03-22 09:57:16,032 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: using BLOCKREPORT_INTERVAL
of 3600000msec Initial delay: 0msec
2010-03-22 09:57:16,243 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 0 blocks got
processed in 16 msecs
2010-03-22 09:57:16,258 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Starting Periodic block
scanner.
2010-03-22 10:34:44,492 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 0 blocks got
processed in 3 msecs




On Mon, Mar 22, 2010 at 10:33 AM, Jay Booth <jaybooth@gmail.com> wrote:

> Usually when I see this in pseudo-distributed, it means the datanode isn't
> up.  Check logs?
>
> On Mon, Mar 22, 2010 at 1:11 PM, Something Something <
> mailinglists19@gmail.com> wrote:
>
> > After upgrading to hadoop-0.20.2, I started seeing these messages in the
> > log:
> >
> >
> > 2010-03-22 09:56:28,393 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 1 on 9000, call addBlock(/home/training/hadoop/mapred/system/
> > jobtracker.info, DFSClient_-1956918169) from 127.0.0.1:41067: error:
> > java.io.IOException: File /home/training/hadoop/mapred/system/
> > jobtracker.info could only be replicated to 0 nodes, instead of 1
> > java.io.IOException: File /home/training/hadoop/mapred/system/
> > jobtracker.info could only be replicated to 0 nodes, instead of 1
> >        at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
> >        at
> >
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
> >        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >        at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >        at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >        at java.lang.reflect.Method.invoke(Method.java:597)
> >        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> >        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> >        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> >        at java.security.AccessController.doPrivileged(Native Method)
> >        at javax.security.auth.Subject.doAs(Subject.java:396)
> >        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> >
> >
> > I am running in Psuedo-distributed mode.  Before upgrading I used to see
> > these messages occasionally, but they were harmless.  Now I see these
> when
> > I
> > try to copy files to DFS using -copyFromLocal, and the files no longer
> get
> > copied :(
> >
> > I mean, it's quite possible, this has nothing to do with the Hadoop
> upgrade
> > but something changed in my environment, but any help will be
> appreciated.
> >  Thanks.
> >
>

--000e0cd1acf01300dd0482672e8f--

From general-return-1237-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 22 20:58:27 2010
Return-Path: <general-return-1237-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 36857 invoked from network); 22 Mar 2010 20:58:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Mar 2010 20:58:27 -0000
Received: (qmail 2913 invoked by uid 500); 22 Mar 2010 20:58:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 2780 invoked by uid 500); 22 Mar 2010 20:58:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 2772 invoked by uid 99); 22 Mar 2010 20:58:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Mar 2010 20:58:26 +0000
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=AWL,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,FS_REPLICA,HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mailinglists19@gmail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 22 Mar 2010 20:58:20 +0000
Received: by wyf19 with SMTP id 19so2567861wyf.35
        for <general@hadoop.apache.org>; Mon, 22 Mar 2010 13:57:59 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=X3jqs6qkFBRwWVHu8EW5B9J12YtggAUlg5UbMB4Xiqo=;
        b=nzwTvIRoSAi8W07htUppCz8K+uNY9gUC8ZajNhTnIjHO9rJwuTyyFu8J6nFGiBBAmZ
         zjn3Qienx85mAI4PC75KXsPaCicZ7atbXbmFp6gZnfBQpbuRHorJRPVRdHiRj8kSl8S4
         4Xr2SXzVDYqRz7h6CwU/A6Z1ABrF5hAA++5Zo=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=K0nwBhIrBZWOeDx6f/Og/BjQIMgA24PVX42Nmw2bVA09+uSiPITNEEvGQEhj5L9IDZ
         QK9D6fIDRIRaTLBPFjkn9RBK6M9OxY1myjgoCtnx39fjMt3y0rAeT1Xh1T9xzvTswUlO
         aedPsctwyV0xI7kDgkf71zu7M59Wrd2vV0Ao0=
MIME-Version: 1.0
Received: by 10.216.87.12 with SMTP id x12mr53501wee.185.1269291478924; Mon, 
	22 Mar 2010 13:57:58 -0700 (PDT)
In-Reply-To: <1eabbac31003221039kda88dd6pcf9e59fcf4d48634@mail.gmail.com>
References: <1eabbac31003221011x468ebbabw5776550bbaf2d9e5@mail.gmail.com>
	 <54dc3c51003221033i5d9a4299iddb239a03ae7a763@mail.gmail.com>
	 <1eabbac31003221039kda88dd6pcf9e59fcf4d48634@mail.gmail.com>
Date: Mon, 22 Mar 2010 13:57:58 -0700
Message-ID: <1eabbac31003221357x2cd5616avfbb687c6e0f03864@mail.gmail.com>
Subject: Re: Error: could only be replicated to 0 nodes, instead of 1
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6d7e0a9b7c043048269f3ce

--0016e6d7e0a9b7c043048269f3ce
Content-Type: text/plain; charset=ISO-8859-1

Seems like this was a space problem.  I deleted bunch of files and now I am
able to copy files to DFS.  I still see the error message in the log, but
it's harmless - I guess!

On Mon, Mar 22, 2010 at 10:39 AM, Something Something <
mailinglists19@gmail.com> wrote:

> Everything in datanode log looks normal to me.  Here it is..  Let me know
> if you see any problem.  Thanks.
>
> 2010-03-22 09:55:02,578 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG:   host = training-vm/127.0.0.1
> STARTUP_MSG:   args = []
> STARTUP_MSG:   version = 0.20.2
> STARTUP_MSG:   build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> ************************************************************/
> 2010-03-22 09:55:18,091 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Storage directory /home/training/hadoop/dfs/data is not formatted.
> 2010-03-22 09:55:18,093 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Formatting ...
> 2010-03-22 09:55:18,527 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Registered
> FSDatasetStatusMBean
> 2010-03-22 09:55:18,533 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Opened info server at 50010
> 2010-03-22 09:55:18,560 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is
> 1048576 bytes/s
> 2010-03-22 09:55:33,986 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2010-03-22 09:55:34,386 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open() is -1.
> Opening the listener on 50075
> 2010-03-22 09:55:34,386 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50075
> webServer.getConnectors()[0].getLocalPort() returned 50075
> 2010-03-22 09:55:34,386 INFO org.apache.hadoop.http.HttpServer: Jetty bound
> to port 50075
> 2010-03-22 09:55:34,387 INFO org.mortbay.log: jetty-6.1.14
> 2010-03-22 09:57:00,445 INFO org.mortbay.log: Started
> SelectChannelConnector@0.0.0.0:50075
> 2010-03-22 09:57:00,480 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=DataNode, sessionId=null
> 2010-03-22 09:57:15,652 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> Initializing RPC Metrics with hostName=DataNode, port=50020
> 2010-03-22 09:57:15,676 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
> 2010-03-22 09:57:15,713 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 50020: starting
> 2010-03-22 09:57:15,738 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 0 on 50020: starting
> 2010-03-22 09:57:15,766 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 1 on 50020: starting
> 2010-03-22 09:57:15,836 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration =
> DatanodeRegistration(localhost:50010, storageID=, infoPort=50075,
> ipcPort=50020)
> 2010-03-22 09:57:15,880 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 2 on 50020: starting
> 2010-03-22 09:57:15,988 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: New storage id
> DS-3114065-127.0.0.1-50010-1269277035840 is assigned to data-node
> 127.0.0.1:50010
> 2010-03-22 09:57:16,022 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
> 127.0.0.1:50010, storageID=DS-3114065-127.0.0.1-50010-1269277035840,
> infoPort=50075, ipcPort=50020)In DataNode.run, data =
> FSDataset{dirpath='/home/training/hadoop/dfs/data/current'}
> 2010-03-22 09:57:16,032 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: using BLOCKREPORT_INTERVAL
> of 3600000msec Initial delay: 0msec
> 2010-03-22 09:57:16,243 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 0 blocks got
> processed in 16 msecs
> 2010-03-22 09:57:16,258 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Starting Periodic block
> scanner.
> 2010-03-22 10:34:44,492 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 0 blocks got
> processed in 3 msecs
>
>
>
>
> On Mon, Mar 22, 2010 at 10:33 AM, Jay Booth <jaybooth@gmail.com> wrote:
>
>> Usually when I see this in pseudo-distributed, it means the datanode isn't
>> up.  Check logs?
>>
>> On Mon, Mar 22, 2010 at 1:11 PM, Something Something <
>> mailinglists19@gmail.com> wrote:
>>
>> > After upgrading to hadoop-0.20.2, I started seeing these messages in the
>> > log:
>> >
>> >
>> > 2010-03-22 09:56:28,393 INFO org.apache.hadoop.ipc.Server: IPC Server
>> > handler 1 on 9000, call addBlock(/home/training/hadoop/mapred/system/
>> > jobtracker.info, DFSClient_-1956918169) from 127.0.0.1:41067: error:
>> > java.io.IOException: File /home/training/hadoop/mapred/system/
>> > jobtracker.info could only be replicated to 0 nodes, instead of 1
>> > java.io.IOException: File /home/training/hadoop/mapred/system/
>> > jobtracker.info could only be replicated to 0 nodes, instead of 1
>> >        at
>> >
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
>> >        at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>> >        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >        at
>> >
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >        at
>> >
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >        at java.lang.reflect.Method.invoke(Method.java:597)
>> >        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>> >        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>> >        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>> >        at java.security.AccessController.doPrivileged(Native Method)
>> >        at javax.security.auth.Subject.doAs(Subject.java:396)
>> >        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>> >
>> >
>> > I am running in Psuedo-distributed mode.  Before upgrading I used to see
>> > these messages occasionally, but they were harmless.  Now I see these
>> when
>> > I
>> > try to copy files to DFS using -copyFromLocal, and the files no longer
>> get
>> > copied :(
>> >
>> > I mean, it's quite possible, this has nothing to do with the Hadoop
>> upgrade
>> > but something changed in my environment, but any help will be
>> appreciated.
>> >  Thanks.
>> >
>>
>
>

--0016e6d7e0a9b7c043048269f3ce--

From general-return-1238-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 23 19:19:54 2010
Return-Path: <general-return-1238-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 30415 invoked from network); 23 Mar 2010 19:19:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Mar 2010 19:19:53 -0000
Received: (qmail 6459 invoked by uid 500); 23 Mar 2010 19:19:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6397 invoked by uid 500); 23 Mar 2010 19:19:52 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 6389 invoked by uid 99); 23 Mar 2010 19:19:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Mar 2010 19:19:52 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.191.91.166] (HELO web37904.mail.mud.yahoo.com) (209.191.91.166)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 23 Mar 2010 19:19:44 +0000
Received: (qmail 69771 invoked by uid 60001); 23 Mar 2010 19:19:23 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1269371963; bh=uhAEk15QSAAvZsZj5jO2y9IujJm7035trmRafi+7jBI=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=ayeDIPkL1mNcYThcpyYUMdo7oAo4CKUHJhibTHmtDKjN5Z2IOK0AzCTX46VmFqSYijtyoX1LgUvQxnpEGN11yKk5yMGjf76aq57MhCnbZia6LclXBE2s1/arfKKgXpZ6zZUJx4uZu2mckz4EDJ4TqWGBaz3P/hA1phjX29kwN6U=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=xJkrLoBEcVX6xlA0Gi3g4qhTtejodmWPEbVpUURv9YBdfW6GrH8wVQvpKc+McsHGj0DWpr0mQ8kfTlkz7Ivw0vpEduqfALSMEfC2d5pPkS7ASQcA50L7iO900pRvczA+A3icz552o4+1pSD2XcwseXG4/hSxFm6+vuk72mwsr1M=;
Message-ID: <169646.69593.qm@web37904.mail.mud.yahoo.com>
X-YMail-OSG: iBvx_z4VM1nPe16uCpNMMPn42YpQuRASSvVxjUkH39ttoTK
 q1OdyfUtk7f3Y4ZHGnZkPWgL2IPAf34ktzDtAjhyZ7VG7gQaXCe9czIPqZVJ
 MpaTDlzZoO3IASZGtglxyjAvQsv7HvxDxyJDs5e0GCG_w32_0MUXbZiLIPPm
 Rc0KAaKub5txpfsxTpMiwTFX28eFAUbqCb78SugoVNd7_VKgr99OgDV9ON7g
 DURKQwvSCTbwM9bM6f8OTjSMVEGHooJLRVx3CpIKuu9w9IISIaV53PdJgXSG
 BWppFmwFgQdqFFXJ_KfVGuf3C7fTX5jlQEuxKR7Gs1zqhBj.UOgkcENVDFrI
 jkOyIwlk2e3IxDBXIrevOkr1phMhGwNA-
Received: from [63.147.59.14] by web37904.mail.mud.yahoo.com via HTTP; Tue, 23 Mar 2010 12:19:23 PDT
X-Mailer: YahooMailClassic/10.0.8 YahooMailWebService/0.8.100.260964
Date: Tue, 23 Mar 2010 12:19:23 -0700 (PDT)
From: Gary Yang <garyyang6@yahoo.com>
Subject: bin/hadoop namenode -format          IOException: Invalid argument
To: general@hadoop.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii

Hi All,

I got exception below. I followed the Quick Start for the Pseudo-Distributed Operation. I did what the quick start says. Can someone help?

cat conf/core-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

  <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
  </property>

</configuration>


cat conf/hdfs-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>


cat conf/mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <property>
    <name>mapred.job.tracker</name>
    <value>localhost:9001</value>
  </property> 
</configuration>



Below are errors:


-bash-3.2# bin/hadoop namenode -format
10/03/23 11:54:54 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = Board2/192.168.0.185
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 0.20.2
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
10/03/23 11:54:56 INFO namenode.FSNamesystem: fsOwner=root,root,bin,daemon,sys,adm,disk,wheel
10/03/23 11:54:56 INFO namenode.FSNamesystem: supergroup=supergroup
10/03/23 11:54:56 INFO namenode.FSNamesystem: isPermissionEnabled=true
10/03/23 11:54:56 INFO common.Storage: java.io.IOException: Invalid argument
        at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:900)
        at java.nio.channels.FileChannel.tryLock(FileChannel.java:974)
        at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:527)
        at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1087)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1110)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:856)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:948)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)

10/03/23 11:54:56 ERROR namenode.NameNode: java.io.IOException: Invalid argument
        at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:900)
        at java.nio.channels.FileChannel.tryLock(FileChannel.java:974)
        at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:527)
        at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1087)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1110)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:856)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:948)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)

10/03/23 11:54:56 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Board2/192.168.0.185
************************************************************/




Thanks,


Gary





      

From general-return-1239-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 23 19:28:41 2010
Return-Path: <general-return-1239-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 58985 invoked from network); 23 Mar 2010 19:28:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Mar 2010 19:28:40 -0000
Received: (qmail 22656 invoked by uid 500); 23 Mar 2010 19:28:40 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 22605 invoked by uid 500); 23 Mar 2010 19:28:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 22596 invoked by uid 99); 23 Mar 2010 19:28:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Mar 2010 19:28:39 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Mar 2010 19:28:31 +0000
Received: from sp1-ex07cas01.ds.corp.yahoo.com (sp1-ex07cas01.ds.corp.yahoo.com [216.252.116.137])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o2NJRokY024185
	for <general@hadoop.apache.org>; Tue, 23 Mar 2010 12:27:50 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:in-reply-to:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=NJBCdbANdAMasvfi3c/TZWNEKCn9gbWV1ibcYmFs5hCoZMn4LOY1TejcMedfNntn
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas01.ds.corp.yahoo.com ([216.252.116.137]) with mapi; Tue, 23 Mar
 2010 12:27:50 -0700
From: Suresh Srinivas <sureshms@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 23 Mar 2010 12:27:49 -0700
Subject: Re: bin/hadoop namenode -format          IOException: Invalid
 argument
Thread-Topic: bin/hadoop namenode -format          IOException: Invalid
 argument
Thread-Index: AcrKvgDea/jHWfWmSGmnbpuw7TcPUAAAOrOb
Message-ID: <C7CE6444.2067F%sureshms@yahoo-inc.com>
In-Reply-To: <169646.69593.qm@web37904.mail.mud.yahoo.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C7CE64442067Fsureshmsyahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C7CE64442067Fsureshmsyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Is namenode running when you tried this operation? If so shutdown the namen=
ode process and try it again.


On 3/23/10 11:19 AM, "Gary Yang" <garyyang6@yahoo.com> wrote:

Hi All,

I got exception below. I followed the Quick Start for the Pseudo-Distribute=
d Operation. I did what the quick start says. Can someone help?

cat conf/core-site.xml
<?xml version=3D"1.0"?>
<?xml-stylesheet type=3D"text/xsl" href=3D"configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

  <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
  </property>

</configuration>


cat conf/hdfs-site.xml
<?xml version=3D"1.0"?>
<?xml-stylesheet type=3D"text/xsl" href=3D"configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>


cat conf/mapred-site.xml
<?xml version=3D"1.0"?>
<?xml-stylesheet type=3D"text/xsl" href=3D"configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <property>
    <name>mapred.job.tracker</name>
    <value>localhost:9001</value>
  </property>
</configuration>



Below are errors:


-bash-3.2# bin/hadoop namenode -format
10/03/23 11:54:54 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host =3D Board2/192.168.0.185
STARTUP_MSG:   args =3D [-format]
STARTUP_MSG:   version =3D 0.20.2
STARTUP_MSG:   build =3D https://svn.apache.org/repos/asf/hadoop/common/bra=
nches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 U=
TC 2010
************************************************************/
10/03/23 11:54:56 INFO namenode.FSNamesystem: fsOwner=3Droot,root,bin,daemo=
n,sys,adm,disk,wheel
10/03/23 11:54:56 INFO namenode.FSNamesystem: supergroup=3Dsupergroup
10/03/23 11:54:56 INFO namenode.FSNamesystem: isPermissionEnabled=3Dtrue
10/03/23 11:54:56 INFO common.Storage: java.io.IOException: Invalid argumen=
t
        at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:900)
        at java.nio.channels.FileChannel.tryLock(FileChannel.java:974)
        at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tr=
yLock(Storage.java:527)
        at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lo=
ck(Storage.java:505)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.ja=
va:1087)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.ja=
va:1110)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.=
java:856)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(N=
ameNode.java:948)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.ja=
va:965)

10/03/23 11:54:56 ERROR namenode.NameNode: java.io.IOException: Invalid arg=
ument
        at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:900)
        at java.nio.channels.FileChannel.tryLock(FileChannel.java:974)
        at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tr=
yLock(Storage.java:527)
        at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lo=
ck(Storage.java:505)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.ja=
va:1087)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.ja=
va:1110)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.=
java:856)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(N=
ameNode.java:948)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.ja=
va:965)

10/03/23 11:54:56 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Board2/192.168.0.185
************************************************************/




Thanks,


Gary








--_000_C7CE64442067Fsureshmsyahooinccom_--

From general-return-1240-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 23 19:41:27 2010
Return-Path: <general-return-1240-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 64393 invoked from network); 23 Mar 2010 19:41:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Mar 2010 19:41:27 -0000
Received: (qmail 40403 invoked by uid 500); 23 Mar 2010 19:41:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 40363 invoked by uid 500); 23 Mar 2010 19:41:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 40355 invoked by uid 99); 23 Mar 2010 19:41:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Mar 2010 19:41:26 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [209.191.91.167] (HELO web37905.mail.mud.yahoo.com) (209.191.91.167)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 23 Mar 2010 19:41:20 +0000
Received: (qmail 59209 invoked by uid 60001); 23 Mar 2010 19:40:58 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1269373258; bh=VRVz8i2WiOvFq1nfArXndSG6opnk3TM/bw8aD9xqsn8=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=y2EpSjxnWtHRSBMrSzk8HwzQysJqCjvxPr4LIZWCvqgbz12R3JHCfwJ/AZrRERtMdGAFh+f7lHH4tRYtllWodZlLo3eYUpE15w0LtWkfZ2OsNUd9uKwPlqVDXNzLBarNzSUiguvB5tDl7N4ujxnjowdSAPS6ALzgAqWZjV/DUE0=
DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding;
  b=GEGKTpwAq5mZxEHnEKYTOhlhHgMS3SubHjUsacQDAgLUlvpR0XEYKr0Oeaia3ENIQu08pOz/td5I6GP4UBsIAabymUWON+IBcFreewQU5L46cpfdyX5DFPZ9JqVMV6RWAtuU5VOfN5WsoYdTTJ2gQJ7xKZOVnkPQyPc9HHwftDs=;
Message-ID: <424193.53878.qm@web37905.mail.mud.yahoo.com>
X-YMail-OSG: hV5rU_YVM1l9glrHaOvY2ncvstreZybMawQkCEJ6l2cJuZF
 3QySnzmP1_D3W7Tghe4XTc5gq3XATipYpuwpXyUsCf.9E4MHjHqC1XS2wwgW
 LfoyM0YvwATcnx5lAtn5m1OpB5Ob5tq_WUIIbHLA7DyXx48yJ44NW5nl_Tz7
 0DQ2bPiGeCNIGAj2aYPnD2UnNEYcxpHpbm5tc7briaViNba2jqrtnmBPky91
 aP6lvX_KoIbIwrmSECcN_XTkcXJ6hhZk.NU3Kl2F2JA.zdBvTbXyDI_rGBqq
 xdFuywFaIpOkXEw71So7eSmbZiAqqPId1fpQOFfZoCQLFd98cUUzo_8sRWNa
 TVB9O6jfkNE.6Ya_dMHM7pvNErFHHlbU-
Received: from [63.147.59.14] by web37905.mail.mud.yahoo.com via HTTP; Tue, 23 Mar 2010 12:40:58 PDT
X-Mailer: YahooMailClassic/10.0.8 YahooMailWebService/0.8.100.260964
Date: Tue, 23 Mar 2010 12:40:58 -0700 (PDT)
From: Gary Yang <garyyang6@yahoo.com>
Subject: Re: bin/hadoop namenode -format          IOException: Invalid argument
To: general@hadoop.apache.org
In-Reply-To: <C7CE6444.2067F%sureshms@yahoo-inc.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

No. The namenode is not running. "bin/hadoop namenode -format" was the very=
 first command. I have not got chance to start the namenode yet. Any idea?

--- On Tue, 3/23/10, Suresh Srinivas <sureshms@yahoo-inc.com> wrote:

> From: Suresh Srinivas <sureshms@yahoo-inc.com>
> Subject: Re: bin/hadoop namenode -format          IOException: Invalid ar=
gument
> To: "general@hadoop.apache.org" <general@hadoop.apache.org>
> Date: Tuesday, March 23, 2010, 12:27 PM
> Is namenode running when you tried
> this operation? If so shutdown the namenode process and try
> it again.
>=20
>=20
> On 3/23/10 11:19 AM, "Gary Yang" <garyyang6@yahoo.com>
> wrote:
>=20
> Hi All,
>=20
> I got exception below. I followed the Quick Start for the
> Pseudo-Distributed Operation. I did what the quick start
> says. Can someone help?
>=20
> cat conf/core-site.xml
> <?xml version=3D"1.0"?>
> <?xml-stylesheet type=3D"text/xsl"
> href=3D"configuration.xsl"?>
>=20
> <!-- Put site-specific property overrides in this file.
> -->
>=20
> <configuration>
>=20
> =A0 <property>
> =A0 =A0 <name>fs.default.name</name>
> =A0 =A0
> <value>hdfs://localhost:9000</value>
> =A0 </property>
>=20
> </configuration>
>=20
>=20
> cat conf/hdfs-site.xml
> <?xml version=3D"1.0"?>
> <?xml-stylesheet type=3D"text/xsl"
> href=3D"configuration.xsl"?>
>=20
> <!-- Put site-specific property overrides in this file.
> -->
>=20
> <configuration>
> =A0 <property>
> =A0 =A0 <name>dfs.replication</name>
> =A0 =A0 <value>1</value>
> =A0 </property>
> </configuration>
>=20
>=20
> cat conf/mapred-site.xml
> <?xml version=3D"1.0"?>
> <?xml-stylesheet type=3D"text/xsl"
> href=3D"configuration.xsl"?>
>=20
> <!-- Put site-specific property overrides in this file.
> -->
>=20
> <configuration>
> =A0 <property>
> =A0 =A0 <name>mapred.job.tracker</name>
> =A0 =A0 <value>localhost:9001</value>
> =A0 </property>
> </configuration>
>=20
>=20
>=20
> Below are errors:
>=20
>=20
> -bash-3.2# bin/hadoop namenode -format
> 10/03/23 11:54:54 INFO namenode.NameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG:=A0=A0=A0host =3D Board2/192.168.0.185
> STARTUP_MSG:=A0=A0=A0args =3D [-format]
> STARTUP_MSG:=A0=A0=A0version =3D 0.20.2
> STARTUP_MSG:=A0=A0=A0build =3D https://svn.apache.org/repos/asf/hadoop/co=
mmon/branches/branch-0.20
> -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC
> 2010
> ************************************************************/
> 10/03/23 11:54:56 INFO namenode.FSNamesystem:
> fsOwner=3Droot,root,bin,daemon,sys,adm,disk,wheel
> 10/03/23 11:54:56 INFO namenode.FSNamesystem:
> supergroup=3Dsupergroup
> 10/03/23 11:54:56 INFO namenode.FSNamesystem:
> isPermissionEnabled=3Dtrue
> 10/03/23 11:54:56 INFO common.Storage: java.io.IOException:
> Invalid argument
> =A0 =A0 =A0 =A0 at
> sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:900)
> =A0 =A0 =A0 =A0 at
> java.nio.channels.FileChannel.tryLock(FileChannel.java:974)
> =A0 =A0 =A0 =A0 at
> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Sto=
rage.java:527)
> =A0 =A0 =A0 =A0 at
> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storag=
e.java:505)
> =A0 =A0 =A0 =A0 at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1087)
> =A0 =A0 =A0 =A0 at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1110)
> =A0 =A0 =A0 =A0 at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:856)
> =A0 =A0 =A0 =A0 at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.j=
ava:948)
> =A0 =A0 =A0 =A0 at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
>=20
> 10/03/23 11:54:56 ERROR namenode.NameNode:
> java.io.IOException: Invalid argument
> =A0 =A0 =A0 =A0 at
> sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:900)
> =A0 =A0 =A0 =A0 at
> java.nio.channels.FileChannel.tryLock(FileChannel.java:974)
> =A0 =A0 =A0 =A0 at
> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Sto=
rage.java:527)
> =A0 =A0 =A0 =A0 at
> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storag=
e.java:505)
> =A0 =A0 =A0 =A0 at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1087)
> =A0 =A0 =A0 =A0 at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1110)
> =A0 =A0 =A0 =A0 at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:856)
> =A0 =A0 =A0 =A0 at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.j=
ava:948)
> =A0 =A0 =A0 =A0 at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
>=20
> 10/03/23 11:54:56 INFO namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at
> Board2/192.168.0.185
> ************************************************************/
>=20
>=20
>=20
>=20
> Thanks,
>=20
>=20
> Gary
>=20
>=20
>=20
>=20
>=20
>=20
>=20
> =0A=0A=0A      

From general-return-1241-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 24 10:17:51 2010
Return-Path: <general-return-1241-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 91861 invoked from network); 24 Mar 2010 10:17:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Mar 2010 10:17:51 -0000
Received: (qmail 52458 invoked by uid 500); 24 Mar 2010 10:17:50 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 52309 invoked by uid 500); 24 Mar 2010 10:17:50 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 52301 invoked by uid 99); 24 Mar 2010 10:17:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Mar 2010 10:17:49 +0000
X-ASF-Spam-Status: No, hits=-2.9 required=10.0
	tests=AWL,RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [192.6.10.60] (HELO tobor.hpl.hp.com) (192.6.10.60)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Mar 2010 10:17:40 +0000
Received: from localhost (localhost [127.0.0.1])
	by tobor.hpl.hp.com (Postfix) with ESMTP id 62DF0B7E49
	for <general@hadoop.apache.org>; Wed, 24 Mar 2010 10:17:18 +0000 (GMT)
X-Virus-Scanned: amavisd-new at hplb.hpl.hp.com
Received: from tobor.hpl.hp.com ([127.0.0.1])
	by localhost (tobor.hpl.hp.com [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id xA0wQuynL5GC for <general@hadoop.apache.org>;
	Wed, 24 Mar 2010 10:17:17 +0000 (GMT)
Received: from 0-imap-br1.hpl.hp.com (0-imap-br1.hpl.hp.com [16.25.144.60])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by tobor.hpl.hp.com (Postfix) with ESMTPS id A3B1EB7E0B
	for <general@hadoop.apache.org>; Wed, 24 Mar 2010 10:17:17 +0000 (GMT)
MailScanner-NULL-Check: 1270030625.41948@4z9RTqmw+yCmC+vetAZ9fQ
Received: from [16.25.175.158] (morzine.hpl.hp.com [16.25.175.158])
	by 0-imap-br1.hpl.hp.com (8.14.1/8.13.4) with ESMTP id o2OAH3oN007750
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <general@hadoop.apache.org>; Wed, 24 Mar 2010 10:17:05 GMT
Message-ID: <4BA9E6AA.0@apache.org>
Date: Wed, 24 Mar 2010 10:17:14 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100228)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: bin/hadoop namenode -format          IOException: Invalid argument
References: <424193.53878.qm@web37905.mail.mud.yahoo.com>
In-Reply-To: <424193.53878.qm@web37905.mail.mud.yahoo.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-HPL-MailScanner-Information: Please contact the ISP for more information
X-MailScanner-ID: o2OAH3oN007750
X-HPL-MailScanner: Found to be clean
X-HPL-MailScanner-From: stevel@apache.org

Gary Yang wrote:
> No. The namenode is not running. "bin/hadoop namenode -format" was the very first command. I have not got chance to start the namenode yet. Any idea?

>>
>> 10/03/23 11:54:56 ERROR namenode.NameNode:
>> java.io.IOException: Invalid argument
>>         at
>> sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:900)
>>         at
>> java.nio.channels.FileChannel.tryLock(FileChannel.java:974)
>>         at
>> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:527)
>>         at
>> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1087)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1110)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:856)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:948)
>>         at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)


That could be the filesystem being unhappy about some directory

Check all your namenode dir settings, make sure they are valid paths, 
try to create them as the hadoop user

From general-return-1242-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 24 16:01:25 2010
Return-Path: <general-return-1242-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68765 invoked from network); 24 Mar 2010 16:01:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Mar 2010 16:01:24 -0000
Received: (qmail 3251 invoked by uid 500); 24 Mar 2010 16:01:23 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 3050 invoked by uid 500); 24 Mar 2010 16:01:23 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 57955 invoked by uid 99); 23 Mar 2010 13:26:46 -0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of madhusona@gmail.com designates 74.125.82.48 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:message-id:subject
         :from:to:content-type;
        bh=tSDqHOo4E+Fpq/JsU+G+ZYc8aMsA6EABP3kNwHetgIk=;
        b=DSDqdZTws46zdzhW4qMwDgNK70SrvWThpxiAtNSKHgntKDChnVu0NQNNqeO0SgbB4P
         Ip+SF0Ik8vGODo20plRd/WrKNUGTx0gildUjcqqT2SY/jQY8t0eZhfzUzmz+3WHiPd65
         1rkX9R5vpaiJEbeoNFBRNot5M76Br5kzc5vj8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=sSMjB6ubYdehf/xRQaJ2hzyTWv4L0LehuT6qIrEI5w3t5TQPBu5Zy224ONkeMk6lXw
         FXl1gJcBVXmnsNWAlvRSjQyyURZT3uHof5HF4TJncH0oO5jm48BkNDxg4l+Hv05Job8u
         Z/lhKA952Tv2yRBStN26cp+Dmt2nSWtcY3CTk=
MIME-Version: 1.0
Date: Tue, 23 Mar 2010 14:26:17 +0100
Message-ID: <975b93dd1003230626y77808d60xdcb26c5682f82a7@mail.gmail.com>
Subject: Difference
From: madhusona@gmail.com
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e649849a2bb07b048277c247
X-Virus-Checked: Checked by ClamAV on apache.org

--0016e649849a2bb07b048277c247
Content-Type: text/plain; charset=ISO-8859-1

Dear Users

I want to know the difference between Hadoop and clusters(Torque,SGE)
-- 
with regards,
B.Madusudhanan,
Research Fellow.

--0016e649849a2bb07b048277c247--

From general-return-1243-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 24 16:05:35 2010
Return-Path: <general-return-1243-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 70575 invoked from network); 24 Mar 2010 16:05:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Mar 2010 16:05:35 -0000
Received: (qmail 14413 invoked by uid 500); 24 Mar 2010 16:05:34 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14338 invoked by uid 500); 24 Mar 2010 16:05:34 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 14330 invoked by uid 99); 24 Mar 2010 16:05:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Mar 2010 16:05:34 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [67.59.59.114] (HELO trironport1.altair.com) (67.59.59.114)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Mar 2010 16:05:26 +0000
X-IronPort-AV: E=Sophos;i="4.51,301,1267419600"; 
   d="scan'208";a="10563234"
Received: from unknown (HELO TR-EXCH07.prog.altair.com) ([204.235.24.175])
  by trironport1.altair.com with ESMTP; 24 Mar 2010 12:04:54 -0400
Received: from TR-EXCH07.prog.altair.com ([204.235.24.175]) by
 TR-EXCH07.prog.altair.com ([204.235.24.175]) with mapi; Wed, 24 Mar 2010
 12:05:16 -0400
From: Lori Ann Martin <lmartin@altair.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Wed, 24 Mar 2010 12:04:19 -0400
Subject: RE: Difference
Thread-Topic: Difference
Thread-Index: AcrLa1VI7fYwZKSnSaWOA8a/Lowl6gAAFPaI
Message-ID: <7232A3A7C53F634B8E91E65BC7833C5112D59A2BD8@TR-EXCH07.prog.altair.com>
References: <975b93dd1003230626y77808d60xdcb26c5682f82a7@mail.gmail.com>
In-Reply-To: <975b93dd1003230626y77808d60xdcb26c5682f82a7@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

I would suggest taking a look at PBS Works by Altair.  www.pbsworks.com.  Y=
ou can go to the website
and download a temp key.  What will you use the WLM for specifically?

________________________________________
From: madhusona@gmail.com [madhusona@gmail.com]
Sent: Tuesday, March 23, 2010 6:26 AM
To: general@hadoop.apache.org
Subject: Difference

Dear Users

I want to know the difference between Hadoop and clusters(Torque,SGE)
--
with regards,
B.Madusudhanan,
Research Fellow.=

From general-return-1244-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 24 18:14:04 2010
Return-Path: <general-return-1244-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 33417 invoked from network); 24 Mar 2010 18:14:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Mar 2010 18:14:04 -0000
Received: (qmail 18156 invoked by uid 500); 24 Mar 2010 18:14:03 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18115 invoked by uid 500); 24 Mar 2010 18:14:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18107 invoked by uid 99); 24 Mar 2010 18:14:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Mar 2010 18:14:03 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Mar 2010 18:13:54 +0000
Received: from [127.0.0.1] ([10.72.185.127])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o2OID5Qk058868
	for <general@hadoop.apache.org>; Wed, 24 Mar 2010 11:13:10 -0700 (PDT)
Message-ID: <4BAA5631.5090202@yahoo-inc.com>
Date: Wed, 24 Mar 2010 11:13:05 -0700
From: Konstantin Shvachko <shv@yahoo-inc.com>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: bin/hadoop namenode -format          IOException: Invalid argument
References: <424193.53878.qm@web37905.mail.mud.yahoo.com> <4BA9E6AA.0@apache.org>
In-Reply-To: <4BA9E6AA.0@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

This may be a problem with the underlying local file system.
Some file systems just don't support locks. Some NFS, e.g.
Some may have buggy native java implementation.

Are your name-node directories in /tmp, which is the default?
/tmp can behave strangely.
You should set "dfs.name.dir" pointing to a local HD directory in hdfs-site.xml.

--Konstantin

On 3/24/2010 3:17 AM, Steve Loughran wrote:
> Gary Yang wrote:
>> No. The namenode is not running. "bin/hadoop namenode -format" was the
>> very first command. I have not got chance to start the namenode yet.
>> Any idea?
>
>>>
>>> 10/03/23 11:54:56 ERROR namenode.NameNode:
>>> java.io.IOException: Invalid argument
>>>         at
>>> sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:900)
>>>         at
>>> java.nio.channels.FileChannel.tryLock(FileChannel.java:974)
>>>         at
>>> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:527)
>>>
>>>         at
>>> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
>>>
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1087)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:1110)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:856)
>>>
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:948)
>>>
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
>
>
> That could be the filesystem being unhappy about some directory
>
> Check all your namenode dir settings, make sure they are valid paths,
> try to create them as the hadoop user


From general-return-1245-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 25 03:59:45 2010
Return-Path: <general-return-1245-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 18740 invoked from network); 25 Mar 2010 03:59:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Mar 2010 03:59:45 -0000
Received: (qmail 13857 invoked by uid 500); 25 Mar 2010 03:59:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 13720 invoked by uid 500); 25 Mar 2010 03:59:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13705 invoked by uid 99); 25 Mar 2010 03:59:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Mar 2010 03:59:43 +0000
X-ASF-Spam-Status: No, hits=3.3 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Mar 2010 03:59:35 +0000
Received: from sp1-ex07cas01.ds.corp.yahoo.com (sp1-ex07cas01.ds.corp.yahoo.com [216.252.116.137])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o2P3wMCa050612;
	Wed, 24 Mar 2010 20:58:22 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:accept-language:content-language:x-ms-has-attach:
	x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=O+qbQhJoL7VnwVzQ1521hHjD+BpdmQEzfsUeYLMx9kVGAVxmF1E943BmX/c2RK/+
Received: from SP1-EX07VS01.ds.corp.yahoo.com ([216.252.116.139]) by
 sp1-ex07cas01.ds.corp.yahoo.com ([216.252.116.137]) with mapi; Wed, 24 Mar
 2010 20:58:22 -0700
From: Dekel Tankel <dekel@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>,
        "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Date: Wed, 24 Mar 2010 20:58:21 -0700
Subject: Hadoop Summit 2010 - June 29th, Santa Clara - Registration and call
 of papers is now open!
Thread-Topic: Hadoop Summit 2010 - June 29th, Santa Clara - Registration and
 call of papers is now open!
Thread-Index: AcrLz2jgF/z5hhfGSFy7H/+BpV0+EA==
Message-ID: <46E2672895E8744A97D7577A6110858B4FE03CC34F@SP1-EX07VS01.ds.corp.yahoo.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_46E2672895E8744A97D7577A6110858B4FE03CC34FSP1EX07VS01ds_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_46E2672895E8744A97D7577A6110858B4FE03CC34FSP1EX07VS01ds_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi All

Yahoo! is proud to host the 3rd Annual Hadoop Summit - June 29th, Santa Cla=
ra, California

Registration and call for papers are now open at  - http://www.hadoopsummit=
.org

Attend keynotes by technology leaders from Yahoo!, Amazon Web Services, Clo=
udera and Facebook, to hear exciting announcements on new capabilities and =
applications using Hadoop.

We are introducing three track sessions:

*Developers - Presentations and discussions about Hadoop programming and be=
st practices.

*Applications - Case studies about exciting new deployments and innovative =
applications.

*Research - Discussions on cutting-edge research based on Hadoop technology=
.

Get an opportunity to share your experience with hundreds of Hadoop users b=
y submitting a session abstract  for one of the tracks above. Submission de=
adline - April 30th.

Register early, space is limited!
Looking forward to see you at the summit.

                Dekel

--_000_46E2672895E8744A97D7577A6110858B4FE03CC34FSP1EX07VS01ds_--

From general-return-1247-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 25 14:01:36 2010
Return-Path: <general-return-1247-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 32107 invoked from network); 25 Mar 2010 14:01:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Mar 2010 14:01:36 -0000
Received: (qmail 15357 invoked by uid 500); 25 Mar 2010 12:12:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 15036 invoked by uid 500); 25 Mar 2010 12:12:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 15012 invoked by uid 99); 25 Mar 2010 12:12:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Mar 2010 12:12:12 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [88.198.2.104] (HELO koch.ro) (88.198.2.104)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Mar 2010 12:12:05 +0000
Received: from 84-72-85-88.dclient.hispeed.ch ([84.72.85.88] helo=jona.localnet)
	by koch.ro with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
	(Exim 4.69)
	(envelope-from <thomas@koch.ro>)
	id 1NulrA-0006Zl-Om; Thu, 25 Mar 2010 13:08:12 +0100
From: Thomas Koch <thomas@koch.ro>
Reply-To: thomas@koch.ro
To: general@hadoop.apache.org,
 hbase-user@hadoop.apache.org,
 common-user@hadoop.apache.org
Subject: [ANN] Hadoop 0.20.2 in Debian unstable
Date: Thu, 25 Mar 2010 13:11:40 +0100
User-Agent: KMail/1.12.4 (Linux/2.6.32-2-amd64; KDE/4.3.4; x86_64; ; )
MIME-Version: 1.0
Content-Type: text/plain;
  charset="us-ascii"
Content-Transfer-Encoding: 7bit
Message-Id: <201003251311.40203.thomas@koch.ro>
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

hadoop has entered Debian unstable a few days ago:
http://packages.qa.debian.org/h/hadoop.html

Please use it, test it, spread the word about it! Having Hadoop in a major 
linux distribution can significantly improve the visibility and lower the 
entry barrier.

There are still some TODO items to improve the packaging:

- build more contribs
- build the C++ nativ libraries
- build the C bindings
- build KFS and Amazon Filesystem bindings

More informations on the Debian Wiki page for Hadoop:
http://wiki.debian.org/Hadoop

Best regards,

Thomas Koch, http://www.koch.ro

From general-return-1248-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 25 15:01:25 2010
Return-Path: <general-return-1248-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 92585 invoked from network); 25 Mar 2010 15:01:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Mar 2010 15:01:25 -0000
Received: (qmail 85695 invoked by uid 500); 25 Mar 2010 15:01:24 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 85631 invoked by uid 500); 25 Mar 2010 15:01:24 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 85623 invoked by uid 99); 25 Mar 2010 15:01:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Mar 2010 15:01:24 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [156.148.72.33] (HELO raffaello.crs4.it) (156.148.72.33)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Mar 2010 15:01:15 +0000
Received: from pflip (pflip.crs4.it [156.148.70.62])
	by raffaello.crs4.it (Postfix) with SMTP id 731556BBA0
	for <general@hadoop.apache.org>; Thu, 25 Mar 2010 15:59:12 +0100 (CET)
Received: by pflip (sSMTP sendmail emulation); Thu, 25 Mar 2010 16:00:53 +0100
Subject: SGE hod support wiki
From: Gianluigi Zanetti <gianluigi.zanetti@crs4.it>
Reply-To: gianluigi.zanetti@crs4.it
To: general@hadoop.apache.org
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Organization: CRS4
Date: Thu, 25 Mar 2010 16:00:53 +0100
Message-Id: <1269529253.18834.195.camel@pflip>
Mime-Version: 1.0
X-Mailer: Evolution 2.26.3 
X-Virus-Checked: Checked by ClamAV on apache.org

Greetings.
Some time ago we submitted a patch for HOD that adds Sun Grid Engine
integration (I believe it was HADOOP-6369). It appears that this patch
is now used in production in more than one site, and we start to have
the need to manage its documentation and bug fixes/extensions in an
'externally' visible way. 
The first idea would be to open a hod-sge specific project on, say,
sourceforge. Would this be considered proper form?

--gianluigi


From general-return-1246-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 25 15:14:34 2010
Return-Path: <general-return-1246-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 46936 invoked from network); 25 Mar 2010 15:14:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Mar 2010 15:14:34 -0000
Received: (qmail 44327 invoked by uid 500); 25 Mar 2010 09:43:12 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44103 invoked by uid 500); 25 Mar 2010 09:43:12 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44067 invoked by uid 99); 25 Mar 2010 09:43:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Mar 2010 09:43:11 +0000
X-ASF-Spam-Status: No, hits=-1.7 required=10.0
	tests=AWL,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: local policy)
Received: from [88.198.2.104] (HELO koch.ro) (88.198.2.104)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Mar 2010 09:43:06 +0000
Received: from 84-72-85-88.dclient.hispeed.ch ([84.72.85.88] helo=jona.localnet)
	by koch.ro with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
	(Exim 4.69)
	(envelope-from <thomas@koch.ro>)
	id 1NujWz-0004fQ-6f; Thu, 25 Mar 2010 10:39:13 +0100
From: Thomas Koch <thomas@koch.ro>
Reply-To: thomas@koch.ro
To: java-dev@lucene.apache.org,
 hbase-user@hadoop.apache.org,
 katta-developer@lists.sourceforge.net,
 solr-dev@lucene.apache.org,
 general@hadoop.apache.org
Subject: ported lucandra: lucene index on HBase
Date: Thu, 25 Mar 2010 10:42:17 +0100
User-Agent: KMail/1.12.4 (Linux/2.6.32-2-amd64; KDE/4.3.4; x86_64; ; )
MIME-Version: 1.0
Content-Type: text/plain;
  charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Message-Id: <201003251042.17623.thomas@koch.ro>

Hi,

Lucandra stores a lucene index on cassandra:
http://blog.sematext.com/2010/02/09/lucandra-a-cassandra-based-lucene-backe=
nd

As the author of lucandra writes: "I=E2=80=99m sure something similar could=
 be built=20
on hbase."

So here it is:
http://github.com/thkoch2001/lucehbase

This is only a first prototype which has not been tested on anything real y=
et.=20
But if you're interested, please join me to get it production ready!

I propose to keep this thread on hbase-user and java-dev only.
Would it make sense to aim this project to become an hbase contrib? Or a=20
lucene contrib?

Best regards,

Thomas Koch, http://www.koch.ro

From general-return-1249-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 25 17:10:51 2010
Return-Path: <general-return-1249-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 27827 invoked from network); 25 Mar 2010 17:10:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Mar 2010 17:10:50 -0000
Received: (qmail 14093 invoked by uid 500); 25 Mar 2010 17:10:48 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 14010 invoked by uid 500); 25 Mar 2010 17:10:48 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 13994 invoked by uid 99); 25 Mar 2010 17:10:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Mar 2010 17:10:48 +0000
X-ASF-Spam-Status: No, hits=-1.1 required=10.0
	tests=AWL,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of jdcryans@gmail.com designates 209.85.223.198 as permitted sender)
Received: from [209.85.223.198] (HELO mail-iw0-f198.google.com) (209.85.223.198)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Mar 2010 17:10:44 +0000
Received: by iwn36 with SMTP id 36so1136343iwn.29
        for <multiple recipients>; Thu, 25 Mar 2010 10:10:23 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:sender:received:in-reply-to
         :references:date:x-google-sender-auth:message-id:subject:from:to:cc
         :content-type:content-transfer-encoding;
        bh=PyIbQzt9uQQidjEEr1Eo/u1NwzCOR/5wGAQxAV8c5vI=;
        b=e+yNQD9feo6ftCwguygkTQ2ElJmHB5C8xDCLhGv3jJWni3ssqcrSVotfJiNasVTCag
         PNL4Zp5Wk7L2NR4o23V8pfAoMCwx0VezynXBwO6mChUve1goLRs3erijFRgqOZfy8tnh
         hokNGEUAuuxn33Pdti26y/+3hsa+GDD/Yik3o=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:cc:content-type
         :content-transfer-encoding;
        b=CdJStQrAzEhpqH9xTVFuVCHb/hAfDc1NYmwIa3SVI3Oagv4ytFaa4xU5UrYixvkIo/
         2gUJg8aT1R3CVjLxz6ZtZyozTxk5bxMby27qrenFtuvOgd6BMIOMt2vh2fJOraoBFfGl
         i00jYNIrtbN2LFe64FwHYKWh3+E3nXlXGGA4g=
MIME-Version: 1.0
Sender: jdcryans@gmail.com
Received: by 10.114.6.7 with SMTP id 7mr3765558waf.90.1269537022936; Thu, 25 
	Mar 2010 10:10:22 -0700 (PDT)
In-Reply-To: <201003251042.17623.thomas@koch.ro>
References: <201003251042.17623.thomas@koch.ro>
Date: Thu, 25 Mar 2010 10:10:22 -0700
X-Google-Sender-Auth: 51e6ca9125b87e29
Message-ID: <31a243e71003251010o53c804f2gd7b4318fd2eb0012@mail.gmail.com>
Subject: Re: ported lucandra: lucene index on HBase
From: Jean-Daniel Cryans <jdcryans@apache.org>
To: general@hadoop.apache.org, thomas@koch.ro
Cc: java-dev@lucene.apache.org, hbase-user@hadoop.apache.org, 
	katta-developer@lists.sourceforge.net, solr-dev@lucene.apache.org
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

That sounds great Thomas! You can start by adding an entry here
http://wiki.apache.org/hadoop/SupportingProjects

WRT becoming an HBase contrib, we have a rule that at least one
committer (or a very active contributor) must be in charge and be
available to fix anything broken in it due to changes in core HBase.
For example, if a contrib doesn't compile before a release, we will
exclude it.

J-D

On Thu, Mar 25, 2010 at 2:42 AM, Thomas Koch <thomas@koch.ro> wrote:
> Hi,
>
> Lucandra stores a lucene index on cassandra:
> http://blog.sematext.com/2010/02/09/lucandra-a-cassandra-based-lucene-bac=
kend
>
> As the author of lucandra writes: "I=92m sure something similar could be =
built
> on hbase."
>
> So here it is:
> http://github.com/thkoch2001/lucehbase
>
> This is only a first prototype which has not been tested on anything real=
 yet.
> But if you're interested, please join me to get it production ready!
>
> I propose to keep this thread on hbase-user and java-dev only.
> Would it make sense to aim this project to become an hbase contrib? Or a
> lucene contrib?
>
> Best regards,
>
> Thomas Koch, http://www.koch.ro
>

From general-return-1250-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 25 17:38:00 2010
Return-Path: <general-return-1250-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 53791 invoked from network); 25 Mar 2010 17:38:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Mar 2010 17:38:00 -0000
Received: (qmail 63253 invoked by uid 500); 25 Mar 2010 17:37:59 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 63209 invoked by uid 500); 25 Mar 2010 17:37:59 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 63201 invoked by uid 99); 25 Mar 2010 17:37:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Mar 2010 17:37:59 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Mar 2010 17:37:53 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:Received:Received:From:To:Subject:
   Thread-Topic:Thread-Index:Date:Message-ID:In-Reply-To:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:user-agent:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version:Return-Path:
   X-OriginalArrivalTime;
  b=BM/tZyb893klu0e2cpVJ4p/LVODvCUmQZ9zz7/Og7lqp2uRMKt9NI/VI
   r9czKP9m8cP4pbfE3/NBR0AoWpbPwURwky/VzJR9BIMYTyMrN0xezyOpf
   BAuO2Wjuf/5BwMY;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1269538673; x=1301074673;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20Re:=20SGE=20hod=20support=20wiki|Date:=20Thu,
   =2025=20Mar=202010=2017:37:31=20+0000|Message-ID:=20<C7D0
   ED6B.3C89%awittenauer@linkedin.com>|To:=20"general@hadoop
   .apache.org"=20<general@hadoop.apache.org>,=0D=0A=09"gian
   luigi.zanetti@crs4.it"=20<gianluigi.zanetti@crs4.it>
   |MIME-Version:=201.0|Content-Transfer-Encoding:=20quoted-
   printable|Content-ID:=20<60debeb4-9ad7-4863-9967-d6ceb715
   e112>|In-Reply-To:=20<1269529253.18834.195.camel@pflip>;
  bh=QT3cAItv3UhwbRwMDcR7gQxD+/lA/tEjWrM+CJqpLF8=;
  b=UYwuIXCuHVVOJRek8yWYF5x3fi5kuxOOFRQyFEJu/jg+LwAYgncRz1h2
   mJ74oMqI1RmM+1DzP6OiKlp+qbkjLJ9C288oZfO4bnugawD/pv2SXMXdm
   eR10vlhU64opHDL;
X-IronPort-AV: E=Sophos;i="4.51,308,1267430400"; 
   d="scan'208";a="11936562"
Received: from esv4-exctest.linkedin.biz ([172.18.46.60]) by CORP-MAIL.linkedin.biz with Microsoft SMTPSVC(6.0.3790.3959);
	 Thu, 25 Mar 2010 10:37:31 -0700
Received: from ESV4-CAS01.linkedin.biz (172.18.46.140) by
 esv4-exctest.linkedin.biz (172.18.46.60) with Microsoft SMTP Server (TLS) id
 14.0.682.1; Thu, 25 Mar 2010 10:37:31 -0700
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas01.linkedin.biz ([172.18.46.140]) with mapi; Thu, 25 Mar 2010
 10:37:31 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>,
	"gianluigi.zanetti@crs4.it" <gianluigi.zanetti@crs4.it>
Subject: Re: SGE hod support wiki
Thread-Topic: SGE hod support wiki
Thread-Index: AQHKzCwKxq0iSY8KikC4ErkuF4Vid5IC61iA
Date: Thu, 25 Mar 2010 17:37:31 +0000
Message-ID: <C7D0ED6B.3C89%awittenauer@linkedin.com>
In-Reply-To: <1269529253.18834.195.camel@pflip>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-Entourage/13.4.0.100208
Content-Type: text/plain; charset="us-ascii"
Content-ID: <60debeb4-9ad7-4863-9967-d6ceb715e112>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginalArrivalTime: 25 Mar 2010 17:37:31.0791 (UTC) FILETIME=[D8EE31F0:01CACC41]
X-Virus-Checked: Checked by ClamAV on apache.org




On 3/25/10 8:00 AM, "Gianluigi Zanetti" <gianluigi.zanetti@crs4.it> wrote:

> Greetings.
> Some time ago we submitted a patch for HOD that adds Sun Grid Engine
> integration (I believe it was HADOOP-6369). It appears that this patch
> is now used in production in more than one site, and we start to have
> the need to manage its documentation and bug fixes/extensions in an
> 'externally' visible way.
> The first idea would be to open a hod-sge specific project on, say,
> sourceforge. Would this be considered proper form?

Why not use the hadoop wiki and just use apache jira?  It would be better i=
f
hod was modified to handle more than torque in the mainline tree than have =
a
custom patch floating around.  Obviously, if there are legal reasons (see
the LZO stuff), then that's a different story.


From general-return-1251-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 25 18:17:20 2010
Return-Path: <general-return-1251-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 78927 invoked from network); 25 Mar 2010 18:17:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Mar 2010 18:17:20 -0000
Received: (qmail 47298 invoked by uid 500); 25 Mar 2010 18:17:19 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 47145 invoked by uid 500); 25 Mar 2010 18:17:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 47137 invoked by uid 99); 25 Mar 2010 18:17:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Mar 2010 18:17:19 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [156.148.72.33] (HELO raffaello.crs4.it) (156.148.72.33)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Mar 2010 18:17:11 +0000
Received: from pflip (pflip.crs4.it [156.148.70.62])
	by raffaello.crs4.it (Postfix) with SMTP id 59DA36BBA6
	for <general@hadoop.apache.org>; Thu, 25 Mar 2010 19:15:09 +0100 (CET)
Received: by pflip (sSMTP sendmail emulation); Thu, 25 Mar 2010 19:16:50 +0100
Subject: Re: SGE hod support wiki
From: Gianluigi Zanetti <gianluigi.zanetti@crs4.it>
Reply-To: gianluigi.zanetti@crs4.it
To: general@hadoop.apache.org
In-Reply-To: <C7D0ED6B.3C89%awittenauer@linkedin.com>
References: <C7D0ED6B.3C89%awittenauer@linkedin.com>
Content-Type: text/plain
Content-Transfer-Encoding: 7bit
Organization: CRS4
Date: Thu, 25 Mar 2010 19:16:50 +0100
Message-Id: <1269541010.18834.343.camel@pflip>
Mime-Version: 1.0
X-Mailer: Evolution 2.26.3 
X-Virus-Checked: Checked by ClamAV on apache.org

To tell the truth, we already have a patch sent in (HADOOP-6369).
On the other hand, silly of me, I did not consider the hadoop wiki
itself. 

--gianluigi


On Thu, 2010-03-25 at 17:37 +0000, Allen Wittenauer wrote:
> 
> 
> On 3/25/10 8:00 AM, "Gianluigi Zanetti" <gianluigi.zanetti@crs4.it> wrote:
> 
> > Greetings.
> > Some time ago we submitted a patch for HOD that adds Sun Grid Engine
> > integration (I believe it was HADOOP-6369). It appears that this patch
> > is now used in production in more than one site, and we start to have
> > the need to manage its documentation and bug fixes/extensions in an
> > 'externally' visible way.
> > The first idea would be to open a hod-sge specific project on, say,
> > sourceforge. Would this be considered proper form?
> 
> Why not use the hadoop wiki and just use apache jira?  It would be better if
> hod was modified to handle more than torque in the mainline tree than have a
> custom patch floating around.  Obviously, if there are legal reasons (see
> the LZO stuff), then that's a different story.
> 

From general-return-1252-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 26 23:55:46 2010
Return-Path: <general-return-1252-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17195 invoked from network); 26 Mar 2010 23:55:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Mar 2010 23:55:45 -0000
Received: (qmail 6425 invoked by uid 500); 26 Mar 2010 23:55:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 6016 invoked by uid 500); 26 Mar 2010 23:55:44 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5814 invoked by uid 99); 26 Mar 2010 23:55:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Mar 2010 23:55:44 +0000
X-ASF-Spam-Status: No, hits=-0.6 required=10.0
	tests=AWL,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.172] (HELO mrout2.yahoo.com) (216.145.54.172)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Mar 2010 23:55:38 +0000
Received: from [10.73.135.250] ([10.73.135.250])
	by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o2QNsrZq091353;
	Fri, 26 Mar 2010 16:54:53 -0700 (PDT)
Message-ID: <4BAD494E.9010900@apache.org>
Date: Fri, 26 Mar 2010 16:54:54 -0700
From: Patrick Hunt <phunt@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100317)
MIME-Version: 1.0
To: announce@apache.org,
        "zookeeper-dev@hadoop.apache.org" <zookeeper-dev@hadoop.apache.org>,
        "zookeeper-user@hadoop.apache.org" <zookeeper-user@hadoop.apache.org>,
        general@hadoop.apache.org
Subject: [ANNOUNCE] Apache ZooKeeper 3.3.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

The Apache ZooKeeper team is proud to announce Apache ZooKeeper version 
3.3.0.

ZooKeeper is a high-performance coordination service for distributed 
applications. It exposes common services - such as naming, configuration 
management, synchronization, and group services - in a simple interface 
so you don't have to write them from scratch. You can use it 
off-the-shelf to implement consensus, group management, leader election, 
and presence protocols. And you can build on it for your own, specific 
needs.

Key features of the 3.3.0 release:
* observers - non-voting members of the ensemble, scale reads 
http://bit.ly/9wPnpq
* distributed queue recipe implementation (c/java)
* additional 4letterword/jmx features to support operations
* maven repository support
* build support for eclipse project generation
* ivy based build

Changes in contrib:
* zookeeper-tree: export/import znode namespace
* zooinspector: gui browser for znode namespace
* bookkeeper client rewrite, use of netty

For ZooKeeper release details and downloads, visit:
http://hadoop.apache.org/zookeeper/releases.html

ZooKeeper 3.3.0 Release Notes are at:
http://hadoop.apache.org/zookeeper/docs/r3.3.0/releasenotes.html

Regards,

The ZooKeeper Team

From general-return-1253-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 27 02:40:22 2010
Return-Path: <general-return-1253-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 64362 invoked from network); 27 Mar 2010 02:40:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Mar 2010 02:40:22 -0000
Received: (qmail 9601 invoked by uid 500); 27 Mar 2010 02:40:21 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 9365 invoked by uid 500); 27 Mar 2010 02:40:19 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 36397 invoked by uid 99); 25 Mar 2010 14:31:45 -0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
From: Isabel Drost <isabel@apache.org>
Reply-To: isabel@apache.org
To: common-user@hadoop.apache.org
Subject: Re: [ANN] Hadoop 0.20.2 in Debian unstable
Date: Thu, 25 Mar 2010 15:31:03 +0100
User-Agent: KMail/1.12.4 (Linux/2.6.32-3-amd64; KDE/4.3.4; x86_64; ; )
Cc: Thomas Koch <thomas@koch.ro>,
 general@hadoop.apache.org,
 hbase-user@hadoop.apache.org
References: <201003251311.40203.thomas@koch.ro>
In-Reply-To: <201003251311.40203.thomas@koch.ro>
MIME-Version: 1.0
Content-Type: multipart/signed;
  boundary="nextPart6077567.SjRRpaTRjs";
  protocol="application/pgp-signature";
  micalg=pgp-sha1
Content-Transfer-Encoding: 7bit
Message-Id: <201003251531.06621.isabel@apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org

--nextPart6077567.SjRRpaTRjs
Content-Type: Text/Plain;
  charset="us-ascii"
Content-Transfer-Encoding: 7bit

On 25.03.2010 Thomas Koch wrote:
> hadoop has entered Debian unstable a few days ago:
> http://packages.qa.debian.org/h/hadoop.html

Congratulations!

Isabel

--nextPart6077567.SjRRpaTRjs
Content-Type: application/pgp-signature; name=signature.asc 
Content-Description: This is a digitally signed message part.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEABECAAYFAkurc6oACgkQPJpCBAwIhbSwvACgjO8iIfb5iyUW0hGGZTOgy5ST
1/gAoIrbDBbFy+if2ReiVcR9Md3BmIJu
=1dte
-----END PGP SIGNATURE-----

--nextPart6077567.SjRRpaTRjs--

From general-return-1254-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 27 02:40:27 2010
Return-Path: <general-return-1254-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 64436 invoked from network); 27 Mar 2010 02:40:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Mar 2010 02:40:26 -0000
Received: (qmail 10353 invoked by uid 500); 27 Mar 2010 02:40:26 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 10322 invoked by uid 500); 27 Mar 2010 02:40:26 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 1974 invoked by uid 99); 26 Mar 2010 23:51:46 -0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=AWL,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: 216.145.54.171 is neither permitted nor denied by domain of phunt@yahoo-inc.com)
Message-ID: <4BAD4859.9010809@yahoo-inc.com>
Date: Fri, 26 Mar 2010 16:50:49 -0700
From: Patrick Hunt <phunt@yahoo-inc.com>
User-Agent: Thunderbird 2.0.0.24 (X11/20100317)
MIME-Version: 1.0
To: announce@apache.org,
        "zookeeper-dev@hadoop.apache.org" <zookeeper-dev@hadoop.apache.org>,
        "zookeeper-user@hadoop.apache.org" <zookeeper-user@hadoop.apache.org>,
        general@hadoop.apache.org
Subject: [ANNOUNCE] Apache ZooKeeper 3.3.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

The Apache ZooKeeper team is proud to announce Apache ZooKeeper version 
3.3.0.

ZooKeeper is a high-performance coordination service for distributed 
applications. It exposes common services - such as naming, configuration 
management, synchronization, and group services - in a simple interface 
so you don't have to write them from scratch. You can use it 
off-the-shelf to implement consensus, group management, leader election, 
and presence protocols. And you can build on it for your own, specific 
needs.

Key features of the 3.3.0 release:
* observers - non-voting members of the ensemble, scale reads 
http://bit.ly/9wPnpq
* distributed queue recipe implementation (c/java)
* additional 4letterword/jmx features to support operations
* maven repository support
* build support for eclipse project generation
* ivy based build

Changes in contrib:
* zookeeper-tree: export/import znode namespace
* zooinspector: gui browser for znode namespace
* bookkeeper client rewrite, use of netty

For ZooKeeper release details and downloads, visit:
http://hadoop.apache.org/zookeeper/releases.html

ZooKeeper 3.3.0 Release Notes are at:
http://hadoop.apache.org/zookeeper/docs/r3.3.0/releasenotes.html

Regards,

The ZooKeeper Team

From general-return-1255-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Mar 28 19:10:54 2010
Return-Path: <general-return-1255-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 8775 invoked from network); 28 Mar 2010 19:10:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Mar 2010 19:10:54 -0000
Received: (qmail 16959 invoked by uid 500); 28 Mar 2010 19:10:52 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 16854 invoked by uid 500); 28 Mar 2010 19:10:51 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16839 invoked by uid 99); 28 Mar 2010 19:10:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 28 Mar 2010 19:10:51 +0000
X-ASF-Spam-Status: No, hits=2.4 required=10.0
	tests=AWL,FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of mailinglists19@gmail.com designates 74.125.82.48 as permitted sender)
Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 28 Mar 2010 19:10:44 +0000
Received: by wwi18 with SMTP id 18so2513233wwi.35
        for <multiple recipients>; Sun, 28 Mar 2010 12:10:23 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:received:message-id
         :subject:from:to:content-type;
        bh=HgoF2zmXsF9tXWKZmXYe8I2uZqJyfYnjflGb0WlIjDc=;
        b=Safb2Enc1jrMq2CQnnTrOJ2Kk6eusJs7YFMCtcumJ+/bLmeFoO2rsuA2mpGiixJxBD
         z+nJdRmC+xHEXhahR5UHa3AWQv9FAKOmabe+rqpu+RKmGb0ypF8LqM3O++Q/ZTIJy00r
         4cOZkuCBrBdgsQm+ItOq49UDS9csFsf6BSeec=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=j9GymAfCVFCxTl7GnCLM1hVg2yNcFW7KNgq0lN5ffIv8j5c58rWdsBuHcsmcL87SNY
         8+GHswOb8EPOI9LRQTc6P0W/vmuq3CFN6HLrK/LagkEFV6UerqjJXwotlcMNWijtfan3
         eBBdftSMIthA9HeLyJH/O37f4Tzmu08urp6Lg=
MIME-Version: 1.0
Received: by 10.216.27.72 with HTTP; Sun, 28 Mar 2010 12:10:23 -0700 (PDT)
Date: Sun, 28 Mar 2010 12:10:23 -0700
Received: by 10.216.85.19 with SMTP id t19mr2430773wee.107.1269803423127; Sun, 
	28 Mar 2010 12:10:23 -0700 (PDT)
Message-ID: <1eabbac31003281210xd5ad959v313ac785bf6fb7b1@mail.gmail.com>
Subject: Improving performance of Hadoop job
From: Something Something <mailinglists19@gmail.com>
To: general@hadoop.apache.org, hbase-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6dab6f5f85c6d0482e125f7

--0016e6dab6f5f85c6d0482e125f7
Content-Type: text/plain; charset=ISO-8859-1

When I run a Hadoop job in a command shell it runs in 21 secs, but when I
run the same from within Eclipse IDE it runs in 1 second!  Trying to figure
out what the reason is.  The output is shown below:

1)  It appears the FILE_BYTES_READ, HDFS_BYTES_WRITTEN etc values are
totally different.  Sounds like that's what is causing the performance
degradation.
2)  I tried setting -Xmx1000m in the command shell but that doesn't help.
Are there other JVM arguments that affect values of these variables
(FILE_BYTES_READ etc)?

Any hints would be GREATLY appreciated.  Thanks.


*Command Shell log:*

INFO: Starting top scores writing job....
10/03/28 11:19:49 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
10/03/28 11:19:50 INFO mapred.JobClient: Running job: job_201003281008_0006
10/03/28 11:19:51 INFO mapred.JobClient:  map 0% reduce 0%
10/03/28 11:19:59 INFO mapred.JobClient:  map 100% reduce 0%
10/03/28 11:20:04 INFO mapred.JobClient:  map 100% reduce 100%
10/03/28 11:20:10 INFO mapred.JobClient: Job complete: job_201003281008_0006
10/03/28 11:20:10 INFO mapred.JobClient: Counters: 16
10/03/28 11:20:10 INFO mapred.JobClient:   Job Counters
10/03/28 11:20:10 INFO mapred.JobClient:     Launched reduce tasks=1
10/03/28 11:20:10 INFO mapred.JobClient:     Launched map tasks=1
10/03/28 11:20:10 INFO mapred.JobClient:     Data-local map tasks=1
10/03/28 11:20:10 INFO mapred.JobClient:   FileSystemCounters
*10/03/28 11:20:10 INFO mapred.JobClient:     FILE_BYTES_READ=1736
10/03/28 11:20:10 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=3504
10/03/28 11:20:10 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=3706
*10/03/28 11:20:10 INFO mapred.JobClient:   Map-Reduce Framework
10/03/28 11:20:10 INFO mapred.JobClient:     Reduce input groups=87
10/03/28 11:20:10 INFO mapred.JobClient:     Combine output records=0
10/03/28 11:20:10 INFO mapred.JobClient:     Map input records=100
10/03/28 11:20:10 INFO mapred.JobClient:     Reduce shuffle bytes=0
10/03/28 11:20:10 INFO mapred.JobClient:     Reduce output records=100
10/03/28 11:20:10 INFO mapred.JobClient:     Spilled Records=200
10/03/28 11:20:10 INFO mapred.JobClient:     Map output bytes=1530
10/03/28 11:20:10 INFO mapred.JobClient:     Combine input records=0
10/03/28 11:20:10 INFO mapred.JobClient:     Map output records=100
10/03/28 11:20:10 INFO mapred.JobClient:     Reduce input records=100
*Total time for writing scores: 21 secs.*



*Eclipse console log:*

INFO: Starting top scores writing job....
10/03/28 11:29:15 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with
processName=JobTracker, sessionId= - already initialized
10/03/28 11:29:15 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
10/03/28 11:29:15 WARN mapred.JobClient: No job jar file set.  User classes
may not be found. See JobConf(Class) or JobConf#setJar(String).
10/03/28 11:29:15 INFO mapred.JobClient: Running job: job_local_0006
10/03/28 11:29:15 INFO mapred.MapTask: io.sort.mb = 100
10/03/28 11:29:16 INFO mapred.MapTask: data buffer = 79691776/99614720
10/03/28 11:29:16 INFO mapred.MapTask: record buffer = 262144/327680
10/03/28 11:29:16 INFO mapred.MapTask: Starting flush of map output
10/03/28 11:29:16 INFO mapred.MapTask: Finished spill 0
10/03/28 11:29:16 INFO mapred.TaskRunner: Task:attempt_local_0006_m_000000_0
is done. And is in the process of commiting
10/03/28 11:29:16 INFO mapred.LocalJobRunner:
10/03/28 11:29:16 INFO mapred.TaskRunner: Task
'attempt_local_0006_m_000000_0' done.
10/03/28 11:29:16 INFO mapred.LocalJobRunner:
10/03/28 11:29:16 INFO mapred.Merger: Merging 1 sorted segments
10/03/28 11:29:16 INFO mapred.Merger: Down to the last merge-pass, with 1
segments left of total size: 1732 bytes
10/03/28 11:29:16 INFO mapred.LocalJobRunner:
10/03/28 11:29:16 INFO mapred.TaskRunner: Task:attempt_local_0006_r_000000_0
is done. And is in the process of commiting
10/03/28 11:29:16 INFO mapred.LocalJobRunner:
10/03/28 11:29:16 INFO mapred.TaskRunner: Task attempt_local_0006_r_000000_0
is allowed to commit now
10/03/28 11:29:16 INFO output.FileOutputCommitter: Saved output of task
'attempt_local_0006_r_000000_0' to markers/results
10/03/28 11:29:16 INFO mapred.LocalJobRunner: reduce > reduce
10/03/28 11:29:16 INFO mapred.TaskRunner: Task
'attempt_local_0006_r_000000_0' done.
10/03/28 11:29:16 INFO mapred.JobClient:  map 100% reduce 100%
10/03/28 11:29:16 INFO mapred.JobClient: Job complete: job_local_0006
10/03/28 11:29:16 INFO mapred.JobClient: Counters: 14
10/03/28 11:29:16 INFO mapred.JobClient:   FileSystemCounters
*10/03/28 11:29:16 INFO mapred.JobClient:     FILE_BYTES_READ=95848
10/03/28 11:29:16 INFO mapred.JobClient:     HDFS_BYTES_READ=277286
10/03/28 11:29:16 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=304886
10/03/28 11:29:16 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=209112
*10/03/28 11:29:16 INFO mapred.JobClient:   Map-Reduce Framework
10/03/28 11:29:16 INFO mapred.JobClient:     Reduce input groups=87
10/03/28 11:29:16 INFO mapred.JobClient:     Combine output records=0
10/03/28 11:29:16 INFO mapred.JobClient:     Map input records=100
10/03/28 11:29:16 INFO mapred.JobClient:     Reduce shuffle bytes=0
10/03/28 11:29:16 INFO mapred.JobClient:     Reduce output records=100
10/03/28 11:29:16 INFO mapred.JobClient:     Spilled Records=200
10/03/28 11:29:16 INFO mapred.JobClient:     Map output bytes=1530
10/03/28 11:29:16 INFO mapred.JobClient:     Combine input records=0
10/03/28 11:29:16 INFO mapred.JobClient:     Map output records=100
10/03/28 11:29:16 INFO mapred.JobClient:     Reduce input records=100
*Total time for writing scores: 1 secs.
*

--0016e6dab6f5f85c6d0482e125f7--

From general-return-1256-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Sun Mar 28 23:39:05 2010
Return-Path: <general-return-1256-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 64102 invoked from network); 28 Mar 2010 23:39:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Mar 2010 23:39:05 -0000
Received: (qmail 2090 invoked by uid 500); 28 Mar 2010 23:39:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1835 invoked by uid 500); 28 Mar 2010 23:39:03 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1821 invoked by uid 99); 28 Mar 2010 23:39:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 28 Mar 2010 23:39:03 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of vermaabhishekp@gmail.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 28 Mar 2010 23:38:55 +0000
Received: by gyd8 with SMTP id 8so5431150gyd.35
        for <multiple recipients>; Sun, 28 Mar 2010 16:38:34 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:cc:content-type;
        bh=qm0qI/fwtD63/9QjpNgHc1wdHJps8MLzDxWoSoRCKCA=;
        b=x43lp5HQDKW9bArDkFd/B2Lec7HWGyPfCV7NHM2wTR5Mr29YJnjrHGx24RT2k/BGUr
         JeZuBNEOWNS+NG/nS7dI5Ws67M28oMn1TdXNrde3CkZL4nuGa2GA07wtg+aBsKe04r82
         /9+rzCgzfD7zDh4qE6P3soiTe3SpJ261ZpIHs=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=QCgEvAVlkjUR3rfOCpiqX9nvjsVbmR0aCK7qxBjtcPB3yvhI0Irsv9agdWHTy7sGnB
         fpEcXwMI0IF5BpVlnB4tM8MOf2ocLlzyjeA4vT53Ku4YKAV+v8LDAIYT0IEDGGMZX35V
         slvIfDKp0QLclxl/IB530K83kXGP5CKXNqpDA=
MIME-Version: 1.0
Received: by 10.151.79.16 with HTTP; Sun, 28 Mar 2010 16:38:34 -0700 (PDT)
In-Reply-To: <1eabbac31003281210xd5ad959v313ac785bf6fb7b1@mail.gmail.com>
References: <1eabbac31003281210xd5ad959v313ac785bf6fb7b1@mail.gmail.com>
Date: Sun, 28 Mar 2010 18:38:34 -0500
Received: by 10.150.117.26 with SMTP id p26mr4129657ybc.348.1269819514174; 
	Sun, 28 Mar 2010 16:38:34 -0700 (PDT)
Message-ID: <3c682ecd1003281638u7551d74by62c06ddbd4c0de82@mail.gmail.com>
Subject: Re: Improving performance of Hadoop job
From: Abhishek Verma <vermaabhishekp@gmail.com>
To: general@hadoop.apache.org
Cc: hbase-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd72a18123d190482e4e510
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd72a18123d190482e4e510
Content-Type: text/plain; charset=ISO-8859-1

The log shows that Eclipse is running the job locally:
0/03/28 11:29:16 INFO mapred.*LocalJobRunner*:

Hence the difference in the run times. Normally, Hadoop jobs have a start up
time of around 10 seconds atleast.

-Abhishek.

On Sun, Mar 28, 2010 at 2:10 PM, Something Something <
mailinglists19@gmail.com> wrote:

> When I run a Hadoop job in a command shell it runs in 21 secs, but when I
> run the same from within Eclipse IDE it runs in 1 second!  Trying to figure
> out what the reason is.  The output is shown below:
>
> 1)  It appears the FILE_BYTES_READ, HDFS_BYTES_WRITTEN etc values are
> totally different.  Sounds like that's what is causing the performance
> degradation.
> 2)  I tried setting -Xmx1000m in the command shell but that doesn't help.
> Are there other JVM arguments that affect values of these variables
> (FILE_BYTES_READ etc)?
>
> Any hints would be GREATLY appreciated.  Thanks.
>
>
> *Command Shell log:*
>
> INFO: Starting top scores writing job....
> 10/03/28 11:19:49 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
> 10/03/28 11:19:50 INFO mapred.JobClient: Running job: job_201003281008_0006
> 10/03/28 11:19:51 INFO mapred.JobClient:  map 0% reduce 0%
> 10/03/28 11:19:59 INFO mapred.JobClient:  map 100% reduce 0%
> 10/03/28 11:20:04 INFO mapred.JobClient:  map 100% reduce 100%
> 10/03/28 11:20:10 INFO mapred.JobClient: Job complete:
> job_201003281008_0006
> 10/03/28 11:20:10 INFO mapred.JobClient: Counters: 16
> 10/03/28 11:20:10 INFO mapred.JobClient:   Job Counters
> 10/03/28 11:20:10 INFO mapred.JobClient:     Launched reduce tasks=1
> 10/03/28 11:20:10 INFO mapred.JobClient:     Launched map tasks=1
> 10/03/28 11:20:10 INFO mapred.JobClient:     Data-local map tasks=1
> 10/03/28 11:20:10 INFO mapred.JobClient:   FileSystemCounters
> *10/03/28 11:20:10 INFO mapred.JobClient:     FILE_BYTES_READ=1736
> 10/03/28 11:20:10 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=3504
> 10/03/28 11:20:10 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=3706
> *10/03/28 11:20:10 INFO mapred.JobClient:   Map-Reduce Framework
> 10/03/28 11:20:10 INFO mapred.JobClient:     Reduce input groups=87
> 10/03/28 11:20:10 INFO mapred.JobClient:     Combine output records=0
> 10/03/28 11:20:10 INFO mapred.JobClient:     Map input records=100
> 10/03/28 11:20:10 INFO mapred.JobClient:     Reduce shuffle bytes=0
> 10/03/28 11:20:10 INFO mapred.JobClient:     Reduce output records=100
> 10/03/28 11:20:10 INFO mapred.JobClient:     Spilled Records=200
> 10/03/28 11:20:10 INFO mapred.JobClient:     Map output bytes=1530
> 10/03/28 11:20:10 INFO mapred.JobClient:     Combine input records=0
> 10/03/28 11:20:10 INFO mapred.JobClient:     Map output records=100
> 10/03/28 11:20:10 INFO mapred.JobClient:     Reduce input records=100
> *Total time for writing scores: 21 secs.*
>
>
>
> *Eclipse console log:*
>
> INFO: Starting top scores writing job....
> 10/03/28 11:29:15 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with
> processName=JobTracker, sessionId= - already initialized
> 10/03/28 11:29:15 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
> 10/03/28 11:29:15 WARN mapred.JobClient: No job jar file set.  User classes
> may not be found. See JobConf(Class) or JobConf#setJar(String).
> 10/03/28 11:29:15 INFO mapred.JobClient: Running job: job_local_0006
> 10/03/28 11:29:15 INFO mapred.MapTask: io.sort.mb = 100
> 10/03/28 11:29:16 INFO mapred.MapTask: data buffer = 79691776/99614720
> 10/03/28 11:29:16 INFO mapred.MapTask: record buffer = 262144/327680
> 10/03/28 11:29:16 INFO mapred.MapTask: Starting flush of map output
> 10/03/28 11:29:16 INFO mapred.MapTask: Finished spill 0
> 10/03/28 11:29:16 INFO mapred.TaskRunner:
> Task:attempt_local_0006_m_000000_0
> is done. And is in the process of commiting
> 10/03/28 11:29:16 INFO mapred.LocalJobRunner:
> 10/03/28 11:29:16 INFO mapred.TaskRunner: Task
> 'attempt_local_0006_m_000000_0' done.
> 10/03/28 11:29:16 INFO mapred.LocalJobRunner:
> 10/03/28 11:29:16 INFO mapred.Merger: Merging 1 sorted segments
> 10/03/28 11:29:16 INFO mapred.Merger: Down to the last merge-pass, with 1
> segments left of total size: 1732 bytes
> 10/03/28 11:29:16 INFO mapred.LocalJobRunner:
> 10/03/28 11:29:16 INFO mapred.TaskRunner:
> Task:attempt_local_0006_r_000000_0
> is done. And is in the process of commiting
> 10/03/28 11:29:16 INFO mapred.LocalJobRunner:
> 10/03/28 11:29:16 INFO mapred.TaskRunner: Task
> attempt_local_0006_r_000000_0
> is allowed to commit now
> 10/03/28 11:29:16 INFO output.FileOutputCommitter: Saved output of task
> 'attempt_local_0006_r_000000_0' to markers/results
> 10/03/28 11:29:16 INFO mapred.LocalJobRunner: reduce > reduce
> 10/03/28 11:29:16 INFO mapred.TaskRunner: Task
> 'attempt_local_0006_r_000000_0' done.
> 10/03/28 11:29:16 INFO mapred.JobClient:  map 100% reduce 100%
> 10/03/28 11:29:16 INFO mapred.JobClient: Job complete: job_local_0006
> 10/03/28 11:29:16 INFO mapred.JobClient: Counters: 14
> 10/03/28 11:29:16 INFO mapred.JobClient:   FileSystemCounters
> *10/03/28 11:29:16 INFO mapred.JobClient:     FILE_BYTES_READ=95848
> 10/03/28 11:29:16 INFO mapred.JobClient:     HDFS_BYTES_READ=277286
> 10/03/28 11:29:16 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=304886
> 10/03/28 11:29:16 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=209112
> *10/03/28 11:29:16 INFO mapred.JobClient:   Map-Reduce Framework
> 10/03/28 11:29:16 INFO mapred.JobClient:     Reduce input groups=87
> 10/03/28 11:29:16 INFO mapred.JobClient:     Combine output records=0
> 10/03/28 11:29:16 INFO mapred.JobClient:     Map input records=100
> 10/03/28 11:29:16 INFO mapred.JobClient:     Reduce shuffle bytes=0
> 10/03/28 11:29:16 INFO mapred.JobClient:     Reduce output records=100
> 10/03/28 11:29:16 INFO mapred.JobClient:     Spilled Records=200
> 10/03/28 11:29:16 INFO mapred.JobClient:     Map output bytes=1530
> 10/03/28 11:29:16 INFO mapred.JobClient:     Combine input records=0
> 10/03/28 11:29:16 INFO mapred.JobClient:     Map output records=100
> 10/03/28 11:29:16 INFO mapred.JobClient:     Reduce input records=100
> *Total time for writing scores: 1 secs.
> *
>

--000e0cd72a18123d190482e4e510--

From general-return-1257-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 29 19:03:51 2010
Return-Path: <general-return-1257-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 1089 invoked from network); 29 Mar 2010 19:03:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Mar 2010 19:03:51 -0000
Received: (qmail 52257 invoked by uid 500); 29 Mar 2010 19:03:46 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 51350 invoked by uid 500); 29 Mar 2010 19:03:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 51286 invoked by uid 99); 29 Mar 2010 19:03:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Mar 2010 19:03:45 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.92.25] (HELO qw-out-2122.google.com) (74.125.92.25)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Mar 2010 19:03:39 +0000
Received: by qw-out-2122.google.com with SMTP id 8so3874983qwh.35
        for <multiple recipients>; Mon, 29 Mar 2010 12:03:15 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.80.149 with HTTP; Mon, 29 Mar 2010 12:02:54 -0700 (PDT)
From: Aaron Kimball <aaron@cloudera.com>
Date: Mon, 29 Mar 2010 12:02:54 -0700
Received: by 10.229.214.74 with SMTP id gz10mr64696qcb.25.1269889395272; Mon, 
	29 Mar 2010 12:03:15 -0700 (PDT)
Message-ID: <d6d7c4411003291202l3a199257y13cef504070b9d38@mail.gmail.com>
Subject: Sqoop is moving to github!
To: hive-user@hadoop.apache.org, common-user@hadoop.apache.org, 
	general@hadoop.apache.org, mapreduce-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016362848204f2e8b0482f52a7f
X-Virus-Checked: Checked by ClamAV on apache.org

--0016362848204f2e8b0482f52a7f
Content-Type: text/plain; charset=ISO-8859-1

Hi Hadoop, Hive, and Sqoop users,

For the past year, the Apache Hadoop MapReduce project has played host to
Sqoop, a command-line tool that performs parallel imports and exports
between relational databases and HDFS. We've developed a lot of features and
gotten a lot of great feedback from users. While Sqoop was a contrib project
in Hadoop, it has been steadily improved and grown.

But the contrib directory is a home for new or small projects incubating
underneath Hadoop's umbrella. Sqoop is starting to look less like a small
project these days. In particular, a feature that has been growing in
importance for Sqoop is its ability to integrate with Hive. In order to
facilitate this integration from a compilation and testing standpoint, we've
pulled Sqoop out of contrib and into its own repository hosted on github.

You can download all the relevant bits here:
http://www.github.com/cloudera/sqoop

The code there will run in conjunction with the Apache Hadoop trunk source.
(Compatibility with other distributions/versions is forthcoming.)

While we've changed hosts, Sqoop will keep the same license -- future
improvements will continue to remain Apache 2.0-licensed. We welcome the
contributions of all in the open source community; there's a lot of exciting
work still to be done! If you'd like to help out but aren't sure where to
start, send me an email and I can recommend a few areas where improvements
would be appreciated.

Want some more information about Sqoop? An introduction is available here:
http://www.cloudera.com/sqoop
A ready-to-run release of Sqoop is included with Cloudera's Distribution for
Hadoop: http://archive.cloudera.com
And its reference manual is available for browsing at
http://archive.cloudera.com/docs/sqoop

If you have any questions about this move process, please ask me.

Regards,
- Aaron Kimball
Cloudera, Inc.

--0016362848204f2e8b0482f52a7f--

From general-return-1258-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 29 23:07:19 2010
Return-Path: <general-return-1258-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 83488 invoked from network); 29 Mar 2010 23:07:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Mar 2010 23:07:19 -0000
Received: (qmail 21137 invoked by uid 500); 29 Mar 2010 23:07:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 21041 invoked by uid 500); 29 Mar 2010 23:07:17 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 21033 invoked by uid 99); 29 Mar 2010 23:07:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Mar 2010 23:07:17 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of andrew.schlaikjer@gmail.com designates 209.85.160.176 as permitted sender)
Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Mar 2010 23:07:11 +0000
Received: by gyd8 with SMTP id 8so6064902gyd.35
        for <general@hadoop.apache.org>; Mon, 29 Mar 2010 16:06:50 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:date:received:message-id
         :subject:from:to:content-type;
        bh=ypy3paEjwCvfzeoZnAZiEDSJgFVDWkX+bpsSk9bXhv0=;
        b=QIo5+duLRtatnxzYo+mIYvSivT7C8cl8heBxultT+AFwaYXrJmUWbz+VduwlFYofGR
         dVHEdKBTBWTlVCRBdLphhNvgLOEiOBqCJHWjal5SdpHt7anuJvp7X0Hz9y1Lcs457LU8
         xLAHIGNbf5RkRME3yKxgvFOAcPQLlesTVzO5I=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=EP4d+zAoTu70AcTQHXvTvgQPTLqsOtFuY6qicEIXYDWKwA0tg8Wz/1E6EeanF5Maf1
         SJn2Mqvhijg85ygX9I/W0M7Hico/MOo5+MwyZMIIM3lcGwyrYaDsBcME3qXfjcdtnSrR
         PJmZzv8mwRJzeLVtJBMF31NS0Ds3EHO0cgLwE=
MIME-Version: 1.0
Received: by 10.150.219.20 with HTTP; Mon, 29 Mar 2010 16:06:50 -0700 (PDT)
Date: Mon, 29 Mar 2010 19:06:50 -0400
Received: by 10.151.5.11 with SMTP id h11mr5101533ybi.13.1269904010865; Mon, 
	29 Mar 2010 16:06:50 -0700 (PDT)
Message-ID: <76c0da9a1003291606h53e46a8gfa75f786b84db834@mail.gmail.com>
Subject: specification of type params in org.apache.hadoop.io.*Writable impls
From: Andy Schlaikjer <andrew.schlaikjer@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Can anyone comment on why the base *Writable implementations (e.g.
IntWritable, LongWritable, etc) don't specify WritableComparable<T>'s
type param T?

I noticed this when I tried to use DoubleWritable [1] within a custom
WritableComparable parameterized type which constrains its own type
params:

public class Pair<
  First extends WritableComparable<First>,
  Second extends WritableComparable<Second>>
  implements WritableComparable<Pair<First, Second>> {
...
}

With this declaration, something like the following will not compile:

Pair<DoubleWritable, DoubleWritable> pair = null;

ERROR: Bound mismatch: The type DoubleWritable is not a valid
substitute for the bounded parameter <First extends
WritableComparable<First>> of the type Pair<First,Second>

However, if DoubleWritable were declared as follows, the above line
would work just fine:

public class DoubleWritable implements WritableComparable<DoubleWritable> {
...
}

[1] http://svn.apache.org/viewvc/hadoop/common/trunk/src/java/org/apache/hadoop/io/DoubleWritable.java?revision=786726&view=markup

Best,

From general-return-1259-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 30 00:15:12 2010
Return-Path: <general-return-1259-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 96843 invoked from network); 30 Mar 2010 00:15:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Mar 2010 00:15:12 -0000
Received: (qmail 90262 invoked by uid 500); 30 Mar 2010 00:15:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 90227 invoked by uid 500); 30 Mar 2010 00:15:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 90219 invoked by uid 99); 30 Mar 2010 00:15:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Mar 2010 00:15:11 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of anthony.urso@gmail.com designates 209.85.217.225 as permitted sender)
Received: from [209.85.217.225] (HELO mail-gx0-f225.google.com) (209.85.217.225)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Mar 2010 00:15:03 +0000
Received: by gxk25 with SMTP id 25so6332010gxk.11
        for <general@hadoop.apache.org>; Mon, 29 Mar 2010 17:14:43 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type;
        bh=AeIwJ9mZKL4FhDSH/RpiIL8KaAwS/BR0AzjBdl8PMWM=;
        b=buGEvjDeZqbPO+fqzkk3KTfOj00HMHFK6SLGzNrt/Ue7DueG7krLkMlyTcEh3GixHK
         ktmAn2k2cCLcY4ytW+Gs27FmlFvUkeEWbTf9llTyPOPNpSkKq/L5Xf6fW6etmuyhmvSc
         6PDk6w2kP2ziNHOO3Xtzhb3Cnv79iYj2OL0R8=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=TR8koqylIpjMV9zQiwuG2w72hO9hxjx380Bcpv0xP/ykgpYJ/DhgGg74FhyY38yfqz
         LJFj1zUiz9EA+0UbQ0FMZbXH9ZszehD6MNVqcbINsW6tlu3YMuq/n6NkHglpVcl8aIGW
         Wr2+S1p6PdkJM7drwudDuwCUC87mvjG+fZxH4=
MIME-Version: 1.0
Received: by 10.231.124.31 with HTTP; Mon, 29 Mar 2010 17:14:41 -0700 (PDT)
In-Reply-To: <bd7d2ecd1003051950jc21e4d0pe682b29866dce7b4@mail.gmail.com>
References: <bd7d2ecd1003051950jc21e4d0pe682b29866dce7b4@mail.gmail.com>
Date: Mon, 29 Mar 2010 17:14:41 -0700
Received: by 10.101.148.6 with SMTP id a6mr4508968ano.150.1269908081518; Mon, 
	29 Mar 2010 17:14:41 -0700 (PDT)
Message-ID: <ba72f77f1003291714w3890949ch5da5e0bcf7668663@mail.gmail.com>
Subject: Re: How to send KV pair to a reduce task on a particular machine?
From: Anthony Urso <anthony.urso@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

The dirty way to do this is to have the reducer throw an exception if
it receives a key that was not intended for the node it is running on.
 It will be rescheduled on another node, and eventually it will land
on the correct one.

Depending on the total number of nodes and reducers in the job, you
may have to increase the max failed tasks configuration parameter in
order that the whole job does not fail while the reducers are bouncing
around from node to node.

Cheers,
Anthony

On Fri, Mar 5, 2010 at 8:50 PM, Yanfeng Zhang <zhangyf14@gmail.com> wrote:
> Hi, all
>
> The KV pairs (kv1, kv2, kv3 kv4) out from mapper would be partitioned into R
> parts (e.g. R=2) by a partitioner. For example, kv1 and kv2 are in
> partition1, while kv3 and kv4 are in partition2, the reducers will get KV
> pairs from these two partitions, reducer1 get KV pairs from partition1 and
> reducer2 get KV pairs from partition2.
>
> I want to let machine1 get KV pairs from partition1 and machine2 get KV
> pairs from partition2. But reducer1 is not always on machine1, reducer2 is
> not always on machine2. Is there any way to make sure kv1 and kv2 are sent
> to machine1 and kv3, kv4 are sent to machine2?
>
> Thank you in advance!
>
> Sincerely,
> --
> Yanfeng Zhang
>

From general-return-1260-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 30 01:22:02 2010
Return-Path: <general-return-1260-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 17195 invoked from network); 30 Mar 2010 01:22:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Mar 2010 01:22:02 -0000
Received: (qmail 74643 invoked by uid 500); 30 Mar 2010 01:22:01 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74602 invoked by uid 500); 30 Mar 2010 01:22:01 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74594 invoked by uid 99); 30 Mar 2010 01:22:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Mar 2010 01:22:01 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of invisiblesun@mac.com designates 17.148.16.101 as permitted sender)
Received: from [17.148.16.101] (HELO asmtpout026.mac.com) (17.148.16.101)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Mar 2010 01:21:52 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; charset=us-ascii
Received: from [172.27.35.143]
 (c-67-169-39-169.hsd1.ca.comcast.net [67.169.39.169])
 by asmtp026.mac.com (Sun Java(tm) System Messaging Server 6.3-8.01 (built Dec
 16 2008; 32bit)) with ESMTPSA id <0L020083LMFF3L00@asmtp026.mac.com> for
 general@hadoop.apache.org; Mon, 29 Mar 2010 18:21:16 -0700 (PDT)
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0
 reason=mlx engine=5.0.0-0908210000 definitions=main-1003290299
Subject: Re: specification of type params in org.apache.hadoop.io.*Writable
 impls
From: Julia Smith <invisiblesun@mac.com>
In-reply-to: <76c0da9a1003291606h53e46a8gfa75f786b84db834@mail.gmail.com>
Date: Mon, 29 Mar 2010 18:21:18 -0700
Message-id: <5F413AFC-D115-4561-8F48-73A42986477A@mac.com>
References: <76c0da9a1003291606h53e46a8gfa75f786b84db834@mail.gmail.com>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1077)
X-Virus-Checked: Checked by ClamAV on apache.org

Java Generics are finicky. Basically the parameterized generic is not considered the same thing (type) as a class extended from it. I went through this when writing a deeply generic-ed package just to test its limits with a heavily templatized c++ package.

On Mar 29, 2010, at 4:06 PM, Andy Schlaikjer wrote:

> Can anyone comment on why the base *Writable implementations (e.g.
> IntWritable, LongWritable, etc) don't specify WritableComparable<T>'s
> type param T?
> 
> I noticed this when I tried to use DoubleWritable [1] within a custom
> WritableComparable parameterized type which constrains its own type
> params:
> 
> public class Pair<
>  First extends WritableComparable<First>,
>  Second extends WritableComparable<Second>>
>  implements WritableComparable<Pair<First, Second>> {
> ...
> }
> 
> With this declaration, something like the following will not compile:
> 
> Pair<DoubleWritable, DoubleWritable> pair = null;
> 
> ERROR: Bound mismatch: The type DoubleWritable is not a valid
> substitute for the bounded parameter <First extends
> WritableComparable<First>> of the type Pair<First,Second>
> 
> However, if DoubleWritable were declared as follows, the above line
> would work just fine:
> 
> public class DoubleWritable implements WritableComparable<DoubleWritable> {
> ...
> }
> 
> [1] http://svn.apache.org/viewvc/hadoop/common/trunk/src/java/org/apache/hadoop/io/DoubleWritable.java?revision=786726&view=markup
> 
> Best,


From general-return-1261-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 30 08:55:32 2010
Return-Path: <general-return-1261-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 8040 invoked from network); 30 Mar 2010 08:55:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Mar 2010 08:55:32 -0000
Received: (qmail 6180 invoked by uid 500); 30 Mar 2010 08:55:31 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 5955 invoked by uid 500); 30 Mar 2010 08:55:29 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 5943 invoked by uid 99); 30 Mar 2010 08:55:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Mar 2010 08:55:28 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of bernd.fondermann@googlemail.com designates 74.125.82.176 as permitted sender)
Received: from [74.125.82.176] (HELO mail-wy0-f176.google.com) (74.125.82.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Mar 2010 08:55:22 +0000
Received: by wyf19 with SMTP id 19so5402982wyf.35
        for <general@hadoop.apache.org>; Tue, 30 Mar 2010 01:55:02 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type;
        bh=VfSE8TqigkGnBDtKUnnXTT7YFO3COf3AWA7Z6NILBGs=;
        b=BTPDQN/GpYV2TywFBNUGIQEXTlReqt9tLXxkl1bgf4T7cUSu8qmj2N4XxOwsT8XSVR
         DbUsSiQnZslUMr9L/JmjvVgZbGMw7grrEkosbesBvlr/7sYcEBew6n7V7YCRVGiL5VjE
         AGWAMGNUY0nxdGzNMr2fjb+HGfY3Te/KjydnM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=C/Iyz73DLEY5avI0LlvnmCF6GJBTBrn5+E4pnH8x6R2VW4LOI8Dg9eNPXtVZQIEYjt
         MFta0hqDw+K0jJeSYfkta4X+kMqE+Z3JmE+lBDBRcc97E8af7D6OvR4vQ0ceARcYi/uM
         rVaBg1ZCfPbphpuRAd9Z3MxCH18SrXFPCfE34=
MIME-Version: 1.0
Received: by 10.216.16.35 with HTTP; Tue, 30 Mar 2010 01:55:02 -0700 (PDT)
In-Reply-To: <d6d7c4411003291202l3a199257y13cef504070b9d38@mail.gmail.com>
References: <d6d7c4411003291202l3a199257y13cef504070b9d38@mail.gmail.com>
Date: Tue, 30 Mar 2010 10:55:02 +0200
Received: by 10.216.86.71 with SMTP id v49mr3400561wee.14.1269939302068; Tue, 
	30 Mar 2010 01:55:02 -0700 (PDT)
Message-ID: <88f6e29a1003300155v1a553afcy59b2a5b7a34fbf5@mail.gmail.com>
Subject: Re: Sqoop is moving to github!
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Aaron,

Good to see you are a contributor to sqoop. Are you a committer yet?
Do you haven an ICLA on file with the ASF? I cannot find any record of it.
I must be missing something here, since PMCs are normally requesting
ICLAs from people making such substantial code contributions.

On Mon, Mar 29, 2010 at 21:02, Aaron Kimball <aaron@cloudera.com> wrote:
> Hi Hadoop, Hive, and Sqoop users,
>
> For the past year, the Apache Hadoop MapReduce project has played host to
> Sqoop, a command-line tool that performs parallel imports and exports
> between relational databases and HDFS. We've developed a lot of features and
> gotten a lot of great feedback from users.

Who is "we" exactly? Cloudera? Hadoop? You?

> While Sqoop was a contrib project
> in Hadoop, it has been steadily improved and grown.

Cool.

> But the contrib directory is a home for new or small projects incubating
> underneath Hadoop's umbrella. Sqoop is starting to look less like a small
> project these days. In particular, a feature that has been growing in
> importance for Sqoop is its ability to integrate with Hive. In order to
> facilitate this integration from a compilation and testing standpoint, we've
> pulled Sqoop out of contrib and into its own repository hosted on github.

So, you are forking sqoop. To facilitate that an Hadoop project can
work with another Hadoop project.
What are the issues with Hadoop that you cannot do it within Hadoop itself?

> You can download all the relevant bits here:
> http://www.github.com/cloudera/sqoop
>
> The code there will run in conjunction with the Apache Hadoop trunk source.
> (Compatibility with other distributions/versions is forthcoming.)

Sqoop is in ASF svn. What do you do when someone is going to continue
developing it here.
Then there's a naming clash. Do you intend to rename your fork?

> While we've changed hosts, Sqoop will keep the same license -- future
> improvements will continue to remain Apache 2.0-licensed. We welcome the
> contributions of all in the open source community; there's a lot of exciting
> work still to be done! If you'd like to help out but aren't sure where to
> start, send me an email and I can recommend a few areas where improvements
> would be appreciated.

Who is "we" in this case? The same "we" as above?

@Hadoop PMC: What is your statement on this fork announcement?

Thanks for clarifying,

  Bernd

From general-return-1262-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 30 13:48:58 2010
Return-Path: <general-return-1262-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 93283 invoked from network); 30 Mar 2010 13:48:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Mar 2010 13:48:58 -0000
Received: (qmail 26813 invoked by uid 500); 30 Mar 2010 13:48:57 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26755 invoked by uid 500); 30 Mar 2010 13:48:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26747 invoked by uid 99); 30 Mar 2010 13:48:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Mar 2010 13:48:56 +0000
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of owen.omalley@gmail.com designates 209.85.160.48 as permitted sender)
Received: from [209.85.160.48] (HELO mail-pw0-f48.google.com) (209.85.160.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Mar 2010 13:48:49 +0000
Received: by pwi7 with SMTP id 7so8842267pwi.35
        for <general@hadoop.apache.org>; Tue, 30 Mar 2010 06:48:28 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type;
        bh=bHKYbUmSWixyksVlCkpB7iiZ472zchp+wrETtFUSwvs=;
        b=hNAvi76KfI0bWrCeLXUJO2V5lEb/UeuKKKqwEKZSuL5nvKy9AhcnI3Pm9suJF/1ip3
         7+NYlgKqUMlQn/IKO2jeO+qu/v6TMetxXWQH4oiLaS3pI7wRAGpsn30aSdbS5wODE3NX
         GXR1sWIUlkTu0WFgvlKcvU5VleBeZvhwwKWiM=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=raJ6tIjLh127BPLE0it2MAc55BSIM+AatKGuccIJQgXkwUCA+MefCZV68QGVZOyYLo
         S1cuGqlgGVsc7Bw/B0ylSPH19Q8WP9vhsJmMqSQFVrhGN5liO5O1u5oIHCbb6RTOvHG9
         TOOXh95MgAetb7Zh35oJPJHnDR6FjF5c2/XRw=
MIME-Version: 1.0
Received: by 10.231.58.211 with HTTP; Tue, 30 Mar 2010 06:48:28 -0700 (PDT)
In-Reply-To: <88f6e29a1003300155v1a553afcy59b2a5b7a34fbf5@mail.gmail.com>
References: <d6d7c4411003291202l3a199257y13cef504070b9d38@mail.gmail.com>
	 <88f6e29a1003300155v1a553afcy59b2a5b7a34fbf5@mail.gmail.com>
Date: Tue, 30 Mar 2010 06:48:28 -0700
Received: by 10.140.251.8 with SMTP id y8mr5740680rvh.231.1269956908297; Tue, 
	30 Mar 2010 06:48:28 -0700 (PDT)
Message-ID: <5f7f740b1003300648u6a9bbb5fj9a2e4eddbe737b8c@mail.gmail.com>
Subject: Re: Sqoop is moving to github!
From: "Owen O'Malley" <owen.omalley@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd17fa6662e00048304e26e
X-Virus-Checked: Checked by ClamAV on apache.org

--000e0cd17fa6662e00048304e26e
Content-Type: text/plain; charset=UTF-8

On Tue, Mar 30, 2010 at 1:55 AM, Bernd Fondermann <
bernd.fondermann@googlemail.com> wrote:

>
> @Hadoop PMC: What is your statement on this fork announcement?


Aaron has been the primary developer of Sqoop. He has asked for it to be
removed from MapReduce's contrib. If a committer -1's the code removal, it
will cause a fork. But that hasn't happened and I don't think it will. In
order for it to make sense to keep the code in Hadoop, someone needs to be
working on it and the only person working on it is Aaron. If someone wants
to make a case for keeping Sqoop in MapReduce, please make it in the jira:
https://issues.apache.org/jira/browse/MAPREDUCE-1644

The sub-projects must avoid circular dependencies. We must be able to build
the sub-projects in order without having to go back and update a previous
project. Making a component of MapReduce depend on Hive (or Pig) would cause
that kind of cycle. Some projects like Zebra just live in the upper project
(Pig in that case). The other option is to ask to be a sub-project, ask to
join Apache incubator, or go to Github. Github doesn't give him the legal or
community aspects of Apache, but it also doesn't require asking permission
or introduce any process requirements.

-- Owen

--000e0cd17fa6662e00048304e26e--

From general-return-1263-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 30 14:02:11 2010
Return-Path: <general-return-1263-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 98274 invoked from network); 30 Mar 2010 14:02:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Mar 2010 14:02:11 -0000
Received: (qmail 45263 invoked by uid 500); 30 Mar 2010 14:02:10 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45190 invoked by uid 500); 30 Mar 2010 14:02:10 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45182 invoked by uid 99); 30 Mar 2010 14:02:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Mar 2010 14:02:10 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of andrew.schlaikjer@gmail.com designates 209.85.212.48 as permitted sender)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Mar 2010 14:02:04 +0000
Received: by vws14 with SMTP id 14so4888494vws.35
        for <general@hadoop.apache.org>; Tue, 30 Mar 2010 07:01:43 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type;
        bh=ov2SylEhwbS4GjKRwk/AlbSlbG4ba3sxTZxziVKyCrQ=;
        b=fFflOU0Mj9Rt6OaL9keg0ZBoJw6yWY7gAi82TdVqwYElZseO/ciIdxXH3CIbpADqkW
         xjVaNVU9gNhbEV6SDEcG7JTlPBE76HaVFMjyAeQP6GJt+BVI0eEidfF2yocqYVTaUXcs
         qtXT006NnPj8SHIxHD7vUpNSDE+0TlYGkrKTA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=GGYgZz0USRhcFApLlFtMMPWgC5XBRmTk5tY+lGC+kPuqWovhdOKFni99SeaTeYzqtu
         9lZZFrDRPylC+/VTeV4neEocjC8tohZHYKjGlK/erqw38JDHEAUSIMqI1ZjHr2UiCZiH
         YwUzQMor9N5BCAM03IwlK4GZriPeY2x+36z2k=
MIME-Version: 1.0
Received: by 10.150.219.20 with HTTP; Tue, 30 Mar 2010 07:01:43 -0700 (PDT)
In-Reply-To: <5F413AFC-D115-4561-8F48-73A42986477A@mac.com>
References: <76c0da9a1003291606h53e46a8gfa75f786b84db834@mail.gmail.com>
	 <5F413AFC-D115-4561-8F48-73A42986477A@mac.com>
Date: Tue, 30 Mar 2010 10:01:43 -0400
Received: by 10.220.126.200 with SMTP id d8mr3991117vcs.166.1269957703252; 
	Tue, 30 Mar 2010 07:01:43 -0700 (PDT)
Message-ID: <76c0da9a1003300701o5e7f8a2ifd4a2d262bf92651@mail.gmail.com>
Subject: Re: specification of type params in org.apache.hadoop.io.*Writable 
	impls
From: Andy Schlaikjer <andrew.schlaikjer@gmail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

I've received a number of responses from helpful Hadoop list members. Thanks!

But I should clarify here; I'm not looking for workarounds for my Pair
declaration or explanations of Java's generics facilities. I'm looking
for justifications for Hadoop's approach.

Looking at Java's core library, there are 86 classes which implement
java.lang.Comparable<T> [1]. 80 of these specify T = the type being
declared, or some parent of the type being declared.

[1] http://java.sun.com/javase/6/docs/api/java/lang/Comparable.html

So, I'm curious, are there specific use cases the Hadoop community has
run into which support the current design of the primitive *Writable
types which excludes specification of Comparable<T>'s T param? If not
I'll try and find time to work on a patch to push more conventional
use of generics into Hadoop Common.

Best,

From general-return-1264-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 30 14:54:44 2010
Return-Path: <general-return-1264-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 15072 invoked from network); 30 Mar 2010 14:54:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Mar 2010 14:54:44 -0000
Received: (qmail 34144 invoked by uid 500); 30 Mar 2010 14:54:43 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 34093 invoked by uid 500); 30 Mar 2010 14:54:43 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 34085 invoked by uid 99); 30 Mar 2010 14:54:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Mar 2010 14:54:43 +0000
X-ASF-Spam-Status: No, hits=0.2 required=10.0
	tests=AWL,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bernd.fondermann@googlemail.com designates 74.125.78.26 as permitted sender)
Received: from [74.125.78.26] (HELO ey-out-2122.google.com) (74.125.78.26)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Mar 2010 14:54:38 +0000
Received: by ey-out-2122.google.com with SMTP id 4so992611eyf.23
        for <general@hadoop.apache.org>; Tue, 30 Mar 2010 07:54:17 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type;
        bh=FAe8F3DxV6xNvQtatotBq5/GoQCyDzvz1Z4Yc+HF/XI=;
        b=rxiPA9yPgkKmiIAZQ4S/IgGANS/gS1a1mjQEePfy6+lPUBOFqnd38Fr/iG6Ysfl8H8
         aFQZ7MLPa1UT4PFP4XpWKbGBz6IGuVTs3O0MmrBaps7aEhi3EGUGOOaBA0Ib7waY/TPS
         /jfO1oRUulVjGVc4USEHIKuBrsrpSyYk6HZsA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=MS2MR1+YG6p8QzCKEuW8DVPH0mR9nKpl8qKFaKrR6zRI2t1kY/VApuLzVXgH4sXp/T
         BM3r5PuhAy2BI5UlqTWpjv0oW+PEbmsDAzrX2jdlBEcgUWXwNkCS6KuvVJeNj48U7+J+
         hPKZg0/iT3ZCCNTgxg2s62G7Q5ea7GumpJIO8=
MIME-Version: 1.0
Received: by 10.216.16.35 with HTTP; Tue, 30 Mar 2010 07:54:17 -0700 (PDT)
In-Reply-To: <5f7f740b1003300648u6a9bbb5fj9a2e4eddbe737b8c@mail.gmail.com>
References: <d6d7c4411003291202l3a199257y13cef504070b9d38@mail.gmail.com>
	 <88f6e29a1003300155v1a553afcy59b2a5b7a34fbf5@mail.gmail.com>
	 <5f7f740b1003300648u6a9bbb5fj9a2e4eddbe737b8c@mail.gmail.com>
Date: Tue, 30 Mar 2010 16:54:17 +0200
Received: by 10.216.87.146 with SMTP id y18mr762904wee.127.1269960857086; Tue, 
	30 Mar 2010 07:54:17 -0700 (PDT)
Message-ID: <88f6e29a1003300754m1826b4f0ve664df8e6b270ae0@mail.gmail.com>
Subject: Re: Sqoop is moving to github!
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Mar 30, 2010 at 15:48, Owen O'Malley <owen.omalley@gmail.com> wrote:
> On Tue, Mar 30, 2010 at 1:55 AM, Bernd Fondermann <
> bernd.fondermann@googlemail.com> wrote:
>
>>
>> @Hadoop PMC: What is your statement on this fork announcement?
>
>
> Aaron has been the primary developer of Sqoop. He has asked for it to be
> removed from MapReduce's contrib. If a committer -1's the code removal, it
> will cause a fork. But that hasn't happened and I don't think it will.

It's not the contributor's code alone anymore, once it's been committed.
Where has this been discussed? Where's the PMC-level vote/record of
consensus to ratify to abandon the code?
(ML + TS will be sufficient and I'll find my way.)

> In
> order for it to make sense to keep the code in Hadoop, someone needs to be
> working on it and the only person working on it is Aaron. If someone wants
> to make a case for keeping Sqoop in MapReduce, please make it in the jira:
> https://issues.apache.org/jira/browse/MAPREDUCE-1644

Well, at least someone reviewed and committed all the contributions to
svn in the first place, he can't be the only one working on it.

> The sub-projects must avoid circular dependencies. We must be able to build
> the sub-projects in order without having to go back and update a previous
> project. Making a component of MapReduce depend on Hive (or Pig) would cause
> that kind of cycle. Some projects like Zebra just live in the upper project
> (Pig in that case). The other option is to ask to be a sub-project, ask to
> join Apache incubator, or go to Github. Github doesn't give him the legal or
> community aspects of Apache, but it also doesn't require asking permission
> or introduce any process requirements.

Well, I'm not looking at this from the contributor's perspective here.
He can do with the code whatever he likes, basically.
It's only the PMC's viewpoint which is interesting for me.

  Bernd

From general-return-1265-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 30 16:51:36 2010
Return-Path: <general-return-1265-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 55477 invoked from network); 30 Mar 2010 16:51:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Mar 2010 16:51:36 -0000
Received: (qmail 93429 invoked by uid 500); 30 Mar 2010 16:51:35 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 93391 invoked by uid 500); 30 Mar 2010 16:51:35 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 93380 invoked by uid 99); 30 Mar 2010 16:51:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Mar 2010 16:51:35 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 30 Mar 2010 16:51:33 +0000
Received: (qmail 55399 invoked by uid 99); 30 Mar 2010 16:51:10 -0000
Received: from localhost.apache.org (HELO [192.168.168.100]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Mar 2010 16:51:10 +0000
Message-ID: <4BB22BFE.30906@apache.org>
Date: Tue, 30 Mar 2010 09:51:10 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100317)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: specification of type params in org.apache.hadoop.io.*Writable
 impls
References: <76c0da9a1003291606h53e46a8gfa75f786b84db834@mail.gmail.com>
In-Reply-To: <76c0da9a1003291606h53e46a8gfa75f786b84db834@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Andy Schlaikjer wrote:
> Can anyone comment on why the base *Writable implementations (e.g.
> IntWritable, LongWritable, etc) don't specify WritableComparable<T>'s
> type param T?

I suspect it's just historic.  When they were first written we were not 
using Java generics, and no one has ever retrofitted them with generics.

Doug

From general-return-1266-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 30 17:28:56 2010
Return-Path: <general-return-1266-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 62740 invoked from network); 30 Mar 2010 17:28:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Mar 2010 17:28:56 -0000
Received: (qmail 58199 invoked by uid 500); 30 Mar 2010 17:28:55 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 58127 invoked by uid 500); 30 Mar 2010 17:28:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 58119 invoked by uid 99); 30 Mar 2010 17:28:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Mar 2010 17:28:55 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of invisiblesun@mac.com designates 17.148.16.103 as permitted sender)
Received: from [17.148.16.103] (HELO asmtpout028.mac.com) (17.148.16.103)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Mar 2010 17:28:47 +0000
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; charset=us-ascii
Received: from [172.27.35.143]
 (c-67-169-39-169.hsd1.ca.comcast.net [67.169.39.169])
 by asmtp028.mac.com (Sun Java(tm) System Messaging Server 6.3-8.01 (built Dec
 16 2008; 32bit)) with ESMTPSA id <0L030059FV76RS80@asmtp028.mac.com> for
 general@hadoop.apache.org; Tue, 30 Mar 2010 10:28:19 -0700 (PDT)
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
 ipscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0
 reason=mlx engine=5.0.0-0908210000 definitions=main-1003300170
Subject: Re: specification of type params in org.apache.hadoop.io.*Writable
 impls
From: Julia Smith <invisiblesun@mac.com>
In-reply-to: <4BB22BFE.30906@apache.org>
Date: Tue, 30 Mar 2010 10:28:18 -0700
Message-id: <EEFFFD0C-A114-4369-BE1A-46084D47DA84@mac.com>
References: <76c0da9a1003291606h53e46a8gfa75f786b84db834@mail.gmail.com>
 <4BB22BFE.30906@apache.org>
To: general@hadoop.apache.org
X-Mailer: Apple Mail (2.1078)

It's easy enough to change the prototype to Comparable<?>, should fix it.

On Mar 30, 2010, at 9:51 AM, Doug Cutting wrote:

> Andy Schlaikjer wrote:
>> Can anyone comment on why the base *Writable implementations (e.g.
>> IntWritable, LongWritable, etc) don't specify WritableComparable<T>'s
>> type param T?
> 
> I suspect it's just historic.  When they were first written we were not using Java generics, and no one has ever retrofitted them with generics.
> 
> Doug


From general-return-1267-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 30 17:53:40 2010
Return-Path: <general-return-1267-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 67921 invoked from network); 30 Mar 2010 17:53:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Mar 2010 17:53:40 -0000
Received: (qmail 91372 invoked by uid 500); 30 Mar 2010 17:53:39 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 91336 invoked by uid 500); 30 Mar 2010 17:53:39 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 91328 invoked by uid 99); 30 Mar 2010 17:53:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Mar 2010 17:53:39 +0000
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [74.125.92.26] (HELO qw-out-2122.google.com) (74.125.92.26)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Mar 2010 17:53:33 +0000
Received: by qw-out-2122.google.com with SMTP id 8so4531277qwh.35
        for <general@hadoop.apache.org>; Tue, 30 Mar 2010 10:53:11 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.80.149 with HTTP; Tue, 30 Mar 2010 10:52:51 -0700 (PDT)
In-Reply-To: <88f6e29a1003300155v1a553afcy59b2a5b7a34fbf5@mail.gmail.com>
References: <d6d7c4411003291202l3a199257y13cef504070b9d38@mail.gmail.com> 
	<88f6e29a1003300155v1a553afcy59b2a5b7a34fbf5@mail.gmail.com>
From: Aaron Kimball <aaron@cloudera.com>
Date: Tue, 30 Mar 2010 10:52:51 -0700
Received: by 10.229.38.147 with SMTP id b19mr1681212qce.102.1269971591391; 
	Tue, 30 Mar 2010 10:53:11 -0700 (PDT)
Message-ID: <d6d7c4411003301052i9dcf686y20c14d588fdf2554@mail.gmail.com>
Subject: Re: Sqoop is moving to github!
To: Bernd Fondermann <bernd.fondermann@googlemail.com>
Cc: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016363b8be89477de0483084d90
X-Virus-Checked: Checked by ClamAV on apache.org

--0016363b8be89477de0483084d90
Content-Type: text/plain; charset=ISO-8859-1

Hi Bernd,

These are some important questions about the status of the project; thanks
for asking them. I'll go through them inline to preserve context on each
one.

On Tue, Mar 30, 2010 at 1:55 AM, Bernd Fondermann <
bernd.fondermann@googlemail.com> wrote:

> Hi Aaron,
>
> Good to see you are a contributor to sqoop. Are you a committer yet?
> Do you haven an ICLA on file with the ASF? I cannot find any record of it.
> I must be missing something here, since PMCs are normally requesting
> ICLAs from people making such substantial code contributions.
>
>
It might be more precise to call me *the* contributor to Sqoop. I've written
about 98% of the code for it; a few other individuals have provided me with
small enhancements or bugfixes, but the overwhelming amount of its care and
feeding has been under my watch.

I am not a committer on the Hadoop MapReduce (or any other ASF) project.
Thus far, nobody has invited me to sign an ICLA with my contributor-only
status. I have relied on others (primarily Tom White) to actually commit all
the Sqoop patches to svn.



> On Mon, Mar 29, 2010 at 21:02, Aaron Kimball <aaron@cloudera.com> wrote:
> > Hi Hadoop, Hive, and Sqoop users,
> >
> > For the past year, the Apache Hadoop MapReduce project has played host to
> > Sqoop, a command-line tool that performs parallel imports and exports
> > between relational databases and HDFS. We've developed a lot of features
> and
> > gotten a lot of great feedback from users.
>
> Who is "we" exactly? Cloudera? Hadoop? You?
>
>
Both myself and Cloudera. As said above, the vast majority of the direct
work on the project has been my own. But there are others at Cloudera who
have helped in less visible fashion with feature prioritization, design
input, code review, QA, user support, etc. And the contributions I make to
Sqoop, I do so as an employee of Cloudera.


> > While Sqoop was a contrib project
> > in Hadoop, it has been steadily improved and grown.
>
> Cool.
>
> > But the contrib directory is a home for new or small projects incubating
> > underneath Hadoop's umbrella. Sqoop is starting to look less like a small
> > project these days. In particular, a feature that has been growing in
> > importance for Sqoop is its ability to integrate with Hive. In order to
> > facilitate this integration from a compilation and testing standpoint,
> we've
> > pulled Sqoop out of contrib and into its own repository hosted on github.
>
> So, you are forking sqoop. To facilitate that an Hadoop project can
> work with another Hadoop project.
> What are the issues with Hadoop that you cannot do it within Hadoop itself?
>
>
When you put it like that, "forking" seems like a bit of a strong term. As
said in my original email, I prefer to think of it as "moving." (See the
next answer below for more on this.)

I believe Owen has already described some of the technical problems.
Conflating Sqoop's source repository with Hadoop's causes unnecessary
circular dependencies that build tools cannot easily work around. The more
straightforward method is to factor out Sqoop into a separate source
repository.


> > You can download all the relevant bits here:
> > http://www.github.com/cloudera/sqoop
> >
> > The code there will run in conjunction with the Apache Hadoop trunk
> source.
> > (Compatibility with other distributions/versions is forthcoming.)
>
> Sqoop is in ASF svn. What do you do when someone is going to continue
> developing it here.
> Then there's a naming clash. Do you intend to rename your fork?
>


I have filed MAPREDUCE-1644 with a patch that completely removes Sqoop from
the MapReduce repository. This will remove Sqoop from the working copy of
the repository, but of course, it will still belong to the ASF's repository
history. (Thus, I hope this will be seen as a straightforward lateral move
more than a fork.)

It's worth pointing out that Sqoop was originally introduced in HADOOP-5815,
committed after 0.20 was branched for release and closed to new features. So
Sqoop has only existed on unreleased development branches in ASF svn. As
such, removing new features from the working copy is still allowed. As
Hadoop is gearing up for a new release, now is the time to consider whether
side-projects like this belong in the same umbrella project.

Others can -1 the removal patch and force a copy of Sqoop to remain in ASF.
This will force Sqoop to be bundled with the impending Hadoop 0.21 release.
However, I do not intend to rename Sqoop. I also intend to do all feature
and bugfix development in the new repository on github. I will be monitoring
the issue tracker on github for bug reports and feature requests. For
someone else to seriously -1 MAPREDUCE-1644, they'd need to be willing to
fix bugs in the ASF copy themselves, or cross-port the patches I develop at
github and graft them on to the ASF copy.

If others are interested in contributing to Sqoop and would like to take on
a role in the project, I welcome them to come help me out at github, rather
than force a true fork to occur and work within MapReduce svn. If enough
people want to work on Sqoop but feel strongly that we should remain in the
ASF (e.g., by introducing a new project in the incubator), I'm certainly
open to listening to that point of view. But that's a separate discussion
from this one.


>
> > While we've changed hosts, Sqoop will keep the same license -- future
> > improvements will continue to remain Apache 2.0-licensed. We welcome the
> > contributions of all in the open source community; there's a lot of
> exciting
> > work still to be done! If you'd like to help out but aren't sure where to
> > start, send me an email and I can recommend a few areas where
> improvements
> > would be appreciated.
>
> Who is "we" in this case? The same "we" as above?
>
>
Indeed.


> @Hadoop PMC: What is your statement on this fork announcement?
>
> Thanks for clarifying,
>
>  Bernd
>

Regards,
- Aaron Kimball

--0016363b8be89477de0483084d90--

From general-return-1268-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 30 17:58:59 2010
Return-Path: <general-return-1268-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68929 invoked from network); 30 Mar 2010 17:58:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Mar 2010 17:58:59 -0000
Received: (qmail 96444 invoked by uid 500); 30 Mar 2010 17:58:58 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 96373 invoked by uid 500); 30 Mar 2010 17:58:58 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 96365 invoked by uid 99); 30 Mar 2010 17:58:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Mar 2010 17:58:58 +0000
X-ASF-Spam-Status: No, hits=0.3 required=10.0
	tests=AWL,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [209.85.212.48] (HELO mail-vw0-f48.google.com) (209.85.212.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Mar 2010 17:58:53 +0000
Received: by vws14 with SMTP id 14so5129737vws.35
        for <general@hadoop.apache.org>; Tue, 30 Mar 2010 10:58:32 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.229.80.149 with HTTP; Tue, 30 Mar 2010 10:58:12 -0700 (PDT)
In-Reply-To: <88f6e29a1003300754m1826b4f0ve664df8e6b270ae0@mail.gmail.com>
References: <d6d7c4411003291202l3a199257y13cef504070b9d38@mail.gmail.com> 
	<88f6e29a1003300155v1a553afcy59b2a5b7a34fbf5@mail.gmail.com> 
	<5f7f740b1003300648u6a9bbb5fj9a2e4eddbe737b8c@mail.gmail.com> 
	<88f6e29a1003300754m1826b4f0ve664df8e6b270ae0@mail.gmail.com>
From: Aaron Kimball <aaron@cloudera.com>
Date: Tue, 30 Mar 2010 10:58:12 -0700
Received: by 10.229.192.142 with SMTP id dq14mr1123412qcb.81.1269971912139; 
	Tue, 30 Mar 2010 10:58:32 -0700 (PDT)
Message-ID: <d6d7c4411003301058w715d3161k62c2990e2458cd84@mail.gmail.com>
Subject: Re: Sqoop is moving to github!
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636285362b2a5990483086016

--001636285362b2a5990483086016
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Mar 30, 2010 at 7:54 AM, Bernd Fondermann <
bernd.fondermann@googlemail.com> wrote:

> On Tue, Mar 30, 2010 at 15:48, Owen O'Malley <owen.omalley@gmail.com>
> wrote:
> > On Tue, Mar 30, 2010 at 1:55 AM, Bernd Fondermann <
> > bernd.fondermann@googlemail.com> wrote:
> >
> >>
> >> @Hadoop PMC: What is your statement on this fork announcement?
> >
> >
> > Aaron has been the primary developer of Sqoop. He has asked for it to be
> > removed from MapReduce's contrib. If a committer -1's the code removal,
> it
> > will cause a fork. But that hasn't happened and I don't think it will.
>
> It's not the contributor's code alone anymore, once it's been committed.
> Where has this been discussed? Where's the PMC-level vote/record of
> consensus to ratify to abandon the code?
> (ML + TS will be sufficient and I'll find my way.)
>
>
I believe that technically the discussion (such as it is) on MAPREDUCE-1644
will stand as the record of vote. It's still open for new comments.


> > In
> > order for it to make sense to keep the code in Hadoop, someone needs to
> be
> > working on it and the only person working on it is Aaron. If someone
> wants
> > to make a case for keeping Sqoop in MapReduce, please make it in the
> jira:
> > https://issues.apache.org/jira/browse/MAPREDUCE-1644
>
> Well, at least someone reviewed and committed all the contributions to
> svn in the first place, he can't be the only one working on it.
>
>
Indeed. Hadoop committers have checked in all my contributions; Tom White
has done the lion's share of this work.



>  > The sub-projects must avoid circular dependencies. We must be able to
> build
> > the sub-projects in order without having to go back and update a previous
> > project. Making a component of MapReduce depend on Hive (or Pig) would
> cause
> > that kind of cycle. Some projects like Zebra just live in the upper
> project
> > (Pig in that case). The other option is to ask to be a sub-project, ask
> to
> > join Apache incubator, or go to Github. Github doesn't give him the legal
> or
> > community aspects of Apache, but it also doesn't require asking
> permission
> > or introduce any process requirements.
>
> Well, I'm not looking at this from the contributor's perspective here.
> He can do with the code whatever he likes, basically.
> It's only the PMC's viewpoint which is interesting for me.
>
>  Bernd
>

- Aaron

--001636285362b2a5990483086016--

From general-return-1269-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 30 18:08:05 2010
Return-Path: <general-return-1269-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 72330 invoked from network); 30 Mar 2010 18:08:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Mar 2010 18:08:05 -0000
Received: (qmail 17055 invoked by uid 500); 30 Mar 2010 18:08:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17002 invoked by uid 500); 30 Mar 2010 18:08:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 16994 invoked by uid 99); 30 Mar 2010 18:08:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Mar 2010 18:08:04 +0000
X-ASF-Spam-Status: No, hits=0.4 required=10.0
	tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of owen.omalley@gmail.com designates 209.85.222.177 as permitted sender)
Received: from [209.85.222.177] (HELO mail-pz0-f177.google.com) (209.85.222.177)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Mar 2010 18:07:58 +0000
Received: by pzk7 with SMTP id 7so114836pzk.30
        for <general@hadoop.apache.org>; Tue, 30 Mar 2010 11:07:38 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type;
        bh=g9DtwsbVFco4IzPEtdTq9S/DYPQbzwGGVGVlA4NU8Uo=;
        b=URg8ruSidpUHQ/aYPxEbCpdGC7NrQj9jFs6/ean0uWzFiwxArNAauCwqt7FZ3yRqsV
         RObB/H34a+cSXiE0mh5YfscEw3CJKzn3JFFHJNwpj+PahANGRUjiu3knGPAIVpsQJMq1
         AOY6XiDUccupLead5kSO1UtF2aKFTHhW/k6lE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=eQps9MAHaFwjSCIbhKElCXuCn3D9a/b+xDULJ+OLfbG6QiZ7uqU5Ah9My2kT6CJ8YT
         9Lr6WiR7Lp4bfXExZOzfltj1GXQyzMeWoVVXZmOvvrHkZ4yv9lQiX1Ce+QzHeO+tr7pR
         l7MfeK6oGaSLGh4fsp56dUJ2xG0U3XdTuEFS0=
MIME-Version: 1.0
Received: by 10.231.168.20 with HTTP; Tue, 30 Mar 2010 11:07:36 -0700 (PDT)
In-Reply-To: <88f6e29a1003300754m1826b4f0ve664df8e6b270ae0@mail.gmail.com>
References: <d6d7c4411003291202l3a199257y13cef504070b9d38@mail.gmail.com>
	 <88f6e29a1003300155v1a553afcy59b2a5b7a34fbf5@mail.gmail.com>
	 <5f7f740b1003300648u6a9bbb5fj9a2e4eddbe737b8c@mail.gmail.com>
	 <88f6e29a1003300754m1826b4f0ve664df8e6b270ae0@mail.gmail.com>
Date: Tue, 30 Mar 2010 11:07:36 -0700
Received: by 10.141.22.16 with SMTP id z16mr1237376rvi.20.1269972456751; Tue, 
	30 Mar 2010 11:07:36 -0700 (PDT)
Message-ID: <5f7f740b1003301107p6d25ec27k69acc27286fec8c7@mail.gmail.com>
Subject: Re: Sqoop is moving to github!
From: "Owen O'Malley" <owen.omalley@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0cd1855028c4bf0483088183

--000e0cd1855028c4bf0483088183
Content-Type: text/plain; charset=UTF-8

On Tue, Mar 30, 2010 at 7:54 AM, Bernd Fondermann <
bernd.fondermann@googlemail.com> wrote:

Where has this been discussed? Where's the PMC-level vote/record of
> consensus to ratify to abandon the code?
>

The (lack of) discussion is in the jira and this thread. In Hadoop, we treat
adding and removing contrib projects as simple code additions or removals
that use lazy consensus among the committers. If someone wants to block it,
the previously mentioned MAPREDUCE-1644 is the proper avenue. So far, no one
in the project has vetoed the removal.

-- Owen

--000e0cd1855028c4bf0483088183--

From general-return-1270-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 30 19:22:34 2010
Return-Path: <general-return-1270-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 94572 invoked from network); 30 Mar 2010 19:22:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Mar 2010 19:22:34 -0000
Received: (qmail 18506 invoked by uid 500); 30 Mar 2010 19:22:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 18382 invoked by uid 500); 30 Mar 2010 19:22:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 18374 invoked by uid 99); 30 Mar 2010 19:22:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Mar 2010 19:22:33 +0000
X-ASF-Spam-Status: No, hits=0.2 required=10.0
	tests=AWL,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of bernd.fondermann@googlemail.com designates 74.125.78.24 as permitted sender)
Received: from [74.125.78.24] (HELO ey-out-2122.google.com) (74.125.78.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Mar 2010 19:22:28 +0000
Received: by ey-out-2122.google.com with SMTP id 4so1037960eyf.23
        for <general@hadoop.apache.org>; Tue, 30 Mar 2010 12:22:07 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=googlemail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:received:message-id:subject:from:to:content-type;
        bh=R0MvOgl/veFaeudBnOfKuvroaopAbzy/ViwrqKYaAKE=;
        b=PwNY/FVKXyEzSbi8iHu3LmYpKwNO4W1KrfcrrPAevctyrUYTYJy8KSnauwLiCmWA29
         g4vrG9Oblcom4XDFanWJcRA7y2Ye5g7A5lzUPbzvnXUzOnmybh1LQiUoMk4fcEe9duPT
         /SWTO7xeCOqU9cOUXtv/G1e4Dn3iXA30JNDV4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=uMC9cpSM09dsOQd1uiRa8sAX6SZ2skN1By4SORxd9yzEHMC8LZApjhbR7OJLv+4m1M
         jz584Fv2wri4LII8tiBoYBRC3dF1uI7RO2qJCvAtun6TozqtnHXp8/0E+YzBEgFK56Is
         mFokCiluqHtawiM3pu9SM18Cr+GYyejDkKoNk=
MIME-Version: 1.0
Received: by 10.216.16.35 with HTTP; Tue, 30 Mar 2010 12:22:07 -0700 (PDT)
In-Reply-To: <d6d7c4411003301058w715d3161k62c2990e2458cd84@mail.gmail.com>
References: <d6d7c4411003291202l3a199257y13cef504070b9d38@mail.gmail.com>
	 <88f6e29a1003300155v1a553afcy59b2a5b7a34fbf5@mail.gmail.com>
	 <5f7f740b1003300648u6a9bbb5fj9a2e4eddbe737b8c@mail.gmail.com>
	 <88f6e29a1003300754m1826b4f0ve664df8e6b270ae0@mail.gmail.com>
	 <d6d7c4411003301058w715d3161k62c2990e2458cd84@mail.gmail.com>
Date: Tue, 30 Mar 2010 21:22:07 +0200
Received: by 10.216.90.9 with SMTP id d9mr1820515wef.95.1269976927381; Tue, 30 
	Mar 2010 12:22:07 -0700 (PDT)
Message-ID: <88f6e29a1003301222u2ccac962p6a0cbe03b8544930@mail.gmail.com>
Subject: Re: Sqoop is moving to github!
From: Bernd Fondermann <bernd.fondermann@googlemail.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Mar 30, 2010 at 19:58, Aaron Kimball <aaron@cloudera.com> wrote:
> On Tue, Mar 30, 2010 at 7:54 AM, Bernd Fondermann <
> bernd.fondermann@googlemail.com> wrote:
>
>> On Tue, Mar 30, 2010 at 15:48, Owen O'Malley <owen.omalley@gmail.com>
>> wrote:
>> > On Tue, Mar 30, 2010 at 1:55 AM, Bernd Fondermann <
>> > bernd.fondermann@googlemail.com> wrote:
>> >
>> >>
>> >> @Hadoop PMC: What is your statement on this fork announcement?
>> >
>> >
>> > Aaron has been the primary developer of Sqoop. He has asked for it to be
>> > removed from MapReduce's contrib. If a committer -1's the code removal,
>> it
>> > will cause a fork. But that hasn't happened and I don't think it will.
>>
>> It's not the contributor's code alone anymore, once it's been committed.
>> Where has this been discussed? Where's the PMC-level vote/record of
>> consensus to ratify to abandon the code?
>> (ML + TS will be sufficient and I'll find my way.)
>>
>>
> I believe that technically the discussion (such as it is) on MAPREDUCE-1644
> will stand as the record of vote. It's still open for new comments.

No, I don't think so. This isn't the record of anything else than your
intentions.
But that's shouldn't be your concern.

  Bernd

From general-return-1271-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 30 21:32:34 2010
Return-Path: <general-return-1271-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 24104 invoked from network); 30 Mar 2010 21:32:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Mar 2010 21:32:34 -0000
Received: (qmail 76847 invoked by uid 500); 30 Mar 2010 21:32:33 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 76684 invoked by uid 500); 30 Mar 2010 21:32:33 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 76676 invoked by uid 99); 30 Mar 2010 21:32:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Mar 2010 21:32:33 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of awittenauer@linkedin.com designates 69.28.149.24 as permitted sender)
Received: from [69.28.149.24] (HELO esv4-mav02.corp.linkedin.com) (69.28.149.24)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Mar 2010 21:32:28 +0000
DomainKey-Signature: s=prod; d=linkedin.com; c=nofws; q=dns;
  h=X-IronPort-AV:Received:Received:Received:From:To:CC:
   Subject:Thread-Topic:Thread-Index:Date:Message-ID:
   Accept-Language:Content-Language:X-MS-Has-Attach:
   X-MS-TNEF-Correlator:user-agent:Content-Type:Content-ID:
   Content-Transfer-Encoding:MIME-Version:Return-Path:
   X-OriginalArrivalTime;
  b=f8YCdvMQXsp/jVTwwox3kkohRow06dGtI5iN0xTInwUHWK0C8dVw3e1f
   VsfPgQa5tPz3p8ORH7xtnYZTZRlAl4M1toL6ukTWOlSTqoCbkAGrbKmJz
   PluqMtnuNqi7tJX;
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;
  d=linkedin.com; i=awittenauer@linkedin.com; q=dns/txt;
  s=proddkim; t=1269984748; x=1301520748;
  h=from:sender:reply-to:subject:date:message-id:to:cc:
   mime-version:content-transfer-encoding:content-id:
   content-description:resent-date:resent-from:resent-sender:
   resent-to:resent-cc:resent-message-id:in-reply-to:
   references:list-id:list-help:list-unsubscribe:
   list-subscribe:list-post:list-owner:list-archive;
  z=From:=20Allen=20Wittenauer=20<awittenauer@linkedin.com>
   |Subject:=20[ANNOUNCEMENT]=20Azkaban=20is=20now=20open=20
   source!|Date:=20Tue,=2030=20Mar=202010=2021:31:06=20+0000
   |Message-ID:=20<C7D7BBAA.3E97%awittenauer@linkedin.com>
   |To:=20"general@hadoop.apache.org"=20<general@hadoop.apac
   he.org>|CC:=20"azkaban-dev@googlegroups.com"=20<azkaban-d
   ev@googlegroups.com>|MIME-Version:=201.0
   |Content-Transfer-Encoding:=20quoted-printable
   |Content-ID:=20<94be957b-4f74-41fe-bf09-46fa3041e0ac>;
  bh=D+9uv+InQWpNznPw1wzOaClRspL8MkmlGAHg2vFWmys=;
  b=NdMq0SYNF5D6/ftXFOWgEBw51hla9sQ8vYJhXjcKA4GRjW8JhRaZsEss
   dVJdxFazdGgzaNiIG4KcuBYW5n+wfJgD+9cYyRoaHSNun0Im2LNJfgUBF
   vkbi7gAEFL9EvzN;
X-IronPort-AV: E=Sophos;i="4.51,336,1267430400"; 
   d="scan'208";a="12038084"
Received: from esv4-exctest.linkedin.biz ([172.18.46.60]) by CORP-MAIL.linkedin.biz with Microsoft SMTPSVC(6.0.3790.3959);
	 Tue, 30 Mar 2010 14:31:08 -0700
Received: from ESV4-CAS02.linkedin.biz (172.18.46.142) by
 esv4-exctest.linkedin.biz (172.18.46.60) with Microsoft SMTP Server (TLS) id
 14.0.682.1; Tue, 30 Mar 2010 14:31:08 -0700
Received: from ESV4-EXC01.linkedin.biz ([fe80::d7c:dc04:aea1:97d7]) by
 esv4-cas02.linkedin.biz ([172.18.46.142]) with mapi; Tue, 30 Mar 2010
 14:31:08 -0700
From: Allen Wittenauer <awittenauer@linkedin.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
CC: "azkaban-dev@googlegroups.com" <azkaban-dev@googlegroups.com>
Subject: [ANNOUNCEMENT] Azkaban is now open source!
Thread-Topic: [ANNOUNCEMENT] Azkaban is now open source!
Thread-Index: AcrQUE4dHU2SOcm3LEiKRPBpvGZKOA==
Date: Tue, 30 Mar 2010 21:31:06 +0000
Message-ID: <C7D7BBAA.3E97%awittenauer@linkedin.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-Entourage/13.4.0.100208
Content-Type: text/plain; charset="us-ascii"
Content-ID: <94be957b-4f74-41fe-bf09-46fa3041e0ac>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginalArrivalTime: 30 Mar 2010 21:31:08.0662 (UTC) FILETIME=[4FB3B960:01CAD050]
X-Virus-Checked: Checked by ClamAV on apache.org



    Many of you that attended the Bay Area HUG meeting in January* wanted t=
o
know if and when LinkedIn would be open sourcing our simple batch scheduler
for Hadoop.  I'm pleased to announce that, yes, Azkaban is now available
under the Apache License!

        http://sna-projects.com/azkaban/

    For those that aren't familiar, Azkaban is LinkedIn's solution to
running multiple, scheduled batch jobs (workflows) against our Hadoop grids=
.
It supports dependency chains, a web UI, job upload capability, Pig,
external-to-Hadoop steps, and more.

    While the project is still in its infancy, we have been using it
successfully for a variety of uses for about 9 months.  It drives some of
our most complex job flows and manages our ETL processes.

    We look forward to any questions/comments/bugs/features/patches you
might have.=20

* - slides and video of the January HUG are available here:

    http://bit.ly/4Wjf6f

<http://developer.yahoo.net/blogs/hadoop/2010/01/hadoop_bay_area_january_20=
1
0_u.html>


From general-return-1272-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 30 23:51:50 2010
Return-Path: <general-return-1272-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 57100 invoked from network); 30 Mar 2010 23:51:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Mar 2010 23:51:50 -0000
Received: (qmail 17330 invoked by uid 500); 30 Mar 2010 23:51:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 17257 invoked by uid 500); 30 Mar 2010 23:51:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 17249 invoked by uid 99); 30 Mar 2010 23:51:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Mar 2010 23:51:49 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [140.211.11.9] (HELO minotaur.apache.org) (140.211.11.9)
    by apache.org (qpsmtpd/0.29) with SMTP; Tue, 30 Mar 2010 23:51:47 +0000
Received: (qmail 57068 invoked by uid 99); 30 Mar 2010 23:51:25 -0000
Received: from localhost.apache.org (HELO [192.168.168.100]) (127.0.0.1)
  (smtp-auth username cutting, mechanism plain)
  by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Mar 2010 23:51:25 +0000
Message-ID: <4BB28E7C.90300@apache.org>
Date: Tue, 30 Mar 2010 16:51:24 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.24 (X11/20100317)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: [DISCUSS] Avro as TLP?
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

The Avro subproject has voted to become a TLP:

http://mail-archives.apache.org/mod_mbox/hadoop-avro-dev/201003.mbox/%3C4BAA70A6.10907@apache.org%3E

Does the Hadoop community have any questions or concerns about this 
proposal?

Please don't vote yet in response to this.  I'll call a formal vote 
after questions, if any, have been resolved.

Thanks,

Doug

From general-return-1273-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 31 07:50:56 2010
Return-Path: <general-return-1273-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 40003 invoked from network); 31 Mar 2010 07:50:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Mar 2010 07:50:55 -0000
Received: (qmail 29825 invoked by uid 500); 31 Mar 2010 07:50:44 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 27750 invoked by uid 500); 31 Mar 2010 07:50:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 27606 invoked by uid 99); 31 Mar 2010 07:50:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 31 Mar 2010 07:50:40 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [88.198.2.104] (HELO koch.ro) (88.198.2.104)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 31 Mar 2010 07:50:33 +0000
Received: from 84-72-85-88.dclient.hispeed.ch ([84.72.85.88] helo=jona.localnet)
	by koch.ro with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
	(Exim 4.69)
	(envelope-from <thomas@koch.ro>)
	id 1Nwscy-0007mq-5h; Wed, 31 Mar 2010 09:46:16 +0200
From: Thomas Koch <thomas@koch.ro>
Reply-To: thomas@koch.ro
To: common-user@hadoop.apache.org,
 general@hadoop.apache.org,
 "hbase-user@hadoop.apache.org" <hbase-user@hadoop.apache.org>,
 java-user@lucene.apache.org,
 java-dev@lucene.apache.org,
 solr-dev@lucene.apache.org,
 solr-user@lucene.apache.org,
 zookeeper-dev@hadoop.apache.org,
 zookeeper-user@hadoop.apache.org
Subject: [ANN] Eclipse GIT plugin beta version released
Date: Wed, 31 Mar 2010 09:50:02 +0200
User-Agent: KMail/1.12.4 (Linux/2.6.32-2-amd64; KDE/4.3.4; x86_64; ; )
MIME-Version: 1.0
Content-Type: text/plain;
  charset="us-ascii"
Content-Transfer-Encoding: 7bit
Message-Id: <201003310950.03579.thomas@koch.ro>
X-Virus-Checked: Checked by ClamAV on apache.org

GIT is one of the most popular distributed version control system. 
In the hope, that more Java developers may want to explore the world of easy 
branching, merging and patch management, I'd like to inform you, that a beta 
version of the upcoming Eclipse GIT plugin is available:

http://www.infoq.com/news/2010/03/egit-released
http://aniszczyk.org/2010/03/22/the-start-of-an-adventure-egitjgit-0-7-1/

Maybe, one day, some apache / hadoop projects will use GIT... :-)

(Yes, I know git.apache.org.)

Best regards,

Thomas Koch, http://www.koch.ro

From general-return-1274-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 31 20:07:15 2010
Return-Path: <general-return-1274-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 38645 invoked from network); 31 Mar 2010 20:07:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Mar 2010 20:07:14 -0000
Received: (qmail 26622 invoked by uid 500); 31 Mar 2010 20:07:14 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 26456 invoked by uid 500); 31 Mar 2010 20:07:13 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 26448 invoked by uid 99); 31 Mar 2010 20:07:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 31 Mar 2010 20:07:13 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of mona@apture.com designates 67.221.235.209 as permitted sender)
Received: from [67.221.235.209] (HELO web06.apture.com) (67.221.235.209)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 31 Mar 2010 20:07:04 +0000
Received: (qmail 26555 invoked by uid 89); 31 Mar 2010 20:06:43 -0000
Received: by simscan 1.1.0 ppid: 26549, pid: 26551, t: 0.9966s
         scanners: regex: 1.1.0 spam: 3.2.5
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
	morpheus.contegix.com
X-Spam-Level: *
Received: from unknown (HELO ?192.168.42.165?) (mona@apture.com@71.133.225.222)
  by 67-221-235-209.contegix.com with ESMTPA; 31 Mar 2010 20:06:42 -0000
From: Mona Gandhi <mona@apture.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: Q about use of avro logs with hadoop streaming
Date: Wed, 31 Mar 2010 13:06:40 -0700
Message-Id: <859D76D2-50AE-4E66-974A-3B47621323F6@apture.com>
To: general@hadoop.apache.org
Mime-Version: 1.0 (Apple Message framework v1077)
X-Mailer: Apple Mail (2.1077)
X-Virus-Checked: Checked by ClamAV on apache.org
X-Old-Spam-Status: No, score=1.9 required=10.0 tests=ALL_TRUSTED,FH_DATE_PAST_20XX
	autolearn=no version=3.2.5

i currently use avro version 1.3.0 to log data. I am having difficulty =
processing these avro logs via a map reduce job written in Python using =
hadoop streaming(v 0.21.0).=20

Any help here is much appreciated.

Thanks,
Mona




From general-return-1275-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 31 22:21:46 2010
Return-Path: <general-return-1275-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 79123 invoked from network); 31 Mar 2010 22:21:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Mar 2010 22:21:46 -0000
Received: (qmail 59562 invoked by uid 500); 31 Mar 2010 22:21:45 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 59508 invoked by uid 500); 31 Mar 2010 22:21:45 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 59500 invoked by uid 99); 31 Mar 2010 22:21:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 31 Mar 2010 22:21:45 +0000
X-ASF-Spam-Status: No, hits=0.3 required=10.0
	tests=AWL,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [76.96.59.243] (HELO qmta13.westchester.pa.mail.comcast.net) (76.96.59.243)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 31 Mar 2010 22:21:37 +0000
Received: from omta17.westchester.pa.mail.comcast.net ([76.96.62.89])
	by qmta13.westchester.pa.mail.comcast.net with comcast
	id zmxt1d0041vXlb85DyMG29; Wed, 31 Mar 2010 22:21:16 +0000
Received: from wlan-snve-152-239.corp.yahoo.com ([209.131.62.115])
	by omta17.westchester.pa.mail.comcast.net with comcast
	id zyRK1d00N2VBGtd3dyRPaD; Wed, 31 Mar 2010 22:25:27 +0000
Message-Id: <FAC2B17D-7A3F-4157-AEC4-E600BCDB5E49@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v936)
Subject: Welcome Namit Jain to the Hadoop PMC
Date: Wed, 31 Mar 2010 15:21:06 -0700
X-Mailer: Apple Mail (2.936)

The Hadoop PMC has voted to add Namit Jain to the Hadoop PMC.  
Congratulations, Namit!

-- Owen

From general-return-103-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 08 20:10:52 2009
Return-Path: <general-return-103-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 37506 invoked from network); 8 Jan 2009 20:10:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 8 Jan 2009 20:10:52 -0000
Received: (qmail 54125 invoked by uid 500); 8 Jan 2009 20:10:41 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 53984 invoked by uid 500); 8 Jan 2009 20:10:41 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 53867 invoked by uid 99); 8 Jan 2009 20:10:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jan 2009 12:10:41 -0800
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jan 2009 20:10:31 +0000
Received: from SNV-EXBH01.ds.corp.yahoo.com (snv-exbh01.ds.corp.yahoo.com [207.126.227.249])
	by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n08KA0ju064183;
	Thu, 8 Jan 2009 12:10:00 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:x-mimeole:content-class:mime-version:
	content-type:subject:date:message-id:x-ms-has-attach:
	x-ms-tnef-correlator:thread-topic:thread-index:from:to:return-path:x-originalarrivaltime;
	b=jSJ/BpSMD2KQOua1bcN0K69I19oYYLvws0NqElGV/vZHG7Yr6ywBLPGagZ6nEejD
Received: from SNV-EXVS02.ds.corp.yahoo.com ([216.145.51.196]) by SNV-EXBH01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Thu, 8 Jan 2009 12:10:00 -0800
x-mimeole: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C971CD.15AE268E"
Subject: Hadoop User Group Meeting (Bay Area) 1/21
Date: Thu, 8 Jan 2009 12:10:00 -0800
Message-ID: <DD27C9769EA63D43BE3CF4CCCC12DD970200B843@SNV-EXVS02.ds.corp.yahoo.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Hadoop User Group Meeting (Bay Area) 1/21
Thread-Index: AclxzRWJr3aRXeNpT8mNevp3DrZJWg==
From: "Ajay Anand" <aanand@yahoo-inc.com>
To: "core-user@hadoop.apache.org" <'core-user@hadoop.apache.org'>,
        "general@hadoop.apache.org" <'general@hadoop.apache.org'>,
        "zookeeper-user@hadoop.apache.org" <'zookeeper-user@hadoop.apache.org'>,
        "hbase-user@hadoop.apache.org" <'hbase-user@hadoop.apache.org'>,
        <pig-user@hadoop.apache.org>
X-OriginalArrivalTime: 08 Jan 2009 20:10:00.0614 (UTC) FILETIME=[15E20C60:01C971CD]
X-Virus-Checked: Checked by ClamAV on apache.org

------_=_NextPart_001_01C971CD.15AE268E
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

The next Bay Area Hadoop User Group meeting is scheduled for Wednesday,
January 21st at Yahoo! 2821 Mission College Blvd, Santa Clara, Building
1, Training Rooms 3 & 4 from 6:00-7:30 pm.

=20

Agenda:

Hadoop 0.20 Overview - Sameer Paranjpye

Hadoop 1.0 discussion - Sanjay Radia.

=20

Please send me any other suggestions for topics for future meetings.

=20

Registration: http://upcoming.yahoo.com/event/1478614

=20

Look forward to seeing you there!

Ajay

=20

=20


------_=_NextPart_001_01C971CD.15AE268E--

From general-return-104-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Jan 12 15:24:01 2009
Return-Path: <general-return-104-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 61723 invoked from network); 12 Jan 2009 15:24:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 12 Jan 2009 15:24:00 -0000
Received: (qmail 60415 invoked by uid 500); 12 Jan 2009 15:23:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 60338 invoked by uid 500); 12 Jan 2009 15:23:55 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Delivered-To: moderator for general@hadoop.apache.org
Received: (qmail 48510 invoked by uid 99); 12 Jan 2009 11:13:51 -0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of hbjerryzl@gmail.com designates 209.85.142.188 as permitted sender)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from
         :user-agent:mime-version:to:subject:content-type
         :content-transfer-encoding;
        bh=syRpSbpQxM67/xeFqyl0yp4mFBBd0hX8fph3TnYz8/E=;
        b=rsbguJ0YddHbgy5+k5OJHILc46VNBUjuS+4D7/qXxV7OycOqgTUR8LPsjJg3iSPrO/
         rHhajK8qdtbn+rtWw8VdfS4y8ZBXGddAaNpnQxaHd6nEYPlwv1bcH2W9w0CxZP+yd1/Z
         l07dhRX4TqH4UULg+cLrzksa5e0njOqw5iFBU=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject
         :content-type:content-transfer-encoding;
        b=vp8/gHFll/eqsAhd5bU0FgWzlTevbX5bXnt79tuzorH32ZTu7yfYNzOLDS6cRpOLKV
         uQXaMRSlDzQ0FHkgcze+3LwUz2kYQW9PJaRO1Z5TDu0627XTBjp89sNMr/8AZAgpFvLD
         Vm9vW8znaExC+CJOIJiWo3fUKB2/Jo+dFzjAo=
Message-ID: <496B25D3.6030504@gmail.com>
Date: Mon, 12 Jan 2009 19:13:23 +0800
From: jerry <hbjerryzl@gmail.com>
User-Agent: Thunderbird 2.0.0.19 (Windows/20081209)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: libhdfs  skipping incompatible jvm
Content-Type: text/plain; charset=GB2312
Content-Transfer-Encoding: 8bit
X-Virus-Checked: Checked by ClamAV on apache.org

my OS is Linux 2.6.18-92.el5xen #1 SMP Tue Jun 10 19:20:18 EDT 2008
x86_64 x86_64 x86_64 GNU/Linux (64bit)

my java version is jdk-6u11-linux-x64-rpm.bin (64bit)

i change the "-m32" config to "-m64".
when i build the "libhdfs", it said that:

*compile-libhdfs:
[exec] gcc -L/home/jerry/jdk1.6.0_12/jre/lib/i386/server -ljvm -shared
-m64 -o /home/jerry/hadoop-0.19.0/build/libhdfs/libhdfs.so.1
-Wl,-soname,libhdfs.so /home/jerry/hadoop-0.19.0/build/libhdfs/hdfs.o
/home/jerry/hadoop-0.19.0/build/libhdfs/hdfsJniHelper.o \
[exec] && ln -sf /home/jerry/hadoop-0.19.0/build/libhdfs/libhdfs.so.1
/home/jerry/hadoop-0.19.0/build/libhdfs/libhdfs.so
[exec] /usr/bin/ld: skipping incompatible
/home/jerry/jdk1.6.0_12/jre/lib/i386/server/libjvm.so when searching for
-ljvm
[exec] /usr/bin/ld: cannot find -ljvm
[exec] collect2: ld ·µ»Ø 1
[exec] make: *** [/home/jerry/hadoop-0.19.0/build/libhdfs/libhdfs.so.1]


is this a bug?
*

From general-return-105-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 13 00:19:50 2009
Return-Path: <general-return-105-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 1122 invoked from network); 13 Jan 2009 00:19:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 13 Jan 2009 00:19:49 -0000
Received: (qmail 82478 invoked by uid 500); 13 Jan 2009 00:19:49 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 82459 invoked by uid 500); 13 Jan 2009 00:19:49 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 82448 invoked by uid 99); 13 Jan 2009 00:19:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jan 2009 16:19:49 -0800
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received-SPF: unknown amx-all (athena.apache.org: encountered unrecognized mechanism during SPF processing of domain of tim.hawkins@bejant.com)
Received: from [209.85.219.21] (HELO mail-ew0-f21.google.com) (209.85.219.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jan 2009 00:19:41 +0000
Received: by ewy14 with SMTP id 14so11995041ewy.5
        for <general@hadoop.apache.org>; Mon, 12 Jan 2009 16:19:18 -0800 (PST)
Received: by 10.210.90.20 with SMTP id n20mr4392569ebb.198.1231805958571;
        Mon, 12 Jan 2009 16:19:18 -0800 (PST)
Received: from MainMacbook.lan (thawkins-adsl.demon.co.uk [80.176.168.49])
        by mx.google.com with ESMTPS id g9sm692846gvc.17.2009.01.12.16.19.17
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Mon, 12 Jan 2009 16:19:17 -0800 (PST)
Message-Id: <E3B392DF-9F38-4686-9859-0FC27202C146@bejant.com>
From: Tim Hawkins <tim.hawkins@bejant.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: New User Questions
Date: Tue, 13 Jan 2009 00:19:16 +0000
X-Mailer: Apple Mail (2.930.3)
X-Virus-Checked: Checked by ClamAV on apache.org

I have just recently started working with hadoop and I have a few  
questions I would like to submit to the community.

1. Is there any remote-able interface to the jobtracker, so that I can  
create an application that would remotely be able to track jobs. Ie  
something like a rest or XMLRPC interface?.

2. Is it possible to attach additional metadata to a job for tracking  
purposes, we have a set of tasks that result in a large number of jobs  
per task, and I would like to tag jobs with an overall task id for  
tracking purposes.

A usage case would be a nutch crawl, our application uses nutch to  
scan domains on demand, each scan is a distinct task, and nutch spawns  
a set of mapred tasks for each stage of the crawl.

I would like to be able to determine completion status for a set of  
nutch crawls using an interface to job-tracker. 

From general-return-106-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 16 15:57:12 2009
Return-Path: <general-return-106-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 42261 invoked from network); 16 Jan 2009 15:57:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 16 Jan 2009 15:57:11 -0000
Received: (qmail 96062 invoked by uid 500); 16 Jan 2009 15:57:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 95876 invoked by uid 500); 16 Jan 2009 15:57:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 95864 invoked by uid 99); 16 Jan 2009 15:57:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jan 2009 07:57:10 -0800
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received-SPF: unknown amx-all (nike.apache.org: encountered unrecognized mechanism during SPF processing of domain of tim.hawkins@bejant.com)
Received: from [209.85.219.21] (HELO mail-ew0-f21.google.com) (209.85.219.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jan 2009 15:57:01 +0000
Received: by ewy14 with SMTP id 14so1950651ewy.5
        for <general@hadoop.apache.org>; Fri, 16 Jan 2009 07:56:41 -0800 (PST)
Received: by 10.210.109.10 with SMTP id h10mr3352537ebc.39.1232121400867;
        Fri, 16 Jan 2009 07:56:40 -0800 (PST)
Received: from ?192.168.1.14? (82-35-84-223.cable.ubr03.dals.blueyonder.co.uk [82.35.84.223])
        by mx.google.com with ESMTPS id b36sm2327425ika.7.2009.01.16.07.56.40
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Fri, 16 Jan 2009 07:56:40 -0800 (PST)
Message-Id: <AA16CB7E-87B2-491A-A9D9-C031A23BE88C@bejant.com>
From: Tim Hawkins <tim.hawkins@bejant.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: Security Groups in Hadoop-ec2
Date: Fri, 16 Jan 2009 15:56:39 +0000
X-Mailer: Apple Mail (2.930.3)
X-Virus-Checked: Checked by ClamAV on apache.org

I have been playing with hadoop-ec2 (src/contrib/ec2) and have found a  
minor problem

When a cluster is launched, it creates two security groups and cross  
links them, ie for cluster-name xxxx it creates EC2 security groups  
xxxx and xxxx-master , the cross-linking is i believe to allow all the  
masters and slaves to be able to communicate with each other.

Unfortunately this means that the security groups are now mutually  
dependant on each other, and the amazon API will no longer allow them  
to be deleted, none of the GUI tools (rightscale, elasticFox, and  
consol.aws) or the command-line tools seem to be able to remove the  
security groups either, presumably because they too are dependant on  
the API.

I believe the solution would be to create 3 security groups, not two,  
xxxx, xxxx-slave and xxxx-master , and only inherit permissions from  
xxxx into the other two, which would achieve the same result but be  
more "friendly". It would also potentially offer the ability for  
multiple clusters to share base security descriptors with other  
subsystems without having to open publicly accessible holes, by  
allowing the name of xxxx to be set independent of the cluster-name,  
allowing it to be shared as a base.

I can make this change to the scripts and test it on our set-up, but  
am not sure how to contribute the changes back to ensure that this  
problem does not effect others.





From general-return-107-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 16 17:45:12 2009
Return-Path: <general-return-107-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 88422 invoked from network); 16 Jan 2009 17:45:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 16 Jan 2009 17:45:12 -0000
Received: (qmail 74794 invoked by uid 500); 16 Jan 2009 17:45:11 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 74775 invoked by uid 500); 16 Jan 2009 17:45:11 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 74764 invoked by uid 99); 16 Jan 2009 17:45:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jan 2009 09:45:11 -0800
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of tom.e.white@gmail.com designates 72.14.220.156 as permitted sender)
Received: from [72.14.220.156] (HELO fg-out-1718.google.com) (72.14.220.156)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jan 2009 17:45:03 +0000
Received: by fg-out-1718.google.com with SMTP id l26so988163fgb.35
        for <general@hadoop.apache.org>; Fri, 16 Jan 2009 09:44:43 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:received:received:message-id:date:from:to
         :subject:in-reply-to:mime-version:content-type
         :content-transfer-encoding:content-disposition:references;
        bh=LM8GFgNKTQ1xbRy2wAUV9C8CMDU4gDxl4eFLiaqv4ww=;
        b=cn631esQvYAdt7MTps47XG9U1ljGwbDmKZ0pac+pUKfNUcAjuLLGzXRzlT8TrLZDoQ
         OUS7DKMVN6O4SXMQhCE8OyNCSvy+d+Y8zgluDnxZbqk1nlH2Zoeia1eLW7bNdjf83ojU
         QezqfshzPUFE7r5oYp1Za29LgTvxWmQYwxJtY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version
         :content-type:content-transfer-encoding:content-disposition
         :references;
        b=JNSm3m9dvLahhp87l6PNAyor2QqIGjq+YSa9IjKp2KtEfUc9VSymWFMv0jBUqIWZmR
         Tk18KDCuJp17KEpqIDTjDx/g298mIqiFNBrf+VfPtY6ba7lbesT7wONH8+Rhnwfx5Rum
         ENkRPM7wBSiBEc94J+5MNoWA33y1QMYZQmpro=
Received: by 10.86.92.9 with SMTP id p9mr534160fgb.15.1232127883445;
        Fri, 16 Jan 2009 09:44:43 -0800 (PST)
Received: by 10.86.62.11 with HTTP; Fri, 16 Jan 2009 09:44:43 -0800 (PST)
Message-ID: <b23e01b50901160944h41557c48n994a7eea0152a87a@mail.gmail.com>
Date: Fri, 16 Jan 2009 17:44:43 +0000
From: "Tom White" <tom.e.white@gmail.com>
To: general@hadoop.apache.org
Subject: Re: Security Groups in Hadoop-ec2
In-Reply-To: <AA16CB7E-87B2-491A-A9D9-C031A23BE88C@bejant.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <AA16CB7E-87B2-491A-A9D9-C031A23BE88C@bejant.com>
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Tim,

You can use ec2-revoke to disassociate the groups from each other,
then delete the groups. In fact, the "hadoop-ec2 delete-cluster"
command does exactly this. Does this solve your problem?

Inheriting from a base group might be a useful enhancement - would you
like to start a Jira for this?

Thanks,

Tom

BTW questions about Hadoop on EC2 are best posted to core-user.

On Fri, Jan 16, 2009 at 3:56 PM, Tim Hawkins <tim.hawkins@bejant.com> wrote:
> I have been playing with hadoop-ec2 (src/contrib/ec2) and have found a minor
> problem
>
> When a cluster is launched, it creates two security groups and cross links
> them, ie for cluster-name xxxx it creates EC2 security groups xxxx and
> xxxx-master , the cross-linking is i believe to allow all the masters and
> slaves to be able to communicate with each other.
>
> Unfortunately this means that the security groups are now mutually dependant
> on each other, and the amazon API will no longer allow them to be deleted,
> none of the GUI tools (rightscale, elasticFox, and consol.aws) or the
> command-line tools seem to be able to remove the security groups either,
> presumably because they too are dependant on the API.
>
> I believe the solution would be to create 3 security groups, not two, xxxx,
> xxxx-slave and xxxx-master , and only inherit permissions from xxxx into the
> other two, which would achieve the same result but be more "friendly". It
> would also potentially offer the ability for multiple clusters to share base
> security descriptors with other subsystems without having to open publicly
> accessible holes, by allowing the name of xxxx to be set independent of the
> cluster-name, allowing it to be shared as a base.
>
> I can make this change to the scripts and test it on our set-up, but am not
> sure how to contribute the changes back to ensure that this problem does not
> effect others.
>
>
>
>
>

From general-return-108-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Wed Jan 21 22:44:19 2009
Return-Path: <general-return-108-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 97781 invoked from network); 21 Jan 2009 22:44:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 21 Jan 2009 22:44:19 -0000
Received: (qmail 2223 invoked by uid 500); 21 Jan 2009 22:44:09 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 1134 invoked by uid 500); 21 Jan 2009 22:44:07 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 1027 invoked by uid 99); 21 Jan 2009 22:44:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jan 2009 14:44:06 -0800
X-ASF-Spam-Status: No, hits=3.4 required=10.0
	tests=HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jan 2009 22:43:55 +0000
Received: from SNV-EXBH01.ds.corp.yahoo.com (snv-exbh01.ds.corp.yahoo.com [207.126.227.249])
	by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n0LMfear035186;
	Wed, 21 Jan 2009 14:41:40 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:x-mimeole:content-class:mime-version:
	content-type:subject:date:message-id:in-reply-to:x-ms-has-attach:
	x-ms-tnef-correlator:thread-topic:thread-index:references:from:to:return-path:x-originalarrivaltime;
	b=EpvR7RUiZY4KFOFSB6xU0gA6sDhFikGLYntDuaXD6H8YsLNZ1uNDlqfEg/AlHfDm
Received: from SNV-EXVS02.ds.corp.yahoo.com ([216.145.51.196]) by SNV-EXBH01.ds.corp.yahoo.com with Microsoft SMTPSVC(6.0.3790.3959);
	 Wed, 21 Jan 2009 14:41:40 -0800
x-mimeole: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C97C19.6C833DD5"
Subject: RE: Hadoop User Group Meeting (Bay Area) 1/21
Date: Wed, 21 Jan 2009 14:41:38 -0800
Message-ID: <DD27C9769EA63D43BE3CF4CCCC12DD970224852C@SNV-EXVS02.ds.corp.yahoo.com>
In-Reply-To: <AD0C16AAE446924586B68FC0EB421C9BAE15439C@SNV-EXVS02.ds.corp.yahoo.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Hadoop User Group Meeting (Bay Area) 1/21
Thread-Index: AclxzRWJr3aRXeNpT8mNevp3DrZJWgKTADdg
References: <AD0C16AAE446924586B68FC0EB421C9BAE15439C@SNV-EXVS02.ds.corp.yahoo.com>
From: "Ajay Anand" <aanand@yahoo-inc.com>
To: "core-user@hadoop.apache.org" <'core-user@hadoop.apache.org'>,
        "general@hadoop.apache.org" <'general@hadoop.apache.org'>,
        "zookeeper-user@hadoop.apache.org" <'zookeeper-user@hadoop.apache.org'>,
        "hbase-user@hadoop.apache.org" <'hbase-user@hadoop.apache.org'>,
        <pig-user@hadoop.apache.org>
X-OriginalArrivalTime: 21 Jan 2009 22:41:40.0407 (UTC) FILETIME=[6D270470:01C97C19]
X-Virus-Checked: Checked by ClamAV on apache.org

------_=_NextPart_001_01C97C19.6C833DD5
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Reminder - the Bay Area Hadoop User Group meeting is today at 6 pm.

=20

________________________________

From: Ajay Anand=20
Sent: Thursday, January 08, 2009 12:10 PM
To: 'core-user@hadoop.apache.org'; 'general@hadoop.apache.org';
'zookeeper-user@hadoop.apache.org'; 'hbase-user@hadoop.apache.org';
pig-user@hadoop.apache.org
Subject: Hadoop User Group Meeting (Bay Area) 1/21

=20

The next Bay Area Hadoop User Group meeting is scheduled for Wednesday,
January 21st at Yahoo! 2821 Mission College Blvd, Santa Clara, Building
1, Training Rooms 3 & 4 from 6:00-7:30 pm.

=20

Agenda:

Hadoop 0.20 Overview - Sameer Paranjpye

Hadoop 1.0 discussion - Sanjay Radia.

=20

Please send me any other suggestions for topics for future meetings.

=20

Registration: http://upcoming.yahoo.com/event/1478614

=20

Look forward to seeing you there!

Ajay

=20

=20


------_=_NextPart_001_01C97C19.6C833DD5--

From general-return-109-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 22 01:30:57 2009
Return-Path: <general-return-109-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@locus.apache.org
Received: (qmail 66263 invoked from network); 22 Jan 2009 01:30:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 22 Jan 2009 01:30:57 -0000
Received: (qmail 94110 invoked by uid 500); 22 Jan 2009 01:30:56 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 94079 invoked by uid 500); 22 Jan 2009 01:30:56 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 94068 invoked by uid 99); 22 Jan 2009 01:30:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jan 2009 17:30:56 -0800
X-ASF-Spam-Status: No, hits=2.2 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of feu.teston@gmail.com designates 209.85.146.177 as permitted sender)
Received: from [209.85.146.177] (HELO wa-out-1112.google.com) (209.85.146.177)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jan 2009 01:30:49 +0000
Received: by wa-out-1112.google.com with SMTP id m38so1169533waf.29
        for <general@hadoop.apache.org>; Wed, 21 Jan 2009 17:30:29 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:received:in-reply-to:references
         :date:message-id:subject:from:to:content-type;
        bh=rzotyO94rV7r7ig3RVKrehkaJUJC3qU0AgK3+0ebF/Y=;
        b=aemWLBIuhpah3h/NyvfeiW4q/0HZ9DMEiceM0EOh1p4CI8MUSjb1I41y/XzdFQV642
         BBSELrCB5j7vUPprCY3b8JVenFmrB+0DCy0BH6WciHmJ5g4OF1sLhUNWaX0i8a6doUJQ
         siny4SePAhlH5bYeWyuwETms6373dX8ERatyA=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=x48wIII38G1WiuQzHjkLoOU2CfuU9dlHgApI+OTAUPwHhJvUS0OGWwOprmlVTpSgC7
         WbE48dmJiE/VCwL52Q/fXFPZett4b77JbtyBgKqB42lCfetrHhEMPZ239kxTxwAjLevi
         vNdgtppeh2YbtnJGINyoS0/9cBrqviJp5c02U=
MIME-Version: 1.0
Received: by 10.114.157.1 with SMTP id f1mr592425wae.221.1232587828710; Wed, 
	21 Jan 2009 17:30:28 -0800 (PST)
In-Reply-To: <DD27C9769EA63D43BE3CF4CCCC12DD970224852C@SNV-EXVS02.ds.corp.yahoo.com>
References: <AD0C16AAE446924586B68FC0EB421C9BAE15439C@SNV-EXVS02.ds.corp.yahoo.com>
	 <DD27C9769EA63D43BE3CF4CCCC12DD970224852C@SNV-EXVS02.ds.corp.yahoo.com>
Date: Wed, 21 Jan 2009 23:30:28 -0200
Message-ID: <11261fa0901211730r4115ebc2wd439381d2673dc55@mail.gmail.com>
Subject: Re: Hadoop User Group Meeting (Bay Area) 1/21
From: Feu Teston <feu.teston@gmail.com>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001636456fd6af4e88046108374f
X-Virus-Checked: Checked by ClamAV on apache.org

--001636456fd6af4e88046108374f
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi Guys,


I wanna know if this meeting will be recorded. I'm living so far so I can't
go to the meeting. But I'll appreciate if you guys can record the meeting
and put this video on youtube after that. I think other people also should
like that.

Thanks,


Feu Teston

2009/1/21 Ajay Anand <aanand@yahoo-inc.com>

> Reminder - the Bay Area Hadoop User Group meeting is today at 6 pm.
>
>
>
> ________________________________
>
> From: Ajay Anand
> Sent: Thursday, January 08, 2009 12:10 PM
> To: 'core-user@hadoop.apache.org'; 'general@hadoop.apache.org';
> 'zookeeper-user@hadoop.apache.org'; 'hbase-user@hadoop.apache.org';
> pig-user@hadoop.apache.org
> Subject: Hadoop User Group Meeting (Bay Area) 1/21
>
>
>
> The next Bay Area Hadoop User Group meeting is scheduled for Wednesday,
> January 21st at Yahoo! 2821 Mission College Blvd, Santa Clara, Building
> 1, Training Rooms 3 & 4 from 6:00-7:30 pm.
>
>
>
> Agenda:
>
> Hadoop 0.20 Overview - Sameer Paranjpye
>
> Hadoop 1.0 discussion - Sanjay Radia.
>
>
>
> Please send me any other suggestions for topics for future meetings.
>
>
>
> Registration: http://upcoming.yahoo.com/event/1478614
>
>
>
> Look forward to seeing you there!
>
> Ajay
>
>
>
>
>
>


-- 
Fernando Teston
http://malditoprogramadorbebado.blogspot.com/

--001636456fd6af4e88046108374f--

From general-return-110-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 27 16:36:20 2009
Return-Path: <general-return-110-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 73724 invoked from network); 27 Jan 2009 16:36:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 27 Jan 2009 16:36:20 -0000
Received: (qmail 4597 invoked by uid 500); 27 Jan 2009 16:36:18 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 4463 invoked by uid 500); 27 Jan 2009 16:36:18 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 4355 invoked by uid 99); 27 Jan 2009 16:36:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jan 2009 08:36:18 -0800
X-ASF-Spam-Status: No, hits=4.2 required=10.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [76.96.62.16] (HELO QMTA01.westchester.pa.mail.comcast.net) (76.96.62.16)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jan 2009 16:36:06 +0000
Received: from OMTA09.westchester.pa.mail.comcast.net ([76.96.62.20])
	by QMTA01.westchester.pa.mail.comcast.net with comcast
	id 8amC1b0030SCNGk51gbJNX; Tue, 27 Jan 2009 16:35:18 +0000
Received: from battlerock-lm.corp.yahoo.com ([209.131.62.113])
	by OMTA09.westchester.pa.mail.comcast.net with comcast
	id 8gbE1b00C2SbwD53VgbHuF; Tue, 27 Jan 2009 16:35:43 +0000
Cc: core-user@hadoop.apache.org,
 core-dev@hadoop.apache.org,
 hive-user@hadoop.apache.org,
 hive-dev@hadoop.apache.org,
 zookeeper-user@hadoop.apache.org,
 zookeeper-dev@yahoo-inc.com,
 pig-user@hadoop.apache.org,
 pig-dev@hadoop.apache.org,
 hbase-dev@hadoop.apache.org,
 hbase-user@hadoop.apache.org
Message-Id: <A7BD0A92-8510-403E-9ECB-8E55CA990D7F@apache.org>
From: Owen O'Malley <omalley@apache.org>
To: general@hadoop.apache.org
Content-Type: multipart/alternative; boundary=Apple-Mail-1--783641243
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: [ANNOUNCE] Registration for ApacheCon Europe 2009 is now open!
Date: Tue, 27 Jan 2009 08:35:12 -0800
References: <497F16FD.3090103@shanecurcuru.org>
X-Mailer: Apple Mail (2.930.3)
X-Virus-Checked: Checked by ClamAV on apache.org

--Apple-Mail-1--783641243
Content-Type: text/plain;
	charset=ISO-8859-1;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: quoted-printable

All,
    I'm broadcasting this to all of the Hadoop dev and users lists, =20
however, in the future I'll only send cross-subproject announcements =20
to general@hadoop.apache.org. Please subscribe over there too! It is =20
very low traffic.
   Anyways, ApacheCon Europe is coming up in March. There are a range =20=

of Hadoop talks being given:

Introduction to Hadoop by Owen O'Malley
Hadoop Map/Reduce: Tuning and Debugging by Arun Murthy
Pig - Making Hadoop Easy by Olga Natkovich
Running Hadoop in the Cloud by Tom White
Architectures for the Cloud by Steve Loughran
Configuring Hadoop for Grid Services by Allen Wittenauer
Dynamic Hadoop Clusters by Steve Loughran
HBasics: An Introduction to Hadoop's Bid Data Database by Michael Stack
Hadoop Tools and Tricks for Data Pipelines by Christophe Bisciglia
Introducing Mahout: Apache Machine Learning by Grant Ingersoll

-- Owen

Begin forwarded message:

> From: Shane Curcuru <asf@shanecurcuru.org>
> Date: January 27, 2009 6:15:25 AM PST
> Subject: [ANN] Registration for ApacheCon Europe 2009 is now open!
>
> PMC moderators - please forward the below to any appropriate dev@ or =20=

> users@ lists so your larger community can hear about ApacheCon =20
> Europe. Remember, ACEU09 has scheduled sessions spanning the breadth =20=

> of the ASF's projects, subprojects, and podlings, including at =20
> least: ActiveMQ, SerivceMix, CXF, Axis2, Hadoop, Felix, Sling, =20
> Maven, Struts, Roller, Shindig, Geronimo, Lucene, Solr, BSF, Mina, =20
> Directory, Tomcat, httpd, Mahout, Bayeux, CouchDB, AntUnit, =20
> Jackrabbit, Archiva, Wicket, POI, Pig, Synapse, Droids, Continuum.
>
>
> ApacheCon EU 2009 registration is now open!
> 23-27 March -- M=F6venpick Hotel, Amsterdam, Netherlands
> http://www.eu.apachecon.com/
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D
>
> Registration for ApacheCon Europe 2009 is now open - act before early
> bird prices expire 6 February.  Remember to book a room at the =20
> M=F6venpick
> and use the Registration Code: Special package attendees for the
> conference registration, and get 150 Euros off your full conference
> registration.
>
> Lower Costs - Thanks to new VAT tax laws, our prices this year are 19%
> lower than last year in Europe!  We've also negotiated a M=F6venpick =20=

> rate
> of a maximum of 155 Euros per night for attendees in our room block.
>
> Quick Links:
>
>   http://xrl.us/aceu09sp  See the schedule
>   http://xrl.us/aceu09hp  Get your hotel room
>   http://xrl.us/aceu09rp  Register for the conference
>
> Other important notes:
>
> - Geeks for Geeks is a new mini-track where we can feature advanced
> technical content from project committers.  And our Hackathon on =20
> Monday
> and Tuesday is open to all attendees - be sure to check it off in your
> registration.
>
> - The Call for Papers for ApacheCon US 2009, held 2-6 November
> 2009 in Oakland, CA, is open through 28 February, so get your
> submissions in now.  This ApacheCon will feature special events with
> some of the ASF's original founders in celebration of the 10th
> anniversary of The Apache Software Foundation.
>
>   http://www.us.apachecon.com/c/acus2009/
>
> - Interested in sponsoring the ApacheCon conferences?  There are =20
> plenty
> of sponsor packages available - please contact Delia Frees at
> delia@apachecon.com for further information.
>
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D
> ApacheCon EU 2008: A week of Open Source at it's best!
>
> Hackathon - open to all! | Geeks for Geeks | Lunchtime Sessions
> In-Depth Trainings | Multi-Track Sessions | BOFs | Business Panel
> Lightning Talks | Receptions | Fast Feather Track | Expo... and more!
>
> - Shane Curcuru, on behalf of
>  Noirin Shirley, Conference Lead,
>  and the whole ApacheCon Europe 2009 Team
>  http://www.eu.apachecon.com/  23-27 March -- Amsterdam, Netherlands
>
>


--Apple-Mail-1--783641243--

From general-return-111-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Tue Jan 27 20:06:04 2009
Return-Path: <general-return-111-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 8447 invoked from network); 27 Jan 2009 20:06:04 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 27 Jan 2009 20:06:04 -0000
Received: (qmail 45512 invoked by uid 500); 27 Jan 2009 20:06:04 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 45483 invoked by uid 500); 27 Jan 2009 20:06:04 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 45466 invoked by uid 99); 27 Jan 2009 20:06:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jan 2009 12:06:03 -0800
X-ASF-Spam-Status: No, hits=-1.8 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of qiujie@cn.ibm.com designates 202.81.18.163 as permitted sender)
Received: from [202.81.18.163] (HELO e23smtp02.au.ibm.com) (202.81.18.163)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jan 2009 20:05:53 +0000
Received: from d23relay03.au.ibm.com (d23relay03.au.ibm.com [202.81.18.234])
	by e23smtp02.au.ibm.com (8.13.1/8.13.1) with ESMTP id n0RK4P3h027824
	for <general@hadoop.apache.org>; Wed, 28 Jan 2009 07:04:25 +1100
Received: from d23av02.au.ibm.com (d23av02.au.ibm.com [9.190.235.138])
	by d23relay03.au.ibm.com (8.13.8/8.13.8/NCO v9.1) with ESMTP id n0RK5Uxb3878922
	for <general@hadoop.apache.org>; Wed, 28 Jan 2009 07:05:30 +1100
Received: from d23av02.au.ibm.com (loopback [127.0.0.1])
	by d23av02.au.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id n0RK5TiD028041
	for <general@hadoop.apache.org>; Wed, 28 Jan 2009 07:05:29 +1100
Received: from d23m0037.cn.ibm.com (d23m0037.cn.ibm.com [9.181.2.105])
	by d23av02.au.ibm.com (8.12.11.20060308/8.12.11) with ESMTP id n0RK5Tf3028031
	for <general@hadoop.apache.org>; Wed, 28 Jan 2009 07:05:29 +1100
Subject: Jie Qiu is out of the office. Any urgent case, please contact Jie Yang
Auto-Submitted: auto-generated
From: Jie Qiu <qiujie@cn.ibm.com>
To: general@hadoop.apache.org
Message-ID: <OF5466062C.2A8CA1BA-ON4825754B.006E5EF0-4825754B.006E5EF0@cn.ibm.com>
Date: Wed, 28 Jan 2009 04:05:32 +0800
X-MIMETrack: Serialize by Router on D23M0037/23/M/IBM(Release 8.0.1|February 07, 2008) at
 01/28/2009 04:05:33
MIME-Version: 1.0
Content-type: multipart/alternative; 
	Boundary="0__=C7BBFFD8DFFDD8608f9e8a93df938690918cC7BBFFD8DFFDD860"
Content-Disposition: inline
X-Virus-Checked: Checked by ClamAV on apache.org

--0__=C7BBFFD8DFFDD8608f9e8a93df938690918cC7BBFFD8DFFDD860
Content-type: text/plain; charset=US-ASCII


I will be out of the office starting  2009-01-23 and will not return until
2009-02-03.

I have limited access to notes.
--0__=C7BBFFD8DFFDD8608f9e8a93df938690918cC7BBFFD8DFFDD860--


From general-return-112-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Fri Jan 30 03:39:44 2009
Return-Path: <general-return-112-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org
Received: (qmail 68328 invoked from network); 30 Jan 2009 03:39:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 30 Jan 2009 03:39:44 -0000
Received: (qmail 44866 invoked by uid 500); 30 Jan 2009 03:39:37 -0000
Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org
Received: (qmail 44681 invoked by uid 500); 30 Jan 2009 03:39:36 -0000
Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:general-help@hadoop.apache.org>
List-Unsubscribe: <mailto:general-unsubscribe@hadoop.apache.org>
List-Post: <mailto:general@hadoop.apache.org>
List-Id: <general.hadoop.apache.org>
Reply-To: general@hadoop.apache.org
Delivered-To: mailing list general@hadoop.apache.org
Received: (qmail 44577 invoked by uid 99); 30 Jan 2009 03:39:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jan 2009 19:39:36 -0800
X-ASF-Spam-Status: No, hits=1.2 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jan 2009 03:39:27 +0000
Received: from [10.0.1.195] (snvvpn1-10-72-73-c195.hq.corp.yahoo.com [10.72.73.195])
	by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n0U3cknn038160;
	Thu, 29 Jan 2009 19:38:47 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:content-type:
	content-transfer-encoding:mime-version:subject:date:x-mailer;
	b=LmqSMoTiRdfKuEEzaRApnhyCzrndW5mfbrgZG1atTAiXsoG1ctjiHWwfVmXeuIjc
Message-Id: <AD1FCFE1-72BC-471A-8778-449FE8EE9FA2@yahoo-inc.com>
From: Nigel Daley <ndaley@yahoo-inc.com>
To: core-user@hadoop.apache.org, general@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: [ANNOUNCE] Hadoop release 0.18.3 available
Date: Thu, 29 Jan 2009 19:38:46 -0800
X-Mailer: Apple Mail (2.930.3)
X-Virus-Checked: Checked by ClamAV on apache.org

Release 0.18.3 fixes many critical bugs in 0.18.2.

For Hadoop release details and downloads, visit:
http://hadoop.apache.org/core/releases.html

Hadoop 0.18.3 Release Notes are at
http://hadoop.apache.org/core/docs/r0.18.3/releasenotes.html

Thanks to all who contributed to this release!

Nigel

From common-issues-return-10872-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 01 00:34:58 2010
Return-Path: <common-issues-return-10872-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86749 invoked from network); 1 Oct 2010 00:34:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Oct 2010 00:34:58 -0000
Received: (qmail 12006 invoked by uid 500); 1 Oct 2010 00:34:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11957 invoked by uid 500); 1 Oct 2010 00:34:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11949 invoked by uid 99); 1 Oct 2010 00:34:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Oct 2010 00:34:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Oct 2010 00:34:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o910YXPe007232
	for <common-issues@hadoop.apache.org>; Fri, 1 Oct 2010 00:34:34 GMT
Message-ID: <8873031.489741285893273967.JavaMail.jira@thor>
Date: Thu, 30 Sep 2010 20:34:33 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6971) Clover build doesn't generate
 per-test coverage
In-Reply-To: <24019340.389891285355492616.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12916743#action_12916743 ] 

Giridharan Kesavan commented on HADOOP-6971:
--------------------------------------------

+1 Looks good.
Tested on 0.21 and 0.22

> Clover build doesn't generate per-test coverage
> -----------------------------------------------
>
>                 Key: HADOOP-6971
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6971
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build, test
>    Affects Versions: 0.20.3, 0.21.1, 0.22.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-6971.patch, HADOOP-6971.y20S.patch
>
>
> Because of the way the structure of Hadoop's builds is done Clover can't properly detect test classes among the sources. As the result clover reports are incomplete and do not provide viral per-test coverage info.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10873-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 01 07:45:58 2010
Return-Path: <common-issues-return-10873-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16497 invoked from network); 1 Oct 2010 07:45:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Oct 2010 07:45:57 -0000
Received: (qmail 99375 invoked by uid 500); 1 Oct 2010 07:45:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99218 invoked by uid 500); 1 Oct 2010 07:45:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99204 invoked by uid 99); 1 Oct 2010 07:45:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Oct 2010 07:45:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Oct 2010 07:45:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o917jXhO015541
	for <common-issues@hadoop.apache.org>; Fri, 1 Oct 2010 07:45:33 GMT
Message-ID: <16099624.493351285919133235.JavaMail.jira@thor>
Date: Fri, 1 Oct 2010 03:45:33 -0400 (EDT)
From: "Aniket Ray (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-3213) Provide Hadoop Pipes tutorial
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-3213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12916814#action_12916814 ] 

Aniket Ray commented on HADOOP-3213:
------------------------------------

Any reasons why this won't be fixed?
C++ word count is seriously lacking.

> Provide Hadoop Pipes tutorial
> -----------------------------
>
>                 Key: HADOOP-3213
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3213
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: documentation
>    Affects Versions: 0.16.2
>         Environment: All
>            Reporter: Milind Bhandarkar
>            Assignee: Edward J. Yoon
>
> Hadoop pipes is a neat (and more efficient than streaming) way of writing C++ Hadoop map-reduce applications. It would be great to have a tutorial for beginners and advanced users, similar to the Java MapRed tutorial.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10874-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 01 08:45:58 2010
Return-Path: <common-issues-return-10874-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30744 invoked from network); 1 Oct 2010 08:45:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Oct 2010 08:45:58 -0000
Received: (qmail 44025 invoked by uid 500); 1 Oct 2010 08:45:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43868 invoked by uid 500); 1 Oct 2010 08:45:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43850 invoked by uid 99); 1 Oct 2010 08:45:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Oct 2010 08:45:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Oct 2010 08:45:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o918jXbC016452
	for <common-issues@hadoop.apache.org>; Fri, 1 Oct 2010 08:45:33 GMT
Message-ID: <4921632.493901285922733320.JavaMail.jira@thor>
Date: Fri, 1 Oct 2010 04:45:33 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6944) [Herriot] Implement a functionality
 for getting proxy users definitions like groups and hosts.
In-Reply-To: <13358164.74161283948073125.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12916828#action_12916828 ] 

Vinay Kumar Thota commented on HADOOP-6944:
-------------------------------------------

+1 for the patch.

> [Herriot] Implement a functionality for getting proxy users definitions like groups and hosts.
> ----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6944
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6944
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: HADOOP-6944.patch, HADOOP-6944.patch, HADOOP-6944.patch, HADOOP-6944.patch, HADOOP-6944.patch
>
>
> Gridmix should require a proxy user's file for impersonating various jobs. So, implement couple of methods for getting the proxy users list and a proxy users file (it's a combination of proxy users and groups) based on cluster configuration.
> The proxy users list should require for map reduce jobs and proxy users file should require for gridmix jobs.
> The following are methods signature,
> public ProxyUserDefinitions getHadoopProxyUsers() - get the list of proxy users list based on cluster configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10875-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 02 01:45:03 2010
Return-Path: <common-issues-return-10875-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53156 invoked from network); 2 Oct 2010 01:45:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Oct 2010 01:45:02 -0000
Received: (qmail 25756 invoked by uid 500); 2 Oct 2010 01:45:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25653 invoked by uid 500); 2 Oct 2010 01:45:02 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25645 invoked by uid 99); 2 Oct 2010 01:45:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 01:45:01 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 01:45:01 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o921iewK002588
	for <common-issues@hadoop.apache.org>; Sat, 2 Oct 2010 01:44:41 GMT
Message-ID: <25023898.508571285983880790.JavaMail.jira@thor>
Date: Fri, 1 Oct 2010 21:44:40 -0400 (EDT)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6728) Overhaul metrics framework
In-Reply-To: <17506266.48501272420692262.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917138#action_12917138 ] 

Eric Yang commented on HADOOP-6728:
-----------------------------------

The code looks ok.  Is it is standard to do package version in the package name for Hadoop?

I prefer to have the code in *org.apache.hadoop.metrics* to avoid package version in the package name.

Otherwise +1 on the patch.

> Overhaul metrics framework
> --------------------------
>
>                 Key: HADOOP-6728
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6728
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.2
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6728-y20.104.patch, metrics1-flow.png, metrics2-builder-r2.png, metrics2-builder.png, metrics2-flow.png, metrics2-mutable-r2.png, metrics2-mutable.png, metrics2-record-r2.png, metrics2-record.png, metrics2-uml-r2.png, metrics2-uml.png
>
>
> Per discussions with Arun, Chris, Hong and Rajiv, et al, we concluded that the current metrics framework needs an overhaul to:
> * Allow multiple plugins for different monitoring systems simultaneously (see also: HADOOP-6508).
> * Refresh metrics plugin config without server restart.
> ** Including filtering of metrics per plugin.
> * Support metrics schema for plugins.
> The jira will be resolved when core hadoop components (hdfs, mapreduce) are updated to use the new framework . Updates to external components that use the existing metrics framework will be tracked by different issues.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10876-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Oct 03 07:52:00 2010
Return-Path: <common-issues-return-10876-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91429 invoked from network); 3 Oct 2010 07:52:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Oct 2010 07:52:00 -0000
Received: (qmail 14097 invoked by uid 500); 3 Oct 2010 07:52:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12831 invoked by uid 500); 3 Oct 2010 07:51:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12822 invoked by uid 99); 3 Oct 2010 07:51:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 03 Oct 2010 07:51:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 03 Oct 2010 07:51:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o937pXOh023635
	for <common-issues@hadoop.apache.org>; Sun, 3 Oct 2010 07:51:33 GMT
Message-ID: <10251121.516661286092293049.JavaMail.jira@thor>
Date: Sun, 3 Oct 2010 03:51:33 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6984) NPE from
 SequenceFile::Writer.CompressionCodecOption
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

NPE from SequenceFile::Writer.CompressionCodecOption
----------------------------------------------------

                 Key: HADOOP-6984
                 URL: https://issues.apache.org/jira/browse/HADOOP-6984
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.22.0
            Reporter: Chris Douglas
            Priority: Minor


The deprecated HADOOP-6856 constructors can create a compressed writers with a null-wrapped {{CompressionCodecOption}}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10877-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Oct 03 07:53:56 2010
Return-Path: <common-issues-return-10877-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91544 invoked from network); 3 Oct 2010 07:53:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Oct 2010 07:53:55 -0000
Received: (qmail 16336 invoked by uid 500); 3 Oct 2010 07:53:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16305 invoked by uid 500); 3 Oct 2010 07:53:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16297 invoked by uid 99); 3 Oct 2010 07:53:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 03 Oct 2010 07:53:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 03 Oct 2010 07:53:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o937rWMo023664
	for <common-issues@hadoop.apache.org>; Sun, 3 Oct 2010 07:53:32 GMT
Message-ID: <3673745.516711286092412930.JavaMail.jira@thor>
Date: Sun, 3 Oct 2010 03:53:32 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6984) NPE from
 SequenceFile::Writer.CompressionCodecOption
In-Reply-To: <10251121.516661286092293049.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-6984:
----------------------------------

    Status: Patch Available  (was: Open)

> NPE from SequenceFile::Writer.CompressionCodecOption
> ----------------------------------------------------
>
>                 Key: HADOOP-6984
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6984
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Chris Douglas
>            Priority: Minor
>         Attachments: 6984-0.patch
>
>
> The deprecated HADOOP-6856 constructors can create a compressed writers with a null-wrapped {{CompressionCodecOption}}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10878-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Oct 03 07:53:58 2010
Return-Path: <common-issues-return-10878-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91597 invoked from network); 3 Oct 2010 07:53:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Oct 2010 07:53:58 -0000
Received: (qmail 16524 invoked by uid 500); 3 Oct 2010 07:53:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16495 invoked by uid 500); 3 Oct 2010 07:53:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16486 invoked by uid 99); 3 Oct 2010 07:53:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 03 Oct 2010 07:53:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 03 Oct 2010 07:53:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o937rW5n023658
	for <common-issues@hadoop.apache.org>; Sun, 3 Oct 2010 07:53:32 GMT
Message-ID: <388048.516691286092412664.JavaMail.jira@thor>
Date: Sun, 3 Oct 2010 03:53:32 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6984) NPE from
 SequenceFile::Writer.CompressionCodecOption
In-Reply-To: <10251121.516661286092293049.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-6984:
----------------------------------

    Attachment: 6984-0.patch

> NPE from SequenceFile::Writer.CompressionCodecOption
> ----------------------------------------------------
>
>                 Key: HADOOP-6984
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6984
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Chris Douglas
>            Priority: Minor
>         Attachments: 6984-0.patch
>
>
> The deprecated HADOOP-6856 constructors can create a compressed writers with a null-wrapped {{CompressionCodecOption}}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10879-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 04 04:38:03 2010
Return-Path: <common-issues-return-10879-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49334 invoked from network); 4 Oct 2010 04:38:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Oct 2010 04:38:01 -0000
Received: (qmail 36291 invoked by uid 500); 4 Oct 2010 04:38:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35028 invoked by uid 500); 4 Oct 2010 04:37:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34953 invoked by uid 99); 4 Oct 2010 04:37:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 04:37:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 04:37:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o944bXFa016200
	for <common-issues@hadoop.apache.org>; Mon, 4 Oct 2010 04:37:33 GMT
Message-ID: <14100841.523581286167053173.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 00:37:33 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6971) Clover build doesn't generate
 per-test coverage
In-Reply-To: <24019340.389891285355492616.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik resolved HADOOP-6971.
----------------------------------------

     Hadoop Flags: [Reviewed]
    Fix Version/s: 0.21.1
       Resolution: Fixed

I have committed this to the trunk and 0.21.

This chance requires Clover 3.0+ to be used for test coverage runs.

> Clover build doesn't generate per-test coverage
> -----------------------------------------------
>
>                 Key: HADOOP-6971
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6971
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build, test
>    Affects Versions: 0.20.3, 0.21.1, 0.22.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>             Fix For: 0.21.1
>
>         Attachments: HADOOP-6971.patch, HADOOP-6971.y20S.patch
>
>
> Because of the way the structure of Hadoop's builds is done Clover can't properly detect test classes among the sources. As the result clover reports are incomplete and do not provide viral per-test coverage info.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10880-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 04 04:56:00 2010
Return-Path: <common-issues-return-10880-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53551 invoked from network); 4 Oct 2010 04:56:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Oct 2010 04:56:00 -0000
Received: (qmail 49649 invoked by uid 500); 4 Oct 2010 04:55:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49494 invoked by uid 500); 4 Oct 2010 04:55:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49485 invoked by uid 99); 4 Oct 2010 04:55:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 04:55:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 04:55:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o944tXxu016439
	for <common-issues@hadoop.apache.org>; Mon, 4 Oct 2010 04:55:33 GMT
Message-ID: <27334867.523801286168133088.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 00:55:33 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6971) Clover build doesn't generate
 per-test coverage
In-Reply-To: <24019340.389891285355492616.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6971:
---------------------------------------

    Release Note: This fix requires that test coverage is running under Clover 3.0+ 

> Clover build doesn't generate per-test coverage
> -----------------------------------------------
>
>                 Key: HADOOP-6971
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6971
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build, test
>    Affects Versions: 0.20.3, 0.21.1, 0.22.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>             Fix For: 0.21.1
>
>         Attachments: HADOOP-6971.patch, HADOOP-6971.y20S.patch
>
>
> Because of the way the structure of Hadoop's builds is done Clover can't properly detect test classes among the sources. As the result clover reports are incomplete and do not provide viral per-test coverage info.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10881-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 04 05:12:01 2010
Return-Path: <common-issues-return-10881-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60690 invoked from network); 4 Oct 2010 05:12:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Oct 2010 05:12:01 -0000
Received: (qmail 54203 invoked by uid 500); 4 Oct 2010 05:12:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54078 invoked by uid 500); 4 Oct 2010 05:11:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54070 invoked by uid 99); 4 Oct 2010 05:11:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 05:11:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 05:11:56 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o945BYt6016776
	for <common-issues@hadoop.apache.org>; Mon, 4 Oct 2010 05:11:34 GMT
Message-ID: <33347710.524261286169094658.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 01:11:34 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6879) Provide SSH based (Jsch) remote
 execution API for system tests
In-Reply-To: <11760779.565921279956469755.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6879?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6879:
---------------------------------------

           Status: Resolved  (was: Patch Available)
     Hadoop Flags: [Reviewed]
    Fix Version/s: 0.22.0
       Resolution: Fixed

I have just committed it.

> Provide SSH based (Jsch) remote execution API for system tests
> --------------------------------------------------------------
>
>                 Key: HADOOP-6879
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6879
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>    Affects Versions: 0.22.0
>            Reporter: Iyappan Srinivasan
>            Assignee: Konstantin Boudnik
>             Fix For: 0.22.0
>
>         Attachments: 6879-ydist-security-patch.txt, HADOOP-6879.patch, HA=
DOOP-6879.patch, HADOOP-6879.patch, HADOOP-6879.patch, HADOOP-6879.patch, H=
ADOOP-6879.patch, HADOOP-6879.patch, HADOOP-6879.patch, HADOOP-6879.patch, =
HADOOP-6879.patch, HADOOP-6879.patch, HADOOP-6879.y20.patch, HADOOP-6879.y2=
0.patch, HADOOP-6879.y20.patch, HADOOP-6879.y20.patch, HADOOP-6879.y20.patc=
h, HADOOP-6879.y20.patch
>
>
> http://mvnrepository.com/
> com.jcraft =C2=BB jsch=20
> 0.1.42 version needs to be included in the build. This is  needed to faci=
litate implementation of some system (Herriot) testcases .
> Please include this in ivy.
> jsch is originally located in http://www.jcraft.com/jsch/

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10882-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 04 17:23:58 2010
Return-Path: <common-issues-return-10882-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22450 invoked from network); 4 Oct 2010 17:23:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Oct 2010 17:23:58 -0000
Received: (qmail 12569 invoked by uid 500); 4 Oct 2010 17:23:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12522 invoked by uid 500); 4 Oct 2010 17:23:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12514 invoked by uid 99); 4 Oct 2010 17:23:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 17:23:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 17:23:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o94HNXW6028132
	for <common-issues@hadoop.apache.org>; Mon, 4 Oct 2010 17:23:33 GMT
Message-ID: <13291892.532661286213013400.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 13:23:33 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6984) NPE from
 SequenceFile::Writer.CompressionCodecOption
In-Reply-To: <10251121.516661286092293049.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-6984:
----------------------------------

    Status: Open  (was: Patch Available)

> NPE from SequenceFile::Writer.CompressionCodecOption
> ----------------------------------------------------
>
>                 Key: HADOOP-6984
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6984
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Chris Douglas
>            Priority: Minor
>         Attachments: 6984-0.patch
>
>
> The deprecated HADOOP-6856 constructors can create a compressed writers with a null-wrapped {{CompressionCodecOption}}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10883-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 04 19:10:01 2010
Return-Path: <common-issues-return-10883-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80705 invoked from network); 4 Oct 2010 19:10:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Oct 2010 19:10:00 -0000
Received: (qmail 3904 invoked by uid 500); 4 Oct 2010 19:10:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3747 invoked by uid 500); 4 Oct 2010 19:09:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3737 invoked by uid 99); 4 Oct 2010 19:09:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 19:09:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 19:09:59 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o94J9c50000043
	for <common-issues@hadoop.apache.org>; Mon, 4 Oct 2010 19:09:39 GMT
Message-ID: <3900252.534971286219378940.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 15:09:38 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6728) Overhaul metrics framework
In-Reply-To: <17506266.48501272420692262.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917714#action_12917714 ] 

Luke Lu commented on HADOOP-6728:
---------------------------------

Hi Eric,

Thanks for the +1 :) the main reason we decided to use metrics2 is that we anticipated alternative evolution paths preferred by the community (see: 
http://goo.gl/Rjb1 and http://goo.gl/NLLs). It looks like like old and new metrics packages are going to coexist for a while, i.e. taking the #2 path:

* Port all hadoop core metrics (common, hdfs and mapreduce) to new framework.
* Deprecating the old metrics package so that external package (e.g. HBase etc.) can still function (in the old way)

BTW, there is already implicit versioning in hadoop as well, e.g., mapred vs mapreduce package, which I think is more confusing to (newer) people as it's not immediately clear which one is the new version.

IMO, it's quite reasonable to have some package versioning scheme, when there is a coexisting period for old and new packages that are not compatible.


> Overhaul metrics framework
> --------------------------
>
>                 Key: HADOOP-6728
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6728
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.2
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6728-y20.104.patch, metrics1-flow.png, metrics2-builder-r2.png, metrics2-builder.png, metrics2-flow.png, metrics2-mutable-r2.png, metrics2-mutable.png, metrics2-record-r2.png, metrics2-record.png, metrics2-uml-r2.png, metrics2-uml.png
>
>
> Per discussions with Arun, Chris, Hong and Rajiv, et al, we concluded that the current metrics framework needs an overhaul to:
> * Allow multiple plugins for different monitoring systems simultaneously (see also: HADOOP-6508).
> * Refresh metrics plugin config without server restart.
> ** Including filtering of metrics per plugin.
> * Support metrics schema for plugins.
> The jira will be resolved when core hadoop components (hdfs, mapreduce) are updated to use the new framework . Updates to external components that use the existing metrics framework will be tracked by different issues.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10884-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 04 19:11:58 2010
Return-Path: <common-issues-return-10884-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82945 invoked from network); 4 Oct 2010 19:11:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Oct 2010 19:11:58 -0000
Received: (qmail 8431 invoked by uid 500); 4 Oct 2010 19:11:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8361 invoked by uid 500); 4 Oct 2010 19:11:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8353 invoked by uid 99); 4 Oct 2010 19:11:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 19:11:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 19:11:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o94JBXV9000088
	for <common-issues@hadoop.apache.org>; Mon, 4 Oct 2010 19:11:33 GMT
Message-ID: <9521717.535011286219493458.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 15:11:33 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6984) NPE from
 SequenceFile::Writer.CompressionCodecOption
In-Reply-To: <10251121.516661286092293049.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6984:
--------------------------------

      Component/s: fs
    Fix Version/s: 0.21.1
         Assignee: Chris Douglas
     Hadoop Flags: [Reviewed]

+1

> NPE from SequenceFile::Writer.CompressionCodecOption
> ----------------------------------------------------
>
>                 Key: HADOOP-6984
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6984
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Chris Douglas
>            Assignee: Chris Douglas
>            Priority: Minor
>             Fix For: 0.21.1
>
>         Attachments: 6984-0.patch
>
>
> The deprecated HADOOP-6856 constructors can create a compressed writers with a null-wrapped {{CompressionCodecOption}}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10885-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 04 19:11:58 2010
Return-Path: <common-issues-return-10885-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82996 invoked from network); 4 Oct 2010 19:11:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Oct 2010 19:11:58 -0000
Received: (qmail 8585 invoked by uid 500); 4 Oct 2010 19:11:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8377 invoked by uid 500); 4 Oct 2010 19:11:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8362 invoked by uid 99); 4 Oct 2010 19:11:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 19:11:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 19:11:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o94JBYhv000095
	for <common-issues@hadoop.apache.org>; Mon, 4 Oct 2010 19:11:34 GMT
Message-ID: <30452980.535031286219494207.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 15:11:34 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6984) NPE from
 SequenceFile::Writer.CompressionCodecOption
In-Reply-To: <10251121.516661286092293049.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6984:
--------------------------------

    Status: Patch Available  (was: Open)

> NPE from SequenceFile::Writer.CompressionCodecOption
> ----------------------------------------------------
>
>                 Key: HADOOP-6984
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6984
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Chris Douglas
>            Assignee: Chris Douglas
>            Priority: Minor
>             Fix For: 0.21.1
>
>         Attachments: 6984-0.patch
>
>
> The deprecated HADOOP-6856 constructors can create a compressed writers with a null-wrapped {{CompressionCodecOption}}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10886-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 04 19:15:58 2010
Return-Path: <common-issues-return-10886-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84292 invoked from network); 4 Oct 2010 19:15:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Oct 2010 19:15:57 -0000
Received: (qmail 9928 invoked by uid 500); 4 Oct 2010 19:15:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9857 invoked by uid 500); 4 Oct 2010 19:15:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9848 invoked by uid 99); 4 Oct 2010 19:15:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 19:15:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 19:15:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o94JFXhq000171
	for <common-issues@hadoop.apache.org>; Mon, 4 Oct 2010 19:15:33 GMT
Message-ID: <4417221.535121286219733304.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 15:15:33 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6984) NPE from
 SequenceFile::Writer.CompressionCodecOption
In-Reply-To: <10251121.516661286092293049.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6984:
--------------------------------

    Fix Version/s:     (was: 0.21.1)
                   0.22.0

> NPE from SequenceFile::Writer.CompressionCodecOption
> ----------------------------------------------------
>
>                 Key: HADOOP-6984
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6984
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Chris Douglas
>            Assignee: Chris Douglas
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: 6984-0.patch
>
>
> The deprecated HADOOP-6856 constructors can create a compressed writers with a null-wrapped {{CompressionCodecOption}}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10887-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 04 20:39:54 2010
Return-Path: <common-issues-return-10887-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14298 invoked from network); 4 Oct 2010 20:39:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Oct 2010 20:39:54 -0000
Received: (qmail 46448 invoked by uid 500); 4 Oct 2010 20:39:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46285 invoked by uid 500); 4 Oct 2010 20:39:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46277 invoked by uid 99); 4 Oct 2010 20:39:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 20:39:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 20:39:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o94KdWPe001534
	for <common-issues@hadoop.apache.org>; Mon, 4 Oct 2010 20:39:33 GMT
Message-ID: <26576141.536571286224772607.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 16:39:32 -0400 (EDT)
From: "Ramkumar Vadali (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6985) Suggest that HADOOP_OPTS be preserved
 in hadoop-env.sh.template
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Suggest that HADOOP_OPTS be preserved in hadoop-env.sh.template
---------------------------------------------------------------

                 Key: HADOOP-6985
                 URL: https://issues.apache.org/jira/browse/HADOOP-6985
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Ramkumar Vadali
            Priority: Minor


For an administrator who wants to customize HADOOP_OPTS, it would be better to have

# if [ "$HADOOP_OPTS" == "" ]; then export HADOOP_OPTS=-server; else FOO+=" -server"; fi

instead of

# Extra Java runtime options.  Empty by default.
# export HADOOP_OPTS=-server


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10888-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 04 21:30:59 2010
Return-Path: <common-issues-return-10888-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35362 invoked from network); 4 Oct 2010 21:30:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Oct 2010 21:30:59 -0000
Received: (qmail 19535 invoked by uid 500); 4 Oct 2010 21:30:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19480 invoked by uid 500); 4 Oct 2010 21:30:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19472 invoked by uid 99); 4 Oct 2010 21:30:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 21:30:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 21:30:56 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o94LUZ3X002584
	for <common-issues@hadoop.apache.org>; Mon, 4 Oct 2010 21:30:35 GMT
Message-ID: <21754660.537831286227835135.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 17:30:35 -0400 (EDT)
From: "Al Thompson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6977) Herriot daemon clients should vend
 statistics
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917775#action_12917775 ] 

Al Thompson commented on HADOOP-6977:
-------------------------------------

I have reviewed the patch in the most recent attachment. In doing so, it occurred to me that the for loop to search for JMX options in the HADOOP_OPTS environment variable could be simplified. This can be done by using either List.contains(..) or Arrays.binarySearch(..). The former may be preferred, as the latter requires the collection to be sorted before invoking binarySearch(). I will attach a small sample that illustrates both of these.

I don't think changing this implementation will impact performance much, good or bad. IMHO the code change would result in cleaner code and provide an examplar to other committers on how not to re-implement the Java libraries in Hadoop code. 

> Herriot daemon clients should vend statistics
> ---------------------------------------------
>
>                 Key: HADOOP-6977
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6977
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-6977.patch, HADOOP-6977.patch, HADOOP-6977.y20S.patch, HADOOP-6977.y20S.patch
>
>
> The HDFS web user interface serves useful information through dfshealth.jsp and dfsnodelist.jsp.
> The Herriot interface to Hadoop cluster daemons would benefit from the addition of some way to channel metics information.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10889-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 04 21:34:57 2010
Return-Path: <common-issues-return-10889-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36804 invoked from network); 4 Oct 2010 21:34:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Oct 2010 21:34:57 -0000
Received: (qmail 23032 invoked by uid 500); 4 Oct 2010 21:34:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22949 invoked by uid 500); 4 Oct 2010 21:34:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22931 invoked by uid 99); 4 Oct 2010 21:34:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 21:34:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 21:34:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o94LYXIb002645
	for <common-issues@hadoop.apache.org>; Mon, 4 Oct 2010 21:34:33 GMT
Message-ID: <18033379.537891286228073037.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 17:34:33 -0400 (EDT)
From: "Al Thompson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6977) Herriot daemon clients should vend
 statistics
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Al Thompson updated HADOOP-6977:
--------------------------------

    Attachment: FinderTest.java

A simple JUnit test which illustrates alternatives to searching for a key in a String array.

> Herriot daemon clients should vend statistics
> ---------------------------------------------
>
>                 Key: HADOOP-6977
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6977
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: FinderTest.java, HADOOP-6977.patch, HADOOP-6977.patch, HADOOP-6977.y20S.patch, HADOOP-6977.y20S.patch
>
>
> The HDFS web user interface serves useful information through dfshealth.jsp and dfsnodelist.jsp.
> The Herriot interface to Hadoop cluster daemons would benefit from the addition of some way to channel metics information.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10890-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 04 21:36:59 2010
Return-Path: <common-issues-return-10890-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37382 invoked from network); 4 Oct 2010 21:36:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Oct 2010 21:36:59 -0000
Received: (qmail 25069 invoked by uid 500); 4 Oct 2010 21:36:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25009 invoked by uid 500); 4 Oct 2010 21:36:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25001 invoked by uid 99); 4 Oct 2010 21:36:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 21:36:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 21:36:56 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o94LaZ1v002683
	for <common-issues@hadoop.apache.org>; Mon, 4 Oct 2010 21:36:35 GMT
Message-ID: <31980992.537951286228195160.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 17:36:35 -0400 (EDT)
From: "Nicolas Spiegelberg (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6986) SequenceFile.Reader should
 distinguish between Network IOE and Parsing IOE
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

SequenceFile.Reader should distinguish between Network IOE and Parsing IOE
--------------------------------------------------------------------------

                 Key: HADOOP-6986
                 URL: https://issues.apache.org/jira/browse/HADOOP-6986
             Project: Hadoop Common
          Issue Type: Bug
          Components: io
    Affects Versions: 0.21.1, 0.22.0, 0.20-append
            Reporter: Nicolas Spiegelberg
            Priority: Minor
             Fix For: 0.21.1, 0.22.0, 0.20-append


The SequenceFile.Reader api should give the user an easy way to distinguish between a Network/Low-level IOE and a Parsing IOE.  The use case appeared recently in the HBase project:

Originally, if a RegionServer got an IOE from HDFS while opening a region file, it would abort the open and let the HMaster reassign the region.  The assumption being that this is a network failure that will likely disappear at a later time or different partition of the network.  However, if HBase gets parsing exceptions, we want to log the problem and continue opening the region anyways, because parsing is an idempotent problem and retries won't fix this issue.

Although this problem was found in HBase, it seems to be a generic problem of being able to more easily identify idempotent vs transient errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10891-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 04 21:50:59 2010
Return-Path: <common-issues-return-10891-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44018 invoked from network); 4 Oct 2010 21:50:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Oct 2010 21:50:59 -0000
Received: (qmail 43081 invoked by uid 500); 4 Oct 2010 21:50:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42923 invoked by uid 500); 4 Oct 2010 21:50:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42914 invoked by uid 99); 4 Oct 2010 21:50:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 21:50:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 21:50:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o94LoYtQ002889
	for <common-issues@hadoop.apache.org>; Mon, 4 Oct 2010 21:50:34 GMT
Message-ID: <18686330.538171286229034539.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 17:50:34 -0400 (EDT)
From: "Nicolas Spiegelberg (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6986) SequenceFile.Reader should
 distinguish between Network IOE and Parsing IOE
In-Reply-To: <31980992.537951286228195160.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicolas Spiegelberg updated HADOOP-6986:
----------------------------------------

    Attachment: HADOOP-6986_0.21.patch
                HADOOP-6986_20-append.patch

2 patch versions: one works for 20-append branch, the 0.21 works for both 0.21 & 0.22

> SequenceFile.Reader should distinguish between Network IOE and Parsing IOE
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6986
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6986
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.21.1, 0.22.0, 0.20-append
>            Reporter: Nicolas Spiegelberg
>            Priority: Minor
>             Fix For: 0.21.1, 0.22.0, 0.20-append
>
>         Attachments: HADOOP-6986_0.21.patch, HADOOP-6986_20-append.patch
>
>
> The SequenceFile.Reader api should give the user an easy way to distinguish between a Network/Low-level IOE and a Parsing IOE.  The use case appeared recently in the HBase project:
> Originally, if a RegionServer got an IOE from HDFS while opening a region file, it would abort the open and let the HMaster reassign the region.  The assumption being that this is a network failure that will likely disappear at a later time or different partition of the network.  However, if HBase gets parsing exceptions, we want to log the problem and continue opening the region anyways, because parsing is an idempotent problem and retries won't fix this issue.
> Although this problem was found in HBase, it seems to be a generic problem of being able to more easily identify idempotent vs transient errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10892-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 04 22:02:57 2010
Return-Path: <common-issues-return-10892-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49472 invoked from network); 4 Oct 2010 22:02:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Oct 2010 22:02:57 -0000
Received: (qmail 62092 invoked by uid 500); 4 Oct 2010 22:02:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61992 invoked by uid 500); 4 Oct 2010 22:02:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61982 invoked by uid 99); 4 Oct 2010 22:02:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 22:02:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 22:02:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o94M2Za2003549
	for <common-issues@hadoop.apache.org>; Mon, 4 Oct 2010 22:02:35 GMT
Message-ID: <19393305.538541286229755620.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 18:02:35 -0400 (EDT)
From: "Nicolas Spiegelberg (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6986) SequenceFile.Reader should
 distinguish between Network IOE and Parsing IOE
In-Reply-To: <31980992.537951286228195160.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917795#action_12917795 ] 

Nicolas Spiegelberg commented on HADOOP-6986:
---------------------------------------------

To fix this issue, I kept all the existing error types & messages, but I added ParseException as the cause to all parsing-related IOEs.  None of the changed exceptions had an associated cause prior.  This will allow us to maintain 100% backwards compatibility (in case any users were doing deep inspection of the IOE text) while allowing new users and easy way to check:  if(ioe.getCause() instanceof ParseException)

> SequenceFile.Reader should distinguish between Network IOE and Parsing IOE
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6986
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6986
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.21.1, 0.22.0, 0.20-append
>            Reporter: Nicolas Spiegelberg
>            Priority: Minor
>             Fix For: 0.21.1, 0.22.0, 0.20-append
>
>         Attachments: HADOOP-6986_0.21.patch, HADOOP-6986_20-append.patch
>
>
> The SequenceFile.Reader api should give the user an easy way to distinguish between a Network/Low-level IOE and a Parsing IOE.  The use case appeared recently in the HBase project:
> Originally, if a RegionServer got an IOE from HDFS while opening a region file, it would abort the open and let the HMaster reassign the region.  The assumption being that this is a network failure that will likely disappear at a later time or different partition of the network.  However, if HBase gets parsing exceptions, we want to log the problem and continue opening the region anyways, because parsing is an idempotent problem and retries won't fix this issue.
> Although this problem was found in HBase, it seems to be a generic problem of being able to more easily identify idempotent vs transient errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10893-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 04 22:22:58 2010
Return-Path: <common-issues-return-10893-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58554 invoked from network); 4 Oct 2010 22:22:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Oct 2010 22:22:58 -0000
Received: (qmail 80831 invoked by uid 500); 4 Oct 2010 22:22:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80244 invoked by uid 500); 4 Oct 2010 22:22:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80046 invoked by uid 99); 4 Oct 2010 22:22:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 22:22:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 22:22:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o94MMX96003876
	for <common-issues@hadoop.apache.org>; Mon, 4 Oct 2010 22:22:33 GMT
Message-ID: <26687627.538921286230953013.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 18:22:33 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6987) Use JUnit Rule to optionally fail
 test cases that run more than 10 seconds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Use JUnit Rule to optionally fail test cases that run more than 10 seconds
--------------------------------------------------------------------------

                 Key: HADOOP-6987
                 URL: https://issues.apache.org/jira/browse/HADOOP-6987
             Project: Hadoop Common
          Issue Type: Improvement
          Components: test
    Affects Versions: 0.21.0
            Reporter: Jakob Homan
            Assignee: Jakob Homan
             Fix For: 0.22.0


Using JUnit Rules annotations we can fail tests cases that take longer than 10 seconds (for instance) to run.  This provides a regression check against test cases taking longer than they had previously due to unintended code changes, as well as provides a membership criteria for unit tests versus integration tests in HDFS and MR.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10894-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 04 22:27:55 2010
Return-Path: <common-issues-return-10894-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60507 invoked from network); 4 Oct 2010 22:27:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Oct 2010 22:27:55 -0000
Received: (qmail 87939 invoked by uid 500); 4 Oct 2010 22:27:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87506 invoked by uid 500); 4 Oct 2010 22:27:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87495 invoked by uid 99); 4 Oct 2010 22:27:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 22:27:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 22:27:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o94MRXt6003947
	for <common-issues@hadoop.apache.org>; Mon, 4 Oct 2010 22:27:34 GMT
Message-ID: <13381880.538971286231253687.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 18:27:33 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6987) Use JUnit Rule to optionally fail
 test cases that run more than 10 seconds
In-Reply-To: <26687627.538921286230953013.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6987:
--------------------------------

    Attachment: HADOOP-6897.patch

Patch creates a new base class, TenSecondsTimeoutPerTest, which tests can extend to 'mix in' (as much as Java allows) this behavior.  Tests extending this class will automatically be failed if their cases run more than 10 seconds, like so:
{noformat}
@Test public void willTimeOut() throws InterruptedException {
  Thread.sleep(12 * 1000);
}
{noformat}
results in
{noformat}
 Testcase: willTimeOut took 10.007 sec
   Caused an ERROR
 test timed out after 10000 milliseconds
 java.lang.Exception: test timed out after 10000 milliseconds
   at java.lang.Thread.sleep(Native Method)
   at org.apache.hadoop.util.TestStringUtils.willTimeOut(TestStringUtils.java:45)
{noformat}
Tests cases need to be in JUnit 4 style to do this, and I've converted one from Core at random, to test its functionality.  Most - though not all - all of the Core tests behave well, so this feature is intended more for HDFS and MR. 

> Use JUnit Rule to optionally fail test cases that run more than 10 seconds
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6987
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6987
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6897.patch
>
>
> Using JUnit Rules annotations we can fail tests cases that take longer than 10 seconds (for instance) to run.  This provides a regression check against test cases taking longer than they had previously due to unintended code changes, as well as provides a membership criteria for unit tests versus integration tests in HDFS and MR.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10895-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 04 22:27:56 2010
Return-Path: <common-issues-return-10895-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60550 invoked from network); 4 Oct 2010 22:27:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Oct 2010 22:27:56 -0000
Received: (qmail 88106 invoked by uid 500); 4 Oct 2010 22:27:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87911 invoked by uid 500); 4 Oct 2010 22:27:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87503 invoked by uid 99); 4 Oct 2010 22:27:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 22:27:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 22:27:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o94MRYfa003953
	for <common-issues@hadoop.apache.org>; Mon, 4 Oct 2010 22:27:34 GMT
Message-ID: <7197831.538991286231254331.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 18:27:34 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6987) Use JUnit Rule to optionally fail
 test cases that run more than 10 seconds
In-Reply-To: <26687627.538921286230953013.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6987:
--------------------------------

    Status: Patch Available  (was: Open)

Submitting patch.

> Use JUnit Rule to optionally fail test cases that run more than 10 seconds
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6987
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6987
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6897.patch
>
>
> Using JUnit Rules annotations we can fail tests cases that take longer than 10 seconds (for instance) to run.  This provides a regression check against test cases taking longer than they had previously due to unintended code changes, as well as provides a membership criteria for unit tests versus integration tests in HDFS and MR.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10896-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 04 23:28:55 2010
Return-Path: <common-issues-return-10896-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75991 invoked from network); 4 Oct 2010 23:28:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Oct 2010 23:28:55 -0000
Received: (qmail 33781 invoked by uid 500); 4 Oct 2010 23:28:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33637 invoked by uid 500); 4 Oct 2010 23:28:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33629 invoked by uid 99); 4 Oct 2010 23:28:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 23:28:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 23:28:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o94NSXtC004729
	for <common-issues@hadoop.apache.org>; Mon, 4 Oct 2010 23:28:33 GMT
Message-ID: <17443930.539541286234913119.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 19:28:33 -0400 (EDT)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6988) Add support for reading multiple
 hadoop delegation token files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Add support for reading multiple hadoop delegation token files
--------------------------------------------------------------

                 Key: HADOOP-6988
                 URL: https://issues.apache.org/jira/browse/HADOOP-6988
             Project: Hadoop Common
          Issue Type: Improvement
          Components: security
    Affects Versions: 0.22.0
            Reporter: Aaron T. Myers
            Assignee: Aaron T. Myers


It would be nice if there were a way to specify multiple delegation token files via the HADOOP_TOKEN_FILE_LOCATION environment variable and the "mapreduce.job.credentials.binary" configuration value. I suggest a colon-separated list of paths, each of which is read as a separate delegation token file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10897-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 04 23:56:58 2010
Return-Path: <common-issues-return-10897-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80489 invoked from network); 4 Oct 2010 23:56:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Oct 2010 23:56:57 -0000
Received: (qmail 48520 invoked by uid 500); 4 Oct 2010 23:56:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48482 invoked by uid 500); 4 Oct 2010 23:56:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48474 invoked by uid 99); 4 Oct 2010 23:56:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 23:56:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 23:56:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o94NuWje005189
	for <common-issues@hadoop.apache.org>; Mon, 4 Oct 2010 23:56:33 GMT
Message-ID: <31394265.540181286236592866.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 19:56:32 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6984) NPE from
 SequenceFile::Writer.CompressionCodecOption
In-Reply-To: <10251121.516661286092293049.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-6984:
----------------------------------

    Attachment: 6984-1.patch

Collapsed CompressionTypeOption and CompressionCodecOption into a single Option type.

> NPE from SequenceFile::Writer.CompressionCodecOption
> ----------------------------------------------------
>
>                 Key: HADOOP-6984
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6984
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Chris Douglas
>            Assignee: Chris Douglas
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: 6984-0.patch, 6984-1.patch
>
>
> The deprecated HADOOP-6856 constructors can create a compressed writers with a null-wrapped {{CompressionCodecOption}}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10898-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 04 23:56:59 2010
Return-Path: <common-issues-return-10898-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80536 invoked from network); 4 Oct 2010 23:56:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Oct 2010 23:56:58 -0000
Received: (qmail 48736 invoked by uid 500); 4 Oct 2010 23:56:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48656 invoked by uid 500); 4 Oct 2010 23:56:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48548 invoked by uid 99); 4 Oct 2010 23:56:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 23:56:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 23:56:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o94NuYYj005195
	for <common-issues@hadoop.apache.org>; Mon, 4 Oct 2010 23:56:34 GMT
Message-ID: <4408858.540201286236594170.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 19:56:34 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6989) TestSetFile is failing on trunk
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

TestSetFile is failing on trunk
-------------------------------

                 Key: HADOOP-6989
                 URL: https://issues.apache.org/jira/browse/HADOOP-6989
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.22.0
            Reporter: Jakob Homan
             Fix For: 0.22.0


Testsuite: org.apache.hadoop.io.TestSetFile
Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 1.015 sec
------------- Standard Output ---------------
2010-10-04 16:32:01,030 INFO  io.TestSetFile (TestSetFile.java:generate(56)) - generating 10000 records in memory
2010-10-04 16:32:01,249 INFO  io.TestSetFile (TestSetFile.java:generate(63)) - sorting 10000 records
2010-10-04 16:32:01,350 INFO  io.TestSetFile (TestSetFile.java:writeTest(72)) - creating with 10000 records
------------- ---------------- ---------------

Testcase: testSetFile took 0.964 sec
	Caused an ERROR
key class or comparator option must be set
java.lang.IllegalArgumentException: key class or comparator option must be set
	at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:247)
	at org.apache.hadoop.io.SetFile$Writer.<init>(SetFile.java:60)
	at org.apache.hadoop.io.TestSetFile.writeTest(TestSetFile.java:73)
	at org.apache.hadoop.io.TestSetFile.testSetFile(TestSetFile.java:45)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10899-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 04 23:56:59 2010
Return-Path: <common-issues-return-10899-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80544 invoked from network); 4 Oct 2010 23:56:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Oct 2010 23:56:58 -0000
Received: (qmail 48762 invoked by uid 500); 4 Oct 2010 23:56:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48667 invoked by uid 500); 4 Oct 2010 23:56:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48658 invoked by uid 99); 4 Oct 2010 23:56:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 23:56:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 23:56:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o94NuYFa005204
	for <common-issues@hadoop.apache.org>; Mon, 4 Oct 2010 23:56:34 GMT
Message-ID: <9953465.540231286236594563.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 19:56:34 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6989) TestSetFile is failing on trunk
In-Reply-To: <4408858.540201286236594170.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917824#action_12917824 ] 

Jakob Homan commented on HADOOP-6989:
-------------------------------------

git bisect blames 3612cf60b989f7d9140efd3e5d2b6b13d792599f, HADOOP-6856. 

> TestSetFile is failing on trunk
> -------------------------------
>
>                 Key: HADOOP-6989
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6989
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Jakob Homan
>             Fix For: 0.22.0
>
>
> Testsuite: org.apache.hadoop.io.TestSetFile
> Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 1.015 sec
> ------------- Standard Output ---------------
> 2010-10-04 16:32:01,030 INFO  io.TestSetFile (TestSetFile.java:generate(56)) - generating 10000 records in memory
> 2010-10-04 16:32:01,249 INFO  io.TestSetFile (TestSetFile.java:generate(63)) - sorting 10000 records
> 2010-10-04 16:32:01,350 INFO  io.TestSetFile (TestSetFile.java:writeTest(72)) - creating with 10000 records
> ------------- ---------------- ---------------
> Testcase: testSetFile took 0.964 sec
> 	Caused an ERROR
> key class or comparator option must be set
> java.lang.IllegalArgumentException: key class or comparator option must be set
> 	at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:247)
> 	at org.apache.hadoop.io.SetFile$Writer.<init>(SetFile.java:60)
> 	at org.apache.hadoop.io.TestSetFile.writeTest(TestSetFile.java:73)
> 	at org.apache.hadoop.io.TestSetFile.testSetFile(TestSetFile.java:45)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10900-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 00:00:59 2010
Return-Path: <common-issues-return-10900-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81449 invoked from network); 5 Oct 2010 00:00:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 00:00:58 -0000
Received: (qmail 52289 invoked by uid 500); 5 Oct 2010 00:00:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51877 invoked by uid 500); 5 Oct 2010 00:00:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51869 invoked by uid 99); 5 Oct 2010 00:00:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 00:00:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 00:00:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9500X8q005262
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 00:00:33 GMT
Message-ID: <30428642.540261286236833649.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 20:00:33 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6987) Use JUnit Rule to optionally fail
 test cases that run more than 10 seconds
In-Reply-To: <26687627.538921286230953013.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917825#action_12917825 ] 

Jakob Homan commented on HADOOP-6987:
-------------------------------------

Not getting my hopes up about Hudson.  Tests all pass except known-bad TestSetFile (HADOOP-6989).  Test-patch is good:
{noformat}
     [exec] +1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 5 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec] 
     [exec]     +1 system tests framework.  The patch passed system tests framework compile.{noformat}

> Use JUnit Rule to optionally fail test cases that run more than 10 seconds
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6987
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6987
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6897.patch
>
>
> Using JUnit Rules annotations we can fail tests cases that take longer than 10 seconds (for instance) to run.  This provides a regression check against test cases taking longer than they had previously due to unintended code changes, as well as provides a membership criteria for unit tests versus integration tests in HDFS and MR.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10901-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 00:05:58 2010
Return-Path: <common-issues-return-10901-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86735 invoked from network); 5 Oct 2010 00:05:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 00:05:58 -0000
Received: (qmail 55858 invoked by uid 500); 5 Oct 2010 00:05:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55752 invoked by uid 500); 5 Oct 2010 00:05:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55744 invoked by uid 99); 5 Oct 2010 00:05:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 00:05:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 00:05:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9505XwS005337
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 00:05:33 GMT
Message-ID: <26427040.540331286237133714.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 20:05:33 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6856) SequenceFile and MapFile need
 cleanup to remove redundant constructors
In-Reply-To: <19355698.285171278691549769.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917826#action_12917826 ] 

Konstantin Boudnik commented on HADOOP-6856:
--------------------------------------------

I wonder if all tests were executed in the light of this HADOOP-6989?

> SequenceFile and MapFile need cleanup to remove redundant constructors
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6856
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6856
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.22.0
>
>         Attachments: h-6856.patch, h-6856.patch
>
>
> Currently there are 2 public SequenceFile.Reader constructors, 3 public SequenceFile.Writer constructors, 9 public SequenceFile.createWriter, 2 public MapFile.Reader constructors, and 8 public MapFile.Writer constructors. All of with various historical combinations of parameters that don't cover the entire space.
> All of this makes it *very* difficult to add new optional parameters to SequenceFile and MapFile. 
> I'd like change to the style of FileContext.create with option parameters. I'll implement one public SequenceFile.Reader constructor and one public SequenceFile.createWriter and implement all of the current variants based on those two. I'll do the same for MapFile.Reader and MapFile.Writer including passing parameters down to the underlying SequenceFile.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10902-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 00:09:55 2010
Return-Path: <common-issues-return-10902-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87038 invoked from network); 5 Oct 2010 00:09:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 00:09:54 -0000
Received: (qmail 56867 invoked by uid 500); 5 Oct 2010 00:09:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56743 invoked by uid 500); 5 Oct 2010 00:09:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56735 invoked by uid 99); 5 Oct 2010 00:09:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 00:09:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 00:09:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9509Xx4005380
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 00:09:33 GMT
Message-ID: <8558916.540381286237373087.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 20:09:33 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6989) TestSetFile is failing on trunk
In-Reply-To: <4408858.540201286236594170.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-6989:
----------------------------------

    Attachment: 6989-0.patch

Transcription error. {{s/keyClass/valueClass}}

> TestSetFile is failing on trunk
> -------------------------------
>
>                 Key: HADOOP-6989
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6989
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: 6989-0.patch
>
>
> Testsuite: org.apache.hadoop.io.TestSetFile
> Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 1.015 sec
> ------------- Standard Output ---------------
> 2010-10-04 16:32:01,030 INFO  io.TestSetFile (TestSetFile.java:generate(56)) - generating 10000 records in memory
> 2010-10-04 16:32:01,249 INFO  io.TestSetFile (TestSetFile.java:generate(63)) - sorting 10000 records
> 2010-10-04 16:32:01,350 INFO  io.TestSetFile (TestSetFile.java:writeTest(72)) - creating with 10000 records
> ------------- ---------------- ---------------
> Testcase: testSetFile took 0.964 sec
> 	Caused an ERROR
> key class or comparator option must be set
> java.lang.IllegalArgumentException: key class or comparator option must be set
> 	at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:247)
> 	at org.apache.hadoop.io.SetFile$Writer.<init>(SetFile.java:60)
> 	at org.apache.hadoop.io.TestSetFile.writeTest(TestSetFile.java:73)
> 	at org.apache.hadoop.io.TestSetFile.testSetFile(TestSetFile.java:45)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10903-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 00:11:55 2010
Return-Path: <common-issues-return-10903-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87339 invoked from network); 5 Oct 2010 00:11:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 00:11:54 -0000
Received: (qmail 57129 invoked by uid 500); 5 Oct 2010 00:11:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57063 invoked by uid 500); 5 Oct 2010 00:11:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57054 invoked by uid 99); 5 Oct 2010 00:11:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 00:11:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 00:11:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o950BXa9005409
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 00:11:33 GMT
Message-ID: <13146942.540411286237493229.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 20:11:33 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6989) TestSetFile is failing on trunk
In-Reply-To: <4408858.540201286236594170.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-6989:
----------------------------------

    Status: Patch Available  (was: Open)

> TestSetFile is failing on trunk
> -------------------------------
>
>                 Key: HADOOP-6989
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6989
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: 6989-0.patch
>
>
> Testsuite: org.apache.hadoop.io.TestSetFile
> Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 1.015 sec
> ------------- Standard Output ---------------
> 2010-10-04 16:32:01,030 INFO  io.TestSetFile (TestSetFile.java:generate(56)) - generating 10000 records in memory
> 2010-10-04 16:32:01,249 INFO  io.TestSetFile (TestSetFile.java:generate(63)) - sorting 10000 records
> 2010-10-04 16:32:01,350 INFO  io.TestSetFile (TestSetFile.java:writeTest(72)) - creating with 10000 records
> ------------- ---------------- ---------------
> Testcase: testSetFile took 0.964 sec
> 	Caused an ERROR
> key class or comparator option must be set
> java.lang.IllegalArgumentException: key class or comparator option must be set
> 	at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:247)
> 	at org.apache.hadoop.io.SetFile$Writer.<init>(SetFile.java:60)
> 	at org.apache.hadoop.io.TestSetFile.writeTest(TestSetFile.java:73)
> 	at org.apache.hadoop.io.TestSetFile.testSetFile(TestSetFile.java:45)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10904-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 00:21:58 2010
Return-Path: <common-issues-return-10904-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88439 invoked from network); 5 Oct 2010 00:21:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 00:21:57 -0000
Received: (qmail 61771 invoked by uid 500); 5 Oct 2010 00:21:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61642 invoked by uid 500); 5 Oct 2010 00:21:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61634 invoked by uid 99); 5 Oct 2010 00:21:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 00:21:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 00:21:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o950LXh2005515
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 00:21:33 GMT
Message-ID: <19644367.540481286238093364.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 20:21:33 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6977) Herriot daemon clients should vend
 statistics
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6977:
---------------------------------------

    Attachment: HADOOP-6977.patch

Thanks Al: totally makes sense. The code looks much cleaner now. I guess I need to abandon my JDK1.1 code habits and learn how big guys are doing it nowadays. 

> Herriot daemon clients should vend statistics
> ---------------------------------------------
>
>                 Key: HADOOP-6977
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6977
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: FinderTest.java, HADOOP-6977.patch, HADOOP-6977.patch, HADOOP-6977.patch, HADOOP-6977.y20S.patch, HADOOP-6977.y20S.patch
>
>
> The HDFS web user interface serves useful information through dfshealth.jsp and dfsnodelist.jsp.
> The Herriot interface to Hadoop cluster daemons would benefit from the addition of some way to channel metics information.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10905-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 00:29:54 2010
Return-Path: <common-issues-return-10905-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95428 invoked from network); 5 Oct 2010 00:29:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 00:29:54 -0000
Received: (qmail 64082 invoked by uid 500); 5 Oct 2010 00:29:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64023 invoked by uid 500); 5 Oct 2010 00:29:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64015 invoked by uid 99); 5 Oct 2010 00:29:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 00:29:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 00:29:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o950TWup005590
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 00:29:33 GMT
Message-ID: <14810559.540511286238572876.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 20:29:32 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6987) Use JUnit Rule to optionally fail
 test cases that run more than 10 seconds
In-Reply-To: <26687627.538921286230953013.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917830#action_12917830 ] 

Konstantin Boudnik commented on HADOOP-6987:
--------------------------------------------

I don't like unnecessary inheritance in tests. This is, in fact, why I can't stand JUnit3 ;)
The solution in the patch has one more disadvantage: what if we'll have tests which shouldn't go over 10 secs, and others (i.e. true unit tests) which shouldn't exceed 2 secs? Would offered solution force us to have different implementations for different timeouts?

In this particular case I'd suggest to simply annotate test cases with expected timeout as in 
{noformat}
@Test(timeout=10000)
public void testEscapeString() throws Exception {
{noformat}

> Use JUnit Rule to optionally fail test cases that run more than 10 seconds
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6987
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6987
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6897.patch
>
>
> Using JUnit Rules annotations we can fail tests cases that take longer than 10 seconds (for instance) to run.  This provides a regression check against test cases taking longer than they had previously due to unintended code changes, as well as provides a membership criteria for unit tests versus integration tests in HDFS and MR.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10906-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 00:33:58 2010
Return-Path: <common-issues-return-10906-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96948 invoked from network); 5 Oct 2010 00:33:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 00:33:57 -0000
Received: (qmail 66405 invoked by uid 500); 5 Oct 2010 00:33:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66341 invoked by uid 500); 5 Oct 2010 00:33:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66331 invoked by uid 99); 5 Oct 2010 00:33:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 00:33:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 00:33:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o950XWRL005654
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 00:33:33 GMT
Message-ID: <26082779.540571286238812923.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 20:33:32 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6989) TestSetFile is failing on trunk
In-Reply-To: <4408858.540201286236594170.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917832#action_12917832 ] 

Konstantin Boudnik commented on HADOOP-6989:
--------------------------------------------

Shall HADOOP-6856 be reverted and reapplied with all corrections in place, considering that it already had two separate commits and this one will be #3 ?

> TestSetFile is failing on trunk
> -------------------------------
>
>                 Key: HADOOP-6989
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6989
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: 6989-0.patch
>
>
> Testsuite: org.apache.hadoop.io.TestSetFile
> Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 1.015 sec
> ------------- Standard Output ---------------
> 2010-10-04 16:32:01,030 INFO  io.TestSetFile (TestSetFile.java:generate(56)) - generating 10000 records in memory
> 2010-10-04 16:32:01,249 INFO  io.TestSetFile (TestSetFile.java:generate(63)) - sorting 10000 records
> 2010-10-04 16:32:01,350 INFO  io.TestSetFile (TestSetFile.java:writeTest(72)) - creating with 10000 records
> ------------- ---------------- ---------------
> Testcase: testSetFile took 0.964 sec
> 	Caused an ERROR
> key class or comparator option must be set
> java.lang.IllegalArgumentException: key class or comparator option must be set
> 	at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:247)
> 	at org.apache.hadoop.io.SetFile$Writer.<init>(SetFile.java:60)
> 	at org.apache.hadoop.io.TestSetFile.writeTest(TestSetFile.java:73)
> 	at org.apache.hadoop.io.TestSetFile.testSetFile(TestSetFile.java:45)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10907-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 00:43:54 2010
Return-Path: <common-issues-return-10907-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98534 invoked from network); 5 Oct 2010 00:43:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 00:43:54 -0000
Received: (qmail 71844 invoked by uid 500); 5 Oct 2010 00:43:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71738 invoked by uid 500); 5 Oct 2010 00:43:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71730 invoked by uid 99); 5 Oct 2010 00:43:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 00:43:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 00:43:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o950hWD9005799
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 00:43:33 GMT
Message-ID: <2957964.540761286239412964.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 20:43:32 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6987) Use JUnit Rule to optionally fail
 test cases that run more than 10 seconds
In-Reply-To: <26687627.538921286230953013.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917835#action_12917835 ] 

Jakob Homan commented on HADOOP-6987:
-------------------------------------

bq. I don't like unnecessary inheritance in tests. 
I'm not wild about them either, but one works with what one's got.

bq. what if we'll have tests which shouldn't go over 10 secs, and others (i.e. true unit tests) which shouldn't exceed 2 secs
The current layout in HDFS is to have separate directory trees for the integration and (currently sparsely populated) unit tests.  This implies we wouldn't be commingling the two types in the same file, and this approach is predicated on that assumption.  I'm ok with either approach, but if we're going to be combining types, we probably should collapse the directory trees sooner than later.

If, on the other hand, we're separating unit and integration tests into different files, this approach has the advantage that one can apply the rule to an entire file with only one line (assuming the test is JUnit 4), rather than having to mark each and every test case with the same annotation.  This approach thus has a smaller burden on the test writer and on any refactoring we may want to do to implement it.

Also, it would be simple enough to write a script as part of test-patch to verify that all tests in /unit are so marked and thus eligible to be in the unit category (although, it also wouldn't be prohibitively difficult to write one to check each @Test annotation to include a timeout value).



> Use JUnit Rule to optionally fail test cases that run more than 10 seconds
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6987
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6987
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6897.patch
>
>
> Using JUnit Rules annotations we can fail tests cases that take longer than 10 seconds (for instance) to run.  This provides a regression check against test cases taking longer than they had previously due to unintended code changes, as well as provides a membership criteria for unit tests versus integration tests in HDFS and MR.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10908-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 00:46:55 2010
Return-Path: <common-issues-return-10908-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98823 invoked from network); 5 Oct 2010 00:46:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 00:46:55 -0000
Received: (qmail 73401 invoked by uid 500); 5 Oct 2010 00:46:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73244 invoked by uid 500); 5 Oct 2010 00:46:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73236 invoked by uid 99); 5 Oct 2010 00:46:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 00:46:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 00:46:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o950kXdL005841
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 00:46:34 GMT
Message-ID: <4012406.540811286239593897.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 20:46:33 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6987) Use JUnit Rule to optionally fail
 test cases that run more than 10 seconds
In-Reply-To: <26687627.538921286230953013.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917837#action_12917837 ] 

Jakob Homan commented on HADOOP-6987:
-------------------------------------

Also, this approach doesn't rule out the other.  It just adds a convenience class that can be included, or not, as needed.  Test writers could certainly not use it and annotate individual test cases, as may be appropriate.

> Use JUnit Rule to optionally fail test cases that run more than 10 seconds
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6987
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6987
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6897.patch
>
>
> Using JUnit Rules annotations we can fail tests cases that take longer than 10 seconds (for instance) to run.  This provides a regression check against test cases taking longer than they had previously due to unintended code changes, as well as provides a membership criteria for unit tests versus integration tests in HDFS and MR.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10909-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 00:58:55 2010
Return-Path: <common-issues-return-10909-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2391 invoked from network); 5 Oct 2010 00:58:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 00:58:54 -0000
Received: (qmail 83444 invoked by uid 500); 5 Oct 2010 00:58:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83388 invoked by uid 500); 5 Oct 2010 00:58:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83380 invoked by uid 99); 5 Oct 2010 00:58:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 00:58:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 00:58:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o950wWu3006005
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 00:58:33 GMT
Message-ID: <21211285.541001286240312885.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 20:58:32 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6987) Use JUnit Rule to optionally fail
 test cases that run more than 10 seconds
In-Reply-To: <26687627.538921286230953013.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917844#action_12917844 ] 

Konstantin Boudnik commented on HADOOP-6987:
--------------------------------------------

Effectively, you offer to have a common class which has to be extended instead of having an explicit statement like
{{@Rule public MethodRule globalTimeout = new Timeout(10 * 1000);}} in each test class which needs to have a special timeout clause. I vouch that having an explicit {{@Rule}} statement like above (if the same timeout applies for all class' cases) is clearer and doesn't require inheritance. 

I think it worth noting [here|http://wiki.apache.org/hadoop/HowToDevelopUnitTests] about best practices of timeout settings as soon as we'll agreed on the approach.

> Use JUnit Rule to optionally fail test cases that run more than 10 seconds
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6987
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6987
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6897.patch
>
>
> Using JUnit Rules annotations we can fail tests cases that take longer than 10 seconds (for instance) to run.  This provides a regression check against test cases taking longer than they had previously due to unintended code changes, as well as provides a membership criteria for unit tests versus integration tests in HDFS and MR.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10910-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 01:09:55 2010
Return-Path: <common-issues-return-10910-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8385 invoked from network); 5 Oct 2010 01:09:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 01:09:55 -0000
Received: (qmail 88962 invoked by uid 500); 5 Oct 2010 01:09:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88888 invoked by uid 500); 5 Oct 2010 01:09:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88880 invoked by uid 99); 5 Oct 2010 01:09:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 01:09:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 01:09:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9519XCw006186
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 01:09:33 GMT
Message-ID: <21436411.541171286240973271.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 21:09:33 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6977) Herriot daemon clients should vend
 statistics
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917848#action_12917848 ] 

Konstantin Boudnik commented on HADOOP-6977:
--------------------------------------------

I have ran {{test-patch.sh}} locally and all look Ok
{noformat}
+1 overall.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 system tests framework.  The patch passed system tests framework compile.
{noformat}

Also, I used the version of Common built with this patch to run TestHL040 from HDFS-1408 and it passed. So, it seems that patch is ready to be committed to trunk.

> Herriot daemon clients should vend statistics
> ---------------------------------------------
>
>                 Key: HADOOP-6977
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6977
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: FinderTest.java, HADOOP-6977.patch, HADOOP-6977.patch, HADOOP-6977.patch, HADOOP-6977.y20S.patch, HADOOP-6977.y20S.patch
>
>
> The HDFS web user interface serves useful information through dfshealth.jsp and dfsnodelist.jsp.
> The Herriot interface to Hadoop cluster daemons would benefit from the addition of some way to channel metics information.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10911-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 01:31:58 2010
Return-Path: <common-issues-return-10911-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11197 invoked from network); 5 Oct 2010 01:31:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 01:31:58 -0000
Received: (qmail 93875 invoked by uid 500); 5 Oct 2010 01:31:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93817 invoked by uid 500); 5 Oct 2010 01:31:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93809 invoked by uid 99); 5 Oct 2010 01:31:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 01:31:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 01:31:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o951VXpB006425
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 01:31:34 GMT
Message-ID: <33408582.541281286242293786.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 21:31:33 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6944) [Herriot] Implement a functionality
 for getting proxy users definitions like groups and hosts.
In-Reply-To: <13358164.74161283948073125.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6944:
---------------------------------------

    Affects Version/s: 0.20.3

> [Herriot] Implement a functionality for getting proxy users definitions like groups and hosts.
> ----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6944
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6944
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>    Affects Versions: 0.20.3
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: HADOOP-6944.patch, HADOOP-6944.patch, HADOOP-6944.patch, HADOOP-6944.patch, HADOOP-6944.patch
>
>
> Gridmix should require a proxy user's file for impersonating various jobs. So, implement couple of methods for getting the proxy users list and a proxy users file (it's a combination of proxy users and groups) based on cluster configuration.
> The proxy users list should require for map reduce jobs and proxy users file should require for gridmix jobs.
> The following are methods signature,
> public ProxyUserDefinitions getHadoopProxyUsers() - get the list of proxy users list based on cluster configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10912-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 02:21:58 2010
Return-Path: <common-issues-return-10912-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24189 invoked from network); 5 Oct 2010 02:21:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 02:21:58 -0000
Received: (qmail 18447 invoked by uid 500); 5 Oct 2010 02:21:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18379 invoked by uid 500); 5 Oct 2010 02:21:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18371 invoked by uid 99); 5 Oct 2010 02:21:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 02:21:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 02:21:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o952LX6h007045
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 02:21:34 GMT
Message-ID: <26853266.541731286245293961.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 22:21:33 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6944) [Herriot] Implement a functionality
 for getting proxy users definitions like groups and hosts.
In-Reply-To: <13358164.74161283948073125.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917862#action_12917862 ] 

Konstantin Boudnik commented on HADOOP-6944:
--------------------------------------------

I have manually ran {{test-patch.sh}} to make sure that patch is Ok
{noformat}
+1 overall.  

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 5 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 system tests framework.  The patch passed system tests framework compile.
{noformat}

> [Herriot] Implement a functionality for getting proxy users definitions like groups and hosts.
> ----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6944
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6944
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>    Affects Versions: 0.20.3
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: HADOOP-6944.patch, HADOOP-6944.patch, HADOOP-6944.patch, HADOOP-6944.patch, HADOOP-6944.patch
>
>
> Gridmix should require a proxy user's file for impersonating various jobs. So, implement couple of methods for getting the proxy users list and a proxy users file (it's a combination of proxy users and groups) based on cluster configuration.
> The proxy users list should require for map reduce jobs and proxy users file should require for gridmix jobs.
> The following are methods signature,
> public ProxyUserDefinitions getHadoopProxyUsers() - get the list of proxy users list based on cluster configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10913-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 02:26:57 2010
Return-Path: <common-issues-return-10913-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24863 invoked from network); 5 Oct 2010 02:26:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 02:26:57 -0000
Received: (qmail 21704 invoked by uid 500); 5 Oct 2010 02:26:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21634 invoked by uid 500); 5 Oct 2010 02:26:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21626 invoked by uid 99); 5 Oct 2010 02:26:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 02:26:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 02:26:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o952QWrK007130
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 02:26:33 GMT
Message-ID: <23983344.541871286245592779.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 22:26:32 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6990) Move TestSequenceFile from MapReduce
 to Common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Move TestSequenceFile from MapReduce to Common
----------------------------------------------

                 Key: HADOOP-6990
                 URL: https://issues.apache.org/jira/browse/HADOOP-6990
             Project: Hadoop Common
          Issue Type: Task
          Components: io
    Affects Versions: 0.21.0, 0.22.0
            Reporter: Chris Douglas
            Priority: Trivial


{{TestSequenceFile}} escaped from Common and ended up in the MapReduce tree, causing several regressions in {{SequenceFile}} to go unnoticed

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10914-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 02:28:54 2010
Return-Path: <common-issues-return-10914-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25234 invoked from network); 5 Oct 2010 02:28:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 02:28:54 -0000
Received: (qmail 22369 invoked by uid 500); 5 Oct 2010 02:28:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22330 invoked by uid 500); 5 Oct 2010 02:28:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22322 invoked by uid 99); 5 Oct 2010 02:28:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 02:28:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 02:28:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o952SWDp007156
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 02:28:33 GMT
Message-ID: <22609732.541901286245712924.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 22:28:32 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6991) SequenceFile::Reader discards length
 for files, does not call openFile
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

SequenceFile::Reader discards length for files, does not call openFile
----------------------------------------------------------------------

                 Key: HADOOP-6991
                 URL: https://issues.apache.org/jira/browse/HADOOP-6991
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.22.0
            Reporter: Chris Douglas
            Priority: Minor


While the sorting and merging in {{SequenceFile}} is deprecated and {{openFile}} is archaic, the semantics should remain consistent.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10915-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 02:31:09 2010
Return-Path: <common-issues-return-10915-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25698 invoked from network); 5 Oct 2010 02:31:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 02:31:09 -0000
Received: (qmail 23832 invoked by uid 500); 5 Oct 2010 02:31:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23794 invoked by uid 500); 5 Oct 2010 02:31:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23786 invoked by uid 99); 5 Oct 2010 02:31:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 02:31:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 02:31:08 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o952UlpV007187
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 02:30:48 GMT
Message-ID: <23936790.541931286245847944.JavaMail.jira@thor>
Date: Mon, 4 Oct 2010 22:30:47 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6991) SequenceFile::Reader discards length
 for files, does not call openFile
In-Reply-To: <22609732.541901286245712924.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-6991:
----------------------------------

    Attachment: 6991-0.patch

> SequenceFile::Reader discards length for files, does not call openFile
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6991
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6991
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Chris Douglas
>            Priority: Minor
>         Attachments: 6991-0.patch
>
>
> While the sorting and merging in {{SequenceFile}} is deprecated and {{openFile}} is archaic, the semantics should remain consistent.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10916-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 05:55:01 2010
Return-Path: <common-issues-return-10916-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61740 invoked from network); 5 Oct 2010 05:55:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 05:55:01 -0000
Received: (qmail 18900 invoked by uid 500); 5 Oct 2010 05:55:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18731 invoked by uid 500); 5 Oct 2010 05:54:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18723 invoked by uid 99); 5 Oct 2010 05:54:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 05:54:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 05:54:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o955sWxQ009947
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 05:54:32 GMT
Message-ID: <16638636.543131286258072872.JavaMail.jira@thor>
Date: Tue, 5 Oct 2010 01:54:32 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6984) NPE from
 SequenceFile::Writer.CompressionCodecOption
In-Reply-To: <10251121.516661286092293049.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917885#action_12917885 ] 

Owen O'Malley commented on HADOOP-6984:
---------------------------------------

I really like the second iteration of the patch, but maybe we could have compress(kind) and compress(kind, codec) instead of using compress(kind, null) for the default codec. The implementation could be the same.

> NPE from SequenceFile::Writer.CompressionCodecOption
> ----------------------------------------------------
>
>                 Key: HADOOP-6984
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6984
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Chris Douglas
>            Assignee: Chris Douglas
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: 6984-0.patch, 6984-1.patch
>
>
> The deprecated HADOOP-6856 constructors can create a compressed writers with a null-wrapped {{CompressionCodecOption}}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10917-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 06:56:12 2010
Return-Path: <common-issues-return-10917-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80566 invoked from network); 5 Oct 2010 06:56:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 06:56:11 -0000
Received: (qmail 77541 invoked by uid 500); 5 Oct 2010 06:56:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77386 invoked by uid 500); 5 Oct 2010 06:56:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77374 invoked by uid 99); 5 Oct 2010 06:56:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 06:56:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 06:56:05 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o956thGX010655
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 06:55:44 GMT
Message-ID: <12688143.543351286261743972.JavaMail.jira@thor>
Date: Tue, 5 Oct 2010 02:55:43 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6989) TestSetFile is failing on trunk
In-Reply-To: <4408858.540201286236594170.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6989:
----------------------------------

      Resolution: Fixed
        Assignee: Jakob Homan
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I just committed this. Thanks, Jakob.

> TestSetFile is failing on trunk
> -------------------------------
>
>                 Key: HADOOP-6989
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6989
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: 6989-0.patch
>
>
> Testsuite: org.apache.hadoop.io.TestSetFile
> Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 1.015 sec
> ------------- Standard Output ---------------
> 2010-10-04 16:32:01,030 INFO  io.TestSetFile (TestSetFile.java:generate(56)) - generating 10000 records in memory
> 2010-10-04 16:32:01,249 INFO  io.TestSetFile (TestSetFile.java:generate(63)) - sorting 10000 records
> 2010-10-04 16:32:01,350 INFO  io.TestSetFile (TestSetFile.java:writeTest(72)) - creating with 10000 records
> ------------- ---------------- ---------------
> Testcase: testSetFile took 0.964 sec
> 	Caused an ERROR
> key class or comparator option must be set
> java.lang.IllegalArgumentException: key class or comparator option must be set
> 	at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:247)
> 	at org.apache.hadoop.io.SetFile$Writer.<init>(SetFile.java:60)
> 	at org.apache.hadoop.io.TestSetFile.writeTest(TestSetFile.java:73)
> 	at org.apache.hadoop.io.TestSetFile.testSetFile(TestSetFile.java:45)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10918-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 07:01:00 2010
Return-Path: <common-issues-return-10918-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81855 invoked from network); 5 Oct 2010 07:01:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 07:01:00 -0000
Received: (qmail 79749 invoked by uid 500); 5 Oct 2010 07:01:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79581 invoked by uid 500); 5 Oct 2010 07:00:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79554 invoked by uid 99); 5 Oct 2010 07:00:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 07:00:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 07:00:56 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9570aAo010734
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 07:00:36 GMT
Message-ID: <2444265.543401286262036567.JavaMail.jira@thor>
Date: Tue, 5 Oct 2010 03:00:36 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6989) TestSetFile is failing on trunk
In-Reply-To: <4408858.540201286236594170.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley reassigned HADOOP-6989:
-------------------------------------

    Assignee: Chris Douglas  (was: Jakob Homan)

> TestSetFile is failing on trunk
> -------------------------------
>
>                 Key: HADOOP-6989
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6989
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Jakob Homan
>            Assignee: Chris Douglas
>             Fix For: 0.22.0
>
>         Attachments: 6989-0.patch
>
>
> Testsuite: org.apache.hadoop.io.TestSetFile
> Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 1.015 sec
> ------------- Standard Output ---------------
> 2010-10-04 16:32:01,030 INFO  io.TestSetFile (TestSetFile.java:generate(56)) - generating 10000 records in memory
> 2010-10-04 16:32:01,249 INFO  io.TestSetFile (TestSetFile.java:generate(63)) - sorting 10000 records
> 2010-10-04 16:32:01,350 INFO  io.TestSetFile (TestSetFile.java:writeTest(72)) - creating with 10000 records
> ------------- ---------------- ---------------
> Testcase: testSetFile took 0.964 sec
> 	Caused an ERROR
> key class or comparator option must be set
> java.lang.IllegalArgumentException: key class or comparator option must be set
> 	at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:247)
> 	at org.apache.hadoop.io.SetFile$Writer.<init>(SetFile.java:60)
> 	at org.apache.hadoop.io.TestSetFile.writeTest(TestSetFile.java:73)
> 	at org.apache.hadoop.io.TestSetFile.testSetFile(TestSetFile.java:45)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10919-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 07:15:03 2010
Return-Path: <common-issues-return-10919-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87840 invoked from network); 5 Oct 2010 07:15:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 07:15:02 -0000
Received: (qmail 88338 invoked by uid 500); 5 Oct 2010 07:15:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88180 invoked by uid 500); 5 Oct 2010 07:15:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88164 invoked by uid 99); 5 Oct 2010 07:14:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 07:14:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 07:14:57 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o957EZkh010936
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 07:14:35 GMT
Message-ID: <6005029.543511286262875372.JavaMail.jira@thor>
Date: Tue, 5 Oct 2010 03:14:35 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6988) Add support for reading multiple
 hadoop delegation token files
In-Reply-To: <17443930.539541286234913119.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917891#action_12917891 ] 

Allen Wittenauer commented on HADOOP-6988:
------------------------------------------

This seems like a great way to inject tokens into a process that it shouldn't have.

> Add support for reading multiple hadoop delegation token files
> --------------------------------------------------------------
>
>                 Key: HADOOP-6988
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6988
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>
> It would be nice if there were a way to specify multiple delegation token files via the HADOOP_TOKEN_FILE_LOCATION environment variable and the "mapreduce.job.credentials.binary" configuration value. I suggest a colon-separated list of paths, each of which is read as a separate delegation token file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10920-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 07:15:03 2010
Return-Path: <common-issues-return-10920-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87888 invoked from network); 5 Oct 2010 07:15:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 07:15:03 -0000
Received: (qmail 88408 invoked by uid 500); 5 Oct 2010 07:15:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88205 invoked by uid 500); 5 Oct 2010 07:15:02 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88167 invoked by uid 99); 5 Oct 2010 07:14:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 07:14:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 07:14:57 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o957Ea6p010942
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 07:14:36 GMT
Message-ID: <12579599.543531286262876266.JavaMail.jira@thor>
Date: Tue, 5 Oct 2010 03:14:36 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6984) NPE from
 SequenceFile::Writer.CompressionCodecOption
In-Reply-To: <10251121.516661286092293049.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-6984:
----------------------------------

    Attachment: 6984-2.patch

*nod* That does look cleaner.

> NPE from SequenceFile::Writer.CompressionCodecOption
> ----------------------------------------------------
>
>                 Key: HADOOP-6984
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6984
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Chris Douglas
>            Assignee: Chris Douglas
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: 6984-0.patch, 6984-1.patch, 6984-2.patch
>
>
> The deprecated HADOOP-6856 constructors can create a compressed writers with a null-wrapped {{CompressionCodecOption}}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10921-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 10:25:00 2010
Return-Path: <common-issues-return-10921-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47429 invoked from network); 5 Oct 2010 10:24:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 10:24:59 -0000
Received: (qmail 77303 invoked by uid 500); 5 Oct 2010 10:24:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77114 invoked by uid 500); 5 Oct 2010 10:24:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77101 invoked by uid 99); 5 Oct 2010 10:24:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 10:24:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 10:24:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o95AOZbt014005
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 10:24:35 GMT
Message-ID: <13421409.545261286274275135.JavaMail.jira@thor>
Date: Tue, 5 Oct 2010 06:24:35 -0400 (EDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6929) RPC should have a way to pass
 Security information other than protocol annotations
In-Reply-To: <29776327.15071282855734948.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sharad Agarwal updated HADOOP-6929:
-----------------------------------

    Attachment: Hadoop-6929_v1.patch

Patch for review. This allows to pass Security information via Avro tunnel to Hadoop RPC. Adds tests to Avro RPC (for reflect and specific).

> RPC should have a way to pass Security information other than protocol annotations
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6929
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6929
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>             Fix For: 0.22.0
>
>         Attachments: Hadoop-6929_v1.patch
>
>
> Currently Hadoop RPC allows protocol annotations as the only way to pass security information. This becomes a problem if protocols are generated and not hand written. For example protocols generated via Avro and passed over Avro tunnel (AvroRpcEngine.java) can't pass the security information.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10922-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 16:09:59 2010
Return-Path: <common-issues-return-10922-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21547 invoked from network); 5 Oct 2010 16:09:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 16:09:59 -0000
Received: (qmail 3240 invoked by uid 500); 5 Oct 2010 16:09:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3072 invoked by uid 500); 5 Oct 2010 16:09:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3061 invoked by uid 99); 5 Oct 2010 16:09:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 16:09:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 16:09:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o95G9Xeo019595
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 16:09:34 GMT
Message-ID: <1218449.549741286294973883.JavaMail.jira@thor>
Date: Tue, 5 Oct 2010 12:09:33 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6988) Add support for reading multiple
 hadoop delegation token files
In-Reply-To: <17443930.539541286234913119.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918042#action_12918042 ] 

Allen Wittenauer commented on HADOOP-6988:
------------------------------------------

At a minimum, -1 on the environment variable.

Shouldn't HADOOP_CLIENT_OPTS be sufficient for passing extra -D params?  We have an abundance of environment variables that users can't handle as it is.  

> Add support for reading multiple hadoop delegation token files
> --------------------------------------------------------------
>
>                 Key: HADOOP-6988
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6988
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>
> It would be nice if there were a way to specify multiple delegation token files via the HADOOP_TOKEN_FILE_LOCATION environment variable and the "mapreduce.job.credentials.binary" configuration value. I suggest a colon-separated list of paths, each of which is read as a separate delegation token file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10923-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 16:22:58 2010
Return-Path: <common-issues-return-10923-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25533 invoked from network); 5 Oct 2010 16:22:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 16:22:58 -0000
Received: (qmail 33757 invoked by uid 500); 5 Oct 2010 16:22:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33713 invoked by uid 500); 5 Oct 2010 16:22:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33705 invoked by uid 99); 5 Oct 2010 16:22:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 16:22:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 16:22:56 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o95GMa5K019887
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 16:22:36 GMT
Message-ID: <19303087.550171286295756399.JavaMail.jira@thor>
Date: Tue, 5 Oct 2010 12:22:36 -0400 (EDT)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6988) Add support for reading multiple
 hadoop delegation token files
In-Reply-To: <17443930.539541286234913119.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918053#action_12918053 ] 

Aaron T. Myers commented on HADOOP-6988:
----------------------------------------

Thanks for the comments, Allen.

The environment variable already exists in hadoop trunk; it's just only capable of specifying a single path to a token file. I'd like to be able to specify multiple token files via this variable. This is really only for convenience, as it's entirely possible to stuff multiple delegation token objects into a single credentials object, which is then serialized to a file. I considered creating a tool which would be capable of merging multiple delegation token files into one, but this seemed like a cleaner solution.

> Add support for reading multiple hadoop delegation token files
> --------------------------------------------------------------
>
>                 Key: HADOOP-6988
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6988
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>
> It would be nice if there were a way to specify multiple delegation token files via the HADOOP_TOKEN_FILE_LOCATION environment variable and the "mapreduce.job.credentials.binary" configuration value. I suggest a colon-separated list of paths, each of which is read as a separate delegation token file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10924-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 17:01:59 2010
Return-Path: <common-issues-return-10924-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34121 invoked from network); 5 Oct 2010 17:01:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 17:01:59 -0000
Received: (qmail 80327 invoked by uid 500); 5 Oct 2010 17:01:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80263 invoked by uid 500); 5 Oct 2010 17:01:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80253 invoked by uid 99); 5 Oct 2010 17:01:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 17:01:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 17:01:56 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o95H1Y0t020626
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 17:01:34 GMT
Message-ID: <4517390.550981286298094812.JavaMail.jira@thor>
Date: Tue, 5 Oct 2010 13:01:34 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6988) Add support for reading multiple
 hadoop delegation token files
In-Reply-To: <17443930.539541286234913119.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918073#action_12918073 ] 

Allen Wittenauer commented on HADOOP-6988:
------------------------------------------

Grr.  I really wish we'd stop creating pet environment variables.  This is ridiculous.

Can we remove this env var as part of this JIRA?  What takes precendence the env var or the mapreduce.job.credentials.binary jobconf setting?  What is the interaction?  If the answer is "we have to look at the code" then we've failed.

It makes much more sense to have mapreduce.job.credentials.binary to support a comma delimited set (to be consistent with the rest of the job conf.  Never mind that colon is the traditional directory delimiter on OS X.)

> Add support for reading multiple hadoop delegation token files
> --------------------------------------------------------------
>
>                 Key: HADOOP-6988
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6988
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>
> It would be nice if there were a way to specify multiple delegation token files via the HADOOP_TOKEN_FILE_LOCATION environment variable and the "mapreduce.job.credentials.binary" configuration value. I suggest a colon-separated list of paths, each of which is read as a separate delegation token file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10925-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 18:04:54 2010
Return-Path: <common-issues-return-10925-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59831 invoked from network); 5 Oct 2010 18:04:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 18:04:54 -0000
Received: (qmail 55977 invoked by uid 500); 5 Oct 2010 18:04:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55884 invoked by uid 500); 5 Oct 2010 18:04:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55876 invoked by uid 99); 5 Oct 2010 18:04:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 18:04:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 18:04:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o95I4XZ7021780
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 18:04:33 GMT
Message-ID: <31885092.552181286301873315.JavaMail.jira@thor>
Date: Tue, 5 Oct 2010 14:04:33 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6984) NPE from
 SequenceFile::Writer.CompressionCodecOption
In-Reply-To: <10251121.516661286092293049.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-6984:
----------------------------------

    Attachment: 6984-3.patch

Merge with trunk

> NPE from SequenceFile::Writer.CompressionCodecOption
> ----------------------------------------------------
>
>                 Key: HADOOP-6984
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6984
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Chris Douglas
>            Assignee: Chris Douglas
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: 6984-0.patch, 6984-1.patch, 6984-2.patch, 6984-3.patch
>
>
> The deprecated HADOOP-6856 constructors can create a compressed writers with a null-wrapped {{CompressionCodecOption}}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10926-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 18:38:17 2010
Return-Path: <common-issues-return-10926-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85640 invoked from network); 5 Oct 2010 18:38:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 18:38:16 -0000
Received: (qmail 10105 invoked by uid 500); 5 Oct 2010 18:38:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10050 invoked by uid 500); 5 Oct 2010 18:38:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10042 invoked by uid 99); 5 Oct 2010 18:38:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 18:38:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 18:38:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o95IbtnA022473
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 18:37:55 GMT
Message-ID: <20611797.553111286303875137.JavaMail.jira@thor>
Date: Tue, 5 Oct 2010 14:37:55 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6991) SequenceFile::Reader discards length
 for files, does not call openFile
In-Reply-To: <22609732.541901286245712924.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-6991:
----------------------------------

    Attachment: 6991-1.patch

Patch on top of HADOOP-6984

> SequenceFile::Reader discards length for files, does not call openFile
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6991
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6991
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Chris Douglas
>            Priority: Minor
>         Attachments: 6991-0.patch, 6991-1.patch
>
>
> While the sorting and merging in {{SequenceFile}} is deprecated and {{openFile}} is archaic, the semantics should remain consistent.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10927-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 20:13:00 2010
Return-Path: <common-issues-return-10927-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59942 invoked from network); 5 Oct 2010 20:13:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 20:13:00 -0000
Received: (qmail 69528 invoked by uid 500); 5 Oct 2010 20:13:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69398 invoked by uid 500); 5 Oct 2010 20:12:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69389 invoked by uid 99); 5 Oct 2010 20:12:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 20:12:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 20:12:57 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o95KCZwY024061
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 20:12:35 GMT
Message-ID: <23895502.554711286309555842.JavaMail.jira@thor>
Date: Tue, 5 Oct 2010 16:12:35 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6988) Add support for reading multiple
 hadoop delegation token files
In-Reply-To: <17443930.539541286234913119.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918145#action_12918145 ] 

Owen O'Malley commented on HADOOP-6988:
---------------------------------------

The environment variable should *not* be multi-valued. It is used to communicate the job's token store to sub-processes of the task. Since a task can't be in more than one job, there isn't any need.

What is the use case for having multiple token files? The rest of the lists use commas, so this should be the same. Wouldn't it be easier to write a tool that allows you to combine multiple token files together into a single one?

> Add support for reading multiple hadoop delegation token files
> --------------------------------------------------------------
>
>                 Key: HADOOP-6988
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6988
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>
> It would be nice if there were a way to specify multiple delegation token files via the HADOOP_TOKEN_FILE_LOCATION environment variable and the "mapreduce.job.credentials.binary" configuration value. I suggest a colon-separated list of paths, each of which is read as a separate delegation token file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10928-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 22:02:58 2010
Return-Path: <common-issues-return-10928-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12639 invoked from network); 5 Oct 2010 22:02:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 22:02:58 -0000
Received: (qmail 26694 invoked by uid 500); 5 Oct 2010 22:02:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26640 invoked by uid 500); 5 Oct 2010 22:02:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26632 invoked by uid 99); 5 Oct 2010 22:02:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 22:02:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 22:02:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o95M2XZp026839
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 22:02:33 GMT
Message-ID: <32187961.559881286316153481.JavaMail.jira@thor>
Date: Tue, 5 Oct 2010 18:02:33 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6969) CHANGES.txt does not reflect the
 release of version 0.21.0.
In-Reply-To: <12538682.369991285269513067.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White resolved HADOOP-6969.
-------------------------------

    Resolution: Fixed

I've fixed this.

> CHANGES.txt does not reflect the release of version 0.21.0.
> -----------------------------------------------------------
>
>                 Key: HADOOP-6969
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6969
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>         Environment: CHANGES.txt should show the release date for 0.21.0 and include section for for 0.21.1 - Unreleased. Latest changes, that did not make into 0.21.0, should be moved under 0.21.1 section.
>            Reporter: Konstantin Shvachko
>            Assignee: Tom White
>             Fix For: 0.21.1
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10929-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 23:17:00 2010
Return-Path: <common-issues-return-10929-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46417 invoked from network); 5 Oct 2010 23:17:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 23:17:00 -0000
Received: (qmail 16636 invoked by uid 500); 5 Oct 2010 23:17:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16555 invoked by uid 500); 5 Oct 2010 23:17:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16547 invoked by uid 99); 5 Oct 2010 23:16:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 23:16:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 23:16:57 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o95NGZ0N028475
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 23:16:36 GMT
Message-ID: <28234767.562781286320595762.JavaMail.jira@thor>
Date: Tue, 5 Oct 2010 19:16:35 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6356) Add a Cache for AbstractFileSystem
 in the new FileContext/AbstractFileSystem framework.
In-Reply-To: <2126828437.1257203939433.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918313#action_12918313 ] 

Sanjay Radia commented on HADOOP-6356:
--------------------------------------

Dhruba says > I think it is appropriate to do the compatible-version-check for each RPC instead of doing it per creation-of-FileSystem object.

Having a FileSystem cache => RPCProxy is also cached => version check is done once per file system object. Dhruba is right in that this is incorrect since the NN may reboot with a different version.

However we will see a more serious performance problem with FileContext under certain usage patterns. FileContext keeps a pointer to the default file system; hence all file operations to the default filesystem will not require a version check (but it is susceptible to the NN reboot with different verison problem). However operations to other file systems such as fc.open("hdfs://nn/foo/bar)  will create a new AbstractFileSystem object that will then create a new dfsClient which will create a new rpcProxy which will do a getVersion() - this is an extra round trip. Good news is that the connection at the lowest layer is cached. 

We can fix this by having each rpc call also piggy back the version number so that the version check is not needed when a proxy is created. This however will break wire compatibility.  

> Add a Cache for AbstractFileSystem in the new FileContext/AbstractFileSystem framework.
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6356
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6356
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.22.0
>
>
> The new filesystem framework, FileContext and AbstractFileSystem does not implement a cache for AbstractFileSystem.
> This Jira proposes to add a cache to the new framework just like with the old FileSystem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10930-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 05 23:20:56 2010
Return-Path: <common-issues-return-10930-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46947 invoked from network); 5 Oct 2010 23:20:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Oct 2010 23:20:56 -0000
Received: (qmail 21485 invoked by uid 500); 5 Oct 2010 23:20:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21457 invoked by uid 500); 5 Oct 2010 23:20:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21449 invoked by uid 99); 5 Oct 2010 23:20:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 23:20:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Oct 2010 23:20:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o95NKYwc028537
	for <common-issues@hadoop.apache.org>; Tue, 5 Oct 2010 23:20:35 GMT
Message-ID: <3118408.562871286320834965.JavaMail.jira@thor>
Date: Tue, 5 Oct 2010 19:20:34 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6356) Add a Cache for AbstractFileSystem
 in the new FileContext/AbstractFileSystem framework.
In-Reply-To: <2126828437.1257203939433.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918314#action_12918314 ] 

Sanjay Radia commented on HADOOP-6356:
--------------------------------------

We could add a cache at the rpcProxy layer. 
This will avoid the extra round trip for FileContext operations to a file system other then the default filesystem. However the current bug of where the NN reboots with the different version will remain.

> Add a Cache for AbstractFileSystem in the new FileContext/AbstractFileSystem framework.
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6356
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6356
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.22.0
>
>
> The new filesystem framework, FileContext and AbstractFileSystem does not implement a cache for AbstractFileSystem.
> This Jira proposes to add a cache to the new framework just like with the old FileSystem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10931-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 06 01:10:56 2010
Return-Path: <common-issues-return-10931-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92854 invoked from network); 6 Oct 2010 01:10:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Oct 2010 01:10:56 -0000
Received: (qmail 10629 invoked by uid 500); 6 Oct 2010 01:10:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10520 invoked by uid 500); 6 Oct 2010 01:10:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10512 invoked by uid 99); 6 Oct 2010 01:10:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 01:10:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 01:10:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o961AYWB000461
	for <common-issues@hadoop.apache.org>; Wed, 6 Oct 2010 01:10:34 GMT
Message-ID: <12731626.565251286327434588.JavaMail.jira@thor>
Date: Tue, 5 Oct 2010 21:10:34 -0400 (EDT)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6988) Add support for reading multiple
 hadoop delegation token files
In-Reply-To: <17443930.539541286234913119.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918368#action_12918368 ] 

Aaron T. Myers commented on HADOOP-6988:
----------------------------------------

Allen: I agree with Owen - it doesn't make sense to remove this environment variable. It is primarily used internally, though its use is documented in hdfs_user_guide.xml and mapred_tutorial.xml. The interaction between this and the conf var is that all tokens specified via either method are added to a single credentials object - nothing's going to take precedence.

Owen: the motivation for this change is not with regard to passing delegation tokens from job to tasks, but rather with submitting jobs in the first place, which is another use for HADOOP_TOKEN_FILE_LOCATION. I'd like to be able to specify multiple tokens (gotten from fetchdt, or via other means) which a job could use to, for example, authenticate to multiple NNs and JTs. I considered creating a tool which would be capable of merging multiple delegation token files into one, but this seemed like a cleaner solution.

Good point on making this comma-separated. I'll definitely do that.

> Add support for reading multiple hadoop delegation token files
> --------------------------------------------------------------
>
>                 Key: HADOOP-6988
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6988
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>
> It would be nice if there were a way to specify multiple delegation token files via the HADOOP_TOKEN_FILE_LOCATION environment variable and the "mapreduce.job.credentials.binary" configuration value. I suggest a colon-separated list of paths, each of which is read as a separate delegation token file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10932-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 06 01:40:54 2010
Return-Path: <common-issues-return-10932-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2086 invoked from network); 6 Oct 2010 01:40:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Oct 2010 01:40:54 -0000
Received: (qmail 27224 invoked by uid 500); 6 Oct 2010 01:40:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27194 invoked by uid 500); 6 Oct 2010 01:40:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27186 invoked by uid 99); 6 Oct 2010 01:40:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 01:40:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 01:40:52 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o961eWqW001068
	for <common-issues@hadoop.apache.org>; Wed, 6 Oct 2010 01:40:32 GMT
Message-ID: <21563065.541286329232015.JavaMail.jira@thor>
Date: Tue, 5 Oct 2010 21:40:32 -0400 (EDT)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6988) Add support for reading multiple
 hadoop delegation token files
In-Reply-To: <17443930.539541286234913119.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918379#action_12918379 ] 

Alejandro Abdelnur commented on HADOOP-6988:
--------------------------------------------

PLEASE, DO NOT REMOVE THIS VARIABLE:

Oozie relies on this variable to be able to dispatch mapreduce/streaming/pipes/pig/hive/sqoop actions. Oozie uses a launcher MR job to start all those action types. This launcher MR job uses the variable to propagate the delegation token to the code that launches the corresponding action.

> Add support for reading multiple hadoop delegation token files
> --------------------------------------------------------------
>
>                 Key: HADOOP-6988
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6988
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>
> It would be nice if there were a way to specify multiple delegation token files via the HADOOP_TOKEN_FILE_LOCATION environment variable and the "mapreduce.job.credentials.binary" configuration value. I suggest a colon-separated list of paths, each of which is read as a separate delegation token file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10933-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 06 03:05:00 2010
Return-Path: <common-issues-return-10933-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38476 invoked from network); 6 Oct 2010 03:04:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Oct 2010 03:04:59 -0000
Received: (qmail 90988 invoked by uid 500); 6 Oct 2010 03:04:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90729 invoked by uid 500); 6 Oct 2010 03:04:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90720 invoked by uid 99); 6 Oct 2010 03:04:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 03:04:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 03:04:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9634WwH005515
	for <common-issues@hadoop.apache.org>; Wed, 6 Oct 2010 03:04:33 GMT
Message-ID: <2058929.1211286334272837.JavaMail.jira@thor>
Date: Tue, 5 Oct 2010 23:04:32 -0400 (EDT)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6988) Add support for reading multiple
 hadoop delegation token files
In-Reply-To: <17443930.539541286234913119.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-6988:
-----------------------------------

    Attachment: hadoop-6988.0.txt

Adding support for HADOOP_TOKEN_FILE_LOCATION being interpreted as a comma-separated list of paths to delegation token files.

> Add support for reading multiple hadoop delegation token files
> --------------------------------------------------------------
>
>                 Key: HADOOP-6988
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6988
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>         Attachments: hadoop-6988.0.txt
>
>
> It would be nice if there were a way to specify multiple delegation token files via the HADOOP_TOKEN_FILE_LOCATION environment variable and the "mapreduce.job.credentials.binary" configuration value. I suggest a colon-separated list of paths, each of which is read as a separate delegation token file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10934-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 06 03:14:59 2010
Return-Path: <common-issues-return-10934-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47352 invoked from network); 6 Oct 2010 03:14:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Oct 2010 03:14:58 -0000
Received: (qmail 4707 invoked by uid 500); 6 Oct 2010 03:14:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4432 invoked by uid 500); 6 Oct 2010 03:14:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4416 invoked by uid 99); 6 Oct 2010 03:14:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 03:14:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 03:14:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o963EVW5007481
	for <common-issues@hadoop.apache.org>; Wed, 6 Oct 2010 03:14:32 GMT
Message-ID: <15331317.1311286334871862.JavaMail.jira@thor>
Date: Tue, 5 Oct 2010 23:14:31 -0400 (EDT)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6988) Add support for reading multiple
 hadoop delegation token files
In-Reply-To: <17443930.539541286234913119.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-6988:
-----------------------------------

    Attachment: hadoop-6988.1.txt

Same patch, this time with -p0.

> Add support for reading multiple hadoop delegation token files
> --------------------------------------------------------------
>
>                 Key: HADOOP-6988
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6988
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>         Attachments: hadoop-6988.0.txt, hadoop-6988.1.txt
>
>
> It would be nice if there were a way to specify multiple delegation token files via the HADOOP_TOKEN_FILE_LOCATION environment variable and the "mapreduce.job.credentials.binary" configuration value. I suggest a colon-separated list of paths, each of which is read as a separate delegation token file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10935-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 06 04:22:00 2010
Return-Path: <common-issues-return-10935-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78894 invoked from network); 6 Oct 2010 04:22:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Oct 2010 04:22:00 -0000
Received: (qmail 68843 invoked by uid 500); 6 Oct 2010 04:22:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68690 invoked by uid 500); 6 Oct 2010 04:21:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68676 invoked by uid 99); 6 Oct 2010 04:21:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 04:21:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 04:21:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o964LWuL014217
	for <common-issues@hadoop.apache.org>; Wed, 6 Oct 2010 04:21:32 GMT
Message-ID: <6973099.1771286338892414.JavaMail.jira@thor>
Date: Wed, 6 Oct 2010 00:21:32 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6895) Native Libraries do not load if a
 different platform signature is returned from
 org.apache.hadoop.util.PlatformName
In-Reply-To: <8210696.123741280779279985.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918412#action_12918412 ] 

Tom White commented on HADOOP-6895:
-----------------------------------

+1 looks good to me. It's worth adding a comment in the file which explains why this is needed. Same for HADOOP-6923.

> Native Libraries do not load if a different platform signature is returned from org.apache.hadoop.util.PlatformName
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6895
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6895
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>         Environment: SLES 10, IBM Java 6, Hadoop 0.21.0-rc0
>            Reporter: Stephen Watt
>            Priority: Minor
>             Fix For: 0.21.1, 0.22.0
>
>         Attachments: HADOOP-6895.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> bin/hadoop-config.sh has an environment variable called JAVA_PLATFORM which is set to to the results returned by org.apache.hadoop.util.PlatformName . These results are sometimes unique to the JRE being used. Although the value returned for 64 Bit Sun/Oracle Java and 64 Bit IBM Java is the same, it is different for the corresponding 32 Bit JREs. 
> The issue is that the value returned is used in creating the path to the native libraries on disk, i.e ${HADOOP_COMMON_HOME}/lib/native/${JAVA_PLATFORM}
> Since the path on disk is fixed with the Sun JRE value /lib/native/Linux-i386-32 it therefore fails when it attempts to load the native libraries with the value returned with 32 Bit IBM Java, /lib/native/Linux-x86-32

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10936-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 06 04:33:59 2010
Return-Path: <common-issues-return-10936-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80566 invoked from network); 6 Oct 2010 04:33:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Oct 2010 04:33:59 -0000
Received: (qmail 78439 invoked by uid 500); 6 Oct 2010 04:33:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78282 invoked by uid 500); 6 Oct 2010 04:33:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78274 invoked by uid 99); 6 Oct 2010 04:33:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 04:33:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 04:33:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o964XYe3014388
	for <common-issues@hadoop.apache.org>; Wed, 6 Oct 2010 04:33:34 GMT
Message-ID: <4112371.1981286339614467.JavaMail.jira@thor>
Date: Wed, 6 Oct 2010 00:33:34 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6941) Hadoop 0.21 will not work on non-SUN
 JREs due to use of com.sun.security in
 org/apache/hadoop/security/UserGroupInformation.java
In-Reply-To: <20695342.63471283897793901.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6941:
------------------------------

    Fix Version/s:     (was: 0.21.0)

> Hadoop 0.21 will not work on non-SUN JREs due to use of com.sun.security in org/apache/hadoop/security/UserGroupInformation.java
> --------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6941
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6941
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>         Environment: SLES 11, Apache Harmony 6 and SLES 11, IBM Java 6
>            Reporter: Stephen Watt
>             Fix For: 0.21.1, 0.22.0
>
>
> Attempting to format the namenode or attempting to start Hadoop using Apache Harmony or the IBM Java JREs results in the following exception:
> 10/09/07 16:35:05 ERROR namenode.NameNode: java.lang.NoClassDefFoundError: com.sun.security.auth.UnixPrincipal
> 	at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:223)
> 	at java.lang.J9VMInternals.initializeImpl(Native Method)
> 	at java.lang.J9VMInternals.initialize(J9VMInternals.java:200)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setConfigurationParameters(FSNamesystem.java:420)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:391)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1240)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1348)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368)
> Caused by: java.lang.ClassNotFoundException: com.sun.security.auth.UnixPrincipal
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:421)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:652)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:346)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:618)
> 	... 8 more
> This is a negative regression as previous versions of Hadoop worked with these JREs

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10937-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 06 04:39:58 2010
Return-Path: <common-issues-return-10937-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81357 invoked from network); 6 Oct 2010 04:39:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Oct 2010 04:39:58 -0000
Received: (qmail 86985 invoked by uid 500); 6 Oct 2010 04:39:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86872 invoked by uid 500); 6 Oct 2010 04:39:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86864 invoked by uid 99); 6 Oct 2010 04:39:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 04:39:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 04:39:52 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o964dVPN014461
	for <common-issues@hadoop.apache.org>; Wed, 6 Oct 2010 04:39:31 GMT
Message-ID: <10502408.2021286339971154.JavaMail.jira@thor>
Date: Wed, 6 Oct 2010 00:39:31 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6954) Sources JARs are not correctly
 published to the Maven repository
In-Reply-To: <19843693.216391284595892946.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918415#action_12918415 ] 

Tom White commented on HADOOP-6954:
-----------------------------------

I'd like to commit this unless there are any objections.

> Sources JARs are not correctly published to the Maven repository
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6954
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6954
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.21.1
>
>         Attachments: HADOOP-6954.patch
>
>
> When you try to "close" the staging repository to make it visible to the public the operation fails (see https://issues.apache.org/jira/browse/HDFS-1292?focusedCommentId=12909953&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12909953 for the type of error).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10938-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 06 04:47:57 2010
Return-Path: <common-issues-return-10938-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82384 invoked from network); 6 Oct 2010 04:47:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Oct 2010 04:47:57 -0000
Received: (qmail 91797 invoked by uid 500); 6 Oct 2010 04:47:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91646 invoked by uid 500); 6 Oct 2010 04:47:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91636 invoked by uid 99); 6 Oct 2010 04:47:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 04:47:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 04:47:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o964lWSB014612
	for <common-issues@hadoop.apache.org>; Wed, 6 Oct 2010 04:47:32 GMT
Message-ID: <10436909.2261286340452823.JavaMail.jira@thor>
Date: Wed, 6 Oct 2010 00:47:32 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6988) Add support for reading multiple
 hadoop delegation token files
In-Reply-To: <17443930.539541286234913119.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918419#action_12918419 ] 

Allen Wittenauer commented on HADOOP-6988:
------------------------------------------

I'm sorry, but it is completely short sighted to have a single use env var like this.  If we need to modify the tasks environment for something else, are we going to introduce another environment variable?  How many are too many?



> Add support for reading multiple hadoop delegation token files
> --------------------------------------------------------------
>
>                 Key: HADOOP-6988
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6988
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>         Attachments: hadoop-6988.0.txt, hadoop-6988.1.txt
>
>
> It would be nice if there were a way to specify multiple delegation token files via the HADOOP_TOKEN_FILE_LOCATION environment variable and the "mapreduce.job.credentials.binary" configuration value. I suggest a colon-separated list of paths, each of which is read as a separate delegation token file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10939-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 06 06:19:55 2010
Return-Path: <common-issues-return-10939-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7276 invoked from network); 6 Oct 2010 06:19:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Oct 2010 06:19:55 -0000
Received: (qmail 48682 invoked by uid 500); 6 Oct 2010 06:19:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48577 invoked by uid 500); 6 Oct 2010 06:19:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48569 invoked by uid 99); 6 Oct 2010 06:19:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 06:19:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 06:19:52 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o966JVJM015870
	for <common-issues@hadoop.apache.org>; Wed, 6 Oct 2010 06:19:31 GMT
Message-ID: <30631831.2791286345971699.JavaMail.jira@thor>
Date: Wed, 6 Oct 2010 02:19:31 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6984) NPE from
 SequenceFile::Writer.CompressionCodecOption
In-Reply-To: <10251121.516661286092293049.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6984:
----------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

This looks great. It passes unit tests.

I just committed this. Thanks, Chris!

> NPE from SequenceFile::Writer.CompressionCodecOption
> ----------------------------------------------------
>
>                 Key: HADOOP-6984
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6984
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Chris Douglas
>            Assignee: Chris Douglas
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: 6984-0.patch, 6984-1.patch, 6984-2.patch, 6984-3.patch
>
>
> The deprecated HADOOP-6856 constructors can create a compressed writers with a null-wrapped {{CompressionCodecOption}}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10940-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 06 06:19:59 2010
Return-Path: <common-issues-return-10940-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7334 invoked from network); 6 Oct 2010 06:19:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Oct 2010 06:19:59 -0000
Received: (qmail 48799 invoked by uid 500); 6 Oct 2010 06:19:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48773 invoked by uid 500); 6 Oct 2010 06:19:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48765 invoked by uid 99); 6 Oct 2010 06:19:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 06:19:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 06:19:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o966JY4A015883
	for <common-issues@hadoop.apache.org>; Wed, 6 Oct 2010 06:19:34 GMT
Message-ID: <22821360.2831286345974138.JavaMail.jira@thor>
Date: Wed, 6 Oct 2010 02:19:34 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6984) NPE from
 SequenceFile::Writer.CompressionCodecOption
In-Reply-To: <10251121.516661286092293049.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6984:
----------------------------------

    Component/s:     (was: fs)
                 io

> NPE from SequenceFile::Writer.CompressionCodecOption
> ----------------------------------------------------
>
>                 Key: HADOOP-6984
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6984
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Chris Douglas
>            Assignee: Chris Douglas
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: 6984-0.patch, 6984-1.patch, 6984-2.patch, 6984-3.patch
>
>
> The deprecated HADOOP-6856 constructors can create a compressed writers with a null-wrapped {{CompressionCodecOption}}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10941-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 06 17:25:56 2010
Return-Path: <common-issues-return-10941-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27682 invoked from network); 6 Oct 2010 17:25:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Oct 2010 17:25:55 -0000
Received: (qmail 40233 invoked by uid 500); 6 Oct 2010 17:25:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40155 invoked by uid 500); 6 Oct 2010 17:25:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40147 invoked by uid 99); 6 Oct 2010 17:25:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 17:25:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 17:25:52 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o96HPVgD010425
	for <common-issues@hadoop.apache.org>; Wed, 6 Oct 2010 17:25:31 GMT
Message-ID: <21352235.10671286385931110.JavaMail.jira@thor>
Date: Wed, 6 Oct 2010 13:25:31 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6991) SequenceFile::Reader discards length
 for files, does not call openFile
In-Reply-To: <22609732.541901286245712924.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-6991:
----------------------------------

    Status: Patch Available  (was: Open)

> SequenceFile::Reader discards length for files, does not call openFile
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6991
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6991
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Chris Douglas
>            Priority: Minor
>         Attachments: 6991-0.patch, 6991-1.patch
>
>
> While the sorting and merging in {{SequenceFile}} is deprecated and {{openFile}} is archaic, the semantics should remain consistent.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10942-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 06 20:05:57 2010
Return-Path: <common-issues-return-10942-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85771 invoked from network); 6 Oct 2010 20:05:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Oct 2010 20:05:56 -0000
Received: (qmail 44573 invoked by uid 500); 6 Oct 2010 20:05:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44539 invoked by uid 500); 6 Oct 2010 20:05:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44530 invoked by uid 99); 6 Oct 2010 20:05:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 20:05:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 20:05:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o96K5Wc8013102
	for <common-issues@hadoop.apache.org>; Wed, 6 Oct 2010 20:05:32 GMT
Message-ID: <24562186.13231286395532223.JavaMail.jira@thor>
Date: Wed, 6 Oct 2010 16:05:32 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6988) Add support for reading multiple
 hadoop delegation token files
In-Reply-To: <17443930.539541286234913119.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918648#action_12918648 ] 

Eli Collins commented on HADOOP-6988:
-------------------------------------

Being able to specify both MR and HDFS delegation token files upfront when submitting a job seems reasonable, avoids requiring all clients kinit or use a tool to merge token files.   Env variable philosophy aside (Allen, want to open a jira with an alternative?) does anyone object to modifying this existing variable so more than one file can be specified?


> Add support for reading multiple hadoop delegation token files
> --------------------------------------------------------------
>
>                 Key: HADOOP-6988
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6988
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>         Attachments: hadoop-6988.0.txt, hadoop-6988.1.txt
>
>
> It would be nice if there were a way to specify multiple delegation token files via the HADOOP_TOKEN_FILE_LOCATION environment variable and the "mapreduce.job.credentials.binary" configuration value. I suggest a colon-separated list of paths, each of which is read as a separate delegation token file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10943-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 06 21:28:54 2010
Return-Path: <common-issues-return-10943-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11605 invoked from network); 6 Oct 2010 21:28:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Oct 2010 21:28:53 -0000
Received: (qmail 54268 invoked by uid 500); 6 Oct 2010 21:28:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54228 invoked by uid 500); 6 Oct 2010 21:28:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54220 invoked by uid 99); 6 Oct 2010 21:28:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 21:28:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 21:28:52 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o96LSWow014631
	for <common-issues@hadoop.apache.org>; Wed, 6 Oct 2010 21:28:32 GMT
Message-ID: <24758682.15361286400512441.JavaMail.jira@thor>
Date: Wed, 6 Oct 2010 17:28:32 -0400 (EDT)
From: "Nicolas Spiegelberg (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6986) SequenceFile.Reader should
 distinguish between Network IOE and Parsing IOE
In-Reply-To: <31980992.537951286228195160.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicolas Spiegelberg updated HADOOP-6986:
----------------------------------------

    Status: Patch Available  (was: Open)

> SequenceFile.Reader should distinguish between Network IOE and Parsing IOE
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6986
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6986
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.21.1, 0.22.0, 0.20-append
>            Reporter: Nicolas Spiegelberg
>            Priority: Minor
>             Fix For: 0.21.1, 0.22.0, 0.20-append
>
>         Attachments: HADOOP-6986_0.21.patch, HADOOP-6986_20-append.patch
>
>
> The SequenceFile.Reader api should give the user an easy way to distinguish between a Network/Low-level IOE and a Parsing IOE.  The use case appeared recently in the HBase project:
> Originally, if a RegionServer got an IOE from HDFS while opening a region file, it would abort the open and let the HMaster reassign the region.  The assumption being that this is a network failure that will likely disappear at a later time or different partition of the network.  However, if HBase gets parsing exceptions, we want to log the problem and continue opening the region anyways, because parsing is an idempotent problem and retries won't fix this issue.
> Although this problem was found in HBase, it seems to be a generic problem of being able to more easily identify idempotent vs transient errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10944-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 06 22:44:54 2010
Return-Path: <common-issues-return-10944-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37443 invoked from network); 6 Oct 2010 22:44:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Oct 2010 22:44:54 -0000
Received: (qmail 67976 invoked by uid 500); 6 Oct 2010 22:44:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67837 invoked by uid 500); 6 Oct 2010 22:44:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67829 invoked by uid 99); 6 Oct 2010 22:44:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 22:44:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 22:44:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o96MiWSg015739
	for <common-issues@hadoop.apache.org>; Wed, 6 Oct 2010 22:44:32 GMT
Message-ID: <20906155.16341286405072485.JavaMail.jira@thor>
Date: Wed, 6 Oct 2010 18:44:32 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6916) Implement append operation for
 S3FileSystem
In-Reply-To: <6991431.365871281969257661.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6916:
--------------------------------

    Affects Version/s: 0.22.0
        Fix Version/s: 0.22.0
           Issue Type: New Feature  (was: Bug)

> Implement append operation for S3FileSystem
> -------------------------------------------
>
>                 Key: HADOOP-6916
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6916
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs/s3
>    Affects Versions: 0.20.2, 0.22.0
>            Reporter: Oleg Aleshko
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: s3_append1.patch
>
>
> Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException("Not supported");
> S3FileSystem should be able to support appending, possibly via common block storage logic.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10945-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 06 23:02:56 2010
Return-Path: <common-issues-return-10945-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39677 invoked from network); 6 Oct 2010 23:02:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Oct 2010 23:02:56 -0000
Received: (qmail 80203 invoked by uid 500); 6 Oct 2010 23:02:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80145 invoked by uid 500); 6 Oct 2010 23:02:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80135 invoked by uid 99); 6 Oct 2010 23:02:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 23:02:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 23:02:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o96N2VQ4015982
	for <common-issues@hadoop.apache.org>; Wed, 6 Oct 2010 23:02:31 GMT
Message-ID: <26161850.16451286406151616.JavaMail.jira@thor>
Date: Wed, 6 Oct 2010 19:02:31 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6992) Update website for recent subproject
 departures
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Update website for recent subproject departures
-----------------------------------------------

                 Key: HADOOP-6992
                 URL: https://issues.apache.org/jira/browse/HADOOP-6992
             Project: Hadoop Common
          Issue Type: Improvement
          Components: documentation
    Affects Versions: site
            Reporter: Doug Cutting
            Assignee: Doug Cutting
             Fix For: site


A number of subprojects have left Hadoop yet the website's not been fully updated to reflect that.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10946-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 06 23:04:57 2010
Return-Path: <common-issues-return-10946-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40140 invoked from network); 6 Oct 2010 23:04:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Oct 2010 23:04:57 -0000
Received: (qmail 85450 invoked by uid 500); 6 Oct 2010 23:04:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85104 invoked by uid 500); 6 Oct 2010 23:04:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85053 invoked by uid 99); 6 Oct 2010 23:04:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 23:04:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 23:04:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o96N4W86016057
	for <common-issues@hadoop.apache.org>; Wed, 6 Oct 2010 23:04:32 GMT
Message-ID: <8661907.16561286406272393.JavaMail.jira@thor>
Date: Wed, 6 Oct 2010 19:04:32 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6992) Update website for recent subproject
 departures
In-Reply-To: <26161850.16451286406151616.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-6992:
---------------------------------

    Attachment: HADOOP-6992.patch

Here's a patch that:
 - removes tabs for non-subprojects
 - moves former sub-projects to the related-projects nav menu
 - adds a related project listing to content of the front page
 - updates the list of PMC members
 - adds redirects for former subprojects to their new homes

> Update website for recent subproject departures
> -----------------------------------------------
>
>                 Key: HADOOP-6992
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6992
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: site
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: site
>
>         Attachments: HADOOP-6992.patch
>
>
> A number of subprojects have left Hadoop yet the website's not been fully updated to reflect that.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10947-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 06 23:14:56 2010
Return-Path: <common-issues-return-10947-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46124 invoked from network); 6 Oct 2010 23:14:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Oct 2010 23:14:56 -0000
Received: (qmail 95698 invoked by uid 500); 6 Oct 2010 23:14:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95632 invoked by uid 500); 6 Oct 2010 23:14:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95624 invoked by uid 99); 6 Oct 2010 23:14:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 23:14:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 23:14:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o96NEYLN016281
	for <common-issues@hadoop.apache.org>; Wed, 6 Oct 2010 23:14:35 GMT
Message-ID: <28209305.16831286406874960.JavaMail.jira@thor>
Date: Wed, 6 Oct 2010 19:14:34 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6916) Implement append operation for
 S3FileSystem
In-Reply-To: <6991431.365871281969257661.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918729#action_12918729 ] 

Eli Collins commented on HADOOP-6916:
-------------------------------------

Hey Oleg,

* Doesn't pos and filePos need to be updated when opening for append?
* Doesn't S3OutputStream#setBlocks need to be synchronized? Why does it copy the blocks?
* Make sure the unit tests exercise calling close after append.
* Have you tested this on S3? eg by setting test.fs.s3.name in src/test/core-site.xml?

Hudson currently running, please use ant test-patch from an svn checkout.

Thanks,
Eli

> Implement append operation for S3FileSystem
> -------------------------------------------
>
>                 Key: HADOOP-6916
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6916
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs/s3
>    Affects Versions: 0.20.2, 0.22.0
>            Reporter: Oleg Aleshko
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: s3_append1.patch
>
>
> Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException("Not supported");
> S3FileSystem should be able to support appending, possibly via common block storage logic.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10948-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 06 23:26:57 2010
Return-Path: <common-issues-return-10948-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48239 invoked from network); 6 Oct 2010 23:26:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Oct 2010 23:26:57 -0000
Received: (qmail 3536 invoked by uid 500); 6 Oct 2010 23:26:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3381 invoked by uid 500); 6 Oct 2010 23:26:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3373 invoked by uid 99); 6 Oct 2010 23:26:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 23:26:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 23:26:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o96NQXcg016417
	for <common-issues@hadoop.apache.org>; Wed, 6 Oct 2010 23:26:33 GMT
Message-ID: <17409998.16911286407592979.JavaMail.jira@thor>
Date: Wed, 6 Oct 2010 19:26:32 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6916) Implement append operation for
 S3FileSystem
In-Reply-To: <6991431.365871281969257661.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918731#action_12918731 ] 

Eli Collins commented on HADOOP-6916:
-------------------------------------

Forgot to mention this could use some additional tests, eg appending to a file already open for append, appending more than a block, etc. See the hdfs append tests for additional ideas.

> Implement append operation for S3FileSystem
> -------------------------------------------
>
>                 Key: HADOOP-6916
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6916
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs/s3
>    Affects Versions: 0.20.2, 0.22.0
>            Reporter: Oleg Aleshko
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: s3_append1.patch
>
>
> Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException("Not supported");
> S3FileSystem should be able to support appending, possibly via common block storage logic.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10949-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 07 00:51:57 2010
Return-Path: <common-issues-return-10949-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73682 invoked from network); 7 Oct 2010 00:51:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Oct 2010 00:51:57 -0000
Received: (qmail 60231 invoked by uid 500); 7 Oct 2010 00:51:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60173 invoked by uid 500); 7 Oct 2010 00:51:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60165 invoked by uid 99); 7 Oct 2010 00:51:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 00:51:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 00:51:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o970pW3a017566
	for <common-issues@hadoop.apache.org>; Thu, 7 Oct 2010 00:51:32 GMT
Message-ID: <31243551.18031286412692617.JavaMail.jira@thor>
Date: Wed, 6 Oct 2010 20:51:32 -0400 (EDT)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6977) Herriot daemon clients should vend
 statistics
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6977?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D129=
18751#action_12918751 ]=20

Tanping Wang commented on HADOOP-6977:
--------------------------------------

# I looked at HADOOP-6977.patch.  Overall I think it is good.  One concern =
that I had was that in the methods of=20
-- getJmxPortNumber()=20
and
-- isJmxEnabled() ,=20

"HADOOP_OPTS" was hard-coded and searched. =20

This require us to enable remote JMX by putting all the JMX options into HA=
DOOP_OPTS.  In fact, we can enable JMX ( with specific port number ) by mod=
ifying command specific options, e.g.

# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS=3D"-Dcom.sun.management.jmxremote $HADOOP_NAMEN=
ODE_OPTS -Dcom.sun.management.jmxremote.port=3D8004=E2=80=B3
export HADOOP_SECONDARYNAMENODE_OPTS=3D"-Dcom.sun.management.jmxremote $HAD=
OOP_SECONDARYNAMENODE_OPTS -Dcom.sun.management.jmxremote.port=3D8005=E2=80=
=B3
export HADOOP_DATANODE_OPTS=3D"-Dcom.sun.management.jmxremote $HADOOP_DATAN=
ODE_OPTS -Dcom.sun.management.jmxremote.port=3D8006=E2=80=B3
export HADOOP_BALANCER_OPTS=3D"-Dcom.sun.management.jmxremote $HADOOP_BALAN=
CER_OPTS -Dcom.sun.management.jmxremote.port=3D8007=E2=80=B3
export HADOOP_JOBTRACKER_OPTS=3D"-Dcom.sun.management.jmxremote $HADOOP_JOB=
TRACKER_OPTS -Dcom.sun.management.jmxremote.port=3D8008=E2=80=B3
export HADOOP_TASKTRACKER_OPTS=3D"-Dcom.sun.management.jmxremote.port=3D800=
9=E2=80=B3


There is also a typo
/**=20
Create connection with the remove JMX server at given host and port

I believe you mean remote but not remove?

# FinderTest.java
This test should be refactor to follow Junit 4 ?



> Herriot daemon clients should vend statistics
> ---------------------------------------------
>
>                 Key: HADOOP-6977
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6977
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: FinderTest.java, HADOOP-6977.patch, HADOOP-6977.patc=
h, HADOOP-6977.patch, HADOOP-6977.y20S.patch, HADOOP-6977.y20S.patch
>
>
> The HDFS web user interface serves useful information through dfshealth.j=
sp and dfsnodelist.jsp.
> The Herriot interface to Hadoop cluster daemons would benefit from the ad=
dition of some way to channel metics information.

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10950-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 07 04:18:59 2010
Return-Path: <common-issues-return-10950-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31961 invoked from network); 7 Oct 2010 04:18:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Oct 2010 04:18:58 -0000
Received: (qmail 78454 invoked by uid 500); 7 Oct 2010 04:18:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77237 invoked by uid 500); 7 Oct 2010 04:18:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77220 invoked by uid 99); 7 Oct 2010 04:18:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 04:18:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 04:18:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o974IWST020582
	for <common-issues@hadoop.apache.org>; Thu, 7 Oct 2010 04:18:32 GMT
Message-ID: <22210278.19631286425112055.JavaMail.jira@thor>
Date: Thu, 7 Oct 2010 00:18:32 -0400 (EDT)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6993) Broken link on cluster setup page of
 docs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Broken link on cluster setup page of docs
-----------------------------------------

                 Key: HADOOP-6993
                 URL: https://issues.apache.org/jira/browse/HADOOP-6993
             Project: Hadoop Common
          Issue Type: Task
          Components: documentation
            Reporter: Aaron T. Myers


The link on http://hadoop.apache.org/common/docs/current/cluster_setup.html#Configuring+the+Hadoop+Daemons to core-default.xml is presently:

{quote}
http://hadoop.apache.org/common/docs/current/common-default.html
{quote}

but it should be:

{quote}
http://hadoop.apache.org/common/docs/current/core-default.html
{quote}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10951-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 07 04:54:58 2010
Return-Path: <common-issues-return-10951-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41665 invoked from network); 7 Oct 2010 04:54:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Oct 2010 04:54:58 -0000
Received: (qmail 99404 invoked by uid 500); 7 Oct 2010 04:54:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99244 invoked by uid 500); 7 Oct 2010 04:54:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99236 invoked by uid 99); 7 Oct 2010 04:54:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 04:54:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 04:54:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o974sVBg021019
	for <common-issues@hadoop.apache.org>; Thu, 7 Oct 2010 04:54:32 GMT
Message-ID: <12867933.19861286427271956.JavaMail.jira@thor>
Date: Thu, 7 Oct 2010 00:54:31 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6977) Herriot daemon clients should vend
 statistics
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918786#action_12918786 ] 

Konstantin Boudnik commented on HADOOP-6977:
--------------------------------------------

bq. In fact, we can enable JMX ( with specific port number ) by modifying command specific options
I think this is valid and I'll make the change to look for JMX specific parameters in a daemon specific options (i.e. HADOOP_NAMENODE_OPTS) as well as in HADOOP_OPTS. That should cover all the bases.

Typo: check, good catch.

FinderTest was an example of loops optimization and isn't a part of the patch.

> Herriot daemon clients should vend statistics
> ---------------------------------------------
>
>                 Key: HADOOP-6977
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6977
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: FinderTest.java, HADOOP-6977.patch, HADOOP-6977.patch, HADOOP-6977.patch, HADOOP-6977.y20S.patch, HADOOP-6977.y20S.patch
>
>
> The HDFS web user interface serves useful information through dfshealth.jsp and dfsnodelist.jsp.
> The Herriot interface to Hadoop cluster daemons would benefit from the addition of some way to channel metics information.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10952-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 07 05:56:56 2010
Return-Path: <common-issues-return-10952-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54394 invoked from network); 7 Oct 2010 05:56:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Oct 2010 05:56:56 -0000
Received: (qmail 42980 invoked by uid 500); 7 Oct 2010 05:56:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42883 invoked by uid 500); 7 Oct 2010 05:56:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42249 invoked by uid 99); 7 Oct 2010 05:56:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 05:56:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 05:56:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o975uWms021924
	for <common-issues@hadoop.apache.org>; Thu, 7 Oct 2010 05:56:33 GMT
Message-ID: <18921037.20601286430992803.JavaMail.jira@thor>
Date: Thu, 7 Oct 2010 01:56:32 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6992) Update website for recent
 subproject departures
In-Reply-To: <26161850.16451286406151616.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918792#action_12918792 ] 

Owen O'Malley commented on HADOOP-6992:
---------------------------------------

It looks good, except that we need to wait for their new web site to be up before we redirect their current url to it.

> Update website for recent subproject departures
> -----------------------------------------------
>
>                 Key: HADOOP-6992
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6992
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: site
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: site
>
>         Attachments: HADOOP-6992.patch
>
>
> A number of subprojects have left Hadoop yet the website's not been fully updated to reflect that.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10953-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 07 05:58:54 2010
Return-Path: <common-issues-return-10953-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55006 invoked from network); 7 Oct 2010 05:58:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Oct 2010 05:58:54 -0000
Received: (qmail 44481 invoked by uid 500); 7 Oct 2010 05:58:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44448 invoked by uid 500); 7 Oct 2010 05:58:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44440 invoked by uid 99); 7 Oct 2010 05:58:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 05:58:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 05:58:51 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o975wVG3021956
	for <common-issues@hadoop.apache.org>; Thu, 7 Oct 2010 05:58:31 GMT
Message-ID: <23806310.20641286431111196.JavaMail.jira@thor>
Date: Thu, 7 Oct 2010 01:58:31 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6993) Broken link on cluster setup page of
 docs
In-Reply-To: <22210278.19631286425112055.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6993:
--------------------------------

    Affects Version/s: 0.21.0
        Fix Version/s: 0.22.0
                       0.21.1
             Assignee: Eli Collins
           Issue Type: Bug  (was: Task)

> Broken link on cluster setup page of docs
> -----------------------------------------
>
>                 Key: HADOOP-6993
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6993
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.21.0
>            Reporter: Aaron T. Myers
>            Assignee: Eli Collins
>             Fix For: 0.21.1, 0.22.0
>
>
> The link on http://hadoop.apache.org/common/docs/current/cluster_setup.html#Configuring+the+Hadoop+Daemons to core-default.xml is presently:
> {quote}
> http://hadoop.apache.org/common/docs/current/common-default.html
> {quote}
> but it should be:
> {quote}
> http://hadoop.apache.org/common/docs/current/core-default.html
> {quote}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10954-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 07 06:00:58 2010
Return-Path: <common-issues-return-10954-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55536 invoked from network); 7 Oct 2010 06:00:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Oct 2010 06:00:57 -0000
Received: (qmail 46168 invoked by uid 500); 7 Oct 2010 06:00:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46014 invoked by uid 500); 7 Oct 2010 06:00:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46001 invoked by uid 99); 7 Oct 2010 06:00:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 06:00:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 06:00:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9760Wnc022004
	for <common-issues@hadoop.apache.org>; Thu, 7 Oct 2010 06:00:32 GMT
Message-ID: <24823572.20671286431232241.JavaMail.jira@thor>
Date: Thu, 7 Oct 2010 02:00:32 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6993) Broken link on cluster setup page of
 docs
In-Reply-To: <22210278.19631286425112055.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6993:
--------------------------------

    Attachment: hadoop-6993-22-1.patch
                hadoop-6993-21-1.patch

Thanks Aaron.  Patches for 21 and similar issue on trunk attached.

> Broken link on cluster setup page of docs
> -----------------------------------------
>
>                 Key: HADOOP-6993
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6993
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.21.0
>            Reporter: Aaron T. Myers
>            Assignee: Eli Collins
>             Fix For: 0.21.1, 0.22.0
>
>         Attachments: hadoop-6993-21-1.patch, hadoop-6993-22-1.patch
>
>
> The link on http://hadoop.apache.org/common/docs/current/cluster_setup.html#Configuring+the+Hadoop+Daemons to core-default.xml is presently:
> {quote}
> http://hadoop.apache.org/common/docs/current/common-default.html
> {quote}
> but it should be:
> {quote}
> http://hadoop.apache.org/common/docs/current/core-default.html
> {quote}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10955-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 07 06:08:56 2010
Return-Path: <common-issues-return-10955-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60538 invoked from network); 7 Oct 2010 06:08:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Oct 2010 06:08:55 -0000
Received: (qmail 48228 invoked by uid 500); 7 Oct 2010 06:08:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47448 invoked by uid 500); 7 Oct 2010 06:08:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47004 invoked by uid 99); 7 Oct 2010 06:08:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 06:08:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 06:08:52 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9768VdB022109
	for <common-issues@hadoop.apache.org>; Thu, 7 Oct 2010 06:08:32 GMT
Message-ID: <2269008.20701286431711879.JavaMail.jira@thor>
Date: Thu, 7 Oct 2010 02:08:31 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6993) Broken link on cluster setup page
 of docs
In-Reply-To: <22210278.19631286425112055.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918794#action_12918794 ] 

Eli Collins commented on HADOOP-6993:
-------------------------------------

Verified the links are fixed in the generated docs.

> Broken link on cluster setup page of docs
> -----------------------------------------
>
>                 Key: HADOOP-6993
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6993
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.21.0
>            Reporter: Aaron T. Myers
>            Assignee: Eli Collins
>             Fix For: 0.21.1, 0.22.0
>
>         Attachments: hadoop-6993-21-1.patch, hadoop-6993-22-1.patch
>
>
> The link on http://hadoop.apache.org/common/docs/current/cluster_setup.html#Configuring+the+Hadoop+Daemons to core-default.xml is presently:
> {quote}
> http://hadoop.apache.org/common/docs/current/common-default.html
> {quote}
> but it should be:
> {quote}
> http://hadoop.apache.org/common/docs/current/core-default.html
> {quote}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10956-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 07 16:02:07 2010
Return-Path: <common-issues-return-10956-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68327 invoked from network); 7 Oct 2010 16:02:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Oct 2010 16:02:07 -0000
Received: (qmail 39225 invoked by uid 500); 7 Oct 2010 16:02:06 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39139 invoked by uid 500); 7 Oct 2010 16:02:06 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39131 invoked by uid 99); 7 Oct 2010 16:02:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 16:02:05 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 16:02:03 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o97G1gsO001406
	for <common-issues@hadoop.apache.org>; Thu, 7 Oct 2010 16:01:42 GMT
Message-ID: <25530780.26661286467302150.JavaMail.jira@thor>
Date: Thu, 7 Oct 2010 12:01:42 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6987) Use JUnit Rule to optionally fail
 test cases that run more than 10 seconds
In-Reply-To: <26687627.538921286230953013.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918951#action_12918951 ] 

Konstantin Boudnik commented on HADOOP-6987:
--------------------------------------------

I had an offline conversation with Jakob about this and he expressed that the approach is about to enforce max time of a true unit test execution. Unit tests are clearly suppose to be very short.

I still have a couple comments about this:
 - let's call class {{UnitTestTimeLimit}} rather then {{TenSecondsTimeoutPerTest}} to make it more generic
 - let's make timeout to be a constant and move it to a config or util class
 - let's update [Unit test page|http://wiki.apache.org/hadoop/HowToDevelopUnitTests]
 - as we are going to establish a precedent for setting rules of engagement I'd suggest to update  HADOOP-6399 with that rule

> Use JUnit Rule to optionally fail test cases that run more than 10 seconds
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6987
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6987
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6897.patch
>
>
> Using JUnit Rules annotations we can fail tests cases that take longer than 10 seconds (for instance) to run.  This provides a regression check against test cases taking longer than they had previously due to unintended code changes, as well as provides a membership criteria for unit tests versus integration tests in HDFS and MR.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10957-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 07 16:28:56 2010
Return-Path: <common-issues-return-10957-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78567 invoked from network); 7 Oct 2010 16:28:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Oct 2010 16:28:56 -0000
Received: (qmail 69345 invoked by uid 500); 7 Oct 2010 16:28:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69308 invoked by uid 500); 7 Oct 2010 16:28:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69300 invoked by uid 99); 7 Oct 2010 16:28:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 16:28:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 16:28:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o97GSVWE001811
	for <common-issues@hadoop.apache.org>; Thu, 7 Oct 2010 16:28:32 GMT
Message-ID: <17445785.26961286468911653.JavaMail.jira@thor>
Date: Thu, 7 Oct 2010 12:28:31 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6992) Update website for recent
 subproject departures
In-Reply-To: <26161850.16451286406151616.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918960#action_12918960 ] 

Doug Cutting commented on HADOOP-6992:
--------------------------------------

> we need to wait for their new web site to be up

Indeed.  Pig's is up, although it's still branded as a Hadoop subproject.  Hive got their virtual host configured yesterday but has not yet put any content at hive.apache.org, so that's currently the gating factor.  Once that happens, I'll commit this.

> Update website for recent subproject departures
> -----------------------------------------------
>
>                 Key: HADOOP-6992
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6992
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: site
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: site
>
>         Attachments: HADOOP-6992.patch
>
>
> A number of subprojects have left Hadoop yet the website's not been fully updated to reflect that.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10958-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 07 17:33:56 2010
Return-Path: <common-issues-return-10958-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24010 invoked from network); 7 Oct 2010 17:33:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Oct 2010 17:33:56 -0000
Received: (qmail 71354 invoked by uid 500); 7 Oct 2010 17:33:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71313 invoked by uid 500); 7 Oct 2010 17:33:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71305 invoked by uid 99); 7 Oct 2010 17:33:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 17:33:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 17:33:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o97HXWI8002898
	for <common-issues@hadoop.apache.org>; Thu, 7 Oct 2010 17:33:32 GMT
Message-ID: <23017183.27881286472812026.JavaMail.jira@thor>
Date: Thu, 7 Oct 2010 13:33:32 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6992) Update website for recent
 subproject departures
In-Reply-To: <26161850.16451286406151616.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918985#action_12918985 ] 

Owen O'Malley commented on HADOOP-6992:
---------------------------------------

Alan asked for me to commit the redirect for pig now, so I committed that part of the patch.

> Update website for recent subproject departures
> -----------------------------------------------
>
>                 Key: HADOOP-6992
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6992
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: site
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: site
>
>         Attachments: HADOOP-6992.patch
>
>
> A number of subprojects have left Hadoop yet the website's not been fully updated to reflect that.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10959-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 07 17:47:52 2010
Return-Path: <common-issues-return-10959-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32524 invoked from network); 7 Oct 2010 17:47:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Oct 2010 17:47:52 -0000
Received: (qmail 89922 invoked by uid 500); 7 Oct 2010 17:47:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89875 invoked by uid 500); 7 Oct 2010 17:47:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89867 invoked by uid 99); 7 Oct 2010 17:47:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 17:47:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 17:47:51 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o97HlVf7003112
	for <common-issues@hadoop.apache.org>; Thu, 7 Oct 2010 17:47:31 GMT
Message-ID: <18413578.28061286473651317.JavaMail.jira@thor>
Date: Thu, 7 Oct 2010 13:47:31 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6993) Broken link on cluster setup page
 of docs
In-Reply-To: <22210278.19631286425112055.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918987#action_12918987 ] 

Tom White commented on HADOOP-6993:
-----------------------------------

+1

> Broken link on cluster setup page of docs
> -----------------------------------------
>
>                 Key: HADOOP-6993
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6993
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.21.0
>            Reporter: Aaron T. Myers
>            Assignee: Eli Collins
>             Fix For: 0.21.1, 0.22.0
>
>         Attachments: hadoop-6993-21-1.patch, hadoop-6993-22-1.patch
>
>
> The link on http://hadoop.apache.org/common/docs/current/cluster_setup.html#Configuring+the+Hadoop+Daemons to core-default.xml is presently:
> {quote}
> http://hadoop.apache.org/common/docs/current/common-default.html
> {quote}
> but it should be:
> {quote}
> http://hadoop.apache.org/common/docs/current/core-default.html
> {quote}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10960-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 07 17:55:56 2010
Return-Path: <common-issues-return-10960-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36328 invoked from network); 7 Oct 2010 17:55:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Oct 2010 17:55:56 -0000
Received: (qmail 98838 invoked by uid 500); 7 Oct 2010 17:55:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98800 invoked by uid 500); 7 Oct 2010 17:55:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98792 invoked by uid 99); 7 Oct 2010 17:55:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 17:55:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 17:55:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o97HtVED003221
	for <common-issues@hadoop.apache.org>; Thu, 7 Oct 2010 17:55:31 GMT
Message-ID: <2997551.28151286474131924.JavaMail.jira@thor>
Date: Thu, 7 Oct 2010 13:55:31 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6993) Broken link on cluster setup page of
 docs
In-Reply-To: <22210278.19631286425112055.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins resolved HADOOP-6993.
---------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]

Thanks Tom.  I've committed this to trunk and 21.

> Broken link on cluster setup page of docs
> -----------------------------------------
>
>                 Key: HADOOP-6993
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6993
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.21.0
>            Reporter: Aaron T. Myers
>            Assignee: Eli Collins
>             Fix For: 0.21.1, 0.22.0
>
>         Attachments: hadoop-6993-21-1.patch, hadoop-6993-22-1.patch
>
>
> The link on http://hadoop.apache.org/common/docs/current/cluster_setup.html#Configuring+the+Hadoop+Daemons to core-default.xml is presently:
> {quote}
> http://hadoop.apache.org/common/docs/current/common-default.html
> {quote}
> but it should be:
> {quote}
> http://hadoop.apache.org/common/docs/current/core-default.html
> {quote}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10961-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 07 18:25:54 2010
Return-Path: <common-issues-return-10961-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61679 invoked from network); 7 Oct 2010 18:25:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Oct 2010 18:25:54 -0000
Received: (qmail 29584 invoked by uid 500); 7 Oct 2010 18:25:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29530 invoked by uid 500); 7 Oct 2010 18:25:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29522 invoked by uid 99); 7 Oct 2010 18:25:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 18:25:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 18:25:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o97IPWCA003737
	for <common-issues@hadoop.apache.org>; Thu, 7 Oct 2010 18:25:32 GMT
Message-ID: <6593025.28721286475932719.JavaMail.jira@thor>
Date: Thu, 7 Oct 2010 14:25:32 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6994) Api in FileContext to get delegation
 tokens.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Api in FileContext to get delegation tokens.
--------------------------------------------

                 Key: HADOOP-6994
                 URL: https://issues.apache.org/jira/browse/HADOOP-6994
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Jitendra Nath Pandey
            Assignee: Jitendra Nath Pandey


APIs to get delegation tokens is required in FileContext. A path may refer to several file systems and delegation tokens would be needed for many of them for a client to be able to successfully access the path. This jira will also add an API to get list of AbstractFileSystems accessed in a path which could be more than one for symlinks.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10962-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 07 18:29:53 2010
Return-Path: <common-issues-return-10962-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64419 invoked from network); 7 Oct 2010 18:29:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Oct 2010 18:29:53 -0000
Received: (qmail 37083 invoked by uid 500); 7 Oct 2010 18:29:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37054 invoked by uid 500); 7 Oct 2010 18:29:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37031 invoked by uid 99); 7 Oct 2010 18:29:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 18:29:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 18:29:52 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o97ITVUJ003837
	for <common-issues@hadoop.apache.org>; Thu, 7 Oct 2010 18:29:32 GMT
Message-ID: <28608338.28881286476171947.JavaMail.jira@thor>
Date: Thu, 7 Oct 2010 14:29:31 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6994) Api to get delegation token in
 AbstractFileSystem
In-Reply-To: <6593025.28721286475932719.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6994:
-----------------------------------------

    Description: 
APIs to get delegation tokens is required in AbstractFileSystem. AbstractFileSystems are accessed via file context therefore an API to get list of AbstractFileSystems accessed in a path is also needed. 
A path may refer to several file systems and delegation tokens could be needed for many of them for a client to be able to successfully access the path. 

  was:APIs to get delegation tokens is required in FileContext. A path may refer to several file systems and delegation tokens would be needed for many of them for a client to be able to successfully access the path. This jira will also add an API to get list of AbstractFileSystems accessed in a path which could be more than one for symlinks.

        Summary: Api to get delegation token in AbstractFileSystem  (was: Api in FileContext to get delegation tokens.)

> Api to get delegation token in AbstractFileSystem
> -------------------------------------------------
>
>                 Key: HADOOP-6994
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6994
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>
> APIs to get delegation tokens is required in AbstractFileSystem. AbstractFileSystems are accessed via file context therefore an API to get list of AbstractFileSystems accessed in a path is also needed. 
> A path may refer to several file systems and delegation tokens could be needed for many of them for a client to be able to successfully access the path. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10963-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 07 18:29:54 2010
Return-Path: <common-issues-return-10963-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64462 invoked from network); 7 Oct 2010 18:29:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Oct 2010 18:29:53 -0000
Received: (qmail 37263 invoked by uid 500); 7 Oct 2010 18:29:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37231 invoked by uid 500); 7 Oct 2010 18:29:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37223 invoked by uid 99); 7 Oct 2010 18:29:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 18:29:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 18:29:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o97ITWUL003849
	for <common-issues@hadoop.apache.org>; Thu, 7 Oct 2010 18:29:32 GMT
Message-ID: <14648928.28921286476172957.JavaMail.jira@thor>
Date: Thu, 7 Oct 2010 14:29:32 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6992) Update website for recent subproject
 departures
In-Reply-To: <26161850.16451286406151616.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6992:
----------------------------------

    Attachment: hadoop-tlp.pdf

For those that don't want to run forrest, here is what the front page would look like.

> Update website for recent subproject departures
> -----------------------------------------------
>
>                 Key: HADOOP-6992
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6992
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: site
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: site
>
>         Attachments: HADOOP-6992.patch, hadoop-tlp.pdf
>
>
> A number of subprojects have left Hadoop yet the website's not been fully updated to reflect that.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10964-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 07 18:34:55 2010
Return-Path: <common-issues-return-10964-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69018 invoked from network); 7 Oct 2010 18:34:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Oct 2010 18:34:55 -0000
Received: (qmail 42457 invoked by uid 500); 7 Oct 2010 18:34:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42414 invoked by uid 500); 7 Oct 2010 18:34:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42406 invoked by uid 99); 7 Oct 2010 18:34:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 18:34:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 18:34:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o97IYV4o003978
	for <common-issues@hadoop.apache.org>; Thu, 7 Oct 2010 18:34:31 GMT
Message-ID: <19247321.28991286476471616.JavaMail.jira@thor>
Date: Thu, 7 Oct 2010 14:34:31 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6994) Api to get delegation token in
 AbstractFileSystem
In-Reply-To: <6593025.28721286475932719.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6994:
-----------------------------------------

    Attachment: HADOOP-6994.1.patch

> Api to get delegation token in AbstractFileSystem
> -------------------------------------------------
>
>                 Key: HADOOP-6994
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6994
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6994.1.patch
>
>
> APIs to get delegation tokens is required in AbstractFileSystem. AbstractFileSystems are accessed via file context therefore an API to get list of AbstractFileSystems accessed in a path is also needed. 
> A path may refer to several file systems and delegation tokens could be needed for many of them for a client to be able to successfully access the path. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10965-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 07 18:44:56 2010
Return-Path: <common-issues-return-10965-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74927 invoked from network); 7 Oct 2010 18:44:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Oct 2010 18:44:55 -0000
Received: (qmail 47804 invoked by uid 500); 7 Oct 2010 18:44:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47672 invoked by uid 500); 7 Oct 2010 18:44:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47664 invoked by uid 99); 7 Oct 2010 18:44:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 18:44:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 18:44:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o97IiVTZ004114
	for <common-issues@hadoop.apache.org>; Thu, 7 Oct 2010 18:44:31 GMT
Message-ID: <25747659.29151286477071733.JavaMail.jira@thor>
Date: Thu, 7 Oct 2010 14:44:31 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6994) Api to get delegation token in
 AbstractFileSystem
In-Reply-To: <6593025.28721286475932719.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12919012#action_12919012 ] 

Tom White commented on HADOOP-6994:
-----------------------------------

The new methods on AbstractFileSystem are public, yet Token and AbstractDelegationTokenIdentifier are LimitedPrivate, which seems inconsistent.

> Api to get delegation token in AbstractFileSystem
> -------------------------------------------------
>
>                 Key: HADOOP-6994
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6994
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6994.1.patch
>
>
> APIs to get delegation tokens is required in AbstractFileSystem. AbstractFileSystems are accessed via file context therefore an API to get list of AbstractFileSystems accessed in a path is also needed. 
> A path may refer to several file systems and delegation tokens could be needed for many of them for a client to be able to successfully access the path. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10966-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 07 19:10:54 2010
Return-Path: <common-issues-return-10966-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96510 invoked from network); 7 Oct 2010 19:10:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Oct 2010 19:10:54 -0000
Received: (qmail 90709 invoked by uid 500); 7 Oct 2010 19:10:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90675 invoked by uid 500); 7 Oct 2010 19:10:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90667 invoked by uid 99); 7 Oct 2010 19:10:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 19:10:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 19:10:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o97JAXwY004643
	for <common-issues@hadoop.apache.org>; Thu, 7 Oct 2010 19:10:33 GMT
Message-ID: <14244057.29781286478633154.JavaMail.jira@thor>
Date: Thu, 7 Oct 2010 15:10:33 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6933) TestListFiles is flaky
In-Reply-To: <14615079.42351282946633507.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12919024#action_12919024 ] 

Tom White commented on HADOOP-6933:
-----------------------------------

Result of test-patch:

{noformat}
[exec] +1 overall.  
[exec] 
[exec]     +1 @author.  The patch does not contain any @author tags.
[exec] 
[exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
[exec] 
[exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
[exec] 
[exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
[exec] 
[exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
[exec] 
[exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
[exec] 
[exec]     +1 system tests framework.  The patch passed system tests framework compile.
[exec] 
{noformat}

> TestListFiles is flaky
> ----------------------
>
>                 Key: HADOOP-6933
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6933
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-6933.txt
>
>
> TestListFiles assumes a particular order of the files returned by the directory iterator. There's no such guarantee made by the underlying API, so the test fails on some hosts.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10967-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 07 19:12:54 2010
Return-Path: <common-issues-return-10967-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97799 invoked from network); 7 Oct 2010 19:12:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Oct 2010 19:12:53 -0000
Received: (qmail 95287 invoked by uid 500); 7 Oct 2010 19:12:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95258 invoked by uid 500); 7 Oct 2010 19:12:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95250 invoked by uid 99); 7 Oct 2010 19:12:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 19:12:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 19:12:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o97JCWeg004698
	for <common-issues@hadoop.apache.org>; Thu, 7 Oct 2010 19:12:32 GMT
Message-ID: <15255434.29841286478752790.JavaMail.jira@thor>
Date: Thu, 7 Oct 2010 15:12:32 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6994) Api to get delegation token in
 AbstractFileSystem
In-Reply-To: <6593025.28721286475932719.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12919026#action_12919026 ] 

Jakob Homan commented on HADOOP-6994:
-------------------------------------

bq. The new methods on AbstractFileSystem are public, yet Token and AbstractDelegationTokenIdentifier are LimitedPrivate, which seems inconsistent. 
I also have this concern, particularly in regard to HADOOP-6988. 

> Api to get delegation token in AbstractFileSystem
> -------------------------------------------------
>
>                 Key: HADOOP-6994
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6994
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6994.1.patch
>
>
> APIs to get delegation tokens is required in AbstractFileSystem. AbstractFileSystems are accessed via file context therefore an API to get list of AbstractFileSystems accessed in a path is also needed. 
> A path may refer to several file systems and delegation tokens could be needed for many of them for a client to be able to successfully access the path. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10968-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 07 19:12:55 2010
Return-Path: <common-issues-return-10968-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97884 invoked from network); 7 Oct 2010 19:12:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Oct 2010 19:12:55 -0000
Received: (qmail 95655 invoked by uid 500); 7 Oct 2010 19:12:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95627 invoked by uid 500); 7 Oct 2010 19:12:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95619 invoked by uid 99); 7 Oct 2010 19:12:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 19:12:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 19:12:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o97JCYrF004720
	for <common-issues@hadoop.apache.org>; Thu, 7 Oct 2010 19:12:34 GMT
Message-ID: <15559596.29911286478754648.JavaMail.jira@thor>
Date: Thu, 7 Oct 2010 15:12:34 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6933) TestListFiles is flaky
In-Reply-To: <14615079.42351282946633507.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6933:
------------------------------

       Resolution: Fixed
    Fix Version/s: 0.22.0
           Status: Resolved  (was: Patch Available)

I've just committed this. Thanks, Todd!

> TestListFiles is flaky
> ----------------------
>
>                 Key: HADOOP-6933
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6933
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6933.txt
>
>
> TestListFiles assumes a particular order of the files returned by the directory iterator. There's no such guarantee made by the underlying API, so the test fails on some hosts.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10969-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 07 19:46:53 2010
Return-Path: <common-issues-return-10969-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15889 invoked from network); 7 Oct 2010 19:46:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Oct 2010 19:46:53 -0000
Received: (qmail 33499 invoked by uid 500); 7 Oct 2010 19:46:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33392 invoked by uid 500); 7 Oct 2010 19:46:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33383 invoked by uid 99); 7 Oct 2010 19:46:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 19:46:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 19:46:52 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o97JkVFp005177
	for <common-issues@hadoop.apache.org>; Thu, 7 Oct 2010 19:46:32 GMT
Message-ID: <20830868.30251286480791796.JavaMail.jira@thor>
Date: Thu, 7 Oct 2010 15:46:31 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6987) Use JUnit Rule to optionally fail
 test cases that run more than 10 seconds
In-Reply-To: <26687627.538921286230953013.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6987:
--------------------------------

    Attachment: HADOOP-6987-2.patch

Updated patch with Cos' suggestion.  There's not really a util class to put the value in, so it's now a final field on the rule class.  I'll update the unit test guidelines when the patch is committed.  I've linked this JIRA to HADOOP-6399.  Once this is committed, I'll open an HDFS JIRA to tag the tests currently in the unit directory.

> Use JUnit Rule to optionally fail test cases that run more than 10 seconds
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6987
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6987
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6897.patch, HADOOP-6987-2.patch
>
>
> Using JUnit Rules annotations we can fail tests cases that take longer than 10 seconds (for instance) to run.  This provides a regression check against test cases taking longer than they had previously due to unintended code changes, as well as provides a membership criteria for unit tests versus integration tests in HDFS and MR.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10970-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 07 22:00:58 2010
Return-Path: <common-issues-return-10970-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91061 invoked from network); 7 Oct 2010 22:00:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Oct 2010 22:00:58 -0000
Received: (qmail 19729 invoked by uid 500); 7 Oct 2010 22:00:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19696 invoked by uid 500); 7 Oct 2010 22:00:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19688 invoked by uid 99); 7 Oct 2010 22:00:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 22:00:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 22:00:57 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o97M0bED007201
	for <common-issues@hadoop.apache.org>; Thu, 7 Oct 2010 22:00:37 GMT
Message-ID: <13456585.32211286488837249.JavaMail.jira@thor>
Date: Thu, 7 Oct 2010 18:00:37 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6988) Add support for reading multiple
 hadoop delegation token files
In-Reply-To: <17443930.539541286234913119.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12919078#action_12919078 ] 

Owen O'Malley commented on HADOOP-6988:
---------------------------------------

The intent of the environment variable is *not* for job submission. I really don't see any value in making it multi-valued.

> Add support for reading multiple hadoop delegation token files
> --------------------------------------------------------------
>
>                 Key: HADOOP-6988
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6988
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>         Attachments: hadoop-6988.0.txt, hadoop-6988.1.txt
>
>
> It would be nice if there were a way to specify multiple delegation token files via the HADOOP_TOKEN_FILE_LOCATION environment variable and the "mapreduce.job.credentials.binary" configuration value. I suggest a colon-separated list of paths, each of which is read as a separate delegation token file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10971-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 07 22:22:55 2010
Return-Path: <common-issues-return-10971-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98922 invoked from network); 7 Oct 2010 22:22:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Oct 2010 22:22:53 -0000
Received: (qmail 47072 invoked by uid 500); 7 Oct 2010 22:22:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47008 invoked by uid 500); 7 Oct 2010 22:22:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46587 invoked by uid 99); 7 Oct 2010 22:22:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 22:22:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 22:22:52 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o97MMVvo007532
	for <common-issues@hadoop.apache.org>; Thu, 7 Oct 2010 22:22:32 GMT
Message-ID: <8155666.32571286490151977.JavaMail.jira@thor>
Date: Thu, 7 Oct 2010 18:22:31 -0400 (EDT)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6988) Add support for reading multiple
 hadoop delegation token files
In-Reply-To: <17443930.539541286234913119.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12919082#action_12919082 ] 

Aaron T. Myers commented on HADOOP-6988:
----------------------------------------

Even if it's not intended for job submission, it's documented as being the preferred way to pass token files to bin/hadoop for the purpose of running HDFS commands.

>From the HDFS user guide:

{quote}
The HDFS fetchdt command is not a Hadoop shell command. It can be run as 'bin/hadoop fetchdt DTfile'. After you got the token you can run an HDFS command without having Kerberos tickets, by pointing HADOOP_TOKEN_FILE_LOCATION environmental variable to the delegation token file.
{quote}

Thus, this change could be useful for any HDFS command which is capable of communicating with multiple distinct NNs.

> Add support for reading multiple hadoop delegation token files
> --------------------------------------------------------------
>
>                 Key: HADOOP-6988
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6988
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>         Attachments: hadoop-6988.0.txt, hadoop-6988.1.txt
>
>
> It would be nice if there were a way to specify multiple delegation token files via the HADOOP_TOKEN_FILE_LOCATION environment variable and the "mapreduce.job.credentials.binary" configuration value. I suggest a colon-separated list of paths, each of which is read as a separate delegation token file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10972-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 07 22:37:55 2010
Return-Path: <common-issues-return-10972-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2681 invoked from network); 7 Oct 2010 22:37:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Oct 2010 22:37:55 -0000
Received: (qmail 71729 invoked by uid 500); 7 Oct 2010 22:37:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71673 invoked by uid 500); 7 Oct 2010 22:37:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71665 invoked by uid 99); 7 Oct 2010 22:37:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 22:37:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 22:37:52 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o97MbVIY007751
	for <common-issues@hadoop.apache.org>; Thu, 7 Oct 2010 22:37:31 GMT
Message-ID: <28126371.32871286491051402.JavaMail.jira@thor>
Date: Thu, 7 Oct 2010 18:37:31 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6994) Api to get delegation token in
 AbstractFileSystem
In-Reply-To: <6593025.28721286475932719.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6994:
-----------------------------------------

    Attachment: HADOOP-6994.2.patch

New patch addressing the comment.

The InterfaceAudience for the new APIs are annotated to be LimitedPrivate.
The getDelegationToken API in FileSystem is also changed to LimitedPrivate.


> Api to get delegation token in AbstractFileSystem
> -------------------------------------------------
>
>                 Key: HADOOP-6994
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6994
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6994.1.patch, HADOOP-6994.2.patch
>
>
> APIs to get delegation tokens is required in AbstractFileSystem. AbstractFileSystems are accessed via file context therefore an API to get list of AbstractFileSystems accessed in a path is also needed. 
> A path may refer to several file systems and delegation tokens could be needed for many of them for a client to be able to successfully access the path. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10973-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 07 23:04:57 2010
Return-Path: <common-issues-return-10973-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7475 invoked from network); 7 Oct 2010 23:04:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Oct 2010 23:04:57 -0000
Received: (qmail 88829 invoked by uid 500); 7 Oct 2010 23:04:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88793 invoked by uid 500); 7 Oct 2010 23:04:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88785 invoked by uid 99); 7 Oct 2010 23:04:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 23:04:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Oct 2010 23:04:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o97N4XFK008301
	for <common-issues@hadoop.apache.org>; Thu, 7 Oct 2010 23:04:33 GMT
Message-ID: <16164275.33651286492673006.JavaMail.jira@thor>
Date: Thu, 7 Oct 2010 19:04:33 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6987) Use JUnit Rule to optionally fail
 test cases that run more than 10 seconds
In-Reply-To: <26687627.538921286230953013.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12919096#action_12919096 ] 

Konstantin Boudnik commented on HADOOP-6987:
--------------------------------------------

+1 patch looks good. 
I have ran it with latest trunk and it seems to be doing the job.

> Use JUnit Rule to optionally fail test cases that run more than 10 seconds
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6987
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6987
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6897.patch, HADOOP-6987-2.patch
>
>
> Using JUnit Rules annotations we can fail tests cases that take longer than 10 seconds (for instance) to run.  This provides a regression check against test cases taking longer than they had previously due to unintended code changes, as well as provides a membership criteria for unit tests versus integration tests in HDFS and MR.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10974-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 08 06:11:56 2010
Return-Path: <common-issues-return-10974-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48253 invoked from network); 8 Oct 2010 06:11:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Oct 2010 06:11:56 -0000
Received: (qmail 82637 invoked by uid 500); 8 Oct 2010 06:11:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82594 invoked by uid 500); 8 Oct 2010 06:11:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82579 invoked by uid 99); 8 Oct 2010 06:11:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Oct 2010 06:11:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Oct 2010 06:11:51 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o986BV1W014335
	for <common-issues@hadoop.apache.org>; Fri, 8 Oct 2010 06:11:31 GMT
Message-ID: <29275260.37121286518291556.JavaMail.jira@thor>
Date: Fri, 8 Oct 2010 02:11:31 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6995) Allow wildcards to be used in
 ProxyUsers configurations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Allow wildcards to be used in ProxyUsers configurations
-------------------------------------------------------

                 Key: HADOOP-6995
                 URL: https://issues.apache.org/jira/browse/HADOOP-6995
             Project: Hadoop Common
          Issue Type: Improvement
          Components: security
    Affects Versions: 0.22.0
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
            Priority: Minor


There are some cases where the full tightness of the ProxyUsers configuration is not required or available -- for example, not all users of oozie may share a common "oozie-users" group, and the operators would prefer to allow oozie on a given host to act proxy for any user. We should allow the operator to specify a wildcard for hosts or groups in the proxyuser configurations.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10975-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 08 06:13:57 2010
Return-Path: <common-issues-return-10975-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48462 invoked from network); 8 Oct 2010 06:13:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Oct 2010 06:13:57 -0000
Received: (qmail 84036 invoked by uid 500); 8 Oct 2010 06:13:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84000 invoked by uid 500); 8 Oct 2010 06:13:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83991 invoked by uid 99); 8 Oct 2010 06:13:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Oct 2010 06:13:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Oct 2010 06:13:52 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o986DUSr014360
	for <common-issues@hadoop.apache.org>; Fri, 8 Oct 2010 06:13:31 GMT
Message-ID: <4097627.37151286518410976.JavaMail.jira@thor>
Date: Fri, 8 Oct 2010 02:13:30 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6995) Allow wildcards to be used in
 ProxyUsers configurations
In-Reply-To: <29275260.37121286518291556.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6995:
--------------------------------

    Attachment: hadoop-6995-branch20.txt

For 0.20S, not for commit.

> Allow wildcards to be used in ProxyUsers configurations
> -------------------------------------------------------
>
>                 Key: HADOOP-6995
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6995
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-6995-branch20.txt
>
>
> There are some cases where the full tightness of the ProxyUsers configuration is not required or available -- for example, not all users of oozie may share a common "oozie-users" group, and the operators would prefer to allow oozie on a given host to act proxy for any user. We should allow the operator to specify a wildcard for hosts or groups in the proxyuser configurations.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10976-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 08 06:17:58 2010
Return-Path: <common-issues-return-10976-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49187 invoked from network); 8 Oct 2010 06:17:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Oct 2010 06:17:58 -0000
Received: (qmail 86524 invoked by uid 500); 8 Oct 2010 06:17:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86383 invoked by uid 500); 8 Oct 2010 06:17:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86374 invoked by uid 99); 8 Oct 2010 06:17:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Oct 2010 06:17:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Oct 2010 06:17:52 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o986HVDI014409
	for <common-issues@hadoop.apache.org>; Fri, 8 Oct 2010 06:17:31 GMT
Message-ID: <5801938.37191286518651409.JavaMail.jira@thor>
Date: Fri, 8 Oct 2010 02:17:31 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6995) Allow wildcards to be used in
 ProxyUsers configurations
In-Reply-To: <29275260.37121286518291556.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6995:
--------------------------------

    Attachment: hadoop-6995.txt

Patch for trunk

> Allow wildcards to be used in ProxyUsers configurations
> -------------------------------------------------------
>
>                 Key: HADOOP-6995
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6995
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-6995-branch20.txt, hadoop-6995.txt
>
>
> There are some cases where the full tightness of the ProxyUsers configuration is not required or available -- for example, not all users of oozie may share a common "oozie-users" group, and the operators would prefer to allow oozie on a given host to act proxy for any user. We should allow the operator to specify a wildcard for hosts or groups in the proxyuser configurations.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10977-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 08 06:18:01 2010
Return-Path: <common-issues-return-10977-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49314 invoked from network); 8 Oct 2010 06:18:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Oct 2010 06:18:01 -0000
Received: (qmail 86741 invoked by uid 500); 8 Oct 2010 06:18:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86693 invoked by uid 500); 8 Oct 2010 06:18:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86685 invoked by uid 99); 8 Oct 2010 06:17:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Oct 2010 06:17:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Oct 2010 06:17:57 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o986HVd2014415
	for <common-issues@hadoop.apache.org>; Fri, 8 Oct 2010 06:17:31 GMT
Message-ID: <29679466.37211286518651745.JavaMail.jira@thor>
Date: Fri, 8 Oct 2010 02:17:31 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6995) Allow wildcards to be used in
 ProxyUsers configurations
In-Reply-To: <29275260.37121286518291556.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6995:
--------------------------------

    Status: Patch Available  (was: Open)

> Allow wildcards to be used in ProxyUsers configurations
> -------------------------------------------------------
>
>                 Key: HADOOP-6995
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6995
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-6995-branch20.txt, hadoop-6995.txt
>
>
> There are some cases where the full tightness of the ProxyUsers configuration is not required or available -- for example, not all users of oozie may share a common "oozie-users" group, and the operators would prefer to allow oozie on a given host to act proxy for any user. We should allow the operator to specify a wildcard for hosts or groups in the proxyuser configurations.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10978-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 08 07:20:57 2010
Return-Path: <common-issues-return-10978-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72510 invoked from network); 8 Oct 2010 07:20:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Oct 2010 07:20:56 -0000
Received: (qmail 29893 invoked by uid 500); 8 Oct 2010 07:20:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29739 invoked by uid 500); 8 Oct 2010 07:20:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29731 invoked by uid 99); 8 Oct 2010 07:20:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Oct 2010 07:20:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Oct 2010 07:20:52 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o987KWOR015220
	for <common-issues@hadoop.apache.org>; Fri, 8 Oct 2010 07:20:32 GMT
Message-ID: <33340944.37721286522432215.JavaMail.jira@thor>
Date: Fri, 8 Oct 2010 03:20:32 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6988) Add support for reading multiple
 hadoop delegation token files
In-Reply-To: <17443930.539541286234913119.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12919181#action_12919181 ] 

Devaraj Das commented on HADOOP-6988:
-------------------------------------

Although it is true that HADOOP_TOKEN_FILE_LOCATION can be used to make normal hdfs commands work, the intent for having this was to support security for Map/Reduce tasks, and, hadoop streaming apps that internally invoke command-line hdfs operations (as Owen had pointed out earlier). If you want to pass multiple tokens during job submission, the preferred approach would be to write the tokens into a file (using the Credentials class's utilities), and then point mapreduce.job.credentials.binary to that file. 
Thinking about it, wouldn't the option of defining mapreduce.job.hdfs-servers in the job configuration work for you. The JobClient will automatically get delegation tokens from those namenodes and all tasks of the job can use those tokens..

> Add support for reading multiple hadoop delegation token files
> --------------------------------------------------------------
>
>                 Key: HADOOP-6988
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6988
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>         Attachments: hadoop-6988.0.txt, hadoop-6988.1.txt
>
>
> It would be nice if there were a way to specify multiple delegation token files via the HADOOP_TOKEN_FILE_LOCATION environment variable and the "mapreduce.job.credentials.binary" configuration value. I suggest a colon-separated list of paths, each of which is read as a separate delegation token file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10979-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 08 07:51:08 2010
Return-Path: <common-issues-return-10979-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77800 invoked from network); 8 Oct 2010 07:51:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Oct 2010 07:51:07 -0000
Received: (qmail 59357 invoked by uid 500); 8 Oct 2010 07:51:07 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59279 invoked by uid 500); 8 Oct 2010 07:51:05 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59259 invoked by uid 99); 8 Oct 2010 07:51:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Oct 2010 07:51:04 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Oct 2010 07:51:02 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o987oeYx015705
	for <common-issues@hadoop.apache.org>; Fri, 8 Oct 2010 07:50:40 GMT
Message-ID: <2461564.38101286524240563.JavaMail.jira@thor>
Date: Fri, 8 Oct 2010 03:50:40 -0400 (EDT)
From: "Volodymyr Orlov (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6767) Patch for running Hadoop on Windows
 without Cygwin
In-Reply-To: <3999817.74761274049343093.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Volodymyr Orlov updated HADOOP-6767:
------------------------------------

    Attachment: hadoop-6767-r1.tar.gz

Hi,

After a pause I am continuing my work on porting Hadoop for Windows. In the attached archive I've added batch scripts for running Hadoop on Windows, made small changes to build.xml file to be able to build on Windows and added batch files and Apache Commons Daemon configuration files to run Hadoop as Windows services. All changes are against trunk

Best Regards,
Vladimir

> Patch for running Hadoop on Windows without Cygwin
> --------------------------------------------------
>
>                 Key: HADOOP-6767
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6767
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, conf, filecache, fs, io, scripts, security, test, util
>    Affects Versions: 0.20.2
>         Environment: Windows XP, 2003, 7, 2008
>            Reporter: Volodymyr Orlov
>             Fix For: 0.20.3
>
>         Attachments: Hadoop-0.20.2-patched.zip, hadoop-6767-r1.tar.gz, HADOOP-6767.patch
>
>
> Proposed patch from Codeminders adds a possibility to run Hadoop on Windows without Cygwin. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10980-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 08 07:53:22 2010
Return-Path: <common-issues-return-10980-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78135 invoked from network); 8 Oct 2010 07:53:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Oct 2010 07:53:22 -0000
Received: (qmail 60406 invoked by uid 500); 8 Oct 2010 07:53:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60385 invoked by uid 500); 8 Oct 2010 07:53:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60377 invoked by uid 99); 8 Oct 2010 07:53:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Oct 2010 07:53:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Oct 2010 07:53:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o987qxfE015783
	for <common-issues@hadoop.apache.org>; Fri, 8 Oct 2010 07:52:59 GMT
Message-ID: <33515070.38261286524379575.JavaMail.jira@thor>
Date: Fri, 8 Oct 2010 03:52:59 -0400 (EDT)
From: "Volodymyr Orlov (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6767) Patch for running Hadoop on Windows
 without Cygwin
In-Reply-To: <3999817.74761274049343093.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Volodymyr Orlov updated HADOOP-6767:
------------------------------------

          Component/s:     (was: security)
                           (was: filecache)
                           (was: test)
                           (was: util)
                           (was: fs)
                           (was: io)
    Affects Version/s:     (was: 0.20.2)
                       0.22.0
        Fix Version/s:     (was: 0.20.3)
                       0.22.0

> Patch for running Hadoop on Windows without Cygwin
> --------------------------------------------------
>
>                 Key: HADOOP-6767
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6767
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, conf, scripts
>    Affects Versions: 0.22.0
>         Environment: Windows XP, 2003, 7, 2008
>            Reporter: Volodymyr Orlov
>             Fix For: 0.22.0
>
>         Attachments: Hadoop-0.20.2-patched.zip, hadoop-6767-r1.tar.gz, HADOOP-6767.patch
>
>
> Proposed patch from Codeminders adds a possibility to run Hadoop on Windows without Cygwin. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10981-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 08 09:49:58 2010
Return-Path: <common-issues-return-10981-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 160 invoked from network); 8 Oct 2010 09:49:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Oct 2010 09:49:58 -0000
Received: (qmail 87521 invoked by uid 500); 8 Oct 2010 09:49:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87363 invoked by uid 500); 8 Oct 2010 09:49:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87354 invoked by uid 99); 8 Oct 2010 09:49:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Oct 2010 09:49:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Oct 2010 09:49:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o989nZKX017633
	for <common-issues@hadoop.apache.org>; Fri, 8 Oct 2010 09:49:35 GMT
Message-ID: <3097321.39411286531375112.JavaMail.jira@thor>
Date: Fri, 8 Oct 2010 05:49:35 -0400 (EDT)
From: "Volodymyr Orlov (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6767) Patch for running Hadoop on Windows
 without Cygwin
In-Reply-To: <3999817.74761274049343093.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Volodymyr Orlov updated HADOOP-6767:
------------------------------------

    Release Note: Batch scripts for running Hadoop on windows, scripts for setting Hadoop as windows service using Apache Commons Daemon, and fixes in build  (was: Patch uses JNA (https://jna.dev.java.net/) to perform operations such as getting folder size, getting disk  space statistics, getting current user, current user groups, setting and getting FS permissions, setting and getting file or folder owner and group. Patch has been tested on Windows XP, Windows 2008, Windows 7. Later on I will add to the issue link to the description of how to set up Hadoop cluster on Windows without Cygwin)
          Status: Patch Available  (was: Open)

please see README.txt file in the archive I've attached to the issue

> Patch for running Hadoop on Windows without Cygwin
> --------------------------------------------------
>
>                 Key: HADOOP-6767
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6767
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, conf, scripts
>    Affects Versions: 0.22.0
>         Environment: Windows XP, 2003, 7, 2008
>            Reporter: Volodymyr Orlov
>             Fix For: 0.22.0
>
>         Attachments: Hadoop-0.20.2-patched.zip, hadoop-6767-r1.tar.gz, HADOOP-6767.patch
>
>
> Proposed patch from Codeminders adds a possibility to run Hadoop on Windows without Cygwin. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10982-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 08 18:30:55 2010
Return-Path: <common-issues-return-10982-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71521 invoked from network); 8 Oct 2010 18:30:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Oct 2010 18:30:55 -0000
Received: (qmail 67006 invoked by uid 500); 8 Oct 2010 18:30:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66946 invoked by uid 500); 8 Oct 2010 18:30:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66938 invoked by uid 99); 8 Oct 2010 18:30:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Oct 2010 18:30:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Oct 2010 18:30:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o98IUXfn025428
	for <common-issues@hadoop.apache.org>; Fri, 8 Oct 2010 18:30:33 GMT
Message-ID: <24940573.44861286562633053.JavaMail.jira@thor>
Date: Fri, 8 Oct 2010 14:30:33 -0400 (EDT)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6988) Add support for reading multiple
 hadoop delegation token files
In-Reply-To: <17443930.539541286234913119.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12919344#action_12919344 ] 

Aaron T. Myers commented on HADOOP-6988:
----------------------------------------

Thanks for the thoughtful comments, Devaraj.

As I said earlier, this is really only for convenience, as it's entirely possible to stuff multiple delegation token objects into a single credentials object, which is then serialized to a file. I considered creating a tool which would be capable of merging multiple delegation token files into one, but this seemed like a cleaner solution. Rather than having every script/job/program that wants to pass multiple independently-fetched delegation token files first invoke some command to merge them, just specify them all via the method that already exists.

The problem with specifying mapreduce.job.hdfs-servers for my particular use-case is that delegation tokens can't be fetched if the application which is submitting the job is only authenticated via a delegation token in the first place. That said, I see this issue as being largely orthogonal from the core question of whether or not it is reasonable to want to specify multiple delegation token files via the system that already exists.

> Add support for reading multiple hadoop delegation token files
> --------------------------------------------------------------
>
>                 Key: HADOOP-6988
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6988
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>         Attachments: hadoop-6988.0.txt, hadoop-6988.1.txt
>
>
> It would be nice if there were a way to specify multiple delegation token files via the HADOOP_TOKEN_FILE_LOCATION environment variable and the "mapreduce.job.credentials.binary" configuration value. I suggest a colon-separated list of paths, each of which is read as a separate delegation token file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10983-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 08 19:23:56 2010
Return-Path: <common-issues-return-10983-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86264 invoked from network); 8 Oct 2010 19:23:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Oct 2010 19:23:56 -0000
Received: (qmail 19176 invoked by uid 500); 8 Oct 2010 19:23:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19099 invoked by uid 500); 8 Oct 2010 19:23:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19091 invoked by uid 99); 8 Oct 2010 19:23:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Oct 2010 19:23:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Oct 2010 19:23:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o98JNVNb026080
	for <common-issues@hadoop.apache.org>; Fri, 8 Oct 2010 19:23:32 GMT
Message-ID: <29489262.45181286565811914.JavaMail.jira@thor>
Date: Fri, 8 Oct 2010 15:23:31 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6987) Use JUnit Rule to optionally fail
 test cases that run more than 10 seconds
In-Reply-To: <26687627.538921286230953013.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6987:
--------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I re-ran test-patch, which was clean.  I've committed this.  Resolving as fixed.

> Use JUnit Rule to optionally fail test cases that run more than 10 seconds
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6987
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6987
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6897.patch, HADOOP-6987-2.patch
>
>
> Using JUnit Rules annotations we can fail tests cases that take longer than 10 seconds (for instance) to run.  This provides a regression check against test cases taking longer than they had previously due to unintended code changes, as well as provides a membership criteria for unit tests versus integration tests in HDFS and MR.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10985-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Oct 10 21:55:59 2010
Return-Path: <common-issues-return-10985-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40141 invoked from network); 10 Oct 2010 21:55:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Oct 2010 21:55:58 -0000
Received: (qmail 1036 invoked by uid 500); 10 Oct 2010 21:55:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 991 invoked by uid 500); 10 Oct 2010 21:55:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 945 invoked by uid 99); 10 Oct 2010 21:55:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Oct 2010 21:55:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Oct 2010 21:55:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9ALtX5q013125
	for <common-issues@hadoop.apache.org>; Sun, 10 Oct 2010 21:55:33 GMT
Message-ID: <31036773.68371286747733904.JavaMail.jira@thor>
Date: Sun, 10 Oct 2010 17:55:33 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6944) [Herriot] Implement a functionality
 for getting proxy users definitions like groups and hosts.
In-Reply-To: <13358164.74161283948073125.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6944:
---------------------------------------

    Affects Version/s:     (was: 0.20.3)
                       0.21.1

> [Herriot] Implement a functionality for getting proxy users definitions like groups and hosts.
> ----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6944
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6944
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>    Affects Versions: 0.21.1
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>             Fix For: 0.21.1
>
>         Attachments: HADOOP-6944.patch, HADOOP-6944.patch, HADOOP-6944.patch, HADOOP-6944.patch, HADOOP-6944.patch
>
>
> Gridmix should require a proxy user's file for impersonating various jobs. So, implement couple of methods for getting the proxy users list and a proxy users file (it's a combination of proxy users and groups) based on cluster configuration.
> The proxy users list should require for map reduce jobs and proxy users file should require for gridmix jobs.
> The following are methods signature,
> public ProxyUserDefinitions getHadoopProxyUsers() - get the list of proxy users list based on cluster configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10984-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Oct 10 21:55:59 2010
Return-Path: <common-issues-return-10984-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40139 invoked from network); 10 Oct 2010 21:55:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Oct 2010 21:55:58 -0000
Received: (qmail 928 invoked by uid 500); 10 Oct 2010 21:55:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 390 invoked by uid 500); 10 Oct 2010 21:55:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 374 invoked by uid 99); 10 Oct 2010 21:55:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Oct 2010 21:55:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Oct 2010 21:55:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9ALtVBs013104
	for <common-issues@hadoop.apache.org>; Sun, 10 Oct 2010 21:55:32 GMT
Message-ID: <32071085.68301286747731773.JavaMail.jira@thor>
Date: Sun, 10 Oct 2010 17:55:31 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6944) [Herriot] Implement a functionality
 for getting proxy users definitions like groups and hosts.
In-Reply-To: <13358164.74161283948073125.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik resolved HADOOP-6944.
----------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.21.1
     Release Note: I have just committed this to 0.21 and trunk. Thanks Vinay.
     Hadoop Flags: [Reviewed]

> [Herriot] Implement a functionality for getting proxy users definitions like groups and hosts.
> ----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6944
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6944
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>    Affects Versions: 0.21.1
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>             Fix For: 0.21.1
>
>         Attachments: HADOOP-6944.patch, HADOOP-6944.patch, HADOOP-6944.patch, HADOOP-6944.patch, HADOOP-6944.patch
>
>
> Gridmix should require a proxy user's file for impersonating various jobs. So, implement couple of methods for getting the proxy users list and a proxy users file (it's a combination of proxy users and groups) based on cluster configuration.
> The proxy users list should require for map reduce jobs and proxy users file should require for gridmix jobs.
> The following are methods signature,
> public ProxyUserDefinitions getHadoopProxyUsers() - get the list of proxy users list based on cluster configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10986-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 11 05:02:18 2010
Return-Path: <common-issues-return-10986-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43207 invoked from network); 11 Oct 2010 05:02:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Oct 2010 05:02:18 -0000
Received: (qmail 95491 invoked by uid 500); 11 Oct 2010 05:02:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95335 invoked by uid 500); 11 Oct 2010 05:02:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95326 invoked by uid 99); 11 Oct 2010 05:02:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Oct 2010 05:02:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Oct 2010 05:02:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9B51sVR022802
	for <common-issues@hadoop.apache.org>; Mon, 11 Oct 2010 05:01:54 GMT
Message-ID: <33550283.70791286773314034.JavaMail.jira@thor>
Date: Mon, 11 Oct 2010 01:01:54 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6472) add tokenCache option to
 GenericOptionsParser for passing file with secret keys to a map reduce job
In-Reply-To: <1407870644.1262042429493.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6472:
----------------------------------

       Resolution: Fixed
    Fix Version/s: 0.22.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

Nobody closed this when it was committed.

> add tokenCache option to GenericOptionsParser for passing file with secret keys to a map reduce job
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6472
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6472
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6472-1.patch, HADOOP-6472-2.patch, HADOOP-6472.patch
>
>
> for MAPRED-1338 - we need an option to pass a file with secreteKeys (tokens) to a mapreduce Job.
> Name of the file is set into the config and will be picked up by JobSubmiter.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10987-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 11 18:31:56 2010
Return-Path: <common-issues-return-10987-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92967 invoked from network); 11 Oct 2010 18:31:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Oct 2010 18:31:54 -0000
Received: (qmail 95451 invoked by uid 500); 11 Oct 2010 18:31:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95069 invoked by uid 500); 11 Oct 2010 18:31:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95060 invoked by uid 99); 11 Oct 2010 18:31:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Oct 2010 18:31:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Oct 2010 18:31:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9BIVWZg005278
	for <common-issues@hadoop.apache.org>; Mon, 11 Oct 2010 18:31:32 GMT
Message-ID: <1541599.80951286821892815.JavaMail.jira@thor>
Date: Mon, 11 Oct 2010 14:31:32 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6996) Allow CodecFactory to return a codec
 object given a codec' class name
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Allow CodecFactory to return a codec object given a codec' class name
---------------------------------------------------------------------

                 Key: HADOOP-6996
                 URL: https://issues.apache.org/jira/browse/HADOOP-6996
             Project: Hadoop Common
          Issue Type: New Feature
          Components: io
    Affects Versions: 0.22.0
            Reporter: Hairong Kuang
            Assignee: Hairong Kuang
             Fix For: 0.22.0


CodecFactory specify the list of codec that are supported by Hadoop. However, it returns a codec only by a file's name. I would like to make getCodec method to alternatively take a codec's class name.

This is required by  HDFS-1435, where
1. it allows an HDFS admin to configure which codec to use to save an image. 
2. It stores the codec class name in its on-disk image instead of a file's suffix.

When saving and reading an image, I'd like to get an codec from CodecFactory by its class name. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10988-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 11 18:35:59 2010
Return-Path: <common-issues-return-10988-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94262 invoked from network); 11 Oct 2010 18:35:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Oct 2010 18:35:58 -0000
Received: (qmail 655 invoked by uid 500); 11 Oct 2010 18:35:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 622 invoked by uid 500); 11 Oct 2010 18:35:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 614 invoked by uid 99); 11 Oct 2010 18:35:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Oct 2010 18:35:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Oct 2010 18:35:56 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9BIZY5X005371
	for <common-issues@hadoop.apache.org>; Mon, 11 Oct 2010 18:35:35 GMT
Message-ID: <25894800.81141286822134744.JavaMail.jira@thor>
Date: Mon, 11 Oct 2010 14:35:34 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6996) Allow CodecFactory to return a codec
 object given a codec' class name
In-Reply-To: <1541599.80951286821892815.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6996:
----------------------------------

    Attachment: getCodecByClassName.patch

This patch
1. adds a classname to codec map to the CodecFactory;
2. adds a new method getCodecByClassName that returns a codec object given a codec class name;
3. Adds a unit test to test this new method.

> Allow CodecFactory to return a codec object given a codec' class name
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6996
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6996
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: getCodecByClassName.patch
>
>
> CodecFactory specify the list of codec that are supported by Hadoop. However, it returns a codec only by a file's name. I would like to make getCodec method to alternatively take a codec's class name.
> This is required by  HDFS-1435, where
> 1. it allows an HDFS admin to configure which codec to use to save an image. 
> 2. It stores the codec class name in its on-disk image instead of a file's suffix.
> When saving and reading an image, I'd like to get an codec from CodecFactory by its class name. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10989-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 11 18:43:54 2010
Return-Path: <common-issues-return-10989-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96399 invoked from network); 11 Oct 2010 18:43:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Oct 2010 18:43:53 -0000
Received: (qmail 11111 invoked by uid 500); 11 Oct 2010 18:43:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10998 invoked by uid 500); 11 Oct 2010 18:43:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10990 invoked by uid 99); 11 Oct 2010 18:43:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Oct 2010 18:43:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Oct 2010 18:43:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9BIhWq7005481
	for <common-issues@hadoop.apache.org>; Mon, 11 Oct 2010 18:43:33 GMT
Message-ID: <22169371.81271286822612441.JavaMail.jira@thor>
Date: Mon, 11 Oct 2010 14:43:32 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6996) Allow CodecFactory to return a codec
 object given a codec' class name
In-Reply-To: <1541599.80951286821892815.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6996:
----------------------------------

    Status: Patch Available  (was: Open)

> Allow CodecFactory to return a codec object given a codec' class name
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6996
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6996
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: getCodecByClassName.patch
>
>
> CodecFactory specify the list of codec that are supported by Hadoop. However, it returns a codec only by a file's name. I would like to make getCodec method to alternatively take a codec's class name.
> This is required by  HDFS-1435, where
> 1. it allows an HDFS admin to configure which codec to use to save an image. 
> 2. It stores the codec class name in its on-disk image instead of a file's suffix.
> When saving and reading an image, I'd like to get an codec from CodecFactory by its class name. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10990-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 11 20:15:58 2010
Return-Path: <common-issues-return-10990-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25411 invoked from network); 11 Oct 2010 20:15:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Oct 2010 20:15:58 -0000
Received: (qmail 13298 invoked by uid 500); 11 Oct 2010 20:15:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13249 invoked by uid 500); 11 Oct 2010 20:15:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13241 invoked by uid 99); 11 Oct 2010 20:15:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Oct 2010 20:15:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Oct 2010 20:15:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9BKFXv6006956
	for <common-issues@hadoop.apache.org>; Mon, 11 Oct 2010 20:15:33 GMT
Message-ID: <6021015.82591286828133483.JavaMail.jira@thor>
Date: Mon, 11 Oct 2010 16:15:33 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6996) Allow CodecFactory to return a
 codec object given a codec' class name
In-Reply-To: <1541599.80951286821892815.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12919970#action_12919970 ] 

Hairong Kuang commented on HADOOP-6996:
---------------------------------------

Ran ant patch and test on my dev machine:

     [exec] -1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to i
     [exec] nclude 3 new or modified tests.
     [exec] 
     [exec]     -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec] 
     [exec]     +1 system tests framework.  The patch passed system tests framework compile.
     [exec] 

BUILD SUCCESSFUL
Total time: 8 minutes 8 seconds

For -1 on java doc, all 6 javadoc warnings seem not related to my patch.

> Allow CodecFactory to return a codec object given a codec' class name
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6996
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6996
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: getCodecByClassName.patch
>
>
> CodecFactory specify the list of codec that are supported by Hadoop. However, it returns a codec only by a file's name. I would like to make getCodec method to alternatively take a codec's class name.
> This is required by  HDFS-1435, where
> 1. it allows an HDFS admin to configure which codec to use to save an image. 
> 2. It stores the codec class name in its on-disk image instead of a file's suffix.
> When saving and reading an image, I'd like to get an codec from CodecFactory by its class name. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10991-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 11 21:39:55 2010
Return-Path: <common-issues-return-10991-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66998 invoked from network); 11 Oct 2010 21:39:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Oct 2010 21:39:55 -0000
Received: (qmail 23967 invoked by uid 500); 11 Oct 2010 21:39:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23944 invoked by uid 500); 11 Oct 2010 21:39:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23936 invoked by uid 99); 11 Oct 2010 21:39:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Oct 2010 21:39:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Oct 2010 21:39:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9BLdYm3008248
	for <common-issues@hadoop.apache.org>; Mon, 11 Oct 2010 21:39:34 GMT
Message-ID: <3422965.83821286833174224.JavaMail.jira@thor>
Date: Mon, 11 Oct 2010 17:39:34 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6996) Allow CodecFactory to return a codec
 object given a codec' class name
In-Reply-To: <1541599.80951286821892815.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6996:
----------------------------------

    Attachment: getCodecByClassName1.patch

This patch additionally adds a default constructor to CompressionCodecFactory.

> Allow CodecFactory to return a codec object given a codec' class name
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6996
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6996
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: getCodecByClassName.patch, getCodecByClassName1.patch
>
>
> CodecFactory specify the list of codec that are supported by Hadoop. However, it returns a codec only by a file's name. I would like to make getCodec method to alternatively take a codec's class name.
> This is required by  HDFS-1435, where
> 1. it allows an HDFS admin to configure which codec to use to save an image. 
> 2. It stores the codec class name in its on-disk image instead of a file's suffix.
> When saving and reading an image, I'd like to get an codec from CodecFactory by its class name. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10992-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 12 04:30:03 2010
Return-Path: <common-issues-return-10992-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19747 invoked from network); 12 Oct 2010 04:30:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Oct 2010 04:30:02 -0000
Received: (qmail 63019 invoked by uid 500); 12 Oct 2010 04:30:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62766 invoked by uid 500); 12 Oct 2010 04:29:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62751 invoked by uid 99); 12 Oct 2010 04:29:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Oct 2010 04:29:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Oct 2010 04:29:56 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9C4TYxg014705
	for <common-issues@hadoop.apache.org>; Tue, 12 Oct 2010 04:29:36 GMT
Message-ID: <29443531.89531286857774868.JavaMail.jira@thor>
Date: Tue, 12 Oct 2010 00:29:34 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6997) 'hadoop' script should set LANG or
 LC_COLLATE explicitly for classpath order
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

'hadoop' script should set LANG or LC_COLLATE explicitly for classpath order
----------------------------------------------------------------------------

                 Key: HADOOP-6997
                 URL: https://issues.apache.org/jira/browse/HADOOP-6997
             Project: Hadoop Common
          Issue Type: Bug
          Components: scripts
    Affects Versions: 0.21.0, 0.20.2, 0.22.0
            Reporter: Greg Roelofs


The 'hadoop' script builds the classpath in pieces, including the following bit for the bulk of it:

{noformat}
# add libs to CLASSPATH
for f in $HADOOP_HOME/lib/*.jar; do
  CLASSPATH=${CLASSPATH}:$f;
done
{noformat}

The ordering of "*.jar", i.e., the collation order, depends on either LANG or LC_COLLATE on Linux systems.  In the absence of either one, the script will default to whatever the user's environment specifies; for Red Hat, the default is "en_US", which is a case-insensitive (and punctuation-insensitive?) ordering.  If LANG is set to "C" instead, the ordering changes to the ASCII/UTF-8 byte ordering.

The key issue here is that $HADOOP_HOME/lib contains both upper- and lowercase jar names (e.g., "SimonTool.jar" and "commons-logging-1.1.1.jar", to pick a completely random pair), which will have an inverted order depending on which setting is used.

'hadoop' should explicitly set LANG and/or LC_COLLATE to whatever setting it's implicitly assuming.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10993-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 12 05:14:57 2010
Return-Path: <common-issues-return-10993-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30577 invoked from network); 12 Oct 2010 05:14:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Oct 2010 05:14:57 -0000
Received: (qmail 84890 invoked by uid 500); 12 Oct 2010 05:14:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84859 invoked by uid 500); 12 Oct 2010 05:14:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84850 invoked by uid 99); 12 Oct 2010 05:14:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Oct 2010 05:14:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Oct 2010 05:14:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9C5EVm7015501
	for <common-issues@hadoop.apache.org>; Tue, 12 Oct 2010 05:14:33 GMT
Message-ID: <26842967.90301286860471693.JavaMail.jira@thor>
Date: Tue, 12 Oct 2010 01:14:31 -0400 (EDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6996) Allow CodecFactory to return a
 codec object given a codec' class name
In-Reply-To: <1541599.80951286821892815.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920096#action_12920096 ] 

dhruba borthakur commented on HADOOP-6996:
------------------------------------------

Code looks good. One minor comment:

{code}
public CompressionCodec getCodecByClassName(String classname) {
    if (codecsByClassName == null) {
     return null;
   }
    return codecsByClassName.get(classname);
}

{code}

do we really need to check for codecsByClassName == null? I see that it always initialized in the constructor.

> Allow CodecFactory to return a codec object given a codec' class name
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6996
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6996
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: getCodecByClassName.patch, getCodecByClassName1.patch
>
>
> CodecFactory specify the list of codec that are supported by Hadoop. However, it returns a codec only by a file's name. I would like to make getCodec method to alternatively take a codec's class name.
> This is required by  HDFS-1435, where
> 1. it allows an HDFS admin to configure which codec to use to save an image. 
> 2. It stores the codec class name in its on-disk image instead of a file's suffix.
> When saving and reading an image, I'd like to get an codec from CodecFactory by its class name. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10994-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 12 12:19:18 2010
Return-Path: <common-issues-return-10994-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21183 invoked from network); 12 Oct 2010 12:19:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Oct 2010 12:19:05 -0000
Received: (qmail 89044 invoked by uid 500); 12 Oct 2010 12:19:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87897 invoked by uid 500); 12 Oct 2010 12:18:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87872 invoked by uid 99); 12 Oct 2010 12:18:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Oct 2010 12:18:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Oct 2010 12:18:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9CCIWYT021548
	for <common-issues@hadoop.apache.org>; Tue, 12 Oct 2010 12:18:33 GMT
Message-ID: <3791054.93961286885912898.JavaMail.jira@thor>
Date: Tue, 12 Oct 2010 08:18:32 -0400 (EDT)
From: "Alex Baranau (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6998) Add alternative search-provider to
 site
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Add alternative search-provider to site
---------------------------------------

                 Key: HADOOP-6998
                 URL: https://issues.apache.org/jira/browse/HADOOP-6998
             Project: Hadoop Common
          Issue Type: Improvement
          Components: documentation
            Reporter: Alex Baranau
            Priority: Minor


Use search-hadoop.com service to make available search in HDFS sources, MLs, wiki, etc.
This was initially proposed on user mailing list. The search service was already added in site's skin (common for all Hadoop related projects) before so this issue is about enabling it for Common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10995-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 12 12:20:59 2010
Return-Path: <common-issues-return-10995-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22714 invoked from network); 12 Oct 2010 12:20:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Oct 2010 12:20:59 -0000
Received: (qmail 93819 invoked by uid 500); 12 Oct 2010 12:20:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93731 invoked by uid 500); 12 Oct 2010 12:20:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93719 invoked by uid 99); 12 Oct 2010 12:20:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Oct 2010 12:20:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Oct 2010 12:20:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9CCKWiM021584
	for <common-issues@hadoop.apache.org>; Tue, 12 Oct 2010 12:20:33 GMT
Message-ID: <25031022.93991286886032949.JavaMail.jira@thor>
Date: Tue, 12 Oct 2010 08:20:32 -0400 (EDT)
From: "Alex Baranau (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6998) Add alternative search-provider to
 site
In-Reply-To: <3791054.93961286885912898.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alex Baranau updated HADOOP-6998:
---------------------------------

    Status: Patch Available  (was: Open)

> Add alternative search-provider to site
> ---------------------------------------
>
>                 Key: HADOOP-6998
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6998
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Alex Baranau
>            Priority: Minor
>
> Use search-hadoop.com service to make available search in HDFS sources, MLs, wiki, etc.
> This was initially proposed on user mailing list. The search service was already added in site's skin (common for all Hadoop related projects) before so this issue is about enabling it for Common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10996-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 12 12:25:05 2010
Return-Path: <common-issues-return-10996-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30571 invoked from network); 12 Oct 2010 12:24:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Oct 2010 12:24:58 -0000
Received: (qmail 445 invoked by uid 500); 12 Oct 2010 12:24:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 369 invoked by uid 500); 12 Oct 2010 12:24:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 361 invoked by uid 99); 12 Oct 2010 12:24:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Oct 2010 12:24:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Oct 2010 12:24:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9CCOWWr021624
	for <common-issues@hadoop.apache.org>; Tue, 12 Oct 2010 12:24:33 GMT
Message-ID: <17351953.94011286886272896.JavaMail.jira@thor>
Date: Tue, 12 Oct 2010 08:24:32 -0400 (EDT)
From: "Alex Baranau (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6998) Add alternative search-provider to
 site
In-Reply-To: <3791054.93961286885912898.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alex Baranau updated HADOOP-6998:
---------------------------------

    Attachment: HADOOP-6998-common.patch
                HADOOP-6998.patch

Patches applicable to:

https://svn.apache.org/repos/asf/hadoop/site/
https://svn.apache.org/repos/asf/hadoop/common/site/

> Add alternative search-provider to site
> ---------------------------------------
>
>                 Key: HADOOP-6998
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6998
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Alex Baranau
>            Priority: Minor
>         Attachments: HADOOP-6998-common.patch, HADOOP-6998.patch
>
>
> Use search-hadoop.com service to make available search in HDFS sources, MLs, wiki, etc.
> This was initially proposed on user mailing list. The search service was already added in site's skin (common for all Hadoop related projects) before so this issue is about enabling it for Common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10997-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 12 12:28:59 2010
Return-Path: <common-issues-return-10997-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34918 invoked from network); 12 Oct 2010 12:28:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Oct 2010 12:28:59 -0000
Received: (qmail 7161 invoked by uid 500); 12 Oct 2010 12:28:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6813 invoked by uid 500); 12 Oct 2010 12:28:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6804 invoked by uid 99); 12 Oct 2010 12:28:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Oct 2010 12:28:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Oct 2010 12:28:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9CCSYko021674
	for <common-issues@hadoop.apache.org>; Tue, 12 Oct 2010 12:28:35 GMT
Message-ID: <10148533.94051286886514830.JavaMail.jira@thor>
Date: Tue, 12 Oct 2010 08:28:34 -0400 (EDT)
From: "Alex Baranau (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6998) Add alternative search-provider to
 site
In-Reply-To: <3791054.93961286885912898.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alex Baranau updated HADOOP-6998:
---------------------------------

    Description: 
Use search-hadoop.com service to make available search in sources, MLs, wiki, etc.
This was initially proposed on user mailing list. The search service was already added in site's skin (common for all Hadoop related projects) before so this issue is about enabling it for Common.

  was:
Use search-hadoop.com service to make available search in HDFS sources, MLs, wiki, etc.
This was initially proposed on user mailing list. The search service was already added in site's skin (common for all Hadoop related projects) before so this issue is about enabling it for Common.


> Add alternative search-provider to site
> ---------------------------------------
>
>                 Key: HADOOP-6998
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6998
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Alex Baranau
>            Priority: Minor
>         Attachments: HADOOP-6998-common.patch, HADOOP-6998.patch
>
>
> Use search-hadoop.com service to make available search in sources, MLs, wiki, etc.
> This was initially proposed on user mailing list. The search service was already added in site's skin (common for all Hadoop related projects) before so this issue is about enabling it for Common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10998-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 12 16:49:56 2010
Return-Path: <common-issues-return-10998-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30988 invoked from network); 12 Oct 2010 16:49:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Oct 2010 16:49:55 -0000
Received: (qmail 27641 invoked by uid 500); 12 Oct 2010 16:49:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27584 invoked by uid 500); 12 Oct 2010 16:49:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27575 invoked by uid 99); 12 Oct 2010 16:49:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Oct 2010 16:49:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Oct 2010 16:49:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9CGnX4O026001
	for <common-issues@hadoop.apache.org>; Tue, 12 Oct 2010 16:49:33 GMT
Message-ID: <4191358.98271286902173375.JavaMail.jira@thor>
Date: Tue, 12 Oct 2010 12:49:33 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6996) Allow CodecFactory to return a
 codec object given a codec' class name
In-Reply-To: <1541599.80951286821892815.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920254#action_12920254 ] 

Hairong Kuang commented on HADOOP-6996:
---------------------------------------

You are right. But before I add the default class constructor, there was a possibility that codecClassName is null. Now I realize that adding the default constructor is an incompatible change. Let me revert this change by deleting the 2nd patch: getCodecByClassName1.patch.

> Allow CodecFactory to return a codec object given a codec' class name
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6996
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6996
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: getCodecByClassName.patch
>
>
> CodecFactory specify the list of codec that are supported by Hadoop. However, it returns a codec only by a file's name. I would like to make getCodec method to alternatively take a codec's class name.
> This is required by  HDFS-1435, where
> 1. it allows an HDFS admin to configure which codec to use to save an image. 
> 2. It stores the codec class name in its on-disk image instead of a file's suffix.
> When saving and reading an image, I'd like to get an codec from CodecFactory by its class name. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10999-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 12 16:49:58 2010
Return-Path: <common-issues-return-10999-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31123 invoked from network); 12 Oct 2010 16:49:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Oct 2010 16:49:58 -0000
Received: (qmail 28049 invoked by uid 500); 12 Oct 2010 16:49:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27975 invoked by uid 500); 12 Oct 2010 16:49:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27967 invoked by uid 99); 12 Oct 2010 16:49:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Oct 2010 16:49:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Oct 2010 16:49:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9CGnY2k026017
	for <common-issues@hadoop.apache.org>; Tue, 12 Oct 2010 16:49:34 GMT
Message-ID: <1518379.98321286902174467.JavaMail.jira@thor>
Date: Tue, 12 Oct 2010 12:49:34 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6996) Allow CodecFactory to return a codec
 object given a codec' class name
In-Reply-To: <1541599.80951286821892815.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6996:
----------------------------------

    Attachment:     (was: getCodecByClassName1.patch)

> Allow CodecFactory to return a codec object given a codec' class name
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6996
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6996
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: getCodecByClassName.patch
>
>
> CodecFactory specify the list of codec that are supported by Hadoop. However, it returns a codec only by a file's name. I would like to make getCodec method to alternatively take a codec's class name.
> This is required by  HDFS-1435, where
> 1. it allows an HDFS admin to configure which codec to use to save an image. 
> 2. It stores the codec class name in its on-disk image instead of a file's suffix.
> When saving and reading an image, I'd like to get an codec from CodecFactory by its class name. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11000-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 12 17:53:57 2010
Return-Path: <common-issues-return-11000-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80022 invoked from network); 12 Oct 2010 17:53:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Oct 2010 17:53:57 -0000
Received: (qmail 49426 invoked by uid 500); 12 Oct 2010 17:53:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49391 invoked by uid 500); 12 Oct 2010 17:53:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49383 invoked by uid 99); 12 Oct 2010 17:53:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Oct 2010 17:53:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Oct 2010 17:53:57 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9CHraF9027244
	for <common-issues@hadoop.apache.org>; Tue, 12 Oct 2010 17:53:36 GMT
Message-ID: <2198347.99931286906016556.JavaMail.jira@thor>
Date: Tue, 12 Oct 2010 13:53:36 -0400 (EDT)
From: "Krishna Ramachandran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6999) security implementation for new
 FileSystem (FileContext) API
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

security implementation for new FileSystem (FileContext) API
------------------------------------------------------------

                 Key: HADOOP-6999
                 URL: https://issues.apache.org/jira/browse/HADOOP-6999
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Krishna Ramachandran


New file system API (HADOOP 4952) should implement security features currently provided by  FileSystem APIs
This is a critical requirement for MapReduce components to  migrate and use new APIs for internal  filesystem operations (MAPREDUCE 2020)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11003-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 12 22:39:04 2010
Return-Path: <common-issues-return-11003-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86362 invoked from network); 12 Oct 2010 22:38:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Oct 2010 22:38:58 -0000
Received: (qmail 29210 invoked by uid 500); 12 Oct 2010 22:38:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29181 invoked by uid 500); 12 Oct 2010 22:38:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29173 invoked by uid 99); 12 Oct 2010 22:38:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Oct 2010 22:38:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Oct 2010 22:38:56 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9CMcYSZ002685
	for <common-issues@hadoop.apache.org>; Tue, 12 Oct 2010 22:38:35 GMT
Message-ID: <25878961.105831286923114967.JavaMail.jira@thor>
Date: Tue, 12 Oct 2010 18:38:34 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6991) SequenceFile::Reader discards length
 for files, does not call openFile
In-Reply-To: <22609732.541901286245712924.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6991:
----------------------------------

    Attachment: h-6991-2.patch

Forgot test case.

> SequenceFile::Reader discards length for files, does not call openFile
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6991
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6991
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Chris Douglas
>            Priority: Minor
>         Attachments: 6991-0.patch, 6991-1.patch, h-6991-2.patch
>
>
> While the sorting and merging in {{SequenceFile}} is deprecated and {{openFile}} is archaic, the semantics should remain consistent.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11001-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 12 22:39:05 2010
Return-Path: <common-issues-return-11001-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86264 invoked from network); 12 Oct 2010 22:38:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Oct 2010 22:38:57 -0000
Received: (qmail 28751 invoked by uid 500); 12 Oct 2010 22:38:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28689 invoked by uid 500); 12 Oct 2010 22:38:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28675 invoked by uid 99); 12 Oct 2010 22:38:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Oct 2010 22:38:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Oct 2010 22:38:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9CMcW2i002657
	for <common-issues@hadoop.apache.org>; Tue, 12 Oct 2010 22:38:33 GMT
Message-ID: <27332509.105741286923112973.JavaMail.jira@thor>
Date: Tue, 12 Oct 2010 18:38:32 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6991) SequenceFile::Reader discards length
 for files, does not call openFile
In-Reply-To: <22609732.541901286245712924.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6991:
----------------------------------

    Attachment: h-6991-2.patch

Updated Chris' patch with the test case moved from MapRed and turning on overwrite.

> SequenceFile::Reader discards length for files, does not call openFile
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6991
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6991
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Chris Douglas
>            Priority: Minor
>         Attachments: 6991-0.patch, 6991-1.patch, h-6991-2.patch
>
>
> While the sorting and merging in {{SequenceFile}} is deprecated and {{openFile}} is archaic, the semantics should remain consistent.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11002-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 12 22:39:06 2010
Return-Path: <common-issues-return-11002-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86275 invoked from network); 12 Oct 2010 22:38:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Oct 2010 22:38:57 -0000
Received: (qmail 28773 invoked by uid 500); 12 Oct 2010 22:38:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28701 invoked by uid 500); 12 Oct 2010 22:38:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28690 invoked by uid 99); 12 Oct 2010 22:38:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Oct 2010 22:38:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Oct 2010 22:38:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9CMcXfB002666
	for <common-issues@hadoop.apache.org>; Tue, 12 Oct 2010 22:38:33 GMT
Message-ID: <5994033.105771286923113516.JavaMail.jira@thor>
Date: Tue, 12 Oct 2010 18:38:33 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6991) SequenceFile::Reader discards length
 for files, does not call openFile
In-Reply-To: <22609732.541901286245712924.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6991:
----------------------------------

    Attachment:     (was: h-6991-2.patch)

> SequenceFile::Reader discards length for files, does not call openFile
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6991
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6991
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Chris Douglas
>            Priority: Minor
>         Attachments: 6991-0.patch, 6991-1.patch, h-6991-2.patch
>
>
> While the sorting and merging in {{SequenceFile}} is deprecated and {{openFile}} is archaic, the semantics should remain consistent.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11004-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 12 23:32:56 2010
Return-Path: <common-issues-return-11004-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40782 invoked from network); 12 Oct 2010 23:32:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Oct 2010 23:32:55 -0000
Received: (qmail 6514 invoked by uid 500); 12 Oct 2010 23:32:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6471 invoked by uid 500); 12 Oct 2010 23:32:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6463 invoked by uid 99); 12 Oct 2010 23:32:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Oct 2010 23:32:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Oct 2010 23:32:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9CNWYDS003376
	for <common-issues@hadoop.apache.org>; Tue, 12 Oct 2010 23:32:34 GMT
Message-ID: <18207176.106251286926354632.JavaMail.jira@thor>
Date: Tue, 12 Oct 2010 19:32:34 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6818) Provide a JNI-based implementation of
 GroupMappingServiceProvider
In-Reply-To: <17262845.29701276200793990.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6818:
--------------------------------

    Attachment: 6818-trunk-1.patch

Addresses the comments from Todd.

> Provide a JNI-based implementation of GroupMappingServiceProvider
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6818
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6818
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6818-trunk-1.patch, 6818-trunk.patch, hadoop-6818-1.patch, hadoop-6818-2.patch, JNIGroupMapping.patch
>
>
> The default implementation of GroupMappingServiceProvider does a fork of a unix command to get the groups of a user. Since the group resolution happens in the servers, this might be costly. This jira aims at providing a JNI-based implementation for GroupMappingServiceProvider.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11005-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 13 00:24:57 2010
Return-Path: <common-issues-return-11005-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78544 invoked from network); 13 Oct 2010 00:24:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Oct 2010 00:24:57 -0000
Received: (qmail 68196 invoked by uid 500); 13 Oct 2010 00:24:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68170 invoked by uid 500); 13 Oct 2010 00:24:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68162 invoked by uid 99); 13 Oct 2010 00:24:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 00:24:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 00:24:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9D0OXjs004064
	for <common-issues@hadoop.apache.org>; Wed, 13 Oct 2010 00:24:33 GMT
Message-ID: <16879323.106841286929473131.JavaMail.jira@thor>
Date: Tue, 12 Oct 2010 20:24:33 -0400 (EDT)
From: "Krishna Ramachandran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-7000) Options.createOpts should provide API
 to set Progressable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Options.createOpts should provide API to set Progressable
---------------------------------------------------------

                 Key: HADOOP-7000
                 URL: https://issues.apache.org/jira/browse/HADOOP-7000
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Krishna Ramachandran


Hadoop common FileContext APIs provide means to create OutPutStream with write-progress reporting using CreateOpts. 

In existing Options.CreateOpts.Progess the constructor (with Progressable as argument) is protected.  A method to obtain a Progress instance is missing. 

Attaching a preliminary suggested fix


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11006-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 13 00:26:57 2010
Return-Path: <common-issues-return-11006-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78798 invoked from network); 13 Oct 2010 00:26:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Oct 2010 00:26:57 -0000
Received: (qmail 69043 invoked by uid 500); 13 Oct 2010 00:26:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69018 invoked by uid 500); 13 Oct 2010 00:26:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69010 invoked by uid 99); 13 Oct 2010 00:26:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 00:26:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 00:26:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9D0QXN1004095
	for <common-issues@hadoop.apache.org>; Wed, 13 Oct 2010 00:26:33 GMT
Message-ID: <8368005.106871286929593264.JavaMail.jira@thor>
Date: Tue, 12 Oct 2010 20:26:33 -0400 (EDT)
From: "Krishna Ramachandran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7000) Options.createOpts should provide API
 to set Progressable
In-Reply-To: <16879323.106841286929473131.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Krishna Ramachandran updated HADOOP-7000:
-----------------------------------------

    Attachment: HADOOP-7000.patch

simple suggested fix

> Options.createOpts should provide API to set Progressable
> ---------------------------------------------------------
>
>                 Key: HADOOP-7000
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7000
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Krishna Ramachandran
>         Attachments: HADOOP-7000.patch
>
>
> Hadoop common FileContext APIs provide means to create OutPutStream with write-progress reporting using CreateOpts. 
> In existing Options.CreateOpts.Progess the constructor (with Progressable as argument) is protected.  A method to obtain a Progress instance is missing. 
> Attaching a preliminary suggested fix

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11007-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 13 00:30:57 2010
Return-Path: <common-issues-return-11007-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83589 invoked from network); 13 Oct 2010 00:30:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Oct 2010 00:30:57 -0000
Received: (qmail 72704 invoked by uid 500); 13 Oct 2010 00:30:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72673 invoked by uid 500); 13 Oct 2010 00:30:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72647 invoked by uid 99); 13 Oct 2010 00:30:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 00:30:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 00:30:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9D0UXGk004137
	for <common-issues@hadoop.apache.org>; Wed, 13 Oct 2010 00:30:33 GMT
Message-ID: <13215799.106891286929833272.JavaMail.jira@thor>
Date: Tue, 12 Oct 2010 20:30:33 -0400 (EDT)
From: "Krishna Ramachandran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7000) Options.createOpts should provide API
 to access Progress
In-Reply-To: <16879323.106841286929473131.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Krishna Ramachandran updated HADOOP-7000:
-----------------------------------------

    Summary: Options.createOpts should provide API to access Progress  (was: Options.createOpts should provide API to set Progressable)

> Options.createOpts should provide API to access Progress
> --------------------------------------------------------
>
>                 Key: HADOOP-7000
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7000
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Krishna Ramachandran
>         Attachments: HADOOP-7000.patch
>
>
> Hadoop common FileContext APIs provide means to create OutPutStream with write-progress reporting using CreateOpts. 
> In existing Options.CreateOpts.Progess the constructor (with Progressable as argument) is protected.  A method to obtain a Progress instance is missing. 
> Attaching a preliminary suggested fix

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11008-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 13 13:43:04 2010
Return-Path: <common-issues-return-11008-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50009 invoked from network); 13 Oct 2010 13:43:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Oct 2010 13:43:03 -0000
Received: (qmail 69748 invoked by uid 500); 13 Oct 2010 13:43:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68032 invoked by uid 500); 13 Oct 2010 13:43:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67883 invoked by uid 99); 13 Oct 2010 13:42:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 13:42:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 13:42:57 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9DDgZpD015310
	for <common-issues@hadoop.apache.org>; Wed, 13 Oct 2010 13:42:35 GMT
Message-ID: <30314810.113111286977355852.JavaMail.jira@thor>
Date: Wed, 13 Oct 2010 09:42:35 -0400 (EDT)
From: "Otis Gospodnetic (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6998) Add alternative search-provider to
 site
In-Reply-To: <3791054.93961286885912898.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920546#action_12920546 ] 

Otis Gospodnetic commented on HADOOP-6998:
------------------------------------------

For what it's worth, the following issues for the same functionality on other sub-projects have been committed so far: PIG-1661 HBASE-2886 HDFS-1367.

> Add alternative search-provider to site
> ---------------------------------------
>
>                 Key: HADOOP-6998
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6998
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Alex Baranau
>            Priority: Minor
>         Attachments: HADOOP-6998-common.patch, HADOOP-6998.patch
>
>
> Use search-hadoop.com service to make available search in sources, MLs, wiki, etc.
> This was initially proposed on user mailing list. The search service was already added in site's skin (common for all Hadoop related projects) before so this issue is about enabling it for Common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11009-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 13 16:01:04 2010
Return-Path: <common-issues-return-11009-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8716 invoked from network); 13 Oct 2010 16:01:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Oct 2010 16:01:04 -0000
Received: (qmail 33779 invoked by uid 500); 13 Oct 2010 16:01:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33751 invoked by uid 500); 13 Oct 2010 16:01:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33737 invoked by uid 99); 13 Oct 2010 16:01:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 16:01:03 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 16:01:01 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9DG0dgh017883
	for <common-issues@hadoop.apache.org>; Wed, 13 Oct 2010 16:00:39 GMT
Message-ID: <33236541.115761286985639492.JavaMail.jira@thor>
Date: Wed, 13 Oct 2010 12:00:39 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6998) Add alternative search-provider to
 site
In-Reply-To: <3791054.93961286885912898.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920625#action_12920625 ] 

Doug Cutting commented on HADOOP-6998:
--------------------------------------

Unless there are objections, I'll commit this in the next few days.

> Add alternative search-provider to site
> ---------------------------------------
>
>                 Key: HADOOP-6998
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6998
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Alex Baranau
>            Priority: Minor
>         Attachments: HADOOP-6998-common.patch, HADOOP-6998.patch
>
>
> Use search-hadoop.com service to make available search in sources, MLs, wiki, etc.
> This was initially proposed on user mailing list. The search service was already added in site's skin (common for all Hadoop related projects) before so this issue is about enabling it for Common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11010-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 13 20:19:55 2010
Return-Path: <common-issues-return-11010-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33509 invoked from network); 13 Oct 2010 20:19:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Oct 2010 20:19:55 -0000
Received: (qmail 38284 invoked by uid 500); 13 Oct 2010 20:19:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38224 invoked by uid 500); 13 Oct 2010 20:19:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38216 invoked by uid 99); 13 Oct 2010 20:19:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 20:19:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 20:19:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9DKJXFv023569
	for <common-issues@hadoop.apache.org>; Wed, 13 Oct 2010 20:19:33 GMT
Message-ID: <33337630.131381287001173475.JavaMail.jira@thor>
Date: Wed, 13 Oct 2010 16:19:33 -0400 (EDT)
From: "Krishna Ramachandran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7000) Options.createOpts should provide API
 to access Progress
In-Reply-To: <16879323.106841286929473131.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Krishna Ramachandran updated HADOOP-7000:
-----------------------------------------

    Attachment: HADOOP-7000-1.patch

Updated test to include Progress

> Options.createOpts should provide API to access Progress
> --------------------------------------------------------
>
>                 Key: HADOOP-7000
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7000
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Krishna Ramachandran
>         Attachments: HADOOP-7000-1.patch, HADOOP-7000.patch
>
>
> Hadoop common FileContext APIs provide means to create OutPutStream with write-progress reporting using CreateOpts. 
> In existing Options.CreateOpts.Progess the constructor (with Progressable as argument) is protected.  A method to obtain a Progress instance is missing. 
> Attaching a preliminary suggested fix

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11011-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 13 23:16:56 2010
Return-Path: <common-issues-return-11011-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25476 invoked from network); 13 Oct 2010 23:16:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Oct 2010 23:16:56 -0000
Received: (qmail 79779 invoked by uid 500); 13 Oct 2010 23:16:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79754 invoked by uid 500); 13 Oct 2010 23:16:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79745 invoked by uid 99); 13 Oct 2010 23:16:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 23:16:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Oct 2010 23:16:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9DNGYfn026318
	for <common-issues@hadoop.apache.org>; Wed, 13 Oct 2010 23:16:34 GMT
Message-ID: <7600255.133881287011794656.JavaMail.jira@thor>
Date: Wed, 13 Oct 2010 19:16:34 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6818) Provide a JNI-based implementation
 of GroupMappingServiceProvider
In-Reply-To: <17262845.29701276200793990.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920802#action_12920802 ] 

Erik Steffl commented on HADOOP-6818:
-------------------------------------

6818-trunk-1.patch review, nothing major, just few details to consider.

  - line length over 80 chars, mismatched intendation in few cases (e.g. JniBasedUnixGroupsMapping.java line: "be loaded");) etc., not sure how much we care about detials like that

  - TestJNIGroupsMapping.java - test compares groups returned by ShellBasedUnixGroupsMapping and JniBasedUnixGroupsMapping, first it compares group lists for size then it looks up whether all groups returned by ShellBasedUnixGroupsMapping are also in JniBasedUnixGroupsMapping. This comparison would fail for lists like (a, a, b) and (a, b, c). It's very unlikely that ShellBasedUnixGroupsMapping would list the same group twice but it's fairly cheap to do exact comparison (i.e. sort the lists and compare the items one by one)

  - JniBasedUnixGroupsMapping.java getGroups: is it not possible to return empty array from getGroupForUser and get rid of if (groups != null && groups.length != 0) and the rest of the function?

  - configure: what's the point of (unset CDPATH)? As far as I can tell it creates a subshell in which it unsets CDPATH which has no effect in current shell. I see the result is used to determine whether unset CDPATH is executed but why? Given that it's not clear maybe a comment would help. (or is this file generated?)

  - JniBasedUnixGroupsMapping.c: it looks like code would be a bit simpler of cleanup label did CHECK_ERROR and code that jumps to cleanup just set the error. If cuser is NULL it could also jump to cleanup. That way there would be only ene exit point of the function, one place responsible for resource cleanup etc.

  - getGroup.c: functions in this file do lot of pointer manipulation, it seems they would also benefit from same techinzue used in JniBasedUnixGroupsMapping.c, i.e. initialize all pointers to NULL, in case of error jump to cleanup label that cleans everything up. Would free maintainer from thinking about whether pwbuf or group (or both or neither) should be freed or not etc.

  -  getGroup.c getGroupIDList: in case of error ngroup is not reset to zero, shouldn't matter but would be more consistent (i.e. ngroups should always be same as number of groups in groups)

> Provide a JNI-based implementation of GroupMappingServiceProvider
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6818
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6818
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6818-trunk-1.patch, 6818-trunk.patch, hadoop-6818-1.patch, hadoop-6818-2.patch, JNIGroupMapping.patch
>
>
> The default implementation of GroupMappingServiceProvider does a fork of a unix command to get the groups of a user. Since the group resolution happens in the servers, this might be costly. This jira aims at providing a JNI-based implementation for GroupMappingServiceProvider.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11012-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 14 15:48:58 2010
Return-Path: <common-issues-return-11012-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39434 invoked from network); 14 Oct 2010 15:48:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Oct 2010 15:48:58 -0000
Received: (qmail 58049 invoked by uid 500); 14 Oct 2010 15:48:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58024 invoked by uid 500); 14 Oct 2010 15:48:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58016 invoked by uid 99); 14 Oct 2010 15:48:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Oct 2010 15:48:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Oct 2010 15:48:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9EFmXZw010753
	for <common-issues@hadoop.apache.org>; Thu, 14 Oct 2010 15:48:33 GMT
Message-ID: <29441159.143271287071313827.JavaMail.jira@thor>
Date: Thu, 14 Oct 2010 11:48:33 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6992) Update website for recent subproject
 departures
In-Reply-To: <26161850.16451286406151616.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting resolved HADOOP-6992.
----------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]

Hive has now moved its website, so I have now committed this.

> Update website for recent subproject departures
> -----------------------------------------------
>
>                 Key: HADOOP-6992
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6992
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: site
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: site
>
>         Attachments: HADOOP-6992.patch, hadoop-tlp.pdf
>
>
> A number of subprojects have left Hadoop yet the website's not been fully updated to reflect that.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11013-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 14 18:39:02 2010
Return-Path: <common-issues-return-11013-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90333 invoked from network); 14 Oct 2010 18:39:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Oct 2010 18:39:01 -0000
Received: (qmail 73551 invoked by uid 500); 14 Oct 2010 18:39:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73504 invoked by uid 500); 14 Oct 2010 18:39:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73496 invoked by uid 99); 14 Oct 2010 18:39:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Oct 2010 18:39:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Oct 2010 18:38:58 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9EIcakp014007
	for <common-issues@hadoop.apache.org>; Thu, 14 Oct 2010 18:38:36 GMT
Message-ID: <33220874.146971287081516580.JavaMail.jira@thor>
Date: Thu, 14 Oct 2010 14:38:36 -0400 (EDT)
From: "Patrick Kling (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-7001) Allow configuration changes without
 restarting configured nodes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Allow configuration changes without restarting configured nodes
---------------------------------------------------------------

                 Key: HADOOP-7001
                 URL: https://issues.apache.org/jira/browse/HADOOP-7001
             Project: Hadoop Common
          Issue Type: Task
            Reporter: Patrick Kling


Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:

interface ChangeableConfigured extends Configured {
   void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
}

The contract of changeConfiguration is as follows:
The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.

It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11014-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 14 19:19:59 2010
Return-Path: <common-issues-return-11014-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14693 invoked from network); 14 Oct 2010 19:19:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Oct 2010 19:19:58 -0000
Received: (qmail 83020 invoked by uid 500); 14 Oct 2010 19:19:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82972 invoked by uid 500); 14 Oct 2010 19:19:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82964 invoked by uid 99); 14 Oct 2010 19:19:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Oct 2010 19:19:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Oct 2010 19:19:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9EJJYrK014784
	for <common-issues@hadoop.apache.org>; Thu, 14 Oct 2010 19:19:34 GMT
Message-ID: <22328346.147801287083974191.JavaMail.jira@thor>
Date: Thu, 14 Oct 2010 15:19:34 -0400 (EDT)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7001) Allow configuration changes without
 restarting configured nodes
In-Reply-To: <33220874.146971287081516580.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921086#action_12921086 ] 

Aaron T. Myers commented on HADOOP-7001:
----------------------------------------

+1 - this would be extremely useful.

May I suggest, though, that we change the name of the interface to "Reconfigurable" ?

> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7001
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7001
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Patrick Kling
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
>    void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11015-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 14 19:35:58 2010
Return-Path: <common-issues-return-11015-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24171 invoked from network); 14 Oct 2010 19:35:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Oct 2010 19:35:58 -0000
Received: (qmail 23509 invoked by uid 500); 14 Oct 2010 19:35:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23468 invoked by uid 500); 14 Oct 2010 19:35:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23458 invoked by uid 99); 14 Oct 2010 19:35:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Oct 2010 19:35:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Oct 2010 19:35:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9EJZYIh015072
	for <common-issues@hadoop.apache.org>; Thu, 14 Oct 2010 19:35:34 GMT
Message-ID: <16986.148121287084934041.JavaMail.jira@thor>
Date: Thu, 14 Oct 2010 15:35:34 -0400 (EDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6996) Allow CodecFactory to return a
 codec object given a codec' class name
In-Reply-To: <1541599.80951286821892815.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921093#action_12921093 ] 

dhruba borthakur commented on HADOOP-6996:
------------------------------------------

+1 looks good to me.

> Allow CodecFactory to return a codec object given a codec' class name
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6996
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6996
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: getCodecByClassName.patch
>
>
> CodecFactory specify the list of codec that are supported by Hadoop. However, it returns a codec only by a file's name. I would like to make getCodec method to alternatively take a codec's class name.
> This is required by  HDFS-1435, where
> 1. it allows an HDFS admin to configure which codec to use to save an image. 
> 2. It stores the codec class name in its on-disk image instead of a file's suffix.
> When saving and reading an image, I'd like to get an codec from CodecFactory by its class name. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11016-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 14 19:49:56 2010
Return-Path: <common-issues-return-11016-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32464 invoked from network); 14 Oct 2010 19:49:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Oct 2010 19:49:55 -0000
Received: (qmail 58968 invoked by uid 500); 14 Oct 2010 19:49:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58811 invoked by uid 500); 14 Oct 2010 19:49:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58803 invoked by uid 99); 14 Oct 2010 19:49:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Oct 2010 19:49:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Oct 2010 19:49:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9EJnYsf015314
	for <common-issues@hadoop.apache.org>; Thu, 14 Oct 2010 19:49:35 GMT
Message-ID: <9748207.148421287085774917.JavaMail.jira@thor>
Date: Thu, 14 Oct 2010 15:49:34 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6628) Make Cloudera commit logs and Yahoo!
 change logs into web-readable formats
In-Reply-To: <1089769602.209181268333067243.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Allen Wittenauer resolved HADOOP-6628.
--------------------------------------

    Resolution: Fixed

> Make Cloudera commit logs and Yahoo! change logs into web-readable formats
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6628
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6628
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Allen Wittenauer
>            Assignee: Allen Wittenauer
>            Priority: Trivial
>         Attachments: mk-cdh-readable.pl, mk-y-readable.pl
>
>
> Here are some quick hacks to help people read the various "release notes" from the various distributions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11017-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 14 20:03:55 2010
Return-Path: <common-issues-return-11017-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38112 invoked from network); 14 Oct 2010 20:03:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Oct 2010 20:03:55 -0000
Received: (qmail 77467 invoked by uid 500); 14 Oct 2010 20:03:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77433 invoked by uid 500); 14 Oct 2010 20:03:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77425 invoked by uid 99); 14 Oct 2010 20:03:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Oct 2010 20:03:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Oct 2010 20:03:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9EK3YZK015586
	for <common-issues@hadoop.apache.org>; Thu, 14 Oct 2010 20:03:34 GMT
Message-ID: <13916308.148561287086614426.JavaMail.jira@thor>
Date: Thu, 14 Oct 2010 16:03:34 -0400 (EDT)
From: "Patrick Kling (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7001) Allow configuration changes without
 restarting configured nodes
In-Reply-To: <33220874.146971287081516580.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921102#action_12921102 ] 

Patrick Kling commented on HADOOP-7001:
---------------------------------------

I agree, Reconfigurable sounds better. Also, we would be extending the interface Configurable (not the class Configured, I got confused there). I also wanted to point out that Reconfigurable would introduce the new method changeConf in addition to the method setConf from Configurable because there is a difference in semantics:
changeConf would make sure that all changes immediately affect the behaviour of the node (or failing that an exception would be thrown), whereas setConf as it is used now does not give any such guarantee (since some configuration values might be cached in other variables and some behaviour might not be changeable without a restart).


> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7001
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7001
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Patrick Kling
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
>    void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11018-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 14 20:28:59 2010
Return-Path: <common-issues-return-11018-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50041 invoked from network); 14 Oct 2010 20:28:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Oct 2010 20:28:59 -0000
Received: (qmail 30904 invoked by uid 500); 14 Oct 2010 20:28:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30866 invoked by uid 500); 14 Oct 2010 20:28:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30858 invoked by uid 99); 14 Oct 2010 20:28:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Oct 2010 20:28:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Oct 2010 20:28:56 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9EKSY23015921
	for <common-issues@hadoop.apache.org>; Thu, 14 Oct 2010 20:28:35 GMT
Message-ID: <29000552.148771287088114891.JavaMail.jira@thor>
Date: Thu, 14 Oct 2010 16:28:34 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7001) Allow configuration changes without
 restarting configured nodes
In-Reply-To: <33220874.146971287081516580.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921106#action_12921106 ] 

Konstantin Boudnik commented on HADOOP-7001:
--------------------------------------------

Also, this Herriot system testing framework will be highly benefited from such a change. +1 on the idea.

> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7001
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7001
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Patrick Kling
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
>    void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11019-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 14 20:40:59 2010
Return-Path: <common-issues-return-11019-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55001 invoked from network); 14 Oct 2010 20:40:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Oct 2010 20:40:59 -0000
Received: (qmail 58431 invoked by uid 500); 14 Oct 2010 20:40:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58399 invoked by uid 500); 14 Oct 2010 20:40:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58391 invoked by uid 99); 14 Oct 2010 20:40:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Oct 2010 20:40:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Oct 2010 20:40:56 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9EKeZGA016141
	for <common-issues@hadoop.apache.org>; Thu, 14 Oct 2010 20:40:35 GMT
Message-ID: <32046555.149121287088835285.JavaMail.jira@thor>
Date: Thu, 14 Oct 2010 16:40:35 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6628) Make Cloudera commit logs and
 Yahoo! change logs into web-readable formats
In-Reply-To: <1089769602.209181268333067243.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921113#action_12921113 ] 

Eli Collins commented on HADOOP-6628:
-------------------------------------

Hey Allen,

There's a jira now for any issues not covered on issues.apache.org:  https://issues.cloudera.org/browse/DISTRO

Thanks,
Eli

> Make Cloudera commit logs and Yahoo! change logs into web-readable formats
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6628
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6628
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Allen Wittenauer
>            Assignee: Allen Wittenauer
>            Priority: Trivial
>         Attachments: mk-cdh-readable.pl, mk-y-readable.pl
>
>
> Here are some quick hacks to help people read the various "release notes" from the various distributions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11020-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 14 21:09:03 2010
Return-Path: <common-issues-return-11020-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66552 invoked from network); 14 Oct 2010 21:09:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Oct 2010 21:09:03 -0000
Received: (qmail 14783 invoked by uid 500); 14 Oct 2010 21:09:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14755 invoked by uid 500); 14 Oct 2010 21:09:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14747 invoked by uid 99); 14 Oct 2010 21:09:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Oct 2010 21:09:03 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Oct 2010 21:09:01 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9EL8dEH016731
	for <common-issues@hadoop.apache.org>; Thu, 14 Oct 2010 21:08:39 GMT
Message-ID: <16365865.149821287090519117.JavaMail.jira@thor>
Date: Thu, 14 Oct 2010 17:08:39 -0400 (EDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7001) Allow configuration changes without
 restarting configured nodes
In-Reply-To: <33220874.146971287081516580.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921128#action_12921128 ] 

dhruba borthakur commented on HADOOP-7001:
------------------------------------------

It would be nice if you can paste the javadocs for this new interface and the method(s) in it. This will help understand the new API.

> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7001
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7001
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Patrick Kling
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
>    void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11021-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 14 21:19:01 2010
Return-Path: <common-issues-return-11021-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70484 invoked from network); 14 Oct 2010 21:19:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Oct 2010 21:19:00 -0000
Received: (qmail 30632 invoked by uid 500); 14 Oct 2010 21:19:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30604 invoked by uid 500); 14 Oct 2010 21:19:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30596 invoked by uid 99); 14 Oct 2010 21:19:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Oct 2010 21:19:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Oct 2010 21:18:57 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9ELIaZ0016906
	for <common-issues@hadoop.apache.org>; Thu, 14 Oct 2010 21:18:36 GMT
Message-ID: <22075339.150041287091115980.JavaMail.jira@thor>
Date: Thu, 14 Oct 2010 17:18:35 -0400 (EDT)
From: "Patrick Kling (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7001) Allow configuration changes without
 restarting configured nodes
In-Reply-To: <33220874.146971287081516580.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921133#action_12921133 ] 

Patrick Kling commented on HADOOP-7001:
---------------------------------------

org.apache.hadoop.conf
Interface Reconfigurable

   All Superinterfaces:
          Configurable  
     ___________________________________________________________________________________________________________________________________________

public interface Reconfigurable

   extends Configurable
   Something whose Configuration can be changed at run time.
     ___________________________________________________________________________________________________________________________________________

   Method Summary
    void changeConf(Configuration conf)
             changes the configuration to the configuration passed if it is not possible to change a configuration option a ConfigurationException is
   thrown and no changes are made to the current configuration



   Methods inherited from interface org.apache.hadoop.conf.Configurable
   getConf, setConf



   Method Detail

  changeConf

void changeConf(Configuration conf)
                throws ConfigurationChangeException

          changes the configuration to the configuration passed if it is not possible to change a configuration option a ConfigurationException is thrown
          and no changes are made to the current configuration

        Throws:
                ConfigurationChangeException



Class ConfigurationChangeException

java.lang.Object
   extended by  java.lang.Throwable
       extended by  java.lang.Exception
           extended by  org.apache.hadoop.conf.ConfigurationChangeException

   All Implemented Interfaces:
          Serializable
     ___________________________________________________________________________________________________________________________________________

public class ConfigurationChangeException

   extends Exception

   exception indicating that configuration property cannot be changed at run time

   See Also:
          Serialized Form
     ___________________________________________________________________________________________________________________________________________

   Constructor Summary
   ConfigurationChangeException(String property)
             Creates a new instance of ConfigurationChangeException
   ConfigurationChangeException(String property, String newVal)
             Creates a new instance of ConfigurationChangeException
   ConfigurationChangeException(String property, String newVal, String oldVal)
             Creates a new instance of ConfigurationChangeException



   Method Summary
    String getMessage()
             gets message describing exception
    String getNewValue()
             gets value to which property was supposed to be changed
    String getOldValue()
             gets old value of property that cannot be changed
    String getProperty()
             gets property that cannot be changed



   Methods inherited from class java.lang.Throwable
   fillInStackTrace, getCause, getLocalizedMessage, getStackTrace, initCause, printStackTrace, printStackTrace, printStackTrace, setStackTrace, toString



   Methods inherited from class java.lang.Object
   clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait



   Constructor Detail

  ConfigurationChangeException

public ConfigurationChangeException(String property,
                                    String newVal,
                                    String oldVal)

          Creates a new instance of ConfigurationChangeException
     ___________________________________________________________________________________________________________________________________________

  ConfigurationChangeException

public ConfigurationChangeException(String property,
                                    String newVal)

          Creates a new instance of ConfigurationChangeException
     ___________________________________________________________________________________________________________________________________________

  ConfigurationChangeException

public ConfigurationChangeException(String property)

          Creates a new instance of ConfigurationChangeException

  getProperty

public String getProperty()

          gets property that cannot be changed
     ___________________________________________________________________________________________________________________________________________

  getNewValue

public String getNewValue()

          gets value to which property was supposed to be changed
     ___________________________________________________________________________________________________________________________________________

  getOldValue

public String getOldValue()

          gets old value of property that cannot be changed
     ___________________________________________________________________________________________________________________________________________

  getMessage

public String getMessage()

          gets message describing exception

        Overrides:
                getMessage in class Throwable





> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7001
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7001
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Patrick Kling
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
>    void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11022-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 14 21:34:55 2010
Return-Path: <common-issues-return-11022-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75676 invoked from network); 14 Oct 2010 21:34:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Oct 2010 21:34:55 -0000
Received: (qmail 60372 invoked by uid 500); 14 Oct 2010 21:34:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60336 invoked by uid 500); 14 Oct 2010 21:34:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60328 invoked by uid 99); 14 Oct 2010 21:34:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Oct 2010 21:34:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Oct 2010 21:34:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9ELYYlH017184
	for <common-issues@hadoop.apache.org>; Thu, 14 Oct 2010 21:34:34 GMT
Message-ID: <22619162.150491287092074006.JavaMail.jira@thor>
Date: Thu, 14 Oct 2010 17:34:34 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6998) Add alternative search-provider to
 site
In-Reply-To: <3791054.93961286885912898.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-6998:
---------------------------------

       Resolution: Fixed
    Fix Version/s: site
         Assignee: Alex Baranau
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I just committed this.  Thanks, Alex.

> Add alternative search-provider to site
> ---------------------------------------
>
>                 Key: HADOOP-6998
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6998
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Alex Baranau
>            Assignee: Alex Baranau
>            Priority: Minor
>             Fix For: site
>
>         Attachments: HADOOP-6998-common.patch, HADOOP-6998.patch
>
>
> Use search-hadoop.com service to make available search in sources, MLs, wiki, etc.
> This was initially proposed on user mailing list. The search service was already added in site's skin (common for all Hadoop related projects) before so this issue is about enabling it for Common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11023-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 15 04:40:00 2010
Return-Path: <common-issues-return-11023-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52262 invoked from network); 15 Oct 2010 04:40:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Oct 2010 04:40:00 -0000
Received: (qmail 73974 invoked by uid 500); 15 Oct 2010 04:40:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73943 invoked by uid 500); 15 Oct 2010 04:39:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73935 invoked by uid 99); 15 Oct 2010 04:39:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 04:39:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 04:39:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9F4dXgm023329
	for <common-issues@hadoop.apache.org>; Fri, 15 Oct 2010 04:39:33 GMT
Message-ID: <16645437.154731287117573453.JavaMail.jira@thor>
Date: Fri, 15 Oct 2010 00:39:33 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6977) Herriot daemon clients should vend
 statistics
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6977:
---------------------------------------

    Attachment: HADOOP-6977.patch

Making some changes to allow for daemon specific lookups into process environment. Specific implementations will be posted as hdfs/mr patches.

> Herriot daemon clients should vend statistics
> ---------------------------------------------
>
>                 Key: HADOOP-6977
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6977
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: FinderTest.java, HADOOP-6977.patch, HADOOP-6977.patch, HADOOP-6977.patch, HADOOP-6977.patch, HADOOP-6977.y20S.patch, HADOOP-6977.y20S.patch
>
>
> The HDFS web user interface serves useful information through dfshealth.jsp and dfsnodelist.jsp.
> The Herriot interface to Hadoop cluster daemons would benefit from the addition of some way to channel metics information.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11024-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 15 18:40:00 2010
Return-Path: <common-issues-return-11024-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25292 invoked from network); 15 Oct 2010 18:40:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Oct 2010 18:40:00 -0000
Received: (qmail 50354 invoked by uid 500); 15 Oct 2010 18:40:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50302 invoked by uid 500); 15 Oct 2010 18:39:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50291 invoked by uid 99); 15 Oct 2010 18:39:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 18:39:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 18:39:58 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9FIdc8p006972
	for <common-issues@hadoop.apache.org>; Fri, 15 Oct 2010 18:39:38 GMT
Message-ID: <775609.166691287167978285.JavaMail.jira@thor>
Date: Fri, 15 Oct 2010 14:39:38 -0400 (EDT)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6977) Herriot daemon clients should vend
 statistics
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6977?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D129=
21487#action_12921487 ]=20

Tanping Wang commented on HADOOP-6977:
--------------------------------------

In the newer patch, "HADOOP_OPTS" is passed in as parameter to check isJmxE=
nabled and getJmxPort.  I assume if user uses specific options for each dem=
on, for example enabling JMX for name node daemon by passing in HADOOP_NAME=
NODE_OPTS,=20
  export HADOOP_NAMENODE_OPTS=3D"-Dcom.sun.management.jmxremote $HADOOP_NAM=
ENODE_OPTS -Dcom.sun.management.jmxremote.port=3D8004=E2=80=B3
then the parameter to be searched will be changed from "HADOOP_OPTS" to "HA=
DOOP_NAMENODE_OPTS".  If my understanding is corret, I think the patch is g=
ood.

> Herriot daemon clients should vend statistics
> ---------------------------------------------
>
>                 Key: HADOOP-6977
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6977
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: FinderTest.java, HADOOP-6977.patch, HADOOP-6977.patc=
h, HADOOP-6977.patch, HADOOP-6977.patch, HADOOP-6977.y20S.patch, HADOOP-697=
7.y20S.patch
>
>
> The HDFS web user interface serves useful information through dfshealth.j=
sp and dfsnodelist.jsp.
> The Herriot interface to Hadoop cluster daemons would benefit from the ad=
dition of some way to channel metics information.

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11025-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 15 18:48:01 2010
Return-Path: <common-issues-return-11025-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28060 invoked from network); 15 Oct 2010 18:47:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Oct 2010 18:47:56 -0000
Received: (qmail 74446 invoked by uid 500); 15 Oct 2010 18:47:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74408 invoked by uid 500); 15 Oct 2010 18:47:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74400 invoked by uid 99); 15 Oct 2010 18:47:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 18:47:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 18:47:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9FIlZmB007216
	for <common-issues@hadoop.apache.org>; Fri, 15 Oct 2010 18:47:35 GMT
Message-ID: <8685742.167171287168455560.JavaMail.jira@thor>
Date: Fri, 15 Oct 2010 14:47:35 -0400 (EDT)
From: "Philip Zeyliger (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7001) Allow configuration changes without
 restarting configured nodes
In-Reply-To: <33220874.146971287081516580.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921493#action_12921493 ] 

Philip Zeyliger commented on HADOOP-7001:
-----------------------------------------

I might be grumpy, but I think the right way to deal with configuration changes in a distributed, fault-tolerant system is to just restart the daemon entirely.  We already have to deal with the daemon suddenly crashing and that not affecting the system too much, so I'm wary of extra complexity of yet another process.  In practice, many configuration variables are loaded at start time and then stored as statics: stuff like threadpool sizes, for example.  Not to mention that Configuration objects get copied along, so it's hard to make sure that a configuration change propagates to all possible children.

I'll point out that the namenode and the jobtracker's fair scheduler already have mechanisms for dynamic configuration changes.  In namenode, that's -refreshNodes.  In the jt, I think the fair scheduler re-reads its XML configuration file on occasion.  Both of these make a lot of sense: these are specific endpoints for managing specific data, and the semantics of those changes are easy to understand.

-- Philip

> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7001
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7001
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Patrick Kling
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
>    void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11026-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 15 20:51:58 2010
Return-Path: <common-issues-return-11026-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83093 invoked from network); 15 Oct 2010 20:51:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Oct 2010 20:51:58 -0000
Received: (qmail 34883 invoked by uid 500); 15 Oct 2010 20:51:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34793 invoked by uid 500); 15 Oct 2010 20:51:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34778 invoked by uid 99); 15 Oct 2010 20:51:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 20:51:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 20:51:57 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9FKpaL2009469
	for <common-issues@hadoop.apache.org>; Fri, 15 Oct 2010 20:51:36 GMT
Message-ID: <16221323.169571287175896839.JavaMail.jira@thor>
Date: Fri, 15 Oct 2010 16:51:36 -0400 (EDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7001) Allow configuration changes without
 restarting configured nodes
In-Reply-To: <33220874.146971287081516580.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921544#action_12921544 ] 

dhruba borthakur commented on HADOOP-7001:
------------------------------------------

But a simple thing like increasing the number of handler threads in a server require the downtime of the server. If we can make the namenode implement Reconfigurable, then we will have the ability to change a few select of parameters dynamically, isn't it?

> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7001
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7001
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Patrick Kling
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
>    void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11027-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 15 21:08:59 2010
Return-Path: <common-issues-return-11027-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92225 invoked from network); 15 Oct 2010 21:08:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Oct 2010 21:08:59 -0000
Received: (qmail 65764 invoked by uid 500); 15 Oct 2010 21:08:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65737 invoked by uid 500); 15 Oct 2010 21:08:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65729 invoked by uid 99); 15 Oct 2010 21:08:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 21:08:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 21:08:58 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9FL8bav009868
	for <common-issues@hadoop.apache.org>; Fri, 15 Oct 2010 21:08:38 GMT
Message-ID: <5588192.170011287176917638.JavaMail.jira@thor>
Date: Fri, 15 Oct 2010 17:08:37 -0400 (EDT)
From: "Patrick Kling (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7001) Allow configuration changes without
 restarting configured nodes
In-Reply-To: <33220874.146971287081516580.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921552#action_12921552 ] 

Patrick Kling commented on HADOOP-7001:
---------------------------------------

I absolutely agree that there will likely be many configuration options that cannot be changed at run time and implementing Reconfigurable does not require that we can make arbitrary changes. For the configuration changes that are possible at run time (such as the existing examples mentioned by Philip), it would be nice to have a consistent mechanism. The interface could also make it easier to propagate configuration changes (by calling changeConf on objects to which the current configuration was propagated).

This should be a bit easier to read than the JavaDoc I posted before:

{code}
package org.apache.hadoop.conf;

/**                                                                                                                                                            
 * Something whose {@link Configuration} can be changed at run time.                                                                                           
 */
public interface Reconfigurable extends Configurable {

  /**                                                                                                                                                          
   * Change the configuration of this object.                                                                                                                  
   *                                                                                                                                                           
   * Change the configuration of this object  to the configuration                                                                                             
   * passed as conf.                                                                                                                                           
   * If it is not possible to change a configuration option,                                                                                                    
   * a {@link ConfigurationChangeException} is thrown                                                                                                          
   * and no changes are made to the current configuration                                                                                                      
   */
  void changeConf(Configuration conf) throws ConfigurationChangeException;

}
{code}

{code}
package org.apache.hadoop.conf;


/**                                                                                                                                                            
 * Exception indicating that configuration property cannot be changed                                                                                          
 * at run time.                                                                                                                                                
 */
public class ConfigurationChangeException extends Exception {

  private String property;
  private String newVal;
  private String oldVal;


  /**                                                                                                                                                          
   * Construct the exception message.                                                                                                                          
   */
  private static String constructMessage(String property,
                                         String newVal, String oldVal) {
    String message = "Could not change property " + property;
    if (oldVal != null) {
      message += " from " + oldVal;
    }
    if (newVal != null) {
      message += " to " + newVal;
    }
    return message;
  }

  /**                                                                                                                                                          
   * Create a new instance of {@link ConfigurationChangeException}.                                                                                            
   */
  public ConfigurationChangeException(String property,
                                      String newVal, String oldVal) {
    super(constructMessage(property, newVal, oldVal));
    this.property = property;
    this.newVal = newVal;
    this.oldVal = oldVal;
  }

  /**                                                                                                                                                          
   * Create a new instance of {@link ConfigurationChangeException}.                                                                                            
   */
  public ConfigurationChangeException(String property,
                                      String newVal, String oldVal,
                                      Throwable cause) {
    super(constructMessage(property, newVal, oldVal), cause);
    this.property = property;
    this.newVal = newVal;
    this.oldVal = oldVal;
  }

  /**                                                                                                                                                          
   * Get property that cannot be changed.                                                                                                                      
   */
  public String getProperty() {
    return property;
  }

  /**                                                                                                                                                          
   * Get value to which property was supposed to be changed.                                                                                                   
   */
  public String getNewValue() {
    return newVal;
  }

  /**                                                                                                                                                          
   * Get old value of property that cannot be changed.                                                                                                         
   */
  public String getOldValue() {
    return oldVal;
  }
{code}

> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7001
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7001
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Patrick Kling
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
>    void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11028-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 15 21:09:04 2010
Return-Path: <common-issues-return-11028-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92321 invoked from network); 15 Oct 2010 21:09:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Oct 2010 21:09:03 -0000
Received: (qmail 66137 invoked by uid 500); 15 Oct 2010 21:09:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66106 invoked by uid 500); 15 Oct 2010 21:09:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66098 invoked by uid 99); 15 Oct 2010 21:09:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 21:09:03 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 21:09:01 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9FL8dVn009908
	for <common-issues@hadoop.apache.org>; Fri, 15 Oct 2010 21:08:39 GMT
Message-ID: <7658735.170141287176919880.JavaMail.jira@thor>
Date: Fri, 15 Oct 2010 17:08:39 -0400 (EDT)
From: "Philip Zeyliger (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7001) Allow configuration changes without
 restarting configured nodes
In-Reply-To: <33220874.146971287081516580.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921553#action_12921553 ] 

Philip Zeyliger commented on HADOOP-7001:
-----------------------------------------

I think that the number of reconfigurable parameters will be quite small, and that they will require extra code.  (e.g., changing the size of a thread pool requires a line or two of code).  So maybe it's just better to expose those things in some more direct way?  At the very least, we should make it quite clear where it's going to be documented what things are re-configurable.

> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7001
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7001
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Patrick Kling
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
>    void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11029-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 15 21:17:41 2010
Return-Path: <common-issues-return-11029-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96868 invoked from network); 15 Oct 2010 21:17:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Oct 2010 21:17:40 -0000
Received: (qmail 83493 invoked by uid 500); 15 Oct 2010 21:17:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83456 invoked by uid 500); 15 Oct 2010 21:17:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83448 invoked by uid 99); 15 Oct 2010 21:17:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 21:17:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 21:17:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9FLHJvF010085
	for <common-issues@hadoop.apache.org>; Fri, 15 Oct 2010 21:17:20 GMT
Message-ID: <15796991.170441287177439792.JavaMail.jira@thor>
Date: Fri, 15 Oct 2010 17:17:19 -0400 (EDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-7001) Allow configuration changes without
 restarting configured nodes
In-Reply-To: <33220874.146971287081516580.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur reassigned HADOOP-7001:
----------------------------------------

    Assignee: Patrick Kling

> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7001
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7001
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Patrick Kling
>            Assignee: Patrick Kling
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
>    void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11030-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 15 21:28:54 2010
Return-Path: <common-issues-return-11030-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2982 invoked from network); 15 Oct 2010 21:28:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Oct 2010 21:28:54 -0000
Received: (qmail 2773 invoked by uid 500); 15 Oct 2010 21:28:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2749 invoked by uid 500); 15 Oct 2010 21:28:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2741 invoked by uid 99); 15 Oct 2010 21:28:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 21:28:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 21:28:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9FLSXax010216
	for <common-issues@hadoop.apache.org>; Fri, 15 Oct 2010 21:28:33 GMT
Message-ID: <2967006.170541287178113545.JavaMail.jira@thor>
Date: Fri, 15 Oct 2010 17:28:33 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6977) Herriot daemon clients should vend
 statistics
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921561#action_12921561 ] 

Konstantin Boudnik commented on HADOOP-6977:
--------------------------------------------

Thanks for the review, Tanping. That's exactly the idea. If you take a look at the latest HDFS-1408 it implements this approach.

> Herriot daemon clients should vend statistics
> ---------------------------------------------
>
>                 Key: HADOOP-6977
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6977
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: FinderTest.java, HADOOP-6977.patch, HADOOP-6977.patch, HADOOP-6977.patch, HADOOP-6977.patch, HADOOP-6977.y20S.patch, HADOOP-6977.y20S.patch
>
>
> The HDFS web user interface serves useful information through dfshealth.jsp and dfsnodelist.jsp.
> The Herriot interface to Hadoop cluster daemons would benefit from the addition of some way to channel metics information.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11031-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 15 22:41:00 2010
Return-Path: <common-issues-return-11031-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31462 invoked from network); 15 Oct 2010 22:41:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Oct 2010 22:41:00 -0000
Received: (qmail 88112 invoked by uid 500); 15 Oct 2010 22:41:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88079 invoked by uid 500); 15 Oct 2010 22:41:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88071 invoked by uid 99); 15 Oct 2010 22:41:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 22:41:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 22:40:57 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9FMeaBe011411
	for <common-issues@hadoop.apache.org>; Fri, 15 Oct 2010 22:40:36 GMT
Message-ID: <19978371.171881287182436037.JavaMail.jira@thor>
Date: Fri, 15 Oct 2010 18:40:36 -0400 (EDT)
From: "Patrick Kling (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7001) Allow configuration changes without
 restarting configured nodes
In-Reply-To: <33220874.146971287081516580.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Patrick Kling updated HADOOP-7001:
----------------------------------

    Attachment: reconfigurable.patch

> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7001
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7001
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Patrick Kling
>            Assignee: Patrick Kling
>         Attachments: reconfigurable.patch
>
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
>    void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11032-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 15 22:50:59 2010
Return-Path: <common-issues-return-11032-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32884 invoked from network); 15 Oct 2010 22:50:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Oct 2010 22:50:59 -0000
Received: (qmail 95615 invoked by uid 500); 15 Oct 2010 22:50:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95594 invoked by uid 500); 15 Oct 2010 22:50:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95586 invoked by uid 99); 15 Oct 2010 22:50:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 22:50:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 22:50:58 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9FMocqI011578
	for <common-issues@hadoop.apache.org>; Fri, 15 Oct 2010 22:50:38 GMT
Message-ID: <18525242.172141287183038547.JavaMail.jira@thor>
Date: Fri, 15 Oct 2010 18:50:38 -0400 (EDT)
From: "Patrick Kling (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7001) Allow configuration changes without
 restarting configured nodes
In-Reply-To: <33220874.146971287081516580.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Patrick Kling updated HADOOP-7001:
----------------------------------

    Attachment: reconfigurable.patch

> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7001
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7001
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Patrick Kling
>            Assignee: Patrick Kling
>         Attachments: reconfigurable.patch
>
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
>    void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11033-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 15 22:51:01 2010
Return-Path: <common-issues-return-11033-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32930 invoked from network); 15 Oct 2010 22:51:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Oct 2010 22:51:00 -0000
Received: (qmail 95804 invoked by uid 500); 15 Oct 2010 22:51:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95773 invoked by uid 500); 15 Oct 2010 22:51:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95765 invoked by uid 99); 15 Oct 2010 22:51:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 22:51:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 22:50:58 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9FMoaKk011539
	for <common-issues@hadoop.apache.org>; Fri, 15 Oct 2010 22:50:36 GMT
Message-ID: <24720232.172011287183036319.JavaMail.jira@thor>
Date: Fri, 15 Oct 2010 18:50:36 -0400 (EDT)
From: "Patrick Kling (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7001) Allow configuration changes without
 restarting configured nodes
In-Reply-To: <33220874.146971287081516580.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Patrick Kling updated HADOOP-7001:
----------------------------------

    Attachment:     (was: reconfigurable.patch)

> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7001
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7001
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Patrick Kling
>            Assignee: Patrick Kling
>         Attachments: reconfigurable.patch
>
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
>    void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11034-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 15 22:58:57 2010
Return-Path: <common-issues-return-11034-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35091 invoked from network); 15 Oct 2010 22:58:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Oct 2010 22:58:56 -0000
Received: (qmail 99233 invoked by uid 500); 15 Oct 2010 22:58:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99160 invoked by uid 500); 15 Oct 2010 22:58:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99152 invoked by uid 99); 15 Oct 2010 22:58:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 22:58:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Oct 2010 22:58:56 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9FMwZDp011710
	for <common-issues@hadoop.apache.org>; Fri, 15 Oct 2010 22:58:35 GMT
Message-ID: <15798614.172321287183515746.JavaMail.jira@thor>
Date: Fri, 15 Oct 2010 18:58:35 -0400 (EDT)
From: "Patrick Kling (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7001) Allow configuration changes without
 restarting configured nodes
In-Reply-To: <33220874.146971287081516580.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921584#action_12921584 ] 

Patrick Kling commented on HADOOP-7001:
---------------------------------------

I have uploaded a patch with the changes proposed in this JIRA. The included base class can be used to easily implement the interface by overriding the method changeProperty, which modifies a single configuration property:

{code}
/**
 * Change a property or throw an exception.
 *
 * Subclasses should overrride this.
 * If newVal is null, change the property to its default value.
 */
protected boolean changeProperty(String property, String newVal) 
  throws ConfigurationChangeException {
  throw new ConfigurationChangeException(property, newVal,
                                         getConf().get(property));
  
}
{code}

This should enable us to include whatever code is necessary to make a configuration change. The overridden version of this method should also serve as a good documentation of what changes are supported by a particular class.

Below is a version of the Reconfigurable interface with more detailed comments. Unfortunately, I cannot edit my previous comments, so I apologize if this is somewhat redundant:

{code}
package org.apache.hadoop.conf;

/**
 * Something whose {@link Configuration} can be changed at run time.
 */
public interface Reconfigurable extends Configurable {

  /**
   * Change the configuration of this object.
   *
   * Change the configuration of this object  to the configuration
   * passed as conf.
   * If it is not possible to change a configuration option,
   * a {@link ConfigurationChangeException} is thrown
   * and no changes are made to the current configuration.
   *
   * Detailed semantics:
   * * If a property is set in the current configuration and in newConf, then 
   *   the configuration must be updated to reflect the value in newConf.
   * * If a property is not set in the current configuration but is set in
   *   newConf, then the configuration must be set to the value in newConf.
   * * If a property is set in the current configuration but is not set in 
   *   newConf, then the configuration must revert to the default value for
   *   this property.
   * * If the change required by these rules is not possible, a
   *   ConfigurationChangeException must be thrown. 
   */
  void changeConf(Configuration newConf) throws ConfigurationChangeException;
}
{code}

> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7001
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7001
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Patrick Kling
>            Assignee: Patrick Kling
>         Attachments: reconfigurable.patch
>
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
>    void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11035-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 18 13:24:49 2010
Return-Path: <common-issues-return-11035-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46661 invoked from network); 18 Oct 2010 13:24:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Oct 2010 13:24:48 -0000
Received: (qmail 42520 invoked by uid 500); 18 Oct 2010 13:24:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42118 invoked by uid 500); 18 Oct 2010 13:24:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42106 invoked by uid 99); 18 Oct 2010 13:24:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Oct 2010 13:24:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Oct 2010 13:24:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9IDOMTH005743
	for <common-issues@hadoop.apache.org>; Mon, 18 Oct 2010 13:24:22 GMT
Message-ID: <33487257.21331287408262876.JavaMail.jira@thor>
Date: Mon, 18 Oct 2010 09:24:22 -0400 (EDT)
From: "Jingguo Yao (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-7002) copy
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

copy
----

                 Key: HADOOP-7002
                 URL: https://issues.apache.org/jira/browse/HADOOP-7002
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Jingguo Yao




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11036-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 18 13:37:47 2010
Return-Path: <common-issues-return-11036-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51544 invoked from network); 18 Oct 2010 13:37:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Oct 2010 13:37:46 -0000
Received: (qmail 68732 invoked by uid 500); 18 Oct 2010 13:37:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68563 invoked by uid 500); 18 Oct 2010 13:37:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68555 invoked by uid 99); 18 Oct 2010 13:37:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Oct 2010 13:37:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Oct 2010 13:37:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9IDbNGG005946
	for <common-issues@hadoop.apache.org>; Mon, 18 Oct 2010 13:37:23 GMT
Message-ID: <25380438.21551287409043514.JavaMail.jira@thor>
Date: Mon, 18 Oct 2010 09:37:23 -0400 (EDT)
From: "Jingguo Yao (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7002) Wrong description of copyFromLocal
 and copyToLocal in documentation
In-Reply-To: <33487257.21331287408262876.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jingguo Yao updated HADOOP-7002:
--------------------------------

           Description: 
The descriptions of copyFromLocal and copyToLocal are wrong. 

For copyFromLocal, the documentation says "Similar to put command, except that the source is restricted to a local file reference." But from the source code of FsShell.java, I can see that copyFromLocal is the sames as put. 

For copyToLocal, the documentation says "Similar to get command, except that the destination is restricted to a local file reference.". But from the source code of FsShell.java, I can see that copyToLocal is the same as get.

And this problem exist for both English and Chinese documentation.
              Priority: Minor  (was: Major)
    Remaining Estimate: 0.33h
     Original Estimate: 0.33h
               Summary: Wrong description of copyFromLocal and copyToLocal in documentation  (was: copy)

> Wrong description of copyFromLocal and copyToLocal in documentation
> -------------------------------------------------------------------
>
>                 Key: HADOOP-7002
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7002
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jingguo Yao
>            Priority: Minor
>   Original Estimate: 0.33h
>  Remaining Estimate: 0.33h
>
> The descriptions of copyFromLocal and copyToLocal are wrong. 
> For copyFromLocal, the documentation says "Similar to put command, except that the source is restricted to a local file reference." But from the source code of FsShell.java, I can see that copyFromLocal is the sames as put. 
> For copyToLocal, the documentation says "Similar to get command, except that the destination is restricted to a local file reference.". But from the source code of FsShell.java, I can see that copyToLocal is the same as get.
> And this problem exist for both English and Chinese documentation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11037-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 18 13:47:50 2010
Return-Path: <common-issues-return-11037-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54028 invoked from network); 18 Oct 2010 13:47:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Oct 2010 13:47:50 -0000
Received: (qmail 85860 invoked by uid 500); 18 Oct 2010 13:47:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85796 invoked by uid 500); 18 Oct 2010 13:47:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85782 invoked by uid 99); 18 Oct 2010 13:47:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Oct 2010 13:47:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Oct 2010 13:47:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9IDlO3o006085
	for <common-issues@hadoop.apache.org>; Mon, 18 Oct 2010 13:47:25 GMT
Message-ID: <6463293.21601287409644910.JavaMail.jira@thor>
Date: Mon, 18 Oct 2010 09:47:24 -0400 (EDT)
From: "Jingguo Yao (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7002) Wrong description of copyFromLocal
 and copyToLocal in documentation
In-Reply-To: <33487257.21331287408262876.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922073#action_12922073 ] 

Jingguo Yao commented on HADOOP-7002:
-------------------------------------

If some committer think that I am right and want to fix it, I am happy to create a patch.

> Wrong description of copyFromLocal and copyToLocal in documentation
> -------------------------------------------------------------------
>
>                 Key: HADOOP-7002
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7002
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jingguo Yao
>            Priority: Minor
>   Original Estimate: 0.33h
>  Remaining Estimate: 0.33h
>
> The descriptions of copyFromLocal and copyToLocal are wrong. 
> For copyFromLocal, the documentation says "Similar to put command, except that the source is restricted to a local file reference." But from the source code of FsShell.java, I can see that copyFromLocal is the sames as put. 
> For copyToLocal, the documentation says "Similar to get command, except that the destination is restricted to a local file reference.". But from the source code of FsShell.java, I can see that copyToLocal is the same as get.
> And this problem exist for both English and Chinese documentation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11038-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 18 18:05:52 2010
Return-Path: <common-issues-return-11038-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15026 invoked from network); 18 Oct 2010 18:05:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Oct 2010 18:05:52 -0000
Received: (qmail 30667 invoked by uid 500); 18 Oct 2010 18:05:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30580 invoked by uid 500); 18 Oct 2010 18:05:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30572 invoked by uid 99); 18 Oct 2010 18:05:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Oct 2010 18:05:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Oct 2010 18:05:49 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9II5Rq7010771
	for <common-issues@hadoop.apache.org>; Mon, 18 Oct 2010 18:05:27 GMT
Message-ID: <4493614.26461287425127395.JavaMail.jira@thor>
Date: Mon, 18 Oct 2010 14:05:27 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6818) Provide a JNI-based implementation
 of GroupMappingServiceProvider
In-Reply-To: <17262845.29701276200793990.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922184#action_12922184 ] 

Todd Lipcon commented on HADOOP-6818:
-------------------------------------

Took a look at the updated patch, looks good from my point of view. +1.

> Provide a JNI-based implementation of GroupMappingServiceProvider
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6818
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6818
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6818-trunk-1.patch, 6818-trunk.patch, hadoop-6818-1.patch, hadoop-6818-2.patch, JNIGroupMapping.patch
>
>
> The default implementation of GroupMappingServiceProvider does a fork of a unix command to get the groups of a user. Since the group resolution happens in the servers, this might be costly. This jira aims at providing a JNI-based implementation for GroupMappingServiceProvider.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11039-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 18 18:32:49 2010
Return-Path: <common-issues-return-11039-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31146 invoked from network); 18 Oct 2010 18:32:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Oct 2010 18:32:49 -0000
Received: (qmail 60665 invoked by uid 500); 18 Oct 2010 18:32:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60617 invoked by uid 500); 18 Oct 2010 18:32:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60609 invoked by uid 99); 18 Oct 2010 18:32:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Oct 2010 18:32:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Oct 2010 18:32:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9IIWPIY011243
	for <common-issues@hadoop.apache.org>; Mon, 18 Oct 2010 18:32:25 GMT
Message-ID: <18157421.27001287426745349.JavaMail.jira@thor>
Date: Mon, 18 Oct 2010 14:32:25 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6917) Remove empty
 FTPFileSystemConfigKeys.java file
In-Reply-To: <5727198.474711282328956153.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922204#action_12922204 ] 

Tom White commented on HADOOP-6917:
-----------------------------------

It's not possible to produce a patch for this since the file is empty. The fix is to run {{svn rm src/java/org/apache/hadoop/fs/ftp/FTPFileSystemConfigKeys.java}} and commit.

> Remove empty FTPFileSystemConfigKeys.java file
> ----------------------------------------------
>
>                 Key: HADOOP-6917
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6917
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Tom White
>
> FTPFileSystemConfigKeys.java is empty and not used so should be deleted.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11040-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 18 18:32:49 2010
Return-Path: <common-issues-return-11040-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31175 invoked from network); 18 Oct 2010 18:32:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Oct 2010 18:32:49 -0000
Received: (qmail 60832 invoked by uid 500); 18 Oct 2010 18:32:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60795 invoked by uid 500); 18 Oct 2010 18:32:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60787 invoked by uid 99); 18 Oct 2010 18:32:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Oct 2010 18:32:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Oct 2010 18:32:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9IIWP5G011249
	for <common-issues@hadoop.apache.org>; Mon, 18 Oct 2010 18:32:25 GMT
Message-ID: <5062509.27021287426745661.JavaMail.jira@thor>
Date: Mon, 18 Oct 2010 14:32:25 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6917) Remove empty
 FTPFileSystemConfigKeys.java file
In-Reply-To: <5727198.474711282328956153.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White reassigned HADOOP-6917:
---------------------------------

    Assignee: Tom White

> Remove empty FTPFileSystemConfigKeys.java file
> ----------------------------------------------
>
>                 Key: HADOOP-6917
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6917
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Tom White
>            Assignee: Tom White
>
> FTPFileSystemConfigKeys.java is empty and not used so should be deleted.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11041-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 18 21:34:51 2010
Return-Path: <common-issues-return-11041-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31357 invoked from network); 18 Oct 2010 21:34:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Oct 2010 21:34:51 -0000
Received: (qmail 40634 invoked by uid 500); 18 Oct 2010 21:34:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40606 invoked by uid 500); 18 Oct 2010 21:34:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40598 invoked by uid 99); 18 Oct 2010 21:34:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Oct 2010 21:34:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Oct 2010 21:34:50 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9ILYTLp014358
	for <common-issues@hadoop.apache.org>; Mon, 18 Oct 2010 21:34:30 GMT
Message-ID: <5367887.30271287437669145.JavaMail.jira@thor>
Date: Mon, 18 Oct 2010 17:34:29 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6954) Sources JARs are not correctly
 published to the Maven repository
In-Reply-To: <19843693.216391284595892946.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922279#action_12922279 ] 

Konstantin Boudnik commented on HADOOP-6954:
--------------------------------------------

+1 patch looks good.

> Sources JARs are not correctly published to the Maven repository
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6954
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6954
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.21.1
>
>         Attachments: HADOOP-6954.patch
>
>
> When you try to "close" the staging repository to make it visible to the public the operation fails (see https://issues.apache.org/jira/browse/HDFS-1292?focusedCommentId=12909953&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12909953 for the type of error).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11042-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 19 00:10:51 2010
Return-Path: <common-issues-return-11042-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86825 invoked from network); 19 Oct 2010 00:10:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Oct 2010 00:10:51 -0000
Received: (qmail 42454 invoked by uid 500); 19 Oct 2010 00:10:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42399 invoked by uid 500); 19 Oct 2010 00:10:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42391 invoked by uid 99); 19 Oct 2010 00:10:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Oct 2010 00:10:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Oct 2010 00:10:50 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9J0ATY0017395
	for <common-issues@hadoop.apache.org>; Tue, 19 Oct 2010 00:10:30 GMT
Message-ID: <2815899.34171287447029909.JavaMail.jira@thor>
Date: Mon, 18 Oct 2010 20:10:29 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6977) Herriot daemon clients should vend
 statistics
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922375#action_12922375 ] 

Konstantin Boudnik commented on HADOOP-6977:
--------------------------------------------

Running test-patch on my laptop.

{noformat}
     [exec] -1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
     [exec] 
     [exec]     -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec] 
     [exec]     +1 system tests framework.  The patch passed system tests framework compile.
{noformat}

Warning seems to be coming from security related code as in last N test-patch runs.

> Herriot daemon clients should vend statistics
> ---------------------------------------------
>
>                 Key: HADOOP-6977
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6977
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: FinderTest.java, HADOOP-6977.patch, HADOOP-6977.patch, HADOOP-6977.patch, HADOOP-6977.patch, HADOOP-6977.y20S.patch, HADOOP-6977.y20S.patch
>
>
> The HDFS web user interface serves useful information through dfshealth.jsp and dfsnodelist.jsp.
> The Herriot interface to Hadoop cluster daemons would benefit from the addition of some way to channel metics information.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11043-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 20 20:54:48 2010
Return-Path: <common-issues-return-11043-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64823 invoked from network); 20 Oct 2010 20:54:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Oct 2010 20:54:48 -0000
Received: (qmail 4602 invoked by uid 500); 20 Oct 2010 20:54:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4562 invoked by uid 500); 20 Oct 2010 20:54:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4444 invoked by uid 99); 20 Oct 2010 20:54:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 20:54:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Oct 2010 20:54:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9KKsQso003309
	for <common-issues@hadoop.apache.org>; Wed, 20 Oct 2010 20:54:26 GMT
Message-ID: <30184462.19001287608066472.JavaMail.jira@thor>
Date: Wed, 20 Oct 2010 16:54:26 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-7003) Fix hadoop patch testing using
 jira_cli tool
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Fix hadoop patch testing using jira_cli tool
--------------------------------------------

                 Key: HADOOP-7003
                 URL: https://issues.apache.org/jira/browse/HADOOP-7003
             Project: Hadoop Common
          Issue Type: New Feature
          Components: build
            Reporter: Giridharan Kesavan




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11044-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 06:40:42 2010
Return-Path: <common-issues-return-11044-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47730 invoked from network); 21 Oct 2010 06:40:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 06:40:42 -0000
Received: (qmail 30880 invoked by uid 500); 21 Oct 2010 06:40:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30826 invoked by uid 500); 21 Oct 2010 06:40:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30818 invoked by uid 99); 21 Oct 2010 06:40:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 06:40:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 06:40:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9L6eHna012863
	for <common-issues@hadoop.apache.org>; Thu, 21 Oct 2010 06:40:17 GMT
Message-ID: <20649000.2011287643217154.JavaMail.jira@thor>
Date: Thu, 21 Oct 2010 02:40:17 -0400 (EDT)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7003) Fix hadoop patch testing using
 jira_cli tool
In-Reply-To: <30184462.19001287608066472.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nigel Daley updated HADOOP-7003:
--------------------------------

    Attachment: hudsonPatchQueueAdmin.sh

Attaching a replacement for nightly/hudsonPatchQueueAdmin.sh.  This is not yet a patch file.  Want to make this easier to review.  The script is written for linux (tweaks are needed for MacOS) and requires xpath cmd line tool to be installed (sudo apt-get install libxml-xpath-perl).

> Fix hadoop patch testing using jira_cli tool
> --------------------------------------------
>
>                 Key: HADOOP-7003
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7003
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>            Reporter: Giridharan Kesavan
>         Attachments: hudsonPatchQueueAdmin.sh
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11045-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 06:42:44 2010
Return-Path: <common-issues-return-11045-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48173 invoked from network); 21 Oct 2010 06:42:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 06:42:44 -0000
Received: (qmail 33026 invoked by uid 500); 21 Oct 2010 06:42:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32907 invoked by uid 500); 21 Oct 2010 06:42:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32895 invoked by uid 99); 21 Oct 2010 06:42:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 06:42:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 06:42:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9L6gHeZ012903
	for <common-issues@hadoop.apache.org>; Thu, 21 Oct 2010 06:42:17 GMT
Message-ID: <9190855.2091287643336977.JavaMail.jira@thor>
Date: Thu, 21 Oct 2010 02:42:16 -0400 (EDT)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7003) Fix hadoop patch testing using
 jira_cli tool
In-Reply-To: <30184462.19001287608066472.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923345#action_12923345 ] 

Nigel Daley commented on HADOOP-7003:
-------------------------------------

The current patch testing process has a number of flaws:
# reliance on email to determine when a patch is available
# a job per *slave* that you want to run the testing on
# because of 2, poor sharing of resources across many projects
# an admin job that tracks the queue and running jobs but can get confused if jobs fail or hudson is rebooted

Hudson has come a long way since I first set it up 4 years ago to run our patch testing.  A recent Hudson feature allows the *same* job to be executed at the *same* time.  A more established feature is the ability to pass parameters between jobs.  These feature now enable a simpler architecture for patch testing.

Here are the new Hudson jobs needed:

*PreCommit-Admin*
This job replaces the current patch testing admin job and runs every 5 to 10 minutes.  It runs a new version of the hudsonPatchQueueAdmin.sh script (attached).  The new script roughly follows these steps:
# grabs the XML output of a Jira filter that contains all the issues that should be tested by the system.  For instance, this filter contains all the HADOOP, HDFS, and MAPREDUCE issues that are in Patch Available state: https://issues.apache.org/jira/sr/jira.issueviews:searchrequest-xml/12313474/SearchRequest-12313474.xml?tempMax=100  By changing the filter (eg. adding PIG and ZOOKEEPER for instance), this hudson job will run tests for those project issues as well.  
# processes the XML into a list of <project-issue>, <attachment id> pairs, where attachment id is the id for the newest attachment
# grabs the one build artifact that this job keeps: a list of pairs: <project-issue>, <attachment id>.  This file is called patch_tested.txt and contains the pairs already submitted for testing.
# for each pair from the XML, see if the pair is in patch_tested.txt.  If not, kick a new project build (next job below) and append the pair to the file.  When kicking off the new build, pass the issue number (and optional the attachment id) to the build.  The build to kick off is computed as "PreCommit-<PROJECT NAME>-Build".  This same project build can be kicked off many times by this execution of the admin job -- each instance sitting in the Hudson queue until a slave is available.
# when this admin job completes, archive the latest patch_tested.txt

*PreCommit-<PROJECT>-Build*
A job with this name (eg PreCommit-MAPREDUCE-Build) must exist for each Jira project administered by PreCommit-Admin.  The job must enable concurrent builds in it's configuration.  This job downloads the patch, does the actual testing and puts a comment in Jira.  It does not communicate back to the admin job.  The job is tied to slaves that are configured with a certain label.  The label is the same as the project name.

*Slaves*
Slaves are configured to run a certain project job (like PreCommit-HDFS-Build) by being labeled with the project (eg HDFS).  This means that any properly configured slave can be easily added to the build pool and start taking on load.  It also means that most slaves should be able to run most patch tests for most projects.


> Fix hadoop patch testing using jira_cli tool
> --------------------------------------------
>
>                 Key: HADOOP-7003
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7003
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>            Reporter: Giridharan Kesavan
>         Attachments: hudsonPatchQueueAdmin.sh
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11046-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 08:14:46 2010
Return-Path: <common-issues-return-11046-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74697 invoked from network); 21 Oct 2010 08:14:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 08:14:45 -0000
Received: (qmail 28608 invoked by uid 500); 21 Oct 2010 08:14:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28470 invoked by uid 500); 21 Oct 2010 08:14:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28462 invoked by uid 99); 21 Oct 2010 08:14:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 08:14:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 08:14:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9L8EJKX014397
	for <common-issues@hadoop.apache.org>; Thu, 21 Oct 2010 08:14:19 GMT
Message-ID: <5419931.3001287648859175.JavaMail.jira@thor>
Date: Thu, 21 Oct 2010 04:14:19 -0400 (EDT)
From: "Henning Blohm (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-7004) Problem with
 org.apache.hadoop.conf.Configuration.REGISTRY
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Problem with org.apache.hadoop.conf.Configuration.REGISTRY
----------------------------------------------------------

                 Key: HADOOP-7004
                 URL: https://issues.apache.org/jira/browse/HADOOP-7004
             Project: Hadoop Common
          Issue Type: Bug
         Environment: hadoop 0.20.2, hbase 0.20.6
            Reporter: Henning Blohm
            Priority: Minor


When reusing Configuration that has an added addResource(InputStream) a
reload of configuration will fail as the stream has been read before.

The reload gets triggered when addDefaultResource is called. That method
uses the REGISTRY static WeakHashMap to reach out to all reachable Configuration 
instances to reset their properties.

The method addDefaultResource is called by e.g. ConfigUtil in org.apache.hadoop.mapreduce.util (hadoop trunk) or 
JobConf (hadoop 0.20.2).

The problem has been observed in Hadoop 0.20.2 but the code in trunk has
essentially the same structure. 

There are a few problems here:

1. You cannot safely use addResource(InputStream), if Configuration
objects are to be re-used (you can however use addResource(URL) instead)

2. Modifying the state of Configuration instances at some later point in
time as a side effect of some class initialization in some completely
unrelated thread leads to unpredictable behavior (properties change under the hood)

3. Configuration instances keep context classloaders to find resources.  After redeployment these may not be "valid" anymore. 
As long as the Configuration instance has not been collected, addDefaultResource will still invoke reloadConfiguration on them. 
While that is harmless today (only resetting members), this looks like a ticking time bomb.

Suggestion:
Define all default resources in Configuration once. Do not hold on to
other configuration instances and do not modify their state as a side
effect of some other activity.

See also: http://osdir.com/ml/general/2010-10/msg25893.html



-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11047-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 19:26:40 2010
Return-Path: <common-issues-return-11047-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41194 invoked from network); 21 Oct 2010 19:26:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 19:26:39 -0000
Received: (qmail 48134 invoked by uid 500); 21 Oct 2010 19:26:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48089 invoked by uid 500); 21 Oct 2010 19:26:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48081 invoked by uid 99); 21 Oct 2010 19:26:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 19:26:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 19:26:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9LJQIEN020982
	for <common-issues@hadoop.apache.org>; Thu, 21 Oct 2010 19:26:18 GMT
Message-ID: <3209187.13611287689178137.JavaMail.jira@thor>
Date: Thu, 21 Oct 2010 15:26:18 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6971) Clover build doesn't generate
 per-test coverage
In-Reply-To: <24019340.389891285355492616.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923584#action_12923584 ] 

Hudson commented on HADOOP-6971:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #395 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/395/])
    

> Clover build doesn't generate per-test coverage
> -----------------------------------------------
>
>                 Key: HADOOP-6971
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6971
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build, test
>    Affects Versions: 0.20.3, 0.21.1, 0.22.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>             Fix For: 0.21.1
>
>         Attachments: HADOOP-6971.patch, HADOOP-6971.y20S.patch
>
>
> Because of the way the structure of Hadoop's builds is done Clover can't properly detect test classes among the sources. As the result clover reports are incomplete and do not provide viral per-test coverage info.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11048-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 19:26:44 2010
Return-Path: <common-issues-return-11048-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41276 invoked from network); 21 Oct 2010 19:26:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 19:26:43 -0000
Received: (qmail 48999 invoked by uid 500); 21 Oct 2010 19:26:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48964 invoked by uid 500); 21 Oct 2010 19:26:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48917 invoked by uid 99); 21 Oct 2010 19:26:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 19:26:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 19:26:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9LJQHbo020976
	for <common-issues@hadoop.apache.org>; Thu, 21 Oct 2010 19:26:17 GMT
Message-ID: <214277.13591287689177706.JavaMail.jira@thor>
Date: Thu, 21 Oct 2010 15:26:17 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6984) NPE from
 SequenceFile::Writer.CompressionCodecOption
In-Reply-To: <10251121.516661286092293049.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923583#action_12923583 ] 

Hudson commented on HADOOP-6984:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #395 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/395/])
    

> NPE from SequenceFile::Writer.CompressionCodecOption
> ----------------------------------------------------
>
>                 Key: HADOOP-6984
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6984
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Chris Douglas
>            Assignee: Chris Douglas
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: 6984-0.patch, 6984-1.patch, 6984-2.patch, 6984-3.patch
>
>
> The deprecated HADOOP-6856 constructors can create a compressed writers with a null-wrapped {{CompressionCodecOption}}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11049-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 19:26:44 2010
Return-Path: <common-issues-return-11049-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41316 invoked from network); 21 Oct 2010 19:26:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 19:26:43 -0000
Received: (qmail 49437 invoked by uid 500); 21 Oct 2010 19:26:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49407 invoked by uid 500); 21 Oct 2010 19:26:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49376 invoked by uid 99); 21 Oct 2010 19:26:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 19:26:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 19:26:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9LJQLjR021028
	for <common-issues@hadoop.apache.org>; Thu, 21 Oct 2010 19:26:21 GMT
Message-ID: <19171615.13761287689181609.JavaMail.jira@thor>
Date: Thu, 21 Oct 2010 15:26:21 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6993) Broken link on cluster setup page
 of docs
In-Reply-To: <22210278.19631286425112055.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923588#action_12923588 ] 

Hudson commented on HADOOP-6993:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #395 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/395/])
    

> Broken link on cluster setup page of docs
> -----------------------------------------
>
>                 Key: HADOOP-6993
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6993
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.21.0
>            Reporter: Aaron T. Myers
>            Assignee: Eli Collins
>             Fix For: 0.21.1, 0.22.0
>
>         Attachments: hadoop-6993-21-1.patch, hadoop-6993-22-1.patch
>
>
> The link on http://hadoop.apache.org/common/docs/current/cluster_setup.html#Configuring+the+Hadoop+Daemons to core-default.xml is presently:
> {quote}
> http://hadoop.apache.org/common/docs/current/common-default.html
> {quote}
> but it should be:
> {quote}
> http://hadoop.apache.org/common/docs/current/core-default.html
> {quote}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11050-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 19:26:45 2010
Return-Path: <common-issues-return-11050-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41338 invoked from network); 21 Oct 2010 19:26:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 19:26:44 -0000
Received: (qmail 49981 invoked by uid 500); 21 Oct 2010 19:26:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49947 invoked by uid 500); 21 Oct 2010 19:26:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49915 invoked by uid 99); 21 Oct 2010 19:26:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 19:26:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 19:26:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9LJQM63021043
	for <common-issues@hadoop.apache.org>; Thu, 21 Oct 2010 19:26:22 GMT
Message-ID: <17694183.13811287689182363.JavaMail.jira@thor>
Date: Thu, 21 Oct 2010 15:26:22 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6933) TestListFiles is flaky
In-Reply-To: <14615079.42351282946633507.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923589#action_12923589 ] 

Hudson commented on HADOOP-6933:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #395 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/395/])
    

> TestListFiles is flaky
> ----------------------
>
>                 Key: HADOOP-6933
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6933
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6933.txt
>
>
> TestListFiles assumes a particular order of the files returned by the directory iterator. There's no such guarantee made by the underlying API, so the test fails on some hosts.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11051-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 19:26:45 2010
Return-Path: <common-issues-return-11051-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41358 invoked from network); 21 Oct 2010 19:26:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 19:26:44 -0000
Received: (qmail 50341 invoked by uid 500); 21 Oct 2010 19:26:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50308 invoked by uid 500); 21 Oct 2010 19:26:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50272 invoked by uid 99); 21 Oct 2010 19:26:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 19:26:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 19:26:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9LJQJRA020995
	for <common-issues@hadoop.apache.org>; Thu, 21 Oct 2010 19:26:19 GMT
Message-ID: <24484275.13651287689179806.JavaMail.jira@thor>
Date: Thu, 21 Oct 2010 15:26:19 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6944) [Herriot] Implement a functionality
 for getting proxy users definitions like groups and hosts.
In-Reply-To: <13358164.74161283948073125.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923585#action_12923585 ] 

Hudson commented on HADOOP-6944:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #395 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/395/])
    

> [Herriot] Implement a functionality for getting proxy users definitions like groups and hosts.
> ----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6944
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6944
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>    Affects Versions: 0.21.1
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>             Fix For: 0.21.1
>
>         Attachments: HADOOP-6944.patch, HADOOP-6944.patch, HADOOP-6944.patch, HADOOP-6944.patch, HADOOP-6944.patch
>
>
> Gridmix should require a proxy user's file for impersonating various jobs. So, implement couple of methods for getting the proxy users list and a proxy users file (it's a combination of proxy users and groups) based on cluster configuration.
> The proxy users list should require for map reduce jobs and proxy users file should require for gridmix jobs.
> The following are methods signature,
> public ProxyUserDefinitions getHadoopProxyUsers() - get the list of proxy users list based on cluster configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11052-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 19:26:46 2010
Return-Path: <common-issues-return-11052-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41397 invoked from network); 21 Oct 2010 19:26:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 19:26:44 -0000
Received: (qmail 50968 invoked by uid 500); 21 Oct 2010 19:26:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50936 invoked by uid 500); 21 Oct 2010 19:26:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50908 invoked by uid 99); 21 Oct 2010 19:26:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 19:26:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 19:26:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9LJQK61021007
	for <common-issues@hadoop.apache.org>; Thu, 21 Oct 2010 19:26:20 GMT
Message-ID: <9677656.13691287689180598.JavaMail.jira@thor>
Date: Thu, 21 Oct 2010 15:26:20 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6879) Provide SSH based (Jsch) remote
 execution API for system tests
In-Reply-To: <11760779.565921279956469755.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6879?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D129=
23586#action_12923586 ]=20

Hudson commented on HADOOP-6879:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #395 (See [https://hudson.apache.o=
rg/hudson/job/Hadoop-Common-trunk-Commit/395/])
   =20

> Provide SSH based (Jsch) remote execution API for system tests
> --------------------------------------------------------------
>
>                 Key: HADOOP-6879
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6879
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>    Affects Versions: 0.22.0
>            Reporter: Iyappan Srinivasan
>            Assignee: Konstantin Boudnik
>             Fix For: 0.22.0
>
>         Attachments: 6879-ydist-security-patch.txt, HADOOP-6879.patch, HA=
DOOP-6879.patch, HADOOP-6879.patch, HADOOP-6879.patch, HADOOP-6879.patch, H=
ADOOP-6879.patch, HADOOP-6879.patch, HADOOP-6879.patch, HADOOP-6879.patch, =
HADOOP-6879.patch, HADOOP-6879.patch, HADOOP-6879.y20.patch, HADOOP-6879.y2=
0.patch, HADOOP-6879.y20.patch, HADOOP-6879.y20.patch, HADOOP-6879.y20.patc=
h, HADOOP-6879.y20.patch
>
>
> http://mvnrepository.com/
> com.jcraft =C2=BB jsch=20
> 0.1.42 version needs to be included in the build. This is  needed to faci=
litate implementation of some system (Herriot) testcases .
> Please include this in ivy.
> jsch is originally located in http://www.jcraft.com/jsch/

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11053-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 19:26:46 2010
Return-Path: <common-issues-return-11053-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41457 invoked from network); 21 Oct 2010 19:26:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 19:26:45 -0000
Received: (qmail 51181 invoked by uid 500); 21 Oct 2010 19:26:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51131 invoked by uid 500); 21 Oct 2010 19:26:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51117 invoked by uid 99); 21 Oct 2010 19:26:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 19:26:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 19:26:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9LJQOeW021064
	for <common-issues@hadoop.apache.org>; Thu, 21 Oct 2010 19:26:24 GMT
Message-ID: <26559547.13881287689184114.JavaMail.jira@thor>
Date: Thu, 21 Oct 2010 15:26:24 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6856) SequenceFile and MapFile need
 cleanup to remove redundant constructors
In-Reply-To: <19355698.285171278691549769.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923590#action_12923590 ] 

Hudson commented on HADOOP-6856:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #395 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/395/])
    

> SequenceFile and MapFile need cleanup to remove redundant constructors
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6856
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6856
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.22.0
>
>         Attachments: h-6856.patch, h-6856.patch
>
>
> Currently there are 2 public SequenceFile.Reader constructors, 3 public SequenceFile.Writer constructors, 9 public SequenceFile.createWriter, 2 public MapFile.Reader constructors, and 8 public MapFile.Writer constructors. All of with various historical combinations of parameters that don't cover the entire space.
> All of this makes it *very* difficult to add new optional parameters to SequenceFile and MapFile. 
> I'd like change to the style of FileContext.create with option parameters. I'll implement one public SequenceFile.Reader constructor and one public SequenceFile.createWriter and implement all of the current variants based on those two. I'll do the same for MapFile.Reader and MapFile.Writer including passing parameters down to the underlying SequenceFile.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11054-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 19:26:46 2010
Return-Path: <common-issues-return-11054-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41465 invoked from network); 21 Oct 2010 19:26:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 19:26:45 -0000
Received: (qmail 51198 invoked by uid 500); 21 Oct 2010 19:26:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51135 invoked by uid 500); 21 Oct 2010 19:26:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51121 invoked by uid 99); 21 Oct 2010 19:26:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 19:26:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 19:26:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9LJQLtW021019
	for <common-issues@hadoop.apache.org>; Thu, 21 Oct 2010 19:26:21 GMT
Message-ID: <28778464.13731287689181186.JavaMail.jira@thor>
Date: Thu, 21 Oct 2010 15:26:21 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6969) CHANGES.txt does not reflect the
 release of version 0.21.0.
In-Reply-To: <12538682.369991285269513067.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923587#action_12923587 ] 

Hudson commented on HADOOP-6969:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #395 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/395/])
    

> CHANGES.txt does not reflect the release of version 0.21.0.
> -----------------------------------------------------------
>
>                 Key: HADOOP-6969
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6969
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>         Environment: CHANGES.txt should show the release date for 0.21.0 and include section for for 0.21.1 - Unreleased. Latest changes, that did not make into 0.21.0, should be moved under 0.21.1 section.
>            Reporter: Konstantin Shvachko
>            Assignee: Tom White
>             Fix For: 0.21.1
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11055-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 19:26:48 2010
Return-Path: <common-issues-return-11055-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41686 invoked from network); 21 Oct 2010 19:26:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 19:26:48 -0000
Received: (qmail 51543 invoked by uid 500); 21 Oct 2010 19:26:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51514 invoked by uid 500); 21 Oct 2010 19:26:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51506 invoked by uid 99); 21 Oct 2010 19:26:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 19:26:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 19:26:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9LJQOfl021073
	for <common-issues@hadoop.apache.org>; Thu, 21 Oct 2010 19:26:24 GMT
Message-ID: <32767006.13911287689184511.JavaMail.jira@thor>
Date: Thu, 21 Oct 2010 15:26:24 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6996) Allow CodecFactory to return a
 codec object given a codec' class name
In-Reply-To: <1541599.80951286821892815.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923591#action_12923591 ] 

Hudson commented on HADOOP-6996:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #395 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/395/])
    

> Allow CodecFactory to return a codec object given a codec' class name
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6996
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6996
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: getCodecByClassName.patch
>
>
> CodecFactory specify the list of codec that are supported by Hadoop. However, it returns a codec only by a file's name. I would like to make getCodec method to alternatively take a codec's class name.
> This is required by  HDFS-1435, where
> 1. it allows an HDFS admin to configure which codec to use to save an image. 
> 2. It stores the codec class name in its on-disk image instead of a file's suffix.
> When saving and reading an image, I'd like to get an codec from CodecFactory by its class name. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11056-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 19:27:19 2010
Return-Path: <common-issues-return-11056-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41915 invoked from network); 21 Oct 2010 19:27:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 19:27:18 -0000
Received: (qmail 52737 invoked by uid 500); 21 Oct 2010 19:27:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52703 invoked by uid 500); 21 Oct 2010 19:27:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52695 invoked by uid 99); 21 Oct 2010 19:27:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 19:27:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 19:27:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9LJQsG3021109
	for <common-issues@hadoop.apache.org>; Thu, 21 Oct 2010 19:26:54 GMT
Message-ID: <13861962.14011287689214713.JavaMail.jira@thor>
Date: Thu, 21 Oct 2010 15:26:54 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6989) TestSetFile is failing on trunk
In-Reply-To: <4408858.540201286236594170.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923593#action_12923593 ] 

Hudson commented on HADOOP-6989:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #395 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/395/])
    

> TestSetFile is failing on trunk
> -------------------------------
>
>                 Key: HADOOP-6989
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6989
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Jakob Homan
>            Assignee: Chris Douglas
>             Fix For: 0.22.0
>
>         Attachments: 6989-0.patch
>
>
> Testsuite: org.apache.hadoop.io.TestSetFile
> Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 1.015 sec
> ------------- Standard Output ---------------
> 2010-10-04 16:32:01,030 INFO  io.TestSetFile (TestSetFile.java:generate(56)) - generating 10000 records in memory
> 2010-10-04 16:32:01,249 INFO  io.TestSetFile (TestSetFile.java:generate(63)) - sorting 10000 records
> 2010-10-04 16:32:01,350 INFO  io.TestSetFile (TestSetFile.java:writeTest(72)) - creating with 10000 records
> ------------- ---------------- ---------------
> Testcase: testSetFile took 0.964 sec
> 	Caused an ERROR
> key class or comparator option must be set
> java.lang.IllegalArgumentException: key class or comparator option must be set
> 	at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:247)
> 	at org.apache.hadoop.io.SetFile$Writer.<init>(SetFile.java:60)
> 	at org.apache.hadoop.io.TestSetFile.writeTest(TestSetFile.java:73)
> 	at org.apache.hadoop.io.TestSetFile.testSetFile(TestSetFile.java:45)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11057-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 19:27:19 2010
Return-Path: <common-issues-return-11057-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41941 invoked from network); 21 Oct 2010 19:27:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 19:27:19 -0000
Received: (qmail 52918 invoked by uid 500); 21 Oct 2010 19:27:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52881 invoked by uid 500); 21 Oct 2010 19:27:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52873 invoked by uid 99); 21 Oct 2010 19:27:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 19:27:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 19:27:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9LJQtou021121
	for <common-issues@hadoop.apache.org>; Thu, 21 Oct 2010 19:26:55 GMT
Message-ID: <25512333.14051287689215291.JavaMail.jira@thor>
Date: Thu, 21 Oct 2010 15:26:55 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6987) Use JUnit Rule to optionally fail
 test cases that run more than 10 seconds
In-Reply-To: <26687627.538921286230953013.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923594#action_12923594 ] 

Hudson commented on HADOOP-6987:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #395 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/395/])
    

> Use JUnit Rule to optionally fail test cases that run more than 10 seconds
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6987
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6987
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6897.patch, HADOOP-6987-2.patch
>
>
> Using JUnit Rules annotations we can fail tests cases that take longer than 10 seconds (for instance) to run.  This provides a regression check against test cases taking longer than they had previously due to unintended code changes, as well as provides a membership criteria for unit tests versus integration tests in HDFS and MR.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11058-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 21:02:42 2010
Return-Path: <common-issues-return-11058-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91775 invoked from network); 21 Oct 2010 21:02:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 21:02:42 -0000
Received: (qmail 49686 invoked by uid 500); 21 Oct 2010 21:02:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49656 invoked by uid 500); 21 Oct 2010 21:02:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49648 invoked by uid 99); 21 Oct 2010 21:02:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 21:02:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 21:02:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9LL2Iq4022975
	for <common-issues@hadoop.apache.org>; Thu, 21 Oct 2010 21:02:18 GMT
Message-ID: <33188746.16591287694938378.JavaMail.jira@thor>
Date: Thu, 21 Oct 2010 17:02:18 -0400 (EDT)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7003) Fix hadoop patch testing using
 jira_cli tool
In-Reply-To: <30184462.19001287608066472.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nigel Daley updated HADOOP-7003:
--------------------------------

    Attachment: hudsonPatchQueueAdmin.sh

Here's a new version with a few tweaks after talking with Giri.  I'll go ahead and commit this.  Giri agreed to setup new Hudson jobs based on this.  Once it's up and running, we can delete the old jobs.

> Fix hadoop patch testing using jira_cli tool
> --------------------------------------------
>
>                 Key: HADOOP-7003
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7003
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>            Reporter: Giridharan Kesavan
>         Attachments: hudsonPatchQueueAdmin.sh, hudsonPatchQueueAdmin.sh
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11059-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 22:19:43 2010
Return-Path: <common-issues-return-11059-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48696 invoked from network); 21 Oct 2010 22:19:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 22:19:43 -0000
Received: (qmail 68773 invoked by uid 500); 21 Oct 2010 22:19:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68744 invoked by uid 500); 21 Oct 2010 22:19:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68736 invoked by uid 99); 21 Oct 2010 22:19:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 22:19:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 22:19:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9LMJJmF024353
	for <common-issues@hadoop.apache.org>; Thu, 21 Oct 2010 22:19:19 GMT
Message-ID: <29655880.18051287699559943.JavaMail.jira@thor>
Date: Thu, 21 Oct 2010 18:19:19 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6832) Provide a web server plugin that uses
 a static user for the web UI
In-Reply-To: <19001973.112581277141003401.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6832:
----------------------------------

    Attachment: h-6382.patch

Here's the trivial update.

> Provide a web server plugin that uses a static user for the web UI
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6832
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6832
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.22.0
>
>         Attachments: h-6382.patch, static-web-user.patch
>
>
> We need a simple plugin that uses a static user for clusters with security that don't want to authenticate users on the web UI.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11060-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 21 22:19:44 2010
Return-Path: <common-issues-return-11060-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48738 invoked from network); 21 Oct 2010 22:19:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Oct 2010 22:19:44 -0000
Received: (qmail 68962 invoked by uid 500); 21 Oct 2010 22:19:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68922 invoked by uid 500); 21 Oct 2010 22:19:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68914 invoked by uid 99); 21 Oct 2010 22:19:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 22:19:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Oct 2010 22:19:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9LMJKLA024371
	for <common-issues@hadoop.apache.org>; Thu, 21 Oct 2010 22:19:20 GMT
Message-ID: <16145244.18111287699560719.JavaMail.jira@thor>
Date: Thu, 21 Oct 2010 18:19:20 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6832) Provide a web server plugin that uses
 a static user for the web UI
In-Reply-To: <19001973.112581277141003401.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6832:
----------------------------------

    Status: Patch Available  (was: Open)

> Provide a web server plugin that uses a static user for the web UI
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6832
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6832
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.22.0
>
>         Attachments: h-6382.patch, static-web-user.patch
>
>
> We need a simple plugin that uses a static user for clusters with security that don't want to authenticate users on the web UI.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11061-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 00:59:00 2010
Return-Path: <common-issues-return-11061-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15733 invoked from network); 22 Oct 2010 00:59:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 00:59:00 -0000
Received: (qmail 9242 invoked by uid 500); 22 Oct 2010 00:59:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9202 invoked by uid 500); 22 Oct 2010 00:59:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9194 invoked by uid 99); 22 Oct 2010 00:59:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 00:59:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 00:59:00 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9M0wdHa026660
	for <common-issues@hadoop.apache.org>; Fri, 22 Oct 2010 00:58:39 GMT
Message-ID: <27712658.20141287709119551.JavaMail.jira@thor>
Date: Thu, 21 Oct 2010 20:58:39 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6947) Kerberos relogin should set
 refreshKrb5Config to true
In-Reply-To: <15312872.90501284007413131.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923722#action_12923722 ] 

Todd Lipcon commented on HADOOP-6947:
-------------------------------------

Kan, mind committing this? It got +1 from you and Hudson.

> Kerberos relogin should set refreshKrb5Config to true
> -----------------------------------------------------
>
>                 Key: HADOOP-6947
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6947
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-6947-branch20.txt, hadoop-6947.txt
>
>
> In working on securing a daemon that uses two different principals from different threads, I found that I wasn't able to login from a second keytab after I'd logged in from the first. This is because we don't set the refreshKrb5Config in the Configuration for the Krb5LoginModule - hence it won't switch over to the correct keytab file if it's different than the first.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11062-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 06:20:46 2010
Return-Path: <common-issues-return-11062-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3416 invoked from network); 22 Oct 2010 06:20:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 06:20:45 -0000
Received: (qmail 234 invoked by uid 500); 22 Oct 2010 06:20:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 130 invoked by uid 500); 22 Oct 2010 06:20:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 120 invoked by uid 99); 22 Oct 2010 06:20:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 06:20:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 06:20:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9M6KJF9001061
	for <common-issues@hadoop.apache.org>; Fri, 22 Oct 2010 06:20:19 GMT
Message-ID: <22078635.22271287728419072.JavaMail.jira@thor>
Date: Fri, 22 Oct 2010 02:20:19 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7003) Fix hadoop patch testing using
 jira_cli tool
In-Reply-To: <30184462.19001287608066472.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923771#action_12923771 ] 

Giridharan Kesavan commented on HADOOP-7003:
--------------------------------------------

PreCommit-Admin and the PreCommit-<PROJECT>-Build  jobs are ready.

> Fix hadoop patch testing using jira_cli tool
> --------------------------------------------
>
>                 Key: HADOOP-7003
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7003
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>            Reporter: Giridharan Kesavan
>         Attachments: hudsonPatchQueueAdmin.sh, hudsonPatchQueueAdmin.sh
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11063-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 06:53:41 2010
Return-Path: <common-issues-return-11063-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15297 invoked from network); 22 Oct 2010 06:53:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 06:53:41 -0000
Received: (qmail 22046 invoked by uid 500); 22 Oct 2010 06:53:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21857 invoked by uid 500); 22 Oct 2010 06:53:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21843 invoked by uid 99); 22 Oct 2010 06:53:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 06:53:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 06:53:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9M6rGNt001473
	for <common-issues@hadoop.apache.org>; Fri, 22 Oct 2010 06:53:16 GMT
Message-ID: <10205943.22621287730396161.JavaMail.jira@thor>
Date: Fri, 22 Oct 2010 02:53:16 -0400 (EDT)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-7003) Fix hadoop patch testing using
 jira_cli tool
In-Reply-To: <30184462.19001287608066472.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nigel Daley resolved HADOOP-7003.
---------------------------------

       Resolution: Fixed
    Fix Version/s: hudson
         Assignee: Nigel Daley

I just committed this.

> Fix hadoop patch testing using jira_cli tool
> --------------------------------------------
>
>                 Key: HADOOP-7003
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7003
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>            Reporter: Giridharan Kesavan
>            Assignee: Nigel Daley
>             Fix For: hudson
>
>         Attachments: hudsonPatchQueueAdmin.sh, hudsonPatchQueueAdmin.sh
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11064-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 07:12:40 2010
Return-Path: <common-issues-return-11064-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22874 invoked from network); 22 Oct 2010 07:12:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 07:12:40 -0000
Received: (qmail 41121 invoked by uid 500); 22 Oct 2010 07:12:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41044 invoked by uid 500); 22 Oct 2010 07:12:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41034 invoked by uid 99); 22 Oct 2010 07:12:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 07:12:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 07:12:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9M7CG9i001887
	for <common-issues@hadoop.apache.org>; Fri, 22 Oct 2010 07:12:16 GMT
Message-ID: <22110062.22991287731536485.JavaMail.jira@thor>
Date: Fri, 22 Oct 2010 03:12:16 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7003) Fix hadoop patch testing using
 jira_cli tool
In-Reply-To: <30184462.19001287608066472.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923786#action_12923786 ] 

Hudson commented on HADOOP-7003:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #396 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/396/])
    HADOOP-7003: New script for managing queue of pre-commit patches that need testing.  Contributed by nigel.


> Fix hadoop patch testing using jira_cli tool
> --------------------------------------------
>
>                 Key: HADOOP-7003
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7003
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>            Reporter: Giridharan Kesavan
>            Assignee: Nigel Daley
>             Fix For: hudson
>
>         Attachments: hudsonPatchQueueAdmin.sh, hudsonPatchQueueAdmin.sh
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11065-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 08:12:46 2010
Return-Path: <common-issues-return-11065-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39535 invoked from network); 22 Oct 2010 08:12:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 08:12:46 -0000
Received: (qmail 81257 invoked by uid 500); 22 Oct 2010 08:12:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81155 invoked by uid 500); 22 Oct 2010 08:12:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81147 invoked by uid 99); 22 Oct 2010 08:12:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 08:12:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 08:12:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9M8CICo002800
	for <common-issues@hadoop.apache.org>; Fri, 22 Oct 2010 08:12:19 GMT
Message-ID: <12940800.23461287735138947.JavaMail.jira@thor>
Date: Fri, 22 Oct 2010 04:12:18 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7003) Fix hadoop patch testing using
 jira_cli tool
In-Reply-To: <30184462.19001287608066472.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923798#action_12923798 ] 

Hudson commented on HADOOP-7003:
--------------------------------

Integrated in Hadoop-Hdfs-trunk-Commit #413 (See [https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/413/])
    HADOOP-7003: New script for managing queue of pre-commit patches that need testing.  Contributed by nigel.


> Fix hadoop patch testing using jira_cli tool
> --------------------------------------------
>
>                 Key: HADOOP-7003
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7003
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>            Reporter: Giridharan Kesavan
>            Assignee: Nigel Daley
>             Fix For: hudson
>
>         Attachments: hudsonPatchQueueAdmin.sh, hudsonPatchQueueAdmin.sh
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11066-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 10:52:44 2010
Return-Path: <common-issues-return-11066-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86649 invoked from network); 22 Oct 2010 10:52:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 10:52:43 -0000
Received: (qmail 39634 invoked by uid 500); 22 Oct 2010 10:52:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39471 invoked by uid 500); 22 Oct 2010 10:52:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39373 invoked by uid 99); 22 Oct 2010 10:52:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 10:52:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 10:52:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9MAqJes004941
	for <common-issues@hadoop.apache.org>; Fri, 22 Oct 2010 10:52:19 GMT
Message-ID: <25889744.24441287744739441.JavaMail.jira@thor>
Date: Fri, 22 Oct 2010 06:52:19 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7003) Fix hadoop patch testing using
 jira_cli tool
In-Reply-To: <30184462.19001287608066472.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923815#action_12923815 ] 

Hudson commented on HADOOP-7003:
--------------------------------

Integrated in ZooKeeper-trunk #976 (See [https://hudson.apache.org/hudson/job/ZooKeeper-trunk/976/])
    HADOOP-7003: New script for managing queue of pre-commit patches that need testing.  Contributed by nigel.


> Fix hadoop patch testing using jira_cli tool
> --------------------------------------------
>
>                 Key: HADOOP-7003
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7003
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>            Reporter: Giridharan Kesavan
>            Assignee: Nigel Daley
>             Fix For: hudson
>
>         Attachments: hudsonPatchQueueAdmin.sh, hudsonPatchQueueAdmin.sh
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11067-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 12:56:48 2010
Return-Path: <common-issues-return-11067-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47335 invoked from network); 22 Oct 2010 12:56:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 12:56:48 -0000
Received: (qmail 87610 invoked by uid 500); 22 Oct 2010 12:56:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86875 invoked by uid 500); 22 Oct 2010 12:56:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86306 invoked by uid 99); 22 Oct 2010 12:56:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 12:56:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 12:56:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9MCuOCt006545
	for <common-issues@hadoop.apache.org>; Fri, 22 Oct 2010 12:56:24 GMT
Message-ID: <11172354.25141287752184130.JavaMail.jira@thor>
Date: Fri, 22 Oct 2010 08:56:24 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6984) NPE from
 SequenceFile::Writer.CompressionCodecOption
In-Reply-To: <10251121.516661286092293049.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923831#action_12923831 ] 

Hudson commented on HADOOP-6984:
--------------------------------

Integrated in Hadoop-Common-trunk #489 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/489/])
    

> NPE from SequenceFile::Writer.CompressionCodecOption
> ----------------------------------------------------
>
>                 Key: HADOOP-6984
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6984
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Chris Douglas
>            Assignee: Chris Douglas
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: 6984-0.patch, 6984-1.patch, 6984-2.patch, 6984-3.patch
>
>
> The deprecated HADOOP-6856 constructors can create a compressed writers with a null-wrapped {{CompressionCodecOption}}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11069-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 12:56:49 2010
Return-Path: <common-issues-return-11069-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47388 invoked from network); 22 Oct 2010 12:56:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 12:56:48 -0000
Received: (qmail 87926 invoked by uid 500); 22 Oct 2010 12:56:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87698 invoked by uid 500); 22 Oct 2010 12:56:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87129 invoked by uid 99); 22 Oct 2010 12:56:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 12:56:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 12:56:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9MCuPgr006566
	for <common-issues@hadoop.apache.org>; Fri, 22 Oct 2010 12:56:25 GMT
Message-ID: <31328021.25201287752185546.JavaMail.jira@thor>
Date: Fri, 22 Oct 2010 08:56:25 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6944) [Herriot] Implement a functionality
 for getting proxy users definitions like groups and hosts.
In-Reply-To: <13358164.74161283948073125.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923833#action_12923833 ] 

Hudson commented on HADOOP-6944:
--------------------------------

Integrated in Hadoop-Common-trunk #489 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/489/])
    

> [Herriot] Implement a functionality for getting proxy users definitions like groups and hosts.
> ----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6944
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6944
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>    Affects Versions: 0.21.1
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>             Fix For: 0.21.1
>
>         Attachments: HADOOP-6944.patch, HADOOP-6944.patch, HADOOP-6944.patch, HADOOP-6944.patch, HADOOP-6944.patch
>
>
> Gridmix should require a proxy user's file for impersonating various jobs. So, implement couple of methods for getting the proxy users list and a proxy users file (it's a combination of proxy users and groups) based on cluster configuration.
> The proxy users list should require for map reduce jobs and proxy users file should require for gridmix jobs.
> The following are methods signature,
> public ProxyUserDefinitions getHadoopProxyUsers() - get the list of proxy users list based on cluster configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11068-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 12:56:49 2010
Return-Path: <common-issues-return-11068-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47369 invoked from network); 22 Oct 2010 12:56:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 12:56:48 -0000
Received: (qmail 87736 invoked by uid 500); 22 Oct 2010 12:56:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87495 invoked by uid 500); 22 Oct 2010 12:56:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86312 invoked by uid 99); 22 Oct 2010 12:56:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 12:56:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 12:56:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9MCuOaE006551
	for <common-issues@hadoop.apache.org>; Fri, 22 Oct 2010 12:56:24 GMT
Message-ID: <22241544.25161287752184505.JavaMail.jira@thor>
Date: Fri, 22 Oct 2010 08:56:24 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6971) Clover build doesn't generate
 per-test coverage
In-Reply-To: <24019340.389891285355492616.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923832#action_12923832 ] 

Hudson commented on HADOOP-6971:
--------------------------------

Integrated in Hadoop-Common-trunk #489 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/489/])
    

> Clover build doesn't generate per-test coverage
> -----------------------------------------------
>
>                 Key: HADOOP-6971
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6971
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build, test
>    Affects Versions: 0.20.3, 0.21.1, 0.22.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>             Fix For: 0.21.1
>
>         Attachments: HADOOP-6971.patch, HADOOP-6971.y20S.patch
>
>
> Because of the way the structure of Hadoop's builds is done Clover can't properly detect test classes among the sources. As the result clover reports are incomplete and do not provide viral per-test coverage info.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11072-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 12:56:50 2010
Return-Path: <common-issues-return-11072-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47470 invoked from network); 22 Oct 2010 12:56:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 12:56:49 -0000
Received: (qmail 88445 invoked by uid 500); 22 Oct 2010 12:56:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88396 invoked by uid 500); 22 Oct 2010 12:56:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88356 invoked by uid 99); 22 Oct 2010 12:56:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 12:56:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 12:56:48 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9MCuSqE006629
	for <common-issues@hadoop.apache.org>; Fri, 22 Oct 2010 12:56:28 GMT
Message-ID: <12303142.25401287752188643.JavaMail.jira@thor>
Date: Fri, 22 Oct 2010 08:56:28 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6989) TestSetFile is failing on trunk
In-Reply-To: <4408858.540201286236594170.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923838#action_12923838 ] 

Hudson commented on HADOOP-6989:
--------------------------------

Integrated in Hadoop-Common-trunk #489 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/489/])
    

> TestSetFile is failing on trunk
> -------------------------------
>
>                 Key: HADOOP-6989
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6989
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Jakob Homan
>            Assignee: Chris Douglas
>             Fix For: 0.22.0
>
>         Attachments: 6989-0.patch
>
>
> Testsuite: org.apache.hadoop.io.TestSetFile
> Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 1.015 sec
> ------------- Standard Output ---------------
> 2010-10-04 16:32:01,030 INFO  io.TestSetFile (TestSetFile.java:generate(56)) - generating 10000 records in memory
> 2010-10-04 16:32:01,249 INFO  io.TestSetFile (TestSetFile.java:generate(63)) - sorting 10000 records
> 2010-10-04 16:32:01,350 INFO  io.TestSetFile (TestSetFile.java:writeTest(72)) - creating with 10000 records
> ------------- ---------------- ---------------
> Testcase: testSetFile took 0.964 sec
> 	Caused an ERROR
> key class or comparator option must be set
> java.lang.IllegalArgumentException: key class or comparator option must be set
> 	at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:247)
> 	at org.apache.hadoop.io.SetFile$Writer.<init>(SetFile.java:60)
> 	at org.apache.hadoop.io.TestSetFile.writeTest(TestSetFile.java:73)
> 	at org.apache.hadoop.io.TestSetFile.testSetFile(TestSetFile.java:45)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11070-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 12:56:50 2010
Return-Path: <common-issues-return-11070-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47438 invoked from network); 22 Oct 2010 12:56:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 12:56:49 -0000
Received: (qmail 88027 invoked by uid 500); 22 Oct 2010 12:56:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87883 invoked by uid 500); 22 Oct 2010 12:56:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87466 invoked by uid 99); 22 Oct 2010 12:56:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 12:56:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 12:56:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9MCuQFD006589
	for <common-issues@hadoop.apache.org>; Fri, 22 Oct 2010 12:56:26 GMT
Message-ID: <23584626.25271287752186765.JavaMail.jira@thor>
Date: Fri, 22 Oct 2010 08:56:26 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6969) CHANGES.txt does not reflect the
 release of version 0.21.0.
In-Reply-To: <12538682.369991285269513067.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923835#action_12923835 ] 

Hudson commented on HADOOP-6969:
--------------------------------

Integrated in Hadoop-Common-trunk #489 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/489/])
    

> CHANGES.txt does not reflect the release of version 0.21.0.
> -----------------------------------------------------------
>
>                 Key: HADOOP-6969
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6969
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>         Environment: CHANGES.txt should show the release date for 0.21.0 and include section for for 0.21.1 - Unreleased. Latest changes, that did not make into 0.21.0, should be moved under 0.21.1 section.
>            Reporter: Konstantin Shvachko
>            Assignee: Tom White
>             Fix For: 0.21.1
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11071-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 12:56:51 2010
Return-Path: <common-issues-return-11071-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47458 invoked from network); 22 Oct 2010 12:56:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 12:56:49 -0000
Received: (qmail 88179 invoked by uid 500); 22 Oct 2010 12:56:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87958 invoked by uid 500); 22 Oct 2010 12:56:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87485 invoked by uid 99); 22 Oct 2010 12:56:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 12:56:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 12:56:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9MCuRlZ006605
	for <common-issues@hadoop.apache.org>; Fri, 22 Oct 2010 12:56:27 GMT
Message-ID: <3965433.25321287752187456.JavaMail.jira@thor>
Date: Fri, 22 Oct 2010 08:56:27 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6933) TestListFiles is flaky
In-Reply-To: <14615079.42351282946633507.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923836#action_12923836 ] 

Hudson commented on HADOOP-6933:
--------------------------------

Integrated in Hadoop-Common-trunk #489 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/489/])
    

> TestListFiles is flaky
> ----------------------
>
>                 Key: HADOOP-6933
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6933
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6933.txt
>
>
> TestListFiles assumes a particular order of the files returned by the directory iterator. There's no such guarantee made by the underlying API, so the test fails on some hosts.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11073-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 12:56:52 2010
Return-Path: <common-issues-return-11073-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47537 invoked from network); 22 Oct 2010 12:56:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 12:56:50 -0000
Received: (qmail 88646 invoked by uid 500); 22 Oct 2010 12:56:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88603 invoked by uid 500); 22 Oct 2010 12:56:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88577 invoked by uid 99); 22 Oct 2010 12:56:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 12:56:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 12:56:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9MCuQAj006577
	for <common-issues@hadoop.apache.org>; Fri, 22 Oct 2010 12:56:26 GMT
Message-ID: <33185788.25231287752186105.JavaMail.jira@thor>
Date: Fri, 22 Oct 2010 08:56:26 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6993) Broken link on cluster setup page
 of docs
In-Reply-To: <22210278.19631286425112055.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923834#action_12923834 ] 

Hudson commented on HADOOP-6993:
--------------------------------

Integrated in Hadoop-Common-trunk #489 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/489/])
    

> Broken link on cluster setup page of docs
> -----------------------------------------
>
>                 Key: HADOOP-6993
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6993
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.21.0
>            Reporter: Aaron T. Myers
>            Assignee: Eli Collins
>             Fix For: 0.21.1, 0.22.0
>
>         Attachments: hadoop-6993-21-1.patch, hadoop-6993-22-1.patch
>
>
> The link on http://hadoop.apache.org/common/docs/current/cluster_setup.html#Configuring+the+Hadoop+Daemons to core-default.xml is presently:
> {quote}
> http://hadoop.apache.org/common/docs/current/common-default.html
> {quote}
> but it should be:
> {quote}
> http://hadoop.apache.org/common/docs/current/core-default.html
> {quote}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11074-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 12:56:52 2010
Return-Path: <common-issues-return-11074-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47553 invoked from network); 22 Oct 2010 12:56:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 12:56:50 -0000
Received: (qmail 88808 invoked by uid 500); 22 Oct 2010 12:56:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88617 invoked by uid 500); 22 Oct 2010 12:56:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88608 invoked by uid 99); 22 Oct 2010 12:56:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 12:56:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 12:56:49 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9MCuTlZ006641
	for <common-issues@hadoop.apache.org>; Fri, 22 Oct 2010 12:56:29 GMT
Message-ID: <18475000.25441287752189254.JavaMail.jira@thor>
Date: Fri, 22 Oct 2010 08:56:29 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6987) Use JUnit Rule to optionally fail
 test cases that run more than 10 seconds
In-Reply-To: <26687627.538921286230953013.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923839#action_12923839 ] 

Hudson commented on HADOOP-6987:
--------------------------------

Integrated in Hadoop-Common-trunk #489 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/489/])
    

> Use JUnit Rule to optionally fail test cases that run more than 10 seconds
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6987
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6987
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6897.patch, HADOOP-6987-2.patch
>
>
> Using JUnit Rules annotations we can fail tests cases that take longer than 10 seconds (for instance) to run.  This provides a regression check against test cases taking longer than they had previously due to unintended code changes, as well as provides a membership criteria for unit tests versus integration tests in HDFS and MR.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11075-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 12:56:53 2010
Return-Path: <common-issues-return-11075-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47732 invoked from network); 22 Oct 2010 12:56:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 12:56:52 -0000
Received: (qmail 89025 invoked by uid 500); 22 Oct 2010 12:56:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88983 invoked by uid 500); 22 Oct 2010 12:56:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88961 invoked by uid 99); 22 Oct 2010 12:56:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 12:56:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 12:56:50 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9MCuUIg006663
	for <common-issues@hadoop.apache.org>; Fri, 22 Oct 2010 12:56:30 GMT
Message-ID: <2616826.25511287752190265.JavaMail.jira@thor>
Date: Fri, 22 Oct 2010 08:56:30 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6951) Distinct minicluster services (e.g.
 NN and JT) overwrite each other's service policies
In-Reply-To: <3990308.158511284404253537.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923840#action_12923840 ] 

Hudson commented on HADOOP-6951:
--------------------------------

Integrated in Hadoop-Common-trunk #489 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/489/])
    

> Distinct minicluster services (e.g. NN and JT) overwrite each other's service policies
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6951
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6951
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6951.1.txt, hadoop-6951.2.txt, hadoop-6951.txt.0
>
>
> Because the protocol -> ACL mapping in ServiceAuthorizationManager is static, services which are run in the same JVM have the potential to clobber the other's service authorization ACLs whenever ServiceAuthorizationManager.refresh() is called. This causes authorization failures if one tries to launch a 2NN connected to a minicluster with hadoop.security.authorization enabled. Seems like each service should have its own instance of a ServiceAuthorizationManager, instead of using static methods.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11076-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 12:56:53 2010
Return-Path: <common-issues-return-11076-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47753 invoked from network); 22 Oct 2010 12:56:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 12:56:52 -0000
Received: (qmail 89201 invoked by uid 500); 22 Oct 2010 12:56:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89065 invoked by uid 500); 22 Oct 2010 12:56:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88969 invoked by uid 99); 22 Oct 2010 12:56:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 12:56:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 12:56:49 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9MCuRIW006614
	for <common-issues@hadoop.apache.org>; Fri, 22 Oct 2010 12:56:27 GMT
Message-ID: <30095452.25351287752187851.JavaMail.jira@thor>
Date: Fri, 22 Oct 2010 08:56:27 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6996) Allow CodecFactory to return a
 codec object given a codec' class name
In-Reply-To: <1541599.80951286821892815.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923837#action_12923837 ] 

Hudson commented on HADOOP-6996:
--------------------------------

Integrated in Hadoop-Common-trunk #489 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/489/])
    

> Allow CodecFactory to return a codec object given a codec' class name
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6996
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6996
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: getCodecByClassName.patch
>
>
> CodecFactory specify the list of codec that are supported by Hadoop. However, it returns a codec only by a file's name. I would like to make getCodec method to alternatively take a codec's class name.
> This is required by  HDFS-1435, where
> 1. it allows an HDFS admin to configure which codec to use to save an image. 
> 2. It stores the codec class name in its on-disk image instead of a file's suffix.
> When saving and reading an image, I'd like to get an codec from CodecFactory by its class name. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11077-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 12:56:53 2010
Return-Path: <common-issues-return-11077-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47786 invoked from network); 22 Oct 2010 12:56:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 12:56:52 -0000
Received: (qmail 89375 invoked by uid 500); 22 Oct 2010 12:56:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89326 invoked by uid 500); 22 Oct 2010 12:56:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89199 invoked by uid 99); 22 Oct 2010 12:56:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 12:56:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 12:56:51 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9MCuU13006675
	for <common-issues@hadoop.apache.org>; Fri, 22 Oct 2010 12:56:30 GMT
Message-ID: <23038314.25551287752190875.JavaMail.jira@thor>
Date: Fri, 22 Oct 2010 08:56:30 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6879) Provide SSH based (Jsch) remote
 execution API for system tests
In-Reply-To: <11760779.565921279956469755.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6879?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D129=
23841#action_12923841 ]=20

Hudson commented on HADOOP-6879:
--------------------------------

Integrated in Hadoop-Common-trunk #489 (See [https://hudson.apache.org/huds=
on/job/Hadoop-Common-trunk/489/])
   =20

> Provide SSH based (Jsch) remote execution API for system tests
> --------------------------------------------------------------
>
>                 Key: HADOOP-6879
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6879
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>    Affects Versions: 0.22.0
>            Reporter: Iyappan Srinivasan
>            Assignee: Konstantin Boudnik
>             Fix For: 0.22.0
>
>         Attachments: 6879-ydist-security-patch.txt, HADOOP-6879.patch, HA=
DOOP-6879.patch, HADOOP-6879.patch, HADOOP-6879.patch, HADOOP-6879.patch, H=
ADOOP-6879.patch, HADOOP-6879.patch, HADOOP-6879.patch, HADOOP-6879.patch, =
HADOOP-6879.patch, HADOOP-6879.patch, HADOOP-6879.y20.patch, HADOOP-6879.y2=
0.patch, HADOOP-6879.y20.patch, HADOOP-6879.y20.patch, HADOOP-6879.y20.patc=
h, HADOOP-6879.y20.patch
>
>
> http://mvnrepository.com/
> com.jcraft =C2=BB jsch=20
> 0.1.42 version needs to be included in the build. This is  needed to faci=
litate implementation of some system (Herriot) testcases .
> Please include this in ivy.
> jsch is originally located in http://www.jcraft.com/jsch/

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11078-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 12:56:54 2010
Return-Path: <common-issues-return-11078-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47899 invoked from network); 22 Oct 2010 12:56:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 12:56:54 -0000
Received: (qmail 89561 invoked by uid 500); 22 Oct 2010 12:56:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89525 invoked by uid 500); 22 Oct 2010 12:56:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89516 invoked by uid 99); 22 Oct 2010 12:56:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 12:56:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 12:56:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9MCuXxU006702
	for <common-issues@hadoop.apache.org>; Fri, 22 Oct 2010 12:56:33 GMT
Message-ID: <11116455.25641287752193419.JavaMail.jira@thor>
Date: Fri, 22 Oct 2010 08:56:33 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7003) Fix hadoop patch testing using
 jira_cli tool
In-Reply-To: <30184462.19001287608066472.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923842#action_12923842 ] 

Hudson commented on HADOOP-7003:
--------------------------------

Integrated in Hadoop-Common-trunk #489 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/489/])
    HADOOP-7003: New script for managing queue of pre-commit patches that need testing.  Contributed by nigel.


> Fix hadoop patch testing using jira_cli tool
> --------------------------------------------
>
>                 Key: HADOOP-7003
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7003
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>            Reporter: Giridharan Kesavan
>            Assignee: Nigel Daley
>             Fix For: hudson
>
>         Attachments: hudsonPatchQueueAdmin.sh, hudsonPatchQueueAdmin.sh
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11079-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 12:56:56 2010
Return-Path: <common-issues-return-11079-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47974 invoked from network); 22 Oct 2010 12:56:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 12:56:56 -0000
Received: (qmail 89754 invoked by uid 500); 22 Oct 2010 12:56:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89726 invoked by uid 500); 22 Oct 2010 12:56:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89718 invoked by uid 99); 22 Oct 2010 12:56:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 12:56:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 12:56:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9MCuZGU006723
	for <common-issues@hadoop.apache.org>; Fri, 22 Oct 2010 12:56:35 GMT
Message-ID: <25306448.25711287752195070.JavaMail.jira@thor>
Date: Fri, 22 Oct 2010 08:56:35 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6856) SequenceFile and MapFile need
 cleanup to remove redundant constructors
In-Reply-To: <19355698.285171278691549769.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923843#action_12923843 ] 

Hudson commented on HADOOP-6856:
--------------------------------

Integrated in Hadoop-Common-trunk #489 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/489/])
    

> SequenceFile and MapFile need cleanup to remove redundant constructors
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6856
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6856
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.22.0
>
>         Attachments: h-6856.patch, h-6856.patch
>
>
> Currently there are 2 public SequenceFile.Reader constructors, 3 public SequenceFile.Writer constructors, 9 public SequenceFile.createWriter, 2 public MapFile.Reader constructors, and 8 public MapFile.Writer constructors. All of with various historical combinations of parameters that don't cover the entire space.
> All of this makes it *very* difficult to add new optional parameters to SequenceFile and MapFile. 
> I'd like change to the style of FileContext.create with option parameters. I'll implement one public SequenceFile.Reader constructor and one public SequenceFile.createWriter and implement all of the current variants based on those two. I'll do the same for MapFile.Reader and MapFile.Writer including passing parameters down to the underlying SequenceFile.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11080-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 18:34:33 2010
Return-Path: <common-issues-return-11080-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85620 invoked from network); 22 Oct 2010 18:34:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 18:34:33 -0000
Received: (qmail 95903 invoked by uid 500); 22 Oct 2010 18:34:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95867 invoked by uid 500); 22 Oct 2010 18:34:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95859 invoked by uid 99); 22 Oct 2010 18:34:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 18:34:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 18:34:30 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9MIY8P8014247
	for <common-issues@hadoop.apache.org>; Fri, 22 Oct 2010 18:34:08 GMT
Message-ID: <17316828.31121287772448177.JavaMail.jira@thor>
Date: Fri, 22 Oct 2010 14:34:08 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6317) Serialize the 'final'ness of
 Configuration keys
In-Reply-To: <985698151.1255715791730.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-6317:
---------------------------------

    Status: Open  (was: Patch Available)

Found this in the patch queue.  The patch no longer applies cleanly.  I'll try to repair it.

> Serialize the 'final'ness of Configuration keys
> -----------------------------------------------
>
>                 Key: HADOOP-6317
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6317
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Arun C Murthy
>            Assignee: Aaron Kimball
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6317.2.patch, HADOOP-6317.3.patch, HADOOP-6317.4.patch, HADOOP-6317.patch
>
>
> Currently Configuration.writeXml doesn't serialize the 'final' attribute. There are some scenarios where it is useful:
> # TaskTracker should mark the tasks' 'localized' parameters as 'final' 
> # GUI tools

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11081-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 18:42:28 2010
Return-Path: <common-issues-return-11081-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87730 invoked from network); 22 Oct 2010 18:42:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 18:42:28 -0000
Received: (qmail 6673 invoked by uid 500); 22 Oct 2010 18:42:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6635 invoked by uid 500); 22 Oct 2010 18:42:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6627 invoked by uid 99); 22 Oct 2010 18:42:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 18:42:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 18:42:27 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9MIg7jP014402
	for <common-issues@hadoop.apache.org>; Fri, 22 Oct 2010 18:42:07 GMT
Message-ID: <28550512.31341287772927514.JavaMail.jira@thor>
Date: Fri, 22 Oct 2010 14:42:07 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6317) Serialize the 'final'ness of
 Configuration keys
In-Reply-To: <985698151.1255715791730.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-6317:
---------------------------------

    Attachment: HADOOP-6317.patch

Fixed patch to apply to current trunk.

> Serialize the 'final'ness of Configuration keys
> -----------------------------------------------
>
>                 Key: HADOOP-6317
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6317
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Arun C Murthy
>            Assignee: Aaron Kimball
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6317.2.patch, HADOOP-6317.3.patch, HADOOP-6317.4.patch, HADOOP-6317.patch, HADOOP-6317.patch
>
>
> Currently Configuration.writeXml doesn't serialize the 'final' attribute. There are some scenarios where it is useful:
> # TaskTracker should mark the tasks' 'localized' parameters as 'final' 
> # GUI tools

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11082-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 19:12:38 2010
Return-Path: <common-issues-return-11082-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98180 invoked from network); 22 Oct 2010 19:12:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 19:12:37 -0000
Received: (qmail 39223 invoked by uid 500); 22 Oct 2010 19:12:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39184 invoked by uid 500); 22 Oct 2010 19:12:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39176 invoked by uid 99); 22 Oct 2010 19:12:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 19:12:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 19:12:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9MJCGq4014920
	for <common-issues@hadoop.apache.org>; Fri, 22 Oct 2010 19:12:17 GMT
Message-ID: <424787.31921287774736794.JavaMail.jira@thor>
Date: Fri, 22 Oct 2010 15:12:16 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6996) Allow CodecFactory to return a
 codec object given a codec' class name
In-Reply-To: <1541599.80951286821892815.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12923979#action_12923979 ] 

Hadoop QA commented on HADOOP-6996:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12456881/getCodecByClassName.patch
  against trunk revision 1022697.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/PreCommit-HADOOP-Build/2/console

This message is automatically generated.

> Allow CodecFactory to return a codec object given a codec' class name
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6996
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6996
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: getCodecByClassName.patch
>
>
> CodecFactory specify the list of codec that are supported by Hadoop. However, it returns a codec only by a file's name. I would like to make getCodec method to alternatively take a codec's class name.
> This is required by  HDFS-1435, where
> 1. it allows an HDFS admin to configure which codec to use to save an image. 
> 2. It stores the codec class name in its on-disk image instead of a file's suffix.
> When saving and reading an image, I'd like to get an codec from CodecFactory by its class name. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11083-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 19:20:39 2010
Return-Path: <common-issues-return-11083-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99859 invoked from network); 22 Oct 2010 19:20:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 19:20:39 -0000
Received: (qmail 48667 invoked by uid 500); 22 Oct 2010 19:20:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48623 invoked by uid 500); 22 Oct 2010 19:20:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48615 invoked by uid 99); 22 Oct 2010 19:20:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 19:20:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 19:20:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9MJKISe015065
	for <common-issues@hadoop.apache.org>; Fri, 22 Oct 2010 19:20:18 GMT
Message-ID: <28225139.32151287775218036.JavaMail.jira@thor>
Date: Fri, 22 Oct 2010 15:20:18 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6317) Serialize the 'final'ness of
 Configuration keys
In-Reply-To: <985698151.1255715791730.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-6317:
---------------------------------

    Hadoop Flags: [Reviewed]
          Status: Patch Available  (was: Open)

{code}
-1 overall.  

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 7 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 system tests framework.  The patch passed system tests framework compile.
{code}

I don't get the javadoc failure, since there are the same 6 javadoc warnings with or without this patch.

Arun, do you have concerns with this patch?  If not, I'll commit it.

> Serialize the 'final'ness of Configuration keys
> -----------------------------------------------
>
>                 Key: HADOOP-6317
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6317
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Arun C Murthy
>            Assignee: Aaron Kimball
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6317.2.patch, HADOOP-6317.3.patch, HADOOP-6317.4.patch, HADOOP-6317.patch, HADOOP-6317.patch
>
>
> Currently Configuration.writeXml doesn't serialize the 'final' attribute. There are some scenarios where it is useful:
> # TaskTracker should mark the tasks' 'localized' parameters as 'final' 
> # GUI tools

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11084-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 22 20:42:46 2010
Return-Path: <common-issues-return-11084-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26317 invoked from network); 22 Oct 2010 20:42:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Oct 2010 20:42:46 -0000
Received: (qmail 50560 invoked by uid 500); 22 Oct 2010 20:42:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50537 invoked by uid 500); 22 Oct 2010 20:42:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50529 invoked by uid 99); 22 Oct 2010 20:42:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 20:42:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Oct 2010 20:42:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9MKgMZM016299
	for <common-issues@hadoop.apache.org>; Fri, 22 Oct 2010 20:42:22 GMT
Message-ID: <2462530.33481287780142022.JavaMail.jira@thor>
Date: Fri, 22 Oct 2010 16:42:22 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6317) Serialize the 'final'ness of
 Configuration keys
In-Reply-To: <985698151.1255715791730.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924014#action_12924014 ] 

Hadoop QA commented on HADOOP-6317:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12457858/HADOOP-6317.patch
  against trunk revision 1022697.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 7 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system tests framework.  The patch passed system tests framework compile.

Test results: http://hudson.zones.apache.org/hudson/job/PreCommit-HADOOP-Build/4/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/PreCommit-HADOOP-Build/4/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/PreCommit-HADOOP-Build/4/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/PreCommit-HADOOP-Build/4/console

This message is automatically generated.

> Serialize the 'final'ness of Configuration keys
> -----------------------------------------------
>
>                 Key: HADOOP-6317
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6317
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Arun C Murthy
>            Assignee: Aaron Kimball
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6317.2.patch, HADOOP-6317.3.patch, HADOOP-6317.4.patch, HADOOP-6317.patch, HADOOP-6317.patch
>
>
> Currently Configuration.writeXml doesn't serialize the 'final' attribute. There are some scenarios where it is useful:
> # TaskTracker should mark the tasks' 'localized' parameters as 'final' 
> # GUI tools

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11085-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 06:33:49 2010
Return-Path: <common-issues-return-11085-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93402 invoked from network); 25 Oct 2010 06:33:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 06:33:48 -0000
Received: (qmail 91579 invoked by uid 500); 25 Oct 2010 06:33:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90585 invoked by uid 500); 25 Oct 2010 06:33:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90558 invoked by uid 99); 25 Oct 2010 06:33:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 06:33:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 06:33:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9P6XJrd009945
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 06:33:20 GMT
Message-ID: <17555034.54181287988399911.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 02:33:19 -0400 (EDT)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-7005) Update test-patch.sh to remove
 callback to Hudson master
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Update test-patch.sh to remove callback to Hudson master
--------------------------------------------------------

                 Key: HADOOP-7005
                 URL: https://issues.apache.org/jira/browse/HADOOP-7005
             Project: Hadoop Common
          Issue Type: Improvement
          Components: test
            Reporter: Nigel Daley
             Fix For: hudson




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11086-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 06:41:46 2010
Return-Path: <common-issues-return-11086-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94848 invoked from network); 25 Oct 2010 06:41:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 06:41:45 -0000
Received: (qmail 97537 invoked by uid 500); 25 Oct 2010 06:41:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96723 invoked by uid 500); 25 Oct 2010 06:41:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96715 invoked by uid 99); 25 Oct 2010 06:41:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 06:41:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 06:41:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9P6fMK2010086
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 06:41:22 GMT
Message-ID: <3335187.54381287988882433.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 02:41:22 -0400 (EDT)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7005) Update test-patch.sh to remove
 callback to Hudson master
In-Reply-To: <17555034.54181287988399911.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nigel Daley updated HADOOP-7005:
--------------------------------

    Attachment: HADOOP-7005.patch

This removes the callback and uses Hudson's BUILD_URL env variable instead of hardcoded url.  Once we get this right, it will need to be copied to MR and HDFS too or we need to go back to using svn externals to pulls this script in from nightly/test-patch dir.

> Update test-patch.sh to remove callback to Hudson master
> --------------------------------------------------------
>
>                 Key: HADOOP-7005
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7005
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>            Reporter: Nigel Daley
>             Fix For: hudson
>
>         Attachments: HADOOP-7005.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11087-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 06:41:47 2010
Return-Path: <common-issues-return-11087-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94864 invoked from network); 25 Oct 2010 06:41:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 06:41:46 -0000
Received: (qmail 97689 invoked by uid 500); 25 Oct 2010 06:41:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97476 invoked by uid 500); 25 Oct 2010 06:41:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97388 invoked by uid 99); 25 Oct 2010 06:41:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 06:41:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 06:41:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9P6fL0r010077
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 06:41:21 GMT
Message-ID: <21950654.54351287988881239.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 02:41:21 -0400 (EDT)
From: "Chris Nauroth (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-7006) hadoop fs -getmerge does not work
 using codebase from trunk.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

hadoop fs -getmerge does not work using codebase from trunk.
------------------------------------------------------------

                 Key: HADOOP-7006
                 URL: https://issues.apache.org/jira/browse/HADOOP-7006
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
    Affects Versions: 0.22.0
            Reporter: Chris Nauroth
             Fix For: 0.22.0


Running the codebase from trunk, the hadoop fs -getmerge command does not work.  As implemented in prior versions (i.e. 0.20.2), I could run hadoop fs -getmerge pointed at a directory containing multiple files.  It would merge all files into a single file on the local file system.  Running the same command using the codebase from trunk, it looks like nothing happens.


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11088-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 06:49:47 2010
Return-Path: <common-issues-return-11088-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96776 invoked from network); 25 Oct 2010 06:49:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 06:49:46 -0000
Received: (qmail 10220 invoked by uid 500); 25 Oct 2010 06:49:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10099 invoked by uid 500); 25 Oct 2010 06:49:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10090 invoked by uid 99); 25 Oct 2010 06:49:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 06:49:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 06:49:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9P6nJb3010205
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 06:49:20 GMT
Message-ID: <27689874.54541287989359909.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 02:49:19 -0400 (EDT)
From: "Chris Nauroth (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7006) hadoop fs -getmerge does not work
 using codebase from trunk.
In-Reply-To: <21950654.54351287988881239.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924465#action_12924465 ] 

Chris Nauroth commented on HADOOP-7006:
---------------------------------------

It appears that this was introduced with changeset 949658.  Here is a partial diff of org.apache.hadoop.fs.FileUtil:

{noformat}
@@ -258,7 +258,7 @@
                                   Configuration conf, String addString) throws IOException {
     dstFile = checkDest(srcDir.getName(), dstFS, dstFile, false);
 
-    if (!srcFS.getFileStatus(srcDir).isDir())
+    if (srcFS.getFileStatus(srcDir).isDirectory())
       return false;
{noformat}

Notice that in addition to switching from isDir() to isDirectory(), this change also dropped the negation on the front of the condition.  I'll attach a simple one-line patch to restore functionality.  I've also added a unit test to cover the FileUtil.copyMerge API.  I'd like to volunteer for HADOOP-6387 next, and I want this unit test in place as a regression test while I work on that.


> hadoop fs -getmerge does not work using codebase from trunk.
> ------------------------------------------------------------
>
>                 Key: HADOOP-7006
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7006
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Chris Nauroth
>             Fix For: 0.22.0
>
>
> Running the codebase from trunk, the hadoop fs -getmerge command does not work.  As implemented in prior versions (i.e. 0.20.2), I could run hadoop fs -getmerge pointed at a directory containing multiple files.  It would merge all files into a single file on the local file system.  Running the same command using the codebase from trunk, it looks like nothing happens.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11089-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 06:53:44 2010
Return-Path: <common-issues-return-11089-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97434 invoked from network); 25 Oct 2010 06:53:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 06:53:44 -0000
Received: (qmail 12974 invoked by uid 500); 25 Oct 2010 06:53:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12810 invoked by uid 500); 25 Oct 2010 06:53:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12802 invoked by uid 99); 25 Oct 2010 06:53:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 06:53:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 06:53:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9P6rK5q010277
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 06:53:20 GMT
Message-ID: <19342497.54561287989600153.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 02:53:20 -0400 (EDT)
From: "Chris Nauroth (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7006) hadoop fs -getmerge does not work
 using codebase from trunk.
In-Reply-To: <21950654.54351287988881239.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Nauroth updated HADOOP-7006:
----------------------------------

    Attachment: HADOOP-7006.patch

I've attached the patch.  Could I please have a code review?  Thank you.

> hadoop fs -getmerge does not work using codebase from trunk.
> ------------------------------------------------------------
>
>                 Key: HADOOP-7006
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7006
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Chris Nauroth
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7006.patch
>
>
> Running the codebase from trunk, the hadoop fs -getmerge command does not work.  As implemented in prior versions (i.e. 0.20.2), I could run hadoop fs -getmerge pointed at a directory containing multiple files.  It would merge all files into a single file on the local file system.  Running the same command using the codebase from trunk, it looks like nothing happens.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11090-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 06:55:46 2010
Return-Path: <common-issues-return-11090-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98982 invoked from network); 25 Oct 2010 06:55:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 06:55:46 -0000
Received: (qmail 18841 invoked by uid 500); 25 Oct 2010 06:55:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18730 invoked by uid 500); 25 Oct 2010 06:55:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18721 invoked by uid 99); 25 Oct 2010 06:55:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 06:55:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 06:55:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9P6tOG0010340
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 06:55:24 GMT
Message-ID: <26168514.54681287989724056.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 02:55:24 -0400 (EDT)
From: "Chris Nauroth (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6387) FsShell -getmerge source file
 pattern is broken
In-Reply-To: <299744839.1259028040128.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924468#action_12924468 ] 

Chris Nauroth commented on HADOOP-6387:
---------------------------------------

I'd like to volunteer for this issue, after my patch for HADOOP-7006, which involves the same part of the codebase.


> FsShell -getmerge source file pattern is broken
> -----------------------------------------------
>
>                 Key: HADOOP-6387
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6387
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>
> The FsShell -getmerge command doesn't work if the "source file pattern" matches files. See below. If the current behavior is intended then we should update the help documentation and java docs to match, but it would be nice if the user could specify a set of files in a directory rather than just directories.
> {code}
> $ hadoop fs -help getmerge
> -getmerge <src> <localdst>:  Get all the files in the directories that 
> 		match the source file pattern and merge and sort them to only
> 		one file on local fs. <src> is kept.
> $ hadoop fs -ls
> Found 3 items
> -rw-r--r--   1 eli supergroup          2 2009-11-23 17:39 /user/eli/1.txt
> -rw-r--r--   1 eli supergroup          2 2009-11-23 17:39 /user/eli/2.txt
> -rw-r--r--   1 eli supergroup          2 2009-11-23 17:39 /user/eli/3.txt
> $ hadoop fs -getmerge /user/eli/*.txt sorted.txt
> $ cat sorted.txt
> cat: sorted.txt: No such file or directory
> $ hadoop fs -getmerge /user/eli/* sorted.txt
> $ cat sorted.txt
> cat: sorted.txt: No such file or directory
> $ hadoop fs -getmerge /user/* sorted.txt
> $ cat sorted.txt 
> 1
> 2
> 3
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11091-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 07:11:46 2010
Return-Path: <common-issues-return-11091-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7878 invoked from network); 25 Oct 2010 07:11:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 07:11:46 -0000
Received: (qmail 34687 invoked by uid 500); 25 Oct 2010 07:11:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34644 invoked by uid 500); 25 Oct 2010 07:11:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34624 invoked by uid 99); 25 Oct 2010 07:11:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 07:11:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 07:11:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9P7BLOT010745
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 07:11:21 GMT
Message-ID: <1120688.55141287990681767.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 03:11:21 -0400 (EDT)
From: "Ramkumar Vadali (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6985) Suggest that HADOOP_OPTS be preserved
 in hadoop-env.sh.template
In-Reply-To: <26576141.536571286224772607.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ramkumar Vadali updated HADOOP-6985:
------------------------------------

    Attachment: HADOOP-6985.patch

> Suggest that HADOOP_OPTS be preserved in hadoop-env.sh.template
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6985
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Ramkumar Vadali
>            Priority: Minor
>         Attachments: HADOOP-6985.patch
>
>
> For an administrator who wants to customize HADOOP_OPTS, it would be better to have
> # if [ "$HADOOP_OPTS" == "" ]; then export HADOOP_OPTS=-server; else FOO+=" -server"; fi
> instead of
> # Extra Java runtime options.  Empty by default.
> # export HADOOP_OPTS=-server

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11092-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 07:11:46 2010
Return-Path: <common-issues-return-11092-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7896 invoked from network); 25 Oct 2010 07:11:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 07:11:46 -0000
Received: (qmail 34845 invoked by uid 500); 25 Oct 2010 07:11:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34698 invoked by uid 500); 25 Oct 2010 07:11:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34621 invoked by uid 99); 25 Oct 2010 07:11:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 07:11:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 07:11:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9P7BMeL010760
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 07:11:22 GMT
Message-ID: <28681621.55191287990682309.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 03:11:22 -0400 (EDT)
From: "Ramkumar Vadali (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6985) Suggest that HADOOP_OPTS be preserved
 in hadoop-env.sh.template
In-Reply-To: <26576141.536571286224772607.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ramkumar Vadali updated HADOOP-6985:
------------------------------------

    Status: Patch Available  (was: Open)

> Suggest that HADOOP_OPTS be preserved in hadoop-env.sh.template
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6985
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Ramkumar Vadali
>            Priority: Minor
>         Attachments: HADOOP-6985.patch
>
>
> For an administrator who wants to customize HADOOP_OPTS, it would be better to have
> # if [ "$HADOOP_OPTS" == "" ]; then export HADOOP_OPTS=-server; else FOO+=" -server"; fi
> instead of
> # Extra Java runtime options.  Empty by default.
> # export HADOOP_OPTS=-server

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11093-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 16:33:47 2010
Return-Path: <common-issues-return-11093-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7602 invoked from network); 25 Oct 2010 16:33:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 16:33:47 -0000
Received: (qmail 18117 invoked by uid 500); 25 Oct 2010 16:33:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18070 invoked by uid 500); 25 Oct 2010 16:33:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18062 invoked by uid 99); 25 Oct 2010 16:33:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 16:33:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 16:33:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PGXPTI019717
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 16:33:25 GMT
Message-ID: <26555000.61321288024405671.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 12:33:25 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6947) Kerberos relogin should set
 refreshKrb5Config to true
In-Reply-To: <15312872.90501284007413131.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924621#action_12924621 ] 

Kan Zhang commented on HADOOP-6947:
-----------------------------------

Todd, sorry for late reply. I just got back after a long leave. And sorry for not being able to commit it as I'm not a committer yet. :)

> Kerberos relogin should set refreshKrb5Config to true
> -----------------------------------------------------
>
>                 Key: HADOOP-6947
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6947
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-6947-branch20.txt, hadoop-6947.txt
>
>
> In working on securing a daemon that uses two different principals from different threads, I found that I wasn't able to login from a second keytab after I'd logged in from the first. This is because we don't set the refreshKrb5Config in the Configuration for the Krb5LoginModule - hence it won't switch over to the correct keytab file if it's different than the first.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11094-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 16:41:43 2010
Return-Path: <common-issues-return-11094-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10591 invoked from network); 25 Oct 2010 16:41:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 16:41:42 -0000
Received: (qmail 32251 invoked by uid 500); 25 Oct 2010 16:41:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31998 invoked by uid 500); 25 Oct 2010 16:41:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31986 invoked by uid 99); 25 Oct 2010 16:41:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 16:41:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 16:41:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PGfKed019857
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 16:41:21 GMT
Message-ID: <6533357.61451288024880905.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 12:41:20 -0400 (EDT)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7006) hadoop fs -getmerge does not work
 using codebase from trunk.
In-Reply-To: <21950654.54351287988881239.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924625#action_12924625 ] 

Aaron T. Myers commented on HADOOP-7006:
----------------------------------------

I've reviewed the patch, and it looks good to me.

> hadoop fs -getmerge does not work using codebase from trunk.
> ------------------------------------------------------------
>
>                 Key: HADOOP-7006
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7006
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Chris Nauroth
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7006.patch
>
>
> Running the codebase from trunk, the hadoop fs -getmerge command does not work.  As implemented in prior versions (i.e. 0.20.2), I could run hadoop fs -getmerge pointed at a directory containing multiple files.  It would merge all files into a single file on the local file system.  Running the same command using the codebase from trunk, it looks like nothing happens.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11095-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 18:54:41 2010
Return-Path: <common-issues-return-11095-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78156 invoked from network); 25 Oct 2010 18:54:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 18:54:41 -0000
Received: (qmail 33915 invoked by uid 500); 25 Oct 2010 18:54:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33793 invoked by uid 500); 25 Oct 2010 18:54:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33785 invoked by uid 99); 25 Oct 2010 18:54:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 18:54:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 18:54:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PIsJ0v022329
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 18:54:20 GMT
Message-ID: <4604213.64351288032859734.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 14:54:19 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7005) Update test-patch.sh to remove
 callback to Hudson master
In-Reply-To: <17555034.54181287988399911.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924680#action_12924680 ] 

Giridharan Kesavan commented on HADOOP-7005:
--------------------------------------------

this patch looks good. +1
hdfs and mapreduce is already using the svn:externals to refer to the test-patch.sh script located in hadoop common. 

> Update test-patch.sh to remove callback to Hudson master
> --------------------------------------------------------
>
>                 Key: HADOOP-7005
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7005
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>            Reporter: Nigel Daley
>             Fix For: hudson
>
>         Attachments: HADOOP-7005.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11096-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 19:02:43 2010
Return-Path: <common-issues-return-11096-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80862 invoked from network); 25 Oct 2010 19:02:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 19:02:42 -0000
Received: (qmail 41641 invoked by uid 500); 25 Oct 2010 19:02:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41584 invoked by uid 500); 25 Oct 2010 19:02:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41576 invoked by uid 99); 25 Oct 2010 19:02:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 19:02:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 19:02:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PJ2LsK022648
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 19:02:22 GMT
Message-ID: <19401211.64511288033341723.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 15:02:21 -0400 (EDT)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-7005) Update test-patch.sh to remove
 callback to Hudson master
In-Reply-To: <17555034.54181287988399911.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nigel Daley resolved HADOOP-7005.
---------------------------------

       Resolution: Fixed
    Fix Version/s:     (was: hudson)
                   0.22.0
         Assignee: Nigel Daley
     Release Note: N/A

I just committed this.

> Update test-patch.sh to remove callback to Hudson master
> --------------------------------------------------------
>
>                 Key: HADOOP-7005
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7005
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7005.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11097-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 19:22:44 2010
Return-Path: <common-issues-return-11097-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92282 invoked from network); 25 Oct 2010 19:22:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 19:22:44 -0000
Received: (qmail 69565 invoked by uid 500); 25 Oct 2010 19:22:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69463 invoked by uid 500); 25 Oct 2010 19:22:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69454 invoked by uid 99); 25 Oct 2010 19:22:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 19:22:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 19:22:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PJMK1O023081
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 19:22:20 GMT
Message-ID: <16415448.64671288034540380.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 15:22:20 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7005) Update test-patch.sh to remove
 callback to Hudson master
In-Reply-To: <17555034.54181287988399911.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924686#action_12924686 ] 

Hudson commented on HADOOP-7005:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #398 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/398/])
    HADOOP-7005: Update test-patch.sh to remove callback to Hudson master. Contributed by nigel.


> Update test-patch.sh to remove callback to Hudson master
> --------------------------------------------------------
>
>                 Key: HADOOP-7005
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7005
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7005.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11098-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 19:44:42 2010
Return-Path: <common-issues-return-11098-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99003 invoked from network); 25 Oct 2010 19:44:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 19:44:42 -0000
Received: (qmail 95636 invoked by uid 500); 25 Oct 2010 19:44:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95568 invoked by uid 500); 25 Oct 2010 19:44:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95560 invoked by uid 99); 25 Oct 2010 19:44:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 19:44:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 19:44:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PJiLXh023381
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 19:44:21 GMT
Message-ID: <1578704.64951288035861606.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 15:44:21 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7006) hadoop fs -getmerge does not work
 using codebase from trunk.
In-Reply-To: <21950654.54351287988881239.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-7006:
---------------------------------

        Assignee: Chris Nauroth
    Hadoop Flags: [Reviewed]
          Status: Patch Available  (was: Open)

> hadoop fs -getmerge does not work using codebase from trunk.
> ------------------------------------------------------------
>
>                 Key: HADOOP-7006
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7006
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7006.patch, HADOOP-7006.patch
>
>
> Running the codebase from trunk, the hadoop fs -getmerge command does not work.  As implemented in prior versions (i.e. 0.20.2), I could run hadoop fs -getmerge pointed at a directory containing multiple files.  It would merge all files into a single file on the local file system.  Running the same command using the codebase from trunk, it looks like nothing happens.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11099-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 19:44:44 2010
Return-Path: <common-issues-return-11099-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99055 invoked from network); 25 Oct 2010 19:44:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 19:44:44 -0000
Received: (qmail 95784 invoked by uid 500); 25 Oct 2010 19:44:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95755 invoked by uid 500); 25 Oct 2010 19:44:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95747 invoked by uid 99); 25 Oct 2010 19:44:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 19:44:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 19:44:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PJiKma023369
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 19:44:21 GMT
Message-ID: <16426407.64911288035860659.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 15:44:20 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7006) hadoop fs -getmerge does not work
 using codebase from trunk.
In-Reply-To: <21950654.54351287988881239.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-7006:
---------------------------------

    Attachment: HADOOP-7006.patch

Tests failed for me until I also changed the file merge to sort the files.  This is probably a good idea anyway.  Here's a new version of the patch with that change.

> hadoop fs -getmerge does not work using codebase from trunk.
> ------------------------------------------------------------
>
>                 Key: HADOOP-7006
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7006
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Chris Nauroth
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7006.patch, HADOOP-7006.patch
>
>
> Running the codebase from trunk, the hadoop fs -getmerge command does not work.  As implemented in prior versions (i.e. 0.20.2), I could run hadoop fs -getmerge pointed at a directory containing multiple files.  It would merge all files into a single file on the local file system.  Running the same command using the codebase from trunk, it looks like nothing happens.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11100-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 19:48:45 2010
Return-Path: <common-issues-return-11100-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1826 invoked from network); 25 Oct 2010 19:48:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 19:48:45 -0000
Received: (qmail 2969 invoked by uid 500); 25 Oct 2010 19:48:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2864 invoked by uid 500); 25 Oct 2010 19:48:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2856 invoked by uid 99); 25 Oct 2010 19:48:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 19:48:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 19:48:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PJmLXI023443
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 19:48:21 GMT
Message-ID: <2969789.65001288036101093.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 15:48:21 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6985) Suggest that HADOOP_OPTS be
 preserved in hadoop-env.sh.template
In-Reply-To: <26576141.536571286224772607.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924693#action_12924693 ] 

Doug Cutting commented on HADOOP-6985:
--------------------------------------

This looks reasonable to me.  It's arguably not back-compatible, since settings of HADOOP_OPTS would previously be ignored.  Does that concern anyone?  If not, I'll commit this.

> Suggest that HADOOP_OPTS be preserved in hadoop-env.sh.template
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6985
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Ramkumar Vadali
>            Priority: Minor
>         Attachments: HADOOP-6985.patch
>
>
> For an administrator who wants to customize HADOOP_OPTS, it would be better to have
> # if [ "$HADOOP_OPTS" == "" ]; then export HADOOP_OPTS=-server; else FOO+=" -server"; fi
> instead of
> # Extra Java runtime options.  Empty by default.
> # export HADOOP_OPTS=-server

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11101-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 19:48:46 2010
Return-Path: <common-issues-return-11101-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1884 invoked from network); 25 Oct 2010 19:48:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 19:48:45 -0000
Received: (qmail 3086 invoked by uid 500); 25 Oct 2010 19:48:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3047 invoked by uid 500); 25 Oct 2010 19:48:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3039 invoked by uid 99); 25 Oct 2010 19:48:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 19:48:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 19:48:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PJmMOZ023458
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 19:48:22 GMT
Message-ID: <23234657.65051288036102147.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 15:48:22 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6985) Suggest that HADOOP_OPTS be
 preserved in hadoop-env.sh.template
In-Reply-To: <26576141.536571286224772607.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting reassigned HADOOP-6985:
------------------------------------

    Assignee: Ramkumar Vadali

> Suggest that HADOOP_OPTS be preserved in hadoop-env.sh.template
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6985
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Ramkumar Vadali
>            Assignee: Ramkumar Vadali
>            Priority: Minor
>         Attachments: HADOOP-6985.patch
>
>
> For an administrator who wants to customize HADOOP_OPTS, it would be better to have
> # if [ "$HADOOP_OPTS" == "" ]; then export HADOOP_OPTS=-server; else FOO+=" -server"; fi
> instead of
> # Extra Java runtime options.  Empty by default.
> # export HADOOP_OPTS=-server

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11102-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 20:24:44 2010
Return-Path: <common-issues-return-11102-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20848 invoked from network); 25 Oct 2010 20:24:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 20:24:43 -0000
Received: (qmail 55760 invoked by uid 500); 25 Oct 2010 20:24:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55736 invoked by uid 500); 25 Oct 2010 20:24:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55728 invoked by uid 99); 25 Oct 2010 20:24:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 20:24:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 20:24:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PKOJ46026593
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 20:24:20 GMT
Message-ID: <6582704.65731288038259857.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 16:24:19 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7005) Update test-patch.sh to remove
 callback to Hudson master
In-Reply-To: <17555034.54181287988399911.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924706#action_12924706 ] 

Hudson commented on HADOOP-7005:
--------------------------------

Integrated in Hadoop-Hdfs-trunk-Commit #417 (See [https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/417/])
    HADOOP-7005: Update test-patch.sh to remove callback to Hudson master. Contributed by nigel.


> Update test-patch.sh to remove callback to Hudson master
> --------------------------------------------------------
>
>                 Key: HADOOP-7005
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7005
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7005.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11103-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 21:18:52 2010
Return-Path: <common-issues-return-11103-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42223 invoked from network); 25 Oct 2010 21:18:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 21:18:51 -0000
Received: (qmail 16156 invoked by uid 500); 25 Oct 2010 21:18:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16112 invoked by uid 500); 25 Oct 2010 21:18:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16104 invoked by uid 99); 25 Oct 2010 21:18:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 21:18:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 21:18:49 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PLIRwN027595
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 21:18:27 GMT
Message-ID: <9525389.66951288041507318.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 17:18:27 -0400 (EDT)
From: "Patrick Kling (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7001) Allow configuration changes without
 restarting configured nodes
In-Reply-To: <33220874.146971287081516580.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Patrick Kling updated HADOOP-7001:
----------------------------------

    Attachment: HADOOP-7001.patch

Here is a new version of my patch. I have changed the interface Reconfigurable somewhat: Since only a subset of configuration properties can be changed at run time, it makes more sense to have an interface that allows us to change a single property at a time, rather than passing a Configuration object and then detecting all changes that need to be made. I have also added methods for detecting which configuration properties can be changed at runtime.

{code}
/**                                                                                                                                                         
 * Something whose {@link Configuration} can be changed at run time.                                                                                        
 */
public interface Reconfigurable extends Configurable {

  /**                                                                                                                                                       
   * Change a the configuration property on this object to the value specified.                                                                             
   *                                                                                                                                                        
   * Change a the configuration property on this object to the value specified                                                                              
   * and return the previous value that the configuration property was set to                                                                               
   * (or null if it was not previously set).                                                                                                                
   *                                                                                                                                                        
   * If the property cannot be changed, throw a                                                                                                             
   * {@link ConfigurationChangeException}.                                                                                                                  
   */
  public String changeConfProperty(String property, String newVal)
    throws ConfigurationChangeException;

  /**                                                                                                                                                       
   * Return whether a given property is changeable at run time.                                                                                             
   *                                                                                                                                                        
   * If isConfPropertyChangeable returns true for a property,                                                                                               
   * then changeConf should not throw an exception when changing                                                                                            
   * this property.                                                                                                                                         
   */
  public boolean isConfPropertyChangeable(String property);

  /**                                                                                                                                                       
   * Return all the properties that can be changed at run time.                                                                                             
   */
  public List<String> getChangeableConfProperties();
}
{code}

The patch also includes a simple servlet for changing the configuration of a Reconfigurable. In order to allow configuration changes at run time, I made all the getters and setters in Configuration synchronized.

> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7001
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7001
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Patrick Kling
>            Assignee: Patrick Kling
>         Attachments: HADOOP-7001.patch, reconfigurable.patch
>
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
>    void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11104-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 21:30:47 2010
Return-Path: <common-issues-return-11104-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46797 invoked from network); 25 Oct 2010 21:30:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 21:30:47 -0000
Received: (qmail 32174 invoked by uid 500); 25 Oct 2010 21:30:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32118 invoked by uid 500); 25 Oct 2010 21:30:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32086 invoked by uid 99); 25 Oct 2010 21:30:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 21:30:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 21:30:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PLUK9K027783
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 21:30:20 GMT
Message-ID: <7411320.67141288042220266.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 17:30:20 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-7007) update the hudson-test-patch target
 to work with the latest test-patch script.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

update the hudson-test-patch target to work with the latest test-patch script.
------------------------------------------------------------------------------

                 Key: HADOOP-7007
                 URL: https://issues.apache.org/jira/browse/HADOOP-7007
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Giridharan Kesavan


The hudson-test-patch target has to be updated to work with the current test-patch.sh script. Since the callback login in the test-patch.sh is removed. by hadoop-7005

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11105-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 21:34:46 2010
Return-Path: <common-issues-return-11105-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47828 invoked from network); 25 Oct 2010 21:34:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 21:34:46 -0000
Received: (qmail 35254 invoked by uid 500); 25 Oct 2010 21:34:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35207 invoked by uid 500); 25 Oct 2010 21:34:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35199 invoked by uid 99); 25 Oct 2010 21:34:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 21:34:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 21:34:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PLYMh6027849
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 21:34:22 GMT
Message-ID: <232975.67251288042462231.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 17:34:22 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7007) update the hudson-test-patch target
 to work with the latest test-patch script.
In-Reply-To: <7411320.67141288042220266.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-7007:
---------------------------------------

    Attachment: HADOOP-7007.patch

> update the hudson-test-patch target to work with the latest test-patch script.
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7007
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7007
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Giridharan Kesavan
>         Attachments: HADOOP-7007.patch
>
>
> The hudson-test-patch target has to be updated to work with the current test-patch.sh script. Since the callback login in the test-patch.sh is removed. by hadoop-7005

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11106-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 21:40:42 2010
Return-Path: <common-issues-return-11106-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49311 invoked from network); 25 Oct 2010 21:40:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 21:40:41 -0000
Received: (qmail 39787 invoked by uid 500); 25 Oct 2010 21:40:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39749 invoked by uid 500); 25 Oct 2010 21:40:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39736 invoked by uid 99); 25 Oct 2010 21:40:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 21:40:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 21:40:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PLeKdl027938
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 21:40:20 GMT
Message-ID: <1911445.67351288042820407.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 17:40:20 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7007) update the hudson-test-patch target
 to work with the latest test-patch script.
In-Reply-To: <7411320.67141288042220266.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-7007:
---------------------------------------

          Component/s: build
    Affects Version/s: 0.22.0
             Assignee: Giridharan Kesavan

updating the jira with the right set to version and component

> update the hudson-test-patch target to work with the latest test-patch script.
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7007
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7007
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>         Attachments: HADOOP-7007.patch
>
>
> The hudson-test-patch target has to be updated to work with the current test-patch.sh script. Since the callback login in the test-patch.sh is removed. by hadoop-7005

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11107-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 21:46:48 2010
Return-Path: <common-issues-return-11107-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51549 invoked from network); 25 Oct 2010 21:46:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 21:46:47 -0000
Received: (qmail 45345 invoked by uid 500); 25 Oct 2010 21:46:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45317 invoked by uid 500); 25 Oct 2010 21:46:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45309 invoked by uid 99); 25 Oct 2010 21:46:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 21:46:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 21:46:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PLkNTK028070
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 21:46:23 GMT
Message-ID: <25120118.67581288043183662.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 17:46:23 -0400 (EDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7001) Allow configuration changes without
 restarting configured nodes
In-Reply-To: <33220874.146971287081516580.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924738#action_12924738 ] 

dhruba borthakur commented on HADOOP-7001:
------------------------------------------

I added Doug and Tom to the list of watchers. I am hoping that they provide some feedback to this JIRA.

> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7001
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7001
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Patrick Kling
>            Assignee: Patrick Kling
>         Attachments: HADOOP-7001.patch, reconfigurable.patch
>
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
>    void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11108-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 21:50:41 2010
Return-Path: <common-issues-return-11108-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52183 invoked from network); 25 Oct 2010 21:50:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 21:50:41 -0000
Received: (qmail 47132 invoked by uid 500); 25 Oct 2010 21:50:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47102 invoked by uid 500); 25 Oct 2010 21:50:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47094 invoked by uid 99); 25 Oct 2010 21:50:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 21:50:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 21:50:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PLoKC7028123
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 21:50:20 GMT
Message-ID: <29311284.67641288043420546.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 17:50:20 -0400 (EDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6985) Suggest that HADOOP_OPTS be
 preserved in hadoop-env.sh.template
In-Reply-To: <26576141.536571286224772607.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924740#action_12924740 ] 

dhruba borthakur commented on HADOOP-6985:
------------------------------------------

+1, the setting of HADOOP_OPTS would have been ignored prior to this patch.

> Suggest that HADOOP_OPTS be preserved in hadoop-env.sh.template
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6985
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Ramkumar Vadali
>            Assignee: Ramkumar Vadali
>            Priority: Minor
>         Attachments: HADOOP-6985.patch
>
>
> For an administrator who wants to customize HADOOP_OPTS, it would be better to have
> # if [ "$HADOOP_OPTS" == "" ]; then export HADOOP_OPTS=-server; else FOO+=" -server"; fi
> instead of
> # Extra Java runtime options.  Empty by default.
> # export HADOOP_OPTS=-server

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11109-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 21:52:41 2010
Return-Path: <common-issues-return-11109-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52467 invoked from network); 25 Oct 2010 21:52:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 21:52:41 -0000
Received: (qmail 48512 invoked by uid 500); 25 Oct 2010 21:52:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48464 invoked by uid 500); 25 Oct 2010 21:52:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48456 invoked by uid 99); 25 Oct 2010 21:52:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 21:52:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 21:52:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PLqKQB028146
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 21:52:20 GMT
Message-ID: <9027374.67661288043540077.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 17:52:20 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7007) update the hudson-test-patch target
 to work with the latest test-patch script.
In-Reply-To: <7411320.67141288042220266.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-7007:
---------------------------------------

    Attachment: HADOOP-7007-test-patch.patch
                HADOOP-7007-build.xml.patch

HADOOP-7007-build.xml.patch - is for the build.xml for common, hdfs and mapreduce projects
HADOOP-7007-test-patch.patch - is just for the common project, as test-patch.sh is reference by hdfs and mr by svn:extenals.

> update the hudson-test-patch target to work with the latest test-patch script.
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7007
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7007
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>         Attachments: HADOOP-7007-build.xml.patch, HADOOP-7007-test-patch.patch, HADOOP-7007.patch
>
>
> The hudson-test-patch target has to be updated to work with the current test-patch.sh script. Since the callback login in the test-patch.sh is removed. by hadoop-7005

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11110-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 21:56:44 2010
Return-Path: <common-issues-return-11110-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53497 invoked from network); 25 Oct 2010 21:56:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 21:56:43 -0000
Received: (qmail 52038 invoked by uid 500); 25 Oct 2010 21:56:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51955 invoked by uid 500); 25 Oct 2010 21:56:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51947 invoked by uid 99); 25 Oct 2010 21:56:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 21:56:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 21:56:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PLuKDZ028225
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 21:56:20 GMT
Message-ID: <27957391.67761288043780042.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 17:56:20 -0400 (EDT)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7007) update the hudson-test-patch target
 to work with the latest test-patch script.
In-Reply-To: <7411320.67141288042220266.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924745#action_12924745 ] 

Nigel Daley commented on HADOOP-7007:
-------------------------------------

+1, looks good.  Note that the unified patch (HADOOP-7007.patch) shouldn't be applied.

> update the hudson-test-patch target to work with the latest test-patch script.
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7007
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7007
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>         Attachments: HADOOP-7007-build.xml.patch, HADOOP-7007-test-patch.patch, HADOOP-7007.patch
>
>
> The hudson-test-patch target has to be updated to work with the current test-patch.sh script. Since the callback login in the test-patch.sh is removed. by hadoop-7005

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11111-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 22:14:46 2010
Return-Path: <common-issues-return-11111-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62197 invoked from network); 25 Oct 2010 22:14:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 22:14:46 -0000
Received: (qmail 66907 invoked by uid 500); 25 Oct 2010 22:14:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66866 invoked by uid 500); 25 Oct 2010 22:14:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66857 invoked by uid 99); 25 Oct 2010 22:14:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 22:14:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 22:14:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PMEP4O028591
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 22:14:25 GMT
Message-ID: <7805845.68161288044865009.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 18:14:25 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7001) Allow configuration changes without
 restarting configured nodes
In-Reply-To: <33220874.146971287081516580.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924755#action_12924755 ] 

Doug Cutting commented on HADOOP-7001:
--------------------------------------

It doesn't seem unreasonable to me for some limited changes might be permitted without restarting.  Restarting a namenode is not a lightweight operation.

If the interface is 'Reconfigurable' then the method might be named 'reconfigureProp(String,String)', no?

I don't like adding new interfaces that aren't yet used.  So it might be good to file issues for at least one the daemons that you expect to implement this and at least draft those patches before this is committed.  If we can't agree on some concrete usages of this interface within Hadoop, then there's no point in adding it.

The servlet should probably return an HTTP error code when reconfiguration fails.

A command-line client might be useful.  Perhaps this will be added to, e.g.. the existing HDFS admin commands.  But each command-line tool that uses this will share client-side HTTP call, handling of error codes, etc.  So perhaps that should be encapsulated in a shared utility.


> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7001
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7001
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Patrick Kling
>            Assignee: Patrick Kling
>         Attachments: HADOOP-7001.patch, reconfigurable.patch
>
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
>    void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11112-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 22:22:48 2010
Return-Path: <common-issues-return-11112-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67765 invoked from network); 25 Oct 2010 22:22:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 22:22:44 -0000
Received: (qmail 74015 invoked by uid 500); 25 Oct 2010 22:22:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73980 invoked by uid 500); 25 Oct 2010 22:22:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73968 invoked by uid 99); 25 Oct 2010 22:22:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 22:22:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 22:22:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PMMK0h028682
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 22:22:20 GMT
Message-ID: <33501972.68231288045340035.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 18:22:20 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6985) Suggest that HADOOP_OPTS be preserved
 in hadoop-env.sh.template
In-Reply-To: <26576141.536571286224772607.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-6985:
---------------------------------

       Resolution: Fixed
    Fix Version/s: 0.22.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I committed this.  Thanks, Ramkumar.

> Suggest that HADOOP_OPTS be preserved in hadoop-env.sh.template
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6985
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Ramkumar Vadali
>            Assignee: Ramkumar Vadali
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6985.patch
>
>
> For an administrator who wants to customize HADOOP_OPTS, it would be better to have
> # if [ "$HADOOP_OPTS" == "" ]; then export HADOOP_OPTS=-server; else FOO+=" -server"; fi
> instead of
> # Extra Java runtime options.  Empty by default.
> # export HADOOP_OPTS=-server

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11113-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 22:46:41 2010
Return-Path: <common-issues-return-11113-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75120 invoked from network); 25 Oct 2010 22:46:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 22:46:41 -0000
Received: (qmail 89450 invoked by uid 500); 25 Oct 2010 22:46:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89345 invoked by uid 500); 25 Oct 2010 22:46:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89337 invoked by uid 99); 25 Oct 2010 22:46:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 22:46:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 22:46:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PMkKxs028982
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 22:46:20 GMT
Message-ID: <23496849.68471288046780624.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 18:46:20 -0400 (EDT)
From: "Koji Noguchi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6985) Suggest that HADOOP_OPTS be
 preserved in hadoop-env.sh.template
In-Reply-To: <26576141.536571286224772607.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924763#action_12924763 ] 

Koji Noguchi commented on HADOOP-6985:
--------------------------------------

Curious.

bq. else FOO+=" -server"; fi

Where is FOO being used?

> Suggest that HADOOP_OPTS be preserved in hadoop-env.sh.template
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6985
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Ramkumar Vadali
>            Assignee: Ramkumar Vadali
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6985.patch
>
>
> For an administrator who wants to customize HADOOP_OPTS, it would be better to have
> # if [ "$HADOOP_OPTS" == "" ]; then export HADOOP_OPTS=-server; else FOO+=" -server"; fi
> instead of
> # Extra Java runtime options.  Empty by default.
> # export HADOOP_OPTS=-server

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11114-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 23:04:44 2010
Return-Path: <common-issues-return-11114-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81704 invoked from network); 25 Oct 2010 23:04:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 23:04:44 -0000
Received: (qmail 5036 invoked by uid 500); 25 Oct 2010 23:04:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4997 invoked by uid 500); 25 Oct 2010 23:04:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4988 invoked by uid 99); 25 Oct 2010 23:04:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 23:04:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 23:04:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PN4KM7029301
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 23:04:20 GMT
Message-ID: <30208748.68741288047860497.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 19:04:20 -0400 (EDT)
From: "Ramkumar Vadali (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Reopened: (HADOOP-6985) Suggest that HADOOP_OPTS be
 preserved in hadoop-env.sh.template
In-Reply-To: <26576141.536571286224772607.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ramkumar Vadali reopened HADOOP-6985:
-------------------------------------


Ugh! FOO -> HADOOP_OPTS
Thats embarassing!

> Suggest that HADOOP_OPTS be preserved in hadoop-env.sh.template
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6985
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Ramkumar Vadali
>            Assignee: Ramkumar Vadali
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6985.patch
>
>
> For an administrator who wants to customize HADOOP_OPTS, it would be better to have
> # if [ "$HADOOP_OPTS" == "" ]; then export HADOOP_OPTS=-server; else FOO+=" -server"; fi
> instead of
> # Extra Java runtime options.  Empty by default.
> # export HADOOP_OPTS=-server

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11115-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 23:06:42 2010
Return-Path: <common-issues-return-11115-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86559 invoked from network); 25 Oct 2010 23:06:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 23:06:42 -0000
Received: (qmail 9895 invoked by uid 500); 25 Oct 2010 23:06:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9856 invoked by uid 500); 25 Oct 2010 23:06:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9847 invoked by uid 99); 25 Oct 2010 23:06:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 23:06:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 23:06:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PN6Ku8029339
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 23:06:21 GMT
Message-ID: <27949714.68801288047980858.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 19:06:20 -0400 (EDT)
From: "Ramkumar Vadali (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6985) Suggest that HADOOP_OPTS be preserved
 in hadoop-env.sh.template
In-Reply-To: <26576141.536571286224772607.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ramkumar Vadali updated HADOOP-6985:
------------------------------------

    Attachment: HADOOP-6985.patch

FOO->HADOOP_OPTS

> Suggest that HADOOP_OPTS be preserved in hadoop-env.sh.template
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6985
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Ramkumar Vadali
>            Assignee: Ramkumar Vadali
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6985.patch
>
>
> For an administrator who wants to customize HADOOP_OPTS, it would be better to have
> # if [ "$HADOOP_OPTS" == "" ]; then export HADOOP_OPTS=-server; else FOO+=" -server"; fi
> instead of
> # Extra Java runtime options.  Empty by default.
> # export HADOOP_OPTS=-server

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11116-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 23:06:42 2010
Return-Path: <common-issues-return-11116-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86609 invoked from network); 25 Oct 2010 23:06:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 23:06:42 -0000
Received: (qmail 10081 invoked by uid 500); 25 Oct 2010 23:06:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10046 invoked by uid 500); 25 Oct 2010 23:06:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10038 invoked by uid 99); 25 Oct 2010 23:06:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 23:06:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 23:06:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PN6LXt029354
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 23:06:21 GMT
Message-ID: <19232474.68851288047981830.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 19:06:21 -0400 (EDT)
From: "Ramkumar Vadali (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6985) Suggest that HADOOP_OPTS be preserved
 in hadoop-env.sh.template
In-Reply-To: <26576141.536571286224772607.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ramkumar Vadali updated HADOOP-6985:
------------------------------------

    Attachment:     (was: HADOOP-6985.patch)

> Suggest that HADOOP_OPTS be preserved in hadoop-env.sh.template
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6985
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Ramkumar Vadali
>            Assignee: Ramkumar Vadali
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6985.patch
>
>
> For an administrator who wants to customize HADOOP_OPTS, it would be better to have
> # if [ "$HADOOP_OPTS" == "" ]; then export HADOOP_OPTS=-server; else FOO+=" -server"; fi
> instead of
> # Extra Java runtime options.  Empty by default.
> # export HADOOP_OPTS=-server

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11117-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 23:06:46 2010
Return-Path: <common-issues-return-11117-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86677 invoked from network); 25 Oct 2010 23:06:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 23:06:46 -0000
Received: (qmail 10289 invoked by uid 500); 25 Oct 2010 23:06:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10238 invoked by uid 500); 25 Oct 2010 23:06:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10230 invoked by uid 99); 25 Oct 2010 23:06:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 23:06:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 23:06:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PN6MMm029369
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 23:06:22 GMT
Message-ID: <27605347.68901288047982665.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 19:06:22 -0400 (EDT)
From: "Ramkumar Vadali (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6985) Suggest that HADOOP_OPTS be preserved
 in hadoop-env.sh.template
In-Reply-To: <26576141.536571286224772607.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ramkumar Vadali updated HADOOP-6985:
------------------------------------

    Attachment:     (was: HADOOP-6985.patch)

> Suggest that HADOOP_OPTS be preserved in hadoop-env.sh.template
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6985
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Ramkumar Vadali
>            Assignee: Ramkumar Vadali
>            Priority: Minor
>             Fix For: 0.22.0
>
>
> For an administrator who wants to customize HADOOP_OPTS, it would be better to have
> # if [ "$HADOOP_OPTS" == "" ]; then export HADOOP_OPTS=-server; else FOO+=" -server"; fi
> instead of
> # Extra Java runtime options.  Empty by default.
> # export HADOOP_OPTS=-server

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11118-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 23:08:42 2010
Return-Path: <common-issues-return-11118-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87074 invoked from network); 25 Oct 2010 23:08:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 23:08:41 -0000
Received: (qmail 11270 invoked by uid 500); 25 Oct 2010 23:08:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11245 invoked by uid 500); 25 Oct 2010 23:08:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11237 invoked by uid 99); 25 Oct 2010 23:08:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 23:08:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 23:08:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PN8L98029406
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 23:08:21 GMT
Message-ID: <7981731.68971288048100989.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 19:08:20 -0400 (EDT)
From: "Ramkumar Vadali (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6985) Suggest that HADOOP_OPTS be preserved
 in hadoop-env.sh.template
In-Reply-To: <26576141.536571286224772607.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ramkumar Vadali updated HADOOP-6985:
------------------------------------

    Attachment: HADOOP_6985.2.patch

FOO -> HADOOP_OPTS

> Suggest that HADOOP_OPTS be preserved in hadoop-env.sh.template
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6985
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Ramkumar Vadali
>            Assignee: Ramkumar Vadali
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP_6985.2.patch
>
>
> For an administrator who wants to customize HADOOP_OPTS, it would be better to have
> # if [ "$HADOOP_OPTS" == "" ]; then export HADOOP_OPTS=-server; else FOO+=" -server"; fi
> instead of
> # Extra Java runtime options.  Empty by default.
> # export HADOOP_OPTS=-server

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11119-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 23:08:46 2010
Return-Path: <common-issues-return-11119-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87205 invoked from network); 25 Oct 2010 23:08:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 23:08:46 -0000
Received: (qmail 11976 invoked by uid 500); 25 Oct 2010 23:08:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11946 invoked by uid 500); 25 Oct 2010 23:08:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11938 invoked by uid 99); 25 Oct 2010 23:08:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 23:08:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 23:08:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PN8MFY029430
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 23:08:22 GMT
Message-ID: <15901950.69051288048102635.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 19:08:22 -0400 (EDT)
From: "Ramkumar Vadali (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6985) Suggest that HADOOP_OPTS be preserved
 in hadoop-env.sh.template
In-Reply-To: <26576141.536571286224772607.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ramkumar Vadali updated HADOOP-6985:
------------------------------------

    Status: Patch Available  (was: Reopened)

> Suggest that HADOOP_OPTS be preserved in hadoop-env.sh.template
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6985
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Ramkumar Vadali
>            Assignee: Ramkumar Vadali
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP_6985.2.patch
>
>
> For an administrator who wants to customize HADOOP_OPTS, it would be better to have
> # if [ "$HADOOP_OPTS" == "" ]; then export HADOOP_OPTS=-server; else FOO+=" -server"; fi
> instead of
> # Extra Java runtime options.  Empty by default.
> # export HADOOP_OPTS=-server

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11120-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 23:10:43 2010
Return-Path: <common-issues-return-11120-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88688 invoked from network); 25 Oct 2010 23:10:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 23:10:43 -0000
Received: (qmail 13923 invoked by uid 500); 25 Oct 2010 23:10:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13836 invoked by uid 500); 25 Oct 2010 23:10:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13828 invoked by uid 99); 25 Oct 2010 23:10:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 23:10:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 23:10:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PNAMYQ029472
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 23:10:22 GMT
Message-ID: <17987193.69111288048222296.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 19:10:22 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-7007) update the hudson-test-patch target
 to work with the latest test-patch script.
In-Reply-To: <7411320.67141288042220266.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan resolved HADOOP-7007.
----------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.22.0

Patch committed.

> update the hudson-test-patch target to work with the latest test-patch script.
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7007
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7007
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7007-build.xml.patch, HADOOP-7007-test-patch.patch, HADOOP-7007.patch
>
>
> The hudson-test-patch target has to be updated to work with the current test-patch.sh script. Since the callback login in the test-patch.sh is removed. by hadoop-7005

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11121-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 23:18:47 2010
Return-Path: <common-issues-return-11121-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91706 invoked from network); 25 Oct 2010 23:18:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 23:18:47 -0000
Received: (qmail 27383 invoked by uid 500); 25 Oct 2010 23:18:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27352 invoked by uid 500); 25 Oct 2010 23:18:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27344 invoked by uid 99); 25 Oct 2010 23:18:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 23:18:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 23:18:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PNINEr029603
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 23:18:23 GMT
Message-ID: <26851743.69321288048703325.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 19:18:23 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6985) Suggest that HADOOP_OPTS be preserved
 in hadoop-env.sh.template
In-Reply-To: <26576141.536571286224772607.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-6985:
---------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

> Suggest that HADOOP_OPTS be preserved in hadoop-env.sh.template
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6985
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Ramkumar Vadali
>            Assignee: Ramkumar Vadali
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP_6985.2.patch
>
>
> For an administrator who wants to customize HADOOP_OPTS, it would be better to have
> # if [ "$HADOOP_OPTS" == "" ]; then export HADOOP_OPTS=-server; else FOO+=" -server"; fi
> instead of
> # Extra Java runtime options.  Empty by default.
> # export HADOOP_OPTS=-server

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11122-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 23:22:46 2010
Return-Path: <common-issues-return-11122-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92366 invoked from network); 25 Oct 2010 23:22:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 23:22:45 -0000
Received: (qmail 29195 invoked by uid 500); 25 Oct 2010 23:22:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29155 invoked by uid 500); 25 Oct 2010 23:22:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29147 invoked by uid 99); 25 Oct 2010 23:22:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 23:22:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 23:22:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PNMLgo029668
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 23:22:22 GMT
Message-ID: <7419425.69411288048941842.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 19:22:21 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6916) Implement append operation for
 S3FileSystem
In-Reply-To: <6991431.365871281969257661.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6916:
------------------------------

    Status: Open  (was: Patch Available)

Marking as open while feedback is addressed.

> Implement append operation for S3FileSystem
> -------------------------------------------
>
>                 Key: HADOOP-6916
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6916
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs/s3
>    Affects Versions: 0.20.2, 0.22.0
>            Reporter: Oleg Aleshko
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: s3_append1.patch
>
>
> Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException("Not supported");
> S3FileSystem should be able to support appending, possibly via common block storage logic.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11123-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 23:28:45 2010
Return-Path: <common-issues-return-11123-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95824 invoked from network); 25 Oct 2010 23:28:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 23:28:45 -0000
Received: (qmail 39109 invoked by uid 500); 25 Oct 2010 23:28:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39069 invoked by uid 500); 25 Oct 2010 23:28:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39061 invoked by uid 99); 25 Oct 2010 23:28:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 23:28:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 23:28:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PNSNuP029795
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 23:28:24 GMT
Message-ID: <5059717.69661288049303860.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 19:28:23 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6317) Serialize the 'final'ness of
 Configuration keys
In-Reply-To: <985698151.1255715791730.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924781#action_12924781 ] 

Hadoop QA commented on HADOOP-6317:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12457858/HADOOP-6317.patch
  against trunk revision 1027301.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 7 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/8//console

This message is automatically generated.

> Serialize the 'final'ness of Configuration keys
> -----------------------------------------------
>
>                 Key: HADOOP-6317
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6317
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Arun C Murthy
>            Assignee: Aaron Kimball
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6317.2.patch, HADOOP-6317.3.patch, HADOOP-6317.4.patch, HADOOP-6317.patch, HADOOP-6317.patch
>
>
> Currently Configuration.writeXml doesn't serialize the 'final' attribute. There are some scenarios where it is useful:
> # TaskTracker should mark the tasks' 'localized' parameters as 'final' 
> # GUI tools

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11124-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 23:38:58 2010
Return-Path: <common-issues-return-11124-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98950 invoked from network); 25 Oct 2010 23:38:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 23:38:58 -0000
Received: (qmail 47471 invoked by uid 500); 25 Oct 2010 23:38:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47447 invoked by uid 500); 25 Oct 2010 23:38:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47439 invoked by uid 99); 25 Oct 2010 23:38:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 23:38:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 23:38:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PNcX3Y029908
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 23:38:34 GMT
Message-ID: <32614174.69721288049913832.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 19:38:33 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6663) BlockDecompressorStream get EOF
 exception when decompressing the file compressed from empty file
In-Reply-To: <1619883340.543171269850767204.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6663:
------------------------------

       Resolution: Fixed
    Fix Version/s: 0.22.0
         Assignee: Kang Xiao
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I've just committed this. Thanks Kang Xiao!

> BlockDecompressorStream get EOF exception when decompressing the file compressed from empty file
> ------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6663
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6663
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Kang Xiao
>            Assignee: Kang Xiao
>             Fix For: 0.22.0
>
>         Attachments: BlockDecompressorStream.java.patch, BlockDecompressorStream.java.patch, BlockDecompressorStream.patch
>
>
> An empty file can be compressed using BlockDecompressorStream, which is for block-based compressiong algorithm such as LZO. However, when decompressing the compressed file, BlockDecompressorStream get EOF exception.
> Here is a typical exception stack:
> java.io.EOFException
> at org.apache.hadoop.io.compress.BlockDecompressorStream.rawReadInt(BlockDecompressorStream.java:125)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:96)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:82)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at java.io.InputStream.read(InputStream.java:85)
> at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:39)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:186)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:170)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:18)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
> at org.apache.hadoop.mapred.Child.main(Child.java:196)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11125-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 23:59:14 2010
Return-Path: <common-issues-return-11125-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1930 invoked from network); 25 Oct 2010 23:59:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 23:59:14 -0000
Received: (qmail 60163 invoked by uid 500); 25 Oct 2010 23:59:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60088 invoked by uid 500); 25 Oct 2010 23:59:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60080 invoked by uid 99); 25 Oct 2010 23:59:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 23:59:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 23:59:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PNwrt4000173
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 23:58:53 GMT
Message-ID: <24302897.69861288051133156.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 19:58:53 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6663) BlockDecompressorStream get EOF
 exception when decompressing the file compressed from empty file
In-Reply-To: <1619883340.543171269850767204.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6663:
------------------------------

    Attachment: HADOOP-6663.patch

> BlockDecompressorStream get EOF exception when decompressing the file compressed from empty file
> ------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6663
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6663
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Kang Xiao
>            Assignee: Kang Xiao
>             Fix For: 0.22.0
>
>         Attachments: BlockDecompressorStream.java.patch, BlockDecompressorStream.java.patch, BlockDecompressorStream.patch, HADOOP-6663.patch
>
>
> An empty file can be compressed using BlockDecompressorStream, which is for block-based compressiong algorithm such as LZO. However, when decompressing the compressed file, BlockDecompressorStream get EOF exception.
> Here is a typical exception stack:
> java.io.EOFException
> at org.apache.hadoop.io.compress.BlockDecompressorStream.rawReadInt(BlockDecompressorStream.java:125)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:96)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:82)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at java.io.InputStream.read(InputStream.java:85)
> at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:39)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:186)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:170)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:18)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
> at org.apache.hadoop.mapred.Child.main(Child.java:196)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11126-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 23:59:18 2010
Return-Path: <common-issues-return-11126-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2089 invoked from network); 25 Oct 2010 23:59:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 23:59:17 -0000
Received: (qmail 61205 invoked by uid 500); 25 Oct 2010 23:59:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61168 invoked by uid 500); 25 Oct 2010 23:59:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61137 invoked by uid 99); 25 Oct 2010 23:59:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 23:59:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 23:59:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PNwqTo000157
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 23:58:52 GMT
Message-ID: <22183248.69811288051132369.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 19:58:52 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Reopened: (HADOOP-6663) BlockDecompressorStream get EOF
 exception when decompressing the file compressed from empty file
In-Reply-To: <1619883340.543171269850767204.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White reopened HADOOP-6663:
-------------------------------


Re-opening as there was a compilation problem.

> BlockDecompressorStream get EOF exception when decompressing the file compressed from empty file
> ------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6663
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6663
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Kang Xiao
>            Assignee: Kang Xiao
>             Fix For: 0.22.0
>
>         Attachments: BlockDecompressorStream.java.patch, BlockDecompressorStream.java.patch, BlockDecompressorStream.patch, HADOOP-6663.patch
>
>
> An empty file can be compressed using BlockDecompressorStream, which is for block-based compressiong algorithm such as LZO. However, when decompressing the compressed file, BlockDecompressorStream get EOF exception.
> Here is a typical exception stack:
> java.io.EOFException
> at org.apache.hadoop.io.compress.BlockDecompressorStream.rawReadInt(BlockDecompressorStream.java:125)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:96)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:82)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at java.io.InputStream.read(InputStream.java:85)
> at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:39)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:186)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:170)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:18)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
> at org.apache.hadoop.mapred.Child.main(Child.java:196)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11127-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Oct 25 23:59:20 2010
Return-Path: <common-issues-return-11127-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2293 invoked from network); 25 Oct 2010 23:59:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Oct 2010 23:59:19 -0000
Received: (qmail 62814 invoked by uid 500); 25 Oct 2010 23:59:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62787 invoked by uid 500); 25 Oct 2010 23:59:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62779 invoked by uid 99); 25 Oct 2010 23:59:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 23:59:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Oct 2010 23:59:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9PNwupU000199
	for <common-issues@hadoop.apache.org>; Mon, 25 Oct 2010 23:58:56 GMT
Message-ID: <13255918.69941288051135984.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 19:58:55 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6663) BlockDecompressorStream get EOF
 exception when decompressing the file compressed from empty file
In-Reply-To: <1619883340.543171269850767204.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6663:
------------------------------

    Status: Patch Available  (was: Reopened)

Running updated patch through Hudson.

> BlockDecompressorStream get EOF exception when decompressing the file compressed from empty file
> ------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6663
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6663
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Kang Xiao
>            Assignee: Kang Xiao
>             Fix For: 0.22.0
>
>         Attachments: BlockDecompressorStream.java.patch, BlockDecompressorStream.java.patch, BlockDecompressorStream.patch, HADOOP-6663.patch
>
>
> An empty file can be compressed using BlockDecompressorStream, which is for block-based compressiong algorithm such as LZO. However, when decompressing the compressed file, BlockDecompressorStream get EOF exception.
> Here is a typical exception stack:
> java.io.EOFException
> at org.apache.hadoop.io.compress.BlockDecompressorStream.rawReadInt(BlockDecompressorStream.java:125)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:96)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:82)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at java.io.InputStream.read(InputStream.java:85)
> at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:39)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:186)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:170)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:18)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
> at org.apache.hadoop.mapred.Child.main(Child.java:196)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11128-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 00:10:44 2010
Return-Path: <common-issues-return-11128-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10383 invoked from network); 26 Oct 2010 00:10:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 00:10:43 -0000
Received: (qmail 71728 invoked by uid 500); 26 Oct 2010 00:10:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71684 invoked by uid 500); 26 Oct 2010 00:10:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71676 invoked by uid 99); 26 Oct 2010 00:10:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 00:10:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 00:10:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9Q0AN90000446
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 00:10:23 GMT
Message-ID: <8032189.70181288051823156.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 20:10:23 -0400 (EDT)
From: "Milind Bhandarkar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6985) Suggest that HADOOP_OPTS be
 preserved in hadoop-env.sh.template
In-Reply-To: <26576141.536571286224772607.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924794#action_12924794 ] 

Milind Bhandarkar commented on HADOOP-6985:
-------------------------------------------

sorry, checked commits. looks like the correct version was committed, but comment was not updated.

> Suggest that HADOOP_OPTS be preserved in hadoop-env.sh.template
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6985
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Ramkumar Vadali
>            Assignee: Ramkumar Vadali
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP_6985.2.patch
>
>
> For an administrator who wants to customize HADOOP_OPTS, it would be better to have
> # if [ "$HADOOP_OPTS" == "" ]; then export HADOOP_OPTS=-server; else FOO+=" -server"; fi
> instead of
> # Extra Java runtime options.  Empty by default.
> # export HADOOP_OPTS=-server

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11129-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 00:10:46 2010
Return-Path: <common-issues-return-11129-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10450 invoked from network); 26 Oct 2010 00:10:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 00:10:46 -0000
Received: (qmail 71991 invoked by uid 500); 26 Oct 2010 00:10:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71959 invoked by uid 500); 26 Oct 2010 00:10:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71951 invoked by uid 99); 26 Oct 2010 00:10:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 00:10:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 00:10:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9Q0AMaF000431
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 00:10:22 GMT
Message-ID: <19676258.70131288051822378.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 20:10:22 -0400 (EDT)
From: "Milind Bhandarkar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6985) Suggest that HADOOP_OPTS be
 preserved in hadoop-env.sh.template
In-Reply-To: <26576141.536571286224772607.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924793#action_12924793 ] 

Milind Bhandarkar commented on HADOOP-6985:
-------------------------------------------

so, which version was committed ?

> Suggest that HADOOP_OPTS be preserved in hadoop-env.sh.template
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6985
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Ramkumar Vadali
>            Assignee: Ramkumar Vadali
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP_6985.2.patch
>
>
> For an administrator who wants to customize HADOOP_OPTS, it would be better to have
> # if [ "$HADOOP_OPTS" == "" ]; then export HADOOP_OPTS=-server; else FOO+=" -server"; fi
> instead of
> # Extra Java runtime options.  Empty by default.
> # export HADOOP_OPTS=-server

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11130-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 00:24:53 2010
Return-Path: <common-issues-return-11130-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18887 invoked from network); 26 Oct 2010 00:24:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 00:24:53 -0000
Received: (qmail 83168 invoked by uid 500); 26 Oct 2010 00:24:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83135 invoked by uid 500); 26 Oct 2010 00:24:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83127 invoked by uid 99); 26 Oct 2010 00:24:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 00:24:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 00:24:52 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9Q0OWYO000601
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 00:24:32 GMT
Message-ID: <475562.70241288052672385.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 20:24:32 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7007) update the hudson-test-patch target
 to work with the latest test-patch script.
In-Reply-To: <7411320.67141288042220266.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924796#action_12924796 ] 

Hudson commented on HADOOP-7007:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #399 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/399/])
    HADOOP-7007: Update the hudson-test-patch ant target. Contributed by Giridharan Kesavan.


> update the hudson-test-patch target to work with the latest test-patch script.
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7007
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7007
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7007-build.xml.patch, HADOOP-7007-test-patch.patch, HADOOP-7007.patch
>
>
> The hudson-test-patch target has to be updated to work with the current test-patch.sh script. Since the callback login in the test-patch.sh is removed. by hadoop-7005

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11131-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 00:24:55 2010
Return-Path: <common-issues-return-11131-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18944 invoked from network); 26 Oct 2010 00:24:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 00:24:54 -0000
Received: (qmail 83352 invoked by uid 500); 26 Oct 2010 00:24:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83323 invoked by uid 500); 26 Oct 2010 00:24:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83315 invoked by uid 99); 26 Oct 2010 00:24:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 00:24:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 00:24:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9Q0OY6W000616
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 00:24:34 GMT
Message-ID: <24969424.70291288052674005.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 20:24:34 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6985) Suggest that HADOOP_OPTS be
 preserved in hadoop-env.sh.template
In-Reply-To: <26576141.536571286224772607.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924797#action_12924797 ] 

Hudson commented on HADOOP-6985:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #399 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/399/])
    HADOOP-6985.  Fix example to make more sense.
HADOOP-6985.  Suggest that HADOOP-OPTS be preserved in hadoop-env.sh.template.  Contributed by Ramkumar Vadali.


> Suggest that HADOOP_OPTS be preserved in hadoop-env.sh.template
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6985
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Ramkumar Vadali
>            Assignee: Ramkumar Vadali
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP_6985.2.patch
>
>
> For an administrator who wants to customize HADOOP_OPTS, it would be better to have
> # if [ "$HADOOP_OPTS" == "" ]; then export HADOOP_OPTS=-server; else FOO+=" -server"; fi
> instead of
> # Extra Java runtime options.  Empty by default.
> # export HADOOP_OPTS=-server

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11132-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 00:24:56 2010
Return-Path: <common-issues-return-11132-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18997 invoked from network); 26 Oct 2010 00:24:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 00:24:56 -0000
Received: (qmail 83550 invoked by uid 500); 26 Oct 2010 00:24:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83507 invoked by uid 500); 26 Oct 2010 00:24:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83499 invoked by uid 99); 26 Oct 2010 00:24:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 00:24:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 00:24:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9Q0OVYg000595
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 00:24:31 GMT
Message-ID: <26148436.70221288052671556.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 20:24:31 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6663) BlockDecompressorStream get EOF
 exception when decompressing the file compressed from empty file
In-Reply-To: <1619883340.543171269850767204.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924795#action_12924795 ] 

Hudson commented on HADOOP-6663:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #399 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/399/])
    Reverting HADOOP-6663.
HADOOP-6663.  BlockDecompressorStream get EOF exception when decompressing the file compressed from empty file.  Contributed by Kang Xiao.


> BlockDecompressorStream get EOF exception when decompressing the file compressed from empty file
> ------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6663
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6663
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Kang Xiao
>            Assignee: Kang Xiao
>             Fix For: 0.22.0
>
>         Attachments: BlockDecompressorStream.java.patch, BlockDecompressorStream.java.patch, BlockDecompressorStream.patch, HADOOP-6663.patch
>
>
> An empty file can be compressed using BlockDecompressorStream, which is for block-based compressiong algorithm such as LZO. However, when decompressing the compressed file, BlockDecompressorStream get EOF exception.
> Here is a typical exception stack:
> java.io.EOFException
> at org.apache.hadoop.io.compress.BlockDecompressorStream.rawReadInt(BlockDecompressorStream.java:125)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:96)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:82)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at java.io.InputStream.read(InputStream.java:85)
> at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:39)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:186)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:170)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:18)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
> at org.apache.hadoop.mapred.Child.main(Child.java:196)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11133-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 00:30:46 2010
Return-Path: <common-issues-return-11133-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22207 invoked from network); 26 Oct 2010 00:30:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 00:30:46 -0000
Received: (qmail 89372 invoked by uid 500); 26 Oct 2010 00:30:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89292 invoked by uid 500); 26 Oct 2010 00:30:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89284 invoked by uid 99); 26 Oct 2010 00:30:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 00:30:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 00:30:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9Q0UMTi000724
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 00:30:22 GMT
Message-ID: <2515577.70431288053022168.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 20:30:22 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6317) Serialize the 'final'ness of
 Configuration keys
In-Reply-To: <985698151.1255715791730.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924799#action_12924799 ] 

Hadoop QA commented on HADOOP-6317:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12457858/HADOOP-6317.patch
  against trunk revision 1027316.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 7 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system tests framework.  The patch passed system tests framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/10//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/10//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/10//artifact/trunk/build/test/checkstyle-errors.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/10//console

This message is automatically generated.

> Serialize the 'final'ness of Configuration keys
> -----------------------------------------------
>
>                 Key: HADOOP-6317
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6317
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Arun C Murthy
>            Assignee: Aaron Kimball
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6317.2.patch, HADOOP-6317.3.patch, HADOOP-6317.4.patch, HADOOP-6317.patch, HADOOP-6317.patch
>
>
> Currently Configuration.writeXml doesn't serialize the 'final' attribute. There are some scenarios where it is useful:
> # TaskTracker should mark the tasks' 'localized' parameters as 'final' 
> # GUI tools

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11134-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 00:58:43 2010
Return-Path: <common-issues-return-11134-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28724 invoked from network); 26 Oct 2010 00:58:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 00:58:43 -0000
Received: (qmail 15038 invoked by uid 500); 26 Oct 2010 00:58:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14999 invoked by uid 500); 26 Oct 2010 00:58:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14991 invoked by uid 99); 26 Oct 2010 00:58:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 00:58:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 00:58:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9Q0wMFc001198
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 00:58:22 GMT
Message-ID: <4719108.70641288054702070.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 20:58:22 -0400 (EDT)
From: "Kang Xiao (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6683) the first optimization:
 ZlibCompressor does not fully utilize the buffer
In-Reply-To: <2121624364.706511270518987284.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924803#action_12924803 ] 

Kang Xiao commented on HADOOP-6683:
-----------------------------------

This optimization for zlibcompressor has been deployed in our cluster and works as expected.

Is there any more work needed for this patch to be reviewed or resolved?

> the first optimization: ZlibCompressor does not fully utilize the buffer
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6683
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6683
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Kang Xiao
>         Attachments: ZlibCompressor.java.patch
>
>
> Thanks for Hong Tang's advice.
> Sub task created for the first optimization. HADOOP-6662 closed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11135-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 01:12:42 2010
Return-Path: <common-issues-return-11135-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36915 invoked from network); 26 Oct 2010 01:12:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 01:12:41 -0000
Received: (qmail 27689 invoked by uid 500); 26 Oct 2010 01:12:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27648 invoked by uid 500); 26 Oct 2010 01:12:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27640 invoked by uid 99); 26 Oct 2010 01:12:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 01:12:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 01:12:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9Q1CLKw001417
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 01:12:21 GMT
Message-ID: <30595442.70751288055540985.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 21:12:20 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6947) Kerberos relogin should set
 refreshKrb5Config to true
In-Reply-To: <15312872.90501284007413131.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924804#action_12924804 ] 

Todd Lipcon commented on HADOOP-6947:
-------------------------------------

Ah, I misremembered, I guess we are in the same boat :) Hopefully someone else will pick this up and commit. Thanks Kan!

> Kerberos relogin should set refreshKrb5Config to true
> -----------------------------------------------------
>
>                 Key: HADOOP-6947
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6947
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-6947-branch20.txt, hadoop-6947.txt
>
>
> In working on securing a daemon that uses two different principals from different threads, I found that I wasn't able to login from a second keytab after I'd logged in from the first. This is because we don't set the refreshKrb5Config in the Configuration for the Krb5LoginModule - hence it won't switch over to the correct keytab file if it's different than the first.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11136-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 01:22:43 2010
Return-Path: <common-issues-return-11136-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39630 invoked from network); 26 Oct 2010 01:22:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 01:22:43 -0000
Received: (qmail 35849 invoked by uid 500); 26 Oct 2010 01:22:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35816 invoked by uid 500); 26 Oct 2010 01:22:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35807 invoked by uid 99); 26 Oct 2010 01:22:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 01:22:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 01:22:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9Q1MMFB001565
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 01:22:22 GMT
Message-ID: <24642748.70831288056142170.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 21:22:22 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7007) update the hudson-test-patch target
 to work with the latest test-patch script.
In-Reply-To: <7411320.67141288042220266.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924806#action_12924806 ] 

Hudson commented on HADOOP-7007:
--------------------------------

Integrated in Hadoop-Hdfs-trunk-Commit #418 (See [https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/418/])
    HADOOP-7007: Update the hudson-test-patch ant target. Contributed by Giridharan Kesavan.
HADOOP-7007: Update the hudson-test-patch ant target. Contributed by Giridharan Kesavan.


> update the hudson-test-patch target to work with the latest test-patch script.
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7007
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7007
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7007-build.xml.patch, HADOOP-7007-test-patch.patch, HADOOP-7007.patch
>
>
> The hudson-test-patch target has to be updated to work with the current test-patch.sh script. Since the callback login in the test-patch.sh is removed. by hadoop-7005

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11137-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 01:38:47 2010
Return-Path: <common-issues-return-11137-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42702 invoked from network); 26 Oct 2010 01:38:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 01:38:46 -0000
Received: (qmail 46893 invoked by uid 500); 26 Oct 2010 01:38:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46856 invoked by uid 500); 26 Oct 2010 01:38:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46848 invoked by uid 99); 26 Oct 2010 01:38:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 01:38:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 01:38:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9Q1cM9N003183
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 01:38:23 GMT
Message-ID: <16878480.70941288057102911.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 21:38:22 -0400 (EDT)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-7008) Enable test-patch.sh to have a
 configured number of acceptable findbugs and javadoc warnings
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Enable test-patch.sh to have a configured number of acceptable findbugs and javadoc warnings
--------------------------------------------------------------------------------------------

                 Key: HADOOP-7008
                 URL: https://issues.apache.org/jira/browse/HADOOP-7008
             Project: Hadoop Common
          Issue Type: Improvement
          Components: test
            Reporter: Nigel Daley


test-patch.sh should be able to accept a properties file containing an acceptable number of findbugs and javadoc warnings.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11138-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 01:40:41 2010
Return-Path: <common-issues-return-11138-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42951 invoked from network); 26 Oct 2010 01:40:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 01:40:40 -0000
Received: (qmail 51605 invoked by uid 500); 26 Oct 2010 01:40:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51579 invoked by uid 500); 26 Oct 2010 01:40:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51571 invoked by uid 99); 26 Oct 2010 01:40:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 01:40:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 01:40:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9Q1eJAu003216
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 01:40:20 GMT
Message-ID: <28914818.70981288057219685.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 21:40:19 -0400 (EDT)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7008) Enable test-patch.sh to have a
 configured number of acceptable findbugs and javadoc warnings
In-Reply-To: <16878480.70941288057102911.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nigel Daley updated HADOOP-7008:
--------------------------------

    Attachment: HADOOP-7008.patch

Attaching a patch for review.  Giri,  hoping you can review this.  It hasn't been well tested yet.

> Enable test-patch.sh to have a configured number of acceptable findbugs and javadoc warnings
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7008
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7008
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>            Reporter: Nigel Daley
>         Attachments: HADOOP-7008.patch
>
>
> test-patch.sh should be able to accept a properties file containing an acceptable number of findbugs and javadoc warnings.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11139-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 02:04:44 2010
Return-Path: <common-issues-return-11139-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47144 invoked from network); 26 Oct 2010 02:04:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 02:04:44 -0000
Received: (qmail 69218 invoked by uid 500); 26 Oct 2010 02:04:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69154 invoked by uid 500); 26 Oct 2010 02:04:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69146 invoked by uid 99); 26 Oct 2010 02:04:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 02:04:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 02:04:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9Q24Kc4003622
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 02:04:20 GMT
Message-ID: <26387620.71321288058660456.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 22:04:20 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6991) SequenceFile::Reader discards
 length for files, does not call openFile
In-Reply-To: <22609732.541901286245712924.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924817#action_12924817 ] 

Chris Douglas commented on HADOOP-6991:
---------------------------------------

+1

Ran the {{SequenceFile*}} unit tests in MapReduce, all passed.

> SequenceFile::Reader discards length for files, does not call openFile
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6991
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6991
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Chris Douglas
>            Priority: Minor
>         Attachments: 6991-0.patch, 6991-1.patch, h-6991-2.patch
>
>
> While the sorting and merging in {{SequenceFile}} is deprecated and {{openFile}} is archaic, the semantics should remain consistent.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11140-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 02:32:48 2010
Return-Path: <common-issues-return-11140-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57260 invoked from network); 26 Oct 2010 02:32:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 02:32:48 -0000
Received: (qmail 86444 invoked by uid 500); 26 Oct 2010 02:32:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86303 invoked by uid 500); 26 Oct 2010 02:32:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86295 invoked by uid 99); 26 Oct 2010 02:32:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 02:32:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 02:32:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9Q2WMZW004002
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 02:32:23 GMT
Message-ID: <25269404.71641288060342804.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 22:32:22 -0400 (EDT)
From: "Patrick Kling (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7001) Allow configuration changes without
 restarting configured nodes
In-Reply-To: <33220874.146971287081516580.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Patrick Kling updated HADOOP-7001:
----------------------------------

    Attachment: HADOOP-7001.2.patch

As suggested by Doug, I have renamed the methods in Reconfigurable. I have also renamed ConfigurationChangeException to ReconfigurationException and ConfigurationChangeServlet to ReconfigurationServlet to keep everything consistent.

Please see HDFS-1477 for a patch that makes NameNode Reconfigurable and allows us to change 2 properties.

The servlet now returns error code 500 if a change fails.

A command line tool for changing configuration properties sounds like a good idea but I am not quite sure how to provide a generic version of such a tool. If the command line tool runs locally, very little functionality beyond calling the interface methods would be needed. Maybe it would be best to leave command line utilities to a later JIRA once it becomes more clear if there is a need for additional support.

> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7001
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7001
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Patrick Kling
>            Assignee: Patrick Kling
>         Attachments: HADOOP-7001.2.patch, HADOOP-7001.patch, reconfigurable.patch
>
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
>    void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11141-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 04:11:48 2010
Return-Path: <common-issues-return-11141-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84851 invoked from network); 26 Oct 2010 04:11:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 04:11:48 -0000
Received: (qmail 64711 invoked by uid 500); 26 Oct 2010 04:11:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64558 invoked by uid 500); 26 Oct 2010 04:11:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64545 invoked by uid 99); 26 Oct 2010 04:11:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 04:11:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 04:11:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9Q4BOoJ005728
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 04:11:24 GMT
Message-ID: <5049801.72321288066284092.JavaMail.jira@thor>
Date: Tue, 26 Oct 2010 00:11:24 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7001) Allow configuration changes without
 restarting configured nodes
In-Reply-To: <33220874.146971287081516580.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924832#action_12924832 ] 

Todd Lipcon commented on HADOOP-7001:
-------------------------------------

I'm not sold on the interface being one of "set conf var 'foo' to value 'bar'". Internally, this seems fine, but exposing this as a servlet seems like it will encourage the case where runtime configuration diverges from what's in the XML configuration files on disk.

Could we instead have the servlet/API be more similar to the existing refresh* functions, where it reloads the configuration off disk, and applies a diff? I'm imagining the servlet would show you a "preview" of the diff to be applied, along with a list of warnings for configuration variables that have been changed on disk but are *not* reconfigurable. Then the administrator can hit an "apply" button to reconfigure the settings that are reconfigurable.

> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7001
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7001
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Patrick Kling
>            Assignee: Patrick Kling
>         Attachments: HADOOP-7001.2.patch, HADOOP-7001.patch, reconfigurable.patch
>
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
>    void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11142-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 05:33:54 2010
Return-Path: <common-issues-return-11142-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2104 invoked from network); 26 Oct 2010 05:33:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 05:33:54 -0000
Received: (qmail 3790 invoked by uid 500); 26 Oct 2010 05:33:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3696 invoked by uid 500); 26 Oct 2010 05:33:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3686 invoked by uid 99); 26 Oct 2010 05:33:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 05:33:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 05:33:48 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9Q5XRmi006856
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 05:33:27 GMT
Message-ID: <9618115.73201288071207106.JavaMail.jira@thor>
Date: Tue, 26 Oct 2010 01:33:27 -0400 (EDT)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7001) Allow configuration changes without
 restarting configured nodes
In-Reply-To: <33220874.146971287081516580.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924842#action_12924842 ] 

Alejandro Abdelnur commented on HADOOP-7001:
--------------------------------------------

I'd also prefer a refresh request that forces a reload of the config from disk.

To detect diffs a memory copy of the current config could be used.

Properties in the configuration XML should be flagged as *reconfigurable* (similar to *final*), if a property is not flagged as *reconfigurable* hot changes for that property are ignored and are not communicated via the Property listener (have you considered using the PropertyChangeListener API from the JDK?).




> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7001
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7001
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Patrick Kling
>            Assignee: Patrick Kling
>         Attachments: HADOOP-7001.2.patch, HADOOP-7001.patch, reconfigurable.patch
>
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
>    void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11143-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 13:08:53 2010
Return-Path: <common-issues-return-11143-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9507 invoked from network); 26 Oct 2010 13:08:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 13:08:53 -0000
Received: (qmail 94693 invoked by uid 500); 26 Oct 2010 13:08:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94621 invoked by uid 500); 26 Oct 2010 13:08:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94612 invoked by uid 99); 26 Oct 2010 13:08:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 13:08:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 13:08:51 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9QD8Vd3014357
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 13:08:31 GMT
Message-ID: <8444150.78891288098511020.JavaMail.jira@thor>
Date: Tue, 26 Oct 2010 09:08:31 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7007) update the hudson-test-patch target
 to work with the latest test-patch script.
In-Reply-To: <7411320.67141288042220266.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924956#action_12924956 ] 

Hudson commented on HADOOP-7007:
--------------------------------

Integrated in Hadoop-Common-trunk #493 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/493/])
    HADOOP-7007: Update the hudson-test-patch ant target. Contributed by Giridharan Kesavan.


> update the hudson-test-patch target to work with the latest test-patch script.
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7007
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7007
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7007-build.xml.patch, HADOOP-7007-test-patch.patch, HADOOP-7007.patch
>
>
> The hudson-test-patch target has to be updated to work with the current test-patch.sh script. Since the callback login in the test-patch.sh is removed. by hadoop-7005

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11144-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 13:08:55 2010
Return-Path: <common-issues-return-11144-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9562 invoked from network); 26 Oct 2010 13:08:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 13:08:54 -0000
Received: (qmail 94864 invoked by uid 500); 26 Oct 2010 13:08:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94668 invoked by uid 500); 26 Oct 2010 13:08:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94604 invoked by uid 99); 26 Oct 2010 13:08:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 13:08:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 13:08:50 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9QD8UL3014339
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 13:08:30 GMT
Message-ID: <8519300.78831288098510024.JavaMail.jira@thor>
Date: Tue, 26 Oct 2010 09:08:30 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6663) BlockDecompressorStream get EOF
 exception when decompressing the file compressed from empty file
In-Reply-To: <1619883340.543171269850767204.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924954#action_12924954 ] 

Hudson commented on HADOOP-6663:
--------------------------------

Integrated in Hadoop-Common-trunk #493 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/493/])
    Reverting HADOOP-6663.
HADOOP-6663.  BlockDecompressorStream get EOF exception when decompressing the file compressed from empty file.  Contributed by Kang Xiao.


> BlockDecompressorStream get EOF exception when decompressing the file compressed from empty file
> ------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6663
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6663
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Kang Xiao
>            Assignee: Kang Xiao
>             Fix For: 0.22.0
>
>         Attachments: BlockDecompressorStream.java.patch, BlockDecompressorStream.java.patch, BlockDecompressorStream.patch, HADOOP-6663.patch
>
>
> An empty file can be compressed using BlockDecompressorStream, which is for block-based compressiong algorithm such as LZO. However, when decompressing the compressed file, BlockDecompressorStream get EOF exception.
> Here is a typical exception stack:
> java.io.EOFException
> at org.apache.hadoop.io.compress.BlockDecompressorStream.rawReadInt(BlockDecompressorStream.java:125)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:96)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:82)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at java.io.InputStream.read(InputStream.java:85)
> at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:39)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:186)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:170)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:18)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
> at org.apache.hadoop.mapred.Child.main(Child.java:196)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11145-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 13:08:55 2010
Return-Path: <common-issues-return-11145-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9586 invoked from network); 26 Oct 2010 13:08:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 13:08:54 -0000
Received: (qmail 95042 invoked by uid 500); 26 Oct 2010 13:08:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94867 invoked by uid 500); 26 Oct 2010 13:08:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94624 invoked by uid 99); 26 Oct 2010 13:08:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 13:08:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 13:08:51 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9QD8VYQ014372
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 13:08:31 GMT
Message-ID: <16731606.78941288098511807.JavaMail.jira@thor>
Date: Tue, 26 Oct 2010 09:08:31 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6985) Suggest that HADOOP_OPTS be
 preserved in hadoop-env.sh.template
In-Reply-To: <26576141.536571286224772607.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924957#action_12924957 ] 

Hudson commented on HADOOP-6985:
--------------------------------

Integrated in Hadoop-Common-trunk #493 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/493/])
    HADOOP-6985.  Fix example to make more sense.
HADOOP-6985.  Suggest that HADOOP-OPTS be preserved in hadoop-env.sh.template.  Contributed by Ramkumar Vadali.


> Suggest that HADOOP_OPTS be preserved in hadoop-env.sh.template
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6985
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6985
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Ramkumar Vadali
>            Assignee: Ramkumar Vadali
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP_6985.2.patch
>
>
> For an administrator who wants to customize HADOOP_OPTS, it would be better to have
> # if [ "$HADOOP_OPTS" == "" ]; then export HADOOP_OPTS=-server; else FOO+=" -server"; fi
> instead of
> # Extra Java runtime options.  Empty by default.
> # export HADOOP_OPTS=-server

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11146-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 13:08:55 2010
Return-Path: <common-issues-return-11146-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9604 invoked from network); 26 Oct 2010 13:08:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 13:08:55 -0000
Received: (qmail 95220 invoked by uid 500); 26 Oct 2010 13:08:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95061 invoked by uid 500); 26 Oct 2010 13:08:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94816 invoked by uid 99); 26 Oct 2010 13:08:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 13:08:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 13:08:52 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9QD8UL9014351
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 13:08:30 GMT
Message-ID: <19746442.78871288098510736.JavaMail.jira@thor>
Date: Tue, 26 Oct 2010 09:08:30 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7005) Update test-patch.sh to remove
 callback to Hudson master
In-Reply-To: <17555034.54181287988399911.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924955#action_12924955 ] 

Hudson commented on HADOOP-7005:
--------------------------------

Integrated in Hadoop-Common-trunk #493 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/493/])
    HADOOP-7005: Update test-patch.sh to remove callback to Hudson master. Contributed by nigel.


> Update test-patch.sh to remove callback to Hudson master
> --------------------------------------------------------
>
>                 Key: HADOOP-7005
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7005
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7005.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11147-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 17:06:51 2010
Return-Path: <common-issues-return-11147-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50904 invoked from network); 26 Oct 2010 17:06:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 17:06:51 -0000
Received: (qmail 99934 invoked by uid 500); 26 Oct 2010 17:06:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99892 invoked by uid 500); 26 Oct 2010 17:06:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99884 invoked by uid 99); 26 Oct 2010 17:06:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 17:06:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 17:06:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9QH6QJJ018465
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 17:06:26 GMT
Message-ID: <26268812.82581288112786251.JavaMail.jira@thor>
Date: Tue, 26 Oct 2010 13:06:26 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6947) Kerberos relogin should set
 refreshKrb5Config to true
In-Reply-To: <15312872.90501284007413131.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6947:
------------------------------

       Resolution: Fixed
    Fix Version/s: 0.22.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I've just committed this. Thanks Todd!

> Kerberos relogin should set refreshKrb5Config to true
> -----------------------------------------------------
>
>                 Key: HADOOP-6947
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6947
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6947-branch20.txt, hadoop-6947.txt
>
>
> In working on securing a daemon that uses two different principals from different threads, I found that I wasn't able to login from a second keytab after I'd logged in from the first. This is because we don't set the refreshKrb5Config in the Configuration for the Krb5LoginModule - hence it won't switch over to the correct keytab file if it's different than the first.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11148-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 17:08:47 2010
Return-Path: <common-issues-return-11148-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51436 invoked from network); 26 Oct 2010 17:08:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 17:08:46 -0000
Received: (qmail 8288 invoked by uid 500); 26 Oct 2010 17:08:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8221 invoked by uid 500); 26 Oct 2010 17:08:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8124 invoked by uid 99); 26 Oct 2010 17:08:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 17:08:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 17:08:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9QH8MQ0018502
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 17:08:22 GMT
Message-ID: <29723996.82631288112902783.JavaMail.jira@thor>
Date: Tue, 26 Oct 2010 13:08:22 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6954) Sources JARs are not correctly
 published to the Maven repository
In-Reply-To: <19843693.216391284595892946.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925042#action_12925042 ] 

Tom White commented on HADOOP-6954:
-----------------------------------

Here are the results of running test-patch:

{noformat}
     [exec] -1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
     [exec]                         Please justify why no new tests are needed for this patch.
     [exec]                         Also please list what manual steps were performed to verify this patch.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec] 
     [exec]     +1 system tests framework.  The patch passed system tests framework compile.
{noformat}

> Sources JARs are not correctly published to the Maven repository
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6954
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6954
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.21.1
>
>         Attachments: HADOOP-6954.patch
>
>
> When you try to "close" the staging repository to make it visible to the public the operation fails (see https://issues.apache.org/jira/browse/HDFS-1292?focusedCommentId=12909953&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12909953 for the type of error).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11149-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 17:12:48 2010
Return-Path: <common-issues-return-11149-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53087 invoked from network); 26 Oct 2010 17:12:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 17:12:48 -0000
Received: (qmail 11460 invoked by uid 500); 26 Oct 2010 17:12:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11431 invoked by uid 500); 26 Oct 2010 17:12:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11423 invoked by uid 99); 26 Oct 2010 17:12:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 17:12:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 17:12:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9QHCOBX018577
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 17:12:24 GMT
Message-ID: <10633354.82721288113144094.JavaMail.jira@thor>
Date: Tue, 26 Oct 2010 13:12:24 -0400 (EDT)
From: "Chris Nauroth (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7006) hadoop fs -getmerge does not work
 using codebase from trunk.
In-Reply-To: <21950654.54351287988881239.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925045#action_12925045 ] 

Chris Nauroth commented on HADOOP-7006:
---------------------------------------

Thanks, Aaron and Doug.

I'm following the directions from http://wiki.apache.org/hadoop/HowToContribute .  I don't see the "Submit Patch" link to trigger Hudson though.  Is there something else that I need to do?


> hadoop fs -getmerge does not work using codebase from trunk.
> ------------------------------------------------------------
>
>                 Key: HADOOP-7006
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7006
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7006.patch, HADOOP-7006.patch
>
>
> Running the codebase from trunk, the hadoop fs -getmerge command does not work.  As implemented in prior versions (i.e. 0.20.2), I could run hadoop fs -getmerge pointed at a directory containing multiple files.  It would merge all files into a single file on the local file system.  Running the same command using the codebase from trunk, it looks like nothing happens.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11150-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 17:18:57 2010
Return-Path: <common-issues-return-11150-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59068 invoked from network); 26 Oct 2010 17:18:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 17:18:47 -0000
Received: (qmail 23466 invoked by uid 500); 26 Oct 2010 17:18:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23420 invoked by uid 500); 26 Oct 2010 17:18:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23362 invoked by uid 99); 26 Oct 2010 17:18:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 17:18:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 17:18:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9QHINiu018712
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 17:18:23 GMT
Message-ID: <14989088.82871288113503784.JavaMail.jira@thor>
Date: Tue, 26 Oct 2010 13:18:23 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6954) Sources JARs are not correctly
 published to the Maven repository
In-Reply-To: <19843693.216391284595892946.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6954:
------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I've just committed this.

> Sources JARs are not correctly published to the Maven repository
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6954
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6954
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.21.1
>
>         Attachments: HADOOP-6954.patch
>
>
> When you try to "close" the staging repository to make it visible to the public the operation fails (see https://issues.apache.org/jira/browse/HDFS-1292?focusedCommentId=12909953&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12909953 for the type of error).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11151-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 17:22:46 2010
Return-Path: <common-issues-return-11151-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60953 invoked from network); 26 Oct 2010 17:22:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 17:22:46 -0000
Received: (qmail 27853 invoked by uid 500); 26 Oct 2010 17:22:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27813 invoked by uid 500); 26 Oct 2010 17:22:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27805 invoked by uid 99); 26 Oct 2010 17:22:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 17:22:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 17:22:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9QHMOH0018821
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 17:22:25 GMT
Message-ID: <3514825.83091288113744946.JavaMail.jira@thor>
Date: Tue, 26 Oct 2010 13:22:24 -0400 (EDT)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7006) hadoop fs -getmerge does not work
 using codebase from trunk.
In-Reply-To: <21950654.54351287988881239.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925053#action_12925053 ] 

Aaron T. Myers commented on HADOOP-7006:
----------------------------------------

Doug already marked the ticket patch available.

Not quite sure why Hudson hasn't picked it up yet - might just be taking some time to get around to it (the HDFS tests take quite a while to run.)

> hadoop fs -getmerge does not work using codebase from trunk.
> ------------------------------------------------------------
>
>                 Key: HADOOP-7006
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7006
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7006.patch, HADOOP-7006.patch
>
>
> Running the codebase from trunk, the hadoop fs -getmerge command does not work.  As implemented in prior versions (i.e. 0.20.2), I could run hadoop fs -getmerge pointed at a directory containing multiple files.  It would merge all files into a single file on the local file system.  Running the same command using the codebase from trunk, it looks like nothing happens.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11152-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 17:28:44 2010
Return-Path: <common-issues-return-11152-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65012 invoked from network); 26 Oct 2010 17:28:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 17:28:43 -0000
Received: (qmail 37781 invoked by uid 500); 26 Oct 2010 17:28:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37746 invoked by uid 500); 26 Oct 2010 17:28:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37716 invoked by uid 99); 26 Oct 2010 17:28:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 17:28:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 17:28:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9QHSLeP018982
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 17:28:21 GMT
Message-ID: <26879604.83321288114101567.JavaMail.jira@thor>
Date: Tue, 26 Oct 2010 13:28:21 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6947) Kerberos relogin should set
 refreshKrb5Config to true
In-Reply-To: <15312872.90501284007413131.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925055#action_12925055 ] 

Hudson commented on HADOOP-6947:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #400 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/400/])
    HADOOP-6947.  Kerberos relogin should set refreshKrb5Config to true.  Contributed by Todd Lipcon.


> Kerberos relogin should set refreshKrb5Config to true
> -----------------------------------------------------
>
>                 Key: HADOOP-6947
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6947
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6947-branch20.txt, hadoop-6947.txt
>
>
> In working on securing a daemon that uses two different principals from different threads, I found that I wasn't able to login from a second keytab after I'd logged in from the first. This is because we don't set the refreshKrb5Config in the Configuration for the Krb5LoginModule - hence it won't switch over to the correct keytab file if it's different than the first.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11153-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 17:32:41 2010
Return-Path: <common-issues-return-11153-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70252 invoked from network); 26 Oct 2010 17:32:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 17:32:41 -0000
Received: (qmail 48915 invoked by uid 500); 26 Oct 2010 17:32:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48824 invoked by uid 500); 26 Oct 2010 17:32:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48816 invoked by uid 99); 26 Oct 2010 17:32:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 17:32:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 17:32:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9QHWKm5019333
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 17:32:20 GMT
Message-ID: <31195542.83341288114340062.JavaMail.jira@thor>
Date: Tue, 26 Oct 2010 13:32:20 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6724) IPC doesn't properly handle IOEs
 thrown by socket factory
In-Reply-To: <4486259.186941272248270321.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins resolved HADOOP-6724.
---------------------------------

       Resolution: Fixed
    Fix Version/s: 0.22.0
                   0.20.3

I ran test-patch and committed this to branch 20.

> IPC doesn't properly handle IOEs thrown by socket factory
> ---------------------------------------------------------
>
>                 Key: HADOOP-6724
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6724
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.20.3, 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.20.3, 0.22.0, 0.21.0
>
>         Attachments: hadoop-6724-20-1.patch, hadoop-6724.txt
>
>
> If the socket factory throws an IOE inside setupIOStreams, then handleConnectionFailure will be called with socket still null, and thus generate an NPE on socket.close(). This ends up orphaning clients, etc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11154-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 17:46:41 2010
Return-Path: <common-issues-return-11154-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79867 invoked from network); 26 Oct 2010 17:46:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 17:46:41 -0000
Received: (qmail 75943 invoked by uid 500); 26 Oct 2010 17:46:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75832 invoked by uid 500); 26 Oct 2010 17:46:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75824 invoked by uid 99); 26 Oct 2010 17:46:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 17:46:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 17:46:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9QHkKNS019707
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 17:46:20 GMT
Message-ID: <12905109.83411288115180063.JavaMail.jira@thor>
Date: Tue, 26 Oct 2010 13:46:20 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6954) Sources JARs are not correctly
 published to the Maven repository
In-Reply-To: <19843693.216391284595892946.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925058#action_12925058 ] 

Hudson commented on HADOOP-6954:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #401 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/401/])
    HADOOP-6954.  Sources JARs are not correctly published to the Maven repository.


> Sources JARs are not correctly published to the Maven repository
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6954
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6954
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.21.1
>
>         Attachments: HADOOP-6954.patch
>
>
> When you try to "close" the staging repository to make it visible to the public the operation fails (see https://issues.apache.org/jira/browse/HDFS-1292?focusedCommentId=12909953&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12909953 for the type of error).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11155-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 18:57:45 2010
Return-Path: <common-issues-return-11155-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21684 invoked from network); 26 Oct 2010 18:57:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 18:57:45 -0000
Received: (qmail 94510 invoked by uid 500); 26 Oct 2010 18:57:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94483 invoked by uid 500); 26 Oct 2010 18:57:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94475 invoked by uid 99); 26 Oct 2010 18:57:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 18:57:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 18:57:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9QIvNbn023104
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 18:57:24 GMT
Message-ID: <26457473.85351288119443917.JavaMail.jira@thor>
Date: Tue, 26 Oct 2010 14:57:23 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6757) NullPointerException for hadoop
 clients launched from streaming tasks
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6757:
------------------------------

    Status: Open  (was: Patch Available)

Marking as open while Sharad's comment is addressed.

> NullPointerException for hadoop clients launched from streaming tasks
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6757
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6757
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Amar Kamat
>            Assignee: Amar Kamat
>         Attachments: BZ-3620565-v1.0.patch, HADOOP-6757-v1.0.patch
>
>
> TaskRunner sets HADOOP_ROOT_LOGGER to info,TLA while launching the child tasks. TLA implicitly assumes that that task-id information will be made available via the 'hadoop.tasklog.taskid' parameter. 'hadoop.tasklog.taskid' is passed to the child task by the TaskRunner via HADOOP_CLIENT_OPTS. When the streaming task launches a hadoop client (say hadoop job -list), the HADOOP_ROOT_LOGGER of the hadoop client is set to 'info,TLA' but hadoop.tasklog.taskid is not set resulting into NPE.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11156-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 19:30:44 2010
Return-Path: <common-issues-return-11156-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33637 invoked from network); 26 Oct 2010 19:30:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 19:30:44 -0000
Received: (qmail 64197 invoked by uid 500); 26 Oct 2010 19:30:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64169 invoked by uid 500); 26 Oct 2010 19:30:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64161 invoked by uid 99); 26 Oct 2010 19:30:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 19:30:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 19:30:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9QJUKsQ023711
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 19:30:20 GMT
Message-ID: <13861625.86011288121420391.JavaMail.jira@thor>
Date: Tue, 26 Oct 2010 15:30:20 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7006) hadoop fs -getmerge does not work
 using codebase from trunk.
In-Reply-To: <21950654.54351287988881239.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925099#action_12925099 ] 

Doug Cutting commented on HADOOP-7006:
--------------------------------------

Ran test-patch by hand.

{code}
-1 overall.  

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 system tests framework.  The patch passed system tests framework compile.
{code}

The javadoc test seems broken, as it fails even when I run test-patch with an empty patch file.

> hadoop fs -getmerge does not work using codebase from trunk.
> ------------------------------------------------------------
>
>                 Key: HADOOP-7006
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7006
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7006.patch, HADOOP-7006.patch
>
>
> Running the codebase from trunk, the hadoop fs -getmerge command does not work.  As implemented in prior versions (i.e. 0.20.2), I could run hadoop fs -getmerge pointed at a directory containing multiple files.  It would merge all files into a single file on the local file system.  Running the same command using the codebase from trunk, it looks like nothing happens.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11157-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 19:34:45 2010
Return-Path: <common-issues-return-11157-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34811 invoked from network); 26 Oct 2010 19:34:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 19:34:44 -0000
Received: (qmail 70887 invoked by uid 500); 26 Oct 2010 19:34:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70845 invoked by uid 500); 26 Oct 2010 19:34:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70837 invoked by uid 99); 26 Oct 2010 19:34:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 19:34:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 19:34:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9QJYNFo023765
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 19:34:24 GMT
Message-ID: <12537211.86071288121663946.JavaMail.jira@thor>
Date: Tue, 26 Oct 2010 15:34:23 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6724) IPC doesn't properly handle IOEs
 thrown by socket factory
In-Reply-To: <4486259.186941272248270321.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925101#action_12925101 ] 

Owen O'Malley commented on HADOOP-6724:
---------------------------------------

Why didn't you apply this to 0.21?

> IPC doesn't properly handle IOEs thrown by socket factory
> ---------------------------------------------------------
>
>                 Key: HADOOP-6724
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6724
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.20.3, 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>         Attachments: hadoop-6724-20-1.patch, hadoop-6724.txt
>
>
> If the socket factory throws an IOE inside setupIOStreams, then handleConnectionFailure will be called with socket still null, and thus generate an NPE on socket.close(). This ends up orphaning clients, etc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11158-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 21:17:43 2010
Return-Path: <common-issues-return-11158-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70272 invoked from network); 26 Oct 2010 21:17:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 21:17:42 -0000
Received: (qmail 58060 invoked by uid 500); 26 Oct 2010 21:17:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58029 invoked by uid 500); 26 Oct 2010 21:17:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58021 invoked by uid 99); 26 Oct 2010 21:17:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 21:17:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 21:17:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9QLHLH6025583
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 21:17:22 GMT
Message-ID: <19824466.88161288127841887.JavaMail.jira@thor>
Date: Tue, 26 Oct 2010 17:17:21 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7006) hadoop fs -getmerge does not work
 using codebase from trunk.
In-Reply-To: <21950654.54351287988881239.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-7006:
---------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I just committed this.  Thanks, Chris!

> hadoop fs -getmerge does not work using codebase from trunk.
> ------------------------------------------------------------
>
>                 Key: HADOOP-7006
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7006
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7006.patch, HADOOP-7006.patch
>
>
> Running the codebase from trunk, the hadoop fs -getmerge command does not work.  As implemented in prior versions (i.e. 0.20.2), I could run hadoop fs -getmerge pointed at a directory containing multiple files.  It would merge all files into a single file on the local file system.  Running the same command using the codebase from trunk, it looks like nothing happens.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11159-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 21:27:48 2010
Return-Path: <common-issues-return-11159-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71834 invoked from network); 26 Oct 2010 21:27:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 21:27:48 -0000
Received: (qmail 70689 invoked by uid 500); 26 Oct 2010 21:27:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70655 invoked by uid 500); 26 Oct 2010 21:27:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70647 invoked by uid 99); 26 Oct 2010 21:27:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 21:27:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 21:27:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9QLRO92025730
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 21:27:24 GMT
Message-ID: <246488.88311288128444362.JavaMail.jira@thor>
Date: Tue, 26 Oct 2010 17:27:24 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-7009) MD5Hash provides a public factory
 method that creates an instance of MessageDigest
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

MD5Hash provides a public factory method that creates an instance of MessageDigest
----------------------------------------------------------------------------------

                 Key: HADOOP-7009
                 URL: https://issues.apache.org/jira/browse/HADOOP-7009
             Project: Hadoop Common
          Issue Type: Improvement
          Components: io
    Affects Versions: 0.22.0
            Reporter: Hairong Kuang
            Assignee: Hairong Kuang
             Fix For: 0.22.0


MD5Hash has a private way of creating a MessageDigest object that's thread local. I'd like to have such a method which is public so that checksuming fsimage (HDFS-903) could use it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11160-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 21:33:42 2010
Return-Path: <common-issues-return-11160-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73786 invoked from network); 26 Oct 2010 21:33:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 21:33:42 -0000
Received: (qmail 78398 invoked by uid 500); 26 Oct 2010 21:33:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78240 invoked by uid 500); 26 Oct 2010 21:33:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78232 invoked by uid 99); 26 Oct 2010 21:33:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 21:33:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 21:33:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9QLXM0I025823
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 21:33:22 GMT
Message-ID: <26577612.88391288128801994.JavaMail.jira@thor>
Date: Tue, 26 Oct 2010 17:33:21 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7009) MD5Hash provides a public factory
 method that creates an instance of MessageDigest
In-Reply-To: <246488.88311288128444362.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-7009:
----------------------------------

    Attachment: md5hash.patch

Here is a simple patch that does this.

> MD5Hash provides a public factory method that creates an instance of MessageDigest
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7009
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7009
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: md5hash.patch
>
>
> MD5Hash has a private way of creating a MessageDigest object that's thread local. I'd like to have such a method which is public so that checksuming fsimage (HDFS-903) could use it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11161-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 21:45:48 2010
Return-Path: <common-issues-return-11161-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76868 invoked from network); 26 Oct 2010 21:45:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 21:45:48 -0000
Received: (qmail 95558 invoked by uid 500); 26 Oct 2010 21:45:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95526 invoked by uid 500); 26 Oct 2010 21:45:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95518 invoked by uid 99); 26 Oct 2010 21:45:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 21:45:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 21:45:48 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9QLjRlM026140
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 21:45:27 GMT
Message-ID: <11286474.88981288129527669.JavaMail.jira@thor>
Date: Tue, 26 Oct 2010 17:45:27 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7006) hadoop fs -getmerge does not work
 using codebase from trunk.
In-Reply-To: <21950654.54351287988881239.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925159#action_12925159 ] 

Hudson commented on HADOOP-7006:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #402 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/402/])
    HADOOP-7006. Fix 'fs -getmerge' command to not be a no-op.  Contributed by Chris Nauroth.


> hadoop fs -getmerge does not work using codebase from trunk.
> ------------------------------------------------------------
>
>                 Key: HADOOP-7006
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7006
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7006.patch, HADOOP-7006.patch
>
>
> Running the codebase from trunk, the hadoop fs -getmerge command does not work.  As implemented in prior versions (i.e. 0.20.2), I could run hadoop fs -getmerge pointed at a directory containing multiple files.  It would merge all files into a single file on the local file system.  Running the same command using the codebase from trunk, it looks like nothing happens.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11162-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Oct 26 21:53:46 2010
Return-Path: <common-issues-return-11162-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80008 invoked from network); 26 Oct 2010 21:53:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Oct 2010 21:53:45 -0000
Received: (qmail 10436 invoked by uid 500); 26 Oct 2010 21:53:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10403 invoked by uid 500); 26 Oct 2010 21:53:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10395 invoked by uid 99); 26 Oct 2010 21:53:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 21:53:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Oct 2010 21:53:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9QLrMXD026288
	for <common-issues@hadoop.apache.org>; Tue, 26 Oct 2010 21:53:24 GMT
Message-ID: <7608273.89221288130002110.JavaMail.jira@thor>
Date: Tue, 26 Oct 2010 17:53:22 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6724) IPC doesn't properly handle IOEs
 thrown by socket factory
In-Reply-To: <4486259.186941272248270321.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925163#action_12925163 ] 

Eli Collins commented on HADOOP-6724:
-------------------------------------

Because it was already committed to the 21 branch.

> IPC doesn't properly handle IOEs thrown by socket factory
> ---------------------------------------------------------
>
>                 Key: HADOOP-6724
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6724
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.20.3, 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>         Attachments: hadoop-6724-20-1.patch, hadoop-6724.txt
>
>
> If the socket factory throws an IOE inside setupIOStreams, then handleConnectionFailure will be called with socket still null, and thus generate an NPE on socket.close(). This ends up orphaning clients, etc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11163-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 11:23:47 2010
Return-Path: <common-issues-return-11163-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55502 invoked from network); 27 Oct 2010 11:23:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 11:23:47 -0000
Received: (qmail 45139 invoked by uid 500); 27 Oct 2010 11:23:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44954 invoked by uid 500); 27 Oct 2010 11:23:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44945 invoked by uid 99); 27 Oct 2010 11:23:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 11:23:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 11:23:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9RBNN92009556
	for <common-issues@hadoop.apache.org>; Wed, 27 Oct 2010 11:23:23 GMT
Message-ID: <9894942.97691288178603234.JavaMail.jira@thor>
Date: Wed, 27 Oct 2010 07:23:23 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6947) Kerberos relogin should set
 refreshKrb5Config to true
In-Reply-To: <15312872.90501284007413131.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925328#action_12925328 ] 

Hudson commented on HADOOP-6947:
--------------------------------

Integrated in Hadoop-Common-trunk #494 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/494/])
    HADOOP-6947.  Kerberos relogin should set refreshKrb5Config to true.  Contributed by Todd Lipcon.


> Kerberos relogin should set refreshKrb5Config to true
> -----------------------------------------------------
>
>                 Key: HADOOP-6947
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6947
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6947-branch20.txt, hadoop-6947.txt
>
>
> In working on securing a daemon that uses two different principals from different threads, I found that I wasn't able to login from a second keytab after I'd logged in from the first. This is because we don't set the refreshKrb5Config in the Configuration for the Krb5LoginModule - hence it won't switch over to the correct keytab file if it's different than the first.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11164-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 11:23:48 2010
Return-Path: <common-issues-return-11164-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55540 invoked from network); 27 Oct 2010 11:23:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 11:23:48 -0000
Received: (qmail 45187 invoked by uid 500); 27 Oct 2010 11:23:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44976 invoked by uid 500); 27 Oct 2010 11:23:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44963 invoked by uid 99); 27 Oct 2010 11:23:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 11:23:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 11:23:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9RBNOjL009574
	for <common-issues@hadoop.apache.org>; Wed, 27 Oct 2010 11:23:24 GMT
Message-ID: <6529809.97751288178604903.JavaMail.jira@thor>
Date: Wed, 27 Oct 2010 07:23:24 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7006) hadoop fs -getmerge does not work
 using codebase from trunk.
In-Reply-To: <21950654.54351287988881239.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925330#action_12925330 ] 

Hudson commented on HADOOP-7006:
--------------------------------

Integrated in Hadoop-Common-trunk #494 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/494/])
    HADOOP-7006. Fix 'fs -getmerge' command to not be a no-op.  Contributed by Chris Nauroth.


> hadoop fs -getmerge does not work using codebase from trunk.
> ------------------------------------------------------------
>
>                 Key: HADOOP-7006
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7006
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7006.patch, HADOOP-7006.patch
>
>
> Running the codebase from trunk, the hadoop fs -getmerge command does not work.  As implemented in prior versions (i.e. 0.20.2), I could run hadoop fs -getmerge pointed at a directory containing multiple files.  It would merge all files into a single file on the local file system.  Running the same command using the codebase from trunk, it looks like nothing happens.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11165-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 11:23:48 2010
Return-Path: <common-issues-return-11165-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55550 invoked from network); 27 Oct 2010 11:23:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 11:23:48 -0000
Received: (qmail 45283 invoked by uid 500); 27 Oct 2010 11:23:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45154 invoked by uid 500); 27 Oct 2010 11:23:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45145 invoked by uid 99); 27 Oct 2010 11:23:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 11:23:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 11:23:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9RBNNqM009562
	for <common-issues@hadoop.apache.org>; Wed, 27 Oct 2010 11:23:23 GMT
Message-ID: <18595872.97711288178603714.JavaMail.jira@thor>
Date: Wed, 27 Oct 2010 07:23:23 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6954) Sources JARs are not correctly
 published to the Maven repository
In-Reply-To: <19843693.216391284595892946.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925329#action_12925329 ] 

Hudson commented on HADOOP-6954:
--------------------------------

Integrated in Hadoop-Common-trunk #494 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/494/])
    HADOOP-6954.  Sources JARs are not correctly published to the Maven repository.


> Sources JARs are not correctly published to the Maven repository
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6954
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6954
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.21.1
>
>         Attachments: HADOOP-6954.patch
>
>
> When you try to "close" the staging repository to make it visible to the public the operation fails (see https://issues.apache.org/jira/browse/HDFS-1292?focusedCommentId=12909953&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12909953 for the type of error).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11168-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 16:41:37 2010
Return-Path: <common-issues-return-11168-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1410 invoked from network); 27 Oct 2010 16:41:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 16:41:05 -0000
Received: (qmail 77904 invoked by uid 500); 27 Oct 2010 16:35:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74526 invoked by uid 500); 27 Oct 2010 16:35:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64997 invoked by uid 99); 27 Oct 2010 15:34:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 15:34:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 15:34:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9RFYLmm018724
	for <common-issues@hadoop.apache.org>; Wed, 27 Oct 2010 15:34:21 GMT
Message-ID: <18878677.101321288193661432.JavaMail.jira@thor>
Date: Wed, 27 Oct 2010 11:34:21 -0400 (EDT)
From: "Jingguo Yao (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7010) Typo in FileSystem.java
In-Reply-To: <26367340.101261288193659612.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jingguo Yao updated HADOOP-7010:
--------------------------------

    Attachment: HADOOP-7010.patch

Patch available.

> Typo in FileSystem.java
> -----------------------
>
>                 Key: HADOOP-7010
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7010
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Jingguo Yao
>            Priority: Minor
>         Attachments: HADOOP-7010.patch
>
>   Original Estimate: 0.08h
>  Remaining Estimate: 0.08h
>
> For the Javadoc for getLocal method, "Get the local file syste" should be "Get the local file system.".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11166-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 16:41:46 2010
Return-Path: <common-issues-return-11166-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2313 invoked from network); 27 Oct 2010 16:41:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 16:41:13 -0000
Received: (qmail 78652 invoked by uid 500); 27 Oct 2010 16:35:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67736 invoked by uid 500); 27 Oct 2010 16:33:25 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65127 invoked by uid 99); 27 Oct 2010 15:36:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 15:36:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 15:36:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9RFaK69018757
	for <common-issues@hadoop.apache.org>; Wed, 27 Oct 2010 15:36:21 GMT
Message-ID: <14731951.101341288193780815.JavaMail.jira@thor>
Date: Wed, 27 Oct 2010 11:36:20 -0400 (EDT)
From: "Jingguo Yao (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7010) Typo in FileSystem.java
In-Reply-To: <26367340.101261288193659612.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jingguo Yao updated HADOOP-7010:
--------------------------------

    Status: Patch Available  (was: Open)

> Typo in FileSystem.java
> -----------------------
>
>                 Key: HADOOP-7010
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7010
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Jingguo Yao
>            Priority: Minor
>         Attachments: HADOOP-7010.patch
>
>   Original Estimate: 0.08h
>  Remaining Estimate: 0.08h
>
> For the Javadoc for getLocal method, "Get the local file syste" should be "Get the local file system.".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11167-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 16:41:48 2010
Return-Path: <common-issues-return-11167-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2649 invoked from network); 27 Oct 2010 16:41:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 16:41:17 -0000
Received: (qmail 78869 invoked by uid 500); 27 Oct 2010 16:36:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71330 invoked by uid 500); 27 Oct 2010 16:34:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64993 invoked by uid 99); 27 Oct 2010 15:34:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 15:34:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 15:34:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9RFYJ6U018706
	for <common-issues@hadoop.apache.org>; Wed, 27 Oct 2010 15:34:19 GMT
Message-ID: <26367340.101261288193659612.JavaMail.jira@thor>
Date: Wed, 27 Oct 2010 11:34:19 -0400 (EDT)
From: "Jingguo Yao (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-7010) Typo in FileSystem.java
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Typo in FileSystem.java
-----------------------

                 Key: HADOOP-7010
                 URL: https://issues.apache.org/jira/browse/HADOOP-7010
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs
    Affects Versions: 0.21.0
            Reporter: Jingguo Yao
            Priority: Minor
         Attachments: HADOOP-7010.patch

For the Javadoc for getLocal method, "Get the local file syste" should be "Get the local file system.".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11169-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 19:45:46 2010
Return-Path: <common-issues-return-11169-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16125 invoked from network); 27 Oct 2010 19:45:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 19:45:45 -0000
Received: (qmail 19590 invoked by uid 500); 27 Oct 2010 19:45:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19550 invoked by uid 500); 27 Oct 2010 19:45:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19520 invoked by uid 99); 27 Oct 2010 19:45:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 19:45:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 19:45:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9RJjKMG022931
	for <common-issues@hadoop.apache.org>; Wed, 27 Oct 2010 19:45:20 GMT
Message-ID: <2032276.105871288208720568.JavaMail.jira@thor>
Date: Wed, 27 Oct 2010 15:45:20 -0400 (EDT)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-7011) KerberosName.main(...) throws NPE
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

KerberosName.main(...) throws NPE
---------------------------------

                 Key: HADOOP-7011
                 URL: https://issues.apache.org/jira/browse/HADOOP-7011
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.22.0
            Reporter: Aaron T. Myers
            Assignee: Aaron T. Myers


The main method of KerberosName attempts to do short name translation before calling KerberosName.setConfiguration(...).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11170-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 21:56:44 2010
Return-Path: <common-issues-return-11170-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60118 invoked from network); 27 Oct 2010 21:56:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 21:56:44 -0000
Received: (qmail 11532 invoked by uid 500); 27 Oct 2010 21:56:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11499 invoked by uid 500); 27 Oct 2010 21:56:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11491 invoked by uid 99); 27 Oct 2010 21:56:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 21:56:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 21:56:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9RLuKot024940
	for <common-issues@hadoop.apache.org>; Wed, 27 Oct 2010 21:56:20 GMT
Message-ID: <24875557.107991288216580255.JavaMail.jira@thor>
Date: Wed, 27 Oct 2010 17:56:20 -0400 (EDT)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7011) KerberosName.main(...) throws NPE
In-Reply-To: <2032276.105871288208720568.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7011:
-----------------------------------

    Attachment: hadoop-7011.0.txt

> KerberosName.main(...) throws NPE
> ---------------------------------
>
>                 Key: HADOOP-7011
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7011
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7011.0.txt
>
>
> The main method of KerberosName attempts to do short name translation before calling KerberosName.setConfiguration(...).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11171-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Oct 27 21:56:44 2010
Return-Path: <common-issues-return-11171-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60150 invoked from network); 27 Oct 2010 21:56:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Oct 2010 21:56:44 -0000
Received: (qmail 11726 invoked by uid 500); 27 Oct 2010 21:56:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11678 invoked by uid 500); 27 Oct 2010 21:56:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11670 invoked by uid 99); 27 Oct 2010 21:56:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 21:56:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Oct 2010 21:56:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9RLuKdP024949
	for <common-issues@hadoop.apache.org>; Wed, 27 Oct 2010 21:56:20 GMT
Message-ID: <29290182.108021288216580868.JavaMail.jira@thor>
Date: Wed, 27 Oct 2010 17:56:20 -0400 (EDT)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7011) KerberosName.main(...) throws NPE
In-Reply-To: <2032276.105871288208720568.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7011:
-----------------------------------

    Fix Version/s: 0.22.0
           Status: Patch Available  (was: Open)

> KerberosName.main(...) throws NPE
> ---------------------------------
>
>                 Key: HADOOP-7011
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7011
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7011.0.txt
>
>
> The main method of KerberosName attempts to do short name translation before calling KerberosName.setConfiguration(...).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11172-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 28 01:55:44 2010
Return-Path: <common-issues-return-11172-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62732 invoked from network); 28 Oct 2010 01:55:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Oct 2010 01:55:44 -0000
Received: (qmail 45867 invoked by uid 500); 28 Oct 2010 01:55:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45721 invoked by uid 500); 28 Oct 2010 01:55:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45713 invoked by uid 99); 28 Oct 2010 01:55:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 01:55:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 01:55:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9S1tOOp003376
	for <common-issues@hadoop.apache.org>; Thu, 28 Oct 2010 01:55:24 GMT
Message-ID: <29727295.111011288230924025.JavaMail.jira@thor>
Date: Wed, 27 Oct 2010 21:55:24 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7010) Typo in FileSystem.java
In-Reply-To: <26367340.101261288193659612.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7010:
--------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

> Typo in FileSystem.java
> -----------------------
>
>                 Key: HADOOP-7010
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7010
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Jingguo Yao
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7010.patch
>
>   Original Estimate: 0.08h
>  Remaining Estimate: 0.08h
>
> For the Javadoc for getLocal method, "Get the local file syste" should be "Get the local file system.".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11173-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 28 02:02:29 2010
Return-Path: <common-issues-return-11173-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65350 invoked from network); 28 Oct 2010 02:02:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Oct 2010 02:02:28 -0000
Received: (qmail 47126 invoked by uid 500); 28 Oct 2010 01:55:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47060 invoked by uid 500); 28 Oct 2010 01:55:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46123 invoked by uid 99); 28 Oct 2010 01:55:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 01:55:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 01:55:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9S1tNHn003370
	for <common-issues@hadoop.apache.org>; Thu, 28 Oct 2010 01:55:23 GMT
Message-ID: <21113646.110991288230923478.JavaMail.jira@thor>
Date: Wed, 27 Oct 2010 21:55:23 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7010) Typo in FileSystem.java
In-Reply-To: <26367340.101261288193659612.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7010:
--------------------------------

    Fix Version/s: 0.22.0
     Hadoop Flags: [Reviewed]

+1

I've committed this to trunk. Thanks Jingguo.

> Typo in FileSystem.java
> -----------------------
>
>                 Key: HADOOP-7010
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7010
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Jingguo Yao
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7010.patch
>
>   Original Estimate: 0.08h
>  Remaining Estimate: 0.08h
>
> For the Javadoc for getLocal method, "Get the local file syste" should be "Get the local file system.".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11174-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 28 02:25:41 2010
Return-Path: <common-issues-return-11174-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74794 invoked from network); 28 Oct 2010 02:25:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Oct 2010 02:25:41 -0000
Received: (qmail 71699 invoked by uid 500); 28 Oct 2010 02:25:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71671 invoked by uid 500); 28 Oct 2010 02:25:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71663 invoked by uid 99); 28 Oct 2010 02:25:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 02:25:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 02:25:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9S2PKJn021120
	for <common-issues@hadoop.apache.org>; Thu, 28 Oct 2010 02:25:20 GMT
Message-ID: <1219800.111341288232720401.JavaMail.jira@thor>
Date: Wed, 27 Oct 2010 22:25:20 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7009) MD5Hash provides a public factory
 method that creates an instance of MessageDigest
In-Reply-To: <246488.88311288128444362.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925659#action_12925659 ] 

Eli Collins commented on HADOOP-7009:
-------------------------------------

+1

Nit: getDigester needs a javadoc, and perhaps a name that suggests it returns a thread local?

> MD5Hash provides a public factory method that creates an instance of MessageDigest
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7009
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7009
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: md5hash.patch
>
>
> MD5Hash has a private way of creating a MessageDigest object that's thread local. I'd like to have such a method which is public so that checksuming fsimage (HDFS-903) could use it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11175-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 28 02:25:42 2010
Return-Path: <common-issues-return-11175-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74813 invoked from network); 28 Oct 2010 02:25:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Oct 2010 02:25:41 -0000
Received: (qmail 71957 invoked by uid 500); 28 Oct 2010 02:25:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71851 invoked by uid 500); 28 Oct 2010 02:25:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71810 invoked by uid 99); 28 Oct 2010 02:25:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 02:25:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 02:25:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9S2PL5G021138
	for <common-issues@hadoop.apache.org>; Thu, 28 Oct 2010 02:25:21 GMT
Message-ID: <29264328.111361288232721110.JavaMail.jira@thor>
Date: Wed, 27 Oct 2010 22:25:21 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7010) Typo in FileSystem.java
In-Reply-To: <26367340.101261288193659612.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925660#action_12925660 ] 

Hudson commented on HADOOP-7010:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #403 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/403/])
    HADOOP-7010. Typo in FileSystem.java. Contributed by Jingguo Yao.


> Typo in FileSystem.java
> -----------------------
>
>                 Key: HADOOP-7010
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7010
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Jingguo Yao
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7010.patch
>
>   Original Estimate: 0.08h
>  Remaining Estimate: 0.08h
>
> For the Javadoc for getLocal method, "Get the local file syste" should be "Get the local file system.".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11176-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 28 04:27:50 2010
Return-Path: <common-issues-return-11176-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28250 invoked from network); 28 Oct 2010 04:27:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Oct 2010 04:27:50 -0000
Received: (qmail 47629 invoked by uid 500); 28 Oct 2010 04:27:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47470 invoked by uid 500); 28 Oct 2010 04:27:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47462 invoked by uid 99); 28 Oct 2010 04:27:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 04:27:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 04:27:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9S4RP9B029210
	for <common-issues@hadoop.apache.org>; Thu, 28 Oct 2010 04:27:25 GMT
Message-ID: <21914114.112221288240045251.JavaMail.jira@thor>
Date: Thu, 28 Oct 2010 00:27:25 -0400 (EDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7001) Allow configuration changes without
 restarting configured nodes
In-Reply-To: <33220874.146971287081516580.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925684#action_12925684 ] 

dhruba borthakur commented on HADOOP-7001:
------------------------------------------

I too like the idea of having "bin/hadoop dfs -refreshConfig" that will reload the config from the conf file, diff the newly read ones with the ones in the NameNode. 
If the only ones that are different are Recofinfigurable, then the refreshConfig command will be successful, otherwise it is an error. The servlet could take a similar approach.

> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7001
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7001
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Patrick Kling
>            Assignee: Patrick Kling
>         Attachments: HADOOP-7001.2.patch, HADOOP-7001.patch, reconfigurable.patch
>
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
>    void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11177-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 28 04:47:42 2010
Return-Path: <common-issues-return-11177-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33750 invoked from network); 28 Oct 2010 04:47:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Oct 2010 04:47:42 -0000
Received: (qmail 56501 invoked by uid 500); 28 Oct 2010 04:47:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56342 invoked by uid 500); 28 Oct 2010 04:47:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56334 invoked by uid 99); 28 Oct 2010 04:47:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 04:47:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 04:47:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9S4lKuB029323
	for <common-issues@hadoop.apache.org>; Thu, 28 Oct 2010 04:47:20 GMT
Message-ID: <9693068.112321288241240245.JavaMail.jira@thor>
Date: Thu, 28 Oct 2010 00:47:20 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7008) Enable test-patch.sh to have a
 configured number of acceptable findbugs and javadoc warnings
In-Reply-To: <16878480.70941288057102911.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925687#action_12925687 ] 

Giridharan Kesavan commented on HADOOP-7008:
--------------------------------------------

comments:

1) we should add OK_RELEASEAUDIT_WARNINGS to the properties file.. 

2) Every project should define these numbers even if the project has zero warnings for findbugs/releaseaudit/javadoc. Thatway we dont have to pre-build the trunk to determine the numbers.

3) HADOOP-7008.patch patch seem to run findbug on trunk if the OK_FINDBUGS_WARNINGS is not defined or the warnings not equal to zero.
Is there any reason why we are running findbugs if its already defined?  

I think all the projects should just define the values for these 3 properties. test-patch.sh should directly apply the patch and determine the warnings and do a +1 or -1 (if the numbers are greater than the numbers defined in the properties file.) 



> Enable test-patch.sh to have a configured number of acceptable findbugs and javadoc warnings
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7008
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7008
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>            Reporter: Nigel Daley
>         Attachments: HADOOP-7008.patch
>
>
> test-patch.sh should be able to accept a properties file containing an acceptable number of findbugs and javadoc warnings.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11178-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 28 05:46:27 2010
Return-Path: <common-issues-return-11178-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50767 invoked from network); 28 Oct 2010 05:46:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Oct 2010 05:46:27 -0000
Received: (qmail 86289 invoked by uid 500); 28 Oct 2010 05:46:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86132 invoked by uid 500); 28 Oct 2010 05:46:26 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86124 invoked by uid 99); 28 Oct 2010 05:46:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 05:46:26 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 05:46:25 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9S5k5SW029802
	for <common-issues@hadoop.apache.org>; Thu, 28 Oct 2010 05:46:05 GMT
Message-ID: <29379978.113071288244765262.JavaMail.jira@thor>
Date: Thu, 28 Oct 2010 01:46:05 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7009) MD5Hash provides a public factory
 method that creates an instance of MessageDigest
In-Reply-To: <246488.88311288128444362.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-7009:
----------------------------------

    Attachment: md5hash1.patch

Eli, thanks for your review! This patch adds javadoc to the method.

> MD5Hash provides a public factory method that creates an instance of MessageDigest
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7009
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7009
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: md5hash.patch, md5hash1.patch
>
>
> MD5Hash has a private way of creating a MessageDigest object that's thread local. I'd like to have such a method which is public so that checksuming fsimage (HDFS-903) could use it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11179-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 28 06:24:31 2010
Return-Path: <common-issues-return-11179-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72803 invoked from network); 28 Oct 2010 06:24:31 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Oct 2010 06:24:31 -0000
Received: (qmail 23470 invoked by uid 500); 28 Oct 2010 06:24:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23308 invoked by uid 500); 28 Oct 2010 06:24:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23296 invoked by uid 99); 28 Oct 2010 06:24:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 06:24:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 06:24:27 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9S6O5eR001029
	for <common-issues@hadoop.apache.org>; Thu, 28 Oct 2010 06:24:05 GMT
Message-ID: <18402718.116331288247045737.JavaMail.jira@thor>
Date: Thu, 28 Oct 2010 02:24:05 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7009) MD5Hash provides a public factory
 method that creates an instance of MessageDigest
In-Reply-To: <246488.88311288128444362.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925707#action_12925707 ] 

Hairong Kuang commented on HADOOP-7009:
---------------------------------------

     [exec] -1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     -1 tests included.  The patch doesn't appe
     [exec] ar to include any new or modified tests.
     [exec]                         Please justify why no new tests are needed for this patch.
     [exec]                         Also please list what manual steps were performed to verify this patch.
     [exec] 
     [exec]     -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec] 
     [exec]     +1 system tests framework.  The patch passed system tests framework compile.
     [exec] 


> MD5Hash provides a public factory method that creates an instance of MessageDigest
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7009
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7009
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: md5hash.patch, md5hash1.patch
>
>
> MD5Hash has a private way of creating a MessageDigest object that's thread local. I'd like to have such a method which is public so that checksuming fsimage (HDFS-903) could use it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11180-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 28 06:52:30 2010
Return-Path: <common-issues-return-11180-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80758 invoked from network); 28 Oct 2010 06:52:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Oct 2010 06:52:30 -0000
Received: (qmail 52099 invoked by uid 500); 28 Oct 2010 06:52:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51941 invoked by uid 500); 28 Oct 2010 06:52:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51933 invoked by uid 99); 28 Oct 2010 06:52:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 06:52:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 06:52:27 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9S6q5au001294
	for <common-issues@hadoop.apache.org>; Thu, 28 Oct 2010 06:52:05 GMT
Message-ID: <26358414.116861288248725215.JavaMail.jira@thor>
Date: Thu, 28 Oct 2010 02:52:05 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7009) MD5Hash provides a public factory
 method that creates an instance of MessageDigest
In-Reply-To: <246488.88311288128444362.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925722#action_12925722 ] 

Hairong Kuang commented on HADOOP-7009:
---------------------------------------

All unit tests were passed. All 6 javadoc warnings are security-related and are unrelated to this patch. The patch is too trivial to have a unit test. I will commit it.

> MD5Hash provides a public factory method that creates an instance of MessageDigest
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7009
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7009
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: md5hash.patch, md5hash1.patch
>
>
> MD5Hash has a private way of creating a MessageDigest object that's thread local. I'd like to have such a method which is public so that checksuming fsimage (HDFS-903) could use it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11181-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 28 08:44:28 2010
Return-Path: <common-issues-return-11181-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35941 invoked from network); 28 Oct 2010 08:44:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Oct 2010 08:44:28 -0000
Received: (qmail 69750 invoked by uid 500); 28 Oct 2010 08:44:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69644 invoked by uid 500); 28 Oct 2010 08:44:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69636 invoked by uid 99); 28 Oct 2010 08:44:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 08:44:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 08:44:27 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9S8i616002272
	for <common-issues@hadoop.apache.org>; Thu, 28 Oct 2010 08:44:06 GMT
Message-ID: <20466061.118031288255446457.JavaMail.jira@thor>
Date: Thu, 28 Oct 2010 04:44:06 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7008) Enable test-patch.sh to have a
 configured number of acceptable findbugs and javadoc warnings
In-Reply-To: <16878480.70941288057102911.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-7008:
---------------------------------------

    Attachment: HADOOP-7008-V1.patch

V1 patch address the comments.

> Enable test-patch.sh to have a configured number of acceptable findbugs and javadoc warnings
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7008
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7008
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>            Reporter: Nigel Daley
>         Attachments: HADOOP-7008-V1.patch, HADOOP-7008.patch
>
>
> test-patch.sh should be able to accept a properties file containing an acceptable number of findbugs and javadoc warnings.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11182-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 28 09:56:46 2010
Return-Path: <common-issues-return-11182-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75075 invoked from network); 28 Oct 2010 09:56:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Oct 2010 09:56:46 -0000
Received: (qmail 60226 invoked by uid 500); 28 Oct 2010 09:56:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60174 invoked by uid 500); 28 Oct 2010 09:56:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60166 invoked by uid 99); 28 Oct 2010 09:56:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 09:56:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 09:56:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9S9uLkn003098
	for <common-issues@hadoop.apache.org>; Thu, 28 Oct 2010 09:56:22 GMT
Message-ID: <18244450.119111288259781768.JavaMail.jira@thor>
Date: Thu, 28 Oct 2010 05:56:21 -0400 (EDT)
From: "tom liu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-7012) IPC.Client may have some connection
 for each a given host/port
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

IPC.Client may have some connection for each a given host/port
--------------------------------------------------------------

                 Key: HADOOP-7012
                 URL: https://issues.apache.org/jira/browse/HADOOP-7012
             Project: Hadoop Common
          Issue Type: Improvement
          Components: ipc
         Environment: JDK1.6.0_17/Tomcat6
            Reporter: tom liu


ipc.Client class holds connection pool, that connection would be reused for a given host/port.
and WritableRpcEngine holds Client pool, that client would be reused for a given SocketFactory.
so, RPC.getProxy method returns proxy object, that only hold one connection or socket connected to a host/port, if or not, we create many proxy objects.

if large requests are sent to RPC Server through proxy, the request would be wait long time to be sent.

i think, for each host/port, ipc.Client would hold many connections.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11183-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 28 09:58:47 2010
Return-Path: <common-issues-return-11183-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75405 invoked from network); 28 Oct 2010 09:58:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Oct 2010 09:58:46 -0000
Received: (qmail 64358 invoked by uid 500); 28 Oct 2010 09:58:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64161 invoked by uid 500); 28 Oct 2010 09:58:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63906 invoked by uid 99); 28 Oct 2010 09:58:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 09:58:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 09:58:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9S9wNpi003112
	for <common-issues@hadoop.apache.org>; Thu, 28 Oct 2010 09:58:23 GMT
Message-ID: <29319698.119141288259903361.JavaMail.jira@thor>
Date: Thu, 28 Oct 2010 05:58:23 -0400 (EDT)
From: "tom liu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7012) IPC.Client may have some connections
 for each given host/port
In-Reply-To: <18244450.119111288259781768.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

tom liu updated HADOOP-7012:
----------------------------

    Summary: IPC.Client may have some connections for each given host/port  (was: IPC.Client may have some connection for each a given host/port)

> IPC.Client may have some connections for each given host/port
> -------------------------------------------------------------
>
>                 Key: HADOOP-7012
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7012
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>         Environment: JDK1.6.0_17/Tomcat6
>            Reporter: tom liu
>
> ipc.Client class holds connection pool, that connection would be reused for a given host/port.
> and WritableRpcEngine holds Client pool, that client would be reused for a given SocketFactory.
> so, RPC.getProxy method returns proxy object, that only hold one connection or socket connected to a host/port, if or not, we create many proxy objects.
> if large requests are sent to RPC Server through proxy, the request would be wait long time to be sent.
> i think, for each host/port, ipc.Client would hold many connections.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11184-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 28 11:22:48 2010
Return-Path: <common-issues-return-11184-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12840 invoked from network); 28 Oct 2010 11:22:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Oct 2010 11:22:48 -0000
Received: (qmail 64754 invoked by uid 500); 28 Oct 2010 11:22:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64602 invoked by uid 500); 28 Oct 2010 11:22:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64594 invoked by uid 99); 28 Oct 2010 11:22:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 11:22:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 11:22:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9SBMNMn007654
	for <common-issues@hadoop.apache.org>; Thu, 28 Oct 2010 11:22:23 GMT
Message-ID: <20199517.119651288264943497.JavaMail.jira@thor>
Date: Thu, 28 Oct 2010 07:22:23 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7010) Typo in FileSystem.java
In-Reply-To: <26367340.101261288193659612.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925778#action_12925778 ] 

Hudson commented on HADOOP-7010:
--------------------------------

Integrated in Hadoop-Common-trunk #495 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/495/])
    HADOOP-7010. Typo in FileSystem.java. Contributed by Jingguo Yao.


> Typo in FileSystem.java
> -----------------------
>
>                 Key: HADOOP-7010
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7010
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Jingguo Yao
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7010.patch
>
>   Original Estimate: 0.08h
>  Remaining Estimate: 0.08h
>
> For the Javadoc for getLocal method, "Get the local file syste" should be "Get the local file system.".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11185-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 28 17:16:55 2010
Return-Path: <common-issues-return-11185-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48648 invoked from network); 28 Oct 2010 17:16:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Oct 2010 17:16:55 -0000
Received: (qmail 33509 invoked by uid 500); 28 Oct 2010 17:16:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33398 invoked by uid 500); 28 Oct 2010 17:16:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33386 invoked by uid 99); 28 Oct 2010 17:16:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 17:16:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 17:16:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9SHGWnJ011102
	for <common-issues@hadoop.apache.org>; Thu, 28 Oct 2010 17:16:32 GMT
Message-ID: <11857712.123921288286192577.JavaMail.jira@thor>
Date: Thu, 28 Oct 2010 13:16:32 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6663) BlockDecompressorStream get EOF
 exception when decompressing the file compressed from empty file
In-Reply-To: <1619883340.543171269850767204.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925875#action_12925875 ] 

Tom White commented on HADOOP-6663:
-----------------------------------

I ran the tests and test-patch manually:

{noformat}
     [exec] +1 overall. 
     [exec]
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec]
     [exec]     +1 tests included.  The patch appears to include 4 new or modified tests.
     [exec]
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec]
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec]
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec]
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec]
     [exec]     +1 system tests framework.  The patch passed system tests framework compile.
{noformat}

> BlockDecompressorStream get EOF exception when decompressing the file compressed from empty file
> ------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6663
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6663
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Kang Xiao
>            Assignee: Kang Xiao
>             Fix For: 0.22.0
>
>         Attachments: BlockDecompressorStream.java.patch, BlockDecompressorStream.java.patch, BlockDecompressorStream.patch, HADOOP-6663.patch
>
>
> An empty file can be compressed using BlockDecompressorStream, which is for block-based compressiong algorithm such as LZO. However, when decompressing the compressed file, BlockDecompressorStream get EOF exception.
> Here is a typical exception stack:
> java.io.EOFException
> at org.apache.hadoop.io.compress.BlockDecompressorStream.rawReadInt(BlockDecompressorStream.java:125)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:96)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:82)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at java.io.InputStream.read(InputStream.java:85)
> at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:39)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:186)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:170)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:18)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
> at org.apache.hadoop.mapred.Child.main(Child.java:196)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11186-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 28 17:18:58 2010
Return-Path: <common-issues-return-11186-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49326 invoked from network); 28 Oct 2010 17:18:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Oct 2010 17:18:58 -0000
Received: (qmail 35069 invoked by uid 500); 28 Oct 2010 17:18:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35017 invoked by uid 500); 28 Oct 2010 17:18:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35009 invoked by uid 99); 28 Oct 2010 17:18:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 17:18:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 17:18:57 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9SHIbBX011148
	for <common-issues@hadoop.apache.org>; Thu, 28 Oct 2010 17:18:37 GMT
Message-ID: <15819198.124061288286317083.JavaMail.jira@thor>
Date: Thu, 28 Oct 2010 13:18:37 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6663) BlockDecompressorStream get EOF
 exception when decompressing the file compressed from empty file
In-Reply-To: <1619883340.543171269850767204.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6663:
------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I've just committed this.

> BlockDecompressorStream get EOF exception when decompressing the file compressed from empty file
> ------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6663
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6663
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Kang Xiao
>            Assignee: Kang Xiao
>             Fix For: 0.22.0
>
>         Attachments: BlockDecompressorStream.java.patch, BlockDecompressorStream.java.patch, BlockDecompressorStream.patch, HADOOP-6663.patch
>
>
> An empty file can be compressed using BlockDecompressorStream, which is for block-based compressiong algorithm such as LZO. However, when decompressing the compressed file, BlockDecompressorStream get EOF exception.
> Here is a typical exception stack:
> java.io.EOFException
> at org.apache.hadoop.io.compress.BlockDecompressorStream.rawReadInt(BlockDecompressorStream.java:125)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:96)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:82)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at java.io.InputStream.read(InputStream.java:85)
> at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:39)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:186)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:170)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:18)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
> at org.apache.hadoop.mapred.Child.main(Child.java:196)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11187-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 28 17:38:52 2010
Return-Path: <common-issues-return-11187-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61918 invoked from network); 28 Oct 2010 17:38:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Oct 2010 17:38:52 -0000
Received: (qmail 62727 invoked by uid 500); 28 Oct 2010 17:38:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62698 invoked by uid 500); 28 Oct 2010 17:38:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62690 invoked by uid 99); 28 Oct 2010 17:38:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 17:38:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 17:38:49 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9SHcR19011389
	for <common-issues@hadoop.apache.org>; Thu, 28 Oct 2010 17:38:28 GMT
Message-ID: <9231331.124601288287507951.JavaMail.jira@thor>
Date: Thu, 28 Oct 2010 13:38:27 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6663) BlockDecompressorStream get EOF
 exception when decompressing the file compressed from empty file
In-Reply-To: <1619883340.543171269850767204.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925888#action_12925888 ] 

Hudson commented on HADOOP-6663:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #404 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/404/])
    HADOOP-6663.  BlockDecompressorStream get EOF exception when decompressing the file compressed from empty file.  Contributed by Kang Xiao.


> BlockDecompressorStream get EOF exception when decompressing the file compressed from empty file
> ------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6663
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6663
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Kang Xiao
>            Assignee: Kang Xiao
>             Fix For: 0.22.0
>
>         Attachments: BlockDecompressorStream.java.patch, BlockDecompressorStream.java.patch, BlockDecompressorStream.patch, HADOOP-6663.patch
>
>
> An empty file can be compressed using BlockDecompressorStream, which is for block-based compressiong algorithm such as LZO. However, when decompressing the compressed file, BlockDecompressorStream get EOF exception.
> Here is a typical exception stack:
> java.io.EOFException
> at org.apache.hadoop.io.compress.BlockDecompressorStream.rawReadInt(BlockDecompressorStream.java:125)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:96)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:82)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at java.io.InputStream.read(InputStream.java:85)
> at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:39)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:186)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:170)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:18)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
> at org.apache.hadoop.mapred.Child.main(Child.java:196)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11188-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 28 18:52:42 2010
Return-Path: <common-issues-return-11188-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13230 invoked from network); 28 Oct 2010 18:52:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Oct 2010 18:52:42 -0000
Received: (qmail 64170 invoked by uid 500); 28 Oct 2010 18:52:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64030 invoked by uid 500); 28 Oct 2010 18:52:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64022 invoked by uid 99); 28 Oct 2010 18:52:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 18:52:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 18:52:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9SIqLDa012111
	for <common-issues@hadoop.apache.org>; Thu, 28 Oct 2010 18:52:21 GMT
Message-ID: <27723498.125621288291941562.JavaMail.jira@thor>
Date: Thu, 28 Oct 2010 14:52:21 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-7009) MD5Hash provides a public factory
 method that creates an instance of MessageDigest
In-Reply-To: <246488.88311288128444362.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang resolved HADOOP-7009.
-----------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]

I've committed this!

> MD5Hash provides a public factory method that creates an instance of MessageDigest
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7009
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7009
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: md5hash.patch, md5hash1.patch
>
>
> MD5Hash has a private way of creating a MessageDigest object that's thread local. I'd like to have such a method which is public so that checksuming fsimage (HDFS-903) could use it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11189-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 28 19:14:42 2010
Return-Path: <common-issues-return-11189-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28356 invoked from network); 28 Oct 2010 19:14:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Oct 2010 19:14:42 -0000
Received: (qmail 87863 invoked by uid 500); 28 Oct 2010 19:14:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87799 invoked by uid 500); 28 Oct 2010 19:14:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87791 invoked by uid 99); 28 Oct 2010 19:14:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 19:14:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 19:14:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9SJEK8L012373
	for <common-issues@hadoop.apache.org>; Thu, 28 Oct 2010 19:14:21 GMT
Message-ID: <1977043.126031288293260939.JavaMail.jira@thor>
Date: Thu, 28 Oct 2010 15:14:20 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7009) MD5Hash provides a public factory
 method that creates an instance of MessageDigest
In-Reply-To: <246488.88311288128444362.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925917#action_12925917 ] 

Hudson commented on HADOOP-7009:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #405 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/405/])
    HADOOP-7009. MD5Hash provides a public factory method that creates an instance of MessageDigest. Contributed by Hairong Kuang.


> MD5Hash provides a public factory method that creates an instance of MessageDigest
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7009
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7009
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: md5hash.patch, md5hash1.patch
>
>
> MD5Hash has a private way of creating a MessageDigest object that's thread local. I'd like to have such a method which is public so that checksuming fsimage (HDFS-903) could use it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11190-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 28 20:55:44 2010
Return-Path: <common-issues-return-11190-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67901 invoked from network); 28 Oct 2010 20:55:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Oct 2010 20:55:44 -0000
Received: (qmail 90497 invoked by uid 500); 28 Oct 2010 20:55:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90339 invoked by uid 500); 28 Oct 2010 20:55:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90331 invoked by uid 99); 28 Oct 2010 20:55:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 20:55:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 20:55:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9SKtNiM013183
	for <common-issues@hadoop.apache.org>; Thu, 28 Oct 2010 20:55:24 GMT
Message-ID: <28742743.127101288299323666.JavaMail.jira@thor>
Date: Thu, 28 Oct 2010 16:55:23 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6991) SequenceFile::Reader discards length
 for files, does not call openFile
In-Reply-To: <22609732.541901286245712924.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley reassigned HADOOP-6991:
-------------------------------------

    Assignee: Chris Douglas

> SequenceFile::Reader discards length for files, does not call openFile
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6991
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6991
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Chris Douglas
>            Assignee: Chris Douglas
>            Priority: Minor
>         Attachments: 6991-0.patch, 6991-1.patch, h-6991-2.patch
>
>
> While the sorting and merging in {{SequenceFile}} is deprecated and {{openFile}} is archaic, the semantics should remain consistent.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11191-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 28 20:57:44 2010
Return-Path: <common-issues-return-11191-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70146 invoked from network); 28 Oct 2010 20:57:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Oct 2010 20:57:44 -0000
Received: (qmail 93347 invoked by uid 500); 28 Oct 2010 20:57:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93203 invoked by uid 500); 28 Oct 2010 20:57:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93195 invoked by uid 99); 28 Oct 2010 20:57:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 20:57:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 20:57:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9SKvK97013202
	for <common-issues@hadoop.apache.org>; Thu, 28 Oct 2010 20:57:20 GMT
Message-ID: <22937827.127131288299440826.JavaMail.jira@thor>
Date: Thu, 28 Oct 2010 16:57:20 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6991) SequenceFile::Reader discards length
 for files, does not call openFile
In-Reply-To: <22609732.541901286245712924.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6991:
----------------------------------

       Resolution: Fixed
    Fix Version/s: 0.22.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I just committed this. Thanks, Chris!

> SequenceFile::Reader discards length for files, does not call openFile
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6991
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6991
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Chris Douglas
>            Assignee: Chris Douglas
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: 6991-0.patch, 6991-1.patch, h-6991-2.patch
>
>
> While the sorting and merging in {{SequenceFile}} is deprecated and {{openFile}} is archaic, the semantics should remain consistent.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11192-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 28 21:21:41 2010
Return-Path: <common-issues-return-11192-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87097 invoked from network); 28 Oct 2010 21:21:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Oct 2010 21:21:40 -0000
Received: (qmail 27848 invoked by uid 500); 28 Oct 2010 21:21:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27749 invoked by uid 500); 28 Oct 2010 21:21:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27741 invoked by uid 99); 28 Oct 2010 21:21:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 21:21:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 21:21:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9SLLKOE013464
	for <common-issues@hadoop.apache.org>; Thu, 28 Oct 2010 21:21:20 GMT
Message-ID: <29057905.127421288300880006.JavaMail.jira@thor>
Date: Thu, 28 Oct 2010 17:21:20 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6991) SequenceFile::Reader discards
 length for files, does not call openFile
In-Reply-To: <22609732.541901286245712924.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925950#action_12925950 ] 

Hudson commented on HADOOP-6991:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #407 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/407/])
    HADOOP-6991.  Fix SequenceFile::Reader to honor file lengths and call
openFile (cdouglas via omalley)


> SequenceFile::Reader discards length for files, does not call openFile
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6991
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6991
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Chris Douglas
>            Assignee: Chris Douglas
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: 6991-0.patch, 6991-1.patch, h-6991-2.patch
>
>
> While the sorting and merging in {{SequenceFile}} is deprecated and {{openFile}} is archaic, the semantics should remain consistent.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11193-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 28 23:11:45 2010
Return-Path: <common-issues-return-11193-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61132 invoked from network); 28 Oct 2010 23:11:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Oct 2010 23:11:45 -0000
Received: (qmail 80490 invoked by uid 500); 28 Oct 2010 23:11:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80447 invoked by uid 500); 28 Oct 2010 23:11:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80439 invoked by uid 99); 28 Oct 2010 23:11:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 23:11:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 23:11:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9SNBL8F014455
	for <common-issues@hadoop.apache.org>; Thu, 28 Oct 2010 23:11:21 GMT
Message-ID: <3217110.128951288307481096.JavaMail.jira@thor>
Date: Thu, 28 Oct 2010 19:11:21 -0400 (EDT)
From: "Patrick Kling (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-7013) Add boolean field isCorrupt to
 BlockLocation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Add boolean field isCorrupt to BlockLocation
--------------------------------------------

                 Key: HADOOP-7013
                 URL: https://issues.apache.org/jira/browse/HADOOP-7013
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Patrick Kling


This is needed to allow DFSClient.getBlockLocations to notify the calling application when returning a BlockLocation that corresponds to a corrupt block. Currently, this happens when there are no uncorrupted replicas of a requested block.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11194-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Oct 28 23:49:46 2010
Return-Path: <common-issues-return-11194-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76169 invoked from network); 28 Oct 2010 23:49:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Oct 2010 23:49:46 -0000
Received: (qmail 29057 invoked by uid 500); 28 Oct 2010 23:49:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29031 invoked by uid 500); 28 Oct 2010 23:49:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29023 invoked by uid 99); 28 Oct 2010 23:49:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 23:49:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Oct 2010 23:49:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9SNnPB9014801
	for <common-issues@hadoop.apache.org>; Thu, 28 Oct 2010 23:49:25 GMT
Message-ID: <19505596.129651288309765035.JavaMail.jira@thor>
Date: Thu, 28 Oct 2010 19:49:25 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6996) Allow CodecFactory to return a codec
 object given a codec' class name
In-Reply-To: <1541599.80951286821892815.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6996:
----------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I've committed this!

> Allow CodecFactory to return a codec object given a codec' class name
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6996
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6996
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: getCodecByClassName.patch
>
>
> CodecFactory specify the list of codec that are supported by Hadoop. However, it returns a codec only by a file's name. I would like to make getCodec method to alternatively take a codec's class name.
> This is required by  HDFS-1435, where
> 1. it allows an HDFS admin to configure which codec to use to save an image. 
> 2. It stores the codec class name in its on-disk image instead of a file's suffix.
> When saving and reading an image, I'd like to get an codec from CodecFactory by its class name. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11195-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 29 00:49:44 2010
Return-Path: <common-issues-return-11195-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7891 invoked from network); 29 Oct 2010 00:49:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Oct 2010 00:49:44 -0000
Received: (qmail 80140 invoked by uid 500); 29 Oct 2010 00:49:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80027 invoked by uid 500); 29 Oct 2010 00:49:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80019 invoked by uid 99); 29 Oct 2010 00:49:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 00:49:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 00:49:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9T0nKFe015265
	for <common-issues@hadoop.apache.org>; Fri, 29 Oct 2010 00:49:20 GMT
Message-ID: <21599126.130371288313360183.JavaMail.jira@thor>
Date: Thu, 28 Oct 2010 20:49:20 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6977) Herriot daemon clients should vend
 statistics
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6977:
---------------------------------------

    Attachment: HADOOP-6977.patch

Addressing good finding Boris made at HDFS-1408.

> Herriot daemon clients should vend statistics
> ---------------------------------------------
>
>                 Key: HADOOP-6977
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6977
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: FinderTest.java, HADOOP-6977.patch, HADOOP-6977.patch, HADOOP-6977.patch, HADOOP-6977.patch, HADOOP-6977.patch, HADOOP-6977.y20S.patch, HADOOP-6977.y20S.patch
>
>
> The HDFS web user interface serves useful information through dfshealth.jsp and dfsnodelist.jsp.
> The Herriot interface to Hadoop cluster daemons would benefit from the addition of some way to channel metics information.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11196-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 29 02:06:41 2010
Return-Path: <common-issues-return-11196-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44533 invoked from network); 29 Oct 2010 02:06:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Oct 2010 02:06:41 -0000
Received: (qmail 35883 invoked by uid 500); 29 Oct 2010 02:06:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35846 invoked by uid 500); 29 Oct 2010 02:06:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35787 invoked by uid 99); 29 Oct 2010 02:06:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 02:06:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 02:06:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9T26IlY016443
	for <common-issues@hadoop.apache.org>; Fri, 29 Oct 2010 02:06:18 GMT
Message-ID: <10468298.133111288317978213.JavaMail.jira@thor>
Date: Thu, 28 Oct 2010 22:06:18 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6873) using delegation token over hftp
 for long running clients (part of hdfs 1296)
In-Reply-To: <3672204.508361279757392441.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926090#action_12926090 ] 

Hudson commented on HADOOP-6873:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/])
    

> using delegation token over hftp for long running clients (part of hdfs 1296)
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-6873
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6873
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6873-2.patch, HADOOP-6873.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11197-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 29 02:06:45 2010
Return-Path: <common-issues-return-11197-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44799 invoked from network); 29 Oct 2010 02:06:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Oct 2010 02:06:45 -0000
Received: (qmail 36592 invoked by uid 500); 29 Oct 2010 02:06:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36555 invoked by uid 500); 29 Oct 2010 02:06:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36547 invoked by uid 99); 29 Oct 2010 02:06:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 02:06:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 02:06:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9T26LYd016504
	for <common-issues@hadoop.apache.org>; Fri, 29 Oct 2010 02:06:21 GMT
Message-ID: <13980391.133311288317981791.JavaMail.jira@thor>
Date: Thu, 28 Oct 2010 22:06:21 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6826) Revert FileSystem create method
 that takes CreateFlags
In-Reply-To: <6937853.16451276636883446.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926097#action_12926097 ] 

Hudson commented on HADOOP-6826:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/])
    

> Revert FileSystem create method that takes CreateFlags
> ------------------------------------------------------
>
>                 Key: HADOOP-6826
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6826
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6826.patch
>
>
> As discussed in HDFS-609 and HADOOP-5438 we should back out the FileSystem create() method that takes a set of CreateFlag objects, until the interface has been agreed upon and fully tested.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11198-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 29 02:06:47 2010
Return-Path: <common-issues-return-11198-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45032 invoked from network); 29 Oct 2010 02:06:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Oct 2010 02:06:47 -0000
Received: (qmail 37419 invoked by uid 500); 29 Oct 2010 02:06:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37383 invoked by uid 500); 29 Oct 2010 02:06:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37375 invoked by uid 99); 29 Oct 2010 02:06:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 02:06:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 02:06:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9T26NmE016537
	for <common-issues@hadoop.apache.org>; Fri, 29 Oct 2010 02:06:23 GMT
Message-ID: <20762671.133421288317983744.JavaMail.jira@thor>
Date: Thu, 28 Oct 2010 22:06:23 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6991) SequenceFile::Reader discards
 length for files, does not call openFile
In-Reply-To: <22609732.541901286245712924.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926101#action_12926101 ] 

Hudson commented on HADOOP-6991:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/])
    

> SequenceFile::Reader discards length for files, does not call openFile
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6991
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6991
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Chris Douglas
>            Assignee: Chris Douglas
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: 6991-0.patch, 6991-1.patch, h-6991-2.patch
>
>
> While the sorting and merging in {{SequenceFile}} is deprecated and {{openFile}} is archaic, the semantics should remain consistent.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11199-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 29 02:07:14 2010
Return-Path: <common-issues-return-11199-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46034 invoked from network); 29 Oct 2010 02:07:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Oct 2010 02:07:14 -0000
Received: (qmail 40412 invoked by uid 500); 29 Oct 2010 02:07:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40378 invoked by uid 500); 29 Oct 2010 02:07:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40368 invoked by uid 99); 29 Oct 2010 02:07:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 02:07:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 02:07:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9T26rl1017025
	for <common-issues@hadoop.apache.org>; Fri, 29 Oct 2010 02:06:53 GMT
Message-ID: <32953749.135041288318013800.JavaMail.jira@thor>
Date: Thu, 28 Oct 2010 22:06:53 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7003) Fix hadoop patch testing using
 jira_cli tool
In-Reply-To: <30184462.19001287608066472.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926127#action_12926127 ] 

Hudson commented on HADOOP-7003:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/])
    

> Fix hadoop patch testing using jira_cli tool
> --------------------------------------------
>
>                 Key: HADOOP-7003
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7003
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>            Reporter: Giridharan Kesavan
>            Assignee: Nigel Daley
>             Fix For: hudson
>
>         Attachments: hudsonPatchQueueAdmin.sh, hudsonPatchQueueAdmin.sh
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11200-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 29 02:07:15 2010
Return-Path: <common-issues-return-11200-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46123 invoked from network); 29 Oct 2010 02:07:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Oct 2010 02:07:15 -0000
Received: (qmail 40729 invoked by uid 500); 29 Oct 2010 02:07:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40699 invoked by uid 500); 29 Oct 2010 02:07:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40691 invoked by uid 99); 29 Oct 2010 02:07:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 02:07:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 02:07:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9T26tnA017052
	for <common-issues@hadoop.apache.org>; Fri, 29 Oct 2010 02:06:55 GMT
Message-ID: <24557971.135131288318015025.JavaMail.jira@thor>
Date: Thu, 28 Oct 2010 22:06:55 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7005) Update test-patch.sh to remove
 callback to Hudson master
In-Reply-To: <17555034.54181287988399911.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926129#action_12926129 ] 

Hudson commented on HADOOP-7005:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/])
    

> Update test-patch.sh to remove callback to Hudson master
> --------------------------------------------------------
>
>                 Key: HADOOP-7005
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7005
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7005.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11201-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 29 02:07:16 2010
Return-Path: <common-issues-return-11201-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46156 invoked from network); 29 Oct 2010 02:07:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Oct 2010 02:07:15 -0000
Received: (qmail 40908 invoked by uid 500); 29 Oct 2010 02:07:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40882 invoked by uid 500); 29 Oct 2010 02:07:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40873 invoked by uid 99); 29 Oct 2010 02:07:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 02:07:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 02:07:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9T26ql2016985
	for <common-issues@hadoop.apache.org>; Fri, 29 Oct 2010 02:06:52 GMT
Message-ID: <12528441.134911288318011994.JavaMail.jira@thor>
Date: Thu, 28 Oct 2010 22:06:51 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6632) Support for using different
 Kerberos keys for different instances of Hadoop services
In-Reply-To: <1542465557.251491268540967230.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926125#action_12926125 ] 

Hudson commented on HADOOP-6632:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/])
    

> Support for using different Kerberos keys for different instances of Hadoop services
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6632
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6632
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>             Fix For: 0.22.0
>
>         Attachments: 6632.mr.patch, c6632-05.patch, c6632-07.patch, HADOOP-6632-Y20S-18.patch, HADOOP-6632-Y20S-22.patch
>
>
> We tested using the same Kerberos key for all datanodes in a HDFS cluster or the same Kerberos key for all TaskTarckers in a MapRed cluster. But it doesn't work. The reason is that when datanodes try to authenticate to the namenode all at once, the Kerberos authenticators they send to the namenode may have the same timestamp and will be rejected as replay requests. This JIRA makes it possible to use a unique key for each service instance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11202-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 29 02:07:17 2010
Return-Path: <common-issues-return-11202-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46202 invoked from network); 29 Oct 2010 02:07:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Oct 2010 02:07:16 -0000
Received: (qmail 41244 invoked by uid 500); 29 Oct 2010 02:07:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41202 invoked by uid 500); 29 Oct 2010 02:07:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41180 invoked by uid 99); 29 Oct 2010 02:07:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 02:07:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 02:07:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9T26tCx017064
	for <common-issues@hadoop.apache.org>; Fri, 29 Oct 2010 02:06:55 GMT
Message-ID: <2972773.135171288318015560.JavaMail.jira@thor>
Date: Thu, 28 Oct 2010 22:06:55 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6845) TokenStorage renamed to
 Credentials.
In-Reply-To: <11027372.98871277758850634.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926131#action_12926131 ] 

Hudson commented on HADOOP-6845:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/])
    

> TokenStorage renamed to Credentials.
> ------------------------------------
>
>                 Key: HADOOP-6845
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6845
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6845.1.patch, HADOOP-6845.2.patch, HADOOP-6845.3.patch
>
>
> This jira tracks common changes for MAPREDUCE-1528.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11203-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 29 02:07:19 2010
Return-Path: <common-issues-return-11203-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46423 invoked from network); 29 Oct 2010 02:07:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Oct 2010 02:07:18 -0000
Received: (qmail 41728 invoked by uid 500); 29 Oct 2010 02:07:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41696 invoked by uid 500); 29 Oct 2010 02:07:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41688 invoked by uid 99); 29 Oct 2010 02:07:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 02:07:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 02:07:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9T26thJ017070
	for <common-issues@hadoop.apache.org>; Fri, 29 Oct 2010 02:06:55 GMT
Message-ID: <24499031.135191288318015845.JavaMail.jira@thor>
Date: Thu, 28 Oct 2010 22:06:55 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7007) update the hudson-test-patch target
 to work with the latest test-patch script.
In-Reply-To: <7411320.67141288042220266.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926132#action_12926132 ] 

Hudson commented on HADOOP-7007:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/])
    

> update the hudson-test-patch target to work with the latest test-patch script.
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7007
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7007
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7007-build.xml.patch, HADOOP-7007-test-patch.patch, HADOOP-7007.patch
>
>
> The hudson-test-patch target has to be updated to work with the current test-patch.sh script. Since the callback login in the test-patch.sh is removed. by hadoop-7005

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11204-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 29 02:07:22 2010
Return-Path: <common-issues-return-11204-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46694 invoked from network); 29 Oct 2010 02:07:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Oct 2010 02:07:22 -0000
Received: (qmail 42655 invoked by uid 500); 29 Oct 2010 02:07:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42608 invoked by uid 500); 29 Oct 2010 02:07:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42600 invoked by uid 99); 29 Oct 2010 02:07:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 02:07:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 02:07:21 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9T271bK017169
	for <common-issues@hadoop.apache.org>; Fri, 29 Oct 2010 02:07:01 GMT
Message-ID: <20818307.135521288318021319.JavaMail.jira@thor>
Date: Thu, 28 Oct 2010 22:07:01 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6786) test-patch needs to verify Herriot
 integrity
In-Reply-To: <19336182.46451274829212745.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926140#action_12926140 ] 

Hudson commented on HADOOP-6786:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/])
    

> test-patch needs to verify Herriot integrity
> --------------------------------------------
>
>                 Key: HADOOP-6786
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6786
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>             Fix For: 0.21.1
>
>         Attachments: HADOOP-6786.patch, HADOOP-6786.patch, HADOOP-6786.patch
>
>
> Whenever a new patch is submitted for verification {{test-patch}} process has to make sure that none of Herriot bindings were broken.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11205-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 29 11:22:48 2010
Return-Path: <common-issues-return-11205-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35778 invoked from network); 29 Oct 2010 11:22:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Oct 2010 11:22:48 -0000
Received: (qmail 11400 invoked by uid 500); 29 Oct 2010 11:22:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11240 invoked by uid 500); 29 Oct 2010 11:22:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11232 invoked by uid 99); 29 Oct 2010 11:22:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 11:22:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 11:22:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9TBMNKm023591
	for <common-issues@hadoop.apache.org>; Fri, 29 Oct 2010 11:22:23 GMT
Message-ID: <11097044.139871288351343310.JavaMail.jira@thor>
Date: Fri, 29 Oct 2010 07:22:23 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6991) SequenceFile::Reader discards
 length for files, does not call openFile
In-Reply-To: <22609732.541901286245712924.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926239#action_12926239 ] 

Hudson commented on HADOOP-6991:
--------------------------------

Integrated in Hadoop-Common-trunk #496 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/496/])
    HADOOP-6991.  Fix SequenceFile::Reader to honor file lengths and call
openFile (cdouglas via omalley)
HADOOP-6991. Move testcase from mapreduce to common, since it tests 
SequenceFile.


> SequenceFile::Reader discards length for files, does not call openFile
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6991
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6991
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Chris Douglas
>            Assignee: Chris Douglas
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: 6991-0.patch, 6991-1.patch, h-6991-2.patch
>
>
> While the sorting and merging in {{SequenceFile}} is deprecated and {{openFile}} is archaic, the semantics should remain consistent.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11206-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 29 11:22:59 2010
Return-Path: <common-issues-return-11206-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35879 invoked from network); 29 Oct 2010 11:22:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Oct 2010 11:22:59 -0000
Received: (qmail 11994 invoked by uid 500); 29 Oct 2010 11:22:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11974 invoked by uid 500); 29 Oct 2010 11:22:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11958 invoked by uid 99); 29 Oct 2010 11:22:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 11:22:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 11:22:56 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9TBMYt4023603
	for <common-issues@hadoop.apache.org>; Fri, 29 Oct 2010 11:22:34 GMT
Message-ID: <6255850.139911288351354912.JavaMail.jira@thor>
Date: Fri, 29 Oct 2010 07:22:34 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6663) BlockDecompressorStream get EOF
 exception when decompressing the file compressed from empty file
In-Reply-To: <1619883340.543171269850767204.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926240#action_12926240 ] 

Hudson commented on HADOOP-6663:
--------------------------------

Integrated in Hadoop-Common-trunk #496 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/496/])
    HADOOP-6663.  BlockDecompressorStream get EOF exception when decompressing the file compressed from empty file.  Contributed by Kang Xiao.


> BlockDecompressorStream get EOF exception when decompressing the file compressed from empty file
> ------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6663
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6663
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Kang Xiao
>            Assignee: Kang Xiao
>             Fix For: 0.22.0
>
>         Attachments: BlockDecompressorStream.java.patch, BlockDecompressorStream.java.patch, BlockDecompressorStream.patch, HADOOP-6663.patch
>
>
> An empty file can be compressed using BlockDecompressorStream, which is for block-based compressiong algorithm such as LZO. However, when decompressing the compressed file, BlockDecompressorStream get EOF exception.
> Here is a typical exception stack:
> java.io.EOFException
> at org.apache.hadoop.io.compress.BlockDecompressorStream.rawReadInt(BlockDecompressorStream.java:125)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:96)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:82)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at java.io.InputStream.read(InputStream.java:85)
> at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:39)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:186)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:170)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:18)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
> at org.apache.hadoop.mapred.Child.main(Child.java:196)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11207-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 29 11:23:00 2010
Return-Path: <common-issues-return-11207-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35909 invoked from network); 29 Oct 2010 11:22:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Oct 2010 11:22:59 -0000
Received: (qmail 12186 invoked by uid 500); 29 Oct 2010 11:22:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12001 invoked by uid 500); 29 Oct 2010 11:22:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11965 invoked by uid 99); 29 Oct 2010 11:22:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 11:22:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 11:22:56 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9TBMZcA023612
	for <common-issues@hadoop.apache.org>; Fri, 29 Oct 2010 11:22:35 GMT
Message-ID: <20189203.139941288351355378.JavaMail.jira@thor>
Date: Fri, 29 Oct 2010 07:22:35 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7009) MD5Hash provides a public factory
 method that creates an instance of MessageDigest
In-Reply-To: <246488.88311288128444362.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926241#action_12926241 ] 

Hudson commented on HADOOP-7009:
--------------------------------

Integrated in Hadoop-Common-trunk #496 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/496/])
    HADOOP-7009. MD5Hash provides a public factory method that creates an instance of MessageDigest. Contributed by Hairong Kuang.


> MD5Hash provides a public factory method that creates an instance of MessageDigest
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7009
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7009
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: md5hash.patch, md5hash1.patch
>
>
> MD5Hash has a private way of creating a MessageDigest object that's thread local. I'd like to have such a method which is public so that checksuming fsimage (HDFS-903) could use it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11208-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 29 19:28:47 2010
Return-Path: <common-issues-return-11208-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18651 invoked from network); 29 Oct 2010 19:28:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Oct 2010 19:28:47 -0000
Received: (qmail 64410 invoked by uid 500); 29 Oct 2010 19:28:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64388 invoked by uid 500); 29 Oct 2010 19:28:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64380 invoked by uid 99); 29 Oct 2010 19:28:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 19:28:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 19:28:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9TJSNSS029586
	for <common-issues@hadoop.apache.org>; Fri, 29 Oct 2010 19:28:23 GMT
Message-ID: <1387343.148221288380503431.JavaMail.jira@thor>
Date: Fri, 29 Oct 2010 15:28:23 -0400 (EDT)
From: "Gerald Guo (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6953) start-{dfs,mapred}.sh scripts fail
 if HADOOP_HOME is not set
In-Reply-To: <4106516.214881284590314802.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926418#action_12926418 ] 

Gerald Guo commented on HADOOP-6953:
------------------------------------


You can add following code to files "hdfs-config.sh" and "mapred-config.sh".

this="${BASH_SOURCE-$0}"
while [ -h "$this" ]; do
  ls=`ls -ld "$this"`
  link=`expr "$ls" : '.*-> \(.*\)$'`
  if expr "$link" : '.*/.*' > /dev/null; then
    this="$link"
  else
    this=`dirname "$this"`/"$link"
  fi  
done

# convert relative path to absolute path
common_bin=`dirname "$this"`
script=`basename "$this"`
common_bin=`cd "$common_bin"; pwd`
this="$common_bin/$script"

# the root of the Hadoop installation
#TODO: change the env variable when dir structure is changed
export HADOOP_HOME=`dirname "$this"`/..
export HADOOP_COMMON_HOME="${HADOOP_HOME}"


> start-{dfs,mapred}.sh scripts fail if HADOOP_HOME is not set
> ------------------------------------------------------------
>
>                 Key: HADOOP-6953
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6953
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.21.0
>            Reporter: Tom White
>             Fix For: 0.21.1
>
>
> If the HADOOP_HOME environment variable is not set then the start and stop scripts for HDFS and MapReduce fail with "Hadoop common not found.". The start-all.sh and stop-all.sh scripts are not affected.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11209-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 29 19:32:47 2010
Return-Path: <common-issues-return-11209-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19759 invoked from network); 29 Oct 2010 19:32:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Oct 2010 19:32:47 -0000
Received: (qmail 70012 invoked by uid 500); 29 Oct 2010 19:32:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69978 invoked by uid 500); 29 Oct 2010 19:32:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69969 invoked by uid 99); 29 Oct 2010 19:32:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 19:32:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 19:32:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9TJWNEo029639
	for <common-issues@hadoop.apache.org>; Fri, 29 Oct 2010 19:32:23 GMT
Message-ID: <15769023.148351288380743351.JavaMail.jira@thor>
Date: Fri, 29 Oct 2010 15:32:23 -0400 (EDT)
From: "Gerald Guo (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6953) start-{dfs,mapred}.sh scripts fail
 if HADOOP_HOME is not set
In-Reply-To: <4106516.214881284590314802.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926421#action_12926421 ] 

Gerald Guo commented on HADOOP-6953:
------------------------------------

Sorry for the weird formatting in my last comment.

this="${BASH_SOURCE-$0}"
while [ -h "$this" ]; do
  ls=`ls -ld "$this"`
  link=`expr "$ls" : '.*-> \(.*\)$'`
  if expr "$link" : '.*/.*' > /dev/null; then
    this="$link"
  else
    this=`dirname "$this"`/"$link"
  fi  
done

\# convert relative path to absolute path
common_bin=`dirname "$this"`
script=`basename "$this"`
common_bin=`cd "$common_bin"; pwd`
this="$common_bin/$script"

\# the root of the Hadoop installation
\#TODO: change the env variable when dir structure is changed
export HADOOP_HOME=`dirname "$this"`/..
export HADOOP_COMMON_HOME="${HADOOP_HOME}"

> start-{dfs,mapred}.sh scripts fail if HADOOP_HOME is not set
> ------------------------------------------------------------
>
>                 Key: HADOOP-6953
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6953
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.21.0
>            Reporter: Tom White
>             Fix For: 0.21.1
>
>
> If the HADOOP_HOME environment variable is not set then the start and stop scripts for HDFS and MapReduce fail with "Hadoop common not found.". The start-all.sh and stop-all.sh scripts are not affected.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11210-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 29 21:08:43 2010
Return-Path: <common-issues-return-11210-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65002 invoked from network); 29 Oct 2010 21:08:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Oct 2010 21:08:43 -0000
Received: (qmail 96639 invoked by uid 500); 29 Oct 2010 21:08:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96610 invoked by uid 500); 29 Oct 2010 21:08:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96602 invoked by uid 99); 29 Oct 2010 21:08:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 21:08:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 21:08:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9TL8MEV000733
	for <common-issues@hadoop.apache.org>; Fri, 29 Oct 2010 21:08:22 GMT
Message-ID: <6667763.150251288386502076.JavaMail.jira@thor>
Date: Fri, 29 Oct 2010 17:08:22 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6990) Move TestSequenceFile from MapReduce
 to Common
In-Reply-To: <23983344.541871286245592779.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas resolved HADOOP-6990.
-----------------------------------

    Resolution: Duplicate

Fixed in HADOOP-6991

> Move TestSequenceFile from MapReduce to Common
> ----------------------------------------------
>
>                 Key: HADOOP-6990
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6990
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: io
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Chris Douglas
>            Priority: Trivial
>
> {{TestSequenceFile}} escaped from Common and ended up in the MapReduce tree, causing several regressions in {{SequenceFile}} to go unnoticed

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11211-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Oct 29 22:57:44 2010
Return-Path: <common-issues-return-11211-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97799 invoked from network); 29 Oct 2010 22:57:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Oct 2010 22:57:44 -0000
Received: (qmail 97546 invoked by uid 500); 29 Oct 2010 22:57:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97504 invoked by uid 500); 29 Oct 2010 22:57:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97496 invoked by uid 99); 29 Oct 2010 22:57:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 22:57:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 22:57:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9TMvKAU001694
	for <common-issues@hadoop.apache.org>; Fri, 29 Oct 2010 22:57:20 GMT
Message-ID: <23645476.151831288393040910.JavaMail.jira@thor>
Date: Fri, 29 Oct 2010 18:57:20 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7011) KerberosName.main(...) throws NPE
In-Reply-To: <2032276.105871288208720568.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-7011:
---------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

This looks right to me & tests pass.  Nothing appears to call this, so it's just useful for debugging, but still worth fixing.

I committed this.  Thanks, Aaron!


> KerberosName.main(...) throws NPE
> ---------------------------------
>
>                 Key: HADOOP-7011
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7011
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7011.0.txt
>
>
> The main method of KerberosName attempts to do short name translation before calling KerberosName.setConfiguration(...).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11212-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 30 04:12:46 2010
Return-Path: <common-issues-return-11212-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27538 invoked from network); 30 Oct 2010 04:12:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Oct 2010 04:12:46 -0000
Received: (qmail 15485 invoked by uid 500); 30 Oct 2010 04:12:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15322 invoked by uid 500); 30 Oct 2010 04:12:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15314 invoked by uid 99); 30 Oct 2010 04:12:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 30 Oct 2010 04:12:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 30 Oct 2010 04:12:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9U4CKT4005223
	for <common-issues@hadoop.apache.org>; Sat, 30 Oct 2010 04:12:20 GMT
Message-ID: <14414782.154171288411940612.JavaMail.jira@thor>
Date: Sat, 30 Oct 2010 00:12:20 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-7014) Generalize CLITest structure and
 interfaces to faciliate upstream adoption (e.g. for web testing)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Generalize CLITest structure and interfaces to faciliate upstream adoption (e.g. for web testing)
-------------------------------------------------------------------------------------------------

                 Key: HADOOP-7014
                 URL: https://issues.apache.org/jira/browse/HADOOP-7014
             Project: Hadoop Common
          Issue Type: Improvement
          Components: test
    Affects Versions: 0.22.0
            Reporter: Konstantin Boudnik
            Assignee: Konstantin Boudnik


There's at least one use case where TestCLI infrastructure is helpful for testing projects outside of core Hadoop (e.g. Owl web testing). In order to make this acceptance easier for upstream project TestCLI needs to be refactored.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11213-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 30 04:22:48 2010
Return-Path: <common-issues-return-11213-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28829 invoked from network); 30 Oct 2010 04:22:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Oct 2010 04:22:48 -0000
Received: (qmail 20562 invoked by uid 500); 30 Oct 2010 04:22:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20405 invoked by uid 500); 30 Oct 2010 04:22:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20397 invoked by uid 99); 30 Oct 2010 04:22:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 30 Oct 2010 04:22:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 30 Oct 2010 04:22:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9U4MN03005305
	for <common-issues@hadoop.apache.org>; Sat, 30 Oct 2010 04:22:23 GMT
Message-ID: <30942355.154301288412543444.JavaMail.jira@thor>
Date: Sat, 30 Oct 2010 00:22:23 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-7014) Generalize CLITest structure and
 interfaces to faciliate upstream adoption (e.g. for web testing)
In-Reply-To: <14414782.154171288411940612.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-7014:
---------------------------------------

    Attachment: HADOOP-7014.patch

Fix.

> Generalize CLITest structure and interfaces to faciliate upstream adoption (e.g. for web testing)
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7014
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7014
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-7014.patch
>
>
> There's at least one use case where TestCLI infrastructure is helpful for testing projects outside of core Hadoop (e.g. Owl web testing). In order to make this acceptance easier for upstream project TestCLI needs to be refactored.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11214-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Oct 30 11:22:45 2010
Return-Path: <common-issues-return-11214-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79502 invoked from network); 30 Oct 2010 11:22:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Oct 2010 11:22:45 -0000
Received: (qmail 16716 invoked by uid 500); 30 Oct 2010 11:22:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16549 invoked by uid 500); 30 Oct 2010 11:22:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16541 invoked by uid 99); 30 Oct 2010 11:22:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 30 Oct 2010 11:22:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 30 Oct 2010 11:22:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9UBMLQE015152
	for <common-issues@hadoop.apache.org>; Sat, 30 Oct 2010 11:22:21 GMT
Message-ID: <2835121.155971288437741171.JavaMail.jira@thor>
Date: Sat, 30 Oct 2010 07:22:21 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-7011) KerberosName.main(...) throws NPE
In-Reply-To: <2032276.105871288208720568.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926565#action_12926565 ] 

Hudson commented on HADOOP-7011:
--------------------------------

Integrated in Hadoop-Common-trunk #497 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/497/])
    HADOOP-7011.  Fix KerberosName.main() to not throw an NPE.  Contributed by Aaron T. Myers.


> KerberosName.main(...) throws NPE
> ---------------------------------
>
>                 Key: HADOOP-7011
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7011
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7011.0.txt
>
>
> The main method of KerberosName attempts to do short name translation before calling KerberosName.setConfiguration(...).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11215-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Oct 31 04:49:47 2010
Return-Path: <common-issues-return-11215-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21632 invoked from network); 31 Oct 2010 04:49:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Oct 2010 04:49:47 -0000
Received: (qmail 85881 invoked by uid 500); 31 Oct 2010 04:49:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85719 invoked by uid 500); 31 Oct 2010 04:49:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85710 invoked by uid 99); 31 Oct 2010 04:49:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 31 Oct 2010 04:49:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 31 Oct 2010 04:49:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o9V4nK68010158
	for <common-issues@hadoop.apache.org>; Sun, 31 Oct 2010 04:49:20 GMT
Message-ID: <32009476.160611288500560055.JavaMail.jira@thor>
Date: Sun, 31 Oct 2010 00:49:20 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6977) Herriot daemon clients should vend
 statistics
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926679#action_12926679 ] 

Boris Shkolnik commented on HADOOP-6977:
----------------------------------------

+1 for latest change

> Herriot daemon clients should vend statistics
> ---------------------------------------------
>
>                 Key: HADOOP-6977
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6977
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: FinderTest.java, HADOOP-6977.patch, HADOOP-6977.patch, HADOOP-6977.patch, HADOOP-6977.patch, HADOOP-6977.patch, HADOOP-6977.y20S.patch, HADOOP-6977.y20S.patch
>
>
> The HDFS web user interface serves useful information through dfshealth.jsp and dfsnodelist.jsp.
> The Herriot interface to Hadoop cluster daemons would benefit from the addition of some way to channel metics information.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 02:58:24 2009
Return-Path: <common-issues-return-1-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44833 invoked from network); 1 Jul 2009 02:58:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 02:58:24 -0000
Received: (qmail 46153 invoked by uid 500); 1 Jul 2009 02:58:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46129 invoked by uid 500); 1 Jul 2009 02:58:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Delivered-To: moderator for common-issues@hadoop.apache.org
Received: (qmail 91845 invoked by uid 99); 1 Jul 2009 00:43:14 -0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Message-ID: <1252650899.1246408967176.JavaMail.jira@brutus>
Date: Tue, 30 Jun 2009 17:42:47 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6122) 64 javac compiler warnings
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12725867#action_12725867 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6122:
------------------------------------------------

> ... For an empty patch, why it says 64 javac warnings but not 124?

Look like that this is another problem in common but not in hdfs and mapreduce.

> 64 javac compiler warnings
> --------------------------
>
>                 Key: HADOOP-6122
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6122
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6122_20090630.patch
>
>
> Ran "ant test-patch" with a empty patch.  Got 
> {noformat}
>      [exec]     -1 javac.  The applied patch generated 64 javac compiler warnings (more than the trunk's current 124 warnings).
>  {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-2-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 04:19:02 2009
Return-Path: <common-issues-return-2-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61359 invoked from network); 1 Jul 2009 04:19:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 04:19:02 -0000
Received: (qmail 86058 invoked by uid 500); 1 Jul 2009 04:19:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85989 invoked by uid 500); 1 Jul 2009 04:19:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85979 invoked by uid 99); 1 Jul 2009 04:19:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 04:19:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 04:19:09 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4D910234C004
	for <common-issues@hadoop.apache.org>; Tue, 30 Jun 2009 21:18:47 -0700 (PDT)
Message-ID: <1017868693.1246421927302.JavaMail.jira@brutus>
Date: Tue, 30 Jun 2009 21:18:47 -0700 (PDT)
From: "rahul k singh (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5170) Set max map/reduce tasks on a per-job
 basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

rahul k singh updated HADOOP-5170:
----------------------------------

    Attachment: tasklimits-v4-20.patch

patch for yahoo internal repo

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-3-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 05:29:05 2009
Return-Path: <common-issues-return-3-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75706 invoked from network); 1 Jul 2009 05:29:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 05:29:05 -0000
Received: (qmail 15477 invoked by uid 500); 1 Jul 2009 05:29:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15446 invoked by uid 500); 1 Jul 2009 05:29:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15436 invoked by uid 99); 1 Jul 2009 05:29:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 05:29:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 05:29:07 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4976E234C004
	for <common-issues@hadoop.apache.org>; Tue, 30 Jun 2009 22:28:47 -0700 (PDT)
Message-ID: <1115140547.1246426127286.JavaMail.jira@brutus>
Date: Tue, 30 Jun 2009 22:28:47 -0700 (PDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6123) hdfs script does not work after
 project split.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12725929#action_12725929 ] 

Sharad Agarwal commented on HADOOP-6123:
----------------------------------------

The scripts won't run as it is for common and hdfs as the ivy lib path has changed. Classpath in hadoop-config.sh needs to include new paths.

> hdfs script does not work after project split.
> ----------------------------------------------
>
>                 Key: HADOOP-6123
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6123
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>             Fix For: 0.21.0
>
>
> There are problems running hdfs script from common.
> # Usage message for hdfs does not have "fs" option anymore.
> # There seem to be an undocumented option "dfs", but using it throws NoClassDefFoundError or ClassNotFoundException.
> # Same with other options e.g. name-node.
> May I am missing something here. How do we call ls these days?
> Do we need to move hdfs script to hdfs project.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-4-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 05:31:00 2009
Return-Path: <common-issues-return-4-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77166 invoked from network); 1 Jul 2009 05:31:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 05:31:00 -0000
Received: (qmail 16598 invoked by uid 500); 1 Jul 2009 05:31:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16576 invoked by uid 500); 1 Jul 2009 05:31:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16566 invoked by uid 99); 1 Jul 2009 05:31:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 05:31:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 05:31:09 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CE2A9234C004
	for <common-issues@hadoop.apache.org>; Tue, 30 Jun 2009 22:30:48 -0700 (PDT)
Message-ID: <1269670427.1246426248830.JavaMail.jira@brutus>
Date: Tue, 30 Jun 2009 22:30:48 -0700 (PDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6123) hdfs script does not work after
 project split.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sharad Agarwal updated HADOOP-6123:
-----------------------------------

    Attachment: 6123_v1.patch

This patch updates the classpath after the project split.
Apply this patch and copy the bin folder in hdfs project. All scripts should work as it is. This eventually has to be fixed in build file of mapred and hdfs which can extract the bin folder out of the common jar.


> hdfs script does not work after project split.
> ----------------------------------------------
>
>                 Key: HADOOP-6123
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6123
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>             Fix For: 0.21.0
>
>         Attachments: 6123_v1.patch
>
>
> There are problems running hdfs script from common.
> # Usage message for hdfs does not have "fs" option anymore.
> # There seem to be an undocumented option "dfs", but using it throws NoClassDefFoundError or ClassNotFoundException.
> # Same with other options e.g. name-node.
> May I am missing something here. How do we call ls these days?
> Do we need to move hdfs script to hdfs project.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-5-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 09:47:01 2009
Return-Path: <common-issues-return-5-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59638 invoked from network); 1 Jul 2009 09:47:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 09:47:01 -0000
Received: (qmail 46447 invoked by uid 500); 1 Jul 2009 09:47:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46423 invoked by uid 500); 1 Jul 2009 09:47:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46413 invoked by uid 99); 1 Jul 2009 09:47:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 09:47:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 09:47:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 8DDDF234C004
	for <common-issues@hadoop.apache.org>; Wed,  1 Jul 2009 02:46:47 -0700 (PDT)
Message-ID: <1548246809.1246441607576.JavaMail.jira@brutus>
Date: Wed, 1 Jul 2009 02:46:47 -0700 (PDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6090) GridMix is broke after upgrading
 random(text)writer to newer mapreduce apis
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amareshwari Sriramadasu updated HADOOP-6090:
--------------------------------------------

    Attachment: 6090.txt

Patch modifying data generation script to use output format from new api.

> GridMix is broke after upgrading random(text)writer to newer mapreduce apis
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-6090
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6090
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: benchmarks
>    Affects Versions: 0.21.0
>            Reporter: Arun C Murthy
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.21.0
>
>         Attachments: 6090.txt
>
>
> GridMix data generation scripts need to use the newer mapreduce api.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-6-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 09:47:06 2009
Return-Path: <common-issues-return-6-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59701 invoked from network); 1 Jul 2009 09:47:06 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 09:47:06 -0000
Received: (qmail 46502 invoked by uid 500); 1 Jul 2009 09:47:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46474 invoked by uid 500); 1 Jul 2009 09:47:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46463 invoked by uid 99); 1 Jul 2009 09:47:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 09:47:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 09:47:07 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 96D94234C046
	for <common-issues@hadoop.apache.org>; Wed,  1 Jul 2009 02:46:47 -0700 (PDT)
Message-ID: <1146116981.1246441607617.JavaMail.jira@brutus>
Date: Wed, 1 Jul 2009 02:46:47 -0700 (PDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6090) GridMix is broke after upgrading
 random(text)writer to newer mapreduce apis
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amareshwari Sriramadasu updated HADOOP-6090:
--------------------------------------------

    Status: Patch Available  (was: Open)

Tested the patch manually by running generateGridmix2data.sh

> GridMix is broke after upgrading random(text)writer to newer mapreduce apis
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-6090
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6090
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: benchmarks
>    Affects Versions: 0.21.0
>            Reporter: Arun C Murthy
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.21.0
>
>         Attachments: 6090.txt
>
>
> GridMix data generation scripts need to use the newer mapreduce api.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 11:10:59 2009
Return-Path: <common-issues-return-7-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85582 invoked from network); 1 Jul 2009 11:10:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 11:10:59 -0000
Received: (qmail 22274 invoked by uid 500); 1 Jul 2009 11:11:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22252 invoked by uid 500); 1 Jul 2009 11:11:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22242 invoked by uid 99); 1 Jul 2009 11:11:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 11:11:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 11:11:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id F0D3B234C1EE
	for <common-issues@hadoop.apache.org>; Wed,  1 Jul 2009 04:10:47 -0700 (PDT)
Message-ID: <2140358728.1246446647985.JavaMail.jira@brutus>
Date: Wed, 1 Jul 2009 04:10:47 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-2366) Space in the value for dfs.data.dir
 can cause great problems
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-2366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726013#action_12726013 ] 

Hudson commented on HADOOP-2366:
--------------------------------

Integrated in Hadoop-Common-trunk #13 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/13/])
    . Support trimmed strings in Configuration.  Contributed by Michele Catasta


> Space in the value for dfs.data.dir can cause great problems
> ------------------------------------------------------------
>
>                 Key: HADOOP-2366
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2366
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>            Reporter: Ted Dunning
>            Assignee: Michele (aka pirroh) Catasta
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-2366.patch
>
>
> The following configuration causes problems:
> <property>
>   <name>dfs.data.dir</name>
>   <value>/mnt/hstore2/hdfs, /home/foo/dfs</value>  
>   <description>
>   Determines where on the local filesystem an DFS data node  should store its bl
> ocks.  If this is a comma-delimited  list of directories, then data will be stor
> ed in all named  directories, typically on different devices.  Directories that 
> do not exist are ignored.  
>   </description>
> </property>
> The problem is that the space after the comma causes the second directory for storage to be " /home/foo/dfs" which is in a directory named <SPACE> which contains a sub-dir named "home" in the hadoop datanodes default directory.  This will typically cause the user's home partition to fill, but will be very hard for the user to understand since a directory with a whitespace name is hard to understand.
> My proposed solution would be to trimLeft all path names from this and similar property after splitting on comma.  This still allows spaces in file and directory names but avoids this problem. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 11:11:06 2009
Return-Path: <common-issues-return-8-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85678 invoked from network); 1 Jul 2009 11:11:06 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 11:11:06 -0000
Received: (qmail 22401 invoked by uid 500); 1 Jul 2009 11:11:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22374 invoked by uid 500); 1 Jul 2009 11:11:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22364 invoked by uid 99); 1 Jul 2009 11:11:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 11:11:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 11:11:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E022A234C04B
	for <common-issues@hadoop.apache.org>; Wed,  1 Jul 2009 04:10:47 -0700 (PDT)
Message-ID: <438678916.1246446647917.JavaMail.jira@brutus>
Date: Wed, 1 Jul 2009 04:10:47 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6080) Handling of  Trash with quota
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726011#action_12726011 ] 

Hudson commented on HADOOP-6080:
--------------------------------

Integrated in Hadoop-Common-trunk #13 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/13/])
     is going to 0.20.
. Introduce -skipTrash option to rm and rmr. Contributed by Jakob Homan.


> Handling of  Trash with quota
> -----------------------------
>
>                 Key: HADOOP-6080
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6080
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>            Reporter: Koji Noguchi
>            Assignee: Jakob Homan
>             Fix For: 0.20.1
>
>         Attachments: HADOOP-6080-v20.patch, HADOOP-6080.patch, javac_warnings_diff.txt
>
>
> Currently with quota turned on, user cannot call '-rmr' on large directory that causes over quota.
> {noformat}
> [knoguchi src]$ hadoop dfs -rmr /tmp/net2
> rmr: Failed to move to trash: hdfs://abc.def.com/tmp/net2
> [knoguchi src]$ hadoop dfs -mv /tmp/net2 /user/knoguchi/.Trash/Current
> mv: org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of /user/knoguchi is exceeded: namespace
> quota=37500 file count=37757, diskspace quota=-1 diskspace=1991250043353
> {noformat}
> Besides from error message being unfriendly, how should this be handled?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 11:11:09 2009
Return-Path: <common-issues-return-9-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85726 invoked from network); 1 Jul 2009 11:11:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 11:11:08 -0000
Received: (qmail 22446 invoked by uid 500); 1 Jul 2009 11:11:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22423 invoked by uid 500); 1 Jul 2009 11:11:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22413 invoked by uid 99); 1 Jul 2009 11:11:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 11:11:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 11:11:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EAF5F234C1EA
	for <common-issues@hadoop.apache.org>; Wed,  1 Jul 2009 04:10:47 -0700 (PDT)
Message-ID: <2083099887.1246446647961.JavaMail.jira@brutus>
Date: Wed, 1 Jul 2009 04:10:47 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6122) 64 javac compiler warnings
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726012#action_12726012 ] 

Hudson commented on HADOOP-6122:
--------------------------------

Integrated in Hadoop-Common-trunk #13 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/13/])
    . The great than operator in test-patch.sh should be "-gt" but not ">".


> 64 javac compiler warnings
> --------------------------
>
>                 Key: HADOOP-6122
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6122
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.21.0
>
>         Attachments: c6122_20090630.patch
>
>
> Ran "ant test-patch" with a empty patch.  Got 
> {noformat}
>      [exec]     -1 javac.  The applied patch generated 64 javac compiler warnings (more than the trunk's current 124 warnings).
>  {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 12:49:02 2009
Return-Path: <common-issues-return-10-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11844 invoked from network); 1 Jul 2009 12:49:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 12:49:01 -0000
Received: (qmail 38904 invoked by uid 500); 1 Jul 2009 12:49:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38879 invoked by uid 500); 1 Jul 2009 12:49:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38869 invoked by uid 99); 1 Jul 2009 12:49:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 12:49:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 12:49:09 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 83D41234C04B
	for <common-issues@hadoop.apache.org>; Wed,  1 Jul 2009 05:48:47 -0700 (PDT)
Message-ID: <688823703.1246452527539.JavaMail.jira@brutus>
Date: Wed, 1 Jul 2009 05:48:47 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6106) Provide an option in
 ShellCommandExecutor to timeout commands that do not complete within a
 certain amount of time.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726039#action_12726039 ] 

Hudson commented on HADOOP-6106:
--------------------------------

Integrated in Hadoop-Hdfs-trunk #9 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/9/])
    . Updated hadoop-core and test jars from hudson trunk #12


> Provide an option in ShellCommandExecutor to timeout commands that do not complete within a certain amount of time.
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Hemanth Yamijala
>            Assignee: Sreekanth Ramakrishnan
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6106-1.patch, HADOOP-6106-2.patch, HADOOP-6106.patch, mapred-211-common-3.patch
>
>
> In MAPREDUCE-211 we came across a need to provide an option to timeout commands launched via the ShellCommandExecutor. The use case is for the health check script being developed in MAPREDUCE-211. We would like the TaskTracker thread to not be blocked by a problematic script or in instances where fork()+exec() has hung (which apparently has been observed in large clusters).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-11-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 12:49:08 2009
Return-Path: <common-issues-return-11-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11884 invoked from network); 1 Jul 2009 12:49:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 12:49:08 -0000
Received: (qmail 38962 invoked by uid 500); 1 Jul 2009 12:49:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38938 invoked by uid 500); 1 Jul 2009 12:49:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38928 invoked by uid 99); 1 Jul 2009 12:49:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 12:49:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 12:49:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4E874234C004
	for <common-issues@hadoop.apache.org>; Wed,  1 Jul 2009 05:48:47 -0700 (PDT)
Message-ID: <2071225828.1246452527316.JavaMail.jira@brutus>
Date: Wed, 1 Jul 2009 05:48:47 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6122) 64 javac compiler warnings
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726038#action_12726038 ] 

Hudson commented on HADOOP-6122:
--------------------------------

Integrated in Hadoop-Hdfs-trunk #9 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/9/])
    . The great than operator in test-patch.sh should be "-gt" but not ">".


> 64 javac compiler warnings
> --------------------------
>
>                 Key: HADOOP-6122
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6122
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.21.0
>
>         Attachments: c6122_20090630.patch
>
>
> Ran "ant test-patch" with a empty patch.  Got 
> {noformat}
>      [exec]     -1 javac.  The applied patch generated 64 javac compiler warnings (more than the trunk's current 124 warnings).
>  {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-12-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 13:55:59 2009
Return-Path: <common-issues-return-12-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43417 invoked from network); 1 Jul 2009 13:55:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 13:55:59 -0000
Received: (qmail 45255 invoked by uid 500); 1 Jul 2009 13:56:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45188 invoked by uid 500); 1 Jul 2009 13:56:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45167 invoked by uid 99); 1 Jul 2009 13:56:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 13:56:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 13:56:07 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 679B9234C004
	for <common-issues@hadoop.apache.org>; Wed,  1 Jul 2009 06:55:47 -0700 (PDT)
Message-ID: <1169957006.1246456547410.JavaMail.jira@brutus>
Date: Wed, 1 Jul 2009 06:55:47 -0700 (PDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6121) High Availability support for
 Hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726067#action_12726067 ] 

Steve Loughran commented on HADOOP-6121:
----------------------------------------

The JobTracker can persist its state, but for HA you'd need to do failover in the clients- currently to achieve that you need to switch the replacement to the same hostname/IP address of the original.

The issue of HA for HDFS is raised in HDFS-243

> High Availability support for Hadoop
> ------------------------------------
>
>                 Key: HADOOP-6121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6121
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: dfs, mapred
>            Reporter: Jie Qiu
>
> Currently, We look at the HA of Hadoop cluster. We need to consider the NameNode HA as well as Jobtracker HA. For NameNode, we want to build primary/standy or master-slaves pattern to provide NameNode HA. Therefore, we need to consider how to ship log between primary/standby/slaves and how commit "write" operation to NameNode after the agreement among primary/standby/slaves on log. Whether will we use Linux HA package or NameNode-built-in HA package without the help of outter Linux HA package. 
> After NameNode become high availability, is it necessary to provide HA for Jobtracker? Can Jobtracker  persist the states of Jobs and tasks into HA NameNode? Or Jobtracker also needs the same approach from NameNode for HA support.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-13-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 17:21:59 2009
Return-Path: <common-issues-return-13-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39502 invoked from network); 1 Jul 2009 17:21:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 17:21:59 -0000
Received: (qmail 15004 invoked by uid 500); 1 Jul 2009 17:22:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14978 invoked by uid 500); 1 Jul 2009 17:22:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14918 invoked by uid 99); 1 Jul 2009 17:22:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 17:22:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 17:22:07 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3A61D234C045
	for <common-issues@hadoop.apache.org>; Wed,  1 Jul 2009 10:21:47 -0700 (PDT)
Message-ID: <330149198.1246468907238.JavaMail.jira@brutus>
Date: Wed, 1 Jul 2009 10:21:47 -0700 (PDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6114) bug in documentation:
 org.apache.hadoop.fs.FileStatus.getLen()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-6114:
-------------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I just committed this. Thanks Dmitry.


> bug in documentation: org.apache.hadoop.fs.FileStatus.getLen() 
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6114
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6114
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.18.3, 0.20.0
>            Reporter: Dmitry Rzhevskiy
>            Assignee: Dmitry Rzhevskiy
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6114.patch
>
>
> In javadoc method  org.apache.hadoop.fs.FileStatus.getLen()  writtend that this method "return the length of this file, in blocks"
> But method return size in bytes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-14-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 17:52:01 2009
Return-Path: <common-issues-return-14-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61041 invoked from network); 1 Jul 2009 17:52:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 17:52:01 -0000
Received: (qmail 56432 invoked by uid 500); 1 Jul 2009 17:52:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56404 invoked by uid 500); 1 Jul 2009 17:52:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56390 invoked by uid 99); 1 Jul 2009 17:52:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 17:52:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 17:52:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 57CD3234C04B
	for <common-issues@hadoop.apache.org>; Wed,  1 Jul 2009 10:51:47 -0700 (PDT)
Message-ID: <1756911292.1246470707358.JavaMail.jira@brutus>
Date: Wed, 1 Jul 2009 10:51:47 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726151#action_12726151 ] 

Owen O'Malley commented on HADOOP-5170:
---------------------------------------

I'm sorry I'm coming in late on this. Are these new knobs at the job level? I think this the wrong direction. In particular, just limiting the number of slots won't do any good. The high ram job processing is a much better model. So rather than declaring max number of maps or reduces, we should allow "large task" jobs where each task is given multiple slots.I think we should revert this before it is released.

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-15-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 18:05:59 2009
Return-Path: <common-issues-return-15-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69748 invoked from network); 1 Jul 2009 18:05:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 18:05:59 -0000
Received: (qmail 73336 invoked by uid 500); 1 Jul 2009 18:06:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73308 invoked by uid 500); 1 Jul 2009 18:06:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73298 invoked by uid 99); 1 Jul 2009 18:06:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 18:06:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 18:06:07 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4A51A234C04B
	for <common-issues@hadoop.apache.org>; Wed,  1 Jul 2009 11:05:47 -0700 (PDT)
Message-ID: <223402576.1246471547303.JavaMail.jira@brutus>
Date: Wed, 1 Jul 2009 11:05:47 -0700 (PDT)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6121) High Availability support for
 Hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726160#action_12726160 ] 

Scott Carey commented on HADOOP-6121:
-------------------------------------

If you really want to go very robust with the JobTracker, you could implement it by embedding ZooKeeper and using it for all the RPC coordination and state tracking.  This is a more natural fit than ZK for the NN, IMO.

A job could be a node in ZK, TaskTrackers would be able to see what work they should do by looking in specific nodes  -- and they can register their status and health as well.  Something like a Reduce job getting the map outputs it needs would simply be listing children from a reduce node for a job. 
Fault tolerance would come for free, and this form of RPC would likely be more efficient than the current one.

Furthermore, with watches firing off, there would be no ping-response sleep delays which would improve performance, and significantly improve latency for latency sensitive jobs.

> High Availability support for Hadoop
> ------------------------------------
>
>                 Key: HADOOP-6121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6121
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: dfs, mapred
>            Reporter: Jie Qiu
>
> Currently, We look at the HA of Hadoop cluster. We need to consider the NameNode HA as well as Jobtracker HA. For NameNode, we want to build primary/standy or master-slaves pattern to provide NameNode HA. Therefore, we need to consider how to ship log between primary/standby/slaves and how commit "write" operation to NameNode after the agreement among primary/standby/slaves on log. Whether will we use Linux HA package or NameNode-built-in HA package without the help of outter Linux HA package. 
> After NameNode become high availability, is it necessary to provide HA for Jobtracker? Can Jobtracker  persist the states of Jobs and tasks into HA NameNode? Or Jobtracker also needs the same approach from NameNode for HA support.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-16-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 18:20:07 2009
Return-Path: <common-issues-return-16-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87032 invoked from network); 1 Jul 2009 18:20:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 18:20:07 -0000
Received: (qmail 90917 invoked by uid 500); 1 Jul 2009 18:20:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90888 invoked by uid 500); 1 Jul 2009 18:20:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90878 invoked by uid 99); 1 Jul 2009 18:20:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 18:20:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 18:20:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 35CB7234C004
	for <common-issues@hadoop.apache.org>; Wed,  1 Jul 2009 11:19:47 -0700 (PDT)
Message-ID: <1916530659.1246472387203.JavaMail.jira@brutus>
Date: Wed, 1 Jul 2009 11:19:47 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6124) patchJavacWarnings and
 trunkJavacWarnings are not consistent.
In-Reply-To: <1339799848.1246409207795.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6124:
-------------------------------------------

    Attachment: c6124_20090701.patch

> ...  HDFS and MapReduce seem not having this problem.
I was wrong.  HDFS and MapReduce also have this problem.

The project is not cleaned when patchJavacWarnings.txt is generated.  Some java source files may be already compiled and they will not be compiled again.  As a result, the warnings inside those files won't be reported.  Note that there is a clean when trunkJavacWarnings.txt is generated.

c6124_20090701.patch: add clean before tar

> patchJavacWarnings and trunkJavacWarnings are not consistent.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6124
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6124
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>         Attachments: c6124_20090701.patch
>
>
> The values of patchJavacWarnings and trunkJavacWarnings are not consistent when running test-patch.sh with an empty patch over Common.  HDFS and MapReduce seem not having this problem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-17-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 18:52:02 2009
Return-Path: <common-issues-return-17-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1589 invoked from network); 1 Jul 2009 18:52:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 18:52:01 -0000
Received: (qmail 38463 invoked by uid 500); 1 Jul 2009 18:52:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38433 invoked by uid 500); 1 Jul 2009 18:52:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38393 invoked by uid 99); 1 Jul 2009 18:52:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 18:52:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 18:52:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 76E3C234C004
	for <common-issues@hadoop.apache.org>; Wed,  1 Jul 2009 11:51:47 -0700 (PDT)
Message-ID: <2146752588.1246474307473.JavaMail.jira@brutus>
Date: Wed, 1 Jul 2009 11:51:47 -0700 (PDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726179#action_12726179 ] 

Arun C Murthy commented on HADOOP-5170:
---------------------------------------

bq. I'm sorry I'm coming in late on this.

Ditto.

bq. Are these new knobs at the job level? I think this the wrong direction. In particular, just limiting the number of slots won't do any good. The high ram job processing is a much better model. So rather than declaring max number of maps or reduces, we should allow "large task" jobs where each task is given multiple slots.

Couldn't agree more. See MAPREDUCE-516.

I see the following issues:

mapred.max.{maps|reduces}.per.node

 As of today, the framework will not be able to guard against users who take up a node and consume all resources (cpu, memory, disk etc.) and starve other user tasks running on the machines. This goes against the spirit of shared compute/storage clusters. I can see an argument being made for this feature once we can figure how to *charge* the user based on the task's total resource consumption; however we are a long way away from this. We have taken a step along this direction by introducing the High RAM Jobs feature in the Capacity Scheduler, we have a long way to go.

mapred.max.{maps|reduces}.per.cluster

Given that we mostly agree that the high-ram jobs are the right model, these features interact very badly with each other.

Consider a high-ram job which has mapred.max.maps.per.cluster set to 100 and a few thousand tasks (a fraction of which is sufficient to exhaust the capacity of it's queue).
Currently the CapacityScheduler starts 'reserving' slots (after charging the user, queue etc.) to satisfy the resource requirements of the job.

Now we have 2 choices if we choose to incorporate mapred.max.{map|reduces}.per.cluster:
# Do not 'reserve' more tasktrackers than mapred.max.maps.per.cluster
# Reserve as much as needed but start 'unreserving' as soon as we have scheduled 'mapred.max.maps.per.cluster' number of maps.

Either way we are in trouble.

# The high-ram job is seriously starved. The scheduler has to pick only 100 nodes and no more. If they happened to be bad bets (other long running tasks etc.) the high-ram job needs to wait for a long while. Furthermore, the other bad side-effect is that the high-ram job gets *pinned* to the first 100 nodes and this really hurts locality of its tasks.
# The cluster is severely under-utilized since we may reserve all of the queue's capacity and then realize that we have to start 'unreserving'. Once the first wave of maps of the high-ram job are done we rinse and repeat for each wave of mapred.max.maps.per.cluster maps, there-by keeping the cluster idle for large amounts of time.


bq. I think we should revert this before it is released.

Unfortunately, yes. +1

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-18-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 20:18:01 2009
Return-Path: <common-issues-return-18-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39418 invoked from network); 1 Jul 2009 20:18:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 20:18:01 -0000
Received: (qmail 93439 invoked by uid 500); 1 Jul 2009 20:18:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93412 invoked by uid 500); 1 Jul 2009 20:18:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93402 invoked by uid 99); 1 Jul 2009 20:18:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 20:18:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 20:18:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3FE62234C004
	for <common-issues@hadoop.apache.org>; Wed,  1 Jul 2009 13:17:47 -0700 (PDT)
Message-ID: <999068112.1246479467247.JavaMail.jira@brutus>
Date: Wed, 1 Jul 2009 13:17:47 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5640) Allow ServicePlugins to hook
 callbacks into key service events
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-5640:
--------------------------------

    Attachment: hadoop-5640.txt

> Allow ServicePlugins to hook callbacks into key service events
> --------------------------------------------------------------
>
>                 Key: HADOOP-5640
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5640
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-5640.txt, hadoop-5640.txt, HADOOP-5640.v2.txt, hadoop-5640.v3.txt
>
>
> HADOOP-5257 added the ability for NameNode and DataNode to start and stop ServicePlugin implementations at NN/DN start/stop. However, this is insufficient integration for some common use cases.
> We should add some functionality for Plugins to subscribe to events generated by the service they're plugging into. Some potential hook points are:
> NameNode:
>   - new datanode registered
>   - datanode has died
>   - exception caught
>   - etc?
> DataNode:
>   - startup
>   - initial registration with NN complete (this is important for HADOOP-4707 to sync up datanode.dnRegistration.name with the NN-side registration)
>   - namenode reconnect
>   - some block transfer hooks?
>   - exception caught
> I see two potential routes for implementation:
> 1) We make an enum for the types of hookpoints and have a general function in the ServicePlugin interface. Something like:
> {code:java}
> enum HookPoint {
>   DN_STARTUP,
>   DN_RECEIVED_NEW_BLOCK,
>   DN_CAUGHT_EXCEPTION,
>  ...
> }
> void runHook(HookPoint hp, Object value);
> {code}
> 2) We make classes specific to each "pluggable" as was originally suggested in HADDOP-5257. Something like:
> {code:java}
> class DataNodePlugin {
>   void datanodeStarted() {}
>   void receivedNewBlock(block info, etc) {}
>   void caughtException(Exception e) {}
>   ...
> }
> {code}
> I personally prefer option (2) since we can ensure plugin API compatibility at compile-time, and we avoid an ugly switch statement in a runHook() function.
> Interested to hear what people's thoughts are here.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-19-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 20:24:04 2009
Return-Path: <common-issues-return-19-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41055 invoked from network); 1 Jul 2009 20:24:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 20:24:01 -0000
Received: (qmail 3650 invoked by uid 500); 1 Jul 2009 20:24:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3633 invoked by uid 500); 1 Jul 2009 20:24:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3623 invoked by uid 99); 1 Jul 2009 20:24:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 20:24:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 20:24:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 38889234C004
	for <common-issues@hadoop.apache.org>; Wed,  1 Jul 2009 13:23:47 -0700 (PDT)
Message-ID: <1652736510.1246479827217.JavaMail.jira@brutus>
Date: Wed, 1 Jul 2009 13:23:47 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5640) Allow ServicePlugins to hook
 callbacks into key service events
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726212#action_12726212 ] 

Todd Lipcon commented on HADOOP-5640:
-------------------------------------

bq. The plugin design is actually the listener pattern. For API extensibility it might be worth considering a ServiceEvent class that is passed to the methods of ServicePlugin and subclasses so that extra context information can be passed without breaking plugins. Also, ServicePlugin should be an abstract class.

The ServicePlugin interface is a holdover from a previous JIRA HADOOP-5257, so I didn't want to modify that. Hopefully Carlos can chime in with regard to this, since he wrote that original patch. Can you explain more about the ServiceEvent idea? I think this is similar to one of the ideas discussed above, but I think it's cleaner to use actual methods with full signatures rather than exposing a single "handleEvent(ServiceEvent e)" and switching on "e instanceof FooEvent" vs "e instanceof BarEvent".

bq. Rename PluginDispatcher to ServicePluginDispatcher. There are other plugins (e.g. MemoryCalculatorPlugin) so it's worth being more precise.

Fixed in attached patch.

bq. SingleArgumentRunnable might be named ServicePluginCallback to better indicate its use.

I disagree - this is a very generic interface that is quite usable for a lot of different things. Since this code is going in util, we might as well make it available for other use cases as well rather than proliferating several interfaces that have the same use.


The new patch I just attached is for the Common portion. I'll make a new JIRA in HDFS for the NameNodePlugin and DataNodePlugin portions of the patch.

> Allow ServicePlugins to hook callbacks into key service events
> --------------------------------------------------------------
>
>                 Key: HADOOP-5640
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5640
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-5640.txt, hadoop-5640.txt, HADOOP-5640.v2.txt, hadoop-5640.v3.txt
>
>
> HADOOP-5257 added the ability for NameNode and DataNode to start and stop ServicePlugin implementations at NN/DN start/stop. However, this is insufficient integration for some common use cases.
> We should add some functionality for Plugins to subscribe to events generated by the service they're plugging into. Some potential hook points are:
> NameNode:
>   - new datanode registered
>   - datanode has died
>   - exception caught
>   - etc?
> DataNode:
>   - startup
>   - initial registration with NN complete (this is important for HADOOP-4707 to sync up datanode.dnRegistration.name with the NN-side registration)
>   - namenode reconnect
>   - some block transfer hooks?
>   - exception caught
> I see two potential routes for implementation:
> 1) We make an enum for the types of hookpoints and have a general function in the ServicePlugin interface. Something like:
> {code:java}
> enum HookPoint {
>   DN_STARTUP,
>   DN_RECEIVED_NEW_BLOCK,
>   DN_CAUGHT_EXCEPTION,
>  ...
> }
> void runHook(HookPoint hp, Object value);
> {code}
> 2) We make classes specific to each "pluggable" as was originally suggested in HADDOP-5257. Something like:
> {code:java}
> class DataNodePlugin {
>   void datanodeStarted() {}
>   void receivedNewBlock(block info, etc) {}
>   void caughtException(Exception e) {}
>   ...
> }
> {code}
> I personally prefer option (2) since we can ensure plugin API compatibility at compile-time, and we avoid an ugly switch statement in a runHook() function.
> Interested to hear what people's thoughts are here.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-20-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 21:04:31 2009
Return-Path: <common-issues-return-20-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59020 invoked from network); 1 Jul 2009 21:04:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 21:04:30 -0000
Received: (qmail 81640 invoked by uid 500); 1 Jul 2009 21:04:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81619 invoked by uid 500); 1 Jul 2009 21:04:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Delivered-To: moderator for common-issues@hadoop.apache.org
Received: (qmail 94251 invoked by uid 99); 1 Jul 2009 00:47:18 -0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Message-ID: <1339799848.1246409207795.JavaMail.jira@brutus>
Date: Tue, 30 Jun 2009 17:46:47 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6124) patchJavacWarnings and
 trunkJavacWarnings are not consistent.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

patchJavacWarnings and trunkJavacWarnings are not consistent.
-------------------------------------------------------------

                 Key: HADOOP-6124
                 URL: https://issues.apache.org/jira/browse/HADOOP-6124
             Project: Hadoop Common
          Issue Type: Bug
          Components: build
            Reporter: Tsz Wo (Nicholas), SZE


The values of patchJavacWarnings and trunkJavacWarnings are not consistent when running test-patch.sh with an empty patch over Common.  HDFS and MapReduce seem not having this problem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-21-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 21:04:48 2009
Return-Path: <common-issues-return-21-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59618 invoked from network); 1 Jul 2009 21:04:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 21:04:48 -0000
Received: (qmail 83019 invoked by uid 500); 1 Jul 2009 21:04:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83000 invoked by uid 500); 1 Jul 2009 21:04:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Delivered-To: moderator for common-issues@hadoop.apache.org
Received: (qmail 5509 invoked by uid 99); 1 Jul 2009 00:57:18 -0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Message-ID: <1342457673.1246409807273.JavaMail.jira@brutus>
Date: Tue, 30 Jun 2009 17:56:47 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6122) 64 javac compiler warnings
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12725872#action_12725872 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6122:
------------------------------------------------

Filed HADOOP-6124 for the inconsistency problem.

Tried test-patch.sh with the patch.  It worked fine.

> 64 javac compiler warnings
> --------------------------
>
>                 Key: HADOOP-6122
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6122
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6122_20090630.patch
>
>
> Ran "ant test-patch" with a empty patch.  Got 
> {noformat}
>      [exec]     -1 javac.  The applied patch generated 64 javac compiler warnings (more than the trunk's current 124 warnings).
>  {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-22-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 21:05:01 2009
Return-Path: <common-issues-return-22-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59716 invoked from network); 1 Jul 2009 21:05:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 21:05:00 -0000
Received: (qmail 83176 invoked by uid 500); 1 Jul 2009 21:05:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83150 invoked by uid 500); 1 Jul 2009 21:05:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Delivered-To: moderator for common-issues@hadoop.apache.org
Received: (qmail 17704 invoked by uid 99); 1 Jul 2009 01:27:11 -0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Message-ID: <1031327.1246411607264.JavaMail.jira@brutus>
Date: Tue, 30 Jun 2009 18:26:47 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6122) 64 javac compiler warnings
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE resolved HADOOP-6122.
--------------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.21.0
     Hadoop Flags: [Reviewed]

I have committed this.

> 64 javac compiler warnings
> --------------------------
>
>                 Key: HADOOP-6122
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6122
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.21.0
>
>         Attachments: c6122_20090630.patch
>
>
> Ran "ant test-patch" with a empty patch.  Got 
> {noformat}
>      [exec]     -1 javac.  The applied patch generated 64 javac compiler warnings (more than the trunk's current 124 warnings).
>  {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-23-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 21:52:50 2009
Return-Path: <common-issues-return-23-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91695 invoked from network); 1 Jul 2009 21:52:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 21:52:50 -0000
Received: (qmail 31138 invoked by uid 500); 1 Jul 2009 21:38:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31094 invoked by uid 500); 1 Jul 2009 21:38:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31083 invoked by uid 99); 1 Jul 2009 21:38:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 21:38:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 21:38:07 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 46B86234C053
	for <common-issues@hadoop.apache.org>; Wed,  1 Jul 2009 14:37:47 -0700 (PDT)
Message-ID: <682341596.1246484267288.JavaMail.jira@brutus>
Date: Wed, 1 Jul 2009 14:37:47 -0700 (PDT)
From: "Philip Zeyliger (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6125) extend DistributedCache to work
 locally (LocalJobRunner) (common half)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

extend DistributedCache to work locally (LocalJobRunner) (common half)
----------------------------------------------------------------------

                 Key: HADOOP-6125
                 URL: https://issues.apache.org/jira/browse/HADOOP-6125
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Philip Zeyliger
            Assignee: Philip Zeyliger
            Priority: Minor


This is the co-ticket to MAPREDUCE-476, covering the significant part of DistributedCache that's in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-24-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 21:52:58 2009
Return-Path: <common-issues-return-24-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92238 invoked from network); 1 Jul 2009 21:52:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 21:52:57 -0000
Received: (qmail 31858 invoked by uid 500); 1 Jul 2009 21:42:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31828 invoked by uid 500); 1 Jul 2009 21:42:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31818 invoked by uid 99); 1 Jul 2009 21:42:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 21:42:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 21:42:07 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 97021234C004
	for <common-issues@hadoop.apache.org>; Wed,  1 Jul 2009 14:41:47 -0700 (PDT)
Message-ID: <1231587512.1246484507603.JavaMail.jira@brutus>
Date: Wed, 1 Jul 2009 14:41:47 -0700 (PDT)
From: "Philip Zeyliger (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6125) extend DistributedCache to work
 locally (LocalJobRunner) (common half)
In-Reply-To: <682341596.1246484267288.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Philip Zeyliger updated HADOOP-6125:
------------------------------------

    Attachment: HADOOP-6125.patch

Regenerated patches from HADOOP-2914.  I used:

bq. cat HADOOP-2914-v3.patch | sed -e 's,src/core/,src/java/,' | patch -p0

> extend DistributedCache to work locally (LocalJobRunner) (common half)
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6125
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6125
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Philip Zeyliger
>            Assignee: Philip Zeyliger
>            Priority: Minor
>         Attachments: HADOOP-6125.patch
>
>
> This is the co-ticket to MAPREDUCE-476, covering the significant part of DistributedCache that's in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-25-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 01 22:23:01 2009
Return-Path: <common-issues-return-25-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11999 invoked from network); 1 Jul 2009 22:23:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Jul 2009 22:23:01 -0000
Received: (qmail 98159 invoked by uid 500); 1 Jul 2009 22:23:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98130 invoked by uid 500); 1 Jul 2009 22:23:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98120 invoked by uid 99); 1 Jul 2009 22:23:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 22:23:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jul 2009 22:23:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EED66234C04B
	for <common-issues@hadoop.apache.org>; Wed,  1 Jul 2009 15:22:47 -0700 (PDT)
Message-ID: <884777006.1246486967977.JavaMail.jira@brutus>
Date: Wed, 1 Jul 2009 15:22:47 -0700 (PDT)
From: "Philip Zeyliger (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6125) extend DistributedCache to work
 locally (LocalJobRunner) (common half)
In-Reply-To: <682341596.1246484267288.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Philip Zeyliger updated HADOOP-6125:
------------------------------------

    Status: Patch Available  (was: Open)

> extend DistributedCache to work locally (LocalJobRunner) (common half)
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6125
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6125
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Philip Zeyliger
>            Assignee: Philip Zeyliger
>            Priority: Minor
>         Attachments: HADOOP-6125.patch
>
>
> This is the co-ticket to MAPREDUCE-476, covering the significant part of DistributedCache that's in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-26-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 02:10:59 2009
Return-Path: <common-issues-return-26-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3321 invoked from network); 2 Jul 2009 02:10:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 02:10:59 -0000
Received: (qmail 51501 invoked by uid 500); 2 Jul 2009 02:11:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51471 invoked by uid 500); 2 Jul 2009 02:11:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51461 invoked by uid 99); 2 Jul 2009 02:11:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 02:11:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 02:11:07 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 70B00234C004
	for <common-issues@hadoop.apache.org>; Wed,  1 Jul 2009 19:10:47 -0700 (PDT)
Message-ID: <599207977.1246500647448.JavaMail.jira@brutus>
Date: Wed, 1 Jul 2009 19:10:47 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-3315) New binary file format
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-3315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-3315:
------------------------------

    Attachment: hadoop-3315-0701-yhadoop-20.patch

Patch attached for yahoo 0.20 branch.

> New binary file format
> ----------------------
>
>                 Key: HADOOP-3315
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3315
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: Owen O'Malley
>            Assignee: Hong Tang
>             Fix For: 0.21.0
>
>         Attachments: hadoop-3315-0507.patch, hadoop-3315-0509-2.patch, hadoop-3315-0509.patch, hadoop-3315-0513.patch, hadoop-3315-0514.patch, hadoop-3315-0601.patch, hadoop-3315-0602.patch, hadoop-3315-0605.patch, hadoop-3315-0612.patch, hadoop-3315-0623-2.patch, hadoop-3315-0701-yhadoop-20.patch, HADOOP-3315_20080908_TFILE_PREVIEW_WITH_LZO_TESTS.patch, HADOOP-3315_20080915_TFILE.patch, hadoop-trunk-tfile.patch, hadoop-trunk-tfile.patch, TFile Specification 20081217.pdf
>
>
> SequenceFile's block compression format is too complex and requires 4 codecs to compress or decompress. It would be good to have a file format that only needs 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-27-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 03:37:06 2009
Return-Path: <common-issues-return-27-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32491 invoked from network); 2 Jul 2009 03:37:06 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 03:37:06 -0000
Received: (qmail 5311 invoked by uid 500); 2 Jul 2009 03:37:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5284 invoked by uid 500); 2 Jul 2009 03:37:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5274 invoked by uid 99); 2 Jul 2009 03:37:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 03:37:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 03:37:07 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id A9DDA234C046
	for <common-issues@hadoop.apache.org>; Wed,  1 Jul 2009 20:36:47 -0700 (PDT)
Message-ID: <572617315.1246505807694.JavaMail.jira@brutus>
Date: Wed, 1 Jul 2009 20:36:47 -0700 (PDT)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6124) patchJavacWarnings and
 trunkJavacWarnings are not consistent.
In-Reply-To: <1339799848.1246409207795.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nigel Daley reassigned HADOOP-6124:
-----------------------------------

    Assignee: Giridharan Kesavan

> patchJavacWarnings and trunkJavacWarnings are not consistent.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6124
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6124
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Giridharan Kesavan
>         Attachments: c6124_20090701.patch
>
>
> The values of patchJavacWarnings and trunkJavacWarnings are not consistent when running test-patch.sh with an empty patch over Common.  HDFS and MapReduce seem not having this problem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-28-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 06:05:01 2009
Return-Path: <common-issues-return-28-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95885 invoked from network); 2 Jul 2009 06:05:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 06:05:01 -0000
Received: (qmail 34014 invoked by uid 500); 2 Jul 2009 06:05:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33878 invoked by uid 500); 2 Jul 2009 06:05:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33868 invoked by uid 99); 2 Jul 2009 06:05:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 06:05:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 06:05:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 9AA62234C004
	for <common-issues@hadoop.apache.org>; Wed,  1 Jul 2009 23:04:47 -0700 (PDT)
Message-ID: <1570482149.1246514687619.JavaMail.jira@brutus>
Date: Wed, 1 Jul 2009 23:04:47 -0700 (PDT)
From: "Jie Qiu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6121) High Availability support for
 Hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726339#action_12726339 ] 

Jie Qiu commented on HADOOP-6121:
---------------------------------

Good suggestion on ZK with Jobtracker. 

Do you know who is implementing ZK for the NameNode? How doese it works?  Currently, we employed the database-like replication approach to ship log from primary to standby(multiple standbys).  We also plan to support different level synchnization, i.e. tight, loose. 

For tight mode, we guarantee primary disk write and standby disk write. For loose mode, we guarantee primary disk write and standby receive log.

> High Availability support for Hadoop
> ------------------------------------
>
>                 Key: HADOOP-6121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6121
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: dfs, mapred
>            Reporter: Jie Qiu
>
> Currently, We look at the HA of Hadoop cluster. We need to consider the NameNode HA as well as Jobtracker HA. For NameNode, we want to build primary/standy or master-slaves pattern to provide NameNode HA. Therefore, we need to consider how to ship log between primary/standby/slaves and how commit "write" operation to NameNode after the agreement among primary/standby/slaves on log. Whether will we use Linux HA package or NameNode-built-in HA package without the help of outter Linux HA package. 
> After NameNode become high availability, is it necessary to provide HA for Jobtracker? Can Jobtracker  persist the states of Jobs and tasks into HA NameNode? Or Jobtracker also needs the same approach from NameNode for HA support.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-29-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 07:38:59 2009
Return-Path: <common-issues-return-29-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34647 invoked from network); 2 Jul 2009 07:38:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 07:38:59 -0000
Received: (qmail 47729 invoked by uid 500); 2 Jul 2009 07:39:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47701 invoked by uid 500); 2 Jul 2009 07:39:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47612 invoked by uid 99); 2 Jul 2009 07:39:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 07:39:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 07:39:07 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 48AD5234C004
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 00:38:47 -0700 (PDT)
Message-ID: <1497450433.1246520327283.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 00:38:47 -0700 (PDT)
From: "Carlos Valiente (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5640) Allow ServicePlugins to hook
 callbacks into key service events
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726366#action_12726366 ] 

Carlos Valiente commented on HADOOP-5640:
-----------------------------------------

{quote}
bq. The plugin design is actually the listener pattern. For API extensibility it might be worth considering a ServiceEvent class that is passed to the methods of ServicePlugin and subclasses so that extra context information can be passed without breaking plugins.

The ServicePlugin interface is a holdover from a previous JIRA HADOOP-5257, so I didn't want to modify that. Hopefully Carlos can chime in with regard to this, since he wrote that original patch.
{quote}

Sounds good to me. We could even go further and have just one single method in {{ServicePlugin}}:

{code}
public abstract class ServicePlugin {

    public abstract void onEvent(ServiceEvent event);
}
{code}

,instead of the current:

{code}
public interface ServicePlugin extends Closeable {

    void start(Object service);

    void stop();
}
{code}

I guess the {{ServiceEvent}} field would have a reference to the {{Object service}} parameter, among other things.

> Allow ServicePlugins to hook callbacks into key service events
> --------------------------------------------------------------
>
>                 Key: HADOOP-5640
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5640
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-5640.txt, hadoop-5640.txt, HADOOP-5640.v2.txt, hadoop-5640.v3.txt
>
>
> HADOOP-5257 added the ability for NameNode and DataNode to start and stop ServicePlugin implementations at NN/DN start/stop. However, this is insufficient integration for some common use cases.
> We should add some functionality for Plugins to subscribe to events generated by the service they're plugging into. Some potential hook points are:
> NameNode:
>   - new datanode registered
>   - datanode has died
>   - exception caught
>   - etc?
> DataNode:
>   - startup
>   - initial registration with NN complete (this is important for HADOOP-4707 to sync up datanode.dnRegistration.name with the NN-side registration)
>   - namenode reconnect
>   - some block transfer hooks?
>   - exception caught
> I see two potential routes for implementation:
> 1) We make an enum for the types of hookpoints and have a general function in the ServicePlugin interface. Something like:
> {code:java}
> enum HookPoint {
>   DN_STARTUP,
>   DN_RECEIVED_NEW_BLOCK,
>   DN_CAUGHT_EXCEPTION,
>  ...
> }
> void runHook(HookPoint hp, Object value);
> {code}
> 2) We make classes specific to each "pluggable" as was originally suggested in HADDOP-5257. Something like:
> {code:java}
> class DataNodePlugin {
>   void datanodeStarted() {}
>   void receivedNewBlock(block info, etc) {}
>   void caughtException(Exception e) {}
>   ...
> }
> {code}
> I personally prefer option (2) since we can ensure plugin API compatibility at compile-time, and we avoid an ugly switch statement in a runHook() function.
> Interested to hear what people's thoughts are here.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-30-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 07:45:01 2009
Return-Path: <common-issues-return-30-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36671 invoked from network); 2 Jul 2009 07:45:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 07:45:01 -0000
Received: (qmail 53889 invoked by uid 500); 2 Jul 2009 07:45:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53859 invoked by uid 500); 2 Jul 2009 07:45:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53849 invoked by uid 99); 2 Jul 2009 07:45:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 07:45:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 07:45:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 36539234C004
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 00:44:47 -0700 (PDT)
Message-ID: <338098663.1246520687208.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 00:44:47 -0700 (PDT)
From: "Jie Qiu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6121) High Availability support for
 Hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726370#action_12726370 ] 

Jie Qiu commented on HADOOP-6121:
---------------------------------

Currently, in my experiment, my approach follows:

1. Hook Hadoop for Memory write and Disk write.  
     Memory write means changes to the BTree, the corresponding log operations (OP_ADD, OP_RENAME, OP_DELETE,....)
     Disk write means write the log to disks

2. Replication log from primary node to slaves(one standy at this stage)

3. Replay log in slaves

4. Standby heartbeat primary and take over if it feels primary down

I would like to know other approaches for Hadoop HA on NameNode. What's additional requirements on Hadoop HA?


> High Availability support for Hadoop
> ------------------------------------
>
>                 Key: HADOOP-6121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6121
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: dfs, mapred
>            Reporter: Jie Qiu
>
> Currently, We look at the HA of Hadoop cluster. We need to consider the NameNode HA as well as Jobtracker HA. For NameNode, we want to build primary/standy or master-slaves pattern to provide NameNode HA. Therefore, we need to consider how to ship log between primary/standby/slaves and how commit "write" operation to NameNode after the agreement among primary/standby/slaves on log. Whether will we use Linux HA package or NameNode-built-in HA package without the help of outter Linux HA package. 
> After NameNode become high availability, is it necessary to provide HA for Jobtracker? Can Jobtracker  persist the states of Jobs and tasks into HA NameNode? Or Jobtracker also needs the same approach from NameNode for HA support.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-31-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 11:52:01 2009
Return-Path: <common-issues-return-31-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35887 invoked from network); 2 Jul 2009 11:52:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 11:52:01 -0000
Received: (qmail 43143 invoked by uid 500); 2 Jul 2009 11:52:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43113 invoked by uid 500); 2 Jul 2009 11:52:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43103 invoked by uid 99); 2 Jul 2009 11:52:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 11:52:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 11:52:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 70F27234C004
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 04:51:47 -0700 (PDT)
Message-ID: <594915485.1246535507448.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 04:51:47 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6114) bug in documentation:
 org.apache.hadoop.fs.FileStatus.getLen()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726442#action_12726442 ] 

Hudson commented on HADOOP-6114:
--------------------------------

Integrated in Hadoop-Common-trunk #14 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/14/])
    . Fix javadoc documentation for FileStatus.getLen.
(Dmitry Rzhevskiy via dhruba)


> bug in documentation: org.apache.hadoop.fs.FileStatus.getLen() 
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6114
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6114
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.18.3, 0.20.0
>            Reporter: Dmitry Rzhevskiy
>            Assignee: Dmitry Rzhevskiy
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6114.patch
>
>
> In javadoc method  org.apache.hadoop.fs.FileStatus.getLen()  writtend that this method "return the length of this file, in blocks"
> But method return size in bytes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-32-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 17:09:59 2009
Return-Path: <common-issues-return-32-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9895 invoked from network); 2 Jul 2009 17:09:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 17:09:59 -0000
Received: (qmail 76935 invoked by uid 500); 2 Jul 2009 17:10:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76889 invoked by uid 500); 2 Jul 2009 17:10:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76872 invoked by uid 99); 2 Jul 2009 17:10:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 17:10:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 17:10:07 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4DA42234C045
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 10:09:47 -0700 (PDT)
Message-ID: <6773949.1246554587317.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 10:09:47 -0700 (PDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6121) High Availability support for
 Hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726560#action_12726560 ] 

Steve Loughran commented on HADOOP-6121:
----------------------------------------

whatever you come up with, can it work on virtualised clusters where clocks are jittery? 

I encounter lots of interesting timeouts with VMs, depending on the underlying infrastructure and its load, and its easy for anything that assumes that multicast works or that clocks move forward at the same rate to get confused. At this point Lamport will probably post a comment implying that timeouts are essential, and I agree -things just need to work well in a world where time is jerky, and the virtual HDDs may not be flushed when calling sync(). 

> High Availability support for Hadoop
> ------------------------------------
>
>                 Key: HADOOP-6121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6121
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: dfs, mapred
>            Reporter: Jie Qiu
>
> Currently, We look at the HA of Hadoop cluster. We need to consider the NameNode HA as well as Jobtracker HA. For NameNode, we want to build primary/standy or master-slaves pattern to provide NameNode HA. Therefore, we need to consider how to ship log between primary/standby/slaves and how commit "write" operation to NameNode after the agreement among primary/standby/slaves on log. Whether will we use Linux HA package or NameNode-built-in HA package without the help of outter Linux HA package. 
> After NameNode become high availability, is it necessary to provide HA for Jobtracker? Can Jobtracker  persist the states of Jobs and tasks into HA NameNode? Or Jobtracker also needs the same approach from NameNode for HA support.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-33-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 17:38:01 2009
Return-Path: <common-issues-return-33-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33853 invoked from network); 2 Jul 2009 17:38:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 17:38:01 -0000
Received: (qmail 18493 invoked by uid 500); 2 Jul 2009 17:38:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18461 invoked by uid 500); 2 Jul 2009 17:38:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18446 invoked by uid 99); 2 Jul 2009 17:38:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 17:38:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 17:38:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 52B16234C046
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 10:37:47 -0700 (PDT)
Message-ID: <423916970.1246556267337.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 10:37:47 -0700 (PDT)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6121) High Availability support for
 Hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726573#action_12726573 ] 

Scott Carey commented on HADOOP-6121:
-------------------------------------

{quote}Do you know who is implementing ZK for the NameNode?{quote}

I don't even know if such a thing is happening, but given that Google's MapReduce paper and BigTable paper indicate that Chubby is leveraged for HA on various hadoop equivalents I figured that was part of the discussion.

I am in the process of using ZK for an internal project that has some similarities with the JT.  Previously it was using ping-response for both state communication and triggering (distributed) events.  Switching to ZK proved to be a bit outside the bounds of its typical use case but is very fast, reliable, and light-weight.  State management and persistence is much easier, and communication with watches very reliable and fast.

Anyhow, its an option.  The reliability, availability, and time sensitivity issues it addresses.  Other special aspects of the JT use case would probably drive some ZK features -- like an api to return only 'new' children of a node after a specific transaction id, easier out-of-the-box embedding, and some higher level APIs for read-only/watch-only use cases.

> High Availability support for Hadoop
> ------------------------------------
>
>                 Key: HADOOP-6121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6121
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: dfs, mapred
>            Reporter: Jie Qiu
>
> Currently, We look at the HA of Hadoop cluster. We need to consider the NameNode HA as well as Jobtracker HA. For NameNode, we want to build primary/standy or master-slaves pattern to provide NameNode HA. Therefore, we need to consider how to ship log between primary/standby/slaves and how commit "write" operation to NameNode after the agreement among primary/standby/slaves on log. Whether will we use Linux HA package or NameNode-built-in HA package without the help of outter Linux HA package. 
> After NameNode become high availability, is it necessary to provide HA for Jobtracker? Can Jobtracker  persist the states of Jobs and tasks into HA NameNode? Or Jobtracker also needs the same approach from NameNode for HA support.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-34-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 17:51:59 2009
Return-Path: <common-issues-return-34-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43139 invoked from network); 2 Jul 2009 17:51:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 17:51:59 -0000
Received: (qmail 36577 invoked by uid 500); 2 Jul 2009 17:52:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36475 invoked by uid 500); 2 Jul 2009 17:52:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36417 invoked by uid 99); 2 Jul 2009 17:52:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 17:52:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 17:52:07 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4EB5B234C052
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 10:51:47 -0700 (PDT)
Message-ID: <806380709.1246557107321.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 10:51:47 -0700 (PDT)
From: "Mahadev konar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6121) High Availability support for
 Hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726578#action_12726578 ] 

Mahadev konar commented on HADOOP-6121:
---------------------------------------

bq. whatever you come up with, can it work on virtualised clusters where clocks are jittery? I encounter lots of interesting timeouts with VMs, depending on the underlying infrastructure and its load, and its easy for anything that assumes that multicast works or that clocks move forward at the same rate to get confused. At this point Lamport will probably post a comment implying that timeouts are essential, and I agree -things just need to work well in a world where time is jerky, and the virtual HDDs may not be flushed when calling sync().

steve, 
   We have seen ZooKeeper being used on EC2 in virtualized environment. As far I can say, it works for those folks.  

> High Availability support for Hadoop
> ------------------------------------
>
>                 Key: HADOOP-6121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6121
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: dfs, mapred
>            Reporter: Jie Qiu
>
> Currently, We look at the HA of Hadoop cluster. We need to consider the NameNode HA as well as Jobtracker HA. For NameNode, we want to build primary/standy or master-slaves pattern to provide NameNode HA. Therefore, we need to consider how to ship log between primary/standby/slaves and how commit "write" operation to NameNode after the agreement among primary/standby/slaves on log. Whether will we use Linux HA package or NameNode-built-in HA package without the help of outter Linux HA package. 
> After NameNode become high availability, is it necessary to provide HA for Jobtracker? Can Jobtracker  persist the states of Jobs and tasks into HA NameNode? Or Jobtracker also needs the same approach from NameNode for HA support.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-35-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 18:29:59 2009
Return-Path: <common-issues-return-35-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74973 invoked from network); 2 Jul 2009 18:29:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 18:29:59 -0000
Received: (qmail 23153 invoked by uid 500); 2 Jul 2009 18:30:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23102 invoked by uid 500); 2 Jul 2009 18:30:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23092 invoked by uid 99); 2 Jul 2009 18:30:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 18:30:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 18:30:07 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 45FD8234C004
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 11:29:47 -0700 (PDT)
Message-ID: <1104778121.1246559387271.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 11:29:47 -0700 (PDT)
From: "Matei Zaharia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726602#action_12726602 ] 

Matei Zaharia commented on HADOOP-5170:
---------------------------------------

I wrote this patch because of demand for this feature from many users, including Cloudera customers, who had reasons to limit the number of tasks per job. If the patch interacts poorly with the capacity scheduler, wouldn't it make more sense to tell users of the capacity scheduler not to use this feature, or to improve MAPREDUCE-516 so that it can take into account task limits?

I agree that task limits are not the best solution for dealing with high-memory jobs, but that's not the only situation that this patch addresses. Here are some others:
* The situation described in the original task was that a task is very CPU-intensive. The user wanted to limit the number of those tasks running to less than the number of cores. However, there is no need to fill up all slots on the machine - the other slots could be used for less CPU-intensive tasks. Until there is a model for all system resources in the capacity scheduler, this patch lets users with that kind of problem achieve reasonable behavior.
* One situation in which you want to limit tasks on the whole cluster is if you have a housekeeping job with long tasks, e.g. a database import or export. If you don't have a limit on running tasks, such a job can take up the whole cluster and hold it for a long time. The capacity and fair  schedulers include a preemption feature to kill tasks, but preemption has some problems: First, the preemption timeout generally shouldn't be set to less than 5-10 minutes so it doesn't interact badly with faulty tasktracker timeouts, and second, preemption might complicate writing certain jobs (e.g. database import/export).

I also think the problems Arun brought up in the capacity scheduler aren't insurmountable. To deal with the locality problem, you can give up slots when you run out of local data on them, like MAPREDUCE-548. To deal with the pinning problem, preemption might help.

More importantly, I think Arun's issues will come up even if task limits aren't used. For example, suppose that a given queue's capacity is 25% of the cluster, say 250 slots, and that all the other queues are filled with jobs, so there is never excess capacity. Then if a high-memory job is submitted to the queue with 25% capacity, it has to behave exactly as if it has a limit of 250 tasks cluster-wide. The same problems that Arun brought up will happen: locality will be poor, the job might make bad bets, etc. This situation will be far more common than a user explicitly setting a limit on running tasks for the job, so if these two concerns are serious, maybe more work needs to be put into MAPREDUCE-516 to prevent them from happening. In particular, I think limiting the number of "bets" based on the job's capacity or task limit, allowing the job to kill tasks if it has waited for too long on a bet, and implementing MAPREDUCE-548 would solve much of the problem.

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-36-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 18:50:02 2009
Return-Path: <common-issues-return-36-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85268 invoked from network); 2 Jul 2009 18:49:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 18:49:59 -0000
Received: (qmail 62942 invoked by uid 500); 2 Jul 2009 18:50:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62909 invoked by uid 500); 2 Jul 2009 18:50:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62899 invoked by uid 99); 2 Jul 2009 18:50:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 18:50:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 18:50:07 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3F17C234C004
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 11:49:47 -0700 (PDT)
Message-ID: <1338667609.1246560587244.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 11:49:47 -0700 (PDT)
From: "Jonathan Gray (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726622#action_12726622 ] 

Jonathan Gray commented on HADOOP-5170:
---------------------------------------

I am using this patch on 0.19 and 0.20, and will have to continue patching it if taken out of the 0.21 release.  I've also recommended this patch and know of others who are using it... It solves a number of scheduling issues we have.

The idea of having "heavy" tasks works great for high-memory jobs.  But we have a number of network-io bound jobs and cpu bound jobs.  We can make the network bound jobs light, and cpu bound jobs heavy, but what we really want is to have one cpu bound job run per node (we have one core for it), and lots of network jobs per node... simultaneously.  With task weights, how can I prevent the cpu bound job from taking up all the slots and letting in high numbers of network bound jobs at the same time?

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-37-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 19:02:02 2009
Return-Path: <common-issues-return-37-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91340 invoked from network); 2 Jul 2009 19:02:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 19:02:02 -0000
Received: (qmail 84247 invoked by uid 500); 2 Jul 2009 19:02:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84213 invoked by uid 500); 2 Jul 2009 19:02:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84193 invoked by uid 99); 2 Jul 2009 19:02:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 19:02:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 19:02:09 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 14392234C004
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 12:01:48 -0700 (PDT)
Message-ID: <1138049620.1246561308068.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 12:01:48 -0700 (PDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726633#action_12726633 ] 

Arun C Murthy commented on HADOOP-5170:
---------------------------------------

bq. If the patch interacts poorly with the capacity scheduler, wouldn't it make more sense to tell users of the capacity scheduler not to use this feature, or to improve MAPREDUCE-516 so that it can take into account task limits?

Like I said, this patch introduced 2 features: per-node limits and per-job limits. The per-job cluster limit interacts poorly with the MAPREDUCE-516, not per-node limits.

bq. The situation described in the original task was that a task is very CPU-intensive. The user wanted to limit the number of those tasks running to less than the number of cores. However, there is no need to fill up all slots on the machine - the other slots could be used for less CPU-intensive tasks. Until there is a model for all system resources in the capacity scheduler, this patch lets users with that kind of problem achieve reasonable behavior.

We are missing the woods for the trees here. The user whose task is CPU-intensive is happy. But, what about the other users whose tasks need CPU too? How do we keep their tasks from starving on the same node? In particular there are no checks and balances on preventing multiple CPU-intensive tasks from being scheduled on the same node. If we *knew* that the other tasks were going to be IO intensive we could co-schedule them on this node, but we don't. This is the reason why Owen and I continue to insist that per-node task limits are a poor substitute for modelling resource usage and that a resource model is a *pre-requisite* for this feature.

This feature works well for clusters with single users, but not in shared clusters.

bq. One situation in which you want to limit tasks on the whole cluster is if you have a housekeeping job with long tasks, e.g. a database import or export. If you don't have a limit on running tasks, such a job can take up the whole cluster and hold it for a long time.

The short-term fix is to submit these jobs to a special queue with limited capacity, possibly to queues with a hard upper-limit on their capacity: MAPREDUCE-532. 

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-38-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 19:26:02 2009
Return-Path: <common-issues-return-38-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2700 invoked from network); 2 Jul 2009 19:26:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 19:26:01 -0000
Received: (qmail 30454 invoked by uid 500); 2 Jul 2009 19:26:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30421 invoked by uid 500); 2 Jul 2009 19:26:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30406 invoked by uid 99); 2 Jul 2009 19:26:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 19:26:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 19:26:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 329F0234C004
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 12:25:47 -0700 (PDT)
Message-ID: <1334497635.1246562747202.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 12:25:47 -0700 (PDT)
From: "Matei Zaharia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726645#action_12726645 ] 

Matei Zaharia commented on HADOOP-5170:
---------------------------------------

If the capacity scheduler will need to support cluster-wide task limits through MAPREDUCE-532 eventually, why can't it support them through this per-job property?

I fully agree that per-node limits aren't a solution to resource sharing in a multi-user environment. However, most Hadoop clusters outside Yahoo, Facebook and a few other large installations are essentially single-user. For these clusters, the per-node limit solves real problems until the time when we introduce a resource model to Hadoop (which will be a serious undertaking). This is why there were many votes on this feature. Whenever I've talked to people about the fair scheduler, task limits have been one of the most requested features.

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-39-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 20:30:01 2009
Return-Path: <common-issues-return-39-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27048 invoked from network); 2 Jul 2009 20:30:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 20:30:01 -0000
Received: (qmail 27582 invoked by uid 500); 2 Jul 2009 20:30:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27540 invoked by uid 500); 2 Jul 2009 20:30:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27488 invoked by uid 99); 2 Jul 2009 20:30:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 20:30:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 20:30:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 61C75234C052
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 13:29:47 -0700 (PDT)
Message-ID: <81915344.1246566587399.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 13:29:47 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726664#action_12726664 ] 

Tom White commented on HADOOP-5170:
-----------------------------------

This is an optional feature - users don't have to use it and it is turned off by default. It has no performance impact for those who don't enable it. On the other hand there are a substantial number of users who are using it (or interested in using it), and would be left with no immediate alternative if it were pulled from the next release.

We can leave this feature in until there is a suitable replacement, after which time it can be deprecated so users can migrate to the replacement. Could that work?

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-40-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 20:55:59 2009
Return-Path: <common-issues-return-40-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38149 invoked from network); 2 Jul 2009 20:55:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 20:55:59 -0000
Received: (qmail 71147 invoked by uid 500); 2 Jul 2009 20:56:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71112 invoked by uid 500); 2 Jul 2009 20:56:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70955 invoked by uid 99); 2 Jul 2009 20:56:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 20:56:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 20:56:07 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4AEC2234C055
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 13:55:47 -0700 (PDT)
Message-ID: <415486475.1246568147306.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 13:55:47 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726674#action_12726674 ] 

Owen O'Malley commented on HADOOP-5170:
---------------------------------------

The problem with this patch is that it doesn't do anything useful and gets in the way of real fixes to the problem. Limiting the number of running tasks per a job is *not* what any users need. It is being used as an expedient way to hack in global resource limits. If I launch two cpu hungry jobs (even with a single user!), I will kill those nodes. That is *not* ok. While MAPREDUCE-516 addresses high-ram jobs, it takes the view that each slot has a set of resources and that a given job's tasks may require multiple slots. This *is* a reasonable model that is applicable for cpu, ram, or disk. It would be reasonable to change the interface to MAPREDUCE-516 to be in terms of the slots per a task instead of memory.

Clearly in the long term, I think that a more dynamic model that tracks the resources being consumed and launches new tasks appropriately. In the mean time, MAPREDUCE-516 is a much better approach to this.

{quote}
This is an optional feature - users don't have to use it and it is turned off by default. It has no performance impact for those who don't enable it. On the other hand there are a substantial number of users who are using it (or interested in using it), and would be left with no immediate alternative if it were pulled from the next release.

We can leave this feature in until there is a suitable replacement, after which time it can be deprecated so users can migrate to the replacement. Could that work?
{quote}

-1 Because once it is in, we need to support it. When it doesn't work for the users, they'll try and fix it. We already have users confused by too many knobs...

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-41-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 20:56:05 2009
Return-Path: <common-issues-return-41-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38266 invoked from network); 2 Jul 2009 20:56:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 20:56:05 -0000
Received: (qmail 71650 invoked by uid 500); 2 Jul 2009 20:56:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71617 invoked by uid 500); 2 Jul 2009 20:56:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71607 invoked by uid 99); 2 Jul 2009 20:56:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 20:56:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 20:56:07 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 450C7234C046
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 13:55:47 -0700 (PDT)
Message-ID: <249233974.1246568147281.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 13:55:47 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6124) patchJavacWarnings and
 trunkJavacWarnings are not consistent.
In-Reply-To: <1339799848.1246409207795.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726673#action_12726673 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6124:
------------------------------------------------

Giridharan, c6124_20090701.patch does not fix the entire problem.  There are still some additional bugs in test-patch.sh.  Since you are the assignee, I will leave this issue to you.  Thanks for working on this.

> patchJavacWarnings and trunkJavacWarnings are not consistent.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6124
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6124
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Giridharan Kesavan
>         Attachments: c6124_20090701.patch
>
>
> The values of patchJavacWarnings and trunkJavacWarnings are not consistent when running test-patch.sh with an empty patch over Common.  HDFS and MapReduce seem not having this problem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-42-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 20:58:01 2009
Return-Path: <common-issues-return-42-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38597 invoked from network); 2 Jul 2009 20:58:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 20:58:01 -0000
Received: (qmail 76201 invoked by uid 500); 2 Jul 2009 20:58:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76166 invoked by uid 500); 2 Jul 2009 20:58:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76155 invoked by uid 99); 2 Jul 2009 20:58:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 20:58:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 20:58:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 48F41234C044
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 13:57:47 -0700 (PDT)
Message-ID: <1456491324.1246568267297.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 13:57:47 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Reopened: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley reopened HADOOP-5170:
-----------------------------------


This is *not* resolved and will likely be reverted.

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-43-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 21:12:01 2009
Return-Path: <common-issues-return-43-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42830 invoked from network); 2 Jul 2009 21:12:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 21:12:01 -0000
Received: (qmail 88070 invoked by uid 500); 2 Jul 2009 21:12:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88038 invoked by uid 500); 2 Jul 2009 21:12:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88028 invoked by uid 99); 2 Jul 2009 21:12:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 21:12:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 21:12:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 35E79234C004
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 14:11:47 -0700 (PDT)
Message-ID: <1072426585.1246569107209.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 14:11:47 -0700 (PDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726683#action_12726683 ] 

Arun C Murthy commented on HADOOP-5170:
---------------------------------------

bq. If the capacity scheduler will need to support cluster-wide task limits through MAPREDUCE-532 eventually, why can't it support them through this per-job property? 

Sorry, I should have clarified that we are simplifying the scope of MAPREDUCE-532 to implement a 'hard-limit' on a queue. Thus, we can limit the fan-in to the resource in question via the hard-limit on the special queue for the resource.

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-44-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 21:43:59 2009
Return-Path: <common-issues-return-44-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65473 invoked from network); 2 Jul 2009 21:43:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 21:43:59 -0000
Received: (qmail 18056 invoked by uid 500); 2 Jul 2009 21:44:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18024 invoked by uid 500); 2 Jul 2009 21:44:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18014 invoked by uid 99); 2 Jul 2009 21:44:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 21:44:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 21:44:07 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 35D06234C004
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 14:43:47 -0700 (PDT)
Message-ID: <2073335369.1246571027206.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 14:43:47 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726694#action_12726694 ] 

Tom White commented on HADOOP-5170:
-----------------------------------

Thanks for the clarification Arun. Are you targeting MAPREDUCE-532 for 0.21?


> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-45-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 21:48:03 2009
Return-Path: <common-issues-return-45-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66977 invoked from network); 2 Jul 2009 21:48:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 21:48:01 -0000
Received: (qmail 19540 invoked by uid 500); 2 Jul 2009 21:48:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19504 invoked by uid 500); 2 Jul 2009 21:48:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19451 invoked by uid 99); 2 Jul 2009 21:48:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 21:48:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 21:48:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4764D234C045
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 14:47:47 -0700 (PDT)
Message-ID: <1779429919.1246571267291.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 14:47:47 -0700 (PDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726696#action_12726696 ] 

Arun C Murthy commented on HADOOP-5170:
---------------------------------------

Yes, it should be ready in a day or two. We will release a patch for the yahoo 0.20 branch too.

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-46-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 21:58:02 2009
Return-Path: <common-issues-return-46-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72545 invoked from network); 2 Jul 2009 21:58:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 21:58:01 -0000
Received: (qmail 30662 invoked by uid 500); 2 Jul 2009 21:58:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30627 invoked by uid 500); 2 Jul 2009 21:58:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30617 invoked by uid 99); 2 Jul 2009 21:58:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 21:58:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 21:58:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 9FFCC234C045
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 14:57:47 -0700 (PDT)
Message-ID: <812364339.1246571867654.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 14:57:47 -0700 (PDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726703#action_12726703 ] 

dhruba borthakur commented on HADOOP-5170:
------------------------------------------

@Arun: is the yahoo 0.20 branch (that you refer to) an existing branch in apache svn? how soon can I access it :-) ?

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-47-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 22:04:01 2009
Return-Path: <common-issues-return-47-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75980 invoked from network); 2 Jul 2009 22:04:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 22:04:01 -0000
Received: (qmail 37452 invoked by uid 500); 2 Jul 2009 22:04:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37417 invoked by uid 500); 2 Jul 2009 22:04:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37392 invoked by uid 99); 2 Jul 2009 22:04:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 22:04:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 22:04:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4558C234C004
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 15:03:47 -0700 (PDT)
Message-ID: <2026526339.1246572227268.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 15:03:47 -0700 (PDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726706#action_12726706 ] 

Arun C Murthy commented on HADOOP-5170:
---------------------------------------

@Dhruba: http://developer.yahoo.com/hadoop/distribution/

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-48-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 22:18:01 2009
Return-Path: <common-issues-return-48-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83975 invoked from network); 2 Jul 2009 22:18:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 22:18:01 -0000
Received: (qmail 54457 invoked by uid 500); 2 Jul 2009 22:18:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54423 invoked by uid 500); 2 Jul 2009 22:18:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54412 invoked by uid 99); 2 Jul 2009 22:18:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 22:18:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 22:18:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3895B234C004
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 15:17:47 -0700 (PDT)
Message-ID: <1069477120.1246573067217.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 15:17:47 -0700 (PDT)
From: "Matei Zaharia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726709#action_12726709 ] 

Matei Zaharia commented on HADOOP-5170:
---------------------------------------

Arun, can you explain what the hard limit on capacity means? Is it option 2 in MAPREDUCE-532 that just disallows a queue from taking excess capacity? I'm still not sure how this can solve the problems you brought up. In fact, as I pointed out in my earlier comment, even *without* any limits defined, MAPREDUCE-516 can harm locality and utilization in its current form. Am I missing something about per-job limits that makes them different from queue limits or from no excess capacity being available?

Owen, I agree that having MAPREDUCE-516 be in terms of multi-slot tasks would solve some issues, but it doesn't sound like it would solve Jonathan's problem for example. I think that coming up with a general multi-resource sharing model will be difficult. Is there anything we can do in the meantime to support this use case? For example, it would be trivial for me to implement the per-node limits in the fair scheduler, but then they wouldn't be available to users of other schedulers.

This might also be a good time to figure out exactly what scheduling functionality can go into JobInProgress/TaskTracker/etc and what should go into schedulers. I didn't think the limits in this patch added much complexity. They obey the contract of obtainNewMapTask, which is that it may or may not return a task. Schedulers already have to deal with jobs that have no tasks to launch on one heartbeat, and then have tasks on the next, because of speculation. So any scheduler that works with that should more or less be okay if the job chooses to launch a task based on its total number of running tasks. If we agreed on some kind of contract, then we would be able to implement common scheduling functionality in the mapreduce package rather than having it be contrib. Otherwise, as long as there are multiple groups working on scheduling on Hadoop, everyone will be worried that someone else's change will break their future work.


> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-49-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 22:55:59 2009
Return-Path: <common-issues-return-49-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94163 invoked from network); 2 Jul 2009 22:55:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 22:55:59 -0000
Received: (qmail 87839 invoked by uid 500); 2 Jul 2009 22:56:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87801 invoked by uid 500); 2 Jul 2009 22:56:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87789 invoked by uid 99); 2 Jul 2009 22:56:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 22:56:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 22:56:07 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 44E85234C045
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 15:55:47 -0700 (PDT)
Message-ID: <650091643.1246575347281.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 15:55:47 -0700 (PDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726727#action_12726727 ] 

Arun C Murthy commented on HADOOP-5170:
---------------------------------------

bq. Arun, can you explain what the hard limit on capacity means? Is it option 2 in MAPREDUCE-532 that just disallows a queue from taking excess capacity?

Yes, it will *only* had a hard-limit per-queue. This feature is aimed at solving a certain class of problem alone i.e. limiting fan-in for a specific resource. 

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-50-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 22:58:02 2009
Return-Path: <common-issues-return-50-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94488 invoked from network); 2 Jul 2009 22:58:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 22:58:02 -0000
Received: (qmail 91570 invoked by uid 500); 2 Jul 2009 22:58:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91532 invoked by uid 500); 2 Jul 2009 22:58:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91522 invoked by uid 99); 2 Jul 2009 22:58:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 22:58:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 22:58:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 35B99234C004
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 15:57:47 -0700 (PDT)
Message-ID: <917570151.1246575467206.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 15:57:47 -0700 (PDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726728#action_12726728 ] 

Arun C Murthy commented on HADOOP-5170:
---------------------------------------

bq. Yes, it will only had a hard-limit per-queue. This feature is aimed at solving a certain class of problem alone i.e. limiting fan-in for a specific resource. 

s/had/add

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-51-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 23:46:01 2009
Return-Path: <common-issues-return-51-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5079 invoked from network); 2 Jul 2009 23:46:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 23:46:01 -0000
Received: (qmail 28609 invoked by uid 500); 2 Jul 2009 23:46:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28574 invoked by uid 500); 2 Jul 2009 23:46:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28564 invoked by uid 99); 2 Jul 2009 23:46:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 23:46:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 23:46:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 67768234C004
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 16:45:47 -0700 (PDT)
Message-ID: <1785951011.1246578347410.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 16:45:47 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726737#action_12726737 ] 

Owen O'Malley commented on HADOOP-5170:
---------------------------------------

I think those limits are a very bad idea. If you think they are useful, I'd be ok with putting them into the fair share scheduler. Then you can experiment with them and see if they are useful outside of a single user research cluster. I suspect you'll quickly discover the lack of utility in this patch. I certainly don't think this is appropriate to go into the main map/reduce framework.

You're right that we can't support Jonathan's use case, but it isn't a valid use case for Hadoop. Hadoop is about *sharing* a cluster with other users. It is not a personal supercomputer operating system. (Although Arun and I did *have* a personal Hadoop supercomputer for a couple months while running the sort benchmarks. *smile*)

I plan on reverting this and closing this as wont fix.

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-52-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 23:55:59 2009
Return-Path: <common-issues-return-52-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6637 invoked from network); 2 Jul 2009 23:55:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 23:55:59 -0000
Received: (qmail 31450 invoked by uid 500); 2 Jul 2009 23:56:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31417 invoked by uid 500); 2 Jul 2009 23:56:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31407 invoked by uid 99); 2 Jul 2009 23:56:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 23:56:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 23:56:07 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id B0E36234C004
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 16:55:47 -0700 (PDT)
Message-ID: <1881140605.1246578947712.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 16:55:47 -0700 (PDT)
From: "Jonathan Gray (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726740#action_12726740 ] 

Jonathan Gray commented on HADOOP-5170:
---------------------------------------

I don't see how my use case is invalid.  

And if decisions are always made for the betterment of very large clusters with multiple users rather than smaller clusters with a single user (or at least, in a controlled environment), you're going to alienate a vast majority of new users and even seasoned users who happen to not care about multi-user environments.

I don't particularly care _how_ I get done what I need to get done.  But I have thousands of jobs submitted to my cluster every day and prior to this patch I was unable to get to 25% the capacity I have now because of the characteristics of a very small minority of the total number of jobs.

I need a way to prevent CPU intensive jobs from eating all the cores.  If I have 20 nodes and 100 CPU bound jobs, i only want 1 per node at a time.  That strikes me as a common use case, not an invalid one.  At the same time I'd like a way to allow 10 jobs per node of network bound jobs that do little else but wait around.

>From a boots on the ground perspective, I can say that this jira comes up more than once a week in discussion with new users, etc.  It's easy to understand and gives the user an enormous amount of control with very few knobs.  At least stick this functionality into a corner somewhere so those of us who want it can still use it.  When there is more mature scheduling down the road, I'm more than happy to switch.

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-53-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 02 23:58:02 2009
Return-Path: <common-issues-return-53-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6880 invoked from network); 2 Jul 2009 23:58:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Jul 2009 23:58:02 -0000
Received: (qmail 31909 invoked by uid 500); 2 Jul 2009 23:58:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31876 invoked by uid 500); 2 Jul 2009 23:58:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31777 invoked by uid 99); 2 Jul 2009 23:58:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 23:58:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jul 2009 23:58:09 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C5E3E234C004
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 16:57:47 -0700 (PDT)
Message-ID: <686624790.1246579067796.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 16:57:47 -0700 (PDT)
From: "Chris K Wensel (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726741#action_12726741 ] 

Chris K Wensel commented on HADOOP-5170:
----------------------------------------

bq. Hadoop is about sharing a cluster with other users.

I beg to differ. many of my users run many concurrent single purpose Hadoop clusters in AWS. each tuned and sized to the particular load. i believe, HOD exists for this purpose as well, to some extent.

re this patch. 

many users, I expect the bulk of Hadoop users, have small constrained clusters and are happy with hand crafting their workloads to get the best utilization and performance. whether or not this patch is 'correct', reading above I get the impression it is being used and the users find it useful. flat out reverting it seems heavy handed without an alternative available.

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-54-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 00:33:59 2009
Return-Path: <common-issues-return-54-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25738 invoked from network); 3 Jul 2009 00:33:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 00:33:59 -0000
Received: (qmail 49546 invoked by uid 500); 3 Jul 2009 00:34:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49512 invoked by uid 500); 3 Jul 2009 00:34:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49502 invoked by uid 99); 3 Jul 2009 00:34:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 00:34:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 00:34:07 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3800E234C004
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 17:33:47 -0700 (PDT)
Message-ID: <2105061098.1246581227216.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 17:33:47 -0700 (PDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726751#action_12726751 ] 

Arun C Murthy commented on HADOOP-5170:
---------------------------------------

{quote}
I have thousands of jobs submitted to my cluster every day and prior to this patch I was unable to get to 25% the capacity I have now because of the characteristics of a very small minority of the total number of jobs.

I need a way to prevent CPU intensive jobs from eating all the cores. If I have 20 nodes and 100 CPU bound jobs, i only want 1 per node at a time. That strikes me as a common use case, not an invalid one. At the same time I'd like a way to allow 10 jobs per node of network bound jobs that do little else but wait around.
{quote}

Jonathan, Can you please sketch the scenario you face in more detail?

I'll hazard a *guess* and ask you to consider using high-ram jobs for your CPU intensive jobs and use MAPREDUCE-532  to limit how much of the cluster they occupy. (I'm reading that you have a small minority of such jobs from your comment).

For e.g. create a 'cpu-heavy' queue and limit it's capacity as a fraction of your total capacity using MAPREDUCE-532. Then configure your CPU-intensive jobs to use up a majority of the slots in each node (thus preventing 2 slots of the same job from being scheduled on the same node) and submit these jobs to the 'cpu-heavy' queue.

Yes, you will not be able to run too many other tasks on those nodes. But with MAPREDUCE-532 you can ensure that it doesn't affect too many nodes in your cluster.

Will that work?



> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-55-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 00:40:01 2009
Return-Path: <common-issues-return-55-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26357 invoked from network); 3 Jul 2009 00:40:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 00:40:01 -0000
Received: (qmail 50806 invoked by uid 500); 3 Jul 2009 00:40:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50773 invoked by uid 500); 3 Jul 2009 00:40:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50763 invoked by uid 99); 3 Jul 2009 00:40:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 00:40:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 00:40:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 7D3E9234C045
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 17:39:47 -0700 (PDT)
Message-ID: <148561964.1246581587512.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 17:39:47 -0700 (PDT)
From: "Matei Zaharia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726753#action_12726753 ] 

Matei Zaharia commented on HADOOP-5170:
---------------------------------------

Jonathan, would you be okay if this feature was placed in the fair scheduler?

I agree with Owen that it's a good idea to remove task limits from the Hadoop core interface so as not to lock in the API. However, I will support them in the fair scheduler. Apart from the demand for this feature from the community, we have paying customers at Cloudera who asked for this patch and run it on multi-rack clusters.

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-56-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 00:54:01 2009
Return-Path: <common-issues-return-56-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28377 invoked from network); 3 Jul 2009 00:54:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 00:54:01 -0000
Received: (qmail 61393 invoked by uid 500); 3 Jul 2009 00:54:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61359 invoked by uid 500); 3 Jul 2009 00:54:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61349 invoked by uid 99); 3 Jul 2009 00:54:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 00:54:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 00:54:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 34185234C004
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 17:53:47 -0700 (PDT)
Message-ID: <450020554.1246582427199.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 17:53:47 -0700 (PDT)
From: "Jonathan Gray (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726757#action_12726757 ] 

Jonathan Gray commented on HADOOP-5170:
---------------------------------------

@Matei +1

@Arun I need to dig further into MAPREDUCE-532 before I can make an educated assessment.  I'm not that familiar with scheduling and queues; part of the reason I like this issue is the simplicity.

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-57-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 01:29:00 2009
Return-Path: <common-issues-return-57-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40177 invoked from network); 3 Jul 2009 01:28:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 01:28:59 -0000
Received: (qmail 79405 invoked by uid 500); 3 Jul 2009 01:29:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79372 invoked by uid 500); 3 Jul 2009 01:29:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79362 invoked by uid 99); 3 Jul 2009 01:29:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 01:29:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 01:29:07 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 56248234C004
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 18:28:47 -0700 (PDT)
Message-ID: <919096066.1246584527206.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 18:28:47 -0700 (PDT)
From: "Amr Awadallah (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6121) High Availability support for
 Hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726762#action_12726762 ] 

Amr Awadallah commented on HADOOP-6121:
---------------------------------------


bq. Do you know who is implementing ZK for the NameNode?

see bookkeeper:

https://issues.apache.org/jira/browse/ZOOKEEPER-276


> High Availability support for Hadoop
> ------------------------------------
>
>                 Key: HADOOP-6121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6121
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: dfs, mapred
>            Reporter: Jie Qiu
>
> Currently, We look at the HA of Hadoop cluster. We need to consider the NameNode HA as well as Jobtracker HA. For NameNode, we want to build primary/standy or master-slaves pattern to provide NameNode HA. Therefore, we need to consider how to ship log between primary/standby/slaves and how commit "write" operation to NameNode after the agreement among primary/standby/slaves on log. Whether will we use Linux HA package or NameNode-built-in HA package without the help of outter Linux HA package. 
> After NameNode become high availability, is it necessary to provide HA for Jobtracker? Can Jobtracker  persist the states of Jobs and tasks into HA NameNode? Or Jobtracker also needs the same approach from NameNode for HA support.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-58-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 02:17:04 2009
Return-Path: <common-issues-return-58-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67524 invoked from network); 3 Jul 2009 02:17:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 02:17:01 -0000
Received: (qmail 8831 invoked by uid 500); 3 Jul 2009 02:17:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8798 invoked by uid 500); 3 Jul 2009 02:17:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8788 invoked by uid 99); 3 Jul 2009 02:17:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 02:17:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 02:17:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 66DF829A0018
	for <common-issues@hadoop.apache.org>; Thu,  2 Jul 2009 19:16:47 -0700 (PDT)
Message-ID: <207056657.1246587407420.JavaMail.jira@brutus>
Date: Thu, 2 Jul 2009 19:16:47 -0700 (PDT)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726768#action_12726768 ] 

Scott Carey commented on HADOOP-5170:
-------------------------------------

bq. You're right that we can't support Jonathan's use case, but it isn't a valid use case for Hadoop. Hadoop is about sharing a cluster with other users.

Please don't assume everyone uses Hadoop the same way.  Many users are more than willing to tune a specific job flow on a cluster for max compute power per $ and have little or no ad-hoc job submission.  For some such a workflow is business critical and the cost/benefit of heavy hand-tuning is obvious.

Yes, these tuning knobs are not good for all use cases and especially ad-hoc shared cluster use.

Being able to specifically tune a business critical workflow is a very good thing for some.  There is a reason this JIRA gained a lot of watchers fast and has the most votes for it.

A resource model will be very useful, but it will never beat hand tuning for performance -- it will only get part way there -- and the two can be cooperative as well.
Some clusters will be used in ways such that hand tuning is possible, others won't.

Perhaps in the long term there could be a shared scheduler component that contains all the task limits with a default implementation, and each scheduler can optionally use, or override it.



> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-59-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 08:27:59 2009
Return-Path: <common-issues-return-59-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83599 invoked from network); 3 Jul 2009 08:27:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 08:27:59 -0000
Received: (qmail 75173 invoked by uid 500); 3 Jul 2009 08:28:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75138 invoked by uid 500); 3 Jul 2009 08:28:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75128 invoked by uid 99); 3 Jul 2009 08:28:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 08:28:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 08:28:07 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3DA98234C044
	for <common-issues@hadoop.apache.org>; Fri,  3 Jul 2009 01:27:47 -0700 (PDT)
Message-ID: <353679909.1246609667251.JavaMail.jira@brutus>
Date: Fri, 3 Jul 2009 01:27:47 -0700 (PDT)
From: "Bo Dong (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6121) High Availability support for
 Hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726825#action_12726825 ] 

Bo Dong commented on HADOOP-6121:
---------------------------------

scott,
We find that the JobTracker only persist the state of competed and dead jobs in DFS, not including the state of in-process jobs. 
However, we can use the existing approach by which JT persists the competed and dead jobs to persist the in-process jobs in JT. 
I feel it is much simpler than ZK-based approach, since we already make NN high availability.

> High Availability support for Hadoop
> ------------------------------------
>
>                 Key: HADOOP-6121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6121
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: dfs, mapred
>            Reporter: Jie Qiu
>
> Currently, We look at the HA of Hadoop cluster. We need to consider the NameNode HA as well as Jobtracker HA. For NameNode, we want to build primary/standy or master-slaves pattern to provide NameNode HA. Therefore, we need to consider how to ship log between primary/standby/slaves and how commit "write" operation to NameNode after the agreement among primary/standby/slaves on log. Whether will we use Linux HA package or NameNode-built-in HA package without the help of outter Linux HA package. 
> After NameNode become high availability, is it necessary to provide HA for Jobtracker? Can Jobtracker  persist the states of Jobs and tasks into HA NameNode? Or Jobtracker also needs the same approach from NameNode for HA support.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-60-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 08:56:02 2009
Return-Path: <common-issues-return-60-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90019 invoked from network); 3 Jul 2009 08:56:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 08:56:02 -0000
Received: (qmail 11375 invoked by uid 500); 3 Jul 2009 08:56:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11345 invoked by uid 500); 3 Jul 2009 08:56:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11307 invoked by uid 99); 3 Jul 2009 08:56:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 08:56:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 08:56:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E92C7234C004
	for <common-issues@hadoop.apache.org>; Fri,  3 Jul 2009 01:55:47 -0700 (PDT)
Message-ID: <490182956.1246611347940.JavaMail.jira@brutus>
Date: Fri, 3 Jul 2009 01:55:47 -0700 (PDT)
From: "Bo Dong (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6121) High Availability support for
 Hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726835#action_12726835 ] 

Bo Dong commented on HADOOP-6121:
---------------------------------

To Amr:
We found that Bookkeeper said Hadoop used Write-Ahead logging approach. 
However, based on my understanding, in 0.19, Hadoop modifies the contents in memory first, and then persists the log. And there is no transaction relationship between modifying memory and persisting log.
For example, in the Create operation (Create a new file entry in the namespace)
[FSNamesystem.java 998] startFileInternal(src, permissions, holder, clientMachine, overwrite, false, replication, blockSize); 
// Add a node child to the namespace, and write the log to a buffer.
[FSNamesystem.java 1000] getEditLog().logSync(); 
// persist the log to the disk.


> High Availability support for Hadoop
> ------------------------------------
>
>                 Key: HADOOP-6121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6121
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: dfs, mapred
>            Reporter: Jie Qiu
>
> Currently, We look at the HA of Hadoop cluster. We need to consider the NameNode HA as well as Jobtracker HA. For NameNode, we want to build primary/standy or master-slaves pattern to provide NameNode HA. Therefore, we need to consider how to ship log between primary/standby/slaves and how commit "write" operation to NameNode after the agreement among primary/standby/slaves on log. Whether will we use Linux HA package or NameNode-built-in HA package without the help of outter Linux HA package. 
> After NameNode become high availability, is it necessary to provide HA for Jobtracker? Can Jobtracker  persist the states of Jobs and tasks into HA NameNode? Or Jobtracker also needs the same approach from NameNode for HA support.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-61-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 09:20:01 2009
Return-Path: <common-issues-return-61-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96135 invoked from network); 3 Jul 2009 09:20:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 09:20:01 -0000
Received: (qmail 42493 invoked by uid 500); 3 Jul 2009 09:20:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42459 invoked by uid 500); 3 Jul 2009 09:20:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42449 invoked by uid 99); 3 Jul 2009 09:20:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 09:20:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 09:20:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 954F1234C004
	for <common-issues@hadoop.apache.org>; Fri,  3 Jul 2009 02:19:47 -0700 (PDT)
Message-ID: <1139664967.1246612787604.JavaMail.jira@brutus>
Date: Fri, 3 Jul 2009 02:19:47 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5640) Allow ServicePlugins to hook
 callbacks into key service events
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726838#action_12726838 ] 

Tom White commented on HADOOP-5640:
-----------------------------------

{quote}
Can you explain more about the ServiceEvent idea? I think this is similar to one of the ideas discussed above, but I think it's cleaner to use actual methods with full signatures rather than exposing a single "handleEvent(ServiceEvent e)" and switching on "e instanceof FooEvent" vs "e instanceof BarEvent".
{quote}

I agree, I was just suggesting having a ServiceEvent parameter to hang other contextual information off, to ease API evolution in the future.

{quote}
bq. SingleArgumentRunnable might be named ServicePluginCallback to better indicate its use.

I disagree - this is a very generic interface that is quite usable for a lot of different things. Since this code is going in util, we might as well make it available for other use cases as well rather than proliferating several interfaces that have the same use.
{quote}

I would argue that code goes into common once it has been demonstrated that it is needed by more than one other subproject. This is true in this case since there will be HDFS and MapReduce plugins, so it makes sense for the service plugin stuff to live in util in common. Whether SingleArgumentRunnable is better called ServicePluginCallback or not depends on whether the interface gains further adoption outside service plugins. This is hard to predict so I'm +0 on calling it SingleArgumentRunnable.


> Allow ServicePlugins to hook callbacks into key service events
> --------------------------------------------------------------
>
>                 Key: HADOOP-5640
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5640
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-5640.txt, hadoop-5640.txt, HADOOP-5640.v2.txt, hadoop-5640.v3.txt
>
>
> HADOOP-5257 added the ability for NameNode and DataNode to start and stop ServicePlugin implementations at NN/DN start/stop. However, this is insufficient integration for some common use cases.
> We should add some functionality for Plugins to subscribe to events generated by the service they're plugging into. Some potential hook points are:
> NameNode:
>   - new datanode registered
>   - datanode has died
>   - exception caught
>   - etc?
> DataNode:
>   - startup
>   - initial registration with NN complete (this is important for HADOOP-4707 to sync up datanode.dnRegistration.name with the NN-side registration)
>   - namenode reconnect
>   - some block transfer hooks?
>   - exception caught
> I see two potential routes for implementation:
> 1) We make an enum for the types of hookpoints and have a general function in the ServicePlugin interface. Something like:
> {code:java}
> enum HookPoint {
>   DN_STARTUP,
>   DN_RECEIVED_NEW_BLOCK,
>   DN_CAUGHT_EXCEPTION,
>  ...
> }
> void runHook(HookPoint hp, Object value);
> {code}
> 2) We make classes specific to each "pluggable" as was originally suggested in HADDOP-5257. Something like:
> {code:java}
> class DataNodePlugin {
>   void datanodeStarted() {}
>   void receivedNewBlock(block info, etc) {}
>   void caughtException(Exception e) {}
>   ...
> }
> {code}
> I personally prefer option (2) since we can ensure plugin API compatibility at compile-time, and we avoid an ugly switch statement in a runHook() function.
> Interested to hear what people's thoughts are here.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-62-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 09:28:01 2009
Return-Path: <common-issues-return-62-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 667 invoked from network); 3 Jul 2009 09:28:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 09:28:01 -0000
Received: (qmail 55501 invoked by uid 500); 3 Jul 2009 09:28:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55464 invoked by uid 500); 3 Jul 2009 09:28:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55454 invoked by uid 99); 3 Jul 2009 09:28:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 09:28:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 09:28:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 6B9B5234C004
	for <common-issues@hadoop.apache.org>; Fri,  3 Jul 2009 02:27:47 -0700 (PDT)
Message-ID: <2030116184.1246613267427.JavaMail.jira@brutus>
Date: Fri, 3 Jul 2009 02:27:47 -0700 (PDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6121) High Availability support for
 Hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726843#action_12726843 ] 

Hemanth Yamijala commented on HADOOP-6121:
------------------------------------------

bq. We find that the JobTracker only persist the state of competed and dead jobs in DFS, not including the state of in-process jobs.

Umm. This is not true. JobTracker persists the state of running jobs as well in the job history files and can recover from this state upon a restart. This was done as part of HADOOP-3245.

> High Availability support for Hadoop
> ------------------------------------
>
>                 Key: HADOOP-6121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6121
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: dfs, mapred
>            Reporter: Jie Qiu
>
> Currently, We look at the HA of Hadoop cluster. We need to consider the NameNode HA as well as Jobtracker HA. For NameNode, we want to build primary/standy or master-slaves pattern to provide NameNode HA. Therefore, we need to consider how to ship log between primary/standby/slaves and how commit "write" operation to NameNode after the agreement among primary/standby/slaves on log. Whether will we use Linux HA package or NameNode-built-in HA package without the help of outter Linux HA package. 
> After NameNode become high availability, is it necessary to provide HA for Jobtracker? Can Jobtracker  persist the states of Jobs and tasks into HA NameNode? Or Jobtracker also needs the same approach from NameNode for HA support.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-63-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 09:48:01 2009
Return-Path: <common-issues-return-63-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14713 invoked from network); 3 Jul 2009 09:48:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 09:48:01 -0000
Received: (qmail 79466 invoked by uid 500); 3 Jul 2009 09:48:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79430 invoked by uid 500); 3 Jul 2009 09:48:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79420 invoked by uid 99); 3 Jul 2009 09:48:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 09:48:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 09:48:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 2F951234C004
	for <common-issues@hadoop.apache.org>; Fri,  3 Jul 2009 02:47:47 -0700 (PDT)
Message-ID: <141304229.1246614467180.JavaMail.jira@brutus>
Date: Fri, 3 Jul 2009 02:47:47 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6125) extend DistributedCache to work
 locally (LocalJobRunner) (common half)
In-Reply-To: <682341596.1246484267288.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12726848#action_12726848 ] 

Hadoop QA commented on HADOOP-6125:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12412324/HADOOP-6125.patch
  against trunk revision 790831.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 4 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs.  The patch appears to introduce 3 new Findbugs warnings.

    -1 release audit.  The applied patch generated 259 release audit warnings (more than the trunk's current 258 warnings).

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/554/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/554/artifact/trunk/current/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/554/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/554/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/554/console

This message is automatically generated.

> extend DistributedCache to work locally (LocalJobRunner) (common half)
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6125
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6125
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Philip Zeyliger
>            Assignee: Philip Zeyliger
>            Priority: Minor
>         Attachments: HADOOP-6125.patch
>
>
> This is the co-ticket to MAPREDUCE-476, covering the significant part of DistributedCache that's in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-64-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 10:40:02 2009
Return-Path: <common-issues-return-64-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44122 invoked from network); 3 Jul 2009 10:40:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 10:40:02 -0000
Received: (qmail 67101 invoked by uid 500); 3 Jul 2009 10:40:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67080 invoked by uid 500); 3 Jul 2009 10:40:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67026 invoked by uid 99); 3 Jul 2009 10:40:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 10:40:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 10:40:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 57B31234C053
	for <common-issues@hadoop.apache.org>; Fri,  3 Jul 2009 03:39:47 -0700 (PDT)
Message-ID: <1355474846.1246617587358.JavaMail.jira@brutus>
Date: Fri, 3 Jul 2009 03:39:47 -0700 (PDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4491) Per-job local data on the TaskTracker
 node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V updated HADOOP-4491:
------------------------------

    Attachment: HADOOP-4491-20090703-common.txt
                HADOOP-4491-20090703.txt

Adding patches for quick review. Still a work in progress. Incorporated most of Devaraj's comments.

Two major changes from the previous patch:
 - Pulled _taskTracker/jobcache/attemp-id/work_ to _taskTracker/jobcache/task-work/attempt-id/work_. This is done to solve issues with jvm reuse. The work dir needs to be shared across tasks with jvm reuse and so cannot be finalized to be owned back by the TT on task finish. Pulling this out makes things clean, the original attempt-dir is finalized, and the task-work directory is cleaned up on jvm exit. TODO: task-work can have better name, perhaps jvmcache.

 - _log-dir/userlogs/attempt-id_ will still have 777 permissions with all the files inside created by linuxTaskController but owned by the user. This is done because
    -- attempt-id dir has to be writable by the child for log.tmp which is done periodically in synclogs.
    -- attempt-dir has to be readable by TT for log serving and writable by TT also by cleanup.

Long time, the correct solution is for this directory to be owned by the child, but cleaned up by TT using a LinuxTaskController binary lauch to swith user.

Pending items from Devaraj's comments:
 -  It will be nice to combine the APIs for creating files/directories and setting appropriate permissions all in one API
 - DiskChecker.java has lot of code to do with permissions handling to make it generic but not everything would be actually used. In fact, the approach taken in making some APIs generic is debatable. We might as well keep it simple for now and extend those APIs as and when required.
 - LocalDirAllocator.getSecureLocalPathForWrite could be renamed as getPrivateLocalPathForEWrite 

Extra:
 - Test TaskTracker.initializeJobDirs
 - Extra tests in taskcontroller.c
 - Test various scenarios of finalizeTaskDirs
 - Finish tests for jvm reuse

There are some minor TODO's also left to do.

> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: mapred, security
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-65-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 11:43:01 2009
Return-Path: <common-issues-return-65-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63474 invoked from network); 3 Jul 2009 11:43:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 11:43:01 -0000
Received: (qmail 31281 invoked by uid 500); 3 Jul 2009 11:43:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31242 invoked by uid 500); 3 Jul 2009 11:43:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31232 invoked by uid 99); 3 Jul 2009 11:43:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 11:43:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 11:43:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 32A32234C004
	for <common-issues@hadoop.apache.org>; Fri,  3 Jul 2009 04:42:47 -0700 (PDT)
Message-ID: <1949476113.1246621367192.JavaMail.jira@brutus>
Date: Fri, 3 Jul 2009 04:42:47 -0700 (PDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4491) Per-job local data on the TaskTracker
 node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V updated HADOOP-4491:
------------------------------

    Attachment: HADOOP-4491-20090703-common.1.txt
                HADOOP-4491-20090703.1.txt

Uploading patches that include the rest of the comments, except
bq. # It will be nice to combine the APIs for creating files/directories and setting appropriate permissions all in one API
There are very few occurrences of above now in the code. So dropping this.

The patches can be reviewed, while I try any possible code reuse/refactoring and making test-cases comprehensive.

> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: mapred, security
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, HADOOP-4491-20090703.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-66-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 17:20:43 2009
Return-Path: <common-issues-return-66-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5729 invoked from network); 3 Jul 2009 17:20:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 17:20:43 -0000
Received: (qmail 48182 invoked by uid 500); 3 Jul 2009 17:14:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48164 invoked by uid 500); 3 Jul 2009 17:14:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48139 invoked by uid 99); 3 Jul 2009 17:14:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 17:14:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 17:14:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 5B715234C055
	for <common-issues@hadoop.apache.org>; Fri,  3 Jul 2009 10:13:47 -0700 (PDT)
Message-ID: <105865310.1246641227373.JavaMail.jira@brutus>
Date: Fri, 3 Jul 2009 10:13:47 -0700 (PDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727052#action_12727052 ] 

Arun C Murthy commented on HADOOP-5170:
---------------------------------------

I think we need to take a step back.

The contention is that the specific knobs in question have long-term ramifications. In particular putting these in the framework proper, as opposed to specific schedulers, impose constraints on _all_ schedulers regardless. We are happy to have specific schedulers targeted for specific use-cases/workloads support them, but we are opposed to allowing features which jeopardize one over the other, especially in the rather obvious way as we've highlighted above. 

In the past, and possibly in the future, we will make some choices which hurt use-case one over the other; in which case we will almost always be biased towards large, multi-user clusters.

----

Having said that, there are *some* workarounds even with the CapacityScheduler (https://issues.apache.org/jira/browse/HADOOP-5170?focusedCommentId=12726751&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12726751) if people are so inclined.



> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-67-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 18:08:01 2009
Return-Path: <common-issues-return-67-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33888 invoked from network); 3 Jul 2009 18:08:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 18:08:01 -0000
Received: (qmail 6680 invoked by uid 500); 3 Jul 2009 18:08:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6642 invoked by uid 500); 3 Jul 2009 18:08:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6557 invoked by uid 99); 3 Jul 2009 18:08:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 18:08:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 18:08:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 48069234C055
	for <common-issues@hadoop.apache.org>; Fri,  3 Jul 2009 11:07:47 -0700 (PDT)
Message-ID: <1463158320.1246644467294.JavaMail.jira@brutus>
Date: Fri, 3 Jul 2009 11:07:47 -0700 (PDT)
From: "Matei Zaharia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727072#action_12727072 ] 

Matei Zaharia commented on HADOOP-5170:
---------------------------------------

I've opened MAPREDUCE-704 to add per-node task limits in the fair scheduler. In addition, MAPREDUCE-698 is for per-pool task limits. Are people alright with having per-pool limits instead of per-job limits? Pools allow you to group multiple jobs under the same limit.

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-68-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 18:29:59 2009
Return-Path: <common-issues-return-68-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52427 invoked from network); 3 Jul 2009 18:29:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 18:29:59 -0000
Received: (qmail 29872 invoked by uid 500); 3 Jul 2009 18:30:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29831 invoked by uid 500); 3 Jul 2009 18:30:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29821 invoked by uid 99); 3 Jul 2009 18:30:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 18:30:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 18:30:07 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3878C234C004
	for <common-issues@hadoop.apache.org>; Fri,  3 Jul 2009 11:29:47 -0700 (PDT)
Message-ID: <1448788359.1246645787223.JavaMail.jira@brutus>
Date: Fri, 3 Jul 2009 11:29:47 -0700 (PDT)
From: "Jonathan Gray (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727081#action_12727081 ] 

Jonathan Gray commented on HADOOP-5170:
---------------------------------------

I can just put all my jobs into a single pool, and have the same functionality I would have now, correct?

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-69-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 18:39:59 2009
Return-Path: <common-issues-return-69-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57997 invoked from network); 3 Jul 2009 18:39:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 18:39:59 -0000
Received: (qmail 36819 invoked by uid 500); 3 Jul 2009 18:40:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36781 invoked by uid 500); 3 Jul 2009 18:40:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36765 invoked by uid 99); 3 Jul 2009 18:40:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 18:40:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 18:40:07 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 38D4E234C004
	for <common-issues@hadoop.apache.org>; Fri,  3 Jul 2009 11:39:47 -0700 (PDT)
Message-ID: <1107413393.1246646387219.JavaMail.jira@brutus>
Date: Fri, 3 Jul 2009 11:39:47 -0700 (PDT)
From: "Matei Zaharia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727087#action_12727087 ] 

Matei Zaharia commented on HADOOP-5170:
---------------------------------------

You'd be able to place a limit on the whole pool - for example, 100 maps. If you submit one job at a time to the pool, then that job will get the whole limit. However, if you submit two jobs, they will share it, and they will get 100 maps in total, not 100 each. Right now the jobs will do fair sharing, meaning that each job will get 50 maps. However, FIFO scheduling within a pool will also be supported by the fair scheduler in the future (I have a fairly well-tested patch for it that I am porting to trunk).

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-70-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 18:48:05 2009
Return-Path: <common-issues-return-70-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59317 invoked from network); 3 Jul 2009 18:48:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 18:48:02 -0000
Received: (qmail 41947 invoked by uid 500); 3 Jul 2009 18:48:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41910 invoked by uid 500); 3 Jul 2009 18:48:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41750 invoked by uid 99); 3 Jul 2009 18:48:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 18:48:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 18:48:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 53B31234C053
	for <common-issues@hadoop.apache.org>; Fri,  3 Jul 2009 11:47:47 -0700 (PDT)
Message-ID: <2037891107.1246646867341.JavaMail.jira@brutus>
Date: Fri, 3 Jul 2009 11:47:47 -0700 (PDT)
From: "Jonathan Gray (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727093#action_12727093 ] 

Jonathan Gray commented on HADOOP-5170:
---------------------------------------

So pooling is just a one-time thing when I submit the job?  It's not something that persists and I submit things into?

I'm a big consumer of MR but have been on a need-to-know basis with respect to these things.  I guess I now need to know.  Again, part of what I liked about this issue/solution was that it's powerful, accessible, and easy to understand.  I understand the concerns of larger users and the need to support this... And I would again ask if we could stick it into a corner somewhere so that it's still easy to access but does not get in the way of everything else.

Otherwise, what I'd be interested in is an explanation / example of how users of this patch might accomplish the same types of things.  For example, only allowing a particular job to use one task per node (or even total tasks at a time = total nodes).  And at the same time, having other jobs that I allow 10s of tasks per node.  I'm not following how that would work.

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-71-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 20:31:03 2009
Return-Path: <common-issues-return-71-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93529 invoked from network); 3 Jul 2009 20:31:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 20:31:01 -0000
Received: (qmail 22889 invoked by uid 500); 3 Jul 2009 20:31:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22852 invoked by uid 500); 3 Jul 2009 20:31:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22842 invoked by uid 99); 3 Jul 2009 20:31:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 20:31:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 20:31:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 53000234C053
	for <common-issues@hadoop.apache.org>; Fri,  3 Jul 2009 13:30:47 -0700 (PDT)
Message-ID: <1193262499.1246653047339.JavaMail.jira@brutus>
Date: Fri, 3 Jul 2009 13:30:47 -0700 (PDT)
From: "Matei Zaharia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727122#action_12727122 ] 

Matei Zaharia commented on HADOOP-5170:
---------------------------------------

No, pools are persistent. You submit a job to a particular pool by setting a jobconf property (e.g. set pool.name="my_pool"). Then you'll be able to have caps on total maps or total reduces running for each pool. For example, you could limit your DB import pool to 20 mappers, and then *all* DB import jobs together will get no more than 10 mappers.

I was planning to make the per-node limits be on a per job basis as in the current patch. However, for the per-job limits, it seemed to make more sense to let them apply across multiple jobs by placing them on a pool.

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-72-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 20:31:03 2009
Return-Path: <common-issues-return-72-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93542 invoked from network); 3 Jul 2009 20:31:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 20:31:01 -0000
Received: (qmail 22955 invoked by uid 500); 3 Jul 2009 20:31:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22919 invoked by uid 500); 3 Jul 2009 20:31:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22887 invoked by uid 99); 3 Jul 2009 20:31:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 20:31:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 20:31:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 7E7E6234C4B2
	for <common-issues@hadoop.apache.org>; Fri,  3 Jul 2009 13:30:47 -0700 (PDT)
Message-ID: <148761204.1246653047517.JavaMail.jira@brutus>
Date: Fri, 3 Jul 2009 13:30:47 -0700 (PDT)
From: "Matei Zaharia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727124#action_12727124 ] 

Matei Zaharia commented on HADOOP-5170:
---------------------------------------

So in other words, to answer your second question, limiting one job to one task/node will be done as in the current patch (with mapred.max.maps.per.node).

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-73-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 20:37:01 2009
Return-Path: <common-issues-return-73-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95624 invoked from network); 3 Jul 2009 20:37:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 20:37:01 -0000
Received: (qmail 25900 invoked by uid 500); 3 Jul 2009 20:37:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25863 invoked by uid 500); 3 Jul 2009 20:37:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25848 invoked by uid 99); 3 Jul 2009 20:37:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 20:37:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 20:37:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3B678234C004
	for <common-issues@hadoop.apache.org>; Fri,  3 Jul 2009 13:36:47 -0700 (PDT)
Message-ID: <1756707390.1246653407229.JavaMail.jira@brutus>
Date: Fri, 3 Jul 2009 13:36:47 -0700 (PDT)
From: "Jonathan Gray (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727127#action_12727127 ] 

Jonathan Gray commented on HADOOP-5170:
---------------------------------------

Makes sense.  Thanks Matei.

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-74-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 03 21:23:14 2009
Return-Path: <common-issues-return-74-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7889 invoked from network); 3 Jul 2009 21:23:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Jul 2009 21:23:14 -0000
Received: (qmail 56246 invoked by uid 500); 3 Jul 2009 21:23:25 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56208 invoked by uid 500); 3 Jul 2009 21:23:25 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56198 invoked by uid 99); 3 Jul 2009 21:23:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 21:23:25 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jul 2009 21:23:14 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 7F4DF234C004
	for <common-issues@hadoop.apache.org>; Fri,  3 Jul 2009 14:22:53 -0700 (PDT)
Message-ID: <704374866.1246656173507.JavaMail.jira@brutus>
Date: Fri, 3 Jul 2009 14:22:53 -0700 (PDT)
From: "Philip Zeyliger (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6105) Provide a way to automatically
 handle backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727143#action_12727143 ] 

Philip Zeyliger commented on HADOOP-6105:
-----------------------------------------

I'm not enamored of this approach and would like to propose
a slightly heavier-weight, but, I think, cleaner approach
than stuffing more logic into the Configuration class.
My apologies for coming to this conversation a bit late.

If you don't want to read a long e-mail, skip down to the code examples
at the bottom. :)

Before I get to the proposal, I wanted to lay out what I think the goals
are.  Note that HADOOP-475 is also related.

* Standardization of configuration names, documentation, and 
value formats.  Today, the names tend to appear in the code, or, at best,
in constants in the code, and the documentation, when it exists,
may be in -default.xml.  It would be nice if it was very difficult
to avoid writing documentation for the variable you're introducing.
Right now there are and have been a handful of bugs where the default
in the code is different than the default in the XML file, and
that gets really confusing.

* Backwards compatibility.  We'd love to rename "mapred.foo" and "mr.bar"
to be consistent, but we want to maintain backwards compatibility.
This ticket is all about that.

* Availability to user code.  Users should be able to use configuration the same way the core does.
Users pass information to their jobs via Configuration, and they should
use the same mechanism.  This is true today.

* Type-safety. Configurations have a handful of recurring types: number of bytes,
filename, URI, hostport combination, arrays of paths, etc.  The parsing
is done in an ad-hoc fashion, which is a shame, since it doesn't have to be.
It would be nice to have some generic runtime checking of configuration
parameters, too, and perhaps even ranges (that number can't be negative!).

* Upgradeability to a different configuration format.  I don't think we'll
leave a place where configuration has to be a key->value map (especially
because of "availability to user code", but it would eventually be nice
if configuration could be queried from other places, or if the
values could have a bit more structure.  (For example, we could use XML
to separate out a list of paths, instead of blindly using comma-delimited,
unescaped text.)

* Development ease.  It ought to be easier to find the places where configuration
is used.  Today the best we can do is a grep, and then follow references
manually.

* Autogenerated documentation.  No-brainer.

* Ability to specify visibility, scope, and stability.  Alogn the lines of HADOOP-5073, configuration
variables should be classified as deprecated, unstable, evolving, and stable.  It would be
nice to introduce variables (say, that were used for tuning), with the expectation that they are
not part of the public API.  Use at your own risk sort of thing.

My proposal is to represent every configuration variable that's accessed in
the Hadoop code by a static instance of a ConfigVariable<T> class.  The interface
is something like:

{code}
public interface ConfigValue<T> {
  T get(Configuration conf);
  T getDefault();
  void set(Configuration conf, T value);
  String getHelp();
}
{code}

There's more than one way to implement this.  Here's one proposal that uses
Java annotations:

{code}
  @ConfigDescription(help="Some help text", 
      visibility=Visibility.PUBLIC)
  @ConfigAccessors({
    @ConfigAccessor(name="common.sample"),
    @ConfigAccessor(name="core.sample", deprecated="Use common.sample instead")
  })
  public final static ConfigVariable<Integer> myConfigVariable = 
    ConfigVariables.newIntConfigVariable(15 /* default value */);
{code}
This approach would require pre-processing (at build time) the annotations
into a data file, and then, at runtime, querying this data file.
(It's not easily possible to get at the annotations on the
field from within myConfigVariable.)

I'm half-way to getting this working, and I actually think something
like the following would be better:
{code}
  @ConfigVariableDeclaration
  public final static ConfigVariable<URI> fsDefaultName = 
    ConfigVariableBuilder.newURI()
      .setDefault(null)
      .setHelp("Default filesystem")
      .setVisibility(Visibility.PUBLIC)
      .addAccessor("fs.default.name")
      .addDeprecatedAccessor("core.default.fs", "Use foo instead")
      .addValidator(new ValidateSupportedFilesystem());
{code}
This would still require build-time preprocessing (javac supports
this) to find the variables, instantiate them, and output
the documentation, but the rest of the processing is easy
at runtime.

A drawback of this approach is how to handle the defaults that
default to other variables.  Perhaps the easiest thing to do 
is to handle the same syntax we support now, like 'addIndirectDefault("${default.dir}/mapred")',
but something that references the other variable directly is more appealing, e.g.: 'addIndirectDefault(OtherClass.class, "fieldname")'.

I think this can be implemented relatively quickly, with little impact on
breaking stuff (because the old way of using Configuration continues to work).

What do you think?


> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-75-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 04 03:51:02 2009
Return-Path: <common-issues-return-75-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32825 invoked from network); 4 Jul 2009 03:51:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Jul 2009 03:51:02 -0000
Received: (qmail 27315 invoked by uid 500); 4 Jul 2009 03:51:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27279 invoked by uid 500); 4 Jul 2009 03:51:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27262 invoked by uid 99); 4 Jul 2009 03:51:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jul 2009 03:51:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jul 2009 03:51:08 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 9FB37234C004
	for <common-issues@hadoop.apache.org>; Fri,  3 Jul 2009 20:50:47 -0700 (PDT)
Message-ID: <432329017.1246679447640.JavaMail.jira@brutus>
Date: Fri, 3 Jul 2009 20:50:47 -0700 (PDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727182#action_12727182 ] 

Devaraj Das commented on HADOOP-5170:
-------------------------------------

Thought about it and i buy the argument that these knobs should not have been added in the core framework. So +1 for reverting this patch.

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-76-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 06 10:06:27 2009
Return-Path: <common-issues-return-76-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78966 invoked from network); 6 Jul 2009 10:06:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jul 2009 10:06:27 -0000
Received: (qmail 79411 invoked by uid 500); 6 Jul 2009 10:06:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79372 invoked by uid 500); 6 Jul 2009 10:06:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79362 invoked by uid 99); 6 Jul 2009 10:06:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jul 2009 10:06:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jul 2009 10:06:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EEA07234C04C
	for <common-issues@hadoop.apache.org>; Mon,  6 Jul 2009 03:06:14 -0700 (PDT)
Message-ID: <178650067.1246874774976.JavaMail.jira@brutus>
Date: Mon, 6 Jul 2009 03:06:14 -0700 (PDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6105) Provide a way to automatically
 handle backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727460#action_12727460 ] 

Hemanth Yamijala commented on HADOOP-6105:
------------------------------------------

Philip, this seems a very ambitious change to the Configuration framework. Before I can comment further, I would like to take a look at the exact JIRA where this is being discussed. Though I could find out a couple of JIRAs that seem related, none had all the points that you've mentioned here. Can you point me to such a jira, if it exists ?

If not, I think it is important to open a new JIRA to discuss these points. That way it will find a better audience and we can discuss better. Please let me know about this.

> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-77-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 06 11:22:29 2009
Return-Path: <common-issues-return-77-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6017 invoked from network); 6 Jul 2009 11:22:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jul 2009 11:22:29 -0000
Received: (qmail 54703 invoked by uid 500); 6 Jul 2009 11:22:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54672 invoked by uid 500); 6 Jul 2009 11:22:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54662 invoked by uid 99); 6 Jul 2009 11:22:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jul 2009 11:22:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jul 2009 11:22:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E2060234C053
	for <common-issues@hadoop.apache.org>; Mon,  6 Jul 2009 04:22:14 -0700 (PDT)
Message-ID: <1085003647.1246879334924.JavaMail.jira@brutus>
Date: Mon, 6 Jul 2009 04:22:14 -0700 (PDT)
From: "Jothi Padmanabhan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6126) Hadoop QA mails on Patch tests do not
 have links to test failures
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Hadoop QA mails on Patch tests do not have links to test failures
-----------------------------------------------------------------

                 Key: HADOOP-6126
                 URL: https://issues.apache.org/jira/browse/HADOOP-6126
             Project: Hadoop Common
          Issue Type: Bug
          Components: build
            Reporter: Jothi Padmanabhan


The recent hadoopqa results on test patches do not seem to have the links to the actual tests that failed. 
For example, http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/356/testReport/

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-78-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 06 16:07:27 2009
Return-Path: <common-issues-return-78-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80562 invoked from network); 6 Jul 2009 16:07:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jul 2009 16:07:26 -0000
Received: (qmail 44657 invoked by uid 500); 6 Jul 2009 16:07:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44627 invoked by uid 500); 6 Jul 2009 16:07:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44617 invoked by uid 99); 6 Jul 2009 16:07:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jul 2009 16:07:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jul 2009 16:07:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DC9E8234C051
	for <common-issues@hadoop.apache.org>; Mon,  6 Jul 2009 09:07:14 -0700 (PDT)
Message-ID: <1673508918.1246896434902.JavaMail.jira@brutus>
Date: Mon, 6 Jul 2009 09:07:14 -0700 (PDT)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6126) Hadoop QA mails on Patch tests do
 not have links to test failures
In-Reply-To: <1085003647.1246879334924.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nigel Daley reassigned HADOOP-6126:
-----------------------------------

    Assignee: Giridharan Kesavan

> Hadoop QA mails on Patch tests do not have links to test failures
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6126
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6126
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Jothi Padmanabhan
>            Assignee: Giridharan Kesavan
>
> The recent hadoopqa results on test patches do not seem to have the links to the actual tests that failed. 
> For example, http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/356/testReport/

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-79-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 06 16:47:27 2009
Return-Path: <common-issues-return-79-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3095 invoked from network); 6 Jul 2009 16:47:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jul 2009 16:47:27 -0000
Received: (qmail 17820 invoked by uid 500); 6 Jul 2009 16:47:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17785 invoked by uid 500); 6 Jul 2009 16:47:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17775 invoked by uid 99); 6 Jul 2009 16:47:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jul 2009 16:47:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jul 2009 16:47:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 72C16234C004
	for <common-issues@hadoop.apache.org>; Mon,  6 Jul 2009 09:47:15 -0700 (PDT)
Message-ID: <437553404.1246898835465.JavaMail.jira@brutus>
Date: Mon, 6 Jul 2009 09:47:15 -0700 (PDT)
From: "Philip Zeyliger (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6105) Provide a way to automatically
 handle backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727615#action_12727615 ] 

Philip Zeyliger commented on HADOOP-6105:
-----------------------------------------

Hemanth,

This JIRA is about backwards-compatibility of deprecated keys, which is something my comment addresses, so I thought it fit in well here.  Think of it as an alternative solution to the problem you're trying to solve by keeping the map of deprecated keys in Configuration.java.  Keeping a deprecation map is expedient and simple, but I think it may hamper a better, longer-term solution.

The design goals above are "out of thin air" (in the sense that they haven't been discussed on JIRA outside of the JIRAs mentioned above and MAPREDUCE-475), though I hope they're reasonable.  They were discussed a bit at http://wiki.apache.org/hadoop/DeveloperOffsite20090612, too.  That said, I hope they help to frame the conversation a bit

I very very much want there to be a path to be able to rename configuration keys, but I want to make sure that the solution that comes out of this JIRA is compatible with some future work.

-- Philip

> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-80-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 06 17:07:27 2009
Return-Path: <common-issues-return-80-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13220 invoked from network); 6 Jul 2009 17:07:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jul 2009 17:07:27 -0000
Received: (qmail 49532 invoked by uid 500); 6 Jul 2009 17:07:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49496 invoked by uid 500); 6 Jul 2009 17:07:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49482 invoked by uid 99); 6 Jul 2009 17:07:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jul 2009 17:07:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jul 2009 17:07:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D1DAC234C04C
	for <common-issues@hadoop.apache.org>; Mon,  6 Jul 2009 10:07:14 -0700 (PDT)
Message-ID: <676967224.1246900034858.JavaMail.jira@brutus>
Date: Mon, 6 Jul 2009 10:07:14 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6127) The real user name should be used by
 bin/hadoop fs (ie. FsShell) instead of the one in the configuration.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

The real user name should be used by bin/hadoop fs (ie. FsShell) instead of the one in the configuration.
---------------------------------------------------------------------------------------------------------

                 Key: HADOOP-6127
                 URL: https://issues.apache.org/jira/browse/HADOOP-6127
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
            Reporter: Owen O'Malley


The real user name should be used by FsShell instead of the one in the configuration. This will make it a tiny bit harder for someone to pretend to be someone else to the file system.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-81-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 06 17:13:36 2009
Return-Path: <common-issues-return-81-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16439 invoked from network); 6 Jul 2009 17:13:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jul 2009 17:13:36 -0000
Received: (qmail 54894 invoked by uid 500); 6 Jul 2009 17:13:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54857 invoked by uid 500); 6 Jul 2009 17:13:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54847 invoked by uid 99); 6 Jul 2009 17:13:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jul 2009 17:13:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jul 2009 17:13:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C7B85234C004
	for <common-issues@hadoop.apache.org>; Mon,  6 Jul 2009 10:13:14 -0700 (PDT)
Message-ID: <782288749.1246900394803.JavaMail.jira@brutus>
Date: Mon, 6 Jul 2009 10:13:14 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6127) The real user name should be used
 by bin/hadoop fs (ie. FsShell) instead of the one in the configuration.
In-Reply-To: <676967224.1246900034858.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727624#action_12727624 ] 

Todd Lipcon commented on HADOOP-6127:
-------------------------------------

How can you get the "real" username without being subject to spoofing? It seems to me that the user can always play LD_PRELOAD tricks so that the call out to "whoami" goes to "evil-whoami.sh". Without something like identd or real token-based authentication I don't know that it's really possible to add any security here.

> The real user name should be used by bin/hadoop fs (ie. FsShell) instead of the one in the configuration.
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6127
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6127
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Owen O'Malley
>
> The real user name should be used by FsShell instead of the one in the configuration. This will make it a tiny bit harder for someone to pretend to be someone else to the file system.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-82-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 06 17:45:32 2009
Return-Path: <common-issues-return-82-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46885 invoked from network); 6 Jul 2009 17:45:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jul 2009 17:45:31 -0000
Received: (qmail 20075 invoked by uid 500); 6 Jul 2009 17:45:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20039 invoked by uid 500); 6 Jul 2009 17:45:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20029 invoked by uid 99); 6 Jul 2009 17:45:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jul 2009 17:45:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jul 2009 17:45:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id ED0F3234C055
	for <common-issues@hadoop.apache.org>; Mon,  6 Jul 2009 10:45:14 -0700 (PDT)
Message-ID: <446837415.1246902314969.JavaMail.jira@brutus>
Date: Mon, 6 Jul 2009 10:45:14 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6127) The real user name should be used
 by bin/hadoop fs (ie. FsShell) instead of the one in the configuration.
In-Reply-To: <676967224.1246900034858.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727637#action_12727637 ] 

Owen O'Malley commented on HADOOP-6127:
---------------------------------------

This isn't about making it secure. It is just about removing the ability to spoof from the command line interface to Hadoop. Even after this change, it is still easy to spoof without getting to LD_PRELOAD. Of course to get to real security, you need authentication via Kerberos or something similar.

> The real user name should be used by bin/hadoop fs (ie. FsShell) instead of the one in the configuration.
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6127
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6127
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Owen O'Malley
>
> The real user name should be used by FsShell instead of the one in the configuration. This will make it a tiny bit harder for someone to pretend to be someone else to the file system.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-83-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 06 18:01:31 2009
Return-Path: <common-issues-return-83-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62442 invoked from network); 6 Jul 2009 18:01:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Jul 2009 18:01:28 -0000
Received: (qmail 42912 invoked by uid 500); 6 Jul 2009 18:01:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42877 invoked by uid 500); 6 Jul 2009 18:01:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42867 invoked by uid 99); 6 Jul 2009 18:01:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jul 2009 18:01:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jul 2009 18:01:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CB544234C004
	for <common-issues@hadoop.apache.org>; Mon,  6 Jul 2009 11:01:14 -0700 (PDT)
Message-ID: <1461589198.1246903274828.JavaMail.jira@brutus>
Date: Mon, 6 Jul 2009 11:01:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5837) TestHdfsProxy fails in Linux
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727649#action_12727649 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-5837:
------------------------------------------------

Zhiyong, thanks for checking this.

BTW, TestHdfsProxy keeps failing from [the first build|http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hdfs-Patch-vesta.apache.org/0/testReport/] to [the latest build|http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hdfs-Patch-vesta.apache.org/4/testReport/] in [Hudson|http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hdfs-Patch-vesta.apache.org/] after the project split.  Could you take a look?  Thanks again.

> TestHdfsProxy fails in Linux
> ----------------------------
>
>                 Key: HADOOP-5837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5837
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: contrib/hdfsproxy
>         Environment: Linux hostname 2.6.9-55.ELsmp #1 SMP Fri Apr 20 16:36:54 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
>            Reporter: Tsz Wo (Nicholas), SZE
>
> {noformat}
> test-junit:
>     [junit] Running org.apache.hadoop.hdfsproxy.TestHdfsProxy
>     [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 4.397 sec
>     [junit] Test org.apache.hadoop.hdfsproxy.TestHdfsProxy FAILED
>     [junit] Running org.apache.hadoop.hdfsproxy.TestProxyUgiManager
>     [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 4.219 sec
> BUILD FAILED
> /home/tsz/hadoop/latest/build.xml:1022: The following error occurred while executing this line:
> /home/tsz/hadoop/latest/src/contrib/build.xml:48: The following error occurred while executing this line:
> /home/tsz/hadoop/latest/src/contrib/hdfsproxy/build.xml:224: Tests failed!
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-84-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 00:33:27 2009
Return-Path: <common-issues-return-84-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60172 invoked from network); 7 Jul 2009 00:33:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 00:33:26 -0000
Received: (qmail 13380 invoked by uid 500); 7 Jul 2009 00:33:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13343 invoked by uid 500); 7 Jul 2009 00:33:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13333 invoked by uid 99); 7 Jul 2009 00:33:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 00:33:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 00:33:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D9326234C004
	for <common-issues@hadoop.apache.org>; Mon,  6 Jul 2009 17:33:14 -0700 (PDT)
Message-ID: <2022618474.1246926794878.JavaMail.jira@brutus>
Date: Mon, 6 Jul 2009 17:33:14 -0700 (PDT)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6121) High Availability support for
 Hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727848#action_12727848 ] 

Konstantin Shvachko commented on HADOOP-6121:
---------------------------------------------

Jie Qiu> we need to consider how to ship log between primary/standby/slaves 

Are you aware of BackupNode introduced in HADOOP-4539?
I thought the log shipping part has been solved by that. Or did you mean anything else?

Jie Qiu> Memory write means changes to the BTree,

What B-tree did you mean?

Jie Qiu> Standby heartbeat primary and take over if it feels primary down

How exactly the standby will decide ("feel" in your terms) that the primary is down?

Also it would be really good to have a document, which gives a general overview and details of your design. You can refer to other jiras for examples. We used to have design documents for large changes like the one you are attempting here.

> High Availability support for Hadoop
> ------------------------------------
>
>                 Key: HADOOP-6121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6121
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: dfs, mapred
>            Reporter: Jie Qiu
>
> Currently, We look at the HA of Hadoop cluster. We need to consider the NameNode HA as well as Jobtracker HA. For NameNode, we want to build primary/standy or master-slaves pattern to provide NameNode HA. Therefore, we need to consider how to ship log between primary/standby/slaves and how commit "write" operation to NameNode after the agreement among primary/standby/slaves on log. Whether will we use Linux HA package or NameNode-built-in HA package without the help of outter Linux HA package. 
> After NameNode become high availability, is it necessary to provide HA for Jobtracker? Can Jobtracker  persist the states of Jobs and tasks into HA NameNode? Or Jobtracker also needs the same approach from NameNode for HA support.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-85-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 01:47:26 2009
Return-Path: <common-issues-return-85-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85375 invoked from network); 7 Jul 2009 01:47:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 01:47:26 -0000
Received: (qmail 49337 invoked by uid 500); 7 Jul 2009 01:47:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49298 invoked by uid 500); 7 Jul 2009 01:47:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49288 invoked by uid 99); 7 Jul 2009 01:47:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 01:47:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 01:47:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DE64E234C1E9
	for <common-issues@hadoop.apache.org>; Mon,  6 Jul 2009 18:47:14 -0700 (PDT)
Message-ID: <647684764.1246931234910.JavaMail.jira@brutus>
Date: Mon, 6 Jul 2009 18:47:14 -0700 (PDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5482)
 org.apache.hadoop.mapred.lib.db.DBInputformat not working with oracle
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron Kimball updated HADOOP-5482:
----------------------------------

    Attachment: HADOOP-5482.trunk.patch

> org.apache.hadoop.mapred.lib.db.DBInputformat not working with oracle
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-5482
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5482
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.18.2, 0.18.3, 0.19.0, 0.19.1
>         Environment: Java 1.6, HAdoop0.19.0, Linux..Oracle, 
>            Reporter: evanand
>            Priority: Blocker
>         Attachments: HADOOP-5482.20-branch.patch, HADOOP-5482.patch, HADOOP-5482.trunk.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> org.apache.hadoop.mapred.lib.db.DBInputformat not working with oracle.
> The out of the box implementation of the Hadoop is working properly with mysql/hsqldb, but NOT with oracle.
> Reason is DBInputformat is implemented with mysql/hsqldb specific query constructs like "LIMIT", "OFFSET".
> FIX:
> building a database provider specific logic based on the database providername (which we can get using connection).
> I HAVE ALREADY IMPLEMENTED IT FOR ORACLE...READY TO CHECK_IN CODE

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-86-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 01:47:27 2009
Return-Path: <common-issues-return-86-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85426 invoked from network); 7 Jul 2009 01:47:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 01:47:27 -0000
Received: (qmail 49450 invoked by uid 500); 7 Jul 2009 01:47:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49375 invoked by uid 500); 7 Jul 2009 01:47:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49355 invoked by uid 99); 7 Jul 2009 01:47:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 01:47:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 01:47:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DB882234C056
	for <common-issues@hadoop.apache.org>; Mon,  6 Jul 2009 18:47:14 -0700 (PDT)
Message-ID: <1736854615.1246931234898.JavaMail.jira@brutus>
Date: Mon, 6 Jul 2009 18:47:14 -0700 (PDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5482)
 org.apache.hadoop.mapred.lib.db.DBInputformat not working with oracle
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron Kimball updated HADOOP-5482:
----------------------------------

    Attachment: HADOOP-5482.20-branch.patch

> org.apache.hadoop.mapred.lib.db.DBInputformat not working with oracle
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-5482
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5482
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.18.2, 0.18.3, 0.19.0, 0.19.1
>         Environment: Java 1.6, HAdoop0.19.0, Linux..Oracle, 
>            Reporter: evanand
>            Priority: Blocker
>         Attachments: HADOOP-5482.20-branch.patch, HADOOP-5482.patch, HADOOP-5482.trunk.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> org.apache.hadoop.mapred.lib.db.DBInputformat not working with oracle.
> The out of the box implementation of the Hadoop is working properly with mysql/hsqldb, but NOT with oracle.
> Reason is DBInputformat is implemented with mysql/hsqldb specific query constructs like "LIMIT", "OFFSET".
> FIX:
> building a database provider specific logic based on the database providername (which we can get using connection).
> I HAVE ALREADY IMPLEMENTED IT FOR ORACLE...READY TO CHECK_IN CODE

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-87-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 01:47:27 2009
Return-Path: <common-issues-return-87-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85435 invoked from network); 7 Jul 2009 01:47:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 01:47:27 -0000
Received: (qmail 49459 invoked by uid 500); 7 Jul 2009 01:47:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49377 invoked by uid 500); 7 Jul 2009 01:47:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49356 invoked by uid 99); 7 Jul 2009 01:47:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 01:47:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 01:47:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D89AB234C04C
	for <common-issues@hadoop.apache.org>; Mon,  6 Jul 2009 18:47:14 -0700 (PDT)
Message-ID: <1145550975.1246931234886.JavaMail.jira@brutus>
Date: Mon, 6 Jul 2009 18:47:14 -0700 (PDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Reopened: (HADOOP-5482)
 org.apache.hadoop.mapred.lib.db.DBInputformat not working with oracle
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron Kimball reopened HADOOP-5482:
-----------------------------------


This issue was marked "Resolved" but it was never applied to the 20 branch or trunk (for that matter, it never seems to have gone through Hudson).

I need Oracle support on 20/trunk, so I'm reopening this issue to revisit this patch.

> org.apache.hadoop.mapred.lib.db.DBInputformat not working with oracle
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-5482
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5482
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.18.2, 0.18.3, 0.19.0, 0.19.1
>         Environment: Java 1.6, HAdoop0.19.0, Linux..Oracle, 
>            Reporter: evanand
>            Priority: Blocker
>         Attachments: HADOOP-5482.20-branch.patch, HADOOP-5482.patch, HADOOP-5482.trunk.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> org.apache.hadoop.mapred.lib.db.DBInputformat not working with oracle.
> The out of the box implementation of the Hadoop is working properly with mysql/hsqldb, but NOT with oracle.
> Reason is DBInputformat is implemented with mysql/hsqldb specific query constructs like "LIMIT", "OFFSET".
> FIX:
> building a database provider specific logic based on the database providername (which we can get using connection).
> I HAVE ALREADY IMPLEMENTED IT FOR ORACLE...READY TO CHECK_IN CODE

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-88-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 01:49:27 2009
Return-Path: <common-issues-return-88-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87303 invoked from network); 7 Jul 2009 01:49:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 01:49:26 -0000
Received: (qmail 52257 invoked by uid 500); 7 Jul 2009 01:49:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52220 invoked by uid 500); 7 Jul 2009 01:49:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52210 invoked by uid 99); 7 Jul 2009 01:49:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 01:49:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 01:49:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D5233234C056
	for <common-issues@hadoop.apache.org>; Mon,  6 Jul 2009 18:49:14 -0700 (PDT)
Message-ID: <1758403885.1246931354872.JavaMail.jira@brutus>
Date: Mon, 6 Jul 2009 18:49:14 -0700 (PDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5482)
 org.apache.hadoop.mapred.lib.db.DBInputformat not working with oracle
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron Kimball updated HADOOP-5482:
----------------------------------

    Status: Patch Available  (was: Reopened)

> org.apache.hadoop.mapred.lib.db.DBInputformat not working with oracle
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-5482
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5482
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.19.1, 0.19.0, 0.18.3, 0.18.2
>         Environment: Java 1.6, HAdoop0.19.0, Linux..Oracle, 
>            Reporter: evanand
>            Priority: Blocker
>         Attachments: HADOOP-5482.20-branch.patch, HADOOP-5482.patch, HADOOP-5482.trunk.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> org.apache.hadoop.mapred.lib.db.DBInputformat not working with oracle.
> The out of the box implementation of the Hadoop is working properly with mysql/hsqldb, but NOT with oracle.
> Reason is DBInputformat is implemented with mysql/hsqldb specific query constructs like "LIMIT", "OFFSET".
> FIX:
> building a database provider specific logic based on the database providername (which we can get using connection).
> I HAVE ALREADY IMPLEMENTED IT FOR ORACLE...READY TO CHECK_IN CODE

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-89-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 01:49:29 2009
Return-Path: <common-issues-return-89-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87622 invoked from network); 7 Jul 2009 01:49:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 01:49:28 -0000
Received: (qmail 52361 invoked by uid 500); 7 Jul 2009 01:49:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52323 invoked by uid 500); 7 Jul 2009 01:49:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52312 invoked by uid 99); 7 Jul 2009 01:49:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 01:49:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 01:49:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CB331234C004
	for <common-issues@hadoop.apache.org>; Mon,  6 Jul 2009 18:49:14 -0700 (PDT)
Message-ID: <830728616.1246931354817.JavaMail.jira@brutus>
Date: Mon, 6 Jul 2009 18:49:14 -0700 (PDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5482)
 org.apache.hadoop.mapred.lib.db.DBInputformat not working with oracle
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727870#action_12727870 ] 

Aaron Kimball commented on HADOOP-5482:
---------------------------------------

Attaching patches for 20-branch and trunk based on the original patch posted here. This rebases the work onto the new API in trunk. Also fixes indentation problems in the initial version of the patch. (The new code was indented at the wrong level.)

Since this adds a db product-specific field, I also used this to set a flag necessary in MySQL, but that's a trivial difference.

I have tested this and verified that it works with Oracle and MySQL locally. No new tests because we don't have an Oracle driver that can be checked in to Hadoop.


> org.apache.hadoop.mapred.lib.db.DBInputformat not working with oracle
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-5482
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5482
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.18.2, 0.18.3, 0.19.0, 0.19.1
>         Environment: Java 1.6, HAdoop0.19.0, Linux..Oracle, 
>            Reporter: evanand
>            Priority: Blocker
>         Attachments: HADOOP-5482.20-branch.patch, HADOOP-5482.patch, HADOOP-5482.trunk.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> org.apache.hadoop.mapred.lib.db.DBInputformat not working with oracle.
> The out of the box implementation of the Hadoop is working properly with mysql/hsqldb, but NOT with oracle.
> Reason is DBInputformat is implemented with mysql/hsqldb specific query constructs like "LIMIT", "OFFSET".
> FIX:
> building a database provider specific logic based on the database providername (which we can get using connection).
> I HAVE ALREADY IMPLEMENTED IT FOR ORACLE...READY TO CHECK_IN CODE

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-90-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 01:51:27 2009
Return-Path: <common-issues-return-90-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88183 invoked from network); 7 Jul 2009 01:51:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 01:51:27 -0000
Received: (qmail 56259 invoked by uid 500); 7 Jul 2009 01:51:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56222 invoked by uid 500); 7 Jul 2009 01:51:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56212 invoked by uid 99); 7 Jul 2009 01:51:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 01:51:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 01:51:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CB87D234C004
	for <common-issues@hadoop.apache.org>; Mon,  6 Jul 2009 18:51:14 -0700 (PDT)
Message-ID: <968067235.1246931474818.JavaMail.jira@brutus>
Date: Mon, 6 Jul 2009 18:51:14 -0700 (PDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5482)
 org.apache.hadoop.mapred.lib.db.DBInputformat not working with oracle
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron Kimball updated HADOOP-5482:
----------------------------------

        Priority: Major  (was: Blocker)
    Hadoop Flags:   (was: [Reviewed])

Removing "Reviewed" flag since my code hasn't been reviewed. Also downgrading priority from "BLOCKER" to Major.

> org.apache.hadoop.mapred.lib.db.DBInputformat not working with oracle
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-5482
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5482
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.18.2, 0.18.3, 0.19.0, 0.19.1
>         Environment: Java 1.6, HAdoop0.19.0, Linux..Oracle, 
>            Reporter: evanand
>         Attachments: HADOOP-5482.20-branch.patch, HADOOP-5482.patch, HADOOP-5482.trunk.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> org.apache.hadoop.mapred.lib.db.DBInputformat not working with oracle.
> The out of the box implementation of the Hadoop is working properly with mysql/hsqldb, but NOT with oracle.
> Reason is DBInputformat is implemented with mysql/hsqldb specific query constructs like "LIMIT", "OFFSET".
> FIX:
> building a database provider specific logic based on the database providername (which we can get using connection).
> I HAVE ALREADY IMPLEMENTED IT FOR ORACLE...READY TO CHECK_IN CODE

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-91-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 02:56:28 2009
Return-Path: <common-issues-return-91-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12119 invoked from network); 7 Jul 2009 02:56:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 02:56:28 -0000
Received: (qmail 99717 invoked by uid 500); 7 Jul 2009 02:56:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99678 invoked by uid 500); 7 Jul 2009 02:56:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99668 invoked by uid 99); 7 Jul 2009 02:56:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 02:56:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 02:56:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C8B02234C004
	for <common-issues@hadoop.apache.org>; Mon,  6 Jul 2009 19:56:14 -0700 (PDT)
Message-ID: <903874219.1246935374817.JavaMail.jira@brutus>
Date: Mon, 6 Jul 2009 19:56:14 -0700 (PDT)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6099) Allow configuring the IPC module to
 send pings
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727888#action_12727888 ] 

Konstantin Shvachko commented on HADOOP-6099:
---------------------------------------------

I see. So you want to be able to set
ipc.client.ping = false - no ping, and at the same time
ipc.ping.interval = 20, which is now treated not as ping interva because there are no pingsl, but as a client timeout.

> Allow configuring the IPC module to send pings
> ----------------------------------------------
>
>                 Key: HADOOP-6099
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6099
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>         Attachments: ipcPing.txt
>
>
> The IPC Client sets a socketTimeout for the time period specified by the pingInterval and then sends a ping every pingInterval. This means that if a RPC server does not respond to a RPC client, then the RPC client blocks forever. This is a problem for applications that wants to switch quickly from one un-responsive HDFS cluster  to a good one. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-92-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 03:02:29 2009
Return-Path: <common-issues-return-92-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15203 invoked from network); 7 Jul 2009 03:02:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 03:02:28 -0000
Received: (qmail 6196 invoked by uid 500); 7 Jul 2009 03:02:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6161 invoked by uid 500); 7 Jul 2009 03:02:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6151 invoked by uid 99); 7 Jul 2009 03:02:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 03:02:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 03:02:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C9C08234C004
	for <common-issues@hadoop.apache.org>; Mon,  6 Jul 2009 20:02:14 -0700 (PDT)
Message-ID: <848373541.1246935734812.JavaMail.jira@brutus>
Date: Mon, 6 Jul 2009 20:02:14 -0700 (PDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6099) Allow configuring the IPC module to
 send pings
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727891#action_12727891 ] 

dhruba borthakur commented on HADOOP-6099:
------------------------------------------

@Konstantin: your assumption is right. is this ok with you?

> Allow configuring the IPC module to send pings
> ----------------------------------------------
>
>                 Key: HADOOP-6099
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6099
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>         Attachments: ipcPing.txt
>
>
> The IPC Client sets a socketTimeout for the time period specified by the pingInterval and then sends a ping every pingInterval. This means that if a RPC server does not respond to a RPC client, then the RPC client blocks forever. This is a problem for applications that wants to switch quickly from one un-responsive HDFS cluster  to a good one. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-93-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 05:00:27 2009
Return-Path: <common-issues-return-93-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50671 invoked from network); 7 Jul 2009 05:00:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 05:00:27 -0000
Received: (qmail 69536 invoked by uid 500); 7 Jul 2009 05:00:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69494 invoked by uid 500); 7 Jul 2009 05:00:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69479 invoked by uid 99); 7 Jul 2009 05:00:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 05:00:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 05:00:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DFCC7234C056
	for <common-issues@hadoop.apache.org>; Mon,  6 Jul 2009 22:00:14 -0700 (PDT)
Message-ID: <809176167.1246942814915.JavaMail.jira@brutus>
Date: Mon, 6 Jul 2009 22:00:14 -0700 (PDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4491) Per-job local data on the TaskTracker
 node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V updated HADOOP-4491:
------------------------------

    Attachment: HADOOP-4491-20090707.txt
                HADOOP-4491-20090707-common.txt

Adding new patches:
 - Cleaned up the common part of the code a little. After going through the patch, realized that the changes to LocalDirAllocator can be done away with and so reverted most of the changes. Now we only need a way to set permissions and so retained setPermissions and related api but moved them to FileUtil.
 - Found a bug related to task localization. With the attached patch, we create attempt directories also on all the disks. This has to be done because job directory is writable only by the TT and so the child cannot create non-existing attempt dirs on certain disks on demand.

Freezing up any further changes till review comments are put up.

> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: mapred, security
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, HADOOP-4491-20090707.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-94-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 06:12:29 2009
Return-Path: <common-issues-return-94-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86913 invoked from network); 7 Jul 2009 06:12:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 06:12:29 -0000
Received: (qmail 18313 invoked by uid 500); 7 Jul 2009 06:12:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18281 invoked by uid 500); 7 Jul 2009 06:12:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18248 invoked by uid 99); 7 Jul 2009 06:12:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 06:12:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 06:12:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CDED3234C004
	for <common-issues@hadoop.apache.org>; Mon,  6 Jul 2009 23:12:14 -0700 (PDT)
Message-ID: <1676321853.1246947134828.JavaMail.jira@brutus>
Date: Mon, 6 Jul 2009 23:12:14 -0700 (PDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4493) Localized files from
 DistributedCache should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727939#action_12727939 ] 

Vinod K V commented on HADOOP-4493:
-----------------------------------

*This issue's focus*

Given the state of the art,  what do we mean by restrictive permissions for files in distributed cache?

*Case I: The cache files are completely private to the job-owner*
 - The job owner wants his/her files only for himself/herself and doesn't want access to anyone else.
 - This means 700 permissions and owned by the job owner, so only the job owner can access it.
 - To facilitate the above, the cache should be localized under a directory with user name.
 - The files should be usable by subsequent jobs of the same user.
 - The configurable size limit of the cache or in other words the disk quota for the cache files should be per user.
 - Because the files are owned by the user, we will need a task-controller process launch during cleaning-up/purging.

*Case II: The job owner is fine with sharing his/her distributed cache files with other users*
 - A possible use case for this, as mentioned on HADOOP-4490, is a multiple of users, perhaps working on the same project that requires the same data files, want to share these files across multiple jobs of possibly other users.
 - The above would mean _55 permissions for everyone.
 - The files should be localized under a common directory not associated to any user name.
 - There should be a configuration knob(per cache URI?) to be specified by the user so as to distinguish these files from the ones that belong to CASE I.
 - The disk quota for the cache files should be a global one and not specific to the user.
 -  The files can be owned by the job owner or the TT itself.
    -- Having TT own the files makes it easy for cleaning up.
    -- If the ownership is with the job-owner, we would need a task-controller process launch. We we also need user information with the files, perhaps through a sub-directory associated with a user name.

So, do we want only CASE I or only CASE II or a combination of both? Thoughts/comments?

> Localized files from DistributedCache should have right access-control
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-4493
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4493
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: mapred, security
>            Reporter: Arun C Murthy
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-95-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 06:44:31 2009
Return-Path: <common-issues-return-95-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6630 invoked from network); 7 Jul 2009 06:44:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 06:44:28 -0000
Received: (qmail 34239 invoked by uid 500); 7 Jul 2009 06:44:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34197 invoked by uid 500); 7 Jul 2009 06:44:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34187 invoked by uid 99); 7 Jul 2009 06:44:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 06:44:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 06:44:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DE44A234C051
	for <common-issues@hadoop.apache.org>; Mon,  6 Jul 2009 23:44:14 -0700 (PDT)
Message-ID: <1238693994.1246949054909.JavaMail.jira@brutus>
Date: Mon, 6 Jul 2009 23:44:14 -0700 (PDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4493) Localized files from
 DistributedCache should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727944#action_12727944 ] 

Vinod K V commented on HADOOP-4493:
-----------------------------------

One more (hybrid) case is possible.

*CASE III: When job owner wants to share with a specific list of users that he mentions.*
 - A minor variant of CASE II.
 - Job owner either specifies the list of users or a group of users with which he wants to share libraries.
 - With most of the other things being the same as in CASE II, the api in DistributedCache should make sure that the access is restricted only to the list of users or the group specified.
 - Even here, the access list should be per cache uri.

> Localized files from DistributedCache should have right access-control
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-4493
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4493
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: mapred, security
>            Reporter: Arun C Murthy
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-96-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 12:26:28 2009
Return-Path: <common-issues-return-96-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61515 invoked from network); 7 Jul 2009 12:26:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 12:26:28 -0000
Received: (qmail 89418 invoked by uid 500); 7 Jul 2009 12:26:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89390 invoked by uid 500); 7 Jul 2009 12:26:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89380 invoked by uid 99); 7 Jul 2009 12:26:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 12:26:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 12:26:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D1BB7234C04C
	for <common-issues@hadoop.apache.org>; Tue,  7 Jul 2009 05:26:14 -0700 (PDT)
Message-ID: <1331823974.1246969574858.JavaMail.jira@brutus>
Date: Tue, 7 Jul 2009 05:26:14 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6128) Serializer and Deserializer should
 extend java.io.Closeable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Serializer and Deserializer should extend java.io.Closeable
-----------------------------------------------------------

                 Key: HADOOP-6128
                 URL: https://issues.apache.org/jira/browse/HADOOP-6128
             Project: Hadoop Common
          Issue Type: Improvement
          Components: io
            Reporter: Tom White


This change wouldn't change behaviour or the API, but would make it possible to use such utilities as IOUtils#closeStream() on Serializers and Deserializers.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-97-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 17:26:29 2009
Return-Path: <common-issues-return-97-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41198 invoked from network); 7 Jul 2009 17:26:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 17:26:29 -0000
Received: (qmail 87398 invoked by uid 500); 7 Jul 2009 17:26:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87360 invoked by uid 500); 7 Jul 2009 17:26:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87350 invoked by uid 99); 7 Jul 2009 17:26:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 17:26:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 17:26:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CA4AE234C004
	for <common-issues@hadoop.apache.org>; Tue,  7 Jul 2009 10:26:14 -0700 (PDT)
Message-ID: <1435702500.1246987574814.JavaMail.jira@brutus>
Date: Tue, 7 Jul 2009 10:26:14 -0700 (PDT)
From: "gary murry (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5587) Feature Designs and Test Plans
 templates in HTML
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

gary murry updated HADOOP-5587:
-------------------------------

    Attachment: TestPlanTemplate.html

The new version includes more examples.

> Feature Designs and Test Plans templates in HTML
> ------------------------------------------------
>
>                 Key: HADOOP-5587
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5587
>             Project: Hadoop Common
>          Issue Type: Wish
>            Reporter: gary murry
>         Attachments: DesignDocTemplate.html, DesignDocTemplate.html, TestPlanTemplate.html, TestPlanTemplate.html, TestPlanTemplate.html, TestPlanTemplate.html
>
>
> This Jira is to address an existing email thread which requested a Feature Design template and Test Plan template in HTML format.  
> Past Design Doc Examples
> http://issues.apache.org/jira/secure/attachment/12348296/DFSUpgradeProposal3.html
> Past Test Plan Examples
> https://issues.apache.org/jira/secure/attachment/12373559/PermissionsTestPlan2.pdf
> https://issues.apache.org/jira/secure/attachment/12363605/BlockCrcFeatureTestPlan.pdf
> https://issues.apache.org/jira/secure/attachment/12351299/TestPlan-HdfsUpgrade.html
>  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-98-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 17:37:11 2009
Return-Path: <common-issues-return-98-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17608 invoked from network); 7 Jul 2009 17:37:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 17:37:11 -0000
Received: (qmail 875 invoked by uid 500); 7 Jul 2009 17:37:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 831 invoked by uid 500); 7 Jul 2009 17:37:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 813 invoked by uid 99); 7 Jul 2009 17:37:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 17:37:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 17:37:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C24E9234C4DF
	for <common-issues@hadoop.apache.org>; Tue,  7 Jul 2009 10:36:16 -0700 (PDT)
Message-ID: <562696707.1246988176795.JavaMail.jira@brutus>
Date: Tue, 7 Jul 2009 10:36:16 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-516) Eclipse-based GUI: DFS explorer and
 basic Map/Reduce job launcher
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-516?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1272=
8223#action_12728223 ]=20

Hudson commented on HADOOP-516:
-------------------------------

Integrated in Hadoop-Mapreduce-trunk #15 (See [http://hudson.zones.apache.o=
rg/hudson/job/Hadoop-Mapreduce-trunk/15/])
   =20

> Eclipse-based GUI: DFS explorer and basic Map/Reduce job launcher
> -----------------------------------------------------------------
>
>                 Key: HADOOP-516
>                 URL: https://issues.apache.org/jira/browse/HADOOP-516
>             Project: Hadoop Common
>          Issue Type: New Feature
>         Environment: Eclipse 3.2
> JDK 1.5
>            Reporter: Fr=C3=A9d=C3=A9ric Bertin
>         Attachments: hdfsExplorer.zip, hdfsExplorer2.zip
>
>
> to increase productivity in our current project (which makes a heavy use =
of Hadoop), we wrote a small Eclipse-based GUI application which basically =
consists in 2 views:
>    *  a HDFS explorer adapted from Eclipse filesystem explorer example.
>      For now, it includes the following features:
>          o classical tree-based browsing interface, with directory
>            content being detailed in a 3 columns table (file name, file
>            size, file type)
>          o refresh button
>          o delete file or directory (with confirm dialog): select files
>            in the tree or table and click the "Delete" button
>          o rename file or directory: simple click on the file in the
>            table, type the new name and validate
>          o open file with system editor: select the file in the table
>            and click "Open" button (works on Windows, not on Linux)
>          o internal drag & drop
>          o external drag & drop from the local filesystem to the HDFS
>            (the opposite doesn't work)
>    * a MapReduce *very* simple job launcher:
>          o select the job XML configuration file
>          o run the job
>          o kill the job
>          o visualize map and reduce progress with progress bars
>          o open a browser on the Hadoop job tracker web interface=20
> INSTALLATION NOTES:
>  - Eclipse 3.2
>  - JDK 1.5
>  - import the archive in Eclipse
>  - copy your hadoop conf file (hadoop-default.xml in "src" folder) -> thi=
s step should be moved in the GUI later
>  - right-click on the project and Run As -> Eclipse Application
>  - enjoy...

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-99-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 17:37:12 2009
Return-Path: <common-issues-return-99-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17831 invoked from network); 7 Jul 2009 17:37:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 17:37:12 -0000
Received: (qmail 947 invoked by uid 500); 7 Jul 2009 17:37:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 908 invoked by uid 500); 7 Jul 2009 17:37:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 898 invoked by uid 99); 7 Jul 2009 17:37:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 17:37:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 17:37:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D5720234C4AC
	for <common-issues@hadoop.apache.org>; Tue,  7 Jul 2009 10:36:16 -0700 (PDT)
Message-ID: <267415001.1246988176873.JavaMail.jira@brutus>
Date: Tue, 7 Jul 2009 10:36:16 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6112) to fix hudsonPatchQueueAdmin for
 different projects
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728225#action_12728225 ] 

Hudson commented on HADOOP-6112:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk #15 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/15/])
    

> to fix hudsonPatchQueueAdmin for different projects
> ---------------------------------------------------
>
>                 Key: HADOOP-6112
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6112
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6112.patch
>
>
> To fix hudsonPatchQueueAdmin process for different hadoop projects.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-100-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 17:37:19 2009
Return-Path: <common-issues-return-100-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18966 invoked from network); 7 Jul 2009 17:37:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 17:37:14 -0000
Received: (qmail 1639 invoked by uid 500); 7 Jul 2009 17:37:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1608 invoked by uid 500); 7 Jul 2009 17:37:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1598 invoked by uid 99); 7 Jul 2009 17:37:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 17:37:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 17:37:22 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1EBBF234C1EE
	for <common-issues@hadoop.apache.org>; Tue,  7 Jul 2009 10:36:18 -0700 (PDT)
Message-ID: <1566034566.1246988178124.JavaMail.jira@brutus>
Date: Tue, 7 Jul 2009 10:36:18 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6122) 64 javac compiler warnings
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728237#action_12728237 ] 

Hudson commented on HADOOP-6122:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk #15 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/15/])
    

> 64 javac compiler warnings
> --------------------------
>
>                 Key: HADOOP-6122
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6122
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.21.0
>
>         Attachments: c6122_20090630.patch
>
>
> Ran "ant test-patch" with a empty patch.  Got 
> {noformat}
>      [exec]     -1 javac.  The applied patch generated 64 javac compiler warnings (more than the trunk's current 124 warnings).
>  {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-101-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 17:37:35 2009
Return-Path: <common-issues-return-101-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26300 invoked from network); 7 Jul 2009 17:37:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 17:37:35 -0000
Received: (qmail 2464 invoked by uid 500); 7 Jul 2009 17:37:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2427 invoked by uid 500); 7 Jul 2009 17:37:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2417 invoked by uid 99); 7 Jul 2009 17:37:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 17:37:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 17:37:42 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id ACCB5234C4C3
	for <common-issues@hadoop.apache.org>; Tue,  7 Jul 2009 10:36:18 -0700 (PDT)
Message-ID: <1318175713.1246988178707.JavaMail.jira@brutus>
Date: Tue, 7 Jul 2009 10:36:18 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6106) Provide an option in
 ShellCommandExecutor to timeout commands that do not complete within a
 certain amount of time.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728246#action_12728246 ] 

Hudson commented on HADOOP-6106:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk #15 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/15/])
    

> Provide an option in ShellCommandExecutor to timeout commands that do not complete within a certain amount of time.
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Hemanth Yamijala
>            Assignee: Sreekanth Ramakrishnan
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6106-1.patch, HADOOP-6106-2.patch, HADOOP-6106.patch, mapred-211-common-3.patch
>
>
> In MAPREDUCE-211 we came across a need to provide an option to timeout commands launched via the ShellCommandExecutor. The use case is for the health check script being developed in MAPREDUCE-211. We would like the TaskTracker thread to not be blocked by a problematic script or in instances where fork()+exec() has hung (which apparently has been observed in large clusters).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-102-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 18:20:29 2009
Return-Path: <common-issues-return-102-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83257 invoked from network); 7 Jul 2009 18:20:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 18:20:29 -0000
Received: (qmail 77752 invoked by uid 500); 7 Jul 2009 18:20:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77662 invoked by uid 500); 7 Jul 2009 18:20:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77626 invoked by uid 99); 7 Jul 2009 18:20:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 18:20:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 18:20:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0F652234C1F0
	for <common-issues@hadoop.apache.org>; Tue,  7 Jul 2009 11:20:15 -0700 (PDT)
Message-ID: <1896745446.1246990815062.JavaMail.jira@brutus>
Date: Tue, 7 Jul 2009 11:20:15 -0700 (PDT)
From: "gary murry (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5976) create script to provide classpath
 for external tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728275#action_12728275 ] 

gary murry commented on HADOOP-5976:
------------------------------------

Also, the -1 javac is related to HADOOP-6122.

> create script to provide classpath for external tools
> -----------------------------------------------------
>
>                 Key: HADOOP-5976
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5976
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: h5976.patch, Hadoop-5976.patch
>
>
> It would be useful for tools building on top of Hadoop to have a script that returns the class path that is needed to get all of the Hadoop jars and the needed libraries.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-103-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 18:42:27 2009
Return-Path: <common-issues-return-103-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80261 invoked from network); 7 Jul 2009 18:42:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 18:42:27 -0000
Received: (qmail 16239 invoked by uid 500); 7 Jul 2009 18:42:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16202 invoked by uid 500); 7 Jul 2009 18:42:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16192 invoked by uid 99); 7 Jul 2009 18:42:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 18:42:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 18:42:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E0541234C1EB
	for <common-issues@hadoop.apache.org>; Tue,  7 Jul 2009 11:42:14 -0700 (PDT)
Message-ID: <622653073.1246992134918.JavaMail.jira@brutus>
Date: Tue, 7 Jul 2009 11:42:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5976) create script to provide classpath
 for external tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-5976:
-------------------------------------------

    Hadoop Flags: [Reviewed]

+1 patch looks good.

> create script to provide classpath for external tools
> -----------------------------------------------------
>
>                 Key: HADOOP-5976
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5976
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: h5976.patch, Hadoop-5976.patch
>
>
> It would be useful for tools building on top of Hadoop to have a script that returns the class path that is needed to get all of the Hadoop jars and the needed libraries.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-104-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 19:08:05 2009
Return-Path: <common-issues-return-104-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9465 invoked from network); 7 Jul 2009 19:08:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 19:08:03 -0000
Received: (qmail 27755 invoked by uid 500); 7 Jul 2009 18:52:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27739 invoked by uid 500); 7 Jul 2009 18:52:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27728 invoked by uid 99); 7 Jul 2009 18:52:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 18:52:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 18:52:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D526B234C051
	for <common-issues@hadoop.apache.org>; Tue,  7 Jul 2009 11:52:14 -0700 (PDT)
Message-ID: <1311128733.1246992734862.JavaMail.jira@brutus>
Date: Tue, 7 Jul 2009 11:52:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-5976) create script to provide classpath
 for external tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE resolved HADOOP-5976.
--------------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.21.0
     Release Note: Add a new hadoop script command, classpath, to print the class path needed to get the Hadoop jar and the required libraries.

I have committed this.  Thanks, Owen and Gary!

> create script to provide classpath for external tools
> -----------------------------------------------------
>
>                 Key: HADOOP-5976
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5976
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.21.0
>
>         Attachments: h5976.patch, Hadoop-5976.patch
>
>
> It would be useful for tools building on top of Hadoop to have a script that returns the class path that is needed to get all of the Hadoop jars and the needed libraries.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-105-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 19:22:29 2009
Return-Path: <common-issues-return-105-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79588 invoked from network); 7 Jul 2009 19:22:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 19:22:29 -0000
Received: (qmail 79145 invoked by uid 500); 7 Jul 2009 19:22:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79111 invoked by uid 500); 7 Jul 2009 19:22:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79101 invoked by uid 99); 7 Jul 2009 19:22:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 19:22:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 19:22:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0AA47234C004
	for <common-issues@hadoop.apache.org>; Tue,  7 Jul 2009 12:22:15 -0700 (PDT)
Message-ID: <491348910.1246994535031.JavaMail.jira@brutus>
Date: Tue, 7 Jul 2009 12:22:15 -0700 (PDT)
From: "gary murry (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5976) create script to provide classpath
 for external tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728306#action_12728306 ] 

gary murry commented on HADOOP-5976:
------------------------------------

Great, thanks!
-Gary



> create script to provide classpath for external tools
> -----------------------------------------------------
>
>                 Key: HADOOP-5976
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5976
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.21.0
>
>         Attachments: h5976.patch, Hadoop-5976.patch
>
>
> It would be useful for tools building on top of Hadoop to have a script that returns the class path that is needed to get all of the Hadoop jars and the needed libraries.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-106-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 19:36:28 2009
Return-Path: <common-issues-return-106-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14408 invoked from network); 7 Jul 2009 19:36:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 19:36:28 -0000
Received: (qmail 11808 invoked by uid 500); 7 Jul 2009 19:36:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11769 invoked by uid 500); 7 Jul 2009 19:36:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11759 invoked by uid 99); 7 Jul 2009 19:36:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 19:36:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 19:36:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D731E234C1E9
	for <common-issues@hadoop.apache.org>; Tue,  7 Jul 2009 12:36:14 -0700 (PDT)
Message-ID: <1542829149.1246995374880.JavaMail.jira@brutus>
Date: Tue, 7 Jul 2009 12:36:14 -0700 (PDT)
From: "Justin Patterson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6129) MapFile doesn't worh with
 serializables other than Writables
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

MapFile doesn't worh with serializables other than Writables
------------------------------------------------------------

                 Key: HADOOP-6129
                 URL: https://issues.apache.org/jira/browse/HADOOP-6129
             Project: Hadoop Common
          Issue Type: Improvement
          Components: io, mapred
    Affects Versions: 0.20.0
            Reporter: Justin Patterson


Since 0.18 (I think), SequenceFiles have supported serializing arbitrary objects through the serialization framework.  MapFiles still don't.  They require WritableComparable keys and Writable values.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-107-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 19:38:30 2009
Return-Path: <common-issues-return-107-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15504 invoked from network); 7 Jul 2009 19:38:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 19:38:29 -0000
Received: (qmail 17740 invoked by uid 500); 7 Jul 2009 19:38:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17696 invoked by uid 500); 7 Jul 2009 19:38:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17605 invoked by uid 99); 7 Jul 2009 19:38:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 19:38:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 19:38:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 61218234C004
	for <common-issues@hadoop.apache.org>; Tue,  7 Jul 2009 12:38:15 -0700 (PDT)
Message-ID: <1540923342.1246995495386.JavaMail.jira@brutus>
Date: Tue, 7 Jul 2009 12:38:15 -0700 (PDT)
From: "Justin Patterson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6129) MapFile doesn't work with
 serializables other than Writables
In-Reply-To: <1542829149.1246995374880.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Justin Patterson updated HADOOP-6129:
-------------------------------------

    Summary: MapFile doesn't work with serializables other than Writables  (was: MapFile doesn't worh with serializables other than Writables)

> MapFile doesn't work with serializables other than Writables
> ------------------------------------------------------------
>
>                 Key: HADOOP-6129
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6129
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, mapred
>    Affects Versions: 0.20.0
>            Reporter: Justin Patterson
>
> Since 0.18 (I think), SequenceFiles have supported serializing arbitrary objects through the serialization framework.  MapFiles still don't.  They require WritableComparable keys and Writable values.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-108-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 20:08:27 2009
Return-Path: <common-issues-return-108-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49751 invoked from network); 7 Jul 2009 20:08:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 20:08:27 -0000
Received: (qmail 74293 invoked by uid 500); 7 Jul 2009 20:08:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74258 invoked by uid 500); 7 Jul 2009 20:08:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74199 invoked by uid 99); 7 Jul 2009 20:08:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 20:08:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 20:08:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CCA82234C004
	for <common-issues@hadoop.apache.org>; Tue,  7 Jul 2009 13:08:14 -0700 (PDT)
Message-ID: <1330948080.1246997294837.JavaMail.jira@brutus>
Date: Tue, 7 Jul 2009 13:08:14 -0700 (PDT)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6099) Allow configuring the IPC module to
 send pings
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728332#action_12728332 ] 

Konstantin Shvachko commented on HADOOP-6099:
---------------------------------------------

Yes I think this is fine.
+1

> Allow configuring the IPC module to send pings
> ----------------------------------------------
>
>                 Key: HADOOP-6099
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6099
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>         Attachments: ipcPing.txt
>
>
> The IPC Client sets a socketTimeout for the time period specified by the pingInterval and then sends a ping every pingInterval. This means that if a RPC server does not respond to a RPC client, then the RPC client blocks forever. This is a problem for applications that wants to switch quickly from one un-responsive HDFS cluster  to a good one. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-109-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 20:10:26 2009
Return-Path: <common-issues-return-109-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55712 invoked from network); 7 Jul 2009 20:10:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 20:10:26 -0000
Received: (qmail 78372 invoked by uid 500); 7 Jul 2009 20:10:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78333 invoked by uid 500); 7 Jul 2009 20:10:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78323 invoked by uid 99); 7 Jul 2009 20:10:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 20:10:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 20:10:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E3BA0234C1F0
	for <common-issues@hadoop.apache.org>; Tue,  7 Jul 2009 13:10:14 -0700 (PDT)
Message-ID: <296905026.1246997414932.JavaMail.jira@brutus>
Date: Tue, 7 Jul 2009 13:10:14 -0700 (PDT)
From: "Justin Patterson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6129) MapFile doesn't work with
 serializables other than Writables
In-Reply-To: <1542829149.1246995374880.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Justin Patterson updated HADOOP-6129:
-------------------------------------

    Attachment: HADOOP-6129.patch

Patch attached.

> MapFile doesn't work with serializables other than Writables
> ------------------------------------------------------------
>
>                 Key: HADOOP-6129
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6129
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, mapred
>    Affects Versions: 0.20.0
>            Reporter: Justin Patterson
>         Attachments: HADOOP-6129.patch
>
>
> Since 0.18 (I think), SequenceFiles have supported serializing arbitrary objects through the serialization framework.  MapFiles still don't.  They require WritableComparable keys and Writable values.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-110-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 20:10:29 2009
Return-Path: <common-issues-return-110-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55771 invoked from network); 7 Jul 2009 20:10:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 20:10:29 -0000
Received: (qmail 78509 invoked by uid 500); 7 Jul 2009 20:10:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78474 invoked by uid 500); 7 Jul 2009 20:10:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78454 invoked by uid 99); 7 Jul 2009 20:10:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 20:10:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 20:10:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C9EFF234C004
	for <common-issues@hadoop.apache.org>; Tue,  7 Jul 2009 13:10:14 -0700 (PDT)
Message-ID: <1374119191.1246997414822.JavaMail.jira@brutus>
Date: Tue, 7 Jul 2009 13:10:14 -0700 (PDT)
From: "Justin Patterson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6129) MapFile doesn't work with
 serializables other than Writables
In-Reply-To: <1542829149.1246995374880.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Justin Patterson updated HADOOP-6129:
-------------------------------------

    Status: Patch Available  (was: Open)

Here's the patch that I used to get this working in my own app.

It basically uses the SerializationFactory for key/value serialization.  The key comparator is either specified by the Writer creator or, for legacy reasons, is created through WritableComparator.get().

The MapFileOutputFormat creates the key comparator using job.getMapOutputKeyComparator().

> MapFile doesn't work with serializables other than Writables
> ------------------------------------------------------------
>
>                 Key: HADOOP-6129
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6129
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, mapred
>    Affects Versions: 0.20.0
>            Reporter: Justin Patterson
>         Attachments: HADOOP-6129.patch
>
>
> Since 0.18 (I think), SequenceFiles have supported serializing arbitrary objects through the serialization framework.  MapFiles still don't.  They require WritableComparable keys and Writable values.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-111-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 20:38:27 2009
Return-Path: <common-issues-return-111-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15066 invoked from network); 7 Jul 2009 20:38:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 20:38:27 -0000
Received: (qmail 18272 invoked by uid 500); 7 Jul 2009 20:38:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18232 invoked by uid 500); 7 Jul 2009 20:38:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18222 invoked by uid 99); 7 Jul 2009 20:38:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 20:38:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 20:38:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DFDD2234C004
	for <common-issues@hadoop.apache.org>; Tue,  7 Jul 2009 13:38:14 -0700 (PDT)
Message-ID: <1422588907.1246999094912.JavaMail.jira@brutus>
Date: Tue, 7 Jul 2009 13:38:14 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5170) Set max map/reduce tasks on a per-job
 basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-5170:
----------------------------------

    Attachment: h5170.patch

This is the patch to rollback the change.

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: h5170.patch, HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-112-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 20:46:32 2009
Return-Path: <common-issues-return-112-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18963 invoked from network); 7 Jul 2009 20:46:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 20:46:30 -0000
Received: (qmail 32122 invoked by uid 500); 7 Jul 2009 20:46:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32078 invoked by uid 500); 7 Jul 2009 20:46:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31948 invoked by uid 99); 7 Jul 2009 20:46:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 20:46:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 20:46:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E5F6B234C1E6
	for <common-issues@hadoop.apache.org>; Tue,  7 Jul 2009 13:46:14 -0700 (PDT)
Message-ID: <1600385821.1246999574941.JavaMail.jira@brutus>
Date: Tue, 7 Jul 2009 13:46:14 -0700 (PDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728358#action_12728358 ] 

Arun C Murthy commented on HADOOP-5170:
---------------------------------------

+1

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: h5170.patch, HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-113-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 21:42:29 2009
Return-Path: <common-issues-return-113-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7058 invoked from network); 7 Jul 2009 21:42:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 21:42:28 -0000
Received: (qmail 99919 invoked by uid 500); 7 Jul 2009 21:42:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99881 invoked by uid 500); 7 Jul 2009 21:42:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99871 invoked by uid 99); 7 Jul 2009 21:42:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 21:42:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 21:42:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D4092234C1E6
	for <common-issues@hadoop.apache.org>; Tue,  7 Jul 2009 14:42:14 -0700 (PDT)
Message-ID: <1683137722.1247002934867.JavaMail.jira@brutus>
Date: Tue, 7 Jul 2009 14:42:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6129) MapFile doesn't work with
 serializables other than Writables
In-Reply-To: <1542829149.1246995374880.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728374#action_12728374 ] 

Hadoop QA commented on HADOOP-6129:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12412780/HADOOP-6129.patch
  against trunk revision 791937.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 patch.  The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/556/console

This message is automatically generated.

> MapFile doesn't work with serializables other than Writables
> ------------------------------------------------------------
>
>                 Key: HADOOP-6129
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6129
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, mapred
>    Affects Versions: 0.20.0
>            Reporter: Justin Patterson
>         Attachments: HADOOP-6129.patch
>
>
> Since 0.18 (I think), SequenceFiles have supported serializing arbitrary objects through the serialization framework.  MapFiles still don't.  They require WritableComparable keys and Writable values.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-114-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 22:41:29 2009
Return-Path: <common-issues-return-114-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34107 invoked from network); 7 Jul 2009 22:41:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 22:41:29 -0000
Received: (qmail 58771 invoked by uid 500); 7 Jul 2009 22:41:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58734 invoked by uid 500); 7 Jul 2009 22:41:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58724 invoked by uid 99); 7 Jul 2009 22:41:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 22:41:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 22:41:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C6B87234C004
	for <common-issues@hadoop.apache.org>; Tue,  7 Jul 2009 15:41:14 -0700 (PDT)
Message-ID: <327888318.1247006474801.JavaMail.jira@brutus>
Date: Tue, 7 Jul 2009 15:41:14 -0700 (PDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6123) hdfs script does not work after
 project split.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728399#action_12728399 ] 

Suresh Srinivas commented on HADOOP-6123:
-----------------------------------------

I tried the change. It works.

I noticed that unlike before I have to setup JAVA_HOME in hadoop-env.sh. This is not picked up from JAVA_HOME environment variable from my shell.

> hdfs script does not work after project split.
> ----------------------------------------------
>
>                 Key: HADOOP-6123
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6123
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>             Fix For: 0.21.0
>
>         Attachments: 6123_v1.patch
>
>
> There are problems running hdfs script from common.
> # Usage message for hdfs does not have "fs" option anymore.
> # There seem to be an undocumented option "dfs", but using it throws NoClassDefFoundError or ClassNotFoundException.
> # Same with other options e.g. name-node.
> May I am missing something here. How do we call ls these days?
> Do we need to move hdfs script to hdfs project.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-115-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 07 23:05:41 2009
Return-Path: <common-issues-return-115-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62909 invoked from network); 7 Jul 2009 23:05:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Jul 2009 23:05:30 -0000
Received: (qmail 82093 invoked by uid 500); 7 Jul 2009 23:05:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82020 invoked by uid 500); 7 Jul 2009 23:05:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81728 invoked by uid 99); 7 Jul 2009 23:05:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 23:05:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jul 2009 23:05:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0D8F0234C4A9
	for <common-issues@hadoop.apache.org>; Tue,  7 Jul 2009 16:05:15 -0700 (PDT)
Message-ID: <281553019.1247007915054.JavaMail.jira@brutus>
Date: Tue, 7 Jul 2009 16:05:15 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6123) hdfs script does not work after
 project split.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728409#action_12728409 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6123:
------------------------------------------------

{code}
+if [ -d "$HADOOP_CORE_HOME/build/ivy/lib/Hadoop-Core/common" ]; then
+for f in $HADOOP_CORE_HOME/build/ivy/lib/Hadoop-Core/common/*.jar; do
+  CLASSPATH=${CLASSPATH}:$f;
+done
+fi
{code}

The for-loop above can be eliminated by keeping the wild card (shown below) as what we does in the bin/hadoop script.
{code}
if [ -d "$HADOOP_CORE_HOME/build/ivy/lib/Hadoop-Core/common" ]; then
  CLASSPATH=${CLASSPATH}:$HADOOP_CORE_HOME/build/ivy/lib/Hadoop-Core/common/*.jar;
fi
{code}

Try also the new "bin/hadoop classpath" command committed (HADOOP-5976) recently.

> hdfs script does not work after project split.
> ----------------------------------------------
>
>                 Key: HADOOP-6123
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6123
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>             Fix For: 0.21.0
>
>         Attachments: 6123_v1.patch
>
>
> There are problems running hdfs script from common.
> # Usage message for hdfs does not have "fs" option anymore.
> # There seem to be an undocumented option "dfs", but using it throws NoClassDefFoundError or ClassNotFoundException.
> # Same with other options e.g. name-node.
> May I am missing something here. How do we call ls these days?
> Do we need to move hdfs script to hdfs project.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-116-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 00:35:29 2009
Return-Path: <common-issues-return-116-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58549 invoked from network); 8 Jul 2009 00:35:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 00:35:27 -0000
Received: (qmail 16962 invoked by uid 500); 8 Jul 2009 00:35:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16918 invoked by uid 500); 8 Jul 2009 00:35:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16908 invoked by uid 99); 8 Jul 2009 00:35:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 00:35:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 00:35:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E5663234C1E9
	for <common-issues@hadoop.apache.org>; Tue,  7 Jul 2009 17:35:14 -0700 (PDT)
Message-ID: <536621156.1247013314938.JavaMail.jira@brutus>
Date: Tue, 7 Jul 2009 17:35:14 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4901) Upgrade to JUnit 4
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728443#action_12728443 ] 

Konstantin Boudnik commented on HADOOP-4901:
--------------------------------------------

Here's a comment on JUnitExt remarks. It seems that JUnit Extension project supports the categorization of some kind. However, it seems that the semantic of their @Category is very different from our desire for tagging is. Here's the excerpt from JUnitExt tutorial:

{panel:title=JUnitExt tutorial's excerpt}
How to use categories

Tests can be categorized, to sort them for different puporses. JUnitExt provides a CategoryTextListerner, which prints out at end of test run all categories, with status of tests listed: May be Success, Ignored, Failed. You can plugin the category text listener using following code: (see also org.junitext.samples package)

	JUnitCore core = new JUnitCore();
	// use for categories special listener, give some statistics
	core.addListener(new CategoryTextListener(System.out));
	core.run(SimpleTest.class);

The output when running thes tests will be:

I.
Time: 0

OK (1 test)

Category: equal tests
  Success testEquals(org.junitext.samples.SimpleTest)
Category: math tests
  Ignored divideByZero(org.junitext.samples.SimpleTest)
{panel}

It might be possible to extend or develop new ClassRunner which will be capable of running only tests from a specified @Category, however it isn't provided by JUnitExt out of the box or the documentation fails no mention it.


> Upgrade to JUnit 4
> ------------------
>
>                 Key: HADOOP-4901
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4901
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>            Reporter: Tom White
>            Assignee: Alex Loddengaard
>
> Amongst other things, JUnit 4 has better support for class-wide set up and tear down (via @BeforeClass and @AfterClass annotations), and more flexible assertions (http://junit.sourceforge.net/doc/ReleaseNotes4.4.html). It would be nice to be able to take advantage of these features in tests we write.
> JUnit 4 can run tests written for JUnit 3.8.1 without any changes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-117-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 01:03:34 2009
Return-Path: <common-issues-return-117-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67394 invoked from network); 8 Jul 2009 01:03:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 01:03:33 -0000
Received: (qmail 29950 invoked by uid 500); 8 Jul 2009 01:03:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29911 invoked by uid 500); 8 Jul 2009 01:03:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29901 invoked by uid 99); 8 Jul 2009 01:03:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 01:03:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 01:03:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CC678234C004
	for <common-issues@hadoop.apache.org>; Tue,  7 Jul 2009 18:03:14 -0700 (PDT)
Message-ID: <1434776397.1247014994824.JavaMail.jira@brutus>
Date: Tue, 7 Jul 2009 18:03:14 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4901) Upgrade to JUnit 4
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728453#action_12728453 ] 

Konstantin Boudnik commented on HADOOP-4901:
--------------------------------------------

I've done some additional investigation on Spring-test framework and it seems that the tagging solution is already in place there. Here's an example:

{code:title=MyTests.java}
@RunWith(SpringJUnit4ClassRunner.class)
@TestExecutionListeners( {})
public class MyTests {
	@BeforeClass
	public static void setUp() {
		System.out.println("test-group is set to: " + System.getProperty("test-group"));
	}

	@Test
	@IfProfileValue(name = "test-group", values = { "fast", "quick" })
	public void testOne() {
		System.out.println("testOne(): Quick stuff");
	}

	@Test
	@IfProfileValue(name = "test-group", value = "fast")
	public void testTwo() {
		System.out.println("testTwo(): Fast stuff");
	}
}
{code}

then to run say 'fast' tests one just need to specify -Dtest-group=fast at run time.

This solution requires a couple of jars to be pulled down from Maven repo, but other than that is it quite clear and easy to implement.

I have found a couple of the issues with Spring-test framework which will affect this approach or at least might slightly postpone it. See http://jira.springframework.org/browse/SPR-5145 and but the fix is coming in the release 3.0 of Spring, which is around the corner. 

Other minor issues are: http://jira.springframework.org/browse/SPR-5902, http://jira.springframework.org/browse/SPR-5903 but these are rather improvements than bugs.



> Upgrade to JUnit 4
> ------------------
>
>                 Key: HADOOP-4901
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4901
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>            Reporter: Tom White
>            Assignee: Alex Loddengaard
>
> Amongst other things, JUnit 4 has better support for class-wide set up and tear down (via @BeforeClass and @AfterClass annotations), and more flexible assertions (http://junit.sourceforge.net/doc/ReleaseNotes4.4.html). It would be nice to be able to take advantage of these features in tests we write.
> JUnit 4 can run tests written for JUnit 3.8.1 without any changes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-118-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 04:22:28 2009
Return-Path: <common-issues-return-118-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82228 invoked from network); 8 Jul 2009 04:22:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 04:22:27 -0000
Received: (qmail 53282 invoked by uid 500); 8 Jul 2009 04:22:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53184 invoked by uid 500); 8 Jul 2009 04:22:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53166 invoked by uid 99); 8 Jul 2009 04:22:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 04:22:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 04:22:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1B8BE29A0016
	for <common-issues@hadoop.apache.org>; Tue,  7 Jul 2009 21:22:15 -0700 (PDT)
Message-ID: <391264787.1247026935111.JavaMail.jira@brutus>
Date: Tue, 7 Jul 2009 21:22:15 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6105) Provide a way to automatically
 handle backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728497#action_12728497 ] 

Owen O'Malley commented on HADOOP-6105:
---------------------------------------

Phillip,
   I think this is too much structure for what is gained. In particular, it replaces a relatively simple string to string map with a lot of code. Where does all of the code go? How does it interact with user's job conf? How do we document it?

I think we should go ahead with the simple approach for now...

> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-119-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 04:40:29 2009
Return-Path: <common-issues-return-119-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51757 invoked from network); 8 Jul 2009 04:40:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 04:40:29 -0000
Received: (qmail 71516 invoked by uid 500); 8 Jul 2009 04:40:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71479 invoked by uid 500); 8 Jul 2009 04:40:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71466 invoked by uid 99); 8 Jul 2009 04:40:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 04:40:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 04:40:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E73E5234C1E6
	for <common-issues@hadoop.apache.org>; Tue,  7 Jul 2009 21:40:14 -0700 (PDT)
Message-ID: <770941932.1247028014946.JavaMail.jira@brutus>
Date: Tue, 7 Jul 2009 21:40:14 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4493) Localized files from
 DistributedCache should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728503#action_12728503 ] 

Owen O'Malley commented on HADOOP-4493:
---------------------------------------

I think the default should be case 1, with completely private distributed caches.

I believe that we don't need sharing with a group, and can probably delay doing case 2 (global sharing) until later.

Thoughts?

> Localized files from DistributedCache should have right access-control
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-4493
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4493
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: mapred, security
>            Reporter: Arun C Murthy
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-120-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 05:44:27 2009
Return-Path: <common-issues-return-120-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6050 invoked from network); 8 Jul 2009 05:44:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 05:44:27 -0000
Received: (qmail 13482 invoked by uid 500); 8 Jul 2009 05:44:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13442 invoked by uid 500); 8 Jul 2009 05:44:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13432 invoked by uid 99); 8 Jul 2009 05:44:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 05:44:37 +0000
X-ASF-Spam-Status: No, hits=-1999.6 required=10.0
	tests=ALL_TRUSTED,SUBJECT_FUZZY_TION
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 05:44:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D66DD234C051
	for <common-issues@hadoop.apache.org>; Tue,  7 Jul 2009 22:44:14 -0700 (PDT)
Message-ID: <2099190480.1247031854877.JavaMail.jira@brutus>
Date: Tue, 7 Jul 2009 22:44:14 -0700 (PDT)
From: "Suman Sehgal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6130) ArrayIndexOutOfBoundsException is
 thrown by KeyFieldBasedPartitioner
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

ArrayIndexOutOfBoundsException is thrown by KeyFieldBasedPartitioner
--------------------------------------------------------------------

                 Key: HADOOP-6130
                 URL: https://issues.apache.org/jira/browse/HADOOP-6130
             Project: Hadoop Common
          Issue Type: Bug
          Components: mapred
    Affects Versions: 0.20.0
            Reporter: Suman Sehgal
            Priority: Critical


KeyFieldBasedPartitioner throws "KeyFieldBasedPartitioner" when some part of the specified key is missing. 
Scenario :
=======
when  value of num.key.fields.for.partition is greater than the separators provided in the input.
Command:
========
hadoop jar streaming.jar -Dmapred.reduce.tasks=3 -Dnum.key.fields.for.partition=5 -input <input-dir>  -output <output-dir> -mapper org.apache.hadoop.mapred.lib.IdentityMapper -reducer org.apache.hadoop.mapred.lib.IdentityReducer -inputformat org.apache.hadoop.mapred.KeyValueTextInputFormat -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner



-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-121-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 05:58:27 2009
Return-Path: <common-issues-return-121-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37650 invoked from network); 8 Jul 2009 05:58:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 05:58:27 -0000
Received: (qmail 27220 invoked by uid 500); 8 Jul 2009 05:58:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27181 invoked by uid 500); 8 Jul 2009 05:58:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27171 invoked by uid 99); 8 Jul 2009 05:58:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 05:58:37 +0000
X-ASF-Spam-Status: No, hits=-1999.6 required=10.0
	tests=ALL_TRUSTED,SUBJECT_FUZZY_TION
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 05:58:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C26D5234C004
	for <common-issues@hadoop.apache.org>; Tue,  7 Jul 2009 22:58:14 -0700 (PDT)
Message-ID: <1784601717.1247032694791.JavaMail.jira@brutus>
Date: Tue, 7 Jul 2009 22:58:14 -0700 (PDT)
From: "Suman Sehgal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6130) ArrayIndexOutOfBoundsException is
 thrown by KeyFieldBasedPartitioner
In-Reply-To: <2099190480.1247031854877.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728521#action_12728521 ] 

Suman Sehgal commented on HADOOP-6130:
--------------------------------------

Stack trace:
=========
java.lang.ArrayIndexOutOfBoundsException: 42
	at org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner.hashCode(KeyFieldBasedPartitioner.java:95)
	at org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner.getPartition(KeyFieldBasedPartitioner.java:87)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:801)
	at org.apache.hadoop.mapred.lib.IdentityMapper.map(IdentityMapper.java:40)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:356)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
	at org.apache.hadoop.mapred.Child.main(Child.java:170)


> ArrayIndexOutOfBoundsException is thrown by KeyFieldBasedPartitioner
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6130
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6130
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Suman Sehgal
>            Priority: Critical
>
> KeyFieldBasedPartitioner throws "KeyFieldBasedPartitioner" when some part of the specified key is missing. 
> Scenario :
> =======
> when  value of num.key.fields.for.partition is greater than the separators provided in the input.
> Command:
> ========
> hadoop jar streaming.jar -Dmapred.reduce.tasks=3 -Dnum.key.fields.for.partition=5 -input <input-dir>  -output <output-dir> -mapper org.apache.hadoop.mapred.lib.IdentityMapper -reducer org.apache.hadoop.mapred.lib.IdentityReducer -inputformat org.apache.hadoop.mapred.KeyValueTextInputFormat -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-122-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 09:18:29 2009
Return-Path: <common-issues-return-122-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88287 invoked from network); 8 Jul 2009 09:18:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 09:18:29 -0000
Received: (qmail 54844 invoked by uid 500); 8 Jul 2009 09:18:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54818 invoked by uid 500); 8 Jul 2009 09:18:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54808 invoked by uid 99); 8 Jul 2009 09:18:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 09:18:38 +0000
X-ASF-Spam-Status: No, hits=-1999.6 required=10.0
	tests=ALL_TRUSTED,SUBJECT_FUZZY_TION
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 09:18:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C40FE234C004
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 02:18:14 -0700 (PDT)
Message-ID: <2062532079.1247044694798.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 02:18:14 -0700 (PDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6130) ArrayIndexOutOfBoundsException is
 thrown by KeyFieldBasedPartitioner
In-Reply-To: <2099190480.1247031854877.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6130:
--------------------------------

    Priority: Major  (was: Critical)
    Assignee: Amar Kamat

Lowering the priority since this bug would affect only those jobs that specify partitioning based on some key fields, and the generated keys doesn't have enough fields.

> ArrayIndexOutOfBoundsException is thrown by KeyFieldBasedPartitioner
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6130
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6130
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Suman Sehgal
>            Assignee: Amar Kamat
>
> KeyFieldBasedPartitioner throws "KeyFieldBasedPartitioner" when some part of the specified key is missing. 
> Scenario :
> =======
> when  value of num.key.fields.for.partition is greater than the separators provided in the input.
> Command:
> ========
> hadoop jar streaming.jar -Dmapred.reduce.tasks=3 -Dnum.key.fields.for.partition=5 -input <input-dir>  -output <output-dir> -mapper org.apache.hadoop.mapred.lib.IdentityMapper -reducer org.apache.hadoop.mapred.lib.IdentityReducer -inputformat org.apache.hadoop.mapred.KeyValueTextInputFormat -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-123-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 10:30:30 2009
Return-Path: <common-issues-return-123-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39784 invoked from network); 8 Jul 2009 10:30:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 10:30:29 -0000
Received: (qmail 37908 invoked by uid 500); 8 Jul 2009 10:30:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37869 invoked by uid 500); 8 Jul 2009 10:30:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37859 invoked by uid 99); 8 Jul 2009 10:30:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 10:30:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 10:30:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1FA9F234C4AC
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 03:30:15 -0700 (PDT)
Message-ID: <1590118913.1247049015128.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 03:30:15 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6131) A sysproperty should not be set
 unless the property is set on the ant command line in build.xml.
In-Reply-To: <661438531.1247049014928.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang reassigned HADOOP-6131:
---------------------------------

    Assignee: Hong Tang

> A sysproperty should not be set unless the property is set on the ant command line in build.xml.
> ------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6131
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6131
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Trivial
>
> Patch for HADOOP-3315 contains an improper usage of setting a sysproperty. What it does now:
> {code}
>       <sysproperty key="io.compression.codec.lzo.class"
>           value="${io.compression.codec.lzo.class}"/>
> {code}
> What should be:
> {code} 
>      <syspropertyset dynamic="no">
>          <propertyref name="io.compression.codec.lzo.class"/>
>       </syspropertyset>
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-124-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 10:30:30 2009
Return-Path: <common-issues-return-124-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39841 invoked from network); 8 Jul 2009 10:30:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 10:30:30 -0000
Received: (qmail 38028 invoked by uid 500); 8 Jul 2009 10:30:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37952 invoked by uid 500); 8 Jul 2009 10:30:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37923 invoked by uid 99); 8 Jul 2009 10:30:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 10:30:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 10:30:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E2FCB234C1EA
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 03:30:14 -0700 (PDT)
Message-ID: <661438531.1247049014928.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 03:30:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6131) A sysproperty should not be set
 unless the property is set on the ant command line in build.xml.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

A sysproperty should not be set unless the property is set on the ant command line in build.xml.
------------------------------------------------------------------------------------------------

                 Key: HADOOP-6131
                 URL: https://issues.apache.org/jira/browse/HADOOP-6131
             Project: Hadoop Common
          Issue Type: Bug
          Components: build
    Affects Versions: 0.21.0
            Reporter: Hong Tang
            Priority: Trivial


Patch for HADOOP-3315 contains an improper usage of setting a sysproperty. What it does now:
{code}
      <sysproperty key="io.compression.codec.lzo.class"
          value="${io.compression.codec.lzo.class}"/>
{code}
What should be:
{code} 
     <syspropertyset dynamic="no">
         <propertyref name="io.compression.codec.lzo.class"/>
      </syspropertyset>
{code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-125-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 10:32:28 2009
Return-Path: <common-issues-return-125-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42702 invoked from network); 8 Jul 2009 10:32:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 10:32:28 -0000
Received: (qmail 43002 invoked by uid 500); 8 Jul 2009 10:32:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42975 invoked by uid 500); 8 Jul 2009 10:32:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42935 invoked by uid 99); 8 Jul 2009 10:32:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 10:32:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 10:32:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 14778234C1F2
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 03:32:16 -0700 (PDT)
Message-ID: <1585179252.1247049136083.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 03:32:16 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6131) A sysproperty should not be set
 unless the property is set on the ant command line in build.xml.
In-Reply-To: <661438531.1247049014928.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-6131:
------------------------------

    Fix Version/s: 0.21.0
           Status: Patch Available  (was: Open)

Attached patch made the minor changes to build.xml. No test necessary.

> A sysproperty should not be set unless the property is set on the ant command line in build.xml.
> ------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6131
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6131
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Trivial
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6131.patch
>
>
> Patch for HADOOP-3315 contains an improper usage of setting a sysproperty. What it does now:
> {code}
>       <sysproperty key="io.compression.codec.lzo.class"
>           value="${io.compression.codec.lzo.class}"/>
> {code}
> What should be:
> {code} 
>      <syspropertyset dynamic="no">
>          <propertyref name="io.compression.codec.lzo.class"/>
>       </syspropertyset>
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-126-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 10:32:30 2009
Return-Path: <common-issues-return-126-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42798 invoked from network); 8 Jul 2009 10:32:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 10:32:30 -0000
Received: (qmail 43277 invoked by uid 500); 8 Jul 2009 10:32:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43239 invoked by uid 500); 8 Jul 2009 10:32:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43182 invoked by uid 99); 8 Jul 2009 10:32:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 10:32:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 10:32:37 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DB8B7234C1EB
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 03:32:15 -0700 (PDT)
Message-ID: <1488414269.1247049135898.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 03:32:15 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6131) A sysproperty should not be set
 unless the property is set on the ant command line in build.xml.
In-Reply-To: <661438531.1247049014928.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-6131:
------------------------------

    Attachment: hadoop-6131.patch

> A sysproperty should not be set unless the property is set on the ant command line in build.xml.
> ------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6131
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6131
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Trivial
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6131.patch
>
>
> Patch for HADOOP-3315 contains an improper usage of setting a sysproperty. What it does now:
> {code}
>       <sysproperty key="io.compression.codec.lzo.class"
>           value="${io.compression.codec.lzo.class}"/>
> {code}
> What should be:
> {code} 
>      <syspropertyset dynamic="no">
>          <propertyref name="io.compression.codec.lzo.class"/>
>       </syspropertyset>
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-127-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 11:12:27 2009
Return-Path: <common-issues-return-127-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97912 invoked from network); 8 Jul 2009 11:12:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 11:12:27 -0000
Received: (qmail 88627 invoked by uid 500); 8 Jul 2009 11:12:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88588 invoked by uid 500); 8 Jul 2009 11:12:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88577 invoked by uid 99); 8 Jul 2009 11:12:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 11:12:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 11:12:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C8553234C004
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 04:12:14 -0700 (PDT)
Message-ID: <617180769.1247051534806.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 04:12:14 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5976) create script to provide classpath
 for external tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728630#action_12728630 ] 

Hudson commented on HADOOP-5976:
--------------------------------

Integrated in Hadoop-Common-trunk #20 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/20/])
    . Add a new command, classpath, to the hadoop script.  Contributed by Owen O'Malley and Gary Murry


> create script to provide classpath for external tools
> -----------------------------------------------------
>
>                 Key: HADOOP-5976
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5976
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.21.0
>
>         Attachments: h5976.patch, Hadoop-5976.patch
>
>
> It would be useful for tools building on top of Hadoop to have a script that returns the class path that is needed to get all of the Hadoop jars and the needed libraries.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-128-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 12:13:35 2009
Return-Path: <common-issues-return-128-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13032 invoked from network); 8 Jul 2009 12:13:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 12:13:31 -0000
Received: (qmail 66651 invoked by uid 500); 8 Jul 2009 12:13:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66601 invoked by uid 500); 8 Jul 2009 12:13:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65875 invoked by uid 99); 8 Jul 2009 12:13:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 12:13:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 12:13:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CDE33234C004
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 05:13:14 -0700 (PDT)
Message-ID: <1288288922.1247055194828.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 05:13:14 -0700 (PDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4493) Localized files from
 DistributedCache should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728647#action_12728647 ] 

Vinod K V commented on HADOOP-4493:
-----------------------------------

One comment that came out of a Hemanth's discussion with Devaraj/Owen is w.r.t access of user files on the file system by the DistributedCache api on the TT. The user files will be assumed to be securely owned by the user and so TT has the onus to use the job's configuration and hence the user's identity to download files from the file system.

> Localized files from DistributedCache should have right access-control
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-4493
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4493
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: mapred, security
>            Reporter: Arun C Murthy
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-129-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 13:11:27 2009
Return-Path: <common-issues-return-129-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38229 invoked from network); 8 Jul 2009 13:11:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 13:11:27 -0000
Received: (qmail 50128 invoked by uid 500); 8 Jul 2009 13:11:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50091 invoked by uid 500); 8 Jul 2009 13:11:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50081 invoked by uid 99); 8 Jul 2009 13:11:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 13:11:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 13:11:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D698B234C044
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 06:11:14 -0700 (PDT)
Message-ID: <1682185659.1247058674877.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 06:11:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6131) A sysproperty should not be set
 unless the property is set on the ant command line in build.xml.
In-Reply-To: <661438531.1247049014928.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728680#action_12728680 ] 

Hadoop QA commented on HADOOP-6131:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12412848/hadoop-6131.patch
  against trunk revision 791937.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/557/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/557/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/557/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/557/console

This message is automatically generated.

> A sysproperty should not be set unless the property is set on the ant command line in build.xml.
> ------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6131
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6131
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Trivial
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6131.patch
>
>
> Patch for HADOOP-3315 contains an improper usage of setting a sysproperty. What it does now:
> {code}
>       <sysproperty key="io.compression.codec.lzo.class"
>           value="${io.compression.codec.lzo.class}"/>
> {code}
> What should be:
> {code} 
>      <syspropertyset dynamic="no">
>          <propertyref name="io.compression.codec.lzo.class"/>
>       </syspropertyset>
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-130-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 13:13:27 2009
Return-Path: <common-issues-return-130-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38686 invoked from network); 8 Jul 2009 13:13:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 13:13:27 -0000
Received: (qmail 50746 invoked by uid 500); 8 Jul 2009 13:13:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50708 invoked by uid 500); 8 Jul 2009 13:13:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50698 invoked by uid 99); 8 Jul 2009 13:13:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 13:13:37 +0000
X-ASF-Spam-Status: No, hits=-1999.6 required=10.0
	tests=ALL_TRUSTED,SUBJECT_FUZZY_TION
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 13:13:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C3B67234C004
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 06:13:14 -0700 (PDT)
Message-ID: <892459876.1247058794797.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 06:13:14 -0700 (PDT)
From: "Amar Kamat (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6130) ArrayIndexOutOfBoundsException is
 thrown by KeyFieldBasedPartitioner
In-Reply-To: <2099190480.1247031854877.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amar Kamat updated HADOOP-6130:
-------------------------------

    Attachment: HADOOP-6130-v1.0.patch

Attaching a patch the solves the issue which I could reproduce. Testing in progress. 

> ArrayIndexOutOfBoundsException is thrown by KeyFieldBasedPartitioner
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6130
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6130
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.20.0
>            Reporter: Suman Sehgal
>            Assignee: Amar Kamat
>         Attachments: HADOOP-6130-v1.0.patch
>
>
> KeyFieldBasedPartitioner throws "KeyFieldBasedPartitioner" when some part of the specified key is missing. 
> Scenario :
> =======
> when  value of num.key.fields.for.partition is greater than the separators provided in the input.
> Command:
> ========
> hadoop jar streaming.jar -Dmapred.reduce.tasks=3 -Dnum.key.fields.for.partition=5 -input <input-dir>  -output <output-dir> -mapper org.apache.hadoop.mapred.lib.IdentityMapper -reducer org.apache.hadoop.mapred.lib.IdentityReducer -inputformat org.apache.hadoop.mapred.KeyValueTextInputFormat -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-131-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 18:43:27 2009
Return-Path: <common-issues-return-131-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27593 invoked from network); 8 Jul 2009 18:43:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 18:43:27 -0000
Received: (qmail 75390 invoked by uid 500); 8 Jul 2009 18:43:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75360 invoked by uid 500); 8 Jul 2009 18:43:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75339 invoked by uid 99); 8 Jul 2009 18:43:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 18:43:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 18:43:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C62BF234C004
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 11:43:14 -0700 (PDT)
Message-ID: <1507256274.1247078594799.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 11:43:14 -0700 (PDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6099) Allow configuring the IPC module to
 send pings
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-6099:
-------------------------------------

    Attachment: ipcPing2.txt

Thanks for the review Konstantin. I also attached a unit test.

> Allow configuring the IPC module to send pings
> ----------------------------------------------
>
>                 Key: HADOOP-6099
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6099
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>         Attachments: ipcPing.txt, ipcPing2.txt
>
>
> The IPC Client sets a socketTimeout for the time period specified by the pingInterval and then sends a ping every pingInterval. This means that if a RPC server does not respond to a RPC client, then the RPC client blocks forever. This is a problem for applications that wants to switch quickly from one un-responsive HDFS cluster  to a good one. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-132-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 18:45:27 2009
Return-Path: <common-issues-return-132-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27887 invoked from network); 8 Jul 2009 18:45:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 18:45:27 -0000
Received: (qmail 77265 invoked by uid 500); 8 Jul 2009 18:45:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77228 invoked by uid 500); 8 Jul 2009 18:45:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77216 invoked by uid 99); 8 Jul 2009 18:45:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 18:45:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 18:45:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D8896234C004
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 11:45:14 -0700 (PDT)
Message-ID: <917190778.1247078714872.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 11:45:14 -0700 (PDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6099) Allow configuring the IPC module to
 send pings
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-6099:
-------------------------------------

    Fix Version/s: 0.21.0
           Status: Patch Available  (was: Open)

> Allow configuring the IPC module to send pings
> ----------------------------------------------
>
>                 Key: HADOOP-6099
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6099
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>             Fix For: 0.21.0
>
>         Attachments: ipcPing.txt, ipcPing2.txt
>
>
> The IPC Client sets a socketTimeout for the time period specified by the pingInterval and then sends a ping every pingInterval. This means that if a RPC server does not respond to a RPC client, then the RPC client blocks forever. This is a problem for applications that wants to switch quickly from one un-responsive HDFS cluster  to a good one. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-133-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 19:01:28 2009
Return-Path: <common-issues-return-133-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47434 invoked from network); 8 Jul 2009 19:01:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 19:01:28 -0000
Received: (qmail 9602 invoked by uid 500); 8 Jul 2009 19:01:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9563 invoked by uid 500); 8 Jul 2009 19:01:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9548 invoked by uid 99); 8 Jul 2009 19:01:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 19:01:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 19:01:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 250AF29A0011
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 12:01:16 -0700 (PDT)
Message-ID: <1000055472.1247079676150.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 12:01:16 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6099) Allow configuring the IPC module to
 send pings
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728851#action_12728851 ] 

Hadoop QA commented on HADOOP-6099:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12412894/ipcPing2.txt
  against trunk revision 791937.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    -1 release audit.  The applied patch generated 260 release audit warnings (more than the trunk's current 258 warnings).

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/558/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/558/artifact/trunk/current/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/558/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/558/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/558/console

This message is automatically generated.

> Allow configuring the IPC module to send pings
> ----------------------------------------------
>
>                 Key: HADOOP-6099
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6099
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>             Fix For: 0.21.0
>
>         Attachments: ipcPing.txt, ipcPing2.txt
>
>
> The IPC Client sets a socketTimeout for the time period specified by the pingInterval and then sends a ping every pingInterval. This means that if a RPC server does not respond to a RPC client, then the RPC client blocks forever. This is a problem for applications that wants to switch quickly from one un-responsive HDFS cluster  to a good one. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-134-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 20:31:27 2009
Return-Path: <common-issues-return-134-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10750 invoked from network); 8 Jul 2009 20:31:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 20:31:27 -0000
Received: (qmail 8092 invoked by uid 500); 8 Jul 2009 20:31:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8055 invoked by uid 500); 8 Jul 2009 20:31:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8045 invoked by uid 99); 8 Jul 2009 20:31:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 20:31:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 20:31:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 2825C234C052
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 13:31:15 -0700 (PDT)
Message-ID: <840919907.1247085075163.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 13:31:15 -0700 (PDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6132) RPC client opens 2 connections per
 real protocol
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

RPC client opens 2 connections per real protocol
------------------------------------------------

                 Key: HADOOP-6132
                 URL: https://issues.apache.org/jira/browse/HADOOP-6132
             Project: Hadoop Common
          Issue Type: Bug
          Components: ipc
            Reporter: Kan Zhang
            Assignee: Kan Zhang


RPC client caches connections per protocol. However, since all of our real protocols are subclasses of VersionedProtocol, a bug in the implementation makes the client opens an extra connection just for the VersionedProtocol, which is not needed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-135-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 20:31:29 2009
Return-Path: <common-issues-return-135-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10805 invoked from network); 8 Jul 2009 20:31:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 20:31:29 -0000
Received: (qmail 8170 invoked by uid 500); 8 Jul 2009 20:31:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8130 invoked by uid 500); 8 Jul 2009 20:31:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8117 invoked by uid 99); 8 Jul 2009 20:31:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 20:31:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 20:31:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 14A89234C004
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 13:31:15 -0700 (PDT)
Message-ID: <1522632601.1247085075080.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 13:31:15 -0700 (PDT)
From: "Mahadev konar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6131) A sysproperty should not be set
 unless the property is set on the ant command line in build.xml.
In-Reply-To: <661438531.1247049014928.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728883#action_12728883 ] 

Mahadev konar commented on HADOOP-6131:
---------------------------------------

+1 to the patch. the -1 from tests is because hudson is running create-c++-configure which isnt a target in common. 
      
{code} [exec] /home/hudson/tools/ant/latest/bin/ant -Dversion=791937_HADOOP-6131_PATCH-12412848 -DHadoopPatchProcess= -Dtest.junit.output.format=xml -Dtest.output=yes -Dcompile.c++=yes -Dforrest.home=/home/nigel/tools/forrest/latest -Djava5.home=/home/hudson/tools/java/latest1.5 create-c++-configure test-core 
{code}

> A sysproperty should not be set unless the property is set on the ant command line in build.xml.
> ------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6131
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6131
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Trivial
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6131.patch
>
>
> Patch for HADOOP-3315 contains an improper usage of setting a sysproperty. What it does now:
> {code}
>       <sysproperty key="io.compression.codec.lzo.class"
>           value="${io.compression.codec.lzo.class}"/>
> {code}
> What should be:
> {code} 
>      <syspropertyset dynamic="no">
>          <propertyref name="io.compression.codec.lzo.class"/>
>       </syspropertyset>
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-136-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 20:33:28 2009
Return-Path: <common-issues-return-136-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15324 invoked from network); 8 Jul 2009 20:33:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 20:33:27 -0000
Received: (qmail 9602 invoked by uid 500); 8 Jul 2009 20:33:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9559 invoked by uid 500); 8 Jul 2009 20:33:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9540 invoked by uid 99); 8 Jul 2009 20:33:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 20:33:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 20:33:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1557A234C004
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 13:33:15 -0700 (PDT)
Message-ID: <1830982095.1247085195082.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 13:33:15 -0700 (PDT)
From: "Mahadev konar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6131) A sysproperty should not be set
 unless the property is set on the ant command line in build.xml.
In-Reply-To: <661438531.1247049014928.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728884#action_12728884 ] 

Mahadev konar commented on HADOOP-6131:
---------------------------------------

+1 for the patch I meant... 

> A sysproperty should not be set unless the property is set on the ant command line in build.xml.
> ------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6131
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6131
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Trivial
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6131.patch
>
>
> Patch for HADOOP-3315 contains an improper usage of setting a sysproperty. What it does now:
> {code}
>       <sysproperty key="io.compression.codec.lzo.class"
>           value="${io.compression.codec.lzo.class}"/>
> {code}
> What should be:
> {code} 
>      <syspropertyset dynamic="no">
>          <propertyref name="io.compression.codec.lzo.class"/>
>       </syspropertyset>
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-137-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 20:35:28 2009
Return-Path: <common-issues-return-137-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22551 invoked from network); 8 Jul 2009 20:35:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 20:35:28 -0000
Received: (qmail 12948 invoked by uid 500); 8 Jul 2009 20:35:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12910 invoked by uid 500); 8 Jul 2009 20:35:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12900 invoked by uid 99); 8 Jul 2009 20:35:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 20:35:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 20:35:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C3FF5234C004
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 13:35:14 -0700 (PDT)
Message-ID: <1768965523.1247085314801.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 13:35:14 -0700 (PDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6132) RPC client opens 2 connections per
 real protocol
In-Reply-To: <840919907.1247085075163.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6132:
------------------------------

    Attachment: 1.patch

attaching a patch. Manually verified that the client only opens one connection per protocol.

> RPC client opens 2 connections per real protocol
> ------------------------------------------------
>
>                 Key: HADOOP-6132
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6132
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>
> RPC client caches connections per protocol. However, since all of our real protocols are subclasses of VersionedProtocol, a bug in the implementation makes the client opens an extra connection just for the VersionedProtocol, which is not needed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-138-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 20:35:29 2009
Return-Path: <common-issues-return-138-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22644 invoked from network); 8 Jul 2009 20:35:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 20:35:29 -0000
Received: (qmail 13017 invoked by uid 500); 8 Jul 2009 20:35:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12974 invoked by uid 500); 8 Jul 2009 20:35:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12964 invoked by uid 99); 8 Jul 2009 20:35:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 20:35:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 20:35:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D41BE234C1E6
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 13:35:14 -0700 (PDT)
Message-ID: <1056526444.1247085314867.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 13:35:14 -0700 (PDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6132) RPC client opens 2 connections per
 real protocol
In-Reply-To: <840919907.1247085075163.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6132:
------------------------------

    Attachment:     (was: 1.patch)

> RPC client opens 2 connections per real protocol
> ------------------------------------------------
>
>                 Key: HADOOP-6132
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6132
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>
> RPC client caches connections per protocol. However, since all of our real protocols are subclasses of VersionedProtocol, a bug in the implementation makes the client opens an extra connection just for the VersionedProtocol, which is not needed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-139-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 20:37:29 2009
Return-Path: <common-issues-return-139-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26794 invoked from network); 8 Jul 2009 20:37:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 20:37:29 -0000
Received: (qmail 22153 invoked by uid 500); 8 Jul 2009 20:37:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22115 invoked by uid 500); 8 Jul 2009 20:37:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22105 invoked by uid 99); 8 Jul 2009 20:37:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 20:37:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 20:37:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 45D16234C004
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 13:37:15 -0700 (PDT)
Message-ID: <1702173255.1247085435271.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 13:37:15 -0700 (PDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6132) RPC client opens 2 connections per
 real protocol
In-Reply-To: <840919907.1247085075163.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6132:
------------------------------

    Attachment: 1.patch

> RPC client opens 2 connections per real protocol
> ------------------------------------------------
>
>                 Key: HADOOP-6132
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6132
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: 1.patch
>
>
> RPC client caches connections per protocol. However, since all of our real protocols are subclasses of VersionedProtocol, a bug in the implementation makes the client opens an extra connection just for the VersionedProtocol, which is not needed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-140-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 20:37:29 2009
Return-Path: <common-issues-return-140-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26822 invoked from network); 8 Jul 2009 20:37:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 20:37:29 -0000
Received: (qmail 22219 invoked by uid 500); 8 Jul 2009 20:37:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22179 invoked by uid 500); 8 Jul 2009 20:37:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22161 invoked by uid 99); 8 Jul 2009 20:37:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 20:37:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 20:37:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 55F17234C051
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 13:37:15 -0700 (PDT)
Message-ID: <2066560201.1247085435351.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 13:37:15 -0700 (PDT)
From: "Mahadev konar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6131) A sysproperty should not be set
 unless the property is set on the ant command line in build.xml.
In-Reply-To: <661438531.1247049014928.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mahadev konar updated HADOOP-6131:
----------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I just committed this. thanks hong.

> A sysproperty should not be set unless the property is set on the ant command line in build.xml.
> ------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6131
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6131
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Trivial
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6131.patch
>
>
> Patch for HADOOP-3315 contains an improper usage of setting a sysproperty. What it does now:
> {code}
>       <sysproperty key="io.compression.codec.lzo.class"
>           value="${io.compression.codec.lzo.class}"/>
> {code}
> What should be:
> {code} 
>      <syspropertyset dynamic="no">
>          <propertyref name="io.compression.codec.lzo.class"/>
>       </syspropertyset>
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-141-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 20:41:29 2009
Return-Path: <common-issues-return-141-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28451 invoked from network); 8 Jul 2009 20:41:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 20:41:29 -0000
Received: (qmail 29829 invoked by uid 500); 8 Jul 2009 20:41:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29793 invoked by uid 500); 8 Jul 2009 20:41:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29783 invoked by uid 99); 8 Jul 2009 20:41:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 20:41:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 20:41:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id F0A2429A0011
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 13:41:14 -0700 (PDT)
Message-ID: <835576799.1247085674984.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 13:41:14 -0700 (PDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6099) Allow configuring the IPC module to
 send pings
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728892#action_12728892 ] 

dhruba borthakur commented on HADOOP-6099:
------------------------------------------

The core test failed beucase of this reason

{quote}
Warning: bad ps syntax, perhaps a bogus '-'? See http://procps.sf.net/faq.html
     [exec] /usr/bin/kill -9 22462 
     [exec] /usr/bin/xargs: /usr/bin/kill: No such file or directory
     [exec] /home/hudson/tools/ant/latest/bin/ant -Dversion=791937_HADOOP-6099_PATCH-12412894 -DHadoopPatchProcess= -Dtest.junit.output.format=xml -Dtest.output=yes -Dcompile.c++=yes -Dforrest.home=/home/nigel/tools/forrest/latest -Djava5.home=/home/hudson/tools/java/latest1.5 create-c++-configure test-core
     [exec] Buildfile: build.xml
     [exec] 
     [exec] BUILD FAILED
     [exec] Target "create-c++-configure" d
{quote}

Is this a problem with the HadoopQA framework?

> Allow configuring the IPC module to send pings
> ----------------------------------------------
>
>                 Key: HADOOP-6099
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6099
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>             Fix For: 0.21.0
>
>         Attachments: ipcPing.txt, ipcPing2.txt
>
>
> The IPC Client sets a socketTimeout for the time period specified by the pingInterval and then sends a ping every pingInterval. This means that if a RPC server does not respond to a RPC client, then the RPC client blocks forever. This is a problem for applications that wants to switch quickly from one un-responsive HDFS cluster  to a good one. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-142-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 20:47:29 2009
Return-Path: <common-issues-return-142-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30479 invoked from network); 8 Jul 2009 20:47:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 20:47:29 -0000
Received: (qmail 39355 invoked by uid 500); 8 Jul 2009 20:47:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39315 invoked by uid 500); 8 Jul 2009 20:47:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39305 invoked by uid 99); 8 Jul 2009 20:47:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 20:47:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 20:47:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D8829234C052
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 13:47:14 -0700 (PDT)
Message-ID: <731641136.1247086034885.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 13:47:14 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6132) RPC client opens 2 connections per
 real protocol
In-Reply-To: <840919907.1247085075163.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728895#action_12728895 ] 

Owen O'Malley commented on HADOOP-6132:
---------------------------------------

Shouldn't the protocol be defined as:

Class<? extends VersionedProtocol> protocol;



> RPC client opens 2 connections per real protocol
> ------------------------------------------------
>
>                 Key: HADOOP-6132
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6132
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: 1.patch
>
>
> RPC client caches connections per protocol. However, since all of our real protocols are subclasses of VersionedProtocol, a bug in the implementation makes the client opens an extra connection just for the VersionedProtocol, which is not needed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-143-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 21:55:27 2009
Return-Path: <common-issues-return-143-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97063 invoked from network); 8 Jul 2009 21:55:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 21:55:27 -0000
Received: (qmail 29690 invoked by uid 500); 8 Jul 2009 21:55:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29649 invoked by uid 500); 8 Jul 2009 21:55:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29639 invoked by uid 99); 8 Jul 2009 21:55:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 21:55:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 21:55:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E1CCE234C044
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 14:55:14 -0700 (PDT)
Message-ID: <138984806.1247090114923.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 14:55:14 -0700 (PDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6132) RPC client opens 2 connections per
 real protocol
In-Reply-To: <840919907.1247085075163.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6132:
------------------------------

    Attachment: 2.patch

Absolutely. An updated patch attached.

> RPC client opens 2 connections per real protocol
> ------------------------------------------------
>
>                 Key: HADOOP-6132
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6132
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: 1.patch, 2.patch
>
>
> RPC client caches connections per protocol. However, since all of our real protocols are subclasses of VersionedProtocol, a bug in the implementation makes the client opens an extra connection just for the VersionedProtocol, which is not needed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-144-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 21:57:29 2009
Return-Path: <common-issues-return-144-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98111 invoked from network); 8 Jul 2009 21:57:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 21:57:29 -0000
Received: (qmail 32556 invoked by uid 500); 8 Jul 2009 21:57:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32518 invoked by uid 500); 8 Jul 2009 21:57:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32508 invoked by uid 99); 8 Jul 2009 21:57:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 21:57:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 21:57:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 12B23234C044
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 14:57:15 -0700 (PDT)
Message-ID: <536187650.1247090235075.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 14:57:15 -0700 (PDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6132) RPC client opens an extra connection
 for VersionedProtocol
In-Reply-To: <840919907.1247085075163.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6132:
------------------------------

    Summary: RPC client opens an extra connection for VersionedProtocol  (was: RPC client opens 2 connections per real protocol)

> RPC client opens an extra connection for VersionedProtocol
> ----------------------------------------------------------
>
>                 Key: HADOOP-6132
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6132
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: 1.patch, 2.patch
>
>
> RPC client caches connections per protocol. However, since all of our real protocols are subclasses of VersionedProtocol, a bug in the implementation makes the client opens an extra connection just for the VersionedProtocol, which is not needed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-146-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 22:07:27 2009
Return-Path: <common-issues-return-146-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2451 invoked from network); 8 Jul 2009 22:07:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 22:07:27 -0000
Received: (qmail 37638 invoked by uid 500); 8 Jul 2009 22:07:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37599 invoked by uid 500); 8 Jul 2009 22:07:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37536 invoked by uid 99); 8 Jul 2009 22:07:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 22:07:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 22:07:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0DA9229A0018
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 15:07:15 -0700 (PDT)
Message-ID: <1827849174.1247090835055.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 15:07:15 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6133) ReflectionUtils performance
 regression
In-Reply-To: <1343704732.1247090834933.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6133:
--------------------------------

    Attachment: Test.java

> ReflectionUtils performance regression
> --------------------------------------
>
>                 Key: HADOOP-6133
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6133
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: Test.java
>
>
> HADOOP-4187 introduced extra calls to Class.forName in ReflectionUtils.setConf. This caused a fairly large performance regression. Attached is a microbenchmark that shows the following timings (ms) for 100M constructions of new instances:
> Explicit construction (new Test): around ~1.6sec
> Using Test.class.newInstance: around ~2.6sec
> ReflectionUtils on 0.18.3: ~8.0sec
> ReflectionUtils on 0.20.0: ~200sec
> This illustrates the ~80x slowdown caused by HADOOP-4187.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-145-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 22:07:27 2009
Return-Path: <common-issues-return-145-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2481 invoked from network); 8 Jul 2009 22:07:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 22:07:27 -0000
Received: (qmail 37575 invoked by uid 500); 8 Jul 2009 22:07:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37535 invoked by uid 500); 8 Jul 2009 22:07:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37520 invoked by uid 99); 8 Jul 2009 22:07:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 22:07:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 22:07:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E4072234C1EB
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 15:07:14 -0700 (PDT)
Message-ID: <1343704732.1247090834933.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 15:07:14 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6133) ReflectionUtils performance
 regression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

ReflectionUtils performance regression
--------------------------------------

                 Key: HADOOP-6133
                 URL: https://issues.apache.org/jira/browse/HADOOP-6133
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
         Attachments: Test.java

HADOOP-4187 introduced extra calls to Class.forName in ReflectionUtils.setConf. This caused a fairly large performance regression. Attached is a microbenchmark that shows the following timings (ms) for 100M constructions of new instances:

Explicit construction (new Test): around ~1.6sec
Using Test.class.newInstance: around ~2.6sec
ReflectionUtils on 0.18.3: ~8.0sec
ReflectionUtils on 0.20.0: ~200sec

This illustrates the ~80x slowdown caused by HADOOP-4187.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-147-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 22:13:28 2009
Return-Path: <common-issues-return-147-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5558 invoked from network); 8 Jul 2009 22:13:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 22:13:28 -0000
Received: (qmail 46460 invoked by uid 500); 8 Jul 2009 22:13:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46428 invoked by uid 500); 8 Jul 2009 22:13:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46393 invoked by uid 99); 8 Jul 2009 22:13:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 22:13:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 22:13:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DE52F234C052
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 15:13:14 -0700 (PDT)
Message-ID: <1525938192.1247091194909.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 15:13:14 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6133) ReflectionUtils performance
 regression
In-Reply-To: <1343704732.1247090834933.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6133:
--------------------------------

          Component/s: conf
    Affects Version/s: 0.20.0

> ReflectionUtils performance regression
> --------------------------------------
>
>                 Key: HADOOP-6133
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6133
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>    Affects Versions: 0.20.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-6133-0.20.patch, Test.java
>
>
> HADOOP-4187 introduced extra calls to Class.forName in ReflectionUtils.setConf. This caused a fairly large performance regression. Attached is a microbenchmark that shows the following timings (ms) for 100M constructions of new instances:
> Explicit construction (new Test): around ~1.6sec
> Using Test.class.newInstance: around ~2.6sec
> ReflectionUtils on 0.18.3: ~8.0sec
> ReflectionUtils on 0.20.0: ~200sec
> This illustrates the ~80x slowdown caused by HADOOP-4187.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-148-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 22:13:29 2009
Return-Path: <common-issues-return-148-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5664 invoked from network); 8 Jul 2009 22:13:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 22:13:29 -0000
Received: (qmail 47296 invoked by uid 500); 8 Jul 2009 22:13:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47265 invoked by uid 500); 8 Jul 2009 22:13:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47248 invoked by uid 99); 8 Jul 2009 22:13:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 22:13:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 22:13:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DB703234C045
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 15:13:14 -0700 (PDT)
Message-ID: <10111321.1247091194897.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 15:13:14 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6133) ReflectionUtils performance
 regression
In-Reply-To: <1343704732.1247090834933.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6133:
--------------------------------

    Attachment: hadoop-6133-0.20.patch

Here is one possible patch to fix this issue. The benchmark results in:

ReflectionUtils on post-patch branch-0.20: ~18.1sec

(still slower than 0.18.3 by about 2.5x but at least tolerable)

This is not the most elegant fix, but the ClassLoader inside Configuration makes it slightly difficult to do this at a different layer.

As for the importance of this - despite advice not to use ReflectionUtils in any hot path, there are cases when this happens. For example, MapWritable and GenericWritable do so for every deserialization. Outside libraries like Cascading also seem to not reuse objects in WritableDeserialization, and we have reports of some jobs spending nearly 100% CPU in Class.forName when profiled.

This patch is against branch-0.20 but should also work post-split after the file rename.

> ReflectionUtils performance regression
> --------------------------------------
>
>                 Key: HADOOP-6133
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6133
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>    Affects Versions: 0.20.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-6133-0.20.patch, Test.java
>
>
> HADOOP-4187 introduced extra calls to Class.forName in ReflectionUtils.setConf. This caused a fairly large performance regression. Attached is a microbenchmark that shows the following timings (ms) for 100M constructions of new instances:
> Explicit construction (new Test): around ~1.6sec
> Using Test.class.newInstance: around ~2.6sec
> ReflectionUtils on 0.18.3: ~8.0sec
> ReflectionUtils on 0.20.0: ~200sec
> This illustrates the ~80x slowdown caused by HADOOP-4187.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-149-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 22:25:29 2009
Return-Path: <common-issues-return-149-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12250 invoked from network); 8 Jul 2009 22:25:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 22:25:29 -0000
Received: (qmail 62616 invoked by uid 500); 8 Jul 2009 22:25:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62577 invoked by uid 500); 8 Jul 2009 22:25:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62567 invoked by uid 99); 8 Jul 2009 22:25:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 22:25:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 22:25:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1CD06234C045
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 15:25:15 -0700 (PDT)
Message-ID: <49150135.1247091915117.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 15:25:15 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6133) ReflectionUtils performance
 regression
In-Reply-To: <1343704732.1247090834933.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728936#action_12728936 ] 

Owen O'Malley commented on HADOOP-6133:
---------------------------------------

Have you tried just caching the JobConfigurable class in a static? That should substantially speed up the code rather than looking it up over and over again. 

> ReflectionUtils performance regression
> --------------------------------------
>
>                 Key: HADOOP-6133
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6133
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>    Affects Versions: 0.20.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-6133-0.20.patch, Test.java
>
>
> HADOOP-4187 introduced extra calls to Class.forName in ReflectionUtils.setConf. This caused a fairly large performance regression. Attached is a microbenchmark that shows the following timings (ms) for 100M constructions of new instances:
> Explicit construction (new Test): around ~1.6sec
> Using Test.class.newInstance: around ~2.6sec
> ReflectionUtils on 0.18.3: ~8.0sec
> ReflectionUtils on 0.20.0: ~200sec
> This illustrates the ~80x slowdown caused by HADOOP-4187.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-150-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 22:31:27 2009
Return-Path: <common-issues-return-150-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14835 invoked from network); 8 Jul 2009 22:31:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 22:31:27 -0000
Received: (qmail 66796 invoked by uid 500); 8 Jul 2009 22:31:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66755 invoked by uid 500); 8 Jul 2009 22:31:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66745 invoked by uid 99); 8 Jul 2009 22:31:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 22:31:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 22:31:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C4CE8234C044
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 15:31:14 -0700 (PDT)
Message-ID: <577798293.1247092274798.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 15:31:14 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6133) ReflectionUtils performance
 regression
In-Reply-To: <1343704732.1247090834933.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728941#action_12728941 ] 

Todd Lipcon commented on HADOOP-6133:
-------------------------------------

Owen: the issue with that is that the JobConfigurable.class has to be loaded through Configuration's ClassLoader. In order to cache it in ReflectionUtils we'd need to have a Map<ClassLoader, Class> CACHE_JOBCONFIGURABLE_CLASSES as well. Otherwise we risk having multiple instances of the JobConfigurable.class and checking isAssignableFrom against the wrong one.

> ReflectionUtils performance regression
> --------------------------------------
>
>                 Key: HADOOP-6133
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6133
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>    Affects Versions: 0.20.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-6133-0.20.patch, Test.java
>
>
> HADOOP-4187 introduced extra calls to Class.forName in ReflectionUtils.setConf. This caused a fairly large performance regression. Attached is a microbenchmark that shows the following timings (ms) for 100M constructions of new instances:
> Explicit construction (new Test): around ~1.6sec
> Using Test.class.newInstance: around ~2.6sec
> ReflectionUtils on 0.18.3: ~8.0sec
> ReflectionUtils on 0.20.0: ~200sec
> This illustrates the ~80x slowdown caused by HADOOP-4187.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-151-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 23:31:30 2009
Return-Path: <common-issues-return-151-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54791 invoked from network); 8 Jul 2009 23:31:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 23:31:28 -0000
Received: (qmail 21285 invoked by uid 500); 8 Jul 2009 23:31:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21247 invoked by uid 500); 8 Jul 2009 23:31:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21237 invoked by uid 99); 8 Jul 2009 23:31:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 23:31:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 23:31:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id F1ACA234C044
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 16:31:15 -0700 (PDT)
Message-ID: <1659559291.1247095875988.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 16:31:15 -0700 (PDT)
From: "Corinne Chandel (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6134) New Hadoop Common Site
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

New Hadoop Common Site
----------------------

                 Key: HADOOP-6134
                 URL: https://issues.apache.org/jira/browse/HADOOP-6134
             Project: Hadoop Common
          Issue Type: Task
          Components: documentation
            Reporter: Corinne Chandel


New Hadoop Common Site

Set up site, initial pass. 
May need to add more content. 
May need to update some links.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-152-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 23:31:34 2009
Return-Path: <common-issues-return-152-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54932 invoked from network); 8 Jul 2009 23:31:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 23:31:34 -0000
Received: (qmail 22194 invoked by uid 500); 8 Jul 2009 23:31:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22152 invoked by uid 500); 8 Jul 2009 23:31:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22142 invoked by uid 99); 8 Jul 2009 23:31:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 23:31:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 23:31:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1294029A0012
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 16:31:16 -0700 (PDT)
Message-ID: <452911042.1247095876075.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 16:31:16 -0700 (PDT)
From: "Zheng Shao (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6135) Fix hadoop-config.sh to work with
 symlinked bin directory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Fix hadoop-config.sh to work with symlinked bin directory
---------------------------------------------------------

                 Key: HADOOP-6135
                 URL: https://issues.apache.org/jira/browse/HADOOP-6135
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.20.0
            Reporter: Zheng Shao
            Assignee: Zheng Shao


The old code does not work if "hadoop/bin" is a symlink. In that case, "hadoop/bin/../conf" may not represent "hadoop/conf".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-153-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 08 23:35:29 2009
Return-Path: <common-issues-return-153-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58884 invoked from network); 8 Jul 2009 23:35:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Jul 2009 23:35:29 -0000
Received: (qmail 25127 invoked by uid 500); 8 Jul 2009 23:35:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25088 invoked by uid 500); 8 Jul 2009 23:35:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25073 invoked by uid 99); 8 Jul 2009 23:35:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 23:35:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jul 2009 23:35:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E7096234C1EB
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 16:35:14 -0700 (PDT)
Message-ID: <70673565.1247096114945.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 16:35:14 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6069) Remove explicit dynamic loading of
 libz in native code
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728964#action_12728964 ] 

Chris Douglas commented on HADOOP-6069:
---------------------------------------

This conflicts with the long-suffering HADOOP-4652, so I hesitate to commit it. What are the advantages?

> Remove explicit dynamic loading of libz in native code
> ------------------------------------------------------
>
>                 Key: HADOOP-6069
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6069
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-6069-guts.txt, hadoop-6069.txt
>
>
> The native zlib code currently uses dlopen/dlsym to dynamically load libz. This used to make sense when there was an lzo option (so you could load libhadoop for lzo purposes without requiring libz as well). Now that libhadoop only has zlib as an option, it makes sense to just add it as an ld flag and let it be automatically loaded as a shlib dependency. I also doubt that there are any distros where libz isn't required by the base system.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-154-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 00:07:27 2009
Return-Path: <common-issues-return-154-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68073 invoked from network); 9 Jul 2009 00:07:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 00:07:27 -0000
Received: (qmail 43016 invoked by uid 500); 9 Jul 2009 00:07:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42978 invoked by uid 500); 9 Jul 2009 00:07:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42967 invoked by uid 99); 9 Jul 2009 00:07:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 00:07:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 00:07:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C8646234C004
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 17:07:14 -0700 (PDT)
Message-ID: <693587578.1247098034813.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 17:07:14 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6069) Remove explicit dynamic loading of
 libz in native code
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728969#action_12728969 ] 

Todd Lipcon commented on HADOOP-6069:
-------------------------------------

The advantage is a significant cleanup of both the autotools and source. I don't see any advantage to dynamically loading libz - zlib is installed on pretty much every Linux system I've ever seen. HADOOP-4652 also seems to depend on libz. If it needs to be updated so it doesn't conflict, I'm happy to do that.

> Remove explicit dynamic loading of libz in native code
> ------------------------------------------------------
>
>                 Key: HADOOP-6069
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6069
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-6069-guts.txt, hadoop-6069.txt
>
>
> The native zlib code currently uses dlopen/dlsym to dynamically load libz. This used to make sense when there was an lzo option (so you could load libhadoop for lzo purposes without requiring libz as well). Now that libhadoop only has zlib as an option, it makes sense to just add it as an ld flag and let it be automatically loaded as a shlib dependency. I also doubt that there are any distros where libz isn't required by the base system.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-155-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 00:33:27 2009
Return-Path: <common-issues-return-155-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 850 invoked from network); 9 Jul 2009 00:33:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 00:33:27 -0000
Received: (qmail 54141 invoked by uid 500); 9 Jul 2009 00:33:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54099 invoked by uid 500); 9 Jul 2009 00:33:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54089 invoked by uid 99); 9 Jul 2009 00:33:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 00:33:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 00:33:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CCF87234C051
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 17:33:14 -0700 (PDT)
Message-ID: <593562115.1247099594838.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 17:33:14 -0700 (PDT)
From: "Corinne Chandel (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6134) New Hadoop Common Site
In-Reply-To: <1659559291.1247095875988.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Corinne Chandel updated HADOOP-6134:
------------------------------------

    Status: Patch Available  (was: Open)

Patch submitted.

> New Hadoop Common Site
> ----------------------
>
>                 Key: HADOOP-6134
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6134
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>            Reporter: Corinne Chandel
>         Attachments: Hadoop-6134.patch
>
>
> New Hadoop Common Site
> Set up site, initial pass. 
> May need to add more content. 
> May need to update some links.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-156-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 00:33:29 2009
Return-Path: <common-issues-return-156-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 906 invoked from network); 9 Jul 2009 00:33:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 00:33:28 -0000
Received: (qmail 54206 invoked by uid 500); 9 Jul 2009 00:33:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54166 invoked by uid 500); 9 Jul 2009 00:33:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54156 invoked by uid 99); 9 Jul 2009 00:33:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 00:33:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 00:33:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C1551234C004
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 17:33:14 -0700 (PDT)
Message-ID: <1557801892.1247099594783.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 17:33:14 -0700 (PDT)
From: "Corinne Chandel (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6134) New Hadoop Common Site
In-Reply-To: <1659559291.1247095875988.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Corinne Chandel updated HADOOP-6134:
------------------------------------

    Attachment: Hadoop-6134.patch

Patch for hadoop-6134.

Apply this patch to: https://svn.apache.org/repos/asf/hadoop/common/site

Note: No new test code requried; changes to documentation only.

> New Hadoop Common Site
> ----------------------
>
>                 Key: HADOOP-6134
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6134
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>            Reporter: Corinne Chandel
>         Attachments: Hadoop-6134.patch
>
>
> New Hadoop Common Site
> Set up site, initial pass. 
> May need to add more content. 
> May need to update some links.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-157-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 00:38:29 2009
Return-Path: <common-issues-return-157-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3131 invoked from network); 9 Jul 2009 00:38:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 00:38:29 -0000
Received: (qmail 84722 invoked by uid 500); 9 Jul 2009 00:38:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84692 invoked by uid 500); 9 Jul 2009 00:38:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84682 invoked by uid 99); 9 Jul 2009 00:38:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 00:38:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 00:38:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 25119234C044
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 17:38:15 -0700 (PDT)
Message-ID: <1288436321.1247099895150.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 17:38:15 -0700 (PDT)
From: "Corinne Chandel (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6136) Hadoop TOP site
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Hadoop TOP site
---------------

                 Key: HADOOP-6136
                 URL: https://issues.apache.org/jira/browse/HADOOP-6136
             Project: Hadoop Common
          Issue Type: Task
          Components: documentation
            Reporter: Corinne Chandel
            Priority: Minor


Hadoop TOP site 

Modification to css styles (to match Hadoop common/hdfs/mapreduce sites).




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-158-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 00:40:27 2009
Return-Path: <common-issues-return-158-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3463 invoked from network); 9 Jul 2009 00:40:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 00:40:27 -0000
Received: (qmail 85584 invoked by uid 500); 9 Jul 2009 00:40:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85546 invoked by uid 500); 9 Jul 2009 00:40:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85536 invoked by uid 99); 9 Jul 2009 00:40:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 00:40:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 00:40:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CB1E9234C004
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 17:40:14 -0700 (PDT)
Message-ID: <546372467.1247100014817.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 17:40:14 -0700 (PDT)
From: "Corinne Chandel (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6136) Hadoop TOP site
In-Reply-To: <1288436321.1247099895150.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Corinne Chandel updated HADOOP-6136:
------------------------------------

    Attachment: Hadoop-6136.patch

Patch for Hadoop-6136.

Apply this patch to the hadoop TOP site: https://svn.apache.org/repos/asf/hadoop/site

Note: No new test code requried; changes to documentation only.

> Hadoop TOP site
> ---------------
>
>                 Key: HADOOP-6136
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6136
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>            Reporter: Corinne Chandel
>            Priority: Minor
>         Attachments: Hadoop-6136.patch
>
>
> Hadoop TOP site 
> Modification to css styles (to match Hadoop common/hdfs/mapreduce sites).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-159-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 00:40:29 2009
Return-Path: <common-issues-return-159-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3507 invoked from network); 9 Jul 2009 00:40:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 00:40:28 -0000
Received: (qmail 85649 invoked by uid 500); 9 Jul 2009 00:40:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85612 invoked by uid 500); 9 Jul 2009 00:40:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85602 invoked by uid 99); 9 Jul 2009 00:40:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 00:40:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 00:40:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DBF92234C045
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 17:40:14 -0700 (PDT)
Message-ID: <147941098.1247100014900.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 17:40:14 -0700 (PDT)
From: "Corinne Chandel (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6136) Hadoop TOP site
In-Reply-To: <1288436321.1247099895150.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Corinne Chandel updated HADOOP-6136:
------------------------------------

    Status: Patch Available  (was: Open)

Patch submitted.

> Hadoop TOP site
> ---------------
>
>                 Key: HADOOP-6136
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6136
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>            Reporter: Corinne Chandel
>            Priority: Minor
>         Attachments: Hadoop-6136.patch
>
>
> Hadoop TOP site 
> Modification to css styles (to match Hadoop common/hdfs/mapreduce sites).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-160-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 00:46:34 2009
Return-Path: <common-issues-return-160-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6147 invoked from network); 9 Jul 2009 00:46:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 00:46:34 -0000
Received: (qmail 90235 invoked by uid 500); 9 Jul 2009 00:46:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90197 invoked by uid 500); 9 Jul 2009 00:46:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90183 invoked by uid 99); 9 Jul 2009 00:46:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 00:46:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 00:46:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D5B65234C1EB
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 17:46:15 -0700 (PDT)
Message-ID: <1445129373.1247100375874.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 17:46:15 -0700 (PDT)
From: "Zheng Shao (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6135) Fix hadoop-config.sh to work with
 symlinked bin directory
In-Reply-To: <452911042.1247095876075.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zheng Shao updated HADOOP-6135:
-------------------------------

    Affects Version/s: 0.17.2
                       0.18.3
                       0.19.1

> Fix hadoop-config.sh to work with symlinked bin directory
> ---------------------------------------------------------
>
>                 Key: HADOOP-6135
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6135
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.17.2, 0.18.3, 0.19.1, 0.20.0
>            Reporter: Zheng Shao
>            Assignee: Zheng Shao
>         Attachments: HADOOP-6135.1.patch
>
>
> The old code does not work if "hadoop/bin" is a symlink. In that case, "hadoop/bin/../conf" may not represent "hadoop/conf".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-161-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 00:46:36 2009
Return-Path: <common-issues-return-161-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6214 invoked from network); 9 Jul 2009 00:46:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 00:46:36 -0000
Received: (qmail 90313 invoked by uid 500); 9 Jul 2009 00:46:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90274 invoked by uid 500); 9 Jul 2009 00:46:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90264 invoked by uid 99); 9 Jul 2009 00:46:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 00:46:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 00:46:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C22E1234C045
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 17:46:15 -0700 (PDT)
Message-ID: <792719340.1247100375794.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 17:46:15 -0700 (PDT)
From: "Zheng Shao (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6135) Fix hadoop-config.sh to work with
 symlinked bin directory
In-Reply-To: <452911042.1247095876075.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zheng Shao updated HADOOP-6135:
-------------------------------

    Attachment: HADOOP-6135.1.patch

This patch fixes the problem for hadoop-0.17/0.18/0.19/0.20.

> Fix hadoop-config.sh to work with symlinked bin directory
> ---------------------------------------------------------
>
>                 Key: HADOOP-6135
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6135
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.17.2, 0.18.3, 0.19.1, 0.20.0
>            Reporter: Zheng Shao
>            Assignee: Zheng Shao
>         Attachments: HADOOP-6135.1.patch
>
>
> The old code does not work if "hadoop/bin" is a symlink. In that case, "hadoop/bin/../conf" may not represent "hadoop/conf".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-162-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 01:46:27 2009
Return-Path: <common-issues-return-162-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79071 invoked from network); 9 Jul 2009 01:46:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 01:46:27 -0000
Received: (qmail 31792 invoked by uid 500); 9 Jul 2009 01:46:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31767 invoked by uid 500); 9 Jul 2009 01:46:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31757 invoked by uid 99); 9 Jul 2009 01:46:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 01:46:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 01:46:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C75FC234C004
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 18:46:14 -0700 (PDT)
Message-ID: <781131784.1247103974801.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 18:46:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6134) New Hadoop Common Site
In-Reply-To: <1659559291.1247095875988.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728996#action_12728996 ] 

Hadoop QA commented on HADOOP-6134:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12412941/Hadoop-6134.patch
  against trunk revision 792302.

    +1 @author.  The patch does not contain any @author tags.

    +0 tests included.  The patch appears to be a documentation patch that doesn't require tests.

    -1 patch.  The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/559/console

This message is automatically generated.

> New Hadoop Common Site
> ----------------------
>
>                 Key: HADOOP-6134
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6134
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>            Reporter: Corinne Chandel
>         Attachments: Hadoop-6134.patch
>
>
> New Hadoop Common Site
> Set up site, initial pass. 
> May need to add more content. 
> May need to update some links.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-163-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 04:40:29 2009
Return-Path: <common-issues-return-163-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25828 invoked from network); 9 Jul 2009 04:40:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 04:40:29 -0000
Received: (qmail 43669 invoked by uid 500); 9 Jul 2009 04:40:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43629 invoked by uid 500); 9 Jul 2009 04:40:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43619 invoked by uid 99); 9 Jul 2009 04:40:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 04:40:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 04:40:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id F0EE6234C004
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 21:40:14 -0700 (PDT)
Message-ID: <353082398.1247114414972.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 21:40:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6136) Hadoop TOP site
In-Reply-To: <1288436321.1247099895150.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729047#action_12729047 ] 

Hadoop QA commented on HADOOP-6136:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12412942/Hadoop-6136.patch
  against trunk revision 792302.

    +1 @author.  The patch does not contain any @author tags.

    +0 tests included.  The patch appears to be a documentation patch that doesn't require tests.

    -1 patch.  The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/560/console

This message is automatically generated.

> Hadoop TOP site
> ---------------
>
>                 Key: HADOOP-6136
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6136
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>            Reporter: Corinne Chandel
>            Priority: Minor
>         Attachments: Hadoop-6136.patch
>
>
> Hadoop TOP site 
> Modification to css styles (to match Hadoop common/hdfs/mapreduce sites).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-164-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 06:14:27 2009
Return-Path: <common-issues-return-164-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40603 invoked from network); 9 Jul 2009 06:14:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 06:14:27 -0000
Received: (qmail 30980 invoked by uid 500); 9 Jul 2009 06:14:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30939 invoked by uid 500); 9 Jul 2009 06:14:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30921 invoked by uid 99); 9 Jul 2009 06:14:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 06:14:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 06:14:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E1153234C1EE
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 23:14:14 -0700 (PDT)
Message-ID: <2044264383.1247120054920.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 23:14:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6137) to fix project specific test-patch
 requirements
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

to fix project specific test-patch requirements 
------------------------------------------------

                 Key: HADOOP-6137
                 URL: https://issues.apache.org/jira/browse/HADOOP-6137
             Project: Hadoop Common
          Issue Type: Bug
          Components: build
            Reporter: Giridharan Kesavan
            Priority: Critical


only mapreduce project needs create-c++-configure target which needs to be executed as part to the test-core ant target.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-165-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 06:16:36 2009
Return-Path: <common-issues-return-165-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49400 invoked from network); 9 Jul 2009 06:16:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 06:16:36 -0000
Received: (qmail 34890 invoked by uid 500); 9 Jul 2009 06:16:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34856 invoked by uid 500); 9 Jul 2009 06:16:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34832 invoked by uid 99); 9 Jul 2009 06:16:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 06:16:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 06:16:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C6D9D234C004
	for <common-issues@hadoop.apache.org>; Wed,  8 Jul 2009 23:16:14 -0700 (PDT)
Message-ID: <334432598.1247120174800.JavaMail.jira@brutus>
Date: Wed, 8 Jul 2009 23:16:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6137) to fix project specific test-patch
 requirements
In-Reply-To: <2044264383.1247120054920.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan reassigned HADOOP-6137:
------------------------------------------

    Assignee: Giridharan Kesavan

> to fix project specific test-patch requirements 
> ------------------------------------------------
>
>                 Key: HADOOP-6137
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6137
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>            Priority: Critical
>
> only mapreduce project needs create-c++-configure target which needs to be executed as part to the test-core ant target.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-166-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 07:52:29 2009
Return-Path: <common-issues-return-166-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90785 invoked from network); 9 Jul 2009 07:52:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 07:52:29 -0000
Received: (qmail 33677 invoked by uid 500); 9 Jul 2009 07:52:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33638 invoked by uid 500); 9 Jul 2009 07:52:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33628 invoked by uid 99); 9 Jul 2009 07:52:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 07:52:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 07:52:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 14BB0234C004
	for <common-issues@hadoop.apache.org>; Thu,  9 Jul 2009 00:52:15 -0700 (PDT)
Message-ID: <1935847253.1247125935070.JavaMail.jira@brutus>
Date: Thu, 9 Jul 2009 00:52:15 -0700 (PDT)
From: "Yongqiang He (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5438) Merge FileSystem.create and
 FileSystem.append
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729083#action_12729083 ] 

Yongqiang He commented on HADOOP-5438:
--------------------------------------

does these two problems be resolved? If not, i will open another jira and resolve it.

> Merge FileSystem.create and FileSystem.append
> ---------------------------------------------
>
>                 Key: HADOOP-5438
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5438
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Yongqiang He
>            Assignee: Yongqiang He
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-5438(2009-04-06).patch, Hadoop-5438-2009-03-30.patch, Hadoop-5438-2009-03-31-2.patch, Hadoop-5438-2009-03-31.patch, Hadoop-5438-2009-05-10.patch, Hadoop-5438-2009-05-15.patch, Hadoop-5438-2009-05-19.patch, Hadoop-5438-2009-05-5.patch
>
>
> Currently, when a user wants to modify a file, the user first calls exists() to know if this file is already there. And then uses create() or append() according to whether the file exists or not.
> the code looks like:
> {code}
> FSDataOutputStream out_1 = null;
> if (fs.exists(path_1))
>    out_1 = fs.append(path_1);
> else
>    out_1 = fs.create(path_1);
> {code}
> . On the performace side,It involes two RPCs. On the easy-of-use side, it is not very convient in contrast to the traditional open interface.
> It will more complicate if there is a overwrite parameter specified. I donot know whether there is a bug about 'overwrite' in 0.19, some times it takes a long time for overwrite creates to reture. So i make the write file code with overwrite param works like:
> {code}
> boolean exists = fs.exists(name);
> if (overwrite) {
>     if (exists)
>        fs.delete(name, true);
>      this.out = fs.create(name, overwrite, bufferSize, replication,
> 				    blockSize, progress);
>      this.currentRowID = 0;
>  } else {
>    if (!exists)
> 	this.out = fs.create(name, overwrite, bufferSize,
> 					replication, blockSize, progress);
>    else
> 	this.out = fs.append(name, bufferSize, progress);
> {code}
> Some code statements there are really redundant and not needed, especialy with the delete(). But without deleting first, the overwrite takes a long time to reture.
> BTW, i will create another issue about the overwrite problem. If it is not a bug at all or a duplicate, someone please close it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-167-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 08:06:27 2009
Return-Path: <common-issues-return-167-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94691 invoked from network); 9 Jul 2009 08:06:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 08:06:27 -0000
Received: (qmail 43324 invoked by uid 500); 9 Jul 2009 08:06:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43287 invoked by uid 500); 9 Jul 2009 08:06:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43273 invoked by uid 99); 9 Jul 2009 08:06:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 08:06:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 08:06:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D900B234C1EE
	for <common-issues@hadoop.apache.org>; Thu,  9 Jul 2009 01:06:14 -0700 (PDT)
Message-ID: <767725942.1247126774888.JavaMail.jira@brutus>
Date: Thu, 9 Jul 2009 01:06:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6137) to fix project specific test-patch
 requirements
In-Reply-To: <2044264383.1247120054920.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729091#action_12729091 ] 

Giridharan Kesavan commented on HADOOP-6137:
--------------------------------------------

this patch cannot be tested using test-patch script as this patches the test-patch script itself.


> to fix project specific test-patch requirements 
> ------------------------------------------------
>
>                 Key: HADOOP-6137
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6137
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>            Priority: Critical
>         Attachments: HADOOP-6137.patch
>
>
> only mapreduce project needs create-c++-configure target which needs to be executed as part to the test-core ant target.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-168-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 08:06:29 2009
Return-Path: <common-issues-return-168-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94755 invoked from network); 9 Jul 2009 08:06:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 08:06:29 -0000
Received: (qmail 43398 invoked by uid 500); 9 Jul 2009 08:06:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43360 invoked by uid 500); 9 Jul 2009 08:06:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43347 invoked by uid 99); 9 Jul 2009 08:06:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 08:06:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 08:06:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C8BD3234C004
	for <common-issues@hadoop.apache.org>; Thu,  9 Jul 2009 01:06:14 -0700 (PDT)
Message-ID: <1772322680.1247126774808.JavaMail.jira@brutus>
Date: Thu, 9 Jul 2009 01:06:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6137) to fix project specific test-patch
 requirements
In-Reply-To: <2044264383.1247120054920.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6137:
---------------------------------------

    Attachment: HADOOP-6137.patch

this patch lets the test-patch to call create-c++-configure target only for mapreduce 

> to fix project specific test-patch requirements 
> ------------------------------------------------
>
>                 Key: HADOOP-6137
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6137
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>            Priority: Critical
>         Attachments: HADOOP-6137.patch
>
>
> only mapreduce project needs create-c++-configure target which needs to be executed as part to the test-core ant target.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-169-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 08:10:27 2009
Return-Path: <common-issues-return-169-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95490 invoked from network); 9 Jul 2009 08:10:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 08:10:27 -0000
Received: (qmail 46327 invoked by uid 500); 9 Jul 2009 08:10:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46289 invoked by uid 500); 9 Jul 2009 08:10:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46279 invoked by uid 99); 9 Jul 2009 08:10:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 08:10:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 08:10:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E9828234C044
	for <common-issues@hadoop.apache.org>; Thu,  9 Jul 2009 01:10:14 -0700 (PDT)
Message-ID: <394630081.1247127014955.JavaMail.jira@brutus>
Date: Thu, 9 Jul 2009 01:10:14 -0700 (PDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6090) GridMix is broke after upgrading
 random(text)writer to newer mapreduce apis
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sharad Agarwal updated HADOOP-6090:
-----------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I committed this. Thanks Amareshwari!

> GridMix is broke after upgrading random(text)writer to newer mapreduce apis
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-6090
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6090
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: benchmarks
>    Affects Versions: 0.21.0
>            Reporter: Arun C Murthy
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.21.0
>
>         Attachments: 6090.txt
>
>
> GridMix data generation scripts need to use the newer mapreduce api.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-170-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 08:12:27 2009
Return-Path: <common-issues-return-170-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96388 invoked from network); 9 Jul 2009 08:12:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 08:12:27 -0000
Received: (qmail 51428 invoked by uid 500); 9 Jul 2009 08:12:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51362 invoked by uid 500); 9 Jul 2009 08:12:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51342 invoked by uid 99); 9 Jul 2009 08:12:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 08:12:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 08:12:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 56246234C48C
	for <common-issues@hadoop.apache.org>; Thu,  9 Jul 2009 01:12:15 -0700 (PDT)
Message-ID: <48227826.1247127135351.JavaMail.jira@brutus>
Date: Thu, 9 Jul 2009 01:12:15 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6137) to fix project specific test-patch
 requirements
In-Reply-To: <2044264383.1247120054920.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan resolved HADOOP-6137.
----------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.21.0

I just committed this!

> to fix project specific test-patch requirements 
> ------------------------------------------------
>
>                 Key: HADOOP-6137
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6137
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>            Priority: Critical
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6137.patch
>
>
> only mapreduce project needs create-c++-configure target which needs to be executed as part to the test-core ant target.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-171-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 09:46:29 2009
Return-Path: <common-issues-return-171-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93616 invoked from network); 9 Jul 2009 09:46:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 09:46:29 -0000
Received: (qmail 75140 invoked by uid 500); 9 Jul 2009 09:46:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75112 invoked by uid 500); 9 Jul 2009 09:46:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75059 invoked by uid 99); 9 Jul 2009 09:46:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 09:46:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 09:46:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 00854234C48C
	for <common-issues@hadoop.apache.org>; Thu,  9 Jul 2009 02:46:15 -0700 (PDT)
Message-ID: <640317395.1247132775001.JavaMail.jira@brutus>
Date: Thu, 9 Jul 2009 02:46:15 -0700 (PDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4493) Localized files from
 DistributedCache should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729127#action_12729127 ] 

Vinod K V commented on HADOOP-4493:
-----------------------------------

Talked with Milind offline, and he too agrees to the proposal of doing Case I first and Case II later. He adds that Case II nevertheless is useful and is on the long term wish-list.

> Localized files from DistributedCache should have right access-control
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-4493
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4493
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-172-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 09:52:28 2009
Return-Path: <common-issues-return-172-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22841 invoked from network); 9 Jul 2009 09:52:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 09:52:28 -0000
Received: (qmail 85646 invoked by uid 500); 9 Jul 2009 09:52:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85539 invoked by uid 500); 9 Jul 2009 09:52:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85514 invoked by uid 99); 9 Jul 2009 09:52:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 09:52:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 09:52:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D1D3E234C004
	for <common-issues@hadoop.apache.org>; Thu,  9 Jul 2009 02:52:14 -0700 (PDT)
Message-ID: <1101670675.1247133134845.JavaMail.jira@brutus>
Date: Thu, 9 Jul 2009 02:52:14 -0700 (PDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4493) Localized files from
 DistributedCache should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729128#action_12729128 ] 

Vinod K V commented on HADOOP-4493:
-----------------------------------

Created  MAPREDUCE-744 for Case II.

> Localized files from DistributedCache should have right access-control
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-4493
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4493
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-173-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 09:57:29 2009
Return-Path: <common-issues-return-173-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26532 invoked from network); 9 Jul 2009 09:57:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 09:57:29 -0000
Received: (qmail 90704 invoked by uid 500); 9 Jul 2009 09:57:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90666 invoked by uid 500); 9 Jul 2009 09:57:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90656 invoked by uid 99); 9 Jul 2009 09:57:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 09:57:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 09:57:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1A904234C1F2
	for <common-issues@hadoop.apache.org>; Thu,  9 Jul 2009 02:57:15 -0700 (PDT)
Message-ID: <1256995069.1247133435107.JavaMail.jira@brutus>
Date: Thu, 9 Jul 2009 02:57:15 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6126) Hadoop QA mails on Patch tests do
 not have links to test failures
In-Reply-To: <1085003647.1246879334924.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729132#action_12729132 ] 

Giridharan Kesavan commented on HADOOP-6126:
--------------------------------------------

By looking at the example build #356; I could see that tests target never ran for this build for some reason.

http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/356/console

 [exec] compile-mapred-test:
     [exec]     [javac] Compiling 152 source files to /home/hudson/hudson-slave/workspace/Mapreduce-Patch-vesta.apache.org/trunk/build/test/mapred/classes
     [exec]     [javac] /home/hudson/hudson-slave/workspace/Mapreduce-Patch-vesta.apache.org/trunk/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:178: method does not override or implement a method from a supertype
     [exec]     [javac]     @Override
     [exec]     [javac]     ^
     [exec]     [javac] Note: Some input files use or override a deprecated API.
     [exec]     [javac] Note: Recompile with -Xlint:deprecation for details.
     [exec]     [javac] 1 error
     [exec] 
     [exec] BUILD FAILED
     [exec] /home/hudson/hudson-slave/workspace/Mapreduce-Patch-vesta.apache.org/trunk/build.xml:470: Compile failed; see the compiler error output for details.
     [exec] 
     [exec] Total time: 6 seconds
     [exec] 

Hence the no test results. The test-patch has a standard template for testreport urls. All it does is changes the build number.

Except for this build #356, I could see test results for other builds.

http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/362/testReport and to the latest builds.

Please reopen if you see this issue again.

Tnx 

> Hadoop QA mails on Patch tests do not have links to test failures
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6126
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6126
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Jothi Padmanabhan
>            Assignee: Giridharan Kesavan
>
> The recent hadoopqa results on test patches do not seem to have the links to the actual tests that failed. 
> For example, http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/356/testReport/

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-174-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 09:59:30 2009
Return-Path: <common-issues-return-174-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27675 invoked from network); 9 Jul 2009 09:59:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 09:59:30 -0000
Received: (qmail 93628 invoked by uid 500); 9 Jul 2009 09:59:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93578 invoked by uid 500); 9 Jul 2009 09:59:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93384 invoked by uid 99); 9 Jul 2009 09:59:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 09:59:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 09:59:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E0653234C044
	for <common-issues@hadoop.apache.org>; Thu,  9 Jul 2009 02:59:14 -0700 (PDT)
Message-ID: <794298398.1247133554918.JavaMail.jira@brutus>
Date: Thu, 9 Jul 2009 02:59:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6126) Hadoop QA mails on Patch tests do
 not have links to test failures
In-Reply-To: <1085003647.1246879334924.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan resolved HADOOP-6126.
----------------------------------------

    Resolution: Cannot Reproduce

> Hadoop QA mails on Patch tests do not have links to test failures
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6126
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6126
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Jothi Padmanabhan
>            Assignee: Giridharan Kesavan
>
> The recent hadoopqa results on test patches do not seem to have the links to the actual tests that failed. 
> For example, http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/356/testReport/

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-175-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 11:11:31 2009
Return-Path: <common-issues-return-175-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32687 invoked from network); 9 Jul 2009 11:11:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 11:11:28 -0000
Received: (qmail 86273 invoked by uid 500); 9 Jul 2009 11:11:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86157 invoked by uid 500); 9 Jul 2009 11:11:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86011 invoked by uid 99); 9 Jul 2009 11:11:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 11:11:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 11:11:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 11AB8234C48D
	for <common-issues@hadoop.apache.org>; Thu,  9 Jul 2009 04:11:15 -0700 (PDT)
Message-ID: <1638618452.1247137875071.JavaMail.jira@brutus>
Date: Thu, 9 Jul 2009 04:11:15 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6137) to fix project specific test-patch
 requirements
In-Reply-To: <2044264383.1247120054920.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729185#action_12729185 ] 

Hudson commented on HADOOP-6137:
--------------------------------

Integrated in Hadoop-Common-trunk #21 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/21/])
    . Fix project specific test-patch requirements


> to fix project specific test-patch requirements 
> ------------------------------------------------
>
>                 Key: HADOOP-6137
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6137
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>            Priority: Critical
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6137.patch
>
>
> only mapreduce project needs create-c++-configure target which needs to be executed as part to the test-core ant target.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-176-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 11:11:32 2009
Return-Path: <common-issues-return-176-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32827 invoked from network); 9 Jul 2009 11:11:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 11:11:29 -0000
Received: (qmail 86975 invoked by uid 500); 9 Jul 2009 11:11:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86939 invoked by uid 500); 9 Jul 2009 11:11:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86924 invoked by uid 99); 9 Jul 2009 11:11:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 11:11:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 11:11:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 16841234C4A8
	for <common-issues@hadoop.apache.org>; Thu,  9 Jul 2009 04:11:15 -0700 (PDT)
Message-ID: <2044651330.1247137875091.JavaMail.jira@brutus>
Date: Thu, 9 Jul 2009 04:11:15 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6131) A sysproperty should not be set
 unless the property is set on the ant command line in build.xml.
In-Reply-To: <661438531.1247049014928.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729186#action_12729186 ] 

Hudson commented on HADOOP-6131:
--------------------------------

Integrated in Hadoop-Common-trunk #21 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/21/])
    . A sysproperty should not be set unless the property is set on the ant command line in build.xml (hong tang via mahadev)


> A sysproperty should not be set unless the property is set on the ant command line in build.xml.
> ------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6131
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6131
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Trivial
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6131.patch
>
>
> Patch for HADOOP-3315 contains an improper usage of setting a sysproperty. What it does now:
> {code}
>       <sysproperty key="io.compression.codec.lzo.class"
>           value="${io.compression.codec.lzo.class}"/>
> {code}
> What should be:
> {code} 
>      <syspropertyset dynamic="no">
>          <propertyref name="io.compression.codec.lzo.class"/>
>       </syspropertyset>
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-177-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 15:22:27 2009
Return-Path: <common-issues-return-177-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43542 invoked from network); 9 Jul 2009 15:22:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 15:22:27 -0000
Received: (qmail 99331 invoked by uid 500); 9 Jul 2009 15:22:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99237 invoked by uid 500); 9 Jul 2009 15:22:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99211 invoked by uid 99); 9 Jul 2009 15:22:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 15:22:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 15:22:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id ECB0729A001E
	for <common-issues@hadoop.apache.org>; Thu,  9 Jul 2009 08:22:14 -0700 (PDT)
Message-ID: <316321051.1247152934968.JavaMail.jira@brutus>
Date: Thu, 9 Jul 2009 08:22:14 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6137) to fix project specific test-patch
 requirements
In-Reply-To: <2044264383.1247120054920.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729291#action_12729291 ] 

Hudson commented on HADOOP-6137:
--------------------------------

Integrated in Hadoop-Hdfs-trunk #17 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/17/])
    . Fix project specific test-patch requirements


> to fix project specific test-patch requirements 
> ------------------------------------------------
>
>                 Key: HADOOP-6137
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6137
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>            Priority: Critical
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6137.patch
>
>
> only mapreduce project needs create-c++-configure target which needs to be executed as part to the test-core ant target.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-178-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 17:41:29 2009
Return-Path: <common-issues-return-178-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2702 invoked from network); 9 Jul 2009 17:41:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 17:41:29 -0000
Received: (qmail 84957 invoked by uid 500); 9 Jul 2009 17:41:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84927 invoked by uid 500); 9 Jul 2009 17:41:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84882 invoked by uid 99); 9 Jul 2009 17:41:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 17:41:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 17:41:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C7FA229A0012
	for <common-issues@hadoop.apache.org>; Thu,  9 Jul 2009 10:41:14 -0700 (PDT)
Message-ID: <1711861660.1247161274814.JavaMail.jira@brutus>
Date: Thu, 9 Jul 2009 10:41:14 -0700 (PDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6132) RPC client opens an extra connection
 for VersionedProtocol
In-Reply-To: <840919907.1247085075163.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6132:
------------------------------

    Status: Patch Available  (was: Open)

> RPC client opens an extra connection for VersionedProtocol
> ----------------------------------------------------------
>
>                 Key: HADOOP-6132
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6132
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: 1.patch, 2.patch
>
>
> RPC client caches connections per protocol. However, since all of our real protocols are subclasses of VersionedProtocol, a bug in the implementation makes the client opens an extra connection just for the VersionedProtocol, which is not needed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-179-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 17:59:27 2009
Return-Path: <common-issues-return-179-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11830 invoked from network); 9 Jul 2009 17:59:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 17:59:27 -0000
Received: (qmail 4129 invoked by uid 500); 9 Jul 2009 17:59:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4086 invoked by uid 500); 9 Jul 2009 17:59:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4075 invoked by uid 99); 9 Jul 2009 17:59:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 17:59:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 17:59:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D23D029A0011
	for <common-issues@hadoop.apache.org>; Thu,  9 Jul 2009 10:59:14 -0700 (PDT)
Message-ID: <853258378.1247162354847.JavaMail.jira@brutus>
Date: Thu, 9 Jul 2009 10:59:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5438) Merge FileSystem.create and
 FileSystem.append
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729356#action_12729356 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-5438:
------------------------------------------------

Hi Yongqiang, I fixed HDFS-440.  But the deprecated warnings are still there.  It would be great if you can fix them.

> Merge FileSystem.create and FileSystem.append
> ---------------------------------------------
>
>                 Key: HADOOP-5438
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5438
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Yongqiang He
>            Assignee: Yongqiang He
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-5438(2009-04-06).patch, Hadoop-5438-2009-03-30.patch, Hadoop-5438-2009-03-31-2.patch, Hadoop-5438-2009-03-31.patch, Hadoop-5438-2009-05-10.patch, Hadoop-5438-2009-05-15.patch, Hadoop-5438-2009-05-19.patch, Hadoop-5438-2009-05-5.patch
>
>
> Currently, when a user wants to modify a file, the user first calls exists() to know if this file is already there. And then uses create() or append() according to whether the file exists or not.
> the code looks like:
> {code}
> FSDataOutputStream out_1 = null;
> if (fs.exists(path_1))
>    out_1 = fs.append(path_1);
> else
>    out_1 = fs.create(path_1);
> {code}
> . On the performace side,It involes two RPCs. On the easy-of-use side, it is not very convient in contrast to the traditional open interface.
> It will more complicate if there is a overwrite parameter specified. I donot know whether there is a bug about 'overwrite' in 0.19, some times it takes a long time for overwrite creates to reture. So i make the write file code with overwrite param works like:
> {code}
> boolean exists = fs.exists(name);
> if (overwrite) {
>     if (exists)
>        fs.delete(name, true);
>      this.out = fs.create(name, overwrite, bufferSize, replication,
> 				    blockSize, progress);
>      this.currentRowID = 0;
>  } else {
>    if (!exists)
> 	this.out = fs.create(name, overwrite, bufferSize,
> 					replication, blockSize, progress);
>    else
> 	this.out = fs.append(name, bufferSize, progress);
> {code}
> Some code statements there are really redundant and not needed, especialy with the delete(). But without deleting first, the overwrite takes a long time to reture.
> BTW, i will create another issue about the overwrite problem. If it is not a bug at all or a duplicate, someone please close it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-180-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 18:16:30 2009
Return-Path: <common-issues-return-180-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22518 invoked from network); 9 Jul 2009 18:16:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 18:16:30 -0000
Received: (qmail 25276 invoked by uid 500); 9 Jul 2009 18:16:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25225 invoked by uid 500); 9 Jul 2009 18:16:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25200 invoked by uid 99); 9 Jul 2009 18:16:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 18:16:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 18:16:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EF00B29A001F
	for <common-issues@hadoop.apache.org>; Thu,  9 Jul 2009 11:16:14 -0700 (PDT)
Message-ID: <45514930.1247163374978.JavaMail.jira@brutus>
Date: Thu, 9 Jul 2009 11:16:14 -0700 (PDT)
From: "Philip Zeyliger (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6125) extend DistributedCache to work
 locally (LocalJobRunner) (common half)
In-Reply-To: <682341596.1246484267288.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Philip Zeyliger updated HADOOP-6125:
------------------------------------

    Resolution: Won't Fix
        Status: Resolved  (was: Patch Available)

MAPREDUCE-711 moves the distributed cache to common, which makes separating the patch in MAPREDUCE-476 into two (this being one half of it) is no longer necessary.

> extend DistributedCache to work locally (LocalJobRunner) (common half)
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6125
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6125
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Philip Zeyliger
>            Assignee: Philip Zeyliger
>            Priority: Minor
>         Attachments: HADOOP-6125.patch
>
>
> This is the co-ticket to MAPREDUCE-476, covering the significant part of DistributedCache that's in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-182-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 18:32:27 2009
Return-Path: <common-issues-return-182-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43766 invoked from network); 9 Jul 2009 18:32:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 18:32:27 -0000
Received: (qmail 53890 invoked by uid 500); 9 Jul 2009 18:32:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53809 invoked by uid 500); 9 Jul 2009 18:32:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53794 invoked by uid 99); 9 Jul 2009 18:32:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 18:32:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 18:32:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id F3B2C29A0018
	for <common-issues@hadoop.apache.org>; Thu,  9 Jul 2009 11:32:14 -0700 (PDT)
Message-ID: <647266706.1247164334997.JavaMail.jira@brutus>
Date: Thu, 9 Jul 2009 11:32:14 -0700 (PDT)
From: "Yongqiang He (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5438) Merge FileSystem.create and
 FileSystem.append
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729372#action_12729372 ] 

Yongqiang He commented on HADOOP-5438:
--------------------------------------

np. Filed a jira HADOOP-6138.

> Merge FileSystem.create and FileSystem.append
> ---------------------------------------------
>
>                 Key: HADOOP-5438
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5438
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Yongqiang He
>            Assignee: Yongqiang He
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-5438(2009-04-06).patch, Hadoop-5438-2009-03-30.patch, Hadoop-5438-2009-03-31-2.patch, Hadoop-5438-2009-03-31.patch, Hadoop-5438-2009-05-10.patch, Hadoop-5438-2009-05-15.patch, Hadoop-5438-2009-05-19.patch, Hadoop-5438-2009-05-5.patch
>
>
> Currently, when a user wants to modify a file, the user first calls exists() to know if this file is already there. And then uses create() or append() according to whether the file exists or not.
> the code looks like:
> {code}
> FSDataOutputStream out_1 = null;
> if (fs.exists(path_1))
>    out_1 = fs.append(path_1);
> else
>    out_1 = fs.create(path_1);
> {code}
> . On the performace side,It involes two RPCs. On the easy-of-use side, it is not very convient in contrast to the traditional open interface.
> It will more complicate if there is a overwrite parameter specified. I donot know whether there is a bug about 'overwrite' in 0.19, some times it takes a long time for overwrite creates to reture. So i make the write file code with overwrite param works like:
> {code}
> boolean exists = fs.exists(name);
> if (overwrite) {
>     if (exists)
>        fs.delete(name, true);
>      this.out = fs.create(name, overwrite, bufferSize, replication,
> 				    blockSize, progress);
>      this.currentRowID = 0;
>  } else {
>    if (!exists)
> 	this.out = fs.create(name, overwrite, bufferSize,
> 					replication, blockSize, progress);
>    else
> 	this.out = fs.append(name, bufferSize, progress);
> {code}
> Some code statements there are really redundant and not needed, especialy with the delete(). But without deleting first, the overwrite takes a long time to reture.
> BTW, i will create another issue about the overwrite problem. If it is not a bug at all or a duplicate, someone please close it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-181-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 18:32:27 2009
Return-Path: <common-issues-return-181-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43760 invoked from network); 9 Jul 2009 18:32:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 18:32:27 -0000
Received: (qmail 53857 invoked by uid 500); 9 Jul 2009 18:32:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53807 invoked by uid 500); 9 Jul 2009 18:32:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53787 invoked by uid 99); 9 Jul 2009 18:32:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 18:32:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 18:32:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EC5B129A0012
	for <common-issues@hadoop.apache.org>; Thu,  9 Jul 2009 11:32:14 -0700 (PDT)
Message-ID: <883885751.1247164334967.JavaMail.jira@brutus>
Date: Thu, 9 Jul 2009 11:32:14 -0700 (PDT)
From: "Yongqiang He (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6138) eliminate the depracate warnings
 introduced by H-5438
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

eliminate the depracate warnings introduced by H-5438
-----------------------------------------------------

                 Key: HADOOP-6138
                 URL: https://issues.apache.org/jira/browse/HADOOP-6138
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Yongqiang He
            Priority: Minor




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-183-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 21:22:28 2009
Return-Path: <common-issues-return-183-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31480 invoked from network); 9 Jul 2009 21:22:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 21:22:27 -0000
Received: (qmail 92666 invoked by uid 500); 9 Jul 2009 21:22:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92621 invoked by uid 500); 9 Jul 2009 21:22:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92611 invoked by uid 99); 9 Jul 2009 21:22:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 21:22:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 21:22:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3C59929A0012
	for <common-issues@hadoop.apache.org>; Thu,  9 Jul 2009 14:22:15 -0700 (PDT)
Message-ID: <1592736719.1247174535246.JavaMail.jira@brutus>
Date: Thu, 9 Jul 2009 14:22:15 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley resolved HADOOP-5170.
-----------------------------------

    Resolution: Won't Fix

I reverted this. Matei may move this into the Fair Share Scheduler.

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: h5170.patch, HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-184-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 23:44:27 2009
Return-Path: <common-issues-return-184-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14714 invoked from network); 9 Jul 2009 23:44:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 23:44:27 -0000
Received: (qmail 61825 invoked by uid 500); 9 Jul 2009 23:44:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61788 invoked by uid 500); 9 Jul 2009 23:44:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61778 invoked by uid 99); 9 Jul 2009 23:44:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 23:44:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 23:44:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id F097C29A0024
	for <common-issues@hadoop.apache.org>; Thu,  9 Jul 2009 16:44:14 -0700 (PDT)
Message-ID: <1529884455.1247183054984.JavaMail.jira@brutus>
Date: Thu, 9 Jul 2009 16:44:14 -0700 (PDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6099) Allow configuring the IPC module to
 send pings
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-6099:
-------------------------------------

    Status: Patch Available  (was: Open)

> Allow configuring the IPC module to send pings
> ----------------------------------------------
>
>                 Key: HADOOP-6099
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6099
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>             Fix For: 0.21.0
>
>         Attachments: ipcPing.txt, ipcPing2.txt
>
>
> The IPC Client sets a socketTimeout for the time period specified by the pingInterval and then sends a ping every pingInterval. This means that if a RPC server does not respond to a RPC client, then the RPC client blocks forever. This is a problem for applications that wants to switch quickly from one un-responsive HDFS cluster  to a good one. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-185-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 23:44:29 2009
Return-Path: <common-issues-return-185-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14764 invoked from network); 9 Jul 2009 23:44:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 23:44:29 -0000
Received: (qmail 61908 invoked by uid 500); 9 Jul 2009 23:44:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61866 invoked by uid 500); 9 Jul 2009 23:44:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61848 invoked by uid 99); 9 Jul 2009 23:44:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 23:44:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 23:44:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EC23F29A0021
	for <common-issues@hadoop.apache.org>; Thu,  9 Jul 2009 16:44:14 -0700 (PDT)
Message-ID: <1935500505.1247183054966.JavaMail.jira@brutus>
Date: Thu, 9 Jul 2009 16:44:14 -0700 (PDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6099) Allow configuring the IPC module to
 send pings
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-6099:
-------------------------------------

    Status: Open  (was: Patch Available)

Retry test

> Allow configuring the IPC module to send pings
> ----------------------------------------------
>
>                 Key: HADOOP-6099
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6099
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>             Fix For: 0.21.0
>
>         Attachments: ipcPing.txt, ipcPing2.txt
>
>
> The IPC Client sets a socketTimeout for the time period specified by the pingInterval and then sends a ping every pingInterval. This means that if a RPC server does not respond to a RPC client, then the RPC client blocks forever. This is a problem for applications that wants to switch quickly from one un-responsive HDFS cluster  to a good one. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-186-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 23:54:29 2009
Return-Path: <common-issues-return-186-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17684 invoked from network); 9 Jul 2009 23:54:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 23:54:29 -0000
Received: (qmail 69390 invoked by uid 500); 9 Jul 2009 23:54:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69354 invoked by uid 500); 9 Jul 2009 23:54:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69344 invoked by uid 99); 9 Jul 2009 23:54:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 23:54:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 23:54:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CDF1F29A0011
	for <common-issues@hadoop.apache.org>; Thu,  9 Jul 2009 16:54:14 -0700 (PDT)
Message-ID: <2063384390.1247183654820.JavaMail.jira@brutus>
Date: Thu, 9 Jul 2009 16:54:14 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Moved: (HADOOP-6139) Incomplete help message is displayed
 for rm and rmr options.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley moved HDFS-478 to HADOOP-6139:
--------------------------------------------

          Component/s:     (was: hdfs client)
    Affects Version/s:     (was: 0.20.1)
                       0.20.1
                  Key: HADOOP-6139  (was: HDFS-478)
              Project: Hadoop Common  (was: Hadoop HDFS)

> Incomplete help message is displayed for rm and rmr options.
> ------------------------------------------------------------
>
>                 Key: HADOOP-6139
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6139
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.1
>            Reporter: Suman Sehgal
>            Assignee: Jakob Homan
>            Priority: Minor
>         Attachments: COMMON-478.patch, HADOOP-478-v20.patch
>
>
> If path/src is missing from rm or rmr option then help message is displayed for these options.
> On giving command "hadoop dfs -rm" 
> help message displayed is "Usage: java FsShell [-rm <path>]"
> while it should be "Usage: java FsShell [-rm [-skipTrash] <path>]"
> Same issue is there with rmr option also.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-187-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 23:56:29 2009
Return-Path: <common-issues-return-187-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20081 invoked from network); 9 Jul 2009 23:56:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 23:56:29 -0000
Received: (qmail 75096 invoked by uid 500); 9 Jul 2009 23:56:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75057 invoked by uid 500); 9 Jul 2009 23:56:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75047 invoked by uid 99); 9 Jul 2009 23:56:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 23:56:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 23:56:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4F84029A001B
	for <common-issues@hadoop.apache.org>; Thu,  9 Jul 2009 16:56:15 -0700 (PDT)
Message-ID: <616154213.1247183775324.JavaMail.jira@brutus>
Date: Thu, 9 Jul 2009 16:56:15 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6139) Incomplete help message is
 displayed for rm and rmr options.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729459#action_12729459 ] 

Jakob Homan commented on HADOOP-6139:
-------------------------------------

I had the issue moved to Common rather than HDFS since the affected file is FsShell.java, within the Common subproject.

> Incomplete help message is displayed for rm and rmr options.
> ------------------------------------------------------------
>
>                 Key: HADOOP-6139
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6139
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.1
>            Reporter: Suman Sehgal
>            Assignee: Jakob Homan
>            Priority: Minor
>         Attachments: COMMON-478.patch, HADOOP-478-v20.patch
>
>
> If path/src is missing from rm or rmr option then help message is displayed for these options.
> On giving command "hadoop dfs -rm" 
> help message displayed is "Usage: java FsShell [-rm <path>]"
> while it should be "Usage: java FsShell [-rm [-skipTrash] <path>]"
> Same issue is there with rmr option also.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-188-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 09 23:58:29 2009
Return-Path: <common-issues-return-188-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20330 invoked from network); 9 Jul 2009 23:58:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Jul 2009 23:58:29 -0000
Received: (qmail 75938 invoked by uid 500); 9 Jul 2009 23:58:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75899 invoked by uid 500); 9 Jul 2009 23:58:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75889 invoked by uid 99); 9 Jul 2009 23:58:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 23:58:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jul 2009 23:58:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C583429A0011
	for <common-issues@hadoop.apache.org>; Thu,  9 Jul 2009 16:58:14 -0700 (PDT)
Message-ID: <1502464703.1247183894797.JavaMail.jira@brutus>
Date: Thu, 9 Jul 2009 16:58:14 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6139) Incomplete help message is displayed
 for rm and rmr options.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6139:
--------------------------------

    Status: Patch Available  (was: Open)

submitting patch.

> Incomplete help message is displayed for rm and rmr options.
> ------------------------------------------------------------
>
>                 Key: HADOOP-6139
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6139
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.1
>            Reporter: Suman Sehgal
>            Assignee: Jakob Homan
>            Priority: Minor
>         Attachments: COMMON-478.patch, HADOOP-478-v20.patch
>
>
> If path/src is missing from rm or rmr option then help message is displayed for these options.
> On giving command "hadoop dfs -rm" 
> help message displayed is "Usage: java FsShell [-rm <path>]"
> while it should be "Usage: java FsShell [-rm [-skipTrash] <path>]"
> Same issue is there with rmr option also.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-189-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 00:04:27 2009
Return-Path: <common-issues-return-189-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24988 invoked from network); 10 Jul 2009 00:04:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 00:04:27 -0000
Received: (qmail 80443 invoked by uid 500); 10 Jul 2009 00:04:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80406 invoked by uid 500); 10 Jul 2009 00:04:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80396 invoked by uid 99); 10 Jul 2009 00:04:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 00:04:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 00:04:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E42DB29A001B
	for <common-issues@hadoop.apache.org>; Thu,  9 Jul 2009 17:04:14 -0700 (PDT)
Message-ID: <762001986.1247184254933.JavaMail.jira@brutus>
Date: Thu, 9 Jul 2009 17:04:14 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6139) Incomplete help message is
 displayed for rm and rmr options.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729464#action_12729464 ] 

Jakob Homan commented on HADOOP-6139:
-------------------------------------

Ran unit tests locally.  Passes all common unit tests.

> Incomplete help message is displayed for rm and rmr options.
> ------------------------------------------------------------
>
>                 Key: HADOOP-6139
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6139
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.1
>            Reporter: Suman Sehgal
>            Assignee: Jakob Homan
>            Priority: Minor
>         Attachments: COMMON-478.patch, HADOOP-478-v20.patch
>
>
> If path/src is missing from rm or rmr option then help message is displayed for these options.
> On giving command "hadoop dfs -rm" 
> help message displayed is "Usage: java FsShell [-rm <path>]"
> while it should be "Usage: java FsShell [-rm [-skipTrash] <path>]"
> Same issue is there with rmr option also.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-190-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 01:18:27 2009
Return-Path: <common-issues-return-190-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57872 invoked from network); 10 Jul 2009 01:18:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 01:18:27 -0000
Received: (qmail 29405 invoked by uid 500); 10 Jul 2009 01:18:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29367 invoked by uid 500); 10 Jul 2009 01:18:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29357 invoked by uid 99); 10 Jul 2009 01:18:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 01:18:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 01:18:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0E1C329A0011
	for <common-issues@hadoop.apache.org>; Thu,  9 Jul 2009 18:18:15 -0700 (PDT)
Message-ID: <202017664.1247188695056.JavaMail.jira@brutus>
Date: Thu, 9 Jul 2009 18:18:15 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6139) Incomplete help message is
 displayed for rm and rmr options.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729486#action_12729486 ] 

Jakob Homan commented on HADOOP-6139:
-------------------------------------

test-patch:
{noformat}
     [exec] -1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
     [exec]                         Please justify why no new tests are needed for this patch.
     [exec]                         Also please list what manual steps were performed to verify this patch.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.{noformat}
No unit tests because is documentation change.  Manual step as documented in jira description.  

> Incomplete help message is displayed for rm and rmr options.
> ------------------------------------------------------------
>
>                 Key: HADOOP-6139
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6139
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.1
>            Reporter: Suman Sehgal
>            Assignee: Jakob Homan
>            Priority: Minor
>         Attachments: COMMON-478.patch, HADOOP-478-v20.patch
>
>
> If path/src is missing from rm or rmr option then help message is displayed for these options.
> On giving command "hadoop dfs -rm" 
> help message displayed is "Usage: java FsShell [-rm <path>]"
> while it should be "Usage: java FsShell [-rm [-skipTrash] <path>]"
> Same issue is there with rmr option also.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-191-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 05:26:29 2009
Return-Path: <common-issues-return-191-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29597 invoked from network); 10 Jul 2009 05:26:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 05:26:29 -0000
Received: (qmail 63888 invoked by uid 500); 10 Jul 2009 05:26:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63851 invoked by uid 500); 10 Jul 2009 05:26:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63841 invoked by uid 99); 10 Jul 2009 05:26:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 05:26:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 05:26:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 070B8234C004
	for <common-issues@hadoop.apache.org>; Thu,  9 Jul 2009 22:26:15 -0700 (PDT)
Message-ID: <634215241.1247203575014.JavaMail.jira@brutus>
Date: Thu, 9 Jul 2009 22:26:15 -0700 (PDT)
From: "Sreekanth Ramakrishnan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5771) Create unit test for
 LinuxTaskController
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sreekanth Ramakrishnan updated HADOOP-5771:
-------------------------------------------

    Attachment: hadoop-5771-ydist.patch

Y! distribution patch.

> Create unit test for LinuxTaskController
> ----------------------------------------
>
>                 Key: HADOOP-5771
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5771
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sreekanth Ramakrishnan
>            Assignee: Sreekanth Ramakrishnan
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5771-1.patch, HADOOP-5771-2.patch, HADOOP-5771-3.patch, HADOOP-5771-4.txt, HADOOP-5771-5.patch, hadoop-5771-ydist.patch
>
>
> Add unit tests to test {{LinuxTaskController}} functionality introduced by HADOOP-4490

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-192-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 06:14:31 2009
Return-Path: <common-issues-return-192-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51608 invoked from network); 10 Jul 2009 06:14:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 06:14:30 -0000
Received: (qmail 6631 invoked by uid 500); 10 Jul 2009 06:14:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6594 invoked by uid 500); 10 Jul 2009 06:14:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6584 invoked by uid 99); 10 Jul 2009 06:14:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 06:14:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 06:14:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1C4BE29A001E
	for <common-issues@hadoop.apache.org>; Thu,  9 Jul 2009 23:14:15 -0700 (PDT)
Message-ID: <1336714069.1247206455114.JavaMail.jira@brutus>
Date: Thu, 9 Jul 2009 23:14:15 -0700 (PDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6099) Allow configuring the IPC module to
 send pings
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729537#action_12729537 ] 

dhruba borthakur commented on HADOOP-6099:
------------------------------------------

Hudson is stuck, so I ran the tests myself:

{quote}
run-test-core:
    [mkdir] Created dir: /mnt/vol/devrs004.snc1/dhruba/raincommon/build/test/data
    [mkdir] Created dir: /mnt/vol/devrs004.snc1/dhruba/raincommon/build/test/logs
     [copy] Copying 1 file to /mnt/vol/devrs004.snc1/dhruba/raincommon/build/test/extraconf
    [junit] Running org.apache.hadoop.cli.TestCLI
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.095 sec
    [junit] Running org.apache.hadoop.conf.TestConfiguration
    [junit] Tests run: 12, Failures: 0, Errors: 0, Time elapsed: 0.364 sec
    [junit] Running org.apache.hadoop.conf.TestConfigurationSubclass
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.192 sec
    [junit] Running org.apache.hadoop.conf.TestGetInstances
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.146 sec
    [junit] Running org.apache.hadoop.filecache.TestDistributedCache
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.39 sec
    [junit] Running org.apache.hadoop.fs.TestBlockLocation
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.095 sec
    [junit] Running org.apache.hadoop.fs.TestChecksumFileSystem
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.263 sec
    [junit] Running org.apache.hadoop.fs.TestDFVariations
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.192 sec
    [junit] Running org.apache.hadoop.fs.TestDU
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 5.125 sec
    [junit] Running org.apache.hadoop.fs.TestGetFileBlockLocations
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.45 sec
    [junit] Running org.apache.hadoop.fs.TestGlobExpander
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.018 sec
    [junit] Running org.apache.hadoop.fs.TestLocalDirAllocator
    [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 0.158 sec
    [junit] Running org.apache.hadoop.fs.TestLocalFileSystem
    [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.51 sec
    [junit] Running org.apache.hadoop.fs.TestLocalFileSystemPermission
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.415 sec
    [junit] Running org.apache.hadoop.fs.TestPath
    [junit] Tests run: 8, Failures: 0, Errors: 0, Time elapsed: 0.03 sec
    [junit] Running org.apache.hadoop.fs.TestTrash
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.511 sec
    [junit] Running org.apache.hadoop.fs.TestTruncatedInputBug
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.256 sec
    [junit] Running org.apache.hadoop.fs.kfs.TestKosmosFileSystem
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.348 sec
    [junit] Running org.apache.hadoop.fs.permission.TestFsPermission
    [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.036 sec
    [junit] Running org.apache.hadoop.fs.s3.TestINode
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.019 sec
    [junit] Running org.apache.hadoop.fs.s3.TestInMemoryS3FileSystemContract
    [junit] Tests run: 29, Failures: 0, Errors: 0, Time elapsed: 0.621 sec
    [junit] Running org.apache.hadoop.fs.s3.TestS3Credentials
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.096 sec
    [junit] Running org.apache.hadoop.fs.s3.TestS3FileSystem
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.112 sec
    [junit] Running org.apache.hadoop.fs.s3native.TestInMemoryNativeS3FileSystemContract
    [junit] Tests run: 35, Failures: 0, Errors: 0, Time elapsed: 0.655 sec
    [junit] Running org.apache.hadoop.http.TestGlobalFilter
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.704 sec
    [junit] Running org.apache.hadoop.http.TestServletFilter
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.729 sec
    [junit] Running org.apache.hadoop.io.TestArrayFile
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 3.057 sec
    [junit] Running org.apache.hadoop.io.TestArrayWritable
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.099 sec
    [junit] Running org.apache.hadoop.io.TestBloomMapFile
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.358 sec
    [junit] Running org.apache.hadoop.io.TestBytesWritable
    [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.081 sec
    [junit] Running org.apache.hadoop.io.TestDefaultStringifier
    [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.157 sec
    [junit] Running org.apache.hadoop.io.TestEnumSetWritable
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.093 sec
    [junit] Running org.apache.hadoop.io.TestGenericWritable
    [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 0.24 sec
    [junit] Running org.apache.hadoop.io.TestMD5Hash
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.03 sec
    [junit] Running org.apache.hadoop.io.TestMapFile
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.31 sec
    [junit] Running org.apache.hadoop.io.TestMapWritable
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.121 sec
    [junit] Running org.apache.hadoop.io.TestSequenceFileSerialization
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.312 sec
    [junit] Running org.apache.hadoop.io.TestSetFile
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 2.978 sec
    [junit] Running org.apache.hadoop.io.TestSortedMapWritable
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.107 sec
    [junit] Running org.apache.hadoop.io.TestText
    [junit] Tests run: 9, Failures: 0, Errors: 0, Time elapsed: 0.569 sec
    [junit] Running org.apache.hadoop.io.TestTextNonUTF8
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.019 sec
    [junit] Running org.apache.hadoop.io.TestUTF8
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.097 sec
    [junit] Running org.apache.hadoop.io.TestVersionedWritable
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.028 sec
    [junit] Running org.apache.hadoop.io.TestWritable
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.027 sec
    [junit] Running org.apache.hadoop.io.TestWritableName
    [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.098 sec
    [junit] Running org.apache.hadoop.io.TestWritableUtils
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.016 sec
    [junit] Running org.apache.hadoop.io.compress.TestCodec
    [junit] Tests run: 6, Failures: 0, Errors: 0, Time elapsed: 8.667 sec
    [junit] Running org.apache.hadoop.io.compress.TestCodecFactory
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.205 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestTFile
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 1.074 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestTFileByteArrays
    [junit] Tests run: 25, Failures: 0, Errors: 0, Time elapsed: 2.378 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestTFileComparators
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.261 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestTFileJClassComparatorByteArrays
    [junit] Tests run: 25, Failures: 0, Errors: 0, Time elapsed: 2.327 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestTFileLzoCodecsByteArrays
    [junit] Tests run: 25, Failures: 0, Errors: 0, Time elapsed: 0.069 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestTFileLzoCodecsStreams
    [junit] Tests run: 19, Failures: 0, Errors: 0, Time elapsed: 0.08 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestTFileNoneCodecsByteArrays
    [junit] Tests run: 25, Failures: 0, Errors: 0, Time elapsed: 1.154 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestTFileNoneCodecsJClassComparatorByteArrays
    [junit] Tests run: 25, Failures: 0, Errors: 0, Time elapsed: 1.137 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestTFileNoneCodecsStreams
    [junit] Tests run: 19, Failures: 0, Errors: 0, Time elapsed: 1.36 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestTFileSeek
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 3.536 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestTFileSeqFileComparison
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 10.179 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestTFileSplit
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 4.519 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestTFileStreams
    [junit] Tests run: 19, Failures: 0, Errors: 0, Time elapsed: 1.355 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestTFileUnsortedByteArrays
    [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.366 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestVLong
    [junit] Tests run: 9, Failures: 0, Errors: 0, Time elapsed: 1.749 sec
    [junit] Running org.apache.hadoop.io.retry.TestRetryProxy
    [junit] Tests run: 9, Failures: 0, Errors: 0, Time elapsed: 0.18 sec
    [junit] Running org.apache.hadoop.io.serializer.TestWritableSerialization
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.038 sec
    [junit] Running org.apache.hadoop.ipc.TestIPC
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 12.743 sec
    [junit] Running org.apache.hadoop.ipc.TestIPCServerResponder
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 11.831 sec
    [junit] Running org.apache.hadoop.ipc.TestRPC
    [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 32.536 sec
    [junit] Running org.apache.hadoop.log.TestLogLevel
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.675 sec
    [junit] Running org.apache.hadoop.metrics.TestMetricsServlet
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.048 sec
    [junit] Running org.apache.hadoop.metrics.spi.TestOutputRecord
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.017 sec
    [junit] Running org.apache.hadoop.net.TestDNS
    [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 0.156 sec
    [junit] Running org.apache.hadoop.net.TestScriptBasedMapping
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.388 sec
    [junit] Running org.apache.hadoop.net.TestSocketIOWithTimeout
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 3.032 sec
    [junit] Running org.apache.hadoop.record.TestBuffer
    [junit] Tests run: 6, Failures: 0, Errors: 0, Time elapsed: 0.018 sec
    [junit] Running org.apache.hadoop.record.TestRecordIO
    [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 0.073 sec
    [junit] Running org.apache.hadoop.record.TestRecordVersioning
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.052 sec
    [junit] Running org.apache.hadoop.security.TestAccessControlList
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.019 sec
    [junit] Running org.apache.hadoop.security.TestAccessToken
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.54 sec
    [junit] Running org.apache.hadoop.security.TestUnixUserGroupInformation
    [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.279 sec
    [junit] Running org.apache.hadoop.security.authorize.TestConfiguredPolicy
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.18 sec
    [junit] Running org.apache.hadoop.util.TestCyclicIteration
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.021 sec
    [junit] Running org.apache.hadoop.util.TestGenericsUtil
    [junit] Tests run: 6, Failures: 0, Errors: 0, Time elapsed: 0.116 sec
    [junit] Running org.apache.hadoop.util.TestIndexedSort
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.716 sec
    [junit] Running org.apache.hadoop.util.TestProcfsBasedProcessTree
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 7.54 sec
    [junit] Running org.apache.hadoop.util.TestShell
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 4.13 sec
    [junit] Running org.apache.hadoop.util.TestStringUtils
    [junit] Tests run: 6, Failures: 0, Errors: 0, Time elapsed: 0.025 sec

     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     -1 release audit.  The applied patch generated 265 release audit warnings (more than the trunk's current 263 warnings
{quote}



> Allow configuring the IPC module to send pings
> ----------------------------------------------
>
>                 Key: HADOOP-6099
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6099
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>             Fix For: 0.21.0
>
>         Attachments: ipcPing.txt, ipcPing2.txt
>
>
> The IPC Client sets a socketTimeout for the time period specified by the pingInterval and then sends a ping every pingInterval. This means that if a RPC server does not respond to a RPC client, then the RPC client blocks forever. This is a problem for applications that wants to switch quickly from one un-responsive HDFS cluster  to a good one. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-193-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 06:20:27 2009
Return-Path: <common-issues-return-193-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60645 invoked from network); 10 Jul 2009 06:20:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 06:20:27 -0000
Received: (qmail 9764 invoked by uid 500); 10 Jul 2009 06:20:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9726 invoked by uid 500); 10 Jul 2009 06:20:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9715 invoked by uid 99); 10 Jul 2009 06:20:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 06:20:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 06:20:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C775F234C004
	for <common-issues@hadoop.apache.org>; Thu,  9 Jul 2009 23:20:14 -0700 (PDT)
Message-ID: <1862263160.1247206814802.JavaMail.jira@brutus>
Date: Thu, 9 Jul 2009 23:20:14 -0700 (PDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6099) Allow configuring the IPC module to
 send pings
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-6099:
-------------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I just committed this. 

> Allow configuring the IPC module to send pings
> ----------------------------------------------
>
>                 Key: HADOOP-6099
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6099
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>             Fix For: 0.21.0
>
>         Attachments: ipcPing.txt, ipcPing2.txt
>
>
> The IPC Client sets a socketTimeout for the time period specified by the pingInterval and then sends a ping every pingInterval. This means that if a RPC server does not respond to a RPC client, then the RPC client blocks forever. This is a problem for applications that wants to switch quickly from one un-responsive HDFS cluster  to a good one. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-194-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 07:35:30 2009
Return-Path: <common-issues-return-194-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92687 invoked from network); 10 Jul 2009 07:35:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 07:35:30 -0000
Received: (qmail 83160 invoked by uid 500); 10 Jul 2009 07:35:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83122 invoked by uid 500); 10 Jul 2009 07:35:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83099 invoked by uid 99); 10 Jul 2009 07:35:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 07:35:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 07:35:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C6CC1234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 00:35:14 -0700 (PDT)
Message-ID: <149324871.1247211314802.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 00:35:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6132) RPC client opens an extra
 connection for VersionedProtocol
In-Reply-To: <840919907.1247085075163.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729558#action_12729558 ] 

Hadoop QA commented on HADOOP-6132:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12412921/2.patch
  against trunk revision 792812.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/561/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/561/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/561/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/561/console

This message is automatically generated.

> RPC client opens an extra connection for VersionedProtocol
> ----------------------------------------------------------
>
>                 Key: HADOOP-6132
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6132
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: 1.patch, 2.patch
>
>
> RPC client caches connections per protocol. However, since all of our real protocols are subclasses of VersionedProtocol, a bug in the implementation makes the client opens an extra connection just for the VersionedProtocol, which is not needed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-195-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 11:59:27 2009
Return-Path: <common-issues-return-195-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13008 invoked from network); 10 Jul 2009 11:59:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 11:59:27 -0000
Received: (qmail 10300 invoked by uid 500); 10 Jul 2009 11:59:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10258 invoked by uid 500); 10 Jul 2009 11:59:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10243 invoked by uid 99); 10 Jul 2009 11:59:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 11:59:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 11:59:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D2F2F234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 04:59:14 -0700 (PDT)
Message-ID: <1433088142.1247227154849.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 04:59:14 -0700 (PDT)
From: "Amar Kamat (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5436) job history directory grows without
 bound, locks up job tracker on new job submission
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729617#action_12729617 ] 

Amar Kamat commented on HADOOP-5436:
------------------------------------

Tim,
HADOOP-4372 removes  the job-dir searching for new jobs and MAPREDUCE-416 moves completed jobs to *done* folder hence keeping the job-history folder clean. MAPREDUCE-416 also introduced file-cache that caches the filenames that were used and hence makes the job-completion for recovered jobs faster. MAPREDUCE-11 intends to remove the searching completely. Still the locking is still an issue. Hopefully we will fix it sometime soon. MR doesnt depend on the files in the *done* folder and hence the *done* folder can be archived/cleanedup etc while the jobtracker is running. 

> job history directory grows without bound, locks up job tracker on new job submission
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5436
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5436
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.19.0
>            Reporter: Tim Williamson
>         Attachments: HADOOP-5436.patch
>
>
> An unpleasant surprise upgrading to 0.19: requests to jobtracker.jsp would take a long time or even time out whenever new jobs where submitted.  Investigation showed the call to JobInProgress.initTasks() was calling JobHistory.JobInfo.logSubmitted() which in turn was calling JobHistory.getJobHistoryFileName() which was pegging the CPU for a couple minutes.  Further investigation showed the were 200,000+ files in the job history folder -- and every submission was creating a FileStatus for them all, then applying a regular expression to just the name.  All this just on the off chance the job tracker had been restarted (see HADOOP-3245).  To make matters worse, these files cannot be safely deleted while the job tracker is running, as the disappearance of a history file at the wrong time causes a FileNotFoundException.
> So to summarize the issues:
> - having Hadoop default to storing all the history files in a single directory is a Bad Idea
> - doing expensive processing of every history file on every job submission is a Worse Idea
> - doing expensive processing of every history file on every job submission while holding a lock on the JobInProgress object and thereby blocking the jobtracker.jsp from rendering is a Terrible Idea (note: haven't confirmed this, but a cursory glance suggests that's what's going on)
> - not being able to clean up the mess without taking down the job tracker is just Unfortunate

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-196-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 12:27:27 2009
Return-Path: <common-issues-return-196-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65264 invoked from network); 10 Jul 2009 12:27:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 12:27:27 -0000
Received: (qmail 48690 invoked by uid 500); 10 Jul 2009 12:27:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48656 invoked by uid 500); 10 Jul 2009 12:27:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48646 invoked by uid 99); 10 Jul 2009 12:27:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 12:27:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 12:27:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C6D16234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 05:27:14 -0700 (PDT)
Message-ID: <996856429.1247228834807.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 05:27:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6099) Allow configuring the IPC module to
 send pings
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729620#action_12729620 ] 

Hadoop QA commented on HADOOP-6099:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12412894/ipcPing2.txt
  against trunk revision 792812.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/562/console

This message is automatically generated.

> Allow configuring the IPC module to send pings
> ----------------------------------------------
>
>                 Key: HADOOP-6099
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6099
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>             Fix For: 0.21.0
>
>         Attachments: ipcPing.txt, ipcPing2.txt
>
>
> The IPC Client sets a socketTimeout for the time period specified by the pingInterval and then sends a ping every pingInterval. This means that if a RPC server does not respond to a RPC client, then the RPC client blocks forever. This is a problem for applications that wants to switch quickly from one un-responsive HDFS cluster  to a good one. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-197-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 14:35:28 2009
Return-Path: <common-issues-return-197-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91067 invoked from network); 10 Jul 2009 14:35:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 14:35:28 -0000
Received: (qmail 53815 invoked by uid 500); 10 Jul 2009 14:35:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53757 invoked by uid 500); 10 Jul 2009 14:35:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53746 invoked by uid 99); 10 Jul 2009 14:35:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 14:35:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 14:35:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 323C7234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 07:35:15 -0700 (PDT)
Message-ID: <1699381653.1247236515194.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 07:35:15 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729650#action_12729650 ] 

Hudson commented on HADOOP-5170:
--------------------------------

Integrated in Hadoop-Common-trunk #22 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/22/])
    . Removed change log entry because  was reverted from
mapreduce.


> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: h5170.patch, HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-198-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 14:35:29 2009
Return-Path: <common-issues-return-198-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91177 invoked from network); 10 Jul 2009 14:35:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 14:35:29 -0000
Received: (qmail 54213 invoked by uid 500); 10 Jul 2009 14:35:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54179 invoked by uid 500); 10 Jul 2009 14:35:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54169 invoked by uid 99); 10 Jul 2009 14:35:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 14:35:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 14:35:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 6FBF4234C4B6
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 07:35:15 -0700 (PDT)
Message-ID: <806891496.1247236515456.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 07:35:15 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6099) Allow configuring the IPC module to
 send pings
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729651#action_12729651 ] 

Hudson commented on HADOOP-6099:
--------------------------------

Integrated in Hadoop-Common-trunk #22 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/22/])
    . The RPC module can be configured to not send period pings.
The default behaviour remains unchanged. (dhruba)


> Allow configuring the IPC module to send pings
> ----------------------------------------------
>
>                 Key: HADOOP-6099
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6099
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>             Fix For: 0.21.0
>
>         Attachments: ipcPing.txt, ipcPing2.txt
>
>
> The IPC Client sets a socketTimeout for the time period specified by the pingInterval and then sends a ping every pingInterval. This means that if a RPC server does not respond to a RPC client, then the RPC client blocks forever. This is a problem for applications that wants to switch quickly from one un-responsive HDFS cluster  to a good one. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-199-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 16:09:27 2009
Return-Path: <common-issues-return-199-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46016 invoked from network); 10 Jul 2009 16:09:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 16:09:27 -0000
Received: (qmail 39318 invoked by uid 500); 10 Jul 2009 16:09:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39299 invoked by uid 500); 10 Jul 2009 16:09:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39289 invoked by uid 99); 10 Jul 2009 16:09:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 16:09:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 16:09:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CC373234C044
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 09:09:14 -0700 (PDT)
Message-ID: <221985591.1247242154831.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 09:09:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6139) Incomplete help message is
 displayed for rm and rmr options.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729696#action_12729696 ] 

Hadoop QA commented on HADOOP-6139:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12413065/HADOOP-478-v20.patch
  against trunk revision 792812.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 patch.  The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/563/console

This message is automatically generated.

> Incomplete help message is displayed for rm and rmr options.
> ------------------------------------------------------------
>
>                 Key: HADOOP-6139
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6139
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.1
>            Reporter: Suman Sehgal
>            Assignee: Jakob Homan
>            Priority: Minor
>         Attachments: COMMON-478.patch, HADOOP-478-v20.patch
>
>
> If path/src is missing from rm or rmr option then help message is displayed for these options.
> On giving command "hadoop dfs -rm" 
> help message displayed is "Usage: java FsShell [-rm <path>]"
> while it should be "Usage: java FsShell [-rm [-skipTrash] <path>]"
> Same issue is there with rmr option also.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-200-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 17:11:29 2009
Return-Path: <common-issues-return-200-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67496 invoked from network); 10 Jul 2009 17:11:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 17:11:29 -0000
Received: (qmail 47976 invoked by uid 500); 10 Jul 2009 17:11:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47792 invoked by uid 500); 10 Jul 2009 17:11:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47765 invoked by uid 99); 10 Jul 2009 17:11:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 17:11:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 17:11:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0B445234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 10:11:15 -0700 (PDT)
Message-ID: <1393504557.1247245875035.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 10:11:15 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6139) Incomplete help message is displayed
 for rm and rmr options.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6139:
-------------------------------------------

    Hadoop Flags: [Reviewed]

+1 patch looks good.

> Incomplete help message is displayed for rm and rmr options.
> ------------------------------------------------------------
>
>                 Key: HADOOP-6139
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6139
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.1
>            Reporter: Suman Sehgal
>            Assignee: Jakob Homan
>            Priority: Minor
>         Attachments: COMMON-478.patch, HADOOP-478-v20.patch
>
>
> If path/src is missing from rm or rmr option then help message is displayed for these options.
> On giving command "hadoop dfs -rm" 
> help message displayed is "Usage: java FsShell [-rm <path>]"
> while it should be "Usage: java FsShell [-rm [-skipTrash] <path>]"
> Same issue is there with rmr option also.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-201-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 17:11:29 2009
Return-Path: <common-issues-return-201-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67509 invoked from network); 10 Jul 2009 17:11:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 17:11:29 -0000
Received: (qmail 48003 invoked by uid 500); 10 Jul 2009 17:11:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47904 invoked by uid 500); 10 Jul 2009 17:11:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47768 invoked by uid 99); 10 Jul 2009 17:11:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 17:11:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 17:11:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1A70A234C1EE
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 10:11:15 -0700 (PDT)
Message-ID: <1165046874.1247245875107.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 10:11:15 -0700 (PDT)
From: "Vladimir Klimontovich (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6140) addArchiveToClassPath doesn't work in
 0.18.x branch
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

addArchiveToClassPath doesn't work in 0.18.x branch
---------------------------------------------------

                 Key: HADOOP-6140
                 URL: https://issues.apache.org/jira/browse/HADOOP-6140
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
    Affects Versions: 0.18.3
            Reporter: Vladimir Klimontovich
            Priority: Minor


addArchiveToClassPath is a method of DistributedCache class. It should be called before running a task. It accepts path to a jar file on a DFS. After it
this method should put this jar file on sitribuuted cache and than add this file to classpath to each map/reduce process on job tracker.

This method didn't work.

Bug 1:

addArchiveToClassPath adds DFS-path to archive to mapred.job.classpath.archives property. It uses System.getProperty("path.separator") as delimiter of multiple path.

getFileClassPaths that is called from TaskRunner uses splits mapred.job.classpath.archives using System.getProperty("path.separator").

In unix systems System.getProperty("path.separator") equals to ":". DFS-path urls is hdfs://host:port/path. It means that a result of split will be
[ hdfs,//host,port/path].

Suggested solution: use "," instead of  

Bug 2:

in TaskRunner there is an algorithm that looks for correspondence between DFS paths and local paths in distributed cache. 
It compares

   if (archives[i].getPath().equals(
                                               archiveClasspaths[j].toString())){

instead of

    if (archives[i].toString().equals(
                                               archiveClasspaths[j].toString())) 


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-202-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 17:13:30 2009
Return-Path: <common-issues-return-202-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68351 invoked from network); 10 Jul 2009 17:13:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 17:13:29 -0000
Received: (qmail 51539 invoked by uid 500); 10 Jul 2009 17:13:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51509 invoked by uid 500); 10 Jul 2009 17:13:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51499 invoked by uid 99); 10 Jul 2009 17:13:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 17:13:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 17:13:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C8C2A234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 10:13:14 -0700 (PDT)
Message-ID: <1593619452.1247245994817.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 10:13:14 -0700 (PDT)
From: "Vladimir Klimontovich (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6140) addArchiveToClassPath doesn't work in
 0.18.x branch
In-Reply-To: <1165046874.1247245875107.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vladimir Klimontovich updated HADOOP-6140:
------------------------------------------

    Attachment: HADOOP-6140.patch

This patch should fix this bug.

> addArchiveToClassPath doesn't work in 0.18.x branch
> ---------------------------------------------------
>
>                 Key: HADOOP-6140
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6140
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.18.3
>            Reporter: Vladimir Klimontovich
>            Priority: Minor
>         Attachments: HADOOP-6140.patch
>
>
> addArchiveToClassPath is a method of DistributedCache class. It should be called before running a task. It accepts path to a jar file on a DFS. After it
> this method should put this jar file on sitribuuted cache and than add this file to classpath to each map/reduce process on job tracker.
> This method didn't work.
> Bug 1:
> addArchiveToClassPath adds DFS-path to archive to mapred.job.classpath.archives property. It uses System.getProperty("path.separator") as delimiter of multiple path.
> getFileClassPaths that is called from TaskRunner uses splits mapred.job.classpath.archives using System.getProperty("path.separator").
> In unix systems System.getProperty("path.separator") equals to ":". DFS-path urls is hdfs://host:port/path. It means that a result of split will be
> [ hdfs,//host,port/path].
> Suggested solution: use "," instead of  
> Bug 2:
> in TaskRunner there is an algorithm that looks for correspondence between DFS paths and local paths in distributed cache. 
> It compares
>    if (archives[i].getPath().equals(
>                                                archiveClasspaths[j].toString())){
> instead of
>     if (archives[i].toString().equals(
>                                                archiveClasspaths[j].toString())) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-203-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 17:35:34 2009
Return-Path: <common-issues-return-203-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81572 invoked from network); 10 Jul 2009 17:35:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 17:35:34 -0000
Received: (qmail 88161 invoked by uid 500); 10 Jul 2009 17:35:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88131 invoked by uid 500); 10 Jul 2009 17:35:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88121 invoked by uid 99); 10 Jul 2009 17:35:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 17:35:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 17:35:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CE933234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 10:35:14 -0700 (PDT)
Message-ID: <1479294126.1247247314831.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 10:35:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6124) patchJavacWarnings and
 trunkJavacWarnings are not consistent.
In-Reply-To: <1339799848.1246409207795.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6124:
-------------------------------------------

    Priority: Critical  (was: Major)

Hope this can be fixed soon.  Otherwise, patches with javac warnings may get in without being noticed.  Changed the priority to "critical".

> patchJavacWarnings and trunkJavacWarnings are not consistent.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6124
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6124
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Giridharan Kesavan
>            Priority: Critical
>         Attachments: c6124_20090701.patch
>
>
> The values of patchJavacWarnings and trunkJavacWarnings are not consistent when running test-patch.sh with an empty patch over Common.  HDFS and MapReduce seem not having this problem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-204-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 17:37:27 2009
Return-Path: <common-issues-return-204-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82386 invoked from network); 10 Jul 2009 17:37:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 17:37:27 -0000
Received: (qmail 88725 invoked by uid 500); 10 Jul 2009 17:37:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88687 invoked by uid 500); 10 Jul 2009 17:37:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88677 invoked by uid 99); 10 Jul 2009 17:37:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 17:37:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 17:37:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C9B3C234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 10:37:14 -0700 (PDT)
Message-ID: <1522348079.1247247434811.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 10:37:14 -0700 (PDT)
From: "Vladimir Klimontovich (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6140)
 DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
In-Reply-To: <1165046874.1247245875107.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vladimir Klimontovich updated HADOOP-6140:
------------------------------------------

    Priority: Major  (was: Minor)
     Summary: DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch  (was: addArchiveToClassPath doesn't work in 0.18.x branch)

> DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6140
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6140
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.18.3
>            Reporter: Vladimir Klimontovich
>         Attachments: HADOOP-6140.patch
>
>
> addArchiveToClassPath is a method of DistributedCache class. It should be called before running a task. It accepts path to a jar file on a DFS. After it
> this method should put this jar file on sitribuuted cache and than add this file to classpath to each map/reduce process on job tracker.
> This method didn't work.
> Bug 1:
> addArchiveToClassPath adds DFS-path to archive to mapred.job.classpath.archives property. It uses System.getProperty("path.separator") as delimiter of multiple path.
> getFileClassPaths that is called from TaskRunner uses splits mapred.job.classpath.archives using System.getProperty("path.separator").
> In unix systems System.getProperty("path.separator") equals to ":". DFS-path urls is hdfs://host:port/path. It means that a result of split will be
> [ hdfs,//host,port/path].
> Suggested solution: use "," instead of  
> Bug 2:
> in TaskRunner there is an algorithm that looks for correspondence between DFS paths and local paths in distributed cache. 
> It compares
>    if (archives[i].getPath().equals(
>                                                archiveClasspaths[j].toString())){
> instead of
>     if (archives[i].toString().equals(
>                                                archiveClasspaths[j].toString())) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-205-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 17:37:29 2009
Return-Path: <common-issues-return-205-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82444 invoked from network); 10 Jul 2009 17:37:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 17:37:29 -0000
Received: (qmail 88805 invoked by uid 500); 10 Jul 2009 17:37:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88763 invoked by uid 500); 10 Jul 2009 17:37:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88753 invoked by uid 99); 10 Jul 2009 17:37:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 17:37:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 17:37:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D8F8C234C052
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 10:37:14 -0700 (PDT)
Message-ID: <572861873.1247247434887.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 10:37:14 -0700 (PDT)
From: "Vladimir Klimontovich (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6140)
 DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
In-Reply-To: <1165046874.1247245875107.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vladimir Klimontovich updated HADOOP-6140:
------------------------------------------

    Tags: DistributedCache

> DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6140
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6140
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.18.3
>            Reporter: Vladimir Klimontovich
>         Attachments: HADOOP-6140.patch
>
>
> addArchiveToClassPath is a method of DistributedCache class. It should be called before running a task. It accepts path to a jar file on a DFS. After it
> this method should put this jar file on sitribuuted cache and than add this file to classpath to each map/reduce process on job tracker.
> This method didn't work.
> Bug 1:
> addArchiveToClassPath adds DFS-path to archive to mapred.job.classpath.archives property. It uses System.getProperty("path.separator") as delimiter of multiple path.
> getFileClassPaths that is called from TaskRunner uses splits mapred.job.classpath.archives using System.getProperty("path.separator").
> In unix systems System.getProperty("path.separator") equals to ":". DFS-path urls is hdfs://host:port/path. It means that a result of split will be
> [ hdfs,//host,port/path].
> Suggested solution: use "," instead of  
> Bug 2:
> in TaskRunner there is an algorithm that looks for correspondence between DFS paths and local paths in distributed cache. 
> It compares
>    if (archives[i].getPath().equals(
>                                                archiveClasspaths[j].toString())){
> instead of
>     if (archives[i].toString().equals(
>                                                archiveClasspaths[j].toString())) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-206-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 18:45:27 2009
Return-Path: <common-issues-return-206-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25830 invoked from network); 10 Jul 2009 18:45:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 18:45:27 -0000
Received: (qmail 95458 invoked by uid 500); 10 Jul 2009 18:45:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95411 invoked by uid 500); 10 Jul 2009 18:45:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95396 invoked by uid 99); 10 Jul 2009 18:45:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 18:45:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 18:45:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1389C234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 11:45:15 -0700 (PDT)
Message-ID: <237212183.1247251515075.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 11:45:15 -0700 (PDT)
From: "Yuri Pradkin (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4012) Providing splitting support for
 bzip2 compressed files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729764#action_12729764 ] 

Yuri Pradkin commented on HADOOP-4012:
--------------------------------------

Abdul, I think at this point it's OK to assume that Chris has accepted your explanation.  Can you please post your latest patch with all the fixes.  Beware that repository has changed: you'll need to do switch to
http://svn.apache.org/repos/asf/hadoop/common/trunk/ or checkout a fresh copy and reapply your patch.

Guys, can we please wrap this up?  It's taking way too long for this patch to make it.  Please.  Thanks.

> Providing splitting support for bzip2 compressed files
> ------------------------------------------------------
>
>                 Key: HADOOP-4012
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4012
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Abdul Qadeer
>            Assignee: Abdul Qadeer
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-4012-version1.patch, Hadoop-4012-version2.patch, Hadoop-4012-version3.patch, Hadoop-4012-version4.patch, Hadoop-4012-version5.patch, Hadoop-4012-version6.patch, Hadoop-4012-version7.patch, Hadoop-4012-version8.patch, Hadoop-4012-version9.patch
>
>
> Hadoop assumes that if the input data is compressed, it can not be split (mainly due to the limitation of many codecs that they need the whole input stream to decompress successfully).  So in such a case, Hadoop prepares only one split per compressed file, where the lower split limit is at 0 while the upper limit is the end of the file.  The consequence of this decision is that, one compress file goes to a single mapper. Although it circumvents the limitation of codecs (as mentioned above) but reduces the parallelism substantially, as it was possible otherwise in case of splitting.
> BZip2 is a compression / De-Compression algorithm which does compression on blocks of data and later these compressed blocks can be decompressed independent of each other.  This is indeed an opportunity that instead of one BZip2 compressed file going to one mapper, we can process chunks of file in parallel.  The correctness criteria of such a processing is that for a bzip2 compressed file, each compressed block should be processed by only one mapper and ultimately all the blocks of the file should be processed.  (By processing we mean the actual utilization of that un-compressed data (coming out of the codecs) in a mapper).
> We are writing the code to implement this suggested functionality.  Although we have used bzip2 as an example, but we have tried to extend Hadoop's compression interfaces so that any other codecs with the same capability as that of bzip2, could easily use the splitting support.  The details of these changes will be posted when we submit the code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-207-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 19:09:29 2009
Return-Path: <common-issues-return-207-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38000 invoked from network); 10 Jul 2009 19:09:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 19:09:29 -0000
Received: (qmail 44277 invoked by uid 500); 10 Jul 2009 19:09:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44173 invoked by uid 500); 10 Jul 2009 19:09:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44163 invoked by uid 99); 10 Jul 2009 19:09:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 19:09:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 19:09:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DBC0A234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 12:09:14 -0700 (PDT)
Message-ID: <2112632214.1247252954885.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 12:09:14 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6139) Incomplete help message is
 displayed for rm and rmr options.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729776#action_12729776 ] 

Jakob Homan commented on HADOOP-6139:
-------------------------------------

Hudson tried to apply v20 patch, which is why I ran test-patch manually.  Both pathces apply fine.  Patch is ready to be committed.

> Incomplete help message is displayed for rm and rmr options.
> ------------------------------------------------------------
>
>                 Key: HADOOP-6139
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6139
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.1
>            Reporter: Suman Sehgal
>            Assignee: Jakob Homan
>            Priority: Minor
>         Attachments: COMMON-478.patch, HADOOP-478-v20.patch
>
>
> If path/src is missing from rm or rmr option then help message is displayed for these options.
> On giving command "hadoop dfs -rm" 
> help message displayed is "Usage: java FsShell [-rm <path>]"
> while it should be "Usage: java FsShell [-rm [-skipTrash] <path>]"
> Same issue is there with rmr option also.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-208-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 19:51:27 2009
Return-Path: <common-issues-return-208-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54621 invoked from network); 10 Jul 2009 19:51:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 19:51:27 -0000
Received: (qmail 96385 invoked by uid 500); 10 Jul 2009 19:51:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96343 invoked by uid 500); 10 Jul 2009 19:51:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96333 invoked by uid 99); 10 Jul 2009 19:51:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 19:51:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 19:51:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C5C9F234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 12:51:14 -0700 (PDT)
Message-ID: <1150437269.1247255474795.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 12:51:14 -0700 (PDT)
From: "Vladimir Klimontovich (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6140)
 DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
In-Reply-To: <1165046874.1247245875107.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vladimir Klimontovich updated HADOOP-6140:
------------------------------------------

    Status: Patch Available  (was: Open)

> DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6140
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6140
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.18.3
>            Reporter: Vladimir Klimontovich
>         Attachments: HADOOP-6140.patch
>
>
> addArchiveToClassPath is a method of DistributedCache class. It should be called before running a task. It accepts path to a jar file on a DFS. After it
> this method should put this jar file on sitribuuted cache and than add this file to classpath to each map/reduce process on job tracker.
> This method didn't work.
> Bug 1:
> addArchiveToClassPath adds DFS-path to archive to mapred.job.classpath.archives property. It uses System.getProperty("path.separator") as delimiter of multiple path.
> getFileClassPaths that is called from TaskRunner uses splits mapred.job.classpath.archives using System.getProperty("path.separator").
> In unix systems System.getProperty("path.separator") equals to ":". DFS-path urls is hdfs://host:port/path. It means that a result of split will be
> [ hdfs,//host,port/path].
> Suggested solution: use "," instead of  
> Bug 2:
> in TaskRunner there is an algorithm that looks for correspondence between DFS paths and local paths in distributed cache. 
> It compares
>    if (archives[i].getPath().equals(
>                                                archiveClasspaths[j].toString())){
> instead of
>     if (archives[i].toString().equals(
>                                                archiveClasspaths[j].toString())) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-209-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 20:29:29 2009
Return-Path: <common-issues-return-209-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66248 invoked from network); 10 Jul 2009 20:29:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 20:29:29 -0000
Received: (qmail 32868 invoked by uid 500); 10 Jul 2009 20:29:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32831 invoked by uid 500); 10 Jul 2009 20:29:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32821 invoked by uid 99); 10 Jul 2009 20:29:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 20:29:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 20:29:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D9908234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 13:29:14 -0700 (PDT)
Message-ID: <566346492.1247257754876.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 13:29:14 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4012) Providing splitting support for
 bzip2 compressed files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729801#action_12729801 ] 

Owen O'Malley commented on HADOOP-4012:
---------------------------------------

-1 to changing LineRecordReader. In particular, you've undone changes that were made by other jiras. This is very very touchy code that the current version of this patch breaks. 

I really think that this is better done by using a separate BzipTextInputFormat and BzipLineRecordReader.

> Providing splitting support for bzip2 compressed files
> ------------------------------------------------------
>
>                 Key: HADOOP-4012
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4012
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Abdul Qadeer
>            Assignee: Abdul Qadeer
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-4012-version1.patch, Hadoop-4012-version2.patch, Hadoop-4012-version3.patch, Hadoop-4012-version4.patch, Hadoop-4012-version5.patch, Hadoop-4012-version6.patch, Hadoop-4012-version7.patch, Hadoop-4012-version8.patch, Hadoop-4012-version9.patch
>
>
> Hadoop assumes that if the input data is compressed, it can not be split (mainly due to the limitation of many codecs that they need the whole input stream to decompress successfully).  So in such a case, Hadoop prepares only one split per compressed file, where the lower split limit is at 0 while the upper limit is the end of the file.  The consequence of this decision is that, one compress file goes to a single mapper. Although it circumvents the limitation of codecs (as mentioned above) but reduces the parallelism substantially, as it was possible otherwise in case of splitting.
> BZip2 is a compression / De-Compression algorithm which does compression on blocks of data and later these compressed blocks can be decompressed independent of each other.  This is indeed an opportunity that instead of one BZip2 compressed file going to one mapper, we can process chunks of file in parallel.  The correctness criteria of such a processing is that for a bzip2 compressed file, each compressed block should be processed by only one mapper and ultimately all the blocks of the file should be processed.  (By processing we mean the actual utilization of that un-compressed data (coming out of the codecs) in a mapper).
> We are writing the code to implement this suggested functionality.  Although we have used bzip2 as an example, but we have tried to extend Hadoop's compression interfaces so that any other codecs with the same capability as that of bzip2, could easily use the splitting support.  The details of these changes will be posted when we submit the code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-210-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 20:31:27 2009
Return-Path: <common-issues-return-210-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67013 invoked from network); 10 Jul 2009 20:31:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 20:31:27 -0000
Received: (qmail 39148 invoked by uid 500); 10 Jul 2009 20:31:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39113 invoked by uid 500); 10 Jul 2009 20:31:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39103 invoked by uid 99); 10 Jul 2009 20:31:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 20:31:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 20:31:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D2988234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 13:31:14 -0700 (PDT)
Message-ID: <640352979.1247257874829.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 13:31:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6139) Incomplete help message is displayed
 for rm and rmr options.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6139:
-------------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.21.0
                   0.20.1
           Status: Resolved  (was: Patch Available)

I have committed this.  Thanks, Jakob!

> Incomplete help message is displayed for rm and rmr options.
> ------------------------------------------------------------
>
>                 Key: HADOOP-6139
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6139
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.1
>            Reporter: Suman Sehgal
>            Assignee: Jakob Homan
>            Priority: Minor
>             Fix For: 0.20.1, 0.21.0
>
>         Attachments: COMMON-478.patch, HADOOP-478-v20.patch
>
>
> If path/src is missing from rm or rmr option then help message is displayed for these options.
> On giving command "hadoop dfs -rm" 
> help message displayed is "Usage: java FsShell [-rm <path>]"
> while it should be "Usage: java FsShell [-rm [-skipTrash] <path>]"
> Same issue is there with rmr option also.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-211-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 20:51:30 2009
Return-Path: <common-issues-return-211-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70369 invoked from network); 10 Jul 2009 20:51:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 20:51:29 -0000
Received: (qmail 66947 invoked by uid 500); 10 Jul 2009 20:51:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66907 invoked by uid 500); 10 Jul 2009 20:51:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66897 invoked by uid 99); 10 Jul 2009 20:51:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 20:51:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 20:51:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EDCFA234C1F1
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 13:51:14 -0700 (PDT)
Message-ID: <2029229069.1247259074973.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 13:51:14 -0700 (PDT)
From: "Philip Zeyliger (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6140)
 DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
In-Reply-To: <1165046874.1247245875107.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729812#action_12729812 ] 

Philip Zeyliger commented on HADOOP-6140:
-----------------------------------------

Vladimir,

Patch looks good.  It would be nice to have a test for (2).  It may also be appropriate to add an exception if someone passes in a filename with a comma.

-- Philip

> DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6140
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6140
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.18.3
>            Reporter: Vladimir Klimontovich
>         Attachments: HADOOP-6140.patch
>
>
> addArchiveToClassPath is a method of DistributedCache class. It should be called before running a task. It accepts path to a jar file on a DFS. After it
> this method should put this jar file on sitribuuted cache and than add this file to classpath to each map/reduce process on job tracker.
> This method didn't work.
> Bug 1:
> addArchiveToClassPath adds DFS-path to archive to mapred.job.classpath.archives property. It uses System.getProperty("path.separator") as delimiter of multiple path.
> getFileClassPaths that is called from TaskRunner uses splits mapred.job.classpath.archives using System.getProperty("path.separator").
> In unix systems System.getProperty("path.separator") equals to ":". DFS-path urls is hdfs://host:port/path. It means that a result of split will be
> [ hdfs,//host,port/path].
> Suggested solution: use "," instead of  
> Bug 2:
> in TaskRunner there is an algorithm that looks for correspondence between DFS paths and local paths in distributed cache. 
> It compares
>    if (archives[i].getPath().equals(
>                                                archiveClasspaths[j].toString())){
> instead of
>     if (archives[i].toString().equals(
>                                                archiveClasspaths[j].toString())) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-212-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 20:53:30 2009
Return-Path: <common-issues-return-212-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70779 invoked from network); 10 Jul 2009 20:53:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 20:53:29 -0000
Received: (qmail 70162 invoked by uid 500); 10 Jul 2009 20:53:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70122 invoked by uid 500); 10 Jul 2009 20:53:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70112 invoked by uid 99); 10 Jul 2009 20:53:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 20:53:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 20:53:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C3291234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 13:53:14 -0700 (PDT)
Message-ID: <1605722789.1247259194798.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 13:53:14 -0700 (PDT)
From: "Vladimir Klimontovich (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6140)
 DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
In-Reply-To: <1165046874.1247245875107.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729813#action_12729813 ] 

Vladimir Klimontovich commented on HADOOP-6140:
-----------------------------------------------

Ok, will make the test.

> DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6140
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6140
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.18.3
>            Reporter: Vladimir Klimontovich
>         Attachments: HADOOP-6140.patch
>
>
> addArchiveToClassPath is a method of DistributedCache class. It should be called before running a task. It accepts path to a jar file on a DFS. After it
> this method should put this jar file on sitribuuted cache and than add this file to classpath to each map/reduce process on job tracker.
> This method didn't work.
> Bug 1:
> addArchiveToClassPath adds DFS-path to archive to mapred.job.classpath.archives property. It uses System.getProperty("path.separator") as delimiter of multiple path.
> getFileClassPaths that is called from TaskRunner uses splits mapred.job.classpath.archives using System.getProperty("path.separator").
> In unix systems System.getProperty("path.separator") equals to ":". DFS-path urls is hdfs://host:port/path. It means that a result of split will be
> [ hdfs,//host,port/path].
> Suggested solution: use "," instead of  
> Bug 2:
> in TaskRunner there is an algorithm that looks for correspondence between DFS paths and local paths in distributed cache. 
> It compares
>    if (archives[i].getPath().equals(
>                                                archiveClasspaths[j].toString())){
> instead of
>     if (archives[i].toString().equals(
>                                                archiveClasspaths[j].toString())) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-213-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 20:57:27 2009
Return-Path: <common-issues-return-213-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71555 invoked from network); 10 Jul 2009 20:57:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 20:57:27 -0000
Received: (qmail 77461 invoked by uid 500); 10 Jul 2009 20:57:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77424 invoked by uid 500); 10 Jul 2009 20:57:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77414 invoked by uid 99); 10 Jul 2009 20:57:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 20:57:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 20:57:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EAAD7234C052
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 13:57:14 -0700 (PDT)
Message-ID: <912077286.1247259434960.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 13:57:14 -0700 (PDT)
From: "Yuri Pradkin (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4012) Providing splitting support for
 bzip2 compressed files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729814#action_12729814 ] 

Yuri Pradkin commented on HADOOP-4012:
--------------------------------------

bq. -1 to changing LineRecordReader. In particular, you've undone changes that were made by other jiras. This is very very touchy code that the current version of this patch breaks.

Can you please be more specific: what changes were undone?  Can you please elaborate on what exactly this patch breaks (perhaps we need more tests?)?

The code indeed is very heavily used, but I think it's also very well beaten upon by various tests.


bq. I really think that this is better done by using a separate BzipTextInputFormat and BzipLineRecordReader.

I think the point of having splitting built in - is that all readers/formats can avoid re-implementing common things.  We are currently using a binary format reader that works just fine with this patch with only minor changes.  Moreover, the idea is to work out  a common framework where other block-compressed formats could be processed in a similar manner.  The alternative that you're suggesting is to have BzipTextInputFormat, XXXBlockCompressInputFormat, YYYBlockCompressInputFormat, and so on.



> Providing splitting support for bzip2 compressed files
> ------------------------------------------------------
>
>                 Key: HADOOP-4012
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4012
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Abdul Qadeer
>            Assignee: Abdul Qadeer
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-4012-version1.patch, Hadoop-4012-version2.patch, Hadoop-4012-version3.patch, Hadoop-4012-version4.patch, Hadoop-4012-version5.patch, Hadoop-4012-version6.patch, Hadoop-4012-version7.patch, Hadoop-4012-version8.patch, Hadoop-4012-version9.patch
>
>
> Hadoop assumes that if the input data is compressed, it can not be split (mainly due to the limitation of many codecs that they need the whole input stream to decompress successfully).  So in such a case, Hadoop prepares only one split per compressed file, where the lower split limit is at 0 while the upper limit is the end of the file.  The consequence of this decision is that, one compress file goes to a single mapper. Although it circumvents the limitation of codecs (as mentioned above) but reduces the parallelism substantially, as it was possible otherwise in case of splitting.
> BZip2 is a compression / De-Compression algorithm which does compression on blocks of data and later these compressed blocks can be decompressed independent of each other.  This is indeed an opportunity that instead of one BZip2 compressed file going to one mapper, we can process chunks of file in parallel.  The correctness criteria of such a processing is that for a bzip2 compressed file, each compressed block should be processed by only one mapper and ultimately all the blocks of the file should be processed.  (By processing we mean the actual utilization of that un-compressed data (coming out of the codecs) in a mapper).
> We are writing the code to implement this suggested functionality.  Although we have used bzip2 as an example, but we have tried to extend Hadoop's compression interfaces so that any other codecs with the same capability as that of bzip2, could easily use the splitting support.  The details of these changes will be posted when we submit the code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-214-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 21:27:31 2009
Return-Path: <common-issues-return-214-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80285 invoked from network); 10 Jul 2009 21:27:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 21:27:31 -0000
Received: (qmail 2098 invoked by uid 500); 10 Jul 2009 21:27:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2055 invoked by uid 500); 10 Jul 2009 21:27:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1935 invoked by uid 99); 10 Jul 2009 21:27:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 21:27:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 21:27:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 6AE62234C1EB
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 14:27:15 -0700 (PDT)
Message-ID: <995656697.1247261235436.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 14:27:15 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4012) Providing splitting support for
 bzip2 compressed files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729826#action_12729826 ] 

Owen O'Malley commented on HADOOP-4012:
---------------------------------------

{quote}
Can you please be more specific: what changes were undone? 
{quote}

It was the feature from HADOOP-3144 that was dropped by mistake. But that is just indicative of the problem. LineRecordReader is very delicate code that is very easy to break. Changes to this code are hard to verify and it is used by many many Hadoop applications.

{quote}
I think the point of having splitting built in - is that all readers/formats can avoid re-implementing common things.
{quote}

This patch only seems to support TextInputFormat and Bzip. I don't see abstractions that would make this portable to other codecs or input formats. Maybe combining your binary format with BzipTextInputFormat makes sense? For binary files, SequenceFiles is already splittable with bzip and *no* changes.

> Providing splitting support for bzip2 compressed files
> ------------------------------------------------------
>
>                 Key: HADOOP-4012
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4012
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Abdul Qadeer
>            Assignee: Abdul Qadeer
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-4012-version1.patch, Hadoop-4012-version2.patch, Hadoop-4012-version3.patch, Hadoop-4012-version4.patch, Hadoop-4012-version5.patch, Hadoop-4012-version6.patch, Hadoop-4012-version7.patch, Hadoop-4012-version8.patch, Hadoop-4012-version9.patch
>
>
> Hadoop assumes that if the input data is compressed, it can not be split (mainly due to the limitation of many codecs that they need the whole input stream to decompress successfully).  So in such a case, Hadoop prepares only one split per compressed file, where the lower split limit is at 0 while the upper limit is the end of the file.  The consequence of this decision is that, one compress file goes to a single mapper. Although it circumvents the limitation of codecs (as mentioned above) but reduces the parallelism substantially, as it was possible otherwise in case of splitting.
> BZip2 is a compression / De-Compression algorithm which does compression on blocks of data and later these compressed blocks can be decompressed independent of each other.  This is indeed an opportunity that instead of one BZip2 compressed file going to one mapper, we can process chunks of file in parallel.  The correctness criteria of such a processing is that for a bzip2 compressed file, each compressed block should be processed by only one mapper and ultimately all the blocks of the file should be processed.  (By processing we mean the actual utilization of that un-compressed data (coming out of the codecs) in a mapper).
> We are writing the code to implement this suggested functionality.  Although we have used bzip2 as an example, but we have tried to extend Hadoop's compression interfaces so that any other codecs with the same capability as that of bzip2, could easily use the splitting support.  The details of these changes will be posted when we submit the code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-215-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 22:23:50 2009
Return-Path: <common-issues-return-215-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8913 invoked from network); 10 Jul 2009 22:23:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 22:23:50 -0000
Received: (qmail 57655 invoked by uid 500); 10 Jul 2009 22:23:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57550 invoked by uid 500); 10 Jul 2009 22:23:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57473 invoked by uid 99); 10 Jul 2009 22:23:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 22:23:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 22:23:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 05AE6234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 15:23:15 -0700 (PDT)
Message-ID: <965264797.1247264595009.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 15:23:15 -0700 (PDT)
From: "Vladimir Klimontovich (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6140)
 DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
In-Reply-To: <1165046874.1247245875107.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729866#action_12729866 ] 

Vladimir Klimontovich commented on HADOOP-6140:
-----------------------------------------------

Philip,

I tried to create junit test for (2), but I met some difficulties. Seems that I need to start whole cluster from junit test (jobtracker, tasktracker) to
test running job with additional classpath in distributed cache. I'm not familiar with hadoop code, but maybe you or someone could point to the test that I could use as example.

Or there is another option. I can extract piece of code with bug to separate method and test it separately. But I'm not sure it's reasonable to in in 0.18 branch.

Also, I noticed that a comma bug is already fixed in trunk. 


> DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6140
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6140
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.18.3
>            Reporter: Vladimir Klimontovich
>         Attachments: HADOOP-6140.patch
>
>
> addArchiveToClassPath is a method of DistributedCache class. It should be called before running a task. It accepts path to a jar file on a DFS. After it
> this method should put this jar file on sitribuuted cache and than add this file to classpath to each map/reduce process on job tracker.
> This method didn't work.
> Bug 1:
> addArchiveToClassPath adds DFS-path to archive to mapred.job.classpath.archives property. It uses System.getProperty("path.separator") as delimiter of multiple path.
> getFileClassPaths that is called from TaskRunner uses splits mapred.job.classpath.archives using System.getProperty("path.separator").
> In unix systems System.getProperty("path.separator") equals to ":". DFS-path urls is hdfs://host:port/path. It means that a result of split will be
> [ hdfs,//host,port/path].
> Suggested solution: use "," instead of  
> Bug 2:
> in TaskRunner there is an algorithm that looks for correspondence between DFS paths and local paths in distributed cache. 
> It compares
>    if (archives[i].getPath().equals(
>                                                archiveClasspaths[j].toString())){
> instead of
>     if (archives[i].toString().equals(
>                                                archiveClasspaths[j].toString())) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-216-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 22:23:50 2009
Return-Path: <common-issues-return-216-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8929 invoked from network); 10 Jul 2009 22:23:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 22:23:50 -0000
Received: (qmail 59303 invoked by uid 500); 10 Jul 2009 22:23:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59276 invoked by uid 500); 10 Jul 2009 22:23:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59246 invoked by uid 99); 10 Jul 2009 22:23:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 22:23:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 22:23:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 182F5234C1EF
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 15:23:15 -0700 (PDT)
Message-ID: <982828317.1247264595098.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 15:23:15 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6141) hadoop 0.20 branch "test-patch" is
 broken
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

hadoop 0.20 branch "test-patch" is broken
-----------------------------------------

                 Key: HADOOP-6141
                 URL: https://issues.apache.org/jira/browse/HADOOP-6141
             Project: Hadoop Common
          Issue Type: Bug
          Components: build
    Affects Versions: 0.20.0
            Reporter: Hong Tang
            Assignee: Hong Tang


There were two problems found in src/test/bin/test-patch.sh while I am doing the backporting of TFile patch (HADOOP-3315):
- java5.home and forrest.home is not defined for the ant command in pre-build stage, which leads to the following error message (in file trunkJavacWarnings.txt):
{code}
java5.check:

BUILD FAILED
/home/htang/workspace/test-patch/branch-0.20/build.xml:891: 'java5.home' is not defined.  Forrest requires Java 5.  Please pass -Djava5.home=<base of Java 5 distribution> to Ant on the command-line.
{code}
- When referring 10-th argument from the command line, it should use "${10}" instead of "$10" (which is $1 with a zero appended).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-217-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 22:25:28 2009
Return-Path: <common-issues-return-217-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9696 invoked from network); 10 Jul 2009 22:25:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 22:25:27 -0000
Received: (qmail 66841 invoked by uid 500); 10 Jul 2009 22:25:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66804 invoked by uid 500); 10 Jul 2009 22:25:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66794 invoked by uid 99); 10 Jul 2009 22:25:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 22:25:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 22:25:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C49DA234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 15:25:14 -0700 (PDT)
Message-ID: <715841804.1247264714801.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 15:25:14 -0700 (PDT)
From: "Vladimir Klimontovich (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6140)
 DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
In-Reply-To: <1165046874.1247245875107.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vladimir Klimontovich updated HADOOP-6140:
------------------------------------------

    Attachment: HADOOP-6140-ver2.patch

This patch contains unit-test for (1) and it validates if path to classpath entry contains CLASSPATH_ARCHIVES_SEPARATOR (",")

> DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6140
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6140
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.18.3
>            Reporter: Vladimir Klimontovich
>         Attachments: HADOOP-6140-ver2.patch, HADOOP-6140.patch
>
>
> addArchiveToClassPath is a method of DistributedCache class. It should be called before running a task. It accepts path to a jar file on a DFS. After it
> this method should put this jar file on sitribuuted cache and than add this file to classpath to each map/reduce process on job tracker.
> This method didn't work.
> Bug 1:
> addArchiveToClassPath adds DFS-path to archive to mapred.job.classpath.archives property. It uses System.getProperty("path.separator") as delimiter of multiple path.
> getFileClassPaths that is called from TaskRunner uses splits mapred.job.classpath.archives using System.getProperty("path.separator").
> In unix systems System.getProperty("path.separator") equals to ":". DFS-path urls is hdfs://host:port/path. It means that a result of split will be
> [ hdfs,//host,port/path].
> Suggested solution: use "," instead of  
> Bug 2:
> in TaskRunner there is an algorithm that looks for correspondence between DFS paths and local paths in distributed cache. 
> It compares
>    if (archives[i].getPath().equals(
>                                                archiveClasspaths[j].toString())){
> instead of
>     if (archives[i].toString().equals(
>                                                archiveClasspaths[j].toString())) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-218-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 22:35:31 2009
Return-Path: <common-issues-return-218-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15274 invoked from network); 10 Jul 2009 22:35:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 22:35:29 -0000
Received: (qmail 84640 invoked by uid 500); 10 Jul 2009 22:35:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84603 invoked by uid 500); 10 Jul 2009 22:35:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84593 invoked by uid 99); 10 Jul 2009 22:35:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 22:35:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 22:35:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D21B3234C044
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 15:35:14 -0700 (PDT)
Message-ID: <1068131679.1247265314859.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 15:35:14 -0700 (PDT)
From: "Philip Zeyliger (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6140)
 DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
In-Reply-To: <1165046874.1247245875107.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729870#action_12729870 ] 

Philip Zeyliger commented on HADOOP-6140:
-----------------------------------------

One creates clusters in unit tests using MiniMRCluster, but that's too heavy-weight: I think it's ok to extract the relevant function and test it via unit test.

+1 to the tests for (1).

-- Philip

> DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6140
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6140
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.18.3
>            Reporter: Vladimir Klimontovich
>         Attachments: HADOOP-6140-ver2.patch, HADOOP-6140.patch
>
>
> addArchiveToClassPath is a method of DistributedCache class. It should be called before running a task. It accepts path to a jar file on a DFS. After it
> this method should put this jar file on sitribuuted cache and than add this file to classpath to each map/reduce process on job tracker.
> This method didn't work.
> Bug 1:
> addArchiveToClassPath adds DFS-path to archive to mapred.job.classpath.archives property. It uses System.getProperty("path.separator") as delimiter of multiple path.
> getFileClassPaths that is called from TaskRunner uses splits mapred.job.classpath.archives using System.getProperty("path.separator").
> In unix systems System.getProperty("path.separator") equals to ":". DFS-path urls is hdfs://host:port/path. It means that a result of split will be
> [ hdfs,//host,port/path].
> Suggested solution: use "," instead of  
> Bug 2:
> in TaskRunner there is an algorithm that looks for correspondence between DFS paths and local paths in distributed cache. 
> It compares
>    if (archives[i].getPath().equals(
>                                                archiveClasspaths[j].toString())){
> instead of
>     if (archives[i].toString().equals(
>                                                archiveClasspaths[j].toString())) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-219-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 22:39:37 2009
Return-Path: <common-issues-return-219-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15997 invoked from network); 10 Jul 2009 22:39:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 22:39:37 -0000
Received: (qmail 89761 invoked by uid 500); 10 Jul 2009 22:39:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89640 invoked by uid 500); 10 Jul 2009 22:39:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89630 invoked by uid 99); 10 Jul 2009 22:39:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 22:39:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 22:39:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 04E20234C48D
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 15:39:15 -0700 (PDT)
Message-ID: <436012761.1247265555019.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 15:39:15 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6141) hadoop 0.20 branch "test-patch" is
 broken
In-Reply-To: <982828317.1247264595098.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729874#action_12729874 ] 

Hong Tang commented on HADOOP-6141:
-----------------------------------

Note that the problems in test-patch.sh seem to be fixed in trunk, but interestingly it is not backported to 0.20.

> hadoop 0.20 branch "test-patch" is broken
> -----------------------------------------
>
>                 Key: HADOOP-6141
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6141
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.20.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>
> There were two problems found in src/test/bin/test-patch.sh while I am doing the backporting of TFile patch (HADOOP-3315):
> - java5.home and forrest.home is not defined for the ant command in pre-build stage, which leads to the following error message (in file trunkJavacWarnings.txt):
> {code}
> java5.check:
> BUILD FAILED
> /home/htang/workspace/test-patch/branch-0.20/build.xml:891: 'java5.home' is not defined.  Forrest requires Java 5.  Please pass -Djava5.home=<base of Java 5 distribution> to Ant on the command-line.
> {code}
> - When referring 10-th argument from the command line, it should use "${10}" instead of "$10" (which is $1 with a zero appended).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-220-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 22:47:27 2009
Return-Path: <common-issues-return-220-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16991 invoked from network); 10 Jul 2009 22:47:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 22:47:27 -0000
Received: (qmail 513 invoked by uid 500); 10 Jul 2009 22:47:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 472 invoked by uid 500); 10 Jul 2009 22:47:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 462 invoked by uid 99); 10 Jul 2009 22:47:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 22:47:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 22:47:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id ED9E2234C1EE
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 15:47:14 -0700 (PDT)
Message-ID: <1194201800.1247266034972.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 15:47:14 -0700 (PDT)
From: "Mahadev konar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6142) archives relative path changes in
 common.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

archives relative path changes in common.
-----------------------------------------

                 Key: HADOOP-6142
                 URL: https://issues.apache.org/jira/browse/HADOOP-6142
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Mahadev konar
            Assignee: Mahadev konar
             Fix For: 0.21.0


HADOOP-3663 was moved to mapreduce accidentally. Opening this jira to upload my chanegs to common --- changes are rleated to MAPREDUCE-739.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-221-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 22:49:27 2009
Return-Path: <common-issues-return-221-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17404 invoked from network); 10 Jul 2009 22:49:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 22:49:27 -0000
Received: (qmail 2621 invoked by uid 500); 10 Jul 2009 22:49:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2584 invoked by uid 500); 10 Jul 2009 22:49:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2574 invoked by uid 99); 10 Jul 2009 22:49:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 22:49:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 22:49:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 46C1B234C1EE
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 15:49:15 -0700 (PDT)
Message-ID: <1899768008.1247266155289.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 15:49:15 -0700 (PDT)
From: "Mahadev konar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6142) archives relative path changes in
 common.
In-Reply-To: <1194201800.1247266034972.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mahadev konar updated HADOOP-6142:
----------------------------------

    Status: Patch Available  (was: Open)

> archives relative path changes in common.
> -----------------------------------------
>
>                 Key: HADOOP-6142
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6142
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Mahadev konar
>            Assignee: Mahadev konar
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6142.patch
>
>
> HADOOP-3663 was moved to mapreduce accidentally. Opening this jira to upload my chanegs to common --- changes are rleated to MAPREDUCE-739.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-222-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 22:49:29 2009
Return-Path: <common-issues-return-222-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17465 invoked from network); 10 Jul 2009 22:49:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 22:49:29 -0000
Received: (qmail 2707 invoked by uid 500); 10 Jul 2009 22:49:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2665 invoked by uid 500); 10 Jul 2009 22:49:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2655 invoked by uid 99); 10 Jul 2009 22:49:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 22:49:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 22:49:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 43BB3234C052
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 15:49:15 -0700 (PDT)
Message-ID: <1926544142.1247266155276.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 15:49:15 -0700 (PDT)
From: "Mahadev konar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6142) archives relative path changes in
 common.
In-Reply-To: <1194201800.1247266034972.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mahadev konar updated HADOOP-6142:
----------------------------------

    Attachment: HADOOP-6142.patch

> archives relative path changes in common.
> -----------------------------------------
>
>                 Key: HADOOP-6142
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6142
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Mahadev konar
>            Assignee: Mahadev konar
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6142.patch
>
>
> HADOOP-3663 was moved to mapreduce accidentally. Opening this jira to upload my chanegs to common --- changes are rleated to MAPREDUCE-739.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-223-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 23:25:36 2009
Return-Path: <common-issues-return-223-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23282 invoked from network); 10 Jul 2009 23:25:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 23:25:36 -0000
Received: (qmail 24423 invoked by uid 500); 10 Jul 2009 23:25:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24380 invoked by uid 500); 10 Jul 2009 23:25:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24370 invoked by uid 99); 10 Jul 2009 23:25:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 23:25:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 23:25:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DA5F2234C046
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 16:25:14 -0700 (PDT)
Message-ID: <1081341136.1247268314893.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 16:25:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6141) hadoop 0.20 branch "test-patch" is
 broken
In-Reply-To: <982828317.1247264595098.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-6141:
------------------------------

    Fix Version/s: 0.20.0
           Status: Patch Available  (was: Open)

Added java5.home and forrest.home definitions to all ant commands. Added curly braces for ${10} (JAVA5_HOME).

> hadoop 0.20 branch "test-patch" is broken
> -----------------------------------------
>
>                 Key: HADOOP-6141
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6141
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.20.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>             Fix For: 0.20.0
>
>         Attachments: hadoop-6141.patch
>
>
> There were two problems found in src/test/bin/test-patch.sh while I am doing the backporting of TFile patch (HADOOP-3315):
> - java5.home and forrest.home is not defined for the ant command in pre-build stage, which leads to the following error message (in file trunkJavacWarnings.txt):
> {code}
> java5.check:
> BUILD FAILED
> /home/htang/workspace/test-patch/branch-0.20/build.xml:891: 'java5.home' is not defined.  Forrest requires Java 5.  Please pass -Djava5.home=<base of Java 5 distribution> to Ant on the command-line.
> {code}
> - When referring 10-th argument from the command line, it should use "${10}" instead of "$10" (which is $1 with a zero appended).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-224-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 23:25:36 2009
Return-Path: <common-issues-return-224-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23331 invoked from network); 10 Jul 2009 23:25:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 23:25:36 -0000
Received: (qmail 24489 invoked by uid 500); 10 Jul 2009 23:25:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24444 invoked by uid 500); 10 Jul 2009 23:25:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24434 invoked by uid 99); 10 Jul 2009 23:25:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 23:25:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 23:25:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C7F81234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 16:25:14 -0700 (PDT)
Message-ID: <906552672.1247268314804.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 16:25:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6141) hadoop 0.20 branch "test-patch" is
 broken
In-Reply-To: <982828317.1247264595098.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-6141:
------------------------------

    Attachment: hadoop-6141.patch

> hadoop 0.20 branch "test-patch" is broken
> -----------------------------------------
>
>                 Key: HADOOP-6141
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6141
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.20.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>             Fix For: 0.20.0
>
>         Attachments: hadoop-6141.patch
>
>
> There were two problems found in src/test/bin/test-patch.sh while I am doing the backporting of TFile patch (HADOOP-3315):
> - java5.home and forrest.home is not defined for the ant command in pre-build stage, which leads to the following error message (in file trunkJavacWarnings.txt):
> {code}
> java5.check:
> BUILD FAILED
> /home/htang/workspace/test-patch/branch-0.20/build.xml:891: 'java5.home' is not defined.  Forrest requires Java 5.  Please pass -Djava5.home=<base of Java 5 distribution> to Ant on the command-line.
> {code}
> - When referring 10-th argument from the command line, it should use "${10}" instead of "$10" (which is $1 with a zero appended).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-225-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 23:31:34 2009
Return-Path: <common-issues-return-225-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23951 invoked from network); 10 Jul 2009 23:31:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 23:31:33 -0000
Received: (qmail 27197 invoked by uid 500); 10 Jul 2009 23:31:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27160 invoked by uid 500); 10 Jul 2009 23:31:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27150 invoked by uid 99); 10 Jul 2009 23:31:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 23:31:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 23:31:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 09B0F234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 16:31:15 -0700 (PDT)
Message-ID: <999576098.1247268675032.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 16:31:15 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6141) hadoop 0.20 branch "test-patch" is
 broken
In-Reply-To: <982828317.1247264595098.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729888#action_12729888 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6141:
------------------------------------------------

+1 patch looks good.

> hadoop 0.20 branch "test-patch" is broken
> -----------------------------------------
>
>                 Key: HADOOP-6141
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6141
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.20.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>             Fix For: 0.20.0
>
>         Attachments: hadoop-6141.patch
>
>
> There were two problems found in src/test/bin/test-patch.sh while I am doing the backporting of TFile patch (HADOOP-3315):
> - java5.home and forrest.home is not defined for the ant command in pre-build stage, which leads to the following error message (in file trunkJavacWarnings.txt):
> {code}
> java5.check:
> BUILD FAILED
> /home/htang/workspace/test-patch/branch-0.20/build.xml:891: 'java5.home' is not defined.  Forrest requires Java 5.  Please pass -Djava5.home=<base of Java 5 distribution> to Ant on the command-line.
> {code}
> - When referring 10-th argument from the command line, it should use "${10}" instead of "$10" (which is $1 with a zero appended).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-226-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 23:35:36 2009
Return-Path: <common-issues-return-226-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24438 invoked from network); 10 Jul 2009 23:35:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 23:35:36 -0000
Received: (qmail 27991 invoked by uid 500); 10 Jul 2009 23:35:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27953 invoked by uid 500); 10 Jul 2009 23:35:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27943 invoked by uid 99); 10 Jul 2009 23:35:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 23:35:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 23:35:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C90B3234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 16:35:14 -0700 (PDT)
Message-ID: <855287160.1247268914810.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 16:35:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4074) releaseaudit target must be
 re-written so that the rat jar file isn't in the classpath
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729889#action_12729889 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-4074:
------------------------------------------------

What should we do for the following comments found in 0.20 test-patch.sh?
{code}
  ### DISABLE RELEASE AUDIT UNTIL HADOOP-4074 IS FIXED
  ### Do not call releaseaudit when run by a developer
  ### if [[ $HUDSON == "true" ]] ; then
    ### echo "$ANT_HOME/bin/ant -Dversion="${VERSION}" -DHadoopPatchProcess= releaseaudit > $PATCH_DIR/trunkReleaseAuditWarnings.txt 2>&1"
    ### $ANT_HOME/bin/ant -Dversion="${VERSION}" -DHadoopPatchProcess= releaseaudit > $PATCH_DIR/trunkReleaseAuditWarnings.txt 2>&1
  ### fi
{code}

> releaseaudit target must be re-written so that the rat jar file isn't in the classpath
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4074
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4074
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.19.0
>            Reporter: Nigel Daley
>
> The nightly build and the Hudson patch testing process copies the rat-0.5.1.jar into the hadoop lib directory so that the releaseaudit target works.  This puts it in the general classpath.  Since Hive was committed, this jar file contains older versions of classes needed by Hive.  This rat jar file should be put elsewhere or configured in another way so that it doesn't need to be in the classpath. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-227-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 10 23:39:29 2009
Return-Path: <common-issues-return-227-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24748 invoked from network); 10 Jul 2009 23:39:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Jul 2009 23:39:29 -0000
Received: (qmail 31334 invoked by uid 500); 10 Jul 2009 23:39:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31296 invoked by uid 500); 10 Jul 2009 23:39:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31286 invoked by uid 99); 10 Jul 2009 23:39:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 23:39:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jul 2009 23:39:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CE8F9234C044
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 16:39:14 -0700 (PDT)
Message-ID: <636161787.1247269154845.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 16:39:14 -0700 (PDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6143) FS shell commands returns incorrect
 exit code  when error occurs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

FS shell commands returns incorrect exit code  when error occurs
----------------------------------------------------------------

                 Key: HADOOP-6143
                 URL: https://issues.apache.org/jira/browse/HADOOP-6143
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
            Reporter: Ravi Phulari


HDFS documentation ( http://hadoop.apache.org/core/docs/current/hdfs_shell.html#du )  mentions that 
{noformat}
Exit Code:

Returns 0 on success and -1 on error. 
{noformat} 

Current Fs shell behavior is buggy with this agreement.

{code}
statepick-lm:Hadoop rphulari$ bin/hadoop fs -ls foo
ls: Cannot access foo: No such file or directory.
statepick-lm:Hadoop rphulari$ echo $?
255
statepick-lm:Hadoop rphulari$ bin/hadoop fs -lsr foo
lsr: Cannot access foo: No such file or directory.
statepick-lm:Hadoop rphulari$ echo $?
255
statepick-lm:Hadoop rphulari$ bin/hadoop fs -du foo
du: Cannot access foo: No such file or directory.
statepick-lm:Hadoop rphulari$ echo $?
255
statepick-lm:Hadoop rphulari$ bin/hadoop fs -dus foo
dus: Cannot access foo: No such file or directory.
statepick-lm:Hadoop rphulari$ echo $?
255
statepick-lm:Hadoop rphulari$ bin/hadoop fs -cp foo f2
cp: File does not exist: foo
statepick-lm:Hadoop rphulari$ echo $?
255
statepick-lm:Hadoop rphulari$ bin/hadoop fs -copyToLocal foo f2
copyToLocal: null
statepick-lm:Hadoop rphulari$ echo $?
255
statepick-lm:Hadoop rphulari$ bin/hadoop fs -copyFromLocal foo f2
copyFromLocal: File foo does not exist.
statepick-lm:Hadoop rphulari$ echo $?
255
{code}  

In all above cases exit code on error should be -1 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-228-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 11 00:01:27 2009
Return-Path: <common-issues-return-228-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28447 invoked from network); 11 Jul 2009 00:01:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jul 2009 00:01:27 -0000
Received: (qmail 44018 invoked by uid 500); 11 Jul 2009 00:01:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43976 invoked by uid 500); 11 Jul 2009 00:01:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43966 invoked by uid 99); 11 Jul 2009 00:01:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 00:01:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 00:01:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D66B3234C046
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 17:01:14 -0700 (PDT)
Message-ID: <546024233.1247270474877.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 17:01:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6140)
 DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
In-Reply-To: <1165046874.1247245875107.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729900#action_12729900 ] 

Hadoop QA commented on HADOOP-6140:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12413164/HADOOP-6140-ver2.patch
  against trunk revision 793098.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/564/console

This message is automatically generated.

> DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6140
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6140
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.18.3
>            Reporter: Vladimir Klimontovich
>         Attachments: HADOOP-6140-ver2.patch, HADOOP-6140.patch
>
>
> addArchiveToClassPath is a method of DistributedCache class. It should be called before running a task. It accepts path to a jar file on a DFS. After it
> this method should put this jar file on sitribuuted cache and than add this file to classpath to each map/reduce process on job tracker.
> This method didn't work.
> Bug 1:
> addArchiveToClassPath adds DFS-path to archive to mapred.job.classpath.archives property. It uses System.getProperty("path.separator") as delimiter of multiple path.
> getFileClassPaths that is called from TaskRunner uses splits mapred.job.classpath.archives using System.getProperty("path.separator").
> In unix systems System.getProperty("path.separator") equals to ":". DFS-path urls is hdfs://host:port/path. It means that a result of split will be
> [ hdfs,//host,port/path].
> Suggested solution: use "," instead of  
> Bug 2:
> in TaskRunner there is an algorithm that looks for correspondence between DFS paths and local paths in distributed cache. 
> It compares
>    if (archives[i].getPath().equals(
>                                                archiveClasspaths[j].toString())){
> instead of
>     if (archives[i].toString().equals(
>                                                archiveClasspaths[j].toString())) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-229-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 11 00:17:27 2009
Return-Path: <common-issues-return-229-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31627 invoked from network); 11 Jul 2009 00:17:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jul 2009 00:17:27 -0000
Received: (qmail 56733 invoked by uid 500); 11 Jul 2009 00:17:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56696 invoked by uid 500); 11 Jul 2009 00:17:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56674 invoked by uid 99); 11 Jul 2009 00:17:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 00:17:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 00:17:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DF172234C052
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 17:17:14 -0700 (PDT)
Message-ID: <1140007707.1247271434912.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 17:17:14 -0700 (PDT)
From: "Alok Singh (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6144) streaming doesn't support
 jobclient.output.filter
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

streaming doesn't support jobclient.output.filter
-------------------------------------------------

                 Key: HADOOP-6144
                 URL: https://issues.apache.org/jira/browse/HADOOP-6144
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.20.0
         Environment: Linux
            Reporter: Alok Singh


the streaming Jobclient implementation i.e contrib/streaming/src/java/org/apache/hadoop/streaming/StreamJob.java is significantly different than the core hadoop mapred/org/apache/hadoop/mapred/JobClient.java.

for example unlike StreamJob.java, JobClient.java it gets tasks log when jobclient.output.filter=ALL is specified .

With hod-logs going away in hadoop 0.20 (due to new scheduler) user has no good way of programmitically getting logs

We should have intermediate adaptor class to implement Tools for the purpose of submitting jobs via m/r, streaming, pipes so that we don't miss some core functionality.
 
GenericJobClient implements Tools and then StreamJob extends GenericJobClient, JobClient extends GenericJobClient

Alok

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-230-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 11 00:23:35 2009
Return-Path: <common-issues-return-230-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39657 invoked from network); 11 Jul 2009 00:23:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jul 2009 00:23:35 -0000
Received: (qmail 61658 invoked by uid 500); 11 Jul 2009 00:23:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61616 invoked by uid 500); 11 Jul 2009 00:23:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61606 invoked by uid 99); 11 Jul 2009 00:23:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 00:23:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 00:23:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C75C4234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 17:23:14 -0700 (PDT)
Message-ID: <1548582403.1247271794802.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 17:23:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6141) hadoop 0.20 branch "test-patch" is
 broken
In-Reply-To: <982828317.1247264595098.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6141:
-------------------------------------------

       Resolution: Fixed
    Fix Version/s:     (was: 0.20.0)
                   0.20.1
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I have committed this to 0.20 only.  Thanks, Hong!

> hadoop 0.20 branch "test-patch" is broken
> -----------------------------------------
>
>                 Key: HADOOP-6141
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6141
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.20.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>             Fix For: 0.20.1
>
>         Attachments: hadoop-6141.patch
>
>
> There were two problems found in src/test/bin/test-patch.sh while I am doing the backporting of TFile patch (HADOOP-3315):
> - java5.home and forrest.home is not defined for the ant command in pre-build stage, which leads to the following error message (in file trunkJavacWarnings.txt):
> {code}
> java5.check:
> BUILD FAILED
> /home/htang/workspace/test-patch/branch-0.20/build.xml:891: 'java5.home' is not defined.  Forrest requires Java 5.  Please pass -Djava5.home=<base of Java 5 distribution> to Ant on the command-line.
> {code}
> - When referring 10-th argument from the command line, it should use "${10}" instead of "$10" (which is $1 with a zero appended).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-231-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 11 00:25:27 2009
Return-Path: <common-issues-return-231-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40302 invoked from network); 11 Jul 2009 00:25:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jul 2009 00:25:27 -0000
Received: (qmail 63113 invoked by uid 500); 11 Jul 2009 00:25:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63073 invoked by uid 500); 11 Jul 2009 00:25:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63057 invoked by uid 99); 11 Jul 2009 00:25:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 00:25:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 00:25:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C7A98234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 17:25:14 -0700 (PDT)
Message-ID: <1365437642.1247271914802.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 17:25:14 -0700 (PDT)
From: "Vladimir Klimontovich (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6140)
 DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
In-Reply-To: <1165046874.1247245875107.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vladimir Klimontovich updated HADOOP-6140:
------------------------------------------

    Attachment: HADOOP-6140-ver3.patch

HADOOP-6140-ver3.patch contains patch with junit-test for (2)

> DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6140
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6140
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.18.3
>            Reporter: Vladimir Klimontovich
>         Attachments: HADOOP-6140-ver2.patch, HADOOP-6140-ver3.patch, HADOOP-6140.patch
>
>
> addArchiveToClassPath is a method of DistributedCache class. It should be called before running a task. It accepts path to a jar file on a DFS. After it
> this method should put this jar file on sitribuuted cache and than add this file to classpath to each map/reduce process on job tracker.
> This method didn't work.
> Bug 1:
> addArchiveToClassPath adds DFS-path to archive to mapred.job.classpath.archives property. It uses System.getProperty("path.separator") as delimiter of multiple path.
> getFileClassPaths that is called from TaskRunner uses splits mapred.job.classpath.archives using System.getProperty("path.separator").
> In unix systems System.getProperty("path.separator") equals to ":". DFS-path urls is hdfs://host:port/path. It means that a result of split will be
> [ hdfs,//host,port/path].
> Suggested solution: use "," instead of  
> Bug 2:
> in TaskRunner there is an algorithm that looks for correspondence between DFS paths and local paths in distributed cache. 
> It compares
>    if (archives[i].getPath().equals(
>                                                archiveClasspaths[j].toString())){
> instead of
>     if (archives[i].toString().equals(
>                                                archiveClasspaths[j].toString())) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-232-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 11 00:31:27 2009
Return-Path: <common-issues-return-232-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42090 invoked from network); 11 Jul 2009 00:31:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jul 2009 00:31:27 -0000
Received: (qmail 66508 invoked by uid 500); 11 Jul 2009 00:31:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66468 invoked by uid 500); 11 Jul 2009 00:31:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66458 invoked by uid 99); 11 Jul 2009 00:31:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 00:31:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 00:31:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3D080234C052
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 17:31:15 -0700 (PDT)
Message-ID: <2017781583.1247272275249.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 17:31:15 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-3315) New binary file format
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-3315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-3315:
------------------------------

    Attachment: hadoop-3315-0710-1-hadoop-20.patch

Patch to hadoop 0.20. Also included the minor fix in HADOOP-6131.

> New binary file format
> ----------------------
>
>                 Key: HADOOP-3315
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3315
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: Owen O'Malley
>            Assignee: Hong Tang
>             Fix For: 0.21.0
>
>         Attachments: hadoop-3315-0507.patch, hadoop-3315-0509-2.patch, hadoop-3315-0509.patch, hadoop-3315-0513.patch, hadoop-3315-0514.patch, hadoop-3315-0601.patch, hadoop-3315-0602.patch, hadoop-3315-0605.patch, hadoop-3315-0612.patch, hadoop-3315-0623-2.patch, hadoop-3315-0701-yhadoop-20.patch, hadoop-3315-0710-1-hadoop-20.patch, HADOOP-3315_20080908_TFILE_PREVIEW_WITH_LZO_TESTS.patch, HADOOP-3315_20080915_TFILE.patch, hadoop-trunk-tfile.patch, hadoop-trunk-tfile.patch, TFile Specification 20081217.pdf
>
>
> SequenceFile's block compression format is too complex and requires 4 codecs to compress or decompress. It would be good to have a file format that only needs 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-233-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 11 00:31:29 2009
Return-Path: <common-issues-return-233-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42155 invoked from network); 11 Jul 2009 00:31:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jul 2009 00:31:29 -0000
Received: (qmail 66588 invoked by uid 500); 11 Jul 2009 00:31:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66548 invoked by uid 500); 11 Jul 2009 00:31:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66538 invoked by uid 99); 11 Jul 2009 00:31:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 00:31:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 00:31:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E7F91234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 17:31:14 -0700 (PDT)
Message-ID: <2012874597.1247272274935.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 17:31:14 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Moved: (HADOOP-6145) No error message for deleting
 non-existant file or directory.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan moved HDFS-479 to HADOOP-6145:
------------------------------------------

          Component/s:     (was: hdfs client)
                       fs
    Affects Version/s:     (was: 0.20.1)
                       0.20.1
                  Key: HADOOP-6145  (was: HDFS-479)
              Project: Hadoop Common  (was: Hadoop HDFS)

> No error message for deleting non-existant file or directory.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6145
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6145
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.1
>            Reporter: Suman Sehgal
>            Assignee: Jakob Homan
>
> If non-existant path or src is provided with rm/rmr option then no error message is displayed 
> command: hadoop dfs -rm <src>  (where src is non-existant)
> dfs displays "rm:  <src>"
> while it should display "No such file or directory".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-234-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 11 00:31:30 2009
Return-Path: <common-issues-return-234-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42218 invoked from network); 11 Jul 2009 00:31:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jul 2009 00:31:29 -0000
Received: (qmail 66661 invoked by uid 500); 11 Jul 2009 00:31:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66623 invoked by uid 500); 11 Jul 2009 00:31:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66613 invoked by uid 99); 11 Jul 2009 00:31:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 00:31:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 00:31:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DD97E234C4CC
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 17:31:15 -0700 (PDT)
Message-ID: <2046448094.1247272275906.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 17:31:15 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6145) No error message for deleting
 non-existant file or directory.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729912#action_12729912 ] 

Jakob Homan commented on HADOOP-6145:
-------------------------------------

Moved to Common as bug was tracked to FsShell, which is in Common.

> No error message for deleting non-existant file or directory.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6145
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6145
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.1
>            Reporter: Suman Sehgal
>            Assignee: Jakob Homan
>
> If non-existant path or src is provided with rm/rmr option then no error message is displayed 
> command: hadoop dfs -rm <src>  (where src is non-existant)
> dfs displays "rm:  <src>"
> while it should display "No such file or directory".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-235-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 11 00:33:36 2009
Return-Path: <common-issues-return-235-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42573 invoked from network); 11 Jul 2009 00:33:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jul 2009 00:33:36 -0000
Received: (qmail 67484 invoked by uid 500); 11 Jul 2009 00:33:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67446 invoked by uid 500); 11 Jul 2009 00:33:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67436 invoked by uid 99); 11 Jul 2009 00:33:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 00:33:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 00:33:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 00356234C052
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 17:33:16 -0700 (PDT)
Message-ID: <1425638920.1247272396000.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 17:33:15 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-3315) New binary file format
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-3315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729914#action_12729914 ] 

Hong Tang commented on HADOOP-3315:
-----------------------------------

ant test and test-patch passed (after applying patch to test-patch.sh from HADOOP-6141).

     [exec] +1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 63 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

> New binary file format
> ----------------------
>
>                 Key: HADOOP-3315
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3315
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: Owen O'Malley
>            Assignee: Hong Tang
>             Fix For: 0.21.0
>
>         Attachments: hadoop-3315-0507.patch, hadoop-3315-0509-2.patch, hadoop-3315-0509.patch, hadoop-3315-0513.patch, hadoop-3315-0514.patch, hadoop-3315-0601.patch, hadoop-3315-0602.patch, hadoop-3315-0605.patch, hadoop-3315-0612.patch, hadoop-3315-0623-2.patch, hadoop-3315-0701-yhadoop-20.patch, hadoop-3315-0710-1-hadoop-20.patch, HADOOP-3315_20080908_TFILE_PREVIEW_WITH_LZO_TESTS.patch, HADOOP-3315_20080915_TFILE.patch, hadoop-trunk-tfile.patch, hadoop-trunk-tfile.patch, TFile Specification 20081217.pdf
>
>
> SequenceFile's block compression format is too complex and requires 4 codecs to compress or decompress. It would be good to have a file format that only needs 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-236-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 11 00:51:36 2009
Return-Path: <common-issues-return-236-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45923 invoked from network); 11 Jul 2009 00:51:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jul 2009 00:51:36 -0000
Received: (qmail 76670 invoked by uid 500); 11 Jul 2009 00:51:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76630 invoked by uid 500); 11 Jul 2009 00:51:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76620 invoked by uid 99); 11 Jul 2009 00:51:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 00:51:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 00:51:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E9137234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 17:51:14 -0700 (PDT)
Message-ID: <791042932.1247273474940.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 17:51:14 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6143) FS shell commands returns incorrect
 exit code  when error occurs
In-Reply-To: <636161787.1247269154845.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729915#action_12729915 ] 

Todd Lipcon commented on HADOOP-6143:
-------------------------------------

-1 exit code is equivalent to 255 exit code on Linux - exit codes are a single byte and range from 0-255. See http://www.faqs.org/docs/abs/HTML/exit-status.html

> FS shell commands returns incorrect exit code  when error occurs
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6143
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6143
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Ravi Phulari
>
> HDFS documentation ( http://hadoop.apache.org/core/docs/current/hdfs_shell.html#du )  mentions that 
> {noformat}
> Exit Code:
> Returns 0 on success and -1 on error. 
> {noformat} 
> Current Fs shell behavior is buggy with this agreement.
> {code}
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -ls foo
> ls: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -lsr foo
> lsr: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -du foo
> du: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -dus foo
> dus: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -cp foo f2
> cp: File does not exist: foo
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -copyToLocal foo f2
> copyToLocal: null
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -copyFromLocal foo f2
> copyFromLocal: File foo does not exist.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> {code}  
> In all above cases exit code on error should be -1 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-237-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 11 00:53:27 2009
Return-Path: <common-issues-return-237-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46157 invoked from network); 11 Jul 2009 00:53:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jul 2009 00:53:27 -0000
Received: (qmail 77865 invoked by uid 500); 11 Jul 2009 00:53:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77825 invoked by uid 500); 11 Jul 2009 00:53:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77813 invoked by uid 99); 11 Jul 2009 00:53:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 00:53:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 00:53:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E0D3A234C1EB
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 17:53:14 -0700 (PDT)
Message-ID: <254127384.1247273594920.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 17:53:14 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6143) FS shell commands returns incorrect
 exit code  when error occurs
In-Reply-To: <636161787.1247269154845.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon resolved HADOOP-6143.
---------------------------------

    Resolution: Invalid
      Assignee: Todd Lipcon

Resolving as invalid. Please reopen if you disagree.

> FS shell commands returns incorrect exit code  when error occurs
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6143
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6143
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Ravi Phulari
>            Assignee: Todd Lipcon
>
> HDFS documentation ( http://hadoop.apache.org/core/docs/current/hdfs_shell.html#du )  mentions that 
> {noformat}
> Exit Code:
> Returns 0 on success and -1 on error. 
> {noformat} 
> Current Fs shell behavior is buggy with this agreement.
> {code}
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -ls foo
> ls: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -lsr foo
> lsr: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -du foo
> du: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -dus foo
> dus: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -cp foo f2
> cp: File does not exist: foo
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -copyToLocal foo f2
> copyToLocal: null
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -copyFromLocal foo f2
> copyFromLocal: File foo does not exist.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> {code}  
> In all above cases exit code on error should be -1 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-238-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 11 01:03:27 2009
Return-Path: <common-issues-return-238-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49533 invoked from network); 11 Jul 2009 01:03:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jul 2009 01:03:27 -0000
Received: (qmail 82520 invoked by uid 500); 11 Jul 2009 01:03:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82483 invoked by uid 500); 11 Jul 2009 01:03:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82473 invoked by uid 99); 11 Jul 2009 01:03:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 01:03:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 01:03:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C5C6C234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 18:03:14 -0700 (PDT)
Message-ID: <1791891147.1247274194796.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 18:03:14 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6145) No error message for deleting
 non-existant file or directory.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6145:
--------------------------------

    Attachment: HADOOP-6145.patch

Attaching patch for v20

> No error message for deleting non-existant file or directory.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6145
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6145
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.1
>            Reporter: Suman Sehgal
>            Assignee: Jakob Homan
>         Attachments: HADOOP-6145.patch
>
>
> If non-existant path or src is provided with rm/rmr option then no error message is displayed 
> command: hadoop dfs -rm <src>  (where src is non-existant)
> dfs displays "rm:  <src>"
> while it should display "No such file or directory".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-239-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 11 01:05:29 2009
Return-Path: <common-issues-return-239-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50093 invoked from network); 11 Jul 2009 01:05:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jul 2009 01:05:29 -0000
Received: (qmail 82974 invoked by uid 500); 11 Jul 2009 01:05:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82936 invoked by uid 500); 11 Jul 2009 01:05:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82883 invoked by uid 99); 11 Jul 2009 01:05:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 01:05:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 01:05:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0C9F4234C4B3
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 18:05:15 -0700 (PDT)
Message-ID: <1627837515.1247274315050.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 18:05:15 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6145) No error message for deleting
 non-existant file or directory.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6145:
--------------------------------

    Attachment: COMMON-6145.patch

Attaching patch for trunk.
Problem was the trash method was throwing FileNotFound, which was not being handled.  There are three different places where the non-existence of the file being deleted could conceivably be handled.  Fixed by just checking for the file's existence at the beginning and exiting if not found.

No unit test because the affected unit test is TestHDFSCLI, which is in HDFS project.  Manually tested and it works.  The reason this wasn't detected previously is that it only manifests itself when the trash feature is enabled, which apparently it isn't on the minidfscluster that powers TestCLI.  We should probably looking at running Test*CLI with both trash on and trash off.

> No error message for deleting non-existant file or directory.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6145
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6145
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.1
>            Reporter: Suman Sehgal
>            Assignee: Jakob Homan
>         Attachments: COMMON-6145.patch, HADOOP-6145.patch
>
>
> If non-existant path or src is provided with rm/rmr option then no error message is displayed 
> command: hadoop dfs -rm <src>  (where src is non-existant)
> dfs displays "rm:  <src>"
> while it should display "No such file or directory".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-240-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 11 01:07:27 2009
Return-Path: <common-issues-return-240-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50809 invoked from network); 11 Jul 2009 01:07:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jul 2009 01:07:27 -0000
Received: (qmail 85865 invoked by uid 500); 11 Jul 2009 01:07:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85827 invoked by uid 500); 11 Jul 2009 01:07:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85817 invoked by uid 99); 11 Jul 2009 01:07:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 01:07:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 01:07:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C4C1A234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 18:07:14 -0700 (PDT)
Message-ID: <1624791477.1247274434791.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 18:07:14 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6145) No error message for deleting
 non-existant file or directory.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6145:
--------------------------------

    Status: Patch Available  (was: Open)

submitting patch.  Hudson should pick up the correct one.

> No error message for deleting non-existant file or directory.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6145
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6145
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.1
>            Reporter: Suman Sehgal
>            Assignee: Jakob Homan
>         Attachments: COMMON-6145.patch, HADOOP-6145.patch
>
>
> If non-existant path or src is provided with rm/rmr option then no error message is displayed 
> command: hadoop dfs -rm <src>  (where src is non-existant)
> dfs displays "rm:  <src>"
> while it should display "No such file or directory".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-241-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 11 01:23:27 2009
Return-Path: <common-issues-return-241-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54890 invoked from network); 11 Jul 2009 01:23:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jul 2009 01:23:27 -0000
Received: (qmail 96630 invoked by uid 500); 11 Jul 2009 01:23:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96593 invoked by uid 500); 11 Jul 2009 01:23:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96582 invoked by uid 99); 11 Jul 2009 01:23:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 01:23:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 01:23:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EEC58234C052
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 18:23:14 -0700 (PDT)
Message-ID: <1688789528.1247275394977.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 18:23:14 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-3315) New binary file format
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-3315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-3315:
----------------------------------

    Fix Version/s:     (was: 0.21.0)
                   0.20.1

Committed to 0.20 branch

> New binary file format
> ----------------------
>
>                 Key: HADOOP-3315
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3315
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: Owen O'Malley
>            Assignee: Hong Tang
>             Fix For: 0.20.1
>
>         Attachments: hadoop-3315-0507.patch, hadoop-3315-0509-2.patch, hadoop-3315-0509.patch, hadoop-3315-0513.patch, hadoop-3315-0514.patch, hadoop-3315-0601.patch, hadoop-3315-0602.patch, hadoop-3315-0605.patch, hadoop-3315-0612.patch, hadoop-3315-0623-2.patch, hadoop-3315-0701-yhadoop-20.patch, hadoop-3315-0710-1-hadoop-20.patch, HADOOP-3315_20080908_TFILE_PREVIEW_WITH_LZO_TESTS.patch, HADOOP-3315_20080915_TFILE.patch, hadoop-trunk-tfile.patch, hadoop-trunk-tfile.patch, TFile Specification 20081217.pdf
>
>
> SequenceFile's block compression format is too complex and requires 4 codecs to compress or decompress. It would be good to have a file format that only needs 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-242-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 11 01:23:31 2009
Return-Path: <common-issues-return-242-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54957 invoked from network); 11 Jul 2009 01:23:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jul 2009 01:23:29 -0000
Received: (qmail 96708 invoked by uid 500); 11 Jul 2009 01:23:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96670 invoked by uid 500); 11 Jul 2009 01:23:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96660 invoked by uid 99); 11 Jul 2009 01:23:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 01:23:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 01:23:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E2805234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 18:23:14 -0700 (PDT)
Message-ID: <1773117517.1247275394913.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 18:23:14 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6145) No error message for deleting
 non-existant file or directory.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729927#action_12729927 ] 

Jakob Homan commented on HADOOP-6145:
-------------------------------------

also, the trunk patch has a quick fix to bring the rm documentation in line with what was committed in HADOOP-6139.

> No error message for deleting non-existant file or directory.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6145
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6145
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.1
>            Reporter: Suman Sehgal
>            Assignee: Jakob Homan
>         Attachments: COMMON-6145.patch, HADOOP-6145.patch
>
>
> If non-existant path or src is provided with rm/rmr option then no error message is displayed 
> command: hadoop dfs -rm <src>  (where src is non-existant)
> dfs displays "rm:  <src>"
> while it should display "No such file or directory".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-243-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 11 02:05:29 2009
Return-Path: <common-issues-return-243-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74269 invoked from network); 11 Jul 2009 02:05:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jul 2009 02:05:27 -0000
Received: (qmail 23095 invoked by uid 500); 11 Jul 2009 02:05:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23050 invoked by uid 500); 11 Jul 2009 02:05:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23032 invoked by uid 99); 11 Jul 2009 02:05:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 02:05:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 02:05:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D21B7234C044
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 19:05:14 -0700 (PDT)
Message-ID: <1485403267.1247277914859.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 19:05:14 -0700 (PDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Reopened: (HADOOP-6143) FS shell commands returns incorrect
 exit code  when error occurs
In-Reply-To: <636161787.1247269154845.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari reopened HADOOP-6143:
----------------------------------


As I wrote in the above description HDFS shell guide misleads to user  that exit code -1 is returned on error. 
( http://hadoop.apache.org/core/docs/current/hdfs_shell.html#du ) 
Either we should return exact exit value equal to -1 on error or we should  correct documentation as  "exit code returns non zero value on error"

OS X manual for CP  command explicitly mentions that "cp utility exits >0 if an error occurs"
{code}
statepick-lm:hadoop-hdfs rphulari$ man cp | grep exit
           is displayed and the exit value is not altered.
     The cp utility exits 0 on success, and >0 if an error occurs.
{code}

> FS shell commands returns incorrect exit code  when error occurs
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6143
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6143
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Ravi Phulari
>            Assignee: Todd Lipcon
>
> HDFS documentation ( http://hadoop.apache.org/core/docs/current/hdfs_shell.html#du )  mentions that 
> {noformat}
> Exit Code:
> Returns 0 on success and -1 on error. 
> {noformat} 
> Current Fs shell behavior is buggy with this agreement.
> {code}
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -ls foo
> ls: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -lsr foo
> lsr: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -du foo
> du: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -dus foo
> dus: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -cp foo f2
> cp: File does not exist: foo
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -copyToLocal foo f2
> copyToLocal: null
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -copyFromLocal foo f2
> copyFromLocal: File foo does not exist.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> {code}  
> In all above cases exit code on error should be -1 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-244-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 11 02:27:29 2009
Return-Path: <common-issues-return-244-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82804 invoked from network); 11 Jul 2009 02:27:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jul 2009 02:27:29 -0000
Received: (qmail 28779 invoked by uid 500); 11 Jul 2009 02:27:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28741 invoked by uid 500); 11 Jul 2009 02:27:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28731 invoked by uid 99); 11 Jul 2009 02:27:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 02:27:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 02:27:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C86B8234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 19:27:14 -0700 (PDT)
Message-ID: <1623762892.1247279234805.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 19:27:14 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6143) FS shell commands returns incorrect
 exit code  when error occurs
In-Reply-To: <636161787.1247269154845.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729937#action_12729937 ] 

Todd Lipcon commented on HADOOP-6143:
-------------------------------------

{code}
todd@todd-laptop:/tmp$ cat Test.java 
public class Test {
  public static void main(String []args) {
    System.exit(-1);
  }
}
todd@todd-laptop:/tmp$ javac Test.java 
todd@todd-laptop:/tmp$ java -cp . Test ; echo $?
255
{code}

It's possible this might actually return -1 on some other OS though (does Windows have a concept of an exit code? I think maybe?). Changing the docs to say "nonzero" vs "zero" instead of -1 vs 0 does make sense, though.

> FS shell commands returns incorrect exit code  when error occurs
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6143
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6143
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Ravi Phulari
>            Assignee: Todd Lipcon
>
> HDFS documentation ( http://hadoop.apache.org/core/docs/current/hdfs_shell.html#du )  mentions that 
> {noformat}
> Exit Code:
> Returns 0 on success and -1 on error. 
> {noformat} 
> Current Fs shell behavior is buggy with this agreement.
> {code}
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -ls foo
> ls: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -lsr foo
> lsr: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -du foo
> du: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -dus foo
> dus: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -cp foo f2
> cp: File does not exist: foo
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -copyToLocal foo f2
> copyToLocal: null
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -copyFromLocal foo f2
> copyFromLocal: File foo does not exist.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> {code}  
> In all above cases exit code on error should be -1 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-245-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 11 02:31:29 2009
Return-Path: <common-issues-return-245-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84288 invoked from network); 11 Jul 2009 02:31:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jul 2009 02:31:29 -0000
Received: (qmail 29164 invoked by uid 500); 11 Jul 2009 02:31:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29122 invoked by uid 500); 11 Jul 2009 02:31:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29105 invoked by uid 99); 11 Jul 2009 02:31:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 02:31:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 02:31:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CADCF234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 19:31:14 -0700 (PDT)
Message-ID: <743723373.1247279474816.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 19:31:14 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6143) FS shell commands returns incorrect
 exit code  when error occurs
In-Reply-To: <636161787.1247269154845.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729938#action_12729938 ] 

Jakob Homan commented on HADOOP-6143:
-------------------------------------

bq. we should correct documentation as "exit code returns non zero value on error" 

+1

> FS shell commands returns incorrect exit code  when error occurs
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6143
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6143
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Ravi Phulari
>            Assignee: Todd Lipcon
>
> HDFS documentation ( http://hadoop.apache.org/core/docs/current/hdfs_shell.html#du )  mentions that 
> {noformat}
> Exit Code:
> Returns 0 on success and -1 on error. 
> {noformat} 
> Current Fs shell behavior is buggy with this agreement.
> {code}
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -ls foo
> ls: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -lsr foo
> lsr: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -du foo
> du: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -dus foo
> dus: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -cp foo f2
> cp: File does not exist: foo
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -copyToLocal foo f2
> copyToLocal: null
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -copyFromLocal foo f2
> copyFromLocal: File foo does not exist.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> {code}  
> In all above cases exit code on error should be -1 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-246-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 11 03:01:28 2009
Return-Path: <common-issues-return-246-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94775 invoked from network); 11 Jul 2009 03:01:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jul 2009 03:01:27 -0000
Received: (qmail 54429 invoked by uid 500); 11 Jul 2009 03:01:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54391 invoked by uid 500); 11 Jul 2009 03:01:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54381 invoked by uid 99); 11 Jul 2009 03:01:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 03:01:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 03:01:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C8A64234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 20:01:14 -0700 (PDT)
Message-ID: <374264735.1247281274810.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 20:01:14 -0700 (PDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6143) FS shell commands returns incorrect
 exit code  when error occurs
In-Reply-To: <636161787.1247269154845.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729940#action_12729940 ] 

Ravi Phulari commented on HADOOP-6143:
--------------------------------------

+1 For changing the docs . 


> FS shell commands returns incorrect exit code  when error occurs
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6143
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6143
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Ravi Phulari
>            Assignee: Todd Lipcon
>
> HDFS documentation ( http://hadoop.apache.org/core/docs/current/hdfs_shell.html#du )  mentions that 
> {noformat}
> Exit Code:
> Returns 0 on success and -1 on error. 
> {noformat} 
> Current Fs shell behavior is buggy with this agreement.
> {code}
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -ls foo
> ls: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -lsr foo
> lsr: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -du foo
> du: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -dus foo
> dus: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -cp foo f2
> cp: File does not exist: foo
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -copyToLocal foo f2
> copyToLocal: null
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -copyFromLocal foo f2
> copyFromLocal: File foo does not exist.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> {code}  
> In all above cases exit code on error should be -1 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-247-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 11 03:47:31 2009
Return-Path: <common-issues-return-247-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11987 invoked from network); 11 Jul 2009 03:47:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jul 2009 03:47:30 -0000
Received: (qmail 82793 invoked by uid 500); 11 Jul 2009 03:47:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82756 invoked by uid 500); 11 Jul 2009 03:47:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82746 invoked by uid 99); 11 Jul 2009 03:47:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 03:47:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 03:47:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C5090234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 20:47:14 -0700 (PDT)
Message-ID: <1522680834.1247284034793.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 20:47:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6142) archives relative path changes in
 common.
In-Reply-To: <1194201800.1247266034972.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729944#action_12729944 ] 

Hadoop QA commented on HADOOP-6142:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12413166/HADOOP-6142.patch
  against trunk revision 793162.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/565/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/565/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/565/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/565/console

This message is automatically generated.

> archives relative path changes in common.
> -----------------------------------------
>
>                 Key: HADOOP-6142
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6142
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Mahadev konar
>            Assignee: Mahadev konar
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6142.patch
>
>
> HADOOP-3663 was moved to mapreduce accidentally. Opening this jira to upload my chanegs to common --- changes are rleated to MAPREDUCE-739.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-248-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 11 03:51:36 2009
Return-Path: <common-issues-return-248-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12236 invoked from network); 11 Jul 2009 03:51:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jul 2009 03:51:36 -0000
Received: (qmail 83222 invoked by uid 500); 11 Jul 2009 03:51:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83184 invoked by uid 500); 11 Jul 2009 03:51:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83174 invoked by uid 99); 11 Jul 2009 03:51:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 03:51:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 03:51:42 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 60F4D234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 20:51:20 -0700 (PDT)
Message-ID: <1174100427.1247284280261.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 20:51:20 -0700 (PDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6143) FS shell commands returns incorrect
 exit code  when error occurs
In-Reply-To: <636161787.1247269154845.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6143:
---------------------------------

    Attachment: HADOOP-6143.patch

Attaching patch . 

> FS shell commands returns incorrect exit code  when error occurs
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6143
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6143
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Ravi Phulari
>            Assignee: Todd Lipcon
>         Attachments: HADOOP-6143.patch
>
>
> HDFS documentation ( http://hadoop.apache.org/core/docs/current/hdfs_shell.html#du )  mentions that 
> {noformat}
> Exit Code:
> Returns 0 on success and -1 on error. 
> {noformat} 
> Current Fs shell behavior is buggy with this agreement.
> {code}
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -ls foo
> ls: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -lsr foo
> lsr: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -du foo
> du: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -dus foo
> dus: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -cp foo f2
> cp: File does not exist: foo
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -copyToLocal foo f2
> copyToLocal: null
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -copyFromLocal foo f2
> copyFromLocal: File foo does not exist.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> {code}  
> In all above cases exit code on error should be -1 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-249-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 11 03:55:36 2009
Return-Path: <common-issues-return-249-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12781 invoked from network); 11 Jul 2009 03:55:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jul 2009 03:55:36 -0000
Received: (qmail 84759 invoked by uid 500); 11 Jul 2009 03:55:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84721 invoked by uid 500); 11 Jul 2009 03:55:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84709 invoked by uid 99); 11 Jul 2009 03:55:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 03:55:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 03:55:41 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 27E8A234C004
	for <common-issues@hadoop.apache.org>; Fri, 10 Jul 2009 20:55:20 -0700 (PDT)
Message-ID: <2033331204.1247284520044.JavaMail.jira@brutus>
Date: Fri, 10 Jul 2009 20:55:20 -0700 (PDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6143) FS shell commands returns incorrect
 exit code  when error occurs
In-Reply-To: <636161787.1247269154845.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6143:
---------------------------------

    Assignee: Ravi Phulari  (was: Todd Lipcon)
      Status: Patch Available  (was: Reopened)

This patch consists of only documentation changes . Compiled ant docs and verified HTML docs . 
No Tests included . 

> FS shell commands returns incorrect exit code  when error occurs
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6143
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6143
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>         Attachments: HADOOP-6143.patch
>
>
> HDFS documentation ( http://hadoop.apache.org/core/docs/current/hdfs_shell.html#du )  mentions that 
> {noformat}
> Exit Code:
> Returns 0 on success and -1 on error. 
> {noformat} 
> Current Fs shell behavior is buggy with this agreement.
> {code}
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -ls foo
> ls: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -lsr foo
> lsr: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -du foo
> du: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -dus foo
> dus: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -cp foo f2
> cp: File does not exist: foo
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -copyToLocal foo f2
> copyToLocal: null
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -copyFromLocal foo f2
> copyFromLocal: File foo does not exist.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> {code}  
> In all above cases exit code on error should be -1 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-250-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 11 10:18:30 2009
Return-Path: <common-issues-return-250-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81631 invoked from network); 11 Jul 2009 10:18:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Jul 2009 10:18:30 -0000
Received: (qmail 15923 invoked by uid 500); 11 Jul 2009 10:18:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15885 invoked by uid 500); 11 Jul 2009 10:18:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15875 invoked by uid 99); 11 Jul 2009 10:18:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 10:18:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jul 2009 10:18:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D0616234C004
	for <common-issues@hadoop.apache.org>; Sat, 11 Jul 2009 03:18:14 -0700 (PDT)
Message-ID: <821632605.1247307494839.JavaMail.jira@brutus>
Date: Sat, 11 Jul 2009 03:18:14 -0700 (PDT)
From: "Abdul Qadeer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4012) Providing splitting support for
 bzip2 compressed files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729967#action_12729967 ] 

Abdul Qadeer commented on HADOOP-4012:
--------------------------------------

I totally agree with Owen that LineRecordReader (LRR) is a heavily used code in Hadoop and changes to such a code should be very carefully done. But that doesn't mean that, LRR code is closed for any improvements and feature enhancements. This LRR deals with Text data and Hadoop uses it as a default because probably Text is the most frequently used kind on input. I think same is true for BZip2 compressed files. If not all, most of them are Text compressed data. This patch provides the following feature to the end user.


User puts his BZip2 compressed text files in an input directory and submits the Hadoop job. Soon he gets the result in the output directory. That is it!

He hadn't had to write or mention

any input format
any record reader
And all this happens using full cluster CPU power (due to BZip2 splitting).
Also our algorithm doesn't demand any specific kind of splits from Hadoop. It works with whatever splits provided to it.
.............................................................................

Now comes the correctness of LRR code.

testFormat() test case in org.apache.hadoop.mapred.TestTextInputFormat is a very stringent test case to ensure the correctness of LRR. And since this test case is frequently run whenever someone submits and tests a patch, so I dont think that any correctness problem in LRR can escape.

Now comes the things like HADOOP-3114. That was my mistake as mentioned by Chris also in his comments. I have corrected that and will upload the new patch soon after running the tests locally. I think these are the places where the vision and knowledge of Hadoop commmiters comes handy. If you see that other than HADOOP-3114, I have missed something else, please tell me that and I will fix it. Additionally Hadoop has a very active user community and any latent bug in the code (especially the one used heavily) can not hide for long. Also the serious businesses using Hadoop usually use only the stable version (e.g I think Yahoo yet uses 0.18? while 0.20 is out).
So I see a safe transition to the changed LRR code.

........................................................
Now comes the point of making a new BZip2 specific input format / record reader.


If we make a separate reader for BZip2 e.g BZip2LineRecordReader, in that class most of the code will be the replica of LRR. And doing that way, GZip support should also come out of LRR and there will be a GZipLineRecordReader, again mostly the code a replica of LRR. Whenever there comes a new codec in Hadoop we will make a new reader whose 99% code is the replica of LRR. So in my view it makes more sense to handle Line related text (be it plain or compressed) in LRR.

When we were adding splitting support for BZip2, we felt that there might be situations when codecs want to change/manipulate the split start or end. So we added support for that. Similarly instead LRR counting read bytes itself, it now asks the stream about the position. This feature makes it possible for a codec to manipulate indirectly that how long a reader should keep on reading. We think these features can help for other block based codecs to do splitting easily without further changes in LRR.
..............................................................

So in summary this patch might need relatively stringent reviewing by the committers due to its heavy usage but it does add useful functionality in a seamless way for the end user.

> Providing splitting support for bzip2 compressed files
> ------------------------------------------------------
>
>                 Key: HADOOP-4012
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4012
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Abdul Qadeer
>            Assignee: Abdul Qadeer
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-4012-version1.patch, Hadoop-4012-version2.patch, Hadoop-4012-version3.patch, Hadoop-4012-version4.patch, Hadoop-4012-version5.patch, Hadoop-4012-version6.patch, Hadoop-4012-version7.patch, Hadoop-4012-version8.patch, Hadoop-4012-version9.patch
>
>
> Hadoop assumes that if the input data is compressed, it can not be split (mainly due to the limitation of many codecs that they need the whole input stream to decompress successfully).  So in such a case, Hadoop prepares only one split per compressed file, where the lower split limit is at 0 while the upper limit is the end of the file.  The consequence of this decision is that, one compress file goes to a single mapper. Although it circumvents the limitation of codecs (as mentioned above) but reduces the parallelism substantially, as it was possible otherwise in case of splitting.
> BZip2 is a compression / De-Compression algorithm which does compression on blocks of data and later these compressed blocks can be decompressed independent of each other.  This is indeed an opportunity that instead of one BZip2 compressed file going to one mapper, we can process chunks of file in parallel.  The correctness criteria of such a processing is that for a bzip2 compressed file, each compressed block should be processed by only one mapper and ultimately all the blocks of the file should be processed.  (By processing we mean the actual utilization of that un-compressed data (coming out of the codecs) in a mapper).
> We are writing the code to implement this suggested functionality.  Although we have used bzip2 as an example, but we have tried to extend Hadoop's compression interfaces so that any other codecs with the same capability as that of bzip2, could easily use the splitting support.  The details of these changes will be posted when we submit the code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-251-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jul 12 13:37:28 2009
Return-Path: <common-issues-return-251-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10052 invoked from network); 12 Jul 2009 13:37:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jul 2009 13:37:27 -0000
Received: (qmail 719 invoked by uid 500); 12 Jul 2009 13:37:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 681 invoked by uid 500); 12 Jul 2009 13:37:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 671 invoked by uid 99); 12 Jul 2009 13:37:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jul 2009 13:37:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jul 2009 13:37:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DC667234C053
	for <common-issues@hadoop.apache.org>; Sun, 12 Jul 2009 06:37:14 -0700 (PDT)
Message-ID: <1894238801.1247405834901.JavaMail.jira@brutus>
Date: Sun, 12 Jul 2009 06:37:14 -0700 (PDT)
From: "Vladimir Klimontovich (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6140)
 DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
In-Reply-To: <1165046874.1247245875107.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730083#action_12730083 ] 

Vladimir Klimontovich commented on HADOOP-6140:
-----------------------------------------------

The same fix, but in trunk (0.21): MAPREDUCE-752

> DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6140
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6140
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.18.3
>            Reporter: Vladimir Klimontovich
>         Attachments: HADOOP-6140-ver2.patch, HADOOP-6140-ver3.patch, HADOOP-6140.patch
>
>
> addArchiveToClassPath is a method of DistributedCache class. It should be called before running a task. It accepts path to a jar file on a DFS. After it
> this method should put this jar file on sitribuuted cache and than add this file to classpath to each map/reduce process on job tracker.
> This method didn't work.
> Bug 1:
> addArchiveToClassPath adds DFS-path to archive to mapred.job.classpath.archives property. It uses System.getProperty("path.separator") as delimiter of multiple path.
> getFileClassPaths that is called from TaskRunner uses splits mapred.job.classpath.archives using System.getProperty("path.separator").
> In unix systems System.getProperty("path.separator") equals to ":". DFS-path urls is hdfs://host:port/path. It means that a result of split will be
> [ hdfs,//host,port/path].
> Suggested solution: use "," instead of  
> Bug 2:
> in TaskRunner there is an algorithm that looks for correspondence between DFS paths and local paths in distributed cache. 
> It compares
>    if (archives[i].getPath().equals(
>                                                archiveClasspaths[j].toString())){
> instead of
>     if (archives[i].toString().equals(
>                                                archiveClasspaths[j].toString())) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-252-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jul 12 13:47:27 2009
Return-Path: <common-issues-return-252-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13567 invoked from network); 12 Jul 2009 13:47:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jul 2009 13:47:27 -0000
Received: (qmail 6759 invoked by uid 500); 12 Jul 2009 13:47:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6717 invoked by uid 500); 12 Jul 2009 13:47:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6707 invoked by uid 99); 12 Jul 2009 13:47:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jul 2009 13:47:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jul 2009 13:47:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 2C782234C1EB
	for <common-issues@hadoop.apache.org>; Sun, 12 Jul 2009 06:47:15 -0700 (PDT)
Message-ID: <1675423876.1247406435181.JavaMail.jira@brutus>
Date: Sun, 12 Jul 2009 06:47:15 -0700 (PDT)
From: "Vladimir Klimontovich (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6140)
 DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
In-Reply-To: <1165046874.1247245875107.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vladimir Klimontovich updated HADOOP-6140:
------------------------------------------

    Attachment:     (was: HADOOP-6140-ver3.patch)

> DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6140
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6140
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.18.3
>            Reporter: Vladimir Klimontovich
>         Attachments: HADOOP-6140-ver4.patch
>
>
> addArchiveToClassPath is a method of DistributedCache class. It should be called before running a task. It accepts path to a jar file on a DFS. After it
> this method should put this jar file on sitribuuted cache and than add this file to classpath to each map/reduce process on job tracker.
> This method didn't work.
> Bug 1:
> addArchiveToClassPath adds DFS-path to archive to mapred.job.classpath.archives property. It uses System.getProperty("path.separator") as delimiter of multiple path.
> getFileClassPaths that is called from TaskRunner uses splits mapred.job.classpath.archives using System.getProperty("path.separator").
> In unix systems System.getProperty("path.separator") equals to ":". DFS-path urls is hdfs://host:port/path. It means that a result of split will be
> [ hdfs,//host,port/path].
> Suggested solution: use "," instead of  
> Bug 2:
> in TaskRunner there is an algorithm that looks for correspondence between DFS paths and local paths in distributed cache. 
> It compares
>    if (archives[i].getPath().equals(
>                                                archiveClasspaths[j].toString())){
> instead of
>     if (archives[i].toString().equals(
>                                                archiveClasspaths[j].toString())) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-253-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jul 12 13:47:32 2009
Return-Path: <common-issues-return-253-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13627 invoked from network); 12 Jul 2009 13:47:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jul 2009 13:47:29 -0000
Received: (qmail 6826 invoked by uid 500); 12 Jul 2009 13:47:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6785 invoked by uid 500); 12 Jul 2009 13:47:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6775 invoked by uid 99); 12 Jul 2009 13:47:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jul 2009 13:47:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jul 2009 13:47:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3915D234C4AC
	for <common-issues@hadoop.apache.org>; Sun, 12 Jul 2009 06:47:15 -0700 (PDT)
Message-ID: <76337531.1247406435233.JavaMail.jira@brutus>
Date: Sun, 12 Jul 2009 06:47:15 -0700 (PDT)
From: "Vladimir Klimontovich (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6140)
 DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
In-Reply-To: <1165046874.1247245875107.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vladimir Klimontovich updated HADOOP-6140:
------------------------------------------

    Attachment:     (was: HADOOP-6140.patch)

> DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6140
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6140
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.18.3
>            Reporter: Vladimir Klimontovich
>         Attachments: HADOOP-6140-ver4.patch
>
>
> addArchiveToClassPath is a method of DistributedCache class. It should be called before running a task. It accepts path to a jar file on a DFS. After it
> this method should put this jar file on sitribuuted cache and than add this file to classpath to each map/reduce process on job tracker.
> This method didn't work.
> Bug 1:
> addArchiveToClassPath adds DFS-path to archive to mapred.job.classpath.archives property. It uses System.getProperty("path.separator") as delimiter of multiple path.
> getFileClassPaths that is called from TaskRunner uses splits mapred.job.classpath.archives using System.getProperty("path.separator").
> In unix systems System.getProperty("path.separator") equals to ":". DFS-path urls is hdfs://host:port/path. It means that a result of split will be
> [ hdfs,//host,port/path].
> Suggested solution: use "," instead of  
> Bug 2:
> in TaskRunner there is an algorithm that looks for correspondence between DFS paths and local paths in distributed cache. 
> It compares
>    if (archives[i].getPath().equals(
>                                                archiveClasspaths[j].toString())){
> instead of
>     if (archives[i].toString().equals(
>                                                archiveClasspaths[j].toString())) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-254-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jul 12 13:47:32 2009
Return-Path: <common-issues-return-254-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13635 invoked from network); 12 Jul 2009 13:47:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jul 2009 13:47:29 -0000
Received: (qmail 6893 invoked by uid 500); 12 Jul 2009 13:47:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6849 invoked by uid 500); 12 Jul 2009 13:47:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6822 invoked by uid 99); 12 Jul 2009 13:47:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jul 2009 13:47:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jul 2009 13:47:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3250C234C48D
	for <common-issues@hadoop.apache.org>; Sun, 12 Jul 2009 06:47:15 -0700 (PDT)
Message-ID: <462055843.1247406435205.JavaMail.jira@brutus>
Date: Sun, 12 Jul 2009 06:47:15 -0700 (PDT)
From: "Vladimir Klimontovich (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6140)
 DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
In-Reply-To: <1165046874.1247245875107.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vladimir Klimontovich updated HADOOP-6140:
------------------------------------------

    Attachment:     (was: HADOOP-6140-ver2.patch)

> DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6140
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6140
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.18.3
>            Reporter: Vladimir Klimontovich
>         Attachments: HADOOP-6140-ver4.patch
>
>
> addArchiveToClassPath is a method of DistributedCache class. It should be called before running a task. It accepts path to a jar file on a DFS. After it
> this method should put this jar file on sitribuuted cache and than add this file to classpath to each map/reduce process on job tracker.
> This method didn't work.
> Bug 1:
> addArchiveToClassPath adds DFS-path to archive to mapred.job.classpath.archives property. It uses System.getProperty("path.separator") as delimiter of multiple path.
> getFileClassPaths that is called from TaskRunner uses splits mapred.job.classpath.archives using System.getProperty("path.separator").
> In unix systems System.getProperty("path.separator") equals to ":". DFS-path urls is hdfs://host:port/path. It means that a result of split will be
> [ hdfs,//host,port/path].
> Suggested solution: use "," instead of  
> Bug 2:
> in TaskRunner there is an algorithm that looks for correspondence between DFS paths and local paths in distributed cache. 
> It compares
>    if (archives[i].getPath().equals(
>                                                archiveClasspaths[j].toString())){
> instead of
>     if (archives[i].toString().equals(
>                                                archiveClasspaths[j].toString())) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-255-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jul 12 13:47:32 2009
Return-Path: <common-issues-return-255-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13648 invoked from network); 12 Jul 2009 13:47:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jul 2009 13:47:29 -0000
Received: (qmail 6951 invoked by uid 500); 12 Jul 2009 13:47:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6913 invoked by uid 500); 12 Jul 2009 13:47:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6897 invoked by uid 99); 12 Jul 2009 13:47:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jul 2009 13:47:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jul 2009 13:47:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 24CAE234C052
	for <common-issues@hadoop.apache.org>; Sun, 12 Jul 2009 06:47:15 -0700 (PDT)
Message-ID: <31200023.1247406435149.JavaMail.jira@brutus>
Date: Sun, 12 Jul 2009 06:47:15 -0700 (PDT)
From: "Vladimir Klimontovich (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6140)
 DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
In-Reply-To: <1165046874.1247245875107.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vladimir Klimontovich updated HADOOP-6140:
------------------------------------------

    Attachment: HADOOP-6140-ver4.patch

HADOOP-6140-ver4.patch is a enhancement of HADOOP-6140-ver3.patch: it extracts comparison statement to separate method and improves junit-tests 

> DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6140
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6140
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.18.3
>            Reporter: Vladimir Klimontovich
>         Attachments: HADOOP-6140-ver4.patch
>
>
> addArchiveToClassPath is a method of DistributedCache class. It should be called before running a task. It accepts path to a jar file on a DFS. After it
> this method should put this jar file on sitribuuted cache and than add this file to classpath to each map/reduce process on job tracker.
> This method didn't work.
> Bug 1:
> addArchiveToClassPath adds DFS-path to archive to mapred.job.classpath.archives property. It uses System.getProperty("path.separator") as delimiter of multiple path.
> getFileClassPaths that is called from TaskRunner uses splits mapred.job.classpath.archives using System.getProperty("path.separator").
> In unix systems System.getProperty("path.separator") equals to ":". DFS-path urls is hdfs://host:port/path. It means that a result of split will be
> [ hdfs,//host,port/path].
> Suggested solution: use "," instead of  
> Bug 2:
> in TaskRunner there is an algorithm that looks for correspondence between DFS paths and local paths in distributed cache. 
> It compares
>    if (archives[i].getPath().equals(
>                                                archiveClasspaths[j].toString())){
> instead of
>     if (archives[i].toString().equals(
>                                                archiveClasspaths[j].toString())) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-256-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jul 12 15:57:30 2009
Return-Path: <common-issues-return-256-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69605 invoked from network); 12 Jul 2009 15:57:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jul 2009 15:57:29 -0000
Received: (qmail 71510 invoked by uid 500); 12 Jul 2009 15:57:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71472 invoked by uid 500); 12 Jul 2009 15:57:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71462 invoked by uid 99); 12 Jul 2009 15:57:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jul 2009 15:57:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jul 2009 15:57:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 978A8234C4B9
	for <common-issues@hadoop.apache.org>; Sun, 12 Jul 2009 08:57:15 -0700 (PDT)
Message-ID: <1012480694.1247414235619.JavaMail.jira@brutus>
Date: Sun, 12 Jul 2009 08:57:15 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6137) to fix project specific test-patch
 requirements
In-Reply-To: <2044264383.1247120054920.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730102#action_12730102 ] 

Hudson commented on HADOOP-6137:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk #20 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/20/])
    

> to fix project specific test-patch requirements 
> ------------------------------------------------
>
>                 Key: HADOOP-6137
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6137
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>            Priority: Critical
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6137.patch
>
>
> only mapreduce project needs create-c++-configure target which needs to be executed as part to the test-core ant target.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-257-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jul 12 15:57:30 2009
Return-Path: <common-issues-return-257-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69652 invoked from network); 12 Jul 2009 15:57:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jul 2009 15:57:30 -0000
Received: (qmail 71613 invoked by uid 500); 12 Jul 2009 15:57:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71551 invoked by uid 500); 12 Jul 2009 15:57:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71529 invoked by uid 99); 12 Jul 2009 15:57:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jul 2009 15:57:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jul 2009 15:57:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0F5FE234C004
	for <common-issues@hadoop.apache.org>; Sun, 12 Jul 2009 08:57:15 -0700 (PDT)
Message-ID: <313832992.1247414235049.JavaMail.jira@brutus>
Date: Sun, 12 Jul 2009 08:57:15 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6090) GridMix is broke after upgrading
 random(text)writer to newer mapreduce apis
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730099#action_12730099 ] 

Hudson commented on HADOOP-6090:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk #20 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/20/])
    

> GridMix is broke after upgrading random(text)writer to newer mapreduce apis
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-6090
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6090
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: benchmarks
>    Affects Versions: 0.21.0
>            Reporter: Arun C Murthy
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.21.0
>
>         Attachments: 6090.txt
>
>
> GridMix data generation scripts need to use the newer mapreduce api.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-258-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jul 12 15:57:30 2009
Return-Path: <common-issues-return-258-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69673 invoked from network); 12 Jul 2009 15:57:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Jul 2009 15:57:30 -0000
Received: (qmail 71636 invoked by uid 500); 12 Jul 2009 15:57:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71553 invoked by uid 500); 12 Jul 2009 15:57:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71540 invoked by uid 99); 12 Jul 2009 15:57:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jul 2009 15:57:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jul 2009 15:57:37 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 332D6234C053
	for <common-issues@hadoop.apache.org>; Sun, 12 Jul 2009 08:57:15 -0700 (PDT)
Message-ID: <1226707272.1247414235208.JavaMail.jira@brutus>
Date: Sun, 12 Jul 2009 08:57:15 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5170) Set max map/reduce tasks on a
 per-job basis, either per-node or cluster-wide
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730100#action_12730100 ] 

Hudson commented on HADOOP-5170:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk #20 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/20/])
    

> Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5170
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Jonathan Gray
>            Assignee: Matei Zaharia
>             Fix For: 0.21.0
>
>         Attachments: h5170.patch, HADOOP-5170-tasklimits-v3-0.18.3.patch, tasklimits-v2.patch, tasklimits-v3-0.19.patch, tasklimits-v3.patch, tasklimits-v4-20.patch, tasklimits-v4.patch, tasklimits.patch
>
>
> There are a number of use cases for being able to do this.  The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.
> This could be implemented as either a per-node maximum or a cluster-wide maximum.  It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.
> Some of the reasons for allowing this feature (mine and from others on list):
> - I have some very large CPU-bound jobs.  I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver.  I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node.  Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
> - Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none.  A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
> - Poor mans MR job virtualization.  Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster.  (makes most sense w/ cluster-wide limit)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-259-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 13 05:08:28 2009
Return-Path: <common-issues-return-259-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97445 invoked from network); 13 Jul 2009 05:08:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jul 2009 05:08:28 -0000
Received: (qmail 51012 invoked by uid 500); 13 Jul 2009 05:08:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50975 invoked by uid 500); 13 Jul 2009 05:08:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50961 invoked by uid 99); 13 Jul 2009 05:08:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 05:08:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 05:08:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D5832234C004
	for <common-issues@hadoop.apache.org>; Sun, 12 Jul 2009 22:08:14 -0700 (PDT)
Message-ID: <266843207.1247461694860.JavaMail.jira@brutus>
Date: Sun, 12 Jul 2009 22:08:14 -0700 (PDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4491) Per-job local data on the
 TaskTracker node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730216#action_12730216 ] 

Hemanth Yamijala commented on HADOOP-4491:
------------------------------------------

I would like to call out some approaches this patch is taking explicitly, so that there's an opportunity for discussion:

h4. 1. Setting secure permissions by default vs doing so only in the LinuxTaskController

The patch sets permissions for localized job files and directories as part of the localization process itself, that is as soon as the files are localized, rather than in the TaskController. I believe this approach was taken primarily to secure permissions on the localized files on the disk ASAP. The effect of doing so is that now all configurations of hadoop, including those which do not worry about security will have secure permissions for some of the files. While this should be OK in principle, it must be noted that without the entire solution offered by the LinuxTaskController, full security is not achieved. Hence, does it make sense to make changes to the permissions only when using the LinuxTaskController (with a small window where files could have potentially insecure permissions), and leave the default case alone ?

Either way, I think the current method in which permissions are changed for the localized files causes these to be splattered across the code. It may lead to mistakes in future if somebody is making changes (say to localize a new file) and is not fully aware of the permissions issue. Ideally it should be abstracted from the process. In speaking with Vinod about this, I realized this may not be very easy to achieve the way LocalDirAllocator currently works. If we keep the changes to the task controller alone, I believe this will be very easy to accomplish.

h4. 2. Approach for sharing common files between TT and child

The patch juggles around with permissions on the localized and intermediate files by first setting ownership to the job owner, and after the task is complete, changing the ownership back to the tasktracker user. The reason it does this, is for two purposes:
-- To enable intermediate outputs to be served by the tasktracker, once the child has created them.
-- To allow cleanup as the tasktracker once the tasks are done.

Note that change of ownership is done by launching the task-controller setuid binary as root privileges are required for the purpose.

The above two problems can be solved in another way as well (and this is what was discussed on HADOOP-4490). In that approach, we'd thought of moving the intermediate outputs to a different directory owned by the tasktracker, and we'd thought of removing files and directories as the user, as part of the cleanup thread. I think one advantage this alternative approach brings is that it gives an opportunity to not launch the task-controller for reduces and hence slots can become free sooner. This does mean the cleanup might take more time, but that happens asynchronously.

On the other hand, the current model seems to be simpler to fit into mind. If a task is not complete, its files are owned by the job owner, if not, they are owned by TT. It is easy to check these rules on a running system. So, is it worth the change in approach to bring in a little optimization (which might actually not matter that much).

h4. 3. Sandboxing task runtime environment by changing values of mapred.local.dir for the child

The patch 'sandboxes' the task runtime environment, by changing the mapred.local.dir values from the original configured values to ${original mapred.local.dir}/taskTracker/jobCache/${job-id}/${task-id} and pass the new values to the child. Vinod told me that this was primarily required because checks in the LocalDirAllocator require the current process to have write permissions on the context passed to it (which is basically the mapred local directories). When the child calls LocalDirAllocator, it would now fail because the original mapred local directories will not have write permissions to world. Hence, the need to sandbox. Of course, it is also probably more secure.

Two questions this raises: Is this change in contract acceptable ? If yes, is it acceptable in the default case as well, or should it be restricted to the case of LinuxTaskController alone.

Either way, the current patch, in order to implement this behavior juggles around the values of the mapred.local.dir variable in conf file at 2-3 locations in the tasktracker code. I think that approach is error prone and needs to change.

Thoughts on these three points ?

> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, HADOOP-4491-20090707.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-260-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 13 05:18:30 2009
Return-Path: <common-issues-return-260-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99187 invoked from network); 13 Jul 2009 05:18:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jul 2009 05:18:30 -0000
Received: (qmail 56290 invoked by uid 500); 13 Jul 2009 05:18:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56252 invoked by uid 500); 13 Jul 2009 05:18:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56242 invoked by uid 99); 13 Jul 2009 05:18:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 05:18:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 05:18:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C8065234C004
	for <common-issues@hadoop.apache.org>; Sun, 12 Jul 2009 22:18:14 -0700 (PDT)
Message-ID: <737525313.1247462294805.JavaMail.jira@brutus>
Date: Sun, 12 Jul 2009 22:18:14 -0700 (PDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4491) Per-job local data on the
 TaskTracker node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730220#action_12730220 ] 

Hemanth Yamijala commented on HADOOP-4491:
------------------------------------------

Some other comments on the code:

- The code models some of the commands passed to the taskcontroller as 'first task' for JVM or 'not first task' for JVM. This complicates the contract between the TT and the task-controller. I think at a minimum these decisions should be restricted only to the JVM manager, and the contract between the TT and task-controller should be handled by modelling new commands like FINALIZE_JVM and FINALIZE_TASK.
- The patch introduces a log directory argument to the task-controller. I think this is not required as there is only one value for the hadoop.log.dir for all tasks.
- Code related to creation of work dirs is removed from TaskRunner. Where is it created now ?
- Permissions for job directory is changed multiple times - once per each task.
- finalizeTaskDirs (or parts of it) needs to be synchronized.
- In the case of jvm reuse, we still need to make the output available. Because it seems the task completion event is sent as soon as the task is done not after jvm exits. This is only a theoretical case possibly, but it still will be good to keep the code paths identical.
- Path permissions should be taken care of ? so, if mapred.local.dir doesn't have execute permissions for others, we need to set them.
- In setup, we seem to be setting permissions even if directory creation fails. also shouldn't we set permissions even if it exists.. so that it is right as per our requirement.
- Didn't understand the purpose of initStatus. Since it starts out being true, wouldn't it always remain true ?
- Cache directory probably doesn't need 777 because it is not written to by the tasks. We can probably retain this if HADOOP-4490 set like permissions, since this will be addressed in HADOOP-4493.
- getBaseIntermediateOutputDir seems an overkill if it is just returning a constant.
- Changes in SpillRecord seem to be unnecessary.

Some nits:
- There are some else blocks without code, but with a code comment explaining why there's no else block. While useful, it is somewhat unconventional. I would recommend the reason be moved to a comment starting the if block itself.
- TODOs in the patch must be discussed and resolved.
- TaskController.FILE_PERMISSIONS doesn't seem to be used anywhere.
- Rename finalizeTaskDirs as finalizeTask - it matches with the naming convention for the other apis in taskcontroller.
- Add a comment about why task-work is 755.
- mapred.child.local.dir has an extra comma at the end.

> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, HADOOP-4491-20090707.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-261-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 13 07:59:27 2009
Return-Path: <common-issues-return-261-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55388 invoked from network); 13 Jul 2009 07:59:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jul 2009 07:59:27 -0000
Received: (qmail 85935 invoked by uid 500); 13 Jul 2009 07:59:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85894 invoked by uid 500); 13 Jul 2009 07:59:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85878 invoked by uid 99); 13 Jul 2009 07:59:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 07:59:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 07:59:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C51D3234C004
	for <common-issues@hadoop.apache.org>; Mon, 13 Jul 2009 00:59:14 -0700 (PDT)
Message-ID: <891603154.1247471954802.JavaMail.jira@brutus>
Date: Mon, 13 Jul 2009 00:59:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6145) No error message for deleting
 non-existant file or directory.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730246#action_12730246 ] 

Hadoop QA commented on HADOOP-6145:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12413186/COMMON-6145.patch
  against trunk revision 793162.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/568/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/568/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/568/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/568/console

This message is automatically generated.

> No error message for deleting non-existant file or directory.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6145
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6145
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.1
>            Reporter: Suman Sehgal
>            Assignee: Jakob Homan
>         Attachments: COMMON-6145.patch, HADOOP-6145.patch
>
>
> If non-existant path or src is provided with rm/rmr option then no error message is displayed 
> command: hadoop dfs -rm <src>  (where src is non-existant)
> dfs displays "rm:  <src>"
> while it should display "No such file or directory".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-262-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 13 10:53:30 2009
Return-Path: <common-issues-return-262-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18056 invoked from network); 13 Jul 2009 10:53:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jul 2009 10:53:29 -0000
Received: (qmail 89926 invoked by uid 500); 13 Jul 2009 10:53:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89889 invoked by uid 500); 13 Jul 2009 10:53:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89879 invoked by uid 99); 13 Jul 2009 10:53:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 10:53:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 10:53:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id F1A26234C056
	for <common-issues@hadoop.apache.org>; Mon, 13 Jul 2009 03:53:14 -0700 (PDT)
Message-ID: <1921752137.1247482394988.JavaMail.jira@brutus>
Date: Mon, 13 Jul 2009 03:53:14 -0700 (PDT)
From: "He Yongqiang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5879) GzipCodec should read compression
 level etc from configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

He Yongqiang updated HADOOP-5879:
---------------------------------

    Attachment: hadoop-5879-7-13-2.patch

upload a new patch integrated Chris' suggestions (Thanks, Chris).

> GzipCodec should read compression level etc from configuration
> --------------------------------------------------------------
>
>                 Key: HADOOP-5879
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5879
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Zheng Shao
>         Attachments: hadoop-5879-5-21.patch, hadoop-5879-7-13-2.patch
>
>
> GzipCodec currently uses the default compression level. We should allow overriding the default value from Configuration.
> {code}
>   static final class GzipZlibCompressor extends ZlibCompressor {
>     public GzipZlibCompressor() {
>       super(ZlibCompressor.CompressionLevel.DEFAULT_COMPRESSION,
>           ZlibCompressor.CompressionStrategy.DEFAULT_STRATEGY,
>           ZlibCompressor.CompressionHeader.GZIP_FORMAT, 64*1024);
>     }
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-263-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 13 11:09:29 2009
Return-Path: <common-issues-return-263-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23085 invoked from network); 13 Jul 2009 11:09:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jul 2009 11:09:29 -0000
Received: (qmail 3700 invoked by uid 500); 13 Jul 2009 11:09:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3661 invoked by uid 500); 13 Jul 2009 11:09:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3646 invoked by uid 99); 13 Jul 2009 11:09:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 11:09:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 11:09:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C75EC234C004
	for <common-issues@hadoop.apache.org>; Mon, 13 Jul 2009 04:09:14 -0700 (PDT)
Message-ID: <1322739965.1247483354802.JavaMail.jira@brutus>
Date: Mon, 13 Jul 2009 04:09:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6143) FS shell commands returns incorrect
 exit code  when error occurs
In-Reply-To: <636161787.1247269154845.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730287#action_12730287 ] 

Hadoop QA commented on HADOOP-6143:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12413189/HADOOP-6143.patch
  against trunk revision 793162.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/569/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/569/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/569/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/569/console

This message is automatically generated.

> FS shell commands returns incorrect exit code  when error occurs
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6143
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6143
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>         Attachments: HADOOP-6143.patch
>
>
> HDFS documentation ( http://hadoop.apache.org/core/docs/current/hdfs_shell.html#du )  mentions that 
> {noformat}
> Exit Code:
> Returns 0 on success and -1 on error. 
> {noformat} 
> Current Fs shell behavior is buggy with this agreement.
> {code}
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -ls foo
> ls: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -lsr foo
> lsr: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -du foo
> du: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -dus foo
> dus: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -cp foo f2
> cp: File does not exist: foo
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -copyToLocal foo f2
> copyToLocal: null
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -copyFromLocal foo f2
> copyFromLocal: File foo does not exist.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> {code}  
> In all above cases exit code on error should be -1 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-264-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 13 11:53:30 2009
Return-Path: <common-issues-return-264-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35302 invoked from network); 13 Jul 2009 11:53:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jul 2009 11:53:27 -0000
Received: (qmail 59203 invoked by uid 500); 13 Jul 2009 11:53:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59163 invoked by uid 500); 13 Jul 2009 11:53:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59148 invoked by uid 99); 13 Jul 2009 11:53:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 11:53:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 11:53:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E8327234C053
	for <common-issues@hadoop.apache.org>; Mon, 13 Jul 2009 04:53:14 -0700 (PDT)
Message-ID: <576780672.1247485994950.JavaMail.jira@brutus>
Date: Mon, 13 Jul 2009 04:53:14 -0700 (PDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6120) Add support for Avro types in hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sharad Agarwal reassigned HADOOP-6120:
--------------------------------------

    Assignee: Sharad Agarwal

> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-265-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 13 12:19:30 2009
Return-Path: <common-issues-return-265-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44911 invoked from network); 13 Jul 2009 12:19:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jul 2009 12:19:29 -0000
Received: (qmail 95988 invoked by uid 500); 13 Jul 2009 12:19:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95939 invoked by uid 500); 13 Jul 2009 12:19:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95708 invoked by uid 99); 13 Jul 2009 12:19:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 12:19:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 12:19:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E522D234C052
	for <common-issues@hadoop.apache.org>; Mon, 13 Jul 2009 05:19:14 -0700 (PDT)
Message-ID: <965637874.1247487554937.JavaMail.jira@brutus>
Date: Mon, 13 Jul 2009 05:19:14 -0700 (PDT)
From: "He Yongqiang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5879) GzipCodec should read compression
 level etc from configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

He Yongqiang updated HADOOP-5879:
---------------------------------

    Attachment: hadoop-5879-7-13-3.patch

hadoop-5879-7-13-3.patch changed the conf string io.compress.* to zlib.compress.*

Supported conf strings are: 
zlib.compress.level         values can be: NO_COMPRESSION / BEST_SPEED / BEST_COMPRESSION / DEFAULT_COMPRESSION
zlib.compress.strategy     values can be:  FILTERED / HUFFMAN_ONLY / RLE / FIXED / DEFAULT_STRATEGY
zlib.compress.buffer.size    values can be any positive integer

> GzipCodec should read compression level etc from configuration
> --------------------------------------------------------------
>
>                 Key: HADOOP-5879
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5879
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Zheng Shao
>         Attachments: hadoop-5879-5-21.patch, hadoop-5879-7-13-2.patch, hadoop-5879-7-13-3.patch
>
>
> GzipCodec currently uses the default compression level. We should allow overriding the default value from Configuration.
> {code}
>   static final class GzipZlibCompressor extends ZlibCompressor {
>     public GzipZlibCompressor() {
>       super(ZlibCompressor.CompressionLevel.DEFAULT_COMPRESSION,
>           ZlibCompressor.CompressionStrategy.DEFAULT_STRATEGY,
>           ZlibCompressor.CompressionHeader.GZIP_FORMAT, 64*1024);
>     }
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-266-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 13 13:01:31 2009
Return-Path: <common-issues-return-266-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63383 invoked from network); 13 Jul 2009 13:01:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jul 2009 13:01:31 -0000
Received: (qmail 51178 invoked by uid 500); 13 Jul 2009 13:01:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51140 invoked by uid 500); 13 Jul 2009 13:01:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51130 invoked by uid 99); 13 Jul 2009 13:01:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 13:01:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 13:01:37 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 7D840234C052
	for <common-issues@hadoop.apache.org>; Mon, 13 Jul 2009 06:01:16 -0700 (PDT)
Message-ID: <13490839.1247490076513.JavaMail.jira@brutus>
Date: Mon, 13 Jul 2009 06:01:16 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6146) Upgrade to JetS3t version 0.7.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Upgrade to JetS3t version 0.7.1
-------------------------------

                 Key: HADOOP-6146
                 URL: https://issues.apache.org/jira/browse/HADOOP-6146
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs/s3
            Reporter: Tom White
            Assignee: Tom White
             Fix For: 0.21.0


The JetS3t library is used for the S3 filesystems. We should upgrade to the latest version (0.7.1) which has support for Requester Pays buckets. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-267-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 13 13:03:27 2009
Return-Path: <common-issues-return-267-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64205 invoked from network); 13 Jul 2009 13:03:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jul 2009 13:03:27 -0000
Received: (qmail 57343 invoked by uid 500); 13 Jul 2009 13:03:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57305 invoked by uid 500); 13 Jul 2009 13:03:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57295 invoked by uid 99); 13 Jul 2009 13:03:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 13:03:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 13:03:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3EA0F234C004
	for <common-issues@hadoop.apache.org>; Mon, 13 Jul 2009 06:03:15 -0700 (PDT)
Message-ID: <1060836433.1247490195242.JavaMail.jira@brutus>
Date: Mon, 13 Jul 2009 06:03:15 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6146) Upgrade to JetS3t version 0.7.1
In-Reply-To: <13490839.1247490076513.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6146:
------------------------------

    Attachment: HADOOP-6146.patch

Patch for this issue. I have successfully run both Jets3tNativeS3FileSystemContractTest and Jets3tS3FileSystemContractTest with this patch.

> Upgrade to JetS3t version 0.7.1
> -------------------------------
>
>                 Key: HADOOP-6146
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6146
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6146.patch
>
>
> The JetS3t library is used for the S3 filesystems. We should upgrade to the latest version (0.7.1) which has support for Requester Pays buckets. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-268-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 13 13:03:29 2009
Return-Path: <common-issues-return-268-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64244 invoked from network); 13 Jul 2009 13:03:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jul 2009 13:03:29 -0000
Received: (qmail 57415 invoked by uid 500); 13 Jul 2009 13:03:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57377 invoked by uid 500); 13 Jul 2009 13:03:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57367 invoked by uid 99); 13 Jul 2009 13:03:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 13:03:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 13:03:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 60101234C052
	for <common-issues@hadoop.apache.org>; Mon, 13 Jul 2009 06:03:15 -0700 (PDT)
Message-ID: <1572263846.1247490195392.JavaMail.jira@brutus>
Date: Mon, 13 Jul 2009 06:03:15 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6146) Upgrade to JetS3t version 0.7.1
In-Reply-To: <13490839.1247490076513.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6146:
------------------------------

    Status: Patch Available  (was: Open)

> Upgrade to JetS3t version 0.7.1
> -------------------------------
>
>                 Key: HADOOP-6146
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6146
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6146.patch
>
>
> The JetS3t library is used for the S3 filesystems. We should upgrade to the latest version (0.7.1) which has support for Requester Pays buckets. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-269-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 13 17:21:28 2009
Return-Path: <common-issues-return-269-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85566 invoked from network); 13 Jul 2009 17:21:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jul 2009 17:21:28 -0000
Received: (qmail 14019 invoked by uid 500); 13 Jul 2009 17:21:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13884 invoked by uid 500); 13 Jul 2009 17:21:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13867 invoked by uid 99); 13 Jul 2009 17:21:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 17:21:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 17:21:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 556CA234C055
	for <common-issues@hadoop.apache.org>; Mon, 13 Jul 2009 10:21:15 -0700 (PDT)
Message-ID: <1687778269.1247505675349.JavaMail.jira@brutus>
Date: Mon, 13 Jul 2009 10:21:15 -0700 (PDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6147) Hadoop Filesystem Plugin for Amazon
 Elastic Block Storage (EBS)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Hadoop Filesystem Plugin for Amazon Elastic Block Storage (EBS)
---------------------------------------------------------------

                 Key: HADOOP-6147
                 URL: https://issues.apache.org/jira/browse/HADOOP-6147
             Project: Hadoop Common
          Issue Type: New Feature
            Reporter: Sanjay Radia


Amazon has introduced a new storage mechanism called EBS. Supposedly it is a lot more efficient than S3 for Hadoop.
A filesystem plugin for hadoop would we be very useful for the community.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-270-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 13 17:33:37 2009
Return-Path: <common-issues-return-270-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94043 invoked from network); 13 Jul 2009 17:33:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jul 2009 17:33:37 -0000
Received: (qmail 35335 invoked by uid 500); 13 Jul 2009 17:33:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35299 invoked by uid 500); 13 Jul 2009 17:33:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35289 invoked by uid 99); 13 Jul 2009 17:33:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 17:33:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 17:33:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EF47329A001B
	for <common-issues@hadoop.apache.org>; Mon, 13 Jul 2009 10:33:14 -0700 (PDT)
Message-ID: <1863036025.1247506394979.JavaMail.jira@brutus>
Date: Mon, 13 Jul 2009 10:33:14 -0700 (PDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6147) Hadoop Filesystem Plugin for Amazon
 Elastic Block Storage (EBS)
In-Reply-To: <1687778269.1247505675349.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730415#action_12730415 ] 

Sanjay Radia commented on HADOOP-6147:
--------------------------------------

>From what I understand about EBS it will not give local access to storage but was developed by Amazon to be more efficient raw storage.
Cloudera reported that the EBS plugin they have written for their customers, released in early June,  is a lot more efficient than S3.
(See http://apache.sys-con.com/node/997306)
Cloudera how about please contributing this back to the community?
If not any volunteers for writing this plugin?

> Hadoop Filesystem Plugin for Amazon Elastic Block Storage (EBS)
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6147
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6147
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sanjay Radia
>
> Amazon has introduced a new storage mechanism called EBS. Supposedly it is a lot more efficient than S3 for Hadoop.
> A filesystem plugin for hadoop would we be very useful for the community.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-271-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 13 17:35:30 2009
Return-Path: <common-issues-return-271-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95953 invoked from network); 13 Jul 2009 17:35:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jul 2009 17:35:30 -0000
Received: (qmail 38141 invoked by uid 500); 13 Jul 2009 17:35:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38051 invoked by uid 500); 13 Jul 2009 17:35:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37924 invoked by uid 99); 13 Jul 2009 17:35:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 17:35:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 17:35:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 084A2234C1E7
	for <common-issues@hadoop.apache.org>; Mon, 13 Jul 2009 10:35:15 -0700 (PDT)
Message-ID: <1396993252.1247506515031.JavaMail.jira@brutus>
Date: Mon, 13 Jul 2009 10:35:15 -0700 (PDT)
From: "Abdul Qadeer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4012) Providing splitting support for
 bzip2 compressed files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730418#action_12730418 ] 

Abdul Qadeer commented on HADOOP-4012:
--------------------------------------

I am making a new patch to accommodate concerns raised by Chris and Owen.  Few files are in the "common" sub-project while the rest are in the "mapreduce".  Since the older Hadoop project has been split into 3 sub-projects, how should I make the new patch?

Should I make two patches or merge them to make one?

> Providing splitting support for bzip2 compressed files
> ------------------------------------------------------
>
>                 Key: HADOOP-4012
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4012
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Abdul Qadeer
>            Assignee: Abdul Qadeer
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-4012-version1.patch, Hadoop-4012-version2.patch, Hadoop-4012-version3.patch, Hadoop-4012-version4.patch, Hadoop-4012-version5.patch, Hadoop-4012-version6.patch, Hadoop-4012-version7.patch, Hadoop-4012-version8.patch, Hadoop-4012-version9.patch
>
>
> Hadoop assumes that if the input data is compressed, it can not be split (mainly due to the limitation of many codecs that they need the whole input stream to decompress successfully).  So in such a case, Hadoop prepares only one split per compressed file, where the lower split limit is at 0 while the upper limit is the end of the file.  The consequence of this decision is that, one compress file goes to a single mapper. Although it circumvents the limitation of codecs (as mentioned above) but reduces the parallelism substantially, as it was possible otherwise in case of splitting.
> BZip2 is a compression / De-Compression algorithm which does compression on blocks of data and later these compressed blocks can be decompressed independent of each other.  This is indeed an opportunity that instead of one BZip2 compressed file going to one mapper, we can process chunks of file in parallel.  The correctness criteria of such a processing is that for a bzip2 compressed file, each compressed block should be processed by only one mapper and ultimately all the blocks of the file should be processed.  (By processing we mean the actual utilization of that un-compressed data (coming out of the codecs) in a mapper).
> We are writing the code to implement this suggested functionality.  Although we have used bzip2 as an example, but we have tried to extend Hadoop's compression interfaces so that any other codecs with the same capability as that of bzip2, could easily use the splitting support.  The details of these changes will be posted when we submit the code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-272-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 13 17:45:30 2009
Return-Path: <common-issues-return-272-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 407 invoked from network); 13 Jul 2009 17:45:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jul 2009 17:45:30 -0000
Received: (qmail 47554 invoked by uid 500); 13 Jul 2009 17:45:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47514 invoked by uid 500); 13 Jul 2009 17:45:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47504 invoked by uid 99); 13 Jul 2009 17:45:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 17:45:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 17:45:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4465D234C004
	for <common-issues@hadoop.apache.org>; Mon, 13 Jul 2009 10:45:15 -0700 (PDT)
Message-ID: <931975189.1247507115266.JavaMail.jira@brutus>
Date: Mon, 13 Jul 2009 10:45:15 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Moved: (HADOOP-6148) Implement a pure Java CRC32 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon moved HDFS-297 to HADOOP-6148:
------------------------------------------

        Key: HADOOP-6148  (was: HDFS-297)
    Project: Hadoop Common  (was: Hadoop HDFS)

> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-273-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 13 17:49:27 2009
Return-Path: <common-issues-return-273-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2394 invoked from network); 13 Jul 2009 17:49:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jul 2009 17:49:27 -0000
Received: (qmail 51922 invoked by uid 500); 13 Jul 2009 17:49:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51884 invoked by uid 500); 13 Jul 2009 17:49:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51874 invoked by uid 99); 13 Jul 2009 17:49:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 17:49:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 17:49:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DF6BA234C052
	for <common-issues@hadoop.apache.org>; Mon, 13 Jul 2009 10:49:14 -0700 (PDT)
Message-ID: <222326463.1247507354914.JavaMail.jira@brutus>
Date: Mon, 13 Jul 2009 10:49:14 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730423#action_12730423 ] 

Todd Lipcon commented on HADOOP-6148:
-------------------------------------

Moved this back to Common since this patch is for util. Just to make sure, I ran the current patch with 1M iterations instead of just 24 and it still passed (but took 44 seconds).

I believe the IP issues look to be clean, so this should be ready for commit.

> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-274-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 13 18:51:34 2009
Return-Path: <common-issues-return-274-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52084 invoked from network); 13 Jul 2009 18:51:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jul 2009 18:51:33 -0000
Received: (qmail 63671 invoked by uid 500); 13 Jul 2009 18:51:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63613 invoked by uid 500); 13 Jul 2009 18:51:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63573 invoked by uid 99); 13 Jul 2009 18:51:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 18:51:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 18:51:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DE8D9234C04B
	for <common-issues@hadoop.apache.org>; Mon, 13 Jul 2009 11:51:14 -0700 (PDT)
Message-ID: <933768347.1247511074910.JavaMail.jira@brutus>
Date: Mon, 13 Jul 2009 11:51:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6146) Upgrade to JetS3t version 0.7.1
In-Reply-To: <13490839.1247490076513.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730458#action_12730458 ] 

Hadoop QA commented on HADOOP-6146:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12413292/HADOOP-6146.patch
  against trunk revision 793162.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/570/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/570/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/570/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/570/console

This message is automatically generated.

> Upgrade to JetS3t version 0.7.1
> -------------------------------
>
>                 Key: HADOOP-6146
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6146
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6146.patch
>
>
> The JetS3t library is used for the S3 filesystems. We should upgrade to the latest version (0.7.1) which has support for Requester Pays buckets. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-275-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 13 18:57:28 2009
Return-Path: <common-issues-return-275-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58100 invoked from network); 13 Jul 2009 18:57:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jul 2009 18:57:28 -0000
Received: (qmail 73735 invoked by uid 500); 13 Jul 2009 18:57:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73699 invoked by uid 500); 13 Jul 2009 18:57:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73681 invoked by uid 99); 13 Jul 2009 18:57:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 18:57:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 18:57:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 42517234C004
	for <common-issues@hadoop.apache.org>; Mon, 13 Jul 2009 11:57:15 -0700 (PDT)
Message-ID: <1699106268.1247511435257.JavaMail.jira@brutus>
Date: Mon, 13 Jul 2009 11:57:15 -0700 (PDT)
From: "Jay Booth (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6147) Hadoop Filesystem Plugin for Amazon
 Elastic Block Storage (EBS)
In-Reply-To: <1687778269.1247505675349.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730460#action_12730460 ] 

Jay Booth commented on HADOOP-6147:
-----------------------------------

It's more efficient raw storage and you can mount it however you want -- from what I understand (only having messed with it very little), this would be something in how you set up your AMI, i.e. you set it up with a couple EBS stores that are striped RAID 0 and it will outperform the 'ephemeral' storage that you get for free with the instance.  Major gain is that since you pay by gigabyte, you can do 4 5GB stores for the same price as 1 20GB store, so you can go either RAID 0 or RAID 5 depending on how you feel about node failure, and get faster IO for the same (or almost the same) cost per GB.  After that setup's done, though, you just mount it to /mnt or /opt or wherever and use it like you would any other device, so I don't see much need to hadoop specific code.  If I had to guess, what cloudera's distributing is a custom AMI, not any significant modification to Hadoop.  Any of the above could be way off, though.

> Hadoop Filesystem Plugin for Amazon Elastic Block Storage (EBS)
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6147
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6147
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sanjay Radia
>
> Amazon has introduced a new storage mechanism called EBS. Supposedly it is a lot more efficient than S3 for Hadoop.
> A filesystem plugin for hadoop would we be very useful for the community.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-276-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 13 20:29:30 2009
Return-Path: <common-issues-return-276-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99308 invoked from network); 13 Jul 2009 20:29:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jul 2009 20:29:30 -0000
Received: (qmail 52156 invoked by uid 500); 13 Jul 2009 20:29:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52018 invoked by uid 500); 13 Jul 2009 20:29:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51913 invoked by uid 99); 13 Jul 2009 20:29:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 20:29:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 20:29:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DB3EF234C055
	for <common-issues@hadoop.apache.org>; Mon, 13 Jul 2009 13:29:14 -0700 (PDT)
Message-ID: <344507574.1247516954897.JavaMail.jira@brutus>
Date: Mon, 13 Jul 2009 13:29:14 -0700 (PDT)
From: "Jeff Hammerbacher (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6147) Hadoop Filesystem Plugin for Amazon
 Elastic Block Storage (EBS)
In-Reply-To: <1687778269.1247505675349.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jeff Hammerbacher resolved HADOOP-6147.
---------------------------------------

    Resolution: Duplicate

Dupe of https://issues.apache.org/jira/browse/HADOOP-6108.

> Hadoop Filesystem Plugin for Amazon Elastic Block Storage (EBS)
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6147
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6147
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sanjay Radia
>
> Amazon has introduced a new storage mechanism called EBS. Supposedly it is a lot more efficient than S3 for Hadoop.
> A filesystem plugin for hadoop would we be very useful for the community.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-277-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 13 22:56:31 2009
Return-Path: <common-issues-return-277-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70162 invoked from network); 13 Jul 2009 22:56:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jul 2009 22:56:30 -0000
Received: (qmail 56021 invoked by uid 500); 13 Jul 2009 22:56:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55880 invoked by uid 500); 13 Jul 2009 22:56:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55866 invoked by uid 99); 13 Jul 2009 22:56:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 22:56:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 22:56:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 2E80E234C1E7
	for <common-issues@hadoop.apache.org>; Mon, 13 Jul 2009 15:56:15 -0700 (PDT)
Message-ID: <1197158103.1247525775189.JavaMail.jira@brutus>
Date: Mon, 13 Jul 2009 15:56:15 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6145) No error message for deleting
 non-existant file or directory.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730574#action_12730574 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6145:
------------------------------------------------

srcFs.exists(src) and srcFs.isDirectory(src) both call srcFs.getFileStatus(src).  We could call srcFs.getFileStatus(src) once in FsShell.delete(..) and save a rpc.

> No error message for deleting non-existant file or directory.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6145
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6145
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.1
>            Reporter: Suman Sehgal
>            Assignee: Jakob Homan
>         Attachments: COMMON-6145.patch, HADOOP-6145.patch
>
>
> If non-existant path or src is provided with rm/rmr option then no error message is displayed 
> command: hadoop dfs -rm <src>  (where src is non-existant)
> dfs displays "rm:  <src>"
> while it should display "No such file or directory".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-278-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 13 23:58:30 2009
Return-Path: <common-issues-return-278-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84674 invoked from network); 13 Jul 2009 23:58:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Jul 2009 23:58:29 -0000
Received: (qmail 28222 invoked by uid 500); 13 Jul 2009 23:58:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28172 invoked by uid 500); 13 Jul 2009 23:58:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28010 invoked by uid 99); 13 Jul 2009 23:58:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 23:58:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jul 2009 23:58:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CA294234C004
	for <common-issues@hadoop.apache.org>; Mon, 13 Jul 2009 16:58:14 -0700 (PDT)
Message-ID: <1356270373.1247529494813.JavaMail.jira@brutus>
Date: Mon, 13 Jul 2009 16:58:14 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6145) No error message for deleting
 non-existant file or directory.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6145:
--------------------------------

    Status: Open  (was: Patch Available)

canceling patch

> No error message for deleting non-existant file or directory.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6145
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6145
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.1
>            Reporter: Suman Sehgal
>            Assignee: Jakob Homan
>         Attachments: COMMON-6145.patch, HADOOP-6145.patch
>
>
> If non-existant path or src is provided with rm/rmr option then no error message is displayed 
> command: hadoop dfs -rm <src>  (where src is non-existant)
> dfs displays "rm:  <src>"
> while it should display "No such file or directory".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-279-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 14 00:00:28 2009
Return-Path: <common-issues-return-279-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85396 invoked from network); 14 Jul 2009 00:00:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jul 2009 00:00:27 -0000
Received: (qmail 28611 invoked by uid 500); 14 Jul 2009 00:00:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28574 invoked by uid 500); 14 Jul 2009 00:00:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28564 invoked by uid 99); 14 Jul 2009 00:00:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 00:00:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 00:00:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DF16D234C055
	for <common-issues@hadoop.apache.org>; Mon, 13 Jul 2009 17:00:14 -0700 (PDT)
Message-ID: <1177421481.1247529614912.JavaMail.jira@brutus>
Date: Mon, 13 Jul 2009 17:00:14 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6145) No error message for deleting
 non-existant file or directory.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6145:
--------------------------------

    Attachment: COMMON-6145.patch

Attaching new Common patch for hudson.

> No error message for deleting non-existant file or directory.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6145
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6145
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.1
>            Reporter: Suman Sehgal
>            Assignee: Jakob Homan
>         Attachments: COMMON-6145.patch, COMMON-6145.patch, HADOOP-6145.patch, HADOOP-6145.patch
>
>
> If non-existant path or src is provided with rm/rmr option then no error message is displayed 
> command: hadoop dfs -rm <src>  (where src is non-existant)
> dfs displays "rm:  <src>"
> while it should display "No such file or directory".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-281-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 14 00:00:29 2009
Return-Path: <common-issues-return-281-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85468 invoked from network); 14 Jul 2009 00:00:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jul 2009 00:00:29 -0000
Received: (qmail 28741 invoked by uid 500); 14 Jul 2009 00:00:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28704 invoked by uid 500); 14 Jul 2009 00:00:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28694 invoked by uid 99); 14 Jul 2009 00:00:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 00:00:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 00:00:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E4D4829A0011
	for <common-issues@hadoop.apache.org>; Mon, 13 Jul 2009 17:00:14 -0700 (PDT)
Message-ID: <716705018.1247529614936.JavaMail.jira@brutus>
Date: Mon, 13 Jul 2009 17:00:14 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6145) No error message for deleting
 non-existant file or directory.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6145:
--------------------------------

    Status: Patch Available  (was: Open)

submitting patch

> No error message for deleting non-existant file or directory.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6145
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6145
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.1
>            Reporter: Suman Sehgal
>            Assignee: Jakob Homan
>         Attachments: COMMON-6145.patch, COMMON-6145.patch, HADOOP-6145.patch, HADOOP-6145.patch
>
>
> If non-existant path or src is provided with rm/rmr option then no error message is displayed 
> command: hadoop dfs -rm <src>  (where src is non-existant)
> dfs displays "rm:  <src>"
> while it should display "No such file or directory".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-280-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 14 00:00:29 2009
Return-Path: <common-issues-return-280-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85447 invoked from network); 14 Jul 2009 00:00:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jul 2009 00:00:29 -0000
Received: (qmail 28678 invoked by uid 500); 14 Jul 2009 00:00:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28640 invoked by uid 500); 14 Jul 2009 00:00:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28630 invoked by uid 99); 14 Jul 2009 00:00:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 00:00:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 00:00:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D56BD234C04B
	for <common-issues@hadoop.apache.org>; Mon, 13 Jul 2009 17:00:14 -0700 (PDT)
Message-ID: <1508728355.1247529614865.JavaMail.jira@brutus>
Date: Mon, 13 Jul 2009 17:00:14 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6145) No error message for deleting
 non-existant file or directory.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6145:
--------------------------------

    Attachment: HADOOP-6145.patch

attaching new v20 patch 

> No error message for deleting non-existant file or directory.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6145
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6145
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.1
>            Reporter: Suman Sehgal
>            Assignee: Jakob Homan
>         Attachments: COMMON-6145.patch, COMMON-6145.patch, HADOOP-6145.patch, HADOOP-6145.patch
>
>
> If non-existant path or src is provided with rm/rmr option then no error message is displayed 
> command: hadoop dfs -rm <src>  (where src is non-existant)
> dfs displays "rm:  <src>"
> while it should display "No such file or directory".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-282-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 14 03:29:13 2009
Return-Path: <common-issues-return-282-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67393 invoked from network); 14 Jul 2009 03:29:13 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jul 2009 03:29:13 -0000
Received: (qmail 25517 invoked by uid 500); 14 Jul 2009 03:29:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25344 invoked by uid 500); 14 Jul 2009 03:29:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25314 invoked by uid 99); 14 Jul 2009 03:29:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 03:29:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 03:29:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E5D05234C04B
	for <common-issues@hadoop.apache.org>; Mon, 13 Jul 2009 20:28:58 -0700 (PDT)
Message-ID: <499719900.1247542138940.JavaMail.jira@brutus>
Date: Mon, 13 Jul 2009 20:28:58 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6145) No error message for deleting
 non-existant file or directory.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730668#action_12730668 ] 

Hadoop QA commented on HADOOP-6145:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12413359/COMMON-6145.patch
  against trunk revision 793162.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/571/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/571/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/571/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/571/console

This message is automatically generated.

> No error message for deleting non-existant file or directory.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6145
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6145
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.1
>            Reporter: Suman Sehgal
>            Assignee: Jakob Homan
>         Attachments: COMMON-6145.patch, COMMON-6145.patch, HADOOP-6145.patch, HADOOP-6145.patch
>
>
> If non-existant path or src is provided with rm/rmr option then no error message is displayed 
> command: hadoop dfs -rm <src>  (where src is non-existant)
> dfs displays "rm:  <src>"
> while it should display "No such file or directory".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-283-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 14 07:57:29 2009
Return-Path: <common-issues-return-283-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51953 invoked from network); 14 Jul 2009 07:57:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jul 2009 07:57:29 -0000
Received: (qmail 20056 invoked by uid 500); 14 Jul 2009 07:57:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20018 invoked by uid 500); 14 Jul 2009 07:57:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20001 invoked by uid 99); 14 Jul 2009 07:57:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 07:57:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 07:57:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 401BF234C004
	for <common-issues@hadoop.apache.org>; Tue, 14 Jul 2009 00:57:15 -0700 (PDT)
Message-ID: <747032300.1247558235248.JavaMail.jira@brutus>
Date: Tue, 14 Jul 2009 00:57:15 -0700 (PDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6120) Add support for Avro types in hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sharad Agarwal updated HADOOP-6120:
-----------------------------------

    Attachment: 6120_v2.patch

This patch:
- addresses Doug's concerns
- moves the common code in abstract base class AvroSerialization
- implements 'reflect' serialization based on Reflect#getSchema. The accept call is based on two mechanisms: 
     tag interface - AvroReflectSerializable.
     package names. Applications can configure the package to be considered as reflect type. Applications then can use it with their existing classes without the need for implementing AvroReflectSerializable interface.
- adds testcase


> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-284-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 14 08:27:30 2009
Return-Path: <common-issues-return-284-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59793 invoked from network); 14 Jul 2009 08:27:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jul 2009 08:27:30 -0000
Received: (qmail 46017 invoked by uid 500); 14 Jul 2009 08:27:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45980 invoked by uid 500); 14 Jul 2009 08:27:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45970 invoked by uid 99); 14 Jul 2009 08:27:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 08:27:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 08:27:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 39B53234C004
	for <common-issues@hadoop.apache.org>; Tue, 14 Jul 2009 01:27:15 -0700 (PDT)
Message-ID: <1981967536.1247560035231.JavaMail.jira@brutus>
Date: Tue, 14 Jul 2009 01:27:15 -0700 (PDT)
From: "Sreekanth Ramakrishnan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5420) Support killing of process groups in
 LinuxTaskController binary
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sreekanth Ramakrishnan updated HADOOP-5420:
-------------------------------------------

    Attachment: 5420-fix-ydist.patch

Fix for issue which was found internally.

> Support killing of process groups in LinuxTaskController binary
> ---------------------------------------------------------------
>
>                 Key: HADOOP-5420
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5420
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Sreekanth Ramakrishnan
>            Assignee: Sreekanth Ramakrishnan
>             Fix For: 0.21.0
>
>         Attachments: 5420-fix-ydist.patch, hadoop-5420-1.patch, hadoop-5420-10.patch, hadoop-5420-11.patch, hadoop-5420-12.patch, hadoop-5420-2.patch, hadoop-5420-3.patch, hadoop-5420-4.patch, hadoop-5420-5.patch, hadoop-5420-6.patch, hadoop-5420-7.patch, hadoop-5420-8.patch, hadoop-5420-9.patch, HADOOP-5420-v20.patch, hadoop-5420.patch
>
>
> Support setsid based kill in LinuxTaskController.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-285-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 14 09:29:45 2009
Return-Path: <common-issues-return-285-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78064 invoked from network); 14 Jul 2009 09:29:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jul 2009 09:29:45 -0000
Received: (qmail 11090 invoked by uid 500); 14 Jul 2009 09:29:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11048 invoked by uid 500); 14 Jul 2009 09:29:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11038 invoked by uid 99); 14 Jul 2009 09:29:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 09:29:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 09:29:51 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id BE7C1234C004
	for <common-issues@hadoop.apache.org>; Tue, 14 Jul 2009 02:29:30 -0700 (PDT)
Message-ID: <441814058.1247563770766.JavaMail.jira@brutus>
Date: Tue, 14 Jul 2009 02:29:30 -0700 (PDT)
From: "He Yongqiang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5879) GzipCodec should read compression
 level etc from configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

He Yongqiang updated HADOOP-5879:
---------------------------------

    Status: Patch Available  (was: Open)

> GzipCodec should read compression level etc from configuration
> --------------------------------------------------------------
>
>                 Key: HADOOP-5879
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5879
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Zheng Shao
>         Attachments: hadoop-5879-5-21.patch, hadoop-5879-7-13-2.patch, hadoop-5879-7-13-3.patch
>
>
> GzipCodec currently uses the default compression level. We should allow overriding the default value from Configuration.
> {code}
>   static final class GzipZlibCompressor extends ZlibCompressor {
>     public GzipZlibCompressor() {
>       super(ZlibCompressor.CompressionLevel.DEFAULT_COMPRESSION,
>           ZlibCompressor.CompressionStrategy.DEFAULT_STRATEGY,
>           ZlibCompressor.CompressionHeader.GZIP_FORMAT, 64*1024);
>     }
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-286-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 14 11:31:11 2009
Return-Path: <common-issues-return-286-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33116 invoked from network); 14 Jul 2009 11:31:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jul 2009 11:31:11 -0000
Received: (qmail 69014 invoked by uid 500); 14 Jul 2009 11:17:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68996 invoked by uid 500); 14 Jul 2009 11:17:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68986 invoked by uid 99); 14 Jul 2009 11:17:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 11:17:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 11:17:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3B4B7234C004
	for <common-issues@hadoop.apache.org>; Tue, 14 Jul 2009 04:17:15 -0700 (PDT)
Message-ID: <660338592.1247570235229.JavaMail.jira@brutus>
Date: Tue, 14 Jul 2009 04:17:15 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5879) GzipCodec should read compression
 level etc from configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730787#action_12730787 ] 

Hadoop QA commented on HADOOP-5879:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12413291/hadoop-5879-7-13-3.patch
  against trunk revision 793162.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs.  The patch appears to introduce 4 new Findbugs warnings.

    -1 release audit.  The applied patch generated 266 release audit warnings (more than the trunk's current 260 warnings).

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/572/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/572/artifact/trunk/current/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/572/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/572/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/572/console

This message is automatically generated.

> GzipCodec should read compression level etc from configuration
> --------------------------------------------------------------
>
>                 Key: HADOOP-5879
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5879
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Zheng Shao
>         Attachments: hadoop-5879-5-21.patch, hadoop-5879-7-13-2.patch, hadoop-5879-7-13-3.patch
>
>
> GzipCodec currently uses the default compression level. We should allow overriding the default value from Configuration.
> {code}
>   static final class GzipZlibCompressor extends ZlibCompressor {
>     public GzipZlibCompressor() {
>       super(ZlibCompressor.CompressionLevel.DEFAULT_COMPRESSION,
>           ZlibCompressor.CompressionStrategy.DEFAULT_STRATEGY,
>           ZlibCompressor.CompressionHeader.GZIP_FORMAT, 64*1024);
>     }
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-287-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 14 17:13:33 2009
Return-Path: <common-issues-return-287-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84458 invoked from network); 14 Jul 2009 17:13:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jul 2009 17:13:29 -0000
Received: (qmail 95642 invoked by uid 500); 14 Jul 2009 17:13:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95604 invoked by uid 500); 14 Jul 2009 17:13:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95587 invoked by uid 99); 14 Jul 2009 17:13:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 17:13:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 17:13:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E78C9234C004
	for <common-issues@hadoop.apache.org>; Tue, 14 Jul 2009 10:13:14 -0700 (PDT)
Message-ID: <2015748604.1247591594934.JavaMail.jira@brutus>
Date: Tue, 14 Jul 2009 10:13:14 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6120) Add support for Avro types in
 hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730978#action_12730978 ] 

Doug Cutting commented on HADOOP-6120:
--------------------------------------

Looks good.  Some comments:
 - in AvroSerialization, s/GenericDatum{Reader,Writer}/Datum{Reader,Writer}/?  All implementations will probably always use a subclass of Generic, but it feels to me like this API should be more abstract, no?
 - in AvroReflectSerialization, 'List<String> packages' might be more efficient as Set<String>?
 - in AvroReflectSerialization, getPackages() could use Configuration#getStrings().
 - does Avro's reflect API actually require that fields are public?  Specific always generates public fields, but I think reflect should be able to serialize and deserialize private fields.  If not, maybe we should fix that in Avro?

> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-288-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 14 17:33:29 2009
Return-Path: <common-issues-return-288-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94871 invoked from network); 14 Jul 2009 17:33:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jul 2009 17:33:29 -0000
Received: (qmail 36694 invoked by uid 500); 14 Jul 2009 17:33:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36634 invoked by uid 500); 14 Jul 2009 17:33:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36521 invoked by uid 99); 14 Jul 2009 17:33:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 17:33:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 17:33:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D12D9234C004
	for <common-issues@hadoop.apache.org>; Tue, 14 Jul 2009 10:33:14 -0700 (PDT)
Message-ID: <342813882.1247592794849.JavaMail.jira@brutus>
Date: Tue, 14 Jul 2009 10:33:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6145) No error message for deleting
 non-existant file or directory.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6145:
-------------------------------------------

    Hadoop Flags: [Reviewed]

+1 patch looks good.

> No error message for deleting non-existant file or directory.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6145
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6145
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.1
>            Reporter: Suman Sehgal
>            Assignee: Jakob Homan
>         Attachments: COMMON-6145.patch, COMMON-6145.patch, HADOOP-6145.patch, HADOOP-6145.patch
>
>
> If non-existant path or src is provided with rm/rmr option then no error message is displayed 
> command: hadoop dfs -rm <src>  (where src is non-existant)
> dfs displays "rm:  <src>"
> while it should display "No such file or directory".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-289-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 14 17:43:29 2009
Return-Path: <common-issues-return-289-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3201 invoked from network); 14 Jul 2009 17:43:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jul 2009 17:43:29 -0000
Received: (qmail 55137 invoked by uid 500); 14 Jul 2009 17:43:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55099 invoked by uid 500); 14 Jul 2009 17:43:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55089 invoked by uid 99); 14 Jul 2009 17:43:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 17:43:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 17:43:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 38A74234C004
	for <common-issues@hadoop.apache.org>; Tue, 14 Jul 2009 10:43:15 -0700 (PDT)
Message-ID: <1446638366.1247593395217.JavaMail.jira@brutus>
Date: Tue, 14 Jul 2009 10:43:15 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6145) No error message for deleting
 non-existant file or directory.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6145:
-------------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.21.0
                   0.20.1
           Status: Resolved  (was: Patch Available)

I have committed this to 0.20 and above.  Thanks, Jakob!

> No error message for deleting non-existant file or directory.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6145
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6145
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.1
>            Reporter: Suman Sehgal
>            Assignee: Jakob Homan
>             Fix For: 0.20.1, 0.21.0
>
>         Attachments: COMMON-6145.patch, COMMON-6145.patch, HADOOP-6145.patch, HADOOP-6145.patch
>
>
> If non-existant path or src is provided with rm/rmr option then no error message is displayed 
> command: hadoop dfs -rm <src>  (where src is non-existant)
> dfs displays "rm:  <src>"
> while it should display "No such file or directory".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-290-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 14 20:43:27 2009
Return-Path: <common-issues-return-290-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98585 invoked from network); 14 Jul 2009 20:43:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jul 2009 20:43:27 -0000
Received: (qmail 70610 invoked by uid 500); 14 Jul 2009 20:43:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70573 invoked by uid 500); 14 Jul 2009 20:43:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70563 invoked by uid 99); 14 Jul 2009 20:43:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 20:43:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 20:43:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CDADC234C004
	for <common-issues@hadoop.apache.org>; Tue, 14 Jul 2009 13:43:14 -0700 (PDT)
Message-ID: <1369633779.1247604194835.JavaMail.jira@brutus>
Date: Tue, 14 Jul 2009 13:43:14 -0700 (PDT)
From: "Raghu Angadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731119#action_12731119 ] 

Raghu Angadi commented on HADOOP-6148:
--------------------------------------

Scott and Todd, any plans to inform OpenJDK guys about this? 
 

> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-291-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 14 22:05:30 2009
Return-Path: <common-issues-return-291-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56517 invoked from network); 14 Jul 2009 22:05:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jul 2009 22:05:30 -0000
Received: (qmail 87998 invoked by uid 500); 14 Jul 2009 22:05:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87922 invoked by uid 500); 14 Jul 2009 22:05:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87885 invoked by uid 99); 14 Jul 2009 22:05:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 22:05:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 22:05:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E7A3029A0015
	for <common-issues@hadoop.apache.org>; Tue, 14 Jul 2009 15:05:14 -0700 (PDT)
Message-ID: <1932342868.1247609114947.JavaMail.jira@brutus>
Date: Tue, 14 Jul 2009 15:05:14 -0700 (PDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6149) FileStatus can support a fileid per
 path
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

FileStatus can support a fileid per path
----------------------------------------

                 Key: HADOOP-6149
                 URL: https://issues.apache.org/jira/browse/HADOOP-6149
             Project: Hadoop Common
          Issue Type: New Feature
            Reporter: dhruba borthakur
            Assignee: dhruba borthakur
             Fix For: 0.21.0


FileStatus should expose a id that uniquely identifies a file. This helps in developing applications that work correctly even when files are moved from one directory to another.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-292-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 14 22:09:27 2009
Return-Path: <common-issues-return-292-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58834 invoked from network); 14 Jul 2009 22:09:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Jul 2009 22:09:27 -0000
Received: (qmail 94993 invoked by uid 500); 14 Jul 2009 22:09:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94956 invoked by uid 500); 14 Jul 2009 22:09:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94946 invoked by uid 99); 14 Jul 2009 22:09:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 22:09:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jul 2009 22:09:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E3B4829A0018
	for <common-issues@hadoop.apache.org>; Tue, 14 Jul 2009 15:09:14 -0700 (PDT)
Message-ID: <765623826.1247609354931.JavaMail.jira@brutus>
Date: Tue, 14 Jul 2009 15:09:14 -0700 (PDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6149) FileStatus can support a fileid per
 path
In-Reply-To: <1932342868.1247609114947.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731160#action_12731160 ] 

dhruba borthakur commented on HADOOP-6149:
------------------------------------------

Please see patch attached to HDFS-487.

> FileStatus can support a fileid per path
> ----------------------------------------
>
>                 Key: HADOOP-6149
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6149
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>             Fix For: 0.21.0
>
>
> FileStatus should expose a id that uniquely identifies a file. This helps in developing applications that work correctly even when files are moved from one directory to another.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-293-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 00:10:29 2009
Return-Path: <common-issues-return-293-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16035 invoked from network); 15 Jul 2009 00:10:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 00:10:27 -0000
Received: (qmail 48136 invoked by uid 500); 15 Jul 2009 00:10:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48097 invoked by uid 500); 15 Jul 2009 00:10:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48087 invoked by uid 99); 15 Jul 2009 00:10:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 00:10:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 00:10:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3018129A001D
	for <common-issues@hadoop.apache.org>; Tue, 14 Jul 2009 17:10:15 -0700 (PDT)
Message-ID: <1948836692.1247616615196.JavaMail.jira@brutus>
Date: Tue, 14 Jul 2009 17:10:15 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6148:
-------------------------------------------

    Attachment: benchmarks20090714.txt

benchmarks20090714.txt: benchmarks results

> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: benchmarks20090714.txt, crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-294-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 00:10:29 2009
Return-Path: <common-issues-return-294-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16102 invoked from network); 15 Jul 2009 00:10:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 00:10:29 -0000
Received: (qmail 48219 invoked by uid 500); 15 Jul 2009 00:10:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48177 invoked by uid 500); 15 Jul 2009 00:10:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48167 invoked by uid 99); 15 Jul 2009 00:10:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 00:10:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 00:10:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 104E9234C004
	for <common-issues@hadoop.apache.org>; Tue, 14 Jul 2009 17:10:15 -0700 (PDT)
Message-ID: <139034399.1247616615055.JavaMail.jira@brutus>
Date: Tue, 14 Jul 2009 17:10:15 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6148:
-------------------------------------------

    Attachment: PureJavaCrc32.java

I think the codes could be improved a little bit as following.
{code}
    public void update(byte[] b, int off, final int len) {
      for(final int end = off + (len & ~3); off < end; ) {
        final int c0 = crc ^ b[off++];
        final int c1 = (crc >>>= 8) ^ b[off++];
        final int c2 = (crc >>>= 8) ^ b[off++];
        final int c3 = (crc >>>= 8) ^ b[off++];
        crc = T4[c0 & 0xff] ^ T3[c1 & 0xff] ^ T2[c2 & 0xff] ^ T1[c3 & 0xff];
      }
      
      for(final int end = off + (len & 3); off < end; ) {
        crc = (crc >>> 8) ^ T1[(crc ^ b[off++]) & 0xff];
      }
    }
{code}
PureJavaCrc32.java: I moved the codes around and did some benchmarks.

> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: benchmarks20090714.txt, crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-295-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 00:34:29 2009
Return-Path: <common-issues-return-295-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29158 invoked from network); 15 Jul 2009 00:34:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 00:34:29 -0000
Received: (qmail 59757 invoked by uid 500); 15 Jul 2009 00:34:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59718 invoked by uid 500); 15 Jul 2009 00:34:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59708 invoked by uid 99); 15 Jul 2009 00:34:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 00:34:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 00:34:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CEBCE29A0011
	for <common-issues@hadoop.apache.org>; Tue, 14 Jul 2009 17:34:14 -0700 (PDT)
Message-ID: <1804574098.1247618054845.JavaMail.jira@brutus>
Date: Tue, 14 Jul 2009 17:34:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6150) Need to be able to instantiate a
 comparator instance from a comparator string without creating a
 TFile.Reader object
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Need to be able to instantiate a comparator instance from a comparator string without creating a TFile.Reader object
--------------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-6150
                 URL: https://issues.apache.org/jira/browse/HADOOP-6150
             Project: Hadoop Common
          Issue Type: Improvement
          Components: io
    Affects Versions: 0.20.0, 0.21.0
            Reporter: Hong Tang
            Assignee: Hong Tang
            Priority: Minor


Occasionally, we want have the same instance of comparator object represented by the TFile comparator string. We should be able to do that without requiring to first open up a tfile that was previously written use the same comparator.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-296-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 03:18:30 2009
Return-Path: <common-issues-return-296-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94003 invoked from network); 15 Jul 2009 03:18:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 03:18:30 -0000
Received: (qmail 97366 invoked by uid 500); 15 Jul 2009 03:18:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97330 invoked by uid 500); 15 Jul 2009 03:18:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97317 invoked by uid 99); 15 Jul 2009 03:18:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 03:18:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 03:18:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 36468234C004
	for <common-issues@hadoop.apache.org>; Tue, 14 Jul 2009 20:18:15 -0700 (PDT)
Message-ID: <1178128312.1247627895210.JavaMail.jira@brutus>
Date: Tue, 14 Jul 2009 20:18:15 -0700 (PDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6120) Add support for Avro types in
 hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731254#action_12731254 ] 

Sharad Agarwal commented on HADOOP-6120:
----------------------------------------

bq. All implementations will probably always use a subclass of Generic, but it feels to me like this API should be more abstract, no?
Absolutely. I realized that but missed to use it in the patch.

bq. reflect should be able to serialize and deserialize private fields. If not, maybe we should fix that in Avro?
I have filed AVRO-78. At places the avro code uses Class#getFields()  instead of Class#getDeclaredFields() 

> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-297-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 04:23:29 2009
Return-Path: <common-issues-return-297-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5073 invoked from network); 15 Jul 2009 04:23:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 04:23:29 -0000
Received: (qmail 43926 invoked by uid 500); 15 Jul 2009 04:23:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43888 invoked by uid 500); 15 Jul 2009 04:23:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43878 invoked by uid 99); 15 Jul 2009 04:23:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 04:23:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 04:23:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D0A06234C004
	for <common-issues@hadoop.apache.org>; Tue, 14 Jul 2009 21:23:14 -0700 (PDT)
Message-ID: <78035294.1247631794847.JavaMail.jira@brutus>
Date: Tue, 14 Jul 2009 21:23:14 -0700 (PDT)
From: "Rajiv Chittajallu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5420) Support killing of process groups
 in LinuxTaskController binary
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731262#action_12731262 ] 

Rajiv Chittajallu commented on HADOOP-5420:
-------------------------------------------

bq. Fix for issue which was found internally. 

This issue is marked as Fixed. Open a new JIRA with a description about the issue and attach the patch.

> Support killing of process groups in LinuxTaskController binary
> ---------------------------------------------------------------
>
>                 Key: HADOOP-5420
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5420
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Sreekanth Ramakrishnan
>            Assignee: Sreekanth Ramakrishnan
>             Fix For: 0.21.0
>
>         Attachments: 5420-fix-ydist.patch, hadoop-5420-1.patch, hadoop-5420-10.patch, hadoop-5420-11.patch, hadoop-5420-12.patch, hadoop-5420-2.patch, hadoop-5420-3.patch, hadoop-5420-4.patch, hadoop-5420-5.patch, hadoop-5420-6.patch, hadoop-5420-7.patch, hadoop-5420-8.patch, hadoop-5420-9.patch, HADOOP-5420-v20.patch, hadoop-5420.patch
>
>
> Support setsid based kill in LinuxTaskController.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-298-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 04:47:27 2009
Return-Path: <common-issues-return-298-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8437 invoked from network); 15 Jul 2009 04:47:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 04:47:27 -0000
Received: (qmail 64150 invoked by uid 500); 15 Jul 2009 04:47:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64110 invoked by uid 500); 15 Jul 2009 04:47:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64099 invoked by uid 99); 15 Jul 2009 04:47:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 04:47:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 04:47:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CF0E8234C004
	for <common-issues@hadoop.apache.org>; Tue, 14 Jul 2009 21:47:14 -0700 (PDT)
Message-ID: <2027491984.1247633234834.JavaMail.jira@brutus>
Date: Tue, 14 Jul 2009 21:47:14 -0700 (PDT)
From: "Ramya R (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6113) AccessControlException should display
 the full path
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ramya R updated HADOOP-6113:
----------------------------

    Tags: ygridqa

> AccessControlException should display the full path
> ---------------------------------------------------
>
>                 Key: HADOOP-6113
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6113
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Ramya R
>            Priority: Minor
>
> org.apache.hadoop.security.AccessControlException should display the full path for which the access is denied. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-299-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 04:55:30 2009
Return-Path: <common-issues-return-299-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10313 invoked from network); 15 Jul 2009 04:55:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 04:55:29 -0000
Received: (qmail 67065 invoked by uid 500); 15 Jul 2009 04:55:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66981 invoked by uid 500); 15 Jul 2009 04:55:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66961 invoked by uid 99); 15 Jul 2009 04:55:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 04:55:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 04:55:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3E87629A0014
	for <common-issues@hadoop.apache.org>; Tue, 14 Jul 2009 21:55:15 -0700 (PDT)
Message-ID: <1261770437.1247633715255.JavaMail.jira@brutus>
Date: Tue, 14 Jul 2009 21:55:15 -0700 (PDT)
From: "Suman Sehgal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5323) Trash documentation needs to be more
 elaborated.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suman Sehgal updated HADOOP-5323:
---------------------------------

    Tags: ygridqa

> Trash documentation needs to be more elaborated.
> ------------------------------------------------
>
>                 Key: HADOOP-5323
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5323
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: 0.18.3
>            Reporter: Suman Sehgal
>            Priority: Minor
>
> Trash documentation should mention the significance of "Current" and "<time-stamp>" directories which get generated inside Trash directory. The documentation should also incorporate modifications done in HADOOP: 4970.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-300-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 05:31:31 2009
Return-Path: <common-issues-return-300-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20620 invoked from network); 15 Jul 2009 05:31:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 05:31:31 -0000
Received: (qmail 90454 invoked by uid 500); 15 Jul 2009 05:31:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90412 invoked by uid 500); 15 Jul 2009 05:31:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90402 invoked by uid 99); 15 Jul 2009 05:31:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 05:31:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 05:31:37 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3E2A6234C004
	for <common-issues@hadoop.apache.org>; Tue, 14 Jul 2009 22:31:16 -0700 (PDT)
Message-ID: <1754599294.1247635876247.JavaMail.jira@brutus>
Date: Tue, 14 Jul 2009 22:31:16 -0700 (PDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6120) Add support for Avro types in hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sharad Agarwal updated HADOOP-6120:
-----------------------------------

    Attachment: 6120_v3.patch

Incorporated review comments.

> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch, 6120_v3.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-301-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 05:41:30 2009
Return-Path: <common-issues-return-301-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24600 invoked from network); 15 Jul 2009 05:41:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 05:41:30 -0000
Received: (qmail 98492 invoked by uid 500); 15 Jul 2009 05:41:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98414 invoked by uid 500); 15 Jul 2009 05:41:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98381 invoked by uid 99); 15 Jul 2009 05:41:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 05:41:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 05:41:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D4E27234C004
	for <common-issues@hadoop.apache.org>; Tue, 14 Jul 2009 22:41:14 -0700 (PDT)
Message-ID: <1800047364.1247636474856.JavaMail.jira@brutus>
Date: Tue, 14 Jul 2009 22:41:14 -0700 (PDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6120) Add support for Avro types in hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sharad Agarwal updated HADOOP-6120:
-----------------------------------

    Status: Patch Available  (was: Open)

> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch, 6120_v3.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-302-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 08:27:29 2009
Return-Path: <common-issues-return-302-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81202 invoked from network); 15 Jul 2009 08:27:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 08:27:29 -0000
Received: (qmail 59468 invoked by uid 500); 15 Jul 2009 08:27:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59430 invoked by uid 500); 15 Jul 2009 08:27:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59413 invoked by uid 99); 15 Jul 2009 08:27:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 08:27:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 08:27:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CD584234C004
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 01:27:14 -0700 (PDT)
Message-ID: <1359424504.1247646434825.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 01:27:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6120) Add support for Avro types in
 hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731342#action_12731342 ] 

Hadoop QA commented on HADOOP-6120:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12413524/6120_v3.patch
  against trunk revision 793987.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 9 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs.  The patch appears to cause Findbugs to fail.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    -1 contrib tests.  The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/573/testReport/
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/573/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/573/console

This message is automatically generated.

> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch, 6120_v3.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-303-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 08:49:27 2009
Return-Path: <common-issues-return-303-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85712 invoked from network); 15 Jul 2009 08:49:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 08:49:27 -0000
Received: (qmail 80364 invoked by uid 500); 15 Jul 2009 08:49:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80325 invoked by uid 500); 15 Jul 2009 08:49:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80315 invoked by uid 99); 15 Jul 2009 08:49:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 08:49:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 08:49:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 04880234C045
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 01:49:15 -0700 (PDT)
Message-ID: <955978941.1247647755013.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 01:49:15 -0700 (PDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6120) Add support for Avro types in
 hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731352#action_12731352 ] 

Sharad Agarwal commented on HADOOP-6120:
----------------------------------------

Hudson failed because patch needs avro jar.

All tests passed locally.
test-patch output:
{code}
-1 overall.
     [exec]
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec]
     [exec]     +1 tests included.  The patch appears to include 9 new or modified tests.
     [exec]
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec]
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec]
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec]
     [exec]     -1 release audit.  The applied patch generated 265 release audit warnings (more than the trunk's current 264 warnings).
{code}
Release audit warnings are due to test file avroRecord.avsc not having license header.

> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch, 6120_v3.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-304-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 11:47:28 2009
Return-Path: <common-issues-return-304-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44034 invoked from network); 15 Jul 2009 11:47:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 11:47:27 -0000
Received: (qmail 85394 invoked by uid 500); 15 Jul 2009 11:47:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85356 invoked by uid 500); 15 Jul 2009 11:47:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85344 invoked by uid 99); 15 Jul 2009 11:47:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 11:47:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 11:47:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 39C36234C052
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 04:47:15 -0700 (PDT)
Message-ID: <801390646.1247658435235.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 04:47:15 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6145) No error message for deleting
 non-existant file or directory.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731404#action_12731404 ] 

Hudson commented on HADOOP-6145:
--------------------------------

Integrated in Hadoop-Common-trunk #27 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/27/])
    . Fix FsShell rm/rmr error messages when there is a FNFE.  Contributed by Jakob Homan


> No error message for deleting non-existant file or directory.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6145
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6145
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.1
>            Reporter: Suman Sehgal
>            Assignee: Jakob Homan
>             Fix For: 0.20.1, 0.21.0
>
>         Attachments: COMMON-6145.patch, COMMON-6145.patch, HADOOP-6145.patch, HADOOP-6145.patch
>
>
> If non-existant path or src is provided with rm/rmr option then no error message is displayed 
> command: hadoop dfs -rm <src>  (where src is non-existant)
> dfs displays "rm:  <src>"
> while it should display "No such file or directory".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-305-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 12:57:29 2009
Return-Path: <common-issues-return-305-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86297 invoked from network); 15 Jul 2009 12:57:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 12:57:29 -0000
Received: (qmail 89182 invoked by uid 500); 15 Jul 2009 12:57:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89139 invoked by uid 500); 15 Jul 2009 12:57:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89128 invoked by uid 99); 15 Jul 2009 12:57:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 12:57:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 12:57:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D18F7234C004
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 05:57:14 -0700 (PDT)
Message-ID: <2110270720.1247662634844.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 05:57:14 -0700 (PDT)
From: "He Yongqiang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6138) eliminate the depracate warnings
 introduced by H-5438
In-Reply-To: <883885751.1247164334967.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

He Yongqiang updated HADOOP-6138:
---------------------------------

    Attachment: hadoop-6138-2009-07-15.patch

Sorry for the delay. Will open new jira for hdfs and mapreduce.

> eliminate the depracate warnings introduced by H-5438
> -----------------------------------------------------
>
>                 Key: HADOOP-6138
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6138
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: He Yongqiang
>            Priority: Minor
>         Attachments: hadoop-6138-2009-07-15.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-306-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 13:23:30 2009
Return-Path: <common-issues-return-306-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96675 invoked from network); 15 Jul 2009 13:23:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 13:23:29 -0000
Received: (qmail 31785 invoked by uid 500); 15 Jul 2009 13:23:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31753 invoked by uid 500); 15 Jul 2009 13:23:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31743 invoked by uid 99); 15 Jul 2009 13:23:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 13:23:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 13:23:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CBCE4234C004
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 06:23:14 -0700 (PDT)
Message-ID: <113943841.1247664194820.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 06:23:14 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6120) Add support for Avro types in
 hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731433#action_12731433 ] 

Tom White commented on HADOOP-6120:
-----------------------------------

Looks good. Minor nits:

* Move the testSerialization() method from TestWritableSerialization since it is now being used from the Avro tests. Something like SerializationTestUtil.
* Add a package.html to the org.apache.hadoop.io.serializer.avro package.

Have you tried using either of these serializations with MapReduce? Something like TestJavaSerialization would be good (but this could be another issue).

> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch, 6120_v3.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-307-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 13:33:29 2009
Return-Path: <common-issues-return-307-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2649 invoked from network); 15 Jul 2009 13:33:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 13:33:29 -0000
Received: (qmail 57861 invoked by uid 500); 15 Jul 2009 13:33:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57830 invoked by uid 500); 15 Jul 2009 13:33:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57820 invoked by uid 99); 15 Jul 2009 13:33:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 13:33:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 13:33:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E5295234C004
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 06:33:14 -0700 (PDT)
Message-ID: <868654812.1247664794926.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 06:33:14 -0700 (PDT)
From: "He Yongqiang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6138) eliminate the depracate warnings
 introduced by H-5438
In-Reply-To: <883885751.1247164334967.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731436#action_12731436 ] 

He Yongqiang commented on HADOOP-6138:
--------------------------------------

filed hdfs-490 and mapreduce-765 accordingly.
https://issues.apache.org/jira/browse/HDFS-490
https://issues.apache.org/jira/browse/MAPREDUCE-765

> eliminate the depracate warnings introduced by H-5438
> -----------------------------------------------------
>
>                 Key: HADOOP-6138
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6138
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: He Yongqiang
>            Priority: Minor
>         Attachments: hadoop-6138-2009-07-15.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-308-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 16:26:29 2009
Return-Path: <common-issues-return-308-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81976 invoked from network); 15 Jul 2009 16:26:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 16:26:29 -0000
Received: (qmail 29675 invoked by uid 500); 15 Jul 2009 16:26:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29653 invoked by uid 500); 15 Jul 2009 16:26:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29643 invoked by uid 99); 15 Jul 2009 16:26:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 16:26:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 16:26:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D1B01234C045
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 09:26:14 -0700 (PDT)
Message-ID: <2103411870.1247675174858.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 09:26:14 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6151) The servlets should quote html
 characters
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

The servlets should quote html characters
-----------------------------------------

                 Key: HADOOP-6151
                 URL: https://issues.apache.org/jira/browse/HADOOP-6151
             Project: Hadoop Common
          Issue Type: Bug
          Components: security
            Reporter: Owen O'Malley


We need to quote html characters that come from user generated data. Otherwise, all of the web ui's have cross site scripting attack, etc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-309-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 16:53:27 2009
Return-Path: <common-issues-return-309-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90825 invoked from network); 15 Jul 2009 16:53:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 16:53:27 -0000
Received: (qmail 75892 invoked by uid 500); 15 Jul 2009 16:53:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75855 invoked by uid 500); 15 Jul 2009 16:53:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75843 invoked by uid 99); 15 Jul 2009 16:53:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 16:53:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 16:53:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CA578234C004
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 09:53:14 -0700 (PDT)
Message-ID: <1315763494.1247676794816.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 09:53:14 -0700 (PDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5420) Support killing of process groups
 in LinuxTaskController binary
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731554#action_12731554 ] 

Hemanth Yamijala commented on HADOOP-5420:
------------------------------------------

bq. This issue is marked as Fixed. Open a new JIRA with a description about the issue and attach the patch.

Rajiv, the reason why there is no new jira is because this is a bug that occurs only on the Yahoo! distribution and not on trunk.

To give an explanation, the patch HADOOP-5420-v20.patch introduced some code to make sure the pid files written by the task-controller were owned by the tasktracker process, as a security check. Inadvertently, in this patch, we removed some code that changed the ownership of the pid file (which was written as root) to be owned by the TT user. As a result, pid files were created as root, but the new check introduced in the patch failed during kill because it found the PID files were not owned by the TT user and hence treated them as suspect. Hence tasks failed to be killed causing runaway processes on the cluster.

The attached patch re-introduces the code that changes ownership of the pid file to the TT user so that during killing the security check would pass and processes would be killed.

> Support killing of process groups in LinuxTaskController binary
> ---------------------------------------------------------------
>
>                 Key: HADOOP-5420
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5420
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Sreekanth Ramakrishnan
>            Assignee: Sreekanth Ramakrishnan
>             Fix For: 0.21.0
>
>         Attachments: 5420-fix-ydist.patch, hadoop-5420-1.patch, hadoop-5420-10.patch, hadoop-5420-11.patch, hadoop-5420-12.patch, hadoop-5420-2.patch, hadoop-5420-3.patch, hadoop-5420-4.patch, hadoop-5420-5.patch, hadoop-5420-6.patch, hadoop-5420-7.patch, hadoop-5420-8.patch, hadoop-5420-9.patch, HADOOP-5420-v20.patch, hadoop-5420.patch
>
>
> Support setsid based kill in LinuxTaskController.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-310-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 17:05:28 2009
Return-Path: <common-issues-return-310-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94013 invoked from network); 15 Jul 2009 17:05:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 17:05:27 -0000
Received: (qmail 92322 invoked by uid 500); 15 Jul 2009 17:05:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92298 invoked by uid 500); 15 Jul 2009 17:05:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92283 invoked by uid 99); 15 Jul 2009 17:05:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 17:05:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 17:05:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 2CCE4234C051
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 10:05:15 -0700 (PDT)
Message-ID: <1736966924.1247677515182.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 10:05:15 -0700 (PDT)
From: "Philip Zeyliger (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6105) Provide a way to automatically
 handle backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731560#action_12731560 ] 

Philip Zeyliger commented on HADOOP-6105:
-----------------------------------------

Owen,

Apologies for missing this e-mail for so long.  I'm behind on the "all-jira" bucket, and I failed to set a watch.

Hemanth, you should definitely forge ahead with the simple, expedient solution.  

I'd like to convince you and Owen that the more complicated proposal is a net win (and I've used a similar system in the past), but I think the best way to do that is to actually write the code and transform a few usages.  I've been busy with some other deadlines, so when I get there, I'll file a JIRA and bother you all again.

(To answer Owen's questions: the couple of classes for ConfigVariable go into the configuration package; users are welcome to use the same classes to set their variables, or they can set them manually; the documentation for the variables themselves is generated, the documentation for the system lives in JavaDoc on the individual classes and the package.)

-- Philip



> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-311-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 18:35:38 2009
Return-Path: <common-issues-return-311-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55110 invoked from network); 15 Jul 2009 18:35:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 18:35:38 -0000
Received: (qmail 40088 invoked by uid 500); 15 Jul 2009 18:20:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40071 invoked by uid 500); 15 Jul 2009 18:20:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40060 invoked by uid 99); 15 Jul 2009 18:20:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 18:20:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 18:20:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CBE75234C004
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 11:20:14 -0700 (PDT)
Message-ID: <1176339992.1247682014821.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 11:20:14 -0700 (PDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6146) Upgrade to JetS3t version 0.7.1
In-Reply-To: <13490839.1247490076513.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731604#action_12731604 ] 

Aaron Kimball commented on HADOOP-6146:
---------------------------------------

+1

> Upgrade to JetS3t version 0.7.1
> -------------------------------
>
>                 Key: HADOOP-6146
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6146
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6146.patch
>
>
> The JetS3t library is used for the S3 filesystems. We should upgrade to the latest version (0.7.1) which has support for Requester Pays buckets. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-312-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 18:36:29 2009
Return-Path: <common-issues-return-312-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60469 invoked from network); 15 Jul 2009 18:36:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 18:36:27 -0000
Received: (qmail 69273 invoked by uid 500); 15 Jul 2009 18:36:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69252 invoked by uid 500); 15 Jul 2009 18:36:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69242 invoked by uid 99); 15 Jul 2009 18:36:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 18:36:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 18:36:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 05D0A234C004
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 11:36:15 -0700 (PDT)
Message-ID: <1050711742.1247682975009.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 11:36:15 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6120) Add support for Avro types in
 hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731612#action_12731612 ] 

Doug Cutting commented on HADOOP-6120:
--------------------------------------

Looks great!
 - shouldn't Serializer/Desrializer#close() call encoder/decoder.close()?
 - should AvroReflectSerialization#accept() be synchronized?  I don't know if we generally require Serializers to be thread-safe, but it's probably best and should not have much performance impact.

> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch, 6120_v3.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-313-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 18:54:30 2009
Return-Path: <common-issues-return-313-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68068 invoked from network); 15 Jul 2009 18:54:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 18:54:29 -0000
Received: (qmail 92805 invoked by uid 500); 15 Jul 2009 18:54:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92781 invoked by uid 500); 15 Jul 2009 18:54:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92766 invoked by uid 99); 15 Jul 2009 18:54:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 18:54:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 18:54:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D63FE234C004
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 11:54:14 -0700 (PDT)
Message-ID: <1647543361.1247684054866.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 11:54:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6148:
-------------------------------------------

    Attachment: benchmarks20090715.txt

Changed the benchmark as below such that there are ~4GB data in each run.
{code}
    for(int j = 10; j < 24; j += 2) {
      for(int k = 0; k < 4; k++) {
        final int bytelen = (1 << j) + k;
        final byte[] b = new byte[bytelen];
        final int n = (int)((1L << 32) / bytelen);
    
        ran.nextBytes(b);
        t.tick("ran.nextBytes, bytelen=" + bytelen);
    
        final SortedMap<Long, Checksum> rank = new TreeMap<Long, Checksum>(); 
        test(pure, b, n, t, rank);
        test(test, b, n, t, rank);
        test(zip, b, n, t, rank);

        System.out.println("rank = " + rank);
        final Checksum c = rank.entrySet().iterator().next().getValue();
        fastest.put(c, fastest.get(c) + 1);
      }
    }
{code}

benchmarks20090715.txt: new results

It is consistent that TestCrc32 is faster than zip.CRC32, which is faster than PureJavaCrc32.  There are ~13% improvement by TestCrc32 over PureJavaCrc32.

> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: benchmarks20090714.txt, benchmarks20090715.txt, crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-314-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 19:04:29 2009
Return-Path: <common-issues-return-314-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73354 invoked from network); 15 Jul 2009 19:04:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 19:04:29 -0000
Received: (qmail 10666 invoked by uid 500); 15 Jul 2009 19:04:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10627 invoked by uid 500); 15 Jul 2009 19:04:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10617 invoked by uid 99); 15 Jul 2009 19:04:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 19:04:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 19:04:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DEB6B29A0013
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 12:04:14 -0700 (PDT)
Message-ID: <1312737691.1247684654911.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 12:04:14 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731631#action_12731631 ] 

Todd Lipcon commented on HADOOP-6148:
-------------------------------------

Thanks for the improvements, Nicholas. I should have a chance to verify your findings today or tomorrow.

At first glance, it seems odd that zip.CRC32 is faster than PureJavaCrc32. Are you using the Sun JDK or OpenJDK? In my tests for large block sizes, Sun's JDK is slower than PureJavaCrc32, but OpenJDK's is faster.

> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: benchmarks20090714.txt, benchmarks20090715.txt, crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-315-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 20:19:30 2009
Return-Path: <common-issues-return-315-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1649 invoked from network); 15 Jul 2009 20:19:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 20:19:29 -0000
Received: (qmail 44154 invoked by uid 500); 15 Jul 2009 20:19:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44116 invoked by uid 500); 15 Jul 2009 20:19:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44106 invoked by uid 99); 15 Jul 2009 20:19:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 20:19:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 20:19:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CF287234C045
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 13:19:14 -0700 (PDT)
Message-ID: <106664792.1247689154847.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 13:19:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731657#action_12731657 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6148:
------------------------------------------------

I am using Sun JDK on Windows.
{noformat}
bash-3.2$ java -version
java version "1.6.0_13"
Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
Java HotSpot(TM) Client VM (build 11.3-b02, mixed mode, sharing)
{noformat}

> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: benchmarks20090714.txt, benchmarks20090715.txt, crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-316-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 20:43:07 2009
Return-Path: <common-issues-return-316-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8366 invoked from network); 15 Jul 2009 20:43:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 20:43:07 -0000
Received: (qmail 65917 invoked by uid 500); 15 Jul 2009 20:43:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65509 invoked by uid 500); 15 Jul 2009 20:42:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64549 invoked by uid 99); 15 Jul 2009 20:31:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 20:31:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 20:31:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 31ED829A002F
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 13:31:15 -0700 (PDT)
Message-ID: <758436615.1247689875203.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 13:31:15 -0700 (PDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6152) Hadoop scripts do not correctly put
 jars on the classpath
In-Reply-To: <283943906.1247689875148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron Kimball updated HADOOP-6152:
----------------------------------

    Attachment: HADOOP-6152.patch

This patch fixes the various bin/* scripts.

> Hadoop scripts do not correctly put jars on the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-6152
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6152
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>            Priority: Blocker
>         Attachments: HADOOP-6152.patch
>
>
> The various Hadoop scripts (bin/hadoop, bin/hdfs, bin/mapred) do not properly identify the jars needed to run Hadoop. They try to include hadoop-*-hdfs.jar, etc, rather than the hadoop-hdfs-*.jar that is actually created by the 'ant jar' and 'ant package' targets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-318-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 20:43:28 2009
Return-Path: <common-issues-return-318-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9114 invoked from network); 15 Jul 2009 20:43:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 20:43:27 -0000
Received: (qmail 66865 invoked by uid 500); 15 Jul 2009 20:43:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66105 invoked by uid 500); 15 Jul 2009 20:43:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64548 invoked by uid 99); 15 Jul 2009 20:31:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 20:31:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 20:31:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 2483229A002B
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 13:31:15 -0700 (PDT)
Message-ID: <283943906.1247689875148.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 13:31:15 -0700 (PDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6152) Hadoop scripts do not correctly put
 jars on the classpath
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Hadoop scripts do not correctly put jars on the classpath
---------------------------------------------------------

                 Key: HADOOP-6152
                 URL: https://issues.apache.org/jira/browse/HADOOP-6152
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Aaron Kimball
            Assignee: Aaron Kimball
            Priority: Blocker
         Attachments: HADOOP-6152.patch

The various Hadoop scripts (bin/hadoop, bin/hdfs, bin/mapred) do not properly identify the jars needed to run Hadoop. They try to include hadoop-*-hdfs.jar, etc, rather than the hadoop-hdfs-*.jar that is actually created by the 'ant jar' and 'ant package' targets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-317-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 20:43:29 2009
Return-Path: <common-issues-return-317-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9105 invoked from network); 15 Jul 2009 20:43:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 20:43:27 -0000
Received: (qmail 66796 invoked by uid 500); 15 Jul 2009 20:43:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65915 invoked by uid 500); 15 Jul 2009 20:43:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64566 invoked by uid 99); 15 Jul 2009 20:32:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 20:32:01 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 20:31:57 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3D68C29A0031
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 13:31:15 -0700 (PDT)
Message-ID: <1128125379.1247689875250.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 13:31:15 -0700 (PDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6152) Hadoop scripts do not correctly put
 jars on the classpath
In-Reply-To: <283943906.1247689875148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron Kimball updated HADOOP-6152:
----------------------------------

    Status: Patch Available  (was: Open)

> Hadoop scripts do not correctly put jars on the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-6152
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6152
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>            Priority: Blocker
>         Attachments: HADOOP-6152.patch
>
>
> The various Hadoop scripts (bin/hadoop, bin/hdfs, bin/mapred) do not properly identify the jars needed to run Hadoop. They try to include hadoop-*-hdfs.jar, etc, rather than the hadoop-hdfs-*.jar that is actually created by the 'ant jar' and 'ant package' targets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-319-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 21:07:29 2009
Return-Path: <common-issues-return-319-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19525 invoked from network); 15 Jul 2009 21:07:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 21:07:29 -0000
Received: (qmail 17672 invoked by uid 500); 15 Jul 2009 21:07:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17630 invoked by uid 500); 15 Jul 2009 21:07:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17567 invoked by uid 99); 15 Jul 2009 21:07:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 21:07:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 21:07:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C4E64234C004
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 14:07:14 -0700 (PDT)
Message-ID: <2132640084.1247692034801.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 14:07:14 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6120) Add support for Avro types in
 hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731686#action_12731686 ] 

Doug Cutting commented on HADOOP-6120:
--------------------------------------

> Hudson failed because patch needs avro jar.

The avro jar should soon appear in the Maven repo.  Then we'll be able to just add it as a dependency in ivy.xml.

> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch, 6120_v3.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-320-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 21:09:28 2009
Return-Path: <common-issues-return-320-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20355 invoked from network); 15 Jul 2009 21:09:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 21:09:27 -0000
Received: (qmail 19302 invoked by uid 500); 15 Jul 2009 21:09:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19258 invoked by uid 500); 15 Jul 2009 21:09:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19244 invoked by uid 99); 15 Jul 2009 21:09:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 21:09:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 21:09:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CE59D234C004
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 14:09:14 -0700 (PDT)
Message-ID: <1466883101.1247692154840.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 14:09:14 -0700 (PDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6151) The servlets should quote html
 characters
In-Reply-To: <2103411870.1247675174858.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arun C Murthy updated HADOOP-6151:
----------------------------------

    Priority: Critical  (was: Major)

> The servlets should quote html characters
> -----------------------------------------
>
>                 Key: HADOOP-6151
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6151
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Priority: Critical
>
> We need to quote html characters that come from user generated data. Otherwise, all of the web ui's have cross site scripting attack, etc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-321-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 22:47:30 2009
Return-Path: <common-issues-return-321-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70436 invoked from network); 15 Jul 2009 22:47:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 22:47:30 -0000
Received: (qmail 77667 invoked by uid 500); 15 Jul 2009 22:47:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77630 invoked by uid 500); 15 Jul 2009 22:47:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77620 invoked by uid 99); 15 Jul 2009 22:47:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 22:47:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 22:47:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EECC7234C045
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 15:47:14 -0700 (PDT)
Message-ID: <1323016769.1247698034977.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 15:47:14 -0700 (PDT)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731721#action_12731721 ] 

Scott Carey commented on HADOOP-6148:
-------------------------------------

Whatever that Timer class is that is being used to measure, it is using the system millisecond time, which has only ~15ms accuracy so the test run time needs to be long to be accurate.  To get more accurate results, use System.nanoTime().
Also, I have found that a couple 'warmup' iterations of a test make the results much more consistent, to avoid the JIT interfering.

Using the benchmark I did before (previously attached, TestCrc32Performance.java), the new version is consistently 10% slower than the previous on my machine (Java 6, Max OS X, Core2 Duo processor, 64 bit JVM).  On Sun JRE 6.0_u14 on Linux (64 bit) with different CPUs, results vary.  I'll dig into those details below.

Results should be normalized to a metric we can compare to -- we have been using MB/sec so far.  Additionally, the JVM and environment used is critical.  Java 1.5 behaves VERY differently and I would expect changes like this to behave differently there (as well as with OpenJDK, IBM, or JRockit).

There are two changes here -- the loop format and termination, and how the shifts and masks are packed.

Adding or removing the final declarations makes no difference in my testing -- the compiler easily identifies whether variables change or not.

These two changes, taken alone, or taken together have varied results.  This is probably because loop unrolling in the JIT behaves differently.  Generally, the loop change helped the least, it only avoids a decrement on a loop condition, which is essentially free for some processors, and it has higher start-up cost and more variables.

Here are results on my laptop, followed by two different servers.

Note, that the first two results in any test are always lower than the rest, regardless of what order i do the tests, so consider the 'size 1' scores suspect.

Here are my results with the new version posted on my laptop Core2 Duo -- 2.5Ghz, 4GB 667Mhz DDR2 SDRAM, OSX 10.5.7 Java 6 (Apple):

$ java -Xmx512m -cp testall.jar org.apache.hadoop.util.TestCrc32Performance
||num bytes||NewLoopOnly MB/sec||NewInnerOnly MB/sec||NewPureJava MB/sec||PureJava MB/sec||Native MB/sec||
| 1	|85.195 	|94.160 	|121.054 	|117.785 	|6.580 	|
| 2	|123.629 	|133.855 	|131.112 	|164.309 	|12.354 	|
| 4	|226.677 	|220.528 	|194.145 	|287.520 	|24.309 	|
| 8	|240.169 	|262.491 	|253.665 	|283.975 	|45.353 	|
| 16	|343.329 	|364.299 	|354.749 	|383.966 	|77.207 	|
| 32	|441.347 	|445.800 	|433.381 	|462.508 	|122.829 	|
| 64	|522.195 	|522.210 	|502.894 	|528.559 	|188.184 	|
| 128	|570.476 	|551.149 	|541.540 	|555.572 	|194.542 	|
| 256	|596.147 	|577.042 	|558.884 	|591.713 	|289.183 	|
| 512	|601.956 	|583.593 	|561.882 	|593.323 	|315.714 	|
| 1024	|623.217 	|592.406 	|577.292 	|603.304 	|332.319 	|
| 2048	|623.979 	|594.163 	|581.419 	|606.365 	|341.448 	|
| 4096	|624.345 	|596.289 	|584.365 	|610.018 	|344.685 	|
| 8192	|626.711 	|593.323 	|585.424 	|607.542 	|347.891 	|
| 16384	|625.938 	|599.650 	|584.414 	|607.903 	|350.221 	|
| 32768	|623.995 	|583.516 	|577.771 	|609.930 	|349.754 	|
| 65536	|623.906 	|594.321 	|578.602 	|610.338 	|347.915 	|
| 131072	|624.308 	|595.950 	|577.024 	|610.308 	|350.647 	|
| 262144	|629.946 	|590.831 	|577.453 	|610.285 	|351.603 	|
| 524288	|623.757 	|597.578 	|575.428 	|610.399 	|349.063 	|
| 1048576	|624.043 	|596.817 	|577.521 	|610.798 	|352.213 	|
| 2097152	|627.303 	|591.817 	|573.852 	|610.981 	|352.406 	|
| 4194304	|623.168 	|593.048 	|570.512 	|605.679 	|347.898 	|
| 8388608	|609.118 	|587.879 	|562.892 	|600.384 	|344.380 	|
| 16777216	|610.116 	|585.480 	|555.988 	|601.001 	|348.796 	|

For the above, the new loop helps, the new inner code hurts, and combined it is worst.  For small checksum sizes all are worse.

For a Linux server (Dual quad core Xeon E5335  @ 2.00GHz), Sun JDK 1.6.0_u14 the results are different.
The Loop change does not help, changing the inner code alone helps the most, and combining the two is somewhere in-between, and for very small sizes its always slower:

java -Xmx512m -XX:+UseParallelOldGC -XX:+UseCompressedOops -cp testall.jar org.apache.hadoop.util.TestCrc32Performance
||num bytes||NewLoopOnly MB/sec||NewInnerOnly MB/sec||NewPureJava MB/sec||PureJava MB/sec||Native MB/sec||
| 1	|66.281 	|66.264 	|85.825 	|93.888 	|7.331 	|
| 2	|93.812 	|94.895 	|92.638 	|129.917 	|13.931 	|
| 4	|155.431 	|144.819 	|161.540 	|178.275 	|26.586 	|
| 8	|174.531 	|185.222 	|206.577 	|185.253 	|47.892 	|
| 16	|244.490 	|275.026 	|292.237 	|256.664 	|81.922 	|
| 32	|306.768 	|351.271 	|345.710 	|313.523 	|127.560 	|
| 64	|350.763 	|407.539 	|382.370 	|352.997 	|175.000 	|
| 128	|377.402 	|442.239 	|406.650 	|376.721 	|218.138 	|
| 256	|392.901 	|461.884 	|420.526 	|390.215 	|247.586 	|
| 512	|397.682 	|467.375 	|425.522 	|393.319 	|264.732 	|
| 1024	|404.145 	|474.192 	|430.791 	|391.678 	|272.883 	|
| 2048	|407.673 	|478.888 	|433.719 	|400.236 	|277.638 	|
| 4096	|409.028 	|480.907 	|435.645 	|401.338 	|280.214 	|
| 8192	|409.890 	|482.769 	|436.209 	|402.037 	|281.782 	|
| 16384	|409.565 	|482.726 	|436.340 	|401.767 	|282.409 	|
| 32768	|407.943 	|481.176 	|436.373 	|399.955 	|282.455 	|
| 65536	|407.933 	|481.746 	|435.970 	|400.228 	|282.761 	|
| 131072	|408.003 	|481.723 	|436.516 	|399.973 	|282.749 	|
| 262144	|407.412 	|481.357 	|436.067 	|400.962 	|283.193 	|
| 524288	|408.077 	|481.335 	|436.416 	|401.280 	|283.137 	|
| 1048576	|408.016 	|481.625 	|436.086 	|402.039 	|283.308 	|
| 2097152	|407.397 	|481.386 	|436.131 	|401.394 	|283.353 	|
| 4194304	|406.609 	|479.130 	|434.960 	|400.632 	|282.376 	|
| 8388608	|403.235 	|475.130 	|430.770 	|397.797 	|280.904 	|
| 16777216	|402.891 	|474.464 	|430.427 	|397.324 	|280.951 	|


Lastly, I have access to a dual quad core Nehalem system Xeon X5550  @ 2.67GHz.   The trend is similar to the other Linux server.
||num bytes||NewLoopOnly MB/sec||NewInnerOnly MB/sec||NewPureJava MB/sec||PureJava MB/sec||Native MB/sec||
| 1	|100.809 	|105.911 	|124.574 	|168.577 	|17.863 	|
| 2	|144.671 	|157.576 	|150.993 	|203.188 	|33.722 	|
| 4	|230.455 	|226.380 	|250.517 	|276.617 	|64.370 	|
| 8	|263.961 	|288.727 	|308.807 	|292.203 	|103.247 	|
| 16	|345.860 	|419.419 	|454.893 	|394.811 	|158.828 	|
| 32	|437.694 	|536.016 	|534.576 	|470.360 	|226.527 	|
| 64	|503.768 	|611.457 	|579.010 	|470.782 	|281.814 	|
| 128	|537.179 	|650.089 	|614.528 	|522.099 	|317.293 	|
| 256	|557.598 	|672.534 	|630.303 	|558.545 	|344.511 	|
| 512	|561.642 	|677.876 	|611.557 	|463.531 	|345.938 	|
| 1024	|565.044 	|690.742 	|615.121 	|558.021 	|367.634 	|
| 2048	|583.641 	|714.832 	|634.747 	|602.027 	|372.008 	|
| 4096	|594.691 	|717.332 	|643.951 	|598.466 	|369.619 	|
| 8192	|562.813 	|639.987 	|596.214 	|560.489 	|366.227 	|
| 16384	|540.490 	|682.933 	|481.701 	|571.237 	|373.531 	|
| 32768	|520.446 	|636.502 	|563.346 	|539.832 	|333.878 	|
| 65536	|498.553 	|617.019 	|556.558 	|528.404 	|343.788 	|
| 131072	|526.779 	|625.905 	|585.023 	|520.933 	|333.155 	|
| 262144	|535.428 	|617.910 	|568.184 	|530.084 	|337.989 	|
| 524288	|563.442 	|637.368 	|623.922 	|578.144 	|379.028 	|
| 1048576	|578.139 	|709.259 	|632.558 	|596.539 	|374.711 	|
| 2097152	|537.883 	|662.951 	|634.672 	|583.292 	|363.072 	|
| 4194304	|551.819 	|677.231 	|632.805 	|580.628 	|372.185 	|
| 8388608	|581.055 	|689.236 	|624.939 	|584.958 	|358.719 	|
| 16777216	|563.078 	|675.309 	|566.319 	|579.324 	|365.306 	|

So it looks like the winner is to change the inner part of the loop only.  Although this hurts a bit on Apple's VM, that isn't a production VM.  Unless there is data for some other VM's that suggests this is a poor choice it looks best to me.  It is slightly slower for sizes less than about 8 bytes however  The code is:

{code}  public void update(byte[] b, int off, int len) {
    while(len > 3) {
      int c0 = crc ^ b[off++];
      int c1 = (crc >>>= 8) ^ b[off++];
      int c2 = (crc >>>= 8) ^ b[off++];
      int c3 = (crc >>>= 8) ^ b[off++];
      crc = T4[c0 & 0xff] ^ T3[c1 & 0xff] ^ T2[c2 & 0xff] ^ T1[c3 & 0xff];
      len -=4;
    }
    while(len > 0) {
      crc = (crc >>> 8) ^ T1[(crc ^ b[off++]) & 0xff];
      len --;
    }
  }{code}


> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: benchmarks20090714.txt, benchmarks20090715.txt, crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-322-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 22:51:27 2009
Return-Path: <common-issues-return-322-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72073 invoked from network); 15 Jul 2009 22:51:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 22:51:27 -0000
Received: (qmail 81617 invoked by uid 500); 15 Jul 2009 22:51:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81577 invoked by uid 500); 15 Jul 2009 22:51:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81567 invoked by uid 99); 15 Jul 2009 22:51:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 22:51:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 22:51:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CEFED234C004
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 15:51:14 -0700 (PDT)
Message-ID: <220200760.1247698274832.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 15:51:14 -0700 (PDT)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Scott Carey updated HADOOP-6148:
--------------------------------

    Attachment: PureJavaCrc32NewLoop.java
                PureJavaCrc32NewInner.java
                PureJavaCrc32New.java

variations of the CRC for testing.

> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: benchmarks20090714.txt, benchmarks20090715.txt, crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32New.java, PureJavaCrc32NewInner.java, PureJavaCrc32NewLoop.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-323-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 22:53:29 2009
Return-Path: <common-issues-return-323-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72637 invoked from network); 15 Jul 2009 22:53:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 22:53:29 -0000
Received: (qmail 86529 invoked by uid 500); 15 Jul 2009 22:53:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86493 invoked by uid 500); 15 Jul 2009 22:53:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86483 invoked by uid 99); 15 Jul 2009 22:53:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 22:53:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 22:53:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C8F74234C004
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 15:53:14 -0700 (PDT)
Message-ID: <45040754.1247698394809.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 15:53:14 -0700 (PDT)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Scott Carey updated HADOOP-6148:
--------------------------------

    Attachment: TestCrc32Performance.java

New test, testing performance of many crc variants and spending more time warming up each of them to make sure the JIT has compiled them to native.  System.gc() calls were experimented with to make results more consistent from run to run.   Typically, I have been running this with -Xmx512m

> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: benchmarks20090714.txt, benchmarks20090715.txt, crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32New.java, PureJavaCrc32NewInner.java, PureJavaCrc32NewLoop.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-324-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 22:57:27 2009
Return-Path: <common-issues-return-324-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73562 invoked from network); 15 Jul 2009 22:57:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 22:57:27 -0000
Received: (qmail 89358 invoked by uid 500); 15 Jul 2009 22:57:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89322 invoked by uid 500); 15 Jul 2009 22:57:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89274 invoked by uid 99); 15 Jul 2009 22:57:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 22:57:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 22:57:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E2DEB29A0015
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 15:57:14 -0700 (PDT)
Message-ID: <1618262767.1247698634928.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 15:57:14 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Issue Comment Edited: (HADOOP-6151) The servlets should
 quote html characters
In-Reply-To: <2103411870.1247675174858.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731727#action_12731727 ] 

Owen O'Malley edited comment on HADOOP-6151 at 7/15/09 3:56 PM:
----------------------------------------------------------------

I believe the transforms should be:
  1. & -> &amp;amp;
  2. < -> &amp;lt;
  3. > -> &amp;gt;
  4. ' -> &amp;apos;
  5. "-> &amp;quot;

As long as we do those transforms, any html that the user includes in their data will just be treated as literal text rather than html commands.

      was (Author: owen.omalley):
    I believe the transforms should be:
  1. & -> &amp;
  2. < -> &lt;
  3. > -> &gt;
  4. ' -> &apos;
  5. "-> &quot;

As long as we do those transforms, any html that the user includes in their data will just be treated as literal text rather than html commands.
  
> The servlets should quote html characters
> -----------------------------------------
>
>                 Key: HADOOP-6151
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6151
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Priority: Critical
>
> We need to quote html characters that come from user generated data. Otherwise, all of the web ui's have cross site scripting attack, etc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-325-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 22:57:30 2009
Return-Path: <common-issues-return-325-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73600 invoked from network); 15 Jul 2009 22:57:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 22:57:29 -0000
Received: (qmail 89429 invoked by uid 500); 15 Jul 2009 22:57:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89392 invoked by uid 500); 15 Jul 2009 22:57:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89382 invoked by uid 99); 15 Jul 2009 22:57:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 22:57:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 22:57:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CAA4B234C004
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 15:57:14 -0700 (PDT)
Message-ID: <972627625.1247698634825.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 15:57:14 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6151) The servlets should quote html
 characters
In-Reply-To: <2103411870.1247675174858.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731727#action_12731727 ] 

Owen O'Malley commented on HADOOP-6151:
---------------------------------------

I believe the transforms should be:
  1. & -> &amp;
  2. < -> &lt;
  3. > -> &gt;
  4. ' -> &apos;
  5. "-> &quot;

As long as we do those transforms, any html that the user includes in their data will just be treated as literal text rather than html commands.

> The servlets should quote html characters
> -----------------------------------------
>
>                 Key: HADOOP-6151
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6151
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Priority: Critical
>
> We need to quote html characters that come from user generated data. Otherwise, all of the web ui's have cross site scripting attack, etc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-326-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 23:05:29 2009
Return-Path: <common-issues-return-326-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75130 invoked from network); 15 Jul 2009 23:05:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 23:05:29 -0000
Received: (qmail 95919 invoked by uid 500); 15 Jul 2009 23:05:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95882 invoked by uid 500); 15 Jul 2009 23:05:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95869 invoked by uid 99); 15 Jul 2009 23:05:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 23:05:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 23:05:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E212829A0011
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 16:05:14 -0700 (PDT)
Message-ID: <941989418.1247699114925.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 16:05:14 -0700 (PDT)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731732#action_12731732 ] 

Scott Carey commented on HADOOP-6148:
-------------------------------------

{quote}
I am using Sun JDK on Windows.
{noformat}
bash-3.2$ java -version
java version "1.6.0_13"
Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
Java HotSpot(TM) Client VM (build 11.3-b02, mixed mode, sharing)
{noformat}{quote}

You are  using the Client VM, not the Server VM.  Try some tests with the -server flag.
You are also using a 32 bit VM, which may have a small effect here.  But the Server versus Client VM makes a big difference here.


> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: benchmarks20090714.txt, benchmarks20090715.txt, crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32New.java, PureJavaCrc32NewInner.java, PureJavaCrc32NewLoop.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-327-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 23:07:28 2009
Return-Path: <common-issues-return-327-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75631 invoked from network); 15 Jul 2009 23:07:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 23:07:28 -0000
Received: (qmail 1066 invoked by uid 500); 15 Jul 2009 23:07:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 999 invoked by uid 500); 15 Jul 2009 23:07:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 975 invoked by uid 99); 15 Jul 2009 23:07:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 23:07:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 23:07:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DC52F234C04B
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 16:07:14 -0700 (PDT)
Message-ID: <749050436.1247699234901.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 16:07:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6152) Hadoop scripts do not correctly put
 jars on the classpath
In-Reply-To: <283943906.1247689875148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731733#action_12731733 ] 

Hadoop QA commented on HADOOP-6152:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12413596/HADOOP-6152.patch
  against trunk revision 793987.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/574/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/574/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/574/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/574/console

This message is automatically generated.

> Hadoop scripts do not correctly put jars on the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-6152
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6152
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>            Priority: Blocker
>         Attachments: HADOOP-6152.patch
>
>
> The various Hadoop scripts (bin/hadoop, bin/hdfs, bin/mapred) do not properly identify the jars needed to run Hadoop. They try to include hadoop-*-hdfs.jar, etc, rather than the hadoop-hdfs-*.jar that is actually created by the 'ant jar' and 'ant package' targets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-328-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 23:13:29 2009
Return-Path: <common-issues-return-328-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76813 invoked from network); 15 Jul 2009 23:13:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 23:13:29 -0000
Received: (qmail 6498 invoked by uid 500); 15 Jul 2009 23:13:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6473 invoked by uid 500); 15 Jul 2009 23:13:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6463 invoked by uid 99); 15 Jul 2009 23:13:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 23:13:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 23:13:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C6B87234C004
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 16:13:14 -0700 (PDT)
Message-ID: <207955473.1247699594800.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 16:13:14 -0700 (PDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6152) Hadoop scripts do not correctly put
 jars on the classpath
In-Reply-To: <283943906.1247689875148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731736#action_12731736 ] 

Aaron Kimball commented on HADOOP-6152:
---------------------------------------

No tests added because we don't have a test framework that works around the end-user package target. 

> Hadoop scripts do not correctly put jars on the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-6152
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6152
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>            Priority: Blocker
>         Attachments: HADOOP-6152.patch
>
>
> The various Hadoop scripts (bin/hadoop, bin/hdfs, bin/mapred) do not properly identify the jars needed to run Hadoop. They try to include hadoop-*-hdfs.jar, etc, rather than the hadoop-hdfs-*.jar that is actually created by the 'ant jar' and 'ant package' targets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-329-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 23:21:30 2009
Return-Path: <common-issues-return-329-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77818 invoked from network); 15 Jul 2009 23:21:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 23:21:30 -0000
Received: (qmail 14022 invoked by uid 500); 15 Jul 2009 23:21:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13983 invoked by uid 500); 15 Jul 2009 23:21:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13967 invoked by uid 99); 15 Jul 2009 23:21:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 23:21:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 23:21:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E51BF234C004
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 16:21:14 -0700 (PDT)
Message-ID: <757199186.1247700074910.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 16:21:14 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731739#action_12731739 ] 

Todd Lipcon commented on HADOOP-6148:
-------------------------------------

Here are the results from my laptop. PureJavaTW is Tsz Wo's updated version.

First, 64-bit JVM:

||num bytes||NewLoopOnly MB/sec||NewInnerOnly MB/sec||NewPureJava MB/sec||PureJava MB/sec||||PureJavaTW MB/sec||Native MB/sec||
| 1     |79.721         |87.113         |111.889        |112.642        |88.684         |9.001  |
| 2     |91.820         |130.441        |130.855        |133.645        |131.948        |17.536         |
| 4     |167.315        |223.087        |198.984        |243.936        |198.387        |33.488         |
| 8     |188.153        |254.671        |243.899        |248.944        |242.746        |59.258         |
| 16    |311.118        |353.191        |350.826        |327.231        |333.589        |99.213         |
| 32    |401.696        |416.339        |417.406        |427.676        |408.610        |148.742        |
| 64    |499.747        |483.148        |445.437        |472.814        |487.970        |208.478        |
| 128   |530.055        |520.043        |505.709        |513.645        |457.080        |253.756        |
| 256   |489.407        |541.459        |523.867        |547.794        |510.867        |283.360        |
| 512   |561.871        |528.383        |528.368        |553.071        |530.421        |300.134        |
| 1024  |579.227        |549.401        |537.941        |551.488        |536.391        |293.307        |
| 2048  |586.443        |551.685        |540.289        |564.328        |539.412        |319.766        |
| 4096  |608.470        |573.333        |560.746        |586.014        |560.349        |332.661        |
| 8192  |590.123        |554.975        |543.089        |424.834        |517.325        |322.192        |
| 16384 |583.385        |539.704        |542.656        |567.484        |542.567        |324.026        |
| 32768 |583.592        |551.508        |533.585        |561.559        |529.811        |321.858        |
| 65536 |584.476        |553.217        |537.679        |544.507        |512.978        |310.739        |
| 131072        |548.941        |529.097        |534.430        |564.858        |533.955        |324.626        |
| 262144        |584.379        |551.733        |536.386        |564.063        |479.038        |324.117        |
| 524288        |583.262        |553.893        |536.770        |563.518        |532.404        |324.924        |
| 1048576       |581.947        |550.572        |533.850        |561.049        |512.846        |294.452        |
| 2097152       |566.543        |534.695        |484.256        |551.693        |527.730        |320.850        |
| 4194304       |569.545        |537.748        |520.731        |547.608        |522.762        |318.084        |
| 8388608       |593.932        |563.233        |530.600        |571.098        |545.905        |310.115        |
| 16777216      |573.095        |560.361        |545.069        |576.036        |529.475        |331.984        |

And 32-bit JVM on the same machine:

||num bytes||NewLoopOnly MB/sec||NewInnerOnly MB/sec||NewPureJava MB/sec||PureJava MB/sec||||PureJavaTW MB/sec||Native MB/sec||
| 1     |56.832         |51.138         |60.116         |60.826         |47.624         |7.612  |
| 2     |84.111         |82.710         |83.672         |81.422         |82.630         |15.075         |
| 4     |167.204        |138.900        |147.189        |163.798        |147.762        |30.063         |
| 8     |175.720        |177.396        |184.865        |180.241        |186.981        |54.772         |
| 16    |227.769        |257.112        |245.795        |247.928        |245.016        |94.794         |
| 32    |279.098        |336.251        |267.332        |298.998        |286.251        |147.808        |
| 64    |304.806        |381.515        |321.722        |339.217        |318.546        |207.577        |
| 128   |316.745        |433.474        |339.159        |356.571        |337.731        |255.225        |
| 256   |331.208        |454.923        |315.551        |374.849        |342.356        |291.948        |
| 512   |333.462        |451.160        |351.006        |374.545        |344.597        |312.074        |
| 1024  |332.338        |462.433        |350.361        |375.159        |359.086        |324.510        |
| 2048  |329.805        |472.755        |361.305        |379.140        |338.142        |317.101        |
| 4096  |326.613        |466.729        |349.653        |345.593        |337.487        |317.728        |
| 8192  |313.368        |458.838        |357.077        |384.962        |348.174        |332.353        |
| 16384 |338.060        |448.738        |341.744        |371.819        |337.915        |295.606        |
| 32768 |301.724        |451.107        |346.410        |381.720        |322.610        |332.183        |
| 65536 |337.599        |472.797        |328.112        |383.809        |324.456        |336.955        |
| 131072        |338.017        |471.498        |351.307        |383.234        |350.165        |338.728        |
| 262144        |338.063        |472.881        |351.411        |383.875        |350.652        |338.079        |
| 524288        |338.175        |471.574        |349.477        |381.680        |348.984        |334.452        |
| 1048576       |333.453        |460.829        |343.706        |381.459        |346.565        |334.384        |
| 2097152       |335.291        |465.923        |347.260        |374.896        |330.102        |330.254        |
| 4194304       |332.436        |460.711        |340.488        |378.804        |346.389        |334.329        |
| 8388608       |334.700        |464.714        |347.837        |378.336        |346.230        |324.550        |
| 16777216      |316.521        |431.736        |342.807        |373.638        |341.928        |328.748        |

It seems to me that, on the 64-bit JVM, most of the implementations are within margin of error at the sizes that are most often exercised (128 to 256 bytes). On 32-bit, the NewInnerOnly wins by a reasonable amount.

I say we commit the NewInnerOnly version.

> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: benchmarks20090714.txt, benchmarks20090715.txt, crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32New.java, PureJavaCrc32NewInner.java, PureJavaCrc32NewLoop.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-330-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 23:39:30 2009
Return-Path: <common-issues-return-330-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80635 invoked from network); 15 Jul 2009 23:39:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 23:39:29 -0000
Received: (qmail 25977 invoked by uid 500); 15 Jul 2009 23:39:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25954 invoked by uid 500); 15 Jul 2009 23:39:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25944 invoked by uid 99); 15 Jul 2009 23:39:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 23:39:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 23:39:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CE98E234C004
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 16:39:14 -0700 (PDT)
Message-ID: <120105708.1247701154831.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 16:39:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731743#action_12731743 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6148:
------------------------------------------------

Thanks, Scott and Todd for checking the performance in such great details!

> I say we commit the NewInnerOnly version.
+1

BTW, the path for TestPureJavaCrc32 is wrong in the previous patch.  We don't have "core" directory anymore.
{code}
--- /dev/null
+++ src/test/core/org/apache/hadoop/util/TestPureJavaCrc32.java
@@ -0,0 +1,94 @@
{code}

> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: benchmarks20090714.txt, benchmarks20090715.txt, crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32New.java, PureJavaCrc32NewInner.java, PureJavaCrc32NewLoop.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-331-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 23:45:28 2009
Return-Path: <common-issues-return-331-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82167 invoked from network); 15 Jul 2009 23:45:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 23:45:27 -0000
Received: (qmail 28696 invoked by uid 500); 15 Jul 2009 23:45:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28658 invoked by uid 500); 15 Jul 2009 23:45:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28646 invoked by uid 99); 15 Jul 2009 23:45:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 23:45:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 23:45:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CFFDE234C004
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 16:45:14 -0700 (PDT)
Message-ID: <2013429545.1247701514838.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 16:45:14 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731746#action_12731746 ] 

Todd Lipcon commented on HADOOP-6148:
-------------------------------------

Good catch on the patch. I'll prepare a new one shortly that fixes that and includes the NewInnerOnly implementation, unless you're already on it.

> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: benchmarks20090714.txt, benchmarks20090715.txt, crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32New.java, PureJavaCrc32NewInner.java, PureJavaCrc32NewLoop.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-332-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 15 23:45:28 2009
Return-Path: <common-issues-return-332-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82194 invoked from network); 15 Jul 2009 23:45:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Jul 2009 23:45:27 -0000
Received: (qmail 28761 invoked by uid 500); 15 Jul 2009 23:45:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28720 invoked by uid 500); 15 Jul 2009 23:45:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28708 invoked by uid 99); 15 Jul 2009 23:45:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 23:45:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jul 2009 23:45:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 483F629A001B
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 16:45:15 -0700 (PDT)
Message-ID: <156566319.1247701515295.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 16:45:15 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731747#action_12731747 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6148:
------------------------------------------------

Since the benchmark program is very useful, we may combine TestCrc32Performance with TestPureJavaCrc32.  Then, the benchmark can be executed by something like "java TestPureJavaCrc32" in the future.

Todd, could you post a new patch?

> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: benchmarks20090714.txt, benchmarks20090715.txt, crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32New.java, PureJavaCrc32NewInner.java, PureJavaCrc32NewLoop.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-333-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 00:11:29 2009
Return-Path: <common-issues-return-333-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88491 invoked from network); 16 Jul 2009 00:11:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 00:11:29 -0000
Received: (qmail 52911 invoked by uid 500); 16 Jul 2009 00:11:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52872 invoked by uid 500); 16 Jul 2009 00:11:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52862 invoked by uid 99); 16 Jul 2009 00:11:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 00:11:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 00:11:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C9AB2234C004
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 17:11:14 -0700 (PDT)
Message-ID: <1960073109.1247703074811.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 17:11:14 -0700 (PDT)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731750#action_12731750 ] 

Scott Carey commented on HADOOP-6148:
-------------------------------------

Interesting results.
Todd, were you using -server on the 32 bit version?  If its Windows, client is the default, if its Linux, server is, and on a Mac 32 bit = Java 1.5.

In this benchmark, the Mac Java 1.5 32 bit JVM has equal performance between the previous version and the InnerOnly version -- but slower than native except for sizes 128 and under -- so its a somewhat useless test for the expected usage of this code.

I wasn't able to run against the newest 32 bit Sun JVM on a Linux server or to try out -client.  There is no -client for the 64 bit Sun JVMs.



> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: benchmarks20090714.txt, benchmarks20090715.txt, crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32New.java, PureJavaCrc32NewInner.java, PureJavaCrc32NewLoop.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-334-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 00:17:29 2009
Return-Path: <common-issues-return-334-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89244 invoked from network); 16 Jul 2009 00:17:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 00:17:29 -0000
Received: (qmail 57483 invoked by uid 500); 16 Jul 2009 00:17:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57441 invoked by uid 500); 16 Jul 2009 00:17:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57425 invoked by uid 99); 16 Jul 2009 00:17:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 00:17:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 00:17:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CC0E6234C004
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 17:17:14 -0700 (PDT)
Message-ID: <792362145.1247703434821.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 17:17:14 -0700 (PDT)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731752#action_12731752 ] 

Scott Carey commented on HADOOP-6148:
-------------------------------------

{quote}It seems to me that, on the 64-bit JVM, most of the implementations are within margin of error at the sizes that are most often exercised (128 to 256 bytes).{quote}

What are the most common use cases, and where else should this code be used other than HDFS?  For HDFS, the default checksum block size is 512 bytes.  For the bzip2 code, it is using its own CRC32  -- perhaps that should change.  For any .zip file compression or decompression, I'm not sure what the typical use case is.  

> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: benchmarks20090714.txt, benchmarks20090715.txt, crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32New.java, PureJavaCrc32NewInner.java, PureJavaCrc32NewLoop.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-335-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 03:19:33 2009
Return-Path: <common-issues-return-335-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8630 invoked from network); 16 Jul 2009 03:19:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 03:19:33 -0000
Received: (qmail 47689 invoked by uid 500); 16 Jul 2009 02:20:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47649 invoked by uid 500); 16 Jul 2009 02:20:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47632 invoked by uid 99); 16 Jul 2009 02:20:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 02:20:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 02:20:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C75E3234C004
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 19:20:14 -0700 (PDT)
Message-ID: <587382899.1247710814808.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 19:20:14 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731772#action_12731772 ] 

Owen O'Malley commented on HADOOP-6148:
---------------------------------------

Since map/reduce uses CRC to check the spills, the size can be arbitrary length.

> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: benchmarks20090714.txt, benchmarks20090715.txt, crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32New.java, PureJavaCrc32NewInner.java, PureJavaCrc32NewLoop.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-336-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 03:22:37 2009
Return-Path: <common-issues-return-336-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9272 invoked from network); 16 Jul 2009 03:22:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 03:22:34 -0000
Received: (qmail 51938 invoked by uid 500); 16 Jul 2009 02:23:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51900 invoked by uid 500); 16 Jul 2009 02:23:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51659 invoked by uid 99); 16 Jul 2009 02:23:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 02:23:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 02:23:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CF76F234C045
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 19:23:14 -0700 (PDT)
Message-ID: <926568478.1247710994848.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 19:23:14 -0700 (PDT)
From: "Daehyun Kim (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6153) RAgzip: multiple map tasks for a
 large gzipped file
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

RAgzip: multiple map tasks for a large gzipped file
---------------------------------------------------

                 Key: HADOOP-6153
                 URL: https://issues.apache.org/jira/browse/HADOOP-6153
             Project: Hadoop Common
          Issue Type: Bug
          Components: io, native
         Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
            Reporter: Daehyun Kim
            Assignee: Daehyun Kim
            Priority: Minor


It support to enable multiple map tasks for one large gzipped file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-337-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 03:26:34 2009
Return-Path: <common-issues-return-337-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10737 invoked from network); 16 Jul 2009 03:26:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 03:26:33 -0000
Received: (qmail 52914 invoked by uid 500); 16 Jul 2009 02:27:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52878 invoked by uid 500); 16 Jul 2009 02:27:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52864 invoked by uid 99); 16 Jul 2009 02:27:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 02:27:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 02:27:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C1F47234C004
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 19:27:14 -0700 (PDT)
Message-ID: <761227278.1247711234780.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 19:27:14 -0700 (PDT)
From: "Daehyun Kim (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6153) RAgzip: multiple map tasks for a
 large gzipped file
In-Reply-To: <926568478.1247710994848.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daehyun Kim updated HADOOP-6153:
--------------------------------

    Issue Type: Improvement  (was: Bug)

> RAgzip: multiple map tasks for a large gzipped file
> ---------------------------------------------------
>
>                 Key: HADOOP-6153
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6153
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, native
>         Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
>            Reporter: Daehyun Kim
>            Assignee: Daehyun Kim
>            Priority: Minor
>
> It support to enable multiple map tasks for one large gzipped file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-338-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 04:10:34 2009
Return-Path: <common-issues-return-338-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23219 invoked from network); 16 Jul 2009 04:10:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 04:10:34 -0000
Received: (qmail 23063 invoked by uid 500); 16 Jul 2009 04:11:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22958 invoked by uid 500); 16 Jul 2009 04:11:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22926 invoked by uid 99); 16 Jul 2009 04:11:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 04:11:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 04:11:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E3E26234C045
	for <common-issues@hadoop.apache.org>; Wed, 15 Jul 2009 21:11:14 -0700 (PDT)
Message-ID: <109977329.1247717474932.JavaMail.jira@brutus>
Date: Wed, 15 Jul 2009 21:11:14 -0700 (PDT)
From: "Jerome Boulon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6154) MultiSortedTFileScanner
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

MultiSortedTFileScanner
-----------------------

                 Key: HADOOP-6154
                 URL: https://issues.apache.org/jira/browse/HADOOP-6154
             Project: Hadoop Common
          Issue Type: New Feature
          Components: io
            Reporter: Jerome Boulon


Given a set of sorted Tfiles or a directory with several sorted Tfiles, it would be nice to have an helper class that help reading those Tfiles in a total sort order.
The MultiSortedTFileScanner class could take a directory as input or Tfile could be dynamically added/removed to/from the MultiSortedTFileScanner.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-339-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 07:02:34 2009
Return-Path: <common-issues-return-339-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86550 invoked from network); 16 Jul 2009 07:02:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 07:02:34 -0000
Received: (qmail 46814 invoked by uid 500); 16 Jul 2009 07:03:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46777 invoked by uid 500); 16 Jul 2009 07:03:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46767 invoked by uid 99); 16 Jul 2009 07:03:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 07:03:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 07:03:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DDAD6234C045
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 00:03:14 -0700 (PDT)
Message-ID: <277363383.1247727794893.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 00:03:14 -0700 (PDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4491) Per-job local data on the TaskTracker
 node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V updated HADOOP-4491:
------------------------------

    Attachment: HADOOP-4491-20090716-mapred.txt

Uploading mapred part of the patch that incorporates most of the review comments.
{quote}
 - In the case of jvm reuse, we still need to make the output available. Because it seems the task completion event is sent as soon as the task is done not after jvm exits. This is only a theoretical case possibly, but it still will be good to keep the code paths identical.
 - finalizeTaskDirs (or parts of it) needs to be synchronized.
 - Path permissions should be taken care of ? so, if mapred.local.dir doesn't have execute permissions for others, we need to set them.
 - In setup, we seem to be setting permissions even if directory creation fails. also shouldn't we set permissions even if it exists.. so that it is right as per our requirement.
 - Cache directory probably doesn't need 777 because it is not written to by the tasks. We can probably retain this if HADOOP-4490 set like permissions, since this will be addressed in HADOOP-4493.
 - getBaseIntermediateOutputDir seems an overkill if it is just returning a constant.
 - Changes in SpillRecord seem to be unnecessary.
 - TaskController.FILE_PERMISSIONS doesn't seem to be used anywhere.
 - Rename finalizeTaskDirs as finalizeTask - it matches with the naming convention for the other apis in taskcontroller.
 - Add a comment about why task-work is 755.
 - mapred.child.local.dir has an extra comma at the end.
{quote}
     -- Done

    * Permissions for job directory is changed multiple times - once per each task.
     -- Done. Added a new INITIALIZE_JOB command.

    * Code related to creation of work dirs is removed from TaskRunner. Where is it created now ?
      -- This is actually not needed. The work dirs are already created before at TaskTracker.localizeTask()  (creation of cwd)

    * Didn't understand the purpose of initStatus. Since it starts out being true, wouldn't it always remain true ?
     -- initStatus is used to track if directory creation is failing on all the disks. It was incorrectly initialized to true. Fixed this.

Things to be done:
{quote}
    - The code models some of the commands passed to the taskcontroller as 'first task' for JVM or 'not first task' for JVM. This complicates the contract between the TT and the task-controller. I think at a minimum these decisions should be restricted only to the JVM manager, and the contract between the TT and task-controller should be handled by modelling new commands like FINALIZE_JVM and FINALIZE_TASK.
    - The patch introduces a log directory argument to the task-controller. I think this is not required as there is only one value for the hadoop.log.dir for all tasks.
    - TODOs in the patch must be discussed and resolved.      
    - Rename finalizeTaskDirs as finalizeTask. To be done in the task-controller code

1. Setting secure permissions by default vs doing so only in the LinuxTaskController
2. Approach for sharing common files between TT and child
3. Sandboxing task runtime environment by changing values of mapred.local.dir for the child
{quote}


> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-340-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 07:38:34 2009
Return-Path: <common-issues-return-340-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96668 invoked from network); 16 Jul 2009 07:38:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 07:38:34 -0000
Received: (qmail 90627 invoked by uid 500); 16 Jul 2009 07:39:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90590 invoked by uid 500); 16 Jul 2009 07:39:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90580 invoked by uid 99); 16 Jul 2009 07:39:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 07:39:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 07:39:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3061E234C045
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 00:39:15 -0700 (PDT)
Message-ID: <1275290190.1247729955193.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 00:39:15 -0700 (PDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6149) FileStatus can support a fileid per
 path
In-Reply-To: <1932342868.1247609114947.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-6149:
-------------------------------------

    Attachment: fileidc1.txt

This patch introduces a new filed called fileid in the FileStatus object. There are setter and getter methods on FileStatus for fileid.

> FileStatus can support a fileid per path
> ----------------------------------------
>
>                 Key: HADOOP-6149
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6149
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>             Fix For: 0.21.0
>
>         Attachments: fileidc1.txt
>
>
> FileStatus should expose a id that uniquely identifies a file. This helps in developing applications that work correctly even when files are moved from one directory to another.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-341-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 10:13:32 2009
Return-Path: <common-issues-return-341-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46684 invoked from network); 16 Jul 2009 10:13:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 10:13:32 -0000
Received: (qmail 99751 invoked by uid 500); 16 Jul 2009 10:14:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99715 invoked by uid 500); 16 Jul 2009 10:14:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99705 invoked by uid 99); 16 Jul 2009 10:14:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 10:14:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 10:14:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 51E36234C004
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 03:14:15 -0700 (PDT)
Message-ID: <184258659.1247739255321.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 03:14:15 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-5107) split the core, hdfs, and mapred
 jars from each other and publish them independently to the Maven repository
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan reassigned HADOOP-5107:
------------------------------------------

    Assignee: Giridharan Kesavan

> split the core, hdfs, and mapred jars from each other and publish them independently to the Maven repository
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5107
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5107
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.0
>            Reporter: Owen O'Malley
>            Assignee: Giridharan Kesavan
>
> I think to support splitting the projects, we should publish the jars for 0.20.0 as independent jars to the Maven repository 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-342-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 10:15:33 2009
Return-Path: <common-issues-return-342-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47473 invoked from network); 16 Jul 2009 10:15:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 10:15:31 -0000
Received: (qmail 2580 invoked by uid 500); 16 Jul 2009 10:16:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2496 invoked by uid 500); 16 Jul 2009 10:16:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2465 invoked by uid 99); 16 Jul 2009 10:16:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 10:16:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 10:16:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E6101234C045
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 03:16:14 -0700 (PDT)
Message-ID: <2094177475.1247739374941.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 03:16:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5107) split the core, hdfs, and mapred jars
 from each other and publish them independently to the Maven repository
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-5107:
---------------------------------------

    Attachment: mapreduce-trunk.patch
                hdfs-trunk.patch
                common-trunk.patch

> split the core, hdfs, and mapred jars from each other and publish them independently to the Maven repository
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5107
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5107
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.0
>            Reporter: Owen O'Malley
>            Assignee: Giridharan Kesavan
>         Attachments: common-trunk.patch, hdfs-trunk.patch, mapreduce-trunk.patch
>
>
> I think to support splitting the projects, we should publish the jars for 0.20.0 as independent jars to the Maven repository 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-343-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 10:23:32 2009
Return-Path: <common-issues-return-343-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52305 invoked from network); 16 Jul 2009 10:23:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 10:23:31 -0000
Received: (qmail 17619 invoked by uid 500); 16 Jul 2009 10:24:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17585 invoked by uid 500); 16 Jul 2009 10:24:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17575 invoked by uid 99); 16 Jul 2009 10:24:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 10:24:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 10:24:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 10CDA234C1E7
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 03:24:15 -0700 (PDT)
Message-ID: <221676377.1247739855067.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 03:24:15 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5107) split the core, hdfs, and mapred
 jars from each other and publish them independently to the Maven repository
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731884#action_12731884 ] 

Giridharan Kesavan commented on HADOOP-5107:
--------------------------------------------

steps:

To publish common's jar's to the local repository : ie : /home/<username>/ivyrepo
cd common-trunk
apply common-trunk.patch
ant ivy-publish-local
this would publish hadoop-core and hadoop-core-test jar to the local filesystem based repository.

cd hdfs-trunk
apply hdfs-trunk.patch 
ant ivy-publish-local -Dresolver=local 
this would publish hdfs jars to the local filesystem based repository 
-Dresolver=local option tells ivy to resolve the common jars from the local filesystem based repository

cd mapreduce-trunk
apply mapreduce-trunk.patch
ant ivy-publish-local -Dresolver=local 
this would publish mapred jars to the local filesystem based repository 
-Dresolver=local option tells ivy to resolve the common and hdfs jars from the local filesystem based repository

this patch also has a ssh based resolver that publishes artifacts to the people server's home folder but that requires authentication.

> split the core, hdfs, and mapred jars from each other and publish them independently to the Maven repository
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5107
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5107
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.0
>            Reporter: Owen O'Malley
>            Assignee: Giridharan Kesavan
>         Attachments: common-trunk.patch, hdfs-trunk.patch, mapreduce-trunk.patch
>
>
> I think to support splitting the projects, we should publish the jars for 0.20.0 as independent jars to the Maven repository 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-344-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 10:23:34 2009
Return-Path: <common-issues-return-344-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52388 invoked from network); 16 Jul 2009 10:23:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 10:23:34 -0000
Received: (qmail 17726 invoked by uid 500); 16 Jul 2009 10:24:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17684 invoked by uid 500); 16 Jul 2009 10:24:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17674 invoked by uid 99); 16 Jul 2009 10:24:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 10:24:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 10:24:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D6BCE234C004
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 03:24:14 -0700 (PDT)
Message-ID: <863476739.1247739854864.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 03:24:14 -0700 (PDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4491) Per-job local data on the
 TaskTracker node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731883#action_12731883 ] 

Hemanth Yamijala commented on HADOOP-4491:
------------------------------------------

I looked at the C code that was pending from previous review. Looking good overall. I have a few comments - mostly minor:

- Should the pointer returned by strdup in configuration.get_value() be freed ? It doesn't seem to be in some cases.
- Return value of any code path calling concatenate should be checked, as it can be null.
- I think we can use stat instead of opendir to check for existence of path.
- Check return value of fts calls.
- I think it may be good to use the flag FTS_NOSTAT as we don't use stat info and it looks like it is an optimization to do so.
- It may be safe to handle a default case in secure_path, just in case versions of FTS have types we are not handling. For e.g we are not handling FTS_ERR.
- The value of the process_flag seems against the name. It should be true (non zero) if we want to process and false (zero) otherwise. Then code like if (!process_path) continue; can be used. Similarly for the variables 'failed' and 'first_task'
- We can use strerror that will print a standard message per errno itself, for calls like chown etc. rather than printing the error message ourselves based on errno.
- What are the permissions for the tasklog attempt directories ? Since they're not being set, will they take the value of the default umask ?
- check_owner is removed. Need to make sure this is ok.
- Let's add debug prints into #ifdef DEBUG or something similar.
- Looks like run_task_as_user shares code with initialize_task. Can it call initialize_task directly ?
- In TERMINATE_TASK_JVM, we are not using first_task - remove the argument to task-controller. Also for clarity, I think we can call the argument as cleanup_work_dirs or something like that, rather than first_task.
- There seems to be a spurious call to open_log_file in TERMINATE_TASK_JVM.
- display_usage seems out of place in task-controller.c. Can we move it to main ?

> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-345-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 10:35:35 2009
Return-Path: <common-issues-return-345-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59461 invoked from network); 16 Jul 2009 10:35:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 10:35:33 -0000
Received: (qmail 45830 invoked by uid 500); 16 Jul 2009 10:36:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45793 invoked by uid 500); 16 Jul 2009 10:36:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45783 invoked by uid 99); 16 Jul 2009 10:36:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 10:36:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 10:36:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D0201234C004
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 03:36:14 -0700 (PDT)
Message-ID: <1884041079.1247740574845.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 03:36:14 -0700 (PDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6120) Add support for Avro types in
 hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731891#action_12731891 ] 

Sharad Agarwal commented on HADOOP-6120:
----------------------------------------

bq. shouldn't Serializer/Desrializer#close() call encoder/decoder.close()?
I don't see encoder/decoder has close() method. Am I missing something?

> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch, 6120_v3.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-346-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 10:39:33 2009
Return-Path: <common-issues-return-346-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60344 invoked from network); 16 Jul 2009 10:39:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 10:39:31 -0000
Received: (qmail 57824 invoked by uid 500); 16 Jul 2009 10:40:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57773 invoked by uid 500); 16 Jul 2009 10:40:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57570 invoked by uid 99); 16 Jul 2009 10:40:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 10:40:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 10:40:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D0E8D234C004
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 03:40:14 -0700 (PDT)
Message-ID: <1624232401.1247740814851.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 03:40:14 -0700 (PDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6120) Add support for Avro types in hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sharad Agarwal updated HADOOP-6120:
-----------------------------------

    Attachment: 6120_v4.patch

Taken care of Tom and Doug's comments:
# Moved the testSerialization() method from TestWritableSerialization
# Added package.html to the org.apache.hadoop.io.serializer.avro
# Made AvroReflectSerialization#accept() as synchronized

> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch, 6120_v3.patch, 6120_v4.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-347-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 10:45:33 2009
Return-Path: <common-issues-return-347-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61980 invoked from network); 16 Jul 2009 10:45:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 10:45:33 -0000
Received: (qmail 66516 invoked by uid 500); 16 Jul 2009 10:46:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66478 invoked by uid 500); 16 Jul 2009 10:46:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66468 invoked by uid 99); 16 Jul 2009 10:46:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 10:46:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 10:46:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E25E5234C04C
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 03:46:14 -0700 (PDT)
Message-ID: <432710077.1247741174926.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 03:46:14 -0700 (PDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6120) Add support for Avro types in
 hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731897#action_12731897 ] 

Sharad Agarwal commented on HADOOP-6120:
----------------------------------------

bq. Have you tried using either of these serializations with MapReduce? Something like TestJavaSerialization would be good (but this could be another issue).
Now after the project split this *has* to be another issue. *smile*

> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch, 6120_v3.patch, 6120_v4.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-348-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 10:57:32 2009
Return-Path: <common-issues-return-348-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65373 invoked from network); 16 Jul 2009 10:57:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 10:57:31 -0000
Received: (qmail 89857 invoked by uid 500); 16 Jul 2009 10:58:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89796 invoked by uid 500); 16 Jul 2009 10:58:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89778 invoked by uid 99); 16 Jul 2009 10:58:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 10:58:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 10:58:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3B94E234C004
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 03:58:15 -0700 (PDT)
Message-ID: <1659293976.1247741895235.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 03:58:15 -0700 (PDT)
From: "He Yongqiang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6138) eliminate the depracate warnings
 introduced by H-5438
In-Reply-To: <883885751.1247164334967.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

He Yongqiang updated HADOOP-6138:
---------------------------------

    Status: Patch Available  (was: Open)

> eliminate the depracate warnings introduced by H-5438
> -----------------------------------------------------
>
>                 Key: HADOOP-6138
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6138
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: He Yongqiang
>            Priority: Minor
>         Attachments: hadoop-6138-2009-07-15.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-349-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 11:39:34 2009
Return-Path: <common-issues-return-349-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76584 invoked from network); 16 Jul 2009 11:39:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 11:39:34 -0000
Received: (qmail 63143 invoked by uid 500); 16 Jul 2009 11:40:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63104 invoked by uid 500); 16 Jul 2009 11:40:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63094 invoked by uid 99); 16 Jul 2009 11:40:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 11:40:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 11:40:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 21EAA234C004
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 04:40:15 -0700 (PDT)
Message-ID: <1681158606.1247744415114.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 04:40:15 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6105) Provide a way to automatically
 handle backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731908#action_12731908 ] 

V.V.Chaitanya Krishna commented on HADOOP-6105:
-----------------------------------------------

 Assumptions:
 1. None of the *-default.xml files would have deprecated keys.
 
========
Changes to set and get method in configuration.

get(name): the key(name) is checked in the deprecation map. If present,
the method returns the value of the first key in the list of new keys.

set(name,value): the key (name) is checked in the deprecation map. If
present, the method sets all the new replacing keys with "value".

========
Following table describes the set and get behaviour:

||Old Key ||New Key ||get(oldkey) ||get(newKey)||
|set|set|old/newValue(whichever is recently set)|old/new value(whichever
is recently set)|
|unSet|unSet|default value of newKey|default value of new key|
|set|unSet|oldValue|old value|
|unSet|set|newValue|newValue|

Note: the set and unset in the above table refers to the keys being
          set dynamically i.e calling Configuration setter methods.


> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-350-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 13:47:34 2009
Return-Path: <common-issues-return-350-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41547 invoked from network); 16 Jul 2009 13:47:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 13:47:33 -0000
Received: (qmail 46218 invoked by uid 500); 16 Jul 2009 13:48:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46198 invoked by uid 500); 16 Jul 2009 13:48:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46188 invoked by uid 99); 16 Jul 2009 13:48:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 13:48:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 13:48:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CCF1D234C044
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 06:48:14 -0700 (PDT)
Message-ID: <1966042385.1247752094818.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 06:48:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6138) eliminate the depracate warnings
 introduced by H-5438
In-Reply-To: <883885751.1247164334967.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12731941#action_12731941 ] 

Hadoop QA commented on HADOOP-6138:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12413546/hadoop-6138-2009-07-15.patch
  against trunk revision 793987.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    -1 release audit.  The applied patch generated 262 release audit warnings (more than the trunk's current 260 warnings).

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/575/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/575/artifact/trunk/current/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/575/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/575/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/575/console

This message is automatically generated.

> eliminate the depracate warnings introduced by H-5438
> -----------------------------------------------------
>
>                 Key: HADOOP-6138
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6138
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: He Yongqiang
>            Priority: Minor
>         Attachments: hadoop-6138-2009-07-15.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-351-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 16:08:32 2009
Return-Path: <common-issues-return-351-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30712 invoked from network); 16 Jul 2009 16:08:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 16:08:32 -0000
Received: (qmail 34714 invoked by uid 500); 16 Jul 2009 16:09:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34685 invoked by uid 500); 16 Jul 2009 16:09:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34648 invoked by uid 99); 16 Jul 2009 16:09:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 16:09:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 16:09:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CAC27234C004
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 09:09:14 -0700 (PDT)
Message-ID: <953989948.1247760554818.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 09:09:14 -0700 (PDT)
From: "Philip Zeyliger (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6105) Provide a way to automatically
 handle backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732002#action_12732002 ] 

Philip Zeyliger commented on HADOOP-6105:
-----------------------------------------

You might consider logging a warning every time you run into (either via get or set) a "set" deprecated key.

> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-352-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 17:00:34 2009
Return-Path: <common-issues-return-352-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45686 invoked from network); 16 Jul 2009 17:00:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 17:00:31 -0000
Received: (qmail 98887 invoked by uid 500); 16 Jul 2009 17:01:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98851 invoked by uid 500); 16 Jul 2009 17:01:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98840 invoked by uid 99); 16 Jul 2009 17:01:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 17:01:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 17:01:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1F5BA29A0023
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 10:01:15 -0700 (PDT)
Message-ID: <532291399.1247763675127.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 10:01:15 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6120) Add support for Avro types in hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-6120:
---------------------------------

    Attachment: HADOOP-6120.patch

Attaching a patch that's identical to Sharad's last patch, except that Avro is now retrieved from the Maven repo.  This should now hopefully pass Hudson tests.

> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch, 6120_v3.patch, 6120_v4.patch, HADOOP-6120.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-353-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 17:00:35 2009
Return-Path: <common-issues-return-353-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45703 invoked from network); 16 Jul 2009 17:00:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 17:00:33 -0000
Received: (qmail 99013 invoked by uid 500); 16 Jul 2009 17:01:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98977 invoked by uid 500); 16 Jul 2009 17:01:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98967 invoked by uid 99); 16 Jul 2009 17:01:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 17:01:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 17:01:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E528D29A0011
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 10:01:14 -0700 (PDT)
Message-ID: <1218210590.1247763674937.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 10:01:14 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6120) Add support for Avro types in hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-6120:
---------------------------------

    Status: Open  (was: Patch Available)

> I don't see encoder/decoder has close() method. Am I missing something?

No, it seems Avro is.  I think the contract for serializers and deserializers is that they can close the underlying stream.  So, until we add Encoder/Decoder#close() to Avro, we should explicitly save a pointer to the stream passed in and close it after flushing the encoder/decoder.

A few other comments.
 - AvroSerialization should be public, since public classes extend it.
 - Every public or protected method and field in public classes should have javadoc, unless its inherited.
 - We should add a bit more documentation at the package and class level about how this is to be used, e.g., explain that reflect serialization will work for any class that's either in the configured package list or that implements the tag interface, and that specific serialization will just work for generated classes.
 - Is there any reason not to add these serializers to io.serializations by default?


> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch, 6120_v3.patch, 6120_v4.patch, HADOOP-6120.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-354-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 17:02:33 2009
Return-Path: <common-issues-return-354-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46721 invoked from network); 16 Jul 2009 17:02:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 17:02:33 -0000
Received: (qmail 2491 invoked by uid 500); 16 Jul 2009 17:03:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2452 invoked by uid 500); 16 Jul 2009 17:03:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2439 invoked by uid 99); 16 Jul 2009 17:03:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 17:03:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 17:03:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D1F2E234C004
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 10:03:14 -0700 (PDT)
Message-ID: <1851467476.1247763794845.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 10:03:14 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6120) Add support for Avro types in hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-6120:
---------------------------------

    Status: Patch Available  (was: Open)

Marking as "Patch Available" to see if Hudson can now find Avro.

> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch, 6120_v3.patch, 6120_v4.patch, HADOOP-6120.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-355-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 19:03:35 2009
Return-Path: <common-issues-return-355-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11045 invoked from network); 16 Jul 2009 19:03:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 19:03:35 -0000
Received: (qmail 25649 invoked by uid 500); 16 Jul 2009 19:04:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25563 invoked by uid 500); 16 Jul 2009 19:04:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25531 invoked by uid 99); 16 Jul 2009 19:04:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 19:04:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 19:04:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 355FF29A0011
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 12:04:15 -0700 (PDT)
Message-ID: <1959763833.1247771055217.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 12:04:15 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6120) Add support for Avro types in
 hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732088#action_12732088 ] 

Hadoop QA commented on HADOOP-6120:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12413707/HADOOP-6120.patch
  against trunk revision 793987.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 15 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    -1 release audit.  The applied patch generated 261 release audit warnings (more than the trunk's current 260 warnings).

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/576/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/576/artifact/trunk/current/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/576/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/576/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/576/console

This message is automatically generated.

> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch, 6120_v3.patch, 6120_v4.patch, HADOOP-6120.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-356-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 19:25:33 2009
Return-Path: <common-issues-return-356-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20820 invoked from network); 16 Jul 2009 19:25:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 19:25:33 -0000
Received: (qmail 54986 invoked by uid 500); 16 Jul 2009 19:26:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54955 invoked by uid 500); 16 Jul 2009 19:26:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54940 invoked by uid 99); 16 Jul 2009 19:26:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 19:26:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 19:26:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E780729A0012
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 12:26:14 -0700 (PDT)
Message-ID: <570309215.1247772374947.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 12:26:14 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6120) Add support for Avro types in hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-6120:
---------------------------------

    Status: Open  (was: Patch Available)

Everything passed but the release audit.  Not bad.

The "release audit warnings" link above is broken, but the offending file is avroRecord.avsc.  We should be able to put an Apache license at top of that file, since Jackson, the JSON parser that Avro uses, permits C-style comments.

> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch, 6120_v3.patch, 6120_v4.patch, HADOOP-6120.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-357-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 19:37:35 2009
Return-Path: <common-issues-return-357-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26658 invoked from network); 16 Jul 2009 19:37:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 19:37:35 -0000
Received: (qmail 75307 invoked by uid 500); 16 Jul 2009 19:38:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75265 invoked by uid 500); 16 Jul 2009 19:38:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74960 invoked by uid 99); 16 Jul 2009 19:38:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 19:38:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 19:38:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id F32C429A0013
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 12:38:14 -0700 (PDT)
Message-ID: <1828549601.1247773094995.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 12:38:14 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6148:
--------------------------------

    Attachment: hadoop-6148.txt

Attached is an updated patch. The changes here from the previous version:

- Replaced the implementation with the NewInner version by Scott.
- Cleaned up the Performance Test a bit and included it as a static inner class of TestPureJavaCrc32 - it can be run using java 'org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest'
- The patch also includes changes to ChecksumFileSystem and util.DataChecksum

Note that I did not move the test out of src/test/core - it looks like src/test/core still exists in the common repository - all of the tests for common are still in there.

As for testing, I reran the unit test, temporarily modified to do several million trials and it passed. I also ran TestChecksumFileSystem and it passed. I re-ran the performance test and made sure it was consistent with the results we've been seeing all along.

After this is committed, there will be a small patch to HDFS and a small patch to MapReduce to sub out PureJavaCrc32 for java.util.zip.CRC32.

> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: benchmarks20090714.txt, benchmarks20090715.txt, crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hadoop-6148.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32New.java, PureJavaCrc32NewInner.java, PureJavaCrc32NewLoop.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-358-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 19:41:34 2009
Return-Path: <common-issues-return-358-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28257 invoked from network); 16 Jul 2009 19:41:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 19:41:33 -0000
Received: (qmail 78950 invoked by uid 500); 16 Jul 2009 19:42:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78865 invoked by uid 500); 16 Jul 2009 19:42:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78547 invoked by uid 99); 16 Jul 2009 19:42:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 19:42:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 19:42:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 42181234C044
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 12:42:15 -0700 (PDT)
Message-ID: <1314387060.1247773335269.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 12:42:15 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6148:
--------------------------------

    Status: Patch Available  (was: Open)

> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: benchmarks20090714.txt, benchmarks20090715.txt, crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hadoop-6148.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32New.java, PureJavaCrc32NewInner.java, PureJavaCrc32NewLoop.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-359-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 21:55:33 2009
Return-Path: <common-issues-return-359-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84484 invoked from network); 16 Jul 2009 21:55:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 21:55:33 -0000
Received: (qmail 78401 invoked by uid 500); 16 Jul 2009 21:56:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78363 invoked by uid 500); 16 Jul 2009 21:56:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78352 invoked by uid 99); 16 Jul 2009 21:56:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 21:56:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 21:56:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CACC6234C004
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 14:56:14 -0700 (PDT)
Message-ID: <1181794958.1247781374819.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 14:56:14 -0700 (PDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5864) Fix DMI and OBL findbugs in packages
 hdfs and metrics
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-5864:
------------------------------------

    Attachment: findbugs3.hdfs.patch

After project split, the change made to findbugExcludeFile.xml is missing in the HDFS subproject. Attaching a patch to introduce the change again.

> Fix DMI and OBL findbugs in packages hdfs and metrics
> -----------------------------------------------------
>
>                 Key: HADOOP-5864
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5864
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.21.0
>
>         Attachments: findBugs.patch, findBugs1.patch, findBugs2.patch, findbugs3.hdfs.patch
>
>
> This issue is to fix the following findbugs:
> DMI in org.apache.hadoop.hdfs.server.namenode.INodeDirectory.getExistingPathINodes(byte[][], INode[]) 
> OBL in org.apache.hadoop.hdfs.server.datanode.DataStorage.linkBlocks(File, File, int)
> OBL in org.apache.hadoop.hdfs.server.datanode.FSDataset.createBlockWriteStreams(File, File)
> OBL in org.apache.hadoop.hdfs.server.datanode.FSDataset.getTmpInputStreams(Block, long, long)
> OBL in org.apache.hadoop.metrics.ContextFactory.setAttributes() 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-360-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 22:33:32 2009
Return-Path: <common-issues-return-360-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1455 invoked from network); 16 Jul 2009 22:33:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 22:33:32 -0000
Received: (qmail 20768 invoked by uid 500); 16 Jul 2009 22:34:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20730 invoked by uid 500); 16 Jul 2009 22:34:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20716 invoked by uid 99); 16 Jul 2009 22:34:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 22:34:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 22:34:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 08600234C004
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 15:34:15 -0700 (PDT)
Message-ID: <263502489.1247783655020.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 15:34:15 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732210#action_12732210 ] 

Hadoop QA commented on HADOOP-6148:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12413728/hadoop-6148.txt
  against trunk revision 793987.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/577/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/577/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/577/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/577/console

This message is automatically generated.

> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: benchmarks20090714.txt, benchmarks20090715.txt, crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hadoop-6148.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32New.java, PureJavaCrc32NewInner.java, PureJavaCrc32NewLoop.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-361-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 22:33:35 2009
Return-Path: <common-issues-return-361-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1517 invoked from network); 16 Jul 2009 22:33:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 22:33:35 -0000
Received: (qmail 20890 invoked by uid 500); 16 Jul 2009 22:34:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20811 invoked by uid 500); 16 Jul 2009 22:34:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20792 invoked by uid 99); 16 Jul 2009 22:34:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 22:34:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 22:34:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 29B1829A001D
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 15:34:15 -0700 (PDT)
Message-ID: <365169664.1247783655169.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 15:34:15 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6155) deprecate Record IO
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

deprecate Record IO
-------------------

                 Key: HADOOP-6155
                 URL: https://issues.apache.org/jira/browse/HADOOP-6155
             Project: Hadoop Common
          Issue Type: Task
          Components: record
            Reporter: Owen O'Malley


With the advent of Avro, I think we should deprecate Record IO.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-362-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 22:35:32 2009
Return-Path: <common-issues-return-362-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2369 invoked from network); 16 Jul 2009 22:35:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 22:35:31 -0000
Received: (qmail 23318 invoked by uid 500); 16 Jul 2009 22:36:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23256 invoked by uid 500); 16 Jul 2009 22:36:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23237 invoked by uid 99); 16 Jul 2009 22:36:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 22:36:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 22:36:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E053929A0012
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 15:36:14 -0700 (PDT)
Message-ID: <388230924.1247783774918.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 15:36:14 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6156) Move map/reduce specific classes out
 of common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Move map/reduce specific classes out of common
----------------------------------------------

                 Key: HADOOP-6156
                 URL: https://issues.apache.org/jira/browse/HADOOP-6156
             Project: Hadoop Common
          Issue Type: Task
          Components: util
            Reporter: Owen O'Malley


Move Sort functions, process tree, and memory calculator classes out of Common and into Map/Reduce.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-363-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 22:39:34 2009
Return-Path: <common-issues-return-363-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3879 invoked from network); 16 Jul 2009 22:39:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 22:39:32 -0000
Received: (qmail 33853 invoked by uid 500); 16 Jul 2009 22:40:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33824 invoked by uid 500); 16 Jul 2009 22:40:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33794 invoked by uid 99); 16 Jul 2009 22:40:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 22:40:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 22:40:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DA14E29A0012
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 15:40:14 -0700 (PDT)
Message-ID: <2120909175.1247784014892.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 15:40:14 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6157) Deprecate the Daemon class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Deprecate the Daemon class
--------------------------

                 Key: HADOOP-6157
                 URL: https://issues.apache.org/jira/browse/HADOOP-6157
             Project: Hadoop Common
          Issue Type: Task
          Components: util
            Reporter: Owen O'Malley


The Daemon class has very little value and should be deprecated and then removed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-364-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 22:47:34 2009
Return-Path: <common-issues-return-364-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5181 invoked from network); 16 Jul 2009 22:47:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 22:47:33 -0000
Received: (qmail 42983 invoked by uid 500); 16 Jul 2009 22:48:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42942 invoked by uid 500); 16 Jul 2009 22:48:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42932 invoked by uid 99); 16 Jul 2009 22:48:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 22:48:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 22:48:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D83B629A0012
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 15:48:14 -0700 (PDT)
Message-ID: <1823292187.1247784494884.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 15:48:14 -0700 (PDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6132) RPC client opens an extra
 connection for VersionedProtocol
In-Reply-To: <840919907.1247085075163.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732218#action_12732218 ] 

Kan Zhang commented on HADOOP-6132:
-----------------------------------

The manual verification I used was running a test that involves the RPC layer (e.g., a test that uses the MiniDFSCluster), with log level of ipc.Server set to debug. Without this patch, one would observe the following line in the log, which means the client is establishing a connection for the VersionedProtocol. It happens whenever a client sets up the first connection for a protocol (e.g., ClientProtocol or DatanodeProtocol). When the bug is fixed, no connection for the VersionedProtocol should ever be set up. Hence, no such log messages should have been observed in the log.
{noformat}
2009-07-16 15:41:37,763 DEBUG ipc.Server (Server.java:readAndProcess(861)) - Successfully authorized org.apache.hadoop.ipc.VersionedProtocol-......
{noformat}

> RPC client opens an extra connection for VersionedProtocol
> ----------------------------------------------------------
>
>                 Key: HADOOP-6132
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6132
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: 1.patch, 2.patch
>
>
> RPC client caches connections per protocol. However, since all of our real protocols are subclasses of VersionedProtocol, a bug in the implementation makes the client opens an extra connection just for the VersionedProtocol, which is not needed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-365-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 22:55:32 2009
Return-Path: <common-issues-return-365-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6363 invoked from network); 16 Jul 2009 22:55:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 22:55:31 -0000
Received: (qmail 47987 invoked by uid 500); 16 Jul 2009 22:56:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47951 invoked by uid 500); 16 Jul 2009 22:56:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47939 invoked by uid 99); 16 Jul 2009 22:56:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 22:56:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 22:56:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E4EF1234C044
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 15:56:14 -0700 (PDT)
Message-ID: <2041057326.1247784974936.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 15:56:14 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6158) Move CyclicIteration to HDFS
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Move CyclicIteration to HDFS
----------------------------

                 Key: HADOOP-6158
                 URL: https://issues.apache.org/jira/browse/HADOOP-6158
             Project: Hadoop Common
          Issue Type: Task
          Components: util
            Reporter: Owen O'Malley


I think we should move CyclicIteration from Common utils to HDFS.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-366-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 22:55:32 2009
Return-Path: <common-issues-return-366-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6409 invoked from network); 16 Jul 2009 22:55:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 22:55:32 -0000
Received: (qmail 48101 invoked by uid 500); 16 Jul 2009 22:56:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48022 invoked by uid 500); 16 Jul 2009 22:56:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48003 invoked by uid 99); 16 Jul 2009 22:56:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 22:56:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 22:56:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E96B629A0012
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 15:56:14 -0700 (PDT)
Message-ID: <1789331991.1247784974955.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 15:56:14 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6148:
--------------------------------

    Attachment: hadoop-6148.txt

Attached is a slightly updated patch which fixes the javadoc warning (there was an @inheritDoc on a constructor, which isn't allowed)

> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: benchmarks20090714.txt, benchmarks20090715.txt, crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hadoop-6148.txt, hadoop-6148.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32New.java, PureJavaCrc32NewInner.java, PureJavaCrc32NewLoop.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-367-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 22:55:32 2009
Return-Path: <common-issues-return-367-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6403 invoked from network); 16 Jul 2009 22:55:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 22:55:32 -0000
Received: (qmail 48100 invoked by uid 500); 16 Jul 2009 22:56:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48024 invoked by uid 500); 16 Jul 2009 22:56:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48002 invoked by uid 99); 16 Jul 2009 22:56:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 22:56:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 22:56:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 19A7929A001F
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 15:56:15 -0700 (PDT)
Message-ID: <1532138457.1247784975104.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 15:56:15 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6148:
--------------------------------

    Status: Open  (was: Patch Available)

> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: benchmarks20090714.txt, benchmarks20090715.txt, crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hadoop-6148.txt, hadoop-6148.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32New.java, PureJavaCrc32NewInner.java, PureJavaCrc32NewLoop.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-368-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 22:55:35 2009
Return-Path: <common-issues-return-368-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6606 invoked from network); 16 Jul 2009 22:55:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 22:55:35 -0000
Received: (qmail 49063 invoked by uid 500); 16 Jul 2009 22:56:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49025 invoked by uid 500); 16 Jul 2009 22:56:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49015 invoked by uid 99); 16 Jul 2009 22:56:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 22:56:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 22:56:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 6F8CE29A002D
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 15:56:15 -0700 (PDT)
Message-ID: <514903313.1247784975456.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 15:56:15 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6148:
--------------------------------

    Status: Patch Available  (was: Open)

> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: benchmarks20090714.txt, benchmarks20090715.txt, crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hadoop-6148.txt, hadoop-6148.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32New.java, PureJavaCrc32NewInner.java, PureJavaCrc32NewLoop.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-369-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 22:57:33 2009
Return-Path: <common-issues-return-369-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6904 invoked from network); 16 Jul 2009 22:57:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 22:57:32 -0000
Received: (qmail 50086 invoked by uid 500); 16 Jul 2009 22:58:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50017 invoked by uid 500); 16 Jul 2009 22:58:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49702 invoked by uid 99); 16 Jul 2009 22:58:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 22:58:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 22:58:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0EBE3234C004
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 15:58:15 -0700 (PDT)
Message-ID: <175487592.1247785095033.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 15:58:15 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5864) Fix DMI and OBL findbugs in
 packages hdfs and metrics
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732228#action_12732228 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-5864:
------------------------------------------------

The changed was committed by Hairong before.  We should investigate why the change disappeared.

> Fix DMI and OBL findbugs in packages hdfs and metrics
> -----------------------------------------------------
>
>                 Key: HADOOP-5864
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5864
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.21.0
>
>         Attachments: findBugs.patch, findBugs1.patch, findBugs2.patch, findbugs3.hdfs.patch
>
>
> This issue is to fix the following findbugs:
> DMI in org.apache.hadoop.hdfs.server.namenode.INodeDirectory.getExistingPathINodes(byte[][], INode[]) 
> OBL in org.apache.hadoop.hdfs.server.datanode.DataStorage.linkBlocks(File, File, int)
> OBL in org.apache.hadoop.hdfs.server.datanode.FSDataset.createBlockWriteStreams(File, File)
> OBL in org.apache.hadoop.hdfs.server.datanode.FSDataset.getTmpInputStreams(Block, long, long)
> OBL in org.apache.hadoop.metrics.ContextFactory.setAttributes() 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-370-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 22:57:34 2009
Return-Path: <common-issues-return-370-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7072 invoked from network); 16 Jul 2009 22:57:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 22:57:34 -0000
Received: (qmail 50291 invoked by uid 500); 16 Jul 2009 22:58:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50261 invoked by uid 500); 16 Jul 2009 22:58:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50247 invoked by uid 99); 16 Jul 2009 22:58:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 22:58:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 22:58:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1EAE729A0012
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 15:58:15 -0700 (PDT)
Message-ID: <762347884.1247785095124.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 15:58:15 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5864) Fix DMI and OBL findbugs in
 packages hdfs and metrics
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732229#action_12732229 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-5864:
------------------------------------------------

The changed was committed by Hairong before.  We should investigate why the change disappeared.

> Fix DMI and OBL findbugs in packages hdfs and metrics
> -----------------------------------------------------
>
>                 Key: HADOOP-5864
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5864
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.21.0
>
>         Attachments: findBugs.patch, findBugs1.patch, findBugs2.patch, findbugs3.hdfs.patch
>
>
> This issue is to fix the following findbugs:
> DMI in org.apache.hadoop.hdfs.server.namenode.INodeDirectory.getExistingPathINodes(byte[][], INode[]) 
> OBL in org.apache.hadoop.hdfs.server.datanode.DataStorage.linkBlocks(File, File, int)
> OBL in org.apache.hadoop.hdfs.server.datanode.FSDataset.createBlockWriteStreams(File, File)
> OBL in org.apache.hadoop.hdfs.server.datanode.FSDataset.getTmpInputStreams(Block, long, long)
> OBL in org.apache.hadoop.metrics.ContextFactory.setAttributes() 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-371-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 23:35:35 2009
Return-Path: <common-issues-return-371-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21884 invoked from network); 16 Jul 2009 23:35:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 23:35:34 -0000
Received: (qmail 95776 invoked by uid 500); 16 Jul 2009 23:36:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95715 invoked by uid 500); 16 Jul 2009 23:36:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95498 invoked by uid 99); 16 Jul 2009 23:36:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 23:36:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 23:36:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D447D234C044
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 16:36:14 -0700 (PDT)
Message-ID: <1326160022.1247787374868.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 16:36:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6159) Remove "core" from build.xml
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Remove "core" from build.xml
----------------------------

                 Key: HADOOP-6159
                 URL: https://issues.apache.org/jira/browse/HADOOP-6159
             Project: Hadoop Common
          Issue Type: Bug
          Components: build
            Reporter: Tsz Wo (Nicholas), SZE


build.xml still creating "core" directories and other stuffs such as
{code}
  <property name="test.core.build.classes" value="${test.build.dir}/core/classes"/>

...

  <property name="jdiff.stable.javadoc" 
            value="http://hadoop.apache.org/core/docs/r${jdiff.stable}/api/"/>
{code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-372-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 16 23:43:33 2009
Return-Path: <common-issues-return-372-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23238 invoked from network); 16 Jul 2009 23:43:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Jul 2009 23:43:33 -0000
Received: (qmail 5179 invoked by uid 500); 16 Jul 2009 23:44:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5140 invoked by uid 500); 16 Jul 2009 23:44:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5130 invoked by uid 99); 16 Jul 2009 23:44:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 23:44:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jul 2009 23:44:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D14C7234C045
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 16:44:14 -0700 (PDT)
Message-ID: <864618968.1247787854856.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 16:44:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6138) eliminate the depracate warnings
 introduced by H-5438
In-Reply-To: <883885751.1247164334967.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6138:
-------------------------------------------

        Assignee: He Yongqiang
    Hadoop Flags: [Reviewed]

+1 patch looks good.

Does anyone know why we got "-1 release audit."?

> eliminate the depracate warnings introduced by H-5438
> -----------------------------------------------------
>
>                 Key: HADOOP-6138
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6138
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: He Yongqiang
>            Assignee: He Yongqiang
>            Priority: Minor
>         Attachments: hadoop-6138-2009-07-15.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-373-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 01:00:38 2009
Return-Path: <common-issues-return-373-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46818 invoked from network); 17 Jul 2009 01:00:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 01:00:34 -0000
Received: (qmail 62637 invoked by uid 500); 17 Jul 2009 01:01:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62598 invoked by uid 500); 17 Jul 2009 01:01:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62584 invoked by uid 99); 17 Jul 2009 01:01:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 01:01:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 01:01:37 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id A60FD29A0012
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 18:01:17 -0700 (PDT)
Message-ID: <1482299059.1247792477679.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 18:01:17 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6148:
-------------------------------------------

     Component/s: util
    Hadoop Flags: [Reviewed]

{code}
diff --git a/src/java/org/apache/hadoop/fs/ChecksumFileSystem.java b/src/java/...
index 72a09bd..6f9701e 100644
--- a/src/java/org/apache/hadoop/fs/ChecksumFileSystem.java
+++ b/src/java/org/apache/hadoop/fs/ChecksumFileSystem.java
{code}
There are "a" and  "b" directories in the patch.  Hudson probably cannot handle it.

+1 Patch looks good other than that.


> BTW, the path for TestPureJavaCrc32 is wrong in the previous patch. We don't have "core" directory anymore.
I was wrong about it.  We still have "core" for the tests.  See also HADOOP-6159.

> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: benchmarks20090714.txt, benchmarks20090715.txt, crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hadoop-6148.txt, hadoop-6148.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32New.java, PureJavaCrc32NewInner.java, PureJavaCrc32NewLoop.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-374-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 01:10:34 2009
Return-Path: <common-issues-return-374-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49565 invoked from network); 17 Jul 2009 01:10:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 01:10:34 -0000
Received: (qmail 68079 invoked by uid 500); 17 Jul 2009 01:11:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68037 invoked by uid 500); 17 Jul 2009 01:11:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68026 invoked by uid 99); 17 Jul 2009 01:11:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 01:11:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 01:11:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CF5FC234C004
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 18:11:14 -0700 (PDT)
Message-ID: <1391540205.1247793074835.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 18:11:14 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6148:
--------------------------------

    Attachment: hadoop-6148.txt

Woops! Forgot to do --no-prefix on my git-diff. Same patch, just without the a/ and b/ prefixes.

> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: benchmarks20090714.txt, benchmarks20090715.txt, crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hadoop-6148.txt, hadoop-6148.txt, hadoop-6148.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32New.java, PureJavaCrc32NewInner.java, PureJavaCrc32NewLoop.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-375-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 01:43:33 2009
Return-Path: <common-issues-return-375-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74511 invoked from network); 17 Jul 2009 01:43:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 01:43:33 -0000
Received: (qmail 1150 invoked by uid 500); 17 Jul 2009 01:44:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1110 invoked by uid 500); 17 Jul 2009 01:44:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1099 invoked by uid 99); 17 Jul 2009 01:44:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 01:44:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 01:44:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D03FD234C004
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 18:44:14 -0700 (PDT)
Message-ID: <1677647803.1247795054838.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 18:44:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732298#action_12732298 ] 

Hadoop QA commented on HADOOP-6148:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12413763/hadoop-6148.txt
  against trunk revision 793987.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/578/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/578/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/578/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/578/console

This message is automatically generated.

> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: benchmarks20090714.txt, benchmarks20090715.txt, crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hadoop-6148.txt, hadoop-6148.txt, hadoop-6148.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32New.java, PureJavaCrc32NewInner.java, PureJavaCrc32NewLoop.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-376-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 02:07:31 2009
Return-Path: <common-issues-return-376-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88291 invoked from network); 17 Jul 2009 02:07:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 02:07:31 -0000
Received: (qmail 18089 invoked by uid 500); 17 Jul 2009 02:08:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18052 invoked by uid 500); 17 Jul 2009 02:08:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18041 invoked by uid 99); 17 Jul 2009 02:08:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 02:08:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 02:08:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CB141234C004
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 19:08:14 -0700 (PDT)
Message-ID: <1002817233.1247796494827.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 19:08:14 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6142) archives relative path changes in
 common.
In-Reply-To: <1194201800.1247266034972.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-6142:
----------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I committed this. Thanks, Mahadev

> archives relative path changes in common.
> -----------------------------------------
>
>                 Key: HADOOP-6142
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6142
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Mahadev konar
>            Assignee: Mahadev konar
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6142.patch
>
>
> HADOOP-3663 was moved to mapreduce accidentally. Opening this jira to upload my chanegs to common --- changes are rleated to MAPREDUCE-739.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-377-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 02:19:31 2009
Return-Path: <common-issues-return-377-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91588 invoked from network); 17 Jul 2009 02:19:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 02:19:31 -0000
Received: (qmail 20026 invoked by uid 500); 17 Jul 2009 02:20:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19989 invoked by uid 500); 17 Jul 2009 02:20:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19979 invoked by uid 99); 17 Jul 2009 02:20:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 02:20:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 02:20:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 21F4C29A0016
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 19:20:15 -0700 (PDT)
Message-ID: <208333450.1247797215137.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 19:20:15 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6148:
-------------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.21.0
           Status: Resolved  (was: Patch Available)

I have committed this.  Thanks, Todd and Scott!

> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>             Fix For: 0.21.0
>
>         Attachments: benchmarks20090714.txt, benchmarks20090715.txt, crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hadoop-6148.txt, hadoop-6148.txt, hadoop-6148.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32New.java, PureJavaCrc32NewInner.java, PureJavaCrc32NewLoop.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-378-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 03:49:34 2009
Return-Path: <common-issues-return-378-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13943 invoked from network); 17 Jul 2009 03:49:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 03:49:34 -0000
Received: (qmail 69905 invoked by uid 500); 17 Jul 2009 03:50:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69863 invoked by uid 500); 17 Jul 2009 03:50:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69844 invoked by uid 99); 17 Jul 2009 03:50:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 03:50:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 03:50:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C6F25234C004
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 20:50:14 -0700 (PDT)
Message-ID: <1561768311.1247802614801.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 20:50:14 -0700 (PDT)
From: "Raghu Angadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6132) RPC client opens an extra
 connection for VersionedProtocol
In-Reply-To: <840919907.1247085075163.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732327#action_12732327 ] 

Raghu Angadi commented on HADOOP-6132:
--------------------------------------

+1 for the fix.

There is another instance of use of getDelaringClass() in the same file (though that part of the code is not used except in some tests). 
We need to fix that.


> RPC client opens an extra connection for VersionedProtocol
> ----------------------------------------------------------
>
>                 Key: HADOOP-6132
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6132
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: 1.patch, 2.patch
>
>
> RPC client caches connections per protocol. However, since all of our real protocols are subclasses of VersionedProtocol, a bug in the implementation makes the client opens an extra connection just for the VersionedProtocol, which is not needed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-379-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 04:28:33 2009
Return-Path: <common-issues-return-379-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18995 invoked from network); 17 Jul 2009 04:28:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 04:28:33 -0000
Received: (qmail 81936 invoked by uid 500); 17 Jul 2009 04:29:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81893 invoked by uid 500); 17 Jul 2009 04:29:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81881 invoked by uid 99); 17 Jul 2009 04:29:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 04:29:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 04:29:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D2AAA234C004
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 21:29:14 -0700 (PDT)
Message-ID: <934162556.1247804954848.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 21:29:14 -0700 (PDT)
From: "Jothi Padmanabhan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4927) Part files on the output filesystem
 are created irrespective of whether the corresponding task has anything to
 write there
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jothi Padmanabhan updated HADOOP-4927:
--------------------------------------

    Attachment: hadoop-4927-y20.patch

Patch for the Y! 20 distribution.

> Part files on the output filesystem are created irrespective of whether the corresponding task has anything to write there
> --------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4927
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4927
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Devaraj Das
>            Assignee: Jothi Padmanabhan
>             Fix For: 0.21.0
>
>         Attachments: hadoop-4927-v1.patch, hadoop-4927-v2.patch, hadoop-4927-v3.patch, hadoop-4927-v4.patch, hadoop-4927-v5.patch, hadoop-4927-v6.patch, hadoop-4927-y20.patch, hadoop-4927.patch
>
>
> When OutputFormat.getRecordWriter is invoked, a part file is created on the output filesystem. But the created RecordWriter is not used until the OutputCollector.collect call is made by the task (user's code). This results in empty part files even if the OutputCollector.collect is never invoked by the corresponding tasks.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-380-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 05:23:31 2009
Return-Path: <common-issues-return-380-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28035 invoked from network); 17 Jul 2009 05:23:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 05:23:31 -0000
Received: (qmail 3337 invoked by uid 500); 17 Jul 2009 05:24:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3297 invoked by uid 500); 17 Jul 2009 05:24:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3287 invoked by uid 99); 17 Jul 2009 05:24:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 05:24:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 05:24:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CE92F234C004
	for <common-issues@hadoop.apache.org>; Thu, 16 Jul 2009 22:24:14 -0700 (PDT)
Message-ID: <1802549969.1247808254831.JavaMail.jira@brutus>
Date: Thu, 16 Jul 2009 22:24:14 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas reassigned HADOOP-6148:
-------------------------------------

    Assignee: Scott Carey  (was: Todd Lipcon)

> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Owen O'Malley
>            Assignee: Scott Carey
>             Fix For: 0.21.0
>
>         Attachments: benchmarks20090714.txt, benchmarks20090715.txt, crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hadoop-6148.txt, hadoop-6148.txt, hadoop-6148.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32New.java, PureJavaCrc32NewInner.java, PureJavaCrc32NewLoop.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-381-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 09:04:34 2009
Return-Path: <common-issues-return-381-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27425 invoked from network); 17 Jul 2009 09:04:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 09:04:34 -0000
Received: (qmail 49230 invoked by uid 500); 17 Jul 2009 09:05:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49191 invoked by uid 500); 17 Jul 2009 09:05:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49181 invoked by uid 99); 17 Jul 2009 09:05:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 09:05:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 09:05:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 25662234C051
	for <common-issues@hadoop.apache.org>; Fri, 17 Jul 2009 02:05:15 -0700 (PDT)
Message-ID: <1678953847.1247821515152.JavaMail.jira@brutus>
Date: Fri, 17 Jul 2009 02:05:15 -0700 (PDT)
From: "rahul k singh (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6105) Provide a way to automatically
 handle backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732420#action_12732420 ] 

rahul k singh commented on HADOOP-6105:
---------------------------------------

The first comment defines the initial proposal , Which says that when both new and old values are set in configuration we give preference
to new values. This we felt is not correct , we should give preference to old value.As we are trying to be backward compatible here.

So in configuration if have both deprecated key and new key , new key's values would be replaced by the deprecated key's values. 

Now in taking the above consideration in mind we have 2 ways of solving this.

1.Try to maintain a single set of key value always .When we have both deprecated key and new key , only maintain new key .So when we are trying to read the Configuration xmls , if we encounter any deprecated key , we simply replace new key's value with the deprecated key's value.
when we do "set" in configuration for the depercated keys, we simply set the new key's with the values passed . When we do "get" we simply return the new key mapping.

2. Always give preference to the deprecated key . While set of new values check if deprecated keys are present , if yes , then simply set deprecated values.


We would like to go ahead with option 1. Where we make sure that old is preferred at the start while loading the configuration , after that it
is simply the whichever key is set recently.


 

> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-382-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 09:34:36 2009
Return-Path: <common-issues-return-382-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39358 invoked from network); 17 Jul 2009 09:34:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 09:34:36 -0000
Received: (qmail 97996 invoked by uid 500); 17 Jul 2009 09:35:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97914 invoked by uid 500); 17 Jul 2009 09:35:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97865 invoked by uid 99); 17 Jul 2009 09:35:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 09:35:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 09:35:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DF63E234C044
	for <common-issues@hadoop.apache.org>; Fri, 17 Jul 2009 02:35:14 -0700 (PDT)
Message-ID: <824935473.1247823314901.JavaMail.jira@brutus>
Date: Fri, 17 Jul 2009 02:35:14 -0700 (PDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6120) Add support for Avro types in hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sharad Agarwal updated HADOOP-6120:
-----------------------------------

    Attachment: 6120_v5.patch

Implemented AvroSerialization#close, added more javadoc, made AvroSerialization as public.

bq. Is there any reason not to add these serializers to io.serializations by default?
I didn't add considering JavaSerialization also not there in default. But I think its ok to have these. Added both Avro and java serializations to core-default.xml and SerializationFactory.

> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch, 6120_v3.patch, 6120_v4.patch, 6120_v5.patch, HADOOP-6120.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-383-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 09:36:35 2009
Return-Path: <common-issues-return-383-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40211 invoked from network); 17 Jul 2009 09:36:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 09:36:33 -0000
Received: (qmail 403 invoked by uid 500); 17 Jul 2009 09:37:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 363 invoked by uid 500); 17 Jul 2009 09:37:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 348 invoked by uid 99); 17 Jul 2009 09:37:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 09:37:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 09:37:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D24E0234C044
	for <common-issues@hadoop.apache.org>; Fri, 17 Jul 2009 02:37:14 -0700 (PDT)
Message-ID: <598490270.1247823434847.JavaMail.jira@brutus>
Date: Fri, 17 Jul 2009 02:37:14 -0700 (PDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6120) Add support for Avro types in hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sharad Agarwal updated HADOOP-6120:
-----------------------------------

    Status: Patch Available  (was: Open)

> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch, 6120_v3.patch, 6120_v4.patch, 6120_v5.patch, HADOOP-6120.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-384-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 09:52:33 2009
Return-Path: <common-issues-return-384-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48635 invoked from network); 17 Jul 2009 09:52:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 09:52:33 -0000
Received: (qmail 11375 invoked by uid 500); 17 Jul 2009 09:53:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11334 invoked by uid 500); 17 Jul 2009 09:53:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11324 invoked by uid 99); 17 Jul 2009 09:53:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 09:53:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 09:53:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 5E8D0234C044
	for <common-issues@hadoop.apache.org>; Fri, 17 Jul 2009 02:53:15 -0700 (PDT)
Message-ID: <1470136567.1247824395379.JavaMail.jira@brutus>
Date: Fri, 17 Jul 2009 02:53:15 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6120) Add support for Avro types in
 hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732440#action_12732440 ] 

Tom White commented on HADOOP-6120:
-----------------------------------

JavaSerialization was written more to demonstrate the generality of the abstraction (HADOOP-1986) than to be used in real-world applications, since its performance is considerably less than Writables (and presumably Avro). So I would be inclined not to add it to the default list, preferring users to explicitly enable it. Does this sound reasonable?

> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch, 6120_v3.patch, 6120_v4.patch, 6120_v5.patch, HADOOP-6120.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-385-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 10:14:31 2009
Return-Path: <common-issues-return-385-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59003 invoked from network); 17 Jul 2009 10:14:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 10:14:31 -0000
Received: (qmail 43063 invoked by uid 500); 17 Jul 2009 10:15:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43023 invoked by uid 500); 17 Jul 2009 10:15:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42951 invoked by uid 99); 17 Jul 2009 10:15:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 10:15:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 10:15:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E996E234C045
	for <common-issues@hadoop.apache.org>; Fri, 17 Jul 2009 03:15:14 -0700 (PDT)
Message-ID: <1694178380.1247825714955.JavaMail.jira@brutus>
Date: Fri, 17 Jul 2009 03:15:14 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Reopened: (HADOOP-5935) Hudson's release audit warnings link
 is broken
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White reopened HADOOP-5935:
-------------------------------


Looks like this change was lost during the project split. See link in HADOOP-6138 for example.

> Hudson's release audit warnings link is broken
> ----------------------------------------------
>
>                 Key: HADOOP-5935
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5935
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: hudson
>            Reporter: Tom White
>            Assignee: Giridharan Kesavan
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5935.patch
>
>
> For example, on HADOOP-5170 the link http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/392/artifact/trunk/current/releaseAuditDiffWarnings.txt gives a 404. This makes it hard to work out which file or files are causing a problem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-386-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 10:16:35 2009
Return-Path: <common-issues-return-386-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60176 invoked from network); 17 Jul 2009 10:16:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 10:16:33 -0000
Received: (qmail 45463 invoked by uid 500); 17 Jul 2009 10:17:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45425 invoked by uid 500); 17 Jul 2009 10:17:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45414 invoked by uid 99); 17 Jul 2009 10:17:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 10:17:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 10:17:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CB582234C044
	for <common-issues@hadoop.apache.org>; Fri, 17 Jul 2009 03:17:14 -0700 (PDT)
Message-ID: <1621177109.1247825834828.JavaMail.jira@brutus>
Date: Fri, 17 Jul 2009 03:17:14 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6138) eliminate the depracate warnings
 introduced by H-5438
In-Reply-To: <883885751.1247164334967.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732447#action_12732447 ] 

Tom White commented on HADOOP-6138:
-----------------------------------

Looks like it was jdiff, which shouldn't really be checked by RAT. (Correct link is http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/575/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt, see also HADOOP-5935.) 

> eliminate the depracate warnings introduced by H-5438
> -----------------------------------------------------
>
>                 Key: HADOOP-6138
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6138
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: He Yongqiang
>            Assignee: He Yongqiang
>            Priority: Minor
>         Attachments: hadoop-6138-2009-07-15.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-387-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 10:20:33 2009
Return-Path: <common-issues-return-387-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62021 invoked from network); 17 Jul 2009 10:20:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 10:20:33 -0000
Received: (qmail 51764 invoked by uid 500); 17 Jul 2009 10:21:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51726 invoked by uid 500); 17 Jul 2009 10:21:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51716 invoked by uid 99); 17 Jul 2009 10:21:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 10:21:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 10:21:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D917B234C044
	for <common-issues@hadoop.apache.org>; Fri, 17 Jul 2009 03:21:14 -0700 (PDT)
Message-ID: <1831580547.1247826074884.JavaMail.jira@brutus>
Date: Fri, 17 Jul 2009 03:21:14 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6146) Upgrade to JetS3t version 0.7.1
In-Reply-To: <13490839.1247490076513.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6146:
------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I've just committed this.

> Upgrade to JetS3t version 0.7.1
> -------------------------------
>
>                 Key: HADOOP-6146
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6146
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6146.patch
>
>
> The JetS3t library is used for the S3 filesystems. We should upgrade to the latest version (0.7.1) which has support for Requester Pays buckets. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-388-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 10:38:34 2009
Return-Path: <common-issues-return-388-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68177 invoked from network); 17 Jul 2009 10:38:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 10:38:33 -0000
Received: (qmail 79892 invoked by uid 500); 17 Jul 2009 10:39:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79853 invoked by uid 500); 17 Jul 2009 10:39:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79828 invoked by uid 99); 17 Jul 2009 10:39:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 10:39:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 10:39:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id F0771234C044
	for <common-issues@hadoop.apache.org>; Fri, 17 Jul 2009 03:39:14 -0700 (PDT)
Message-ID: <1081086164.1247827154970.JavaMail.jira@brutus>
Date: Fri, 17 Jul 2009 03:39:14 -0700 (PDT)
From: "rahul k singh (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6105) Provide a way to automatically
 handle backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732453#action_12732453 ] 

rahul k singh commented on HADOOP-6105:
---------------------------------------

To make the above proposals more clear:

for example:
==========
A (a deprecated key) 
B(new key mapping for A) , 
SA (setValueForA) , 
SB(setValueForB) ,
GA(get("A)) 
GB(get("B")).

1. Always maintain new keys only. At the time of loading configuration xml , if A and B are present , we simply , replace B's values with A's value.
* Advantages with this approach:
 ** The whole system is consistent. we always maintain the single set of values , hence whole behavior is deterministic w.r.t SA or SB which ever is called latest.
* Disadvantage:
  ** Deprecated values would be over written. so if we call SB then A's values are also changed. //Is this a issue??

2.Always give preference to deprecated keys. If we have both A and B present in configuration , give preference to A always.
* Advantage:
** System is deterministic as we always get deprecated value if present.
* Disadvantage: 
** GB would check if GA is available and if yes , would return the A's values.So if user calls SB , and then calls GB , they would expect B's values, instead they would get A's value 

Any comments ?


> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-389-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 11:02:33 2009
Return-Path: <common-issues-return-389-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75192 invoked from network); 17 Jul 2009 11:02:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 11:02:33 -0000
Received: (qmail 15750 invoked by uid 500); 17 Jul 2009 11:03:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15712 invoked by uid 500); 17 Jul 2009 11:03:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15702 invoked by uid 99); 17 Jul 2009 11:03:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 11:03:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 11:03:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D377D234C045
	for <common-issues@hadoop.apache.org>; Fri, 17 Jul 2009 04:03:14 -0700 (PDT)
Message-ID: <1675092760.1247828594854.JavaMail.jira@brutus>
Date: Fri, 17 Jul 2009 04:03:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6120) Add support for Avro types in
 hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732459#action_12732459 ] 

Hadoop QA commented on HADOOP-6120:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12413792/6120_v5.patch
  against trunk revision 795028.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 15 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/579/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/579/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/579/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/579/console

This message is automatically generated.

> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch, 6120_v3.patch, 6120_v4.patch, 6120_v5.patch, HADOOP-6120.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-390-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 11:10:33 2009
Return-Path: <common-issues-return-390-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77194 invoked from network); 17 Jul 2009 11:10:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 11:10:33 -0000
Received: (qmail 25224 invoked by uid 500); 17 Jul 2009 11:11:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25186 invoked by uid 500); 17 Jul 2009 11:11:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25169 invoked by uid 99); 17 Jul 2009 11:11:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 11:11:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 11:11:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E3CB6234C1E7
	for <common-issues@hadoop.apache.org>; Fri, 17 Jul 2009 04:11:16 -0700 (PDT)
Message-ID: <939464689.1247829076932.JavaMail.jira@brutus>
Date: Fri, 17 Jul 2009 04:11:16 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6148) Implement a pure Java CRC32
 calculator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732465#action_12732465 ] 

Hudson commented on HADOOP-6148:
--------------------------------

Integrated in Hadoop-Common-trunk #29 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/29/])
    . Implement a fast, pure Java CRC32 calculator which outperforms java.util.zip.CRC32.  Contributed by Todd Lipcon and Scott Carey


> Implement a pure Java CRC32 calculator
> --------------------------------------
>
>                 Key: HADOOP-6148
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6148
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Owen O'Malley
>            Assignee: Scott Carey
>             Fix For: 0.21.0
>
>         Attachments: benchmarks20090714.txt, benchmarks20090715.txt, crc32-results.txt, hadoop-5598-evil.txt, hadoop-5598-hybrid.txt, hadoop-5598.txt, hadoop-5598.txt, hadoop-6148.txt, hadoop-6148.txt, hadoop-6148.txt, hdfs-297.txt, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32.java, PureJavaCrc32New.java, PureJavaCrc32NewInner.java, PureJavaCrc32NewLoop.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestCrc32Performance.java, TestPureJavaCrc32.java
>
>
> We've seen a reducer writing 200MB to HDFS with replication = 1 spending a long time in crc calculation. In particular, it was spending 5 seconds in crc calculation out of a total of 6 for the write. I suspect that it is the java-jni border that is causing us grief.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-391-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 11:10:34 2009
Return-Path: <common-issues-return-391-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77254 invoked from network); 17 Jul 2009 11:10:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 11:10:33 -0000
Received: (qmail 25340 invoked by uid 500); 17 Jul 2009 11:11:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25313 invoked by uid 500); 17 Jul 2009 11:11:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25290 invoked by uid 99); 17 Jul 2009 11:11:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 11:11:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 11:11:37 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 5445D234C4AA
	for <common-issues@hadoop.apache.org>; Fri, 17 Jul 2009 04:11:17 -0700 (PDT)
Message-ID: <383406953.1247829077344.JavaMail.jira@brutus>
Date: Fri, 17 Jul 2009 04:11:17 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6142) archives relative path changes in
 common.
In-Reply-To: <1194201800.1247266034972.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732467#action_12732467 ] 

Hudson commented on HADOOP-6142:
--------------------------------

Integrated in Hadoop-Common-trunk #29 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/29/])
    . Update documentation and use of harchives for relative paths added
in MAPREDUCE-739. Contributed by Mahadev Konar


> archives relative path changes in common.
> -----------------------------------------
>
>                 Key: HADOOP-6142
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6142
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Mahadev konar
>            Assignee: Mahadev konar
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6142.patch
>
>
> HADOOP-3663 was moved to mapreduce accidentally. Opening this jira to upload my chanegs to common --- changes are rleated to MAPREDUCE-739.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-392-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 11:10:35 2009
Return-Path: <common-issues-return-392-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77312 invoked from network); 17 Jul 2009 11:10:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 11:10:35 -0000
Received: (qmail 25421 invoked by uid 500); 17 Jul 2009 11:11:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25383 invoked by uid 500); 17 Jul 2009 11:11:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25373 invoked by uid 99); 17 Jul 2009 11:11:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 11:11:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 11:11:37 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C6DE0234C055
	for <common-issues@hadoop.apache.org>; Fri, 17 Jul 2009 04:11:16 -0700 (PDT)
Message-ID: <1988943968.1247829076813.JavaMail.jira@brutus>
Date: Fri, 17 Jul 2009 04:11:16 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6146) Upgrade to JetS3t version 0.7.1
In-Reply-To: <13490839.1247490076513.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732464#action_12732464 ] 

Hudson commented on HADOOP-6146:
--------------------------------

Integrated in Hadoop-Common-trunk #29 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/29/])
    . Upgrade to JetS3t version 0.7.1.


> Upgrade to JetS3t version 0.7.1
> -------------------------------
>
>                 Key: HADOOP-6146
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6146
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6146.patch
>
>
> The JetS3t library is used for the S3 filesystems. We should upgrade to the latest version (0.7.1) which has support for Requester Pays buckets. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-393-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 11:26:31 2009
Return-Path: <common-issues-return-393-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80651 invoked from network); 17 Jul 2009 11:26:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 11:26:31 -0000
Received: (qmail 42496 invoked by uid 500); 17 Jul 2009 11:27:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42459 invoked by uid 500); 17 Jul 2009 11:27:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42448 invoked by uid 99); 17 Jul 2009 11:27:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 11:27:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 11:27:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 18B74234C045
	for <common-issues@hadoop.apache.org>; Fri, 17 Jul 2009 04:27:15 -0700 (PDT)
Message-ID: <1162830772.1247830035100.JavaMail.jira@brutus>
Date: Fri, 17 Jul 2009 04:27:15 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6160) releaseaudit (rats) should not be run
 over the entire release binary
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

releaseaudit (rats) should not be run over the entire release binary
--------------------------------------------------------------------

                 Key: HADOOP-6160
                 URL: https://issues.apache.org/jira/browse/HADOOP-6160
             Project: Hadoop Common
          Issue Type: Improvement
          Components: build
            Reporter: Giridharan Kesavan




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-394-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 11:26:33 2009
Return-Path: <common-issues-return-394-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80686 invoked from network); 17 Jul 2009 11:26:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 11:26:32 -0000
Received: (qmail 42635 invoked by uid 500); 17 Jul 2009 11:27:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42537 invoked by uid 500); 17 Jul 2009 11:27:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42516 invoked by uid 99); 17 Jul 2009 11:27:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 11:27:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 11:27:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 2DAF1234C04C
	for <common-issues@hadoop.apache.org>; Fri, 17 Jul 2009 04:27:15 -0700 (PDT)
Message-ID: <2109787534.1247830035186.JavaMail.jira@brutus>
Date: Fri, 17 Jul 2009 04:27:15 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6160) releaseaudit (rats) should not be
 run over the entire release binary
In-Reply-To: <1162830772.1247830035100.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan reassigned HADOOP-6160:
------------------------------------------

    Assignee: Giridharan Kesavan

> releaseaudit (rats) should not be run over the entire release binary
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6160
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6160
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-395-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 11:26:34 2009
Return-Path: <common-issues-return-395-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80835 invoked from network); 17 Jul 2009 11:26:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 11:26:33 -0000
Received: (qmail 43481 invoked by uid 500); 17 Jul 2009 11:27:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43440 invoked by uid 500); 17 Jul 2009 11:27:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43430 invoked by uid 99); 17 Jul 2009 11:27:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 11:27:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 11:27:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 306AA234C052
	for <common-issues@hadoop.apache.org>; Fri, 17 Jul 2009 04:27:15 -0700 (PDT)
Message-ID: <1717017862.1247830035197.JavaMail.jira@brutus>
Date: Fri, 17 Jul 2009 04:27:15 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6160) releaseaudit (rats) should not be run
 againt the entire release binary
In-Reply-To: <1162830772.1247830035100.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6160:
---------------------------------------

    Summary: releaseaudit (rats) should not be run againt the entire release binary  (was: releaseaudit (rats) should not be run over the entire release binary)

> releaseaudit (rats) should not be run againt the entire release binary
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6160
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6160
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-396-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 15:37:34 2009
Return-Path: <common-issues-return-396-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7256 invoked from network); 17 Jul 2009 15:37:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 15:37:33 -0000
Received: (qmail 5767 invoked by uid 500); 17 Jul 2009 15:38:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5734 invoked by uid 500); 17 Jul 2009 15:38:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5724 invoked by uid 99); 17 Jul 2009 15:38:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 15:38:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 15:38:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C60C0234C045
	for <common-issues@hadoop.apache.org>; Fri, 17 Jul 2009 08:38:14 -0700 (PDT)
Message-ID: <1225103000.1247845094805.JavaMail.jira@brutus>
Date: Fri, 17 Jul 2009 08:38:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6160) releaseaudit (rats) should not be
 run againt the entire release binary
In-Reply-To: <1162830772.1247830035100.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732579#action_12732579 ] 

Giridharan Kesavan commented on HADOOP-6160:
--------------------------------------------

use the rat anttask to configure rats to run against the specific directory.

I'm thinking of excluding the following dirs/files for releaseaudit.

build/hadoop-core-0.21.0-dev/CHANGES.txt
build/hadoop-core-0.21.0-dev/docs
build/hadoop-core-0.21.0-dev/lib/jdiff





> releaseaudit (rats) should not be run againt the entire release binary
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6160
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6160
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-397-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 15:55:31 2009
Return-Path: <common-issues-return-397-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13404 invoked from network); 17 Jul 2009 15:55:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 15:55:31 -0000
Received: (qmail 25973 invoked by uid 500); 17 Jul 2009 15:56:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25934 invoked by uid 500); 17 Jul 2009 15:56:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25923 invoked by uid 99); 17 Jul 2009 15:56:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 15:56:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 15:56:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D67D4234C044
	for <common-issues@hadoop.apache.org>; Fri, 17 Jul 2009 08:56:14 -0700 (PDT)
Message-ID: <138493939.1247846174870.JavaMail.jira@brutus>
Date: Fri, 17 Jul 2009 08:56:14 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6134) New Hadoop Common Site
In-Reply-To: <1659559291.1247095875988.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-6134:
---------------------------------

    Resolution: Fixed
      Assignee: Corinne Chandel
        Status: Resolved  (was: Patch Available)

I just committed this.  Thanks, Corinne!

> New Hadoop Common Site
> ----------------------
>
>                 Key: HADOOP-6134
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6134
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>            Reporter: Corinne Chandel
>            Assignee: Corinne Chandel
>         Attachments: Hadoop-6134.patch
>
>
> New Hadoop Common Site
> Set up site, initial pass. 
> May need to add more content. 
> May need to update some links.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-398-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 17:40:32 2009
Return-Path: <common-issues-return-398-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56531 invoked from network); 17 Jul 2009 17:40:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 17:40:32 -0000
Received: (qmail 47049 invoked by uid 500); 17 Jul 2009 17:41:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47010 invoked by uid 500); 17 Jul 2009 17:41:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47000 invoked by uid 99); 17 Jul 2009 17:41:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 17:41:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 17:41:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C7717234C044
	for <common-issues@hadoop.apache.org>; Fri, 17 Jul 2009 10:41:14 -0700 (PDT)
Message-ID: <1429051737.1247852474802.JavaMail.jira@brutus>
Date: Fri, 17 Jul 2009 10:41:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6138) eliminate the depracate warnings
 introduced by H-5438
In-Reply-To: <883885751.1247164334967.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732626#action_12732626 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6138:
------------------------------------------------

Thanks Tom for the info.

No new tests needed because the changes is simple.  I will commit this soon.

> eliminate the depracate warnings introduced by H-5438
> -----------------------------------------------------
>
>                 Key: HADOOP-6138
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6138
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: He Yongqiang
>            Assignee: He Yongqiang
>            Priority: Minor
>         Attachments: hadoop-6138-2009-07-15.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-399-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 18:03:32 2009
Return-Path: <common-issues-return-399-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71777 invoked from network); 17 Jul 2009 18:03:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 18:03:32 -0000
Received: (qmail 87207 invoked by uid 500); 17 Jul 2009 18:04:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87161 invoked by uid 500); 17 Jul 2009 18:04:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86916 invoked by uid 99); 17 Jul 2009 18:04:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 18:04:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 18:04:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E0B35234C055
	for <common-issues@hadoop.apache.org>; Fri, 17 Jul 2009 11:04:14 -0700 (PDT)
Message-ID: <1109690120.1247853854919.JavaMail.jira@brutus>
Date: Fri, 17 Jul 2009 11:04:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6138) eliminate the depracate warnings
 introduced by H-5438
In-Reply-To: <883885751.1247164334967.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6138:
-------------------------------------------

          Component/s: fs
          Description: Eliminate the deprecated warnings introduced by HADOOP-5438.
    Affects Version/s: 0.21.0
        Fix Version/s: 0.21.0
           Issue Type: Bug  (was: Improvement)

> eliminate the depracate warnings introduced by H-5438
> -----------------------------------------------------
>
>                 Key: HADOOP-6138
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6138
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: He Yongqiang
>            Assignee: He Yongqiang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6138-2009-07-15.patch
>
>
> Eliminate the deprecated warnings introduced by HADOOP-5438.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-400-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 18:05:33 2009
Return-Path: <common-issues-return-400-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73123 invoked from network); 17 Jul 2009 18:05:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 18:05:33 -0000
Received: (qmail 90776 invoked by uid 500); 17 Jul 2009 18:06:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90749 invoked by uid 500); 17 Jul 2009 18:06:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90739 invoked by uid 99); 17 Jul 2009 18:06:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 18:06:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 18:06:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0555D234C051
	for <common-issues@hadoop.apache.org>; Fri, 17 Jul 2009 11:06:15 -0700 (PDT)
Message-ID: <160089176.1247853975020.JavaMail.jira@brutus>
Date: Fri, 17 Jul 2009 11:06:15 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6138) eliminate the depracate warnings
 introduced by H-5438
In-Reply-To: <883885751.1247164334967.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6138:
-------------------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I have committed this.  Thanks, Yongqiang!

> eliminate the depracate warnings introduced by H-5438
> -----------------------------------------------------
>
>                 Key: HADOOP-6138
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6138
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: He Yongqiang
>            Assignee: He Yongqiang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6138-2009-07-15.patch
>
>
> Eliminate the deprecated warnings introduced by HADOOP-5438.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-401-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 20:11:32 2009
Return-Path: <common-issues-return-401-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31239 invoked from network); 17 Jul 2009 20:11:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 20:11:31 -0000
Received: (qmail 71361 invoked by uid 500); 17 Jul 2009 20:12:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71323 invoked by uid 500); 17 Jul 2009 20:12:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71313 invoked by uid 99); 17 Jul 2009 20:12:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 20:12:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 20:12:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DBA6D234C045
	for <common-issues@hadoop.apache.org>; Fri, 17 Jul 2009 13:12:14 -0700 (PDT)
Message-ID: <582258206.1247861534898.JavaMail.jira@brutus>
Date: Fri, 17 Jul 2009 13:12:14 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6120) Add support for Avro types in
 hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732716#action_12732716 ] 

Doug Cutting commented on HADOOP-6120:
--------------------------------------

Sigh.  I just reallized this is really fragile, since we don't permit one to specify a different schema to read things.

We need to get metadata from the serializer used to write data to the deserializer used to deserialize it.  For even Writable, we should check that the data was indeed written with WritableSerializer, since folks can configure a given class to be serialized in different ways.

So we might add methods like:
 - Deserializer Serialization#getDeserializer(Map<String,String> meta);
 - Serializer Serialization#getSerializer(Map<String,String> meta);
 - Map<String,String> Serializer#getMeta();
And alter, e.g. SequenceFile and other containers to use these.

Perhaps this belongs in another Jira, but without it we're missing out on a lot of Avro's functionality.


> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch, 6120_v3.patch, 6120_v4.patch, 6120_v5.patch, HADOOP-6120.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-402-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 17 21:26:32 2009
Return-Path: <common-issues-return-402-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74710 invoked from network); 17 Jul 2009 21:26:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Jul 2009 21:26:32 -0000
Received: (qmail 83651 invoked by uid 500); 17 Jul 2009 21:27:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83590 invoked by uid 500); 17 Jul 2009 21:27:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83570 invoked by uid 99); 17 Jul 2009 21:27:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 21:27:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jul 2009 21:27:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C89F4234C044
	for <common-issues@hadoop.apache.org>; Fri, 17 Jul 2009 14:27:14 -0700 (PDT)
Message-ID: <1133905064.1247866034808.JavaMail.jira@brutus>
Date: Fri, 17 Jul 2009 14:27:14 -0700 (PDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6132) RPC client opens an extra
 connection for VersionedProtocol
In-Reply-To: <840919907.1247085075163.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732743#action_12732743 ] 

Kan Zhang commented on HADOOP-6132:
-----------------------------------

> There is another instance of use of getDelaringClass() in the same file

Yes, it is used in a public static convenience method whereby users can make a call by passing only the method. Unless we want to change the 
interface contract and require the user to pass the protocol as well, we have to use getDeclaringClass() to figure out the protocol. I'm not sure 
what the original purpose of this interface was. But I feel it serves some purpose (as long as users are aware of the twist of using 
getDeclaringClass()) and since it is only used in tests, I don't feel making the changes in this patch. What do you think?

> RPC client opens an extra connection for VersionedProtocol
> ----------------------------------------------------------
>
>                 Key: HADOOP-6132
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6132
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: 1.patch, 2.patch
>
>
> RPC client caches connections per protocol. However, since all of our real protocols are subclasses of VersionedProtocol, a bug in the implementation makes the client opens an extra connection just for the VersionedProtocol, which is not needed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-403-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 18 01:49:31 2009
Return-Path: <common-issues-return-403-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66548 invoked from network); 18 Jul 2009 01:49:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jul 2009 01:49:31 -0000
Received: (qmail 69006 invoked by uid 500); 18 Jul 2009 01:50:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68968 invoked by uid 500); 18 Jul 2009 01:50:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68958 invoked by uid 99); 18 Jul 2009 01:50:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jul 2009 01:50:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jul 2009 01:50:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CF890234C044
	for <common-issues@hadoop.apache.org>; Fri, 17 Jul 2009 18:50:14 -0700 (PDT)
Message-ID: <236654617.1247881814836.JavaMail.jira@brutus>
Date: Fri, 17 Jul 2009 18:50:14 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5879) GzipCodec should read compression
 level etc from configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-5879:
----------------------------------

    Assignee: He Yongqiang
      Status: Open  (was: Patch Available)

The latest patch looks good. A few notes:
* BuiltInZlibDeflater can use the setLevel and setStrategy methods on java.util.zip.Deflater to implement reset
* This should definitely have a unit test, particularly involving CodecPool
* The javadoc for reinit should describe the contract for that method; the implementations should also describe what happens for that particular codec.
* In ZlibCompression::reinit, shouldn't end() be called before reconfiguring the codec?
* The direct buffer size should not be reconfigured. Codecs are pooled because recreating direct buffers is very expensive; allocating new direct buffers on a cache hit defeats the purpose of pooling them. The buffer size doesn't affect correctness of zlib compression.
* This should add methods like ZlibCompression::setCompressionLevel(Configuration, <value>) to set the {{zlib.compress.*}} properties

> GzipCodec should read compression level etc from configuration
> --------------------------------------------------------------
>
>                 Key: HADOOP-5879
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5879
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Zheng Shao
>            Assignee: He Yongqiang
>         Attachments: hadoop-5879-5-21.patch, hadoop-5879-7-13-2.patch, hadoop-5879-7-13-3.patch
>
>
> GzipCodec currently uses the default compression level. We should allow overriding the default value from Configuration.
> {code}
>   static final class GzipZlibCompressor extends ZlibCompressor {
>     public GzipZlibCompressor() {
>       super(ZlibCompressor.CompressionLevel.DEFAULT_COMPRESSION,
>           ZlibCompressor.CompressionStrategy.DEFAULT_STRATEGY,
>           ZlibCompressor.CompressionHeader.GZIP_FORMAT, 64*1024);
>     }
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-404-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 18 03:47:32 2009
Return-Path: <common-issues-return-404-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2738 invoked from network); 18 Jul 2009 03:47:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jul 2009 03:47:31 -0000
Received: (qmail 9422 invoked by uid 500); 18 Jul 2009 03:48:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9382 invoked by uid 500); 18 Jul 2009 03:48:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9367 invoked by uid 99); 18 Jul 2009 03:48:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jul 2009 03:48:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jul 2009 03:48:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C8219234C044
	for <common-issues@hadoop.apache.org>; Fri, 17 Jul 2009 20:48:14 -0700 (PDT)
Message-ID: <1767828979.1247888894812.JavaMail.jira@brutus>
Date: Fri, 17 Jul 2009 20:48:14 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5879) GzipCodec should read compression
 level etc from configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732833#action_12732833 ] 

Chris Douglas commented on HADOOP-5879:
---------------------------------------

Whoops; ZlibCompression isn't a class. Maybe setCompressionLevel(conf, value) should be added to GzipCodec instead.

> GzipCodec should read compression level etc from configuration
> --------------------------------------------------------------
>
>                 Key: HADOOP-5879
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5879
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Zheng Shao
>            Assignee: He Yongqiang
>         Attachments: hadoop-5879-5-21.patch, hadoop-5879-7-13-2.patch, hadoop-5879-7-13-3.patch
>
>
> GzipCodec currently uses the default compression level. We should allow overriding the default value from Configuration.
> {code}
>   static final class GzipZlibCompressor extends ZlibCompressor {
>     public GzipZlibCompressor() {
>       super(ZlibCompressor.CompressionLevel.DEFAULT_COMPRESSION,
>           ZlibCompressor.CompressionStrategy.DEFAULT_STRATEGY,
>           ZlibCompressor.CompressionHeader.GZIP_FORMAT, 64*1024);
>     }
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-405-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 18 11:15:32 2009
Return-Path: <common-issues-return-405-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22103 invoked from network); 18 Jul 2009 11:15:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jul 2009 11:15:31 -0000
Received: (qmail 55662 invoked by uid 500); 18 Jul 2009 11:16:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55617 invoked by uid 500); 18 Jul 2009 11:16:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55607 invoked by uid 99); 18 Jul 2009 11:16:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jul 2009 11:16:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jul 2009 11:16:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E116F234C044
	for <common-issues@hadoop.apache.org>; Sat, 18 Jul 2009 04:16:14 -0700 (PDT)
Message-ID: <1897231587.1247915774907.JavaMail.jira@brutus>
Date: Sat, 18 Jul 2009 04:16:14 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6138) eliminate the depracate warnings
 introduced by H-5438
In-Reply-To: <883885751.1247164334967.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732879#action_12732879 ] 

Hudson commented on HADOOP-6138:
--------------------------------

Integrated in Hadoop-Common-trunk #30 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/30/])
    . Eliminate the depracate warnings introduced by H-5438.  Contributed by He Yongqiang


> eliminate the depracate warnings introduced by H-5438
> -----------------------------------------------------
>
>                 Key: HADOOP-6138
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6138
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: He Yongqiang
>            Assignee: He Yongqiang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6138-2009-07-15.patch
>
>
> Eliminate the deprecated warnings introduced by HADOOP-5438.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-406-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 18 14:35:35 2009
Return-Path: <common-issues-return-406-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85965 invoked from network); 18 Jul 2009 14:35:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jul 2009 14:35:35 -0000
Received: (qmail 47611 invoked by uid 500); 18 Jul 2009 14:36:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47571 invoked by uid 500); 18 Jul 2009 14:36:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47556 invoked by uid 99); 18 Jul 2009 14:36:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jul 2009 14:36:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jul 2009 14:36:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C88DD234C044
	for <common-issues@hadoop.apache.org>; Sat, 18 Jul 2009 07:36:14 -0700 (PDT)
Message-ID: <795653157.1247927774816.JavaMail.jira@brutus>
Date: Sat, 18 Jul 2009 07:36:14 -0700 (PDT)
From: "He Yongqiang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5879) GzipCodec should read compression
 level etc from configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

He Yongqiang updated HADOOP-5879:
---------------------------------

    Attachment: hadoop-5879-7-18-3.patch

A new patch incorporates Chris' comments (Thanks, Chirs!).

> GzipCodec should read compression level etc from configuration
> --------------------------------------------------------------
>
>                 Key: HADOOP-5879
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5879
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Zheng Shao
>            Assignee: He Yongqiang
>         Attachments: hadoop-5879-5-21.patch, hadoop-5879-7-13-2.patch, hadoop-5879-7-13-3.patch, hadoop-5879-7-18-3.patch
>
>
> GzipCodec currently uses the default compression level. We should allow overriding the default value from Configuration.
> {code}
>   static final class GzipZlibCompressor extends ZlibCompressor {
>     public GzipZlibCompressor() {
>       super(ZlibCompressor.CompressionLevel.DEFAULT_COMPRESSION,
>           ZlibCompressor.CompressionStrategy.DEFAULT_STRATEGY,
>           ZlibCompressor.CompressionHeader.GZIP_FORMAT, 64*1024);
>     }
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-407-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 18 23:03:31 2009
Return-Path: <common-issues-return-407-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82724 invoked from network); 18 Jul 2009 23:03:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jul 2009 23:03:31 -0000
Received: (qmail 74594 invoked by uid 500); 18 Jul 2009 23:04:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74554 invoked by uid 500); 18 Jul 2009 23:04:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74536 invoked by uid 99); 18 Jul 2009 23:04:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jul 2009 23:04:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jul 2009 23:04:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D6609234C045
	for <common-issues@hadoop.apache.org>; Sat, 18 Jul 2009 16:04:14 -0700 (PDT)
Message-ID: <666455284.1247958254877.JavaMail.jira@brutus>
Date: Sat, 18 Jul 2009 16:04:14 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6161) Add get/setEnum to Configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Add get/setEnum to Configuration
--------------------------------

                 Key: HADOOP-6161
                 URL: https://issues.apache.org/jira/browse/HADOOP-6161
             Project: Hadoop Common
          Issue Type: Improvement
          Components: conf
            Reporter: Chris Douglas
            Priority: Minor


It would be useful if Configuration had helper get/set methods for enumerated types.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-408-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 18 23:05:33 2009
Return-Path: <common-issues-return-408-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83201 invoked from network); 18 Jul 2009 23:05:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jul 2009 23:05:31 -0000
Received: (qmail 76635 invoked by uid 500); 18 Jul 2009 23:06:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76592 invoked by uid 500); 18 Jul 2009 23:06:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76582 invoked by uid 99); 18 Jul 2009 23:06:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jul 2009 23:06:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jul 2009 23:06:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0C7D1234C044
	for <common-issues@hadoop.apache.org>; Sat, 18 Jul 2009 16:06:15 -0700 (PDT)
Message-ID: <763325377.1247958375036.JavaMail.jira@brutus>
Date: Sat, 18 Jul 2009 16:06:15 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6161) Add get/setEnum to Configuration
In-Reply-To: <666455284.1247958254877.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-6161:
----------------------------------

    Attachment: C6161-0.patch

> Add get/setEnum to Configuration
> --------------------------------
>
>                 Key: HADOOP-6161
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6161
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Chris Douglas
>            Priority: Minor
>         Attachments: C6161-0.patch
>
>
> It would be useful if Configuration had helper get/set methods for enumerated types.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-409-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 18 23:05:34 2009
Return-Path: <common-issues-return-409-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83218 invoked from network); 18 Jul 2009 23:05:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jul 2009 23:05:32 -0000
Received: (qmail 76711 invoked by uid 500); 18 Jul 2009 23:06:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76669 invoked by uid 500); 18 Jul 2009 23:06:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76655 invoked by uid 99); 18 Jul 2009 23:06:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jul 2009 23:06:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jul 2009 23:06:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 12A1F234C046
	for <common-issues@hadoop.apache.org>; Sat, 18 Jul 2009 16:06:15 -0700 (PDT)
Message-ID: <1850200318.1247958375075.JavaMail.jira@brutus>
Date: Sat, 18 Jul 2009 16:06:15 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6161) Add get/setEnum to Configuration
In-Reply-To: <666455284.1247958254877.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-6161:
----------------------------------

    Assignee: Chris Douglas
      Status: Patch Available  (was: Open)

> Add get/setEnum to Configuration
> --------------------------------
>
>                 Key: HADOOP-6161
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6161
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Chris Douglas
>            Assignee: Chris Douglas
>            Priority: Minor
>         Attachments: C6161-0.patch
>
>
> It would be useful if Configuration had helper get/set methods for enumerated types.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-410-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 18 23:25:31 2009
Return-Path: <common-issues-return-410-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85987 invoked from network); 18 Jul 2009 23:25:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jul 2009 23:25:31 -0000
Received: (qmail 81622 invoked by uid 500); 18 Jul 2009 23:26:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81584 invoked by uid 500); 18 Jul 2009 23:26:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81574 invoked by uid 99); 18 Jul 2009 23:26:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jul 2009 23:26:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jul 2009 23:26:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C58D4234C044
	for <common-issues@hadoop.apache.org>; Sat, 18 Jul 2009 16:26:14 -0700 (PDT)
Message-ID: <43626980.1247959574804.JavaMail.jira@brutus>
Date: Sat, 18 Jul 2009 16:26:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6161) Add get/setEnum to Configuration
In-Reply-To: <666455284.1247958254877.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732942#action_12732942 ] 

Hadoop QA commented on HADOOP-6161:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12413934/C6161-0.patch
  against trunk revision 795172.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/580/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/580/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/580/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/580/console

This message is automatically generated.

> Add get/setEnum to Configuration
> --------------------------------
>
>                 Key: HADOOP-6161
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6161
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Chris Douglas
>            Assignee: Chris Douglas
>            Priority: Minor
>         Attachments: C6161-0.patch
>
>
> It would be useful if Configuration had helper get/set methods for enumerated types.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-411-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 18 23:39:33 2009
Return-Path: <common-issues-return-411-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87550 invoked from network); 18 Jul 2009 23:39:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jul 2009 23:39:33 -0000
Received: (qmail 84333 invoked by uid 500); 18 Jul 2009 23:40:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84293 invoked by uid 500); 18 Jul 2009 23:40:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84283 invoked by uid 99); 18 Jul 2009 23:40:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jul 2009 23:40:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jul 2009 23:40:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CC72C234C044
	for <common-issues@hadoop.apache.org>; Sat, 18 Jul 2009 16:40:14 -0700 (PDT)
Message-ID: <1464038677.1247960414823.JavaMail.jira@brutus>
Date: Sat, 18 Jul 2009 16:40:14 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5879) GzipCodec should read compression
 level etc from configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732943#action_12732943 ] 

Chris Douglas commented on HADOOP-5879:
---------------------------------------

Thanks for updating the patch
* Rather than having special semantics for construct, I'd suggest removing the directBufferSize formal from construct and returning the allocation to the constructor
* Sorry, my last comment was not clear. By "implementations should also describe..." I meant that the classes implementing reinit should include a description of what it effects in its javadoc for reinit, as you did in BuiltInZlibDeflater. I didn't mean that more logging was required. Compressor::reinit should also describe the contract for future implementers, "Prepare the compressor to be used in a new stream with settings defined in the given Configuration" or something like that
* I think it's appropriate to fail if the configuration is invalid rather than taking the default in ZlibFactory::getCompression\* (why ZlibFactory?). I don't think the getDefault\* methods are necessary. setCompression\* should take the appropriate enum, rather than String. Filed HADOOP-6161 for get/setEnum to simplify some of the conf-related code
* In ZlibCompressor::reinit, reset is not called if the Configuration is null. For users calling these methods not via CodecPool, reset() should probably be called
* CompressionLevel::compressionLevel() and CompressionStrategy::compressionStrategy() should remain package-private; the integers are implementation details
* The unit test doesn't really verify the functionality added by this patch, save that it doesn't throw exceptions. That said, it would be hard to verify that this is working as expected without adding get\* methods to ZlibCompressor. Can you describe how you verified the patch's correctness, both for the native and non-native codecs?

> GzipCodec should read compression level etc from configuration
> --------------------------------------------------------------
>
>                 Key: HADOOP-5879
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5879
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Zheng Shao
>            Assignee: He Yongqiang
>         Attachments: hadoop-5879-5-21.patch, hadoop-5879-7-13-2.patch, hadoop-5879-7-13-3.patch, hadoop-5879-7-18-3.patch
>
>
> GzipCodec currently uses the default compression level. We should allow overriding the default value from Configuration.
> {code}
>   static final class GzipZlibCompressor extends ZlibCompressor {
>     public GzipZlibCompressor() {
>       super(ZlibCompressor.CompressionLevel.DEFAULT_COMPRESSION,
>           ZlibCompressor.CompressionStrategy.DEFAULT_STRATEGY,
>           ZlibCompressor.CompressionHeader.GZIP_FORMAT, 64*1024);
>     }
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-412-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 18 23:57:37 2009
Return-Path: <common-issues-return-412-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91281 invoked from network); 18 Jul 2009 23:57:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jul 2009 23:57:37 -0000
Received: (qmail 95541 invoked by uid 500); 18 Jul 2009 23:58:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95502 invoked by uid 500); 18 Jul 2009 23:58:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95492 invoked by uid 99); 18 Jul 2009 23:58:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jul 2009 23:58:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jul 2009 23:58:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D9CDB234C044
	for <common-issues@hadoop.apache.org>; Sat, 18 Jul 2009 16:58:14 -0700 (PDT)
Message-ID: <917138917.1247961494880.JavaMail.jira@brutus>
Date: Sat, 18 Jul 2009 16:58:14 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5732) SFTP FileSystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5732?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-5732:
----------------------------------

    Status: Open  (was: Patch Available)

> SFTP FileSystem
> ---------------
>
>                 Key: HADOOP-5732
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5732
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.20.0
>         Environment: Any environment
>            Reporter: =C3=8D=C3=B1igo Goiri
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5732.patch, HADOOP-5732.patch, HADOOP-5732.pa=
tch, HADOOP-5732.patch, ivy-for-hadoop-7532.patch, ivy-for-hadoop-7532.patc=
h
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> I have implemented a FileSystem that supports SFTP. It uses JSch (http://=
www.jcraft.com/jsch/) in order to manage SFTP.

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-413-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 18 23:57:37 2009
Return-Path: <common-issues-return-413-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91289 invoked from network); 18 Jul 2009 23:57:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jul 2009 23:57:37 -0000
Received: (qmail 95607 invoked by uid 500); 18 Jul 2009 23:58:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95569 invoked by uid 500); 18 Jul 2009 23:58:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95548 invoked by uid 99); 18 Jul 2009 23:58:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jul 2009 23:58:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jul 2009 23:58:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E88BC234C056
	for <common-issues@hadoop.apache.org>; Sat, 18 Jul 2009 16:58:14 -0700 (PDT)
Message-ID: <353381511.1247961494951.JavaMail.jira@brutus>
Date: Sat, 18 Jul 2009 16:58:14 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5732) SFTP FileSystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5732?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-5732:
----------------------------------

    Status: Patch Available  (was: Open)

Running patch through hudson

> SFTP FileSystem
> ---------------
>
>                 Key: HADOOP-5732
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5732
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.20.0
>         Environment: Any environment
>            Reporter: =C3=8D=C3=B1igo Goiri
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5732.patch, HADOOP-5732.patch, HADOOP-5732.pa=
tch, HADOOP-5732.patch, ivy-for-hadoop-7532.patch, ivy-for-hadoop-7532.patc=
h
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> I have implemented a FileSystem that supports SFTP. It uses JSch (http://=
www.jcraft.com/jsch/) in order to manage SFTP.

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-414-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 18 23:57:38 2009
Return-Path: <common-issues-return-414-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91357 invoked from network); 18 Jul 2009 23:57:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Jul 2009 23:57:38 -0000
Received: (qmail 95674 invoked by uid 500); 18 Jul 2009 23:58:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95636 invoked by uid 500); 18 Jul 2009 23:58:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95626 invoked by uid 99); 18 Jul 2009 23:58:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jul 2009 23:58:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jul 2009 23:58:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id F0AE2234C1EC
	for <common-issues@hadoop.apache.org>; Sat, 18 Jul 2009 16:58:14 -0700 (PDT)
Message-ID: <1276682894.1247961494985.JavaMail.jira@brutus>
Date: Sat, 18 Jul 2009 16:58:14 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-5732) SFTP FileSystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5732?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas reassigned HADOOP-5732:
-------------------------------------

    Assignee: =C3=8D=C3=B1igo Goiri

> SFTP FileSystem
> ---------------
>
>                 Key: HADOOP-5732
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5732
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.20.0
>         Environment: Any environment
>            Reporter: =C3=8D=C3=B1igo Goiri
>            Assignee: =C3=8D=C3=B1igo Goiri
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5732.patch, HADOOP-5732.patch, HADOOP-5732.pa=
tch, HADOOP-5732.patch, ivy-for-hadoop-7532.patch, ivy-for-hadoop-7532.patc=
h
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> I have implemented a FileSystem that supports SFTP. It uses JSch (http://=
www.jcraft.com/jsch/) in order to manage SFTP.

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-415-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jul 19 00:07:39 2009
Return-Path: <common-issues-return-415-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94499 invoked from network); 19 Jul 2009 00:07:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jul 2009 00:07:39 -0000
Received: (qmail 222 invoked by uid 500); 19 Jul 2009 00:08:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 179 invoked by uid 500); 19 Jul 2009 00:08:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 169 invoked by uid 99); 19 Jul 2009 00:08:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jul 2009 00:08:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jul 2009 00:08:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C7BB4234C044
	for <common-issues@hadoop.apache.org>; Sat, 18 Jul 2009 17:08:14 -0700 (PDT)
Message-ID: <1704850204.1247962094804.JavaMail.jira@brutus>
Date: Sat, 18 Jul 2009 17:08:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5732) SFTP FileSystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5732?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D127=
32947#action_12732947 ]=20

Hadoop QA commented on HADOOP-5732:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment=20
  http://issues.apache.org/jira/secure/attachment/12411419/HADOOP-5732.patc=
h
  against trunk revision 795172.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 11 new or modified tes=
ts.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of java=
c compiler warnings.

    -1 findbugs.  The patch appears to cause Findbugs to fail.

    +1 release audit.  The applied patch does not increase the total number=
 of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    -1 contrib tests.  The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.=
apache.org/581/testReport/
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-=
vesta.apache.org/581/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vest=
a.apache.org/581/console

This message is automatically generated.

> SFTP FileSystem
> ---------------
>
>                 Key: HADOOP-5732
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5732
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.20.0
>         Environment: Any environment
>            Reporter: =C3=8D=C3=B1igo Goiri
>            Assignee: =C3=8D=C3=B1igo Goiri
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5732.patch, HADOOP-5732.patch, HADOOP-5732.pa=
tch, HADOOP-5732.patch, ivy-for-hadoop-7532.patch, ivy-for-hadoop-7532.patc=
h
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> I have implemented a FileSystem that supports SFTP. It uses JSch (http://=
www.jcraft.com/jsch/) in order to manage SFTP.

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-416-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jul 19 04:53:34 2009
Return-Path: <common-issues-return-416-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67585 invoked from network); 19 Jul 2009 04:53:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jul 2009 04:53:34 -0000
Received: (qmail 39400 invoked by uid 500); 19 Jul 2009 04:54:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39362 invoked by uid 500); 19 Jul 2009 04:54:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39352 invoked by uid 99); 19 Jul 2009 04:54:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jul 2009 04:54:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jul 2009 04:54:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E9EA5234C044
	for <common-issues@hadoop.apache.org>; Sat, 18 Jul 2009 21:54:14 -0700 (PDT)
Message-ID: <1124983580.1247979254943.JavaMail.jira@brutus>
Date: Sat, 18 Jul 2009 21:54:14 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6143) FS shell commands returns incorrect
 exit code  when error occurs
In-Reply-To: <636161787.1247269154845.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-6143:
----------------------------------

    Status: Open  (was: Patch Available)

{noformat}
-		   <code> Returns 0 on success and -1 on error. </code></p>
+		   <code> Returns 0 on success , and >0 if an error occurs. </code></p>
{noformat}

Rather than test- or care- whether other platforms return -1, I would prefer "nonzero on error" or more symmetrically, "nonzero on failure".

> FS shell commands returns incorrect exit code  when error occurs
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6143
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6143
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>         Attachments: HADOOP-6143.patch
>
>
> HDFS documentation ( http://hadoop.apache.org/core/docs/current/hdfs_shell.html#du )  mentions that 
> {noformat}
> Exit Code:
> Returns 0 on success and -1 on error. 
> {noformat} 
> Current Fs shell behavior is buggy with this agreement.
> {code}
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -ls foo
> ls: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -lsr foo
> lsr: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -du foo
> du: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -dus foo
> dus: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -cp foo f2
> cp: File does not exist: foo
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -copyToLocal foo f2
> copyToLocal: null
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -copyFromLocal foo f2
> copyFromLocal: File foo does not exist.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> {code}  
> In all above cases exit code on error should be -1 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-417-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jul 19 05:23:33 2009
Return-Path: <common-issues-return-417-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73698 invoked from network); 19 Jul 2009 05:23:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jul 2009 05:23:33 -0000
Received: (qmail 47137 invoked by uid 500); 19 Jul 2009 05:24:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47046 invoked by uid 500); 19 Jul 2009 05:24:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47027 invoked by uid 99); 19 Jul 2009 05:24:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jul 2009 05:24:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jul 2009 05:24:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0142C234C055
	for <common-issues@hadoop.apache.org>; Sat, 18 Jul 2009 22:24:15 -0700 (PDT)
Message-ID: <1618049917.1247981055004.JavaMail.jira@brutus>
Date: Sat, 18 Jul 2009 22:24:15 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Moved: (HADOOP-6162) MapFile doesn't work with serializables
 other than Writables
In-Reply-To: <1542829149.1246995374880.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas moved MAPREDUCE-738 to HADOOP-6162:
-------------------------------------------------

        Key: HADOOP-6162  (was: MAPREDUCE-738)
    Project: Hadoop Common  (was: Hadoop Map/Reduce)

> MapFile doesn't work with serializables other than Writables
> ------------------------------------------------------------
>
>                 Key: HADOOP-6162
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6162
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Justin Patterson
>         Attachments: HADOOP-6129.patch
>
>
> Since 0.18 (I think), SequenceFiles have supported serializing arbitrary objects through the serialization framework.  MapFiles still don't.  They require WritableComparable keys and Writable values.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-418-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jul 19 05:25:33 2009
Return-Path: <common-issues-return-418-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74488 invoked from network); 19 Jul 2009 05:25:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jul 2009 05:25:33 -0000
Received: (qmail 47465 invoked by uid 500); 19 Jul 2009 05:26:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47428 invoked by uid 500); 19 Jul 2009 05:26:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47418 invoked by uid 99); 19 Jul 2009 05:26:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jul 2009 05:26:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jul 2009 05:26:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C4253234C044
	for <common-issues@hadoop.apache.org>; Sat, 18 Jul 2009 22:26:14 -0700 (PDT)
Message-ID: <706365959.1247981174789.JavaMail.jira@brutus>
Date: Sat, 18 Jul 2009 22:26:14 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6162) MapFile doesn't work with
 serializables other than Writables
In-Reply-To: <1542829149.1246995374880.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732972#action_12732972 ] 

Chris Douglas commented on HADOOP-6162:
---------------------------------------

On second thought, this issue should be in common; the MapFileOutputFormat changes can be moved to a MR issue.

> MapFile doesn't work with serializables other than Writables
> ------------------------------------------------------------
>
>                 Key: HADOOP-6162
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6162
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Justin Patterson
>         Attachments: HADOOP-6129.patch
>
>
> Since 0.18 (I think), SequenceFiles have supported serializing arbitrary objects through the serialization framework.  MapFiles still don't.  They require WritableComparable keys and Writable values.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-419-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jul 19 21:13:34 2009
Return-Path: <common-issues-return-419-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47616 invoked from network); 19 Jul 2009 21:13:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jul 2009 21:13:34 -0000
Received: (qmail 33254 invoked by uid 500); 19 Jul 2009 21:14:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33210 invoked by uid 500); 19 Jul 2009 21:14:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33200 invoked by uid 99); 19 Jul 2009 21:14:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jul 2009 21:14:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jul 2009 21:14:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 190A0234C044
	for <common-issues@hadoop.apache.org>; Sun, 19 Jul 2009 14:14:15 -0700 (PDT)
Message-ID: <940132753.1248038055097.JavaMail.jira@brutus>
Date: Sun, 19 Jul 2009 14:14:15 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4010) Chaging LineRecordReader algo so that
 it does not need to skip backwards in the stream
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-4010:
----------------------------------

    Attachment: 4010-mapreduce.patch

The changes to mapred.LineRecordReader should have been propagated to mapreduce.lib.input.LineRecordReader

> Chaging LineRecordReader algo so that it does not need to skip backwards in the stream
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4010
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4010
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.19.0
>            Reporter: Abdul Qadeer
>            Assignee: Abdul Qadeer
>             Fix For: 0.21.0
>
>         Attachments: 4010-mapreduce.patch, Hadoop-4010.patch, Hadoop-4010_version2.patch, Hadoop-4010_version3.patch
>
>
> The current algorithm of the LineRecordReader needs to move backwards in the stream (in its constructor) to correctly position itself in the stream.  So it moves back one byte from the start of its split and try to read a record (i.e. a line) and throws that away.  This is so because it is sure that, this line would be taken care of by some other mapper.  This algorithm is difficult and in-efficient if used for compressed stream where data is coming to the LineRecordReader via some codecs. (Although in the current implementation, Hadoop does not split a compressed file and only makes one split from the start to the end of the file and so only one mapper handles it.  We are currently working on BZip2 codecs where splitting is possible to work with Hadoop.  So this proposed change will make it possible to uniformly handle plain as well as compressed stream.)
> In the new algorithm, each mapper always skips its first line because it is sure that, that line would have been read by some other mapper.  So now each mapper must finish its reading at a record boundary which is always beyond its upper split limit.  Due to this change, LineRecordReader does not need to move backwards in the stream.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-420-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jul 19 21:15:32 2009
Return-Path: <common-issues-return-420-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48031 invoked from network); 19 Jul 2009 21:15:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jul 2009 21:15:32 -0000
Received: (qmail 33892 invoked by uid 500); 19 Jul 2009 21:16:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33850 invoked by uid 500); 19 Jul 2009 21:16:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33830 invoked by uid 99); 19 Jul 2009 21:16:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jul 2009 21:16:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jul 2009 21:16:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 2B8AD29A0014
	for <common-issues@hadoop.apache.org>; Sun, 19 Jul 2009 14:16:15 -0700 (PDT)
Message-ID: <1313832882.1248038175177.JavaMail.jira@brutus>
Date: Sun, 19 Jul 2009 14:16:15 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Reopened: (HADOOP-4010) Chaging LineRecordReader algo so
 that it does not need to skip backwards in the stream
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas reopened HADOOP-4010:
-----------------------------------


> Chaging LineRecordReader algo so that it does not need to skip backwards in the stream
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4010
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4010
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.19.0
>            Reporter: Abdul Qadeer
>            Assignee: Abdul Qadeer
>             Fix For: 0.21.0
>
>         Attachments: 4010-mapreduce.patch, Hadoop-4010.patch, Hadoop-4010_version2.patch, Hadoop-4010_version3.patch
>
>
> The current algorithm of the LineRecordReader needs to move backwards in the stream (in its constructor) to correctly position itself in the stream.  So it moves back one byte from the start of its split and try to read a record (i.e. a line) and throws that away.  This is so because it is sure that, this line would be taken care of by some other mapper.  This algorithm is difficult and in-efficient if used for compressed stream where data is coming to the LineRecordReader via some codecs. (Although in the current implementation, Hadoop does not split a compressed file and only makes one split from the start to the end of the file and so only one mapper handles it.  We are currently working on BZip2 codecs where splitting is possible to work with Hadoop.  So this proposed change will make it possible to uniformly handle plain as well as compressed stream.)
> In the new algorithm, each mapper always skips its first line because it is sure that, that line would have been read by some other mapper.  So now each mapper must finish its reading at a record boundary which is always beyond its upper split limit.  Due to this change, LineRecordReader does not need to move backwards in the stream.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-421-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jul 19 21:47:31 2009
Return-Path: <common-issues-return-421-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61777 invoked from network); 19 Jul 2009 21:47:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jul 2009 21:47:31 -0000
Received: (qmail 50343 invoked by uid 500); 19 Jul 2009 21:48:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50304 invoked by uid 500); 19 Jul 2009 21:48:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50289 invoked by uid 99); 19 Jul 2009 21:48:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jul 2009 21:48:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jul 2009 21:48:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 06F5129A001D
	for <common-issues@hadoop.apache.org>; Sun, 19 Jul 2009 14:48:15 -0700 (PDT)
Message-ID: <550759029.1248040095027.JavaMail.jira@brutus>
Date: Sun, 19 Jul 2009 14:48:15 -0700 (PDT)
From: "Justin Patterson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6162) MapFile doesn't work with
 serializables other than Writables
In-Reply-To: <1542829149.1246995374880.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Justin Patterson updated HADOOP-6162:
-------------------------------------

    Attachment:     (was: HADOOP-6129.patch)

> MapFile doesn't work with serializables other than Writables
> ------------------------------------------------------------
>
>                 Key: HADOOP-6162
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6162
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Justin Patterson
>         Attachments: HADOOP-6162.patch
>
>
> Since 0.18 (I think), SequenceFiles have supported serializing arbitrary objects through the serialization framework.  MapFiles still don't.  They require WritableComparable keys and Writable values.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-422-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jul 19 21:47:33 2009
Return-Path: <common-issues-return-422-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61859 invoked from network); 19 Jul 2009 21:47:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Jul 2009 21:47:33 -0000
Received: (qmail 50531 invoked by uid 500); 19 Jul 2009 21:48:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50492 invoked by uid 500); 19 Jul 2009 21:48:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50477 invoked by uid 99); 19 Jul 2009 21:48:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jul 2009 21:48:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jul 2009 21:48:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D6B0D234C044
	for <common-issues@hadoop.apache.org>; Sun, 19 Jul 2009 14:48:14 -0700 (PDT)
Message-ID: <1227594242.1248040094864.JavaMail.jira@brutus>
Date: Sun, 19 Jul 2009 14:48:14 -0700 (PDT)
From: "Justin Patterson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6162) MapFile doesn't work with
 serializables other than Writables
In-Reply-To: <1542829149.1246995374880.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Justin Patterson updated HADOOP-6162:
-------------------------------------

    Attachment: HADOOP-6162.patch

Here's another patch that works with the new (divided) file structure and includes a unit test to prove that it works.

> MapFile doesn't work with serializables other than Writables
> ------------------------------------------------------------
>
>                 Key: HADOOP-6162
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6162
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Justin Patterson
>         Attachments: HADOOP-6162.patch
>
>
> Since 0.18 (I think), SequenceFiles have supported serializing arbitrary objects through the serialization framework.  MapFiles still don't.  They require WritableComparable keys and Writable values.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-423-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 20 05:03:33 2009
Return-Path: <common-issues-return-423-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25864 invoked from network); 20 Jul 2009 05:03:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jul 2009 05:03:33 -0000
Received: (qmail 13436 invoked by uid 500); 20 Jul 2009 05:04:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13412 invoked by uid 500); 20 Jul 2009 05:04:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13402 invoked by uid 99); 20 Jul 2009 05:04:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 05:04:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 05:04:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EA59D29A0011
	for <common-issues@hadoop.apache.org>; Sun, 19 Jul 2009 22:04:14 -0700 (PDT)
Message-ID: <2055548189.1248066254958.JavaMail.jira@brutus>
Date: Sun, 19 Jul 2009 22:04:14 -0700 (PDT)
From: "Ravi Gummadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6163) Progress class should provide an api
 if phases exist
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Progress class should provide an api if phases exist
----------------------------------------------------

                 Key: HADOOP-6163
                 URL: https://issues.apache.org/jira/browse/HADOOP-6163
             Project: Hadoop Common
          Issue Type: Improvement
          Components: util
    Affects Versions: 0.21.0
            Reporter: Ravi Gummadi
            Assignee: Ravi Gummadi
             Fix For: 0.21.0


Progress class needs to provide an api for client to know if there are phases.
This is needed for Task.setProgress() to decide whether to update progress of task or progress of current phase in the task.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-424-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 20 05:05:33 2009
Return-Path: <common-issues-return-424-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26409 invoked from network); 20 Jul 2009 05:05:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jul 2009 05:05:33 -0000
Received: (qmail 14618 invoked by uid 500); 20 Jul 2009 05:06:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14575 invoked by uid 500); 20 Jul 2009 05:06:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14565 invoked by uid 99); 20 Jul 2009 05:06:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 05:06:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 05:06:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D8D56234C044
	for <common-issues@hadoop.apache.org>; Sun, 19 Jul 2009 22:06:14 -0700 (PDT)
Message-ID: <290777667.1248066374873.JavaMail.jira@brutus>
Date: Sun, 19 Jul 2009 22:06:14 -0700 (PDT)
From: "Ravi Gummadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6163) Progress class should provide an api
 if phases exist
In-Reply-To: <2055548189.1248066254958.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated HADOOP-6163:
---------------------------------

    Attachment: HADOOP-6163.patch

Attaching patch that adds   boolean phasesExist() method to Progress.java.

Please review and provide your comments.

> Progress class should provide an api if phases exist
> ----------------------------------------------------
>
>                 Key: HADOOP-6163
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6163
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6163.patch
>
>
> Progress class needs to provide an api for client to know if there are phases.
> This is needed for Task.setProgress() to decide whether to update progress of task or progress of current phase in the task.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-425-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 20 11:38:32 2009
Return-Path: <common-issues-return-425-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69780 invoked from network); 20 Jul 2009 11:38:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jul 2009 11:38:32 -0000
Received: (qmail 21019 invoked by uid 500); 20 Jul 2009 11:39:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20975 invoked by uid 500); 20 Jul 2009 11:39:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20964 invoked by uid 99); 20 Jul 2009 11:39:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 11:39:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 11:39:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 5F9C9234C055
	for <common-issues@hadoop.apache.org>; Mon, 20 Jul 2009 04:39:15 -0700 (PDT)
Message-ID: <186282748.1248089955390.JavaMail.jira@brutus>
Date: Mon, 20 Jul 2009 04:39:15 -0700 (PDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6120) Add support for Avro types in
 hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12733194#action_12733194 ] 

Sharad Agarwal commented on HADOOP-6120:
----------------------------------------

bq. JavaSerialization was written more to demonstrate the generality of the abstraction
I am fine removing JavaSerialization from the defaults, if folks think it will discourage people to use java serialization.

bq. folks can configure a given class to be serialized in different ways.
Do you mean that accept(Class<?> c) should select serialization not only based on class but also based on the meta data passed ? We need to change accept as well, no ?

bq. Perhaps this belongs in another Jira, but without it we're missing out on a lot of Avro's functionality.
I think this can be discussed in another Jira, as the current implementation should be fine in case reader and writer are for same schema. We can start using this in mapreduce jobs.

> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch, 6120_v3.patch, 6120_v4.patch, 6120_v5.patch, HADOOP-6120.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-426-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 20 11:38:34 2009
Return-Path: <common-issues-return-426-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69922 invoked from network); 20 Jul 2009 11:38:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jul 2009 11:38:34 -0000
Received: (qmail 21929 invoked by uid 500); 20 Jul 2009 11:39:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21891 invoked by uid 500); 20 Jul 2009 11:39:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21881 invoked by uid 99); 20 Jul 2009 11:39:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 11:39:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 11:39:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 2EA38234C045
	for <common-issues@hadoop.apache.org>; Mon, 20 Jul 2009 04:39:15 -0700 (PDT)
Message-ID: <967301748.1248089955190.JavaMail.jira@brutus>
Date: Mon, 20 Jul 2009 04:39:15 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-5935) Hudson's release audit warnings link
 is broken
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan resolved HADOOP-5935.
----------------------------------------

    Resolution: Fixed

committed with the latest trunk code.

> Hudson's release audit warnings link is broken
> ----------------------------------------------
>
>                 Key: HADOOP-5935
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5935
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: hudson
>            Reporter: Tom White
>            Assignee: Giridharan Kesavan
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5935.patch
>
>
> For example, on HADOOP-5170 the link http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/392/artifact/trunk/current/releaseAuditDiffWarnings.txt gives a 404. This makes it hard to work out which file or files are causing a problem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-427-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 20 18:57:36 2009
Return-Path: <common-issues-return-427-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87753 invoked from network); 20 Jul 2009 18:57:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jul 2009 18:57:36 -0000
Received: (qmail 58920 invoked by uid 500); 20 Jul 2009 18:58:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58881 invoked by uid 500); 20 Jul 2009 18:58:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58811 invoked by uid 99); 20 Jul 2009 18:58:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 18:58:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 18:58:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C869E234C044
	for <common-issues@hadoop.apache.org>; Mon, 20 Jul 2009 11:58:14 -0700 (PDT)
Message-ID: <78432043.1248116294809.JavaMail.jira@brutus>
Date: Mon, 20 Jul 2009 11:58:14 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6161) Add get/setEnum to Configuration
In-Reply-To: <666455284.1247958254877.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12733316#action_12733316 ] 

Owen O'Malley commented on HADOOP-6161:
---------------------------------------

+1

> Add get/setEnum to Configuration
> --------------------------------
>
>                 Key: HADOOP-6161
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6161
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Chris Douglas
>            Assignee: Chris Douglas
>            Priority: Minor
>         Attachments: C6161-0.patch
>
>
> It would be useful if Configuration had helper get/set methods for enumerated types.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-428-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 20 19:37:32 2009
Return-Path: <common-issues-return-428-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2918 invoked from network); 20 Jul 2009 19:37:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jul 2009 19:37:32 -0000
Received: (qmail 32996 invoked by uid 500); 20 Jul 2009 19:38:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32948 invoked by uid 500); 20 Jul 2009 19:38:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32938 invoked by uid 99); 20 Jul 2009 19:38:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 19:38:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 19:38:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 01C0A234C1E6
	for <common-issues@hadoop.apache.org>; Mon, 20 Jul 2009 12:38:15 -0700 (PDT)
Message-ID: <1169680547.1248118695006.JavaMail.jira@brutus>
Date: Mon, 20 Jul 2009 12:38:15 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6120) Add support for Avro types in
 hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12733331#action_12733331 ] 

Doug Cutting commented on HADOOP-6120:
--------------------------------------

If, e.g., a class both implements Writable and also either implements AvroReflectSerializeable or whose package is listed in the config, then which serialization is used is determined by the order of the serializers in the config, which could change.  Similarly for the Serializeable interface and JavaSerialization.  So its not safe to assume that the Class->Serialization map is fixed, and we should really be storing at least the serialization's name in container metadata, and probably also a version (e.g. serialVersionUID for JavaSerialization).  And once we have a mechanism to support that, we can also store other metadata, like the schema, so that we can read older versions and generic data.

+1 for removing JavaSerialization from the defaults.

+1 for a separate Jira on serialization metadata.  But it would be best to have such metadata by the 0.21 freeze, in two weeks.


> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch, 6120_v3.patch, 6120_v4.patch, 6120_v5.patch, HADOOP-6120.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-429-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 20 22:34:32 2009
Return-Path: <common-issues-return-429-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19545 invoked from network); 20 Jul 2009 22:34:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jul 2009 22:34:32 -0000
Received: (qmail 61358 invoked by uid 500); 20 Jul 2009 22:35:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61316 invoked by uid 500); 20 Jul 2009 22:35:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61306 invoked by uid 99); 20 Jul 2009 22:35:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 22:35:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 22:35:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CA23F234C044
	for <common-issues@hadoop.apache.org>; Mon, 20 Jul 2009 15:35:14 -0700 (PDT)
Message-ID: <634330581.1248129314813.JavaMail.jira@brutus>
Date: Mon, 20 Jul 2009 15:35:14 -0700 (PDT)
From: "Raghu Angadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6132) RPC client opens an extra
 connection for VersionedProtocol
In-Reply-To: <840919907.1247085075163.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12733381#action_12733381 ] 

Raghu Angadi commented on HADOOP-6132:
--------------------------------------

sure. That is fine. If someone actually ends up using the interface, they will be similarly surprised. But for now I guess it is fine. If we want to fix it, we need to make the interface take class object as one of the arguments (since that is part of definition of what a connection is).

+1.


> RPC client opens an extra connection for VersionedProtocol
> ----------------------------------------------------------
>
>                 Key: HADOOP-6132
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6132
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: 1.patch, 2.patch
>
>
> RPC client caches connections per protocol. However, since all of our real protocols are subclasses of VersionedProtocol, a bug in the implementation makes the client opens an extra connection just for the VersionedProtocol, which is not needed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-430-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 20 23:00:42 2009
Return-Path: <common-issues-return-430-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30587 invoked from network); 20 Jul 2009 23:00:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Jul 2009 23:00:41 -0000
Received: (qmail 89923 invoked by uid 500); 20 Jul 2009 23:01:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89881 invoked by uid 500); 20 Jul 2009 23:01:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89871 invoked by uid 99); 20 Jul 2009 23:01:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 23:01:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2009 23:01:45 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E1A3F234C044
	for <common-issues@hadoop.apache.org>; Mon, 20 Jul 2009 16:01:24 -0700 (PDT)
Message-ID: <468504763.1248130884915.JavaMail.jira@brutus>
Date: Mon, 20 Jul 2009 16:01:24 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6161) Add get/setEnum to Configuration
In-Reply-To: <666455284.1247958254877.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-6161:
----------------------------------

       Resolution: Fixed
    Fix Version/s: 0.21.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I committed this and updated the core and core-test jars in HDFS and MAPREDUCE.

> Add get/setEnum to Configuration
> --------------------------------
>
>                 Key: HADOOP-6161
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6161
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Chris Douglas
>            Assignee: Chris Douglas
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: C6161-0.patch
>
>
> It would be useful if Configuration had helper get/set methods for enumerated types.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-431-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 21 03:54:34 2009
Return-Path: <common-issues-return-431-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27758 invoked from network); 21 Jul 2009 03:54:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Jul 2009 03:54:34 -0000
Received: (qmail 36697 invoked by uid 500); 21 Jul 2009 03:55:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36657 invoked by uid 500); 21 Jul 2009 03:55:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36647 invoked by uid 99); 21 Jul 2009 03:55:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 03:55:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 03:55:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E1D99234C044
	for <common-issues@hadoop.apache.org>; Mon, 20 Jul 2009 20:55:14 -0700 (PDT)
Message-ID: <466254032.1248148514910.JavaMail.jira@brutus>
Date: Mon, 20 Jul 2009 20:55:14 -0700 (PDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6120) Add support for Avro types in hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sharad Agarwal updated HADOOP-6120:
-----------------------------------

    Status: Open  (was: Patch Available)

> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch, 6120_v3.patch, 6120_v4.patch, 6120_v5.patch, HADOOP-6120.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-432-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 21 03:56:32 2009
Return-Path: <common-issues-return-432-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27914 invoked from network); 21 Jul 2009 03:56:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Jul 2009 03:56:32 -0000
Received: (qmail 37343 invoked by uid 500); 21 Jul 2009 03:57:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37302 invoked by uid 500); 21 Jul 2009 03:57:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37292 invoked by uid 99); 21 Jul 2009 03:57:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 03:57:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 03:57:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1F7B029A0022
	for <common-issues@hadoop.apache.org>; Mon, 20 Jul 2009 20:57:15 -0700 (PDT)
Message-ID: <843579526.1248148635128.JavaMail.jira@brutus>
Date: Mon, 20 Jul 2009 20:57:15 -0700 (PDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6120) Add support for Avro types in hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sharad Agarwal updated HADOOP-6120:
-----------------------------------

    Status: Patch Available  (was: Open)

> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch, 6120_v3.patch, 6120_v4.patch, 6120_v5.patch, 6120_v6.patch, HADOOP-6120.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-433-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 21 03:56:34 2009
Return-Path: <common-issues-return-433-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27945 invoked from network); 21 Jul 2009 03:56:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Jul 2009 03:56:33 -0000
Received: (qmail 37409 invoked by uid 500); 21 Jul 2009 03:57:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37369 invoked by uid 500); 21 Jul 2009 03:57:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37359 invoked by uid 99); 21 Jul 2009 03:57:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 03:57:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 03:57:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E0A9C234C044
	for <common-issues@hadoop.apache.org>; Mon, 20 Jul 2009 20:57:14 -0700 (PDT)
Message-ID: <1553647912.1248148634905.JavaMail.jira@brutus>
Date: Mon, 20 Jul 2009 20:57:14 -0700 (PDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6120) Add support for Avro types in hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sharad Agarwal updated HADOOP-6120:
-----------------------------------

    Attachment: 6120_v6.patch

Removed JavaSerialization from defaults

> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch, 6120_v3.patch, 6120_v4.patch, 6120_v5.patch, 6120_v6.patch, HADOOP-6120.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-434-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 21 04:18:34 2009
Return-Path: <common-issues-return-434-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31487 invoked from network); 21 Jul 2009 04:18:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Jul 2009 04:18:33 -0000
Received: (qmail 49608 invoked by uid 500); 21 Jul 2009 04:19:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49565 invoked by uid 500); 21 Jul 2009 04:19:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49553 invoked by uid 99); 21 Jul 2009 04:19:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 04:19:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 04:19:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0E27129A0016
	for <common-issues@hadoop.apache.org>; Mon, 20 Jul 2009 21:19:15 -0700 (PDT)
Message-ID: <344904807.1248149955057.JavaMail.jira@brutus>
Date: Mon, 20 Jul 2009 21:19:15 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6120) Add support for Avro types in
 hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12733466#action_12733466 ] 

Hadoop QA commented on HADOOP-6120:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12414061/6120_v6.patch
  against trunk revision 796052.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 15 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/582/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/582/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/582/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/582/console

This message is automatically generated.

> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch, 6120_v3.patch, 6120_v4.patch, 6120_v5.patch, 6120_v6.patch, HADOOP-6120.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-435-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 21 04:24:33 2009
Return-Path: <common-issues-return-435-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32193 invoked from network); 21 Jul 2009 04:24:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Jul 2009 04:24:33 -0000
Received: (qmail 51695 invoked by uid 500); 21 Jul 2009 04:25:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51655 invoked by uid 500); 21 Jul 2009 04:25:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51645 invoked by uid 99); 21 Jul 2009 04:25:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 04:25:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 04:25:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DA0C9234C044
	for <common-issues@hadoop.apache.org>; Mon, 20 Jul 2009 21:25:14 -0700 (PDT)
Message-ID: <274655062.1248150314878.JavaMail.jira@brutus>
Date: Mon, 20 Jul 2009 21:25:14 -0700 (PDT)
From: "Sreekanth Ramakrishnan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5488) HADOOP-2721 doesn't clean up
 descendant processes of a jvm that exits cleanly after running a task
 successfully
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sreekanth Ramakrishnan updated HADOOP-5488:
-------------------------------------------

    Attachment: hadoop-5488-ydist.patch

Attaching patch for Yahoo distribution.

> HADOOP-2721 doesn't clean up descendant processes of a jvm that exits cleanly after running a task successfully
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5488
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5488
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Vinod K V
>            Assignee: Ravi Gummadi
>             Fix For: 0.21.0
>
>         Attachments: hadoop-5488-ydist.patch, HADOOP-5488.patch, HADOOP_5488_5614.patch, HADOOP_5488_5614.v1.2.patch, HADOOP_5488_5614.v1.3.patch, HADOOP_5488_5614.v1.patch, testcase5488.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-436-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 21 04:26:34 2009
Return-Path: <common-issues-return-436-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32429 invoked from network); 21 Jul 2009 04:26:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Jul 2009 04:26:34 -0000
Received: (qmail 52310 invoked by uid 500); 21 Jul 2009 04:27:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52269 invoked by uid 500); 21 Jul 2009 04:27:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52194 invoked by uid 99); 21 Jul 2009 04:27:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 04:27:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 04:27:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DF90229A0018
	for <common-issues@hadoop.apache.org>; Mon, 20 Jul 2009 21:27:14 -0700 (PDT)
Message-ID: <214187820.1248150434914.JavaMail.jira@brutus>
Date: Mon, 20 Jul 2009 21:27:14 -0700 (PDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6120) Add support for Avro types in
 hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12733470#action_12733470 ] 

Sharad Agarwal commented on HADOOP-6120:
----------------------------------------

Test failure is unrelated.

> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: 6120_v1.patch, 6120_v2.patch, 6120_v3.patch, 6120_v4.patch, 6120_v5.patch, 6120_v6.patch, HADOOP-6120.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-437-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 21 04:58:32 2009
Return-Path: <common-issues-return-437-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37381 invoked from network); 21 Jul 2009 04:58:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Jul 2009 04:58:31 -0000
Received: (qmail 62457 invoked by uid 500); 21 Jul 2009 04:59:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62417 invoked by uid 500); 21 Jul 2009 04:59:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62405 invoked by uid 99); 21 Jul 2009 04:59:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 04:59:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 04:59:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CFFFC234C044
	for <common-issues@hadoop.apache.org>; Mon, 20 Jul 2009 21:59:14 -0700 (PDT)
Message-ID: <887346771.1248152354838.JavaMail.jira@brutus>
Date: Mon, 20 Jul 2009 21:59:14 -0700 (PDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6140)
 DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
In-Reply-To: <1165046874.1247245875107.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12733477#action_12733477 ] 

Amareshwari Sriramadasu commented on HADOOP-6140:
-------------------------------------------------

path.separator bug is fixed by HADOOP-4864.

> DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6140
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6140
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.18.3
>            Reporter: Vladimir Klimontovich
>         Attachments: HADOOP-6140-ver4.patch
>
>
> addArchiveToClassPath is a method of DistributedCache class. It should be called before running a task. It accepts path to a jar file on a DFS. After it
> this method should put this jar file on sitribuuted cache and than add this file to classpath to each map/reduce process on job tracker.
> This method didn't work.
> Bug 1:
> addArchiveToClassPath adds DFS-path to archive to mapred.job.classpath.archives property. It uses System.getProperty("path.separator") as delimiter of multiple path.
> getFileClassPaths that is called from TaskRunner uses splits mapred.job.classpath.archives using System.getProperty("path.separator").
> In unix systems System.getProperty("path.separator") equals to ":". DFS-path urls is hdfs://host:port/path. It means that a result of split will be
> [ hdfs,//host,port/path].
> Suggested solution: use "," instead of  
> Bug 2:
> in TaskRunner there is an algorithm that looks for correspondence between DFS paths and local paths in distributed cache. 
> It compares
>    if (archives[i].getPath().equals(
>                                                archiveClasspaths[j].toString())){
> instead of
>     if (archives[i].toString().equals(
>                                                archiveClasspaths[j].toString())) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-438-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 21 08:31:34 2009
Return-Path: <common-issues-return-438-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14130 invoked from network); 21 Jul 2009 08:31:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Jul 2009 08:31:34 -0000
Received: (qmail 43472 invoked by uid 500); 21 Jul 2009 08:32:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43428 invoked by uid 500); 21 Jul 2009 08:32:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43418 invoked by uid 99); 21 Jul 2009 08:32:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 08:32:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 08:32:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4866B234C055
	for <common-issues@hadoop.apache.org>; Tue, 21 Jul 2009 01:32:15 -0700 (PDT)
Message-ID: <46756721.1248165135295.JavaMail.jira@brutus>
Date: Tue, 21 Jul 2009 01:32:15 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6150) Need to be able to instantiate a
 comparator instance from a comparator string without creating a
 TFile.Reader object
In-Reply-To: <1804574098.1247618054845.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-6150:
------------------------------

    Attachment: HADOOP-6150-0721.patch

> Need to be able to instantiate a comparator instance from a comparator string without creating a TFile.Reader object
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6150
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6150
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.0, 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6150-0721.patch
>
>
> Occasionally, we want have the same instance of comparator object represented by the TFile comparator string. We should be able to do that without requiring to first open up a tfile that was previously written use the same comparator.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-439-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 21 08:31:34 2009
Return-Path: <common-issues-return-439-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14142 invoked from network); 21 Jul 2009 08:31:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Jul 2009 08:31:34 -0000
Received: (qmail 43536 invoked by uid 500); 21 Jul 2009 08:32:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43492 invoked by uid 500); 21 Jul 2009 08:32:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43467 invoked by uid 99); 21 Jul 2009 08:32:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 08:32:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 08:32:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4CEB7234C1E9
	for <common-issues@hadoop.apache.org>; Tue, 21 Jul 2009 01:32:15 -0700 (PDT)
Message-ID: <1257415101.1248165135314.JavaMail.jira@brutus>
Date: Tue, 21 Jul 2009 01:32:15 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6150) Need to be able to instantiate a
 comparator instance from a comparator string without creating a
 TFile.Reader object
In-Reply-To: <1804574098.1247618054845.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-6150:
------------------------------

    Fix Version/s: 0.21.0
           Status: Patch Available  (was: Open)

Trivial patch that exposes an internal API to public. No unit test needed.

> Need to be able to instantiate a comparator instance from a comparator string without creating a TFile.Reader object
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6150
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6150
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.0, 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6150-0721.patch
>
>
> Occasionally, we want have the same instance of comparator object represented by the TFile comparator string. We should be able to do that without requiring to first open up a tfile that was previously written use the same comparator.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-440-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 21 09:29:35 2009
Return-Path: <common-issues-return-440-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30057 invoked from network); 21 Jul 2009 09:29:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Jul 2009 09:29:34 -0000
Received: (qmail 94078 invoked by uid 500); 21 Jul 2009 09:30:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93992 invoked by uid 500); 21 Jul 2009 09:30:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93886 invoked by uid 99); 21 Jul 2009 09:30:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 09:30:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 09:30:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DD2C0234C1E6
	for <common-issues@hadoop.apache.org>; Tue, 21 Jul 2009 02:30:14 -0700 (PDT)
Message-ID: <326979076.1248168614905.JavaMail.jira@brutus>
Date: Tue, 21 Jul 2009 02:30:14 -0700 (PDT)
From: "Ravi Gummadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6163) Progress class should provide an api
 if phases exist
In-Reply-To: <2055548189.1248066254958.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated HADOOP-6163:
---------------------------------

    Attachment: HADOOP-6163.v1.patch

Attaching new patch that checks if phases exist in phase() method and returns null if there are no phases in this Progress object instead of throwing ArrayIndexOutOfBoundsException.

> Progress class should provide an api if phases exist
> ----------------------------------------------------
>
>                 Key: HADOOP-6163
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6163
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6163.patch, HADOOP-6163.v1.patch
>
>
> Progress class needs to provide an api for client to know if there are phases.
> This is needed for Task.setProgress() to decide whether to update progress of task or progress of current phase in the task.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-441-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 21 09:45:32 2009
Return-Path: <common-issues-return-441-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38082 invoked from network); 21 Jul 2009 09:45:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Jul 2009 09:45:31 -0000
Received: (qmail 12493 invoked by uid 500); 21 Jul 2009 09:46:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12449 invoked by uid 500); 21 Jul 2009 09:46:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12435 invoked by uid 99); 21 Jul 2009 09:46:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 09:46:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 09:46:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C90EA234C044
	for <common-issues@hadoop.apache.org>; Tue, 21 Jul 2009 02:46:14 -0700 (PDT)
Message-ID: <1195629918.1248169574809.JavaMail.jira@brutus>
Date: Tue, 21 Jul 2009 02:46:14 -0700 (PDT)
From: "Ravi Gummadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6163) Progress class should provide an api
 if phases exist
In-Reply-To: <2055548189.1248066254958.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated HADOOP-6163:
---------------------------------

    Status: Patch Available  (was: Open)

> Progress class should provide an api if phases exist
> ----------------------------------------------------
>
>                 Key: HADOOP-6163
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6163
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6163.patch, HADOOP-6163.v1.patch
>
>
> Progress class needs to provide an api for client to know if there are phases.
> This is needed for Task.setProgress() to decide whether to update progress of task or progress of current phase in the task.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-442-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 21 10:19:34 2009
Return-Path: <common-issues-return-442-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57238 invoked from network); 21 Jul 2009 10:19:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Jul 2009 10:19:34 -0000
Received: (qmail 60417 invoked by uid 500); 21 Jul 2009 10:20:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60390 invoked by uid 500); 21 Jul 2009 10:20:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60369 invoked by uid 99); 21 Jul 2009 10:20:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 10:20:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 10:20:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DE418234C051
	for <common-issues@hadoop.apache.org>; Tue, 21 Jul 2009 03:20:14 -0700 (PDT)
Message-ID: <1109553295.1248171614909.JavaMail.jira@brutus>
Date: Tue, 21 Jul 2009 03:20:14 -0700 (PDT)
From: "Ravi Gummadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6163) Progress class should provide an
 api if phases exist
In-Reply-To: <2055548189.1248066254958.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12733574#action_12733574 ] 

Ravi Gummadi commented on HADOOP-6163:
--------------------------------------

ant tet-patch gave

     [exec] -1 overall.
     [exec]
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec]
     [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
     [exec]                         Please justify why no new tests are needed for this patch.
     [exec]                         Also please list what manual steps were performed to verify this patch.
     [exec]
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec]
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec]
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec]
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.



This doesn't need a testcase as the testcase in MAPREDUCE-743 covers this code change.

> Progress class should provide an api if phases exist
> ----------------------------------------------------
>
>                 Key: HADOOP-6163
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6163
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6163.patch, HADOOP-6163.v1.patch
>
>
> Progress class needs to provide an api for client to know if there are phases.
> This is needed for Task.setProgress() to decide whether to update progress of task or progress of current phase in the task.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-443-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 21 11:00:33 2009
Return-Path: <common-issues-return-443-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69541 invoked from network); 21 Jul 2009 11:00:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Jul 2009 11:00:32 -0000
Received: (qmail 2958 invoked by uid 500); 21 Jul 2009 11:01:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2916 invoked by uid 500); 21 Jul 2009 11:01:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2906 invoked by uid 99); 21 Jul 2009 11:01:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 11:01:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 11:01:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CC93F234C046
	for <common-issues@hadoop.apache.org>; Tue, 21 Jul 2009 04:01:15 -0700 (PDT)
Message-ID: <593290691.1248174075837.JavaMail.jira@brutus>
Date: Tue, 21 Jul 2009 04:01:15 -0700 (PDT)
From: "Ravi Gummadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6163) Progress class should provide an
 api if phases exist
In-Reply-To: <2055548189.1248066254958.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12733588#action_12733588 ] 

Ravi Gummadi commented on HADOOP-6163:
--------------------------------------

Unit tests  passed on my local machine.

> Progress class should provide an api if phases exist
> ----------------------------------------------------
>
>                 Key: HADOOP-6163
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6163
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6163.patch, HADOOP-6163.v1.patch
>
>
> Progress class needs to provide an api for client to know if there are phases.
> This is needed for Task.setProgress() to decide whether to update progress of task or progress of current phase in the task.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-444-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 21 21:32:34 2009
Return-Path: <common-issues-return-444-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38147 invoked from network); 21 Jul 2009 21:32:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Jul 2009 21:32:34 -0000
Received: (qmail 17725 invoked by uid 500); 21 Jul 2009 21:33:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17686 invoked by uid 500); 21 Jul 2009 21:33:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17676 invoked by uid 99); 21 Jul 2009 21:33:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 21:33:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 21:33:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D45A4234C044
	for <common-issues@hadoop.apache.org>; Tue, 21 Jul 2009 14:33:14 -0700 (PDT)
Message-ID: <53791890.1248211994862.JavaMail.jira@brutus>
Date: Tue, 21 Jul 2009 14:33:14 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-1257) a distributed junit test runner
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12733838#action_12733838 ] 

Konstantin Boudnik commented on HADOOP-1257:
--------------------------------------------

I'm slightly confused with this issue: is there a need for a special MR tests runner? Or this is all about a general unit test runner which will allow to speed up the testing process on the Hudson's side? 

I believe that it is important to make such a distinguish because it would influence the solution a lot. E.g. for a generic parallel execution we don't need any of MR pieces, proposed by the patch in the attachment (unless I've totally misunderstood its purpose)

> a distributed junit test runner
> -------------------------------
>
>                 Key: HADOOP-1257
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1257
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: test
>    Affects Versions: 0.12.3
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: HADOOP-1257.patch
>
>
> It would be nice to have a distributed junit runner that would run 100 instances of each test a separate map, and use counters to count how many times each test passed/failed/hung.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-445-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 21 23:54:31 2009
Return-Path: <common-issues-return-445-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84347 invoked from network); 21 Jul 2009 23:54:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Jul 2009 23:54:31 -0000
Received: (qmail 46576 invoked by uid 500); 21 Jul 2009 23:55:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46535 invoked by uid 500); 21 Jul 2009 23:55:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46525 invoked by uid 99); 21 Jul 2009 23:55:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 23:55:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jul 2009 23:55:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E0F9E29A0016
	for <common-issues@hadoop.apache.org>; Tue, 21 Jul 2009 16:55:14 -0700 (PDT)
Message-ID: <1144829364.1248220514920.JavaMail.jira@brutus>
Date: Tue, 21 Jul 2009 16:55:14 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6164) Test-patch's method of checking for
 new tests is not correct
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Test-patch's method of checking for new tests is not correct
------------------------------------------------------------

                 Key: HADOOP-6164
                 URL: https://issues.apache.org/jira/browse/HADOOP-6164
             Project: Hadoop Common
          Issue Type: Bug
          Components: test
            Reporter: Jakob Homan


As happened in HDFS-458, test-patch/hudson saw the previous patch(HDFS-492) add a new test, bringing the total number of tests to 48.  Then it tested 458, which had no new tests as it was a change to the build file.  492 had not yet been committed yet, so there were still only 48 tests in trunk.  But test-patch failed the patch, expecting 49 tests.  test-patch should check the correct number of tests based on trunk, not on whatever number it last saw.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-446-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 22 00:30:34 2009
Return-Path: <common-issues-return-446-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97303 invoked from network); 22 Jul 2009 00:30:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jul 2009 00:30:31 -0000
Received: (qmail 86910 invoked by uid 500); 22 Jul 2009 00:31:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86870 invoked by uid 500); 22 Jul 2009 00:31:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86860 invoked by uid 99); 22 Jul 2009 00:31:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 00:31:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 00:31:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C9434234C044
	for <common-issues@hadoop.apache.org>; Tue, 21 Jul 2009 17:31:14 -0700 (PDT)
Message-ID: <475960326.1248222674813.JavaMail.jira@brutus>
Date: Tue, 21 Jul 2009 17:31:14 -0700 (PDT)
From: "Philip Zeyliger (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6105) Provide a way to automatically
 handle backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12733919#action_12733919 ] 

Philip Zeyliger commented on HADOOP-6105:
-----------------------------------------

I think you have to think about how Hadoop's notion of a "final" flag interacts with this, too.  If a system administrator has set either A or B to be final, then that value must override any user-submitted value, regardless of which was set first.

> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-447-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 22 00:54:31 2009
Return-Path: <common-issues-return-447-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3512 invoked from network); 22 Jul 2009 00:54:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jul 2009 00:54:31 -0000
Received: (qmail 98793 invoked by uid 500); 22 Jul 2009 00:55:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98753 invoked by uid 500); 22 Jul 2009 00:55:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98743 invoked by uid 99); 22 Jul 2009 00:55:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 00:55:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 00:55:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C68C4234C044
	for <common-issues@hadoop.apache.org>; Tue, 21 Jul 2009 17:55:14 -0700 (PDT)
Message-ID: <887636184.1248224114806.JavaMail.jira@brutus>
Date: Tue, 21 Jul 2009 17:55:14 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6164) Test-patch's method of checking for
 new tests is not correct
In-Reply-To: <1144829364.1248220514920.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6164:
--------------------------------

    Description: As happened in HDFS-458, test-patch/hudson saw the previous patch(HDFS-492) add a new test, bringing the total number of tests to 49.  Then it tested 458, which had no new tests as it was a change to the build file.  492 had not yet been committed yet, so there were still only 48 tests in trunk.  But test-patch failed the patch, expecting 49 tests.  test-patch should check the correct number of tests based on trunk, not on whatever number it last saw.  (was: As happened in HDFS-458, test-patch/hudson saw the previous patch(HDFS-492) add a new test, bringing the total number of tests to 48.  Then it tested 458, which had no new tests as it was a change to the build file.  492 had not yet been committed yet, so there were still only 48 tests in trunk.  But test-patch failed the patch, expecting 49 tests.  test-patch should check the correct number of tests based on trunk, not on whatever number it last saw.)

Fixed error in description in re: # of tests.  Otherwise issue made no sense.

> Test-patch's method of checking for new tests is not correct
> ------------------------------------------------------------
>
>                 Key: HADOOP-6164
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6164
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Jakob Homan
>
> As happened in HDFS-458, test-patch/hudson saw the previous patch(HDFS-492) add a new test, bringing the total number of tests to 49.  Then it tested 458, which had no new tests as it was a change to the build file.  492 had not yet been committed yet, so there were still only 48 tests in trunk.  But test-patch failed the patch, expecting 49 tests.  test-patch should check the correct number of tests based on trunk, not on whatever number it last saw.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-448-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 22 01:31:32 2009
Return-Path: <common-issues-return-448-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16061 invoked from network); 22 Jul 2009 01:31:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jul 2009 01:31:32 -0000
Received: (qmail 24075 invoked by uid 500); 22 Jul 2009 01:32:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24035 invoked by uid 500); 22 Jul 2009 01:32:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24025 invoked by uid 99); 22 Jul 2009 01:32:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 01:32:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 01:32:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CDFBB234C044
	for <common-issues@hadoop.apache.org>; Tue, 21 Jul 2009 18:32:14 -0700 (PDT)
Message-ID: <1594585786.1248226334829.JavaMail.jira@brutus>
Date: Tue, 21 Jul 2009 18:32:14 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-1257) a distributed junit test runner
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12733942#action_12733942 ] 

Konstantin Boudnik commented on HADOOP-1257:
--------------------------------------------

Also, it seems like a possibility that parallel tests execution will be included into JUnit 4.7 (see HADOOP-4901 comments on that)

> a distributed junit test runner
> -------------------------------
>
>                 Key: HADOOP-1257
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1257
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: test
>    Affects Versions: 0.12.3
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: HADOOP-1257.patch
>
>
> It would be nice to have a distributed junit runner that would run 100 instances of each test a separate map, and use counters to count how many times each test passed/failed/hung.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-449-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 22 06:05:36 2009
Return-Path: <common-issues-return-449-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87774 invoked from network); 22 Jul 2009 06:05:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jul 2009 06:05:34 -0000
Received: (qmail 46498 invoked by uid 500); 22 Jul 2009 06:06:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46156 invoked by uid 500); 22 Jul 2009 06:06:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46017 invoked by uid 99); 22 Jul 2009 06:06:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 06:06:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 06:06:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id B78B629A001C
	for <common-issues@hadoop.apache.org>; Tue, 21 Jul 2009 23:06:15 -0700 (PDT)
Message-ID: <91935080.1248242775750.JavaMail.jira@brutus>
Date: Tue, 21 Jul 2009 23:06:15 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6164) Test-patch's method of checking for
 new tests is not correct
In-Reply-To: <1144829364.1248220514920.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734003#action_12734003 ] 

Giridharan Kesavan commented on HADOOP-6164:
--------------------------------------------

test patch doesn make any decision on the no of tests run. If the tests fail it fails the build. It is hudson which parses the junit test results from the xml file.

BTW hudson didnt run any core-test for HDFS-450
it failed for create-configure-c++ so didnt run test-core target
so -1 for core test
and hdfsproxy test failed so -1 for contrib test.

test-patch script doesnt do any count on the no of test run.



> Test-patch's method of checking for new tests is not correct
> ------------------------------------------------------------
>
>                 Key: HADOOP-6164
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6164
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Jakob Homan
>
> As happened in HDFS-458, test-patch/hudson saw the previous patch(HDFS-492) add a new test, bringing the total number of tests to 49.  Then it tested 458, which had no new tests as it was a change to the build file.  492 had not yet been committed yet, so there were still only 48 tests in trunk.  But test-patch failed the patch, expecting 49 tests.  test-patch should check the correct number of tests based on trunk, not on whatever number it last saw.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-450-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 22 06:11:31 2009
Return-Path: <common-issues-return-450-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90775 invoked from network); 22 Jul 2009 06:11:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jul 2009 06:11:31 -0000
Received: (qmail 52499 invoked by uid 500); 22 Jul 2009 06:12:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52459 invoked by uid 500); 22 Jul 2009 06:12:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52448 invoked by uid 99); 22 Jul 2009 06:12:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 06:12:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 06:12:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DE54429A0011
	for <common-issues@hadoop.apache.org>; Tue, 21 Jul 2009 23:12:14 -0700 (PDT)
Message-ID: <884612214.1248243134909.JavaMail.jira@brutus>
Date: Tue, 21 Jul 2009 23:12:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6164) Test-patch's method of checking for
 new tests is not correct
In-Reply-To: <1144829364.1248220514920.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan resolved HADOOP-6164.
----------------------------------------

    Resolution: Invalid
      Assignee: Giridharan Kesavan

Im closing this resolved invalid.
This is how hudson works and this has nothing to do with the test-patch script.

> Test-patch's method of checking for new tests is not correct
> ------------------------------------------------------------
>
>                 Key: HADOOP-6164
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6164
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Jakob Homan
>            Assignee: Giridharan Kesavan
>
> As happened in HDFS-458, test-patch/hudson saw the previous patch(HDFS-492) add a new test, bringing the total number of tests to 49.  Then it tested 458, which had no new tests as it was a change to the build file.  492 had not yet been committed yet, so there were still only 48 tests in trunk.  But test-patch failed the patch, expecting 49 tests.  test-patch should check the correct number of tests based on trunk, not on whatever number it last saw.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-451-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 22 11:24:33 2009
Return-Path: <common-issues-return-451-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98978 invoked from network); 22 Jul 2009 11:24:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jul 2009 11:24:33 -0000
Received: (qmail 50020 invoked by uid 500); 22 Jul 2009 11:25:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49995 invoked by uid 500); 22 Jul 2009 11:25:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49985 invoked by uid 99); 22 Jul 2009 11:25:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 11:25:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 11:25:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D6C7F234C044
	for <common-issues@hadoop.apache.org>; Wed, 22 Jul 2009 04:25:14 -0700 (PDT)
Message-ID: <1308804598.1248261914878.JavaMail.jira@brutus>
Date: Wed, 22 Jul 2009 04:25:14 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6165) Add metadata to Serializations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Add metadata to Serializations
------------------------------

                 Key: HADOOP-6165
                 URL: https://issues.apache.org/jira/browse/HADOOP-6165
             Project: Hadoop Common
          Issue Type: New Feature
          Components: contrib/serialization
            Reporter: Tom White
            Priority: Blocker
             Fix For: 0.21.0


The Serialization framework only allows a class to be passed as metadata. This assumes there is a one-to-one mapping between types and Serializations, which is overly restrictive. By permitting applications to pass arbitrary metadata to Serializations, they can get more control over which Serialization is used, and would also allow, for example, one to pass an Avro schema to an Avro Serialization.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-452-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 22 11:44:32 2009
Return-Path: <common-issues-return-452-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3028 invoked from network); 22 Jul 2009 11:44:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jul 2009 11:44:32 -0000
Received: (qmail 86819 invoked by uid 500); 22 Jul 2009 11:45:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86775 invoked by uid 500); 22 Jul 2009 11:45:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86763 invoked by uid 99); 22 Jul 2009 11:45:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 11:45:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 11:45:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 9669E234C1EA
	for <common-issues@hadoop.apache.org>; Wed, 22 Jul 2009 04:45:15 -0700 (PDT)
Message-ID: <1822663906.1248263115615.JavaMail.jira@brutus>
Date: Wed, 22 Jul 2009 04:45:15 -0700 (PDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4491) Per-job local data on the
 TaskTracker node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734079#action_12734079 ] 

Vinod K V commented on HADOOP-4491:
-----------------------------------


Broadly, there are two directory strucutres - system and users

 - system directory will be owned by mapreduce, thereby protecting the contents.
 - users is 755, owned by mapreduce
 - users/$jobid is clearly 700 and owned by the user.

 - system/$jobid/outputs can be directly $ttroot/ as was discussed offline. But I've left it inside system/$jobid as the $jobid directory seemed reduntant to me. In any case, the outputs once moved need to owned by the TT.

 - all of the files localized by the TT are written into system/$jobid
 - After job localization is done, all files under system/$jobid/userfiles are moved to users/$jobid to be consumed by the user's task and so owned by the user.
 - After task localization is done, the whole directory system/$jobid/$taskid is moved to users/$jobid/ and owned by the user.

 - when the task finishes, the whole users/$user/$jobid/$attemptid/output directory needs to be moved to outputs/$jobid/$attemptid.

 - cleaning up of a task is removal of  users//$jobid/$attemptid
 - cleaning up a job is removal of users//$jobid, system/$jobid

These changs will be needed for both DefaultTaskController as well as the LinuxTaskController.

LinuxTaskController uses the setuid binary to do the move operations as the root and changing ownership of the target files to the user.

Distributed cache files and the log files still need to be baked into this structure.

> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-453-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 22 11:44:38 2009
Return-Path: <common-issues-return-453-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3080 invoked from network); 22 Jul 2009 11:44:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jul 2009 11:44:38 -0000
Received: (qmail 86894 invoked by uid 500); 22 Jul 2009 11:45:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86854 invoked by uid 500); 22 Jul 2009 11:45:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86844 invoked by uid 99); 22 Jul 2009 11:45:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 11:45:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 11:45:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DD4FF234C004
	for <common-issues@hadoop.apache.org>; Wed, 22 Jul 2009 04:45:14 -0700 (PDT)
Message-ID: <1483627323.1248263114891.JavaMail.jira@brutus>
Date: Wed, 22 Jul 2009 04:45:14 -0700 (PDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4491) Per-job local data on the
 TaskTracker node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734078#action_12734078 ] 

Vinod K V commented on HADOOP-4491:
-----------------------------------

I had some offline discussions with Hemanth/Owen/Arun. Those discussions resulted in a change of proposal for this JIRA.

The TaskTracker's directory structure will be like the following:
{noformat}
$ttroot
   |
   |- distcache
   |
   |- users
   |     |
   |     '-- $jobid
   |            |
   |            |- jars
   |            |    '-- job.jar
   |            |- work
   |            |
   |            `-- $attemptid
   |                  |- job.xml
   |                  |- taskjvm.sh
   |                  |- work
   |                  |      '-- tmp
   |                  `-- output
   |
   '-- system
        |
        `-- $jobid
             |
             |- job.xml
             |
             |- userfiles
             |       |- jars
             |       |    '-- job.jar
             |       '-- work
             |
             |- $attemptid
             |        |- job.xml
             |        |- taskjvm.sh
             |        '-- work
             |              '-- tmp
             |
             '-- outputs
                     `-- $attemptid 
{noformat}

> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-454-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 22 11:52:33 2009
Return-Path: <common-issues-return-454-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4676 invoked from network); 22 Jul 2009 11:52:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jul 2009 11:52:32 -0000
Received: (qmail 95201 invoked by uid 500); 22 Jul 2009 11:53:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95162 invoked by uid 500); 22 Jul 2009 11:53:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95152 invoked by uid 99); 22 Jul 2009 11:53:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 11:53:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 11:53:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CFB0C234C004
	for <common-issues@hadoop.apache.org>; Wed, 22 Jul 2009 04:53:14 -0700 (PDT)
Message-ID: <1921850559.1248263594834.JavaMail.jira@brutus>
Date: Wed, 22 Jul 2009 04:53:14 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6165) Add metadata to Serializations
In-Reply-To: <1308804598.1248261914878.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6165:
------------------------------

    Attachment: HADOOP-6165.patch

Here's a patch with some ideas about how to go about this. It is very preliminary.



1. One of the problems is that Serialization/Serializer/Deserializer are all interfaces, which makes it difficult to evolve them. One way to manage this is to introduce Base{Serialization,Serializer,Deserializer} abstract classes that implement the corresponding interface. SerializationFactory will read the io.serializations configuration property and if a serialization implements BaseSerialization it will use that directly, while if it is a (legacy) Serialization it will wrap it in a BaseSerialization. The trick here is to put legacy Serializations at the end, since they have less metadata and are therefore less discriminating.

The Serialization/Serializer/Deserializer interfaces are all deprecated and can be removed in a future release, leaving only Base{Serialization,Serializer,Deserializer}.

2. In addition to the Map<String, String> metadata do we need to keep the class metadata? That is, do we need 

public abstract boolean accept(Class<?> c, Map<String, String> metadata);

or is the following sufficient?

public abstract boolean accept(Map<String, String> metadata); 

We could have a "class" entry in the map which stores this information, but we'd have to convert it to a Class object to do the isAssignableFrom check that some serializations need to do, e.g. Writable.class.isAssignableFrom(c). Perhaps this is OK.

3. Should we have a Metadata class to permit evolution of beyond Map<String, String>? (E.g. to keep a Class property.)

4. Where does the metadata come from? In the context of MapReduce, the answer depends on the stage of MapReduce. (None of these changes have been implemented in the patch.)

i. Map input

The metadata comes from the container. For example, in SequenceFiles the metadata comes from the key-value class types, and the SequenceFile metadata (a Map<Text, Text>, which is ideally suited for this scheme).

To support this, SequenceFile.Reader would pass its metadata to the deserializer. Similarly, SequenceFile.Writer would add metadata from the BaseSerializer to the SequenceFile's writer.

ii. Map output/Reduce input

The metadata would have to be supplied by the MapReduce framework. Just like we have mapred.mapoutput.{key,value}.class, we could have properties to specify extra metadata. Metadata is a map, so something like mapred.mapoutput.{key,value}.metadata.K where K can be an arbitrary string.

For example, one might define mapred.mapoutput.key.metadata.avroSchema to be the Avro schema for map output key types. To get this to work we would need support from Configuration to get a Map from a property prefix. So conf.getMap("mapred.mapoutput.key.metadata") would return a Map<String, String> of all the properties under the mapred.mapoutput.key.metadata prefix.

iii. Reduce output

The metadata would have to be supplied by the MapReduce framework. Just like the map output we could have mapred.output.{key,value}.metadata.K properties.

5. Resolution process

To take an Avro example: AvroReflectSerialization's accept method would look for a "serialization" key of org.apache.hadoop.io.serializer.avro.AvroReflectSerialization. The nice thing about this is that we don't need a list of packages, or even a base type (AvroReflectSerializeable). This would only work if we had the mechanisms in 4 so that the metadata was passed around correctly.

Writables are an existing Serialization, so the implementation is different, since there is plenty of existing data with no extra metadata (in SequenceFiles for instance). So its accept method would check to see if the "serialization" key is set, and if it is, that it is "org.apache.hadoop.io.serializer.WritableSerialization". If not set, it would fall back to the existing check: Writable.class.isAssignableFrom(c).


> Add metadata to Serializations
> ------------------------------
>
>                 Key: HADOOP-6165
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6165
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: contrib/serialization
>            Reporter: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6165.patch
>
>
> The Serialization framework only allows a class to be passed as metadata. This assumes there is a one-to-one mapping between types and Serializations, which is overly restrictive. By permitting applications to pass arbitrary metadata to Serializations, they can get more control over which Serialization is used, and would also allow, for example, one to pass an Avro schema to an Avro Serialization.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-455-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 22 12:10:32 2009
Return-Path: <common-issues-return-455-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12169 invoked from network); 22 Jul 2009 12:10:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jul 2009 12:10:32 -0000
Received: (qmail 31612 invoked by uid 500); 22 Jul 2009 12:11:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31584 invoked by uid 500); 22 Jul 2009 12:11:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31574 invoked by uid 99); 22 Jul 2009 12:11:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 12:11:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 12:11:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C7F1F234C004
	for <common-issues@hadoop.apache.org>; Wed, 22 Jul 2009 05:11:14 -0700 (PDT)
Message-ID: <1579922700.1248264674814.JavaMail.jira@brutus>
Date: Wed, 22 Jul 2009 05:11:14 -0700 (PDT)
From: "Ravi Gummadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6163) Progress class should provide an api
 if phases exist
In-Reply-To: <2055548189.1248066254958.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated HADOOP-6163:
---------------------------------

    Resolution: Invalid
        Status: Resolved  (was: Patch Available)

As we don't have tasks without phases(both map tasks and reduce tasks have phases) for which we need progress, this is not an issue.

> Progress class should provide an api if phases exist
> ----------------------------------------------------
>
>                 Key: HADOOP-6163
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6163
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6163.patch, HADOOP-6163.v1.patch
>
>
> Progress class needs to provide an api for client to know if there are phases.
> This is needed for Task.setProgress() to decide whether to update progress of task or progress of current phase in the task.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-456-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 22 17:36:32 2009
Return-Path: <common-issues-return-456-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56523 invoked from network); 22 Jul 2009 17:36:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jul 2009 17:36:32 -0000
Received: (qmail 16027 invoked by uid 500); 22 Jul 2009 17:37:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15986 invoked by uid 500); 22 Jul 2009 17:37:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15972 invoked by uid 99); 22 Jul 2009 17:37:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 17:37:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 17:37:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D76A529A0014
	for <common-issues@hadoop.apache.org>; Wed, 22 Jul 2009 10:37:14 -0700 (PDT)
Message-ID: <1363337689.1248284234881.JavaMail.jira@brutus>
Date: Wed, 22 Jul 2009 10:37:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6164) Test-patch's method of checking for
 new tests is not correct
In-Reply-To: <1144829364.1248220514920.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734208#action_12734208 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6164:
------------------------------------------------

> BTW hudson didnt run any core-test for HDFS-450
Why it is related to BTW HDFS-450?  Typo?

I think Jakob was talking about [this test results|https://issues.apache.org/jira/browse/HDFS-458?focusedCommentId=12732863&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12732863].  It got +1 on contrib test.

Giridharan, could you take a second look?

> Test-patch's method of checking for new tests is not correct
> ------------------------------------------------------------
>
>                 Key: HADOOP-6164
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6164
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Jakob Homan
>            Assignee: Giridharan Kesavan
>
> As happened in HDFS-458, test-patch/hudson saw the previous patch(HDFS-492) add a new test, bringing the total number of tests to 49.  Then it tested 458, which had no new tests as it was a change to the build file.  492 had not yet been committed yet, so there were still only 48 tests in trunk.  But test-patch failed the patch, expecting 49 tests.  test-patch should check the correct number of tests based on trunk, not on whatever number it last saw.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-457-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 22 17:38:32 2009
Return-Path: <common-issues-return-457-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57659 invoked from network); 22 Jul 2009 17:38:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jul 2009 17:38:31 -0000
Received: (qmail 18988 invoked by uid 500); 22 Jul 2009 17:39:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18944 invoked by uid 500); 22 Jul 2009 17:39:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18931 invoked by uid 99); 22 Jul 2009 17:39:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 17:39:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 17:39:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D047A234C004
	for <common-issues@hadoop.apache.org>; Wed, 22 Jul 2009 10:39:14 -0700 (PDT)
Message-ID: <427061340.1248284354839.JavaMail.jira@brutus>
Date: Wed, 22 Jul 2009 10:39:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6164) Test-patch's method of checking for
 new tests is not correct
In-Reply-To: <1144829364.1248220514920.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734209#action_12734209 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6164:
------------------------------------------------

> Why it is related to BTW HDFS-450? Typo? 
Oops, I got a typo. It should be "Why it is related to HDFS-450? Typo? "

> Test-patch's method of checking for new tests is not correct
> ------------------------------------------------------------
>
>                 Key: HADOOP-6164
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6164
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Jakob Homan
>            Assignee: Giridharan Kesavan
>
> As happened in HDFS-458, test-patch/hudson saw the previous patch(HDFS-492) add a new test, bringing the total number of tests to 49.  Then it tested 458, which had no new tests as it was a change to the build file.  492 had not yet been committed yet, so there were still only 48 tests in trunk.  But test-patch failed the patch, expecting 49 tests.  test-patch should check the correct number of tests based on trunk, not on whatever number it last saw.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-458-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 22 17:40:35 2009
Return-Path: <common-issues-return-458-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58498 invoked from network); 22 Jul 2009 17:40:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jul 2009 17:40:34 -0000
Received: (qmail 20020 invoked by uid 500); 22 Jul 2009 17:41:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19982 invoked by uid 500); 22 Jul 2009 17:41:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19966 invoked by uid 99); 22 Jul 2009 17:41:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 17:41:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 17:41:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id BDAC829A0027
	for <common-issues@hadoop.apache.org>; Wed, 22 Jul 2009 10:41:15 -0700 (PDT)
Message-ID: <830397505.1248284475776.JavaMail.jira@brutus>
Date: Wed, 22 Jul 2009 10:41:15 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6164) Test-patch's method of checking for
 new tests is not correct
In-Reply-To: <1144829364.1248220514920.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734212#action_12734212 ] 

Jakob Homan commented on HADOOP-6164:
-------------------------------------

I've spoken with Giri about this offline.  Hopefully he'll re-open the issue soon.

> Test-patch's method of checking for new tests is not correct
> ------------------------------------------------------------
>
>                 Key: HADOOP-6164
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6164
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Jakob Homan
>            Assignee: Giridharan Kesavan
>
> As happened in HDFS-458, test-patch/hudson saw the previous patch(HDFS-492) add a new test, bringing the total number of tests to 49.  Then it tested 458, which had no new tests as it was a change to the build file.  492 had not yet been committed yet, so there were still only 48 tests in trunk.  But test-patch failed the patch, expecting 49 tests.  test-patch should check the correct number of tests based on trunk, not on whatever number it last saw.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-459-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 22 18:08:34 2009
Return-Path: <common-issues-return-459-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74218 invoked from network); 22 Jul 2009 18:08:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jul 2009 18:08:34 -0000
Received: (qmail 72807 invoked by uid 500); 22 Jul 2009 18:09:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72778 invoked by uid 500); 22 Jul 2009 18:09:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72763 invoked by uid 99); 22 Jul 2009 18:09:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 18:09:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 18:09:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E403629A0016
	for <common-issues@hadoop.apache.org>; Wed, 22 Jul 2009 11:09:14 -0700 (PDT)
Message-ID: <2045430377.1248286154932.JavaMail.jira@brutus>
Date: Wed, 22 Jul 2009 11:09:14 -0700 (PDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6123) hdfs script does not work after
 project split.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734231#action_12734231 ] 

Suresh Srinivas commented on HADOOP-6123:
-----------------------------------------

when is this change going to be committed?

> hdfs script does not work after project split.
> ----------------------------------------------
>
>                 Key: HADOOP-6123
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6123
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>             Fix For: 0.21.0
>
>         Attachments: 6123_v1.patch
>
>
> There are problems running hdfs script from common.
> # Usage message for hdfs does not have "fs" option anymore.
> # There seem to be an undocumented option "dfs", but using it throws NoClassDefFoundError or ClassNotFoundException.
> # Same with other options e.g. name-node.
> May I am missing something here. How do we call ls these days?
> Do we need to move hdfs script to hdfs project.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-460-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 22 18:18:31 2009
Return-Path: <common-issues-return-460-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78210 invoked from network); 22 Jul 2009 18:18:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jul 2009 18:18:31 -0000
Received: (qmail 87169 invoked by uid 500); 22 Jul 2009 18:19:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87122 invoked by uid 500); 22 Jul 2009 18:19:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87111 invoked by uid 99); 22 Jul 2009 18:19:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 18:19:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 18:19:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C3C45234C004
	for <common-issues@hadoop.apache.org>; Wed, 22 Jul 2009 11:19:14 -0700 (PDT)
Message-ID: <1654799633.1248286754797.JavaMail.jira@brutus>
Date: Wed, 22 Jul 2009 11:19:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6123) hdfs script does not work after
 project split.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734232#action_12734232 ] 

Hong Tang commented on HADOOP-6123:
-----------------------------------

You never mark the jira "patch available".

> hdfs script does not work after project split.
> ----------------------------------------------
>
>                 Key: HADOOP-6123
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6123
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>             Fix For: 0.21.0
>
>         Attachments: 6123_v1.patch
>
>
> There are problems running hdfs script from common.
> # Usage message for hdfs does not have "fs" option anymore.
> # There seem to be an undocumented option "dfs", but using it throws NoClassDefFoundError or ClassNotFoundException.
> # Same with other options e.g. name-node.
> May I am missing something here. How do we call ls these days?
> Do we need to move hdfs script to hdfs project.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-461-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 22 18:22:34 2009
Return-Path: <common-issues-return-461-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86412 invoked from network); 22 Jul 2009 18:22:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jul 2009 18:22:34 -0000
Received: (qmail 89903 invoked by uid 500); 22 Jul 2009 18:23:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89853 invoked by uid 500); 22 Jul 2009 18:23:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89590 invoked by uid 99); 22 Jul 2009 18:23:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 18:23:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 18:23:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D6EFA234C004
	for <common-issues@hadoop.apache.org>; Wed, 22 Jul 2009 11:23:14 -0700 (PDT)
Message-ID: <1458244412.1248286994865.JavaMail.jira@brutus>
Date: Wed, 22 Jul 2009 11:23:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6123) hdfs script does not work after
 project split.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6123:
-------------------------------------------

    Assignee: Sharad Agarwal

Hi Sharad, do you think [my previous comment|https://issues.apache.org/jira/browse/HADOOP-6123?focusedCommentId=12728409&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12728409] making sense?

> hdfs script does not work after project split.
> ----------------------------------------------
>
>                 Key: HADOOP-6123
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6123
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Sharad Agarwal
>             Fix For: 0.21.0
>
>         Attachments: 6123_v1.patch
>
>
> There are problems running hdfs script from common.
> # Usage message for hdfs does not have "fs" option anymore.
> # There seem to be an undocumented option "dfs", but using it throws NoClassDefFoundError or ClassNotFoundException.
> # Same with other options e.g. name-node.
> May I am missing something here. How do we call ls these days?
> Do we need to move hdfs script to hdfs project.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-462-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 22 18:38:33 2009
Return-Path: <common-issues-return-462-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95195 invoked from network); 22 Jul 2009 18:38:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jul 2009 18:38:33 -0000
Received: (qmail 9613 invoked by uid 500); 22 Jul 2009 18:39:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9545 invoked by uid 500); 22 Jul 2009 18:39:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9304 invoked by uid 99); 22 Jul 2009 18:39:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 18:39:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 18:39:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id ED381234C1EA
	for <common-issues@hadoop.apache.org>; Wed, 22 Jul 2009 11:39:14 -0700 (PDT)
Message-ID: <384029420.1248287954970.JavaMail.jira@brutus>
Date: Wed, 22 Jul 2009 11:39:14 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6165) Add metadata to Serializations
In-Reply-To: <1308804598.1248261914878.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734242#action_12734242 ] 

Doug Cutting commented on HADOOP-6165:
--------------------------------------

>  is the following sufficient?
> 
> public abstract boolean accept(Map<String, String> metadata); 

I think it is, since this is called once per container, not per object.  In some cases there may not be a more distinguished class than Object and/or the class may not be known.

> Should we have a Metadata class to permit evolution of beyond Map<String, String>?

No, metadata should be trivially serializeable.

> we could have properties to specify extra metadata. Metadata is a map, so something like mapred.mapoutput.{key,value}.metadata.K

Alternately we can have a single key with a complex value, e.g.:
  mapred.mapoutput.metadata.key="a=b,b=c"
We'd have to process escapes if we want values to be able to contain comma and equals.

Or we could extend Configuration to fully support nested maps, e.g., a nested configuration in a value's XML would create a Map value.

Or we could pass these through outside of the Configuration, e.g., in IFile.



> Add metadata to Serializations
> ------------------------------
>
>                 Key: HADOOP-6165
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6165
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: contrib/serialization
>            Reporter: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6165.patch
>
>
> The Serialization framework only allows a class to be passed as metadata. This assumes there is a one-to-one mapping between types and Serializations, which is overly restrictive. By permitting applications to pass arbitrary metadata to Serializations, they can get more control over which Serialization is used, and would also allow, for example, one to pass an Avro schema to an Avro Serialization.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-463-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 22 19:04:38 2009
Return-Path: <common-issues-return-463-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5547 invoked from network); 22 Jul 2009 19:04:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jul 2009 19:04:38 -0000
Received: (qmail 51902 invoked by uid 500); 22 Jul 2009 19:05:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51861 invoked by uid 500); 22 Jul 2009 19:05:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51851 invoked by uid 99); 22 Jul 2009 19:05:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 19:05:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 19:05:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CAFC6234C004
	for <common-issues@hadoop.apache.org>; Wed, 22 Jul 2009 12:05:14 -0700 (PDT)
Message-ID: <1472445229.1248289514817.JavaMail.jira@brutus>
Date: Wed, 22 Jul 2009 12:05:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6124) patchJavacWarnings and
 trunkJavacWarnings are not consistent.
In-Reply-To: <1339799848.1246409207795.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734250#action_12734250 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6124:
------------------------------------------------

Quite a few javac warnings are getting in without being detected.  See MAPREDUCE-716 and MAPREDUCE-743.

Giridharan, if you are not going to work on this soon, please un-assign yourself.  Someone else may want to work on this.

> patchJavacWarnings and trunkJavacWarnings are not consistent.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6124
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6124
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Giridharan Kesavan
>            Priority: Critical
>         Attachments: c6124_20090701.patch
>
>
> The values of patchJavacWarnings and trunkJavacWarnings are not consistent when running test-patch.sh with an empty patch over Common.  HDFS and MapReduce seem not having this problem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-464-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 22 19:30:32 2009
Return-Path: <common-issues-return-464-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11627 invoked from network); 22 Jul 2009 19:30:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jul 2009 19:30:32 -0000
Received: (qmail 90398 invoked by uid 500); 22 Jul 2009 19:31:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90358 invoked by uid 500); 22 Jul 2009 19:31:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90348 invoked by uid 99); 22 Jul 2009 19:31:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 19:31:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 19:31:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 13F0729A0014
	for <common-issues@hadoop.apache.org>; Wed, 22 Jul 2009 12:31:15 -0700 (PDT)
Message-ID: <992574177.1248291075080.JavaMail.jira@brutus>
Date: Wed, 22 Jul 2009 12:31:15 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5318) Poor IO Performance due to AtomicLong
 operations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-5318:
--------------------------------

    Attachment: TestWriteConcurrency.java
                Rplot001.png

Attaching a benchmark and the resulting box plots. This benchmark writes 1M pairs of <ByteWritable, ByteWritable> into a SequenceFile with varying number of threads. The three series are:

green: Before HADOOP-6148's improvement of CRC32
red: With HADOOP-6148's CRC committed (as it is in trunk)
blue: With HADOOP-6148's CRC plus hadoop-5318.txt from this JIRA

The results clearly show that the CRC implementation made the majority of the difference. Adding this patch seems to result in a very slight improvement. Here's the R script I used to generate the plot:

{code}
d.before.crc32 <- read.table(file="results_before_crc32.tsv", header=F)
d.before <- read.table(file="results_before.tsv", header=F)
d.after <- read.table(file="results_after.tsv", header=F)

lims <- c(1000000, 6400000)
log <- "" /* change to "y" to get log scale */

boxplot(V2 ~ V1, d.before.crc32, col="green", ylim=lims, at=(1:8)-0.4, log=log, boxwex=0.2)
boxplot(V2 ~ V1, d.before, col="red", ylim=lims, at=(1:8)-0.2,boxwex=0.2, log=log, add=T)
boxplot(V2 ~ V1, d.after, col="blue", ylim=lims, at=1:8+0.2,boxwex=0.2, log=log,  add=T)
legend(legend=c("before CRC32", "crc, no 5318", "5318+crc"), fill=c("green", "red", "blue"), x="topright",col="black")
{code}

Ran the benchmarks on:

Java(TM) SE Runtime Environment (build 1.6.0_14-b08)
Java HotSpot(TM) 64-Bit Server VM (build 14.0-b16, mixed mode)

Machine is a Nehalem box, dual quad core with hyperthreading (/proc/cpuinfo shows 16 CPUs: Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz)

> Poor IO Performance due to AtomicLong operations
> ------------------------------------------------
>
>                 Key: HADOOP-5318
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5318
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.19.0
>         Environment: 2x quad core xeon linux 64 bit
>            Reporter: Ben Maurer
>            Assignee: Todd Lipcon
>         Attachments: buf.patch, buffer-output.patch, hadoop-5318.txt, hadoop-5318.txt, Rplot001.png, TestWriteConcurrency.java, TestWriteConcurrency.java, TestWriteConcurrency.java
>
>
> The AtomicLong operations in counting file system statistics can cause high levels of contention with multiple threads. This test demonstrates having multiple threads writing to different sequence files:
> {code:java}
> import java.io.IOException;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.io.ByteWritable;
> import org.apache.hadoop.io.SequenceFile;
> import org.apache.hadoop.io.SequenceFile.Writer;
> import org.apache.hadoop.io.SequenceFile.CompressionType;
> public class Test {
> 	public static void main(String[] args) throws IOException {
> 		final Configuration c = new Configuration();
> 		final FileSystem fs = FileSystem.get(c);
> 		
> 		final int NUM = 1000*1000;
> 		for (int i = 0; i < Integer.valueOf(args[0]); i ++) {
> 			final int ii = i;
> 			new Thread(new Runnable() {
> 				@Override
> 				public void run() {
> 					
> 					try {
> 						Writer f = SequenceFile.createWriter(fs, c, new Path("/test/" + ii ), ByteWritable.class, ByteWritable.class, CompressionType.NONE);
> 						ByteWritable v = new ByteWritable();
> 						
> 						long time = System.currentTimeMillis();
> 						for (int i = 0; i < NUM; i ++)
> 							f.append(v, v);
> 						f.close();
> 						long end = System.currentTimeMillis();
> 						
> 						System.out.printf("%d opartions in %d msec. %f/second\n", NUM, end - time, (float)(1000 * NUM)/(end - time));
> 						
> 					} catch (Exception e) {
> 						// TODO Auto-generated catch block
> 						e.printStackTrace();
> 					}
> 					
> 				}
> 			}).start();
> 		}
> 	}
> }
> {code}
> The results of this benchmark are
> {code}
> ==== 1 threads ====
> 1000000 opartions in 1431 msec. 698812.000000/second
> ==== 2 threads ====
> 1000000 opartions in 3001 msec. 333222.250000/second
> 1000000 opartions in 2985 msec. 335008.375000/second
> ==== 3 threads ====
> 1000000 opartions in 4923 msec. 203128.171875/second
> 1000000 opartions in 4924 msec. 203086.921875/second
> 1000000 opartions in 4981 msec. 200762.906250/second
> ==== 4 threads ====
> 1000000 opartions in 6716 msec. 148898.156250/second
> 1000000 opartions in 7048 msec. 141884.218750/second
> 1000000 opartions in 7342 msec. 136202.671875/second
> 1000000 opartions in 7344 msec. 136165.578125/second
> ==== 5 threads ====
> 1000000 opartions in 10366 msec. 96469.226563/second
> 1000000 opartions in 11085 msec. 90212.000000/second
> 1000000 opartions in 11121 msec. 89919.968750/second
> 1000000 opartions in 11464 msec. 87229.585938/second
> 1000000 opartions in 11538 msec. 86670.132813/second
> ==== 6 threads ====
> 1000000 opartions in 16513 msec. 60558.347656/second
> 1000000 opartions in 17704 msec. 56484.410156/second
> 1000000 opartions in 18219 msec. 54887.753906/second
> 1000000 opartions in 18550 msec. 53908.355469/second
> 1000000 opartions in 18605 msec. 53748.992188/second
> 1000000 opartions in 18663 msec. 53581.953125/second
> ==== 7 threads ====
> 1000000 opartions in 22207 msec. 45030.847656/second
> 1000000 opartions in 23275 msec. 42964.554688/second
> 1000000 opartions in 23484 msec. 42582.183594/second
> 1000000 opartions in 24378 msec. 41020.593750/second
> 1000000 opartions in 24425 msec. 40941.656250/second
> 1000000 opartions in 24533 msec. 40761.421875/second
> 1000000 opartions in 24645 msec. 40576.183594/second
> ==== 8 threads ====
> 1000000 opartions in 26375 msec. 37914.691406/second
> 1000000 opartions in 26420 msec. 37850.113281/second
> 1000000 opartions in 26532 msec. 37690.335938/second
> 1000000 opartions in 26670 msec. 37495.312500/second
> 1000000 opartions in 29772 msec. 33588.605469/second
> 1000000 opartions in 29859 msec. 33490.738281/second
> 1000000 opartions in 30098 msec. 33224.800781/second
> 1000000 opartions in 30082 msec. 33242.468750/second
> {code}
> However, if I comment out the file system statistics increments, the benchmark improves to:
> {code}
> ==== 1 threads ====
> 1000000 opartions in 1194 msec. 837520.937500/second
> ==== 2 threads ====
> 1000000 opartions in 1433 msec. 697836.687500/second
> 1000000 opartions in 1433 msec. 697836.687500/second
> ==== 3 threads ====
> 1000000 opartions in 1643 msec. 608642.750000/second
> 1000000 opartions in 1643 msec. 608642.750000/second
> 1000000 opartions in 1639 msec. 610128.125000/second
> ==== 4 threads ====
> 1000000 opartions in 1886 msec. 530222.687500/second
> 1000000 opartions in 1886 msec. 530222.687500/second
> 1000000 opartions in 1886 msec. 530222.687500/second
> 1000000 opartions in 1899 msec. 526592.937500/second
> ==== 5 threads ====
> 1000000 opartions in 2065 msec. 484261.500000/second
> 1000000 opartions in 2066 msec. 484027.093750/second
> 1000000 opartions in 2067 msec. 483792.937500/second
> 1000000 opartions in 2066 msec. 484027.093750/second
> 1000000 opartions in 2066 msec. 484027.093750/second
> ==== 6 threads ====
> 1000000 opartions in 2151 msec. 464900.031250/second
> 1000000 opartions in 2111 msec. 473709.156250/second
> 1000000 opartions in 2153 msec. 464468.187500/second
> 1000000 opartions in 2114 msec. 473036.906250/second
> 1000000 opartions in 2113 msec. 473260.781250/second
> 1000000 opartions in 2112 msec. 473484.843750/second
> ==== 7 threads ====
> 1000000 opartions in 2368 msec. 422297.312500/second
> 1000000 opartions in 2334 msec. 428449.000000/second
> 1000000 opartions in 2332 msec. 428816.468750/second
> 1000000 opartions in 2330 msec. 429184.562500/second
> 1000000 opartions in 2332 msec. 428816.468750/second
> 1000000 opartions in 2375 msec. 421052.625000/second
> 1000000 opartions in 2394 msec. 417710.937500/second
> ==== 8 threads ====
> 1000000 opartions in 2517 msec. 397298.375000/second
> 1000000 opartions in 2538 msec. 394011.031250/second
> 1000000 opartions in 2538 msec. 394011.031250/second
> 1000000 opartions in 2538 msec. 394011.031250/second
> 1000000 opartions in 2539 msec. 393855.843750/second
> 1000000 opartions in 2614 msec. 382555.468750/second
> 1000000 opartions in 2666 msec. 375093.781250/second
> 1000000 opartions in 2701 msec. 370233.250000/second
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-465-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 22 20:51:49 2009
Return-Path: <common-issues-return-465-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32857 invoked from network); 22 Jul 2009 20:51:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jul 2009 20:51:49 -0000
Received: (qmail 91576 invoked by uid 500); 22 Jul 2009 20:27:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91532 invoked by uid 500); 22 Jul 2009 20:27:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91509 invoked by uid 99); 22 Jul 2009 20:27:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 20:27:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 20:27:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0BB9129A0027
	for <common-issues@hadoop.apache.org>; Wed, 22 Jul 2009 13:27:15 -0700 (PDT)
Message-ID: <1958528591.1248294435047.JavaMail.jira@brutus>
Date: Wed, 22 Jul 2009 13:27:15 -0700 (PDT)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5318) Poor IO Performance due to
 AtomicLong operations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734299#action_12734299 ] 

Scott Carey commented on HADOOP-5318:
-------------------------------------

Since the CRC code reaches to near peak throughput (90%) at about 128 or 256  byte chunks, a small buffer may be beneficial.  Copying extra data is cheap for this size (2 to 4 cache lines -- the copy is in L1 cache), and extra delay before the CRC is also not very risky since its in L1 cache very briefly, not sitting off CPU in RAM.  On the other hand, if all clients are buffering by at least this size, it is irrelevant.  Writing tiny chunks however will impact throughput as the CRC per-call overhead adds up.

See HADOOP-6148 for CRC32 benchmark numbers.  Key breakpoints are about 128 to 256 byte chunk size for ~90% peak throughput, and 2k to 4k chunks where peak throughput is reached. 

> Poor IO Performance due to AtomicLong operations
> ------------------------------------------------
>
>                 Key: HADOOP-5318
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5318
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.19.0
>         Environment: 2x quad core xeon linux 64 bit
>            Reporter: Ben Maurer
>            Assignee: Todd Lipcon
>         Attachments: buf.patch, buffer-output.patch, hadoop-5318.txt, hadoop-5318.txt, Rplot001.png, TestWriteConcurrency.java, TestWriteConcurrency.java, TestWriteConcurrency.java
>
>
> The AtomicLong operations in counting file system statistics can cause high levels of contention with multiple threads. This test demonstrates having multiple threads writing to different sequence files:
> {code:java}
> import java.io.IOException;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.io.ByteWritable;
> import org.apache.hadoop.io.SequenceFile;
> import org.apache.hadoop.io.SequenceFile.Writer;
> import org.apache.hadoop.io.SequenceFile.CompressionType;
> public class Test {
> 	public static void main(String[] args) throws IOException {
> 		final Configuration c = new Configuration();
> 		final FileSystem fs = FileSystem.get(c);
> 		
> 		final int NUM = 1000*1000;
> 		for (int i = 0; i < Integer.valueOf(args[0]); i ++) {
> 			final int ii = i;
> 			new Thread(new Runnable() {
> 				@Override
> 				public void run() {
> 					
> 					try {
> 						Writer f = SequenceFile.createWriter(fs, c, new Path("/test/" + ii ), ByteWritable.class, ByteWritable.class, CompressionType.NONE);
> 						ByteWritable v = new ByteWritable();
> 						
> 						long time = System.currentTimeMillis();
> 						for (int i = 0; i < NUM; i ++)
> 							f.append(v, v);
> 						f.close();
> 						long end = System.currentTimeMillis();
> 						
> 						System.out.printf("%d opartions in %d msec. %f/second\n", NUM, end - time, (float)(1000 * NUM)/(end - time));
> 						
> 					} catch (Exception e) {
> 						// TODO Auto-generated catch block
> 						e.printStackTrace();
> 					}
> 					
> 				}
> 			}).start();
> 		}
> 	}
> }
> {code}
> The results of this benchmark are
> {code}
> ==== 1 threads ====
> 1000000 opartions in 1431 msec. 698812.000000/second
> ==== 2 threads ====
> 1000000 opartions in 3001 msec. 333222.250000/second
> 1000000 opartions in 2985 msec. 335008.375000/second
> ==== 3 threads ====
> 1000000 opartions in 4923 msec. 203128.171875/second
> 1000000 opartions in 4924 msec. 203086.921875/second
> 1000000 opartions in 4981 msec. 200762.906250/second
> ==== 4 threads ====
> 1000000 opartions in 6716 msec. 148898.156250/second
> 1000000 opartions in 7048 msec. 141884.218750/second
> 1000000 opartions in 7342 msec. 136202.671875/second
> 1000000 opartions in 7344 msec. 136165.578125/second
> ==== 5 threads ====
> 1000000 opartions in 10366 msec. 96469.226563/second
> 1000000 opartions in 11085 msec. 90212.000000/second
> 1000000 opartions in 11121 msec. 89919.968750/second
> 1000000 opartions in 11464 msec. 87229.585938/second
> 1000000 opartions in 11538 msec. 86670.132813/second
> ==== 6 threads ====
> 1000000 opartions in 16513 msec. 60558.347656/second
> 1000000 opartions in 17704 msec. 56484.410156/second
> 1000000 opartions in 18219 msec. 54887.753906/second
> 1000000 opartions in 18550 msec. 53908.355469/second
> 1000000 opartions in 18605 msec. 53748.992188/second
> 1000000 opartions in 18663 msec. 53581.953125/second
> ==== 7 threads ====
> 1000000 opartions in 22207 msec. 45030.847656/second
> 1000000 opartions in 23275 msec. 42964.554688/second
> 1000000 opartions in 23484 msec. 42582.183594/second
> 1000000 opartions in 24378 msec. 41020.593750/second
> 1000000 opartions in 24425 msec. 40941.656250/second
> 1000000 opartions in 24533 msec. 40761.421875/second
> 1000000 opartions in 24645 msec. 40576.183594/second
> ==== 8 threads ====
> 1000000 opartions in 26375 msec. 37914.691406/second
> 1000000 opartions in 26420 msec. 37850.113281/second
> 1000000 opartions in 26532 msec. 37690.335938/second
> 1000000 opartions in 26670 msec. 37495.312500/second
> 1000000 opartions in 29772 msec. 33588.605469/second
> 1000000 opartions in 29859 msec. 33490.738281/second
> 1000000 opartions in 30098 msec. 33224.800781/second
> 1000000 opartions in 30082 msec. 33242.468750/second
> {code}
> However, if I comment out the file system statistics increments, the benchmark improves to:
> {code}
> ==== 1 threads ====
> 1000000 opartions in 1194 msec. 837520.937500/second
> ==== 2 threads ====
> 1000000 opartions in 1433 msec. 697836.687500/second
> 1000000 opartions in 1433 msec. 697836.687500/second
> ==== 3 threads ====
> 1000000 opartions in 1643 msec. 608642.750000/second
> 1000000 opartions in 1643 msec. 608642.750000/second
> 1000000 opartions in 1639 msec. 610128.125000/second
> ==== 4 threads ====
> 1000000 opartions in 1886 msec. 530222.687500/second
> 1000000 opartions in 1886 msec. 530222.687500/second
> 1000000 opartions in 1886 msec. 530222.687500/second
> 1000000 opartions in 1899 msec. 526592.937500/second
> ==== 5 threads ====
> 1000000 opartions in 2065 msec. 484261.500000/second
> 1000000 opartions in 2066 msec. 484027.093750/second
> 1000000 opartions in 2067 msec. 483792.937500/second
> 1000000 opartions in 2066 msec. 484027.093750/second
> 1000000 opartions in 2066 msec. 484027.093750/second
> ==== 6 threads ====
> 1000000 opartions in 2151 msec. 464900.031250/second
> 1000000 opartions in 2111 msec. 473709.156250/second
> 1000000 opartions in 2153 msec. 464468.187500/second
> 1000000 opartions in 2114 msec. 473036.906250/second
> 1000000 opartions in 2113 msec. 473260.781250/second
> 1000000 opartions in 2112 msec. 473484.843750/second
> ==== 7 threads ====
> 1000000 opartions in 2368 msec. 422297.312500/second
> 1000000 opartions in 2334 msec. 428449.000000/second
> 1000000 opartions in 2332 msec. 428816.468750/second
> 1000000 opartions in 2330 msec. 429184.562500/second
> 1000000 opartions in 2332 msec. 428816.468750/second
> 1000000 opartions in 2375 msec. 421052.625000/second
> 1000000 opartions in 2394 msec. 417710.937500/second
> ==== 8 threads ====
> 1000000 opartions in 2517 msec. 397298.375000/second
> 1000000 opartions in 2538 msec. 394011.031250/second
> 1000000 opartions in 2538 msec. 394011.031250/second
> 1000000 opartions in 2538 msec. 394011.031250/second
> 1000000 opartions in 2539 msec. 393855.843750/second
> 1000000 opartions in 2614 msec. 382555.468750/second
> 1000000 opartions in 2666 msec. 375093.781250/second
> 1000000 opartions in 2701 msec. 370233.250000/second
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-466-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 22 21:08:33 2009
Return-Path: <common-issues-return-466-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43823 invoked from network); 22 Jul 2009 21:08:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Jul 2009 21:08:32 -0000
Received: (qmail 72755 invoked by uid 500); 22 Jul 2009 21:09:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72699 invoked by uid 500); 22 Jul 2009 21:09:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72542 invoked by uid 99); 22 Jul 2009 21:09:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 21:09:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jul 2009 21:09:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 10F3C234C053
	for <common-issues@hadoop.apache.org>; Wed, 22 Jul 2009 14:09:15 -0700 (PDT)
Message-ID: <517792960.1248296955068.JavaMail.jira@brutus>
Date: Wed, 22 Jul 2009 14:09:15 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5318) Poor IO Performance due to
 AtomicLong operations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734321#action_12734321 ] 

Todd Lipcon commented on HADOOP-5318:
-------------------------------------

Scott: I think if we add buffering it should be at the "client" level - eg in SequenceFileOutputStream or anywhere else that very small records will be written at a high rate. Doing so in the FSDataOutputStream seems heavy-handed, since we don't know what size the writes will usually be, and I'd imagine the usual is much larger than this pathological test case we used for the benchmark.

I'm also not sure I see your point about the L1 cache - in between the small writes there may be arbitrary amounts of computation (eg in the mapper function) that would evict the buffer from cache.

> Poor IO Performance due to AtomicLong operations
> ------------------------------------------------
>
>                 Key: HADOOP-5318
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5318
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.19.0
>         Environment: 2x quad core xeon linux 64 bit
>            Reporter: Ben Maurer
>            Assignee: Todd Lipcon
>         Attachments: buf.patch, buffer-output.patch, hadoop-5318.txt, hadoop-5318.txt, Rplot001.png, TestWriteConcurrency.java, TestWriteConcurrency.java, TestWriteConcurrency.java
>
>
> The AtomicLong operations in counting file system statistics can cause high levels of contention with multiple threads. This test demonstrates having multiple threads writing to different sequence files:
> {code:java}
> import java.io.IOException;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.io.ByteWritable;
> import org.apache.hadoop.io.SequenceFile;
> import org.apache.hadoop.io.SequenceFile.Writer;
> import org.apache.hadoop.io.SequenceFile.CompressionType;
> public class Test {
> 	public static void main(String[] args) throws IOException {
> 		final Configuration c = new Configuration();
> 		final FileSystem fs = FileSystem.get(c);
> 		
> 		final int NUM = 1000*1000;
> 		for (int i = 0; i < Integer.valueOf(args[0]); i ++) {
> 			final int ii = i;
> 			new Thread(new Runnable() {
> 				@Override
> 				public void run() {
> 					
> 					try {
> 						Writer f = SequenceFile.createWriter(fs, c, new Path("/test/" + ii ), ByteWritable.class, ByteWritable.class, CompressionType.NONE);
> 						ByteWritable v = new ByteWritable();
> 						
> 						long time = System.currentTimeMillis();
> 						for (int i = 0; i < NUM; i ++)
> 							f.append(v, v);
> 						f.close();
> 						long end = System.currentTimeMillis();
> 						
> 						System.out.printf("%d opartions in %d msec. %f/second\n", NUM, end - time, (float)(1000 * NUM)/(end - time));
> 						
> 					} catch (Exception e) {
> 						// TODO Auto-generated catch block
> 						e.printStackTrace();
> 					}
> 					
> 				}
> 			}).start();
> 		}
> 	}
> }
> {code}
> The results of this benchmark are
> {code}
> ==== 1 threads ====
> 1000000 opartions in 1431 msec. 698812.000000/second
> ==== 2 threads ====
> 1000000 opartions in 3001 msec. 333222.250000/second
> 1000000 opartions in 2985 msec. 335008.375000/second
> ==== 3 threads ====
> 1000000 opartions in 4923 msec. 203128.171875/second
> 1000000 opartions in 4924 msec. 203086.921875/second
> 1000000 opartions in 4981 msec. 200762.906250/second
> ==== 4 threads ====
> 1000000 opartions in 6716 msec. 148898.156250/second
> 1000000 opartions in 7048 msec. 141884.218750/second
> 1000000 opartions in 7342 msec. 136202.671875/second
> 1000000 opartions in 7344 msec. 136165.578125/second
> ==== 5 threads ====
> 1000000 opartions in 10366 msec. 96469.226563/second
> 1000000 opartions in 11085 msec. 90212.000000/second
> 1000000 opartions in 11121 msec. 89919.968750/second
> 1000000 opartions in 11464 msec. 87229.585938/second
> 1000000 opartions in 11538 msec. 86670.132813/second
> ==== 6 threads ====
> 1000000 opartions in 16513 msec. 60558.347656/second
> 1000000 opartions in 17704 msec. 56484.410156/second
> 1000000 opartions in 18219 msec. 54887.753906/second
> 1000000 opartions in 18550 msec. 53908.355469/second
> 1000000 opartions in 18605 msec. 53748.992188/second
> 1000000 opartions in 18663 msec. 53581.953125/second
> ==== 7 threads ====
> 1000000 opartions in 22207 msec. 45030.847656/second
> 1000000 opartions in 23275 msec. 42964.554688/second
> 1000000 opartions in 23484 msec. 42582.183594/second
> 1000000 opartions in 24378 msec. 41020.593750/second
> 1000000 opartions in 24425 msec. 40941.656250/second
> 1000000 opartions in 24533 msec. 40761.421875/second
> 1000000 opartions in 24645 msec. 40576.183594/second
> ==== 8 threads ====
> 1000000 opartions in 26375 msec. 37914.691406/second
> 1000000 opartions in 26420 msec. 37850.113281/second
> 1000000 opartions in 26532 msec. 37690.335938/second
> 1000000 opartions in 26670 msec. 37495.312500/second
> 1000000 opartions in 29772 msec. 33588.605469/second
> 1000000 opartions in 29859 msec. 33490.738281/second
> 1000000 opartions in 30098 msec. 33224.800781/second
> 1000000 opartions in 30082 msec. 33242.468750/second
> {code}
> However, if I comment out the file system statistics increments, the benchmark improves to:
> {code}
> ==== 1 threads ====
> 1000000 opartions in 1194 msec. 837520.937500/second
> ==== 2 threads ====
> 1000000 opartions in 1433 msec. 697836.687500/second
> 1000000 opartions in 1433 msec. 697836.687500/second
> ==== 3 threads ====
> 1000000 opartions in 1643 msec. 608642.750000/second
> 1000000 opartions in 1643 msec. 608642.750000/second
> 1000000 opartions in 1639 msec. 610128.125000/second
> ==== 4 threads ====
> 1000000 opartions in 1886 msec. 530222.687500/second
> 1000000 opartions in 1886 msec. 530222.687500/second
> 1000000 opartions in 1886 msec. 530222.687500/second
> 1000000 opartions in 1899 msec. 526592.937500/second
> ==== 5 threads ====
> 1000000 opartions in 2065 msec. 484261.500000/second
> 1000000 opartions in 2066 msec. 484027.093750/second
> 1000000 opartions in 2067 msec. 483792.937500/second
> 1000000 opartions in 2066 msec. 484027.093750/second
> 1000000 opartions in 2066 msec. 484027.093750/second
> ==== 6 threads ====
> 1000000 opartions in 2151 msec. 464900.031250/second
> 1000000 opartions in 2111 msec. 473709.156250/second
> 1000000 opartions in 2153 msec. 464468.187500/second
> 1000000 opartions in 2114 msec. 473036.906250/second
> 1000000 opartions in 2113 msec. 473260.781250/second
> 1000000 opartions in 2112 msec. 473484.843750/second
> ==== 7 threads ====
> 1000000 opartions in 2368 msec. 422297.312500/second
> 1000000 opartions in 2334 msec. 428449.000000/second
> 1000000 opartions in 2332 msec. 428816.468750/second
> 1000000 opartions in 2330 msec. 429184.562500/second
> 1000000 opartions in 2332 msec. 428816.468750/second
> 1000000 opartions in 2375 msec. 421052.625000/second
> 1000000 opartions in 2394 msec. 417710.937500/second
> ==== 8 threads ====
> 1000000 opartions in 2517 msec. 397298.375000/second
> 1000000 opartions in 2538 msec. 394011.031250/second
> 1000000 opartions in 2538 msec. 394011.031250/second
> 1000000 opartions in 2538 msec. 394011.031250/second
> 1000000 opartions in 2539 msec. 393855.843750/second
> 1000000 opartions in 2614 msec. 382555.468750/second
> 1000000 opartions in 2666 msec. 375093.781250/second
> 1000000 opartions in 2701 msec. 370233.250000/second
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-467-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 01:40:31 2009
Return-Path: <common-issues-return-467-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37085 invoked from network); 23 Jul 2009 01:40:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 01:40:31 -0000
Received: (qmail 82362 invoked by uid 500); 23 Jul 2009 01:41:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82333 invoked by uid 500); 23 Jul 2009 01:41:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82323 invoked by uid 99); 23 Jul 2009 01:41:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 01:41:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 01:41:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CCDBA29A0014
	for <common-issues@hadoop.apache.org>; Wed, 22 Jul 2009 18:41:14 -0700 (PDT)
Message-ID: <1460486577.1248313274837.JavaMail.jira@brutus>
Date: Wed, 22 Jul 2009 18:41:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6166) Improve PureJavaCrc32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Improve PureJavaCrc32
---------------------

                 Key: HADOOP-6166
                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
             Project: Hadoop Common
          Issue Type: Improvement
          Components: util
            Reporter: Tsz Wo (Nicholas), SZE
            Assignee: Tsz Wo (Nicholas), SZE


Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-468-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 01:44:32 2009
Return-Path: <common-issues-return-468-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38615 invoked from network); 23 Jul 2009 01:44:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 01:44:31 -0000
Received: (qmail 83966 invoked by uid 500); 23 Jul 2009 01:45:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83922 invoked by uid 500); 23 Jul 2009 01:45:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83859 invoked by uid 99); 23 Jul 2009 01:45:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 01:45:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 01:45:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D4E9B234C004
	for <common-issues@hadoop.apache.org>; Wed, 22 Jul 2009 18:45:14 -0700 (PDT)
Message-ID: <1684083349.1248313514857.JavaMail.jira@brutus>
Date: Wed, 22 Jul 2009 18:45:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6166:
-------------------------------------------

    Attachment: c6166_20090722.patch

c6166_20090722.patch: Tried a few CRC32 implementations.

> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-469-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 01:52:36 2009
Return-Path: <common-issues-return-469-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42257 invoked from network); 23 Jul 2009 01:52:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 01:52:36 -0000
Received: (qmail 88301 invoked by uid 500); 23 Jul 2009 01:53:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88215 invoked by uid 500); 23 Jul 2009 01:53:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87859 invoked by uid 99); 23 Jul 2009 01:53:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 01:53:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 01:53:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 45A2C29A0017
	for <common-issues@hadoop.apache.org>; Wed, 22 Jul 2009 18:53:15 -0700 (PDT)
Message-ID: <128618595.1248313995284.JavaMail.jira@brutus>
Date: Wed, 22 Jul 2009 18:53:15 -0700 (PDT)
From: "Fernando (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6167) bin/hadoop script doesn't allow for
 different memory settings for each daemon type
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

bin/hadoop script doesn't allow for different memory settings for each daemon type
----------------------------------------------------------------------------------

                 Key: HADOOP-6167
                 URL: https://issues.apache.org/jira/browse/HADOOP-6167
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.20.0
            Reporter: Fernando


bin/hadoop assumes that all daemon types ( namenode, datanode, jobtracker, tasktracker ), all use the same memory settings.. (HADOOP_HEAPSIZE).

I propose changes to that script to allow overriding the default memory ( HADOOP_HEAPSIZE ), with daemon specific OPTS (HADOOP_NAMENODE_OPTS, etc ).

Basically at the bottom of the bin/hadoop script, it will check to see if the user has already set "-Xmx" in the HADOOP_OPTS variable.. if so, then it will ignore the JAVA_HEAP_SIZE variable..

as such:

# run it
if [[ $HADOOP_OPTS == *-Xmx* ]]; then
  exec "$JAVA" $HADOOP_OPTS -classpath "$CLASSPATH" $CLASS "$@"
else
  exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS -classpath "$CLASSPATH" $CLASS "$@"
fi


I will attach the file as I have modified it..

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-470-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 01:54:32 2009
Return-Path: <common-issues-return-470-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43040 invoked from network); 23 Jul 2009 01:54:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 01:54:32 -0000
Received: (qmail 90527 invoked by uid 500); 23 Jul 2009 01:55:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90463 invoked by uid 500); 23 Jul 2009 01:55:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90205 invoked by uid 99); 23 Jul 2009 01:55:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 01:55:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 01:55:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DF86729A0015
	for <common-issues@hadoop.apache.org>; Wed, 22 Jul 2009 18:55:14 -0700 (PDT)
Message-ID: <1940428464.1248314114914.JavaMail.jira@brutus>
Date: Wed, 22 Jul 2009 18:55:14 -0700 (PDT)
From: "Fernando (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6167) bin/hadoop script doesn't allow for
 different memory settings for each daemon type
In-Reply-To: <128618595.1248313995284.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Fernando updated HADOOP-6167:
-----------------------------

    Attachment: hadoop

proposed new bin/hadoop script file.

The only change is in the last few lines.

> bin/hadoop script doesn't allow for different memory settings for each daemon type
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6167
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.0
>            Reporter: Fernando
>         Attachments: hadoop
>
>
> bin/hadoop assumes that all daemon types ( namenode, datanode, jobtracker, tasktracker ), all use the same memory settings.. (HADOOP_HEAPSIZE).
> I propose changes to that script to allow overriding the default memory ( HADOOP_HEAPSIZE ), with daemon specific OPTS (HADOOP_NAMENODE_OPTS, etc ).
> Basically at the bottom of the bin/hadoop script, it will check to see if the user has already set "-Xmx" in the HADOOP_OPTS variable.. if so, then it will ignore the JAVA_HEAP_SIZE variable..
> as such:
> # run it
> if [[ $HADOOP_OPTS == *-Xmx* ]]; then
>   exec "$JAVA" $HADOOP_OPTS -classpath "$CLASSPATH" $CLASS "$@"
> else
>   exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS -classpath "$CLASSPATH" $CLASS "$@"
> fi
> I will attach the file as I have modified it..

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-471-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 01:54:33 2009
Return-Path: <common-issues-return-471-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43151 invoked from network); 23 Jul 2009 01:54:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 01:54:33 -0000
Received: (qmail 90829 invoked by uid 500); 23 Jul 2009 01:55:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90785 invoked by uid 500); 23 Jul 2009 01:55:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90775 invoked by uid 99); 23 Jul 2009 01:55:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 01:55:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 01:55:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CD82F234C004
	for <common-issues@hadoop.apache.org>; Wed, 22 Jul 2009 18:55:14 -0700 (PDT)
Message-ID: <1243249652.1248314114827.JavaMail.jira@brutus>
Date: Wed, 22 Jul 2009 18:55:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6166:
-------------------------------------------

    Attachment: c6166_20090722_benchmark_32VM.txt
                c6166_20090722_benchmark_64VM.txt

c6166_20090722_benchmark_64VM.txt, c6166_20090722_benchmark_32VM.txt: Tested the implementations on both 32-bit and 64-bit VMs.  Seems that Crc32_4_3 are faster than current PureJavaCrc32 in both cases.

(Sorry that I still have not tested with TestPureJavaCrc32.PerformanceTest.)

> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-472-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 03:36:42 2009
Return-Path: <common-issues-return-472-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70495 invoked from network); 23 Jul 2009 03:36:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 03:36:42 -0000
Received: (qmail 73455 invoked by uid 500); 23 Jul 2009 03:37:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73413 invoked by uid 500); 23 Jul 2009 03:37:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73403 invoked by uid 99); 23 Jul 2009 03:37:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 03:37:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 03:37:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 40305234C004
	for <common-issues@hadoop.apache.org>; Wed, 22 Jul 2009 20:37:15 -0700 (PDT)
Message-ID: <1938396138.1248320235248.JavaMail.jira@brutus>
Date: Wed, 22 Jul 2009 20:37:15 -0700 (PDT)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734445#action_12734445 ] 

Scott Carey commented on HADOOP-6166:
-------------------------------------

One could take Intel's idea, and go 8 (or 6 or 12?) bytes at a time instead of 4. This would line up with the 12 bit lookup tables a bit better.  6 bytes might be the easier boundary, and require 4 12 bit lookup tables. which is 32K, the size of the tables in the inner loop in your "4_3" are 17K, and the current version is 4K.

Going 12 bytes at a time would require 8 tables and 64K of space, and then we're randomly jumping around lookup tables that don't fit in a L1 D-cache on some processors.

The 8 bytes at a time approach is in C code (BSD license) here:
http://sourceforge.net/projects/slicing-by-8/files/

The trick with going over 4 bytes in a loop has to do with how the cycle works on the CRC.  I think, after the first four bytes of lookups, it changes a bit.
But I didn't read that much into that code to be sure of what it is doing.  They do it with 8 1K lookup tables each of one byte indexes.  They also get to directly Xor out of the byte array a single int, rather than grabbing one byte at a time and shifting.  We can do that with a ByteBuffer with getInt(), but when I tried that (with getInt) in the previous case the byte buffer creation overhead was too large, and the ByteBuffer access seemed very inefficient for some reason.  Oh how nice it would be if you could grab the next 4 bytes in a byte[] as an Int (or the next 8 as a Long) without wrapping it in a ByteBuffer, and let the compiler figure out the optimal processor load instruction.

I think a 6 byte at a time, 4 lookup of 12 bit LUT would be like this:

{code}
public class Crc32_6_4 extends Crc32Base {
  /** {@inheritDoc} */
  public void update(byte[] b, int off, int len) {
    while(len > 3) {
      crc ^= b[off++] & 0xff;
      crc ^= (b[off++] & 0xff) << 8;
      crc ^= (b[off++] & 0xff) << 16;
      crc ^= (b[off++] & 0xff) << 24;
      int c0 = b[off++] & 0xff;;
      c0 ^= (b[off++] & 0xff) << 8;

      crc = T12_3[crc & 0xfff] ^ T12_2[(crc >>> 12) & 0xfff] ^ T12_1[((crc >>> 24) & (c0 << 8)) & 0xfff] ^ T12_0[c0 >> 4];
      len -= 6;
    }

    for (; len > 0; len--) {
      crc = (crc >>> 8) ^ Table8_0[(crc ^ b[off++]) & 0xff];
    }
  }
{code}

Assuming the tables were built right and the "wrap past 4 bytes means don't xor with crc" is correct.  I haven't tried this at all.

All of these with larger lookup tables run the risk of performing worse under concurrency, even if faster single threaded.  Cache pressure is greater under concurrency.  We might want to use the benchmark in HADOOP-5318, which is heavily CRC reliant, as a check to make sure we aren't regressing under higher concurrency due to cache pressure.


For reference, Intel's C code (referenced above, snippet below) with 8 tables looks like this in the inner loop:

{code}
/*++
 *
 * Copyright (c) 2004-2006 Intel Corporation - All Rights Reserved
 *
 * This software program is licensed subject to the BSD License, 
 * available at http://www.opensource.org/licenses/bsd-license.html
 *
 * Abstract: The main routine
 * 
 --*/
crc32c_sb8_64_bit(
	uint32_t* p_running_crc,
    const uint8_t*	p_buf,
    const uint32_t length,
	const uint32_t init_bytes,
	uint8_t			mode)
{
	uint32_t li;
	uint32_t crc, term1, term2;
	uint32_t running_length;
	uint32_t end_bytes;
	if(mode ==  MODE_CONT)
		crc = *p_running_crc;
	else	
		crc = CRC32C_INIT_REFLECTED;
	running_length = ((length - init_bytes)/8)*8;
	end_bytes = length - init_bytes - running_length; 

	for(li=0; li < init_bytes; li++) 
		crc = crc_tableil8_o32[(crc ^ *p_buf++) & 0x000000FF] ^ (crc >> 8);	
	for(li=0; li < running_length/8; li++) 
	{
		crc ^= *(uint32_t *)p_buf;
		p_buf += 4;
		term1 = crc_tableil8_o88[crc & 0x000000FF] ^
				crc_tableil8_o80[(crc >> 8) & 0x000000FF];
		term2 = crc >> 16;
		crc = term1 ^
			  crc_tableil8_o72[term2 & 0x000000FF] ^ 
			  crc_tableil8_o64[(term2 >> 8) & 0x000000FF];
		term1 = crc_tableil8_o56[(*(uint32_t *)p_buf) & 0x000000FF] ^
				crc_tableil8_o48[((*(uint32_t *)p_buf) >> 8) & 0x000000FF];
		
		term2 = (*(uint32_t *)p_buf) >> 16;
		crc =	crc ^ 
				term1 ^		
				crc_tableil8_o40[term2  & 0x000000FF] ^	
				crc_tableil8_o32[(term2 >> 8) & 0x000000FF];	
		p_buf += 4;
	}
	for(li=0; li < end_bytes; li++) 
		crc = crc_tableil8_o32[(crc ^ *p_buf++) & 0x000000FF] ^ (crc >> 8);
	if((mode == MODE_BEGIN) || (mode ==  MODE_CONT))
		return crc;		
    return crc ^ XOROT;	
}
{code}

That is pretty straightforward to implement if the next 4 tables (T5, T6, T7, T8 in the terminology of the current PureJavaCRC32) are generated the same way as the previous 4.
Intel makes sure the inner loop is on an 8 byte boundary, because the C compiler can make the load needed for the
{code}crc ^= *(uint32_t *)p_buf{code} part faster if that is the case.  They also tend to favor shifting by 16 and 8 and avoiding shifting by 24 for some reason.

I may try out the eight bytes at once with 8 lookup tables version next week.

> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-473-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 04:12:34 2009
Return-Path: <common-issues-return-473-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75047 invoked from network); 23 Jul 2009 04:12:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 04:12:34 -0000
Received: (qmail 99855 invoked by uid 500); 23 Jul 2009 04:13:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99678 invoked by uid 500); 23 Jul 2009 04:13:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99668 invoked by uid 99); 23 Jul 2009 04:13:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 04:13:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 04:13:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 17D3129A0015
	for <common-issues@hadoop.apache.org>; Wed, 22 Jul 2009 21:13:15 -0700 (PDT)
Message-ID: <226879173.1248322395096.JavaMail.jira@brutus>
Date: Wed, 22 Jul 2009 21:13:15 -0700 (PDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6123) hdfs script does not work after
 project split.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734452#action_12734452 ] 

Sharad Agarwal commented on HADOOP-6123:
----------------------------------------

bq. do you think my previous comment making sense?
I don't think the "for loop" can be replaced by using the wildcard. It won't expand the path in CLASSPATH.

> hdfs script does not work after project split.
> ----------------------------------------------
>
>                 Key: HADOOP-6123
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6123
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Sharad Agarwal
>             Fix For: 0.21.0
>
>         Attachments: 6123_v1.patch
>
>
> There are problems running hdfs script from common.
> # Usage message for hdfs does not have "fs" option anymore.
> # There seem to be an undocumented option "dfs", but using it throws NoClassDefFoundError or ClassNotFoundException.
> # Same with other options e.g. name-node.
> May I am missing something here. How do we call ls these days?
> Do we need to move hdfs script to hdfs project.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-474-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 04:14:32 2009
Return-Path: <common-issues-return-474-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75307 invoked from network); 23 Jul 2009 04:14:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 04:14:32 -0000
Received: (qmail 1243 invoked by uid 500); 23 Jul 2009 04:15:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1192 invoked by uid 500); 23 Jul 2009 04:15:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1178 invoked by uid 99); 23 Jul 2009 04:15:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 04:15:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 04:15:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 48BE7234C004
	for <common-issues@hadoop.apache.org>; Wed, 22 Jul 2009 21:15:15 -0700 (PDT)
Message-ID: <571503760.1248322515283.JavaMail.jira@brutus>
Date: Wed, 22 Jul 2009 21:15:15 -0700 (PDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6123) hdfs script does not work after
 project split.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sharad Agarwal updated HADOOP-6123:
-----------------------------------

    Status: Patch Available  (was: Open)

> hdfs script does not work after project split.
> ----------------------------------------------
>
>                 Key: HADOOP-6123
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6123
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Sharad Agarwal
>             Fix For: 0.21.0
>
>         Attachments: 6123_v1.patch
>
>
> There are problems running hdfs script from common.
> # Usage message for hdfs does not have "fs" option anymore.
> # There seem to be an undocumented option "dfs", but using it throws NoClassDefFoundError or ClassNotFoundException.
> # Same with other options e.g. name-node.
> May I am missing something here. How do we call ls these days?
> Do we need to move hdfs script to hdfs project.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-475-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 04:26:32 2009
Return-Path: <common-issues-return-475-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76404 invoked from network); 23 Jul 2009 04:26:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 04:26:32 -0000
Received: (qmail 6524 invoked by uid 500); 23 Jul 2009 04:27:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6480 invoked by uid 500); 23 Jul 2009 04:27:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6470 invoked by uid 99); 23 Jul 2009 04:27:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 04:27:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 04:27:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E0C7C29A0016
	for <common-issues@hadoop.apache.org>; Wed, 22 Jul 2009 21:27:14 -0700 (PDT)
Message-ID: <719917983.1248323234919.JavaMail.jira@brutus>
Date: Wed, 22 Jul 2009 21:27:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6164) Test-patch's method of checking for
 new tests is not correct
In-Reply-To: <1144829364.1248220514920.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734456#action_12734456 ] 

Giridharan Kesavan commented on HADOOP-6164:
--------------------------------------------

my bad, I mistook hdfs-458 for hdfs-450 and made the inference.

But looking the core-test console log:
http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/25/console

     [exec]     [junit] Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 3.745 sec
     [exec]     [junit] Test org.apache.hadoop.hdfs.server.datanode.TestInterDatanodeProtocol FAILED

TestInterDatanodeProtocol test failed for some reason; and hudson while parsing the xml results didn't consider the test failure.

since there is a failure in the test-core ant target test-patch script gave a -1 for this patch for core test.

I'm not sure why hudson didn't consider this failure while generating the test report.
I closed this invalid b'coz I pretty much know test-patch doesn't consider the no of tests being run by previous and current hudson build jobs.

http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/25/testReport/org.apache.hadoop.hdfs.server.datanode/TestInterDatanodeProtocol/

If you think I'm missing something please reopen this jira.

Thanks,

> Test-patch's method of checking for new tests is not correct
> ------------------------------------------------------------
>
>                 Key: HADOOP-6164
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6164
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Jakob Homan
>            Assignee: Giridharan Kesavan
>
> As happened in HDFS-458, test-patch/hudson saw the previous patch(HDFS-492) add a new test, bringing the total number of tests to 49.  Then it tested 458, which had no new tests as it was a change to the build file.  492 had not yet been committed yet, so there were still only 48 tests in trunk.  But test-patch failed the patch, expecting 49 tests.  test-patch should check the correct number of tests based on trunk, not on whatever number it last saw.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-476-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 06:52:38 2009
Return-Path: <common-issues-return-476-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40697 invoked from network); 23 Jul 2009 06:52:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 06:52:38 -0000
Received: (qmail 50874 invoked by uid 500); 23 Jul 2009 06:53:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50830 invoked by uid 500); 23 Jul 2009 06:53:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50820 invoked by uid 99); 23 Jul 2009 06:53:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 06:53:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 06:53:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D78D229A0015
	for <common-issues@hadoop.apache.org>; Wed, 22 Jul 2009 23:53:14 -0700 (PDT)
Message-ID: <1342936440.1248331994882.JavaMail.jira@brutus>
Date: Wed, 22 Jul 2009 23:53:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6124) patchJavacWarnings and
 trunkJavacWarnings are not consistent.
In-Reply-To: <1339799848.1246409207795.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6124:
---------------------------------------

    Attachment: HADOOP-6124.patch

this patch adds the clean target before calling the tar while searching for javac warnings and also address the issue mentioned as part of HADOOP-6077.

> patchJavacWarnings and trunkJavacWarnings are not consistent.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6124
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6124
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Giridharan Kesavan
>            Priority: Critical
>         Attachments: c6124_20090701.patch, HADOOP-6124.patch
>
>
> The values of patchJavacWarnings and trunkJavacWarnings are not consistent when running test-patch.sh with an empty patch over Common.  HDFS and MapReduce seem not having this problem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-477-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 08:04:32 2009
Return-Path: <common-issues-return-477-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55548 invoked from network); 23 Jul 2009 08:04:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 08:04:32 -0000
Received: (qmail 12864 invoked by uid 500); 23 Jul 2009 08:05:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12829 invoked by uid 500); 23 Jul 2009 08:05:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12819 invoked by uid 99); 23 Jul 2009 08:05:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 08:05:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 08:05:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id F233C29A0018
	for <common-issues@hadoop.apache.org>; Thu, 23 Jul 2009 01:05:14 -0700 (PDT)
Message-ID: <1254746822.1248336314991.JavaMail.jira@brutus>
Date: Thu, 23 Jul 2009 01:05:14 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6164) Test-patch's method of checking for
 new tests is not correct
In-Reply-To: <1144829364.1248220514920.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734509#action_12734509 ] 

Jakob Homan commented on HADOOP-6164:
-------------------------------------

Well I imagine it's worth finding out why Hudson let through a patch that failed a test without reporting it.  This is a pretty big issue if it's happening with other patches; can you please follow up on it?

> Test-patch's method of checking for new tests is not correct
> ------------------------------------------------------------
>
>                 Key: HADOOP-6164
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6164
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Jakob Homan
>            Assignee: Giridharan Kesavan
>
> As happened in HDFS-458, test-patch/hudson saw the previous patch(HDFS-492) add a new test, bringing the total number of tests to 49.  Then it tested 458, which had no new tests as it was a change to the build file.  492 had not yet been committed yet, so there were still only 48 tests in trunk.  But test-patch failed the patch, expecting 49 tests.  test-patch should check the correct number of tests based on trunk, not on whatever number it last saw.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-478-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 09:38:32 2009
Return-Path: <common-issues-return-478-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85315 invoked from network); 23 Jul 2009 09:38:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 09:38:32 -0000
Received: (qmail 14430 invoked by uid 500); 23 Jul 2009 09:39:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14387 invoked by uid 500); 23 Jul 2009 09:39:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14377 invoked by uid 99); 23 Jul 2009 09:39:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 09:39:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 09:39:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3E15E234C004
	for <common-issues@hadoop.apache.org>; Thu, 23 Jul 2009 02:39:15 -0700 (PDT)
Message-ID: <425777178.1248341955239.JavaMail.jira@brutus>
Date: Thu, 23 Jul 2009 02:39:15 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6105) Provide a way to automatically
 handle backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734532#action_12734532 ] 

V.V.Chaitanya Krishna commented on HADOOP-6105:
-----------------------------------------------

Assumptions/Requirements:

*  Deprecated keys never occur in default.xml files.
*  There won't be any storage for deprecated keys in the Configuration object. Instead, new key mappings will be used.
*  The precedence order is as follows:
*# Value which comes with the final attribute as true.
*# Value occurring with the deprecated key.

The following table depicts the various cases arising, keeping in view whether the key in concern is final and if the key is deprecated:

*NOTE:* 
# *Key* - the current key that is to be loaded in loadResource method. It can be a deprecated key (*old*), key in default.xml (*key_default*) or key in site.xml (*key_site*).
#  *Prev. Occurrence* - The attribute which changed the value of the *Key* most recently.
# *isFinal* - true if  *Key* is having the _final_ property set to true.
# *isPrevFinal* - true if  *Prev. Occurrence* is having the _final_ property set to true.
# *Value* - Value to be expected after *Key* is loaded. It can be either of the values corresponding to *old*, *key_default* or *key_site*.
# _warn_ - logs a warning message indicating that a final parameter is being attempted to override.
Also, a warning message which indicates the usage of a deprecated key is logged whenever *Key* is a deprecated one.

||Key||isFinal||Prev.Occurrence||isPrevFinal||Value||
|old|true|key_default|true|old|
|old|true|key_default|false|old|
|old|true|key_site|true|old|
|old|true|key_site|false|old|
|old|false|key_default|true|key_default (warn)|
|old|false|key_default|false|old|
|old|false|key_site|true|key_site (warn)|
|old|false|key_site|false|old|
|key_site|true|old|true|old|
|key_site|true|old|false|key_site|
|key_site|true|key_default|true|key_default (warn)|
|key_site|true|key_default|false|key_site|
|key_site|false|old|true|old|
|key_site|false|old|false|old|
|key_site|false|key_default|true|key_default (warn)|
|key_site|false|key_default|false|key_site|




> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-479-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 10:30:34 2009
Return-Path: <common-issues-return-479-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16148 invoked from network); 23 Jul 2009 10:30:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 10:30:33 -0000
Received: (qmail 93245 invoked by uid 500); 23 Jul 2009 10:31:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93202 invoked by uid 500); 23 Jul 2009 10:31:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93192 invoked by uid 99); 23 Jul 2009 10:31:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 10:31:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 10:31:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E829F29A001E
	for <common-issues@hadoop.apache.org>; Thu, 23 Jul 2009 03:31:14 -0700 (PDT)
Message-ID: <1980566547.1248345074950.JavaMail.jira@brutus>
Date: Thu, 23 Jul 2009 03:31:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6164) Test-patch's method of checking for
 new tests is not correct
In-Reply-To: <1144829364.1248220514920.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734543#action_12734543 ] 

Giridharan Kesavan commented on HADOOP-6164:
--------------------------------------------

I agree with you and I would follow up with this issue. 
But this is for the first time I'm hearing something like this.

If you agree that this not something to do with test-patch then I can openup a different jira and deal with hudson issue or we can file a bug with hudson dev.

thanks

> Test-patch's method of checking for new tests is not correct
> ------------------------------------------------------------
>
>                 Key: HADOOP-6164
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6164
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Jakob Homan
>            Assignee: Giridharan Kesavan
>
> As happened in HDFS-458, test-patch/hudson saw the previous patch(HDFS-492) add a new test, bringing the total number of tests to 49.  Then it tested 458, which had no new tests as it was a change to the build file.  492 had not yet been committed yet, so there were still only 48 tests in trunk.  But test-patch failed the patch, expecting 49 tests.  test-patch should check the correct number of tests based on trunk, not on whatever number it last saw.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-480-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 16:22:35 2009
Return-Path: <common-issues-return-480-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14317 invoked from network); 23 Jul 2009 16:22:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 16:22:35 -0000
Received: (qmail 54265 invoked by uid 500); 23 Jul 2009 16:23:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54187 invoked by uid 500); 23 Jul 2009 16:23:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54138 invoked by uid 99); 23 Jul 2009 16:23:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 16:23:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 16:23:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 100FA234C046
	for <common-issues@hadoop.apache.org>; Thu, 23 Jul 2009 09:23:15 -0700 (PDT)
Message-ID: <1529253317.1248366195064.JavaMail.jira@brutus>
Date: Thu, 23 Jul 2009 09:23:15 -0700 (PDT)
From: "Amar Kamat (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4490) Map and Reduce tasks should run as
 the user who submitted the job
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amar Kamat updated HADOOP-4490:
-------------------------------

    Attachment: 2701949.4490.9.patch

Attaching an example patch for branch 20 not to be committed.

> Map and Reduce tasks should run as the user who submitted the job
> -----------------------------------------------------------------
>
>                 Key: HADOOP-4490
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4490
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>            Assignee: Hemanth Yamijala
>             Fix For: 0.21.0
>
>         Attachments: 2701949.4490.9.patch, cluster_setup.pdf, cluster_setup.pdf, HADOOP-4490-1.patch, HADOOP-4490-1.patch, hadoop-4490-10.patch, hadoop-4490-11.patch, hadoop-4490-12.patch, hadoop-4490-13.patch, hadoop-4490-14.patch, HADOOP-4490-2.patch, HADOOP-4490-3.patch, hadoop-4490-4.patch, hadoop-4490-5.patch, hadoop-4490-6.patch, hadoop-4490-7.patch, hadoop-4490-8.patch, hadoop-4490-9.patch, hadoop-4490-br20-3.2.patch, hadoop-4490-br20-3.patch, hadoop-4490-design.pdf, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490_streaming.patch
>
>
> Currently the TaskTracker spawns the map/reduce tasks, resulting in them running as the user who started the TaskTracker.
> For security and accounting purposes the tasks should be run as the job-owner.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-481-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 17:36:32 2009
Return-Path: <common-issues-return-481-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58649 invoked from network); 23 Jul 2009 17:36:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 17:36:32 -0000
Received: (qmail 35595 invoked by uid 500); 23 Jul 2009 17:37:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35560 invoked by uid 500); 23 Jul 2009 17:37:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35545 invoked by uid 99); 23 Jul 2009 17:37:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 17:37:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 17:37:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E41A2234C04B
	for <common-issues@hadoop.apache.org>; Thu, 23 Jul 2009 10:37:14 -0700 (PDT)
Message-ID: <1446445809.1248370634933.JavaMail.jira@brutus>
Date: Thu, 23 Jul 2009 10:37:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6123) hdfs script does not work after
 project split.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734657#action_12734657 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6123:
------------------------------------------------

> I don't think the "for loop" can be replaced by using the wildcard. It won't expand the path in CLASSPATH.
In the [java man page|http://java.sun.com/javase/6/docs/technotes/tools/solaris/java.html], it says,
{quote}
-classpath classpath
-cp classpath

... As a special convenience, a class path element containing a basename of * is considered equivalent to specifying a list of all the files in the directory with the extension .jar or .JAR (a java program cannot tell the difference between the two invocations).

For example, if directory foo contains a.jar and b.JAR, then the class path element foo/* is expanded to a A.jar:b.JAR, except that the order of jar files is unspecified.  ...
{quote}

> hdfs script does not work after project split.
> ----------------------------------------------
>
>                 Key: HADOOP-6123
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6123
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Sharad Agarwal
>             Fix For: 0.21.0
>
>         Attachments: 6123_v1.patch
>
>
> There are problems running hdfs script from common.
> # Usage message for hdfs does not have "fs" option anymore.
> # There seem to be an undocumented option "dfs", but using it throws NoClassDefFoundError or ClassNotFoundException.
> # Same with other options e.g. name-node.
> May I am missing something here. How do we call ls these days?
> Do we need to move hdfs script to hdfs project.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-482-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 18:33:32 2009
Return-Path: <common-issues-return-482-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3633 invoked from network); 23 Jul 2009 18:33:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 18:33:31 -0000
Received: (qmail 55678 invoked by uid 500); 23 Jul 2009 18:34:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55635 invoked by uid 500); 23 Jul 2009 18:34:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55625 invoked by uid 99); 23 Jul 2009 18:34:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 18:34:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 18:34:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D6F5B234C046
	for <common-issues@hadoop.apache.org>; Thu, 23 Jul 2009 11:34:14 -0700 (PDT)
Message-ID: <2065865070.1248374054879.JavaMail.jira@brutus>
Date: Thu, 23 Jul 2009 11:34:14 -0700 (PDT)
From: "Fernando (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6167) bin/hadoop script doesn't allow for
 different memory settings for each daemon type
In-Reply-To: <128618595.1248313995284.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734691#action_12734691 ] 

Fernando commented on HADOOP-6167:
----------------------------------

HBase scripts have the same issue since they were just copying things from Hadoop.. so they have the same bug.

Here is the related bug in HBase: https://issues.apache.org/jira/browse/HBASE-1687


> bin/hadoop script doesn't allow for different memory settings for each daemon type
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6167
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.0
>            Reporter: Fernando
>         Attachments: hadoop
>
>
> bin/hadoop assumes that all daemon types ( namenode, datanode, jobtracker, tasktracker ), all use the same memory settings.. (HADOOP_HEAPSIZE).
> I propose changes to that script to allow overriding the default memory ( HADOOP_HEAPSIZE ), with daemon specific OPTS (HADOOP_NAMENODE_OPTS, etc ).
> Basically at the bottom of the bin/hadoop script, it will check to see if the user has already set "-Xmx" in the HADOOP_OPTS variable.. if so, then it will ignore the JAVA_HEAP_SIZE variable..
> as such:
> # run it
> if [[ $HADOOP_OPTS == *-Xmx* ]]; then
>   exec "$JAVA" $HADOOP_OPTS -classpath "$CLASSPATH" $CLASS "$@"
> else
>   exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS -classpath "$CLASSPATH" $CLASS "$@"
> fi
> I will attach the file as I have modified it..

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-483-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 18:39:32 2009
Return-Path: <common-issues-return-483-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6134 invoked from network); 23 Jul 2009 18:39:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 18:39:32 -0000
Received: (qmail 66982 invoked by uid 500); 23 Jul 2009 18:40:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66938 invoked by uid 500); 23 Jul 2009 18:40:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66928 invoked by uid 99); 23 Jul 2009 18:40:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 18:40:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 18:40:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DA184234C046
	for <common-issues@hadoop.apache.org>; Thu, 23 Jul 2009 11:40:14 -0700 (PDT)
Message-ID: <1009432768.1248374414892.JavaMail.jira@brutus>
Date: Thu, 23 Jul 2009 11:40:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734693#action_12734693 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6166:
------------------------------------------------

Yeah, the speed depends on many platform dependent details like cache size, 32-bit/64-bit, etc.  So reducing the number of operations in the CRC algorithm may not lead to a better performance.

I tried more varieties like Crc32_6_4 but my implementation did not perform well.  We should run benchmark on Scott's.

> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-484-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 18:39:39 2009
Return-Path: <common-issues-return-484-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6209 invoked from network); 23 Jul 2009 18:39:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 18:39:39 -0000
Received: (qmail 67334 invoked by uid 500); 23 Jul 2009 18:40:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67307 invoked by uid 500); 23 Jul 2009 18:40:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67297 invoked by uid 99); 23 Jul 2009 18:40:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 18:40:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 18:40:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CA4C2234C004
	for <common-issues@hadoop.apache.org>; Thu, 23 Jul 2009 11:40:14 -0700 (PDT)
Message-ID: <1278669946.1248374414813.JavaMail.jira@brutus>
Date: Thu, 23 Jul 2009 11:40:14 -0700 (PDT)
From: "ryan rawson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6167) bin/hadoop script doesn't allow for
 different memory settings for each daemon type
In-Reply-To: <128618595.1248313995284.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734692#action_12734692 ] 

ryan rawson commented on HADOOP-6167:
-------------------------------------

I'm not sure this is true, my datanode processes are running like so:

/usr/java/jdk1.6.0_14/bin/java -Xmx1000m -Dcom.sun.management.jmxremote -Xmx2000m -XX:+DoEscapeAnalysis (all the other arguments)

in hadoop-env.sh i have:
export HADOOP_DATANODE_OPTS="-Xmx2000m -XX:+DoEscapeAnalysis ...."


how does this not work?

> bin/hadoop script doesn't allow for different memory settings for each daemon type
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6167
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.0
>            Reporter: Fernando
>         Attachments: hadoop
>
>
> bin/hadoop assumes that all daemon types ( namenode, datanode, jobtracker, tasktracker ), all use the same memory settings.. (HADOOP_HEAPSIZE).
> I propose changes to that script to allow overriding the default memory ( HADOOP_HEAPSIZE ), with daemon specific OPTS (HADOOP_NAMENODE_OPTS, etc ).
> Basically at the bottom of the bin/hadoop script, it will check to see if the user has already set "-Xmx" in the HADOOP_OPTS variable.. if so, then it will ignore the JAVA_HEAP_SIZE variable..
> as such:
> # run it
> if [[ $HADOOP_OPTS == *-Xmx* ]]; then
>   exec "$JAVA" $HADOOP_OPTS -classpath "$CLASSPATH" $CLASS "$@"
> else
>   exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS -classpath "$CLASSPATH" $CLASS "$@"
> fi
> I will attach the file as I have modified it..

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-485-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 19:17:34 2009
Return-Path: <common-issues-return-485-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31657 invoked from network); 23 Jul 2009 19:17:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 19:17:34 -0000
Received: (qmail 68798 invoked by uid 500); 23 Jul 2009 19:18:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68752 invoked by uid 500); 23 Jul 2009 19:18:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68677 invoked by uid 99); 23 Jul 2009 19:18:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 19:18:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 19:18:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 2C4CD29A0015
	for <common-issues@hadoop.apache.org>; Thu, 23 Jul 2009 12:18:15 -0700 (PDT)
Message-ID: <406796382.1248376695180.JavaMail.jira@brutus>
Date: Thu, 23 Jul 2009 12:18:15 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6167) bin/hadoop script doesn't allow for
 different memory settings for each daemon type
In-Reply-To: <128618595.1248313995284.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734735#action_12734735 ] 

Doug Cutting commented on HADOOP-6167:
--------------------------------------

This might be better done by:
 - adding JAVA_HEAP_MAX to HADOOP_OPTS if it does not already contain -Xmx
 - removing JAVA_HEAP_MAX from the java command

Also, please attach a patch file, not the altered source file.

http://wiki.apache.org/hadoop/HowToContribute



> bin/hadoop script doesn't allow for different memory settings for each daemon type
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6167
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.0
>            Reporter: Fernando
>         Attachments: hadoop
>
>
> bin/hadoop assumes that all daemon types ( namenode, datanode, jobtracker, tasktracker ), all use the same memory settings.. (HADOOP_HEAPSIZE).
> I propose changes to that script to allow overriding the default memory ( HADOOP_HEAPSIZE ), with daemon specific OPTS (HADOOP_NAMENODE_OPTS, etc ).
> Basically at the bottom of the bin/hadoop script, it will check to see if the user has already set "-Xmx" in the HADOOP_OPTS variable.. if so, then it will ignore the JAVA_HEAP_SIZE variable..
> as such:
> # run it
> if [[ $HADOOP_OPTS == *-Xmx* ]]; then
>   exec "$JAVA" $HADOOP_OPTS -classpath "$CLASSPATH" $CLASS "$@"
> else
>   exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS -classpath "$CLASSPATH" $CLASS "$@"
> fi
> I will attach the file as I have modified it..

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-486-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 19:21:35 2009
Return-Path: <common-issues-return-486-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32462 invoked from network); 23 Jul 2009 19:21:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 19:21:35 -0000
Received: (qmail 71849 invoked by uid 500); 23 Jul 2009 19:22:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71779 invoked by uid 500); 23 Jul 2009 19:22:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71756 invoked by uid 99); 23 Jul 2009 19:22:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 19:22:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 19:22:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3D18129A001D
	for <common-issues@hadoop.apache.org>; Thu, 23 Jul 2009 12:22:15 -0700 (PDT)
Message-ID: <1350230471.1248376935249.JavaMail.jira@brutus>
Date: Thu, 23 Jul 2009 12:22:15 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6120) Add support for Avro types in hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-6120:
---------------------------------

       Resolution: Fixed
    Fix Version/s: 0.21.0
           Status: Resolved  (was: Patch Available)

I just committed this.  Thanks, Sharad!

> Add support for Avro types in hadoop
> ------------------------------------
>
>                 Key: HADOOP-6120
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6120
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>             Fix For: 0.21.0
>
>         Attachments: 6120_v1.patch, 6120_v2.patch, 6120_v3.patch, 6120_v4.patch, 6120_v5.patch, 6120_v6.patch, HADOOP-6120.patch
>
>
> Support to serialize and deserialize Avro types in Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-487-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 20:06:31 2009
Return-Path: <common-issues-return-487-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51882 invoked from network); 23 Jul 2009 20:06:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 20:06:31 -0000
Received: (qmail 63535 invoked by uid 500); 23 Jul 2009 20:07:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63489 invoked by uid 500); 23 Jul 2009 20:07:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63471 invoked by uid 99); 23 Jul 2009 20:07:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 20:07:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 20:07:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D63DF234C046
	for <common-issues@hadoop.apache.org>; Thu, 23 Jul 2009 13:07:14 -0700 (PDT)
Message-ID: <1003483123.1248379634876.JavaMail.jira@brutus>
Date: Thu, 23 Jul 2009 13:07:14 -0700 (PDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6168) HADOOP_HEAPSIZE cannot be done
 per-server easily
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

HADOOP_HEAPSIZE cannot be done per-server easily
------------------------------------------------

                 Key: HADOOP-6168
                 URL: https://issues.apache.org/jira/browse/HADOOP-6168
             Project: Hadoop Common
          Issue Type: Bug
          Components: conf
    Affects Versions: 0.18.3
            Reporter: Allen Wittenauer


The hadoop script forces a heap that cannot be easily overridden if one wants to push the same config everywhere.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-488-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 20:12:43 2009
Return-Path: <common-issues-return-488-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55678 invoked from network); 23 Jul 2009 20:12:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 20:12:40 -0000
Received: (qmail 71399 invoked by uid 500); 23 Jul 2009 20:13:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71355 invoked by uid 500); 23 Jul 2009 20:13:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71345 invoked by uid 99); 23 Jul 2009 20:13:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 20:13:45 +0000
X-ASF-Spam-Status: No, hits=-1997.2 required=10.0
	tests=ALL_TRUSTED,WEIRD_QUOTING
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 20:13:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CA20A234C004
	for <common-issues@hadoop.apache.org>; Thu, 23 Jul 2009 13:13:14 -0700 (PDT)
Message-ID: <101070509.1248379994813.JavaMail.jira@brutus>
Date: Thu, 23 Jul 2009 13:13:14 -0700 (PDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6168) HADOOP_HEAPSIZE cannot be done
 per-server easily
In-Reply-To: <1003483123.1248379634876.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734759#action_12734759 ] 

Allen Wittenauer commented on HADOOP-6168:
------------------------------------------

So bin/hadoop has this logic:

JAVA_HEAP_MAX=-Xmx1000m

# check envvars which might override default args
if [ "$HADOOP_HEAPSIZE" != "" ]; then
  #echo "run with heapsize $HADOOP_HEAPSIZE"
  JAVA_HEAP_MAX="-Xmx""$HADOOP_HEAPSIZE""m"
  #echo $JAVA_HEAP_MAX
fi


This makes it impossible to create a hadoop-env.sh that is truly generic to all nodes.  It would be better to do this HEAPSIZE management in hadoop-env.sh to allow it to be easily override-able.

> HADOOP_HEAPSIZE cannot be done per-server easily
> ------------------------------------------------
>
>                 Key: HADOOP-6168
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6168
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.18.3
>            Reporter: Allen Wittenauer
>
> The hadoop script forces a heap that cannot be easily overridden if one wants to push the same config everywhere.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-489-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 20:20:32 2009
Return-Path: <common-issues-return-489-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58205 invoked from network); 23 Jul 2009 20:20:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 20:20:32 -0000
Received: (qmail 85484 invoked by uid 500); 23 Jul 2009 20:21:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85441 invoked by uid 500); 23 Jul 2009 20:21:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85431 invoked by uid 99); 23 Jul 2009 20:21:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 20:21:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 20:21:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CAA85234C004
	for <common-issues@hadoop.apache.org>; Thu, 23 Jul 2009 13:21:14 -0700 (PDT)
Message-ID: <848066752.1248380474825.JavaMail.jira@brutus>
Date: Thu, 23 Jul 2009 13:21:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734762#action_12734762 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6166:
------------------------------------------------

Unfortunately, Crc32_4_3 only wins on a 32-bit vm over TestPureJavaCrc32.PerformanceTest but not 64-bit vm.

- 32-bit vm
||num bytes||CRC32 MB/sec||Crc32_4_2 MB/sec||Crc32_4_3 MB/sec||Crc32_3_2 MB/sec||PureJavaCrc32 MB/sec||
| 1	|4.504 	|52.409 	|55.779 	|46.603 	|59.146 	|
| 2	|8.825 	|86.590 	|87.938 	|77.267 	|89.942 	|
| 4	|17.254 	|119.824 	|151.929 	|120.808 	|146.983 	|
| 8	|32.037 	|147.222 	|202.527 	|161.984 	|174.844 	|
| 16	|59.078 	|161.879 	|231.467 	|195.635 	|228.018 	|
| 32	|100.267 	|176.767 	|276.844 	|241.295 	|244.502 	|
| 64	|148.985 	|178.250 	|283.511 	|269.368 	|263.209 	|
| 128	|199.763 	|185.639 	|294.116 	|271.943 	|259.021 	|
| 256	|232.751 	|179.525 	|290.357 	|259.453 	|256.891 	|
| 512	|255.430 	|178.217 	|296.907 	|280.763 	|257.362 	|
| 1024	|262.274 	|172.033 	|289.806 	|277.863 	|261.793 	|
| 2048	|273.744 	|187.468 	|299.271 	|286.387 	|272.611 	|
| 4096	|289.373 	|186.306 	|293.845 	|276.021 	|266.067 	|
| 8192	|290.282 	|184.625 	|296.723 	|285.097 	|271.503 	|
| 16384	|298.959 	|180.863 	|291.081 	|250.583 	|199.536 	|
| 32768	|277.718 	|184.078 	|293.156 	|285.722 	|270.377 	|
| 65536	|300.016 	|186.439 	|298.946 	|283.990 	|271.268 	|
| 131072	|298.971 	|186.754 	|298.417 	|283.949 	|268.240 	|
| 262144	|299.688 	|184.124 	|296.014 	|281.633 	|265.799 	|
| 524288	|282.488 	|176.217 	|288.120 	|284.030 	|267.917 	|
| 1048576	|294.852 	|185.167 	|291.499 	|279.362 	|267.438 	|
| 2097152	|296.117 	|174.667 	|281.145 	|272.180 	|260.837 	|
| 4194304	|283.934 	|173.777 	|279.931 	|271.393 	|259.805 	|
| 8388608	|289.455 	|177.829 	|291.535 	|269.850 	|259.513 	|
| 16777216	|284.204 	|177.449 	|290.489 	|276.586 	|265.657 	|

- 64-bit vm
||num bytes||CRC32 MB/sec||Crc32_4_2 MB/sec||Crc32_4_3 MB/sec||Crc32_3_2 MB/sec||PureJavaCrc32 MB/sec||
| 1     |7.636  |80.107         |99.658         |77.283         |34.446         |
| 2     |14.598         |116.202        |110.091        |94.056         |106.498        |
| 4     |27.786         |152.932        |197.294        |147.532        |175.766        |
| 8     |50.159         |153.598        |194.617        |163.596        |197.350        |
| 16    |85.036         |177.761        |258.683        |237.917        |278.764        |
| 32    |130.278        |180.486        |310.024        |281.343        |342.374        |
| 64    |177.501        |181.663        |343.592        |320.385        |384.938        |
| 128   |217.128        |181.836        |366.965        |338.893        |411.724        |
| 256   |245.690        |182.637        |379.003        |348.981        |425.874        |
| 512   |262.085        |181.103        |381.961        |355.506        |428.103        |
| 1024  |271.307        |179.753        |381.658        |356.488        |433.800        |
| 2048  |276.640        |180.451        |378.667        |351.067        |437.275        |
| 4096  |278.435        |179.881        |372.762        |347.728        |437.209        |
| 8192  |279.883        |180.776        |377.241        |351.178        |439.571        |
| 16384 |281.385        |180.775        |377.493        |353.361        |439.606        |
| 32768 |281.434        |180.703        |378.047        |353.656        |438.703        |
| 65536 |281.354        |180.914        |377.805        |353.130        |437.152        |
| 131072        |280.941        |180.288        |377.164        |353.340        |438.806        |
| 262144        |282.056        |180.910        |378.514        |354.320        |438.208        |
| 524288        |281.066        |180.177        |377.148        |352.832        |437.183        |
| 1048576       |281.668        |180.790        |378.412        |354.059        |438.755        |
| 2097152       |282.162        |180.545        |377.918        |353.841        |438.497        |
| 4194304       |281.379        |179.018        |376.287        |352.240        |436.963        |
| 8388608       |279.929        |178.058        |371.618        |349.405        |430.993        |
| 16777216      |278.974        |177.577        |371.729        |347.971        |429.326        |

> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-490-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 21:22:36 2009
Return-Path: <common-issues-return-490-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80506 invoked from network); 23 Jul 2009 21:22:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 21:22:35 -0000
Received: (qmail 71859 invoked by uid 500); 23 Jul 2009 21:23:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71805 invoked by uid 500); 23 Jul 2009 21:23:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71531 invoked by uid 99); 23 Jul 2009 21:23:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 21:23:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 21:23:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E9CAC234C053
	for <common-issues@hadoop.apache.org>; Thu, 23 Jul 2009 14:23:14 -0700 (PDT)
Message-ID: <1992376851.1248384194956.JavaMail.jira@brutus>
Date: Thu, 23 Jul 2009 14:23:14 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5438) Merge FileSystem.create and
 FileSystem.append
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734792#action_12734792 ] 

Jakob Homan commented on HADOOP-5438:
-------------------------------------

So, as committed there are now six create methods that take a boolean value as to whether or not overwrite (and converts them to an enumset) and one create method that takes an EnumSet.  HADOOP-3252 will definitely help with this, but this still seems odd and counterintuitive.  If we have the CreateFlag enum, we should use it everywhere.  Otherwise the methods are a bit confusing - a user will sit there going 'why are there so many create methods and why is there is this odd one out - isn't overwrite boolean the same as the createflag enum?'

If there weren't so many different create methods, I would suggest having one each for just the Enum value and and equivalent one that takes an enumset, so that if you just wanted to overwrite you could just use the one that takes the enum, and the convenience method would jam it into an enumset.  It's too bad Java won't automatically convert a single enum into an enumset when necessary.

> Merge FileSystem.create and FileSystem.append
> ---------------------------------------------
>
>                 Key: HADOOP-5438
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5438
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: He Yongqiang
>            Assignee: He Yongqiang
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-5438(2009-04-06).patch, Hadoop-5438-2009-03-30.patch, Hadoop-5438-2009-03-31-2.patch, Hadoop-5438-2009-03-31.patch, Hadoop-5438-2009-05-10.patch, Hadoop-5438-2009-05-15.patch, Hadoop-5438-2009-05-19.patch, Hadoop-5438-2009-05-5.patch
>
>
> Currently, when a user wants to modify a file, the user first calls exists() to know if this file is already there. And then uses create() or append() according to whether the file exists or not.
> the code looks like:
> {code}
> FSDataOutputStream out_1 = null;
> if (fs.exists(path_1))
>    out_1 = fs.append(path_1);
> else
>    out_1 = fs.create(path_1);
> {code}
> . On the performace side,It involes two RPCs. On the easy-of-use side, it is not very convient in contrast to the traditional open interface.
> It will more complicate if there is a overwrite parameter specified. I donot know whether there is a bug about 'overwrite' in 0.19, some times it takes a long time for overwrite creates to reture. So i make the write file code with overwrite param works like:
> {code}
> boolean exists = fs.exists(name);
> if (overwrite) {
>     if (exists)
>        fs.delete(name, true);
>      this.out = fs.create(name, overwrite, bufferSize, replication,
> 				    blockSize, progress);
>      this.currentRowID = 0;
>  } else {
>    if (!exists)
> 	this.out = fs.create(name, overwrite, bufferSize,
> 					replication, blockSize, progress);
>    else
> 	this.out = fs.append(name, bufferSize, progress);
> {code}
> Some code statements there are really redundant and not needed, especialy with the delete(). But without deleting first, the overwrite takes a long time to reture.
> BTW, i will create another issue about the overwrite problem. If it is not a bug at all or a duplicate, someone please close it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-491-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 21:32:32 2009
Return-Path: <common-issues-return-491-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86406 invoked from network); 23 Jul 2009 21:32:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 21:32:32 -0000
Received: (qmail 80839 invoked by uid 500); 23 Jul 2009 21:33:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80796 invoked by uid 500); 23 Jul 2009 21:33:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80786 invoked by uid 99); 23 Jul 2009 21:33:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 21:33:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 21:33:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C86DF234C004
	for <common-issues@hadoop.apache.org>; Thu, 23 Jul 2009 14:33:14 -0700 (PDT)
Message-ID: <1989670659.1248384794807.JavaMail.jira@brutus>
Date: Thu, 23 Jul 2009 14:33:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6150) Need to be able to instantiate a
 comparator instance from a comparator string without creating a
 TFile.Reader object
In-Reply-To: <1804574098.1247618054845.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-6150:
------------------------------

    Attachment: HADOOP-6150-0723-for-20.patch

Same patch for hadoop 0.20.

> Need to be able to instantiate a comparator instance from a comparator string without creating a TFile.Reader object
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6150
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6150
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.0, 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6150-0721.patch, HADOOP-6150-0723-for-20.patch
>
>
> Occasionally, we want have the same instance of comparator object represented by the TFile comparator string. We should be able to do that without requiring to first open up a tfile that was previously written use the same comparator.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-492-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 21:38:38 2009
Return-Path: <common-issues-return-492-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90243 invoked from network); 23 Jul 2009 21:38:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 21:38:38 -0000
Received: (qmail 86733 invoked by uid 500); 23 Jul 2009 21:39:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86616 invoked by uid 500); 23 Jul 2009 21:39:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86558 invoked by uid 99); 23 Jul 2009 21:39:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 21:39:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 21:39:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E68EF234C053
	for <common-issues@hadoop.apache.org>; Thu, 23 Jul 2009 14:39:14 -0700 (PDT)
Message-ID: <905730474.1248385154943.JavaMail.jira@brutus>
Date: Thu, 23 Jul 2009 14:39:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Reopened: (HADOOP-5864) Fix DMI and OBL findbugs in packages
 hdfs and metrics
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE reopened HADOOP-5864:
--------------------------------------------


Talked to Suresh.  Look like that the changes was lost during project split.

> Fix DMI and OBL findbugs in packages hdfs and metrics
> -----------------------------------------------------
>
>                 Key: HADOOP-5864
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5864
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.21.0
>
>         Attachments: findBugs.patch, findBugs1.patch, findBugs2.patch, findbugs3.hdfs.patch
>
>
> This issue is to fix the following findbugs:
> DMI in org.apache.hadoop.hdfs.server.namenode.INodeDirectory.getExistingPathINodes(byte[][], INode[]) 
> OBL in org.apache.hadoop.hdfs.server.datanode.DataStorage.linkBlocks(File, File, int)
> OBL in org.apache.hadoop.hdfs.server.datanode.FSDataset.createBlockWriteStreams(File, File)
> OBL in org.apache.hadoop.hdfs.server.datanode.FSDataset.getTmpInputStreams(Block, long, long)
> OBL in org.apache.hadoop.metrics.ContextFactory.setAttributes() 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-493-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 21:42:32 2009
Return-Path: <common-issues-return-493-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91929 invoked from network); 23 Jul 2009 21:42:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 21:42:31 -0000
Received: (qmail 89654 invoked by uid 500); 23 Jul 2009 21:43:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89611 invoked by uid 500); 23 Jul 2009 21:43:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89601 invoked by uid 99); 23 Jul 2009 21:43:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 21:43:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 21:43:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CDD30234C046
	for <common-issues@hadoop.apache.org>; Thu, 23 Jul 2009 14:43:14 -0700 (PDT)
Message-ID: <1525779092.1248385394842.JavaMail.jira@brutus>
Date: Thu, 23 Jul 2009 14:43:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-5864) Fix DMI and OBL findbugs in packages
 hdfs and metrics
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE resolved HADOOP-5864.
--------------------------------------------

    Resolution: Fixed

I have re-committed the missing changes to hdfs trunk.  Thanks, Suresh!

> Fix DMI and OBL findbugs in packages hdfs and metrics
> -----------------------------------------------------
>
>                 Key: HADOOP-5864
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5864
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.21.0
>
>         Attachments: findBugs.patch, findBugs1.patch, findBugs2.patch, findbugs3.hdfs.patch
>
>
> This issue is to fix the following findbugs:
> DMI in org.apache.hadoop.hdfs.server.namenode.INodeDirectory.getExistingPathINodes(byte[][], INode[]) 
> OBL in org.apache.hadoop.hdfs.server.datanode.DataStorage.linkBlocks(File, File, int)
> OBL in org.apache.hadoop.hdfs.server.datanode.FSDataset.createBlockWriteStreams(File, File)
> OBL in org.apache.hadoop.hdfs.server.datanode.FSDataset.getTmpInputStreams(Block, long, long)
> OBL in org.apache.hadoop.metrics.ContextFactory.setAttributes() 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-494-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 21:48:34 2009
Return-Path: <common-issues-return-494-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96214 invoked from network); 23 Jul 2009 21:48:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 21:48:34 -0000
Received: (qmail 98127 invoked by uid 500); 23 Jul 2009 21:49:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98080 invoked by uid 500); 23 Jul 2009 21:49:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98070 invoked by uid 99); 23 Jul 2009 21:49:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 21:49:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 21:49:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C89DB234C004
	for <common-issues@hadoop.apache.org>; Thu, 23 Jul 2009 14:49:14 -0700 (PDT)
Message-ID: <1018878767.1248385754817.JavaMail.jira@brutus>
Date: Thu, 23 Jul 2009 14:49:14 -0700 (PDT)
From: "gary murry (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6164) Test-patch's method of checking for
 new tests is not correct
In-Reply-To: <1144829364.1248220514920.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734805#action_12734805 ] 

gary murry commented on HADOOP-6164:
------------------------------------

Hi Giri,

If another jira is being opened for this, can we make sure an link the issues for tracking?  I think it is agreed that this is not a test-patch issue, but rather a hudson issue.

Thanks,
-Gary

> Test-patch's method of checking for new tests is not correct
> ------------------------------------------------------------
>
>                 Key: HADOOP-6164
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6164
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Jakob Homan
>            Assignee: Giridharan Kesavan
>
> As happened in HDFS-458, test-patch/hudson saw the previous patch(HDFS-492) add a new test, bringing the total number of tests to 49.  Then it tested 458, which had no new tests as it was a change to the build file.  492 had not yet been committed yet, so there were still only 48 tests in trunk.  But test-patch failed the patch, expecting 49 tests.  test-patch should check the correct number of tests based on trunk, not on whatever number it last saw.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-495-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 23 22:29:32 2009
Return-Path: <common-issues-return-495-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13912 invoked from network); 23 Jul 2009 22:29:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Jul 2009 22:29:31 -0000
Received: (qmail 26502 invoked by uid 500); 23 Jul 2009 22:30:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26459 invoked by uid 500); 23 Jul 2009 22:30:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26449 invoked by uid 99); 23 Jul 2009 22:30:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 22:30:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jul 2009 22:30:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D0568234C004
	for <common-issues@hadoop.apache.org>; Thu, 23 Jul 2009 15:30:14 -0700 (PDT)
Message-ID: <1775980808.1248388214839.JavaMail.jira@brutus>
Date: Thu, 23 Jul 2009 15:30:14 -0700 (PDT)
From: "Raghu Angadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6132) RPC client opens an extra connection
 for VersionedProtocol
In-Reply-To: <840919907.1247085075163.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Raghu Angadi updated HADOOP-6132:
---------------------------------

             Tags: RPC
       Resolution: Fixed
    Fix Version/s: 0.21.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I just committed this. Thanks Kan.

> RPC client opens an extra connection for VersionedProtocol
> ----------------------------------------------------------
>
>                 Key: HADOOP-6132
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6132
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>             Fix For: 0.21.0
>
>         Attachments: 1.patch, 2.patch
>
>
> RPC client caches connections per protocol. However, since all of our real protocols are subclasses of VersionedProtocol, a bug in the implementation makes the client opens an extra connection just for the VersionedProtocol, which is not needed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-496-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 24 00:05:34 2009
Return-Path: <common-issues-return-496-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35642 invoked from network); 24 Jul 2009 00:05:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jul 2009 00:05:33 -0000
Received: (qmail 11104 invoked by uid 500); 24 Jul 2009 00:06:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11060 invoked by uid 500); 24 Jul 2009 00:06:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11049 invoked by uid 99); 24 Jul 2009 00:06:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 00:06:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 00:06:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C6FFF234C004
	for <common-issues@hadoop.apache.org>; Thu, 23 Jul 2009 17:06:14 -0700 (PDT)
Message-ID: <1633438328.1248393974800.JavaMail.jira@brutus>
Date: Thu, 23 Jul 2009 17:06:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6150) Need to be able to instantiate a
 comparator instance from a comparator string without creating a
 TFile.Reader object
In-Reply-To: <1804574098.1247618054845.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734839#action_12734839 ] 

Hong Tang commented on HADOOP-6150:
-----------------------------------

test-patch output for patch for hadoop trunk:

     [exec] -1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
     [exec]                         Please justify why no new tests are needed for this patch.
     [exec]                         Also please list what manual steps were performed to verify this patch.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
 

> Need to be able to instantiate a comparator instance from a comparator string without creating a TFile.Reader object
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6150
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6150
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.0, 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6150-0721.patch, HADOOP-6150-0723-for-20.patch
>
>
> Occasionally, we want have the same instance of comparator object represented by the TFile comparator string. We should be able to do that without requiring to first open up a tfile that was previously written use the same comparator.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-497-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 24 00:09:31 2009
Return-Path: <common-issues-return-497-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36192 invoked from network); 24 Jul 2009 00:09:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jul 2009 00:09:31 -0000
Received: (qmail 15562 invoked by uid 500); 24 Jul 2009 00:10:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15518 invoked by uid 500); 24 Jul 2009 00:10:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15508 invoked by uid 99); 24 Jul 2009 00:10:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 00:10:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 00:10:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C659B234C004
	for <common-issues@hadoop.apache.org>; Thu, 23 Jul 2009 17:10:14 -0700 (PDT)
Message-ID: <1116363998.1248394214798.JavaMail.jira@brutus>
Date: Thu, 23 Jul 2009 17:10:14 -0700 (PDT)
From: "Raghu Angadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6150) Need to be able to instantiate a
 comparator instance from a comparator string without creating a
 TFile.Reader object
In-Reply-To: <1804574098.1247618054845.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734841#action_12734841 ] 

Raghu Angadi commented on HADOOP-6150:
--------------------------------------

+1.

> Need to be able to instantiate a comparator instance from a comparator string without creating a TFile.Reader object
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6150
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6150
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.0, 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6150-0721.patch, HADOOP-6150-0723-for-20.patch
>
>
> Occasionally, we want have the same instance of comparator object represented by the TFile comparator string. We should be able to do that without requiring to first open up a tfile that was previously written use the same comparator.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-498-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 24 00:37:35 2009
Return-Path: <common-issues-return-498-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47123 invoked from network); 24 Jul 2009 00:37:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jul 2009 00:37:34 -0000
Received: (qmail 33240 invoked by uid 500); 24 Jul 2009 00:38:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33198 invoked by uid 500); 24 Jul 2009 00:38:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33188 invoked by uid 99); 24 Jul 2009 00:38:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 00:38:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 00:38:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 41359234C004
	for <common-issues@hadoop.apache.org>; Thu, 23 Jul 2009 17:38:15 -0700 (PDT)
Message-ID: <1113947182.1248395895259.JavaMail.jira@brutus>
Date: Thu, 23 Jul 2009 17:38:15 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6150) Need to be able to instantiate a
 comparator instance from a comparator string without creating a
 TFile.Reader object
In-Reply-To: <1804574098.1247618054845.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734849#action_12734849 ] 

Hong Tang commented on HADOOP-6150:
-----------------------------------

test-patch output for patch for hadoop 20: 

     [exec] -1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
     [exec]                         Please justify why no tests are needed for this patch.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
 

> Need to be able to instantiate a comparator instance from a comparator string without creating a TFile.Reader object
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6150
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6150
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.0, 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6150-0721.patch, HADOOP-6150-0723-for-20.patch
>
>
> Occasionally, we want have the same instance of comparator object represented by the TFile comparator string. We should be able to do that without requiring to first open up a tfile that was previously written use the same comparator.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-499-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 24 01:16:34 2009
Return-Path: <common-issues-return-499-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69856 invoked from network); 24 Jul 2009 01:16:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jul 2009 01:16:34 -0000
Received: (qmail 75889 invoked by uid 500); 24 Jul 2009 01:17:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75845 invoked by uid 500); 24 Jul 2009 01:17:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75835 invoked by uid 99); 24 Jul 2009 01:17:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 01:17:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 01:17:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 12666234C004
	for <common-issues@hadoop.apache.org>; Thu, 23 Jul 2009 18:17:15 -0700 (PDT)
Message-ID: <1549950480.1248398235058.JavaMail.jira@brutus>
Date: Thu, 23 Jul 2009 18:17:15 -0700 (PDT)
From: "Raghu Angadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6150) Need to be able to instantiate a
 comparator instance from a comparator string without creating a
 TFile.Reader object
In-Reply-To: <1804574098.1247618054845.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734863#action_12734863 ] 

Raghu Angadi commented on HADOOP-6150:
--------------------------------------

I just committed this. Thanks Hong.

> Need to be able to instantiate a comparator instance from a comparator string without creating a TFile.Reader object
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6150
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6150
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.0, 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6150-0721.patch, HADOOP-6150-0723-for-20.patch
>
>
> Occasionally, we want have the same instance of comparator object represented by the TFile comparator string. We should be able to do that without requiring to first open up a tfile that was previously written use the same comparator.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-500-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 24 01:50:34 2009
Return-Path: <common-issues-return-500-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92078 invoked from network); 24 Jul 2009 01:50:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jul 2009 01:50:34 -0000
Received: (qmail 97567 invoked by uid 500); 24 Jul 2009 01:51:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97530 invoked by uid 500); 24 Jul 2009 01:51:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97516 invoked by uid 99); 24 Jul 2009 01:51:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 01:51:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 01:51:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C5A5B234C004
	for <common-issues@hadoop.apache.org>; Thu, 23 Jul 2009 18:51:14 -0700 (PDT)
Message-ID: <7152271.1248400274793.JavaMail.jira@brutus>
Date: Thu, 23 Jul 2009 18:51:14 -0700 (PDT)
From: "Fernando (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6167) bin/hadoop script doesn't allow for
 different memory settings for each daemon type
In-Reply-To: <128618595.1248313995284.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Fernando updated HADOOP-6167:
-----------------------------

    Attachment: hadoop-script.diff

Ok. this is the diff now.  Have a look, I cleaned it up a little to do what you suggested.

> bin/hadoop script doesn't allow for different memory settings for each daemon type
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6167
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.0
>            Reporter: Fernando
>         Attachments: hadoop, hadoop-script.diff
>
>
> bin/hadoop assumes that all daemon types ( namenode, datanode, jobtracker, tasktracker ), all use the same memory settings.. (HADOOP_HEAPSIZE).
> I propose changes to that script to allow overriding the default memory ( HADOOP_HEAPSIZE ), with daemon specific OPTS (HADOOP_NAMENODE_OPTS, etc ).
> Basically at the bottom of the bin/hadoop script, it will check to see if the user has already set "-Xmx" in the HADOOP_OPTS variable.. if so, then it will ignore the JAVA_HEAP_SIZE variable..
> as such:
> # run it
> if [[ $HADOOP_OPTS == *-Xmx* ]]; then
>   exec "$JAVA" $HADOOP_OPTS -classpath "$CLASSPATH" $CLASS "$@"
> else
>   exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS -classpath "$CLASSPATH" $CLASS "$@"
> fi
> I will attach the file as I have modified it..

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-501-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 24 03:42:34 2009
Return-Path: <common-issues-return-501-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28046 invoked from network); 24 Jul 2009 03:42:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jul 2009 03:42:34 -0000
Received: (qmail 67597 invoked by uid 500); 24 Jul 2009 03:43:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67549 invoked by uid 500); 24 Jul 2009 03:43:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67539 invoked by uid 99); 24 Jul 2009 03:43:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 03:43:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 03:43:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E0719234C051
	for <common-issues@hadoop.apache.org>; Thu, 23 Jul 2009 20:43:14 -0700 (PDT)
Message-ID: <775761630.1248406994918.JavaMail.jira@brutus>
Date: Thu, 23 Jul 2009 20:43:14 -0700 (PDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6123) hdfs script does not work after
 project split.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734891#action_12734891 ] 

Sharad Agarwal commented on HADOOP-6123:
----------------------------------------

@Tsz When I tried, it didn't work for me. May be I am missing something. Have you tried it? If it works for you, can you post a patch?

> hdfs script does not work after project split.
> ----------------------------------------------
>
>                 Key: HADOOP-6123
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6123
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Sharad Agarwal
>             Fix For: 0.21.0
>
>         Attachments: 6123_v1.patch
>
>
> There are problems running hdfs script from common.
> # Usage message for hdfs does not have "fs" option anymore.
> # There seem to be an undocumented option "dfs", but using it throws NoClassDefFoundError or ClassNotFoundException.
> # Same with other options e.g. name-node.
> May I am missing something here. How do we call ls these days?
> Do we need to move hdfs script to hdfs project.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-502-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 24 03:51:32 2009
Return-Path: <common-issues-return-502-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29099 invoked from network); 24 Jul 2009 03:51:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jul 2009 03:51:32 -0000
Received: (qmail 74088 invoked by uid 500); 24 Jul 2009 03:52:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74046 invoked by uid 500); 24 Jul 2009 03:52:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74036 invoked by uid 99); 24 Jul 2009 03:52:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 03:52:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 03:52:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C3A19234C004
	for <common-issues@hadoop.apache.org>; Thu, 23 Jul 2009 20:52:14 -0700 (PDT)
Message-ID: <1680582874.1248407534786.JavaMail.jira@brutus>
Date: Thu, 23 Jul 2009 20:52:14 -0700 (PDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6168) HADOOP_HEAPSIZE cannot be done
 per-server easily
In-Reply-To: <1003483123.1248379634876.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734892#action_12734892 ] 

Vinod K V commented on HADOOP-6168:
-----------------------------------


Though not the cleanest, there is a way to do this using various HADOOP_*_OPTS. We can set heap size also in these opts, for e.g. HADOOP_JOBTRACKER_OPTS=-Xmx2000m, it will be eventually picked up by the running command. It works this way because jvm takes up the last -Xmx value that occurs in the command line which would be from HADOOP_OPTS.

{code} exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS $CLASS "$@"{code}

> HADOOP_HEAPSIZE cannot be done per-server easily
> ------------------------------------------------
>
>                 Key: HADOOP-6168
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6168
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.18.3
>            Reporter: Allen Wittenauer
>
> The hadoop script forces a heap that cannot be easily overridden if one wants to push the same config everywhere.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-503-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 24 03:59:34 2009
Return-Path: <common-issues-return-503-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29789 invoked from network); 24 Jul 2009 03:59:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jul 2009 03:59:34 -0000
Received: (qmail 77997 invoked by uid 500); 24 Jul 2009 04:00:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77959 invoked by uid 500); 24 Jul 2009 04:00:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77948 invoked by uid 99); 24 Jul 2009 04:00:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 04:00:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 04:00:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C2981234C004
	for <common-issues@hadoop.apache.org>; Thu, 23 Jul 2009 21:00:14 -0700 (PDT)
Message-ID: <1583349091.1248408014784.JavaMail.jira@brutus>
Date: Thu, 23 Jul 2009 21:00:14 -0700 (PDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6168) HADOOP_HEAPSIZE cannot be done
 per-server easily
In-Reply-To: <1003483123.1248379634876.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734893#action_12734893 ] 

Vinod K V commented on HADOOP-6168:
-----------------------------------

Oh, and I think this issue is a duplicate of HADOOP-6167.

> HADOOP_HEAPSIZE cannot be done per-server easily
> ------------------------------------------------
>
>                 Key: HADOOP-6168
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6168
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.18.3
>            Reporter: Allen Wittenauer
>
> The hadoop script forces a heap that cannot be easily overridden if one wants to push the same config everywhere.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-504-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 24 04:17:33 2009
Return-Path: <common-issues-return-504-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32165 invoked from network); 24 Jul 2009 04:17:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jul 2009 04:17:33 -0000
Received: (qmail 85336 invoked by uid 500); 24 Jul 2009 04:18:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85264 invoked by uid 500); 24 Jul 2009 04:18:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85013 invoked by uid 99); 24 Jul 2009 04:18:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 04:18:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 04:18:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0D0F6234C053
	for <common-issues@hadoop.apache.org>; Thu, 23 Jul 2009 21:18:15 -0700 (PDT)
Message-ID: <1006773588.1248409095052.JavaMail.jira@brutus>
Date: Thu, 23 Jul 2009 21:18:15 -0700 (PDT)
From: "Amar Kamat (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4490) Map and Reduce tasks should run as
 the user who submitted the job
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amar Kamat updated HADOOP-4490:
-------------------------------

    Attachment:     (was: 2701949.4490.9.patch)

> Map and Reduce tasks should run as the user who submitted the job
> -----------------------------------------------------------------
>
>                 Key: HADOOP-4490
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4490
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>            Assignee: Hemanth Yamijala
>             Fix For: 0.21.0
>
>         Attachments: cluster_setup.pdf, cluster_setup.pdf, HADOOP-4490-1.patch, HADOOP-4490-1.patch, hadoop-4490-10.patch, hadoop-4490-11.patch, hadoop-4490-12.patch, hadoop-4490-13.patch, hadoop-4490-14.patch, HADOOP-4490-2.patch, HADOOP-4490-3.patch, hadoop-4490-4.patch, hadoop-4490-5.patch, hadoop-4490-6.patch, hadoop-4490-7.patch, hadoop-4490-8.patch, hadoop-4490-9.patch, hadoop-4490-br20-3.2.patch, hadoop-4490-br20-3.patch, hadoop-4490-design.pdf, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490_streaming.patch
>
>
> Currently the TaskTracker spawns the map/reduce tasks, resulting in them running as the user who started the TaskTracker.
> For security and accounting purposes the tasks should be run as the job-owner.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-505-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 24 05:21:32 2009
Return-Path: <common-issues-return-505-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43152 invoked from network); 24 Jul 2009 05:21:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jul 2009 05:21:32 -0000
Received: (qmail 14981 invoked by uid 500); 24 Jul 2009 05:22:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14943 invoked by uid 500); 24 Jul 2009 05:22:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14931 invoked by uid 99); 24 Jul 2009 05:22:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 05:22:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 05:22:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D684A234C051
	for <common-issues@hadoop.apache.org>; Thu, 23 Jul 2009 22:22:14 -0700 (PDT)
Message-ID: <1768250141.1248412934877.JavaMail.jira@brutus>
Date: Thu, 23 Jul 2009 22:22:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6164) Test-patch's method of checking for
 new tests is not correct
In-Reply-To: <1144829364.1248220514920.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734905#action_12734905 ] 

Giridharan Kesavan commented on HADOOP-6164:
--------------------------------------------

filed a jira for trackin 
https://issues.apache.org/jira/browse/HDFS-502
tnx!


> Test-patch's method of checking for new tests is not correct
> ------------------------------------------------------------
>
>                 Key: HADOOP-6164
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6164
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Jakob Homan
>            Assignee: Giridharan Kesavan
>
> As happened in HDFS-458, test-patch/hudson saw the previous patch(HDFS-492) add a new test, bringing the total number of tests to 49.  Then it tested 458, which had no new tests as it was a change to the build file.  492 had not yet been committed yet, so there were still only 48 tests in trunk.  But test-patch failed the patch, expecting 49 tests.  test-patch should check the correct number of tests based on trunk, not on whatever number it last saw.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-506-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 24 10:12:34 2009
Return-Path: <common-issues-return-506-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65928 invoked from network); 24 Jul 2009 10:12:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jul 2009 10:12:34 -0000
Received: (qmail 9612 invoked by uid 500); 24 Jul 2009 10:13:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9563 invoked by uid 500); 24 Jul 2009 10:13:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9540 invoked by uid 99); 24 Jul 2009 10:13:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 10:13:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 10:13:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DBD5C234C051
	for <common-issues@hadoop.apache.org>; Fri, 24 Jul 2009 03:13:14 -0700 (PDT)
Message-ID: <1239173575.1248430394899.JavaMail.jira@brutus>
Date: Fri, 24 Jul 2009 03:13:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6160) releaseaudit (rats) should not be run
 againt the entire release binary
In-Reply-To: <1162830772.1247830035100.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6160:
---------------------------------------

    Attachment: HADOOP-6160.patch

This patch is applicable for common, hdfs and mapred.


> releaseaudit (rats) should not be run againt the entire release binary
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6160
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6160
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>         Attachments: HADOOP-6160.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-507-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 24 10:12:34 2009
Return-Path: <common-issues-return-507-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65980 invoked from network); 24 Jul 2009 10:12:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jul 2009 10:12:34 -0000
Received: (qmail 9706 invoked by uid 500); 24 Jul 2009 10:13:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9629 invoked by uid 500); 24 Jul 2009 10:13:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9541 invoked by uid 99); 24 Jul 2009 10:13:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 10:13:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 10:13:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DEA70234C1E6
	for <common-issues@hadoop.apache.org>; Fri, 24 Jul 2009 03:13:14 -0700 (PDT)
Message-ID: <1395439769.1248430394911.JavaMail.jira@brutus>
Date: Fri, 24 Jul 2009 03:13:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6160) releaseaudit (rats) should not be
 run againt the entire release binary
In-Reply-To: <1162830772.1247830035100.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735016#action_12735016 ] 

Giridharan Kesavan commented on HADOOP-6160:
--------------------------------------------

can someone review this please?

> releaseaudit (rats) should not be run againt the entire release binary
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6160
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6160
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>         Attachments: HADOOP-6160.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-508-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 24 10:12:35 2009
Return-Path: <common-issues-return-508-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65991 invoked from network); 24 Jul 2009 10:12:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jul 2009 10:12:34 -0000
Received: (qmail 9728 invoked by uid 500); 24 Jul 2009 10:13:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9633 invoked by uid 500); 24 Jul 2009 10:13:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9542 invoked by uid 99); 24 Jul 2009 10:13:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 10:13:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 10:13:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E1570234C1E9
	for <common-issues@hadoop.apache.org>; Fri, 24 Jul 2009 03:13:14 -0700 (PDT)
Message-ID: <307256804.1248430394922.JavaMail.jira@brutus>
Date: Fri, 24 Jul 2009 03:13:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6160) releaseaudit (rats) should not be run
 againt the entire release binary
In-Reply-To: <1162830772.1247830035100.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6160:
---------------------------------------

    Status: Patch Available  (was: Open)

> releaseaudit (rats) should not be run againt the entire release binary
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6160
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6160
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>         Attachments: HADOOP-6160.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-509-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 24 17:57:53 2009
Return-Path: <common-issues-return-509-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47283 invoked from network); 24 Jul 2009 17:57:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jul 2009 17:57:53 -0000
Received: (qmail 16913 invoked by uid 500); 24 Jul 2009 17:58:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16869 invoked by uid 500); 24 Jul 2009 17:58:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16859 invoked by uid 99); 24 Jul 2009 17:58:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 17:58:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 17:58:56 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 03AB4234C1EF
	for <common-issues@hadoop.apache.org>; Fri, 24 Jul 2009 10:58:15 -0700 (PDT)
Message-ID: <1639348581.1248458295014.JavaMail.jira@brutus>
Date: Fri, 24 Jul 2009 10:58:15 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6123) hdfs script does not work after
 project split.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735111#action_12735111 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6123:
------------------------------------------------

I tried your for-loop patch.  It does not work.
{noformat}
bash-3.2$ ./bin/hdfs namenode
java.lang.NoClassDefFoundError: org/apache/hadoop/hdfs/server/namenode/NameNode
...
{noformat}

> ... Have you tried it? If it works for you, can you post a patch? 
I have not tried it.

> hdfs script does not work after project split.
> ----------------------------------------------
>
>                 Key: HADOOP-6123
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6123
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Sharad Agarwal
>             Fix For: 0.21.0
>
>         Attachments: 6123_v1.patch
>
>
> There are problems running hdfs script from common.
> # Usage message for hdfs does not have "fs" option anymore.
> # There seem to be an undocumented option "dfs", but using it throws NoClassDefFoundError or ClassNotFoundException.
> # Same with other options e.g. name-node.
> May I am missing something here. How do we call ls these days?
> Do we need to move hdfs script to hdfs project.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-510-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 24 19:44:32 2009
Return-Path: <common-issues-return-510-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91243 invoked from network); 24 Jul 2009 19:44:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jul 2009 19:44:32 -0000
Received: (qmail 70655 invoked by uid 500); 24 Jul 2009 19:45:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70607 invoked by uid 500); 24 Jul 2009 19:45:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70597 invoked by uid 99); 24 Jul 2009 19:45:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 19:45:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 19:45:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DE68B234C004
	for <common-issues@hadoop.apache.org>; Fri, 24 Jul 2009 12:45:14 -0700 (PDT)
Message-ID: <698749201.1248464714897.JavaMail.jira@brutus>
Date: Fri, 24 Jul 2009 12:45:14 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5318) Poor IO Performance due to
 AtomicLong operations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735138#action_12735138 ] 

Todd Lipcon commented on HADOOP-5318:
-------------------------------------

Given that we got the majority of speedup out of the CRC changes, I think we should resolve this ticket as wontfix. I just ran the same benchmark of trunk vs patched on my dual core laptop and the performance was essentially the same. On the Nehalem box it looked like there might have been a few percent speedup, but it seems not-worth-it for the added complexity.

If anyone disagrees (or has a benchmark to back up this patch), please speak up. I'll upload an updated patch against trunk so people can give it a try.

> Poor IO Performance due to AtomicLong operations
> ------------------------------------------------
>
>                 Key: HADOOP-5318
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5318
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.19.0
>         Environment: 2x quad core xeon linux 64 bit
>            Reporter: Ben Maurer
>            Assignee: Todd Lipcon
>         Attachments: buf.patch, buffer-output.patch, hadoop-5318.txt, hadoop-5318.txt, hadoop-5318.txt, Rplot001.png, TestWriteConcurrency.java, TestWriteConcurrency.java, TestWriteConcurrency.java
>
>
> The AtomicLong operations in counting file system statistics can cause high levels of contention with multiple threads. This test demonstrates having multiple threads writing to different sequence files:
> {code:java}
> import java.io.IOException;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.io.ByteWritable;
> import org.apache.hadoop.io.SequenceFile;
> import org.apache.hadoop.io.SequenceFile.Writer;
> import org.apache.hadoop.io.SequenceFile.CompressionType;
> public class Test {
> 	public static void main(String[] args) throws IOException {
> 		final Configuration c = new Configuration();
> 		final FileSystem fs = FileSystem.get(c);
> 		
> 		final int NUM = 1000*1000;
> 		for (int i = 0; i < Integer.valueOf(args[0]); i ++) {
> 			final int ii = i;
> 			new Thread(new Runnable() {
> 				@Override
> 				public void run() {
> 					
> 					try {
> 						Writer f = SequenceFile.createWriter(fs, c, new Path("/test/" + ii ), ByteWritable.class, ByteWritable.class, CompressionType.NONE);
> 						ByteWritable v = new ByteWritable();
> 						
> 						long time = System.currentTimeMillis();
> 						for (int i = 0; i < NUM; i ++)
> 							f.append(v, v);
> 						f.close();
> 						long end = System.currentTimeMillis();
> 						
> 						System.out.printf("%d opartions in %d msec. %f/second\n", NUM, end - time, (float)(1000 * NUM)/(end - time));
> 						
> 					} catch (Exception e) {
> 						// TODO Auto-generated catch block
> 						e.printStackTrace();
> 					}
> 					
> 				}
> 			}).start();
> 		}
> 	}
> }
> {code}
> The results of this benchmark are
> {code}
> ==== 1 threads ====
> 1000000 opartions in 1431 msec. 698812.000000/second
> ==== 2 threads ====
> 1000000 opartions in 3001 msec. 333222.250000/second
> 1000000 opartions in 2985 msec. 335008.375000/second
> ==== 3 threads ====
> 1000000 opartions in 4923 msec. 203128.171875/second
> 1000000 opartions in 4924 msec. 203086.921875/second
> 1000000 opartions in 4981 msec. 200762.906250/second
> ==== 4 threads ====
> 1000000 opartions in 6716 msec. 148898.156250/second
> 1000000 opartions in 7048 msec. 141884.218750/second
> 1000000 opartions in 7342 msec. 136202.671875/second
> 1000000 opartions in 7344 msec. 136165.578125/second
> ==== 5 threads ====
> 1000000 opartions in 10366 msec. 96469.226563/second
> 1000000 opartions in 11085 msec. 90212.000000/second
> 1000000 opartions in 11121 msec. 89919.968750/second
> 1000000 opartions in 11464 msec. 87229.585938/second
> 1000000 opartions in 11538 msec. 86670.132813/second
> ==== 6 threads ====
> 1000000 opartions in 16513 msec. 60558.347656/second
> 1000000 opartions in 17704 msec. 56484.410156/second
> 1000000 opartions in 18219 msec. 54887.753906/second
> 1000000 opartions in 18550 msec. 53908.355469/second
> 1000000 opartions in 18605 msec. 53748.992188/second
> 1000000 opartions in 18663 msec. 53581.953125/second
> ==== 7 threads ====
> 1000000 opartions in 22207 msec. 45030.847656/second
> 1000000 opartions in 23275 msec. 42964.554688/second
> 1000000 opartions in 23484 msec. 42582.183594/second
> 1000000 opartions in 24378 msec. 41020.593750/second
> 1000000 opartions in 24425 msec. 40941.656250/second
> 1000000 opartions in 24533 msec. 40761.421875/second
> 1000000 opartions in 24645 msec. 40576.183594/second
> ==== 8 threads ====
> 1000000 opartions in 26375 msec. 37914.691406/second
> 1000000 opartions in 26420 msec. 37850.113281/second
> 1000000 opartions in 26532 msec. 37690.335938/second
> 1000000 opartions in 26670 msec. 37495.312500/second
> 1000000 opartions in 29772 msec. 33588.605469/second
> 1000000 opartions in 29859 msec. 33490.738281/second
> 1000000 opartions in 30098 msec. 33224.800781/second
> 1000000 opartions in 30082 msec. 33242.468750/second
> {code}
> However, if I comment out the file system statistics increments, the benchmark improves to:
> {code}
> ==== 1 threads ====
> 1000000 opartions in 1194 msec. 837520.937500/second
> ==== 2 threads ====
> 1000000 opartions in 1433 msec. 697836.687500/second
> 1000000 opartions in 1433 msec. 697836.687500/second
> ==== 3 threads ====
> 1000000 opartions in 1643 msec. 608642.750000/second
> 1000000 opartions in 1643 msec. 608642.750000/second
> 1000000 opartions in 1639 msec. 610128.125000/second
> ==== 4 threads ====
> 1000000 opartions in 1886 msec. 530222.687500/second
> 1000000 opartions in 1886 msec. 530222.687500/second
> 1000000 opartions in 1886 msec. 530222.687500/second
> 1000000 opartions in 1899 msec. 526592.937500/second
> ==== 5 threads ====
> 1000000 opartions in 2065 msec. 484261.500000/second
> 1000000 opartions in 2066 msec. 484027.093750/second
> 1000000 opartions in 2067 msec. 483792.937500/second
> 1000000 opartions in 2066 msec. 484027.093750/second
> 1000000 opartions in 2066 msec. 484027.093750/second
> ==== 6 threads ====
> 1000000 opartions in 2151 msec. 464900.031250/second
> 1000000 opartions in 2111 msec. 473709.156250/second
> 1000000 opartions in 2153 msec. 464468.187500/second
> 1000000 opartions in 2114 msec. 473036.906250/second
> 1000000 opartions in 2113 msec. 473260.781250/second
> 1000000 opartions in 2112 msec. 473484.843750/second
> ==== 7 threads ====
> 1000000 opartions in 2368 msec. 422297.312500/second
> 1000000 opartions in 2334 msec. 428449.000000/second
> 1000000 opartions in 2332 msec. 428816.468750/second
> 1000000 opartions in 2330 msec. 429184.562500/second
> 1000000 opartions in 2332 msec. 428816.468750/second
> 1000000 opartions in 2375 msec. 421052.625000/second
> 1000000 opartions in 2394 msec. 417710.937500/second
> ==== 8 threads ====
> 1000000 opartions in 2517 msec. 397298.375000/second
> 1000000 opartions in 2538 msec. 394011.031250/second
> 1000000 opartions in 2538 msec. 394011.031250/second
> 1000000 opartions in 2538 msec. 394011.031250/second
> 1000000 opartions in 2539 msec. 393855.843750/second
> 1000000 opartions in 2614 msec. 382555.468750/second
> 1000000 opartions in 2666 msec. 375093.781250/second
> 1000000 opartions in 2701 msec. 370233.250000/second
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-511-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 24 19:44:35 2009
Return-Path: <common-issues-return-511-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91291 invoked from network); 24 Jul 2009 19:44:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jul 2009 19:44:35 -0000
Received: (qmail 70734 invoked by uid 500); 24 Jul 2009 19:45:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70690 invoked by uid 500); 24 Jul 2009 19:45:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70680 invoked by uid 99); 24 Jul 2009 19:45:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 19:45:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 19:45:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 17F54234C495
	for <common-issues@hadoop.apache.org>; Fri, 24 Jul 2009 12:45:15 -0700 (PDT)
Message-ID: <970895747.1248464715097.JavaMail.jira@brutus>
Date: Fri, 24 Jul 2009 12:45:15 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5318) Poor IO Performance due to AtomicLong
 operations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-5318:
--------------------------------

    Attachment: hadoop-5318.txt

Patch against trunk common

> Poor IO Performance due to AtomicLong operations
> ------------------------------------------------
>
>                 Key: HADOOP-5318
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5318
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.19.0
>         Environment: 2x quad core xeon linux 64 bit
>            Reporter: Ben Maurer
>            Assignee: Todd Lipcon
>         Attachments: buf.patch, buffer-output.patch, hadoop-5318.txt, hadoop-5318.txt, hadoop-5318.txt, Rplot001.png, TestWriteConcurrency.java, TestWriteConcurrency.java, TestWriteConcurrency.java
>
>
> The AtomicLong operations in counting file system statistics can cause high levels of contention with multiple threads. This test demonstrates having multiple threads writing to different sequence files:
> {code:java}
> import java.io.IOException;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.io.ByteWritable;
> import org.apache.hadoop.io.SequenceFile;
> import org.apache.hadoop.io.SequenceFile.Writer;
> import org.apache.hadoop.io.SequenceFile.CompressionType;
> public class Test {
> 	public static void main(String[] args) throws IOException {
> 		final Configuration c = new Configuration();
> 		final FileSystem fs = FileSystem.get(c);
> 		
> 		final int NUM = 1000*1000;
> 		for (int i = 0; i < Integer.valueOf(args[0]); i ++) {
> 			final int ii = i;
> 			new Thread(new Runnable() {
> 				@Override
> 				public void run() {
> 					
> 					try {
> 						Writer f = SequenceFile.createWriter(fs, c, new Path("/test/" + ii ), ByteWritable.class, ByteWritable.class, CompressionType.NONE);
> 						ByteWritable v = new ByteWritable();
> 						
> 						long time = System.currentTimeMillis();
> 						for (int i = 0; i < NUM; i ++)
> 							f.append(v, v);
> 						f.close();
> 						long end = System.currentTimeMillis();
> 						
> 						System.out.printf("%d opartions in %d msec. %f/second\n", NUM, end - time, (float)(1000 * NUM)/(end - time));
> 						
> 					} catch (Exception e) {
> 						// TODO Auto-generated catch block
> 						e.printStackTrace();
> 					}
> 					
> 				}
> 			}).start();
> 		}
> 	}
> }
> {code}
> The results of this benchmark are
> {code}
> ==== 1 threads ====
> 1000000 opartions in 1431 msec. 698812.000000/second
> ==== 2 threads ====
> 1000000 opartions in 3001 msec. 333222.250000/second
> 1000000 opartions in 2985 msec. 335008.375000/second
> ==== 3 threads ====
> 1000000 opartions in 4923 msec. 203128.171875/second
> 1000000 opartions in 4924 msec. 203086.921875/second
> 1000000 opartions in 4981 msec. 200762.906250/second
> ==== 4 threads ====
> 1000000 opartions in 6716 msec. 148898.156250/second
> 1000000 opartions in 7048 msec. 141884.218750/second
> 1000000 opartions in 7342 msec. 136202.671875/second
> 1000000 opartions in 7344 msec. 136165.578125/second
> ==== 5 threads ====
> 1000000 opartions in 10366 msec. 96469.226563/second
> 1000000 opartions in 11085 msec. 90212.000000/second
> 1000000 opartions in 11121 msec. 89919.968750/second
> 1000000 opartions in 11464 msec. 87229.585938/second
> 1000000 opartions in 11538 msec. 86670.132813/second
> ==== 6 threads ====
> 1000000 opartions in 16513 msec. 60558.347656/second
> 1000000 opartions in 17704 msec. 56484.410156/second
> 1000000 opartions in 18219 msec. 54887.753906/second
> 1000000 opartions in 18550 msec. 53908.355469/second
> 1000000 opartions in 18605 msec. 53748.992188/second
> 1000000 opartions in 18663 msec. 53581.953125/second
> ==== 7 threads ====
> 1000000 opartions in 22207 msec. 45030.847656/second
> 1000000 opartions in 23275 msec. 42964.554688/second
> 1000000 opartions in 23484 msec. 42582.183594/second
> 1000000 opartions in 24378 msec. 41020.593750/second
> 1000000 opartions in 24425 msec. 40941.656250/second
> 1000000 opartions in 24533 msec. 40761.421875/second
> 1000000 opartions in 24645 msec. 40576.183594/second
> ==== 8 threads ====
> 1000000 opartions in 26375 msec. 37914.691406/second
> 1000000 opartions in 26420 msec. 37850.113281/second
> 1000000 opartions in 26532 msec. 37690.335938/second
> 1000000 opartions in 26670 msec. 37495.312500/second
> 1000000 opartions in 29772 msec. 33588.605469/second
> 1000000 opartions in 29859 msec. 33490.738281/second
> 1000000 opartions in 30098 msec. 33224.800781/second
> 1000000 opartions in 30082 msec. 33242.468750/second
> {code}
> However, if I comment out the file system statistics increments, the benchmark improves to:
> {code}
> ==== 1 threads ====
> 1000000 opartions in 1194 msec. 837520.937500/second
> ==== 2 threads ====
> 1000000 opartions in 1433 msec. 697836.687500/second
> 1000000 opartions in 1433 msec. 697836.687500/second
> ==== 3 threads ====
> 1000000 opartions in 1643 msec. 608642.750000/second
> 1000000 opartions in 1643 msec. 608642.750000/second
> 1000000 opartions in 1639 msec. 610128.125000/second
> ==== 4 threads ====
> 1000000 opartions in 1886 msec. 530222.687500/second
> 1000000 opartions in 1886 msec. 530222.687500/second
> 1000000 opartions in 1886 msec. 530222.687500/second
> 1000000 opartions in 1899 msec. 526592.937500/second
> ==== 5 threads ====
> 1000000 opartions in 2065 msec. 484261.500000/second
> 1000000 opartions in 2066 msec. 484027.093750/second
> 1000000 opartions in 2067 msec. 483792.937500/second
> 1000000 opartions in 2066 msec. 484027.093750/second
> 1000000 opartions in 2066 msec. 484027.093750/second
> ==== 6 threads ====
> 1000000 opartions in 2151 msec. 464900.031250/second
> 1000000 opartions in 2111 msec. 473709.156250/second
> 1000000 opartions in 2153 msec. 464468.187500/second
> 1000000 opartions in 2114 msec. 473036.906250/second
> 1000000 opartions in 2113 msec. 473260.781250/second
> 1000000 opartions in 2112 msec. 473484.843750/second
> ==== 7 threads ====
> 1000000 opartions in 2368 msec. 422297.312500/second
> 1000000 opartions in 2334 msec. 428449.000000/second
> 1000000 opartions in 2332 msec. 428816.468750/second
> 1000000 opartions in 2330 msec. 429184.562500/second
> 1000000 opartions in 2332 msec. 428816.468750/second
> 1000000 opartions in 2375 msec. 421052.625000/second
> 1000000 opartions in 2394 msec. 417710.937500/second
> ==== 8 threads ====
> 1000000 opartions in 2517 msec. 397298.375000/second
> 1000000 opartions in 2538 msec. 394011.031250/second
> 1000000 opartions in 2538 msec. 394011.031250/second
> 1000000 opartions in 2538 msec. 394011.031250/second
> 1000000 opartions in 2539 msec. 393855.843750/second
> 1000000 opartions in 2614 msec. 382555.468750/second
> 1000000 opartions in 2666 msec. 375093.781250/second
> 1000000 opartions in 2701 msec. 370233.250000/second
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-512-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 24 21:43:39 2009
Return-Path: <common-issues-return-512-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30958 invoked from network); 24 Jul 2009 21:43:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jul 2009 21:43:39 -0000
Received: (qmail 15855 invoked by uid 500); 24 Jul 2009 21:44:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15811 invoked by uid 500); 24 Jul 2009 21:44:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15797 invoked by uid 99); 24 Jul 2009 21:44:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 21:44:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 21:44:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E03BB234C004
	for <common-issues@hadoop.apache.org>; Fri, 24 Jul 2009 14:44:14 -0700 (PDT)
Message-ID: <1150274237.1248471854902.JavaMail.jira@brutus>
Date: Fri, 24 Jul 2009 14:44:14 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4491) Per-job local data on the
 TaskTracker node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735175#action_12735175 ] 

Owen O'Malley commented on HADOOP-4491:
---------------------------------------

After a little playing around with Linux, it looks like it blocks using hard links for directories, even for root. *frown*

That leaves us two options for dealing with the logs directory:
  1. Create the files and make hard links on the files and make sure the files are appended to.
  2. Use acls on the directories to limit access to just the mapred and job user.

I'm leaning toward the second one.

{noformat}
$ttroot - mapred, 755
   |
   |- jobs - mapred 755
   |     |
   |     '-- $jobid - mapred 700 + rw access by $user
   |            |
   |            |- distcache - mapred 755
   |            |- jars - mapred 755
   |            |    '-- job.jar
   |            `-- $attemptid - mapred 755
   |                  |- job.xml
   |                  |- taskjvm.sh
   |                  |- work - $user 700
   |                  |- logs - $user 755
   |                  `-- output - $user 755
   '-- system
        |
        `-- $jobid
             |
             |- job.xml
{noformat}

So all of the protection is at the $jobid level. The user only has write access to the work, logs, and output directories.

Thoughts?

> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-513-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 24 22:22:41 2009
Return-Path: <common-issues-return-513-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45869 invoked from network); 24 Jul 2009 22:22:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Jul 2009 22:22:40 -0000
Received: (qmail 43731 invoked by uid 500); 24 Jul 2009 22:23:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43688 invoked by uid 500); 24 Jul 2009 22:23:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43677 invoked by uid 99); 24 Jul 2009 22:23:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 22:23:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jul 2009 22:23:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0D9C2234C004
	for <common-issues@hadoop.apache.org>; Fri, 24 Jul 2009 15:23:15 -0700 (PDT)
Message-ID: <1364409221.1248474195040.JavaMail.jira@brutus>
Date: Fri, 24 Jul 2009 15:23:15 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6124) patchJavacWarnings and
 trunkJavacWarnings are not consistent.
In-Reply-To: <1339799848.1246409207795.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735189#action_12735189 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6124:
------------------------------------------------

HADOOP-6124.patch has some good changes for HADOOP-6077 but it did not fix the problem in this issue.
{noformat}
     [exec] There appear to be 122 javac compiler warnings before the patch and  
64 javac compiler warnings after applying the patch.
{noformat}

> patchJavacWarnings and trunkJavacWarnings are not consistent.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6124
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6124
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Giridharan Kesavan
>            Priority: Critical
>         Attachments: c6124_20090701.patch, HADOOP-6124.patch
>
>
> The values of patchJavacWarnings and trunkJavacWarnings are not consistent when running test-patch.sh with an empty patch over Common.  HDFS and MapReduce seem not having this problem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-514-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 25 02:01:35 2009
Return-Path: <common-issues-return-514-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9540 invoked from network); 25 Jul 2009 02:01:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Jul 2009 02:01:33 -0000
Received: (qmail 85730 invoked by uid 500); 25 Jul 2009 02:02:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85621 invoked by uid 500); 25 Jul 2009 02:02:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85511 invoked by uid 99); 25 Jul 2009 02:02:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jul 2009 02:02:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jul 2009 02:02:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CDBA8234C046
	for <common-issues@hadoop.apache.org>; Fri, 24 Jul 2009 19:02:14 -0700 (PDT)
Message-ID: <927826654.1248487334841.JavaMail.jira@brutus>
Date: Fri, 24 Jul 2009 19:02:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6169) Removing deprecated method calls in
 TFile
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Removing deprecated method calls in TFile
-----------------------------------------

                 Key: HADOOP-6169
                 URL: https://issues.apache.org/jira/browse/HADOOP-6169
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Hong Tang
            Priority: Minor


We should remove the use of deprecated APIs in TFile.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-515-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 25 09:27:37 2009
Return-Path: <common-issues-return-515-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98857 invoked from network); 25 Jul 2009 09:27:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Jul 2009 09:27:34 -0000
Received: (qmail 25320 invoked by uid 500); 25 Jul 2009 09:28:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25274 invoked by uid 500); 25 Jul 2009 09:28:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25264 invoked by uid 99); 25 Jul 2009 09:28:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jul 2009 09:28:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jul 2009 09:28:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C2040234C004
	for <common-issues@hadoop.apache.org>; Sat, 25 Jul 2009 02:28:14 -0700 (PDT)
Message-ID: <1305824929.1248514094780.JavaMail.jira@brutus>
Date: Sat, 25 Jul 2009 02:28:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6169) Removing deprecated method calls in
 TFile
In-Reply-To: <927826654.1248487334841.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-6169:
------------------------------

    Attachment: hadoop-6169-20090725.patch

> Removing deprecated method calls in TFile
> -----------------------------------------
>
>                 Key: HADOOP-6169
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6169
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hong Tang
>            Priority: Minor
>         Attachments: hadoop-6169-20090725.patch
>
>
> We should remove the use of deprecated APIs in TFile.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-516-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 25 09:31:34 2009
Return-Path: <common-issues-return-516-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1404 invoked from network); 25 Jul 2009 09:31:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Jul 2009 09:31:34 -0000
Received: (qmail 25911 invoked by uid 500); 25 Jul 2009 09:32:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25867 invoked by uid 500); 25 Jul 2009 09:32:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25857 invoked by uid 99); 25 Jul 2009 09:32:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jul 2009 09:32:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jul 2009 09:32:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1366F234C004
	for <common-issues@hadoop.apache.org>; Sat, 25 Jul 2009 02:32:15 -0700 (PDT)
Message-ID: <418718750.1248514335064.JavaMail.jira@brutus>
Date: Sat, 25 Jul 2009 02:32:15 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6169) Removing deprecated method calls in
 TFile
In-Reply-To: <927826654.1248487334841.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang reassigned HADOOP-6169:
---------------------------------

    Assignee: Hong Tang

> Removing deprecated method calls in TFile
> -----------------------------------------
>
>                 Key: HADOOP-6169
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6169
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>         Attachments: hadoop-6169-20090725.patch
>
>
> We should remove the use of deprecated APIs in TFile.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-517-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 25 09:45:32 2009
Return-Path: <common-issues-return-517-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6994 invoked from network); 25 Jul 2009 09:45:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Jul 2009 09:45:32 -0000
Received: (qmail 33212 invoked by uid 500); 25 Jul 2009 09:46:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33166 invoked by uid 500); 25 Jul 2009 09:46:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33155 invoked by uid 99); 25 Jul 2009 09:46:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jul 2009 09:46:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jul 2009 09:46:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C2DC2234C004
	for <common-issues@hadoop.apache.org>; Sat, 25 Jul 2009 02:46:14 -0700 (PDT)
Message-ID: <2095452071.1248515174783.JavaMail.jira@brutus>
Date: Sat, 25 Jul 2009 02:46:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6169) Removing deprecated method calls in
 TFile
In-Reply-To: <927826654.1248487334841.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735255#action_12735255 ] 

Hong Tang commented on HADOOP-6169:
-----------------------------------

 test-patch output:

     [exec] +1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 9 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec] 

> Removing deprecated method calls in TFile
> -----------------------------------------
>
>                 Key: HADOOP-6169
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6169
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>         Attachments: hadoop-6169-20090725.patch
>
>
> We should remove the use of deprecated APIs in TFile.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-518-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 25 09:47:32 2009
Return-Path: <common-issues-return-518-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7992 invoked from network); 25 Jul 2009 09:47:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Jul 2009 09:47:32 -0000
Received: (qmail 36150 invoked by uid 500); 25 Jul 2009 09:48:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36106 invoked by uid 500); 25 Jul 2009 09:48:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36096 invoked by uid 99); 25 Jul 2009 09:48:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jul 2009 09:48:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jul 2009 09:48:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C7E96234C004
	for <common-issues@hadoop.apache.org>; Sat, 25 Jul 2009 02:48:14 -0700 (PDT)
Message-ID: <1721978382.1248515294804.JavaMail.jira@brutus>
Date: Sat, 25 Jul 2009 02:48:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6169) Removing deprecated method calls in
 TFile
In-Reply-To: <927826654.1248487334841.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-6169:
------------------------------

        Fix Version/s: 0.21.0
    Affects Version/s: 0.21.0
               Status: Patch Available  (was: Open)

Trivial patch that eliminates the use of deprecated apis. I also fixed a few findbugs warnings along the way.

> Removing deprecated method calls in TFile
> -----------------------------------------
>
>                 Key: HADOOP-6169
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6169
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6169-20090725.patch
>
>
> We should remove the use of deprecated APIs in TFile.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-519-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 25 13:39:35 2009
Return-Path: <common-issues-return-519-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66838 invoked from network); 25 Jul 2009 13:39:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Jul 2009 13:39:35 -0000
Received: (qmail 40470 invoked by uid 500); 25 Jul 2009 13:40:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40425 invoked by uid 500); 25 Jul 2009 13:40:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40415 invoked by uid 99); 25 Jul 2009 13:40:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jul 2009 13:40:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jul 2009 13:40:37 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 40AFC234C004
	for <common-issues@hadoop.apache.org>; Sat, 25 Jul 2009 06:40:16 -0700 (PDT)
Message-ID: <442782326.1248529216117.JavaMail.jira@brutus>
Date: Sat, 25 Jul 2009 06:40:16 -0700 (PDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4491) Per-job local data on the
 TaskTracker node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735272#action_12735272 ] 

Hemanth Yamijala commented on HADOOP-4491:
------------------------------------------

bq. Use acls on the directories to limit access to just the mapred and job user.

Are you referring to the ACLs that use extended attributes on the file system. Their support should be explicitly enabled in the kernel, and also supported by the file system type, right ? If yes, doesn't this approach make the utility of this feature much more restrictive ?

It seems like option 1 is a slightly better option in that case... 

> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-520-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jul 26 12:40:34 2009
Return-Path: <common-issues-return-520-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59287 invoked from network); 26 Jul 2009 12:40:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Jul 2009 12:40:34 -0000
Received: (qmail 39594 invoked by uid 500); 26 Jul 2009 12:41:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39567 invoked by uid 500); 26 Jul 2009 12:41:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39557 invoked by uid 99); 26 Jul 2009 12:41:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jul 2009 12:41:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jul 2009 12:41:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4697A234C004
	for <common-issues@hadoop.apache.org>; Sun, 26 Jul 2009 05:41:15 -0700 (PDT)
Message-ID: <1446874848.1248612075274.JavaMail.jira@brutus>
Date: Sun, 26 Jul 2009 05:41:15 -0700 (PDT)
From: "He Yongqiang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5879) GzipCodec should read compression
 level etc from configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

He Yongqiang updated HADOOP-5879:
---------------------------------

    Attachment: hadoop-5879-7-26.patch

hadoop-5879-7-26.patch incorporates all Chris's suggestions (Thanks for the detailed comments, Chris!). 
I have to admit that the testcase can not fully verify the functionality. And to know that it works, what i did was by looking the log and debugging in my IDE. Please let me know if you know how we can test it automatically.

> GzipCodec should read compression level etc from configuration
> --------------------------------------------------------------
>
>                 Key: HADOOP-5879
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5879
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Zheng Shao
>            Assignee: He Yongqiang
>         Attachments: hadoop-5879-5-21.patch, hadoop-5879-7-13-2.patch, hadoop-5879-7-13-3.patch, hadoop-5879-7-18-3.patch, hadoop-5879-7-26.patch
>
>
> GzipCodec currently uses the default compression level. We should allow overriding the default value from Configuration.
> {code}
>   static final class GzipZlibCompressor extends ZlibCompressor {
>     public GzipZlibCompressor() {
>       super(ZlibCompressor.CompressionLevel.DEFAULT_COMPRESSION,
>           ZlibCompressor.CompressionStrategy.DEFAULT_STRATEGY,
>           ZlibCompressor.CompressionHeader.GZIP_FORMAT, 64*1024);
>     }
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-521-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 05:27:34 2009
Return-Path: <common-issues-return-521-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54024 invoked from network); 27 Jul 2009 05:27:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 05:27:33 -0000
Received: (qmail 7897 invoked by uid 500); 27 Jul 2009 05:28:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7851 invoked by uid 500); 27 Jul 2009 05:28:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7841 invoked by uid 99); 27 Jul 2009 05:28:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 05:28:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 05:28:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D3845234C04B
	for <common-issues@hadoop.apache.org>; Sun, 26 Jul 2009 22:28:14 -0700 (PDT)
Message-ID: <1585515553.1248672494865.JavaMail.jira@brutus>
Date: Sun, 26 Jul 2009 22:28:14 -0700 (PDT)
From: "Edward J. Yoon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5988) Add a command to ' FsShell stat '
 to get a file's block location information
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735467#action_12735467 ] 

Edward J. Yoon commented on HADOOP-5988:
----------------------------------------

Do you mean the "-stat" option of 'ls' command?

> Add a command to ' FsShell stat ' to get a file's block location information
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5988
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5988
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: He Yongqiang
>
> Adding an option to ' FsShell stat ' to get a file's block location information will be very useful.
> we can print the block location information in this format:
> blockID:XXXXX  byte-range:YYYY-ZZZZ  location:dn1;dn2;
> blockID:XXXXX  byte-range:YYYY-ZZZZ  location:dn1;dn2;

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-522-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 05:35:34 2009
Return-Path: <common-issues-return-522-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57724 invoked from network); 27 Jul 2009 05:35:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 05:35:34 -0000
Received: (qmail 8893 invoked by uid 500); 27 Jul 2009 05:36:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8845 invoked by uid 500); 27 Jul 2009 05:36:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8830 invoked by uid 99); 27 Jul 2009 05:36:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 05:36:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 05:36:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 33CF2234C004
	for <common-issues@hadoop.apache.org>; Sun, 26 Jul 2009 22:36:15 -0700 (PDT)
Message-ID: <1128261947.1248672975198.JavaMail.jira@brutus>
Date: Sun, 26 Jul 2009 22:36:15 -0700 (PDT)
From: "He Yongqiang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5988) Add a command to ' FsShell stat '
 to get a file's block location information
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735469#action_12735469 ] 

He Yongqiang commented on HADOOP-5988:
--------------------------------------

yeah.  like "hadoop dfs -stat %B path_to_file".  

> Add a command to ' FsShell stat ' to get a file's block location information
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5988
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5988
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: He Yongqiang
>
> Adding an option to ' FsShell stat ' to get a file's block location information will be very useful.
> we can print the block location information in this format:
> blockID:XXXXX  byte-range:YYYY-ZZZZ  location:dn1;dn2;
> blockID:XXXXX  byte-range:YYYY-ZZZZ  location:dn1;dn2;

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-523-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 06:17:36 2009
Return-Path: <common-issues-return-523-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75377 invoked from network); 27 Jul 2009 06:17:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 06:17:34 -0000
Received: (qmail 39557 invoked by uid 500); 27 Jul 2009 06:18:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39496 invoked by uid 500); 27 Jul 2009 06:18:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39473 invoked by uid 99); 27 Jul 2009 06:18:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 06:18:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 06:18:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id ECE47234C052
	for <common-issues@hadoop.apache.org>; Sun, 26 Jul 2009 23:18:14 -0700 (PDT)
Message-ID: <460680504.1248675494955.JavaMail.jira@brutus>
Date: Sun, 26 Jul 2009 23:18:14 -0700 (PDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6123) hdfs script does not work after
 project split.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735490#action_12735490 ] 

Sharad Agarwal commented on HADOOP-6123:
----------------------------------------

bq. I tried your for-loop patch. It does not work.
It works for me with latest trunk. By any chance you are running the command in common itself ? It should be run in HDFS. See my [comment |https://issues.apache.org/jira/browse/HADOOP-6123?focusedCommentId=12725932&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12725932] above.

> hdfs script does not work after project split.
> ----------------------------------------------
>
>                 Key: HADOOP-6123
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6123
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Sharad Agarwal
>             Fix For: 0.21.0
>
>         Attachments: 6123_v1.patch
>
>
> There are problems running hdfs script from common.
> # Usage message for hdfs does not have "fs" option anymore.
> # There seem to be an undocumented option "dfs", but using it throws NoClassDefFoundError or ClassNotFoundException.
> # Same with other options e.g. name-node.
> May I am missing something here. How do we call ls these days?
> Do we need to move hdfs script to hdfs project.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-524-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 06:55:34 2009
Return-Path: <common-issues-return-524-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93082 invoked from network); 27 Jul 2009 06:55:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 06:55:34 -0000
Received: (qmail 57531 invoked by uid 500); 27 Jul 2009 06:56:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57485 invoked by uid 500); 27 Jul 2009 06:56:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57475 invoked by uid 99); 27 Jul 2009 06:56:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 06:56:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 06:56:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3DB7D234C046
	for <common-issues@hadoop.apache.org>; Sun, 26 Jul 2009 23:56:15 -0700 (PDT)
Message-ID: <660573122.1248677775237.JavaMail.jira@brutus>
Date: Sun, 26 Jul 2009 23:56:15 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6105) Provide a way to automatically handle
 backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna updated HADOOP-6105:
------------------------------------------

    Attachment: HADOOP-6105.patch

Uploading the patch

> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>         Attachments: HADOOP-6105.patch
>
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-525-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 06:59:34 2009
Return-Path: <common-issues-return-525-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93769 invoked from network); 27 Jul 2009 06:59:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 06:59:34 -0000
Received: (qmail 59439 invoked by uid 500); 27 Jul 2009 07:00:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59395 invoked by uid 500); 27 Jul 2009 07:00:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59385 invoked by uid 99); 27 Jul 2009 07:00:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 07:00:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 07:00:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CBF9C234C046
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 00:00:14 -0700 (PDT)
Message-ID: <1221739052.1248678014826.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 00:00:14 -0700 (PDT)
From: "Edward J. Yoon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5988) Add a command to ' FsShell stat '
 to get a file's block location information
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735498#action_12735498 ] 

Edward J. Yoon commented on HADOOP-5988:
----------------------------------------

I see. I tried using getFileBlockLocations().

BlockLocation[] locations = srcFs.getFileBlockLocations(f, 0, f.getLen());
for(int ii = 0; ii < locations.length; ii++) {
  buf.append(locations[ii].toString());
}

But, I can't get a blockid. Should I use the DFSClient?

> Add a command to ' FsShell stat ' to get a file's block location information
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5988
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5988
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: He Yongqiang
>
> Adding an option to ' FsShell stat ' to get a file's block location information will be very useful.
> we can print the block location information in this format:
> blockID:XXXXX  byte-range:YYYY-ZZZZ  location:dn1;dn2;
> blockID:XXXXX  byte-range:YYYY-ZZZZ  location:dn1;dn2;

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-526-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 07:19:35 2009
Return-Path: <common-issues-return-526-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97248 invoked from network); 27 Jul 2009 07:19:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 07:19:32 -0000
Received: (qmail 69877 invoked by uid 500); 27 Jul 2009 07:20:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69831 invoked by uid 500); 27 Jul 2009 07:20:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69821 invoked by uid 99); 27 Jul 2009 07:20:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 07:20:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 07:20:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D3307234C046
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 00:20:14 -0700 (PDT)
Message-ID: <1443371822.1248679214853.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 00:20:14 -0700 (PDT)
From: "He Yongqiang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5988) Add a command to ' FsShell stat '
 to get a file's block location information
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735508#action_12735508 ] 

He Yongqiang commented on HADOOP-5988:
--------------------------------------

DFSClient is from hdfs, and FsShell is in common. After project splitting, it becomes difficuty to modify the core and hdfs at the same time. 
And FsShell should works for all file systems, which DFSClient is only for hdfs. 
I think if can not get blocks' ids, sortting the output according to byte-range is pretty much the same.

> Add a command to ' FsShell stat ' to get a file's block location information
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5988
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5988
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: He Yongqiang
>
> Adding an option to ' FsShell stat ' to get a file's block location information will be very useful.
> we can print the block location information in this format:
> blockID:XXXXX  byte-range:YYYY-ZZZZ  location:dn1;dn2;
> blockID:XXXXX  byte-range:YYYY-ZZZZ  location:dn1;dn2;

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-527-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 08:08:35 2009
Return-Path: <common-issues-return-527-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9971 invoked from network); 27 Jul 2009 08:08:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 08:08:35 -0000
Received: (qmail 8870 invoked by uid 500); 27 Jul 2009 08:09:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8817 invoked by uid 500); 27 Jul 2009 08:09:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8807 invoked by uid 99); 27 Jul 2009 08:09:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 08:09:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 08:09:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DB4CE234C04B
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 01:09:14 -0700 (PDT)
Message-ID: <469525524.1248682154897.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 01:09:14 -0700 (PDT)
From: "Edward J. Yoon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5988) Add a command to ' FsShell stat ' to
 get a file's block location information
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Edward J. Yoon updated HADOOP-5988:
-----------------------------------

    Attachment: HADOOP-5988.patch

OK, thanks for your comments. Attach my patch.

> Add a command to ' FsShell stat ' to get a file's block location information
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5988
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5988
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: He Yongqiang
>         Attachments: HADOOP-5988.patch
>
>
> Adding an option to ' FsShell stat ' to get a file's block location information will be very useful.
> we can print the block location information in this format:
> blockID:XXXXX  byte-range:YYYY-ZZZZ  location:dn1;dn2;
> blockID:XXXXX  byte-range:YYYY-ZZZZ  location:dn1;dn2;

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-528-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 08:16:34 2009
Return-Path: <common-issues-return-528-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12062 invoked from network); 27 Jul 2009 08:16:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 08:16:34 -0000
Received: (qmail 13321 invoked by uid 500); 27 Jul 2009 08:17:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13275 invoked by uid 500); 27 Jul 2009 08:17:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13265 invoked by uid 99); 27 Jul 2009 08:17:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 08:17:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 08:17:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1CCEC234C046
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 01:17:15 -0700 (PDT)
Message-ID: <1899776844.1248682635105.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 01:17:15 -0700 (PDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6162) MapFile doesn't work with
 serializables other than Writables
In-Reply-To: <1542829149.1246995374880.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735517#action_12735517 ] 

Sharad Agarwal commented on HADOOP-6162:
----------------------------------------

Few comments on the patch:
There are lot of javac warnings. We should get rid of them.
Tests for ArrayFile, BloomMapFile and SetFile also should be added to their respective junit tests.
Should we deprecate constructors takings Writables ?


> MapFile doesn't work with serializables other than Writables
> ------------------------------------------------------------
>
>                 Key: HADOOP-6162
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6162
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Justin Patterson
>         Attachments: HADOOP-6162.patch
>
>
> Since 0.18 (I think), SequenceFiles have supported serializing arbitrary objects through the serialization framework.  MapFiles still don't.  They require WritableComparable keys and Writable values.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-529-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 08:20:35 2009
Return-Path: <common-issues-return-529-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12574 invoked from network); 27 Jul 2009 08:20:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 08:20:35 -0000
Received: (qmail 16004 invoked by uid 500); 27 Jul 2009 08:21:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15955 invoked by uid 500); 27 Jul 2009 08:21:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15945 invoked by uid 99); 27 Jul 2009 08:21:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 08:21:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 08:21:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CFEB5234C04B
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 01:21:14 -0700 (PDT)
Message-ID: <2062775542.1248682874850.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 01:21:14 -0700 (PDT)
From: "He Yongqiang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5988) Add a command to ' FsShell stat '
 to get a file's block location information
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735519#action_12735519 ] 

He Yongqiang commented on HADOOP-5988:
--------------------------------------

I think the patch can work for normal situations.
can you give some example outputs? like: stat a local path, stat a hdfs path, stat a non-exist path, stat a directory etc. For more eaier to read, i think the output should be indented (especially the byte range part), what do you think?

> Add a command to ' FsShell stat ' to get a file's block location information
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5988
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5988
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: He Yongqiang
>         Attachments: HADOOP-5988.patch
>
>
> Adding an option to ' FsShell stat ' to get a file's block location information will be very useful.
> we can print the block location information in this format:
> blockID:XXXXX  byte-range:YYYY-ZZZZ  location:dn1;dn2;
> blockID:XXXXX  byte-range:YYYY-ZZZZ  location:dn1;dn2;

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-530-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 08:24:34 2009
Return-Path: <common-issues-return-530-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13061 invoked from network); 27 Jul 2009 08:24:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 08:24:34 -0000
Received: (qmail 18465 invoked by uid 500); 27 Jul 2009 08:25:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18416 invoked by uid 500); 27 Jul 2009 08:25:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18406 invoked by uid 99); 27 Jul 2009 08:25:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 08:25:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 08:25:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CD9AA234C046
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 01:25:14 -0700 (PDT)
Message-ID: <30399517.1248683114827.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 01:25:14 -0700 (PDT)
From: "Daehyun Kim (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6153) RAgzip: multiple map tasks for a
 large gzipped file
In-Reply-To: <926568478.1247710994848.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daehyun Kim updated HADOOP-6153:
--------------------------------

    Attachment: HADOOP-6153.patch

1. To merge RAGzip with GzipCodec class, I add a method 'getDecompressorForRandomAccess(...)' to GzipCodec class. This method return the Decompressor for random access.

2. I add the '-makeap' option that is FS Shell Command. This option supports to make the access point of the gzip.

> RAgzip: multiple map tasks for a large gzipped file
> ---------------------------------------------------
>
>                 Key: HADOOP-6153
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6153
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, native
>         Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
>            Reporter: Daehyun Kim
>            Assignee: Daehyun Kim
>            Priority: Minor
>         Attachments: HADOOP-6153.patch
>
>
> It support to enable multiple map tasks for one large gzipped file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-531-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 08:28:33 2009
Return-Path: <common-issues-return-531-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13946 invoked from network); 27 Jul 2009 08:28:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 08:28:33 -0000
Received: (qmail 20450 invoked by uid 500); 27 Jul 2009 08:29:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20404 invoked by uid 500); 27 Jul 2009 08:29:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20394 invoked by uid 99); 27 Jul 2009 08:29:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 08:29:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 08:29:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D9DE2234C04B
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 01:29:14 -0700 (PDT)
Message-ID: <83024448.1248683354891.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 01:29:14 -0700 (PDT)
From: "Daehyun Kim (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Work started: (HADOOP-6153) RAgzip: multiple map tasks for a
 large gzipped file
In-Reply-To: <926568478.1247710994848.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Work on HADOOP-6153 started by Daehyun Kim.

> RAgzip: multiple map tasks for a large gzipped file
> ---------------------------------------------------
>
>                 Key: HADOOP-6153
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6153
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, native
>         Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
>            Reporter: Daehyun Kim
>            Assignee: Daehyun Kim
>            Priority: Minor
>         Attachments: HADOOP-6153.patch
>
>
> It support to enable multiple map tasks for one large gzipped file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-533-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 08:30:32 2009
Return-Path: <common-issues-return-533-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14499 invoked from network); 27 Jul 2009 08:30:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 08:30:32 -0000
Received: (qmail 21783 invoked by uid 500); 27 Jul 2009 08:31:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21697 invoked by uid 500); 27 Jul 2009 08:31:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21675 invoked by uid 99); 27 Jul 2009 08:31:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 08:31:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 08:31:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0F505234C055
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 01:31:15 -0700 (PDT)
Message-ID: <1752372067.1248683475061.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 01:31:15 -0700 (PDT)
From: "Daehyun Kim (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6153) RAgzip: multiple map tasks for a
 large gzipped file
In-Reply-To: <926568478.1247710994848.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daehyun Kim updated HADOOP-6153:
--------------------------------

    Hadoop Flags: [Reviewed]
          Status: Patch Available  (was: In Progress)

> RAgzip: multiple map tasks for a large gzipped file
> ---------------------------------------------------
>
>                 Key: HADOOP-6153
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6153
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, native
>         Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
>            Reporter: Daehyun Kim
>            Assignee: Daehyun Kim
>            Priority: Minor
>         Attachments: HADOOP-6153.patch
>
>
> It support to enable multiple map tasks for one large gzipped file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-532-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 08:30:32 2009
Return-Path: <common-issues-return-532-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14497 invoked from network); 27 Jul 2009 08:30:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 08:30:32 -0000
Received: (qmail 21762 invoked by uid 500); 27 Jul 2009 08:31:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21695 invoked by uid 500); 27 Jul 2009 08:31:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21674 invoked by uid 99); 27 Jul 2009 08:31:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 08:31:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 08:31:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CF32D234C046
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 01:31:14 -0700 (PDT)
Message-ID: <714407519.1248683474847.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 01:31:14 -0700 (PDT)
From: "Ramya R (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6160) releaseaudit (rats) should not be
 run againt the entire release binary
In-Reply-To: <1162830772.1247830035100.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735523#action_12735523 ] 

Ramya R commented on HADOOP-6160:
---------------------------------

+1. Patch looks fine. 

> releaseaudit (rats) should not be run againt the entire release binary
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6160
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6160
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>         Attachments: HADOOP-6160.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-534-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 08:45:31 2009
Return-Path: <common-issues-return-534-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17406 invoked from network); 27 Jul 2009 08:45:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 08:45:31 -0000
Received: (qmail 31091 invoked by uid 500); 27 Jul 2009 08:46:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31042 invoked by uid 500); 27 Jul 2009 08:46:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31032 invoked by uid 99); 27 Jul 2009 08:46:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 08:46:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 08:46:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D9195234C04C
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 01:46:14 -0700 (PDT)
Message-ID: <1958067341.1248684374888.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 01:46:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6160) releaseaudit (rats) should not be
 run againt the entire release binary
In-Reply-To: <1162830772.1247830035100.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735530#action_12735530 ] 

Giridharan Kesavan commented on HADOOP-6160:
--------------------------------------------

this patch runs rats-ant task the release binary excluding docs and lib/jdiff dirs.
Will file follow up jiras to add AL headers for files missing them and hence we can reduce the releaseaudit warnings count to zero.

> releaseaudit (rats) should not be run againt the entire release binary
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6160
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6160
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>         Attachments: HADOOP-6160.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-535-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 08:47:32 2009
Return-Path: <common-issues-return-535-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17724 invoked from network); 27 Jul 2009 08:47:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 08:47:32 -0000
Received: (qmail 32091 invoked by uid 500); 27 Jul 2009 08:48:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32042 invoked by uid 500); 27 Jul 2009 08:48:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32028 invoked by uid 99); 27 Jul 2009 08:48:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 08:48:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 08:48:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 16C0B234C1F0
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 01:48:15 -0700 (PDT)
Message-ID: <692750034.1248684495092.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 01:48:15 -0700 (PDT)
From: "Daehyun Kim (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6153) RAgzip: multiple map tasks for a
 large gzipped file
In-Reply-To: <926568478.1247710994848.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daehyun Kim updated HADOOP-6153:
--------------------------------

    Hadoop Flags:   (was: [Reviewed])

> RAgzip: multiple map tasks for a large gzipped file
> ---------------------------------------------------
>
>                 Key: HADOOP-6153
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6153
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, native
>         Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
>            Reporter: Daehyun Kim
>            Assignee: Daehyun Kim
>            Priority: Minor
>         Attachments: HADOOP-6153.patch
>
>
> It support to enable multiple map tasks for one large gzipped file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-536-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 09:02:32 2009
Return-Path: <common-issues-return-536-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20495 invoked from network); 27 Jul 2009 09:02:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 09:02:32 -0000
Received: (qmail 44285 invoked by uid 500); 27 Jul 2009 09:03:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44250 invoked by uid 500); 27 Jul 2009 09:03:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44240 invoked by uid 99); 27 Jul 2009 09:03:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 09:03:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 09:03:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C818C234C046
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 02:03:14 -0700 (PDT)
Message-ID: <1688398539.1248685394803.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 02:03:14 -0700 (PDT)
From: "Edward J. Yoon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5988) Add a command to ' FsShell stat '
 to get a file's block location information
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735534#action_12735534 ] 

Edward J. Yoon commented on HADOOP-5988:
----------------------------------------

Thanks for your review. I'll check them.

>> For more eaier to read, i think the output should be indented (especially the byte range part), what do you think? 

And, I agree with you. Below is the output of the '# bin/hadoop fs -stat %B tmp/src.tar.gz'.

----
09/07/27 17:50:01 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
byte-range:0-67108864   location:dev1.nm;
byte-range:67108864-134217728   location:dev1.nm;
byte-range:134217728-201326592  location:dev1.nm;
byte-range:201326592-268435456  location:dev1.nm;
byte-range:268435456-335544320  location:dev1.nm;
byte-range:335544320-402653184  location:dev1.nm;
byte-range:402653184-469762048  location:dev1.nm;
byte-range:469762048-517866001  location:dev1.nm;


> Add a command to ' FsShell stat ' to get a file's block location information
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5988
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5988
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: He Yongqiang
>         Attachments: HADOOP-5988.patch
>
>
> Adding an option to ' FsShell stat ' to get a file's block location information will be very useful.
> we can print the block location information in this format:
> blockID:XXXXX  byte-range:YYYY-ZZZZ  location:dn1;dn2;
> blockID:XXXXX  byte-range:YYYY-ZZZZ  location:dn1;dn2;

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-537-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 09:44:35 2009
Return-Path: <common-issues-return-537-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34373 invoked from network); 27 Jul 2009 09:44:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 09:44:32 -0000
Received: (qmail 71768 invoked by uid 500); 27 Jul 2009 09:45:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71742 invoked by uid 500); 27 Jul 2009 09:45:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71732 invoked by uid 99); 27 Jul 2009 09:45:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 09:45:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 09:45:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CBFDD234C046
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 02:45:14 -0700 (PDT)
Message-ID: <2109969689.1248687914819.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 02:45:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6160) releaseaudit (rats) should not be
 run againt the entire release binary
In-Reply-To: <1162830772.1247830035100.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735540#action_12735540 ] 

Hadoop QA commented on HADOOP-6160:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12414427/HADOOP-6160.patch
  against trunk revision 797300.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/584/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/584/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/584/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/584/console

This message is automatically generated.

> releaseaudit (rats) should not be run againt the entire release binary
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6160
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6160
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>         Attachments: HADOOP-6160.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-538-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 10:12:35 2009
Return-Path: <common-issues-return-538-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48066 invoked from network); 27 Jul 2009 10:12:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 10:12:35 -0000
Received: (qmail 14228 invoked by uid 500); 27 Jul 2009 10:13:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14207 invoked by uid 500); 27 Jul 2009 10:13:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14197 invoked by uid 99); 27 Jul 2009 10:13:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 10:13:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 10:13:37 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D2D46234C046
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 03:13:14 -0700 (PDT)
Message-ID: <829820393.1248689594848.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 03:13:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6160) releaseaudit (rats) should not be run
 againt the entire release binary
In-Reply-To: <1162830772.1247830035100.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6160:
---------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.21.0
           Status: Resolved  (was: Patch Available)

this patch doesn't requires a test as this just changes to the build.xml file.

I just committed this!

> releaseaudit (rats) should not be run againt the entire release binary
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6160
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6160
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6160.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-539-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 10:18:37 2009
Return-Path: <common-issues-return-539-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50319 invoked from network); 27 Jul 2009 10:18:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 10:18:37 -0000
Received: (qmail 21632 invoked by uid 500); 27 Jul 2009 10:19:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21605 invoked by uid 500); 27 Jul 2009 10:19:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21595 invoked by uid 99); 27 Jul 2009 10:19:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 10:19:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 10:19:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CBC76234C046
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 03:19:14 -0700 (PDT)
Message-ID: <1805123261.1248689954819.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 03:19:14 -0700 (PDT)
From: "rahul k singh (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6105) Provide a way to automatically
 handle backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735552#action_12735552 ] 

rahul k singh commented on HADOOP-6105:
---------------------------------------

Changes  look good , 
some minor comments:

Configuration.java

1.Check for null in addDeprecation method.
2.Some extra lines are there in front of some of the methods , remove them

3.Try to use foreach kind of syntax where ever possible.It makes code look simpler.

TestConfiguration.java

1.In tearDown method , you should delete the test-config.xml file also
2.make sure that CONFIG3 is deleted.


1.Diff has Index: junitvmwatcher8758949811953274592.properties
I dont think it is required.

Diff also has test-config3.xml
i think this is becoz of CONFIG3 not being deleted in testcase.

> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>         Attachments: HADOOP-6105.patch
>
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-540-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 11:10:34 2009
Return-Path: <common-issues-return-540-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65066 invoked from network); 27 Jul 2009 11:10:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 11:10:34 -0000
Received: (qmail 83360 invoked by uid 500); 27 Jul 2009 11:11:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83324 invoked by uid 500); 27 Jul 2009 11:11:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83314 invoked by uid 99); 27 Jul 2009 11:11:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 11:11:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 11:11:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 33E0A234C046
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 04:11:15 -0700 (PDT)
Message-ID: <1222099290.1248693075198.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 04:11:15 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6160) releaseaudit (rats) should not be
 run againt the entire release binary
In-Reply-To: <1162830772.1247830035100.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735560#action_12735560 ] 

Hudson commented on HADOOP-6160:
--------------------------------

Integrated in Hadoop-Common-trunk #36 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/36/])
    . Fix ant releaseaudit target to run on specific directories. (Contributed by Giridharan Kesavan)


> releaseaudit (rats) should not be run againt the entire release binary
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6160
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6160
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6160.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-541-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 12:30:34 2009
Return-Path: <common-issues-return-541-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89426 invoked from network); 27 Jul 2009 12:30:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 12:30:34 -0000
Received: (qmail 77824 invoked by uid 500); 27 Jul 2009 12:31:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77775 invoked by uid 500); 27 Jul 2009 12:31:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77765 invoked by uid 99); 27 Jul 2009 12:31:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 12:31:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 12:31:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 81BB6234C051
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 05:31:15 -0700 (PDT)
Message-ID: <1624044572.1248697875530.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 05:31:15 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6160) releaseaudit (rats) should not be
 run againt the entire release binary
In-Reply-To: <1162830772.1247830035100.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735585#action_12735585 ] 

Hudson commented on HADOOP-6160:
--------------------------------

Integrated in Hadoop-Hdfs-trunk #31 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/31/])
    . Fix ant releaseaudit target to run on specific directories. (Contributed by Giridharan Kesavan)


> releaseaudit (rats) should not be run againt the entire release binary
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6160
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6160
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6160.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-542-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 12:41:34 2009
Return-Path: <common-issues-return-542-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92656 invoked from network); 27 Jul 2009 12:41:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 12:41:34 -0000
Received: (qmail 94056 invoked by uid 500); 27 Jul 2009 12:42:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94014 invoked by uid 500); 27 Jul 2009 12:42:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93996 invoked by uid 99); 27 Jul 2009 12:42:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 12:42:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 12:42:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D3C15234C046
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 05:42:14 -0700 (PDT)
Message-ID: <1539622322.1248698534855.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 05:42:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6123) hdfs script does not work after
 project split.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735590#action_12735590 ] 

Hadoop QA commented on HADOOP-6123:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12412236/6123_v1.patch
  against trunk revision 798093.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/586/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/586/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/586/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/586/console

This message is automatically generated.

> hdfs script does not work after project split.
> ----------------------------------------------
>
>                 Key: HADOOP-6123
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6123
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Sharad Agarwal
>             Fix For: 0.21.0
>
>         Attachments: 6123_v1.patch
>
>
> There are problems running hdfs script from common.
> # Usage message for hdfs does not have "fs" option anymore.
> # There seem to be an undocumented option "dfs", but using it throws NoClassDefFoundError or ClassNotFoundException.
> # Same with other options e.g. name-node.
> May I am missing something here. How do we call ls these days?
> Do we need to move hdfs script to hdfs project.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-543-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 13:03:32 2009
Return-Path: <common-issues-return-543-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 691 invoked from network); 27 Jul 2009 13:03:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 13:03:32 -0000
Received: (qmail 24354 invoked by uid 500); 27 Jul 2009 13:04:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24314 invoked by uid 500); 27 Jul 2009 13:04:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24304 invoked by uid 99); 27 Jul 2009 13:04:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 13:04:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 13:04:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E582B234C04B
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 06:04:14 -0700 (PDT)
Message-ID: <332025333.1248699854939.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 06:04:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6169) Removing deprecated method calls in
 TFile
In-Reply-To: <927826654.1248487334841.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735598#action_12735598 ] 

Hadoop QA commented on HADOOP-6169:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12414508/hadoop-6169-20090725.patch
  against trunk revision 798093.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 9 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/588/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/588/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/588/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/588/console

This message is automatically generated.

> Removing deprecated method calls in TFile
> -----------------------------------------
>
>                 Key: HADOOP-6169
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6169
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6169-20090725.patch
>
>
> We should remove the use of deprecated APIs in TFile.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-544-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 15:44:33 2009
Return-Path: <common-issues-return-544-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94083 invoked from network); 27 Jul 2009 15:44:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 15:44:33 -0000
Received: (qmail 70162 invoked by uid 500); 27 Jul 2009 15:45:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70117 invoked by uid 500); 27 Jul 2009 15:45:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70075 invoked by uid 99); 27 Jul 2009 15:45:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 15:45:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 15:45:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 24E2B234C04B
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 08:45:15 -0700 (PDT)
Message-ID: <645941162.1248709515150.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 08:45:15 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6160) releaseaudit (rats) should not be
 run againt the entire release binary
In-Reply-To: <1162830772.1247830035100.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735630#action_12735630 ] 

Hudson commented on HADOOP-6160:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk #31 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/31/])
    . Fix ant releaseaudit target to run on specific directories. (Contributed by Giridharan Kesavan)


> releaseaudit (rats) should not be run againt the entire release binary
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6160
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6160
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6160.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-545-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 15:48:32 2009
Return-Path: <common-issues-return-545-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95578 invoked from network); 27 Jul 2009 15:48:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 15:48:31 -0000
Received: (qmail 72229 invoked by uid 500); 27 Jul 2009 15:49:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72188 invoked by uid 500); 27 Jul 2009 15:49:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72178 invoked by uid 99); 27 Jul 2009 15:49:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 15:49:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 15:49:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D0BB2234C044
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 08:49:14 -0700 (PDT)
Message-ID: <625452059.1248709754840.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 08:49:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6153) RAgzip: multiple map tasks for a
 large gzipped file
In-Reply-To: <926568478.1247710994848.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735632#action_12735632 ] 

Hadoop QA commented on HADOOP-6153:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12414595/HADOOP-6153.patch
  against trunk revision 798093.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/589/console

This message is automatically generated.

> RAgzip: multiple map tasks for a large gzipped file
> ---------------------------------------------------
>
>                 Key: HADOOP-6153
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6153
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, native
>         Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
>            Reporter: Daehyun Kim
>            Assignee: Daehyun Kim
>            Priority: Minor
>         Attachments: HADOOP-6153.patch
>
>
> It support to enable multiple map tasks for one large gzipped file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-546-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 17:25:34 2009
Return-Path: <common-issues-return-546-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28184 invoked from network); 27 Jul 2009 17:25:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 17:25:34 -0000
Received: (qmail 15033 invoked by uid 500); 27 Jul 2009 17:26:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15003 invoked by uid 500); 27 Jul 2009 17:26:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14838 invoked by uid 99); 27 Jul 2009 17:26:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 17:26:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 17:26:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 00BA5234C051
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 10:26:15 -0700 (PDT)
Message-ID: <409673876.1248715575002.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 10:26:15 -0700 (PDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4491) Per-job local data on the
 TaskTracker node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735669#action_12735669 ] 

Hemanth Yamijala commented on HADOOP-4491:
------------------------------------------

Owen and I had an offline discussion about this, and we felt another approach to try out was to see if we could have the directories and files owned by the user and group-owned by the tasktracker. The group ownership should be sticky so permissions are inherited. The permissions must apply for all the relevant components in the paths.

So, $jobid and $attemptid in the examples above would be  owned by the user, group-owned by mapred, and have permissions like 570 or similar.

This might also remove the need to have parallel directory structures.

The rationale for this approach follows from the fact that for maximum security the task-controller executable needs to be group owned by the tasktracker (to prevent other users from launching it). Hence, this almost means that the tasktracker user is a special user in the system that is required for secure installations. And it can be setup such that the user is in a separate group on its own.

Thoughts ?

> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-547-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 17:43:34 2009
Return-Path: <common-issues-return-547-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41631 invoked from network); 27 Jul 2009 17:43:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 17:43:34 -0000
Received: (qmail 30999 invoked by uid 500); 27 Jul 2009 17:44:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30954 invoked by uid 500); 27 Jul 2009 17:44:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30944 invoked by uid 99); 27 Jul 2009 17:44:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 17:44:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 17:44:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D738C234C044
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 10:44:14 -0700 (PDT)
Message-ID: <113189766.1248716654867.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 10:44:14 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5438) Merge FileSystem.create and
 FileSystem.append
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735680#action_12735680 ] 

Doug Cutting commented on HADOOP-5438:
--------------------------------------

> It's too bad Java won't automatically convert a single enum into an enumset when necessary.

We could use a variable arity parameter, e.g.:
  FSDataOutputStream create(Path file, CreateFlag... flags);


> Merge FileSystem.create and FileSystem.append
> ---------------------------------------------
>
>                 Key: HADOOP-5438
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5438
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: He Yongqiang
>            Assignee: He Yongqiang
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-5438(2009-04-06).patch, Hadoop-5438-2009-03-30.patch, Hadoop-5438-2009-03-31-2.patch, Hadoop-5438-2009-03-31.patch, Hadoop-5438-2009-05-10.patch, Hadoop-5438-2009-05-15.patch, Hadoop-5438-2009-05-19.patch, Hadoop-5438-2009-05-5.patch
>
>
> Currently, when a user wants to modify a file, the user first calls exists() to know if this file is already there. And then uses create() or append() according to whether the file exists or not.
> the code looks like:
> {code}
> FSDataOutputStream out_1 = null;
> if (fs.exists(path_1))
>    out_1 = fs.append(path_1);
> else
>    out_1 = fs.create(path_1);
> {code}
> . On the performace side,It involes two RPCs. On the easy-of-use side, it is not very convient in contrast to the traditional open interface.
> It will more complicate if there is a overwrite parameter specified. I donot know whether there is a bug about 'overwrite' in 0.19, some times it takes a long time for overwrite creates to reture. So i make the write file code with overwrite param works like:
> {code}
> boolean exists = fs.exists(name);
> if (overwrite) {
>     if (exists)
>        fs.delete(name, true);
>      this.out = fs.create(name, overwrite, bufferSize, replication,
> 				    blockSize, progress);
>      this.currentRowID = 0;
>  } else {
>    if (!exists)
> 	this.out = fs.create(name, overwrite, bufferSize,
> 					replication, blockSize, progress);
>    else
> 	this.out = fs.append(name, bufferSize, progress);
> {code}
> Some code statements there are really redundant and not needed, especialy with the delete(). But without deleting first, the overwrite takes a long time to reture.
> BTW, i will create another issue about the overwrite problem. If it is not a bug at all or a duplicate, someone please close it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-548-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 18:30:41 2009
Return-Path: <common-issues-return-548-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78055 invoked from network); 27 Jul 2009 18:30:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 18:30:40 -0000
Received: (qmail 81999 invoked by uid 500); 27 Jul 2009 18:31:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81956 invoked by uid 500); 27 Jul 2009 18:31:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81944 invoked by uid 99); 27 Jul 2009 18:31:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 18:31:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 18:31:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C9A7A234C044
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 11:31:14 -0700 (PDT)
Message-ID: <1248416554.1248719474811.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 11:31:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6123) hdfs script does not work after
 project split.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6123:
-------------------------------------------

    Hadoop Flags: [Reviewed]

Tried again the patch.  It worked fine.  I probably did something wrong last time.

Since the for-loop approach is already used in hadoop-config.sh, we should keep the new codes consistent with the existing codes.

+1 patch looks good.

> hdfs script does not work after project split.
> ----------------------------------------------
>
>                 Key: HADOOP-6123
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6123
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Sharad Agarwal
>             Fix For: 0.21.0
>
>         Attachments: 6123_v1.patch
>
>
> There are problems running hdfs script from common.
> # Usage message for hdfs does not have "fs" option anymore.
> # There seem to be an undocumented option "dfs", but using it throws NoClassDefFoundError or ClassNotFoundException.
> # Same with other options e.g. name-node.
> May I am missing something here. How do we call ls these days?
> Do we need to move hdfs script to hdfs project.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-549-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 18:36:34 2009
Return-Path: <common-issues-return-549-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81680 invoked from network); 27 Jul 2009 18:36:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 18:36:34 -0000
Received: (qmail 92853 invoked by uid 500); 27 Jul 2009 18:37:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92804 invoked by uid 500); 27 Jul 2009 18:37:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92794 invoked by uid 99); 27 Jul 2009 18:37:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 18:37:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 18:37:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E3492234C04B
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 11:37:14 -0700 (PDT)
Message-ID: <1678565713.1248719834930.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 11:37:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6123) hdfs script does not work after
 project split.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6123:
-------------------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

No new tests since this is a script change.

I have committed this.  Thanks, Sharad!

> hdfs script does not work after project split.
> ----------------------------------------------
>
>                 Key: HADOOP-6123
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6123
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Sharad Agarwal
>             Fix For: 0.21.0
>
>         Attachments: 6123_v1.patch
>
>
> There are problems running hdfs script from common.
> # Usage message for hdfs does not have "fs" option anymore.
> # There seem to be an undocumented option "dfs", but using it throws NoClassDefFoundError or ClassNotFoundException.
> # Same with other options e.g. name-node.
> May I am missing something here. How do we call ls these days?
> Do we need to move hdfs script to hdfs project.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-550-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 19:06:32 2009
Return-Path: <common-issues-return-550-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91674 invoked from network); 27 Jul 2009 19:06:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 19:06:32 -0000
Received: (qmail 34303 invoked by uid 500); 27 Jul 2009 19:07:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34258 invoked by uid 500); 27 Jul 2009 19:07:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34248 invoked by uid 99); 27 Jul 2009 19:07:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 19:07:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 19:07:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 026CF234C044
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 12:07:15 -0700 (PDT)
Message-ID: <1776381968.1248721634994.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 12:07:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6166:
-------------------------------------------

    Attachment: c6166_20090727.patch

Not yet able to improve PureJavaCrc32 in my 64-bit machine but had a lot of fun last weekends.

c6166_20090727.patch: moved the codes to common (finally).  Please try it when you have time.

- 64-bit
java.version = 1.6.0_10
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_10-b33
java.vm.version = 11.0-b15
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = amd64
os.name = Linux
os.version = 2.6.9-55.ELsmp
||num bytes||PureJavaCrc32 MB/sec||PureJavaCrc32New MB/sec||Crc32_3_2 MB/sec||Crc32_4_3 MB/sec||Crc32_5_5 MB/sec||Crc32_6_6 MB/sec||Crc32_8_8 MB/sec||Crc32_12_12 MB/sec||
|        8 |  157.986 |  102.628 |  135.926 |  204.207 |  218.584 |  239.862 |  253.886 |  213.072 |
|       16 |  245.381 |  214.363 |  238.798 |  202.284 |  261.246 |  207.342 |  219.935 |  245.648 |
|       32 |  331.296 |  290.766 |  283.689 |  218.582 |  310.499 |  329.800 |  283.757 |  273.074 |
|       64 |  405.822 |  345.573 |  325.067 |  224.623 |  344.732 |  345.010 |  311.538 |  346.970 |
|      128 |  451.240 |  378.875 |  343.498 |  226.853 |  391.824 |  392.462 |  323.504 |  391.573 |
|      256 |  479.728 |  410.574 |  352.432 |  226.939 |  416.448 |  396.344 |  331.537 |  415.233 |
|      512 |  488.917 |  425.214 |  355.640 |  227.120 |  424.109 |  409.781 |  335.965 |  427.655 |
|     1024 |  499.820 |  433.135 |  358.441 |  225.953 |  430.212 |  414.286 |  337.652 |  431.440 |
|     2048 |  504.199 |  438.373 |  352.913 |  223.754 |  435.921 |  417.888 |  339.190 |  438.454 |
|     4096 |  509.100 |  441.553 |  351.305 |  222.355 |  438.667 |  420.657 |  341.057 |  441.063 |
|     8192 |  511.632 |  439.242 |  352.058 |  222.469 |  439.568 |  422.009 |  341.427 |  447.972 |
|    16384 |  510.829 |  444.631 |  354.097 |  222.488 |  439.707 |  419.661 |  341.200 |  451.286 |
|    32768 |  507.353 |  437.758 |  354.601 |  222.503 |  436.775 |  416.266 |  339.704 |  449.830 |
|    65536 |  507.335 |  434.042 |  354.837 |  222.682 |  436.742 |  417.825 |  339.761 |  449.868 |
|   131072 |  507.473 |  431.449 |  355.014 |  222.748 |  436.477 |  417.910 |  339.835 |  449.958 |
|   262144 |  507.548 |  429.451 |  354.932 |  222.632 |  436.698 |  417.783 |  339.852 |  449.936 |
|   524288 |  507.322 |  428.618 |  355.142 |  222.491 |  436.584 |  417.715 |  339.826 |  450.146 |
|  1048576 |  507.148 |  428.778 |  354.769 |  222.534 |  436.506 |  417.819 |  339.830 |  450.032 |
|  2097152 |  506.610 |  432.981 |  354.596 |  222.261 |  436.080 |  417.573 |  339.933 |  449.623 |
|  4194304 |  504.503 |  432.501 |  352.918 |  221.669 |  432.956 |  414.626 |  337.668 |  445.489 |
|  8388608 |  498.208 |  428.943 |  348.488 |  219.868 |  429.455 |  411.497 |  335.448 |  440.899 |
| 16777216 |  497.184 |  423.245 |  348.105 |  219.657 |  427.288 |  410.788 |  334.992 |  440.603 |

- 32-bit
java.version = 1.6.0_14
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_14-b08
java.vm.version = 14.0-b16
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) Client VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = x86
os.name = Windows XP
os.version = 5.1
||num bytes||PureJavaCrc32 MB/sec||PureJavaCrc32New MB/sec||Crc32_3_2 MB/sec||Crc32_4_3 MB/sec||Crc32_5_5 MB/sec||Crc32_6_6 MB/sec||Crc32_8_8 MB/sec||Crc32_12_12 MB/sec||
|        8 |  192.776 |  167.821 |  174.386 |  207.658 |  184.504 |  196.480 |  222.191 |  155.052 |
|       16 |  227.810 |  212.454 |  224.370 |  250.812 |  236.371 |  229.370 |  267.719 |  248.392 |
|       32 |  250.230 |  239.881 |  251.688 |  280.112 |  257.619 |  270.357 |  298.951 |  268.026 |
|       64 |  263.695 |  257.785 |  275.683 |  296.554 |  269.474 |  284.615 |  316.642 |  319.204 |
|      128 |  270.873 |  266.500 |  286.942 |  305.744 |  282.172 |  298.166 |  325.899 |  325.806 |
|      256 |  282.205 |  270.751 |  294.155 |  306.224 |  288.460 |  303.445 |  330.728 |  342.959 |
|      512 |  282.529 |  270.063 |  295.103 |  309.134 |  290.043 |  300.008 |  332.325 |  343.466 |
|     1024 |  279.710 |  273.680 |  298.489 |  308.905 |  289.993 |  305.850 |  331.706 |  347.761 |
|     2048 |  279.753 |  274.073 |  293.911 |  304.972 |  285.538 |  308.030 |  334.459 |  350.296 |
|     4096 |  278.830 |  275.688 |  290.520 |  302.634 |  293.201 |  308.455 |  334.498 |  351.761 |
|     8192 |  279.518 |  274.829 |  289.088 |  299.036 |  292.383 |  305.940 |  333.512 |  352.584 |
|    16384 |  278.609 |  251.000 |  287.964 |  303.862 |  293.782 |  308.534 |  333.397 |  347.889 |
|    32768 |  276.124 |  272.805 |  290.125 |  300.458 |  289.993 |  306.447 |  334.985 |  350.833 |
|    65536 |  274.212 |  273.606 |  288.872 |  303.673 |  286.457 |  307.196 |  332.591 |  349.273 |
|   131072 |  275.371 |  272.257 |  289.985 |  303.490 |  289.126 |  303.128 |  330.630 |  349.575 |
|   262144 |  275.607 |  273.878 |  288.080 |  302.439 |  285.862 |  304.965 |  330.775 |  347.044 |
|   524288 |  274.578 |  270.549 |  286.745 |  299.832 |  287.063 |  304.299 |  332.160 |  346.131 |
|  1048576 |  270.002 |  272.333 |  285.866 |  299.702 |  284.005 |  304.845 |  329.455 |  343.377 |
|  2097152 |  268.254 |  265.650 |  285.905 |  297.428 |  286.168 |  302.749 |  329.962 |  344.515 |
|  4194304 |  272.093 |  268.692 |  285.552 |  299.619 |  285.262 |  299.311 |  327.311 |  338.847 |
|  8388608 |  268.156 |  265.971 |  283.967 |  291.225 |  282.974 |  301.162 |  323.698 |  343.482 |
| 16777216 |  271.428 |  270.893 |  285.939 |  299.171 |  284.946 |  302.520 |  328.465 |  343.694 |



> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt, c6166_20090727.patch
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-551-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 22:11:32 2009
Return-Path: <common-issues-return-551-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80035 invoked from network); 27 Jul 2009 22:11:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 22:11:32 -0000
Received: (qmail 87078 invoked by uid 500); 27 Jul 2009 22:12:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87032 invoked by uid 500); 27 Jul 2009 22:12:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87022 invoked by uid 99); 27 Jul 2009 22:12:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 22:12:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 22:12:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D8EC0234C04C
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 15:12:14 -0700 (PDT)
Message-ID: <704093981.1248732734887.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 15:12:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735806#action_12735806 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6166:
------------------------------------------------

Got a different story after updated to the latest jdk.

java.version = 1.6.0_14
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_14-b08
java.vm.version = 14.0-b16
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = amd64
os.name = Linux
os.version = 2.6.9-55.ELsmp
||num bytes||PureJavaCrc32 MB/sec||PureJavaCrc32New MB/sec||Crc32_3_2 MB/sec||Crc32_4_3 MB/sec||Crc32_5_5 MB/sec||Crc32_6_6 MB/sec||Crc32_8_8 MB/sec||Crc32_12_12 MB/sec||
|        8 |  190.202 |  170.445 |  170.560 |  228.872 |  215.769 |  206.454 |  231.685 |  214.802 |
|       16 |  257.234 |  209.434 |  225.336 |  267.450 |  263.165 |  187.785 |  253.917 |  232.935 |
|       32 |  309.992 |  271.358 |  243.099 |  309.840 |  309.997 |  304.166 |  319.013 |  270.716 |
|       64 |  348.461 |  326.343 |  265.435 |  338.049 |  330.947 |  333.372 |  358.959 |  334.240 |
|      128 |  369.745 |  362.989 |  271.531 |  354.615 |  382.880 |  371.822 |  383.919 |  370.870 |
|      256 |  382.773 |  385.201 |  279.028 |  364.379 |  402.521 |  378.506 |  400.110 |  399.904 |
|      512 |  384.597 |  397.015 |  279.898 |  364.408 |  406.135 |  389.814 |  407.351 |  405.674 |
|     1024 |  390.181 |  405.577 |  281.035 |  364.413 |  412.159 |  395.506 |  408.046 |  416.599 |
|     2048 |  392.820 |  408.548 |  275.382 |  360.259 |  412.941 |  395.982 |  409.498 |  422.795 |
|     4096 |  392.362 |  408.593 |  273.375 |  355.012 |  414.489 |  397.885 |  410.857 |  422.115 |
|     8192 |  393.152 |  409.355 |  273.973 |  355.846 |  415.358 |  398.532 |  411.701 |  423.370 |
|    16384 |  393.094 |  409.406 |  274.759 |  355.500 |  415.657 |  398.542 |  411.813 |  422.864 |
|    32768 |  392.515 |  408.989 |  276.169 |  357.135 |  415.965 |  400.295 |  411.622 |  422.887 |
|    65536 |  393.323 |  408.997 |  276.594 |  357.448 |  416.075 |  400.896 |  411.966 |  422.850 |
|   131072 |  393.531 |  408.982 |  276.566 |  357.490 |  416.059 |  400.959 |  412.037 |  422.953 |
|   262144 |  393.638 |  409.070 |  276.585 |  357.407 |  416.030 |  401.040 |  412.046 |  423.034 |
|   524288 |  393.629 |  408.982 |  276.511 |  357.462 |  416.123 |  400.994 |  411.924 |  423.010 |
|  1048576 |  393.652 |  408.943 |  276.397 |  357.050 |  415.844 |  400.785 |  411.927 |  422.808 |
|  2097152 |  393.408 |  408.558 |  276.024 |  356.452 |  415.633 |  400.296 |  411.594 |  422.426 |
|  4194304 |  391.575 |  405.148 |  275.157 |  354.772 |  413.680 |  397.834 |  409.809 |  420.100 |
|  8388608 |  389.204 |  404.179 |  273.648 |  351.896 |  411.007 |  395.309 |  407.030 |  417.661 |
| 16777216 |  388.753 |  403.422 |  273.343 |  351.298 |  410.380 |  394.783 |  406.396 |  416.995 |

The above table makes more sense since it is easy to tell from the codes that Crc32_N_N for N > 4 is more efficient than PureJavaCrc32 (i.e. Crc32_4_4).  Note that N cannot be increased arbitrary.  Otherwise, the tables may not fit into the cpu cache as explained previously by Scott.  (Tried Crc32_16_16 but it got worst.)

As shown above, Crc32_12_12 has 7% and 26% improvement on my 64-bit and 32-bit machines with jdk 1.6.0_14-b08, respectively.  I cannot explain why the numbers were generally better in 1.6.0_10-b33, 64-bit vm.  Specific jdk feature/bug?

> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt, c6166_20090727.patch
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-552-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 27 23:55:32 2009
Return-Path: <common-issues-return-552-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11290 invoked from network); 27 Jul 2009 23:55:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Jul 2009 23:55:32 -0000
Received: (qmail 57235 invoked by uid 500); 27 Jul 2009 23:56:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57192 invoked by uid 500); 27 Jul 2009 23:56:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57177 invoked by uid 99); 27 Jul 2009 23:56:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 23:56:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jul 2009 23:56:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CC3B3234C044
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 16:56:14 -0700 (PDT)
Message-ID: <1423769270.1248738974821.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 16:56:14 -0700 (PDT)
From: "Koji Noguchi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6167) bin/hadoop script doesn't allow for
 different memory settings for each daemon type
In-Reply-To: <128618595.1248313995284.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735846#action_12735846 ] 

Koji Noguchi commented on HADOOP-6167:
--------------------------------------

bq. how does this not work?

I agree with ryan.  It does work as is (at least in our environment).
If the goal is to make it look better, I'm fine.

In our rhel environment, jvm gives preference to latter if argument is provided twice.


> bin/hadoop script doesn't allow for different memory settings for each daemon type
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6167
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.0
>            Reporter: Fernando
>         Attachments: hadoop, hadoop-script.diff
>
>
> bin/hadoop assumes that all daemon types ( namenode, datanode, jobtracker, tasktracker ), all use the same memory settings.. (HADOOP_HEAPSIZE).
> I propose changes to that script to allow overriding the default memory ( HADOOP_HEAPSIZE ), with daemon specific OPTS (HADOOP_NAMENODE_OPTS, etc ).
> Basically at the bottom of the bin/hadoop script, it will check to see if the user has already set "-Xmx" in the HADOOP_OPTS variable.. if so, then it will ignore the JAVA_HEAP_SIZE variable..
> as such:
> # run it
> if [[ $HADOOP_OPTS == *-Xmx* ]]; then
>   exec "$JAVA" $HADOOP_OPTS -classpath "$CLASSPATH" $CLASS "$@"
> else
>   exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS -classpath "$CLASSPATH" $CLASS "$@"
> fi
> I will attach the file as I have modified it..

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-553-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 28 00:37:33 2009
Return-Path: <common-issues-return-553-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26023 invoked from network); 28 Jul 2009 00:37:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Jul 2009 00:37:33 -0000
Received: (qmail 80173 invoked by uid 500); 28 Jul 2009 00:38:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80105 invoked by uid 500); 28 Jul 2009 00:38:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79833 invoked by uid 99); 28 Jul 2009 00:38:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 00:38:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 00:38:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1B846234C04C
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 17:38:15 -0700 (PDT)
Message-ID: <1550479875.1248741495111.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 17:38:15 -0700 (PDT)
From: "Fernando (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6167) bin/hadoop script doesn't allow for
 different memory settings for each daemon type
In-Reply-To: <128618595.1248313995284.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735857#action_12735857 ] 

Fernando commented on HADOOP-6167:
----------------------------------

Ok. So is this -Xmx behavior documented?

I guess not, but I guess it is working without any code change.. so you can mark this bug as "invalid" or some such..

sorry for the confusion.

> bin/hadoop script doesn't allow for different memory settings for each daemon type
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6167
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.0
>            Reporter: Fernando
>         Attachments: hadoop, hadoop-script.diff
>
>
> bin/hadoop assumes that all daemon types ( namenode, datanode, jobtracker, tasktracker ), all use the same memory settings.. (HADOOP_HEAPSIZE).
> I propose changes to that script to allow overriding the default memory ( HADOOP_HEAPSIZE ), with daemon specific OPTS (HADOOP_NAMENODE_OPTS, etc ).
> Basically at the bottom of the bin/hadoop script, it will check to see if the user has already set "-Xmx" in the HADOOP_OPTS variable.. if so, then it will ignore the JAVA_HEAP_SIZE variable..
> as such:
> # run it
> if [[ $HADOOP_OPTS == *-Xmx* ]]; then
>   exec "$JAVA" $HADOOP_OPTS -classpath "$CLASSPATH" $CLASS "$@"
> else
>   exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS -classpath "$CLASSPATH" $CLASS "$@"
> fi
> I will attach the file as I have modified it..

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-554-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 28 01:15:36 2009
Return-Path: <common-issues-return-554-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40381 invoked from network); 28 Jul 2009 01:15:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Jul 2009 01:15:34 -0000
Received: (qmail 6549 invoked by uid 500); 28 Jul 2009 01:16:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6519 invoked by uid 500); 28 Jul 2009 01:16:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6473 invoked by uid 99); 28 Jul 2009 01:16:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 01:16:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 01:16:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E00B5234C046
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 18:16:14 -0700 (PDT)
Message-ID: <601879567.1248743774916.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 18:16:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-2851) Bogus logging messages in
 Configration object constructors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-2851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE resolved HADOOP-2851.
--------------------------------------------

    Resolution: Won't Fix

Seem that this is not needed.  Closing this as "Won't Fix".

> Bogus logging messages in Configration object constructors
> ----------------------------------------------------------
>
>                 Key: HADOOP-2851
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2851
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.16.0
>            Reporter: Jingkei Ly
>            Priority: Trivial
>         Attachments: 2851_20080925.patch, 2851_20081103.patch
>
>
> The constructors for the Configuration object contains a superfluous logging message that logs an IOException whenever logging is enabled for the debug level. Basically both constructors have the statement:
> if (LOG.isDebugEnabled()) {
>       LOG.debug(StringUtils.stringifyException(new IOException("config()")));
>     }
> I can' t see any reason for it to be there and it just ends up leaving bogus IOExceptions in log files. It looks like its an old debug print statement which has accidentally been left in.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-555-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 28 01:41:32 2009
Return-Path: <common-issues-return-555-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58825 invoked from network); 28 Jul 2009 01:41:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Jul 2009 01:41:32 -0000
Received: (qmail 19132 invoked by uid 500); 28 Jul 2009 01:42:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19049 invoked by uid 500); 28 Jul 2009 01:42:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19035 invoked by uid 99); 28 Jul 2009 01:42:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 01:42:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 01:42:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 68029234C044
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 18:42:15 -0700 (PDT)
Message-ID: <975819897.1248745335411.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 18:42:15 -0700 (PDT)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6167) bin/hadoop script doesn't allow for
 different memory settings for each daemon type
In-Reply-To: <128618595.1248313995284.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735884#action_12735884 ] 

Scott Carey commented on HADOOP-6167:
-------------------------------------

Having -Xmx listed multiple times is misleading and confusing.  It should be listed once, preferably near the beginning.  Using tools like 'top' with the front of the command line printed shouldn't lie.  IMO listing things as important as -Xmx twice is a bad idea regardless of the actual behavior.

> bin/hadoop script doesn't allow for different memory settings for each daemon type
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6167
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.0
>            Reporter: Fernando
>         Attachments: hadoop, hadoop-script.diff
>
>
> bin/hadoop assumes that all daemon types ( namenode, datanode, jobtracker, tasktracker ), all use the same memory settings.. (HADOOP_HEAPSIZE).
> I propose changes to that script to allow overriding the default memory ( HADOOP_HEAPSIZE ), with daemon specific OPTS (HADOOP_NAMENODE_OPTS, etc ).
> Basically at the bottom of the bin/hadoop script, it will check to see if the user has already set "-Xmx" in the HADOOP_OPTS variable.. if so, then it will ignore the JAVA_HEAP_SIZE variable..
> as such:
> # run it
> if [[ $HADOOP_OPTS == *-Xmx* ]]; then
>   exec "$JAVA" $HADOOP_OPTS -classpath "$CLASSPATH" $CLASS "$@"
> else
>   exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS -classpath "$CLASSPATH" $CLASS "$@"
> fi
> I will attach the file as I have modified it..

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-556-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 28 03:55:32 2009
Return-Path: <common-issues-return-556-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5521 invoked from network); 28 Jul 2009 03:55:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Jul 2009 03:55:32 -0000
Received: (qmail 58104 invoked by uid 500); 28 Jul 2009 02:56:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58056 invoked by uid 500); 28 Jul 2009 02:56:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58046 invoked by uid 99); 28 Jul 2009 02:56:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 02:56:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 02:56:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D3AF7234C044
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 19:56:14 -0700 (PDT)
Message-ID: <1809798463.1248749774852.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 19:56:14 -0700 (PDT)
From: "Edward J. Yoon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5988) Add a command to ' FsShell stat ' to
 get a file's block location information
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Edward J. Yoon updated HADOOP-5988:
-----------------------------------

    Attachment: HADOOP-5988_v01.patch

I just changed output like this.

{code}
[root@udanax hadoop-common-trunk]# bin/hadoop fs -stat %B tmp/aa.tar
09/07/28 11:48:26 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
+----------------------------------+
| byte-range:0-67108864            |
| location:udanax.org;             |
+----------------------------------+
| byte-range:67108864-134217728    |
| location:udanax.org;             |
+----------------------------------+
 ..
+----------------------------------+
| byte-range:2147483648-2165155840 |
| location:udanax.org;             |
+----------------------------------+

[root@udanax hadoop-common-trunk]# bin/hadoop fs -stat %B tmp/aa.ta
09/07/28 11:50:00 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
stat: cannot stat `tmp/aa.ta': No such file or directory
{code}

> Add a command to ' FsShell stat ' to get a file's block location information
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5988
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5988
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: He Yongqiang
>         Attachments: HADOOP-5988.patch, HADOOP-5988_v01.patch
>
>
> Adding an option to ' FsShell stat ' to get a file's block location information will be very useful.
> we can print the block location information in this format:
> blockID:XXXXX  byte-range:YYYY-ZZZZ  location:dn1;dn2;
> blockID:XXXXX  byte-range:YYYY-ZZZZ  location:dn1;dn2;

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-557-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 28 03:56:23 2009
Return-Path: <common-issues-return-557-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5689 invoked from network); 28 Jul 2009 03:56:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Jul 2009 03:56:22 -0000
Received: (qmail 91562 invoked by uid 500); 28 Jul 2009 03:57:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91518 invoked by uid 500); 28 Jul 2009 03:57:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91508 invoked by uid 99); 28 Jul 2009 03:57:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 03:57:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 03:57:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C72F6234C044
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 20:57:14 -0700 (PDT)
Message-ID: <23700578.1248753434801.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 20:57:14 -0700 (PDT)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735908#action_12735908 ] 

Scott Carey commented on HADOOP-6166:
-------------------------------------

For the 32 bit results, try passing -server on the command line.  It behaves quite differently with loop unrolling and certain low level optimizations in the JIT versus -client (which is only default on 32 bit windows, and anyone who would run Hadoop there and wanted better performance would pass -server to speed it up).

Are you specifying a -Xmx memory value?  What about -Xms?  On windows with -client, the VM has unusual default memory and GC values, I've found that setting its NewRatio more like the other platforms helps a lot: -XX:NewRatio=4 or something like that may make your results more consistent across the platforms (and faster on 32 bit windows).

On my environment, on the previous set of tests, changing from _10 to _12 to _14 on JDK6 did not seem to do much.  But I was manually setting -Xmx512m for all of my tests.  I can try again later, but there is something odd about the results slowing down so much on the 1.6.0_14 version.

It is also curious that the PureJavaCrc32New -- which only changes the loop style --also slows down but not as much as the older PureJavaCrc32 and goes from always about 15% slower to a little bit faster.  My guess is something configuration related has changed with respect to some default JVM settings. 

I think there may be some improvement possible in the 8_8 case in how the 9 XORs at the end are done.  Perhaps all in one line?  or in 3 sets of 3?  Or more likely the compiler is smart enough to do the register optimization itself?  Perhaps not, Intel's C code even avoids a single line with more than 4 XORs at once for some reason.

> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt, c6166_20090727.patch
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-558-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 28 04:59:22 2009
Return-Path: <common-issues-return-558-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13436 invoked from network); 28 Jul 2009 04:59:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Jul 2009 04:59:22 -0000
Received: (qmail 26751 invoked by uid 500); 28 Jul 2009 05:00:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26708 invoked by uid 500); 28 Jul 2009 05:00:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26698 invoked by uid 99); 28 Jul 2009 05:00:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 05:00:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 05:00:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C3AA8234C044
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 22:00:14 -0700 (PDT)
Message-ID: <1452364760.1248757214789.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 22:00:14 -0700 (PDT)
From: "Daehyun Kim (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6153) RAgzip: multiple map tasks for a
 large gzipped file
In-Reply-To: <926568478.1247710994848.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daehyun Kim updated HADOOP-6153:
--------------------------------

    Attachment:     (was: HADOOP-6153.patch)

> RAgzip: multiple map tasks for a large gzipped file
> ---------------------------------------------------
>
>                 Key: HADOOP-6153
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6153
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, native
>         Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
>            Reporter: Daehyun Kim
>            Assignee: Daehyun Kim
>            Priority: Minor
>
> It support to enable multiple map tasks for one large gzipped file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-559-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 28 05:07:21 2009
Return-Path: <common-issues-return-559-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14125 invoked from network); 28 Jul 2009 05:07:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Jul 2009 05:07:21 -0000
Received: (qmail 30393 invoked by uid 500); 28 Jul 2009 05:08:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30347 invoked by uid 500); 28 Jul 2009 05:08:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30337 invoked by uid 99); 28 Jul 2009 05:08:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 05:08:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 05:08:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 53656234C1E6
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 22:08:15 -0700 (PDT)
Message-ID: <2019488582.1248757695340.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 22:08:15 -0700 (PDT)
From: "Daehyun Kim (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6153) RAgzip: multiple map tasks for a
 large gzipped file
In-Reply-To: <926568478.1247710994848.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daehyun Kim updated HADOOP-6153:
--------------------------------

    Status: In Progress  (was: Patch Available)

> RAgzip: multiple map tasks for a large gzipped file
> ---------------------------------------------------
>
>                 Key: HADOOP-6153
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6153
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, native
>         Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
>            Reporter: Daehyun Kim
>            Assignee: Daehyun Kim
>            Priority: Minor
>         Attachments: HADOOP-6153.patch
>
>
> It support to enable multiple map tasks for one large gzipped file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-560-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 28 05:07:21 2009
Return-Path: <common-issues-return-560-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14156 invoked from network); 28 Jul 2009 05:07:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Jul 2009 05:07:21 -0000
Received: (qmail 30463 invoked by uid 500); 28 Jul 2009 05:08:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30415 invoked by uid 500); 28 Jul 2009 05:08:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30378 invoked by uid 99); 28 Jul 2009 05:08:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 05:08:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 05:08:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 59649234C1EB
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 22:08:15 -0700 (PDT)
Message-ID: <941098028.1248757695365.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 22:08:15 -0700 (PDT)
From: "Daehyun Kim (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6153) RAgzip: multiple map tasks for a
 large gzipped file
In-Reply-To: <926568478.1247710994848.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daehyun Kim updated HADOOP-6153:
--------------------------------

    Status: Patch Available  (was: In Progress)

> RAgzip: multiple map tasks for a large gzipped file
> ---------------------------------------------------
>
>                 Key: HADOOP-6153
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6153
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, native
>         Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
>            Reporter: Daehyun Kim
>            Assignee: Daehyun Kim
>            Priority: Minor
>         Attachments: HADOOP-6153.patch
>
>
> It support to enable multiple map tasks for one large gzipped file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-561-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 28 05:07:24 2009
Return-Path: <common-issues-return-561-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14208 invoked from network); 28 Jul 2009 05:07:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Jul 2009 05:07:24 -0000
Received: (qmail 30710 invoked by uid 500); 28 Jul 2009 05:08:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30499 invoked by uid 500); 28 Jul 2009 05:08:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30479 invoked by uid 99); 28 Jul 2009 05:08:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 05:08:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 05:08:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3B4AE234C044
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 22:08:15 -0700 (PDT)
Message-ID: <1006199778.1248757695226.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 22:08:15 -0700 (PDT)
From: "Daehyun Kim (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6153) RAgzip: multiple map tasks for a
 large gzipped file
In-Reply-To: <926568478.1247710994848.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daehyun Kim updated HADOOP-6153:
--------------------------------

    Attachment: HADOOP-6153.patch

My patch brings the hunk error.
I think the reason I copied the text of the patch in my linux terminal and pasted the text to notepad.

> RAgzip: multiple map tasks for a large gzipped file
> ---------------------------------------------------
>
>                 Key: HADOOP-6153
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6153
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, native
>         Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
>            Reporter: Daehyun Kim
>            Assignee: Daehyun Kim
>            Priority: Minor
>         Attachments: HADOOP-6153.patch
>
>
> It support to enable multiple map tasks for one large gzipped file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-562-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 28 05:59:22 2009
Return-Path: <common-issues-return-562-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21170 invoked from network); 28 Jul 2009 05:59:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Jul 2009 05:59:21 -0000
Received: (qmail 63502 invoked by uid 500); 28 Jul 2009 06:00:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63454 invoked by uid 500); 28 Jul 2009 06:00:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63444 invoked by uid 99); 28 Jul 2009 06:00:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 06:00:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 06:00:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EDCDC234C046
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 23:00:14 -0700 (PDT)
Message-ID: <1570876304.1248760814965.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 23:00:14 -0700 (PDT)
From: "He Yongqiang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5438) Merge FileSystem.create and
 FileSystem.append
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735950#action_12735950 ] 

He Yongqiang commented on HADOOP-5438:
--------------------------------------

{quote}
If there weren't so many different create methods, I would suggest having one each for just the Enum value and and equivalent one that takes an enumset, so that if you just wanted to overwrite you could just use the one that takes the enum, and the convenience method would jam it into an enumset. It's too bad Java won't automatically convert a single enum into an enumset when necessary.
{quote}
Agree. It will be better we can provide a create method which takes one CreateFlag enum value and equivalent one that takes an enumset (i think it's already there). But how about the other already existing create methods? Remove them? And this will be a incompatible change.

>>We could use a variable arity parameter, e.g.: FSDataOutputStream create(Path file, CreateFlag... flags);
This is a very good suggestion. some programmer(like me) does not use EnumSet.of() very often. But the situation can be mitigated if we provide a method that accept one CreateFlag parameter.

> Merge FileSystem.create and FileSystem.append
> ---------------------------------------------
>
>                 Key: HADOOP-5438
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5438
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: He Yongqiang
>            Assignee: He Yongqiang
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-5438(2009-04-06).patch, Hadoop-5438-2009-03-30.patch, Hadoop-5438-2009-03-31-2.patch, Hadoop-5438-2009-03-31.patch, Hadoop-5438-2009-05-10.patch, Hadoop-5438-2009-05-15.patch, Hadoop-5438-2009-05-19.patch, Hadoop-5438-2009-05-5.patch
>
>
> Currently, when a user wants to modify a file, the user first calls exists() to know if this file is already there. And then uses create() or append() according to whether the file exists or not.
> the code looks like:
> {code}
> FSDataOutputStream out_1 = null;
> if (fs.exists(path_1))
>    out_1 = fs.append(path_1);
> else
>    out_1 = fs.create(path_1);
> {code}
> . On the performace side,It involes two RPCs. On the easy-of-use side, it is not very convient in contrast to the traditional open interface.
> It will more complicate if there is a overwrite parameter specified. I donot know whether there is a bug about 'overwrite' in 0.19, some times it takes a long time for overwrite creates to reture. So i make the write file code with overwrite param works like:
> {code}
> boolean exists = fs.exists(name);
> if (overwrite) {
>     if (exists)
>        fs.delete(name, true);
>      this.out = fs.create(name, overwrite, bufferSize, replication,
> 				    blockSize, progress);
>      this.currentRowID = 0;
>  } else {
>    if (!exists)
> 	this.out = fs.create(name, overwrite, bufferSize,
> 					replication, blockSize, progress);
>    else
> 	this.out = fs.append(name, bufferSize, progress);
> {code}
> Some code statements there are really redundant and not needed, especialy with the delete(). But without deleting first, the overwrite takes a long time to reture.
> BTW, i will create another issue about the overwrite problem. If it is not a bug at all or a duplicate, someone please close it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-563-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 28 07:19:01 2009
Return-Path: <common-issues-return-563-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9940 invoked from network); 28 Jul 2009 07:19:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Jul 2009 07:19:01 -0000
Received: (qmail 13649 invoked by uid 500); 28 Jul 2009 06:53:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13603 invoked by uid 500); 28 Jul 2009 06:53:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13593 invoked by uid 99); 28 Jul 2009 06:53:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 06:53:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 06:53:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D5E32234C044
	for <common-issues@hadoop.apache.org>; Mon, 27 Jul 2009 23:53:14 -0700 (PDT)
Message-ID: <1466861872.1248763994861.JavaMail.jira@brutus>
Date: Mon, 27 Jul 2009 23:53:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6153) RAgzip: multiple map tasks for a
 large gzipped file
In-Reply-To: <926568478.1247710994848.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735962#action_12735962 ] 

Hadoop QA commented on HADOOP-6153:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12414706/HADOOP-6153.patch
  against trunk revision 798247.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/590/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/590/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/590/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/590/console

This message is automatically generated.

> RAgzip: multiple map tasks for a large gzipped file
> ---------------------------------------------------
>
>                 Key: HADOOP-6153
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6153
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, native
>         Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
>            Reporter: Daehyun Kim
>            Assignee: Daehyun Kim
>            Priority: Minor
>         Attachments: HADOOP-6153.patch
>
>
> It support to enable multiple map tasks for one large gzipped file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-564-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 28 08:22:20 2009
Return-Path: <common-issues-return-564-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32875 invoked from network); 28 Jul 2009 08:22:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Jul 2009 08:22:20 -0000
Received: (qmail 89040 invoked by uid 500); 28 Jul 2009 08:23:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88993 invoked by uid 500); 28 Jul 2009 08:23:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88983 invoked by uid 99); 28 Jul 2009 08:23:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 08:23:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 08:23:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D1A19234C044
	for <common-issues@hadoop.apache.org>; Tue, 28 Jul 2009 01:23:14 -0700 (PDT)
Message-ID: <1756655081.1248769394844.JavaMail.jira@brutus>
Date: Tue, 28 Jul 2009 01:23:14 -0700 (PDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4491) Per-job local data on the
 TaskTracker node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735980#action_12735980 ] 

Vinod K V commented on HADOOP-4491:
-----------------------------------

Going forward with the latest proposal.

Salient points:
 - Same directory structure as what is in the trunk.
 - Actions taken up the linux-taskcontroller:
    -- initializeJob
          --- sudo chown user:mapred -R taskTracker/jobcache/$jobid
          --- sudo chmod 570 -R taskTracker/jobcache/$jobid
    -- initializeTask
          --- sudo chown user:mapred -R taskTracker/jobcache/$jobid/$attemptid/
          --- sudo chmod 770 -R taskTracker/jobcache/$jobid/$attemptid/
          --- sudo chmod +s taskTracker/jobcache/$jobid/$attemptid
          --- 
          --- sudo chown user:mapred log-dir/userlogs/$attemptid
          --- sudo chmod 770 log-dir/userlogs/$attemptid
          --- sudo chmod +s log-dir/userlogs/$attemptid

child umask 0007


> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-565-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 28 10:36:20 2009
Return-Path: <common-issues-return-565-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91308 invoked from network); 28 Jul 2009 10:36:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Jul 2009 10:36:20 -0000
Received: (qmail 55092 invoked by uid 500); 28 Jul 2009 10:37:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55051 invoked by uid 500); 28 Jul 2009 10:37:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55041 invoked by uid 99); 28 Jul 2009 10:37:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 10:37:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 10:37:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C7AEC234C044
	for <common-issues@hadoop.apache.org>; Tue, 28 Jul 2009 03:37:14 -0700 (PDT)
Message-ID: <1794771284.1248777434803.JavaMail.jira@brutus>
Date: Tue, 28 Jul 2009 03:37:14 -0700 (PDT)
From: "Sreekanth Ramakrishnan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5420) Support killing of process groups in
 LinuxTaskController binary
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sreekanth Ramakrishnan updated HADOOP-5420:
-------------------------------------------

    Attachment: 5420-ydist.patch.txt

Patch to be used for Yahoo distribution

> Support killing of process groups in LinuxTaskController binary
> ---------------------------------------------------------------
>
>                 Key: HADOOP-5420
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5420
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Sreekanth Ramakrishnan
>            Assignee: Sreekanth Ramakrishnan
>             Fix For: 0.21.0
>
>         Attachments: 5420-fix-ydist.patch, 5420-ydist.patch.txt, hadoop-5420-1.patch, hadoop-5420-10.patch, hadoop-5420-11.patch, hadoop-5420-12.patch, hadoop-5420-2.patch, hadoop-5420-3.patch, hadoop-5420-4.patch, hadoop-5420-5.patch, hadoop-5420-6.patch, hadoop-5420-7.patch, hadoop-5420-8.patch, hadoop-5420-9.patch, HADOOP-5420-v20.patch, hadoop-5420.patch
>
>
> Support setsid based kill in LinuxTaskController.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-566-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 28 11:10:20 2009
Return-Path: <common-issues-return-566-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2049 invoked from network); 28 Jul 2009 11:10:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Jul 2009 11:10:20 -0000
Received: (qmail 16033 invoked by uid 500); 28 Jul 2009 11:11:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15994 invoked by uid 500); 28 Jul 2009 11:11:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15984 invoked by uid 99); 28 Jul 2009 11:11:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 11:11:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 11:11:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3E69E234C044
	for <common-issues@hadoop.apache.org>; Tue, 28 Jul 2009 04:11:15 -0700 (PDT)
Message-ID: <247027378.1248779475238.JavaMail.jira@brutus>
Date: Tue, 28 Jul 2009 04:11:15 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6123) hdfs script does not work after
 project split.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12736034#action_12736034 ] 

Hudson commented on HADOOP-6123:
--------------------------------

Integrated in Hadoop-Common-trunk #37 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/37/])
    . Add missing classpaths in hadoop-config.sh.  Contributed by Sharad Agarwal


> hdfs script does not work after project split.
> ----------------------------------------------
>
>                 Key: HADOOP-6123
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6123
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Sharad Agarwal
>             Fix For: 0.21.0
>
>         Attachments: 6123_v1.patch
>
>
> There are problems running hdfs script from common.
> # Usage message for hdfs does not have "fs" option anymore.
> # There seem to be an undocumented option "dfs", but using it throws NoClassDefFoundError or ClassNotFoundException.
> # Same with other options e.g. name-node.
> May I am missing something here. How do we call ls these days?
> Do we need to move hdfs script to hdfs project.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-567-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 28 18:39:19 2009
Return-Path: <common-issues-return-567-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11513 invoked from network); 28 Jul 2009 18:39:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Jul 2009 18:39:19 -0000
Received: (qmail 68982 invoked by uid 500); 28 Jul 2009 18:40:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68938 invoked by uid 500); 28 Jul 2009 18:40:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68918 invoked by uid 99); 28 Jul 2009 18:40:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 18:40:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 18:40:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DAB04234C04B
	for <common-issues@hadoop.apache.org>; Tue, 28 Jul 2009 11:40:14 -0700 (PDT)
Message-ID: <1294736091.1248806414894.JavaMail.jira@brutus>
Date: Tue, 28 Jul 2009 11:40:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6146) Upgrade to JetS3t version 0.7.1
In-Reply-To: <13490839.1247490076513.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12736212#action_12736212 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6146:
------------------------------------------------

trunk/.eclipse.templates/.classpath still has
{code}
	<classpathentry kind="lib" path="build/ivy/lib/Hadoop-Core/common/jets3t-0.6.1.jar"/>
{code}
It would be great if the .classpath file is generated by a script.  I guess it may not be easy to write such script.

> Upgrade to JetS3t version 0.7.1
> -------------------------------
>
>                 Key: HADOOP-6146
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6146
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6146.patch
>
>
> The JetS3t library is used for the S3 filesystems. We should upgrade to the latest version (0.7.1) which has support for Requester Pays buckets. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-568-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 28 18:53:19 2009
Return-Path: <common-issues-return-568-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15317 invoked from network); 28 Jul 2009 18:53:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Jul 2009 18:53:19 -0000
Received: (qmail 98289 invoked by uid 500); 28 Jul 2009 18:54:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98244 invoked by uid 500); 28 Jul 2009 18:54:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98234 invoked by uid 99); 28 Jul 2009 18:54:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 18:54:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 18:54:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DD44B234C04B
	for <common-issues@hadoop.apache.org>; Tue, 28 Jul 2009 11:54:14 -0700 (PDT)
Message-ID: <2071586940.1248807254905.JavaMail.jira@brutus>
Date: Tue, 28 Jul 2009 11:54:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6166:
-------------------------------------------

    Attachment: c6166_20090728.patch

c6166_20090728.patch: included Crc32_16_16


> For the 32 bit results, try passing -server on the command line. ...
Here is the result:

java.version = 1.6.0_14
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_14-b08
java.vm.version = 14.0-b16
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = x86
os.name = Windows XP
os.version = 5.1
||num bytes||PureJavaCrc32 MB/sec||PureJavaCrc32New MB/sec||Crc32_3_2 MB/sec||Crc32_4_3 MB/sec||Crc32_5_5 MB/sec||Crc32_6_6 MB/sec||Crc32_8_8 MB/sec||Crc32_12_12 MB/sec||Crc32_16_16 MB/sec||
|        8 |  138.935 |  148.510 |  133.888 |  174.420 |  142.309 |  148.559 |  202.270 |  125.889 |  117.607 |
|       16 |  195.238 |  179.688 |  194.082 |  196.024 |  202.448 |  174.408 |  231.516 |  181.476 |  249.847 |
|       32 |  239.042 |  212.647 |  218.873 |  214.975 |  238.313 |  234.569 |  285.546 |  222.713 |  282.443 |
|       64 |  267.240 |  236.977 |  248.711 |  224.998 |  272.373 |  259.976 |  314.990 |  268.683 |  306.000 |
|      128 |  282.564 |  261.874 |  258.325 |  195.558 |  183.524 |  290.901 |  339.453 |  307.891 |  285.557 |
|      256 |  286.647 |  271.146 |  270.484 |  224.961 |  288.691 |  307.519 |  352.148 |  337.360 |  312.192 |
|      512 |  298.539 |  276.192 |  274.773 |  236.895 |  336.279 |  315.217 |  361.232 |  346.809 |  319.615 |
|     1024 |  303.658 |  279.882 |  276.542 |  236.183 |  340.919 |  325.135 |  364.909 |  352.689 |  319.080 |
|     2048 |  309.358 |  285.787 |  273.328 |  236.416 |  345.868 |  327.777 |  368.106 |  357.019 |  321.033 |
|     4096 |  306.306 |  285.192 |  272.680 |  237.541 |  343.045 |  327.025 |  368.837 |  358.270 |  322.088 |
|     8192 |  307.772 |  288.171 |  272.977 |  237.316 |  348.833 |  328.908 |  373.525 |  361.827 |  322.454 |
|    16384 |  307.900 |  286.654 |  273.482 |  236.011 |  332.936 |  328.303 |  370.397 |  359.706 |  320.460 |
|    32768 |  302.599 |  285.929 |  273.000 |  237.496 |  343.129 |  328.161 |  368.144 |  360.141 |  320.854 |
|    65536 |  305.564 |  285.796 |  273.027 |  236.645 |  342.567 |  329.054 |  369.318 |  360.611 |  322.333 |
|   131072 |  306.763 |  285.466 |  274.336 |  237.648 |  344.286 |  329.910 |  373.027 |  360.100 |  320.236 |
|   262144 |  302.322 |  286.444 |  273.267 |  236.971 |  345.512 |  327.882 |  370.549 |  358.936 |  320.964 |
|   524288 |  304.555 |  284.659 |  272.150 |  235.174 |  342.026 |  327.074 |  369.213 |  359.436 |  316.547 |
|  1048576 |  301.722 |  279.686 |  271.529 |  235.130 |  338.665 |  324.743 |  365.818 |  352.513 |  315.451 |
|  2097152 |  301.360 |  282.853 |  270.846 |  232.843 |  336.175 |  322.065 |  362.790 |  356.372 |  317.965 |
|  4194304 |  298.921 |  283.021 |  269.376 |  233.498 |  336.376 |  321.957 |  365.402 |  354.546 |  299.699 |
|  8388608 |  250.164 |  281.916 |  269.071 |  234.353 |  338.636 |  325.124 |  365.995 |  353.549 |  312.460 |
| 16777216 |  290.762 |  264.850 |  270.366 |  235.145 |  338.756 |  321.101 |  364.583 |  353.767 |  316.974 |

> Are you specifying a -Xmx memory value? What about -Xms?
I have -Xmx512m but no -Xms.  Any suggestion?

> It is also curious that the PureJavaCrc32New - which only changes the loop style ...
This trick does not always work: PureJavaCrc32New was slower in the results shown above.

> I think there may be some improvement possible in the 8_8 case in how the 9 XORs at the end are done. ...
Yeah, we should try.

Thanks, Scott.

> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt, c6166_20090727.patch, c6166_20090728.patch
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-569-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 28 22:02:39 2009
Return-Path: <common-issues-return-569-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42976 invoked from network); 28 Jul 2009 22:02:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Jul 2009 22:02:38 -0000
Received: (qmail 56964 invoked by uid 500); 28 Jul 2009 22:02:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56940 invoked by uid 500); 28 Jul 2009 22:02:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56930 invoked by uid 99); 28 Jul 2009 22:02:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 22:02:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 22:02:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E05E7234C045
	for <common-issues@hadoop.apache.org>; Tue, 28 Jul 2009 15:02:14 -0700 (PDT)
Message-ID: <1566370037.1248818534917.JavaMail.jira@brutus>
Date: Tue, 28 Jul 2009 15:02:14 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6170) add Avro-based RPC serialization
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

add Avro-based RPC serialization
--------------------------------

                 Key: HADOOP-6170
                 URL: https://issues.apache.org/jira/browse/HADOOP-6170
             Project: Hadoop Common
          Issue Type: New Feature
            Reporter: Doug Cutting
            Assignee: Doug Cutting
             Fix For: 0.21.0


Permit RPC protocols to use Avro to serialize requests and responses, so that protocols may better evolve without breaking compatibility.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-570-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 28 22:13:37 2009
Return-Path: <common-issues-return-570-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49214 invoked from network); 28 Jul 2009 22:13:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Jul 2009 22:13:36 -0000
Received: (qmail 69826 invoked by uid 500); 28 Jul 2009 22:13:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69792 invoked by uid 500); 28 Jul 2009 22:13:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69725 invoked by uid 99); 28 Jul 2009 22:13:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 22:13:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jul 2009 22:13:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CC450234C044
	for <common-issues@hadoop.apache.org>; Tue, 28 Jul 2009 15:13:14 -0700 (PDT)
Message-ID: <2017280446.1248819194819.JavaMail.jira@brutus>
Date: Tue, 28 Jul 2009 15:13:14 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6170) add Avro-based RPC serialization
In-Reply-To: <1566370037.1248818534917.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12736331#action_12736331 ] 

Doug Cutting commented on HADOOP-6170:
--------------------------------------

To implement this, an Avro Transciever can be implemented using Hadoop's Client and Server classes.  The request and response Writable would contain just List<ByteBuffer>.

> add Avro-based RPC serialization
> --------------------------------
>
>                 Key: HADOOP-6170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6170
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: 0.21.0
>
>
> Permit RPC protocols to use Avro to serialize requests and responses, so that protocols may better evolve without breaking compatibility.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-571-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 29 10:35:39 2009
Return-Path: <common-issues-return-571-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98447 invoked from network); 29 Jul 2009 10:35:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Jul 2009 10:35:39 -0000
Received: (qmail 26219 invoked by uid 500); 29 Jul 2009 10:35:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26199 invoked by uid 500); 29 Jul 2009 10:35:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26189 invoked by uid 99); 29 Jul 2009 10:35:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jul 2009 10:35:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jul 2009 10:35:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 25079234C1E6
	for <common-issues@hadoop.apache.org>; Wed, 29 Jul 2009 03:35:15 -0700 (PDT)
Message-ID: <616222003.1248863715150.JavaMail.jira@brutus>
Date: Wed, 29 Jul 2009 03:35:15 -0700 (PDT)
From: "Amar Kamat (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-5436) job history directory grows without
 bound, locks up job tracker on new job submission
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amar Kamat resolved HADOOP-5436.
--------------------------------

    Resolution: Fixed

I think locking the jobtracker cannot be avoided as its inline with heartbeat. MAPREDUCE-786 should make the JobHistory calls faster.

> job history directory grows without bound, locks up job tracker on new job submission
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5436
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5436
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.19.0
>            Reporter: Tim Williamson
>         Attachments: HADOOP-5436.patch
>
>
> An unpleasant surprise upgrading to 0.19: requests to jobtracker.jsp would take a long time or even time out whenever new jobs where submitted.  Investigation showed the call to JobInProgress.initTasks() was calling JobHistory.JobInfo.logSubmitted() which in turn was calling JobHistory.getJobHistoryFileName() which was pegging the CPU for a couple minutes.  Further investigation showed the were 200,000+ files in the job history folder -- and every submission was creating a FileStatus for them all, then applying a regular expression to just the name.  All this just on the off chance the job tracker had been restarted (see HADOOP-3245).  To make matters worse, these files cannot be safely deleted while the job tracker is running, as the disappearance of a history file at the wrong time causes a FileNotFoundException.
> So to summarize the issues:
> - having Hadoop default to storing all the history files in a single directory is a Bad Idea
> - doing expensive processing of every history file on every job submission is a Worse Idea
> - doing expensive processing of every history file on every job submission while holding a lock on the JobInProgress object and thereby blocking the jobtracker.jsp from rendering is a Terrible Idea (note: haven't confirmed this, but a cursory glance suggests that's what's going on)
> - not being able to clean up the mess without taking down the job tracker is just Unfortunate

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-572-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 29 19:08:36 2009
Return-Path: <common-issues-return-572-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 157 invoked from network); 29 Jul 2009 19:08:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Jul 2009 19:08:36 -0000
Received: (qmail 73383 invoked by uid 500); 29 Jul 2009 19:08:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73365 invoked by uid 500); 29 Jul 2009 19:08:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73347 invoked by uid 99); 29 Jul 2009 19:08:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jul 2009 19:08:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jul 2009 19:08:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D5257234C04C
	for <common-issues@hadoop.apache.org>; Wed, 29 Jul 2009 12:08:14 -0700 (PDT)
Message-ID: <490201341.1248894494872.JavaMail.jira@brutus>
Date: Wed, 29 Jul 2009 12:08:14 -0700 (PDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5436) job history directory grows without
 bound, locks up job tracker on new job submission
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12736758#action_12736758 ] 

Arun C Murthy commented on HADOOP-5436:
---------------------------------------

I don't see a commit here... how is this issue 'FIXED' ? Is it a duplicate of some other bug?

If none of the above, we should mark this jira as 'wontfix'/'invalid' or keep it open if we haven't satisfied the request...

> job history directory grows without bound, locks up job tracker on new job submission
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5436
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5436
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.19.0
>            Reporter: Tim Williamson
>         Attachments: HADOOP-5436.patch
>
>
> An unpleasant surprise upgrading to 0.19: requests to jobtracker.jsp would take a long time or even time out whenever new jobs where submitted.  Investigation showed the call to JobInProgress.initTasks() was calling JobHistory.JobInfo.logSubmitted() which in turn was calling JobHistory.getJobHistoryFileName() which was pegging the CPU for a couple minutes.  Further investigation showed the were 200,000+ files in the job history folder -- and every submission was creating a FileStatus for them all, then applying a regular expression to just the name.  All this just on the off chance the job tracker had been restarted (see HADOOP-3245).  To make matters worse, these files cannot be safely deleted while the job tracker is running, as the disappearance of a history file at the wrong time causes a FileNotFoundException.
> So to summarize the issues:
> - having Hadoop default to storing all the history files in a single directory is a Bad Idea
> - doing expensive processing of every history file on every job submission is a Worse Idea
> - doing expensive processing of every history file on every job submission while holding a lock on the JobInProgress object and thereby blocking the jobtracker.jsp from rendering is a Terrible Idea (note: haven't confirmed this, but a cursory glance suggests that's what's going on)
> - not being able to clean up the mess without taking down the job tracker is just Unfortunate

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-573-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 30 06:56:36 2009
Return-Path: <common-issues-return-573-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56326 invoked from network); 30 Jul 2009 06:56:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Jul 2009 06:56:36 -0000
Received: (qmail 83270 invoked by uid 500); 30 Jul 2009 06:56:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83234 invoked by uid 500); 30 Jul 2009 06:56:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83224 invoked by uid 99); 30 Jul 2009 06:56:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jul 2009 06:56:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jul 2009 06:56:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D1B29234C046
	for <common-issues@hadoop.apache.org>; Wed, 29 Jul 2009 23:56:14 -0700 (PDT)
Message-ID: <1937055044.1248936974858.JavaMail.jira@brutus>
Date: Wed, 29 Jul 2009 23:56:14 -0700 (PDT)
From: "Nathan Marz (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-5330) Zombie tasks remain after jobs
 finish/fail/get killed
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nathan Marz resolved HADOOP-5330.
---------------------------------

    Resolution: Invalid

As it turns out, this was caused by an application problem. We were running embedded Solr instances in the tasks that were preventing the process from exiting. The fix was to close the Solr instances at task completion.

> Zombie tasks remain after jobs finish/fail/get killed
> -----------------------------------------------------
>
>                 Key: HADOOP-5330
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5330
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.19.1
>            Reporter: Nathan Marz
>
> I'm seeing a lot of "task attempts" around our hadoop cluster for jobs that are no longer around. The attempts seem to be "hung", as they sit there forever. Additionally, they seem to take up map and reduce slots in the cluster unless MapReduce is restarted. This causes real jobs to be unable to utilize the whole cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-574-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 30 23:21:38 2009
Return-Path: <common-issues-return-574-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38339 invoked from network); 30 Jul 2009 23:21:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Jul 2009 23:21:38 -0000
Received: (qmail 68968 invoked by uid 500); 30 Jul 2009 23:21:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68924 invoked by uid 500); 30 Jul 2009 23:21:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68914 invoked by uid 99); 30 Jul 2009 23:21:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jul 2009 23:21:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jul 2009 23:21:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CE5D5234C1E9
	for <common-issues@hadoop.apache.org>; Thu, 30 Jul 2009 16:21:14 -0700 (PDT)
Message-ID: <1734084494.1248996074844.JavaMail.jira@brutus>
Date: Thu, 30 Jul 2009 16:21:14 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6171) "package" task in build.xml should
 copy source with preservelastmodified
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

"package" task in build.xml should copy source with preservelastmodified
------------------------------------------------------------------------

                 Key: HADOOP-6171
                 URL: https://issues.apache.org/jira/browse/HADOOP-6171
             Project: Hadoop Common
          Issue Type: Bug
          Components: build
            Reporter: Todd Lipcon


In the package task, it copies the source to dist.dir without using preservelastmodified=true. This can cause issues with the autotools configure process for the C++ builds. Namely, it will sometimes think the configure script is out of date with respect to its source files and try to re-bootstrap, which relies on particular versions of autotools on the building computer. This isn't something that should be required for those wanting to build from a distribution tarball.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-575-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 00:11:38 2009
Return-Path: <common-issues-return-575-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45430 invoked from network); 31 Jul 2009 00:11:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 00:11:38 -0000
Received: (qmail 1319 invoked by uid 500); 31 Jul 2009 00:11:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1276 invoked by uid 500); 31 Jul 2009 00:11:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1266 invoked by uid 99); 31 Jul 2009 00:11:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 00:11:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 00:11:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C81BA234C046
	for <common-issues@hadoop.apache.org>; Thu, 30 Jul 2009 17:11:14 -0700 (PDT)
Message-ID: <474244237.1248999074804.JavaMail.jira@brutus>
Date: Thu, 30 Jul 2009 17:11:14 -0700 (PDT)
From: "Raghu Angadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6169) Removing deprecated method calls in
 TFile
In-Reply-To: <927826654.1248487334841.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737373#action_12737373 ] 

Raghu Angadi commented on HADOOP-6169:
--------------------------------------

+1.

This seems to contain quite a few findbugs warnings also, which is good.

> Removing deprecated method calls in TFile
> -----------------------------------------
>
>                 Key: HADOOP-6169
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6169
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6169-20090725.patch
>
>
> We should remove the use of deprecated APIs in TFile.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-576-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 02:10:38 2009
Return-Path: <common-issues-return-576-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80296 invoked from network); 31 Jul 2009 02:10:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 02:10:38 -0000
Received: (qmail 64934 invoked by uid 500); 31 Jul 2009 02:10:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64889 invoked by uid 500); 31 Jul 2009 02:10:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64878 invoked by uid 99); 31 Jul 2009 02:10:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 02:10:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 02:10:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CFAA0234C046
	for <common-issues@hadoop.apache.org>; Thu, 30 Jul 2009 19:10:14 -0700 (PDT)
Message-ID: <848627495.1249006214836.JavaMail.jira@brutus>
Date: Thu, 30 Jul 2009 19:10:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6150) Need to be able to instantiate a
 comparator instance from a comparator string without creating a
 TFile.Reader object
In-Reply-To: <1804574098.1247618054845.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-6150:
------------------------------

       Resolution: Fixed
    Fix Version/s: 0.20.0
           Status: Resolved  (was: Patch Available)

> Need to be able to instantiate a comparator instance from a comparator string without creating a TFile.Reader object
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6150
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6150
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.0, 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0, 0.20.0
>
>         Attachments: HADOOP-6150-0721.patch, HADOOP-6150-0723-for-20.patch
>
>
> Occasionally, we want have the same instance of comparator object represented by the TFile comparator string. We should be able to do that without requiring to first open up a tfile that was previously written use the same comparator.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-577-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 02:12:38 2009
Return-Path: <common-issues-return-577-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81401 invoked from network); 31 Jul 2009 02:12:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 02:12:38 -0000
Received: (qmail 68198 invoked by uid 500); 31 Jul 2009 02:12:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68136 invoked by uid 500); 31 Jul 2009 02:12:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68117 invoked by uid 99); 31 Jul 2009 02:12:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 02:12:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 02:12:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E69F2234C1EB
	for <common-issues@hadoop.apache.org>; Thu, 30 Jul 2009 19:12:14 -0700 (PDT)
Message-ID: <1788812602.1249006334943.JavaMail.jira@brutus>
Date: Thu, 30 Jul 2009 19:12:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4162)
 CodecPool.getDecompressor(LzopCodec) always creates a brand-new
 decompressor.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737396#action_12737396 ] 

Hong Tang commented on HADOOP-4162:
-----------------------------------

LZO is not part of hadoop any more... Will mark the issue invalid.

> CodecPool.getDecompressor(LzopCodec) always creates a brand-new decompressor.
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-4162
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4162
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.18.0
>            Reporter: Hong Tang
>            Assignee: Arun C Murthy
>         Attachments: HADOOP-4162_0_20080911.patch
>
>
> CodecPool.getDecompressor(LzopCodec) always creates a brand-new decompressor. I investigated the code, the reason seems to be the following:
> LzopCodec inherits from LzoCodec. The getDecompressorType() method is supposed to return the concrete Decompressor class type the specific Codec class creates. In this case, LzopCodec creates LzopDecompressors and should return LzopDecompressor.class. But instead, it uses the getDecompressorType() method defined in the parent and returns LzoDecompressor.class.
> This leads to CodecPool unable to properly recycle the decompressors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-578-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 02:12:38 2009
Return-Path: <common-issues-return-578-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81404 invoked from network); 31 Jul 2009 02:12:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 02:12:38 -0000
Received: (qmail 68216 invoked by uid 500); 31 Jul 2009 02:12:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68138 invoked by uid 500); 31 Jul 2009 02:12:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68116 invoked by uid 99); 31 Jul 2009 02:12:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 02:12:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 02:12:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id F41E929A0018
	for <common-issues@hadoop.apache.org>; Thu, 30 Jul 2009 19:12:14 -0700 (PDT)
Message-ID: <257215299.1249006334999.JavaMail.jira@brutus>
Date: Thu, 30 Jul 2009 19:12:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-4162) CodecPool.getDecompressor(LzopCodec)
 always creates a brand-new decompressor.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang resolved HADOOP-4162.
-------------------------------

    Resolution: Invalid

> CodecPool.getDecompressor(LzopCodec) always creates a brand-new decompressor.
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-4162
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4162
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.18.0
>            Reporter: Hong Tang
>            Assignee: Arun C Murthy
>         Attachments: HADOOP-4162_0_20080911.patch
>
>
> CodecPool.getDecompressor(LzopCodec) always creates a brand-new decompressor. I investigated the code, the reason seems to be the following:
> LzopCodec inherits from LzoCodec. The getDecompressorType() method is supposed to return the concrete Decompressor class type the specific Codec class creates. In this case, LzopCodec creates LzopDecompressors and should return LzopDecompressor.class. But instead, it uses the getDecompressorType() method defined in the parent and returns LzoDecompressor.class.
> This leads to CodecPool unable to properly recycle the decompressors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-579-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 07:34:47 2009
Return-Path: <common-issues-return-579-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65562 invoked from network); 31 Jul 2009 07:34:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 07:34:46 -0000
Received: (qmail 89592 invoked by uid 500); 31 Jul 2009 07:34:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89442 invoked by uid 500); 31 Jul 2009 07:34:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89422 invoked by uid 99); 31 Jul 2009 07:34:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 07:34:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 07:34:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0333C234C04C
	for <common-issues@hadoop.apache.org>; Fri, 31 Jul 2009 00:34:15 -0700 (PDT)
Message-ID: <816666106.1249025655012.JavaMail.jira@brutus>
Date: Fri, 31 Jul 2009 00:34:15 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6172) bin/hadoop version not working
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

bin/hadoop version not working
------------------------------

                 Key: HADOOP-6172
                 URL: https://issues.apache.org/jira/browse/HADOOP-6172
             Project: Hadoop Common
          Issue Type: Bug
          Components: build
    Affects Versions: 0.21.0
            Reporter: Hong Tang
            Priority: Minor


Two problems found:
- ${build.src} not included in ant target "compile-core-classes", thus o.a.h.package-info.java is not compiled, which contains the version annotation.
- bin/hadoop-config.sh attempts to include jar files matching pattern hadoop-*-core.jar rather than hadoop-core-*.jar.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-580-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 07:40:43 2009
Return-Path: <common-issues-return-580-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66789 invoked from network); 31 Jul 2009 07:40:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 07:40:43 -0000
Received: (qmail 93996 invoked by uid 500); 31 Jul 2009 07:40:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93950 invoked by uid 500); 31 Jul 2009 07:40:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93939 invoked by uid 99); 31 Jul 2009 07:40:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 07:40:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 07:40:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D2457234C1E6
	for <common-issues@hadoop.apache.org>; Fri, 31 Jul 2009 00:40:14 -0700 (PDT)
Message-ID: <2043251489.1249026014860.JavaMail.jira@brutus>
Date: Fri, 31 Jul 2009 00:40:14 -0700 (PDT)
From: "Sreekanth Ramakrishnan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6172) bin/hadoop version not working
In-Reply-To: <816666106.1249025655012.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737449#action_12737449 ] 

Sreekanth Ramakrishnan commented on HADOOP-6172:
------------------------------------------------

HADOOP-4503 actually talks about the first problem which is mentioned in this JIRA.


> bin/hadoop version not working
> ------------------------------
>
>                 Key: HADOOP-6172
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6172
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Priority: Minor
>
> Two problems found:
> - ${build.src} not included in ant target "compile-core-classes", thus o.a.h.package-info.java is not compiled, which contains the version annotation.
> - bin/hadoop-config.sh attempts to include jar files matching pattern hadoop-*-core.jar rather than hadoop-core-*.jar.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-581-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 07:40:44 2009
Return-Path: <common-issues-return-581-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66825 invoked from network); 31 Jul 2009 07:40:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 07:40:44 -0000
Received: (qmail 94065 invoked by uid 500); 31 Jul 2009 07:40:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94020 invoked by uid 500); 31 Jul 2009 07:40:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94010 invoked by uid 99); 31 Jul 2009 07:40:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 07:40:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 07:40:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C667D234C046
	for <common-issues@hadoop.apache.org>; Fri, 31 Jul 2009 00:40:14 -0700 (PDT)
Message-ID: <689112236.1249026014805.JavaMail.jira@brutus>
Date: Fri, 31 Jul 2009 00:40:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6172) bin/hadoop version not working
In-Reply-To: <816666106.1249025655012.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737448#action_12737448 ] 

Hong Tang commented on HADOOP-6172:
-----------------------------------

Problem symptoms:
% bin/hadoop version
Hadoop Unknown
Subversion Unknown -r Unknown
Compiled by Unknown on Unknown
>From source with checksum Unknown

% build/hadoop-core-0.21.0-dev/bin/hadoop version
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/VersionInfo
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.VersionInfo
        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:319)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:330)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:254)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:402)

> bin/hadoop version not working
> ------------------------------
>
>                 Key: HADOOP-6172
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6172
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Priority: Minor
>
> Two problems found:
> - ${build.src} not included in ant target "compile-core-classes", thus o.a.h.package-info.java is not compiled, which contains the version annotation.
> - bin/hadoop-config.sh attempts to include jar files matching pattern hadoop-*-core.jar rather than hadoop-core-*.jar.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-582-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 07:42:43 2009
Return-Path: <common-issues-return-582-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67276 invoked from network); 31 Jul 2009 07:42:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 07:42:43 -0000
Received: (qmail 96756 invoked by uid 500); 31 Jul 2009 07:42:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96710 invoked by uid 500); 31 Jul 2009 07:42:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96698 invoked by uid 99); 31 Jul 2009 07:42:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 07:42:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 07:42:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C72C8234C046
	for <common-issues@hadoop.apache.org>; Fri, 31 Jul 2009 00:42:14 -0700 (PDT)
Message-ID: <1707678108.1249026134811.JavaMail.jira@brutus>
Date: Fri, 31 Jul 2009 00:42:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6172) bin/hadoop version not working
In-Reply-To: <816666106.1249025655012.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-6172:
------------------------------

    Attachment: hadoop-6172-20090731.patch

> bin/hadoop version not working
> ------------------------------
>
>                 Key: HADOOP-6172
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6172
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6172-20090731.patch
>
>
> Two problems found:
> - ${build.src} not included in ant target "compile-core-classes", thus o.a.h.package-info.java is not compiled, which contains the version annotation.
> - bin/hadoop-config.sh attempts to include jar files matching pattern hadoop-*-core.jar rather than hadoop-core-*.jar.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-583-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 07:42:45 2009
Return-Path: <common-issues-return-583-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67295 invoked from network); 31 Jul 2009 07:42:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 07:42:45 -0000
Received: (qmail 96834 invoked by uid 500); 31 Jul 2009 07:42:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96787 invoked by uid 500); 31 Jul 2009 07:42:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96777 invoked by uid 99); 31 Jul 2009 07:42:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 07:42:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 07:42:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D6482234C1E9
	for <common-issues@hadoop.apache.org>; Fri, 31 Jul 2009 00:42:14 -0700 (PDT)
Message-ID: <81545953.1249026134876.JavaMail.jira@brutus>
Date: Fri, 31 Jul 2009 00:42:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6172) bin/hadoop version not working
In-Reply-To: <816666106.1249025655012.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-6172:
------------------------------

    Fix Version/s: 0.21.0
         Assignee: Hong Tang
           Status: Patch Available  (was: Open)

Trivial patch that fixes both problems. No junit test required. Manually tested and confirmed to resolve the problems.

> bin/hadoop version not working
> ------------------------------
>
>                 Key: HADOOP-6172
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6172
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6172-20090731.patch
>
>
> Two problems found:
> - ${build.src} not included in ant target "compile-core-classes", thus o.a.h.package-info.java is not compiled, which contains the version annotation.
> - bin/hadoop-config.sh attempts to include jar files matching pattern hadoop-*-core.jar rather than hadoop-core-*.jar.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-584-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 07:48:45 2009
Return-Path: <common-issues-return-584-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68376 invoked from network); 31 Jul 2009 07:48:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 07:48:45 -0000
Received: (qmail 6785 invoked by uid 500); 31 Jul 2009 07:48:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6767 invoked by uid 500); 31 Jul 2009 07:48:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6757 invoked by uid 99); 31 Jul 2009 07:48:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 07:48:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 07:48:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C5593234C046
	for <common-issues@hadoop.apache.org>; Fri, 31 Jul 2009 00:48:14 -0700 (PDT)
Message-ID: <206496305.1249026494804.JavaMail.jira@brutus>
Date: Fri, 31 Jul 2009 00:48:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6172) bin/hadoop version not working
In-Reply-To: <816666106.1249025655012.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737452#action_12737452 ] 

Hong Tang commented on HADOOP-6172:
-----------------------------------

@sreekanth, I am not sure if they are the same problem. For HADOOP-4503, call "ant jar" twice would resolve the problem. But for this one, a java file (build/src/org/apache/hadoop/package-info.java) is simply not compiled and deposited to build/classes directory.

> bin/hadoop version not working
> ------------------------------
>
>                 Key: HADOOP-6172
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6172
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6172-20090731.patch
>
>
> Two problems found:
> - ${build.src} not included in ant target "compile-core-classes", thus o.a.h.package-info.java is not compiled, which contains the version annotation.
> - bin/hadoop-config.sh attempts to include jar files matching pattern hadoop-*-core.jar rather than hadoop-core-*.jar.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-585-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 08:04:47 2009
Return-Path: <common-issues-return-585-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72489 invoked from network); 31 Jul 2009 08:04:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 08:04:45 -0000
Received: (qmail 35019 invoked by uid 500); 31 Jul 2009 08:04:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34981 invoked by uid 500); 31 Jul 2009 08:04:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34971 invoked by uid 99); 31 Jul 2009 08:04:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 08:04:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 08:04:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E2FCD234C1EB
	for <common-issues@hadoop.apache.org>; Fri, 31 Jul 2009 01:04:14 -0700 (PDT)
Message-ID: <210252632.1249027454928.JavaMail.jira@brutus>
Date: Fri, 31 Jul 2009 01:04:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6173) src/native/packageNativeHadoop.sh
 only packages files with "hadoop" in the name
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name
-------------------------------------------------------------------------------

                 Key: HADOOP-6173
                 URL: https://issues.apache.org/jira/browse/HADOOP-6173
             Project: Hadoop Common
          Issue Type: Bug
          Components: build
    Affects Versions: 0.21.0
            Reporter: Hong Tang
            Priority: Minor


src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name. This becomes too restrictive when a user wants to inject third-party native libraries into his/her own tar build.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-586-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 08:10:43 2009
Return-Path: <common-issues-return-586-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73341 invoked from network); 31 Jul 2009 08:10:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 08:10:43 -0000
Received: (qmail 38397 invoked by uid 500); 31 Jul 2009 08:10:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38351 invoked by uid 500); 31 Jul 2009 08:10:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38341 invoked by uid 99); 31 Jul 2009 08:10:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 08:10:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 08:10:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3A661234C046
	for <common-issues@hadoop.apache.org>; Fri, 31 Jul 2009 01:10:15 -0700 (PDT)
Message-ID: <1084443946.1249027815224.JavaMail.jira@brutus>
Date: Fri, 31 Jul 2009 01:10:15 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6173) src/native/packageNativeHadoop.sh
 only packages files with "hadoop" in the name
In-Reply-To: <210252632.1249027454928.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737457#action_12737457 ] 

Hong Tang commented on HADOOP-6173:
-----------------------------------

The context of the problem is that I want to write a external script to package jars and native libraries from hadoop-gpl-compression to hadoop trunk tarball. The brief procedure is as follows:
- build hadoop-gpl-compression, put jar file under $hadoop-src-dir/lib, and native libraries under $hadoop-src-dir/lib/native.
- run "ant tar" under $hadoop-src-dir.

However, in the tarball, hadoop-gpl-compression-0.1.0-dev.jar is included, but none of the native libraries are included (libgplcompression.*).

Close examination reveals that it is because the script src/native/packageNativeHadoop.sh has the following lines:
Line  45     $TAR *hadoop* | (cd $DIST_LIB_DIR/$platform/; $UNTAR)
Line  61     $TAR *hadoop* | (cd $DIST_LIB_DIR/$platform/; $UNTAR)

> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6173
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6173
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Priority: Minor
>
> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name. This becomes too restrictive when a user wants to inject third-party native libraries into his/her own tar build.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-587-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 08:12:48 2009
Return-Path: <common-issues-return-587-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73720 invoked from network); 31 Jul 2009 08:12:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 08:12:48 -0000
Received: (qmail 41108 invoked by uid 500); 31 Jul 2009 08:12:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41061 invoked by uid 500); 31 Jul 2009 08:12:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41051 invoked by uid 99); 31 Jul 2009 08:12:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 08:12:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 08:12:38 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E8555234C1E6
	for <common-issues@hadoop.apache.org>; Fri, 31 Jul 2009 01:12:17 -0700 (PDT)
Message-ID: <69638855.1249027937950.JavaMail.jira@brutus>
Date: Fri, 31 Jul 2009 01:12:17 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6173) src/native/packageNativeHadoop.sh
 only packages files with "hadoop" in the name
In-Reply-To: <210252632.1249027454928.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-6173:
------------------------------

    Attachment: hadoop-6174-20090731.patch

> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6173
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6173
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Priority: Minor
>         Attachments: hadoop-6174-20090731.patch
>
>
> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name. This becomes too restrictive when a user wants to inject third-party native libraries into his/her own tar build.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-588-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 08:12:48 2009
Return-Path: <common-issues-return-588-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73748 invoked from network); 31 Jul 2009 08:12:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 08:12:48 -0000
Received: (qmail 41176 invoked by uid 500); 31 Jul 2009 08:12:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41130 invoked by uid 500); 31 Jul 2009 08:12:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41120 invoked by uid 99); 31 Jul 2009 08:12:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 08:12:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 08:12:39 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 24918234C1F1
	for <common-issues@hadoop.apache.org>; Fri, 31 Jul 2009 01:12:18 -0700 (PDT)
Message-ID: <899938108.1249027938148.JavaMail.jira@brutus>
Date: Fri, 31 Jul 2009 01:12:18 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6173) src/native/packageNativeHadoop.sh
 only packages files with "hadoop" in the name
In-Reply-To: <210252632.1249027454928.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-6173:
------------------------------

    Status: Patch Available  (was: Open)

Trivial patch that relaxes the file name restriction.

> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6173
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6173
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Priority: Minor
>         Attachments: hadoop-6174-20090731.patch
>
>
> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name. This becomes too restrictive when a user wants to inject third-party native libraries into his/her own tar build.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-589-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 08:20:45 2009
Return-Path: <common-issues-return-589-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75087 invoked from network); 31 Jul 2009 08:20:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 08:20:44 -0000
Received: (qmail 46546 invoked by uid 500); 31 Jul 2009 08:20:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46500 invoked by uid 500); 31 Jul 2009 08:20:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46490 invoked by uid 99); 31 Jul 2009 08:20:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 08:20:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 08:20:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 05C4A234C046
	for <common-issues@hadoop.apache.org>; Fri, 31 Jul 2009 01:20:15 -0700 (PDT)
Message-ID: <1700961911.1249028415019.JavaMail.jira@brutus>
Date: Fri, 31 Jul 2009 01:20:15 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6173) src/native/packageNativeHadoop.sh
 only packages files with "hadoop" in the name
In-Reply-To: <210252632.1249027454928.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang reassigned HADOOP-6173:
---------------------------------

    Assignee: Hong Tang

> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6173
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6173
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>         Attachments: hadoop-6174-20090731.patch
>
>
> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name. This becomes too restrictive when a user wants to inject third-party native libraries into his/her own tar build.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-590-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 17:42:43 2009
Return-Path: <common-issues-return-590-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63147 invoked from network); 31 Jul 2009 17:42:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 17:42:43 -0000
Received: (qmail 18582 invoked by uid 500); 31 Jul 2009 17:42:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18542 invoked by uid 500); 31 Jul 2009 17:42:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18532 invoked by uid 99); 31 Jul 2009 17:42:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 17:42:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 17:42:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C8012234C046
	for <common-issues@hadoop.apache.org>; Fri, 31 Jul 2009 10:42:14 -0700 (PDT)
Message-ID: <1242812014.1249062134805.JavaMail.jira@brutus>
Date: Fri, 31 Jul 2009 10:42:14 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6173) src/native/packageNativeHadoop.sh
 only packages files with "hadoop" in the name
In-Reply-To: <210252632.1249027454928.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-6173:
---------------------------------

    Issue Type: New Feature  (was: Bug)

This seems to me more like a new feature request than a bug: existing code is designed to package what's to be included in a release and no more.  You'd like to extend the package task to support inclusion of third party binaries.  If we intend to maintain support for such a feature, should we add a test for it?

Alternately you might be able to layer your own Ant build file that imports Hadoop's build.xml has a task that depends on the "package" target that then copies extra things into the dist directory, then a task that depends on that task and the "tar" task to bundle everything up.


> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6173
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6173
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>         Attachments: hadoop-6174-20090731.patch
>
>
> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name. This becomes too restrictive when a user wants to inject third-party native libraries into his/her own tar build.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-591-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 18:16:43 2009
Return-Path: <common-issues-return-591-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81715 invoked from network); 31 Jul 2009 18:16:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 18:16:43 -0000
Received: (qmail 71295 invoked by uid 500); 31 Jul 2009 18:16:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71255 invoked by uid 500); 31 Jul 2009 18:16:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71245 invoked by uid 99); 31 Jul 2009 18:16:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 18:16:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 18:16:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EF95A234C1F2
	for <common-issues@hadoop.apache.org>; Fri, 31 Jul 2009 11:16:14 -0700 (PDT)
Message-ID: <1793638136.1249064174980.JavaMail.jira@brutus>
Date: Fri, 31 Jul 2009 11:16:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6172) bin/hadoop version not working
In-Reply-To: <816666106.1249025655012.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737631#action_12737631 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6172:
------------------------------------------------

+1 patch looks good.

> bin/hadoop version not working
> ------------------------------
>
>                 Key: HADOOP-6172
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6172
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6172-20090731.patch
>
>
> Two problems found:
> - ${build.src} not included in ant target "compile-core-classes", thus o.a.h.package-info.java is not compiled, which contains the version annotation.
> - bin/hadoop-config.sh attempts to include jar files matching pattern hadoop-*-core.jar rather than hadoop-core-*.jar.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-592-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 18:18:47 2009
Return-Path: <common-issues-return-592-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89754 invoked from network); 31 Jul 2009 18:18:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 18:18:46 -0000
Received: (qmail 73528 invoked by uid 500); 31 Jul 2009 18:18:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73479 invoked by uid 500); 31 Jul 2009 18:18:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73469 invoked by uid 99); 31 Jul 2009 18:18:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 18:18:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 18:18:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3717F234C495
	for <common-issues@hadoop.apache.org>; Fri, 31 Jul 2009 11:18:15 -0700 (PDT)
Message-ID: <283776342.1249064295224.JavaMail.jira@brutus>
Date: Fri, 31 Jul 2009 11:18:15 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6172) bin/hadoop version not working
In-Reply-To: <816666106.1249025655012.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737633#action_12737633 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6172:
------------------------------------------------

Since this is a script changes, test-patch and unit tests are not related.  I am going to commit this without submitting it to hudson.

> bin/hadoop version not working
> ------------------------------
>
>                 Key: HADOOP-6172
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6172
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6172-20090731.patch
>
>
> Two problems found:
> - ${build.src} not included in ant target "compile-core-classes", thus o.a.h.package-info.java is not compiled, which contains the version annotation.
> - bin/hadoop-config.sh attempts to include jar files matching pattern hadoop-*-core.jar rather than hadoop-core-*.jar.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-593-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 18:26:45 2009
Return-Path: <common-issues-return-593-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93810 invoked from network); 31 Jul 2009 18:26:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 18:26:45 -0000
Received: (qmail 81092 invoked by uid 500); 31 Jul 2009 18:26:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81061 invoked by uid 500); 31 Jul 2009 18:26:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81051 invoked by uid 99); 31 Jul 2009 18:26:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 18:26:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 18:26:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id F1762234C48C
	for <common-issues@hadoop.apache.org>; Fri, 31 Jul 2009 11:26:14 -0700 (PDT)
Message-ID: <1308284166.1249064774988.JavaMail.jira@brutus>
Date: Fri, 31 Jul 2009 11:26:14 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6172) bin/hadoop version not working
In-Reply-To: <816666106.1249025655012.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737641#action_12737641 ] 

Konstantin Boudnik commented on HADOOP-6172:
--------------------------------------------

Shouldn't we have a primitive unit test to check bin/hadoop version result? I believe this is a very basic sanity test too.

> bin/hadoop version not working
> ------------------------------
>
>                 Key: HADOOP-6172
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6172
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6172-20090731.patch
>
>
> Two problems found:
> - ${build.src} not included in ant target "compile-core-classes", thus o.a.h.package-info.java is not compiled, which contains the version annotation.
> - bin/hadoop-config.sh attempts to include jar files matching pattern hadoop-*-core.jar rather than hadoop-core-*.jar.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-594-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 18:34:43 2009
Return-Path: <common-issues-return-594-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98209 invoked from network); 31 Jul 2009 18:34:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 18:34:43 -0000
Received: (qmail 89402 invoked by uid 500); 31 Jul 2009 18:34:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89354 invoked by uid 500); 31 Jul 2009 18:34:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89344 invoked by uid 99); 31 Jul 2009 18:34:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 18:34:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 18:34:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 061E2234C04C
	for <common-issues@hadoop.apache.org>; Fri, 31 Jul 2009 11:34:15 -0700 (PDT)
Message-ID: <854931886.1249065255002.JavaMail.jira@brutus>
Date: Fri, 31 Jul 2009 11:34:15 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6172) bin/hadoop version not working
In-Reply-To: <816666106.1249025655012.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737648#action_12737648 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6172:
------------------------------------------------

> Shouldn't we have a primitive unit test to check bin/hadoop version result? ..

I think this is a good idea but we do not have any unit test to test any script (correct me if I am wrong).  Not sure why we are not creating unit tests for scripts.  I guess we usually do manual tests for scripts.  Are you proposing to add unit tests for scripts?

Holding on the commit.

> bin/hadoop version not working
> ------------------------------
>
>                 Key: HADOOP-6172
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6172
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6172-20090731.patch
>
>
> Two problems found:
> - ${build.src} not included in ant target "compile-core-classes", thus o.a.h.package-info.java is not compiled, which contains the version annotation.
> - bin/hadoop-config.sh attempts to include jar files matching pattern hadoop-*-core.jar rather than hadoop-core-*.jar.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-595-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 18:36:42 2009
Return-Path: <common-issues-return-595-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98528 invoked from network); 31 Jul 2009 18:36:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 18:36:42 -0000
Received: (qmail 93240 invoked by uid 500); 31 Jul 2009 18:36:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93212 invoked by uid 500); 31 Jul 2009 18:36:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93197 invoked by uid 99); 31 Jul 2009 18:36:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 18:36:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 18:36:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E02A4234C1ED
	for <common-issues@hadoop.apache.org>; Fri, 31 Jul 2009 11:36:14 -0700 (PDT)
Message-ID: <818249399.1249065374917.JavaMail.jira@brutus>
Date: Fri, 31 Jul 2009 11:36:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6173) src/native/packageNativeHadoop.sh
 only packages files with "hadoop" in the name
In-Reply-To: <210252632.1249027454928.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737650#action_12737650 ] 

Hong Tang commented on HADOOP-6173:
-----------------------------------

Just to clarify, the current build.xml behavior seems to be incorrect/inconsistent with what you described. If I put in a 3rd party jar in lib/, it will be copied to release tar, and if I put a 3rd party native library under lib/native with "hadoop" in their names (like libhadoopgplcompression.so), they will also be included.

So I would argue that it is reasonable to include all jars and native libraries under lib/ to tar ball for two reasons:
- it is a user's conscience decision to copy data under lib/ and thus the inclusion of these files is "by-choice".
- the hadoop script currently includes all jars under lib/ in classpath, and all native libraries under lib/native/<arch>/ in sysproperty java.library.path. And it is reasonable for user to expect that if he/she runs an "ant tar" and untar the tarball somewhere else, it should behave exactly the same as the original place.

I am fine to add a test to verify the behavior. How about just running md5sum over the set of files under lib and under package final destination?

> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6173
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6173
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>         Attachments: hadoop-6174-20090731.patch
>
>
> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name. This becomes too restrictive when a user wants to inject third-party native libraries into his/her own tar build.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-596-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 19:03:45 2009
Return-Path: <common-issues-return-596-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4133 invoked from network); 31 Jul 2009 19:03:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 19:03:45 -0000
Received: (qmail 19531 invoked by uid 500); 31 Jul 2009 19:03:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19485 invoked by uid 500); 31 Jul 2009 19:03:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19475 invoked by uid 99); 31 Jul 2009 19:03:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 19:03:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 19:03:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C6087234C046
	for <common-issues@hadoop.apache.org>; Fri, 31 Jul 2009 12:03:14 -0700 (PDT)
Message-ID: <1590882067.1249066994797.JavaMail.jira@brutus>
Date: Fri, 31 Jul 2009 12:03:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6172) bin/hadoop version not working
In-Reply-To: <816666106.1249025655012.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737660#action_12737660 ] 

Hong Tang commented on HADOOP-6172:
-----------------------------------

Such test could change the dependency, because it is testing whether the tar target runs correctly. Or we probably could incorporate it as part of the "test-patch" target (because that would build a tar in the process).

> bin/hadoop version not working
> ------------------------------
>
>                 Key: HADOOP-6172
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6172
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6172-20090731.patch
>
>
> Two problems found:
> - ${build.src} not included in ant target "compile-core-classes", thus o.a.h.package-info.java is not compiled, which contains the version annotation.
> - bin/hadoop-config.sh attempts to include jar files matching pattern hadoop-*-core.jar rather than hadoop-core-*.jar.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-597-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 19:26:45 2009
Return-Path: <common-issues-return-597-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11433 invoked from network); 31 Jul 2009 19:26:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 19:26:45 -0000
Received: (qmail 58686 invoked by uid 500); 31 Jul 2009 19:26:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58640 invoked by uid 500); 31 Jul 2009 19:26:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58630 invoked by uid 99); 31 Jul 2009 19:26:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 19:26:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 19:26:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C9FCD234C046
	for <common-issues@hadoop.apache.org>; Fri, 31 Jul 2009 12:26:14 -0700 (PDT)
Message-ID: <2022332602.1249068374812.JavaMail.jira@brutus>
Date: Fri, 31 Jul 2009 12:26:14 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6172) bin/hadoop version not working
In-Reply-To: <816666106.1249025655012.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737675#action_12737675 ] 

Konstantin Boudnik commented on HADOOP-6172:
--------------------------------------------

I'm not sure why the dependency would be changed - it seems that scripts are coming directly from SVN and nothing special is required to run
{noformat}
bin/hadoop version
{noformat}
but compilation. Thus certain script tests won't require anything special too. Am I mistaken?

Also, I think whatever tests might be automated with a relatively low effort should be automated. Manual tests aren't that efficient nor error proof. E.g. in the current common trunk bin/hadoop version produces a lot of 'unknown' version tags which might've been caught by a routinely executed test. 

Shall we start adding tests for scripts right now? Why not - this day isn't worst than any consequent one :-)

> bin/hadoop version not working
> ------------------------------
>
>                 Key: HADOOP-6172
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6172
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6172-20090731.patch
>
>
> Two problems found:
> - ${build.src} not included in ant target "compile-core-classes", thus o.a.h.package-info.java is not compiled, which contains the version annotation.
> - bin/hadoop-config.sh attempts to include jar files matching pattern hadoop-*-core.jar rather than hadoop-core-*.jar.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-598-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 20:19:45 2009
Return-Path: <common-issues-return-598-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25091 invoked from network); 31 Jul 2009 20:19:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 20:19:45 -0000
Received: (qmail 11002 invoked by uid 500); 31 Jul 2009 20:19:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10953 invoked by uid 500); 31 Jul 2009 20:19:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10943 invoked by uid 99); 31 Jul 2009 20:19:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 20:19:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 20:19:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C8D3D234C046
	for <common-issues@hadoop.apache.org>; Fri, 31 Jul 2009 13:19:14 -0700 (PDT)
Message-ID: <423779046.1249071554807.JavaMail.jira@brutus>
Date: Fri, 31 Jul 2009 13:19:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6172) bin/hadoop version not working
In-Reply-To: <816666106.1249025655012.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737692#action_12737692 ] 

Hong Tang commented on HADOOP-6172:
-----------------------------------

@Cos, in this particular case, I want to conduct two tests:
- $hadoop-src-dir/bin/hadoop version
- $hadoop-src-dir/build/hadoop-xxx/bin/hadoop version

The second test requires hadoop being copied to build/bin, which would not happen until "ant tar" is called. So the second test would depend on "tar" target.

> bin/hadoop version not working
> ------------------------------
>
>                 Key: HADOOP-6172
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6172
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6172-20090731.patch
>
>
> Two problems found:
> - ${build.src} not included in ant target "compile-core-classes", thus o.a.h.package-info.java is not compiled, which contains the version annotation.
> - bin/hadoop-config.sh attempts to include jar files matching pattern hadoop-*-core.jar rather than hadoop-core-*.jar.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-599-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 20:45:43 2009
Return-Path: <common-issues-return-599-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28834 invoked from network); 31 Jul 2009 20:45:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 20:45:43 -0000
Received: (qmail 31770 invoked by uid 500); 31 Jul 2009 20:45:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31726 invoked by uid 500); 31 Jul 2009 20:45:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31716 invoked by uid 99); 31 Jul 2009 20:45:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 20:45:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 20:45:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D26E9234C04C
	for <common-issues@hadoop.apache.org>; Fri, 31 Jul 2009 13:45:14 -0700 (PDT)
Message-ID: <1201808291.1249073114861.JavaMail.jira@brutus>
Date: Fri, 31 Jul 2009 13:45:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6172) bin/hadoop version not working
In-Reply-To: <816666106.1249025655012.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737701#action_12737701 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6172:
------------------------------------------------

> Also, I think whatever tests might be automated with a relatively low effort should be automated. ...

Since our build process does not support script unit tests, I think it is not a relatively low effort to add a test for this issue.  The fixes in this issue are simple but important.  Even we decide to create unit tests for script, how about do it in a separated issue, so that we can commit this first?

> bin/hadoop version not working
> ------------------------------
>
>                 Key: HADOOP-6172
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6172
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6172-20090731.patch
>
>
> Two problems found:
> - ${build.src} not included in ant target "compile-core-classes", thus o.a.h.package-info.java is not compiled, which contains the version annotation.
> - bin/hadoop-config.sh attempts to include jar files matching pattern hadoop-*-core.jar rather than hadoop-core-*.jar.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-600-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 21:12:38 2009
Return-Path: <common-issues-return-600-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35916 invoked from network); 31 Jul 2009 21:12:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 21:12:38 -0000
Received: (qmail 55382 invoked by uid 500); 31 Jul 2009 21:12:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55353 invoked by uid 500); 31 Jul 2009 21:12:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55326 invoked by uid 99); 31 Jul 2009 21:12:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 21:12:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 21:12:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D033A234C04C
	for <common-issues@hadoop.apache.org>; Fri, 31 Jul 2009 14:12:14 -0700 (PDT)
Message-ID: <860131482.1249074734852.JavaMail.jira@brutus>
Date: Fri, 31 Jul 2009 14:12:14 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6174) Create tests for scripts to verify
 basic user experience, i.e. version, etc.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Create tests for scripts to verify basic user experience, i.e. version, etc.
----------------------------------------------------------------------------

                 Key: HADOOP-6174
                 URL: https://issues.apache.org/jira/browse/HADOOP-6174
             Project: Hadoop Common
          Issue Type: Test
            Reporter: Konstantin Boudnik


As per HADOOP-6172 we need to have a set of scripts to verify the correctness of some very basic functionality accessible through bin/ scripts. 


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-601-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 21:12:43 2009
Return-Path: <common-issues-return-601-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35990 invoked from network); 31 Jul 2009 21:12:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 21:12:43 -0000
Received: (qmail 55469 invoked by uid 500); 31 Jul 2009 21:12:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55424 invoked by uid 500); 31 Jul 2009 21:12:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55414 invoked by uid 99); 31 Jul 2009 21:12:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 21:12:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 21:12:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EBECC234C1F2
	for <common-issues@hadoop.apache.org>; Fri, 31 Jul 2009 14:12:14 -0700 (PDT)
Message-ID: <1893808199.1249074734965.JavaMail.jira@brutus>
Date: Fri, 31 Jul 2009 14:12:14 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6172) bin/hadoop version not working
In-Reply-To: <816666106.1249025655012.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737711#action_12737711 ] 

Konstantin Boudnik commented on HADOOP-6172:
--------------------------------------------

Fine with me. Let's commit this first. To track the scripts testing I've created HADOOP-6172

> bin/hadoop version not working
> ------------------------------
>
>                 Key: HADOOP-6172
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6172
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6172-20090731.patch
>
>
> Two problems found:
> - ${build.src} not included in ant target "compile-core-classes", thus o.a.h.package-info.java is not compiled, which contains the version annotation.
> - bin/hadoop-config.sh attempts to include jar files matching pattern hadoop-*-core.jar rather than hadoop-core-*.jar.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-602-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 21:24:36 2009
Return-Path: <common-issues-return-602-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38782 invoked from network); 31 Jul 2009 21:24:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 21:24:36 -0000
Received: (qmail 69091 invoked by uid 500); 31 Jul 2009 21:24:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69045 invoked by uid 500); 31 Jul 2009 21:24:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69035 invoked by uid 99); 31 Jul 2009 21:24:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 21:24:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 21:24:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C5288234C046
	for <common-issues@hadoop.apache.org>; Fri, 31 Jul 2009 14:24:14 -0700 (PDT)
Message-ID: <1940824441.1249075454793.JavaMail.jira@brutus>
Date: Fri, 31 Jul 2009 14:24:14 -0700 (PDT)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6174) Create tests for scripts to verify
 basic user experience, i.e. version, etc.
In-Reply-To: <860131482.1249074734852.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737717#action_12737717 ] 

Nigel Daley commented on HADOOP-6174:
-------------------------------------

Hopefully TestCLI can be enhanced for these.

> Create tests for scripts to verify basic user experience, i.e. version, etc.
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-6174
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6174
>             Project: Hadoop Common
>          Issue Type: Test
>            Reporter: Konstantin Boudnik
>
> As per HADOOP-6172 we need to have a set of scripts to verify the correctness of some very basic functionality accessible through bin/ scripts. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-603-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 21:40:45 2009
Return-Path: <common-issues-return-603-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47153 invoked from network); 31 Jul 2009 21:40:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 21:40:44 -0000
Received: (qmail 81150 invoked by uid 500); 31 Jul 2009 21:40:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81105 invoked by uid 500); 31 Jul 2009 21:40:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81095 invoked by uid 99); 31 Jul 2009 21:40:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 21:40:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 21:40:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C61F6234C046
	for <common-issues@hadoop.apache.org>; Fri, 31 Jul 2009 14:40:14 -0700 (PDT)
Message-ID: <1539486865.1249076414798.JavaMail.jira@brutus>
Date: Fri, 31 Jul 2009 14:40:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6172) bin/hadoop version not working
In-Reply-To: <816666106.1249025655012.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6172:
-------------------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

Tested manually as shown below:
{noformat}
bash-3.2$ ./bin/hadoop version
Hadoop Unknown
Subversion Unknown -r Unknown
Compiled by Unknown on Unknown
>From source with checksum Unknown

# apply patch and recompile
bash-3.2$ patch -p0 -i ../hadoop-6172-20090731.patch 
...

bash-3.2$ ./bin/hadoop version
Hadoop 0.21.0-dev
Subversion https://svn.apache.org/repos/asf/hadoop/common/trunk -r 798247
Compiled by tsz on Fri Jul 31 14:40:51 PDT 2009
>From source with checksum 4cec386d7ec492be92e14fc12a5a50a4
{noformat}

I have committed this.  Thanks, Hong!

> bin/hadoop version not working
> ------------------------------
>
>                 Key: HADOOP-6172
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6172
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6172-20090731.patch
>
>
> Two problems found:
> - ${build.src} not included in ant target "compile-core-classes", thus o.a.h.package-info.java is not compiled, which contains the version annotation.
> - bin/hadoop-config.sh attempts to include jar files matching pattern hadoop-*-core.jar rather than hadoop-core-*.jar.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-604-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 21:58:43 2009
Return-Path: <common-issues-return-604-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53844 invoked from network); 31 Jul 2009 21:58:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 21:58:43 -0000
Received: (qmail 92581 invoked by uid 500); 31 Jul 2009 21:58:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92534 invoked by uid 500); 31 Jul 2009 21:58:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92524 invoked by uid 99); 31 Jul 2009 21:58:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 21:58:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 21:58:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DFBDD234C1EF
	for <common-issues@hadoop.apache.org>; Fri, 31 Jul 2009 14:58:14 -0700 (PDT)
Message-ID: <1978873028.1249077494915.JavaMail.jira@brutus>
Date: Fri, 31 Jul 2009 14:58:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6173) src/native/packageNativeHadoop.sh
 only packages files with "hadoop" in the name
In-Reply-To: <210252632.1249027454928.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737731#action_12737731 ] 

Hong Tang commented on HADOOP-6173:
-----------------------------------

BTW, the attached patch file name is misleading (I mistyped).

> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6173
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6173
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>         Attachments: hadoop-6174-20090731.patch
>
>
> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name. This becomes too restrictive when a user wants to inject third-party native libraries into his/her own tar build.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-605-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 31 21:58:44 2009
Return-Path: <common-issues-return-605-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53882 invoked from network); 31 Jul 2009 21:58:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Jul 2009 21:58:43 -0000
Received: (qmail 92657 invoked by uid 500); 31 Jul 2009 21:58:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92604 invoked by uid 500); 31 Jul 2009 21:58:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92583 invoked by uid 99); 31 Jul 2009 21:58:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 21:58:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2009 21:58:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DA0A5234C04C
	for <common-issues@hadoop.apache.org>; Fri, 31 Jul 2009 14:58:14 -0700 (PDT)
Message-ID: <1631238047.1249077494892.JavaMail.jira@brutus>
Date: Fri, 31 Jul 2009 14:58:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6173) src/native/packageNativeHadoop.sh
 only packages files with "hadoop" in the name
In-Reply-To: <210252632.1249027454928.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737730#action_12737730 ] 

Hong Tang commented on HADOOP-6173:
-----------------------------------

Looks like we need some basic support to test the scripts, particularly because the verification must be done after running "tar" or "package". Similar situation happens for HADOOP-6172. And Cos created a new issue HADOOP-6174 to track the unit tests for scripts. Could we resolve this issue for now?

> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6173
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6173
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>         Attachments: hadoop-6174-20090731.patch
>
>
> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name. This becomes too restrictive when a user wants to inject third-party native libraries into his/her own tar build.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-15651-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 00:25:28 2011
Return-Path: <common-issues-return-15651-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8097B4049
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 00:25:28 +0000 (UTC)
Received: (qmail 64221 invoked by uid 500); 1 Jun 2011 00:25:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64101 invoked by uid 500); 1 Jun 2011 00:25:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64093 invoked by uid 99); 1 Jun 2011 00:25:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 00:25:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 00:25:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5C2DDEC80C
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 00:24:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 00:24:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1570148867.57902.1306887887374.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7345) listStatus for local files throws
 NPE instead of permission denied
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

listStatus for local files throws NPE instead of permission denied
------------------------------------------------------------------

                 Key: HADOOP-7345
                 URL: https://issues.apache.org/jira/browse/HADOOP-7345
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
    Affects Versions: 0.23.0
            Reporter: Daryn Sharp


Calling {{fs.listStatus}} on a local directory where the user does not have permissions generates a {{NullPointerException}}:

{noformat}
Exception in thread "main" java.lang.NullPointerException
	at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1115)
	at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1150)
	at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:494)
{noformat}

FileSystem.java:
{code}
1111:  private void listStatus(ArrayList<FileStatus> results, Path f,
1112:      PathFilter filter) throws FileNotFoundException, IOException {
1113:    FileStatus listing[] = listStatus(f);
1114:
1115:    for (int i = 0; i < listing.length; i++) {
{code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15652-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 00:27:29 2011
Return-Path: <common-issues-return-15652-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 58841409A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 00:27:29 +0000 (UTC)
Received: (qmail 67102 invoked by uid 500); 1 Jun 2011 00:27:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67072 invoked by uid 500); 1 Jun 2011 00:27:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67063 invoked by uid 99); 1 Jun 2011 00:27:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 00:27:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 00:27:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0B4AEEC891
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 00:26:48 +0000 (UTC)
Date: Wed, 1 Jun 2011 00:26:48 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1136514016.57917.1306888008042.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041907#comment-13041907 ] 

Luke Lu commented on HADOOP-7144:
---------------------------------

LOG.error is always called before handleThrowable. It looked reasonable (it's not swallowing all throwables and log the errors) to me at the time, as we should never let jetty handle the exceptions/errors as stack traces would be display as part of the result, which is a security no no.

OTOH, I think the logging should be handled by handleThrowable calls for better readability and DRYness.

Looking closely though, I found follow correctness issues:

# content type should be set to "application/json; charset=utf8"
# need to response.setStatus(response.SC_INTERNAL_SERVER_ERROR) for all error cases


> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Robert Joseph Evans
>              Labels: jmx
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7411-0.20.20X-V1.patch, HADOOP-7411-trunk-V1.patch, HADOOP-7411-trunk-alpha.patch, jmx.json
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15653-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 00:41:31 2011
Return-Path: <common-issues-return-15653-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4C0DE4963
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 00:41:31 +0000 (UTC)
Received: (qmail 80926 invoked by uid 500); 1 Jun 2011 00:41:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80726 invoked by uid 500); 1 Jun 2011 00:41:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80718 invoked by uid 99); 1 Jun 2011 00:41:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 00:41:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 00:41:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9883FECDF3
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 00:40:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 00:40:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <328650881.57984.1306888847620.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <3255382.248681296161204958.JavaMail.jira@thor>
Subject: [jira] [Updated] (HADOOP-7121) Exceptions while serializing IPC
 call response are not handled well
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7121:
--------------------------------

    Attachment: hadoop-7121.txt

Attached patch adds coverage to all parts of the IPC lifecycle:
- client side write param
- server side read param
- server side write response
- client side read response

In the cases that the error occurs server side, it sends back the exception to the client as well as logging it.


> Exceptions while serializing IPC call response are not handled well
> -------------------------------------------------------------------
>
>                 Key: HADOOP-7121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7121
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>              Labels: newbie
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7121.txt
>
>
> We had a situation where for some reason the serialization of an RPC call's response was throwing OOME. When this happens, the exception is not caught, and the call never gets a response - the client just hangs. Additionally, the OOME propagated all the way to the top of the IPC handler and caused the handler. Plus, the Handler upon exit only logged to stdout and not to the log4j logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15654-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 00:43:31 2011
Return-Path: <common-issues-return-15654-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3F10449B2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 00:43:31 +0000 (UTC)
Received: (qmail 83828 invoked by uid 500); 1 Jun 2011 00:43:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83792 invoked by uid 500); 1 Jun 2011 00:43:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83784 invoked by uid 99); 1 Jun 2011 00:43:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 00:43:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 00:43:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A5561ECEF3
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 00:42:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 00:42:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <562956220.57997.1306888967674.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <3255382.248681296161204958.JavaMail.jira@thor>
Subject: [jira] [Updated] (HADOOP-7121) Exceptions while serializing IPC
 call response are not handled well
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7121:
--------------------------------

    Status: Patch Available  (was: Open)

> Exceptions while serializing IPC call response are not handled well
> -------------------------------------------------------------------
>
>                 Key: HADOOP-7121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7121
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>              Labels: newbie
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7121.txt
>
>
> We had a situation where for some reason the serialization of an RPC call's response was throwing OOME. When this happens, the exception is not caught, and the call never gets a response - the client just hangs. Additionally, the OOME propagated all the way to the top of the IPC handler and caused the handler. Plus, the Handler upon exit only logged to stdout and not to the log4j logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15655-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 00:55:30 2011
Return-Path: <common-issues-return-15655-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AD4F140BC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 00:55:30 +0000 (UTC)
Received: (qmail 93627 invoked by uid 500); 1 Jun 2011 00:55:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93494 invoked by uid 500); 1 Jun 2011 00:55:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93486 invoked by uid 99); 1 Jun 2011 00:55:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 00:55:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 00:55:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5B2ABEC241
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 00:54:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 00:54:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1028508876.58011.1306889687369.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <3255382.248681296161204958.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7121) Exceptions while serializing IPC
 call response are not handled well
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041919#comment-13041919 ] 

Hadoop QA commented on HADOOP-7121:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481021/hadoop-7121.txt
  against trunk revision 1129905.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/549//console

This message is automatically generated.

> Exceptions while serializing IPC call response are not handled well
> -------------------------------------------------------------------
>
>                 Key: HADOOP-7121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7121
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>              Labels: newbie
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7121.txt
>
>
> We had a situation where for some reason the serialization of an RPC call's response was throwing OOME. When this happens, the exception is not caught, and the call never gets a response - the client just hangs. Additionally, the OOME propagated all the way to the top of the IPC handler and caused the handler. Plus, the Handler upon exit only logged to stdout and not to the log4j logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15656-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 01:02:44 2011
Return-Path: <common-issues-return-15656-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 56C104FB3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 01:02:44 +0000 (UTC)
Received: (qmail 2310 invoked by uid 500); 1 Jun 2011 01:02:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2260 invoked by uid 500); 1 Jun 2011 01:02:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2168 invoked by uid 99); 1 Jun 2011 01:02:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 01:02:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 01:02:37 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 11508EC3DE
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 01:01:56 +0000 (UTC)
Date: Wed, 1 Jun 2011 01:01:56 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <647927887.58055.1306890116067.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7346) Send back nicer error to clients
 using outdated IPC version
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Send back nicer error to clients using outdated IPC version
-----------------------------------------------------------

                 Key: HADOOP-7346
                 URL: https://issues.apache.org/jira/browse/HADOOP-7346
             Project: Hadoop Common
          Issue Type: Improvement
          Components: ipc
    Affects Versions: 0.22.0
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
             Fix For: 0.22.0


When an older Hadoop version tries to contact a newer Hadoop version across an IPC protocol version bump, the client currently just gets a non-useful error message like "EOFException".

Instead, the IPC server code can speak just enough of prior IPC protocols to send back a "fatal" message indicating the version mismatch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15657-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 01:16:32 2011
Return-Path: <common-issues-return-15657-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E59064E37
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 01:16:30 +0000 (UTC)
Received: (qmail 10659 invoked by uid 500); 1 Jun 2011 01:16:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10582 invoked by uid 500); 1 Jun 2011 01:16:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10570 invoked by uid 99); 1 Jun 2011 01:16:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 01:16:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 01:16:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5D17CEC70B
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 01:15:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 01:15:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <454687198.58064.1306890947377.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <3255382.248681296161204958.JavaMail.jira@thor>
Subject: [jira] [Updated] (HADOOP-7121) Exceptions while serializing IPC
 call response are not handled well
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7121:
--------------------------------

    Attachment: hadoop-7121.txt

Updated patch against trunk

> Exceptions while serializing IPC call response are not handled well
> -------------------------------------------------------------------
>
>                 Key: HADOOP-7121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7121
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>              Labels: newbie
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7121.txt, hadoop-7121.txt
>
>
> We had a situation where for some reason the serialization of an RPC call's response was throwing OOME. When this happens, the exception is not caught, and the call never gets a response - the client just hangs. Additionally, the OOME propagated all the way to the top of the IPC handler and caused the handler. Plus, the Handler upon exit only logged to stdout and not to the log4j logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15658-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 01:20:28 2011
Return-Path: <common-issues-return-15658-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B20174086
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 01:20:28 +0000 (UTC)
Received: (qmail 13297 invoked by uid 500); 1 Jun 2011 01:20:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13243 invoked by uid 500); 1 Jun 2011 01:20:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13235 invoked by uid 99); 1 Jun 2011 01:20:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 01:20:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 01:20:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6A6F1EC81E
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 01:19:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 01:19:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1585729560.58072.1306891187432.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <647927887.58055.1306890116067.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7346) Send back nicer error to clients
 using outdated IPC version
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7346:
--------------------------------

    Attachment: hadoop-7346.txt

I manually tested this patch using an 0.18.3 client, 0.20.0 client, and CDH3 client (which has the same IPC stack as 0.20.20x series)

I also looped an old client for a few minutes hitting the new server, then took a jstack on the new server, to make sure there weren't any thread leaks.

Unfortunately it's not easy to automatically test it since we can't load old versions of the IPC clients within the context of a unit test.

> Send back nicer error to clients using outdated IPC version
> -----------------------------------------------------------
>
>                 Key: HADOOP-7346
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7346
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7346.txt
>
>
> When an older Hadoop version tries to contact a newer Hadoop version across an IPC protocol version bump, the client currently just gets a non-useful error message like "EOFException".
> Instead, the IPC server code can speak just enough of prior IPC protocols to send back a "fatal" message indicating the version mismatch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15659-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 01:20:31 2011
Return-Path: <common-issues-return-15659-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DAC114094
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 01:20:30 +0000 (UTC)
Received: (qmail 13525 invoked by uid 500); 1 Jun 2011 01:20:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13496 invoked by uid 500); 1 Jun 2011 01:20:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13488 invoked by uid 99); 1 Jun 2011 01:20:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 01:20:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 01:20:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 78537EC820
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 01:19:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 01:19:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1966313814.58074.1306891187489.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <647927887.58055.1306890116067.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7346) Send back nicer error to clients
 using outdated IPC version
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7346:
--------------------------------

    Status: Patch Available  (was: Open)

> Send back nicer error to clients using outdated IPC version
> -----------------------------------------------------------
>
>                 Key: HADOOP-7346
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7346
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7346.txt
>
>
> When an older Hadoop version tries to contact a newer Hadoop version across an IPC protocol version bump, the client currently just gets a non-useful error message like "EOFException".
> Instead, the IPC server code can speak just enough of prior IPC protocols to send back a "fatal" message indicating the version mismatch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15660-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 01:26:29 2011
Return-Path: <common-issues-return-15660-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E440E4130
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 01:26:28 +0000 (UTC)
Received: (qmail 17164 invoked by uid 500); 1 Jun 2011 01:26:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17085 invoked by uid 500); 1 Jun 2011 01:26:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17077 invoked by uid 99); 1 Jun 2011 01:26:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 01:26:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 01:26:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 830FBEC94D
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 01:25:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 01:25:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1667378050.58082.1306891547533.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1992675736.52742.1306733207390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7339) Introduce a buffered checksum for
 avoiding frequently calls on Checksum.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7339:
--------------------------------

    Fix Version/s:     (was: 0.22.0)
                   0.23.0

bumping to 0.23 since this is an optimization

> Introduce a buffered checksum for avoiding frequently calls on Checksum.update()
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-7339
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7339
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Min Zhou
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7339-v1.diff, HADOOP-7339-v2.diff
>
>
> We found that PureJavaCRC32/CRC32.update() is the TOP 1 of the methods consuming CPU in a map side, and in reduce side, it cost a lots of CPU too.
> IFileOutputStream would frequently call Checksum.update() during writing a record. It's very common a MR key/value less than 512 bytes. Checksum.update() would be called every time writing a key/value. 
> Test case: terasort 100MB. 
> Checksum.update() calls has be reduced from 4030348 to 28069.  This method is not a hotspot anymore.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15661-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 01:29:35 2011
Return-Path: <common-issues-return-15661-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 935334193
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 01:29:35 +0000 (UTC)
Received: (qmail 21011 invoked by uid 500); 1 Jun 2011 01:29:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20981 invoked by uid 500); 1 Jun 2011 01:29:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20973 invoked by uid 99); 1 Jun 2011 01:29:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 01:29:35 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 01:29:33 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 557FDECA20
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 01:28:52 +0000 (UTC)
Date: Wed, 1 Jun 2011 01:28:52 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <359309548.58088.1306891732346.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <531829213.32234.1305934307559.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7316) Add public javadocs to
 FSDataInputStream and FSDataOutputStream
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041928#comment-13041928 ] 

Eli Collins commented on HADOOP-7316:
-------------------------------------

Hey Owen - is the latest patch acceptable to you?

> Add public javadocs to FSDataInputStream and FSDataOutputStream
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7316
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7316
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Jonathan Hsieh
>            Assignee: Eli Collins
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7316-1.patch, hadoop-7316-2.patch
>
>
> This is a method made public for testing.  In comments in HADOOP-7301 after commit, adding javadoc comments was requested.  This is a follow up jira to address it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15662-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 01:35:28 2011
Return-Path: <common-issues-return-15662-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EBC5F47DA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 01:35:28 +0000 (UTC)
Received: (qmail 23679 invoked by uid 500); 1 Jun 2011 01:35:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23638 invoked by uid 500); 1 Jun 2011 01:35:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23629 invoked by uid 99); 1 Jun 2011 01:35:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 01:35:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 01:35:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 99319ECB99
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 01:34:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 01:34:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1764487900.58097.1306892087624.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <647927887.58055.1306890116067.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7346) Send back nicer error to clients
 using outdated IPC version
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041929#comment-13041929 ] 

Hadoop QA commented on HADOOP-7346:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481025/hadoop-7346.txt
  against trunk revision 1129905.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/550//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/550//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/550//console

This message is automatically generated.

> Send back nicer error to clients using outdated IPC version
> -----------------------------------------------------------
>
>                 Key: HADOOP-7346
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7346
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7346.txt
>
>
> When an older Hadoop version tries to contact a newer Hadoop version across an IPC protocol version bump, the client currently just gets a non-useful error message like "EOFException".
> Instead, the IPC server code can speak just enough of prior IPC protocols to send back a "fatal" message indicating the version mismatch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15663-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 01:35:29 2011
Return-Path: <common-issues-return-15663-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3FB7747E9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 01:35:29 +0000 (UTC)
Received: (qmail 23925 invoked by uid 500); 1 Jun 2011 01:35:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23877 invoked by uid 500); 1 Jun 2011 01:35:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23723 invoked by uid 99); 1 Jun 2011 01:35:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 01:35:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 01:35:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AD072ECB9B
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 01:34:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 01:34:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1227029855.58099.1306892087705.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <3255382.248681296161204958.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7121) Exceptions while serializing IPC
 call response are not handled well
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041930#comment-13041930 ] 

Hadoop QA commented on HADOOP-7121:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481024/hadoop-7121.txt
  against trunk revision 1129905.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/551//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/551//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/551//console

This message is automatically generated.

> Exceptions while serializing IPC call response are not handled well
> -------------------------------------------------------------------
>
>                 Key: HADOOP-7121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7121
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>              Labels: newbie
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7121.txt, hadoop-7121.txt
>
>
> We had a situation where for some reason the serialization of an RPC call's response was throwing OOME. When this happens, the exception is not caught, and the call never gets a response - the client just hangs. Additionally, the OOME propagated all the way to the top of the IPC handler and caused the handler. Plus, the Handler upon exit only logged to stdout and not to the log4j logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15664-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 01:49:31 2011
Return-Path: <common-issues-return-15664-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2E6064EC3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 01:49:31 +0000 (UTC)
Received: (qmail 39335 invoked by uid 500); 1 Jun 2011 01:49:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39292 invoked by uid 500); 1 Jun 2011 01:49:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39274 invoked by uid 99); 1 Jun 2011 01:49:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 01:49:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 01:49:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9A34AECF95
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 01:48:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 01:48:47 +0000 (UTC)
From: "Priyo Mustafi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <209080988.58127.1306892927628.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1145070693.28819.1305845387456.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7311) Port remaining metrics v1 from
 trunk to branch-0.20-security
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041933#comment-13041933 ] 

Priyo Mustafi commented on HADOOP-7311:
---------------------------------------

Hi Luke, I need to have Ganglia working with Metrics2.  I guess I need to code something like GangliaSink which writes/adapts to say GangliaContext41. Can you please provide me some hints so that I don't spend lot of time going the wrong way?  Your help much appreciated.  
Thanks
Priyo

> Port remaining metrics v1 from trunk to branch-0.20-security
> ------------------------------------------------------------
>
>                 Key: HADOOP-7311
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7311
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.203.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: 0.20.204.0
>
>         Attachments: metrics-0.20-security.patch
>
>
> HADOOP-7190 added metrics packages/classes. This is a port from trunk to make them actually work for the security branch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15665-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 01:53:29 2011
Return-Path: <common-issues-return-15665-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 74AD54F45
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 01:53:29 +0000 (UTC)
Received: (qmail 44537 invoked by uid 500); 1 Jun 2011 01:53:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44509 invoked by uid 500); 1 Jun 2011 01:53:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44501 invoked by uid 99); 1 Jun 2011 01:53:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 01:53:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 01:53:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1393AEC1E9
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 01:52:48 +0000 (UTC)
Date: Wed, 1 Jun 2011 01:52:48 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <863808477.58162.1306893168076.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <3255382.248681296161204958.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7121) Exceptions while serializing IPC
 call response are not handled well
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041938#comment-13041938 ] 

Eli Collins commented on HADOOP-7121:
-------------------------------------

+1  lgtm

Nit: lines 1364, 1366, 1612 don't need to wrap

> Exceptions while serializing IPC call response are not handled well
> -------------------------------------------------------------------
>
>                 Key: HADOOP-7121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7121
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>              Labels: newbie
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7121.txt, hadoop-7121.txt
>
>
> We had a situation where for some reason the serialization of an RPC call's response was throwing OOME. When this happens, the exception is not caught, and the call never gets a response - the client just hangs. Additionally, the OOME propagated all the way to the top of the IPC handler and caused the handler. Plus, the Handler upon exit only logged to stdout and not to the log4j logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15666-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 02:01:31 2011
Return-Path: <common-issues-return-15666-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C67506150
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 02:01:31 +0000 (UTC)
Received: (qmail 50731 invoked by uid 500); 1 Jun 2011 02:01:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50506 invoked by uid 500); 1 Jun 2011 02:01:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50498 invoked by uid 99); 1 Jun 2011 02:01:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 02:01:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 02:01:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B53D1EC3EE
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 02:00:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 02:00:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <660790573.58180.1306893647738.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <3255382.248681296161204958.JavaMail.jira@thor>
Subject: [jira] [Updated] (HADOOP-7121) Exceptions while serializing IPC
 call response are not handled well
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7121:
--------------------------------

    Attachment: hadoop-7121.txt

nits fixed. Since it was just a whitespace change from previous patch I will commit this version based on Eli's +1.

> Exceptions while serializing IPC call response are not handled well
> -------------------------------------------------------------------
>
>                 Key: HADOOP-7121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7121
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>              Labels: newbie
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7121.txt, hadoop-7121.txt, hadoop-7121.txt
>
>
> We had a situation where for some reason the serialization of an RPC call's response was throwing OOME. When this happens, the exception is not caught, and the call never gets a response - the client just hangs. Additionally, the OOME propagated all the way to the top of the IPC handler and caused the handler. Plus, the Handler upon exit only logged to stdout and not to the log4j logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15667-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 02:03:31 2011
Return-Path: <common-issues-return-15667-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B40B16AB4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 02:03:31 +0000 (UTC)
Received: (qmail 51779 invoked by uid 500); 1 Jun 2011 02:03:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51726 invoked by uid 500); 1 Jun 2011 02:03:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51628 invoked by uid 99); 1 Jun 2011 02:03:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 02:03:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 02:03:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F3FCAEC524
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 02:02:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 02:02:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <825097249.58201.1306893767996.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <3255382.248681296161204958.JavaMail.jira@thor>
Subject: [jira] [Updated] (HADOOP-7121) Exceptions while serializing IPC
 call response are not handled well
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7121:
--------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

Committed to trunk and 22. Thanks for reviewing, Eli

> Exceptions while serializing IPC call response are not handled well
> -------------------------------------------------------------------
>
>                 Key: HADOOP-7121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7121
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>              Labels: newbie
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7121.txt, hadoop-7121.txt, hadoop-7121.txt
>
>
> We had a situation where for some reason the serialization of an RPC call's response was throwing OOME. When this happens, the exception is not caught, and the call never gets a response - the client just hangs. Additionally, the OOME propagated all the way to the top of the IPC handler and caused the handler. Plus, the Handler upon exit only logged to stdout and not to the log4j logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15668-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 02:05:29 2011
Return-Path: <common-issues-return-15668-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 005B46EC0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 02:05:29 +0000 (UTC)
Received: (qmail 54257 invoked by uid 500); 1 Jun 2011 02:05:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54180 invoked by uid 500); 1 Jun 2011 02:05:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54171 invoked by uid 99); 1 Jun 2011 02:05:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 02:05:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 02:05:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8D54AEC5EC
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 02:04:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 02:04:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <545810429.58213.1306893887575.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <3255382.248681296161204958.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7121) Exceptions while serializing IPC
 call response are not handled well
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041942#comment-13041942 ] 

Hadoop QA commented on HADOOP-7121:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481031/hadoop-7121.txt
  against trunk revision 1129982.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/552//console

This message is automatically generated.

> Exceptions while serializing IPC call response are not handled well
> -------------------------------------------------------------------
>
>                 Key: HADOOP-7121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7121
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>              Labels: newbie
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7121.txt, hadoop-7121.txt, hadoop-7121.txt
>
>
> We had a situation where for some reason the serialization of an RPC call's response was throwing OOME. When this happens, the exception is not caught, and the call never gets a response - the client just hangs. Additionally, the OOME propagated all the way to the top of the IPC handler and caused the handler. Plus, the Handler upon exit only logged to stdout and not to the log4j logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15669-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 02:05:29 2011
Return-Path: <common-issues-return-15669-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2C7AB6ED2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 02:05:29 +0000 (UTC)
Received: (qmail 54438 invoked by uid 500); 1 Jun 2011 02:05:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54385 invoked by uid 500); 1 Jun 2011 02:05:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54178 invoked by uid 99); 1 Jun 2011 02:05:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 02:05:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 02:05:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9D66FEC5EE
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 02:04:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 02:04:47 +0000 (UTC)
From: "XieXianshan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1237235509.58215.1306893887641.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <53044194.55753.1306837788451.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7340) incomplete help message is
 displayed for getmerge [addnl] option
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041943#comment-13041943 ] 

XieXianshan commented on HADOOP-7340:
-------------------------------------

Thanks for the comments,i`ll update the patch as soon as possible.
And meanwhile, i`ll submit another patch to "make it a flag like -addnl".

> incomplete help message is displayed for getmerge [addnl] option
> ----------------------------------------------------------------
>
>                 Key: HADOOP-7340
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7340
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Assignee: XieXianshan
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7340
>
>
> The help message for the command "hdfs dfs -help getmerge" is displayed like this:
> "-getmerge <src> <localdst> [addnl]:	Get all the files in the directories that
> 		match the source file pattern and merge and sort them to only
> 		one file on local fs. <src> is kept."
> and the information about [addnl] option is missed,despite the fact that [addnl] option is implemented.
> Therefore,the expected message should be displayed like this:
> "-getmerge <src> <localdst> [addnl]:	Get all the files in the directories that
> 		match the source file pattern and merge and sort them to only
> 		one file on local fs. <src> is kept.
> 		  addnl   Optionally addnl can be set to enable adding a newline 
> 		          character at the end of each file."

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15670-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 02:09:31 2011
Return-Path: <common-issues-return-15670-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9CDAB6F3B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 02:09:31 +0000 (UTC)
Received: (qmail 58840 invoked by uid 500); 1 Jun 2011 02:09:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58801 invoked by uid 500); 1 Jun 2011 02:09:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58786 invoked by uid 99); 1 Jun 2011 02:09:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 02:09:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 02:09:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 01471EC73A
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 02:08:48 +0000 (UTC)
Date: Wed, 1 Jun 2011 02:08:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1369625164.58220.1306894128002.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <314496369.49888.1306530467371.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7335) Force entropy to come from non-true
 random for tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7335:
--------------------------------

    Attachment: hadoop-7335.txt

Here's a patch to pass that flag to tests.

I'd appreciate if someone can test this on Windows/OSX to make sure it doesn't break their platforms.

> Force entropy to come from non-true random for tests
> ----------------------------------------------------
>
>                 Key: HADOOP-7335
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7335
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7335.txt
>
>
> Passing the system property {{-Djava.security.egd=file:///dev/urandom}} forces the JVM to seed its PRNG from non-true random (/dev/urandom) instead of the true random (/dev/random). This makes the tests run faster, since without it they often hang waiting for entropy while Jetty is initializing.
> We should turn this on for the test targets by default, so developers/hudson boxes don't have to make this change system-wide or use workarounds like rngtools.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15671-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 02:09:31 2011
Return-Path: <common-issues-return-15671-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C67746F4A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 02:09:31 +0000 (UTC)
Received: (qmail 59005 invoked by uid 500); 1 Jun 2011 02:09:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58804 invoked by uid 500); 1 Jun 2011 02:09:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58787 invoked by uid 99); 1 Jun 2011 02:09:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 02:09:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 02:09:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0E7E5EC73C
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 02:08:48 +0000 (UTC)
Date: Wed, 1 Jun 2011 02:08:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1412574628.58222.1306894128056.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <314496369.49888.1306530467371.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7335) Force entropy to come from non-true
 random for tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7335:
--------------------------------

    Status: Patch Available  (was: Open)

> Force entropy to come from non-true random for tests
> ----------------------------------------------------
>
>                 Key: HADOOP-7335
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7335
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7335.txt
>
>
> Passing the system property {{-Djava.security.egd=file:///dev/urandom}} forces the JVM to seed its PRNG from non-true random (/dev/urandom) instead of the true random (/dev/random). This makes the tests run faster, since without it they often hang waiting for entropy while Jetty is initializing.
> We should turn this on for the test targets by default, so developers/hudson boxes don't have to make this change system-wide or use workarounds like rngtools.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15672-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 02:15:31 2011
Return-Path: <common-issues-return-15672-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 210E5612D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 02:15:31 +0000 (UTC)
Received: (qmail 61506 invoked by uid 500); 1 Jun 2011 02:15:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61422 invoked by uid 500); 1 Jun 2011 02:15:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61409 invoked by uid 99); 1 Jun 2011 02:15:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 02:15:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 02:15:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 73ACAEC8AC
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 02:14:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 02:14:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1298467416.58228.1306894487470.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <3255382.248681296161204958.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7121) Exceptions while serializing IPC
 call response are not handled well
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041946#comment-13041946 ] 

Hudson commented on HADOOP-7121:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #631 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/631/])
    HADOOP-7121. Exceptions while serializing IPC call responses are not handled well. Contributed by Todd Lipcon.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1129982
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/ipc/Server.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/ipc/TestIPC.java


> Exceptions while serializing IPC call response are not handled well
> -------------------------------------------------------------------
>
>                 Key: HADOOP-7121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7121
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>              Labels: newbie
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7121.txt, hadoop-7121.txt, hadoop-7121.txt
>
>
> We had a situation where for some reason the serialization of an RPC call's response was throwing OOME. When this happens, the exception is not caught, and the call never gets a response - the client just hangs. Additionally, the OOME propagated all the way to the top of the IPC handler and caused the handler. Plus, the Handler upon exit only logged to stdout and not to the log4j logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15673-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 02:25:28 2011
Return-Path: <common-issues-return-15673-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AEC776547
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 02:25:28 +0000 (UTC)
Received: (qmail 70683 invoked by uid 500); 1 Jun 2011 02:25:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70605 invoked by uid 500); 1 Jun 2011 02:25:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70597 invoked by uid 99); 1 Jun 2011 02:25:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 02:25:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 02:25:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 59871ECB2C
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 02:24:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 02:24:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1385417673.58247.1306895087363.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <314496369.49888.1306530467371.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7335) Force entropy to come from
 non-true random for tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041949#comment-13041949 ] 

Hadoop QA commented on HADOOP-7335:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481032/hadoop-7335.txt
  against trunk revision 1129982.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/553//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/553//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/553//console

This message is automatically generated.

> Force entropy to come from non-true random for tests
> ----------------------------------------------------
>
>                 Key: HADOOP-7335
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7335
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7335.txt
>
>
> Passing the system property {{-Djava.security.egd=file:///dev/urandom}} forces the JVM to seed its PRNG from non-true random (/dev/urandom) instead of the true random (/dev/random). This makes the tests run faster, since without it they often hang waiting for entropy while Jetty is initializing.
> We should turn this on for the test targets by default, so developers/hudson boxes don't have to make this change system-wide or use workarounds like rngtools.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15674-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 02:31:31 2011
Return-Path: <common-issues-return-15674-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 534EB6CD8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 02:31:31 +0000 (UTC)
Received: (qmail 75448 invoked by uid 500); 1 Jun 2011 02:31:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75333 invoked by uid 500); 1 Jun 2011 02:31:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75311 invoked by uid 99); 1 Jun 2011 02:31:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 02:31:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 02:31:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 58592ECCB8
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 02:30:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 02:30:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <578625602.58257.1306895447358.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1145070693.28819.1305845387456.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7311) Port remaining metrics v1 from
 trunk to branch-0.20-security
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041951#comment-13041951 ] 

Luke Lu commented on HADOOP-7311:
---------------------------------

Hi Priyo, let's discuss the design of the Ganglia plugin on HADOOP-7324. Since I'm not familiar with Ganglia internals, I have some questions to ask. Can you answer them there?



> Port remaining metrics v1 from trunk to branch-0.20-security
> ------------------------------------------------------------
>
>                 Key: HADOOP-7311
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7311
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.203.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: 0.20.204.0
>
>         Attachments: metrics-0.20-security.patch
>
>
> HADOOP-7190 added metrics packages/classes. This is a port from trunk to make them actually work for the security branch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15675-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 03:12:32 2011
Return-Path: <common-issues-return-15675-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CA58269A7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 03:12:32 +0000 (UTC)
Received: (qmail 13601 invoked by uid 500); 1 Jun 2011 03:12:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13571 invoked by uid 500); 1 Jun 2011 03:12:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13563 invoked by uid 99); 1 Jun 2011 03:12:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 03:12:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 03:12:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5EA2AEC69A
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 03:11:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 03:11:47 +0000 (UTC)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <51879504.58292.1306897907384.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <314496369.49888.1306530467371.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7335) Force entropy to come from
 non-true random for tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041958#comment-13041958 ] 

Konstantin Boudnik commented on HADOOP-7335:
--------------------------------------------

Looks good, except that according to [JAAS reference guide|http://download.oracle.com/javase/1.4.2/docs/guide/security/jaas/JAASRefGuide.html] it suppose to have a single '/' in the file URL. Which  doesn't look right to me ;)

> Force entropy to come from non-true random for tests
> ----------------------------------------------------
>
>                 Key: HADOOP-7335
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7335
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7335.txt
>
>
> Passing the system property {{-Djava.security.egd=file:///dev/urandom}} forces the JVM to seed its PRNG from non-true random (/dev/urandom) instead of the true random (/dev/random). This makes the tests run faster, since without it they often hang waiting for entropy while Jetty is initializing.
> We should turn this on for the test targets by default, so developers/hudson boxes don't have to make this change system-wide or use workarounds like rngtools.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15676-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 03:30:31 2011
Return-Path: <common-issues-return-15676-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 66A0D63DE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 03:30:31 +0000 (UTC)
Received: (qmail 26929 invoked by uid 500); 1 Jun 2011 03:30:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26689 invoked by uid 500); 1 Jun 2011 03:30:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26669 invoked by uid 99); 1 Jun 2011 03:30:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 03:30:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 03:30:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1C224ECB12
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 03:29:48 +0000 (UTC)
Date: Wed, 1 Jun 2011 03:29:48 +0000 (UTC)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <822987426.58328.1306898988111.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <647927887.58055.1306890116067.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7346) Send back nicer error to clients
 using outdated IPC version
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041963#comment-13041963 ] 

Konstantin Boudnik commented on HADOOP-7346:
--------------------------------------------

+1 patch looks good.

Do you think it is possible to test this new behavior with perhaps mocks?

> Send back nicer error to clients using outdated IPC version
> -----------------------------------------------------------
>
>                 Key: HADOOP-7346
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7346
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7346.txt
>
>
> When an older Hadoop version tries to contact a newer Hadoop version across an IPC protocol version bump, the client currently just gets a non-useful error message like "EOFException".
> Instead, the IPC server code can speak just enough of prior IPC protocols to send back a "fatal" message indicating the version mismatch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15677-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 03:40:30 2011
Return-Path: <common-issues-return-15677-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E95046644
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 03:40:29 +0000 (UTC)
Received: (qmail 30833 invoked by uid 500); 1 Jun 2011 03:40:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30719 invoked by uid 500); 1 Jun 2011 03:40:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30688 invoked by uid 99); 1 Jun 2011 03:40:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 03:40:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 03:40:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7EC03ECED4
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 03:39:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 03:39:47 +0000 (UTC)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <965088684.58368.1306899587515.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <397648942.57126.1306869588398.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7342) Add an utility API in FileUtil for
 JDK File.list
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041967#comment-13041967 ] 

Konstantin Boudnik commented on HADOOP-7342:
--------------------------------------------

Coupla comments Bharath:
- this sounds like you are trying to use exceptions to replace return value checks. Now you'd have to check IOException around all the calls to the new API. Besides, you're kinda violating original {{File.list()}} contract which doesn't declare IOException

Also, asserts in the tests are better to have messages to ease detection of test failures


> Add an utility API in FileUtil for JDK File.list
> ------------------------------------------------
>
>                 Key: HADOOP-7342
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7342
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7342-1.patch
>
>
> Java File.list API can return null when disk is bad or directory is not a directory. This utility API in FileUtil will throw an exception when this happens rather than returning null. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15678-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 03:40:30 2011
Return-Path: <common-issues-return-15678-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B9108664F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 03:40:30 +0000 (UTC)
Received: (qmail 31035 invoked by uid 500); 1 Jun 2011 03:40:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31005 invoked by uid 500); 1 Jun 2011 03:40:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30992 invoked by uid 99); 1 Jun 2011 03:40:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 03:40:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 03:40:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 64781ECED1
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 03:39:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 03:39:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1285409152.58365.1306899587408.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <557825167.38099.1306198007453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7324) Ganglia plugins for metrics v2
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041966#comment-13041966 ] 

Luke Lu commented on HADOOP-7324:
---------------------------------

My limited knowledge with gmond wire format is from a quick scan of the GangliaContext\*.java source.

I have a question that I hope Ganglia experts can answer: the wire format seems _sparse_, in that it contains full key value pairs. I wonder if the update can be sparse as well, i.e., the metrics that haven't changed since the last snapshot doesn't need to be sent to gmond as well. The metrics2 system allows sparse update to reduce object/bandwidth usage.

If the update can be sparse, it quite straight forward to adapt FileSink and GangliaContext code. OTOH, If only dense update is allowed, {{o.a.h.metrics2.util.MetricsCache}} can be used.

> Ganglia plugins for metrics v2
> ------------------------------
>
>                 Key: HADOOP-7324
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7324
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: metrics
>    Affects Versions: 0.20.203.0, 0.23.0
>            Reporter: Luke Lu
>             Fix For: 0.20.204.0, 0.23.0
>
>
> Although, all metrics in metrics v2 are exposed via the standard JMX mechanisms, there are a few users who prefer using Ganglia to collect metrics.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15679-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 03:45:32 2011
Return-Path: <common-issues-return-15679-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 54FB96A1A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 03:45:32 +0000 (UTC)
Received: (qmail 33683 invoked by uid 500); 1 Jun 2011 03:45:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33575 invoked by uid 500); 1 Jun 2011 03:45:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33526 invoked by uid 99); 1 Jun 2011 03:45:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 03:45:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 03:45:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 85B15ED110
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 03:44:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 03:44:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1747017124.58392.1306899887544.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041971#comment-13041971 ] 

Hudson commented on HADOOP-7284:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #632 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/632/])
    HADOOP-7284 Trash and shell's rm does not work for viewfs

sradia : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1129989
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/viewfs/ViewFs.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/viewfs/ChRootedFileSystem.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/viewfs/InodeTree.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Delete.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/ViewFileSystemTestSetup.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/TestTrash.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/Trash.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/TestChRootedFs.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/TestChRootedFileSystem.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/TestViewFsTrash.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/viewfs/Constants.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/TestFSMainOperationsLocalFileSystem.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/viewfs/ConfigUtil.java


> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash10.patch, trash11.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch, trash8.patch, trash9.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15680-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 04:15:30 2011
Return-Path: <common-issues-return-15680-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9C94763E5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 04:15:30 +0000 (UTC)
Received: (qmail 52958 invoked by uid 500); 1 Jun 2011 04:15:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52739 invoked by uid 500); 1 Jun 2011 04:15:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52720 invoked by uid 99); 1 Jun 2011 04:15:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 04:15:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 04:15:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6233FED6D0
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 04:14:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 04:14:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <686004280.58440.1306901687398.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7347) HDFS Wire compatibility
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

HDFS Wire compatibility
-----------------------

                 Key: HADOOP-7347
                 URL: https://issues.apache.org/jira/browse/HADOOP-7347
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Sanjay Radia
            Assignee: Sanjay Radia
             Fix For: 0.23.0




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15681-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 04:21:33 2011
Return-Path: <common-issues-return-15681-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F1E166899
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 04:21:32 +0000 (UTC)
Received: (qmail 63513 invoked by uid 500); 1 Jun 2011 04:21:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63388 invoked by uid 500); 1 Jun 2011 04:21:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63366 invoked by uid 99); 1 Jun 2011 04:21:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 04:21:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 04:21:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 86FADED895
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 04:20:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 04:20:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <821279707.58447.1306902047549.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <686004280.58440.1306901687398.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7347) HDFS Wire compatibility
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7347?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7347:
---------------------------------

    Attachment: Wire Compatibility =E2=80=93 Separating wire types.pdf

The attached document proposes a design to achieve wire compatibility.=20
* The design is complementary to using AVRO/PB like serialization technolog=
ies.=20
* Further some of the steps such a separating data types are necessary to u=
se such serialization technologies.
* The design, can be used to provide compatibility from non-avro/pb (say 23=
) to avro/pb release



> HDFS Wire compatibility
> -----------------------
>
>                 Key: HADOOP-7347
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7347
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: Wire Compatibility =E2=80=93 Separating wire types.p=
df
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15682-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 04:23:31 2011
Return-Path: <common-issues-return-15682-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B7EC46903
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 04:23:31 +0000 (UTC)
Received: (qmail 68696 invoked by uid 500); 1 Jun 2011 04:23:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68364 invoked by uid 500); 1 Jun 2011 04:23:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68309 invoked by uid 99); 1 Jun 2011 04:23:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 04:23:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 04:23:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 82176ED939
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 04:22:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 04:22:47 +0000 (UTC)
From: "XieXianshan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1142137633.58455.1306902167529.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7348) Modify the option of FsShell
 getmerge from [addnl] to [-nl] for more comprehensive
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Modify the option of FsShell getmerge from [addnl] to [-nl] for more comprehensive
----------------------------------------------------------------------------------

                 Key: HADOOP-7348
                 URL: https://issues.apache.org/jira/browse/HADOOP-7348
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs
    Affects Versions: 0.23.0
            Reporter: XieXianshan
            Assignee: XieXianshan
             Fix For: 0.23.0


The [addnl] option of FsShell getmerge should be either "true" or "false",but it is very hard to understand by users, especially  who`s never used this option before. 
So,the [addnl] option should be changed to [-nl] for more comprehensive.


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15683-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 04:31:29 2011
Return-Path: <common-issues-return-15683-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EF14363B3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 04:31:28 +0000 (UTC)
Received: (qmail 71173 invoked by uid 500); 1 Jun 2011 04:31:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71048 invoked by uid 500); 1 Jun 2011 04:31:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71040 invoked by uid 99); 1 Jun 2011 04:31:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 04:31:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 04:31:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 584CEEDABA
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 04:30:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 04:30:47 +0000 (UTC)
From: "XieXianshan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1133750328.58463.1306902647357.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1142137633.58455.1306902167529.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7348) Modify the option of FsShell
 getmerge from [addnl] to [-nl] for more comprehensive
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

XieXianshan updated HADOOP-7348:
--------------------------------

    Attachment: HADOOP-7348.patch

> Modify the option of FsShell getmerge from [addnl] to [-nl] for more comprehensive
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7348
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7348
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Assignee: XieXianshan
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7348.patch
>
>
> The [addnl] option of FsShell getmerge should be either "true" or "false",but it is very hard to understand by users, especially  who`s never used this option before. 
> So,the [addnl] option should be changed to [-nl] for more comprehensive.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15684-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 04:31:31 2011
Return-Path: <common-issues-return-15684-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2A59863F0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 04:31:31 +0000 (UTC)
Received: (qmail 71355 invoked by uid 500); 1 Jun 2011 04:31:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71309 invoked by uid 500); 1 Jun 2011 04:31:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71301 invoked by uid 99); 1 Jun 2011 04:31:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 04:31:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 04:31:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 688C2EDABC
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 04:30:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 04:30:47 +0000 (UTC)
From: "XieXianshan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <760772240.58465.1306902647425.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1142137633.58455.1306902167529.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7348) Modify the option of FsShell
 getmerge from [addnl] to [-nl] for more comprehensive
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

XieXianshan updated HADOOP-7348:
--------------------------------

    Status: Patch Available  (was: Open)

fix this issue.

> Modify the option of FsShell getmerge from [addnl] to [-nl] for more comprehensive
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7348
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7348
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Assignee: XieXianshan
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7348.patch
>
>
> The [addnl] option of FsShell getmerge should be either "true" or "false",but it is very hard to understand by users, especially  who`s never used this option before. 
> So,the [addnl] option should be changed to [-nl] for more comprehensive.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15685-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 04:45:39 2011
Return-Path: <common-issues-return-15685-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7772A643E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 04:45:39 +0000 (UTC)
Received: (qmail 76996 invoked by uid 500); 1 Jun 2011 04:45:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76738 invoked by uid 500); 1 Jun 2011 04:45:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76710 invoked by uid 99); 1 Jun 2011 04:45:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 04:45:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 04:45:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A5508EDD96
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 04:44:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 04:44:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <955557719.58471.1306903487674.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1142137633.58455.1306902167529.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7348) Modify the option of FsShell
 getmerge from [addnl] to [-nl] for more comprehensive
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041983#comment-13041983 ] 

Hadoop QA commented on HADOOP-7348:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481040/HADOOP-7348.patch
  against trunk revision 1129989.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.cli.TestCLI

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/554//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/554//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/554//console

This message is automatically generated.

> Modify the option of FsShell getmerge from [addnl] to [-nl] for more comprehensive
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7348
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7348
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Assignee: XieXianshan
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7348.patch
>
>
> The [addnl] option of FsShell getmerge should be either "true" or "false",but it is very hard to understand by users, especially  who`s never used this option before. 
> So,the [addnl] option should be changed to [-nl] for more comprehensive.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15686-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 09:46:32 2011
Return-Path: <common-issues-return-15686-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 14BAB6664
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 09:46:32 +0000 (UTC)
Received: (qmail 8530 invoked by uid 500); 1 Jun 2011 09:46:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7905 invoked by uid 500); 1 Jun 2011 09:46:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7831 invoked by uid 99); 1 Jun 2011 09:46:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 09:46:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 09:46:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9EAD2ED122
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 09:45:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 09:45:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <439313255.59042.1306921547646.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <3255382.248681296161204958.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7121) Exceptions while serializing IPC
 call response are not handled well
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042068#comment-13042068 ] 

Hudson commented on HADOOP-7121:
--------------------------------

Integrated in Hadoop-Common-22-branch #59 (See [https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/59/])
    HADOOP-7121. Exceptions while serializing IPC call responses are not handled well. Contributed by Todd Lipcon.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1129983
Files : 
* /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/ipc/Server.java
* /hadoop/common/branches/branch-0.22/src/test/core/org/apache/hadoop/ipc/TestIPC.java
* /hadoop/common/branches/branch-0.22/CHANGES.txt


> Exceptions while serializing IPC call response are not handled well
> -------------------------------------------------------------------
>
>                 Key: HADOOP-7121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7121
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>              Labels: newbie
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7121.txt, hadoop-7121.txt, hadoop-7121.txt
>
>
> We had a situation where for some reason the serialization of an RPC call's response was throwing OOME. When this happens, the exception is not caught, and the call never gets a response - the client just hangs. Additionally, the OOME propagated all the way to the top of the IPC handler and caused the handler. Plus, the Handler upon exit only logged to stdout and not to the log4j logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15687-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 11:16:29 2011
Return-Path: <common-issues-return-15687-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 349246D93
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 11:16:29 +0000 (UTC)
Received: (qmail 32120 invoked by uid 500); 1 Jun 2011 11:16:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32067 invoked by uid 500); 1 Jun 2011 11:16:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32058 invoked by uid 99); 1 Jun 2011 11:16:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 11:16:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 11:16:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C7DD7ED7A6
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 11:15:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 11:15:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1072563231.59247.1306926947815.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <3255382.248681296161204958.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7121) Exceptions while serializing IPC
 call response are not handled well
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042112#comment-13042112 ] 

Hudson commented on HADOOP-7121:
--------------------------------

Integrated in Hadoop-Common-trunk #706 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/706/])
    HADOOP-7121. Exceptions while serializing IPC call responses are not handled well. Contributed by Todd Lipcon.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1129982
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/ipc/Server.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/ipc/TestIPC.java


> Exceptions while serializing IPC call response are not handled well
> -------------------------------------------------------------------
>
>                 Key: HADOOP-7121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7121
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>              Labels: newbie
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7121.txt, hadoop-7121.txt, hadoop-7121.txt
>
>
> We had a situation where for some reason the serialization of an RPC call's response was throwing OOME. When this happens, the exception is not caught, and the call never gets a response - the client just hangs. Additionally, the OOME propagated all the way to the top of the IPC handler and caused the handler. Plus, the Handler upon exit only logged to stdout and not to the log4j logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15688-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 11:16:31 2011
Return-Path: <common-issues-return-15688-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 438D86DA9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 11:16:31 +0000 (UTC)
Received: (qmail 32392 invoked by uid 500); 1 Jun 2011 11:16:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32354 invoked by uid 500); 1 Jun 2011 11:16:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32345 invoked by uid 99); 1 Jun 2011 11:16:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 11:16:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 11:16:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 81CDAED79D
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 11:15:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 11:15:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1273325975.59239.1306926947527.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1195662636.6130.1300891805766.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7208) equals() and hashCode()
 implementation need to change in StandardSocketFactory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042109#comment-13042109 ] 

Hudson commented on HADOOP-7208:
--------------------------------

Integrated in Hadoop-Common-trunk #706 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/706/])
    HADOOP-7208. Fix implementation of equals() and hashCode() in StandardSocketFactory. Contributed by Uma Maheswara Rao G.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1129840
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/ipc/TestSocketFactory.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/net/StandardSocketFactory.java


> equals() and hashCode() implementation need to change in StandardSocketFactory
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7208
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7208
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7208-2.patch, HADOOP-7208-3.patch, HADOOP-7208.patch
>
>
>   In Hadoop IPC Client, we are using ClientCache which will maintain the HashMap to keep the Client references.
> private Map<SocketFactory, Client> clients =
>       new HashMap<SocketFactory, Client>();
>  Now let us say, we want use two standard factories with Hadoop. MyStandardSocketFactory (which extends StandardSocketFactory), and StandardSocketFactory. In this case, because of equals and hashcode implementation, MyStandardSocketFactory client can be overridden by StandardSocketFactoryClient

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15689-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 11:16:31 2011
Return-Path: <common-issues-return-15689-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 77F926DB9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 11:16:31 +0000 (UTC)
Received: (qmail 32630 invoked by uid 500); 1 Jun 2011 11:16:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32595 invoked by uid 500); 1 Jun 2011 11:16:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32502 invoked by uid 99); 1 Jun 2011 11:16:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 11:16:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 11:16:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B0F33ED7A3
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 11:15:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 11:15:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <55634536.59244.1306926947721.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042110#comment-13042110 ] 

Hudson commented on HADOOP-7284:
--------------------------------

Integrated in Hadoop-Common-trunk #706 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/706/])
    HADOOP-7284 Trash and shell's rm does not work for viewfs

sradia : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1129989
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/viewfs/ViewFs.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/viewfs/ChRootedFileSystem.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/viewfs/InodeTree.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Delete.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/ViewFileSystemTestSetup.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/TestTrash.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/Trash.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/TestChRootedFs.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/TestChRootedFileSystem.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/TestViewFsTrash.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/viewfs/Constants.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/TestFSMainOperationsLocalFileSystem.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/viewfs/ConfigUtil.java


> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash10.patch, trash11.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch, trash8.patch, trash9.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15690-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 11:16:32 2011
Return-Path: <common-issues-return-15690-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DC4DF6DC9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 11:16:31 +0000 (UTC)
Received: (qmail 32889 invoked by uid 500); 1 Jun 2011 11:16:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32860 invoked by uid 500); 1 Jun 2011 11:16:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32852 invoked by uid 99); 1 Jun 2011 11:16:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 11:16:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 11:16:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C01ABED7A5
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 11:15:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 11:15:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1739056643.59246.1306926947783.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1435375533.50176.1306536707416.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7336) TestFileContextResolveAfs will
 fail with default test.build.data property.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042111#comment-13042111 ] 

Hudson commented on HADOOP-7336:
--------------------------------

Integrated in Hadoop-Common-trunk #706 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/706/])
    HADOOP-7336. TestFileContextResolveAfs will fail with default test.build.data property.

jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1129905
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/TestFileContextResolveAfs.java


> TestFileContextResolveAfs will fail with default test.build.data property.
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7336
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7336
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7336.2.patch
>
>
> In TestFileContextResolveAfs if test.build.data property is not set and default is used, the test case will try to create that in the root directory and that will fail. /tmp should be used as default as in many other test cases. Normally, test.build.data will be set and this issue should not occur.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15691-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 15:21:31 2011
Return-Path: <common-issues-return-15691-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 38B6F6950
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 15:21:31 +0000 (UTC)
Received: (qmail 18329 invoked by uid 500); 1 Jun 2011 15:21:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18279 invoked by uid 500); 1 Jun 2011 15:21:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18271 invoked by uid 99); 1 Jun 2011 15:21:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 15:21:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 15:21:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AF54DEE5FB
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 15:20:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 15:20:47 +0000 (UTC)
From: "Robert Joseph Evans (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <652717924.59909.1306941647715.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Joseph Evans updated HADOOP-7144:
----------------------------------------

    Status: Open  (was: Patch Available)

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Robert Joseph Evans
>              Labels: jmx
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7411-0.20.20X-V1.patch, HADOOP-7411-trunk-V1.patch, HADOOP-7411-trunk-alpha.patch, jmx.json
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15692-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 15:23:28 2011
Return-Path: <common-issues-return-15692-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CDCFC6C20
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 15:23:28 +0000 (UTC)
Received: (qmail 24225 invoked by uid 500); 1 Jun 2011 15:23:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24190 invoked by uid 500); 1 Jun 2011 15:23:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24182 invoked by uid 99); 1 Jun 2011 15:23:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 15:23:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 15:23:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8DE9EEE6E5
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 15:22:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 15:22:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <599137111.59916.1306941767578.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1142137633.58455.1306902167529.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7348) Modify the option of FsShell
 getmerge from [addnl] to [-nl] for more comprehensive
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042227#comment-13042227 ] 

Daryn Sharp commented on HADOOP-7348:
-------------------------------------

I see you beat me to removing my TODO, nice!  I'd suggest using a tertiary for assignment, but that's just me, so up to you:
{code}delimiter = cf.getOpt("nl") ? "\n" : null;{code}

Please update the DESCRIPTION to include the {{-nl}} option, update the text in TestCLI, and update the hdfs tests (link the jira to this one).  The applicable tests will be in TestDFSShell and/or TestHDFSCLI.


> Modify the option of FsShell getmerge from [addnl] to [-nl] for more comprehensive
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7348
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7348
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Assignee: XieXianshan
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7348.patch
>
>
> The [addnl] option of FsShell getmerge should be either "true" or "false",but it is very hard to understand by users, especially  who`s never used this option before. 
> So,the [addnl] option should be changed to [-nl] for more comprehensive.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15693-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 15:25:29 2011
Return-Path: <common-issues-return-15693-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CC9AD6F4C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 15:25:29 +0000 (UTC)
Received: (qmail 33984 invoked by uid 500); 1 Jun 2011 15:25:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33940 invoked by uid 500); 1 Jun 2011 15:25:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33878 invoked by uid 99); 1 Jun 2011 15:25:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 15:25:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 15:25:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4AE63EE7C9
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 15:24:48 +0000 (UTC)
Date: Wed, 1 Jun 2011 15:24:48 +0000 (UTC)
From: "Robert Joseph Evans (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1490617000.59934.1306941888303.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Joseph Evans updated HADOOP-7144:
----------------------------------------

    Status: Patch Available  (was: Open)

The following are the results of test-patch on the security branch.

     [exec] -1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 4 new or modified tests.
     [exec] 
     [exec]     -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     -1 Eclipse classpath. The patch causes the Eclipse classpath to differ from the contents of the lib directories.


The eclipse path is bogus, and so is the findbugs.  I don't know why they keep complaining but they do.

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Robert Joseph Evans
>              Labels: jmx
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7411-0.20.20X-V1.patch, HADOOP-7411-0.20.20X-V2.patch, HADOOP-7411-trunk-V1.patch, HADOOP-7411-trunk-V2.patch, HADOOP-7411-trunk-alpha.patch, jmx.json
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15694-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 15:25:31 2011
Return-Path: <common-issues-return-15694-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2247C6F5A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 15:25:31 +0000 (UTC)
Received: (qmail 34321 invoked by uid 500); 1 Jun 2011 15:25:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34287 invoked by uid 500); 1 Jun 2011 15:25:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34279 invoked by uid 99); 1 Jun 2011 15:25:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 15:25:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 15:25:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id ACB59EE7B7
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 15:24:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 15:24:47 +0000 (UTC)
From: "Robert Joseph Evans (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1499847784.59920.1306941887704.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Joseph Evans updated HADOOP-7144:
----------------------------------------

    Attachment: HADOOP-7411-trunk-V2.patch
                HADOOP-7411-0.20.20X-V2.patch

I believe that I have addressed all of the comments.  The code no longer catches a Throwable, but enumerates the errors that can occur at each location, and handles them appropriately.  IOExceptions are passed up to doGet which will return an INTERNAL SERVER ERROR.  If the query is malformed then a BAD REQUEST is returned.  In no case is the json being used to return the error. 

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Robert Joseph Evans
>              Labels: jmx
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7411-0.20.20X-V1.patch, HADOOP-7411-0.20.20X-V2.patch, HADOOP-7411-trunk-V1.patch, HADOOP-7411-trunk-V2.patch, HADOOP-7411-trunk-alpha.patch, jmx.json
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15695-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 17:33:28 2011
Return-Path: <common-issues-return-15695-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C0158629F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 17:33:28 +0000 (UTC)
Received: (qmail 47477 invoked by uid 500); 1 Jun 2011 17:33:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47424 invoked by uid 500); 1 Jun 2011 17:33:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47415 invoked by uid 99); 1 Jun 2011 17:33:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 17:33:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 17:33:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6A01EEE652
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 17:32:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 17:32:47 +0000 (UTC)
From: "Bharath Mundlapudi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1992011655.60357.1306949567427.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <397648942.57126.1306869588398.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7342) Add an utility API in FileUtil for
 JDK File.list
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042309#comment-13042309 ] 

Bharath Mundlapudi commented on HADOOP-7342:
--------------------------------------------

Hi Konstantin,

Thanks for your comments and for reviewing the code. 

My comments:
If you see most of the code in HDFS does have IOException handling around most of the methods. Only the problem is NPE are not handled. We have seen in our tests, NPE can have bad impact on certain cases when disks go bad. 

IMO, this is the problem with JDK API. Instead of asking user to have these null checks everywhere for these calls, API should have been consistent with the return values.

Regarding asserts, I have not put the messages for assertEquals only because this call will assert saying what is expected and actual values should be if it fails. This will be clear from the logs that assertEquals failed. 






> Add an utility API in FileUtil for JDK File.list
> ------------------------------------------------
>
>                 Key: HADOOP-7342
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7342
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7342-1.patch
>
>
> Java File.list API can return null when disk is bad or directory is not a directory. This utility API in FileUtil will throw an exception when this happens rather than returning null. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15696-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 17:35:31 2011
Return-Path: <common-issues-return-15696-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 56C126788
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 17:35:31 +0000 (UTC)
Received: (qmail 49245 invoked by uid 500); 1 Jun 2011 17:35:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49160 invoked by uid 500); 1 Jun 2011 17:35:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49151 invoked by uid 99); 1 Jun 2011 17:35:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 17:35:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 17:35:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A1886EE6E9
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 17:34:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 17:34:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <90700817.60370.1306949687657.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042311#comment-13042311 ] 

Hadoop QA commented on HADOOP-7144:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481100/HADOOP-7411-trunk-V2.patch
  against trunk revision 1129989.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 4 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/555//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/555//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/555//console

This message is automatically generated.

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Robert Joseph Evans
>              Labels: jmx
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7411-0.20.20X-V1.patch, HADOOP-7411-0.20.20X-V2.patch, HADOOP-7411-trunk-V1.patch, HADOOP-7411-trunk-V2.patch, HADOOP-7411-trunk-alpha.patch, jmx.json
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15697-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 17:41:32 2011
Return-Path: <common-issues-return-15697-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DAECE697E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 17:41:31 +0000 (UTC)
Received: (qmail 58964 invoked by uid 500); 1 Jun 2011 17:41:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58912 invoked by uid 500); 1 Jun 2011 17:41:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58904 invoked by uid 99); 1 Jun 2011 17:41:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 17:41:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 17:41:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9AC1FEE931
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 17:40:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 17:40:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1348718755.60401.1306950047630.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042315#comment-13042315 ] 

Luke Lu commented on HADOOP-7144:
---------------------------------

+1. It's good enough for commit, even though IMO, it's better to catch and log Throwable and setStatus 500 for security reasons (it's usually recommended to disallow stack traces in responses in production mode, although it's not a big deal in this particular case). BTW, if you submit the trunk patch last and make patch available, hudson/jenkins will pickup the right patch.

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Robert Joseph Evans
>              Labels: jmx
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7411-0.20.20X-V1.patch, HADOOP-7411-0.20.20X-V2.patch, HADOOP-7411-trunk-V1.patch, HADOOP-7411-trunk-V2.patch, HADOOP-7411-trunk-alpha.patch, jmx.json
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15698-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 17:59:30 2011
Return-Path: <common-issues-return-15698-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D764361DA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 17:59:30 +0000 (UTC)
Received: (qmail 80053 invoked by uid 500); 1 Jun 2011 17:59:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80016 invoked by uid 500); 1 Jun 2011 17:59:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80008 invoked by uid 99); 1 Jun 2011 17:59:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 17:59:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 17:59:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7C8A3EE019
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 17:58:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 17:58:47 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <919435663.60441.1306951127506.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <504299282.44039.1306374047470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7331) Makes hadoop-daemon.sh to return 1
 if daemon processes did not get started
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042323#comment-13042323 ] 

Tanping Wang commented on HADOOP-7331:
--------------------------------------

If no furture comments, would this be committed?

> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7331
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7331
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7331.2.patch, HADOOP-7331.patch
>
>
> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15699-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 17:59:31 2011
Return-Path: <common-issues-return-15699-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 84E3961FB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 17:59:31 +0000 (UTC)
Received: (qmail 80348 invoked by uid 500); 1 Jun 2011 17:59:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80260 invoked by uid 500); 1 Jun 2011 17:59:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80210 invoked by uid 99); 1 Jun 2011 17:59:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 17:59:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 17:59:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A5D92EE021
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 17:58:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 17:58:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1978049135.60447.1306951127676.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <504299282.44039.1306374047470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7331) Makes hadoop-daemon.sh to return 1
 if daemon processes did not get started
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042324#comment-13042324 ] 

Todd Lipcon commented on HADOOP-7331:
-------------------------------------

Steve, just to check: you're OK with committing this as is, right?

> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7331
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7331
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7331.2.patch, HADOOP-7331.patch
>
>
> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15700-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 18:20:31 2011
Return-Path: <common-issues-return-15700-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E56106512
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 18:20:30 +0000 (UTC)
Received: (qmail 24464 invoked by uid 500); 1 Jun 2011 18:20:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24428 invoked by uid 500); 1 Jun 2011 18:20:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24420 invoked by uid 99); 1 Jun 2011 18:20:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 18:20:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 18:20:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 99937EE8ED
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 18:19:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 18:19:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1904727755.60504.1306952387625.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <397648942.57126.1306869588398.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7342) Add an utility API in FileUtil for
 JDK File.list
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042338#comment-13042338 ] 

Matt Foley commented on HADOOP-7342:
------------------------------------

Cos, the reason I support this new utility api, is that Hadoop is full of constructs like
{code}
// in context of a "throws IOException" or try/catch IOE:
File[] files = dir.list();
if (files.length > 0) {...}
{code}

This code handles empty and non-empty directory lists, and is ready to handle IOExceptions.  Unfortunately, the posix-style File.list() api returns null when it would reasonably be expected to throw IOException, causing NPE on the .length method call, which as a run-time exception isn't typically even looked for.

Rather than inject null-checking code with a conditional throw statement into every place like this, Bharath has suggested calling this new FileUtil method.  Developers can still use the JDK method when the null return on IOE is desirable.  I think this is a good idea.

> Add an utility API in FileUtil for JDK File.list
> ------------------------------------------------
>
>                 Key: HADOOP-7342
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7342
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7342-1.patch
>
>
> Java File.list API can return null when disk is bad or directory is not a directory. This utility API in FileUtil will throw an exception when this happens rather than returning null. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15701-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 18:22:30 2011
Return-Path: <common-issues-return-15701-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D73936532
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 18:22:30 +0000 (UTC)
Received: (qmail 25411 invoked by uid 500); 1 Jun 2011 18:22:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25352 invoked by uid 500); 1 Jun 2011 18:22:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25100 invoked by uid 99); 1 Jun 2011 18:22:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 18:22:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 18:22:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3061DEEAB3
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 18:21:48 +0000 (UTC)
Date: Wed, 1 Jun 2011 18:21:48 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <334649763.60517.1306952508194.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <397648942.57126.1306869588398.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7342) Add an utility API in FileUtil for
 JDK File.list
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042340#comment-13042340 ] 

Matt Foley commented on HADOOP-7342:
------------------------------------

oops, that's String[] not File[].  Same logic for .listFiles() :-)

> Add an utility API in FileUtil for JDK File.list
> ------------------------------------------------
>
>                 Key: HADOOP-7342
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7342
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7342-1.patch
>
>
> Java File.list API can return null when disk is bad or directory is not a directory. This utility API in FileUtil will throw an exception when this happens rather than returning null. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15702-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 18:38:31 2011
Return-Path: <common-issues-return-15702-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C759C6FC9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 18:38:31 +0000 (UTC)
Received: (qmail 56288 invoked by uid 500); 1 Jun 2011 18:38:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56249 invoked by uid 500); 1 Jun 2011 18:38:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56241 invoked by uid 99); 1 Jun 2011 18:38:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 18:38:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 18:38:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 73CD2EE14D
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 18:37:48 +0000 (UTC)
Date: Wed, 1 Jun 2011 18:37:48 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1795640380.60595.1306953468471.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042352#comment-13042352 ] 

Luke Lu commented on HADOOP-7144:
---------------------------------

Another improvement would be to setStatus 404 if the queryNames result is empty.

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Robert Joseph Evans
>              Labels: jmx
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7411-0.20.20X-V1.patch, HADOOP-7411-0.20.20X-V2.patch, HADOOP-7411-trunk-V1.patch, HADOOP-7411-trunk-V2.patch, HADOOP-7411-trunk-alpha.patch, jmx.json
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15703-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 21:39:31 2011
Return-Path: <common-issues-return-15703-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4DD656A40
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 21:39:31 +0000 (UTC)
Received: (qmail 51936 invoked by uid 500); 1 Jun 2011 21:39:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51892 invoked by uid 500); 1 Jun 2011 21:39:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51883 invoked by uid 99); 1 Jun 2011 21:39:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 21:39:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 21:39:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AC329EE854
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 21:38:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 21:38:47 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1818551636.61033.1306964327702.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <654871262.37888.1306191347393.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7323) Add capability to resolve
 compression codec based on codec name
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042454#comment-13042454 ] 

Tom White commented on HADOOP-7323:
-----------------------------------

Thinking about this more, overloading getCodecByClassName() may be misleading, so it might be better to add a new method called getCodecByName() which returns codecs based on class name or alias. There are only a couple of callers of getCodecByClassName() (in HDFS) so it doesn't make much difference in terms of changing code to use the new method.

To take advantage of the new method expressions of the form

{code}
conf.getClassByName(name).asSubclass(CompressionCodec.class)
{code}

should be replaced with

{code}
CompressionCodecFactory.getCodecByName(name)
{code}

This mainly applies in the MapReduce project.

We should also add a getCodecClassByName() method at the same time, since sometimes only the class is needed.

> Add capability to resolve compression codec based on codec name
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7323
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7323
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7323.patch, HADOOP-7323b.patch
>
>
> When setting up a compression codec in an MR job the full class name of the codec must be used.
> To ease usability, compression codecs should be resolved by their codec name (ie 'gzip', 'deflate', 'zlib', 'bzip2') instead their full codec class name.
> Besides easy of use for Hadoop users who would use the codec alias instead the full codec class name, it could simplify how HBase resolves loads the codecs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15704-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 21:45:33 2011
Return-Path: <common-issues-return-15704-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4AAB46EFD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 21:45:33 +0000 (UTC)
Received: (qmail 60505 invoked by uid 500); 1 Jun 2011 21:45:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60474 invoked by uid 500); 1 Jun 2011 21:45:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60466 invoked by uid 99); 1 Jun 2011 21:45:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 21:45:33 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 21:45:30 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 20C80EEAEC
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 21:44:49 +0000 (UTC)
Date: Wed, 1 Jun 2011 21:44:49 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1134527432.61073.1306964689131.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7001) Allow configuration changes
 without restarting configured nodes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042459#comment-13042459 ] 

Hudson commented on HADOOP-7001:
--------------------------------

Integrated in Hadoop-Common-22-branch #60 (See [https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/60/])
    Merging changes r1038493 and r1038480 for HADOOP-7001 from trunk

suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130306
Files : 
* /hadoop/common/branches/branch-0.22/src/docs
* /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/conf/ReconfigurableBase.java
* /hadoop/common/branches/branch-0.22
* /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/conf/Reconfigurable.java
* /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/conf/ReconfigurationException.java
* /hadoop/common/branches/branch-0.22/src/test/core
* /hadoop/common/branches/branch-0.22/src/test/core/org/apache/hadoop/io/TestSequenceFile.java
* /hadoop/common/branches/branch-0.22/CHANGES.txt
* /hadoop/common/branches/branch-0.22/src/java
* /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/conf/Configuration.java
* /hadoop/common/branches/branch-0.22/src/test/core/org/apache/hadoop/conf/TestReconfiguration.java
* /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/conf/ReconfigurationUtil.java
* /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/conf/ReconfigurationServlet.java


> Allow configuration changes without restarting configured nodes
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7001
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7001
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: conf
>    Affects Versions: 0.23.0
>            Reporter: Patrick Kling
>            Assignee: Patrick Kling
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7001.2.patch, HADOOP-7001.3.patch, HADOOP-7001.4.patch, HADOOP-7001.5.patch, HADOOP-7001.patch, reconfigurable.patch
>
>
> Currently, changing the configuration on a node (e.g., the name node) requires that we restart the node. We propose a change that would allow us to make configuration changes without restarting. Nodes that support configuration changes at run time should implement the following interface:
> interface ChangeableConfigured extends Configured {
>    void changeConfiguration(Configuration newConf) throws ConfigurationChangeException;
> }
> The contract of changeConfiguration is as follows:
> The node will compare newConf to the existing configuration. For each configuration property that is set to a different value than in the current configuration, the node will either adjust its behaviour to conform to the new configuration or throw a ConfigurationChangeException if this change is not possible at run time. If a configuration property is set in the current configuration but is unset in newConf, the node should use its default value for this property. After a successful invocation of changeConfiguration, the behaviour of the configured node should be indistinguishable from the behaviour of a node that was configured with newConf at creation.
> It should be easy to change existing nodes to implement this interface. We can start by throwing the exception for all changes and then gradually start supporting more and more changes at run time. (We might even consider replacing Configured with ChangeableConfigured entirely, but I think the proposal above afford greater flexibility).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15705-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 21:59:34 2011
Return-Path: <common-issues-return-15705-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 761BD655E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 21:59:34 +0000 (UTC)
Received: (qmail 87683 invoked by uid 500); 1 Jun 2011 21:59:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87611 invoked by uid 500); 1 Jun 2011 21:59:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87465 invoked by uid 99); 1 Jun 2011 21:59:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 21:59:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 21:59:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D7AF7EE27F
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 21:58:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 21:58:47 +0000 (UTC)
From: "Bharath Mundlapudi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1193750186.61135.1306965527880.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1570148867.57902.1306887887374.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7345) listStatus for local files throws
 NPE instead of permission denied
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bharath Mundlapudi resolved HADOOP-7345.
----------------------------------------

    Resolution: Duplicate

> listStatus for local files throws NPE instead of permission denied
> ------------------------------------------------------------------
>
>                 Key: HADOOP-7345
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7345
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>
> Calling {{fs.listStatus}} on a local directory where the user does not have permissions generates a {{NullPointerException}}:
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> 	at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1115)
> 	at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1150)
> 	at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:494)
> {noformat}
> FileSystem.java:
> {code}
> 1111:  private void listStatus(ArrayList<FileStatus> results, Path f,
> 1112:      PathFilter filter) throws FileNotFoundException, IOException {
> 1113:    FileStatus listing[] = listStatus(f);
> 1114:
> 1115:    for (int i = 0; i < listing.length; i++) {
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15706-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 22:55:30 2011
Return-Path: <common-issues-return-15706-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 92EB86A29
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 22:55:30 +0000 (UTC)
Received: (qmail 56623 invoked by uid 500); 1 Jun 2011 22:55:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56591 invoked by uid 500); 1 Jun 2011 22:55:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56581 invoked by uid 99); 1 Jun 2011 22:55:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 22:55:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 22:55:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 61EB8EF34A
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 22:54:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 22:54:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1967033465.61217.1306968887397.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <164637914.38616.1306225908771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7327) FileSystem.listStatus() throws
 NullPointerException instead of IOException upon access permission failure
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042494#comment-13042494 ] 

Daryn Sharp commented on HADOOP-7327:
-------------------------------------

A generic IOE with a message of "Error accessing", while definitively preferable to a NPE that tears down FsShell, affords the user no idea of the underlying error.  At a minimum, it would be nice to distinguish a permission denied from a "something went wrong".  I think that would still conform to NIO behavior since the security manager will throw an exception.

> FileSystem.listStatus() throws NullPointerException instead of IOException upon access permission failure
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7327
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7327
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>         Attachments: hadoop-7327-1.patch
>
>
> Many processes that call listStatus() expect to handle IOException, but instead are getting runtime error NullPointerException, if the directory being scanned is visible but no-access to the running user id.  For example, if directory foo is drwxr-xr-x, and subdirectory foo/bar is drwx------, then trying to do listStatus(Path(foo/bar)) will cause a NullPointerException.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15707-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 22:57:30 2011
Return-Path: <common-issues-return-15707-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A96876A58
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 22:57:30 +0000 (UTC)
Received: (qmail 57344 invoked by uid 500); 1 Jun 2011 22:57:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57314 invoked by uid 500); 1 Jun 2011 22:57:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57306 invoked by uid 99); 1 Jun 2011 22:57:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 22:57:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 22:57:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 85D4AEF3E6
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 22:56:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 22:56:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <567243826.61224.1306969007544.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <164637914.38616.1306225908771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7327) FileSystem.listStatus() throws
 NullPointerException instead of IOException upon access permission failure
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042495#comment-13042495 ] 

Daryn Sharp commented on HADOOP-7327:
-------------------------------------

Also, I'm pretty sure acl exceptions were being thrown a few months.  Did something maybe change in the NN?

> FileSystem.listStatus() throws NullPointerException instead of IOException upon access permission failure
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7327
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7327
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>         Attachments: hadoop-7327-1.patch
>
>
> Many processes that call listStatus() expect to handle IOException, but instead are getting runtime error NullPointerException, if the directory being scanned is visible but no-access to the running user id.  For example, if directory foo is drwxr-xr-x, and subdirectory foo/bar is drwx------, then trying to do listStatus(Path(foo/bar)) will cause a NullPointerException.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15708-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 23:15:30 2011
Return-Path: <common-issues-return-15708-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CAC2C62DA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 23:15:30 +0000 (UTC)
Received: (qmail 75163 invoked by uid 500); 1 Jun 2011 23:15:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75122 invoked by uid 500); 1 Jun 2011 23:15:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75110 invoked by uid 99); 1 Jun 2011 23:15:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 23:15:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 23:15:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A4623EF8AC
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 23:14:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 23:14:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1897325592.61264.1306970087669.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <314496369.49888.1306530467371.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7335) Force entropy to come from
 non-true random for tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042504#comment-13042504 ] 

Todd Lipcon commented on HADOOP-7335:
-------------------------------------

Yea, I just tried changing my environment to file:/dev/urandom and it doesn't work correctly - I see Jetty hanging at startup. jstack shows it trying to read entropy. If I use file:///dev/urandom as I did in this patch, it starts immediately every time.

> Force entropy to come from non-true random for tests
> ----------------------------------------------------
>
>                 Key: HADOOP-7335
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7335
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7335.txt
>
>
> Passing the system property {{-Djava.security.egd=file:///dev/urandom}} forces the JVM to seed its PRNG from non-true random (/dev/urandom) instead of the true random (/dev/random). This makes the tests run faster, since without it they often hang waiting for entropy while Jetty is initializing.
> We should turn this on for the test targets by default, so developers/hudson boxes don't have to make this change system-wide or use workarounds like rngtools.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15709-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 23:58:30 2011
Return-Path: <common-issues-return-15709-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A6BE46C05
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 23:58:30 +0000 (UTC)
Received: (qmail 28512 invoked by uid 500); 1 Jun 2011 23:58:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28478 invoked by uid 500); 1 Jun 2011 23:58:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28470 invoked by uid 99); 1 Jun 2011 23:58:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 23:58:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 23:58:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6BF55EE245
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 23:57:47 +0000 (UTC)
Date: Wed, 1 Jun 2011 23:57:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1016265931.61342.1306972667438.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7349) HADOOP-7121 accidentally disabled
 some tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

HADOOP-7121 accidentally disabled some tests
--------------------------------------------

                 Key: HADOOP-7349
                 URL: https://issues.apache.org/jira/browse/HADOOP-7349
             Project: Hadoop Common
          Issue Type: Bug
          Components: ipc, test
    Affects Versions: 0.22.0
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
             Fix For: 0.22.0


When I converted TestIPC to JUnit 4, I missed a couple of tests towards the bottom of the file when adding the @Test annotation.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15710-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 23:59:29 2011
Return-Path: <common-issues-return-15710-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A09836C18
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 23:59:29 +0000 (UTC)
Received: (qmail 28903 invoked by uid 500); 1 Jun 2011 23:59:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28876 invoked by uid 500); 1 Jun 2011 23:59:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28868 invoked by uid 99); 1 Jun 2011 23:59:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 23:59:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 23:59:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4C268EE2D5
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 23:58:48 +0000 (UTC)
Date: Wed, 1 Jun 2011 23:58:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1149229970.61344.1306972728309.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1016265931.61342.1306972667438.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7349) HADOOP-7121 accidentally disabled
 some tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7349:
--------------------------------

    Attachment: hadoop-7349.txt

> HADOOP-7121 accidentally disabled some tests
> --------------------------------------------
>
>                 Key: HADOOP-7349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7349
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7349.txt
>
>
> When I converted TestIPC to JUnit 4, I missed a couple of tests towards the bottom of the file when adding the @Test annotation.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15711-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 23:59:29 2011
Return-Path: <common-issues-return-15711-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D2D1F6C25
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 23:59:29 +0000 (UTC)
Received: (qmail 29143 invoked by uid 500); 1 Jun 2011 23:59:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29116 invoked by uid 500); 1 Jun 2011 23:59:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29103 invoked by uid 99); 1 Jun 2011 23:59:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 23:59:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 23:59:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6FB1BEE2DC
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 23:58:48 +0000 (UTC)
Date: Wed, 1 Jun 2011 23:58:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <575890928.61349.1306972728454.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1016265931.61342.1306972667438.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7349) HADOOP-7121 accidentally disabled
 some tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7349:
--------------------------------

    Status: Patch Available  (was: Open)

> HADOOP-7121 accidentally disabled some tests
> --------------------------------------------
>
>                 Key: HADOOP-7349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7349
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7349.txt
>
>
> When I converted TestIPC to JUnit 4, I missed a couple of tests towards the bottom of the file when adding the @Test annotation.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15712-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  1 23:59:31 2011
Return-Path: <common-issues-return-15712-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9C0746C34
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  1 Jun 2011 23:59:31 +0000 (UTC)
Received: (qmail 29440 invoked by uid 500); 1 Jun 2011 23:59:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29396 invoked by uid 500); 1 Jun 2011 23:59:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29388 invoked by uid 99); 1 Jun 2011 23:59:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 23:59:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Jun 2011 23:59:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 51BFCEE2D6
	for <common-issues@hadoop.apache.org>; Wed,  1 Jun 2011 23:58:48 +0000 (UTC)
Date: Wed, 1 Jun 2011 23:58:48 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <640184672.61345.1306972728331.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7350) Use ServiceLoader to discover
 compression codec classes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Use ServiceLoader to discover compression codec classes
-------------------------------------------------------

                 Key: HADOOP-7350
                 URL: https://issues.apache.org/jira/browse/HADOOP-7350
             Project: Hadoop Common
          Issue Type: Improvement
          Components: conf, io
            Reporter: Tom White


By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15713-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 00:05:37 2011
Return-Path: <common-issues-return-15713-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5E29C656F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 00:05:37 +0000 (UTC)
Received: (qmail 38626 invoked by uid 500); 2 Jun 2011 00:05:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38509 invoked by uid 500); 2 Jun 2011 00:05:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38501 invoked by uid 99); 2 Jun 2011 00:05:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 00:05:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 00:05:35 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4156DEE4ED
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 00:04:54 +0000 (UTC)
Date: Thu, 2 Jun 2011 00:04:54 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <633786880.61355.1306973094264.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1016265931.61342.1306972667438.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7349) HADOOP-7121 accidentally disabled
 some tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042518#comment-13042518 ] 

Eli Collins commented on HADOOP-7349:
-------------------------------------

+1

> HADOOP-7121 accidentally disabled some tests
> --------------------------------------------
>
>                 Key: HADOOP-7349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7349
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7349.txt
>
>
> When I converted TestIPC to JUnit 4, I missed a couple of tests towards the bottom of the file when adding the @Test annotation.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15714-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 00:07:29 2011
Return-Path: <common-issues-return-15714-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2EC0F6599
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 00:07:29 +0000 (UTC)
Received: (qmail 39498 invoked by uid 500); 2 Jun 2011 00:07:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39434 invoked by uid 500); 2 Jun 2011 00:07:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39200 invoked by uid 99); 2 Jun 2011 00:07:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 00:07:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 00:07:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 955D3EE5AC
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 00:06:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 00:06:47 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <257664967.61363.1306973207608.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <640184672.61345.1306972728331.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7350) Use ServiceLoader to discover
 compression codec classes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-7350:
------------------------------

    Attachment: HADOOP-7350.patch

This patch modifies CompressionCodecFactory.getCodecClasses() to use a service loader in addition to reading class names from io.compression.codecs. 

> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
>                 Key: HADOOP-7350
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7350
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, io
>            Reporter: Tom White
>         Attachments: HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15715-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 00:07:29 2011
Return-Path: <common-issues-return-15715-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B98B965DB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 00:07:29 +0000 (UTC)
Received: (qmail 39583 invoked by uid 500); 2 Jun 2011 00:07:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39516 invoked by uid 500); 2 Jun 2011 00:07:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39284 invoked by uid 99); 2 Jun 2011 00:07:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 00:07:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 00:07:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A166EEE5AE
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 00:06:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 00:06:47 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1806381470.61365.1306973207658.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <640184672.61345.1306972728331.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7350) Use ServiceLoader to discover
 compression codec classes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-7350:
------------------------------

    Assignee: Tom White
      Status: Patch Available  (was: Open)

> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
>                 Key: HADOOP-7350
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7350
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, io
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15716-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 00:11:30 2011
Return-Path: <common-issues-return-15716-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 16F9466DC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 00:11:30 +0000 (UTC)
Received: (qmail 41966 invoked by uid 500); 2 Jun 2011 00:11:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41913 invoked by uid 500); 2 Jun 2011 00:11:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41721 invoked by uid 99); 2 Jun 2011 00:11:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 00:11:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 00:11:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 93643EE761
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 00:10:48 +0000 (UTC)
Date: Thu, 2 Jun 2011 00:10:48 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1645916018.61396.1306973448600.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1067116917.37592.1306187867429.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7322) Adding a util method in FileUtil for
 JDK File.listFiles
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated HADOOP-7322:
-------------------------------

    Summary: Adding a util method in FileUtil for JDK File.listFiles  (was: Adding a util method in FileUtil for directory listing)

> Adding a util method in FileUtil for JDK File.listFiles
> -------------------------------------------------------
>
>                 Key: HADOOP-7322
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7322
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7322-1.patch, HADOOP-7322-2.patch, HADOOP-7322-3.patch
>
>
> While testing Disk Fail Inplace, we encountered lots of NPE from Dir.listFiles API. This API can return null when Dir is not directory or disk is bad. I am proposing to have a File Util which can be used consistently across to deal with disk issues. This util api will do the following:
> 1. When error happens it will throw IOException
> 2. Else it will return empty list or list of files.
> Signature:
> File[] FileUtil.listFiles(File dir) throws IOException {}
> This way we no need to write wrapper code every where. Also, API is consistent with the signature.
>  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15717-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 00:15:30 2011
Return-Path: <common-issues-return-15717-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A506E6878
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 00:15:30 +0000 (UTC)
Received: (qmail 46274 invoked by uid 500); 2 Jun 2011 00:15:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46246 invoked by uid 500); 2 Jun 2011 00:15:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46238 invoked by uid 99); 2 Jun 2011 00:15:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 00:15:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 00:15:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 706D5EE8E6
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 00:14:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 00:14:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1088725496.61424.1306973687457.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1016265931.61342.1306972667438.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7349) HADOOP-7121 accidentally disabled
 some tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042522#comment-13042522 ] 

Hadoop QA commented on HADOOP-7349:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481169/hadoop-7349.txt
  against trunk revision 1129989.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/556//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/556//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/556//console

This message is automatically generated.

> HADOOP-7121 accidentally disabled some tests
> --------------------------------------------
>
>                 Key: HADOOP-7349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7349
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7349.txt
>
>
> When I converted TestIPC to JUnit 4, I missed a couple of tests towards the bottom of the file when adding the @Test annotation.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15718-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 00:25:30 2011
Return-Path: <common-issues-return-15718-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BBAC16131
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 00:25:30 +0000 (UTC)
Received: (qmail 58523 invoked by uid 500); 2 Jun 2011 00:25:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58460 invoked by uid 500); 2 Jun 2011 00:25:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58407 invoked by uid 99); 2 Jun 2011 00:25:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 00:25:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 00:25:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 58DEDEED04
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 00:24:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 00:24:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <876506214.61443.1306974287360.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <640184672.61345.1306972728331.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
 compression codec classes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042528#comment-13042528 ] 

Hadoop QA commented on HADOOP-7350:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481172/HADOOP-7350.patch
  against trunk revision 1129989.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/557//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/557//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/557//console

This message is automatically generated.

> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
>                 Key: HADOOP-7350
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7350
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, io
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15719-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 00:53:28 2011
Return-Path: <common-issues-return-15719-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BFC646D40
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 00:53:28 +0000 (UTC)
Received: (qmail 80341 invoked by uid 500); 2 Jun 2011 00:53:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80262 invoked by uid 500); 2 Jun 2011 00:53:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80197 invoked by uid 99); 2 Jun 2011 00:53:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 00:53:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 00:53:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 73657EF445
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 00:52:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 00:52:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <43396362.61498.1306975967469.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <647927887.58055.1306890116067.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7346) Send back nicer error to clients
 using outdated IPC version
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7346:
--------------------------------

    Attachment: hadoop-7346.txt

I couldn't think of any way to do it with mocks, so I did brute force and wrote some tests which just replay RPC traces that I captured by hand from 0.18.3, branch-0.20, and 0.21.0. They then verify that the response matches my captured response (which I verified generated a reasonable exception on the client)

> Send back nicer error to clients using outdated IPC version
> -----------------------------------------------------------
>
>                 Key: HADOOP-7346
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7346
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7346.txt, hadoop-7346.txt
>
>
> When an older Hadoop version tries to contact a newer Hadoop version across an IPC protocol version bump, the client currently just gets a non-useful error message like "EOFException".
> Instead, the IPC server code can speak just enough of prior IPC protocols to send back a "fatal" message indicating the version mismatch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15720-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 01:05:31 2011
Return-Path: <common-issues-return-15720-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2136763FE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 01:05:31 +0000 (UTC)
Received: (qmail 93798 invoked by uid 500); 2 Jun 2011 01:05:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93745 invoked by uid 500); 2 Jun 2011 01:05:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93729 invoked by uid 99); 2 Jun 2011 01:05:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 01:05:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 01:05:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8615BEF824
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 01:04:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 01:04:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1551320005.61560.1306976687546.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <647927887.58055.1306890116067.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7346) Send back nicer error to clients
 using outdated IPC version
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042544#comment-13042544 ] 

Hadoop QA commented on HADOOP-7346:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481179/hadoop-7346.txt
  against trunk revision 1129989.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/558//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/558//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/558//console

This message is automatically generated.

> Send back nicer error to clients using outdated IPC version
> -----------------------------------------------------------
>
>                 Key: HADOOP-7346
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7346
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7346.txt, hadoop-7346.txt
>
>
> When an older Hadoop version tries to contact a newer Hadoop version across an IPC protocol version bump, the client currently just gets a non-useful error message like "EOFException".
> Instead, the IPC server code can speak just enough of prior IPC protocols to send back a "fatal" message indicating the version mismatch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15721-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 01:25:29 2011
Return-Path: <common-issues-return-15721-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E40FB6BCB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 01:25:28 +0000 (UTC)
Received: (qmail 18362 invoked by uid 500); 2 Jun 2011 01:25:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18333 invoked by uid 500); 2 Jun 2011 01:25:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18324 invoked by uid 99); 2 Jun 2011 01:25:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 01:25:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 01:25:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A3CE2EFDB4
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 01:24:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 01:24:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <512237990.61615.1306977887667.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <640184672.61345.1306972728331.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
 compression codec classes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042554#comment-13042554 ] 

Todd Lipcon commented on HADOOP-7350:
-------------------------------------

- We should probably remove the codecs from core-default.xml now that they're loaded via ServiceLoader
- Is there a way to inject a new codec programatically through the ServiceLoader interface? If so, we could entirely deprecate io.compression.codecs. If not, maybe we should rename it to something like io.compression.extra.codecs and specify that it's only necessary if you have a codec that doesn't expose itself through ServiceLoader?
- hdfs-default.xml has an item dfs.image.compression.codec that needs to be updated

> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
>                 Key: HADOOP-7350
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7350
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, io
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15722-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 01:51:31 2011
Return-Path: <common-issues-return-15722-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 147436996
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 01:51:31 +0000 (UTC)
Received: (qmail 44997 invoked by uid 500); 2 Jun 2011 01:51:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44929 invoked by uid 500); 2 Jun 2011 01:51:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44921 invoked by uid 99); 2 Jun 2011 01:51:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 01:51:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 01:51:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A90D4EF41E
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 01:50:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 01:50:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <71746544.61661.1306979447688.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <164637914.38616.1306225908771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7327) FileSystem.listStatus() throws
 NullPointerException instead of IOException upon access permission failure
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042568#comment-13042568 ] 

Matt Foley commented on HADOOP-7327:
------------------------------------

Daryn, at the point of this patch insertion, all we know is that the underlying call (via the abstract API) returned null.  How do you propose to distinguish among the possible causes?

> FileSystem.listStatus() throws NullPointerException instead of IOException upon access permission failure
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7327
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7327
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>         Attachments: hadoop-7327-1.patch
>
>
> Many processes that call listStatus() expect to handle IOException, but instead are getting runtime error NullPointerException, if the directory being scanned is visible but no-access to the running user id.  For example, if directory foo is drwxr-xr-x, and subdirectory foo/bar is drwx------, then trying to do listStatus(Path(foo/bar)) will cause a NullPointerException.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15723-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 06:06:33 2011
Return-Path: <common-issues-return-15723-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7E83A46F4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 06:06:33 +0000 (UTC)
Received: (qmail 34772 invoked by uid 500); 2 Jun 2011 06:06:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34555 invoked by uid 500); 2 Jun 2011 06:06:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34536 invoked by uid 99); 2 Jun 2011 06:06:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 06:06:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 06:06:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4D136EFC66
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 06:05:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 06:05:47 +0000 (UTC)
From: "stack (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <99930859.62060.1306994747296.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7351) Regression:
 HttpServer#getWebAppsPath used to be protected so subclasses could supply
 alternate webapps path but it was made private by HADOOP-6461
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Regression: HttpServer#getWebAppsPath used to be protected so subclasses could supply alternate webapps path but it was made private by HADOOP-6461
---------------------------------------------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-7351
                 URL: https://issues.apache.org/jira/browse/HADOOP-7351
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: stack


It USED to be protected rather than private but its access was changed by HADOOP-6461.  It did the following:

{code}
-  protected String getWebAppsPath() throws IOException {
-    URL url = getClass().getClassLoader().getResource("webapps");
+  private String getWebAppsPath(String appName) throws FileNotFoundException {
+    URL url = getClass().getClassLoader().getResource("webapps/" + appName);
...
{code}

HBase subclasses HttpServer providing its UI.  This change makes it so we can no longer do so.

This change made it into 0.21.  I'd like to get a fix committed to 0.22 as well as TRUNK.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15724-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 06:09:32 2011
Return-Path: <common-issues-return-15724-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 42FA2479C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 06:09:32 +0000 (UTC)
Received: (qmail 37292 invoked by uid 500); 2 Jun 2011 06:09:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37199 invoked by uid 500); 2 Jun 2011 06:09:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37181 invoked by uid 99); 2 Jun 2011 06:09:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 06:09:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 06:09:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5A7B4EFD52
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 06:08:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 06:08:47 +0000 (UTC)
From: "stack (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <452639219.62063.1306994927367.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <99930859.62060.1306994747296.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7351) Regression:
 HttpServer#getWebAppsPath used to be protected so subclasses could supply
 alternate webapps path but it was made private by HADOOP-6461
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HADOOP-7351:
--------------------------

    Assignee: stack
      Status: Patch Available  (was: Open)

> Regression: HttpServer#getWebAppsPath used to be protected so subclasses could supply alternate webapps path but it was made private by HADOOP-6461
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7351
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7351
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>         Attachments: 7351.txt
>
>
> It USED to be protected rather than private but its access was changed by HADOOP-6461.  It did the following:
> {code}
> -  protected String getWebAppsPath() throws IOException {
> -    URL url = getClass().getClassLoader().getResource("webapps");
> +  private String getWebAppsPath(String appName) throws FileNotFoundException {
> +    URL url = getClass().getClassLoader().getResource("webapps/" + appName);
> ...
> {code}
> HBase subclasses HttpServer providing its UI.  This change makes it so we can no longer do so.
> This change made it into 0.21.  I'd like to get a fix committed to 0.22 as well as TRUNK.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15725-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 06:09:33 2011
Return-Path: <common-issues-return-15725-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3940247AC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 06:09:33 +0000 (UTC)
Received: (qmail 37479 invoked by uid 500); 2 Jun 2011 06:09:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37303 invoked by uid 500); 2 Jun 2011 06:09:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37196 invoked by uid 99); 2 Jun 2011 06:09:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 06:09:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 06:09:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 53223EFD51
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 06:08:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 06:08:47 +0000 (UTC)
From: "stack (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1648811901.62062.1306994927323.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <99930859.62060.1306994747296.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7351) Regression:
 HttpServer#getWebAppsPath used to be protected so subclasses could supply
 alternate webapps path but it was made private by HADOOP-6461
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HADOOP-7351:
--------------------------

    Attachment: 7351.txt

Patch that does this:

{code}
-  private String getWebAppsPath(String appName) throws FileNotFoundException {
+  protected String getWebAppsPath(String appName) throws FileNotFoundException {
{code}

only.

> Regression: HttpServer#getWebAppsPath used to be protected so subclasses could supply alternate webapps path but it was made private by HADOOP-6461
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7351
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7351
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: stack
>         Attachments: 7351.txt
>
>
> It USED to be protected rather than private but its access was changed by HADOOP-6461.  It did the following:
> {code}
> -  protected String getWebAppsPath() throws IOException {
> -    URL url = getClass().getClassLoader().getResource("webapps");
> +  private String getWebAppsPath(String appName) throws FileNotFoundException {
> +    URL url = getClass().getClassLoader().getResource("webapps/" + appName);
> ...
> {code}
> HBase subclasses HttpServer providing its UI.  This change makes it so we can no longer do so.
> This change made it into 0.21.  I'd like to get a fix committed to 0.22 as well as TRUNK.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15726-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 06:17:32 2011
Return-Path: <common-issues-return-15726-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 353314156
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 06:17:32 +0000 (UTC)
Received: (qmail 40362 invoked by uid 500); 2 Jun 2011 06:17:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40228 invoked by uid 500); 2 Jun 2011 06:17:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40214 invoked by uid 99); 2 Jun 2011 06:17:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 06:17:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 06:17:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B8016EFF3F
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 06:16:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 06:16:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <958686885.62073.1306995407750.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7352) Contracts of LocalFileSystem and
 DistributedFileSystem should require FileSystem::listStatus throw
 IOException not return null upon access error
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Contracts of LocalFileSystem and DistributedFileSystem should require FileSystem::listStatus throw IOException not return null upon access error
------------------------------------------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-7352
                 URL: https://issues.apache.org/jira/browse/HADOOP-7352
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs, fs/s3
            Reporter: Matt Foley
            Assignee: Matt Foley


In HADOOP-6201 and HDFS-538 it was agreed that FileSystem::listStatus should throw FileNotFoundException instead of returning null, when the target directory did not exist.

However, in LocalFileSystem implementation today, FileSystem::listStatus still may return null, when the target directory exists but does not grant read permission.  This causes NPE in many callers, for all the reasons cited in HADOOP-6201 and HDFS-538.  See HADOOP-7327 and its linked issues for examples.


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15727-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 06:19:31 2011
Return-Path: <common-issues-return-15727-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8FAA04B56
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 06:19:31 +0000 (UTC)
Received: (qmail 43084 invoked by uid 500); 2 Jun 2011 06:19:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43013 invoked by uid 500); 2 Jun 2011 06:19:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42987 invoked by uid 99); 2 Jun 2011 06:19:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 06:19:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 06:19:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 735A4EFFEF
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 06:18:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 06:18:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <796040710.62077.1306995527456.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <958686885.62073.1306995407750.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7352) Contracts of LocalFileSystem and
 DistributedFileSystem should require FileSystem::listStatus throw
 IOException not return null upon access error
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042622#comment-13042622 ] 

Matt Foley commented on HADOOP-7352:
------------------------------------

This behavior, of returning null on access error, is following the pattern of File::list and File::listFiles, which are defined by the JDK.  However, those cause NPE in a lot of Hadoop code too, which is why they've recently been fixed by HADOOP-7342, HADOOP-7322, HDFS-1934, and HDFS-2019.  Those patches provided FileUtil alternatives to the JDK methods.  That approach isn't applicable to FileSystem::listStatus because it is an abstract method in an important contract class defined by Hadoop.  It has a massive number of callers in Common and Mapred (although I found none in HDFS).

I believe this change in semantics, to throw IOException instead of returning null, would have no negative impact.  I've examined the 36 non-trivial callers in Common, and only 2 checked for null result.  All the others would definitely get NPE on null result, and the two that did check had faulty logic!  In Mapred there are far more callers, but almost all of them also will throw NPE on null result.  In going through about half of them, I found one correct and one incorrect null check in about 20 callers.

Therefore, I believe there is no downside to changing the semantics of the base method in this way, and we'll get rid of some mystery NPEs.  We will need to scan for any callers that check for null and use it as a non-error condition; there won't be many but there will be a couple.

Please give feedback.  If this is acceptable I will provide a patch for each of the underlying FileSystem.listStatus implementations, and do the careful scan for callers that actually expect null as an allowed result.

> Contracts of LocalFileSystem and DistributedFileSystem should require FileSystem::listStatus throw IOException not return null upon access error
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7352
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7352
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs, fs/s3
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>
> In HADOOP-6201 and HDFS-538 it was agreed that FileSystem::listStatus should throw FileNotFoundException instead of returning null, when the target directory did not exist.
> However, in LocalFileSystem implementation today, FileSystem::listStatus still may return null, when the target directory exists but does not grant read permission.  This causes NPE in many callers, for all the reasons cited in HADOOP-6201 and HDFS-538.  See HADOOP-7327 and its linked issues for examples.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15728-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 06:25:31 2011
Return-Path: <common-issues-return-15728-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 987164E44
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 06:25:31 +0000 (UTC)
Received: (qmail 45442 invoked by uid 500); 2 Jun 2011 06:25:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45304 invoked by uid 500); 2 Jun 2011 06:25:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45293 invoked by uid 99); 2 Jun 2011 06:25:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 06:25:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 06:25:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 61C09EF197
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 06:24:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 06:24:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1234431402.62080.1306995887396.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <99930859.62060.1306994747296.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7351) Regression:
 HttpServer#getWebAppsPath used to be protected so subclasses could supply
 alternate webapps path but it was made private by HADOOP-6461
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042623#comment-13042623 ] 

Hadoop QA commented on HADOOP-7351:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481198/7351.txt
  against trunk revision 1129989.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/559//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/559//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/559//console

This message is automatically generated.

> Regression: HttpServer#getWebAppsPath used to be protected so subclasses could supply alternate webapps path but it was made private by HADOOP-6461
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7351
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7351
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>         Attachments: 7351.txt
>
>
> It USED to be protected rather than private but its access was changed by HADOOP-6461.  It did the following:
> {code}
> -  protected String getWebAppsPath() throws IOException {
> -    URL url = getClass().getClassLoader().getResource("webapps");
> +  private String getWebAppsPath(String appName) throws FileNotFoundException {
> +    URL url = getClass().getClassLoader().getResource("webapps/" + appName);
> ...
> {code}
> HBase subclasses HttpServer providing its UI.  This change makes it so we can no longer do so.
> This change made it into 0.21.  I'd like to get a fix committed to 0.22 as well as TRUNK.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15729-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 06:29:31 2011
Return-Path: <common-issues-return-15729-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 24FCB4EC4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 06:29:31 +0000 (UTC)
Received: (qmail 49327 invoked by uid 500); 2 Jun 2011 06:29:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49280 invoked by uid 500); 2 Jun 2011 06:29:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49267 invoked by uid 99); 2 Jun 2011 06:29:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 06:29:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 06:29:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5A4AAEF2E2
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 06:28:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 06:28:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1772281569.62086.1306996127366.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <958686885.62073.1306995407750.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7352) Contracts of LocalFileSystem and
 DistributedFileSystem should require FileSystem::listStatus throw
 IOException not return null upon access error
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042625#comment-13042625 ] 

Matt Foley commented on HADOOP-7352:
------------------------------------

Correction:  there were calls to FileSystem::listStatus in HDFS, but they were all in Test classes, except for two pass-through calls in HftpFileSystem.listStatus() and HftpFileSystem.LsParser.listStatus().

> Contracts of LocalFileSystem and DistributedFileSystem should require FileSystem::listStatus throw IOException not return null upon access error
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7352
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7352
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs, fs/s3
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>
> In HADOOP-6201 and HDFS-538 it was agreed that FileSystem::listStatus should throw FileNotFoundException instead of returning null, when the target directory did not exist.
> However, in LocalFileSystem implementation today, FileSystem::listStatus still may return null, when the target directory exists but does not grant read permission.  This causes NPE in many callers, for all the reasons cited in HADOOP-6201 and HDFS-538.  See HADOOP-7327 and its linked issues for examples.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15730-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 14:16:32 2011
Return-Path: <common-issues-return-15730-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C54C94451
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 14:16:32 +0000 (UTC)
Received: (qmail 1904 invoked by uid 500); 2 Jun 2011 14:16:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1833 invoked by uid 500); 2 Jun 2011 14:16:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1819 invoked by uid 99); 2 Jun 2011 14:16:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 14:16:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 14:16:30 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AF8E4F0523
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 14:15:50 +0000 (UTC)
Date: Thu, 2 Jun 2011 14:15:50 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1729859746.62851.1307024150714.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <164637914.38616.1306225908771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7327) FileSystem.listStatus() throws
 NullPointerException instead of IOException upon access permission failure
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042787#comment-13042787 ] 

Daryn Sharp commented on HADOOP-7327:
-------------------------------------

I checked an old path from months ago that I never submitted for FsShell.  I see that I was catching AccessControlExceptions to standardized the output format.  File access is throwing ACEs but I'm not sure whether dirs were too.  It would stand to reason that both file and dir access should throw ACEs, which I think is consistent with NIO behavior?

That said, I'm only commenting/lamenting so my intention isn't to block this patch since a NPE will prematurely grind many FsShell commands to a halt.  For instance, "du" is worthless if just one dir in the tree is unreadable.  In many cases this patch is unquestionably better than nothing.

> FileSystem.listStatus() throws NullPointerException instead of IOException upon access permission failure
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7327
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7327
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>         Attachments: hadoop-7327-1.patch
>
>
> Many processes that call listStatus() expect to handle IOException, but instead are getting runtime error NullPointerException, if the directory being scanned is visible but no-access to the running user id.  For example, if directory foo is drwxr-xr-x, and subdirectory foo/bar is drwx------, then trying to do listStatus(Path(foo/bar)) will cause a NullPointerException.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15731-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 14:31:29 2011
Return-Path: <common-issues-return-15731-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E5215420D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 14:31:28 +0000 (UTC)
Received: (qmail 24001 invoked by uid 500); 2 Jun 2011 14:31:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23966 invoked by uid 500); 2 Jun 2011 14:31:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23956 invoked by uid 99); 2 Jun 2011 14:31:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 14:31:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 14:31:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9BEFFF0A87
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 14:30:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 14:30:47 +0000 (UTC)
From: "ramkrishna.s.vasudevan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1376414453.62883.1307025047635.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1430617657.560.1300108409748.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7191)  BackUpNameNode is using 100% CPU
 and not accepting any requests.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ramkrishna.s.vasudevan updated HADOOP-7191:
-------------------------------------------

    Attachment: HADOOP-7191.patch

>  BackUpNameNode is using 100% CPU and not accepting any requests.
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7191
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7191
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>            Reporter: Uma Maheswara Rao G
>         Attachments: HADOOP-7191.patch
>
>
> In our environment, after 3 days long run Backup NameNode is using 100% CPU and not accepting any calls. 
> *Thread dump*
> "IPC Server Responder" daemon prio=10 tid=0x00007f86c41c6800 nid=0x3b2a runnable [0x00007f86ce579000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> locked <0x00007f86d67e2a20> (a sun.nio.ch.Util$1) 
> locked <0x00007f86d67e2a08> (a java.util.Collections$UnmodifiableSet) 
> locked <0x00007f86d67e26a8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.hadoop.ipc.Server$Responder.run(Server.java:501) 
> Looks like we are running into similar issue like this Jetty one. http://jira.codehaus.org/browse/JETTY-937

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15732-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 14:41:29 2011
Return-Path: <common-issues-return-15732-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B26C2456A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 14:41:29 +0000 (UTC)
Received: (qmail 51784 invoked by uid 500); 2 Jun 2011 14:41:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51737 invoked by uid 500); 2 Jun 2011 14:41:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51683 invoked by uid 99); 2 Jun 2011 14:41:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 14:41:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 14:41:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D0F84F0FBC
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 14:40:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 14:40:47 +0000 (UTC)
From: "ramkrishna.s.vasudevan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <156841823.62922.1307025647852.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1430617657.560.1300108409748.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7191)  BackUpNameNode is using 100% CPU
 and not accepting any requests.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ramkrishna.s.vasudevan updated HADOOP-7191:
-------------------------------------------

    Affects Version/s: 0.23.0
                       0.20-append
               Status: Patch Available  (was: Open)

As per the patch in DIRMINA-678

>  BackUpNameNode is using 100% CPU and not accepting any requests.
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7191
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7191
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.20-append, 0.23.0
>            Reporter: Uma Maheswara Rao G
>         Attachments: HADOOP-7191.patch
>
>
> In our environment, after 3 days long run Backup NameNode is using 100% CPU and not accepting any calls. 
> *Thread dump*
> "IPC Server Responder" daemon prio=10 tid=0x00007f86c41c6800 nid=0x3b2a runnable [0x00007f86ce579000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> locked <0x00007f86d67e2a20> (a sun.nio.ch.Util$1) 
> locked <0x00007f86d67e2a08> (a java.util.Collections$UnmodifiableSet) 
> locked <0x00007f86d67e26a8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.hadoop.ipc.Server$Responder.run(Server.java:501) 
> Looks like we are running into similar issue like this Jetty one. http://jira.codehaus.org/browse/JETTY-937

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15733-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 14:55:29 2011
Return-Path: <common-issues-return-15733-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7146444CA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 14:55:29 +0000 (UTC)
Received: (qmail 79299 invoked by uid 500); 2 Jun 2011 14:55:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79272 invoked by uid 500); 2 Jun 2011 14:55:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79264 invoked by uid 99); 2 Jun 2011 14:55:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 14:55:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 14:55:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E383FF0543
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 14:54:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 14:54:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1504865748.62967.1307026487928.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1430617657.560.1300108409748.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7191)  BackUpNameNode is using 100% CPU
 and not accepting any requests.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042804#comment-13042804 ] 

Hadoop QA commented on HADOOP-7191:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481236/HADOOP-7191.patch
  against trunk revision 1129989.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs.  The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/560//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/560//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/560//console

This message is automatically generated.

>  BackUpNameNode is using 100% CPU and not accepting any requests.
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7191
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7191
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.20-append, 0.23.0
>            Reporter: Uma Maheswara Rao G
>         Attachments: HADOOP-7191.patch
>
>
> In our environment, after 3 days long run Backup NameNode is using 100% CPU and not accepting any calls. 
> *Thread dump*
> "IPC Server Responder" daemon prio=10 tid=0x00007f86c41c6800 nid=0x3b2a runnable [0x00007f86ce579000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> locked <0x00007f86d67e2a20> (a sun.nio.ch.Util$1) 
> locked <0x00007f86d67e2a08> (a java.util.Collections$UnmodifiableSet) 
> locked <0x00007f86d67e26a8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.hadoop.ipc.Server$Responder.run(Server.java:501) 
> Looks like we are running into similar issue like this Jetty one. http://jira.codehaus.org/browse/JETTY-937

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15734-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 15:04:31 2011
Return-Path: <common-issues-return-15734-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 12CC64F07
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 15:04:31 +0000 (UTC)
Received: (qmail 95352 invoked by uid 500); 2 Jun 2011 15:04:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95320 invoked by uid 500); 2 Jun 2011 15:04:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95289 invoked by uid 99); 2 Jun 2011 15:04:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 15:04:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 15:04:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5D561F08F3
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 15:03:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 15:03:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <671647059.63008.1307027027379.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7353) Display of errors masks details of
 some exceptions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Display of errors masks details of some exceptions
--------------------------------------------------

                 Key: HADOOP-7353
                 URL: https://issues.apache.org/jira/browse/HADOOP-7353
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
    Affects Versions: 0.23.0
            Reporter: Daryn Sharp
            Assignee: Daryn Sharp


{{FsShell}}'s top level exception handler catches and displays exceptions.  Unfortunately it displays only the first line of an exception, which means an unexpected {{RuntimeExceptions}} like {{NullPointerException}} only display "{{cmd: NullPointerException}}".  This user has no context to understand and/or accurately report the issue.

Found due to bugs such as {{HADOOP-7327}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15735-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 15:10:32 2011
Return-Path: <common-issues-return-15735-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C03B342A6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 15:10:32 +0000 (UTC)
Received: (qmail 7773 invoked by uid 500); 2 Jun 2011 15:10:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7719 invoked by uid 500); 2 Jun 2011 15:10:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7664 invoked by uid 99); 2 Jun 2011 15:10:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 15:10:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 15:10:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 81C11F0C16
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 15:09:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 15:09:47 +0000 (UTC)
From: "Devaraj K (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <293705748.63018.1307027387527.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Moved] (HADOOP-7354) NullPointerException in the job
 tracker UI, when we perform kill or change the priority of jobs without
 selecting the any job.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj K moved MAPREDUCE-2528 to HADOOP-7354:
----------------------------------------------

          Component/s:     (was: jobtracker)
        Fix Version/s:     (was: 0.23.0)
                       0.23.0
    Affects Version/s:     (was: 0.23.0)
                       0.23.0
                  Key: HADOOP-7354  (was: MAPREDUCE-2528)
              Project: Hadoop Common  (was: Hadoop Map/Reduce)

> NullPointerException in the job tracker UI, when we perform kill or change the priority of jobs without selecting the any job.
> ------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7354
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7354
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Devaraj K
>            Assignee: Devaraj K
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2528-1.patch, MAPREDUCE-2528.patch
>
>
> If we click on Kill Selected Jobs or Change button without selecting any job, it is giving the below exception in the UI.
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.http.HttpServer$QuotingInputFilter$RequestQuoter.getParameterValues(HttpServer.java:798)
> at org.apache.hadoop.mapred.JSPUtil.processButtons(JSPUtil.java:209)
> at org.apache.hadoop.mapred.jobtracker_jsp._jspService(jobtracker_jsp.java:146)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1124)
> at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:871)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1115)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:361)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:879)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:741)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:213)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15736-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 15:12:28 2011
Return-Path: <common-issues-return-15736-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CD74744A8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 15:12:28 +0000 (UTC)
Received: (qmail 13543 invoked by uid 500); 2 Jun 2011 15:12:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13515 invoked by uid 500); 2 Jun 2011 15:12:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13507 invoked by uid 99); 2 Jun 2011 15:12:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 15:12:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 15:12:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7752EF0CF6
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 15:11:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 15:11:47 +0000 (UTC)
From: "Devaraj K (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1682913859.63047.1307027507485.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7354) NullPointerException in the job
 tracker UI, when we perform kill or change the priority of jobs without
 selecting the any job.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042816#comment-13042816 ] 

Devaraj K commented on HADOOP-7354:
-----------------------------------

Issue is moved to Hadoop Common because changes need to be done in common for this issue. 
Provided patch with server side fix.


> NullPointerException in the job tracker UI, when we perform kill or change the priority of jobs without selecting the any job.
> ------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7354
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7354
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Devaraj K
>            Assignee: Devaraj K
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2528-1.patch, MAPREDUCE-2528.patch
>
>
> If we click on Kill Selected Jobs or Change button without selecting any job, it is giving the below exception in the UI.
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.http.HttpServer$QuotingInputFilter$RequestQuoter.getParameterValues(HttpServer.java:798)
> at org.apache.hadoop.mapred.JSPUtil.processButtons(JSPUtil.java:209)
> at org.apache.hadoop.mapred.jobtracker_jsp._jspService(jobtracker_jsp.java:146)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1124)
> at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:871)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1115)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:361)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:879)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:741)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:213)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15737-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 15:12:29 2011
Return-Path: <common-issues-return-15737-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 05D5E44B1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 15:12:29 +0000 (UTC)
Received: (qmail 13795 invoked by uid 500); 2 Jun 2011 15:12:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13718 invoked by uid 500); 2 Jun 2011 15:12:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13577 invoked by uid 99); 2 Jun 2011 15:12:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 15:12:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 15:12:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 90433F0CF9
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 15:11:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 15:11:47 +0000 (UTC)
From: "Devaraj K (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1409139511.63050.1307027507587.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7354) NullPointerException in the job
 tracker UI, when we perform kill or change the priority of jobs without
 selecting the any job.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj K updated HADOOP-7354:
------------------------------

    Status: Patch Available  (was: Open)

> NullPointerException in the job tracker UI, when we perform kill or change the priority of jobs without selecting the any job.
> ------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7354
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7354
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Devaraj K
>            Assignee: Devaraj K
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2528-1.patch, MAPREDUCE-2528.patch
>
>
> If we click on Kill Selected Jobs or Change button without selecting any job, it is giving the below exception in the UI.
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.http.HttpServer$QuotingInputFilter$RequestQuoter.getParameterValues(HttpServer.java:798)
> at org.apache.hadoop.mapred.JSPUtil.processButtons(JSPUtil.java:209)
> at org.apache.hadoop.mapred.jobtracker_jsp._jspService(jobtracker_jsp.java:146)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1124)
> at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:871)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1115)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:361)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:879)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:741)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:213)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15738-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 15:35:31 2011
Return-Path: <common-issues-return-15738-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B29B64B8C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 15:35:31 +0000 (UTC)
Received: (qmail 65687 invoked by uid 500); 2 Jun 2011 15:35:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65654 invoked by uid 500); 2 Jun 2011 15:35:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65646 invoked by uid 99); 2 Jun 2011 15:35:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 15:35:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 15:35:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8C71AF06E3
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 15:34:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 15:34:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <917112299.63105.1307028887571.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7354) NullPointerException in the job
 tracker UI, when we perform kill or change the priority of jobs without
 selecting the any job.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042829#comment-13042829 ] 

Hadoop QA commented on HADOOP-7354:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481233/MAPREDUCE-2528-1.patch
  against trunk revision 1129989.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/561//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/561//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/561//console

This message is automatically generated.

> NullPointerException in the job tracker UI, when we perform kill or change the priority of jobs without selecting the any job.
> ------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7354
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7354
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Devaraj K
>            Assignee: Devaraj K
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2528-1.patch, MAPREDUCE-2528.patch
>
>
> If we click on Kill Selected Jobs or Change button without selecting any job, it is giving the below exception in the UI.
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.http.HttpServer$QuotingInputFilter$RequestQuoter.getParameterValues(HttpServer.java:798)
> at org.apache.hadoop.mapred.JSPUtil.processButtons(JSPUtil.java:209)
> at org.apache.hadoop.mapred.jobtracker_jsp._jspService(jobtracker_jsp.java:146)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1124)
> at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:871)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1115)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:361)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:879)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:741)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:213)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15739-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 16:46:31 2011
Return-Path: <common-issues-return-15739-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F013E406B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 16:46:30 +0000 (UTC)
Received: (qmail 14943 invoked by uid 500); 2 Jun 2011 16:46:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14893 invoked by uid 500); 2 Jun 2011 16:46:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14885 invoked by uid 99); 2 Jun 2011 16:46:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 16:46:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 16:46:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 97239F01D3
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 16:45:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 16:45:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <135051868.63252.1307033147615.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <99930859.62060.1306994747296.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7351) Regression:
 HttpServer#getWebAppsPath used to be protected so subclasses could supply
 alternate webapps path but it was made private by HADOOP-6461
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042867#comment-13042867 ] 

Todd Lipcon commented on HADOOP-7351:
-------------------------------------

We should figure out what the intended interface audience and classification is for HttpServer... I'm thinking:
@InterfaceAudience.LimitedPrivate({"HDFS", "MapReduce", "HBase"})
@InterfaceStability.Evolving

does that sound right?

> Regression: HttpServer#getWebAppsPath used to be protected so subclasses could supply alternate webapps path but it was made private by HADOOP-6461
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7351
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7351
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>         Attachments: 7351.txt
>
>
> It USED to be protected rather than private but its access was changed by HADOOP-6461.  It did the following:
> {code}
> -  protected String getWebAppsPath() throws IOException {
> -    URL url = getClass().getClassLoader().getResource("webapps");
> +  private String getWebAppsPath(String appName) throws FileNotFoundException {
> +    URL url = getClass().getClassLoader().getResource("webapps/" + appName);
> ...
> {code}
> HBase subclasses HttpServer providing its UI.  This change makes it so we can no longer do so.
> This change made it into 0.21.  I'd like to get a fix committed to 0.22 as well as TRUNK.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15740-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 17:07:31 2011
Return-Path: <common-issues-return-15740-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E0EF64C1F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 17:07:30 +0000 (UTC)
Received: (qmail 63543 invoked by uid 500); 2 Jun 2011 17:07:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63516 invoked by uid 500); 2 Jun 2011 17:07:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63508 invoked by uid 99); 2 Jun 2011 17:07:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 17:07:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 17:07:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 673F4F0A79
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 17:06:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 17:06:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1382063703.63329.1307034407419.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <958686885.62073.1306995407750.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7352) Contracts of LocalFileSystem and
 DistributedFileSystem should require FileSystem::listStatus throw
 IOException not return null upon access error
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042884#comment-13042884 ] 

Matt Foley commented on HADOOP-7352:
------------------------------------

In a comment in another bug (<a href="https://issues.apache.org/jira/browse/HADOOP-7327?focusedCommentId=13042494&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13042494">here</a>), Daryn points out that if the exceptions are thrown from a lower level with knowledge of the cause of the problem, it would be good to throw something informative such as AccessControlException, rather than just a generic IOException.

I agree in principle, but AccessControlException is a subclass of SecurityException, which is a RuntimeException and therefore no better than NPE.  All of the callers of listStatus() are prepared to get an IOException, but none of the standard sub-classes of IOException include access failures.  Is there a Hadoop or Apache defined sub-class of IOException that would be appropriate?

> Contracts of LocalFileSystem and DistributedFileSystem should require FileSystem::listStatus throw IOException not return null upon access error
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7352
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7352
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs, fs/s3
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>
> In HADOOP-6201 and HDFS-538 it was agreed that FileSystem::listStatus should throw FileNotFoundException instead of returning null, when the target directory did not exist.
> However, in LocalFileSystem implementation today, FileSystem::listStatus still may return null, when the target directory exists but does not grant read permission.  This causes NPE in many callers, for all the reasons cited in HADOOP-6201 and HDFS-538.  See HADOOP-7327 and its linked issues for examples.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15741-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 17:09:28 2011
Return-Path: <common-issues-return-15741-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CF2224C74
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 17:09:28 +0000 (UTC)
Received: (qmail 66360 invoked by uid 500); 2 Jun 2011 17:09:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66322 invoked by uid 500); 2 Jun 2011 17:09:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66314 invoked by uid 99); 2 Jun 2011 17:09:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 17:09:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 17:09:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6AAD1F0B4D
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 17:08:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 17:08:47 +0000 (UTC)
From: "stack (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <48310626.63336.1307034527433.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <99930859.62060.1306994747296.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7351) Regression:
 HttpServer#getWebAppsPath used to be protected so subclasses could supply
 alternate webapps path but it was made private by HADOOP-6461
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042886#comment-13042886 ] 

stack commented on HADOOP-7351:
-------------------------------

Sounds fine to me.  Want me to add this to patch?

> Regression: HttpServer#getWebAppsPath used to be protected so subclasses could supply alternate webapps path but it was made private by HADOOP-6461
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7351
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7351
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>         Attachments: 7351.txt
>
>
> It USED to be protected rather than private but its access was changed by HADOOP-6461.  It did the following:
> {code}
> -  protected String getWebAppsPath() throws IOException {
> -    URL url = getClass().getClassLoader().getResource("webapps");
> +  private String getWebAppsPath(String appName) throws FileNotFoundException {
> +    URL url = getClass().getClassLoader().getResource("webapps/" + appName);
> ...
> {code}
> HBase subclasses HttpServer providing its UI.  This change makes it so we can no longer do so.
> This change made it into 0.21.  I'd like to get a fix committed to 0.22 as well as TRUNK.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15742-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 17:09:29 2011
Return-Path: <common-issues-return-15742-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 097144C7B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 17:09:29 +0000 (UTC)
Received: (qmail 66619 invoked by uid 500); 2 Jun 2011 17:09:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66558 invoked by uid 500); 2 Jun 2011 17:09:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66382 invoked by uid 99); 2 Jun 2011 17:09:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 17:09:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 17:09:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7C739F0B50
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 17:08:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 17:08:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1906647090.63338.1307034527506.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <164637914.38616.1306225908771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7327) FileSystem.listStatus() throws
 NullPointerException instead of IOException upon access permission failure
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042887#comment-13042887 ] 

Matt Foley commented on HADOOP-7327:
------------------------------------

@Daryn, for discussion of what exception to throw in the more complete solution, please see https://issues.apache.org/jira/browse/HADOOP-7352?focusedCommentId=13042884&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13042884

> FileSystem.listStatus() throws NullPointerException instead of IOException upon access permission failure
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7327
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7327
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>         Attachments: hadoop-7327-1.patch
>
>
> Many processes that call listStatus() expect to handle IOException, but instead are getting runtime error NullPointerException, if the directory being scanned is visible but no-access to the running user id.  For example, if directory foo is drwxr-xr-x, and subdirectory foo/bar is drwx------, then trying to do listStatus(Path(foo/bar)) will cause a NullPointerException.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15743-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 17:11:28 2011
Return-Path: <common-issues-return-15743-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CE1F74D90
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 17:11:28 +0000 (UTC)
Received: (qmail 70408 invoked by uid 500); 2 Jun 2011 17:11:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70355 invoked by uid 500); 2 Jun 2011 17:11:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70347 invoked by uid 99); 2 Jun 2011 17:11:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 17:11:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 17:11:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 61106F0C69
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 17:10:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 17:10:47 +0000 (UTC)
From: "stack (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1912493675.63344.1307034647394.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <99930859.62060.1306994747296.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7351) Regression:
 HttpServer#getWebAppsPath used to be protected so subclasses could supply
 alternate webapps path but it was made private by HADOOP-6461
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042889#comment-13042889 ] 

stack commented on HADOOP-7351:
-------------------------------

Though, Todd, your suggestions are a little out of scope for this JIRA?  Should I do annotations in new issue?

> Regression: HttpServer#getWebAppsPath used to be protected so subclasses could supply alternate webapps path but it was made private by HADOOP-6461
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7351
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7351
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>         Attachments: 7351.txt
>
>
> It USED to be protected rather than private but its access was changed by HADOOP-6461.  It did the following:
> {code}
> -  protected String getWebAppsPath() throws IOException {
> -    URL url = getClass().getClassLoader().getResource("webapps");
> +  private String getWebAppsPath(String appName) throws FileNotFoundException {
> +    URL url = getClass().getClassLoader().getResource("webapps/" + appName);
> ...
> {code}
> HBase subclasses HttpServer providing its UI.  This change makes it so we can no longer do so.
> This change made it into 0.21.  I'd like to get a fix committed to 0.22 as well as TRUNK.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15744-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 17:32:29 2011
Return-Path: <common-issues-return-15744-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EA24548D0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 17:32:28 +0000 (UTC)
Received: (qmail 8031 invoked by uid 500); 2 Jun 2011 17:32:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8001 invoked by uid 500); 2 Jun 2011 17:32:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7993 invoked by uid 99); 2 Jun 2011 17:32:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 17:32:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 17:32:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9F196F0A0B
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 17:31:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 17:31:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <347375324.63409.1307035907648.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <99930859.62060.1306994747296.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7351) Regression:
 HttpServer#getWebAppsPath used to be protected so subclasses could supply
 alternate webapps path but it was made private by HADOOP-6461
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042900#comment-13042900 ] 

Todd Lipcon commented on HADOOP-7351:
-------------------------------------

Yea, other issue is fine. +1 for this issue, but let's try to get the annotations in 0.22 as well. Maintaining APIs that allow subclassing by other projects is tough.

> Regression: HttpServer#getWebAppsPath used to be protected so subclasses could supply alternate webapps path but it was made private by HADOOP-6461
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7351
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7351
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>         Attachments: 7351.txt
>
>
> It USED to be protected rather than private but its access was changed by HADOOP-6461.  It did the following:
> {code}
> -  protected String getWebAppsPath() throws IOException {
> -    URL url = getClass().getClassLoader().getResource("webapps");
> +  private String getWebAppsPath(String appName) throws FileNotFoundException {
> +    URL url = getClass().getClassLoader().getResource("webapps/" + appName);
> ...
> {code}
> HBase subclasses HttpServer providing its UI.  This change makes it so we can no longer do so.
> This change made it into 0.21.  I'd like to get a fix committed to 0.22 as well as TRUNK.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15745-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 17:50:31 2011
Return-Path: <common-issues-return-15745-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7F66444AF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 17:50:31 +0000 (UTC)
Received: (qmail 44636 invoked by uid 500); 2 Jun 2011 17:50:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44596 invoked by uid 500); 2 Jun 2011 17:50:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44588 invoked by uid 99); 2 Jun 2011 17:50:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 17:50:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 17:50:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3338EF09B2
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 17:49:48 +0000 (UTC)
Date: Thu, 2 Jun 2011 17:49:48 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <790882.63604.1307036988206.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <958686885.62073.1306995407750.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7352) Contracts of LocalFileSystem and
 DistributedFileSystem should require FileSystem::listStatus throw
 IOException not return null upon access error
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042932#comment-13042932 ] 

Daryn Sharp commented on HADOOP-7352:
-------------------------------------

Hadoop common has a {{org.apache.hadoop.fs.permission.AccessControlException}} derives from {{IOException}}.

> Contracts of LocalFileSystem and DistributedFileSystem should require FileSystem::listStatus throw IOException not return null upon access error
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7352
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7352
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs, fs/s3
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>
> In HADOOP-6201 and HDFS-538 it was agreed that FileSystem::listStatus should throw FileNotFoundException instead of returning null, when the target directory did not exist.
> However, in LocalFileSystem implementation today, FileSystem::listStatus still may return null, when the target directory exists but does not grant read permission.  This causes NPE in many callers, for all the reasons cited in HADOOP-6201 and HDFS-538.  See HADOOP-7327 and its linked issues for examples.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15746-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 18:16:30 2011
Return-Path: <common-issues-return-15746-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F071D4977
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 18:16:29 +0000 (UTC)
Received: (qmail 189 invoked by uid 500); 2 Jun 2011 18:16:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 133 invoked by uid 500); 2 Jun 2011 18:16:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99862 invoked by uid 99); 2 Jun 2011 18:16:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 18:16:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 18:16:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 92FC3F08E2
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 18:15:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 18:15:47 +0000 (UTC)
From: "stack (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <620892300.63733.1307038547598.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7355) Add audience and stability
 annotations to HttpServer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Add audience and stability annotations to HttpServer class
----------------------------------------------------------

                 Key: HADOOP-7355
                 URL: https://issues.apache.org/jira/browse/HADOOP-7355
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: stack
            Assignee: stack


HttpServer has at least one subclasser in HBase.  Flag this class w/ annotations that make this plain so we avoid regressions like HADOOP-7351

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15747-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 18:24:29 2011
Return-Path: <common-issues-return-15747-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 41EFD4062
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 18:24:29 +0000 (UTC)
Received: (qmail 13104 invoked by uid 500); 2 Jun 2011 18:24:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13066 invoked by uid 500); 2 Jun 2011 18:24:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13054 invoked by uid 99); 2 Jun 2011 18:24:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 18:24:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 18:24:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 774AFF0C66
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 18:23:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 18:23:47 +0000 (UTC)
From: "stack (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1159492490.63746.1307039027485.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <620892300.63733.1307038547598.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7355) Add audience and stability
 annotations to HttpServer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HADOOP-7355:
--------------------------

    Attachment: 7355.txt

> Add audience and stability annotations to HttpServer class
> ----------------------------------------------------------
>
>                 Key: HADOOP-7355
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7355
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: stack
>            Assignee: stack
>         Attachments: 7355.txt
>
>
> HttpServer has at least one subclasser in HBase.  Flag this class w/ annotations that make this plain so we avoid regressions like HADOOP-7351

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15748-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 18:24:29 2011
Return-Path: <common-issues-return-15748-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 42C5E4068
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 18:24:29 +0000 (UTC)
Received: (qmail 13179 invoked by uid 500); 2 Jun 2011 18:24:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13077 invoked by uid 500); 2 Jun 2011 18:24:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13057 invoked by uid 99); 2 Jun 2011 18:24:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 18:24:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 18:24:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6C7F0F0C65
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 18:23:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 18:23:47 +0000 (UTC)
From: "stack (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1212636360.63745.1307039027441.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <99930859.62060.1306994747296.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7351) Regression:
 HttpServer#getWebAppsPath used to be protected so subclasses could supply
 alternate webapps path but it was made private by HADOOP-6461
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042959#comment-13042959 ] 

stack commented on HADOOP-7351:
-------------------------------

Added HADOOP-7355 to add annotations.

> Regression: HttpServer#getWebAppsPath used to be protected so subclasses could supply alternate webapps path but it was made private by HADOOP-6461
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7351
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7351
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>         Attachments: 7351.txt
>
>
> It USED to be protected rather than private but its access was changed by HADOOP-6461.  It did the following:
> {code}
> -  protected String getWebAppsPath() throws IOException {
> -    URL url = getClass().getClassLoader().getResource("webapps");
> +  private String getWebAppsPath(String appName) throws FileNotFoundException {
> +    URL url = getClass().getClassLoader().getResource("webapps/" + appName);
> ...
> {code}
> HBase subclasses HttpServer providing its UI.  This change makes it so we can no longer do so.
> This change made it into 0.21.  I'd like to get a fix committed to 0.22 as well as TRUNK.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15749-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 18:24:29 2011
Return-Path: <common-issues-return-15749-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B01BB4088
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 18:24:29 +0000 (UTC)
Received: (qmail 13549 invoked by uid 500); 2 Jun 2011 18:24:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13487 invoked by uid 500); 2 Jun 2011 18:24:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13245 invoked by uid 99); 2 Jun 2011 18:24:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 18:24:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 18:24:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9B33BF0C6C
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 18:23:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 18:23:47 +0000 (UTC)
From: "stack (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1079644619.63750.1307039027632.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <620892300.63733.1307038547598.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7355) Add audience and stability
 annotations to HttpServer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HADOOP-7355:
--------------------------

    Status: Patch Available  (was: Open)

> Add audience and stability annotations to HttpServer class
> ----------------------------------------------------------
>
>                 Key: HADOOP-7355
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7355
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: stack
>            Assignee: stack
>         Attachments: 7355.txt
>
>
> HttpServer has at least one subclasser in HBase.  Flag this class w/ annotations that make this plain so we avoid regressions like HADOOP-7351

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15750-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 18:28:31 2011
Return-Path: <common-issues-return-15750-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7E419457E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 18:28:31 +0000 (UTC)
Received: (qmail 20169 invoked by uid 500); 2 Jun 2011 18:28:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20134 invoked by uid 500); 2 Jun 2011 18:28:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20119 invoked by uid 99); 2 Jun 2011 18:28:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 18:28:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 18:28:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AE9BAF0ECA
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 18:27:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 18:27:47 +0000 (UTC)
From: "stack (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <766793301.63769.1307039267712.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <620892300.63733.1307038547598.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7355) Add audience and stability
 annotations to HttpServer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HADOOP-7355:
--------------------------

    Attachment: 7355-v2.txt

> Add audience and stability annotations to HttpServer class
> ----------------------------------------------------------
>
>                 Key: HADOOP-7355
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7355
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: stack
>            Assignee: stack
>         Attachments: 7355-v2.txt, 7355.txt
>
>
> HttpServer has at least one subclasser in HBase.  Flag this class w/ annotations that make this plain so we avoid regressions like HADOOP-7351

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15751-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 18:28:32 2011
Return-Path: <common-issues-return-15751-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 83404458B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 18:28:31 +0000 (UTC)
Received: (qmail 20230 invoked by uid 500); 2 Jun 2011 18:28:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20137 invoked by uid 500); 2 Jun 2011 18:28:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20120 invoked by uid 99); 2 Jun 2011 18:28:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 18:28:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 18:28:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7E5C1F0EC6
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 18:27:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 18:27:47 +0000 (UTC)
From: "stack (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <729315901.63766.1307039267514.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <620892300.63733.1307038547598.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7355) Add audience and stability
 annotations to HttpServer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HADOOP-7355:
--------------------------

    Status: Open  (was: Patch Available)

Missed import

> Add audience and stability annotations to HttpServer class
> ----------------------------------------------------------
>
>                 Key: HADOOP-7355
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7355
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: stack
>            Assignee: stack
>         Attachments: 7355-v2.txt, 7355.txt
>
>
> HttpServer has at least one subclasser in HBase.  Flag this class w/ annotations that make this plain so we avoid regressions like HADOOP-7351

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15752-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 18:30:28 2011
Return-Path: <common-issues-return-15752-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C00DC4968
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 18:30:28 +0000 (UTC)
Received: (qmail 23748 invoked by uid 500); 2 Jun 2011 18:30:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23719 invoked by uid 500); 2 Jun 2011 18:30:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23710 invoked by uid 99); 2 Jun 2011 18:30:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 18:30:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 18:30:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5B08BF0F95
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 18:29:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 18:29:47 +0000 (UTC)
From: "stack (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2020346558.63772.1307039387369.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <620892300.63733.1307038547598.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7355) Add audience and stability
 annotations to HttpServer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HADOOP-7355:
--------------------------

    Status: Patch Available  (was: Open)

Added missing import.  Resubmit.

> Add audience and stability annotations to HttpServer class
> ----------------------------------------------------------
>
>                 Key: HADOOP-7355
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7355
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: stack
>            Assignee: stack
>         Attachments: 7355-v2.txt, 7355.txt
>
>
> HttpServer has at least one subclasser in HBase.  Flag this class w/ annotations that make this plain so we avoid regressions like HADOOP-7351

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15753-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 18:41:28 2011
Return-Path: <common-issues-return-15753-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9D1624525
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 18:41:28 +0000 (UTC)
Received: (qmail 45413 invoked by uid 500); 2 Jun 2011 18:41:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45372 invoked by uid 500); 2 Jun 2011 18:41:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45363 invoked by uid 99); 2 Jun 2011 18:41:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 18:41:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 18:41:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6BBD3F03A5
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 18:40:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 18:40:47 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <322002804.63786.1307040047437.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <620892300.63733.1307038547598.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7355) Add audience and stability
 annotations to HttpServer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042971#comment-13042971 ] 

Tom White commented on HADOOP-7355:
-----------------------------------

+1

> Add audience and stability annotations to HttpServer class
> ----------------------------------------------------------
>
>                 Key: HADOOP-7355
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7355
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: stack
>            Assignee: stack
>         Attachments: 7355-v2.txt, 7355.txt
>
>
> HttpServer has at least one subclasser in HBase.  Flag this class w/ annotations that make this plain so we avoid regressions like HADOOP-7351

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15754-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 18:45:31 2011
Return-Path: <common-issues-return-15754-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2B9104A43
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 18:45:31 +0000 (UTC)
Received: (qmail 55197 invoked by uid 500); 2 Jun 2011 18:45:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55137 invoked by uid 500); 2 Jun 2011 18:45:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55129 invoked by uid 99); 2 Jun 2011 18:45:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 18:45:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 18:45:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 96C86F05B2
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 18:44:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 18:44:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2040142587.63803.1307040287614.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <620892300.63733.1307038547598.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7355) Add audience and stability
 annotations to HttpServer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042975#comment-13042975 ] 

Hadoop QA commented on HADOOP-7355:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481272/7355-v2.txt
  against trunk revision 1129989.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/562//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/562//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/562//console

This message is automatically generated.

> Add audience and stability annotations to HttpServer class
> ----------------------------------------------------------
>
>                 Key: HADOOP-7355
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7355
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: stack
>            Assignee: stack
>         Attachments: 7355-v2.txt, 7355.txt
>
>
> HttpServer has at least one subclasser in HBase.  Flag this class w/ annotations that make this plain so we avoid regressions like HADOOP-7351

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15755-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 19:12:29 2011
Return-Path: <common-issues-return-15755-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DD80B43FA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 19:12:28 +0000 (UTC)
Received: (qmail 10407 invoked by uid 500); 2 Jun 2011 19:12:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10371 invoked by uid 500); 2 Jun 2011 19:12:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10362 invoked by uid 99); 2 Jun 2011 19:12:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 19:12:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 19:12:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7D1A4F01C6
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 19:11:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 19:11:47 +0000 (UTC)
From: "stack (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <362341760.63859.1307041907509.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <99930859.62060.1306994747296.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7351) Regression:
 HttpServer#getWebAppsPath used to be protected so subclasses could supply
 alternate webapps path but it was made private by HADOOP-6461
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HADOOP-7351:
--------------------------

       Resolution: Fixed
    Fix Version/s: 0.22.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

Committed to 0.22 branch and to TRUNK.  Thanks for review Todd.

This small change makes it so HBase does not have to redo how it does its UI when it runs against hadoop 0.22; we subclass HttpServer so we can have the 'log level', 'thread dump', etc., servlets available in HBase UI. 

> Regression: HttpServer#getWebAppsPath used to be protected so subclasses could supply alternate webapps path but it was made private by HADOOP-6461
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7351
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7351
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>             Fix For: 0.22.0
>
>         Attachments: 7351.txt
>
>
> It USED to be protected rather than private but its access was changed by HADOOP-6461.  It did the following:
> {code}
> -  protected String getWebAppsPath() throws IOException {
> -    URL url = getClass().getClassLoader().getResource("webapps");
> +  private String getWebAppsPath(String appName) throws FileNotFoundException {
> +    URL url = getClass().getClassLoader().getResource("webapps/" + appName);
> ...
> {code}
> HBase subclasses HttpServer providing its UI.  This change makes it so we can no longer do so.
> This change made it into 0.21.  I'd like to get a fix committed to 0.22 as well as TRUNK.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15756-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 19:12:31 2011
Return-Path: <common-issues-return-15756-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 036504414
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 19:12:31 +0000 (UTC)
Received: (qmail 10717 invoked by uid 500); 2 Jun 2011 19:12:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10680 invoked by uid 500); 2 Jun 2011 19:12:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10672 invoked by uid 99); 2 Jun 2011 19:12:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 19:12:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 19:12:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9AF4CF01C9
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 19:11:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 19:11:47 +0000 (UTC)
From: "stack (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <664679266.63862.1307041907631.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <620892300.63733.1307038547598.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7355) Add audience and stability
 annotations to HttpServer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HADOOP-7355:
--------------------------

       Resolution: Fixed
    Fix Version/s: 0.22.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

Committed to branch and trunk.  Thanks for the review Tom.

> Add audience and stability annotations to HttpServer class
> ----------------------------------------------------------
>
>                 Key: HADOOP-7355
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7355
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: stack
>            Assignee: stack
>             Fix For: 0.22.0
>
>         Attachments: 7355-v2.txt, 7355.txt
>
>
> HttpServer has at least one subclasser in HBase.  Flag this class w/ annotations that make this plain so we avoid regressions like HADOOP-7351

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15758-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 19:14:30 2011
Return-Path: <common-issues-return-15758-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 48A884480
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 19:14:30 +0000 (UTC)
Received: (qmail 13905 invoked by uid 500); 2 Jun 2011 19:14:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13872 invoked by uid 500); 2 Jun 2011 19:14:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13830 invoked by uid 99); 2 Jun 2011 19:14:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 19:14:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 19:14:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A6121F03A6
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 19:13:48 +0000 (UTC)
Date: Thu, 2 Jun 2011 19:13:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1121771945.63881.1307042028677.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <99930859.62060.1306994747296.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7351) Regression:
 HttpServer#getWebAppsPath used to be protected so subclasses could supply
 alternate webapps path but it was made private by HADOOP-6461
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042998#comment-13042998 ] 

Hudson commented on HADOOP-7351:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #633 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/633/])
    HADOOP-7351 Regression: HttpServer#getWebAppsPath used to be protected so subclasses could supply alternate webapps path but it was made private by HADOOP-6461

stack : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130730
Files : 
* /hadoop/common/trunk/src/java/org/apache/hadoop/http/HttpServer.java
* /hadoop/common/trunk/CHANGES.txt


> Regression: HttpServer#getWebAppsPath used to be protected so subclasses could supply alternate webapps path but it was made private by HADOOP-6461
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7351
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7351
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>             Fix For: 0.22.0
>
>         Attachments: 7351.txt
>
>
> It USED to be protected rather than private but its access was changed by HADOOP-6461.  It did the following:
> {code}
> -  protected String getWebAppsPath() throws IOException {
> -    URL url = getClass().getClassLoader().getResource("webapps");
> +  private String getWebAppsPath(String appName) throws FileNotFoundException {
> +    URL url = getClass().getClassLoader().getResource("webapps/" + appName);
> ...
> {code}
> HBase subclasses HttpServer providing its UI.  This change makes it so we can no longer do so.
> This change made it into 0.21.  I'd like to get a fix committed to 0.22 as well as TRUNK.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15757-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 19:14:31 2011
Return-Path: <common-issues-return-15757-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 49EFE4485
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 19:14:30 +0000 (UTC)
Received: (qmail 13742 invoked by uid 500); 2 Jun 2011 19:14:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13701 invoked by uid 500); 2 Jun 2011 19:14:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13664 invoked by uid 99); 2 Jun 2011 19:14:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 19:14:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 19:14:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8B4FDF03A4
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 19:13:48 +0000 (UTC)
Date: Thu, 2 Jun 2011 19:13:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <477573083.63879.1307042028567.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <620892300.63733.1307038547598.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7355) Add audience and stability
 annotations to HttpServer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042997#comment-13042997 ] 

Hudson commented on HADOOP-7355:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #633 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/633/])
    HADOOP-7355 Add audience and stability annotations to HttpServer class

stack : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130729
Files : 
* /hadoop/common/trunk/src/java/org/apache/hadoop/http/HttpServer.java
* /hadoop/common/trunk/CHANGES.txt


> Add audience and stability annotations to HttpServer class
> ----------------------------------------------------------
>
>                 Key: HADOOP-7355
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7355
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: stack
>            Assignee: stack
>             Fix For: 0.22.0
>
>         Attachments: 7355-v2.txt, 7355.txt
>
>
> HttpServer has at least one subclasser in HBase.  Flag this class w/ annotations that make this plain so we avoid regressions like HADOOP-7351

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15759-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 19:14:32 2011
Return-Path: <common-issues-return-15759-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0B5AF44E3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 19:14:32 +0000 (UTC)
Received: (qmail 15056 invoked by uid 500); 2 Jun 2011 19:14:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15023 invoked by uid 500); 2 Jun 2011 19:14:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15015 invoked by uid 99); 2 Jun 2011 19:14:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 19:14:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 19:14:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 78167F03A2
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 19:13:48 +0000 (UTC)
Date: Thu, 2 Jun 2011 19:13:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1350904890.63877.1307042028488.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6461) webapps aren't located correctly
 post-split
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042996#comment-13042996 ] 

Hudson commented on HADOOP-6461:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #633 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/633/])
    HADOOP-7351 Regression: HttpServer#getWebAppsPath used to be protected so subclasses could supply alternate webapps path but it was made private by HADOOP-6461

stack : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130730
Files : 
* /hadoop/common/trunk/src/java/org/apache/hadoop/http/HttpServer.java
* /hadoop/common/trunk/CHANGES.txt


> webapps aren't located correctly post-split
> -------------------------------------------
>
>                 Key: HADOOP-6461
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6461
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Steve Loughran
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6461.patch, HADOOP-6461.patch, HADOOP-6461.patch, HADOOP-6461.patch, HADOOP-6461.txt, hadoop-6461.patch, hadoop-6461.patch, hadoop-6461.patch, hadoop-6461.txt, hadoop-6461.txt
>
>
> Post-split, when one builds common it creates an empty build/webapps dir. If you then try to launch the NN for example using HADOOP_HDFS_HOME=/path/to/hdfs hdfs namenode, HttpServer.getWebAppsPath locates the empty common webapps dir, and the NN web UI fails to load.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15760-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 19:24:28 2011
Return-Path: <common-issues-return-15760-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E4A3C4E8C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 19:24:28 +0000 (UTC)
Received: (qmail 26937 invoked by uid 500); 2 Jun 2011 19:24:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26907 invoked by uid 500); 2 Jun 2011 19:24:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26897 invoked by uid 99); 2 Jun 2011 19:24:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 19:24:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 19:24:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 602BAF0662
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 19:23:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 19:23:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1519302869.63899.1307042627390.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1016265931.61342.1306972667438.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7349) HADOOP-7121 accidentally disabled
 some tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7349:
--------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

Committed to 22 and trunk, thanks Eli

> HADOOP-7121 accidentally disabled some tests
> --------------------------------------------
>
>                 Key: HADOOP-7349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7349
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7349.txt
>
>
> When I converted TestIPC to JUnit 4, I missed a couple of tests towards the bottom of the file when adding the @Test annotation.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15761-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 19:28:29 2011
Return-Path: <common-issues-return-15761-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 562234F31
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 19:28:29 +0000 (UTC)
Received: (qmail 30736 invoked by uid 500); 2 Jun 2011 19:28:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30685 invoked by uid 500); 2 Jun 2011 19:28:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30677 invoked by uid 99); 2 Jun 2011 19:28:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 19:28:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 19:28:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B9860F0826
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 19:27:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 19:27:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <902840117.63910.1307042867756.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <99930859.62060.1306994747296.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7351) Regression:
 HttpServer#getWebAppsPath used to be protected so subclasses could supply
 alternate webapps path but it was made private by HADOOP-6461
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043005#comment-13043005 ] 

Hudson commented on HADOOP-7351:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #634 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/634/])
    HADOOP-7351 Regression: HttpServer#getWebAppsPath used to be protected so subclasses could supply alternate webapps path but it was made private by HADOOP-6461 -- Fix commit message; I'd put it in wrong location under 'new features' rather than under 'bugs'

stack : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130738
Files : 
* /hadoop/common/trunk/CHANGES.txt


> Regression: HttpServer#getWebAppsPath used to be protected so subclasses could supply alternate webapps path but it was made private by HADOOP-6461
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7351
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7351
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>             Fix For: 0.22.0
>
>         Attachments: 7351.txt
>
>
> It USED to be protected rather than private but its access was changed by HADOOP-6461.  It did the following:
> {code}
> -  protected String getWebAppsPath() throws IOException {
> -    URL url = getClass().getClassLoader().getResource("webapps");
> +  private String getWebAppsPath(String appName) throws FileNotFoundException {
> +    URL url = getClass().getClassLoader().getResource("webapps/" + appName);
> ...
> {code}
> HBase subclasses HttpServer providing its UI.  This change makes it so we can no longer do so.
> This change made it into 0.21.  I'd like to get a fix committed to 0.22 as well as TRUNK.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15762-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 19:28:31 2011
Return-Path: <common-issues-return-15762-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 528754F48
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 19:28:31 +0000 (UTC)
Received: (qmail 31211 invoked by uid 500); 2 Jun 2011 19:28:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31181 invoked by uid 500); 2 Jun 2011 19:28:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31173 invoked by uid 99); 2 Jun 2011 19:28:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 19:28:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 19:28:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A919EF0824
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 19:27:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 19:27:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <276558850.63908.1307042867689.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6461) webapps aren't located correctly
 post-split
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043004#comment-13043004 ] 

Hudson commented on HADOOP-6461:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #634 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/634/])
    HADOOP-7351 Regression: HttpServer#getWebAppsPath used to be protected so subclasses could supply alternate webapps path but it was made private by HADOOP-6461 -- Fix commit message; I'd put it in wrong location under 'new features' rather than under 'bugs'

stack : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130738
Files : 
* /hadoop/common/trunk/CHANGES.txt


> webapps aren't located correctly post-split
> -------------------------------------------
>
>                 Key: HADOOP-6461
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6461
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Steve Loughran
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6461.patch, HADOOP-6461.patch, HADOOP-6461.patch, HADOOP-6461.patch, HADOOP-6461.txt, hadoop-6461.patch, hadoop-6461.patch, hadoop-6461.patch, hadoop-6461.txt, hadoop-6461.txt
>
>
> Post-split, when one builds common it creates an empty build/webapps dir. If you then try to launch the NN for example using HADOOP_HDFS_HOME=/path/to/hdfs hdfs namenode, HttpServer.getWebAppsPath locates the empty common webapps dir, and the NN web UI fails to load.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15763-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 19:46:28 2011
Return-Path: <common-issues-return-15763-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B250F489D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 19:46:28 +0000 (UTC)
Received: (qmail 63394 invoked by uid 500); 2 Jun 2011 19:46:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63321 invoked by uid 500); 2 Jun 2011 19:46:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63313 invoked by uid 99); 2 Jun 2011 19:46:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 19:46:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 19:46:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7075BF0F1C
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 19:45:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 19:45:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <694023600.63942.1307043947457.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1016265931.61342.1306972667438.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7349) HADOOP-7121 accidentally disabled
 some tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043012#comment-13043012 ] 

Hudson commented on HADOOP-7349:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #635 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/635/])
    HADOOP-7349. HADOOP-7121 accidentally disabled some tests in TestIPC. Contributed by Todd Lipcon.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130758
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/ipc/TestIPC.java


> HADOOP-7121 accidentally disabled some tests
> --------------------------------------------
>
>                 Key: HADOOP-7349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7349
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7349.txt
>
>
> When I converted TestIPC to JUnit 4, I missed a couple of tests towards the bottom of the file when adding the @Test annotation.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15764-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 19:46:31 2011
Return-Path: <common-issues-return-15764-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4A67348F6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 19:46:31 +0000 (UTC)
Received: (qmail 63649 invoked by uid 500); 2 Jun 2011 19:46:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63608 invoked by uid 500); 2 Jun 2011 19:46:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63596 invoked by uid 99); 2 Jun 2011 19:46:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 19:46:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 19:46:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BAE4BF0F22
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 19:45:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 19:45:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1945182169.63948.1307043947762.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <3255382.248681296161204958.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7121) Exceptions while serializing IPC
 call response are not handled well
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043013#comment-13043013 ] 

Hudson commented on HADOOP-7121:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #635 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/635/])
    HADOOP-7349. HADOOP-7121 accidentally disabled some tests in TestIPC. Contributed by Todd Lipcon.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130758
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/ipc/TestIPC.java


> Exceptions while serializing IPC call response are not handled well
> -------------------------------------------------------------------
>
>                 Key: HADOOP-7121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7121
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>              Labels: newbie
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7121.txt, hadoop-7121.txt, hadoop-7121.txt
>
>
> We had a situation where for some reason the serialization of an RPC call's response was throwing OOME. When this happens, the exception is not caught, and the call never gets a response - the client just hangs. Additionally, the OOME propagated all the way to the top of the IPC handler and caused the handler. Plus, the Handler upon exit only logged to stdout and not to the log4j logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15765-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 19:46:31 2011
Return-Path: <common-issues-return-15765-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 741684904
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 19:46:31 +0000 (UTC)
Received: (qmail 63883 invoked by uid 500); 2 Jun 2011 19:46:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63816 invoked by uid 500); 2 Jun 2011 19:46:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63650 invoked by uid 99); 2 Jun 2011 19:46:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 19:46:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 19:46:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CB3FBF0F27
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 19:45:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 19:45:47 +0000 (UTC)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <98325961.63950.1307043947829.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <958686885.62073.1306995407750.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7352) Contracts of LocalFileSystem and
 DistributedFileSystem should require FileSystem::listStatus throw
 IOException not return null upon access error
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043014#comment-13043014 ] 

Jakob Homan commented on HADOOP-7352:
-------------------------------------

It's correct to make the contract consistent.  Trying to make IOExceptions useful at this point is probably a lost cause.  Otherwise, the changes proposed sound reasonable.

> Contracts of LocalFileSystem and DistributedFileSystem should require FileSystem::listStatus throw IOException not return null upon access error
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7352
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7352
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs, fs/s3
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>
> In HADOOP-6201 and HDFS-538 it was agreed that FileSystem::listStatus should throw FileNotFoundException instead of returning null, when the target directory did not exist.
> However, in LocalFileSystem implementation today, FileSystem::listStatus still may return null, when the target directory exists but does not grant read permission.  This causes NPE in many callers, for all the reasons cited in HADOOP-6201 and HDFS-538.  See HADOOP-7327 and its linked issues for examples.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15766-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 20:08:40 2011
Return-Path: <common-issues-return-15766-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E17B5440B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 20:08:39 +0000 (UTC)
Received: (qmail 97787 invoked by uid 500); 2 Jun 2011 20:08:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97744 invoked by uid 500); 2 Jun 2011 20:08:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97736 invoked by uid 99); 2 Jun 2011 20:08:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 20:08:39 +0000
X-ASF-Spam-Status: No, hits=2.0 required=5.0
	tests=FREEMAIL_FROM,RFC_ABUSE_POST,SPF_NEUTRAL,T_TO_NO_BRKTS_FREEMAIL,URI_HEX
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: 216.139.236.26 is neither permitted nor denied by domain of gokrazzzy@gmail.com)
Received: from [216.139.236.26] (HELO sam.nabble.com) (216.139.236.26)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 20:08:33 +0000
Received: from ben.nabble.com ([192.168.236.152])
	by sam.nabble.com with esmtp (Exim 4.72)
	(envelope-from <gokrazzzy@gmail.com>)
	id 1QSEBg-0005MD-IE
	for common-issues@hadoop.apache.org; Thu, 02 Jun 2011 13:08:12 -0700
Date: Thu, 2 Jun 2011 13:08:12 -0700 (PDT)
From: gokul <gokrazzzy@gmail.com>
To: common-issues@hadoop.apache.org
Message-ID: <1307045292541-3016972.post@n3.nabble.com>
Subject: Hadoop QuickStart Standalone Operation error
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Dear all,

I have been trying to get a hang of Hadoop (noob/ newbie). After
installation, I followed the Quick Start guide of Hadoop to execute the
sample example given (
http://hadoop.apache.org/common/docs/r0.20.2/quickstart.html#Local
Standalone Operation ). 
After executing the third step ( $ bin/hadoop jar hadoop-*-examples.jar grep
input output 'dfs[a-z.]+' ), 

I got the error saying that

"Exception in thread "main" java.lang.NoClassDefFoundError:
Internship/hadoop-0/20/2/bin////logs
Caused by: java.lang.ClassNotFoundException:
Internship.hadoop-0.20.2.bin....logs
	at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
Could not find the main class: Internship/hadoop-0.20.2/bin/../logs. Program
will exit."

I tried a lot of different things and nothing worked. 
Some of the solutions that didn't help: 
1) Set the $JAVA_HOME path correctly (works fine)
2) Command incorrect:  $ bin/hadoop jar hadoop-*-examples.jar grep input
output 'dfs[a-z.]+' (correct as per Hadoop QuickStart)


Any form of help is very very gratefully appreciated.

Thanks in advance.


--
View this message in context: http://hadoop-common.472056.n3.nabble.com/Hadoop-QuickStart-Standalone-Operation-error-tp3016972p3016972.html
Sent from the Issues mailing list archive at Nabble.com.

From common-issues-return-15767-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 20:24:29 2011
Return-Path: <common-issues-return-15767-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 336394462
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 20:24:29 +0000 (UTC)
Received: (qmail 31570 invoked by uid 500); 2 Jun 2011 20:24:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31541 invoked by uid 500); 2 Jun 2011 20:24:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31530 invoked by uid 99); 2 Jun 2011 20:24:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 20:24:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 20:24:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9DF75F18D5
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 20:23:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 20:23:47 +0000 (UTC)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <601228460.64008.1307046227643.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7106) Re-organize hadoop subversion
 layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043029#comment-13043029 ] 

Arun C Murthy commented on HADOOP-7106:
---------------------------------------

bq. MR-279 out of (yahoo-merge,MR-279,yahoo-merge).

+1, thanks Todd!

> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15768-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 20:50:31 2011
Return-Path: <common-issues-return-15768-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1026C4E6C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 20:50:31 +0000 (UTC)
Received: (qmail 81617 invoked by uid 500); 2 Jun 2011 20:50:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81590 invoked by uid 500); 2 Jun 2011 20:50:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81582 invoked by uid 99); 2 Jun 2011 20:50:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 20:50:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 20:50:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 90074F0001
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 20:49:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 20:49:47 +0000 (UTC)
From: "Bharath Mundlapudi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1138665100.64049.1307047787586.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <958686885.62073.1306995407750.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7352) Contracts of LocalFileSystem and
 DistributedFileSystem should require FileSystem::listStatus throw
 IOException not return null upon access error
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043037#comment-13043037 ] 

Bharath Mundlapudi commented on HADOOP-7352:
--------------------------------------------

Throwing an IOException is reasonable when listStatus returns null. The reason for this null case can be multiple things like file may not be present, permission issue, disk is bad, OS decided to make file system readonly etc etc. Since at FileSystem level we may not know all the causes for this so we should throw an IOException. Do you agree?

> Contracts of LocalFileSystem and DistributedFileSystem should require FileSystem::listStatus throw IOException not return null upon access error
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7352
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7352
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs, fs/s3
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>
> In HADOOP-6201 and HDFS-538 it was agreed that FileSystem::listStatus should throw FileNotFoundException instead of returning null, when the target directory did not exist.
> However, in LocalFileSystem implementation today, FileSystem::listStatus still may return null, when the target directory exists but does not grant read permission.  This causes NPE in many callers, for all the reasons cited in HADOOP-6201 and HDFS-538.  See HADOOP-7327 and its linked issues for examples.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15769-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 21:15:30 2011
Return-Path: <common-issues-return-15769-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 400864BD5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 21:15:30 +0000 (UTC)
Received: (qmail 27483 invoked by uid 500); 2 Jun 2011 21:15:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27426 invoked by uid 500); 2 Jun 2011 21:15:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Delivered-To: moderator for common-issues@hadoop.apache.org
Received: (qmail 97988 invoked by uid 99); 2 Jun 2011 20:08:40 -0000
X-ASF-Spam-Status: No, hits=3.5 required=5.0
	tests=HTML_MESSAGE,SPF_PASS,URI_HEX
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of subscription-bounces+404756@n3.nabble.com designates 216.139.236.26 as permitted sender)
Date: Thu, 2 Jun 2011 13:08:12 -0700 (PDT)
From: "gokul [via Hadoop Common]" <ml-node+3016972-1372104908-404756@n3.nabble.com>
To: common-issues <common-issues@hadoop.apache.org>
Message-ID: <1307045292541-3016972.post@n3.nabble.com>
Subject: Hadoop QuickStart Standalone Operation error
MIME-Version: 1.0
Content-Type: multipart/alternative; 
	boundary="----=_Part_23956_3812415.1307045292543"
X-Virus-Checked: Checked by ClamAV on apache.org

------=_Part_23956_3812415.1307045292543
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Dear all,

I have been trying to get a hang of Hadoop (noob/ newbie). After
installation, I followed the Quick Start guide of Hadoop to execute the
sample example given (
http://hadoop.apache.org/common/docs/r0.20.2/quickstart.html#Local
Standalone Operation ). 
After executing the third step ( $ bin/hadoop jar hadoop-*-examples.jar grep
input output 'dfs[a-z.]+' ), 

I got the error saying that

"Exception in thread "main" java.lang.NoClassDefFoundError:
Internship/hadoop-0/20/2/bin////logs
Caused by: java.lang.ClassNotFoundException:
Internship.hadoop-0.20.2.bin....logs
	at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
Could not find the main class: Internship/hadoop-0.20.2/bin/../logs. Program
will exit."

I tried a lot of different things and nothing worked. 
Some of the solutions that didn't help: 
1) Set the $JAVA_HOME path correctly (works fine)
2) Command incorrect:  $ bin/hadoop jar hadoop-*-examples.jar grep input
output 'dfs[a-z.]+' (correct as per Hadoop QuickStart)


Any form of help is very very gratefully appreciated.

Thanks in advance.



______________________________________
If you reply to this email, your message will be added to the discussion below:
http://hadoop-common.472056.n3.nabble.com/Hadoop-QuickStart-Standalone-Operation-error-tp3016972p3016972.html
This email was sent by gokul (via Nabble)
To receive all replies by email, subscribe to this discussion: http://hadoop-common.472056.n3.nabble.com/template/NamlServlet.jtp?macro=subscribe_by_code&node=3016972&code=Y29tbW9uLWlzc3Vlc0BoYWRvb3AuYXBhY2hlLm9yZ3wzMDE2OTcyfDQyNDMxNTUzOQ==
------=_Part_23956_3812415.1307045292543--

From common-issues-return-15770-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 21:20:36 2011
Return-Path: <common-issues-return-15770-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F05A843F3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 21:20:35 +0000 (UTC)
Received: (qmail 35224 invoked by uid 500); 2 Jun 2011 21:20:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35190 invoked by uid 500); 2 Jun 2011 21:20:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35096 invoked by uid 99); 2 Jun 2011 21:20:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 21:20:35 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 21:20:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A1DB8F0C31
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 21:19:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 21:19:47 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <701793763.64135.1307049587659.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <640184672.61345.1306972728331.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7350) Use ServiceLoader to discover
 compression codec classes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-7350:
------------------------------

    Attachment: HADOOP-7350.patch

{quote}
- We should probably remove the codecs from core-default.xml now that they're loaded via ServiceLoader
{quote}

Done - see new patch.

{quote}
- Is there a way to inject a new codec programatically through the ServiceLoader interface? If so, we could entirely deprecate io.compression.codecs. If not, maybe we should rename it to something like io.compression.extra.codecs and specify that it's only necessary if you have a codec that doesn't expose itself through ServiceLoader?
{quote}

I don't think we need to deprecate or rename io.compression.codecs - it's just used to specify _additional_ codecs to the ones that are loaded through a ServiceLoader. Note that duplicates are ignored, so there's no problem with users older configs having codecs that could be loaded through ServiceLoader.

{quote}
- hdfs-default.xml has an item dfs.image.compression.codec that needs to be updated
{quote}

This doesn't need to be updated, although with HADOOP-7323 (and a corresponding HDFS change) it could be changed to "default".

> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
>                 Key: HADOOP-7350
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7350
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, io
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15771-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 21:34:31 2011
Return-Path: <common-issues-return-15771-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F348B4FBF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 21:34:30 +0000 (UTC)
Received: (qmail 52658 invoked by uid 500); 2 Jun 2011 21:34:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52612 invoked by uid 500); 2 Jun 2011 21:34:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52604 invoked by uid 99); 2 Jun 2011 21:34:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 21:34:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 21:34:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6C805F10A7
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 21:33:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 21:33:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1619379070.64157.1307050427440.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <640184672.61345.1306972728331.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
 compression codec classes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043063#comment-13043063 ] 

Hadoop QA commented on HADOOP-7350:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481287/HADOOP-7350.patch
  against trunk revision 1130758.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/563//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/563//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/563//console

This message is automatically generated.

> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
>                 Key: HADOOP-7350
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7350
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, io
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15772-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 21:38:28 2011
Return-Path: <common-issues-return-15772-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B27C44097
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 21:38:28 +0000 (UTC)
Received: (qmail 56603 invoked by uid 500); 2 Jun 2011 21:38:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56578 invoked by uid 500); 2 Jun 2011 21:38:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56570 invoked by uid 99); 2 Jun 2011 21:38:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 21:38:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 21:38:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6E3CFF11F1
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 21:37:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 21:37:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <287817546.64167.1307050667448.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <640184672.61345.1306972728331.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
 compression codec classes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043066#comment-13043066 ] 

Todd Lipcon commented on HADOOP-7350:
-------------------------------------

bq. This doesn't need to be updated

Sorry, the default doesn't need to change, but the docs should be updated: it no longer needs to be one of the codecs listed in io.compression.codecs. It just needs to be any codec that's "registered" (either via conf or via ServiceLoader). I don't know the best verbiage to succinctly document it, though.

The rest of your points make sense.

> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
>                 Key: HADOOP-7350
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7350
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, io
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15773-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 21:42:28 2011
Return-Path: <common-issues-return-15773-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CE9DF4475
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 21:42:28 +0000 (UTC)
Received: (qmail 67071 invoked by uid 500); 2 Jun 2011 21:42:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67020 invoked by uid 500); 2 Jun 2011 21:42:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66983 invoked by uid 99); 2 Jun 2011 21:42:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 21:42:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 21:42:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 702CFF1313
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 21:41:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 21:41:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <975262325.64173.1307050907456.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <640184672.61345.1306972728331.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
 compression codec classes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043067#comment-13043067 ] 

Todd Lipcon commented on HADOOP-7350:
-------------------------------------

+1 on most recent patch

> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
>                 Key: HADOOP-7350
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7350
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, io
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15774-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 21:44:29 2011
Return-Path: <common-issues-return-15774-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2E26A479B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 21:44:29 +0000 (UTC)
Received: (qmail 73567 invoked by uid 500); 2 Jun 2011 21:44:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73531 invoked by uid 500); 2 Jun 2011 21:44:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73523 invoked by uid 99); 2 Jun 2011 21:44:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 21:44:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 21:44:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D52A2F149C
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 21:43:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 21:43:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <145690903.64181.1307051027870.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6461) webapps aren't located correctly
 post-split
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043069#comment-13043069 ] 

Hudson commented on HADOOP-6461:
--------------------------------

Integrated in Hadoop-Common-22-branch #61 (See [https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/61/])
    HADOOP-7351 Regression: HttpServer#getWebAppsPath used to be protected so subclasses could supply alternate webapps path but it was made private by HADOOP-6461

stack : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130740
Files : 
* /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/http/HttpServer.java
* /hadoop/common/branches/branch-0.22/CHANGES.txt


> webapps aren't located correctly post-split
> -------------------------------------------
>
>                 Key: HADOOP-6461
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6461
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Steve Loughran
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6461.patch, HADOOP-6461.patch, HADOOP-6461.patch, HADOOP-6461.patch, HADOOP-6461.txt, hadoop-6461.patch, hadoop-6461.patch, hadoop-6461.patch, hadoop-6461.txt, hadoop-6461.txt
>
>
> Post-split, when one builds common it creates an empty build/webapps dir. If you then try to launch the NN for example using HADOOP_HDFS_HOME=/path/to/hdfs hdfs namenode, HttpServer.getWebAppsPath locates the empty common webapps dir, and the NN web UI fails to load.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15775-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 21:44:29 2011
Return-Path: <common-issues-return-15775-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6498A47A3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 21:44:29 +0000 (UTC)
Received: (qmail 73839 invoked by uid 500); 2 Jun 2011 21:44:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73773 invoked by uid 500); 2 Jun 2011 21:44:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73561 invoked by uid 99); 2 Jun 2011 21:44:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 21:44:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 21:44:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E2E69F149E
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 21:43:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 21:43:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <286046074.64183.1307051027926.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <3255382.248681296161204958.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7121) Exceptions while serializing IPC
 call response are not handled well
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043070#comment-13043070 ] 

Hudson commented on HADOOP-7121:
--------------------------------

Integrated in Hadoop-Common-22-branch #61 (See [https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/61/])
    HADOOP-7349. HADOOP-7121 accidentally disabled some tests in TestIPC. Contributed by Todd Lipcon.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130757
Files : 
* /hadoop/common/branches/branch-0.22/src/test/core/org/apache/hadoop/ipc/TestIPC.java
* /hadoop/common/branches/branch-0.22/CHANGES.txt


> Exceptions while serializing IPC call response are not handled well
> -------------------------------------------------------------------
>
>                 Key: HADOOP-7121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7121
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>              Labels: newbie
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7121.txt, hadoop-7121.txt, hadoop-7121.txt
>
>
> We had a situation where for some reason the serialization of an RPC call's response was throwing OOME. When this happens, the exception is not caught, and the call never gets a response - the client just hangs. Additionally, the OOME propagated all the way to the top of the IPC handler and caused the handler. Plus, the Handler upon exit only logged to stdout and not to the log4j logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15776-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 21:44:29 2011
Return-Path: <common-issues-return-15776-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9C8E147B1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 21:44:29 +0000 (UTC)
Received: (qmail 73943 invoked by uid 500); 2 Jun 2011 21:44:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73889 invoked by uid 500); 2 Jun 2011 21:44:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73753 invoked by uid 99); 2 Jun 2011 21:44:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 21:44:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 21:44:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 26851F14A6
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 21:43:48 +0000 (UTC)
Date: Thu, 2 Jun 2011 21:43:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1655689487.64190.1307051028154.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <620892300.63733.1307038547598.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7355) Add audience and stability
 annotations to HttpServer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043071#comment-13043071 ] 

Hudson commented on HADOOP-7355:
--------------------------------

Integrated in Hadoop-Common-22-branch #61 (See [https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/61/])
    HADOOP-7355 Add audience and stability annotations to HttpServer class

stack : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130741
Files : 
* /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/http/HttpServer.java
* /hadoop/common/branches/branch-0.22/CHANGES.txt


> Add audience and stability annotations to HttpServer class
> ----------------------------------------------------------
>
>                 Key: HADOOP-7355
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7355
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: stack
>            Assignee: stack
>             Fix For: 0.22.0
>
>         Attachments: 7355-v2.txt, 7355.txt
>
>
> HttpServer has at least one subclasser in HBase.  Flag this class w/ annotations that make this plain so we avoid regressions like HADOOP-7351

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15777-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 21:44:31 2011
Return-Path: <common-issues-return-15777-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ECA2947C8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 21:44:31 +0000 (UTC)
Received: (qmail 74402 invoked by uid 500); 2 Jun 2011 21:44:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74345 invoked by uid 500); 2 Jun 2011 21:44:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74328 invoked by uid 99); 2 Jun 2011 21:44:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 21:44:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 21:44:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 508C7F14AA
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 21:43:48 +0000 (UTC)
Date: Thu, 2 Jun 2011 21:43:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1367556399.64194.1307051028326.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <99930859.62060.1306994747296.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7351) Regression:
 HttpServer#getWebAppsPath used to be protected so subclasses could supply
 alternate webapps path but it was made private by HADOOP-6461
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043072#comment-13043072 ] 

Hudson commented on HADOOP-7351:
--------------------------------

Integrated in Hadoop-Common-22-branch #61 (See [https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/61/])
    HADOOP-7351 Regression: HttpServer#getWebAppsPath used to be protected so subclasses could supply alternate webapps path but it was made private by HADOOP-6461

stack : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130740
Files : 
* /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/http/HttpServer.java
* /hadoop/common/branches/branch-0.22/CHANGES.txt


> Regression: HttpServer#getWebAppsPath used to be protected so subclasses could supply alternate webapps path but it was made private by HADOOP-6461
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7351
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7351
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>             Fix For: 0.22.0
>
>         Attachments: 7351.txt
>
>
> It USED to be protected rather than private but its access was changed by HADOOP-6461.  It did the following:
> {code}
> -  protected String getWebAppsPath() throws IOException {
> -    URL url = getClass().getClassLoader().getResource("webapps");
> +  private String getWebAppsPath(String appName) throws FileNotFoundException {
> +    URL url = getClass().getClassLoader().getResource("webapps/" + appName);
> ...
> {code}
> HBase subclasses HttpServer providing its UI.  This change makes it so we can no longer do so.
> This change made it into 0.21.  I'd like to get a fix committed to 0.22 as well as TRUNK.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15778-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 21:44:33 2011
Return-Path: <common-issues-return-15778-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 12FDE47DE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 21:44:33 +0000 (UTC)
Received: (qmail 74662 invoked by uid 500); 2 Jun 2011 21:44:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74626 invoked by uid 500); 2 Jun 2011 21:44:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74618 invoked by uid 99); 2 Jun 2011 21:44:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 21:44:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 21:44:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9F908F1496
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 21:43:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 21:43:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2135754297.64175.1307051027650.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1016265931.61342.1306972667438.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7349) HADOOP-7121 accidentally disabled
 some tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043068#comment-13043068 ] 

Hudson commented on HADOOP-7349:
--------------------------------

Integrated in Hadoop-Common-22-branch #61 (See [https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/61/])
    HADOOP-7349. HADOOP-7121 accidentally disabled some tests in TestIPC. Contributed by Todd Lipcon.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130757
Files : 
* /hadoop/common/branches/branch-0.22/src/test/core/org/apache/hadoop/ipc/TestIPC.java
* /hadoop/common/branches/branch-0.22/CHANGES.txt


> HADOOP-7121 accidentally disabled some tests
> --------------------------------------------
>
>                 Key: HADOOP-7349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7349
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7349.txt
>
>
> When I converted TestIPC to JUnit 4, I missed a couple of tests towards the bottom of the file when adding the @Test annotation.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15779-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 21:44:52 2011
Return-Path: <common-issues-return-15779-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3146B4893
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 21:44:52 +0000 (UTC)
Received: (qmail 76369 invoked by uid 500); 2 Jun 2011 21:44:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76336 invoked by uid 500); 2 Jun 2011 21:44:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76328 invoked by uid 99); 2 Jun 2011 21:44:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 21:44:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 21:44:49 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 967F3F14B1
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 21:43:48 +0000 (UTC)
Date: Thu, 2 Jun 2011 21:43:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2024260301.64201.1307051028613.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <504299282.44039.1306374047470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7331) Makes hadoop-daemon.sh to return 1
 if daemon processes did not get started
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7331:
--------------------------------

      Resolution: Fixed
    Release Note: hadoop-daemon.sh now returns a non-zero exit code if it detects that the daemon was not still running after 3 seconds.
    Hadoop Flags: [Incompatible change, Reviewed]
          Status: Resolved  (was: Patch Available)

Committed this to trunk. Thanks, Tanping!

I marked this as an incompatible change and added a release note just in case anyone was relying on the previously broken behavior.

> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7331
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7331
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7331.2.patch, HADOOP-7331.patch
>
>
> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15780-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 21:46:29 2011
Return-Path: <common-issues-return-15780-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BFEEE44BE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 21:46:29 +0000 (UTC)
Received: (qmail 79149 invoked by uid 500); 2 Jun 2011 21:46:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79110 invoked by uid 500); 2 Jun 2011 21:46:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79102 invoked by uid 99); 2 Jun 2011 21:46:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 21:46:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 21:46:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 62050F1633
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 21:45:48 +0000 (UTC)
Date: Thu, 2 Jun 2011 21:45:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <670428312.64210.1307051148398.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <504299282.44039.1306374047470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7331) Makes hadoop-daemon.sh to return 1
 if daemon processes did not get started
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043076#comment-13043076 ] 

Todd Lipcon commented on HADOOP-7331:
-------------------------------------

(should have mentioned, I realized Steve is on vacation at the moment. Since Tanping had addressed his comment, I decided to commit - if there's any issue we can revert or follow up later with another patch)

> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7331
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7331
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7331.2.patch, HADOOP-7331.patch
>
>
> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15781-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 21:48:31 2011
Return-Path: <common-issues-return-15781-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 760404FB9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 21:48:31 +0000 (UTC)
Received: (qmail 85683 invoked by uid 500); 2 Jun 2011 21:48:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85629 invoked by uid 500); 2 Jun 2011 21:48:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85621 invoked by uid 99); 2 Jun 2011 21:48:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 21:48:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 21:48:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 12645F178D
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 21:47:48 +0000 (UTC)
Date: Thu, 2 Jun 2011 21:47:48 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1047433099.64245.1307051268072.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <654871262.37888.1306191347393.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7323) Add capability to resolve
 compression codec based on codec name
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-7323:
---------------------------------------

    Attachment: HADOOP-7323.patch

Tom,

Thank for your feedback.

Attached is patch that includes your suggestions.

> Add capability to resolve compression codec based on codec name
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7323
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7323
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7323.patch, HADOOP-7323.patch, HADOOP-7323b.patch
>
>
> When setting up a compression codec in an MR job the full class name of the codec must be used.
> To ease usability, compression codecs should be resolved by their codec name (ie 'gzip', 'deflate', 'zlib', 'bzip2') instead their full codec class name.
> Besides easy of use for Hadoop users who would use the codec alias instead the full codec class name, it could simplify how HBase resolves loads the codecs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15782-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 21:58:29 2011
Return-Path: <common-issues-return-15782-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0B36641CE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 21:58:29 +0000 (UTC)
Received: (qmail 95467 invoked by uid 500); 2 Jun 2011 21:58:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95386 invoked by uid 500); 2 Jun 2011 21:58:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95378 invoked by uid 99); 2 Jun 2011 21:58:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 21:58:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 21:58:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6084DF1A0A
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 21:57:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 21:57:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <400248794.64253.1307051867391.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <654871262.37888.1306191347393.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7323) Add capability to resolve
 compression codec based on codec name
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043083#comment-13043083 ] 

Hadoop QA commented on HADOOP-7323:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481292/HADOOP-7323.patch
  against trunk revision 1130833.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The patch appears to cause tar ant target to fail.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:


    -1 system test framework.  The patch failed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/564//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/564//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/564//console

This message is automatically generated.

> Add capability to resolve compression codec based on codec name
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7323
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7323
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7323.patch, HADOOP-7323.patch, HADOOP-7323b.patch
>
>
> When setting up a compression codec in an MR job the full class name of the codec must be used.
> To ease usability, compression codecs should be resolved by their codec name (ie 'gzip', 'deflate', 'zlib', 'bzip2') instead their full codec class name.
> Besides easy of use for Hadoop users who would use the codec alias instead the full codec class name, it could simplify how HBase resolves loads the codecs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15783-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 22:06:31 2011
Return-Path: <common-issues-return-15783-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EBD1E49E9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 22:06:30 +0000 (UTC)
Received: (qmail 6074 invoked by uid 500); 2 Jun 2011 22:06:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6041 invoked by uid 500); 2 Jun 2011 22:06:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6033 invoked by uid 99); 2 Jun 2011 22:06:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 22:06:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 22:06:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 94692F15A5
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 22:05:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 22:05:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <820581336.64285.1307052347603.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <504299282.44039.1306374047470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7331) Makes hadoop-daemon.sh to return 1
 if daemon processes did not get started
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043087#comment-13043087 ] 

Hudson commented on HADOOP-7331:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #636 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/636/])
    HADOOP-7331. Make hadoop-daemon.sh return exit code 1 if daemon processes did not get started. Contributed by Tanping Wang.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130833
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/bin/hadoop-daemon.sh


> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7331
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7331
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7331.2.patch, HADOOP-7331.patch
>
>
> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15784-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 23:16:29 2011
Return-Path: <common-issues-return-15784-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 380694985
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 23:16:29 +0000 (UTC)
Received: (qmail 89969 invoked by uid 500); 2 Jun 2011 23:16:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89942 invoked by uid 500); 2 Jun 2011 23:16:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89933 invoked by uid 99); 2 Jun 2011 23:16:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 23:16:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 23:16:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id ED3C1F1B90
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 23:15:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 23:15:47 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <134648733.64561.1307056547968.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <654871262.37888.1306191347393.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7323) Add capability to resolve
 compression codec based on codec name
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-7323:
---------------------------------------

    Status: Open  (was: Patch Available)

> Add capability to resolve compression codec based on codec name
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7323
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7323
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7323.patch, HADOOP-7323.patch, HADOOP-7323.patch, HADOOP-7323b.patch
>
>
> When setting up a compression codec in an MR job the full class name of the codec must be used.
> To ease usability, compression codecs should be resolved by their codec name (ie 'gzip', 'deflate', 'zlib', 'bzip2') instead their full codec class name.
> Besides easy of use for Hadoop users who would use the codec alias instead the full codec class name, it could simplify how HBase resolves loads the codecs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15785-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 23:16:31 2011
Return-Path: <common-issues-return-15785-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C327749C0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 23:16:31 +0000 (UTC)
Received: (qmail 90227 invoked by uid 500); 2 Jun 2011 23:16:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90201 invoked by uid 500); 2 Jun 2011 23:16:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90193 invoked by uid 99); 2 Jun 2011 23:16:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 23:16:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 23:16:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0AB0CF1B92
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 23:15:48 +0000 (UTC)
Date: Thu, 2 Jun 2011 23:15:48 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1106168305.64563.1307056548040.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <654871262.37888.1306191347393.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7323) Add capability to resolve
 compression codec based on codec name
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-7323:
---------------------------------------

    Attachment: HADOOP-7323.patch

And now a patch with all files :)

> Add capability to resolve compression codec based on codec name
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7323
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7323
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7323.patch, HADOOP-7323.patch, HADOOP-7323.patch, HADOOP-7323b.patch
>
>
> When setting up a compression codec in an MR job the full class name of the codec must be used.
> To ease usability, compression codecs should be resolved by their codec name (ie 'gzip', 'deflate', 'zlib', 'bzip2') instead their full codec class name.
> Besides easy of use for Hadoop users who would use the codec alias instead the full codec class name, it could simplify how HBase resolves loads the codecs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15786-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 23:28:30 2011
Return-Path: <common-issues-return-15786-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0DF7F4F59
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 23:28:30 +0000 (UTC)
Received: (qmail 5981 invoked by uid 500); 2 Jun 2011 23:28:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5811 invoked by uid 500); 2 Jun 2011 23:28:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5617 invoked by uid 99); 2 Jun 2011 23:28:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 23:28:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 23:28:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 618C6F1E53
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 23:27:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 23:27:47 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1043064090.64591.1307057267396.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7356) RPM packages broke bin/hadoop script
 for hadoop 0.20.205
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

RPM packages broke bin/hadoop script for hadoop 0.20.205
--------------------------------------------------------

                 Key: HADOOP-7356
                 URL: https://issues.apache.org/jira/browse/HADOOP-7356
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.20.204.0
         Environment: Java 6, Redhat EL 5.5
            Reporter: Eric Yang
            Assignee: Eric Yang


hadoop-config.sh has been moved to libexec for binary package, but developers prefers to have hadoop-config.sh in bin.  Hadoo shell scripts should be modified to support both scenarios.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15787-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 23:32:29 2011
Return-Path: <common-issues-return-15787-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 20B914BFA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 23:32:29 +0000 (UTC)
Received: (qmail 7705 invoked by uid 500); 2 Jun 2011 23:32:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7668 invoked by uid 500); 2 Jun 2011 23:32:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7660 invoked by uid 99); 2 Jun 2011 23:32:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 23:32:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 23:32:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 57C4EF1F20
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 23:31:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 23:31:47 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1880834417.64593.1307057507355.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <640184672.61345.1306972728331.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7350) Use ServiceLoader to discover
 compression codec classes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-7350:
------------------------------

    Attachment: HADOOP-7350.patch

Slight adjustment to load codec classes only once using a ServiceLoader.

I'll address the HDFS documentation change in another JIRA.

> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
>                 Key: HADOOP-7350
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7350
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, io
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15788-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 23:50:28 2011
Return-Path: <common-issues-return-15788-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C35874D44
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 23:50:28 +0000 (UTC)
Received: (qmail 21016 invoked by uid 500); 2 Jun 2011 23:50:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20988 invoked by uid 500); 2 Jun 2011 23:50:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20980 invoked by uid 99); 2 Jun 2011 23:50:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 23:50:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 23:50:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 71862F1453
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 23:49:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 23:49:47 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1918384273.64620.1307058587461.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1043064090.64591.1307057267396.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7356) RPM packages broke bin/hadoop script
 for hadoop 0.20.205
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-7356:
------------------------------

    Attachment: HADOOP-7356.patch

Preserve hadoop-config.sh and HADOOP_HOME locations for developers.  This only applies to 0.20.204 or 0.20.205 branches.

> RPM packages broke bin/hadoop script for hadoop 0.20.205
> --------------------------------------------------------
>
>                 Key: HADOOP-7356
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7356
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.204.0
>         Environment: Java 6, Redhat EL 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>         Attachments: HADOOP-7356.patch
>
>
> hadoop-config.sh has been moved to libexec for binary package, but developers prefers to have hadoop-config.sh in bin.  Hadoo shell scripts should be modified to support both scenarios.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15789-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 23:50:29 2011
Return-Path: <common-issues-return-15789-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 12ECB4D5A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 23:50:29 +0000 (UTC)
Received: (qmail 21180 invoked by uid 500); 2 Jun 2011 23:50:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21129 invoked by uid 500); 2 Jun 2011 23:50:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21081 invoked by uid 99); 2 Jun 2011 23:50:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 23:50:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 23:50:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 90798F1456
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 23:49:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 23:49:47 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1490460127.64623.1307058587586.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1043064090.64591.1307057267396.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7356) RPM packages broke bin/hadoop script
 for hadoop 0.20.205
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-7356:
------------------------------

    Status: Patch Available  (was: Open)

> RPM packages broke bin/hadoop script for hadoop 0.20.205
> --------------------------------------------------------
>
>                 Key: HADOOP-7356
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7356
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.204.0
>         Environment: Java 6, Redhat EL 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>         Attachments: HADOOP-7356.patch
>
>
> hadoop-config.sh has been moved to libexec for binary package, but developers prefers to have hadoop-config.sh in bin.  Hadoo shell scripts should be modified to support both scenarios.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15790-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 23:54:28 2011
Return-Path: <common-issues-return-15790-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BDC6A4D8C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 23:54:28 +0000 (UTC)
Received: (qmail 22465 invoked by uid 500); 2 Jun 2011 23:54:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22434 invoked by uid 500); 2 Jun 2011 23:54:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22426 invoked by uid 99); 2 Jun 2011 23:54:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 23:54:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 23:54:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 72C35F156C
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 23:53:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 23:53:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1934380502.64629.1307058827467.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <640184672.61345.1306972728331.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
 compression codec classes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043143#comment-13043143 ] 

Hadoop QA commented on HADOOP-7350:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481304/HADOOP-7350.patch
  against trunk revision 1130833.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/565//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/565//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/565//console

This message is automatically generated.

> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
>                 Key: HADOOP-7350
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7350
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, io
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15791-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  2 23:54:30 2011
Return-Path: <common-issues-return-15791-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B9E304DA6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  2 Jun 2011 23:54:30 +0000 (UTC)
Received: (qmail 22757 invoked by uid 500); 2 Jun 2011 23:54:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22716 invoked by uid 500); 2 Jun 2011 23:54:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22708 invoked by uid 99); 2 Jun 2011 23:54:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 23:54:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2011 23:54:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 65917F156A
	for <common-issues@hadoop.apache.org>; Thu,  2 Jun 2011 23:53:47 +0000 (UTC)
Date: Thu, 2 Jun 2011 23:53:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <634015767.64627.1307058827412.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1043064090.64591.1307057267396.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7356) RPM packages broke bin/hadoop
 script for hadoop 0.20.205
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043142#comment-13043142 ] 

Hadoop QA commented on HADOOP-7356:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481305/HADOOP-7356.patch
  against trunk revision 1130833.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/566//console

This message is automatically generated.

> RPM packages broke bin/hadoop script for hadoop 0.20.205
> --------------------------------------------------------
>
>                 Key: HADOOP-7356
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7356
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.204.0
>         Environment: Java 6, Redhat EL 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>         Attachments: HADOOP-7356.patch
>
>
> hadoop-config.sh has been moved to libexec for binary package, but developers prefers to have hadoop-config.sh in bin.  Hadoo shell scripts should be modified to support both scenarios.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15792-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 01:07:28 2011
Return-Path: <common-issues-return-15792-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BE23E4C0A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 01:07:28 +0000 (UTC)
Received: (qmail 84890 invoked by uid 500); 3 Jun 2011 01:07:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84866 invoked by uid 500); 3 Jun 2011 01:07:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84858 invoked by uid 99); 3 Jun 2011 01:07:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 01:07:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 01:07:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 75A26F1536
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 01:06:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 01:06:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1207205763.64768.1307063207478.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <958686885.62073.1306995407750.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7352) Contracts of LocalFileSystem and
 DistributedFileSystem should require FileSystem::listStatus throw
 IOException not return null upon access error
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043163#comment-13043163 ] 

Matt Foley commented on HADOOP-7352:
------------------------------------

@Daryn - thanks, I'll use that where appropriate.

@Bharath - agreed, the exception will always be IOException or one of its subclasses; IOException where there is no detailed info available or no better subclass defined.

> Contracts of LocalFileSystem and DistributedFileSystem should require FileSystem::listStatus throw IOException not return null upon access error
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7352
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7352
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs, fs/s3
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>
> In HADOOP-6201 and HDFS-538 it was agreed that FileSystem::listStatus should throw FileNotFoundException instead of returning null, when the target directory did not exist.
> However, in LocalFileSystem implementation today, FileSystem::listStatus still may return null, when the target directory exists but does not grant read permission.  This causes NPE in many callers, for all the reasons cited in HADOOP-6201 and HDFS-538.  See HADOOP-7327 and its linked issues for examples.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15793-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 01:13:29 2011
Return-Path: <common-issues-return-15793-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F28704E15
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 01:13:28 +0000 (UTC)
Received: (qmail 91784 invoked by uid 500); 3 Jun 2011 01:13:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91757 invoked by uid 500); 3 Jun 2011 01:13:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91749 invoked by uid 99); 3 Jun 2011 01:13:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 01:13:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 01:13:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8E647F16BB
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 01:12:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 01:12:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1359596915.64782.1307063567579.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1470673003.56543.1306860527397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7341) Fix option parsing in
 CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043166#comment-13043166 ] 

Daryn Sharp commented on HADOOP-7341:
-------------------------------------

Are there any remaining issues I need to address?

> Fix option parsing in CommandFormat
> -----------------------------------
>
>                 Key: HADOOP-7341
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7341
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7341-2.patch, HADOOP-7341.patch
>
>
> CommandFormat currently allows options in any location within the args.  This is not the intended behavior for FsShell commands.  Prior to the redesign, the commands used to expect option processing to stop at the first non-option.
> CommandFormat was an existing class prior the redesign, but it was only used by "count" to find the -q flag.  All commands were converted to using this class, thus inherited the unintended behavior.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15794-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 06:01:32 2011
Return-Path: <common-issues-return-15794-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A3FCF61A5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 06:01:32 +0000 (UTC)
Received: (qmail 23421 invoked by uid 500); 3 Jun 2011 06:01:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23398 invoked by uid 500); 3 Jun 2011 06:01:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23385 invoked by uid 99); 3 Jun 2011 06:01:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 06:01:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 06:01:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 63485F1632
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 06:00:49 +0000 (UTC)
Date: Fri, 3 Jun 2011 06:00:49 +0000 (UTC)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <72681940.65118.1307080849403.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <397648942.57126.1306869588398.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7342) Add an utility API in FileUtil for
 JDK File.list
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043222#comment-13043222 ] 

Konstantin Boudnik commented on HADOOP-7342:
--------------------------------------------

Matt, according to [Jacob's comment|https://issues.apache.org/jira/browse/HDFS-1934?focusedCommentId=13039888&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13039888] this API chance has been already discussed and I have missed that part. So, taking it back: let's go with this (although it seems unusual ;)

> Add an utility API in FileUtil for JDK File.list
> ------------------------------------------------
>
>                 Key: HADOOP-7342
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7342
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7342-1.patch
>
>
> Java File.list API can return null when disk is bad or directory is not a directory. This utility API in FileUtil will throw an exception when this happens rather than returning null. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15795-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 06:01:33 2011
Return-Path: <common-issues-return-15795-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6DFC461C4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 06:01:33 +0000 (UTC)
Received: (qmail 23698 invoked by uid 500); 3 Jun 2011 06:01:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23662 invoked by uid 500); 3 Jun 2011 06:01:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23653 invoked by uid 99); 3 Jun 2011 06:01:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 06:01:33 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 06:01:30 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DD3BDF1643
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 06:00:49 +0000 (UTC)
Date: Fri, 3 Jun 2011 06:00:49 +0000 (UTC)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <919591549.65131.1307080849902.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <397648942.57126.1306869588398.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7342) Add an utility API in FileUtil for
 JDK File.list
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043223#comment-13043223 ] 

Konstantin Boudnik commented on HADOOP-7342:
--------------------------------------------

bq. This will be clear from the logs that assertEquals failed. 
Bharath, it will be apparent that it failed, but won't clear why without looking into the test's code.

> Add an utility API in FileUtil for JDK File.list
> ------------------------------------------------
>
>                 Key: HADOOP-7342
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7342
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7342-1.patch
>
>
> Java File.list API can return null when disk is bad or directory is not a directory. This utility API in FileUtil will throw an exception when this happens rather than returning null. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15796-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 06:05:37 2011
Return-Path: <common-issues-return-15796-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BF7956F64
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 06:05:37 +0000 (UTC)
Received: (qmail 27178 invoked by uid 500); 3 Jun 2011 06:05:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27075 invoked by uid 500); 3 Jun 2011 06:05:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27029 invoked by uid 99); 3 Jun 2011 06:05:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 06:05:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 06:05:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 59EE8F1743
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 06:04:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 06:04:47 +0000 (UTC)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1471056846.65135.1307081087364.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <647927887.58055.1306890116067.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7346) Send back nicer error to clients
 using outdated IPC version
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043224#comment-13043224 ] 

Konstantin Boudnik commented on HADOOP-7346:
--------------------------------------------

Creative ;) Thanks. 
There's a couple of white-space changes in the latest patch. 
+1 otherwise.

> Send back nicer error to clients using outdated IPC version
> -----------------------------------------------------------
>
>                 Key: HADOOP-7346
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7346
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7346.txt, hadoop-7346.txt
>
>
> When an older Hadoop version tries to contact a newer Hadoop version across an IPC protocol version bump, the client currently just gets a non-useful error message like "EOFException".
> Instead, the IPC server code can speak just enough of prior IPC protocols to send back a "fatal" message indicating the version mismatch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15797-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 08:22:30 2011
Return-Path: <common-issues-return-15797-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3568162DF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 08:22:30 +0000 (UTC)
Received: (qmail 69294 invoked by uid 500); 3 Jun 2011 08:22:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69266 invoked by uid 500); 3 Jun 2011 08:22:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69258 invoked by uid 99); 3 Jun 2011 08:22:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 08:22:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 08:22:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8D426F2DB9
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 08:21:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 08:21:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <702637021.65228.1307089307575.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <164637914.38616.1306225908771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7327) FileSystem.listStatus() throws
 NullPointerException instead of IOException upon access permission failure
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043247#comment-13043247 ] 

Matt Foley commented on HADOOP-7327:
------------------------------------

We have two instances of this bug actually causing NPEs in the real world (or at least the Test world).

The first is in TestTrash.testTrashEmptier(), which runs on the LocalFileSystem, and launches a Trash$Emptier thread in background.  Every time that thread cycles, it attempts to empty the local .Trash directory in EVERY "home directory" known.  On my Mac, there are four user accounts, each with its own "home" directory, one of which, /Users/test, has a .Trash directory with access permissions set so it is not readable by me.  When it tries to list the .Trash subdirectories with FileSystem.listStatus(), it logs the following:

11/05/23 14:34:54 WARN fs.Trash: RuntimeException during Trash.Emptier.run() java.lang.NullPointerException
	at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1114)
	at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1137)
	at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:494)
	at org.apache.hadoop.fs.Trash.expunge(Trash.java:215)
	at org.apache.hadoop.fs.Trash$Emptier.run(Trash.java:313)
	at java.lang.Thread.run(Thread.java:680)

The second example is from Daryn, and was partly described in HADOOP-7345.  When he tries to use "bin/hadoop fs -ls aaa" on any directory "aaa" for which he does not have access privs, it logs:

java.lang.NullPointerException
    at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1115)
    at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1150)
    at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:494)
    at org.apache.hadoop.fs.shell.PathData.getDirectoryContents(PathData.java:163)

It is seen that both of these cases go through the FileSystem.listStatus(ArrayList<FileStatus>, Path, PathFilter) form of the call, and it is here that the attached patch is targeted.


> FileSystem.listStatus() throws NullPointerException instead of IOException upon access permission failure
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7327
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7327
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>         Attachments: hadoop-7327-1.patch
>
>
> Many processes that call listStatus() expect to handle IOException, but instead are getting runtime error NullPointerException, if the directory being scanned is visible but no-access to the running user id.  For example, if directory foo is drwxr-xr-x, and subdirectory foo/bar is drwx------, then trying to do listStatus(Path(foo/bar)) will cause a NullPointerException.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15798-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 08:34:29 2011
Return-Path: <common-issues-return-15798-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 22E8C6910
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 08:34:29 +0000 (UTC)
Received: (qmail 82631 invoked by uid 500); 3 Jun 2011 08:34:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82602 invoked by uid 500); 3 Jun 2011 08:34:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82593 invoked by uid 99); 3 Jun 2011 08:34:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 08:34:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 08:34:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A654BF212E
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 08:33:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 08:33:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1166556808.65239.1307090027677.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <164637914.38616.1306225908771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7327) FileSystem.listStatus() throws
 NullPointerException instead of IOException upon access permission failure
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043250#comment-13043250 ] 

Matt Foley commented on HADOOP-7327:
------------------------------------

Although this issue was discovered while investigating HDFS-1852, it does not actually block it, so removing the "blocker" link.  It doesn't cause TestTrash to fail, it just junks up the log with NPEs while running TestTrash on a local system.

> FileSystem.listStatus() throws NullPointerException instead of IOException upon access permission failure
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7327
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7327
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>         Attachments: hadoop-7327-1.patch
>
>
> Many processes that call listStatus() expect to handle IOException, but instead are getting runtime error NullPointerException, if the directory being scanned is visible but no-access to the running user id.  For example, if directory foo is drwxr-xr-x, and subdirectory foo/bar is drwx------, then trying to do listStatus(Path(foo/bar)) will cause a NullPointerException.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15799-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 08:34:31 2011
Return-Path: <common-issues-return-15799-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 37BA06943
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 08:34:31 +0000 (UTC)
Received: (qmail 82883 invoked by uid 500); 3 Jun 2011 08:34:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82858 invoked by uid 500); 3 Jun 2011 08:34:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82850 invoked by uid 99); 3 Jun 2011 08:34:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 08:34:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 08:34:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B942DF2130
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 08:33:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 08:33:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <574821798.65241.1307090027755.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <164637914.38616.1306225908771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7327) FileSystem.listStatus() throws
 NullPointerException instead of IOException upon access permission failure
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated HADOOP-7327:
-------------------------------

    Priority: Minor  (was: Major)

> FileSystem.listStatus() throws NullPointerException instead of IOException upon access permission failure
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7327
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7327
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>            Priority: Minor
>         Attachments: hadoop-7327-1.patch
>
>
> Many processes that call listStatus() expect to handle IOException, but instead are getting runtime error NullPointerException, if the directory being scanned is visible but no-access to the running user id.  For example, if directory foo is drwxr-xr-x, and subdirectory foo/bar is drwx------, then trying to do listStatus(Path(foo/bar)) will cause a NullPointerException.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15800-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 11:16:29 2011
Return-Path: <common-issues-return-15800-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 31B70621A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 11:16:29 +0000 (UTC)
Received: (qmail 19814 invoked by uid 500); 3 Jun 2011 11:16:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19794 invoked by uid 500); 3 Jun 2011 11:16:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19786 invoked by uid 99); 3 Jun 2011 11:16:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 11:16:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 11:16:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A1C32F243C
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 11:15:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 11:15:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <655346790.65454.1307099747659.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <504299282.44039.1306374047470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7331) Makes hadoop-daemon.sh to return 1
 if daemon processes did not get started
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043299#comment-13043299 ] 

Hudson commented on HADOOP-7331:
--------------------------------

Integrated in Hadoop-Common-trunk #708 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/708/])
    HADOOP-7331. Make hadoop-daemon.sh return exit code 1 if daemon processes did not get started. Contributed by Tanping Wang.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130833
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/bin/hadoop-daemon.sh


> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7331
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7331
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7331.2.patch, HADOOP-7331.patch
>
>
> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15802-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 11:16:32 2011
Return-Path: <common-issues-return-15802-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E8109624A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 11:16:29 +0000 (UTC)
Received: (qmail 20335 invoked by uid 500); 3 Jun 2011 11:16:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20293 invoked by uid 500); 3 Jun 2011 11:16:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20280 invoked by uid 99); 3 Jun 2011 11:16:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 11:16:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 11:16:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4C652F244D
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 11:15:48 +0000 (UTC)
Date: Fri, 3 Jun 2011 11:15:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <999649012.65471.1307099748309.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <620892300.63733.1307038547598.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7355) Add audience and stability
 annotations to HttpServer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043303#comment-13043303 ] 

Hudson commented on HADOOP-7355:
--------------------------------

Integrated in Hadoop-Common-trunk #708 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/708/])
    HADOOP-7355 Add audience and stability annotations to HttpServer class

stack : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130729
Files : 
* /hadoop/common/trunk/src/java/org/apache/hadoop/http/HttpServer.java
* /hadoop/common/trunk/CHANGES.txt


> Add audience and stability annotations to HttpServer class
> ----------------------------------------------------------
>
>                 Key: HADOOP-7355
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7355
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: stack
>            Assignee: stack
>             Fix For: 0.22.0
>
>         Attachments: 7355-v2.txt, 7355.txt
>
>
> HttpServer has at least one subclasser in HBase.  Flag this class w/ annotations that make this plain so we avoid regressions like HADOOP-7351

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15803-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 11:16:32 2011
Return-Path: <common-issues-return-15803-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5A93C627B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 11:16:31 +0000 (UTC)
Received: (qmail 20838 invoked by uid 500); 3 Jun 2011 11:16:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20797 invoked by uid 500); 3 Jun 2011 11:16:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20789 invoked by uid 99); 3 Jun 2011 11:16:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 11:16:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 11:16:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C2471F2440
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 11:15:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 11:15:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1479850343.65458.1307099747792.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6461) webapps aren't located correctly
 post-split
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043301#comment-13043301 ] 

Hudson commented on HADOOP-6461:
--------------------------------

Integrated in Hadoop-Common-trunk #708 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/708/])
    HADOOP-7351 Regression: HttpServer#getWebAppsPath used to be protected so subclasses could supply alternate webapps path but it was made private by HADOOP-6461 -- Fix commit message; I'd put it in wrong location under 'new features' rather than under 'bugs'
HADOOP-7351 Regression: HttpServer#getWebAppsPath used to be protected so subclasses could supply alternate webapps path but it was made private by HADOOP-6461

stack : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130738
Files : 
* /hadoop/common/trunk/CHANGES.txt

stack : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130730
Files : 
* /hadoop/common/trunk/src/java/org/apache/hadoop/http/HttpServer.java
* /hadoop/common/trunk/CHANGES.txt


> webapps aren't located correctly post-split
> -------------------------------------------
>
>                 Key: HADOOP-6461
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6461
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Steve Loughran
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6461.patch, HADOOP-6461.patch, HADOOP-6461.patch, HADOOP-6461.patch, HADOOP-6461.txt, hadoop-6461.patch, hadoop-6461.patch, hadoop-6461.patch, hadoop-6461.txt, hadoop-6461.txt
>
>
> Post-split, when one builds common it creates an empty build/webapps dir. If you then try to launch the NN for example using HADOOP_HDFS_HOME=/path/to/hdfs hdfs namenode, HttpServer.getWebAppsPath locates the empty common webapps dir, and the NN web UI fails to load.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15804-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 11:16:32 2011
Return-Path: <common-issues-return-15804-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A3D956289
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 11:16:31 +0000 (UTC)
Received: (qmail 21068 invoked by uid 500); 3 Jun 2011 11:16:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21040 invoked by uid 500); 3 Jun 2011 11:16:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21028 invoked by uid 99); 3 Jun 2011 11:16:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 11:16:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 11:16:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3F0E5F244B
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 11:15:48 +0000 (UTC)
Date: Fri, 3 Jun 2011 11:15:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1964047960.65469.1307099748255.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <3255382.248681296161204958.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7121) Exceptions while serializing IPC
 call response are not handled well
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043302#comment-13043302 ] 

Hudson commented on HADOOP-7121:
--------------------------------

Integrated in Hadoop-Common-trunk #708 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/708/])
    HADOOP-7349. HADOOP-7121 accidentally disabled some tests in TestIPC. Contributed by Todd Lipcon.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130758
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/ipc/TestIPC.java


> Exceptions while serializing IPC call response are not handled well
> -------------------------------------------------------------------
>
>                 Key: HADOOP-7121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7121
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>              Labels: newbie
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7121.txt, hadoop-7121.txt, hadoop-7121.txt
>
>
> We had a situation where for some reason the serialization of an RPC call's response was throwing OOME. When this happens, the exception is not caught, and the call never gets a response - the client just hangs. Additionally, the OOME propagated all the way to the top of the IPC handler and caused the handler. Plus, the Handler upon exit only logged to stdout and not to the log4j logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15801-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 11:16:33 2011
Return-Path: <common-issues-return-15801-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 62920622D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 11:16:29 +0000 (UTC)
Received: (qmail 19982 invoked by uid 500); 3 Jun 2011 11:16:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19933 invoked by uid 500); 3 Jun 2011 11:16:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19884 invoked by uid 99); 3 Jun 2011 11:16:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 11:16:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 11:16:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B5A48F243E
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 11:15:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 11:15:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <479584761.65456.1307099747740.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1016265931.61342.1306972667438.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7349) HADOOP-7121 accidentally disabled
 some tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043300#comment-13043300 ] 

Hudson commented on HADOOP-7349:
--------------------------------

Integrated in Hadoop-Common-trunk #708 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/708/])
    HADOOP-7349. HADOOP-7121 accidentally disabled some tests in TestIPC. Contributed by Todd Lipcon.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130758
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/ipc/TestIPC.java


> HADOOP-7121 accidentally disabled some tests
> --------------------------------------------
>
>                 Key: HADOOP-7349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7349
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7349.txt
>
>
> When I converted TestIPC to JUnit 4, I missed a couple of tests towards the bottom of the file when adding the @Test annotation.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15805-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 11:16:52 2011
Return-Path: <common-issues-return-15805-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0F03263DC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 11:16:52 +0000 (UTC)
Received: (qmail 21524 invoked by uid 500); 3 Jun 2011 11:16:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21496 invoked by uid 500); 3 Jun 2011 11:16:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21488 invoked by uid 99); 3 Jun 2011 11:16:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 11:16:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 11:16:49 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 67421F244F
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 11:15:48 +0000 (UTC)
Date: Fri, 3 Jun 2011 11:15:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2057249495.65473.1307099748419.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <99930859.62060.1306994747296.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7351) Regression:
 HttpServer#getWebAppsPath used to be protected so subclasses could supply
 alternate webapps path but it was made private by HADOOP-6461
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043304#comment-13043304 ] 

Hudson commented on HADOOP-7351:
--------------------------------

Integrated in Hadoop-Common-trunk #708 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/708/])
    HADOOP-7351 Regression: HttpServer#getWebAppsPath used to be protected so subclasses could supply alternate webapps path but it was made private by HADOOP-6461 -- Fix commit message; I'd put it in wrong location under 'new features' rather than under 'bugs'
HADOOP-7351 Regression: HttpServer#getWebAppsPath used to be protected so subclasses could supply alternate webapps path but it was made private by HADOOP-6461

stack : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130738
Files : 
* /hadoop/common/trunk/CHANGES.txt

stack : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130730
Files : 
* /hadoop/common/trunk/src/java/org/apache/hadoop/http/HttpServer.java
* /hadoop/common/trunk/CHANGES.txt


> Regression: HttpServer#getWebAppsPath used to be protected so subclasses could supply alternate webapps path but it was made private by HADOOP-6461
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7351
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7351
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>             Fix For: 0.22.0
>
>         Attachments: 7351.txt
>
>
> It USED to be protected rather than private but its access was changed by HADOOP-6461.  It did the following:
> {code}
> -  protected String getWebAppsPath() throws IOException {
> -    URL url = getClass().getClassLoader().getResource("webapps");
> +  private String getWebAppsPath(String appName) throws FileNotFoundException {
> +    URL url = getClass().getClassLoader().getResource("webapps/" + appName);
> ...
> {code}
> HBase subclasses HttpServer providing its UI.  This change makes it so we can no longer do so.
> This change made it into 0.21.  I'd like to get a fix committed to 0.22 as well as TRUNK.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15806-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 15:48:29 2011
Return-Path: <common-issues-return-15806-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 263DF61D3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 15:48:29 +0000 (UTC)
Received: (qmail 95265 invoked by uid 500); 3 Jun 2011 15:48:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95234 invoked by uid 500); 3 Jun 2011 15:48:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95223 invoked by uid 99); 3 Jun 2011 15:48:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 15:48:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 15:48:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BCD82F24FD
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 15:47:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 15:47:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1301941069.66008.1307116067770.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <164637914.38616.1306225908771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7327) FileSystem.listStatus() throws
 NullPointerException instead of IOException upon access permission failure
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043417#comment-13043417 ] 

Daryn Sharp commented on HADOOP-7327:
-------------------------------------

+1  It looks like the issue will be more comprehensively addressed by HADOOP-7352, so I'm fine with this as an incremental improvement.

> FileSystem.listStatus() throws NullPointerException instead of IOException upon access permission failure
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7327
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7327
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>            Priority: Minor
>         Attachments: hadoop-7327-1.patch
>
>
> Many processes that call listStatus() expect to handle IOException, but instead are getting runtime error NullPointerException, if the directory being scanned is visible but no-access to the running user id.  For example, if directory foo is drwxr-xr-x, and subdirectory foo/bar is drwx------, then trying to do listStatus(Path(foo/bar)) will cause a NullPointerException.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15807-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 15:52:28 2011
Return-Path: <common-issues-return-15807-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A6C816413
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 15:52:28 +0000 (UTC)
Received: (qmail 8400 invoked by uid 500); 3 Jun 2011 15:52:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8367 invoked by uid 500); 3 Jun 2011 15:52:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8358 invoked by uid 99); 3 Jun 2011 15:52:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 15:52:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 15:52:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 62905F2719
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 15:51:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 15:51:47 +0000 (UTC)
From: "Thomas Graves (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <835823012.66027.1307116307400.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1738050873.57393.1306875887708.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7343) backport HADOOP-7008 and HADOOP-7042
 to branch-0.20-security
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Graves updated HADOOP-7343:
----------------------------------

    Attachment: HADOOP-7343-20security.patch

> backport HADOOP-7008 and HADOOP-7042 to branch-0.20-security
> ------------------------------------------------------------
>
>                 Key: HADOOP-7343
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7343
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.20.204.0
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>            Priority: Minor
>         Attachments: HADOOP-7343-20security.patch, HADOOP-7343-20security.patch
>
>
> backport HADOOP-7008 and HADOOP-7042 to branch-0.20-security so that we can enable test-patch.sh to have a configured number of acceptable findbugs and javadoc warnings

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15808-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 15:52:28 2011
Return-Path: <common-issues-return-15808-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EE7D36420
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 15:52:28 +0000 (UTC)
Received: (qmail 8647 invoked by uid 500); 3 Jun 2011 15:52:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8608 invoked by uid 500); 3 Jun 2011 15:52:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8600 invoked by uid 99); 3 Jun 2011 15:52:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 15:52:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 15:52:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9B51EF2723
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 15:51:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 15:51:47 +0000 (UTC)
From: "Thomas Graves (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1335334742.66033.1307116307632.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1738050873.57393.1306875887708.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7343) backport HADOOP-7008 and HADOOP-7042
 to branch-0.20-security
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Graves updated HADOOP-7343:
----------------------------------

    Attachment: HADOOP-7343-20security.patch

> backport HADOOP-7008 and HADOOP-7042 to branch-0.20-security
> ------------------------------------------------------------
>
>                 Key: HADOOP-7343
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7343
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.20.204.0
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>            Priority: Minor
>         Attachments: HADOOP-7343-20security.patch, HADOOP-7343-20security.patch
>
>
> backport HADOOP-7008 and HADOOP-7042 to branch-0.20-security so that we can enable test-patch.sh to have a configured number of acceptable findbugs and javadoc warnings

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15809-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 15:56:29 2011
Return-Path: <common-issues-return-15809-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2CB4D6658
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 15:56:29 +0000 (UTC)
Received: (qmail 15417 invoked by uid 500); 3 Jun 2011 15:56:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15380 invoked by uid 500); 3 Jun 2011 15:56:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15370 invoked by uid 99); 3 Jun 2011 15:56:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 15:56:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 15:56:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D3F48F2940
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 15:55:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 15:55:47 +0000 (UTC)
From: "Thomas Graves (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1716844908.66048.1307116547864.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1738050873.57393.1306875887708.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7343) backport HADOOP-7008 and HADOOP-7042
 to branch-0.20-security
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Graves updated HADOOP-7343:
----------------------------------

    Status: Patch Available  (was: Open)

> backport HADOOP-7008 and HADOOP-7042 to branch-0.20-security
> ------------------------------------------------------------
>
>                 Key: HADOOP-7343
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7343
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.20.204.0
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>            Priority: Minor
>         Attachments: HADOOP-7343-20security.patch, HADOOP-7343-20security.patch
>
>
> backport HADOOP-7008 and HADOOP-7042 to branch-0.20-security so that we can enable test-patch.sh to have a configured number of acceptable findbugs and javadoc warnings

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15810-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 15:56:31 2011
Return-Path: <common-issues-return-15810-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4E56A667D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 15:56:31 +0000 (UTC)
Received: (qmail 15870 invoked by uid 500); 3 Jun 2011 15:56:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15827 invoked by uid 500); 3 Jun 2011 15:56:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15819 invoked by uid 99); 3 Jun 2011 15:56:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 15:56:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 15:56:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C4C5DF293E
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 15:55:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 15:55:47 +0000 (UTC)
From: "Thomas Graves (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <340506013.66046.1307116547802.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1738050873.57393.1306875887708.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7343) backport HADOOP-7008 and
 HADOOP-7042 to branch-0.20-security
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043423#comment-13043423 ] 

Thomas Graves commented on HADOOP-7343:
---------------------------------------

tested manually by having it not quit if svn update finds code checked out since otherwise it would fail and run the old test-patch.sh.


[exec]
[exec] -1 overall.     
[exec]     
[exec]     +1 @author.  The patch does not contain any @author tags.     
[exec]     
[exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.     [exec]                         Please justify why no tests are needed for this patch.
[exec]     
[exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
[exec]
[exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec]     +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.
[exec]

> backport HADOOP-7008 and HADOOP-7042 to branch-0.20-security
> ------------------------------------------------------------
>
>                 Key: HADOOP-7343
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7343
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.20.204.0
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>            Priority: Minor
>         Attachments: HADOOP-7343-20security.patch, HADOOP-7343-20security.patch
>
>
> backport HADOOP-7008 and HADOOP-7042 to branch-0.20-security so that we can enable test-patch.sh to have a configured number of acceptable findbugs and javadoc warnings

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15811-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 16:04:31 2011
Return-Path: <common-issues-return-15811-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 08C1F6FF5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 16:04:31 +0000 (UTC)
Received: (qmail 30419 invoked by uid 500); 3 Jun 2011 16:04:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30382 invoked by uid 500); 3 Jun 2011 16:04:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30374 invoked by uid 99); 3 Jun 2011 16:04:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 16:04:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 16:04:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5E92DF2C8B
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 16:03:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 16:03:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <740833217.66076.1307117027384.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1738050873.57393.1306875887708.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7343) backport HADOOP-7008 and
 HADOOP-7042 to branch-0.20-security
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043429#comment-13043429 ] 

Hadoop QA commented on HADOOP-7343:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481360/HADOOP-7343-20security.patch
  against trunk revision 1130833.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 17 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/567//console

This message is automatically generated.

> backport HADOOP-7008 and HADOOP-7042 to branch-0.20-security
> ------------------------------------------------------------
>
>                 Key: HADOOP-7343
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7343
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.20.204.0
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>            Priority: Minor
>         Attachments: HADOOP-7343-20security.patch, HADOOP-7343-20security.patch
>
>
> backport HADOOP-7008 and HADOOP-7042 to branch-0.20-security so that we can enable test-patch.sh to have a configured number of acceptable findbugs and javadoc warnings

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15812-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 16:22:28 2011
Return-Path: <common-issues-return-15812-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DF0116F3E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 16:22:28 +0000 (UTC)
Received: (qmail 79114 invoked by uid 500); 3 Jun 2011 16:22:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79080 invoked by uid 500); 3 Jun 2011 16:22:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79072 invoked by uid 99); 3 Jun 2011 16:22:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 16:22:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 16:22:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A0E3DF2518
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 16:21:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 16:21:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <185570340.66155.1307118107655.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <397648942.57126.1306869588398.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7342) Add an utility API in FileUtil for
 JDK File.list
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated HADOOP-7342:
-------------------------------

    Attachment: HADOOP-7342-2.patch

Added the Assert messages Cos requested, and trigger patch submission.

> Add an utility API in FileUtil for JDK File.list
> ------------------------------------------------
>
>                 Key: HADOOP-7342
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7342
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7342-1.patch, HADOOP-7342-2.patch
>
>
> Java File.list API can return null when disk is bad or directory is not a directory. This utility API in FileUtil will throw an exception when this happens rather than returning null. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15813-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 16:22:31 2011
Return-Path: <common-issues-return-15813-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E2B966F66
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 16:22:31 +0000 (UTC)
Received: (qmail 79955 invoked by uid 500); 3 Jun 2011 16:22:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79906 invoked by uid 500); 3 Jun 2011 16:22:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79845 invoked by uid 99); 3 Jun 2011 16:22:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 16:22:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 16:22:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 18EF1F2521
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 16:21:48 +0000 (UTC)
Date: Fri, 3 Jun 2011 16:21:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2130042012.66162.1307118107927.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <397648942.57126.1306869588398.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7342) Add an utility API in FileUtil for
 JDK File.list
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated HADOOP-7342:
-------------------------------

    Status: Patch Available  (was: Open)

> Add an utility API in FileUtil for JDK File.list
> ------------------------------------------------
>
>                 Key: HADOOP-7342
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7342
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7342-1.patch, HADOOP-7342-2.patch
>
>
> Java File.list API can return null when disk is bad or directory is not a directory. This utility API in FileUtil will throw an exception when this happens rather than returning null. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15814-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 16:45:28 2011
Return-Path: <common-issues-return-15814-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CC4FB601A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 16:45:28 +0000 (UTC)
Received: (qmail 43764 invoked by uid 500); 3 Jun 2011 16:45:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43735 invoked by uid 500); 3 Jun 2011 16:45:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43725 invoked by uid 99); 3 Jun 2011 16:45:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 16:45:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 16:45:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 770FBF20BA
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 16:44:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 16:44:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1954060242.66263.1307119487484.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <397648942.57126.1306869588398.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7342) Add an utility API in FileUtil for
 JDK File.list
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043564#comment-13043564 ] 

Hadoop QA commented on HADOOP-7342:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481365/HADOOP-7342-2.patch
  against trunk revision 1130833.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/568//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/568//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/568//console

This message is automatically generated.

> Add an utility API in FileUtil for JDK File.list
> ------------------------------------------------
>
>                 Key: HADOOP-7342
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7342
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7342-1.patch, HADOOP-7342-2.patch
>
>
> Java File.list API can return null when disk is bad or directory is not a directory. This utility API in FileUtil will throw an exception when this happens rather than returning null. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15815-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 16:47:30 2011
Return-Path: <common-issues-return-15815-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C4E856DE8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 16:47:30 +0000 (UTC)
Received: (qmail 48253 invoked by uid 500); 3 Jun 2011 16:47:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48220 invoked by uid 500); 3 Jun 2011 16:47:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48212 invoked by uid 99); 3 Jun 2011 16:47:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 16:47:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 16:47:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 84E9AF2AB7
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 16:46:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 16:46:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1868245221.66273.1307119607540.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <397648942.57126.1306869588398.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7342) Add an utility API in FileUtil for
 JDK File.list
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043613#comment-13043613 ] 

Matt Foley commented on HADOOP-7342:
------------------------------------

Looks good.  Cos, can you +1 it?

> Add an utility API in FileUtil for JDK File.list
> ------------------------------------------------
>
>                 Key: HADOOP-7342
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7342
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7342-1.patch, HADOOP-7342-2.patch
>
>
> Java File.list API can return null when disk is bad or directory is not a directory. This utility API in FileUtil will throw an exception when this happens rather than returning null. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15816-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 16:57:29 2011
Return-Path: <common-issues-return-15816-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A1A90668A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 16:57:29 +0000 (UTC)
Received: (qmail 60231 invoked by uid 500); 3 Jun 2011 16:57:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60195 invoked by uid 500); 3 Jun 2011 16:57:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60187 invoked by uid 99); 3 Jun 2011 16:57:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 16:57:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 16:57:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 37F0FF233E
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 16:56:48 +0000 (UTC)
Date: Fri, 3 Jun 2011 16:56:48 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <810589611.66302.1307120208225.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <671647059.63008.1307027027379.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7353) Cleanup FsShell and prevent masking
 of RTE stacktraces
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7353:
--------------------------------

    Summary: Cleanup FsShell and prevent masking of RTE stacktraces  (was: Display of errors masks details of some exceptions)

> Cleanup FsShell and prevent masking of RTE stacktraces
> ------------------------------------------------------
>
>                 Key: HADOOP-7353
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7353
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>
> {{FsShell}}'s top level exception handler catches and displays exceptions.  Unfortunately it displays only the first line of an exception, which means an unexpected {{RuntimeExceptions}} like {{NullPointerException}} only display "{{cmd: NullPointerException}}".  This user has no context to understand and/or accurately report the issue.
> Found due to bugs such as {{HADOOP-7327}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15817-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 17:18:29 2011
Return-Path: <common-issues-return-15817-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 367AE6096
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 17:18:29 +0000 (UTC)
Received: (qmail 98248 invoked by uid 500); 3 Jun 2011 17:18:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98212 invoked by uid 500); 3 Jun 2011 17:18:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98204 invoked by uid 99); 3 Jun 2011 17:18:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 17:18:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 17:18:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E7C6CF2BE6
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 17:17:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 17:17:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <590931434.66382.1307121467946.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <671647059.63008.1307027027379.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7353) Cleanup FsShell and prevent masking
 of RTE stacktraces
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7353:
--------------------------------

    Status: Patch Available  (was: Open)

> Cleanup FsShell and prevent masking of RTE stacktraces
> ------------------------------------------------------
>
>                 Key: HADOOP-7353
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7353
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7353.patch
>
>
> {{FsShell}}'s top level exception handler catches and displays exceptions.  Unfortunately it displays only the first line of an exception, which means an unexpected {{RuntimeExceptions}} like {{NullPointerException}} only display "{{cmd: NullPointerException}}".  This user has no context to understand and/or accurately report the issue.
> Found due to bugs such as {{HADOOP-7327}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15818-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 17:18:31 2011
Return-Path: <common-issues-return-15818-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1742D60CC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 17:18:31 +0000 (UTC)
Received: (qmail 98525 invoked by uid 500); 3 Jun 2011 17:18:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98486 invoked by uid 500); 3 Jun 2011 17:18:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98478 invoked by uid 99); 3 Jun 2011 17:18:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 17:18:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 17:18:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 94905F2BD7
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 17:17:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 17:17:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <681738408.66373.1307121467605.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <671647059.63008.1307027027379.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7353) Cleanup FsShell and prevent masking
 of RTE stacktraces
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7353:
--------------------------------

    Attachment: HADOOP-7353.patch

Cleanup exception handling in {{FsShell.run()}} method to print the stack trace of unexpected exceptions such as NPEs.

Converted -help & -usage into {{FsCommand}} subclasses to greatly simplify the aforementioned exception handling block.  All conditional chains are now gone!  Everything is dynamic.

A few trivial changes to {{Command}} & {{CommandFactory}} to support nested classes needed for -help/-usage to access the {{CommandFactory}}.

Break shell object instantiation, and population of commandFactory, into separate methods so DFSAdmin (already a subclass) can eventually take advantage of the modularity.

> Cleanup FsShell and prevent masking of RTE stacktraces
> ------------------------------------------------------
>
>                 Key: HADOOP-7353
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7353
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7353.patch
>
>
> {{FsShell}}'s top level exception handler catches and displays exceptions.  Unfortunately it displays only the first line of an exception, which means an unexpected {{RuntimeExceptions}} like {{NullPointerException}} only display "{{cmd: NullPointerException}}".  This user has no context to understand and/or accurately report the issue.
> Found due to bugs such as {{HADOOP-7327}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15819-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 17:35:29 2011
Return-Path: <common-issues-return-15819-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CD9BD6025
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 17:35:29 +0000 (UTC)
Received: (qmail 16451 invoked by uid 500); 3 Jun 2011 17:35:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16394 invoked by uid 500); 3 Jun 2011 17:35:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16182 invoked by uid 99); 3 Jun 2011 17:35:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 17:35:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 17:35:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BD0EFF32ED
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 17:34:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 17:34:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <945323235.66446.1307122487771.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <671647059.63008.1307027027379.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7353) Cleanup FsShell and prevent
 masking of RTE stacktraces
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043896#comment-13043896 ] 

Daryn Sharp commented on HADOOP-7353:
-------------------------------------

I also removed the slew of pre-existing warnings in {{FsShell}}...

> Cleanup FsShell and prevent masking of RTE stacktraces
> ------------------------------------------------------
>
>                 Key: HADOOP-7353
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7353
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7353.patch
>
>
> {{FsShell}}'s top level exception handler catches and displays exceptions.  Unfortunately it displays only the first line of an exception, which means an unexpected {{RuntimeExceptions}} like {{NullPointerException}} only display "{{cmd: NullPointerException}}".  This user has no context to understand and/or accurately report the issue.
> Found due to bugs such as {{HADOOP-7327}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15820-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 17:35:31 2011
Return-Path: <common-issues-return-15820-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1D48B6057
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 17:35:31 +0000 (UTC)
Received: (qmail 17883 invoked by uid 500); 3 Jun 2011 17:35:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17852 invoked by uid 500); 3 Jun 2011 17:35:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17816 invoked by uid 99); 3 Jun 2011 17:35:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 17:35:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 17:35:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 761FBF32E6
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 17:34:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 17:34:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <39296727.66439.1307122487480.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <671647059.63008.1307027027379.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7353) Cleanup FsShell and prevent
 masking of RTE stacktraces
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043894#comment-13043894 ] 

Hadoop QA commented on HADOOP-7353:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481372/HADOOP-7353.patch
  against trunk revision 1130833.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 4 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/569//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/569//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/569//console

This message is automatically generated.

> Cleanup FsShell and prevent masking of RTE stacktraces
> ------------------------------------------------------
>
>                 Key: HADOOP-7353
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7353
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7353.patch
>
>
> {{FsShell}}'s top level exception handler catches and displays exceptions.  Unfortunately it displays only the first line of an exception, which means an unexpected {{RuntimeExceptions}} like {{NullPointerException}} only display "{{cmd: NullPointerException}}".  This user has no context to understand and/or accurately report the issue.
> Found due to bugs such as {{HADOOP-7327}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15821-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 18:12:29 2011
Return-Path: <common-issues-return-15821-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 12A2D6B47
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 18:12:29 +0000 (UTC)
Received: (qmail 85198 invoked by uid 500); 3 Jun 2011 18:12:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85158 invoked by uid 500); 3 Jun 2011 18:12:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85078 invoked by uid 99); 3 Jun 2011 18:12:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 18:12:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 18:12:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6CC12F20F6
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 18:11:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 18:11:47 +0000 (UTC)
From: "Philip Zeyliger (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1267239290.66594.1307124707441.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7357) hadoop.io.compress.TestCodec#main()
 should exit with non-zero exit code if test failed
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

hadoop.io.compress.TestCodec#main() should exit with non-zero exit code if test failed
--------------------------------------------------------------------------------------

                 Key: HADOOP-7357
                 URL: https://issues.apache.org/jira/browse/HADOOP-7357
             Project: Hadoop Common
          Issue Type: Bug
          Components: test
            Reporter: Philip Zeyliger
            Assignee: Philip Zeyliger
            Priority: Trivial
         Attachments: HADOOP-7357.patch.txt

It's convenient to run something like
{noformat}
HADOOP_CLASSPATH=hadoop-test-0.20.2.jar bin/hadoop org.apache.hadoop.io.compress.TestCodec  -count 3 -codec fo
{noformat}
but the error code it returns isn't interesting.

1-line patch attached fixes that.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15822-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 18:12:29 2011
Return-Path: <common-issues-return-15822-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6CE446B58
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 18:12:29 +0000 (UTC)
Received: (qmail 85645 invoked by uid 500); 3 Jun 2011 18:12:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85616 invoked by uid 500); 3 Jun 2011 18:12:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85332 invoked by uid 99); 3 Jun 2011 18:12:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 18:12:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 18:12:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 93FDCF20F9
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 18:11:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 18:11:47 +0000 (UTC)
From: "Philip Zeyliger (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1863949665.66597.1307124707602.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1267239290.66594.1307124707441.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7357) hadoop.io.compress.TestCodec#main()
 should exit with non-zero exit code if test failed
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Philip Zeyliger updated HADOOP-7357:
------------------------------------

    Attachment: HADOOP-7357.patch.txt

> hadoop.io.compress.TestCodec#main() should exit with non-zero exit code if test failed
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7357
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7357
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Philip Zeyliger
>            Assignee: Philip Zeyliger
>            Priority: Trivial
>         Attachments: HADOOP-7357.patch.txt
>
>
> It's convenient to run something like
> {noformat}
> HADOOP_CLASSPATH=hadoop-test-0.20.2.jar bin/hadoop org.apache.hadoop.io.compress.TestCodec  -count 3 -codec fo
> {noformat}
> but the error code it returns isn't interesting.
> 1-line patch attached fixes that.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15823-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 18:12:32 2011
Return-Path: <common-issues-return-15823-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E79306B75
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 18:12:31 +0000 (UTC)
Received: (qmail 85991 invoked by uid 500); 3 Jun 2011 18:12:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85919 invoked by uid 500); 3 Jun 2011 18:12:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85857 invoked by uid 99); 3 Jun 2011 18:12:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 18:12:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 18:12:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2CC68F2109
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 18:11:48 +0000 (UTC)
Date: Fri, 3 Jun 2011 18:11:48 +0000 (UTC)
From: "Philip Zeyliger (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1308624728.66612.1307124708178.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1267239290.66594.1307124707441.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7357) hadoop.io.compress.TestCodec#main()
 should exit with non-zero exit code if test failed
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Philip Zeyliger updated HADOOP-7357:
------------------------------------

    Status: Patch Available  (was: Open)

> hadoop.io.compress.TestCodec#main() should exit with non-zero exit code if test failed
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7357
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7357
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Philip Zeyliger
>            Assignee: Philip Zeyliger
>            Priority: Trivial
>         Attachments: HADOOP-7357.patch.txt
>
>
> It's convenient to run something like
> {noformat}
> HADOOP_CLASSPATH=hadoop-test-0.20.2.jar bin/hadoop org.apache.hadoop.io.compress.TestCodec  -count 3 -codec fo
> {noformat}
> but the error code it returns isn't interesting.
> 1-line patch attached fixes that.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15824-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 18:16:29 2011
Return-Path: <common-issues-return-15824-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 123AC635E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 18:16:29 +0000 (UTC)
Received: (qmail 99836 invoked by uid 500); 3 Jun 2011 18:16:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99801 invoked by uid 500); 3 Jun 2011 18:16:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99761 invoked by uid 99); 3 Jun 2011 18:16:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 18:16:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 18:16:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8BD1CF2353
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 18:15:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 18:15:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1951159417.66630.1307124947569.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1267239290.66594.1307124707441.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7357)
 hadoop.io.compress.TestCodec#main() should exit with non-zero exit code if
 test failed
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043930#comment-13043930 ] 

Todd Lipcon commented on HADOOP-7357:
-------------------------------------

Why not get rid of the try/catch completely and just let the exception propagate out of main? That would produce the correct error code as well and remove 5 lines of code.

> hadoop.io.compress.TestCodec#main() should exit with non-zero exit code if test failed
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7357
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7357
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Philip Zeyliger
>            Assignee: Philip Zeyliger
>            Priority: Trivial
>         Attachments: HADOOP-7357.patch.txt
>
>
> It's convenient to run something like
> {noformat}
> HADOOP_CLASSPATH=hadoop-test-0.20.2.jar bin/hadoop org.apache.hadoop.io.compress.TestCodec  -count 3 -codec fo
> {noformat}
> but the error code it returns isn't interesting.
> 1-line patch attached fixes that.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15825-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 18:16:31 2011
Return-Path: <common-issues-return-15825-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2C4646391
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 18:16:31 +0000 (UTC)
Received: (qmail 213 invoked by uid 500); 3 Jun 2011 18:16:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 182 invoked by uid 500); 3 Jun 2011 18:16:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 174 invoked by uid 99); 3 Jun 2011 18:16:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 18:16:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 18:16:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A7651F2358
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 18:15:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 18:15:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <575109744.66633.1307124947681.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1267239290.66594.1307124707441.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7357) hadoop.io.compress.TestCodec#main()
 should exit with non-zero exit code if test failed
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7357:
-----------------------------------

    Status: Open  (was: Patch Available)

Hi Philip,

This patch won't apply as-is. Looks to me like you generated it using `git show'. Please regenerate the patch using `git diff'.

> hadoop.io.compress.TestCodec#main() should exit with non-zero exit code if test failed
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7357
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7357
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Philip Zeyliger
>            Assignee: Philip Zeyliger
>            Priority: Trivial
>         Attachments: HADOOP-7357.patch.txt
>
>
> It's convenient to run something like
> {noformat}
> HADOOP_CLASSPATH=hadoop-test-0.20.2.jar bin/hadoop org.apache.hadoop.io.compress.TestCodec  -count 3 -codec fo
> {noformat}
> but the error code it returns isn't interesting.
> 1-line patch attached fixes that.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15826-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 18:18:32 2011
Return-Path: <common-issues-return-15826-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9C1276DEC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 18:18:32 +0000 (UTC)
Received: (qmail 4354 invoked by uid 500); 3 Jun 2011 18:18:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4299 invoked by uid 500); 3 Jun 2011 18:18:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4161 invoked by uid 99); 3 Jun 2011 18:18:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 18:18:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 18:18:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B3218F248F
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 18:17:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 18:17:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <409900072.66640.1307125067730.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1267239290.66594.1307124707441.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7357) hadoop.io.compress.TestCodec#main()
 should exit with non-zero exit code if test failed
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7357:
-----------------------------------

    Status: Patch Available  (was: Open)

I take it all back! `patch' is smarter than I gave it credit for.

> hadoop.io.compress.TestCodec#main() should exit with non-zero exit code if test failed
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7357
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7357
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Philip Zeyliger
>            Assignee: Philip Zeyliger
>            Priority: Trivial
>         Attachments: HADOOP-7357.patch.txt
>
>
> It's convenient to run something like
> {noformat}
> HADOOP_CLASSPATH=hadoop-test-0.20.2.jar bin/hadoop org.apache.hadoop.io.compress.TestCodec  -count 3 -codec fo
> {noformat}
> but the error code it returns isn't interesting.
> 1-line patch attached fixes that.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15827-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 18:24:29 2011
Return-Path: <common-issues-return-15827-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7F8E068EC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 18:24:29 +0000 (UTC)
Received: (qmail 15156 invoked by uid 500); 3 Jun 2011 18:24:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15128 invoked by uid 500); 3 Jun 2011 18:24:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15119 invoked by uid 99); 3 Jun 2011 18:24:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 18:24:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 18:24:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 176AFF27E2
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 18:23:48 +0000 (UTC)
Date: Fri, 3 Jun 2011 18:23:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1816598440.66669.1307125428092.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <531829213.32234.1305934307559.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7316) Add public javadocs to
 FSDataInputStream and FSDataOutputStream
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043937#comment-13043937 ] 

Todd Lipcon commented on HADOOP-7316:
-------------------------------------

Super nitty review on the javadoc:
- Javadoc style guide at http://www.oracle.com/technetwork/java/javase/documentation/index-137868.html says the following about punctuation and capitalization for @param and @return:
{quote}
The description begins with a lowercase letter if it is a phrase (contains no verb), or an uppercase letter if it is a sentence. End the phrase with a period only if another phrase or sentence follows it.
{quote}
(ie the @param and @returns should start with lower case letters)

- following should read "the buffer *into* which data is read" in two places:
+   * @param buffer    The buffer in which data is read.

- for the readFully javadoc, @param length should say "the exact number of bytes to read" or "the number of bytes" rather than "the maximum" given the contract of readFully.
- for readFully, we should also specify that, even if it throws an exception, it may modify an undetermined number of bytes in the target buffer.
- typo: "altenate" should be "alternate"





> Add public javadocs to FSDataInputStream and FSDataOutputStream
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7316
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7316
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Jonathan Hsieh
>            Assignee: Eli Collins
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7316-1.patch, hadoop-7316-2.patch
>
>
> This is a method made public for testing.  In comments in HADOOP-7301 after commit, adding javadoc comments was requested.  This is a follow up jira to address it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15828-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 18:24:31 2011
Return-Path: <common-issues-return-15828-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 903506901
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 18:24:31 +0000 (UTC)
Received: (qmail 15784 invoked by uid 500); 3 Jun 2011 18:24:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15742 invoked by uid 500); 3 Jun 2011 18:24:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15734 invoked by uid 99); 3 Jun 2011 18:24:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 18:24:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 18:24:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 34191F27E6
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 18:23:48 +0000 (UTC)
Date: Fri, 3 Jun 2011 18:23:48 +0000 (UTC)
From: "Philip Zeyliger (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <339156762.66673.1307125428210.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1267239290.66594.1307124707441.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7357) hadoop.io.compress.TestCodec#main()
 should exit with non-zero exit code if test failed
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Philip Zeyliger updated HADOOP-7357:
------------------------------------

    Attachment: HADOOP-7357-v2.patch.txt

Removing the entire try catch, as per Todd's suggestion.

> hadoop.io.compress.TestCodec#main() should exit with non-zero exit code if test failed
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7357
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7357
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Philip Zeyliger
>            Assignee: Philip Zeyliger
>            Priority: Trivial
>         Attachments: HADOOP-7357-v2.patch.txt, HADOOP-7357.patch.txt
>
>
> It's convenient to run something like
> {noformat}
> HADOOP_CLASSPATH=hadoop-test-0.20.2.jar bin/hadoop org.apache.hadoop.io.compress.TestCodec  -count 3 -codec fo
> {noformat}
> but the error code it returns isn't interesting.
> 1-line patch attached fixes that.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15829-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 18:24:33 2011
Return-Path: <common-issues-return-15829-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 19632691E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 18:24:33 +0000 (UTC)
Received: (qmail 16349 invoked by uid 500); 3 Jun 2011 18:24:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16300 invoked by uid 500); 3 Jun 2011 18:24:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16259 invoked by uid 99); 3 Jun 2011 18:24:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 18:24:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 18:24:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6B86BF27EC
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 18:23:48 +0000 (UTC)
Date: Fri, 3 Jun 2011 18:23:48 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1385534528.66679.1307125428437.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1267239290.66594.1307124707441.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7357)
 hadoop.io.compress.TestCodec#main() should exit with non-zero exit code if
 test failed
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043939#comment-13043939 ] 

Hadoop QA commented on HADOOP-7357:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481384/HADOOP-7357.patch.txt
  against trunk revision 1130833.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/570//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/570//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/570//console

This message is automatically generated.

> hadoop.io.compress.TestCodec#main() should exit with non-zero exit code if test failed
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7357
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7357
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Philip Zeyliger
>            Assignee: Philip Zeyliger
>            Priority: Trivial
>         Attachments: HADOOP-7357-v2.patch.txt, HADOOP-7357.patch.txt
>
>
> It's convenient to run something like
> {noformat}
> HADOOP_CLASSPATH=hadoop-test-0.20.2.jar bin/hadoop org.apache.hadoop.io.compress.TestCodec  -count 3 -codec fo
> {noformat}
> but the error code it returns isn't interesting.
> 1-line patch attached fixes that.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15830-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 18:34:29 2011
Return-Path: <common-issues-return-15830-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0FC636731
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 18:34:29 +0000 (UTC)
Received: (qmail 45854 invoked by uid 500); 3 Jun 2011 18:34:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45820 invoked by uid 500); 3 Jun 2011 18:34:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45807 invoked by uid 99); 3 Jun 2011 18:34:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 18:34:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 18:34:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id ABE82F2E4E
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 18:33:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 18:33:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <567245774.66767.1307126027700.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1267239290.66594.1307124707441.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7357)
 hadoop.io.compress.TestCodec#main() should exit with non-zero exit code if
 test failed
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043952#comment-13043952 ] 

Hadoop QA commented on HADOOP-7357:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481388/HADOOP-7357-v2.patch.txt
  against trunk revision 1130833.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/571//console

This message is automatically generated.

> hadoop.io.compress.TestCodec#main() should exit with non-zero exit code if test failed
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7357
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7357
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Philip Zeyliger
>            Assignee: Philip Zeyliger
>            Priority: Trivial
>         Attachments: HADOOP-7357-v2.patch.txt, HADOOP-7357.patch.txt
>
>
> It's convenient to run something like
> {noformat}
> HADOOP_CLASSPATH=hadoop-test-0.20.2.jar bin/hadoop org.apache.hadoop.io.compress.TestCodec  -count 3 -codec fo
> {noformat}
> but the error code it returns isn't interesting.
> 1-line patch attached fixes that.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15831-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 19:47:29 2011
Return-Path: <common-issues-return-15831-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1C9A8624E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 19:47:29 +0000 (UTC)
Received: (qmail 1498 invoked by uid 500); 3 Jun 2011 19:47:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1361 invoked by uid 500); 3 Jun 2011 19:47:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1328 invoked by uid 99); 3 Jun 2011 19:47:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 19:47:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 19:47:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BF433F377D
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 19:46:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 19:46:47 +0000 (UTC)
From: "Robert Joseph Evans (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <977670648.66991.1307130407780.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <467948518.30516.1305901367779.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7314) Add support for throwing
 UnknownHostException when a host doesn't resolve
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043998#comment-13043998 ] 

Robert Joseph Evans commented on HADOOP-7314:
---------------------------------------------

new UrlValidator(UrlValidator.ALLOW_ALL_SCHEMES); is called every time verifyHostnames is called.  It would probably be better to have a private member for this, and if it is reentrant have a static private instance.

> Add support for throwing UnknownHostException when a host doesn't resolve
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-7314
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7314
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jeffrey Naisbitt
>            Assignee: Jeffrey Naisbitt
>         Attachments: HADOOP-7314.patch
>
>
> As part of MAPREDUCE-2489, we need support for having the resolve methods (for DNS mapping) throw UnknownHostExceptions.  (Currently, they hide the exception).  Since the existing 'resolve' method is ultimately used by several other locations/components, I propose we add a new 'resolveValidHosts' method.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15832-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 20:37:30 2011
Return-Path: <common-issues-return-15832-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AA2EC6ACB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 20:37:30 +0000 (UTC)
Received: (qmail 80459 invoked by uid 500); 3 Jun 2011 20:37:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80402 invoked by uid 500); 3 Jun 2011 20:37:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80393 invoked by uid 99); 3 Jun 2011 20:37:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 20:37:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 20:37:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5561EF3817
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 20:36:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 20:36:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1956847682.67114.1307133407346.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1160101136.38164.1306200167436.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7325) hadoop command - do not accept
 class names starting with a hyphen
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044024#comment-13044024 ] 

Todd Lipcon commented on HADOOP-7325:
-------------------------------------

nits:
- I don't think you need the comments explaining bash [[ vs [
- if you do want to keep them, there's a typo: bra*c*ket is missing its 'c'.


> hadoop command - do not accept class names starting with a hyphen
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7325
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Brock Noland
>            Assignee: Brock Noland
>            Priority: Minor
>         Attachments: hadoop-illegal-class-name-0.patch
>
>
> If this is committed I will look at patches for hdfs and mapred.
> When teaching a good portion of the students in every single class execute:
> {code}
> $ hadoop -fs ls /
> {code}
> The -fs is passed directly to the JVM and the JVM fails to start:
> {code}
> $ ./bin/hadoop -fs ls /
> Unrecognized option: -fs
> Could not create the Java virtual machine.
> {code}
> Which is confusing and typically requires explanation. The attached patch improves that behavior:
> {code}
> $ ./bin/hadoop -fs ls /
> Error: No command named `-fs' was found. Perhaps you meant `hadoop fs'
> {code}
> The only risk I can see is if someone is abusing the implementation of hadoop command doing something like so:
> {code}
> $ ./bin/hadoop -Xmx1g  org.apache.hadoop.util.RunJar
> RunJar jarFile [mainClass] args...
> {code}
> The hadoop command does not appear to advertise allowing JVM options before the classname.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15833-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 20:49:31 2011
Return-Path: <common-issues-return-15833-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 183A0645B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 20:49:31 +0000 (UTC)
Received: (qmail 4250 invoked by uid 500); 3 Jun 2011 20:49:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4212 invoked by uid 500); 3 Jun 2011 20:49:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4204 invoked by uid 99); 3 Jun 2011 20:49:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 20:49:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 20:49:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BE8A1F3C02
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 20:48:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 20:48:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1620300939.67157.1307134127777.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1267239290.66594.1307124707441.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7357)
 hadoop.io.compress.TestCodec#main() should exit with non-zero exit code if
 test failed
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044035#comment-13044035 ] 

Todd Lipcon commented on HADOOP-7357:
-------------------------------------

Can you re-upload latest patch with --no-prefix?

> hadoop.io.compress.TestCodec#main() should exit with non-zero exit code if test failed
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7357
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7357
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Philip Zeyliger
>            Assignee: Philip Zeyliger
>            Priority: Trivial
>         Attachments: HADOOP-7357-v2.patch.txt, HADOOP-7357.patch.txt
>
>
> It's convenient to run something like
> {noformat}
> HADOOP_CLASSPATH=hadoop-test-0.20.2.jar bin/hadoop org.apache.hadoop.io.compress.TestCodec  -count 3 -codec fo
> {noformat}
> but the error code it returns isn't interesting.
> 1-line patch attached fixes that.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15834-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 20:49:31 2011
Return-Path: <common-issues-return-15834-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8DD3F6474
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 20:49:31 +0000 (UTC)
Received: (qmail 4528 invoked by uid 500); 3 Jun 2011 20:49:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4462 invoked by uid 500); 3 Jun 2011 20:49:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4441 invoked by uid 99); 3 Jun 2011 20:49:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 20:49:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 20:49:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C5D96F3C03
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 20:48:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 20:48:47 +0000 (UTC)
From: "Brock Noland (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <157754166.67158.1307134127807.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1160101136.38164.1306200167436.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7325) hadoop command - do not accept class
 names starting with a hyphen
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Brock Noland updated HADOOP-7325:
---------------------------------

    Attachment: hadoop-illegal-class-name-1.patch

nit's resolved.

> hadoop command - do not accept class names starting with a hyphen
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7325
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Brock Noland
>            Assignee: Brock Noland
>            Priority: Minor
>         Attachments: hadoop-illegal-class-name-0.patch, hadoop-illegal-class-name-1.patch
>
>
> If this is committed I will look at patches for hdfs and mapred.
> When teaching a good portion of the students in every single class execute:
> {code}
> $ hadoop -fs ls /
> {code}
> The -fs is passed directly to the JVM and the JVM fails to start:
> {code}
> $ ./bin/hadoop -fs ls /
> Unrecognized option: -fs
> Could not create the Java virtual machine.
> {code}
> Which is confusing and typically requires explanation. The attached patch improves that behavior:
> {code}
> $ ./bin/hadoop -fs ls /
> Error: No command named `-fs' was found. Perhaps you meant `hadoop fs'
> {code}
> The only risk I can see is if someone is abusing the implementation of hadoop command doing something like so:
> {code}
> $ ./bin/hadoop -Xmx1g  org.apache.hadoop.util.RunJar
> RunJar jarFile [mainClass] args...
> {code}
> The hadoop command does not appear to advertise allowing JVM options before the classname.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15835-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 20:55:29 2011
Return-Path: <common-issues-return-15835-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 268016998
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 20:55:29 +0000 (UTC)
Received: (qmail 19869 invoked by uid 500); 3 Jun 2011 20:55:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19843 invoked by uid 500); 3 Jun 2011 20:55:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19835 invoked by uid 99); 3 Jun 2011 20:55:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 20:55:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 20:55:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BAA78F3DAB
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 20:54:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 20:54:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <342452528.67171.1307134487761.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <647927887.58055.1306890116067.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7346) Send back nicer error to clients
 using outdated IPC version
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7346:
--------------------------------

    Attachment: hadoop-7346.txt

Rebased against trunk and removed spurious whitespace change. I'll commit based on Cos's +1 if this passes Hudson (no code change since previous patch)

> Send back nicer error to clients using outdated IPC version
> -----------------------------------------------------------
>
>                 Key: HADOOP-7346
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7346
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7346.txt, hadoop-7346.txt, hadoop-7346.txt
>
>
> When an older Hadoop version tries to contact a newer Hadoop version across an IPC protocol version bump, the client currently just gets a non-useful error message like "EOFException".
> Instead, the IPC server code can speak just enough of prior IPC protocols to send back a "fatal" message indicating the version mismatch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15836-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 21:05:29 2011
Return-Path: <common-issues-return-15836-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 037D86772
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 21:05:29 +0000 (UTC)
Received: (qmail 47927 invoked by uid 500); 3 Jun 2011 21:05:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47883 invoked by uid 500); 3 Jun 2011 21:05:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47875 invoked by uid 99); 3 Jun 2011 21:05:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 21:05:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 21:05:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 91445F3249
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 21:04:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 21:04:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <979523855.67211.1307135087591.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7284:
--------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

Marking JIRA resolved, since Sanjay committed this last week.

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash10.patch, trash11.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch, trash8.patch, trash9.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15837-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 21:05:32 2011
Return-Path: <common-issues-return-15837-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1688467C5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 21:05:32 +0000 (UTC)
Received: (qmail 48958 invoked by uid 500); 3 Jun 2011 21:05:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48915 invoked by uid 500); 3 Jun 2011 21:05:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48860 invoked by uid 99); 3 Jun 2011 21:05:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 21:05:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 21:05:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B399FF324E
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 21:04:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 21:04:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <445964104.67216.1307135087732.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1160101136.38164.1306200167436.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7325) hadoop command - do not accept
 class names starting with a hyphen
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044055#comment-13044055 ] 

Hadoop QA commented on HADOOP-7325:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481405/hadoop-illegal-class-name-1.patch
  against trunk revision 1130833.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/572//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/572//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/572//console

This message is automatically generated.

> hadoop command - do not accept class names starting with a hyphen
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7325
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Brock Noland
>            Assignee: Brock Noland
>            Priority: Minor
>         Attachments: hadoop-illegal-class-name-0.patch, hadoop-illegal-class-name-1.patch
>
>
> If this is committed I will look at patches for hdfs and mapred.
> When teaching a good portion of the students in every single class execute:
> {code}
> $ hadoop -fs ls /
> {code}
> The -fs is passed directly to the JVM and the JVM fails to start:
> {code}
> $ ./bin/hadoop -fs ls /
> Unrecognized option: -fs
> Could not create the Java virtual machine.
> {code}
> Which is confusing and typically requires explanation. The attached patch improves that behavior:
> {code}
> $ ./bin/hadoop -fs ls /
> Error: No command named `-fs' was found. Perhaps you meant `hadoop fs'
> {code}
> The only risk I can see is if someone is abusing the implementation of hadoop command doing something like so:
> {code}
> $ ./bin/hadoop -Xmx1g  org.apache.hadoop.util.RunJar
> RunJar jarFile [mainClass] args...
> {code}
> The hadoop command does not appear to advertise allowing JVM options before the classname.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15838-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 21:12:31 2011
Return-Path: <common-issues-return-15838-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E4D316A20
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 21:12:30 +0000 (UTC)
Received: (qmail 63948 invoked by uid 500); 3 Jun 2011 21:12:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63904 invoked by uid 500); 3 Jun 2011 21:12:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63896 invoked by uid 99); 3 Jun 2011 21:12:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 21:12:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 21:12:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 57610F33F1
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 21:11:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 21:11:47 +0000 (UTC)
From: "Brock Noland (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2139418896.67228.1307135507354.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1160101136.38164.1306200167436.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7325) hadoop command - do not accept class
 names starting with a hyphen
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Brock Noland updated HADOOP-7325:
---------------------------------

    Attachment: hadoop-illegal-class-name-2.patch

Slightly too agressive on comment deletion. I think the first comment was useful. Updated.

> hadoop command - do not accept class names starting with a hyphen
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7325
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Brock Noland
>            Assignee: Brock Noland
>            Priority: Minor
>         Attachments: hadoop-illegal-class-name-0.patch, hadoop-illegal-class-name-1.patch, hadoop-illegal-class-name-2.patch
>
>
> If this is committed I will look at patches for hdfs and mapred.
> When teaching a good portion of the students in every single class execute:
> {code}
> $ hadoop -fs ls /
> {code}
> The -fs is passed directly to the JVM and the JVM fails to start:
> {code}
> $ ./bin/hadoop -fs ls /
> Unrecognized option: -fs
> Could not create the Java virtual machine.
> {code}
> Which is confusing and typically requires explanation. The attached patch improves that behavior:
> {code}
> $ ./bin/hadoop -fs ls /
> Error: No command named `-fs' was found. Perhaps you meant `hadoop fs'
> {code}
> The only risk I can see is if someone is abusing the implementation of hadoop command doing something like so:
> {code}
> $ ./bin/hadoop -Xmx1g  org.apache.hadoop.util.RunJar
> RunJar jarFile [mainClass] args...
> {code}
> The hadoop command does not appear to advertise allowing JVM options before the classname.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15839-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 21:15:00 2011
Return-Path: <common-issues-return-15839-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3FF7F6AF5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 21:15:00 +0000 (UTC)
Received: (qmail 66497 invoked by uid 500); 3 Jun 2011 21:15:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66470 invoked by uid 500); 3 Jun 2011 21:15:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66462 invoked by uid 99); 3 Jun 2011 21:15:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 21:15:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 21:14:58 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C2BAAF34AC
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 21:14:18 +0000 (UTC)
Date: Fri, 3 Jun 2011 21:14:18 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <353376378.67233.1307135658794.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <647927887.58055.1306890116067.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7346) Send back nicer error to clients
 using outdated IPC version
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044059#comment-13044059 ] 

Hadoop QA commented on HADOOP-7346:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481406/hadoop-7346.txt
  against trunk revision 1130833.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/573//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/573//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/573//console

This message is automatically generated.

> Send back nicer error to clients using outdated IPC version
> -----------------------------------------------------------
>
>                 Key: HADOOP-7346
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7346
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7346.txt, hadoop-7346.txt, hadoop-7346.txt
>
>
> When an older Hadoop version tries to contact a newer Hadoop version across an IPC protocol version bump, the client currently just gets a non-useful error message like "EOFException".
> Instead, the IPC server code can speak just enough of prior IPC protocols to send back a "fatal" message indicating the version mismatch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15840-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 21:20:32 2011
Return-Path: <common-issues-return-15840-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7E1356299
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 21:20:32 +0000 (UTC)
Received: (qmail 80234 invoked by uid 500); 3 Jun 2011 21:20:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80187 invoked by uid 500); 3 Jun 2011 21:20:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80167 invoked by uid 99); 3 Jun 2011 21:20:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 21:20:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 21:20:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 20FDBF379C
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 21:19:48 +0000 (UTC)
Date: Fri, 3 Jun 2011 21:19:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <994429901.67267.1307135988131.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1142137633.58455.1306902167529.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7348) Modify the option of FsShell
 getmerge from [addnl] to [-nl] for more comprehensive
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7348:
--------------------------------

    Status: Open  (was: Patch Available)

> Modify the option of FsShell getmerge from [addnl] to [-nl] for more comprehensive
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7348
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7348
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Assignee: XieXianshan
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7348.patch
>
>
> The [addnl] option of FsShell getmerge should be either "true" or "false",but it is very hard to understand by users, especially  who`s never used this option before. 
> So,the [addnl] option should be changed to [-nl] for more comprehensive.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15841-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 21:20:32 2011
Return-Path: <common-issues-return-15841-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 97FA6629F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 21:20:32 +0000 (UTC)
Received: (qmail 80279 invoked by uid 500); 3 Jun 2011 21:20:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80190 invoked by uid 500); 3 Jun 2011 21:20:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80168 invoked by uid 99); 3 Jun 2011 21:20:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 21:20:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 21:20:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 53406F37A1
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 21:19:48 +0000 (UTC)
Date: Fri, 3 Jun 2011 21:19:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2111585105.67272.1307135988337.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1992675736.52742.1306733207390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7339) Introduce a buffered checksum for
 avoiding frequently calls on Checksum.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7339:
--------------------------------

    Status: Open  (was: Patch Available)

canceling patch available status pending a benchmark

> Introduce a buffered checksum for avoiding frequently calls on Checksum.update()
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-7339
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7339
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Min Zhou
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7339-v1.diff, HADOOP-7339-v2.diff
>
>
> We found that PureJavaCRC32/CRC32.update() is the TOP 1 of the methods consuming CPU in a map side, and in reduce side, it cost a lots of CPU too.
> IFileOutputStream would frequently call Checksum.update() during writing a record. It's very common a MR key/value less than 512 bytes. Checksum.update() would be called every time writing a key/value. 
> Test case: terasort 100MB. 
> Checksum.update() calls has be reduced from 4030348 to 28069.  This method is not a hotspot anymore.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15842-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 21:24:28 2011
Return-Path: <common-issues-return-15842-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DF4C0658F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 21:24:28 +0000 (UTC)
Received: (qmail 89564 invoked by uid 500); 3 Jun 2011 21:24:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89524 invoked by uid 500); 3 Jun 2011 21:24:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89516 invoked by uid 99); 3 Jun 2011 21:24:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 21:24:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 21:24:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 53193F3907
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 21:23:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 21:23:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1069887525.67287.1307136227337.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1160101136.38164.1306200167436.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7325) hadoop command - do not accept
 class names starting with a hyphen
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044068#comment-13044068 ] 

Hadoop QA commented on HADOOP-7325:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481408/hadoop-illegal-class-name-2.patch
  against trunk revision 1130833.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/574//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/574//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/574//console

This message is automatically generated.

> hadoop command - do not accept class names starting with a hyphen
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7325
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Brock Noland
>            Assignee: Brock Noland
>            Priority: Minor
>         Attachments: hadoop-illegal-class-name-0.patch, hadoop-illegal-class-name-1.patch, hadoop-illegal-class-name-2.patch
>
>
> If this is committed I will look at patches for hdfs and mapred.
> When teaching a good portion of the students in every single class execute:
> {code}
> $ hadoop -fs ls /
> {code}
> The -fs is passed directly to the JVM and the JVM fails to start:
> {code}
> $ ./bin/hadoop -fs ls /
> Unrecognized option: -fs
> Could not create the Java virtual machine.
> {code}
> Which is confusing and typically requires explanation. The attached patch improves that behavior:
> {code}
> $ ./bin/hadoop -fs ls /
> Error: No command named `-fs' was found. Perhaps you meant `hadoop fs'
> {code}
> The only risk I can see is if someone is abusing the implementation of hadoop command doing something like so:
> {code}
> $ ./bin/hadoop -Xmx1g  org.apache.hadoop.util.RunJar
> RunJar jarFile [mainClass] args...
> {code}
> The hadoop command does not appear to advertise allowing JVM options before the classname.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15843-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 21:41:31 2011
Return-Path: <common-issues-return-15843-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CC65B6E69
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 21:41:31 +0000 (UTC)
Received: (qmail 9408 invoked by uid 500); 3 Jun 2011 21:41:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9347 invoked by uid 500); 3 Jun 2011 21:41:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9151 invoked by uid 99); 3 Jun 2011 21:41:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 21:41:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 21:41:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2496FF3D3B
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 21:40:48 +0000 (UTC)
Date: Fri, 3 Jun 2011 21:40:48 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1699201797.67317.1307137248146.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <397648942.57126.1306869588398.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7342) Add an utility API in FileUtil for
 JDK File.list
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044073#comment-13044073 ] 

Matt Foley commented on HADOOP-7342:
------------------------------------

If we don't hear from Cos by end of day, I'll commit this since we accepted his comment and passed Jenkins test, and no one else has commented in four days.  Also, this just mirrors the already committed HADOOP-7322.  Thanks.

> Add an utility API in FileUtil for JDK File.list
> ------------------------------------------------
>
>                 Key: HADOOP-7342
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7342
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7342-1.patch, HADOOP-7342-2.patch
>
>
> Java File.list API can return null when disk is bad or directory is not a directory. This utility API in FileUtil will throw an exception when this happens rather than returning null. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15844-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 21:46:28 2011
Return-Path: <common-issues-return-15844-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B8DC76987
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 21:46:28 +0000 (UTC)
Received: (qmail 15594 invoked by uid 500); 3 Jun 2011 21:46:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15558 invoked by uid 500); 3 Jun 2011 21:46:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15549 invoked by uid 99); 3 Jun 2011 21:46:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 21:46:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 21:46:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6EDCBF3E42
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 21:45:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 21:45:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1597399507.67325.1307137547450.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7358) Improve log levels when exceptions
 caught in RPC handler
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Improve log levels when exceptions caught in RPC handler
--------------------------------------------------------

                 Key: HADOOP-7358
                 URL: https://issues.apache.org/jira/browse/HADOOP-7358
             Project: Hadoop Common
          Issue Type: Improvement
          Components: ipc
    Affects Versions: 0.22.0
            Reporter: Todd Lipcon
            Priority: Minor
             Fix For: 0.22.0


When a server implementation throws an exception handling an RPC, the Handler thread catches it and logs it before responding with the exception over the IPC channel. This is currently done at INFO level.

I'd like to propose that, if the exception is an unchecked exception, it should be logged at WARN level instead. This can help alert operators when they might be hitting some kind of bug.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15845-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 21:52:29 2011
Return-Path: <common-issues-return-15845-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 193456681
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 21:52:29 +0000 (UTC)
Received: (qmail 24868 invoked by uid 500); 3 Jun 2011 21:52:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24840 invoked by uid 500); 3 Jun 2011 21:52:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24832 invoked by uid 99); 3 Jun 2011 21:52:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 21:52:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 21:52:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BCC55F3FBC
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 21:51:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 21:51:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <612451200.67332.1307137907769.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1597399507.67325.1307137547450.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7358) Improve log levels when exceptions
 caught in RPC handler
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7358:
--------------------------------

    Attachment: hadoop-7358.txt

Simple patch attached. No test case since it's just a log level change.

> Improve log levels when exceptions caught in RPC handler
> --------------------------------------------------------
>
>                 Key: HADOOP-7358
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7358
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7358.txt
>
>
> When a server implementation throws an exception handling an RPC, the Handler thread catches it and logs it before responding with the exception over the IPC channel. This is currently done at INFO level.
> I'd like to propose that, if the exception is an unchecked exception, it should be logged at WARN level instead. This can help alert operators when they might be hitting some kind of bug.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15846-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 21:54:29 2011
Return-Path: <common-issues-return-15846-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F06976DED
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 21:54:28 +0000 (UTC)
Received: (qmail 37479 invoked by uid 500); 3 Jun 2011 21:54:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37452 invoked by uid 500); 3 Jun 2011 21:54:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37443 invoked by uid 99); 3 Jun 2011 21:54:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 21:54:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 21:54:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8CAA2F30B7
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 21:53:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 21:53:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1776834545.67338.1307138027572.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <647927887.58055.1306890116067.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7346) Send back nicer error to clients
 using outdated IPC version
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044078#comment-13044078 ] 

Todd Lipcon commented on HADOOP-7346:
-------------------------------------

I'd like to commit this to 0.22 branch, since it really helps with version transitions, and I think 0.22 should see more uptake than 0.21 did.

It applies cleanly, the only issue is that it requires guava as a dependency in the tests. Anyone have an issue with my adding it to ivy? The other option is to not include the tests in the commit (or reimplement the two methods we need). Adding guava seems easiest and will also make it easier to backport other patches from trunk which might use it, so that' my preference.

> Send back nicer error to clients using outdated IPC version
> -----------------------------------------------------------
>
>                 Key: HADOOP-7346
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7346
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7346.txt, hadoop-7346.txt, hadoop-7346.txt
>
>
> When an older Hadoop version tries to contact a newer Hadoop version across an IPC protocol version bump, the client currently just gets a non-useful error message like "EOFException".
> Instead, the IPC server code can speak just enough of prior IPC protocols to send back a "fatal" message indicating the version mismatch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15848-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 21:56:31 2011
Return-Path: <common-issues-return-15848-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 880B76F1C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 21:56:31 +0000 (UTC)
Received: (qmail 42196 invoked by uid 500); 3 Jun 2011 21:56:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42128 invoked by uid 500); 3 Jun 2011 21:56:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42108 invoked by uid 99); 3 Jun 2011 21:56:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 21:56:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 21:56:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BADFCF3213
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 21:55:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 21:55:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <272832913.67353.1307138147761.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <647927887.58055.1306890116067.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7346) Send back nicer error to clients
 using outdated IPC version
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044081#comment-13044081 ] 

Hudson commented on HADOOP-7346:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #637 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/637/])
    HADOOP-7346. Send back nicer error message to clients using outdated IPC version. Contributed by Todd Lipcon.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1131254
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/ipc/Server.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/ipc/TestIPC.java


> Send back nicer error to clients using outdated IPC version
> -----------------------------------------------------------
>
>                 Key: HADOOP-7346
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7346
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7346.txt, hadoop-7346.txt, hadoop-7346.txt
>
>
> When an older Hadoop version tries to contact a newer Hadoop version across an IPC protocol version bump, the client currently just gets a non-useful error message like "EOFException".
> Instead, the IPC server code can speak just enough of prior IPC protocols to send back a "fatal" message indicating the version mismatch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15847-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 21:56:31 2011
Return-Path: <common-issues-return-15847-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A11CD6F25
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 21:56:31 +0000 (UTC)
Received: (qmail 42188 invoked by uid 500); 3 Jun 2011 21:56:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42126 invoked by uid 500); 3 Jun 2011 21:56:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42107 invoked by uid 99); 3 Jun 2011 21:56:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 21:56:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 21:56:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C68C1F3214
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 21:55:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 21:55:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <625182446.67354.1307138147809.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1597399507.67325.1307137547450.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7358) Improve log levels when exceptions
 caught in RPC handler
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044082#comment-13044082 ] 

Eli Collins commented on HADOOP-7358:
-------------------------------------

+1 pending hudson

> Improve log levels when exceptions caught in RPC handler
> --------------------------------------------------------
>
>                 Key: HADOOP-7358
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7358
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7358.txt
>
>
> When a server implementation throws an exception handling an RPC, the Handler thread catches it and logs it before responding with the exception over the IPC channel. This is currently done at INFO level.
> I'd like to propose that, if the exception is an unchecked exception, it should be logged at WARN level instead. This can help alert operators when they might be hitting some kind of bug.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15849-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 22:02:29 2011
Return-Path: <common-issues-return-15849-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DADC36EDB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 22:02:29 +0000 (UTC)
Received: (qmail 53332 invoked by uid 500); 3 Jun 2011 22:02:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53299 invoked by uid 500); 3 Jun 2011 22:02:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53290 invoked by uid 99); 3 Jun 2011 22:02:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 22:02:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 22:02:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1ECA8F33A9
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 22:01:48 +0000 (UTC)
Date: Fri, 3 Jun 2011 22:01:48 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1335100488.67366.1307138508123.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <531829213.32234.1305934307559.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7316) Add public javadocs to
 FSDataInputStream and FSDataOutputStream
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7316:
--------------------------------

    Attachment: hadoop-7316-3.patch

Thanks Todd. Updated patch attached.

> Add public javadocs to FSDataInputStream and FSDataOutputStream
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7316
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7316
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Jonathan Hsieh
>            Assignee: Eli Collins
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7316-1.patch, hadoop-7316-2.patch, hadoop-7316-3.patch
>
>
> This is a method made public for testing.  In comments in HADOOP-7301 after commit, adding javadoc comments was requested.  This is a follow up jira to address it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15850-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 22:04:30 2011
Return-Path: <common-issues-return-15850-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A7B816823
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 22:04:30 +0000 (UTC)
Received: (qmail 54422 invoked by uid 500); 3 Jun 2011 22:04:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54388 invoked by uid 500); 3 Jun 2011 22:04:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54380 invoked by uid 99); 3 Jun 2011 22:04:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 22:04:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 22:04:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 663C7F3457
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 22:03:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 22:03:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <690177636.67376.1307138627415.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <531829213.32234.1305934307559.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7316) Add public javadocs to
 FSDataInputStream and FSDataOutputStream
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7316:
--------------------------------

    Status: Patch Available  (was: Open)

> Add public javadocs to FSDataInputStream and FSDataOutputStream
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7316
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7316
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Jonathan Hsieh
>            Assignee: Eli Collins
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7316-1.patch, hadoop-7316-2.patch, hadoop-7316-3.patch
>
>
> This is a method made public for testing.  In comments in HADOOP-7301 after commit, adding javadoc comments was requested.  This is a follow up jira to address it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15851-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 22:10:30 2011
Return-Path: <common-issues-return-15851-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A06C068EB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 22:10:30 +0000 (UTC)
Received: (qmail 56978 invoked by uid 500); 3 Jun 2011 22:10:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56952 invoked by uid 500); 3 Jun 2011 22:10:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56944 invoked by uid 99); 3 Jun 2011 22:10:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 22:10:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 22:10:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5C63EF35C3
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 22:09:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 22:09:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <74123397.67390.1307138987374.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <531829213.32234.1305934307559.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7316) Add public javadocs to
 FSDataInputStream and FSDataOutputStream
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044088#comment-13044088 ] 

Todd Lipcon commented on HADOOP-7316:
-------------------------------------

+1 pending hudson

> Add public javadocs to FSDataInputStream and FSDataOutputStream
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7316
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7316
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Jonathan Hsieh
>            Assignee: Eli Collins
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7316-1.patch, hadoop-7316-2.patch, hadoop-7316-3.patch
>
>
> This is a method made public for testing.  In comments in HADOOP-7301 after commit, adding javadoc comments was requested.  This is a follow up jira to address it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15852-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 22:21:28 2011
Return-Path: <common-issues-return-15852-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A7A7B6088
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 22:21:28 +0000 (UTC)
Received: (qmail 71463 invoked by uid 500); 3 Jun 2011 22:21:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71441 invoked by uid 500); 3 Jun 2011 22:21:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71433 invoked by uid 99); 3 Jun 2011 22:21:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 22:21:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 22:21:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5D4E1F38AD
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 22:20:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 22:20:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <719129071.67407.1307139647378.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1470673003.56543.1306860527397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7341) Fix option parsing in
 CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044092#comment-13044092 ] 

Aaron T. Myers commented on HADOOP-7341:
----------------------------------------

+1, looks good to me.

One nit: I'd recommend either removing the "@deprecated name is never used" from the comment, or changing the name of the parameter from "n" to "name". Outside of the context of this patch, the comment doesn't make much sense.

> Fix option parsing in CommandFormat
> -----------------------------------
>
>                 Key: HADOOP-7341
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7341
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7341-2.patch, HADOOP-7341.patch
>
>
> CommandFormat currently allows options in any location within the args.  This is not the intended behavior for FsShell commands.  Prior to the redesign, the commands used to expect option processing to stop at the first non-option.
> CommandFormat was an existing class prior the redesign, but it was only used by "count" to find the -q flag.  All commands were converted to using this class, thus inherited the unintended behavior.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15853-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 22:25:28 2011
Return-Path: <common-issues-return-15853-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D302A6141
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 22:25:28 +0000 (UTC)
Received: (qmail 79452 invoked by uid 500); 3 Jun 2011 22:25:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79425 invoked by uid 500); 3 Jun 2011 22:25:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79417 invoked by uid 99); 3 Jun 2011 22:25:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 22:25:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 22:25:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7EA13F39CB
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 22:24:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 22:24:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <6389285.67420.1307139887515.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <531829213.32234.1305934307559.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7316) Add public javadocs to
 FSDataInputStream and FSDataOutputStream
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044094#comment-13044094 ] 

Hadoop QA commented on HADOOP-7316:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481412/hadoop-7316-3.patch
  against trunk revision 1131254.

    +1 @author.  The patch does not contain any @author tags.

    +0 tests included.  The patch appears to be a documentation patch that doesn't require tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/575//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/575//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/575//console

This message is automatically generated.

> Add public javadocs to FSDataInputStream and FSDataOutputStream
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7316
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7316
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Jonathan Hsieh
>            Assignee: Eli Collins
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7316-1.patch, hadoop-7316-2.patch, hadoop-7316-3.patch
>
>
> This is a method made public for testing.  In comments in HADOOP-7301 after commit, adding javadoc comments was requested.  This is a follow up jira to address it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15854-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 22:27:30 2011
Return-Path: <common-issues-return-15854-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D4A766226
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 22:27:30 +0000 (UTC)
Received: (qmail 83471 invoked by uid 500); 3 Jun 2011 22:27:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83442 invoked by uid 500); 3 Jun 2011 22:27:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83434 invoked by uid 99); 3 Jun 2011 22:27:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 22:27:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 22:27:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6B2FEF3A75
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 22:26:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 22:26:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <958558303.67427.1307140007435.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <531829213.32234.1305934307559.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7316) Add public javadocs to
 FSDataInputStream and FSDataOutputStream
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7316:
--------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

Thanks Todd! I've committed this.

> Add public javadocs to FSDataInputStream and FSDataOutputStream
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7316
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7316
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Jonathan Hsieh
>            Assignee: Eli Collins
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7316-1.patch, hadoop-7316-2.patch, hadoop-7316-3.patch
>
>
> This is a method made public for testing.  In comments in HADOOP-7301 after commit, adding javadoc comments was requested.  This is a follow up jira to address it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15855-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 22:46:29 2011
Return-Path: <common-issues-return-15855-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1F2B967AB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 22:46:29 +0000 (UTC)
Received: (qmail 9046 invoked by uid 500); 3 Jun 2011 22:46:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9019 invoked by uid 500); 3 Jun 2011 22:46:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9011 invoked by uid 99); 3 Jun 2011 22:46:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 22:46:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 22:46:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C0B60F316F
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 22:45:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 22:45:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1853364238.67490.1307141147786.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <531829213.32234.1305934307559.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7316) Add public javadocs to
 FSDataInputStream and FSDataOutputStream
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044109#comment-13044109 ] 

Hudson commented on HADOOP-7316:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #638 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/638/])
    HADOOP-7316. Add public javadocs to FSDataInputStream and FSDataOutputStream. Contributed by Eli Collins

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1131268
Files : 
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/FsCommand.java
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Command.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/PositionedReadable.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/FSDataOutputStream.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/FSDataInputStream.java


> Add public javadocs to FSDataInputStream and FSDataOutputStream
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7316
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7316
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Jonathan Hsieh
>            Assignee: Eli Collins
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7316-1.patch, hadoop-7316-2.patch, hadoop-7316-3.patch
>
>
> This is a method made public for testing.  In comments in HADOOP-7301 after commit, adding javadoc comments was requested.  This is a follow up jira to address it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15856-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 23:22:31 2011
Return-Path: <common-issues-return-15856-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 30984695D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 23:22:31 +0000 (UTC)
Received: (qmail 50663 invoked by uid 500); 3 Jun 2011 23:22:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50630 invoked by uid 500); 3 Jun 2011 23:22:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50622 invoked by uid 99); 3 Jun 2011 23:22:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 23:22:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 23:22:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 760A5F3999
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 23:21:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 23:21:47 +0000 (UTC)
From: "John George (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <820092696.67537.1307143307480.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <531829213.32234.1305934307559.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7316) Add public javadocs to
 FSDataInputStream and FSDataOutputStream
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044121#comment-13044121 ] 

John George commented on HADOOP-7316:
-------------------------------------

Just to understand this a little better:
Is the read() function also not expected to throw an EOF Exception if the user tried to read beyond EOF? 

> Add public javadocs to FSDataInputStream and FSDataOutputStream
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7316
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7316
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Jonathan Hsieh
>            Assignee: Eli Collins
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7316-1.patch, hadoop-7316-2.patch, hadoop-7316-3.patch
>
>
> This is a method made public for testing.  In comments in HADOOP-7301 after commit, adding javadoc comments was requested.  This is a follow up jira to address it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15857-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 23:24:28 2011
Return-Path: <common-issues-return-15857-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C6CA669A0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 23:24:28 +0000 (UTC)
Received: (qmail 52582 invoked by uid 500); 3 Jun 2011 23:24:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52554 invoked by uid 500); 3 Jun 2011 23:24:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52546 invoked by uid 99); 3 Jun 2011 23:24:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 23:24:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 23:24:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8B380F3A11
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 23:23:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 23:23:47 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <607210568.67544.1307143427567.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1043064090.64591.1307057267396.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7356) RPM packages broke bin/hadoop script
 for hadoop 0.20.205
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-7356:
------------------------------

    Attachment: HADOOP-7356-1.patch

Fixed incorrect file references.

> RPM packages broke bin/hadoop script for hadoop 0.20.205
> --------------------------------------------------------
>
>                 Key: HADOOP-7356
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7356
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.204.0
>         Environment: Java 6, Redhat EL 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>         Attachments: HADOOP-7356-1.patch, HADOOP-7356.patch
>
>
> hadoop-config.sh has been moved to libexec for binary package, but developers prefers to have hadoop-config.sh in bin.  Hadoo shell scripts should be modified to support both scenarios.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15858-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 23:34:28 2011
Return-Path: <common-issues-return-15858-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D943862C8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 23:34:28 +0000 (UTC)
Received: (qmail 65396 invoked by uid 500); 3 Jun 2011 23:34:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65371 invoked by uid 500); 3 Jun 2011 23:34:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65363 invoked by uid 99); 3 Jun 2011 23:34:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 23:34:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 23:34:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5C25FF3CBB
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 23:33:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 23:33:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1559119365.67568.1307144027374.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1043064090.64591.1307057267396.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7356) RPM packages broke bin/hadoop
 script for hadoop 0.20.205
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044128#comment-13044128 ] 

Hadoop QA commented on HADOOP-7356:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481416/HADOOP-7356-1.patch
  against trunk revision 1131268.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/576//console

This message is automatically generated.

> RPM packages broke bin/hadoop script for hadoop 0.20.205
> --------------------------------------------------------
>
>                 Key: HADOOP-7356
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7356
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.204.0
>         Environment: Java 6, Redhat EL 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>         Attachments: HADOOP-7356-1.patch, HADOOP-7356.patch
>
>
> hadoop-config.sh has been moved to libexec for binary package, but developers prefers to have hadoop-config.sh in bin.  Hadoo shell scripts should be modified to support both scenarios.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15859-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 23:42:30 2011
Return-Path: <common-issues-return-15859-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 17BE46490
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 23:42:30 +0000 (UTC)
Received: (qmail 71890 invoked by uid 500); 3 Jun 2011 23:42:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71860 invoked by uid 500); 3 Jun 2011 23:42:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71851 invoked by uid 99); 3 Jun 2011 23:42:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 23:42:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 23:42:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 89400F3EDF
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 23:41:48 +0000 (UTC)
Date: Fri, 3 Jun 2011 23:41:48 +0000 (UTC)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <177257759.67588.1307144508557.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-7144:
----------------------------------

       Resolution: Fixed
    Fix Version/s: 0.20.205.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

+1

I committed this to trunk and the 20-security branch. Thanks, Robert!

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Robert Joseph Evans
>              Labels: jmx
>             Fix For: 0.20.205.0, 0.23.0
>
>         Attachments: HADOOP-7411-0.20.20X-V1.patch, HADOOP-7411-0.20.20X-V2.patch, HADOOP-7411-trunk-V1.patch, HADOOP-7411-trunk-V2.patch, HADOOP-7411-trunk-alpha.patch, jmx.json
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15860-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 23:44:30 2011
Return-Path: <common-issues-return-15860-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D223C65CD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 23:44:30 +0000 (UTC)
Received: (qmail 75621 invoked by uid 500); 3 Jun 2011 23:44:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75576 invoked by uid 500); 3 Jun 2011 23:44:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75568 invoked by uid 99); 3 Jun 2011 23:44:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 23:44:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 23:44:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 82D30F3F50
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 23:43:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 23:43:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1637771486.67604.1307144627532.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <531829213.32234.1305934307559.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7316) Add public javadocs to
 FSDataInputStream and FSDataOutputStream
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044134#comment-13044134 ] 

Eli Collins commented on HADOOP-7316:
-------------------------------------

Depends on the underlying InputStream. read (and readFully) result in a call to seek, which in the case of DFSInputStream throws an IOException if you try to seek past the length of the files. FSInputStream#readFully however throws an EOFException if the underlying call to read failed while FSInputStream#read will return whatever value the underlying call to read returned (will not throw an EOFException). So, per the javadoc, you only get the EOFException with readFully if you hit the EOF while reading.

DFSInputStream#seek should really throw an EOFException if seeking beyond the end of stream, but changing it to do that now would break compatibility =(

> Add public javadocs to FSDataInputStream and FSDataOutputStream
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7316
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7316
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Jonathan Hsieh
>            Assignee: Eli Collins
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7316-1.patch, hadoop-7316-2.patch, hadoop-7316-3.patch
>
>
> This is a method made public for testing.  In comments in HADOOP-7301 after commit, adding javadoc comments was requested.  This is a follow up jira to address it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15861-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 23:53:31 2011
Return-Path: <common-issues-return-15861-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 242D06D20
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 23:53:31 +0000 (UTC)
Received: (qmail 85198 invoked by uid 500); 3 Jun 2011 23:53:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85173 invoked by uid 500); 3 Jun 2011 23:53:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85165 invoked by uid 99); 3 Jun 2011 23:53:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 23:53:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 23:53:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 92C53F31B6
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 23:52:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 23:52:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1086511869.67627.1307145167598.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <531829213.32234.1305934307559.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7316) Add public javadocs to
 FSDataInputStream and FSDataOutputStream
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044138#comment-13044138 ] 

Todd Lipcon commented on HADOOP-7316:
-------------------------------------

bq. DFSInputStream#seek should really throw an EOFException if seeking beyond the end of stream, but changing it to do that now would break compatibility =(

Actually, Java's RandomAccessFile API lets you seek way past the end of a file with no exception. The next read() call will simply return -1. But asking for "position()" on the file's channel continues to return the past-eof offset.

> Add public javadocs to FSDataInputStream and FSDataOutputStream
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7316
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7316
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Jonathan Hsieh
>            Assignee: Eli Collins
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7316-1.patch, hadoop-7316-2.patch, hadoop-7316-3.patch
>
>
> This is a method made public for testing.  In comments in HADOOP-7301 after commit, adding javadoc comments was requested.  This is a follow up jira to address it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15862-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 23:55:29 2011
Return-Path: <common-issues-return-15862-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0DC236E71
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 23:55:29 +0000 (UTC)
Received: (qmail 87399 invoked by uid 500); 3 Jun 2011 23:55:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87371 invoked by uid 500); 3 Jun 2011 23:55:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87363 invoked by uid 99); 3 Jun 2011 23:55:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 23:55:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 23:55:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A4CF8F323D
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 23:54:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 23:54:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1954153315.67641.1307145287671.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044140#comment-13044140 ] 

Hudson commented on HADOOP-7144:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #639 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/639/])
    HADOOP-7144. Expose JMX metrics via JSON servlet.
Contributed by Robert Joseph Evans

cdouglas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1131289
Files : 
* /hadoop/common/trunk/src/java/org/apache/hadoop/http/HttpServer.java
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/jmx
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/jmx/TestJMXJsonServlet.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/jmx/JMXJsonServlet.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/jmx/package-info.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/jmx


> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Robert Joseph Evans
>              Labels: jmx
>             Fix For: 0.20.205.0, 0.23.0
>
>         Attachments: HADOOP-7411-0.20.20X-V1.patch, HADOOP-7411-0.20.20X-V2.patch, HADOOP-7411-trunk-V1.patch, HADOOP-7411-trunk-V2.patch, HADOOP-7411-trunk-alpha.patch, jmx.json
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15863-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun  3 23:59:29 2011
Return-Path: <common-issues-return-15863-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 26350603E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  3 Jun 2011 23:59:29 +0000 (UTC)
Received: (qmail 93113 invoked by uid 500); 3 Jun 2011 23:59:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93076 invoked by uid 500); 3 Jun 2011 23:59:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93068 invoked by uid 99); 3 Jun 2011 23:59:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 23:59:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Jun 2011 23:59:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D482DF33D8
	for <common-issues@hadoop.apache.org>; Fri,  3 Jun 2011 23:58:47 +0000 (UTC)
Date: Fri, 3 Jun 2011 23:58:47 +0000 (UTC)
From: "John George (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1246510294.67670.1307145527867.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <531829213.32234.1305934307559.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7316) Add public javadocs to
 FSDataInputStream and FSDataOutputStream
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044144#comment-13044144 ] 

John George commented on HADOOP-7316:
-------------------------------------

Thanks a lot Todd and Eli.
Is it okay if I add this as a comment to DFSInputStream at some point? Because Daryn and I were both confused about the behavior and the above information really helps.

> Add public javadocs to FSDataInputStream and FSDataOutputStream
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7316
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7316
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Jonathan Hsieh
>            Assignee: Eli Collins
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7316-1.patch, hadoop-7316-2.patch, hadoop-7316-3.patch
>
>
> This is a method made public for testing.  In comments in HADOOP-7301 after commit, adding javadoc comments was requested.  This is a follow up jira to address it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15864-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun  4 00:03:31 2011
Return-Path: <common-issues-return-15864-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4DF60633B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat,  4 Jun 2011 00:03:31 +0000 (UTC)
Received: (qmail 679 invoked by uid 500); 4 Jun 2011 00:03:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 649 invoked by uid 500); 4 Jun 2011 00:03:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 641 invoked by uid 99); 4 Jun 2011 00:03:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 00:03:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 00:03:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0322CF358F
	for <common-issues@hadoop.apache.org>; Sat,  4 Jun 2011 00:02:48 +0000 (UTC)
Date: Sat, 4 Jun 2011 00:02:48 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <180606880.67691.1307145768009.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <531829213.32234.1305934307559.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7316) Add public javadocs to
 FSDataInputStream and FSDataOutputStream
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044148#comment-13044148 ] 

Eli Collins commented on HADOOP-7316:
-------------------------------------

Go for it!

> Add public javadocs to FSDataInputStream and FSDataOutputStream
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7316
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7316
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Jonathan Hsieh
>            Assignee: Eli Collins
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7316-1.patch, hadoop-7316-2.patch, hadoop-7316-3.patch
>
>
> This is a method made public for testing.  In comments in HADOOP-7301 after commit, adding javadoc comments was requested.  This is a follow up jira to address it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15865-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun  4 01:13:29 2011
Return-Path: <common-issues-return-15865-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1846A6B03
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat,  4 Jun 2011 01:13:29 +0000 (UTC)
Received: (qmail 64803 invoked by uid 500); 4 Jun 2011 01:13:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64761 invoked by uid 500); 4 Jun 2011 01:13:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64610 invoked by uid 99); 4 Jun 2011 01:13:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 01:13:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 01:13:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 70D9EE1175
	for <common-issues@hadoop.apache.org>; Sat,  4 Jun 2011 01:12:47 +0000 (UTC)
Date: Sat, 4 Jun 2011 01:12:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <563852709.67776.1307149967458.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <531829213.32234.1305934307559.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7316) Add public javadocs to
 FSDataInputStream and FSDataOutputStream
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044158#comment-13044158 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7316:
------------------------------------------------

> ... Because Daryn and I were both confused about the behavior ...

Hi John, I think the current behavior is not well designed.  If it is the case, you may also fix it.

> Add public javadocs to FSDataInputStream and FSDataOutputStream
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7316
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7316
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Jonathan Hsieh
>            Assignee: Eli Collins
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7316-1.patch, hadoop-7316-2.patch, hadoop-7316-3.patch
>
>
> This is a method made public for testing.  In comments in HADOOP-7301 after commit, adding javadoc comments was requested.  This is a follow up jira to address it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15866-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun  4 02:09:29 2011
Return-Path: <common-issues-return-15866-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0B0E94D67
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat,  4 Jun 2011 02:09:29 +0000 (UTC)
Received: (qmail 94679 invoked by uid 500); 4 Jun 2011 02:09:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94653 invoked by uid 500); 4 Jun 2011 02:09:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94645 invoked by uid 99); 4 Jun 2011 02:09:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 02:09:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 02:09:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 733A8E1C9C
	for <common-issues@hadoop.apache.org>; Sat,  4 Jun 2011 02:08:47 +0000 (UTC)
Date: Sat, 4 Jun 2011 02:08:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1647883290.67848.1307153327468.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <314496369.49888.1306530467371.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7335) Force entropy to come from
 non-true random for tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044167#comment-13044167 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7335:
------------------------------------------------

> I'd appreciate if someone can test this on Windows/OSX ...

Tried {{TestHttpServer}} and {{TestServletFilter}}.  Seems working fine.

> Force entropy to come from non-true random for tests
> ----------------------------------------------------
>
>                 Key: HADOOP-7335
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7335
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7335.txt
>
>
> Passing the system property {{-Djava.security.egd=file:///dev/urandom}} forces the JVM to seed its PRNG from non-true random (/dev/urandom) instead of the true random (/dev/random). This makes the tests run faster, since without it they often hang waiting for entropy while Jetty is initializing.
> We should turn this on for the test targets by default, so developers/hudson boxes don't have to make this change system-wide or use workarounds like rngtools.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15867-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun  4 02:09:31 2011
Return-Path: <common-issues-return-15867-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0AACE4D74
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat,  4 Jun 2011 02:09:31 +0000 (UTC)
Received: (qmail 94969 invoked by uid 500); 4 Jun 2011 02:09:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94913 invoked by uid 500); 4 Jun 2011 02:09:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94905 invoked by uid 99); 4 Jun 2011 02:09:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 02:09:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 02:09:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 88A5BE1CA2
	for <common-issues@hadoop.apache.org>; Sat,  4 Jun 2011 02:08:47 +0000 (UTC)
Date: Sat, 4 Jun 2011 02:08:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <586366198.67850.1307153327556.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <314496369.49888.1306530467371.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7335) Force entropy to come from
 non-true random for tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044168#comment-13044168 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7335:
------------------------------------------------

I mean tried on Windows/Cygwin.

> Force entropy to come from non-true random for tests
> ----------------------------------------------------
>
>                 Key: HADOOP-7335
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7335
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7335.txt
>
>
> Passing the system property {{-Djava.security.egd=file:///dev/urandom}} forces the JVM to seed its PRNG from non-true random (/dev/urandom) instead of the true random (/dev/random). This makes the tests run faster, since without it they often hang waiting for entropy while Jetty is initializing.
> We should turn this on for the test targets by default, so developers/hudson boxes don't have to make this change system-wide or use workarounds like rngtools.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15868-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun  4 04:11:36 2011
Return-Path: <common-issues-return-15868-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3F3B04DF9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat,  4 Jun 2011 04:11:36 +0000 (UTC)
Received: (qmail 69214 invoked by uid 500); 4 Jun 2011 04:11:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67886 invoked by uid 500); 4 Jun 2011 04:11:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67871 invoked by uid 99); 4 Jun 2011 04:11:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 04:11:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 04:11:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E2661F3078
	for <common-issues@hadoop.apache.org>; Sat,  4 Jun 2011 04:10:47 +0000 (UTC)
Date: Sat, 4 Jun 2011 04:10:47 +0000 (UTC)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2001738969.68013.1307160647923.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Assigned] (HADOOP-7106) Re-organize hadoop subversion
 layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nigel Daley reassigned HADOOP-7106:
-----------------------------------

    Assignee: Todd Lipcon  (was: Nigel Daley)

Todd agreed to take over this issue and coordinate with Jukka. 

> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15869-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun  4 04:37:31 2011
Return-Path: <common-issues-return-15869-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F0F744E74
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat,  4 Jun 2011 04:37:30 +0000 (UTC)
Received: (qmail 81653 invoked by uid 500); 4 Jun 2011 04:37:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81565 invoked by uid 500); 4 Jun 2011 04:37:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81556 invoked by uid 99); 4 Jun 2011 04:37:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 04:37:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 04:37:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 62214F355B
	for <common-issues@hadoop.apache.org>; Sat,  4 Jun 2011 04:36:47 +0000 (UTC)
Date: Sat, 4 Jun 2011 04:36:47 +0000 (UTC)
From: "John George (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1148988945.68061.1307162207398.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <531829213.32234.1305934307559.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7316) Add public javadocs to
 FSDataInputStream and FSDataOutputStream
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044198#comment-13044198 ] 

John George commented on HADOOP-7316:
-------------------------------------

>> ... Because Daryn and I were both confused about the behavior ...

> Hi John, I think the current behavior is not well designed. If it is the case, you may also fix it.

Thanks Nicholas. I created HDFS-2033 to track read path design.

> Add public javadocs to FSDataInputStream and FSDataOutputStream
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7316
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7316
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Jonathan Hsieh
>            Assignee: Eli Collins
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7316-1.patch, hadoop-7316-2.patch, hadoop-7316-3.patch
>
>
> This is a method made public for testing.  In comments in HADOOP-7301 after commit, adding javadoc comments was requested.  This is a follow up jira to address it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15870-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun  4 06:54:32 2011
Return-Path: <common-issues-return-15870-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2884C4CCA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat,  4 Jun 2011 06:54:32 +0000 (UTC)
Received: (qmail 30384 invoked by uid 500); 4 Jun 2011 06:54:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30285 invoked by uid 500); 4 Jun 2011 06:54:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30277 invoked by uid 99); 4 Jun 2011 06:54:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 06:54:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 06:54:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 615B7F480A
	for <common-issues@hadoop.apache.org>; Sat,  4 Jun 2011 06:53:47 +0000 (UTC)
Date: Sat, 4 Jun 2011 06:53:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <190446662.68137.1307170427395.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <397648942.57126.1306869588398.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7342) Add an utility API in FileUtil for
 JDK File.list
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated HADOOP-7342:
-------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

Committed to trunk.  Thanks, Bharath!  And thanks Cos for helping review.

> Add an utility API in FileUtil for JDK File.list
> ------------------------------------------------
>
>                 Key: HADOOP-7342
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7342
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7342-1.patch, HADOOP-7342-2.patch
>
>
> Java File.list API can return null when disk is bad or directory is not a directory. This utility API in FileUtil will throw an exception when this happens rather than returning null. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15871-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun  4 11:16:30 2011
Return-Path: <common-issues-return-15871-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B4DD4474B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat,  4 Jun 2011 11:16:30 +0000 (UTC)
Received: (qmail 7659 invoked by uid 500); 4 Jun 2011 11:16:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7605 invoked by uid 500); 4 Jun 2011 11:16:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7591 invoked by uid 99); 4 Jun 2011 11:16:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 11:16:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 11:16:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 29312F4522
	for <common-issues@hadoop.apache.org>; Sat,  4 Jun 2011 11:15:48 +0000 (UTC)
Date: Sat, 4 Jun 2011 11:15:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <531037777.68390.1307186148165.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <647927887.58055.1306890116067.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7346) Send back nicer error to clients
 using outdated IPC version
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044275#comment-13044275 ] 

Hudson commented on HADOOP-7346:
--------------------------------

Integrated in Hadoop-Common-trunk #709 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/709/])
    HADOOP-7346. Send back nicer error message to clients using outdated IPC version. Contributed by Todd Lipcon.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1131254
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/ipc/Server.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/ipc/TestIPC.java


> Send back nicer error to clients using outdated IPC version
> -----------------------------------------------------------
>
>                 Key: HADOOP-7346
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7346
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7346.txt, hadoop-7346.txt, hadoop-7346.txt
>
>
> When an older Hadoop version tries to contact a newer Hadoop version across an IPC protocol version bump, the client currently just gets a non-useful error message like "EOFException".
> Instead, the IPC server code can speak just enough of prior IPC protocols to send back a "fatal" message indicating the version mismatch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15874-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun  4 11:16:31 2011
Return-Path: <common-issues-return-15874-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0FD754759
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat,  4 Jun 2011 11:16:31 +0000 (UTC)
Received: (qmail 7956 invoked by uid 500); 4 Jun 2011 11:16:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7713 invoked by uid 500); 4 Jun 2011 11:16:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7621 invoked by uid 99); 4 Jun 2011 11:16:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 11:16:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 11:16:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1781BF452D
	for <common-issues@hadoop.apache.org>; Sat,  4 Jun 2011 11:15:49 +0000 (UTC)
Date: Sat, 4 Jun 2011 11:15:49 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1514183650.68401.1307186149092.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <397648942.57126.1306869588398.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7342) Add an utility API in FileUtil for
 JDK File.list
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044277#comment-13044277 ] 

Hudson commented on HADOOP-7342:
--------------------------------

Integrated in Hadoop-Common-trunk #709 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/709/])
    HADOOP-7342. Add an utility API in FileUtil for JDK File.list avoid NPEs on File.list().  Contributed by Bharath Mundlapudi.

mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1131330
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/FileUtil.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/TestFileUtil.java


> Add an utility API in FileUtil for JDK File.list
> ------------------------------------------------
>
>                 Key: HADOOP-7342
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7342
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7342-1.patch, HADOOP-7342-2.patch
>
>
> Java File.list API can return null when disk is bad or directory is not a directory. This utility API in FileUtil will throw an exception when this happens rather than returning null. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15872-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun  4 11:16:31 2011
Return-Path: <common-issues-return-15872-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4712E477B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat,  4 Jun 2011 11:16:31 +0000 (UTC)
Received: (qmail 7698 invoked by uid 500); 4 Jun 2011 11:16:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7630 invoked by uid 500); 4 Jun 2011 11:16:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7581 invoked by uid 99); 4 Jun 2011 11:16:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 11:16:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 11:16:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 010B6F451D
	for <common-issues@hadoop.apache.org>; Sat,  4 Jun 2011 11:15:48 +0000 (UTC)
Date: Sat, 4 Jun 2011 11:15:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1218761597.68386.1307186148001.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044274#comment-13044274 ] 

Hudson commented on HADOOP-7144:
--------------------------------

Integrated in Hadoop-Common-trunk #709 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/709/])
    HADOOP-7144. Expose JMX metrics via JSON servlet.
Contributed by Robert Joseph Evans

cdouglas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1131289
Files : 
* /hadoop/common/trunk/src/java/org/apache/hadoop/http/HttpServer.java
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/jmx
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/jmx/TestJMXJsonServlet.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/jmx/JMXJsonServlet.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/jmx/package-info.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/jmx


> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Robert Joseph Evans
>              Labels: jmx
>             Fix For: 0.20.205.0, 0.23.0
>
>         Attachments: HADOOP-7411-0.20.20X-V1.patch, HADOOP-7411-0.20.20X-V2.patch, HADOOP-7411-trunk-V1.patch, HADOOP-7411-trunk-V2.patch, HADOOP-7411-trunk-alpha.patch, jmx.json
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15873-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun  4 11:16:31 2011
Return-Path: <common-issues-return-15873-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 94603478E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat,  4 Jun 2011 11:16:31 +0000 (UTC)
Received: (qmail 7820 invoked by uid 500); 4 Jun 2011 11:16:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7663 invoked by uid 500); 4 Jun 2011 11:16:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7613 invoked by uid 99); 4 Jun 2011 11:16:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 11:16:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 11:16:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id ED551F4529
	for <common-issues@hadoop.apache.org>; Sat,  4 Jun 2011 11:15:48 +0000 (UTC)
Date: Sat, 4 Jun 2011 11:15:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <889828892.68397.1307186148968.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <531829213.32234.1305934307559.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7316) Add public javadocs to
 FSDataInputStream and FSDataOutputStream
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044276#comment-13044276 ] 

Hudson commented on HADOOP-7316:
--------------------------------

Integrated in Hadoop-Common-trunk #709 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/709/])
    HADOOP-7316. Add public javadocs to FSDataInputStream and FSDataOutputStream. Contributed by Eli Collins

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1131268
Files : 
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/FsCommand.java
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Command.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/PositionedReadable.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/FSDataOutputStream.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/FSDataInputStream.java


> Add public javadocs to FSDataInputStream and FSDataOutputStream
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7316
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7316
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Jonathan Hsieh
>            Assignee: Eli Collins
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7316-1.patch, hadoop-7316-2.patch, hadoop-7316-3.patch
>
>
> This is a method made public for testing.  In comments in HADOOP-7301 after commit, adding javadoc comments was requested.  This is a follow up jira to address it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15875-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun  4 17:26:32 2011
Return-Path: <common-issues-return-15875-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4120D4355
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat,  4 Jun 2011 17:26:32 +0000 (UTC)
Received: (qmail 49953 invoked by uid 500); 4 Jun 2011 17:26:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49903 invoked by uid 500); 4 Jun 2011 17:26:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49859 invoked by uid 99); 4 Jun 2011 17:26:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 17:26:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 17:26:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BA372F4369
	for <common-issues@hadoop.apache.org>; Sat,  4 Jun 2011 17:25:47 +0000 (UTC)
Date: Sat, 4 Jun 2011 17:25:47 +0000 (UTC)
From: "Travis Crawford (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <710160053.68701.1307208347759.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7359) Pluggable interface for cluster
 membership
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Pluggable interface for cluster membership
------------------------------------------

                 Key: HADOOP-7359
                 URL: https://issues.apache.org/jira/browse/HADOOP-7359
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Travis Crawford


Currently Hadoop uses local files to determine cluster membership. With HDFS for example, dfs.hosts and dfs.hosts.exclude are used.

To enable tighter integrations cluster membership should be an interface, with the current file-based functionality provided as the default implementation. The common case would be no functional change, however, sites could plug an alternative implementation in, such as pulling the machine lists from a machine database.

DETAILS:

Two machine lists, includes and excludes, are used to define cluster membership and state. HostsFileReader currently handles reading these lists from files, who's names are passed in by FSNamesystem for HDFS and JobTracker for MR.

The proposed change is adding a HostsReader interface to common, and changing HostsFileReader to an abstract class that functions the same as today.

Two new classes, DFSHostsFileReader and MRHostsFileReader, extend HostsFileReader and simply pass the appropriate file names in. These new classes are needed because config key names live outside common.

Two new conf keys, defaulting to the file-based readers, would be added to choose a different hosts reader: dfs.namenode.hosts.reader.class mapreduce.jobtracker.hosts.reader.class

Comments/suggestions? I have most of this written already but would love some feedback on the general idea before posting the diff.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15876-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun  4 20:26:28 2011
Return-Path: <common-issues-return-15876-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CE5874DE7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat,  4 Jun 2011 20:26:28 +0000 (UTC)
Received: (qmail 34522 invoked by uid 500); 4 Jun 2011 20:26:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34491 invoked by uid 500); 4 Jun 2011 20:26:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34483 invoked by uid 99); 4 Jun 2011 20:26:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 20:26:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 20:26:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5C9AE1005BD
	for <common-issues@hadoop.apache.org>; Sat,  4 Jun 2011 20:25:47 +0000 (UTC)
Date: Sat, 4 Jun 2011 20:25:47 +0000 (UTC)
From: "Niels Basjes (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <860037502.68989.1307219147375.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1008788632.26385.1305807047638.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7305) Eclipse project files are incomplete
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Niels Basjes updated HADOOP-7305:
---------------------------------

    Attachment: HADOOP-7305-2011-05-30.patch

This is an updated patch to fix the problems that were reported on OS X.
I've tried to make this as "fail safe" as possible. 
However do not have access to OS X to try it out.

This should be tested by others before comitting.

> Eclipse project files are incomplete
> ------------------------------------
>
>                 Key: HADOOP-7305
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7305
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Niels Basjes
>            Assignee: Niels Basjes
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7305-2011-05-19.patch, HADOOP-7305-2011-05-30.patch
>
>
> After a fresh checkout of hadoop-common I do 'ant compile eclipse'.
> I open eclipse, set ANT_HOME and build the project. 
> At that point the following error appears:
> {quote}
> The type com.sun.javadoc.RootDoc cannot be resolved. It is indirectly referenced from required .class files	ExcludePrivateAnnotationsJDiffDoclet.java	/common/src/java/org/apache/hadoop/classification/tools	line 1	Java Problem
> {quote}
> The solution is to add the "tools.jar" from the JDK to the buildpath/classpath.
> This should be fixed in the build.xml.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15877-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun  4 20:40:29 2011
Return-Path: <common-issues-return-15877-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 01DEE4547
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat,  4 Jun 2011 20:40:29 +0000 (UTC)
Received: (qmail 42002 invoked by uid 500); 4 Jun 2011 20:40:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41970 invoked by uid 500); 4 Jun 2011 20:40:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41962 invoked by uid 99); 4 Jun 2011 20:40:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 20:40:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 20:40:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9CA85100771
	for <common-issues@hadoop.apache.org>; Sat,  4 Jun 2011 20:39:47 +0000 (UTC)
Date: Sat, 4 Jun 2011 20:39:47 +0000 (UTC)
From: "Niels Basjes (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <257999243.69000.1307219987638.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1008788632.26385.1305807047638.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Reopened] (HADOOP-7305) Eclipse project files are
 incomplete
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Niels Basjes reopened HADOOP-7305:
----------------------------------


Apparently there are some issues with the first version in combination with OS X. 

> Eclipse project files are incomplete
> ------------------------------------
>
>                 Key: HADOOP-7305
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7305
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Niels Basjes
>            Assignee: Niels Basjes
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7305-2011-05-19.patch, HADOOP-7305-2011-05-30.patch
>
>
> After a fresh checkout of hadoop-common I do 'ant compile eclipse'.
> I open eclipse, set ANT_HOME and build the project. 
> At that point the following error appears:
> {quote}
> The type com.sun.javadoc.RootDoc cannot be resolved. It is indirectly referenced from required .class files	ExcludePrivateAnnotationsJDiffDoclet.java	/common/src/java/org/apache/hadoop/classification/tools	line 1	Java Problem
> {quote}
> The solution is to add the "tools.jar" from the JDK to the buildpath/classpath.
> This should be fixed in the build.xml.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15878-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun  4 20:42:30 2011
Return-Path: <common-issues-return-15878-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ADB0245DC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat,  4 Jun 2011 20:42:30 +0000 (UTC)
Received: (qmail 45325 invoked by uid 500); 4 Jun 2011 20:42:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45302 invoked by uid 500); 4 Jun 2011 20:42:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45294 invoked by uid 99); 4 Jun 2011 20:42:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 20:42:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 20:42:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 632DC1007C2
	for <common-issues@hadoop.apache.org>; Sat,  4 Jun 2011 20:41:47 +0000 (UTC)
Date: Sat, 4 Jun 2011 20:41:47 +0000 (UTC)
From: "Niels Basjes (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1301308449.69002.1307220107403.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1008788632.26385.1305807047638.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7305) Eclipse project files are incomplete
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Niels Basjes updated HADOOP-7305:
---------------------------------

    Hadoop Flags:   (was: [Reviewed])
          Status: Patch Available  (was: Reopened)

This new patch should fix the reported issues. This needs to be teste by someone with OS X because I do not have access to such a system.

> Eclipse project files are incomplete
> ------------------------------------
>
>                 Key: HADOOP-7305
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7305
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Niels Basjes
>            Assignee: Niels Basjes
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7305-2011-05-19.patch, HADOOP-7305-2011-05-30.patch
>
>
> After a fresh checkout of hadoop-common I do 'ant compile eclipse'.
> I open eclipse, set ANT_HOME and build the project. 
> At that point the following error appears:
> {quote}
> The type com.sun.javadoc.RootDoc cannot be resolved. It is indirectly referenced from required .class files	ExcludePrivateAnnotationsJDiffDoclet.java	/common/src/java/org/apache/hadoop/classification/tools	line 1	Java Problem
> {quote}
> The solution is to add the "tools.jar" from the JDK to the buildpath/classpath.
> This should be fixed in the build.xml.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15879-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun  4 21:05:31 2011
Return-Path: <common-issues-return-15879-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3F8DE4A70
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat,  4 Jun 2011 21:05:31 +0000 (UTC)
Received: (qmail 76111 invoked by uid 500); 4 Jun 2011 21:05:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75991 invoked by uid 500); 4 Jun 2011 21:05:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75983 invoked by uid 99); 4 Jun 2011 21:05:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 21:05:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 21:05:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 953AF100B2C
	for <common-issues@hadoop.apache.org>; Sat,  4 Jun 2011 21:04:47 +0000 (UTC)
Date: Sat, 4 Jun 2011 21:04:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1978564108.69032.1307221487607.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1008788632.26385.1305807047638.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7305) Eclipse project files are
 incomplete
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044394#comment-13044394 ] 

Hadoop QA commented on HADOOP-7305:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481469/HADOOP-7305-2011-05-30.patch
  against trunk revision 1131330.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/577//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/577//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/577//console

This message is automatically generated.

> Eclipse project files are incomplete
> ------------------------------------
>
>                 Key: HADOOP-7305
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7305
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Niels Basjes
>            Assignee: Niels Basjes
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7305-2011-05-19.patch, HADOOP-7305-2011-05-30.patch
>
>
> After a fresh checkout of hadoop-common I do 'ant compile eclipse'.
> I open eclipse, set ANT_HOME and build the project. 
> At that point the following error appears:
> {quote}
> The type com.sun.javadoc.RootDoc cannot be resolved. It is indirectly referenced from required .class files	ExcludePrivateAnnotationsJDiffDoclet.java	/common/src/java/org/apache/hadoop/classification/tools	line 1	Java Problem
> {quote}
> The solution is to add the "tools.jar" from the JDK to the buildpath/classpath.
> This should be fixed in the build.xml.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15880-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun  4 23:39:31 2011
Return-Path: <common-issues-return-15880-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8BB104BEA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat,  4 Jun 2011 23:39:31 +0000 (UTC)
Received: (qmail 69686 invoked by uid 500); 4 Jun 2011 23:39:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69649 invoked by uid 500); 4 Jun 2011 23:39:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69641 invoked by uid 99); 4 Jun 2011 23:39:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 23:39:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 04 Jun 2011 23:39:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F3EEC1002AB
	for <common-issues@hadoop.apache.org>; Sat,  4 Jun 2011 23:38:47 +0000 (UTC)
Date: Sat, 4 Jun 2011 23:38:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1562375312.69142.1307230727995.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <710160053.68701.1307208347759.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7359) Pluggable interface for cluster
 membership
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044414#comment-13044414 ] 

Todd Lipcon commented on HADOOP-7359:
-------------------------------------

Sounds like a reasonable design to me!

> Pluggable interface for cluster membership
> ------------------------------------------
>
>                 Key: HADOOP-7359
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7359
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Travis Crawford
>
> Currently Hadoop uses local files to determine cluster membership. With HDFS for example, dfs.hosts and dfs.hosts.exclude are used.
> To enable tighter integrations cluster membership should be an interface, with the current file-based functionality provided as the default implementation. The common case would be no functional change, however, sites could plug an alternative implementation in, such as pulling the machine lists from a machine database.
> DETAILS:
> Two machine lists, includes and excludes, are used to define cluster membership and state. HostsFileReader currently handles reading these lists from files, who's names are passed in by FSNamesystem for HDFS and JobTracker for MR.
> The proposed change is adding a HostsReader interface to common, and changing HostsFileReader to an abstract class that functions the same as today.
> Two new classes, DFSHostsFileReader and MRHostsFileReader, extend HostsFileReader and simply pass the appropriate file names in. These new classes are needed because config key names live outside common.
> Two new conf keys, defaulting to the file-based readers, would be added to choose a different hosts reader: dfs.namenode.hosts.reader.class mapreduce.jobtracker.hosts.reader.class
> Comments/suggestions? I have most of this written already but would love some feedback on the general idea before posting the diff.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15881-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun  5 05:20:12 2011
Return-Path: <common-issues-return-15881-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 811F56BDD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun,  5 Jun 2011 05:20:12 +0000 (UTC)
Received: (qmail 97353 invoked by uid 500); 5 Jun 2011 05:20:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97322 invoked by uid 500); 5 Jun 2011 05:20:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97313 invoked by uid 99); 5 Jun 2011 05:20:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jun 2011 05:20:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jun 2011 05:20:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 57D591005A9
	for <common-issues@hadoop.apache.org>; Sun,  5 Jun 2011 05:19:47 +0000 (UTC)
Date: Sun, 5 Jun 2011 05:19:47 +0000 (UTC)
From: "Travis Crawford (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1431870602.69375.1307251187356.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <710160053.68701.1307208347759.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7359) Pluggable interface for cluster
 membership
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Travis Crawford updated HADOOP-7359:
------------------------------------

    Attachment: HADOOP-7359.diff

Common portion of proposed change.

> Pluggable interface for cluster membership
> ------------------------------------------
>
>                 Key: HADOOP-7359
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7359
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Travis Crawford
>         Attachments: HADOOP-7359.diff
>
>
> Currently Hadoop uses local files to determine cluster membership. With HDFS for example, dfs.hosts and dfs.hosts.exclude are used.
> To enable tighter integrations cluster membership should be an interface, with the current file-based functionality provided as the default implementation. The common case would be no functional change, however, sites could plug an alternative implementation in, such as pulling the machine lists from a machine database.
> DETAILS:
> Two machine lists, includes and excludes, are used to define cluster membership and state. HostsFileReader currently handles reading these lists from files, who's names are passed in by FSNamesystem for HDFS and JobTracker for MR.
> The proposed change is adding a HostsReader interface to common, and changing HostsFileReader to an abstract class that functions the same as today.
> Two new classes, DFSHostsFileReader and MRHostsFileReader, extend HostsFileReader and simply pass the appropriate file names in. These new classes are needed because config key names live outside common.
> Two new conf keys, defaulting to the file-based readers, would be added to choose a different hosts reader: dfs.namenode.hosts.reader.class mapreduce.jobtracker.hosts.reader.class
> Comments/suggestions? I have most of this written already but would love some feedback on the general idea before posting the diff.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15882-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun  5 21:44:09 2011
Return-Path: <common-issues-return-15882-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7ECBB61AA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun,  5 Jun 2011 21:44:09 +0000 (UTC)
Received: (qmail 74616 invoked by uid 500); 5 Jun 2011 21:44:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74574 invoked by uid 500); 5 Jun 2011 21:44:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74566 invoked by uid 99); 5 Jun 2011 21:44:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jun 2011 21:44:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jun 2011 21:44:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 57570102859
	for <common-issues@hadoop.apache.org>; Sun,  5 Jun 2011 21:43:47 +0000 (UTC)
Date: Sun, 5 Jun 2011 21:43:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1100898366.70175.1307310227354.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1470673003.56543.1306860527397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7341) Fix option parsing in CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7341:
--------------------------------

    Attachment: HADOOP-7341-3.patch

Per Aaron, alter text related to deprecation.

> Fix option parsing in CommandFormat
> -----------------------------------
>
>                 Key: HADOOP-7341
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7341
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7341-2.patch, HADOOP-7341-3.patch, HADOOP-7341.patch
>
>
> CommandFormat currently allows options in any location within the args.  This is not the intended behavior for FsShell commands.  Prior to the redesign, the commands used to expect option processing to stop at the first non-option.
> CommandFormat was an existing class prior the redesign, but it was only used by "count" to find the -q flag.  All commands were converted to using this class, thus inherited the unintended behavior.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15883-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun  5 22:00:11 2011
Return-Path: <common-issues-return-15883-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 821C7687F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun,  5 Jun 2011 22:00:11 +0000 (UTC)
Received: (qmail 87339 invoked by uid 500); 5 Jun 2011 22:00:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87200 invoked by uid 500); 5 Jun 2011 22:00:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87192 invoked by uid 99); 5 Jun 2011 22:00:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jun 2011 22:00:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jun 2011 22:00:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 45EA9102B87
	for <common-issues@hadoop.apache.org>; Sun,  5 Jun 2011 21:59:48 +0000 (UTC)
Date: Sun, 5 Jun 2011 21:59:48 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <152940744.70215.1307311188282.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-6385) dfs does not support -rmdir (was
 HDFS-639)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp reassigned HADOOP-6385:
-----------------------------------

    Assignee: Daryn Sharp

> dfs does not support -rmdir (was HDFS-639)
> ------------------------------------------
>
>                 Key: HADOOP-6385
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6385
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>            Reporter: Scott Phillips
>            Assignee: Daryn Sharp
>            Priority: Minor
>         Attachments: HADOOP-6385.patch
>
>
> From HDFS-639
> > Given we have a mkdir, we should have a rmdir. Using rmr is not
> > a reasonable substitute when you only want to delete empty
> > directories.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15884-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun  5 22:02:10 2011
Return-Path: <common-issues-return-15884-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E51776254
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun,  5 Jun 2011 22:02:10 +0000 (UTC)
Received: (qmail 89213 invoked by uid 500); 5 Jun 2011 22:02:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89170 invoked by uid 500); 5 Jun 2011 22:02:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89162 invoked by uid 99); 5 Jun 2011 22:02:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jun 2011 22:02:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jun 2011 22:02:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7E744102BE2
	for <common-issues@hadoop.apache.org>; Sun,  5 Jun 2011 22:01:47 +0000 (UTC)
Date: Sun, 5 Jun 2011 22:01:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1765230389.70218.1307311307514.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6385) dfs does not support -rmdir (was
 HDFS-639)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-6385:
--------------------------------

    Attachment: HADOOP-6385-2.patch

> dfs does not support -rmdir (was HDFS-639)
> ------------------------------------------
>
>                 Key: HADOOP-6385
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6385
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>            Reporter: Scott Phillips
>            Assignee: Daryn Sharp
>            Priority: Minor
>         Attachments: HADOOP-6385-2.patch, HADOOP-6385.patch
>
>
> From HDFS-639
> > Given we have a mkdir, we should have a rmdir. Using rmr is not
> > a reasonable substitute when you only want to delete empty
> > directories.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15885-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun  5 22:04:10 2011
Return-Path: <common-issues-return-15885-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8CB656BA2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun,  5 Jun 2011 22:04:10 +0000 (UTC)
Received: (qmail 89668 invoked by uid 500); 5 Jun 2011 22:04:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89632 invoked by uid 500); 5 Jun 2011 22:04:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89624 invoked by uid 99); 5 Jun 2011 22:04:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jun 2011 22:04:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jun 2011 22:04:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4F693102C55
	for <common-issues@hadoop.apache.org>; Sun,  5 Jun 2011 22:03:47 +0000 (UTC)
Date: Sun, 5 Jun 2011 22:03:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1523854424.70223.1307311427322.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1470673003.56543.1306860527397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7341) Fix option parsing in
 CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044632#comment-13044632 ] 

Hadoop QA commented on HADOOP-7341:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481523/HADOOP-7341-3.patch
  against trunk revision 1131330.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/578//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/578//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/578//console

This message is automatically generated.

> Fix option parsing in CommandFormat
> -----------------------------------
>
>                 Key: HADOOP-7341
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7341
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7341-2.patch, HADOOP-7341-3.patch, HADOOP-7341.patch
>
>
> CommandFormat currently allows options in any location within the args.  This is not the intended behavior for FsShell commands.  Prior to the redesign, the commands used to expect option processing to stop at the first non-option.
> CommandFormat was an existing class prior the redesign, but it was only used by "count" to find the -q flag.  All commands were converted to using this class, thus inherited the unintended behavior.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15886-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun  5 22:08:08 2011
Return-Path: <common-issues-return-15886-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A4DF86F12
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun,  5 Jun 2011 22:08:08 +0000 (UTC)
Received: (qmail 93117 invoked by uid 500); 5 Jun 2011 22:08:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93090 invoked by uid 500); 5 Jun 2011 22:08:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93082 invoked by uid 99); 5 Jun 2011 22:08:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jun 2011 22:08:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jun 2011 22:08:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 87508102D03
	for <common-issues@hadoop.apache.org>; Sun,  5 Jun 2011 22:07:47 +0000 (UTC)
Date: Sun, 5 Jun 2011 22:07:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1849684169.70230.1307311667550.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6385) dfs does not support -rmdir (was
 HDFS-639)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-6385:
--------------------------------

    Attachment: HADOOP-6385-2.patch

> dfs does not support -rmdir (was HDFS-639)
> ------------------------------------------
>
>                 Key: HADOOP-6385
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6385
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>            Reporter: Scott Phillips
>            Assignee: Daryn Sharp
>            Priority: Minor
>         Attachments: HADOOP-6385-2.patch, HADOOP-6385-2.patch, HADOOP-6385.patch
>
>
> From HDFS-639
> > Given we have a mkdir, we should have a rmdir. Using rmr is not
> > a reasonable substitute when you only want to delete empty
> > directories.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15887-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun  5 22:10:10 2011
Return-Path: <common-issues-return-15887-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AB67D6F51
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun,  5 Jun 2011 22:10:10 +0000 (UTC)
Received: (qmail 95569 invoked by uid 500); 5 Jun 2011 22:10:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95533 invoked by uid 500); 5 Jun 2011 22:10:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95525 invoked by uid 99); 5 Jun 2011 22:10:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jun 2011 22:10:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jun 2011 22:10:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 580E7102D6C
	for <common-issues@hadoop.apache.org>; Sun,  5 Jun 2011 22:09:47 +0000 (UTC)
Date: Sun, 5 Jun 2011 22:09:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1793545812.70235.1307311787357.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6385) dfs does not support -rmdir (was
 HDFS-639)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-6385:
--------------------------------

    Status: Patch Available  (was: Reopened)

> dfs does not support -rmdir (was HDFS-639)
> ------------------------------------------
>
>                 Key: HADOOP-6385
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6385
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>            Reporter: Scott Phillips
>            Assignee: Daryn Sharp
>            Priority: Minor
>         Attachments: HADOOP-6385-2.patch, HADOOP-6385-2.patch, HADOOP-6385.patch
>
>
> From HDFS-639
> > Given we have a mkdir, we should have a rmdir. Using rmr is not
> > a reasonable substitute when you only want to delete empty
> > directories.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15888-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun  5 22:24:11 2011
Return-Path: <common-issues-return-15888-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E471C6785
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun,  5 Jun 2011 22:24:10 +0000 (UTC)
Received: (qmail 7496 invoked by uid 500); 5 Jun 2011 22:24:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7358 invoked by uid 500); 5 Jun 2011 22:24:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7281 invoked by uid 99); 5 Jun 2011 22:24:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jun 2011 22:24:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jun 2011 22:24:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 66F39102F48
	for <common-issues@hadoop.apache.org>; Sun,  5 Jun 2011 22:23:47 +0000 (UTC)
Date: Sun, 5 Jun 2011 22:23:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <624510738.70252.1307312627418.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6385) dfs does not support -rmdir (was
 HDFS-639)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044637#comment-13044637 ] 

Hadoop QA commented on HADOOP-6385:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481525/HADOOP-6385-2.patch
  against trunk revision 1131330.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/579//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/579//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/579//console

This message is automatically generated.

> dfs does not support -rmdir (was HDFS-639)
> ------------------------------------------
>
>                 Key: HADOOP-6385
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6385
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>            Reporter: Scott Phillips
>            Assignee: Daryn Sharp
>            Priority: Minor
>         Attachments: HADOOP-6385-2.patch, HADOOP-6385-2.patch, HADOOP-6385.patch
>
>
> From HDFS-639
> > Given we have a mkdir, we should have a rmdir. Using rmr is not
> > a reasonable substitute when you only want to delete empty
> > directories.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15889-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun  5 23:42:08 2011
Return-Path: <common-issues-return-15889-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BA0C06BF9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun,  5 Jun 2011 23:42:08 +0000 (UTC)
Received: (qmail 90776 invoked by uid 500); 5 Jun 2011 23:42:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90740 invoked by uid 500); 5 Jun 2011 23:42:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90732 invoked by uid 99); 5 Jun 2011 23:42:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jun 2011 23:42:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jun 2011 23:42:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5CDF410290A
	for <common-issues@hadoop.apache.org>; Sun,  5 Jun 2011 23:41:47 +0000 (UTC)
Date: Sun, 5 Jun 2011 23:41:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2004248917.70295.1307317307377.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1470673003.56543.1306860527397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7341) Fix option parsing in
 CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044649#comment-13044649 ] 

Aaron T. Myers commented on HADOOP-7341:
----------------------------------------

+1, looks good to me.

> Fix option parsing in CommandFormat
> -----------------------------------
>
>                 Key: HADOOP-7341
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7341
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7341-2.patch, HADOOP-7341-3.patch, HADOOP-7341.patch
>
>
> CommandFormat currently allows options in any location within the args.  This is not the intended behavior for FsShell commands.  Prior to the redesign, the commands used to expect option processing to stop at the first non-option.
> CommandFormat was an existing class prior the redesign, but it was only used by "count" to find the -q flag.  All commands were converted to using this class, thus inherited the unintended behavior.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15890-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun  5 23:44:18 2011
Return-Path: <common-issues-return-15890-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CF21B6D98
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun,  5 Jun 2011 23:44:18 +0000 (UTC)
Received: (qmail 93521 invoked by uid 500); 5 Jun 2011 23:44:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93490 invoked by uid 500); 5 Jun 2011 23:44:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93468 invoked by uid 99); 5 Jun 2011 23:44:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jun 2011 23:44:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Jun 2011 23:44:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E86D9102997
	for <common-issues@hadoop.apache.org>; Sun,  5 Jun 2011 23:43:47 +0000 (UTC)
Date: Sun, 5 Jun 2011 23:43:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <382631848.70301.1307317427949.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1470673003.56543.1306860527397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7341) Fix option parsing in CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7341:
--------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

Committed based on atm's +1 (he's still waiting on his ASF account to get set up)

> Fix option parsing in CommandFormat
> -----------------------------------
>
>                 Key: HADOOP-7341
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7341
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7341-2.patch, HADOOP-7341-3.patch, HADOOP-7341.patch
>
>
> CommandFormat currently allows options in any location within the args.  This is not the intended behavior for FsShell commands.  Prior to the redesign, the commands used to expect option processing to stop at the first non-option.
> CommandFormat was an existing class prior the redesign, but it was only used by "count" to find the -q flag.  All commands were converted to using this class, thus inherited the unintended behavior.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15891-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 00:06:11 2011
Return-Path: <common-issues-return-15891-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B114E6209
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 00:06:11 +0000 (UTC)
Received: (qmail 23217 invoked by uid 500); 6 Jun 2011 00:06:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23183 invoked by uid 500); 6 Jun 2011 00:06:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23169 invoked by uid 99); 6 Jun 2011 00:06:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 00:06:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 00:06:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 212C2102CC0
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 00:05:48 +0000 (UTC)
Date: Mon, 6 Jun 2011 00:05:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1924883569.70314.1307318748132.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1470673003.56543.1306860527397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7341) Fix option parsing in
 CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044656#comment-13044656 ] 

Hudson commented on HADOOP-7341:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #641 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/641/])
    HADOOP-7341. Fix options parsing in CommandFormat. Contributed by Daryn Sharp.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132505
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/CopyCommands.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Touchz.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/MoveCommands.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/FsUsage.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/TestCommandFormat.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Stat.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Test.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Mkdir.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Count.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Tail.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Delete.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/SetReplication.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Ls.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Display.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/CommandFormat.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/FsShellPermissions.java


> Fix option parsing in CommandFormat
> -----------------------------------
>
>                 Key: HADOOP-7341
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7341
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7341-2.patch, HADOOP-7341-3.patch, HADOOP-7341.patch
>
>
> CommandFormat currently allows options in any location within the args.  This is not the intended behavior for FsShell commands.  Prior to the redesign, the commands used to expect option processing to stop at the first non-option.
> CommandFormat was an existing class prior the redesign, but it was only used by "count" to find the -q flag.  All commands were converted to using this class, thus inherited the unintended behavior.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15892-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 00:06:11 2011
Return-Path: <common-issues-return-15892-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B00436207
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 00:06:11 +0000 (UTC)
Received: (qmail 23267 invoked by uid 500); 6 Jun 2011 00:06:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23187 invoked by uid 500); 6 Jun 2011 00:06:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23172 invoked by uid 99); 6 Jun 2011 00:06:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 00:06:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 00:06:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 41756102CC4
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 00:05:48 +0000 (UTC)
Date: Mon, 6 Jun 2011 00:05:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1844967408.70318.1307318748265.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <397648942.57126.1306869588398.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7342) Add an utility API in FileUtil for
 JDK File.list
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044657#comment-13044657 ] 

Hudson commented on HADOOP-7342:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #641 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/641/])
    

> Add an utility API in FileUtil for JDK File.list
> ------------------------------------------------
>
>                 Key: HADOOP-7342
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7342
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7342-1.patch, HADOOP-7342-2.patch
>
>
> Java File.list API can return null when disk is bad or directory is not a directory. This utility API in FileUtil will throw an exception when this happens rather than returning null. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15893-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 01:10:09 2011
Return-Path: <common-issues-return-15893-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 037B168C4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 01:10:09 +0000 (UTC)
Received: (qmail 44969 invoked by uid 500); 6 Jun 2011 01:10:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44904 invoked by uid 500); 6 Jun 2011 01:10:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44896 invoked by uid 99); 6 Jun 2011 01:10:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 01:10:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 01:10:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C4A51102687
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 01:09:47 +0000 (UTC)
Date: Mon, 6 Jun 2011 01:09:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <218273058.70401.1307322587802.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <314496369.49888.1306530467371.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7335) Force entropy to come from
 non-true random for tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044670#comment-13044670 ] 

Eli Collins commented on HADOOP-7335:
-------------------------------------

Forgot to mention, /dev/urandom has been in Solaris since 2.6.

> Force entropy to come from non-true random for tests
> ----------------------------------------------------
>
>                 Key: HADOOP-7335
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7335
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7335.txt
>
>
> Passing the system property {{-Djava.security.egd=file:///dev/urandom}} forces the JVM to seed its PRNG from non-true random (/dev/urandom) instead of the true random (/dev/random). This makes the tests run faster, since without it they often hang waiting for entropy while Jetty is initializing.
> We should turn this on for the test targets by default, so developers/hudson boxes don't have to make this change system-wide or use workarounds like rngtools.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15894-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 01:10:10 2011
Return-Path: <common-issues-return-15894-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 681B868D1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 01:10:10 +0000 (UTC)
Received: (qmail 45212 invoked by uid 500); 6 Jun 2011 01:10:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45168 invoked by uid 500); 6 Jun 2011 01:10:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45160 invoked by uid 99); 6 Jun 2011 01:10:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 01:10:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 01:10:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8D51E102681
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 01:09:47 +0000 (UTC)
Date: Mon, 6 Jun 2011 01:09:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1375210162.70395.1307322587575.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <314496369.49888.1306530467371.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7335) Force entropy to come from
 non-true random for tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044668#comment-13044668 ] 

Eli Collins commented on HADOOP-7335:
-------------------------------------

+1  lgtm

OSX has supported /dev/urandom since at least 10.2. From http://developer.apple.com/library/mac/#documentation/darwin/reference/manpages/man4/random.4.html

{quote}
 /dev/urandom is a compatibility nod to Linux. On Linux, /dev/urandom will produce lower quality output if the entropy pool drains, while /dev/random will prefer to block and wait for additional entropy to be collected.  With Yarrow, this choice and distinction is not necessary, and the two devices behave identically. You may use either.
{quote}


> Force entropy to come from non-true random for tests
> ----------------------------------------------------
>
>                 Key: HADOOP-7335
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7335
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7335.txt
>
>
> Passing the system property {{-Djava.security.egd=file:///dev/urandom}} forces the JVM to seed its PRNG from non-true random (/dev/urandom) instead of the true random (/dev/random). This makes the tests run faster, since without it they often hang waiting for entropy while Jetty is initializing.
> We should turn this on for the test targets by default, so developers/hudson boxes don't have to make this change system-wide or use workarounds like rngtools.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15895-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 01:38:11 2011
Return-Path: <common-issues-return-15895-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 036E26AFB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 01:38:11 +0000 (UTC)
Received: (qmail 70640 invoked by uid 500); 6 Jun 2011 01:38:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70614 invoked by uid 500); 6 Jun 2011 01:38:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70606 invoked by uid 99); 6 Jun 2011 01:38:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 01:38:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 01:38:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8CAF1102A5E
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 01:37:47 +0000 (UTC)
Date: Mon, 6 Jun 2011 01:37:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1748376894.70409.1307324267573.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <314496369.49888.1306530467371.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7335) Force entropy to come from
 non-true random for tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044671#comment-13044671 ] 

Hudson commented on HADOOP-7335:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #642 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/642/])
    HADOOP-7335. Force entropy to come from non-true random for tests. Contributed by Todd Lipcon

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132511
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/build.xml


> Force entropy to come from non-true random for tests
> ----------------------------------------------------
>
>                 Key: HADOOP-7335
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7335
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7335.txt
>
>
> Passing the system property {{-Djava.security.egd=file:///dev/urandom}} forces the JVM to seed its PRNG from non-true random (/dev/urandom) instead of the true random (/dev/random). This makes the tests run faster, since without it they often hang waiting for entropy while Jetty is initializing.
> We should turn this on for the test targets by default, so developers/hudson boxes don't have to make this change system-wide or use workarounds like rngtools.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15896-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 01:42:11 2011
Return-Path: <common-issues-return-15896-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F0C8C6C52
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 01:42:10 +0000 (UTC)
Received: (qmail 73578 invoked by uid 500); 6 Jun 2011 01:42:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73539 invoked by uid 500); 6 Jun 2011 01:42:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73531 invoked by uid 99); 6 Jun 2011 01:42:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 01:42:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 01:42:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B857E102B42
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 01:41:47 +0000 (UTC)
Date: Mon, 6 Jun 2011 01:41:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2135608068.70419.1307324507752.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <314496369.49888.1306530467371.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7335) Force entropy to come from non-true
 random for tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7335:
--------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I've committed this to branch 22 and trunk. Thanks Todd!

(the hudson results are green btw: https://builds.apache.org/job/Hadoop-Common-trunk-Commit/642. Not sure why they haven't been posted to jira)

> Force entropy to come from non-true random for tests
> ----------------------------------------------------
>
>                 Key: HADOOP-7335
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7335
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7335.txt
>
>
> Passing the system property {{-Djava.security.egd=file:///dev/urandom}} forces the JVM to seed its PRNG from non-true random (/dev/urandom) instead of the true random (/dev/random). This makes the tests run faster, since without it they often hang waiting for entropy while Jetty is initializing.
> We should turn this on for the test targets by default, so developers/hudson boxes don't have to make this change system-wide or use workarounds like rngtools.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15897-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 05:33:12 2011
Return-Path: <common-issues-return-15897-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 565FF46D9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 05:33:12 +0000 (UTC)
Received: (qmail 11160 invoked by uid 500); 6 Jun 2011 05:33:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10977 invoked by uid 500); 6 Jun 2011 05:33:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10957 invoked by uid 99); 6 Jun 2011 05:33:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 05:33:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 05:33:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A50C81027B6
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 05:32:47 +0000 (UTC)
Date: Mon, 6 Jun 2011 05:32:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1695882960.70852.1307338367672.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <647927887.58055.1306890116067.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7346) Send back nicer error to clients
 using outdated IPC version
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044719#comment-13044719 ] 

Todd Lipcon commented on HADOOP-7346:
-------------------------------------

I checked with Nigel, he said he was cool with adding guava as a dependency in 0.22. So I've committed this to 22 as well.

> Send back nicer error to clients using outdated IPC version
> -----------------------------------------------------------
>
>                 Key: HADOOP-7346
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7346
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7346.txt, hadoop-7346.txt, hadoop-7346.txt
>
>
> When an older Hadoop version tries to contact a newer Hadoop version across an IPC protocol version bump, the client currently just gets a non-useful error message like "EOFException".
> Instead, the IPC server code can speak just enough of prior IPC protocols to send back a "fatal" message indicating the version mismatch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15898-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 05:33:12 2011
Return-Path: <common-issues-return-15898-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5E18746DB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 05:33:12 +0000 (UTC)
Received: (qmail 11245 invoked by uid 500); 6 Jun 2011 05:33:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10986 invoked by uid 500); 6 Jun 2011 05:33:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10964 invoked by uid 99); 6 Jun 2011 05:33:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 05:33:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 05:33:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B86C11027BB
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 05:32:47 +0000 (UTC)
Date: Mon, 6 Jun 2011 05:32:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1091105669.70854.1307338367752.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <647927887.58055.1306890116067.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7346) Send back nicer error to clients
 using outdated IPC version
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7346:
--------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

Thanks for the review, Cos!

> Send back nicer error to clients using outdated IPC version
> -----------------------------------------------------------
>
>                 Key: HADOOP-7346
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7346
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7346.txt, hadoop-7346.txt, hadoop-7346.txt
>
>
> When an older Hadoop version tries to contact a newer Hadoop version across an IPC protocol version bump, the client currently just gets a non-useful error message like "EOFException".
> Instead, the IPC server code can speak just enough of prior IPC protocols to send back a "fatal" message indicating the version mismatch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15899-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 08:04:39 2011
Return-Path: <common-issues-return-15899-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7DCD440E2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 08:04:39 +0000 (UTC)
Received: (qmail 75323 invoked by uid 500); 6 Jun 2011 08:04:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70900 invoked by uid 500); 6 Jun 2011 08:04:25 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69127 invoked by uid 99); 6 Jun 2011 08:04:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 08:04:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 08:04:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5C4EB102B19
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 08:03:47 +0000 (UTC)
Date: Mon, 6 Jun 2011 08:03:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <92042485.70969.1307347427375.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7360) FsShell does not preserve relative
 paths with globs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

FsShell does not preserve relative paths with globs
---------------------------------------------------

                 Key: HADOOP-7360
                 URL: https://issues.apache.org/jira/browse/HADOOP-7360
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs
    Affects Versions: 0.23.0
            Reporter: Daryn Sharp
            Assignee: Daryn Sharp


FsShell currently preserves relative paths that do not contain globs.  Unfortunately the method {{fs.globStatus()}} is fully qualifying all returned paths.  This is causing inconsistent display of paths.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15900-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 08:06:09 2011
Return-Path: <common-issues-return-15900-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E6E6A40FC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 08:06:08 +0000 (UTC)
Received: (qmail 75994 invoked by uid 500); 6 Jun 2011 08:06:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75893 invoked by uid 500); 6 Jun 2011 08:06:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75885 invoked by uid 99); 6 Jun 2011 08:06:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 08:06:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 08:06:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 665E3102C12
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 08:05:47 +0000 (UTC)
Date: Mon, 6 Jun 2011 08:05:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1655224328.70972.1307347547415.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <671647059.63008.1307027027379.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7353) Cleanup FsShell and prevent
 masking of RTE stacktraces
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044740#comment-13044740 ] 

Aaron T. Myers commented on HADOOP-7353:
----------------------------------------

Looks pretty good, Daryn. A few small nits:

* There's no need to expressly initialize the instance variables in the {{FsShell} ctor to {{null}}. Java will do that automatically.
* "The follow methods are syntactic sugar." - They're not really "_syntactic_" sugar, as much as they are just helpers.
* "if (instance == null) throw new UnknownCommandException(cmd);" - coding guidelines say to put always use braces on {{if}} statements, and always put the body on a new line.
* There seems to be an inconsistency with printing to stdout vs. stderr, or at least I can't tell why you chose to use one in some places, and the other in other places.
* "// historical abstract method in Command " - move this comment above the method.

> Cleanup FsShell and prevent masking of RTE stacktraces
> ------------------------------------------------------
>
>                 Key: HADOOP-7353
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7353
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7353.patch
>
>
> {{FsShell}}'s top level exception handler catches and displays exceptions.  Unfortunately it displays only the first line of an exception, which means an unexpected {{RuntimeExceptions}} like {{NullPointerException}} only display "{{cmd: NullPointerException}}".  This user has no context to understand and/or accurately report the issue.
> Found due to bugs such as {{HADOOP-7327}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15901-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 08:12:08 2011
Return-Path: <common-issues-return-15901-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9D6EC4417
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 08:12:08 +0000 (UTC)
Received: (qmail 85514 invoked by uid 500); 6 Jun 2011 08:12:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85436 invoked by uid 500); 6 Jun 2011 08:12:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85425 invoked by uid 99); 6 Jun 2011 08:12:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 08:12:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 08:12:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 57810102E16
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 08:11:47 +0000 (UTC)
Date: Mon, 6 Jun 2011 08:11:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <998005864.70980.1307347907355.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <92042485.70969.1307347427375.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7360) FsShell does not preserve relative
 paths with globs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7360:
--------------------------------

    Attachment: HADOOP-7360.patch

{{PathData}} retains all forms of relativity in a glob, including whether the authority is present or absent (ie. hdfs://host/path or hdfs:///path).

Streamlines the {{PathData}} ctors, and converts most to private since they were only public to ease the redesign.

Changed {{CLITestHelper}} to expand keywords not only in the commands, but also in the expected output.  Notably this allows NAMENODE to be used in the expected output instead of fuzzy regexps.



> FsShell does not preserve relative paths with globs
> ---------------------------------------------------
>
>                 Key: HADOOP-7360
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7360
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7360.patch
>
>
> FsShell currently preserves relative paths that do not contain globs.  Unfortunately the method {{fs.globStatus()}} is fully qualifying all returned paths.  This is causing inconsistent display of paths.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15902-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 08:14:10 2011
Return-Path: <common-issues-return-15902-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D64214537
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 08:14:10 +0000 (UTC)
Received: (qmail 87901 invoked by uid 500); 6 Jun 2011 08:14:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87867 invoked by uid 500); 6 Jun 2011 08:14:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87859 invoked by uid 99); 6 Jun 2011 08:14:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 08:14:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 08:14:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AB47C102F25
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 08:13:47 +0000 (UTC)
Date: Mon, 6 Jun 2011 08:13:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1576993123.70984.1307348027698.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <92042485.70969.1307347427375.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7360) FsShell does not preserve relative
 paths with globs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7360:
--------------------------------

    Status: Patch Available  (was: Open)

> FsShell does not preserve relative paths with globs
> ---------------------------------------------------
>
>                 Key: HADOOP-7360
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7360
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7360.patch
>
>
> FsShell currently preserves relative paths that do not contain globs.  Unfortunately the method {{fs.globStatus()}} is fully qualifying all returned paths.  This is causing inconsistent display of paths.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15903-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 09:45:20 2011
Return-Path: <common-issues-return-15903-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AFBA94BD8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 09:45:20 +0000 (UTC)
Received: (qmail 83445 invoked by uid 500); 6 Jun 2011 09:45:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83403 invoked by uid 500); 6 Jun 2011 09:45:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83395 invoked by uid 99); 6 Jun 2011 09:45:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 09:45:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 09:45:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5FE6A36681
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 09:44:59 +0000 (UTC)
Date: Mon, 6 Jun 2011 09:44:59 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1864654885.54.1307353499389.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <647927887.58055.1306890116067.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7346) Send back nicer error to clients
 using outdated IPC version
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044763#comment-13044763 ] 

Hudson commented on HADOOP-7346:
--------------------------------

Integrated in Hadoop-Common-22-branch #62 (See [https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/62/])
    HADOOP-7346. Send back nicer error message to clients using outdated IPC version. Contributed by Todd Lipcon.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132528
Files : 
* /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/ipc/Server.java
* /hadoop/common/branches/branch-0.22/ivy/libraries.properties
* /hadoop/common/branches/branch-0.22/src/test/core/org/apache/hadoop/ipc/TestIPC.java
* /hadoop/common/branches/branch-0.22/CHANGES.txt
* /hadoop/common/branches/branch-0.22/ivy.xml


> Send back nicer error to clients using outdated IPC version
> -----------------------------------------------------------
>
>                 Key: HADOOP-7346
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7346
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7346.txt, hadoop-7346.txt, hadoop-7346.txt
>
>
> When an older Hadoop version tries to contact a newer Hadoop version across an IPC protocol version bump, the client currently just gets a non-useful error message like "EOFException".
> Instead, the IPC server code can speak just enough of prior IPC protocols to send back a "fatal" message indicating the version mismatch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15904-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 09:45:22 2011
Return-Path: <common-issues-return-15904-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 905BD4C19
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 09:45:22 +0000 (UTC)
Received: (qmail 83692 invoked by uid 500); 6 Jun 2011 09:45:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83660 invoked by uid 500); 6 Jun 2011 09:45:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83652 invoked by uid 99); 6 Jun 2011 09:45:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 09:45:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 09:45:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 58B7836680
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 09:44:59 +0000 (UTC)
Date: Mon, 6 Jun 2011 09:44:59 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2118717856.53.1307353499360.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <314496369.49888.1306530467371.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7335) Force entropy to come from
 non-true random for tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044762#comment-13044762 ] 

Hudson commented on HADOOP-7335:
--------------------------------

Integrated in Hadoop-Common-22-branch #62 (See [https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/62/])
    HADOOP-7335. svn merge -c 1132511 from trunk

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132512
Files : 
* /hadoop/common/branches/branch-0.22/src/docs
* /hadoop/common/branches/branch-0.22
* /hadoop/common/branches/branch-0.22/src/test/core
* /hadoop/common/branches/branch-0.22/src/test/core/org/apache/hadoop/io/TestSequenceFile.java
* /hadoop/common/branches/branch-0.22/CHANGES.txt
* /hadoop/common/branches/branch-0.22/src/java
* /hadoop/common/branches/branch-0.22/build.xml


> Force entropy to come from non-true random for tests
> ----------------------------------------------------
>
>                 Key: HADOOP-7335
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7335
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7335.txt
>
>
> Passing the system property {{-Djava.security.egd=file:///dev/urandom}} forces the JVM to seed its PRNG from non-true random (/dev/urandom) instead of the true random (/dev/random). This makes the tests run faster, since without it they often hang waiting for entropy while Jetty is initializing.
> We should turn this on for the test targets by default, so developers/hudson boxes don't have to make this change system-wide or use workarounds like rngtools.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15905-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 11:16:22 2011
Return-Path: <common-issues-return-15905-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7C8D14A94
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 11:16:22 +0000 (UTC)
Received: (qmail 99827 invoked by uid 500); 6 Jun 2011 11:16:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99791 invoked by uid 500); 6 Jun 2011 11:16:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99783 invoked by uid 99); 6 Jun 2011 11:16:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 11:16:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 11:16:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1F6CC36475
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 11:15:59 +0000 (UTC)
Date: Mon, 6 Jun 2011 11:15:59 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1427323806.143.1307358959125.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <314496369.49888.1306530467371.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7335) Force entropy to come from
 non-true random for tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044786#comment-13044786 ] 

Hudson commented on HADOOP-7335:
--------------------------------

Integrated in Hadoop-Common-trunk #711 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/711/])
    HADOOP-7335. Force entropy to come from non-true random for tests. Contributed by Todd Lipcon

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132511
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/build.xml


> Force entropy to come from non-true random for tests
> ----------------------------------------------------
>
>                 Key: HADOOP-7335
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7335
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7335.txt
>
>
> Passing the system property {{-Djava.security.egd=file:///dev/urandom}} forces the JVM to seed its PRNG from non-true random (/dev/urandom) instead of the true random (/dev/random). This makes the tests run faster, since without it they often hang waiting for entropy while Jetty is initializing.
> We should turn this on for the test targets by default, so developers/hudson boxes don't have to make this change system-wide or use workarounds like rngtools.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15906-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 11:16:22 2011
Return-Path: <common-issues-return-15906-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AB3AF4AA7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 11:16:22 +0000 (UTC)
Received: (qmail 191 invoked by uid 500); 6 Jun 2011 11:16:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 129 invoked by uid 500); 6 Jun 2011 11:16:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99898 invoked by uid 99); 6 Jun 2011 11:16:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 11:16:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 11:16:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2DCE836477
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 11:15:59 +0000 (UTC)
Date: Mon, 6 Jun 2011 11:15:59 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <870430342.145.1307358959184.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1470673003.56543.1306860527397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7341) Fix option parsing in
 CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044787#comment-13044787 ] 

Hudson commented on HADOOP-7341:
--------------------------------

Integrated in Hadoop-Common-trunk #711 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/711/])
    HADOOP-7341. Fix options parsing in CommandFormat. Contributed by Daryn Sharp.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132505
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/CopyCommands.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Touchz.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/MoveCommands.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/FsUsage.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/TestCommandFormat.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Stat.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Test.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Mkdir.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Count.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Tail.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Delete.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/SetReplication.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Ls.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Display.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/CommandFormat.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/FsShellPermissions.java


> Fix option parsing in CommandFormat
> -----------------------------------
>
>                 Key: HADOOP-7341
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7341
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7341-2.patch, HADOOP-7341-3.patch, HADOOP-7341.patch
>
>
> CommandFormat currently allows options in any location within the args.  This is not the intended behavior for FsShell commands.  Prior to the redesign, the commands used to expect option processing to stop at the first non-option.
> CommandFormat was an existing class prior the redesign, but it was only used by "count" to find the -q flag.  All commands were converted to using this class, thus inherited the unintended behavior.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15907-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 15:33:22 2011
Return-Path: <common-issues-return-15907-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6E6C14CC6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 15:33:22 +0000 (UTC)
Received: (qmail 77619 invoked by uid 500); 6 Jun 2011 15:33:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77589 invoked by uid 500); 6 Jun 2011 15:33:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77581 invoked by uid 99); 6 Jun 2011 15:33:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 15:33:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 15:33:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2548C103C69
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 15:32:59 +0000 (UTC)
Date: Mon, 6 Jun 2011 15:32:59 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <595565442.702.1307374379149.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <671647059.63008.1307027027379.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7353) Cleanup FsShell and prevent masking
 of RTE stacktraces
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7353:
--------------------------------

    Attachment: HADOOP-7353-2.patch

Thanks Aaron!

bq. There's no need to expressly initialize the instance variables in the {{FsShell}} ctor to {{null}}. Java will do that automatically.
Sure, I removed the initialization.

bq. "The follow methods are syntactic sugar." - They're not really "syntactic" sugar, as much as they are just helpers.
Changed.

bq. "if (instance == null) throw new UnknownCommandException(cmd);" - coding guidelines say to put always use braces on if statements, and always put the body on a new line.
Changed.  I found a few more, so I changed them too.

bq. There seems to be an inconsistency with printing to stdout vs. stderr, or at least I can't tell why you chose to use one in some places, and the other in other places.
The -help/-usage commands will print to stdout.  The usage goes to stderr when no command is given, or because of an illegal argument.

bq. "// historical abstract method in Command " - move this comment above the method.
Moved.

Thanks again!


> Cleanup FsShell and prevent masking of RTE stacktraces
> ------------------------------------------------------
>
>                 Key: HADOOP-7353
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7353
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7353-2.patch, HADOOP-7353.patch
>
>
> {{FsShell}}'s top level exception handler catches and displays exceptions.  Unfortunately it displays only the first line of an exception, which means an unexpected {{RuntimeExceptions}} like {{NullPointerException}} only display "{{cmd: NullPointerException}}".  This user has no context to understand and/or accurately report the issue.
> Found due to bugs such as {{HADOOP-7327}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15908-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 15:55:22 2011
Return-Path: <common-issues-return-15908-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 42A3F49D3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 15:55:22 +0000 (UTC)
Received: (qmail 44515 invoked by uid 500); 6 Jun 2011 15:55:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44402 invoked by uid 500); 6 Jun 2011 15:55:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44394 invoked by uid 99); 6 Jun 2011 15:55:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 15:55:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 15:55:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E512E10354A
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 15:54:58 +0000 (UTC)
Date: Mon, 6 Jun 2011 15:54:58 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1885923335.775.1307375698935.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <671647059.63008.1307027027379.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7353) Cleanup FsShell and prevent
 masking of RTE stacktraces
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044923#comment-13044923 ] 

Hadoop QA commented on HADOOP-7353:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481578/HADOOP-7353-2.patch
  against trunk revision 1132511.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 4 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/581//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/581//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/581//console

This message is automatically generated.

> Cleanup FsShell and prevent masking of RTE stacktraces
> ------------------------------------------------------
>
>                 Key: HADOOP-7353
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7353
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7353-2.patch, HADOOP-7353.patch
>
>
> {{FsShell}}'s top level exception handler catches and displays exceptions.  Unfortunately it displays only the first line of an exception, which means an unexpected {{RuntimeExceptions}} like {{NullPointerException}} only display "{{cmd: NullPointerException}}".  This user has no context to understand and/or accurately report the issue.
> Found due to bugs such as {{HADOOP-7327}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15909-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 16:51:21 2011
Return-Path: <common-issues-return-15909-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3531546DC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 16:51:21 +0000 (UTC)
Received: (qmail 78335 invoked by uid 500); 6 Jun 2011 16:51:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78303 invoked by uid 500); 6 Jun 2011 16:51:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78287 invoked by uid 99); 6 Jun 2011 16:51:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 16:51:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 16:51:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E68B7103DBA
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 16:50:59 +0000 (UTC)
Date: Mon, 6 Jun 2011 16:50:59 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1389573820.914.1307379059940.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-6671:
---------------------------------------

    Attachment: mvn-layout-e.sh
                HADOOP-6671-e.patch

Patch rebased to HEAD of trunk (use the

Now jdiff works (same as in Ant build)

Build options in BUILDING.txt file.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>         Attachments: HADOOP-6671-e.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15910-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 16:51:22 2011
Return-Path: <common-issues-return-15910-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4412F46EB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 16:51:22 +0000 (UTC)
Received: (qmail 78615 invoked by uid 500); 6 Jun 2011 16:51:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78578 invoked by uid 500); 6 Jun 2011 16:51:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78570 invoked by uid 99); 6 Jun 2011 16:51:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 16:51:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 16:51:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0421F103DD5
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 16:51:01 +0000 (UTC)
Date: Mon, 6 Jun 2011 16:51:01 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1157166038.939.1307379061013.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044950#comment-13044950 ] 

Alejandro Abdelnur commented on HADOOP-6671:
--------------------------------------------

If folks agree I'd like to work with Tom in a new branch to get all the Jenkins hook scripts working.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>         Attachments: HADOOP-6671-e.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15911-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 16:54:26 2011
Return-Path: <common-issues-return-15911-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5AA794908
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 16:54:26 +0000 (UTC)
Received: (qmail 86813 invoked by uid 500); 6 Jun 2011 16:54:26 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86768 invoked by uid 500); 6 Jun 2011 16:54:26 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86760 invoked by uid 99); 6 Jun 2011 16:54:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 16:54:26 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 16:54:23 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0DADF103EF9
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 16:54:03 +0000 (UTC)
Date: Mon, 6 Jun 2011 16:54:03 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1268078813.964.1307379243052.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044951#comment-13044951 ] 

Hadoop QA commented on HADOOP-6671:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481585/mvn-layout-e.sh
  against trunk revision 1132511.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 11 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/582//console

This message is automatically generated.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>         Attachments: HADOOP-6671-e.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15912-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 17:17:22 2011
Return-Path: <common-issues-return-15912-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 21DAB473A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 17:17:22 +0000 (UTC)
Received: (qmail 47889 invoked by uid 500); 6 Jun 2011 17:17:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47865 invoked by uid 500); 6 Jun 2011 17:17:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47857 invoked by uid 99); 6 Jun 2011 17:17:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 17:17:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 17:17:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DFF89103981
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 17:16:58 +0000 (UTC)
Date: Mon, 6 Jun 2011 17:16:58 +0000 (UTC)
From: "Jeffrey Naisbitt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1272005493.1015.1307380618913.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <467948518.30516.1305901367779.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7314) Add support for throwing
 UnknownHostException when a host doesn't resolve
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044963#comment-13044963 ] 

Jeffrey Naisbitt commented on HADOOP-7314:
------------------------------------------

Thanks for the review, Robert!  I think you meant this comment to go with the MAPREDUCE-2489 patch though, and I will address it there.  :)

> Add support for throwing UnknownHostException when a host doesn't resolve
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-7314
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7314
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jeffrey Naisbitt
>            Assignee: Jeffrey Naisbitt
>         Attachments: HADOOP-7314.patch
>
>
> As part of MAPREDUCE-2489, we need support for having the resolve methods (for DNS mapping) throw UnknownHostExceptions.  (Currently, they hide the exception).  Since the existing 'resolve' method is ultimately used by several other locations/components, I propose we add a new 'resolveValidHosts' method.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15913-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 18:23:20 2011
Return-Path: <common-issues-return-15913-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 466D94413
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 18:23:20 +0000 (UTC)
Received: (qmail 86104 invoked by uid 500); 6 Jun 2011 18:23:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86072 invoked by uid 500); 6 Jun 2011 18:23:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86064 invoked by uid 99); 6 Jun 2011 18:23:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 18:23:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 18:23:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 11180104F95
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 18:22:59 +0000 (UTC)
Date: Mon, 6 Jun 2011 18:22:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <411325994.1238.1307384579066.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon reassigned HADOOP-6671:
-----------------------------------

    Assignee: Alejandro Abdelnur

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-6671-e.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15914-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 19:43:23 2011
Return-Path: <common-issues-return-15914-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 005464B09
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 19:43:23 +0000 (UTC)
Received: (qmail 7146 invoked by uid 500); 6 Jun 2011 19:43:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7076 invoked by uid 500); 6 Jun 2011 19:43:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7068 invoked by uid 99); 6 Jun 2011 19:43:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 19:43:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 19:43:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9F862104E5F
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 19:42:59 +0000 (UTC)
Date: Mon, 6 Jun 2011 19:42:59 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2029521589.1587.1307389379650.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <21049979.212171296037723374.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7119) add Kerberos HTTP SPNEGO
 authentication support to Hadoop JT/NN/DN/TT web-consoles
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045045#comment-13045045 ] 

Alejandro Abdelnur commented on HADOOP-7119:
--------------------------------------------

I'm working on an update for this patch that would bring all Alfredo code into Hadoop.


> add Kerberos HTTP SPNEGO authentication support to Hadoop JT/NN/DN/TT web-consoles
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7119
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7119
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>    Affects Versions: 0.23.0
>         Environment: all
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>         Attachments: ha-common-01.patch, ha-common-02.patch, ha-commons.patch
>
>
> Currently the JT/NN/DN/TT web-consoles don't support any form of authentication.
> Hadoop RPC API already supports Kerberos authentication.
> Kerberos enables single sign-on.
> Popular browsers (Firefox and Internet Explorer) have support for Kerberos HTTP SPNEGO.
> Adding support for Kerberos HTTP SPNEGO to Hadoop web consoles would provide a unified authentication mechanism and single sign-on for Hadoop web UI and Hadoop RPC.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15915-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 20:23:22 2011
Return-Path: <common-issues-return-15915-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 66F7D4225
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 20:23:22 +0000 (UTC)
Received: (qmail 77981 invoked by uid 500); 6 Jun 2011 20:23:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77946 invoked by uid 500); 6 Jun 2011 20:23:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77938 invoked by uid 99); 6 Jun 2011 20:23:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 20:23:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 20:23:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3C004104D14
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 20:22:59 +0000 (UTC)
Date: Mon, 6 Jun 2011 20:22:59 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <340734413.1734.1307391779242.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <671647059.63008.1307027027379.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7353) Cleanup FsShell and prevent
 masking of RTE stacktraces
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045068#comment-13045068 ] 

Aaron T. Myers commented on HADOOP-7353:
----------------------------------------

+1, latest patch looks good to me.

> Cleanup FsShell and prevent masking of RTE stacktraces
> ------------------------------------------------------
>
>                 Key: HADOOP-7353
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7353
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7353-2.patch, HADOOP-7353.patch
>
>
> {{FsShell}}'s top level exception handler catches and displays exceptions.  Unfortunately it displays only the first line of an exception, which means an unexpected {{RuntimeExceptions}} like {{NullPointerException}} only display "{{cmd: NullPointerException}}".  This user has no context to understand and/or accurately report the issue.
> Found due to bugs such as {{HADOOP-7327}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15916-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 20:49:22 2011
Return-Path: <common-issues-return-15916-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 21F3749F3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 20:49:22 +0000 (UTC)
Received: (qmail 40486 invoked by uid 500); 6 Jun 2011 20:49:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40460 invoked by uid 500); 6 Jun 2011 20:49:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40452 invoked by uid 99); 6 Jun 2011 20:49:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 20:49:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 20:49:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F381A1047AA
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 20:48:58 +0000 (UTC)
Date: Mon, 6 Jun 2011 20:48:58 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1938764894.1836.1307393338994.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1043064090.64591.1307057267396.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7356) RPM packages broke bin/hadoop
 script for hadoop 0.20.205
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045088#comment-13045088 ] 

Todd Lipcon commented on HADOOP-7356:
-------------------------------------

we also need this for trunk, right?
{code}
todd@todd-w510:~/svn/hadoop-common-trunk-commit$ ./bin/hadoop fs -cat /asdf
./bin/hadoop: line 24: /home/todd/svn/hadoop-common-trunk-commit/bin/../libexec/hadoop-config.sh: No such file or directory
...
{code}

> RPM packages broke bin/hadoop script for hadoop 0.20.205
> --------------------------------------------------------
>
>                 Key: HADOOP-7356
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7356
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.204.0
>         Environment: Java 6, Redhat EL 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>         Attachments: HADOOP-7356-1.patch, HADOOP-7356.patch
>
>
> hadoop-config.sh has been moved to libexec for binary package, but developers prefers to have hadoop-config.sh in bin.  Hadoo shell scripts should be modified to support both scenarios.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15917-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 20:55:20 2011
Return-Path: <common-issues-return-15917-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6772C4B6B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 20:55:20 +0000 (UTC)
Received: (qmail 50501 invoked by uid 500); 6 Jun 2011 20:55:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50452 invoked by uid 500); 6 Jun 2011 20:55:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50422 invoked by uid 99); 6 Jun 2011 20:55:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 20:55:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 20:55:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1E8E910499C
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 20:54:59 +0000 (UTC)
Date: Mon, 6 Jun 2011 20:54:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1497643120.1845.1307393699121.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <671647059.63008.1307027027379.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7353) Cleanup FsShell and prevent masking
 of RTE stacktraces
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7353:
--------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

Committed to trunk based on atm's +1 and a quick sanity check. Thanks Daryn/Aaron.

> Cleanup FsShell and prevent masking of RTE stacktraces
> ------------------------------------------------------
>
>                 Key: HADOOP-7353
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7353
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7353-2.patch, HADOOP-7353.patch
>
>
> {{FsShell}}'s top level exception handler catches and displays exceptions.  Unfortunately it displays only the first line of an exception, which means an unexpected {{RuntimeExceptions}} like {{NullPointerException}} only display "{{cmd: NullPointerException}}".  This user has no context to understand and/or accurately report the issue.
> Found due to bugs such as {{HADOOP-7327}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15918-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 20:59:20 2011
Return-Path: <common-issues-return-15918-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 888BB4DB4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 20:59:20 +0000 (UTC)
Received: (qmail 58236 invoked by uid 500); 6 Jun 2011 20:59:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58153 invoked by uid 500); 6 Jun 2011 20:59:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58145 invoked by uid 99); 6 Jun 2011 20:59:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 20:59:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 20:59:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 67807104BC8
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 20:58:59 +0000 (UTC)
Date: Mon, 6 Jun 2011 20:58:59 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <418964100.1864.1307393939420.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <654871262.37888.1306191347393.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7323) Add capability to resolve
 compression codec based on codec name
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-7323:
------------------------------

    Fix Version/s:     (was: 0.22.0)
     Hadoop Flags: [Reviewed]
           Status: Patch Available  (was: Open)

> Add capability to resolve compression codec based on codec name
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7323
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7323
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-7323.patch, HADOOP-7323.patch, HADOOP-7323.patch, HADOOP-7323b.patch
>
>
> When setting up a compression codec in an MR job the full class name of the codec must be used.
> To ease usability, compression codecs should be resolved by their codec name (ie 'gzip', 'deflate', 'zlib', 'bzip2') instead their full codec class name.
> Besides easy of use for Hadoop users who would use the codec alias instead the full codec class name, it could simplify how HBase resolves loads the codecs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15919-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 20:59:22 2011
Return-Path: <common-issues-return-15919-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8D3AA4DC6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 20:59:22 +0000 (UTC)
Received: (qmail 58471 invoked by uid 500); 6 Jun 2011 20:59:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58436 invoked by uid 500); 6 Jun 2011 20:59:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58428 invoked by uid 99); 6 Jun 2011 20:59:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 20:59:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 20:59:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2931F104BC3
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 20:58:59 +0000 (UTC)
Date: Mon, 6 Jun 2011 20:58:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1758092272.1859.1307393939165.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1160101136.38164.1306200167436.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7325) hadoop command - do not accept class
 names starting with a hyphen
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7325:
--------------------------------

    Hadoop Flags: [Reviewed]

+1

> hadoop command - do not accept class names starting with a hyphen
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7325
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Brock Noland
>            Assignee: Brock Noland
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-illegal-class-name-0.patch, hadoop-illegal-class-name-1.patch, hadoop-illegal-class-name-2.patch
>
>
> If this is committed I will look at patches for hdfs and mapred.
> When teaching a good portion of the students in every single class execute:
> {code}
> $ hadoop -fs ls /
> {code}
> The -fs is passed directly to the JVM and the JVM fails to start:
> {code}
> $ ./bin/hadoop -fs ls /
> Unrecognized option: -fs
> Could not create the Java virtual machine.
> {code}
> Which is confusing and typically requires explanation. The attached patch improves that behavior:
> {code}
> $ ./bin/hadoop -fs ls /
> Error: No command named `-fs' was found. Perhaps you meant `hadoop fs'
> {code}
> The only risk I can see is if someone is abusing the implementation of hadoop command doing something like so:
> {code}
> $ ./bin/hadoop -Xmx1g  org.apache.hadoop.util.RunJar
> RunJar jarFile [mainClass] args...
> {code}
> The hadoop command does not appear to advertise allowing JVM options before the classname.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15920-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 20:59:23 2011
Return-Path: <common-issues-return-15920-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 14E2C4DD6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 20:59:23 +0000 (UTC)
Received: (qmail 58721 invoked by uid 500); 6 Jun 2011 20:59:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58677 invoked by uid 500); 6 Jun 2011 20:59:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58669 invoked by uid 99); 6 Jun 2011 20:59:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 20:59:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 20:59:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B3F3E104BD3
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 20:58:59 +0000 (UTC)
Date: Mon, 6 Jun 2011 20:58:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1183989455.1874.1307393939734.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1160101136.38164.1306200167436.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7325) hadoop command - do not accept class
 names starting with a hyphen
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7325:
--------------------------------

       Resolution: Fixed
    Fix Version/s: 0.22.0
           Status: Resolved  (was: Patch Available)

Committed to trunk and 0.22. Thanks, Brock! Please do file patches for bin/hdfs and bin/mapred in those projects as well.

> hadoop command - do not accept class names starting with a hyphen
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7325
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Brock Noland
>            Assignee: Brock Noland
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-illegal-class-name-0.patch, hadoop-illegal-class-name-1.patch, hadoop-illegal-class-name-2.patch
>
>
> If this is committed I will look at patches for hdfs and mapred.
> When teaching a good portion of the students in every single class execute:
> {code}
> $ hadoop -fs ls /
> {code}
> The -fs is passed directly to the JVM and the JVM fails to start:
> {code}
> $ ./bin/hadoop -fs ls /
> Unrecognized option: -fs
> Could not create the Java virtual machine.
> {code}
> Which is confusing and typically requires explanation. The attached patch improves that behavior:
> {code}
> $ ./bin/hadoop -fs ls /
> Error: No command named `-fs' was found. Perhaps you meant `hadoop fs'
> {code}
> The only risk I can see is if someone is abusing the implementation of hadoop command doing something like so:
> {code}
> $ ./bin/hadoop -Xmx1g  org.apache.hadoop.util.RunJar
> RunJar jarFile [mainClass] args...
> {code}
> The hadoop command does not appear to advertise allowing JVM options before the classname.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15921-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 21:05:22 2011
Return-Path: <common-issues-return-15921-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 948574742
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 21:05:22 +0000 (UTC)
Received: (qmail 74035 invoked by uid 500); 6 Jun 2011 21:05:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74005 invoked by uid 500); 6 Jun 2011 21:05:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73997 invoked by uid 99); 6 Jun 2011 21:05:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:05:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:05:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5F1C4104EEE
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 21:04:59 +0000 (UTC)
Date: Mon, 6 Jun 2011 21:04:59 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1727294441.1911.1307394299386.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1043064090.64591.1307057267396.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7356) RPM packages broke bin/hadoop script
 for hadoop 0.20.205
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-7356:
------------------------------

    Attachment: HADOOP-7356-trunk.patch

Same patch for trunk.

> RPM packages broke bin/hadoop script for hadoop 0.20.205
> --------------------------------------------------------
>
>                 Key: HADOOP-7356
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7356
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.204.0
>         Environment: Java 6, Redhat EL 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>         Attachments: HADOOP-7356-1.patch, HADOOP-7356-trunk.patch, HADOOP-7356.patch
>
>
> hadoop-config.sh has been moved to libexec for binary package, but developers prefers to have hadoop-config.sh in bin.  Hadoo shell scripts should be modified to support both scenarios.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15922-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 21:15:53 2011
Return-Path: <common-issues-return-15922-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3175C4F3C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 21:15:53 +0000 (UTC)
Received: (qmail 98556 invoked by uid 500); 6 Jun 2011 21:15:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98500 invoked by uid 500); 6 Jun 2011 21:15:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98372 invoked by uid 99); 6 Jun 2011 21:15:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:15:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:15:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CBB141043EE
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 21:15:31 +0000 (UTC)
Date: Mon, 6 Jun 2011 21:15:31 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <961978861.1976.1307394931831.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <671647059.63008.1307027027379.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7353) Cleanup FsShell and prevent
 masking of RTE stacktraces
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045119#comment-13045119 ] 

Hudson commented on HADOOP-7353:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #643 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/643/])
    HADOOP-7353. Cleanup FsShell and prevent masking of RTE stack traces. Contributed by Daryn Sharp.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132764
Files : 
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Count.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/FsCommand.java
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/CommandFactory.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/cli/testConf.xml
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Command.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Display.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/FsShell.java


> Cleanup FsShell and prevent masking of RTE stacktraces
> ------------------------------------------------------
>
>                 Key: HADOOP-7353
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7353
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7353-2.patch, HADOOP-7353.patch
>
>
> {{FsShell}}'s top level exception handler catches and displays exceptions.  Unfortunately it displays only the first line of an exception, which means an unexpected {{RuntimeExceptions}} like {{NullPointerException}} only display "{{cmd: NullPointerException}}".  This user has no context to understand and/or accurately report the issue.
> Found due to bugs such as {{HADOOP-7327}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15923-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 21:15:54 2011
Return-Path: <common-issues-return-15923-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C47B44F62
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 21:15:54 +0000 (UTC)
Received: (qmail 98900 invoked by uid 500); 6 Jun 2011 21:15:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98862 invoked by uid 500); 6 Jun 2011 21:15:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98854 invoked by uid 99); 6 Jun 2011 21:15:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:15:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:15:52 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6B3C11043DD
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 21:15:31 +0000 (UTC)
Date: Mon, 6 Jun 2011 21:15:31 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <621194069.1964.1307394931435.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1043064090.64591.1307057267396.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7356) RPM packages broke bin/hadoop
 script for hadoop 0.20.205
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045116#comment-13045116 ] 

Todd Lipcon commented on HADOOP-7356:
-------------------------------------

+1 for trunk patch. I'll commit there, and leave others to review the 205 branch.

> RPM packages broke bin/hadoop script for hadoop 0.20.205
> --------------------------------------------------------
>
>                 Key: HADOOP-7356
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7356
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.204.0
>         Environment: Java 6, Redhat EL 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>         Attachments: HADOOP-7356-1.patch, HADOOP-7356-trunk.patch, HADOOP-7356.patch
>
>
> hadoop-config.sh has been moved to libexec for binary package, but developers prefers to have hadoop-config.sh in bin.  Hadoo shell scripts should be modified to support both scenarios.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15924-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 21:15:55 2011
Return-Path: <common-issues-return-15924-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 00CAD4F71
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 21:15:55 +0000 (UTC)
Received: (qmail 99122 invoked by uid 500); 6 Jun 2011 21:15:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99098 invoked by uid 500); 6 Jun 2011 21:15:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99046 invoked by uid 99); 6 Jun 2011 21:15:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:15:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:15:52 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7365B1043DE
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 21:15:31 +0000 (UTC)
Date: Mon, 6 Jun 2011 21:15:31 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1490709616.1965.1307394931469.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <654871262.37888.1306191347393.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7323) Add capability to resolve
 compression codec based on codec name
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045117#comment-13045117 ] 

Hadoop QA commented on HADOOP-7323:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481301/HADOOP-7323.patch
  against trunk revision 1132769.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/584//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/584//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/584//console

This message is automatically generated.

> Add capability to resolve compression codec based on codec name
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7323
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7323
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-7323.patch, HADOOP-7323.patch, HADOOP-7323.patch, HADOOP-7323b.patch
>
>
> When setting up a compression codec in an MR job the full class name of the codec must be used.
> To ease usability, compression codecs should be resolved by their codec name (ie 'gzip', 'deflate', 'zlib', 'bzip2') instead their full codec class name.
> Besides easy of use for Hadoop users who would use the codec alias instead the full codec class name, it could simplify how HBase resolves loads the codecs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15925-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 21:15:55 2011
Return-Path: <common-issues-return-15925-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5B3DC4F85
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 21:15:55 +0000 (UTC)
Received: (qmail 99384 invoked by uid 500); 6 Jun 2011 21:15:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99339 invoked by uid 500); 6 Jun 2011 21:15:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99331 invoked by uid 99); 6 Jun 2011 21:15:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:15:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:15:52 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B9DA91043EA
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 21:15:31 +0000 (UTC)
Date: Mon, 6 Jun 2011 21:15:31 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <26782368.1974.1307394931757.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1160101136.38164.1306200167436.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7325) hadoop command - do not accept
 class names starting with a hyphen
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045118#comment-13045118 ] 

Hudson commented on HADOOP-7325:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #643 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/643/])
    HADOOP-7325. The hadoop command should not accept class names starting with a hyphen. Contributed by Brock Noland.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132769
Files : 
* /hadoop/common/trunk/bin/hadoop
* /hadoop/common/trunk/CHANGES.txt


> hadoop command - do not accept class names starting with a hyphen
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7325
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Brock Noland
>            Assignee: Brock Noland
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-illegal-class-name-0.patch, hadoop-illegal-class-name-1.patch, hadoop-illegal-class-name-2.patch
>
>
> If this is committed I will look at patches for hdfs and mapred.
> When teaching a good portion of the students in every single class execute:
> {code}
> $ hadoop -fs ls /
> {code}
> The -fs is passed directly to the JVM and the JVM fails to start:
> {code}
> $ ./bin/hadoop -fs ls /
> Unrecognized option: -fs
> Could not create the Java virtual machine.
> {code}
> Which is confusing and typically requires explanation. The attached patch improves that behavior:
> {code}
> $ ./bin/hadoop -fs ls /
> Error: No command named `-fs' was found. Perhaps you meant `hadoop fs'
> {code}
> The only risk I can see is if someone is abusing the implementation of hadoop command doing something like so:
> {code}
> $ ./bin/hadoop -Xmx1g  org.apache.hadoop.util.RunJar
> RunJar jarFile [mainClass] args...
> {code}
> The hadoop command does not appear to advertise allowing JVM options before the classname.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15926-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 21:19:21 2011
Return-Path: <common-issues-return-15926-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4458E4FD4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 21:19:21 +0000 (UTC)
Received: (qmail 4550 invoked by uid 500); 6 Jun 2011 21:19:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4460 invoked by uid 500); 6 Jun 2011 21:19:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4452 invoked by uid 99); 6 Jun 2011 21:19:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:19:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:19:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0556910455B
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 21:19:00 +0000 (UTC)
Date: Mon, 6 Jun 2011 21:19:00 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <797409138.1985.1307395140018.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1043064090.64591.1307057267396.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7356) RPM packages broke bin/hadoop
 script for hadoop 0.20.205
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045122#comment-13045122 ] 

Hadoop QA commented on HADOOP-7356:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481624/HADOOP-7356-trunk.patch
  against trunk revision 1132776.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/585//console

This message is automatically generated.

> RPM packages broke bin/hadoop script for hadoop 0.20.205
> --------------------------------------------------------
>
>                 Key: HADOOP-7356
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7356
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.204.0
>         Environment: Java 6, Redhat EL 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>         Attachments: HADOOP-7356-1.patch, HADOOP-7356-trunk.patch, HADOOP-7356.patch
>
>
> hadoop-config.sh has been moved to libexec for binary package, but developers prefers to have hadoop-config.sh in bin.  Hadoo shell scripts should be modified to support both scenarios.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15927-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 21:21:22 2011
Return-Path: <common-issues-return-15927-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7E58D458C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 21:21:22 +0000 (UTC)
Received: (qmail 8762 invoked by uid 500); 6 Jun 2011 21:21:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8710 invoked by uid 500); 6 Jun 2011 21:21:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8618 invoked by uid 99); 6 Jun 2011 21:21:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:21:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:21:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 219B710463E
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 21:20:59 +0000 (UTC)
Date: Mon, 6 Jun 2011 21:20:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1035210085.1994.1307395259134.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1043064090.64591.1307057267396.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7356) RPM packages broke bin/hadoop
 script for hadoop 0.20.205
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045124#comment-13045124 ] 

Todd Lipcon commented on HADOOP-7356:
-------------------------------------

Hudson's complaining above because it tried to apply the patch after I already did. I didn't wait for Hudson since it only affects the scripts which are tested manually anyway.

> RPM packages broke bin/hadoop script for hadoop 0.20.205
> --------------------------------------------------------
>
>                 Key: HADOOP-7356
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7356
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.204.0
>         Environment: Java 6, Redhat EL 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>         Attachments: HADOOP-7356-1.patch, HADOOP-7356-trunk.patch, HADOOP-7356.patch
>
>
> hadoop-config.sh has been moved to libexec for binary package, but developers prefers to have hadoop-config.sh in bin.  Hadoo shell scripts should be modified to support both scenarios.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15928-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 21:29:20 2011
Return-Path: <common-issues-return-15928-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 333F8489E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 21:29:20 +0000 (UTC)
Received: (qmail 32939 invoked by uid 500); 6 Jun 2011 21:29:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32909 invoked by uid 500); 6 Jun 2011 21:29:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32900 invoked by uid 99); 6 Jun 2011 21:29:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:29:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:29:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F187F10490D
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 21:28:58 +0000 (UTC)
Date: Mon, 6 Jun 2011 21:28:58 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1662758371.2010.1307395738985.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <640184672.61345.1306972728331.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
 compression codec classes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045127#comment-13045127 ] 

Todd Lipcon commented on HADOOP-7350:
-------------------------------------

hmm, the two most recent patches are identical best I can see...

> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
>                 Key: HADOOP-7350
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7350
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, io
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15929-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 21:31:20 2011
Return-Path: <common-issues-return-15929-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A6E7D4FE7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 21:31:20 +0000 (UTC)
Received: (qmail 37985 invoked by uid 500); 6 Jun 2011 21:31:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37949 invoked by uid 500); 6 Jun 2011 21:31:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37941 invoked by uid 99); 6 Jun 2011 21:31:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:31:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:31:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 65F83104A29
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 21:30:59 +0000 (UTC)
Date: Mon, 6 Jun 2011 21:30:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1288284645.2021.1307395859414.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Moved] (HADOOP-7361) Provide overwrite option
 (-overwrite/-f) in put and copyFromLocal command line options
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon moved HDFS-1608 to HADOOP-7361:
-------------------------------------------

          Component/s:     (was: hdfs client)
                       fs
    Affects Version/s:     (was: 0.20.2)
                           (was: 0.20.1)
                  Key: HADOOP-7361  (was: HDFS-1608)
              Project: Hadoop Common  (was: Hadoop HDFS)

> Provide overwrite option (-overwrite/-f) in put and copyFromLocal command line options
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7361
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7361
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Uma Maheswara Rao G
>            Priority: Minor
>         Attachments: HDFS-1608-src.patch, HDFS-1608-test.patch
>
>
> FileSystem has the API 
> *public void copyFromLocalFile(boolean delSrc, boolean overwrite, Path[] srcs, Path dst)*
>      
>                          
> This API provides overwrite option. But the mapping command line doesn't have this option. To maintain the consistency and better usage  the command line option also can support the overwrite option like to put the files forcefully. ( put [-f] <srcpath> <dstPath>) and also for copyFromLocal command line option.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15930-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 21:31:23 2011
Return-Path: <common-issues-return-15930-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EDFB84049
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 21:31:22 +0000 (UTC)
Received: (qmail 38765 invoked by uid 500); 6 Jun 2011 21:31:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38731 invoked by uid 500); 6 Jun 2011 21:31:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38723 invoked by uid 99); 6 Jun 2011 21:31:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:31:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:31:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 73D80104A2B
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 21:30:59 +0000 (UTC)
Date: Mon, 6 Jun 2011 21:30:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1507757141.2023.1307395859471.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7361) Provide overwrite option
 (-overwrite/-f) in put and copyFromLocal command line options
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7361:
--------------------------------

    Status: Open  (was: Patch Available)

moved the JIRA to common

Can you please provide an up-to-date patch against trunk? thx

> Provide overwrite option (-overwrite/-f) in put and copyFromLocal command line options
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7361
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7361
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Uma Maheswara Rao G
>            Priority: Minor
>         Attachments: HDFS-1608-src.patch, HDFS-1608-test.patch
>
>
> FileSystem has the API 
> *public void copyFromLocalFile(boolean delSrc, boolean overwrite, Path[] srcs, Path dst)*
>      
>                          
> This API provides overwrite option. But the mapping command line doesn't have this option. To maintain the consistency and better usage  the command line option also can support the overwrite option like to put the files forcefully. ( put [-f] <srcpath> <dstPath>) and also for copyFromLocal command line option.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15931-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 21:31:23 2011
Return-Path: <common-issues-return-15931-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AA0CE4070
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 21:31:23 +0000 (UTC)
Received: (qmail 39177 invoked by uid 500); 6 Jun 2011 21:31:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39133 invoked by uid 500); 6 Jun 2011 21:31:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39080 invoked by uid 99); 6 Jun 2011 21:31:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:31:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:31:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F19EB104A39
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 21:30:59 +0000 (UTC)
Date: Mon, 6 Jun 2011 21:30:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1126260074.2036.1307395859986.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7269) S3 Native should allow customizable
 file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7269:
--------------------------------

    Status: Open  (was: Patch Available)

canceling patch status while waiting on test improvement

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: HADOOP-7269-S3-metadata-001.diff, HADOOP-7269-S3-metadata-002.diff, HADOOP-7269-S3-metadata-003.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15932-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 21:35:21 2011
Return-Path: <common-issues-return-15932-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2DEAF400C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 21:35:21 +0000 (UTC)
Received: (qmail 43470 invoked by uid 500); 6 Jun 2011 21:35:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43412 invoked by uid 500); 6 Jun 2011 21:35:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43132 invoked by uid 99); 6 Jun 2011 21:35:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:35:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:35:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8B786104C5D
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 21:34:59 +0000 (UTC)
Date: Mon, 6 Jun 2011 21:34:59 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1795750470.2063.1307396099568.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1043064090.64591.1307057267396.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7356) RPM packages broke bin/hadoop
 script for hadoop 0.20.205
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045136#comment-13045136 ] 

Hudson commented on HADOOP-7356:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #644 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/644/])
    HADOOP-7356. RPM packages broke bin/hadoop script in developer environment. Contributed by Eric Yang.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132776
Files : 
* /hadoop/common/trunk/bin/hadoop-daemons.sh
* /hadoop/common/trunk/bin/stop-all.sh
* /hadoop/common/trunk/bin/hadoop
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/bin/hadoop-daemon.sh
* /hadoop/common/trunk/bin/start-all.sh
* /hadoop/common/trunk/bin/slaves.sh


> RPM packages broke bin/hadoop script for hadoop 0.20.205
> --------------------------------------------------------
>
>                 Key: HADOOP-7356
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7356
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.204.0
>         Environment: Java 6, Redhat EL 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>         Attachments: HADOOP-7356-1.patch, HADOOP-7356-trunk.patch, HADOOP-7356.patch
>
>
> hadoop-config.sh has been moved to libexec for binary package, but developers prefers to have hadoop-config.sh in bin.  Hadoo shell scripts should be modified to support both scenarios.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15933-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 21:43:22 2011
Return-Path: <common-issues-return-15933-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 537D64274
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 21:43:22 +0000 (UTC)
Received: (qmail 53453 invoked by uid 500); 6 Jun 2011 21:43:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53426 invoked by uid 500); 6 Jun 2011 21:43:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53418 invoked by uid 99); 6 Jun 2011 21:43:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:43:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:43:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0B244104F5A
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 21:42:59 +0000 (UTC)
Date: Mon, 6 Jun 2011 21:42:59 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <667184657.2110.1307396579042.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <640184672.61345.1306972728331.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7350) Use ServiceLoader to discover
 compression codec classes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-7350:
------------------------------

    Attachment: HADOOP-7350.patch

Sorry - this should be the right patch now.

> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
>                 Key: HADOOP-7350
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7350
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, io
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15934-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 21:45:22 2011
Return-Path: <common-issues-return-15934-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8EB1445AF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 21:45:22 +0000 (UTC)
Received: (qmail 54989 invoked by uid 500); 6 Jun 2011 21:45:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54961 invoked by uid 500); 6 Jun 2011 21:45:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54949 invoked by uid 99); 6 Jun 2011 21:45:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:45:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:45:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 15EF6104FF7
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 21:44:59 +0000 (UTC)
Date: Mon, 6 Jun 2011 21:44:59 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1097193812.2116.1307396699086.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1160101136.38164.1306200167436.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7325) hadoop command - do not accept
 class names starting with a hyphen
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045146#comment-13045146 ] 

Hudson commented on HADOOP-7325:
--------------------------------

Integrated in Hadoop-Common-22-branch #63 (See [https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/63/])
    HADOOP-7325. The hadoop command should not accept class names starting with a hyphen. Contributed by Brock Noland.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132768
Files : 
* /hadoop/common/branches/branch-0.22/bin/hadoop
* /hadoop/common/branches/branch-0.22/CHANGES.txt


> hadoop command - do not accept class names starting with a hyphen
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7325
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Brock Noland
>            Assignee: Brock Noland
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-illegal-class-name-0.patch, hadoop-illegal-class-name-1.patch, hadoop-illegal-class-name-2.patch
>
>
> If this is committed I will look at patches for hdfs and mapred.
> When teaching a good portion of the students in every single class execute:
> {code}
> $ hadoop -fs ls /
> {code}
> The -fs is passed directly to the JVM and the JVM fails to start:
> {code}
> $ ./bin/hadoop -fs ls /
> Unrecognized option: -fs
> Could not create the Java virtual machine.
> {code}
> Which is confusing and typically requires explanation. The attached patch improves that behavior:
> {code}
> $ ./bin/hadoop -fs ls /
> Error: No command named `-fs' was found. Perhaps you meant `hadoop fs'
> {code}
> The only risk I can see is if someone is abusing the implementation of hadoop command doing something like so:
> {code}
> $ ./bin/hadoop -Xmx1g  org.apache.hadoop.util.RunJar
> RunJar jarFile [mainClass] args...
> {code}
> The hadoop command does not appear to advertise allowing JVM options before the classname.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15935-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 21:53:20 2011
Return-Path: <common-issues-return-15935-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6E5614AD9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 21:53:20 +0000 (UTC)
Received: (qmail 67055 invoked by uid 500); 6 Jun 2011 21:53:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67026 invoked by uid 500); 6 Jun 2011 21:53:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67018 invoked by uid 99); 6 Jun 2011 21:53:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:53:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 21:53:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2D5121042BD
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 21:52:59 +0000 (UTC)
Date: Mon, 6 Jun 2011 21:52:59 +0000 (UTC)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1329992262.2136.1307397179181.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1738050873.57393.1306875887708.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7343) backport HADOOP-7008 and HADOOP-7042
 to branch-0.20-security
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-7343:
----------------------------------

       Resolution: Fixed
    Fix Version/s: 0.20.206.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

+1

I committed this. Thanks, Tom!

> backport HADOOP-7008 and HADOOP-7042 to branch-0.20-security
> ------------------------------------------------------------
>
>                 Key: HADOOP-7343
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7343
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.20.204.0
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>            Priority: Minor
>             Fix For: 0.20.206.0
>
>         Attachments: HADOOP-7343-20security.patch, HADOOP-7343-20security.patch
>
>
> backport HADOOP-7008 and HADOOP-7042 to branch-0.20-security so that we can enable test-patch.sh to have a configured number of acceptable findbugs and javadoc warnings

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15936-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun  6 22:05:20 2011
Return-Path: <common-issues-return-15936-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 66D33478E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  6 Jun 2011 22:05:20 +0000 (UTC)
Received: (qmail 93729 invoked by uid 500); 6 Jun 2011 22:05:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93707 invoked by uid 500); 6 Jun 2011 22:05:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93699 invoked by uid 99); 6 Jun 2011 22:05:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 22:05:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 22:05:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 118D4104680
	for <common-issues@hadoop.apache.org>; Mon,  6 Jun 2011 22:04:59 +0000 (UTC)
Date: Mon, 6 Jun 2011 22:04:59 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1829634008.2160.1307397899068.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <640184672.61345.1306972728331.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
 compression codec classes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045155#comment-13045155 ] 

Hadoop QA commented on HADOOP-7350:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481630/HADOOP-7350.patch
  against trunk revision 1132776.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/586//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/586//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/586//console

This message is automatically generated.

> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
>                 Key: HADOOP-7350
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7350
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, io
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15937-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 00:07:20 2011
Return-Path: <common-issues-return-15937-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 796E741EE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 00:07:20 +0000 (UTC)
Received: (qmail 81150 invoked by uid 500); 7 Jun 2011 00:07:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81114 invoked by uid 500); 7 Jun 2011 00:07:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81106 invoked by uid 99); 7 Jun 2011 00:07:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 00:07:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 00:07:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 47E3B104A6F
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 00:06:59 +0000 (UTC)
Date: Tue, 7 Jun 2011 00:06:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1596151778.2498.1307405219291.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <640184672.61345.1306972728331.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
 compression codec classes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045216#comment-13045216 ] 

Todd Lipcon commented on HADOOP-7350:
-------------------------------------

Curious: why the change to only load once in the static initializer?

> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
>                 Key: HADOOP-7350
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7350
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, io
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15938-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 10:54:23 2011
Return-Path: <common-issues-return-15938-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 586836149
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 10:54:23 +0000 (UTC)
Received: (qmail 3578 invoked by uid 500); 7 Jun 2011 10:54:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3394 invoked by uid 500); 7 Jun 2011 10:54:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3382 invoked by uid 99); 7 Jun 2011 10:54:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 10:54:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 10:54:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D43BE8949E
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 10:53:58 +0000 (UTC)
Date: Tue, 7 Jun 2011 10:53:58 +0000 (UTC)
From: "Min Zhou (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <405621887.55.1307444038866.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1992675736.52742.1306733207390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7339) Introduce a buffered checksum for
 avoiding frequently calls on Checksum.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045358#comment-13045358 ] 

Min Zhou commented on HADOOP-7339:
----------------------------------

terasort 1GB on the same cluster , each job is ran more than 10 times, using intel vtune for profiling. 

# buffer(8KB) + pure java crc32 , 58.6s (CPU)
# buffer(1MB) + intel ipp optimized zlib crc32 (JNI), 60.8s (CPU)
# buffer(64KB) + intel ipp optimized zlib crc32 (JNI), 61.3s (CPU)
# pure java crc32, 62.0s (CPU)
# buffer(8KB) + intel ipp optimized zlib crc32, 64.1s (CPU)
# zlib crc32(JNI), 69.8s (CPU)
# intel ipp optimized zlib crc32, 703.s (CPU) 

It shows that BufferedChecksum + PureJavaCrc32 should be the best choice.
Thanks Kungong for sharing and testing the original patch.

> Introduce a buffered checksum for avoiding frequently calls on Checksum.update()
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-7339
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7339
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Min Zhou
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7339-v1.diff, HADOOP-7339-v2.diff
>
>
> We found that PureJavaCRC32/CRC32.update() is the TOP 1 of the methods consuming CPU in a map side, and in reduce side, it cost a lots of CPU too.
> IFileOutputStream would frequently call Checksum.update() during writing a record. It's very common a MR key/value less than 512 bytes. Checksum.update() would be called every time writing a key/value. 
> Test case: terasort 100MB. 
> Checksum.update() calls has be reduced from 4030348 to 28069.  This method is not a hotspot anymore.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15939-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 10:56:22 2011
Return-Path: <common-issues-return-15939-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3E0DA619B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 10:56:22 +0000 (UTC)
Received: (qmail 4963 invoked by uid 500); 7 Jun 2011 10:56:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4933 invoked by uid 500); 7 Jun 2011 10:56:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4924 invoked by uid 99); 7 Jun 2011 10:56:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 10:56:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 10:56:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D6F9D8960D
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 10:55:58 +0000 (UTC)
Date: Tue, 7 Jun 2011 10:55:58 +0000 (UTC)
From: "Min Zhou (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <378456417.62.1307444158877.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1992675736.52742.1306733207390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7339) Introduce a buffered checksum for
 avoiding frequently calls on Checksum.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045359#comment-13045359 ] 

Min Zhou commented on HADOOP-7339:
----------------------------------

oops, some typos in the previous comment.

terasort 1GB on the same cluster , each job is ran more than 10 times, using intel vtune for profiling.

buffer(8KB) + pure java crc32 , 58.6s (CPU)
buffer(1MB) + intel ipp optimized zlib crc32 (JNI), 60.8s (CPU)
buffer(64KB) + intel ipp optimized zlib crc32 (JNI), 61.3s (CPU)
pure java crc32, 62.0s (CPU)
buffer(8KB) + intel ipp optimized zlib crc32, 64.1s (CPU)
zlib crc32(JNI), 69.8s (CPU)
intel ipp optimized zlib crc32, 70.3 s (CPU)
It shows that BufferedChecksum + PureJavaCrc32 should be the best choice.
Thanks Kungu for sharing and testing the original patch.

> Introduce a buffered checksum for avoiding frequently calls on Checksum.update()
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-7339
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7339
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Min Zhou
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7339-v1.diff, HADOOP-7339-v2.diff
>
>
> We found that PureJavaCRC32/CRC32.update() is the TOP 1 of the methods consuming CPU in a map side, and in reduce side, it cost a lots of CPU too.
> IFileOutputStream would frequently call Checksum.update() during writing a record. It's very common a MR key/value less than 512 bytes. Checksum.update() would be called every time writing a key/value. 
> Test case: terasort 100MB. 
> Checksum.update() calls has be reduced from 4030348 to 28069.  This method is not a hotspot anymore.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15940-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 10:58:23 2011
Return-Path: <common-issues-return-15940-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1A5DE6215
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 10:58:23 +0000 (UTC)
Received: (qmail 8360 invoked by uid 500); 7 Jun 2011 10:58:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8325 invoked by uid 500); 7 Jun 2011 10:58:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8317 invoked by uid 99); 7 Jun 2011 10:58:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 10:58:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 10:58:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4FB3A896D2
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 10:57:59 +0000 (UTC)
Date: Tue, 7 Jun 2011 10:57:59 +0000 (UTC)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1799999999.69.1307444279323.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7362) Create a Lock annotation to document
 locking (hierarchies) for methods
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Create a Lock annotation to document locking (hierarchies) for methods
----------------------------------------------------------------------

                 Key: HADOOP-7362
                 URL: https://issues.apache.org/jira/browse/HADOOP-7362
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Arun C Murthy
            Assignee: Arun C Murthy


It will be useful to have better developer docs via a Lock annotation to document locking (hierarchies) for methods.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15941-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 11:00:20 2011
Return-Path: <common-issues-return-15941-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 53DB3644E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 11:00:20 +0000 (UTC)
Received: (qmail 12439 invoked by uid 500); 7 Jun 2011 11:00:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12372 invoked by uid 500); 7 Jun 2011 11:00:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12358 invoked by uid 99); 7 Jun 2011 11:00:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 11:00:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 11:00:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D1112897CA
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 10:59:58 +0000 (UTC)
Date: Tue, 7 Jun 2011 10:59:58 +0000 (UTC)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1059055041.73.1307444398852.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1799999999.69.1307444279323.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7362) Create a Lock annotation to
 document locking (hierarchies) for methods
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045361#comment-13045361 ] 

Arun C Murthy commented on HADOOP-7362:
---------------------------------------

I'm open to other suggestions in-lieu of (custom) annotations...

Eventually it will be nice to hack jcarder or some such tool to understand these!

Thoughts?

> Create a Lock annotation to document locking (hierarchies) for methods
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-7362
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7362
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>
> It will be useful to have better developer docs via a Lock annotation to document locking (hierarchies) for methods.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15942-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 11:04:20 2011
Return-Path: <common-issues-return-15942-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 118C0680B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 11:04:20 +0000 (UTC)
Received: (qmail 15930 invoked by uid 500); 7 Jun 2011 11:04:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15897 invoked by uid 500); 7 Jun 2011 11:04:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15889 invoked by uid 99); 7 Jun 2011 11:04:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 11:04:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 11:04:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BFC058994B
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 11:03:58 +0000 (UTC)
Date: Tue, 7 Jun 2011 11:03:58 +0000 (UTC)
From: "Min Zhou (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1718904113.77.1307444638781.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1799999999.69.1307444279323.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7362) Create a Lock annotation to
 document locking (hierarchies) for methods
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045362#comment-13045362 ] 

Min Zhou commented on HADOOP-7362:
----------------------------------

+1 for the idea. This would be very nice for developers inspecting whether a method of a class(or its  inheritors) have a lock or not.


> Create a Lock annotation to document locking (hierarchies) for methods
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-7362
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7362
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>
> It will be useful to have better developer docs via a Lock annotation to document locking (hierarchies) for methods.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15943-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 11:16:20 2011
Return-Path: <common-issues-return-15943-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 371766E1E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 11:16:20 +0000 (UTC)
Received: (qmail 39900 invoked by uid 500); 7 Jun 2011 11:16:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39863 invoked by uid 500); 7 Jun 2011 11:16:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39855 invoked by uid 99); 7 Jun 2011 11:16:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 11:16:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 11:16:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EE37C89DFD
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 11:15:58 +0000 (UTC)
Date: Tue, 7 Jun 2011 11:15:58 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <338455421.90.1307445358972.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1160101136.38164.1306200167436.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7325) hadoop command - do not accept
 class names starting with a hyphen
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045365#comment-13045365 ] 

Hudson commented on HADOOP-7325:
--------------------------------

Integrated in Hadoop-Common-trunk #712 (See [https://builds.apache.org/job/Hadoop-Common-trunk/712/])
    HADOOP-7325. The hadoop command should not accept class names starting with a hyphen. Contributed by Brock Noland.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132769
Files : 
* /hadoop/common/trunk/bin/hadoop
* /hadoop/common/trunk/CHANGES.txt


> hadoop command - do not accept class names starting with a hyphen
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7325
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Brock Noland
>            Assignee: Brock Noland
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-illegal-class-name-0.patch, hadoop-illegal-class-name-1.patch, hadoop-illegal-class-name-2.patch
>
>
> If this is committed I will look at patches for hdfs and mapred.
> When teaching a good portion of the students in every single class execute:
> {code}
> $ hadoop -fs ls /
> {code}
> The -fs is passed directly to the JVM and the JVM fails to start:
> {code}
> $ ./bin/hadoop -fs ls /
> Unrecognized option: -fs
> Could not create the Java virtual machine.
> {code}
> Which is confusing and typically requires explanation. The attached patch improves that behavior:
> {code}
> $ ./bin/hadoop -fs ls /
> Error: No command named `-fs' was found. Perhaps you meant `hadoop fs'
> {code}
> The only risk I can see is if someone is abusing the implementation of hadoop command doing something like so:
> {code}
> $ ./bin/hadoop -Xmx1g  org.apache.hadoop.util.RunJar
> RunJar jarFile [mainClass] args...
> {code}
> The hadoop command does not appear to advertise allowing JVM options before the classname.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15944-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 11:16:21 2011
Return-Path: <common-issues-return-15944-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 02C7F6E38
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 11:16:21 +0000 (UTC)
Received: (qmail 40157 invoked by uid 500); 7 Jun 2011 11:16:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40116 invoked by uid 500); 7 Jun 2011 11:16:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40108 invoked by uid 99); 7 Jun 2011 11:16:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 11:16:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 11:16:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1800A89E02
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 11:15:59 +0000 (UTC)
Date: Tue, 7 Jun 2011 11:15:59 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <930634834.93.1307445359095.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1043064090.64591.1307057267396.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7356) RPM packages broke bin/hadoop
 script for hadoop 0.20.205
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045366#comment-13045366 ] 

Hudson commented on HADOOP-7356:
--------------------------------

Integrated in Hadoop-Common-trunk #712 (See [https://builds.apache.org/job/Hadoop-Common-trunk/712/])
    HADOOP-7356. RPM packages broke bin/hadoop script in developer environment. Contributed by Eric Yang.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132776
Files : 
* /hadoop/common/trunk/bin/hadoop-daemons.sh
* /hadoop/common/trunk/bin/stop-all.sh
* /hadoop/common/trunk/bin/hadoop
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/bin/hadoop-daemon.sh
* /hadoop/common/trunk/bin/start-all.sh
* /hadoop/common/trunk/bin/slaves.sh


> RPM packages broke bin/hadoop script for hadoop 0.20.205
> --------------------------------------------------------
>
>                 Key: HADOOP-7356
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7356
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.204.0
>         Environment: Java 6, Redhat EL 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>         Attachments: HADOOP-7356-1.patch, HADOOP-7356-trunk.patch, HADOOP-7356.patch
>
>
> hadoop-config.sh has been moved to libexec for binary package, but developers prefers to have hadoop-config.sh in bin.  Hadoo shell scripts should be modified to support both scenarios.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15945-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 11:16:22 2011
Return-Path: <common-issues-return-15945-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6F1726E66
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 11:16:22 +0000 (UTC)
Received: (qmail 40413 invoked by uid 500); 7 Jun 2011 11:16:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40380 invoked by uid 500); 7 Jun 2011 11:16:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40370 invoked by uid 99); 7 Jun 2011 11:16:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 11:16:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 11:16:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 26CE589E05
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 11:15:59 +0000 (UTC)
Date: Tue, 7 Jun 2011 11:15:59 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1360093057.95.1307445359155.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <671647059.63008.1307027027379.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7353) Cleanup FsShell and prevent
 masking of RTE stacktraces
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045367#comment-13045367 ] 

Hudson commented on HADOOP-7353:
--------------------------------

Integrated in Hadoop-Common-trunk #712 (See [https://builds.apache.org/job/Hadoop-Common-trunk/712/])
    HADOOP-7353. Cleanup FsShell and prevent masking of RTE stack traces. Contributed by Daryn Sharp.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132764
Files : 
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Count.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/FsCommand.java
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/CommandFactory.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/cli/testConf.xml
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Command.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Display.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/FsShell.java


> Cleanup FsShell and prevent masking of RTE stacktraces
> ------------------------------------------------------
>
>                 Key: HADOOP-7353
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7353
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7353-2.patch, HADOOP-7353.patch
>
>
> {{FsShell}}'s top level exception handler catches and displays exceptions.  Unfortunately it displays only the first line of an exception, which means an unexpected {{RuntimeExceptions}} like {{NullPointerException}} only display "{{cmd: NullPointerException}}".  This user has no context to understand and/or accurately report the issue.
> Found due to bugs such as {{HADOOP-7327}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15946-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 12:01:22 2011
Return-Path: <common-issues-return-15946-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3B82860A3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 12:01:22 +0000 (UTC)
Received: (qmail 16455 invoked by uid 500); 7 Jun 2011 12:01:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16411 invoked by uid 500); 7 Jun 2011 12:01:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16403 invoked by uid 99); 7 Jun 2011 12:01:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 12:01:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 12:01:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5B637B81BD
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 12:01:00 +0000 (UTC)
Date: Tue, 7 Jun 2011 12:01:00 +0000 (UTC)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <568806838.165.1307448060370.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1799999999.69.1307444279323.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7362) Create a Lock annotation to
 document locking (hierarchies) for methods
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045382#comment-13045382 ] 

Arun C Murthy commented on HADOOP-7362:
---------------------------------------

bq. This would be very nice for developers inspecting whether a method of a class(or its inheritors) have a lock or not.

Precisely - it will aid developers to understand and reason locking, and serve as a documentation aid. 

I hope, eventually, we can tie this to a tool to do static/dynamic analysis e.g. jcarder. 

----

OTOH, is there an existing mechanism we should be using in-lieu of the proposed annotation?

> Create a Lock annotation to document locking (hierarchies) for methods
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-7362
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7362
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>
> It will be useful to have better developer docs via a Lock annotation to document locking (hierarchies) for methods.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15947-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 14:54:23 2011
Return-Path: <common-issues-return-15947-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 46059665B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 14:54:23 +0000 (UTC)
Received: (qmail 44656 invoked by uid 500); 7 Jun 2011 14:54:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44593 invoked by uid 500); 7 Jun 2011 14:54:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44584 invoked by uid 99); 7 Jun 2011 14:54:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 14:54:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 14:54:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 09648103539
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 14:53:59 +0000 (UTC)
Date: Tue, 7 Jun 2011 14:53:59 +0000 (UTC)
From: "jiraposter@reviews.apache.org (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <217543967.528.1307458439035.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <467948518.30516.1305901367779.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7314) Add support for throwing
 UnknownHostException when a host doesn't resolve
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045463#comment-13045463 ] 

jiraposter@reviews.apache.org commented on HADOOP-7314:
-------------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/775/#review775
-----------------------------------------------------------

Ship it!


+1 Looks good.

- Tom


On 2011-05-23 15:49:39, Jeffrey Naisbitt wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/775/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2011-05-23 15:49:39)
bq.  
bq.  
bq.  Review request for hadoop-common and hadoop-mapreduce.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  As part of MAPREDUCE-2489, we need support for having the resolve methods (for DNS mapping) throw UnknownHostExceptions. (Currently, they hide the exception). Since the existing 'resolve' method is ultimately used by several other locations/components, I propose we add a new 'resolveValidHosts' method.
bq.  
bq.  
bq.  This addresses bug HADOOP-7314.
bq.      https://issues.apache.org/jira/browse/HADOOP-7314
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    trunk/src/java/org/apache/hadoop/net/DNSToSwitchMapping.java 1125067 
bq.    trunk/src/java/org/apache/hadoop/net/ScriptBasedMapping.java 1125067 
bq.    trunk/src/test/core/org/apache/hadoop/net/StaticMapping.java 1125067 
bq.    trunk/src/test/core/org/apache/hadoop/net/TestScriptBasedMapping.java 1125067 
bq.    trunk/src/java/org/apache/hadoop/net/CachedDNSToSwitchMapping.java 1125067 
bq.  
bq.  Diff: https://reviews.apache.org/r/775/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Jeffrey
bq.  
bq.



> Add support for throwing UnknownHostException when a host doesn't resolve
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-7314
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7314
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jeffrey Naisbitt
>            Assignee: Jeffrey Naisbitt
>         Attachments: HADOOP-7314.patch
>
>
> As part of MAPREDUCE-2489, we need support for having the resolve methods (for DNS mapping) throw UnknownHostExceptions.  (Currently, they hide the exception).  Since the existing 'resolve' method is ultimately used by several other locations/components, I propose we add a new 'resolveValidHosts' method.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15948-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 16:30:22 2011
Return-Path: <common-issues-return-15948-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 591FC6DCF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 16:30:22 +0000 (UTC)
Received: (qmail 85484 invoked by uid 500); 7 Jun 2011 16:30:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85452 invoked by uid 500); 7 Jun 2011 16:30:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85443 invoked by uid 99); 7 Jun 2011 16:30:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 16:30:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 16:30:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DEE26103037
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 16:29:58 +0000 (UTC)
Date: Tue, 7 Jun 2011 16:29:58 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <241587118.771.1307464198909.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1799999999.69.1307444279323.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7362) Create a Lock annotation to
 document locking (hierarchies) for methods
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045521#comment-13045521 ] 

Tom White commented on HADOOP-7362:
-----------------------------------

The annotations from "Java Concurrency in Practice" may be appropriate here: http://jcip.net/annotations/doc/net/jcip/annotations/package-summary.html, in partciular http://jcip.net/annotations/doc/net/jcip/annotations/GuardedBy.html

FindBugs has support for these annotations, not sure about JCarder.

> Create a Lock annotation to document locking (hierarchies) for methods
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-7362
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7362
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>
> It will be useful to have better developer docs via a Lock annotation to document locking (hierarchies) for methods.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15949-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 16:50:29 2011
Return-Path: <common-issues-return-15949-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2ADF96209
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 16:50:29 +0000 (UTC)
Received: (qmail 26157 invoked by uid 500); 7 Jun 2011 16:50:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26085 invoked by uid 500); 7 Jun 2011 16:50:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26077 invoked by uid 99); 7 Jun 2011 16:50:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 16:50:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 16:50:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BB6CE1038C4
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 16:50:05 +0000 (UTC)
Date: Tue, 7 Jun 2011 16:50:05 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <127653748.807.1307465405763.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1799999999.69.1307444279323.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7362) Create a Lock annotation to
 document locking (hierarchies) for methods
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045532#comment-13045532 ] 

Todd Lipcon commented on HADOOP-7362:
-------------------------------------

JCarder doesn't currently support annotations, but I know that codebase pretty well, and could probably hack it in.

One very simple thing you could do that doesn't require any analysis at all is: assert Thread.holdsLock(x)

> Create a Lock annotation to document locking (hierarchies) for methods
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-7362
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7362
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>
> It will be useful to have better developer docs via a Lock annotation to document locking (hierarchies) for methods.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15950-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 16:54:21 2011
Return-Path: <common-issues-return-15950-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ECA90635D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 16:54:21 +0000 (UTC)
Received: (qmail 29040 invoked by uid 500); 7 Jun 2011 16:54:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29003 invoked by uid 500); 7 Jun 2011 16:54:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28994 invoked by uid 99); 7 Jun 2011 16:54:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 16:54:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 16:54:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C98CB103A40
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 16:53:58 +0000 (UTC)
Date: Tue, 7 Jun 2011 16:53:58 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1920403017.813.1307465638822.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1799999999.69.1307444279323.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7362) Create a Lock annotation to
 document locking (hierarchies) for methods
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045534#comment-13045534 ] 

Luke Lu commented on HADOOP-7362:
---------------------------------

JCarder doesn't use annotations, which is touted as its strength.

+1 for using JCIP annotations and maybe other findbugs annotations, especially @CheckForNull etc.


> Create a Lock annotation to document locking (hierarchies) for methods
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-7362
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7362
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>
> It will be useful to have better developer docs via a Lock annotation to document locking (hierarchies) for methods.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15951-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 17:01:20 2011
Return-Path: <common-issues-return-15951-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7A45A6D09
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 17:01:20 +0000 (UTC)
Received: (qmail 33807 invoked by uid 500); 7 Jun 2011 17:01:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33771 invoked by uid 500); 7 Jun 2011 17:01:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33763 invoked by uid 99); 7 Jun 2011 17:01:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 17:01:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 17:01:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 44A04103D1B
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 17:00:59 +0000 (UTC)
Date: Tue, 7 Jun 2011 17:00:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1841439711.824.1307466059277.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1799999999.69.1307444279323.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7362) Create a Lock annotation to
 document locking (hierarchies) for methods
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045536#comment-13045536 ] 

Todd Lipcon commented on HADOOP-7362:
-------------------------------------

I like the idea of using JCIP where possible, especially for its documentation value.

But in code paths that aren't hot/performance sensitive, please let's not rely on things like static analysis and @CheckForNull to catch bugs. Good old fashioned asserts and checks like Preconditions.checkNotNull() have the nice side effect that anyone running tests will find the bug, and you can't just ignore it like you can static analysis results :)

> Create a Lock annotation to document locking (hierarchies) for methods
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-7362
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7362
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>
> It will be useful to have better developer docs via a Lock annotation to document locking (hierarchies) for methods.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15952-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 17:32:22 2011
Return-Path: <common-issues-return-15952-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 393D66D25
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 17:32:22 +0000 (UTC)
Received: (qmail 3610 invoked by uid 500); 7 Jun 2011 17:32:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3563 invoked by uid 500); 7 Jun 2011 17:32:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3555 invoked by uid 99); 7 Jun 2011 17:32:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 17:32:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 17:32:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C9FB010683E
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 17:31:58 +0000 (UTC)
Date: Tue, 7 Jun 2011 17:31:58 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <158298796.897.1307467918824.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7363) TestRawLocalFileSystemContract is
 needed
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

TestRawLocalFileSystemContract is needed
----------------------------------------

                 Key: HADOOP-7363
                 URL: https://issues.apache.org/jira/browse/HADOOP-7363
             Project: Hadoop Common
          Issue Type: Test
          Components: fs
    Affects Versions: 0.23.0
            Reporter: Matt Foley
            Assignee: Matt Foley


FileSystemContractBaseTest is supposed to be run with each concrete FileSystem implementation to insure adherence to the "contract" for FileSystem behavior.  However, currently only HDFS and S3 do so.  RawLocalFileSystem, at least, needs to be added. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15953-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 17:42:22 2011
Return-Path: <common-issues-return-15953-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4D4026B29
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 17:42:22 +0000 (UTC)
Received: (qmail 24690 invoked by uid 500); 7 Jun 2011 17:42:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24647 invoked by uid 500); 7 Jun 2011 17:42:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24639 invoked by uid 99); 7 Jun 2011 17:42:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 17:42:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 17:42:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 149C8106C0F
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 17:41:59 +0000 (UTC)
Date: Tue, 7 Jun 2011 17:41:59 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <509619184.917.1307468519081.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1799999999.69.1307444279323.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7362) Create a Lock annotation to
 document locking (hierarchies) for methods
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045555#comment-13045555 ] 

Luke Lu commented on HADOOP-7362:
---------------------------------

The findbug annotations and runtime asserts are complimentary. You can use @CheckForNull on a field and findbugs will warn if checks (e.g., via Preconditions.checkNotNull) is not actually performed.

> Create a Lock annotation to document locking (hierarchies) for methods
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-7362
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7362
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>
> It will be useful to have better developer docs via a Lock annotation to document locking (hierarchies) for methods.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15954-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 17:58:20 2011
Return-Path: <common-issues-return-15954-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3831867B1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 17:58:20 +0000 (UTC)
Received: (qmail 50106 invoked by uid 500); 7 Jun 2011 17:58:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50068 invoked by uid 500); 7 Jun 2011 17:58:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50060 invoked by uid 99); 7 Jun 2011 17:58:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 17:58:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 17:58:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DDDC3106044
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 17:57:58 +0000 (UTC)
Date: Tue, 7 Jun 2011 17:57:58 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <649116224.935.1307469478905.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <640184672.61345.1306972728331.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7350) Use ServiceLoader to discover
 compression codec classes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-7350:
------------------------------

    Attachment: HADOOP-7350.patch

I moved it to take advantage of the ServiceLoader's caching of providers. This new patch does that but moves the iteration back to the getCodecClasses() method. (This is now like the example in the ServiceLoader documentation.)


> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
>                 Key: HADOOP-7350
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7350
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, io
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15955-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 18:14:22 2011
Return-Path: <common-issues-return-15955-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4BA9F62B8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 18:14:22 +0000 (UTC)
Received: (qmail 86238 invoked by uid 500); 7 Jun 2011 18:14:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86180 invoked by uid 500); 7 Jun 2011 18:14:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86171 invoked by uid 99); 7 Jun 2011 18:14:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 18:14:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 18:14:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D665B1065D1
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 18:13:58 +0000 (UTC)
Date: Tue, 7 Jun 2011 18:13:58 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1260460150.971.1307470438874.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <640184672.61345.1306972728331.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7350) Use ServiceLoader to discover
 compression codec classes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045564#comment-13045564 ] 

Hadoop QA commented on HADOOP-7350:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481726/HADOOP-7350.patch
  against trunk revision 1132776.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/587//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/587//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/587//console

This message is automatically generated.

> Use ServiceLoader to discover compression codec classes
> -------------------------------------------------------
>
>                 Key: HADOOP-7350
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7350
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, io
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch, HADOOP-7350.patch
>
>
> By using a ServiceLoader users wouldn't have to add codec classes to io.compression.codecs for codecs that aren't shipped with Hadoop (e.g. LZO), since they would be automatically picked up from the classpath.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15956-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 18:34:22 2011
Return-Path: <common-issues-return-15956-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B5E2F64FC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 18:34:22 +0000 (UTC)
Received: (qmail 38367 invoked by uid 500); 7 Jun 2011 18:34:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38337 invoked by uid 500); 7 Jun 2011 18:34:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38329 invoked by uid 99); 7 Jun 2011 18:34:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 18:34:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 18:34:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 51E44106D77
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 18:33:59 +0000 (UTC)
Date: Tue, 7 Jun 2011 18:33:59 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2034582327.1061.1307471639332.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <654871262.37888.1306191347393.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7323) Add capability to resolve
 compression codec based on codec name
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-7323:
------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I've just committed this. Thanks, Alejandro!

> Add capability to resolve compression codec based on codec name
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7323
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7323
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-7323.patch, HADOOP-7323.patch, HADOOP-7323.patch, HADOOP-7323b.patch
>
>
> When setting up a compression codec in an MR job the full class name of the codec must be used.
> To ease usability, compression codecs should be resolved by their codec name (ie 'gzip', 'deflate', 'zlib', 'bzip2') instead their full codec class name.
> Besides easy of use for Hadoop users who would use the codec alias instead the full codec class name, it could simplify how HBase resolves loads the codecs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15957-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 18:56:20 2011
Return-Path: <common-issues-return-15957-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 95CF26FB8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 18:56:20 +0000 (UTC)
Received: (qmail 71991 invoked by uid 500); 7 Jun 2011 18:56:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71958 invoked by uid 500); 7 Jun 2011 18:56:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71950 invoked by uid 99); 7 Jun 2011 18:56:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 18:56:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 18:56:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E9A7D10651D
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 18:55:58 +0000 (UTC)
Date: Tue, 7 Jun 2011 18:55:58 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <120748403.1093.1307472958953.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <654871262.37888.1306191347393.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7323) Add capability to resolve
 compression codec based on codec name
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045590#comment-13045590 ] 

Hudson commented on HADOOP-7323:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #645 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/645/])
    HADOOP-7323. Add capability to resolve compression codec based on codec name. Contributed by Alejandro Abdelnur.

tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1133125
Files : 
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/io/compress/TestCodecFactory.java
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/io/compress/TestCodec.java
* /hadoop/common/trunk/src/java/core-default.xml
* /hadoop/common/trunk/src/java/org/apache/hadoop/io/compress/CompressionCodecFactory.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/io/compress/DeflateCodec.java


> Add capability to resolve compression codec based on codec name
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7323
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7323
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-7323.patch, HADOOP-7323.patch, HADOOP-7323.patch, HADOOP-7323b.patch
>
>
> When setting up a compression codec in an MR job the full class name of the codec must be used.
> To ease usability, compression codecs should be resolved by their codec name (ie 'gzip', 'deflate', 'zlib', 'bzip2') instead their full codec class name.
> Besides easy of use for Hadoop users who would use the codec alias instead the full codec class name, it could simplify how HBase resolves loads the codecs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15958-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 20:26:25 2011
Return-Path: <common-issues-return-15958-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A28836D05
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 20:26:25 +0000 (UTC)
Received: (qmail 38546 invoked by uid 500); 7 Jun 2011 20:26:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38500 invoked by uid 500); 7 Jun 2011 20:26:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38448 invoked by uid 99); 7 Jun 2011 20:26:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 20:26:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 20:26:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C705D10624A
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 20:25:58 +0000 (UTC)
Date: Tue, 7 Jun 2011 20:25:58 +0000 (UTC)
From: "Thomas Graves (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <879608400.1240.1307478358811.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7364) TestMiniMRDFSCaching fails if
 test.build.dir is set to something other than build/test
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test
--------------------------------------------------------------------------------------

                 Key: HADOOP-7364
                 URL: https://issues.apache.org/jira/browse/HADOOP-7364
             Project: Hadoop Common
          Issue Type: Bug
          Components: test
    Affects Versions: 0.20.205.0
            Reporter: Thomas Graves
            Assignee: Thomas Graves


TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15959-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 21:17:27 2011
Return-Path: <common-issues-return-15959-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E7B4C6723
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 21:17:27 +0000 (UTC)
Received: (qmail 34021 invoked by uid 500); 7 Jun 2011 21:17:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33991 invoked by uid 500); 7 Jun 2011 21:17:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33976 invoked by uid 99); 7 Jun 2011 21:17:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 21:17:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 21:17:25 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 36BE5106576
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 21:17:04 +0000 (UTC)
Date: Tue, 7 Jun 2011 21:17:04 +0000 (UTC)
From: "Thomas Graves (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <653059739.1352.1307481424220.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <879608400.1240.1307478358811.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7364) TestMiniMRDFSCaching fails if
 test.build.dir is set to something other than build/test
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Graves updated HADOOP-7364:
----------------------------------

    Attachment: HADOOP-7364.patch

> TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7364
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7364
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.20.205.0
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>         Attachments: HADOOP-7364.patch
>
>
> TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15960-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 21:17:28 2011
Return-Path: <common-issues-return-15960-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EB37A672A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 21:17:27 +0000 (UTC)
Received: (qmail 34059 invoked by uid 500); 7 Jun 2011 21:17:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33994 invoked by uid 500); 7 Jun 2011 21:17:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33977 invoked by uid 99); 7 Jun 2011 21:17:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 21:17:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 21:17:25 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 24B97106574
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 21:17:04 +0000 (UTC)
Date: Tue, 7 Jun 2011 21:17:04 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1400491455.1350.1307481424146.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2063003633.1960.1297925244935.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7147) Native code does not build on
 BSD/OSX
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045630#comment-13045630 ] 

Allen Wittenauer commented on HADOOP-7147:
------------------------------------------

Actually, we probably want to ignore the call anyway.  Looks like setnetgrent suffers from some really bizarro portability issues. On Linux, setnetgrent returns 1 on success.  On Solaris, it returns 0.  

There is a side issue as well, at least on Solaris, where setnetgrent() will actually reset for *all* threads that are in the midst of enumerating netgroups.  I haven't verified if this is true on Linux or other platforms.

> Native code does not build on BSD/OSX
> -------------------------------------
>
>                 Key: HADOOP-7147
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7147
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>
> HADOOP-6864 uses the setnetgrent function in a way which is not compatible with BSD APIs, where the call returns void rather than int. This prevents the native libs from building on OSX, for example.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15961-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 21:27:24 2011
Return-Path: <common-issues-return-15961-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0268565E7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 21:27:24 +0000 (UTC)
Received: (qmail 47363 invoked by uid 500); 7 Jun 2011 21:27:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47321 invoked by uid 500); 7 Jun 2011 21:27:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47255 invoked by uid 99); 7 Jun 2011 21:27:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 21:27:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 21:27:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D3847106871
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 21:26:58 +0000 (UTC)
Date: Tue, 7 Jun 2011 21:26:58 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <714460080.1372.1307482018863.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7365) Clean up CHANGES.txt
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Clean up CHANGES.txt 
---------------------

                 Key: HADOOP-7365
                 URL: https://issues.apache.org/jira/browse/HADOOP-7365
             Project: Hadoop Common
          Issue Type: Task
            Reporter: Owen O'Malley
            Assignee: Owen O'Malley


CHANGES.txt has some typos and standardizing the formatting.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15962-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 21:31:20 2011
Return-Path: <common-issues-return-15962-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0045D6D57
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 21:31:20 +0000 (UTC)
Received: (qmail 53418 invoked by uid 500); 7 Jun 2011 21:31:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53390 invoked by uid 500); 7 Jun 2011 21:31:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53382 invoked by uid 99); 7 Jun 2011 21:31:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 21:31:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 21:31:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BEB23106955
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 21:30:58 +0000 (UTC)
Date: Tue, 7 Jun 2011 21:30:58 +0000 (UTC)
From: "Thomas Graves (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1021687240.1383.1307482258777.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <879608400.1240.1307478358811.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7364) TestMiniMRDFSCaching fails if
 test.build.dir is set to something other than build/test
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Graves updated HADOOP-7364:
----------------------------------

    Fix Version/s: 0.20.205.0
           Status: Patch Available  (was: Open)

> TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7364
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7364
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.20.205.0
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.20.205.0
>
>         Attachments: HADOOP-7364.patch
>
>
> TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15963-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 21:34:20 2011
Return-Path: <common-issues-return-15963-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4760B6BE0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 21:34:20 +0000 (UTC)
Received: (qmail 55491 invoked by uid 500); 7 Jun 2011 21:34:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55462 invoked by uid 500); 7 Jun 2011 21:34:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55452 invoked by uid 99); 7 Jun 2011 21:34:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 21:34:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 21:34:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DE805106A3B
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 21:33:58 +0000 (UTC)
Date: Tue, 7 Jun 2011 21:33:58 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1892899819.1387.1307482438908.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <879608400.1240.1307478358811.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7364) TestMiniMRDFSCaching fails if
 test.build.dir is set to something other than build/test
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045638#comment-13045638 ] 

Hadoop QA commented on HADOOP-7364:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481744/HADOOP-7364.patch
  against trunk revision 1133125.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 5 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/588//console

This message is automatically generated.

> TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7364
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7364
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.20.205.0
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.20.205.0
>
>         Attachments: HADOOP-7364.patch
>
>
> TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15964-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 21:48:22 2011
Return-Path: <common-issues-return-15964-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BE1836605
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 21:48:22 +0000 (UTC)
Received: (qmail 83885 invoked by uid 500); 7 Jun 2011 21:48:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83832 invoked by uid 500); 7 Jun 2011 21:48:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83824 invoked by uid 99); 7 Jun 2011 21:48:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 21:48:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 21:48:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 291B5106E1F
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 21:47:59 +0000 (UTC)
Date: Tue, 7 Jun 2011 21:47:59 +0000 (UTC)
From: "Thomas Graves (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1039693965.1392.1307483279165.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <879608400.1240.1307478358811.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7364) TestMiniMRDFSCaching fails if
 test.build.dir is set to something other than build/test
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045641#comment-13045641 ] 

Thomas Graves commented on HADOOP-7364:
---------------------------------------

patch is for branch-0.20-205 not trunk. 

> TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7364
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7364
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.20.205.0
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.20.205.0
>
>         Attachments: HADOOP-7364.patch
>
>
> TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15965-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 21:50:20 2011
Return-Path: <common-issues-return-15965-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F39C468C5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 21:50:19 +0000 (UTC)
Received: (qmail 89611 invoked by uid 500); 7 Jun 2011 21:50:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89575 invoked by uid 500); 7 Jun 2011 21:50:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89566 invoked by uid 99); 7 Jun 2011 21:50:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 21:50:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 21:50:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C5217106F17
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 21:49:58 +0000 (UTC)
Date: Tue, 7 Jun 2011 21:49:58 +0000 (UTC)
From: "Kihwal Lee (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <986267073.1402.1307483398804.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <879608400.1240.1307478358811.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7364) TestMiniMRDFSCaching fails if
 test.build.dir is set to something other than build/test
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045643#comment-13045643 ] 

Kihwal Lee commented on HADOOP-7364:
------------------------------------

The patch won't apply for trunk. It should be against "src/test/mapred/org/apache/hadoop/mapred/MRCaching.java" of the mapreduce repo.

+1 for the fix itself. I applied the patch to 0.20-security and the test passed. I ran the test with test.build.dir set to something other than "build/test".

> TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7364
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7364
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.20.205.0
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.20.205.0
>
>         Attachments: HADOOP-7364.patch
>
>
> TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15966-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 21:58:22 2011
Return-Path: <common-issues-return-15966-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E9AE86D94
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 21:58:22 +0000 (UTC)
Received: (qmail 10949 invoked by uid 500); 7 Jun 2011 21:58:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10915 invoked by uid 500); 7 Jun 2011 21:58:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10907 invoked by uid 99); 7 Jun 2011 21:58:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 21:58:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 21:58:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A86201062C6
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 21:57:59 +0000 (UTC)
Date: Tue, 7 Jun 2011 21:57:59 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1858395266.1440.1307483879685.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2063003633.1960.1297925244935.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7147) Native code does not build on
 BSD/OSX
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Allen Wittenauer updated HADOOP-7147:
-------------------------------------

    Attachment: hadoop-7147.patch

This patch cleans up the variable re-use as well as changes setnetgrent by ignoring the return code.

> Native code does not build on BSD/OSX
> -------------------------------------
>
>                 Key: HADOOP-7147
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7147
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Todd Lipcon
>         Attachments: hadoop-7147.patch
>
>
> HADOOP-6864 uses the setnetgrent function in a way which is not compatible with BSD APIs, where the call returns void rather than int. This prevents the native libs from building on OSX, for example.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15967-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 21:58:23 2011
Return-Path: <common-issues-return-15967-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EE1EC6DA4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 21:58:23 +0000 (UTC)
Received: (qmail 11211 invoked by uid 500); 7 Jun 2011 21:58:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11174 invoked by uid 500); 7 Jun 2011 21:58:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11162 invoked by uid 99); 7 Jun 2011 21:58:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 21:58:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 21:58:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 17F491062D7
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 21:58:00 +0000 (UTC)
Date: Tue, 7 Jun 2011 21:58:00 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1742860930.1448.1307483880094.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2063003633.1960.1297925244935.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7147) Native code does not build on
 BSD/OSX
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Allen Wittenauer updated HADOOP-7147:
-------------------------------------

             Assignee: Allen Wittenauer
    Affects Version/s: 0.22.0
               Status: Patch Available  (was: Open)

> Native code does not build on BSD/OSX
> -------------------------------------
>
>                 Key: HADOOP-7147
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7147
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Allen Wittenauer
>         Attachments: hadoop-7147.patch
>
>
> HADOOP-6864 uses the setnetgrent function in a way which is not compatible with BSD APIs, where the call returns void rather than int. This prevents the native libs from building on OSX, for example.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15968-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 22:05:23 2011
Return-Path: <common-issues-return-15968-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 27DE3659E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 22:05:23 +0000 (UTC)
Received: (qmail 28099 invoked by uid 500); 7 Jun 2011 22:05:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28063 invoked by uid 500); 7 Jun 2011 22:05:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28016 invoked by uid 99); 7 Jun 2011 22:05:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:05:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:05:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F1A5F106639
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 22:04:58 +0000 (UTC)
Date: Tue, 7 Jun 2011 22:04:58 +0000 (UTC)
From: "Thomas Graves (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1157739578.1480.1307484298986.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <879608400.1240.1307478358811.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7364) TestMiniMRDFSCaching fails if
 test.build.dir is set to something other than build/test
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Graves updated HADOOP-7364:
----------------------------------

    Attachment: HADOOP-7364-trunk.patch

patch for trunk.

> TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7364
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7364
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.20.205.0
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.20.205.0
>
>         Attachments: HADOOP-7364-trunk.patch, HADOOP-7364.patch
>
>
> TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15969-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 22:09:22 2011
Return-Path: <common-issues-return-15969-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AB279699F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 22:09:22 +0000 (UTC)
Received: (qmail 45454 invoked by uid 500); 7 Jun 2011 22:09:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45425 invoked by uid 500); 7 Jun 2011 22:09:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45417 invoked by uid 99); 7 Jun 2011 22:09:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:09:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:09:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7371D106879
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 22:08:59 +0000 (UTC)
Date: Tue, 7 Jun 2011 22:08:59 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1756372488.1517.1307484539469.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2063003633.1960.1297925244935.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7147) setnetgrent in native code is not
 portable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Allen Wittenauer updated HADOOP-7147:
-------------------------------------

    Summary: setnetgrent in native code is not portable  (was: Native code does not build on BSD/OSX)

> setnetgrent in native code is not portable
> ------------------------------------------
>
>                 Key: HADOOP-7147
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7147
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Allen Wittenauer
>         Attachments: hadoop-7147.patch
>
>
> HADOOP-6864 uses the setnetgrent function in a way which is not compatible with BSD APIs, where the call returns void rather than int. This prevents the native libs from building on OSX, for example.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15970-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 22:11:24 2011
Return-Path: <common-issues-return-15970-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9CEA46A88
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 22:11:24 +0000 (UTC)
Received: (qmail 49513 invoked by uid 500); 7 Jun 2011 22:11:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49352 invoked by uid 500); 7 Jun 2011 22:11:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49315 invoked by uid 99); 7 Jun 2011 22:11:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:11:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:11:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3B187106972
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 22:10:59 +0000 (UTC)
Date: Tue, 7 Jun 2011 22:10:59 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <768308636.1527.1307484659239.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7366) OS X puts Java headers in
 non-standard place
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

OS X puts Java headers in non-standard place
--------------------------------------------

                 Key: HADOOP-7366
                 URL: https://issues.apache.org/jira/browse/HADOOP-7366
             Project: Hadoop Common
          Issue Type: Bug
          Components: native
    Affects Versions: 0.22.0, 0.23.0
            Reporter: Allen Wittenauer
            Priority: Minor


On OS X, $JAVA_HOME doesn't have JNI headers.  Another check is needed to locate them.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15971-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 22:15:20 2011
Return-Path: <common-issues-return-15971-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 27A586CFC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 22:15:20 +0000 (UTC)
Received: (qmail 71675 invoked by uid 500); 7 Jun 2011 22:15:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71640 invoked by uid 500); 7 Jun 2011 22:15:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71632 invoked by uid 99); 7 Jun 2011 22:15:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:15:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:15:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E34FE106B94
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 22:14:58 +0000 (UTC)
Date: Tue, 7 Jun 2011 22:14:58 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <866739793.1546.1307484898927.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <879608400.1240.1307478358811.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7364) TestMiniMRDFSCaching fails if
 test.build.dir is set to something other than build/test
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045664#comment-13045664 ] 

Hadoop QA commented on HADOOP-7364:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481749/HADOOP-7364-trunk.patch
  against trunk revision 1133125.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 5 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/590//console

This message is automatically generated.

> TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7364
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7364
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.20.205.0
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.20.205.0
>
>         Attachments: HADOOP-7364-trunk.patch, HADOOP-7364.patch
>
>
> TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15972-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 22:15:21 2011
Return-Path: <common-issues-return-15972-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EACE56D1A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 22:15:20 +0000 (UTC)
Received: (qmail 71948 invoked by uid 500); 7 Jun 2011 22:15:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71915 invoked by uid 500); 7 Jun 2011 22:15:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71907 invoked by uid 99); 7 Jun 2011 22:15:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:15:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:15:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 79BC3106B9C
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 22:14:59 +0000 (UTC)
Date: Tue, 7 Jun 2011 22:14:59 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <852261344.1554.1307484899495.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2063003633.1960.1297925244935.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7147) setnetgrent in native code is not
 portable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045665#comment-13045665 ] 

Hadoop QA commented on HADOOP-7147:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481747/hadoop-7147.patch
  against trunk revision 1133125.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/589//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/589//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/589//console

This message is automatically generated.

> setnetgrent in native code is not portable
> ------------------------------------------
>
>                 Key: HADOOP-7147
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7147
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Allen Wittenauer
>         Attachments: hadoop-7147.patch
>
>
> HADOOP-6864 uses the setnetgrent function in a way which is not compatible with BSD APIs, where the call returns void rather than int. This prevents the native libs from building on OSX, for example.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15973-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 22:15:21 2011
Return-Path: <common-issues-return-15973-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2156F6D28
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 22:15:21 +0000 (UTC)
Received: (qmail 72145 invoked by uid 500); 7 Jun 2011 22:15:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72095 invoked by uid 500); 7 Jun 2011 22:15:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72001 invoked by uid 99); 7 Jun 2011 22:15:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:15:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:15:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id ACC40106BA3
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 22:14:59 +0000 (UTC)
Date: Tue, 7 Jun 2011 22:14:59 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <904358418.1560.1307484899704.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <768308636.1527.1307484659239.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7366) OS X puts Java headers in
 non-standard place
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Allen Wittenauer updated HADOOP-7366:
-------------------------------------

    Status: Patch Available  (was: Open)

> OS X puts Java headers in non-standard place
> --------------------------------------------
>
>                 Key: HADOOP-7366
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7366
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Allen Wittenauer
>            Priority: Minor
>         Attachments: hadoop-7366.patch
>
>
> On OS X, $JAVA_HOME doesn't have JNI headers.  Another check is needed to locate them.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15974-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 22:15:22 2011
Return-Path: <common-issues-return-15974-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CDDDE6D4D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 22:15:22 +0000 (UTC)
Received: (qmail 72473 invoked by uid 500); 7 Jun 2011 22:15:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72441 invoked by uid 500); 7 Jun 2011 22:15:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72433 invoked by uid 99); 7 Jun 2011 22:15:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:15:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:15:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8CC4D106B9F
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 22:14:59 +0000 (UTC)
Date: Tue, 7 Jun 2011 22:14:59 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <235540572.1556.1307484899573.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <768308636.1527.1307484659239.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7366) OS X puts Java headers in
 non-standard place
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Allen Wittenauer updated HADOOP-7366:
-------------------------------------

    Attachment: hadoop-7366.patch

If $JAVA_HOME/../Headers is a directory, we add this to the JNI_CPPFLAGS

> OS X puts Java headers in non-standard place
> --------------------------------------------
>
>                 Key: HADOOP-7366
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7366
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Allen Wittenauer
>            Priority: Minor
>         Attachments: hadoop-7366.patch
>
>
> On OS X, $JAVA_HOME doesn't have JNI headers.  Another check is needed to locate them.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15975-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 22:15:25 2011
Return-Path: <common-issues-return-15975-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 242436D60
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 22:15:23 +0000 (UTC)
Received: (qmail 72722 invoked by uid 500); 7 Jun 2011 22:15:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72691 invoked by uid 500); 7 Jun 2011 22:15:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72682 invoked by uid 99); 7 Jun 2011 22:15:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:15:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:15:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C3BA1106BA7
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 22:14:59 +0000 (UTC)
Date: Tue, 7 Jun 2011 22:14:59 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1745218163.1563.1307484899798.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2063003633.1960.1297925244935.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7147) setnetgrent in native code is not
 portable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Allen Wittenauer updated HADOOP-7147:
-------------------------------------

    Component/s: native

> setnetgrent in native code is not portable
> ------------------------------------------
>
>                 Key: HADOOP-7147
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7147
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Allen Wittenauer
>         Attachments: hadoop-7147.patch
>
>
> HADOOP-6864 uses the setnetgrent function in a way which is not compatible with BSD APIs, where the call returns void rather than int. This prevents the native libs from building on OSX, for example.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15976-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 22:17:20 2011
Return-Path: <common-issues-return-15976-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3D4AB671E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 22:17:20 +0000 (UTC)
Received: (qmail 78992 invoked by uid 500); 7 Jun 2011 22:17:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78958 invoked by uid 500); 7 Jun 2011 22:17:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78950 invoked by uid 99); 7 Jun 2011 22:17:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:17:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:17:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DB761106CB5
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 22:16:58 +0000 (UTC)
Date: Tue, 7 Jun 2011 22:16:58 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1164614461.1572.1307485018895.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <768308636.1527.1307484659239.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7366) OS X puts Java headers in
 non-standard place
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Allen Wittenauer reassigned HADOOP-7366:
----------------------------------------

    Assignee: Allen Wittenauer

> OS X puts Java headers in non-standard place
> --------------------------------------------
>
>                 Key: HADOOP-7366
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7366
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Allen Wittenauer
>            Assignee: Allen Wittenauer
>            Priority: Minor
>         Attachments: hadoop-7366.patch
>
>
> On OS X, $JAVA_HOME doesn't have JNI headers.  Another check is needed to locate them.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15977-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 22:33:20 2011
Return-Path: <common-issues-return-15977-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 02DCF67AE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 22:33:20 +0000 (UTC)
Received: (qmail 20433 invoked by uid 500); 7 Jun 2011 22:33:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20402 invoked by uid 500); 7 Jun 2011 22:33:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20394 invoked by uid 99); 7 Jun 2011 22:33:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:33:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:33:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C3504106430
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 22:32:58 +0000 (UTC)
Date: Tue, 7 Jun 2011 22:32:58 +0000 (UTC)
From: "Thomas Graves (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1034745950.1628.1307485978796.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <879608400.1240.1307478358811.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7364) TestMiniMRDFSCaching fails if
 test.build.dir is set to something other than build/test
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045680#comment-13045680 ] 

Thomas Graves commented on HADOOP-7364:
---------------------------------------

hudson failed because patch applies to MAPREDUCE for trunk, going to file a separate JIRA for that and attach patch for trunk there.

> TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7364
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7364
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.20.205.0
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.20.205.0
>
>         Attachments: HADOOP-7364-trunk.patch, HADOOP-7364.patch
>
>
> TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15978-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 22:35:20 2011
Return-Path: <common-issues-return-15978-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4C4136DC4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 22:35:20 +0000 (UTC)
Received: (qmail 22576 invoked by uid 500); 7 Jun 2011 22:35:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22501 invoked by uid 500); 7 Jun 2011 22:35:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22493 invoked by uid 99); 7 Jun 2011 22:35:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:35:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:35:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E6DA2106507
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 22:34:58 +0000 (UTC)
Date: Tue, 7 Jun 2011 22:34:58 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <386433635.1639.1307486098942.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <768308636.1527.1307484659239.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7366) OS X puts Java headers in
 non-standard place
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045681#comment-13045681 ] 

Hadoop QA commented on HADOOP-7366:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481753/hadoop-7366.patch
  against trunk revision 1133125.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/591//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/591//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/591//console

This message is automatically generated.

> OS X puts Java headers in non-standard place
> --------------------------------------------
>
>                 Key: HADOOP-7366
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7366
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Allen Wittenauer
>            Assignee: Allen Wittenauer
>            Priority: Minor
>         Attachments: hadoop-7366.patch
>
>
> On OS X, $JAVA_HOME doesn't have JNI headers.  Another check is needed to locate them.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15979-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 22:37:21 2011
Return-Path: <common-issues-return-15979-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C786B6E52
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 22:37:21 +0000 (UTC)
Received: (qmail 24421 invoked by uid 500); 7 Jun 2011 22:37:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24374 invoked by uid 500); 7 Jun 2011 22:37:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24319 invoked by uid 99); 7 Jun 2011 22:37:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:37:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:37:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 477691066A3
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 22:36:59 +0000 (UTC)
Date: Tue, 7 Jun 2011 22:36:59 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1573829836.1656.1307486219289.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <714460080.1372.1307482018863.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7365) Clean up CHANGES.txt
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7365:
----------------------------------

    Attachment: c.patch

> Clean up CHANGES.txt 
> ---------------------
>
>                 Key: HADOOP-7365
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7365
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: c.patch
>
>
> CHANGES.txt has some typos and standardizing the formatting.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15980-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 22:37:25 2011
Return-Path: <common-issues-return-15980-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 803B86EA9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 22:37:25 +0000 (UTC)
Received: (qmail 26133 invoked by uid 500); 7 Jun 2011 22:37:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26091 invoked by uid 500); 7 Jun 2011 22:37:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26049 invoked by uid 99); 7 Jun 2011 22:37:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:37:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:37:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6F4E71066AA
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 22:36:59 +0000 (UTC)
Date: Tue, 7 Jun 2011 22:36:59 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1895298949.1661.1307486219452.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <714460080.1372.1307482018863.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7365) Clean up CHANGES.txt
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7365:
----------------------------------

    Status: Patch Available  (was: Open)

> Clean up CHANGES.txt 
> ---------------------
>
>                 Key: HADOOP-7365
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7365
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: c.patch
>
>
> CHANGES.txt has some typos and standardizing the formatting.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15981-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 22:39:22 2011
Return-Path: <common-issues-return-15981-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 729A36FB0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 22:39:22 +0000 (UTC)
Received: (qmail 31305 invoked by uid 500); 7 Jun 2011 22:39:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31276 invoked by uid 500); 7 Jun 2011 22:39:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31268 invoked by uid 99); 7 Jun 2011 22:39:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:39:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:39:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1BB7810675B
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 22:38:59 +0000 (UTC)
Date: Tue, 7 Jun 2011 22:38:59 +0000 (UTC)
From: "Thomas Graves (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1355797565.1667.1307486339110.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <879608400.1240.1307478358811.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7364) TestMiniMRDFSCaching fails if
 test.build.dir is set to something other than build/test
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045687#comment-13045687 ] 

Thomas Graves commented on HADOOP-7364:
---------------------------------------

MAPREDUCE-2575 has patch for trunk.

> TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7364
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7364
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.20.205.0
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.20.205.0
>
>         Attachments: HADOOP-7364-trunk.patch, HADOOP-7364.patch
>
>
> TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15982-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 22:55:21 2011
Return-Path: <common-issues-return-15982-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0CB546E61
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 22:55:21 +0000 (UTC)
Received: (qmail 55270 invoked by uid 500); 7 Jun 2011 22:55:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55219 invoked by uid 500); 7 Jun 2011 22:55:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55153 invoked by uid 99); 7 Jun 2011 22:55:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:55:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:55:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F090E106B0E
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 22:54:58 +0000 (UTC)
Date: Tue, 7 Jun 2011 22:54:58 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1185920213.1690.1307487298981.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <714460080.1372.1307482018863.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7365) Clean up CHANGES.txt
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045693#comment-13045693 ] 

Hadoop QA commented on HADOOP-7365:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481758/c.patch
  against trunk revision 1133125.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/592//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/592//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/592//console

This message is automatically generated.

> Clean up CHANGES.txt 
> ---------------------
>
>                 Key: HADOOP-7365
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7365
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: c.patch
>
>
> CHANGES.txt has some typos and standardizing the formatting.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15983-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun  7 22:57:19 2011
Return-Path: <common-issues-return-15983-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DB5C46EA4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  7 Jun 2011 22:57:19 +0000 (UTC)
Received: (qmail 58023 invoked by uid 500); 7 Jun 2011 22:57:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57994 invoked by uid 500); 7 Jun 2011 22:57:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57986 invoked by uid 99); 7 Jun 2011 22:57:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:57:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2011 22:57:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B89A2106B79
	for <common-issues@hadoop.apache.org>; Tue,  7 Jun 2011 22:56:58 +0000 (UTC)
Date: Tue, 7 Jun 2011 22:56:58 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <812783938.1699.1307487418752.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <714460080.1372.1307482018863.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7365) Clean up CHANGES.txt
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045695#comment-13045695 ] 

Allen Wittenauer commented on HADOOP-7365:
------------------------------------------

Other than minor spacing issues and the fact that lots of people can't spell (myself included), it is much better. :)

> Clean up CHANGES.txt 
> ---------------------
>
>                 Key: HADOOP-7365
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7365
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: c.patch
>
>
> CHANGES.txt has some typos and standardizing the formatting.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15984-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 01:30:23 2011
Return-Path: <common-issues-return-15984-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0582A6187
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 01:30:23 +0000 (UTC)
Received: (qmail 83941 invoked by uid 500); 8 Jun 2011 01:30:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83888 invoked by uid 500); 8 Jun 2011 01:30:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83880 invoked by uid 99); 8 Jun 2011 01:30:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 01:30:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 01:30:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 86496106288
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 01:29:59 +0000 (UTC)
Date: Wed, 8 Jun 2011 01:29:59 +0000 (UTC)
From: "Joep Rottinghuis (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1760112850.2026.1307496599546.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-5611) C++ libraries do not build on Debian
 Lenny
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joep Rottinghuis updated HADOOP-5611:
-------------------------------------

    Affects Version/s: 0.20.205.0

This is not Debian specific. The same happens on RHEL6.
The root cause is a cleanup of header files in Gcc 4.3 which exposes missing includes.
As Sreekanth pointed out in HADOOP-5678, this is documented in http://www.cyrius.com/journal/2007/05/10#gcc-4.3-include

RHEL5 uses gcc 4.1.2
RHEL6 uses gcc 4.4.5
Ubuntu Natty Narwhal uses gcc 4.5.2

This should be applied to branch-0.20-security and the 0.20.205 branch.

Note that HADOOP-5611 must also be applied after this patch.

> C++ libraries do not build on Debian Lenny
> ------------------------------------------
>
>                 Key: HADOOP-5611
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5611
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.18.3, 0.19.0, 0.19.1, 0.20.205.0
>         Environment: Debian Lenny, 64 bit
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.20.2
>
>         Attachments: 0001-HADOOP-5611-Add-some-missing-includes-to-c-code-t.patch, HADOOP-5611-fixed.patch
>
>
> The compilation of the C++ util and Hadoop Pipes code generates some errors about undefined functions. It appears there are just some missing includes that must not affect some other systems. Attaching a patch that fixes this issue.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15985-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 01:34:22 2011
Return-Path: <common-issues-return-15985-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 75A0963F1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 01:34:22 +0000 (UTC)
Received: (qmail 85646 invoked by uid 500); 8 Jun 2011 01:34:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85609 invoked by uid 500); 8 Jun 2011 01:34:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85601 invoked by uid 99); 8 Jun 2011 01:34:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 01:34:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 01:34:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1D6721063CB
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 01:33:59 +0000 (UTC)
Date: Wed, 8 Jun 2011 01:33:59 +0000 (UTC)
From: "Joep Rottinghuis (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1518516072.2029.1307496839117.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-5611) C++ libraries do not build on
 Debian Lenny
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045743#comment-13045743 ] 

Joep Rottinghuis commented on HADOOP-5611:
------------------------------------------

Uhm, the note should read:

MAPREDUCE-1251 must also be applied after this patch.

> C++ libraries do not build on Debian Lenny
> ------------------------------------------
>
>                 Key: HADOOP-5611
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5611
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.18.3, 0.19.0, 0.19.1, 0.20.205.0
>         Environment: Debian Lenny, 64 bit
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.20.2
>
>         Attachments: 0001-HADOOP-5611-Add-some-missing-includes-to-c-code-t.patch, HADOOP-5611-fixed.patch
>
>
> The compilation of the C++ util and Hadoop Pipes code generates some errors about undefined functions. It appears there are just some missing includes that must not affect some other systems. Attaching a patch that fixes this issue.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15986-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 04:54:24 2011
Return-Path: <common-issues-return-15986-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 30514420D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 04:54:24 +0000 (UTC)
Received: (qmail 18419 invoked by uid 500); 8 Jun 2011 04:54:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18379 invoked by uid 500); 8 Jun 2011 04:54:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18369 invoked by uid 99); 8 Jun 2011 04:54:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 04:54:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 04:54:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 084A51070FB
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 04:53:59 +0000 (UTC)
Date: Wed, 8 Jun 2011 04:53:59 +0000 (UTC)
From: "Travis Crawford (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1946943168.2233.1307508839030.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <710160053.68701.1307208347759.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7359) Pluggable interface for cluster
 membership
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045778#comment-13045778 ] 

Travis Crawford commented on HADOOP-7359:
-----------------------------------------

Would anyone object to allowing the HostsReader to trigger refreshNodes? That would let Hadoop scan for or be notified of cluster membership changes and automagically do the Right Thing.

DETAILS

Taking a step back, this change would be the most useful if your authoritative source for machine roles is stored Somewhere Else and you want Hadoop to integrate. The posted diff simply lets you pull the lists of included/excluded hosts from such a source, but does not activate the new lists - you still need to refreshNodes.

Imagine you update the authoritative source with new/removed machines and want Hadoop to learn about the change (ZK watch, polling, etc.). It would be very handy for cluster membership & state changes to propagate without manual intervention as is needed today. Permitting HostsReader to call refreshNodes would accomplish this goal.

PROPOSED IMPLEMENTATION

Introduce a "Refreshable" interface that both FSNamesystem and JobTracker implement, that only defines a refreshNodes method. HostsReader would have an initialize method that takes a Refreshable and users could choose to call refreshNodes.

The current file-based cluster membership would continue to work exactly as it does today.

Sort of a bigger change, but potentially very useful at larger sites. If there's general agreement this would be useful I'll post a diff. If not, I still think there's value in this change as it means no more copy/pasting lists of machines from the machine database :)

> Pluggable interface for cluster membership
> ------------------------------------------
>
>                 Key: HADOOP-7359
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7359
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Travis Crawford
>         Attachments: HADOOP-7359.diff
>
>
> Currently Hadoop uses local files to determine cluster membership. With HDFS for example, dfs.hosts and dfs.hosts.exclude are used.
> To enable tighter integrations cluster membership should be an interface, with the current file-based functionality provided as the default implementation. The common case would be no functional change, however, sites could plug an alternative implementation in, such as pulling the machine lists from a machine database.
> DETAILS:
> Two machine lists, includes and excludes, are used to define cluster membership and state. HostsFileReader currently handles reading these lists from files, who's names are passed in by FSNamesystem for HDFS and JobTracker for MR.
> The proposed change is adding a HostsReader interface to common, and changing HostsFileReader to an abstract class that functions the same as today.
> Two new classes, DFSHostsFileReader and MRHostsFileReader, extend HostsFileReader and simply pass the appropriate file names in. These new classes are needed because config key names live outside common.
> Two new conf keys, defaulting to the file-based readers, would be added to choose a different hosts reader: dfs.namenode.hosts.reader.class mapreduce.jobtracker.hosts.reader.class
> Comments/suggestions? I have most of this written already but would love some feedback on the general idea before posting the diff.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15987-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 05:37:25 2011
Return-Path: <common-issues-return-15987-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 48EDD4044
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 05:37:25 +0000 (UTC)
Received: (qmail 44414 invoked by uid 500); 8 Jun 2011 05:37:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44376 invoked by uid 500); 8 Jun 2011 05:37:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44364 invoked by uid 99); 8 Jun 2011 05:37:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 05:37:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 05:37:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E1FEB107B7A
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 05:36:58 +0000 (UTC)
Date: Wed, 8 Jun 2011 05:36:58 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <519231866.2295.1307511418922.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1043064090.64591.1307057267396.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7356) RPM packages broke bin/hadoop script
 for hadoop 0.20.205
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7356:
----------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

Todd committed this.

> RPM packages broke bin/hadoop script for hadoop 0.20.205
> --------------------------------------------------------
>
>                 Key: HADOOP-7356
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7356
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.204.0
>         Environment: Java 6, Redhat EL 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7356-1.patch, HADOOP-7356-trunk.patch, HADOOP-7356.patch
>
>
> hadoop-config.sh has been moved to libexec for binary package, but developers prefers to have hadoop-config.sh in bin.  Hadoo shell scripts should be modified to support both scenarios.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15988-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 05:43:23 2011
Return-Path: <common-issues-return-15988-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D07314095
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 05:43:23 +0000 (UTC)
Received: (qmail 46440 invoked by uid 500); 8 Jun 2011 05:43:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46391 invoked by uid 500); 8 Jun 2011 05:43:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46374 invoked by uid 99); 8 Jun 2011 05:43:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 05:43:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 05:43:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CE895107CF6
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 05:42:58 +0000 (UTC)
Date: Wed, 8 Jun 2011 05:42:58 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1942644481.2302.1307511778842.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1043064090.64591.1307057267396.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7356) RPM packages broke bin/hadoop
 script for hadoop 0.20.205
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045795#comment-13045795 ] 

Todd Lipcon commented on HADOOP-7356:
-------------------------------------

Hey Owen. As I commented [here|https://issues.apache.org/jira/browse/HADOOP-7356?focusedCommentId=13045116&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13045116] I left this open since it hasn't been committed to 0.20-security yet. Given the title I thought you might want to commit there before marking resolved.

> RPM packages broke bin/hadoop script for hadoop 0.20.205
> --------------------------------------------------------
>
>                 Key: HADOOP-7356
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7356
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.204.0
>         Environment: Java 6, Redhat EL 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7356-1.patch, HADOOP-7356-trunk.patch, HADOOP-7356.patch
>
>
> hadoop-config.sh has been moved to libexec for binary package, but developers prefers to have hadoop-config.sh in bin.  Hadoo shell scripts should be modified to support both scenarios.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15989-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 06:09:28 2011
Return-Path: <common-issues-return-15989-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AFC944066
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 06:09:28 +0000 (UTC)
Received: (qmail 64814 invoked by uid 500); 8 Jun 2011 06:09:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64725 invoked by uid 500); 8 Jun 2011 06:09:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64698 invoked by uid 99); 8 Jun 2011 06:09:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 06:09:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 06:09:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F159E107389
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 06:08:58 +0000 (UTC)
Date: Wed, 8 Jun 2011 06:08:58 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <608244590.2388.1307513338985.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-5647) TestJobHistory fails if /tmp/_logs
 is not writable to. Testcase should not depend on /tmp
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley resolved HADOOP-5647.
-----------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
                   0.20.203.0

This was committed by Sharad.

> TestJobHistory fails if /tmp/_logs is not writable to. Testcase should not depend on /tmp
> -----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5647
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5647
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: HADOOP-5647.patch, yhadoop20-5647.patch
>
>
> TestJobHistory sets /tmp as hadoop.job.history.user.location to check if the history file is created in that directory or not. If /tmp/_logs is already created by some other user, this test will fail because of not having write permission.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15990-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 08:08:22 2011
Return-Path: <common-issues-return-15990-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C71214F9C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 08:08:22 +0000 (UTC)
Received: (qmail 56609 invoked by uid 500); 8 Jun 2011 08:08:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56581 invoked by uid 500); 8 Jun 2011 08:08:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56573 invoked by uid 99); 8 Jun 2011 08:08:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 08:08:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 08:08:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D7ABB107CA9
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 08:07:58 +0000 (UTC)
Date: Wed, 8 Jun 2011 08:07:58 +0000 (UTC)
From: "XieXianshan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1574380138.2512.1307520478880.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1142137633.58455.1306902167529.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7348) Modify the option of FsShell
 getmerge from [addnl] to [-nl] for more comprehensive
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

XieXianshan updated HADOOP-7348:
--------------------------------

    Attachment: HADOOP-7348.patch_2

> Modify the option of FsShell getmerge from [addnl] to [-nl] for more comprehensive
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7348
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7348
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Assignee: XieXianshan
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7348.patch, HADOOP-7348.patch_2
>
>
> The [addnl] option of FsShell getmerge should be either "true" or "false",but it is very hard to understand by users, especially  who`s never used this option before. 
> So,the [addnl] option should be changed to [-nl] for more comprehensive.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15991-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 08:20:20 2011
Return-Path: <common-issues-return-15991-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6F3744BAF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 08:20:20 +0000 (UTC)
Received: (qmail 84886 invoked by uid 500); 8 Jun 2011 08:20:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84770 invoked by uid 500); 8 Jun 2011 08:20:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84762 invoked by uid 99); 8 Jun 2011 08:20:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 08:20:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 08:20:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 057AD107251
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 08:19:59 +0000 (UTC)
Date: Wed, 8 Jun 2011 08:19:59 +0000 (UTC)
From: "XieXianshan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <939353310.2546.1307521199019.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1142137633.58455.1306902167529.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7348) Modify the option of FsShell
 getmerge from [addnl] to [-nl] for more comprehensive
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

XieXianshan updated HADOOP-7348:
--------------------------------

    Status: Patch Available  (was: Open)

Reuploading the patch.Thanks Daryn,you teach me so many things.

> Modify the option of FsShell getmerge from [addnl] to [-nl] for more comprehensive
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7348
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7348
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Assignee: XieXianshan
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7348.patch, HADOOP-7348.patch_2
>
>
> The [addnl] option of FsShell getmerge should be either "true" or "false",but it is very hard to understand by users, especially  who`s never used this option before. 
> So,the [addnl] option should be changed to [-nl] for more comprehensive.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15992-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 08:34:20 2011
Return-Path: <common-issues-return-15992-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0916F4338
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 08:34:20 +0000 (UTC)
Received: (qmail 2648 invoked by uid 500); 8 Jun 2011 08:34:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2621 invoked by uid 500); 8 Jun 2011 08:34:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2613 invoked by uid 99); 8 Jun 2011 08:34:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 08:34:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 08:34:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C3386107744
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 08:33:58 +0000 (UTC)
Date: Wed, 8 Jun 2011 08:33:58 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1156619647.2561.1307522038796.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1142137633.58455.1306902167529.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7348) Modify the option of FsShell
 getmerge from [addnl] to [-nl] for more comprehensive
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045836#comment-13045836 ] 

Hadoop QA commented on HADOOP-7348:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481783/HADOOP-7348.patch_2
  against trunk revision 1133125.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 4 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The applied patch generated 1068 javac compiler warnings (more than the trunk's current 1067 warnings).

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/593//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/593//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/593//console

This message is automatically generated.

> Modify the option of FsShell getmerge from [addnl] to [-nl] for more comprehensive
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7348
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7348
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Assignee: XieXianshan
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7348.patch, HADOOP-7348.patch_2
>
>
> The [addnl] option of FsShell getmerge should be either "true" or "false",but it is very hard to understand by users, especially  who`s never used this option before. 
> So,the [addnl] option should be changed to [-nl] for more comprehensive.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15993-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 08:38:20 2011
Return-Path: <common-issues-return-15993-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2F81D44BF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 08:38:20 +0000 (UTC)
Received: (qmail 10054 invoked by uid 500); 8 Jun 2011 08:38:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10016 invoked by uid 500); 8 Jun 2011 08:38:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10008 invoked by uid 99); 8 Jun 2011 08:38:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 08:38:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 08:38:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DC8E11078F9
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 08:37:58 +0000 (UTC)
Date: Wed, 8 Jun 2011 08:37:58 +0000 (UTC)
From: "Mahadev konar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <464986632.2564.1307522278899.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <879608400.1240.1307478358811.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7364) TestMiniMRDFSCaching fails if
 test.build.dir is set to something other than build/test
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mahadev konar updated HADOOP-7364:
----------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I just committed this to the branch. Thanks Tom. Will review/commit the trunk patch soon.

> TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7364
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7364
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.20.205.0
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.20.205.0
>
>         Attachments: HADOOP-7364-trunk.patch, HADOOP-7364.patch
>
>
> TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15994-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 11:16:20 2011
Return-Path: <common-issues-return-15994-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 894DB4AB3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 11:16:20 +0000 (UTC)
Received: (qmail 88306 invoked by uid 500); 8 Jun 2011 11:16:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88268 invoked by uid 500); 8 Jun 2011 11:16:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88257 invoked by uid 99); 8 Jun 2011 11:16:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 11:16:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 11:16:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DAF15107ACC
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 11:15:58 +0000 (UTC)
Date: Wed, 8 Jun 2011 11:15:58 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <150971706.2859.1307531758889.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <654871262.37888.1306191347393.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7323) Add capability to resolve
 compression codec based on codec name
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045905#comment-13045905 ] 

Hudson commented on HADOOP-7323:
--------------------------------

Integrated in Hadoop-Common-trunk #713 (See [https://builds.apache.org/job/Hadoop-Common-trunk/713/])
    HADOOP-7323. Add capability to resolve compression codec based on codec name. Contributed by Alejandro Abdelnur.

tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1133125
Files : 
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/io/compress/TestCodecFactory.java
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/io/compress/TestCodec.java
* /hadoop/common/trunk/src/java/core-default.xml
* /hadoop/common/trunk/src/java/org/apache/hadoop/io/compress/CompressionCodecFactory.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/io/compress/DeflateCodec.java


> Add capability to resolve compression codec based on codec name
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7323
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7323
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-7323.patch, HADOOP-7323.patch, HADOOP-7323.patch, HADOOP-7323b.patch
>
>
> When setting up a compression codec in an MR job the full class name of the codec must be used.
> To ease usability, compression codecs should be resolved by their codec name (ie 'gzip', 'deflate', 'zlib', 'bzip2') instead their full codec class name.
> Besides easy of use for Hadoop users who would use the codec alias instead the full codec class name, it could simplify how HBase resolves loads the codecs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15995-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 15:43:20 2011
Return-Path: <common-issues-return-15995-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 51DDD43F1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 15:43:20 +0000 (UTC)
Received: (qmail 19840 invoked by uid 500); 8 Jun 2011 15:43:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19720 invoked by uid 500); 8 Jun 2011 15:43:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19712 invoked by uid 99); 8 Jun 2011 15:43:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 15:43:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 15:43:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 223CB1075E4
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 15:42:59 +0000 (UTC)
Date: Wed, 8 Jun 2011 15:42:59 +0000 (UTC)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <218298693.3455.1307547779136.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1799999999.69.1307444279323.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7362) Create a Lock annotation to
 document locking (hierarchies) for methods
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046021#comment-13046021 ] 

Arun C Murthy commented on HADOOP-7362:
---------------------------------------

Thanks for the suggestion Tom.

The issue with GuardedBy is that it only takes only argument, which is a problem when you have multiple guarding a single method... any other ideas?

> Create a Lock annotation to document locking (hierarchies) for methods
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-7362
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7362
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>
> It will be useful to have better developer docs via a Lock annotation to document locking (hierarchies) for methods.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15996-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 16:28:20 2011
Return-Path: <common-issues-return-15996-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A8F1B4916
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 16:28:20 +0000 (UTC)
Received: (qmail 14378 invoked by uid 500); 8 Jun 2011 16:28:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14330 invoked by uid 500); 8 Jun 2011 16:28:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14322 invoked by uid 99); 8 Jun 2011 16:28:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 16:28:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 16:28:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6796F108CE1
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 16:27:59 +0000 (UTC)
Date: Wed, 8 Jun 2011 16:27:59 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1528308285.3669.1307550479421.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1799999999.69.1307444279323.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7362) Create a Lock annotation to
 document locking (hierarchies) for methods
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046051#comment-13046051 ] 

Tom White commented on HADOOP-7362:
-----------------------------------

You're right that GuardedBy is not suitable for documenting locking order. How about introducing a LockingOrder annotation that takes an array of values?

> Create a Lock annotation to document locking (hierarchies) for methods
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-7362
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7362
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>
> It will be useful to have better developer docs via a Lock annotation to document locking (hierarchies) for methods.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15997-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 16:38:22 2011
Return-Path: <common-issues-return-15997-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 51184450C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 16:38:22 +0000 (UTC)
Received: (qmail 45243 invoked by uid 500); 8 Jun 2011 16:38:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45202 invoked by uid 500); 8 Jun 2011 16:38:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45194 invoked by uid 99); 8 Jun 2011 16:38:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 16:38:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 16:38:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0EDD71080DA
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 16:37:59 +0000 (UTC)
Date: Wed, 8 Jun 2011 16:37:59 +0000 (UTC)
From: "Nicholas Telford (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1309266507.3683.1307551079057.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7269) S3 Native should allow customizable
 file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicholas Telford updated HADOOP-7269:
-------------------------------------

    Attachment: 0002-Added-check-that-metadata-was-set-to-unit-test.patch
                0001-Added-support-for-metadata-to-be-applied-to-objects-.patch

Added test that metadata was actually set on the file.

Re-based against trunk. The patches have been rebuilt, this time according to the guidelines here: http://www.apache.org/dev/git.html

Please disregard the old ".diff" patches I supplied and only use these.

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: 0001-Added-support-for-metadata-to-be-applied-to-objects-.patch, 0002-Added-check-that-metadata-was-set-to-unit-test.patch, HADOOP-7269-S3-metadata-001.diff, HADOOP-7269-S3-metadata-002.diff, HADOOP-7269-S3-metadata-003.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15998-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 16:40:22 2011
Return-Path: <common-issues-return-15998-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2D56646AE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 16:40:22 +0000 (UTC)
Received: (qmail 55163 invoked by uid 500); 8 Jun 2011 16:40:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55127 invoked by uid 500); 8 Jun 2011 16:40:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55119 invoked by uid 99); 8 Jun 2011 16:40:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 16:40:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 16:40:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DBADB1081BE
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 16:39:58 +0000 (UTC)
Date: Wed, 8 Jun 2011 16:39:58 +0000 (UTC)
From: "Nicholas Telford (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <906288486.3686.1307551198896.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7269) S3 Native should allow customizable
 file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicholas Telford updated HADOOP-7269:
-------------------------------------

    Status: Patch Available  (was: Open)

This patch only allows the writing of metadata for a new file. You cannot arbitrarily add metadata for an existing file, nor can you read existing metadata.

The major use-case for this is so that you can provide S3 with instructions about the file (e.g. Expires, Content-Disposition, etc.) so it's not actually as useless as it sounds.

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: 0001-Added-support-for-metadata-to-be-applied-to-objects-.patch, 0002-Added-check-that-metadata-was-set-to-unit-test.patch, HADOOP-7269-S3-metadata-001.diff, HADOOP-7269-S3-metadata-002.diff, HADOOP-7269-S3-metadata-003.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15999-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 16:44:22 2011
Return-Path: <common-issues-return-15999-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 27A1B486D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 16:44:22 +0000 (UTC)
Received: (qmail 65172 invoked by uid 500); 8 Jun 2011 16:44:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65132 invoked by uid 500); 8 Jun 2011 16:44:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65124 invoked by uid 99); 8 Jun 2011 16:44:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 16:44:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 16:44:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C8975108358
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 16:43:58 +0000 (UTC)
Date: Wed, 8 Jun 2011 16:43:58 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1896254372.3707.1307551438817.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7269) S3 Native should allow
 customizable file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046059#comment-13046059 ] 

Hadoop QA commented on HADOOP-7269:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481835/0002-Added-check-that-metadata-was-set-to-unit-test.patch
  against trunk revision 1133125.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 4 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/594//console

This message is automatically generated.

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: 0001-Added-support-for-metadata-to-be-applied-to-objects-.patch, 0002-Added-check-that-metadata-was-set-to-unit-test.patch, HADOOP-7269-S3-metadata-001.diff, HADOOP-7269-S3-metadata-002.diff, HADOOP-7269-S3-metadata-003.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16000-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 16:48:23 2011
Return-Path: <common-issues-return-16000-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7496B4173
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 16:48:23 +0000 (UTC)
Received: (qmail 79324 invoked by uid 500); 8 Jun 2011 16:48:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79262 invoked by uid 500); 8 Jun 2011 16:48:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79254 invoked by uid 99); 8 Jun 2011 16:48:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 16:48:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 16:48:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1369C1084B3
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 16:48:00 +0000 (UTC)
Date: Wed, 8 Jun 2011 16:48:00 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1611216763.3722.1307551680076.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7269) S3 Native should allow
 customizable file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046061#comment-13046061 ] 

Todd Lipcon commented on HADOOP-7269:
-------------------------------------

Hi Nicholas. Sorry, our patch-bot is stupid and can only apply one patch at a time. Can you just upload a single combined "p0" level diff?

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: 0001-Added-support-for-metadata-to-be-applied-to-objects-.patch, 0002-Added-check-that-metadata-was-set-to-unit-test.patch, HADOOP-7269-S3-metadata-001.diff, HADOOP-7269-S3-metadata-002.diff, HADOOP-7269-S3-metadata-003.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16001-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 16:52:20 2011
Return-Path: <common-issues-return-16001-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 972FA43EA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 16:52:20 +0000 (UTC)
Received: (qmail 96069 invoked by uid 500); 8 Jun 2011 16:52:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96046 invoked by uid 500); 8 Jun 2011 16:52:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96031 invoked by uid 99); 8 Jun 2011 16:52:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 16:52:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 16:52:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E87AA1086EE
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 16:51:58 +0000 (UTC)
Date: Wed, 8 Jun 2011 16:51:58 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <564626948.3730.1307551918948.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <714460080.1372.1307482018863.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7365) Clean up CHANGES.txt
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7365:
----------------------------------

    Attachment: c.patch

This patch cleans up CHANGES.txt by:
1. Normalizing committers names to their apache login.
2. Fixing typos.
3. Limiting the line length to 79 chars.
4. Normalizing a few of the jira names for jiras that moved projects.
5. Creating virtual-jira NOJIRA- ids for 3 changes that didn't have jiras.
6. Adding jiras for hadoop 1.0.
7. Normalizing the format in the old releases to what we currently use.

> Clean up CHANGES.txt 
> ---------------------
>
>                 Key: HADOOP-7365
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7365
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: c.patch, c.patch
>
>
> CHANGES.txt has some typos and standardizing the formatting.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16002-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 16:52:20 2011
Return-Path: <common-issues-return-16002-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 974E343EB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 16:52:20 +0000 (UTC)
Received: (qmail 96098 invoked by uid 500); 8 Jun 2011 16:52:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96049 invoked by uid 500); 8 Jun 2011 16:52:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96032 invoked by uid 99); 8 Jun 2011 16:52:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 16:52:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 16:52:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 28B7C108701
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 16:51:59 +0000 (UTC)
Date: Wed, 8 Jun 2011 16:51:59 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <543397601.3736.1307551919163.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1043064090.64591.1307057267396.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7356) RPM packages broke bin/hadoop script
 for hadoop 0.20.205
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-7356:
------------------------------

    Status: Patch Available  (was: Reopened)

Owen, please review and commit to 0.20.20[4-5] branch.  Thanks

> RPM packages broke bin/hadoop script for hadoop 0.20.205
> --------------------------------------------------------
>
>                 Key: HADOOP-7356
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7356
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.204.0
>         Environment: Java 6, Redhat EL 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7356-1.patch, HADOOP-7356-trunk.patch, HADOOP-7356.patch
>
>
> hadoop-config.sh has been moved to libexec for binary package, but developers prefers to have hadoop-config.sh in bin.  Hadoo shell scripts should be modified to support both scenarios.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16003-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 16:52:22 2011
Return-Path: <common-issues-return-16003-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 51B3B43F6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 16:52:22 +0000 (UTC)
Received: (qmail 96599 invoked by uid 500); 8 Jun 2011 16:52:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96570 invoked by uid 500); 8 Jun 2011 16:52:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96562 invoked by uid 99); 8 Jun 2011 16:52:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 16:52:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 16:52:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0C6D21086F8
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 16:51:59 +0000 (UTC)
Date: Wed, 8 Jun 2011 16:51:59 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <156801986.3733.1307551919047.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1043064090.64591.1307057267396.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Reopened] (HADOOP-7356) RPM packages broke bin/hadoop
 script for hadoop 0.20.205
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang reopened HADOOP-7356:
-------------------------------


Reopen for not yet committed to 0.20.20x branch.

> RPM packages broke bin/hadoop script for hadoop 0.20.205
> --------------------------------------------------------
>
>                 Key: HADOOP-7356
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7356
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.204.0
>         Environment: Java 6, Redhat EL 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7356-1.patch, HADOOP-7356-trunk.patch, HADOOP-7356.patch
>
>
> hadoop-config.sh has been moved to libexec for binary package, but developers prefers to have hadoop-config.sh in bin.  Hadoo shell scripts should be modified to support both scenarios.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16004-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 16:54:20 2011
Return-Path: <common-issues-return-16004-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 67E4544D2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 16:54:20 +0000 (UTC)
Received: (qmail 4889 invoked by uid 500); 8 Jun 2011 16:54:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4846 invoked by uid 500); 8 Jun 2011 16:54:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4838 invoked by uid 99); 8 Jun 2011 16:54:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 16:54:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 16:54:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F31151087D4
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 16:53:58 +0000 (UTC)
Date: Wed, 8 Jun 2011 16:53:58 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <276097238.3743.1307552038992.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <714460080.1372.1307482018863.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7365) Clean up CHANGES.txt
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046067#comment-13046067 ] 

Hadoop QA commented on HADOOP-7365:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481840/c.patch
  against trunk revision 1133125.

    -1 @author.  The patch appears to contain 2 @author tags which the Hadoop community has agreed to not allow in code contributions.

    +1 tests included.  The patch appears to include 2 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/595//console

This message is automatically generated.

> Clean up CHANGES.txt 
> ---------------------
>
>                 Key: HADOOP-7365
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7365
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: c.patch, c.patch
>
>
> CHANGES.txt has some typos and standardizing the formatting.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16005-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 17:20:20 2011
Return-Path: <common-issues-return-16005-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0B788456E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 17:20:20 +0000 (UTC)
Received: (qmail 69718 invoked by uid 500); 8 Jun 2011 17:20:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69673 invoked by uid 500); 8 Jun 2011 17:20:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69663 invoked by uid 99); 8 Jun 2011 17:20:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 17:20:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 17:20:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C81E71084B1
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 17:19:58 +0000 (UTC)
Date: Wed, 8 Jun 2011 17:19:58 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <34642033.3863.1307553598815.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7367) getgrouplist() in getGroup.c is not
 portable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

getgrouplist() in getGroup.c is not portable
--------------------------------------------

                 Key: HADOOP-7367
                 URL: https://issues.apache.org/jira/browse/HADOOP-7367
             Project: Hadoop Common
          Issue Type: Bug
          Components: native
    Affects Versions: 0.22.0, 0.23.0
         Environment: System V
            Reporter: Allen Wittenauer


getGroupIDList uses getgrouplist() to fetch the groups for a user.  Unfortunately, this routine is a BSD-specific call and is not present in most System V-based operating systems. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16006-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 17:22:22 2011
Return-Path: <common-issues-return-16006-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 01147462E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 17:22:22 +0000 (UTC)
Received: (qmail 76895 invoked by uid 500); 8 Jun 2011 17:22:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76867 invoked by uid 500); 8 Jun 2011 17:22:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76859 invoked by uid 99); 8 Jun 2011 17:22:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 17:22:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 17:22:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C1BC61085DF
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 17:21:58 +0000 (UTC)
Date: Wed, 8 Jun 2011 17:21:58 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <700052073.3870.1307553718790.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <34642033.3863.1307553598815.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7367) getgrouplist() in getGroup.c is
 not portable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046090#comment-13046090 ] 

Allen Wittenauer commented on HADOOP-7367:
------------------------------------------

A few points:

* The inclusion of this routine is preventing the native code from being compiled on platforms where it previously worked.

* We have multiple ways to try to solve this:
a) Rewrite the entire routine to remove the dependency on getgrouplist()
b) Do auto-detection to use the built-in getgrouplist, otherwise use one we supply
c) Use the current version when getgrouplist is present, otherwise use a completely different code stream that uses getgrent() etc.

> getgrouplist() in getGroup.c is not portable
> --------------------------------------------
>
>                 Key: HADOOP-7367
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7367
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.22.0, 0.23.0
>         Environment: System V
>            Reporter: Allen Wittenauer
>
> getGroupIDList uses getgrouplist() to fetch the groups for a user.  Unfortunately, this routine is a BSD-specific call and is not present in most System V-based operating systems. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16007-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 17:24:20 2011
Return-Path: <common-issues-return-16007-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4ABEC4705
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 17:24:20 +0000 (UTC)
Received: (qmail 80238 invoked by uid 500); 8 Jun 2011 17:24:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80211 invoked by uid 500); 8 Jun 2011 17:24:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80202 invoked by uid 99); 8 Jun 2011 17:24:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 17:24:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 17:24:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 22D4B1086F8
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 17:23:59 +0000 (UTC)
Date: Wed, 8 Jun 2011 17:23:59 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <580905232.3875.1307553839139.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1992675736.52742.1306733207390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7339) Introduce a buffered checksum for
 avoiding frequently calls on Checksum.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046092#comment-13046092 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7339:
------------------------------------------------

Hi Min, your results look good.  Could you also modify {{org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest}} and run it to illustrate the performance improvement?

> Introduce a buffered checksum for avoiding frequently calls on Checksum.update()
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-7339
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7339
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Min Zhou
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7339-v1.diff, HADOOP-7339-v2.diff
>
>
> We found that PureJavaCRC32/CRC32.update() is the TOP 1 of the methods consuming CPU in a map side, and in reduce side, it cost a lots of CPU too.
> IFileOutputStream would frequently call Checksum.update() during writing a record. It's very common a MR key/value less than 512 bytes. Checksum.update() would be called every time writing a key/value. 
> Test case: terasort 100MB. 
> Checksum.update() calls has be reduced from 4030348 to 28069.  This method is not a hotspot anymore.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16008-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 17:28:22 2011
Return-Path: <common-issues-return-16008-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A99BC4943
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 17:28:22 +0000 (UTC)
Received: (qmail 88442 invoked by uid 500); 8 Jun 2011 17:28:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88368 invoked by uid 500); 8 Jun 2011 17:28:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88350 invoked by uid 99); 8 Jun 2011 17:28:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 17:28:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 17:28:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D63461088B2
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 17:27:58 +0000 (UTC)
Date: Wed, 8 Jun 2011 17:27:58 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <17167584.3898.1307554078874.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <34642033.3863.1307553598815.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7367) getgrouplist() in getGroup.c is
 not portable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046096#comment-13046096 ] 

Allen Wittenauer commented on HADOOP-7367:
------------------------------------------

It also appears this code doesn't work on OS X... which is... interesting.

> getgrouplist() in getGroup.c is not portable
> --------------------------------------------
>
>                 Key: HADOOP-7367
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7367
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.22.0, 0.23.0
>         Environment: System V
>            Reporter: Allen Wittenauer
>
> getGroupIDList uses getgrouplist() to fetch the groups for a user.  Unfortunately, this routine is a BSD-specific call and is not present in most System V-based operating systems. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16009-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 17:59:20 2011
Return-Path: <common-issues-return-16009-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0EA044D56
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 17:59:20 +0000 (UTC)
Received: (qmail 74898 invoked by uid 500); 8 Jun 2011 17:59:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74871 invoked by uid 500); 8 Jun 2011 17:59:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74863 invoked by uid 99); 8 Jun 2011 17:59:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 17:59:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 17:59:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D1B3D108491
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 17:58:58 +0000 (UTC)
Date: Wed, 8 Jun 2011 17:58:58 +0000 (UTC)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1631715552.3947.1307555938855.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1799999999.69.1307444279323.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7362) Create a Lock annotation to
 document locking (hierarchies) for methods
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046109#comment-13046109 ] 

Arun C Murthy commented on HADOOP-7362:
---------------------------------------

Yep, that is precisely what I just put into MR-279 for now. I think I'll change it to take String[] rather than Class[] ala GuardedBy.

> Create a Lock annotation to document locking (hierarchies) for methods
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-7362
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7362
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>
> It will be useful to have better developer docs via a Lock annotation to document locking (hierarchies) for methods.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16010-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 19:16:23 2011
Return-Path: <common-issues-return-16010-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0D9B54483
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 19:16:23 +0000 (UTC)
Received: (qmail 6458 invoked by uid 500); 8 Jun 2011 19:16:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6426 invoked by uid 500); 8 Jun 2011 19:16:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6418 invoked by uid 99); 8 Jun 2011 19:16:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 19:16:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 19:16:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4A6DB1091CE
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 19:15:59 +0000 (UTC)
Date: Wed, 8 Jun 2011 19:15:59 +0000 (UTC)
From: "Joep Rottinghuis (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <74308544.4114.1307560559301.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-5759) IllegalArgumentException when
 CombineFileInputFormat is used as job InputFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joep Rottinghuis updated HADOOP-5759:
-------------------------------------

    Affects Version/s: 0.20.205.0

The "patch-5759-ydist.txt" patch should be applied to branch-0.20-security and the 0.20.205 branch.


> IllegalArgumentException when CombineFileInputFormat is used as job InputFormat
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-5759
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5759
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.1, 0.20.205.0
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.20.2
>
>         Attachments: patch-5759-1.txt, patch-5759-2.txt, patch-5759-ydist.txt, patch-5759.txt
>
>
> As per my understanding, CombineFileInputFormat is creating splits with rackname as split location. 
> When I use CombineFileInputFormat as the InputFormat for job, job initialization fails with following exception :
> 2009-04-28 14:10:40,162 ERROR mapred.EagerTaskInitializationListener (EagerTaskInitializationListener.java:run(83)) - Job initialization failed:
> java.lang.IllegalArgumentException: Network location name contains /: /default-rack
>   at org.apache.hadoop.net.NodeBase.set(NodeBase.java:76)
>   at org.apache.hadoop.net.NodeBase.<init>(NodeBase.java:57)
>   at org.apache.hadoop.mapred.JobTracker.addHostToNodeMapping(JobTracker.java:2342)
>   at org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2336)
>   at org.apache.hadoop.mapred.JobInProgress.createCache(JobInProgress.java:344)
>   at org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:441)
>   at org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:81)
>   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
>   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>   at java.lang.Thread.run(Thread.java:619)
> When I changed CombineFileInputFormat to pass just rackname (without '/'), JT wrongly resolves  the node as /default-rack/<rack-name>.
> Solution is to pass hostnames holding the block(on the rack),  instead of rackname.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16011-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 19:46:22 2011
Return-Path: <common-issues-return-16011-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 276F04EA9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 19:46:22 +0000 (UTC)
Received: (qmail 55292 invoked by uid 500); 8 Jun 2011 19:46:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55269 invoked by uid 500); 8 Jun 2011 19:46:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55260 invoked by uid 99); 8 Jun 2011 19:46:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 19:46:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 19:46:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C57A8109E38
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 19:46:00 +0000 (UTC)
Date: Wed, 8 Jun 2011 19:46:00 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1042204471.4341.1307562360805.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1043064090.64591.1307057267396.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7356) RPM packages broke bin/hadoop script
 for hadoop 0.20.205
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7356:
----------------------------------

       Resolution: Fixed
    Fix Version/s: 0.20.204.0
           Status: Resolved  (was: Patch Available)

I committed this to 0.20.204.0. I took out the support for running out of non-deployed developer directories.

> RPM packages broke bin/hadoop script for hadoop 0.20.205
> --------------------------------------------------------
>
>                 Key: HADOOP-7356
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7356
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.204.0
>         Environment: Java 6, Redhat EL 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>             Fix For: 0.20.204.0, 0.23.0
>
>         Attachments: HADOOP-7356-1.patch, HADOOP-7356-trunk.patch, HADOOP-7356.patch
>
>
> hadoop-config.sh has been moved to libexec for binary package, but developers prefers to have hadoop-config.sh in bin.  Hadoo shell scripts should be modified to support both scenarios.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16012-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 20:49:22 2011
Return-Path: <common-issues-return-16012-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7AB464636
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 20:49:22 +0000 (UTC)
Received: (qmail 65851 invoked by uid 500); 8 Jun 2011 20:49:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65809 invoked by uid 500); 8 Jun 2011 20:49:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65611 invoked by uid 99); 8 Jun 2011 20:49:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 20:49:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 20:49:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DC659109B6B
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 20:48:58 +0000 (UTC)
Date: Wed, 8 Jun 2011 20:48:58 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <741721305.4621.1307566138899.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <34642033.3863.1307553598815.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7367) getgrouplist() in getGroup.c is
 not portable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046209#comment-13046209 ] 

Allen Wittenauer commented on HADOOP-7367:
------------------------------------------

>From the bit of playing I've done, getgrouplist on OS X doesn't update ngroups to have the total number of groups the user is in.  Since the first call uses 0, it returns 0, which then dumps out of the rest of the loop.

So a fix here needs to take that into consideration.  (It may be worthwhile to just use sysconf(_POSIX_NGROUPS_MAX) for the first call so that getgrouplist mostly works on OS X and others with different semantics than the Linux version).

> getgrouplist() in getGroup.c is not portable
> --------------------------------------------
>
>                 Key: HADOOP-7367
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7367
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.22.0, 0.23.0
>         Environment: System V
>            Reporter: Allen Wittenauer
>
> getGroupIDList uses getgrouplist() to fetch the groups for a user.  Unfortunately, this routine is a BSD-specific call and is not present in most System V-based operating systems. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16013-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 20:49:24 2011
Return-Path: <common-issues-return-16013-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6EA93465E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 20:49:24 +0000 (UTC)
Received: (qmail 66173 invoked by uid 500); 8 Jun 2011 20:49:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66058 invoked by uid 500); 8 Jun 2011 20:49:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66017 invoked by uid 99); 8 Jun 2011 20:49:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 20:49:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 20:49:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2F073109B71
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 20:48:59 +0000 (UTC)
Date: Wed, 8 Jun 2011 20:48:59 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1533357109.4627.1307566139189.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <34642033.3863.1307553598815.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7367) getgrouplist() in getGroup.c is not
 portable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Allen Wittenauer updated HADOOP-7367:
-------------------------------------

    Environment: 
System V (Solaris, HP-UX, AIX?)
Mac OS X

  was:System V


> getgrouplist() in getGroup.c is not portable
> --------------------------------------------
>
>                 Key: HADOOP-7367
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7367
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.22.0, 0.23.0
>         Environment: System V (Solaris, HP-UX, AIX?)
> Mac OS X
>            Reporter: Allen Wittenauer
>
> getGroupIDList uses getgrouplist() to fetch the groups for a user.  Unfortunately, this routine is a BSD-specific call and is not present in most System V-based operating systems. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16014-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 21:24:22 2011
Return-Path: <common-issues-return-16014-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 54BDF434E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 21:24:22 +0000 (UTC)
Received: (qmail 41314 invoked by uid 500); 8 Jun 2011 21:24:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41277 invoked by uid 500); 8 Jun 2011 21:24:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41267 invoked by uid 99); 8 Jun 2011 21:24:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 21:24:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 21:24:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DB9D710999B
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 21:23:58 +0000 (UTC)
Date: Wed, 8 Jun 2011 21:23:58 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1269276399.4788.1307568238896.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7368) Mechanism for providing version info
 of hadoop jars
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Mechanism for providing version info of hadoop jars
---------------------------------------------------

                 Key: HADOOP-7368
                 URL: https://issues.apache.org/jira/browse/HADOOP-7368
             Project: Hadoop Common
          Issue Type: Improvement
          Components: build
    Affects Versions: 0.23.0
            Reporter: Luke Lu
            Assignee: Luke Lu
             Fix For: 0.23.0


In 0.20.x, only one jar (hadoop-core-*.jar) really matters. The o.a.h.util.VersionInfo combined with saveVersion.sh script (generating a package level annotation) served us well. For 0.23+, due to the project split and modularization of mapreduce (currently in MR-279), a lot more essential hadoop jars are created. The potential of mixing up the jars is significantly increased as well. We need a simple way to list the version info (version, branch, source checksum etc.) for all the jars involved. This is essential for QE and tracking down various issues.

I propose that we use a VersionProvider interface (similar to the current VersionInfo util class) and ServiceLoader to enumerate the version providers in the jars.
{code}
public interface VersionProvider {
  String getJar(); 
  String getPackage();
  String getVersion();
  String getRevision();
  String getBranch();
  String getDate();
  String getUrl();
  String getSourceChecksum();
}
{code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16015-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 21:30:22 2011
Return-Path: <common-issues-return-16015-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1D8194792
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 21:30:22 +0000 (UTC)
Received: (qmail 51382 invoked by uid 500); 8 Jun 2011 21:30:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51261 invoked by uid 500); 8 Jun 2011 21:30:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51250 invoked by uid 99); 8 Jun 2011 21:30:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 21:30:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 21:30:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C7EE3109B6B
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 21:29:58 +0000 (UTC)
Date: Wed, 8 Jun 2011 21:29:58 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <308395773.4851.1307568598815.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1269276399.4788.1307568238896.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7368) Mechanism for providing version
 info of hadoop jars
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046238#comment-13046238 ] 

Todd Lipcon commented on HADOOP-7368:
-------------------------------------

Nice idea. This will also be handy for things like codec plugins too!

> Mechanism for providing version info of hadoop jars
> ---------------------------------------------------
>
>                 Key: HADOOP-7368
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7368
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.23.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>
> In 0.20.x, only one jar (hadoop-core-*.jar) really matters. The o.a.h.util.VersionInfo combined with saveVersion.sh script (generating a package level annotation) served us well. For 0.23+, due to the project split and modularization of mapreduce (currently in MR-279), a lot more essential hadoop jars are created. The potential of mixing up the jars is significantly increased as well. We need a simple way to list the version info (version, branch, source checksum etc.) for all the jars involved. This is essential for QE and tracking down various issues.
> I propose that we use a VersionProvider interface (similar to the current VersionInfo util class) and ServiceLoader to enumerate the version providers in the jars.
> {code}
> public interface VersionProvider {
>   String getJar(); 
>   String getPackage();
>   String getVersion();
>   String getRevision();
>   String getBranch();
>   String getDate();
>   String getUrl();
>   String getSourceChecksum();
> }
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16016-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 21:36:21 2011
Return-Path: <common-issues-return-16016-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8F4B34DFD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 21:36:21 +0000 (UTC)
Received: (qmail 66565 invoked by uid 500); 8 Jun 2011 21:36:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66518 invoked by uid 500); 8 Jun 2011 21:36:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66450 invoked by uid 99); 8 Jun 2011 21:36:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 21:36:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 21:36:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0503B109B1A
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 21:35:59 +0000 (UTC)
Date: Wed, 8 Jun 2011 21:35:59 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1849782588.4878.1307568959017.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1269276399.4788.1307568238896.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7368) Mechanism for providing version
 info of hadoop jars
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046244#comment-13046244 ] 

Alejandro Abdelnur commented on HADOOP-7368:
--------------------------------------------

In addition, would be possible to store this info in a properties file in the JAR instead calling a script to generate a class?

Then, VersionProvider could be a class that reads a /version.properties' file to get all the values.

> Mechanism for providing version info of hadoop jars
> ---------------------------------------------------
>
>                 Key: HADOOP-7368
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7368
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.23.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>
> In 0.20.x, only one jar (hadoop-core-*.jar) really matters. The o.a.h.util.VersionInfo combined with saveVersion.sh script (generating a package level annotation) served us well. For 0.23+, due to the project split and modularization of mapreduce (currently in MR-279), a lot more essential hadoop jars are created. The potential of mixing up the jars is significantly increased as well. We need a simple way to list the version info (version, branch, source checksum etc.) for all the jars involved. This is essential for QE and tracking down various issues.
> I propose that we use a VersionProvider interface (similar to the current VersionInfo util class) and ServiceLoader to enumerate the version providers in the jars.
> {code}
> public interface VersionProvider {
>   String getJar(); 
>   String getPackage();
>   String getVersion();
>   String getRevision();
>   String getBranch();
>   String getDate();
>   String getUrl();
>   String getSourceChecksum();
> }
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16017-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 21:46:20 2011
Return-Path: <common-issues-return-16017-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 29F3D4A2B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 21:46:20 +0000 (UTC)
Received: (qmail 79387 invoked by uid 500); 8 Jun 2011 21:46:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79352 invoked by uid 500); 8 Jun 2011 21:46:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79341 invoked by uid 99); 8 Jun 2011 21:46:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 21:46:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 21:46:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E5C7B109F1A
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 21:45:58 +0000 (UTC)
Date: Wed, 8 Jun 2011 21:45:58 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <182555283.4891.1307569558937.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7369) tarball has incorrect permissions
 for sbin and libexec
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

tarball has incorrect permissions for sbin and libexec
------------------------------------------------------

                 Key: HADOOP-7369
                 URL: https://issues.apache.org/jira/browse/HADOOP-7369
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Owen O'Malley
         Attachments: libexec.patch

The tarball currently doesn't have execute set on libexec/* or sbin/*.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16018-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 21:46:22 2011
Return-Path: <common-issues-return-16018-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4092C4A8E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 21:46:22 +0000 (UTC)
Received: (qmail 79819 invoked by uid 500); 8 Jun 2011 21:46:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79778 invoked by uid 500); 8 Jun 2011 21:46:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79480 invoked by uid 99); 8 Jun 2011 21:46:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 21:46:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 21:46:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0ED67109F1E
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 21:45:59 +0000 (UTC)
Date: Wed, 8 Jun 2011 21:45:59 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1353320576.4894.1307569559057.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <182555283.4891.1307569558937.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7369) tarball has incorrect permissions
 for sbin and libexec
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley reassigned HADOOP-7369:
-------------------------------------

    Assignee: Owen O'Malley

> tarball has incorrect permissions for sbin and libexec
> ------------------------------------------------------
>
>                 Key: HADOOP-7369
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7369
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: libexec.patch
>
>
> The tarball currently doesn't have execute set on libexec/* or sbin/*.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16019-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 21:46:22 2011
Return-Path: <common-issues-return-16019-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 43B434A9E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 21:46:22 +0000 (UTC)
Received: (qmail 79895 invoked by uid 500); 8 Jun 2011 21:46:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79843 invoked by uid 500); 8 Jun 2011 21:46:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79593 invoked by uid 99); 8 Jun 2011 21:46:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 21:46:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 21:46:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1D4A9109F20
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 21:45:59 +0000 (UTC)
Date: Wed, 8 Jun 2011 21:45:59 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1850374885.4896.1307569559116.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <182555283.4891.1307569558937.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7369) tarball has incorrect permissions
 for sbin and libexec
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7369:
----------------------------------

    Attachment: libexec.patch

This sets the permissions correctly.

> tarball has incorrect permissions for sbin and libexec
> ------------------------------------------------------
>
>                 Key: HADOOP-7369
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7369
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: libexec.patch
>
>
> The tarball currently doesn't have execute set on libexec/* or sbin/*.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16020-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 21:46:22 2011
Return-Path: <common-issues-return-16020-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EE9544ACD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 21:46:22 +0000 (UTC)
Received: (qmail 81652 invoked by uid 500); 8 Jun 2011 21:46:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81620 invoked by uid 500); 8 Jun 2011 21:46:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81600 invoked by uid 99); 8 Jun 2011 21:46:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 21:46:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 21:46:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 584A1109F2A
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 21:45:59 +0000 (UTC)
Date: Wed, 8 Jun 2011 21:45:59 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2096969188.4903.1307569559358.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <182555283.4891.1307569558937.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7369) tarball has incorrect permissions
 for sbin and libexec
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7369:
----------------------------------

    Status: Patch Available  (was: Open)

> tarball has incorrect permissions for sbin and libexec
> ------------------------------------------------------
>
>                 Key: HADOOP-7369
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7369
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: libexec.patch
>
>
> The tarball currently doesn't have execute set on libexec/* or sbin/*.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16021-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 21:46:40 2011
Return-Path: <common-issues-return-16021-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C79854D22
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 21:46:40 +0000 (UTC)
Received: (qmail 82003 invoked by uid 500); 8 Jun 2011 21:46:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81976 invoked by uid 500); 8 Jun 2011 21:46:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81968 invoked by uid 99); 8 Jun 2011 21:46:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 21:46:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 21:46:39 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 64E5C109F2C
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 21:45:59 +0000 (UTC)
Date: Wed, 8 Jun 2011 21:45:59 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1704868676.4905.1307569559410.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <182555283.4891.1307569558937.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7369) tarball has incorrect permissions
 for sbin and libexec
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7369:
----------------------------------

    Status: Open  (was: Patch Available)

> tarball has incorrect permissions for sbin and libexec
> ------------------------------------------------------
>
>                 Key: HADOOP-7369
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7369
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: libexec.patch
>
>
> The tarball currently doesn't have execute set on libexec/* or sbin/*.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16022-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 21:48:21 2011
Return-Path: <common-issues-return-16022-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F078E4799
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 21:48:21 +0000 (UTC)
Received: (qmail 82574 invoked by uid 500); 8 Jun 2011 21:48:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82544 invoked by uid 500); 8 Jun 2011 21:48:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82536 invoked by uid 99); 8 Jun 2011 21:48:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 21:48:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 21:48:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C75FB10901D
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 21:47:58 +0000 (UTC)
Date: Wed, 8 Jun 2011 21:47:58 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1975607691.4907.1307569678813.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <182555283.4891.1307569558937.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7369) tarball has incorrect permissions
 for sbin and libexec
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7369:
----------------------------------

    Attachment:     (was: libexec.patch)

> tarball has incorrect permissions for sbin and libexec
> ------------------------------------------------------
>
>                 Key: HADOOP-7369
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7369
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>
> The tarball currently doesn't have execute set on libexec/* or sbin/*.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16023-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 21:54:22 2011
Return-Path: <common-issues-return-16023-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2CF964D5C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 21:54:22 +0000 (UTC)
Received: (qmail 94464 invoked by uid 500); 8 Jun 2011 21:54:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94437 invoked by uid 500); 8 Jun 2011 21:54:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94427 invoked by uid 99); 8 Jun 2011 21:54:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 21:54:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 21:54:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D315A1092AD
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 21:53:58 +0000 (UTC)
Date: Wed, 8 Jun 2011 21:53:58 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <954038211.4929.1307570038861.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <182555283.4891.1307569558937.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7369) tarball has incorrect permissions
 for sbin and libexec
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7369:
----------------------------------

    Attachment: libexec.patch

just the right fix

> tarball has incorrect permissions for sbin and libexec
> ------------------------------------------------------
>
>                 Key: HADOOP-7369
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7369
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: libexec.patch
>
>
> The tarball currently doesn't have execute set on libexec/* or sbin/*.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16024-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 22:04:22 2011
Return-Path: <common-issues-return-16024-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8BF24452F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 22:04:22 +0000 (UTC)
Received: (qmail 4925 invoked by uid 500); 8 Jun 2011 22:04:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4900 invoked by uid 500); 8 Jun 2011 22:04:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4889 invoked by uid 99); 8 Jun 2011 22:04:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 22:04:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 22:04:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3718F109673
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 22:03:59 +0000 (UTC)
Date: Wed, 8 Jun 2011 22:03:59 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <375501769.4957.1307570639222.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1269276399.4788.1307568238896.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7368) Mechanism for providing version
 info of hadoop jars
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046255#comment-13046255 ] 

Luke Lu commented on HADOOP-7368:
---------------------------------

bq. In addition, would be possible to store this info in a properties file in the JAR instead calling a script to generate a class?

Of course, the implementation of the VersionProvider interface is module dependent. The current saveVersion.sh script generates a package level annotation, which is actually simpler and less error prone then the reading version.properties.

> Mechanism for providing version info of hadoop jars
> ---------------------------------------------------
>
>                 Key: HADOOP-7368
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7368
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.23.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>
> In 0.20.x, only one jar (hadoop-core-*.jar) really matters. The o.a.h.util.VersionInfo combined with saveVersion.sh script (generating a package level annotation) served us well. For 0.23+, due to the project split and modularization of mapreduce (currently in MR-279), a lot more essential hadoop jars are created. The potential of mixing up the jars is significantly increased as well. We need a simple way to list the version info (version, branch, source checksum etc.) for all the jars involved. This is essential for QE and tracking down various issues.
> I propose that we use a VersionProvider interface (similar to the current VersionInfo util class) and ServiceLoader to enumerate the version providers in the jars.
> {code}
> public interface VersionProvider {
>   String getJar(); 
>   String getPackage();
>   String getVersion();
>   String getRevision();
>   String getBranch();
>   String getDate();
>   String getUrl();
>   String getSourceChecksum();
> }
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16025-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 22:06:22 2011
Return-Path: <common-issues-return-16025-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1AAF045B5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 22:06:22 +0000 (UTC)
Received: (qmail 6716 invoked by uid 500); 8 Jun 2011 22:06:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6648 invoked by uid 500); 8 Jun 2011 22:06:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6640 invoked by uid 99); 8 Jun 2011 22:06:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 22:06:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 22:06:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E52A9109731
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 22:05:58 +0000 (UTC)
Date: Wed, 8 Jun 2011 22:05:58 +0000 (UTC)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1273315475.4966.1307570758935.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <182555283.4891.1307569558937.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7369) tarball has incorrect permissions
 for sbin and libexec
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046256#comment-13046256 ] 

Giridharan Kesavan commented on HADOOP-7369:
--------------------------------------------

+1 looks good 

> tarball has incorrect permissions for sbin and libexec
> ------------------------------------------------------
>
>                 Key: HADOOP-7369
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7369
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: libexec.patch
>
>
> The tarball currently doesn't have execute set on libexec/* or sbin/*.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16026-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 22:08:22 2011
Return-Path: <common-issues-return-16026-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 40E7045F0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 22:08:22 +0000 (UTC)
Received: (qmail 7733 invoked by uid 500); 8 Jun 2011 22:08:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7698 invoked by uid 500); 8 Jun 2011 22:08:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7690 invoked by uid 99); 8 Jun 2011 22:08:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 22:08:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 22:08:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D90871097EA
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 22:07:58 +0000 (UTC)
Date: Wed, 8 Jun 2011 22:07:58 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1659391110.4972.1307570878885.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1269276399.4788.1307568238896.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7368) Mechanism for providing version
 info of hadoop jars
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046257#comment-13046257 ] 

Luke Lu commented on HADOOP-7368:
---------------------------------

bq. less error prone then the reading version.properties.

ugh, s/then the/than/

> Mechanism for providing version info of hadoop jars
> ---------------------------------------------------
>
>                 Key: HADOOP-7368
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7368
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.23.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>
> In 0.20.x, only one jar (hadoop-core-*.jar) really matters. The o.a.h.util.VersionInfo combined with saveVersion.sh script (generating a package level annotation) served us well. For 0.23+, due to the project split and modularization of mapreduce (currently in MR-279), a lot more essential hadoop jars are created. The potential of mixing up the jars is significantly increased as well. We need a simple way to list the version info (version, branch, source checksum etc.) for all the jars involved. This is essential for QE and tracking down various issues.
> I propose that we use a VersionProvider interface (similar to the current VersionInfo util class) and ServiceLoader to enumerate the version providers in the jars.
> {code}
> public interface VersionProvider {
>   String getJar(); 
>   String getPackage();
>   String getVersion();
>   String getRevision();
>   String getBranch();
>   String getDate();
>   String getUrl();
>   String getSourceChecksum();
> }
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16027-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun  8 22:40:20 2011
Return-Path: <common-issues-return-16027-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 043024A03
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  8 Jun 2011 22:40:20 +0000 (UTC)
Received: (qmail 34909 invoked by uid 500); 8 Jun 2011 22:40:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34877 invoked by uid 500); 8 Jun 2011 22:40:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34869 invoked by uid 99); 8 Jun 2011 22:40:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 22:40:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 08 Jun 2011 22:40:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D09CD109013
	for <common-issues@hadoop.apache.org>; Wed,  8 Jun 2011 22:39:58 +0000 (UTC)
Date: Wed, 8 Jun 2011 22:39:58 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1503522857.5067.1307572798850.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <182555283.4891.1307569558937.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7369) tarball has incorrect permissions
 for sbin and libexec
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046264#comment-13046264 ] 

Eric Yang commented on HADOOP-7369:
-----------------------------------

+1 looks good.  Thanks Owen.

> tarball has incorrect permissions for sbin and libexec
> ------------------------------------------------------
>
>                 Key: HADOOP-7369
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7369
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: libexec.patch
>
>
> The tarball currently doesn't have execute set on libexec/* or sbin/*.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16028-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 00:57:23 2011
Return-Path: <common-issues-return-16028-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0D9224CFE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 00:57:23 +0000 (UTC)
Received: (qmail 65775 invoked by uid 500); 9 Jun 2011 00:57:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65749 invoked by uid 500); 9 Jun 2011 00:57:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65656 invoked by uid 99); 9 Jun 2011 00:57:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 00:57:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 00:57:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BABA5109226
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 00:56:59 +0000 (UTC)
Date: Thu, 9 Jun 2011 00:56:59 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <522851026.5293.1307581019761.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <34642033.3863.1307553598815.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7367) getgrouplist() in getGroup.c is not
 portable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Allen Wittenauer updated HADOOP-7367:
-------------------------------------

    Attachment: hadoop-7367.patch

> getgrouplist() in getGroup.c is not portable
> --------------------------------------------
>
>                 Key: HADOOP-7367
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7367
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.22.0, 0.23.0
>         Environment: System V (Solaris, HP-UX, AIX?)
> Mac OS X
>            Reporter: Allen Wittenauer
>         Attachments: hadoop-7367.patch
>
>
> getGroupIDList uses getgrouplist() to fetch the groups for a user.  Unfortunately, this routine is a BSD-specific call and is not present in most System V-based operating systems. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16029-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 01:01:25 2011
Return-Path: <common-issues-return-16029-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6CF3E4554
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 01:01:25 +0000 (UTC)
Received: (qmail 71025 invoked by uid 500); 9 Jun 2011 01:01:25 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70987 invoked by uid 500); 9 Jun 2011 01:01:25 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70979 invoked by uid 99); 9 Jun 2011 01:01:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 01:01:25 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 01:01:24 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 03B1210932A
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 01:01:04 +0000 (UTC)
Date: Thu, 9 Jun 2011 01:01:04 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <188169949.5299.1307581264011.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <34642033.3863.1307553598815.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7367) getgrouplist() in getGroup.c is not
 portable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Allen Wittenauer updated HADOOP-7367:
-------------------------------------

    Status: Patch Available  (was: Open)

Several things about this patch:

* A check has been added to configure to look for a getgrouplist routine
* If it is found, we use a getgrouplist routine that tries to do it all in one pass, based upon allocating POSIX_NGROUPS_MAX groups (up to 32) for the initial buffer.  If the user has more than 32, we depend upon getgrouplist properly returning the number of groups for the user.  (So this will only match the first 32 on Darwin)
* If getgrouplist isn't there, we run through the entire groups db, looking for entries.

Please note that at least Solaris requires _POSIX_PTHREAD_SEMANTICS defined for the 5 param version of get*_r.  I'll cover that in a separate patch.

> getgrouplist() in getGroup.c is not portable
> --------------------------------------------
>
>                 Key: HADOOP-7367
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7367
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.22.0, 0.23.0
>         Environment: System V (Solaris, HP-UX, AIX?)
> Mac OS X
>            Reporter: Allen Wittenauer
>         Attachments: hadoop-7367.patch
>
>
> getGroupIDList uses getgrouplist() to fetch the groups for a user.  Unfortunately, this routine is a BSD-specific call and is not present in most System V-based operating systems. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16030-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 01:07:22 2011
Return-Path: <common-issues-return-16030-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3F6244546
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 01:07:22 +0000 (UTC)
Received: (qmail 75092 invoked by uid 500); 9 Jun 2011 01:07:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75060 invoked by uid 500); 9 Jun 2011 01:07:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75052 invoked by uid 99); 9 Jun 2011 01:07:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 01:07:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 01:07:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E89D41094A1
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 01:06:58 +0000 (UTC)
Date: Thu, 9 Jun 2011 01:06:58 +0000 (UTC)
From: "Min Zhou (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <204142867.5331.1307581618949.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1992675736.52742.1306733207390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7339) Introduce a buffered checksum for
 avoiding frequently calls on Checksum.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046296#comment-13046296 ] 

Min Zhou commented on HADOOP-7339:
----------------------------------

Hi Nicholas,

Thank you for your feedback. Since right now BufferedChecksum is a non-public inner class, it should be taken out from DataChecksum as a public class like PureJavaCrc32. 

Could I change the testcase name from org.apache.hadoop.util.TestPureJavaCrc32 to org.apache.hadoop.util.TestChecksum? Cause it would become a testcase on 4 checksum implementations(or their combinations), jdk builtin CRC32, PureJavaCrc32, Buffered jdk builtin CRC32, Buffered PureJavaCrc32. 

> Introduce a buffered checksum for avoiding frequently calls on Checksum.update()
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-7339
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7339
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Min Zhou
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7339-v1.diff, HADOOP-7339-v2.diff
>
>
> We found that PureJavaCRC32/CRC32.update() is the TOP 1 of the methods consuming CPU in a map side, and in reduce side, it cost a lots of CPU too.
> IFileOutputStream would frequently call Checksum.update() during writing a record. It's very common a MR key/value less than 512 bytes. Checksum.update() would be called every time writing a key/value. 
> Test case: terasort 100MB. 
> Checksum.update() calls has be reduced from 4030348 to 28069.  This method is not a hotspot anymore.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16031-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 01:17:20 2011
Return-Path: <common-issues-return-16031-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 12B8D4E79
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 01:17:20 +0000 (UTC)
Received: (qmail 79463 invoked by uid 500); 9 Jun 2011 01:17:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79427 invoked by uid 500); 9 Jun 2011 01:17:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79409 invoked by uid 99); 9 Jun 2011 01:17:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 01:17:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 01:17:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C33F710970B
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 01:16:58 +0000 (UTC)
Date: Thu, 9 Jun 2011 01:16:58 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <389881229.5342.1307582218796.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1992675736.52742.1306733207390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7339) Introduce a buffered checksum for
 avoiding frequently calls on Checksum.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7339:
-------------------------------------------

    Assignee: Min Zhou

Let's focus on the code and minimize changes here so that it is easier to be reviewed.  We may work on renaming/refactoring on a separated issue later.

> Introduce a buffered checksum for avoiding frequently calls on Checksum.update()
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-7339
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7339
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Min Zhou
>            Assignee: Min Zhou
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7339-v1.diff, HADOOP-7339-v2.diff
>
>
> We found that PureJavaCRC32/CRC32.update() is the TOP 1 of the methods consuming CPU in a map side, and in reduce side, it cost a lots of CPU too.
> IFileOutputStream would frequently call Checksum.update() during writing a record. It's very common a MR key/value less than 512 bytes. Checksum.update() would be called every time writing a key/value. 
> Test case: terasort 100MB. 
> Checksum.update() calls has be reduced from 4030348 to 28069.  This method is not a hotspot anymore.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16032-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 01:55:20 2011
Return-Path: <common-issues-return-16032-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 79F54413A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 01:55:20 +0000 (UTC)
Received: (qmail 3528 invoked by uid 500); 9 Jun 2011 01:55:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3491 invoked by uid 500); 9 Jun 2011 01:55:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3483 invoked by uid 99); 9 Jun 2011 01:55:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 01:55:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 01:55:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CEAF1109F02
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 01:54:58 +0000 (UTC)
Date: Thu, 9 Jun 2011 01:54:58 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1286432479.5378.1307584498843.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <34642033.3863.1307553598815.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7367) getgrouplist() in getGroup.c is
 not portable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046303#comment-13046303 ] 

Hadoop QA commented on HADOOP-7367:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481884/hadoop-7367.patch
  against trunk revision 1133125.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/596//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/596//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/596//console

This message is automatically generated.

> getgrouplist() in getGroup.c is not portable
> --------------------------------------------
>
>                 Key: HADOOP-7367
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7367
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.22.0, 0.23.0
>         Environment: System V (Solaris, HP-UX, AIX?)
> Mac OS X
>            Reporter: Allen Wittenauer
>         Attachments: hadoop-7367.patch
>
>
> getGroupIDList uses getgrouplist() to fetch the groups for a user.  Unfortunately, this routine is a BSD-specific call and is not present in most System V-based operating systems. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16033-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 07:54:21 2011
Return-Path: <common-issues-return-16033-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2383E665B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 07:54:21 +0000 (UTC)
Received: (qmail 13661 invoked by uid 500); 9 Jun 2011 07:54:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13628 invoked by uid 500); 9 Jun 2011 07:54:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13605 invoked by uid 99); 9 Jun 2011 07:54:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 07:54:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 07:54:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CA49D10A87B
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 07:53:58 +0000 (UTC)
Date: Thu, 9 Jun 2011 07:53:58 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1204582022.5878.1307606038825.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7370) Optimize pread on ChecksumFileSystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Optimize pread on ChecksumFileSystem
------------------------------------

                 Key: HADOOP-7370
                 URL: https://issues.apache.org/jira/browse/HADOOP-7370
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
            Priority: Minor
         Attachments: checksumfs-pread-0.20.txt

Currently the implementation of positional read in ChecksumFileSystem is verify inefficient - it actually re-opens the underlying file and checksum file, then seeks and uses normal read. Instead, it can push down positional read directly to the underlying FS and verify checksum.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16034-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 07:54:22 2011
Return-Path: <common-issues-return-16034-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DB8D7666E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 07:54:22 +0000 (UTC)
Received: (qmail 13820 invoked by uid 500); 9 Jun 2011 07:54:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13632 invoked by uid 500); 9 Jun 2011 07:54:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13616 invoked by uid 99); 9 Jun 2011 07:54:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 07:54:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 07:54:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DF4E610A87E
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 07:53:58 +0000 (UTC)
Date: Thu, 9 Jun 2011 07:53:58 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1094423604.5881.1307606038911.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1204582022.5878.1307606038825.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7370) Optimize pread on ChecksumFileSystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7370:
--------------------------------

    Attachment: checksumfs-pread-0.20.txt

I was hacking on this against 0.20. It seems to have made a slight difference but I didn't do any real benchmarking. This would have to be updated to trunk and then benchmarked before commit.

> Optimize pread on ChecksumFileSystem
> ------------------------------------
>
>                 Key: HADOOP-7370
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7370
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: checksumfs-pread-0.20.txt
>
>
> Currently the implementation of positional read in ChecksumFileSystem is verify inefficient - it actually re-opens the underlying file and checksum file, then seeks and uses normal read. Instead, it can push down positional read directly to the underlying FS and verify checksum.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16035-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 07:56:22 2011
Return-Path: <common-issues-return-16035-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EE8AF672B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 07:56:21 +0000 (UTC)
Received: (qmail 18743 invoked by uid 500); 9 Jun 2011 07:56:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18716 invoked by uid 500); 9 Jun 2011 07:56:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18708 invoked by uid 99); 9 Jun 2011 07:56:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 07:56:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 07:56:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C3D4D10A93D
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 07:55:58 +0000 (UTC)
Date: Thu, 9 Jun 2011 07:55:58 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <354858314.5883.1307606158798.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1204582022.5878.1307606038825.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7370) Optimize pread on
 ChecksumFileSystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046378#comment-13046378 ] 

Todd Lipcon commented on HADOOP-7370:
-------------------------------------

ah, also ignore the diff hunk related to readChunk -- part of HADOOP-3205 slipped in here

> Optimize pread on ChecksumFileSystem
> ------------------------------------
>
>                 Key: HADOOP-7370
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7370
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: checksumfs-pread-0.20.txt
>
>
> Currently the implementation of positional read in ChecksumFileSystem is verify inefficient - it actually re-opens the underlying file and checksum file, then seeks and uses normal read. Instead, it can push down positional read directly to the underlying FS and verify checksum.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16036-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 09:32:20 2011
Return-Path: <common-issues-return-16036-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 513096EAF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 09:32:20 +0000 (UTC)
Received: (qmail 46693 invoked by uid 500); 9 Jun 2011 09:32:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46654 invoked by uid 500); 9 Jun 2011 09:32:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46645 invoked by uid 99); 9 Jun 2011 09:32:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 09:32:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 09:32:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 04EFF10A2E0
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 09:31:59 +0000 (UTC)
Date: Thu, 9 Jun 2011 09:31:59 +0000 (UTC)
From: "Nicholas Telford (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1683549881.6270.1307611919016.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7269) S3 Native should allow customizable
 file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicholas Telford updated HADOOP-7269:
-------------------------------------

    Attachment: 7269-combined.patch

I had a feeling this might be the case. Combined patch attached.

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: 0001-Added-support-for-metadata-to-be-applied-to-objects-.patch, 0002-Added-check-that-metadata-was-set-to-unit-test.patch, 7269-combined.patch, HADOOP-7269-S3-metadata-001.diff, HADOOP-7269-S3-metadata-002.diff, HADOOP-7269-S3-metadata-003.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16037-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 09:34:22 2011
Return-Path: <common-issues-return-16037-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4D0B36905
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 09:34:22 +0000 (UTC)
Received: (qmail 53408 invoked by uid 500); 9 Jun 2011 09:34:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53360 invoked by uid 500); 9 Jun 2011 09:34:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53344 invoked by uid 99); 9 Jun 2011 09:34:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 09:34:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 09:34:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EEDC910A3F8
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 09:33:58 +0000 (UTC)
Date: Thu, 9 Jun 2011 09:33:58 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2105698203.6280.1307612038975.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7269) S3 Native should allow
 customizable file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046426#comment-13046426 ] 

Hadoop QA commented on HADOOP-7269:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481901/7269-combined.patch
  against trunk revision 1133125.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 8 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/597//console

This message is automatically generated.

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: 0001-Added-support-for-metadata-to-be-applied-to-objects-.patch, 0002-Added-check-that-metadata-was-set-to-unit-test.patch, 7269-combined.patch, HADOOP-7269-S3-metadata-001.diff, HADOOP-7269-S3-metadata-002.diff, HADOOP-7269-S3-metadata-003.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16038-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 09:49:22 2011
Return-Path: <common-issues-return-16038-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B46B862C2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 09:49:22 +0000 (UTC)
Received: (qmail 77379 invoked by uid 500); 9 Jun 2011 09:49:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77267 invoked by uid 500); 9 Jun 2011 09:49:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77259 invoked by uid 99); 9 Jun 2011 09:49:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 09:49:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 09:49:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0E82B10A945
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 09:48:59 +0000 (UTC)
Date: Thu, 9 Jun 2011 09:48:59 +0000 (UTC)
From: "Nicholas Telford (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1836661052.6315.1307612939056.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7269) S3 Native should allow customizable
 file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicholas Telford updated HADOOP-7269:
-------------------------------------

    Attachment: 7269-combined-proper.patch

Totally stupid mistake there. I've switched to using Git for all my work on Apache projects and I'm still finding my feet with the workflow.

This should be sorted now, please ignore all previous patches.

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: 0001-Added-support-for-metadata-to-be-applied-to-objects-.patch, 0002-Added-check-that-metadata-was-set-to-unit-test.patch, 7269-combined-proper.patch, 7269-combined.patch, HADOOP-7269-S3-metadata-001.diff, HADOOP-7269-S3-metadata-002.diff, HADOOP-7269-S3-metadata-003.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16039-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 10:05:20 2011
Return-Path: <common-issues-return-16039-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A21896C92
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 10:05:20 +0000 (UTC)
Received: (qmail 12877 invoked by uid 500); 9 Jun 2011 10:05:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12842 invoked by uid 500); 9 Jun 2011 10:05:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12824 invoked by uid 99); 9 Jun 2011 10:05:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 10:05:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 10:05:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2EF9A10AFDF
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 10:04:59 +0000 (UTC)
Date: Thu, 9 Jun 2011 10:04:59 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <463165810.6352.1307613899189.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7269) S3 Native should allow
 customizable file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046444#comment-13046444 ] 

Hadoop QA commented on HADOOP-7269:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481903/7269-combined-proper.patch
  against trunk revision 1133125.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 8 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The applied patch generated 1069 javac compiler warnings (more than the trunk's current 1067 warnings).

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/598//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/598//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/598//console

This message is automatically generated.

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: 0001-Added-support-for-metadata-to-be-applied-to-objects-.patch, 0002-Added-check-that-metadata-was-set-to-unit-test.patch, 7269-combined-proper.patch, 7269-combined.patch, HADOOP-7269-S3-metadata-001.diff, HADOOP-7269-S3-metadata-002.diff, HADOOP-7269-S3-metadata-003.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16040-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 11:28:22 2011
Return-Path: <common-issues-return-16040-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 004B46B71
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 11:28:22 +0000 (UTC)
Received: (qmail 29714 invoked by uid 500); 9 Jun 2011 11:28:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29664 invoked by uid 500); 9 Jun 2011 11:28:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29535 invoked by uid 99); 9 Jun 2011 11:28:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 11:28:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 11:28:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 099DE10AD94
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 11:27:59 +0000 (UTC)
Date: Thu, 9 Jun 2011 11:27:59 +0000 (UTC)
From: "Mark Butler (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <597117606.6541.1307618879036.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-3394) Distributed Lucene Index For Hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-3394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Butler resolved HADOOP-3394.
---------------------------------

    Resolution: Not A Problem

Closing this issue, it is no longer relevant. 

> Distributed Lucene Index For Hadoop
> -----------------------------------
>
>                 Key: HADOOP-3394
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3394
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Mark Butler
>            Priority: Minor
>         Attachments: dlucene080518.tar.gz
>
>
> Here is the current prototype implementation of a distributed free text index using Hadoop based on Doug Cutting's design:
> http://www.mail-archive.com/general@lucene.apache.org/msg00338.html
> There has also been some discussion about this on the Hadoop Wiki:
> http://wiki.apache.org/hadoop/DistributedLucene
> This work is not finished, so it is not intended for inclusion yet. For a description of the contribution and its current status see the report in 
> doc/index.html 
> in the attached archive that gives some details of the implementation. 
> This work was designed as a contrib contribution. However, as there are at least two other projects (Bailey and Katta) with similar goals it seemed a good idea to make this code available for discussion.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16041-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 14:06:20 2011
Return-Path: <common-issues-return-16041-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7FB8D6C86
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 14:06:20 +0000 (UTC)
Received: (qmail 99296 invoked by uid 500); 9 Jun 2011 14:06:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99222 invoked by uid 500); 9 Jun 2011 14:06:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99206 invoked by uid 99); 9 Jun 2011 14:06:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 14:06:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 14:06:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1326F10A319
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 14:05:59 +0000 (UTC)
Date: Thu, 9 Jun 2011 14:05:59 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1847389995.6877.1307628359075.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1142137633.58455.1306902167529.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7348) Modify the option of FsShell
 getmerge from [addnl] to [-nl] for more comprehensive
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046550#comment-13046550 ] 

Daryn Sharp commented on HADOOP-7348:
-------------------------------------

Please revert {{new CommandFormat(null, 2, 3, "nl");}} to omit the first parameter (null).  The {{CommandFormat}} ctor that takes the unused parameter has been deprecated, so that's probably the -1 warning in the QA build.  I'd also recommend moving the -nl to the front of the usage to be consistent with other commands.  Ie. {{USAGE = "[-nl] <src> <localdst>"}}.

Other than that, looks good!  Just be sure to update the aforementioned tests in hdfs.  Please link that jira to this one to make it easier to track.

> Modify the option of FsShell getmerge from [addnl] to [-nl] for more comprehensive
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7348
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7348
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Assignee: XieXianshan
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7348.patch, HADOOP-7348.patch_2
>
>
> The [addnl] option of FsShell getmerge should be either "true" or "false",but it is very hard to understand by users, especially  who`s never used this option before. 
> So,the [addnl] option should be changed to [-nl] for more comprehensive.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16042-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 14:37:20 2011
Return-Path: <common-issues-return-16042-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C22C46142
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 14:37:20 +0000 (UTC)
Received: (qmail 76213 invoked by uid 500); 9 Jun 2011 14:37:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76174 invoked by uid 500); 9 Jun 2011 14:37:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76166 invoked by uid 99); 9 Jun 2011 14:37:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 14:37:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 14:37:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6F19B10A627
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 14:36:59 +0000 (UTC)
Date: Thu, 9 Jun 2011 14:36:59 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1358824149.7043.1307630219450.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1008788632.26385.1305807047638.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7305) Eclipse project files are
 incomplete
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046579#comment-13046579 ] 

Daryn Sharp commented on HADOOP-7305:
-------------------------------------

The patch doesn't work on OS X because the path is ${env.JAVA_HOME}/bundle/Classes/classes.jar (sorry if I mispoke earlier).  Again though, classes.jar is already in the jdk's search path so it isn't strictly necessary.

Another friend and I tested a fresh eclipse install on RHEL5 w/o tools.jar in the .classpath file.  All we did is import the project and define ANT_HOME.  It worked fine, so it calls into question the need for this patch.  Are you sure you don't have a misconfiged environment?  Are other people experiencing the problem?

> Eclipse project files are incomplete
> ------------------------------------
>
>                 Key: HADOOP-7305
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7305
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Niels Basjes
>            Assignee: Niels Basjes
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7305-2011-05-19.patch, HADOOP-7305-2011-05-30.patch
>
>
> After a fresh checkout of hadoop-common I do 'ant compile eclipse'.
> I open eclipse, set ANT_HOME and build the project. 
> At that point the following error appears:
> {quote}
> The type com.sun.javadoc.RootDoc cannot be resolved. It is indirectly referenced from required .class files	ExcludePrivateAnnotationsJDiffDoclet.java	/common/src/java/org/apache/hadoop/classification/tools	line 1	Java Problem
> {quote}
> The solution is to add the "tools.jar" from the JDK to the buildpath/classpath.
> This should be fixed in the build.xml.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16043-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 15:07:22 2011
Return-Path: <common-issues-return-16043-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6B17563B8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 15:07:22 +0000 (UTC)
Received: (qmail 39206 invoked by uid 500); 9 Jun 2011 15:07:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39173 invoked by uid 500); 9 Jun 2011 15:07:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39165 invoked by uid 99); 9 Jun 2011 15:07:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 15:07:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 15:07:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2A8DD10B595
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 15:06:59 +0000 (UTC)
Date: Thu, 9 Jun 2011 15:06:59 +0000 (UTC)
From: "Anonymous (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <662227690.7168.1307632019171.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1250812071.19292.1305645587504.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7297) Error in the documentation regarding
 Checkpoint/Backup Node
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Anonymous updated HADOOP-7297:
------------------------------

    Status: Patch Available  (was: Reopened)

> Error in the documentation regarding Checkpoint/Backup Node
> -----------------------------------------------------------
>
>                 Key: HADOOP-7297
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7297
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.20.203.0
>            Reporter: arnaud p
>            Priority: Trivial
>
> On http://hadoop.apache.org/common/docs/r0.20.203.0/hdfs_user_guide.html#Checkpoint+Node: the command bin/hdfs namenode -checkpoint required to launch the backup/checkpoint node does not exist.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16044-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 15:15:22 2011
Return-Path: <common-issues-return-16044-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9A8FB6793
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 15:15:22 +0000 (UTC)
Received: (qmail 57705 invoked by uid 500); 9 Jun 2011 15:15:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57671 invoked by uid 500); 9 Jun 2011 15:15:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57658 invoked by uid 99); 9 Jun 2011 15:15:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 15:15:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 15:15:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1170B10B8CB
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 15:14:59 +0000 (UTC)
Date: Thu, 9 Jun 2011 15:14:59 +0000 (UTC)
From: =?utf-8?Q?Christoph_B=C3=B6hm_=28JIRA=29?= <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <309134864.7188.1307632499067.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1250812071.19292.1305645587504.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7297) Error in the documentation regarding
 Checkpoint/Backup Node
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7297?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Christoph B=C3=B6hm updated HADOOP-7297:
-----------------------------------

    Description:=20
On http://hadoop.apache.org/common/docs/r0.20.203.0/hdfs_user_guide.html#Ch=
eckpoint+Node: the command bin/hdfs namenode -checkpoint required to launch=
 the backup/checkpoint node does not exist.
I have removed this from the docs.

  was:On http://hadoop.apache.org/common/docs/r0.20.203.0/hdfs_user_guide.h=
tml#Checkpoint+Node: the command bin/hdfs namenode -checkpoint required to =
launch the backup/checkpoint node does not exist.


> Error in the documentation regarding Checkpoint/Backup Node
> -----------------------------------------------------------
>
>                 Key: HADOOP-7297
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7297
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.20.203.0
>            Reporter: arnaud p
>            Priority: Trivial
>
> On http://hadoop.apache.org/common/docs/r0.20.203.0/hdfs_user_guide.html#=
Checkpoint+Node: the command bin/hdfs namenode -checkpoint required to laun=
ch the backup/checkpoint node does not exist.
> I have removed this from the docs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16045-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 15:17:24 2011
Return-Path: <common-issues-return-16045-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A001B6F51
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 15:17:22 +0000 (UTC)
Received: (qmail 65557 invoked by uid 500); 9 Jun 2011 15:17:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65499 invoked by uid 500); 9 Jun 2011 15:17:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65491 invoked by uid 99); 9 Jun 2011 15:17:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 15:17:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 15:17:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 547E210B9B5
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 15:17:01 +0000 (UTC)
Date: Thu, 9 Jun 2011 15:17:01 +0000 (UTC)
From: =?utf-8?Q?Christoph_B=C3=B6hm_=28JIRA=29?= <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <318405068.7205.1307632621342.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1250812071.19292.1305645587504.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7297) Error in the documentation regarding
 Checkpoint/Backup Node
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7297?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Christoph B=C3=B6hm updated HADOOP-7297:
-----------------------------------

    Attachment: hadoop-7297.patch

I have removed this from the docs.

> Error in the documentation regarding Checkpoint/Backup Node
> -----------------------------------------------------------
>
>                 Key: HADOOP-7297
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7297
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.20.203.0
>            Reporter: arnaud p
>            Priority: Trivial
>         Attachments: hadoop-7297.patch
>
>
> On http://hadoop.apache.org/common/docs/r0.20.203.0/hdfs_user_guide.html#=
Checkpoint+Node: the command bin/hdfs namenode -checkpoint required to laun=
ch the backup/checkpoint node does not exist.
> I have removed this from the docs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16046-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 15:25:20 2011
Return-Path: <common-issues-return-16046-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B14696E38
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 15:25:20 +0000 (UTC)
Received: (qmail 79579 invoked by uid 500); 9 Jun 2011 15:25:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79542 invoked by uid 500); 9 Jun 2011 15:25:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79534 invoked by uid 99); 9 Jun 2011 15:25:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 15:25:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 15:25:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 66A9A10BDD6
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 15:24:59 +0000 (UTC)
Date: Thu, 9 Jun 2011 15:24:59 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1678068314.7229.1307633099416.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1250812071.19292.1305645587504.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7297) Error in the documentation
 regarding Checkpoint/Backup Node
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046612#comment-13046612 ] 

Hadoop QA commented on HADOOP-7297:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481942/hadoop-7297.patch
  against trunk revision 1133125.

    +1 @author.  The patch does not contain any @author tags.

    +0 tests included.  The patch appears to be a documentation patch that doesn't require tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/600//console

This message is automatically generated.

> Error in the documentation regarding Checkpoint/Backup Node
> -----------------------------------------------------------
>
>                 Key: HADOOP-7297
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7297
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.20.203.0
>            Reporter: arnaud p
>            Priority: Trivial
>         Attachments: hadoop-7297.patch
>
>
> On http://hadoop.apache.org/common/docs/r0.20.203.0/hdfs_user_guide.html#Checkpoint+Node: the command bin/hdfs namenode -checkpoint required to launch the backup/checkpoint node does not exist.
> I have removed this from the docs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16047-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 15:25:20 2011
Return-Path: <common-issues-return-16047-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E45E96E50
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 15:25:20 +0000 (UTC)
Received: (qmail 79836 invoked by uid 500); 9 Jun 2011 15:25:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79785 invoked by uid 500); 9 Jun 2011 15:25:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79691 invoked by uid 99); 9 Jun 2011 15:25:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 15:25:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 15:25:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 74ECC10BDD8
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 15:24:59 +0000 (UTC)
Date: Thu, 9 Jun 2011 15:24:59 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <808940268.7231.1307633099475.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1894387424.4008.1305137809491.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7278) Automatic Hadoop cluster
 deployment for build validation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046613#comment-13046613 ] 

Hadoop QA commented on HADOOP-7278:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479155/HADOOP-7278.patch
  against trunk revision 1133125.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/599//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/599//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/599//console

This message is automatically generated.

> Automatic Hadoop cluster deployment for build validation
> --------------------------------------------------------
>
>                 Key: HADOOP-7278
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7278
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0, 0.23.0
>         Environment: Apache Jenkins
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>              Labels: newbie
>         Attachments: HADOOP-7278.patch
>
>
> It'd be great to have a way of automatically deploying Hadoop cluster to a set of machine once all components are successfully built (in the form or tar or whatever). Deployed cluster then can be used to run a set of validation jobs to make sure that produced artifacts are viable. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16048-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 15:31:20 2011
Return-Path: <common-issues-return-16048-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 744FF6732
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 15:31:20 +0000 (UTC)
Received: (qmail 89367 invoked by uid 500); 9 Jun 2011 15:31:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89341 invoked by uid 500); 9 Jun 2011 15:31:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89333 invoked by uid 99); 9 Jun 2011 15:31:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 15:31:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 15:31:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1317710B0BC
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 15:30:59 +0000 (UTC)
Date: Thu, 9 Jun 2011 15:30:59 +0000 (UTC)
From: "Frank Conrad (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1437268561.7248.1307633459074.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-1886) Undocumented parameters in
 FilesSystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Frank Conrad updated HADOOP-1886:
---------------------------------

    Attachment: hadoop-1886-1.diff

add java doc

> Undocumented parameters in FilesSystem
> --------------------------------------
>
>                 Key: HADOOP-1886
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1886
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.14.0
>            Reporter: Konstantin Shvachko
>            Priority: Trivial
>         Attachments: hadoop-1886-1.diff
>
>
> Multiple create methods in public FileSystem class lack documentation for the following 2 parameters.
> - long blockSize,
> - Progressable progress

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16049-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 15:31:20 2011
Return-Path: <common-issues-return-16049-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9861D6740
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 15:31:20 +0000 (UTC)
Received: (qmail 89627 invoked by uid 500); 9 Jun 2011 15:31:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89581 invoked by uid 500); 9 Jun 2011 15:31:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89573 invoked by uid 99); 9 Jun 2011 15:31:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 15:31:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 15:31:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 58A8F10B0BF
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 15:30:59 +0000 (UTC)
Date: Thu, 9 Jun 2011 15:30:59 +0000 (UTC)
From: "Frank Conrad (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1342964020.7251.1307633459359.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-1886) Undocumented parameters in
 FilesSystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Frank Conrad updated HADOOP-1886:
---------------------------------

    Status: Patch Available  (was: Open)

> Undocumented parameters in FilesSystem
> --------------------------------------
>
>                 Key: HADOOP-1886
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1886
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.14.0
>            Reporter: Konstantin Shvachko
>            Priority: Trivial
>         Attachments: hadoop-1886-1.diff
>
>
> Multiple create methods in public FileSystem class lack documentation for the following 2 parameters.
> - long blockSize,
> - Progressable progress

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16050-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 15:45:22 2011
Return-Path: <common-issues-return-16050-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 642456E05
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 15:45:22 +0000 (UTC)
Received: (qmail 26628 invoked by uid 500); 9 Jun 2011 15:45:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26579 invoked by uid 500); 9 Jun 2011 15:45:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26528 invoked by uid 99); 9 Jun 2011 15:45:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 15:45:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 15:45:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D378C10B61D
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 15:44:58 +0000 (UTC)
Date: Thu, 9 Jun 2011 15:44:58 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <235357240.7273.1307634298862.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-1886) Undocumented parameters in
 FilesSystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046624#comment-13046624 ] 

Hadoop QA commented on HADOOP-1886:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481945/hadoop-1886-1.diff
  against trunk revision 1133125.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 2 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/601//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/601//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/601//console

This message is automatically generated.

> Undocumented parameters in FilesSystem
> --------------------------------------
>
>                 Key: HADOOP-1886
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1886
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.14.0
>            Reporter: Konstantin Shvachko
>            Priority: Trivial
>         Attachments: hadoop-1886-1.diff
>
>
> Multiple create methods in public FileSystem class lack documentation for the following 2 parameters.
> - long blockSize,
> - Progressable progress

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16051-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 16:58:21 2011
Return-Path: <common-issues-return-16051-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E7B9C660E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 16:58:21 +0000 (UTC)
Received: (qmail 20563 invoked by uid 500); 9 Jun 2011 16:58:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20503 invoked by uid 500); 9 Jun 2011 16:58:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20341 invoked by uid 99); 9 Jun 2011 16:58:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 16:58:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 16:58:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4124C10B368
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 16:57:59 +0000 (UTC)
Date: Thu, 9 Jun 2011 16:57:59 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1415670632.7518.1307638679263.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7371) Improve tarball distributions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Improve tarball distributions
-----------------------------

                 Key: HADOOP-7371
                 URL: https://issues.apache.org/jira/browse/HADOOP-7371
             Project: Hadoop Common
          Issue Type: Improvement
          Components: build
         Environment: Java 6, Redhat 5.5
            Reporter: Eric Yang
            Assignee: Eric Yang


Hadoop release tarball contains both raw source and binary.  This leads users to use the release tarball as base for applying patches, to build custom Hadoop.  This is not the recommended method to develop hadoop because it leads to mixed development system where processed files and raw source are hard to separate.  

To correct the problematic usage of the release tarball, the release build target should be defined as:

"ant source" generates source release tarball.
"ant binary" is binary release without source/javadoc jar files.
"ant tar" is a mirror of binary release with source/javadoc jar files.

Does this sound reasonable?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16052-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 17:11:24 2011
Return-Path: <common-issues-return-16052-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6FEC1611F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 17:11:24 +0000 (UTC)
Received: (qmail 42232 invoked by uid 500); 9 Jun 2011 17:11:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42202 invoked by uid 500); 9 Jun 2011 17:11:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42194 invoked by uid 99); 9 Jun 2011 17:11:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 17:11:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 17:11:22 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 175C810B858
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 17:11:01 +0000 (UTC)
Date: Thu, 9 Jun 2011 17:11:01 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1920693641.7540.1307639461092.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1415670632.7518.1307638679263.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7371) Improve tarball distributions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046678#comment-13046678 ] 

Alejandro Abdelnur commented on HADOOP-7371:
--------------------------------------------

Yes, it makes senses.

As part of the Mavenization HADOOP-6671 this would be trivial.

Because of that, wouldn't make sense to wait till Mavenization is in place for this?


> Improve tarball distributions
> -----------------------------
>
>                 Key: HADOOP-7371
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7371
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>         Environment: Java 6, Redhat 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>
> Hadoop release tarball contains both raw source and binary.  This leads users to use the release tarball as base for applying patches, to build custom Hadoop.  This is not the recommended method to develop hadoop because it leads to mixed development system where processed files and raw source are hard to separate.  
> To correct the problematic usage of the release tarball, the release build target should be defined as:
> "ant source" generates source release tarball.
> "ant binary" is binary release without source/javadoc jar files.
> "ant tar" is a mirror of binary release with source/javadoc jar files.
> Does this sound reasonable?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16053-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 17:35:22 2011
Return-Path: <common-issues-return-16053-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E34F3650A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 17:35:21 +0000 (UTC)
Received: (qmail 793 invoked by uid 500); 9 Jun 2011 17:35:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 766 invoked by uid 500); 9 Jun 2011 17:35:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 758 invoked by uid 99); 9 Jun 2011 17:35:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 17:35:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 17:35:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C61D910B123
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 17:34:58 +0000 (UTC)
Date: Thu, 9 Jun 2011 17:34:58 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1666235636.7586.1307640898807.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1415670632.7518.1307638679263.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7371) Improve tarball distributions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046686#comment-13046686 ] 

Eric Yang commented on HADOOP-7371:
-----------------------------------

{quote}Because of that, wouldn't make sense to wait till Mavenization is in place for this?{quote}

Yes, for trunk, it would.  For 0.20.205 branch, maven work is not going to be back ported to this branch, and it looks like a needed improvement for 0.20.205 if there is plan to make future release from 0.20-security branch.

> Improve tarball distributions
> -----------------------------
>
>                 Key: HADOOP-7371
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7371
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>         Environment: Java 6, Redhat 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>
> Hadoop release tarball contains both raw source and binary.  This leads users to use the release tarball as base for applying patches, to build custom Hadoop.  This is not the recommended method to develop hadoop because it leads to mixed development system where processed files and raw source are hard to separate.  
> To correct the problematic usage of the release tarball, the release build target should be defined as:
> "ant source" generates source release tarball.
> "ant binary" is binary release without source/javadoc jar files.
> "ant tar" is a mirror of binary release with source/javadoc jar files.
> Does this sound reasonable?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16054-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 17:37:22 2011
Return-Path: <common-issues-return-16054-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 409D7652F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 17:37:22 +0000 (UTC)
Received: (qmail 1340 invoked by uid 500); 9 Jun 2011 17:37:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1311 invoked by uid 500); 9 Jun 2011 17:37:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1303 invoked by uid 99); 9 Jun 2011 17:37:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 17:37:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 17:37:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id ECF1D10B1C4
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 17:36:58 +0000 (UTC)
Date: Thu, 9 Jun 2011 17:36:58 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <568169351.7591.1307641018965.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1415670632.7518.1307638679263.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7371) Improve tarball distributions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046687#comment-13046687 ] 

Alejandro Abdelnur commented on HADOOP-7371:
--------------------------------------------

Got it, so this means this JIRA is not for trunk, right?

> Improve tarball distributions
> -----------------------------
>
>                 Key: HADOOP-7371
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7371
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>         Environment: Java 6, Redhat 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>
> Hadoop release tarball contains both raw source and binary.  This leads users to use the release tarball as base for applying patches, to build custom Hadoop.  This is not the recommended method to develop hadoop because it leads to mixed development system where processed files and raw source are hard to separate.  
> To correct the problematic usage of the release tarball, the release build target should be defined as:
> "ant source" generates source release tarball.
> "ant binary" is binary release without source/javadoc jar files.
> "ant tar" is a mirror of binary release with source/javadoc jar files.
> Does this sound reasonable?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16055-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 17:41:22 2011
Return-Path: <common-issues-return-16055-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 061426606
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 17:41:22 +0000 (UTC)
Received: (qmail 8179 invoked by uid 500); 9 Jun 2011 17:41:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8095 invoked by uid 500); 9 Jun 2011 17:41:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8087 invoked by uid 99); 9 Jun 2011 17:41:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 17:41:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 17:41:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D947410B3A0
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 17:40:58 +0000 (UTC)
Date: Thu, 9 Jun 2011 17:40:58 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1368566129.7603.1307641258886.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <34642033.3863.1307553598815.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7367) getgrouplist() in getGroup.c is not
 portable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Allen Wittenauer updated HADOOP-7367:
-------------------------------------

    Affects Version/s: 0.20.203.0
             Assignee: Allen Wittenauer
               Labels: regression  (was: )

Adding the regression and 0.20.203 info, since this broke functionality that previously worked due to the compression codecs being in the same library.  (Maybe libhadoop.so needs to get broken up?)

> getgrouplist() in getGroup.c is not portable
> --------------------------------------------
>
>                 Key: HADOOP-7367
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7367
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.20.203.0, 0.22.0, 0.23.0
>         Environment: System V (Solaris, HP-UX, AIX?)
> Mac OS X
>            Reporter: Allen Wittenauer
>            Assignee: Allen Wittenauer
>              Labels: regression
>         Attachments: hadoop-7367.patch
>
>
> getGroupIDList uses getgrouplist() to fetch the groups for a user.  Unfortunately, this routine is a BSD-specific call and is not present in most System V-based operating systems. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16056-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 17:43:22 2011
Return-Path: <common-issues-return-16056-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1CC7B6642
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 17:43:22 +0000 (UTC)
Received: (qmail 9942 invoked by uid 500); 9 Jun 2011 17:43:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9915 invoked by uid 500); 9 Jun 2011 17:43:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9907 invoked by uid 99); 9 Jun 2011 17:43:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 17:43:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 17:43:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E824A10B45E
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 17:42:58 +0000 (UTC)
Date: Thu, 9 Jun 2011 17:42:58 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1944938259.7610.1307641378947.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1415670632.7518.1307638679263.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7371) Improve tarball distributions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046693#comment-13046693 ] 

Eric Yang commented on HADOOP-7371:
-----------------------------------

If this can be done as part of HADOOP-6671, then please change this jira to 0.20.205.  Otherwise, I will submit a patch for trunk and make another jira for 0.20.205.

> Improve tarball distributions
> -----------------------------
>
>                 Key: HADOOP-7371
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7371
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>         Environment: Java 6, Redhat 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>
> Hadoop release tarball contains both raw source and binary.  This leads users to use the release tarball as base for applying patches, to build custom Hadoop.  This is not the recommended method to develop hadoop because it leads to mixed development system where processed files and raw source are hard to separate.  
> To correct the problematic usage of the release tarball, the release build target should be defined as:
> "ant source" generates source release tarball.
> "ant binary" is binary release without source/javadoc jar files.
> "ant tar" is a mirror of binary release with source/javadoc jar files.
> Does this sound reasonable?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16057-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 17:52:21 2011
Return-Path: <common-issues-return-16057-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 030AB6C95
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 17:52:21 +0000 (UTC)
Received: (qmail 21826 invoked by uid 500); 9 Jun 2011 17:52:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21791 invoked by uid 500); 9 Jun 2011 17:52:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21783 invoked by uid 99); 9 Jun 2011 17:52:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 17:52:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 17:52:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8C3DF10B95D
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 17:51:59 +0000 (UTC)
Date: Thu, 9 Jun 2011 17:51:59 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <961178887.7695.1307641919571.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1250812071.19292.1305645587504.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7297) Error in the documentation
 regarding Checkpoint/Backup Node
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046701#comment-13046701 ] 

Harsh J commented on HADOOP-7297:
---------------------------------

Christoph,

Thank you very much for contributing a patch fix!

Since CN and BN nodes are both not present in this release, all documentation that refers to it around ought to be removed. Could you re-up a fresh patch with all relevant documentation and references removed?

I'm unsure as to how it got in (perhaps a bad documentation cherry-pick while implementing the branch?)

I think the QA failed to apply the patch because of Eclipse specific lines in it (as seen on top). Could you try with a simple {{svn diff}} or a {{git diff --no-prefix}} and re-upload?

> Error in the documentation regarding Checkpoint/Backup Node
> -----------------------------------------------------------
>
>                 Key: HADOOP-7297
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7297
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.20.203.0
>            Reporter: arnaud p
>            Priority: Trivial
>         Attachments: hadoop-7297.patch
>
>
> On http://hadoop.apache.org/common/docs/r0.20.203.0/hdfs_user_guide.html#Checkpoint+Node: the command bin/hdfs namenode -checkpoint required to launch the backup/checkpoint node does not exist.
> I have removed this from the docs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16060-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 17:58:23 2011
Return-Path: <common-issues-return-16060-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 641706EEF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 17:58:23 +0000 (UTC)
Received: (qmail 34185 invoked by uid 500); 9 Jun 2011 17:58:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34116 invoked by uid 500); 9 Jun 2011 17:58:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34059 invoked by uid 99); 9 Jun 2011 17:58:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 17:58:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 17:58:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B2A2610BC85
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 17:58:01 +0000 (UTC)
Date: Thu, 9 Jun 2011 17:58:01 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1877453133.7770.1307642281728.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1415670632.7518.1307638679263.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7371) Improve tarball distributions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046710#comment-13046710 ] 

Eli Collins commented on HADOOP-7371:
-------------------------------------

bq. Because of that, wouldn't make sense to wait till Mavenization is in place for this?

Is common/hdfs/mr mavenization coming soon? We shouldn't block this work on a much larger project.

> Improve tarball distributions
> -----------------------------
>
>                 Key: HADOOP-7371
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7371
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>         Environment: Java 6, Redhat 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>             Fix For: 0.20.205.0
>
>
> Hadoop release tarball contains both raw source and binary.  This leads users to use the release tarball as base for applying patches, to build custom Hadoop.  This is not the recommended method to develop hadoop because it leads to mixed development system where processed files and raw source are hard to separate.  
> To correct the problematic usage of the release tarball, the release build target should be defined as:
> "ant source" generates source release tarball.
> "ant binary" is binary release without source/javadoc jar files.
> "ant tar" is a mirror of binary release with source/javadoc jar files.
> Does this sound reasonable?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16058-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 17:58:23 2011
Return-Path: <common-issues-return-16058-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B4D7F6F0B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 17:58:23 +0000 (UTC)
Received: (qmail 33839 invoked by uid 500); 9 Jun 2011 17:58:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33799 invoked by uid 500); 9 Jun 2011 17:58:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33763 invoked by uid 99); 9 Jun 2011 17:58:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 17:58:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 17:58:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2219010BC72
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 17:58:01 +0000 (UTC)
Date: Thu, 9 Jun 2011 17:58:01 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1051028881.7757.1307642281135.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <710160053.68701.1307208347759.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7359) Pluggable interface for cluster
 membership
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046708#comment-13046708 ] 

Aaron T. Myers commented on HADOOP-7359:
----------------------------------------

bq. Would anyone object to allowing the HostsReader to trigger refreshNodes? That would let Hadoop scan for or be notified of cluster membership changes and automagically do the Right Thing.

In the abstract I think this is a fine change to make.

bq. Introduce a "Refreshable" interface that both FSNamesystem and JobTracker implement, that only defines a refreshNodes method. HostsReader would have an initialize method that takes a Refreshable and users could choose to call refreshNodes.

I think the name "Refreshable" isn't the best. Seems a little too generic to me. How about something like "NodeListRefreshable" ?

Also, the NN and the JT already implement the interfaces {{o.a.h.hdfs.protocol.ClientProtocol}} and {{o.a.h.mapred.AdminOperationsProtocol}}, respectively, both of which require implementation of a {{refreshNodes()}} method which happen to have the same signature. You could just make these interfaces extend your new interface and then you'd get the genericity you'd need without actually having to touch the NN or JT classes at all.

bq. The current file-based cluster membership would continue to work exactly as it does today.

That seems wise to me. This proposed change would also make it easy to potentially make the {{HostsFileReader}} do something like periodically check the mtime of the hosts files and re-read them automatically if they've changed and call {{refreshNodes()}} on the relevant {{NodeListRefreshable}}.

> Pluggable interface for cluster membership
> ------------------------------------------
>
>                 Key: HADOOP-7359
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7359
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Travis Crawford
>         Attachments: HADOOP-7359.diff
>
>
> Currently Hadoop uses local files to determine cluster membership. With HDFS for example, dfs.hosts and dfs.hosts.exclude are used.
> To enable tighter integrations cluster membership should be an interface, with the current file-based functionality provided as the default implementation. The common case would be no functional change, however, sites could plug an alternative implementation in, such as pulling the machine lists from a machine database.
> DETAILS:
> Two machine lists, includes and excludes, are used to define cluster membership and state. HostsFileReader currently handles reading these lists from files, who's names are passed in by FSNamesystem for HDFS and JobTracker for MR.
> The proposed change is adding a HostsReader interface to common, and changing HostsFileReader to an abstract class that functions the same as today.
> Two new classes, DFSHostsFileReader and MRHostsFileReader, extend HostsFileReader and simply pass the appropriate file names in. These new classes are needed because config key names live outside common.
> Two new conf keys, defaulting to the file-based readers, would be added to choose a different hosts reader: dfs.namenode.hosts.reader.class mapreduce.jobtracker.hosts.reader.class
> Comments/suggestions? I have most of this written already but would love some feedback on the general idea before posting the diff.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16059-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 17:58:24 2011
Return-Path: <common-issues-return-16059-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 087EF6F15
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 17:58:24 +0000 (UTC)
Received: (qmail 33925 invoked by uid 500); 9 Jun 2011 17:58:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33888 invoked by uid 500); 9 Jun 2011 17:58:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33803 invoked by uid 99); 9 Jun 2011 17:58:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 17:58:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 17:58:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7292D10BC7F
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 17:58:01 +0000 (UTC)
Date: Thu, 9 Jun 2011 17:58:01 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <723671514.7764.1307642281465.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1250812071.19292.1305645587504.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7297) Error in the documentation
 regarding Checkpoint/Backup Node
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046709#comment-13046709 ] 

Todd Lipcon commented on HADOOP-7297:
-------------------------------------

If the patch is against non-trunk, the QA Bot isn't smart enough to apply a patch.

> Error in the documentation regarding Checkpoint/Backup Node
> -----------------------------------------------------------
>
>                 Key: HADOOP-7297
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7297
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.20.203.0
>            Reporter: arnaud p
>            Priority: Trivial
>         Attachments: hadoop-7297.patch
>
>
> On http://hadoop.apache.org/common/docs/r0.20.203.0/hdfs_user_guide.html#Checkpoint+Node: the command bin/hdfs namenode -checkpoint required to launch the backup/checkpoint node does not exist.
> I have removed this from the docs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16061-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 17:58:24 2011
Return-Path: <common-issues-return-16061-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9D6056F3A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 17:58:24 +0000 (UTC)
Received: (qmail 34282 invoked by uid 500); 9 Jun 2011 17:58:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34244 invoked by uid 500); 9 Jun 2011 17:58:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34111 invoked by uid 99); 9 Jun 2011 17:58:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 17:58:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 17:58:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C220F10BC86
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 17:58:01 +0000 (UTC)
Date: Thu, 9 Jun 2011 17:58:01 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1377580960.7771.1307642281791.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1415670632.7518.1307638679263.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7371) Improve tarball distributions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-7371:
---------------------------------------

    Fix Version/s: 0.20.205.0

HADOOP-6671 (in trunk) will include options to generate source/binary/tar as described in this JIRA

> Improve tarball distributions
> -----------------------------
>
>                 Key: HADOOP-7371
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7371
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>         Environment: Java 6, Redhat 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>             Fix For: 0.20.205.0
>
>
> Hadoop release tarball contains both raw source and binary.  This leads users to use the release tarball as base for applying patches, to build custom Hadoop.  This is not the recommended method to develop hadoop because it leads to mixed development system where processed files and raw source are hard to separate.  
> To correct the problematic usage of the release tarball, the release build target should be defined as:
> "ant source" generates source release tarball.
> "ant binary" is binary release without source/javadoc jar files.
> "ant tar" is a mirror of binary release with source/javadoc jar files.
> Does this sound reasonable?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16062-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 18:02:23 2011
Return-Path: <common-issues-return-16062-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3164C6B30
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 18:02:23 +0000 (UTC)
Received: (qmail 42134 invoked by uid 500); 9 Jun 2011 18:02:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42074 invoked by uid 500); 9 Jun 2011 18:02:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42011 invoked by uid 99); 9 Jun 2011 18:02:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 18:02:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 18:02:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 75D7810BE79
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 18:02:00 +0000 (UTC)
Date: Thu, 9 Jun 2011 18:02:00 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1842020300.7793.1307642520479.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1415670632.7518.1307638679263.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7371) Improve tarball distributions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046713#comment-13046713 ] 

Alejandro Abdelnur commented on HADOOP-7371:
--------------------------------------------

Rg Eli's comment: I'm wiring DEB/RPM in HADOOP-6671, after doing Mavenization of hadoop-common would be equivalent to Ant functionality.

> Improve tarball distributions
> -----------------------------
>
>                 Key: HADOOP-7371
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7371
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>         Environment: Java 6, Redhat 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>             Fix For: 0.20.205.0
>
>
> Hadoop release tarball contains both raw source and binary.  This leads users to use the release tarball as base for applying patches, to build custom Hadoop.  This is not the recommended method to develop hadoop because it leads to mixed development system where processed files and raw source are hard to separate.  
> To correct the problematic usage of the release tarball, the release build target should be defined as:
> "ant source" generates source release tarball.
> "ant binary" is binary release without source/javadoc jar files.
> "ant tar" is a mirror of binary release with source/javadoc jar files.
> Does this sound reasonable?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16063-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 18:12:22 2011
Return-Path: <common-issues-return-16063-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 781146823
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 18:12:22 +0000 (UTC)
Received: (qmail 58178 invoked by uid 500); 9 Jun 2011 18:12:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58145 invoked by uid 500); 9 Jun 2011 18:12:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58137 invoked by uid 99); 9 Jun 2011 18:12:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 18:12:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 18:12:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D4A5F10B267
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 18:11:58 +0000 (UTC)
Date: Thu, 9 Jun 2011 18:11:58 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1479142716.7833.1307643118867.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7372) Remove ref of 20.3 release from
 branch-0.20 CHANGES.txt
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Remove ref of 20.3 release from branch-0.20 CHANGES.txt
-------------------------------------------------------

                 Key: HADOOP-7372
                 URL: https://issues.apache.org/jira/browse/HADOOP-7372
             Project: Hadoop Common
          Issue Type: Task
          Components: documentation
            Reporter: Eli Collins
            Assignee: Eli Collins
             Fix For: 0.20.3
         Attachments: hadoop-7372-1.patch

CHANGES.txt on branch-0.20 claims there was a 0.20.3 release on 1/5. There has not been a 0.20.3 release.

{noformat}
Release 0.20.4 - Unreleased
...
Release 0.20.3 - 2011-1-5
{noformat}

We should update this to indicate 0.20. is unreleased.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16064-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 18:12:24 2011
Return-Path: <common-issues-return-16064-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 87DFA6869
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 18:12:24 +0000 (UTC)
Received: (qmail 59033 invoked by uid 500); 9 Jun 2011 18:12:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58995 invoked by uid 500); 9 Jun 2011 18:12:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58911 invoked by uid 99); 9 Jun 2011 18:12:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 18:12:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 18:12:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 455A610B275
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 18:11:59 +0000 (UTC)
Date: Thu, 9 Jun 2011 18:11:59 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1052252352.7844.1307643119281.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1479142716.7833.1307643118867.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7372) Remove ref of 20.3 release from
 branch-0.20 CHANGES.txt
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7372:
--------------------------------

    Attachment: hadoop-7372-1.patch

Patch attached.

> Remove ref of 20.3 release from branch-0.20 CHANGES.txt
> -------------------------------------------------------
>
>                 Key: HADOOP-7372
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7372
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.20.3
>
>         Attachments: hadoop-7372-1.patch
>
>
> CHANGES.txt on branch-0.20 claims there was a 0.20.3 release on 1/5. There has not been a 0.20.3 release.
> {noformat}
> Release 0.20.4 - Unreleased
> ...
> Release 0.20.3 - 2011-1-5
> {noformat}
> We should update this to indicate 0.20. is unreleased.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16065-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 18:18:23 2011
Return-Path: <common-issues-return-16065-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ECEF06835
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 18:18:22 +0000 (UTC)
Received: (qmail 71195 invoked by uid 500); 9 Jun 2011 18:18:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71165 invoked by uid 500); 9 Jun 2011 18:18:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71157 invoked by uid 99); 9 Jun 2011 18:18:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 18:18:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 18:18:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 410E810B496
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 18:17:59 +0000 (UTC)
Date: Thu, 9 Jun 2011 18:17:59 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <593050139.7874.1307643479263.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1415670632.7518.1307638679263.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7371) Improve tarball distributions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046721#comment-13046721 ] 

Eli Collins commented on HADOOP-7371:
-------------------------------------

But this jira pertains to all the projects (we release one tarball, not three), not just common. Ie we'd need the HDFS and MR side of HADOOP-6671 to use this here.

> Improve tarball distributions
> -----------------------------
>
>                 Key: HADOOP-7371
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7371
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>         Environment: Java 6, Redhat 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>             Fix For: 0.20.205.0
>
>
> Hadoop release tarball contains both raw source and binary.  This leads users to use the release tarball as base for applying patches, to build custom Hadoop.  This is not the recommended method to develop hadoop because it leads to mixed development system where processed files and raw source are hard to separate.  
> To correct the problematic usage of the release tarball, the release build target should be defined as:
> "ant source" generates source release tarball.
> "ant binary" is binary release without source/javadoc jar files.
> "ant tar" is a mirror of binary release with source/javadoc jar files.
> Does this sound reasonable?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16066-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 18:32:20 2011
Return-Path: <common-issues-return-16066-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0F6DA619C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 18:32:20 +0000 (UTC)
Received: (qmail 99190 invoked by uid 500); 9 Jun 2011 18:32:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99156 invoked by uid 500); 9 Jun 2011 18:32:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99148 invoked by uid 99); 9 Jun 2011 18:32:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 18:32:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 18:32:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D0A2610BA26
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 18:31:58 +0000 (UTC)
Date: Thu, 9 Jun 2011 18:31:58 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <92564299.7945.1307644318851.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1415670632.7518.1307638679263.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7371) Improve tarball distributions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046726#comment-13046726 ] 

Alejandro Abdelnur commented on HADOOP-7371:
--------------------------------------------

Yes, we need HDFS & MR mavenized as well.

Those would be done immediately after HADOOP-6671, first HDFS and then MR.

Note that you'll be able to work on non-mavenized HDFS and MR using mavenized COMMON.



> Improve tarball distributions
> -----------------------------
>
>                 Key: HADOOP-7371
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7371
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>         Environment: Java 6, Redhat 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>             Fix For: 0.20.205.0
>
>
> Hadoop release tarball contains both raw source and binary.  This leads users to use the release tarball as base for applying patches, to build custom Hadoop.  This is not the recommended method to develop hadoop because it leads to mixed development system where processed files and raw source are hard to separate.  
> To correct the problematic usage of the release tarball, the release build target should be defined as:
> "ant source" generates source release tarball.
> "ant binary" is binary release without source/javadoc jar files.
> "ant tar" is a mirror of binary release with source/javadoc jar files.
> Does this sound reasonable?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16067-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 19:41:22 2011
Return-Path: <common-issues-return-16067-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B81666E18
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 19:41:22 +0000 (UTC)
Received: (qmail 29411 invoked by uid 500); 9 Jun 2011 19:41:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29344 invoked by uid 500); 9 Jun 2011 19:41:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29336 invoked by uid 99); 9 Jun 2011 19:41:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 19:41:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 19:41:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DAA0110B7AD
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 19:40:58 +0000 (UTC)
Date: Thu, 9 Jun 2011 19:40:58 +0000 (UTC)
From: "Niels Basjes (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <617053419.8365.1307648458891.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1008788632.26385.1305807047638.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7305) Eclipse project files are incomplete
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Niels Basjes updated HADOOP-7305:
---------------------------------

    Attachment: HADOOP-7305-2011-06-09.patch

I double checked and when I just import the project and do a refresh (F5) on the project then I get the error.
I checked this on both 
# my regular CentOS 5.6 with the SUN JDK 1.6.0_22 and manually downloaded Eclipse 
# on a pristine Fedora 15 with the packaged OpenJDK and packaged Eclipse.

Also note that I first checked in the mailing list to verify if anyone else has the same problem and Todd Lipcon immediately stepped forward. So yes, others experience the same issue.

I've updated the patch to reflect the different path you mentioned.
I hope it's right this time.

> Eclipse project files are incomplete
> ------------------------------------
>
>                 Key: HADOOP-7305
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7305
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Niels Basjes
>            Assignee: Niels Basjes
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7305-2011-05-19.patch, HADOOP-7305-2011-05-30.patch, HADOOP-7305-2011-06-09.patch
>
>
> After a fresh checkout of hadoop-common I do 'ant compile eclipse'.
> I open eclipse, set ANT_HOME and build the project. 
> At that point the following error appears:
> {quote}
> The type com.sun.javadoc.RootDoc cannot be resolved. It is indirectly referenced from required .class files	ExcludePrivateAnnotationsJDiffDoclet.java	/common/src/java/org/apache/hadoop/classification/tools	line 1	Java Problem
> {quote}
> The solution is to add the "tools.jar" from the JDK to the buildpath/classpath.
> This should be fixed in the build.xml.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16068-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 19:55:23 2011
Return-Path: <common-issues-return-16068-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8334A6793
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 19:55:23 +0000 (UTC)
Received: (qmail 64492 invoked by uid 500); 9 Jun 2011 19:55:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64454 invoked by uid 500); 9 Jun 2011 19:55:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64446 invoked by uid 99); 9 Jun 2011 19:55:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 19:55:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 19:55:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0F01010BCD8
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 19:55:00 +0000 (UTC)
Date: Thu, 9 Jun 2011 19:55:00 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1554767447.8408.1307649300058.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1008788632.26385.1305807047638.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7305) Eclipse project files are
 incomplete
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046774#comment-13046774 ] 

Hadoop QA commented on HADOOP-7305:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481968/HADOOP-7305-2011-06-09.patch
  against trunk revision 1133125.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/602//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/602//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/602//console

This message is automatically generated.

> Eclipse project files are incomplete
> ------------------------------------
>
>                 Key: HADOOP-7305
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7305
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Niels Basjes
>            Assignee: Niels Basjes
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7305-2011-05-19.patch, HADOOP-7305-2011-05-30.patch, HADOOP-7305-2011-06-09.patch
>
>
> After a fresh checkout of hadoop-common I do 'ant compile eclipse'.
> I open eclipse, set ANT_HOME and build the project. 
> At that point the following error appears:
> {quote}
> The type com.sun.javadoc.RootDoc cannot be resolved. It is indirectly referenced from required .class files	ExcludePrivateAnnotationsJDiffDoclet.java	/common/src/java/org/apache/hadoop/classification/tools	line 1	Java Problem
> {quote}
> The solution is to add the "tools.jar" from the JDK to the buildpath/classpath.
> This should be fixed in the build.xml.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16069-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 20:03:20 2011
Return-Path: <common-issues-return-16069-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 614846986
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 20:03:20 +0000 (UTC)
Received: (qmail 80676 invoked by uid 500); 9 Jun 2011 20:03:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80644 invoked by uid 500); 9 Jun 2011 20:03:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80465 invoked by uid 99); 9 Jun 2011 20:03:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 20:03:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 20:03:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E9D0910B108
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 20:02:58 +0000 (UTC)
Date: Thu, 9 Jun 2011 20:02:58 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2129468871.8460.1307649778954.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7373) Tarball deployment doesn't work with
 {start,stop}-{dfs,mapred}
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Tarball deployment doesn't work with {start,stop}-{dfs,mapred}
--------------------------------------------------------------

                 Key: HADOOP-7373
                 URL: https://issues.apache.org/jira/browse/HADOOP-7373
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Owen O'Malley


The hadoop-config.sh overrides the variable "bin", which makes the scripts use libexec for hadoop-daemon(s).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16070-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 20:05:22 2011
Return-Path: <common-issues-return-16070-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A00306EB1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 20:05:22 +0000 (UTC)
Received: (qmail 84568 invoked by uid 500); 9 Jun 2011 20:05:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84533 invoked by uid 500); 9 Jun 2011 20:05:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84518 invoked by uid 99); 9 Jun 2011 20:05:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 20:05:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 20:05:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C9A1210B1C5
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 20:04:58 +0000 (UTC)
Date: Thu, 9 Jun 2011 20:04:58 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1828424769.8463.1307649898822.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2129468871.8460.1307649778954.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7373) Tarball deployment doesn't work
 with {start,stop}-{dfs,mapred}
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046782#comment-13046782 ] 

Owen O'Malley commented on HADOOP-7373:
---------------------------------------

This also applies to the hadoop-daemons.sh when it is looking up slaves.sh

> Tarball deployment doesn't work with {start,stop}-{dfs,mapred}
> --------------------------------------------------------------
>
>                 Key: HADOOP-7373
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7373
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>
> The hadoop-config.sh overrides the variable "bin", which makes the scripts use libexec for hadoop-daemon(s).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16071-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 20:49:20 2011
Return-Path: <common-issues-return-16071-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 239666508
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 20:49:20 +0000 (UTC)
Received: (qmail 47387 invoked by uid 500); 9 Jun 2011 20:49:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47362 invoked by uid 500); 9 Jun 2011 20:49:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47354 invoked by uid 99); 9 Jun 2011 20:49:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 20:49:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 20:49:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DD66610BFD9
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 20:48:58 +0000 (UTC)
Date: Thu, 9 Jun 2011 20:48:58 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <979138249.8540.1307652538903.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2129468871.8460.1307649778954.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7373) Tarball deployment doesn't work with
 {start,stop}-{dfs,mapred}
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7373:
----------------------------------

    Attachment: c.patch

This patch fixes hadoop-config.sh to use a different variable name, so that the other scripts don't get their environment clobbered.

> Tarball deployment doesn't work with {start,stop}-{dfs,mapred}
> --------------------------------------------------------------
>
>                 Key: HADOOP-7373
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7373
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>         Attachments: c.patch
>
>
> The hadoop-config.sh overrides the variable "bin", which makes the scripts use libexec for hadoop-daemon(s).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16072-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 20:49:22 2011
Return-Path: <common-issues-return-16072-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8EB386518
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 20:49:22 +0000 (UTC)
Received: (qmail 47684 invoked by uid 500); 9 Jun 2011 20:49:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47649 invoked by uid 500); 9 Jun 2011 20:49:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47634 invoked by uid 99); 9 Jun 2011 20:49:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 20:49:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 20:49:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1A7ED10BFDE
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 20:48:59 +0000 (UTC)
Date: Thu, 9 Jun 2011 20:48:59 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <161372953.8544.1307652539105.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2129468871.8460.1307649778954.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7373) Tarball deployment doesn't work
 with {start,stop}-{dfs,mapred}
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley reassigned HADOOP-7373:
-------------------------------------

    Assignee: Owen O'Malley

> Tarball deployment doesn't work with {start,stop}-{dfs,mapred}
> --------------------------------------------------------------
>
>                 Key: HADOOP-7373
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7373
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: c.patch
>
>
> The hadoop-config.sh overrides the variable "bin", which makes the scripts use libexec for hadoop-daemon(s).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16073-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 20:58:20 2011
Return-Path: <common-issues-return-16073-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3E0726799
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 20:58:20 +0000 (UTC)
Received: (qmail 54789 invoked by uid 500); 9 Jun 2011 20:58:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54759 invoked by uid 500); 9 Jun 2011 20:58:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54751 invoked by uid 99); 9 Jun 2011 20:58:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 20:58:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 20:58:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C233D10B2EF
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 20:57:58 +0000 (UTC)
Date: Thu, 9 Jun 2011 20:57:58 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <176931664.8562.1307653078792.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2129468871.8460.1307649778954.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7373) Tarball deployment doesn't work
 with {start,stop}-{dfs,mapred}
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046799#comment-13046799 ] 

Eric Yang commented on HADOOP-7373:
-----------------------------------

We can probably remove the entire block for 0.20.20x:

{noformat}
# convert relative path to absolute path
bin=`dirname "$this"`
script=`basename "$this"`
bin=`cd "$bin"; pwd`
this="$bin/$script"
{noformat}

> Tarball deployment doesn't work with {start,stop}-{dfs,mapred}
> --------------------------------------------------------------
>
>                 Key: HADOOP-7373
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7373
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: c.patch
>
>
> The hadoop-config.sh overrides the variable "bin", which makes the scripts use libexec for hadoop-daemon(s).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16074-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 21:20:21 2011
Return-Path: <common-issues-return-16074-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A1B73661A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 21:20:21 +0000 (UTC)
Received: (qmail 89227 invoked by uid 500); 9 Jun 2011 21:20:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89183 invoked by uid 500); 9 Jun 2011 21:20:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89175 invoked by uid 99); 9 Jun 2011 21:20:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 21:20:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 21:20:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0516010BBA3
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 21:20:00 +0000 (UTC)
Date: Thu, 9 Jun 2011 21:20:00 +0000 (UTC)
From: "E. Sammer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <989466830.8625.1307654400017.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <710160053.68701.1307208347759.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7359) Pluggable interface for cluster
 membership
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046809#comment-13046809 ] 

E. Sammer commented on HADOOP-7359:
-----------------------------------

I like the idea of this, but I would not necessarily attach it to the notion of files, explicitly. I would propose a ClusterTopology SPI-style interface and a ClusterTopologyListener interface for those interested in topology changes. Ideally, all clients (either internal to Hadoop daemons or external tools) would ask implementations of ClusterTopology for the list of hosts.

Off the top of my head API:

ClusterTopology <<interface>>
  getNodes() : Set<Node>
  refresh()
  getListeners() : Set<ClusterTopologyListener>
  addListener(ClusterTopologyListener) : boolean (wasAdded)
  removeListener(ClusterTopologyListener) : boolean (wasRemoved)

ClusterTopologyListener <<interface>>
  onTopologyChange(ClusterTopology)

And *then* have a single class that implements ClusterTopology. Configure the class two ways (no need to have an inheritance hierarchy).

HostFileClusterTopology <<class>>
  /*
     A private member with a base file name. The implementation automatically
     looks for baseFileName + .{include,exclude}
   */
  baseFileName : File

This, to me, seems like it would support the current file based membership but also things like an RDBMS or ZK. In the case of files and an RDBMS (and other non-event based systems) listeners wouldn't be notified of changes until a refresh() occurred. Alternatively, implementations could include a poller which automatically called refresh() when a change is detected or something like that. This is also easy to mock out and test.

> Pluggable interface for cluster membership
> ------------------------------------------
>
>                 Key: HADOOP-7359
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7359
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Travis Crawford
>         Attachments: HADOOP-7359.diff
>
>
> Currently Hadoop uses local files to determine cluster membership. With HDFS for example, dfs.hosts and dfs.hosts.exclude are used.
> To enable tighter integrations cluster membership should be an interface, with the current file-based functionality provided as the default implementation. The common case would be no functional change, however, sites could plug an alternative implementation in, such as pulling the machine lists from a machine database.
> DETAILS:
> Two machine lists, includes and excludes, are used to define cluster membership and state. HostsFileReader currently handles reading these lists from files, who's names are passed in by FSNamesystem for HDFS and JobTracker for MR.
> The proposed change is adding a HostsReader interface to common, and changing HostsFileReader to an abstract class that functions the same as today.
> Two new classes, DFSHostsFileReader and MRHostsFileReader, extend HostsFileReader and simply pass the appropriate file names in. These new classes are needed because config key names live outside common.
> Two new conf keys, defaulting to the file-based readers, would be added to choose a different hosts reader: dfs.namenode.hosts.reader.class mapreduce.jobtracker.hosts.reader.class
> Comments/suggestions? I have most of this written already but would love some feedback on the general idea before posting the diff.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16075-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 21:58:23 2011
Return-Path: <common-issues-return-16075-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D8CDD6DEC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 21:58:23 +0000 (UTC)
Received: (qmail 50238 invoked by uid 500); 9 Jun 2011 21:58:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50202 invoked by uid 500); 9 Jun 2011 21:58:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50186 invoked by uid 99); 9 Jun 2011 21:58:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 21:58:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 21:58:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0469A10BD27
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 21:57:59 +0000 (UTC)
Date: Thu, 9 Jun 2011 21:57:59 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <308187203.8772.1307656679014.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7374) hadoop-config.sh shouldn't add
 tools.jar to the classpath
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

hadoop-config.sh shouldn't add tools.jar to the classpath
---------------------------------------------------------

                 Key: HADOOP-7374
                 URL: https://issues.apache.org/jira/browse/HADOOP-7374
             Project: Hadoop Common
          Issue Type: Improvement
          Components: scripts
            Reporter: Eli Collins
            Assignee: Eli Collins


bin/hadoop-config.sh (and bin/rcc) add lib/tools.jar from JAVA_HOME to the classpath. This has been there since the initial commit of bin/hadoop, but I don't think it's needed. *Executing* Hadoop does not depend on tools.jar (or other libraries only available in the JDK, not the JRE) so let's not automatically add it. Marking this as an incompatible change since a job could potentially have relied on Hadoop adding tools.jar to the CLASSPATH automatically (though such a job would not have run on a system that did not have JAVA_HOME point to a jdk). The build of course still requires a JDK.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16076-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 22:00:20 2011
Return-Path: <common-issues-return-16076-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 852E86FF8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 22:00:20 +0000 (UTC)
Received: (qmail 52318 invoked by uid 500); 9 Jun 2011 22:00:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52287 invoked by uid 500); 9 Jun 2011 22:00:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52272 invoked by uid 99); 9 Jun 2011 22:00:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 22:00:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 22:00:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DB53810BE3E
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 21:59:58 +0000 (UTC)
Date: Thu, 9 Jun 2011 21:59:58 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1449184275.8775.1307656798895.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <308187203.8772.1307656679014.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7374) hadoop-config.sh shouldn't add
 tools.jar to the classpath
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7374:
--------------------------------

    Attachment: hadoop-7374-1.patch

Patch attached. Ran the hdfs and mr daemons and looked at the web UIs for sanity.

> hadoop-config.sh shouldn't add tools.jar to the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-7374
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7374
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>         Attachments: hadoop-7374-1.patch
>
>
> bin/hadoop-config.sh (and bin/rcc) add lib/tools.jar from JAVA_HOME to the classpath. This has been there since the initial commit of bin/hadoop, but I don't think it's needed. *Executing* Hadoop does not depend on tools.jar (or other libraries only available in the JDK, not the JRE) so let's not automatically add it. Marking this as an incompatible change since a job could potentially have relied on Hadoop adding tools.jar to the CLASSPATH automatically (though such a job would not have run on a system that did not have JAVA_HOME point to a jdk). The build of course still requires a JDK.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16077-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 22:26:22 2011
Return-Path: <common-issues-return-16077-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8877C62AD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 22:26:22 +0000 (UTC)
Received: (qmail 87759 invoked by uid 500); 9 Jun 2011 22:26:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87677 invoked by uid 500); 9 Jun 2011 22:26:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87556 invoked by uid 99); 9 Jun 2011 22:26:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 22:26:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 22:26:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0980310B940
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 22:25:59 +0000 (UTC)
Date: Thu, 9 Jun 2011 22:25:59 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1340908608.8852.1307658359035.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2129468871.8460.1307649778954.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7373) Tarball deployment doesn't work
 with {start,stop}-{dfs,mapred}
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046860#comment-13046860 ] 

Eli Collins commented on HADOOP-7373:
-------------------------------------

I think you'll need these if you want the init scripts to run on boot.

> Tarball deployment doesn't work with {start,stop}-{dfs,mapred}
> --------------------------------------------------------------
>
>                 Key: HADOOP-7373
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7373
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: c.patch
>
>
> The hadoop-config.sh overrides the variable "bin", which makes the scripts use libexec for hadoop-daemon(s).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16078-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 22:32:27 2011
Return-Path: <common-issues-return-16078-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 986CB6E75
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 22:32:27 +0000 (UTC)
Received: (qmail 91995 invoked by uid 500); 9 Jun 2011 22:32:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91966 invoked by uid 500); 9 Jun 2011 22:32:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91958 invoked by uid 99); 9 Jun 2011 22:32:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 22:32:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 22:32:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 140A810BB52
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 22:32:06 +0000 (UTC)
Date: Thu, 9 Jun 2011 22:32:06 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1684609090.8865.1307658726079.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <308187203.8772.1307656679014.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7374) hadoop-config.sh shouldn't add
 tools.jar to the classpath
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046863#comment-13046863 ] 

Owen O'Malley commented on HADOOP-7374:
---------------------------------------

The thing to test is that the tools still run: distcp, har, etc.

> hadoop-config.sh shouldn't add tools.jar to the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-7374
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7374
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>         Attachments: hadoop-7374-1.patch
>
>
> bin/hadoop-config.sh (and bin/rcc) add lib/tools.jar from JAVA_HOME to the classpath. This has been there since the initial commit of bin/hadoop, but I don't think it's needed. *Executing* Hadoop does not depend on tools.jar (or other libraries only available in the JDK, not the JRE) so let's not automatically add it. Marking this as an incompatible change since a job could potentially have relied on Hadoop adding tools.jar to the CLASSPATH automatically (though such a job would not have run on a system that did not have JAVA_HOME point to a jdk). The build of course still requires a JDK.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16079-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 22:42:21 2011
Return-Path: <common-issues-return-16079-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DA2DD6AA8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 22:42:20 +0000 (UTC)
Received: (qmail 1680 invoked by uid 500); 9 Jun 2011 22:42:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1637 invoked by uid 500); 9 Jun 2011 22:42:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1624 invoked by uid 99); 9 Jun 2011 22:42:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 22:42:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 22:42:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7D45B10B10D
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 22:41:59 +0000 (UTC)
Date: Thu, 9 Jun 2011 22:41:59 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <697193324.8918.1307659319509.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1269276399.4788.1307568238896.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7368) Mechanism for providing version
 info of hadoop jars
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046881#comment-13046881 ] 

Allen Wittenauer commented on HADOOP-7368:
------------------------------------------

If it was a file, I'd be worried about people just replacing the file in the jar to bypass compatibility checks.  At least with the class, they have to do a bit more work.  (I also have to admit that I like the fact that today it is somewhat non-obvious how this works, at least if one can take the "interesting" version information I've seen from various 3rd party Hadoop distributions as anecdotal evidence).

> Mechanism for providing version info of hadoop jars
> ---------------------------------------------------
>
>                 Key: HADOOP-7368
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7368
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.23.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>
> In 0.20.x, only one jar (hadoop-core-*.jar) really matters. The o.a.h.util.VersionInfo combined with saveVersion.sh script (generating a package level annotation) served us well. For 0.23+, due to the project split and modularization of mapreduce (currently in MR-279), a lot more essential hadoop jars are created. The potential of mixing up the jars is significantly increased as well. We need a simple way to list the version info (version, branch, source checksum etc.) for all the jars involved. This is essential for QE and tracking down various issues.
> I propose that we use a VersionProvider interface (similar to the current VersionInfo util class) and ServiceLoader to enumerate the version providers in the jars.
> {code}
> public interface VersionProvider {
>   String getJar(); 
>   String getPackage();
>   String getVersion();
>   String getRevision();
>   String getBranch();
>   String getDate();
>   String getUrl();
>   String getSourceChecksum();
> }
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16080-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun  9 23:58:20 2011
Return-Path: <common-issues-return-16080-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0200F673B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  9 Jun 2011 23:58:20 +0000 (UTC)
Received: (qmail 76725 invoked by uid 500); 9 Jun 2011 23:58:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76685 invoked by uid 500); 9 Jun 2011 23:58:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76677 invoked by uid 99); 9 Jun 2011 23:58:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 23:58:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Jun 2011 23:58:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CDBA610BA38
	for <common-issues@hadoop.apache.org>; Thu,  9 Jun 2011 23:57:58 +0000 (UTC)
Date: Thu, 9 Jun 2011 23:57:58 +0000 (UTC)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <720092189.9164.1307663878839.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2129468871.8460.1307649778954.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7373) Tarball deployment doesn't work
 with {start,stop}-{dfs,mapred}
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046926#comment-13046926 ] 

Giridharan Kesavan commented on HADOOP-7373:
--------------------------------------------

+1 looks good, thanks Owen.

> Tarball deployment doesn't work with {start,stop}-{dfs,mapred}
> --------------------------------------------------------------
>
>                 Key: HADOOP-7373
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7373
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: c.patch
>
>
> The hadoop-config.sh overrides the variable "bin", which makes the scripts use libexec for hadoop-daemon(s).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16081-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 00:54:20 2011
Return-Path: <common-issues-return-16081-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 73C4C6364
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 00:54:20 +0000 (UTC)
Received: (qmail 18400 invoked by uid 500); 10 Jun 2011 00:54:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18365 invoked by uid 500); 10 Jun 2011 00:54:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18356 invoked by uid 99); 10 Jun 2011 00:54:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 00:54:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 00:54:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1859F10C813
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 00:53:59 +0000 (UTC)
Date: Fri, 10 Jun 2011 00:53:59 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2139110755.9253.1307667239096.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6671:
------------------------------

    Attachment: HADOOP-6671-cross-project-HDFS.patch

I've set up a Jenkins job to build common artifacts using Maven: https://builds.apache.org/job/Hadoop-Common-trunk-maven/. 

It's building the same artifacts as https://builds.apache.org/job/Hadoop-Common-trunk/, including documentation and native libraries, and reports for compiler warnings, tests, FindBugs, and Checkstyle. The only missing report is for Clover which needs adding to the Maven build.

Currently two tests are failing (https://builds.apache.org/job/Hadoop-Common-trunk-maven/8/) - I'm not sure why, as they pass for me locally using Maven, and on Hudson using Ant.

I also tried a cross-project build using Maven for common and Ant for HDFS. I needed the attached patch to get the HDFS build to work - these are changes that are needed anyway that we were getting away with using Ivy. MapReduce will need similar changes.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-6671-cross-project-HDFS.patch, HADOOP-6671-e.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16082-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 01:04:22 2011
Return-Path: <common-issues-return-16082-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 45F236A7F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 01:04:22 +0000 (UTC)
Received: (qmail 27725 invoked by uid 500); 10 Jun 2011 01:04:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27700 invoked by uid 500); 10 Jun 2011 01:04:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27692 invoked by uid 99); 10 Jun 2011 01:04:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 01:04:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 01:04:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D37D110CA3C
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 01:03:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 01:03:58 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1978070403.9304.1307667838862.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046947#comment-13046947 ] 

Hadoop QA commented on HADOOP-6671:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482008/HADOOP-6671-cross-project-HDFS.patch
  against trunk revision 1133125.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/603//console

This message is automatically generated.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-6671-cross-project-HDFS.patch, HADOOP-6671-e.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16083-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 04:18:25 2011
Return-Path: <common-issues-return-16083-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0660E4F5B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 04:18:25 +0000 (UTC)
Received: (qmail 71929 invoked by uid 500); 10 Jun 2011 04:18:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71703 invoked by uid 500); 10 Jun 2011 04:18:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71691 invoked by uid 99); 10 Jun 2011 04:18:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 04:18:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 04:18:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CA4E910C139
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 04:17:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 04:17:58 +0000 (UTC)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2080833602.9490.1307679478825.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1894387424.4008.1305137809491.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7278) Automatic Hadoop cluster
 deployment for build validation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046986#comment-13046986 ] 

Konstantin Boudnik commented on HADOOP-7278:
--------------------------------------------

So, the deployment has been working for at least one month now. Here one can see it in action:
  https://builds.apache.org//view/G-L/view/Hadoop/job/Hadoop-22-cluster-deploy/
It also triggers on cluster tests (a downstream job from the above). 

Does anyone want to comment?

> Automatic Hadoop cluster deployment for build validation
> --------------------------------------------------------
>
>                 Key: HADOOP-7278
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7278
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0, 0.23.0
>         Environment: Apache Jenkins
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>              Labels: newbie
>         Attachments: HADOOP-7278.patch
>
>
> It'd be great to have a way of automatically deploying Hadoop cluster to a set of machine once all components are successfully built (in the form or tar or whatever). Deployed cluster then can be used to run a set of validation jobs to make sure that produced artifacts are viable. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16084-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 04:31:21 2011
Return-Path: <common-issues-return-16084-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5792140D5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 04:31:21 +0000 (UTC)
Received: (qmail 82614 invoked by uid 500); 10 Jun 2011 04:31:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82592 invoked by uid 500); 10 Jun 2011 04:31:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82579 invoked by uid 99); 10 Jun 2011 04:31:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 04:31:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 04:31:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1880810C399
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 04:30:59 +0000 (UTC)
Date: Fri, 10 Jun 2011 04:30:59 +0000 (UTC)
From: "Ted Dunning (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1673967036.9509.1307680259096.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32237646.17061288907081267.JavaMail.jira@thor>
Subject: [jira] [Updated] (HADOOP-7020) establish a "Powered by Hadoop" logo
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ted Dunning updated HADOOP-7020:
--------------------------------

    Attachment: PoweredByHadoop_Small.jpg

Here is a potential powered by logo that was created using the standard small elephant logo.

It would be very helpful for people trying to comply with the trademark guidelines to have an approved powered-by logo.  This would make the powered-by/includes/is distinction much easier to make.

> establish a "Powered by Hadoop" logo
> ------------------------------------
>
>                 Key: HADOOP-7020
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7020
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: site
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: site
>
>         Attachments: PoweredByHadoop_Small.jpg, powered-by-hadoop-small.png, powered-by-hadoop.png
>
>
> We should agree on a Powered By Hadoop logo, as suggested in:
> http://www.apache.org/foundation/marks/pmcs#poweredby

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16085-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 04:33:24 2011
Return-Path: <common-issues-return-16085-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 958D74AED
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 04:33:24 +0000 (UTC)
Received: (qmail 83228 invoked by uid 500); 10 Jun 2011 04:33:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83073 invoked by uid 500); 10 Jun 2011 04:33:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83065 invoked by uid 99); 10 Jun 2011 04:33:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 04:33:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 04:33:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0D27110C456
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 04:32:59 +0000 (UTC)
Date: Fri, 10 Jun 2011 04:32:59 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1691766363.9529.1307680379050.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-1886) Undocumented parameters in
 FilesSystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-1886:
--------------------------------

    Fix Version/s: 0.23.0
         Assignee: Frank Conrad
     Hadoop Flags: [Reviewed]

Thanks for contributing Frank!

> Undocumented parameters in FilesSystem
> --------------------------------------
>
>                 Key: HADOOP-1886
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1886
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.14.0
>            Reporter: Konstantin Shvachko
>            Assignee: Frank Conrad
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: hadoop-1886-1.diff
>
>
> Multiple create methods in public FileSystem class lack documentation for the following 2 parameters.
> - long blockSize,
> - Progressable progress

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16086-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 04:35:21 2011
Return-Path: <common-issues-return-16086-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B961F405A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 04:35:21 +0000 (UTC)
Received: (qmail 86314 invoked by uid 500); 10 Jun 2011 04:35:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86270 invoked by uid 500); 10 Jun 2011 04:35:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86244 invoked by uid 99); 10 Jun 2011 04:35:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 04:35:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 04:35:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DE39210C4DB
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 04:34:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 04:34:58 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1983549182.9532.1307680498906.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-1886) Undocumented parameters in
 FilesSystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-1886:
--------------------------------

    Attachment: hadoop-1886-2.patch

Patch attached is a minor update to Frank's, fixes the two javadoc warnings and a couple minor things.

> Undocumented parameters in FilesSystem
> --------------------------------------
>
>                 Key: HADOOP-1886
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1886
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.14.0
>            Reporter: Konstantin Shvachko
>            Assignee: Frank Conrad
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: hadoop-1886-1.diff, hadoop-1886-2.patch
>
>
> Multiple create methods in public FileSystem class lack documentation for the following 2 parameters.
> - long blockSize,
> - Progressable progress

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16087-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 04:35:22 2011
Return-Path: <common-issues-return-16087-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 66E094065
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 04:35:22 +0000 (UTC)
Received: (qmail 86548 invoked by uid 500); 10 Jun 2011 04:35:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86281 invoked by uid 500); 10 Jun 2011 04:35:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86245 invoked by uid 99); 10 Jun 2011 04:35:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 04:35:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 04:35:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F231010C4DD
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 04:34:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 04:34:58 +0000 (UTC)
From: "Ted Dunning (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <157768534.9534.1307680498988.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32237646.17061288907081267.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7020) establish a "Powered by Hadoop"
 logo
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046991#comment-13046991 ] 

Ted Dunning commented on HADOOP-7020:
-------------------------------------

The logo I just put up as a candidate uses a rastered version of the elephant.  I can get a higher quality, scalable version make if this logo is accepted.

> establish a "Powered by Hadoop" logo
> ------------------------------------
>
>                 Key: HADOOP-7020
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7020
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: site
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: site
>
>         Attachments: PoweredByHadoop_Small.jpg, powered-by-hadoop-small.png, powered-by-hadoop.png
>
>
> We should agree on a Powered By Hadoop logo, as suggested in:
> http://www.apache.org/foundation/marks/pmcs#poweredby

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16088-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 04:41:21 2011
Return-Path: <common-issues-return-16088-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 51B6140B4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 04:41:21 +0000 (UTC)
Received: (qmail 88673 invoked by uid 500); 10 Jun 2011 04:41:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88644 invoked by uid 500); 10 Jun 2011 04:41:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88628 invoked by uid 99); 10 Jun 2011 04:41:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 04:41:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 04:41:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8EA6110C64C
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 04:40:59 +0000 (UTC)
Date: Fri, 10 Jun 2011 04:40:59 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <821468188.9572.1307680859581.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32237646.17061288907081267.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7020) establish a "Powered by Hadoop"
 logo
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046992#comment-13046992 ] 

Owen O'Malley commented on HADOOP-7020:
---------------------------------------

I think that looks reasonable. I would have preferred "Powered by Apache Hadoop," but this looks reasonable.

> establish a "Powered by Hadoop" logo
> ------------------------------------
>
>                 Key: HADOOP-7020
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7020
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: site
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: site
>
>         Attachments: PoweredByHadoop_Small.jpg, powered-by-hadoop-small.png, powered-by-hadoop.png
>
>
> We should agree on a Powered By Hadoop logo, as suggested in:
> http://www.apache.org/foundation/marks/pmcs#poweredby

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16089-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 04:55:24 2011
Return-Path: <common-issues-return-16089-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 67169481D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 04:55:24 +0000 (UTC)
Received: (qmail 96275 invoked by uid 500); 10 Jun 2011 04:55:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96224 invoked by uid 500); 10 Jun 2011 04:55:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96189 invoked by uid 99); 10 Jun 2011 04:55:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 04:55:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 04:55:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F1BC010C8E6
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 04:54:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 04:54:58 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1650227280.9587.1307681698987.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32237646.17061288907081267.JavaMail.jira@thor>
Subject: [jira] [Updated] (HADOOP-7020) establish a "Powered by Hadoop" logo
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7020:
--------------------------------

    Attachment: hadoop-elephant-pb.jpeg

How about just the elephant with "Powered by Apache Hadoop"... something like hadooop-elephant-pb.jpeg but with a nicer font?

I like how the Apache PB logo is similar to the main logo: http://www.apache.org/images/apache_pb.gif

> establish a "Powered by Hadoop" logo
> ------------------------------------
>
>                 Key: HADOOP-7020
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7020
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: site
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: site
>
>         Attachments: PoweredByHadoop_Small.jpg, hadoop-elephant-pb.jpeg, powered-by-hadoop-small.png, powered-by-hadoop.png
>
>
> We should agree on a Powered By Hadoop logo, as suggested in:
> http://www.apache.org/foundation/marks/pmcs#poweredby

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16090-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 04:55:24 2011
Return-Path: <common-issues-return-16090-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BD677482A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 04:55:24 +0000 (UTC)
Received: (qmail 96314 invoked by uid 500); 10 Jun 2011 04:55:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96235 invoked by uid 500); 10 Jun 2011 04:55:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96215 invoked by uid 99); 10 Jun 2011 04:55:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 04:55:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 04:55:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DCAAB10C8E4
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 04:54:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 04:54:58 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <654861694.9585.1307681698900.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-1886) Undocumented parameters in
 FilesSystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046994#comment-13046994 ] 

Hadoop QA commented on HADOOP-1886:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482017/hadoop-1886-2.patch
  against trunk revision 1133125.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/604//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/604//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/604//console

This message is automatically generated.

> Undocumented parameters in FilesSystem
> --------------------------------------
>
>                 Key: HADOOP-1886
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1886
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.14.0
>            Reporter: Konstantin Shvachko
>            Assignee: Frank Conrad
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: hadoop-1886-1.diff, hadoop-1886-2.patch
>
>
> Multiple create methods in public FileSystem class lack documentation for the following 2 parameters.
> - long blockSize,
> - Progressable progress

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16091-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 05:01:21 2011
Return-Path: <common-issues-return-16091-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4566B4F8A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 05:01:21 +0000 (UTC)
Received: (qmail 97919 invoked by uid 500); 10 Jun 2011 05:01:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97872 invoked by uid 500); 10 Jun 2011 05:01:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97862 invoked by uid 99); 10 Jun 2011 05:01:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 05:01:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 05:01:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9C09610CA50
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 05:00:59 +0000 (UTC)
Date: Fri, 10 Jun 2011 05:00:59 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1697576487.9628.1307682059635.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32237646.17061288907081267.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7020) establish a "Powered by Hadoop"
 logo
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046997#comment-13046997 ] 

Owen O'Malley commented on HADOOP-7020:
---------------------------------------

I think it is important that the two logos be easily distinguishable from each other. I like the silver circle.

> establish a "Powered by Hadoop" logo
> ------------------------------------
>
>                 Key: HADOOP-7020
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7020
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: site
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: site
>
>         Attachments: PoweredByHadoop_Small.jpg, hadoop-elephant-pb.jpeg, powered-by-hadoop-small.png, powered-by-hadoop.png
>
>
> We should agree on a Powered By Hadoop logo, as suggested in:
> http://www.apache.org/foundation/marks/pmcs#poweredby

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16093-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 05:13:25 2011
Return-Path: <common-issues-return-16093-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 836FB4034
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 05:13:25 +0000 (UTC)
Received: (qmail 15969 invoked by uid 500); 10 Jun 2011 05:13:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15709 invoked by uid 500); 10 Jun 2011 05:13:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15696 invoked by uid 99); 10 Jun 2011 05:13:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 05:13:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 05:13:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DCD3310CC75
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 05:12:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 05:12:58 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1916181072.9639.1307682778901.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <308187203.8772.1307656679014.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7374) hadoop-config.sh shouldn't add
 tools.jar to the classpath
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046999#comment-13046999 ] 

Eli Collins commented on HADOOP-7374:
-------------------------------------

Thanks Owen. I tested distcp and har. I also grepped the sources for the package names in tools.jar and that didn't turn up anything so I think the code is clear. The only dependency looks like Jasper to compile the jsps, but that's a build-time dependency. 

Attached (trivial) patch look good?

> hadoop-config.sh shouldn't add tools.jar to the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-7374
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7374
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>         Attachments: hadoop-7374-1.patch
>
>
> bin/hadoop-config.sh (and bin/rcc) add lib/tools.jar from JAVA_HOME to the classpath. This has been there since the initial commit of bin/hadoop, but I don't think it's needed. *Executing* Hadoop does not depend on tools.jar (or other libraries only available in the JDK, not the JRE) so let's not automatically add it. Marking this as an incompatible change since a job could potentially have relied on Hadoop adding tools.jar to the CLASSPATH automatically (though such a job would not have run on a system that did not have JAVA_HOME point to a jdk). The build of course still requires a JDK.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16092-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 05:13:25 2011
Return-Path: <common-issues-return-16092-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9C4404035
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 05:13:25 +0000 (UTC)
Received: (qmail 15853 invoked by uid 500); 10 Jun 2011 05:13:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15705 invoked by uid 500); 10 Jun 2011 05:13:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15678 invoked by uid 99); 10 Jun 2011 05:13:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 05:13:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 05:13:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EA02210CC77
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 05:12:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 05:12:58 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1917018235.9641.1307682778955.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <308187203.8772.1307656679014.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7374) hadoop-config.sh shouldn't add
 tools.jar to the classpath
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7374:
--------------------------------

    Status: Patch Available  (was: Open)

> hadoop-config.sh shouldn't add tools.jar to the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-7374
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7374
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>         Attachments: hadoop-7374-1.patch
>
>
> bin/hadoop-config.sh (and bin/rcc) add lib/tools.jar from JAVA_HOME to the classpath. This has been there since the initial commit of bin/hadoop, but I don't think it's needed. *Executing* Hadoop does not depend on tools.jar (or other libraries only available in the JDK, not the JRE) so let's not automatically add it. Marking this as an incompatible change since a job could potentially have relied on Hadoop adding tools.jar to the CLASSPATH automatically (though such a job would not have run on a system that did not have JAVA_HOME point to a jdk). The build of course still requires a JDK.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16094-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 05:25:21 2011
Return-Path: <common-issues-return-16094-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CFDAC483B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 05:25:21 +0000 (UTC)
Received: (qmail 31946 invoked by uid 500); 10 Jun 2011 05:25:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31929 invoked by uid 500); 10 Jun 2011 05:25:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31921 invoked by uid 99); 10 Jun 2011 05:25:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 05:25:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 05:25:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D1BFC10CEBA
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 05:24:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 05:24:58 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <636610095.9647.1307683498855.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-1886) Undocumented parameters in
 FilesSystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-1886:
--------------------------------

    Attachment: hadoop-1886-2.patch

Right patch this time.

> Undocumented parameters in FilesSystem
> --------------------------------------
>
>                 Key: HADOOP-1886
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1886
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.14.0
>            Reporter: Konstantin Shvachko
>            Assignee: Frank Conrad
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: hadoop-1886-1.diff, hadoop-1886-2.patch, hadoop-1886-2.patch
>
>
> Multiple create methods in public FileSystem class lack documentation for the following 2 parameters.
> - long blockSize,
> - Progressable progress

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16095-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 05:27:21 2011
Return-Path: <common-issues-return-16095-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B359449B6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 05:27:21 +0000 (UTC)
Received: (qmail 37658 invoked by uid 500); 10 Jun 2011 05:27:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37533 invoked by uid 500); 10 Jun 2011 05:27:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37523 invoked by uid 99); 10 Jun 2011 05:27:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 05:27:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 05:27:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DCC2710CF7B
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 05:26:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 05:26:58 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <180140321.9651.1307683618900.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-1886) Undocumented parameters in
 FilesSystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047002#comment-13047002 ] 

Eli Collins commented on HADOOP-1886:
-------------------------------------

+1 to Frank's patch (w minor mods).

> Undocumented parameters in FilesSystem
> --------------------------------------
>
>                 Key: HADOOP-1886
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1886
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.14.0
>            Reporter: Konstantin Shvachko
>            Assignee: Frank Conrad
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: hadoop-1886-1.diff, hadoop-1886-2.patch, hadoop-1886-2.patch
>
>
> Multiple create methods in public FileSystem class lack documentation for the following 2 parameters.
> - long blockSize,
> - Progressable progress

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16096-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 05:35:20 2011
Return-Path: <common-issues-return-16096-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C69D24065
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 05:35:20 +0000 (UTC)
Received: (qmail 47359 invoked by uid 500); 10 Jun 2011 05:35:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47232 invoked by uid 500); 10 Jun 2011 05:35:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47219 invoked by uid 99); 10 Jun 2011 05:35:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 05:35:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 05:35:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C34F010C297
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 05:34:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 05:34:58 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2125273108.9689.1307684098796.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <308187203.8772.1307656679014.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7374) hadoop-config.sh shouldn't add
 tools.jar to the classpath
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047009#comment-13047009 ] 

Hadoop QA commented on HADOOP-7374:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481985/hadoop-7374-1.patch
  against trunk revision 1133125.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/605//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/605//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/605//console

This message is automatically generated.

> hadoop-config.sh shouldn't add tools.jar to the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-7374
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7374
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>         Attachments: hadoop-7374-1.patch
>
>
> bin/hadoop-config.sh (and bin/rcc) add lib/tools.jar from JAVA_HOME to the classpath. This has been there since the initial commit of bin/hadoop, but I don't think it's needed. *Executing* Hadoop does not depend on tools.jar (or other libraries only available in the JDK, not the JRE) so let's not automatically add it. Marking this as an incompatible change since a job could potentially have relied on Hadoop adding tools.jar to the CLASSPATH automatically (though such a job would not have run on a system that did not have JAVA_HOME point to a jdk). The build of course still requires a JDK.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16097-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 05:37:23 2011
Return-Path: <common-issues-return-16097-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B4FF540B0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 05:37:23 +0000 (UTC)
Received: (qmail 50097 invoked by uid 500); 10 Jun 2011 05:37:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50009 invoked by uid 500); 10 Jun 2011 05:37:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50001 invoked by uid 99); 10 Jun 2011 05:37:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 05:37:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 05:37:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EAB0B10C31F
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 05:36:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 05:36:58 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <449021261.9692.1307684218957.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1479142716.7833.1307643118867.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7372) Remove ref of 20.3 release from
 branch-0.20 CHANGES.txt
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047010#comment-13047010 ] 

Tom White commented on HADOOP-7372:
-----------------------------------

+1

> Remove ref of 20.3 release from branch-0.20 CHANGES.txt
> -------------------------------------------------------
>
>                 Key: HADOOP-7372
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7372
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.20.3
>
>         Attachments: hadoop-7372-1.patch
>
>
> CHANGES.txt on branch-0.20 claims there was a 0.20.3 release on 1/5. There has not been a 0.20.3 release.
> {noformat}
> Release 0.20.4 - Unreleased
> ...
> Release 0.20.3 - 2011-1-5
> {noformat}
> We should update this to indicate 0.20. is unreleased.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16098-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 05:41:22 2011
Return-Path: <common-issues-return-16098-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E4CAC4114
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 05:41:21 +0000 (UTC)
Received: (qmail 55502 invoked by uid 500); 10 Jun 2011 05:41:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55349 invoked by uid 500); 10 Jun 2011 05:41:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55334 invoked by uid 99); 10 Jun 2011 05:41:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 05:41:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 05:41:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C30B410C3CA
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 05:40:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 05:40:58 +0000 (UTC)
From: "XieXianshan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1395327642.9694.1307684458795.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1142137633.58455.1306902167529.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7348) Modify the option of FsShell
 getmerge from [addnl] to [-nl] for more comprehensive
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047011#comment-13047011 ] 

XieXianshan commented on HADOOP-7348:
-------------------------------------

Thanks for your comments,i`ll renew this patch.
But there`s no any auto-test for FsShell getmerge in hdfs.And there`s a note in the file testHDFSConf.xml:

<!-- Tests for getmerge -->
    <!-- Manual Testing -->

> Modify the option of FsShell getmerge from [addnl] to [-nl] for more comprehensive
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7348
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7348
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Assignee: XieXianshan
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7348.patch, HADOOP-7348.patch_2
>
>
> The [addnl] option of FsShell getmerge should be either "true" or "false",but it is very hard to understand by users, especially  who`s never used this option before. 
> So,the [addnl] option should be changed to [-nl] for more comprehensive.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16099-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 05:45:22 2011
Return-Path: <common-issues-return-16099-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F0FB444A1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 05:45:22 +0000 (UTC)
Received: (qmail 62916 invoked by uid 500); 10 Jun 2011 05:45:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62815 invoked by uid 500); 10 Jun 2011 05:45:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62795 invoked by uid 99); 10 Jun 2011 05:45:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 05:45:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 05:45:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F29C710C4E4
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 05:44:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 05:44:58 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1136068418.9701.1307684698990.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-1886) Undocumented parameters in
 FilesSystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047012#comment-13047012 ] 

Hadoop QA commented on HADOOP-1886:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482019/hadoop-1886-2.patch
  against trunk revision 1133125.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/606//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/606//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/606//console

This message is automatically generated.

> Undocumented parameters in FilesSystem
> --------------------------------------
>
>                 Key: HADOOP-1886
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1886
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.14.0
>            Reporter: Konstantin Shvachko
>            Assignee: Frank Conrad
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: hadoop-1886-1.diff, hadoop-1886-2.patch, hadoop-1886-2.patch
>
>
> Multiple create methods in public FileSystem class lack documentation for the following 2 parameters.
> - long blockSize,
> - Progressable progress

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16100-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 05:49:23 2011
Return-Path: <common-issues-return-16100-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CB6E24811
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 05:49:23 +0000 (UTC)
Received: (qmail 64182 invoked by uid 500); 10 Jun 2011 05:49:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64131 invoked by uid 500); 10 Jun 2011 05:49:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64114 invoked by uid 99); 10 Jun 2011 05:49:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 05:49:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 05:49:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E71EB10C5F2
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 05:48:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 05:48:58 +0000 (UTC)
From: "XieXianshan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <806461777.9709.1307684938943.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1142137633.58455.1306902167529.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7348) Modify the option of FsShell
 getmerge from [addnl] to [-nl] for more comprehensive
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

XieXianshan updated HADOOP-7348:
--------------------------------

    Attachment: HADOOP-7348.patch

> Modify the option of FsShell getmerge from [addnl] to [-nl] for more comprehensive
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7348
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7348
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Assignee: XieXianshan
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7348.patch, HADOOP-7348.patch, HADOOP-7348.patch_2
>
>
> The [addnl] option of FsShell getmerge should be either "true" or "false",but it is very hard to understand by users, especially  who`s never used this option before. 
> So,the [addnl] option should be changed to [-nl] for more comprehensive.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16101-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 05:56:20 2011
Return-Path: <common-issues-return-16101-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D1BD349F8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 05:56:20 +0000 (UTC)
Received: (qmail 75750 invoked by uid 500); 10 Jun 2011 05:56:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75638 invoked by uid 500); 10 Jun 2011 05:56:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75630 invoked by uid 99); 10 Jun 2011 05:56:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 05:56:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 05:56:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 06FD310C80B
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 05:55:59 +0000 (UTC)
Date: Fri, 10 Jun 2011 05:55:59 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1196129422.9720.1307685359025.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <634368540.9715.1307685358867.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7375) Add resolvePath method to
 FileContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7375:
---------------------------------

    Status: Patch Available  (was: Open)

> Add resolvePath method to FileContext
> -------------------------------------
>
>                 Key: HADOOP-7375
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7375
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: resolvePath1.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16102-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 05:56:22 2011
Return-Path: <common-issues-return-16102-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 568744A03
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 05:56:22 +0000 (UTC)
Received: (qmail 75947 invoked by uid 500); 10 Jun 2011 05:56:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75900 invoked by uid 500); 10 Jun 2011 05:56:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75892 invoked by uid 99); 10 Jun 2011 05:56:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 05:56:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 05:56:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id ED66510C809
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 05:55:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 05:55:58 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <122605905.9718.1307685358969.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <634368540.9715.1307685358867.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7375) Add resolvePath method to
 FileContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7375:
---------------------------------

    Attachment: resolvePath1.patch

> Add resolvePath method to FileContext
> -------------------------------------
>
>                 Key: HADOOP-7375
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7375
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: resolvePath1.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16103-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 05:56:22 2011
Return-Path: <common-issues-return-16103-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9347C4A0E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 05:56:22 +0000 (UTC)
Received: (qmail 76196 invoked by uid 500); 10 Jun 2011 05:56:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75998 invoked by uid 500); 10 Jun 2011 05:56:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75902 invoked by uid 99); 10 Jun 2011 05:56:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 05:56:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 05:56:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D491710C806
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 05:55:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 05:55:58 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <634368540.9715.1307685358867.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7375) Add resolvePath method to
 FileContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Add resolvePath method to FileContext
-------------------------------------

                 Key: HADOOP-7375
                 URL: https://issues.apache.org/jira/browse/HADOOP-7375
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Sanjay Radia
            Assignee: Sanjay Radia
             Fix For: 0.23.0
         Attachments: resolvePath1.patch



--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16104-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 06:04:21 2011
Return-Path: <common-issues-return-16104-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D5A0840F7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 06:04:21 +0000 (UTC)
Received: (qmail 88009 invoked by uid 500); 10 Jun 2011 06:04:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87909 invoked by uid 500); 10 Jun 2011 06:04:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87899 invoked by uid 99); 10 Jun 2011 06:04:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 06:04:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 06:04:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C28D710CA16
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 06:03:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 06:03:58 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2035878363.9743.1307685838793.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1142137633.58455.1306902167529.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7348) Modify the option of FsShell
 getmerge from [addnl] to [-nl] for more comprehensive
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047018#comment-13047018 ] 

Hadoop QA commented on HADOOP-7348:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482022/HADOOP-7348.patch
  against trunk revision 1133125.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 4 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/607//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/607//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/607//console

This message is automatically generated.

> Modify the option of FsShell getmerge from [addnl] to [-nl] for more comprehensive
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7348
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7348
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Assignee: XieXianshan
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7348.patch, HADOOP-7348.patch, HADOOP-7348.patch_2
>
>
> The [addnl] option of FsShell getmerge should be either "true" or "false",but it is very hard to understand by users, especially  who`s never used this option before. 
> So,the [addnl] option should be changed to [-nl] for more comprehensive.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16105-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 06:08:20 2011
Return-Path: <common-issues-return-16105-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8937D4288
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 06:08:20 +0000 (UTC)
Received: (qmail 90808 invoked by uid 500); 10 Jun 2011 06:08:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90751 invoked by uid 500); 10 Jun 2011 06:08:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90743 invoked by uid 99); 10 Jun 2011 06:08:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 06:08:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 06:08:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C066310CB35
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 06:07:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 06:07:58 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1484893605.9748.1307686078784.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <634368540.9715.1307685358867.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7375) Add resolvePath method to
 FileContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047020#comment-13047020 ] 

Eli Collins commented on HADOOP-7375:
-------------------------------------

AFS#getFileStatus is now called instead of FileContext#getFileStatus, which means fixRelativePart is no longer used to make the path absolute in FileContext relative to the working dir before passing the path to AFS right? 

Nit: lines 568 and 2231 need indenting.  Otherwise looks great.

> Add resolvePath method to FileContext
> -------------------------------------
>
>                 Key: HADOOP-7375
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7375
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: resolvePath1.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16106-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 06:10:22 2011
Return-Path: <common-issues-return-16106-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 99CF14327
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 06:10:22 +0000 (UTC)
Received: (qmail 92426 invoked by uid 500); 10 Jun 2011 06:10:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92375 invoked by uid 500); 10 Jun 2011 06:10:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92344 invoked by uid 99); 10 Jun 2011 06:10:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 06:10:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 06:10:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B819F10CC26
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 06:09:59 +0000 (UTC)
Date: Fri, 10 Jun 2011 06:09:59 +0000 (UTC)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <93089364.9770.1307686199750.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32237646.17061288907081267.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7020) establish a "Powered by Hadoop"
 logo
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047022#comment-13047022 ] 

Konstantin Boudnik commented on HADOOP-7020:
--------------------------------------------

Silver circle is very classy. I like it as well.

> establish a "Powered by Hadoop" logo
> ------------------------------------
>
>                 Key: HADOOP-7020
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7020
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: site
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: site
>
>         Attachments: PoweredByHadoop_Small.jpg, hadoop-elephant-pb.jpeg, powered-by-hadoop-small.png, powered-by-hadoop.png
>
>
> We should agree on a Powered By Hadoop logo, as suggested in:
> http://www.apache.org/foundation/marks/pmcs#poweredby

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16107-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 06:15:23 2011
Return-Path: <common-issues-return-16107-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0EB18460B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 06:15:23 +0000 (UTC)
Received: (qmail 97892 invoked by uid 500); 10 Jun 2011 06:15:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97746 invoked by uid 500); 10 Jun 2011 06:15:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97736 invoked by uid 99); 10 Jun 2011 06:15:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 06:15:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 06:15:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C2F8110CD38
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 06:14:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 06:14:58 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <98273032.9774.1307686498795.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <634368540.9715.1307685358867.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7375) Add resolvePath method to
 FileContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047024#comment-13047024 ] 

Hadoop QA commented on HADOOP-7375:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482023/resolvePath1.patch
  against trunk revision 1133125.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/608//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/608//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/608//console

This message is automatically generated.

> Add resolvePath method to FileContext
> -------------------------------------
>
>                 Key: HADOOP-7375
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7375
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: resolvePath1.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16108-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 06:25:22 2011
Return-Path: <common-issues-return-16108-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C7EDB4CC1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 06:25:22 +0000 (UTC)
Received: (qmail 3475 invoked by uid 500); 10 Jun 2011 06:25:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3431 invoked by uid 500); 10 Jun 2011 06:25:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3415 invoked by uid 99); 10 Jun 2011 06:25:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 06:25:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 06:25:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AD83AB7014
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 06:24:59 +0000 (UTC)
Date: Fri, 10 Jun 2011 06:24:59 +0000 (UTC)
From: "Ted Dunning (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <609792.9797.1307687099707.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32237646.17061288907081267.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7020) establish a "Powered by Hadoop"
 logo
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047026#comment-13047026 ] 

Ted Dunning commented on HADOOP-7020:
-------------------------------------

Based on the OOM and KB feedback, I think we will be moving forward with the circle logo.

We recognize that this is not yet an official logo, but we really need to be able to make the distinction.  We can update if there is consensus on another option.


> establish a "Powered by Hadoop" logo
> ------------------------------------
>
>                 Key: HADOOP-7020
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7020
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: site
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: site
>
>         Attachments: PoweredByHadoop_Small.jpg, hadoop-elephant-pb.jpeg, powered-by-hadoop-small.png, powered-by-hadoop.png
>
>
> We should agree on a Powered By Hadoop logo, as suggested in:
> http://www.apache.org/foundation/marks/pmcs#poweredby

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16109-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 07:01:26 2011
Return-Path: <common-issues-return-16109-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2A6284451
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 07:01:26 +0000 (UTC)
Received: (qmail 30829 invoked by uid 500); 10 Jun 2011 07:01:26 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30722 invoked by uid 500); 10 Jun 2011 07:01:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30384 invoked by uid 99); 10 Jun 2011 07:01:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 07:01:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 07:01:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 29E29B7F32
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 07:00:59 +0000 (UTC)
Date: Fri, 10 Jun 2011 07:00:59 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1266637257.9881.1307689259168.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1479142716.7833.1307643118867.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7372) Remove ref of 20.3 release from
 branch-0.20 CHANGES.txt
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047040#comment-13047040 ] 

Eli Collins commented on HADOOP-7372:
-------------------------------------

On 2nd thought, wouldn't it make sense to do this by reverting the "Preparing for release 0.20.3" change? Looks like the build.xml and releasenotes.html needs to be edited to.

> Remove ref of 20.3 release from branch-0.20 CHANGES.txt
> -------------------------------------------------------
>
>                 Key: HADOOP-7372
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7372
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.20.3
>
>         Attachments: hadoop-7372-1.patch
>
>
> CHANGES.txt on branch-0.20 claims there was a 0.20.3 release on 1/5. There has not been a 0.20.3 release.
> {noformat}
> Release 0.20.4 - Unreleased
> ...
> Release 0.20.3 - 2011-1-5
> {noformat}
> We should update this to indicate 0.20. is unreleased.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16110-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 07:13:21 2011
Return-Path: <common-issues-return-16110-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 56F2244E4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 07:13:21 +0000 (UTC)
Received: (qmail 47706 invoked by uid 500); 10 Jun 2011 07:13:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47569 invoked by uid 500); 10 Jun 2011 07:13:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47557 invoked by uid 99); 10 Jun 2011 07:13:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 07:13:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 07:13:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D1D2010C3DC
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 07:12:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 07:12:58 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1252719079.9904.1307689978856.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7376) DataTransfer Protocol using
 protobufs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

DataTransfer Protocol using protobufs
-------------------------------------

                 Key: HADOOP-7376
                 URL: https://issues.apache.org/jira/browse/HADOOP-7376
             Project: Hadoop Common
          Issue Type: Sub-task
    Affects Versions: 0.23.0
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon


We've been talking about this for a long time... would be nice to use something like protobufs or Thrift for some of our wire protocols.

I knocked together a prototype of DataTransferProtocol on top of proto bufs that seems to work.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16111-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 07:15:20 2011
Return-Path: <common-issues-return-16111-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3C4734606
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 07:15:20 +0000 (UTC)
Received: (qmail 51398 invoked by uid 500); 10 Jun 2011 07:15:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51371 invoked by uid 500); 10 Jun 2011 07:15:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51363 invoked by uid 99); 10 Jun 2011 07:15:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 07:15:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 07:15:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CB3DB10C5EE
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 07:14:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 07:14:58 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1986610418.9906.1307690098829.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1252719079.9904.1307689978856.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7376) DataTransfer Protocol using
 protobufs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7376:
--------------------------------

    Issue Type: New Feature  (was: Sub-task)
        Parent:     (was: HADOOP-7347)

> DataTransfer Protocol using protobufs
> -------------------------------------
>
>                 Key: HADOOP-7376
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7376
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.23.0
>
>
> We've been talking about this for a long time... would be nice to use something like protobufs or Thrift for some of our wire protocols.
> I knocked together a prototype of DataTransferProtocol on top of proto bufs that seems to work.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16112-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 07:45:20 2011
Return-Path: <common-issues-return-16112-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3BCAB4DEF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 07:45:20 +0000 (UTC)
Received: (qmail 97240 invoked by uid 500); 10 Jun 2011 07:45:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97212 invoked by uid 500); 10 Jun 2011 07:45:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97197 invoked by uid 99); 10 Jun 2011 07:45:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 07:45:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 07:45:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DB5DB10C1A5
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 07:44:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 07:44:58 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1698547793.10005.1307691898895.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-1886) Undocumented parameters in
 FilesSystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-1886:
--------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I've committed this. Thanks Frank!

> Undocumented parameters in FilesSystem
> --------------------------------------
>
>                 Key: HADOOP-1886
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1886
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.14.0
>            Reporter: Konstantin Shvachko
>            Assignee: Frank Conrad
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: hadoop-1886-1.diff, hadoop-1886-2.patch, hadoop-1886-2.patch
>
>
> Multiple create methods in public FileSystem class lack documentation for the following 2 parameters.
> - long blockSize,
> - Progressable progress

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16113-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 08:07:21 2011
Return-Path: <common-issues-return-16113-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5D98F4A0A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 08:07:21 +0000 (UTC)
Received: (qmail 19013 invoked by uid 500); 10 Jun 2011 08:07:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18980 invoked by uid 500); 10 Jun 2011 08:07:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18950 invoked by uid 99); 10 Jun 2011 08:07:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 08:07:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 08:07:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E344310C8DA
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 08:06:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 08:06:58 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <662453607.10028.1307693218927.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-1886) Undocumented parameters in
 FilesSystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047065#comment-13047065 ] 

Hudson commented on HADOOP-1886:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #646 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/646/])
    HADOOP-1886. Undocumented parameters in FilesSystem. Contributed by Frank Conrad

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1134218
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/FileSystem.java


> Undocumented parameters in FilesSystem
> --------------------------------------
>
>                 Key: HADOOP-1886
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1886
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.14.0
>            Reporter: Konstantin Shvachko
>            Assignee: Frank Conrad
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: hadoop-1886-1.diff, hadoop-1886-2.patch, hadoop-1886-2.patch
>
>
> Multiple create methods in public FileSystem class lack documentation for the following 2 parameters.
> - long blockSize,
> - Progressable progress

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16114-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 09:07:22 2011
Return-Path: <common-issues-return-16114-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 310B84E0F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 09:07:22 +0000 (UTC)
Received: (qmail 99713 invoked by uid 500); 10 Jun 2011 09:07:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99670 invoked by uid 500); 10 Jun 2011 09:07:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99662 invoked by uid 99); 10 Jun 2011 09:07:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 09:07:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 09:07:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F085A10CFAF
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 09:06:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 09:06:58 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1360911936.10121.1307696818981.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-1886) Undocumented parameters in
 FilesSystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047087#comment-13047087 ] 

Hudson commented on HADOOP-1886:
--------------------------------

Integrated in Hadoop-Common-trunk-maven #11 (See [https://builds.apache.org/job/Hadoop-Common-trunk-maven/11/])
    HADOOP-1886. Undocumented parameters in FilesSystem. Contributed by Frank Conrad

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1134218
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/FileSystem.java


> Undocumented parameters in FilesSystem
> --------------------------------------
>
>                 Key: HADOOP-1886
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1886
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.14.0
>            Reporter: Konstantin Shvachko
>            Assignee: Frank Conrad
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: hadoop-1886-1.diff, hadoop-1886-2.patch, hadoop-1886-2.patch
>
>
> Multiple create methods in public FileSystem class lack documentation for the following 2 parameters.
> - long blockSize,
> - Progressable progress

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16115-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 11:15:20 2011
Return-Path: <common-issues-return-16115-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 226424C72
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 11:15:20 +0000 (UTC)
Received: (qmail 16569 invoked by uid 500); 10 Jun 2011 11:15:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16542 invoked by uid 500); 10 Jun 2011 11:15:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16534 invoked by uid 99); 10 Jun 2011 11:15:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 11:15:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 11:15:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CAE1B10CE6C
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 11:14:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 11:14:58 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2087964163.10306.1307704498827.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-1886) Undocumented parameters in
 FilesSystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047139#comment-13047139 ] 

Hudson commented on HADOOP-1886:
--------------------------------

Integrated in Hadoop-Common-trunk #715 (See [https://builds.apache.org/job/Hadoop-Common-trunk/715/])
    HADOOP-1886. Undocumented parameters in FilesSystem. Contributed by Frank Conrad

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1134218
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/FileSystem.java


> Undocumented parameters in FilesSystem
> --------------------------------------
>
>                 Key: HADOOP-1886
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1886
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.14.0
>            Reporter: Konstantin Shvachko
>            Assignee: Frank Conrad
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: hadoop-1886-1.diff, hadoop-1886-2.patch, hadoop-1886-2.patch
>
>
> Multiple create methods in public FileSystem class lack documentation for the following 2 parameters.
> - long blockSize,
> - Progressable progress

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16116-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 13:47:20 2011
Return-Path: <common-issues-return-16116-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1918B4E69
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 13:47:20 +0000 (UTC)
Received: (qmail 75563 invoked by uid 500); 10 Jun 2011 13:47:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75533 invoked by uid 500); 10 Jun 2011 13:47:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75525 invoked by uid 99); 10 Jun 2011 13:47:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 13:47:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 13:47:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D9F2210D475
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 13:46:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 13:46:58 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1382834788.10633.1307713618889.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1142137633.58455.1306902167529.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7348) Modify the option of FsShell
 getmerge from [addnl] to [-nl] for more comprehensive
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047186#comment-13047186 ] 

Daryn Sharp commented on HADOOP-7348:
-------------------------------------

+1  Very nice!

It's too bad there aren't hdfs tests since it wouldn't be hard to do...  Would you mind filing a jira for the need to add tests so we don't forget it needs to be done?

> Modify the option of FsShell getmerge from [addnl] to [-nl] for more comprehensive
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7348
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7348
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Assignee: XieXianshan
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7348.patch, HADOOP-7348.patch, HADOOP-7348.patch_2
>
>
> The [addnl] option of FsShell getmerge should be either "true" or "false",but it is very hard to understand by users, especially  who`s never used this option before. 
> So,the [addnl] option should be changed to [-nl] for more comprehensive.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16117-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 16:48:23 2011
Return-Path: <common-issues-return-16117-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 343F94926
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 16:48:23 +0000 (UTC)
Received: (qmail 64159 invoked by uid 500); 10 Jun 2011 16:48:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64115 invoked by uid 500); 10 Jun 2011 16:48:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64106 invoked by uid 99); 10 Jun 2011 16:48:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 16:48:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 16:48:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9214F10D662
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 16:47:59 +0000 (UTC)
Date: Fri, 10 Jun 2011 16:47:59 +0000 (UTC)
From: "Lohit Vijayarenu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1705211241.11147.1307724479595.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32237646.17061288907081267.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7020) establish a "Powered by Hadoop"
 logo
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047283#comment-13047283 ] 

Lohit Vijayarenu commented on HADOOP-7020:
------------------------------------------

Silver circle logo is very nice. Would love to have it as laptop sticker or t-shirt logo :)

> establish a "Powered by Hadoop" logo
> ------------------------------------
>
>                 Key: HADOOP-7020
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7020
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: site
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: site
>
>         Attachments: PoweredByHadoop_Small.jpg, hadoop-elephant-pb.jpeg, powered-by-hadoop-small.png, powered-by-hadoop.png
>
>
> We should agree on a Powered By Hadoop logo, as suggested in:
> http://www.apache.org/foundation/marks/pmcs#poweredby

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16118-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 17:18:21 2011
Return-Path: <common-issues-return-16118-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0482B4E08
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 17:18:21 +0000 (UTC)
Received: (qmail 23382 invoked by uid 500); 10 Jun 2011 17:18:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23334 invoked by uid 500); 10 Jun 2011 17:18:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23189 invoked by uid 99); 10 Jun 2011 17:18:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 17:18:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 17:18:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3E06810DFD7
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 17:17:59 +0000 (UTC)
Date: Fri, 10 Jun 2011 17:17:59 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1451428120.11231.1307726279250.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1269276399.4788.1307568238896.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7368) Mechanism for providing version
 info of hadoop jars
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047301#comment-13047301 ] 

Alejandro Abdelnur commented on HADOOP-7368:
--------------------------------------------

IMO, the current mechanism brings obscurity/complexity to the build process.

And if you open a JAR, replace a file, and reJAR, then you know you are doing something funny. (And you could do that with the annotation class as well)



> Mechanism for providing version info of hadoop jars
> ---------------------------------------------------
>
>                 Key: HADOOP-7368
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7368
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.23.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>
> In 0.20.x, only one jar (hadoop-core-*.jar) really matters. The o.a.h.util.VersionInfo combined with saveVersion.sh script (generating a package level annotation) served us well. For 0.23+, due to the project split and modularization of mapreduce (currently in MR-279), a lot more essential hadoop jars are created. The potential of mixing up the jars is significantly increased as well. We need a simple way to list the version info (version, branch, source checksum etc.) for all the jars involved. This is essential for QE and tracking down various issues.
> I propose that we use a VersionProvider interface (similar to the current VersionInfo util class) and ServiceLoader to enumerate the version providers in the jars.
> {code}
> public interface VersionProvider {
>   String getJar(); 
>   String getPackage();
>   String getVersion();
>   String getRevision();
>   String getBranch();
>   String getDate();
>   String getUrl();
>   String getSourceChecksum();
> }
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16119-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 17:34:20 2011
Return-Path: <common-issues-return-16119-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 52E844107
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 17:34:20 +0000 (UTC)
Received: (qmail 46052 invoked by uid 500); 10 Jun 2011 17:34:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46021 invoked by uid 500); 10 Jun 2011 17:34:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46012 invoked by uid 99); 10 Jun 2011 17:34:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 17:34:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 17:34:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 27A4510D624
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 17:33:59 +0000 (UTC)
Date: Fri, 10 Jun 2011 17:33:59 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1510144945.11296.1307727239159.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <714460080.1372.1307482018863.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7365) Clean up CHANGES.txt
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7365:
----------------------------------

    Attachment: changes.patch

This fixes up some more of the wrong jiras that showed up when i looked for duplicates.

> Clean up CHANGES.txt 
> ---------------------
>
>                 Key: HADOOP-7365
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7365
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: c.patch, c.patch, changes.patch
>
>
> CHANGES.txt has some typos and standardizing the formatting.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16120-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 17:48:21 2011
Return-Path: <common-issues-return-16120-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0CC204B80
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 17:48:21 +0000 (UTC)
Received: (qmail 65055 invoked by uid 500); 10 Jun 2011 17:48:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65012 invoked by uid 500); 10 Jun 2011 17:48:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64977 invoked by uid 99); 10 Jun 2011 17:48:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 17:48:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 17:48:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C673510DC72
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 17:47:59 +0000 (UTC)
Date: Fri, 10 Jun 2011 17:47:59 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1632804688.11352.1307728079809.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <634368540.9715.1307685358867.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7375) Add resolvePath method to
 FileContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047321#comment-13047321 ] 

Suresh Srinivas commented on HADOOP-7375:
-----------------------------------------

+1 with Eli's comments addressed.

> Add resolvePath method to FileContext
> -------------------------------------
>
>                 Key: HADOOP-7375
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7375
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: resolvePath1.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16121-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 17:54:23 2011
Return-Path: <common-issues-return-16121-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F17BE4E34
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 17:54:22 +0000 (UTC)
Received: (qmail 82808 invoked by uid 500); 10 Jun 2011 17:54:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82769 invoked by uid 500); 10 Jun 2011 17:54:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82760 invoked by uid 99); 10 Jun 2011 17:54:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 17:54:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 17:54:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7F23E10D00B
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 17:53:59 +0000 (UTC)
Date: Fri, 10 Jun 2011 17:53:59 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1072202577.11404.1307728439517.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <714460080.1372.1307482018863.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7365) Clean up CHANGES.txt
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047330#comment-13047330 ] 

Hadoop QA commented on HADOOP-7365:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482079/changes.patch
  against trunk revision 1134218.

    -1 @author.  The patch appears to contain 2 @author tags which the Hadoop community has agreed to not allow in code contributions.

    +1 tests included.  The patch appears to include 2 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/609//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/609//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/609//console

This message is automatically generated.

> Clean up CHANGES.txt 
> ---------------------
>
>                 Key: HADOOP-7365
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7365
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: c.patch, c.patch, changes.patch
>
>
> CHANGES.txt has some typos and standardizing the formatting.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16122-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 18:39:23 2011
Return-Path: <common-issues-return-16122-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A05294705
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 18:39:23 +0000 (UTC)
Received: (qmail 79974 invoked by uid 500); 10 Jun 2011 18:39:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79924 invoked by uid 500); 10 Jun 2011 18:39:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79814 invoked by uid 99); 10 Jun 2011 18:39:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 18:39:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 18:39:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E169D10D35F
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 18:38:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 18:38:58 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1196787142.11570.1307731138920.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2129468871.8460.1307649778954.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7373) Tarball deployment doesn't work
 with {start,stop}-{dfs,mapred}
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley resolved HADOOP-7373.
-----------------------------------

       Resolution: Fixed
    Fix Version/s: 0.20.204.0

It looks like trunk didn't have the problem.

> Tarball deployment doesn't work with {start,stop}-{dfs,mapred}
> --------------------------------------------------------------
>
>                 Key: HADOOP-7373
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7373
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.204.0
>
>         Attachments: c.patch
>
>
> The hadoop-config.sh overrides the variable "bin", which makes the scripts use libexec for hadoop-daemon(s).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16123-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 19:37:22 2011
Return-Path: <common-issues-return-16123-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 418B64FAC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 19:37:22 +0000 (UTC)
Received: (qmail 81199 invoked by uid 500); 10 Jun 2011 19:37:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81126 invoked by uid 500); 10 Jun 2011 19:37:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80966 invoked by uid 99); 10 Jun 2011 19:37:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 19:37:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 19:37:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3AD4510D6D8
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 19:36:59 +0000 (UTC)
Date: Fri, 10 Jun 2011 19:36:59 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <151121036.11705.1307734619237.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7377) Fix command name handling affecting
 DFSAdmin
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Fix command name handling affecting DFSAdmin
--------------------------------------------

                 Key: HADOOP-7377
                 URL: https://issues.apache.org/jira/browse/HADOOP-7377
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
    Affects Versions: 0.23.0
            Reporter: Daryn Sharp
            Assignee: Daryn Sharp


When an error occurs in the get/set quota commands in DFSAdmin, they are displaying the following:
setQuota: failed to get SetQuotaCommand.NAME

The {{Command}} class expects the {{NAME}} field to be accessible, but for DFSAdmin, it's not.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16124-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 19:45:21 2011
Return-Path: <common-issues-return-16124-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0D7D6454D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 19:45:21 +0000 (UTC)
Received: (qmail 5553 invoked by uid 500); 10 Jun 2011 19:45:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5509 invoked by uid 500); 10 Jun 2011 19:45:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5501 invoked by uid 99); 10 Jun 2011 19:45:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 19:45:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 19:45:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D58A010D99E
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 19:44:59 +0000 (UTC)
Date: Fri, 10 Jun 2011 19:44:59 +0000 (UTC)
From: "Ian Holsman (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <934434933.11751.1307735099871.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7106) Re-organize hadoop subversion
 layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047394#comment-13047394 ] 

Ian Holsman commented on HADOOP-7106:
-------------------------------------

the asf-authorization patch was committed in 790732.


> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16125-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 19:45:23 2011
Return-Path: <common-issues-return-16125-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 441F74596
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 19:45:23 +0000 (UTC)
Received: (qmail 5844 invoked by uid 500); 10 Jun 2011 19:45:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5805 invoked by uid 500); 10 Jun 2011 19:45:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5797 invoked by uid 99); 10 Jun 2011 19:45:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 19:45:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 19:45:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E8F0210D9A1
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 19:44:59 +0000 (UTC)
Date: Fri, 10 Jun 2011 19:44:59 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <538004229.11753.1307735099951.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32237646.17061288907081267.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7020) establish a "Powered by Hadoop"
 logo
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047395#comment-13047395 ] 

Allen Wittenauer commented on HADOOP-7020:
------------------------------------------

Is the artistic intent to suggest a mouse/gerbil/small rodent wheel?

> establish a "Powered by Hadoop" logo
> ------------------------------------
>
>                 Key: HADOOP-7020
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7020
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: site
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: site
>
>         Attachments: PoweredByHadoop_Small.jpg, hadoop-elephant-pb.jpeg, powered-by-hadoop-small.png, powered-by-hadoop.png
>
>
> We should agree on a Powered By Hadoop logo, as suggested in:
> http://www.apache.org/foundation/marks/pmcs#poweredby

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16126-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 19:55:24 2011
Return-Path: <common-issues-return-16126-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 412DD4C02
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 19:55:24 +0000 (UTC)
Received: (qmail 25928 invoked by uid 500); 10 Jun 2011 19:55:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25869 invoked by uid 500); 10 Jun 2011 19:55:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25796 invoked by uid 99); 10 Jun 2011 19:55:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 19:55:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 19:55:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EEF0210DDCF
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 19:54:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 19:54:58 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <301573562.11798.1307735698975.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1415670632.7518.1307638679263.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7371) Improve tarball distributions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047402#comment-13047402 ] 

Alejandro Abdelnur commented on HADOOP-7371:
--------------------------------------------

Eric, the source TAR, should be the source ready to build, or ...?


> Improve tarball distributions
> -----------------------------
>
>                 Key: HADOOP-7371
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7371
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>         Environment: Java 6, Redhat 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>             Fix For: 0.20.205.0
>
>
> Hadoop release tarball contains both raw source and binary.  This leads users to use the release tarball as base for applying patches, to build custom Hadoop.  This is not the recommended method to develop hadoop because it leads to mixed development system where processed files and raw source are hard to separate.  
> To correct the problematic usage of the release tarball, the release build target should be defined as:
> "ant source" generates source release tarball.
> "ant binary" is binary release without source/javadoc jar files.
> "ant tar" is a mirror of binary release with source/javadoc jar files.
> Does this sound reasonable?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16127-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 20:04:24 2011
Return-Path: <common-issues-return-16127-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EE5214581
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 20:04:23 +0000 (UTC)
Received: (qmail 49371 invoked by uid 500); 10 Jun 2011 20:04:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49330 invoked by uid 500); 10 Jun 2011 20:04:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49318 invoked by uid 99); 10 Jun 2011 20:04:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 20:04:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 20:04:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B48E810D24D
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 20:03:59 +0000 (UTC)
Date: Fri, 10 Jun 2011 20:03:59 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1084885206.11860.1307736239736.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <151121036.11705.1307734619237.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7377) Fix command name handling affecting
 DFSAdmin
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7377:
--------------------------------

    Attachment: HADOOP-7377.patch

Quick cheap way to get the NAME field despite the visibility of the class and/or field.

> Fix command name handling affecting DFSAdmin
> --------------------------------------------
>
>                 Key: HADOOP-7377
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7377
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7377.patch
>
>
> When an error occurs in the get/set quota commands in DFSAdmin, they are displaying the following:
> setQuota: failed to get SetQuotaCommand.NAME
> The {{Command}} class expects the {{NAME}} field to be accessible, but for DFSAdmin, it's not.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16128-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 20:06:20 2011
Return-Path: <common-issues-return-16128-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5342A4685
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 20:06:20 +0000 (UTC)
Received: (qmail 54490 invoked by uid 500); 10 Jun 2011 20:06:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54464 invoked by uid 500); 10 Jun 2011 20:06:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54456 invoked by uid 99); 10 Jun 2011 20:06:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 20:06:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 20:06:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D354810D357
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 20:05:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 20:05:58 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1021369500.11862.1307736358852.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <151121036.11705.1307734619237.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7377) Fix command name handling affecting
 DFSAdmin
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7377:
--------------------------------

    Status: Patch Available  (was: Open)

No tests included because it fixes tests in hdfs.

> Fix command name handling affecting DFSAdmin
> --------------------------------------------
>
>                 Key: HADOOP-7377
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7377
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7377.patch
>
>
> When an error occurs in the get/set quota commands in DFSAdmin, they are displaying the following:
> setQuota: failed to get SetQuotaCommand.NAME
> The {{Command}} class expects the {{NAME}} field to be accessible, but for DFSAdmin, it's not.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16129-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 20:19:22 2011
Return-Path: <common-issues-return-16129-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2AD3F4E31
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 20:19:22 +0000 (UTC)
Received: (qmail 83208 invoked by uid 500); 10 Jun 2011 20:19:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83176 invoked by uid 500); 10 Jun 2011 20:19:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83163 invoked by uid 99); 10 Jun 2011 20:19:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 20:19:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 20:19:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CD60510D8A9
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 20:18:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 20:18:58 +0000 (UTC)
From: "Ted Dunning (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2088815756.11935.1307737138837.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32237646.17061288907081267.JavaMail.jira@thor>
Subject: [jira] [Updated] (HADOOP-7020) establish a "Powered by Hadoop" logo
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ted Dunning updated HADOOP-7020:
--------------------------------

    Attachment: PoweredByApacheHadoop.png

Here is a version with "Powered by Apache Hadoop".  That fits surprisingly well.

Again, it would be good to get some clarity here.

> establish a "Powered by Hadoop" logo
> ------------------------------------
>
>                 Key: HADOOP-7020
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7020
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: site
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: site
>
>         Attachments: PoweredByApacheHadoop.png, PoweredByHadoop_Small.jpg, hadoop-elephant-pb.jpeg, powered-by-hadoop-small.png, powered-by-hadoop.png
>
>
> We should agree on a Powered By Hadoop logo, as suggested in:
> http://www.apache.org/foundation/marks/pmcs#poweredby

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16130-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 20:25:20 2011
Return-Path: <common-issues-return-16130-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4538042EC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 20:25:20 +0000 (UTC)
Received: (qmail 91954 invoked by uid 500); 10 Jun 2011 20:25:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91879 invoked by uid 500); 10 Jun 2011 20:25:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91865 invoked by uid 99); 10 Jun 2011 20:25:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 20:25:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 20:25:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DADC110DA53
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 20:24:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 20:24:58 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1258610207.11959.1307737498893.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <151121036.11705.1307734619237.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7377) Fix command name handling
 affecting DFSAdmin
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047426#comment-13047426 ] 

Hadoop QA commented on HADOOP-7377:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482092/HADOOP-7377.patch
  against trunk revision 1134218.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/610//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/610//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/610//console

This message is automatically generated.

> Fix command name handling affecting DFSAdmin
> --------------------------------------------
>
>                 Key: HADOOP-7377
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7377
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7377.patch
>
>
> When an error occurs in the get/set quota commands in DFSAdmin, they are displaying the following:
> setQuota: failed to get SetQuotaCommand.NAME
> The {{Command}} class expects the {{NAME}} field to be accessible, but for DFSAdmin, it's not.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16131-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 20:25:23 2011
Return-Path: <common-issues-return-16131-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EEECE4322
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 20:25:22 +0000 (UTC)
Received: (qmail 92309 invoked by uid 500); 10 Jun 2011 20:25:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92274 invoked by uid 500); 10 Jun 2011 20:25:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92266 invoked by uid 99); 10 Jun 2011 20:25:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 20:25:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 20:25:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B2FB110DA76
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 20:24:59 +0000 (UTC)
Date: Fri, 10 Jun 2011 20:24:59 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <414687843.11984.1307737499729.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047427#comment-13047427 ] 

Tom White commented on HADOOP-6671:
-----------------------------------

A couple of other things I noticed:

We need to use a custom doclet for Javadoc (to exclude private classes). Something like

{code}
<doclet>org.apache.hadoop.classification.tools.ExcludePrivateAnnotationsStandardDoclet</doclet>
<docletArtifact>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-common</artifactId>
  <version>0.23.0-SNAPSHOT</version>
</docletArtifact>
<useStandardDocletOptions>true</useStandardDocletOptions>
{code}

The releaseaudit target equivalent is needed in Maven.




> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-6671-cross-project-HDFS.patch, HADOOP-6671-e.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16132-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 20:27:23 2011
Return-Path: <common-issues-return-16132-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 88314441B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 20:27:23 +0000 (UTC)
Received: (qmail 94671 invoked by uid 500); 10 Jun 2011 20:27:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94473 invoked by uid 500); 10 Jun 2011 20:27:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94450 invoked by uid 99); 10 Jun 2011 20:27:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 20:27:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 20:27:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1398710DBE1
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 20:27:00 +0000 (UTC)
Date: Fri, 10 Jun 2011 20:27:00 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <24909417.11996.1307737620076.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1415670632.7518.1307638679263.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7371) Improve tarball distributions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047428#comment-13047428 ] 

Eric Yang commented on HADOOP-7371:
-----------------------------------

Alejandro, source tarball should be the source ready to build without hadoop jar files.

> Improve tarball distributions
> -----------------------------
>
>                 Key: HADOOP-7371
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7371
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>         Environment: Java 6, Redhat 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>             Fix For: 0.20.205.0
>
>
> Hadoop release tarball contains both raw source and binary.  This leads users to use the release tarball as base for applying patches, to build custom Hadoop.  This is not the recommended method to develop hadoop because it leads to mixed development system where processed files and raw source are hard to separate.  
> To correct the problematic usage of the release tarball, the release build target should be defined as:
> "ant source" generates source release tarball.
> "ant binary" is binary release without source/javadoc jar files.
> "ant tar" is a mirror of binary release with source/javadoc jar files.
> Does this sound reasonable?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16133-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 20:35:22 2011
Return-Path: <common-issues-return-16133-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 265034B83
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 20:35:22 +0000 (UTC)
Received: (qmail 7468 invoked by uid 500); 10 Jun 2011 20:35:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7435 invoked by uid 500); 10 Jun 2011 20:35:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7422 invoked by uid 99); 10 Jun 2011 20:35:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 20:35:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 20:35:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D5DF510DED6
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 20:34:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 20:34:58 +0000 (UTC)
From: "Ted Dunning (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <669595667.12010.1307738098872.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32237646.17061288907081267.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7020) establish a "Powered by Hadoop"
 logo
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047430#comment-13047430 ] 

Ted Dunning commented on HADOOP-7020:
-------------------------------------

Allen,

I think that the artistic intent was just tie the words and elephant together.

> establish a "Powered by Hadoop" logo
> ------------------------------------
>
>                 Key: HADOOP-7020
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7020
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: site
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: site
>
>         Attachments: PoweredByApacheHadoop.png, PoweredByHadoop_Small.jpg, hadoop-elephant-pb.jpeg, powered-by-hadoop-small.png, powered-by-hadoop.png
>
>
> We should agree on a Powered By Hadoop logo, as suggested in:
> http://www.apache.org/foundation/marks/pmcs#poweredby

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16134-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 21:10:20 2011
Return-Path: <common-issues-return-16134-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0B1A3456C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 21:10:20 +0000 (UTC)
Received: (qmail 83544 invoked by uid 500); 10 Jun 2011 21:10:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83509 invoked by uid 500); 10 Jun 2011 21:10:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83501 invoked by uid 99); 10 Jun 2011 21:10:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 21:10:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 21:10:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CBEF610DE67
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 21:09:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 21:09:58 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2009244920.12207.1307740198831.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <714460080.1372.1307482018863.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7365) Clean up CHANGES.txt
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047457#comment-13047457 ] 

Suresh Srinivas commented on HADOOP-7365:
-----------------------------------------

+1 for the change. Could you please add some unit tests ;-)

> Clean up CHANGES.txt 
> ---------------------
>
>                 Key: HADOOP-7365
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7365
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: c.patch, c.patch, changes.patch
>
>
> CHANGES.txt has some typos and standardizing the formatting.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16135-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 21:20:20 2011
Return-Path: <common-issues-return-16135-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9C8444DD5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 21:20:20 +0000 (UTC)
Received: (qmail 838 invoked by uid 500); 10 Jun 2011 21:20:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 810 invoked by uid 500); 10 Jun 2011 21:20:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 799 invoked by uid 99); 10 Jun 2011 21:20:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 21:20:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 21:20:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C831C10D1E4
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 21:19:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 21:19:58 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1294483310.12227.1307740798816.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <714460080.1372.1307482018863.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7365) Clean up CHANGES.txt
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047461#comment-13047461 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7365:
------------------------------------------------


> ...  Could you please add some unit tests ...

There are already 2 new or modified tests.  :)

bq. +1 tests included. The patch appears to include 2 new or modified tests.

> Clean up CHANGES.txt 
> ---------------------
>
>                 Key: HADOOP-7365
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7365
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: c.patch, c.patch, changes.patch
>
>
> CHANGES.txt has some typos and standardizing the formatting.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16136-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 22:22:20 2011
Return-Path: <common-issues-return-16136-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6F6534B1F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 22:22:20 +0000 (UTC)
Received: (qmail 2384 invoked by uid 500); 10 Jun 2011 22:22:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2355 invoked by uid 500); 10 Jun 2011 22:22:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2347 invoked by uid 99); 10 Jun 2011 22:22:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 22:22:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 22:22:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1EBAC10DAE5
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 22:21:59 +0000 (UTC)
Date: Fri, 10 Jun 2011 22:21:59 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1780011712.12448.1307744519122.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7378) hadoop dfs -ls : Do not expand
 directories (was HDFS-1475)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

hadoop dfs -ls : Do not expand directories (was HDFS-1475)
----------------------------------------------------------

                 Key: HADOOP-7378
                 URL: https://issues.apache.org/jira/browse/HADOOP-7378
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs
    Affects Versions: 0.23.0
            Reporter: Daryn Sharp
            Assignee: Daryn Sharp


In a nutshell, ls needs the ability to list a directory but not its contents.  W/o -d, it is impossible to list the root directory's owner, permissions, etc.  See the original hdfs bug for details.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16137-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 22:35:21 2011
Return-Path: <common-issues-return-16137-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F0D24428F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 22:35:21 +0000 (UTC)
Received: (qmail 17009 invoked by uid 500); 10 Jun 2011 22:35:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16970 invoked by uid 500); 10 Jun 2011 22:35:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16962 invoked by uid 99); 10 Jun 2011 22:35:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 22:35:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 22:35:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C9DAE10DE55
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 22:34:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 22:34:58 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1986319569.12481.1307745298823.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1780011712.12448.1307744519122.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7378) hadoop dfs -ls : Do not expand
 directories (was HDFS-1475)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7378:
--------------------------------

    Attachment: HADOOP-7378.patch

> hadoop dfs -ls : Do not expand directories (was HDFS-1475)
> ----------------------------------------------------------
>
>                 Key: HADOOP-7378
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7378
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7378.patch
>
>
> In a nutshell, ls needs the ability to list a directory but not its contents.  W/o -d, it is impossible to list the root directory's owner, permissions, etc.  See the original hdfs bug for details.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16138-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 22:37:20 2011
Return-Path: <common-issues-return-16138-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 26B3D4423
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 22:37:20 +0000 (UTC)
Received: (qmail 18998 invoked by uid 500); 10 Jun 2011 22:37:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18957 invoked by uid 500); 10 Jun 2011 22:37:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18949 invoked by uid 99); 10 Jun 2011 22:37:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 22:37:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 22:37:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0080410DFC4
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 22:36:59 +0000 (UTC)
Date: Fri, 10 Jun 2011 22:36:58 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <75615708.12485.1307745418998.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1780011712.12448.1307744519122.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7378) hadoop dfs -ls : Do not expand
 directories (was HDFS-1475)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7378:
--------------------------------

    Status: Patch Available  (was: Open)

> hadoop dfs -ls : Do not expand directories (was HDFS-1475)
> ----------------------------------------------------------
>
>                 Key: HADOOP-7378
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7378
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7378.patch
>
>
> In a nutshell, ls needs the ability to list a directory but not its contents.  W/o -d, it is impossible to list the root directory's owner, permissions, etc.  See the original hdfs bug for details.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16139-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 22:55:22 2011
Return-Path: <common-issues-return-16139-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 459804D40
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 22:55:22 +0000 (UTC)
Received: (qmail 32927 invoked by uid 500); 10 Jun 2011 22:55:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32897 invoked by uid 500); 10 Jun 2011 22:55:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32888 invoked by uid 99); 10 Jun 2011 22:55:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 22:55:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 22:55:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CD7B010E289
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 22:54:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 22:54:58 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <555352666.12625.1307746498838.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1780011712.12448.1307744519122.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7378) hadoop dfs -ls : Do not expand
 directories (was HDFS-1475)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047643#comment-13047643 ] 

Hadoop QA commented on HADOOP-7378:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482108/HADOOP-7378.patch
  against trunk revision 1134218.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 4 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/611//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/611//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/611//console

This message is automatically generated.

> hadoop dfs -ls : Do not expand directories (was HDFS-1475)
> ----------------------------------------------------------
>
>                 Key: HADOOP-7378
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7378
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7378.patch
>
>
> In a nutshell, ls needs the ability to list a directory but not its contents.  W/o -d, it is impossible to list the root directory's owner, permissions, etc.  See the original hdfs bug for details.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16140-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 10 23:15:21 2011
Return-Path: <common-issues-return-16140-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 689754778
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 10 Jun 2011 23:15:21 +0000 (UTC)
Received: (qmail 55218 invoked by uid 500); 10 Jun 2011 23:15:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55187 invoked by uid 500); 10 Jun 2011 23:15:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55175 invoked by uid 99); 10 Jun 2011 23:15:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 23:15:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Jun 2011 23:15:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D0D4710E406
	for <common-issues@hadoop.apache.org>; Fri, 10 Jun 2011 23:14:58 +0000 (UTC)
Date: Fri, 10 Jun 2011 23:14:58 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <55510245.12688.1307747698852.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7379) Add ability to include Protobufs in
 ObjectWritable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Add ability to include Protobufs in ObjectWritable
--------------------------------------------------

                 Key: HADOOP-7379
                 URL: https://issues.apache.org/jira/browse/HADOOP-7379
             Project: Hadoop Common
          Issue Type: Improvement
          Components: io, ipc
    Affects Versions: 0.23.0
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
             Fix For: 0.23.0


Per HDFS-2060, it would make it easier to piecemeal switch to protocol buffer based data structures in the wire protocol if we could intermix the two. The IPC framework currently provides the concept of "engines" for RPC, but that doesn't easily allow mixed types within the same framework for ease of transition.

I'd like to add the cases to ObjectWritable to be handle subclasses of {{Message}}, the superclass of codegenned protobufs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16141-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 11 00:21:23 2011
Return-Path: <common-issues-return-16141-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D57544564
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 11 Jun 2011 00:21:23 +0000 (UTC)
Received: (qmail 20589 invoked by uid 500); 11 Jun 2011 00:21:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20542 invoked by uid 500); 11 Jun 2011 00:21:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20527 invoked by uid 99); 11 Jun 2011 00:21:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 00:21:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 00:21:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7866210E458
	for <common-issues@hadoop.apache.org>; Sat, 11 Jun 2011 00:21:00 +0000 (UTC)
Date: Sat, 11 Jun 2011 00:21:00 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2094690652.12980.1307751660489.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32237646.17061288907081267.JavaMail.jira@thor>
Subject: [jira] [Updated] (HADOOP-7020) establish a "Powered by Hadoop" logo
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7020:
--------------------------------

    Attachment: powered_by_hadoop.jpg

Here's a draft from Alison Wong who designed the Apache Whirr logo. It needs to be updated to say "Apache Hadoop" and perhaps make the elephant yellow. If people like the overall look she'll iterate on the design.

> establish a "Powered by Hadoop" logo
> ------------------------------------
>
>                 Key: HADOOP-7020
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7020
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: site
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: site
>
>         Attachments: PoweredByApacheHadoop.png, PoweredByHadoop_Small.jpg, hadoop-elephant-pb.jpeg, powered-by-hadoop-small.png, powered-by-hadoop.png, powered_by_hadoop.jpg
>
>
> We should agree on a Powered By Hadoop logo, as suggested in:
> http://www.apache.org/foundation/marks/pmcs#poweredby

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16142-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 11 00:21:24 2011
Return-Path: <common-issues-return-16142-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0684A456B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 11 Jun 2011 00:21:24 +0000 (UTC)
Received: (qmail 20775 invoked by uid 500); 11 Jun 2011 00:21:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20545 invoked by uid 500); 11 Jun 2011 00:21:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20528 invoked by uid 99); 11 Jun 2011 00:21:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 00:21:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 00:21:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 82CD210E459
	for <common-issues@hadoop.apache.org>; Sat, 11 Jun 2011 00:21:00 +0000 (UTC)
Date: Sat, 11 Jun 2011 00:21:00 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1766362893.12981.1307751660532.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32237646.17061288907081267.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7020) establish a "Powered by Hadoop"
 logo
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047799#comment-13047799 ] 

Eli Collins commented on HADOOP-7020:
-------------------------------------

File is powered_by_hadoop.jpg

> establish a "Powered by Hadoop" logo
> ------------------------------------
>
>                 Key: HADOOP-7020
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7020
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: site
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: site
>
>         Attachments: PoweredByApacheHadoop.png, PoweredByHadoop_Small.jpg, hadoop-elephant-pb.jpeg, powered-by-hadoop-small.png, powered-by-hadoop.png, powered_by_hadoop.jpg
>
>
> We should agree on a Powered By Hadoop logo, as suggested in:
> http://www.apache.org/foundation/marks/pmcs#poweredby

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16143-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 11 00:31:20 2011
Return-Path: <common-issues-return-16143-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 571D64110
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 11 Jun 2011 00:31:20 +0000 (UTC)
Received: (qmail 30401 invoked by uid 500); 11 Jun 2011 00:31:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30375 invoked by uid 500); 11 Jun 2011 00:31:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30367 invoked by uid 99); 11 Jun 2011 00:31:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 00:31:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 00:31:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D1D4610E65B
	for <common-issues@hadoop.apache.org>; Sat, 11 Jun 2011 00:30:58 +0000 (UTC)
Date: Sat, 11 Jun 2011 00:30:58 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1903769667.13020.1307752258856.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <55510245.12688.1307747698852.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7379) Add ability to include Protobufs in
 ObjectWritable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7379:
--------------------------------

    Attachment: hadoop-7379.txt

Here's a patch which does the following:
- protobufs can be serialized by ObjectWritable. They are length-prefixed with a varint and then written on the wire
- adds test for the writable format itself
- adds a call in TestIPC that uses a protobuf as a parameter and response

I see one spot for an optimization by removing a buffer copy on the read side, but I'd like to leave that for later.

> Add ability to include Protobufs in ObjectWritable
> --------------------------------------------------
>
>                 Key: HADOOP-7379
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7379
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, ipc
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7379.txt
>
>
> Per HDFS-2060, it would make it easier to piecemeal switch to protocol buffer based data structures in the wire protocol if we could intermix the two. The IPC framework currently provides the concept of "engines" for RPC, but that doesn't easily allow mixed types within the same framework for ease of transition.
> I'd like to add the cases to ObjectWritable to be handle subclasses of {{Message}}, the superclass of codegenned protobufs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16144-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 11 00:33:20 2011
Return-Path: <common-issues-return-16144-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 149494CAD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 11 Jun 2011 00:33:20 +0000 (UTC)
Received: (qmail 31793 invoked by uid 500); 11 Jun 2011 00:33:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31762 invoked by uid 500); 11 Jun 2011 00:33:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31754 invoked by uid 99); 11 Jun 2011 00:33:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 00:33:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 00:33:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D65DF10E6A5
	for <common-issues@hadoop.apache.org>; Sat, 11 Jun 2011 00:32:58 +0000 (UTC)
Date: Sat, 11 Jun 2011 00:32:58 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <248224096.13022.1307752378874.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <55510245.12688.1307747698852.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7379) Add ability to include Protobufs in
 ObjectWritable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7379:
--------------------------------

    Status: Patch Available  (was: Open)

> Add ability to include Protobufs in ObjectWritable
> --------------------------------------------------
>
>                 Key: HADOOP-7379
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7379
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, ipc
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7379.txt
>
>
> Per HDFS-2060, it would make it easier to piecemeal switch to protocol buffer based data structures in the wire protocol if we could intermix the two. The IPC framework currently provides the concept of "engines" for RPC, but that doesn't easily allow mixed types within the same framework for ease of transition.
> I'd like to add the cases to ObjectWritable to be handle subclasses of {{Message}}, the superclass of codegenned protobufs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16145-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 11 00:54:20 2011
Return-Path: <common-issues-return-16145-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C59C5499D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 11 Jun 2011 00:54:20 +0000 (UTC)
Received: (qmail 42022 invoked by uid 500); 11 Jun 2011 00:54:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41982 invoked by uid 500); 11 Jun 2011 00:54:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41974 invoked by uid 99); 11 Jun 2011 00:54:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 00:54:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 00:54:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6384B10EB0E
	for <common-issues@hadoop.apache.org>; Sat, 11 Jun 2011 00:53:59 +0000 (UTC)
Date: Sat, 11 Jun 2011 00:53:59 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1440012191.13063.1307753639404.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <55510245.12688.1307747698852.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7379) Add ability to include Protobufs
 in ObjectWritable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047810#comment-13047810 ] 

Hadoop QA commented on HADOOP-7379:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482124/hadoop-7379.txt
  against trunk revision 1134218.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 5 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/612//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/612//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/612//console

This message is automatically generated.

> Add ability to include Protobufs in ObjectWritable
> --------------------------------------------------
>
>                 Key: HADOOP-7379
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7379
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, ipc
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7379.txt
>
>
> Per HDFS-2060, it would make it easier to piecemeal switch to protocol buffer based data structures in the wire protocol if we could intermix the two. The IPC framework currently provides the concept of "engines" for RPC, but that doesn't easily allow mixed types within the same framework for ease of transition.
> I'd like to add the cases to ObjectWritable to be handle subclasses of {{Message}}, the superclass of codegenned protobufs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16146-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 11 00:58:23 2011
Return-Path: <common-issues-return-16146-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E16E54A21
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 11 Jun 2011 00:58:22 +0000 (UTC)
Received: (qmail 48412 invoked by uid 500); 11 Jun 2011 00:58:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48382 invoked by uid 500); 11 Jun 2011 00:58:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48374 invoked by uid 99); 11 Jun 2011 00:58:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 00:58:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 00:58:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7462510EB9F
	for <common-issues@hadoop.apache.org>; Sat, 11 Jun 2011 00:57:59 +0000 (UTC)
Date: Sat, 11 Jun 2011 00:57:59 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1752073012.13070.1307753879473.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <55510245.12688.1307747698852.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7379) Add ability to include Protobufs
 in ObjectWritable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047812#comment-13047812 ] 

Luke Lu commented on HADOOP-7379:
---------------------------------

Todd, what PEDs are you taking? I mean, you're on a roll :) 

ProtoUtil.readRawVarint32 is a new implementation vint32 decoding for DataInput. It needs test coverage for 1 to 5 bytes. 

> Add ability to include Protobufs in ObjectWritable
> --------------------------------------------------
>
>                 Key: HADOOP-7379
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7379
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, ipc
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7379.txt
>
>
> Per HDFS-2060, it would make it easier to piecemeal switch to protocol buffer based data structures in the wire protocol if we could intermix the two. The IPC framework currently provides the concept of "engines" for RPC, but that doesn't easily allow mixed types within the same framework for ease of transition.
> I'd like to add the cases to ObjectWritable to be handle subclasses of {{Message}}, the superclass of codegenned protobufs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16147-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 11 02:48:25 2011
Return-Path: <common-issues-return-16147-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 43F806D37
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 11 Jun 2011 02:48:25 +0000 (UTC)
Received: (qmail 18276 invoked by uid 500); 11 Jun 2011 02:48:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18064 invoked by uid 500); 11 Jun 2011 02:48:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18056 invoked by uid 99); 11 Jun 2011 02:48:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 02:48:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 02:48:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C06B210EB66
	for <common-issues@hadoop.apache.org>; Sat, 11 Jun 2011 02:47:58 +0000 (UTC)
Date: Sat, 11 Jun 2011 02:47:58 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1625774222.13162.1307760478784.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <634368540.9715.1307685358867.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7375) Add resolvePath method to
 FileContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047826#comment-13047826 ] 

Sanjay Radia commented on HADOOP-7375:
--------------------------------------

>AFS#getFileStatus is now called instead of FileContext#getFileStatus, which means
>fixRelativePart is no longer used to make the path absolute in FileContext relative to the
>working dir before passing the path to AFS right? 


No sure if your question is with regards to all the methods in FileContext or the single method I changed.
I have changed the FileContext#resolve to call AFS#resolvePath(p) which currently calls
AFS#getFileStatus(p)

> Add resolvePath method to FileContext
> -------------------------------------
>
>                 Key: HADOOP-7375
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7375
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: resolvePath1.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16148-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 11 03:28:33 2011
Return-Path: <common-issues-return-16148-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E37C86A47
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 11 Jun 2011 03:28:32 +0000 (UTC)
Received: (qmail 35823 invoked by uid 500); 11 Jun 2011 03:28:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34370 invoked by uid 500); 11 Jun 2011 03:28:26 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34360 invoked by uid 99); 11 Jun 2011 03:28:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 03:28:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 03:28:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C63BA10E05C
	for <common-issues@hadoop.apache.org>; Sat, 11 Jun 2011 03:27:58 +0000 (UTC)
Date: Sat, 11 Jun 2011 03:27:58 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1903916273.13177.1307762878808.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <55510245.12688.1307747698852.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7379) Add ability to include Protobufs
 in ObjectWritable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047829#comment-13047829 ] 

Todd Lipcon commented on HADOOP-7379:
-------------------------------------

bq. Todd, what PEDs are you taking? I mean, you're on a roll 

I recently learned how to make pretty good cappuccinos from our machine here at the office. No joke, I'm considering a part time job at Blue Bottle. Thanks :)

bq. ProtoUtil.readRawVarint32 is a new implementation vint32 decoding for DataInput. It needs test coverage for 1 to 5 bytes.

I actually copy-pasted it from protobufs (compatible license). I can probably copy-paste some of their unit tests, too.. I'll look around.

> Add ability to include Protobufs in ObjectWritable
> --------------------------------------------------
>
>                 Key: HADOOP-7379
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7379
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, ipc
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7379.txt
>
>
> Per HDFS-2060, it would make it easier to piecemeal switch to protocol buffer based data structures in the wire protocol if we could intermix the two. The IPC framework currently provides the concept of "engines" for RPC, but that doesn't easily allow mixed types within the same framework for ease of transition.
> I'd like to add the cases to ObjectWritable to be handle subclasses of {{Message}}, the superclass of codegenned protobufs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16149-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 11 03:48:24 2011
Return-Path: <common-issues-return-16149-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D477E6447
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 11 Jun 2011 03:48:24 +0000 (UTC)
Received: (qmail 42166 invoked by uid 500); 11 Jun 2011 03:48:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42132 invoked by uid 500); 11 Jun 2011 03:48:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42116 invoked by uid 99); 11 Jun 2011 03:48:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 03:48:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 03:48:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CBB0810E324
	for <common-issues@hadoop.apache.org>; Sat, 11 Jun 2011 03:47:58 +0000 (UTC)
Date: Sat, 11 Jun 2011 03:47:58 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1354187382.13188.1307764078831.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7380) Common portion of HDFS-1973
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Common portion of HDFS-1973
---------------------------

                 Key: HADOOP-7380
                 URL: https://issues.apache.org/jira/browse/HADOOP-7380
             Project: Hadoop Common
          Issue Type: New Feature
          Components: ipc
    Affects Versions: 0.23.0
            Reporter: Aaron T. Myers
            Assignee: Aaron T. Myers
             Fix For: 0.23.0


Implementing client failover will likely require changes to {{o.a.h.io.ipc}} and/or {{o.a.h.io.retry}}. This JIRA is to track those changes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16150-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 11 04:22:25 2011
Return-Path: <common-issues-return-16150-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4BBB46139
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 11 Jun 2011 04:22:25 +0000 (UTC)
Received: (qmail 65065 invoked by uid 500); 11 Jun 2011 04:22:25 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64858 invoked by uid 500); 11 Jun 2011 04:22:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63749 invoked by uid 99); 11 Jun 2011 04:22:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 04:22:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 04:22:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 12C1410E893
	for <common-issues@hadoop.apache.org>; Sat, 11 Jun 2011 04:21:59 +0000 (UTC)
Date: Sat, 11 Jun 2011 04:21:59 +0000 (UTC)
From: "Joep Rottinghuis (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <389414663.13212.1307766119073.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7381) FindBugs OutOfMemoryError
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

FindBugs OutOfMemoryError
-------------------------

                 Key: HADOOP-7381
                 URL: https://issues.apache.org/jira/browse/HADOOP-7381
             Project: Hadoop Common
          Issue Type: Bug
          Components: build
    Affects Versions: 0.20.205.0
         Environment: FindBugs 1.3.9, ant 1.8.2, RHEL6, Jenkins 1.414 in Tomcat 7.0.14, Sun Java HotSpot(TM) 64-Bit Server VM (build 14.2-b01, mixed mode)
            Reporter: Joep Rottinghuis
            Assignee: Joep Rottinghuis
            Priority: Minor


When running the findbugs target from Jenkins, I get an OutOfMemory error.
The "effort" in FindBugs is set to Max which ends up using a lot of memory to go through all the classes. The jvmargs passed to FindBugs is hardcoded to 512 MB max.

We can leave the default to 512M, as long as we pass this as an ant parameter which can be overwritten in individual cases through -D, or in the build.properties file (either basedir, or user's home directory).

This is the error I get:

findbugs:
    [mkdir] Created dir: /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs
 [findbugs] Executing findbugs from ant task
 [findbugs] Running FindBugs...
 [findbugs] Out of memory
 [findbugs] Total memory: 477M
 [findbugs]  free memory: 62M
 [findbugs] Analyzed: /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/hadoop-core-0.20.security-test-3.jar
...

[findbugs] Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
 [findbugs] 	at edu.umd.cs.findbugs.OpcodeStack.pushBySignature(OpcodeStack.java:2589)
 [findbugs] 	at edu.umd.cs.findbugs.OpcodeStack.pushByInvoke(OpcodeStack.java:2565)
 [findbugs] 	at edu.umd.cs.findbugs.OpcodeStack.processMethodCall(OpcodeStack.java:2011)
 [findbugs] 	at edu.umd.cs.findbugs.OpcodeStack.sawOpcode(OpcodeStack.java:1574)
                  ...15 more
 [findbugs] Java Result: 1
 [findbugs] Output saved to /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.xml
     [xslt] Processing /hadoop01/jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.xml to /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.html
     [xslt] Loading stylesheet /usr/local/findbugs/src/xsl/default.xsl
     [xslt] : Error! Premature end of file.
     [xslt] : Error! com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
     [xslt] Failed to process /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.xml

BUILD FAILED
/.../jenkins/jobs/hadoop-common-test-smoke/workspace/build.xml:1164: javax.xml.transform.TransformerException: javax.xml.transform.TransformerException: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:720)
	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:313)
	at org.apache.tools.ant.taskdefs.optional.TraXLiaison.transform(TraXLiaison.java:194)
	at org.apache.tools.ant.taskdefs.XSLTProcess.process(XSLTProcess.java:852)
	at org.apache.tools.ant.taskdefs.XSLTProcess.execute(XSLTProcess.java:388)
	at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
	... 15 more
Caused by: javax.xml.transform.TransformerException: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getDOM(TransformerImpl.java:547)
	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:710)
	... 20 more
Caused by: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
	at com.sun.org.apache.xalan.internal.xsltc.dom.XSLTCDTMManager.getDTM(XSLTCDTMManager.java:446)
	at com.sun.org.apache.xalan.internal.xsltc.dom.XSLTCDTMManager.getDTM(XSLTCDTMManager.java:234)
	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getDOM(TransformerImpl.java:525)
	... 21 more
---------

etc.etc. 
Total time: 8 minutes 35 seconds
Build step 'Execute shell' marked build as failure
[CHECKSTYLE] Collecting checkstyle analysis files...
CHECKSTYLESuccessfully parsed file /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/checkstyle-errors.xml of module  with 13525 warnings.
[FINDBUGS] Skipping publisher since build result is FAILURE
[WARNINGS] Skipping publisher since build result is FAILURE



--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16151-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 11 04:24:21 2011
Return-Path: <common-issues-return-16151-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AA4986182
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 11 Jun 2011 04:24:21 +0000 (UTC)
Received: (qmail 65676 invoked by uid 500); 11 Jun 2011 04:24:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65644 invoked by uid 500); 11 Jun 2011 04:24:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65634 invoked by uid 99); 11 Jun 2011 04:24:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 04:24:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 04:24:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 678F710E8E7
	for <common-issues@hadoop.apache.org>; Sat, 11 Jun 2011 04:23:59 +0000 (UTC)
Date: Sat, 11 Jun 2011 04:23:59 +0000 (UTC)
From: "Joep Rottinghuis (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1849702119.13214.1307766239420.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <389414663.13212.1307766119073.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7381) FindBugs OutOfMemoryError
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047835#comment-13047835 ] 

Joep Rottinghuis commented on HADOOP-7381:
------------------------------------------

Even though this is hardcoded on trunk as well, it is less likely to be a problem due to the project split. After the split each of the three projects will have a much smaller set of files to parse during the FindBugs targets.

> FindBugs OutOfMemoryError
> -------------------------
>
>                 Key: HADOOP-7381
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7381
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.20.205.0
>         Environment: FindBugs 1.3.9, ant 1.8.2, RHEL6, Jenkins 1.414 in Tomcat 7.0.14, Sun Java HotSpot(TM) 64-Bit Server VM (build 14.2-b01, mixed mode)
>            Reporter: Joep Rottinghuis
>            Assignee: Joep Rottinghuis
>            Priority: Minor
>
> When running the findbugs target from Jenkins, I get an OutOfMemory error.
> The "effort" in FindBugs is set to Max which ends up using a lot of memory to go through all the classes. The jvmargs passed to FindBugs is hardcoded to 512 MB max.
> We can leave the default to 512M, as long as we pass this as an ant parameter which can be overwritten in individual cases through -D, or in the build.properties file (either basedir, or user's home directory).
> This is the error I get:
> findbugs:
>     [mkdir] Created dir: /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs
>  [findbugs] Executing findbugs from ant task
>  [findbugs] Running FindBugs...
>  [findbugs] Out of memory
>  [findbugs] Total memory: 477M
>  [findbugs]  free memory: 62M
>  [findbugs] Analyzed: /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/hadoop-core-0.20.security-test-3.jar
> ...
> [findbugs] Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
>  [findbugs] 	at edu.umd.cs.findbugs.OpcodeStack.pushBySignature(OpcodeStack.java:2589)
>  [findbugs] 	at edu.umd.cs.findbugs.OpcodeStack.pushByInvoke(OpcodeStack.java:2565)
>  [findbugs] 	at edu.umd.cs.findbugs.OpcodeStack.processMethodCall(OpcodeStack.java:2011)
>  [findbugs] 	at edu.umd.cs.findbugs.OpcodeStack.sawOpcode(OpcodeStack.java:1574)
>                   ...15 more
>  [findbugs] Java Result: 1
>  [findbugs] Output saved to /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.xml
>      [xslt] Processing /hadoop01/jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.xml to /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.html
>      [xslt] Loading stylesheet /usr/local/findbugs/src/xsl/default.xsl
>      [xslt] : Error! Premature end of file.
>      [xslt] : Error! com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
>      [xslt] Failed to process /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.xml
> BUILD FAILED
> /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build.xml:1164: javax.xml.transform.TransformerException: javax.xml.transform.TransformerException: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
> 	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:720)
> 	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:313)
> 	at org.apache.tools.ant.taskdefs.optional.TraXLiaison.transform(TraXLiaison.java:194)
> 	at org.apache.tools.ant.taskdefs.XSLTProcess.process(XSLTProcess.java:852)
> 	at org.apache.tools.ant.taskdefs.XSLTProcess.execute(XSLTProcess.java:388)
> 	at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
> 	... 15 more
> Caused by: javax.xml.transform.TransformerException: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
> 	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getDOM(TransformerImpl.java:547)
> 	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:710)
> 	... 20 more
> Caused by: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
> 	at com.sun.org.apache.xalan.internal.xsltc.dom.XSLTCDTMManager.getDTM(XSLTCDTMManager.java:446)
> 	at com.sun.org.apache.xalan.internal.xsltc.dom.XSLTCDTMManager.getDTM(XSLTCDTMManager.java:234)
> 	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getDOM(TransformerImpl.java:525)
> 	... 21 more
> ---------
> etc.etc. 
> Total time: 8 minutes 35 seconds
> Build step 'Execute shell' marked build as failure
> [CHECKSTYLE] Collecting checkstyle analysis files...
> CHECKSTYLESuccessfully parsed file /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/checkstyle-errors.xml of module  with 13525 warnings.
> [FINDBUGS] Skipping publisher since build result is FAILURE
> [WARNINGS] Skipping publisher since build result is FAILURE

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16152-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 11 04:30:23 2011
Return-Path: <common-issues-return-16152-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2128A635F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 11 Jun 2011 04:30:23 +0000 (UTC)
Received: (qmail 67209 invoked by uid 500); 11 Jun 2011 04:30:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67176 invoked by uid 500); 11 Jun 2011 04:30:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67166 invoked by uid 99); 11 Jun 2011 04:30:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 04:30:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 04:30:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C7FBF10E9E3
	for <common-issues@hadoop.apache.org>; Sat, 11 Jun 2011 04:29:58 +0000 (UTC)
Date: Sat, 11 Jun 2011 04:29:58 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <80655891.13222.1307766598815.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <55510245.12688.1307747698852.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7379) Add ability to include Protobufs in
 ObjectWritable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7379:
--------------------------------

    Attachment: hadoop-7379.txt

Added a simple unit test that encodes ints with CodedOutputStream and then decodes them with the function in ProtoUtil

> Add ability to include Protobufs in ObjectWritable
> --------------------------------------------------
>
>                 Key: HADOOP-7379
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7379
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, ipc
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7379.txt, hadoop-7379.txt
>
>
> Per HDFS-2060, it would make it easier to piecemeal switch to protocol buffer based data structures in the wire protocol if we could intermix the two. The IPC framework currently provides the concept of "engines" for RPC, but that doesn't easily allow mixed types within the same framework for ease of transition.
> I'd like to add the cases to ObjectWritable to be handle subclasses of {{Message}}, the superclass of codegenned protobufs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16153-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 11 04:40:24 2011
Return-Path: <common-issues-return-16153-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1B5DD66D5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 11 Jun 2011 04:40:24 +0000 (UTC)
Received: (qmail 69656 invoked by uid 500); 11 Jun 2011 04:40:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69621 invoked by uid 500); 11 Jun 2011 04:40:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69604 invoked by uid 99); 11 Jun 2011 04:40:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 04:40:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 04:40:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D53E510EB58
	for <common-issues@hadoop.apache.org>; Sat, 11 Jun 2011 04:39:58 +0000 (UTC)
Date: Sat, 11 Jun 2011 04:39:58 +0000 (UTC)
From: "shanlin (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <793051693.13228.1307767198870.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7382) hadoop 0.20.203.0 Eclipse Plugin
 does not work with Eclipse
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

hadoop 0.20.203.0 Eclipse Plugin does not work with Eclipse
-----------------------------------------------------------

                 Key: HADOOP-7382
                 URL: https://issues.apache.org/jira/browse/HADOOP-7382
             Project: Hadoop Common
          Issue Type: Bug
          Components: contrib/eclipse-plugin
    Affects Versions: 0.20.203.0
         Environment: window 7 ; eclipse3.6-rc2
            Reporter: shanlin
            Priority: Blocker


hadoop 0.20.203.0 Eclipse Plugin does not work with Eclipse,while adding DFS Locations,Advance parameters didin't hava a option hadoop.job.ugi

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16154-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 11 04:46:22 2011
Return-Path: <common-issues-return-16154-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D00D96FA9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 11 Jun 2011 04:46:22 +0000 (UTC)
Received: (qmail 75203 invoked by uid 500); 11 Jun 2011 04:46:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75171 invoked by uid 500); 11 Jun 2011 04:46:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75158 invoked by uid 99); 11 Jun 2011 04:46:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 04:46:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 04:46:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E402810EC3F
	for <common-issues@hadoop.apache.org>; Sat, 11 Jun 2011 04:45:58 +0000 (UTC)
Date: Sat, 11 Jun 2011 04:45:58 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <774305907.13237.1307767558930.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <55510245.12688.1307747698852.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7379) Add ability to include Protobufs
 in ObjectWritable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047838#comment-13047838 ] 

Hadoop QA commented on HADOOP-7379:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482130/hadoop-7379.txt
  against trunk revision 1134218.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 7 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/613//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/613//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/613//console

This message is automatically generated.

> Add ability to include Protobufs in ObjectWritable
> --------------------------------------------------
>
>                 Key: HADOOP-7379
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7379
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, ipc
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7379.txt, hadoop-7379.txt
>
>
> Per HDFS-2060, it would make it easier to piecemeal switch to protocol buffer based data structures in the wire protocol if we could intermix the two. The IPC framework currently provides the concept of "engines" for RPC, but that doesn't easily allow mixed types within the same framework for ease of transition.
> I'd like to add the cases to ObjectWritable to be handle subclasses of {{Message}}, the superclass of codegenned protobufs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16155-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 11 05:06:24 2011
Return-Path: <common-issues-return-16155-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1FC3D62A7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 11 Jun 2011 05:06:24 +0000 (UTC)
Received: (qmail 82865 invoked by uid 500); 11 Jun 2011 05:06:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82815 invoked by uid 500); 11 Jun 2011 05:06:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82804 invoked by uid 99); 11 Jun 2011 05:06:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 05:06:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 05:06:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0317D10EF57
	for <common-issues@hadoop.apache.org>; Sat, 11 Jun 2011 05:05:59 +0000 (UTC)
Date: Sat, 11 Jun 2011 05:05:59 +0000 (UTC)
From: "Joep Rottinghuis (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <8326607.13252.1307768759009.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <389414663.13212.1307766119073.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7381) FindBugs OutOfMemoryError
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joep Rottinghuis updated HADOOP-7381:
-------------------------------------

    Attachment: hadoop-7381.patch

Adjusted build file to make FindBugs jvmargs a ant property with the same default value, but this one can be set from outside build.xml

> FindBugs OutOfMemoryError
> -------------------------
>
>                 Key: HADOOP-7381
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7381
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.20.205.0
>         Environment: FindBugs 1.3.9, ant 1.8.2, RHEL6, Jenkins 1.414 in Tomcat 7.0.14, Sun Java HotSpot(TM) 64-Bit Server VM (build 14.2-b01, mixed mode)
>            Reporter: Joep Rottinghuis
>            Assignee: Joep Rottinghuis
>            Priority: Minor
>         Attachments: hadoop-7381.patch
>
>
> When running the findbugs target from Jenkins, I get an OutOfMemory error.
> The "effort" in FindBugs is set to Max which ends up using a lot of memory to go through all the classes. The jvmargs passed to FindBugs is hardcoded to 512 MB max.
> We can leave the default to 512M, as long as we pass this as an ant parameter which can be overwritten in individual cases through -D, or in the build.properties file (either basedir, or user's home directory).
> This is the error I get:
> findbugs:
>     [mkdir] Created dir: /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs
>  [findbugs] Executing findbugs from ant task
>  [findbugs] Running FindBugs...
>  [findbugs] Out of memory
>  [findbugs] Total memory: 477M
>  [findbugs]  free memory: 62M
>  [findbugs] Analyzed: /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/hadoop-core-0.20.security-test-3.jar
> ...
> [findbugs] Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
>  [findbugs] 	at edu.umd.cs.findbugs.OpcodeStack.pushBySignature(OpcodeStack.java:2589)
>  [findbugs] 	at edu.umd.cs.findbugs.OpcodeStack.pushByInvoke(OpcodeStack.java:2565)
>  [findbugs] 	at edu.umd.cs.findbugs.OpcodeStack.processMethodCall(OpcodeStack.java:2011)
>  [findbugs] 	at edu.umd.cs.findbugs.OpcodeStack.sawOpcode(OpcodeStack.java:1574)
>                   ...15 more
>  [findbugs] Java Result: 1
>  [findbugs] Output saved to /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.xml
>      [xslt] Processing /hadoop01/jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.xml to /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.html
>      [xslt] Loading stylesheet /usr/local/findbugs/src/xsl/default.xsl
>      [xslt] : Error! Premature end of file.
>      [xslt] : Error! com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
>      [xslt] Failed to process /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.xml
> BUILD FAILED
> /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build.xml:1164: javax.xml.transform.TransformerException: javax.xml.transform.TransformerException: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
> 	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:720)
> 	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:313)
> 	at org.apache.tools.ant.taskdefs.optional.TraXLiaison.transform(TraXLiaison.java:194)
> 	at org.apache.tools.ant.taskdefs.XSLTProcess.process(XSLTProcess.java:852)
> 	at org.apache.tools.ant.taskdefs.XSLTProcess.execute(XSLTProcess.java:388)
> 	at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
> 	... 15 more
> Caused by: javax.xml.transform.TransformerException: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
> 	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getDOM(TransformerImpl.java:547)
> 	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:710)
> 	... 20 more
> Caused by: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
> 	at com.sun.org.apache.xalan.internal.xsltc.dom.XSLTCDTMManager.getDTM(XSLTCDTMManager.java:446)
> 	at com.sun.org.apache.xalan.internal.xsltc.dom.XSLTCDTMManager.getDTM(XSLTCDTMManager.java:234)
> 	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getDOM(TransformerImpl.java:525)
> 	... 21 more
> ---------
> etc.etc. 
> Total time: 8 minutes 35 seconds
> Build step 'Execute shell' marked build as failure
> [CHECKSTYLE] Collecting checkstyle analysis files...
> CHECKSTYLESuccessfully parsed file /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/checkstyle-errors.xml of module  with 13525 warnings.
> [FINDBUGS] Skipping publisher since build result is FAILURE
> [WARNINGS] Skipping publisher since build result is FAILURE

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16156-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 11 05:06:25 2011
Return-Path: <common-issues-return-16156-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6CC5662B6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 11 Jun 2011 05:06:25 +0000 (UTC)
Received: (qmail 83106 invoked by uid 500); 11 Jun 2011 05:06:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83041 invoked by uid 500); 11 Jun 2011 05:06:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82813 invoked by uid 99); 11 Jun 2011 05:06:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 05:06:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 05:06:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0A70D10EF58
	for <common-issues@hadoop.apache.org>; Sat, 11 Jun 2011 05:05:59 +0000 (UTC)
Date: Sat, 11 Jun 2011 05:05:59 +0000 (UTC)
From: "Joep Rottinghuis (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <358846794.13253.1307768759039.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <389414663.13212.1307766119073.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7381) FindBugs OutOfMemoryError
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047843#comment-13047843 ] 

Joep Rottinghuis commented on HADOOP-7381:
------------------------------------------

Once patch is applied and I add the following to my build.properties, the error disappears:

# Give FindBugs some more memory due to the large # classes and effort=max parameter.
findbugs.jvmargs=-Xmx1024M

> FindBugs OutOfMemoryError
> -------------------------
>
>                 Key: HADOOP-7381
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7381
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.20.205.0
>         Environment: FindBugs 1.3.9, ant 1.8.2, RHEL6, Jenkins 1.414 in Tomcat 7.0.14, Sun Java HotSpot(TM) 64-Bit Server VM (build 14.2-b01, mixed mode)
>            Reporter: Joep Rottinghuis
>            Assignee: Joep Rottinghuis
>            Priority: Minor
>         Attachments: hadoop-7381.patch
>
>
> When running the findbugs target from Jenkins, I get an OutOfMemory error.
> The "effort" in FindBugs is set to Max which ends up using a lot of memory to go through all the classes. The jvmargs passed to FindBugs is hardcoded to 512 MB max.
> We can leave the default to 512M, as long as we pass this as an ant parameter which can be overwritten in individual cases through -D, or in the build.properties file (either basedir, or user's home directory).
> This is the error I get:
> findbugs:
>     [mkdir] Created dir: /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs
>  [findbugs] Executing findbugs from ant task
>  [findbugs] Running FindBugs...
>  [findbugs] Out of memory
>  [findbugs] Total memory: 477M
>  [findbugs]  free memory: 62M
>  [findbugs] Analyzed: /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/hadoop-core-0.20.security-test-3.jar
> ...
> [findbugs] Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
>  [findbugs] 	at edu.umd.cs.findbugs.OpcodeStack.pushBySignature(OpcodeStack.java:2589)
>  [findbugs] 	at edu.umd.cs.findbugs.OpcodeStack.pushByInvoke(OpcodeStack.java:2565)
>  [findbugs] 	at edu.umd.cs.findbugs.OpcodeStack.processMethodCall(OpcodeStack.java:2011)
>  [findbugs] 	at edu.umd.cs.findbugs.OpcodeStack.sawOpcode(OpcodeStack.java:1574)
>                   ...15 more
>  [findbugs] Java Result: 1
>  [findbugs] Output saved to /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.xml
>      [xslt] Processing /hadoop01/jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.xml to /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.html
>      [xslt] Loading stylesheet /usr/local/findbugs/src/xsl/default.xsl
>      [xslt] : Error! Premature end of file.
>      [xslt] : Error! com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
>      [xslt] Failed to process /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.xml
> BUILD FAILED
> /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build.xml:1164: javax.xml.transform.TransformerException: javax.xml.transform.TransformerException: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
> 	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:720)
> 	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:313)
> 	at org.apache.tools.ant.taskdefs.optional.TraXLiaison.transform(TraXLiaison.java:194)
> 	at org.apache.tools.ant.taskdefs.XSLTProcess.process(XSLTProcess.java:852)
> 	at org.apache.tools.ant.taskdefs.XSLTProcess.execute(XSLTProcess.java:388)
> 	at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
> 	... 15 more
> Caused by: javax.xml.transform.TransformerException: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
> 	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getDOM(TransformerImpl.java:547)
> 	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:710)
> 	... 20 more
> Caused by: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
> 	at com.sun.org.apache.xalan.internal.xsltc.dom.XSLTCDTMManager.getDTM(XSLTCDTMManager.java:446)
> 	at com.sun.org.apache.xalan.internal.xsltc.dom.XSLTCDTMManager.getDTM(XSLTCDTMManager.java:234)
> 	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getDOM(TransformerImpl.java:525)
> 	... 21 more
> ---------
> etc.etc. 
> Total time: 8 minutes 35 seconds
> Build step 'Execute shell' marked build as failure
> [CHECKSTYLE] Collecting checkstyle analysis files...
> CHECKSTYLESuccessfully parsed file /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/checkstyle-errors.xml of module  with 13525 warnings.
> [FINDBUGS] Skipping publisher since build result is FAILURE
> [WARNINGS] Skipping publisher since build result is FAILURE

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16157-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 11 06:37:15 2011
Return-Path: <common-issues-return-16157-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 015AE6627
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 11 Jun 2011 06:37:15 +0000 (UTC)
Received: (qmail 23450 invoked by uid 500); 11 Jun 2011 06:37:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22764 invoked by uid 500); 11 Jun 2011 06:36:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22691 invoked by uid 99); 11 Jun 2011 06:36:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 06:36:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 06:36:22 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4DA7210EBA7
	for <common-issues@hadoop.apache.org>; Sat, 11 Jun 2011 06:36:02 +0000 (UTC)
Date: Sat, 11 Jun 2011 06:36:02 +0000 (UTC)
From: "Ted Dunning (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <808994966.13302.1307774162314.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32237646.17061288907081267.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7020) establish a "Powered by Hadoop"
 logo
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047847#comment-13047847 ] 

Ted Dunning commented on HADOOP-7020:
-------------------------------------

Eli,

That is certainly a powerful looking powered-by logo.

It doesn't fit the stylistic heritage of a playful logo based on a children's toy, though.  My own feeling is that it would be good to have the option of maintaining the continuity to make clear that software that is powered by Hadoop owes its patrinomy to Hadoop.



> establish a "Powered by Hadoop" logo
> ------------------------------------
>
>                 Key: HADOOP-7020
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7020
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: site
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: site
>
>         Attachments: PoweredByApacheHadoop.png, PoweredByHadoop_Small.jpg, hadoop-elephant-pb.jpeg, powered-by-hadoop-small.png, powered-by-hadoop.png, powered_by_hadoop.jpg
>
>
> We should agree on a Powered By Hadoop logo, as suggested in:
> http://www.apache.org/foundation/marks/pmcs#poweredby

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16158-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 11 09:04:20 2011
Return-Path: <common-issues-return-16158-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EBEDA618E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 11 Jun 2011 09:04:20 +0000 (UTC)
Received: (qmail 1654 invoked by uid 500); 11 Jun 2011 09:04:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1523 invoked by uid 500); 11 Jun 2011 09:04:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1509 invoked by uid 99); 11 Jun 2011 09:04:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 09:04:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 09:04:18 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CDB8310E616
	for <common-issues@hadoop.apache.org>; Sat, 11 Jun 2011 09:03:58 +0000 (UTC)
Date: Sat, 11 Jun 2011 09:03:58 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <834230839.13569.1307783038839.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32237646.17061288907081267.JavaMail.jira@thor>
Subject: [jira] [Updated] (HADOOP-7020) establish a "Powered by Hadoop" logo
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7020:
-----------------------------------

    Attachment: powered-by-proposal.jpg

Here's a proposal I'd like to put forth. I tried to incorporate as much as possible of the feedback people have thus far provided. Specifically, it is somewhat visually distinct from the project logo while retaining obvious association, and incorporates the Apache feather.

I also think it works well with the tradition of wearing different hats at Apache. If there were such a thing as an actual "Apache hat" I imagine it would be a headband with a single feather.

> establish a "Powered by Hadoop" logo
> ------------------------------------
>
>                 Key: HADOOP-7020
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7020
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: site
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: site
>
>         Attachments: PoweredByApacheHadoop.png, PoweredByHadoop_Small.jpg, hadoop-elephant-pb.jpeg, powered-by-hadoop-small.png, powered-by-hadoop.png, powered-by-proposal.jpg, powered_by_hadoop.jpg
>
>
> We should agree on a Powered By Hadoop logo, as suggested in:
> http://www.apache.org/foundation/marks/pmcs#poweredby

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16159-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 11 09:16:21 2011
Return-Path: <common-issues-return-16159-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EE05F6724
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 11 Jun 2011 09:16:20 +0000 (UTC)
Received: (qmail 9604 invoked by uid 500); 11 Jun 2011 09:16:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9576 invoked by uid 500); 11 Jun 2011 09:16:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9567 invoked by uid 99); 11 Jun 2011 09:16:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 09:16:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 09:16:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C118C10E889
	for <common-issues@hadoop.apache.org>; Sat, 11 Jun 2011 09:15:59 +0000 (UTC)
Date: Sat, 11 Jun 2011 09:15:59 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <821134800.13614.1307783759787.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32237646.17061288907081267.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7020) establish a "Powered by Hadoop"
 logo
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047870#comment-13047870 ] 

Aaron T. Myers commented on HADOOP-7020:
----------------------------------------

Also, does anyone know what font is used for the word "hadoop" in the project logo? I haven't found a good way to figure that out.

> establish a "Powered by Hadoop" logo
> ------------------------------------
>
>                 Key: HADOOP-7020
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7020
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: site
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: site
>
>         Attachments: PoweredByApacheHadoop.png, PoweredByHadoop_Small.jpg, hadoop-elephant-pb.jpeg, powered-by-hadoop-small.png, powered-by-hadoop.png, powered-by-proposal.jpg, powered_by_hadoop.jpg
>
>
> We should agree on a Powered By Hadoop logo, as suggested in:
> http://www.apache.org/foundation/marks/pmcs#poweredby

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16160-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 11 09:24:26 2011
Return-Path: <common-issues-return-16160-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 71A326877
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 11 Jun 2011 09:24:26 +0000 (UTC)
Received: (qmail 12665 invoked by uid 500); 11 Jun 2011 09:24:26 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12630 invoked by uid 500); 11 Jun 2011 09:24:26 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12622 invoked by uid 99); 11 Jun 2011 09:24:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 09:24:26 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 09:24:24 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 323C710E9D5
	for <common-issues@hadoop.apache.org>; Sat, 11 Jun 2011 09:24:03 +0000 (UTC)
Date: Sat, 11 Jun 2011 09:24:03 +0000 (UTC)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1867765000.13623.1307784243202.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <55510245.12688.1307747698852.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7379) Add ability to include Protobufs
 in ObjectWritable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047872#comment-13047872 ] 

Arun C Murthy commented on HADOOP-7379:
---------------------------------------

Woah, tlipcon, slow down man! :)

In all seriousness... great stuff.

What will it take to get ipc.Server to understand PB rather than Writable?

> Add ability to include Protobufs in ObjectWritable
> --------------------------------------------------
>
>                 Key: HADOOP-7379
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7379
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, ipc
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7379.txt, hadoop-7379.txt
>
>
> Per HDFS-2060, it would make it easier to piecemeal switch to protocol buffer based data structures in the wire protocol if we could intermix the two. The IPC framework currently provides the concept of "engines" for RPC, but that doesn't easily allow mixed types within the same framework for ease of transition.
> I'd like to add the cases to ObjectWritable to be handle subclasses of {{Message}}, the superclass of codegenned protobufs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16161-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 11 18:31:20 2011
Return-Path: <common-issues-return-16161-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 46BB86B8B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 11 Jun 2011 18:31:20 +0000 (UTC)
Received: (qmail 62056 invoked by uid 500); 11 Jun 2011 18:31:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62026 invoked by uid 500); 11 Jun 2011 18:31:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62013 invoked by uid 99); 11 Jun 2011 18:31:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 18:31:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 18:31:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 128C9110194
	for <common-issues@hadoop.apache.org>; Sat, 11 Jun 2011 18:30:59 +0000 (UTC)
Date: Sat, 11 Jun 2011 18:30:59 +0000 (UTC)
From: "Bochun Bai (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2012445929.14160.1307817059072.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6424) Port FsShell to FileContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bochun Bai updated HADOOP-6424:
-------------------------------

    Attachment: HADOOP-6424.patch

FsShell changed a lot these two years.
More FileSystem should be removed from org.apache.hadoop.fs.shell package.

> Port FsShell to FileContext 
> ----------------------------
>
>                 Key: HADOOP-6424
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6424
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: John George
>         Attachments: HADOOP-6424.patch, HADOOP-6424.patch
>
>
> FsShell currently uses FileSystem, needs to be moved over to FileContext.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-16162-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 11 22:12:23 2011
Return-Path: <common-issues-return-16162-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 692196A64
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 11 Jun 2011 22:12:23 +0000 (UTC)
Received: (qmail 39461 invoked by uid 500); 11 Jun 2011 22:12:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39433 invoked by uid 500); 11 Jun 2011 22:12:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39425 invoked by uid 99); 11 Jun 2011 22:12:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 22:12:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 22:12:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 17FE04141F7
	for <common-issues@hadoop.apache.org>; Sat, 11 Jun 2011 22:12:00 +0000 (UTC)
Date: Sat, 11 Jun 2011 22:12:00 +0000 (UTC)
From: "jiraposter@reviews.apache.org (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2132487624.72.1307830320095.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047991#comment-13047991 ] 

jiraposter@reviews.apache.org commented on HADOOP-7328:
-------------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/884/
-----------------------------------------------------------

Review request for hadoop-common and Todd Lipcon.


Summary
-------

Since getSerialization() can possibly return a null, it is only right that getSerializer() and getDeserializer() usage functions do the same, instead of throwing up NPEs.

Related issue to which this improvement is required: https://issues.apache.org/jira/browse/MAPREDUCE-2584


This addresses bug HADOOP-7328.
    http://issues.apache.org/jira/browse/HADOOP-7328


Diffs
-----

  src/java/org/apache/hadoop/io/serializer/SerializationFactory.java dee314a 

Diff: https://reviews.apache.org/r/884/diff


Testing
-------

Existing SequenceFile serialization factory tests pass. The change is merely to make the functions return null instead of throwing an NPE within.


Thanks,

Harsh



> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16163-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 11 22:20:21 2011
Return-Path: <common-issues-return-16163-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8EAB56163
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 11 Jun 2011 22:20:21 +0000 (UTC)
Received: (qmail 46572 invoked by uid 500); 11 Jun 2011 22:20:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46535 invoked by uid 500); 11 Jun 2011 22:20:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46524 invoked by uid 99); 11 Jun 2011 22:20:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 22:20:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Jun 2011 22:20:19 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7553E414350
	for <common-issues@hadoop.apache.org>; Sat, 11 Jun 2011 22:19:58 +0000 (UTC)
Date: Sat, 11 Jun 2011 22:19:58 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1978328307.86.1307830798477.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J updated HADOOP-7328:
----------------------------

    Release Note:   (was: Improve SerializationFactory's handling of cases where a serializer isn't found given a class.)
          Status: Patch Available  (was: Open)

Please review patch at https://reviews.apache.org/r/884/

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16164-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 01:24:14 2011
Return-Path: <common-issues-return-16164-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D78D669D2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 01:24:14 +0000 (UTC)
Received: (qmail 41390 invoked by uid 500); 12 Jun 2011 01:24:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41355 invoked by uid 500); 12 Jun 2011 01:24:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41347 invoked by uid 99); 12 Jun 2011 01:24:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 01:24:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 01:24:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AF1174145B9
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 01:23:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 01:23:51 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1912848852.86.1307841831713.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <634368540.9715.1307685358867.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7375) Add resolvePath method to
 FileContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048193#comment-13048193 ] 

Eli Collins commented on HADOOP-7375:
-------------------------------------

Ah, never mind, I thought it was previously calling FC#getFileStatus.

+1  feel free to address the nits directly in the commit since it's just indentation.

> Add resolvePath method to FileContext
> -------------------------------------
>
>                 Key: HADOOP-7375
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7375
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: resolvePath1.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16165-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 01:38:13 2011
Return-Path: <common-issues-return-16165-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 01ACB6F74
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 01:38:13 +0000 (UTC)
Received: (qmail 48972 invoked by uid 500); 12 Jun 2011 01:38:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48943 invoked by uid 500); 12 Jun 2011 01:38:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48935 invoked by uid 99); 12 Jun 2011 01:38:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 01:38:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 01:38:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 93B024147E3
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 01:37:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 01:37:51 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1019680227.96.1307842671601.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <634368540.9715.1307685358867.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7375) Add resolvePath method to
 FileContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7375:
--------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I've committed this. Thanks Sanjay.

> Add resolvePath method to FileContext
> -------------------------------------
>
>                 Key: HADOOP-7375
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7375
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: resolvePath1.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16166-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 01:56:14 2011
Return-Path: <common-issues-return-16166-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D432D649E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 01:56:14 +0000 (UTC)
Received: (qmail 54051 invoked by uid 500); 12 Jun 2011 01:56:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53940 invoked by uid 500); 12 Jun 2011 01:56:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53932 invoked by uid 99); 12 Jun 2011 01:56:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 01:56:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 01:56:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A3A57414A47
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 01:55:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 01:55:51 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2052603422.104.1307843751667.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <634368540.9715.1307685358867.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7375) Add resolvePath method to
 FileContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048196#comment-13048196 ] 

Hudson commented on HADOOP-7375:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #647 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/647/])
    HADOOP-7375. Add resolvePath method to FileContext. Contributed by Sanjay Radia

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1134854
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/ViewFsBaseTest.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/FileContext.java


> Add resolvePath method to FileContext
> -------------------------------------
>
>                 Key: HADOOP-7375
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7375
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: resolvePath1.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16167-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 02:27:17 2011
Return-Path: <common-issues-return-16167-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6338A496D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 02:27:17 +0000 (UTC)
Received: (qmail 63831 invoked by uid 500); 12 Jun 2011 02:27:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63803 invoked by uid 500); 12 Jun 2011 02:27:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63795 invoked by uid 99); 12 Jun 2011 02:27:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 02:27:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 02:27:14 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AE2DE414E87
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 02:26:53 +0000 (UTC)
Date: Sun, 12 Jun 2011 02:26:53 +0000 (UTC)
From: "jiraposter@reviews.apache.org (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <528013048.117.1307845613709.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048199#comment-13048199 ] 

jiraposter@reviews.apache.org commented on HADOOP-7328:
-------------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/884/#review805
-----------------------------------------------------------

Ship it!


Looks good to me. Can you upload this rev of the patch to the JIRA so the QA Bot runs on it?

- Todd


On 2011-06-11 22:10:17, Harsh Chouraria wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/884/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2011-06-11 22:10:17)
bq.  
bq.  
bq.  Review request for hadoop-common and Todd Lipcon.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  Since getSerialization() can possibly return a null, it is only right that getSerializer() and getDeserializer() usage functions do the same, instead of throwing up NPEs.
bq.  
bq.  Related issue to which this improvement is required: https://issues.apache.org/jira/browse/MAPREDUCE-2584
bq.  
bq.  
bq.  This addresses bug HADOOP-7328.
bq.      http://issues.apache.org/jira/browse/HADOOP-7328
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    src/java/org/apache/hadoop/io/serializer/SerializationFactory.java dee314a 
bq.  
bq.  Diff: https://reviews.apache.org/r/884/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  Existing SequenceFile serialization factory tests pass. The change is merely to make the functions return null instead of throwing an NPE within.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Harsh
bq.  
bq.



> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16168-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 03:09:18 2011
Return-Path: <common-issues-return-16168-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A00EA4DF2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 03:09:18 +0000 (UTC)
Received: (qmail 82753 invoked by uid 500); 12 Jun 2011 03:09:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82698 invoked by uid 500); 12 Jun 2011 03:09:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82681 invoked by uid 99); 12 Jun 2011 03:09:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 03:09:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 03:09:13 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 51617414500
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 03:08:53 +0000 (UTC)
Date: Sun, 12 Jun 2011 03:08:53 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1695397989.143.1307848133330.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Moved] (HADOOP-7383) HDFS needs to export protobuf library
 dependency in pom
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon moved HDFS-2068 to HADOOP-7383:
-------------------------------------------

          Component/s:     (was: build)
                       build
        Fix Version/s:     (was: 0.23.0)
                       0.23.0
    Affects Version/s:     (was: 0.23.0)
                       0.23.0
                  Key: HADOOP-7383  (was: HDFS-2068)
              Project: Hadoop Common  (was: Hadoop HDFS)

> HDFS needs to export protobuf library dependency in pom
> -------------------------------------------------------
>
>                 Key: HADOOP-7383
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7383
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.23.0
>
>
> MR builds are failing since the HDFS protobuf patch went in, since they aren't picking up protobuf as a transitive dependency. I think we just need to add it to the HDFS pom template.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16169-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 03:15:17 2011
Return-Path: <common-issues-return-16169-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 513AC4F0E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 03:15:17 +0000 (UTC)
Received: (qmail 84422 invoked by uid 500); 12 Jun 2011 03:15:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84312 invoked by uid 500); 12 Jun 2011 03:15:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84304 invoked by uid 99); 12 Jun 2011 03:15:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 03:15:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 03:15:15 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2BB5A4146E4
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 03:14:55 +0000 (UTC)
Date: Sun, 12 Jun 2011 03:14:55 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <63164167.161.1307848495175.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7383) HDFS needs to export protobuf
 library dependency in pom
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7383:
--------------------------------

    Attachment: hadoop-7383.txt

Tested this patch with resolvers=internal, it fixed MR build for me.

> HDFS needs to export protobuf library dependency in pom
> -------------------------------------------------------
>
>                 Key: HADOOP-7383
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7383
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7383.txt
>
>
> MR builds are failing since the HDFS protobuf patch went in, since they aren't picking up protobuf as a transitive dependency. I think we just need to add it to the HDFS pom template.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16170-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 03:19:16 2011
Return-Path: <common-issues-return-16170-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EFBBC40E1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 03:19:16 +0000 (UTC)
Received: (qmail 85510 invoked by uid 500); 12 Jun 2011 03:19:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85280 invoked by uid 500); 12 Jun 2011 03:19:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85270 invoked by uid 99); 12 Jun 2011 03:19:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 03:19:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 03:19:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BA85041481C
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 03:18:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 03:18:51 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <143581848.172.1307848731761.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7383) HDFS needs to export protobuf
 library dependency in pom
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7383:
--------------------------------

    Hadoop Flags: [Reviewed]
          Status: Patch Available  (was: Open)

> HDFS needs to export protobuf library dependency in pom
> -------------------------------------------------------
>
>                 Key: HADOOP-7383
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7383
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7383.txt
>
>
> MR builds are failing since the HDFS protobuf patch went in, since they aren't picking up protobuf as a transitive dependency. I think we just need to add it to the HDFS pom template.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16171-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 03:19:17 2011
Return-Path: <common-issues-return-16171-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3C86340F7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 03:19:17 +0000 (UTC)
Received: (qmail 85531 invoked by uid 500); 12 Jun 2011 03:19:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85289 invoked by uid 500); 12 Jun 2011 03:19:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85279 invoked by uid 99); 12 Jun 2011 03:19:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 03:19:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 03:19:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9F5D641481A
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 03:18:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 03:18:51 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <824981297.170.1307848731649.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7383) HDFS needs to export protobuf
 library dependency in pom
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048210#comment-13048210 ] 

Eli Collins commented on HADOOP-7383:
-------------------------------------

+1  (rationale for 2.4.0a btw is that 2.4.1 isn't in maven yet, should be there soon)

> HDFS needs to export protobuf library dependency in pom
> -------------------------------------------------------
>
>                 Key: HADOOP-7383
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7383
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7383.txt
>
>
> MR builds are failing since the HDFS protobuf patch went in, since they aren't picking up protobuf as a transitive dependency. I think we just need to add it to the HDFS pom template.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16172-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 03:32:17 2011
Return-Path: <common-issues-return-16172-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3136645EC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 03:32:17 +0000 (UTC)
Received: (qmail 91786 invoked by uid 500); 12 Jun 2011 03:32:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91750 invoked by uid 500); 12 Jun 2011 03:32:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91739 invoked by uid 99); 12 Jun 2011 03:32:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 03:32:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 03:32:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8CFC0414996
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 03:31:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 03:31:51 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <821277200.176.1307849511574.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7383) HDFS needs to export protobuf
 library dependency in pom
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7383:
--------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I've committed this (before the Hudson results are in because in unbreaks the build). Thanks Todd!


> HDFS needs to export protobuf library dependency in pom
> -------------------------------------------------------
>
>                 Key: HADOOP-7383
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7383
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7383.txt
>
>
> MR builds are failing since the HDFS protobuf patch went in, since they aren't picking up protobuf as a transitive dependency. I think we just need to add it to the HDFS pom template.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16173-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 03:36:16 2011
Return-Path: <common-issues-return-16173-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 98FD24C13
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 03:36:16 +0000 (UTC)
Received: (qmail 92670 invoked by uid 500); 12 Jun 2011 03:36:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92503 invoked by uid 500); 12 Jun 2011 03:36:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92494 invoked by uid 99); 12 Jun 2011 03:36:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 03:36:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 03:36:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C3CC6414A60
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 03:35:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 03:35:51 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <75678170.180.1307849751798.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7383) HDFS needs to export protobuf
 library dependency in pom
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048212#comment-13048212 ] 

Hadoop QA commented on HADOOP-7383:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482271/hadoop-7383.txt
  against trunk revision 1134854.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/614//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/614//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/614//console

This message is automatically generated.

> HDFS needs to export protobuf library dependency in pom
> -------------------------------------------------------
>
>                 Key: HADOOP-7383
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7383
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7383.txt
>
>
> MR builds are failing since the HDFS protobuf patch went in, since they aren't picking up protobuf as a transitive dependency. I think we just need to add it to the HDFS pom template.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16174-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 03:46:19 2011
Return-Path: <common-issues-return-16174-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EADDD45BF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 03:46:18 +0000 (UTC)
Received: (qmail 99828 invoked by uid 500); 12 Jun 2011 03:46:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99767 invoked by uid 500); 12 Jun 2011 03:46:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99587 invoked by uid 99); 12 Jun 2011 03:46:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 03:46:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 03:46:13 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 51851414BF4
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 03:45:52 +0000 (UTC)
Date: Sun, 12 Jun 2011 03:45:52 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <522825405.205.1307850352330.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7383) HDFS needs to export protobuf
 library dependency in pom
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048214#comment-13048214 ] 

Hudson commented on HADOOP-7383:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #648 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/648/])
    HADOOP-7383. HDFS needs to export protobuf library dependency in pom. Contributed by Todd Lipcon

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1134857
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/ivy.xml
* /hadoop/common/trunk/ivy/hadoop-common-template.xml
* /hadoop/common/trunk/ivy/libraries.properties


> HDFS needs to export protobuf library dependency in pom
> -------------------------------------------------------
>
>                 Key: HADOOP-7383
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7383
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7383.txt
>
>
> MR builds are failing since the HDFS protobuf patch went in, since they aren't picking up protobuf as a transitive dependency. I think we just need to add it to the HDFS pom template.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16175-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 03:54:17 2011
Return-Path: <common-issues-return-16175-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0385942D0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 03:54:17 +0000 (UTC)
Received: (qmail 3025 invoked by uid 500); 12 Jun 2011 03:54:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2898 invoked by uid 500); 12 Jun 2011 03:54:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2720 invoked by uid 99); 12 Jun 2011 03:54:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 03:54:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 03:54:13 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 23F57414E29
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 03:53:52 +0000 (UTC)
Date: Sun, 12 Jun 2011 03:53:52 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <693132022.253.1307850832144.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <308187203.8772.1307656679014.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7374) hadoop-config.sh shouldn't add
 tools.jar to the classpath
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048218#comment-13048218 ] 

Todd Lipcon commented on HADOOP-7374:
-------------------------------------

+1

> hadoop-config.sh shouldn't add tools.jar to the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-7374
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7374
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>         Attachments: hadoop-7374-1.patch
>
>
> bin/hadoop-config.sh (and bin/rcc) add lib/tools.jar from JAVA_HOME to the classpath. This has been there since the initial commit of bin/hadoop, but I don't think it's needed. *Executing* Hadoop does not depend on tools.jar (or other libraries only available in the JDK, not the JRE) so let's not automatically add it. Marking this as an incompatible change since a job could potentially have relied on Hadoop adding tools.jar to the CLASSPATH automatically (though such a job would not have run on a system that did not have JAVA_HOME point to a jdk). The build of course still requires a JDK.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16176-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 04:16:17 2011
Return-Path: <common-issues-return-16176-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 07C594DE2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 04:16:17 +0000 (UTC)
Received: (qmail 13149 invoked by uid 500); 12 Jun 2011 04:16:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13117 invoked by uid 500); 12 Jun 2011 04:16:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12979 invoked by uid 99); 12 Jun 2011 04:16:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 04:16:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 04:16:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8351E41427C
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 04:15:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 04:15:51 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1102242304.281.1307852151534.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <308187203.8772.1307656679014.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7374) Don't add tools.jar to the classpath
 when running Hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7374:
--------------------------------

    Fix Version/s: 0.23.0
     Release Note: The scripts that run Hadoop no longer automatically add tools.jar from the JDK to the classpath (if it is present). If your job depends on tools.jar in the JDK you will need to add this dependency in your job.
     Hadoop Flags: [Incompatible change, Reviewed]  (was: [Incompatible change])
          Summary: Don't add tools.jar to the classpath when running Hadoop  (was: hadoop-config.sh shouldn't add tools.jar to the classpath)

Thanks Todd! I've committed this.

> Don't add tools.jar to the classpath when running Hadoop
> --------------------------------------------------------
>
>                 Key: HADOOP-7374
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7374
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7374-1.patch
>
>
> bin/hadoop-config.sh (and bin/rcc) add lib/tools.jar from JAVA_HOME to the classpath. This has been there since the initial commit of bin/hadoop, but I don't think it's needed. *Executing* Hadoop does not depend on tools.jar (or other libraries only available in the JDK, not the JRE) so let's not automatically add it. Marking this as an incompatible change since a job could potentially have relied on Hadoop adding tools.jar to the CLASSPATH automatically (though such a job would not have run on a system that did not have JAVA_HOME point to a jdk). The build of course still requires a JDK.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16177-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 04:18:16 2011
Return-Path: <common-issues-return-16177-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C420147B6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 04:18:16 +0000 (UTC)
Received: (qmail 13585 invoked by uid 500); 12 Jun 2011 04:18:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13546 invoked by uid 500); 12 Jun 2011 04:18:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13534 invoked by uid 99); 12 Jun 2011 04:18:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 04:18:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 04:18:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B77BE4142D5
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 04:17:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 04:17:51 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1873673710.285.1307852271748.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <308187203.8772.1307656679014.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7374) Don't add tools.jar to the classpath
 when running Hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7374:
--------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

> Don't add tools.jar to the classpath when running Hadoop
> --------------------------------------------------------
>
>                 Key: HADOOP-7374
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7374
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7374-1.patch
>
>
> bin/hadoop-config.sh (and bin/rcc) add lib/tools.jar from JAVA_HOME to the classpath. This has been there since the initial commit of bin/hadoop, but I don't think it's needed. *Executing* Hadoop does not depend on tools.jar (or other libraries only available in the JDK, not the JRE) so let's not automatically add it. Marking this as an incompatible change since a job could potentially have relied on Hadoop adding tools.jar to the CLASSPATH automatically (though such a job would not have run on a system that did not have JAVA_HOME point to a jdk). The build of course still requires a JDK.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16178-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 04:20:15 2011
Return-Path: <common-issues-return-16178-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 62C1F4030
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 04:20:15 +0000 (UTC)
Received: (qmail 14275 invoked by uid 500); 12 Jun 2011 04:20:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14068 invoked by uid 500); 12 Jun 2011 04:20:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14046 invoked by uid 99); 12 Jun 2011 04:20:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 04:20:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 04:20:13 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 122514143B5
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 04:19:53 +0000 (UTC)
Date: Sun, 12 Jun 2011 04:19:53 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <962603696.291.1307852393070.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J updated HADOOP-7328:
----------------------------

    Attachment: HADOOP-7328.r2.diff

Reviewed patch attached.

(+ a couple of bad whitespace fixes)

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff, HADOOP-7328.r2.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16179-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 04:20:15 2011
Return-Path: <common-issues-return-16179-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7C8784035
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 04:20:15 +0000 (UTC)
Received: (qmail 14333 invoked by uid 500); 12 Jun 2011 04:20:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14077 invoked by uid 500); 12 Jun 2011 04:20:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14060 invoked by uid 99); 12 Jun 2011 04:20:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 04:20:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 04:20:13 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3A9D54143BA
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 04:19:53 +0000 (UTC)
Date: Sun, 12 Jun 2011 04:19:53 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <494261327.296.1307852393236.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J updated HADOOP-7328:
----------------------------

    Status: Open  (was: Patch Available)

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff, HADOOP-7328.r2.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16180-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 04:20:17 2011
Return-Path: <common-issues-return-16180-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1CE564056
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 04:20:17 +0000 (UTC)
Received: (qmail 14682 invoked by uid 500); 12 Jun 2011 04:20:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14584 invoked by uid 500); 12 Jun 2011 04:20:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14564 invoked by uid 99); 12 Jun 2011 04:20:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 04:20:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 04:20:14 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 46CD24143BC
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 04:19:53 +0000 (UTC)
Date: Sun, 12 Jun 2011 04:19:53 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1542507896.298.1307852393286.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J updated HADOOP-7328:
----------------------------

    Status: Patch Available  (was: Open)

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff, HADOOP-7328.r2.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16181-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 04:36:15 2011
Return-Path: <common-issues-return-16181-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A71254706
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 04:36:15 +0000 (UTC)
Received: (qmail 19925 invoked by uid 500); 12 Jun 2011 04:36:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19856 invoked by uid 500); 12 Jun 2011 04:36:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19844 invoked by uid 99); 12 Jun 2011 04:36:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 04:36:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 04:36:13 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4E31541473C
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 04:35:53 +0000 (UTC)
Date: Sun, 12 Jun 2011 04:35:53 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <446984878.316.1307853353317.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <308187203.8772.1307656679014.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7374) Don't add tools.jar to the
 classpath when running Hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048229#comment-13048229 ] 

Hudson commented on HADOOP-7374:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #649 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/649/])
    HADOOP-7374. Don't add tools.jar to the classpath when running Hadoop. Contributed by Eli Collins

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1134861
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/bin/hadoop-config.sh
* /hadoop/common/trunk/bin/rcc


> Don't add tools.jar to the classpath when running Hadoop
> --------------------------------------------------------
>
>                 Key: HADOOP-7374
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7374
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7374-1.patch
>
>
> bin/hadoop-config.sh (and bin/rcc) add lib/tools.jar from JAVA_HOME to the classpath. This has been there since the initial commit of bin/hadoop, but I don't think it's needed. *Executing* Hadoop does not depend on tools.jar (or other libraries only available in the JDK, not the JRE) so let's not automatically add it. Marking this as an incompatible change since a job could potentially have relied on Hadoop adding tools.jar to the CLASSPATH automatically (though such a job would not have run on a system that did not have JAVA_HOME point to a jdk). The build of course still requires a JDK.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16182-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 04:36:17 2011
Return-Path: <common-issues-return-16182-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 33F8E471B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 04:36:17 +0000 (UTC)
Received: (qmail 20505 invoked by uid 500); 12 Jun 2011 04:36:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20465 invoked by uid 500); 12 Jun 2011 04:36:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20446 invoked by uid 99); 12 Jun 2011 04:36:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 04:36:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 04:36:14 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 35DE0414739
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 04:35:53 +0000 (UTC)
Date: Sun, 12 Jun 2011 04:35:53 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <40917355.313.1307853353217.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048228#comment-13048228 ] 

Hadoop QA commented on HADOOP-7328:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482273/HADOOP-7328.r2.diff
  against trunk revision 1134861.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/615//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/615//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/615//console

This message is automatically generated.

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff, HADOOP-7328.r2.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16183-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 04:44:17 2011
Return-Path: <common-issues-return-16183-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CF6BE47F7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 04:44:17 +0000 (UTC)
Received: (qmail 31329 invoked by uid 500); 12 Jun 2011 04:44:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31300 invoked by uid 500); 12 Jun 2011 04:44:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31285 invoked by uid 99); 12 Jun 2011 04:44:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 04:44:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 04:44:14 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1A475414953
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 04:43:53 +0000 (UTC)
Date: Sun, 12 Jun 2011 04:43:53 +0000 (UTC)
From: "jiraposter@reviews.apache.org (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1398361038.339.1307853833104.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048235#comment-13048235 ] 

jiraposter@reviews.apache.org commented on HADOOP-7328:
-------------------------------------------------------



bq.  On 2011-06-12 02:24:55, Todd Lipcon wrote:
bq.  > Looks good to me. Can you upload this rev of the patch to the JIRA so the QA Bot runs on it?

Submitted on JIRA. Thanks for the review Todd!


- Harsh


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/884/#review805
-----------------------------------------------------------


On 2011-06-11 22:10:17, Harsh J wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/884/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2011-06-11 22:10:17)
bq.  
bq.  
bq.  Review request for hadoop-common and Todd Lipcon.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  Since getSerialization() can possibly return a null, it is only right that getSerializer() and getDeserializer() usage functions do the same, instead of throwing up NPEs.
bq.  
bq.  Related issue to which this improvement is required: https://issues.apache.org/jira/browse/MAPREDUCE-2584
bq.  
bq.  
bq.  This addresses bug HADOOP-7328.
bq.      http://issues.apache.org/jira/browse/HADOOP-7328
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    src/java/org/apache/hadoop/io/serializer/SerializationFactory.java dee314a 
bq.  
bq.  Diff: https://reviews.apache.org/r/884/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  Existing SequenceFile serialization factory tests pass. The change is merely to make the functions return null instead of throwing an NPE within.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Harsh
bq.  
bq.



> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff, HADOOP-7328.r2.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16184-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 05:10:16 2011
Return-Path: <common-issues-return-16184-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D0DBC456C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 05:10:16 +0000 (UTC)
Received: (qmail 39128 invoked by uid 500); 12 Jun 2011 05:10:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39000 invoked by uid 500); 12 Jun 2011 05:10:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38981 invoked by uid 99); 12 Jun 2011 05:10:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 05:10:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 05:10:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B7D55414010
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 05:09:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 05:09:51 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <590218562.378.1307855391749.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Updated] (HADOOP-7106) Re-organize hadoop subversion layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7106:
--------------------------------

    Attachment: gitk-example.png
                HADOOP-7106.sh
                HADOOP-7106-git.sh

Here are the final (I think) versions of the scripts to use. A few changes since previous one:
- uses shopt -s dotglob since the previous version accidentally left common's .gitignore and .eclipse-templates in the root dir
- does combined trees for MR-279, HDFS-1073, yahoo-merge
- had to redo the order so those trees come before "trunk" - otherwise for some reason git-svn's tracking got messed up and HDFS-1073 ended up with a common subtree missing all of its subdirectories except for src

Changes to the git script:
- supports merging trees from upstream branches named different things (for MR-279 and HDFS-1073)
- does a sanity check using git-ls-tree to make sure that the tree checksum in the combined repo matches exactly the tree checksum of the last pre-7106 commit in the old repo (this is how I found the problem with the dotfiles and the missing dirs)

I've also attached a gitk screenshot showing what it looks like in the git mirror

> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, gitk-example.png
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16185-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 06:04:14 2011
Return-Path: <common-issues-return-16185-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 97B804DC1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 06:04:14 +0000 (UTC)
Received: (qmail 53141 invoked by uid 500); 12 Jun 2011 06:04:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53115 invoked by uid 500); 12 Jun 2011 06:04:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53089 invoked by uid 99); 12 Jun 2011 06:04:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 06:04:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 06:04:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8C5DD414A58
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 06:03:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 06:03:51 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <67823723.448.1307858631571.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6605?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6605:
--------------------------------

    Attachment: hadoop-6605-3.patch

Thanks for the feedback everyone. Popping the stack, Hadoop requires the us=
er set JAVA_HOME for two reasons:

# We want to add tools.jar to the classpath, and JAVA_HOME let's the user s=
pecify a base directory to look (other than the default java which may be f=
rom a JRE and therefore not have tools.jar). This is no longer an issue sin=
ce HADOOP-7374 removed it.
# We want to respect JAVA_HOME even if there is already a java in the path.=
 Ie users and admins can easily configure which java should be used with Ha=
doop that's different from the default system java. This makes sense given =
that Hadoop is picky. Therefore it makes sense to only auto-detect JAVA_HOM=
E if it is not set (which all versions of the patch do) and we can determin=
e a reasonable value.

On OSX, they provide an API (java_home(1)) that does this (returns a path s=
uitable for setting JAVA_HOME based on enabled/preferred JVM'S as set by Ja=
va Preferences). I think we agree it makes sense to use this.

On Linux, there is no single API that works across distributions. Even thou=
gh alternatives is widely available it works differently on different distr=
iubtions (also, it indicates where the java binary lives, not where JAVA_HO=
ME is, though you could determine that with readlink). There are well-known=
 locations where JAVA_HOME is installed that you can check to reasonably de=
tect it. This is the approach taken by the previous patch. I've provided da=
ta that shows that checking a set of directories does not measurably impact=
 the execution time (therefore "too much work" sounds like a philosophical =
objection rather than a technical objection to me). I've found that globbin=
g is not an issue in practice because the glob does not match more than one=
 installation on a given system. This is because the JDK was resolved via a=
 packaging dependency and the package updates itself rather than having mul=
tiple versions installed. People who manually install multiple JDKs typical=
ly set JAVA_HOME explicitly and therefore the detection is not used. There =
are no alternative proposals for autodetecting JAVA_HOME on Linux, and I'm =
not going to spend any more time on this part for now so I'm dropping this =
case from the patch.

In any case (ha), there is consensus on the OSX approach so let's just go w=
ith this for now. We can easily implement cases for other OS types in the f=
uture if there's an approach that's acceptable. Patch attached.

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch, hadoop-6605-=
2.patch, hadoop-6605-3.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if=
 JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on =
various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the e=
nvironment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

From common-issues-return-16186-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 06:08:16 2011
Return-Path: <common-issues-return-16186-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B51E14E54
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 06:08:16 +0000 (UTC)
Received: (qmail 53583 invoked by uid 500); 12 Jun 2011 06:08:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53552 invoked by uid 500); 12 Jun 2011 06:08:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53535 invoked by uid 99); 12 Jun 2011 06:08:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 06:08:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 06:08:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 89977414B26
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 06:07:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 06:07:51 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <293728502.457.1307858871560.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6605:
--------------------------------

    Status: Patch Available  (was: Open)

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch, hadoop-6605-2.patch, hadoop-6605-3.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16187-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 06:24:21 2011
Return-Path: <common-issues-return-16187-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DA0AF44F9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 06:24:21 +0000 (UTC)
Received: (qmail 56850 invoked by uid 500); 12 Jun 2011 06:24:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56653 invoked by uid 500); 12 Jun 2011 06:24:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56621 invoked by uid 99); 12 Jun 2011 06:24:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 06:24:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 06:24:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7D839414D9C
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 06:23:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 06:23:51 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <571160235.470.1307859831510.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048252#comment-13048252 ] 

Hadoop QA commented on HADOOP-6605:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482283/hadoop-6605-3.patch
  against trunk revision 1134861.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/616//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/616//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/616//console

This message is automatically generated.

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch, hadoop-6605-2.patch, hadoop-6605-3.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16188-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 09:06:14 2011
Return-Path: <common-issues-return-16188-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0B73A4D7B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 09:06:14 +0000 (UTC)
Received: (qmail 63922 invoked by uid 500); 12 Jun 2011 09:06:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63888 invoked by uid 500); 12 Jun 2011 09:06:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63867 invoked by uid 99); 12 Jun 2011 09:06:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 09:06:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 09:06:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B1EA841534B
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 09:05:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 09:05:51 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <574635695.623.1307869551725.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <634368540.9715.1307685358867.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7375) Add resolvePath method to
 FileContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048274#comment-13048274 ] 

Hudson commented on HADOOP-7375:
--------------------------------

Integrated in Hadoop-Common-trunk-maven #13 (See [https://builds.apache.org/job/Hadoop-Common-trunk-maven/13/])
    HADOOP-7375. Add resolvePath method to FileContext. Contributed by Sanjay Radia

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1134854
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/ViewFsBaseTest.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/FileContext.java


> Add resolvePath method to FileContext
> -------------------------------------
>
>                 Key: HADOOP-7375
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7375
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: resolvePath1.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16190-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 09:06:14 2011
Return-Path: <common-issues-return-16190-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 25CEE4D84
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 09:06:14 +0000 (UTC)
Received: (qmail 64042 invoked by uid 500); 12 Jun 2011 09:06:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63925 invoked by uid 500); 12 Jun 2011 09:06:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63880 invoked by uid 99); 12 Jun 2011 09:06:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 09:06:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 09:06:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C895F41534D
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 09:05:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 09:05:51 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <662307779.625.1307869551818.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <308187203.8772.1307656679014.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7374) Don't add tools.jar to the
 classpath when running Hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048275#comment-13048275 ] 

Hudson commented on HADOOP-7374:
--------------------------------

Integrated in Hadoop-Common-trunk-maven #13 (See [https://builds.apache.org/job/Hadoop-Common-trunk-maven/13/])
    HADOOP-7374. Don't add tools.jar to the classpath when running Hadoop. Contributed by Eli Collins

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1134861
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/bin/hadoop-config.sh
* /hadoop/common/trunk/bin/rcc


> Don't add tools.jar to the classpath when running Hadoop
> --------------------------------------------------------
>
>                 Key: HADOOP-7374
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7374
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7374-1.patch
>
>
> bin/hadoop-config.sh (and bin/rcc) add lib/tools.jar from JAVA_HOME to the classpath. This has been there since the initial commit of bin/hadoop, but I don't think it's needed. *Executing* Hadoop does not depend on tools.jar (or other libraries only available in the JDK, not the JRE) so let's not automatically add it. Marking this as an incompatible change since a job could potentially have relied on Hadoop adding tools.jar to the CLASSPATH automatically (though such a job would not have run on a system that did not have JAVA_HOME point to a jdk). The build of course still requires a JDK.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16189-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 09:06:14 2011
Return-Path: <common-issues-return-16189-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0C5FB4D7E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 09:06:14 +0000 (UTC)
Received: (qmail 63964 invoked by uid 500); 12 Jun 2011 09:06:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63892 invoked by uid 500); 12 Jun 2011 09:06:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63863 invoked by uid 99); 12 Jun 2011 09:06:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 09:06:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 09:06:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 96CAB415348
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 09:05:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 09:05:51 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1763869152.620.1307869551614.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7383) HDFS needs to export protobuf
 library dependency in pom
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048273#comment-13048273 ] 

Hudson commented on HADOOP-7383:
--------------------------------

Integrated in Hadoop-Common-trunk-maven #13 (See [https://builds.apache.org/job/Hadoop-Common-trunk-maven/13/])
    HADOOP-7383. HDFS needs to export protobuf library dependency in pom. Contributed by Todd Lipcon

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1134857
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/ivy.xml
* /hadoop/common/trunk/ivy/hadoop-common-template.xml
* /hadoop/common/trunk/ivy/libraries.properties


> HDFS needs to export protobuf library dependency in pom
> -------------------------------------------------------
>
>                 Key: HADOOP-7383
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7383
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7383.txt
>
>
> MR builds are failing since the HDFS protobuf patch went in, since they aren't picking up protobuf as a transitive dependency. I think we just need to add it to the HDFS pom template.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16191-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 10:00:17 2011
Return-Path: <common-issues-return-16191-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0B86144B1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 10:00:17 +0000 (UTC)
Received: (qmail 94113 invoked by uid 500); 12 Jun 2011 10:00:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94085 invoked by uid 500); 12 Jun 2011 10:00:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94073 invoked by uid 99); 12 Jun 2011 10:00:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 10:00:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 10:00:14 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C11D1415B76
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 09:59:52 +0000 (UTC)
Date: Sun, 12 Jun 2011 09:59:52 +0000 (UTC)
From: "Eric Charles (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <94840952.693.1307872792787.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048284#comment-13048284 ] 

Eric Charles commented on HADOOP-6671:
--------------------------------------

Hi,
Jenkins haddop-common-trunk is blue (builds fine).
I've downloaded https://builds.apache.org/job/Hadoop-Common-trunk-maven/13/artifact/trunk/target/hadoop-common-0.23.0-SNAPSHOT.tar.gz
and imported it in eclipse via m2eclipse. Still need to add 3 folders in my build path to have operation project in eclipse: target/generated-sources/avro-protocol, avro-schema and record-cc (Maybe this can be solved with the build-helper-maven-plugin : http://www.sonatype.com/people/2008/05/adding-additional-source-folders-to-your-maven-build/)

When I run "mvn test" in shell, I've got two failures in TestUserGroupInformation [1], however the TestUserGroupInformation run from eclipse works fine.
Seems like UserGroupInformation behaves differently in shell than in eclipse. Same issue if I run "mvn test -P os.mac" (I run on mac).

[1]
-------------------------------------------------------------------------------
Test set: org.apache.hadoop.security.TestUserGroupInformation
-------------------------------------------------------------------------------
Tests run: 14, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.389 sec <<< FAILURE!
testGetServerSideGroups(org.apache.hadoop.security.TestUserGroupInformation)  Time elapsed: 0.029 sec  <<< FAILURE!
java.lang.AssertionError: expected:<13> but was:<0>
	at org.junit.Assert.fail(Assert.java:91)
	at org.junit.Assert.failNotEquals(Assert.java:645)
	at org.junit.Assert.assertEquals(Assert.java:126)
	at org.junit.Assert.assertEquals(Assert.java:470)
	at org.junit.Assert.assertEquals(Assert.java:454)
	at org.apache.hadoop.security.TestUserGroupInformation.testGetServerSideGroups(TestUserGroupInformation.java:97)
...
testLogin(org.apache.hadoop.security.TestUserGroupInformation)  Time elapsed: 0 sec  <<< FAILURE!
java.lang.AssertionError: 
	at org.junit.Assert.fail(Assert.java:91)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertTrue(Assert.java:54)
	at org.apache.hadoop.security.TestUserGroupInformation.testLogin(TestUserGroupInformation.java:122)
...

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-6671-cross-project-HDFS.patch, HADOOP-6671-e.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16192-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 11:15:13 2011
Return-Path: <common-issues-return-16192-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2E3154699
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 11:15:13 +0000 (UTC)
Received: (qmail 35119 invoked by uid 500); 12 Jun 2011 11:15:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35086 invoked by uid 500); 12 Jun 2011 11:15:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35078 invoked by uid 99); 12 Jun 2011 11:15:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 11:15:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 11:15:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B33E8414B1C
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 11:14:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 11:14:51 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <260028992.737.1307877291730.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7383) HDFS needs to export protobuf
 library dependency in pom
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048295#comment-13048295 ] 

Hudson commented on HADOOP-7383:
--------------------------------

Integrated in Hadoop-Common-trunk #717 (See [https://builds.apache.org/job/Hadoop-Common-trunk/717/])
    HADOOP-7383. HDFS needs to export protobuf library dependency in pom. Contributed by Todd Lipcon

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1134857
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/ivy.xml
* /hadoop/common/trunk/ivy/hadoop-common-template.xml
* /hadoop/common/trunk/ivy/libraries.properties


> HDFS needs to export protobuf library dependency in pom
> -------------------------------------------------------
>
>                 Key: HADOOP-7383
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7383
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7383.txt
>
>
> MR builds are failing since the HDFS protobuf patch went in, since they aren't picking up protobuf as a transitive dependency. I think we just need to add it to the HDFS pom template.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16193-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 11:15:13 2011
Return-Path: <common-issues-return-16193-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9C4CF46A9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 11:15:13 +0000 (UTC)
Received: (qmail 35397 invoked by uid 500); 12 Jun 2011 11:15:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35325 invoked by uid 500); 12 Jun 2011 11:15:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35258 invoked by uid 99); 12 Jun 2011 11:15:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 11:15:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 11:15:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E1341414B21
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 11:14:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 11:14:51 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1040931579.741.1307877291919.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <634368540.9715.1307685358867.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7375) Add resolvePath method to
 FileContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048297#comment-13048297 ] 

Hudson commented on HADOOP-7375:
--------------------------------

Integrated in Hadoop-Common-trunk #717 (See [https://builds.apache.org/job/Hadoop-Common-trunk/717/])
    HADOOP-7375. Add resolvePath method to FileContext. Contributed by Sanjay Radia

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1134854
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/ViewFsBaseTest.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/FileContext.java


> Add resolvePath method to FileContext
> -------------------------------------
>
>                 Key: HADOOP-7375
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7375
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: resolvePath1.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16194-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 11:15:15 2011
Return-Path: <common-issues-return-16194-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4C82646EA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 11:15:15 +0000 (UTC)
Received: (qmail 36881 invoked by uid 500); 12 Jun 2011 11:15:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36844 invoked by uid 500); 12 Jun 2011 11:15:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36836 invoked by uid 99); 12 Jun 2011 11:15:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 11:15:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 11:15:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id ED86A414B22
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 11:14:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 11:14:51 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1535259791.742.1307877291969.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <308187203.8772.1307656679014.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7374) Don't add tools.jar to the
 classpath when running Hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048298#comment-13048298 ] 

Hudson commented on HADOOP-7374:
--------------------------------

Integrated in Hadoop-Common-trunk #717 (See [https://builds.apache.org/job/Hadoop-Common-trunk/717/])
    HADOOP-7374. Don't add tools.jar to the classpath when running Hadoop. Contributed by Eli Collins

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1134861
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/bin/hadoop-config.sh
* /hadoop/common/trunk/bin/rcc


> Don't add tools.jar to the classpath when running Hadoop
> --------------------------------------------------------
>
>                 Key: HADOOP-7374
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7374
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7374-1.patch
>
>
> bin/hadoop-config.sh (and bin/rcc) add lib/tools.jar from JAVA_HOME to the classpath. This has been there since the initial commit of bin/hadoop, but I don't think it's needed. *Executing* Hadoop does not depend on tools.jar (or other libraries only available in the JDK, not the JRE) so let's not automatically add it. Marking this as an incompatible change since a job could potentially have relied on Hadoop adding tools.jar to the CLASSPATH automatically (though such a job would not have run on a system that did not have JAVA_HOME point to a jdk). The build of course still requires a JDK.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16195-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 14:21:12 2011
Return-Path: <common-issues-return-16195-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CF2094265
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 14:21:12 +0000 (UTC)
Received: (qmail 30207 invoked by uid 500); 12 Jun 2011 14:21:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30177 invoked by uid 500); 12 Jun 2011 14:21:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30169 invoked by uid 99); 12 Jun 2011 14:21:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 14:21:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 14:21:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9207E415BAB
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 14:20:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 14:20:51 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1866786569.917.1307888451594.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6936) broken links in
 http://wiki.apache.org/hadoop/FAQ#A12//s
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048331#comment-13048331 ] 

Harsh J commented on HADOOP-6936:
---------------------------------

Fixed the FAQ links.

The HowToConfigure page is highly outdated. I do not know if it still serves a purpose beyond the tutorial (which explains how to too). It could be possibly set up to redirect to a current documentation.

But as Nicholas notes, this is all doable by anyone signed into the Wiki (free account signups, just gotta enter captchas). Filing a JIRA for this is not required.

> broken links in http://wiki.apache.org/hadoop/FAQ#A12//s
> --------------------------------------------------------
>
>                 Key: HADOOP-6936
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6936
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>            Reporter: Eugene Koontz
>            Priority: Trivial
>
> http://wiki.apache.org/hadoop/FAQ#A12//s 
> has links to :
> http://hadoop.apache.org/core/docs/current/hadoop-default.html#dfs.replication.min
> http://hadoop.apache.org/common/docs/current/hadoop-default.html#dfs.safemode.threshold.pct
> both of which are 404 as of time of filing this issue.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16196-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 14:25:14 2011
Return-Path: <common-issues-return-16196-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BDEA442C7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 14:25:14 +0000 (UTC)
Received: (qmail 31250 invoked by uid 500); 12 Jun 2011 14:25:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31188 invoked by uid 500); 12 Jun 2011 14:25:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31179 invoked by uid 99); 12 Jun 2011 14:25:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 14:25:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 14:25:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8D3F0415D2E
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 14:24:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 14:24:51 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <142929921.924.1307888691575.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6574) Commit HADOOP-6414:Add command
 line help for -expunge command.    to Hadoop 0.20
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048334#comment-13048334 ] 

Harsh J commented on HADOOP-6574:
---------------------------------

Ravi, is this still required?

If it is, please list it as a fix version against the upcoming 0.20.x release (204?) and post a new patch if required?

> Commit HADOOP-6414:Add command line help for -expunge command.    to Hadoop 0.20 
> ---------------------------------------------------------------------------------
>
>                 Key: HADOOP-6574
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6574
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.0
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>            Priority: Trivial
>         Attachments: HADOOP-6574.patch
>
>
> HADOOP-6414 : Add command line help for -expunge command. needs to be committed to Hadoop 0.20.
> Opening this new Jira to address this issue.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16197-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 14:31:17 2011
Return-Path: <common-issues-return-16197-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AF2344B9B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 14:31:17 +0000 (UTC)
Received: (qmail 38601 invoked by uid 500); 12 Jun 2011 14:31:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38570 invoked by uid 500); 12 Jun 2011 14:31:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38562 invoked by uid 99); 12 Jun 2011 14:31:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 14:31:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 14:31:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 81E72415EB2
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 14:30:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 14:30:51 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1457617233.932.1307889051528.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6406) hadoop-core.pom contains hardcoded
 and nonexistent commons-cli version and jetty groupId/artifactId mixup
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048337#comment-13048337 ] 

Harsh J commented on HADOOP-6406:
---------------------------------

Marcel,

Is this still valid? I do not see a hadoop-core.pom file anywhere?

> hadoop-core.pom contains hardcoded and nonexistent commons-cli version and jetty groupId/artifactId mixup
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6406
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6406
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Marcel  May
>            Priority: Trivial
>         Attachments: HADOOP-6406.patch
>
>
> hadoop-core.pom in trunk contains
> a) hardcoded non-existing commons-cli version "2.0-20070823"
> b) jetty groupId/artifactId mixup : the given artifactId is the groupId, the groupId is the artifactId

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16198-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 15:07:15 2011
Return-Path: <common-issues-return-16198-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0683A4ACB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 15:07:15 +0000 (UTC)
Received: (qmail 65720 invoked by uid 500); 12 Jun 2011 15:07:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65681 invoked by uid 500); 12 Jun 2011 15:07:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65673 invoked by uid 99); 12 Jun 2011 15:07:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 15:07:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 15:07:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 95E004155D4
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 15:06:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 15:06:51 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <955728521.963.1307891211610.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1311101848.10328.1300399589594.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7197) Support for non-standard ssh port
 for slaves
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048344#comment-13048344 ] 

Harsh J commented on HADOOP-7197:
---------------------------------

I think ~/.ssh/config changes make it a better approach for the site on the whole as well.

> Support for non-standard ssh port for slaves
> --------------------------------------------
>
>                 Key: HADOOP-7197
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7197
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: util
>    Affects Versions: 0.20.2
>         Environment: FreeBSD 8.2-RELEASE
>            Reporter: Vrachnis Ilias-Dimitrios
>            Priority: Trivial
>              Labels: hadoop
>         Attachments: slaves.sh.patch
>
>
> I was trying to add a slave that ran sshd in a non-standard port (eg. 2222 in stead of 22), when I noticed that there was no way to support another port through the configuration for a single node.
> Supporting a different port for all the slaves is possible through the HADOOP_SSH_OPTS variable, but not for a single slave.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16199-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 15:15:25 2011
Return-Path: <common-issues-return-16199-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AEF8F4D1B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 15:15:25 +0000 (UTC)
Received: (qmail 69598 invoked by uid 500); 12 Jun 2011 15:15:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69551 invoked by uid 500); 12 Jun 2011 15:15:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69341 invoked by uid 99); 12 Jun 2011 15:15:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 15:15:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 15:15:23 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 52E71415760
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 15:15:03 +0000 (UTC)
Date: Sun, 12 Jun 2011 15:15:03 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1440412372.968.1307891703336.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-6251) Incorrect sample code in javadoc
 for org.apache.hadoop.mapreduce.Mapper
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J resolved HADOOP-6251.
-----------------------------

    Resolution: Duplicate

Already fixed in MAPREDUCE-1407.

This issue duplicates MAPREDUCE-1407

> Incorrect sample code in javadoc for org.apache.hadoop.mapreduce.Mapper
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-6251
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6251
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.0
>            Reporter: Yasuko Komiyama
>            Priority: Trivial
>
> In the javadoc description about mapreduce.Mapper class, there is an example code "TokenCounterMapper", and it calls context.collect() in the map() method.  It should call context.write().
>   context.collect(word, one);  ---->  context.write(word, one);

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16200-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 15:19:15 2011
Return-Path: <common-issues-return-16200-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 25A9F4EA8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 15:19:15 +0000 (UTC)
Received: (qmail 74151 invoked by uid 500); 12 Jun 2011 15:19:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74107 invoked by uid 500); 12 Jun 2011 15:19:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74070 invoked by uid 99); 12 Jun 2011 15:19:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 15:19:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 15:19:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id ACD4F4158A1
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 15:18:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 15:18:51 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <132484621.970.1307891931704.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-6238) (very) minor typo in javadoc for
 org.apache.hadoop.util.Tool
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J resolved HADOOP-6238.
-----------------------------

    Resolution: Duplicate

Already fixed by HADOOP-6504

Issue duplicates HADOOP-6504

> (very) minor typo in javadoc for org.apache.hadoop.util.Tool
> ------------------------------------------------------------
>
>                 Key: HADOOP-6238
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6238
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.20.0
>            Reporter: Parker Jones
>            Priority: Trivial
>   Original Estimate: 0.25h
>  Remaining Estimate: 0.25h
>
> In the description, "Here is how a typical Tool is implemented:" there's some sample code for a class called "MyApp" that gives a simple implementation of Tool.  In its main method, it references a class "Sort" where I believe MyApp should appear:
>          int res = ToolRunner.run(new Configuration(), new Sort(), args);
> should be
>          int res = ToolRunner.run(new Configuration(), new MyApp(), args);
> I'm brand-new to hadoop (Day 1) and was initially confused how ToolRunner knew which class to run.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16201-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 15:29:15 2011
Return-Path: <common-issues-return-16201-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1A7C1467B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 15:29:15 +0000 (UTC)
Received: (qmail 88678 invoked by uid 500); 12 Jun 2011 15:29:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88648 invoked by uid 500); 12 Jun 2011 15:29:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88640 invoked by uid 99); 12 Jun 2011 15:29:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 15:29:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 15:29:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DE80B415BFD
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 15:28:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 15:28:51 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1475913333.990.1307892531908.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6219) Add dumpConfiguration option in
 hadoop help message
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6219?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J updated HADOOP-6219:
----------------------------

    Attachment: MAPREDUCE-919.patch

Reuploading Chaitanya's patch to mark as PA and get QA to run the right (la=
test timestamp) one instead of Yahoo's internal patches=E2=80=A6

> Add dumpConfiguration option in hadoop help message
> ---------------------------------------------------
>
>                 Key: HADOOP-6219
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6219
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.1
>            Reporter: Ramya R
>            Assignee: V.V.Chaitanya Krishna
>            Priority: Trivial
>         Attachments: HADOOP-6184-ydist.patch, HADOOP-6219-ydist.patch, MA=
PREDUCE-919.patch, MAPREDUCE-919.patch
>
>
> Execution of bin/hadoop should show the -dumpConfiguration option introdu=
ced in MAPREDUCE-768

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

From common-issues-return-16202-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 15:31:12 2011
Return-Path: <common-issues-return-16202-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9D4E14D56
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 15:31:12 +0000 (UTC)
Received: (qmail 89195 invoked by uid 500); 12 Jun 2011 15:31:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89154 invoked by uid 500); 12 Jun 2011 15:31:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89146 invoked by uid 99); 12 Jun 2011 15:31:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 15:31:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 15:31:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8279C415CCA
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 15:30:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 15:30:51 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <325266518.994.1307892651531.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6219) Add dumpConfiguration option in
 hadoop help message
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J updated HADOOP-6219:
----------------------------

    Fix Version/s: 0.23.0
           Status: Patch Available  (was: Open)

Marking as PA. Patch still applies clean to trunk (1 line offset).

> Add dumpConfiguration option in hadoop help message
> ---------------------------------------------------
>
>                 Key: HADOOP-6219
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6219
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.1
>            Reporter: Ramya R
>            Assignee: V.V.Chaitanya Krishna
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-6184-ydist.patch, HADOOP-6219-ydist.patch, MAPREDUCE-919.patch, MAPREDUCE-919.patch
>
>
> Execution of bin/hadoop should show the -dumpConfiguration option introduced in MAPREDUCE-768

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16203-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 15:31:14 2011
Return-Path: <common-issues-return-16203-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D68CD4D8A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 15:31:14 +0000 (UTC)
Received: (qmail 89458 invoked by uid 500); 12 Jun 2011 15:31:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89414 invoked by uid 500); 12 Jun 2011 15:31:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89406 invoked by uid 99); 12 Jun 2011 15:31:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 15:31:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 15:31:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A5F2B415CCE
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 15:30:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 15:30:51 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <484746731.998.1307892651676.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6219) Add dumpConfiguration option in
 hadoop help message
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J updated HADOOP-6219:
----------------------------

    Resolution: Duplicate
        Status: Resolved  (was: Patch Available)

This issue is a duplicate of MAPREDUCE-919

> Add dumpConfiguration option in hadoop help message
> ---------------------------------------------------
>
>                 Key: HADOOP-6219
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6219
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.1
>            Reporter: Ramya R
>            Assignee: V.V.Chaitanya Krishna
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-6184-ydist.patch, HADOOP-6219-ydist.patch, MAPREDUCE-919.patch, MAPREDUCE-919.patch
>
>
> Execution of bin/hadoop should show the -dumpConfiguration option introduced in MAPREDUCE-768

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16204-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 15:33:14 2011
Return-Path: <common-issues-return-16204-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CFD0548E4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 15:33:14 +0000 (UTC)
Received: (qmail 91925 invoked by uid 500); 12 Jun 2011 15:33:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91893 invoked by uid 500); 12 Jun 2011 15:33:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91885 invoked by uid 99); 12 Jun 2011 15:33:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 15:33:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 15:33:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 94527415DF8
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 15:32:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 15:32:51 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <145536999.1002.1307892771604.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Reopened] (HADOOP-6219) Add dumpConfiguration option in
 hadoop help message
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J reopened HADOOP-6219:
-----------------------------


Sorry, pretty strange that both link to same issue? Ideally it should be under mapred project now.

> Add dumpConfiguration option in hadoop help message
> ---------------------------------------------------
>
>                 Key: HADOOP-6219
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6219
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.1
>            Reporter: Ramya R
>            Assignee: V.V.Chaitanya Krishna
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-6184-ydist.patch, HADOOP-6219-ydist.patch, MAPREDUCE-919.patch, MAPREDUCE-919.patch
>
>
> Execution of bin/hadoop should show the -dumpConfiguration option introduced in MAPREDUCE-768

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16205-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 15:57:13 2011
Return-Path: <common-issues-return-16205-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C906A4A75
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 15:57:13 +0000 (UTC)
Received: (qmail 18178 invoked by uid 500); 12 Jun 2011 15:57:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18119 invoked by uid 500); 12 Jun 2011 15:57:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17796 invoked by uid 99); 12 Jun 2011 15:57:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 15:57:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 15:57:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8C093415426
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 15:56:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 15:56:51 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <229916015.1047.1307894211570.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-5624) @Override cleanup for Eclipse
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-5624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J resolved HADOOP-5624.
-----------------------------

    Resolution: Not A Problem

Eclipse does not seem to complain for any of the patch's changes.

I do not see any @Override issues on trunk of all three projects right now. Please do not hesitate to re-open in case it is still an issue as per.

> @Override cleanup for Eclipse
> -----------------------------
>
>                 Key: HADOOP-5624
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5624
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Carlos Valiente
>            Priority: Trivial
>         Attachments: HADOOP-5624.patch
>
>
> Eclipse complains about several methods which are marked as {{@Override}}, but which are not defined in any superclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16206-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 16:01:15 2011
Return-Path: <common-issues-return-16206-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B9F90425A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 16:01:15 +0000 (UTC)
Received: (qmail 23298 invoked by uid 500); 12 Jun 2011 16:01:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23080 invoked by uid 500); 12 Jun 2011 16:01:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23072 invoked by uid 99); 12 Jun 2011 16:01:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 16:01:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 16:01:13 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BDB8D4154F0
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 16:00:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 16:00:51 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1566492830.1051.1307894451773.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-2988) Trash not being deleted on time
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048363#comment-13048363 ] 

Harsh J commented on HADOOP-2988:
---------------------------------

Koji,

Given that the TestTrash class has an Emptier thread test that passes, is this still a problem?

> Trash not being deleted on time
> -------------------------------
>
>                 Key: HADOOP-2988
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2988
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.14.3
>            Reporter: Koji Noguchi
>            Priority: Trivial
>
> On one of our cluster, we set the Trash interval to 6 hrs. 
> However, sometimes the namenode doesn't delete the Trash dir on time.
> -bash-3.00$ hadoop dfs -ls /Trash
> Found 3 items
> /Trash/0711201201       <dir>           2007-11-20 06:00
> /Trash/0711201800       <dir>           2007-11-20 12:15
> /Trash/Current  <dir>           2007-11-20 18:01
> In our current setting, we're supposed to have only one 'current' and one previous snapshot.  
> Grepping shows that /Trash/0711201201 was not even touched.
> My guess is that 1800 - 1201 = 5 hrs 59min  < 6hrs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16207-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 18:00:13 2011
Return-Path: <common-issues-return-16207-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C9D074DC0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 18:00:13 +0000 (UTC)
Received: (qmail 30841 invoked by uid 500); 12 Jun 2011 18:00:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30695 invoked by uid 500); 12 Jun 2011 18:00:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30687 invoked by uid 99); 12 Jun 2011 18:00:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 18:00:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 18:00:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 92BF1415FDB
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 17:59:52 +0000 (UTC)
Date: Sun, 12 Jun 2011 17:59:52 +0000 (UTC)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <27096257.1208.1307901592597.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7106) Re-organize hadoop subversion
 layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048394#comment-13048394 ] 

Nigel Daley commented on HADOOP-7106:
-------------------------------------

Awesome!  Thanks Todd. 

> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, gitk-example.png
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16208-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 18:53:15 2011
Return-Path: <common-issues-return-16208-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DE7A547D4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 18:53:14 +0000 (UTC)
Received: (qmail 73300 invoked by uid 500); 12 Jun 2011 18:53:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73270 invoked by uid 500); 12 Jun 2011 18:53:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73262 invoked by uid 99); 12 Jun 2011 18:53:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 18:53:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 18:53:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9956B415F34
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 18:52:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 18:52:51 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1244652372.1326.1307904771624.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-6936) broken links in
 http://wiki.apache.org/hadoop/FAQ#A12//s
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6936?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J resolved HADOOP-6936.
-----------------------------

    Resolution: Fixed

Just for a history lesson ref. here: There used to be hadoop-*.xml files on=
ce upon a time. Its now split over to core-*, hdfs-*, mapred-* files (* -> =
{site, default}).

Closing as the HowToConfigure link has also been updated by me. Although it=
 needs more love in general (We should switch to confluence=E2=80=A6 its mo=
re encouraging).

> broken links in http://wiki.apache.org/hadoop/FAQ#A12//s
> --------------------------------------------------------
>
>                 Key: HADOOP-6936
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6936
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>            Reporter: Eugene Koontz
>            Priority: Trivial
>
> http://wiki.apache.org/hadoop/FAQ#A12//s=20
> has links to :
> http://hadoop.apache.org/core/docs/current/hadoop-default.html#dfs.replic=
ation.min
> http://hadoop.apache.org/common/docs/current/hadoop-default.html#dfs.safe=
mode.threshold.pct
> both of which are 404 as of time of filing this issue.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

From common-issues-return-16209-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 18:57:14 2011
Return-Path: <common-issues-return-16209-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B87CF488F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 18:57:14 +0000 (UTC)
Received: (qmail 75575 invoked by uid 500); 12 Jun 2011 18:57:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75547 invoked by uid 500); 12 Jun 2011 18:57:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75539 invoked by uid 99); 12 Jun 2011 18:57:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 18:57:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 18:57:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 875A4415060
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 18:56:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 18:56:51 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <589524318.1329.1307905011551.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6936) broken links in
 http://wiki.apache.org/hadoop/FAQ#A12//s
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048413#comment-13048413 ] 

Harsh J commented on HADOOP-6936:
---------------------------------

Failed to add, link that has proper sublinks is: http://hadoop.apache.org/common/docs/current/cluster_setup.html#Configuration+Files

> broken links in http://wiki.apache.org/hadoop/FAQ#A12//s
> --------------------------------------------------------
>
>                 Key: HADOOP-6936
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6936
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>            Reporter: Eugene Koontz
>            Priority: Trivial
>              Labels: wiki
>
> http://wiki.apache.org/hadoop/FAQ#A12//s 
> has links to :
> http://hadoop.apache.org/core/docs/current/hadoop-default.html#dfs.replication.min
> http://hadoop.apache.org/common/docs/current/hadoop-default.html#dfs.safemode.threshold.pct
> both of which are 404 as of time of filing this issue.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16210-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 19:21:12 2011
Return-Path: <common-issues-return-16210-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D9D3148A2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 19:21:12 +0000 (UTC)
Received: (qmail 94491 invoked by uid 500); 12 Jun 2011 19:21:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94451 invoked by uid 500); 12 Jun 2011 19:21:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94443 invoked by uid 99); 12 Jun 2011 19:21:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 19:21:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 19:21:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AD0AF41565B
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 19:20:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 19:20:51 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <717022041.1367.1307906451705.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7361) Provide overwrite option
 (-overwrite/-f) in put and copyFromLocal command line options
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048418#comment-13048418 ] 

Harsh J commented on HADOOP-7361:
---------------------------------

Seems to be in parallel contention with HADOOP-7176 efforts by Daryn Sharp.

> Provide overwrite option (-overwrite/-f) in put and copyFromLocal command line options
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7361
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7361
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Uma Maheswara Rao G
>            Priority: Minor
>         Attachments: HDFS-1608-src.patch, HDFS-1608-test.patch
>
>
> FileSystem has the API 
> *public void copyFromLocalFile(boolean delSrc, boolean overwrite, Path[] srcs, Path dst)*
>      
>                          
> This API provides overwrite option. But the mapping command line doesn't have this option. To maintain the consistency and better usage  the command line option also can support the overwrite option like to put the files forcefully. ( put [-f] <srcpath> <dstPath>) and also for copyFromLocal command line option.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16211-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 22:34:14 2011
Return-Path: <common-issues-return-16211-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D055B4371
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 22:34:14 +0000 (UTC)
Received: (qmail 46702 invoked by uid 500); 12 Jun 2011 22:34:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46670 invoked by uid 500); 12 Jun 2011 22:34:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46662 invoked by uid 99); 12 Jun 2011 22:34:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 22:34:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 22:34:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 902AC4164D0
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 22:33:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 22:33:51 +0000 (UTC)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1650637010.1532.1307918031587.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6344) rm and rmr fail to correctly move
 the user's files to the trash prior to deleting when they are over quota.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nigel Daley updated HADOOP-6344:
--------------------------------

    Fix Version/s: 0.21.0
                   0.22.0

> rm and rmr fail to correctly move the user's files to the trash prior to deleting when they are over quota.  
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6344
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6344
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.0, 0.20.1, 0.21.0, 0.22.0
>            Reporter: gary murry
>            Assignee: Jakob Homan
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HDFS-740-for-Y20.patch, HDFS-740.patch
>
>
> With trash turned on, if a user is over his quota and does a rm (or rmr), the file is deleted without a copy being placed in the trash.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16212-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 22:38:14 2011
Return-Path: <common-issues-return-16212-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D525C441B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 22:38:14 +0000 (UTC)
Received: (qmail 47444 invoked by uid 500); 12 Jun 2011 22:38:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47410 invoked by uid 500); 12 Jun 2011 22:38:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47402 invoked by uid 99); 12 Jun 2011 22:38:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 22:38:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 22:38:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B6B80416574
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 22:37:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 22:37:51 +0000 (UTC)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <101269183.1542.1307918271745.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6599) Split RPC metrics into summary and
 detailed metrics
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nigel Daley updated HADOOP-6599:
--------------------------------

    Fix Version/s: 0.22.0

> Split RPC metrics into summary and detailed metrics
> ---------------------------------------------------
>
>                 Key: HADOOP-6599
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6599
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6599.1.patch, hadoop-6599.2.patch, hadoop-6599.2.patch, hadoop-6599.patch, hadoop-6599.patch, hadoop-6599.rel20.patch
>
>
> Currently RPC metrics tracks items that provides summary of RPC usage along with more detailed per method statistics that tracks number of method calls made, time spent on that call. Combining both summary and detailed together results in large metrics report which metrics collection systems may not handle. Proposal is to split the metrics into summary and detailed metrics.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16213-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 23:28:13 2011
Return-Path: <common-issues-return-16213-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E1F9B41E1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 23:28:12 +0000 (UTC)
Received: (qmail 75091 invoked by uid 500); 12 Jun 2011 23:28:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75064 invoked by uid 500); 12 Jun 2011 23:28:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75056 invoked by uid 99); 12 Jun 2011 23:28:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 23:28:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 23:28:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AC32A416BFC
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 23:27:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 23:27:51 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2086985344.1547.1307921271702.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7106) Re-organize hadoop subversion
 layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048439#comment-13048439 ] 

Hudson commented on HADOOP-7106:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #651 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/651/])
    Add HADOOP-7106 to CHANGES.txt for trunk.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1135001
Files : 
* /hadoop/common/trunk/hdfs/CHANGES.txt
* /hadoop/common/trunk/mapreduce/CHANGES.txt
* /hadoop/common/trunk/common/CHANGES.txt


> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, gitk-example.png
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16214-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 12 23:42:13 2011
Return-Path: <common-issues-return-16214-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 305D64AEF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 12 Jun 2011 23:42:13 +0000 (UTC)
Received: (qmail 83326 invoked by uid 500); 12 Jun 2011 23:42:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83289 invoked by uid 500); 12 Jun 2011 23:42:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83279 invoked by uid 99); 12 Jun 2011 23:42:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 23:42:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jun 2011 23:42:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 99633416E33
	for <common-issues@hadoop.apache.org>; Sun, 12 Jun 2011 23:41:51 +0000 (UTC)
Date: Sun, 12 Jun 2011 23:41:51 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <327361541.1573.1307922111624.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7106) Re-organize hadoop subversion
 layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048440#comment-13048440 ] 

Todd Lipcon commented on HADOOP-7106:
-------------------------------------

I committed this this afternoon. I posted some instructions about how to migrate here:
http://markmail.org/message/cmyx2ojtn6dj6wlp


Next we need to remove the transitional lines from the auth template and mailer config, right? Then, we can resolve this?

> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, gitk-example.png
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16215-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 00:42:13 2011
Return-Path: <common-issues-return-16215-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4BCEE4D53
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 00:42:13 +0000 (UTC)
Received: (qmail 14147 invoked by uid 500); 13 Jun 2011 00:42:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14120 invoked by uid 500); 13 Jun 2011 00:42:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14109 invoked by uid 99); 13 Jun 2011 00:42:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 00:42:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 00:42:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C1F49416779
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 00:41:51 +0000 (UTC)
Date: Mon, 13 Jun 2011 00:41:51 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1507121853.1613.1307925711791.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7384) Allow test-patch to be more flexible
 about patch format
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Allow test-patch to be more flexible about patch format
-------------------------------------------------------

                 Key: HADOOP-7384
                 URL: https://issues.apache.org/jira/browse/HADOOP-7384
             Project: Hadoop Common
          Issue Type: Improvement
    Affects Versions: 0.23.0
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon


Right now the test-patch process only accepts patches that are generated as "-p0" relative to common/, hdfs/, or mapreduce/. This has always been annoying for git users where the default patch format is -p1. It's also now annoying for SVN users who may generate a patch relative to trunk/ instead of the subproject subdirectory. We should auto-detect the correct patch level.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16216-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 00:52:13 2011
Return-Path: <common-issues-return-16216-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A166D43A3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 00:52:13 +0000 (UTC)
Received: (qmail 17163 invoked by uid 500); 13 Jun 2011 00:52:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17139 invoked by uid 500); 13 Jun 2011 00:52:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17131 invoked by uid 99); 13 Jun 2011 00:52:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 00:52:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 00:52:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 04D3E416948
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 00:51:52 +0000 (UTC)
Date: Mon, 13 Jun 2011 00:51:52 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <578994662.1619.1307926312016.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1507121853.1613.1307925711791.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7384) Allow test-patch to be more flexible
 about patch format
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7384:
--------------------------------

    Status: Patch Available  (was: Open)

> Allow test-patch to be more flexible about patch format
> -------------------------------------------------------
>
>                 Key: HADOOP-7384
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7384
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-7384.txt
>
>
> Right now the test-patch process only accepts patches that are generated as "-p0" relative to common/, hdfs/, or mapreduce/. This has always been annoying for git users where the default patch format is -p1. It's also now annoying for SVN users who may generate a patch relative to trunk/ instead of the subproject subdirectory. We should auto-detect the correct patch level.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16217-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 00:52:15 2011
Return-Path: <common-issues-return-16217-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5610F43AD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 00:52:15 +0000 (UTC)
Received: (qmail 17427 invoked by uid 500); 13 Jun 2011 00:52:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17396 invoked by uid 500); 13 Jun 2011 00:52:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17388 invoked by uid 99); 13 Jun 2011 00:52:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 00:52:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 00:52:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id ECAD9416946
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 00:51:51 +0000 (UTC)
Date: Mon, 13 Jun 2011 00:51:51 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1879813115.1617.1307926311966.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1507121853.1613.1307925711791.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7384) Allow test-patch to be more flexible
 about patch format
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7384:
--------------------------------

    Attachment: hadoop-7384.txt

Here's a transcript of my manual testing:

todd@todd-w510:~/git/hadoop-common/common$ git show | PATCH=echo src/test/bin/smart-apply-patch.sh -
Looks like this is a git patch. Stripping a/ and b/ prefixes
and incrementing PLEVEL
Looks like this is relative to project root. Increasing PLEVEL
Going to apply patch with: echo -p2
-p2 -E
todd@todd-w510:~/git/hadoop-common/common$ git show --no-prefix | PATCH=echo src/test/bin/smart-apply-patch.sh -
Looks like this is relative to project root. Increasing PLEVEL
Going to apply patch with: echo -p1
-p1 -E
todd@todd-w510:~/git/hadoop-common/common$ git show --relative | PATCH=echo src/test/bin/smart-apply-patch.sh -
Looks like this is a git patch. Stripping a/ and b/ prefixes
and incrementing PLEVEL
Going to apply patch with: echo -p1
-p1 -E
todd@todd-w510:~/git/hadoop-common/common$ git show --relative --no-prefix | PATCH=echo src/test/bin/smart-apply-patch.sh -
Going to apply patch with: echo -p0
-p0 -E

todd@todd-w510:~/git/hadoop-common/common$ echo > CHANGES.txt ; echo > ../mapreduce/CHANGES.txt 

todd@todd-w510:~/git/hadoop-common/common$ git diff  | PATCH=echo src/test/bin/smart-apply-patch.sh -
Looks like this is a git patch. Stripping a/ and b/ prefixes
and incrementing PLEVEL
Looks like this is a cross-subproject patch. Not supported!
todd@todd-w510:~/git/hadoop-common/common$ echo $?
1


> Allow test-patch to be more flexible about patch format
> -------------------------------------------------------
>
>                 Key: HADOOP-7384
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7384
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-7384.txt
>
>
> Right now the test-patch process only accepts patches that are generated as "-p0" relative to common/, hdfs/, or mapreduce/. This has always been annoying for git users where the default patch format is -p1. It's also now annoying for SVN users who may generate a patch relative to trunk/ instead of the subproject subdirectory. We should auto-detect the correct patch level.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16218-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 03:30:20 2011
Return-Path: <common-issues-return-16218-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A09BB60E5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 03:30:20 +0000 (UTC)
Received: (qmail 95772 invoked by uid 500); 13 Jun 2011 03:30:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95737 invoked by uid 500); 13 Jun 2011 03:30:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95722 invoked by uid 99); 13 Jun 2011 03:30:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 03:30:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 03:30:14 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6F80A4163D0
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 03:29:53 +0000 (UTC)
Date: Mon, 13 Jun 2011 03:29:53 +0000 (UTC)
From: "Vinod Kumar Vavilapalli (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1359391911.1771.1307935793453.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7106) Re-organize hadoop subversion
 layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048450#comment-13048450 ] 

Vinod Kumar Vavilapalli commented on HADOOP-7106:
-------------------------------------------------

Todd, the history for MR-279 branch (http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279) is completely lost. OTOH, I just verified that mapreduce and hdfs history is (correctly) retained in both the trunk and branch-0.21 branches.

Was this a mistake? I am hoping we can get back the lost history. Right?

> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, gitk-example.png
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16219-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 03:32:15 2011
Return-Path: <common-issues-return-16219-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 07BA46D95
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 03:32:15 +0000 (UTC)
Received: (qmail 96348 invoked by uid 500); 13 Jun 2011 03:32:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96280 invoked by uid 500); 13 Jun 2011 03:32:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96269 invoked by uid 99); 13 Jun 2011 03:32:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 03:32:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 03:32:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7FCDF41644B
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 03:31:51 +0000 (UTC)
Date: Mon, 13 Jun 2011 03:31:51 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <742343018.1774.1307935911520.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <55510245.12688.1307747698852.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7379) Add ability to include Protobufs
 in ObjectWritable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048451#comment-13048451 ] 

Todd Lipcon commented on HADOOP-7379:
-------------------------------------

bq. What will it take to get ipc.Server to understand PB rather than Writable?

You mean using PBs for the actual invocations? Or the "header" type info at the beginning? I think most of the value from PB comes from the evolvable parameter types, but moving to PB for the headers would also allow us to do things like add extra metadata to requests without breaking previous versions (eg dapper-style tracing info?)

> Add ability to include Protobufs in ObjectWritable
> --------------------------------------------------
>
>                 Key: HADOOP-7379
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7379
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, ipc
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7379.txt, hadoop-7379.txt
>
>
> Per HDFS-2060, it would make it easier to piecemeal switch to protocol buffer based data structures in the wire protocol if we could intermix the two. The IPC framework currently provides the concept of "engines" for RPC, but that doesn't easily allow mixed types within the same framework for ease of transition.
> I'd like to add the cases to ObjectWritable to be handle subclasses of {{Message}}, the superclass of codegenned protobufs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16220-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 03:34:16 2011
Return-Path: <common-issues-return-16220-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 22A3363BF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 03:34:16 +0000 (UTC)
Received: (qmail 96954 invoked by uid 500); 13 Jun 2011 03:34:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96836 invoked by uid 500); 13 Jun 2011 03:34:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96820 invoked by uid 99); 13 Jun 2011 03:34:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 03:34:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 03:34:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 03665416538
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 03:33:52 +0000 (UTC)
Date: Mon, 13 Jun 2011 03:33:52 +0000 (UTC)
From: "Vinod Kumar Vavilapalli (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <479678623.1785.1307936032010.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7106) Re-organize hadoop subversion
 layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048453#comment-13048453 ] 

Vinod Kumar Vavilapalli commented on HADOOP-7106:
-------------------------------------------------

Sorry, I take my comment back. The history is also split between the projects under MR-279, I see it under mapreduce. I was confused as MR-279 is supposed to be a branch of only mapreduce pre-project split :)

False alarm!

Thanks for the good work!

> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, gitk-example.png
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16221-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 03:38:17 2011
Return-Path: <common-issues-return-16221-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2D671649D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 03:38:17 +0000 (UTC)
Received: (qmail 99895 invoked by uid 500); 13 Jun 2011 03:38:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99855 invoked by uid 500); 13 Jun 2011 03:38:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99832 invoked by uid 99); 13 Jun 2011 03:38:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 03:38:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 03:38:13 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 58920416710
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 03:37:52 +0000 (UTC)
Date: Mon, 13 Jun 2011 03:37:52 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <812082732.1835.1307936272359.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7106) Re-organize hadoop subversion
 layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048455#comment-13048455 ] 

Todd Lipcon commented on HADOOP-7106:
-------------------------------------

Hey Vinod -- yep, MR-279 is now a "cross-project branch" that includes the yahoo-merge versions of common and hdfs. I figured that will make it easier for you to develop since many of the yahoo-merge changes are specifically for MR-279. Glad to hear you found the history, too :)

> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, gitk-example.png
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16222-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 05:00:15 2011
Return-Path: <common-issues-return-16222-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5BD06645D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 05:00:15 +0000 (UTC)
Received: (qmail 33061 invoked by uid 500); 13 Jun 2011 05:00:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33015 invoked by uid 500); 13 Jun 2011 05:00:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32974 invoked by uid 99); 13 Jun 2011 05:00:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 05:00:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 05:00:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C4D71416AB0
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 04:59:51 +0000 (UTC)
Date: Mon, 13 Jun 2011 04:59:51 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1038446292.1991.1307941191802.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048467#comment-13048467 ] 

Tom White commented on HADOOP-6671:
-----------------------------------

Eric, 

I noticed the same test failures when running on Jenkins (https://builds.apache.org/job/Hadoop-Common-trunk-maven/8/), but I can't reproduce locally (on Mac), or on Jenkins with Ant (see https://builds.apache.org/job/Hadoop-Common-trunk/lastCompletedBuild/testReport/org.apache.hadoop.security/TestUserGroupInformation/testGetServerSideGroups/). Seems like some kind of environment issue.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-6671-cross-project-HDFS.patch, HADOOP-6671-e.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16223-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 05:16:17 2011
Return-Path: <common-issues-return-16223-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 24E556D72
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 05:16:17 +0000 (UTC)
Received: (qmail 38186 invoked by uid 500); 13 Jun 2011 05:16:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37968 invoked by uid 500); 13 Jun 2011 05:16:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37959 invoked by uid 99); 13 Jun 2011 05:16:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 05:16:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 05:16:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 824C3416F35
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 05:15:51 +0000 (UTC)
Date: Mon, 13 Jun 2011 05:15:51 +0000 (UTC)
From: "XieXianshan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <623913763.2022.1307942151529.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1142137633.58455.1306902167529.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7348) Modify the option of FsShell
 getmerge from [addnl] to [-nl] for more comprehensive
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048469#comment-13048469 ] 

XieXianshan commented on HADOOP-7348:
-------------------------------------

It`s my pleasure. :D


> Modify the option of FsShell getmerge from [addnl] to [-nl] for more comprehensive
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7348
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7348
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Assignee: XieXianshan
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7348.patch, HADOOP-7348.patch, HADOOP-7348.patch_2
>
>
> The [addnl] option of FsShell getmerge should be either "true" or "false",but it is very hard to understand by users, especially  who`s never used this option before. 
> So,the [addnl] option should be changed to [-nl] for more comprehensive.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16224-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 07:18:15 2011
Return-Path: <common-issues-return-16224-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C65B76144
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 07:18:15 +0000 (UTC)
Received: (qmail 16285 invoked by uid 500); 13 Jun 2011 07:18:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15948 invoked by uid 500); 13 Jun 2011 07:18:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15863 invoked by uid 99); 13 Jun 2011 07:18:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 07:18:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 07:18:13 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 42552416593
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 07:17:53 +0000 (UTC)
Date: Mon, 13 Jun 2011 07:17:53 +0000 (UTC)
From: "Eric Charles (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1809483786.2193.1307949473268.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048496#comment-13048496 ] 

Eric Charles commented on HADOOP-6671:
--------------------------------------

Hi Tom,
Yes, sounds like environment, not code, issue.
I'm used to maven, but completely new to hadoop, especially regarding the way hadoop relies on Operating Systems functions (I see many Runtime calls in the code).

First of all, I will try to let the common project tests work in eclipse. If I run "ant test", all tests are successfull. But if I run them eclipse, about half are failing. I will post on mailing list for this. I think getting the solution for this will help to fix the TestUserGroupInformation test (working in eclipse, but not in command line).

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-6671-cross-project-HDFS.patch, HADOOP-6671-e.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16225-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 11:26:13 2011
Return-Path: <common-issues-return-16225-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1C8BD620C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 11:26:13 +0000 (UTC)
Received: (qmail 42090 invoked by uid 500); 13 Jun 2011 11:26:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42031 invoked by uid 500); 13 Jun 2011 11:26:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42017 invoked by uid 99); 13 Jun 2011 11:26:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 11:26:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 11:26:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B1C0041791F
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 11:25:51 +0000 (UTC)
Date: Mon, 13 Jun 2011 11:25:51 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2090749820.2326.1307964351724.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7106) Re-organize hadoop subversion
 layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048520#comment-13048520 ] 

Hudson commented on HADOOP-7106:
--------------------------------

Integrated in Hadoop-Common-trunk #719 (See [https://builds.apache.org/job/Hadoop-Common-trunk/719/])
    Add HADOOP-7106 to CHANGES.txt for trunk.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1135001
Files : 
* /hadoop/common/trunk/hdfs/CHANGES.txt
* /hadoop/common/trunk/mapreduce/CHANGES.txt
* /hadoop/common/trunk/common/CHANGES.txt


> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, gitk-example.png
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16226-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 12:31:13 2011
Return-Path: <common-issues-return-16226-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9236468D1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 12:31:13 +0000 (UTC)
Received: (qmail 7778 invoked by uid 500); 13 Jun 2011 12:31:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7733 invoked by uid 500); 13 Jun 2011 12:31:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7725 invoked by uid 99); 13 Jun 2011 12:31:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 12:31:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 12:31:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4F618417BB8
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 12:30:52 +0000 (UTC)
Date: Mon, 13 Jun 2011 12:30:52 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2094014708.2451.1307968252321.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32237646.17061288907081267.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7020) establish a "Powered by Hadoop"
 logo
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048535#comment-13048535 ] 

Steve Loughran commented on HADOOP-7020:
----------------------------------------

I prefer the hadoop-elephant-pb image, though of course you need a tm marker on apache and hadoop. The alternative I think might be good is powered-by-hadoop.png, because it clearly differentiates Hadoop itself from things powered by apache hadoop. 

> establish a "Powered by Hadoop" logo
> ------------------------------------
>
>                 Key: HADOOP-7020
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7020
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: site
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: site
>
>         Attachments: PoweredByApacheHadoop.png, PoweredByHadoop_Small.jpg, hadoop-elephant-pb.jpeg, powered-by-hadoop-small.png, powered-by-hadoop.png, powered-by-proposal.jpg, powered_by_hadoop.jpg
>
>
> We should agree on a Powered By Hadoop logo, as suggested in:
> http://www.apache.org/foundation/marks/pmcs#poweredby

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16227-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 14:04:14 2011
Return-Path: <common-issues-return-16227-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C56FB67A9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 14:04:14 +0000 (UTC)
Received: (qmail 8627 invoked by uid 500); 13 Jun 2011 14:04:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8486 invoked by uid 500); 13 Jun 2011 14:04:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8478 invoked by uid 99); 13 Jun 2011 14:04:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 14:04:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 14:04:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8E927417E0D
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 14:03:51 +0000 (UTC)
Date: Mon, 13 Jun 2011 14:03:51 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1711135966.2567.1307973831580.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048566#comment-13048566 ] 

Daryn Sharp commented on HADOOP-6605:
-------------------------------------

+1 Beautiful!  My only suggestion would be to consider use double brackets (ie. [[ ... ]]) so you don't have to add quotes around the variables.  Double brackets do 1-pass expansion, as opposed to single brackets that expanded and re-parse which leads to problem with variables contains space or other metachars unless quotes are added.

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch, hadoop-6605-2.patch, hadoop-6605-3.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16228-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 14:08:16 2011
Return-Path: <common-issues-return-16228-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 02A126AE8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 14:08:16 +0000 (UTC)
Received: (qmail 18091 invoked by uid 500); 13 Jun 2011 14:08:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18051 invoked by uid 500); 13 Jun 2011 14:08:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18035 invoked by uid 99); 13 Jun 2011 14:08:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 14:08:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 14:08:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DEA91417034
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 14:07:51 +0000 (UTC)
Date: Mon, 13 Jun 2011 14:07:51 +0000 (UTC)
From: "T Jake Luciani (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <380982275.2581.1307974071908.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048568#comment-13048568 ] 

T Jake Luciani commented on HADOOP-7206:
----------------------------------------

We have a Snappy codec in brisk based on the java-snappy project: http://code.google.com/p/snappy-java/

We did this because snappy-java is ASL, in maven, and comes with pre-built with shared libs for win, linux, and osx. For end users this is a much lower bar to getting into their system.

https://github.com/riptano/brisk/tree/master/src/java/src/com/hadoop/compression/snappy

I'm confused on how to contribue this back based on the above comments :) should we submit a patch or release this as a standalone github project?



> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>         Attachments: HADOOP-7206.patch
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16229-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 15:56:13 2011
Return-Path: <common-issues-return-16229-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 114146485
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 15:56:13 +0000 (UTC)
Received: (qmail 18874 invoked by uid 500); 13 Jun 2011 15:56:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18846 invoked by uid 500); 13 Jun 2011 15:56:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18805 invoked by uid 99); 13 Jun 2011 15:56:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 15:56:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 15:56:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CA60441715B
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 15:55:51 +0000 (UTC)
Date: Mon, 13 Jun 2011 15:55:51 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <671034748.2828.1307980551825.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048612#comment-13048612 ] 

Allen Wittenauer commented on HADOOP-6605:
------------------------------------------

I still think this is a bad idea, even if we are limiting to OS X.  

BTW, the patch should use -x, not -e.

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch, hadoop-6605-2.patch, hadoop-6605-3.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16230-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 16:38:15 2011
Return-Path: <common-issues-return-16230-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 742F066A0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 16:38:15 +0000 (UTC)
Received: (qmail 24802 invoked by uid 500); 13 Jun 2011 16:38:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24743 invoked by uid 500); 13 Jun 2011 16:38:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24726 invoked by uid 99); 13 Jun 2011 16:38:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 16:38:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 16:38:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C1309417895
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 16:37:51 +0000 (UTC)
Date: Mon, 13 Jun 2011 16:37:51 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1049301347.2996.1307983071788.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6605:
--------------------------------

    Attachment: hadoop-6605-4.patch

Thanks for the feedback guys. Patch attached uses double brackets and -x.

Allen, I think this patch addresses the specific concern you raised (searching too many directories, the Linux stuff). This patch does pretty minimal work (just calls out to java_home). It does still execute on each command, that's intentional and IMO OK because per above it's minimal work and otherwise Hadoop would not notice when the user changed their JAVA_HOME preference. Lemme know if you're -1 or -0.

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch, hadoop-6605-2.patch, hadoop-6605-3.patch, hadoop-6605-4.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16231-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 16:58:14 2011
Return-Path: <common-issues-return-16231-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D866E6C30
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 16:58:14 +0000 (UTC)
Received: (qmail 74423 invoked by uid 500); 13 Jun 2011 16:58:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74397 invoked by uid 500); 13 Jun 2011 16:58:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74389 invoked by uid 99); 13 Jun 2011 16:58:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 16:58:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 16:58:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 922E24174AF
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 16:57:51 +0000 (UTC)
Date: Mon, 13 Jun 2011 16:57:51 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1680665884.3082.1307984271595.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048651#comment-13048651 ] 

Daryn Sharp commented on HADOOP-6605:
-------------------------------------

I'd like to succinctly reassert that it's unexpected behavior, on OS X, for a java app to not honor the user's preference.  FWIW, I've never had to explicitly define JAVA_HOME for an app.  For instance, Tomcat doesn't require the user to explicitly define JAVA_HOME to start the server.

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch, hadoop-6605-2.patch, hadoop-6605-3.patch, hadoop-6605-4.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16232-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 17:15:13 2011
Return-Path: <common-issues-return-16232-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7B0FC6B48
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 17:15:13 +0000 (UTC)
Received: (qmail 19022 invoked by uid 500); 13 Jun 2011 17:15:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18991 invoked by uid 500); 13 Jun 2011 17:15:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18983 invoked by uid 99); 13 Jun 2011 17:15:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 17:15:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 17:15:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 35EC6417D68
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 17:14:52 +0000 (UTC)
Date: Mon, 13 Jun 2011 17:14:52 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <507887964.3143.1307985292217.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32237646.17061288907081267.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7020) establish a "Powered by Hadoop"
 logo
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048661#comment-13048661 ] 

Owen O'Malley commented on HADOOP-7020:
---------------------------------------

I still prefer the silver circle variant [^PoweredByApacheHadoop.png].

Alison's one is very artistic, but looks like a logo for a different project [^powered_by_hadoop.jpg].

It looks like we are developing quite the range of options. I propose that we vote with single transferable vote. If you have any more candidates, upload them here by tomorrow 14 June at 5pm PST. I'll start a thread on general then calling for votes between the options.

> establish a "Powered by Hadoop" logo
> ------------------------------------
>
>                 Key: HADOOP-7020
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7020
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: site
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: site
>
>         Attachments: PoweredByApacheHadoop.png, PoweredByHadoop_Small.jpg, hadoop-elephant-pb.jpeg, powered-by-hadoop-small.png, powered-by-hadoop.png, powered-by-proposal.jpg, powered_by_hadoop.jpg
>
>
> We should agree on a Powered By Hadoop logo, as suggested in:
> http://www.apache.org/foundation/marks/pmcs#poweredby

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16233-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 17:25:15 2011
Return-Path: <common-issues-return-16233-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1BA86644B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 17:25:15 +0000 (UTC)
Received: (qmail 41812 invoked by uid 500); 13 Jun 2011 17:25:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41767 invoked by uid 500); 13 Jun 2011 17:25:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41730 invoked by uid 99); 13 Jun 2011 17:25:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 17:25:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 17:25:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8F4BE418240
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 17:24:51 +0000 (UTC)
Date: Mon, 13 Jun 2011 17:24:51 +0000 (UTC)
From: "Bharath Mundlapudi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <113408089.3186.1307985891583.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7385) Remove
 StringUtils.stringifyException(ie) in logger functions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Remove StringUtils.stringifyException(ie) in logger functions
-------------------------------------------------------------

                 Key: HADOOP-7385
                 URL: https://issues.apache.org/jira/browse/HADOOP-7385
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.23.0
            Reporter: Bharath Mundlapudi
            Assignee: Bharath Mundlapudi
            Priority: Minor
             Fix For: 0.23.0


Apache logger api has an overloaded function which can take the message and exception. I am proposing to clean the logging code with this api.
ie.:
Change the code from LOG.warn(msg, StringUtils.stringifyException(exception)); to LOG.warn(msg, exception);


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16234-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 17:57:15 2011
Return-Path: <common-issues-return-16234-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A99D46E23
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 17:57:15 +0000 (UTC)
Received: (qmail 83532 invoked by uid 500); 13 Jun 2011 17:57:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83468 invoked by uid 500); 13 Jun 2011 17:57:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83460 invoked by uid 99); 13 Jun 2011 17:57:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 17:57:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 17:57:13 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4C725418D9E
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 17:56:52 +0000 (UTC)
Date: Mon, 13 Jun 2011 17:56:52 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1635251930.3251.1307987812309.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048672#comment-13048672 ] 

Allen Wittenauer commented on HADOOP-6605:
------------------------------------------

I'll change my vote to 0 for this particular change to add support for OS X.  (I keep meaning to check Lion but keep forgetting.  Too busy working on other Lion bugs...).  This means we should change the case statement to a single if test.

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch, hadoop-6605-2.patch, hadoop-6605-3.patch, hadoop-6605-4.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16235-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 18:25:14 2011
Return-Path: <common-issues-return-16235-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A4F8C612B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 18:25:14 +0000 (UTC)
Received: (qmail 30854 invoked by uid 500); 13 Jun 2011 18:25:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30815 invoked by uid 500); 13 Jun 2011 18:25:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30807 invoked by uid 99); 13 Jun 2011 18:25:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 18:25:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 18:25:13 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1AD23418B50
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 18:24:53 +0000 (UTC)
Date: Mon, 13 Jun 2011 18:24:53 +0000 (UTC)
From: "jiraposter@reviews.apache.org (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1896926805.3380.1307989493106.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048694#comment-13048694 ] 

jiraposter@reviews.apache.org commented on HADOOP-7328:
-------------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/884/#review817
-----------------------------------------------------------


Sorry if this is out of context, but is it really best to also return a null here?  Shouldn't it check for null result from getSerialization(), then throw a (non-NPE) exception?  Or do you prefer to do that check and throw at a higher level of the code?

- Matt


On 2011-06-11 22:10:17, Harsh J wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/884/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2011-06-11 22:10:17)
bq.  
bq.  
bq.  Review request for hadoop-common and Todd Lipcon.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  Since getSerialization() can possibly return a null, it is only right that getSerializer() and getDeserializer() usage functions do the same, instead of throwing up NPEs.
bq.  
bq.  Related issue to which this improvement is required: https://issues.apache.org/jira/browse/MAPREDUCE-2584
bq.  
bq.  
bq.  This addresses bug HADOOP-7328.
bq.      http://issues.apache.org/jira/browse/HADOOP-7328
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    src/java/org/apache/hadoop/io/serializer/SerializationFactory.java dee314a 
bq.  
bq.  Diff: https://reviews.apache.org/r/884/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  Existing SequenceFile serialization factory tests pass. The change is merely to make the functions return null instead of throwing an NPE within.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Harsh
bq.  
bq.



> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff, HADOOP-7328.r2.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16236-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 18:27:13 2011
Return-Path: <common-issues-return-16236-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 25C0B6291
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 18:27:13 +0000 (UTC)
Received: (qmail 34665 invoked by uid 500); 13 Jun 2011 18:27:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34625 invoked by uid 500); 13 Jun 2011 18:27:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34617 invoked by uid 99); 13 Jun 2011 18:27:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 18:27:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 18:27:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D9364418BF8
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 18:26:51 +0000 (UTC)
Date: Mon, 13 Jun 2011 18:26:51 +0000 (UTC)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1451554905.3385.1307989611886.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048695#comment-13048695 ] 

Chris Douglas commented on HADOOP-6605:
---------------------------------------

+1 on the OS X detection

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch, hadoop-6605-2.patch, hadoop-6605-3.patch, hadoop-6605-4.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16237-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 18:33:16 2011
Return-Path: <common-issues-return-16237-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E24D265E0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 18:33:16 +0000 (UTC)
Received: (qmail 48410 invoked by uid 500); 13 Jun 2011 18:33:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48366 invoked by uid 500); 13 Jun 2011 18:33:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48358 invoked by uid 99); 13 Jun 2011 18:33:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 18:33:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 18:33:14 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 06301418E40
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 18:32:53 +0000 (UTC)
Date: Mon, 13 Jun 2011 18:32:53 +0000 (UTC)
From: "jiraposter@reviews.apache.org (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2089129157.3399.1307989973022.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048698#comment-13048698 ] 

jiraposter@reviews.apache.org commented on HADOOP-7328:
-------------------------------------------------------



bq.  On 2011-06-13 18:23:35, Matt Foley wrote:
bq.  > Sorry if this is out of context, but is it really best to also return a null here?  Shouldn't it check for null result from getSerialization(), then throw a (non-NPE) exception?  Or do you prefer to do that check and throw at a higher level of the code?

I thought about this while doing the review... my thinking was that our other similar APIs do return null -eg CompressionCodecFactory.getCodecByName() returns null if the specified codec isn't found. This API is also marked as LimitedPrivate Evolving so it shouldn't be a problematic breaking change.

Maybe we should add a bit of javadoc saying "@returns null if no serializer is known for the given class." ?


- Todd


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/884/#review817
-----------------------------------------------------------


On 2011-06-11 22:10:17, Harsh J wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/884/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2011-06-11 22:10:17)
bq.  
bq.  
bq.  Review request for hadoop-common and Todd Lipcon.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  Since getSerialization() can possibly return a null, it is only right that getSerializer() and getDeserializer() usage functions do the same, instead of throwing up NPEs.
bq.  
bq.  Related issue to which this improvement is required: https://issues.apache.org/jira/browse/MAPREDUCE-2584
bq.  
bq.  
bq.  This addresses bug HADOOP-7328.
bq.      http://issues.apache.org/jira/browse/HADOOP-7328
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    src/java/org/apache/hadoop/io/serializer/SerializationFactory.java dee314a 
bq.  
bq.  Diff: https://reviews.apache.org/r/884/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  Existing SequenceFile serialization factory tests pass. The change is merely to make the functions return null instead of throwing an NPE within.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Harsh
bq.  
bq.



> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff, HADOOP-7328.r2.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16238-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 18:55:15 2011
Return-Path: <common-issues-return-16238-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 616976938
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 18:55:15 +0000 (UTC)
Received: (qmail 92882 invoked by uid 500); 13 Jun 2011 18:55:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92853 invoked by uid 500); 13 Jun 2011 18:55:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92845 invoked by uid 99); 13 Jun 2011 18:55:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 18:55:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 18:55:13 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 23CB041887B
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 18:54:52 +0000 (UTC)
Date: Mon, 13 Jun 2011 18:54:52 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <830594891.3465.1307991292143.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048707#comment-13048707 ] 

Daryn Sharp commented on HADOOP-6605:
-------------------------------------

Great, thanks Allen!  From everything I've read, Java Prefs and /usr/libexec/java_home are still present in Lion.  Apple is a major contributor to OpenJDK so I don't think Java will be fading soon.  Jobs arm twisted Oracle into releasing and supporting future Java releases.

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch, hadoop-6605-2.patch, hadoop-6605-3.patch, hadoop-6605-4.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16239-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 19:13:16 2011
Return-Path: <common-issues-return-16239-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ED25B6A69
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 19:13:16 +0000 (UTC)
Received: (qmail 39634 invoked by uid 500); 13 Jun 2011 19:13:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39593 invoked by uid 500); 13 Jun 2011 19:13:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39585 invoked by uid 99); 13 Jun 2011 19:13:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 19:13:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 19:13:14 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F0AD941822C
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 19:12:52 +0000 (UTC)
Date: Mon, 13 Jun 2011 19:12:52 +0000 (UTC)
From: "jiraposter@reviews.apache.org (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1412681699.3587.1307992372982.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048722#comment-13048722 ] 

jiraposter@reviews.apache.org commented on HADOOP-7328:
-------------------------------------------------------



bq.  On 2011-06-13 18:23:35, Matt Foley wrote:
bq.  > Sorry if this is out of context, but is it really best to also return a null here?  Shouldn't it check for null result from getSerialization(), then throw a (non-NPE) exception?  Or do you prefer to do that check and throw at a higher level of the code?
bq.  
bq.  Todd Lipcon wrote:
bq.      I thought about this while doing the review... my thinking was that our other similar APIs do return null -eg CompressionCodecFactory.getCodecByName() returns null if the specified codec isn't found. This API is also marked as LimitedPrivate Evolving so it shouldn't be a problematic breaking change.
bq.      
bq.      Maybe we should add a bit of javadoc saying "@returns null if no serializer is known for the given class." ?

The public method getSerialization() returns a null if it does not find an acceptor. I think it is fair that a get{Serializer,Deserializer}() call do the same since they are specific nature calls underneath?

Or we could revamp the entire set if Exceptions are better to have over null returns and null checks, but it ought to be consistent is what I think.


- Harsh


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/884/#review817
-----------------------------------------------------------


On 2011-06-11 22:10:17, Harsh J wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/884/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2011-06-11 22:10:17)
bq.  
bq.  
bq.  Review request for hadoop-common and Todd Lipcon.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  Since getSerialization() can possibly return a null, it is only right that getSerializer() and getDeserializer() usage functions do the same, instead of throwing up NPEs.
bq.  
bq.  Related issue to which this improvement is required: https://issues.apache.org/jira/browse/MAPREDUCE-2584
bq.  
bq.  
bq.  This addresses bug HADOOP-7328.
bq.      http://issues.apache.org/jira/browse/HADOOP-7328
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    src/java/org/apache/hadoop/io/serializer/SerializationFactory.java dee314a 
bq.  
bq.  Diff: https://reviews.apache.org/r/884/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  Existing SequenceFile serialization factory tests pass. The change is merely to make the functions return null instead of throwing an NPE within.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Harsh
bq.  
bq.



> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff, HADOOP-7328.r2.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16240-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 21:22:11 2011
Return-Path: <common-issues-return-16240-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 935406D52
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 21:22:11 +0000 (UTC)
Received: (qmail 86606 invoked by uid 500); 13 Jun 2011 21:22:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86543 invoked by uid 500); 13 Jun 2011 21:22:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86535 invoked by uid 99); 13 Jun 2011 21:22:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 21:22:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 21:22:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8692F4138C6
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 21:21:47 +0000 (UTC)
Date: Mon, 13 Jun 2011 21:21:47 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <593714438.25.1308000107548.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-6671:
---------------------------------------

    Attachment: mvn-layout-f.sh
                HADOOP-6671-f.patch

Patch rebased on top of current (unsplitted) trunk.

new stuff:

* findbugs wired
* javadocs use the Hadoop doclet

Still missing:

* DEB/RPM generation

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-6671-cross-project-HDFS.patch, HADOOP-6671-e.patch, HADOOP-6671-f.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout-f.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16241-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 21:22:11 2011
Return-Path: <common-issues-return-16241-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C21E16D5D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 21:22:11 +0000 (UTC)
Received: (qmail 86759 invoked by uid 500); 13 Jun 2011 21:22:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86705 invoked by uid 500); 13 Jun 2011 21:22:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86643 invoked by uid 99); 13 Jun 2011 21:22:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 21:22:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 21:22:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4D4444138E0
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 21:21:48 +0000 (UTC)
Date: Mon, 13 Jun 2011 21:21:48 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1956595386.51.1308000108313.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048788#comment-13048788 ] 

Alejandro Abdelnur commented on HADOOP-6671:
--------------------------------------------

Some additional info on the TestUserGroupInformation failure, if running the test alone (-Dtest=TestUserGroupInformation) the test passes (both on Mac and Linux)

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-6671-cross-project-HDFS.patch, HADOOP-6671-e.patch, HADOOP-6671-f.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout-f.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16242-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 21:26:08 2011
Return-Path: <common-issues-return-16242-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9EDCD6EA1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 21:26:08 +0000 (UTC)
Received: (qmail 97150 invoked by uid 500); 13 Jun 2011 21:26:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97121 invoked by uid 500); 13 Jun 2011 21:26:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97113 invoked by uid 99); 13 Jun 2011 21:26:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 21:26:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 21:26:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 65328413B15
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 21:25:47 +0000 (UTC)
Date: Mon, 13 Jun 2011 21:25:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <664739753.84.1308000347411.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048793#comment-13048793 ] 

Aaron T. Myers commented on HADOOP-6671:
----------------------------------------

@Alejandro: does this test fail for you when running the full test suite, but not when run in isolation? Or does this test not fail for you at all?

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-6671-cross-project-HDFS.patch, HADOOP-6671-e.patch, HADOOP-6671-f.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout-f.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16243-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 21:32:09 2011
Return-Path: <common-issues-return-16243-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6ECCA6AF7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 21:32:09 +0000 (UTC)
Received: (qmail 5542 invoked by uid 500); 13 Jun 2011 21:32:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5516 invoked by uid 500); 13 Jun 2011 21:32:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5508 invoked by uid 99); 13 Jun 2011 21:32:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 21:32:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 21:32:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 71D8A413CDA
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 21:31:47 +0000 (UTC)
Date: Mon, 13 Jun 2011 21:31:47 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <999591541.125.1308000707462.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6835) Support concatenated gzip files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Allen Wittenauer updated HADOOP-6835:
-------------------------------------

    Summary: Support concatenated gzip files  (was: Support concatenated gzip and bzip2 files)

> Support concatenated gzip files
> -------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>             Fix For: 0.22.0
>
>         Attachments: C6835-9.patch, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v4.trunk-hadoop-common.patch, HADOOP-6835.v4.trunk-hadoop-mapreduce.patch, HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v5.trunk-hadoop-common.patch, HADOOP-6835.v6.trunk-hadoop-common.patch, HADOOP-6835.v7.trunk-hadoop-common.patch, HADOOP-6835.v8.trunk-hadoop-common.patch, HADOOP-6835.v9.yahoo-0.20.2xx-branch.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch, grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16244-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 21:34:08 2011
Return-Path: <common-issues-return-16244-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CFEB26423
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 21:34:08 +0000 (UTC)
Received: (qmail 6403 invoked by uid 500); 13 Jun 2011 21:34:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6326 invoked by uid 500); 13 Jun 2011 21:34:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6318 invoked by uid 99); 13 Jun 2011 21:34:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 21:34:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 21:34:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7C0B2413E3F
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 21:33:47 +0000 (UTC)
Date: Mon, 13 Jun 2011 21:33:47 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1674077446.146.1308000827504.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7386) Support concatenated bzip2 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Support concatenated bzip2 files
--------------------------------

                 Key: HADOOP-7386
                 URL: https://issues.apache.org/jira/browse/HADOOP-7386
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Allen Wittenauer


HADOOP-6835 added the framework and direct support for concatenated gzip files.  We should do the same for bzip files.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16245-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 21:40:11 2011
Return-Path: <common-issues-return-16245-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2788B6747
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 21:40:11 +0000 (UTC)
Received: (qmail 21284 invoked by uid 500); 13 Jun 2011 21:40:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21096 invoked by uid 500); 13 Jun 2011 21:40:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21088 invoked by uid 99); 13 Jun 2011 21:40:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 21:40:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 21:40:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 665984134C2
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 21:39:47 +0000 (UTC)
Date: Mon, 13 Jun 2011 21:39:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1444673059.214.1308001187388.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048800#comment-13048800 ] 

Todd Lipcon commented on HADOOP-6671:
-------------------------------------

Is maven executing tests in a "per-test fork" mode? Or is it trying to execute them all in one JVM instantiation? UGI relies on some static state (the "login user") which might be busted if it runs in the same JVM as other test cases.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-6671-cross-project-HDFS.patch, HADOOP-6671-e.patch, HADOOP-6671-f.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout-f.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16246-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 21:44:11 2011
Return-Path: <common-issues-return-16246-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B41D66929
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 21:44:11 +0000 (UTC)
Received: (qmail 28485 invoked by uid 500); 13 Jun 2011 21:44:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28458 invoked by uid 500); 13 Jun 2011 21:44:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28440 invoked by uid 99); 13 Jun 2011 21:44:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 21:44:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 21:44:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E862B413642
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 21:43:47 +0000 (UTC)
Date: Mon, 13 Jun 2011 21:43:47 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <800822925.265.1308001427948.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048803#comment-13048803 ] 

Alejandro Abdelnur commented on HADOOP-6671:
--------------------------------------------

@ATM: only when running full test suite, just that test passes
@Todd: doing a per-test fork (that is the odd thing, at first thought the same)

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-6671-cross-project-HDFS.patch, HADOOP-6671-e.patch, HADOOP-6671-f.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout-f.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16247-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 21:50:08 2011
Return-Path: <common-issues-return-16247-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C2E286F80
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 21:50:08 +0000 (UTC)
Received: (qmail 39040 invoked by uid 500); 13 Jun 2011 21:50:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38999 invoked by uid 500); 13 Jun 2011 21:50:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38991 invoked by uid 99); 13 Jun 2011 21:50:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 21:50:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 21:50:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 77A124138AC
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 21:49:47 +0000 (UTC)
Date: Mon, 13 Jun 2011 21:49:47 +0000 (UTC)
From: "Brock Noland (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <21611571.277.1308001787486.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1160101136.38164.1306200167436.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7325) hadoop command - do not accept
 class names starting with a hyphen
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048805#comment-13048805 ] 

Brock Noland commented on HADOOP-7325:
--------------------------------------

Looking at the hdfs and mapred commands, they do not seem to allow the "command" to be a direct classname and also have unknown command handling built in, so I do not think they need to be updated. If someone disagrees, please share!

{code}
elif [ "$COMMAND" = "groups" ] ; then
  CLASS=org.apache.hadoop.hdfs.tools.GetGroups
else
  echo $COMMAND - invalid command
  print_usage
  exit
{code}

> hadoop command - do not accept class names starting with a hyphen
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7325
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Brock Noland
>            Assignee: Brock Noland
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-illegal-class-name-0.patch, hadoop-illegal-class-name-1.patch, hadoop-illegal-class-name-2.patch
>
>
> If this is committed I will look at patches for hdfs and mapred.
> When teaching a good portion of the students in every single class execute:
> {code}
> $ hadoop -fs ls /
> {code}
> The -fs is passed directly to the JVM and the JVM fails to start:
> {code}
> $ ./bin/hadoop -fs ls /
> Unrecognized option: -fs
> Could not create the Java virtual machine.
> {code}
> Which is confusing and typically requires explanation. The attached patch improves that behavior:
> {code}
> $ ./bin/hadoop -fs ls /
> Error: No command named `-fs' was found. Perhaps you meant `hadoop fs'
> {code}
> The only risk I can see is if someone is abusing the implementation of hadoop command doing something like so:
> {code}
> $ ./bin/hadoop -Xmx1g  org.apache.hadoop.util.RunJar
> RunJar jarFile [mainClass] args...
> {code}
> The hadoop command does not appear to advertise allowing JVM options before the classname.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16248-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 22:26:12 2011
Return-Path: <common-issues-return-16248-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0C3B06AF2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 22:26:12 +0000 (UTC)
Received: (qmail 20425 invoked by uid 500); 13 Jun 2011 22:26:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20359 invoked by uid 500); 13 Jun 2011 22:26:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20312 invoked by uid 99); 13 Jun 2011 22:26:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 22:26:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 22:26:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B9B8A413D1A
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 22:25:47 +0000 (UTC)
Date: Mon, 13 Jun 2011 22:25:47 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1009900334.382.1308003947757.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7387) Change rpm to ignore configuration
 files when uninstalling
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Change rpm to ignore configuration files when uninstalling
----------------------------------------------------------

                 Key: HADOOP-7387
                 URL: https://issues.apache.org/jira/browse/HADOOP-7387
             Project: Hadoop Common
          Issue Type: Improvement
         Environment: Java 6, RHEL 5.5
            Reporter: Eric Yang
            Assignee: Eric Yang
            Priority: Minor


When uninstalling RPM files, the config files are renamed from core-site.xml to core-site.xml.rpmsave.  It would be nice if config file does not get renamed.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16249-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 22:28:10 2011
Return-Path: <common-issues-return-16249-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BBAD76B47
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 22:28:10 +0000 (UTC)
Received: (qmail 23234 invoked by uid 500); 13 Jun 2011 22:28:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23202 invoked by uid 500); 13 Jun 2011 22:28:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23194 invoked by uid 99); 13 Jun 2011 22:28:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 22:28:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 22:28:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7FD0A413DAC
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 22:27:47 +0000 (UTC)
Date: Mon, 13 Jun 2011 22:27:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1289531762.389.1308004067520.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6605:
--------------------------------

    Attachment: hadoop-6605-5.patch

Thanks guys.  Patch attached, same as the last, just changes the case stmt to an if stmt.

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch, hadoop-6605-2.patch, hadoop-6605-3.patch, hadoop-6605-4.patch, hadoop-6605-5.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16250-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 22:32:08 2011
Return-Path: <common-issues-return-16250-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CB5CF6563
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 22:32:08 +0000 (UTC)
Received: (qmail 26596 invoked by uid 500); 13 Jun 2011 22:32:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26565 invoked by uid 500); 13 Jun 2011 22:32:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26557 invoked by uid 99); 13 Jun 2011 22:32:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 22:32:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 22:32:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8752B413F4E
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 22:31:47 +0000 (UTC)
Date: Mon, 13 Jun 2011 22:31:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1451616256.403.1308004307551.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1160101136.38164.1306200167436.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7325) hadoop command - do not accept
 class names starting with a hyphen
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048835#comment-13048835 ] 

Todd Lipcon commented on HADOOP-7325:
-------------------------------------

Sounds right to me. Thanks for looking.

> hadoop command - do not accept class names starting with a hyphen
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7325
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Brock Noland
>            Assignee: Brock Noland
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-illegal-class-name-0.patch, hadoop-illegal-class-name-1.patch, hadoop-illegal-class-name-2.patch
>
>
> If this is committed I will look at patches for hdfs and mapred.
> When teaching a good portion of the students in every single class execute:
> {code}
> $ hadoop -fs ls /
> {code}
> The -fs is passed directly to the JVM and the JVM fails to start:
> {code}
> $ ./bin/hadoop -fs ls /
> Unrecognized option: -fs
> Could not create the Java virtual machine.
> {code}
> Which is confusing and typically requires explanation. The attached patch improves that behavior:
> {code}
> $ ./bin/hadoop -fs ls /
> Error: No command named `-fs' was found. Perhaps you meant `hadoop fs'
> {code}
> The only risk I can see is if someone is abusing the implementation of hadoop command doing something like so:
> {code}
> $ ./bin/hadoop -Xmx1g  org.apache.hadoop.util.RunJar
> RunJar jarFile [mainClass] args...
> {code}
> The hadoop command does not appear to advertise allowing JVM options before the classname.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16251-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 22:32:09 2011
Return-Path: <common-issues-return-16251-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0519D656F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 22:32:09 +0000 (UTC)
Received: (qmail 26829 invoked by uid 500); 13 Jun 2011 22:32:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26788 invoked by uid 500); 13 Jun 2011 22:32:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26663 invoked by uid 99); 13 Jun 2011 22:32:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 22:32:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 22:32:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AEF22413F54
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 22:31:47 +0000 (UTC)
Date: Mon, 13 Jun 2011 22:31:47 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <589245674.409.1308004307713.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1507121853.1613.1307925711791.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7384) Allow test-patch to be more
 flexible about patch format
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048836#comment-13048836 ] 

Tom White commented on HADOOP-7384:
-----------------------------------

+1 This looks like a useful change.

> Allow test-patch to be more flexible about patch format
> -------------------------------------------------------
>
>                 Key: HADOOP-7384
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7384
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-7384.txt
>
>
> Right now the test-patch process only accepts patches that are generated as "-p0" relative to common/, hdfs/, or mapreduce/. This has always been annoying for git users where the default patch format is -p1. It's also now annoying for SVN users who may generate a patch relative to trunk/ instead of the subproject subdirectory. We should auto-detect the correct patch level.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16252-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 22:36:13 2011
Return-Path: <common-issues-return-16252-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EB6BF618C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 22:36:12 +0000 (UTC)
Received: (qmail 35239 invoked by uid 500); 13 Jun 2011 22:36:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35198 invoked by uid 500); 13 Jun 2011 22:36:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35185 invoked by uid 99); 13 Jun 2011 22:36:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 22:36:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 22:36:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0658C41334F
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 22:35:49 +0000 (UTC)
Date: Mon, 13 Jun 2011 22:35:49 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2082700715.455.1308004549022.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048839#comment-13048839 ] 

Daryn Sharp commented on HADOOP-6605:
-------------------------------------

+1 I'd recommend double square brackets again to eliminate need to quote, but not a huge deal.  Can't wait to enjoy this patch!

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch, hadoop-6605-2.patch, hadoop-6605-3.patch, hadoop-6605-4.patch, hadoop-6605-5.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16253-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 22:38:10 2011
Return-Path: <common-issues-return-16253-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1389C61F5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 22:38:10 +0000 (UTC)
Received: (qmail 36550 invoked by uid 500); 13 Jun 2011 22:38:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36477 invoked by uid 500); 13 Jun 2011 22:38:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36295 invoked by uid 99); 13 Jun 2011 22:38:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 22:38:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 22:38:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5A480413431
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 22:37:47 +0000 (UTC)
Date: Mon, 13 Jun 2011 22:37:47 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <9085317.460.1308004667366.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7388) Remove definition of HADOOP_HOME and
 HADOOP_PREFIX from hadoop-env.sh.template
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Remove definition of HADOOP_HOME and HADOOP_PREFIX from hadoop-env.sh.template
------------------------------------------------------------------------------

                 Key: HADOOP-7388
                 URL: https://issues.apache.org/jira/browse/HADOOP-7388
             Project: Hadoop Common
          Issue Type: Improvement
         Environment: Java 6, RHEL 5.5
            Reporter: Eric Yang
            Assignee: Eric Yang
            Priority: Trivial


The file structure layout proposed in HADOOP-6255 was designed to remove the need of using HADOOP_HOME environment to locate hadoop bits.  The file structure layout should be able to map to /usr or system directories, therefore HADOOP_HOME is renamed to HADOOP_PREFIX to be more concise.  HADOOP_PREFIX should not be exported to the user.  If the user use hadoop-setup-single-node.sh or hadoop-setup-conf.sh to configure hadoop, the current scripts put HADOOP_PREFIX/HADOOP_HOME in hadoop-env.sh.  The config template generation code should remove reference of HADOOP_PREFIX/HADOOP_HOME from hadoop-env.sh.


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16254-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 22:46:08 2011
Return-Path: <common-issues-return-16254-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D2F746B50
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 22:46:08 +0000 (UTC)
Received: (qmail 46765 invoked by uid 500); 13 Jun 2011 22:46:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46743 invoked by uid 500); 13 Jun 2011 22:46:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46735 invoked by uid 99); 13 Jun 2011 22:46:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 22:46:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 22:46:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9A6D641376E
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 22:45:47 +0000 (UTC)
Date: Mon, 13 Jun 2011 22:45:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <787366486.476.1308005147629.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048843#comment-13048843 ] 

Eli Collins commented on HADOOP-6605:
-------------------------------------

Thanks. I left the if as single bracket to be consistent w the rest of the file. Perhaps file a new jira to convert bin/* over to double brackets? 

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch, hadoop-6605-2.patch, hadoop-6605-3.patch, hadoop-6605-4.patch, hadoop-6605-5.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16255-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 22:48:11 2011
Return-Path: <common-issues-return-16255-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 14F956895
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 22:48:11 +0000 (UTC)
Received: (qmail 50107 invoked by uid 500); 13 Jun 2011 22:48:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50075 invoked by uid 500); 13 Jun 2011 22:48:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50067 invoked by uid 99); 13 Jun 2011 22:48:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 22:48:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 22:48:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C935D413879
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 22:47:47 +0000 (UTC)
Date: Mon, 13 Jun 2011 22:47:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1378699221.489.1308005267821.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6605:
--------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I've committed this. Thanks guys!

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch, hadoop-6605-2.patch, hadoop-6605-3.patch, hadoop-6605-4.patch, hadoop-6605-5.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16256-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 22:52:10 2011
Return-Path: <common-issues-return-16256-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 81E7D6AFA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 22:52:10 +0000 (UTC)
Received: (qmail 53609 invoked by uid 500); 13 Jun 2011 22:52:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53568 invoked by uid 500); 13 Jun 2011 22:52:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53560 invoked by uid 99); 13 Jun 2011 22:52:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 22:52:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 22:52:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 155DD413A14
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 22:51:49 +0000 (UTC)
Date: Mon, 13 Jun 2011 22:51:49 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <538357149.547.1308005509083.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048850#comment-13048850 ] 

Alejandro Abdelnur commented on HADOOP-7206:
--------------------------------------------

Jake,

1. Snappy-Java bundles the native libraries in the JAR itself. While that is convenient/clever packaging technique, this is different from how Hadoop handles native libraries (loading them from lib/native/${OS_ARCH}/).

2. The motivation for keeping hadoop-snappy independent of hadoop was that we could use it right the way in other projects (HBase already integrated it).

I would strongly argue that native libraries should handled in a consistent maner in Hadoop.

And, if the preference of the Hadoop folks is to bundle snappy in Hadoop (dismissing #2), then I'd advocate for bringing Hadoop-Snappy into Hadoop as this JIRA originally proposed. By doing this we would have 1 external dependency (snappy) instead 2 (snappy-java and snappy, with the side effect that if we need a new version of snappy we would have to wait for snappy-java to do a release with it).

Thoughts?



> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>         Attachments: HADOOP-7206.patch
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16257-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 22:52:10 2011
Return-Path: <common-issues-return-16257-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AFB856B04
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 22:52:10 +0000 (UTC)
Received: (qmail 53840 invoked by uid 500); 13 Jun 2011 22:52:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53763 invoked by uid 500); 13 Jun 2011 22:52:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53569 invoked by uid 99); 13 Jun 2011 22:52:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 22:52:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 22:52:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1DFCD413A15
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 22:51:49 +0000 (UTC)
Date: Mon, 13 Jun 2011 22:51:49 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2072171727.548.1308005509119.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048851#comment-13048851 ] 

Alejandro Abdelnur commented on HADOOP-7206:
--------------------------------------------

Jake,

1. Snappy-Java bundles the native libraries in the JAR itself. While that is convenient/clever packaging technique, this is different from how Hadoop handles native libraries (loading them from lib/native/${OS_ARCH}/).

2. The motivation for keeping hadoop-snappy independent of hadoop was that we could use it right the way in other projects (HBase already integrated it).

I would strongly argue that native libraries should handled in a consistent maner in Hadoop.

And, if the preference of the Hadoop folks is to bundle snappy in Hadoop (dismissing #2), then I'd advocate for bringing Hadoop-Snappy into Hadoop as this JIRA originally proposed. By doing this we would have 1 external dependency (snappy) instead 2 (snappy-java and snappy, with the side effect that if we need a new version of snappy we would have to wait for snappy-java to do a release with it).

Thoughts?



> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>         Attachments: HADOOP-7206.patch
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16258-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 23:08:08 2011
Return-Path: <common-issues-return-16258-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CADE46345
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 23:08:08 +0000 (UTC)
Received: (qmail 77625 invoked by uid 500); 13 Jun 2011 23:08:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77591 invoked by uid 500); 13 Jun 2011 23:08:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77472 invoked by uid 99); 13 Jun 2011 23:08:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:08:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:08:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 721BF4140CF
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 23:07:47 +0000 (UTC)
Date: Mon, 13 Jun 2011 23:07:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <829548377.644.1308006467463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048860#comment-13048860 ] 

Hudson commented on HADOOP-6605:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #652 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/652/])
    HADOOP-6605. Add JAVA_HOME detection to hadoop-config. Contributed by Eli Collins

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1135333
Files : 
* /hadoop/common/trunk/common/bin/hadoop-config.sh
* /hadoop/common/trunk/common/CHANGES.txt


> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch, hadoop-6605-2.patch, hadoop-6605-3.patch, hadoop-6605-4.patch, hadoop-6605-5.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16259-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 23:16:13 2011
Return-Path: <common-issues-return-16259-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4821B66E4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 23:16:13 +0000 (UTC)
Received: (qmail 81694 invoked by uid 500); 13 Jun 2011 23:16:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81657 invoked by uid 500); 13 Jun 2011 23:16:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81644 invoked by uid 99); 13 Jun 2011 23:16:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:16:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:16:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7AEB04143AA
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 23:15:48 +0000 (UTC)
Date: Mon, 13 Jun 2011 23:15:48 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <705488537.689.1308006948499.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-6671:
---------------------------------------

    Attachment: HADOOP-6671-g.patch

handles TAR creating in the same way other Ant delegated task are handled.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-6671-cross-project-HDFS.patch, HADOOP-6671-e.patch, HADOOP-6671-f.patch, HADOOP-6671-g.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout-f.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16260-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 23:22:10 2011
Return-Path: <common-issues-return-16260-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6FF446A98
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 23:22:10 +0000 (UTC)
Received: (qmail 87675 invoked by uid 500); 13 Jun 2011 23:22:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87645 invoked by uid 500); 13 Jun 2011 23:22:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87637 invoked by uid 99); 13 Jun 2011 23:22:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:22:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:22:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0E5E74146AA
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 23:21:47 +0000 (UTC)
Date: Mon, 13 Jun 2011 23:21:47 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <975289640.729.1308007307055.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048870#comment-13048870 ] 

Tom White commented on HADOOP-6671:
-----------------------------------

Per-test in Ant (and Maven) creates a new JVM per TestCase class (not per method). When I move testGetServerSideGroups() to be the last method in TestUserGroupInformation it consistently fails in Ant, which suggests that it relies on static state (as Todd suggested).

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-6671-cross-project-HDFS.patch, HADOOP-6671-e.patch, HADOOP-6671-f.patch, HADOOP-6671-g.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout-f.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16261-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 23:26:08 2011
Return-Path: <common-issues-return-16261-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 811DB6B93
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 23:26:08 +0000 (UTC)
Received: (qmail 92425 invoked by uid 500); 13 Jun 2011 23:26:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92400 invoked by uid 500); 13 Jun 2011 23:26:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92392 invoked by uid 99); 13 Jun 2011 23:26:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:26:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:26:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3EA1A41489C
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 23:25:47 +0000 (UTC)
Date: Mon, 13 Jun 2011 23:25:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1548816915.763.1308007547253.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Updated] (HADOOP-7106) Re-organize hadoop subversion layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7106:
--------------------------------

    Attachment: mailer-conf.diff

Here's a diff for asf-mailer.conf for the new layout.

I simplified it a bit by using a fancier regular expression in the for_paths config. According to http://opensource.perlig.de/svnmailer/doc-1.0/ this is allowed.

> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, gitk-example.png, mailer-conf.diff
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16262-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 23:32:08 2011
Return-Path: <common-issues-return-16262-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 94DC36741
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 23:32:08 +0000 (UTC)
Received: (qmail 4936 invoked by uid 500); 13 Jun 2011 23:32:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4908 invoked by uid 500); 13 Jun 2011 23:32:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4900 invoked by uid 99); 13 Jun 2011 23:32:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:32:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:32:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2EE4E414B2B
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 23:31:47 +0000 (UTC)
Date: Mon, 13 Jun 2011 23:31:47 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <118449070.832.1308007907188.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur reassigned HADOOP-7206:
------------------------------------------

    Assignee: Alejandro Abdelnur

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-7206.patch
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16263-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 23:34:11 2011
Return-Path: <common-issues-return-16263-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F100C60E7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 23:34:10 +0000 (UTC)
Received: (qmail 9399 invoked by uid 500); 13 Jun 2011 23:34:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9343 invoked by uid 500); 13 Jun 2011 23:34:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9335 invoked by uid 99); 13 Jun 2011 23:34:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:34:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:34:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AB084414C2E
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 23:33:47 +0000 (UTC)
Date: Mon, 13 Jun 2011 23:33:47 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <843446758.904.1308008027697.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048879#comment-13048879 ] 

Alejandro Abdelnur commented on HADOOP-6671:
--------------------------------------------

@Tom: that explains, thxs for the detective work. In my answer to Todd I meant testcase, didn't associate the per test to per test method.

Then it makes sense to open a different JIRA to fix this testcase, right? (I don't think it would be an easy thing to do to get one JVM per test method, not to mention it would be tooooo slow)

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-6671-cross-project-HDFS.patch, HADOOP-6671-e.patch, HADOOP-6671-f.patch, HADOOP-6671-g.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout-f.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16264-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 23:38:10 2011
Return-Path: <common-issues-return-16264-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DA67C632A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 23:38:09 +0000 (UTC)
Received: (qmail 18786 invoked by uid 500); 13 Jun 2011 23:38:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18755 invoked by uid 500); 13 Jun 2011 23:38:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18747 invoked by uid 99); 13 Jun 2011 23:38:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:38:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:38:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 943AB414DBE
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 23:37:48 +0000 (UTC)
Date: Mon, 13 Jun 2011 23:37:48 +0000 (UTC)
From: "T Jake Luciani (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1341615972.958.1308008268603.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048882#comment-13048882 ] 

T Jake Luciani commented on HADOOP-7206:
----------------------------------------

I agree with 1) this should be in the source tree... however, if 2) is the stopgap, then it should be as easy as possible for folks to use these plugins till then (hence the choice of snappy-java).

IMO the apache builds should come with native libraries for common platforms as well as packages.  I'd be happy to see hadoop-snappy in the src but without proper distribution it's a barrier to entry for using native libraries.



> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-7206.patch
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16265-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 23:38:12 2011
Return-Path: <common-issues-return-16265-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 709CB6359
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 23:38:12 +0000 (UTC)
Received: (qmail 19267 invoked by uid 500); 13 Jun 2011 23:38:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19233 invoked by uid 500); 13 Jun 2011 23:38:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19225 invoked by uid 99); 13 Jun 2011 23:38:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:38:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:38:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E8F4D414DCA
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 23:37:48 +0000 (UTC)
Date: Mon, 13 Jun 2011 23:37:48 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1899798280.970.1308008268951.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048883#comment-13048883 ] 

Suresh Srinivas commented on HADOOP-7144:
-----------------------------------------

Is the JMX provided in this jira exposes the document equivalent to the MBean or an MBeam method?

If it is entire MBean, then this can cause issues with Namenode where live node list, dead node list, decom node list etc are returned in one huge json document. Should the access be per mbean property?

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Robert Joseph Evans
>              Labels: jmx
>             Fix For: 0.20.205.0, 0.23.0
>
>         Attachments: HADOOP-7411-0.20.20X-V1.patch, HADOOP-7411-0.20.20X-V2.patch, HADOOP-7411-trunk-V1.patch, HADOOP-7411-trunk-V2.patch, HADOOP-7411-trunk-alpha.patch, jmx.json
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16266-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 23:44:08 2011
Return-Path: <common-issues-return-16266-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 712D76443
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 23:44:08 +0000 (UTC)
Received: (qmail 24744 invoked by uid 500); 13 Jun 2011 23:44:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24640 invoked by uid 500); 13 Jun 2011 23:44:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24632 invoked by uid 99); 13 Jun 2011 23:44:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:44:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:44:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2750C41323E
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 23:43:47 +0000 (UTC)
Date: Mon, 13 Jun 2011 23:43:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1997856959.987.1308008627157.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048889#comment-13048889 ] 

Aaron T. Myers commented on HADOOP-6671:
----------------------------------------

I've figured out the issue. {{UserGroupInformation.createUserForTesting(...)}} replaces {{UserGroupInformation}}'s static reference to {{groups}} with an instance of {{TestingGroups}}. Once this occurs in a JVM, no subsequent calls to {{UserGroupInformation.getGroupNames(...)}} will work for any real users on the system. The test case in question compares the groups of the real user running the tests with the groups of that same user as determined by the {{UserGroupInformation}} class.

The explanation for why this fails under maven when run as part of the suite, but not under maven when run in isolation or under ant is that those must run the test cases in different orders.

The solution is that tests which call {{createUserForTesting}} should reset the static reference when they complete. I can work on a patch for this. Tom/Alejandro - do you think this should be done as a separate JIRA? Or part of this one?

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-6671-cross-project-HDFS.patch, HADOOP-6671-e.patch, HADOOP-6671-f.patch, HADOOP-6671-g.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout-f.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16267-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 23:46:11 2011
Return-Path: <common-issues-return-16267-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D448F6B97
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 23:46:11 +0000 (UTC)
Received: (qmail 27698 invoked by uid 500); 13 Jun 2011 23:46:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27667 invoked by uid 500); 13 Jun 2011 23:46:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27659 invoked by uid 99); 13 Jun 2011 23:46:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:46:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:46:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 452554133DE
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 23:45:47 +0000 (UTC)
Date: Mon, 13 Jun 2011 23:45:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <615882170.1020.1308008747279.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048891#comment-13048891 ] 

Matt Foley commented on HADOOP-7328:
------------------------------------

Moving this discussion back to the Jira to decrease the number and size of emails and jira appends :-)

It appears to me that getSerializer() should throw an exception on failure to return a usable value, because:
* The method is only called on objects for which a serializer is expected to exist, so this truly is an "exceptional" condition.
* The typical caller has no alternate strategy, as far as I know, so it will have to do a check for null then throw an exception anyway.

The argument for consistency has merit, but this is all Hadoop stuff, so it is okay for us to improve the semantics, if such it would be.  Nevertheless, I'm a "0" on this, do what you think best.

And yes, please, any method that might return null should clearly document that fact in the javadocs.

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff, HADOOP-7328.r2.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16268-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 23:53:10 2011
Return-Path: <common-issues-return-16268-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A890D6ABB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 23:53:10 +0000 (UTC)
Received: (qmail 29824 invoked by uid 500); 13 Jun 2011 23:53:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29789 invoked by uid 500); 13 Jun 2011 23:53:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29781 invoked by uid 99); 13 Jun 2011 23:53:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:53:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:53:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 78BBF413622
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 23:52:47 +0000 (UTC)
Date: Mon, 13 Jun 2011 23:52:47 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1893629280.1035.1308009167491.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1415670632.7518.1307638679263.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7371) Improve tarball distributions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-7371:
------------------------------

    Attachment: HADOOP-7371.patch

For "ant tar":

- Sources are compressed to a jar file as $HADOOP_PREFIX/share/hadoop/hadoop-source-[version].jar
- Javadoc is compressed as $HADOOP_PREFIX/share/javadoc/hadoop-javadoc-[version].jar
- Documents are relocated to $HADOOP_PREFIX/share/doc/hadoop
- HADOOP_HOME is set to be the same as HADOOP_PREFIX

> Improve tarball distributions
> -----------------------------
>
>                 Key: HADOOP-7371
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7371
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>         Environment: Java 6, Redhat 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>             Fix For: 0.20.205.0
>
>         Attachments: HADOOP-7371.patch
>
>
> Hadoop release tarball contains both raw source and binary.  This leads users to use the release tarball as base for applying patches, to build custom Hadoop.  This is not the recommended method to develop hadoop because it leads to mixed development system where processed files and raw source are hard to separate.  
> To correct the problematic usage of the release tarball, the release build target should be defined as:
> "ant source" generates source release tarball.
> "ant binary" is binary release without source/javadoc jar files.
> "ant tar" is a mirror of binary release with source/javadoc jar files.
> Does this sound reasonable?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16269-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 23:57:09 2011
Return-Path: <common-issues-return-16269-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 336AA6F73
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 23:57:09 +0000 (UTC)
Received: (qmail 37667 invoked by uid 500); 13 Jun 2011 23:57:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37614 invoked by uid 500); 13 Jun 2011 23:57:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37604 invoked by uid 99); 13 Jun 2011 23:57:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:57:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:57:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B95784137EE
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 23:56:47 +0000 (UTC)
Date: Mon, 13 Jun 2011 23:56:47 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <867250086.1052.1308009407755.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1415670632.7518.1307638679263.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7371) Improve tarball distributions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048900#comment-13048900 ] 

Eric Yang commented on HADOOP-7371:
-----------------------------------

For "ant source"

- Create a hadoop-source-[version].tar.gz in build directory

> Improve tarball distributions
> -----------------------------
>
>                 Key: HADOOP-7371
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7371
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>         Environment: Java 6, Redhat 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>             Fix For: 0.20.205.0
>
>         Attachments: HADOOP-7371.patch
>
>
> Hadoop release tarball contains both raw source and binary.  This leads users to use the release tarball as base for applying patches, to build custom Hadoop.  This is not the recommended method to develop hadoop because it leads to mixed development system where processed files and raw source are hard to separate.  
> To correct the problematic usage of the release tarball, the release build target should be defined as:
> "ant source" generates source release tarball.
> "ant binary" is binary release without source/javadoc jar files.
> "ant tar" is a mirror of binary release with source/javadoc jar files.
> Does this sound reasonable?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16270-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 23:57:09 2011
Return-Path: <common-issues-return-16270-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 633576F84
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 23:57:09 +0000 (UTC)
Received: (qmail 37880 invoked by uid 500); 13 Jun 2011 23:57:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37828 invoked by uid 500); 13 Jun 2011 23:57:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37668 invoked by uid 99); 13 Jun 2011 23:57:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:57:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:57:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D1EC24137F2
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 23:56:47 +0000 (UTC)
Date: Mon, 13 Jun 2011 23:56:47 +0000 (UTC)
From: "Brian Bockelman (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <761331786.1056.1308009407856.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1009900334.382.1308003947757.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7387) Change rpm to ignore configuration
 files when uninstalling
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048902#comment-13048902 ] 

Brian Bockelman commented on HADOOP-7387:
-----------------------------------------

I'm not sure this is a great idea.

Like it or not (I don't really like the behavior), it's the way RPMs are expected to behave on the platform.  To "fix" the behavior will make the RPM stick out.

> Change rpm to ignore configuration files when uninstalling
> ----------------------------------------------------------
>
>                 Key: HADOOP-7387
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7387
>             Project: Hadoop Common
>          Issue Type: Improvement
>         Environment: Java 6, RHEL 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>            Priority: Minor
>
> When uninstalling RPM files, the config files are renamed from core-site.xml to core-site.xml.rpmsave.  It would be nice if config file does not get renamed.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16271-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 13 23:57:11 2011
Return-Path: <common-issues-return-16271-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 46AFA6F91
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 13 Jun 2011 23:57:11 +0000 (UTC)
Received: (qmail 38181 invoked by uid 500); 13 Jun 2011 23:57:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38152 invoked by uid 500); 13 Jun 2011 23:57:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38143 invoked by uid 99); 13 Jun 2011 23:57:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:57:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jun 2011 23:57:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E992C4137F5
	for <common-issues@hadoop.apache.org>; Mon, 13 Jun 2011 23:56:47 +0000 (UTC)
Date: Mon, 13 Jun 2011 23:56:47 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <983225575.1059.1308009407953.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048903#comment-13048903 ] 

Alejandro Abdelnur commented on HADOOP-6671:
--------------------------------------------

@ATM, thanks. 

IMO it should be a separate JIRA

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-6671-cross-project-HDFS.patch, HADOOP-6671-e.patch, HADOOP-6671-f.patch, HADOOP-6671-g.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout-f.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16272-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 00:21:10 2011
Return-Path: <common-issues-return-16272-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B540D6E6D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 00:21:10 +0000 (UTC)
Received: (qmail 59638 invoked by uid 500); 14 Jun 2011 00:21:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59591 invoked by uid 500); 14 Jun 2011 00:21:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59583 invoked by uid 99); 14 Jun 2011 00:21:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 00:21:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 00:21:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 83116414056
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 00:20:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 00:20:47 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <336474648.1143.1308010847533.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1009900334.382.1308003947757.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Work started] (HADOOP-7387) Change rpm to ignore
 configuration files when uninstalling
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Work on HADOOP-7387 started by Eric Yang.

> Change rpm to ignore configuration files when uninstalling
> ----------------------------------------------------------
>
>                 Key: HADOOP-7387
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7387
>             Project: Hadoop Common
>          Issue Type: Improvement
>         Environment: Java 6, RHEL 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>            Priority: Minor
>
> When uninstalling RPM files, the config files are renamed from core-site.xml to core-site.xml.rpmsave.  It would be nice if config file does not get renamed.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16273-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 00:23:10 2011
Return-Path: <common-issues-return-16273-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CE95B6F2D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 00:23:10 +0000 (UTC)
Received: (qmail 61821 invoked by uid 500); 14 Jun 2011 00:23:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61775 invoked by uid 500); 14 Jun 2011 00:23:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61757 invoked by uid 99); 14 Jun 2011 00:23:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 00:23:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 00:23:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 59D15414128
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 00:22:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 00:22:47 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1523789827.1146.1308010967364.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1009900334.382.1308003947757.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7387) Change rpm to ignore configuration
 files when uninstalling
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-7387:
------------------------------

    Status: Patch Available  (was: In Progress)

> Change rpm to ignore configuration files when uninstalling
> ----------------------------------------------------------
>
>                 Key: HADOOP-7387
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7387
>             Project: Hadoop Common
>          Issue Type: Improvement
>         Environment: Java 6, RHEL 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>            Priority: Minor
>
> When uninstalling RPM files, the config files are renamed from core-site.xml to core-site.xml.rpmsave.  It would be nice if config file does not get renamed.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16274-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 00:23:11 2011
Return-Path: <common-issues-return-16274-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 08B086F37
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 00:23:11 +0000 (UTC)
Received: (qmail 62041 invoked by uid 500); 14 Jun 2011 00:23:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61921 invoked by uid 500); 14 Jun 2011 00:23:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61761 invoked by uid 99); 14 Jun 2011 00:23:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 00:23:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 00:23:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6CEA541412B
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 00:22:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 00:22:47 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <403402530.1149.1308010967443.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1009900334.382.1308003947757.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7387) Change rpm to ignore configuration
 files when uninstalling
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-7387:
------------------------------

    Resolution: Won't Fix
        Status: Resolved  (was: Patch Available)

Brian is right, the rpm behavior is not changeable, see:

[http://www-uxsup.csx.cam.ac.uk/~jw35/docs/rpm_config.html]

And there are good reasons for .rpmsave backup files from package manager prospective.  Hence, this is a won't fix.

> Change rpm to ignore configuration files when uninstalling
> ----------------------------------------------------------
>
>                 Key: HADOOP-7387
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7387
>             Project: Hadoop Common
>          Issue Type: Improvement
>         Environment: Java 6, RHEL 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>            Priority: Minor
>
> When uninstalling RPM files, the config files are renamed from core-site.xml to core-site.xml.rpmsave.  It would be nice if config file does not get renamed.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16275-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 01:07:12 2011
Return-Path: <common-issues-return-16275-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4CEA16CF7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 01:07:12 +0000 (UTC)
Received: (qmail 12156 invoked by uid 500); 14 Jun 2011 01:07:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12087 invoked by uid 500); 14 Jun 2011 01:07:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11869 invoked by uid 99); 14 Jun 2011 01:07:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 01:07:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 01:07:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6096A414C92
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 01:06:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 01:06:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <272967917.1210.1308013607392.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7389) Use of TestingGroups by tests causes
 subsequent tests to fail
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Use of TestingGroups by tests causes subsequent tests to fail
-------------------------------------------------------------

                 Key: HADOOP-7389
                 URL: https://issues.apache.org/jira/browse/HADOOP-7389
             Project: Hadoop Common
          Issue Type: Bug
          Components: test
    Affects Versions: 0.23.0
            Reporter: Aaron T. Myers
            Assignee: Aaron T. Myers
             Fix For: 0.23.0


As mentioned in HADOOP-6671, {{UserGroupInformation.createUserForTesting(...)}} manipulates static state which can cause test cases which are run after a call to this function to fail.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16276-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 01:09:08 2011
Return-Path: <common-issues-return-16276-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CE7216D13
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 01:09:08 +0000 (UTC)
Received: (qmail 13958 invoked by uid 500); 14 Jun 2011 01:09:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13880 invoked by uid 500); 14 Jun 2011 01:09:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13872 invoked by uid 99); 14 Jun 2011 01:09:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 01:09:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 01:09:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 95EFB414CF9
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 01:08:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 01:08:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <992977813.1212.1308013727610.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <272967917.1210.1308013607392.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7389) Use of TestingGroups by tests causes
 subsequent tests to fail
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7389:
-----------------------------------

    Attachment: hadoop-7389.0.patch

The only thing I changed in {{TestUserGroupInformation.java}} was to change the order of the test cases. Without the corresponding change to {{UserGroupInformation.java}}, this modified test fails.

> Use of TestingGroups by tests causes subsequent tests to fail
> -------------------------------------------------------------
>
>                 Key: HADOOP-7389
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7389
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7389.0.patch
>
>
> As mentioned in HADOOP-6671, {{UserGroupInformation.createUserForTesting(...)}} manipulates static state which can cause test cases which are run after a call to this function to fail.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16277-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 01:13:09 2011
Return-Path: <common-issues-return-16277-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4AAB26ECC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 01:13:09 +0000 (UTC)
Received: (qmail 16983 invoked by uid 500); 14 Jun 2011 01:13:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16922 invoked by uid 500); 14 Jun 2011 01:13:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16914 invoked by uid 99); 14 Jun 2011 01:13:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 01:13:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 01:13:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 02403414E2E
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 01:12:48 +0000 (UTC)
Date: Tue, 14 Jun 2011 01:12:48 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <844126269.1238.1308013968005.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048929#comment-13048929 ] 

Aaron T. Myers commented on HADOOP-6671:
----------------------------------------

bq. IMO it should be a separate JIRA

Filed and posted a patch at: https://issues.apache.org/jira/browse/HADOOP-7389

After some more thought, I ended up going a slightly different route than "create a cleanup method." Instead, a failure to find any groups for a user created by a call to {{UserGroupInformation.createUserForTesting(...)}} will result in a follow-up call to the original {{Groups}} implementation.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-6671-cross-project-HDFS.patch, HADOOP-6671-e.patch, HADOOP-6671-f.patch, HADOOP-6671-g.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout-f.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16278-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 01:17:11 2011
Return-Path: <common-issues-return-16278-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D93606605
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 01:17:10 +0000 (UTC)
Received: (qmail 21482 invoked by uid 500); 14 Jun 2011 01:17:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21445 invoked by uid 500); 14 Jun 2011 01:17:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21437 invoked by uid 99); 14 Jun 2011 01:17:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 01:17:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 01:17:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 63DB1414F44
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 01:16:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 01:16:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <798012517.1243.1308014207405.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048930#comment-13048930 ] 

Hadoop QA commented on HADOOP-6671:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482491/HADOOP-6671-g.patch
  against trunk revision 1135333.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 35 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/621//console

This message is automatically generated.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-6671-cross-project-HDFS.patch, HADOOP-6671-e.patch, HADOOP-6671-f.patch, HADOOP-6671-g.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout-f.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16279-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 01:25:10 2011
Return-Path: <common-issues-return-16279-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C98506666
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 01:25:10 +0000 (UTC)
Received: (qmail 29780 invoked by uid 500); 14 Jun 2011 01:25:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29717 invoked by uid 500); 14 Jun 2011 01:25:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29709 invoked by uid 99); 14 Jun 2011 01:25:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 01:25:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 01:25:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 57D54414160
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 01:24:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 01:24:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <356309607.1268.1308014687356.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1507121853.1613.1307925711791.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7384) Allow test-patch to be more
 flexible about patch format
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048932#comment-13048932 ] 

Hadoop QA commented on HADOOP-7384:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482315/hadoop-7384.txt
  against trunk revision 1135333.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 7 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/620//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/620//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/620//console

This message is automatically generated.

> Allow test-patch to be more flexible about patch format
> -------------------------------------------------------
>
>                 Key: HADOOP-7384
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7384
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-7384.txt
>
>
> Right now the test-patch process only accepts patches that are generated as "-p0" relative to common/, hdfs/, or mapreduce/. This has always been annoying for git users where the default patch format is -p1. It's also now annoying for SVN users who may generate a patch relative to trunk/ instead of the subproject subdirectory. We should auto-detect the correct patch level.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16280-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 01:35:09 2011
Return-Path: <common-issues-return-16280-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 01D766C89
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 01:35:09 +0000 (UTC)
Received: (qmail 37699 invoked by uid 500); 14 Jun 2011 01:35:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37628 invoked by uid 500); 14 Jun 2011 01:35:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37620 invoked by uid 99); 14 Jun 2011 01:35:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 01:35:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 01:35:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 650EC414418
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 01:34:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 01:34:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1172145388.1284.1308015287410.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048936#comment-13048936 ] 

Luke Lu commented on HADOOP-7144:
---------------------------------

bq. Should the access be per mbean property?

The initial customers for this jira are ops querying metrics mbeans, so the per mbean size was not an issue. I agree that for things like liveNodes from the NameNodeInfo mbean, returning all the attributes of the mbean is too wasteful for mere access of lighter-weight attributes in the same mbean.

Let's open a separate jira to add the get=attribute-or-pattern mechanism. 

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Robert Joseph Evans
>              Labels: jmx
>             Fix For: 0.20.205.0, 0.23.0
>
>         Attachments: HADOOP-7411-0.20.20X-V1.patch, HADOOP-7411-0.20.20X-V2.patch, HADOOP-7411-trunk-V1.patch, HADOOP-7411-trunk-V2.patch, HADOOP-7411-trunk-alpha.patch, jmx.json
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16281-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 01:41:08 2011
Return-Path: <common-issues-return-16281-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C2FDE6DEB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 01:41:08 +0000 (UTC)
Received: (qmail 43510 invoked by uid 500); 14 Jun 2011 01:41:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43479 invoked by uid 500); 14 Jun 2011 01:41:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43471 invoked by uid 99); 14 Jun 2011 01:41:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 01:41:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 01:41:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9C47D414671
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 01:40:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 01:40:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2040224409.1310.1308015647637.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <272967917.1210.1308013607392.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7389) Use of TestingGroups by tests
 causes subsequent tests to fail
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048941#comment-13048941 ] 

Todd Lipcon commented on HADOOP-7389:
-------------------------------------

I don't think we need the following code anymore:
{code}
       if (result == null) {
         result = new ArrayList<String>();
       }
{code}
right?

Also, any reason that you've relocated the testLogin method?

> Use of TestingGroups by tests causes subsequent tests to fail
> -------------------------------------------------------------
>
>                 Key: HADOOP-7389
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7389
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7389.0.patch
>
>
> As mentioned in HADOOP-6671, {{UserGroupInformation.createUserForTesting(...)}} manipulates static state which can cause test cases which are run after a call to this function to fail.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16282-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 01:41:10 2011
Return-Path: <common-issues-return-16282-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C11E86DFB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 01:41:10 +0000 (UTC)
Received: (qmail 43768 invoked by uid 500); 14 Jun 2011 01:41:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43737 invoked by uid 500); 14 Jun 2011 01:41:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43729 invoked by uid 99); 14 Jun 2011 01:41:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 01:41:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 01:41:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8F4CA41466F
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 01:40:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 01:40:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1572900494.1308.1308015647583.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <272967917.1210.1308013607392.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7389) Use of TestingGroups by tests causes
 subsequent tests to fail
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7389:
--------------------------------

    Status: Patch Available  (was: Open)

> Use of TestingGroups by tests causes subsequent tests to fail
> -------------------------------------------------------------
>
>                 Key: HADOOP-7389
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7389
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7389.0.patch
>
>
> As mentioned in HADOOP-6671, {{UserGroupInformation.createUserForTesting(...)}} manipulates static state which can cause test cases which are run after a call to this function to fail.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16283-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 01:47:10 2011
Return-Path: <common-issues-return-16283-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8928A6D6D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 01:47:10 +0000 (UTC)
Received: (qmail 47262 invoked by uid 500); 14 Jun 2011 01:47:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47237 invoked by uid 500); 14 Jun 2011 01:47:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47229 invoked by uid 99); 14 Jun 2011 01:47:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 01:47:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 01:47:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5BAFA41481D
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 01:46:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 01:46:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1124494804.1316.1308016007372.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <272967917.1210.1308013607392.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7389) Use of TestingGroups by tests
 causes subsequent tests to fail
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048943#comment-13048943 ] 

Todd Lipcon commented on HADOOP-7389:
-------------------------------------

oh, now I understand: reordering the tests makes it fail. Then, fixing UGI makes it pass again.

> Use of TestingGroups by tests causes subsequent tests to fail
> -------------------------------------------------------------
>
>                 Key: HADOOP-7389
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7389
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7389.0.patch
>
>
> As mentioned in HADOOP-6671, {{UserGroupInformation.createUserForTesting(...)}} manipulates static state which can cause test cases which are run after a call to this function to fail.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16284-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 01:51:08 2011
Return-Path: <common-issues-return-16284-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8BD97641E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 01:51:08 +0000 (UTC)
Received: (qmail 49485 invoked by uid 500); 14 Jun 2011 01:51:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49459 invoked by uid 500); 14 Jun 2011 01:51:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49451 invoked by uid 99); 14 Jun 2011 01:51:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 01:51:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 01:51:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 61DE14148EF
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 01:50:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 01:50:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <323064739.1322.1308016247397.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <272967917.1210.1308013607392.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7389) Use of TestingGroups by tests
 causes subsequent tests to fail
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048944#comment-13048944 ] 

Aaron T. Myers commented on HADOOP-7389:
----------------------------------------

bq. I don't think we need the following code anymore:

Perhaps, but only incidentally so because the underlying implementations of the {{Groups}} interface happen to do this themselves.

I can remove it if you want, but it seems like leaving it in will only potentially be helpful.

> Use of TestingGroups by tests causes subsequent tests to fail
> -------------------------------------------------------------
>
>                 Key: HADOOP-7389
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7389
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7389.0.patch
>
>
> As mentioned in HADOOP-6671, {{UserGroupInformation.createUserForTesting(...)}} manipulates static state which can cause test cases which are run after a call to this function to fail.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16285-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 02:03:11 2011
Return-Path: <common-issues-return-16285-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0AA5D6578
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 02:03:11 +0000 (UTC)
Received: (qmail 56867 invoked by uid 500); 14 Jun 2011 02:03:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56650 invoked by uid 500); 14 Jun 2011 02:03:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56642 invoked by uid 99); 14 Jun 2011 02:03:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 02:03:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 02:03:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 617BA414C9E
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 02:02:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 02:02:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2074748358.1340.1308016967396.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <272967917.1210.1308013607392.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7389) Use of TestingGroups by tests
 causes subsequent tests to fail
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048948#comment-13048948 ] 

Todd Lipcon commented on HADOOP-7389:
-------------------------------------

well, if the underlying implementation returned null, then it would be a bug, right? In which case this behavior would be masking the bug in the underlying implementation.

> Use of TestingGroups by tests causes subsequent tests to fail
> -------------------------------------------------------------
>
>                 Key: HADOOP-7389
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7389
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7389.0.patch
>
>
> As mentioned in HADOOP-6671, {{UserGroupInformation.createUserForTesting(...)}} manipulates static state which can cause test cases which are run after a call to this function to fail.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16286-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 02:05:08 2011
Return-Path: <common-issues-return-16286-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B99E16CF4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 02:05:08 +0000 (UTC)
Received: (qmail 58660 invoked by uid 500); 14 Jun 2011 02:05:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58630 invoked by uid 500); 14 Jun 2011 02:05:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58622 invoked by uid 99); 14 Jun 2011 02:05:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 02:05:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 02:05:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 76329414DDC
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 02:04:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 02:04:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1420756372.1357.1308017087480.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <272967917.1210.1308013607392.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7389) Use of TestingGroups by tests
 causes subsequent tests to fail
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048950#comment-13048950 ] 

Hadoop QA commented on HADOOP-7389:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482503/hadoop-7389.0.patch
  against trunk revision 1135333.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/622//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/622//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/622//console

This message is automatically generated.

> Use of TestingGroups by tests causes subsequent tests to fail
> -------------------------------------------------------------
>
>                 Key: HADOOP-7389
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7389
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7389.0.patch
>
>
> As mentioned in HADOOP-6671, {{UserGroupInformation.createUserForTesting(...)}} manipulates static state which can cause test cases which are run after a call to this function to fail.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16287-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 03:48:17 2011
Return-Path: <common-issues-return-16287-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 16DA665A4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 03:48:17 +0000 (UTC)
Received: (qmail 32879 invoked by uid 500); 14 Jun 2011 03:48:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32819 invoked by uid 500); 14 Jun 2011 03:48:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32747 invoked by uid 99); 14 Jun 2011 03:48:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 03:48:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 03:48:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CEBC6414E6C
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 03:47:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 03:47:47 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <521842997.1544.1308023267843.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048976#comment-13048976 ] 

Tanping Wang commented on HADOOP-7144:
--------------------------------------

Yes, we want to be able to query each individual property of a mbean using JMXJsonServerlet.   Something like 

http://hostname/jmx?qry=mbeanName:property

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Robert Joseph Evans
>              Labels: jmx
>             Fix For: 0.20.205.0, 0.23.0
>
>         Attachments: HADOOP-7411-0.20.20X-V1.patch, HADOOP-7411-0.20.20X-V2.patch, HADOOP-7411-trunk-V1.patch, HADOOP-7411-trunk-V2.patch, HADOOP-7411-trunk-alpha.patch, jmx.json
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16288-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 04:08:12 2011
Return-Path: <common-issues-return-16288-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3666E6F93
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 04:08:12 +0000 (UTC)
Received: (qmail 50760 invoked by uid 500); 14 Jun 2011 04:08:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50715 invoked by uid 500); 14 Jun 2011 04:08:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50706 invoked by uid 99); 14 Jun 2011 04:08:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 04:08:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 04:08:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 55B4D414811
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 04:07:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 04:07:47 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1919599244.1598.1308024467347.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048978#comment-13048978 ] 

Tanping Wang commented on HADOOP-7144:
--------------------------------------

Suppose we can query each property of a mbean, can we return the JSON string exactly like what the mbean property function returns?  This way it is consistent between querying through JMXJsonServlet and JMX itself.  No additional parsing will be needed between both situations, it will just be a plug-and-play when switching back and forth.

For example,
{code}
	NameNode#getLiveNodes() 
{code}
returns a Json string like, 
{noformat}
"LiveNodes" : "{\"l.yahoo.com\":{\"usedSpace\":49152,\"lastContact\":2,\"adminState\":\"In Service\"},\"4.yahoo.com\":{\"usedSpace\":49152,\"lastContact\":0,\"adminState\":\"Decommissioned\"}}"
{noformat}

It would be helpful if the exact Json string can be returned when querying via JMXJsonServlet.  If possible, can we eliminate the bean name and modelerType from the returned string like how JMXJsonServerlet is currently returning, i.e.

{noformat}
{
  "beans" : [ {
    "name" : "Hadoop:service=NameNode,name=NameNodeInfo",
    "modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
    "Threads" : 34,
....
{noformat}


> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Robert Joseph Evans
>              Labels: jmx
>             Fix For: 0.20.205.0, 0.23.0
>
>         Attachments: HADOOP-7411-0.20.20X-V1.patch, HADOOP-7411-0.20.20X-V2.patch, HADOOP-7411-trunk-V1.patch, HADOOP-7411-trunk-V2.patch, HADOOP-7411-trunk-alpha.patch, jmx.json
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16289-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 05:12:13 2011
Return-Path: <common-issues-return-16289-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 23A936C3C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 05:12:13 +0000 (UTC)
Received: (qmail 85925 invoked by uid 500); 14 Jun 2011 05:12:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85816 invoked by uid 500); 14 Jun 2011 05:12:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85782 invoked by uid 99); 14 Jun 2011 05:12:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 05:12:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 05:12:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4C7F24146B0
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 05:11:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 05:11:47 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2138740653.1648.1308028307310.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1415670632.7518.1307638679263.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7371) Improve tarball distributions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-7371:
------------------------------

    Status: Patch Available  (was: Open)

> Improve tarball distributions
> -----------------------------
>
>                 Key: HADOOP-7371
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7371
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>         Environment: Java 6, Redhat 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>             Fix For: 0.20.205.0
>
>         Attachments: HADOOP-7371.patch
>
>
> Hadoop release tarball contains both raw source and binary.  This leads users to use the release tarball as base for applying patches, to build custom Hadoop.  This is not the recommended method to develop hadoop because it leads to mixed development system where processed files and raw source are hard to separate.  
> To correct the problematic usage of the release tarball, the release build target should be defined as:
> "ant source" generates source release tarball.
> "ant binary" is binary release without source/javadoc jar files.
> "ant tar" is a mirror of binary release with source/javadoc jar files.
> Does this sound reasonable?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16290-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 05:14:09 2011
Return-Path: <common-issues-return-16290-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 55B206C4F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 05:14:09 +0000 (UTC)
Received: (qmail 86345 invoked by uid 500); 14 Jun 2011 05:14:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86301 invoked by uid 500); 14 Jun 2011 05:14:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86289 invoked by uid 99); 14 Jun 2011 05:14:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 05:14:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 05:14:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7EBBC4147D9
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 05:13:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 05:13:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <723180266.1663.1308028427515.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1415670632.7518.1307638679263.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7371) Improve tarball distributions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048980#comment-13048980 ] 

Hadoop QA commented on HADOOP-7371:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482496/HADOOP-7371.patch
  against trunk revision 1135333.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/623//console

This message is automatically generated.

> Improve tarball distributions
> -----------------------------
>
>                 Key: HADOOP-7371
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7371
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>         Environment: Java 6, Redhat 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>             Fix For: 0.20.205.0
>
>         Attachments: HADOOP-7371.patch
>
>
> Hadoop release tarball contains both raw source and binary.  This leads users to use the release tarball as base for applying patches, to build custom Hadoop.  This is not the recommended method to develop hadoop because it leads to mixed development system where processed files and raw source are hard to separate.  
> To correct the problematic usage of the release tarball, the release build target should be defined as:
> "ant source" generates source release tarball.
> "ant binary" is binary release without source/javadoc jar files.
> "ant tar" is a mirror of binary release with source/javadoc jar files.
> Does this sound reasonable?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16291-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 06:41:19 2011
Return-Path: <common-issues-return-16291-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C20C1651F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 06:41:19 +0000 (UTC)
Received: (qmail 42080 invoked by uid 500); 14 Jun 2011 06:41:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41611 invoked by uid 500); 14 Jun 2011 06:41:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41461 invoked by uid 99); 14 Jun 2011 06:41:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 06:41:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 06:41:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A007D4147E2
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 06:40:49 +0000 (UTC)
Date: Tue, 14 Jun 2011 06:40:49 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1762212382.1809.1308033649652.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <272967917.1210.1308013607392.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7389) Use of TestingGroups by tests causes
 subsequent tests to fail
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7389:
-----------------------------------

    Attachment: hadoop-7389.1.patch

Updated patch addressing Todd's comments.

> Use of TestingGroups by tests causes subsequent tests to fail
> -------------------------------------------------------------
>
>                 Key: HADOOP-7389
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7389
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7389.0.patch, hadoop-7389.1.patch
>
>
> As mentioned in HADOOP-6671, {{UserGroupInformation.createUserForTesting(...)}} manipulates static state which can cause test cases which are run after a call to this function to fail.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16292-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 06:55:11 2011
Return-Path: <common-issues-return-16292-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 133876BCD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 06:55:11 +0000 (UTC)
Received: (qmail 54259 invoked by uid 500); 14 Jun 2011 06:55:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54031 invoked by uid 500); 14 Jun 2011 06:55:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54015 invoked by uid 99); 14 Jun 2011 06:55:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 06:55:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 06:55:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C0E02414DAF
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 06:54:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 06:54:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <307910169.1853.1308034487786.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <272967917.1210.1308013607392.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7389) Use of TestingGroups by tests
 causes subsequent tests to fail
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049015#comment-13049015 ] 

Hadoop QA commented on HADOOP-7389:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482516/hadoop-7389.1.patch
  against trunk revision 1135333.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/624//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/624//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/624//console

This message is automatically generated.

> Use of TestingGroups by tests causes subsequent tests to fail
> -------------------------------------------------------------
>
>                 Key: HADOOP-7389
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7389
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7389.0.patch, hadoop-7389.1.patch
>
>
> As mentioned in HADOOP-6671, {{UserGroupInformation.createUserForTesting(...)}} manipulates static state which can cause test cases which are run after a call to this function to fail.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16293-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 08:09:12 2011
Return-Path: <common-issues-return-16293-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 90EBB6671
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 08:09:12 +0000 (UTC)
Received: (qmail 50934 invoked by uid 500); 14 Jun 2011 08:09:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50891 invoked by uid 500); 14 Jun 2011 08:09:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50712 invoked by uid 99); 14 Jun 2011 08:09:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 08:09:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 08:09:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7F92D4153FA
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 08:08:48 +0000 (UTC)
Date: Tue, 14 Jun 2011 08:08:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <174858604.2031.1308038928519.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7106) Re-organize hadoop subversion
 layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049051#comment-13049051 ] 

Hudson commented on HADOOP-7106:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk-Commit #726 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/726/])
    

> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, gitk-example.png, mailer-conf.diff
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16294-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 09:03:10 2011
Return-Path: <common-issues-return-16294-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7E6676F13
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 09:03:10 +0000 (UTC)
Received: (qmail 9526 invoked by uid 500); 14 Jun 2011 09:03:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9494 invoked by uid 500); 14 Jun 2011 09:03:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9486 invoked by uid 99); 14 Jun 2011 09:03:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 09:03:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 09:03:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 31EC8415DD2
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 09:02:49 +0000 (UTC)
Date: Tue, 14 Jun 2011 09:02:49 +0000 (UTC)
From: "Bochun Bai (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <121270719.2122.1308042169200.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6424) Port FsShell to FileContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049067#comment-13049067 ] 

Bochun Bai commented on HADOOP-6424:
------------------------------------

I suggest close this issue with only FsShell.java changes. And issues blocked by this(HADOOP-6732) can start.

Like HDFS-1788, we should create a serial of issues:
  FsShell ln: new command line to create.
  FsShell cp/mv/rm/cat/tail/stat: should support symlink as src and target.
  FsShell du: should correct size of symlink
  FsShell get/put: should support symlink.

Is that ok?

> Port FsShell to FileContext 
> ----------------------------
>
>                 Key: HADOOP-6424
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6424
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: John George
>         Attachments: HADOOP-6424.patch, HADOOP-6424.patch
>
>
> FsShell currently uses FileSystem, needs to be moved over to FileContext.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16295-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 14:46:08 2011
Return-Path: <common-issues-return-16295-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D72AB6922
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 14:46:08 +0000 (UTC)
Received: (qmail 38488 invoked by uid 500); 14 Jun 2011 14:46:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38456 invoked by uid 500); 14 Jun 2011 14:46:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38447 invoked by uid 99); 14 Jun 2011 14:46:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 14:46:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 14:46:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7CFFF415587
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 14:45:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 14:45:47 +0000 (UTC)
From: "Thomas Graves (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1481744229.2736.1308062747508.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7390) VersionInfo not generated properly
 in git after unsplit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

VersionInfo not generated properly in git after unsplit
-------------------------------------------------------

                 Key: HADOOP-7390
                 URL: https://issues.apache.org/jira/browse/HADOOP-7390
             Project: Hadoop Common
          Issue Type: Bug
          Components: build
            Reporter: Thomas Graves
            Priority: Minor


The version information generated during the build of common when running from git has revision and branch Unknown. I believe this started after the unsplit:

@HadoopVersionAnnotation(version="0.22.0-SNAPSHOT", revision="Unknown", branch="Unknown",
                         user="tgraves", date="Tue Jun 14 13:39:10 UTC 2011", url="file:///home/tgraves/git/hadoop-common/common",
                         srcChecksum="0f78ea668971fe51e7ebf4f97f84eed2")

The ./src/saveVersion.sh script which generates the package-info.java file with the version info looks for the presence of .git directory and that is now a level up instead of in the common directory.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16296-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 14:46:09 2011
Return-Path: <common-issues-return-16296-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9CAB36948
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 14:46:09 +0000 (UTC)
Received: (qmail 38772 invoked by uid 500); 14 Jun 2011 14:46:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38734 invoked by uid 500); 14 Jun 2011 14:46:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38725 invoked by uid 99); 14 Jun 2011 14:46:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 14:46:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 14:46:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 67E5C4155BE
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 14:45:48 +0000 (UTC)
Date: Tue, 14 Jun 2011 14:45:48 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1355462639.2769.1308062748422.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6929) RPC should have a way to pass
 Security information other than protocol annotations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6929:
----------------------------------

    Status: Patch Available  (was: Open)

> RPC should have a way to pass Security information other than protocol annotations
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6929
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6929
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: Hadoop-6929_v1.patch, h-6929.patch
>
>
> Currently Hadoop RPC allows protocol annotations as the only way to pass security information. This becomes a problem if protocols are generated and not hand written. For example protocols generated via Avro and passed over Avro tunnel (AvroRpcEngine.java) can't pass the security information.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16297-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 14:46:12 2011
Return-Path: <common-issues-return-16297-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A6EB169E4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 14:46:12 +0000 (UTC)
Received: (qmail 40449 invoked by uid 500); 14 Jun 2011 14:46:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40411 invoked by uid 500); 14 Jun 2011 14:46:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40375 invoked by uid 99); 14 Jun 2011 14:46:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 14:46:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 14:46:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F0EBE4155AC
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 14:45:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 14:45:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <35576490.2753.1308062747983.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6929) RPC should have a way to pass
 Security information other than protocol annotations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6929:
----------------------------------

    Attachment: h-6929.patch

This patch replaces the configured security info classes with security info providers that are listed in the jar.

> RPC should have a way to pass Security information other than protocol annotations
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6929
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6929
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: Hadoop-6929_v1.patch, h-6929.patch
>
>
> Currently Hadoop RPC allows protocol annotations as the only way to pass security information. This becomes a problem if protocols are generated and not hand written. For example protocols generated via Avro and passed over Avro tunnel (AvroRpcEngine.java) can't pass the security information.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16298-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 14:52:14 2011
Return-Path: <common-issues-return-16298-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B80246858
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 14:52:14 +0000 (UTC)
Received: (qmail 54707 invoked by uid 500); 14 Jun 2011 14:52:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54673 invoked by uid 500); 14 Jun 2011 14:52:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54665 invoked by uid 99); 14 Jun 2011 14:52:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 14:52:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 14:52:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1076A415AA1
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 14:51:50 +0000 (UTC)
Date: Tue, 14 Jun 2011 14:51:50 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1343713583.2796.1308063110063.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1481744229.2736.1308062747508.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7390) VersionInfo not generated properly
 in git after unsplit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon reassigned HADOOP-7390:
-----------------------------------

    Assignee: Todd Lipcon

> VersionInfo not generated properly in git after unsplit
> -------------------------------------------------------
>
>                 Key: HADOOP-7390
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7390
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Thomas Graves
>            Assignee: Todd Lipcon
>            Priority: Minor
>
> The version information generated during the build of common when running from git has revision and branch Unknown. I believe this started after the unsplit:
> @HadoopVersionAnnotation(version="0.22.0-SNAPSHOT", revision="Unknown", branch="Unknown",
>                          user="tgraves", date="Tue Jun 14 13:39:10 UTC 2011", url="file:///home/tgraves/git/hadoop-common/common",
>                          srcChecksum="0f78ea668971fe51e7ebf4f97f84eed2")
> The ./src/saveVersion.sh script which generates the package-info.java file with the version info looks for the presence of .git directory and that is now a level up instead of in the common directory.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16299-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 14:54:09 2011
Return-Path: <common-issues-return-16299-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 82EE569A7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 14:54:09 +0000 (UTC)
Received: (qmail 59130 invoked by uid 500); 14 Jun 2011 14:54:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59096 invoked by uid 500); 14 Jun 2011 14:54:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59088 invoked by uid 99); 14 Jun 2011 14:54:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 14:54:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 14:54:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 33213415CD1
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 14:53:48 +0000 (UTC)
Date: Tue, 14 Jun 2011 14:53:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <561990583.2807.1308063228206.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1481744229.2736.1308062747508.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7390) VersionInfo not generated properly
 in git after unsplit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7390:
--------------------------------

    Status: Patch Available  (was: Open)

> VersionInfo not generated properly in git after unsplit
> -------------------------------------------------------
>
>                 Key: HADOOP-7390
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7390
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Thomas Graves
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7390.txt
>
>
> The version information generated during the build of common when running from git has revision and branch Unknown. I believe this started after the unsplit:
> @HadoopVersionAnnotation(version="0.22.0-SNAPSHOT", revision="Unknown", branch="Unknown",
>                          user="tgraves", date="Tue Jun 14 13:39:10 UTC 2011", url="file:///home/tgraves/git/hadoop-common/common",
>                          srcChecksum="0f78ea668971fe51e7ebf4f97f84eed2")
> The ./src/saveVersion.sh script which generates the package-info.java file with the version info looks for the presence of .git directory and that is now a level up instead of in the common directory.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16300-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 14:54:11 2011
Return-Path: <common-issues-return-16300-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5D12169C9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 14:54:11 +0000 (UTC)
Received: (qmail 59771 invoked by uid 500); 14 Jun 2011 14:54:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59731 invoked by uid 500); 14 Jun 2011 14:54:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59723 invoked by uid 99); 14 Jun 2011 14:54:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 14:54:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 14:54:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1FD38415CCE
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 14:53:48 +0000 (UTC)
Date: Tue, 14 Jun 2011 14:53:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1510299179.2804.1308063228127.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1481744229.2736.1308062747508.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7390) VersionInfo not generated properly
 in git after unsplit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7390:
--------------------------------

    Attachment: hadoop-7390.txt

> VersionInfo not generated properly in git after unsplit
> -------------------------------------------------------
>
>                 Key: HADOOP-7390
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7390
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Thomas Graves
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7390.txt
>
>
> The version information generated during the build of common when running from git has revision and branch Unknown. I believe this started after the unsplit:
> @HadoopVersionAnnotation(version="0.22.0-SNAPSHOT", revision="Unknown", branch="Unknown",
>                          user="tgraves", date="Tue Jun 14 13:39:10 UTC 2011", url="file:///home/tgraves/git/hadoop-common/common",
>                          srcChecksum="0f78ea668971fe51e7ebf4f97f84eed2")
> The ./src/saveVersion.sh script which generates the package-info.java file with the version info looks for the presence of .git directory and that is now a level up instead of in the common directory.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16301-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 14:56:11 2011
Return-Path: <common-issues-return-16301-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 686946A33
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 14:56:11 +0000 (UTC)
Received: (qmail 63539 invoked by uid 500); 14 Jun 2011 14:56:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63504 invoked by uid 500); 14 Jun 2011 14:56:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63496 invoked by uid 99); 14 Jun 2011 14:56:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 14:56:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 14:56:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2B0E1415E4F
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 14:55:48 +0000 (UTC)
Date: Tue, 14 Jun 2011 14:55:48 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1790712289.2830.1308063348173.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6929) RPC should have a way to pass
 Security information other than protocol annotations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049215#comment-13049215 ] 

Hadoop QA commented on HADOOP-6929:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482554/h-6929.patch
  against trunk revision 1135333.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/625//console

This message is automatically generated.

> RPC should have a way to pass Security information other than protocol annotations
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6929
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6929
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: Hadoop-6929_v1.patch, h-6929.patch
>
>
> Currently Hadoop RPC allows protocol annotations as the only way to pass security information. This becomes a problem if protocols are generated and not hand written. For example protocols generated via Avro and passed over Avro tunnel (AvroRpcEngine.java) can't pass the security information.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16302-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 15:04:09 2011
Return-Path: <common-issues-return-16302-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EB4656FEA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 15:04:08 +0000 (UTC)
Received: (qmail 79000 invoked by uid 500); 14 Jun 2011 15:04:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78957 invoked by uid 500); 14 Jun 2011 15:04:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78949 invoked by uid 99); 14 Jun 2011 15:04:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 15:04:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 15:04:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A61A041531C
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 15:03:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 15:03:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1685230701.2853.1308063827677.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1481744229.2736.1308062747508.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7390) VersionInfo not generated properly
 in git after unsplit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049219#comment-13049219 ] 

Hadoop QA commented on HADOOP-7390:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482556/hadoop-7390.txt
  against trunk revision 1135333.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/626//console

This message is automatically generated.

> VersionInfo not generated properly in git after unsplit
> -------------------------------------------------------
>
>                 Key: HADOOP-7390
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7390
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Thomas Graves
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7390.txt
>
>
> The version information generated during the build of common when running from git has revision and branch Unknown. I believe this started after the unsplit:
> @HadoopVersionAnnotation(version="0.22.0-SNAPSHOT", revision="Unknown", branch="Unknown",
>                          user="tgraves", date="Tue Jun 14 13:39:10 UTC 2011", url="file:///home/tgraves/git/hadoop-common/common",
>                          srcChecksum="0f78ea668971fe51e7ebf4f97f84eed2")
> The ./src/saveVersion.sh script which generates the package-info.java file with the version info looks for the presence of .git directory and that is now a level up instead of in the common directory.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16303-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 15:08:11 2011
Return-Path: <common-issues-return-16303-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 78EE363D3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 15:08:11 +0000 (UTC)
Received: (qmail 87384 invoked by uid 500); 14 Jun 2011 15:08:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87331 invoked by uid 500); 14 Jun 2011 15:08:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87262 invoked by uid 99); 14 Jun 2011 15:08:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 15:08:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 15:08:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E9DFD415729
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 15:07:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 15:07:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2053616919.2891.1308064067954.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1507121853.1613.1307925711791.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7384) Allow test-patch to be more flexible
 about patch format
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7384:
--------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

Committed to trunk

> Allow test-patch to be more flexible about patch format
> -------------------------------------------------------
>
>                 Key: HADOOP-7384
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7384
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7384.txt
>
>
> Right now the test-patch process only accepts patches that are generated as "-p0" relative to common/, hdfs/, or mapreduce/. This has always been annoying for git users where the default patch format is -p1. It's also now annoying for SVN users who may generate a patch relative to trunk/ instead of the subproject subdirectory. We should auto-detect the correct patch level.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16304-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 15:08:11 2011
Return-Path: <common-issues-return-16304-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A788463E1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 15:08:11 +0000 (UTC)
Received: (qmail 87541 invoked by uid 500); 14 Jun 2011 15:08:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87488 invoked by uid 500); 14 Jun 2011 15:08:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87379 invoked by uid 99); 14 Jun 2011 15:08:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 15:08:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 15:08:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CE685415725
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 15:07:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 15:07:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <33699770.2887.1308064067842.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6929) RPC should have a way to pass
 Security information other than protocol annotations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049225#comment-13049225 ] 

Todd Lipcon commented on HADOOP-6929:
-------------------------------------

I just committed HADOOP-7384 which should enable this patch to work properly. I'll re-trigger the build.

> RPC should have a way to pass Security information other than protocol annotations
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6929
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6929
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: Hadoop-6929_v1.patch, h-6929.patch
>
>
> Currently Hadoop RPC allows protocol annotations as the only way to pass security information. This becomes a problem if protocols are generated and not hand written. For example protocols generated via Avro and passed over Avro tunnel (AvroRpcEngine.java) can't pass the security information.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16305-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 15:10:09 2011
Return-Path: <common-issues-return-16305-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6F0386444
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 15:10:09 +0000 (UTC)
Received: (qmail 89168 invoked by uid 500); 14 Jun 2011 15:10:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89047 invoked by uid 500); 14 Jun 2011 15:10:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89038 invoked by uid 99); 14 Jun 2011 15:10:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 15:10:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 15:10:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1A3F54158B1
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 15:09:48 +0000 (UTC)
Date: Tue, 14 Jun 2011 15:09:48 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <577319932.2911.1308064188103.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6929) RPC should have a way to pass
 Security information other than protocol annotations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049229#comment-13049229 ] 

Hadoop QA commented on HADOOP-6929:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482554/h-6929.patch
  against trunk revision 1135633.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/627//console

This message is automatically generated.

> RPC should have a way to pass Security information other than protocol annotations
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6929
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6929
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: Hadoop-6929_v1.patch, h-6929.patch
>
>
> Currently Hadoop RPC allows protocol annotations as the only way to pass security information. This becomes a problem if protocols are generated and not hand written. For example protocols generated via Avro and passed over Avro tunnel (AvroRpcEngine.java) can't pass the security information.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16306-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 15:10:11 2011
Return-Path: <common-issues-return-16306-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C57666468
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 15:10:11 +0000 (UTC)
Received: (qmail 89627 invoked by uid 500); 14 Jun 2011 15:10:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89571 invoked by uid 500); 14 Jun 2011 15:10:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89563 invoked by uid 99); 14 Jun 2011 15:10:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 15:10:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 15:10:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 27F554158B3
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 15:09:48 +0000 (UTC)
Date: Tue, 14 Jun 2011 15:09:48 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1946142326.2913.1308064188160.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1481744229.2736.1308062747508.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7390) VersionInfo not generated properly
 in git after unsplit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049230#comment-13049230 ] 

Hadoop QA commented on HADOOP-7390:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482556/hadoop-7390.txt
  against trunk revision 1135633.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/628//console

This message is automatically generated.

> VersionInfo not generated properly in git after unsplit
> -------------------------------------------------------
>
>                 Key: HADOOP-7390
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7390
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Thomas Graves
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7390.txt
>
>
> The version information generated during the build of common when running from git has revision and branch Unknown. I believe this started after the unsplit:
> @HadoopVersionAnnotation(version="0.22.0-SNAPSHOT", revision="Unknown", branch="Unknown",
>                          user="tgraves", date="Tue Jun 14 13:39:10 UTC 2011", url="file:///home/tgraves/git/hadoop-common/common",
>                          srcChecksum="0f78ea668971fe51e7ebf4f97f84eed2")
> The ./src/saveVersion.sh script which generates the package-info.java file with the version info looks for the presence of .git directory and that is now a level up instead of in the common directory.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16307-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 15:16:15 2011
Return-Path: <common-issues-return-16307-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DC9446B75
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 15:16:14 +0000 (UTC)
Received: (qmail 7039 invoked by uid 500); 14 Jun 2011 15:16:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6996 invoked by uid 500); 14 Jun 2011 15:16:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6978 invoked by uid 99); 14 Jun 2011 15:16:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 15:16:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 15:16:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3CBE0415C0B
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 15:15:50 +0000 (UTC)
Date: Tue, 14 Jun 2011 15:15:50 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <165844.2935.1308064550245.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6929) RPC should have a way to pass
 Security information other than protocol annotations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049232#comment-13049232 ] 

Hadoop QA commented on HADOOP-6929:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482554/h-6929.patch
  against trunk revision 1135636.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The patch appears to cause tar ant target to fail.

    -1 findbugs.  The patch appears to cause Findbugs (version 1.3.9) to fail.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:


    -1 system test framework.  The patch failed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/629//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/629//console

This message is automatically generated.

> RPC should have a way to pass Security information other than protocol annotations
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6929
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6929
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: Hadoop-6929_v1.patch, h-6929.patch
>
>
> Currently Hadoop RPC allows protocol annotations as the only way to pass security information. This becomes a problem if protocols are generated and not hand written. For example protocols generated via Avro and passed over Avro tunnel (AvroRpcEngine.java) can't pass the security information.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16308-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 15:28:15 2011
Return-Path: <common-issues-return-16308-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0C71B6327
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 15:28:15 +0000 (UTC)
Received: (qmail 31410 invoked by uid 500); 14 Jun 2011 15:28:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31360 invoked by uid 500); 14 Jun 2011 15:28:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31266 invoked by uid 99); 14 Jun 2011 15:28:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 15:28:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 15:28:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D83D54160CE
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 15:27:48 +0000 (UTC)
Date: Tue, 14 Jun 2011 15:27:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <150410867.2977.1308065268882.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1507121853.1613.1307925711791.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7384) Allow test-patch to be more
 flexible about patch format
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049236#comment-13049236 ] 

Hudson commented on HADOOP-7384:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #653 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/653/])
    Amend commit of HADOOP-7384 to include svn:executable bit on smart-apply-patch.sh
HADOOP-7384. Allow test-patch to be more flexible about patch format. Contributed by Todd Lipcon.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1135636
Files : 
* /hadoop/common/trunk/common/src/test/bin/smart-apply-patch.sh

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1135633
Files : 
* /hadoop/common/trunk/common/CHANGES.txt
* /hadoop/common/trunk/common/src/test/bin/smart-apply-patch.sh
* /hadoop/common/trunk/common/src/test/bin/test-patch.sh


> Allow test-patch to be more flexible about patch format
> -------------------------------------------------------
>
>                 Key: HADOOP-7384
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7384
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7384.txt
>
>
> Right now the test-patch process only accepts patches that are generated as "-p0" relative to common/, hdfs/, or mapreduce/. This has always been annoying for git users where the default patch format is -p1. It's also now annoying for SVN users who may generate a patch relative to trunk/ instead of the subproject subdirectory. We should auto-detect the correct patch level.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16309-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 16:50:08 2011
Return-Path: <common-issues-return-16309-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C98B66543
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 16:50:08 +0000 (UTC)
Received: (qmail 94548 invoked by uid 500); 14 Jun 2011 16:50:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94520 invoked by uid 500); 14 Jun 2011 16:50:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94512 invoked by uid 99); 14 Jun 2011 16:50:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 16:50:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 16:50:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7932E41696D
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 16:49:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 16:49:47 +0000 (UTC)
From: "Joep Rottinghuis (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1758858078.3231.1308070187493.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1009900334.382.1308003947757.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7387) Change rpm to ignore configuration
 files when uninstalling
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049264#comment-13049264 ] 

Joep Rottinghuis commented on HADOOP-7387:
------------------------------------------

If I read the rpm_config page you referred to correctly, that specifies the behavior when a package is updated, not what happens if it is uninstalled.
I'd say if a config file has been unchanged from the original installation then uninstall should actually delete the config file (you can always get it back by re-installing the package again).
If a config file has been modified and differs from the version in the pacakge, it could be renamed to .rpmsave.

I think it is important that uninstalling packages leaves the system as close as possible to the state it was in before the package was installed in the first place.
When you want to upgrade from one version to the next you really do not want any artifacts to linger around and mess up things in an unexpected way.

Specifically:
Install version A, uninstall version A, install version B
should be the same as:
Install version B

> Change rpm to ignore configuration files when uninstalling
> ----------------------------------------------------------
>
>                 Key: HADOOP-7387
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7387
>             Project: Hadoop Common
>          Issue Type: Improvement
>         Environment: Java 6, RHEL 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>            Priority: Minor
>
> When uninstalling RPM files, the config files are renamed from core-site.xml to core-site.xml.rpmsave.  It would be nice if config file does not get renamed.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16310-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 17:02:09 2011
Return-Path: <common-issues-return-16310-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BDC726350
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 17:02:09 +0000 (UTC)
Received: (qmail 15867 invoked by uid 500); 14 Jun 2011 17:02:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15834 invoked by uid 500); 14 Jun 2011 17:02:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15826 invoked by uid 99); 14 Jun 2011 17:02:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 17:02:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 17:02:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6F10C416D9E
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 17:01:48 +0000 (UTC)
Date: Tue, 14 Jun 2011 17:01:48 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1725127926.3265.1308070908451.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6929) RPC should have a way to pass
 Security information other than protocol annotations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6929:
----------------------------------

    Attachment: h-6929.patch

I forgot the two new java classes.

> RPC should have a way to pass Security information other than protocol annotations
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6929
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6929
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: Hadoop-6929_v1.patch, h-6929.patch, h-6929.patch
>
>
> Currently Hadoop RPC allows protocol annotations as the only way to pass security information. This becomes a problem if protocols are generated and not hand written. For example protocols generated via Avro and passed over Avro tunnel (AvroRpcEngine.java) can't pass the security information.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16311-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 17:04:11 2011
Return-Path: <common-issues-return-16311-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 50C7F6D5D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 17:04:11 +0000 (UTC)
Received: (qmail 23697 invoked by uid 500); 14 Jun 2011 17:04:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23662 invoked by uid 500); 14 Jun 2011 17:04:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23654 invoked by uid 99); 14 Jun 2011 17:04:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 17:04:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 17:04:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E0611416FE9
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 17:03:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 17:03:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1185672168.3281.1308071027915.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6929) RPC should have a way to pass
 Security information other than protocol annotations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6929:
----------------------------------

    Status: Open  (was: Patch Available)

> RPC should have a way to pass Security information other than protocol annotations
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6929
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6929
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: Hadoop-6929_v1.patch, h-6929.patch, h-6929.patch
>
>
> Currently Hadoop RPC allows protocol annotations as the only way to pass security information. This becomes a problem if protocols are generated and not hand written. For example protocols generated via Avro and passed over Avro tunnel (AvroRpcEngine.java) can't pass the security information.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16312-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 17:04:11 2011
Return-Path: <common-issues-return-16312-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C59846D75
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 17:04:11 +0000 (UTC)
Received: (qmail 23945 invoked by uid 500); 14 Jun 2011 17:04:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23913 invoked by uid 500); 14 Jun 2011 17:04:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23905 invoked by uid 99); 14 Jun 2011 17:04:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 17:04:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 17:04:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5C138416FFA
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 17:03:48 +0000 (UTC)
Date: Tue, 14 Jun 2011 17:03:48 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <581385121.3297.1308071028374.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6929) RPC should have a way to pass
 Security information other than protocol annotations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6929:
----------------------------------

    Status: Patch Available  (was: Open)

> RPC should have a way to pass Security information other than protocol annotations
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6929
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6929
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: Hadoop-6929_v1.patch, h-6929.patch, h-6929.patch
>
>
> Currently Hadoop RPC allows protocol annotations as the only way to pass security information. This becomes a problem if protocols are generated and not hand written. For example protocols generated via Avro and passed over Avro tunnel (AvroRpcEngine.java) can't pass the security information.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16313-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 17:16:10 2011
Return-Path: <common-issues-return-16313-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B6A68669D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 17:16:10 +0000 (UTC)
Received: (qmail 48163 invoked by uid 500); 14 Jun 2011 17:16:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48124 invoked by uid 500); 14 Jun 2011 17:16:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48115 invoked by uid 99); 14 Jun 2011 17:16:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 17:16:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 17:16:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 86394416595
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 17:15:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 17:15:47 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <232002188.3379.1308071747546.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur reassigned HADOOP-7206:
------------------------------------------

    Assignee: issei yoshida  (was: Alejandro Abdelnur)

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206.patch
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16314-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 17:16:11 2011
Return-Path: <common-issues-return-16314-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 35E3666BF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 17:16:11 +0000 (UTC)
Received: (qmail 48478 invoked by uid 500); 14 Jun 2011 17:16:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48375 invoked by uid 500); 14 Jun 2011 17:16:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48367 invoked by uid 99); 14 Jun 2011 17:16:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 17:16:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 17:16:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C1AFA4165E2
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 17:15:49 +0000 (UTC)
Date: Tue, 14 Jun 2011 17:15:49 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <732127666.3445.1308071749790.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6929) RPC should have a way to pass
 Security information other than protocol annotations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049284#comment-13049284 ] 

Hadoop QA commented on HADOOP-6929:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482567/h-6929.patch
  against trunk revision 1135636.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.metrics2.impl.TestMetricsConfig
                  org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/630//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/630//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/630//console

This message is automatically generated.

> RPC should have a way to pass Security information other than protocol annotations
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6929
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6929
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: Hadoop-6929_v1.patch, h-6929.patch, h-6929.patch
>
>
> Currently Hadoop RPC allows protocol annotations as the only way to pass security information. This becomes a problem if protocols are generated and not hand written. For example protocols generated via Avro and passed over Avro tunnel (AvroRpcEngine.java) can't pass the security information.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16315-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 17:28:10 2011
Return-Path: <common-issues-return-16315-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E45EF6D50
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 17:28:10 +0000 (UTC)
Received: (qmail 76469 invoked by uid 500); 14 Jun 2011 17:28:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76440 invoked by uid 500); 14 Jun 2011 17:28:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76432 invoked by uid 99); 14 Jun 2011 17:28:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 17:28:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 17:28:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 62E33416CC6
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 17:27:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 17:27:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <412587682.3482.1308072467401.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1481744229.2736.1308062747508.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7390) VersionInfo not generated properly
 in git after unsplit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049294#comment-13049294 ] 

Hadoop QA commented on HADOOP-7390:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482556/hadoop-7390.txt
  against trunk revision 1135636.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/631//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/631//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/631//console

This message is automatically generated.

> VersionInfo not generated properly in git after unsplit
> -------------------------------------------------------
>
>                 Key: HADOOP-7390
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7390
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Thomas Graves
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7390.txt
>
>
> The version information generated during the build of common when running from git has revision and branch Unknown. I believe this started after the unsplit:
> @HadoopVersionAnnotation(version="0.22.0-SNAPSHOT", revision="Unknown", branch="Unknown",
>                          user="tgraves", date="Tue Jun 14 13:39:10 UTC 2011", url="file:///home/tgraves/git/hadoop-common/common",
>                          srcChecksum="0f78ea668971fe51e7ebf4f97f84eed2")
> The ./src/saveVersion.sh script which generates the package-info.java file with the version info looks for the presence of .git directory and that is now a level up instead of in the common directory.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16316-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 17:46:10 2011
Return-Path: <common-issues-return-16316-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 918116EED
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 17:46:10 +0000 (UTC)
Received: (qmail 18792 invoked by uid 500); 14 Jun 2011 17:46:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18719 invoked by uid 500); 14 Jun 2011 17:46:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18711 invoked by uid 99); 14 Jun 2011 17:46:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 17:46:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 17:46:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5EF4E416329
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 17:45:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 17:45:47 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1854453348.3527.1308073547384.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <9085317.460.1308004667366.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7388) Remove definition of HADOOP_HOME and
 HADOOP_PREFIX from hadoop-env.sh.template
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-7388:
------------------------------

    Attachment: HADOOP-7388.patch

Removed HADOOP_HOME and HADOOP_PREFIX for 0.20-security branch.

> Remove definition of HADOOP_HOME and HADOOP_PREFIX from hadoop-env.sh.template
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7388
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7388
>             Project: Hadoop Common
>          Issue Type: Improvement
>         Environment: Java 6, RHEL 5.5
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>            Priority: Trivial
>         Attachments: HADOOP-7388.patch
>
>
> The file structure layout proposed in HADOOP-6255 was designed to remove the need of using HADOOP_HOME environment to locate hadoop bits.  The file structure layout should be able to map to /usr or system directories, therefore HADOOP_HOME is renamed to HADOOP_PREFIX to be more concise.  HADOOP_PREFIX should not be exported to the user.  If the user use hadoop-setup-single-node.sh or hadoop-setup-conf.sh to configure hadoop, the current scripts put HADOOP_PREFIX/HADOOP_HOME in hadoop-env.sh.  The config template generation code should remove reference of HADOOP_PREFIX/HADOOP_HOME from hadoop-env.sh.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16317-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 17:46:11 2011
Return-Path: <common-issues-return-16317-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DB83C6F02
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 17:46:10 +0000 (UTC)
Received: (qmail 19078 invoked by uid 500); 14 Jun 2011 17:46:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19025 invoked by uid 500); 14 Jun 2011 17:46:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18953 invoked by uid 99); 14 Jun 2011 17:46:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 17:46:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 17:46:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 86BC641632E
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 17:45:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 17:45:47 +0000 (UTC)
From: "Thomas Graves (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1614827927.3532.1308073547548.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1481744229.2736.1308062747508.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7390) VersionInfo not generated properly
 in git after unsplit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049303#comment-13049303 ] 

Thomas Graves commented on HADOOP-7390:
---------------------------------------

+1 downloaded and tested patch and it works now.  Thanks!

> VersionInfo not generated properly in git after unsplit
> -------------------------------------------------------
>
>                 Key: HADOOP-7390
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7390
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Thomas Graves
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7390.txt
>
>
> The version information generated during the build of common when running from git has revision and branch Unknown. I believe this started after the unsplit:
> @HadoopVersionAnnotation(version="0.22.0-SNAPSHOT", revision="Unknown", branch="Unknown",
>                          user="tgraves", date="Tue Jun 14 13:39:10 UTC 2011", url="file:///home/tgraves/git/hadoop-common/common",
>                          srcChecksum="0f78ea668971fe51e7ebf4f97f84eed2")
> The ./src/saveVersion.sh script which generates the package-info.java file with the version info looks for the presence of .git directory and that is now a level up instead of in the common directory.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16318-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 19:19:09 2011
Return-Path: <common-issues-return-16318-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0CF3F6D9D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 19:19:09 +0000 (UTC)
Received: (qmail 94978 invoked by uid 500); 14 Jun 2011 19:19:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94924 invoked by uid 500); 14 Jun 2011 19:19:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94912 invoked by uid 99); 14 Jun 2011 19:19:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 19:19:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 19:19:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A229A416072
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 19:18:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 19:18:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <277360745.3856.1308079127661.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <151121036.11705.1307734619237.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7377) Fix command name handling
 affecting DFSAdmin
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049351#comment-13049351 ] 

Matt Foley commented on HADOOP-7377:
------------------------------------

I'm uncomfortable using reflection with access permission override, to get a datum that seems like it could be fetched using provided methods such as getCommandName().  However, Daryn has convinced me that it would require a massive change of legacy code to make this better:

bq. [Daryn] The CommandFactory is supposed to call setName(...) on the instance it creates, using the name fed to the factory (since commands can have more than one invokable name).  setName()/getName() should be just using the instance field name.  The static NAME is legacy and was not intended to remain. But Count and the DFSAdmin commands, unfortunately, use it. I intend to kill both getCommandName() and NAME, if possible, after DFSAdmin is adapted to better use the framework. Basically, the command should not know it's name, that's the factory's job, and the factory tells the command how it was conjured up. But in the meantime, NAME is needed.

So, +1 on this patch.

Auto-test passes except for new test, but this patch will be tested by existing DFSAdmin unit tests starting to work.
I'll commit this in 24 hours to allow for other comments.

Daryn, please list the exact unit tests that will be fixed by this submission.  Thanks.


> Fix command name handling affecting DFSAdmin
> --------------------------------------------
>
>                 Key: HADOOP-7377
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7377
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7377.patch
>
>
> When an error occurs in the get/set quota commands in DFSAdmin, they are displaying the following:
> setQuota: failed to get SetQuotaCommand.NAME
> The {{Command}} class expects the {{NAME}} field to be accessible, but for DFSAdmin, it's not.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16319-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 19:51:09 2011
Return-Path: <common-issues-return-16319-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 51E2F6519
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 19:51:09 +0000 (UTC)
Received: (qmail 52247 invoked by uid 500); 14 Jun 2011 19:51:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52217 invoked by uid 500); 14 Jun 2011 19:51:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52209 invoked by uid 99); 14 Jun 2011 19:51:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 19:51:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 19:51:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 886A0416DFD
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 19:50:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 19:50:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1928332496.4012.1308081047555.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7391) Copy the interrface classification
 documentation of HADOOP-5073 into javadoc
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Copy the interrface classification documentation of HADOOP-5073 into javadoc
----------------------------------------------------------------------------

                 Key: HADOOP-7391
                 URL: https://issues.apache.org/jira/browse/HADOOP-7391
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Sanjay Radia
            Assignee: Sanjay Radia
             Fix For: 0.22.0


The documentation for interface classification in Jira Hadoop-5073 was not copied to the Javadoc
of the classification.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16320-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 20:55:14 2011
Return-Path: <common-issues-return-16320-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 66FAE664D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 20:55:14 +0000 (UTC)
Received: (qmail 76736 invoked by uid 500); 14 Jun 2011 20:55:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76702 invoked by uid 500); 14 Jun 2011 20:55:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76694 invoked by uid 99); 14 Jun 2011 20:55:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 20:55:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 20:55:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0C204416EAF
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 20:54:51 +0000 (UTC)
Date: Tue, 14 Jun 2011 20:54:51 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <491586685.4234.1308084891046.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049409#comment-13049409 ] 

Alejandro Abdelnur commented on HADOOP-7206:
--------------------------------------------

Following up on this, Issay will update his original patch that bring hadoop-snappy into hadoop-common.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206.patch
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16321-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 21:15:26 2011
Return-Path: <common-issues-return-16321-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 191D96412
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 21:15:25 +0000 (UTC)
Received: (qmail 12150 invoked by uid 500); 14 Jun 2011 21:15:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12105 invoked by uid 500); 14 Jun 2011 21:15:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12097 invoked by uid 99); 14 Jun 2011 21:15:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 21:15:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 21:15:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CF6A0416B6B
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 21:15:00 +0000 (UTC)
Date: Tue, 14 Jun 2011 21:15:00 +0000 (UTC)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1086675387.4333.1308086100846.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <389414663.13212.1307766119073.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7381) FindBugs OutOfMemoryError
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049428#comment-13049428 ] 

Konstantin Shvachko commented on HADOOP-7381:
---------------------------------------------

Let's keep description small.
This looks like a good thing to have. I'd recommend to commit this to trunk as well.

> FindBugs OutOfMemoryError
> -------------------------
>
>                 Key: HADOOP-7381
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7381
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.20.205.0
>         Environment: FindBugs 1.3.9, ant 1.8.2, RHEL6, Jenkins 1.414 in Tomcat 7.0.14, Sun Java HotSpot(TM) 64-Bit Server VM (build 14.2-b01, mixed mode)
>            Reporter: Joep Rottinghuis
>            Assignee: Joep Rottinghuis
>            Priority: Minor
>         Attachments: hadoop-7381.patch
>
>
> When running the findbugs target from Jenkins, I get an OutOfMemory error.
> The "effort" in FindBugs is set to Max which ends up using a lot of memory to go through all the classes. The jvmargs passed to FindBugs is hardcoded to 512 MB max.
> We can leave the default to 512M, as long as we pass this as an ant parameter which can be overwritten in individual cases through -D, or in the build.properties file (either basedir, or user's home directory).
> This is the error I get:
> findbugs:
>     [mkdir] Created dir: /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs
>  [findbugs] Executing findbugs from ant task
>  [findbugs] Running FindBugs...
>  [findbugs] Out of memory
>  [findbugs] Total memory: 477M
>  [findbugs]  free memory: 62M
>  [findbugs] Analyzed: /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/hadoop-core-0.20.security-test-3.jar
> ...
> [findbugs] Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
>  [findbugs] 	at edu.umd.cs.findbugs.OpcodeStack.pushBySignature(OpcodeStack.java:2589)
>  [findbugs] 	at edu.umd.cs.findbugs.OpcodeStack.pushByInvoke(OpcodeStack.java:2565)
>  [findbugs] 	at edu.umd.cs.findbugs.OpcodeStack.processMethodCall(OpcodeStack.java:2011)
>  [findbugs] 	at edu.umd.cs.findbugs.OpcodeStack.sawOpcode(OpcodeStack.java:1574)
>                   ...15 more
>  [findbugs] Java Result: 1
>  [findbugs] Output saved to /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.xml
>      [xslt] Processing /hadoop01/jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.xml to /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.html
>      [xslt] Loading stylesheet /usr/local/findbugs/src/xsl/default.xsl
>      [xslt] : Error! Premature end of file.
>      [xslt] : Error! com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
>      [xslt] Failed to process /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.xml
> BUILD FAILED
> /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build.xml:1164: javax.xml.transform.TransformerException: javax.xml.transform.TransformerException: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
> 	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:720)
> 	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:313)
> 	at org.apache.tools.ant.taskdefs.optional.TraXLiaison.transform(TraXLiaison.java:194)
> 	at org.apache.tools.ant.taskdefs.XSLTProcess.process(XSLTProcess.java:852)
> 	at org.apache.tools.ant.taskdefs.XSLTProcess.execute(XSLTProcess.java:388)
> 	at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
> 	... 15 more
> Caused by: javax.xml.transform.TransformerException: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
> 	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getDOM(TransformerImpl.java:547)
> 	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:710)
> 	... 20 more
> Caused by: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
> 	at com.sun.org.apache.xalan.internal.xsltc.dom.XSLTCDTMManager.getDTM(XSLTCDTMManager.java:446)
> 	at com.sun.org.apache.xalan.internal.xsltc.dom.XSLTCDTMManager.getDTM(XSLTCDTMManager.java:234)
> 	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getDOM(TransformerImpl.java:525)
> 	... 21 more
> ---------
> etc.etc. 
> Total time: 8 minutes 35 seconds
> Build step 'Execute shell' marked build as failure
> [CHECKSTYLE] Collecting checkstyle analysis files...
> CHECKSTYLESuccessfully parsed file /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/checkstyle-errors.xml of module  with 13525 warnings.
> [FINDBUGS] Skipping publisher since build result is FAILURE
> [WARNINGS] Skipping publisher since build result is FAILURE

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16322-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 21:19:11 2011
Return-Path: <common-issues-return-16322-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0609C64A8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 21:19:11 +0000 (UTC)
Received: (qmail 15962 invoked by uid 500); 14 Jun 2011 21:19:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15903 invoked by uid 500); 14 Jun 2011 21:19:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15885 invoked by uid 99); 14 Jun 2011 21:19:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 21:19:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 21:19:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 69861416CEF
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 21:18:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 21:18:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <192001893.4337.1308086327429.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <151121036.11705.1307734619237.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7377) Fix command name handling
 affecting DFSAdmin
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049429#comment-13049429 ] 

Daryn Sharp commented on HADOOP-7377:
-------------------------------------

This patch fixes all of the failing negative tests for {{DFSAdmin}} in {{TestHDFSCLI}}.  As of build #777, the negative {{DFSAdmin}} tests are the only tests that are failing in {{TestHDFSCLI}}.

> Fix command name handling affecting DFSAdmin
> --------------------------------------------
>
>                 Key: HADOOP-7377
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7377
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7377.patch
>
>
> When an error occurs in the get/set quota commands in DFSAdmin, they are displaying the following:
> setQuota: failed to get SetQuotaCommand.NAME
> The {{Command}} class expects the {{NAME}} field to be accessible, but for DFSAdmin, it's not.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16323-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 21:49:08 2011
Return-Path: <common-issues-return-16323-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A95796E08
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 21:49:08 +0000 (UTC)
Received: (qmail 97761 invoked by uid 500); 14 Jun 2011 21:49:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97726 invoked by uid 500); 14 Jun 2011 21:49:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97718 invoked by uid 99); 14 Jun 2011 21:49:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 21:49:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 21:49:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 72B3D416DC8
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 21:48:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 21:48:47 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <164404123.4447.1308088127466.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7392) Implement capability of querying
 individual property of a mbean using JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Implement capability of querying individual property of a mbean using JMXProxyServlet 
--------------------------------------------------------------------------------------

                 Key: HADOOP-7392
                 URL: https://issues.apache.org/jira/browse/HADOOP-7392
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Tanping Wang


Hadoop-7144 provides the capability to query all the properties of a mbean using JMXProxyServlet.  In addition to this, we add the capability to query an individual property of a mbean.  Client will send http request,

http://hostname/jmx?get=meanName#property

to query from server.  


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16324-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 21:49:17 2011
Return-Path: <common-issues-return-16324-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B88096F4C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 21:49:17 +0000 (UTC)
Received: (qmail 99978 invoked by uid 500); 14 Jun 2011 21:49:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99844 invoked by uid 500); 14 Jun 2011 21:49:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99733 invoked by uid 99); 14 Jun 2011 21:49:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 21:49:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 21:49:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A2273416DD0
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 21:48:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 21:48:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1913857146.4454.1308088127661.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7393) PathData saves original string
 value, inviting failure when CWD changes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

PathData saves original string value, inviting failure when CWD changes
-----------------------------------------------------------------------

                 Key: HADOOP-7393
                 URL: https://issues.apache.org/jira/browse/HADOOP-7393
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
    Affects Versions: 0.23.0
            Reporter: Matt Foley


PathData#string stores the pathstring originally used to construct the Path, and returns it from various methods, apparently in an attempt to improve the user experience for the shell.

However, the current working directory may change, and if so this string value becomes meaningless and/or incorrect in context.


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16325-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 21:49:17 2011
Return-Path: <common-issues-return-16325-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C0C246F68
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 21:49:17 +0000 (UTC)
Received: (qmail 215 invoked by uid 500); 14 Jun 2011 21:49:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99971 invoked by uid 500); 14 Jun 2011 21:49:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99753 invoked by uid 99); 14 Jun 2011 21:49:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 21:49:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 21:49:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C4484416DD5
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 21:48:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 21:48:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1022113332.4459.1308088127801.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1913857146.4454.1308088127661.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7393) PathData saves original string
 value, inviting failure when CWD changes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049452#comment-13049452 ] 

Matt Foley commented on HADOOP-7393:
------------------------------------

Three solutions might be valid:
# Like Path, convert the pathstring to an absolute pathstring.  If it is desired to give the user an fs pathstring instead of URI as used by Path, then convert the relative fs pathstring into an absolute fs pathstring, and store that.
# Store the absolute path (or Path) of the CWD at instantiation time.  When #string is referenced, compare the current WD to the stored WD, and if they differ, convert #string to an absolute stringpath wrt the stored WD.
# Don't store #string.  Like Path, discard the pathstring after instantiation.  If needed later, return Path#uri.


> PathData saves original string value, inviting failure when CWD changes
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-7393
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7393
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Matt Foley
>
> PathData#string stores the pathstring originally used to construct the Path, and returns it from various methods, apparently in an attempt to improve the user experience for the shell.
> However, the current working directory may change, and if so this string value becomes meaningless and/or incorrect in context.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16326-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 21:53:10 2011
Return-Path: <common-issues-return-16326-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AD39C60D1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 21:53:10 +0000 (UTC)
Received: (qmail 14752 invoked by uid 500); 14 Jun 2011 21:53:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14662 invoked by uid 500); 14 Jun 2011 21:53:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14654 invoked by uid 99); 14 Jun 2011 21:53:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 21:53:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 21:53:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 71073416FDA
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 21:52:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 21:52:47 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <815018880.4473.1308088367459.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <164404123.4447.1308088127466.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7392) Implement capability of querying
 individual property of a mbean using JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tanping Wang updated HADOOP-7392:
---------------------------------

    Affects Version/s: 0.23.0
        Fix Version/s: 0.23.0
             Assignee: Tanping Wang

> Implement capability of querying individual property of a mbean using JMXProxyServlet 
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7392
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7392
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>             Fix For: 0.23.0
>
>
> Hadoop-7144 provides the capability to query all the properties of a mbean using JMXProxyServlet.  In addition to this, we add the capability to query an individual property of a mbean.  Client will send http request,
> http://hostname/jmx?get=meanName#property
> to query from server.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16327-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 21:55:09 2011
Return-Path: <common-issues-return-16327-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 965046130
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 21:55:09 +0000 (UTC)
Received: (qmail 18654 invoked by uid 500); 14 Jun 2011 21:55:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18618 invoked by uid 500); 14 Jun 2011 21:55:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18610 invoked by uid 99); 14 Jun 2011 21:55:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 21:55:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 21:55:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 609F1416168
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 21:54:48 +0000 (UTC)
Date: Tue, 14 Jun 2011 21:54:48 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1633698435.4538.1308088488392.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <272967917.1210.1308013607392.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7389) Use of TestingGroups by tests causes
 subsequent tests to fail
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-7389:
------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I've just committed this. Thanks, Aaron.

> Use of TestingGroups by tests causes subsequent tests to fail
> -------------------------------------------------------------
>
>                 Key: HADOOP-7389
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7389
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7389.0.patch, hadoop-7389.1.patch
>
>
> As mentioned in HADOOP-6671, {{UserGroupInformation.createUserForTesting(...)}} manipulates static state which can cause test cases which are run after a call to this function to fail.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16328-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 21:57:09 2011
Return-Path: <common-issues-return-16328-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E3FFE62E1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 21:57:08 +0000 (UTC)
Received: (qmail 21646 invoked by uid 500); 14 Jun 2011 21:57:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21608 invoked by uid 500); 14 Jun 2011 21:57:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21600 invoked by uid 99); 14 Jun 2011 21:57:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 21:57:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 21:57:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5BF1B41621B
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 21:56:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 21:56:47 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1922662521.4541.1308088607373.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049459#comment-13049459 ] 

Tanping Wang commented on HADOOP-7144:
--------------------------------------

created HADOOP-7392 to implement querying per mbean property.

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Robert Joseph Evans
>              Labels: jmx
>             Fix For: 0.20.205.0, 0.23.0
>
>         Attachments: HADOOP-7411-0.20.20X-V1.patch, HADOOP-7411-0.20.20X-V2.patch, HADOOP-7411-trunk-V1.patch, HADOOP-7411-trunk-V2.patch, HADOOP-7411-trunk-alpha.patch, jmx.json
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16329-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 22:05:09 2011
Return-Path: <common-issues-return-16329-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 94B946A23
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 22:05:09 +0000 (UTC)
Received: (qmail 35409 invoked by uid 500); 14 Jun 2011 22:05:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35329 invoked by uid 500); 14 Jun 2011 22:05:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35321 invoked by uid 99); 14 Jun 2011 22:05:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 22:05:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 22:05:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B9DC5416519
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 22:04:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 22:04:47 +0000 (UTC)
From: "Joep Rottinghuis (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <509141358.4568.1308089087757.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <389414663.13212.1307766119073.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7381) FindBugs OutOfMemoryError
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joep Rottinghuis updated HADOOP-7381:
-------------------------------------

    Description: 
When running the findbugs target from Jenkins, I get an OutOfMemory error.
The "effort" in FindBugs is set to Max which ends up using a lot of memory to go through all the classes. The jvmargs passed to FindBugs is hardcoded to 512 MB max.

We can leave the default to 512M, as long as we pass this as an ant parameter which can be overwritten in individual cases through -D, or in the build.properties file (either basedir, or user's home directory).


  was:
When running the findbugs target from Jenkins, I get an OutOfMemory error.
The "effort" in FindBugs is set to Max which ends up using a lot of memory to go through all the classes. The jvmargs passed to FindBugs is hardcoded to 512 MB max.

We can leave the default to 512M, as long as we pass this as an ant parameter which can be overwritten in individual cases through -D, or in the build.properties file (either basedir, or user's home directory).

This is the error I get:

findbugs:
    [mkdir] Created dir: /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs
 [findbugs] Executing findbugs from ant task
 [findbugs] Running FindBugs...
 [findbugs] Out of memory
 [findbugs] Total memory: 477M
 [findbugs]  free memory: 62M
 [findbugs] Analyzed: /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/hadoop-core-0.20.security-test-3.jar
...

[findbugs] Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
 [findbugs] 	at edu.umd.cs.findbugs.OpcodeStack.pushBySignature(OpcodeStack.java:2589)
 [findbugs] 	at edu.umd.cs.findbugs.OpcodeStack.pushByInvoke(OpcodeStack.java:2565)
 [findbugs] 	at edu.umd.cs.findbugs.OpcodeStack.processMethodCall(OpcodeStack.java:2011)
 [findbugs] 	at edu.umd.cs.findbugs.OpcodeStack.sawOpcode(OpcodeStack.java:1574)
                  ...15 more
 [findbugs] Java Result: 1
 [findbugs] Output saved to /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.xml
     [xslt] Processing /hadoop01/jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.xml to /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.html
     [xslt] Loading stylesheet /usr/local/findbugs/src/xsl/default.xsl
     [xslt] : Error! Premature end of file.
     [xslt] : Error! com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
     [xslt] Failed to process /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.xml

BUILD FAILED
/.../jenkins/jobs/hadoop-common-test-smoke/workspace/build.xml:1164: javax.xml.transform.TransformerException: javax.xml.transform.TransformerException: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:720)
	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:313)
	at org.apache.tools.ant.taskdefs.optional.TraXLiaison.transform(TraXLiaison.java:194)
	at org.apache.tools.ant.taskdefs.XSLTProcess.process(XSLTProcess.java:852)
	at org.apache.tools.ant.taskdefs.XSLTProcess.execute(XSLTProcess.java:388)
	at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
	... 15 more
Caused by: javax.xml.transform.TransformerException: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getDOM(TransformerImpl.java:547)
	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:710)
	... 20 more
Caused by: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
	at com.sun.org.apache.xalan.internal.xsltc.dom.XSLTCDTMManager.getDTM(XSLTCDTMManager.java:446)
	at com.sun.org.apache.xalan.internal.xsltc.dom.XSLTCDTMManager.getDTM(XSLTCDTMManager.java:234)
	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getDOM(TransformerImpl.java:525)
	... 21 more
---------

etc.etc. 
Total time: 8 minutes 35 seconds
Build step 'Execute shell' marked build as failure
[CHECKSTYLE] Collecting checkstyle analysis files...
CHECKSTYLESuccessfully parsed file /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/checkstyle-errors.xml of module  with 13525 warnings.
[FINDBUGS] Skipping publisher since build result is FAILURE
[WARNINGS] Skipping publisher since build result is FAILURE




Will look to see if a different patch is needed for trunk.


Moving exception stack to a comment to keep description short.
This is the error I get:

findbugs:
    [mkdir] Created dir: /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs
 [findbugs] Executing findbugs from ant task
 [findbugs] Running FindBugs...
 [findbugs] Out of memory
 [findbugs] Total memory: 477M
 [findbugs]  free memory: 62M
 [findbugs] Analyzed: /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/hadoop-core-0.20.security-test-3.jar
...

[findbugs] Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
 [findbugs] 	at edu.umd.cs.findbugs.OpcodeStack.pushBySignature(OpcodeStack.java:2589)
 [findbugs] 	at edu.umd.cs.findbugs.OpcodeStack.pushByInvoke(OpcodeStack.java:2565)
 [findbugs] 	at edu.umd.cs.findbugs.OpcodeStack.processMethodCall(OpcodeStack.java:2011)
 [findbugs] 	at edu.umd.cs.findbugs.OpcodeStack.sawOpcode(OpcodeStack.java:1574)
                  ...15 more
 [findbugs] Java Result: 1
 [findbugs] Output saved to /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.xml
     [xslt] Processing /hadoop01/jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.xml to /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.html
     [xslt] Loading stylesheet /usr/local/findbugs/src/xsl/default.xsl
     [xslt] : Error! Premature end of file.
     [xslt] : Error! com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
     [xslt] Failed to process /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/findbugs/hadoop-findbugs-report.xml

BUILD FAILED
/.../jenkins/jobs/hadoop-common-test-smoke/workspace/build.xml:1164: javax.xml.transform.TransformerException: javax.xml.transform.TransformerException: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:720)
	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:313)
	at org.apache.tools.ant.taskdefs.optional.TraXLiaison.transform(TraXLiaison.java:194)
	at org.apache.tools.ant.taskdefs.XSLTProcess.process(XSLTProcess.java:852)
	at org.apache.tools.ant.taskdefs.XSLTProcess.execute(XSLTProcess.java:388)
	at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291)
	... 15 more
Caused by: javax.xml.transform.TransformerException: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getDOM(TransformerImpl.java:547)
	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(TransformerImpl.java:710)
	... 20 more
Caused by: com.sun.org.apache.xml.internal.utils.WrappedRuntimeException: Premature end of file.
	at com.sun.org.apache.xalan.internal.xsltc.dom.XSLTCDTMManager.getDTM(XSLTCDTMManager.java:446)
	at com.sun.org.apache.xalan.internal.xsltc.dom.XSLTCDTMManager.getDTM(XSLTCDTMManager.java:234)
	at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.getDOM(TransformerImpl.java:525)
	... 21 more
---------

etc.etc. 
Total time: 8 minutes 35 seconds
Build step 'Execute shell' marked build as failure
[CHECKSTYLE] Collecting checkstyle analysis files...
CHECKSTYLESuccessfully parsed file /.../jenkins/jobs/hadoop-common-test-smoke/workspace/build/test/checkstyle-errors.xml of module  with 13525 warnings.
[FINDBUGS] Skipping publisher since build result is FAILURE
[WARNINGS] Skipping publisher since build result is FAILURE



> FindBugs OutOfMemoryError
> -------------------------
>
>                 Key: HADOOP-7381
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7381
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.20.205.0
>         Environment: FindBugs 1.3.9, ant 1.8.2, RHEL6, Jenkins 1.414 in Tomcat 7.0.14, Sun Java HotSpot(TM) 64-Bit Server VM (build 14.2-b01, mixed mode)
>            Reporter: Joep Rottinghuis
>            Assignee: Joep Rottinghuis
>            Priority: Minor
>         Attachments: hadoop-7381.patch
>
>
> When running the findbugs target from Jenkins, I get an OutOfMemory error.
> The "effort" in FindBugs is set to Max which ends up using a lot of memory to go through all the classes. The jvmargs passed to FindBugs is hardcoded to 512 MB max.
> We can leave the default to 512M, as long as we pass this as an ant parameter which can be overwritten in individual cases through -D, or in the build.properties file (either basedir, or user's home directory).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16330-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 22:07:09 2011
Return-Path: <common-issues-return-16330-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 649CE6ACC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 22:07:09 +0000 (UTC)
Received: (qmail 42809 invoked by uid 500); 14 Jun 2011 22:07:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42766 invoked by uid 500); 14 Jun 2011 22:07:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42758 invoked by uid 99); 14 Jun 2011 22:07:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 22:07:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 22:07:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 33B2241665D
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 22:06:48 +0000 (UTC)
Date: Tue, 14 Jun 2011 22:06:48 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1984521321.4585.1308089208208.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6929) RPC should have a way to pass
 Security information other than protocol annotations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6929:
----------------------------------

    Attachment: h-6929.patch

Fix the metrics testcase that was using a hardcoded build/classes path name.

> RPC should have a way to pass Security information other than protocol annotations
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6929
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6929
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: Hadoop-6929_v1.patch, h-6929.patch, h-6929.patch, h-6929.patch
>
>
> Currently Hadoop RPC allows protocol annotations as the only way to pass security information. This becomes a problem if protocols are generated and not hand written. For example protocols generated via Avro and passed over Avro tunnel (AvroRpcEngine.java) can't pass the security information.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16331-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 22:07:12 2011
Return-Path: <common-issues-return-16331-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 89F106ADD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 22:07:12 +0000 (UTC)
Received: (qmail 43109 invoked by uid 500); 14 Jun 2011 22:07:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43080 invoked by uid 500); 14 Jun 2011 22:07:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43064 invoked by uid 99); 14 Jun 2011 22:07:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 22:07:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 22:07:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B4095416670
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 22:06:48 +0000 (UTC)
Date: Tue, 14 Jun 2011 22:06:48 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1436774551.4603.1308089208734.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6929) RPC should have a way to pass
 Security information other than protocol annotations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6929:
----------------------------------

    Status: Patch Available  (was: Open)

> RPC should have a way to pass Security information other than protocol annotations
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6929
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6929
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: Hadoop-6929_v1.patch, h-6929.patch, h-6929.patch, h-6929.patch
>
>
> Currently Hadoop RPC allows protocol annotations as the only way to pass security information. This becomes a problem if protocols are generated and not hand written. For example protocols generated via Avro and passed over Avro tunnel (AvroRpcEngine.java) can't pass the security information.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16332-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 22:07:13 2011
Return-Path: <common-issues-return-16332-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8B8C56AE6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 22:07:12 +0000 (UTC)
Received: (qmail 43161 invoked by uid 500); 14 Jun 2011 22:07:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43082 invoked by uid 500); 14 Jun 2011 22:07:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43065 invoked by uid 99); 14 Jun 2011 22:07:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 22:07:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 22:07:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A600B41666E
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 22:06:48 +0000 (UTC)
Date: Tue, 14 Jun 2011 22:06:48 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <883586894.4601.1308089208676.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6929) RPC should have a way to pass
 Security information other than protocol annotations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6929:
----------------------------------

    Status: Open  (was: Patch Available)

> RPC should have a way to pass Security information other than protocol annotations
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6929
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6929
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: Hadoop-6929_v1.patch, h-6929.patch, h-6929.patch, h-6929.patch
>
>
> Currently Hadoop RPC allows protocol annotations as the only way to pass security information. This becomes a problem if protocols are generated and not hand written. For example protocols generated via Avro and passed over Avro tunnel (AvroRpcEngine.java) can't pass the security information.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16333-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 22:11:11 2011
Return-Path: <common-issues-return-16333-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EA1CA6BD7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 22:11:11 +0000 (UTC)
Received: (qmail 51637 invoked by uid 500); 14 Jun 2011 22:11:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51605 invoked by uid 500); 14 Jun 2011 22:11:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51597 invoked by uid 99); 14 Jun 2011 22:11:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 22:11:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 22:11:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CA8704168A9
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 22:10:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 22:10:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <608995701.4635.1308089447826.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <92042485.70969.1307347427375.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7360) FsShell does not preserve relative
 paths with globs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049467#comment-13049467 ] 

Matt Foley commented on HADOOP-7360:
------------------------------------

TestPathData:

Multiple places: The preferred way to express "new Path(testDir+"/d1")" is "new Path(testDir, "d1")".  This avoids hard-coding the path delimiter for a local filesystem path.

initialize():
* Consider "fs.createNewFile(Path)" instead of "fs.create(Path).close()"
* If you put the "setWorkingDirectory" command near the beginning of the method, wouldn't all the creates and mkdirs get simpler?

I'm concerned that testDir is a relative path.  Can you convert it to an absolute path as early in the process as possible, before anyone might have set a non-default working directory?

Why did you remove testWithFsAndPath()?  It is not a null test.

getParent() should be named testGetParent().

testRelativeGlobBack(): see previous concern about making "testDir" absolute before interacting with second and later setWorkingDirectory() invocations.  This might give you testDir/testDir/d1 ?

PathData:

Agree with your elimination of PathData(FileSystem, Path, FileStatus),
but suggest also eliminating PathData(FileSystem, String, FileStatus)
and PathData(FileSystem, FileStatus).
Basically, the status should be looked up in the FS, not set as an argument.
Should do so at the end of method PathData(FileSystem, String).
In comment for PathData(FileSystem, String), remove "Convenience" adjective.  It becomes the primary ctor.

setStat(boolean ignoreFNF) is counter-intuitive.  Most people seeing "setStat(boolean)" will assume the status is being set to the argument.  Please call it lookupStatus(boolean ignoreFNF).  Or could use two methods, both with zero arguments, called setStat() and setStatIgnoreFNF().

Suggest letting lookupStatus(ignoreFNF) return the stat value instead of void.  Then refreshStatus just becomes "return lookupStatus(false);".

getChecksumFile(): Why is this method needed (returns PathData), vs just using Fs.getChecksumFile() (returns Path)?

getParent(): Several concerns:
* Why is this method needed instead of just using Path.getParent()?  There are currently no callers of this new method.
* Why is it important to preserve the relative-ness of the parent path?  Trying to do so exposes you to serious issues because the current working directory may have changed since PathData#string was stored, which means the value returned by this method could be meaningless if string was relative.  I think I would go so far as to say this can't work.  I would much rather let Path.getParent() do what it was carefully implemented to do.
* The same issue applies to the next two chunks.

This brings up the issue of why PathData saves the value of #string?
This just begs for such mis-use.  I've opened HADOOP-7393 for this.


> FsShell does not preserve relative paths with globs
> ---------------------------------------------------
>
>                 Key: HADOOP-7360
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7360
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7360.patch
>
>
> FsShell currently preserves relative paths that do not contain globs.  Unfortunately the method {{fs.globStatus()}} is fully qualifying all returned paths.  This is causing inconsistent display of paths.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16334-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 22:17:09 2011
Return-Path: <common-issues-return-16334-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1860D6477
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 22:17:09 +0000 (UTC)
Received: (qmail 60521 invoked by uid 500); 14 Jun 2011 22:17:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60487 invoked by uid 500); 14 Jun 2011 22:17:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60477 invoked by uid 99); 14 Jun 2011 22:17:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 22:17:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 22:17:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B661A416AFC
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 22:16:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 22:16:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2018682955.4642.1308089807743.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <272967917.1210.1308013607392.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7389) Use of TestingGroups by tests
 causes subsequent tests to fail
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049469#comment-13049469 ] 

Hudson commented on HADOOP-7389:
--------------------------------

Integrated in Hadoop-Common-trunk-maven #17 (See [https://builds.apache.org/job/Hadoop-Common-trunk-maven/17/])
    HADOOP-7389. Use of TestingGroups by tests causes subsequent tests to fail. Contributed by Aaron T. Myers.

tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1135820
Files : 
* /hadoop/common/trunk/common/CHANGES.txt
* /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/security/TestUserGroupInformation.java
* /hadoop/common/trunk/common/src/java/org/apache/hadoop/security/UserGroupInformation.java


> Use of TestingGroups by tests causes subsequent tests to fail
> -------------------------------------------------------------
>
>                 Key: HADOOP-7389
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7389
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7389.0.patch, hadoop-7389.1.patch
>
>
> As mentioned in HADOOP-6671, {{UserGroupInformation.createUserForTesting(...)}} manipulates static state which can cause test cases which are run after a call to this function to fail.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16336-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 22:17:09 2011
Return-Path: <common-issues-return-16336-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BAB2C6496
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 22:17:09 +0000 (UTC)
Received: (qmail 60824 invoked by uid 500); 14 Jun 2011 22:17:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60754 invoked by uid 500); 14 Jun 2011 22:17:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60738 invoked by uid 99); 14 Jun 2011 22:17:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 22:17:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 22:17:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3B0E2416B0A
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 22:16:48 +0000 (UTC)
Date: Tue, 14 Jun 2011 22:16:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <642332564.4656.1308089808238.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049471#comment-13049471 ] 

Hudson commented on HADOOP-6605:
--------------------------------

Integrated in Hadoop-Common-trunk-maven #17 (See [https://builds.apache.org/job/Hadoop-Common-trunk-maven/17/])
    

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch, hadoop-6605-2.patch, hadoop-6605-3.patch, hadoop-6605-4.patch, hadoop-6605-5.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16335-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 22:17:09 2011
Return-Path: <common-issues-return-16335-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D788664A0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 22:17:09 +0000 (UTC)
Received: (qmail 60825 invoked by uid 500); 14 Jun 2011 22:17:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60751 invoked by uid 500); 14 Jun 2011 22:17:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60676 invoked by uid 99); 14 Jun 2011 22:17:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 22:17:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 22:17:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E263F416B01
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 22:16:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 22:16:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <314357923.4647.1308089807923.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1507121853.1613.1307925711791.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7384) Allow test-patch to be more
 flexible about patch format
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049470#comment-13049470 ] 

Hudson commented on HADOOP-7384:
--------------------------------

Integrated in Hadoop-Common-trunk-maven #17 (See [https://builds.apache.org/job/Hadoop-Common-trunk-maven/17/])
    Amend commit of HADOOP-7384 to include svn:executable bit on smart-apply-patch.sh
HADOOP-7384. Allow test-patch to be more flexible about patch format. Contributed by Todd Lipcon.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1135636
Files : 
* /hadoop/common/trunk/common/src/test/bin/smart-apply-patch.sh

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1135633
Files : 
* /hadoop/common/trunk/common/CHANGES.txt
* /hadoop/common/trunk/common/src/test/bin/smart-apply-patch.sh
* /hadoop/common/trunk/common/src/test/bin/test-patch.sh


> Allow test-patch to be more flexible about patch format
> -------------------------------------------------------
>
>                 Key: HADOOP-7384
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7384
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7384.txt
>
>
> Right now the test-patch process only accepts patches that are generated as "-p0" relative to common/, hdfs/, or mapreduce/. This has always been annoying for git users where the default patch format is -p1. It's also now annoying for SVN users who may generate a patch relative to trunk/ instead of the subproject subdirectory. We should auto-detect the correct patch level.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16337-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 22:25:11 2011
Return-Path: <common-issues-return-16337-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 12DB2647C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 22:25:11 +0000 (UTC)
Received: (qmail 73468 invoked by uid 500); 14 Jun 2011 22:25:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73429 invoked by uid 500); 14 Jun 2011 22:25:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73416 invoked by uid 99); 14 Jun 2011 22:25:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 22:25:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 22:25:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5E4A141601C
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 22:24:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 22:24:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <592503529.4675.1308090287382.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6929) RPC should have a way to pass
 Security information other than protocol annotations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049478#comment-13049478 ] 

Hadoop QA commented on HADOOP-6929:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482605/h-6929.patch
  against trunk revision 1135820.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 10 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/632//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/632//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/632//console

This message is automatically generated.

> RPC should have a way to pass Security information other than protocol annotations
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6929
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6929
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: Hadoop-6929_v1.patch, h-6929.patch, h-6929.patch, h-6929.patch
>
>
> Currently Hadoop RPC allows protocol annotations as the only way to pass security information. This becomes a problem if protocols are generated and not hand written. For example protocols generated via Avro and passed over Avro tunnel (AvroRpcEngine.java) can't pass the security information.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16338-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 22:53:09 2011
Return-Path: <common-issues-return-16338-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 264196732
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 22:53:09 +0000 (UTC)
Received: (qmail 16545 invoked by uid 500); 14 Jun 2011 22:53:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16456 invoked by uid 500); 14 Jun 2011 22:53:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16344 invoked by uid 99); 14 Jun 2011 22:53:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 22:53:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 22:53:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 72310416E65
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 22:52:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 22:52:47 +0000 (UTC)
From: "Bharath Mundlapudi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <295892337.4779.1308091967464.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <113408089.3186.1307985891583.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7385) Remove
 StringUtils.stringifyException(ie) in logger functions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bharath Mundlapudi updated HADOOP-7385:
---------------------------------------

    Attachment: HADOOP-7385-1.patch

Attaching the patch.

> Remove StringUtils.stringifyException(ie) in logger functions
> -------------------------------------------------------------
>
>                 Key: HADOOP-7385
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7385
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7385-1.patch
>
>
> Apache logger api has an overloaded function which can take the message and exception. I am proposing to clean the logging code with this api.
> ie.:
> Change the code from LOG.warn(msg, StringUtils.stringifyException(exception)); to LOG.warn(msg, exception);

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16339-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 22:55:09 2011
Return-Path: <common-issues-return-16339-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 83772676F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 22:55:09 +0000 (UTC)
Received: (qmail 17602 invoked by uid 500); 14 Jun 2011 22:55:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17474 invoked by uid 500); 14 Jun 2011 22:55:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17424 invoked by uid 99); 14 Jun 2011 22:55:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 22:55:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 22:55:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3539F416FD1
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 22:54:48 +0000 (UTC)
Date: Tue, 14 Jun 2011 22:54:48 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1052250844.4787.1308092088214.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1354187382.13188.1307764078831.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7380) Common portion of HDFS-1973
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7380:
-----------------------------------

    Attachment: hadoop-7380.0.patch

Here's a first hack at the Common portion of client failover, not intended for commit.

If people think this approach is roughly sound I'll fix it up and prepare it for commit. It still needs at least the following work:

* A way to mark certain interface methods as idempotent (and thus should be retried upon failover). This will probably take the form of a method annotation.
* Refinement of the precise set of exceptions to deal with.
* Potential refinement of the {{FailoverProxyProvider}} interface. I'm not in love with it as it stands.

I've also created a test which uses this code to actually perform a failover between two federated NNs (which happen to have the same FS metadata) by starting them both up, creating a file in each at the same path, shutting down one, and ensuring the DFSClient properly fails over to the second. I can post that patch on HDFS-1973 but it depends on this patch.

> Common portion of HDFS-1973
> ---------------------------
>
>                 Key: HADOOP-7380
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7380
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7380.0.patch
>
>
> Implementing client failover will likely require changes to {{o.a.h.io.ipc}} and/or {{o.a.h.io.retry}}. This JIRA is to track those changes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16340-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 23:23:08 2011
Return-Path: <common-issues-return-16340-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A6A2F68A6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 23:23:08 +0000 (UTC)
Received: (qmail 49449 invoked by uid 500); 14 Jun 2011 23:23:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49404 invoked by uid 500); 14 Jun 2011 23:23:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49396 invoked by uid 99); 14 Jun 2011 23:23:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 23:23:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 23:23:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5AFDF416BE1
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 23:22:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 23:22:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <829078509.4909.1308093767369.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1913857146.4454.1308088127661.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7393) PathData saves original string
 value, inviting failure when CWD changes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049509#comment-13049509 ] 

Daryn Sharp commented on HADOOP-7393:
-------------------------------------

As currently designed/implemented, I feel the premise of this jira is invalid, however I do understand the sentiment.  The PathData object is only intended for internal use in FsShell.  The cwd does not change during the execution of any of the commands, so the relative path is not invalidated.

bq. Like Path, convert the pathstring to an absolute pathstring. [...]
bq. Don't store #string. [...]

This would unravel the benefit.  The intent of storing the string is that what you give it is +exactly+ what you get back, just like a unix shell, not a path that has been mangled.  If you were to take the old FsShell and feed it an arbitrary path, you would have a hard time reliably matching the path in the output.

bq.  Store the absolute path (or Path) of the CWD at instantiation time. When #string is referenced, compare the current WD to the stored WD, and if they differ, convert #string to an absolute stringpath wrt the stored WD.

I would put forth the opinion that PathData is currently doing the "right thing":  relative paths are invalidated when pwd changes.  At a normal shell, if you were to type {{x=foo.txt}}, {{cat $x}}, {{cd somewhere-else}}, and again {{cat $x}}, you certainly wouldn't expect the original {{foo.txt}} to be displayed.  If the desire is to use PathData outside of FsShell, I could probably be swayed from this position.

My goal has been for FsShell to act and feel just like a real shell.  That said, I do welcome a response.

> PathData saves original string value, inviting failure when CWD changes
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-7393
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7393
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Matt Foley
>
> PathData#string stores the pathstring originally used to construct the Path, and returns it from various methods, apparently in an attempt to improve the user experience for the shell.
> However, the current working directory may change, and if so this string value becomes meaningless and/or incorrect in context.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16341-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 23:29:10 2011
Return-Path: <common-issues-return-16341-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C293A6A74
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 23:29:10 +0000 (UTC)
Received: (qmail 55509 invoked by uid 500); 14 Jun 2011 23:29:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55481 invoked by uid 500); 14 Jun 2011 23:29:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55473 invoked by uid 99); 14 Jun 2011 23:29:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 23:29:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 23:29:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 63859416E2F
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 23:28:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 23:28:47 +0000 (UTC)
From: "Ian Holsman (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <284169479.4914.1308094127404.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7106) Re-organize hadoop subversion
 layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049510#comment-13049510 ] 

Ian Holsman commented on HADOOP-7106:
-------------------------------------

mailer-conf.diff has been applied. revision: 790962.

> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, gitk-example.png, mailer-conf.diff
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16342-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 23:31:09 2011
Return-Path: <common-issues-return-16342-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6D1056042
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 23:31:09 +0000 (UTC)
Received: (qmail 56801 invoked by uid 500); 14 Jun 2011 23:31:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56762 invoked by uid 500); 14 Jun 2011 23:31:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56753 invoked by uid 99); 14 Jun 2011 23:31:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 23:31:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 23:31:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2EB0E416F99
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 23:30:48 +0000 (UTC)
Date: Tue, 14 Jun 2011 23:30:48 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <589876698.4966.1308094248188.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-6671:
---------------------------------------

    Attachment: HADOOP-6671-h.patch

Wiring Clover.

The following property in the pom.xml may have to set to the correct value in the Jenkins servers:
    <cloverLicenseLocation>${user.home}/.clover.license</cloverLicenseLocation>



Build commands avail:

* Clean                     : mvn clean
* Compile                   : mvn compile
* Run tests                 : mvn test
* Create JAR                : mvn package
* Run findbugs              : mvn package -DrunFindbugs
* Run checkstyle            : mvn checkstyle:checkstyle
* TAR wo/docs & wo/source   : mvn package -DmakeTar
* TAR w/docs & w/source     : mvn package -DmakeTar -DgenerateDocs -DincludeSource
* TAR w/source only         : mvn assembly:single -Dsource 
* Install in local M2 cache : mvn install
* Deploy to Maven repo      : mvn deploy
* Run clover                : mvn test -DrunClover



> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-6671-cross-project-HDFS.patch, HADOOP-6671-e.patch, HADOOP-6671-f.patch, HADOOP-6671-g.patch, HADOOP-6671-h.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout-f.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16343-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 23:33:12 2011
Return-Path: <common-issues-return-16343-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6EB6D6A24
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 23:33:12 +0000 (UTC)
Received: (qmail 59911 invoked by uid 500); 14 Jun 2011 23:33:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59884 invoked by uid 500); 14 Jun 2011 23:33:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59876 invoked by uid 99); 14 Jun 2011 23:33:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 23:33:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 23:33:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2549D41613F
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 23:32:49 +0000 (UTC)
Date: Tue, 14 Jun 2011 23:32:49 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <891308148.5025.1308094369149.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32237646.17061288907081267.JavaMail.jira@thor>
Subject: [jira] [Updated] (HADOOP-7020) establish a "Powered by Hadoop" logo
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7020:
-----------------------------------

    Attachment: hadoop.JPG

Here's a proposal for consideration kindly contributed by Stephanie Davidson. I like it because it's quite visually distinct from the project logo, but obviously sticks with the yellow elephant and cartoony theme. I also like this one since it sticks with the color theme of the logos so far associated with the project.

It's also the only logo so far proposed in which the elephant is clearly powering something. :)

> establish a "Powered by Hadoop" logo
> ------------------------------------
>
>                 Key: HADOOP-7020
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7020
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: site
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: site
>
>         Attachments: PoweredByApacheHadoop.png, PoweredByHadoop_Small.jpg, hadoop-elephant-pb.jpeg, hadoop.JPG, powered-by-hadoop-small.png, powered-by-hadoop.png, powered-by-proposal.jpg, powered_by_hadoop.jpg
>
>
> We should agree on a Powered By Hadoop logo, as suggested in:
> http://www.apache.org/foundation/marks/pmcs#poweredby

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16344-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 14 23:35:11 2011
Return-Path: <common-issues-return-16344-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3772160D1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 14 Jun 2011 23:35:11 +0000 (UTC)
Received: (qmail 64686 invoked by uid 500); 14 Jun 2011 23:35:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64650 invoked by uid 500); 14 Jun 2011 23:35:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64613 invoked by uid 99); 14 Jun 2011 23:35:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 23:35:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Jun 2011 23:35:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 636114161E2
	for <common-issues@hadoop.apache.org>; Tue, 14 Jun 2011 23:34:47 +0000 (UTC)
Date: Tue, 14 Jun 2011 23:34:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1058799296.5030.1308094487403.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049519#comment-13049519 ] 

Hadoop QA commented on HADOOP-6671:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482623/HADOOP-6671-h.patch
  against trunk revision 1135820.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 38 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/633//console

This message is automatically generated.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-6671-cross-project-HDFS.patch, HADOOP-6671-e.patch, HADOOP-6671-f.patch, HADOOP-6671-g.patch, HADOOP-6671-h.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout-f.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16345-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 00:37:11 2011
Return-Path: <common-issues-return-16345-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 54379615B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 00:37:11 +0000 (UTC)
Received: (qmail 29586 invoked by uid 500); 15 Jun 2011 00:37:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29556 invoked by uid 500); 15 Jun 2011 00:37:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29548 invoked by uid 99); 15 Jun 2011 00:37:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 00:37:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 00:37:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5711D417648
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 00:36:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 00:36:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1790031637.5219.1308098207353.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <92042485.70969.1307347427375.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7360) FsShell does not preserve relative
 paths with globs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049546#comment-13049546 ] 

Daryn Sharp commented on HADOOP-7360:
-------------------------------------

I can address the test points.  One question: If I need to construct a path of {{testdir+"/d1/f1"}}, is this the preferred way: {{new Path(new Path(testDir, "d1"), "f1")}}?

bq.  Why did you remove testWithFsAndPath()

It's private now.

bq. [...] suggest also eliminating PathData(FileSystem, String, FileStatus) and PathData(FileSystem, FileStatus).  Basically, the status should be looked up in the FS, not set as an argument.

{{PathData(FileSystem, FileStatus)}} was removed in the patch.  Did you mean something else?

It would be a very bad idea to remove the method {{PathData(FileSystem, String, FileStatus)}}.  It's private, and only used in cases like listStatus/globStatus where the result is an array of {{FileStatus}}.  It's used to avoid an unnecessary re-stat of the path -- overall, this class has drastically reduced the unnecessary re-stats occurring in FsShell.

bq. PathData(FileSystem, String) [...] becomes the primary ctor.
{{PathData(FileSystem,String,FileStatus)}} is the primary ctor because a) removing it will cause 2X the stats, b) it extracts path from a non-null status.  The path is a final, so I can't invoke {{super(FileSystem, String)}} and then reset the path.

bq. setStat(boolean ignoreFNF) is counter-intuitive. [...] Please call it lookupStatus(boolean ignoreFNF) [...]
100% agreed!  I'm actually surprised I did that...  Either I was lacking coffee, or slipped up an eclipse refactor...

bq. getChecksumFile(): Why is this method needed (returns PathData), vs just using Fs.getChecksumFile() (returns Path)?
There's another change backed up behind this one that drastically simplifies the copy commands to operate purely on PathData (which incidentally will further reduce stats, but that's not the primary purpose).  Since {{PathData(FileSystem,String)}} is now private, I had to push it into PathData, else I have to re-expose the private ctor which will complicate the switch to {{FileContext}}.  Since commands aren't storing the fs in temporaries, it may be possible to just switch fs to a FileContext and generally have everything work.  Having a command monkeying around getting a raw fs for the crc file and then instantiating thru what what should be a private ctor is a minor setback.

bq. Why is this method needed instead of just using Path.getParent()? There are currently no callers of this new method.
There are backed up changes that depend on it (I didn't expect this jira to languish...).  Remember that everything is operating on PathData, so I can't use Path.getParent() because I'd have to re-expose a private ctor.


The other questions doubting the need for relativity are addressed on the jira you filed.  My design goal has been to mimic a unix shell.

bq. This just begs for such mis-use
I think computer users are well enough acquainted with filesystems to know that if you change the pwd, then relative paths are no longer valid.

> FsShell does not preserve relative paths with globs
> ---------------------------------------------------
>
>                 Key: HADOOP-7360
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7360
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7360.patch
>
>
> FsShell currently preserves relative paths that do not contain globs.  Unfortunately the method {{fs.globStatus()}} is fully qualifying all returned paths.  This is causing inconsistent display of paths.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16346-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 00:47:10 2011
Return-Path: <common-issues-return-16346-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B78CC66D1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 00:47:10 +0000 (UTC)
Received: (qmail 45522 invoked by uid 500); 15 Jun 2011 00:47:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45494 invoked by uid 500); 15 Jun 2011 00:47:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45486 invoked by uid 99); 15 Jun 2011 00:47:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 00:47:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 00:47:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 616F54178AE
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 00:46:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 00:46:47 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1574411268.5244.1308098807395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049548#comment-13049548 ] 

Tom White commented on HADOOP-6671:
-----------------------------------

I updated https://builds.apache.org/job/Hadoop-Common-trunk-maven/ to generate Clover reports.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-6671-cross-project-HDFS.patch, HADOOP-6671-e.patch, HADOOP-6671-f.patch, HADOOP-6671-g.patch, HADOOP-6671-h.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout-f.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16347-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 00:53:09 2011
Return-Path: <common-issues-return-16347-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B09936E54
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 00:53:09 +0000 (UTC)
Received: (qmail 51969 invoked by uid 500); 15 Jun 2011 00:53:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51786 invoked by uid 500); 15 Jun 2011 00:53:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51770 invoked by uid 99); 15 Jun 2011 00:53:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 00:53:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 00:53:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4B93D417A0B
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 00:52:48 +0000 (UTC)
Date: Wed, 15 Jun 2011 00:52:48 +0000 (UTC)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <141353966.5294.1308099168306.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7106) Re-organize hadoop subversion
 layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049549#comment-13049549 ] 

Konstantin Shvachko commented on HADOOP-7106:
---------------------------------------------

Jenkins paths need to be updated. Both PreCommit-HDFS-Build and PreCommit-MAPREDUCE-Build are failing. Other builds are probably affected as well.

> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, gitk-example.png, mailer-conf.diff
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16348-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 00:57:13 2011
Return-Path: <common-issues-return-16348-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 356066F1C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 00:57:13 +0000 (UTC)
Received: (qmail 55728 invoked by uid 500); 15 Jun 2011 00:57:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55692 invoked by uid 500); 15 Jun 2011 00:57:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55684 invoked by uid 99); 15 Jun 2011 00:57:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 00:57:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 00:57:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DDCA3417B5C
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 00:56:49 +0000 (UTC)
Date: Wed, 15 Jun 2011 00:56:49 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1918591724.5337.1308099409905.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7106) Re-organize hadoop subversion
 layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049551#comment-13049551 ] 

Todd Lipcon commented on HADOOP-7106:
-------------------------------------

Fixed the precommit builds and resubmitted all of the patches that had failed due to the problem. I did all of the other builds, but somehow missed these. Sorry about that!

> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, gitk-example.png, mailer-conf.diff
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16349-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 01:15:12 2011
Return-Path: <common-issues-return-16349-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EF14D694E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 01:15:11 +0000 (UTC)
Received: (qmail 66609 invoked by uid 500); 15 Jun 2011 01:15:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66571 invoked by uid 500); 15 Jun 2011 01:15:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66563 invoked by uid 99); 15 Jun 2011 01:15:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 01:15:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 01:15:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8659A41615A
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 01:14:48 +0000 (UTC)
Date: Wed, 15 Jun 2011 01:14:48 +0000 (UTC)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <215762644.5417.1308100488547.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7106) Re-organize hadoop subversion
 layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049558#comment-13049558 ] 

Konstantin Shvachko commented on HADOOP-7106:
---------------------------------------------

Thanks, Tom!

> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, gitk-example.png, mailer-conf.diff
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16350-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 01:15:12 2011
Return-Path: <common-issues-return-16350-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D37F9695F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 01:15:12 +0000 (UTC)
Received: (qmail 66864 invoked by uid 500); 15 Jun 2011 01:15:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66837 invoked by uid 500); 15 Jun 2011 01:15:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66829 invoked by uid 99); 15 Jun 2011 01:15:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 01:15:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 01:15:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 68EBA416181
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 01:14:49 +0000 (UTC)
Date: Wed, 15 Jun 2011 01:14:49 +0000 (UTC)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1801087231.5443.1308100489426.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7106) Re-organize hadoop subversion
 layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049559#comment-13049559 ] 

Konstantin Shvachko commented on HADOOP-7106:
---------------------------------------------

Oops, I meant Todd ;-)

> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, gitk-example.png, mailer-conf.diff
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16351-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 02:13:11 2011
Return-Path: <common-issues-return-16351-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CBD754205
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 02:13:11 +0000 (UTC)
Received: (qmail 22397 invoked by uid 500); 15 Jun 2011 02:13:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22371 invoked by uid 500); 15 Jun 2011 02:13:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22357 invoked by uid 99); 15 Jun 2011 02:13:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 02:13:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 02:13:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 61746417645
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 02:12:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 02:12:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <639440876.5515.1308103967395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7394) Chinese documentation can't be built
 with Forrest 0.9
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Chinese documentation can't be built with Forrest 0.9
-----------------------------------------------------

                 Key: HADOOP-7394
                 URL: https://issues.apache.org/jira/browse/HADOOP-7394
             Project: Hadoop Common
          Issue Type: Bug
          Components: documentation
    Affects Versions: 0.23.0
            Reporter: Aaron T. Myers
            Priority: Minor


Running {{ant cn-docs}} with Forrest 0.8 will work. Running the same thing with Forrest 0.9 prints the following error:

{noformat}
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/fop/messaging/MessageHandler
	at org.apache.cocoon.serialization.FOPSerializer.configure(FOPSerializer.java:122)
	at org.apache.avalon.framework.container.ContainerUtil.configure(ContainerUtil.java:201)
	at org.apache.avalon.excalibur.component.DefaultComponentFactory.newInstance(DefaultComponentFactory.java:289)
	at org.apache.avalon.excalibur.pool.InstrumentedResourceLimitingPool.newPoolable(InstrumentedResourceLimitingPool.java:655)
	at org.apache.avalon.excalibur.pool.InstrumentedResourceLimitingPool.get(InstrumentedResourceLimitingPool.java:371)
	at org.apache.avalon.excalibur.component.PoolableComponentHandler.doGet(PoolableComponentHandler.java:198)
	at org.apache.avalon.excalibur.component.ComponentHandler.get(ComponentHandler.java:381)
	at org.apache.avalon.excalibur.component.ExcaliburComponentSelector.select(ExcaliburComponentSelector.java:215)
	at org.apache.cocoon.components.ExtendedComponentSelector.select(ExtendedComponentSelector.java:268)
	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.setSerializer(AbstractProcessingPipeline.java:311)
	at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.setSerializer(AbstractCachingProcessingPipeline.java:171)
	at org.apache.cocoon.components.treeprocessor.sitemap.SerializeNode.invoke(SerializeNode.java:120)
	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)
	at org.apache.cocoon.components.treeprocessor.sitemap.SelectNode.invoke(SelectNode.java:103)
	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:47)
	at org.apache.cocoon.components.treeprocessor.sitemap.PreparableMatchNode.invoke(PreparableMatchNode.java:131)
	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)
	at org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:143)
	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)
	at org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:93)
	at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:235)
	at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:177)
	at org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:254)
	at org.apache.cocoon.components.treeprocessor.sitemap.MountNode.invoke(MountNode.java:118)
	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)
	at org.apache.cocoon.components.treeprocessor.sitemap.SelectNode.invoke(SelectNode.java:98)
	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)
	at org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:143)
	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)
	at org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:93)
	at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:235)
	at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:177)
	at org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:254)
	at org.apache.cocoon.Cocoon.process(Cocoon.java:699)
	at org.apache.cocoon.bean.CocoonWrapper.getPage(CocoonWrapper.java:514)
	at org.apache.cocoon.bean.CocoonBean.processTarget(CocoonBean.java:499)
	at org.apache.cocoon.bean.CocoonBean.process(CocoonBean.java:356)
	at org.apache.cocoon.Main.main(Main.java:321)
Caused by: java.lang.ClassNotFoundException: org.apache.fop.messaging.MessageHandler
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
	... 38 more
{noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16352-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 02:49:10 2011
Return-Path: <common-issues-return-16352-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5B1B24B94
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 02:49:10 +0000 (UTC)
Received: (qmail 49154 invoked by uid 500); 15 Jun 2011 02:49:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49127 invoked by uid 500); 15 Jun 2011 02:49:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49117 invoked by uid 99); 15 Jun 2011 02:49:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 02:49:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 02:49:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 61F8D4170CA
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 02:48:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 02:48:47 +0000 (UTC)
From: "Bochun Bai (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <550642264.5557.1308106127398.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6424) Port FsShell to FileContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bochun Bai updated HADOOP-6424:
-------------------------------

    Status: Patch Available  (was: Open)

> Port FsShell to FileContext 
> ----------------------------
>
>                 Key: HADOOP-6424
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6424
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: John George
>         Attachments: HADOOP-6424.patch, HADOOP-6424.patch
>
>
> FsShell currently uses FileSystem, needs to be moved over to FileContext.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16353-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 02:51:09 2011
Return-Path: <common-issues-return-16353-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B39744BC1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 02:51:09 +0000 (UTC)
Received: (qmail 49877 invoked by uid 500); 15 Jun 2011 02:51:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49778 invoked by uid 500); 15 Jun 2011 02:51:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49764 invoked by uid 99); 15 Jun 2011 02:51:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 02:51:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 02:51:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 785074171A3
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 02:50:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 02:50:47 +0000 (UTC)
From: "issei yoshida (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1501186456.5568.1308106247489.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

issei yoshida updated HADOOP-7206:
----------------------------------

    Attachment: HADOOP-7206-002.patch

update for the latest Hadoop trunk
and change the compression overhead.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16354-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 02:59:13 2011
Return-Path: <common-issues-return-16354-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 28B464DB9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 02:59:13 +0000 (UTC)
Received: (qmail 57407 invoked by uid 500); 15 Jun 2011 02:59:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57365 invoked by uid 500); 15 Jun 2011 02:59:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57346 invoked by uid 99); 15 Jun 2011 02:59:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 02:59:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 02:59:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 27FF2417397
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 02:58:48 +0000 (UTC)
Date: Wed, 15 Jun 2011 02:58:48 +0000 (UTC)
From: "issei yoshida (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1327126460.5619.1308106728160.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049586#comment-13049586 ] 

issei yoshida commented on HADOOP-7206:
---------------------------------------

I updated the patch for the latest Hadoop trunk.

I would like to hear your advice.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16355-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 04:22:11 2011
Return-Path: <common-issues-return-16355-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C10044DAD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 04:22:11 +0000 (UTC)
Received: (qmail 10491 invoked by uid 500); 15 Jun 2011 04:22:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10257 invoked by uid 500); 15 Jun 2011 04:22:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10249 invoked by uid 99); 15 Jun 2011 04:22:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 04:22:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 04:22:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9FD2541795D
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 04:21:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 04:21:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <391333450.5757.1308111707651.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049602#comment-13049602 ] 

Hadoop QA commented on HADOOP-7206:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482633/HADOOP-7206-002.patch
  against trunk revision 1135820.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.io.compress.TestCodec

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/634//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/634//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/634//console

This message is automatically generated.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16356-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 04:40:10 2011
Return-Path: <common-issues-return-16356-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 65A8F452C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 04:40:10 +0000 (UTC)
Received: (qmail 19470 invoked by uid 500); 15 Jun 2011 04:40:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19441 invoked by uid 500); 15 Jun 2011 04:40:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19426 invoked by uid 99); 15 Jun 2011 04:40:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 04:40:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 04:40:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5C0C9417D31
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 04:39:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 04:39:47 +0000 (UTC)
From: "Sudharsan Sampath (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <317000789.5820.1308112787373.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049604#comment-13049604 ] 

Sudharsan Sampath commented on HADOOP-7328:
-------------------------------------------

Just my thoughts...

To me throwing a specific exception would be better. The checkSerializerSpecs attempts to see if we can initialise a new instance of the serializer/deserializer from the jvm where the job is submitted. How does it guarantee that the libs/jars from which these classes are loaded are available for the jvm that executes the job or vice versa? Even if this methods succeeds isn't there a chance then the original problem might occur due to the libs missing from the actual jvm that executes the job?



> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff, HADOOP-7328.r2.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16357-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 05:48:13 2011
Return-Path: <common-issues-return-16357-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F2B954932
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 05:48:12 +0000 (UTC)
Received: (qmail 54061 invoked by uid 500); 15 Jun 2011 05:48:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53839 invoked by uid 500); 15 Jun 2011 05:48:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53818 invoked by uid 99); 15 Jun 2011 05:48:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 05:48:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 05:48:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 674FD417E16
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 05:47:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 05:47:47 +0000 (UTC)
From: "Yan Jinshuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <740424940.5878.1308116867420.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7395) The infomation returned by the wrong
 usage of the command "job -counter<job-id> <group-name> <counter-name> "is
 not appropriate
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

The infomation returned by the wrong usage of the command "job -counter<job-id> <group-name> <counter-name> "is not appropriate
-------------------------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-7395
                 URL: https://issues.apache.org/jira/browse/HADOOP-7395
             Project: Hadoop Common
          Issue Type: Improvement
    Affects Versions: 0.23.0
            Reporter: Yan Jinshuang
            Priority: Minor
             Fix For: 0.20.3


When use the command "job -counter<job-id> <group-name> <counter-name> " with wrong group-name or wrong counter-name(with correct job-id), the result is always 0. It is better to show the user the detail, like "illegal group-name", "illegal counter-name", etc. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16358-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 05:50:12 2011
Return-Path: <common-issues-return-16358-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3B5FB4A11
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 05:50:12 +0000 (UTC)
Received: (qmail 56027 invoked by uid 500); 15 Jun 2011 05:50:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55868 invoked by uid 500); 15 Jun 2011 05:50:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55748 invoked by uid 99); 15 Jun 2011 05:50:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 05:50:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 05:50:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5A70A417F05
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 05:49:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 05:49:47 +0000 (UTC)
From: "Yan Jinshuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1659840147.5880.1308116987367.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <740424940.5878.1308116867420.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7395) The infomation returned by the wrong
 usage of the command "job -counter<job-id> <group-name> <counter-name> "is
 not appropriate
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yan Jinshuang updated HADOOP-7395:
----------------------------------

    Fix Version/s:     (was: 0.20.3)
                   0.23.0

> The infomation returned by the wrong usage of the command "job -counter<job-id> <group-name> <counter-name> "is not appropriate
> -------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7395
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7395
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Yan Jinshuang
>            Priority: Minor
>             Fix For: 0.23.0
>
>
> When use the command "job -counter<job-id> <group-name> <counter-name> " with wrong group-name or wrong counter-name(with correct job-id), the result is always 0. It is better to show the user the detail, like "illegal group-name", "illegal counter-name", etc. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16359-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 07:54:15 2011
Return-Path: <common-issues-return-16359-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 217A946AD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 07:54:15 +0000 (UTC)
Received: (qmail 86271 invoked by uid 500); 15 Jun 2011 07:54:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86136 invoked by uid 500); 15 Jun 2011 07:54:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86092 invoked by uid 99); 15 Jun 2011 07:54:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 07:54:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 07:54:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9D60341747D
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 07:53:48 +0000 (UTC)
Date: Wed, 15 Jun 2011 07:53:48 +0000 (UTC)
From: "Yan Jinshuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1022068843.6096.1308124428641.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7396) The information returned by the
 wrong usage of the command "hadoop job -events <job-id> <from-event-#>
 <#-of-events>" is not appropriate
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

The information returned by the wrong usage of the command "hadoop job -events <job-id> <from-event-#> <#-of-events>" is not appropriate
----------------------------------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-7396
                 URL: https://issues.apache.org/jira/browse/HADOOP-7396
             Project: Hadoop Common
          Issue Type: Improvement
    Affects Versions: 0.23.0
            Reporter: Yan Jinshuang
            Priority: Minor
             Fix For: 0.23.0


With wrong value of from-event-# and #-of-events, though the from-events-# after the #-of-events for example from 1000 to 1, the command always return 0.It is expected to show detailed information, like "the start number should be less than the end number for range of events".

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16360-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 08:41:10 2011
Return-Path: <common-issues-return-16360-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D5C2041E6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 08:41:10 +0000 (UTC)
Received: (qmail 33155 invoked by uid 500); 15 Jun 2011 08:41:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33116 invoked by uid 500); 15 Jun 2011 08:41:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33108 invoked by uid 99); 15 Jun 2011 08:41:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 08:41:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 08:41:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6D1BA417998
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 08:40:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 08:40:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1220977770.6186.1308127247443.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1354187382.13188.1307764078831.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7380) Common portion of HDFS-1973
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7380:
-----------------------------------

    Attachment: hadoop-7380.1.patch

Updated patch which adds support for marking interface methods as being idempotent. Again, not intended for commit just yet.

I also updated the DFSClient failover test case to ensure it works with this annotation facility, which it does.

> Common portion of HDFS-1973
> ---------------------------
>
>                 Key: HADOOP-7380
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7380
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7380.0.patch, hadoop-7380.1.patch
>
>
> Implementing client failover will likely require changes to {{o.a.h.io.ipc}} and/or {{o.a.h.io.retry}}. This JIRA is to track those changes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16361-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 09:43:12 2011
Return-Path: <common-issues-return-16361-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4AB9A4EF2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 09:43:12 +0000 (UTC)
Received: (qmail 19413 invoked by uid 500); 15 Jun 2011 09:43:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19371 invoked by uid 500); 15 Jun 2011 09:43:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19361 invoked by uid 99); 15 Jun 2011 09:43:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 09:43:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 09:43:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E1BA4417C3A
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 09:42:48 +0000 (UTC)
Date: Wed, 15 Jun 2011 09:42:48 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <295795785.6343.1308130968921.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049696#comment-13049696 ] 

Hadoop QA commented on HADOOP-7206:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482633/HADOOP-7206-002.patch
  against trunk revision 1135820.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.io.compress.TestCodec

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/638//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/638//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/638//console

This message is automatically generated.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16362-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 15:56:11 2011
Return-Path: <common-issues-return-16362-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E4D364F3C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 15:56:10 +0000 (UTC)
Received: (qmail 80346 invoked by uid 500); 15 Jun 2011 15:56:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80311 invoked by uid 500); 15 Jun 2011 15:56:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80303 invoked by uid 99); 15 Jun 2011 15:56:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 15:56:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 15:56:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5FE994193B9
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 15:55:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 15:55:47 +0000 (UTC)
From: "Scott Fines (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <149464743.7145.1308153347389.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7397) Allow configurable timeouts when
 connecting to HDFS via java FileSystem API
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Allow configurable timeouts when connecting to HDFS via java FileSystem API
---------------------------------------------------------------------------

                 Key: HADOOP-7397
                 URL: https://issues.apache.org/jira/browse/HADOOP-7397
             Project: Hadoop Common
          Issue Type: Improvement
          Components: ipc
    Affects Versions: 0.20.2
         Environment: Any
            Reporter: Scott Fines
            Priority: Minor


If the NameNode is not available (in, for example, a network partition event separating the client from the NameNode), and an attempt is made to connect, then the FileSystem api will *eventually* timeout and throw an error. However, that timeout is currently hardcoded to be 20 seconds to connect, with 45 retries, for a total of a 15 minute wait before failure. While in many circumstances this is fine, there are also many circumstances (such as booting a service) where both the connection timeout and the number of retries should be significantly less, so as not to harm availability of other services.

Investigating Client.java, I see that there are two fields in Connection: maxRetries and rpcTimeout. I propose either re-using those fields for initiating the connection as well; alternatively, using the already existing dfs.socket.timeout parameter to set the connection timeout on initialization, and potentially adding a new field such as dfs.connection.retries with a default of 45 to replicate current behaviors.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16363-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 17:06:11 2011
Return-Path: <common-issues-return-16363-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 513CE41BB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 17:06:11 +0000 (UTC)
Received: (qmail 77281 invoked by uid 500); 15 Jun 2011 17:06:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77254 invoked by uid 500); 15 Jun 2011 17:06:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77245 invoked by uid 99); 15 Jun 2011 17:06:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:06:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:06:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F120C419D6E
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 17:05:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 17:05:47 +0000 (UTC)
From: "Mahadev konar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <912662184.7454.1308157547983.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6929) RPC should have a way to pass
 Security information other than protocol annotations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049884#comment-13049884 ] 

Mahadev konar commented on HADOOP-6929:
---------------------------------------

+1 the patch looks good. Ill run through some hdfs tests just to make sure everything is all fine and then commit.

> RPC should have a way to pass Security information other than protocol annotations
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6929
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6929
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: Hadoop-6929_v1.patch, h-6929.patch, h-6929.patch, h-6929.patch
>
>
> Currently Hadoop RPC allows protocol annotations as the only way to pass security information. This becomes a problem if protocols are generated and not hand written. For example protocols generated via Avro and passed over Avro tunnel (AvroRpcEngine.java) can't pass the security information.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16364-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 17:06:11 2011
Return-Path: <common-issues-return-16364-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7E06541CF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 17:06:11 +0000 (UTC)
Received: (qmail 77513 invoked by uid 500); 15 Jun 2011 17:06:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77439 invoked by uid 500); 15 Jun 2011 17:06:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77275 invoked by uid 99); 15 Jun 2011 17:06:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:06:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:06:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F2259419DB5
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 17:05:49 +0000 (UTC)
Date: Wed, 15 Jun 2011 17:05:49 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <399425902.7513.1308157549988.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6424) Port FsShell to FileContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049886#comment-13049886 ] 

Hadoop QA commented on HADOOP-6424:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482156/HADOOP-6424.patch
  against trunk revision 1135820.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/641//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/641//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/641//console

This message is automatically generated.

> Port FsShell to FileContext 
> ----------------------------
>
>                 Key: HADOOP-6424
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6424
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: John George
>         Attachments: HADOOP-6424.patch, HADOOP-6424.patch
>
>
> FsShell currently uses FileSystem, needs to be moved over to FileContext.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16365-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 17:06:11 2011
Return-Path: <common-issues-return-16365-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F156D41D9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 17:06:11 +0000 (UTC)
Received: (qmail 77806 invoked by uid 500); 15 Jun 2011 17:06:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77753 invoked by uid 500); 15 Jun 2011 17:06:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77508 invoked by uid 99); 15 Jun 2011 17:06:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:06:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:06:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 18A75419D73
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 17:05:48 +0000 (UTC)
Date: Wed, 15 Jun 2011 17:05:48 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1152975400.7458.1308157548097.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049885#comment-13049885 ] 

Hadoop QA commented on HADOOP-7206:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482633/HADOOP-7206-002.patch
  against trunk revision 1135820.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.io.compress.TestCodec

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/640//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/640//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/640//console

This message is automatically generated.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16366-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 17:10:12 2011
Return-Path: <common-issues-return-16366-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3E40E4415
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 17:10:12 +0000 (UTC)
Received: (qmail 89612 invoked by uid 500); 15 Jun 2011 17:10:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89584 invoked by uid 500); 15 Jun 2011 17:10:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89574 invoked by uid 99); 15 Jun 2011 17:10:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:10:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:10:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4C493419110
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 17:09:48 +0000 (UTC)
Date: Wed, 15 Jun 2011 17:09:48 +0000 (UTC)
From: "Jeffrey Naisbitt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <13372167.7547.1308157788308.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <467948518.30516.1305901367779.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7314) Add support for throwing
 UnknownHostException when a host doesn't resolve
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049891#comment-13049891 ] 

Jeffrey Naisbitt commented on HADOOP-7314:
------------------------------------------

I'm not sure why I don't see test-patch results, but here are the results from a manual run:

     [exec] +1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 6 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec] 
     [exec]     +1 system test framework.  The patch passed system test framework compile.

> Add support for throwing UnknownHostException when a host doesn't resolve
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-7314
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7314
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jeffrey Naisbitt
>            Assignee: Jeffrey Naisbitt
>         Attachments: HADOOP-7314.patch
>
>
> As part of MAPREDUCE-2489, we need support for having the resolve methods (for DNS mapping) throw UnknownHostExceptions.  (Currently, they hide the exception).  Since the existing 'resolve' method is ultimately used by several other locations/components, I propose we add a new 'resolveValidHosts' method.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16367-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 17:10:13 2011
Return-Path: <common-issues-return-16367-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2E96D4438
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 17:10:13 +0000 (UTC)
Received: (qmail 90082 invoked by uid 500); 15 Jun 2011 17:10:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90051 invoked by uid 500); 15 Jun 2011 17:10:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90043 invoked by uid 99); 15 Jun 2011 17:10:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:10:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:10:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8A51E419132
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 17:09:51 +0000 (UTC)
Date: Wed, 15 Jun 2011 17:09:51 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1764643332.7573.1308157791563.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7106) Re-organize hadoop subversion
 layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049892#comment-13049892 ] 

Hudson commented on HADOOP-7106:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk-Commit #728 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/728/])
    HADOOP-7106. Reorganize SVN layout to combine HDFS, Common, and MR in a single tree (project unsplit)

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1134994
Files : 
* /hadoop/common/branches/branch-0.22/.eclipse.templates
* /hadoop/common/branches/branch-0.21-old/common/lib
* /hadoop/common/branches/branch-0.21/LICENSE.txt
* /hadoop/common/branches/branch-0.22/common
* /hadoop/common/branches/branch-0.21-old/conf
* /hadoop/common/branches/yahoo-merge/mapreduce
* /hadoop/common/trunk/conf
* /hadoop/common/site/hdfs
* /hadoop/common/branches/branch-0.22/CHANGES.txt
* /hadoop/common/site/author
* /hadoop/common/branches/branch-0.22/common/ivy.xml
* /hadoop/common/branches/branch-0.21-old/common
* /hadoop/zookeeper
* /hadoop/common/branches/HDFS-1073/common
* /hadoop/pig
* /hadoop/common/branches/branch-0.21-old/common/ivy.xml
* /hadoop/common/trunk/README.txt
* /hadoop/common/branches/branch-0.21/NOTICE.txt
* /hadoop/common/branches/yahoo-merge/common/README.txt
* /hadoop/common/branches/branch-0.22/common/LICENSE.txt
* /hadoop/common/branches/yahoo-merge/conf
* /hadoop/common/branches/branch-0.21/common/lib
* /hadoop/common/trunk/lib
* /hadoop/common/trunk/common/.eclipse.templates
* /hadoop/common/branches/branch-0.21-old/common/CHANGES.txt
* /hadoop/common/tags/release-0.21.0/mapreduce
* /hadoop/common/branches/branch-0.22/mapreduce
* /hadoop/common/tags/release-0.21.0/lib
* /hadoop/common/branches/branch-0.21/common
* /hadoop/common/branches/HDFS-265/src/test
* /hadoop/site/build.xml
* /hadoop/common/branches/branch-0.21/common/.eclipse.templates
* /hadoop/common/branches/branch-0.22/common/conf
* /hadoop/common/branches/branch-0.21-old/mapreduce/src/test
* /hadoop/common/tags/release-0.21.0/README.txt
* /hadoop/common/branches/branch-0.21/common/ivy.xml
* /hadoop/common/branches/yahoo-merge/.gitignore
* /hadoop/common/site/common/build.xml
* /hadoop/common/trunk/build.xml
* /hadoop/common/tags/release-0.21.0/common/README.txt
* /hadoop/common/tags/release-0.21.0/ivy.xml
* /hadoop/common/branches/branch-0.22/common/README.txt
* /hadoop/common/branches/branch-0.21/build.xml
* /hadoop/common/branches/yahoo-merge/common/src
* /hadoop/common/trunk/NOTICE.txt
* /hadoop/common/branches/branch-0.21/common/CHANGES.txt
* /hadoop/common/site/main/build.xml
* /hadoop/common/branches/branch-0.22/conf
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/branches/branch-0.21-old/lib
* /hadoop/common/branches/yahoo-merge/common/ivy
* /hadoop/common/branches/HDFS-265
* /hadoop/common/branches/yahoo-merge/common/NOTICE.txt
* /hadoop/common/trunk/common/lib
* /hadoop/common/branches/branch-0.21/hdfs
* /hadoop/common/tags/release-0.21.0/hdfs/src/test
* /hadoop/common/branches/branch-0.21/mapreduce/src/test
* /hadoop/common/tags/release-0.21.0/CHANGES.txt
* /hadoop/common/branches/branch-0.22/src
* /hadoop/common/branches/yahoo-merge/lib
* /hadoop/common/branches/branch-0.22/README.txt
* /hadoop/common/site/common/author/src/documentation
* /hadoop/common/branches/branch-0.22/ivy
* /hadoop/common/branches/MAPREDUCE-233
* /hadoop/common/branches/branch-0.21-old/ivy.xml
* /hadoop/common/tags/release-0.21.0/NOTICE.txt
* /hadoop/common/branches/branch-0.21-old/common/.gitignore
* /hadoop/common/trunk/common/build.xml
* /hadoop/common/branches/branch-0.21-old/.gitignore
* /hadoop/common/branches/yahoo-merge/common/bin
* /hadoop/common/trunk/mapreduce
* /hadoop/common/branches/yahoo-merge/.eclipse.templates
* /hadoop/common/tags/release-0.21.0/common/lib
* /hadoop/common/tags/release-0.21.0/common/NOTICE.txt
* /hadoop/common/branches/branch-0.22/common/NOTICE.txt
* /hadoop/common/branches/branch-0.21-old/CHANGES.txt
* /hadoop/site/publish
* /hadoop/common/branches/branch-0.21-old/common/src
* /hadoop/common/trunk/common/conf
* /hadoop/common/branches/branch-0.21/mapreduce
* /hadoop/common/branches/yahoo-merge/common
* /hadoop/common/trunk/common/CHANGES.txt
* /hadoop/common/branches/branch-0.21/lib
* /hadoop/common/tags/release-0.21.0/common
* /hadoop/common/branches/HDFS-1052
* /hadoop/common/site/common/publish
* /hadoop/common/site/common
* /hadoop/common/branches/yahoo-merge/CHANGES.txt
* /hadoop/common/branches/yahoo-merge/common/ivy.xml
* /hadoop/common/tags/release-0.21.0/common/conf
* /hadoop/common/branches/branch-0.22/bin
* /hadoop/common/branches/branch-0.21-old/common/ivy
* /hadoop/common/branches/yahoo-merge/common/build.xml
* /hadoop/common/trunk/mapreduce/src/test
* /hadoop/common/tags/release-0.21.0/common/ivy.xml
* /hadoop/common/branches/HDFS-1073/hdfs
* /hadoop/common/trunk/common/README.txt
* /hadoop/common/site/publish
* /hadoop/common/branches/branch-0.21/common/build.xml
* /hadoop/common/site/hdfs/author/src/documentation
* /hadoop/common/branches/branch-0.21/.gitignore
* /hadoop/common/branches/branch-0.22/NOTICE.txt
* /hadoop/common/branches/branch-0.21/common/README.txt
* /hadoop/common/branches/branch-0.21/ivy.xml
* /hadoop/common/branches/branch-0.22/ivy.xml
* /hadoop/common/branches/yahoo-merge/common/LICENSE.txt
* /hadoop/common/branches/branch-0.21/common/src
* /hadoop/common/branches/HDFS-1073/hdfs/src/test
* /hadoop/common/branches/branch-0.22/common/lib
* /hadoop/common/tags/release-0.21.0/common/CHANGES.txt
* /hadoop/common/trunk/src
* /hadoop/common/site/main/author
* /hadoop/common/branches/branch-0.21/common/ivy
* /hadoop/common/branches/branch-0.21-old/common/.eclipse.templates
* /hadoop/common/branches/branch-0.21-old/common/conf
* /hadoop/common/branches/branch-0.21/hdfs/src/test
* /hadoop/common/tags/release-0.21.0/hdfs
* /hadoop/common/branches/branch-0.21-old/common/bin
* /hadoop/common/tags/release-0.21.0/src
* /hadoop/common/branches/branch-0.21-old/.eclipse.templates
* /hadoop/mapreduce
* /hadoop/common/branches/branch-0.22/LICENSE.txt
* /hadoop/common/tags/release-0.21.0/mapreduce/src/test
* /hadoop/common/trunk/ivy
* /hadoop/common/branches/branch-0.21/CHANGES.txt
* /hadoop/common/branches/HDFS-326
* /hadoop/hive
* /hadoop/common/branches/yahoo-merge/common/conf
* /hadoop/common/tags/release-0.21.0/ivy
* /hadoop/common/branches/branch-0.22/common/build.xml
* /hadoop/common/branches/branch-0.21-old/build.xml
* /hadoop/common/trunk/common/NOTICE.txt
* /hadoop/common/trunk/.gitignore
* /hadoop/common/branches/branch-0.22/common/CHANGES.txt
* /hadoop/common/branches/yahoo-merge/common/.gitignore
* /hadoop/common/branches/branch-0.21/common/bin
* /hadoop/common/branches/branch-0.21/.eclipse.templates
* /hadoop/common/branches/branch-0.21/conf
* /hadoop/common/branches/branch-0.21-old/src
* /hadoop/common/branches/branch-0.21/common/NOTICE.txt
* /hadoop/common/branches/branch-0.21-old/common/LICENSE.txt
* /hadoop/common/branches/HDFS-326/src/test
* /hadoop/common/trunk/bin
* /hadoop/common/trunk/common/src
* /hadoop/common/branches/branch-0.22/mapreduce/src/test
* /hadoop/common/branches/HDFS-1073
* /hadoop/common/branches/branch-0.21-old/ivy
* /hadoop/common/branches/yahoo-merge/README.txt
* /hadoop/common/tags/release-0.21.0/bin
* /hadoop/common/branches/yahoo-merge/src
* /hadoop/common/branches/branch-0.21/common/conf
* /hadoop/common/trunk/common/ivy
* /hadoop/common/branches/branch-0.21-old/hdfs
* /hadoop/common/trunk/hdfs
* /hadoop/common/branches/HDFS-1073/mapreduce/src/test
* /hadoop/common/tags/release-0.21.0/common/build.xml
* /hadoop/common/trunk/ivy.xml
* /hadoop/common/branches/yahoo-merge/ivy
* /hadoop/common/tags/release-0.21.0/.gitignore
* /hadoop/common/branches/MR-279/mapreduce
* /hadoop/common/site/build.xml
* /hadoop/common/branches/branch-0.21/common/LICENSE.txt
* /hadoop/common/tags/release-0.21.0/common/.gitignore
* /hadoop/common/tags/release-0.21.0/common/src
* /hadoop/common/branches/branch-0.22/common/.gitignore
* /hadoop/common/trunk/LICENSE.txt
* /hadoop/common/branches/branch-0.21-old/mapreduce
* /hadoop/common/site/common/publish/docs
* /hadoop/common/branches/MR-279
* /hadoop/common/branches/yahoo-merge/hdfs
* /hadoop/common/branches/branch-0.21-old/bin
* /hadoop/common/tags/release-0.21.0/common/ivy
* /hadoop/common/tags/release-0.21.0/LICENSE.txt
* /hadoop/common/branches/branch-0.21/src
* /hadoop/common/trunk/common/bin
* /hadoop/common/branches/branch-0.22/hdfs/src/test
* /hadoop/common/trunk/.eclipse.templates
* /hadoop/common/branches/yahoo-merge/bin
* /hadoop/common/branches/yahoo-merge/common/.eclipse.templates
* /hadoop/common/branches/branch-0.21/ivy
* /hadoop/common/trunk/common
* /hadoop/common/site/common/author
* /hadoop/common/branches/branch-0.21-old/common/README.txt
* /hadoop/common/branches/branch-0.22/.gitignore
* /hadoop/common/branches/branch-0.21-old/README.txt
* /hadoop/common/branches/yahoo-merge/NOTICE.txt
* /hadoop/common/trunk/common/ivy.xml
* /hadoop/common/branches/branch-0.21-old/common/build.xml
* /hadoop/common/branches/MR-279/hdfs/src/test
* /hadoop/common/branches/MR-279/common
* /hadoop/site/author
* /hadoop/common/branches/yahoo-merge/ivy.xml
* /hadoop/common/branches/MAPREDUCE-233/src/test
* /hadoop/common/branches/yahoo-merge/common/lib
* /hadoop/common/branches/branch-0.22/common/src
* /hadoop/common/branches/HDFS-1073/mapreduce
* /hadoop/common/branches/yahoo-merge/build.xml
* /hadoop/common/tags/release-0.21.0/common/bin
* /hadoop/common/branches/branch-0.21-old/LICENSE.txt
* /hadoop/common/branches/branch-0.21-old/hdfs/src/test
* /hadoop/common/tags/release-0.21.0/conf
* /hadoop/common/tags/release-0.21.0/.eclipse.templates
* /hadoop/common/branches/HDFS-1052/src/test
* /hadoop/common/site/mapreduce
* /hadoop/common/trunk/common/LICENSE.txt
* /hadoop/common/branches/branch-0.22/common/ivy
* /hadoop/hdfs
* /hadoop/common/branches/branch-0.22/hdfs
* /hadoop/common/branches/yahoo-merge/LICENSE.txt
* /hadoop/common/branches/branch-0.22/lib
* /hadoop/common/branches/branch-0.21/bin
* /hadoop/common/tags/release-0.21.0/common/.eclipse.templates
* /hadoop/common/branches/branch-0.22/common/.eclipse.templates
* /hadoop/common/branches/branch-0.21/README.txt
* /hadoop/common/trunk/common/.gitignore
* /hadoop/common/branches/MR-279/hdfs
* /hadoop/common/site/mapreduce/author/src/documentation
* /hadoop/common/branches/MR-279/mapreduce/src/test
* /hadoop/common/site/main/publish
* /hadoop/common/site/main
* /hadoop/common/tags/release-0.21.0/build.xml
* /hadoop/common/branches/branch-0.22/build.xml
* /hadoop/common/branches/yahoo-merge/common/CHANGES.txt
* /hadoop/common/branches/branch-0.21-old/common/NOTICE.txt
* /hadoop/common/tags/release-0.21.0/common/LICENSE.txt
* /hadoop/common/branches/branch-0.21-old/NOTICE.txt
* /hadoop/common/branches/branch-0.21/common/.gitignore
* /hadoop/common/branches/branch-0.22/common/bin
* /hadoop/common/trunk/hdfs/src/test


> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, gitk-example.png, mailer-conf.diff
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16368-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 17:10:15 2011
Return-Path: <common-issues-return-16368-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A1908445A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 17:10:15 +0000 (UTC)
Received: (qmail 91196 invoked by uid 500); 15 Jun 2011 17:10:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91135 invoked by uid 500); 15 Jun 2011 17:10:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91087 invoked by uid 99); 15 Jun 2011 17:10:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:10:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:10:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BDA35419136
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 17:09:51 +0000 (UTC)
Date: Wed, 15 Jun 2011 17:09:51 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1826709279.7577.1308157791773.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1507121853.1613.1307925711791.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7384) Allow test-patch to be more
 flexible about patch format
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049893#comment-13049893 ] 

Hudson commented on HADOOP-7384:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk-Commit #728 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/728/])
    Amend commit of HADOOP-7384 to include svn:executable bit on smart-apply-patch.sh
HADOOP-7384. Allow test-patch to be more flexible about patch format. Contributed by Todd Lipcon.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1135636
Files : 
* /hadoop/common/trunk/common/src/test/bin/smart-apply-patch.sh

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1135633
Files : 
* /hadoop/common/trunk/common/CHANGES.txt
* /hadoop/common/trunk/common/src/test/bin/smart-apply-patch.sh
* /hadoop/common/trunk/common/src/test/bin/test-patch.sh


> Allow test-patch to be more flexible about patch format
> -------------------------------------------------------
>
>                 Key: HADOOP-7384
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7384
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7384.txt
>
>
> Right now the test-patch process only accepts patches that are generated as "-p0" relative to common/, hdfs/, or mapreduce/. This has always been annoying for git users where the default patch format is -p1. It's also now annoying for SVN users who may generate a patch relative to trunk/ instead of the subproject subdirectory. We should auto-detect the correct patch level.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16369-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 17:14:10 2011
Return-Path: <common-issues-return-16369-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EB3D6456D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 17:14:10 +0000 (UTC)
Received: (qmail 97563 invoked by uid 500); 15 Jun 2011 17:14:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97536 invoked by uid 500); 15 Jun 2011 17:14:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97528 invoked by uid 99); 15 Jun 2011 17:14:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:14:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:14:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 757F1419313
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 17:13:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 17:13:47 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1436217384.7589.1308158027478.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-3436) Useless synchronized in JobTracker
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-3436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J resolved HADOOP-3436.
-----------------------------

    Resolution: Not A Problem

Does not appear to be a problem w.r.t. trunk. There is no such variable held (a collection is used instead, and that requires to hold JT lock and is synchronized (per comments)).

Resolving as "Not a problem" (anymore). Stale issue.

> Useless synchronized in JobTracker
> ----------------------------------
>
>                 Key: HADOOP-3436
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3436
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Brice Arnould
>            Assignee: Brice Arnould
>            Priority: Trivial
>
> In the original code, numTaskTrackers is fetch in a synchronized way, which is useless because anyway it might be change during the running of the algorithm.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16370-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 17:16:10 2011
Return-Path: <common-issues-return-16370-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8960A48A4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 17:16:10 +0000 (UTC)
Received: (qmail 1425 invoked by uid 500); 15 Jun 2011 17:16:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1387 invoked by uid 500); 15 Jun 2011 17:16:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1379 invoked by uid 99); 15 Jun 2011 17:16:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:16:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:16:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5D3B3419402
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 17:15:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 17:15:47 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <755648041.7591.1308158147378.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1250812071.19292.1305645587504.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7297) Error in the documentation regarding
 Checkpoint/Backup Node
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J updated HADOOP-7297:
----------------------------

    Fix Version/s: 0.20.203.1
           Status: Open  (was: Patch Available)

(Cancelling patch, needs more removals)

> Error in the documentation regarding Checkpoint/Backup Node
> -----------------------------------------------------------
>
>                 Key: HADOOP-7297
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7297
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.20.203.0
>            Reporter: arnaud p
>            Priority: Trivial
>             Fix For: 0.20.203.1
>
>         Attachments: hadoop-7297.patch
>
>
> On http://hadoop.apache.org/common/docs/r0.20.203.0/hdfs_user_guide.html#Checkpoint+Node: the command bin/hdfs namenode -checkpoint required to launch the backup/checkpoint node does not exist.
> I have removed this from the docs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16371-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 17:18:09 2011
Return-Path: <common-issues-return-16371-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EFA7442AF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 17:18:08 +0000 (UTC)
Received: (qmail 3244 invoked by uid 500); 15 Jun 2011 17:18:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3209 invoked by uid 500); 15 Jun 2011 17:18:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3196 invoked by uid 99); 15 Jun 2011 17:18:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:18:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:18:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9AF0F419587
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 17:17:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 17:17:47 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1876955507.7606.1308158267631.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1812341271.10485.1300405469738.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7198) Hadoop defaults for web UI ports
 often fall smack in the middle of Linux ephemeral port range
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049900#comment-13049900 ] 

Harsh J commented on HADOOP-7198:
---------------------------------

I guess we could make a documentation note at most for this.

Do you have any other ideas in mind?

> Hadoop defaults for web UI ports often fall smack in the middle of Linux ephemeral port range
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7198
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7198
>             Project: Hadoop Common
>          Issue Type: Wish
>            Reporter: Philip Zeyliger
>            Priority: Trivial
>
> It turns out (see http://en.wikipedia.org/wiki/Ephemeral_port and  /proc/sys/net/ipv4/ip_local_port_range) that when you bind to port 0, Linux chooses an ephemeral port.  On my default-ridden Ubuntu Maverick box and on CentOS 5.5, that range is 32768-61000.  So, when HBase binds to 60030 or when mapReduce binds to 50070, there's a small chance that you'll conflict with, say, an FTP session, or with some other Hadoop daemon that's had a listening address configured as :0.
> I don't know that there's a practical resolution here, since changing the defaults seems like an ill-fated effort, but if you have any ephemeral port use, you can run into this.  We've now run into it once.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16372-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 17:34:14 2011
Return-Path: <common-issues-return-16372-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5FC8544E8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 17:34:14 +0000 (UTC)
Received: (qmail 28546 invoked by uid 500); 15 Jun 2011 17:34:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28494 invoked by uid 500); 15 Jun 2011 17:34:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28477 invoked by uid 99); 15 Jun 2011 17:34:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:34:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:34:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AF9F9419C76
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 17:33:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 17:33:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <133442052.7638.1308159227716.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <149464743.7145.1308153347389.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7397) Allow configurable timeouts when
 connecting to HDFS via java FileSystem API
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7397:
-----------------------------------

    Affects Version/s: 0.23.0

This also appears to affect trunk.

> Allow configurable timeouts when connecting to HDFS via java FileSystem API
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-7397
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7397
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.20.2, 0.23.0
>         Environment: Any
>            Reporter: Scott Fines
>            Priority: Minor
>              Labels: hadoop
>
> If the NameNode is not available (in, for example, a network partition event separating the client from the NameNode), and an attempt is made to connect, then the FileSystem api will *eventually* timeout and throw an error. However, that timeout is currently hardcoded to be 20 seconds to connect, with 45 retries, for a total of a 15 minute wait before failure. While in many circumstances this is fine, there are also many circumstances (such as booting a service) where both the connection timeout and the number of retries should be significantly less, so as not to harm availability of other services.
> Investigating Client.java, I see that there are two fields in Connection: maxRetries and rpcTimeout. I propose either re-using those fields for initiating the connection as well; alternatively, using the already existing dfs.socket.timeout parameter to set the connection timeout on initialization, and potentially adding a new field such as dfs.connection.retries with a default of 45 to replicate current behaviors.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16373-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 17:38:11 2011
Return-Path: <common-issues-return-16373-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A0F994857
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 17:38:11 +0000 (UTC)
Received: (qmail 42791 invoked by uid 500); 15 Jun 2011 17:38:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42741 invoked by uid 500); 15 Jun 2011 17:38:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42731 invoked by uid 99); 15 Jun 2011 17:38:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:38:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 17:38:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E7601419E66
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 17:37:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 17:37:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <709933421.7669.1308159467944.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Resolved] (HADOOP-7106) Re-organize hadoop subversion
 layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon resolved HADOOP-7106.
---------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]

I think all the pieces of this are complete now, so marking resolved. Thanks to the many people who contributed: Nigel, Owen, Doug, Ian, Jukka, etc.

> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, gitk-example.png, mailer-conf.diff
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16374-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 18:31:09 2011
Return-Path: <common-issues-return-16374-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 132D14026
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 18:31:09 +0000 (UTC)
Received: (qmail 53660 invoked by uid 500); 15 Jun 2011 18:31:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53628 invoked by uid 500); 15 Jun 2011 18:31:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53620 invoked by uid 99); 15 Jun 2011 18:31:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 18:31:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 18:31:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BD96B4192A8
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 18:30:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 18:30:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1162241110.7961.1308162647772.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <151121036.11705.1307734619237.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7377) Fix command name handling
 affecting DFSAdmin
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049939#comment-13049939 ] 

Matt Foley commented on HADOOP-7377:
------------------------------------

This patch needs to be rebased for trunk.

> Fix command name handling affecting DFSAdmin
> --------------------------------------------
>
>                 Key: HADOOP-7377
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7377
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7377.patch
>
>
> When an error occurs in the get/set quota commands in DFSAdmin, they are displaying the following:
> setQuota: failed to get SetQuotaCommand.NAME
> The {{Command}} class expects the {{NAME}} field to be accessible, but for DFSAdmin, it's not.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16375-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 19:11:08 2011
Return-Path: <common-issues-return-16375-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C070C4BDA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 19:11:08 +0000 (UTC)
Received: (qmail 66588 invoked by uid 500); 15 Jun 2011 19:11:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66547 invoked by uid 500); 15 Jun 2011 19:11:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66538 invoked by uid 99); 15 Jun 2011 19:11:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 19:11:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 19:11:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5BDDA419AD3
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 19:10:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 19:10:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2076503973.8161.1308165047373.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1913857146.4454.1308088127661.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7393) PathData saves original string
 value, inviting failure when CWD changes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049966#comment-13049966 ] 

Matt Foley commented on HADOOP-7393:
------------------------------------

Option 2 seems like the only viable approach, then.

I'll agree that getDirectoryContents() works correctly in that regards.
getPathDataForChild() doesn't seem to.  Nor do I see general logic that invalidates relative paths when the cwd does change, except maybe in the relativize() method.

I also have to point out that (a) all the PathData i/f methods are public, not package-private; (b) code always gets re-used for purposes not expected by the author, and (c) the fact that set/getWorkingDirectory methods exist mean that you can't reasonably expect them to never be called between two calls to PathData references.

So will you humor me to the following extent?  Save 'cwd' at instantiation time, either as a Path or an absolute pathstring.  Provide a method 'String getOrigPathString()' which checks getWorkingDirectory against cwd, and returns 'string' if same, or something derived from 'cwd' + 'string' otherwise.  Then comment out the wazoo that 'string' must never be accessed directly, but always via the getOrigPathString() method.  And do it that way internally in all PathData methods too.

Would that be acceptable?  getWorkingDirectory() should only access in-memory data structures, so it shouldn't be a big cost.


> PathData saves original string value, inviting failure when CWD changes
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-7393
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7393
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Matt Foley
>
> PathData#string stores the pathstring originally used to construct the Path, and returns it from various methods, apparently in an attempt to improve the user experience for the shell.
> However, the current working directory may change, and if so this string value becomes meaningless and/or incorrect in context.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16376-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 19:18:09 2011
Return-Path: <common-issues-return-16376-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 45FA74992
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 19:18:09 +0000 (UTC)
Received: (qmail 79700 invoked by uid 500); 15 Jun 2011 19:18:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79661 invoked by uid 500); 15 Jun 2011 19:18:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79653 invoked by uid 99); 15 Jun 2011 19:18:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 19:18:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 19:18:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 67E4D419DBC
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 19:17:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 19:17:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1905403412.8171.1308165467422.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <92042485.70969.1307347427375.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7360) FsShell does not preserve relative
 paths with globs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049967#comment-13049967 ] 

Matt Foley commented on HADOOP-7360:
------------------------------------

bq. It would be a very bad idea to remove the method PathData(FileSystem, String, FileStatus) ... avoid unnecessary re-stat of the path ...

Understood.  As long as it's private, that's fine.  Please comment it as being for that use.

Ok to the rest of the responses.  Comments on how to make relativity safe are in HDFS-7393.  Thanks, Daryn.





> FsShell does not preserve relative paths with globs
> ---------------------------------------------------
>
>                 Key: HADOOP-7360
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7360
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7360.patch
>
>
> FsShell currently preserves relative paths that do not contain globs.  Unfortunately the method {{fs.globStatus()}} is fully qualifying all returned paths.  This is causing inconsistent display of paths.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16377-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 21:31:09 2011
Return-Path: <common-issues-return-16377-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EC58A4731
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 21:31:08 +0000 (UTC)
Received: (qmail 23622 invoked by uid 500); 15 Jun 2011 21:31:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23601 invoked by uid 500); 15 Jun 2011 21:31:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23593 invoked by uid 99); 15 Jun 2011 21:31:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 21:31:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 21:31:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A19DD419979
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 21:30:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 21:30:47 +0000 (UTC)
From: "Jeffrey Naisbitt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <419988067.8514.1308173447658.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <467948518.30516.1305901367779.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7314) Add support for throwing
 UnknownHostException when a host doesn't resolve
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jeffrey Naisbitt updated HADOOP-7314:
-------------------------------------

    Status: Open  (was: Patch Available)

Going to try using java.net.URI instead of UrlValidator to get rid of all the extra dependency stuff.

> Add support for throwing UnknownHostException when a host doesn't resolve
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-7314
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7314
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jeffrey Naisbitt
>            Assignee: Jeffrey Naisbitt
>         Attachments: HADOOP-7314.patch
>
>
> As part of MAPREDUCE-2489, we need support for having the resolve methods (for DNS mapping) throw UnknownHostExceptions.  (Currently, they hide the exception).  Since the existing 'resolve' method is ultimately used by several other locations/components, I propose we add a new 'resolveValidHosts' method.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16378-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 21:53:11 2011
Return-Path: <common-issues-return-16378-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2649246C6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 21:53:11 +0000 (UTC)
Received: (qmail 56300 invoked by uid 500); 15 Jun 2011 21:53:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56265 invoked by uid 500); 15 Jun 2011 21:53:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56257 invoked by uid 99); 15 Jun 2011 21:53:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 21:53:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 21:53:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CF5FE41AC12
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 21:52:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 21:52:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1488695307.8613.1308174767845.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <55510245.12688.1307747698852.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7379) Add ability to include Protobufs
 in ObjectWritable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050060#comment-13050060 ] 

Todd Lipcon commented on HADOOP-7379:
-------------------------------------

What do you guys think, is this a reasonable path?

> Add ability to include Protobufs in ObjectWritable
> --------------------------------------------------
>
>                 Key: HADOOP-7379
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7379
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, ipc
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7379.txt, hadoop-7379.txt
>
>
> Per HDFS-2060, it would make it easier to piecemeal switch to protocol buffer based data structures in the wire protocol if we could intermix the two. The IPC framework currently provides the concept of "engines" for RPC, but that doesn't easily allow mixed types within the same framework for ease of transition.
> I'd like to add the cases to ObjectWritable to be handle subclasses of {{Message}}, the superclass of codegenned protobufs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16379-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 22:13:09 2011
Return-Path: <common-issues-return-16379-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 27AD04099
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 22:13:09 +0000 (UTC)
Received: (qmail 81007 invoked by uid 500); 15 Jun 2011 22:13:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80967 invoked by uid 500); 15 Jun 2011 22:13:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80950 invoked by uid 99); 15 Jun 2011 22:13:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 22:13:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 22:13:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 96A4641A4D9
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 22:12:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 22:12:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1632789821.8685.1308175967613.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <55510245.12688.1307747698852.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7379) Add ability to include Protobufs
 in ObjectWritable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050075#comment-13050075 ] 

Luke Lu commented on HADOOP-7379:
---------------------------------

+1, lgtm.

> Add ability to include Protobufs in ObjectWritable
> --------------------------------------------------
>
>                 Key: HADOOP-7379
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7379
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, ipc
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7379.txt, hadoop-7379.txt
>
>
> Per HDFS-2060, it would make it easier to piecemeal switch to protocol buffer based data structures in the wire protocol if we could intermix the two. The IPC framework currently provides the concept of "engines" for RPC, but that doesn't easily allow mixed types within the same framework for ease of transition.
> I'd like to add the cases to ObjectWritable to be handle subclasses of {{Message}}, the superclass of codegenned protobufs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16380-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 22:19:09 2011
Return-Path: <common-issues-return-16380-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 20EA14421
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 22:19:09 +0000 (UTC)
Received: (qmail 88271 invoked by uid 500); 15 Jun 2011 22:19:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88243 invoked by uid 500); 15 Jun 2011 22:19:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88235 invoked by uid 99); 15 Jun 2011 22:19:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 22:19:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 22:19:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5B0BB41A663
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 22:18:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 22:18:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1185954194.8694.1308176327369.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1481744229.2736.1308062747508.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7390) VersionInfo not generated properly
 in git after unsplit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7390:
-----------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

+1 - worked like a charm.

I've just committed this. Thanks a lot, Todd.

> VersionInfo not generated properly in git after unsplit
> -------------------------------------------------------
>
>                 Key: HADOOP-7390
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7390
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Thomas Graves
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7390.txt
>
>
> The version information generated during the build of common when running from git has revision and branch Unknown. I believe this started after the unsplit:
> @HadoopVersionAnnotation(version="0.22.0-SNAPSHOT", revision="Unknown", branch="Unknown",
>                          user="tgraves", date="Tue Jun 14 13:39:10 UTC 2011", url="file:///home/tgraves/git/hadoop-common/common",
>                          srcChecksum="0f78ea668971fe51e7ebf4f97f84eed2")
> The ./src/saveVersion.sh script which generates the package-info.java file with the version info looks for the presence of .git directory and that is now a level up instead of in the common directory.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16381-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 22:23:09 2011
Return-Path: <common-issues-return-16381-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3E99C49DE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 22:23:09 +0000 (UTC)
Received: (qmail 94048 invoked by uid 500); 15 Jun 2011 22:23:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93946 invoked by uid 500); 15 Jun 2011 22:23:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93797 invoked by uid 99); 15 Jun 2011 22:23:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 22:23:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 22:23:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BBAA941A7D9
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 22:22:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 22:22:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <204819857.8708.1308176567765.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <55510245.12688.1307747698852.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7379) Add ability to include Protobufs in
 ObjectWritable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7379:
--------------------------------

      Resolution: Fixed
    Release Note: Protocol buffer-generated types may now be used as arguments or return values for Hadoop RPC.
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks for reviewing, Luke!

I haven't marked this as an incompatible change, since it simply adds a new feature to ObjectWritable without changing encoding of any existing features.

> Add ability to include Protobufs in ObjectWritable
> --------------------------------------------------
>
>                 Key: HADOOP-7379
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7379
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, ipc
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7379.txt, hadoop-7379.txt
>
>
> Per HDFS-2060, it would make it easier to piecemeal switch to protocol buffer based data structures in the wire protocol if we could intermix the two. The IPC framework currently provides the concept of "engines" for RPC, but that doesn't easily allow mixed types within the same framework for ease of transition.
> I'd like to add the cases to ObjectWritable to be handle subclasses of {{Message}}, the superclass of codegenned protobufs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16382-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 22:27:09 2011
Return-Path: <common-issues-return-16382-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 052334AB1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 22:27:09 +0000 (UTC)
Received: (qmail 194 invoked by uid 500); 15 Jun 2011 22:27:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 134 invoked by uid 500); 15 Jun 2011 22:27:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 125 invoked by uid 99); 15 Jun 2011 22:27:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 22:27:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 22:27:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C04E441A943
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 22:26:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 22:26:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1807388231.8723.1308176807784.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-6732) Improve FsShell's heap consumption
 by switching to listStatus that returns an iterator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp reassigned HADOOP-6732:
-----------------------------------

    Assignee: Daryn Sharp

> Improve FsShell's heap consumption by switching to listStatus that returns an iterator
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6732
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6732
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Hairong Kuang
>            Assignee: Daryn Sharp
>
> When listing a large directory from the command line using the default heap configuration, FsShell often runs out of memory. This is because all stats of the entries under the directory need to be in memory before printing them. The new API listStatus that returns an iterator of FileStatus, which implemented in HDFS-1091, no longer requires that all entries are fetched first. Thus switching to this new API will greatly improve the use of heap space.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16383-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 22:27:09 2011
Return-Path: <common-issues-return-16383-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 336464AB9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 22:27:09 +0000 (UTC)
Received: (qmail 364 invoked by uid 500); 15 Jun 2011 22:27:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 302 invoked by uid 500); 15 Jun 2011 22:27:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 159 invoked by uid 99); 15 Jun 2011 22:27:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 22:27:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 22:27:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CDA7341A945
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 22:26:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 22:26:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1323421461.8725.1308176807839.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <151121036.11705.1307734619237.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7377) Fix command name handling affecting
 DFSAdmin
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated HADOOP-7377:
-------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
           Status: Resolved  (was: Patch Available)

No it doesn't need to be rebased.

Committed to trunk.  Thanks, Daryn!

> Fix command name handling affecting DFSAdmin
> --------------------------------------------
>
>                 Key: HADOOP-7377
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7377
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7377.patch
>
>
> When an error occurs in the get/set quota commands in DFSAdmin, they are displaying the following:
> setQuota: failed to get SetQuotaCommand.NAME
> The {{Command}} class expects the {{NAME}} field to be accessible, but for DFSAdmin, it's not.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16384-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 22:29:11 2011
Return-Path: <common-issues-return-16384-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 03C7B4B0B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 22:29:11 +0000 (UTC)
Received: (qmail 3617 invoked by uid 500); 15 Jun 2011 22:29:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3581 invoked by uid 500); 15 Jun 2011 22:29:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3573 invoked by uid 99); 15 Jun 2011 22:29:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 22:29:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 22:29:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 56FC241AA07
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 22:28:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 22:28:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1289206768.8728.1308176927353.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6732) Improve FsShell's heap consumption
 by switching to listStatus that returns an iterator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050084#comment-13050084 ] 

Daryn Sharp commented on HADOOP-6732:
-------------------------------------

The PathData changes to FsShell have reduced the RPC load on the NN, but at the expense of a few more objects per path.  Adding iterators will negate the additional memory pressure.

> Improve FsShell's heap consumption by switching to listStatus that returns an iterator
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6732
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6732
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Hairong Kuang
>            Assignee: Daryn Sharp
>
> When listing a large directory from the command line using the default heap configuration, FsShell often runs out of memory. This is because all stats of the entries under the directory need to be in memory before printing them. The new API listStatus that returns an iterator of FileStatus, which implemented in HDFS-1091, no longer requires that all entries are fetched first. Thus switching to this new API will greatly improve the use of heap space.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16385-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 22:37:11 2011
Return-Path: <common-issues-return-16385-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7F6494221
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 22:37:11 +0000 (UTC)
Received: (qmail 11510 invoked by uid 500); 15 Jun 2011 22:37:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11436 invoked by uid 500); 15 Jun 2011 22:37:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11420 invoked by uid 99); 15 Jun 2011 22:37:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 22:37:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 22:37:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 625C441ADAA
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 22:36:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 22:36:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <738025072.8753.1308177407399.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1481744229.2736.1308062747508.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7390) VersionInfo not generated properly
 in git after unsplit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050087#comment-13050087 ] 

Hudson commented on HADOOP-7390:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #656 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/656/])
    HADOOP-7390. VersionInfo not generated properly in git after unsplit. (todd via atm)

atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1136220
Files : 
* /hadoop/common/trunk/common/src/saveVersion.sh
* /hadoop/common/trunk/common/CHANGES.txt


> VersionInfo not generated properly in git after unsplit
> -------------------------------------------------------
>
>                 Key: HADOOP-7390
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7390
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Thomas Graves
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7390.txt
>
>
> The version information generated during the build of common when running from git has revision and branch Unknown. I believe this started after the unsplit:
> @HadoopVersionAnnotation(version="0.22.0-SNAPSHOT", revision="Unknown", branch="Unknown",
>                          user="tgraves", date="Tue Jun 14 13:39:10 UTC 2011", url="file:///home/tgraves/git/hadoop-common/common",
>                          srcChecksum="0f78ea668971fe51e7ebf4f97f84eed2")
> The ./src/saveVersion.sh script which generates the package-info.java file with the version info looks for the presence of .git directory and that is now a level up instead of in the common directory.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16386-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 22:39:12 2011
Return-Path: <common-issues-return-16386-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5965942BC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 22:39:12 +0000 (UTC)
Received: (qmail 14911 invoked by uid 500); 15 Jun 2011 22:39:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14824 invoked by uid 500); 15 Jun 2011 22:39:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14807 invoked by uid 99); 15 Jun 2011 22:39:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 22:39:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 22:39:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 569E041AE79
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 22:38:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 22:38:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1293648970.8758.1308177527351.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7398) create a mechanism to suppress the
 HADOOP_HOME deprecated warning
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

create a mechanism to suppress the HADOOP_HOME deprecated warning
-----------------------------------------------------------------

                 Key: HADOOP-7398
                 URL: https://issues.apache.org/jira/browse/HADOOP-7398
             Project: Hadoop Common
          Issue Type: New Feature
            Reporter: Owen O'Malley
            Assignee: Owen O'Malley


Create a new mechanism to suppress the warning about HADOOP_HOME deprecation.

I'll create a HADOOP_HOME_WARN_SUPPRESS environment variable that suppresses the warning.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16387-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 22:51:09 2011
Return-Path: <common-issues-return-16387-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4F1CA4C4E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 22:51:09 +0000 (UTC)
Received: (qmail 42130 invoked by uid 500); 15 Jun 2011 22:51:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42046 invoked by uid 500); 15 Jun 2011 22:51:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42038 invoked by uid 99); 15 Jun 2011 22:51:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 22:51:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 22:51:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B14CA41A554
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 22:50:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 22:50:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1556987777.8912.1308178247723.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <151121036.11705.1307734619237.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7377) Fix command name handling
 affecting DFSAdmin
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050100#comment-13050100 ] 

Hudson commented on HADOOP-7377:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #657 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/657/])
    HADOOP-7377. Fix command name handling affecting DFSAdmin. Contributed by Daryn Sharp.

mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1136223
Files : 
* /hadoop/common/trunk/common/src/java/org/apache/hadoop/fs/shell/Command.java
* /hadoop/common/trunk/common/CHANGES.txt


> Fix command name handling affecting DFSAdmin
> --------------------------------------------
>
>                 Key: HADOOP-7377
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7377
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7377.patch
>
>
> When an error occurs in the get/set quota commands in DFSAdmin, they are displaying the following:
> setQuota: failed to get SetQuotaCommand.NAME
> The {{Command}} class expects the {{NAME}} field to be accessible, but for DFSAdmin, it's not.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16388-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 22:51:11 2011
Return-Path: <common-issues-return-16388-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 07D194C67
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 22:51:11 +0000 (UTC)
Received: (qmail 42394 invoked by uid 500); 15 Jun 2011 22:51:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42355 invoked by uid 500); 15 Jun 2011 22:51:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42347 invoked by uid 99); 15 Jun 2011 22:51:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 22:51:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 22:51:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9CF9841A550
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 22:50:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 22:50:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <700187311.8909.1308178247639.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <55510245.12688.1307747698852.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7379) Add ability to include Protobufs
 in ObjectWritable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050099#comment-13050099 ] 

Hudson commented on HADOOP-7379:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #657 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/657/])
    HADOOP-7379. Add the ability to serialize and deserialize protocol buffers in ObjectWritable. Contributed by Todd Lipcon.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1136222
Files : 
* /hadoop/common/trunk/common/src/java/org/apache/hadoop/io/DataOutputOutputStream.java
* /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/ipc/TestRPC.java
* /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/util/TestProtoUtil.java
* /hadoop/common/trunk/common/CHANGES.txt
* /hadoop/common/trunk/common/src/java/org/apache/hadoop/util/ProtoUtil.java
* /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/TestObjectWritableProtos.java
* /hadoop/common/trunk/common/src/java/org/apache/hadoop/io/ObjectWritable.java


> Add ability to include Protobufs in ObjectWritable
> --------------------------------------------------
>
>                 Key: HADOOP-7379
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7379
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, ipc
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7379.txt, hadoop-7379.txt
>
>
> Per HDFS-2060, it would make it easier to piecemeal switch to protocol buffer based data structures in the wire protocol if we could intermix the two. The IPC framework currently provides the concept of "engines" for RPC, but that doesn't easily allow mixed types within the same framework for ease of transition.
> I'd like to add the cases to ObjectWritable to be handle subclasses of {{Message}}, the superclass of codegenned protobufs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16389-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 23:05:12 2011
Return-Path: <common-issues-return-16389-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 05EDB458C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 23:05:12 +0000 (UTC)
Received: (qmail 55863 invoked by uid 500); 15 Jun 2011 23:05:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55823 invoked by uid 500); 15 Jun 2011 23:05:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55815 invoked by uid 99); 15 Jun 2011 23:05:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 23:05:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 23:05:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8CCBD41AEA9
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 23:04:48 +0000 (UTC)
Date: Wed, 15 Jun 2011 23:04:48 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <236374698.8931.1308179088573.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1354187382.13188.1307764078831.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7380) Common portion of HDFS-1973
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050104#comment-13050104 ] 

Aaron T. Myers commented on HADOOP-7380:
----------------------------------------

Another open issue which needs to be addressed is synchronization around the {{FailoverProxyProvider}} on failover. This will be necessary since multiple threads may be accessing a single RPC proxy, and we don't want this to effectively cause the client to failover multiple times spuriously.

I'll deal with this once I get confirmation that the general approach is sound.

> Common portion of HDFS-1973
> ---------------------------
>
>                 Key: HADOOP-7380
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7380
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7380.0.patch, hadoop-7380.1.patch
>
>
> Implementing client failover will likely require changes to {{o.a.h.io.ipc}} and/or {{o.a.h.io.retry}}. This JIRA is to track those changes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16390-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 23:05:55 2011
Return-Path: <common-issues-return-16390-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 11E9C47D1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 23:05:54 +0000 (UTC)
Received: (qmail 60958 invoked by uid 500); 15 Jun 2011 23:05:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60884 invoked by uid 500); 15 Jun 2011 23:05:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60836 invoked by uid 99); 15 Jun 2011 23:05:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 23:05:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 23:05:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CA53541AF64
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 23:04:53 +0000 (UTC)
Date: Wed, 15 Jun 2011 23:04:53 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <455199615.9096.1308179093825.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7106) Re-organize hadoop subversion
 layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050119#comment-13050119 ] 

Hudson commented on HADOOP-7106:
--------------------------------

Integrated in Hadoop-Hdfs-trunk-Commit #746 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/746/])
    

> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, gitk-example.png, mailer-conf.diff
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16391-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 23:05:58 2011
Return-Path: <common-issues-return-16391-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1E1F04865
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 23:05:58 +0000 (UTC)
Received: (qmail 62708 invoked by uid 500); 15 Jun 2011 23:05:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62665 invoked by uid 500); 15 Jun 2011 23:05:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62541 invoked by uid 99); 15 Jun 2011 23:05:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 23:05:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 23:05:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 618A841AF79
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 23:04:54 +0000 (UTC)
Date: Wed, 15 Jun 2011 23:04:54 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1684030060.9116.1308179094396.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1507121853.1613.1307925711791.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7384) Allow test-patch to be more
 flexible about patch format
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050124#comment-13050124 ] 

Hudson commented on HADOOP-7384:
--------------------------------

Integrated in Hadoop-Hdfs-trunk-Commit #746 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/746/])
    

> Allow test-patch to be more flexible about patch format
> -------------------------------------------------------
>
>                 Key: HADOOP-7384
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7384
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7384.txt
>
>
> Right now the test-patch process only accepts patches that are generated as "-p0" relative to common/, hdfs/, or mapreduce/. This has always been annoying for git users where the default patch format is -p1. It's also now annoying for SVN users who may generate a patch relative to trunk/ instead of the subproject subdirectory. We should auto-detect the correct patch level.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16392-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 23:33:08 2011
Return-Path: <common-issues-return-16392-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9CD0644EC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 23:33:08 +0000 (UTC)
Received: (qmail 6598 invoked by uid 500); 15 Jun 2011 23:33:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6572 invoked by uid 500); 15 Jun 2011 23:33:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6564 invoked by uid 99); 15 Jun 2011 23:33:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 23:33:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 23:33:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 542CB41A7FD
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 23:32:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 23:32:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2100048801.9251.1308180767341.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1293648970.8758.1308177527351.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7398) create a mechanism to suppress the
 HADOOP_HOME deprecated warning
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7398:
----------------------------------

    Attachment: warn.patch

setting HADOOP_HOME_WARN_SUPPRESS to anything suppresses the warning.

> create a mechanism to suppress the HADOOP_HOME deprecated warning
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7398
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7398
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: warn.patch
>
>
> Create a new mechanism to suppress the warning about HADOOP_HOME deprecation.
> I'll create a HADOOP_HOME_WARN_SUPPRESS environment variable that suppresses the warning.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16393-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 23:35:08 2011
Return-Path: <common-issues-return-16393-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8CBD14B65
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 23:35:08 +0000 (UTC)
Received: (qmail 9408 invoked by uid 500); 15 Jun 2011 23:35:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9368 invoked by uid 500); 15 Jun 2011 23:35:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9355 invoked by uid 99); 15 Jun 2011 23:35:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 23:35:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 23:35:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 54F3F41A8AE
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 23:34:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 23:34:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1772693893.9253.1308180887344.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1293648970.8758.1308177527351.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7398) create a mechanism to suppress the
 HADOOP_HOME deprecated warning
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050143#comment-13050143 ] 

Luke Lu commented on HADOOP-7398:
---------------------------------

+1. lgtm.

> create a mechanism to suppress the HADOOP_HOME deprecated warning
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7398
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7398
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: warn.patch
>
>
> Create a new mechanism to suppress the warning about HADOOP_HOME deprecation.
> I'll create a HADOOP_HOME_WARN_SUPPRESS environment variable that suppresses the warning.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16394-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 23:41:16 2011
Return-Path: <common-issues-return-16394-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 91D5C4C30
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 23:41:16 +0000 (UTC)
Received: (qmail 15400 invoked by uid 500); 15 Jun 2011 23:41:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15362 invoked by uid 500); 15 Jun 2011 23:41:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15352 invoked by uid 99); 15 Jun 2011 23:41:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 23:41:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 23:41:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CAF5341AA95
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 23:40:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 23:40:47 +0000 (UTC)
From: "Mahadev konar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1991491254.9273.1308181247827.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6929) RPC should have a way to pass
 Security information other than protocol annotations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mahadev konar updated HADOOP-6929:
----------------------------------

    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]

I just committed this to trunk. Thanks Sharad and Owen!

> RPC should have a way to pass Security information other than protocol annotations
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6929
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6929
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>             Fix For: 0.23.0
>
>         Attachments: Hadoop-6929_v1.patch, h-6929.patch, h-6929.patch, h-6929.patch
>
>
> Currently Hadoop RPC allows protocol annotations as the only way to pass security information. This becomes a problem if protocols are generated and not hand written. For example protocols generated via Avro and passed over Avro tunnel (AvroRpcEngine.java) can't pass the security information.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16395-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 15 23:41:18 2011
Return-Path: <common-issues-return-16395-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A8E984C41
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 15 Jun 2011 23:41:18 +0000 (UTC)
Received: (qmail 15685 invoked by uid 500); 15 Jun 2011 23:41:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15652 invoked by uid 500); 15 Jun 2011 23:41:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15642 invoked by uid 99); 15 Jun 2011 23:41:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 23:41:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2011 23:41:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D888241AA98
	for <common-issues@hadoop.apache.org>; Wed, 15 Jun 2011 23:40:47 +0000 (UTC)
Date: Wed, 15 Jun 2011 23:40:47 +0000 (UTC)
From: "Mahadev konar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1412229208.9275.1308181247883.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6929) RPC should have a way to pass
 Security information other than protocol annotations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mahadev konar updated HADOOP-6929:
----------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

> RPC should have a way to pass Security information other than protocol annotations
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6929
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6929
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>             Fix For: 0.23.0
>
>         Attachments: Hadoop-6929_v1.patch, h-6929.patch, h-6929.patch, h-6929.patch
>
>
> Currently Hadoop RPC allows protocol annotations as the only way to pass security information. This becomes a problem if protocols are generated and not hand written. For example protocols generated via Avro and passed over Avro tunnel (AvroRpcEngine.java) can't pass the security information.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16396-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 00:47:09 2011
Return-Path: <common-issues-return-16396-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 93E7E4CEB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 00:47:09 +0000 (UTC)
Received: (qmail 81649 invoked by uid 500); 16 Jun 2011 00:47:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81594 invoked by uid 500); 16 Jun 2011 00:47:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81574 invoked by uid 99); 16 Jun 2011 00:47:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 00:47:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 00:47:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D312841ABC9
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 00:46:47 +0000 (UTC)
Date: Thu, 16 Jun 2011 00:46:47 +0000 (UTC)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1760538336.9387.1308185207859.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1250812071.19292.1305645587504.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7297) Error in the documentation
 regarding Checkpoint/Backup Node
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7297?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan reassigned HADOOP-7297:
-----------------------------------

    Assignee: Christoph B=C3=B6hm

> Error in the documentation regarding Checkpoint/Backup Node
> -----------------------------------------------------------
>
>                 Key: HADOOP-7297
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7297
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.20.203.0
>            Reporter: arnaud p
>            Assignee: Christoph B=C3=B6hm
>            Priority: Trivial
>             Fix For: 0.20.203.1
>
>         Attachments: hadoop-7297.patch
>
>
> On http://hadoop.apache.org/common/docs/r0.20.203.0/hdfs_user_guide.html#=
Checkpoint+Node: the command bin/hdfs namenode -checkpoint required to laun=
ch the backup/checkpoint node does not exist.
> I have removed this from the docs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

From common-issues-return-16397-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 00:48:12 2011
Return-Path: <common-issues-return-16397-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9737842F3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 00:48:12 +0000 (UTC)
Received: (qmail 83262 invoked by uid 500); 16 Jun 2011 00:48:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83191 invoked by uid 500); 16 Jun 2011 00:48:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83162 invoked by uid 99); 16 Jun 2011 00:48:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 00:48:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 00:48:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 207AB41AC67
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 00:47:49 +0000 (UTC)
Date: Thu, 16 Jun 2011 00:47:49 +0000 (UTC)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1391931477.9405.1308185269129.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1250812071.19292.1305645587504.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7297) Error in the documentation
 regarding Checkpoint/Backup Node
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7297?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D130=
50162#comment-13050162 ]=20

Jakob Homan commented on HADOOP-7297:
-------------------------------------

Christoph - can you remove the rest of the references to backup and checkpo=
int node? I'll manually test the patch since it's a bit of a pain without J=
enkins.

Thanks. -jg

> Error in the documentation regarding Checkpoint/Backup Node
> -----------------------------------------------------------
>
>                 Key: HADOOP-7297
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7297
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.20.203.0
>            Reporter: arnaud p
>            Assignee: Christoph B=C3=B6hm
>            Priority: Trivial
>             Fix For: 0.20.203.1
>
>         Attachments: hadoop-7297.patch
>
>
> On http://hadoop.apache.org/common/docs/r0.20.203.0/hdfs_user_guide.html#=
Checkpoint+Node: the command bin/hdfs namenode -checkpoint required to laun=
ch the backup/checkpoint node does not exist.
> I have removed this from the docs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

From common-issues-return-16398-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 00:52:10 2011
Return-Path: <common-issues-return-16398-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 97C4844E5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 00:52:10 +0000 (UTC)
Received: (qmail 93941 invoked by uid 500); 16 Jun 2011 00:52:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93913 invoked by uid 500); 16 Jun 2011 00:52:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93905 invoked by uid 99); 16 Jun 2011 00:52:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 00:52:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 00:52:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4BF6641AEEB
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 00:51:49 +0000 (UTC)
Date: Thu, 16 Jun 2011 00:51:49 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <611880176.9469.1308185509307.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050167#comment-13050167 ] 

Alejandro Abdelnur commented on HADOOP-7206:
--------------------------------------------

The test is failing because snappy SO is not installed/available in the build machine.

Options that I see for this are:

1* Install snappy in build machines
2* Drag into hadoop native code snappy code 

While #2 is simpler it means we are forking snappy for Hadoop usage. We could do this until snappy is available as a package (RPM/DEB).

Thoughts?


> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16399-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 00:56:11 2011
Return-Path: <common-issues-return-16399-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 22D9745EF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 00:56:11 +0000 (UTC)
Received: (qmail 99962 invoked by uid 500); 16 Jun 2011 00:56:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99916 invoked by uid 500); 16 Jun 2011 00:56:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99901 invoked by uid 99); 16 Jun 2011 00:56:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 00:56:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 00:56:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7FA5E41A099
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 00:55:47 +0000 (UTC)
Date: Thu, 16 Jun 2011 00:55:47 +0000 (UTC)
From: "T Jake Luciani (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1237229405.9477.1308185747519.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050171#comment-13050171 ] 

T Jake Luciani commented on HADOOP-7206:
----------------------------------------

this is why the snappy-java approach works well IMO.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16400-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 01:46:10 2011
Return-Path: <common-issues-return-16400-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B92694616
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 01:46:10 +0000 (UTC)
Received: (qmail 38166 invoked by uid 500); 16 Jun 2011 01:46:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38140 invoked by uid 500); 16 Jun 2011 01:46:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38132 invoked by uid 99); 16 Jun 2011 01:46:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 01:46:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 01:46:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 86C9041ABC7
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 01:45:47 +0000 (UTC)
Date: Thu, 16 Jun 2011 01:45:47 +0000 (UTC)
From: "Rajiv Chittajallu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1801929813.9574.1308188747548.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1293648970.8758.1308177527351.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7398) create a mechanism to suppress the
 HADOOP_HOME deprecated warning
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050178#comment-13050178 ] 

Rajiv Chittajallu commented on HADOOP-7398:
-------------------------------------------

Warning should go to stderr. If this env is not used outside of bin/hadoop, unset it.

> create a mechanism to suppress the HADOOP_HOME deprecated warning
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7398
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7398
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: warn.patch
>
>
> Create a new mechanism to suppress the warning about HADOOP_HOME deprecation.
> I'll create a HADOOP_HOME_WARN_SUPPRESS environment variable that suppresses the warning.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16401-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 07:03:14 2011
Return-Path: <common-issues-return-16401-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 15B576337
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 07:03:14 +0000 (UTC)
Received: (qmail 9863 invoked by uid 500); 16 Jun 2011 07:03:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9444 invoked by uid 500); 16 Jun 2011 07:03:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8627 invoked by uid 99); 16 Jun 2011 07:03:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 07:03:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 07:03:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A369A41B9EE
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 07:02:47 +0000 (UTC)
Date: Thu, 16 Jun 2011 07:02:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <307925084.10027.1308207767666.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1928332496.4012.1308081047555.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7391) Copy the interrface classification
 documentation of HADOOP-5073 into javadoc
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7391:
---------------------------------

    Attachment: Hadoop Interface Taxonomy.html

I have copied the interface classification stuff from HADOOP-5073 in the attached html.
I plan to add this Overview.html in the package o.a.h.classification.
I have added a couple of FAQs and minor edits but the text is mostly unchanged from HADOOP-5073

> Copy the interrface classification documentation of HADOOP-5073 into javadoc
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-7391
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7391
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.22.0
>
>         Attachments: Hadoop Interface Taxonomy.html
>
>
> The documentation for interface classification in Jira Hadoop-5073 was not copied to the Javadoc
> of the classification.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16402-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 08:16:13 2011
Return-Path: <common-issues-return-16402-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8643C65AD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 08:16:13 +0000 (UTC)
Received: (qmail 20286 invoked by uid 500); 16 Jun 2011 08:16:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20156 invoked by uid 500); 16 Jun 2011 08:16:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19906 invoked by uid 99); 16 Jun 2011 08:16:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 08:16:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 08:16:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4A89E41BC60
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 08:15:49 +0000 (UTC)
Date: Thu, 16 Jun 2011 08:15:49 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1593248389.10179.1308212149301.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7399) Remove Writable from ipc.Client and
 ipc.Server.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Remove Writable from ipc.Client and ipc.Server.
-----------------------------------------------

                 Key: HADOOP-7399
                 URL: https://issues.apache.org/jira/browse/HADOOP-7399
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Jitendra Nath Pandey
            Assignee: Jitendra Nath Pandey


  This jira proposes to remove the dependency of ipc client and server on Writable. I think this will be an important step towards making an RpcEngine truly configurable, without having to use tunnel protocols as in case of AvroRPCEngine.



--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16403-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 09:38:10 2011
Return-Path: <common-issues-return-16403-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 873B1628F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 09:38:10 +0000 (UTC)
Received: (qmail 12210 invoked by uid 500); 16 Jun 2011 09:38:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12165 invoked by uid 500); 16 Jun 2011 09:38:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12157 invoked by uid 99); 16 Jun 2011 09:38:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 09:38:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 09:38:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2F0BD41B11C
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 09:37:49 +0000 (UTC)
Date: Thu, 16 Jun 2011 09:37:49 +0000 (UTC)
From: "issei yoshida (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <714673755.10474.1308217069189.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050313#comment-13050313 ] 

issei yoshida commented on HADOOP-7206:
---------------------------------------

Alejandro,

How do we drag Snappy code into Hadoop trunk?
May we drag the Snappy svn repository of the google code as svn externals?
I think that it'd make Hadoop trunk more complex.

IMO, #1 is simpler approarch.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16404-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 11:48:09 2011
Return-Path: <common-issues-return-16404-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E9BF36B39
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 11:48:08 +0000 (UTC)
Received: (qmail 78990 invoked by uid 500); 16 Jun 2011 11:48:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78867 invoked by uid 500); 16 Jun 2011 11:48:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78858 invoked by uid 99); 16 Jun 2011 11:48:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 11:48:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 11:48:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6971341B46D
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 11:47:47 +0000 (UTC)
Date: Thu, 16 Jun 2011 11:47:47 +0000 (UTC)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1633629904.10678.1308224867428.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050360#comment-13050360 ] 

Doug Cutting commented on HADOOP-7206:
--------------------------------------

What are the downsides of building on snappy-java?  It seems to be updated frequently, is published in Maven, and would avoid loader conflicts with other Java projects that use snappy-java.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16405-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 12:14:12 2011
Return-Path: <common-issues-return-16405-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 956106B78
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 12:14:12 +0000 (UTC)
Received: (qmail 34392 invoked by uid 500); 16 Jun 2011 12:14:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34346 invoked by uid 500); 16 Jun 2011 12:14:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34333 invoked by uid 99); 16 Jun 2011 12:14:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 12:14:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 12:14:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1DE2641B126
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 12:13:49 +0000 (UTC)
Date: Thu, 16 Jun 2011 12:13:49 +0000 (UTC)
From: "jiraposter@reviews.apache.org (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <458150255.10772.1308226429119.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050373#comment-13050373 ] 

jiraposter@reviews.apache.org commented on HADOOP-7328:
-------------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/884/
-----------------------------------------------------------

(Updated 2011-06-16 12:13:34.081758)


Review request for hadoop-common and Todd Lipcon.


Changes
-------

Throw exceptions (getting rid of nulls). Add appropriate javadocs and fix one checkstyle nit.


Summary
-------

Since getSerialization() can possibly return a null, it is only right that getSerializer() and getDeserializer() usage functions do the same, instead of throwing up NPEs.

Related issue to which this improvement is required: https://issues.apache.org/jira/browse/MAPREDUCE-2584


This addresses bug HADOOP-7328.
    http://issues.apache.org/jira/browse/HADOOP-7328


Diffs (updated)
-----

  src/java/org/apache/hadoop/io/serializer/SerializationFactory.java dee314a 

Diff: https://reviews.apache.org/r/884/diff


Testing
-------

Existing SequenceFile serialization factory tests pass. The change is merely to make the functions return null instead of throwing an NPE within.


Thanks,

Harsh



> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff, HADOOP-7328.r2.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16406-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 12:16:08 2011
Return-Path: <common-issues-return-16406-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CCE9F6F2D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 12:16:08 +0000 (UTC)
Received: (qmail 38828 invoked by uid 500); 16 Jun 2011 12:16:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38745 invoked by uid 500); 16 Jun 2011 12:16:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38737 invoked by uid 99); 16 Jun 2011 12:16:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 12:16:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 12:16:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8EA7641B24E
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 12:15:47 +0000 (UTC)
Date: Thu, 16 Jun 2011 12:15:47 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <619429730.10784.1308226547581.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J updated HADOOP-7328:
----------------------------

    Attachment: HADOOP-7328.r3.diff

New patch that throws IOExceptions instead.

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff, HADOOP-7328.r2.diff, HADOOP-7328.r3.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16407-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 16:18:11 2011
Return-Path: <common-issues-return-16407-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 10D8B6863
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 16:18:11 +0000 (UTC)
Received: (qmail 87482 invoked by uid 500); 16 Jun 2011 16:18:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87436 invoked by uid 500); 16 Jun 2011 16:18:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87420 invoked by uid 99); 16 Jun 2011 16:18:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 16:18:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 16:18:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9D1EA41B27E
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 16:17:47 +0000 (UTC)
Date: Thu, 16 Jun 2011 16:17:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1964581123.11440.1308241067640.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050533#comment-13050533 ] 

Owen O'Malley commented on HADOOP-7328:
---------------------------------------

Ugh. I hate the messages from review board.

This is a completely incompatible change. I'd propose adding a new method to SerializationFactory that is about explaining why it wasn't found.

I need to get back to HADOOP-6685 and move that forward.

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff, HADOOP-7328.r2.diff, HADOOP-7328.r3.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16408-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 16:28:08 2011
Return-Path: <common-issues-return-16408-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A55FD6470
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 16:28:08 +0000 (UTC)
Received: (qmail 10817 invoked by uid 500); 16 Jun 2011 16:28:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10775 invoked by uid 500); 16 Jun 2011 16:28:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10767 invoked by uid 99); 16 Jun 2011 16:28:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 16:28:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 16:28:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5D7BE41B7E8
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 16:27:47 +0000 (UTC)
Date: Thu, 16 Jun 2011 16:27:47 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1775819749.11466.1308241667379.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J updated HADOOP-7328:
----------------------------

    Status: Open  (was: Patch Available)

Owen,

Yes I noticed it doesn't compile the base after I did an {{ant clean compile}}. Was gonna cancel the patch right before your comment went in.

Looking into the other issue you mention.

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff, HADOOP-7328.r2.diff, HADOOP-7328.r3.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16409-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 16:46:09 2011
Return-Path: <common-issues-return-16409-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EAB1267D9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 16:46:08 +0000 (UTC)
Received: (qmail 52997 invoked by uid 500); 16 Jun 2011 16:46:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52941 invoked by uid 500); 16 Jun 2011 16:46:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52884 invoked by uid 99); 16 Jun 2011 16:46:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 16:46:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 16:46:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9987341C25A
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 16:45:47 +0000 (UTC)
Date: Thu, 16 Jun 2011 16:45:47 +0000 (UTC)
From: "Scott Fines (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <395796746.11563.1308242747625.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <149464743.7145.1308153347389.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7397) Allow configurable timeouts when
 connecting to HDFS via java FileSystem API
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Scott Fines updated HADOOP-7397:
--------------------------------

    Attachment: timeout.patch

> Allow configurable timeouts when connecting to HDFS via java FileSystem API
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-7397
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7397
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.20.2, 0.23.0
>         Environment: Any
>            Reporter: Scott Fines
>            Priority: Minor
>              Labels: hadoop
>             Fix For: 0.23.0
>
>         Attachments: timeout.patch
>
>
> If the NameNode is not available (in, for example, a network partition event separating the client from the NameNode), and an attempt is made to connect, then the FileSystem api will *eventually* timeout and throw an error. However, that timeout is currently hardcoded to be 20 seconds to connect, with 45 retries, for a total of a 15 minute wait before failure. While in many circumstances this is fine, there are also many circumstances (such as booting a service) where both the connection timeout and the number of retries should be significantly less, so as not to harm availability of other services.
> Investigating Client.java, I see that there are two fields in Connection: maxRetries and rpcTimeout. I propose either re-using those fields for initiating the connection as well; alternatively, using the already existing dfs.socket.timeout parameter to set the connection timeout on initialization, and potentially adding a new field such as dfs.connection.retries with a default of 45 to replicate current behaviors.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16410-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 16:46:10 2011
Return-Path: <common-issues-return-16410-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D33F36837
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 16:46:10 +0000 (UTC)
Received: (qmail 53298 invoked by uid 500); 16 Jun 2011 16:46:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53253 invoked by uid 500); 16 Jun 2011 16:46:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53245 invoked by uid 99); 16 Jun 2011 16:46:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 16:46:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 16:46:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7E2D541C256
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 16:45:47 +0000 (UTC)
Date: Thu, 16 Jun 2011 16:45:47 +0000 (UTC)
From: "Scott Fines (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <856665800.11559.1308242747513.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <149464743.7145.1308153347389.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7397) Allow configurable timeouts when
 connecting to HDFS via java FileSystem API
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Scott Fines updated HADOOP-7397:
--------------------------------

    Fix Version/s: 0.23.0
     Release Note: Allows clients to specify connection timeouts with the conf parameter "ipc.client.connect.timeout"
           Status: Patch Available  (was: Open)

Here's a patch for trunk. I've added an additional Configuration parameter called "ipc.client.connect.timeout" with a default of 20000, and I've changed the max retries from a hardcoded int 45 to the "ipc.client.connection.max.retries" conf parameter.

> Allow configurable timeouts when connecting to HDFS via java FileSystem API
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-7397
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7397
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.20.2, 0.23.0
>         Environment: Any
>            Reporter: Scott Fines
>            Priority: Minor
>              Labels: hadoop
>             Fix For: 0.23.0
>
>         Attachments: timeout.patch
>
>
> If the NameNode is not available (in, for example, a network partition event separating the client from the NameNode), and an attempt is made to connect, then the FileSystem api will *eventually* timeout and throw an error. However, that timeout is currently hardcoded to be 20 seconds to connect, with 45 retries, for a total of a 15 minute wait before failure. While in many circumstances this is fine, there are also many circumstances (such as booting a service) where both the connection timeout and the number of retries should be significantly less, so as not to harm availability of other services.
> Investigating Client.java, I see that there are two fields in Connection: maxRetries and rpcTimeout. I propose either re-using those fields for initiating the connection as well; alternatively, using the already existing dfs.socket.timeout parameter to set the connection timeout on initialization, and potentially adding a new field such as dfs.connection.retries with a default of 45 to replicate current behaviors.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16411-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 17:36:10 2011
Return-Path: <common-issues-return-16411-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CF03F61EA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 17:36:10 +0000 (UTC)
Received: (qmail 44300 invoked by uid 500); 16 Jun 2011 17:36:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44266 invoked by uid 500); 16 Jun 2011 17:36:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44258 invoked by uid 99); 16 Jun 2011 17:36:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 17:36:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 17:36:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 78D3841C737
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 17:35:47 +0000 (UTC)
Date: Thu, 16 Jun 2011 17:35:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2109320174.11686.1308245747491.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050580#comment-13050580 ] 

Todd Lipcon commented on HADOOP-7328:
-------------------------------------

bq. This is a completely incompatible change
The audience for SerializationFactory is private to HDFS and MR, so as long as we have matching patches for those projects, we don't need to consider this a user-facing incompatibility. Right?

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff, HADOOP-7328.r2.diff, HADOOP-7328.r3.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16412-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 18:22:10 2011
Return-Path: <common-issues-return-16412-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 323E9640B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 18:22:10 +0000 (UTC)
Received: (qmail 56970 invoked by uid 500); 16 Jun 2011 18:22:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56937 invoked by uid 500); 16 Jun 2011 18:22:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56929 invoked by uid 99); 16 Jun 2011 18:22:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 18:22:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 18:22:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C152D41C348
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 18:21:48 +0000 (UTC)
Date: Thu, 16 Jun 2011 18:21:48 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <452571753.12052.1308248508788.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050626#comment-13050626 ] 

Harsh J commented on HADOOP-7328:
---------------------------------

I actually do have two comments of concern w.r.t. front-end & throwing:

If we are to make code changes _throw-y_, then it would require that all places that use the said API inside Hadoop add throws IOException to their methods. Upon a quick check, only one single SequenceFile private utility method required this change, and none in MapReduce. But it could've been that the change required subsequent public API changes.

Since we've never had front end checks for MR serializers before; if people relied on passing strings over to job.xml and have it resolve only on worker machines - the related issue may break this functionality. Guideline-wise, its a wrong way to do it, but I think it still breaks something deep below I think.

But I think its worth enforcing this check (null handling later or throw above, either way) since it makes stuff like job setup errors quicker to debug and iterate upon.

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff, HADOOP-7328.r2.diff, HADOOP-7328.r3.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16413-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 18:33:18 2011
Return-Path: <common-issues-return-16413-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BE95367AD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 18:33:18 +0000 (UTC)
Received: (qmail 84104 invoked by uid 500); 16 Jun 2011 18:33:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84078 invoked by uid 500); 16 Jun 2011 18:33:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84070 invoked by uid 99); 16 Jun 2011 18:33:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 18:33:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 18:33:15 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3C43C41C790
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 18:32:48 +0000 (UTC)
Date: Thu, 16 Jun 2011 18:32:48 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1740160151.12080.1308249168243.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050632#comment-13050632 ] 

Owen O'Malley commented on HADOOP-7328:
---------------------------------------

{quote}
The audience for SerializationFactory is private to HDFS and MR.
{quote}

The annotations claiming it is limited private is *NOT* license to change it without severe need. There *are* other users and this change will break them.

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff, HADOOP-7328.r2.diff, HADOOP-7328.r3.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16414-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 18:35:09 2011
Return-Path: <common-issues-return-16414-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2AFE46CEC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 18:35:09 +0000 (UTC)
Received: (qmail 87763 invoked by uid 500); 16 Jun 2011 18:35:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87665 invoked by uid 500); 16 Jun 2011 18:35:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87653 invoked by uid 99); 16 Jun 2011 18:35:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 18:35:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 18:35:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C88C541C8EC
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 18:34:47 +0000 (UTC)
Date: Thu, 16 Jun 2011 18:34:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1970702874.12097.1308249287818.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050636#comment-13050636 ] 

Todd Lipcon commented on HADOOP-7328:
-------------------------------------

Are you OK with replacing the current behavior (throws NPE) with a "return null" instead, like the original patch on this JIRA? Anyone depending on a thrown NPE is just asking for trouble

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff, HADOOP-7328.r2.diff, HADOOP-7328.r3.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16415-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 18:47:12 2011
Return-Path: <common-issues-return-16415-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 152BB6E76
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 18:47:12 +0000 (UTC)
Received: (qmail 12964 invoked by uid 500); 16 Jun 2011 18:47:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12920 invoked by uid 500); 16 Jun 2011 18:47:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12912 invoked by uid 99); 16 Jun 2011 18:47:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 18:47:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 18:47:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C553841C029
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 18:46:50 +0000 (UTC)
Date: Thu, 16 Jun 2011 18:46:50 +0000 (UTC)
From: "T Jake Luciani (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1708763678.12221.1308250010805.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050641#comment-13050641 ] 

T Jake Luciani commented on HADOOP-7206:
----------------------------------------

I've attached the snappy-java version of this codec for consideration

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v1-0001-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16416-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 18:47:12 2011
Return-Path: <common-issues-return-16416-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9CAD56E9A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 18:47:12 +0000 (UTC)
Received: (qmail 13221 invoked by uid 500); 16 Jun 2011 18:47:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13191 invoked by uid 500); 16 Jun 2011 18:47:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13183 invoked by uid 99); 16 Jun 2011 18:47:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 18:47:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 18:47:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2140A41CFC2
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 18:46:49 +0000 (UTC)
Date: Thu, 16 Jun 2011 18:46:49 +0000 (UTC)
From: "T Jake Luciani (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1708333925.12174.1308250009132.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

T Jake Luciani updated HADOOP-7206:
-----------------------------------

    Attachment: v1-0001-HADOOP-7206-snappy-codec-using-snappy-java.txt

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v1-0001-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16417-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 18:47:13 2011
Return-Path: <common-issues-return-16417-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 99FDA6EBA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 18:47:13 +0000 (UTC)
Received: (qmail 13488 invoked by uid 500); 16 Jun 2011 18:47:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13463 invoked by uid 500); 16 Jun 2011 18:47:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13455 invoked by uid 99); 16 Jun 2011 18:47:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 18:47:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 18:47:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F04C641C02F
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 18:46:50 +0000 (UTC)
Date: Thu, 16 Jun 2011 18:46:50 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <560365360.12225.1308250010972.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050643#comment-13050643 ] 

Harsh J commented on HADOOP-7328:
---------------------------------

I'm good with returning (and where it applies, checking) nulls. I do not think that breaks anything at all.

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff, HADOOP-7328.r2.diff, HADOOP-7328.r3.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16418-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 19:35:09 2011
Return-Path: <common-issues-return-16418-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 594F36A54
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 19:35:09 +0000 (UTC)
Received: (qmail 6672 invoked by uid 500); 16 Jun 2011 19:35:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6636 invoked by uid 500); 16 Jun 2011 19:35:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6611 invoked by uid 99); 16 Jun 2011 19:35:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 19:35:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 19:35:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AD73941C450
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 19:34:47 +0000 (UTC)
Date: Thu, 16 Jun 2011 19:34:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1460003215.12348.1308252887706.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050671#comment-13050671 ] 

Hadoop QA commented on HADOOP-7206:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482838/v1-0001-HADOOP-7206-snappy-codec-using-snappy-java.txt
  against trunk revision 1136249.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 4 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/643//console

This message is automatically generated.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v1-0001-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16419-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 19:47:11 2011
Return-Path: <common-issues-return-16419-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C16C76520
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 19:47:11 +0000 (UTC)
Received: (qmail 35492 invoked by uid 500); 16 Jun 2011 19:47:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35456 invoked by uid 500); 16 Jun 2011 19:47:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35448 invoked by uid 99); 16 Jun 2011 19:47:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 19:47:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 19:47:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C902A41CB0D
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 19:46:47 +0000 (UTC)
Date: Thu, 16 Jun 2011 19:46:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1600040888.12447.1308253607819.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <149464743.7145.1308153347389.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7397) Allow configurable timeouts when
 connecting to HDFS via java FileSystem API
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050682#comment-13050682 ] 

Hadoop QA commented on HADOOP-7397:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482821/timeout.patch
  against trunk revision 1136249.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/642//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/642//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/642//console

This message is automatically generated.

> Allow configurable timeouts when connecting to HDFS via java FileSystem API
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-7397
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7397
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.20.2, 0.23.0
>         Environment: Any
>            Reporter: Scott Fines
>            Priority: Minor
>              Labels: hadoop
>             Fix For: 0.23.0
>
>         Attachments: timeout.patch
>
>
> If the NameNode is not available (in, for example, a network partition event separating the client from the NameNode), and an attempt is made to connect, then the FileSystem api will *eventually* timeout and throw an error. However, that timeout is currently hardcoded to be 20 seconds to connect, with 45 retries, for a total of a 15 minute wait before failure. While in many circumstances this is fine, there are also many circumstances (such as booting a service) where both the connection timeout and the number of retries should be significantly less, so as not to harm availability of other services.
> Investigating Client.java, I see that there are two fields in Connection: maxRetries and rpcTimeout. I propose either re-using those fields for initiating the connection as well; alternatively, using the already existing dfs.socket.timeout parameter to set the connection timeout on initialization, and potentially adding a new field such as dfs.connection.retries with a default of 45 to replicate current behaviors.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16420-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 20:15:11 2011
Return-Path: <common-issues-return-16420-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D9D7C6E65
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 20:15:10 +0000 (UTC)
Received: (qmail 87708 invoked by uid 500); 16 Jun 2011 20:15:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87646 invoked by uid 500); 16 Jun 2011 20:15:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87638 invoked by uid 99); 16 Jun 2011 20:15:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:15:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:15:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7F6AF41C732
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 20:14:47 +0000 (UTC)
Date: Thu, 16 Jun 2011 20:14:47 +0000 (UTC)
From: "T Jake Luciani (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1564252336.12495.1308255287518.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

T Jake Luciani updated HADOOP-7206:
-----------------------------------

    Attachment:     (was: v1-0001-HADOOP-7206-snappy-codec-using-snappy-java.txt)

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16421-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 20:19:09 2011
Return-Path: <common-issues-return-16421-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7D8CF601C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 20:19:09 +0000 (UTC)
Received: (qmail 91956 invoked by uid 500); 16 Jun 2011 20:19:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91917 invoked by uid 500); 16 Jun 2011 20:19:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91889 invoked by uid 99); 16 Jun 2011 20:19:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:19:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:19:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D42AC41C9AE
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 20:18:47 +0000 (UTC)
Date: Thu, 16 Jun 2011 20:18:47 +0000 (UTC)
From: "T Jake Luciani (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1902602762.12552.1308255527865.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

T Jake Luciani updated HADOOP-7206:
-----------------------------------

    Attachment: v1-0001-HADOOP-7206-snappy-codec-using-snappy-java.txt

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v1-0001-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16422-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 20:21:12 2011
Return-Path: <common-issues-return-16422-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0C4146578
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 20:21:12 +0000 (UTC)
Received: (qmail 95066 invoked by uid 500); 16 Jun 2011 20:21:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95031 invoked by uid 500); 16 Jun 2011 20:21:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95023 invoked by uid 99); 16 Jun 2011 20:21:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:21:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:21:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BDAFA41CBCA
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 20:20:50 +0000 (UTC)
Date: Thu, 16 Jun 2011 20:20:50 +0000 (UTC)
From: "T Jake Luciani (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1477484639.12698.1308255650773.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

T Jake Luciani updated HADOOP-7206:
-----------------------------------

    Attachment: v2-HADOOP-7206-snappy-codec-using-snappy-java.txt

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16423-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 20:21:12 2011
Return-Path: <common-issues-return-16423-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 903E56584
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 20:21:12 +0000 (UTC)
Received: (qmail 95313 invoked by uid 500); 16 Jun 2011 20:21:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95283 invoked by uid 500); 16 Jun 2011 20:21:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95275 invoked by uid 99); 16 Jun 2011 20:21:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:21:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:21:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1294241CB96
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 20:20:49 +0000 (UTC)
Date: Thu, 16 Jun 2011 20:20:49 +0000 (UTC)
From: "T Jake Luciani (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1942120000.12647.1308255649072.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

T Jake Luciani updated HADOOP-7206:
-----------------------------------

    Attachment:     (was: v1-0001-HADOOP-7206-snappy-codec-using-snappy-java.txt)

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16424-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 20:29:10 2011
Return-Path: <common-issues-return-16424-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C0F20697E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 20:29:10 +0000 (UTC)
Received: (qmail 13371 invoked by uid 500); 16 Jun 2011 20:29:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13348 invoked by uid 500); 16 Jun 2011 20:29:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13340 invoked by uid 99); 16 Jun 2011 20:29:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:29:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:29:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5C42F41CE3F
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 20:28:47 +0000 (UTC)
Date: Thu, 16 Jun 2011 20:28:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1564639476.12717.1308256127374.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1293648970.8758.1308177527351.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7398) create a mechanism to suppress the
 HADOOP_HOME deprecated warning
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7398:
----------------------------------

    Attachment: warn.patch

This sends the warning to stderr.

> create a mechanism to suppress the HADOOP_HOME deprecated warning
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7398
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7398
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: warn.patch, warn.patch
>
>
> Create a new mechanism to suppress the warning about HADOOP_HOME deprecation.
> I'll create a HADOOP_HOME_WARN_SUPPRESS environment variable that suppresses the warning.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16425-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 20:33:11 2011
Return-Path: <common-issues-return-16425-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 54CEA6BE1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 20:33:11 +0000 (UTC)
Received: (qmail 19421 invoked by uid 500); 16 Jun 2011 20:33:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19381 invoked by uid 500); 16 Jun 2011 20:33:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19373 invoked by uid 99); 16 Jun 2011 20:33:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:33:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:33:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1E27A41C07C
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 20:32:48 +0000 (UTC)
Date: Thu, 16 Jun 2011 20:32:48 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <161508832.12723.1308256368120.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1293648970.8758.1308177527351.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7398) create a mechanism to suppress the
 HADOOP_HOME deprecated warning
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7398:
----------------------------------

    Comment: was deleted

(was: This sends the warning to stderr.)

> create a mechanism to suppress the HADOOP_HOME deprecated warning
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7398
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7398
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: warn-204.patch, warn.patch, warn.patch
>
>
> Create a new mechanism to suppress the warning about HADOOP_HOME deprecation.
> I'll create a HADOOP_HOME_WARN_SUPPRESS environment variable that suppresses the warning.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16426-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 20:33:11 2011
Return-Path: <common-issues-return-16426-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AF1CB6BEF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 20:33:11 +0000 (UTC)
Received: (qmail 19657 invoked by uid 500); 16 Jun 2011 20:33:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19629 invoked by uid 500); 16 Jun 2011 20:33:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19621 invoked by uid 99); 16 Jun 2011 20:33:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:33:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:33:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3A21E41C082
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 20:32:48 +0000 (UTC)
Date: Thu, 16 Jun 2011 20:32:48 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <222484047.12727.1308256368235.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1293648970.8758.1308177527351.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7398) create a mechanism to suppress the
 HADOOP_HOME deprecated warning
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7398:
----------------------------------

    Attachment:     (was: warn.patch)

> create a mechanism to suppress the HADOOP_HOME deprecated warning
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7398
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7398
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: warn-204.patch, warn.patch, warn.patch
>
>
> Create a new mechanism to suppress the warning about HADOOP_HOME deprecation.
> I'll create a HADOOP_HOME_WARN_SUPPRESS environment variable that suppresses the warning.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16427-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 20:33:13 2011
Return-Path: <common-issues-return-16427-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BF3A16C3F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 20:33:13 +0000 (UTC)
Received: (qmail 20096 invoked by uid 500); 16 Jun 2011 20:33:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20054 invoked by uid 500); 16 Jun 2011 20:33:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19980 invoked by uid 99); 16 Jun 2011 20:33:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:33:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:33:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DF5CF41C095
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 20:32:48 +0000 (UTC)
Date: Thu, 16 Jun 2011 20:32:48 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <195981494.12744.1308256368911.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1293648970.8758.1308177527351.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7398) create a mechanism to suppress the
 HADOOP_HOME deprecated warning
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7398:
----------------------------------

    Attachment: warn-204.patch
                warn.patch

The warn patch for 20-security adds the suppression and redirects it to stderr.

The 204/205 patch removes the warning.

> create a mechanism to suppress the HADOOP_HOME deprecated warning
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7398
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7398
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: warn-204.patch, warn.patch, warn.patch
>
>
> Create a new mechanism to suppress the warning about HADOOP_HOME deprecation.
> I'll create a HADOOP_HOME_WARN_SUPPRESS environment variable that suppresses the warning.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16428-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 20:35:09 2011
Return-Path: <common-issues-return-16428-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 01E9C64AA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 20:35:09 +0000 (UTC)
Received: (qmail 24207 invoked by uid 500); 16 Jun 2011 20:35:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24146 invoked by uid 500); 16 Jun 2011 20:35:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24125 invoked by uid 99); 16 Jun 2011 20:35:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:35:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:35:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8F6B441C228
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 20:34:47 +0000 (UTC)
Date: Thu, 16 Jun 2011 20:34:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <104084546.12753.1308256487584.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1293648970.8758.1308177527351.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7398) create a mechanism to suppress the
 HADOOP_HOME deprecated warning
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050710#comment-13050710 ] 

Luke Lu commented on HADOOP-7398:
---------------------------------

+1, the new patches lgtm.

> create a mechanism to suppress the HADOOP_HOME deprecated warning
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7398
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7398
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: warn-204.patch, warn.patch, warn.patch
>
>
> Create a new mechanism to suppress the warning about HADOOP_HOME deprecation.
> I'll create a HADOOP_HOME_WARN_SUPPRESS environment variable that suppresses the warning.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16429-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 20:37:12 2011
Return-Path: <common-issues-return-16429-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 62A846535
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 20:37:12 +0000 (UTC)
Received: (qmail 27216 invoked by uid 500); 16 Jun 2011 20:37:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27180 invoked by uid 500); 16 Jun 2011 20:37:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27170 invoked by uid 99); 16 Jun 2011 20:37:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:37:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:37:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0A49141C35C
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 20:36:49 +0000 (UTC)
Date: Thu, 16 Jun 2011 20:36:49 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <591022018.12802.1308256609038.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050711#comment-13050711 ] 

Hadoop QA commented on HADOOP-7206:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482850/v2-HADOOP-7206-snappy-codec-using-snappy-java.txt
  against trunk revision 1136249.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/644//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/644//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/644//console

This message is automatically generated.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16430-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 20:39:11 2011
Return-Path: <common-issues-return-16430-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 820D765A3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 20:39:11 +0000 (UTC)
Received: (qmail 33650 invoked by uid 500); 16 Jun 2011 20:39:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33600 invoked by uid 500); 16 Jun 2011 20:39:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33484 invoked by uid 99); 16 Jun 2011 20:39:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:39:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:39:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9C9D941C4C0
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 20:38:48 +0000 (UTC)
Date: Thu, 16 Jun 2011 20:38:48 +0000 (UTC)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <345461627.12821.1308256728638.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7400) HdfsProxyTests fails when the
 -Dtest.build.dir and -Dbuild.test is set
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

HdfsProxyTests fails when the -Dtest.build.dir and -Dbuild.test is set 
-----------------------------------------------------------------------

                 Key: HADOOP-7400
                 URL: https://issues.apache.org/jira/browse/HADOOP-7400
             Project: Hadoop Common
          Issue Type: Bug
          Components: build
    Affects Versions: 0.20.206.0
            Reporter: Giridharan Kesavan
            Assignee: Giridharan Kesavan


HdfsProxyTests fails when the -Dtest.build.dir and -Dbuild.test is set a dir other than build dir

test-junit:
     [copy] Copying 1 file to /home/y/var/builds/thread2/workspace/Cloud-Hadoop-0.20.1xx-Secondary/src/contrib/hdfsproxy/src/test/resources/proxy-config
    [junit] Running org.apache.hadoop.hdfsproxy.TestHdfsProxy
    [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
    [junit] Test org.apache.hadoop.hdfsproxy.TestHdfsProxy FAILED

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16431-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 20:53:12 2011
Return-Path: <common-issues-return-16431-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6A4496F4E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 20:53:12 +0000 (UTC)
Received: (qmail 57415 invoked by uid 500); 16 Jun 2011 20:53:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57372 invoked by uid 500); 16 Jun 2011 20:53:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57243 invoked by uid 99); 16 Jun 2011 20:53:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:53:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:53:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 838EE41CCFE
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 20:52:47 +0000 (UTC)
Date: Thu, 16 Jun 2011 20:52:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1489280073.12870.1308257567535.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1293648970.8758.1308177527351.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7398) create a mechanism to suppress the
 HADOOP_HOME deprecated warning
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley resolved HADOOP-7398.
-----------------------------------

       Resolution: Fixed
    Fix Version/s: 0.20.205.0
                   0.20.204.0
     Hadoop Flags: [Reviewed]

I just committed this.

> create a mechanism to suppress the HADOOP_HOME deprecated warning
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7398
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7398
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.204.0, 0.20.205.0
>
>         Attachments: warn-204.patch, warn.patch, warn.patch
>
>
> Create a new mechanism to suppress the warning about HADOOP_HOME deprecation.
> I'll create a HADOOP_HOME_WARN_SUPPRESS environment variable that suppresses the warning.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16432-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 20:55:09 2011
Return-Path: <common-issues-return-16432-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3C8B3619F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 20:55:09 +0000 (UTC)
Received: (qmail 62493 invoked by uid 500); 16 Jun 2011 20:55:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62443 invoked by uid 500); 16 Jun 2011 20:55:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62433 invoked by uid 99); 16 Jun 2011 20:55:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:55:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:55:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D6FD241CF03
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 20:54:47 +0000 (UTC)
Date: Thu, 16 Jun 2011 20:54:47 +0000 (UTC)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <24973087.12893.1308257687877.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <345461627.12821.1308256728638.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7400) HdfsProxyTests fails when the
 -Dtest.build.dir and -Dbuild.test is set
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-7400:
---------------------------------------

    Attachment: HADOOP-7400.patch

> HdfsProxyTests fails when the -Dtest.build.dir and -Dbuild.test is set 
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-7400
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7400
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.20.206.0
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>         Attachments: HADOOP-7400.patch
>
>
> HdfsProxyTests fails when the -Dtest.build.dir and -Dbuild.test is set a dir other than build dir
> test-junit:
>      [copy] Copying 1 file to /home/y/var/builds/thread2/workspace/Cloud-Hadoop-0.20.1xx-Secondary/src/contrib/hdfsproxy/src/test/resources/proxy-config
>     [junit] Running org.apache.hadoop.hdfsproxy.TestHdfsProxy
>     [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
>     [junit] Test org.apache.hadoop.hdfsproxy.TestHdfsProxy FAILED

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16433-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 20:57:09 2011
Return-Path: <common-issues-return-16433-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 113616254
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 20:57:09 +0000 (UTC)
Received: (qmail 65182 invoked by uid 500); 16 Jun 2011 20:57:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65144 invoked by uid 500); 16 Jun 2011 20:57:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65136 invoked by uid 99); 16 Jun 2011 20:57:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:57:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 20:57:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AEB3C41C075
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 20:56:47 +0000 (UTC)
Date: Thu, 16 Jun 2011 20:56:47 +0000 (UTC)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1475256669.12907.1308257807712.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <345461627.12821.1308256728638.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7400) HdfsProxyTests fails when the
 -Dtest.build.dir and -Dbuild.test is set
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-7400:
---------------------------------------

    Attachment: HADOOP-7400.patch

> HdfsProxyTests fails when the -Dtest.build.dir and -Dbuild.test is set 
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-7400
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7400
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.20.206.0
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>         Attachments: HADOOP-7400.patch, HADOOP-7400.patch
>
>
> HdfsProxyTests fails when the -Dtest.build.dir and -Dbuild.test is set a dir other than build dir
> test-junit:
>      [copy] Copying 1 file to /home/y/var/builds/thread2/workspace/Cloud-Hadoop-0.20.1xx-Secondary/src/contrib/hdfsproxy/src/test/resources/proxy-config
>     [junit] Running org.apache.hadoop.hdfsproxy.TestHdfsProxy
>     [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
>     [junit] Test org.apache.hadoop.hdfsproxy.TestHdfsProxy FAILED

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16434-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 21:28:56 2011
Return-Path: <common-issues-return-16434-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8150260C8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 21:28:56 +0000 (UTC)
Received: (qmail 75912 invoked by uid 500); 16 Jun 2011 21:07:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75885 invoked by uid 500); 16 Jun 2011 21:07:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75877 invoked by uid 99); 16 Jun 2011 21:07:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 21:07:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 21:07:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DCAC441C4B6
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 21:06:47 +0000 (UTC)
Date: Thu, 16 Jun 2011 21:06:47 +0000 (UTC)
From: "Kihwal Lee (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <437014379.12957.1308258407900.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <345461627.12821.1308256728638.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7400) HdfsProxyTests fails when the
 -Dtest.build.dir and -Dbuild.test is set
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050730#comment-13050730 ] 

Kihwal Lee commented on HADOOP-7400:
------------------------------------

Thanks, Giri. With this patch, the test is launched correctly. Although TestHdfsProxy is a bit flaky in trunk, this patch should still apply. 

+1 on commit to 0.20-security.

For trunk the patch needs to be a bit different due to the tree structure change.

> HdfsProxyTests fails when the -Dtest.build.dir and -Dbuild.test is set 
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-7400
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7400
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.20.206.0
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>         Attachments: HADOOP-7400.patch, HADOOP-7400.patch
>
>
> HdfsProxyTests fails when the -Dtest.build.dir and -Dbuild.test is set a dir other than build dir
> test-junit:
>      [copy] Copying 1 file to /home/y/var/builds/thread2/workspace/Cloud-Hadoop-0.20.1xx-Secondary/src/contrib/hdfsproxy/src/test/resources/proxy-config
>     [junit] Running org.apache.hadoop.hdfsproxy.TestHdfsProxy
>     [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
>     [junit] Test org.apache.hadoop.hdfsproxy.TestHdfsProxy FAILED

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16435-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 21:32:08 2011
Return-Path: <common-issues-return-16435-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C33BB6010
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 21:32:08 +0000 (UTC)
Received: (qmail 11557 invoked by uid 500); 16 Jun 2011 21:32:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11521 invoked by uid 500); 16 Jun 2011 21:32:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11512 invoked by uid 99); 16 Jun 2011 21:32:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 21:32:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 21:32:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 61F2B41CE6D
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 21:31:47 +0000 (UTC)
Date: Thu, 16 Jun 2011 21:31:47 +0000 (UTC)
From: "Kihwal Lee (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1865119655.13000.1308259907398.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <345461627.12821.1308256728638.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7400) HdfsProxyTests fails when the
 -Dtest.build.dir and -Dbuild.test is set
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050744#comment-13050744 ] 

Kihwal Lee commented on HADOOP-7400:
------------------------------------

Hmm. It seems the entire TestHdfsProxy is disabled in trunk. I thought just one thing was disabled in the test. We can still do the same on trunk since the code is still there and may be brought back to life in the future.

> HdfsProxyTests fails when the -Dtest.build.dir and -Dbuild.test is set 
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-7400
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7400
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.20.206.0
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>         Attachments: HADOOP-7400.patch, HADOOP-7400.patch
>
>
> HdfsProxyTests fails when the -Dtest.build.dir and -Dbuild.test is set a dir other than build dir
> test-junit:
>      [copy] Copying 1 file to /home/y/var/builds/thread2/workspace/Cloud-Hadoop-0.20.1xx-Secondary/src/contrib/hdfsproxy/src/test/resources/proxy-config
>     [junit] Running org.apache.hadoop.hdfsproxy.TestHdfsProxy
>     [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
>     [junit] Test org.apache.hadoop.hdfsproxy.TestHdfsProxy FAILED

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16436-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 21:36:08 2011
Return-Path: <common-issues-return-16436-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D8DE26B4A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 21:36:08 +0000 (UTC)
Received: (qmail 17591 invoked by uid 500); 16 Jun 2011 21:36:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17565 invoked by uid 500); 16 Jun 2011 21:36:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17557 invoked by uid 99); 16 Jun 2011 21:36:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 21:36:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 21:36:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6C3CB41C061
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 21:35:47 +0000 (UTC)
Date: Thu, 16 Jun 2011 21:35:47 +0000 (UTC)
From: "Kihwal Lee (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1602363391.13012.1308260147440.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <345461627.12821.1308256728638.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7400) HdfsProxyTests fails when the
 -Dtest.build.dir and -Dbuild.test is set
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050747#comment-13050747 ] 

Kihwal Lee commented on HADOOP-7400:
------------------------------------

FYI, the following was done to trunk in HDFS-1666. If anyone intend to verify hdfsproxy in trunk or any release based off trunk, HDFS-1666 needs to be revisited.

{code}
   <target name="test">
+    <!-- hdfsproxy tests failing due to HDFS-1666
     <subant target="test">
       <fileset dir="." includes="hdfsproxy/build.xml"/>
     </subant>
+      -->
   </target>
{code}


> HdfsProxyTests fails when the -Dtest.build.dir and -Dbuild.test is set 
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-7400
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7400
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.20.206.0
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>         Attachments: HADOOP-7400.patch, HADOOP-7400.patch
>
>
> HdfsProxyTests fails when the -Dtest.build.dir and -Dbuild.test is set a dir other than build dir
> test-junit:
>      [copy] Copying 1 file to /home/y/var/builds/thread2/workspace/Cloud-Hadoop-0.20.1xx-Secondary/src/contrib/hdfsproxy/src/test/resources/proxy-config
>     [junit] Running org.apache.hadoop.hdfsproxy.TestHdfsProxy
>     [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
>     [junit] Test org.apache.hadoop.hdfsproxy.TestHdfsProxy FAILED

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16437-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 22:58:09 2011
Return-Path: <common-issues-return-16437-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 13DF16758
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 22:58:09 +0000 (UTC)
Received: (qmail 31865 invoked by uid 500); 16 Jun 2011 22:58:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31835 invoked by uid 500); 16 Jun 2011 22:58:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31827 invoked by uid 99); 16 Jun 2011 22:58:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 22:58:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 22:58:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A7C7D41C14C
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 22:57:47 +0000 (UTC)
Date: Thu, 16 Jun 2011 22:57:47 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1994815367.13214.1308265067684.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-6671:
---------------------------------------

    Attachment: HADOOP-6671-i.patch
                common-mvn-layout-i.sh

rebased to trunk's head.

script & patch version 'i'

test-patch.sh wired

findbugs/clover/rat/checkstyle work. Refer to BUILDING.txt file for details.


> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-6671-cross-project-HDFS.patch, HADOOP-6671-e.patch, HADOOP-6671-f.patch, HADOOP-6671-g.patch, HADOOP-6671-h.patch, HADOOP-6671-i.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, common-mvn-layout-i.sh, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout-f.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16438-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 23:04:11 2011
Return-Path: <common-issues-return-16438-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 947556D52
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 23:04:11 +0000 (UTC)
Received: (qmail 41169 invoked by uid 500); 16 Jun 2011 23:04:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41129 invoked by uid 500); 16 Jun 2011 23:04:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41121 invoked by uid 99); 16 Jun 2011 23:04:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 23:04:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 23:04:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 326F141C3BF
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 23:03:48 +0000 (UTC)
Date: Thu, 16 Jun 2011 23:03:48 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1715106687.13272.1308265428203.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050787#comment-13050787 ] 

Hadoop QA commented on HADOOP-6671:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482874/HADOOP-6671-i.patch
  against trunk revision 1136249.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 57 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/645//console

This message is automatically generated.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-6671-cross-project-HDFS.patch, HADOOP-6671-e.patch, HADOOP-6671-f.patch, HADOOP-6671-g.patch, HADOOP-6671-h.patch, HADOOP-6671-i.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, common-mvn-layout-i.sh, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout-f.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16439-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 16 23:40:09 2011
Return-Path: <common-issues-return-16439-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5CAA06F15
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 16 Jun 2011 23:40:09 +0000 (UTC)
Received: (qmail 78250 invoked by uid 500); 16 Jun 2011 23:40:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78155 invoked by uid 500); 16 Jun 2011 23:40:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78110 invoked by uid 99); 16 Jun 2011 23:40:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 23:40:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 23:40:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1C1E141C605
	for <common-issues@hadoop.apache.org>; Thu, 16 Jun 2011 23:39:48 +0000 (UTC)
Date: Thu, 16 Jun 2011 23:39:48 +0000 (UTC)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1982428292.13417.1308267588112.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <345461627.12821.1308256728638.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7400) HdfsProxyTests fails when the
 -Dtest.build.dir and -Dbuild.test is set
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan resolved HADOOP-7400.
----------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.20.206.0

> HdfsProxyTests fails when the -Dtest.build.dir and -Dbuild.test is set 
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-7400
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7400
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.20.206.0
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>             Fix For: 0.20.206.0
>
>         Attachments: HADOOP-7400.patch, HADOOP-7400.patch
>
>
> HdfsProxyTests fails when the -Dtest.build.dir and -Dbuild.test is set a dir other than build dir
> test-junit:
>      [copy] Copying 1 file to /home/y/var/builds/thread2/workspace/Cloud-Hadoop-0.20.1xx-Secondary/src/contrib/hdfsproxy/src/test/resources/proxy-config
>     [junit] Running org.apache.hadoop.hdfsproxy.TestHdfsProxy
>     [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
>     [junit] Test org.apache.hadoop.hdfsproxy.TestHdfsProxy FAILED

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16440-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 00:47:10 2011
Return-Path: <common-issues-return-16440-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AD1F56115
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 00:47:10 +0000 (UTC)
Received: (qmail 55963 invoked by uid 500); 17 Jun 2011 00:47:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55933 invoked by uid 500); 17 Jun 2011 00:47:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55925 invoked by uid 99); 17 Jun 2011 00:47:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 00:47:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 00:47:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5F0F641AA85
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 00:46:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 00:46:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1142015754.13505.1308271607386.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1354187382.13188.1307764078831.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7380) Common portion of HDFS-1973
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050831#comment-13050831 ] 

Eli Collins commented on HADOOP-7380:
-------------------------------------

Took a quick look at the patch. Overall the approach looks sane, I'll follow up with more detailed review feedback. 

> Common portion of HDFS-1973
> ---------------------------
>
>                 Key: HADOOP-7380
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7380
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7380.0.patch, hadoop-7380.1.patch
>
>
> Implementing client failover will likely require changes to {{o.a.h.io.ipc}} and/or {{o.a.h.io.retry}}. This JIRA is to track those changes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16441-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 01:07:10 2011
Return-Path: <common-issues-return-16441-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 467E36CDC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 01:07:10 +0000 (UTC)
Received: (qmail 70060 invoked by uid 500); 17 Jun 2011 01:07:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70035 invoked by uid 500); 17 Jun 2011 01:07:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70026 invoked by uid 99); 17 Jun 2011 01:07:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 01:07:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 01:07:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EC5FF41C1BF
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 01:06:48 +0000 (UTC)
Date: Fri, 17 Jun 2011 01:06:48 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <974102316.13598.1308272808964.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050835#comment-13050835 ] 

Alejandro Abdelnur commented on HADOOP-7206:
--------------------------------------------

For both patches, the codec should be defined in the codecs property in commons-default.xml

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16442-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 02:01:12 2011
Return-Path: <common-issues-return-16442-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BB3D140C0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 02:01:12 +0000 (UTC)
Received: (qmail 21584 invoked by uid 500); 17 Jun 2011 02:01:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21550 invoked by uid 500); 17 Jun 2011 02:01:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21537 invoked by uid 99); 17 Jun 2011 02:01:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 02:01:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 02:01:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1EEBD41D1ED
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 02:00:49 +0000 (UTC)
Date: Fri, 17 Jun 2011 02:00:49 +0000 (UTC)
From: "T Jake Luciani (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1128275309.13740.1308276049123.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

T Jake Luciani updated HADOOP-7206:
-----------------------------------

    Attachment: v3-HADOOP-7206-snappy-codec-using-snappy-java.txt

v3 adds SnappyCodec to core-default.xml

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16443-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 02:15:11 2011
Return-Path: <common-issues-return-16443-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 28C5D417E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 02:15:11 +0000 (UTC)
Received: (qmail 30228 invoked by uid 500); 17 Jun 2011 02:15:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30145 invoked by uid 500); 17 Jun 2011 02:15:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30120 invoked by uid 99); 17 Jun 2011 02:15:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 02:15:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 02:15:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A54E341D66C
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 02:14:49 +0000 (UTC)
Date: Fri, 17 Jun 2011 02:14:49 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1859928895.13801.1308276889673.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050859#comment-13050859 ] 

Hadoop QA commented on HADOOP-7206:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482888/v3-HADOOP-7206-snappy-codec-using-snappy-java.txt
  against trunk revision 1136249.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/646//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/646//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/646//console

This message is automatically generated.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16444-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 15:23:11 2011
Return-Path: <common-issues-return-16444-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3F81841EC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 15:23:11 +0000 (UTC)
Received: (qmail 99896 invoked by uid 500); 17 Jun 2011 15:23:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99407 invoked by uid 500); 17 Jun 2011 15:23:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99391 invoked by uid 99); 17 Jun 2011 15:23:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 15:23:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 15:23:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C702841E0D4
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 15:22:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 15:22:47 +0000 (UTC)
From: "monica beckwith (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1229925666.15226.1308324167811.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7401) Unit test TestPureJavaCRC32 warmup
 code warms up the not-so-important loop in PureJavaCRC32.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Unit test TestPureJavaCRC32 warmup code warms up the not-so-important loop in PureJavaCRC32.update()
----------------------------------------------------------------------------------------------------

                 Key: HADOOP-7401
                 URL: https://issues.apache.org/jira/browse/HADOOP-7401
             Project: Hadoop Common
          Issue Type: Improvement
          Components: test
    Affects Versions: 0.21.0
         Environment: Solaris-Sparcv9; Solaris-AMD64
            Reporter: monica beckwith
            Priority: Minor


When the warmup code sequence in TestPureJavaCRC32.java is executed, it sends size=len=2 and due to the value of 'trials' in for loop in doBench(), the crc.update() gets run > compile threshold, thus providing the information that 'while 0>len<7' is a hot loop and 'while len>7' is a cold loop. This brings the MB/s number for len > 7 in PureJavaCRC32.update() way down (e.g. ~28.5% for size=len=65536) .

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16445-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 15:39:10 2011
Return-Path: <common-issues-return-16445-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BB9EE4E1C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 15:39:10 +0000 (UTC)
Received: (qmail 35061 invoked by uid 500); 17 Jun 2011 15:39:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35006 invoked by uid 500); 17 Jun 2011 15:39:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34998 invoked by uid 99); 17 Jun 2011 15:39:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 15:39:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 15:39:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 65FEA41E865
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 15:38:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 15:38:47 +0000 (UTC)
From: "monica beckwith (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1170654712.15266.1308325127414.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1229925666.15226.1308324167811.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7401) Unit test TestPureJavaCRC32 warmup
 code warms up the not-so-important loop in PureJavaCRC32.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

monica beckwith updated HADOOP-7401:
------------------------------------

    Description: 
When the warmup code sequence in TestPureJavaCRC32.java is executed, it sends size=len=2 and due to the value of 'trials' in for loop in doBench(), the crc.update() gets run > the compile threshold, thus providing the information that 'while 0>len<7' is a hot loop and 'while len>7' is a cold loop. This brings the MB/s number for len > 7 in PureJavaCRC32.update() way down (e.g. ~28.5% for size=len=65536).

The workaround would be to use size=len=>7 (so just having size=len=2101 ahead of size=len=2 will do the trick) in the warmup section. 

  was:When the warmup code sequence in TestPureJavaCRC32.java is executed, it sends size=len=2 and due to the value of 'trials' in for loop in doBench(), the crc.update() gets run > compile threshold, thus providing the information that 'while 0>len<7' is a hot loop and 'while len>7' is a cold loop. This brings the MB/s number for len > 7 in PureJavaCRC32.update() way down (e.g. ~28.5% for size=len=65536) .


> Unit test TestPureJavaCRC32 warmup code warms up the not-so-important loop in PureJavaCRC32.update()
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7401
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7401
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>         Environment: Solaris-Sparcv9; Solaris-AMD64
>            Reporter: monica beckwith
>            Priority: Minor
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> When the warmup code sequence in TestPureJavaCRC32.java is executed, it sends size=len=2 and due to the value of 'trials' in for loop in doBench(), the crc.update() gets run > the compile threshold, thus providing the information that 'while 0>len<7' is a hot loop and 'while len>7' is a cold loop. This brings the MB/s number for len > 7 in PureJavaCRC32.update() way down (e.g. ~28.5% for size=len=65536).
> The workaround would be to use size=len=>7 (so just having size=len=2101 ahead of size=len=2 will do the trick) in the warmup section. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16446-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 16:07:11 2011
Return-Path: <common-issues-return-16446-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6F79C49AD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 16:07:11 +0000 (UTC)
Received: (qmail 96552 invoked by uid 500); 17 Jun 2011 16:07:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96486 invoked by uid 500); 17 Jun 2011 16:07:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96478 invoked by uid 99); 17 Jun 2011 16:07:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 16:07:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 16:07:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D6DDF41E84C
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 16:06:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 16:06:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1378269829.15355.1308326807876.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1937737372.43422.1306361387389.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7330) The metrics source mbean
 implementation should return the attribute value instead of the object
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7330:
----------------------------------

    Fix Version/s: 0.20.204.0

> The metrics source mbean implementation should return the attribute value instead of the object
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7330
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7330
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.203.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.20.203.1, 0.20.204.0
>
>         Attachments: hadoop-7330-jmx-v1.patch
>
>
> The MetricsSourceAdapter#getAttribute in 0.20.203 is returning the attribute object instead of the value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16447-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 16:35:11 2011
Return-Path: <common-issues-return-16447-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2967F4454
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 16:35:11 +0000 (UTC)
Received: (qmail 54943 invoked by uid 500); 17 Jun 2011 16:35:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54919 invoked by uid 500); 17 Jun 2011 16:35:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54911 invoked by uid 99); 17 Jun 2011 16:35:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 16:35:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 16:35:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9A96741E5EE
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 16:34:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 16:34:47 +0000 (UTC)
From: "monica beckwith (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1813025616.15448.1308328487630.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1229925666.15226.1308324167811.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7401) Unit test TestPureJavaCRC32 warmup
 code warms up the not-so-important loop in PureJavaCRC32.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

monica beckwith updated HADOOP-7401:
------------------------------------

    Description: 
When the warmup code sequence in TestPureJavaCRC32.java is executed, it sends size=len=2 and due to the value of 'trials' in for loop in doBench(), the crc.update() gets run > the compile threshold, thus providing the information that 'while 0<len<7' is a hot loop and 'while len>7' is a cold loop. This brings the MB/s number for len > 7 in PureJavaCRC32.update() way down (e.g. ~28.5% for size=len=65536).

The workaround would be to use size=len=>7 (so just having size=len=2101 ahead of size=len=2 will do the trick) in the warmup section. 

  was:
When the warmup code sequence in TestPureJavaCRC32.java is executed, it sends size=len=2 and due to the value of 'trials' in for loop in doBench(), the crc.update() gets run > the compile threshold, thus providing the information that 'while 0>len<7' is a hot loop and 'while len>7' is a cold loop. This brings the MB/s number for len > 7 in PureJavaCRC32.update() way down (e.g. ~28.5% for size=len=65536).

The workaround would be to use size=len=>7 (so just having size=len=2101 ahead of size=len=2 will do the trick) in the warmup section. 


> Unit test TestPureJavaCRC32 warmup code warms up the not-so-important loop in PureJavaCRC32.update()
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7401
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7401
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>         Environment: Solaris-Sparcv9; Solaris-AMD64
>            Reporter: monica beckwith
>            Priority: Minor
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> When the warmup code sequence in TestPureJavaCRC32.java is executed, it sends size=len=2 and due to the value of 'trials' in for loop in doBench(), the crc.update() gets run > the compile threshold, thus providing the information that 'while 0<len<7' is a hot loop and 'while len>7' is a cold loop. This brings the MB/s number for len > 7 in PureJavaCRC32.update() way down (e.g. ~28.5% for size=len=65536).
> The workaround would be to use size=len=>7 (so just having size=len=2101 ahead of size=len=2 will do the trick) in the warmup section. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16448-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 17:29:30 2011
Return-Path: <common-issues-return-16448-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9D7FB421F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 17:29:30 +0000 (UTC)
Received: (qmail 55293 invoked by uid 500); 17 Jun 2011 17:29:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55249 invoked by uid 500); 17 Jun 2011 17:29:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55241 invoked by uid 99); 17 Jun 2011 17:29:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 17:29:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 17:29:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C35E941E5C8
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 17:28:48 +0000 (UTC)
Date: Fri, 17 Jun 2011 17:28:48 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <439916422.15691.1308331728797.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <879608400.1240.1307478358811.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7364) TestMiniMRDFSCaching fails if
 test.build.dir is set to something other than build/test
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7364:
----------------------------------

    Fix Version/s: 0.20.204.0

> TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7364
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7364
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.20.205.0
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.20.204.0, 0.20.205.0
>
>         Attachments: HADOOP-7364-trunk.patch, HADOOP-7364.patch
>
>
> TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16449-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 17:29:32 2011
Return-Path: <common-issues-return-16449-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D6D62427B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 17:29:32 +0000 (UTC)
Received: (qmail 56246 invoked by uid 500); 17 Jun 2011 17:29:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56215 invoked by uid 500); 17 Jun 2011 17:29:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56205 invoked by uid 99); 17 Jun 2011 17:29:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 17:29:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 17:29:30 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DD0E141E5CB
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 17:28:48 +0000 (UTC)
Date: Fri, 17 Jun 2011 17:28:48 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1684048523.15694.1308331728902.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <161449362.3682.1305134627390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7277) Add Eclipse launch tasks for the
 0.20-security branch
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7277:
----------------------------------

    Fix Version/s: 0.20.204.0

> Add Eclipse launch tasks for the 0.20-security branch
> -----------------------------------------------------
>
>                 Key: HADOOP-7277
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7277
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.204.0
>            Reporter: Jeffrey Naisbitt
>            Assignee: Jeffrey Naisbitt
>            Priority: Minor
>             Fix For: 0.20.204.0, 0.20.205.0
>
>         Attachments: HADOOP-7277-0.20s.patch
>
>
> This is to add the eclipse launchers from HADOOP-5911 to the 0.20 security branch.
> Eclipse has a notion of "run configuration", which encapsulates what's needed to run or debug an application. I use this quite a bit to start various Hadoop daemons in debug mode, with breakpoints set, to inspect state and what not.
> This is simply configuration, so no tests are provided. After running "ant eclipse" and refreshing your project, you should see entries in the Run Configurations and Debug Configurations for launching the various hadoop daemons from within eclipse. There's a template for testing a specific test, and also templates to run all the tests, the job tracker, and a task tracker. It's likely that some parameters need to be further tweaked to have the same behavior as "ant test", but for most tests, this works.
> This also requires a small change to build.xml for the eclipse classpath.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16450-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 17:29:33 2011
Return-Path: <common-issues-return-16450-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3346C428A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 17:29:33 +0000 (UTC)
Received: (qmail 56514 invoked by uid 500); 17 Jun 2011 17:29:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56479 invoked by uid 500); 17 Jun 2011 17:29:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56469 invoked by uid 99); 17 Jun 2011 17:29:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 17:29:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 17:29:30 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E62CF41E5CC
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 17:28:48 +0000 (UTC)
Date: Fri, 17 Jun 2011 17:28:48 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2116542248.15695.1308331728939.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <88464990.3250.1305126467497.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7274) CLONE - IOUtils.readFully and
 IOUtils.skipFully have typo in exception creation's message
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7274:
----------------------------------

    Fix Version/s: 0.20.204.0

> CLONE - IOUtils.readFully and IOUtils.skipFully have typo in exception creation's message
> -----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7274
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7274
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.20.204.0
>            Reporter: Jonathan Eagles
>            Assignee: Jonathan Eagles
>            Priority: Minor
>             Fix For: 0.20.204.0, 0.20.205.0
>
>         Attachments: HADOOP-7274-0.20-security.patch
>
>
> Same fix as for HADOOP-7057 for the Hadoop security branch
> {noformat}
>         throw new IOException( "Premeture EOF from inputStream");
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16451-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 17:31:09 2011
Return-Path: <common-issues-return-16451-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 04B4B48FF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 17:31:09 +0000 (UTC)
Received: (qmail 59401 invoked by uid 500); 17 Jun 2011 17:31:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59374 invoked by uid 500); 17 Jun 2011 17:31:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59365 invoked by uid 99); 17 Jun 2011 17:31:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 17:31:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 17:31:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C171C41E6FE
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 17:30:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 17:30:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <305373051.15705.1308331847789.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7144:
----------------------------------

    Fix Version/s: 0.20.204.0

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Robert Joseph Evans
>              Labels: jmx
>             Fix For: 0.20.204.0, 0.20.205.0, 0.23.0
>
>         Attachments: HADOOP-7411-0.20.20X-V1.patch, HADOOP-7411-0.20.20X-V2.patch, HADOOP-7411-trunk-V1.patch, HADOOP-7411-trunk-V2.patch, HADOOP-7411-trunk-alpha.patch, jmx.json
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16452-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 17:31:11 2011
Return-Path: <common-issues-return-16452-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 10A5D493C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 17:31:11 +0000 (UTC)
Received: (qmail 59719 invoked by uid 500); 17 Jun 2011 17:31:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59669 invoked by uid 500); 17 Jun 2011 17:31:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59657 invoked by uid 99); 17 Jun 2011 17:31:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 17:31:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 17:31:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 78CC341E6F8
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 17:30:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 17:30:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1743542645.15699.1308331847491.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <285327525.9499.1304014323342.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7248) Have a way to automatically update
 Eclipse .classpath file when new libs are added to the classpath through
 Ivy for 0.20-* based sources
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7248:
----------------------------------

    Fix Version/s: 0.20.204.0

> Have a way to automatically update Eclipse .classpath file when new libs are added to the classpath through Ivy for 0.20-* based sources
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7248
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7248
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.3
>            Reporter: Konstantin Boudnik
>            Assignee: Thomas Graves
>            Priority: Minor
>             Fix For: 0.20.204.0, 0.20.205.0
>
>         Attachments: C7248-1.patch, HADOOP-7248-branch-20-security.patch
>
>
> Backport HADOOP-6407 into 0.20 based source trees

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16453-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 17:39:09 2011
Return-Path: <common-issues-return-16453-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BF7F14D04
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 17:39:09 +0000 (UTC)
Received: (qmail 70173 invoked by uid 500); 17 Jun 2011 17:39:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70073 invoked by uid 500); 17 Jun 2011 17:39:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69851 invoked by uid 99); 17 Jun 2011 17:39:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 17:39:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 17:39:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8DE8D41E98E
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 17:38:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 17:38:47 +0000 (UTC)
From: "Nicholas Telford (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1179062691.15731.1308332327578.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7269) S3 Native should allow customizable
 file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicholas Telford updated HADOOP-7269:
-------------------------------------

    Attachment: 7269-combined-002.patch

Fixed 2 additional compiler warnings. Thanks Todd for helping me track them down.

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: 0001-Added-support-for-metadata-to-be-applied-to-objects-.patch, 0002-Added-check-that-metadata-was-set-to-unit-test.patch, 7269-combined-002.patch, 7269-combined-proper.patch, 7269-combined.patch, HADOOP-7269-S3-metadata-001.diff, HADOOP-7269-S3-metadata-002.diff, HADOOP-7269-S3-metadata-003.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16454-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 17:55:11 2011
Return-Path: <common-issues-return-16454-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2BE384BC3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 17:55:11 +0000 (UTC)
Received: (qmail 93244 invoked by uid 500); 17 Jun 2011 17:55:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93165 invoked by uid 500); 17 Jun 2011 17:55:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93088 invoked by uid 99); 17 Jun 2011 17:55:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 17:55:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 17:55:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9BAD341E46C
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 17:54:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 17:54:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <998425100.15811.1308333287634.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7269) S3 Native should allow
 customizable file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051220#comment-13051220 ] 

Hadoop QA commented on HADOOP-7269:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482960/7269-combined-002.patch
  against trunk revision 1136249.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 8 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/647//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/647//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/647//console

This message is automatically generated.

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: 0001-Added-support-for-metadata-to-be-applied-to-objects-.patch, 0002-Added-check-that-metadata-was-set-to-unit-test.patch, 7269-combined-002.patch, 7269-combined-proper.patch, 7269-combined.patch, HADOOP-7269-S3-metadata-001.diff, HADOOP-7269-S3-metadata-002.diff, HADOOP-7269-S3-metadata-003.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16455-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 18:03:09 2011
Return-Path: <common-issues-return-16455-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 348884021
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 18:03:09 +0000 (UTC)
Received: (qmail 10223 invoked by uid 500); 17 Jun 2011 18:03:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10197 invoked by uid 500); 17 Jun 2011 18:03:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10188 invoked by uid 99); 17 Jun 2011 18:03:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 18:03:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 18:03:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B080A41E934
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 18:02:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 18:02:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <946948744.15845.1308333767719.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <7539864.45541295391523753.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7111) Several TFile tests failing when
 native libraries are present
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051227#comment-13051227 ] 

Todd Lipcon commented on HADOOP-7111:
-------------------------------------

Something in trunk just changed such that the native libs are on the classpath when the tests run, so the tests are now failing:

https://builds.apache.org/job/Hadoop-Common-trunk/721/testReport/junit/org.apache.hadoop.io.file.tfile/TestTFileByteArrays/testOneBlockPlusOneEntry/

> Several TFile tests failing when native libraries are present
> -------------------------------------------------------------
>
>                 Key: HADOOP-7111
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7111
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>
> When running tests with native libraries present, TestTFileByteArrays and TestTFileJClassComparatorByteArrays fail on trunk. They don't seem to fail in 0.20 with native libraries.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16456-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 18:48:11 2011
Return-Path: <common-issues-return-16456-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 74A124218
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 18:48:11 +0000 (UTC)
Received: (qmail 95226 invoked by uid 500); 17 Jun 2011 18:48:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95188 invoked by uid 500); 17 Jun 2011 18:48:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95180 invoked by uid 99); 17 Jun 2011 18:48:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 18:48:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 18:48:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 980E141EE19
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 18:47:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 18:47:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1758842790.15942.1308336467619.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7402) TestConfiguration doesn't clean up
 after itself
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

TestConfiguration doesn't clean up after itself
-----------------------------------------------

                 Key: HADOOP-7402
                 URL: https://issues.apache.org/jira/browse/HADOOP-7402
             Project: Hadoop Common
          Issue Type: Bug
          Components: test
    Affects Versions: 0.23.0
            Reporter: Aaron T. Myers
            Assignee: Aaron T. Myers
            Priority: Trivial
             Fix For: 0.23.0
         Attachments: hadoop-7402.0.patch

{{testGetFile}} and {{testGetLocalPath}} both create directories a, b, and c in the working directory from where the tests are run. They should clean up after themselves.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16457-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 18:48:11 2011
Return-Path: <common-issues-return-16457-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A6E594236
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 18:48:11 +0000 (UTC)
Received: (qmail 95398 invoked by uid 500); 17 Jun 2011 18:48:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95347 invoked by uid 500); 17 Jun 2011 18:48:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95292 invoked by uid 99); 17 Jun 2011 18:48:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 18:48:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 18:48:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AC78441EE1C
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 18:47:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 18:47:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <60905139.15945.1308336467703.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1758842790.15942.1308336467619.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7402) TestConfiguration doesn't clean up
 after itself
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7402:
-----------------------------------

    Attachment: hadoop-7402.0.patch

Patch which fixes the issue. I manually verified that the created directories are cleaned up after a run of this test.

> TestConfiguration doesn't clean up after itself
> -----------------------------------------------
>
>                 Key: HADOOP-7402
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7402
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7402.0.patch
>
>
> {{testGetFile}} and {{testGetLocalPath}} both create directories a, b, and c in the working directory from where the tests are run. They should clean up after themselves.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16458-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 18:56:09 2011
Return-Path: <common-issues-return-16458-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E8FF94641
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 18:56:08 +0000 (UTC)
Received: (qmail 2371 invoked by uid 500); 17 Jun 2011 18:56:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2305 invoked by uid 500); 17 Jun 2011 18:56:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2191 invoked by uid 99); 17 Jun 2011 18:56:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 18:56:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 18:56:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 904E541E2AA
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 18:55:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 18:55:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1714333275.15965.1308336947587.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7403) Fix SVN urls on site after unsplit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Fix SVN urls on site after unsplit
----------------------------------

                 Key: HADOOP-7403
                 URL: https://issues.apache.org/jira/browse/HADOOP-7403
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: site
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
         Attachments: hadoop-7403.txt

We need to update the version control URLs on the site

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16459-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 18:56:10 2011
Return-Path: <common-issues-return-16459-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C2E70466A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 18:56:10 +0000 (UTC)
Received: (qmail 2680 invoked by uid 500); 17 Jun 2011 18:56:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2634 invoked by uid 500); 17 Jun 2011 18:56:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2423 invoked by uid 99); 17 Jun 2011 18:56:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 18:56:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 18:56:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B282E41E2AF
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 18:55:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 18:55:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2047674763.15970.1308336947727.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1714333275.15965.1308336947587.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7403) Fix SVN urls on site after unsplit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7403:
--------------------------------

    Attachment: hadoop-7403.txt

Here's a fix. Someone please sanity-check I didn't flub a URL? :)

> Fix SVN urls on site after unsplit
> ----------------------------------
>
>                 Key: HADOOP-7403
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7403
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: site
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-7403.txt
>
>
> We need to update the version control URLs on the site

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16460-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 18:58:08 2011
Return-Path: <common-issues-return-16460-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9AB0F46B9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 18:58:08 +0000 (UTC)
Received: (qmail 6004 invoked by uid 500); 17 Jun 2011 18:58:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5977 invoked by uid 500); 17 Jun 2011 18:58:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5968 invoked by uid 99); 17 Jun 2011 18:58:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 18:58:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 18:58:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6317441E3EB
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 18:57:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 18:57:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <556104858.15972.1308337067402.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1758842790.15942.1308336467619.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7402) TestConfiguration doesn't clean up
 after itself
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051250#comment-13051250 ] 

Todd Lipcon commented on HADOOP-7402:
-------------------------------------

Rather than doing that, why not fix the test so that it chdirs into the build directory first? Or sets the conf to point to the build dir?

> TestConfiguration doesn't clean up after itself
> -----------------------------------------------
>
>                 Key: HADOOP-7402
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7402
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7402.0.patch
>
>
> {{testGetFile}} and {{testGetLocalPath}} both create directories a, b, and c in the working directory from where the tests are run. They should clean up after themselves.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16461-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 19:00:08 2011
Return-Path: <common-issues-return-16461-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DD22A47C7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 19:00:08 +0000 (UTC)
Received: (qmail 9851 invoked by uid 500); 17 Jun 2011 19:00:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9783 invoked by uid 500); 17 Jun 2011 19:00:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9775 invoked by uid 99); 17 Jun 2011 19:00:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 19:00:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 19:00:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A151E41E55B
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 18:59:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 18:59:47 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <778519518.15985.1308337187657.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1758842790.15942.1308336467619.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7402) TestConfiguration doesn't clean up
 after itself
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051253#comment-13051253 ] 

Alejandro Abdelnur commented on HADOOP-7402:
--------------------------------------------


Those test dirs should be written in the build directory, not to pollute the source tree. 

Using the Java Sys prop 'test.build.data' as the base for those dirs would do.

Thxs.


> TestConfiguration doesn't clean up after itself
> -----------------------------------------------
>
>                 Key: HADOOP-7402
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7402
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7402.0.patch
>
>
> {{testGetFile}} and {{testGetLocalPath}} both create directories a, b, and c in the working directory from where the tests are run. They should clean up after themselves.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16462-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 19:14:08 2011
Return-Path: <common-issues-return-16462-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C6E094327
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 19:14:08 +0000 (UTC)
Received: (qmail 35080 invoked by uid 500); 17 Jun 2011 19:14:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35048 invoked by uid 500); 17 Jun 2011 19:14:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35039 invoked by uid 99); 17 Jun 2011 19:14:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 19:14:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 19:14:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7E94541ED24
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 19:13:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 19:13:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <751034416.16022.1308338027515.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1758842790.15942.1308336467619.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7402) TestConfiguration doesn't clean up
 after itself
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7402:
-----------------------------------

    Attachment: hadoop-7402.1.patch

Patch which uses "test.build.data" as a prefix for the directories.

> TestConfiguration doesn't clean up after itself
> -----------------------------------------------
>
>                 Key: HADOOP-7402
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7402
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7402.0.patch, hadoop-7402.1.patch
>
>
> {{testGetFile}} and {{testGetLocalPath}} both create directories a, b, and c in the working directory from where the tests are run. They should clean up after themselves.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16463-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 19:14:11 2011
Return-Path: <common-issues-return-16463-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0F481433E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 19:14:11 +0000 (UTC)
Received: (qmail 35602 invoked by uid 500); 17 Jun 2011 19:14:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35570 invoked by uid 500); 17 Jun 2011 19:14:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35546 invoked by uid 99); 17 Jun 2011 19:14:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 19:14:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 19:14:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8C43741ED29
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 19:13:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 19:13:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1932809549.16024.1308338027571.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1758842790.15942.1308336467619.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7402) TestConfiguration doesn't clean up
 after itself
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7402:
-----------------------------------

    Status: Patch Available  (was: Open)

> TestConfiguration doesn't clean up after itself
> -----------------------------------------------
>
>                 Key: HADOOP-7402
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7402
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7402.0.patch, hadoop-7402.1.patch
>
>
> {{testGetFile}} and {{testGetLocalPath}} both create directories a, b, and c in the working directory from where the tests are run. They should clean up after themselves.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16464-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 19:34:11 2011
Return-Path: <common-issues-return-16464-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9D75544B3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 19:34:11 +0000 (UTC)
Received: (qmail 67043 invoked by uid 500); 17 Jun 2011 19:34:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66968 invoked by uid 500); 17 Jun 2011 19:34:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66869 invoked by uid 99); 17 Jun 2011 19:34:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 19:34:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 19:34:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0880D41E8E6
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 19:33:48 +0000 (UTC)
Date: Fri, 17 Jun 2011 19:33:48 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <547610371.16092.1308339228031.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1758842790.15942.1308336467619.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7402) TestConfiguration doesn't clean up
 after itself
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051276#comment-13051276 ] 

Hadoop QA commented on HADOOP-7402:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482974/hadoop-7402.1.patch
  against trunk revision 1136249.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/648//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/648//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/648//console

This message is automatically generated.

> TestConfiguration doesn't clean up after itself
> -----------------------------------------------
>
>                 Key: HADOOP-7402
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7402
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7402.0.patch, hadoop-7402.1.patch
>
>
> {{testGetFile}} and {{testGetLocalPath}} both create directories a, b, and c in the working directory from where the tests are run. They should clean up after themselves.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16465-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 20:18:09 2011
Return-Path: <common-issues-return-16465-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 570D94CD1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 20:18:09 +0000 (UTC)
Received: (qmail 34998 invoked by uid 500); 17 Jun 2011 20:18:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34952 invoked by uid 500); 17 Jun 2011 20:18:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34944 invoked by uid 99); 17 Jun 2011 20:18:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 20:18:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 20:18:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 20CEF41E35B
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 20:17:48 +0000 (UTC)
Date: Fri, 17 Jun 2011 20:17:48 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1255628176.16287.1308341868131.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1293648970.8758.1308177527351.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7398) create a mechanism to suppress the
 HADOOP_HOME deprecated warning
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7398:
----------------------------------

    Fix Version/s:     (was: 0.20.205.0)

> create a mechanism to suppress the HADOOP_HOME deprecated warning
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7398
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7398
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.204.0
>
>         Attachments: warn-204.patch, warn.patch, warn.patch
>
>
> Create a new mechanism to suppress the warning about HADOOP_HOME deprecation.
> I'll create a HADOOP_HOME_WARN_SUPPRESS environment variable that suppresses the warning.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16466-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 20:18:29 2011
Return-Path: <common-issues-return-16466-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 922C84EA9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 20:18:29 +0000 (UTC)
Received: (qmail 36175 invoked by uid 500); 17 Jun 2011 20:18:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36145 invoked by uid 500); 17 Jun 2011 20:18:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36136 invoked by uid 99); 17 Jun 2011 20:18:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 20:18:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 20:18:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4FED541E364
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 20:17:48 +0000 (UTC)
Date: Fri, 17 Jun 2011 20:17:48 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1391602129.16294.1308341868324.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <161449362.3682.1305134627390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7277) Add Eclipse launch tasks for the
 0.20-security branch
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7277:
----------------------------------

    Fix Version/s:     (was: 0.20.205.0)

> Add Eclipse launch tasks for the 0.20-security branch
> -----------------------------------------------------
>
>                 Key: HADOOP-7277
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7277
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.204.0
>            Reporter: Jeffrey Naisbitt
>            Assignee: Jeffrey Naisbitt
>            Priority: Minor
>             Fix For: 0.20.204.0
>
>         Attachments: HADOOP-7277-0.20s.patch
>
>
> This is to add the eclipse launchers from HADOOP-5911 to the 0.20 security branch.
> Eclipse has a notion of "run configuration", which encapsulates what's needed to run or debug an application. I use this quite a bit to start various Hadoop daemons in debug mode, with breakpoints set, to inspect state and what not.
> This is simply configuration, so no tests are provided. After running "ant eclipse" and refreshing your project, you should see entries in the Run Configurations and Debug Configurations for launching the various hadoop daemons from within eclipse. There's a template for testing a specific test, and also templates to run all the tests, the job tracker, and a task tracker. It's likely that some parameters need to be further tweaked to have the same behavior as "ant test", but for most tests, this works.
> This also requires a small change to build.xml for the eclipse classpath.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16468-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 20:18:30 2011
Return-Path: <common-issues-return-16468-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 70AA64ECC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 20:18:30 +0000 (UTC)
Received: (qmail 36491 invoked by uid 500); 17 Jun 2011 20:18:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36403 invoked by uid 500); 17 Jun 2011 20:18:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36386 invoked by uid 99); 17 Jun 2011 20:18:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 20:18:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 20:18:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 99C9441E373
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 20:17:48 +0000 (UTC)
Date: Fri, 17 Jun 2011 20:17:48 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2045330380.16304.1308341868626.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <285327525.9499.1304014323342.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7248) Have a way to automatically update
 Eclipse .classpath file when new libs are added to the classpath through
 Ivy for 0.20-* based sources
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7248:
----------------------------------

    Fix Version/s:     (was: 0.20.205.0)

> Have a way to automatically update Eclipse .classpath file when new libs are added to the classpath through Ivy for 0.20-* based sources
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7248
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7248
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.3
>            Reporter: Konstantin Boudnik
>            Assignee: Thomas Graves
>            Priority: Minor
>             Fix For: 0.20.204.0
>
>         Attachments: C7248-1.patch, HADOOP-7248-branch-20-security.patch
>
>
> Backport HADOOP-6407 into 0.20 based source trees

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16467-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 20:18:30 2011
Return-Path: <common-issues-return-16467-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8A8724EDB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 20:18:30 +0000 (UTC)
Received: (qmail 36446 invoked by uid 500); 17 Jun 2011 20:18:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36401 invoked by uid 500); 17 Jun 2011 20:18:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36385 invoked by uid 99); 17 Jun 2011 20:18:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 20:18:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 20:18:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6F48C41E368
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 20:17:48 +0000 (UTC)
Date: Fri, 17 Jun 2011 20:17:48 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <109098382.16298.1308341868452.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <88464990.3250.1305126467497.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7274) CLONE - IOUtils.readFully and
 IOUtils.skipFully have typo in exception creation's message
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7274:
----------------------------------

    Fix Version/s:     (was: 0.20.205.0)

> CLONE - IOUtils.readFully and IOUtils.skipFully have typo in exception creation's message
> -----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7274
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7274
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.20.204.0
>            Reporter: Jonathan Eagles
>            Assignee: Jonathan Eagles
>            Priority: Minor
>             Fix For: 0.20.204.0
>
>         Attachments: HADOOP-7274-0.20-security.patch
>
>
> Same fix as for HADOOP-7057 for the Hadoop security branch
> {noformat}
>         throw new IOException( "Premeture EOF from inputStream");
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16469-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 20:18:31 2011
Return-Path: <common-issues-return-16469-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E9CDB4F22
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 20:18:31 +0000 (UTC)
Received: (qmail 37151 invoked by uid 500); 17 Jun 2011 20:18:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37113 invoked by uid 500); 17 Jun 2011 20:18:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37092 invoked by uid 99); 17 Jun 2011 20:18:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 20:18:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 20:18:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3BA8D41E361
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 20:17:48 +0000 (UTC)
Date: Fri, 17 Jun 2011 20:17:48 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1222347692.16291.1308341868241.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <879608400.1240.1307478358811.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7364) TestMiniMRDFSCaching fails if
 test.build.dir is set to something other than build/test
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7364:
----------------------------------

    Fix Version/s:     (was: 0.20.205.0)

> TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7364
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7364
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.20.205.0
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.20.204.0
>
>         Attachments: HADOOP-7364-trunk.patch, HADOOP-7364.patch
>
>
> TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16470-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 20:18:32 2011
Return-Path: <common-issues-return-16470-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7067A4F37
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 20:18:32 +0000 (UTC)
Received: (qmail 37412 invoked by uid 500); 17 Jun 2011 20:18:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37376 invoked by uid 500); 17 Jun 2011 20:18:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37368 invoked by uid 99); 17 Jun 2011 20:18:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 20:18:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 20:18:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AECFF41E377
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 20:17:48 +0000 (UTC)
Date: Fri, 17 Jun 2011 20:17:48 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1664707300.16307.1308341868711.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7144:
----------------------------------

    Fix Version/s:     (was: 0.20.205.0)

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Robert Joseph Evans
>              Labels: jmx
>             Fix For: 0.20.204.0, 0.23.0
>
>         Attachments: HADOOP-7411-0.20.20X-V1.patch, HADOOP-7411-0.20.20X-V2.patch, HADOOP-7411-trunk-V1.patch, HADOOP-7411-trunk-V2.patch, HADOOP-7411-trunk-alpha.patch, jmx.json
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16471-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 20:23:11 2011
Return-Path: <common-issues-return-16471-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3752749C3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 20:23:11 +0000 (UTC)
Received: (qmail 42693 invoked by uid 500); 17 Jun 2011 20:23:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42647 invoked by uid 500); 17 Jun 2011 20:23:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42636 invoked by uid 99); 17 Jun 2011 20:23:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 20:23:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 20:23:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DD38E41E60B
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 20:22:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 20:22:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1178172617.16334.1308342167902.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <557825167.38099.1306198007453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7324) Ganglia plugins for metrics v2
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7324:
----------------------------------

    Fix Version/s:     (was: 0.20.204.0)

> Ganglia plugins for metrics v2
> ------------------------------
>
>                 Key: HADOOP-7324
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7324
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: metrics
>    Affects Versions: 0.20.203.0, 0.23.0
>            Reporter: Luke Lu
>             Fix For: 0.23.0
>
>
> Although, all metrics in metrics v2 are exposed via the standard JMX mechanisms, there are a few users who prefer using Ganglia to collect metrics.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16472-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 20:23:11 2011
Return-Path: <common-issues-return-16472-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 77F6F49D2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 20:23:11 +0000 (UTC)
Received: (qmail 42954 invoked by uid 500); 17 Jun 2011 20:23:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42886 invoked by uid 500); 17 Jun 2011 20:23:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42822 invoked by uid 99); 17 Jun 2011 20:23:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 20:23:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 20:23:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BB1D741E605
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 20:22:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 20:22:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1150959099.16329.1308342167763.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1145070693.28819.1305845387456.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7311) Port remaining metrics v1 from trunk
 to branch-0.20-security
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7311:
----------------------------------

    Fix Version/s:     (was: 0.20.204.0)

> Port remaining metrics v1 from trunk to branch-0.20-security
> ------------------------------------------------------------
>
>                 Key: HADOOP-7311
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7311
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.203.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>         Attachments: metrics-0.20-security.patch
>
>
> HADOOP-7190 added metrics packages/classes. This is a port from trunk to make them actually work for the security branch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16473-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 20:31:11 2011
Return-Path: <common-issues-return-16473-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 056BA4063
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 20:31:11 +0000 (UTC)
Received: (qmail 50739 invoked by uid 500); 17 Jun 2011 20:31:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50699 invoked by uid 500); 17 Jun 2011 20:31:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50691 invoked by uid 99); 17 Jun 2011 20:31:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 20:31:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 20:31:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8EB4741E955
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 20:30:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 20:30:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <455847575.16360.1308342647581.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1327657029.27279.1304663043230.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7262) Update CS docs with better example
 configs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7262:
----------------------------------

    Fix Version/s:     (was: 0.20.204.0)

> Update CS docs with better example configs
> ------------------------------------------
>
>                 Key: HADOOP-7262
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7262
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>            Priority: Minor
>         Attachments: HADOOP-7262.patch, capacity_scheduler.pdf
>
>
> It will be nice to enhance CS docs with real-world example configs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16474-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 20:39:14 2011
Return-Path: <common-issues-return-16474-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DFD194221
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 20:39:13 +0000 (UTC)
Received: (qmail 61684 invoked by uid 500); 17 Jun 2011 20:39:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61599 invoked by uid 500); 17 Jun 2011 20:39:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61553 invoked by uid 99); 17 Jun 2011 20:39:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 20:39:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 20:39:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 42C1141EC9C
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 20:38:49 +0000 (UTC)
Date: Fri, 17 Jun 2011 20:38:49 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1618155710.16415.1308343129270.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051313#comment-13051313 ] 

Alejandro Abdelnur commented on HADOOP-7206:
--------------------------------------------

Jake, I've tested your patch and it works. 

A few of comments on the patch:

* the change in build.xml seems unnecessary.
* SnappyCompressor.compress() method has a log INFO, it should be removed or it should be a log DEBUG within an if DEBUG_ENABLED block, and what is prints seems wrong (the reported compressed size is always bigger than the outBuf)


> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16475-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 21:18:10 2011
Return-Path: <common-issues-return-16475-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 85B654720
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 21:18:10 +0000 (UTC)
Received: (qmail 16907 invoked by uid 500); 17 Jun 2011 21:18:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16868 invoked by uid 500); 17 Jun 2011 21:18:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16858 invoked by uid 99); 17 Jun 2011 21:18:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 21:18:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 21:18:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 40B2B41EC34
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 21:17:49 +0000 (UTC)
Date: Fri, 17 Jun 2011 21:17:49 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1211221567.16500.1308345469261.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <158298796.897.1307467918824.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7363) TestRawLocalFileSystemContract is
 needed
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051327#comment-13051327 ] 

Matt Foley commented on HADOOP-7363:
------------------------------------

There is overlap between the intent of this ticket and the existing TestFSMainOperationsLocalFileSystem and FSMainOperationsBaseTest.

> TestRawLocalFileSystemContract is needed
> ----------------------------------------
>
>                 Key: HADOOP-7363
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7363
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>
> FileSystemContractBaseTest is supposed to be run with each concrete FileSystem implementation to insure adherence to the "contract" for FileSystem behavior.  However, currently only HDFS and S3 do so.  RawLocalFileSystem, at least, needs to be added. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16476-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 21:28:11 2011
Return-Path: <common-issues-return-16476-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 580554456
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 21:28:11 +0000 (UTC)
Received: (qmail 26738 invoked by uid 500); 17 Jun 2011 21:28:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26691 invoked by uid 500); 17 Jun 2011 21:28:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26676 invoked by uid 99); 17 Jun 2011 21:28:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 21:28:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 21:28:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A298541E14C
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 21:27:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 21:27:47 +0000 (UTC)
From: "monica beckwith (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <532987074.16533.1308346067662.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1229925666.15226.1308324167811.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7401) Unit test TestPureJavaCRC32 warmup
 code warms up the not-so-important loop in PureJavaCRC32.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

monica beckwith updated HADOOP-7401:
------------------------------------

    Attachment: HADOOP-7401.patch

> Unit test TestPureJavaCRC32 warmup code warms up the not-so-important loop in PureJavaCRC32.update()
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7401
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7401
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>         Environment: Solaris-Sparcv9; Solaris-AMD64
>            Reporter: monica beckwith
>            Priority: Minor
>         Attachments: HADOOP-7401.patch
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> When the warmup code sequence in TestPureJavaCRC32.java is executed, it sends size=len=2 and due to the value of 'trials' in for loop in doBench(), the crc.update() gets run > the compile threshold, thus providing the information that 'while 0<len<7' is a hot loop and 'while len>7' is a cold loop. This brings the MB/s number for len > 7 in PureJavaCRC32.update() way down (e.g. ~28.5% for size=len=65536).
> The workaround would be to use size=len=>7 (so just having size=len=2101 ahead of size=len=2 will do the trick) in the warmup section. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16477-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 21:30:10 2011
Return-Path: <common-issues-return-16477-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C81B3457E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 21:30:10 +0000 (UTC)
Received: (qmail 28029 invoked by uid 500); 17 Jun 2011 21:30:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27996 invoked by uid 500); 17 Jun 2011 21:30:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27988 invoked by uid 99); 17 Jun 2011 21:30:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 21:30:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 21:30:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8897A41E223
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 21:29:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 21:29:47 +0000 (UTC)
From: "monica beckwith (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <419025847.16553.1308346187556.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1229925666.15226.1308324167811.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7401) Unit test TestPureJavaCRC32 warmup
 code warms up the not-so-important loop in PureJavaCRC32.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

monica beckwith updated HADOOP-7401:
------------------------------------

    Fix Version/s: 0.21.0
     Release Note: fixed warmup in the perf test.
           Status: Patch Available  (was: Open)

> Unit test TestPureJavaCRC32 warmup code warms up the not-so-important loop in PureJavaCRC32.update()
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7401
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7401
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>         Environment: Solaris-Sparcv9; Solaris-AMD64
>            Reporter: monica beckwith
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-7401.patch
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> When the warmup code sequence in TestPureJavaCRC32.java is executed, it sends size=len=2 and due to the value of 'trials' in for loop in doBench(), the crc.update() gets run > the compile threshold, thus providing the information that 'while 0<len<7' is a hot loop and 'while len>7' is a cold loop. This brings the MB/s number for len > 7 in PureJavaCRC32.update() way down (e.g. ~28.5% for size=len=65536).
> The workaround would be to use size=len=>7 (so just having size=len=2101 ahead of size=len=2 will do the trick) in the warmup section. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16478-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 21:34:09 2011
Return-Path: <common-issues-return-16478-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1DA714A66
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 21:34:09 +0000 (UTC)
Received: (qmail 30326 invoked by uid 500); 17 Jun 2011 21:34:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30279 invoked by uid 500); 17 Jun 2011 21:34:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30271 invoked by uid 99); 17 Jun 2011 21:34:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 21:34:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 21:34:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AE19D41E4FE
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 21:33:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 21:33:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1703238116.16571.1308346427709.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1229925666.15226.1308324167811.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7401) Unit test TestPureJavaCRC32 warmup
 code warms up the not-so-important loop in PureJavaCRC32.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051336#comment-13051336 ] 

Hadoop QA commented on HADOOP-7401:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482990/HADOOP-7401.patch
  against trunk revision 1136249.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/649//console

This message is automatically generated.

> Unit test TestPureJavaCRC32 warmup code warms up the not-so-important loop in PureJavaCRC32.update()
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7401
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7401
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>         Environment: Solaris-Sparcv9; Solaris-AMD64
>            Reporter: monica beckwith
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-7401.patch
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> When the warmup code sequence in TestPureJavaCRC32.java is executed, it sends size=len=2 and due to the value of 'trials' in for loop in doBench(), the crc.update() gets run > the compile threshold, thus providing the information that 'while 0<len<7' is a hot loop and 'while len>7' is a cold loop. This brings the MB/s number for len > 7 in PureJavaCRC32.update() way down (e.g. ~28.5% for size=len=65536).
> The workaround would be to use size=len=>7 (so just having size=len=2101 ahead of size=len=2 will do the trick) in the warmup section. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16479-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 21:34:11 2011
Return-Path: <common-issues-return-16479-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5EB374AB7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 21:34:11 +0000 (UTC)
Received: (qmail 30698 invoked by uid 500); 17 Jun 2011 21:34:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30650 invoked by uid 500); 17 Jun 2011 21:34:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30606 invoked by uid 99); 17 Jun 2011 21:34:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 21:34:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 21:34:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 107B441E50E
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 21:33:48 +0000 (UTC)
Date: Fri, 17 Jun 2011 21:33:48 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1503965830.16584.1308346428064.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1593248389.10179.1308212149301.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7399) Remove Writable from ipc.Client and
 ipc.Server.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7399:
-----------------------------------------

    Attachment: HADOOP-7399.3.patch

This is a very early version of the patch. I still need to do lot more code cleanup. Sasl messages for connection setup also need to be handled consistently. The basic idea is to use ClientCall and ServerCall as the abstractions for interaction between an RpcEngine and ipc.Client/Server.

> Remove Writable from ipc.Client and ipc.Server.
> -----------------------------------------------
>
>                 Key: HADOOP-7399
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7399
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-7399.3.patch
>
>
>   This jira proposes to remove the dependency of ipc client and server on Writable. I think this will be an important step towards making an RpcEngine truly configurable, without having to use tunnel protocols as in case of AvroRPCEngine.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16480-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 21:36:09 2011
Return-Path: <common-issues-return-16480-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5C57A4BE6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 21:36:09 +0000 (UTC)
Received: (qmail 33968 invoked by uid 500); 17 Jun 2011 21:36:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33942 invoked by uid 500); 17 Jun 2011 21:36:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33934 invoked by uid 99); 17 Jun 2011 21:36:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 21:36:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 21:36:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 17BAA41E5AA
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 21:35:48 +0000 (UTC)
Date: Fri, 17 Jun 2011 21:35:48 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1193771986.16586.1308346548093.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1593248389.10179.1308212149301.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7399) Remove Writable from ipc.Client and
 ipc.Server.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7399:
-----------------------------------------

    Hadoop Flags: [Incompatible change]

> Remove Writable from ipc.Client and ipc.Server.
> -----------------------------------------------
>
>                 Key: HADOOP-7399
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7399
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-7399.3.patch
>
>
>   This jira proposes to remove the dependency of ipc client and server on Writable. I think this will be an important step towards making an RpcEngine truly configurable, without having to use tunnel protocols as in case of AvroRPCEngine.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16481-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 21:44:09 2011
Return-Path: <common-issues-return-16481-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EECF24D9A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 21:44:08 +0000 (UTC)
Received: (qmail 42856 invoked by uid 500); 17 Jun 2011 21:44:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42832 invoked by uid 500); 17 Jun 2011 21:44:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42824 invoked by uid 99); 17 Jun 2011 21:44:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 21:44:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 21:44:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A079441EA15
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 21:43:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 21:43:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1886630754.16614.1308347027653.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1229925666.15226.1308324167811.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7401) Unit test TestPureJavaCRC32 warmup
 code warms up the not-so-important loop in PureJavaCRC32.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE reassigned HADOOP-7401:
----------------------------------------------

    Assignee: monica beckwith

Hi Monica, this is an interesting discovery!  Could you also post the results before and after the patch?

BTW, you need to generate patches at the project root directory.  Otherwise, Hadoop QA could not apply it.

> Unit test TestPureJavaCRC32 warmup code warms up the not-so-important loop in PureJavaCRC32.update()
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7401
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7401
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>         Environment: Solaris-Sparcv9; Solaris-AMD64
>            Reporter: monica beckwith
>            Assignee: monica beckwith
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-7401.patch
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> When the warmup code sequence in TestPureJavaCRC32.java is executed, it sends size=len=2 and due to the value of 'trials' in for loop in doBench(), the crc.update() gets run > the compile threshold, thus providing the information that 'while 0<len<7' is a hot loop and 'while len>7' is a cold loop. This brings the MB/s number for len > 7 in PureJavaCRC32.update() way down (e.g. ~28.5% for size=len=65536).
> The workaround would be to use size=len=>7 (so just having size=len=2101 ahead of size=len=2 will do the trick) in the warmup section. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16482-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 21:51:10 2011
Return-Path: <common-issues-return-16482-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A0BB94657
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 21:51:10 +0000 (UTC)
Received: (qmail 54798 invoked by uid 500); 17 Jun 2011 21:51:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54717 invoked by uid 500); 17 Jun 2011 21:51:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54709 invoked by uid 99); 17 Jun 2011 21:51:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 21:51:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 21:51:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 49EBB41ED9F
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 21:50:49 +0000 (UTC)
Date: Fri, 17 Jun 2011 21:50:49 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <591176784.16694.1308347449299.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051346#comment-13051346 ] 

Alejandro Abdelnur commented on HADOOP-7206:
--------------------------------------------

Extra comment: indentation of java code should be 2 spaces, not 4

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16483-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 22:11:08 2011
Return-Path: <common-issues-return-16483-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D99BF4FA6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 22:11:08 +0000 (UTC)
Received: (qmail 73441 invoked by uid 500); 17 Jun 2011 22:11:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73410 invoked by uid 500); 17 Jun 2011 22:11:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73402 invoked by uid 99); 17 Jun 2011 22:11:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 22:11:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 22:11:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 72EE741F29C
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 22:10:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 22:10:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1650452057.16753.1308348647467.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <164637914.38616.1306225908771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7327) FileSystem.listStatus() throws
 NullPointerException instead of IOException upon access permission failure
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated HADOOP-7327:
-------------------------------

    Attachment: hadoop-7327-2.patch

Correct.

I've added a test case to FSMainOperationsBaseTest.  Also touched up a couple apparent errors in that unit test file, such as no @Test annotations on two testcases that work just fine and fast.

> FileSystem.listStatus() throws NullPointerException instead of IOException upon access permission failure
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7327
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7327
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>            Priority: Minor
>         Attachments: hadoop-7327-1.patch, hadoop-7327-2.patch
>
>
> Many processes that call listStatus() expect to handle IOException, but instead are getting runtime error NullPointerException, if the directory being scanned is visible but no-access to the running user id.  For example, if directory foo is drwxr-xr-x, and subdirectory foo/bar is drwx------, then trying to do listStatus(Path(foo/bar)) will cause a NullPointerException.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16484-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 22:25:09 2011
Return-Path: <common-issues-return-16484-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1E748493C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 22:25:09 +0000 (UTC)
Received: (qmail 87187 invoked by uid 500); 17 Jun 2011 22:25:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87158 invoked by uid 500); 17 Jun 2011 22:25:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87150 invoked by uid 99); 17 Jun 2011 22:25:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 22:25:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 22:25:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 78E0941F784
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 22:24:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 22:24:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <949687869.16798.1308349487492.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <164637914.38616.1306225908771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7327) FileSystem.listStatus() throws
 NullPointerException instead of IOException upon access permission failure
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051363#comment-13051363 ] 

Hadoop QA commented on HADOOP-7327:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482998/hadoop-7327-2.patch
  against trunk revision 1136249.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/650//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/650//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/650//console

This message is automatically generated.

> FileSystem.listStatus() throws NullPointerException instead of IOException upon access permission failure
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7327
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7327
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>            Priority: Minor
>         Attachments: hadoop-7327-1.patch, hadoop-7327-2.patch
>
>
> Many processes that call listStatus() expect to handle IOException, but instead are getting runtime error NullPointerException, if the directory being scanned is visible but no-access to the running user id.  For example, if directory foo is drwxr-xr-x, and subdirectory foo/bar is drwx------, then trying to do listStatus(Path(foo/bar)) will cause a NullPointerException.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16485-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 22:31:09 2011
Return-Path: <common-issues-return-16485-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E868F4F3F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 22:31:09 +0000 (UTC)
Received: (qmail 91272 invoked by uid 500); 17 Jun 2011 22:31:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91140 invoked by uid 500); 17 Jun 2011 22:31:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91030 invoked by uid 99); 17 Jun 2011 22:31:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 22:31:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 22:31:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A70B441F905
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 22:30:48 +0000 (UTC)
Date: Fri, 17 Jun 2011 22:30:48 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <833863948.16819.1308349848681.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1593248389.10179.1308212149301.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7399) Remove Writable from ipc.Client and
 ipc.Server.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7399:
-----------------------------------------

    Fix Version/s: 0.23.0
           Status: Patch Available  (was: Open)

> Remove Writable from ipc.Client and ipc.Server.
> -----------------------------------------------
>
>                 Key: HADOOP-7399
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7399
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7399.3.patch
>
>
>   This jira proposes to remove the dependency of ipc client and server on Writable. I think this will be an important step towards making an RpcEngine truly configurable, without having to use tunnel protocols as in case of AvroRPCEngine.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16486-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 22:35:12 2011
Return-Path: <common-issues-return-16486-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7FF6A4047
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 22:35:12 +0000 (UTC)
Received: (qmail 94787 invoked by uid 500); 17 Jun 2011 22:35:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94761 invoked by uid 500); 17 Jun 2011 22:35:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94752 invoked by uid 99); 17 Jun 2011 22:35:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 22:35:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 22:35:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1312441F9FA
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 22:34:49 +0000 (UTC)
Date: Fri, 17 Jun 2011 22:34:49 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2109073898.16828.1308350089074.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7326) TestTrash.testTrashEmptier()
 infinite loops if run on a home directory with stuff in .Trash
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051367#comment-13051367 ] 

Matt Foley commented on HADOOP-7326:
------------------------------------

FSMainOperationsBaseTest is a nice class that seems well-behaved for sharing between unit tests of multiple FileSystems, both Local and HDFS.  It uses FileSystemTestHelper for setting up test data root directories.


> TestTrash.testTrashEmptier() infinite loops if run on a home directory with stuff in .Trash
> -------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7326
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7326
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>         Attachments: hadoop-7326-TestTrash-1.patch
>
>
> This bug was discovered while investigating HDFS-1967, intermittent failure of TestHDFSTrash, which calls TestTrash.  If the user id running the test has a ~/.Trash directory that already has 4 or more files in it, a loop in TestTrash.testTrashEmptier() will never terminate, because it is waiting for the count of objects to increase from zero to exactly three (and it creates one object right away).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16487-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 22:53:10 2011
Return-Path: <common-issues-return-16487-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C906B49ED
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 22:53:10 +0000 (UTC)
Received: (qmail 15400 invoked by uid 500); 17 Jun 2011 22:53:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15369 invoked by uid 500); 17 Jun 2011 22:53:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15359 invoked by uid 99); 17 Jun 2011 22:53:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 22:53:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 22:53:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 61D2341FF76
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 22:52:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 22:52:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1492935609.16872.1308351167397.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1758842790.15942.1308336467619.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7402) TestConfiguration doesn't clean up
 after itself
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051377#comment-13051377 ] 

Eli Collins commented on HADOOP-7402:
-------------------------------------

+1

> TestConfiguration doesn't clean up after itself
> -----------------------------------------------
>
>                 Key: HADOOP-7402
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7402
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7402.0.patch, hadoop-7402.1.patch
>
>
> {{testGetFile}} and {{testGetLocalPath}} both create directories a, b, and c in the working directory from where the tests are run. They should clean up after themselves.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16488-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 22:55:11 2011
Return-Path: <common-issues-return-16488-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CFDA24AA3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 22:55:11 +0000 (UTC)
Received: (qmail 19394 invoked by uid 500); 17 Jun 2011 22:55:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19341 invoked by uid 500); 17 Jun 2011 22:55:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19316 invoked by uid 99); 17 Jun 2011 22:55:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 22:55:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 22:55:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B0A1241E1F0
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 22:54:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 22:54:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1855687681.16884.1308351287720.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1593248389.10179.1308212149301.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7399) Remove Writable from ipc.Client
 and ipc.Server.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051378#comment-13051378 ] 

Hadoop QA commented on HADOOP-7399:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482993/HADOOP-7399.3.patch
  against trunk revision 1136249.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The applied patch generated 1074 javac compiler warnings (more than the trunk's current 1067 warnings).

    -1 findbugs.  The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/651//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/651//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/651//console

This message is automatically generated.

> Remove Writable from ipc.Client and ipc.Server.
> -----------------------------------------------
>
>                 Key: HADOOP-7399
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7399
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7399.3.patch
>
>
>   This jira proposes to remove the dependency of ipc client and server on Writable. I think this will be an important step towards making an RpcEngine truly configurable, without having to use tunnel protocols as in case of AvroRPCEngine.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16489-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 22:55:12 2011
Return-Path: <common-issues-return-16489-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E8A214AAD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 22:55:11 +0000 (UTC)
Received: (qmail 19411 invoked by uid 500); 17 Jun 2011 22:55:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19343 invoked by uid 500); 17 Jun 2011 22:55:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19317 invoked by uid 99); 17 Jun 2011 22:55:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 22:55:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 22:55:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C58AC41E1F3
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 22:54:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 22:54:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <470816890.16887.1308351287806.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1758842790.15942.1308336467619.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7402) TestConfiguration doesn't clean up
 after itself
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7402:
--------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I've committed this. Thanks atm!

> TestConfiguration doesn't clean up after itself
> -----------------------------------------------
>
>                 Key: HADOOP-7402
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7402
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7402.0.patch, hadoop-7402.1.patch
>
>
> {{testGetFile}} and {{testGetLocalPath}} both create directories a, b, and c in the working directory from where the tests are run. They should clean up after themselves.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16490-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 17 23:40:08 2011
Return-Path: <common-issues-return-16490-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A216D4689
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 17 Jun 2011 23:40:08 +0000 (UTC)
Received: (qmail 64689 invoked by uid 500); 17 Jun 2011 23:40:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64658 invoked by uid 500); 17 Jun 2011 23:40:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64649 invoked by uid 99); 17 Jun 2011 23:40:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 23:40:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Jun 2011 23:40:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6DE0641F2B0
	for <common-issues@hadoop.apache.org>; Fri, 17 Jun 2011 23:39:47 +0000 (UTC)
Date: Fri, 17 Jun 2011 23:39:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <131346312.17013.1308353987447.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7326) TestTrash.testTrashEmptier()
 infinite loops if run on a home directory with stuff in .Trash
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051401#comment-13051401 ] 

Matt Foley commented on HADOOP-7326:
------------------------------------

I think I found the problem with TestTrash using one's personal .Trash folder instead of something from the "TestLFS" file system.  The main testcase, #testTrash() sets the TestLFS correctly, then calls #trashShell(FileSystem, Path), which throws away the config object in which the TestLFS was set, and instantiates a new one instead.  I've changed it to call fs.getConf() instead, thereby preserving the use of the TestLFS configuration.

> TestTrash.testTrashEmptier() infinite loops if run on a home directory with stuff in .Trash
> -------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7326
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7326
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>         Attachments: hadoop-7326-TestTrash-1.patch
>
>
> This bug was discovered while investigating HDFS-1967, intermittent failure of TestHDFSTrash, which calls TestTrash.  If the user id running the test has a ~/.Trash directory that already has 4 or more files in it, a loop in TestTrash.testTrashEmptier() will never terminate, because it is waiting for the count of objects to increase from zero to exactly three (and it creates one object right away).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16491-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 00:48:11 2011
Return-Path: <common-issues-return-16491-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 770104170
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 18 Jun 2011 00:48:11 +0000 (UTC)
Received: (qmail 13287 invoked by uid 500); 18 Jun 2011 00:48:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13254 invoked by uid 500); 18 Jun 2011 00:48:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13243 invoked by uid 99); 18 Jun 2011 00:48:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 00:48:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 00:48:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2492F41C06A
	for <common-issues@hadoop.apache.org>; Sat, 18 Jun 2011 00:47:50 +0000 (UTC)
Date: Sat, 18 Jun 2011 00:47:50 +0000 (UTC)
From: "T Jake Luciani (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <257859789.17144.1308358070146.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

T Jake Luciani updated HADOOP-7206:
-----------------------------------

    Attachment: v4-HADOOP-7206-snappy-codec-using-snappy-java.txt

Thanks for the feedback. Incorporated those changes into v4

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16492-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 01:00:13 2011
Return-Path: <common-issues-return-16492-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C1F484CCB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 18 Jun 2011 01:00:13 +0000 (UTC)
Received: (qmail 26720 invoked by uid 500); 18 Jun 2011 01:00:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26657 invoked by uid 500); 18 Jun 2011 01:00:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26457 invoked by uid 99); 18 Jun 2011 01:00:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 01:00:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 01:00:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3604841C40D
	for <common-issues@hadoop.apache.org>; Sat, 18 Jun 2011 00:59:49 +0000 (UTC)
Date: Sat, 18 Jun 2011 00:59:49 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <160595976.17212.1308358789217.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1714333275.15965.1308336947587.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7403) Fix SVN urls on site after unsplit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon resolved HADOOP-7403.
---------------------------------

    Resolution: Duplicate

apparently Luke fixed this while I was working on the patch :)

> Fix SVN urls on site after unsplit
> ----------------------------------
>
>                 Key: HADOOP-7403
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7403
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: site
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-7403.txt
>
>
> We need to update the version control URLs on the site

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16493-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 01:00:14 2011
Return-Path: <common-issues-return-16493-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 118F14CE1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 18 Jun 2011 01:00:14 +0000 (UTC)
Received: (qmail 26866 invoked by uid 500); 18 Jun 2011 01:00:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26847 invoked by uid 500); 18 Jun 2011 01:00:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26838 invoked by uid 99); 18 Jun 2011 01:00:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 01:00:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 01:00:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 19DF041C409
	for <common-issues@hadoop.apache.org>; Sat, 18 Jun 2011 00:59:49 +0000 (UTC)
Date: Sat, 18 Jun 2011 00:59:49 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1184942442.17208.1308358789102.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1714333275.15965.1308336947587.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7403) Fix SVN urls on site after unsplit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051424#comment-13051424 ] 

Aaron T. Myers commented on HADOOP-7403:
----------------------------------------

+1 looks good to me.

> Fix SVN urls on site after unsplit
> ----------------------------------
>
>                 Key: HADOOP-7403
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7403
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: site
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-7403.txt
>
>
> We need to update the version control URLs on the site

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16494-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 01:00:14 2011
Return-Path: <common-issues-return-16494-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CF24F4D05
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 18 Jun 2011 01:00:14 +0000 (UTC)
Received: (qmail 27495 invoked by uid 500); 18 Jun 2011 01:00:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27454 invoked by uid 500); 18 Jun 2011 01:00:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27216 invoked by uid 99); 18 Jun 2011 01:00:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 01:00:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 01:00:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0592241C406
	for <common-issues@hadoop.apache.org>; Sat, 18 Jun 2011 00:59:49 +0000 (UTC)
Date: Sat, 18 Jun 2011 00:59:49 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <526744128.17205.1308358789019.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051423#comment-13051423 ] 

Tom White commented on HADOOP-7206:
-----------------------------------

I noticed that the compression overhead in this patch is {{(bufferSize >> 3) + 128 + 3}} which is less than the maximum possible blowup that Snappy allows for (http://code.google.com/p/snappy/source/browse/trunk/snappy.cc#55). Should this be changed to {{bufferSize / 6 + 32}}?

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16495-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 01:04:08 2011
Return-Path: <common-issues-return-16495-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C219D41CA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 18 Jun 2011 01:04:08 +0000 (UTC)
Received: (qmail 32034 invoked by uid 500); 18 Jun 2011 01:04:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31997 invoked by uid 500); 18 Jun 2011 01:04:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31989 invoked by uid 99); 18 Jun 2011 01:04:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 01:04:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 01:04:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 733C441C4F0
	for <common-issues@hadoop.apache.org>; Sat, 18 Jun 2011 01:03:47 +0000 (UTC)
Date: Sat, 18 Jun 2011 01:03:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <750698678.17238.1308359027468.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051426#comment-13051426 ] 

Hadoop QA commented on HADOOP-7206:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12483020/v4-HADOOP-7206-snappy-codec-using-snappy-java.txt
  against trunk revision 1137065.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/652//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/652//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/652//console

This message is automatically generated.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16496-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 01:20:12 2011
Return-Path: <common-issues-return-16496-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1A5C54146
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 18 Jun 2011 01:20:12 +0000 (UTC)
Received: (qmail 51179 invoked by uid 500); 18 Jun 2011 01:20:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51142 invoked by uid 500); 18 Jun 2011 01:20:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51134 invoked by uid 99); 18 Jun 2011 01:20:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 01:20:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 01:20:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A3F0B41C8B5
	for <common-issues@hadoop.apache.org>; Sat, 18 Jun 2011 01:19:50 +0000 (UTC)
Date: Sat, 18 Jun 2011 01:19:50 +0000 (UTC)
From: "T Jake Luciani (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <492679810.17347.1308359990668.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051429#comment-13051429 ] 

T Jake Luciani commented on HADOOP-7206:
----------------------------------------

Hi Tom.  Good find. Actually snappy-java exposes this call so perhaps it should change to 

{noformat}
Snappy.maxCompressedLength(bufferSize) - bufferSize;
{noformat}

Agree?

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16497-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 01:40:09 2011
Return-Path: <common-issues-return-16497-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4B3764C31
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 18 Jun 2011 01:40:09 +0000 (UTC)
Received: (qmail 63269 invoked by uid 500); 18 Jun 2011 01:40:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63238 invoked by uid 500); 18 Jun 2011 01:40:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63230 invoked by uid 99); 18 Jun 2011 01:40:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 01:40:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 01:40:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 066E941CF43
	for <common-issues@hadoop.apache.org>; Sat, 18 Jun 2011 01:39:48 +0000 (UTC)
Date: Sat, 18 Jun 2011 01:39:48 +0000 (UTC)
From: "T Jake Luciani (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <778622023.17385.1308361188023.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

T Jake Luciani updated HADOOP-7206:
-----------------------------------

    Attachment: v5-HADOOP-7206-snappy-codec-using-snappy-java.txt

v5 includes mentioned change

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16498-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 01:54:09 2011
Return-Path: <common-issues-return-16498-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 31AEA4686
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 18 Jun 2011 01:54:09 +0000 (UTC)
Received: (qmail 70589 invoked by uid 500); 18 Jun 2011 01:54:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70555 invoked by uid 500); 18 Jun 2011 01:54:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70547 invoked by uid 99); 18 Jun 2011 01:54:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 01:54:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 01:54:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7504441F2A7
	for <common-issues@hadoop.apache.org>; Sat, 18 Jun 2011 01:53:47 +0000 (UTC)
Date: Sat, 18 Jun 2011 01:53:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <370337148.17443.1308362027476.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051436#comment-13051436 ] 

Hadoop QA commented on HADOOP-7206:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12483021/v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
  against trunk revision 1137065.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/653//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/653//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/653//console

This message is automatically generated.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: issei yoshida
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16499-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 12:54:10 2011
Return-Path: <common-issues-return-16499-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 60F796836
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 18 Jun 2011 12:54:10 +0000 (UTC)
Received: (qmail 17661 invoked by uid 500); 18 Jun 2011 12:54:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17604 invoked by uid 500); 18 Jun 2011 12:54:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17440 invoked by uid 99); 18 Jun 2011 12:54:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 12:54:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 12:54:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B8BA3420CAC
	for <common-issues@hadoop.apache.org>; Sat, 18 Jun 2011 12:53:47 +0000 (UTC)
Date: Sat, 18 Jun 2011 12:53:47 +0000 (UTC)
From: =?utf-8?Q?Celina_d=C2=B4_=C3=81vila_Samogin_=28JIRA=29?= <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <633001546.18025.1308401627753.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6846) Scripts for building Hadoop 0.22.0
 release
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6846?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D130=
51509#comment-13051509 ]=20

Celina d=C2=B4 =C3=81vila Samogin commented on HADOOP-6846:
-------------------------------------------------

After many tests, I submit some scripts to generate the hadoop tarball with=
out documentation and no source code, only for 32 bit binary code. I have u=
sed in my master's job.

With these scripts, you can get a hadoop tarball with modified and compiled=
 source code for testing.

The scripts were based on patches HADOOP-6342 and HADOOP-6846.

https://github.com/celinasam/Generate-Hadoop-Tarball

> Scripts for building Hadoop 0.22.0 release
> ------------------------------------------
>
>                 Key: HADOOP-6846
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6846
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6846.patch, release-scripts.tar.gz
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

From common-issues-return-16500-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 17:02:12 2011
Return-Path: <common-issues-return-16500-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 65C4D6BEE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 18 Jun 2011 17:02:12 +0000 (UTC)
Received: (qmail 78842 invoked by uid 500); 18 Jun 2011 17:02:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78809 invoked by uid 500); 18 Jun 2011 17:02:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78800 invoked by uid 99); 18 Jun 2011 17:02:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 17:02:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 17:02:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 02188420EEF
	for <common-issues@hadoop.apache.org>; Sat, 18 Jun 2011 17:01:48 +0000 (UTC)
Date: Sat, 18 Jun 2011 17:01:48 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <926418756.18277.1308416508005.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1022068843.6096.1308124428641.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7396) The information returned by the
 wrong usage of the command "hadoop job -events <job-id> <from-event-#>
 <#-of-events>" is not appropriate
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J resolved HADOOP-7396.
-----------------------------

    Resolution: Duplicate

> The information returned by the wrong usage of the command "hadoop job -events <job-id> <from-event-#> <#-of-events>" is not appropriate
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7396
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7396
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Yan Jinshuang
>            Priority: Minor
>             Fix For: 0.23.0
>
>
> With wrong value of from-event-# and #-of-events, though the from-events-# after the #-of-events for example from 1000 to 1, the command always return 0.It is expected to show detailed information, like "the start number should be less than the end number for range of events".

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16501-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 17:06:10 2011
Return-Path: <common-issues-return-16501-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ABFB767CB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 18 Jun 2011 17:06:10 +0000 (UTC)
Received: (qmail 83880 invoked by uid 500); 18 Jun 2011 17:06:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83849 invoked by uid 500); 18 Jun 2011 17:06:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83841 invoked by uid 99); 18 Jun 2011 17:06:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 17:06:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 17:06:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 689D24200DC
	for <common-issues@hadoop.apache.org>; Sat, 18 Jun 2011 17:05:47 +0000 (UTC)
Date: Sat, 18 Jun 2011 17:05:47 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <425713541.18294.1308416747425.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1597399507.67325.1307137547450.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7358) Improve log levels when exceptions
 caught in RPC handler
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J updated HADOOP-7358:
----------------------------

    Status: Patch Available  (was: Open)

(Marking as PA since Todd may have forgotten to -- For Hudson)

> Improve log levels when exceptions caught in RPC handler
> --------------------------------------------------------
>
>                 Key: HADOOP-7358
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7358
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7358.txt
>
>
> When a server implementation throws an exception handling an RPC, the Handler thread catches it and logs it before responding with the exception over the IPC channel. This is currently done at INFO level.
> I'd like to propose that, if the exception is an unchecked exception, it should be logged at WARN level instead. This can help alert operators when they might be hitting some kind of bug.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16502-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 17:26:10 2011
Return-Path: <common-issues-return-16502-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AFCCF6EF1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 18 Jun 2011 17:26:10 +0000 (UTC)
Received: (qmail 97239 invoked by uid 500); 18 Jun 2011 17:26:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97209 invoked by uid 500); 18 Jun 2011 17:26:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97198 invoked by uid 99); 18 Jun 2011 17:26:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 17:26:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 17:26:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 686A94204F3
	for <common-issues@hadoop.apache.org>; Sat, 18 Jun 2011 17:25:47 +0000 (UTC)
Date: Sat, 18 Jun 2011 17:25:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1098345397.18317.1308417947424.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1597399507.67325.1307137547450.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7358) Improve log levels when exceptions
 caught in RPC handler
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051563#comment-13051563 ] 

Hadoop QA commented on HADOOP-7358:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481410/hadoop-7358.txt
  against trunk revision 1137065.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/654//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/654//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/654//console

This message is automatically generated.

> Improve log levels when exceptions caught in RPC handler
> --------------------------------------------------------
>
>                 Key: HADOOP-7358
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7358
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7358.txt
>
>
> When a server implementation throws an exception handling an RPC, the Handler thread catches it and logs it before responding with the exception over the IPC channel. This is currently done at INFO level.
> I'd like to propose that, if the exception is an unchecked exception, it should be logged at WARN level instead. This can help alert operators when they might be hitting some kind of bug.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16503-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 17:28:08 2011
Return-Path: <common-issues-return-16503-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C9D1F6F31
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 18 Jun 2011 17:28:08 +0000 (UTC)
Received: (qmail 98737 invoked by uid 500); 18 Jun 2011 17:28:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98707 invoked by uid 500); 18 Jun 2011 17:28:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98698 invoked by uid 99); 18 Jun 2011 17:28:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 17:28:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 17:28:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9CBF54205AA
	for <common-issues@hadoop.apache.org>; Sat, 18 Jun 2011 17:27:47 +0000 (UTC)
Date: Sat, 18 Jun 2011 17:27:47 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <661700138.18324.1308418067639.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1838851750.12019.1304096043285.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7252) JUnit shows up as a compile time
 dependency
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J reassigned HADOOP-7252:
-------------------------------

    Assignee: Harsh J

> JUnit shows up as a compile time dependency
> -------------------------------------------
>
>                 Key: HADOOP-7252
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7252
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build, conf, test
>    Affects Versions: 0.20.2
>            Reporter: Pony
>            Assignee: Harsh J
>            Priority: Minor
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16504-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 17:28:09 2011
Return-Path: <common-issues-return-16504-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 04F276F3F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 18 Jun 2011 17:28:09 +0000 (UTC)
Received: (qmail 98983 invoked by uid 500); 18 Jun 2011 17:28:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98948 invoked by uid 500); 18 Jun 2011 17:28:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98776 invoked by uid 99); 18 Jun 2011 17:28:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 17:28:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 17:28:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 84B2D4205A7
	for <common-issues@hadoop.apache.org>; Sat, 18 Jun 2011 17:27:47 +0000 (UTC)
Date: Sat, 18 Jun 2011 17:27:47 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1197960407.18321.1308418067540.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1838851750.12019.1304096043285.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7252) JUnit shows up as a compile time
 dependency
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7252?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D130=
51564#comment-13051564 ]=20

Harsh J commented on HADOOP-7252:
---------------------------------

This appears to be fixed on trunk, but remains "common->default" on 0.22. C=
ould be fixed for 0.22 also.

{{git log}} shows Hadoop moves from OO'M last, so can't tell which JIRA her=
e=E2=80=A6

> JUnit shows up as a compile time dependency
> -------------------------------------------
>
>                 Key: HADOOP-7252
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7252
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build, conf, test
>    Affects Versions: 0.20.2
>            Reporter: Pony
>            Priority: Minor
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

From common-issues-return-16505-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 17:34:08 2011
Return-Path: <common-issues-return-16505-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A020A63A6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 18 Jun 2011 17:34:08 +0000 (UTC)
Received: (qmail 3414 invoked by uid 500); 18 Jun 2011 17:34:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3380 invoked by uid 500); 18 Jun 2011 17:34:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3368 invoked by uid 99); 18 Jun 2011 17:34:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 17:34:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 17:34:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 746EC420826
	for <common-issues@hadoop.apache.org>; Sat, 18 Jun 2011 17:33:47 +0000 (UTC)
Date: Sat, 18 Jun 2011 17:33:47 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1584164643.18330.1308418427473.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1838851750.12019.1304096043285.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7252) JUnit shows up as a compile time
 dependency
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J updated HADOOP-7252:
----------------------------

    Attachment: HADOOP-7252bp22.r1.diff

Fix against 0.22 branch.

Patch *not* intended for Trunk/0.23

> JUnit shows up as a compile time dependency
> -------------------------------------------
>
>                 Key: HADOOP-7252
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7252
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build, conf, test
>    Affects Versions: 0.20.2
>            Reporter: Pony
>            Assignee: Harsh J
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7252bp22.r1.diff
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16506-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 17:34:08 2011
Return-Path: <common-issues-return-16506-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CF6B463B6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 18 Jun 2011 17:34:08 +0000 (UTC)
Received: (qmail 3699 invoked by uid 500); 18 Jun 2011 17:34:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3623 invoked by uid 500); 18 Jun 2011 17:34:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3498 invoked by uid 99); 18 Jun 2011 17:34:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 17:34:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 17:34:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8FB2942082C
	for <common-issues@hadoop.apache.org>; Sat, 18 Jun 2011 17:33:47 +0000 (UTC)
Date: Sat, 18 Jun 2011 17:33:47 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1074630760.18334.1308418427585.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1838851750.12019.1304096043285.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7252) JUnit shows up as a compile time
 dependency
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J updated HADOOP-7252:
----------------------------

    Fix Version/s: 0.22.0
           Status: Patch Available  (was: Open)

Patch available for 0.22.

(Please ignore Hudson run, since that would run for trunk where this appears already fixed)

> JUnit shows up as a compile time dependency
> -------------------------------------------
>
>                 Key: HADOOP-7252
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7252
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build, conf, test
>    Affects Versions: 0.20.2
>            Reporter: Pony
>            Assignee: Harsh J
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7252bp22.r1.diff
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16507-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 17:44:08 2011
Return-Path: <common-issues-return-16507-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C958E673B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 18 Jun 2011 17:44:08 +0000 (UTC)
Received: (qmail 9918 invoked by uid 500); 18 Jun 2011 17:44:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9887 invoked by uid 500); 18 Jun 2011 17:44:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9879 invoked by uid 99); 18 Jun 2011 17:44:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 17:44:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 17:44:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 94F664209DD
	for <common-issues@hadoop.apache.org>; Sat, 18 Jun 2011 17:43:47 +0000 (UTC)
Date: Sat, 18 Jun 2011 17:43:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1868372683.18339.1308419027607.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1838851750.12019.1304096043285.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7252) JUnit shows up as a compile time
 dependency
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051571#comment-13051571 ] 

Hadoop QA commented on HADOOP-7252:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12483056/HADOOP-7252bp22.r1.diff
  against trunk revision 1137065.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/655//console

This message is automatically generated.

> JUnit shows up as a compile time dependency
> -------------------------------------------
>
>                 Key: HADOOP-7252
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7252
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build, conf, test
>    Affects Versions: 0.20.2
>            Reporter: Pony
>            Assignee: Harsh J
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7252bp22.r1.diff
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16508-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 18:14:10 2011
Return-Path: <common-issues-return-16508-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BBA65635B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 18 Jun 2011 18:14:10 +0000 (UTC)
Received: (qmail 22194 invoked by uid 500); 18 Jun 2011 18:14:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22153 invoked by uid 500); 18 Jun 2011 18:14:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22145 invoked by uid 99); 18 Jun 2011 18:14:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 18:14:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 18:14:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 568B3420E3A
	for <common-issues@hadoop.apache.org>; Sat, 18 Jun 2011 18:13:47 +0000 (UTC)
Date: Sat, 18 Jun 2011 18:13:47 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <865314287.18346.1308420827351.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6804) allow
 FileSystem.copyFromLocalFile() to execute under specified username
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051573#comment-13051573 ] 

Harsh J commented on HADOOP-6804:
---------------------------------

Ted - Is http://hadoop.apache.org/common/docs/current/Secure_Impersonation.html what you're looking for?

> allow FileSystem.copyFromLocalFile() to execute under specified username
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6804
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6804
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.20.2
>            Reporter: Ted Yu
>            Priority: Minor
>
> When the user calling FileSystem.copyFromLocalFile() doesn't have permission to write to certain hdfs path:
> Thread [main] (Suspended (exception AccessControlException))   
>     DFSClient.mkdirs(String, FsPermission) line: 905   
>     DistributedFileSystem.mkdirs(Path, FsPermission) line: 262   
>     DistributedFileSystem(FileSystem).mkdirs(Path) line: 1162   
>     FileUtil.copy(FileSystem, Path, FileSystem, Path, boolean, boolean, Configuration) line: 194   
>     DistributedFileSystem(FileSystem).copyFromLocalFile(boolean, boolean, Path, Path) line: 1231   
>     DistributedFileSystem(FileSystem).copyFromLocalFile(boolean, Path, Path) line: 1207   
>     DistributedFileSystem(FileSystem).copyFromLocalFile(Path, Path) line: 1179   
>     GridM2mInstallation.copyInputFiles(FlowConfigurations$FlowConf) line: 380   
> Passwordless ssh has been setup for current user, tyu, on localhost and user hadoop on hdfs
> FileSystem.copyFromLocalFile() should be able to execute using a different username than effective user on localhost so that AccessControlException can be avoided.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16509-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 18:18:10 2011
Return-Path: <common-issues-return-16509-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B7B116F1D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 18 Jun 2011 18:18:10 +0000 (UTC)
Received: (qmail 23014 invoked by uid 500); 18 Jun 2011 18:18:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22982 invoked by uid 500); 18 Jun 2011 18:18:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22974 invoked by uid 99); 18 Jun 2011 18:18:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 18:18:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 18:18:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 57668420EF1
	for <common-issues@hadoop.apache.org>; Sat, 18 Jun 2011 18:17:47 +0000 (UTC)
Date: Sat, 18 Jun 2011 18:17:47 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <900011000.18348.1308421067354.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6840) Support non-recursive create() in
 FileSystem & SequenceFile.Writer
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J updated HADOOP-6840:
----------------------------

    Fix Version/s: 0.21.1
                   0.20-append

(Adding in Fix versions to track)

> Support non-recursive create() in FileSystem & SequenceFile.Writer
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6840
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6840
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs, io
>    Affects Versions: 0.20-append, 0.21.0
>            Reporter: Nicolas Spiegelberg
>            Priority: Minor
>             Fix For: 0.20-append, 0.21.1
>
>         Attachments: HADOOP-6840_0.20-append.patch, HADOOP-6840_0.21-2.patch, HADOOP-6840_0.21.patch
>
>
> The proposed solution for HBASE-2312 requires the sequence file to handle a non-recursive create.  This is already supported by HDFS, but needs to have an equivalent FileSystem & SequenceFile.Writer API.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16510-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 18:26:08 2011
Return-Path: <common-issues-return-16510-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8CC16699C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 18 Jun 2011 18:26:08 +0000 (UTC)
Received: (qmail 30720 invoked by uid 500); 18 Jun 2011 18:26:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30696 invoked by uid 500); 18 Jun 2011 18:26:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30688 invoked by uid 99); 18 Jun 2011 18:26:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 18:26:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 18:26:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 569E1420100
	for <common-issues@hadoop.apache.org>; Sat, 18 Jun 2011 18:25:47 +0000 (UTC)
Date: Sat, 18 Jun 2011 18:25:47 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <965386948.18365.1308421547351.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6801) io.sort.mb and io.sort.factor were
 renamed and moved to mapreduce but are still in
 CommonConfigurationKeysPublic.java and used in SequenceFile.java
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051577#comment-13051577 ] 

Harsh J commented on HADOOP-6801:
---------------------------------

One solution would be to make these options sequence-file specific since they can't obviously be related to MapReduce here?

> io.sort.mb and io.sort.factor were renamed and moved to mapreduce but are still in CommonConfigurationKeysPublic.java and used in SequenceFile.java
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6801
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6801
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Priority: Minor
>
> Following configuration keys in CommonConfigurationKeysPublic.java (former CommonConfigurationKeys.java):
> public static final String  IO_SORT_MB_KEY = "io.sort.mb";
> public static final String  IO_SORT_FACTOR_KEY = "io.sort.factor";
> are partially moved:
>   - they were renamed to mapreduce.task.io.sort.mb and mapreduce.task.io.sort.factor respectively
>   - they were moved to mapreduce project, documented in mapred-default.xml
> However:
>   - they are still listed in CommonConfigurationKeysPublic.java as quoted above
>   - strings "io.sort.mb" and "io.sort.factor" are used in SequenceFile.java in Hadoop Common project
> Not sure what the solution is, these constants should probably be removed from CommonConfigurationKeysPublic.java but I am not sure what's the best solution for SequenceFile.java.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16511-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 18:28:09 2011
Return-Path: <common-issues-return-16511-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 09E306A91
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 18 Jun 2011 18:28:09 +0000 (UTC)
Received: (qmail 34427 invoked by uid 500); 18 Jun 2011 18:28:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34399 invoked by uid 500); 18 Jun 2011 18:28:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34390 invoked by uid 99); 18 Jun 2011 18:28:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 18:28:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 18:28:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BA8DB420198
	for <common-issues@hadoop.apache.org>; Sat, 18 Jun 2011 18:27:47 +0000 (UTC)
Date: Sat, 18 Jun 2011 18:27:47 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1266023974.18373.1308421667760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6801) io.sort.mb and io.sort.factor were
 renamed and moved to mapreduce but are still in
 CommonConfigurationKeysPublic.java and used in SequenceFile.java
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051578#comment-13051578 ] 

Harsh J commented on HADOOP-6801:
---------------------------------

Or perhaps add Sorter constructors that let one specify, and fall back to a documented default instead of a configuration carried one.

> io.sort.mb and io.sort.factor were renamed and moved to mapreduce but are still in CommonConfigurationKeysPublic.java and used in SequenceFile.java
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6801
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6801
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Priority: Minor
>
> Following configuration keys in CommonConfigurationKeysPublic.java (former CommonConfigurationKeys.java):
> public static final String  IO_SORT_MB_KEY = "io.sort.mb";
> public static final String  IO_SORT_FACTOR_KEY = "io.sort.factor";
> are partially moved:
>   - they were renamed to mapreduce.task.io.sort.mb and mapreduce.task.io.sort.factor respectively
>   - they were moved to mapreduce project, documented in mapred-default.xml
> However:
>   - they are still listed in CommonConfigurationKeysPublic.java as quoted above
>   - strings "io.sort.mb" and "io.sort.factor" are used in SequenceFile.java in Hadoop Common project
> Not sure what the solution is, these constants should probably be removed from CommonConfigurationKeysPublic.java but I am not sure what's the best solution for SequenceFile.java.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16512-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 18:28:09 2011
Return-Path: <common-issues-return-16512-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 321706A9E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 18 Jun 2011 18:28:09 +0000 (UTC)
Received: (qmail 34622 invoked by uid 500); 18 Jun 2011 18:28:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34574 invoked by uid 500); 18 Jun 2011 18:28:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34470 invoked by uid 99); 18 Jun 2011 18:28:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 18:28:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 18:28:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CD46A42019B
	for <common-issues@hadoop.apache.org>; Sat, 18 Jun 2011 18:27:47 +0000 (UTC)
Date: Sat, 18 Jun 2011 18:27:47 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1676983658.18376.1308421667837.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-6801) io.sort.mb and io.sort.factor were
 renamed and moved to mapreduce but are still in
 CommonConfigurationKeysPublic.java and used in SequenceFile.java
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J reassigned HADOOP-6801:
-------------------------------

    Assignee: Harsh J

> io.sort.mb and io.sort.factor were renamed and moved to mapreduce but are still in CommonConfigurationKeysPublic.java and used in SequenceFile.java
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6801
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6801
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Harsh J
>            Priority: Minor
>
> Following configuration keys in CommonConfigurationKeysPublic.java (former CommonConfigurationKeys.java):
> public static final String  IO_SORT_MB_KEY = "io.sort.mb";
> public static final String  IO_SORT_FACTOR_KEY = "io.sort.factor";
> are partially moved:
>   - they were renamed to mapreduce.task.io.sort.mb and mapreduce.task.io.sort.factor respectively
>   - they were moved to mapreduce project, documented in mapred-default.xml
> However:
>   - they are still listed in CommonConfigurationKeysPublic.java as quoted above
>   - strings "io.sort.mb" and "io.sort.factor" are used in SequenceFile.java in Hadoop Common project
> Not sure what the solution is, these constants should probably be removed from CommonConfigurationKeysPublic.java but I am not sure what's the best solution for SequenceFile.java.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16513-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 18 22:53:09 2011
Return-Path: <common-issues-return-16513-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F21006C6E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 18 Jun 2011 22:53:08 +0000 (UTC)
Received: (qmail 95191 invoked by uid 500); 18 Jun 2011 22:53:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95071 invoked by uid 500); 18 Jun 2011 22:53:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95063 invoked by uid 99); 18 Jun 2011 22:53:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 22:53:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Jun 2011 22:53:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 98939420378
	for <common-issues@hadoop.apache.org>; Sat, 18 Jun 2011 22:52:47 +0000 (UTC)
Date: Sat, 18 Jun 2011 22:52:47 +0000 (UTC)
From: "Ted Yu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <473628834.18553.1308437567621.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6804) allow
 FileSystem.copyFromLocalFile() to execute under specified username
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051615#comment-13051615 ] 

Ted Yu commented on HADOOP-6804:
--------------------------------

Does hadoop commandline utility support this impersonation ?
Thanks

> allow FileSystem.copyFromLocalFile() to execute under specified username
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6804
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6804
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.20.2
>            Reporter: Ted Yu
>            Priority: Minor
>
> When the user calling FileSystem.copyFromLocalFile() doesn't have permission to write to certain hdfs path:
> Thread [main] (Suspended (exception AccessControlException))   
>     DFSClient.mkdirs(String, FsPermission) line: 905   
>     DistributedFileSystem.mkdirs(Path, FsPermission) line: 262   
>     DistributedFileSystem(FileSystem).mkdirs(Path) line: 1162   
>     FileUtil.copy(FileSystem, Path, FileSystem, Path, boolean, boolean, Configuration) line: 194   
>     DistributedFileSystem(FileSystem).copyFromLocalFile(boolean, boolean, Path, Path) line: 1231   
>     DistributedFileSystem(FileSystem).copyFromLocalFile(boolean, Path, Path) line: 1207   
>     DistributedFileSystem(FileSystem).copyFromLocalFile(Path, Path) line: 1179   
>     GridM2mInstallation.copyInputFiles(FlowConfigurations$FlowConf) line: 380   
> Passwordless ssh has been setup for current user, tyu, on localhost and user hadoop on hdfs
> FileSystem.copyFromLocalFile() should be able to execute using a different username than effective user on localhost so that AccessControlException can be avoided.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16514-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 19 03:05:14 2011
Return-Path: <common-issues-return-16514-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 66AA947D6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 19 Jun 2011 03:05:14 +0000 (UTC)
Received: (qmail 59464 invoked by uid 500); 19 Jun 2011 03:05:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59437 invoked by uid 500); 19 Jun 2011 03:05:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59427 invoked by uid 99); 19 Jun 2011 03:05:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jun 2011 03:05:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jun 2011 03:05:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DE6AB41FB12
	for <common-issues@hadoop.apache.org>; Sun, 19 Jun 2011 03:04:49 +0000 (UTC)
Date: Sun, 19 Jun 2011 03:04:49 +0000 (UTC)
From: "John George (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <983342773.18691.1308452689907.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6424) Port FsShell to FileContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051630#comment-13051630 ] 

John George commented on HADOOP-6424:
-------------------------------------

I think it is perfectly fine to create individual tickets for each symlink property. It might even be easier to review in that case. 

> Port FsShell to FileContext 
> ----------------------------
>
>                 Key: HADOOP-6424
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6424
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: John George
>         Attachments: HADOOP-6424.patch, HADOOP-6424.patch
>
>
> FsShell currently uses FileSystem, needs to be moved over to FileContext.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16515-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 19 04:03:12 2011
Return-Path: <common-issues-return-16515-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 008BC4F84
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 19 Jun 2011 04:03:12 +0000 (UTC)
Received: (qmail 77864 invoked by uid 500); 19 Jun 2011 04:03:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77837 invoked by uid 500); 19 Jun 2011 04:03:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77829 invoked by uid 99); 19 Jun 2011 04:03:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jun 2011 04:03:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jun 2011 04:03:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6F900421AF5
	for <common-issues@hadoop.apache.org>; Sun, 19 Jun 2011 04:02:47 +0000 (UTC)
Date: Sun, 19 Jun 2011 04:02:47 +0000 (UTC)
From: "John George (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1036744552.18754.1308456167453.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <18635139.89501293838185702.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7082) Configuration.writeXML should not
 hold lock while outputting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051635#comment-13051635 ] 

John George commented on HADOOP-7082:
-------------------------------------

Can someone pull this patch into MR-279 branch?

> Configuration.writeXML should not hold lock while outputting
> ------------------------------------------------------------
>
>                 Key: HADOOP-7082
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7082
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7082.txt
>
>
> Common side of HDFS-1542

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16516-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 19 05:26:12 2011
Return-Path: <common-issues-return-16516-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3CD7045DA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 19 Jun 2011 05:26:12 +0000 (UTC)
Received: (qmail 3934 invoked by uid 500); 19 Jun 2011 05:26:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3867 invoked by uid 500); 19 Jun 2011 05:26:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3659 invoked by uid 99); 19 Jun 2011 05:26:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jun 2011 05:26:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jun 2011 05:26:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 750A4421B34
	for <common-issues@hadoop.apache.org>; Sun, 19 Jun 2011 05:25:47 +0000 (UTC)
Date: Sun, 19 Jun 2011 05:25:47 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1134372006.18826.1308461147476.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6804) allow
 FileSystem.copyFromLocalFile() to execute under specified username
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051642#comment-13051642 ] 

Harsh J commented on HADOOP-6804:
---------------------------------

No I do not think the CLI utilities have this in their code. They directly try as the user you are running.

Since am not very clear on this one but willing to work a bit on it, could you tell me why is this required?

> allow FileSystem.copyFromLocalFile() to execute under specified username
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6804
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6804
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.20.2
>            Reporter: Ted Yu
>            Priority: Minor
>
> When the user calling FileSystem.copyFromLocalFile() doesn't have permission to write to certain hdfs path:
> Thread [main] (Suspended (exception AccessControlException))   
>     DFSClient.mkdirs(String, FsPermission) line: 905   
>     DistributedFileSystem.mkdirs(Path, FsPermission) line: 262   
>     DistributedFileSystem(FileSystem).mkdirs(Path) line: 1162   
>     FileUtil.copy(FileSystem, Path, FileSystem, Path, boolean, boolean, Configuration) line: 194   
>     DistributedFileSystem(FileSystem).copyFromLocalFile(boolean, boolean, Path, Path) line: 1231   
>     DistributedFileSystem(FileSystem).copyFromLocalFile(boolean, Path, Path) line: 1207   
>     DistributedFileSystem(FileSystem).copyFromLocalFile(Path, Path) line: 1179   
>     GridM2mInstallation.copyInputFiles(FlowConfigurations$FlowConf) line: 380   
> Passwordless ssh has been setup for current user, tyu, on localhost and user hadoop on hdfs
> FileSystem.copyFromLocalFile() should be able to execute using a different username than effective user on localhost so that AccessControlException can be avoided.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16517-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 19 08:21:11 2011
Return-Path: <common-issues-return-16517-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 911BC4D45
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 19 Jun 2011 08:21:11 +0000 (UTC)
Received: (qmail 88672 invoked by uid 500); 19 Jun 2011 08:21:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88242 invoked by uid 500); 19 Jun 2011 08:21:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88234 invoked by uid 99); 19 Jun 2011 08:21:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jun 2011 08:21:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jun 2011 08:21:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6B8BE421E80
	for <common-issues@hadoop.apache.org>; Sun, 19 Jun 2011 08:20:47 +0000 (UTC)
Date: Sun, 19 Jun 2011 08:20:47 +0000 (UTC)
From: "Sunil Goyal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2069949610.19020.1308471647437.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7404) Data Blocks Spliting should be
 record oriented or provided option for give the spliting locations
 (offsets) as input file
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Data Blocks Spliting should be record oriented or provided option for give the spliting locations (offsets) as input file
-------------------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-7404
                 URL: https://issues.apache.org/jira/browse/HADOOP-7404
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Sunil Goyal


Old Bug :  https://issues.apache.org/jira/browse/HADOOP-106

It is difficult to do the padding in the existing records. Due to the following reason:

1. Records are having the different Size (some may be bytes, some may be GB) but in same file.
2. It is having the compatibility issues with the other standard tools.
3. It will increases the file size without any need of other tools (not working on hadoop).

I think there should be option to this splitting process like this:-   

1. File contains information of offsets where should be splitting done. (like 10,100,120, offset it).
2. Hadoop should do the splitting according to it ( 10-0 = 10, 100-10 =90 , etc).
3. This file can be generated easily from the other tools.  



--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16518-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 19 14:10:09 2011
Return-Path: <common-issues-return-16518-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 770754D18
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 19 Jun 2011 14:10:09 +0000 (UTC)
Received: (qmail 96742 invoked by uid 500); 19 Jun 2011 14:10:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96714 invoked by uid 500); 19 Jun 2011 14:10:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96706 invoked by uid 99); 19 Jun 2011 14:10:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jun 2011 14:10:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jun 2011 14:10:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5EE814219DC
	for <common-issues@hadoop.apache.org>; Sun, 19 Jun 2011 14:09:47 +0000 (UTC)
Date: Sun, 19 Jun 2011 14:09:47 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1792743879.19194.1308492587385.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2069949610.19020.1308471647437.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7404) Data Blocks Spliting should be
 record oriented or provided option for give the spliting locations
 (offsets) as input file
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051686#comment-13051686 ] 

Harsh J commented on HADOOP-7404:
---------------------------------

Sunil,

Interesting points here I think :)

Some Qs:
How often do you have to face #1 in practice?
I do not get your "other tools" points, care to explain with an example of sorts?

> Data Blocks Spliting should be record oriented or provided option for give the spliting locations (offsets) as input file
> -------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7404
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7404
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sunil Goyal
>
> Old Bug :  https://issues.apache.org/jira/browse/HADOOP-106
> It is difficult to do the padding in the existing records. Due to the following reason:
> 1. Records are having the different Size (some may be bytes, some may be GB) but in same file.
> 2. It is having the compatibility issues with the other standard tools.
> 3. It will increases the file size without any need of other tools (not working on hadoop).
> I think there should be option to this splitting process like this:-   
> 1. File contains information of offsets where should be splitting done. (like 10,100,120, offset it).
> 2. Hadoop should do the splitting according to it ( 10-0 = 10, 100-10 =90 , etc).
> 3. This file can be generated easily from the other tools.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16519-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 19 16:38:08 2011
Return-Path: <common-issues-return-16519-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D859F4A20
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 19 Jun 2011 16:38:08 +0000 (UTC)
Received: (qmail 85810 invoked by uid 500); 19 Jun 2011 16:38:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85781 invoked by uid 500); 19 Jun 2011 16:38:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85773 invoked by uid 99); 19 Jun 2011 16:38:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jun 2011 16:38:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jun 2011 16:38:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 59A40422110
	for <common-issues@hadoop.apache.org>; Sun, 19 Jun 2011 16:37:47 +0000 (UTC)
Date: Sun, 19 Jun 2011 16:37:47 +0000 (UTC)
From: "Brock Noland (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <912347773.19253.1308501467363.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-310) Additional constructor requested in
 BytesWritable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Brock Noland updated HADOOP-310:
--------------------------------

    Attachment: bytes-writable-zero-copy-interface-0.patch

I think the offset would be ideal, but the extra constructor and a set() method which does the same is a good incremental change. That patch is attached.


> Additional constructor requested in BytesWritable
> -------------------------------------------------
>
>                 Key: HADOOP-310
>                 URL: https://issues.apache.org/jira/browse/HADOOP-310
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: p sutter
>            Assignee: Owen O'Malley
>            Priority: Minor
>         Attachments: bytes-writable-zero-copy-interface-0.patch
>
>
> It would be grand if BytesWritable.java had an additional constructor as below. This allows me to use the BytesWritable class without doing a buffer copy, since we have a less-than-fully-utilized byte array holding our key.
> Thanks!
>  
>  /**
>    * Create a BytesWritable using the byte array as the initial value.
>    * @param bytes This array becomes the backing storage for the object.
>    */
>   public BytesWritable(byte[] bytes, int size) {
>     this.bytes = bytes;
>     this.size = size;
>   }
>   

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16520-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 19 16:50:08 2011
Return-Path: <common-issues-return-16520-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AFE07413F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 19 Jun 2011 16:50:08 +0000 (UTC)
Received: (qmail 89934 invoked by uid 500); 19 Jun 2011 16:50:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89900 invoked by uid 500); 19 Jun 2011 16:50:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89892 invoked by uid 99); 19 Jun 2011 16:50:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jun 2011 16:50:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jun 2011 16:50:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5ADB04222E0
	for <common-issues@hadoop.apache.org>; Sun, 19 Jun 2011 16:49:47 +0000 (UTC)
Date: Sun, 19 Jun 2011 16:49:47 +0000 (UTC)
From: "Sunil Goyal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <633335823.19260.1308502187369.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2069949610.19020.1308471647437.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7404) Data Blocks Spliting should be
 record oriented or provided option for give the spliting locations
 (offsets) as input file
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051706#comment-13051706 ] 

Sunil Goyal commented on HADOOP-7404:
-------------------------------------

Hello Harsh
       I am from EDA (Electronics Design Automation) Domain. There are very large text file of GDS (http://en.wikipedia.org/wiki/GDSII) of 20GB-100GB is common. It is the standard format used by all the tools for exchange of data. All the standard tools (like Cadence, Synopsys, Magma, other free tools) use this format. 
      This file contains the various records of different size. It is difficult to insert the padding in this format. Standard tools will not be able to understand it. It is open standard format. It is difficult to change this format. It is widely accepted by the industry. 
       It is easy to dump out the separate file giving the location of these offsets. It will help to use the Hadoop for these types of format.
        There are also other format like this in EDA (LEF, DEF , Power Optimization format, Simulation Result Format).

I think giving the option as offset file splitting will be helpful in general. Different type of application user knows where they can get the maximum advantage parallelism by spliting of it its own location.


> Data Blocks Spliting should be record oriented or provided option for give the spliting locations (offsets) as input file
> -------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7404
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7404
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sunil Goyal
>
> Old Bug :  https://issues.apache.org/jira/browse/HADOOP-106
> It is difficult to do the padding in the existing records. Due to the following reason:
> 1. Records are having the different Size (some may be bytes, some may be GB) but in same file.
> 2. It is having the compatibility issues with the other standard tools.
> 3. It will increases the file size without any need of other tools (not working on hadoop).
> I think there should be option to this splitting process like this:-   
> 1. File contains information of offsets where should be splitting done. (like 10,100,120, offset it).
> 2. Hadoop should do the splitting according to it ( 10-0 = 10, 100-10 =90 , etc).
> 3. This file can be generated easily from the other tools.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16521-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 19 19:34:09 2011
Return-Path: <common-issues-return-16521-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0DEB347A2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 19 Jun 2011 19:34:09 +0000 (UTC)
Received: (qmail 96787 invoked by uid 500); 19 Jun 2011 19:34:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96763 invoked by uid 500); 19 Jun 2011 19:34:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96755 invoked by uid 99); 19 Jun 2011 19:34:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jun 2011 19:34:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jun 2011 19:34:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 924C042222C
	for <common-issues@hadoop.apache.org>; Sun, 19 Jun 2011 19:33:47 +0000 (UTC)
Date: Sun, 19 Jun 2011 19:33:47 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <790047585.19353.1308512027595.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2069949610.19020.1308471647437.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7404) Data Blocks Spliting should be
 record oriented or provided option for give the spliting locations
 (offsets) as input file
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051722#comment-13051722 ] 

Harsh J commented on HADOOP-7404:
---------------------------------

Sunil,

(I've not worked with such files, so my analysis _may_ be wrong here -- do let me know if it is)

I've found by experience that not all _actual_ file formats are directly suited to HDFS+MapReduce. For your case, how viable do you think it would be to add in transformation phases for pushing such files into HDFS and then pulling them out back in similar formats?

For example, lets say you could transform such large files into chunks of relatively-less-large files (of say 2 GB) with as many records as it may accomodate and then load those into HDFS (or do it as you stream, somehow -- you'll know your record end markers and sizes read thus-far).

These files can be created with large enough block sizes (2-4 GB is good enough I think, perhaps you could go beyond as well but am not aware of many trying to go beyond the 8 GB block size mark, and we haven't tested that much). This way you achieve the locality you're looking for.

Would splitting out your records from such large files into smaller chunks of records per file not be a viable option here?

The issue with providing mechanisms to specify offsets is that you have to maintain all the offsets in the NameNode memory instead of having to maintain just a tuple of the number of blocks, length of file and the multiplier used. Not to mention, it could (if not pluggable as a design/impl.) also make things more complex for people who do not really mind using non-record-based splitting. But there's lot of merits here as well, as the other ticket mentions it.

> Data Blocks Spliting should be record oriented or provided option for give the spliting locations (offsets) as input file
> -------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7404
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7404
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sunil Goyal
>
> Old Bug :  https://issues.apache.org/jira/browse/HADOOP-106
> It is difficult to do the padding in the existing records. Due to the following reason:
> 1. Records are having the different Size (some may be bytes, some may be GB) but in same file.
> 2. It is having the compatibility issues with the other standard tools.
> 3. It will increases the file size without any need of other tools (not working on hadoop).
> I think there should be option to this splitting process like this:-   
> 1. File contains information of offsets where should be splitting done. (like 10,100,120, offset it).
> 2. Hadoop should do the splitting according to it ( 10-0 = 10, 100-10 =90 , etc).
> 3. This file can be generated easily from the other tools.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16522-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 19 19:52:11 2011
Return-Path: <common-issues-return-16522-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5A17B42BA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 19 Jun 2011 19:52:11 +0000 (UTC)
Received: (qmail 13851 invoked by uid 500); 19 Jun 2011 19:52:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13817 invoked by uid 500); 19 Jun 2011 19:52:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13790 invoked by uid 99); 19 Jun 2011 19:52:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jun 2011 19:52:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 19 Jun 2011 19:52:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5A8924225D4
	for <common-issues@hadoop.apache.org>; Sun, 19 Jun 2011 19:51:47 +0000 (UTC)
Date: Sun, 19 Jun 2011 19:51:47 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1733559157.19367.1308513107367.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2069949610.19020.1308471647437.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7404) Data Blocks Spliting should be
 record oriented or provided option for give the spliting locations
 (offsets) as input file
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051727#comment-13051727 ] 

Harsh J commented on HADOOP-7404:
---------------------------------

Even if you go the padding way, you can add operations when you export processed files outside of Hadoop to get rid of things that would make the file inconsistent. Sure, it would be quite some IO wastage but until you have a new feature that can perhaps ease this, it would work well.

Also, have you considered using, or evaluated alternative FSes for the same purpose? Hadoop is designed to work across different FS as well!

> Data Blocks Spliting should be record oriented or provided option for give the spliting locations (offsets) as input file
> -------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7404
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7404
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sunil Goyal
>
> Old Bug :  https://issues.apache.org/jira/browse/HADOOP-106
> It is difficult to do the padding in the existing records. Due to the following reason:
> 1. Records are having the different Size (some may be bytes, some may be GB) but in same file.
> 2. It is having the compatibility issues with the other standard tools.
> 3. It will increases the file size without any need of other tools (not working on hadoop).
> I think there should be option to this splitting process like this:-   
> 1. File contains information of offsets where should be splitting done. (like 10,100,120, offset it).
> 2. Hadoop should do the splitting according to it ( 10-0 = 10, 100-10 =90 , etc).
> 3. This file can be generated easily from the other tools.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16523-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 00:40:10 2011
Return-Path: <common-issues-return-16523-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 56258487E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 00:40:10 +0000 (UTC)
Received: (qmail 77012 invoked by uid 500); 20 Jun 2011 00:40:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76868 invoked by uid 500); 20 Jun 2011 00:40:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76780 invoked by uid 99); 20 Jun 2011 00:40:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 00:40:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 00:40:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 873E9422232
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 00:39:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 00:39:47 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2045427004.19479.1308530387550.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <836774115.14005.1304224263168.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7254) Programmatically start  processes
 with JMX port open
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tanping Wang resolved HADOOP-7254.
----------------------------------

    Resolution: Won't Fix

HDFS-2083 provides the alternative solution which avoids the security issue.

> Programmatically start  processes with JMX port open 
> -----------------------------------------------------
>
>                 Key: HADOOP-7254
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7254
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: scripts
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7254.patch
>
>
> We propose a programmatic way to start processes with JMX enabled.  This is the counter part of HDFS-1874.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16524-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 01:20:09 2011
Return-Path: <common-issues-return-16524-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E502043AF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 01:20:08 +0000 (UTC)
Received: (qmail 8351 invoked by uid 500); 20 Jun 2011 01:20:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8320 invoked by uid 500); 20 Jun 2011 01:20:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8311 invoked by uid 99); 20 Jun 2011 01:20:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 01:20:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 01:20:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A0BB2422D44
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 01:19:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 01:19:47 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <951229516.19553.1308532787655.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <164404123.4447.1308088127466.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7392) Implement capability of querying
 individual property of a mbean using JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tanping Wang updated HADOOP-7392:
---------------------------------

    Attachment: HADOOP-7392.patch

> Implement capability of querying individual property of a mbean using JMXProxyServlet 
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7392
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7392
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7392.patch
>
>
> Hadoop-7144 provides the capability to query all the properties of a mbean using JMXProxyServlet.  In addition to this, we add the capability to query an individual property of a mbean.  Client will send http request,
> http://hostname/jmx?get=meanName::property
> to query from server.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16525-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 01:20:10 2011
Return-Path: <common-issues-return-16525-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B8A5943BB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 01:20:10 +0000 (UTC)
Received: (qmail 8607 invoked by uid 500); 20 Jun 2011 01:20:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8582 invoked by uid 500); 20 Jun 2011 01:20:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8574 invoked by uid 99); 20 Jun 2011 01:20:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 01:20:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 01:20:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 99D0C422D43
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 01:19:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 01:19:47 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1301554094.19552.1308532787626.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <164404123.4447.1308088127466.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7392) Implement capability of querying
 individual property of a mbean using JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tanping Wang updated HADOOP-7392:
---------------------------------

    Description: 
Hadoop-7144 provides the capability to query all the properties of a mbean using JMXProxyServlet.  In addition to this, we add the capability to query an individual property of a mbean.  Client will send http request,

http://hostname/jmx?get=meanName::property

to query from server.  


  was:
Hadoop-7144 provides the capability to query all the properties of a mbean using JMXProxyServlet.  In addition to this, we add the capability to query an individual property of a mbean.  Client will send http request,

http://hostname/jmx?get=meanName#property

to query from server.  



> Implement capability of querying individual property of a mbean using JMXProxyServlet 
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7392
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7392
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7392.patch
>
>
> Hadoop-7144 provides the capability to query all the properties of a mbean using JMXProxyServlet.  In addition to this, we add the capability to query an individual property of a mbean.  Client will send http request,
> http://hostname/jmx?get=meanName::property
> to query from server.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16526-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 04:03:15 2011
Return-Path: <common-issues-return-16526-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3D99A60A7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 04:03:15 +0000 (UTC)
Received: (qmail 96776 invoked by uid 500); 20 Jun 2011 04:03:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96012 invoked by uid 500); 20 Jun 2011 04:03:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95995 invoked by uid 99); 20 Jun 2011 04:03:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 04:03:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 04:03:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id ABFBC423489
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 04:02:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 04:02:47 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <360702556.19709.1308542567701.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <557825167.38099.1306198007453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7324) Ganglia plugins for metrics v2
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Allen Wittenauer updated HADOOP-7324:
-------------------------------------

    Description: Although, all metrics in metrics v2 are exposed via the standard JMX mechanisms, most users are using Ganglia to collect metrics.  (was: Although, all metrics in metrics v2 are exposed via the standard JMX mechanisms, there are a few users who prefer using Ganglia to collect metrics.)
       Priority: Blocker  (was: Major)
         Labels: regression  (was: )
     Issue Type: Bug  (was: New Feature)

Changing this to accurately reflect the world outside Yahoo!.

> Ganglia plugins for metrics v2
> ------------------------------
>
>                 Key: HADOOP-7324
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7324
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.203.0, 0.23.0
>            Reporter: Luke Lu
>            Priority: Blocker
>              Labels: regression
>             Fix For: 0.23.0
>
>
> Although, all metrics in metrics v2 are exposed via the standard JMX mechanisms, most users are using Ganglia to collect metrics.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16527-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 04:07:13 2011
Return-Path: <common-issues-return-16527-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AF75A65E3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 04:07:13 +0000 (UTC)
Received: (qmail 99260 invoked by uid 500); 20 Jun 2011 04:07:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99186 invoked by uid 500); 20 Jun 2011 04:07:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99169 invoked by uid 99); 20 Jun 2011 04:07:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 04:07:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 04:07:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6D68042357D
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 04:06:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 04:06:47 +0000 (UTC)
From: "Ted Yu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <398939111.19724.1308542807444.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6804) allow
 FileSystem.copyFromLocalFile() to execute under specified username
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051790#comment-13051790 ] 

Ted Yu commented on HADOOP-6804:
--------------------------------

The background was that certain operations are submitted from developer's computer (Eclipse, e.g.) to the cluster which involves impersonation.


> allow FileSystem.copyFromLocalFile() to execute under specified username
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6804
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6804
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.20.2
>            Reporter: Ted Yu
>            Priority: Minor
>
> When the user calling FileSystem.copyFromLocalFile() doesn't have permission to write to certain hdfs path:
> Thread [main] (Suspended (exception AccessControlException))   
>     DFSClient.mkdirs(String, FsPermission) line: 905   
>     DistributedFileSystem.mkdirs(Path, FsPermission) line: 262   
>     DistributedFileSystem(FileSystem).mkdirs(Path) line: 1162   
>     FileUtil.copy(FileSystem, Path, FileSystem, Path, boolean, boolean, Configuration) line: 194   
>     DistributedFileSystem(FileSystem).copyFromLocalFile(boolean, boolean, Path, Path) line: 1231   
>     DistributedFileSystem(FileSystem).copyFromLocalFile(boolean, Path, Path) line: 1207   
>     DistributedFileSystem(FileSystem).copyFromLocalFile(Path, Path) line: 1179   
>     GridM2mInstallation.copyInputFiles(FlowConfigurations$FlowConf) line: 380   
> Passwordless ssh has been setup for current user, tyu, on localhost and user hadoop on hdfs
> FileSystem.copyFromLocalFile() should be able to execute using a different username than effective user on localhost so that AccessControlException can be avoided.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16528-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 04:21:14 2011
Return-Path: <common-issues-return-16528-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3C9276C19
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 04:21:14 +0000 (UTC)
Received: (qmail 3889 invoked by uid 500); 20 Jun 2011 04:21:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3665 invoked by uid 500); 20 Jun 2011 04:21:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3651 invoked by uid 99); 20 Jun 2011 04:21:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 04:21:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 04:21:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6A06C423848
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 04:20:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 04:20:47 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1149165875.19738.1308543647431.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7405) libhadoop is all or nothing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

libhadoop is all or nothing
---------------------------

                 Key: HADOOP-7405
                 URL: https://issues.apache.org/jira/browse/HADOOP-7405
             Project: Hadoop Common
          Issue Type: Bug
          Components: native
    Affects Versions: 0.20.203.0, 0.23.0
         Environment: Everything not Linux
            Reporter: Allen Wittenauer
            Priority: Blocker


As a result of a ton of new code in libhadoop being added in 0.20.203/0.22, a lot of features that used to work no longer do reliably.  The most common problem is native compression, but other issues such as Mac OS X's group support broke as well.  The native code checks need to be refactored such that libhadoop.so should report what it supports rather than having the Java-side assume that if it loads, it is all supported.  This would allow us to stub routines until they've been vetted, removing the chances of such regressions appearing in the future.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16529-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 04:43:15 2011
Return-Path: <common-issues-return-16529-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D69516294
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 04:43:15 +0000 (UTC)
Received: (qmail 11784 invoked by uid 500); 20 Jun 2011 04:43:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11565 invoked by uid 500); 20 Jun 2011 04:43:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11557 invoked by uid 99); 20 Jun 2011 04:43:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 04:43:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 04:43:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 58284423B6D
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 04:42:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 04:42:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <182026872.19742.1308544967357.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1149165875.19738.1308543647431.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7405) libhadoop is all or nothing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051796#comment-13051796 ] 

Todd Lipcon commented on HADOOP-7405:
-------------------------------------

It seems the correct answer here is to actually use soversions properly, no?

> libhadoop is all or nothing
> ---------------------------
>
>                 Key: HADOOP-7405
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7405
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.20.203.0, 0.23.0
>         Environment: Everything not Linux
>            Reporter: Allen Wittenauer
>            Priority: Blocker
>              Labels: regression
>
> As a result of a ton of new code in libhadoop being added in 0.20.203/0.22, a lot of features that used to work no longer do reliably.  The most common problem is native compression, but other issues such as Mac OS X's group support broke as well.  The native code checks need to be refactored such that libhadoop.so should report what it supports rather than having the Java-side assume that if it loads, it is all supported.  This would allow us to stub routines until they've been vetted, removing the chances of such regressions appearing in the future.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16530-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 05:02:15 2011
Return-Path: <common-issues-return-16530-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ADBE26588
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 05:02:15 +0000 (UTC)
Received: (qmail 19166 invoked by uid 500); 20 Jun 2011 05:02:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18946 invoked by uid 500); 20 Jun 2011 05:02:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18938 invoked by uid 99); 20 Jun 2011 05:02:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 05:02:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 05:02:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 735BD423F83
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 05:01:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 05:01:47 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1021529783.19756.1308546107469.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1149165875.19738.1308543647431.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7405) libhadoop is all or nothing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051800#comment-13051800 ] 

Allen Wittenauer commented on HADOOP-7405:
------------------------------------------

That's an interesting idea. How would this work? Are you thinking that during the native load, we detect what soversion got loaded and then base available functionality on that?

I was thinking (far) down the road that we could basically dlopen() modules from within libhadoop.so.  Then as functionality got ported, one would just drop in the appropriate shared lib. This also makes codec support very interesting:  instead of having different bootstrap code for every codec, one could generalize it more than it is today.

> libhadoop is all or nothing
> ---------------------------
>
>                 Key: HADOOP-7405
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7405
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.20.203.0, 0.23.0
>         Environment: Everything not Linux
>            Reporter: Allen Wittenauer
>            Priority: Blocker
>              Labels: regression
>
> As a result of a ton of new code in libhadoop being added in 0.20.203/0.22, a lot of features that used to work no longer do reliably.  The most common problem is native compression, but other issues such as Mac OS X's group support broke as well.  The native code checks need to be refactored such that libhadoop.so should report what it supports rather than having the Java-side assume that if it loads, it is all supported.  This would allow us to stub routines until they've been vetted, removing the chances of such regressions appearing in the future.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16531-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 05:36:16 2011
Return-Path: <common-issues-return-16531-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 255E96085
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 05:36:16 +0000 (UTC)
Received: (qmail 36007 invoked by uid 500); 20 Jun 2011 05:36:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35759 invoked by uid 500); 20 Jun 2011 05:36:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35750 invoked by uid 99); 20 Jun 2011 05:36:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 05:36:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 05:36:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6E5DD42388F
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 05:35:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 05:35:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <159318510.19824.1308548147448.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1149165875.19738.1308543647431.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7405) libhadoop is all or nothing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051808#comment-13051808 ] 

Todd Lipcon commented on HADOOP-7405:
-------------------------------------

I'm thinking it would just expect a particular version, and if you have an old version on your system, it should refuse to load with an error logged.

Supporting past versions of the .so on new versions of Hadoop is a large matrix to test -- we already do a bad job of testing native code.

If there are regressions with regard to OS support, we should catch those by having people with strange OSes testing trunk and voting on releases. Building a complicated framework into the code doesn't seem like the easiest solution to me.

> libhadoop is all or nothing
> ---------------------------
>
>                 Key: HADOOP-7405
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7405
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.20.203.0, 0.23.0
>         Environment: Everything not Linux
>            Reporter: Allen Wittenauer
>            Priority: Blocker
>              Labels: regression
>
> As a result of a ton of new code in libhadoop being added in 0.20.203/0.22, a lot of features that used to work no longer do reliably.  The most common problem is native compression, but other issues such as Mac OS X's group support broke as well.  The native code checks need to be refactored such that libhadoop.so should report what it supports rather than having the Java-side assume that if it loads, it is all supported.  This would allow us to stub routines until they've been vetted, removing the chances of such regressions appearing in the future.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16532-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 06:02:25 2011
Return-Path: <common-issues-return-16532-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 927B8658B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 06:02:25 +0000 (UTC)
Received: (qmail 55367 invoked by uid 500); 20 Jun 2011 06:02:25 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55083 invoked by uid 500); 20 Jun 2011 06:02:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55042 invoked by uid 99); 20 Jun 2011 06:02:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 06:02:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 06:02:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D1D1B423027
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 06:01:49 +0000 (UTC)
Date: Mon, 20 Jun 2011 06:01:49 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <689587468.19861.1308549709855.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1149165875.19738.1308543647431.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7405) libhadoop is all or nothing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051813#comment-13051813 ] 

Allen Wittenauer commented on HADOOP-7405:
------------------------------------------

Ah, ok.  The soversion essentially moves lockstep with a given release.  If we were to go to that method, we'd need to split libhadoop in half to fix the regression problem.

Granted, it is simple, but it doesn't necessarily prevent the problem that we've found ourselves.  Unfortunately, I do not believe anyone on the PMC attempts to run non-Linux on a standard basis.  Using release votes to prevent these situations from happening is pretty much a non-starter.


> libhadoop is all or nothing
> ---------------------------
>
>                 Key: HADOOP-7405
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7405
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.20.203.0, 0.23.0
>         Environment: Everything not Linux
>            Reporter: Allen Wittenauer
>            Priority: Blocker
>              Labels: regression
>
> As a result of a ton of new code in libhadoop being added in 0.20.203/0.22, a lot of features that used to work no longer do reliably.  The most common problem is native compression, but other issues such as Mac OS X's group support broke as well.  The native code checks need to be refactored such that libhadoop.so should report what it supports rather than having the Java-side assume that if it loads, it is all supported.  This would allow us to stub routines until they've been vetted, removing the chances of such regressions appearing in the future.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16533-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 11:28:09 2011
Return-Path: <common-issues-return-16533-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 926926568
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 11:28:09 +0000 (UTC)
Received: (qmail 4413 invoked by uid 500); 20 Jun 2011 11:28:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4370 invoked by uid 500); 20 Jun 2011 11:28:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4361 invoked by uid 99); 20 Jun 2011 11:28:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 11:28:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 11:28:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 90AA6423364
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 11:27:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 11:27:47 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <681990115.20486.1308569267589.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2144547827.20484.1308569267525.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Work started] (HADOOP-7406) Add a "I broke the HAdoop
 build" poster
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Work on HADOOP-7406 started by Steve Loughran.

> Add a "I broke the HAdoop build" poster
> ---------------------------------------
>
>                 Key: HADOOP-7406
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7406
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Minor
>
> Some people have been known to check in code that does not work, and hence break the build. These people (and I may be one of them) deserve to have their contribution acknowledged with an "I broke the build" poster. I have a draft slide for this, using an upside down copy of the logo. It may be A4 paper, when US-letter is probably more appropriate. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16534-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 11:28:11 2011
Return-Path: <common-issues-return-16534-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1B1D46578
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 11:28:11 +0000 (UTC)
Received: (qmail 4700 invoked by uid 500); 20 Jun 2011 11:28:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4659 invoked by uid 500); 20 Jun 2011 11:28:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4644 invoked by uid 99); 20 Jun 2011 11:28:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 11:28:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 11:28:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8116F423362
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 11:27:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 11:27:47 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2144547827.20484.1308569267525.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7406) Add a "I broke the HAdoop build"
 poster
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Add a "I broke the HAdoop build" poster
---------------------------------------

                 Key: HADOOP-7406
                 URL: https://issues.apache.org/jira/browse/HADOOP-7406
             Project: Hadoop Common
          Issue Type: New Feature
            Reporter: Steve Loughran
            Assignee: Steve Loughran
            Priority: Minor


Some people have been known to check in code that does not work, and hence break the build. These people (and I may be one of them) deserve to have their contribution acknowledged with an "I broke the build" poster. I have a draft slide for this, using an upside down copy of the logo. It may be A4 paper, when US-letter is probably more appropriate. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16535-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 11:30:10 2011
Return-Path: <common-issues-return-16535-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 96ADE66C1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 11:30:10 +0000 (UTC)
Received: (qmail 10604 invoked by uid 500); 20 Jun 2011 11:30:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10578 invoked by uid 500); 20 Jun 2011 11:30:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10570 invoked by uid 99); 20 Jun 2011 11:30:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 11:30:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 11:30:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 64BCD4234C0
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 11:29:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 11:29:47 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <518014966.20489.1308569387409.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2144547827.20484.1308569267525.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7406) Add a "I broke the HAdoop build"
 poster
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steve Loughran updated HADOOP-7406:
-----------------------------------

    Attachment: breaking_the_hadoop_build.odp

Initial draft

> Add a "I broke the HAdoop build" poster
> ---------------------------------------
>
>                 Key: HADOOP-7406
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7406
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Minor
>         Attachments: breaking_the_hadoop_build.odp
>
>
> Some people have been known to check in code that does not work, and hence break the build. These people (and I may be one of them) deserve to have their contribution acknowledged with an "I broke the build" poster. I have a draft slide for this, using an upside down copy of the logo. It may be A4 paper, when US-letter is probably more appropriate. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16536-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 11:30:10 2011
Return-Path: <common-issues-return-16536-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D65CD66D7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 11:30:10 +0000 (UTC)
Received: (qmail 10866 invoked by uid 500); 20 Jun 2011 11:30:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10821 invoked by uid 500); 20 Jun 2011 11:30:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10666 invoked by uid 99); 20 Jun 2011 11:30:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 11:30:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 11:30:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 785554234C2
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 11:29:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 11:29:47 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1831153860.20491.1308569387489.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2144547827.20484.1308569267525.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7406) Add a "I broke the Hadoop build"
 poster
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steve Loughran updated HADOOP-7406:
-----------------------------------

    Summary: Add a "I broke the Hadoop build" poster  (was: Add a "I broke the HAdoop build" poster)

> Add a "I broke the Hadoop build" poster
> ---------------------------------------
>
>                 Key: HADOOP-7406
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7406
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Minor
>         Attachments: breaking_the_hadoop_build.odp
>
>
> Some people have been known to check in code that does not work, and hence break the build. These people (and I may be one of them) deserve to have their contribution acknowledged with an "I broke the build" poster. I have a draft slide for this, using an upside down copy of the logo. It may be A4 paper, when US-letter is probably more appropriate. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16537-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 11:32:10 2011
Return-Path: <common-issues-return-16537-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9595860C1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 11:32:10 +0000 (UTC)
Received: (qmail 12435 invoked by uid 500); 20 Jun 2011 11:32:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12402 invoked by uid 500); 20 Jun 2011 11:32:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12392 invoked by uid 99); 20 Jun 2011 11:32:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 11:32:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 11:32:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5F99F4235DA
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 11:31:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 11:31:47 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1540981593.20497.1308569507388.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2144547827.20484.1308569267525.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7406) Add a "I broke the Hadoop build"
 poster
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steve Loughran updated HADOOP-7406:
-----------------------------------

    Component/s: build

> Add a "I broke the Hadoop build" poster
> ---------------------------------------
>
>                 Key: HADOOP-7406
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7406
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Minor
>         Attachments: breaking_the_hadoop_build.odp
>
>
> Some people have been known to check in code that does not work, and hence break the build. These people (and I may be one of them) deserve to have their contribution acknowledged with an "I broke the build" poster. I have a draft slide for this, using an upside down copy of the logo. It may be A4 paper, when US-letter is probably more appropriate. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16538-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 12:02:12 2011
Return-Path: <common-issues-return-16538-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B198760E4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 12:02:12 +0000 (UTC)
Received: (qmail 51931 invoked by uid 500); 20 Jun 2011 12:02:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51872 invoked by uid 500); 20 Jun 2011 12:02:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51864 invoked by uid 99); 20 Jun 2011 12:02:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 12:02:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 12:02:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 57A3142311A
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 12:01:49 +0000 (UTC)
Date: Mon, 20 Jun 2011 12:01:49 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <347427305.20515.1308571309355.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2144547827.20484.1308569267525.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7406) Add a "I broke the Hadoop build"
 poster
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051940#comment-13051940 ] 

Harsh J commented on HADOOP-7406:
---------------------------------

I remember there used to be a Gentoo Hall of Shame for people who had done something like {{emerge --unmerge gcc}}.

On Gentoo, since all packages require compilation, even gcc, you're left with a locked down OS (no compiler to compile the compiler to get it back).

Should we have a hall of shame (ranks too perhaps) for build breakage? ;-)

> Add a "I broke the Hadoop build" poster
> ---------------------------------------
>
>                 Key: HADOOP-7406
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7406
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Minor
>         Attachments: breaking_the_hadoop_build.odp
>
>
> Some people have been known to check in code that does not work, and hence break the build. These people (and I may be one of them) deserve to have their contribution acknowledged with an "I broke the build" poster. I have a draft slide for this, using an upside down copy of the logo. It may be A4 paper, when US-letter is probably more appropriate. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16539-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 12:40:08 2011
Return-Path: <common-issues-return-16539-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C6A0E60CD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 12:40:08 +0000 (UTC)
Received: (qmail 32503 invoked by uid 500); 20 Jun 2011 12:40:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32418 invoked by uid 500); 20 Jun 2011 12:40:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32410 invoked by uid 99); 20 Jun 2011 12:40:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 12:40:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 12:40:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8165B42356B
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 12:39:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 12:39:47 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1358851190.20577.1308573587526.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2144547827.20484.1308569267525.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7406) Add a "I broke the Hadoop build"
 poster
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051954#comment-13051954 ] 

Steve Loughran commented on HADOOP-7406:
----------------------------------------

I'm not sure about being that harsh. At work we split things up into build and test -it's OK to break tests for a while, but if you stop things compiling you get the I broke the build poster. You get to keep it until someone else breaks the build, at which point you get to present it and congratulate them on their achievement. 

> Add a "I broke the Hadoop build" poster
> ---------------------------------------
>
>                 Key: HADOOP-7406
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7406
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Minor
>         Attachments: breaking_the_hadoop_build.odp
>
>
> Some people have been known to check in code that does not work, and hence break the build. These people (and I may be one of them) deserve to have their contribution acknowledged with an "I broke the build" poster. I have a draft slide for this, using an upside down copy of the logo. It may be A4 paper, when US-letter is probably more appropriate. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16540-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 16:15:13 2011
Return-Path: <common-issues-return-16540-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AD5886B48
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 16:15:13 +0000 (UTC)
Received: (qmail 52352 invoked by uid 500); 20 Jun 2011 16:15:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52290 invoked by uid 500); 20 Jun 2011 16:15:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52258 invoked by uid 99); 20 Jun 2011 16:15:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 16:15:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 16:15:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 31AB8424BF4
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 16:14:52 +0000 (UTC)
Date: Mon, 20 Jun 2011 16:14:52 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1837487126.21065.1308586492200.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1149165875.19738.1308543647431.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7405) libhadoop is all or nothing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052033#comment-13052033 ] 

Alejandro Abdelnur commented on HADOOP-7405:
--------------------------------------------

Since Hadoop Kerberos Mac OS X support was never fully there, it is not possible to compile libhadoop due to some compiler errors.

Having libhadoop working partially on different platforms does not seem a good idea as it will mislead users to believe that a particular platform is supported when is 'partially' supported. I'd rather go for the principle of least surprise here (all or nothing), this will make Hadoop easier to user.

A while ago I've opened a HADOOP-7083 to enable running Hadoop with Kerberos ON without relying on some libhadoop functionality and the argument there was that doing that was a security risk. 

I'm not arguing for HADOOP-7083 to be considered again, what I'm trying to state is that if we go the partial libhadoop functionality we'll have to come up with a checklist that will have to identify if certain functionality provided by native libs is optional, can hadoop work without it, is a security risk, etc. This will complicate the not only the lives of developers but of users.

Because of this my take is that if we require native code to run Hadoop, we should provide the full set of native code for each platform we are building for. Only for functionality that there is Java fallback (at a cost of performance) native-code should be optional.




> libhadoop is all or nothing
> ---------------------------
>
>                 Key: HADOOP-7405
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7405
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.20.203.0, 0.23.0
>         Environment: Everything not Linux
>            Reporter: Allen Wittenauer
>            Priority: Blocker
>              Labels: regression
>
> As a result of a ton of new code in libhadoop being added in 0.20.203/0.22, a lot of features that used to work no longer do reliably.  The most common problem is native compression, but other issues such as Mac OS X's group support broke as well.  The native code checks need to be refactored such that libhadoop.so should report what it supports rather than having the Java-side assume that if it loads, it is all supported.  This would allow us to stub routines until they've been vetted, removing the chances of such regressions appearing in the future.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16541-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 16:39:09 2011
Return-Path: <common-issues-return-16541-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1B6FC6376
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 16:39:09 +0000 (UTC)
Received: (qmail 98989 invoked by uid 500); 20 Jun 2011 16:39:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98951 invoked by uid 500); 20 Jun 2011 16:39:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98943 invoked by uid 99); 20 Jun 2011 16:39:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 16:39:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 16:39:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D363B42479B
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 16:38:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 16:38:47 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2020416957.21150.1308587927862.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-7206:
------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
         Assignee: T Jake Luciani  (was: issei yoshida)
     Release Note:   (was: This patch bring the snappy compression to Hadoop.

It requires the snappy v1.0.2 or higher,
Because it doesn't use C++ interface but C interface and loads the snappy native library dynamically via dlopen.

Its native library is a part of the libhadoop.)
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I've just committed this. Thanks T Jake and Issei.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: T Jake Luciani
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16542-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 16:53:10 2011
Return-Path: <common-issues-return-16542-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 75DEC6D73
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 16:53:10 +0000 (UTC)
Received: (qmail 17813 invoked by uid 500); 20 Jun 2011 16:53:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17783 invoked by uid 500); 20 Jun 2011 16:53:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17771 invoked by uid 99); 20 Jun 2011 16:53:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 16:53:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 16:53:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5FAFC424EA6
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 16:52:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 16:52:47 +0000 (UTC)
From: "Brock Noland (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <181311564.21227.1308588767388.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-310) Additional constructor requested in
 BytesWritable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Brock Noland updated HADOOP-310:
--------------------------------

    Attachment: bytes-writable-zero-copy-interface-1.patch

Updated one test assert statement with reason for failure.

> Additional constructor requested in BytesWritable
> -------------------------------------------------
>
>                 Key: HADOOP-310
>                 URL: https://issues.apache.org/jira/browse/HADOOP-310
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: p sutter
>            Assignee: Owen O'Malley
>            Priority: Minor
>         Attachments: bytes-writable-zero-copy-interface-0.patch, bytes-writable-zero-copy-interface-1.patch
>
>
> It would be grand if BytesWritable.java had an additional constructor as below. This allows me to use the BytesWritable class without doing a buffer copy, since we have a less-than-fully-utilized byte array holding our key.
> Thanks!
>  
>  /**
>    * Create a BytesWritable using the byte array as the initial value.
>    * @param bytes This array becomes the backing storage for the object.
>    */
>   public BytesWritable(byte[] bytes, int size) {
>     this.bytes = bytes;
>     this.size = size;
>   }
>   

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16543-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 17:35:09 2011
Return-Path: <common-issues-return-16543-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 561AD6354
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 17:35:09 +0000 (UTC)
Received: (qmail 5188 invoked by uid 500); 20 Jun 2011 17:35:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5155 invoked by uid 500); 20 Jun 2011 17:35:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5145 invoked by uid 99); 20 Jun 2011 17:35:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 17:35:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 17:35:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D3060424303
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 17:34:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 17:34:47 +0000 (UTC)
From: "Ivan Kelly (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <680679068.21358.1308591287861.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7407) Snappy integration breaks HDFS
 build.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Snappy integration breaks HDFS build.
-------------------------------------

                 Key: HADOOP-7407
                 URL: https://issues.apache.org/jira/browse/HADOOP-7407
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Ivan Kelly
            Priority: Critical


The common/ivy/hadoop-common-template.xml submitted with 7206 has a typo which breaks anything that depends on the hadoop-common maven package.
Instead of <artifactId>java-snappy</artifactId>, you should have <artifactId>snappy-java</artifactId>

[ivy:resolve] downloading https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.23.0-SNAPSHOT/hadoop-common-0.23.0-20110620.163810-177.jar ...
[ivy:resolve] .......................
[ivy:resolve] ..................................
[ivy:resolve] ...........................................
[ivy:resolve] ...............................................................
[ivy:resolve] ................................................ (1631kB)
[ivy:resolve] .. (0kB)
[ivy:resolve] 	[SUCCESSFUL ] org.apache.hadoop#hadoop-common;0.23.0-SNAPSHOT!hadoop-common.jar (8441ms)
[ivy:resolve] 
[ivy:resolve] :: problems summary ::
[ivy:resolve] :::: WARNINGS
[ivy:resolve] 		module not found: org.xerial.snappy#java-snappy;1.0.3-rc2
[ivy:resolve] 	==== apache-snapshot: tried
[ivy:resolve] 	  https://repository.apache.org/content/repositories/snapshots/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.pom
[ivy:resolve] 	  -- artifact org.xerial.snappy#java-snappy;1.0.3-rc2!java-snappy.jar:
[ivy:resolve] 	  https://repository.apache.org/content/repositories/snapshots/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.jar
[ivy:resolve] 	==== maven2: tried
[ivy:resolve] 	  http://repo1.maven.org/maven2/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.pom
[ivy:resolve] 	  -- artifact org.xerial.snappy#java-snappy;1.0.3-rc2!java-snappy.jar:
[ivy:resolve] 	  http://repo1.maven.org/maven2/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.jar
[ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
[ivy:resolve] 		::          UNRESOLVED DEPENDENCIES         ::
[ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
[ivy:resolve] 		:: org.xerial.snappy#java-snappy;1.0.3-rc2: not found
[ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
[ivy:resolve] 


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16544-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 17:47:10 2011
Return-Path: <common-issues-return-16544-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CDAFA63E9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 17:47:10 +0000 (UTC)
Received: (qmail 25763 invoked by uid 500); 20 Jun 2011 17:47:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25741 invoked by uid 500); 20 Jun 2011 17:47:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25733 invoked by uid 99); 20 Jun 2011 17:47:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 17:47:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 17:47:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 795A64249D1
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 17:46:49 +0000 (UTC)
Date: Mon, 20 Jun 2011 17:46:49 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1905111596.21425.1308592009493.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Reopened] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE reopened HADOOP-7206:
--------------------------------------------


> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: T Jake Luciani
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16545-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 17:47:11 2011
Return-Path: <common-issues-return-16545-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6E1F56405
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 17:47:11 +0000 (UTC)
Received: (qmail 26050 invoked by uid 500); 20 Jun 2011 17:47:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26001 invoked by uid 500); 20 Jun 2011 17:47:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25879 invoked by uid 99); 20 Jun 2011 17:47:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 17:47:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 17:47:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8D8C64249D5
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 17:46:49 +0000 (UTC)
Date: Mon, 20 Jun 2011 17:46:49 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <672321241.21428.1308592009576.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <680679068.21358.1308591287861.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7407) Snappy integration breaks HDFS
 build.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-7407:
---------------------------------------

    Attachment: HADOOP-7407.patch

> Snappy integration breaks HDFS build.
> -------------------------------------
>
>                 Key: HADOOP-7407
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7407
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Ivan Kelly
>            Priority: Critical
>         Attachments: HADOOP-7407.patch
>
>
> The common/ivy/hadoop-common-template.xml submitted with 7206 has a typo which breaks anything that depends on the hadoop-common maven package.
> Instead of <artifactId>java-snappy</artifactId>, you should have <artifactId>snappy-java</artifactId>
> [ivy:resolve] downloading https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.23.0-SNAPSHOT/hadoop-common-0.23.0-20110620.163810-177.jar ...
> [ivy:resolve] .......................
> [ivy:resolve] ..................................
> [ivy:resolve] ...........................................
> [ivy:resolve] ...............................................................
> [ivy:resolve] ................................................ (1631kB)
> [ivy:resolve] .. (0kB)
> [ivy:resolve] 	[SUCCESSFUL ] org.apache.hadoop#hadoop-common;0.23.0-SNAPSHOT!hadoop-common.jar (8441ms)
> [ivy:resolve] 
> [ivy:resolve] :: problems summary ::
> [ivy:resolve] :::: WARNINGS
> [ivy:resolve] 		module not found: org.xerial.snappy#java-snappy;1.0.3-rc2
> [ivy:resolve] 	==== apache-snapshot: tried
> [ivy:resolve] 	  https://repository.apache.org/content/repositories/snapshots/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.pom
> [ivy:resolve] 	  -- artifact org.xerial.snappy#java-snappy;1.0.3-rc2!java-snappy.jar:
> [ivy:resolve] 	  https://repository.apache.org/content/repositories/snapshots/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.jar
> [ivy:resolve] 	==== maven2: tried
> [ivy:resolve] 	  http://repo1.maven.org/maven2/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.pom
> [ivy:resolve] 	  -- artifact org.xerial.snappy#java-snappy;1.0.3-rc2!java-snappy.jar:
> [ivy:resolve] 	  http://repo1.maven.org/maven2/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.jar
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 		::          UNRESOLVED DEPENDENCIES         ::
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 		:: org.xerial.snappy#java-snappy;1.0.3-rc2: not found
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16546-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 17:47:11 2011
Return-Path: <common-issues-return-16546-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 70278640D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 17:47:11 +0000 (UTC)
Received: (qmail 26070 invoked by uid 500); 20 Jun 2011 17:47:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26004 invoked by uid 500); 20 Jun 2011 17:47:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25988 invoked by uid 99); 20 Jun 2011 17:47:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 17:47:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 17:47:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9B4BA4249D7
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 17:46:49 +0000 (UTC)
Date: Mon, 20 Jun 2011 17:46:49 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2067615753.21430.1308592009633.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <680679068.21358.1308591287861.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7407) Snappy integration breaks HDFS
 build.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-7407:
---------------------------------------

    Status: Patch Available  (was: Open)

> Snappy integration breaks HDFS build.
> -------------------------------------
>
>                 Key: HADOOP-7407
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7407
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Ivan Kelly
>            Priority: Critical
>         Attachments: HADOOP-7407.patch
>
>
> The common/ivy/hadoop-common-template.xml submitted with 7206 has a typo which breaks anything that depends on the hadoop-common maven package.
> Instead of <artifactId>java-snappy</artifactId>, you should have <artifactId>snappy-java</artifactId>
> [ivy:resolve] downloading https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.23.0-SNAPSHOT/hadoop-common-0.23.0-20110620.163810-177.jar ...
> [ivy:resolve] .......................
> [ivy:resolve] ..................................
> [ivy:resolve] ...........................................
> [ivy:resolve] ...............................................................
> [ivy:resolve] ................................................ (1631kB)
> [ivy:resolve] .. (0kB)
> [ivy:resolve] 	[SUCCESSFUL ] org.apache.hadoop#hadoop-common;0.23.0-SNAPSHOT!hadoop-common.jar (8441ms)
> [ivy:resolve] 
> [ivy:resolve] :: problems summary ::
> [ivy:resolve] :::: WARNINGS
> [ivy:resolve] 		module not found: org.xerial.snappy#java-snappy;1.0.3-rc2
> [ivy:resolve] 	==== apache-snapshot: tried
> [ivy:resolve] 	  https://repository.apache.org/content/repositories/snapshots/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.pom
> [ivy:resolve] 	  -- artifact org.xerial.snappy#java-snappy;1.0.3-rc2!java-snappy.jar:
> [ivy:resolve] 	  https://repository.apache.org/content/repositories/snapshots/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.jar
> [ivy:resolve] 	==== maven2: tried
> [ivy:resolve] 	  http://repo1.maven.org/maven2/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.pom
> [ivy:resolve] 	  -- artifact org.xerial.snappy#java-snappy;1.0.3-rc2!java-snappy.jar:
> [ivy:resolve] 	  http://repo1.maven.org/maven2/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.jar
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 		::          UNRESOLVED DEPENDENCIES         ::
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 		:: org.xerial.snappy#java-snappy;1.0.3-rc2: not found
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16547-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 17:57:13 2011
Return-Path: <common-issues-return-16547-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C8BB762D7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 17:57:13 +0000 (UTC)
Received: (qmail 47342 invoked by uid 500); 20 Jun 2011 17:57:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47293 invoked by uid 500); 20 Jun 2011 17:57:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47275 invoked by uid 99); 20 Jun 2011 17:57:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 17:57:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 17:57:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0724C424E0A
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 17:56:50 +0000 (UTC)
Date: Mon, 20 Jun 2011 17:56:50 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1603598428.21489.1308592610025.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052103#comment-13052103 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7206:
------------------------------------------------

BTW, who has reviewed the patch?  It seems there is no +1 on the patch.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: T Jake Luciani
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16548-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 18:01:10 2011
Return-Path: <common-issues-return-16548-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 44C7069E6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 18:01:10 +0000 (UTC)
Received: (qmail 52435 invoked by uid 500); 20 Jun 2011 18:01:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52334 invoked by uid 500); 20 Jun 2011 18:01:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52325 invoked by uid 99); 20 Jun 2011 18:01:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 18:01:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 18:01:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8ADF342404B
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 18:00:48 +0000 (UTC)
Date: Mon, 20 Jun 2011 18:00:48 +0000 (UTC)
From: "T Jake Luciani (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1796278942.21505.1308592848565.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052107#comment-13052107 ] 

T Jake Luciani commented on HADOOP-7206:
----------------------------------------

Can we fix #7407 rather than revert this one? looks like a typo. 

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: T Jake Luciani
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16549-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 18:05:09 2011
Return-Path: <common-issues-return-16549-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3CBA56BCB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 18:05:09 +0000 (UTC)
Received: (qmail 63730 invoked by uid 500); 20 Jun 2011 18:05:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63688 invoked by uid 500); 20 Jun 2011 18:05:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63636 invoked by uid 99); 20 Jun 2011 18:05:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 18:05:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 18:05:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 96142424234
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 18:04:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 18:04:47 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1866770488.21562.1308593087611.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <680679068.21358.1308591287861.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7407) Snappy integration breaks HDFS
 build.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-7407:
------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
         Assignee: Alejandro Abdelnur
     Release Note: 
I've just committed this. Thanks Alejandro.

I manually checked that HDFS and MapReduce both build. Sorry for not doing this earlier.
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

> Snappy integration breaks HDFS build.
> -------------------------------------
>
>                 Key: HADOOP-7407
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7407
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Ivan Kelly
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7407.patch
>
>
> The common/ivy/hadoop-common-template.xml submitted with 7206 has a typo which breaks anything that depends on the hadoop-common maven package.
> Instead of <artifactId>java-snappy</artifactId>, you should have <artifactId>snappy-java</artifactId>
> [ivy:resolve] downloading https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.23.0-SNAPSHOT/hadoop-common-0.23.0-20110620.163810-177.jar ...
> [ivy:resolve] .......................
> [ivy:resolve] ..................................
> [ivy:resolve] ...........................................
> [ivy:resolve] ...............................................................
> [ivy:resolve] ................................................ (1631kB)
> [ivy:resolve] .. (0kB)
> [ivy:resolve] 	[SUCCESSFUL ] org.apache.hadoop#hadoop-common;0.23.0-SNAPSHOT!hadoop-common.jar (8441ms)
> [ivy:resolve] 
> [ivy:resolve] :: problems summary ::
> [ivy:resolve] :::: WARNINGS
> [ivy:resolve] 		module not found: org.xerial.snappy#java-snappy;1.0.3-rc2
> [ivy:resolve] 	==== apache-snapshot: tried
> [ivy:resolve] 	  https://repository.apache.org/content/repositories/snapshots/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.pom
> [ivy:resolve] 	  -- artifact org.xerial.snappy#java-snappy;1.0.3-rc2!java-snappy.jar:
> [ivy:resolve] 	  https://repository.apache.org/content/repositories/snapshots/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.jar
> [ivy:resolve] 	==== maven2: tried
> [ivy:resolve] 	  http://repo1.maven.org/maven2/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.pom
> [ivy:resolve] 	  -- artifact org.xerial.snappy#java-snappy;1.0.3-rc2!java-snappy.jar:
> [ivy:resolve] 	  http://repo1.maven.org/maven2/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.jar
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 		::          UNRESOLVED DEPENDENCIES         ::
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 		:: org.xerial.snappy#java-snappy;1.0.3-rc2: not found
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16550-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 18:05:11 2011
Return-Path: <common-issues-return-16550-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 99D0A6BDC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 18:05:11 +0000 (UTC)
Received: (qmail 64010 invoked by uid 500); 20 Jun 2011 18:05:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63979 invoked by uid 500); 20 Jun 2011 18:05:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63971 invoked by uid 99); 20 Jun 2011 18:05:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 18:05:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 18:05:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AFBEB424238
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 18:04:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 18:04:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <106127372.21565.1308593087716.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <680679068.21358.1308591287861.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7407) Snappy integration breaks HDFS
 build.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052110#comment-13052110 ] 

Hadoop QA commented on HADOOP-7407:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12483180/HADOOP-7407.patch
  against trunk revision 1137690.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/656//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/656//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/656//console

This message is automatically generated.

> Snappy integration breaks HDFS build.
> -------------------------------------
>
>                 Key: HADOOP-7407
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7407
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Ivan Kelly
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7407.patch
>
>
> The common/ivy/hadoop-common-template.xml submitted with 7206 has a typo which breaks anything that depends on the hadoop-common maven package.
> Instead of <artifactId>java-snappy</artifactId>, you should have <artifactId>snappy-java</artifactId>
> [ivy:resolve] downloading https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.23.0-SNAPSHOT/hadoop-common-0.23.0-20110620.163810-177.jar ...
> [ivy:resolve] .......................
> [ivy:resolve] ..................................
> [ivy:resolve] ...........................................
> [ivy:resolve] ...............................................................
> [ivy:resolve] ................................................ (1631kB)
> [ivy:resolve] .. (0kB)
> [ivy:resolve] 	[SUCCESSFUL ] org.apache.hadoop#hadoop-common;0.23.0-SNAPSHOT!hadoop-common.jar (8441ms)
> [ivy:resolve] 
> [ivy:resolve] :: problems summary ::
> [ivy:resolve] :::: WARNINGS
> [ivy:resolve] 		module not found: org.xerial.snappy#java-snappy;1.0.3-rc2
> [ivy:resolve] 	==== apache-snapshot: tried
> [ivy:resolve] 	  https://repository.apache.org/content/repositories/snapshots/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.pom
> [ivy:resolve] 	  -- artifact org.xerial.snappy#java-snappy;1.0.3-rc2!java-snappy.jar:
> [ivy:resolve] 	  https://repository.apache.org/content/repositories/snapshots/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.jar
> [ivy:resolve] 	==== maven2: tried
> [ivy:resolve] 	  http://repo1.maven.org/maven2/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.pom
> [ivy:resolve] 	  -- artifact org.xerial.snappy#java-snappy;1.0.3-rc2!java-snappy.jar:
> [ivy:resolve] 	  http://repo1.maven.org/maven2/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.jar
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 		::          UNRESOLVED DEPENDENCIES         ::
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 		:: org.xerial.snappy#java-snappy;1.0.3-rc2: not found
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16551-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 18:07:15 2011
Return-Path: <common-issues-return-16551-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 18EAF6C71
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 18:07:15 +0000 (UTC)
Received: (qmail 66820 invoked by uid 500); 20 Jun 2011 18:07:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66784 invoked by uid 500); 20 Jun 2011 18:07:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66775 invoked by uid 99); 20 Jun 2011 18:07:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 18:07:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 18:07:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2B4284243CF
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 18:06:51 +0000 (UTC)
Date: Mon, 20 Jun 2011 18:06:51 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <237665661.21669.1308593211173.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052112#comment-13052112 ] 

Tom White commented on HADOOP-7206:
-----------------------------------

I've committed the fix to HADOOP-7407 after manually verifying it. 

> BTW, who has reviewed the patch?

I reviewed the patch. I marked it as "Reviewed" when committing it.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: T Jake Luciani
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16552-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 18:13:12 2011
Return-Path: <common-issues-return-16552-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D030E6065
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 18:13:12 +0000 (UTC)
Received: (qmail 80034 invoked by uid 500); 20 Jun 2011 18:13:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79975 invoked by uid 500); 20 Jun 2011 18:13:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79872 invoked by uid 99); 20 Jun 2011 18:13:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 18:13:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 18:13:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3631C42473F
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 18:12:48 +0000 (UTC)
Date: Mon, 20 Jun 2011 18:12:48 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <679321838.21736.1308593568218.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1149165875.19738.1308543647431.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7405) libhadoop is all or nothing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052118#comment-13052118 ] 

Allen Wittenauer commented on HADOOP-7405:
------------------------------------------

bq. Since Hadoop Kerberos Mac OS X support was never fully there, it is not possible to compile libhadoop due to some compiler errors.

The compiler errors are fairly simple to fix on Darwin.  I don't know why, but it seems like 9 times out of 10, we favor BSD functionality when we go with something non-portable. 

bg. Because of this my take is that if we require native code to run Hadoop, we should provide the full set of native code for each platform we are building for. 

Regardless of what happens in this jira, we need a testsuite for the C code anyway.  OS X actually proves out that even if the code compiles, it doesn't necessarily mean it works properly.  (See HADOOP-7367).

bg. A while ago I've opened a HADOOP-7083 to enable running Hadoop with Kerberos ON without relying on some libhadoop functionality and the argument there was that doing that was a security risk. 

Right.  It wanted to create a third security mode where some stuff worked and some stuff didn't.  That's not quite what I'm asking for here and it wouldn't actually fix the problem we're hitting anyway. The security functionality is orthogonal to the compression functionality.  That's the base, surface issue.  Since it is in one big chunk, we broke *both*.

(While I guess it wasn't obvious, I should probably state that I'm not looking for a "partially working" security mode.  The scope of what constitutes a working unit would still need to be defined.  It is more than reasonable to say that all of the functions that are directly security related would need to be ported and treated like one block.  Asking libhadoop.so if it "supports security" seems like a reasonable thing to ask it.)

The problem that we've got is that we have a lot of unrelated code sitting in libhadoop.so.  Every time we add something we run the risk of regressing features out of platforms other than Linux since those other platforms are an afterthought.  HADOOP-7206 may actually be a great example of this:  if we go with a pure native implementation, we won't be able to support Snappy on anything but Linux with the current state of things.  Lack of compression support has a *direct* impact on the client.  I'd be surprised if the majority of shops are only using Linux clients. 

Wouldn't it be great to be able to ask the lib "do you support gzip, do you support snappy, do you support lzo, do you support security, ..."?  Then we could add code as needed, do ports as needed, etc.  An alternative would be that we start breaking libhadoop up into at least related functionality.

I suppose the other outcome might be that we as a community just admit that we don't support Hadoop on anything but Linux and give up on any semblance of portability.  More and more code is being added or rewritten in C.  I would be surprised if this trend changes.

> libhadoop is all or nothing
> ---------------------------
>
>                 Key: HADOOP-7405
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7405
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.20.203.0, 0.23.0
>         Environment: Everything not Linux
>            Reporter: Allen Wittenauer
>            Priority: Blocker
>              Labels: regression
>
> As a result of a ton of new code in libhadoop being added in 0.20.203/0.22, a lot of features that used to work no longer do reliably.  The most common problem is native compression, but other issues such as Mac OS X's group support broke as well.  The native code checks need to be refactored such that libhadoop.so should report what it supports rather than having the Java-side assume that if it loads, it is all supported.  This would allow us to stub routines until they've been vetted, removing the chances of such regressions appearing in the future.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16553-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 18:17:09 2011
Return-Path: <common-issues-return-16553-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EEA746827
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 18:17:08 +0000 (UTC)
Received: (qmail 84049 invoked by uid 500); 20 Jun 2011 18:17:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84013 invoked by uid 500); 20 Jun 2011 18:17:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84005 invoked by uid 99); 20 Jun 2011 18:17:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 18:17:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 18:17:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B9686424A84
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 18:16:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 18:16:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <718256822.21772.1308593807756.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1149165875.19738.1308543647431.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7405) libhadoop is all or nothing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052122#comment-13052122 ] 

Todd Lipcon commented on HADOOP-7405:
-------------------------------------

Allen, if you want to write a patch that adds a call like:
boolean isNativeFeatureSupported(long featureCode);
...with a set of enums for various features, I would happily review it.


> libhadoop is all or nothing
> ---------------------------
>
>                 Key: HADOOP-7405
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7405
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.20.203.0, 0.23.0
>         Environment: Everything not Linux
>            Reporter: Allen Wittenauer
>            Priority: Blocker
>              Labels: regression
>
> As a result of a ton of new code in libhadoop being added in 0.20.203/0.22, a lot of features that used to work no longer do reliably.  The most common problem is native compression, but other issues such as Mac OS X's group support broke as well.  The native code checks need to be refactored such that libhadoop.so should report what it supports rather than having the Java-side assume that if it loads, it is all supported.  This would allow us to stub routines until they've been vetted, removing the chances of such regressions appearing in the future.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16554-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 18:27:10 2011
Return-Path: <common-issues-return-16554-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CBA076C6E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 18:27:10 +0000 (UTC)
Received: (qmail 97658 invoked by uid 500); 20 Jun 2011 18:27:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97619 invoked by uid 500); 20 Jun 2011 18:27:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97611 invoked by uid 99); 20 Jun 2011 18:27:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 18:27:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 18:27:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 97B7F4240F6
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 18:26:49 +0000 (UTC)
Date: Mon, 20 Jun 2011 18:26:49 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1850309523.21856.1308594409618.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052129#comment-13052129 ] 

Hudson commented on HADOOP-7206:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #661 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/661/])
    

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: T Jake Luciani
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16555-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 18:27:11 2011
Return-Path: <common-issues-return-16555-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6E8856C7A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 18:27:11 +0000 (UTC)
Received: (qmail 97935 invoked by uid 500); 20 Jun 2011 18:27:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97889 invoked by uid 500); 20 Jun 2011 18:27:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97880 invoked by uid 99); 20 Jun 2011 18:27:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 18:27:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 18:27:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3D2A042410B
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 18:26:50 +0000 (UTC)
Date: Mon, 20 Jun 2011 18:26:50 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1791153381.21875.1308594410247.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1758842790.15942.1308336467619.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7402) TestConfiguration doesn't clean up
 after itself
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052131#comment-13052131 ] 

Hudson commented on HADOOP-7402:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #661 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/661/])
    

> TestConfiguration doesn't clean up after itself
> -----------------------------------------------
>
>                 Key: HADOOP-7402
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7402
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7402.0.patch, hadoop-7402.1.patch
>
>
> {{testGetFile}} and {{testGetLocalPath}} both create directories a, b, and c in the working directory from where the tests are run. They should clean up after themselves.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16556-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 18:27:11 2011
Return-Path: <common-issues-return-16556-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C8AEF6C84
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 18:27:11 +0000 (UTC)
Received: (qmail 98174 invoked by uid 500); 20 Jun 2011 18:27:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98145 invoked by uid 500); 20 Jun 2011 18:27:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98132 invoked by uid 99); 20 Jun 2011 18:27:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 18:27:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 18:27:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4B08542410D
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 18:26:50 +0000 (UTC)
Date: Mon, 20 Jun 2011 18:26:50 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2065651478.21877.1308594410304.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <680679068.21358.1308591287861.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7407) Snappy integration breaks HDFS
 build.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052132#comment-13052132 ] 

Hudson commented on HADOOP-7407:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #661 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/661/])
    HADOOP-7407. Snappy integration breaks HDFS build. Contributed by Alejandro Abdelnur.

tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1137724
Files : 
* /hadoop/common/trunk/common/ivy/hadoop-common-template.xml
* /hadoop/common/trunk/common/CHANGES.txt


> Snappy integration breaks HDFS build.
> -------------------------------------
>
>                 Key: HADOOP-7407
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7407
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Ivan Kelly
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7407.patch
>
>
> The common/ivy/hadoop-common-template.xml submitted with 7206 has a typo which breaks anything that depends on the hadoop-common maven package.
> Instead of <artifactId>java-snappy</artifactId>, you should have <artifactId>snappy-java</artifactId>
> [ivy:resolve] downloading https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.23.0-SNAPSHOT/hadoop-common-0.23.0-20110620.163810-177.jar ...
> [ivy:resolve] .......................
> [ivy:resolve] ..................................
> [ivy:resolve] ...........................................
> [ivy:resolve] ...............................................................
> [ivy:resolve] ................................................ (1631kB)
> [ivy:resolve] .. (0kB)
> [ivy:resolve] 	[SUCCESSFUL ] org.apache.hadoop#hadoop-common;0.23.0-SNAPSHOT!hadoop-common.jar (8441ms)
> [ivy:resolve] 
> [ivy:resolve] :: problems summary ::
> [ivy:resolve] :::: WARNINGS
> [ivy:resolve] 		module not found: org.xerial.snappy#java-snappy;1.0.3-rc2
> [ivy:resolve] 	==== apache-snapshot: tried
> [ivy:resolve] 	  https://repository.apache.org/content/repositories/snapshots/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.pom
> [ivy:resolve] 	  -- artifact org.xerial.snappy#java-snappy;1.0.3-rc2!java-snappy.jar:
> [ivy:resolve] 	  https://repository.apache.org/content/repositories/snapshots/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.jar
> [ivy:resolve] 	==== maven2: tried
> [ivy:resolve] 	  http://repo1.maven.org/maven2/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.pom
> [ivy:resolve] 	  -- artifact org.xerial.snappy#java-snappy;1.0.3-rc2!java-snappy.jar:
> [ivy:resolve] 	  http://repo1.maven.org/maven2/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.jar
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 		::          UNRESOLVED DEPENDENCIES         ::
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 		:: org.xerial.snappy#java-snappy;1.0.3-rc2: not found
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16557-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 18:27:12 2011
Return-Path: <common-issues-return-16557-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C06B66C9E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 18:27:12 +0000 (UTC)
Received: (qmail 98835 invoked by uid 500); 20 Jun 2011 18:27:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98804 invoked by uid 500); 20 Jun 2011 18:27:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98796 invoked by uid 99); 20 Jun 2011 18:27:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 18:27:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 18:27:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A50634240F8
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 18:26:49 +0000 (UTC)
Date: Mon, 20 Jun 2011 18:26:49 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1312535507.21858.1308594409672.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6929) RPC should have a way to pass
 Security information other than protocol annotations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052130#comment-13052130 ] 

Hudson commented on HADOOP-6929:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #661 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/661/])
    

> RPC should have a way to pass Security information other than protocol annotations
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6929
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6929
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>             Fix For: 0.23.0
>
>         Attachments: Hadoop-6929_v1.patch, h-6929.patch, h-6929.patch, h-6929.patch
>
>
> Currently Hadoop RPC allows protocol annotations as the only way to pass security information. This becomes a problem if protocols are generated and not hand written. For example protocols generated via Avro and passed over Avro tunnel (AvroRpcEngine.java) can't pass the security information.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16558-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 18:29:11 2011
Return-Path: <common-issues-return-16558-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7BA3A6ECE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 18:29:11 +0000 (UTC)
Received: (qmail 7089 invoked by uid 500); 20 Jun 2011 18:29:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7024 invoked by uid 500); 20 Jun 2011 18:29:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6993 invoked by uid 99); 20 Jun 2011 18:29:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 18:29:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 18:29:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BA85C424298
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 18:28:49 +0000 (UTC)
Date: Mon, 20 Jun 2011 18:28:49 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <208045556.21930.1308594529760.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052134#comment-13052134 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7206:
------------------------------------------------

> Can we fix #7407 rather than revert this one? looks like a typo. 

Sure.  Thanks.  :)

> I reviewed the patch. I marked it as "Reviewed" when committing it.

Hi Tom, the patch is missing javadoc for the new public classes/methods.  That was also a reason that I suggested reverting it.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: T Jake Luciani
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16559-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 18:35:09 2011
Return-Path: <common-issues-return-16559-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 50A266788
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 18:35:09 +0000 (UTC)
Received: (qmail 19857 invoked by uid 500); 20 Jun 2011 18:35:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19822 invoked by uid 500); 20 Jun 2011 18:35:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19814 invoked by uid 99); 20 Jun 2011 18:35:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 18:35:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 18:35:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F0179424506
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 18:34:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 18:34:47 +0000 (UTC)
From: "Sunil Goyal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1344120248.21945.1308594887980.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2069949610.19020.1308471647437.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7404) Data Blocks Spliting should be
 record oriented or provided option for give the spliting locations
 (offsets) as input file
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052139#comment-13052139 ] 

Sunil Goyal commented on HADOOP-7404:
-------------------------------------

Hi Harsh
        Thanks for your response and giving the feedback for problem. Yes splitting the file into smaller chunks will be a viable option for me. I do not know how much advantage i will get by using it. Let me explain you the problem in more details (may you suggest some better solution for it). Its simplified version will be like this:-

Consider I am having a directory having lot of files inside it (20,000 to 1millon). Each file contains the data in the same format (say text). I have created a single file this whole directory (20GB to 100GB). Now I wanted to do some operation on some of these files and again wanted to dump the same file again.

  If i am able to splitting it by using the offset, Each of this can be processed independently. I can gain the maximum advantage of Hadoop  processing nodes. Splitting it according to offset will be useful option for me.  If it we can do the splitting of this file according to its offset. It will give the great advantage and remove one extra step. 
 
I have no idea about FSes. Can you please point out come documentation link for it. (Unable to get it by google)

Thanks
Sunil Goyal 



> Data Blocks Spliting should be record oriented or provided option for give the spliting locations (offsets) as input file
> -------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7404
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7404
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sunil Goyal
>
> Old Bug :  https://issues.apache.org/jira/browse/HADOOP-106
> It is difficult to do the padding in the existing records. Due to the following reason:
> 1. Records are having the different Size (some may be bytes, some may be GB) but in same file.
> 2. It is having the compatibility issues with the other standard tools.
> 3. It will increases the file size without any need of other tools (not working on hadoop).
> I think there should be option to this splitting process like this:-   
> 1. File contains information of offsets where should be splitting done. (like 10,100,120, offset it).
> 2. Hadoop should do the splitting according to it ( 10-0 = 10, 100-10 =90 , etc).
> 3. This file can be generated easily from the other tools.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16560-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 19:01:09 2011
Return-Path: <common-issues-return-16560-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 11F846C1B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 19:01:09 +0000 (UTC)
Received: (qmail 61543 invoked by uid 500); 20 Jun 2011 19:01:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61518 invoked by uid 500); 20 Jun 2011 19:01:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61510 invoked by uid 99); 20 Jun 2011 19:01:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 19:01:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 19:01:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 97A9A424237
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 19:00:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 19:00:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1185307468.22009.1308596447617.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <164404123.4447.1308088127466.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7392) Implement capability of querying
 individual property of a mbean using JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052157#comment-13052157 ] 

Luke Lu commented on HADOOP-7392:
---------------------------------

# I think /jmx should preserve the current behavior: list all mbeans, when no parameters are specified.
# Need to document the get syntax in the javadoc.
# Need to handle cases when property does(n't exist and make sure propertyinfo doesn't cause NPE later.
# Need to set the response status to 404 (SC_NOT_FOUND) if the property is not found.

> Implement capability of querying individual property of a mbean using JMXProxyServlet 
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7392
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7392
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7392.patch
>
>
> Hadoop-7144 provides the capability to query all the properties of a mbean using JMXProxyServlet.  In addition to this, we add the capability to query an individual property of a mbean.  Client will send http request,
> http://hostname/jmx?get=meanName::property
> to query from server.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16561-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 19:05:12 2011
Return-Path: <common-issues-return-16561-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CCCF36E55
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 19:05:12 +0000 (UTC)
Received: (qmail 76688 invoked by uid 500); 20 Jun 2011 19:05:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76651 invoked by uid 500); 20 Jun 2011 19:05:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76613 invoked by uid 99); 20 Jun 2011 19:05:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 19:05:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 19:05:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6ADB042445F
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 19:04:49 +0000 (UTC)
Date: Mon, 20 Jun 2011 19:04:49 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <583092230.22051.1308596689434.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052161#comment-13052161 ] 

Tom White commented on HADOOP-7206:
-----------------------------------

Nicholas, I agree that javadoc is needed. Thanks for pointing it out.

T Jake, would you like to create a new patch which adds javadoc? I think a new JIRA would be fine.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: T Jake Luciani
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16562-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 19:43:09 2011
Return-Path: <common-issues-return-16562-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E09DC6A0F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 19:43:08 +0000 (UTC)
Received: (qmail 61991 invoked by uid 500); 20 Jun 2011 19:43:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61915 invoked by uid 500); 20 Jun 2011 19:43:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61901 invoked by uid 99); 20 Jun 2011 19:43:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 19:43:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 19:43:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 875BE4243EE
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 19:42:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 19:42:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1793256765.22217.1308598967551.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32237646.17061288907081267.JavaMail.jira@thor>
Subject: [jira] [Updated] (HADOOP-7020) establish a "Powered by Hadoop" logo
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-7020:
----------------------------

    Attachment: pbh-64-logos.png

Just got a few minutes to look at these. I must say that I'm very disappointed. Some of these are very nice clip art but none of them are appropriate powered-by logos, as they are completely illegible at typical logo size: <= 64 pixels in height.

My gimpfu is weak, but here is an idea, IMHO, is better than the existing attempts at a "powered by apache logo" in several aspects:

# Good continuity with existing Apache and Hadoop logos
# Perfectly legible at logo size (64 pixel height)
# Theme friendly (looks good on black background as well).

> establish a "Powered by Hadoop" logo
> ------------------------------------
>
>                 Key: HADOOP-7020
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7020
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: site
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: site
>
>         Attachments: PoweredByApacheHadoop.png, PoweredByHadoop_Small.jpg, hadoop-elephant-pb.jpeg, hadoop.JPG, pbh-64-logos.png, powered-by-hadoop-small.png, powered-by-hadoop.png, powered-by-proposal.jpg, powered_by_hadoop.jpg
>
>
> We should agree on a Powered By Hadoop logo, as suggested in:
> http://www.apache.org/foundation/marks/pmcs#poweredby

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16563-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 19:45:11 2011
Return-Path: <common-issues-return-16563-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E33BE6C10
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 19:45:10 +0000 (UTC)
Received: (qmail 65573 invoked by uid 500); 20 Jun 2011 19:45:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65516 invoked by uid 500); 20 Jun 2011 19:45:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65492 invoked by uid 99); 20 Jun 2011 19:45:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 19:45:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 19:45:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 895FC424510
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 19:44:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 19:44:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1108334668.22242.1308599087559.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32237646.17061288907081267.JavaMail.jira@thor>
Subject: [jira] [Updated] (HADOOP-7020) establish a "Powered by Hadoop" logo
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-7020:
----------------------------

    Attachment: powered-by-apache-hadoop.png

A larger png for people to play with.

> establish a "Powered by Hadoop" logo
> ------------------------------------
>
>                 Key: HADOOP-7020
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7020
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: site
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: site
>
>         Attachments: PoweredByApacheHadoop.png, PoweredByHadoop_Small.jpg, hadoop-elephant-pb.jpeg, hadoop.JPG, pbh-64-logos.png, powered-by-apache-hadoop.png, powered-by-hadoop-small.png, powered-by-hadoop.png, powered-by-proposal.jpg, powered_by_hadoop.jpg
>
>
> We should agree on a Powered By Hadoop logo, as suggested in:
> http://www.apache.org/foundation/marks/pmcs#poweredby

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16564-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 19:57:11 2011
Return-Path: <common-issues-return-16564-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0ED6E6359
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 19:57:11 +0000 (UTC)
Received: (qmail 86401 invoked by uid 500); 20 Jun 2011 19:57:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86366 invoked by uid 500); 20 Jun 2011 19:57:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86328 invoked by uid 99); 20 Jun 2011 19:57:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 19:57:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 19:57:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8627C425825
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 19:56:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 19:56:47 +0000 (UTC)
From: "T Jake Luciani (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1760011225.22288.1308599807546.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7408) Add javadoc for SnappyCodec
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Add javadoc for SnappyCodec
---------------------------

                 Key: HADOOP-7408
                 URL: https://issues.apache.org/jira/browse/HADOOP-7408
             Project: Hadoop Common
          Issue Type: Bug
          Components: io
            Reporter: T Jake Luciani
            Priority: Trivial
             Fix For: 0.23.0


HADOOP-7206 failed to include a javadoc for public methods.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16565-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 19:59:08 2011
Return-Path: <common-issues-return-16565-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A8ADA63CD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 19:59:08 +0000 (UTC)
Received: (qmail 90238 invoked by uid 500); 20 Jun 2011 19:59:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90173 invoked by uid 500); 20 Jun 2011 19:59:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90165 invoked by uid 99); 20 Jun 2011 19:59:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 19:59:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 19:59:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7EB1D425912
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 19:58:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 19:58:47 +0000 (UTC)
From: "T Jake Luciani (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <89780526.22296.1308599927515.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1760011225.22288.1308599807546.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7408) Add javadoc for SnappyCodec
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

T Jake Luciani updated HADOOP-7408:
-----------------------------------

    Attachment: v1-HADOOP-7408-add-snappy-javadoc.txt

> Add javadoc for SnappyCodec
> ---------------------------
>
>                 Key: HADOOP-7408
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7408
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: T Jake Luciani
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: v1-HADOOP-7408-add-snappy-javadoc.txt
>
>
> HADOOP-7206 failed to include a javadoc for public methods.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16566-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 19:59:10 2011
Return-Path: <common-issues-return-16566-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A96AF63E0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 19:59:10 +0000 (UTC)
Received: (qmail 90509 invoked by uid 500); 20 Jun 2011 19:59:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90463 invoked by uid 500); 20 Jun 2011 19:59:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90455 invoked by uid 99); 20 Jun 2011 19:59:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 19:59:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 19:59:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 691E6425943
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 19:58:49 +0000 (UTC)
Date: Mon, 20 Jun 2011 19:58:49 +0000 (UTC)
From: "T Jake Luciani (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <194819688.22345.1308599929427.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052190#comment-13052190 ] 

T Jake Luciani commented on HADOOP-7206:
----------------------------------------

Sure. Added in HADOOP-7408

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: T Jake Luciani
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16567-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 19:59:12 2011
Return-Path: <common-issues-return-16567-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C66C463F2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 19:59:12 +0000 (UTC)
Received: (qmail 90776 invoked by uid 500); 20 Jun 2011 19:59:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90743 invoked by uid 500); 20 Jun 2011 19:59:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90735 invoked by uid 99); 20 Jun 2011 19:59:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 19:59:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 19:59:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8CDAF425948
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 19:58:49 +0000 (UTC)
Date: Mon, 20 Jun 2011 19:58:49 +0000 (UTC)
From: "T Jake Luciani (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2081471578.22350.1308599929573.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1760011225.22288.1308599807546.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7408) Add javadoc for SnappyCodec
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

T Jake Luciani updated HADOOP-7408:
-----------------------------------

    Assignee: T Jake Luciani
      Status: Patch Available  (was: Open)

> Add javadoc for SnappyCodec
> ---------------------------
>
>                 Key: HADOOP-7408
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7408
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: T Jake Luciani
>            Assignee: T Jake Luciani
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: v1-HADOOP-7408-add-snappy-javadoc.txt
>
>
> HADOOP-7206 failed to include a javadoc for public methods.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16568-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 20:13:09 2011
Return-Path: <common-issues-return-16568-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EBE616C77
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 20:13:08 +0000 (UTC)
Received: (qmail 15634 invoked by uid 500); 20 Jun 2011 20:13:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15605 invoked by uid 500); 20 Jun 2011 20:13:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15597 invoked by uid 99); 20 Jun 2011 20:13:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 20:13:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 20:13:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A220F425F0A
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 20:12:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 20:12:47 +0000 (UTC)
From: "monica beckwith (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1172254581.22393.1308600767660.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1229925666.15226.1308324167811.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7401) Unit test TestPureJavaCRC32 warmup
 code warms up the not-so-important loop in PureJavaCRC32.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

monica beckwith updated HADOOP-7401:
------------------------------------

    Status: Open  (was: Patch Available)

> Unit test TestPureJavaCRC32 warmup code warms up the not-so-important loop in PureJavaCRC32.update()
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7401
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7401
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>         Environment: Solaris-Sparcv9; Solaris-AMD64
>            Reporter: monica beckwith
>            Assignee: monica beckwith
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-7401.patch
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> When the warmup code sequence in TestPureJavaCRC32.java is executed, it sends size=len=2 and due to the value of 'trials' in for loop in doBench(), the crc.update() gets run > the compile threshold, thus providing the information that 'while 0<len<7' is a hot loop and 'while len>7' is a cold loop. This brings the MB/s number for len > 7 in PureJavaCRC32.update() way down (e.g. ~28.5% for size=len=65536).
> The workaround would be to use size=len=>7 (so just having size=len=2101 ahead of size=len=2 will do the trick) in the warmup section. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16569-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 20:13:09 2011
Return-Path: <common-issues-return-16569-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 878726C8D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 20:13:09 +0000 (UTC)
Received: (qmail 15960 invoked by uid 500); 20 Jun 2011 20:13:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15895 invoked by uid 500); 20 Jun 2011 20:13:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15773 invoked by uid 99); 20 Jun 2011 20:13:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 20:13:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 20:13:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C5AF9425F0E
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 20:12:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 20:12:47 +0000 (UTC)
From: "monica beckwith (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1480045949.22397.1308600767806.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1229925666.15226.1308324167811.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7401) Unit test TestPureJavaCRC32 warmup
 code warms up the not-so-important loop in PureJavaCRC32.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

monica beckwith updated HADOOP-7401:
------------------------------------

    Attachment:     (was: HADOOP-7401.patch)

> Unit test TestPureJavaCRC32 warmup code warms up the not-so-important loop in PureJavaCRC32.update()
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7401
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7401
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>         Environment: Solaris-Sparcv9; Solaris-AMD64
>            Reporter: monica beckwith
>            Assignee: monica beckwith
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-7401.patch
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> When the warmup code sequence in TestPureJavaCRC32.java is executed, it sends size=len=2 and due to the value of 'trials' in for loop in doBench(), the crc.update() gets run > the compile threshold, thus providing the information that 'while 0<len<7' is a hot loop and 'while len>7' is a cold loop. This brings the MB/s number for len > 7 in PureJavaCRC32.update() way down (e.g. ~28.5% for size=len=65536).
> The workaround would be to use size=len=>7 (so just having size=len=2101 ahead of size=len=2 will do the trick) in the warmup section. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16570-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 20:13:11 2011
Return-Path: <common-issues-return-16570-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 442366C99
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 20:13:11 +0000 (UTC)
Received: (qmail 16471 invoked by uid 500); 20 Jun 2011 20:13:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16440 invoked by uid 500); 20 Jun 2011 20:13:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16432 invoked by uid 99); 20 Jun 2011 20:13:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 20:13:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 20:13:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 31CD0425F1A
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 20:12:48 +0000 (UTC)
Date: Mon, 20 Jun 2011 20:12:48 +0000 (UTC)
From: "monica beckwith (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <694609826.22409.1308600768200.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1229925666.15226.1308324167811.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7401) Unit test TestPureJavaCRC32 warmup
 code warms up the not-so-important loop in PureJavaCRC32.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

monica beckwith updated HADOOP-7401:
------------------------------------

    Attachment: hadoop-7401.patch

> Unit test TestPureJavaCRC32 warmup code warms up the not-so-important loop in PureJavaCRC32.update()
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7401
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7401
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>         Environment: Solaris-Sparcv9; Solaris-AMD64
>            Reporter: monica beckwith
>            Assignee: monica beckwith
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-7401.patch
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> When the warmup code sequence in TestPureJavaCRC32.java is executed, it sends size=len=2 and due to the value of 'trials' in for loop in doBench(), the crc.update() gets run > the compile threshold, thus providing the information that 'while 0<len<7' is a hot loop and 'while len>7' is a cold loop. This brings the MB/s number for len > 7 in PureJavaCRC32.update() way down (e.g. ~28.5% for size=len=65536).
> The workaround would be to use size=len=>7 (so just having size=len=2101 ahead of size=len=2 will do the trick) in the warmup section. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16571-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 20:13:11 2011
Return-Path: <common-issues-return-16571-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B40456CB8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 20:13:11 +0000 (UTC)
Received: (qmail 16797 invoked by uid 500); 20 Jun 2011 20:13:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16756 invoked by uid 500); 20 Jun 2011 20:13:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16681 invoked by uid 99); 20 Jun 2011 20:13:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 20:13:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 20:13:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5B1EE425F1E
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 20:12:48 +0000 (UTC)
Date: Mon, 20 Jun 2011 20:12:48 +0000 (UTC)
From: "monica beckwith (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <709150972.22413.1308600768370.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1229925666.15226.1308324167811.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7401) Unit test TestPureJavaCRC32 warmup
 code warms up the not-so-important loop in PureJavaCRC32.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

monica beckwith updated HADOOP-7401:
------------------------------------

    Status: Patch Available  (was: Open)

> Unit test TestPureJavaCRC32 warmup code warms up the not-so-important loop in PureJavaCRC32.update()
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7401
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7401
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>         Environment: Solaris-Sparcv9; Solaris-AMD64
>            Reporter: monica beckwith
>            Assignee: monica beckwith
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-7401.patch
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> When the warmup code sequence in TestPureJavaCRC32.java is executed, it sends size=len=2 and due to the value of 'trials' in for loop in doBench(), the crc.update() gets run > the compile threshold, thus providing the information that 'while 0<len<7' is a hot loop and 'while len>7' is a cold loop. This brings the MB/s number for len > 7 in PureJavaCRC32.update() way down (e.g. ~28.5% for size=len=65536).
> The workaround would be to use size=len=>7 (so just having size=len=2101 ahead of size=len=2 will do the trick) in the warmup section. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16572-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 20:15:08 2011
Return-Path: <common-issues-return-16572-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CB9006D71
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 20:15:08 +0000 (UTC)
Received: (qmail 23874 invoked by uid 500); 20 Jun 2011 20:15:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23847 invoked by uid 500); 20 Jun 2011 20:15:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23839 invoked by uid 99); 20 Jun 2011 20:15:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 20:15:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 20:15:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8C4C2424038
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 20:14:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 20:14:47 +0000 (UTC)
From: "monica beckwith (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <910028750.22425.1308600887571.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1229925666.15226.1308324167811.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7401) Unit test TestPureJavaCRC32 warmup
 code warms up the not-so-important loop in PureJavaCRC32.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052200#comment-13052200 ] 

monica beckwith commented on HADOOP-7401:
-----------------------------------------

Thanks! for the comments. I've updated the patch. I'll post the numbers soon.

> Unit test TestPureJavaCRC32 warmup code warms up the not-so-important loop in PureJavaCRC32.update()
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7401
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7401
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>         Environment: Solaris-Sparcv9; Solaris-AMD64
>            Reporter: monica beckwith
>            Assignee: monica beckwith
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-7401.patch
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> When the warmup code sequence in TestPureJavaCRC32.java is executed, it sends size=len=2 and due to the value of 'trials' in for loop in doBench(), the crc.update() gets run > the compile threshold, thus providing the information that 'while 0<len<7' is a hot loop and 'while len>7' is a cold loop. This brings the MB/s number for len > 7 in PureJavaCRC32.update() way down (e.g. ~28.5% for size=len=65536).
> The workaround would be to use size=len=>7 (so just having size=len=2101 ahead of size=len=2 will do the trick) in the warmup section. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16573-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 20:15:09 2011
Return-Path: <common-issues-return-16573-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0A1B06D82
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 20:15:09 +0000 (UTC)
Received: (qmail 24149 invoked by uid 500); 20 Jun 2011 20:15:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24090 invoked by uid 500); 20 Jun 2011 20:15:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23924 invoked by uid 99); 20 Jun 2011 20:15:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 20:15:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 20:15:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9A13B42403A
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 20:14:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 20:14:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1979315054.22427.1308600887628.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1760011225.22288.1308599807546.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7408) Add javadoc for SnappyCodec
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052201#comment-13052201 ] 

Hadoop QA commented on HADOOP-7408:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12483200/v1-HADOOP-7408-add-snappy-javadoc.txt
  against trunk revision 1137724.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/657//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/657//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/657//console

This message is automatically generated.

> Add javadoc for SnappyCodec
> ---------------------------
>
>                 Key: HADOOP-7408
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7408
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: T Jake Luciani
>            Assignee: T Jake Luciani
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: v1-HADOOP-7408-add-snappy-javadoc.txt
>
>
> HADOOP-7206 failed to include a javadoc for public methods.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16574-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 20:33:11 2011
Return-Path: <common-issues-return-16574-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 001D864F5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 20:33:10 +0000 (UTC)
Received: (qmail 47623 invoked by uid 500); 20 Jun 2011 20:33:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47545 invoked by uid 500); 20 Jun 2011 20:33:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47537 invoked by uid 99); 20 Jun 2011 20:33:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 20:33:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 20:33:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DF76A424906
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 20:32:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 20:32:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <77027627.22457.1308601967912.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <557825167.38099.1306198007453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7324) Ganglia plugins for metrics v2
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052206#comment-13052206 ] 

Luke Lu commented on HADOOP-7324:
---------------------------------

Do people care about Gangalia 3.0.x? Would Ganglia 3.1+ support suffice?

> Ganglia plugins for metrics v2
> ------------------------------
>
>                 Key: HADOOP-7324
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7324
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.203.0, 0.23.0
>            Reporter: Luke Lu
>            Priority: Blocker
>              Labels: regression
>             Fix For: 0.23.0
>
>
> Although, all metrics in metrics v2 are exposed via the standard JMX mechanisms, most users are using Ganglia to collect metrics.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16575-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 20:35:08 2011
Return-Path: <common-issues-return-16575-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E92136DAA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 20:35:08 +0000 (UTC)
Received: (qmail 51540 invoked by uid 500); 20 Jun 2011 20:35:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51502 invoked by uid 500); 20 Jun 2011 20:35:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51492 invoked by uid 99); 20 Jun 2011 20:35:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 20:35:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 20:35:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8B29E4249BC
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 20:34:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 20:34:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1421388965.22466.1308602087566.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1229925666.15226.1308324167811.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7401) Unit test TestPureJavaCRC32 warmup
 code warms up the not-so-important loop in PureJavaCRC32.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052210#comment-13052210 ] 

Hadoop QA commented on HADOOP-7401:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12483202/hadoop-7401.patch
  against trunk revision 1137724.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.fs.TestGetFileBlockLocations

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/658//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/658//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/658//console

This message is automatically generated.

> Unit test TestPureJavaCRC32 warmup code warms up the not-so-important loop in PureJavaCRC32.update()
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7401
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7401
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>         Environment: Solaris-Sparcv9; Solaris-AMD64
>            Reporter: monica beckwith
>            Assignee: monica beckwith
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-7401.patch
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> When the warmup code sequence in TestPureJavaCRC32.java is executed, it sends size=len=2 and due to the value of 'trials' in for loop in doBench(), the crc.update() gets run > the compile threshold, thus providing the information that 'while 0<len<7' is a hot loop and 'while len>7' is a cold loop. This brings the MB/s number for len > 7 in PureJavaCRC32.update() way down (e.g. ~28.5% for size=len=65536).
> The workaround would be to use size=len=>7 (so just having size=len=2101 ahead of size=len=2 will do the trick) in the warmup section. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16576-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 21:03:12 2011
Return-Path: <common-issues-return-16576-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B211F6AE9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 21:03:12 +0000 (UTC)
Received: (qmail 4097 invoked by uid 500); 20 Jun 2011 21:03:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3966 invoked by uid 500); 20 Jun 2011 21:03:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3905 invoked by uid 99); 20 Jun 2011 21:03:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 21:03:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 21:03:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 29608424584
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 21:02:49 +0000 (UTC)
Date: Mon, 20 Jun 2011 21:02:49 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1516566096.22569.1308603769166.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE resolved HADOOP-7206.
--------------------------------------------

    Resolution: Fixed

Let's fix the javadoc in HADOOP-7408.  Thanks T Jake and Tom.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: T Jake Luciani
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16577-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 21:41:11 2011
Return-Path: <common-issues-return-16577-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6209D668E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 21:41:11 +0000 (UTC)
Received: (qmail 78554 invoked by uid 500); 20 Jun 2011 21:41:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78532 invoked by uid 500); 20 Jun 2011 21:41:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78524 invoked by uid 99); 20 Jun 2011 21:41:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 21:41:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 21:41:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7569F4244B4
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 21:40:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 21:40:47 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1822093553.22641.1308606047477.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1760011225.22288.1308599807546.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7408) Add javadoc for SnappyCodec
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-7408:
------------------------------

    Attachment: HADOOP-7408.patch

Looks good. I've updated your patch to annotate SnappyCodec as public evolving (like the other codecs in the package), and to provide a link to the snappy homepage from this class.

> Add javadoc for SnappyCodec
> ---------------------------
>
>                 Key: HADOOP-7408
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7408
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: T Jake Luciani
>            Assignee: T Jake Luciani
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7408.patch, v1-HADOOP-7408-add-snappy-javadoc.txt
>
>
> HADOOP-7206 failed to include a javadoc for public methods.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16578-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 20 21:55:11 2011
Return-Path: <common-issues-return-16578-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3DB716F87
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 20 Jun 2011 21:55:11 +0000 (UTC)
Received: (qmail 5096 invoked by uid 500); 20 Jun 2011 21:55:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5043 invoked by uid 500); 20 Jun 2011 21:55:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4861 invoked by uid 99); 20 Jun 2011 21:55:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 21:55:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jun 2011 21:55:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A6F06424AE5
	for <common-issues@hadoop.apache.org>; Mon, 20 Jun 2011 21:54:47 +0000 (UTC)
Date: Mon, 20 Jun 2011 21:54:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <416007355.22668.1308606887680.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1760011225.22288.1308599807546.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7408) Add javadoc for SnappyCodec
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052242#comment-13052242 ] 

Hadoop QA commented on HADOOP-7408:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12483211/HADOOP-7408.patch
  against trunk revision 1137724.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/659//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/659//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/659//console

This message is automatically generated.

> Add javadoc for SnappyCodec
> ---------------------------
>
>                 Key: HADOOP-7408
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7408
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: T Jake Luciani
>            Assignee: T Jake Luciani
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7408.patch, v1-HADOOP-7408-add-snappy-javadoc.txt
>
>
> HADOOP-7206 failed to include a javadoc for public methods.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16579-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 00:29:10 2011
Return-Path: <common-issues-return-16579-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8F5CB6EA9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 21 Jun 2011 00:29:10 +0000 (UTC)
Received: (qmail 84888 invoked by uid 500); 21 Jun 2011 00:29:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84855 invoked by uid 500); 21 Jun 2011 00:29:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84846 invoked by uid 99); 21 Jun 2011 00:29:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 00:29:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 00:29:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7B4C442233E
	for <common-issues@hadoop.apache.org>; Tue, 21 Jun 2011 00:28:47 +0000 (UTC)
Date: Tue, 21 Jun 2011 00:28:47 +0000 (UTC)
From: "T Jake Luciani (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <665392662.22968.1308616127502.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1760011225.22288.1308599807546.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7408) Add javadoc for SnappyCodec
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052290#comment-13052290 ] 

T Jake Luciani commented on HADOOP-7408:
----------------------------------------

+1

> Add javadoc for SnappyCodec
> ---------------------------
>
>                 Key: HADOOP-7408
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7408
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: T Jake Luciani
>            Assignee: T Jake Luciani
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7408.patch, v1-HADOOP-7408-add-snappy-javadoc.txt
>
>
> HADOOP-7206 failed to include a javadoc for public methods.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16580-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 01:37:10 2011
Return-Path: <common-issues-return-16580-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9AA846BBA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 21 Jun 2011 01:37:10 +0000 (UTC)
Received: (qmail 44911 invoked by uid 500); 21 Jun 2011 01:37:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44885 invoked by uid 500); 21 Jun 2011 01:37:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44877 invoked by uid 99); 21 Jun 2011 01:37:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 01:37:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 01:37:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8E13D42550F
	for <common-issues@hadoop.apache.org>; Tue, 21 Jun 2011 01:36:47 +0000 (UTC)
Date: Tue, 21 Jun 2011 01:36:47 +0000 (UTC)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <401221421.23096.1308620207579.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <18635139.89501293838185702.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7082) Configuration.writeXML should not
 hold lock while outputting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052309#comment-13052309 ] 

Chris Douglas commented on HADOOP-7082:
---------------------------------------

bq. Can someone pull this patch into MR-279 branch?

Done.

> Configuration.writeXML should not hold lock while outputting
> ------------------------------------------------------------
>
>                 Key: HADOOP-7082
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7082
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7082.txt
>
>
> Common side of HDFS-1542

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16581-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 02:01:10 2011
Return-Path: <common-issues-return-16581-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 322046444
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 21 Jun 2011 02:01:10 +0000 (UTC)
Received: (qmail 63467 invoked by uid 500); 21 Jun 2011 02:01:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63434 invoked by uid 500); 21 Jun 2011 02:01:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63426 invoked by uid 99); 21 Jun 2011 02:01:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 02:01:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 02:01:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 10AD1425AF3
	for <common-issues@hadoop.apache.org>; Tue, 21 Jun 2011 02:00:49 +0000 (UTC)
Date: Tue, 21 Jun 2011 02:00:49 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <411884783.23135.1308621649064.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1760011225.22288.1308599807546.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7408) Add javadoc for SnappyCodec
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052315#comment-13052315 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7408:
------------------------------------------------

Why passing conf to the following method?
{code}
  public static boolean isNativeSnappyLoaded(Configuration conf) {
    return nativeSnappyLoaded;
  }
{code}
How about remove {{isNativeSnappyLoaded(..)}}  and change {{nativeSnappyLoaded}} to final?

> Add javadoc for SnappyCodec
> ---------------------------
>
>                 Key: HADOOP-7408
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7408
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: T Jake Luciani
>            Assignee: T Jake Luciani
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7408.patch, v1-HADOOP-7408-add-snappy-javadoc.txt
>
>
> HADOOP-7206 failed to include a javadoc for public methods.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16582-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 02:03:10 2011
Return-Path: <common-issues-return-16582-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8AAAC43E9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 21 Jun 2011 02:03:10 +0000 (UTC)
Received: (qmail 64236 invoked by uid 500); 21 Jun 2011 02:03:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64145 invoked by uid 500); 21 Jun 2011 02:03:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64137 invoked by uid 99); 21 Jun 2011 02:03:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 02:03:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 02:03:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8EBA1425BF4
	for <common-issues@hadoop.apache.org>; Tue, 21 Jun 2011 02:02:47 +0000 (UTC)
Date: Tue, 21 Jun 2011 02:02:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <184822398.23144.1308621767581.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1760011225.22288.1308599807546.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7408) Add javadoc for SnappyCodec
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052318#comment-13052318 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7408:
------------------------------------------------

Also, could you add {{@Override}} for overriding methods?

> Add javadoc for SnappyCodec
> ---------------------------
>
>                 Key: HADOOP-7408
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7408
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: T Jake Luciani
>            Assignee: T Jake Luciani
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7408.patch, v1-HADOOP-7408-add-snappy-javadoc.txt
>
>
> HADOOP-7206 failed to include a javadoc for public methods.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16583-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 05:18:20 2011
Return-Path: <common-issues-return-16583-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CB8F7436B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 21 Jun 2011 05:18:20 +0000 (UTC)
Received: (qmail 13336 invoked by uid 500); 21 Jun 2011 05:18:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13091 invoked by uid 500); 21 Jun 2011 05:18:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13045 invoked by uid 99); 21 Jun 2011 05:18:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 05:18:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 05:18:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8F44B425494
	for <common-issues@hadoop.apache.org>; Tue, 21 Jun 2011 05:17:47 +0000 (UTC)
Date: Tue, 21 Jun 2011 05:17:47 +0000 (UTC)
From: "Bob Liu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <839456896.23368.1308633467583.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <557825167.38099.1306198007453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7324) Ganglia plugins for metrics v2
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052352#comment-13052352 ] 

Bob Liu commented on HADOOP-7324:
---------------------------------

Hello Luke,

I think support for Ganglia 3.1.x is suffice... as it's easy to upgrade ganglia from 3.0.x to 3.1.x.

By the way, any ETA as to when the ganglia plugins will be available for hadoop-20.203 using metrics v2?


> Ganglia plugins for metrics v2
> ------------------------------
>
>                 Key: HADOOP-7324
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7324
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.203.0, 0.23.0
>            Reporter: Luke Lu
>            Priority: Blocker
>              Labels: regression
>             Fix For: 0.23.0
>
>
> Although, all metrics in metrics v2 are exposed via the standard JMX mechanisms, most users are using Ganglia to collect metrics.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16584-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 06:34:24 2011
Return-Path: <common-issues-return-16584-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ABE3F418E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 21 Jun 2011 06:34:24 +0000 (UTC)
Received: (qmail 72821 invoked by uid 500); 21 Jun 2011 06:34:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71083 invoked by uid 500); 21 Jun 2011 06:34:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70922 invoked by uid 99); 21 Jun 2011 06:34:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 06:34:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 06:34:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 00934425A6D
	for <common-issues@hadoop.apache.org>; Tue, 21 Jun 2011 06:33:48 +0000 (UTC)
Date: Tue, 21 Jun 2011 06:33:47 +0000 (UTC)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <779737299.23419.1308638027999.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2144547827.20484.1308569267525.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7406) Add a "I broke the Hadoop build"
 poster
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052367#comment-13052367 ] 

Konstantin Boudnik commented on HADOOP-7406:
--------------------------------------------

+1 Steve.

> Add a "I broke the Hadoop build" poster
> ---------------------------------------
>
>                 Key: HADOOP-7406
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7406
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Minor
>         Attachments: breaking_the_hadoop_build.odp
>
>
> Some people have been known to check in code that does not work, and hence break the build. These people (and I may be one of them) deserve to have their contribution acknowledged with an "I broke the build" poster. I have a draft slide for this, using an upside down copy of the logo. It may be A4 paper, when US-letter is probably more appropriate. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16585-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 09:29:10 2011
Return-Path: <common-issues-return-16585-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D8D90469F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 21 Jun 2011 09:29:10 +0000 (UTC)
Received: (qmail 28269 invoked by uid 500); 21 Jun 2011 09:29:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28224 invoked by uid 500); 21 Jun 2011 09:29:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28216 invoked by uid 99); 21 Jun 2011 09:29:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 09:29:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 09:29:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 920EC425066
	for <common-issues@hadoop.apache.org>; Tue, 21 Jun 2011 09:28:48 +0000 (UTC)
Date: Tue, 21 Jun 2011 09:28:48 +0000 (UTC)
From: "XieXianshan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <933619452.23812.1308648528595.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <740424940.5878.1308116867420.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7395) The infomation returned by the wrong
 usage of the command "job -counter<job-id> <group-name> <counter-name> "is
 not appropriate
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

XieXianshan updated HADOOP-7395:
--------------------------------

    Attachment: Hadoop_7395

> The infomation returned by the wrong usage of the command "job -counter<job-id> <group-name> <counter-name> "is not appropriate
> -------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7395
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7395
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Yan Jinshuang
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: Hadoop_7395
>
>
> When use the command "job -counter<job-id> <group-name> <counter-name> " with wrong group-name or wrong counter-name(with correct job-id), the result is always 0. It is better to show the user the detail, like "illegal group-name", "illegal counter-name", etc. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16586-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 09:41:11 2011
Return-Path: <common-issues-return-16586-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 296A4411B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 21 Jun 2011 09:41:11 +0000 (UTC)
Received: (qmail 49296 invoked by uid 500); 21 Jun 2011 09:41:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49256 invoked by uid 500); 21 Jun 2011 09:41:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49248 invoked by uid 99); 21 Jun 2011 09:41:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 09:41:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 09:41:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0A041425864
	for <common-issues@hadoop.apache.org>; Tue, 21 Jun 2011 09:40:48 +0000 (UTC)
Date: Tue, 21 Jun 2011 09:40:48 +0000 (UTC)
From: "XieXianshan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <777363857.23855.1308649248037.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <740424940.5878.1308116867420.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7395) The infomation returned by the wrong
 usage of the command "job -counter<job-id> <group-name> <counter-name> "is
 not appropriate
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

XieXianshan updated HADOOP-7395:
--------------------------------

    Status: Patch Available  (was: Open)

Fix this issue.
The error message is as the following after the patch is done:
[Could not find group FileSystemCounters_err] <--incorrect group name
 or
[Could not find counter FILE_BYTES_READ_err in the group FileSystemCounters] <--incorrect counter name



> The infomation returned by the wrong usage of the command "job -counter<job-id> <group-name> <counter-name> "is not appropriate
> -------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7395
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7395
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Yan Jinshuang
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: Hadoop_7395
>
>
> When use the command "job -counter<job-id> <group-name> <counter-name> " with wrong group-name or wrong counter-name(with correct job-id), the result is always 0. It is better to show the user the detail, like "illegal group-name", "illegal counter-name", etc. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16587-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 09:45:10 2011
Return-Path: <common-issues-return-16587-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 941CB42D8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 21 Jun 2011 09:45:10 +0000 (UTC)
Received: (qmail 52812 invoked by uid 500); 21 Jun 2011 09:45:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52739 invoked by uid 500); 21 Jun 2011 09:45:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52731 invoked by uid 99); 21 Jun 2011 09:45:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 09:45:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 09:45:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7BECE425A3F
	for <common-issues@hadoop.apache.org>; Tue, 21 Jun 2011 09:44:47 +0000 (UTC)
Date: Tue, 21 Jun 2011 09:44:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1312331457.23859.1308649487504.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <740424940.5878.1308116867420.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7395) The infomation returned by the
 wrong usage of the command "job -counter<job-id> <group-name>
 <counter-name> "is not appropriate
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052444#comment-13052444 ] 

Hadoop QA commented on HADOOP-7395:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12483256/Hadoop_7395
  against trunk revision 1137724.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/660//console

This message is automatically generated.

> The infomation returned by the wrong usage of the command "job -counter<job-id> <group-name> <counter-name> "is not appropriate
> -------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7395
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7395
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Yan Jinshuang
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: Hadoop_7395
>
>
> When use the command "job -counter<job-id> <group-name> <counter-name> " with wrong group-name or wrong counter-name(with correct job-id), the result is always 0. It is better to show the user the detail, like "illegal group-name", "illegal counter-name", etc. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16588-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 10:39:09 2011
Return-Path: <common-issues-return-16588-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3B94849F4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 21 Jun 2011 10:39:09 +0000 (UTC)
Received: (qmail 53300 invoked by uid 500); 21 Jun 2011 10:39:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53192 invoked by uid 500); 21 Jun 2011 10:39:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53176 invoked by uid 99); 21 Jun 2011 10:39:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 10:39:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 10:39:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 86EE042568F
	for <common-issues@hadoop.apache.org>; Tue, 21 Jun 2011 10:38:47 +0000 (UTC)
Date: Tue, 21 Jun 2011 10:38:47 +0000 (UTC)
From: "Vitalii Tymchyshyn (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1042435816.23959.1308652727549.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2007918179.15449.1305558467414.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7294) FileUtil uses wrong stat command for
 FreeBSD
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vitalii Tymchyshyn updated HADOOP-7294:
---------------------------------------

    Attachment: patch.diff

This is patch for 0.21 release. I did a branch in my own svn, so revisions are off.

> FileUtil uses wrong stat command for FreeBSD
> --------------------------------------------
>
>                 Key: HADOOP-7294
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7294
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0
>         Environment: FreeBSD 8.0-STABLE
>            Reporter: Vitalii Tymchyshyn
>         Attachments: patch.diff
>
>
> I get next exception when try to use append:
> 2011-05-16 17:07:54,648 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.112.0.207:50010, storageID=DS-1047171559-
> 10.112.0.207-50010-1302796304164, infoPort=50075, ipcPort=50020):DataXceiver
> java.io.IOException: Failed to get link count on file /var/data/hdfs/data/current/finalized/subdir26/subdir17/subdir55/blk_-1266943884751786595:
>  message=null; error=stat: illegal option -- c; exit value=1
>         at org.apache.hadoop.fs.FileUtil.createIOException(FileUtil.java:709)
>         at org.apache.hadoop.fs.FileUtil.access$000(FileUtil.java:42)
>         at org.apache.hadoop.fs.FileUtil$HardLink.getLinkCount(FileUtil.java:682)
>         at org.apache.hadoop.hdfs.server.datanode.ReplicaInfo.unlinkBlock(ReplicaInfo.java:215)
>         at org.apache.hadoop.hdfs.server.datanode.FSDataset.append(FSDataset.java:1116)
> It seems that FreeBSD is treated like UNIX and so calls 'stat -c%h', while FreeBSD is much more like Mac (since they have same BSD roots):
> $ stat --help
> stat: illegal option -- -
> usage: stat [-FlLnqrsx] [-f format] [-t timefmt] [file ...]
> $ stat -f%l a_file
> 1

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16589-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 10:39:10 2011
Return-Path: <common-issues-return-16589-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3B8E849F3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 21 Jun 2011 10:39:09 +0000 (UTC)
Received: (qmail 53318 invoked by uid 500); 21 Jun 2011 10:39:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53194 invoked by uid 500); 21 Jun 2011 10:39:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53182 invoked by uid 99); 21 Jun 2011 10:39:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 10:39:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 10:39:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BC81C425696
	for <common-issues@hadoop.apache.org>; Tue, 21 Jun 2011 10:38:47 +0000 (UTC)
Date: Tue, 21 Jun 2011 10:38:47 +0000 (UTC)
From: "Vitalii Tymchyshyn (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <151652946.23965.1308652727769.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2007918179.15449.1305558467414.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7294) FileUtil uses wrong stat command for
 FreeBSD
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vitalii Tymchyshyn updated HADOOP-7294:
---------------------------------------

    Status: Patch Available  (was: Open)

> FileUtil uses wrong stat command for FreeBSD
> --------------------------------------------
>
>                 Key: HADOOP-7294
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7294
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0
>         Environment: FreeBSD 8.0-STABLE
>            Reporter: Vitalii Tymchyshyn
>         Attachments: patch.diff
>
>
> I get next exception when try to use append:
> 2011-05-16 17:07:54,648 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.112.0.207:50010, storageID=DS-1047171559-
> 10.112.0.207-50010-1302796304164, infoPort=50075, ipcPort=50020):DataXceiver
> java.io.IOException: Failed to get link count on file /var/data/hdfs/data/current/finalized/subdir26/subdir17/subdir55/blk_-1266943884751786595:
>  message=null; error=stat: illegal option -- c; exit value=1
>         at org.apache.hadoop.fs.FileUtil.createIOException(FileUtil.java:709)
>         at org.apache.hadoop.fs.FileUtil.access$000(FileUtil.java:42)
>         at org.apache.hadoop.fs.FileUtil$HardLink.getLinkCount(FileUtil.java:682)
>         at org.apache.hadoop.hdfs.server.datanode.ReplicaInfo.unlinkBlock(ReplicaInfo.java:215)
>         at org.apache.hadoop.hdfs.server.datanode.FSDataset.append(FSDataset.java:1116)
> It seems that FreeBSD is treated like UNIX and so calls 'stat -c%h', while FreeBSD is much more like Mac (since they have same BSD roots):
> $ stat --help
> stat: illegal option -- -
> usage: stat [-FlLnqrsx] [-f format] [-t timefmt] [file ...]
> $ stat -f%l a_file
> 1

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16590-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 10:45:11 2011
Return-Path: <common-issues-return-16590-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 695024DAC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 21 Jun 2011 10:45:11 +0000 (UTC)
Received: (qmail 60437 invoked by uid 500); 21 Jun 2011 10:45:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60364 invoked by uid 500); 21 Jun 2011 10:45:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60356 invoked by uid 99); 21 Jun 2011 10:45:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 10:45:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 10:45:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4C7EE4259E0
	for <common-issues@hadoop.apache.org>; Tue, 21 Jun 2011 10:44:48 +0000 (UTC)
Date: Tue, 21 Jun 2011 10:44:48 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1177074488.23980.1308653088310.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2007918179.15449.1305558467414.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7294) FileUtil uses wrong stat command
 for FreeBSD
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052466#comment-13052466 ] 

Hadoop QA commented on HADOOP-7294:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12483262/patch.diff
  against trunk revision 1137724.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/661//console

This message is automatically generated.

> FileUtil uses wrong stat command for FreeBSD
> --------------------------------------------
>
>                 Key: HADOOP-7294
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7294
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0
>         Environment: FreeBSD 8.0-STABLE
>            Reporter: Vitalii Tymchyshyn
>         Attachments: patch.diff
>
>
> I get next exception when try to use append:
> 2011-05-16 17:07:54,648 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.112.0.207:50010, storageID=DS-1047171559-
> 10.112.0.207-50010-1302796304164, infoPort=50075, ipcPort=50020):DataXceiver
> java.io.IOException: Failed to get link count on file /var/data/hdfs/data/current/finalized/subdir26/subdir17/subdir55/blk_-1266943884751786595:
>  message=null; error=stat: illegal option -- c; exit value=1
>         at org.apache.hadoop.fs.FileUtil.createIOException(FileUtil.java:709)
>         at org.apache.hadoop.fs.FileUtil.access$000(FileUtil.java:42)
>         at org.apache.hadoop.fs.FileUtil$HardLink.getLinkCount(FileUtil.java:682)
>         at org.apache.hadoop.hdfs.server.datanode.ReplicaInfo.unlinkBlock(ReplicaInfo.java:215)
>         at org.apache.hadoop.hdfs.server.datanode.FSDataset.append(FSDataset.java:1116)
> It seems that FreeBSD is treated like UNIX and so calls 'stat -c%h', while FreeBSD is much more like Mac (since they have same BSD roots):
> $ stat --help
> stat: illegal option -- -
> usage: stat [-FlLnqrsx] [-f format] [-t timefmt] [file ...]
> $ stat -f%l a_file
> 1

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16591-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 17:08:11 2011
Return-Path: <common-issues-return-16591-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1B92140AD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 21 Jun 2011 17:08:11 +0000 (UTC)
Received: (qmail 92141 invoked by uid 500); 21 Jun 2011 17:08:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92106 invoked by uid 500); 21 Jun 2011 17:08:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92016 invoked by uid 99); 21 Jun 2011 17:08:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 17:08:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 17:08:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B0DC44268D8
	for <common-issues@hadoop.apache.org>; Tue, 21 Jun 2011 17:07:47 +0000 (UTC)
Date: Tue, 21 Jun 2011 17:07:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <28913034.24893.1308676067721.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <557825167.38099.1306198007453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7324) Ganglia plugins for metrics v2
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052662#comment-13052662 ] 

Luke Lu commented on HADOOP-7324:
---------------------------------

bq. I think support for Ganglia 3.1.x is suffice... as it's easy to upgrade ganglia from 3.0.x to 3.1.x.

Thanks. That's good to know.

I'll try to provide a patch for 204+ and a small hadoop-ganglia31-0.20.203.jar, which can be put into the lib directory of the 0.20.203 release (as is, no patch for 203 is needed), hopefully, sometime this week, to appease the wrath of Allen ;)

> Ganglia plugins for metrics v2
> ------------------------------
>
>                 Key: HADOOP-7324
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7324
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.203.0, 0.23.0
>            Reporter: Luke Lu
>            Priority: Blocker
>              Labels: regression
>             Fix For: 0.23.0
>
>
> Although, all metrics in metrics v2 are exposed via the standard JMX mechanisms, most users are using Ganglia to collect metrics.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16592-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 17:26:09 2011
Return-Path: <common-issues-return-16592-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 89D2D4EB5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 21 Jun 2011 17:26:09 +0000 (UTC)
Received: (qmail 31014 invoked by uid 500); 21 Jun 2011 17:26:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30981 invoked by uid 500); 21 Jun 2011 17:26:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30973 invoked by uid 99); 21 Jun 2011 17:26:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 17:26:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 17:26:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3E80C426CF5
	for <common-issues@hadoop.apache.org>; Tue, 21 Jun 2011 17:25:48 +0000 (UTC)
Date: Tue, 21 Jun 2011 17:25:48 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1875258417.25075.1308677148252.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7409) TestTFileByteArrays is failing on
 Hudson
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

TestTFileByteArrays is failing on Hudson
----------------------------------------

                 Key: HADOOP-7409
                 URL: https://issues.apache.org/jira/browse/HADOOP-7409
             Project: Hadoop Common
          Issue Type: Bug
          Components: io
    Affects Versions: 0.23.0
            Reporter: Aaron T. Myers
             Fix For: 0.23.0


This test has failed in the last 4 nightly builds, as seen here: https://builds.apache.org/job/Hadoop-Common-trunk/

I can't reproduce this failure on my machine, running the test either in isolation or as part of the full suite.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16593-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 17:34:31 2011
Return-Path: <common-issues-return-16593-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 813254D19
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 21 Jun 2011 17:34:31 +0000 (UTC)
Received: (qmail 61630 invoked by uid 500); 21 Jun 2011 17:34:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61537 invoked by uid 500); 21 Jun 2011 17:34:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60536 invoked by uid 99); 21 Jun 2011 17:34:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 17:34:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 17:34:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D18EA426446
	for <common-issues@hadoop.apache.org>; Tue, 21 Jun 2011 17:33:47 +0000 (UTC)
Date: Tue, 21 Jun 2011 17:33:47 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <642991329.25148.1308677627854.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7410) Mavenize common RPM/DEB
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Mavenize common RPM/DEB
-----------------------

                 Key: HADOOP-7410
                 URL: https://issues.apache.org/jira/browse/HADOOP-7410
             Project: Hadoop Common
          Issue Type: Task
            Reporter: Alejandro Abdelnur


Mavenize RPM/DEB generation

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16594-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 17:36:11 2011
Return-Path: <common-issues-return-16594-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D8A684DA1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 21 Jun 2011 17:36:11 +0000 (UTC)
Received: (qmail 68771 invoked by uid 500); 21 Jun 2011 17:36:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68707 invoked by uid 500); 21 Jun 2011 17:36:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68688 invoked by uid 99); 21 Jun 2011 17:36:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 17:36:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 17:36:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 82CE842674B
	for <common-issues@hadoop.apache.org>; Tue, 21 Jun 2011 17:35:50 +0000 (UTC)
Date: Tue, 21 Jun 2011 17:35:50 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1635687927.25170.1308677750532.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7411) Convert remaining Ant based build to
 Maven
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Convert remaining Ant based build to Maven
------------------------------------------

                 Key: HADOOP-7411
                 URL: https://issues.apache.org/jira/browse/HADOOP-7411
             Project: Hadoop Common
          Issue Type: Task
            Reporter: Alejandro Abdelnur


Once Mavenization is complete, do a second iteration to remove antrun calls out, this may require writing some Mojos.

The tricky things are native compilation and symlinks handling (for native libs) when creating packaging.


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16595-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 17:38:09 2011
Return-Path: <common-issues-return-16595-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D821A4E3B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 21 Jun 2011 17:38:08 +0000 (UTC)
Received: (qmail 72996 invoked by uid 500); 21 Jun 2011 17:38:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72953 invoked by uid 500); 21 Jun 2011 17:38:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72944 invoked by uid 99); 21 Jun 2011 17:38:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 17:38:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 17:38:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AA47D42694F
	for <common-issues@hadoop.apache.org>; Tue, 21 Jun 2011 17:37:47 +0000 (UTC)
Date: Tue, 21 Jun 2011 17:37:47 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1221804712.25214.1308677867694.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7412) Mavenization Umbrella
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Mavenization Umbrella
---------------------

                 Key: HADOOP-7412
                 URL: https://issues.apache.org/jira/browse/HADOOP-7412
             Project: Hadoop Common
          Issue Type: Task
            Reporter: Alejandro Abdelnur


Umbrella JIRA for all Mavenization JIRAs

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16596-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 17:38:10 2011
Return-Path: <common-issues-return-16596-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AFD054E6B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 21 Jun 2011 17:38:10 +0000 (UTC)
Received: (qmail 73430 invoked by uid 500); 21 Jun 2011 17:38:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73388 invoked by uid 500); 21 Jun 2011 17:38:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73329 invoked by uid 99); 21 Jun 2011 17:38:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 17:38:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 17:38:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D7805426953
	for <common-issues@hadoop.apache.org>; Tue, 21 Jun 2011 17:37:47 +0000 (UTC)
Date: Tue, 21 Jun 2011 17:37:47 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <737643372.25218.1308677867879.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-6671:
---------------------------------------

    Issue Type: Sub-task  (was: Improvement)
        Parent: HADOOP-7412

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-6671-cross-project-HDFS.patch, HADOOP-6671-e.patch, HADOOP-6671-f.patch, HADOOP-6671-g.patch, HADOOP-6671-h.patch, HADOOP-6671-i.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, common-mvn-layout-i.sh, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout-f.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16597-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 17:50:11 2011
Return-Path: <common-issues-return-16597-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 727BF49C0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 21 Jun 2011 17:50:11 +0000 (UTC)
Received: (qmail 4974 invoked by uid 500); 21 Jun 2011 17:50:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4945 invoked by uid 500); 21 Jun 2011 17:50:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4937 invoked by uid 99); 21 Jun 2011 17:50:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 17:50:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 17:50:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F35FA42611A
	for <common-issues@hadoop.apache.org>; Tue, 21 Jun 2011 17:49:47 +0000 (UTC)
Date: Tue, 21 Jun 2011 17:49:47 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1565148226.25330.1308678587993.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1221804712.25214.1308677867694.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7412) Mavenization Umbrella
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052706#comment-13052706 ] 

Alejandro Abdelnur commented on HADOOP-7412:
--------------------------------------------

Sub-Tasks (JIRAS from other projects cannot be linked as sub-tasks):

* HADOOP-6671
* HDFS-2096
* MAPREDUCE-2607
* MAPREDUCE-2608
* HDFS-2097
* HADOOP-7410
* MAPREDUCE-2609
* HDFS-2098
* HADOOP-7411


> Mavenization Umbrella
> ---------------------
>
>                 Key: HADOOP-7412
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7412
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Alejandro Abdelnur
>
> Umbrella JIRA for all Mavenization JIRAs

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16598-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 18:20:12 2011
Return-Path: <common-issues-return-16598-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2738A41D3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 21 Jun 2011 18:20:12 +0000 (UTC)
Received: (qmail 73157 invoked by uid 500); 21 Jun 2011 18:20:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73122 invoked by uid 500); 21 Jun 2011 18:20:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73104 invoked by uid 99); 21 Jun 2011 18:20:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 18:20:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 18:20:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7D0C8426794
	for <common-issues@hadoop.apache.org>; Tue, 21 Jun 2011 18:19:48 +0000 (UTC)
Date: Tue, 21 Jun 2011 18:19:48 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1838991128.25538.1308680388508.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-6671:
---------------------------------------

    Attachment: HADOOP-6671-j.patch

rebasing to trunks HEAD (needed because of new snappy dependency)

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-6671-cross-project-HDFS.patch, HADOOP-6671-e.patch, HADOOP-6671-f.patch, HADOOP-6671-g.patch, HADOOP-6671-h.patch, HADOOP-6671-i.patch, HADOOP-6671-j.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, common-mvn-layout-i.sh, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout-f.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16599-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 18:24:11 2011
Return-Path: <common-issues-return-16599-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B214B425E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 21 Jun 2011 18:24:11 +0000 (UTC)
Received: (qmail 78144 invoked by uid 500); 21 Jun 2011 18:24:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78108 invoked by uid 500); 21 Jun 2011 18:24:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78099 invoked by uid 99); 21 Jun 2011 18:24:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 18:24:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 18:24:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 27A9E426A1B
	for <common-issues@hadoop.apache.org>; Tue, 21 Jun 2011 18:23:48 +0000 (UTC)
Date: Tue, 21 Jun 2011 18:23:48 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1156896389.25602.1308680628159.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052738#comment-13052738 ] 

Hadoop QA commented on HADOOP-6671:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12483320/HADOOP-6671-j.patch
  against trunk revision 1137724.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 57 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/662//console

This message is automatically generated.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-6671-cross-project-HDFS.patch, HADOOP-6671-e.patch, HADOOP-6671-f.patch, HADOOP-6671-g.patch, HADOOP-6671-h.patch, HADOOP-6671-i.patch, HADOOP-6671-j.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, common-mvn-layout-i.sh, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout-f.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16600-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 20:12:10 2011
Return-Path: <common-issues-return-16600-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BACB14C29
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 21 Jun 2011 20:12:10 +0000 (UTC)
Received: (qmail 75687 invoked by uid 500); 21 Jun 2011 20:12:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75600 invoked by uid 500); 21 Jun 2011 20:12:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75562 invoked by uid 99); 21 Jun 2011 20:12:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 20:12:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 20:12:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2F6EB42760A
	for <common-issues@hadoop.apache.org>; Tue, 21 Jun 2011 20:11:49 +0000 (UTC)
Date: Tue, 21 Jun 2011 20:11:49 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1920700328.25899.1308687109191.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7413) Create Jenkins build for Maven patch
 testing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Create Jenkins build for Maven patch testing
--------------------------------------------

                 Key: HADOOP-7413
                 URL: https://issues.apache.org/jira/browse/HADOOP-7413
             Project: Hadoop Common
          Issue Type: Sub-task
          Components: build
            Reporter: Tom White
            Assignee: Tom White


We need an equivalent of https://builds.apache.org/job/PreCommit-HADOOP-Build for the Maven build. Until this is live it would be triggered manually and wouldn't post comments to JIRA.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16601-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 20:18:08 2011
Return-Path: <common-issues-return-16601-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A5B214A25
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 21 Jun 2011 20:18:08 +0000 (UTC)
Received: (qmail 85515 invoked by uid 500); 21 Jun 2011 20:18:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85485 invoked by uid 500); 21 Jun 2011 20:18:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85477 invoked by uid 99); 21 Jun 2011 20:18:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 20:18:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 20:18:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7B9D442788C
	for <common-issues@hadoop.apache.org>; Tue, 21 Jun 2011 20:17:47 +0000 (UTC)
Date: Tue, 21 Jun 2011 20:17:47 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <44335180.25917.1308687467502.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1920700328.25899.1308687109191.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7413) Create Jenkins build for Maven patch
 testing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-7413:
------------------------------

    Attachment: HADOOP-7413.patch

> Create Jenkins build for Maven patch testing
> --------------------------------------------
>
>                 Key: HADOOP-7413
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7413
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: build
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-7413.patch
>
>
> We need an equivalent of https://builds.apache.org/job/PreCommit-HADOOP-Build for the Maven build. Until this is live it would be triggered manually and wouldn't post comments to JIRA.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16602-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 20:22:10 2011
Return-Path: <common-issues-return-16602-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D9A8B4604
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 21 Jun 2011 20:22:10 +0000 (UTC)
Received: (qmail 95171 invoked by uid 500); 21 Jun 2011 20:22:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95077 invoked by uid 500); 21 Jun 2011 20:22:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95069 invoked by uid 99); 21 Jun 2011 20:22:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 20:22:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 20:22:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 967C24279C5
	for <common-issues@hadoop.apache.org>; Tue, 21 Jun 2011 20:21:47 +0000 (UTC)
Date: Tue, 21 Jun 2011 20:21:47 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <459710720.25924.1308687707613.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1920700328.25899.1308687109191.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7413) Create Jenkins build for Maven patch
 testing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-7413:
------------------------------

    Status: Patch Available  (was: Open)

> Create Jenkins build for Maven patch testing
> --------------------------------------------
>
>                 Key: HADOOP-7413
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7413
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: build
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-7413.patch
>
>
> We need an equivalent of https://builds.apache.org/job/PreCommit-HADOOP-Build for the Maven build. Until this is live it would be triggered manually and wouldn't post comments to JIRA.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16603-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 20:36:11 2011
Return-Path: <common-issues-return-16603-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E26DA4F57
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 21 Jun 2011 20:36:10 +0000 (UTC)
Received: (qmail 23022 invoked by uid 500); 21 Jun 2011 20:36:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22978 invoked by uid 500); 21 Jun 2011 20:36:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22970 invoked by uid 99); 21 Jun 2011 20:36:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 20:36:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 20:36:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 680FC427FFE
	for <common-issues@hadoop.apache.org>; Tue, 21 Jun 2011 20:35:47 +0000 (UTC)
Date: Tue, 21 Jun 2011 20:35:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <758041738.25971.1308688547422.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1920700328.25899.1308687109191.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7413) Create Jenkins build for Maven
 patch testing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052812#comment-13052812 ] 

Hadoop QA commented on HADOOP-7413:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12483337/HADOOP-7413.patch
  against trunk revision 1137724.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/663//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/663//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/663//console

This message is automatically generated.

> Create Jenkins build for Maven patch testing
> --------------------------------------------
>
>                 Key: HADOOP-7413
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7413
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: build
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-7413.patch
>
>
> We need an equivalent of https://builds.apache.org/job/PreCommit-HADOOP-Build for the Maven build. Until this is live it would be triggered manually and wouldn't post comments to JIRA.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16604-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 21:16:13 2011
Return-Path: <common-issues-return-16604-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1CEA04D4F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 21 Jun 2011 21:16:13 +0000 (UTC)
Received: (qmail 95928 invoked by uid 500); 21 Jun 2011 21:16:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95888 invoked by uid 500); 21 Jun 2011 21:16:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95880 invoked by uid 99); 21 Jun 2011 21:16:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 21:16:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 21:16:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B53BB4275C2
	for <common-issues@hadoop.apache.org>; Tue, 21 Jun 2011 21:15:47 +0000 (UTC)
Date: Tue, 21 Jun 2011 21:15:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <891494169.26162.1308690947738.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1920700328.25899.1308687109191.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7413) Create Jenkins build for Maven
 patch testing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052838#comment-13052838 ] 

Eli Collins commented on HADOOP-7413:
-------------------------------------

Wrong patch? This just adds a line to CHANGES.txt

> Create Jenkins build for Maven patch testing
> --------------------------------------------
>
>                 Key: HADOOP-7413
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7413
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: build
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-7413.patch
>
>
> We need an equivalent of https://builds.apache.org/job/PreCommit-HADOOP-Build for the Maven build. Until this is live it would be triggered manually and wouldn't post comments to JIRA.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16605-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 21:24:08 2011
Return-Path: <common-issues-return-16605-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C80374626
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 21 Jun 2011 21:24:08 +0000 (UTC)
Received: (qmail 10339 invoked by uid 500); 21 Jun 2011 21:24:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10307 invoked by uid 500); 21 Jun 2011 21:24:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10299 invoked by uid 99); 21 Jun 2011 21:24:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 21:24:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 21:24:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6A2C6427ADC
	for <common-issues@hadoop.apache.org>; Tue, 21 Jun 2011 21:23:47 +0000 (UTC)
Date: Tue, 21 Jun 2011 21:23:47 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1253294286.26213.1308691427431.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1920700328.25899.1308687109191.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7413) Create Jenkins build for Maven
 patch testing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052846#comment-13052846 ] 

Tom White commented on HADOOP-7413:
-----------------------------------

This is just a test patch for the new Jenkins job to try out.

> Create Jenkins build for Maven patch testing
> --------------------------------------------
>
>                 Key: HADOOP-7413
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7413
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: build
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-7413.patch
>
>
> We need an equivalent of https://builds.apache.org/job/PreCommit-HADOOP-Build for the Maven build. Until this is live it would be triggered manually and wouldn't post comments to JIRA.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16606-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 21:54:10 2011
Return-Path: <common-issues-return-16606-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D73DE43F0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 21 Jun 2011 21:54:10 +0000 (UTC)
Received: (qmail 68629 invoked by uid 500); 21 Jun 2011 21:54:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68602 invoked by uid 500); 21 Jun 2011 21:54:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68593 invoked by uid 99); 21 Jun 2011 21:54:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 21:54:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 21:54:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8DD7E427A0B
	for <common-issues@hadoop.apache.org>; Tue, 21 Jun 2011 21:53:49 +0000 (UTC)
Date: Tue, 21 Jun 2011 21:53:49 +0000 (UTC)
From: "Roman Shaposhnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <724598187.26360.1308693229577.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052867#comment-13052867 ] 

Roman Shaposhnik commented on HADOOP-7206:
------------------------------------------

I'd be curious to find out whether the lack of Solaris binaries in the /org/xerial/snappy/native/ bothers anybody:  
http://maven.xerial.org/repository/artifact/org/xerial/snappy/snappy-java/1.0.3-rc3/snappy-java-1.0.3-rc3.jar
   

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: T Jake Luciani
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16607-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 21 22:18:13 2011
Return-Path: <common-issues-return-16607-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5D3D64DD1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 21 Jun 2011 22:18:13 +0000 (UTC)
Received: (qmail 849 invoked by uid 500); 21 Jun 2011 22:18:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 799 invoked by uid 500); 21 Jun 2011 22:18:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 670 invoked by uid 99); 21 Jun 2011 22:18:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 22:18:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jun 2011 22:18:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8A200427647
	for <common-issues@hadoop.apache.org>; Tue, 21 Jun 2011 22:17:49 +0000 (UTC)
Date: Tue, 21 Jun 2011 22:17:49 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <823265396.26495.1308694669562.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052888#comment-13052888 ] 

Allen Wittenauer commented on HADOOP-7206:
------------------------------------------

With the current state of trunk, native compression only works on Linux anyway without reworking the entire libhadoop structure.  See HADOOP-7405.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: T Jake Luciani
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16608-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 22 00:05:10 2011
Return-Path: <common-issues-return-16608-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D160E4D50
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 22 Jun 2011 00:05:10 +0000 (UTC)
Received: (qmail 17267 invoked by uid 500); 22 Jun 2011 00:05:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17217 invoked by uid 500); 22 Jun 2011 00:05:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17109 invoked by uid 99); 22 Jun 2011 00:05:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 00:05:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 00:05:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DD05042738C
	for <common-issues@hadoop.apache.org>; Wed, 22 Jun 2011 00:04:47 +0000 (UTC)
Date: Wed, 22 Jun 2011 00:04:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1977530863.26762.1308701087902.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052955#comment-13052955 ] 

Todd Lipcon commented on HADOOP-7206:
-------------------------------------

Sorry, I stopped paying attention to this for a while... I have some concerns about the way this ended up:

We're now pulling in a jar which autoexpands its .so dependency into /tmp and then loads native libraries that way. That's (a) messy, (b) potentially insecure without workarounds to change /tmp to some other dir, and (c) inconsistent with how native libraries work. These are the same arguments Alejandro made above

This maven artifact that we now depend on is something that isn't easy to rebuild, and it's not even clear how it gets build. For example, which glibc version is it linked against? Which OSX version is the included dylib built on? Seems a little scary as a dependency

It seems the motivation to switch from the hadoop-snappy approach to the java-snappy approach was that the former approach depended on having snappy.so available at runtime, which isn't always the case. I would propose the following:
- at build time, you can choose (a) disable snappy, (b) enable snappy and dynamically link our JNI shims against snappy.so, or (c) enable snappy and statically link against snappy.so
- those who don't care about snappy choose (a)
- those who care about snappy and plan to deploy on systems where libsnappy.so is deployed system-wide (eg fedora or most recent ubuntu) can choose (b) to pick up the snappy lib off the system
- those who care about snappy and plan to deploy elsewhere choose (c), and just make sure that snappy is available at compile time

Then the hadoopsnappy.so can be included in lib/native just like our other native dependencies without the unjar-to-tmp hackery.

Does this idea address everyone's goals?

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: T Jake Luciani
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16609-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 22 00:33:10 2011
Return-Path: <common-issues-return-16609-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CA5754151
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 22 Jun 2011 00:33:10 +0000 (UTC)
Received: (qmail 55278 invoked by uid 500); 22 Jun 2011 00:33:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55252 invoked by uid 500); 22 Jun 2011 00:33:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55244 invoked by uid 99); 22 Jun 2011 00:33:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 00:33:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 00:33:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8ED19427C0D
	for <common-issues@hadoop.apache.org>; Wed, 22 Jun 2011 00:32:47 +0000 (UTC)
Date: Wed, 22 Jun 2011 00:32:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1714646631.26863.1308702767581.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052971#comment-13052971 ] 

Eli Collins commented on HADOOP-7206:
-------------------------------------

I think so. Seems like we could get away with (a) and (b) for now. Ie those who care about snappy and plan to deploy elsewhere could reasonably use dynamic linking as well.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: T Jake Luciani
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16610-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 22 01:21:09 2011
Return-Path: <common-issues-return-16610-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6CA7E4504
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 22 Jun 2011 01:21:09 +0000 (UTC)
Received: (qmail 89910 invoked by uid 500); 22 Jun 2011 01:21:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89867 invoked by uid 500); 22 Jun 2011 01:21:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89858 invoked by uid 99); 22 Jun 2011 01:21:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 01:21:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 01:21:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2DA754279E9
	for <common-issues@hadoop.apache.org>; Wed, 22 Jun 2011 01:20:48 +0000 (UTC)
Date: Wed, 22 Jun 2011 01:20:48 +0000 (UTC)
From: "XieXianshan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2010151044.26991.1308705648183.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <740424940.5878.1308116867420.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7395) The infomation returned by the
 wrong usage of the command "job -counter<job-id> <group-name>
 <counter-name> "is not appropriate
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052984#comment-13052984 ] 

XieXianshan commented on HADOOP-7395:
-------------------------------------

-1 
This issue should be created in project Map/Reduce.


> The infomation returned by the wrong usage of the command "job -counter<job-id> <group-name> <counter-name> "is not appropriate
> -------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7395
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7395
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Yan Jinshuang
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: Hadoop_7395
>
>
> When use the command "job -counter<job-id> <group-name> <counter-name> " with wrong group-name or wrong counter-name(with correct job-id), the result is always 0. It is better to show the user the detail, like "illegal group-name", "illegal counter-name", etc. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16611-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 22 01:23:08 2011
Return-Path: <common-issues-return-16611-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7A019453B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 22 Jun 2011 01:23:08 +0000 (UTC)
Received: (qmail 91389 invoked by uid 500); 22 Jun 2011 01:23:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91347 invoked by uid 500); 22 Jun 2011 01:23:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91339 invoked by uid 99); 22 Jun 2011 01:23:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 01:23:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 01:23:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 62F41427AA1
	for <common-issues@hadoop.apache.org>; Wed, 22 Jun 2011 01:22:47 +0000 (UTC)
Date: Wed, 22 Jun 2011 01:22:47 +0000 (UTC)
From: "XieXianshan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <397423626.26993.1308705767402.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <740424940.5878.1308116867420.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7395) The infomation returned by the wrong
 usage of the command "job -counter<job-id> <group-name> <counter-name> "is
 not appropriate
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

XieXianshan updated HADOOP-7395:
--------------------------------

    Resolution: Invalid
        Status: Resolved  (was: Patch Available)

> The infomation returned by the wrong usage of the command "job -counter<job-id> <group-name> <counter-name> "is not appropriate
> -------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7395
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7395
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Yan Jinshuang
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: Hadoop_7395
>
>
> When use the command "job -counter<job-id> <group-name> <counter-name> " with wrong group-name or wrong counter-name(with correct job-id), the result is always 0. It is better to show the user the detail, like "illegal group-name", "illegal counter-name", etc. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16612-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 22 03:23:21 2011
Return-Path: <common-issues-return-16612-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AAC956AEE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 22 Jun 2011 03:23:21 +0000 (UTC)
Received: (qmail 99293 invoked by uid 500); 22 Jun 2011 03:23:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98507 invoked by uid 500); 22 Jun 2011 03:23:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98492 invoked by uid 99); 22 Jun 2011 03:23:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 03:23:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 03:23:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8ED42427172
	for <common-issues@hadoop.apache.org>; Wed, 22 Jun 2011 03:22:47 +0000 (UTC)
Date: Wed, 22 Jun 2011 03:22:47 +0000 (UTC)
From: "Taro L. Saito (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1623316998.27187.1308712967581.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053019#comment-13053019 ] 

Taro L. Saito commented on HADOOP-7206:
---------------------------------------

Let me clarify some differences between Issay's hadoop-snappy and my snappy-java:

hadoop-snappy
 * Uses libsnappy.so (available in recent Linux distributions) and libhadoopsnappy.so (JNI code compiled for the target platform)

snappy-java
 * Uses libsnappyjava.so (mixing up the original snappy and JNI code), or snappyjava.dll (for Windows), libsnappyjava.jnilib (for Mac OS X)
 * It copies one of the native library to the directory specified in org.xerial.snappy.tempdir or java.io.tempdir system property.
 * If the dependencies to the glibc (in Linux GLIBC2.3 or higher is required for now) and dylib (in Mac OS X) cause some problems, you can re-compile snappy-java's native library only for your own platform (with make clean-native native). No need to care about building native libraries for the other platforms if you never use them. 

The same thing between hadoop-snappy and snappy-java is:
 * Both approaches need to compile the native code (libhadoopsnappy.so or libsnappyjava.so) somewhere. My snappy-java simply provides pre-compiled libsnappyjava.so for various platforms.

One of the design goals of snappy-java is to avoid troubles in linking against native libraries (e.g., libsnappy.so), such as crashes due to libstdc++ compatibility, missing libraries, etc. But as Alejandro suggested in my discussion group, using separate libsnappy.so and libsnapyjava.so is technically possible even in snappy-java:
 * First, tries to load pre-installed libsnappy.so and libsnappyjava.so (the version not containing libsnappy.so)  
 * If not found, extract these libraries embedded in the JAR to somewhere.
 * Load both native libraries.  

I am not sure supporting such loading mechanism is a right way to go.


> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: T Jake Luciani
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16613-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 22 05:56:14 2011
Return-Path: <common-issues-return-16613-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9CFC365C5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 22 Jun 2011 05:56:14 +0000 (UTC)
Received: (qmail 93121 invoked by uid 500); 22 Jun 2011 05:56:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92867 invoked by uid 500); 22 Jun 2011 05:56:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92842 invoked by uid 99); 22 Jun 2011 05:56:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 05:56:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 05:56:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 68BDB428FF1
	for <common-issues@hadoop.apache.org>; Wed, 22 Jun 2011 05:55:47 +0000 (UTC)
Date: Wed, 22 Jun 2011 05:55:47 +0000 (UTC)
From: "Benjamin Reed (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1443063883.27446.1308722147425.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7414) path to setWar() in HttpServer is
 not setup properly for resources from Jar URLs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

path to setWar() in HttpServer is not setup properly for resources from Jar URLs
--------------------------------------------------------------------------------

                 Key: HADOOP-7414
                 URL: https://issues.apache.org/jira/browse/HADOOP-7414
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Benjamin Reed
         Attachments: HADOOP-7414.patch

the path in setWar() needs to end in a '/' for jetty to properly process resources from Jar URLs. by adding a '/' at the end the webapps can run out of a Jar file instead of unzipping into a file system.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16614-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 22 05:56:19 2011
Return-Path: <common-issues-return-16614-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F25E365F3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 22 Jun 2011 05:56:19 +0000 (UTC)
Received: (qmail 94647 invoked by uid 500); 22 Jun 2011 05:56:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93117 invoked by uid 500); 22 Jun 2011 05:56:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92869 invoked by uid 99); 22 Jun 2011 05:56:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 05:56:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 05:56:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 77A5B428FF3
	for <common-issues@hadoop.apache.org>; Wed, 22 Jun 2011 05:55:47 +0000 (UTC)
Date: Wed, 22 Jun 2011 05:55:47 +0000 (UTC)
From: "Benjamin Reed (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1043495529.27448.1308722147487.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1443063883.27446.1308722147425.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7414) path to setWar() in HttpServer is
 not setup properly for resources from Jar URLs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Benjamin Reed updated HADOOP-7414:
----------------------------------

    Attachment: HADOOP-7414.patch

> path to setWar() in HttpServer is not setup properly for resources from Jar URLs
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-7414
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7414
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Benjamin Reed
>         Attachments: HADOOP-7414.patch
>
>
> the path in setWar() needs to end in a '/' for jetty to properly process resources from Jar URLs. by adding a '/' at the end the webapps can run out of a Jar file instead of unzipping into a file system.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16615-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 22 06:02:22 2011
Return-Path: <common-issues-return-16615-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E2F7F62DE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 22 Jun 2011 06:02:22 +0000 (UTC)
Received: (qmail 3070 invoked by uid 500); 22 Jun 2011 06:02:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2594 invoked by uid 500); 22 Jun 2011 06:02:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2539 invoked by uid 99); 22 Jun 2011 06:02:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 06:02:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 06:02:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 962BF427396
	for <common-issues@hadoop.apache.org>; Wed, 22 Jun 2011 06:01:49 +0000 (UTC)
Date: Wed, 22 Jun 2011 06:01:49 +0000 (UTC)
From: "Benjamin Reed (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1442157395.27476.1308722509611.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1443063883.27446.1308722147425.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7414) path to setWar() in HttpServer is
 not setup properly for resources from Jar URLs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Benjamin Reed updated HADOOP-7414:
----------------------------------

    Status: Patch Available  (was: Open)

> path to setWar() in HttpServer is not setup properly for resources from Jar URLs
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-7414
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7414
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Benjamin Reed
>         Attachments: HADOOP-7414.patch
>
>
> the path in setWar() needs to end in a '/' for jetty to properly process resources from Jar URLs. by adding a '/' at the end the webapps can run out of a Jar file instead of unzipping into a file system.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16616-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 22 07:35:22 2011
Return-Path: <common-issues-return-16616-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 933626F99
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 22 Jun 2011 07:35:22 +0000 (UTC)
Received: (qmail 29154 invoked by uid 500); 22 Jun 2011 07:35:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28162 invoked by uid 500); 22 Jun 2011 07:35:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28126 invoked by uid 99); 22 Jun 2011 07:35:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 07:35:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 07:35:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4F4494289CF
	for <common-issues@hadoop.apache.org>; Wed, 22 Jun 2011 07:34:48 +0000 (UTC)
Date: Wed, 22 Jun 2011 07:34:48 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2114464655.27701.1308728088321.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1443063883.27446.1308722147425.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7414) path to setWar() in HttpServer is
 not setup properly for resources from Jar URLs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053098#comment-13053098 ] 

Hadoop QA commented on HADOOP-7414:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12483406/HADOOP-7414.patch
  against trunk revision 1137724.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/664//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/664//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/664//console

This message is automatically generated.

> path to setWar() in HttpServer is not setup properly for resources from Jar URLs
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-7414
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7414
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Benjamin Reed
>         Attachments: HADOOP-7414.patch
>
>
> the path in setWar() needs to end in a '/' for jetty to properly process resources from Jar URLs. by adding a '/' at the end the webapps can run out of a Jar file instead of unzipping into a file system.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16617-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 22 12:13:14 2011
Return-Path: <common-issues-return-16617-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0833B6F0C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 22 Jun 2011 12:13:14 +0000 (UTC)
Received: (qmail 49052 invoked by uid 500); 22 Jun 2011 12:13:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48202 invoked by uid 500); 22 Jun 2011 12:13:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47822 invoked by uid 99); 22 Jun 2011 12:13:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 12:13:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 12:13:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7C708428366
	for <common-issues@hadoop.apache.org>; Wed, 22 Jun 2011 12:12:47 +0000 (UTC)
Date: Wed, 22 Jun 2011 12:12:47 +0000 (UTC)
From: "Rajesh Kumar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <992091594.29217.1308744767506.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7415) Isolation Runner
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Isolation Runner
----------------

                 Key: HADOOP-7415
                 URL: https://issues.apache.org/jira/browse/HADOOP-7415
             Project: Hadoop Common
          Issue Type: Test
    Affects Versions: 0.20.0
         Environment: Using ubuntu machine
            Reporter: Rajesh Kumar


I am running Isolation Runner. I followed the procedure given in the book Hadoop The Definitive Guide.
But it is giving :
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException.



--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16618-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 22 17:13:09 2011
Return-Path: <common-issues-return-16618-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1EC1A61E1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 22 Jun 2011 17:13:09 +0000 (UTC)
Received: (qmail 3068 invoked by uid 500); 22 Jun 2011 17:13:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2919 invoked by uid 500); 22 Jun 2011 17:13:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2900 invoked by uid 99); 22 Jun 2011 17:13:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 17:13:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 17:13:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 889D1429BC8
	for <common-issues@hadoop.apache.org>; Wed, 22 Jun 2011 17:12:47 +0000 (UTC)
Date: Wed, 22 Jun 2011 17:12:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <748340618.29869.1308762767556.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-310) Additional constructor requested in
 BytesWritable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053344#comment-13053344 ] 

Aaron T. Myers commented on HADOOP-310:
---------------------------------------

Hey Brock, thanks a lot for reviving this issue (after 5 years!) I have a few small concerns about this patch:

# The constructor which only takes a {{byte[]}} can now be implemented in terms of this new constructor you've introduced.
# We now have {{set(byte[] bytes, int length)}} and {{set(byte[] bytes, int offset, int length)}}, which differ not only in signature, but in their semantics with respect to copying of the input data. This seems like it might lead to confusion. Perhaps we should name the new method something like {{setDirect(...)}}? Or perhaps we should amend the existing {{set(...)}} method to not do a copy? The latter is obviously a trickier and backward-incompatible change,  but probably clearer.

> Additional constructor requested in BytesWritable
> -------------------------------------------------
>
>                 Key: HADOOP-310
>                 URL: https://issues.apache.org/jira/browse/HADOOP-310
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: p sutter
>            Assignee: Owen O'Malley
>            Priority: Minor
>         Attachments: bytes-writable-zero-copy-interface-0.patch, bytes-writable-zero-copy-interface-1.patch
>
>
> It would be grand if BytesWritable.java had an additional constructor as below. This allows me to use the BytesWritable class without doing a buffer copy, since we have a less-than-fully-utilized byte array holding our key.
> Thanks!
>  
>  /**
>    * Create a BytesWritable using the byte array as the initial value.
>    * @param bytes This array becomes the backing storage for the object.
>    */
>   public BytesWritable(byte[] bytes, int size) {
>     this.bytes = bytes;
>     this.size = size;
>   }
>   

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16619-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 22 23:09:14 2011
Return-Path: <common-issues-return-16619-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7D6BD6E17
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 22 Jun 2011 23:09:14 +0000 (UTC)
Received: (qmail 64402 invoked by uid 500); 22 Jun 2011 23:09:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64375 invoked by uid 500); 22 Jun 2011 23:09:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64348 invoked by uid 99); 22 Jun 2011 23:09:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 23:09:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 23:09:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D17034296EB
	for <common-issues@hadoop.apache.org>; Wed, 22 Jun 2011 23:08:50 +0000 (UTC)
Date: Wed, 22 Jun 2011 23:08:50 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <330737614.30743.1308784130854.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7416) Allow test-patch to work with cross
 sub-project changes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Allow test-patch to work with cross sub-project changes
-------------------------------------------------------

                 Key: HADOOP-7416
                 URL: https://issues.apache.org/jira/browse/HADOOP-7416
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Aaron T. Myers


Now that the sub-projects are sub-directories in the same repo, we can make {{test-patch.sh}} not barf on cross-project patches. We would need to make {{smart-apply-patch.sh}} not fail hard when it detects this, and probably make {{test-patch.sh}} run all the validation checks against all affected sub-projects.

This was inspired by HDFS-1900, which changes a config in Common that's referenced in a bunch of places in HDFS.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16620-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 22 23:35:11 2011
Return-Path: <common-issues-return-16620-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9998A6300
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 22 Jun 2011 23:35:11 +0000 (UTC)
Received: (qmail 24874 invoked by uid 500); 22 Jun 2011 23:35:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24824 invoked by uid 500); 22 Jun 2011 23:35:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24810 invoked by uid 99); 22 Jun 2011 23:35:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 23:35:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 23:35:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 34B9D4291A9
	for <common-issues@hadoop.apache.org>; Wed, 22 Jun 2011 23:34:48 +0000 (UTC)
Date: Wed, 22 Jun 2011 23:34:48 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2104918972.30802.1308785688212.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1443063883.27446.1308722147425.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7414) path to setWar() in HttpServer is
 not setup properly for resources from Jar URLs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053540#comment-13053540 ] 

Aaron T. Myers commented on HADOOP-7414:
----------------------------------------

Hey Ben, does Jetty mind if there are multiple slashes at the end of the path? That's the only potential problem I can imagine with this patch.

Also, is there a way to write a unit test for this?

> path to setWar() in HttpServer is not setup properly for resources from Jar URLs
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-7414
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7414
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Benjamin Reed
>         Attachments: HADOOP-7414.patch
>
>
> the path in setWar() needs to end in a '/' for jetty to properly process resources from Jar URLs. by adding a '/' at the end the webapps can run out of a Jar file instead of unzipping into a file system.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16621-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 22 23:47:09 2011
Return-Path: <common-issues-return-16621-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 85F0A623A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 22 Jun 2011 23:47:09 +0000 (UTC)
Received: (qmail 50463 invoked by uid 500); 22 Jun 2011 23:47:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50431 invoked by uid 500); 22 Jun 2011 23:47:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50423 invoked by uid 99); 22 Jun 2011 23:47:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 23:47:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 23:47:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BC1A6429643
	for <common-issues@hadoop.apache.org>; Wed, 22 Jun 2011 23:46:47 +0000 (UTC)
Date: Wed, 22 Jun 2011 23:46:47 +0000 (UTC)
From: "Jongwook Woo (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1045246622.30828.1308786407767.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <639440876.5515.1308103967395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7394) Chinese documentation can't be
 built with Forrest 0.9
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053547#comment-13053547 ] 

Jongwook Woo commented on HADOOP-7394:
--------------------------------------

It is because Forrest 0.9 does not have 'fop-0.20.5.jar' that has the class 'org/apache/fop/messaging/MessageHandler'

You may check them out, for example, in my Forrest 0.9:

jongwook@localhost:~/apache/apache-forrest-0.9/lib/core$ ls *fop*
cocoon-fop-block.jar

In my Forrest 0.8:
jongwook@localhost:~/apache/apache-forrest-0.8/lib/core$ ls *fop*
cocoon-fop-block-2.2.0-dev.jar  fop-0.20.5.jar  fop-0.20.5.jar.license.txt


> Chinese documentation can't be built with Forrest 0.9
> -----------------------------------------------------
>
>                 Key: HADOOP-7394
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7394
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Priority: Minor
>
> Running {{ant cn-docs}} with Forrest 0.8 will work. Running the same thing with Forrest 0.9 prints the following error:
> {noformat}
> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/fop/messaging/MessageHandler
> 	at org.apache.cocoon.serialization.FOPSerializer.configure(FOPSerializer.java:122)
> 	at org.apache.avalon.framework.container.ContainerUtil.configure(ContainerUtil.java:201)
> 	at org.apache.avalon.excalibur.component.DefaultComponentFactory.newInstance(DefaultComponentFactory.java:289)
> 	at org.apache.avalon.excalibur.pool.InstrumentedResourceLimitingPool.newPoolable(InstrumentedResourceLimitingPool.java:655)
> 	at org.apache.avalon.excalibur.pool.InstrumentedResourceLimitingPool.get(InstrumentedResourceLimitingPool.java:371)
> 	at org.apache.avalon.excalibur.component.PoolableComponentHandler.doGet(PoolableComponentHandler.java:198)
> 	at org.apache.avalon.excalibur.component.ComponentHandler.get(ComponentHandler.java:381)
> 	at org.apache.avalon.excalibur.component.ExcaliburComponentSelector.select(ExcaliburComponentSelector.java:215)
> 	at org.apache.cocoon.components.ExtendedComponentSelector.select(ExtendedComponentSelector.java:268)
> 	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.setSerializer(AbstractProcessingPipeline.java:311)
> 	at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.setSerializer(AbstractCachingProcessingPipeline.java:171)
> 	at org.apache.cocoon.components.treeprocessor.sitemap.SerializeNode.invoke(SerializeNode.java:120)
> 	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)
> 	at org.apache.cocoon.components.treeprocessor.sitemap.SelectNode.invoke(SelectNode.java:103)
> 	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:47)
> 	at org.apache.cocoon.components.treeprocessor.sitemap.PreparableMatchNode.invoke(PreparableMatchNode.java:131)
> 	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)
> 	at org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:143)
> 	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)
> 	at org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:93)
> 	at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:235)
> 	at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:177)
> 	at org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:254)
> 	at org.apache.cocoon.components.treeprocessor.sitemap.MountNode.invoke(MountNode.java:118)
> 	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)
> 	at org.apache.cocoon.components.treeprocessor.sitemap.SelectNode.invoke(SelectNode.java:98)
> 	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)
> 	at org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:143)
> 	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)
> 	at org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:93)
> 	at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:235)
> 	at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:177)
> 	at org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:254)
> 	at org.apache.cocoon.Cocoon.process(Cocoon.java:699)
> 	at org.apache.cocoon.bean.CocoonWrapper.getPage(CocoonWrapper.java:514)
> 	at org.apache.cocoon.bean.CocoonBean.processTarget(CocoonBean.java:499)
> 	at org.apache.cocoon.bean.CocoonBean.process(CocoonBean.java:356)
> 	at org.apache.cocoon.Main.main(Main.java:321)
> Caused by: java.lang.ClassNotFoundException: org.apache.fop.messaging.MessageHandler
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> 	... 38 more
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16622-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 22 23:53:11 2011
Return-Path: <common-issues-return-16622-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ABB756D9E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 22 Jun 2011 23:53:11 +0000 (UTC)
Received: (qmail 57239 invoked by uid 500); 22 Jun 2011 23:53:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57208 invoked by uid 500); 22 Jun 2011 23:53:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57176 invoked by uid 99); 22 Jun 2011 23:53:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 23:53:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Jun 2011 23:53:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6CBE84297F2
	for <common-issues@hadoop.apache.org>; Wed, 22 Jun 2011 23:52:47 +0000 (UTC)
Date: Wed, 22 Jun 2011 23:52:47 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <407429641.30834.1308786767442.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7417) Hadoop Management System (Umbrella)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Hadoop Management System (Umbrella)
-----------------------------------

                 Key: HADOOP-7417
                 URL: https://issues.apache.org/jira/browse/HADOOP-7417
             Project: Hadoop Common
          Issue Type: New Feature
         Environment: Java 6, Linux
            Reporter: Eric Yang
            Assignee: Eric Yang


The primary goal of Hadoop Management System is to build a component around management and deployment of Hadoop related projects. This includes software installation, configuration, application orchestration, deployment automation and monitoring Hadoop.

Prototype demo source code can be obtained from:

http://github.com/macroadster/hms

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16623-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 00:31:09 2011
Return-Path: <common-issues-return-16623-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 70F6668C7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 00:31:09 +0000 (UTC)
Received: (qmail 96491 invoked by uid 500); 23 Jun 2011 00:31:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96437 invoked by uid 500); 23 Jun 2011 00:31:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96427 invoked by uid 99); 23 Jun 2011 00:31:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 00:31:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 00:31:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E41DC4276CF
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 00:30:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 00:30:47 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1445041033.30924.1308789047931.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur reassigned HADOOP-7206:
------------------------------------------

    Assignee: Alejandro Abdelnur  (was: T Jake Luciani)

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16624-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 00:31:12 2011
Return-Path: <common-issues-return-16624-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 33CAE6927
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 00:31:12 +0000 (UTC)
Received: (qmail 97101 invoked by uid 500); 23 Jun 2011 00:31:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97015 invoked by uid 500); 23 Jun 2011 00:31:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96925 invoked by uid 99); 23 Jun 2011 00:31:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 00:31:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 00:31:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8FB8E427729
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 00:30:50 +0000 (UTC)
Date: Thu, 23 Jun 2011 00:30:50 +0000 (UTC)
From: "Andrew Look (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1931919786.30989.1308789050585.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7418) support for multiple slashes in the
 path separator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Look updated HADOOP-7418:
--------------------------------

    Status: Open  (was: Patch Available)

> support for multiple slashes in the path separator
> --------------------------------------------------
>
>                 Key: HADOOP-7418
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7418
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>         Environment: Linux running JDK 1.6
>            Reporter: Sudharsan Sampath
>            Assignee: Andrew Look
>            Priority: Minor
>              Labels: newbie
>             Fix For: 0.23.0
>
>         Attachments: HDFS-1460.txt, HDFS-1460.txt
>
>
> the parsing of the input path string to identify the uri authority conflicts with the file system paths. For instance the following is a valid path in both the linux file system and the hdfs.
> //user/directory1//directory2.
> While this works perfectly fine in the command line for manipulating hdfs, the same fails when specified as the input path for a mapper class with the following expcetion.
> Exception in thread "main" java.net.UnknownHostException: unknown host: user
>         at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:195)
> as the org.apache.hadoop.fs.Path class assumes the string that follows the '//' to be an uri authority

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16625-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 00:31:13 2011
Return-Path: <common-issues-return-16625-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C2BD2695E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 00:31:13 +0000 (UTC)
Received: (qmail 97427 invoked by uid 500); 23 Jun 2011 00:31:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97402 invoked by uid 500); 23 Jun 2011 00:31:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97391 invoked by uid 99); 23 Jun 2011 00:31:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 00:31:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 00:31:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2B1AD427723
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 00:30:50 +0000 (UTC)
Date: Thu, 23 Jun 2011 00:30:50 +0000 (UTC)
From: "Andrew Look (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1307013660.30983.1308789050171.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Moved] (HADOOP-7418) support for multiple slashes in the
 path separator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Look moved HDFS-1460 to HADOOP-7418:
-------------------------------------------

          Component/s:     (was: hdfs client)
        Fix Version/s:     (was: 0.23.0)
                       0.23.0
    Affects Version/s:     (was: 0.23.0)
                       0.23.0
                  Key: HADOOP-7418  (was: HDFS-1460)
              Project: Hadoop Common  (was: Hadoop HDFS)

> support for multiple slashes in the path separator
> --------------------------------------------------
>
>                 Key: HADOOP-7418
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7418
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>         Environment: Linux running JDK 1.6
>            Reporter: Sudharsan Sampath
>            Assignee: Andrew Look
>            Priority: Minor
>              Labels: newbie
>             Fix For: 0.23.0
>
>         Attachments: HDFS-1460.txt, HDFS-1460.txt
>
>
> the parsing of the input path string to identify the uri authority conflicts with the file system paths. For instance the following is a valid path in both the linux file system and the hdfs.
> //user/directory1//directory2.
> While this works perfectly fine in the command line for manipulating hdfs, the same fails when specified as the input path for a mapper class with the following expcetion.
> Exception in thread "main" java.net.UnknownHostException: unknown host: user
>         at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:195)
> as the org.apache.hadoop.fs.Path class assumes the string that follows the '//' to be an uri authority

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16626-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 00:31:14 2011
Return-Path: <common-issues-return-16626-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6EE8B698B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 00:31:14 +0000 (UTC)
Received: (qmail 97882 invoked by uid 500); 23 Jun 2011 00:31:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97853 invoked by uid 500); 23 Jun 2011 00:31:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97845 invoked by uid 99); 23 Jun 2011 00:31:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 00:31:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 00:31:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A47BE42772D
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 00:30:50 +0000 (UTC)
Date: Thu, 23 Jun 2011 00:30:50 +0000 (UTC)
From: "Andrew Look (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <763408194.30992.1308789050670.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7418) support for multiple slashes in the
 path separator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Look updated HADOOP-7418:
--------------------------------

    Status: Patch Available  (was: Open)

> support for multiple slashes in the path separator
> --------------------------------------------------
>
>                 Key: HADOOP-7418
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7418
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>         Environment: Linux running JDK 1.6
>            Reporter: Sudharsan Sampath
>            Assignee: Andrew Look
>            Priority: Minor
>              Labels: newbie
>             Fix For: 0.23.0
>
>         Attachments: HDFS-1460.txt, HDFS-1460.txt
>
>
> the parsing of the input path string to identify the uri authority conflicts with the file system paths. For instance the following is a valid path in both the linux file system and the hdfs.
> //user/directory1//directory2.
> While this works perfectly fine in the command line for manipulating hdfs, the same fails when specified as the input path for a mapper class with the following expcetion.
> Exception in thread "main" java.net.UnknownHostException: unknown host: user
>         at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:195)
> as the org.apache.hadoop.fs.Path class assumes the string that follows the '//' to be an uri authority

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16627-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 00:45:09 2011
Return-Path: <common-issues-return-16627-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 291A26EC3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 00:45:09 +0000 (UTC)
Received: (qmail 17444 invoked by uid 500); 23 Jun 2011 00:45:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17368 invoked by uid 500); 23 Jun 2011 00:45:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17223 invoked by uid 99); 23 Jun 2011 00:45:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 00:45:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 00:45:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 95EF9427D17
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 00:44:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 00:44:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1174146062.31024.1308789887610.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7418) support for multiple slashes in
 the path separator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053570#comment-13053570 ] 

Hadoop QA commented on HADOOP-7418:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12483526/HDFS-1460.txt
  against trunk revision 1137724.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/665//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/665//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/665//console

This message is automatically generated.

> support for multiple slashes in the path separator
> --------------------------------------------------
>
>                 Key: HADOOP-7418
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7418
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>         Environment: Linux running JDK 1.6
>            Reporter: Sudharsan Sampath
>            Assignee: Andrew Look
>            Priority: Minor
>              Labels: newbie
>             Fix For: 0.23.0
>
>         Attachments: HDFS-1460.txt, HDFS-1460.txt
>
>
> the parsing of the input path string to identify the uri authority conflicts with the file system paths. For instance the following is a valid path in both the linux file system and the hdfs.
> //user/directory1//directory2.
> While this works perfectly fine in the command line for manipulating hdfs, the same fails when specified as the input path for a mapper class with the following expcetion.
> Exception in thread "main" java.net.UnknownHostException: unknown host: user
>         at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:195)
> as the org.apache.hadoop.fs.Path class assumes the string that follows the '//' to be an uri authority

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16628-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 01:03:11 2011
Return-Path: <common-issues-return-16628-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 55930680E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 01:03:11 +0000 (UTC)
Received: (qmail 36625 invoked by uid 500); 23 Jun 2011 01:03:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36591 invoked by uid 500); 23 Jun 2011 01:03:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36583 invoked by uid 99); 23 Jun 2011 01:03:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 01:03:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 01:03:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A307442A26A
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 01:02:49 +0000 (UTC)
Date: Thu, 23 Jun 2011 01:02:49 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1965117194.31128.1308790969664.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Reopened] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur reopened HADOOP-7206:
----------------------------------------


After mulling over this issue a bit more, reading a few times Todd's comment and asking around to folks that deal with nativelibs I'm having second thoughts about the committed patch based on snappy-java.

The snappy-java approach is tempting because it 'just works' (without having to install snappy SO in your system). However, it has a serious drawback; the native code is not built in target OS, only on the same architecture. Because of this the build is not easy reproducible as there is not knowledge of the OS used to build it. In addition, this can lead to not avail dependencies in the running OS.

The hadoop-snappy approach has the drawback that it requires an additional step (to install snappy SO in the platform), but as benefits it takes care of the drawbacks of the snappy-java approach; the native code is built in the target OS. Thus, resulting on easy reproducible builds. Furthermore the drawback is transient, until snappy is avail the different OSes by default or OS driven updates.

A secondary issue is that snappy-java nativelib statically links snappy. As snappy SO makes it to standard Linux distributions, snappy-java will use a private copy of it instead using the one installed in the OS. On the other hand, hadoop-snappy SO dynamically links snappy SO, when snappy SO is available in the OS, it could be consumed directly from it. (this could be taken care by snappy-java if it changes to dynamically link snappy SO).

Because of this I'd like to revert the snappy-java based patch and go for Issay's hadoop-snappy patch.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16629-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 01:09:11 2011
Return-Path: <common-issues-return-16629-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EB2326EDE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 01:09:10 +0000 (UTC)
Received: (qmail 39544 invoked by uid 500); 23 Jun 2011 01:09:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39508 invoked by uid 500); 23 Jun 2011 01:09:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39497 invoked by uid 99); 23 Jun 2011 01:09:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 01:09:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 01:09:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7083142A463
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 01:08:49 +0000 (UTC)
Date: Thu, 23 Jun 2011 01:08:49 +0000 (UTC)
From: "T Jake Luciani (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2114160641.31189.1308791329457.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053579#comment-13053579 ] 

T Jake Luciani commented on HADOOP-7206:
----------------------------------------

bq. However, it has a serious drawback; the native code is not built in target OS, only on the same architecture. Because of this the build is not easy reproducible as there is not knowledge of the OS used to build it. In addition, this can lead to not avail dependencies in the running OS.

Why not let snappy-java implement the (check for libsnappy.so on the system before using the bundled one) as Taro mentioned?

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16630-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 01:23:09 2011
Return-Path: <common-issues-return-16630-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 03FB96713
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 01:23:08 +0000 (UTC)
Received: (qmail 56710 invoked by uid 500); 23 Jun 2011 01:23:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56680 invoked by uid 500); 23 Jun 2011 01:23:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56667 invoked by uid 99); 23 Jun 2011 01:23:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 01:23:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 01:23:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8463842A82B
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 01:22:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 01:22:47 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <227471532.31221.1308792167539.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053586#comment-13053586 ] 

Alejandro Abdelnur commented on HADOOP-7206:
--------------------------------------------

@Jake 

snappy-java does not load vanilla snappy SO, it loads snappy-java SO which is a combination of snappy SO + the JNI bindings for Java. 

Assuming that snappy-java splits the snappy & snappy-java native code in 2 SOs as I suggested before in the snappy-java alias (and Taro is not convinced of that approach, see the end of his comment) ... snappy SO would be loaded from the system but snappy-java SO would still be loaded from the JAR. This would mean that not avail dependencies could still happen for the snappy-java SO. And, if you want to have a snappy-java SO build for your OS, you'd have build it yourself but still consume the JAR that comes an external dependency.

IMO this is a big NO NO. I rather have some extra setup work until snappy SO as commonly available with the OSes.


> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16631-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 01:31:11 2011
Return-Path: <common-issues-return-16631-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 36C1D6DDB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 01:31:11 +0000 (UTC)
Received: (qmail 64627 invoked by uid 500); 23 Jun 2011 01:31:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64576 invoked by uid 500); 23 Jun 2011 01:31:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64568 invoked by uid 99); 23 Jun 2011 01:31:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 01:31:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 01:31:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7785642AAE9
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 01:30:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 01:30:47 +0000 (UTC)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1423692780.31287.1308792647486.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053590#comment-13053590 ] 

Scott Carey commented on HADOOP-7206:
-------------------------------------

bq. However, it has a serious drawback; the native code is not built in target OS, only on the same architecture. Because of this the build is not easy reproducible as there is not knowledge of the OS used to build it.

Sure it is reproducible.  snappy is used as an artifact, not built from source.  The build is reproducible because it _always_ uses the same artifact, and always produces the same output.  Is it a requirement to recompile all Java jars to be reproducible?


hadoop-snappy has another drawback/benefit pair:

Users may have snappy-java in their paths for their own use (for example via Avro, Hive, Hbase, or user code).
Drawback: the library can't be shared, bloating the # of classes and jars
Benefit: the library won't have a version conflict
Unknown(to me): does a snappy-java binding conflict with a hadoop custom one if both are loaded in the same JVM / Classloader?


I think the check for a system available libsnappy.so prior to loading the one in the jar should go into the snappy-java project, then users can optionally compile one and make it available to Hadoop, or use the one in the jar, and Hadoop has to maintain less code and build infrastructure as a result. 


> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16632-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 01:39:11 2011
Return-Path: <common-issues-return-16632-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1FEE46F05
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 01:39:11 +0000 (UTC)
Received: (qmail 75293 invoked by uid 500); 23 Jun 2011 01:39:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75258 invoked by uid 500); 23 Jun 2011 01:39:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75250 invoked by uid 99); 23 Jun 2011 01:39:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 01:39:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 01:39:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C063942ACB9
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 01:38:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 01:38:47 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1672174058.31349.1308793127784.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053592#comment-13053592 ] 

Alejandro Abdelnur commented on HADOOP-7206:
--------------------------------------------

@Scott,

snappy-java downloads snappy source TAR and builds it.

Regarding to your 'unknown' last time I've check, snappy-java SO and snappy SO could not be loaded at the same time in a JVM, the JVM would core dump.


> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16633-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 01:41:11 2011
Return-Path: <common-issues-return-16633-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0FA4F6F86
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 01:41:11 +0000 (UTC)
Received: (qmail 77680 invoked by uid 500); 23 Jun 2011 01:41:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77640 invoked by uid 500); 23 Jun 2011 01:41:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77632 invoked by uid 99); 23 Jun 2011 01:41:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 01:41:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 01:41:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7B3C642AE45
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 01:40:49 +0000 (UTC)
Date: Thu, 23 Jun 2011 01:40:49 +0000 (UTC)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1000923173.31461.1308793249501.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053595#comment-13053595 ] 

Scott Carey commented on HADOOP-7206:
-------------------------------------

bq. IMO this is a big NO NO. I rather have some extra setup work until snappy SO as commonly available with the OSes.

This may happen with Linux and BSD within a couple years, but likely will never happen for anything else.

snappy-java will be in user classpaths anyway.  I doubt every other project will go with this method (and if they did, the work should be shared, not in Hadoop.  If we feel strongly that snappy-java is doing it wrong, fork it into a new project and let everyone benefit!).  Other projects without the manpower to maintain this will just use snappy-java as is.  Avro has done this already and I'm sure others will too.  Including a jar as a dependency and having it 'just work' for 99% of users is an easy win.  Especially since Snappy is an optional compression codec for the vast majority of use cases.



> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16634-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 01:45:11 2011
Return-Path: <common-issues-return-16634-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9F2EA6113
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 01:45:11 +0000 (UTC)
Received: (qmail 81630 invoked by uid 500); 23 Jun 2011 01:45:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81577 invoked by uid 500); 23 Jun 2011 01:45:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81568 invoked by uid 99); 23 Jun 2011 01:45:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 01:45:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 01:45:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7298A42AF7A
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 01:44:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 01:44:47 +0000 (UTC)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1859066644.31467.1308793487466.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053598#comment-13053598 ] 

Scott Carey commented on HADOOP-7206:
-------------------------------------

bq.  Regarding to your 'unknown' last time I've check, snappy-java SO and snappy SO could not be loaded at the same time in a JVM, the JVM would core dump.

-1 to coredumps.  I already have snappy-java in my classpath in M/R jobs.

bq. snappy-java downloads snappy source TAR and builds it.

Ah, you mean that snappy-java's build is not reproducible.  Yes, I agree.  I thought you meant that _Hadoop_ was not reproducible if it used snappy-java.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16635-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 02:37:17 2011
Return-Path: <common-issues-return-16635-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 772054D0A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 02:37:17 +0000 (UTC)
Received: (qmail 33812 invoked by uid 500); 23 Jun 2011 02:37:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33445 invoked by uid 500); 23 Jun 2011 02:37:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33348 invoked by uid 99); 23 Jun 2011 02:37:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 02:37:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 02:37:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A96D442AC9C
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 02:36:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 02:36:47 +0000 (UTC)
From: "Taro L. Saito (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <260680014.31562.1308796607690.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053607#comment-13053607 ] 

Taro L. Saito commented on HADOOP-7206:
---------------------------------------

@Scott @Alejandro

Current version of snappy-java-1.0.3-rc3 can load libsnappy.so and libsnappyjava.so at the same time. I fixed the build options to avoid collision of snappy API between two native libraries.  

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16636-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 02:49:19 2011
Return-Path: <common-issues-return-16636-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8672C469E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 02:49:19 +0000 (UTC)
Received: (qmail 48443 invoked by uid 500); 23 Jun 2011 02:49:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48213 invoked by uid 500); 23 Jun 2011 02:49:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48103 invoked by uid 99); 23 Jun 2011 02:49:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 02:49:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 02:49:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 98E5742A24A
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 02:48:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 02:48:47 +0000 (UTC)
From: "Taro L. Saito (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <353583839.31630.1308797327622.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053610#comment-13053610 ] 

Taro L. Saito commented on HADOOP-7206:
---------------------------------------

I noticed that reusing libsnappy.so between multiple java libraries is not a good approach since JVM cannot load the same native libraries twice. 

If Hadoop loads libsnappy.so, the other third-party libraries cannot use libsnappy.so unless Snappy API used in the initial loader (i.e. Hadoop) is exposed to the users. 

Although I haven't fully confirmed that co-existence of libsnappy.so and libsnappyjava.so does not cause any serious problems, if this approach works well, Hadoop should integrate Issay's hadoop-snappy, which uses pre-installed libsnappy.so. The other Hadoop users can use snappy-java by simply putting the jar into their class path.  



> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16637-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 02:51:14 2011
Return-Path: <common-issues-return-16637-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3700F46D8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 02:51:14 +0000 (UTC)
Received: (qmail 50088 invoked by uid 500); 23 Jun 2011 02:51:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49937 invoked by uid 500); 23 Jun 2011 02:51:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49909 invoked by uid 99); 23 Jun 2011 02:51:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 02:51:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 02:51:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6200F42A430
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 02:50:49 +0000 (UTC)
Date: Thu, 23 Jun 2011 02:50:49 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <165092060.31734.1308797449398.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053611#comment-13053611 ] 

Allen Wittenauer commented on HADOOP-7206:
------------------------------------------

How does this work with 32-bit vs. 64-bit?  You push different jars? 

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16638-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 02:55:13 2011
Return-Path: <common-issues-return-16638-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7CF2D47F4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 02:55:13 +0000 (UTC)
Received: (qmail 56699 invoked by uid 500); 23 Jun 2011 02:55:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55217 invoked by uid 500); 23 Jun 2011 02:55:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54850 invoked by uid 99); 23 Jun 2011 02:55:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 02:55:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 02:55:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 38FF742A691
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 02:54:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 02:54:47 +0000 (UTC)
From: "Taro L. Saito (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1275154176.31753.1308797687230.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053615#comment-13053615 ] 

Taro L. Saito commented on HADOOP-7206:
---------------------------------------

@Allen

snappy-java checks os.arch system property, which shows different values for 32 and 64 bit CPUs. In a single snappy-java's jar file, native libraries for both 32 and 64 bits CPUs are bundled, and one of them is extracted according to the machine environment.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16639-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 03:01:19 2011
Return-Path: <common-issues-return-16639-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7735D4011
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 03:01:19 +0000 (UTC)
Received: (qmail 64039 invoked by uid 500); 23 Jun 2011 03:01:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63599 invoked by uid 500); 23 Jun 2011 03:01:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63551 invoked by uid 99); 23 Jun 2011 03:01:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 03:01:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 03:01:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 18E2142AA10
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 03:00:48 +0000 (UTC)
Date: Thu, 23 Jun 2011 03:00:48 +0000 (UTC)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <416318956.31838.1308798048098.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053622#comment-13053622 ] 

Scott Carey commented on HADOOP-7206:
-------------------------------------

Would it be reasonable to split snappy-java up into multiple maven modules, so that users can choose which parts they want?  Perhaps they want the Java portions, but none of the JNI, or want the JNI wrapper but not the implementation, etc.  A snappy-java could still be built with all artifacts bundled, but a la carte could also be chosen without a custom build of snappy-java.

I think it is best for users if as many use cases as possible were handled in one central library that could be shared by multiple projects.  Any work done for Hadoop is likely to be repeated elsewhere.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16640-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 03:02:27 2011
Return-Path: <common-issues-return-16640-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B3AE347DD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 03:02:27 +0000 (UTC)
Received: (qmail 66984 invoked by uid 500); 23 Jun 2011 03:02:26 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65954 invoked by uid 500); 23 Jun 2011 03:02:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65918 invoked by uid 99); 23 Jun 2011 03:02:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 03:02:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 03:02:15 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7B9B342AB0D
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 03:01:54 +0000 (UTC)
Date: Thu, 23 Jun 2011 03:01:54 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1296559745.31895.1308798114502.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053624#comment-13053624 ] 

Allen Wittenauer commented on HADOOP-7206:
------------------------------------------

How safe is os.arch on non-Sun JVMs?  How safe are the .so's on non-Sun JVMs? 

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16641-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 03:29:20 2011
Return-Path: <common-issues-return-16641-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D3E944B2A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 03:29:20 +0000 (UTC)
Received: (qmail 85996 invoked by uid 500); 23 Jun 2011 03:29:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85752 invoked by uid 500); 23 Jun 2011 03:29:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85714 invoked by uid 99); 23 Jun 2011 03:29:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 03:29:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 03:29:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2985C42A0F4
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 03:28:49 +0000 (UTC)
Date: Thu, 23 Jun 2011 03:28:49 +0000 (UTC)
From: "Taro L. Saito (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <835713009.32012.1308799729166.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053629#comment-13053629 ] 

Taro L. Saito commented on HADOOP-7206:
---------------------------------------

@Scott

JNI interface is tightly coupled with the Java code, so splitting Java/JNI/native libraries would not work as you expected. A reasonable option I can think of is to provide snappy-java without native libraries. In this case, users need to compile/install snappy-java's native library (make install to copy libsnappy.so to /usr/local/lib) and a jar file without native libraries. This style fits to users who need to compile everything from scratch. Snappy-java tries to load native libraries in the path specified in java.library.path if no bundled native library is found. 

@Allen
I am not familiar with non-Sun JVMs, but loading .so files depends only on locally installed library (GLIBC 2.3 or higher). No other dependencies exists in pre-compiled native libraries (for Linix) in snappy-java, because I carefully chosen the compilation options to avoid extra dependencies. 


> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16642-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 03:31:18 2011
Return-Path: <common-issues-return-16642-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2A02C41DF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 03:31:18 +0000 (UTC)
Received: (qmail 91226 invoked by uid 500); 23 Jun 2011 03:31:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88747 invoked by uid 500); 23 Jun 2011 03:31:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88712 invoked by uid 99); 23 Jun 2011 03:31:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 03:31:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 03:31:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7C19442A17C
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 03:30:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 03:30:47 +0000 (UTC)
From: "Taro L. Saito (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <453507571.32015.1308799847504.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053631#comment-13053631 ] 

Taro L. Saito commented on HADOOP-7206:
---------------------------------------

Fix:
make install to copy -libsnappy.so- *libsnappyjava.so* to /usr/local/lib


> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16643-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 04:36:28 2011
Return-Path: <common-issues-return-16643-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2F79544B8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 04:36:28 +0000 (UTC)
Received: (qmail 57221 invoked by uid 500); 23 Jun 2011 04:36:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56982 invoked by uid 500); 23 Jun 2011 04:36:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56961 invoked by uid 99); 23 Jun 2011 04:36:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 04:36:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 04:36:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 36A4042A085
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 04:35:48 +0000 (UTC)
Date: Thu, 23 Jun 2011 04:35:48 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <779082057.32108.1308803748220.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Moved] (HADOOP-7419) new hadoop-config.sh doesn't manage
 classpath for HADOOP_CONF_DIR correctly
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers moved HDFS-2089 to HADOOP-7419:
----------------------------------------------

        Fix Version/s:     (was: 0.23.0)
                       0.23.0
    Affects Version/s:     (was: 0.23.0)
                       0.23.0
                  Key: HADOOP-7419  (was: HDFS-2089)
              Project: Hadoop Common  (was: Hadoop HDFS)

> new hadoop-config.sh doesn't manage classpath for HADOOP_CONF_DIR correctly
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-7419
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7419
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Bing Zheng
>              Labels: newbie
>             Fix For: 0.23.0
>
>         Attachments: HDFS-2089.txt
>
>
> Since the introduction of the RPM packages, hadoop-config.sh incorrectly puts $HADOOP_HDFS_HOME/conf on the classpath regardless of whether HADOOP_CONF_DIR is already defined in the environment.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16644-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 04:38:12 2011
Return-Path: <common-issues-return-16644-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8326E44DB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 04:38:12 +0000 (UTC)
Received: (qmail 57768 invoked by uid 500); 23 Jun 2011 04:38:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57555 invoked by uid 500); 23 Jun 2011 04:38:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57546 invoked by uid 99); 23 Jun 2011 04:38:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 04:38:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 04:38:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 722F342A14F
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 04:37:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 04:37:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1343724135.32113.1308803867464.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7419) new hadoop-config.sh doesn't manage
 classpath for HADOOP_CONF_DIR correctly
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7419:
-----------------------------------

    Attachment: HDFS-2089.txt

Re-uploading the same patch to re-trigger test-patch.

> new hadoop-config.sh doesn't manage classpath for HADOOP_CONF_DIR correctly
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-7419
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7419
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Bing Zheng
>              Labels: newbie
>             Fix For: 0.23.0
>
>         Attachments: HDFS-2089.txt, HDFS-2089.txt
>
>
> Since the introduction of the RPM packages, hadoop-config.sh incorrectly puts $HADOOP_HDFS_HOME/conf on the classpath regardless of whether HADOOP_CONF_DIR is already defined in the environment.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16645-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 04:44:17 2011
Return-Path: <common-issues-return-16645-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0E4DC4544
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 04:44:17 +0000 (UTC)
Received: (qmail 62040 invoked by uid 500); 23 Jun 2011 04:44:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61906 invoked by uid 500); 23 Jun 2011 04:44:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61858 invoked by uid 99); 23 Jun 2011 04:44:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 04:44:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 04:44:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C3A5542A277
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 04:43:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 04:43:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <77760834.32122.1308804227798.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053648#comment-13053648 ] 

Todd Lipcon commented on HADOOP-7206:
-------------------------------------

Regarding the conflicts, it seems this is a visibility bug. In the case that the snappy code gets compiled into the same .so file as the JNI bindings, it should be set up with -fvisibility=hidden, and only the JNI implementations of native functions marked as exported symbols. Anything else is pretty unclean.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16646-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 04:50:16 2011
Return-Path: <common-issues-return-16646-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 85EA14BAA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 04:50:16 +0000 (UTC)
Received: (qmail 68169 invoked by uid 500); 23 Jun 2011 04:50:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67981 invoked by uid 500); 23 Jun 2011 04:50:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67966 invoked by uid 99); 23 Jun 2011 04:50:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 04:50:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 04:50:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 63D6A42A461
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 04:49:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 04:49:47 +0000 (UTC)
From: "Benjamin Reed (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1445204711.32183.1308804587405.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1443063883.27446.1308722147425.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7414) path to setWar() in HttpServer is
 not setup properly for resources from Jar URLs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053651#comment-13053651 ] 

Benjamin Reed commented on HADOOP-7414:
---------------------------------------

the only way there would be multiple slashes is if the name was "" or had a slash at the end of it. i don't think we hit either case.

note that there is a corresponding bug report in jetty: http://jira.codehaus.org/browse/JETTY-1307

the test case would need to run hadoop with the webapps in a jar file. i don't know of any easy way to do that.

> path to setWar() in HttpServer is not setup properly for resources from Jar URLs
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-7414
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7414
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Benjamin Reed
>         Attachments: HADOOP-7414.patch
>
>
> the path in setWar() needs to end in a '/' for jetty to properly process resources from Jar URLs. by adding a '/' at the end the webapps can run out of a Jar file instead of unzipping into a file system.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16647-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 04:54:25 2011
Return-Path: <common-issues-return-16647-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A5D2C4BF3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 04:54:25 +0000 (UTC)
Received: (qmail 69167 invoked by uid 500); 23 Jun 2011 04:54:25 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68912 invoked by uid 500); 23 Jun 2011 04:54:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68860 invoked by uid 99); 23 Jun 2011 04:54:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 04:54:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 04:54:16 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2780D42A5D1
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 04:53:56 +0000 (UTC)
Date: Thu, 23 Jun 2011 04:53:56 +0000 (UTC)
From: "Taro L. Saito (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1086435038.32237.1308804836158.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053652#comment-13053652 ] 

Taro L. Saito commented on HADOOP-7206:
---------------------------------------

@Todd

As you mentioned, snappy-java's native libraries are now built with -fvisibility=hidden, and I already tried to load libsnappy.so, then used API in snappy-java. No JVM crash is observed. 



> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16648-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 04:54:28 2011
Return-Path: <common-issues-return-16648-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D2F1C4BFE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 04:54:28 +0000 (UTC)
Received: (qmail 69254 invoked by uid 500); 23 Jun 2011 04:54:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69169 invoked by uid 500); 23 Jun 2011 04:54:25 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68861 invoked by uid 99); 23 Jun 2011 04:54:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 04:54:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 04:54:16 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 34B5E42A5D3
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 04:53:56 +0000 (UTC)
Date: Thu, 23 Jun 2011 04:53:56 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1846313595.32239.1308804836212.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7419) new hadoop-config.sh doesn't
 manage classpath for HADOOP_CONF_DIR correctly
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053653#comment-13053653 ] 

Hadoop QA commented on HADOOP-7419:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12483545/HDFS-2089.txt
  against trunk revision 1137724.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/666//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/666//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/666//console

This message is automatically generated.

> new hadoop-config.sh doesn't manage classpath for HADOOP_CONF_DIR correctly
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-7419
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7419
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Bing Zheng
>              Labels: newbie
>             Fix For: 0.23.0
>
>         Attachments: HDFS-2089.txt, HDFS-2089.txt
>
>
> Since the introduction of the RPM packages, hadoop-config.sh incorrectly puts $HADOOP_HDFS_HOME/conf on the classpath regardless of whether HADOOP_CONF_DIR is already defined in the environment.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16649-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 08:38:30 2011
Return-Path: <common-issues-return-16649-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5A99C41A6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 08:38:30 +0000 (UTC)
Received: (qmail 11051 invoked by uid 500); 23 Jun 2011 08:38:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10579 invoked by uid 500); 23 Jun 2011 08:38:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10375 invoked by uid 99); 23 Jun 2011 08:38:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 08:38:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 08:38:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7B38242A815
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 08:37:51 +0000 (UTC)
Date: Thu, 23 Jun 2011 08:37:51 +0000 (UTC)
From: "issei yoshida (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1526226641.32635.1308818271501.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053713#comment-13053713 ] 

issei yoshida commented on HADOOP-7206:
---------------------------------------

Taro,

> I noticed that reusing libsnappy.so between multiple java libraries is not a good approach since JVM cannot load the same native libraries twice.

The patch I uploaded before loads libsnappy.so via dlopen in the native code,
so it probably isn't a problem.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16650-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 08:38:31 2011
Return-Path: <common-issues-return-16650-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5AF7941AA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 08:38:30 +0000 (UTC)
Received: (qmail 11053 invoked by uid 500); 23 Jun 2011 08:38:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10768 invoked by uid 500); 23 Jun 2011 08:38:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10340 invoked by uid 99); 23 Jun 2011 08:38:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 08:38:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 08:38:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3C7AD42A7CD
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 08:37:49 +0000 (UTC)
Date: Thu, 23 Jun 2011 08:37:49 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <778872678.32582.1308818269244.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1443063883.27446.1308722147425.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7414) path to setWar() in HttpServer is
 not setup properly for resources from Jar URLs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053712#comment-13053712 ] 

Steve Loughran commented on HADOOP-7414:
----------------------------------------

I had a look through the Jetty source on this, then into jar: URLs, and ended up more confused than ever, so exchanged emails with Greg Wilkins, because it seemed to me that WAR files were overkill compared to creating and configuring a servlet context. His reply

"I tend to agree that Hadoop should not be using WebAppContext, as that
drags in all the war mechanisms of special classloaders, security,
sessions etc. that I suspect that hadoop does not need.   It would be
more light weight to program at the ContextHandler level and add only
the handlers that you need (probably just ServletHandler).   If you
are interested, I can point you at the jetty embedded examples.

For this specific issue, adding a / to the end of the path does
suggest to jetty that it is a directory rather than a WAR file.   If
it is a WAR file, then you don't need the /.  The extraction into temp
files is controlled by the setExtractWar method on WebAppContext and
not the trailing /"

What is trying to be achieved here? To deploy some servlets in the Hadoop JVM? off the normal classpath? If so, I don't think WebApp contexts are the right way. Jetty's context handling may seem a bit trickier to write, but I use them in-JVM regularly and don't have many problems (many - the main one was SLF4J, but I fixed that)

> path to setWar() in HttpServer is not setup properly for resources from Jar URLs
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-7414
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7414
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Benjamin Reed
>         Attachments: HADOOP-7414.patch
>
>
> the path in setWar() needs to end in a '/' for jetty to properly process resources from Jar URLs. by adding a '/' at the end the webapps can run out of a Jar file instead of unzipping into a file system.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16651-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 10:45:12 2011
Return-Path: <common-issues-return-16651-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2B17F4C10
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 10:45:12 +0000 (UTC)
Received: (qmail 93345 invoked by uid 500); 23 Jun 2011 10:45:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93051 invoked by uid 500); 23 Jun 2011 10:45:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92860 invoked by uid 99); 23 Jun 2011 10:45:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 10:45:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 10:45:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 825BD42AAA1
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 10:44:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 10:44:47 +0000 (UTC)
From: "Pushparaj (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1647765621.32935.1308825887530.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7420) HDFS package reference throws error
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

HDFS package reference throws error
-----------------------------------

                 Key: HADOOP-7420
                 URL: https://issues.apache.org/jira/browse/HADOOP-7420
             Project: Hadoop Common
          Issue Type: Bug
          Components: build
    Affects Versions: 0.21.0
         Environment: Build
            Reporter: Pushparaj


I have referenced this package in my Java code. The project needs to be compiled using JDK 1.5 compiler only.

The Hadoop reference throws error from javac as below. 

[javac] class file has wrong version 50.0, should be 49.0

Note: If compiled using Java 1.6 , it goes through.


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16652-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 11:45:08 2011
Return-Path: <common-issues-return-16652-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E4D1D4717
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 11:45:08 +0000 (UTC)
Received: (qmail 82907 invoked by uid 500); 23 Jun 2011 11:45:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82880 invoked by uid 500); 23 Jun 2011 11:45:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82872 invoked by uid 99); 23 Jun 2011 11:45:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 11:45:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 11:45:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6E51642B3DC
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 11:44:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 11:44:47 +0000 (UTC)
From: "Uparis Abeysena (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1956559103.33015.1308829487448.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-1224) "Browse the filesystem" link
 pointing to a dead data-node
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-1224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053803#comment-13053803 ] 

Uparis Abeysena commented on HADOOP-1224:
-----------------------------------------

results see: http://retractable-dog-leashes.net

> "Browse the filesystem" link pointing to a dead data-node
> ---------------------------------------------------------
>
>                 Key: HADOOP-1224
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1224
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.12.3
>            Reporter: Konstantin Shvachko
>             Fix For: 0.13.0
>
>         Attachments: DFSBrowsingDeadNode_v1.0.patch, DFSBrowsingDeadNode_v1.1.patch, DFSBrowsingDeadNode_v1.2.patch
>
>
> On the NameNode status web page "Browse the filesystem" link can point to a dead data-node.
> The reason for that is that FSNamesystem.randomDataNode() selects a random node from the
> list of all nodes rather then selecting among alive nodes only.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16653-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 14:32:10 2011
Return-Path: <common-issues-return-16653-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 53162466B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 14:32:10 +0000 (UTC)
Received: (qmail 50508 invoked by uid 500); 23 Jun 2011 14:32:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50452 invoked by uid 500); 23 Jun 2011 14:32:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50334 invoked by uid 99); 23 Jun 2011 14:32:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 14:32:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 14:32:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7319F42BD41
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 14:31:48 +0000 (UTC)
Date: Thu, 23 Jun 2011 14:31:48 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1947269000.33382.1308839508468.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1635687927.25170.1308677750532.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7411) Convert remaining Ant based build
 to Maven
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur reassigned HADOOP-7411:
------------------------------------------

    Assignee: Alejandro Abdelnur

> Convert remaining Ant based build to Maven
> ------------------------------------------
>
>                 Key: HADOOP-7411
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7411
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>
> Once Mavenization is complete, do a second iteration to remove antrun calls out, this may require writing some Mojos.
> The tricky things are native compilation and symlinks handling (for native libs) when creating packaging.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16654-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 14:32:11 2011
Return-Path: <common-issues-return-16654-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B722C46A3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 14:32:11 +0000 (UTC)
Received: (qmail 51130 invoked by uid 500); 23 Jun 2011 14:32:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51096 invoked by uid 500); 23 Jun 2011 14:32:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51084 invoked by uid 99); 23 Jun 2011 14:32:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 14:32:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 14:32:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CBC7942BD32
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 14:31:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 14:31:47 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1813883862.33368.1308839507831.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1221804712.25214.1308677867694.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7412) Mavenization Umbrella
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur reassigned HADOOP-7412:
------------------------------------------

    Assignee: Alejandro Abdelnur

> Mavenization Umbrella
> ---------------------
>
>                 Key: HADOOP-7412
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7412
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>
> Umbrella JIRA for all Mavenization JIRAs

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16655-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 15:08:12 2011
Return-Path: <common-issues-return-16655-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6D5374B40
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 15:08:12 +0000 (UTC)
Received: (qmail 8972 invoked by uid 500); 23 Jun 2011 15:08:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8853 invoked by uid 500); 23 Jun 2011 15:08:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8703 invoked by uid 99); 23 Jun 2011 15:08:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 15:08:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 15:08:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 72AD342B1B4
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 15:07:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 15:07:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <771981300.33507.1308841667466.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1647765621.32935.1308825887530.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7420) HDFS package reference throws error
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon resolved HADOOP-7420.
---------------------------------

    Resolution: Invalid

Hadoop requires Java 1.6.0 - this is known and expected.

> HDFS package reference throws error
> -----------------------------------
>
>                 Key: HADOOP-7420
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7420
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>         Environment: Build
>            Reporter: Pushparaj
>
> I have referenced this package in my Java code. The project needs to be compiled using JDK 1.5 compiler only.
> The Hadoop reference throws error from javac as below. 
> [javac] class file has wrong version 50.0, should be 49.0
> Note: If compiled using Java 1.6 , it goes through.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16656-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 16:56:11 2011
Return-Path: <common-issues-return-16656-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 11CC346FC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 16:56:11 +0000 (UTC)
Received: (qmail 24481 invoked by uid 500); 23 Jun 2011 16:56:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24328 invoked by uid 500); 23 Jun 2011 16:56:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24320 invoked by uid 99); 23 Jun 2011 16:56:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 16:56:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 16:56:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A9A9A42B3AE
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 16:55:49 +0000 (UTC)
Date: Thu, 23 Jun 2011 16:55:49 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1449616331.33814.1308848149691.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053962#comment-13053962 ] 

Alejandro Abdelnur commented on HADOOP-7206:
--------------------------------------------

@Taro, regarding 'loading native libs twice', if using the same classloader, loading a nativelib a second time is a NOP, the JVM complains only if the classloader is different from the first load.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16657-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 17:06:11 2011
Return-Path: <common-issues-return-16657-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 755A1425C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 17:06:11 +0000 (UTC)
Received: (qmail 49944 invoked by uid 500); 23 Jun 2011 17:06:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49895 invoked by uid 500); 23 Jun 2011 17:06:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49879 invoked by uid 99); 23 Jun 2011 17:06:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 17:06:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 17:06:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1348242B962
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 17:05:50 +0000 (UTC)
Date: Thu, 23 Jun 2011 17:05:50 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2055750991.33895.1308848750075.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053971#comment-13053971 ] 

Allen Wittenauer commented on HADOOP-7206:
------------------------------------------

I'd like to see:

a) OS X (32-bit and 64-bit) support
b) This tested on something other than a Sun JVM, preferably IBM's since a few folks are using that.

Also, how hard is it to add a non-bundled so?

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16658-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 17:38:09 2011
Return-Path: <common-issues-return-16658-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6DE9A4769
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 17:38:09 +0000 (UTC)
Received: (qmail 6514 invoked by uid 500); 23 Jun 2011 17:38:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6458 invoked by uid 500); 23 Jun 2011 17:38:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6293 invoked by uid 99); 23 Jun 2011 17:38:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 17:38:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 17:38:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F16CB42B8C2
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 17:37:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 17:37:47 +0000 (UTC)
From: "Jeffrey Naisbitt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1634947298.33988.1308850667984.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <467948518.30516.1305901367779.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7314) Add support for throwing
 UnknownHostException when a host doesn't resolve
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jeffrey Naisbitt updated HADOOP-7314:
-------------------------------------

    Status: Patch Available  (was: Open)

> Add support for throwing UnknownHostException when a host doesn't resolve
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-7314
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7314
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jeffrey Naisbitt
>            Assignee: Jeffrey Naisbitt
>         Attachments: HADOOP-7314-v2.patch, HADOOP-7314.patch
>
>
> As part of MAPREDUCE-2489, we need support for having the resolve methods (for DNS mapping) throw UnknownHostExceptions.  (Currently, they hide the exception).  Since the existing 'resolve' method is ultimately used by several other locations/components, I propose we add a new 'resolveValidHosts' method.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16659-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 17:38:11 2011
Return-Path: <common-issues-return-16659-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 443934786
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 17:38:11 +0000 (UTC)
Received: (qmail 6970 invoked by uid 500); 23 Jun 2011 17:38:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6938 invoked by uid 500); 23 Jun 2011 17:38:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6930 invoked by uid 99); 23 Jun 2011 17:38:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 17:38:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 17:38:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E2F2442B8C0
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 17:37:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 17:37:47 +0000 (UTC)
From: "Jeffrey Naisbitt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <598046568.33986.1308850667902.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <467948518.30516.1305901367779.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7314) Add support for throwing
 UnknownHostException when a host doesn't resolve
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jeffrey Naisbitt updated HADOOP-7314:
-------------------------------------

    Attachment: HADOOP-7314-v2.patch

Updated patch for trunk

> Add support for throwing UnknownHostException when a host doesn't resolve
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-7314
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7314
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jeffrey Naisbitt
>            Assignee: Jeffrey Naisbitt
>         Attachments: HADOOP-7314-v2.patch, HADOOP-7314.patch
>
>
> As part of MAPREDUCE-2489, we need support for having the resolve methods (for DNS mapping) throw UnknownHostExceptions.  (Currently, they hide the exception).  Since the existing 'resolve' method is ultimately used by several other locations/components, I propose we add a new 'resolveValidHosts' method.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16660-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 17:56:11 2011
Return-Path: <common-issues-return-16660-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0CF6F417A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 17:56:10 +0000 (UTC)
Received: (qmail 38576 invoked by uid 500); 23 Jun 2011 17:56:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38540 invoked by uid 500); 23 Jun 2011 17:56:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38531 invoked by uid 99); 23 Jun 2011 17:56:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 17:56:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 17:56:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5BC8D42B194
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 17:55:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 17:55:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <794665588.34087.1308851747372.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <467948518.30516.1305901367779.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7314) Add support for throwing
 UnknownHostException when a host doesn't resolve
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054005#comment-13054005 ] 

Hadoop QA commented on HADOOP-7314:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12483621/HADOOP-7314-v2.patch
  against trunk revision 1137724.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/667//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/667//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/667//console

This message is automatically generated.

> Add support for throwing UnknownHostException when a host doesn't resolve
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-7314
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7314
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jeffrey Naisbitt
>            Assignee: Jeffrey Naisbitt
>         Attachments: HADOOP-7314-v2.patch, HADOOP-7314.patch
>
>
> As part of MAPREDUCE-2489, we need support for having the resolve methods (for DNS mapping) throw UnknownHostExceptions.  (Currently, they hide the exception).  Since the existing 'resolve' method is ultimately used by several other locations/components, I propose we add a new 'resolveValidHosts' method.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16661-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 19:08:08 2011
Return-Path: <common-issues-return-16661-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DCAA247D7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 19:08:08 +0000 (UTC)
Received: (qmail 96335 invoked by uid 500); 23 Jun 2011 19:08:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96303 invoked by uid 500); 23 Jun 2011 19:08:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96281 invoked by uid 99); 23 Jun 2011 19:08:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 19:08:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 19:08:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 631B042B1BD
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 19:07:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 19:07:47 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <298429641.34286.1308856067402.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6508) Incorrect values for metrics with
 CompositeContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054040#comment-13054040 ] 

Allen Wittenauer commented on HADOOP-6508:
------------------------------------------

Should this really be resolved/fixed or is this JIRA in some other state?

> Incorrect values for metrics with CompositeContext
> --------------------------------------------------
>
>                 Key: HADOOP-6508
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6508
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.0
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: CompositeContext-solution.png, CompositeContext.png
>
>
> In our clusters, when we use CompositeContext with two contexts, second context gets wrong values.
> This problem is consistent on 500 (and above) node cluster.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16662-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 20:27:12 2011
Return-Path: <common-issues-return-16662-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A07624E09
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 20:27:12 +0000 (UTC)
Received: (qmail 16109 invoked by uid 500); 23 Jun 2011 20:27:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16054 invoked by uid 500); 23 Jun 2011 20:27:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15856 invoked by uid 99); 23 Jun 2011 20:27:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 20:27:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 20:27:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 78C6442B58B
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 20:26:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 20:26:47 +0000 (UTC)
From: "Ron Bodkin (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <538925304.34436.1308860807491.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7421) Performance Patch: Disable
 Client-Side Nagle By Default
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Performance Patch: Disable Client-Side Nagle By Default
-------------------------------------------------------

                 Key: HADOOP-7421
                 URL: https://issues.apache.org/jira/browse/HADOOP-7421
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Ron Bodkin


HADOOP-2232 was a valuable contribution, but I would suggest that Hadoop IPC should disable Nagle's algorithm by default, to improve performance system-wide. Nagle just adds delay to network connections, which is unnecessary in the high bandwidth networks that Hadoop uses. As it stands, this becomes an obscure, little known but important performance optimization (especially for HBase), which took us days to track down during a performance tuning exercise. Obviously integrating this optimization into the Hadoop release will require performance testing to validate that it doesn't harm performance, but it should only help.

I will submit a patch for how to address this in trunk.


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16663-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 20:43:11 2011
Return-Path: <common-issues-return-16663-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BA14C47B3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 20:43:11 +0000 (UTC)
Received: (qmail 47164 invoked by uid 500); 23 Jun 2011 20:43:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47117 invoked by uid 500); 23 Jun 2011 20:43:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47109 invoked by uid 99); 23 Jun 2011 20:43:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 20:43:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 20:43:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CCF1842BD96
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 20:42:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 20:42:47 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1370156955.34475.1308861767836.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <407429641.30834.1308786767442.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7417) Hadoop Management System
 (Umbrella)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054087#comment-13054087 ] 

Allen Wittenauer commented on HADOOP-7417:
------------------------------------------

Is the intent to provide bindings to pre-existing software or is the intent to re-invent the wheel?

> Hadoop Management System (Umbrella)
> -----------------------------------
>
>                 Key: HADOOP-7417
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7417
>             Project: Hadoop Common
>          Issue Type: New Feature
>         Environment: Java 6, Linux
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>
> The primary goal of Hadoop Management System is to build a component around management and deployment of Hadoop related projects. This includes software installation, configuration, application orchestration, deployment automation and monitoring Hadoop.
> Prototype demo source code can be obtained from:
> http://github.com/macroadster/hms

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16664-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 22:35:09 2011
Return-Path: <common-issues-return-16664-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8D1C34B88
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 22:35:09 +0000 (UTC)
Received: (qmail 43752 invoked by uid 500); 23 Jun 2011 22:35:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43697 invoked by uid 500); 23 Jun 2011 22:35:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43678 invoked by uid 99); 23 Jun 2011 22:35:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 22:35:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 22:35:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1963742C518
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 22:34:48 +0000 (UTC)
Date: Thu, 23 Jun 2011 22:34:48 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <767799780.34844.1308868488100.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6682) NetUtils:normalizeHostName does
 not process hostnames starting with [a-f] correctly
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054146#comment-13054146 ] 

Eli Collins commented on HADOOP-6682:
-------------------------------------

The Y20 patch here is bogus btw it changes from 10 to 16 instead of vice versa. Doesn't look like it was applied anywhere so not a big deal.

> NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6682
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6682
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6682-Y20.patch, HADOOP-6682.patch
>
>
>   public static String normalizeHostName(String name) {
>     if (Character.digit(name.charAt(0), 16) != -1) {
>       return name;
> This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address.  This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f].  Such names will not be converted to an ip address, but be returned unchanged.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16665-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 22:39:08 2011
Return-Path: <common-issues-return-16665-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C0B234DD9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 22:39:08 +0000 (UTC)
Received: (qmail 50459 invoked by uid 500); 23 Jun 2011 22:39:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50437 invoked by uid 500); 23 Jun 2011 22:39:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50429 invoked by uid 99); 23 Jun 2011 22:39:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 22:39:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 22:39:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8DFD442C67B
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 22:38:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 22:38:47 +0000 (UTC)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <522664711.34856.1308868727578.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6682) NetUtils:normalizeHostName does
 not process hostnames starting with [a-f] correctly
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054148#comment-13054148 ] 

Jakob Homan commented on HADOOP-6682:
-------------------------------------

hmmm... must have run the patch the wrong way.  That was one crazy week.

> NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6682
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6682
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6682-Y20.patch, HADOOP-6682.patch
>
>
>   public static String normalizeHostName(String name) {
>     if (Character.digit(name.charAt(0), 16) != -1) {
>       return name;
> This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address.  This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f].  Such names will not be converted to an ip address, but be returned unchanged.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16666-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 23:13:09 2011
Return-Path: <common-issues-return-16666-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A23EA42AA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 23:13:09 +0000 (UTC)
Received: (qmail 87068 invoked by uid 500); 23 Jun 2011 23:13:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86979 invoked by uid 500); 23 Jun 2011 23:13:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86718 invoked by uid 99); 23 Jun 2011 23:13:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 23:13:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 23:13:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B59DF42C200
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 23:12:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 23:12:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <452979018.34930.1308870767740.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7422) Test that the topology script is
 always passed IP addresses
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Test that the topology script is always passed IP addresses
-----------------------------------------------------------

                 Key: HADOOP-7422
                 URL: https://issues.apache.org/jira/browse/HADOOP-7422
             Project: Hadoop Common
          Issue Type: Test
          Components: test
            Reporter: Eli Collins


Now that HADOOP-6682 has been fixed, Hadoop should always pass the topology script an IP address rather than a hostname. We should write a test that covers this (specifically that DNSToSwitchMapping#resolve is always passed IP addresses) so users can safely write topology scripts that don't handle hostnames.   

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16667-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 23:15:08 2011
Return-Path: <common-issues-return-16667-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A3D2543A8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 23:15:08 +0000 (UTC)
Received: (qmail 90093 invoked by uid 500); 23 Jun 2011 23:15:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89997 invoked by uid 500); 23 Jun 2011 23:15:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89988 invoked by uid 99); 23 Jun 2011 23:15:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 23:15:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 23:15:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 68CCF42C2CE
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 23:14:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 23:14:47 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1950919684.34935.1308870887426.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <113408089.3186.1307985891583.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7385) Remove
 StringUtils.stringifyException(ie) in logger functions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054167#comment-13054167 ] 

Tanping Wang commented on HADOOP-7385:
--------------------------------------

+1 on the patch.

> Remove StringUtils.stringifyException(ie) in logger functions
> -------------------------------------------------------------
>
>                 Key: HADOOP-7385
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7385
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7385-1.patch
>
>
> Apache logger api has an overloaded function which can take the message and exception. I am proposing to clean the logging code with this api.
> ie.:
> Change the code from LOG.warn(msg, StringUtils.stringifyException(exception)); to LOG.warn(msg, exception);

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16668-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 23:15:10 2011
Return-Path: <common-issues-return-16668-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C0F7443D4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 23:15:10 +0000 (UTC)
Received: (qmail 90347 invoked by uid 500); 23 Jun 2011 23:15:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90317 invoked by uid 500); 23 Jun 2011 23:15:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90309 invoked by uid 99); 23 Jun 2011 23:15:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 23:15:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 23:15:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 75F8542C2D0
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 23:14:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 23:14:47 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <168201296.34937.1308870887480.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <113408089.3186.1307985891583.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7385) Remove
 StringUtils.stringifyException(ie) in logger functions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tanping Wang updated HADOOP-7385:
---------------------------------

    Status: Patch Available  (was: Open)

> Remove StringUtils.stringifyException(ie) in logger functions
> -------------------------------------------------------------
>
>                 Key: HADOOP-7385
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7385
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7385-1.patch
>
>
> Apache logger api has an overloaded function which can take the message and exception. I am proposing to clean the logging code with this api.
> ie.:
> Change the code from LOG.warn(msg, StringUtils.stringifyException(exception)); to LOG.warn(msg, exception);

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16669-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 23:17:10 2011
Return-Path: <common-issues-return-16669-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D2D4B4A9D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 23:17:10 +0000 (UTC)
Received: (qmail 94879 invoked by uid 500); 23 Jun 2011 23:17:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94842 invoked by uid 500); 23 Jun 2011 23:17:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94829 invoked by uid 99); 23 Jun 2011 23:17:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 23:17:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 23:17:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6AAF642C3F3
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 23:16:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 23:16:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <814884523.34939.1308871007433.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7423) Document topology script
 requirements
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Document topology script requirements
-------------------------------------

                 Key: HADOOP-7423
                 URL: https://issues.apache.org/jira/browse/HADOOP-7423
             Project: Hadoop Common
          Issue Type: Improvement
          Components: documentation
            Reporter: Eli Collins


The topology script documentation is cluster_setup.xml is unclear. The topology script:
# Only needs to handle IP addresses (not hostnames)
# Needs to handle multiple arguments for caching to work effectively

We should check in an example script or include an example one in the docs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16670-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 23:23:10 2011
Return-Path: <common-issues-return-16670-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D62164B02
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 23:23:10 +0000 (UTC)
Received: (qmail 1999 invoked by uid 500); 23 Jun 2011 23:23:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1644 invoked by uid 500); 23 Jun 2011 23:23:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1382 invoked by uid 99); 23 Jun 2011 23:23:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 23:23:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 23:23:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8EA5E42C62A
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 23:22:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 23:22:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1133748281.34969.1308871367581.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7424) Log an error if the topology script
 doesn't handle multiple args
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Log an error if the topology script doesn't handle multiple args
----------------------------------------------------------------

                 Key: HADOOP-7424
                 URL: https://issues.apache.org/jira/browse/HADOOP-7424
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Eli Collins


ScriptBasedMapping#resolve currently warns and returns null if it passes n arguments to the topology script and gets back a different number of resolutions. This indicates a bug in the topology script (or it's input) and therefore should be an error.

{code}
// invalid number of entries returned by the script
LOG.warn("Script " + scriptName + " returned "
   + Integer.toString(m.size()) + " values when "
   + Integer.toString(names.size()) + " were expected.");
return null;
{code}

There's only one place in Hadoop (FSNamesystem init) where we pass multiple arguments to the topology script, and it only done for performance (to trigger resolution/caching of all the hosts in the includes file on startup). So currently a topology script that doesn't handle multiple arguments just means the initial cache population doesn't work.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16671-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 23:23:11 2011
Return-Path: <common-issues-return-16671-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 231FA4B10
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 23:23:11 +0000 (UTC)
Received: (qmail 3161 invoked by uid 500); 23 Jun 2011 23:23:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3120 invoked by uid 500); 23 Jun 2011 23:23:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3084 invoked by uid 99); 23 Jun 2011 23:23:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 23:23:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 23:23:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6A1AF42C625
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 23:22:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 23:22:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1546283423.34964.1308871367431.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <814884523.34939.1308871007433.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7423) Document topology script
 requirements
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7423:
-----------------------------------

      Tags:   (was: newbie)
    Labels: newbie  (was: )

> Document topology script requirements
> -------------------------------------
>
>                 Key: HADOOP-7423
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7423
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Eli Collins
>              Labels: newbie
>
> The topology script documentation is cluster_setup.xml is unclear. The topology script:
> # Only needs to handle IP addresses (not hostnames)
> # Needs to handle multiple arguments for caching to work effectively
> We should check in an example script or include an example one in the docs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16672-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 23:25:08 2011
Return-Path: <common-issues-return-16672-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BE8CD4B6E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 23:25:08 +0000 (UTC)
Received: (qmail 7522 invoked by uid 500); 23 Jun 2011 23:25:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7484 invoked by uid 500); 23 Jun 2011 23:25:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7475 invoked by uid 99); 23 Jun 2011 23:25:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 23:25:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 23:25:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 666AD42C735
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 23:24:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 23:24:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <735184810.34971.1308871487416.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1133748281.34969.1308871367581.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7424) Log an error if the topology script
 doesn't handle multiple args
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7424:
--------------------------------

      Tags:   (was: newbie)
    Labels: newbie  (was: )

> Log an error if the topology script doesn't handle multiple args
> ----------------------------------------------------------------
>
>                 Key: HADOOP-7424
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7424
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Eli Collins
>              Labels: newbie
>
> ScriptBasedMapping#resolve currently warns and returns null if it passes n arguments to the topology script and gets back a different number of resolutions. This indicates a bug in the topology script (or it's input) and therefore should be an error.
> {code}
> // invalid number of entries returned by the script
> LOG.warn("Script " + scriptName + " returned "
>    + Integer.toString(m.size()) + " values when "
>    + Integer.toString(names.size()) + " were expected.");
> return null;
> {code}
> There's only one place in Hadoop (FSNamesystem init) where we pass multiple arguments to the topology script, and it only done for performance (to trigger resolution/caching of all the hosts in the includes file on startup). So currently a topology script that doesn't handle multiple arguments just means the initial cache population doesn't work.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16673-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 23:25:09 2011
Return-Path: <common-issues-return-16673-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1147E4B7C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 23:25:09 +0000 (UTC)
Received: (qmail 7730 invoked by uid 500); 23 Jun 2011 23:25:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7674 invoked by uid 500); 23 Jun 2011 23:25:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7554 invoked by uid 99); 23 Jun 2011 23:25:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 23:25:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 23:25:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 85D1542C73B
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 23:24:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 23:24:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1738749828.34975.1308871487544.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <452979018.34930.1308870767740.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7422) Test that the topology script is
 always passed IP addresses
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7422:
--------------------------------

      Tags:   (was: newbie)
    Labels: newbie  (was: )

> Test that the topology script is always passed IP addresses
> -----------------------------------------------------------
>
>                 Key: HADOOP-7422
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7422
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>            Reporter: Eli Collins
>              Labels: newbie
>
> Now that HADOOP-6682 has been fixed, Hadoop should always pass the topology script an IP address rather than a hostname. We should write a test that covers this (specifically that DNSToSwitchMapping#resolve is always passed IP addresses) so users can safely write topology scripts that don't handle hostnames.   

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16674-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 23:35:12 2011
Return-Path: <common-issues-return-16674-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 42E7B45DB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 23:35:12 +0000 (UTC)
Received: (qmail 26008 invoked by uid 500); 23 Jun 2011 23:35:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25980 invoked by uid 500); 23 Jun 2011 23:35:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25972 invoked by uid 99); 23 Jun 2011 23:35:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 23:35:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 23:35:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 45AF442CADC
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 23:34:48 +0000 (UTC)
Date: Thu, 23 Jun 2011 23:34:48 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2007686244.35009.1308872088282.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <113408089.3186.1307985891583.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7385) Remove
 StringUtils.stringifyException(ie) in logger functions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054177#comment-13054177 ] 

Hadoop QA commented on HADOOP-7385:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12482614/HADOOP-7385-1.patch
  against trunk revision 1137724.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/668//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/668//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/668//console

This message is automatically generated.

> Remove StringUtils.stringifyException(ie) in logger functions
> -------------------------------------------------------------
>
>                 Key: HADOOP-7385
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7385
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7385-1.patch
>
>
> Apache logger api has an overloaded function which can take the message and exception. I am proposing to clean the logging code with this api.
> ie.:
> Change the code from LOG.warn(msg, StringUtils.stringifyException(exception)); to LOG.warn(msg, exception);

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16675-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 23:57:08 2011
Return-Path: <common-issues-return-16675-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9FE684F45
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 23:57:08 +0000 (UTC)
Received: (qmail 51729 invoked by uid 500); 23 Jun 2011 23:57:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51703 invoked by uid 500); 23 Jun 2011 23:57:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51695 invoked by uid 99); 23 Jun 2011 23:57:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 23:57:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 23:57:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6B19242C02B
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 23:56:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 23:56:47 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <996387520.35050.1308873407435.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <113408089.3186.1307985891583.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7385) Remove
 StringUtils.stringifyException(ie) in logger functions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tanping Wang updated HADOOP-7385:
---------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

> Remove StringUtils.stringifyException(ie) in logger functions
> -------------------------------------------------------------
>
>                 Key: HADOOP-7385
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7385
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7385-1.patch
>
>
> Apache logger api has an overloaded function which can take the message and exception. I am proposing to clean the logging code with this api.
> ie.:
> Change the code from LOG.warn(msg, StringUtils.stringifyException(exception)); to LOG.warn(msg, exception);

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16676-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 23 23:59:11 2011
Return-Path: <common-issues-return-16676-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E7DB84F7E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 23 Jun 2011 23:59:10 +0000 (UTC)
Received: (qmail 55137 invoked by uid 500); 23 Jun 2011 23:59:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55115 invoked by uid 500); 23 Jun 2011 23:59:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55107 invoked by uid 99); 23 Jun 2011 23:59:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 23:59:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 23:59:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9970442C100
	for <common-issues@hadoop.apache.org>; Thu, 23 Jun 2011 23:58:47 +0000 (UTC)
Date: Thu, 23 Jun 2011 23:58:47 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <285596089.35055.1308873527625.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <113408089.3186.1307985891583.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7385) Remove
 StringUtils.stringifyException(ie) in logger functions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054185#comment-13054185 ] 

Tanping Wang commented on HADOOP-7385:
--------------------------------------

Just committed the patch.  Thanks to Bharath Mundlapudi!

> Remove StringUtils.stringifyException(ie) in logger functions
> -------------------------------------------------------------
>
>                 Key: HADOOP-7385
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7385
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7385-1.patch
>
>
> Apache logger api has an overloaded function which can take the message and exception. I am proposing to clean the logging code with this api.
> ie.:
> Change the code from LOG.warn(msg, StringUtils.stringifyException(exception)); to LOG.warn(msg, exception);

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16677-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 00:03:08 2011
Return-Path: <common-issues-return-16677-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C5F7E4078
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 00:03:08 +0000 (UTC)
Received: (qmail 61152 invoked by uid 500); 24 Jun 2011 00:03:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61123 invoked by uid 500); 24 Jun 2011 00:03:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61115 invoked by uid 99); 24 Jun 2011 00:03:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 00:03:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 00:03:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6997B42C22E
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 00:02:47 +0000 (UTC)
Date: Fri, 24 Jun 2011 00:02:47 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <591641687.35065.1308873767429.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <113408089.3186.1307985891583.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7385) Remove
 StringUtils.stringifyException(ie) in logger functions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tanping Wang updated HADOOP-7385:
---------------------------------

    Hadoop Flags: [Reviewed]

> Remove StringUtils.stringifyException(ie) in logger functions
> -------------------------------------------------------------
>
>                 Key: HADOOP-7385
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7385
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7385-1.patch
>
>
> Apache logger api has an overloaded function which can take the message and exception. I am proposing to clean the logging code with this api.
> ie.:
> Change the code from LOG.warn(msg, StringUtils.stringifyException(exception)); to LOG.warn(msg, exception);

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16678-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 02:06:10 2011
Return-Path: <common-issues-return-16678-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7FE096C64
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 02:06:10 +0000 (UTC)
Received: (qmail 68630 invoked by uid 500); 24 Jun 2011 02:06:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68417 invoked by uid 500); 24 Jun 2011 02:06:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68267 invoked by uid 99); 24 Jun 2011 02:06:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 02:06:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 02:06:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AB54542C545
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 02:05:47 +0000 (UTC)
Date: Fri, 24 Jun 2011 02:05:47 +0000 (UTC)
From: "steven zhuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1304349125.35232.1308881147698.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7425) ReflectionUtils.setConf would
 configure the KeyFieldBasedPartitioner twice in Hadoop 0.21.0, when
 KeyFieldBasedPartitioner is an Configurable instance
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

ReflectionUtils.setConf would configure the KeyFieldBasedPartitioner twice in Hadoop 0.21.0, when KeyFieldBasedPartitioner is an Configurable instance
------------------------------------------------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-7425
                 URL: https://issues.apache.org/jira/browse/HADOOP-7425
             Project: Hadoop Common
          Issue Type: Bug
          Components: util
    Affects Versions: 0.21.0
            Reporter: steven zhuang


In  the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number. 


public static void setConf(Object theObject, Configuration conf) {
    if (conf != null) {
      if (theObject instanceof Configurable) {
        ((Configurable) theObject).setConf(conf);
      }
      setJobConf(theObject, conf);
    }
  }


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16679-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 02:20:11 2011
Return-Path: <common-issues-return-16679-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B439C6356
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 02:20:11 +0000 (UTC)
Received: (qmail 79740 invoked by uid 500); 24 Jun 2011 02:20:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79685 invoked by uid 500); 24 Jun 2011 02:20:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79677 invoked by uid 99); 24 Jun 2011 02:20:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 02:20:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 02:20:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 84C1042C92A
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 02:19:47 +0000 (UTC)
Date: Fri, 24 Jun 2011 02:19:47 +0000 (UTC)
From: "Taro L. Saito (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <618064660.35245.1308881987540.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054215#comment-13054215 ] 

Taro L. Saito commented on HADOOP-7206:
---------------------------------------

@Issei @Alejandro
Great. That means as long as using the same classloader (as Hadoop seems to do so), reusing libsnappy.so between hadoop-snappy and snappy-java is no problem. Now, it looks like whether to use libsnappy.so or not is a problem of snappy-java, and I prefer to use libsnappyjava.so (statically linked snappy + JNI code with -fvisibility=hiden option), which can avoid potential API conflict and missing library problems (for some OSes). 

In my experience of developing sqlite-jdbc (http://sqlite-jdbc.googlecode.com/), which uses the same technique to extract .so file at runtime, many people seems to be satisfied with this approach. The problem that can be solved by the runtime library extraction is failures due to misconfiguration (e.g., LD_LIBRARY_PATH, etc.) and library build process (gcc, linker options, etc.) for each OS. For example, I frequently use Windows to develop the code, but run the production code under Linux; no need to switch the library files really helps me a lot. But, looking at HADOOP-7405, current Hadoop's native libraries are not so portable across various OSes. In such a state, motivation for using portable library something like snappy-java might be low.

I don't care which one is used in Hadoop, but the discussion in this thread has been useful for me to improve snappy-java. Thanks!

@Allen
a) OS X (32-bit/64-bit) are already supported.  
b) I need to know os.name and os.arch name system properties that IBM JVM provides. 

Building and embedding non-bundled so file into snappy-java is simple; just do "make".  
As a matter of fact, I do it for 6 types of OS and CPU combinations. And also, by using VMWare FUSION in Mac, all of native libraries currently supported can be compiled in a single machine. Some 64-bit OS can be used to build 32-bit native libraries (e.g., Windows, Linux, etc.) 



> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16680-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 02:36:12 2011
Return-Path: <common-issues-return-16680-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A5CC569C6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 02:36:12 +0000 (UTC)
Received: (qmail 89037 invoked by uid 500); 24 Jun 2011 02:36:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88956 invoked by uid 500); 24 Jun 2011 02:36:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88946 invoked by uid 99); 24 Jun 2011 02:36:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 02:36:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 02:36:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 606F742CCA5
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 02:35:47 +0000 (UTC)
Date: Fri, 24 Jun 2011 02:35:47 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <248050298.35309.1308882947391.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <407429641.30834.1308786767442.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7417) Hadoop Management System
 (Umbrella)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054218#comment-13054218 ] 

Eric Yang commented on HADOOP-7417:
-----------------------------------

HMS is design to deploy existing Hadoop software stack.

> Hadoop Management System (Umbrella)
> -----------------------------------
>
>                 Key: HADOOP-7417
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7417
>             Project: Hadoop Common
>          Issue Type: New Feature
>         Environment: Java 6, Linux
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>
> The primary goal of Hadoop Management System is to build a component around management and deployment of Hadoop related projects. This includes software installation, configuration, application orchestration, deployment automation and monitoring Hadoop.
> Prototype demo source code can be obtained from:
> http://github.com/macroadster/hms

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16681-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 03:36:15 2011
Return-Path: <common-issues-return-16681-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CAAFD67D6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 03:36:15 +0000 (UTC)
Received: (qmail 24304 invoked by uid 500); 24 Jun 2011 03:36:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24070 invoked by uid 500); 24 Jun 2011 03:36:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24025 invoked by uid 99); 24 Jun 2011 03:36:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 03:36:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 03:36:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8887442C98C
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 03:35:47 +0000 (UTC)
Date: Fri, 24 Jun 2011 03:35:47 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2056306331.35399.1308886547556.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-7206:
---------------------------------------

    Attachment: HADOOP-7206revertplusnew.patch

Attached is a patch based on Issay's patch.

* It reverts the current committed HADOOP-7206 & HADDOP-7407
* It uses Snappy native directly
* It adds the JNI bindings to Hadoop native
* Via {{configure.ac}}, if snappy is not available it ignores Snappy JNI bindings
* Snappy lib is looked (by default) at {{/usr/local}}
* Once Snappy is avail by default in different OSes, the default lookup can be changed to {{/usr/}} for automatic detection
* Location of Snappy lib can be altered with {{-Dsnappy.prefix=}} Ant option
* SnappyCodec is defined in {{core-default.xml}}
* If the Snappy JNI bindings and/or Snappy are not present, SnappyCodec warns and continues

IMO this addresses the concerns previously discussed in the JIRA. And it will not break the build in Apache Jenkins machines if Snappy is not install.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16682-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 05:13:16 2011
Return-Path: <common-issues-return-16682-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 812766373
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 05:13:16 +0000 (UTC)
Received: (qmail 72453 invoked by uid 500); 24 Jun 2011 05:13:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72221 invoked by uid 500); 24 Jun 2011 05:13:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72202 invoked by uid 99); 24 Jun 2011 05:13:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 05:13:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 05:13:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 699AD42C1C0
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 05:12:47 +0000 (UTC)
Date: Fri, 24 Jun 2011 05:12:47 +0000 (UTC)
From: "Joep Rottinghuis (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1467102947.35701.1308892367429.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <814884523.34939.1308871007433.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7423) Document topology script
 requirements
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054241#comment-13054241 ] 

Joep Rottinghuis commented on HADOOP-7423:
------------------------------------------

Isn't it true that even with IP addresses used in the topology script, one still has to make sure that Hostname=>IP=>Hostname is always consistent?
We had a situation where some nodes were inserted in each rack all and the numeric part of the names of the data nodes shifted by a few numbers (the domain also changed slighty). Even though the IP addresses stayed the same, for some of the nodes an nslookup resulted in multiple entries. We got some weird behavior.
In a brief glance through the code there seem to be several places where IP is mapped to name and name is mapped to IP, or am I wrong in this?

> Document topology script requirements
> -------------------------------------
>
>                 Key: HADOOP-7423
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7423
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Eli Collins
>              Labels: newbie
>
> The topology script documentation is cluster_setup.xml is unclear. The topology script:
> # Only needs to handle IP addresses (not hostnames)
> # Needs to handle multiple arguments for caching to work effectively
> We should check in an example script or include an example one in the docs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16683-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 05:23:29 2011
Return-Path: <common-issues-return-16683-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 879DB6A06
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 05:23:29 +0000 (UTC)
Received: (qmail 78447 invoked by uid 500); 24 Jun 2011 05:23:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78059 invoked by uid 500); 24 Jun 2011 05:23:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77974 invoked by uid 99); 24 Jun 2011 05:23:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 05:23:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 05:23:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D9E2742C587
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 05:22:47 +0000 (UTC)
Date: Fri, 24 Jun 2011 05:22:47 +0000 (UTC)
From: "Bryant Yao (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <690225560.35713.1308892967888.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-5911) Adding Eclipse "launch" tasks for
 Hadoop daemons
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054242#comment-13054242 ] 

Bryant Yao commented on HADOOP-5911:
------------------------------------

Hi, I am a USC student in Chris Mattmann's class. I wanted to create an Hadoop-specific launch configuration for my class project which will help fix this issue. I was wondering if there was any help you could give me in figuring out the requirements for this issue. Thanks!

> Adding Eclipse "launch" tasks for Hadoop daemons
> ------------------------------------------------
>
>                 Key: HADOOP-5911
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5911
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>         Environment: Eclipse 3.4
>            Reporter: Philip Zeyliger
>            Assignee: Philip Zeyliger
>         Attachments: HADOOP-5911.patch
>
>
> Eclipse has a notion of "run configuration", which encapsulates what's needed to run or debug an application.  I use this quite a bit to start various Hadoop daemons in debug mode, with breakpoints set, to inspect state and what not.
> Next comment will include a patch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16684-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 06:13:21 2011
Return-Path: <common-issues-return-16684-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 02DA06F66
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 06:13:21 +0000 (UTC)
Received: (qmail 17690 invoked by uid 500); 24 Jun 2011 06:13:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16889 invoked by uid 500); 24 Jun 2011 06:13:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16874 invoked by uid 99); 24 Jun 2011 06:13:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 06:13:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 06:13:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6B9D342C4EA
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 06:12:47 +0000 (UTC)
Date: Fri, 24 Jun 2011 06:12:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1897048622.35775.1308895967437.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <639440876.5515.1308103967395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7394) Chinese documentation can't be
 built with Forrest 0.9
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers resolved HADOOP-7394.
------------------------------------

    Resolution: Duplicate

Looks like this is a duplicate of HADOOP-7303.

> Chinese documentation can't be built with Forrest 0.9
> -----------------------------------------------------
>
>                 Key: HADOOP-7394
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7394
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Priority: Minor
>
> Running {{ant cn-docs}} with Forrest 0.8 will work. Running the same thing with Forrest 0.9 prints the following error:
> {noformat}
> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/fop/messaging/MessageHandler
> 	at org.apache.cocoon.serialization.FOPSerializer.configure(FOPSerializer.java:122)
> 	at org.apache.avalon.framework.container.ContainerUtil.configure(ContainerUtil.java:201)
> 	at org.apache.avalon.excalibur.component.DefaultComponentFactory.newInstance(DefaultComponentFactory.java:289)
> 	at org.apache.avalon.excalibur.pool.InstrumentedResourceLimitingPool.newPoolable(InstrumentedResourceLimitingPool.java:655)
> 	at org.apache.avalon.excalibur.pool.InstrumentedResourceLimitingPool.get(InstrumentedResourceLimitingPool.java:371)
> 	at org.apache.avalon.excalibur.component.PoolableComponentHandler.doGet(PoolableComponentHandler.java:198)
> 	at org.apache.avalon.excalibur.component.ComponentHandler.get(ComponentHandler.java:381)
> 	at org.apache.avalon.excalibur.component.ExcaliburComponentSelector.select(ExcaliburComponentSelector.java:215)
> 	at org.apache.cocoon.components.ExtendedComponentSelector.select(ExtendedComponentSelector.java:268)
> 	at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.setSerializer(AbstractProcessingPipeline.java:311)
> 	at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.setSerializer(AbstractCachingProcessingPipeline.java:171)
> 	at org.apache.cocoon.components.treeprocessor.sitemap.SerializeNode.invoke(SerializeNode.java:120)
> 	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)
> 	at org.apache.cocoon.components.treeprocessor.sitemap.SelectNode.invoke(SelectNode.java:103)
> 	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:47)
> 	at org.apache.cocoon.components.treeprocessor.sitemap.PreparableMatchNode.invoke(PreparableMatchNode.java:131)
> 	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)
> 	at org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:143)
> 	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)
> 	at org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:93)
> 	at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:235)
> 	at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:177)
> 	at org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:254)
> 	at org.apache.cocoon.components.treeprocessor.sitemap.MountNode.invoke(MountNode.java:118)
> 	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)
> 	at org.apache.cocoon.components.treeprocessor.sitemap.SelectNode.invoke(SelectNode.java:98)
> 	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)
> 	at org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:143)
> 	at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)
> 	at org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:93)
> 	at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:235)
> 	at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:177)
> 	at org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:254)
> 	at org.apache.cocoon.Cocoon.process(Cocoon.java:699)
> 	at org.apache.cocoon.bean.CocoonWrapper.getPage(CocoonWrapper.java:514)
> 	at org.apache.cocoon.bean.CocoonBean.processTarget(CocoonBean.java:499)
> 	at org.apache.cocoon.bean.CocoonBean.process(CocoonBean.java:356)
> 	at org.apache.cocoon.Main.main(Main.java:321)
> Caused by: java.lang.ClassNotFoundException: org.apache.fop.messaging.MessageHandler
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> 	... 38 more
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16685-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 12:52:15 2011
Return-Path: <common-issues-return-16685-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 263356571
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 12:52:15 +0000 (UTC)
Received: (qmail 17555 invoked by uid 500); 24 Jun 2011 12:52:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16599 invoked by uid 500); 24 Jun 2011 12:52:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16419 invoked by uid 99); 24 Jun 2011 12:52:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 12:52:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 12:52:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CC15A42D9CD
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 12:51:49 +0000 (UTC)
Date: Fri, 24 Jun 2011 12:51:49 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1310013439.36547.1308919909832.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054418#comment-13054418 ] 

Tom White commented on HADOOP-7206:
-----------------------------------

TestCodec passes for me both when snappy is not installed and when snappy is installed and native compilation is enabled. +1

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16686-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 17:06:10 2011
Return-Path: <common-issues-return-16686-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 099246091
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 17:06:10 +0000 (UTC)
Received: (qmail 38210 invoked by uid 500); 24 Jun 2011 17:06:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38136 invoked by uid 500); 24 Jun 2011 17:06:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38127 invoked by uid 99); 24 Jun 2011 17:06:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 17:06:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 17:06:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D58C242DF04
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 17:05:47 +0000 (UTC)
Date: Fri, 24 Jun 2011 17:05:47 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1486205511.37237.1308935147871.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-7206:
---------------------------------------

    Attachment: HADOOP-7206revertplusnew-b.patch

The updated patch allows the Ant build to include the snappy SO next to the hadoop SO.

By default this option is disabled, to enabled invoke Ant with the '-Dbundle.snappy=true' option.


> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206revertplusnew-b.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16687-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 17:06:13 2011
Return-Path: <common-issues-return-16687-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5263E60AE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 17:06:13 +0000 (UTC)
Received: (qmail 38642 invoked by uid 500); 24 Jun 2011 17:06:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38600 invoked by uid 500); 24 Jun 2011 17:06:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38592 invoked by uid 99); 24 Jun 2011 17:06:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 17:06:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 17:06:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CD51C42DF4A
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 17:05:49 +0000 (UTC)
Date: Fri, 24 Jun 2011 17:05:49 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1949635982.37290.1308935149838.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-7206:
---------------------------------------

    Status: Patch Available  (was: Reopened)

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206revertplusnew-b.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16688-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 17:20:11 2011
Return-Path: <common-issues-return-16688-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1C54E6A79
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 17:20:11 +0000 (UTC)
Received: (qmail 65559 invoked by uid 500); 24 Jun 2011 17:20:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65450 invoked by uid 500); 24 Jun 2011 17:20:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65430 invoked by uid 99); 24 Jun 2011 17:20:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 17:20:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 17:20:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8CB6B42E597
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 17:19:49 +0000 (UTC)
Date: Fri, 24 Jun 2011 17:19:49 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <762231158.37424.1308935989573.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054561#comment-13054561 ] 

Eli Collins commented on HADOOP-7206:
-------------------------------------

Hey Alejandro - mind if I revert the previous two patches and then we can apply the new one? Would be nice to have this go in a single commit.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206revertplusnew-b.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16689-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 17:24:09 2011
Return-Path: <common-issues-return-16689-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7323F6CEE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 17:24:09 +0000 (UTC)
Received: (qmail 72665 invoked by uid 500); 24 Jun 2011 17:24:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72589 invoked by uid 500); 24 Jun 2011 17:24:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72581 invoked by uid 99); 24 Jun 2011 17:24:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 17:24:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 17:24:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D7BB742E712
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 17:23:47 +0000 (UTC)
Date: Fri, 24 Jun 2011 17:23:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <399025460.37464.1308936227880.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054566#comment-13054566 ] 

Hadoop QA commented on HADOOP-7206:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12483739/HADOOP-7206revertplusnew-b.patch
  against trunk revision 1139123.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    -1 release audit.  The applied patch generated 2 release audit warnings (more than the trunk's current 1 warnings).

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/669//testReport/
Release audit warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/669//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/669//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/669//console

This message is automatically generated.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206revertplusnew-b.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16690-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 17:40:10 2011
Return-Path: <common-issues-return-16690-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D342964FE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 17:40:10 +0000 (UTC)
Received: (qmail 90978 invoked by uid 500); 24 Jun 2011 17:40:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90940 invoked by uid 500); 24 Jun 2011 17:40:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90932 invoked by uid 99); 24 Jun 2011 17:40:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 17:40:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 17:40:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7E5FE42EED7
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 17:39:49 +0000 (UTC)
Date: Fri, 24 Jun 2011 17:39:49 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <442235887.37611.1308937189514.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054572#comment-13054572 ] 

Alejandro Abdelnur commented on HADOOP-7206:
--------------------------------------------

Eli, I'm good. I'll prepare that patch and I'll add the missing AL

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206revertplusnew-b.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16691-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 17:40:11 2011
Return-Path: <common-issues-return-16691-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 07DEB650C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 17:40:11 +0000 (UTC)
Received: (qmail 91174 invoked by uid 500); 24 Jun 2011 17:40:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91111 invoked by uid 500); 24 Jun 2011 17:40:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90985 invoked by uid 99); 24 Jun 2011 17:40:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 17:40:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 17:40:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9972642EEDA
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 17:39:49 +0000 (UTC)
Date: Fri, 24 Jun 2011 17:39:49 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1895887268.37614.1308937189625.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-7206:
---------------------------------------

    Status: Open  (was: Patch Available)

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206revertplusnew-b.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16692-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 17:44:16 2011
Return-Path: <common-issues-return-16692-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 92A5D65AB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 17:44:16 +0000 (UTC)
Received: (qmail 95908 invoked by uid 500); 24 Jun 2011 17:44:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95729 invoked by uid 500); 24 Jun 2011 17:44:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95700 invoked by uid 99); 24 Jun 2011 17:44:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 17:44:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 17:44:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8A78042D11D
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 17:43:49 +0000 (UTC)
Date: Fri, 24 Jun 2011 17:43:49 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <906341420.37719.1308937429563.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054574#comment-13054574 ] 

Eli Collins commented on HADOOP-7206:
-------------------------------------

Great. I've reverted 7206 and 7407, you can attach a new patch now.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206revertplusnew-b.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16693-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 17:54:09 2011
Return-Path: <common-issues-return-16693-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 28D426E72
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 17:54:09 +0000 (UTC)
Received: (qmail 7108 invoked by uid 500); 24 Jun 2011 17:54:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7071 invoked by uid 500); 24 Jun 2011 17:54:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7049 invoked by uid 99); 24 Jun 2011 17:54:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 17:54:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 17:54:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9833942D5D0
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 17:53:47 +0000 (UTC)
Date: Fri, 24 Jun 2011 17:53:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <776968642.37737.1308938027619.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <680679068.21358.1308591287861.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Reopened] (HADOOP-7407) Snappy integration breaks HDFS
 build.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE reopened HADOOP-7407:
--------------------------------------------


> Snappy integration breaks HDFS build.
> -------------------------------------
>
>                 Key: HADOOP-7407
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7407
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Ivan Kelly
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7407.patch
>
>
> The common/ivy/hadoop-common-template.xml submitted with 7206 has a typo which breaks anything that depends on the hadoop-common maven package.
> Instead of <artifactId>java-snappy</artifactId>, you should have <artifactId>snappy-java</artifactId>
> [ivy:resolve] downloading https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.23.0-SNAPSHOT/hadoop-common-0.23.0-20110620.163810-177.jar ...
> [ivy:resolve] .......................
> [ivy:resolve] ..................................
> [ivy:resolve] ...........................................
> [ivy:resolve] ...............................................................
> [ivy:resolve] ................................................ (1631kB)
> [ivy:resolve] .. (0kB)
> [ivy:resolve] 	[SUCCESSFUL ] org.apache.hadoop#hadoop-common;0.23.0-SNAPSHOT!hadoop-common.jar (8441ms)
> [ivy:resolve] 
> [ivy:resolve] :: problems summary ::
> [ivy:resolve] :::: WARNINGS
> [ivy:resolve] 		module not found: org.xerial.snappy#java-snappy;1.0.3-rc2
> [ivy:resolve] 	==== apache-snapshot: tried
> [ivy:resolve] 	  https://repository.apache.org/content/repositories/snapshots/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.pom
> [ivy:resolve] 	  -- artifact org.xerial.snappy#java-snappy;1.0.3-rc2!java-snappy.jar:
> [ivy:resolve] 	  https://repository.apache.org/content/repositories/snapshots/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.jar
> [ivy:resolve] 	==== maven2: tried
> [ivy:resolve] 	  http://repo1.maven.org/maven2/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.pom
> [ivy:resolve] 	  -- artifact org.xerial.snappy#java-snappy;1.0.3-rc2!java-snappy.jar:
> [ivy:resolve] 	  http://repo1.maven.org/maven2/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.jar
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 		::          UNRESOLVED DEPENDENCIES         ::
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 		:: org.xerial.snappy#java-snappy;1.0.3-rc2: not found
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16694-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 17:54:09 2011
Return-Path: <common-issues-return-16694-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8263A6E7C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 17:54:09 +0000 (UTC)
Received: (qmail 7378 invoked by uid 500); 24 Jun 2011 17:54:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7341 invoked by uid 500); 24 Jun 2011 17:54:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7332 invoked by uid 99); 24 Jun 2011 17:54:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 17:54:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 17:54:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DE0E942D5E4
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 17:53:47 +0000 (UTC)
Date: Fri, 24 Jun 2011 17:53:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <467656216.37745.1308938027906.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <680679068.21358.1308591287861.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7407) Snappy integration breaks HDFS
 build.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE resolved HADOOP-7407.
--------------------------------------------

    Resolution: Not A Problem

It turns out that we have to revert HADOOP-7206 and this (Thanks Eli).  So this is not a problem anymore.

> Snappy integration breaks HDFS build.
> -------------------------------------
>
>                 Key: HADOOP-7407
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7407
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Ivan Kelly
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7407.patch
>
>
> The common/ivy/hadoop-common-template.xml submitted with 7206 has a typo which breaks anything that depends on the hadoop-common maven package.
> Instead of <artifactId>java-snappy</artifactId>, you should have <artifactId>snappy-java</artifactId>
> [ivy:resolve] downloading https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.23.0-SNAPSHOT/hadoop-common-0.23.0-20110620.163810-177.jar ...
> [ivy:resolve] .......................
> [ivy:resolve] ..................................
> [ivy:resolve] ...........................................
> [ivy:resolve] ...............................................................
> [ivy:resolve] ................................................ (1631kB)
> [ivy:resolve] .. (0kB)
> [ivy:resolve] 	[SUCCESSFUL ] org.apache.hadoop#hadoop-common;0.23.0-SNAPSHOT!hadoop-common.jar (8441ms)
> [ivy:resolve] 
> [ivy:resolve] :: problems summary ::
> [ivy:resolve] :::: WARNINGS
> [ivy:resolve] 		module not found: org.xerial.snappy#java-snappy;1.0.3-rc2
> [ivy:resolve] 	==== apache-snapshot: tried
> [ivy:resolve] 	  https://repository.apache.org/content/repositories/snapshots/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.pom
> [ivy:resolve] 	  -- artifact org.xerial.snappy#java-snappy;1.0.3-rc2!java-snappy.jar:
> [ivy:resolve] 	  https://repository.apache.org/content/repositories/snapshots/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.jar
> [ivy:resolve] 	==== maven2: tried
> [ivy:resolve] 	  http://repo1.maven.org/maven2/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.pom
> [ivy:resolve] 	  -- artifact org.xerial.snappy#java-snappy;1.0.3-rc2!java-snappy.jar:
> [ivy:resolve] 	  http://repo1.maven.org/maven2/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.jar
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 		::          UNRESOLVED DEPENDENCIES         ::
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 		:: org.xerial.snappy#java-snappy;1.0.3-rc2: not found
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16695-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 17:58:14 2011
Return-Path: <common-issues-return-16695-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1204460E6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 17:58:14 +0000 (UTC)
Received: (qmail 18518 invoked by uid 500); 24 Jun 2011 17:58:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18485 invoked by uid 500); 24 Jun 2011 17:58:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18477 invoked by uid 99); 24 Jun 2011 17:58:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 17:58:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 17:58:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9487842D89B
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 17:57:50 +0000 (UTC)
Date: Fri, 24 Jun 2011 17:57:50 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <192369022.37780.1308938270605.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7206:
-------------------------------------------

    Fix Version/s:     (was: 0.23.0)
     Hadoop Flags:   (was: [Reviewed])

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206revertplusnew-b.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16696-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 18:08:11 2011
Return-Path: <common-issues-return-16696-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3ECFB68D1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 18:08:11 +0000 (UTC)
Received: (qmail 32826 invoked by uid 500); 24 Jun 2011 18:08:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32803 invoked by uid 500); 24 Jun 2011 18:08:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32793 invoked by uid 99); 24 Jun 2011 18:08:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 18:08:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 18:08:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D321942DED1
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 18:07:49 +0000 (UTC)
Date: Fri, 24 Jun 2011 18:07:49 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2018335596.37913.1308938869861.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-7206:
---------------------------------------

    Attachment: HADOOP-7206new.patch

It works on top of the reverted trunk.

Added missing AL

Replaced System.err.println() with log messages

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206new.patch, HADOOP-7206revertplusnew-b.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16697-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 18:08:13 2011
Return-Path: <common-issues-return-16697-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9D5236913
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 18:08:13 +0000 (UTC)
Received: (qmail 34466 invoked by uid 500); 24 Jun 2011 18:08:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34443 invoked by uid 500); 24 Jun 2011 18:08:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34403 invoked by uid 99); 24 Jun 2011 18:08:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 18:08:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 18:08:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D9CBA42DF0F
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 18:07:51 +0000 (UTC)
Date: Fri, 24 Jun 2011 18:07:51 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <62715422.37966.1308938871888.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-7206:
---------------------------------------

    Status: Patch Available  (was: Open)

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206new.patch, HADOOP-7206revertplusnew-b.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16698-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 18:14:09 2011
Return-Path: <common-issues-return-16698-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 680646A9D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 18:14:09 +0000 (UTC)
Received: (qmail 47860 invoked by uid 500); 24 Jun 2011 18:14:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47823 invoked by uid 500); 24 Jun 2011 18:14:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47722 invoked by uid 99); 24 Jun 2011 18:14:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 18:14:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 18:14:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 04E2D42E2BD
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 18:13:48 +0000 (UTC)
Date: Fri, 24 Jun 2011 18:13:48 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <863327185.38019.1308939228016.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054598#comment-13054598 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7206:
------------------------------------------------

Hi Alejandro, could you add javadoc for the new public method/fields and @Override for the overriding methods?

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206new.patch, HADOOP-7206revertplusnew-b.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16699-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 18:24:10 2011
Return-Path: <common-issues-return-16699-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E385962F4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 18:24:10 +0000 (UTC)
Received: (qmail 65895 invoked by uid 500); 24 Jun 2011 18:24:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65856 invoked by uid 500); 24 Jun 2011 18:24:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65845 invoked by uid 99); 24 Jun 2011 18:24:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 18:24:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 18:24:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7329B42E8B8
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 18:23:49 +0000 (UTC)
Date: Fri, 24 Jun 2011 18:23:49 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <854269091.38161.1308939829468.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054609#comment-13054609 ] 

Hadoop QA commented on HADOOP-7206:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12483753/HADOOP-7206new.patch
  against trunk revision 1139387.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/670//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/670//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/670//console

This message is automatically generated.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206new.patch, HADOOP-7206revertplusnew-b.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16700-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 18:42:08 2011
Return-Path: <common-issues-return-16700-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BFBC96EE2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 18:42:08 +0000 (UTC)
Received: (qmail 96289 invoked by uid 500); 24 Jun 2011 18:42:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96252 invoked by uid 500); 24 Jun 2011 18:42:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96238 invoked by uid 99); 24 Jun 2011 18:42:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 18:42:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 18:42:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6FA4742E009
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 18:41:47 +0000 (UTC)
Date: Fri, 24 Jun 2011 18:41:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1070636631.38224.1308940907454.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1760011225.22288.1308599807546.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7408) Add javadoc for SnappyCodec
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7408:
-------------------------------------------

    Status: Open  (was: Patch Available)

> Add javadoc for SnappyCodec
> ---------------------------
>
>                 Key: HADOOP-7408
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7408
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: T Jake Luciani
>            Assignee: T Jake Luciani
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7408.patch, v1-HADOOP-7408-add-snappy-javadoc.txt
>
>
> HADOOP-7206 failed to include a javadoc for public methods.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16701-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 18:42:11 2011
Return-Path: <common-issues-return-16701-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0AD016F17
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 18:42:11 +0000 (UTC)
Received: (qmail 97859 invoked by uid 500); 24 Jun 2011 18:42:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97801 invoked by uid 500); 24 Jun 2011 18:42:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97764 invoked by uid 99); 24 Jun 2011 18:42:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 18:42:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 18:42:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 89EBA42E00D
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 18:41:47 +0000 (UTC)
Date: Fri, 24 Jun 2011 18:41:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <107862294.38228.1308940907562.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1760011225.22288.1308599807546.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7408) Add javadoc for SnappyCodec
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE resolved HADOOP-7408.
--------------------------------------------

       Resolution: Not A Problem
    Fix Version/s:     (was: 0.23.0)

It turns out that we have to revert HADOOP-7206. So this is not a problem anymore.

> Add javadoc for SnappyCodec
> ---------------------------
>
>                 Key: HADOOP-7408
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7408
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: T Jake Luciani
>            Assignee: T Jake Luciani
>            Priority: Trivial
>         Attachments: HADOOP-7408.patch, v1-HADOOP-7408-add-snappy-javadoc.txt
>
>
> HADOOP-7206 failed to include a javadoc for public methods.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16702-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 18:54:08 2011
Return-Path: <common-issues-return-16702-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CB46966EC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 18:54:08 +0000 (UTC)
Received: (qmail 12809 invoked by uid 500); 24 Jun 2011 18:54:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12775 invoked by uid 500); 24 Jun 2011 18:54:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12767 invoked by uid 99); 24 Jun 2011 18:54:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 18:54:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 18:54:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 68BEB42E424
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 18:53:47 +0000 (UTC)
Date: Fri, 24 Jun 2011 18:53:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <991846466.38245.1308941627425.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1443063883.27446.1308722147425.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7414) path to setWar() in HttpServer is
 not setup properly for resources from Jar URLs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054625#comment-13054625 ] 

Luke Lu commented on HADOOP-7414:
---------------------------------

bq. What is trying to be achieved here?

The problem is not about war. It's about picking up the compiled jsp classes from the webapps directory inside a jar. Without this patch the directory must be a fs directory and not a resource inside a jar, which Ben wants. This is useful to create a single jar installation of hadoop for easy deployment (in Ben's case, show-off :))

As for the patch, I think it'll be prudent to check if the name endsWith("/"), as there might exist code trying to compensate the same problem and that testing whether "//" works with jetty's different versions seems a lot more work.

> path to setWar() in HttpServer is not setup properly for resources from Jar URLs
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-7414
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7414
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Benjamin Reed
>         Attachments: HADOOP-7414.patch
>
>
> the path in setWar() needs to end in a '/' for jetty to properly process resources from Jar URLs. by adding a '/' at the end the webapps can run out of a Jar file instead of unzipping into a file system.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16703-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 21:16:13 2011
Return-Path: <common-issues-return-16703-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2066E6DFC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 21:16:13 +0000 (UTC)
Received: (qmail 25334 invoked by uid 500); 24 Jun 2011 21:16:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24788 invoked by uid 500); 24 Jun 2011 21:16:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24772 invoked by uid 99); 24 Jun 2011 21:16:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 21:16:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 21:16:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8EDE942E10C
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 21:15:47 +0000 (UTC)
Date: Fri, 24 Jun 2011 21:15:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <373123281.38525.1308950147582.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1354187382.13188.1307764078831.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7380) Common portion of HDFS-1973
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7380:
-----------------------------------

    Attachment: hadoop-7380.2.patch

Here's an updated patch which I believe is ready for commit. I mostly just rebased against trunk and added comments since the last one.

Eli, please let me know what you think.

> Common portion of HDFS-1973
> ---------------------------
>
>                 Key: HADOOP-7380
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7380
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7380.0.patch, hadoop-7380.1.patch, hadoop-7380.2.patch
>
>
> Implementing client failover will likely require changes to {{o.a.h.io.ipc}} and/or {{o.a.h.io.retry}}. This JIRA is to track those changes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16704-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 21:18:10 2011
Return-Path: <common-issues-return-16704-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1539768F0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 21:18:10 +0000 (UTC)
Received: (qmail 31068 invoked by uid 500); 24 Jun 2011 21:18:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30929 invoked by uid 500); 24 Jun 2011 21:18:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30921 invoked by uid 99); 24 Jun 2011 21:18:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 21:18:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 21:18:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 81E0342E1BB
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 21:17:48 +0000 (UTC)
Date: Fri, 24 Jun 2011 21:17:48 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <825967375.38532.1308950268528.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-5911) Adding Eclipse "launch" tasks for
 Hadoop daemons
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-5911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-5911:
--------------------------------

    Assignee:     (was: Philip Zeyliger)
      Labels: newbie  (was: )

Hey Bryant,

Welcome!  Phil is not working on this, I'd check out his patch and see if you can get it working on trunk.

Thanks,
Eli

> Adding Eclipse "launch" tasks for Hadoop daemons
> ------------------------------------------------
>
>                 Key: HADOOP-5911
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5911
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>         Environment: Eclipse 3.4
>            Reporter: Philip Zeyliger
>              Labels: newbie
>         Attachments: HADOOP-5911.patch
>
>
> Eclipse has a notion of "run configuration", which encapsulates what's needed to run or debug an application.  I use this quite a bit to start various Hadoop daemons in debug mode, with breakpoints set, to inspect state and what not.
> Next comment will include a patch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16705-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 21:24:12 2011
Return-Path: <common-issues-return-16705-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EEEFD67AD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 21:24:11 +0000 (UTC)
Received: (qmail 41665 invoked by uid 500); 24 Jun 2011 21:24:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41570 invoked by uid 500); 24 Jun 2011 21:24:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41546 invoked by uid 99); 24 Jun 2011 21:24:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 21:24:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 21:24:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C660642E3D8
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 21:23:47 +0000 (UTC)
Date: Fri, 24 Jun 2011 21:23:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1502310367.38551.1308950627809.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <407429641.30834.1308786767442.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7417) Hadoop Management System
 (Umbrella)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054683#comment-13054683 ] 

Eli Collins commented on HADOOP-7417:
-------------------------------------

Hey Eric, HMS is a separate project headed to the incubator (http://wiki.apache.org/incubator/HMSProposal) right?



> Hadoop Management System (Umbrella)
> -----------------------------------
>
>                 Key: HADOOP-7417
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7417
>             Project: Hadoop Common
>          Issue Type: New Feature
>         Environment: Java 6, Linux
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>
> The primary goal of Hadoop Management System is to build a component around management and deployment of Hadoop related projects. This includes software installation, configuration, application orchestration, deployment automation and monitoring Hadoop.
> Prototype demo source code can be obtained from:
> http://github.com/macroadster/hms

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16706-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 23:09:12 2011
Return-Path: <common-issues-return-16706-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 972076A0F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 23:09:12 +0000 (UTC)
Received: (qmail 38783 invoked by uid 500); 24 Jun 2011 23:09:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38570 invoked by uid 500); 24 Jun 2011 23:09:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38465 invoked by uid 99); 24 Jun 2011 23:09:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 23:09:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 23:09:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E6CB142EAED
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 23:08:47 +0000 (UTC)
Date: Fri, 24 Jun 2011 23:08:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1815196256.38852.1308956927942.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054736#comment-13054736 ] 

Eli Collins commented on HADOOP-7206:
-------------------------------------

Looks great!  Thanks for coming up with an implementation that incorporates all the discussion. 

Testing?

Modulo Nicholas' good feedback, minor review feedback follows

* Why is property environment="env" needed? Can we remove that same setting from the findbugs target now that it's global? 
* Indicate is -> Indicate if
* java.library.path should use ${snappy.lib} vs ${snappy.prefix}/lib
* #if shoudldn't be indented in org_apache_hadoop_io_compress_snappy.h, and Snappy*.c, ie everything can be shifted left one stop.
* 1st echo in packageNativeHadoop.sh is missindented
* "io.compression.codec.snappy.buffersize" should be defined in CommonConfigurationKeys
* In LoadSnappy I don't think we need to log if hadoopNativeAvailable because that will is indicated by the output of NativeCodeLoader already

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206new.patch, HADOOP-7206revertplusnew-b.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16707-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 23:25:11 2011
Return-Path: <common-issues-return-16707-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2FE686171
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 23:25:11 +0000 (UTC)
Received: (qmail 51214 invoked by uid 500); 24 Jun 2011 23:25:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51155 invoked by uid 500); 24 Jun 2011 23:25:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51147 invoked by uid 99); 24 Jun 2011 23:25:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 23:25:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 23:25:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5DC3142E1F1
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 23:24:47 +0000 (UTC)
Date: Fri, 24 Jun 2011 23:24:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1117745742.38954.1308957887380.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1479142716.7833.1307643118867.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7372) Remove ref of 20.3 release from
 branch-0.20 CHANGES.txt
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7372:
--------------------------------

    Attachment: hadoop-7372-2.patch

Patch attached, reverts 1055647 and moves the one addition in CHANGES.txt from the 20.4 section to the 20.3 section.

> Remove ref of 20.3 release from branch-0.20 CHANGES.txt
> -------------------------------------------------------
>
>                 Key: HADOOP-7372
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7372
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.20.3
>
>         Attachments: hadoop-7372-1.patch, hadoop-7372-2.patch
>
>
> CHANGES.txt on branch-0.20 claims there was a 0.20.3 release on 1/5. There has not been a 0.20.3 release.
> {noformat}
> Release 0.20.4 - Unreleased
> ...
> Release 0.20.3 - 2011-1-5
> {noformat}
> We should update this to indicate 0.20. is unreleased.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16708-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 23:29:09 2011
Return-Path: <common-issues-return-16708-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CDE626233
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 23:29:09 +0000 (UTC)
Received: (qmail 53975 invoked by uid 500); 24 Jun 2011 23:29:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53917 invoked by uid 500); 24 Jun 2011 23:29:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53902 invoked by uid 99); 24 Jun 2011 23:29:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 23:29:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 23:29:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D397342E326
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 23:28:47 +0000 (UTC)
Date: Fri, 24 Jun 2011 23:28:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1839076211.38970.1308958127863.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1479142716.7833.1307643118867.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7372) Remove ref of 20.3 release from
 branch-0.20 CHANGES.txt
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054751#comment-13054751 ] 

Todd Lipcon commented on HADOOP-7372:
-------------------------------------

+1 seems reasonable to me.

> Remove ref of 20.3 release from branch-0.20 CHANGES.txt
> -------------------------------------------------------
>
>                 Key: HADOOP-7372
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7372
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.20.3
>
>         Attachments: hadoop-7372-1.patch, hadoop-7372-2.patch
>
>
> CHANGES.txt on branch-0.20 claims there was a 0.20.3 release on 1/5. There has not been a 0.20.3 release.
> {noformat}
> Release 0.20.4 - Unreleased
> ...
> Release 0.20.3 - 2011-1-5
> {noformat}
> We should update this to indicate 0.20. is unreleased.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16709-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 23:29:11 2011
Return-Path: <common-issues-return-16709-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7420B624A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 23:29:11 +0000 (UTC)
Received: (qmail 54280 invoked by uid 500); 24 Jun 2011 23:29:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54201 invoked by uid 500); 24 Jun 2011 23:29:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54192 invoked by uid 99); 24 Jun 2011 23:29:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 23:29:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 23:29:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CD21D42E325
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 23:28:47 +0000 (UTC)
Date: Fri, 24 Jun 2011 23:28:47 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1455318042.38969.1308958127837.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <407429641.30834.1308786767442.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7417) Hadoop Management System
 (Umbrella)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054750#comment-13054750 ] 

Eric Yang commented on HADOOP-7417:
-----------------------------------

Hi Eli, I am uncertain where is the best place for this project.  This is a customized manage and deployment system for Hadoop software stack.  Hence, it could be inside Hadoop or in the incubator.  Some people have expressed interests for this to be in the contrib project, and others think this should be an incubator project.  I am fine either way, but I did drafted the incubator proposal.  What do people think?  Let me post a architecture diagram and design.  Hadoop community can decide the proper course of action.

> Hadoop Management System (Umbrella)
> -----------------------------------
>
>                 Key: HADOOP-7417
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7417
>             Project: Hadoop Common
>          Issue Type: New Feature
>         Environment: Java 6, Linux
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>
> The primary goal of Hadoop Management System is to build a component around management and deployment of Hadoop related projects. This includes software installation, configuration, application orchestration, deployment automation and monitoring Hadoop.
> Prototype demo source code can be obtained from:
> http://github.com/macroadster/hms

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16710-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 23:33:09 2011
Return-Path: <common-issues-return-16710-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8BB29619D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 23:33:09 +0000 (UTC)
Received: (qmail 55814 invoked by uid 500); 24 Jun 2011 23:33:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55769 invoked by uid 500); 24 Jun 2011 23:33:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55761 invoked by uid 99); 24 Jun 2011 23:33:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 23:33:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 23:33:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7D3B342E40C
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 23:32:47 +0000 (UTC)
Date: Fri, 24 Jun 2011 23:32:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2057849156.38976.1308958367509.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7418) support for multiple slashes in
 the path separator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054752#comment-13054752 ] 

Aaron T. Myers commented on HADOOP-7418:
----------------------------------------

Hey Andrew, patch looks pretty good to me. One question:

>From manually inspecting the code, I suspect the {{Path}} class will still not correctly handle the case of multiple trailing slashes. Would you mind adding a test case to test some path with a silly number (> 5) of trailing slashes?

You might also consider running the full HDFS and MR test suites with this patch installed, as those obviously exercise all of the {{Path}} code quite a bit. See the "Changes that span projects" section of this page for instructions about how to get HDFS and MR to compile against your patched Common: http://wiki.apache.org/hadoop/HowToContribute

> support for multiple slashes in the path separator
> --------------------------------------------------
>
>                 Key: HADOOP-7418
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7418
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>         Environment: Linux running JDK 1.6
>            Reporter: Sudharsan Sampath
>            Assignee: Andrew Look
>            Priority: Minor
>              Labels: newbie
>             Fix For: 0.23.0
>
>         Attachments: HDFS-1460.txt, HDFS-1460.txt
>
>
> the parsing of the input path string to identify the uri authority conflicts with the file system paths. For instance the following is a valid path in both the linux file system and the hdfs.
> //user/directory1//directory2.
> While this works perfectly fine in the command line for manipulating hdfs, the same fails when specified as the input path for a mapper class with the following expcetion.
> Exception in thread "main" java.net.UnknownHostException: unknown host: user
>         at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:195)
> as the org.apache.hadoop.fs.Path class assumes the string that follows the '//' to be an uri authority

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16711-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 23:35:11 2011
Return-Path: <common-issues-return-16711-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4E746683F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 23:35:11 +0000 (UTC)
Received: (qmail 57060 invoked by uid 500); 24 Jun 2011 23:35:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57017 invoked by uid 500); 24 Jun 2011 23:35:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57009 invoked by uid 99); 24 Jun 2011 23:35:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 23:35:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 23:35:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 82B2242E514
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 23:34:47 +0000 (UTC)
Date: Fri, 24 Jun 2011 23:34:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <265705155.38982.1308958487532.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1479142716.7833.1307643118867.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7372) Remove ref of 20.3 release from
 branch-0.20 CHANGES.txt
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7372:
--------------------------------

    Hadoop Flags: [Reviewed]
          Status: Patch Available  (was: Open)

Thanks Todd. I've committed this and updated the two jiras w 0.20.4 fix version to 0.20.3. 

> Remove ref of 20.3 release from branch-0.20 CHANGES.txt
> -------------------------------------------------------
>
>                 Key: HADOOP-7372
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7372
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.20.3
>
>         Attachments: hadoop-7372-1.patch, hadoop-7372-2.patch
>
>
> CHANGES.txt on branch-0.20 claims there was a 0.20.3 release on 1/5. There has not been a 0.20.3 release.
> {noformat}
> Release 0.20.4 - Unreleased
> ...
> Release 0.20.3 - 2011-1-5
> {noformat}
> We should update this to indicate 0.20. is unreleased.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16712-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 23:35:11 2011
Return-Path: <common-issues-return-16712-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8F9A66854
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 23:35:11 +0000 (UTC)
Received: (qmail 57139 invoked by uid 500); 24 Jun 2011 23:35:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57036 invoked by uid 500); 24 Jun 2011 23:35:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57018 invoked by uid 99); 24 Jun 2011 23:35:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 23:35:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 23:35:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 99A4B42E517
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 23:34:47 +0000 (UTC)
Date: Fri, 24 Jun 2011 23:34:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <830941199.38984.1308958487626.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1479142716.7833.1307643118867.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7372) Remove ref of 20.3 release from
 branch-0.20 CHANGES.txt
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7372:
--------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

> Remove ref of 20.3 release from branch-0.20 CHANGES.txt
> -------------------------------------------------------
>
>                 Key: HADOOP-7372
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7372
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.20.3
>
>         Attachments: hadoop-7372-1.patch, hadoop-7372-2.patch
>
>
> CHANGES.txt on branch-0.20 claims there was a 0.20.3 release on 1/5. There has not been a 0.20.3 release.
> {noformat}
> Release 0.20.4 - Unreleased
> ...
> Release 0.20.3 - 2011-1-5
> {noformat}
> We should update this to indicate 0.20. is unreleased.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16713-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri Jun 24 23:51:09 2011
Return-Path: <common-issues-return-16713-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A6FCF6F31
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 24 Jun 2011 23:51:09 +0000 (UTC)
Received: (qmail 66282 invoked by uid 500); 24 Jun 2011 23:51:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66250 invoked by uid 500); 24 Jun 2011 23:51:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66242 invoked by uid 99); 24 Jun 2011 23:51:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 23:51:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 24 Jun 2011 23:51:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8111742E9AB
	for <common-issues@hadoop.apache.org>; Fri, 24 Jun 2011 23:50:47 +0000 (UTC)
Date: Fri, 24 Jun 2011 23:50:47 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <549943764.39015.1308959447525.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-7206:
---------------------------------------

    Attachment: HADOOP-7206new-b.patch

Incorporating Nicholas's & Eli's comments.

Thxs.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206new-b.patch, HADOOP-7206new.patch, HADOOP-7206revertplusnew-b.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16714-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 00:03:12 2011
Return-Path: <common-issues-return-16714-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7AD166F0E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 25 Jun 2011 00:03:12 +0000 (UTC)
Received: (qmail 71299 invoked by uid 500); 25 Jun 2011 00:03:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70961 invoked by uid 500); 25 Jun 2011 00:03:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70951 invoked by uid 99); 25 Jun 2011 00:03:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 00:03:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 00:03:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D0B2542ED38
	for <common-issues@hadoop.apache.org>; Sat, 25 Jun 2011 00:02:47 +0000 (UTC)
Date: Sat, 25 Jun 2011 00:02:47 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1075703539.39087.1308960167851.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054765#comment-13054765 ] 

Alejandro Abdelnur commented on HADOOP-7206:
--------------------------------------------

Regarding testing, i've tested it in Ubuntu, with and without snappy SO present.

The <property environment="env"/> makes the ENV variables available as properties with prefix "env.".

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206new-b.patch, HADOOP-7206new.patch, HADOOP-7206revertplusnew-b.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16715-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 00:05:11 2011
Return-Path: <common-issues-return-16715-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A0CDC6688
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 25 Jun 2011 00:05:11 +0000 (UTC)
Received: (qmail 72203 invoked by uid 500); 25 Jun 2011 00:05:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72149 invoked by uid 500); 25 Jun 2011 00:05:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72141 invoked by uid 99); 25 Jun 2011 00:05:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 00:05:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 00:05:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D226442EE93
	for <common-issues@hadoop.apache.org>; Sat, 25 Jun 2011 00:04:49 +0000 (UTC)
Date: Sat, 25 Jun 2011 00:04:49 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1407052132.39191.1308960289857.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054766#comment-13054766 ] 

Eli Collins commented on HADOOP-7206:
-------------------------------------

+1  I'll wait for Hudson to finish before committing.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206new-b.patch, HADOOP-7206new.patch, HADOOP-7206revertplusnew-b.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16716-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 00:05:16 2011
Return-Path: <common-issues-return-16716-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 132EB669A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 25 Jun 2011 00:05:16 +0000 (UTC)
Received: (qmail 72483 invoked by uid 500); 25 Jun 2011 00:05:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72435 invoked by uid 500); 25 Jun 2011 00:05:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72427 invoked by uid 99); 25 Jun 2011 00:05:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 00:05:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 00:05:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E7A2E42EED1
	for <common-issues@hadoop.apache.org>; Sat, 25 Jun 2011 00:04:51 +0000 (UTC)
Date: Sat, 25 Jun 2011 00:04:51 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1810130575.39244.1308960291945.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054767#comment-13054767 ] 

Hadoop QA commented on HADOOP-7206:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12483793/HADOOP-7206new-b.patch
  against trunk revision 1139387.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/671//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/671//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/671//console

This message is automatically generated.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206new-b.patch, HADOOP-7206new.patch, HADOOP-7206revertplusnew-b.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16717-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 00:07:15 2011
Return-Path: <common-issues-return-16717-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 88CF36736
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 25 Jun 2011 00:07:15 +0000 (UTC)
Received: (qmail 75939 invoked by uid 500); 25 Jun 2011 00:07:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75662 invoked by uid 500); 25 Jun 2011 00:07:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75628 invoked by uid 99); 25 Jun 2011 00:07:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 00:07:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 00:07:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1E67542C003
	for <common-issues@hadoop.apache.org>; Sat, 25 Jun 2011 00:06:50 +0000 (UTC)
Date: Sat, 25 Jun 2011 00:06:50 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1727637131.39299.1308960410121.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-7206:
---------------------------------------

    Status: Open  (was: Patch Available)

var mistake in build script

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206new-b.patch, HADOOP-7206new.patch, HADOOP-7206revertplusnew-b.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16718-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 00:09:09 2011
Return-Path: <common-issues-return-16718-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AA6036752
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 25 Jun 2011 00:09:09 +0000 (UTC)
Received: (qmail 77463 invoked by uid 500); 25 Jun 2011 00:09:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77397 invoked by uid 500); 25 Jun 2011 00:09:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77382 invoked by uid 99); 25 Jun 2011 00:09:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 00:09:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 00:09:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AAE4442C138
	for <common-issues@hadoop.apache.org>; Sat, 25 Jun 2011 00:08:47 +0000 (UTC)
Date: Sat, 25 Jun 2011 00:08:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1390107011.39312.1308960527696.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7426) User Guide for how to use viewfs
 with federation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

User Guide for how to use viewfs with federation
------------------------------------------------

                 Key: HADOOP-7426
                 URL: https://issues.apache.org/jira/browse/HADOOP-7426
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Sanjay Radia
            Assignee: Sanjay Radia
            Priority: Minor




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16719-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 00:11:10 2011
Return-Path: <common-issues-return-16719-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CFFC5685B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 25 Jun 2011 00:11:10 +0000 (UTC)
Received: (qmail 80741 invoked by uid 500); 25 Jun 2011 00:11:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80375 invoked by uid 500); 25 Jun 2011 00:11:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80362 invoked by uid 99); 25 Jun 2011 00:11:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 00:11:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 00:11:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D6B0842C3C1
	for <common-issues@hadoop.apache.org>; Sat, 25 Jun 2011 00:10:47 +0000 (UTC)
Date: Sat, 25 Jun 2011 00:10:47 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <378306222.39319.1308960647876.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-7206:
---------------------------------------

    Attachment: HADOOP-7206new-c.patch

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206new-b.patch, HADOOP-7206new-c.patch, HADOOP-7206new.patch, HADOOP-7206revertplusnew-b.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16720-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 00:11:11 2011
Return-Path: <common-issues-return-16720-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 708216868
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 25 Jun 2011 00:11:11 +0000 (UTC)
Received: (qmail 81113 invoked by uid 500); 25 Jun 2011 00:11:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81038 invoked by uid 500); 25 Jun 2011 00:11:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81027 invoked by uid 99); 25 Jun 2011 00:11:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 00:11:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 00:11:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AF43E42C3BC
	for <common-issues@hadoop.apache.org>; Sat, 25 Jun 2011 00:10:47 +0000 (UTC)
Date: Sat, 25 Jun 2011 00:10:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <436437289.39315.1308960647714.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1390107011.39312.1308960527696.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7426) User Guide for how to use viewfs
 with federation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7426:
---------------------------------

    Attachment: Viewfs Guide.pdf

> User Guide for how to use viewfs with federation
> ------------------------------------------------
>
>                 Key: HADOOP-7426
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7426
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>            Priority: Minor
>         Attachments: Viewfs Guide.pdf
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16721-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 00:11:13 2011
Return-Path: <common-issues-return-16721-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 14DBC687D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 25 Jun 2011 00:11:13 +0000 (UTC)
Received: (qmail 81457 invoked by uid 500); 25 Jun 2011 00:11:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81412 invoked by uid 500); 25 Jun 2011 00:11:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81404 invoked by uid 99); 25 Jun 2011 00:11:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 00:11:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 00:11:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CC22C42C406
	for <common-issues@hadoop.apache.org>; Sat, 25 Jun 2011 00:10:49 +0000 (UTC)
Date: Sat, 25 Jun 2011 00:10:49 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <966174505.39371.1308960649833.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1390107011.39312.1308960527696.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7426) User Guide for how to use viewfs
 with federation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054774#comment-13054774 ] 

Sanjay Radia commented on HADOOP-7426:
--------------------------------------

I have attached a guide for using viewfs with federation.

> User Guide for how to use viewfs with federation
> ------------------------------------------------
>
>                 Key: HADOOP-7426
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7426
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>            Priority: Minor
>         Attachments: Viewfs Guide.pdf
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16722-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 00:11:15 2011
Return-Path: <common-issues-return-16722-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6454468AD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 25 Jun 2011 00:11:15 +0000 (UTC)
Received: (qmail 82955 invoked by uid 500); 25 Jun 2011 00:11:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82900 invoked by uid 500); 25 Jun 2011 00:11:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82892 invoked by uid 99); 25 Jun 2011 00:11:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 00:11:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 00:11:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C466F42C44E
	for <common-issues@hadoop.apache.org>; Sat, 25 Jun 2011 00:10:51 +0000 (UTC)
Date: Sat, 25 Jun 2011 00:10:51 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1464091340.39424.1308960651801.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-7206:
---------------------------------------

    Status: Patch Available  (was: Open)

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206new-b.patch, HADOOP-7206new-c.patch, HADOOP-7206new.patch, HADOOP-7206revertplusnew-b.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16723-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 00:15:13 2011
Return-Path: <common-issues-return-16723-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 957206995
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 25 Jun 2011 00:15:13 +0000 (UTC)
Received: (qmail 88327 invoked by uid 500); 25 Jun 2011 00:15:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88279 invoked by uid 500); 25 Jun 2011 00:15:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88271 invoked by uid 99); 25 Jun 2011 00:15:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 00:15:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 00:15:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 937B642C654
	for <common-issues@hadoop.apache.org>; Sat, 25 Jun 2011 00:14:49 +0000 (UTC)
Date: Sat, 25 Jun 2011 00:14:49 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1230234416.39481.1308960889600.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054775#comment-13054775 ] 

Eli Collins commented on HADOOP-7206:
-------------------------------------

Ah, thought those snappy.* properties were globally scoped, +1 to fix in the latest patch.  

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206new-b.patch, HADOOP-7206new-c.patch, HADOOP-7206new.patch, HADOOP-7206revertplusnew-b.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16724-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 00:25:12 2011
Return-Path: <common-issues-return-16724-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E00616FD8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 25 Jun 2011 00:25:12 +0000 (UTC)
Received: (qmail 97058 invoked by uid 500); 25 Jun 2011 00:25:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96979 invoked by uid 500); 25 Jun 2011 00:25:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96969 invoked by uid 99); 25 Jun 2011 00:25:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 00:25:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 00:25:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 42A1842C888
	for <common-issues@hadoop.apache.org>; Sat, 25 Jun 2011 00:24:48 +0000 (UTC)
Date: Sat, 25 Jun 2011 00:24:48 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1933976710.39494.1308961488269.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054777#comment-13054777 ] 

Hadoop QA commented on HADOOP-7206:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12483796/HADOOP-7206new-c.patch
  against trunk revision 1139387.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/672//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/672//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/672//console

This message is automatically generated.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206new-b.patch, HADOOP-7206new-c.patch, HADOOP-7206new.patch, HADOOP-7206revertplusnew-b.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16725-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 00:59:09 2011
Return-Path: <common-issues-return-16725-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CBB9C60E3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 25 Jun 2011 00:59:09 +0000 (UTC)
Received: (qmail 19350 invoked by uid 500); 25 Jun 2011 00:59:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19231 invoked by uid 500); 25 Jun 2011 00:59:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19220 invoked by uid 99); 25 Jun 2011 00:59:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 00:59:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 00:59:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7685342E08B
	for <common-issues@hadoop.apache.org>; Sat, 25 Jun 2011 00:58:47 +0000 (UTC)
Date: Sat, 25 Jun 2011 00:58:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <558149280.39602.1308963527482.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054785#comment-13054785 ] 

Eli Collins commented on HADOOP-7206:
-------------------------------------

I verified that with snappy-1.0.3 installed on my host a tarball built with {{ant -Dforrest.home=$FORREST_HOME -Dbundle.snappy=true compile-native tar}} correctly bundles the snappy lib. Also tested that TestCodec run from this tarball runs w/ native snappy enabled when it (and libhadoop) are available on the host. 


> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206new-b.patch, HADOOP-7206new-c.patch, HADOOP-7206new.patch, HADOOP-7206revertplusnew-b.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16726-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 01:01:20 2011
Return-Path: <common-issues-return-16726-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2A20C67AC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 25 Jun 2011 01:01:20 +0000 (UTC)
Received: (qmail 20446 invoked by uid 500); 25 Jun 2011 01:01:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20399 invoked by uid 500); 25 Jun 2011 01:01:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20391 invoked by uid 99); 25 Jun 2011 01:01:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 01:01:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 01:01:17 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 32DEE42E1BD
	for <common-issues@hadoop.apache.org>; Sat, 25 Jun 2011 01:00:56 +0000 (UTC)
Date: Sat, 25 Jun 2011 01:00:56 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <777942870.39683.1308963656204.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054786#comment-13054786 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7206:
------------------------------------------------

@Alejandro, thanks for adding the javadoc.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206new-b.patch, HADOOP-7206new-c.patch, HADOOP-7206new.patch, HADOOP-7206revertplusnew-b.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16727-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 01:03:09 2011
Return-Path: <common-issues-return-16727-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D0DD56124
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 25 Jun 2011 01:03:09 +0000 (UTC)
Received: (qmail 22422 invoked by uid 500); 25 Jun 2011 01:03:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22364 invoked by uid 500); 25 Jun 2011 01:03:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22356 invoked by uid 99); 25 Jun 2011 01:03:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 01:03:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 01:03:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CB6E542E33A
	for <common-issues@hadoop.apache.org>; Sat, 25 Jun 2011 01:02:47 +0000 (UTC)
Date: Sat, 25 Jun 2011 01:02:47 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1275634327.39748.1308963767830.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <407429641.30834.1308786767442.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7417) Hadoop Management System
 (Umbrella)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054787#comment-13054787 ] 

Eric Yang commented on HADOOP-7417:
-----------------------------------

A brief description of Hadoop Management System design:

!http://people.apache.org/~eyang/docs/HMS.svg!

h4. Setup

HMS Agent is a list of rpm packages which can be deployed as part of OS image through PXE boot.  HMS Beacon is a daemon which runs on each zookeeper nodes to broadcast the location of the zookeeper.  HMS Agent and controllers are standalone daemons, which resolve zookeeper location through HMS Beacon (zeroconf).

h4. Operation

Operator can issue command through HMS client and pass through HMS controller REST API.  HMS command is serialized into JSON messages and queued in Zookeeper.  Multiple HMS controllers watch the command queue for commands.  When a command triggers the controller to execute, HMS controllers compete to create a lock for the command, and corresponding cluster to execute the command.  If locks are successfully created, the controller begin to translate the command into a list of actions to perform on the managed nodes.  HMS controller watches for the status queues and coordinate actions to perform on HMS agents.  HMS managed agents download software through yum repository or bit torrent through peer exchange.  HMS agent reports installation status and configuration status back to agent status queue for HMS controller to orchestrate the cluster deployment.  Once, all actions are finalized, HMS controller store the deployment command history in the cluster node.  

In the event of node failures (to be implemented), operator can re-image the defected node. When the agent join back, HMS agent can send status to controller to replay the installation and configuration history to recover.

h4. Monitoring Proposal

For large clusters deployment, monitoring setup could be complex.  HMS can simplify this by orchestrate Hadoop 0.20.2+1 (append branch) + HBase 0.90.3 + Pig 0.8.1 + Chukwa 0.5 deployment using the proposed RPM packages for HADOOP-6255, ZOOKEEPER-999, HBASE-3606, PIG-1857, CHUKWA (HADOOP-5030).

> Hadoop Management System (Umbrella)
> -----------------------------------
>
>                 Key: HADOOP-7417
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7417
>             Project: Hadoop Common
>          Issue Type: New Feature
>         Environment: Java 6, Linux
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>
> The primary goal of Hadoop Management System is to build a component around management and deployment of Hadoop related projects. This includes software installation, configuration, application orchestration, deployment automation and monitoring Hadoop.
> Prototype demo source code can be obtained from:
> http://github.com/macroadster/hms

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16728-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 01:05:11 2011
Return-Path: <common-issues-return-16728-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BC60167D7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 25 Jun 2011 01:05:11 +0000 (UTC)
Received: (qmail 23744 invoked by uid 500); 25 Jun 2011 01:05:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23714 invoked by uid 500); 25 Jun 2011 01:05:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23705 invoked by uid 99); 25 Jun 2011 01:05:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 01:05:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 01:05:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B2E9442E3BA
	for <common-issues@hadoop.apache.org>; Sat, 25 Jun 2011 01:04:47 +0000 (UTC)
Date: Sat, 25 Jun 2011 01:04:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2081065247.39756.1308963887729.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7206:
--------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I've committed this. Thank you Issei and Alejandro!

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206new-b.patch, HADOOP-7206new-c.patch, HADOOP-7206new.patch, HADOOP-7206revertplusnew-b.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16729-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 01:35:09 2011
Return-Path: <common-issues-return-16729-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DB1C9673D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 25 Jun 2011 01:35:09 +0000 (UTC)
Received: (qmail 38159 invoked by uid 500); 25 Jun 2011 01:35:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38120 invoked by uid 500); 25 Jun 2011 01:35:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38110 invoked by uid 99); 25 Jun 2011 01:35:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 01:35:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 01:35:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 68C4D42EC0F
	for <common-issues@hadoop.apache.org>; Sat, 25 Jun 2011 01:34:47 +0000 (UTC)
Date: Sat, 25 Jun 2011 01:34:47 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <384305569.39859.1308965687425.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <504299282.44039.1306374047470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7331) Make hadoop-daemon.sh to return 1 if
 daemon processes did not get started
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tanping Wang updated HADOOP-7331:
---------------------------------

    Summary: Make hadoop-daemon.sh to return 1 if daemon processes did not get started  (was: Makes hadoop-daemon.sh to return 1 if daemon processes did not get started)

> Make hadoop-daemon.sh to return 1 if daemon processes did not get started
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-7331
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7331
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7331.2.patch, HADOOP-7331.patch
>
>
> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16730-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 09:19:27 2011
Return-Path: <common-issues-return-16730-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B29954FE9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 25 Jun 2011 09:19:27 +0000 (UTC)
Received: (qmail 43048 invoked by uid 500); 25 Jun 2011 09:19:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42824 invoked by uid 500); 25 Jun 2011 09:19:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42793 invoked by uid 99); 25 Jun 2011 09:19:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 09:19:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 09:19:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 97A9242FE95
	for <common-issues@hadoop.apache.org>; Sat, 25 Jun 2011 09:18:47 +0000 (UTC)
Date: Sat, 25 Jun 2011 09:18:47 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1247957115.40166.1308993527618.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1484131492.21509.1305696107453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7298) Add test utility for writing
 multi-threaded tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J updated HADOOP-7298:
----------------------------

    Attachment: hadoop-7298.txt

Ditto code patch attached, with following changes:

+JavaDocs
-Whitespaces

> Add test utility for writing multi-threaded tests
> -------------------------------------------------
>
>                 Key: HADOOP-7298
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7298
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7298.txt, hadoop-7298.txt
>
>
> A lot of our tests spawn off multiple threads in order to check various synchronization issues, etc. It's often tedious to write these kinds of tests because you have to manually propagate exceptions back to the main thread, etc.
> In HBase we have developed a testing utility which makes writing these kinds of tests much easier. I'd like to copy that utility into Hadoop so we can use it here as well.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16731-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 09:25:36 2011
Return-Path: <common-issues-return-16731-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9964C4459
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 25 Jun 2011 09:25:36 +0000 (UTC)
Received: (qmail 50102 invoked by uid 500); 25 Jun 2011 09:25:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49664 invoked by uid 500); 25 Jun 2011 09:25:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49616 invoked by uid 99); 25 Jun 2011 09:25:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 09:25:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 09:25:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 98FD442F02B
	for <common-issues@hadoop.apache.org>; Sat, 25 Jun 2011 09:24:47 +0000 (UTC)
Date: Sat, 25 Jun 2011 09:24:47 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <897302862.40183.1308993887623.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1484131492.21509.1305696107453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7298) Add test utility for writing
 multi-threaded tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J updated HADOOP-7298:
----------------------------

    Fix Version/s: 0.23.0
           Status: Patch Available  (was: Open)

> Add test utility for writing multi-threaded tests
> -------------------------------------------------
>
>                 Key: HADOOP-7298
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7298
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: hadoop-7298.txt, hadoop-7298.txt
>
>
> A lot of our tests spawn off multiple threads in order to check various synchronization issues, etc. It's often tedious to write these kinds of tests because you have to manually propagate exceptions back to the main thread, etc.
> In HBase we have developed a testing utility which makes writing these kinds of tests much easier. I'd like to copy that utility into Hadoop so we can use it here as well.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16732-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 09:45:22 2011
Return-Path: <common-issues-return-16732-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BF2EB4E3A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 25 Jun 2011 09:45:22 +0000 (UTC)
Received: (qmail 66259 invoked by uid 500); 25 Jun 2011 09:45:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66026 invoked by uid 500); 25 Jun 2011 09:45:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66004 invoked by uid 99); 25 Jun 2011 09:45:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 09:45:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 09:45:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7144142F380
	for <common-issues@hadoop.apache.org>; Sat, 25 Jun 2011 09:44:47 +0000 (UTC)
Date: Sat, 25 Jun 2011 09:44:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1124953987.40193.1308995087460.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1484131492.21509.1305696107453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7298) Add test utility for writing
 multi-threaded tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054848#comment-13054848 ] 

Hadoop QA commented on HADOOP-7298:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12483809/hadoop-7298.txt
  against trunk revision 1139476.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 4 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/673//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/673//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/673//console

This message is automatically generated.

> Add test utility for writing multi-threaded tests
> -------------------------------------------------
>
>                 Key: HADOOP-7298
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7298
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: hadoop-7298.txt, hadoop-7298.txt
>
>
> A lot of our tests spawn off multiple threads in order to check various synchronization issues, etc. It's often tedious to write these kinds of tests because you have to manually propagate exceptions back to the main thread, etc.
> In HBase we have developed a testing utility which makes writing these kinds of tests much easier. I'd like to copy that utility into Hadoop so we can use it here as well.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16733-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 11:31:10 2011
Return-Path: <common-issues-return-16733-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0A7D44E65
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 25 Jun 2011 11:31:10 +0000 (UTC)
Received: (qmail 38779 invoked by uid 500); 25 Jun 2011 11:31:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38593 invoked by uid 500); 25 Jun 2011 11:31:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38585 invoked by uid 99); 25 Jun 2011 11:31:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 11:31:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 11:31:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6741642F735
	for <common-issues@hadoop.apache.org>; Sat, 25 Jun 2011 11:30:47 +0000 (UTC)
Date: Sat, 25 Jun 2011 11:30:47 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2037008512.40285.1309001447419.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054864#comment-13054864 ] 

Harsh J commented on HADOOP-7328:
---------------------------------

Is using nulls agreed upon? I'd like to scrap the r3 patch if so, and get the r2 one in instead, so I can do the improvement on the MR side appropriately.

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff, HADOOP-7328.r2.diff, HADOOP-7328.r3.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16734-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 12:23:12 2011
Return-Path: <common-issues-return-16734-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DF9454CE8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 25 Jun 2011 12:23:11 +0000 (UTC)
Received: (qmail 70081 invoked by uid 500); 25 Jun 2011 12:23:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70033 invoked by uid 500); 25 Jun 2011 12:23:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70023 invoked by uid 99); 25 Jun 2011 12:23:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 12:23:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 12:23:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CF47E42F295
	for <common-issues@hadoop.apache.org>; Sat, 25 Jun 2011 12:22:47 +0000 (UTC)
Date: Sat, 25 Jun 2011 12:22:47 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1352510570.40343.1309004567846.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6801) io.sort.mb and io.sort.factor were
 renamed and moved to mapreduce but are still in
 CommonConfigurationKeysPublic.java and used in SequenceFile.java
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J updated HADOOP-6801:
----------------------------

    Attachment: HADOOP-6801.r1.diff

Patch that introduces two new options over io.sort.mb and io.sort.factor, specifically for SequenceFile.Sorter:

{code}
seq.io.sort.mb = 100
seq.io.sort.factor = 100
{code}

Descriptions included inside {{core-default.xml}} changes as well.

I did not yet write checks for _deprecated_ io.sort.* properties. Is the double deprecation fine here (MR already deprecates this, and now Common has to too, for new SequenceFile.Sorter specific properties)?

I believe DistCp would need documentation work as well, since it uses SequenceFile.Sorter

> io.sort.mb and io.sort.factor were renamed and moved to mapreduce but are still in CommonConfigurationKeysPublic.java and used in SequenceFile.java
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6801
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6801
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Harsh J
>            Priority: Minor
>         Attachments: HADOOP-6801.r1.diff
>
>
> Following configuration keys in CommonConfigurationKeysPublic.java (former CommonConfigurationKeys.java):
> public static final String  IO_SORT_MB_KEY = "io.sort.mb";
> public static final String  IO_SORT_FACTOR_KEY = "io.sort.factor";
> are partially moved:
>   - they were renamed to mapreduce.task.io.sort.mb and mapreduce.task.io.sort.factor respectively
>   - they were moved to mapreduce project, documented in mapred-default.xml
> However:
>   - they are still listed in CommonConfigurationKeysPublic.java as quoted above
>   - strings "io.sort.mb" and "io.sort.factor" are used in SequenceFile.java in Hadoop Common project
> Not sure what the solution is, these constants should probably be removed from CommonConfigurationKeysPublic.java but I am not sure what's the best solution for SequenceFile.java.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16735-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 15:28:12 2011
Return-Path: <common-issues-return-16735-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0449F4BE1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 25 Jun 2011 15:28:12 +0000 (UTC)
Received: (qmail 62478 invoked by uid 500); 25 Jun 2011 15:28:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62391 invoked by uid 500); 25 Jun 2011 15:28:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62373 invoked by uid 99); 25 Jun 2011 15:28:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 15:28:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 15:28:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E11C6430AA6
	for <common-issues@hadoop.apache.org>; Sat, 25 Jun 2011 15:27:47 +0000 (UTC)
Date: Sat, 25 Jun 2011 15:27:47 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <740829586.40554.1309015667919.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <407429641.30834.1308786767442.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7417) Hadoop Management System
 (Umbrella)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054921#comment-13054921 ] 

Allen Wittenauer commented on HADOOP-7417:
------------------------------------------

bq. This is a customized manage and deployment system for Hadoop software stack.

This makes no sense.  Rather than re-invent a specific type of wheel, one could generalize the wheel and get a potentially larger audience.  The problem of course is that the competition rate goes way up.  

>From the above description, I don't see anything here that makes this particularly unique vs. other config systems.  (In fact, it could be argued that it is several years behind other systems.)

> Hadoop Management System (Umbrella)
> -----------------------------------
>
>                 Key: HADOOP-7417
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7417
>             Project: Hadoop Common
>          Issue Type: New Feature
>         Environment: Java 6, Linux
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>
> The primary goal of Hadoop Management System is to build a component around management and deployment of Hadoop related projects. This includes software installation, configuration, application orchestration, deployment automation and monitoring Hadoop.
> Prototype demo source code can be obtained from:
> http://github.com/macroadster/hms

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16736-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 19:06:10 2011
Return-Path: <common-issues-return-16736-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2DCF546C4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 25 Jun 2011 19:06:10 +0000 (UTC)
Received: (qmail 67729 invoked by uid 500); 25 Jun 2011 19:06:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67638 invoked by uid 500); 25 Jun 2011 19:06:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67630 invoked by uid 99); 25 Jun 2011 19:06:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 19:06:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 19:06:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A2F70430F86
	for <common-issues@hadoop.apache.org>; Sat, 25 Jun 2011 19:05:47 +0000 (UTC)
Date: Sat, 25 Jun 2011 19:05:47 +0000 (UTC)
From: "Jim Plush (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <414366626.40710.1309028747664.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <620925511.8411.1297360858022.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7135) Commit 949660 broke
 FileUtil.copyMerge
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054950#comment-13054950 ] 

Jim Plush commented on HADOOP-7135:
-----------------------------------

the link above is a 404, any chance of updating it?

> Commit 949660 broke FileUtil.copyMerge
> --------------------------------------
>
>                 Key: HADOOP-7135
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7135
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Jeffrey Gerard
>   Original Estimate: 5m
>  Remaining Estimate: 5m
>
> Looking at http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.21/src/java/org/apache/hadoop/fs/FileUtil.java?r1=949659&r2=949660&
> it seems this commit broke FileUtil.copyMerge by omitting NOT operator.  copyMerge only makes sense if srcDir is a directory.
> Should be
>     if (!srcFS.getFileStatus(srcDir).isDirectory())
>        return false;

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16737-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 19:14:11 2011
Return-Path: <common-issues-return-16737-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AE1634798
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 25 Jun 2011 19:14:11 +0000 (UTC)
Received: (qmail 71472 invoked by uid 500); 25 Jun 2011 19:14:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71409 invoked by uid 500); 25 Jun 2011 19:14:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71401 invoked by uid 99); 25 Jun 2011 19:14:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 19:14:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 19:14:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 788F64301F9
	for <common-issues@hadoop.apache.org>; Sat, 25 Jun 2011 19:13:47 +0000 (UTC)
Date: Sat, 25 Jun 2011 19:13:47 +0000 (UTC)
From: "Jim Plush (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1668666733.40729.1309029227490.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <620925511.8411.1297360858022.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7135) Commit 949660 broke
 FileUtil.copyMerge
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054955#comment-13054955 ] 

Jim Plush commented on HADOOP-7135:
-----------------------------------

This appears to be fixed in trunk
FileUtil.copyMerge method now shows:

 dstFile = checkDest(srcDir.getName(), dstFS, dstFile, false);

    if (!srcFS.getFileStatus(srcDir).isDirectory())
      return false;
   
    OutputStream out = dstFS.create(dstFile);

> Commit 949660 broke FileUtil.copyMerge
> --------------------------------------
>
>                 Key: HADOOP-7135
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7135
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Jeffrey Gerard
>   Original Estimate: 5m
>  Remaining Estimate: 5m
>
> Looking at http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.21/src/java/org/apache/hadoop/fs/FileUtil.java?r1=949659&r2=949660&
> it seems this commit broke FileUtil.copyMerge by omitting NOT operator.  copyMerge only makes sense if srcDir is a directory.
> Should be
>     if (!srcFS.getFileStatus(srcDir).isDirectory())
>        return false;

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16738-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 19:16:14 2011
Return-Path: <common-issues-return-16738-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F38314ADC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 25 Jun 2011 19:16:11 +0000 (UTC)
Received: (qmail 74447 invoked by uid 500); 25 Jun 2011 19:16:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74342 invoked by uid 500); 25 Jun 2011 19:16:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74332 invoked by uid 99); 25 Jun 2011 19:16:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 19:16:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 19:16:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0F9194302C3
	for <common-issues@hadoop.apache.org>; Sat, 25 Jun 2011 19:15:48 +0000 (UTC)
Date: Sat, 25 Jun 2011 19:15:48 +0000 (UTC)
From: "Jim Plush (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1639830023.40733.1309029348060.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1031095941.46621.1302367745992.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7218) FileUtil.copyMerge implementation
 error
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054956#comment-13054956 ] 

Jim Plush commented on HADOOP-7218:
-----------------------------------

This seems to be a duplicate of:

https://issues.apache.org/jira/browse/HADOOP-7135

This is also fixed in the trunk:

FileUtil.copyMerge:

 dstFile = checkDest(srcDir.getName(), dstFS, dstFile, false);

    if (!srcFS.getFileStatus(srcDir).isDirectory())
      return false;
   
    OutputStream out = dstFS.create(dstFile);

> FileUtil.copyMerge implementation error
> ---------------------------------------
>
>                 Key: HADOOP-7218
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7218
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0
>         Environment: CENTOS 5.5 64bit
>            Reporter: XiaoboGu
>
> if (srcFS.getFileStatus(srcDir).isDirectory())
> 			return false;
> should be 
> if (!srcFS.getFileStatus(srcDir).isDirectory())
> 			return false;

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16739-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat Jun 25 20:32:12 2011
Return-Path: <common-issues-return-16739-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CB8664D90
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 25 Jun 2011 20:32:12 +0000 (UTC)
Received: (qmail 14291 invoked by uid 500); 25 Jun 2011 20:32:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14064 invoked by uid 500); 25 Jun 2011 20:32:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14052 invoked by uid 99); 25 Jun 2011 20:32:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 20:32:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Jun 2011 20:32:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E6292430B61
	for <common-issues@hadoop.apache.org>; Sat, 25 Jun 2011 20:31:47 +0000 (UTC)
Date: Sat, 25 Jun 2011 20:31:47 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2127217979.40819.1309033907939.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <407429641.30834.1308786767442.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7417) Hadoop Management System
 (Umbrella)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7417?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D130=
54963#comment-13054963 ]=20

Eric Yang commented on HADOOP-7417:
-----------------------------------

Allen, I did extensive studies on all existing systems including puppet, mc=
ollective, chef, cfengine, controlTier, Bcfg2.  Most of the configuration m=
anagement system focus on generating a set of templates and config paramete=
rs and push out changes one node at a time.  This works fine in small numbe=
r of machines, but most of the system fails beyond 1800 nodes or become dif=
ficult to maintain.  i.e. mcollective uses spamming tree model on puppeteer=
, hence the puppet master becomes single point of failure.  One puppet mast=
er failure could take large chunk of the nodes offline.  HMS is designed to=
 remove single point of failures in the deployment system, and improve perf=
ormance.  we found it is more reliable to store system state in Zookeeper f=
or HA.  Zeroconf is great for resolving service location.  Exist config man=
agement system requires installation and configuration before it can be dep=
loyed.  HMS is designed to install and operate without having to configure =
the management system.  Bittorrent is much faster than install software fro=
m yum repository for large scale system.  Granted that this system started =
several years behind existing system, but it solves some scalability and re=
liability issues. =20

To summarize, HMS does the following better:

- Scale
- Reliability
- Cross node application orchestration (action dependencies)
- Speed
- Sophisticate monitoring system (Reuse Chukwa)
- Self healing cluster (Ability to replay history to heal nodes)


> Hadoop Management System (Umbrella)
> -----------------------------------
>
>                 Key: HADOOP-7417
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7417
>             Project: Hadoop Common
>          Issue Type: New Feature
>         Environment: Java 6, Linux
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>
> The primary goal of Hadoop Management System is to build a component arou=
nd management and deployment of Hadoop related projects. This includes soft=
ware installation, configuration, application orchestration, deployment aut=
omation and monitoring Hadoop.
> Prototype demo source code can be obtained from:
> http://github.com/macroadster/hms

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

From common-issues-return-16740-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 26 00:41:12 2011
Return-Path: <common-issues-return-16740-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 20C514143
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 26 Jun 2011 00:41:12 +0000 (UTC)
Received: (qmail 12572 invoked by uid 500); 26 Jun 2011 00:41:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12512 invoked by uid 500); 26 Jun 2011 00:41:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12490 invoked by uid 99); 26 Jun 2011 00:41:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 00:41:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 00:41:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8012B42E47C
	for <common-issues@hadoop.apache.org>; Sun, 26 Jun 2011 00:40:47 +0000 (UTC)
Date: Sun, 26 Jun 2011 00:40:47 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <942858061.40936.1309048847521.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <164404123.4447.1308088127466.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7392) Implement capability of querying
 individual property of a mbean using JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tanping Wang updated HADOOP-7392:
---------------------------------

    Attachment: HADOOP-7392.2.patch

Attach a patch to address the review comments.

> Implement capability of querying individual property of a mbean using JMXProxyServlet 
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7392
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7392
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7392.2.patch, HADOOP-7392.patch
>
>
> Hadoop-7144 provides the capability to query all the properties of a mbean using JMXProxyServlet.  In addition to this, we add the capability to query an individual property of a mbean.  Client will send http request,
> http://hostname/jmx?get=meanName::property
> to query from server.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16741-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 26 00:41:13 2011
Return-Path: <common-issues-return-16741-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 211374144
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 26 Jun 2011 00:41:12 +0000 (UTC)
Received: (qmail 12811 invoked by uid 500); 26 Jun 2011 00:41:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12517 invoked by uid 500); 26 Jun 2011 00:41:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12491 invoked by uid 99); 26 Jun 2011 00:41:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 00:41:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 00:41:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9574942E47E
	for <common-issues@hadoop.apache.org>; Sun, 26 Jun 2011 00:40:47 +0000 (UTC)
Date: Sun, 26 Jun 2011 00:40:47 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1883116628.40938.1309048847609.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <164404123.4447.1308088127466.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7392) Implement capability of querying
 individual property of a mbean using JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tanping Wang updated HADOOP-7392:
---------------------------------

    Status: Patch Available  (was: Open)

> Implement capability of querying individual property of a mbean using JMXProxyServlet 
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7392
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7392
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7392.2.patch, HADOOP-7392.patch
>
>
> Hadoop-7144 provides the capability to query all the properties of a mbean using JMXProxyServlet.  In addition to this, we add the capability to query an individual property of a mbean.  Client will send http request,
> http://hostname/jmx?get=meanName::property
> to query from server.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16742-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 26 00:55:12 2011
Return-Path: <common-issues-return-16742-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 806AD4BE9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 26 Jun 2011 00:55:12 +0000 (UTC)
Received: (qmail 20338 invoked by uid 500); 26 Jun 2011 00:55:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20189 invoked by uid 500); 26 Jun 2011 00:55:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20165 invoked by uid 99); 26 Jun 2011 00:55:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 00:55:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 00:55:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CE3E042E82B
	for <common-issues@hadoop.apache.org>; Sun, 26 Jun 2011 00:54:47 +0000 (UTC)
Date: Sun, 26 Jun 2011 00:54:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1689053382.40953.1309049687841.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <164404123.4447.1308088127466.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7392) Implement capability of querying
 individual property of a mbean using JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054995#comment-13054995 ] 

Hadoop QA commented on HADOOP-7392:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12483831/HADOOP-7392.2.patch
  against trunk revision 1139476.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/674//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/674//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/674//console

This message is automatically generated.

> Implement capability of querying individual property of a mbean using JMXProxyServlet 
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7392
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7392
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7392.2.patch, HADOOP-7392.patch
>
>
> Hadoop-7144 provides the capability to query all the properties of a mbean using JMXProxyServlet.  In addition to this, we add the capability to query an individual property of a mbean.  Client will send http request,
> http://hostname/jmx?get=meanName::property
> to query from server.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16743-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 26 01:05:11 2011
Return-Path: <common-issues-return-16743-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 80E3B42CE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 26 Jun 2011 01:05:11 +0000 (UTC)
Received: (qmail 23391 invoked by uid 500); 26 Jun 2011 01:05:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23288 invoked by uid 500); 26 Jun 2011 01:05:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23280 invoked by uid 99); 26 Jun 2011 01:05:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 01:05:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 01:05:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6305142EA28
	for <common-issues@hadoop.apache.org>; Sun, 26 Jun 2011 01:04:47 +0000 (UTC)
Date: Sun, 26 Jun 2011 01:04:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1456112278.40986.1309050287402.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <164404123.4447.1308088127466.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7392) Implement capability of querying
 individual property of a mbean using JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054998#comment-13054998 ] 

Luke Lu commented on HADOOP-7392:
---------------------------------

+1, lgtm. Thanks Tanping!

> Implement capability of querying individual property of a mbean using JMXProxyServlet 
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7392
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7392
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7392.2.patch, HADOOP-7392.patch
>
>
> Hadoop-7144 provides the capability to query all the properties of a mbean using JMXProxyServlet.  In addition to this, we add the capability to query an individual property of a mbean.  Client will send http request,
> http://hostname/jmx?get=meanName::property
> to query from server.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16744-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 26 05:20:36 2011
Return-Path: <common-issues-return-16744-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ADB966BBE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 26 Jun 2011 05:20:36 +0000 (UTC)
Received: (qmail 3850 invoked by uid 500); 26 Jun 2011 05:20:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2518 invoked by uid 500); 26 Jun 2011 05:20:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2478 invoked by uid 99); 26 Jun 2011 05:20:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 05:20:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 05:20:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EB8DA431108
	for <common-issues@hadoop.apache.org>; Sun, 26 Jun 2011 05:19:47 +0000 (UTC)
Date: Sun, 26 Jun 2011 05:19:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1932268053.41109.1309065587961.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <620925511.8411.1297360858022.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7135) Commit 949660 broke
 FileUtil.copyMerge
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers resolved HADOOP-7135.
------------------------------------

    Resolution: Invalid
      Assignee: Jim Plush

Thanks a lot for the investigation, Jim. Closing this out.

> Commit 949660 broke FileUtil.copyMerge
> --------------------------------------
>
>                 Key: HADOOP-7135
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7135
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Jeffrey Gerard
>            Assignee: Jim Plush
>   Original Estimate: 5m
>  Remaining Estimate: 5m
>
> Looking at http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.21/src/java/org/apache/hadoop/fs/FileUtil.java?r1=949659&r2=949660&
> it seems this commit broke FileUtil.copyMerge by omitting NOT operator.  copyMerge only makes sense if srcDir is a directory.
> Should be
>     if (!srcFS.getFileStatus(srcDir).isDirectory())
>        return false;

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16745-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 26 05:20:43 2011
Return-Path: <common-issues-return-16745-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 240986BD1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 26 Jun 2011 05:20:43 +0000 (UTC)
Received: (qmail 4382 invoked by uid 500); 26 Jun 2011 05:20:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3825 invoked by uid 500); 26 Jun 2011 05:20:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2487 invoked by uid 99); 26 Jun 2011 05:20:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 05:20:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 05:20:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8CE164310FC
	for <common-issues@hadoop.apache.org>; Sun, 26 Jun 2011 05:19:47 +0000 (UTC)
Date: Sun, 26 Jun 2011 05:19:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1606206532.41098.1309065587573.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1031095941.46621.1302367745992.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7218) FileUtil.copyMerge implementation
 error
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers resolved HADOOP-7218.
------------------------------------

    Resolution: Duplicate
      Assignee: Jim Plush

Thanks a lot for the investigation, Jim. Closing this as a dupe.

> FileUtil.copyMerge implementation error
> ---------------------------------------
>
>                 Key: HADOOP-7218
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7218
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0
>         Environment: CENTOS 5.5 64bit
>            Reporter: XiaoboGu
>            Assignee: Jim Plush
>
> if (srcFS.getFileStatus(srcDir).isDirectory())
> 			return false;
> should be 
> if (!srcFS.getFileStatus(srcDir).isDirectory())
> 			return false;

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16746-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 26 09:00:49 2011
Return-Path: <common-issues-return-16746-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3509367D4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 26 Jun 2011 09:00:49 +0000 (UTC)
Received: (qmail 84092 invoked by uid 500); 26 Jun 2011 09:00:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83473 invoked by uid 500); 26 Jun 2011 09:00:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83402 invoked by uid 99); 26 Jun 2011 09:00:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 09:00:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 09:00:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8932B431FE1
	for <common-issues@hadoop.apache.org>; Sun, 26 Jun 2011 08:59:47 +0000 (UTC)
Date: Sun, 26 Jun 2011 08:59:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1513260035.41223.1309078787558.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13055036#comment-13055036 ] 

Owen O'Malley commented on HADOOP-7328:
---------------------------------------

Yes, I think the r2 patch is good. You need to write some unit tests to ensure the behavior doesn't regress.

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff, HADOOP-7328.r2.diff, HADOOP-7328.r3.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16747-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 26 10:35:12 2011
Return-Path: <common-issues-return-16747-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7C00D693E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 26 Jun 2011 10:35:12 +0000 (UTC)
Received: (qmail 36196 invoked by uid 500); 26 Jun 2011 10:35:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36012 invoked by uid 500); 26 Jun 2011 10:35:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35993 invoked by uid 99); 26 Jun 2011 10:35:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 10:35:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 10:35:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 648CE431E8B
	for <common-issues@hadoop.apache.org>; Sun, 26 Jun 2011 10:34:47 +0000 (UTC)
Date: Sun, 26 Jun 2011 10:34:47 +0000 (UTC)
From: "Brock Noland (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <448809532.41257.1309084487408.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-310) Additional constructor requested in
 BytesWritable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Brock Noland updated HADOOP-310:
--------------------------------

    Attachment: bytes-writable-zero-copy-interface-2.patch

Great points. The attached removes the set method which would have been confusing. This is what the original JIRA request was for.

I do not see any need for the set method I had previously written. If you compare the cost of creating of an additional object, due to no zero copy set, versus being required to perform a copy to fill the BW, the extra object should be negligible with a byte array of any size.

> Additional constructor requested in BytesWritable
> -------------------------------------------------
>
>                 Key: HADOOP-310
>                 URL: https://issues.apache.org/jira/browse/HADOOP-310
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: p sutter
>            Assignee: Owen O'Malley
>            Priority: Minor
>         Attachments: bytes-writable-zero-copy-interface-0.patch, bytes-writable-zero-copy-interface-1.patch, bytes-writable-zero-copy-interface-2.patch
>
>
> It would be grand if BytesWritable.java had an additional constructor as below. This allows me to use the BytesWritable class without doing a buffer copy, since we have a less-than-fully-utilized byte array holding our key.
> Thanks!
>  
>  /**
>    * Create a BytesWritable using the byte array as the initial value.
>    * @param bytes This array becomes the backing storage for the object.
>    */
>   public BytesWritable(byte[] bytes, int size) {
>     this.bytes = bytes;
>     this.size = size;
>   }
>   

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16748-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 26 18:10:10 2011
Return-Path: <common-issues-return-16748-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3B71B6EC1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 26 Jun 2011 18:10:10 +0000 (UTC)
Received: (qmail 7028 invoked by uid 500); 26 Jun 2011 18:10:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6963 invoked by uid 500); 26 Jun 2011 18:10:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6689 invoked by uid 99); 26 Jun 2011 18:10:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 18:10:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 18:10:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 979414325CE
	for <common-issues@hadoop.apache.org>; Sun, 26 Jun 2011 18:09:47 +0000 (UTC)
Date: Sun, 26 Jun 2011 18:09:47 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <596015862.41728.1309111787617.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J updated HADOOP-7328:
----------------------------

    Attachment: HADOOP-7328.r4.diff

Revised patch of {{r2}} that adds in a new test as well (per Owen's comment).

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff, HADOOP-7328.r2.diff, HADOOP-7328.r3.diff, HADOOP-7328.r4.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16749-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 26 18:10:11 2011
Return-Path: <common-issues-return-16749-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 674106ED0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 26 Jun 2011 18:10:11 +0000 (UTC)
Received: (qmail 7286 invoked by uid 500); 26 Jun 2011 18:10:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7259 invoked by uid 500); 26 Jun 2011 18:10:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7249 invoked by uid 99); 26 Jun 2011 18:10:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 18:10:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 18:10:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A76824325D0
	for <common-issues@hadoop.apache.org>; Sun, 26 Jun 2011 18:09:47 +0000 (UTC)
Date: Sun, 26 Jun 2011 18:09:47 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <398836262.41730.1309111787682.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J updated HADOOP-7328:
----------------------------

    Status: Patch Available  (was: Open)

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff, HADOOP-7328.r2.diff, HADOOP-7328.r3.diff, HADOOP-7328.r4.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16750-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 26 18:14:09 2011
Return-Path: <common-issues-return-16750-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5CC65622B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 26 Jun 2011 18:14:09 +0000 (UTC)
Received: (qmail 17193 invoked by uid 500); 26 Jun 2011 18:14:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17138 invoked by uid 500); 26 Jun 2011 18:14:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17121 invoked by uid 99); 26 Jun 2011 18:14:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 18:14:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 18:14:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6B428432698
	for <common-issues@hadoop.apache.org>; Sun, 26 Jun 2011 18:13:47 +0000 (UTC)
Date: Sun, 26 Jun 2011 18:13:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <137825466.41736.1309112027435.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13055136#comment-13055136 ] 

Hadoop QA commented on HADOOP-7328:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12483864/HADOOP-7328.r4.diff
  against trunk revision 1139476.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 2 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/675//console

This message is automatically generated.

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff, HADOOP-7328.r2.diff, HADOOP-7328.r3.diff, HADOOP-7328.r4.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16751-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 26 18:24:09 2011
Return-Path: <common-issues-return-16751-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D499868C5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 26 Jun 2011 18:24:09 +0000 (UTC)
Received: (qmail 23692 invoked by uid 500); 26 Jun 2011 18:24:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23400 invoked by uid 500); 26 Jun 2011 18:24:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23388 invoked by uid 99); 26 Jun 2011 18:24:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 18:24:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 18:24:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B688443289A
	for <common-issues@hadoop.apache.org>; Sun, 26 Jun 2011 18:23:47 +0000 (UTC)
Date: Sun, 26 Jun 2011 18:23:47 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1715558479.41760.1309112627744.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J updated HADOOP-7328:
----------------------------

    Attachment: HADOOP-7328.r4.diff

Not sure what went wrong with {{patch}} there.

Re-attaching with {{common/}} prefixes removed.

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff, HADOOP-7328.r2.diff, HADOOP-7328.r3.diff, HADOOP-7328.r4.diff, HADOOP-7328.r4.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16752-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 26 18:26:10 2011
Return-Path: <common-issues-return-16752-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3799768DD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 26 Jun 2011 18:26:10 +0000 (UTC)
Received: (qmail 24108 invoked by uid 500); 26 Jun 2011 18:26:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23991 invoked by uid 500); 26 Jun 2011 18:26:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23983 invoked by uid 99); 26 Jun 2011 18:26:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 18:26:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 18:26:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 118EF4329FA
	for <common-issues@hadoop.apache.org>; Sun, 26 Jun 2011 18:25:48 +0000 (UTC)
Date: Sun, 26 Jun 2011 18:25:48 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <628124890.41777.1309112748068.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J updated HADOOP-7328:
----------------------------

    Status: Patch Available  (was: Open)

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff, HADOOP-7328.r2.diff, HADOOP-7328.r3.diff, HADOOP-7328.r4.diff, HADOOP-7328.r4.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16753-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 26 18:26:11 2011
Return-Path: <common-issues-return-16753-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D022E68F6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 26 Jun 2011 18:26:11 +0000 (UTC)
Received: (qmail 24307 invoked by uid 500); 26 Jun 2011 18:26:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24271 invoked by uid 500); 26 Jun 2011 18:26:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24251 invoked by uid 99); 26 Jun 2011 18:26:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 18:26:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 18:26:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CE3124329F4
	for <common-issues@hadoop.apache.org>; Sun, 26 Jun 2011 18:25:47 +0000 (UTC)
Date: Sun, 26 Jun 2011 18:25:47 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <736875858.41771.1309112747841.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J updated HADOOP-7328:
----------------------------

    Status: Open  (was: Patch Available)

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff, HADOOP-7328.r2.diff, HADOOP-7328.r3.diff, HADOOP-7328.r4.diff, HADOOP-7328.r4.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16754-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 26 18:34:12 2011
Return-Path: <common-issues-return-16754-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EA5886DEE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 26 Jun 2011 18:34:11 +0000 (UTC)
Received: (qmail 27886 invoked by uid 500); 26 Jun 2011 18:34:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27836 invoked by uid 500); 26 Jun 2011 18:34:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27812 invoked by uid 99); 26 Jun 2011 18:34:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 18:34:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 18:34:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BDA05432C16
	for <common-issues@hadoop.apache.org>; Sun, 26 Jun 2011 18:33:47 +0000 (UTC)
Date: Sun, 26 Jun 2011 18:33:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2136446577.41801.1309113227773.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13055147#comment-13055147 ] 

Hadoop QA commented on HADOOP-7328:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12483867/HADOOP-7328.r4.diff
  against trunk revision 1139476.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 2 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/676//console

This message is automatically generated.

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J
>            Assignee: Harsh J
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff, HADOOP-7328.r2.diff, HADOOP-7328.r3.diff, HADOOP-7328.r4.diff, HADOOP-7328.r4.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16755-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 26 21:54:14 2011
Return-Path: <common-issues-return-16755-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EEC3D6BA9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 26 Jun 2011 21:54:13 +0000 (UTC)
Received: (qmail 34827 invoked by uid 500); 26 Jun 2011 21:54:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34760 invoked by uid 500); 26 Jun 2011 21:54:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34750 invoked by uid 99); 26 Jun 2011 21:54:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 21:54:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 21:54:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C51AB432896
	for <common-issues@hadoop.apache.org>; Sun, 26 Jun 2011 21:53:49 +0000 (UTC)
Date: Sun, 26 Jun 2011 21:53:49 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <331529283.42058.1309125229804.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <164404123.4447.1308088127466.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7392) Implement capability of querying
 individual property of a mbean using JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tanping Wang updated HADOOP-7392:
---------------------------------

      Resolution: Fixed
    Release Note: Thanks Luke for reviewing.  Just committed to trunk.
          Status: Resolved  (was: Patch Available)

> Implement capability of querying individual property of a mbean using JMXProxyServlet 
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7392
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7392
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7392.2.patch, HADOOP-7392.patch
>
>
> Hadoop-7144 provides the capability to query all the properties of a mbean using JMXProxyServlet.  In addition to this, we add the capability to query an individual property of a mbean.  Client will send http request,
> http://hostname/jmx?get=meanName::property
> to query from server.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16756-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 26 21:56:09 2011
Return-Path: <common-issues-return-16756-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7B7246BE3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 26 Jun 2011 21:56:09 +0000 (UTC)
Received: (qmail 35422 invoked by uid 500); 26 Jun 2011 21:56:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35286 invoked by uid 500); 26 Jun 2011 21:56:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35278 invoked by uid 99); 26 Jun 2011 21:56:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 21:56:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 21:56:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7B27C43296A
	for <common-issues@hadoop.apache.org>; Sun, 26 Jun 2011 21:55:47 +0000 (UTC)
Date: Sun, 26 Jun 2011 21:55:47 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1939799470.42064.1309125347501.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <164404123.4447.1308088127466.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7392) Implement capability of querying
 individual property of a mbean using JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tanping Wang updated HADOOP-7392:
---------------------------------

    Release Note:   (was: Thanks Luke for reviewing.  Just committed to trunk.)
    Hadoop Flags: [Reviewed]

> Implement capability of querying individual property of a mbean using JMXProxyServlet 
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7392
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7392
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7392.2.patch, HADOOP-7392.patch
>
>
> Hadoop-7144 provides the capability to query all the properties of a mbean using JMXProxyServlet.  In addition to this, we add the capability to query an individual property of a mbean.  Client will send http request,
> http://hostname/jmx?get=meanName::property
> to query from server.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16757-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun Jun 26 21:56:11 2011
Return-Path: <common-issues-return-16757-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 26AFD6BF3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 26 Jun 2011 21:56:11 +0000 (UTC)
Received: (qmail 35602 invoked by uid 500); 26 Jun 2011 21:56:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35560 invoked by uid 500); 26 Jun 2011 21:56:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35546 invoked by uid 99); 26 Jun 2011 21:56:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 21:56:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Jun 2011 21:56:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8165243296B
	for <common-issues@hadoop.apache.org>; Sun, 26 Jun 2011 21:55:47 +0000 (UTC)
Date: Sun, 26 Jun 2011 21:55:47 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1260447183.42065.1309125347527.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <164404123.4447.1308088127466.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7392) Implement capability of querying
 individual property of a mbean using JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13055212#comment-13055212 ] 

Tanping Wang commented on HADOOP-7392:
--------------------------------------

Thanks Luke for reviewing.  Just committed to trunk.

> Implement capability of querying individual property of a mbean using JMXProxyServlet 
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7392
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7392
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7392.2.patch, HADOOP-7392.patch
>
>
> Hadoop-7144 provides the capability to query all the properties of a mbean using JMXProxyServlet.  In addition to this, we add the capability to query an individual property of a mbean.  Client will send http request,
> http://hostname/jmx?get=meanName::property
> to query from server.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16758-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 27 00:10:09 2011
Return-Path: <common-issues-return-16758-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AC8206339
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 27 Jun 2011 00:10:09 +0000 (UTC)
Received: (qmail 33059 invoked by uid 500); 27 Jun 2011 00:10:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32964 invoked by uid 500); 27 Jun 2011 00:10:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32956 invoked by uid 99); 27 Jun 2011 00:10:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 00:10:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 00:10:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 953774327A9
	for <common-issues@hadoop.apache.org>; Mon, 27 Jun 2011 00:09:47 +0000 (UTC)
Date: Mon, 27 Jun 2011 00:09:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <817618820.42234.1309133387608.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7427) syntax error in smart-apply-patch.sh
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

syntax error in smart-apply-patch.sh 
-------------------------------------

                 Key: HADOOP-7427
                 URL: https://issues.apache.org/jira/browse/HADOOP-7427
             Project: Hadoop Common
          Issue Type: Bug
          Components: scripts
            Reporter: Tsz Wo (Nicholas), SZE


{noformat}
     [exec]     Finished build.
     [exec] hdfs/src/test/bin/smart-apply-patch.sh: line 60: syntax error in conditional expression: unexpected token `('

BUILD FAILED
hdfs/build.xml:1595: exec returned: 1
{noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16759-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 27 00:18:09 2011
Return-Path: <common-issues-return-16759-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A04C96F6F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 27 Jun 2011 00:18:09 +0000 (UTC)
Received: (qmail 43135 invoked by uid 500); 27 Jun 2011 00:18:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42927 invoked by uid 500); 27 Jun 2011 00:18:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42916 invoked by uid 99); 27 Jun 2011 00:18:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 00:18:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 00:18:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6F7C243298F
	for <common-issues@hadoop.apache.org>; Mon, 27 Jun 2011 00:17:47 +0000 (UTC)
Date: Mon, 27 Jun 2011 00:17:47 +0000 (UTC)
From: "Rajiv Chittajallu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <645613181.42251.1309133867453.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <407429641.30834.1308786767442.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7417) Hadoop Management System
 (Umbrella)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13055244#comment-13055244 ] 

Rajiv Chittajallu commented on HADOOP-7417:
-------------------------------------------

{quote}
Allen, I did extensive studies on all existing systems including puppet, mcollective, chef, cfengine, controlTier, Bcfg2. Most of the configuration management system focus on generating a set of templates and config parameters and push out changes one node at a time. This works fine in small number of machines, but most of the system fails beyond 1800 nodes or become difficult to maintain.
{quote}

We use tar, ssh, wget, rsync & gpg with custom roles system (https://computing.llnl.gov/linux/genders.html can be an alternative) to manage configuration and packages. Our environment is probably still small to hit the limits of these tools.

Our challenge with managing hadoop cluster is the lack of standard interfaces to reliably monitor the cluster. Standard unix tools expect process to exit with non zero status on error and counters to be positive numbers.

IMHO whats needed here are features like HADOOP-6728 & HADOOP-7144, make them consistent across all components and integrate them with existing tools, HADOOP-7324 .

bq. Zeroconf is great for resolving service location.

As part of this proposal, are there plans to update how hadoop daemon and client configurations are handled or is this specific to HMS?

bq. Bittorrent is much faster than install software from yum repository for large scale system.

Bittorrent is a file sharing protocol and yum is a utility for rpm package management. I guess you mean to say bittorrent is faster to distribute files than http. If RPM is choose as the package format but don't want to use yum, HMS may need to implement another rpm based package management.

Alternatively, this could just be a yum plugin.


thats my 0.2 cents. But hey, if you want to invest your time in writing Yet Another Monitoring System ;-), I wish you all the best!

> Hadoop Management System (Umbrella)
> -----------------------------------
>
>                 Key: HADOOP-7417
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7417
>             Project: Hadoop Common
>          Issue Type: New Feature
>         Environment: Java 6, Linux
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>
> The primary goal of Hadoop Management System is to build a component around management and deployment of Hadoop related projects. This includes software installation, configuration, application orchestration, deployment automation and monitoring Hadoop.
> Prototype demo source code can be obtained from:
> http://github.com/macroadster/hms

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16760-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 27 00:58:10 2011
Return-Path: <common-issues-return-16760-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8C3F96C83
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 27 Jun 2011 00:58:10 +0000 (UTC)
Received: (qmail 81375 invoked by uid 500); 27 Jun 2011 00:58:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81171 invoked by uid 500); 27 Jun 2011 00:58:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81160 invoked by uid 99); 27 Jun 2011 00:58:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 00:58:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 00:58:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 033F74328FB
	for <common-issues@hadoop.apache.org>; Mon, 27 Jun 2011 00:57:48 +0000 (UTC)
Date: Mon, 27 Jun 2011 00:57:48 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <829323455.42387.1309136268010.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <407429641.30834.1308786767442.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7417) Hadoop Management System
 (Umbrella)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13055258#comment-13055258 ] 

Allen Wittenauer commented on HADOOP-7417:
------------------------------------------

The more I think about this and read the comments it is pretty clear what needs to happen.  Eli already hinted at it, but I'm going to expand on his suggestion.  This needs to get remade as a generic toolset and sent to the incubator as a generic Apache configuration management system.  I can't think of any Apache licensed systems and this would be a good thing for the ASF to have in the portfolio.  The biggest hurdle it has is the massive reliance on GPL'd bits.  (There are so many components that this is a configuration management system that requires a configuration management system to actually configure...)

As it currently stands, this functionality is completely outside of core Hadoop.  So I'm already -1 on this going into common.  



> Hadoop Management System (Umbrella)
> -----------------------------------
>
>                 Key: HADOOP-7417
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7417
>             Project: Hadoop Common
>          Issue Type: New Feature
>         Environment: Java 6, Linux
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>
> The primary goal of Hadoop Management System is to build a component around management and deployment of Hadoop related projects. This includes software installation, configuration, application orchestration, deployment automation and monitoring Hadoop.
> Prototype demo source code can be obtained from:
> http://github.com/macroadster/hms

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16761-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 27 02:08:11 2011
Return-Path: <common-issues-return-16761-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 80CCC4C4A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 27 Jun 2011 02:08:11 +0000 (UTC)
Received: (qmail 44652 invoked by uid 500); 27 Jun 2011 02:08:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44592 invoked by uid 500); 27 Jun 2011 02:08:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44584 invoked by uid 99); 27 Jun 2011 02:08:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 02:08:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 02:08:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 67EB7433212
	for <common-issues@hadoop.apache.org>; Mon, 27 Jun 2011 02:07:47 +0000 (UTC)
Date: Mon, 27 Jun 2011 02:07:47 +0000 (UTC)
From: "steven zhuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <504300446.42541.1309140467421.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1304349125.35232.1308881147698.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7425) ReflectionUtils.setConf would
 configure anything Configurable twice
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

steven zhuang updated HADOOP-7425:
----------------------------------

    Description: 
In  the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.

In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number. 


public static void setConf(Object theObject, Configuration conf) {
    if (conf != null) {
      if (theObject instanceof Configurable) {
        ((Configurable) theObject).setConf(conf);
      }
      setJobConf(theObject, conf);
    }
  }


  was:
In  the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number. 


public static void setConf(Object theObject, Configuration conf) {
    if (conf != null) {
      if (theObject instanceof Configurable) {
        ((Configurable) theObject).setConf(conf);
      }
      setJobConf(theObject, conf);
    }
  }


        Summary: ReflectionUtils.setConf would configure anything Configurable twice  (was: ReflectionUtils.setConf would configure the KeyFieldBasedPartitioner twice in Hadoop 0.21.0, when KeyFieldBasedPartitioner is an Configurable instance)

> ReflectionUtils.setConf would configure anything Configurable twice
> -------------------------------------------------------------------
>
>                 Key: HADOOP-7425
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7425
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: steven zhuang
>
> In  the setConf method of org.apache.hadoop.util.ReflectionUtils, any instance of Configurable would be configured twice.
> In 0.21.0, KeyFieldBasedPartitioner implements the Configurable interface. When configured twice, it get two KeyDescription and gives out wrong partition number. 
> public static void setConf(Object theObject, Configuration conf) {
>     if (conf != null) {
>       if (theObject instanceof Configurable) {
>         ((Configurable) theObject).setConf(conf);
>       }
>       setJobConf(theObject, conf);
>     }
>   }

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16762-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 27 02:45:16 2011
Return-Path: <common-issues-return-16762-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4D8E24187
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 27 Jun 2011 02:45:16 +0000 (UTC)
Received: (qmail 67460 invoked by uid 500); 27 Jun 2011 02:45:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67227 invoked by uid 500); 27 Jun 2011 02:45:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67208 invoked by uid 99); 27 Jun 2011 02:45:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 02:45:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 02:45:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 70FB5433B0A
	for <common-issues@hadoop.apache.org>; Mon, 27 Jun 2011 02:44:47 +0000 (UTC)
Date: Mon, 27 Jun 2011 02:44:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1634277813.42585.1309142687459.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-310) Additional constructor requested in
 BytesWritable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-310:
----------------------------------

    Assignee: Brock Noland  (was: Owen O'Malley)
      Status: Patch Available  (was: Open)

+1, the patch looks good to me.

Marking this as patch available so Hudson runs test-patch. I'll commit this pending clean Hudson results.

Also reassigning this to you, Brock.

> Additional constructor requested in BytesWritable
> -------------------------------------------------
>
>                 Key: HADOOP-310
>                 URL: https://issues.apache.org/jira/browse/HADOOP-310
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: p sutter
>            Assignee: Brock Noland
>            Priority: Minor
>         Attachments: bytes-writable-zero-copy-interface-0.patch, bytes-writable-zero-copy-interface-1.patch, bytes-writable-zero-copy-interface-2.patch
>
>
> It would be grand if BytesWritable.java had an additional constructor as below. This allows me to use the BytesWritable class without doing a buffer copy, since we have a less-than-fully-utilized byte array holding our key.
> Thanks!
>  
>  /**
>    * Create a BytesWritable using the byte array as the initial value.
>    * @param bytes This array becomes the backing storage for the object.
>    */
>   public BytesWritable(byte[] bytes, int size) {
>     this.bytes = bytes;
>     this.size = size;
>   }
>   

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16763-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 27 03:05:25 2011
Return-Path: <common-issues-return-16763-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 644614E22
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 27 Jun 2011 03:05:25 +0000 (UTC)
Received: (qmail 73695 invoked by uid 500); 27 Jun 2011 03:05:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73446 invoked by uid 500); 27 Jun 2011 03:05:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73413 invoked by uid 99); 27 Jun 2011 03:05:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 03:05:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 03:05:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A047B433EEA
	for <common-issues@hadoop.apache.org>; Mon, 27 Jun 2011 03:04:47 +0000 (UTC)
Date: Mon, 27 Jun 2011 03:04:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2105186611.42606.1309143887653.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-310) Additional constructor requested in
 BytesWritable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13055303#comment-13055303 ] 

Hadoop QA commented on HADOOP-310:
----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12483841/bytes-writable-zero-copy-interface-2.patch
  against trunk revision 1139947.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/677//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/677//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/677//console

This message is automatically generated.

> Additional constructor requested in BytesWritable
> -------------------------------------------------
>
>                 Key: HADOOP-310
>                 URL: https://issues.apache.org/jira/browse/HADOOP-310
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: p sutter
>            Assignee: Brock Noland
>            Priority: Minor
>         Attachments: bytes-writable-zero-copy-interface-0.patch, bytes-writable-zero-copy-interface-1.patch, bytes-writable-zero-copy-interface-2.patch
>
>
> It would be grand if BytesWritable.java had an additional constructor as below. This allows me to use the BytesWritable class without doing a buffer copy, since we have a less-than-fully-utilized byte array holding our key.
> Thanks!
>  
>  /**
>    * Create a BytesWritable using the byte array as the initial value.
>    * @param bytes This array becomes the backing storage for the object.
>    */
>   public BytesWritable(byte[] bytes, int size) {
>     this.bytes = bytes;
>     this.size = size;
>   }
>   

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16764-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 27 03:41:33 2011
Return-Path: <common-issues-return-16764-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CD4644D90
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 27 Jun 2011 03:41:33 +0000 (UTC)
Received: (qmail 87303 invoked by uid 500); 27 Jun 2011 03:41:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87069 invoked by uid 500); 27 Jun 2011 03:41:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87046 invoked by uid 99); 27 Jun 2011 03:41:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 03:41:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 03:41:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 98ABE4328E5
	for <common-issues@hadoop.apache.org>; Mon, 27 Jun 2011 03:40:50 +0000 (UTC)
Date: Mon, 27 Jun 2011 03:40:50 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <169876195.42637.1309146050622.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-310) Additional constructor requested in
 BytesWritable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-310:
----------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I've just committed this.

Thanks a lot for the contribution, Brock!

> Additional constructor requested in BytesWritable
> -------------------------------------------------
>
>                 Key: HADOOP-310
>                 URL: https://issues.apache.org/jira/browse/HADOOP-310
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: p sutter
>            Assignee: Brock Noland
>            Priority: Minor
>         Attachments: bytes-writable-zero-copy-interface-0.patch, bytes-writable-zero-copy-interface-1.patch, bytes-writable-zero-copy-interface-2.patch
>
>
> It would be grand if BytesWritable.java had an additional constructor as below. This allows me to use the BytesWritable class without doing a buffer copy, since we have a less-than-fully-utilized byte array holding our key.
> Thanks!
>  
>  /**
>    * Create a BytesWritable using the byte array as the initial value.
>    * @param bytes This array becomes the backing storage for the object.
>    */
>   public BytesWritable(byte[] bytes, int size) {
>     this.bytes = bytes;
>     this.size = size;
>   }
>   

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16765-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 27 03:41:35 2011
Return-Path: <common-issues-return-16765-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BB1B84DA7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 27 Jun 2011 03:41:35 +0000 (UTC)
Received: (qmail 87402 invoked by uid 500); 27 Jun 2011 03:41:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87076 invoked by uid 500); 27 Jun 2011 03:41:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87045 invoked by uid 99); 27 Jun 2011 03:41:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 03:41:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 03:41:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7C8664328E2
	for <common-issues@hadoop.apache.org>; Mon, 27 Jun 2011 03:40:50 +0000 (UTC)
Date: Mon, 27 Jun 2011 03:40:50 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <396019667.42634.1309146050506.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <817618820.42234.1309133387608.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7427) syntax error in
 smart-apply-patch.sh
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13055310#comment-13055310 ] 

Todd Lipcon commented on HADOOP-7427:
-------------------------------------

strange, what version of bash is this? is this on one of the build boxes or your own machine?

> syntax error in smart-apply-patch.sh 
> -------------------------------------
>
>                 Key: HADOOP-7427
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7427
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Tsz Wo (Nicholas), SZE
>
> {noformat}
>      [exec]     Finished build.
>      [exec] hdfs/src/test/bin/smart-apply-patch.sh: line 60: syntax error in conditional expression: unexpected token `('
> BUILD FAILED
> hdfs/build.xml:1595: exec returned: 1
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16766-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 27 05:17:21 2011
Return-Path: <common-issues-return-16766-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 23B1A44C0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 27 Jun 2011 05:17:21 +0000 (UTC)
Received: (qmail 32086 invoked by uid 500); 27 Jun 2011 05:17:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31710 invoked by uid 500); 27 Jun 2011 05:17:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31681 invoked by uid 99); 27 Jun 2011 05:17:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 05:17:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 05:17:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C7520433E57
	for <common-issues@hadoop.apache.org>; Mon, 27 Jun 2011 05:16:47 +0000 (UTC)
Date: Mon, 27 Jun 2011 05:16:47 +0000 (UTC)
From: "Priyo Mustafi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <812798706.42710.1309151807812.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <557825167.38099.1306198007453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7324) Ganglia plugins for metrics v2
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Priyo Mustafi updated HADOOP-7324:
----------------------------------

    Attachment: HADOOP-7324.patch

I have done some testing with 3.0 but not testing on 3.1 as we don't have 3.1 on our installation. I couldn't ascertain whether Ganglia can handle sparse updates so kept the default option as dense updates. If somebody can confirm that it can handle, sparse, then I can remove the logic for caching and doing dense updates.

> Ganglia plugins for metrics v2
> ------------------------------
>
>                 Key: HADOOP-7324
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7324
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.203.0, 0.23.0
>            Reporter: Luke Lu
>            Priority: Blocker
>              Labels: regression
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7324.patch
>
>
> Although, all metrics in metrics v2 are exposed via the standard JMX mechanisms, most users are using Ganglia to collect metrics.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16767-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 27 16:06:12 2011
Return-Path: <common-issues-return-16767-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 91599483D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 27 Jun 2011 16:06:12 +0000 (UTC)
Received: (qmail 26189 invoked by uid 500); 27 Jun 2011 16:06:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25981 invoked by uid 500); 27 Jun 2011 16:06:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25965 invoked by uid 99); 27 Jun 2011 16:06:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 16:06:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 16:06:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9A35243437C
	for <common-issues@hadoop.apache.org>; Mon, 27 Jun 2011 16:05:47 +0000 (UTC)
Date: Mon, 27 Jun 2011 16:05:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1410249436.44140.1309190747628.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <557825167.38099.1306198007453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7324) Ganglia plugins for metrics v2
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13055616#comment-13055616 ] 

Luke Lu commented on HADOOP-7324:
---------------------------------

Thanks for the nice patch, Priyo!

Here is a preliminary review:

# Like I mentioned in our email exchange, I'm hesitant to make the impl classes public and use them directly in the sink. It breaks abstraction that makes certain later optimizations impossible. A better approach would be using the builtin visitor interface. That said, it might be OK in 0.20.20x series, as they're mostly in maintenance mode.
# In the #init, please log the exception instead of printStackTrace.
# The #getTag method is only used to get the context of the record, which is unnecessary, as the record has a #context method.
# The #parseSocket method is unnecessary, please reuse metrics2.util.Servers#parse.
# The sparse/dense logic in #putMetrics can be improved a bit: the cache is looked up twice in dense mode. You can have #update method return the record once in dense mode.
# Ganglia treats a metric as counter if slope is positive and gauge for anything else. Having to specify slope explicitly in the conf for every counter is a tedious chore we can avoid. This is a deficiency in the existing metrics1 implementation as well.



> Ganglia plugins for metrics v2
> ------------------------------
>
>                 Key: HADOOP-7324
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7324
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.203.0, 0.23.0
>            Reporter: Luke Lu
>            Priority: Blocker
>              Labels: regression
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7324.patch
>
>
> Although, all metrics in metrics v2 are exposed via the standard JMX mechanisms, most users are using Ganglia to collect metrics.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16768-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 27 16:49:10 2011
Return-Path: <common-issues-return-16768-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 63D8644EE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 27 Jun 2011 16:49:10 +0000 (UTC)
Received: (qmail 88434 invoked by uid 500); 27 Jun 2011 16:49:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88346 invoked by uid 500); 27 Jun 2011 16:49:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88318 invoked by uid 99); 27 Jun 2011 16:49:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 16:49:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 16:49:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1BAEB43485B
	for <common-issues@hadoop.apache.org>; Mon, 27 Jun 2011 16:48:48 +0000 (UTC)
Date: Mon, 27 Jun 2011 16:48:48 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2091353978.44279.1309193328110.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <407429641.30834.1308786767442.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7417) Hadoop Management System
 (Umbrella)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13055645#comment-13055645 ] 

Eric Yang commented on HADOOP-7417:
-----------------------------------

bq. As part of this proposal, are there plans to update how hadoop daemon and client configurations are handled or is this specific to HMS?

It is possible to add zeroconf to hadoop.  There were discussion of zeroconf on and off in hadoop jiras and mailing list.

bq. Bittorrent is a file sharing protocol and yum is a utility for rpm package management. I guess you mean to say bittorrent is faster to distribute files than http. If RPM is choose as the package format but don't want to use yum, HMS may need to implement another rpm based package management.

HMS is currently designed to install software as a stack for the torrent implementation.  If the RPM dependencies are not resolved with in the software stack or the base OS, it will fail the installation instead of resolving the dependency and download from yum.  Hadoop software stack is small.  Hence, Build time package dependencies declaration and release engineer blessed software stack should be sufficient for now.  HMS does not intend to create another package management, but it can utilize platform's package management system. i.e. .rpm/.deb on linux, and .pkg on Mac.

bq. Alternatively, this could just be a yum plugin.

There is a yum bittorrent proposal back in 2008, but it has not come to fruit yet.  We may utilize it, if this is implemented in the future.

bq. The biggest hurdle it has is the massive reliance on GPL'd bits. (There are so many components that this is a configuration management system that requires a configuration management system to actually configure...)

Allen, system call to GPL software does not automatically make caller GPL.  See: 

[http://www.ifross.org/en/program-forks-gpl-licensed-program-system-or-vice-versa-call-derivative-work]

HMS agent can call Linux utilities as long as the software runs in separated processes.  Hence, your concern should be addressed.  HMS has already ensure its libraries are Apache Software License or compatible license.

> Hadoop Management System (Umbrella)
> -----------------------------------
>
>                 Key: HADOOP-7417
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7417
>             Project: Hadoop Common
>          Issue Type: New Feature
>         Environment: Java 6, Linux
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>
> The primary goal of Hadoop Management System is to build a component around management and deployment of Hadoop related projects. This includes software installation, configuration, application orchestration, deployment automation and monitoring Hadoop.
> Prototype demo source code can be obtained from:
> http://github.com/macroadster/hms

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16769-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 27 17:17:12 2011
Return-Path: <common-issues-return-16769-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E6651487F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 27 Jun 2011 17:17:11 +0000 (UTC)
Received: (qmail 32887 invoked by uid 500); 27 Jun 2011 17:17:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32790 invoked by uid 500); 27 Jun 2011 17:17:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32772 invoked by uid 99); 27 Jun 2011 17:17:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 17:17:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 17:17:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6ED9C4344C8
	for <common-issues@hadoop.apache.org>; Mon, 27 Jun 2011 17:16:47 +0000 (UTC)
Date: Mon, 27 Jun 2011 17:16:47 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <765428267.44377.1309195007450.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <407429641.30834.1308786767442.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7417) Hadoop Management System
 (Umbrella)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13055664#comment-13055664 ] 

Allen Wittenauer commented on HADOOP-7417:
------------------------------------------

bq. Allen, system call to GPL software does not automatically make caller GPL.

I'm fully aware of that.  The problem is that there isn't much here outside of GPL'ed and other licensed code.  This would be much more interesting and useful if it was self-contained. A configuration management system that needs a configuration management system to install sort of defeats the purpose.  All of the great ones require a minuscule amount of stub in place before they can install themselves.  I don't see that happening here.

> Hadoop Management System (Umbrella)
> -----------------------------------
>
>                 Key: HADOOP-7417
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7417
>             Project: Hadoop Common
>          Issue Type: New Feature
>         Environment: Java 6, Linux
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>
> The primary goal of Hadoop Management System is to build a component around management and deployment of Hadoop related projects. This includes software installation, configuration, application orchestration, deployment automation and monitoring Hadoop.
> Prototype demo source code can be obtained from:
> http://github.com/macroadster/hms

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16770-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 27 19:00:11 2011
Return-Path: <common-issues-return-16770-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BDE1747E6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 27 Jun 2011 19:00:11 +0000 (UTC)
Received: (qmail 67945 invoked by uid 500); 27 Jun 2011 19:00:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67754 invoked by uid 500); 27 Jun 2011 19:00:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67734 invoked by uid 99); 27 Jun 2011 19:00:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 19:00:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 19:00:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6FCAC434D15
	for <common-issues@hadoop.apache.org>; Mon, 27 Jun 2011 18:59:47 +0000 (UTC)
Date: Mon, 27 Jun 2011 18:59:47 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1834208151.44579.1309201187454.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <407429641.30834.1308786767442.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7417) Hadoop Management System
 (Umbrella)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13055698#comment-13055698 ] 

Eric Yang commented on HADOOP-7417:
-----------------------------------

bq. This would be much more interesting and useful if it was self-contained. A configuration management system that needs a configuration management system to install sort of defeats the purpose. All of the great ones require a minuscule amount of stub in place before they can install themselves. I don't see that happening here.

There is a difference between configuration management system and package management system.  The statements above are mixing package management with configuration management.  For package management system, HMS can tap into existing package management system provided by the OS.  HMS also supports tarball deployment, and role based software stack installation.  Hence, it does not require a package management system to operate, if the customer choose to use tarball only.  

In the base OS, there is nothing manages configuration files for Hadoop.  Configuration management is usually operate through template and scripting.  Operation can use puppet to configure the system, and commit the configuration files to source tree repository.  Chef uses MangoDB to store states.  HMS tracks the configuration history and store the state changes in Zookeeper.  

For both package and config management, there are strong cases to use HMS than generic deployment system because HMS can provide better deployment experience and improve deployment efficiency.

> Hadoop Management System (Umbrella)
> -----------------------------------
>
>                 Key: HADOOP-7417
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7417
>             Project: Hadoop Common
>          Issue Type: New Feature
>         Environment: Java 6, Linux
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>
> The primary goal of Hadoop Management System is to build a component around management and deployment of Hadoop related projects. This includes software installation, configuration, application orchestration, deployment automation and monitoring Hadoop.
> Prototype demo source code can be obtained from:
> http://github.com/macroadster/hms

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16771-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 27 19:40:09 2011
Return-Path: <common-issues-return-16771-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C6A0B41C0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 27 Jun 2011 19:40:09 +0000 (UTC)
Received: (qmail 13292 invoked by uid 500); 27 Jun 2011 19:40:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13035 invoked by uid 500); 27 Jun 2011 19:40:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13027 invoked by uid 99); 27 Jun 2011 19:40:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 19:40:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 19:40:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6F3744352AA
	for <common-issues@hadoop.apache.org>; Mon, 27 Jun 2011 19:39:47 +0000 (UTC)
Date: Mon, 27 Jun 2011 19:39:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <174265054.44671.1309203587452.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7428) IPC connection is orphaned with null
 'out' member
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

IPC connection is orphaned with null 'out' member
-------------------------------------------------

                 Key: HADOOP-7428
                 URL: https://issues.apache.org/jira/browse/HADOOP-7428
             Project: Hadoop Common
          Issue Type: Bug
          Components: ipc
    Affects Versions: 0.23.0
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon


We had a situation a JT ended up in a state where a certain user could not submit a job, due to an NPE on the following line in {{sendParam}}:
{code}
synchronized (Connection.this.out) {
{code}
Looking at the code, my guess is that an RTE was thrown in setupIOstreams, which only catches IOE. This could leave the connection in a half-setup state which is never cleaned up and also cannot perform IPCs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16772-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 27 22:17:11 2011
Return-Path: <common-issues-return-16772-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B7FA04569
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 27 Jun 2011 22:17:11 +0000 (UTC)
Received: (qmail 18529 invoked by uid 500); 27 Jun 2011 22:17:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18286 invoked by uid 500); 27 Jun 2011 22:17:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18266 invoked by uid 99); 27 Jun 2011 22:17:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 22:17:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 22:17:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C3A7D435632
	for <common-issues@hadoop.apache.org>; Mon, 27 Jun 2011 22:16:47 +0000 (UTC)
Date: Mon, 27 Jun 2011 22:16:47 +0000 (UTC)
From: "Priyo Mustafi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1562847727.45084.1309213007798.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <557825167.38099.1306198007453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7324) Ganglia plugins for metrics v2
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Priyo Mustafi reassigned HADOOP-7324:
-------------------------------------

    Assignee: Priyo Mustafi

> Ganglia plugins for metrics v2
> ------------------------------
>
>                 Key: HADOOP-7324
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7324
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.203.0, 0.23.0
>            Reporter: Luke Lu
>            Assignee: Priyo Mustafi
>            Priority: Blocker
>              Labels: regression
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7324.patch
>
>
> Although, all metrics in metrics v2 are exposed via the standard JMX mechanisms, most users are using Ganglia to collect metrics.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16773-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon Jun 27 22:32:59 2011
Return-Path: <common-issues-return-16773-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AB6C348E7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 27 Jun 2011 22:32:59 +0000 (UTC)
Received: (qmail 39702 invoked by uid 500); 27 Jun 2011 22:32:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39569 invoked by uid 500); 27 Jun 2011 22:32:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39561 invoked by uid 99); 27 Jun 2011 22:32:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 22:32:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Jun 2011 22:32:56 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A8B674132B9
	for <common-issues@hadoop.apache.org>; Mon, 27 Jun 2011 22:32:35 +0000 (UTC)
Date: Mon, 27 Jun 2011 22:32:35 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <726006040.15.1309213955687.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <817618820.42234.1309133387608.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7427) syntax error in
 smart-apply-patch.sh
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13055883#comment-13055883 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7427:
------------------------------------------------

{noformat}
$  bash --version
GNU bash, version 3.00.15(1)-release (x86_64-redhat-linux-gnu)
Copyright (C) 2004 Free Software Foundation, Inc.
{noformat}


> syntax error in smart-apply-patch.sh 
> -------------------------------------
>
>                 Key: HADOOP-7427
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7427
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Tsz Wo (Nicholas), SZE
>
> {noformat}
>      [exec]     Finished build.
>      [exec] hdfs/src/test/bin/smart-apply-patch.sh: line 60: syntax error in conditional expression: unexpected token `('
> BUILD FAILED
> hdfs/build.xml:1595: exec returned: 1
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16774-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 00:01:41 2011
Return-Path: <common-issues-return-16774-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8399841FF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 00:01:41 +0000 (UTC)
Received: (qmail 74980 invoked by uid 500); 28 Jun 2011 00:01:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74759 invoked by uid 500); 28 Jun 2011 00:01:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74751 invoked by uid 99); 28 Jun 2011 00:01:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 00:01:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 00:01:38 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 993E34152EE
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 00:01:17 +0000 (UTC)
Date: Tue, 28 Jun 2011 00:01:17 +0000 (UTC)
From: "Jagane Sundar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1589822549.187.1309219277624.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <407429641.30834.1308786767442.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7417) Hadoop Management System
 (Umbrella)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056217#comment-13056217 ] 

Jagane Sundar commented on HADOOP-7417:
---------------------------------------

Let me expand a little bit on Eric's explanation:

HMS is a full lifecycle management system for Hadoop. It can create hadoop clusters, automatically manage the configuration of hadoop clusters, tear down hadoop clusters and upgrade hadoop clusters.

HMS consists of highly available HMS Controller(s). A HMS Agent on each Hadoop Node communicates with the HMS Controller(s). HMS Users issue commands to the HMS Controller using a REST API. Example commands can create/teardown/configure/upgrade Hadoop clusters.

HMS aspires to use OS native packaging mechanisms for installing hadoop, i.e. rpms on RedHat, .debs on Debian/Ubu. tarball installation for unsupported Unix like operating systems is also supported.

HMS differs from Chef/Puppet and other deployment frameworks in several important ways.
1. HMS is built to be highly available from day 1. HMS controllers can die and be restarted. HMS Agents and the machines they run on can spontaneously reboot. HMS itself will continue to be available.
2. HMS is meant to scale to the size of a 20,000 plus server Datacenter.
3. HMS is designed to be the building block for constructing a self service Hadoop cloud, wherein users can create/configure/teardown Hadoops.

> Hadoop Management System (Umbrella)
> -----------------------------------
>
>                 Key: HADOOP-7417
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7417
>             Project: Hadoop Common
>          Issue Type: New Feature
>         Environment: Java 6, Linux
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>
> The primary goal of Hadoop Management System is to build a component around management and deployment of Hadoop related projects. This includes software installation, configuration, application orchestration, deployment automation and monitoring Hadoop.
> Prototype demo source code can be obtained from:
> http://github.com/macroadster/hms

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16775-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 00:13:39 2011
Return-Path: <common-issues-return-16775-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7F6AA4127
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 00:13:39 +0000 (UTC)
Received: (qmail 85218 invoked by uid 500); 28 Jun 2011 00:13:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85166 invoked by uid 500); 28 Jun 2011 00:13:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85155 invoked by uid 99); 28 Jun 2011 00:13:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 00:13:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 00:13:37 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DAD474157E7
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 00:13:17 +0000 (UTC)
Date: Tue, 28 Jun 2011 00:13:17 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1272267415.238.1309219997893.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1593248389.10179.1308212149301.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7399) Remove Writable from ipc.Client and
 ipc.Server.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7399:
-----------------------------------------

    Status: Open  (was: Patch Available)

> Remove Writable from ipc.Client and ipc.Server.
> -----------------------------------------------
>
>                 Key: HADOOP-7399
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7399
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7399.3.patch, HADOOP-7399.9.patch
>
>
>   This jira proposes to remove the dependency of ipc client and server on Writable. I think this will be an important step towards making an RpcEngine truly configurable, without having to use tunnel protocols as in case of AvroRPCEngine.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16776-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 00:13:39 2011
Return-Path: <common-issues-return-16776-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C567B4132
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 00:13:39 +0000 (UTC)
Received: (qmail 85472 invoked by uid 500); 28 Jun 2011 00:13:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85177 invoked by uid 500); 28 Jun 2011 00:13:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85167 invoked by uid 99); 28 Jun 2011 00:13:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 00:13:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 00:13:38 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 40A4E4157F1
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 00:13:18 +0000 (UTC)
Date: Tue, 28 Jun 2011 00:13:18 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <192564212.247.1309219998261.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1593248389.10179.1308212149301.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7399) Remove Writable from ipc.Client and
 ipc.Server.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7399:
-----------------------------------------

    Status: Patch Available  (was: Open)

> Remove Writable from ipc.Client and ipc.Server.
> -----------------------------------------------
>
>                 Key: HADOOP-7399
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7399
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7399.3.patch, HADOOP-7399.9.patch
>
>
>   This jira proposes to remove the dependency of ipc client and server on Writable. I think this will be an important step towards making an RpcEngine truly configurable, without having to use tunnel protocols as in case of AvroRPCEngine.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16777-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 00:13:41 2011
Return-Path: <common-issues-return-16777-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 26C674142
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 00:13:41 +0000 (UTC)
Received: (qmail 85765 invoked by uid 500); 28 Jun 2011 00:13:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85713 invoked by uid 500); 28 Jun 2011 00:13:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85695 invoked by uid 99); 28 Jun 2011 00:13:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 00:13:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 00:13:38 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 76A444157DD
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 00:13:17 +0000 (UTC)
Date: Tue, 28 Jun 2011 00:13:17 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <744599249.229.1309219997482.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1593248389.10179.1308212149301.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7399) Remove Writable from ipc.Client and
 ipc.Server.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7399:
-----------------------------------------

    Attachment: HADOOP-7399.9.patch

Updated patch, with a little more cleaned up interfaces. The server uses special "calls" for sasl messages as well. I have maintained the same mechanism by adding a SaslServerCall that implements ServerCall. SaslServerCall is not RpcEngine specific and still uses Writable because client assumes a writable format. Encoding of sasl messages could also be delegated to the RpcEngine but I would prefer to do that in a separate jira. 

> Remove Writable from ipc.Client and ipc.Server.
> -----------------------------------------------
>
>                 Key: HADOOP-7399
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7399
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7399.3.patch, HADOOP-7399.9.patch
>
>
>   This jira proposes to remove the dependency of ipc client and server on Writable. I think this will be an important step towards making an RpcEngine truly configurable, without having to use tunnel protocols as in case of AvroRPCEngine.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16778-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 00:35:41 2011
Return-Path: <common-issues-return-16778-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5E84D4397
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 00:35:41 +0000 (UTC)
Received: (qmail 5952 invoked by uid 500); 28 Jun 2011 00:35:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5893 invoked by uid 500); 28 Jun 2011 00:35:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5883 invoked by uid 99); 28 Jun 2011 00:35:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 00:35:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 00:35:38 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 66CB3415FF5
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 00:35:17 +0000 (UTC)
Date: Tue, 28 Jun 2011 00:35:17 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <277866513.279.1309221317417.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1593248389.10179.1308212149301.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7399) Remove Writable from ipc.Client
 and ipc.Server.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056229#comment-13056229 ] 

Hadoop QA commented on HADOOP-7399:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12484365/HADOOP-7399.9.patch
  against trunk revision 1140010.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The applied patch generated 1078 javac compiler warnings (more than the trunk's current 1067 warnings).

    -1 findbugs.  The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/678//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/678//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/678//console

This message is automatically generated.

> Remove Writable from ipc.Client and ipc.Server.
> -----------------------------------------------
>
>                 Key: HADOOP-7399
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7399
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7399.3.patch, HADOOP-7399.9.patch
>
>
>   This jira proposes to remove the dependency of ipc client and server on Writable. I think this will be an important step towards making an RpcEngine truly configurable, without having to use tunnel protocols as in case of AvroRPCEngine.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16779-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 00:37:39 2011
Return-Path: <common-issues-return-16779-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2060843EA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 00:37:39 +0000 (UTC)
Received: (qmail 6746 invoked by uid 500); 28 Jun 2011 00:37:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6710 invoked by uid 500); 28 Jun 2011 00:37:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6699 invoked by uid 99); 28 Jun 2011 00:37:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 00:37:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 00:37:37 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 68C544160A1
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 00:37:17 +0000 (UTC)
Date: Tue, 28 Jun 2011 00:37:17 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <480132221.284.1309221437425.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <407429641.30834.1308786767442.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7417) Hadoop Management System
 (Umbrella)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056231#comment-13056231 ] 

Allen Wittenauer commented on HADOOP-7417:
------------------------------------------

Something has to maintain the packages and config settings of HMS itself.  "Kickstart" is not a valid answer.  

Something else being missed here is that Hadoop is just one component of a working data center.  If I have to deploy bcfg2/puppet/chef/cfengine/... to manage my LDAP, Azkaban, Voldemort, ... nodes, why would I have something different for Hadoop when I can just use the pre-existing services?  (Most of these systems are HA and can handle 20k nodes if one knows how to deploy them properly.)

Also, why isn't this just rolled into Whirr?  Isn't the whole point of that project software provisioning for "cloud" services?  Yet another reason this doesn't belong in Hadoop-proper.

> Hadoop Management System (Umbrella)
> -----------------------------------
>
>                 Key: HADOOP-7417
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7417
>             Project: Hadoop Common
>          Issue Type: New Feature
>         Environment: Java 6, Linux
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>
> The primary goal of Hadoop Management System is to build a component around management and deployment of Hadoop related projects. This includes software installation, configuration, application orchestration, deployment automation and monitoring Hadoop.
> Prototype demo source code can be obtained from:
> http://github.com/macroadster/hms

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16780-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 01:27:40 2011
Return-Path: <common-issues-return-16780-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 92D0D40BC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 01:27:40 +0000 (UTC)
Received: (qmail 47208 invoked by uid 500); 28 Jun 2011 01:27:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47138 invoked by uid 500); 28 Jun 2011 01:27:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47123 invoked by uid 99); 28 Jun 2011 01:27:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 01:27:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 01:27:37 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D66E3418273
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 01:27:16 +0000 (UTC)
Date: Tue, 28 Jun 2011 01:27:16 +0000 (UTC)
From: "Priyo Mustafi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1338937605.398.1309224436875.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <557825167.38099.1306198007453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7324) Ganglia plugins for metrics v2
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Priyo Mustafi updated HADOOP-7324:
----------------------------------

    Attachment: HADOOP-7324.patch

New patch with Luke's suggested changes

> Ganglia plugins for metrics v2
> ------------------------------
>
>                 Key: HADOOP-7324
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7324
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.203.0, 0.23.0
>            Reporter: Luke Lu
>            Assignee: Priyo Mustafi
>            Priority: Blocker
>              Labels: regression
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7324.patch, HADOOP-7324.patch
>
>
> Although, all metrics in metrics v2 are exposed via the standard JMX mechanisms, most users are using Ganglia to collect metrics.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16781-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 05:15:00 2011
Return-Path: <common-issues-return-16781-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 92E7862CA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 05:15:00 +0000 (UTC)
Received: (qmail 17110 invoked by uid 500); 28 Jun 2011 05:14:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16838 invoked by uid 500); 28 Jun 2011 05:14:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16820 invoked by uid 99); 28 Jun 2011 05:14:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 05:14:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 05:14:38 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 208C8435E36
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 05:14:17 +0000 (UTC)
Date: Tue, 28 Jun 2011 05:14:17 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <403145852.745.1309238057130.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7429) Common side of HDFS-2110
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Common side of HDFS-2110
------------------------

                 Key: HADOOP-7429
                 URL: https://issues.apache.org/jira/browse/HADOOP-7429
             Project: Hadoop Common
          Issue Type: Improvement
          Components: util
            Reporter: Eli Collins
            Assignee: Eli Collins
            Priority: Minor


Common side of HDFS-2110, adds a new IOUtils copy bytes method and cleanup.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16782-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 05:17:15 2011
Return-Path: <common-issues-return-16782-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 677AC6AD5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 05:17:15 +0000 (UTC)
Received: (qmail 19484 invoked by uid 500); 28 Jun 2011 05:17:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19212 invoked by uid 500); 28 Jun 2011 05:16:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19176 invoked by uid 99); 28 Jun 2011 05:16:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 05:16:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 05:16:38 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 20D36435EF5
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 05:16:17 +0000 (UTC)
Date: Tue, 28 Jun 2011 05:16:17 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <58403908.747.1309238177131.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <403145852.745.1309238057130.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7429) Common side of HDFS-2110
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7429:
--------------------------------

    Attachment: hadoop-7429-1.patch

Common side of the HDFS-2110 patch attached.

> Common side of HDFS-2110
> ------------------------
>
>                 Key: HADOOP-7429
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7429
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>         Attachments: hadoop-7429-1.patch
>
>
> Common side of HDFS-2110, adds a new IOUtils copy bytes method and cleanup.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16783-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 05:17:19 2011
Return-Path: <common-issues-return-16783-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A3C616B49
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 05:17:19 +0000 (UTC)
Received: (qmail 19567 invoked by uid 500); 28 Jun 2011 05:17:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19244 invoked by uid 500); 28 Jun 2011 05:17:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19175 invoked by uid 99); 28 Jun 2011 05:16:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 05:16:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 05:16:38 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 37592435EF7
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 05:16:17 +0000 (UTC)
Date: Tue, 28 Jun 2011 05:16:17 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1883852052.749.1309238177223.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <403145852.745.1309238057130.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7429) Common side of HDFS-2110
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7429:
--------------------------------

    Hadoop Flags: [Reviewed]
          Status: Patch Available  (was: Open)

atm reviewed this on HDFS-2110.

> Common side of HDFS-2110
> ------------------------
>
>                 Key: HADOOP-7429
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7429
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>         Attachments: hadoop-7429-1.patch
>
>
> Common side of HDFS-2110, adds a new IOUtils copy bytes method and cleanup.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16784-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 05:20:55 2011
Return-Path: <common-issues-return-16784-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F37FC6A50
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 05:20:54 +0000 (UTC)
Received: (qmail 21986 invoked by uid 500); 28 Jun 2011 05:20:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21748 invoked by uid 500); 28 Jun 2011 05:20:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21680 invoked by uid 99); 28 Jun 2011 05:20:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 05:20:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 05:20:38 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5E311435088
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 05:20:17 +0000 (UTC)
Date: Tue, 28 Jun 2011 05:20:17 +0000 (UTC)
From: "Jagane Sundar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1006102642.757.1309238417382.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <407429641.30834.1308786767442.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7417) Hadoop Management System
 (Umbrella)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056315#comment-13056315 ] 

Jagane Sundar commented on HADOOP-7417:
---------------------------------------

The goal of HMS is to provide a Hadoop specific management platform. While chef/puppet/bcfg2 and other scripted deployment tools are useful for simpler software, a complex software platform such as Hadoop could benefit from a management platform that is custom built for it.

With the hadoop.next platform starting to take shape, the job of installing and configuring Hadoop is going to need even more sophisticated technology. It is critical to look at management of Hadoop from a holistic perspective.

HMS plays nice with existing packaging mechanisms such as .rpms, .debs and tar files, which are the finest granularity building blocks.

Alan - I don't understand your comparison of Whirr with HMS. HMS has the potential to provide the ability for cloud service providers to provide a clone of Elastic Map Reduce. Correct me if I am wrong, but isn't Whirr a set of scripts for deploying Hadoop on an IaaS cloud?


> Hadoop Management System (Umbrella)
> -----------------------------------
>
>                 Key: HADOOP-7417
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7417
>             Project: Hadoop Common
>          Issue Type: New Feature
>         Environment: Java 6, Linux
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>
> The primary goal of Hadoop Management System is to build a component around management and deployment of Hadoop related projects. This includes software installation, configuration, application orchestration, deployment automation and monitoring Hadoop.
> Prototype demo source code can be obtained from:
> http://github.com/macroadster/hms

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16785-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 05:26:52 2011
Return-Path: <common-issues-return-16785-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 02B156AB7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 05:26:52 +0000 (UTC)
Received: (qmail 23229 invoked by uid 500); 28 Jun 2011 05:26:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23063 invoked by uid 500); 28 Jun 2011 05:26:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23018 invoked by uid 99); 28 Jun 2011 05:26:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 05:26:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 05:26:39 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6F943435290
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 05:26:19 +0000 (UTC)
Date: Tue, 28 Jun 2011 05:26:19 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <718942487.795.1309238779453.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <407429641.30834.1308786767442.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7417) Hadoop Management System
 (Umbrella)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056317#comment-13056317 ] 

Eli Collins commented on HADOOP-7417:
-------------------------------------

HMS is a separate project for the ecosystem of Hadoop related projects, this is out of scope for Hadoop common. How about moving the discussion of Whirr, HMS etc to the incubator thread.

> Hadoop Management System (Umbrella)
> -----------------------------------
>
>                 Key: HADOOP-7417
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7417
>             Project: Hadoop Common
>          Issue Type: New Feature
>         Environment: Java 6, Linux
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>
> The primary goal of Hadoop Management System is to build a component around management and deployment of Hadoop related projects. This includes software installation, configuration, application orchestration, deployment automation and monitoring Hadoop.
> Prototype demo source code can be obtained from:
> http://github.com/macroadster/hms

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16786-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 05:33:01 2011
Return-Path: <common-issues-return-16786-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 184B06C19
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 05:33:01 +0000 (UTC)
Received: (qmail 27623 invoked by uid 500); 28 Jun 2011 05:33:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27338 invoked by uid 500); 28 Jun 2011 05:32:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27300 invoked by uid 99); 28 Jun 2011 05:32:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 05:32:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 05:32:46 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 822B54354C2
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 05:32:26 +0000 (UTC)
Date: Tue, 28 Jun 2011 05:32:26 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1716662269.803.1309239146529.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <403145852.745.1309238057130.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7429) Common side of HDFS-2110
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056320#comment-13056320 ] 

Hadoop QA commented on HADOOP-7429:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12484392/hadoop-7429-1.patch
  against trunk revision 1140010.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/679//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/679//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/679//console

This message is automatically generated.

> Common side of HDFS-2110
> ------------------------
>
>                 Key: HADOOP-7429
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7429
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>         Attachments: hadoop-7429-1.patch
>
>
> Common side of HDFS-2110, adds a new IOUtils copy bytes method and cleanup.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16787-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 05:33:02 2011
Return-Path: <common-issues-return-16787-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E94566C6C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 05:33:02 +0000 (UTC)
Received: (qmail 28155 invoked by uid 500); 28 Jun 2011 05:33:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27356 invoked by uid 500); 28 Jun 2011 05:33:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27299 invoked by uid 99); 28 Jun 2011 05:32:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 05:32:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 05:32:46 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8C1834354C3
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 05:32:26 +0000 (UTC)
Date: Tue, 28 Jun 2011 05:32:26 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <499957638.804.1309239146570.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <403145852.745.1309238057130.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7429) Common side of HDFS-2110
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056321#comment-13056321 ] 

Eli Collins commented on HADOOP-7429:
-------------------------------------

This is mostly cleanup, the new IOUtils method is tested in HDFS-2110.

> Common side of HDFS-2110
> ------------------------
>
>                 Key: HADOOP-7429
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7429
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>         Attachments: hadoop-7429-1.patch
>
>
> Common side of HDFS-2110, adds a new IOUtils copy bytes method and cleanup.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16788-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 05:34:51 2011
Return-Path: <common-issues-return-16788-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4CC09616E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 05:34:51 +0000 (UTC)
Received: (qmail 29146 invoked by uid 500); 28 Jun 2011 05:34:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29062 invoked by uid 500); 28 Jun 2011 05:34:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29026 invoked by uid 99); 28 Jun 2011 05:34:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 05:34:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 05:34:37 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 539934355C4
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 05:34:17 +0000 (UTC)
Date: Tue, 28 Jun 2011 05:34:17 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <15167716.810.1309239257338.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <403145852.745.1309238057130.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7429) Add another IOUtils#copyBytes method
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7429:
--------------------------------

    Summary: Add another IOUtils#copyBytes method  (was: Common side of HDFS-2110)

> Add another IOUtils#copyBytes method
> ------------------------------------
>
>                 Key: HADOOP-7429
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7429
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>         Attachments: hadoop-7429-1.patch
>
>
> Common side of HDFS-2110, adds a new IOUtils copy bytes method and cleanup.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16789-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 05:34:58 2011
Return-Path: <common-issues-return-16789-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 46AF5617C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 05:34:58 +0000 (UTC)
Received: (qmail 29426 invoked by uid 500); 28 Jun 2011 05:34:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29164 invoked by uid 500); 28 Jun 2011 05:34:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29022 invoked by uid 99); 28 Jun 2011 05:34:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 05:34:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 05:34:37 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6760A4355C6
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 05:34:17 +0000 (UTC)
Date: Tue, 28 Jun 2011 05:34:17 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1747673115.812.1309239257420.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <403145852.745.1309238057130.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7429) Add another IOUtils#copyBytes
 method
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056322#comment-13056322 ] 

Hadoop QA commented on HADOOP-7429:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12484392/hadoop-7429-1.patch
  against trunk revision 1140010.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/680//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/680//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/680//console

This message is automatically generated.

> Add another IOUtils#copyBytes method
> ------------------------------------
>
>                 Key: HADOOP-7429
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7429
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>         Attachments: hadoop-7429-1.patch
>
>
> Common side of HDFS-2110, adds a new IOUtils copy bytes method and cleanup.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16790-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 05:36:59 2011
Return-Path: <common-issues-return-16790-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2FDCC61C1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 05:36:59 +0000 (UTC)
Received: (qmail 31779 invoked by uid 500); 28 Jun 2011 05:36:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31448 invoked by uid 500); 28 Jun 2011 05:36:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31423 invoked by uid 99); 28 Jun 2011 05:36:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 05:36:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 05:36:38 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1E7D94356D4
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 05:36:17 +0000 (UTC)
Date: Tue, 28 Jun 2011 05:36:17 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1244595049.814.1309239377120.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <403145852.745.1309238057130.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7429) Add another IOUtils#copyBytes method
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7429:
--------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I've committed this. Thanks atm!

> Add another IOUtils#copyBytes method
> ------------------------------------
>
>                 Key: HADOOP-7429
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7429
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>         Attachments: hadoop-7429-1.patch
>
>
> Common side of HDFS-2110, adds a new IOUtils copy bytes method and cleanup.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16791-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 07:13:00 2011
Return-Path: <common-issues-return-16791-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C9F2761B7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 07:13:00 +0000 (UTC)
Received: (qmail 17681 invoked by uid 500); 28 Jun 2011 07:13:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17267 invoked by uid 500); 28 Jun 2011 07:12:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17215 invoked by uid 99); 28 Jun 2011 07:12:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 07:12:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 07:12:37 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4CCD7436287
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 07:12:17 +0000 (UTC)
Date: Tue, 28 Jun 2011 07:12:17 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1254454082.946.1309245137311.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1875258417.25075.1308677148252.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7409) TestTFileByteArrays is failing on
 Hudson
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins resolved HADOOP-7409.
---------------------------------

    Resolution: Duplicate

Looks like a dupe.

> TestTFileByteArrays is failing on Hudson
> ----------------------------------------
>
>                 Key: HADOOP-7409
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7409
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>             Fix For: 0.23.0
>
>
> This test has failed in the last 4 nightly builds, as seen here: https://builds.apache.org/job/Hadoop-Common-trunk/
> I can't reproduce this failure on my machine, running the test either in isolation or as part of the full suite.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16792-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 07:14:51 2011
Return-Path: <common-issues-return-16792-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1AFD762C3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 07:14:51 +0000 (UTC)
Received: (qmail 20939 invoked by uid 500); 28 Jun 2011 07:14:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20778 invoked by uid 500); 28 Jun 2011 07:14:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20749 invoked by uid 99); 28 Jun 2011 07:14:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 07:14:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 07:14:38 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B9A554363BA
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 07:14:17 +0000 (UTC)
Date: Tue, 28 Jun 2011 07:14:17 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1723276134.955.1309245257757.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <7539864.45541295391523753.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7111) Several TFile tests failing when
 native libraries are present
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056358#comment-13056358 ] 

Eli Collins commented on HADOOP-7111:
-------------------------------------

Strange, bi-secting on my host leads to HADOOP-7206 but that was committed after these tests started failing, must be triggering something.


> Several TFile tests failing when native libraries are present
> -------------------------------------------------------------
>
>                 Key: HADOOP-7111
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7111
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>
> When running tests with native libraries present, TestTFileByteArrays and TestTFileJClassComparatorByteArrays fail on trunk. They don't seem to fail in 0.20 with native libraries.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16793-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 16:20:40 2011
Return-Path: <common-issues-return-16793-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8CBB76233
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 16:20:40 +0000 (UTC)
Received: (qmail 99473 invoked by uid 500); 28 Jun 2011 16:20:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98835 invoked by uid 500); 28 Jun 2011 16:20:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98799 invoked by uid 99); 28 Jun 2011 16:20:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 16:20:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 16:20:37 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5957F4377EE
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 16:20:17 +0000 (UTC)
Date: Tue, 28 Jun 2011 16:20:17 +0000 (UTC)
From: "Ravi Prakash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1225328612.2229.1309278017362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7430) Improve error message when moving to
 trash fails due to quota issue
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Improve error message when moving to trash fails due to quota issue
-------------------------------------------------------------------

                 Key: HADOOP-7430
                 URL: https://issues.apache.org/jira/browse/HADOOP-7430
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs
            Reporter: Ravi Prakash
            Assignee: Ravi Prakash
            Priority: Minor


-rm command doesn't suggest -skipTrash on failure.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16794-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 16:24:40 2011
Return-Path: <common-issues-return-16794-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CEBC263AE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 16:24:40 +0000 (UTC)
Received: (qmail 14271 invoked by uid 500); 28 Jun 2011 16:24:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14214 invoked by uid 500); 28 Jun 2011 16:24:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14201 invoked by uid 99); 28 Jun 2011 16:24:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 16:24:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 16:24:37 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id ED2C4437A8B
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 16:24:16 +0000 (UTC)
Date: Tue, 28 Jun 2011 16:24:16 +0000 (UTC)
From: "Ravi Prakash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <695982236.2244.1309278256968.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1225328612.2229.1309278017362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7430) Improve error message when moving
 to trash fails due to quota issue
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056610#comment-13056610 ] 

Ravi Prakash commented on HADOOP-7430:
--------------------------------------

Courtesy Rajit Saha 
{quote}https://issues.apache.org/jira/browse/HADOOP-6203 surfaces again.

now -rmr is depreciated and using -rm -R and not seeing following messeges in output

Consider using -skipTrash option

and

Problem with Trash.The NameSpace quota (directories and files) of directory /user/someUser is exceeded: quota=5 file
count=6


The only messege coming -
rm: Failed to move to trash: hdfs://localhost:8020/user/someUser/a/b/c
{quote}

Also from Rajit,
{quote}found another scenario when "use -skipTrash" is not promted


$hdfs dfs  -rm -R .
rm: Cannot move "hdfs://<namenode>:8020/user/<user>" to the trash, as it contains the trash

Previously same command use to print the following

Problem with Trash.. Consider using -skipTrash option
rm: Cannot move "hdfs://<namenode>:8020/user/<user>" to the trash, as it contains the trash"
{quote}

> Improve error message when moving to trash fails due to quota issue
> -------------------------------------------------------------------
>
>                 Key: HADOOP-7430
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7430
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Ravi Prakash
>            Assignee: Ravi Prakash
>            Priority: Minor
>
> -rm command doesn't suggest -skipTrash on failure.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16795-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 16:34:38 2011
Return-Path: <common-issues-return-16795-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BDF666C84
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 16:34:38 +0000 (UTC)
Received: (qmail 34750 invoked by uid 500); 28 Jun 2011 16:34:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34686 invoked by uid 500); 28 Jun 2011 16:34:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34675 invoked by uid 99); 28 Jun 2011 16:34:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 16:34:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 16:34:36 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D9068437FB1
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 16:34:16 +0000 (UTC)
Date: Tue, 28 Jun 2011 16:34:16 +0000 (UTC)
From: "Harsh J (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1489441278.2267.1309278856885.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7431) Test DiskChecker's functionality in
 identifying bad directories (Part 2 of testing DiskChecker)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Test DiskChecker's functionality in identifying bad directories (Part 2 of testing DiskChecker)
-----------------------------------------------------------------------------------------------

                 Key: HADOOP-7431
                 URL: https://issues.apache.org/jira/browse/HADOOP-7431
             Project: Hadoop Common
          Issue Type: Test
          Components: test, util
    Affects Versions: 0.23.0
            Reporter: Harsh J
            Assignee: Harsh J
             Fix For: 0.23.0


Add a test for the DiskChecker#checkDir method used in other projects (HDFS).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16796-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 16:59:40 2011
Return-Path: <common-issues-return-16796-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 24537688F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 16:59:40 +0000 (UTC)
Received: (qmail 80942 invoked by uid 500); 28 Jun 2011 16:59:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80885 invoked by uid 500); 28 Jun 2011 16:59:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80873 invoked by uid 99); 28 Jun 2011 16:59:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 16:59:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 16:59:38 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 136ED436ED9
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 16:59:18 +0000 (UTC)
Date: Tue, 28 Jun 2011 16:59:18 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1451299663.2378.1309280358076.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <407429641.30834.1308786767442.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7417) Hadoop Management System
 (Umbrella)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7417?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D130=
56628#comment-13056628 ]=20

Eric Yang commented on HADOOP-7417:
-----------------------------------

Echo on Jagane's comments.  According to Hadoop mission statement:

The Apache=E2=84=A2 Hadoop=E2=84=A2 project develops open-source software f=
or reliable, scalable, distributed computing.

HMS is aligned with Hadoop mission statement.  HMS is a tool to build Infra=
structure as a Service cloud, and Whirr is a tool to build Platform as a Se=
rvice.  There are differences between them.  HMS can provide deep OS integr=
ation for Hadoop, i.e. deploy a Kerbrose enabled Hadoop cluster with LDAP i=
ntegration in cooperate environment.  It will also provide hooks to remove =
defected nodes from operation, and bring up new nodes into the infrastructu=
re.

Whirr provides the other end of spectrum where it provides a platform for d=
evelopers to code, test and experiment new software without the complexity =
of setting up and maintaining test, development and production servers.

Both model should coexist, but HMS will require tighter integration with Ha=
doop to enhance Hadoop's presence in the cooperate environment. Additional =
input from Hadoop PMC can help to decide the right path for HMS.


> Hadoop Management System (Umbrella)
> -----------------------------------
>
>                 Key: HADOOP-7417
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7417
>             Project: Hadoop Common
>          Issue Type: New Feature
>         Environment: Java 6, Linux
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>
> The primary goal of Hadoop Management System is to build a component arou=
nd management and deployment of Hadoop related projects. This includes soft=
ware installation, configuration, application orchestration, deployment aut=
omation and monitoring Hadoop.
> Prototype demo source code can be obtained from:
> http://github.com/macroadster/hms

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

From common-issues-return-16797-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 17:07:39 2011
Return-Path: <common-issues-return-16797-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D6F80611B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 17:07:39 +0000 (UTC)
Received: (qmail 95918 invoked by uid 500); 28 Jun 2011 17:07:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95808 invoked by uid 500); 28 Jun 2011 17:07:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95797 invoked by uid 99); 28 Jun 2011 17:07:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 17:07:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 17:07:38 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E347343735B
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 17:07:17 +0000 (UTC)
Date: Tue, 28 Jun 2011 17:07:17 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <732418322.2423.1309280837927.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <407429641.30834.1308786767442.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7417) Hadoop Management System
 (Umbrella)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056637#comment-13056637 ] 

Eli Collins commented on HADOOP-7417:
-------------------------------------

That's cool, the mailing list is a better forum for discussing HMS and Whirr than a common jira, how about we move this to common-dev? 

> Hadoop Management System (Umbrella)
> -----------------------------------
>
>                 Key: HADOOP-7417
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7417
>             Project: Hadoop Common
>          Issue Type: New Feature
>         Environment: Java 6, Linux
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>
> The primary goal of Hadoop Management System is to build a component around management and deployment of Hadoop related projects. This includes software installation, configuration, application orchestration, deployment automation and monitoring Hadoop.
> Prototype demo source code can be obtained from:
> http://github.com/macroadster/hms

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16798-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 17:07:42 2011
Return-Path: <common-issues-return-16798-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DA557612C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 17:07:41 +0000 (UTC)
Received: (qmail 96144 invoked by uid 500); 28 Jun 2011 17:07:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96100 invoked by uid 500); 28 Jun 2011 17:07:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96092 invoked by uid 99); 28 Jun 2011 17:07:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 17:07:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 17:07:39 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2F099437365
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 17:07:18 +0000 (UTC)
Date: Tue, 28 Jun 2011 17:07:18 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <697240772.2432.1309280838189.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <557825167.38099.1306198007453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7324) Ganglia plugins for metrics v2
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056638#comment-13056638 ] 

Luke Lu commented on HADOOP-7324:
---------------------------------

Thanks Priyo, the patch is coming together nicely.

More issues I missed/noticed:
# Breaking the MetricsCache API requires the approval of release manager for branch-0.20-security. It'd be better if we can keep the existing MetricsCache#getMetric method intact and add a #getMetricInstance for the object.
# You can have the abstract sink implement the visitor interface to avoid creating new instances of the visitor in #putMetrics.
# It'll be nice to have a unit test for these sinks. We can mock the datagramSocket to capture the packet with mockito.

> Ganglia plugins for metrics v2
> ------------------------------
>
>                 Key: HADOOP-7324
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7324
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.203.0, 0.23.0
>            Reporter: Luke Lu
>            Assignee: Priyo Mustafi
>            Priority: Blocker
>              Labels: regression
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7324.patch, HADOOP-7324.patch
>
>
> Although, all metrics in metrics v2 are exposed via the standard JMX mechanisms, most users are using Ganglia to collect metrics.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16799-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 17:28:41 2011
Return-Path: <common-issues-return-16799-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A557D6D39
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 17:28:41 +0000 (UTC)
Received: (qmail 38797 invoked by uid 500); 28 Jun 2011 17:28:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38683 invoked by uid 500); 28 Jun 2011 17:28:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38675 invoked by uid 99); 28 Jun 2011 17:28:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 17:28:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 17:28:38 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 266CF437BEF
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 17:28:17 +0000 (UTC)
Date: Tue, 28 Jun 2011 17:28:17 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1046887389.2495.1309282097153.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <174265054.44671.1309203587452.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7428) IPC connection is orphaned with null
 'out' member
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7428:
--------------------------------

    Attachment: hadoop-7428.txt

Here's a patch which demonstrates the issue. The unit test is a little bit contrived, but does demonstrate the issue and act as a regression test.

> IPC connection is orphaned with null 'out' member
> -------------------------------------------------
>
>                 Key: HADOOP-7428
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7428
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-7428.txt
>
>
> We had a situation a JT ended up in a state where a certain user could not submit a job, due to an NPE on the following line in {{sendParam}}:
> {code}
> synchronized (Connection.this.out) {
> {code}
> Looking at the code, my guess is that an RTE was thrown in setupIOstreams, which only catches IOE. This could leave the connection in a half-setup state which is never cleaned up and also cannot perform IPCs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16800-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 17:30:39 2011
Return-Path: <common-issues-return-16800-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DD0A76152
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 17:30:38 +0000 (UTC)
Received: (qmail 40861 invoked by uid 500); 28 Jun 2011 17:30:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40802 invoked by uid 500); 28 Jun 2011 17:30:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40794 invoked by uid 99); 28 Jun 2011 17:30:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 17:30:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 17:30:37 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 43545437CFF
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 17:30:17 +0000 (UTC)
Date: Tue, 28 Jun 2011 17:30:17 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <951629238.2503.1309282217271.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <174265054.44671.1309203587452.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7428) IPC connection is orphaned with null
 'out' member
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7428:
--------------------------------

    Status: Patch Available  (was: Open)

> IPC connection is orphaned with null 'out' member
> -------------------------------------------------
>
>                 Key: HADOOP-7428
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7428
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-7428.txt
>
>
> We had a situation a JT ended up in a state where a certain user could not submit a job, due to an NPE on the following line in {{sendParam}}:
> {code}
> synchronized (Connection.this.out) {
> {code}
> Looking at the code, my guess is that an RTE was thrown in setupIOstreams, which only catches IOE. This could leave the connection in a half-setup state which is never cleaned up and also cannot perform IPCs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16801-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 17:40:39 2011
Return-Path: <common-issues-return-16801-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9866365F8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 17:40:39 +0000 (UTC)
Received: (qmail 60997 invoked by uid 500); 28 Jun 2011 17:40:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60925 invoked by uid 500); 28 Jun 2011 17:40:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60907 invoked by uid 99); 28 Jun 2011 17:40:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 17:40:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 17:40:37 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 862F14372AA
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 17:40:17 +0000 (UTC)
Date: Tue, 28 Jun 2011 17:40:17 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1574128798.2539.1309282817546.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1008788632.26385.1305807047638.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7305) Eclipse project files are
 incomplete
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056659#comment-13056659 ] 

Daryn Sharp commented on HADOOP-7305:
-------------------------------------

I apologize for the delay -- I was unexpectedly unavailable over a week.  I did test that if JAVA_HOME is set to the JDK dir for Linux that this patch isn't strictly necessary, but it certainly doesn't hurt as a convenience if JAVA_HOME is set to the JRE.

+1 if the OS X section is changed to just the following:

{code}
+    <!-- The tools.jar from the JDK is called classes.jar on OS X. -->
+    <condition property="jdk.tools.jar" value="${java.home}/bundle/Classes/classes.jar">
+      <available file="${java.home}/bundle/Classes/classes.jar"/>
+    </condition>
{code}

Via the web, I found this to be the preferred path and I have confirmed that it works.

> Eclipse project files are incomplete
> ------------------------------------
>
>                 Key: HADOOP-7305
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7305
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Niels Basjes
>            Assignee: Niels Basjes
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7305-2011-05-19.patch, HADOOP-7305-2011-05-30.patch, HADOOP-7305-2011-06-09.patch
>
>
> After a fresh checkout of hadoop-common I do 'ant compile eclipse'.
> I open eclipse, set ANT_HOME and build the project. 
> At that point the following error appears:
> {quote}
> The type com.sun.javadoc.RootDoc cannot be resolved. It is indirectly referenced from required .class files	ExcludePrivateAnnotationsJDiffDoclet.java	/common/src/java/org/apache/hadoop/classification/tools	line 1	Java Problem
> {quote}
> The solution is to add the "tools.jar" from the JDK to the buildpath/classpath.
> This should be fixed in the build.xml.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16802-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 17:44:39 2011
Return-Path: <common-issues-return-16802-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 89F3B66F8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 17:44:39 +0000 (UTC)
Received: (qmail 69933 invoked by uid 500); 28 Jun 2011 17:44:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69748 invoked by uid 500); 28 Jun 2011 17:44:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69718 invoked by uid 99); 28 Jun 2011 17:44:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 17:44:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 17:44:37 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 58BC6437578
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 17:44:17 +0000 (UTC)
Date: Tue, 28 Jun 2011 17:44:17 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1553819637.2587.1309283057360.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <817618820.42234.1309133387608.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7427) syntax error in
 smart-apply-patch.sh
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056669#comment-13056669 ] 

Todd Lipcon commented on HADOOP-7427:
-------------------------------------

wow, that's pretty old. curious, what platform is that? RHEL5 seems to have 3.2.25.

We can probably figure out different syntax to work on your system

> syntax error in smart-apply-patch.sh 
> -------------------------------------
>
>                 Key: HADOOP-7427
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7427
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Tsz Wo (Nicholas), SZE
>
> {noformat}
>      [exec]     Finished build.
>      [exec] hdfs/src/test/bin/smart-apply-patch.sh: line 60: syntax error in conditional expression: unexpected token `('
> BUILD FAILED
> hdfs/build.xml:1595: exec returned: 1
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16803-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 17:44:40 2011
Return-Path: <common-issues-return-16803-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 56541670B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 17:44:40 +0000 (UTC)
Received: (qmail 70261 invoked by uid 500); 28 Jun 2011 17:44:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69878 invoked by uid 500); 28 Jun 2011 17:44:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69743 invoked by uid 99); 28 Jun 2011 17:44:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 17:44:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 17:44:37 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7BC8E43757D
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 17:44:17 +0000 (UTC)
Date: Tue, 28 Jun 2011 17:44:17 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <947619245.2592.1309283057504.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <174265054.44671.1309203587452.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7428) IPC connection is orphaned with
 null 'out' member
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056670#comment-13056670 ] 

Hadoop QA commented on HADOOP-7428:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12484478/hadoop-7428.txt
  against trunk revision 1140442.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/681//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/681//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/681//console

This message is automatically generated.

> IPC connection is orphaned with null 'out' member
> -------------------------------------------------
>
>                 Key: HADOOP-7428
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7428
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-7428.txt
>
>
> We had a situation a JT ended up in a state where a certain user could not submit a job, due to an NPE on the following line in {{sendParam}}:
> {code}
> synchronized (Connection.this.out) {
> {code}
> Looking at the code, my guess is that an RTE was thrown in setupIOstreams, which only catches IOE. This could leave the connection in a half-setup state which is never cleaned up and also cannot perform IPCs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16804-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 17:50:39 2011
Return-Path: <common-issues-return-16804-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9E45E6DFD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 17:50:39 +0000 (UTC)
Received: (qmail 79319 invoked by uid 500); 28 Jun 2011 17:50:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79133 invoked by uid 500); 28 Jun 2011 17:50:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79117 invoked by uid 99); 28 Jun 2011 17:50:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 17:50:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 17:50:37 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D0B03437824
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 17:50:17 +0000 (UTC)
Date: Tue, 28 Jun 2011 17:50:17 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <310854172.2620.1309283417851.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <407429641.30834.1308786767442.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7417) Hadoop Management System
 (Umbrella)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056672#comment-13056672 ] 

Eric Yang commented on HADOOP-7417:
-----------------------------------

bq. That's cool, the mailing list is a better forum for discussing HMS and Whirr than a common jira, how about we move this to common-dev?

Sounds good.

> Hadoop Management System (Umbrella)
> -----------------------------------
>
>                 Key: HADOOP-7417
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7417
>             Project: Hadoop Common
>          Issue Type: New Feature
>         Environment: Java 6, Linux
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>
> The primary goal of Hadoop Management System is to build a component around management and deployment of Hadoop related projects. This includes software installation, configuration, application orchestration, deployment automation and monitoring Hadoop.
> Prototype demo source code can be obtained from:
> http://github.com/macroadster/hms

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16805-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 18:18:42 2011
Return-Path: <common-issues-return-16805-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 351DA6EB1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 18:18:42 +0000 (UTC)
Received: (qmail 24381 invoked by uid 500); 28 Jun 2011 18:18:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23864 invoked by uid 500); 28 Jun 2011 18:18:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23849 invoked by uid 99); 28 Jun 2011 18:18:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 18:18:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 18:18:38 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3694A4375FE
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 18:18:17 +0000 (UTC)
Date: Tue, 28 Jun 2011 18:18:17 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1675021955.2746.1309285097219.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <817618820.42234.1309133387608.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7427) syntax error in
 smart-apply-patch.sh
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056692#comment-13056692 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7427:
------------------------------------------------

Here is the OS information.
{noformat}
Linux 2.6.9-55.ELsmp #1 SMP Fri Apr 20 16:36:54 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
{noformat}

> syntax error in smart-apply-patch.sh 
> -------------------------------------
>
>                 Key: HADOOP-7427
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7427
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Tsz Wo (Nicholas), SZE
>
> {noformat}
>      [exec]     Finished build.
>      [exec] hdfs/src/test/bin/smart-apply-patch.sh: line 60: syntax error in conditional expression: unexpected token `('
> BUILD FAILED
> hdfs/build.xml:1595: exec returned: 1
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16806-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 18:48:42 2011
Return-Path: <common-issues-return-16806-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 623716FA3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 18:48:42 +0000 (UTC)
Received: (qmail 70159 invoked by uid 500); 28 Jun 2011 18:48:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70107 invoked by uid 500); 28 Jun 2011 18:48:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70067 invoked by uid 99); 28 Jun 2011 18:48:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 18:48:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 18:48:38 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7A5EE437276
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 18:48:17 +0000 (UTC)
Date: Tue, 28 Jun 2011 18:48:17 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <993557994.2811.1309286897497.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <817618820.42234.1309133387608.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7427) syntax error in
 smart-apply-patch.sh
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056704#comment-13056704 ] 

Todd Lipcon commented on HADOOP-7427:
-------------------------------------

RHEL4?

> syntax error in smart-apply-patch.sh 
> -------------------------------------
>
>                 Key: HADOOP-7427
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7427
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Tsz Wo (Nicholas), SZE
>
> {noformat}
>      [exec]     Finished build.
>      [exec] hdfs/src/test/bin/smart-apply-patch.sh: line 60: syntax error in conditional expression: unexpected token `('
> BUILD FAILED
> hdfs/build.xml:1595: exec returned: 1
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16807-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 18:58:41 2011
Return-Path: <common-issues-return-16807-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6719F6370
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 18:58:41 +0000 (UTC)
Received: (qmail 96943 invoked by uid 500); 28 Jun 2011 18:58:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96878 invoked by uid 500); 28 Jun 2011 18:58:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96870 invoked by uid 99); 28 Jun 2011 18:58:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 18:58:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 18:58:38 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5CA6243758F
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 18:58:17 +0000 (UTC)
Date: Tue, 28 Jun 2011 18:58:17 +0000 (UTC)
From: "Benjamin Reed (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <594450095.2834.1309287497376.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1443063883.27446.1308722147425.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7414) path to setWar() in HttpServer is
 not setup properly for resources from Jar URLs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Benjamin Reed updated HADOOP-7414:
----------------------------------

    Attachment: HADOOP-7414_2.patch

this patch only adds a / on the end if it is needed.

> path to setWar() in HttpServer is not setup properly for resources from Jar URLs
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-7414
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7414
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Benjamin Reed
>         Attachments: HADOOP-7414.patch, HADOOP-7414_2.patch
>
>
> the path in setWar() needs to end in a '/' for jetty to properly process resources from Jar URLs. by adding a '/' at the end the webapps can run out of a Jar file instead of unzipping into a file system.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16808-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 19:14:39 2011
Return-Path: <common-issues-return-16808-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 29B0A6B9D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 19:14:39 +0000 (UTC)
Received: (qmail 11219 invoked by uid 500); 28 Jun 2011 19:14:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11183 invoked by uid 500); 28 Jun 2011 19:14:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11165 invoked by uid 99); 28 Jun 2011 19:14:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 19:14:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 19:14:37 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 216A2437ABA
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 19:14:17 +0000 (UTC)
Date: Tue, 28 Jun 2011 19:14:17 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <181039336.2871.1309288457133.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1443063883.27446.1308722147425.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7414) path to setWar() in HttpServer is
 not setup properly for resources from Jar URLs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056713#comment-13056713 ] 

Hadoop QA commented on HADOOP-7414:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12484485/HADOOP-7414_2.patch
  against trunk revision 1140442.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/682//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/682//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/682//console

This message is automatically generated.

> path to setWar() in HttpServer is not setup properly for resources from Jar URLs
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-7414
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7414
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Benjamin Reed
>         Attachments: HADOOP-7414.patch, HADOOP-7414_2.patch
>
>
> the path in setWar() needs to end in a '/' for jetty to properly process resources from Jar URLs. by adding a '/' at the end the webapps can run out of a Jar file instead of unzipping into a file system.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16809-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 19:43:39 2011
Return-Path: <common-issues-return-16809-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9647E6E70
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 19:43:39 +0000 (UTC)
Received: (qmail 53319 invoked by uid 500); 28 Jun 2011 19:43:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53069 invoked by uid 500); 28 Jun 2011 19:43:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53058 invoked by uid 99); 28 Jun 2011 19:43:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 19:43:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 19:43:37 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C44D54373FE
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 19:43:17 +0000 (UTC)
Date: Tue, 28 Jun 2011 19:43:17 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1235333878.2911.1309290197800.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1484131492.21509.1305696107453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7298) Add test utility for writing
 multi-threaded tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056720#comment-13056720 ] 

Todd Lipcon commented on HADOOP-7298:
-------------------------------------

Looks good to me. Suresh, mind looking at this latest patch? Harsh's patch for MAPREDUCE-1347 depends on this.

> Add test utility for writing multi-threaded tests
> -------------------------------------------------
>
>                 Key: HADOOP-7298
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7298
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: hadoop-7298.txt, hadoop-7298.txt
>
>
> A lot of our tests spawn off multiple threads in order to check various synchronization issues, etc. It's often tedious to write these kinds of tests because you have to manually propagate exceptions back to the main thread, etc.
> In HBase we have developed a testing utility which makes writing these kinds of tests much easier. I'd like to copy that utility into Hadoop so we can use it here as well.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16810-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 21:43:55 2011
Return-Path: <common-issues-return-16810-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7E593659B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 21:43:55 +0000 (UTC)
Received: (qmail 20657 invoked by uid 500); 28 Jun 2011 21:43:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20197 invoked by uid 500); 28 Jun 2011 21:43:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20188 invoked by uid 99); 28 Jun 2011 21:43:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 21:43:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 21:43:49 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0F9B1413969
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 21:43:29 +0000 (UTC)
Date: Tue, 28 Jun 2011 21:43:29 +0000 (UTC)
From: "Andrew Look (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <492511776.199.1309297409060.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7418) support for multiple slashes in the
 path separator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Look updated HADOOP-7418:
--------------------------------

    Attachment: HADOOP-7418.txt

I redid my patch to handle the case of >2 slashes, and added a couple of unit tests as you suggested.

When running the test suite, I had some tests hang due to timeouts, but no failures. I tried running the pre-patch versions of HDFS's test suite, and for instance, TestHDFSCLI times out because it can't connect to a namenode. Similar outcomes for a number of other tests: TestLargeBlock, TestReplaceDataNodeOnFailure...

Will the Jenkins instance run these for me?

> support for multiple slashes in the path separator
> --------------------------------------------------
>
>                 Key: HADOOP-7418
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7418
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>         Environment: Linux running JDK 1.6
>            Reporter: Sudharsan Sampath
>            Assignee: Andrew Look
>            Priority: Minor
>              Labels: newbie
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7418.txt, HDFS-1460.txt, HDFS-1460.txt
>
>
> the parsing of the input path string to identify the uri authority conflicts with the file system paths. For instance the following is a valid path in both the linux file system and the hdfs.
> //user/directory1//directory2.
> While this works perfectly fine in the command line for manipulating hdfs, the same fails when specified as the input path for a mapper class with the following expcetion.
> Exception in thread "main" java.net.UnknownHostException: unknown host: user
>         at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:195)
> as the org.apache.hadoop.fs.Path class assumes the string that follows the '//' to be an uri authority

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16811-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 21:49:50 2011
Return-Path: <common-issues-return-16811-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9FCC46C2F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 21:49:50 +0000 (UTC)
Received: (qmail 28586 invoked by uid 500); 28 Jun 2011 21:49:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28535 invoked by uid 500); 28 Jun 2011 21:49:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28527 invoked by uid 99); 28 Jun 2011 21:49:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 21:49:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 21:49:48 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E0EE4413C75
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 21:49:28 +0000 (UTC)
Date: Tue, 28 Jun 2011 21:49:28 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <128463663.214.1309297768918.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7418) support for multiple slashes in
 the path separator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056809#comment-13056809 ] 

Aaron T. Myers commented on HADOOP-7418:
----------------------------------------

bq. Will the Jenkins instance run these for me?

Jenkins will only run the tests for the Common sub-project, since this is a JIRA for Common.

I'll take a look at this patch and try to run some of tests you mention that timed out. We see a few notoriously flaky tests fail spuriously pretty often, but not usually these, and not usually with timeouts.

> support for multiple slashes in the path separator
> --------------------------------------------------
>
>                 Key: HADOOP-7418
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7418
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>         Environment: Linux running JDK 1.6
>            Reporter: Sudharsan Sampath
>            Assignee: Andrew Look
>            Priority: Minor
>              Labels: newbie
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7418.txt, HDFS-1460.txt, HDFS-1460.txt
>
>
> the parsing of the input path string to identify the uri authority conflicts with the file system paths. For instance the following is a valid path in both the linux file system and the hdfs.
> //user/directory1//directory2.
> While this works perfectly fine in the command line for manipulating hdfs, the same fails when specified as the input path for a mapper class with the following expcetion.
> Exception in thread "main" java.net.UnknownHostException: unknown host: user
>         at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:195)
> as the org.apache.hadoop.fs.Path class assumes the string that follows the '//' to be an uri authority

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16812-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 22:03:54 2011
Return-Path: <common-issues-return-16812-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 943CC6332
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 22:03:54 +0000 (UTC)
Received: (qmail 52068 invoked by uid 500); 28 Jun 2011 22:03:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51717 invoked by uid 500); 28 Jun 2011 22:03:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51708 invoked by uid 99); 28 Jun 2011 22:03:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 22:03:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 22:03:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4B48841322A
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 22:03:29 +0000 (UTC)
Date: Tue, 28 Jun 2011 22:03:29 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1344860584.251.1309298609305.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7418) support for multiple slashes in
 the path separator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056816#comment-13056816 ] 

Hadoop QA commented on HADOOP-7418:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12484565/HADOOP-7418.txt
  against trunk revision 1140442.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/683//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/683//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/683//console

This message is automatically generated.

> support for multiple slashes in the path separator
> --------------------------------------------------
>
>                 Key: HADOOP-7418
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7418
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>         Environment: Linux running JDK 1.6
>            Reporter: Sudharsan Sampath
>            Assignee: Andrew Look
>            Priority: Minor
>              Labels: newbie
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7418.txt, HDFS-1460.txt, HDFS-1460.txt
>
>
> the parsing of the input path string to identify the uri authority conflicts with the file system paths. For instance the following is a valid path in both the linux file system and the hdfs.
> //user/directory1//directory2.
> While this works perfectly fine in the command line for manipulating hdfs, the same fails when specified as the input path for a mapper class with the following expcetion.
> Exception in thread "main" java.net.UnknownHostException: unknown host: user
>         at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:195)
> as the org.apache.hadoop.fs.Path class assumes the string that follows the '//' to be an uri authority

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16813-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 22:51:53 2011
Return-Path: <common-issues-return-16813-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9E58561B6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 22:51:53 +0000 (UTC)
Received: (qmail 22276 invoked by uid 500); 28 Jun 2011 22:51:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22223 invoked by uid 500); 28 Jun 2011 22:51:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22215 invoked by uid 99); 28 Jun 2011 22:51:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 22:51:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 22:51:49 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8F02B4130FC
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 22:51:28 +0000 (UTC)
Date: Tue, 28 Jun 2011 22:51:28 +0000 (UTC)
From: "Andrew Look (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1399346697.466.1309301488582.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7418) support for multiple slashes in
 the path separator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056865#comment-13056865 ] 

Andrew Look commented on HADOOP-7418:
-------------------------------------

Great, thanks Aaron! 

I'm running OS X Snow Leopard so that may be causing problems on my end... I've heard of others having trouble building the C++ libraries in HDFS (see http://cxwangyi.blogspot.com/2010_04_01_archive.html ) and apparently some mods need to be made to get things to run correctly.

Have you heard of any such issues on OS X?

> support for multiple slashes in the path separator
> --------------------------------------------------
>
>                 Key: HADOOP-7418
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7418
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>         Environment: Linux running JDK 1.6
>            Reporter: Sudharsan Sampath
>            Assignee: Andrew Look
>            Priority: Minor
>              Labels: newbie
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7418.txt, HDFS-1460.txt, HDFS-1460.txt
>
>
> the parsing of the input path string to identify the uri authority conflicts with the file system paths. For instance the following is a valid path in both the linux file system and the hdfs.
> //user/directory1//directory2.
> While this works perfectly fine in the command line for manipulating hdfs, the same fails when specified as the input path for a mapper class with the following expcetion.
> Exception in thread "main" java.net.UnknownHostException: unknown host: user
>         at org.apache.hadoop.ipc.Client$Connection.<init>(Client.java:195)
> as the org.apache.hadoop.fs.Path class assumes the string that follows the '//' to be an uri authority

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16814-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue Jun 28 23:11:53 2011
Return-Path: <common-issues-return-16814-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1C4116CD2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 28 Jun 2011 23:11:53 +0000 (UTC)
Received: (qmail 54975 invoked by uid 500); 28 Jun 2011 23:11:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54905 invoked by uid 500); 28 Jun 2011 23:11:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54894 invoked by uid 99); 28 Jun 2011 23:11:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 23:11:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Jun 2011 23:11:49 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8319841385C
	for <common-issues@hadoop.apache.org>; Tue, 28 Jun 2011 23:11:28 +0000 (UTC)
Date: Tue, 28 Jun 2011 23:11:28 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <347121513.495.1309302688533.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <403145852.745.1309238057130.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7429) Add another IOUtils#copyBytes method
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7429:
--------------------------------

    Attachment: hadoop-7429-2.patch

Minor update, add a close parameter.

> Add another IOUtils#copyBytes method
> ------------------------------------
>
>                 Key: HADOOP-7429
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7429
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>         Attachments: hadoop-7429-1.patch, hadoop-7429-2.patch
>
>
> Common side of HDFS-2110, adds a new IOUtils copy bytes method and cleanup.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16815-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 01:40:51 2011
Return-Path: <common-issues-return-16815-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 61E506F77
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 01:40:51 +0000 (UTC)
Received: (qmail 97074 invoked by uid 500); 29 Jun 2011 01:40:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96850 invoked by uid 500); 29 Jun 2011 01:40:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96840 invoked by uid 99); 29 Jun 2011 01:40:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 01:40:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 01:40:49 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 36AC7438BD1
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 01:40:29 +0000 (UTC)
Date: Wed, 29 Jun 2011 01:40:29 +0000 (UTC)
From: "Min Zhou (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1029638806.870.1309311629220.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1992675736.52742.1306733207390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7339) Introduce a buffered checksum for
 avoiding frequently calls on Checksum.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Min Zhou resolved HADOOP-7339.
------------------------------

    Resolution: Won't Fix

> Introduce a buffered checksum for avoiding frequently calls on Checksum.update()
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-7339
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7339
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Min Zhou
>            Assignee: Min Zhou
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7339-v1.diff, HADOOP-7339-v2.diff
>
>
> We found that PureJavaCRC32/CRC32.update() is the TOP 1 of the methods consuming CPU in a map side, and in reduce side, it cost a lots of CPU too.
> IFileOutputStream would frequently call Checksum.update() during writing a record. It's very common a MR key/value less than 512 bytes. Checksum.update() would be called every time writing a key/value. 
> Test case: terasort 100MB. 
> Checksum.update() calls has be reduced from 4030348 to 28069.  This method is not a hotspot anymore.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16816-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 05:42:14 2011
Return-Path: <common-issues-return-16816-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2EC784759
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 05:42:14 +0000 (UTC)
Received: (qmail 54915 invoked by uid 500); 29 Jun 2011 05:42:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54445 invoked by uid 500); 29 Jun 2011 05:41:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54396 invoked by uid 99); 29 Jun 2011 05:41:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 05:41:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 05:41:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AA87D4380F1
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 05:41:29 +0000 (UTC)
Date: Wed, 29 Jun 2011 05:41:29 +0000 (UTC)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1981182038.1204.1309326089695.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057011#comment-13057011 ] 

Konstantin Boudnik commented on HADOOP-6671:
--------------------------------------------

Can we include Sonar stuff here as well? Here INFRA-3645 is the almost working patch for HDFS which I am sure can be scaled to the Common with ease. If there's no objection I can update the patch.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-6671-cross-project-HDFS.patch, HADOOP-6671-e.patch, HADOOP-6671-f.patch, HADOOP-6671-g.patch, HADOOP-6671-h.patch, HADOOP-6671-i.patch, HADOOP-6671-j.patch, HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, common-mvn-layout-i.sh, hadoop-commons-maven.patch, mvn-layout-e.sh, mvn-layout-f.sh, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16817-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 05:44:09 2011
Return-Path: <common-issues-return-16817-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 844504778
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 05:44:09 +0000 (UTC)
Received: (qmail 55327 invoked by uid 500); 29 Jun 2011 05:44:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55060 invoked by uid 500); 29 Jun 2011 05:43:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55040 invoked by uid 99); 29 Jun 2011 05:43:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 05:43:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 05:43:49 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 72874438199
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 05:43:28 +0000 (UTC)
Date: Wed, 29 Jun 2011 05:43:28 +0000 (UTC)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1308675070.1213.1309326208465.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1221804712.25214.1308677867694.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7412) Mavenization Umbrella
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057012#comment-13057012 ] 

Konstantin Boudnik commented on HADOOP-7412:
--------------------------------------------

I am suggesting to considering the inclusion of INFRA-3645's patch.

> Mavenization Umbrella
> ---------------------
>
>                 Key: HADOOP-7412
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7412
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>
> Umbrella JIRA for all Mavenization JIRAs

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16818-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 09:12:26 2011
Return-Path: <common-issues-return-16818-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1E76C4AC1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 09:12:26 +0000 (UTC)
Received: (qmail 73705 invoked by uid 500); 29 Jun 2011 09:12:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72785 invoked by uid 500); 29 Jun 2011 09:12:04 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72683 invoked by uid 99); 29 Jun 2011 09:11:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 09:11:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 09:11:52 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C4CF14393A3
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 09:11:31 +0000 (UTC)
Date: Wed, 29 Jun 2011 09:11:31 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <955644710.1745.1309338691802.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6929) RPC should have a way to pass
 Security information other than protocol annotations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057095#comment-13057095 ] 

Hudson commented on HADOOP-6929:
--------------------------------

Integrated in Hadoop-Common-trunk-maven #43 (See [https://builds.apache.org/job/Hadoop-Common-trunk-maven/43/])
    

> RPC should have a way to pass Security information other than protocol annotations
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6929
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6929
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>             Fix For: 0.23.0
>
>         Attachments: Hadoop-6929_v1.patch, h-6929.patch, h-6929.patch, h-6929.patch
>
>
> Currently Hadoop RPC allows protocol annotations as the only way to pass security information. This becomes a problem if protocols are generated and not hand written. For example protocols generated via Avro and passed over Avro tunnel (AvroRpcEngine.java) can't pass the security information.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16819-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 09:12:28 2011
Return-Path: <common-issues-return-16819-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DFE224ACB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 09:12:27 +0000 (UTC)
Received: (qmail 74447 invoked by uid 500); 29 Jun 2011 09:12:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73656 invoked by uid 500); 29 Jun 2011 09:12:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72681 invoked by uid 99); 29 Jun 2011 09:11:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 09:11:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 09:11:52 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 44630439391
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 09:11:31 +0000 (UTC)
Date: Wed, 29 Jun 2011 09:11:31 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2146721356.1729.1309338691276.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057094#comment-13057094 ] 

Hudson commented on HADOOP-7206:
--------------------------------

Integrated in Hadoop-Common-trunk-maven #43 (See [https://builds.apache.org/job/Hadoop-Common-trunk-maven/43/])
    

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206new-b.patch, HADOOP-7206new-c.patch, HADOOP-7206new.patch, HADOOP-7206revertplusnew-b.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16820-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 09:12:32 2011
Return-Path: <common-issues-return-16820-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 803314AD3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 09:12:32 +0000 (UTC)
Received: (qmail 74790 invoked by uid 500); 29 Jun 2011 09:12:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74383 invoked by uid 500); 29 Jun 2011 09:12:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72685 invoked by uid 99); 29 Jun 2011 09:11:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 09:11:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 09:11:52 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D782A4393A5
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 09:11:31 +0000 (UTC)
Date: Wed, 29 Jun 2011 09:11:31 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <867835813.1747.1309338691871.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <403145852.745.1309238057130.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7429) Add another IOUtils#copyBytes
 method
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057096#comment-13057096 ] 

Hudson commented on HADOOP-7429:
--------------------------------

Integrated in Hadoop-Common-trunk-maven #43 (See [https://builds.apache.org/job/Hadoop-Common-trunk-maven/43/])
    

> Add another IOUtils#copyBytes method
> ------------------------------------
>
>                 Key: HADOOP-7429
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7429
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>         Attachments: hadoop-7429-1.patch, hadoop-7429-2.patch
>
>
> Common side of HDFS-2110, adds a new IOUtils copy bytes method and cleanup.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16821-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 09:12:39 2011
Return-Path: <common-issues-return-16821-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 26F074AE5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 09:12:39 +0000 (UTC)
Received: (qmail 75089 invoked by uid 500); 29 Jun 2011 09:12:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74741 invoked by uid 500); 29 Jun 2011 09:12:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72655 invoked by uid 99); 29 Jun 2011 09:11:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 09:11:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 09:11:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 062C1439358
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 09:11:29 +0000 (UTC)
Date: Wed, 29 Jun 2011 09:11:29 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1068486762.1677.1309338689021.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-310) Additional constructor requested in
 BytesWritable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057093#comment-13057093 ] 

Hudson commented on HADOOP-310:
-------------------------------

Integrated in Hadoop-Common-trunk-maven #43 (See [https://builds.apache.org/job/Hadoop-Common-trunk-maven/43/])
    

> Additional constructor requested in BytesWritable
> -------------------------------------------------
>
>                 Key: HADOOP-310
>                 URL: https://issues.apache.org/jira/browse/HADOOP-310
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: p sutter
>            Assignee: Brock Noland
>            Priority: Minor
>         Attachments: bytes-writable-zero-copy-interface-0.patch, bytes-writable-zero-copy-interface-1.patch, bytes-writable-zero-copy-interface-2.patch
>
>
> It would be grand if BytesWritable.java had an additional constructor as below. This allows me to use the BytesWritable class without doing a buffer copy, since we have a less-than-fully-utilized byte array holding our key.
> Thanks!
>  
>  /**
>    * Create a BytesWritable using the byte array as the initial value.
>    * @param bytes This array becomes the backing storage for the object.
>    */
>   public BytesWritable(byte[] bytes, int size) {
>     this.bytes = bytes;
>     this.size = size;
>   }
>   

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16822-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 09:12:51 2011
Return-Path: <common-issues-return-16822-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 689EC4AFD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 09:12:51 +0000 (UTC)
Received: (qmail 75404 invoked by uid 500); 29 Jun 2011 09:12:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75088 invoked by uid 500); 29 Jun 2011 09:12:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72659 invoked by uid 99); 29 Jun 2011 09:11:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 09:11:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 09:11:52 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F191C4393A8
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 09:11:31 +0000 (UTC)
Date: Wed, 29 Jun 2011 09:11:31 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <202408076.1750.1309338691986.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <113408089.3186.1307985891583.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7385) Remove
 StringUtils.stringifyException(ie) in logger functions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057097#comment-13057097 ] 

Hudson commented on HADOOP-7385:
--------------------------------

Integrated in Hadoop-Common-trunk-maven #43 (See [https://builds.apache.org/job/Hadoop-Common-trunk-maven/43/])
    

> Remove StringUtils.stringifyException(ie) in logger functions
> -------------------------------------------------------------
>
>                 Key: HADOOP-7385
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7385
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7385-1.patch
>
>
> Apache logger api has an overloaded function which can take the message and exception. I am proposing to clean the logging code with this api.
> ie.:
> Change the code from LOG.warn(msg, StringUtils.stringifyException(exception)); to LOG.warn(msg, exception);

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16823-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 09:12:57 2011
Return-Path: <common-issues-return-16823-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 477684B0C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 09:12:57 +0000 (UTC)
Received: (qmail 75706 invoked by uid 500); 29 Jun 2011 09:12:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75410 invoked by uid 500); 29 Jun 2011 09:12:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72870 invoked by uid 99); 29 Jun 2011 09:12:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 09:12:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 09:12:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 53ED44393B0
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 09:11:32 +0000 (UTC)
Date: Wed, 29 Jun 2011 09:11:32 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <441525181.1758.1309338692340.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1164170325.41524.1306304627421.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7329) incomplete help message  is
 displayed for df -h option
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057099#comment-13057099 ] 

Hudson commented on HADOOP-7329:
--------------------------------

Integrated in Hadoop-Common-trunk-maven #43 (See [https://builds.apache.org/job/Hadoop-Common-trunk-maven/43/])
    

> incomplete help message  is displayed for df -h option
> ------------------------------------------------------
>
>                 Key: HADOOP-7329
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7329
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Assignee: XieXianshan
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7329
>
>
> The help message for the command "hdfs dfs -help df" is displayed like this:
> "-df [<path> ...]:    Shows the capacity, free and used space of the filesystem.
>         If the filesystem has multiple partitions, and no path to a
>         particular partition is specified, then the status of the root
>         partitions will be shown."
> and the information about df -h option is missed,despite the fact that df -h option is implemented.
> Therefore,the expected message should be displayed like this:
> "-df [-h] [<path> ...]:    Shows the capacity, free and used space of the filesystem.
>         If the filesystem has multiple partitions, and no path to a
>         particular partition is specified, then the status of the root
>         partitions will be shown.
>           -h   Formats the sizes of files in a human-readable fashion
>                rather than a number of bytes."

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16824-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 09:13:06 2011
Return-Path: <common-issues-return-16824-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 400F44B1F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 09:13:06 +0000 (UTC)
Received: (qmail 75963 invoked by uid 500); 29 Jun 2011 09:13:05 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75741 invoked by uid 500); 29 Jun 2011 09:12:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72851 invoked by uid 99); 29 Jun 2011 09:12:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 09:12:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 09:12:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B50DA4393BD
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 09:11:32 +0000 (UTC)
Date: Wed, 29 Jun 2011 09:11:32 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <40432215.1768.1309338692738.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <55510245.12688.1307747698852.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7379) Add ability to include Protobufs
 in ObjectWritable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057100#comment-13057100 ] 

Hudson commented on HADOOP-7379:
--------------------------------

Integrated in Hadoop-Common-trunk-maven #43 (See [https://builds.apache.org/job/Hadoop-Common-trunk-maven/43/])
    

> Add ability to include Protobufs in ObjectWritable
> --------------------------------------------------
>
>                 Key: HADOOP-7379
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7379
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, ipc
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7379.txt, hadoop-7379.txt
>
>
> Per HDFS-2060, it would make it easier to piecemeal switch to protocol buffer based data structures in the wire protocol if we could intermix the two. The IPC framework currently provides the concept of "engines" for RPC, but that doesn't easily allow mixed types within the same framework for ease of transition.
> I'd like to add the cases to ObjectWritable to be handle subclasses of {{Message}}, the superclass of codegenned protobufs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16825-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 09:13:09 2011
Return-Path: <common-issues-return-16825-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D15A64B43
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 09:13:09 +0000 (UTC)
Received: (qmail 76639 invoked by uid 500); 29 Jun 2011 09:13:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75988 invoked by uid 500); 29 Jun 2011 09:13:05 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72826 invoked by uid 99); 29 Jun 2011 09:12:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 09:12:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 09:12:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1EFE84393AB
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 09:11:32 +0000 (UTC)
Date: Wed, 29 Jun 2011 09:11:32 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1032632055.1753.1309338692123.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1758842790.15942.1308336467619.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7402) TestConfiguration doesn't clean up
 after itself
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057098#comment-13057098 ] 

Hudson commented on HADOOP-7402:
--------------------------------

Integrated in Hadoop-Common-trunk-maven #43 (See [https://builds.apache.org/job/Hadoop-Common-trunk-maven/43/])
    

> TestConfiguration doesn't clean up after itself
> -----------------------------------------------
>
>                 Key: HADOOP-7402
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7402
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7402.0.patch, hadoop-7402.1.patch
>
>
> {{testGetFile}} and {{testGetLocalPath}} both create directories a, b, and c in the working directory from where the tests are run. They should clean up after themselves.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16826-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 09:13:15 2011
Return-Path: <common-issues-return-16826-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 317034B56
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 09:13:15 +0000 (UTC)
Received: (qmail 76914 invoked by uid 500); 29 Jun 2011 09:13:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76683 invoked by uid 500); 29 Jun 2011 09:13:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72849 invoked by uid 99); 29 Jun 2011 09:12:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 09:12:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 09:12:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D17624393C2
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 09:11:32 +0000 (UTC)
Date: Wed, 29 Jun 2011 09:11:32 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <80265614.1771.1309338692854.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <151121036.11705.1307734619237.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7377) Fix command name handling
 affecting DFSAdmin
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057101#comment-13057101 ] 

Hudson commented on HADOOP-7377:
--------------------------------

Integrated in Hadoop-Common-trunk-maven #43 (See [https://builds.apache.org/job/Hadoop-Common-trunk-maven/43/])
    

> Fix command name handling affecting DFSAdmin
> --------------------------------------------
>
>                 Key: HADOOP-7377
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7377
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7377.patch
>
>
> When an error occurs in the get/set quota commands in DFSAdmin, they are displaying the following:
> setQuota: failed to get SetQuotaCommand.NAME
> The {{Command}} class expects the {{NAME}} field to be accessible, but for DFSAdmin, it's not.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16827-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 09:13:20 2011
Return-Path: <common-issues-return-16827-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2782C4B61
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 09:13:20 +0000 (UTC)
Received: (qmail 77198 invoked by uid 500); 29 Jun 2011 09:13:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76915 invoked by uid 500); 29 Jun 2011 09:13:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72854 invoked by uid 99); 29 Jun 2011 09:12:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 09:12:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 09:12:13 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1A7564393C9
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 09:11:33 +0000 (UTC)
Date: Wed, 29 Jun 2011 09:11:33 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1542819967.1778.1309338693105.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1481744229.2736.1308062747508.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7390) VersionInfo not generated properly
 in git after unsplit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057103#comment-13057103 ] 

Hudson commented on HADOOP-7390:
--------------------------------

Integrated in Hadoop-Common-trunk-maven #43 (See [https://builds.apache.org/job/Hadoop-Common-trunk-maven/43/])
    

> VersionInfo not generated properly in git after unsplit
> -------------------------------------------------------
>
>                 Key: HADOOP-7390
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7390
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Thomas Graves
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7390.txt
>
>
> The version information generated during the build of common when running from git has revision and branch Unknown. I believe this started after the unsplit:
> @HadoopVersionAnnotation(version="0.22.0-SNAPSHOT", revision="Unknown", branch="Unknown",
>                          user="tgraves", date="Tue Jun 14 13:39:10 UTC 2011", url="file:///home/tgraves/git/hadoop-common/common",
>                          srcChecksum="0f78ea668971fe51e7ebf4f97f84eed2")
> The ./src/saveVersion.sh script which generates the package-info.java file with the version info looks for the presence of .git directory and that is now a level up instead of in the common directory.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16828-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 09:13:24 2011
Return-Path: <common-issues-return-16828-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D5B484B65
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 09:13:24 +0000 (UTC)
Received: (qmail 77424 invoked by uid 500); 29 Jun 2011 09:13:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77362 invoked by uid 500); 29 Jun 2011 09:13:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72862 invoked by uid 99); 29 Jun 2011 09:12:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 09:12:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 09:12:13 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F14E24393C6
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 09:11:32 +0000 (UTC)
Date: Wed, 29 Jun 2011 09:11:32 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <966403414.1775.1309338692985.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <680679068.21358.1308591287861.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7407) Snappy integration breaks HDFS
 build.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057102#comment-13057102 ] 

Hudson commented on HADOOP-7407:
--------------------------------

Integrated in Hadoop-Common-trunk-maven #43 (See [https://builds.apache.org/job/Hadoop-Common-trunk-maven/43/])
    

> Snappy integration breaks HDFS build.
> -------------------------------------
>
>                 Key: HADOOP-7407
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7407
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Ivan Kelly
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7407.patch
>
>
> The common/ivy/hadoop-common-template.xml submitted with 7206 has a typo which breaks anything that depends on the hadoop-common maven package.
> Instead of <artifactId>java-snappy</artifactId>, you should have <artifactId>snappy-java</artifactId>
> [ivy:resolve] downloading https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.23.0-SNAPSHOT/hadoop-common-0.23.0-20110620.163810-177.jar ...
> [ivy:resolve] .......................
> [ivy:resolve] ..................................
> [ivy:resolve] ...........................................
> [ivy:resolve] ...............................................................
> [ivy:resolve] ................................................ (1631kB)
> [ivy:resolve] .. (0kB)
> [ivy:resolve] 	[SUCCESSFUL ] org.apache.hadoop#hadoop-common;0.23.0-SNAPSHOT!hadoop-common.jar (8441ms)
> [ivy:resolve] 
> [ivy:resolve] :: problems summary ::
> [ivy:resolve] :::: WARNINGS
> [ivy:resolve] 		module not found: org.xerial.snappy#java-snappy;1.0.3-rc2
> [ivy:resolve] 	==== apache-snapshot: tried
> [ivy:resolve] 	  https://repository.apache.org/content/repositories/snapshots/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.pom
> [ivy:resolve] 	  -- artifact org.xerial.snappy#java-snappy;1.0.3-rc2!java-snappy.jar:
> [ivy:resolve] 	  https://repository.apache.org/content/repositories/snapshots/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.jar
> [ivy:resolve] 	==== maven2: tried
> [ivy:resolve] 	  http://repo1.maven.org/maven2/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.pom
> [ivy:resolve] 	  -- artifact org.xerial.snappy#java-snappy;1.0.3-rc2!java-snappy.jar:
> [ivy:resolve] 	  http://repo1.maven.org/maven2/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.jar
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 		::          UNRESOLVED DEPENDENCIES         ::
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 		:: org.xerial.snappy#java-snappy;1.0.3-rc2: not found
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16829-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 11:17:59 2011
Return-Path: <common-issues-return-16829-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 18B0C4D3F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 11:17:59 +0000 (UTC)
Received: (qmail 71815 invoked by uid 500); 29 Jun 2011 11:17:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71159 invoked by uid 500); 29 Jun 2011 11:17:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70764 invoked by uid 99); 29 Jun 2011 11:17:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 11:17:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 11:17:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0EEFF439EC2
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 11:17:29 +0000 (UTC)
Date: Wed, 29 Jun 2011 11:17:29 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <310120105.2036.1309346249058.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-310) Additional constructor requested in
 BytesWritable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057155#comment-13057155 ] 

Hudson commented on HADOOP-310:
-------------------------------

Integrated in Hadoop-Common-trunk #732 (See [https://builds.apache.org/job/Hadoop-Common-trunk/732/])
    

> Additional constructor requested in BytesWritable
> -------------------------------------------------
>
>                 Key: HADOOP-310
>                 URL: https://issues.apache.org/jira/browse/HADOOP-310
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: p sutter
>            Assignee: Brock Noland
>            Priority: Minor
>         Attachments: bytes-writable-zero-copy-interface-0.patch, bytes-writable-zero-copy-interface-1.patch, bytes-writable-zero-copy-interface-2.patch
>
>
> It would be grand if BytesWritable.java had an additional constructor as below. This allows me to use the BytesWritable class without doing a buffer copy, since we have a less-than-fully-utilized byte array holding our key.
> Thanks!
>  
>  /**
>    * Create a BytesWritable using the byte array as the initial value.
>    * @param bytes This array becomes the backing storage for the object.
>    */
>   public BytesWritable(byte[] bytes, int size) {
>     this.bytes = bytes;
>     this.size = size;
>   }
>   

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16830-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 11:18:00 2011
Return-Path: <common-issues-return-16830-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 502694D73
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 11:18:00 +0000 (UTC)
Received: (qmail 72127 invoked by uid 500); 29 Jun 2011 11:18:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71773 invoked by uid 500); 29 Jun 2011 11:17:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70767 invoked by uid 99); 29 Jun 2011 11:17:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 11:17:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 11:17:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A518E439EFB
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 11:17:31 +0000 (UTC)
Date: Wed, 29 Jun 2011 11:17:31 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <366921909.2092.1309346251673.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <113408089.3186.1307985891583.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7385) Remove
 StringUtils.stringifyException(ie) in logger functions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057158#comment-13057158 ] 

Hudson commented on HADOOP-7385:
--------------------------------

Integrated in Hadoop-Common-trunk #732 (See [https://builds.apache.org/job/Hadoop-Common-trunk/732/])
    

> Remove StringUtils.stringifyException(ie) in logger functions
> -------------------------------------------------------------
>
>                 Key: HADOOP-7385
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7385
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7385-1.patch
>
>
> Apache logger api has an overloaded function which can take the message and exception. I am proposing to clean the logging code with this api.
> ie.:
> Change the code from LOG.warn(msg, StringUtils.stringifyException(exception)); to LOG.warn(msg, exception);

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16831-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 11:18:02 2011
Return-Path: <common-issues-return-16831-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8FD584DBD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 11:18:02 +0000 (UTC)
Received: (qmail 72428 invoked by uid 500); 29 Jun 2011 11:18:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72219 invoked by uid 500); 29 Jun 2011 11:18:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71120 invoked by uid 99); 29 Jun 2011 11:17:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 11:17:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 11:17:52 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7B7AE439EFF
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 11:17:32 +0000 (UTC)
Date: Wed, 29 Jun 2011 11:17:32 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1859867063.2096.1309346252502.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1758842790.15942.1308336467619.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7402) TestConfiguration doesn't clean up
 after itself
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057159#comment-13057159 ] 

Hudson commented on HADOOP-7402:
--------------------------------

Integrated in Hadoop-Common-trunk #732 (See [https://builds.apache.org/job/Hadoop-Common-trunk/732/])
    

> TestConfiguration doesn't clean up after itself
> -----------------------------------------------
>
>                 Key: HADOOP-7402
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7402
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7402.0.patch, hadoop-7402.1.patch
>
>
> {{testGetFile}} and {{testGetLocalPath}} both create directories a, b, and c in the working directory from where the tests are run. They should clean up after themselves.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16832-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 11:18:04 2011
Return-Path: <common-issues-return-16832-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 261094DF1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 11:18:04 +0000 (UTC)
Received: (qmail 72710 invoked by uid 500); 29 Jun 2011 11:18:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72409 invoked by uid 500); 29 Jun 2011 11:18:02 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71160 invoked by uid 99); 29 Jun 2011 11:17:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 11:17:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 11:17:52 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 800C5439EF7
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 11:17:31 +0000 (UTC)
Date: Wed, 29 Jun 2011 11:17:31 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <22865619.2088.1309346251521.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057156#comment-13057156 ] 

Hudson commented on HADOOP-7206:
--------------------------------

Integrated in Hadoop-Common-trunk #732 (See [https://builds.apache.org/job/Hadoop-Common-trunk/732/])
    

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206new-b.patch, HADOOP-7206new-c.patch, HADOOP-7206new.patch, HADOOP-7206revertplusnew-b.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16833-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 11:18:04 2011
Return-Path: <common-issues-return-16833-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CDB664E0E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 11:18:04 +0000 (UTC)
Received: (qmail 72944 invoked by uid 500); 29 Jun 2011 11:18:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72695 invoked by uid 500); 29 Jun 2011 11:18:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71161 invoked by uid 99); 29 Jun 2011 11:17:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 11:17:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 11:17:52 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 93405439EF9
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 11:17:31 +0000 (UTC)
Date: Wed, 29 Jun 2011 11:17:31 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2035351407.2090.1309346251600.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <403145852.745.1309238057130.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7429) Add another IOUtils#copyBytes
 method
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057157#comment-13057157 ] 

Hudson commented on HADOOP-7429:
--------------------------------

Integrated in Hadoop-Common-trunk #732 (See [https://builds.apache.org/job/Hadoop-Common-trunk/732/])
    

> Add another IOUtils#copyBytes method
> ------------------------------------
>
>                 Key: HADOOP-7429
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7429
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>         Attachments: hadoop-7429-1.patch, hadoop-7429-2.patch
>
>
> Common side of HDFS-2110, adds a new IOUtils copy bytes method and cleanup.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16834-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 11:18:15 2011
Return-Path: <common-issues-return-16834-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A60924F49
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 11:18:15 +0000 (UTC)
Received: (qmail 74287 invoked by uid 500); 29 Jun 2011 11:18:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74207 invoked by uid 500); 29 Jun 2011 11:18:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74186 invoked by uid 99); 29 Jun 2011 11:18:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 11:18:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 11:18:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B238D439F05
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 11:17:32 +0000 (UTC)
Date: Wed, 29 Jun 2011 11:17:32 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2100227831.2101.1309346252726.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1164170325.41524.1306304627421.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7329) incomplete help message  is
 displayed for df -h option
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057160#comment-13057160 ] 

Hudson commented on HADOOP-7329:
--------------------------------

Integrated in Hadoop-Common-trunk #732 (See [https://builds.apache.org/job/Hadoop-Common-trunk/732/])
    

> incomplete help message  is displayed for df -h option
> ------------------------------------------------------
>
>                 Key: HADOOP-7329
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7329
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Assignee: XieXianshan
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7329
>
>
> The help message for the command "hdfs dfs -help df" is displayed like this:
> "-df [<path> ...]:    Shows the capacity, free and used space of the filesystem.
>         If the filesystem has multiple partitions, and no path to a
>         particular partition is specified, then the status of the root
>         partitions will be shown."
> and the information about df -h option is missed,despite the fact that df -h option is implemented.
> Therefore,the expected message should be displayed like this:
> "-df [-h] [<path> ...]:    Shows the capacity, free and used space of the filesystem.
>         If the filesystem has multiple partitions, and no path to a
>         particular partition is specified, then the status of the root
>         partitions will be shown.
>           -h   Formats the sizes of files in a human-readable fashion
>                rather than a number of bytes."

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16835-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 11:18:17 2011
Return-Path: <common-issues-return-16835-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BFFAB4F81
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 11:18:17 +0000 (UTC)
Received: (qmail 74578 invoked by uid 500); 29 Jun 2011 11:18:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74289 invoked by uid 500); 29 Jun 2011 11:18:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74191 invoked by uid 99); 29 Jun 2011 11:18:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 11:18:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 11:18:10 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DD05E439F09
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 11:17:32 +0000 (UTC)
Date: Wed, 29 Jun 2011 11:17:32 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1061220275.2105.1309346252902.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <680679068.21358.1308591287861.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7407) Snappy integration breaks HDFS
 build.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057161#comment-13057161 ] 

Hudson commented on HADOOP-7407:
--------------------------------

Integrated in Hadoop-Common-trunk #732 (See [https://builds.apache.org/job/Hadoop-Common-trunk/732/])
    

> Snappy integration breaks HDFS build.
> -------------------------------------
>
>                 Key: HADOOP-7407
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7407
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Ivan Kelly
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7407.patch
>
>
> The common/ivy/hadoop-common-template.xml submitted with 7206 has a typo which breaks anything that depends on the hadoop-common maven package.
> Instead of <artifactId>java-snappy</artifactId>, you should have <artifactId>snappy-java</artifactId>
> [ivy:resolve] downloading https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.23.0-SNAPSHOT/hadoop-common-0.23.0-20110620.163810-177.jar ...
> [ivy:resolve] .......................
> [ivy:resolve] ..................................
> [ivy:resolve] ...........................................
> [ivy:resolve] ...............................................................
> [ivy:resolve] ................................................ (1631kB)
> [ivy:resolve] .. (0kB)
> [ivy:resolve] 	[SUCCESSFUL ] org.apache.hadoop#hadoop-common;0.23.0-SNAPSHOT!hadoop-common.jar (8441ms)
> [ivy:resolve] 
> [ivy:resolve] :: problems summary ::
> [ivy:resolve] :::: WARNINGS
> [ivy:resolve] 		module not found: org.xerial.snappy#java-snappy;1.0.3-rc2
> [ivy:resolve] 	==== apache-snapshot: tried
> [ivy:resolve] 	  https://repository.apache.org/content/repositories/snapshots/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.pom
> [ivy:resolve] 	  -- artifact org.xerial.snappy#java-snappy;1.0.3-rc2!java-snappy.jar:
> [ivy:resolve] 	  https://repository.apache.org/content/repositories/snapshots/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.jar
> [ivy:resolve] 	==== maven2: tried
> [ivy:resolve] 	  http://repo1.maven.org/maven2/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.pom
> [ivy:resolve] 	  -- artifact org.xerial.snappy#java-snappy;1.0.3-rc2!java-snappy.jar:
> [ivy:resolve] 	  http://repo1.maven.org/maven2/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.jar
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 		::          UNRESOLVED DEPENDENCIES         ::
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 		:: org.xerial.snappy#java-snappy;1.0.3-rc2: not found
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16836-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 14:45:15 2011
Return-Path: <common-issues-return-16836-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9271E4AE6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 14:45:15 +0000 (UTC)
Received: (qmail 27121 invoked by uid 500); 29 Jun 2011 14:45:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26529 invoked by uid 500); 29 Jun 2011 14:45:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26341 invoked by uid 99); 29 Jun 2011 14:45:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 14:45:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 14:45:11 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B9106439F69
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 14:44:30 +0000 (UTC)
Date: Wed, 29 Jun 2011 14:44:30 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1393669049.2629.1309358670754.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7106) Re-organize hadoop subversion
 layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057262#comment-13057262 ] 

Hudson commented on HADOOP-7106:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk #722 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/722/])
    

> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, gitk-example.png, mailer-conf.diff
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16837-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 14:45:33 2011
Return-Path: <common-issues-return-16837-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 783974D62
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 14:45:33 +0000 (UTC)
Received: (qmail 28649 invoked by uid 500); 29 Jun 2011 14:45:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27989 invoked by uid 500); 29 Jun 2011 14:45:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27965 invoked by uid 99); 29 Jun 2011 14:45:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 14:45:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 14:45:30 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 71687439FC0
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 14:44:33 +0000 (UTC)
Date: Wed, 29 Jun 2011 14:44:33 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <821144565.2697.1309358673461.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1507121853.1613.1307925711791.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7384) Allow test-patch to be more
 flexible about patch format
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057271#comment-13057271 ] 

Hudson commented on HADOOP-7384:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk #722 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/722/])
    

> Allow test-patch to be more flexible about patch format
> -------------------------------------------------------
>
>                 Key: HADOOP-7384
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7384
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7384.txt
>
>
> Right now the test-patch process only accepts patches that are generated as "-p0" relative to common/, hdfs/, or mapreduce/. This has always been annoying for git users where the default patch format is -p1. It's also now annoying for SVN users who may generate a patch relative to trunk/ instead of the subproject subdirectory. We should auto-detect the correct patch level.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16838-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 19:10:56 2011
Return-Path: <common-issues-return-16838-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6D9C14875
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 19:10:56 +0000 (UTC)
Received: (qmail 91781 invoked by uid 500); 29 Jun 2011 19:10:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91340 invoked by uid 500); 29 Jun 2011 19:10:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91176 invoked by uid 99); 29 Jun 2011 19:10:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 19:10:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 19:10:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id ED19D4393E1
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 19:10:28 +0000 (UTC)
Date: Wed, 29 Jun 2011 19:10:28 +0000 (UTC)
From: "Ravi Prakash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <510946152.3395.1309374628967.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1225328612.2229.1309278017362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7430) Improve error message when moving to
 trash fails due to quota issue
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Prakash updated HADOOP-7430:
---------------------------------

    Attachment: HADOOP-7430.1.patch

Attaching a patch to fix this regression. This patch applies to trunk revision 1141162.

Could someone please answer two of my questions?

1. In Trash.java I'm having to do a {noformat}cause.toString().indexOf("QuotaExceededException") != -1 {noformat} to find if the Exception was a QuotaExceededException (and then too it is not deterministic (what if the message contained that string?)). I would love to do an instanceOf but then HDFS would be added as a dependency to common (QuotaExceededException is defined in HDFS). How can I circumvent this / should I circumvent this? Can you please point me to a pre-existing discussion if one exists?
2. Similarly in TestTrash.java, to test that the message is also displayed when quota is exceeded, I should call setQuota() in hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java. Again, I'd have to add HDFS as a dependency for testing. I haven't done it in this patch.

Also, when I ran test-patch on this, I got
{noformat}
 [exec]     -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.
{noformat}
Here's test-patch's javadoc output
{noformat}
     [exec]   [javadoc] Constructing Javadoc information...
     [exec]   [javadoc] /home/raviprak/Code/hadoop/svn/svnhadoop-common/common/src/java/org/apache/hadoop/security/SecurityUtil.java:40: warning: sun.security.jgss.krb5.Krb5Util is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.jgss.krb5.Krb5Util;
     [exec]   [javadoc]                              ^
     [exec]   [javadoc] /home/raviprak/Code/hadoop/svn/svnhadoop-common/common/src/java/org/apache/hadoop/security/SecurityUtil.java:41: warning: sun.security.krb5.Credentials is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.Credentials;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /home/raviprak/Code/hadoop/svn/svnhadoop-common/common/src/java/org/apache/hadoop/security/SecurityUtil.java:42: warning: sun.security.krb5.PrincipalName is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.PrincipalName;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /home/raviprak/Code/hadoop/svn/svnhadoop-common/common/src/java/org/apache/hadoop/security/KerberosName.java:31: warning: sun.security.krb5.Config is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.Config;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /home/raviprak/Code/hadoop/svn/svnhadoop-common/common/src/java/org/apache/hadoop/security/KerberosName.java:32: warning: sun.security.krb5.KrbException is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.KrbException;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /home/raviprak/Code/hadoop/svn/svnhadoop-common/common/src/java/org/apache/hadoop/security/KerberosName.java:81: warning: sun.security.krb5.Config is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc]   private static Config kerbConf;
     [exec]   [javadoc]                  ^
     [exec]   [javadoc] ExcludePrivateAnnotationsStandardDoclet
     [exec]   [javadoc] javadoc: warning - Error fetching URL: http://java.sun.com/javase/6/docs/api/package-list
     [exec]   [javadoc] Standard Doclet version 1.6.0_01
     [exec]   [javadoc] Building tree for all the packages and classes...
     [exec]   [javadoc] Building index for all the packages and classes...
     [exec]   [javadoc] Building index for all classes...
     [exec]   [javadoc] Generating /home/raviprak/Code/hadoop/svn/svnhadoop-common/common/build/docs/api/stylesheet.css...
     [exec]   [javadoc] 7 warnings
     [exec] 
     [exec] BUILD SUCCESSFUL
     [exec] Total time: 3 minutes 48 seconds
     [exec] 
     [exec] 
     [exec] There appear to be 7 javadoc warnings generated by the patched build.
{noformat}
But I didn't mess with any files. :( Any clues why I'm -1?

Thanks


> Improve error message when moving to trash fails due to quota issue
> -------------------------------------------------------------------
>
>                 Key: HADOOP-7430
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7430
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Ravi Prakash
>            Assignee: Ravi Prakash
>            Priority: Minor
>         Attachments: HADOOP-7430.1.patch
>
>
> -rm command doesn't suggest -skipTrash on failure.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16839-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 21:52:51 2011
Return-Path: <common-issues-return-16839-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4F6504D36
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 21:52:51 +0000 (UTC)
Received: (qmail 46600 invoked by uid 500); 29 Jun 2011 21:52:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46308 invoked by uid 500); 29 Jun 2011 21:52:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46295 invoked by uid 99); 29 Jun 2011 21:52:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 21:52:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 21:52:48 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C164B43A815
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 21:52:28 +0000 (UTC)
Date: Wed, 29 Jun 2011 21:52:28 +0000 (UTC)
From: "Ravi Prakash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <984905995.3866.1309384348789.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1225328612.2229.1309278017362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7430) Improve error message when moving to
 trash fails due to quota issue
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Prakash updated HADOOP-7430:
---------------------------------

    Status: Patch Available  (was: Open)

> Improve error message when moving to trash fails due to quota issue
> -------------------------------------------------------------------
>
>                 Key: HADOOP-7430
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7430
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Ravi Prakash
>            Assignee: Ravi Prakash
>            Priority: Minor
>         Attachments: HADOOP-7430.1.patch
>
>
> -rm command doesn't suggest -skipTrash on failure.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16840-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 22:14:51 2011
Return-Path: <common-issues-return-16840-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C35F64A58
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 22:14:51 +0000 (UTC)
Received: (qmail 74129 invoked by uid 500); 29 Jun 2011 22:14:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73716 invoked by uid 500); 29 Jun 2011 22:14:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73686 invoked by uid 99); 29 Jun 2011 22:14:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 22:14:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 22:14:49 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 04AD943A144
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 22:14:29 +0000 (UTC)
Date: Wed, 29 Jun 2011 22:14:29 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <442901305.3945.1309385669015.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1225328612.2229.1309278017362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7430) Improve error message when moving
 to trash fails due to quota issue
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057512#comment-13057512 ] 

Hadoop QA commented on HADOOP-7430:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12484676/HADOOP-7430.1.patch
  against trunk revision 1140442.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/684//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/684//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/684//console

This message is automatically generated.

> Improve error message when moving to trash fails due to quota issue
> -------------------------------------------------------------------
>
>                 Key: HADOOP-7430
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7430
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Ravi Prakash
>            Assignee: Ravi Prakash
>            Priority: Minor
>         Attachments: HADOOP-7430.1.patch
>
>
> -rm command doesn't suggest -skipTrash on failure.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16841-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 22:39:50 2011
Return-Path: <common-issues-return-16841-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8D60F4ABC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 22:39:50 +0000 (UTC)
Received: (qmail 96968 invoked by uid 500); 29 Jun 2011 22:39:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96901 invoked by uid 500); 29 Jun 2011 22:39:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96887 invoked by uid 99); 29 Jun 2011 22:39:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 22:39:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 22:39:48 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7399143A846
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 22:39:28 +0000 (UTC)
Date: Wed, 29 Jun 2011 22:39:28 +0000 (UTC)
From: "Sherry Chen (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <215422977.3987.1309387168470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7432) Back-port HADOOP-7110 to
 0.20-security
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Back-port HADOOP-7110 to 0.20-security
--------------------------------------

                 Key: HADOOP-7432
                 URL: https://issues.apache.org/jira/browse/HADOOP-7432
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Sherry Chen
            Assignee: Sherry Chen


HADOOP-7110 implemented chmod in the NativeIO library so we can have good performance (ie not fork) and still not be prone to races. This should fix build failures (and probably task failures too).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16842-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 22:43:51 2011
Return-Path: <common-issues-return-16842-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 45D094B62
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 22:43:51 +0000 (UTC)
Received: (qmail 934 invoked by uid 500); 29 Jun 2011 22:43:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 857 invoked by uid 500); 29 Jun 2011 22:43:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 833 invoked by uid 99); 29 Jun 2011 22:43:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 22:43:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 22:43:49 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 077EF43A979
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 22:43:29 +0000 (UTC)
Date: Wed, 29 Jun 2011 22:43:29 +0000 (UTC)
From: "Sherry Chen (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1369468418.4000.1309387409027.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <215422977.3987.1309387168470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7432) Back-port HADOOP-7110 to
 0.20-security
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sherry Chen updated HADOOP-7432:
--------------------------------

    Attachment: HADOOP-7432.patch

Back-ported HADOOP-7110 to 0.20-security.

> Back-port HADOOP-7110 to 0.20-security
> --------------------------------------
>
>                 Key: HADOOP-7432
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7432
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sherry Chen
>            Assignee: Sherry Chen
>         Attachments: HADOOP-7432.patch
>
>
> HADOOP-7110 implemented chmod in the NativeIO library so we can have good performance (ie not fork) and still not be prone to races. This should fix build failures (and probably task failures too).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16843-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 22:43:51 2011
Return-Path: <common-issues-return-16843-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9ADC64B72
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 22:43:51 +0000 (UTC)
Received: (qmail 1191 invoked by uid 500); 29 Jun 2011 22:43:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 868 invoked by uid 500); 29 Jun 2011 22:43:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 835 invoked by uid 99); 29 Jun 2011 22:43:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 22:43:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 22:43:49 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 207CA43A97C
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 22:43:29 +0000 (UTC)
Date: Wed, 29 Jun 2011 22:43:29 +0000 (UTC)
From: "Sherry Chen (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <469896139.4003.1309387409130.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <215422977.3987.1309387168470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7432) Back-port HADOOP-7110 to
 0.20-security
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sherry Chen updated HADOOP-7432:
--------------------------------

    Status: Patch Available  (was: Open)

Manual test result:

     [exec] 
     [exec] +1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.
     [exec] 
     [exec] 
     [exec] 
     [exec] 
     [exec] ======================================================================
     [exec] ======================================================================
     [exec]     Finished build.
     [exec] ======================================================================
     [exec] ======================================================================


> Back-port HADOOP-7110 to 0.20-security
> --------------------------------------
>
>                 Key: HADOOP-7432
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7432
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sherry Chen
>            Assignee: Sherry Chen
>         Attachments: HADOOP-7432.patch
>
>
> HADOOP-7110 implemented chmod in the NativeIO library so we can have good performance (ie not fork) and still not be prone to races. This should fix build failures (and probably task failures too).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16844-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 22:53:53 2011
Return-Path: <common-issues-return-16844-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DDCA24431
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 22:53:52 +0000 (UTC)
Received: (qmail 16689 invoked by uid 500); 29 Jun 2011 22:53:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16345 invoked by uid 500); 29 Jun 2011 22:53:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16330 invoked by uid 99); 29 Jun 2011 22:53:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 22:53:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 22:53:49 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 702C243ACFF
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 22:53:29 +0000 (UTC)
Date: Wed, 29 Jun 2011 22:53:29 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1403843086.4064.1309388009456.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <215422977.3987.1309387168470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7432) Back-port HADOOP-7110 to
 0.20-security
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057536#comment-13057536 ] 

Hadoop QA commented on HADOOP-7432:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12484697/HADOOP-7432.patch
  against trunk revision 1140442.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/685//console

This message is automatically generated.

> Back-port HADOOP-7110 to 0.20-security
> --------------------------------------
>
>                 Key: HADOOP-7432
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7432
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sherry Chen
>            Assignee: Sherry Chen
>         Attachments: HADOOP-7432.patch
>
>
> HADOOP-7110 implemented chmod in the NativeIO library so we can have good performance (ie not fork) and still not be prone to races. This should fix build failures (and probably task failures too).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16845-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 23:13:50 2011
Return-Path: <common-issues-return-16845-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 899774F22
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 23:13:50 +0000 (UTC)
Received: (qmail 49329 invoked by uid 500); 29 Jun 2011 23:13:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49256 invoked by uid 500); 29 Jun 2011 23:13:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49248 invoked by uid 99); 29 Jun 2011 23:13:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 23:13:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 23:13:48 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 79E1543930D
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 23:13:28 +0000 (UTC)
Date: Wed, 29 Jun 2011 23:13:28 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <276265505.4104.1309389208496.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7433) Snappy SO file/links are copied to
 the wrong directory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Snappy SO file/links are copied to the wrong directory
------------------------------------------------------

                 Key: HADOOP-7433
                 URL: https://issues.apache.org/jira/browse/HADOOP-7433
             Project: Hadoop Common
          Issue Type: Bug
          Components: build
            Reporter: Alejandro Abdelnur
            Assignee: Alejandro Abdelnur
             Fix For: 0.23.0


The SO file/links are copied into lib/native/, they should be copied to lib/native/$OS_ARCH

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16846-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 23:19:50 2011
Return-Path: <common-issues-return-16846-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E4BEF4668
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 23:19:50 +0000 (UTC)
Received: (qmail 55908 invoked by uid 500); 29 Jun 2011 23:19:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55873 invoked by uid 500); 29 Jun 2011 23:19:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55851 invoked by uid 99); 29 Jun 2011 23:19:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 23:19:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 23:19:48 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DEAB44395F6
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 23:19:28 +0000 (UTC)
Date: Wed, 29 Jun 2011 23:19:28 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <60281614.4125.1309389568909.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <276265505.4104.1309389208496.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7433) Snappy SO file/links are copied to
 the wrong directory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-7433:
---------------------------------------

    Attachment: HADOOP-7433.patch

> Snappy SO file/links are copied to the wrong directory
> ------------------------------------------------------
>
>                 Key: HADOOP-7433
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7433
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7433.patch
>
>
> The SO file/links are copied into lib/native/, they should be copied to lib/native/$OS_ARCH

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16847-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 23:19:51 2011
Return-Path: <common-issues-return-16847-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 26B534685
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 23:19:51 +0000 (UTC)
Received: (qmail 56122 invoked by uid 500); 29 Jun 2011 23:19:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55894 invoked by uid 500); 29 Jun 2011 23:19:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55858 invoked by uid 99); 29 Jun 2011 23:19:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 23:19:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 23:19:48 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EAD3B4395FC
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 23:19:28 +0000 (UTC)
Date: Wed, 29 Jun 2011 23:19:28 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <367778316.4127.1309389568958.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <276265505.4104.1309389208496.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7433) Snappy SO file/links are copied to
 the wrong directory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-7433:
---------------------------------------

    Status: Patch Available  (was: Open)

> Snappy SO file/links are copied to the wrong directory
> ------------------------------------------------------
>
>                 Key: HADOOP-7433
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7433
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7433.patch
>
>
> The SO file/links are copied into lib/native/, they should be copied to lib/native/$OS_ARCH

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16848-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed Jun 29 23:33:52 2011
Return-Path: <common-issues-return-16848-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C59704EDF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 29 Jun 2011 23:33:52 +0000 (UTC)
Received: (qmail 71568 invoked by uid 500); 29 Jun 2011 23:33:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71544 invoked by uid 500); 29 Jun 2011 23:33:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71535 invoked by uid 99); 29 Jun 2011 23:33:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 23:33:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Jun 2011 23:33:49 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 75F8C43999E
	for <common-issues@hadoop.apache.org>; Wed, 29 Jun 2011 23:33:28 +0000 (UTC)
Date: Wed, 29 Jun 2011 23:33:28 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1771618740.4132.1309390408479.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <276265505.4104.1309389208496.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7433) Snappy SO file/links are copied to
 the wrong directory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057549#comment-13057549 ] 

Hadoop QA commented on HADOOP-7433:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12484703/HADOOP-7433.patch
  against trunk revision 1140442.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/686//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/686//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/686//console

This message is automatically generated.

> Snappy SO file/links are copied to the wrong directory
> ------------------------------------------------------
>
>                 Key: HADOOP-7433
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7433
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7433.patch
>
>
> The SO file/links are copied into lib/native/, they should be copied to lib/native/$OS_ARCH

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16849-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 02:01:52 2011
Return-Path: <common-issues-return-16849-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 605CD6018
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 02:01:52 +0000 (UTC)
Received: (qmail 97022 invoked by uid 500); 30 Jun 2011 02:01:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96973 invoked by uid 500); 30 Jun 2011 02:01:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96957 invoked by uid 99); 30 Jun 2011 02:01:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 02:01:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 02:01:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0D30043AC5F
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 02:01:30 +0000 (UTC)
Date: Thu, 30 Jun 2011 02:01:30 +0000 (UTC)
From: =?utf-8?Q?=E4=B8=A5=E9=87=91=E5=8F=8C_=28JIRA=29?= <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <300592700.4296.1309399290050.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7434) there is no error message displayed
 while using the command "daemonlog -setlevel" with illegal level
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

there is no error message displayed while using the command "daemonlog -set=
level" with illegal level
---------------------------------------------------------------------------=
-------------------------

                 Key: HADOOP-7434
                 URL: https://issues.apache.org/jira/browse/HADOOP-7434
             Project: Hadoop Common
          Issue Type: Improvement
    Affects Versions: 0.23.0
            Reporter: =E4=B8=A5=E9=87=91=E5=8F=8C


While using the command with inexistent "level" like "nomsg", there is no e=
rror message displayed,and the level "DEBUG" is set by default.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

From common-issues-return-16850-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 02:13:52 2011
Return-Path: <common-issues-return-16850-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 656896E74
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 02:13:52 +0000 (UTC)
Received: (qmail 11370 invoked by uid 500); 30 Jun 2011 02:13:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11342 invoked by uid 500); 30 Jun 2011 02:13:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11334 invoked by uid 99); 30 Jun 2011 02:13:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 02:13:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 02:13:49 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5DD1F43AE89
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 02:13:28 +0000 (UTC)
Date: Thu, 30 Jun 2011 02:13:28 +0000 (UTC)
From: "XieXianshan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <253534842.4301.1309400008381.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <300592700.4296.1309399290050.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7434) there is no error message displayed
 while using the command "daemonlog -setlevel" with illegal level
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7434?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

XieXianshan updated HADOOP-7434:
--------------------------------

    Attachment: HADOOP-7434

Fixed this issue.
The error msg will be displayed as follow while the command with illegal le=
vel.

Submitted Level: ERRORLEVEL
Bad level "ERRORLEVEL"

> there is no error message displayed while using the command "daemonlog -s=
etlevel" with illegal level
> -------------------------------------------------------------------------=
---------------------------
>
>                 Key: HADOOP-7434
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7434
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: =E4=B8=A5=E9=87=91=E5=8F=8C
>         Attachments: HADOOP-7434
>
>
> While using the command with inexistent "level" like "nomsg", there is no=
 error message displayed,and the level "DEBUG" is set by default.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

From common-issues-return-16851-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 02:15:52 2011
Return-Path: <common-issues-return-16851-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A15FD60A7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 02:15:52 +0000 (UTC)
Received: (qmail 13511 invoked by uid 500); 30 Jun 2011 02:15:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13362 invoked by uid 500); 30 Jun 2011 02:15:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13351 invoked by uid 99); 30 Jun 2011 02:15:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 02:15:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 02:15:49 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6608843AF7B
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 02:15:28 +0000 (UTC)
Date: Thu, 30 Jun 2011 02:15:28 +0000 (UTC)
From: "XieXianshan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1448314972.4303.1309400128414.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <300592700.4296.1309399290050.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7434) there is no error message displayed
 while using the command "daemonlog -setlevel" with illegal level
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7434?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

XieXianshan updated HADOOP-7434:
--------------------------------

    Fix Version/s: 0.23.0
           Status: Patch Available  (was: Open)

> there is no error message displayed while using the command "daemonlog -s=
etlevel" with illegal level
> -------------------------------------------------------------------------=
---------------------------
>
>                 Key: HADOOP-7434
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7434
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: =E4=B8=A5=E9=87=91=E5=8F=8C
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7434
>
>
> While using the command with inexistent "level" like "nomsg", there is no=
 error message displayed,and the level "DEBUG" is set by default.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

From common-issues-return-16852-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 02:33:56 2011
Return-Path: <common-issues-return-16852-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5D8FD6861
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 02:33:56 +0000 (UTC)
Received: (qmail 29009 invoked by uid 500); 30 Jun 2011 02:33:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27827 invoked by uid 500); 30 Jun 2011 02:33:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27646 invoked by uid 99); 30 Jun 2011 02:33:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 02:33:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 02:33:49 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A1A3443A474
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 02:33:28 +0000 (UTC)
Date: Thu, 30 Jun 2011 02:33:28 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1230569784.4325.1309401208658.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <300592700.4296.1309399290050.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7434) there is no error message
 displayed while using the command "daemonlog -setlevel" with illegal level
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7434?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D130=
57582#comment-13057582 ]=20

Hadoop QA commented on HADOOP-7434:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment=20
  http://issues.apache.org/jira/secure/attachment/12484712/HADOOP-7434
  against trunk revision 1140442.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modi=
fied tests.
                        Please justify why no new tests are needed for this=
 patch.
                        Also please list what manual steps were performed t=
o verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of java=
c compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.=
3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number=
 of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compi=
le.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/687//tes=
tReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/687=
//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/687//c=
onsole

This message is automatically generated.

> there is no error message displayed while using the command "daemonlog -s=
etlevel" with illegal level
> -------------------------------------------------------------------------=
---------------------------
>
>                 Key: HADOOP-7434
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7434
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: =E4=B8=A5=E9=87=91=E5=8F=8C
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7434
>
>
> While using the command with inexistent "level" like "nomsg", there is no=
 error message displayed,and the level "DEBUG" is set by default.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

From common-issues-return-16853-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 04:38:31 2011
Return-Path: <common-issues-return-16853-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 24B4969B3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 04:38:31 +0000 (UTC)
Received: (qmail 24189 invoked by uid 500); 30 Jun 2011 04:38:25 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22296 invoked by uid 500); 30 Jun 2011 04:38:05 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22239 invoked by uid 99); 30 Jun 2011 04:37:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 04:37:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 04:37:49 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 70B1243A8D9
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 04:37:28 +0000 (UTC)
Date: Thu, 30 Jun 2011 04:37:28 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1620802473.4453.1309408648457.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <276265505.4104.1309389208496.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7433) Snappy SO file/links are copied to
 the wrong directory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057611#comment-13057611 ] 

Eli Collins commented on HADOOP-7433:
-------------------------------------

Where does $DIST_LIB_DIR/$BUILD_PLATFORM get created?

testing?  ant -Dbundle.snappy=true compile-native tar blows up for me w this patch.

> Snappy SO file/links are copied to the wrong directory
> ------------------------------------------------------
>
>                 Key: HADOOP-7433
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7433
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7433.patch
>
>
> The SO file/links are copied into lib/native/, they should be copied to lib/native/$OS_ARCH

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16854-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 06:21:06 2011
Return-Path: <common-issues-return-16854-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3CA6463A8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 06:21:06 +0000 (UTC)
Received: (qmail 11584 invoked by uid 500); 30 Jun 2011 06:21:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11341 invoked by uid 500); 30 Jun 2011 06:20:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11311 invoked by uid 99); 30 Jun 2011 06:20:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 06:20:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 06:20:49 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 84D9643A8D5
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 06:20:28 +0000 (UTC)
Date: Thu, 30 Jun 2011 06:20:28 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1612498209.4599.1309414828540.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <403145852.745.1309238057130.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7429) Add another IOUtils#copyBytes
 method
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057640#comment-13057640 ] 

Aaron T. Myers commented on HADOOP-7429:
----------------------------------------

+1, the patch looks good to me.

> Add another IOUtils#copyBytes method
> ------------------------------------
>
>                 Key: HADOOP-7429
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7429
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>         Attachments: hadoop-7429-1.patch, hadoop-7429-2.patch
>
>
> Common side of HDFS-2110, adds a new IOUtils copy bytes method and cleanup.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16855-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 07:27:03 2011
Return-Path: <common-issues-return-16855-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 41F43630A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 07:27:03 +0000 (UTC)
Received: (qmail 75898 invoked by uid 500); 30 Jun 2011 07:27:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75543 invoked by uid 500); 30 Jun 2011 07:26:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75325 invoked by uid 99); 30 Jun 2011 07:26:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 07:26:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 07:26:48 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D574843AF7B
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 07:26:28 +0000 (UTC)
Date: Thu, 30 Jun 2011 07:26:28 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1942287280.4631.1309418788871.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-310) Additional constructor requested in
 BytesWritable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057645#comment-13057645 ] 

Hudson commented on HADOOP-310:
-------------------------------

Integrated in Hadoop-Common-trunk-Commit #668 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/668/])
    

> Additional constructor requested in BytesWritable
> -------------------------------------------------
>
>                 Key: HADOOP-310
>                 URL: https://issues.apache.org/jira/browse/HADOOP-310
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: p sutter
>            Assignee: Brock Noland
>            Priority: Minor
>         Attachments: bytes-writable-zero-copy-interface-0.patch, bytes-writable-zero-copy-interface-1.patch, bytes-writable-zero-copy-interface-2.patch
>
>
> It would be grand if BytesWritable.java had an additional constructor as below. This allows me to use the BytesWritable class without doing a buffer copy, since we have a less-than-fully-utilized byte array holding our key.
> Thanks!
>  
>  /**
>    * Create a BytesWritable using the byte array as the initial value.
>    * @param bytes This array becomes the backing storage for the object.
>    */
>   public BytesWritable(byte[] bytes, int size) {
>     this.bytes = bytes;
>     this.size = size;
>   }
>   

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16856-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 07:27:04 2011
Return-Path: <common-issues-return-16856-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A1174631B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 07:27:04 +0000 (UTC)
Received: (qmail 76226 invoked by uid 500); 30 Jun 2011 07:27:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75861 invoked by uid 500); 30 Jun 2011 07:27:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75353 invoked by uid 99); 30 Jun 2011 07:26:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 07:26:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 07:26:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 940B943AFD2
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 07:26:31 +0000 (UTC)
Date: Thu, 30 Jun 2011 07:26:31 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <305905078.4684.1309418791603.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057646#comment-13057646 ] 

Hudson commented on HADOOP-7206:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #668 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/668/])
    

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>            Assignee: Alejandro Abdelnur
>         Attachments: HADOOP-7206-002.patch, HADOOP-7206.patch, HADOOP-7206new-b.patch, HADOOP-7206new-c.patch, HADOOP-7206new.patch, HADOOP-7206revertplusnew-b.patch, HADOOP-7206revertplusnew.patch, v2-HADOOP-7206-snappy-codec-using-snappy-java.txt, v3-HADOOP-7206-snappy-codec-using-snappy-java.txt, v4-HADOOP-7206-snappy-codec-using-snappy-java.txt, v5-HADOOP-7206-snappy-codec-using-snappy-java.txt
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16857-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 07:27:05 2011
Return-Path: <common-issues-return-16857-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9FA016339
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 07:27:05 +0000 (UTC)
Received: (qmail 76508 invoked by uid 500); 30 Jun 2011 07:27:05 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76132 invoked by uid 500); 30 Jun 2011 07:27:04 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75355 invoked by uid 99); 30 Jun 2011 07:26:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 07:26:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 07:26:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AB6DD43AFD4
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 07:26:31 +0000 (UTC)
Date: Thu, 30 Jun 2011 07:26:31 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <734187603.4686.1309418791698.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <403145852.745.1309238057130.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7429) Add another IOUtils#copyBytes
 method
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057647#comment-13057647 ] 

Hudson commented on HADOOP-7429:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #668 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/668/])
    Minor update to HADOOP-7429.

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1141415
Files : 
* /hadoop/common/trunk/common/src/java/org/apache/hadoop/io/IOUtils.java


> Add another IOUtils#copyBytes method
> ------------------------------------
>
>                 Key: HADOOP-7429
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7429
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>         Attachments: hadoop-7429-1.patch, hadoop-7429-2.patch
>
>
> Common side of HDFS-2110, adds a new IOUtils copy bytes method and cleanup.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16858-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 07:27:06 2011
Return-Path: <common-issues-return-16858-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DF9686340
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 07:27:05 +0000 (UTC)
Received: (qmail 76699 invoked by uid 500); 30 Jun 2011 07:27:05 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76482 invoked by uid 500); 30 Jun 2011 07:27:05 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75354 invoked by uid 99); 30 Jun 2011 07:26:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 07:26:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 07:26:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CA13D43AFD7
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 07:26:31 +0000 (UTC)
Date: Thu, 30 Jun 2011 07:26:31 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <679494199.4689.1309418791824.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <113408089.3186.1307985891583.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7385) Remove
 StringUtils.stringifyException(ie) in logger functions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057648#comment-13057648 ] 

Hudson commented on HADOOP-7385:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #668 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/668/])
    

> Remove StringUtils.stringifyException(ie) in logger functions
> -------------------------------------------------------------
>
>                 Key: HADOOP-7385
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7385
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7385-1.patch
>
>
> Apache logger api has an overloaded function which can take the message and exception. I am proposing to clean the logging code with this api.
> ie.:
> Change the code from LOG.warn(msg, StringUtils.stringifyException(exception)); to LOG.warn(msg, exception);

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16859-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 07:27:06 2011
Return-Path: <common-issues-return-16859-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5FF136356
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 07:27:06 +0000 (UTC)
Received: (qmail 77012 invoked by uid 500); 30 Jun 2011 07:27:05 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76536 invoked by uid 500); 30 Jun 2011 07:27:05 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75616 invoked by uid 99); 30 Jun 2011 07:26:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 07:26:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 07:26:52 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 179FD43AFDF
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 07:26:32 +0000 (UTC)
Date: Thu, 30 Jun 2011 07:26:32 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <512872261.4694.1309418792093.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1164170325.41524.1306304627421.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7329) incomplete help message  is
 displayed for df -h option
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057649#comment-13057649 ] 

Hudson commented on HADOOP-7329:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #668 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/668/])
    

> incomplete help message  is displayed for df -h option
> ------------------------------------------------------
>
>                 Key: HADOOP-7329
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7329
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Assignee: XieXianshan
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7329
>
>
> The help message for the command "hdfs dfs -help df" is displayed like this:
> "-df [<path> ...]:    Shows the capacity, free and used space of the filesystem.
>         If the filesystem has multiple partitions, and no path to a
>         particular partition is specified, then the status of the root
>         partitions will be shown."
> and the information about df -h option is missed,despite the fact that df -h option is implemented.
> Therefore,the expected message should be displayed like this:
> "-df [-h] [<path> ...]:    Shows the capacity, free and used space of the filesystem.
>         If the filesystem has multiple partitions, and no path to a
>         particular partition is specified, then the status of the root
>         partitions will be shown.
>           -h   Formats the sizes of files in a human-readable fashion
>                rather than a number of bytes."

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16860-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 07:27:16 2011
Return-Path: <common-issues-return-16860-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3F16A636A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 07:27:16 +0000 (UTC)
Received: (qmail 77392 invoked by uid 500); 30 Jun 2011 07:27:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77298 invoked by uid 500); 30 Jun 2011 07:27:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77274 invoked by uid 99); 30 Jun 2011 07:27:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 07:27:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 07:27:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4040743AFE4
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 07:26:32 +0000 (UTC)
Date: Thu, 30 Jun 2011 07:26:32 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <359763028.4698.1309418792259.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <680679068.21358.1308591287861.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7407) Snappy integration breaks HDFS
 build.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057650#comment-13057650 ] 

Hudson commented on HADOOP-7407:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #668 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/668/])
    

> Snappy integration breaks HDFS build.
> -------------------------------------
>
>                 Key: HADOOP-7407
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7407
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Ivan Kelly
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7407.patch
>
>
> The common/ivy/hadoop-common-template.xml submitted with 7206 has a typo which breaks anything that depends on the hadoop-common maven package.
> Instead of <artifactId>java-snappy</artifactId>, you should have <artifactId>snappy-java</artifactId>
> [ivy:resolve] downloading https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common/0.23.0-SNAPSHOT/hadoop-common-0.23.0-20110620.163810-177.jar ...
> [ivy:resolve] .......................
> [ivy:resolve] ..................................
> [ivy:resolve] ...........................................
> [ivy:resolve] ...............................................................
> [ivy:resolve] ................................................ (1631kB)
> [ivy:resolve] .. (0kB)
> [ivy:resolve] 	[SUCCESSFUL ] org.apache.hadoop#hadoop-common;0.23.0-SNAPSHOT!hadoop-common.jar (8441ms)
> [ivy:resolve] 
> [ivy:resolve] :: problems summary ::
> [ivy:resolve] :::: WARNINGS
> [ivy:resolve] 		module not found: org.xerial.snappy#java-snappy;1.0.3-rc2
> [ivy:resolve] 	==== apache-snapshot: tried
> [ivy:resolve] 	  https://repository.apache.org/content/repositories/snapshots/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.pom
> [ivy:resolve] 	  -- artifact org.xerial.snappy#java-snappy;1.0.3-rc2!java-snappy.jar:
> [ivy:resolve] 	  https://repository.apache.org/content/repositories/snapshots/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.jar
> [ivy:resolve] 	==== maven2: tried
> [ivy:resolve] 	  http://repo1.maven.org/maven2/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.pom
> [ivy:resolve] 	  -- artifact org.xerial.snappy#java-snappy;1.0.3-rc2!java-snappy.jar:
> [ivy:resolve] 	  http://repo1.maven.org/maven2/org/xerial/snappy/java-snappy/1.0.3-rc2/java-snappy-1.0.3-rc2.jar
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 		::          UNRESOLVED DEPENDENCIES         ::
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 		:: org.xerial.snappy#java-snappy;1.0.3-rc2: not found
> [ivy:resolve] 		::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16861-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 09:11:04 2011
Return-Path: <common-issues-return-16861-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8CED86572
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 09:11:04 +0000 (UTC)
Received: (qmail 24543 invoked by uid 500); 30 Jun 2011 09:11:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24036 invoked by uid 500); 30 Jun 2011 09:10:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24021 invoked by uid 99); 30 Jun 2011 09:10:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 09:10:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 09:10:48 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 73E2743BB2C
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 09:10:28 +0000 (UTC)
Date: Thu, 30 Jun 2011 09:10:28 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1394372660.4935.1309425028471.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <403145852.745.1309238057130.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7429) Add another IOUtils#copyBytes
 method
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057704#comment-13057704 ] 

Hudson commented on HADOOP-7429:
--------------------------------

Integrated in Hadoop-Common-trunk-maven #44 (See [https://builds.apache.org/job/Hadoop-Common-trunk-maven/44/])
    Minor update to HADOOP-7429.

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1141415
Files : 
* /hadoop/common/trunk/common/src/java/org/apache/hadoop/io/IOUtils.java


> Add another IOUtils#copyBytes method
> ------------------------------------
>
>                 Key: HADOOP-7429
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7429
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>         Attachments: hadoop-7429-1.patch, hadoop-7429-2.patch
>
>
> Common side of HDFS-2110, adds a new IOUtils copy bytes method and cleanup.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16862-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 09:33:05 2011
Return-Path: <common-issues-return-16862-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 133EF6F26
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 09:33:05 +0000 (UTC)
Received: (qmail 54924 invoked by uid 500); 30 Jun 2011 09:33:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54667 invoked by uid 500); 30 Jun 2011 09:32:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54629 invoked by uid 99); 30 Jun 2011 09:32:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 09:32:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 09:32:49 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 926FC43BC6A
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 09:32:28 +0000 (UTC)
Date: Thu, 30 Jun 2011 09:32:28 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <694017055.4990.1309426348596.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <28994504.166561290042553993.JavaMail.jira@thor>
Subject: [jira] [Resolved] (HADOOP-7040) DiskChecker:mkdirsWithExistsCheck
 swallows FileNotFoundException.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins resolved HADOOP-7040.
---------------------------------

       Resolution: Fixed
    Fix Version/s:     (was: 0.20.2)
                   0.20.203.1

This was already fixed in 0.20.203.

> DiskChecker:mkdirsWithExistsCheck swallows FileNotFoundException.
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7040
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7040
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.1, 0.20.2, 0.21.0
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>             Fix For: 0.20.203.1
>
>         Attachments: 4132253-3.patch
>
>
> As a result, DataNode.checkDir will miss the exception (it catches DiskErrorException, not FileNotFoundException), and fail instead of ignoring the non-existent directory.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16863-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 11:14:56 2011
Return-Path: <common-issues-return-16863-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2E71F623B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 11:14:56 +0000 (UTC)
Received: (qmail 29479 invoked by uid 500); 30 Jun 2011 11:14:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29312 invoked by uid 500); 30 Jun 2011 11:14:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29298 invoked by uid 99); 30 Jun 2011 11:14:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 11:14:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 11:14:49 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5F11C43C98F
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 11:14:28 +0000 (UTC)
Date: Thu, 30 Jun 2011 11:14:28 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1126969463.5175.1309432468386.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <403145852.745.1309238057130.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7429) Add another IOUtils#copyBytes
 method
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057756#comment-13057756 ] 

Hudson commented on HADOOP-7429:
--------------------------------

Integrated in Hadoop-Common-trunk #733 (See [https://builds.apache.org/job/Hadoop-Common-trunk/733/])
    Minor update to HADOOP-7429.

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1141415
Files : 
* /hadoop/common/trunk/common/src/java/org/apache/hadoop/io/IOUtils.java


> Add another IOUtils#copyBytes method
> ------------------------------------
>
>                 Key: HADOOP-7429
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7429
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>         Attachments: hadoop-7429-1.patch, hadoop-7429-2.patch
>
>
> Common side of HDFS-2110, adds a new IOUtils copy bytes method and cleanup.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16864-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 15:19:51 2011
Return-Path: <common-issues-return-16864-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E7D0B6BBF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 15:19:51 +0000 (UTC)
Received: (qmail 73312 invoked by uid 500); 30 Jun 2011 15:19:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73119 invoked by uid 500); 30 Jun 2011 15:19:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73092 invoked by uid 99); 30 Jun 2011 15:19:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 15:19:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 15:19:48 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A3EE543CC48
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 15:19:28 +0000 (UTC)
Date: Thu, 30 Jun 2011 15:19:28 +0000 (UTC)
From: "Robert Joseph Evans (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <28110088.5806.1309447168667.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <215422977.3987.1309387168470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7432) Back-port HADOOP-7110 to
 0.20-security
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057877#comment-13057877 ] 

Robert Joseph Evans commented on HADOOP-7432:
---------------------------------------------

Why did line 130 of RawLocalFileSystem.java change from "%04o" to "%05o"?

> Back-port HADOOP-7110 to 0.20-security
> --------------------------------------
>
>                 Key: HADOOP-7432
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7432
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sherry Chen
>            Assignee: Sherry Chen
>         Attachments: HADOOP-7432.patch
>
>
> HADOOP-7110 implemented chmod in the NativeIO library so we can have good performance (ie not fork) and still not be prone to races. This should fix build failures (and probably task failures too).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16865-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 16:23:52 2011
Return-Path: <common-issues-return-16865-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 771CF69B7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 16:23:52 +0000 (UTC)
Received: (qmail 30974 invoked by uid 500); 30 Jun 2011 16:23:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30905 invoked by uid 500); 30 Jun 2011 16:23:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30894 invoked by uid 99); 30 Jun 2011 16:23:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 16:23:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 16:23:49 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7B11343CAC8
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 16:23:28 +0000 (UTC)
Date: Thu, 30 Jun 2011 16:23:28 +0000 (UTC)
From: "Sherry Chen (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1962569391.5964.1309451008501.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <215422977.3987.1309387168470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7432) Back-port HADOOP-7110 to
 0.20-security
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057914#comment-13057914 ] 

Sherry Chen commented on HADOOP-7432:
-------------------------------------

Copied from HADOOP:7110. 
Seems trunk has "%05o" but 0.20-security has "%04o".

> Back-port HADOOP-7110 to 0.20-security
> --------------------------------------
>
>                 Key: HADOOP-7432
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7432
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sherry Chen
>            Assignee: Sherry Chen
>         Attachments: HADOOP-7432.patch
>
>
> HADOOP-7110 implemented chmod in the NativeIO library so we can have good performance (ie not fork) and still not be prone to races. This should fix build failures (and probably task failures too).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16866-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 17:14:53 2011
Return-Path: <common-issues-return-16866-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F13B96C3D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 17:14:52 +0000 (UTC)
Received: (qmail 31761 invoked by uid 500); 30 Jun 2011 17:14:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31676 invoked by uid 500); 30 Jun 2011 17:14:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31648 invoked by uid 99); 30 Jun 2011 17:14:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 17:14:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 17:14:49 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7F0CC43CFBF
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 17:14:28 +0000 (UTC)
Date: Thu, 30 Jun 2011 17:14:28 +0000 (UTC)
From: "Robert Joseph Evans (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <96820296.6043.1309454068517.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <215422977.3987.1309387168470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7432) Back-port HADOOP-7110 to
 0.20-security
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057936#comment-13057936 ] 

Robert Joseph Evans commented on HADOOP-7432:
---------------------------------------------

It looks like the change was made as part of HADOOP-3953. Implement sticky bit for directories in HDFS. (Jakob Homan via szetszwo)

There were a lot of other changes too associated with that patch, could you please revert it back to "%04o".  It should not make any difference in the actual operation of the chmod command, but it will be more consistent with what is actually supported by 0.20-security.

> Back-port HADOOP-7110 to 0.20-security
> --------------------------------------
>
>                 Key: HADOOP-7432
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7432
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sherry Chen
>            Assignee: Sherry Chen
>         Attachments: HADOOP-7432.patch
>
>
> HADOOP-7110 implemented chmod in the NativeIO library so we can have good performance (ie not fork) and still not be prone to races. This should fix build failures (and probably task failures too).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16867-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 17:28:54 2011
Return-Path: <common-issues-return-16867-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7650F651D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 17:28:54 +0000 (UTC)
Received: (qmail 50537 invoked by uid 500); 30 Jun 2011 17:28:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50063 invoked by uid 500); 30 Jun 2011 17:28:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49853 invoked by uid 99); 30 Jun 2011 17:28:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 17:28:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 17:28:49 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7B11543C5F2
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 17:28:28 +0000 (UTC)
Date: Thu, 30 Jun 2011 17:28:28 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <943446590.6070.1309454908501.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7435) Make pre-commit checks run against
 the correct branch
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Make pre-commit checks run against the correct branch
-----------------------------------------------------

                 Key: HADOOP-7435
                 URL: https://issues.apache.org/jira/browse/HADOOP-7435
             Project: Hadoop Common
          Issue Type: Improvement
          Components: test
    Affects Versions: 0.23.0
            Reporter: Aaron T. Myers
             Fix For: 0.23.0


The Hudson pre-commit tests are presently only capable of testing a patch against trunk. It'd be nice if this could be extended to automatically run against the correct branch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16868-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 17:32:50 2011
Return-Path: <common-issues-return-16868-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6780762E1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 17:32:50 +0000 (UTC)
Received: (qmail 61739 invoked by uid 500); 30 Jun 2011 17:32:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61697 invoked by uid 500); 30 Jun 2011 17:32:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61689 invoked by uid 99); 30 Jun 2011 17:32:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 17:32:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 17:32:48 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 94B3F43C7AA
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 17:32:28 +0000 (UTC)
Date: Thu, 30 Jun 2011 17:32:28 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1641995401.6077.1309455148605.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <943446590.6070.1309454908501.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7435) Make pre-commit checks run against
 the correct branch
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057941#comment-13057941 ] 

Aaron T. Myers commented on HADOOP-7435:
----------------------------------------

I'm envisioning something like this:

# If there's only a single "fix version" set on the relevant JIRA, try to apply the patch/test against that branch.
# If there's multiple "fix versions" set on the relevant JIRA, we could have a convention for naming the patch to indicate which branch it should be tested against, e.g. "*-22.patch" for 0.22.
# If there's no fix version set, default to testing the patch against trunk.

> Make pre-commit checks run against the correct branch
> -----------------------------------------------------
>
>                 Key: HADOOP-7435
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7435
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>             Fix For: 0.23.0
>
>
> The Hudson pre-commit tests are presently only capable of testing a patch against trunk. It'd be nice if this could be extended to automatically run against the correct branch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16869-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 17:36:52 2011
Return-Path: <common-issues-return-16869-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BE5A66CB6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 17:36:52 +0000 (UTC)
Received: (qmail 67234 invoked by uid 500); 30 Jun 2011 17:36:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67163 invoked by uid 500); 30 Jun 2011 17:36:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67155 invoked by uid 99); 30 Jun 2011 17:36:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 17:36:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 17:36:49 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 97C5D43C96C
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 17:36:28 +0000 (UTC)
Date: Thu, 30 Jun 2011 17:36:28 +0000 (UTC)
From: "Sherry Chen (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <521219569.6085.1309455388618.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <215422977.3987.1309387168470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7432) Back-port HADOOP-7110 to
 0.20-security
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sherry Chen updated HADOOP-7432:
--------------------------------

    Attachment: HADOOP-7432_1.patch

Agree.
Line 130 of RawLocalFileSystem.java reverted back to "%04o".

> Back-port HADOOP-7110 to 0.20-security
> --------------------------------------
>
>                 Key: HADOOP-7432
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7432
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sherry Chen
>            Assignee: Sherry Chen
>         Attachments: HADOOP-7432.patch, HADOOP-7432_1.patch
>
>
> HADOOP-7110 implemented chmod in the NativeIO library so we can have good performance (ie not fork) and still not be prone to races. This should fix build failures (and probably task failures too).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16870-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 17:44:50 2011
Return-Path: <common-issues-return-16870-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 873736EB3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 17:44:50 +0000 (UTC)
Received: (qmail 75927 invoked by uid 500); 30 Jun 2011 17:44:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75866 invoked by uid 500); 30 Jun 2011 17:44:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75849 invoked by uid 99); 30 Jun 2011 17:44:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 17:44:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 17:44:48 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7B77443CE05
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 17:44:28 +0000 (UTC)
Date: Thu, 30 Jun 2011 17:44:28 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1646765376.6122.1309455868502.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <215422977.3987.1309387168470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7432) Back-port HADOOP-7110 to
 0.20-security
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057953#comment-13057953 ] 

Hadoop QA commented on HADOOP-7432:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12484801/HADOOP-7432_1.patch
  against trunk revision 1141415.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/688//console

This message is automatically generated.

> Back-port HADOOP-7110 to 0.20-security
> --------------------------------------
>
>                 Key: HADOOP-7432
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7432
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sherry Chen
>            Assignee: Sherry Chen
>         Attachments: HADOOP-7432.patch, HADOOP-7432_1.patch
>
>
> HADOOP-7110 implemented chmod in the NativeIO library so we can have good performance (ie not fork) and still not be prone to races. This should fix build failures (and probably task failures too).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16871-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 17:54:50 2011
Return-Path: <common-issues-return-16871-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7DF4666FE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 17:54:50 +0000 (UTC)
Received: (qmail 88090 invoked by uid 500); 30 Jun 2011 17:54:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87953 invoked by uid 500); 30 Jun 2011 17:54:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87940 invoked by uid 99); 30 Jun 2011 17:54:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 17:54:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 17:54:48 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9160A43C4BA
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 17:54:28 +0000 (UTC)
Date: Thu, 30 Jun 2011 17:54:28 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1995294234.6159.1309456468591.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <174265054.44671.1309203587452.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7428) IPC connection is orphaned with
 null 'out' member
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057957#comment-13057957 ] 

Eli Collins commented on HADOOP-7428:
-------------------------------------

+1  good catch.

> IPC connection is orphaned with null 'out' member
> -------------------------------------------------
>
>                 Key: HADOOP-7428
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7428
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-7428.txt
>
>
> We had a situation a JT ended up in a state where a certain user could not submit a job, due to an NPE on the following line in {{sendParam}}:
> {code}
> synchronized (Connection.this.out) {
> {code}
> Looking at the code, my guess is that an RTE was thrown in setupIOstreams, which only catches IOE. This could leave the connection in a half-setup state which is never cleaned up and also cannot perform IPCs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16872-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 17:56:59 2011
Return-Path: <common-issues-return-16872-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9FDBB6768
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 17:56:59 +0000 (UTC)
Received: (qmail 92376 invoked by uid 500); 30 Jun 2011 17:56:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92304 invoked by uid 500); 30 Jun 2011 17:56:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92296 invoked by uid 99); 30 Jun 2011 17:56:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 17:56:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 17:56:56 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7F94043C5C2
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 17:56:35 +0000 (UTC)
Date: Thu, 30 Jun 2011 17:56:35 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1444464438.6167.1309456595519.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <174265054.44671.1309203587452.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7428) IPC connection is orphaned with null
 'out' member
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7428:
--------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I've committed this. Thanks Todd!

> IPC connection is orphaned with null 'out' member
> -------------------------------------------------
>
>                 Key: HADOOP-7428
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7428
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-7428.txt
>
>
> We had a situation a JT ended up in a state where a certain user could not submit a job, due to an NPE on the following line in {{sendParam}}:
> {code}
> synchronized (Connection.this.out) {
> {code}
> Looking at the code, my guess is that an RTE was thrown in setupIOstreams, which only catches IOE. This could leave the connection in a half-setup state which is never cleaned up and also cannot perform IPCs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16873-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 18:16:50 2011
Return-Path: <common-issues-return-16873-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ABB7C67A3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 18:16:50 +0000 (UTC)
Received: (qmail 20586 invoked by uid 500); 30 Jun 2011 18:16:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20531 invoked by uid 500); 30 Jun 2011 18:16:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20515 invoked by uid 99); 30 Jun 2011 18:16:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 18:16:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 18:16:48 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AF68243C0FE
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 18:16:28 +0000 (UTC)
Date: Thu, 30 Jun 2011 18:16:28 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1107081138.6220.1309457788715.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <174265054.44671.1309203587452.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7428) IPC connection is orphaned with
 null 'out' member
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057964#comment-13057964 ] 

Hudson commented on HADOOP-7428:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #669 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/669/])
    HADOOP-7428. IPC connection is orphaned with null 'out' member. Contributed by Todd Lipcon

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1141638
Files : 
* /hadoop/common/trunk/common/CHANGES.txt
* /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/ipc/TestIPC.java
* /hadoop/common/trunk/common/src/java/org/apache/hadoop/ipc/Client.java


> IPC connection is orphaned with null 'out' member
> -------------------------------------------------
>
>                 Key: HADOOP-7428
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7428
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-7428.txt
>
>
> We had a situation a JT ended up in a state where a certain user could not submit a job, due to an NPE on the following line in {{sendParam}}:
> {code}
> synchronized (Connection.this.out) {
> {code}
> Looking at the code, my guess is that an RTE was thrown in setupIOstreams, which only catches IOE. This could leave the connection in a half-setup state which is never cleaned up and also cannot perform IPCs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16874-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 19:20:53 2011
Return-Path: <common-issues-return-16874-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 38B7060B0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 19:20:53 +0000 (UTC)
Received: (qmail 49587 invoked by uid 500); 30 Jun 2011 19:20:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49519 invoked by uid 500); 30 Jun 2011 19:20:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49505 invoked by uid 99); 30 Jun 2011 19:20:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 19:20:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 19:20:49 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7FC3143D5C3
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 19:20:28 +0000 (UTC)
Date: Thu, 30 Jun 2011 19:20:28 +0000 (UTC)
From: "Jonathan Eagles (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <585940721.6595.1309461628519.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <943446590.6070.1309454908501.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7435) Make pre-commit checks run against
 the correct branch
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13058008#comment-13058008 ] 

Jonathan Eagles commented on HADOOP-7435:
-----------------------------------------

Thanks, Aaron for this great idea. Here are some of my thoughts on this.

Having patch file naming convention would allow for some interesting use cases. Though a simple version of this patch naming scheme would be a great addition, here is a full blown implementation I was thinking about.

<ISSUE-#>-<full branch name>-v<version #>.patch

(e.g. MAPREDUCE-2569-MR-279-v2.patch)

- This naming scheme makes the patch completely stand-alone, allowing for emailing the patch
- Sanity of managing multiple patch files simultaneously in dev env
- Programmatic detection of which branch to apply
- Versioning of patches helps those who are submitting and testing out patches, and prevents hudson/jenkins from applying several versions of the same patch and instead tests only the latest version of the patch.
- JIRA attach file pre-commit detects "*.patch" and verifies patch naming convention.

Personally, I do not prefer to overload "Fix version/s" since that implies what branches the patch is already committed to, leading to confusion. Additionally, newbie hadoop developers may be able to attach files to issues, but are not allowed to edit issue fields.

> Make pre-commit checks run against the correct branch
> -----------------------------------------------------
>
>                 Key: HADOOP-7435
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7435
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>             Fix For: 0.23.0
>
>
> The Hudson pre-commit tests are presently only capable of testing a patch against trunk. It'd be nice if this could be extended to automatically run against the correct branch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16875-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 19:43:52 2011
Return-Path: <common-issues-return-16875-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 76BAD6D0A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 19:43:52 +0000 (UTC)
Received: (qmail 98780 invoked by uid 500); 30 Jun 2011 19:43:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98557 invoked by uid 500); 30 Jun 2011 19:43:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98540 invoked by uid 99); 30 Jun 2011 19:43:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 19:43:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 19:43:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6D92943C5B8
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 19:43:30 +0000 (UTC)
Date: Thu, 30 Jun 2011 19:43:30 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1554218243.6744.1309463010445.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7436) Bundle Chukwa Metrics plugin in
 Hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Bundle Chukwa Metrics plugin in Hadoop
--------------------------------------

                 Key: HADOOP-7436
                 URL: https://issues.apache.org/jira/browse/HADOOP-7436
             Project: Hadoop Common
          Issue Type: New Feature
         Environment: Java 6
            Reporter: Eric Yang
            Assignee: Eric Yang


For monitoring hadoop cluster with Chukwa, the current step is to copy chukwa-hadoop-*-client.jar and json-simple to hadoop classpath. (i.e. $HADOOP_HOME/lib or $HADOOP_PREFIX/share/hadoop/lib), and modify the hadoop-metrics.properties to use org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext for emitting metrics.  It is preferred to reduce the number of manual steps that is required to enable chukwa monitored hadoop cluster by moving the plugin code into hadoop code base.  It is similar to bundling Ganglia metrics plugin in Hadoop code base.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16876-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 19:51:52 2011
Return-Path: <common-issues-return-16876-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8B839645F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 19:51:52 +0000 (UTC)
Received: (qmail 6866 invoked by uid 500); 30 Jun 2011 19:51:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6819 invoked by uid 500); 30 Jun 2011 19:51:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6811 invoked by uid 99); 30 Jun 2011 19:51:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 19:51:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 19:51:49 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 671E043C869
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 19:51:28 +0000 (UTC)
Date: Thu, 30 Jun 2011 19:51:28 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1175756388.6751.1309463488419.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1554218243.6744.1309463010445.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7436) Bundle Chukwa Metrics plugin in
 Hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13058037#comment-13058037 ] 

Todd Lipcon commented on HADOOP-7436:
-------------------------------------

I disagree -- I think what we need is a better plugin system. Contrib like this tends to aggregate broken code that no one wants to maintain (witness hdfsproxy tests that have been failing >1 year)

> Bundle Chukwa Metrics plugin in Hadoop
> --------------------------------------
>
>                 Key: HADOOP-7436
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7436
>             Project: Hadoop Common
>          Issue Type: New Feature
>         Environment: Java 6
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>
> For monitoring hadoop cluster with Chukwa, the current step is to copy chukwa-hadoop-*-client.jar and json-simple to hadoop classpath. (i.e. $HADOOP_HOME/lib or $HADOOP_PREFIX/share/hadoop/lib), and modify the hadoop-metrics.properties to use org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext for emitting metrics.  It is preferred to reduce the number of manual steps that is required to enable chukwa monitored hadoop cluster by moving the plugin code into hadoop code base.  It is similar to bundling Ganglia metrics plugin in Hadoop code base.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16877-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 20:15:51 2011
Return-Path: <common-issues-return-16877-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 879016057
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 20:15:51 +0000 (UTC)
Received: (qmail 30575 invoked by uid 500); 30 Jun 2011 20:15:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30523 invoked by uid 500); 30 Jun 2011 20:15:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30515 invoked by uid 99); 30 Jun 2011 20:15:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 20:15:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 20:15:49 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5548B43C417
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 20:15:29 +0000 (UTC)
Date: Thu, 30 Jun 2011 20:15:29 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1709227589.6809.1309464929346.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1554218243.6744.1309463010445.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7436) Bundle Chukwa Metrics plugin in
 Hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13058051#comment-13058051 ] 

Eric Yang commented on HADOOP-7436:
-----------------------------------

Metrics plugin is like a driver software.  Linux kernel bundles compatible drivers in the source tarball.  This is no different, it is easier to certify a software driver if it is included in the kernel source with test cases.  If a broken driver is not fixed, it is easy to remove.  The power and responsibilities are in the hand of the Hadoop PMC.  If Hadoop community don't make integration easy, then it depends on third party to make the integration happen.  The distributed integration approach only drains developers from Hadoop community.  This is something I would like to avoid.


> Bundle Chukwa Metrics plugin in Hadoop
> --------------------------------------
>
>                 Key: HADOOP-7436
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7436
>             Project: Hadoop Common
>          Issue Type: New Feature
>         Environment: Java 6
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>
> For monitoring hadoop cluster with Chukwa, the current step is to copy chukwa-hadoop-*-client.jar and json-simple to hadoop classpath. (i.e. $HADOOP_HOME/lib or $HADOOP_PREFIX/share/hadoop/lib), and modify the hadoop-metrics.properties to use org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext for emitting metrics.  It is preferred to reduce the number of manual steps that is required to enable chukwa monitored hadoop cluster by moving the plugin code into hadoop code base.  It is similar to bundling Ganglia metrics plugin in Hadoop code base.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16878-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 20:19:50 2011
Return-Path: <common-issues-return-16878-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CD3FF638C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 20:19:50 +0000 (UTC)
Received: (qmail 38981 invoked by uid 500); 30 Jun 2011 20:19:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38869 invoked by uid 500); 30 Jun 2011 20:19:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38844 invoked by uid 99); 30 Jun 2011 20:19:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 20:19:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 20:19:48 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8475243C7BC
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 20:19:28 +0000 (UTC)
Date: Thu, 30 Jun 2011 20:19:28 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1197380690.6824.1309465168539.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1554218243.6744.1309463010445.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7436) Bundle Chukwa Metrics plugin in
 Hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13058054#comment-13058054 ] 

Todd Lipcon commented on HADOOP-7436:
-------------------------------------

The difference is that Linux has a sane maintainer system, where each subtree has someone who is committed to its maintenance. We don't have such a thing.

In theory, it's easy to remove broken code, but in practice, when we tried to do this with hdfxproxy, people -1ed its removal and then didn't step up to fix the tests (which are still failing).

> Bundle Chukwa Metrics plugin in Hadoop
> --------------------------------------
>
>                 Key: HADOOP-7436
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7436
>             Project: Hadoop Common
>          Issue Type: New Feature
>         Environment: Java 6
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>
> For monitoring hadoop cluster with Chukwa, the current step is to copy chukwa-hadoop-*-client.jar and json-simple to hadoop classpath. (i.e. $HADOOP_HOME/lib or $HADOOP_PREFIX/share/hadoop/lib), and modify the hadoop-metrics.properties to use org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext for emitting metrics.  It is preferred to reduce the number of manual steps that is required to enable chukwa monitored hadoop cluster by moving the plugin code into hadoop code base.  It is similar to bundling Ganglia metrics plugin in Hadoop code base.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16879-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 22:21:52 2011
Return-Path: <common-issues-return-16879-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 924B6625B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 22:21:52 +0000 (UTC)
Received: (qmail 28243 invoked by uid 500); 30 Jun 2011 22:21:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28184 invoked by uid 500); 30 Jun 2011 22:21:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28161 invoked by uid 99); 30 Jun 2011 22:21:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 22:21:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 22:21:49 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6528043D251
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 22:21:28 +0000 (UTC)
Date: Thu, 30 Jun 2011 22:21:28 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1553059972.7175.1309472488411.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1554218243.6744.1309463010445.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7436) Bundle Chukwa Metrics plugin in
 Hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13058105#comment-13058105 ] 

Eric Yang commented on HADOOP-7436:
-----------------------------------

Chukwa uses a simple mechanism, serialize metrics to JSON string, and stream JSON string to external system using Log4J socket appender.  Chukwa metrics plugin is in fact generic.  Hadoop currently has FileSink plugin to demo how to interface with Metrics 2 framework.  It would be useful to have a generic Metrics 2 plugin that can stream metrics to any monitoring system that implements Log4J socket appender server.  Hence, this should not generate unmaintained legacy code but a good interface for external monitoring systems.  If I rename this plugin as Log4J Socket Metrics plugin, would this be a reasonable plugin to have in Hadoop?

> Bundle Chukwa Metrics plugin in Hadoop
> --------------------------------------
>
>                 Key: HADOOP-7436
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7436
>             Project: Hadoop Common
>          Issue Type: New Feature
>         Environment: Java 6
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>
> For monitoring hadoop cluster with Chukwa, the current step is to copy chukwa-hadoop-*-client.jar and json-simple to hadoop classpath. (i.e. $HADOOP_HOME/lib or $HADOOP_PREFIX/share/hadoop/lib), and modify the hadoop-metrics.properties to use org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext for emitting metrics.  It is preferred to reduce the number of manual steps that is required to enable chukwa monitored hadoop cluster by moving the plugin code into hadoop code base.  It is similar to bundling Ganglia metrics plugin in Hadoop code base.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-16880-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu Jun 30 23:17:50 2011
Return-Path: <common-issues-return-16880-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 837086B2E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 30 Jun 2011 23:17:50 +0000 (UTC)
Received: (qmail 94217 invoked by uid 500); 30 Jun 2011 23:17:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94169 invoked by uid 500); 30 Jun 2011 23:17:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94155 invoked by uid 99); 30 Jun 2011 23:17:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 23:17:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jun 2011 23:17:48 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 67ACF43D227
	for <common-issues@hadoop.apache.org>; Thu, 30 Jun 2011 23:17:28 +0000 (UTC)
Date: Thu, 30 Jun 2011 23:17:28 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1642157739.7288.1309475848421.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1554218243.6744.1309463010445.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7436) Bundle Chukwa Metrics plugin in
 Hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13058126#comment-13058126 ] 

Todd Lipcon commented on HADOOP-7436:
-------------------------------------

Yea, if it's that generic, I suppose it makes sense. I had figured it was a Chukwa-specific protocol and would have to pull in chukwa as a dependency.

> Bundle Chukwa Metrics plugin in Hadoop
> --------------------------------------
>
>                 Key: HADOOP-7436
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7436
>             Project: Hadoop Common
>          Issue Type: New Feature
>         Environment: Java 6
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>
> For monitoring hadoop cluster with Chukwa, the current step is to copy chukwa-hadoop-*-client.jar and json-simple to hadoop classpath. (i.e. $HADOOP_HOME/lib or $HADOOP_PREFIX/share/hadoop/lib), and modify the hadoop-metrics.properties to use org.apache.hadoop.chukwa.inputtools.log4j.Log4JMetricsContext for emitting metrics.  It is preferred to reduce the number of manual steps that is required to enable chukwa monitored hadoop cluster by moving the plugin code into hadoop code base.  It is similar to bundling Ganglia metrics plugin in Hadoop code base.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13297-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 01 04:24:05 2011
Return-Path: <common-issues-return-13297-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83828 invoked from network); 1 Mar 2011 04:24:04 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Mar 2011 04:24:04 -0000
Received: (qmail 43397 invoked by uid 500); 1 Mar 2011 04:24:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42779 invoked by uid 500); 1 Mar 2011 04:24:02 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42642 invoked by uid 99); 1 Mar 2011 04:24:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 04:24:01 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 04:23:59 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 65547433F9
	for <common-issues@hadoop.apache.org>; Tue,  1 Mar 2011 04:23:38 +0000 (UTC)
Date: Tue, 1 Mar 2011 04:23:38 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <130492952.3925.1298953418411.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000704#comment-13000704 ] 

Hudson commented on HADOOP-6835:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk-Commit #616 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/616/])
    MAPREDUCE-1927. Unit test for HADOOP-6835 (concatenated gzip support). Contributed by Greg Roelofs.


> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>             Fix For: 0.22.0
>
>         Attachments: C6835-9.patch, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v4.trunk-hadoop-common.patch, HADOOP-6835.v4.trunk-hadoop-mapreduce.patch, HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v5.trunk-hadoop-common.patch, HADOOP-6835.v6.trunk-hadoop-common.patch, HADOOP-6835.v7.trunk-hadoop-common.patch, HADOOP-6835.v8.trunk-hadoop-common.patch, HADOOP-6835.v9.yahoo-0.20.2xx-branch.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch, grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13298-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 01 05:02:02 2011
Return-Path: <common-issues-return-13298-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4073 invoked from network); 1 Mar 2011 05:02:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Mar 2011 05:02:02 -0000
Received: (qmail 60744 invoked by uid 500); 1 Mar 2011 05:02:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60548 invoked by uid 500); 1 Mar 2011 05:01:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60539 invoked by uid 99); 1 Mar 2011 05:01:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 05:01:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 05:01:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E41E343CFD
	for <common-issues@hadoop.apache.org>; Tue,  1 Mar 2011 05:01:36 +0000 (UTC)
Date: Tue, 1 Mar 2011 05:01:36 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1680703382.4025.1298955696931.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6132081.87541295550763969.JavaMail.jira@thor>
Subject: [jira] Commented: (HADOOP-7112) Issue a warning when
 GenericOptionsParser libjars are not on local filesystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000719#comment-13000719 ] 

Hudson commented on HADOOP-7112:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #513 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/513/])
    HADOOP-7112. Issue a warning when GenericOptionsParser libjars are not on local filesystem.


> Issue a warning when GenericOptionsParser libjars are not on local filesystem
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-7112
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7112
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, filecache
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7112.patch
>
>
> In GenericOptionsParser#getLibJars() any jars that are not local filesystem paths are silently ignored. We should issue a warning for users.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13299-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 01 07:18:02 2011
Return-Path: <common-issues-return-13299-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51660 invoked from network); 1 Mar 2011 07:18:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Mar 2011 07:18:02 -0000
Received: (qmail 60627 invoked by uid 500); 1 Mar 2011 07:18:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60413 invoked by uid 500); 1 Mar 2011 07:17:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60384 invoked by uid 99); 1 Mar 2011 07:17:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 07:17:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 07:17:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0C7CC43122
	for <common-issues@hadoop.apache.org>; Tue,  1 Mar 2011 07:17:37 +0000 (UTC)
Date: Tue, 1 Mar 2011 07:17:37 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <564937449.4375.1298963857047.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <626842700.2525.1298091098484.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7151) Document need for stable hashCode()
 in WritableComparable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000758#comment-13000758 ] 

Hudson commented on HADOOP-7151:
--------------------------------

Integrated in Hadoop-Common-trunk #616 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/616/])
    HADOOP-7151. Document need for stable hashCode() in WritableComparable. Contributed by Dmitriy V. Ryaboy.


> Document need for stable hashCode() in WritableComparable
> ---------------------------------------------------------
>
>                 Key: HADOOP-7151
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7151
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Dmitriy V. Ryaboy
>            Assignee: Dmitriy V. Ryaboy
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7151.patch, HADOOP-7151.patch
>
>
> When a Writable is used as a key, HashPartitioner implicitly assumes that hashCode() will return the same value across different instances of the JVM. This is not a guaranteed behavior in Java, and Object's default hashCode() does not in fact do this, which can lead to subtle bugs. This requirement should be explicitly called out.
> In addition the sample MyWritable in the javadoc for WritableComparable does not implement hashCode() and thus has a bug. That should be fixed.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13300-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 01 07:18:04 2011
Return-Path: <common-issues-return-13300-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51811 invoked from network); 1 Mar 2011 07:18:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Mar 2011 07:18:03 -0000
Received: (qmail 60846 invoked by uid 500); 1 Mar 2011 07:18:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60438 invoked by uid 500); 1 Mar 2011 07:18:02 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60387 invoked by uid 99); 1 Mar 2011 07:17:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 07:17:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 07:17:56 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E097D4311F
	for <common-issues@hadoop.apache.org>; Tue,  1 Mar 2011 07:17:36 +0000 (UTC)
Date: Tue, 1 Mar 2011 07:17:36 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1721139740.4372.1298963856916.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6132081.87541295550763969.JavaMail.jira@thor>
Subject: [jira] Commented: (HADOOP-7112) Issue a warning when
 GenericOptionsParser libjars are not on local filesystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000759#comment-13000759 ] 

Hudson commented on HADOOP-7112:
--------------------------------

Integrated in Hadoop-Common-trunk #616 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/616/])
    HADOOP-7112. Issue a warning when GenericOptionsParser libjars are not on local filesystem.


> Issue a warning when GenericOptionsParser libjars are not on local filesystem
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-7112
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7112
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, filecache
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7112.patch
>
>
> In GenericOptionsParser#getLibJars() any jars that are not local filesystem paths are silently ignored. We should issue a warning for users.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13301-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 01 07:18:04 2011
Return-Path: <common-issues-return-13301-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51870 invoked from network); 1 Mar 2011 07:18:04 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Mar 2011 07:18:04 -0000
Received: (qmail 61074 invoked by uid 500); 1 Mar 2011 07:18:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60982 invoked by uid 500); 1 Mar 2011 07:18:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60417 invoked by uid 99); 1 Mar 2011 07:18:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 07:18:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 07:17:58 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3A37E43128
	for <common-issues@hadoop.apache.org>; Tue,  1 Mar 2011 07:17:37 +0000 (UTC)
Date: Tue, 1 Mar 2011 07:17:37 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1419027717.4381.1298963857235.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1803853433.10528.1298458838799.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7153) MapWritable violates contract of
 Map interface for equals() and hashCode()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000757#comment-13000757 ] 

Hudson commented on HADOOP-7153:
--------------------------------

Integrated in Hadoop-Common-trunk #616 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/616/])
    HADOOP-7153. MapWritable violates contract of Map interface for equals() and hashCode(). Contributed by Nicholas Telford.


> MapWritable violates contract of Map interface for equals() and hashCode()
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7153
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7153
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: 7153-mapwritable-equality-test.diff, 7153-mapwritable-equality.combined.diff, 7153-mapwritable-equality.diff
>
>
> o.a.h.io.MapWritable implements the java.util.Map interface, however it does not define an implementation of the equals() or hashCode() methods; instead the default implementations in java.lang.Object are used.
> This violates the contract of the Map interface which defines different behaviour for equals() and hashCode() than Object does. More information here: http://download.oracle.com/javase/6/docs/api/java/util/Map.html#equals(java.lang.Object)
> The practical consequence is that MapWritables containing equal entries cannot be compared properly. We were bitten by this when trying to write an MRUnit test for a Mapper that outputs MapWritables; the MRUnit driver cannot test the equality of the expected and actual MapWritable objects.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13302-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 01 17:16:04 2011
Return-Path: <common-issues-return-13302-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1088 invoked from network); 1 Mar 2011 17:16:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Mar 2011 17:16:03 -0000
Received: (qmail 69410 invoked by uid 500); 1 Mar 2011 17:16:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68658 invoked by uid 500); 1 Mar 2011 17:16:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68624 invoked by uid 99); 1 Mar 2011 17:16:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 17:16:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 17:15:58 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EEAC949F32
	for <common-issues@hadoop.apache.org>; Tue,  1 Mar 2011 17:15:36 +0000 (UTC)
Date: Tue, 1 Mar 2011 17:15:36 +0000 (UTC)
From: "hg (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2031635857.5461.1298999736974.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-5980) LD_LIBRARY_PATH not passed to tasks
 spawned off by LinuxTaskController
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000969#comment-13000969 ] 

hg commented on HADOOP-5980:
----------------------------

Hello,

I tried applying hadoop-5980-v20.patch to hadoop-0.20.2 and I got the following error:
patching file src/mapred/org/apache/hadoop/mapred/LinuxTaskController.java
Hunk #1 FAILED at 24.
Hunk #2 FAILED at 125.
2 out of 2 hunks FAILED -- saving rejects to file src/mapred/org/apache/hadoop/mapred/LinuxTaskController.java.rej

v20.patch seems to end abruptly at line 38.  What should I do?

Best Regards,
H

> LD_LIBRARY_PATH not passed to tasks spawned off by LinuxTaskController
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-5980
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5980
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sreekanth Ramakrishnan
>            Assignee: Sreekanth Ramakrishnan
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5980-1.patch, HADOOP-5980-2.patch, HADOOP-5980-3.patch, hadoop-5980-v20.patch
>
>
> Currently, task spawned off by {{LinuxTaskController}} don't get LD_LIBRARY_PATH in their environment. The tasks should get same LD_LIBRARY_PATH value as when spawned off by {{DefaultTaskController}}

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13303-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 01 22:52:29 2011
Return-Path: <common-issues-return-13303-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51276 invoked from network); 1 Mar 2011 22:52:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Mar 2011 22:52:28 -0000
Received: (qmail 97133 invoked by uid 500); 1 Mar 2011 22:52:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97075 invoked by uid 500); 1 Mar 2011 22:52:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97063 invoked by uid 99); 1 Mar 2011 22:52:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 22:52:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 22:52:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0A4424A420
	for <common-issues@hadoop.apache.org>; Tue,  1 Mar 2011 22:51:37 +0000 (UTC)
Date: Tue, 1 Mar 2011 22:51:37 +0000 (UTC)
From: "Scott Chen (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <824415802.6352.1299019897038.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7159) RPC server should log the client
 hostname when read exception happened
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

RPC server should log the client hostname when read exception happened
----------------------------------------------------------------------

                 Key: HADOOP-7159
                 URL: https://issues.apache.org/jira/browse/HADOOP-7159
             Project: Hadoop Common
          Issue Type: Improvement
          Components: ipc
            Reporter: Scott Chen
            Assignee: Scott Chen
            Priority: Trivial


This makes find mismatched clients easier

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13304-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 01 22:53:58 2011
Return-Path: <common-issues-return-13304-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51804 invoked from network); 1 Mar 2011 22:53:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Mar 2011 22:53:58 -0000
Received: (qmail 683 invoked by uid 500); 1 Mar 2011 22:53:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 631 invoked by uid 500); 1 Mar 2011 22:53:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 618 invoked by uid 99); 1 Mar 2011 22:53:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 22:53:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 22:53:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0486A4A516
	for <common-issues@hadoop.apache.org>; Tue,  1 Mar 2011 22:53:37 +0000 (UTC)
Date: Tue, 1 Mar 2011 22:53:37 +0000 (UTC)
From: "Scott Chen (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <248993550.6356.1299020017015.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <824415802.6352.1299019897038.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7159) RPC server should log the client
 hostname when read exception happened
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Scott Chen updated HADOOP-7159:
-------------------------------

    Attachment: HADOOP-7159.txt

> RPC server should log the client hostname when read exception happened
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-7159
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7159
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>            Priority: Trivial
>         Attachments: HADOOP-7159.txt
>
>
> This makes find mismatched clients easier

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13305-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 01 22:54:00 2011
Return-Path: <common-issues-return-13305-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51860 invoked from network); 1 Mar 2011 22:53:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Mar 2011 22:53:59 -0000
Received: (qmail 930 invoked by uid 500); 1 Mar 2011 22:53:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 704 invoked by uid 500); 1 Mar 2011 22:53:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 614 invoked by uid 99); 1 Mar 2011 22:53:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 22:53:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 22:53:56 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E8E644A513
	for <common-issues@hadoop.apache.org>; Tue,  1 Mar 2011 22:53:36 +0000 (UTC)
Date: Tue, 1 Mar 2011 22:53:36 +0000 (UTC)
From: "Scott Chen (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <435214794.6354.1299020016950.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <824415802.6352.1299019897038.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7159) RPC server should log the client
 hostname when read exception happened
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Scott Chen updated HADOOP-7159:
-------------------------------

    Status: Patch Available  (was: Open)

> RPC server should log the client hostname when read exception happened
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-7159
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7159
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>            Priority: Trivial
>         Attachments: HADOOP-7159.txt
>
>
> This makes find mismatched clients easier

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13306-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 02 08:40:02 2011
Return-Path: <common-issues-return-13306-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41522 invoked from network); 2 Mar 2011 08:40:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Mar 2011 08:40:02 -0000
Received: (qmail 23438 invoked by uid 500); 2 Mar 2011 08:40:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23215 invoked by uid 500); 2 Mar 2011 08:39:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23202 invoked by uid 99); 2 Mar 2011 08:39:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 08:39:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 08:39:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 066624BDC2
	for <common-issues@hadoop.apache.org>; Wed,  2 Mar 2011 08:39:37 +0000 (UTC)
Date: Wed, 2 Mar 2011 08:39:37 +0000 (UTC)
From: "Bharath Mundlapudi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <43301795.7467.1299055177022.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <29584617.273451296248323898.JavaMail.jira@thor>
Subject: [jira] Commented: (HADOOP-7123) Hadoop Disk Fail Inplace
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001334#comment-13001334 ] 

Bharath Mundlapudi commented on HADOOP-7123:
--------------------------------------------


Hi Wang Xu, 

I have some comments regarding HDFS-1362. I will provide my comments on that jira. 


> Hadoop Disk Fail Inplace
> ------------------------
>
>                 Key: HADOOP-7123
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7123
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>
> This is an umbrella Jira for tracking all the issues and work related for handling disk failures across the Hadoop stack. Today, Hadoop handles disk failures to some extent but not all the issues. This involves understanding both startup and runtime issues related to disk failures in the Hadoop components. 
> Scope: The initial scope of work will only be for DataNode and TaskTracker related disk failure issues. 
> Methodology: Injecting disk failures in SCSI layer at various runtime points of Hadoop and observing Hadoop's behavior for improvement. 

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13307-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 02 16:03:04 2011
Return-Path: <common-issues-return-13307-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76701 invoked from network); 2 Mar 2011 16:03:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Mar 2011 16:03:03 -0000
Received: (qmail 32847 invoked by uid 500); 2 Mar 2011 16:03:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32347 invoked by uid 500); 2 Mar 2011 16:02:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32307 invoked by uid 99); 2 Mar 2011 16:02:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 16:02:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 16:02:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8651D4BD31
	for <common-issues@hadoop.apache.org>; Wed,  2 Mar 2011 16:02:37 +0000 (UTC)
Date: Wed, 2 Mar 2011 16:02:37 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <169869926.8025.1299081757546.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-6541) An Interactive Hadoop FS shell
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6541:
------------------------------

    Status: Open  (was: Patch Available)

> An Interactive Hadoop FS shell
> ------------------------------
>
>                 Key: HADOOP-6541
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6541
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>         Attachments: HADOOP-6541.2.patch, HADOOP-6541.3.patch, HADOOP-6541.4.patch, HADOOP-6541.patch
>
>
> A shell that allows the user to execute multiple filesystem operations in a single JVM instance at a prompt.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13308-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 02 17:12:04 2011
Return-Path: <common-issues-return-13308-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15815 invoked from network); 2 Mar 2011 17:12:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Mar 2011 17:12:03 -0000
Received: (qmail 90291 invoked by uid 500); 2 Mar 2011 17:12:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89947 invoked by uid 500); 2 Mar 2011 17:12:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89932 invoked by uid 99); 2 Mar 2011 17:12:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 17:12:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 17:11:59 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 94F864C96C
	for <common-issues@hadoop.apache.org>; Wed,  2 Mar 2011 17:11:39 +0000 (UTC)
Date: Wed, 2 Mar 2011 17:11:39 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2067905911.8217.1299085899606.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <31897278.92301295561324063.JavaMail.jira@thor>
Subject: [jira] Updated: (HADOOP-7114) FsShell should dump all exceptions at
 DEBUG level
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-7114:
------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

+1 I've just committed this. Thanks Todd!

> FsShell should dump all exceptions at DEBUG level
> -------------------------------------------------
>
>                 Key: HADOOP-7114
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7114
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7114.txt
>
>
> Most of the FsShell commands catch exceptions and then just print out an error like "foo: " + e.getLocalizedMessage(). This is fine when the exception is "user-facing" (eg permissions errors) but in the case of a user hitting a bug you get a useless error message with no stack trace. For example, something "chmod: null" in the case of a NullPointerException bug.
> It would help debug these cases for users and developers if we also logged the exception with full trace at DEBUG level.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13309-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 02 17:14:02 2011
Return-Path: <common-issues-return-13309-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16634 invoked from network); 2 Mar 2011 17:14:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Mar 2011 17:14:02 -0000
Received: (qmail 93928 invoked by uid 500); 2 Mar 2011 17:14:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93845 invoked by uid 500); 2 Mar 2011 17:14:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93837 invoked by uid 99); 2 Mar 2011 17:14:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 17:14:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 17:13:58 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1CE714CA69
	for <common-issues@hadoop.apache.org>; Wed,  2 Mar 2011 17:13:37 +0000 (UTC)
Date: Wed, 2 Mar 2011 17:13:37 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <872956917.8227.1299086017115.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-6974) Configurable header buffer size for
 Hadoop HTTP server
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6974:
------------------------------

    Status: Open  (was: Patch Available)

Paul, unfortunately this no longer applies. Can you regenerate the patch and I'll commit it.

> Configurable header buffer size for Hadoop HTTP server
> ------------------------------------------------------
>
>                 Key: HADOOP-6974
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6974
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Paul Butler
>            Assignee: Paul Butler
>         Attachments: HADOOP-6974.3.patch, HADOOP-6974.4.patch, hadoop-6974.2.patch, hadoop-6974.patch
>
>
> This patch adds a configurable parameter dfs.http.header.buffer.size to Hadoop which allows the buffer size to be configured from the xml configuration.
> This fixes an issue that came up in an environment where the Hadoop servers share a domain with other web applications that use domain cookies. The large cookies overwhelmed Jetty's buffer which caused it to return a 413 error.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13310-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 02 17:26:01 2011
Return-Path: <common-issues-return-13310-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54034 invoked from network); 2 Mar 2011 17:26:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Mar 2011 17:26:01 -0000
Received: (qmail 33283 invoked by uid 500); 2 Mar 2011 17:26:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33011 invoked by uid 500); 2 Mar 2011 17:25:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32986 invoked by uid 99); 2 Mar 2011 17:25:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 17:25:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 17:25:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0B5644CF41
	for <common-issues@hadoop.apache.org>; Wed,  2 Mar 2011 17:25:37 +0000 (UTC)
Date: Wed, 2 Mar 2011 17:25:37 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1449256476.8252.1299086737042.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2257860.286501294766747197.JavaMail.jira@thor>
Subject: [jira] Commented: (HADOOP-7098) tasktracker property not set in
 conf/hadoop-env.sh
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001500#comment-13001500 ] 

Tom White commented on HADOOP-7098:
-----------------------------------

+1 The lack of a HADOOP_TASKTRACKER_OPTS definition looks like an oversight (these variables were introduced in HADOOP-2551). HADOOP_TASKTRACKER_OPTS is already honoured by bin/mapred, so no changes are needed there.

Bernd, have you tested this manually? There's currently no easy way to write an automated test for this change.

> tasktracker property not set in conf/hadoop-env.sh
> --------------------------------------------------
>
>                 Key: HADOOP-7098
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7098
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.20.2, 0.21.1, 0.22.0
>            Reporter: Bernd Fondermann
>            Assignee: Bernd Fondermann
>         Attachments: hadoop-7098.patch
>
>
> For all cluster components, except TaskTracker the OPTS environment variable is set like this in hadoop-env.sh:
> export HADOOP_<COMPONENT>_OPTS="-Dcom.sun.management.jmxremote $HADOOP_<COMPONENT>_OPTS"
> The provided patch fixes this.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13311-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 02 18:04:03 2011
Return-Path: <common-issues-return-13311-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85745 invoked from network); 2 Mar 2011 18:04:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Mar 2011 18:04:02 -0000
Received: (qmail 92030 invoked by uid 500); 2 Mar 2011 18:04:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91782 invoked by uid 500); 2 Mar 2011 18:04:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91765 invoked by uid 99); 2 Mar 2011 18:03:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 18:03:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 18:03:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E986C4CA0D
	for <common-issues@hadoop.apache.org>; Wed,  2 Mar 2011 18:03:36 +0000 (UTC)
Date: Wed, 2 Mar 2011 18:03:36 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2069671519.8286.1299089016952.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-6754) DefaultCodec.createOutputStream()
 leaks memory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6754:
------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
           Status: Resolved  (was: Patch Available)

I've just committed this. Thanks Aaron!

> DefaultCodec.createOutputStream() leaks memory
> ----------------------------------------------
>
>                 Key: HADOOP-6754
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6754
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>             Fix For: 0.23.0
>
>         Attachments: CompressionBug.java, HADOOP-6754.2.patch, HADOOP-6754.patch, HADOOP-6754.patch
>
>
> DefaultCodec.createOutputStream() creates a new Compressor instance in each OutputStream. Even if the OutputStream is closed, this leaks memory.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13312-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 02 18:26:58 2011
Return-Path: <common-issues-return-13312-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63779 invoked from network); 2 Mar 2011 18:26:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Mar 2011 18:26:52 -0000
Received: (qmail 28659 invoked by uid 500); 2 Mar 2011 18:26:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28063 invoked by uid 500); 2 Mar 2011 18:26:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27755 invoked by uid 99); 2 Mar 2011 18:26:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 18:26:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 18:20:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E96A94CF7C
	for <common-issues@hadoop.apache.org>; Wed,  2 Mar 2011 18:20:36 +0000 (UTC)
Date: Wed, 2 Mar 2011 18:20:36 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <347646534.8325.1299090036952.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <16141091.88171289847975276.JavaMail.jira@thor>
Subject: [jira] Updated: (HADOOP-7035) Document incompatible API changes
 between releases
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-7035:
------------------------------

    Fix Version/s: 0.22.0
         Assignee: Tom White

> Document incompatible API changes between releases
> --------------------------------------------------
>
>                 Key: HADOOP-7035
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7035
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: jdiff-with-previous-release.sh, jdiff-with-previous-release.sh
>
>
> We can use JDiff to generate a list of incompatible changes for each release. See https://issues.apache.org/jira/browse/HADOOP-6668?focusedCommentId=12860017&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12860017

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13313-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 02 20:16:49 2011
Return-Path: <common-issues-return-13313-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20192 invoked from network); 2 Mar 2011 20:16:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Mar 2011 20:16:47 -0000
Received: (qmail 22168 invoked by uid 500); 2 Mar 2011 20:16:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20853 invoked by uid 500); 2 Mar 2011 20:16:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20598 invoked by uid 99); 2 Mar 2011 20:16:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 20:16:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 19:36:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 93C1E4C063
	for <common-issues@hadoop.apache.org>; Wed,  2 Mar 2011 19:36:37 +0000 (UTC)
Date: Wed, 2 Mar 2011 19:36:37 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <55923473.8590.1299094597601.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-6342) Create a script to squash a common,
 hdfs, and mapreduce tarball into a single hadoop tarball
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6342:
------------------------------

    Status: Open  (was: Patch Available)

We can close this now as it is subsumed by HADOOP-6846 (see https://issues.apache.org/jira/browse/HADOOP-6846?focusedCommentId=13001571&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13001571).

> Create a script to squash a common, hdfs, and mapreduce tarball into a single hadoop tarball
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6342
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6342
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>            Priority: Blocker
>             Fix For: 0.21.1
>
>         Attachments: HADOOP-6342.2.patch, HADOOP-6342.patch, h-6342.patch, tar-munge, tar-munge
>
>
> It would be convenient for the transition if we had a script to take a set of common, hdfs, and mapreduce tarballs and merge them into a single tarball. This is intended just to help users who don't want to transition to split projects for deployment immediately.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13314-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 02 20:22:19 2011
Return-Path: <common-issues-return-13314-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41153 invoked from network); 2 Mar 2011 20:21:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Mar 2011 20:21:57 -0000
Received: (qmail 73214 invoked by uid 500); 2 Mar 2011 20:21:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73173 invoked by uid 500); 2 Mar 2011 20:21:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Delivered-To: moderator for common-issues@hadoop.apache.org
Received: (qmail 11383 invoked by uid 99); 2 Mar 2011 19:34:57 -0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Date: Wed, 2 Mar 2011 19:34:36 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <132961198.8577.1299094476903.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-6846) Scripts for building Hadoop 0.21.0
 release
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6846:
------------------------------

    Fix Version/s: 0.22.0

> Scripts for building Hadoop 0.21.0 release
> ------------------------------------------
>
>                 Key: HADOOP-6846
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6846
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: build
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: release-scripts.tar.gz
>
>


-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13315-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 02 20:23:00 2011
Return-Path: <common-issues-return-13315-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46988 invoked from network); 2 Mar 2011 20:23:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Mar 2011 20:23:00 -0000
Received: (qmail 88723 invoked by uid 500); 2 Mar 2011 20:22:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88693 invoked by uid 500); 2 Mar 2011 20:22:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88685 invoked by uid 99); 2 Mar 2011 20:22:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 20:22:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 20:22:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EC19A4C16F
	for <common-issues@hadoop.apache.org>; Wed,  2 Mar 2011 20:22:36 +0000 (UTC)
Date: Wed, 2 Mar 2011 20:22:36 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <276487681.8733.1299097356963.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <10697353.42141289538553594.JavaMail.jira@thor>
Subject: [jira] Updated: (HADOOP-7030) new topology mapping implementations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-7030:
------------------------------

    Assignee: Patrick Angeles
      Status: Open  (was: Patch Available)

This looks like a useful addition. Here are my comments on the patch:

* Could you combine the two types of file, so that if there are three columns the first two are interpreted as a range, otherwise use the first as a single host. Or just support CIDR notation?
* Have you thought about InetAddress to avoid implementing IP address parsing logic? http://guava-libraries.googlecode.com/svn/tags/release08/javadoc/com/google/common/net/InetAddresses.html might be useful (there was talk of introducing Guava recently).
* RefreshableDNSToSwitchMapping isn't hooked up yet, so perhaps it should go in a follow on JIRA.
* The name "TableMapping" is a bit general. How about "FileBasedMapping", or similar?
* The configuration keys should go in CommonConfigurationKeysPublic.
* Primes are not needed in hashCode implementations. For Ip4 Arrays.hashCode(value) is sufficient.
* The tests swallow exceptions - there should at least be a comment saying that this is expected. Also, fail() with a message is preferable to assertTrue(false).
* The tests should be JUnit 4 style.

> new topology mapping implementations
> ------------------------------------
>
>                 Key: HADOOP-7030
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7030
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0, 0.20.2, 0.20.1
>            Reporter: Patrick Angeles
>            Assignee: Patrick Angeles
>         Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13316-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 02 20:40:58 2011
Return-Path: <common-issues-return-13316-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17013 invoked from network); 2 Mar 2011 20:40:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Mar 2011 20:40:58 -0000
Received: (qmail 55290 invoked by uid 500); 2 Mar 2011 20:40:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55221 invoked by uid 500); 2 Mar 2011 20:40:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55092 invoked by uid 99); 2 Mar 2011 20:40:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 20:40:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 20:40:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2359E4C862
	for <common-issues@hadoop.apache.org>; Wed,  2 Mar 2011 20:40:37 +0000 (UTC)
Date: Wed, 2 Mar 2011 20:40:37 +0000 (UTC)
From: "Bernd Fondermann (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <543821882.8797.1299098437141.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2257860.286501294766747197.JavaMail.jira@thor>
Subject: [jira] Commented: (HADOOP-7098) tasktracker property not set in
 conf/hadoop-env.sh
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001612#comment-13001612 ] 

Bernd Fondermann commented on HADOOP-7098:
------------------------------------------

I did test this manually indeed. The patch is in active use in my env.

There are things like http://code.google.com/p/jbash/ which might help setting up a test, but just for this small improvement it seem like overkill.  

> tasktracker property not set in conf/hadoop-env.sh
> --------------------------------------------------
>
>                 Key: HADOOP-7098
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7098
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.20.2, 0.21.1, 0.22.0
>            Reporter: Bernd Fondermann
>            Assignee: Bernd Fondermann
>         Attachments: hadoop-7098.patch
>
>
> For all cluster components, except TaskTracker the OPTS environment variable is set like this in hadoop-env.sh:
> export HADOOP_<COMPONENT>_OPTS="-Dcom.sun.management.jmxremote $HADOOP_<COMPONENT>_OPTS"
> The provided patch fixes this.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13317-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 02 20:42:59 2011
Return-Path: <common-issues-return-13317-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17879 invoked from network); 2 Mar 2011 20:42:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Mar 2011 20:42:58 -0000
Received: (qmail 61604 invoked by uid 500); 2 Mar 2011 20:42:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61476 invoked by uid 500); 2 Mar 2011 20:42:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61458 invoked by uid 99); 2 Mar 2011 20:42:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 20:42:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 20:42:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 380A04C9F9
	for <common-issues@hadoop.apache.org>; Wed,  2 Mar 2011 20:42:37 +0000 (UTC)
Date: Wed, 2 Mar 2011 20:42:37 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1610427056.8810.1299098557226.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <10330397.93631295565704195.JavaMail.jira@thor>
Subject: [jira] Commented: (HADOOP-7115) Add a cache for getpwuid_r and
 getpwgid_r calls
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001616#comment-13001616 ] 

Todd Lipcon commented on HADOOP-7115:
-------------------------------------

Hey Devaraj. Any news on this patch for trunk? I need to fix HADOOP-7156 which is likely to conflict with this code.

> Add a cache for getpwuid_r and getpwgid_r calls
> -----------------------------------------------
>
>                 Key: HADOOP-7115
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7115
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Arun C Murthy
>            Assignee: Devaraj Das
>
> As discussed in HADOOP-6978, a cache helps a lot.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13318-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 02 21:41:01 2011
Return-Path: <common-issues-return-13318-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15077 invoked from network); 2 Mar 2011 21:41:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Mar 2011 21:41:00 -0000
Received: (qmail 44476 invoked by uid 500); 2 Mar 2011 21:40:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44415 invoked by uid 500); 2 Mar 2011 21:40:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44268 invoked by uid 99); 2 Mar 2011 21:40:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 21:40:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 21:40:58 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2040A41818
	for <common-issues@hadoop.apache.org>; Wed,  2 Mar 2011 21:40:37 +0000 (UTC)
Date: Wed, 2 Mar 2011 21:40:37 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1914981294.9078.1299102037128.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2257860.286501294766747197.JavaMail.jira@thor>
Subject: [jira] Commented: (HADOOP-7098) tasktracker property not set in
 conf/hadoop-env.sh
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001678#comment-13001678 ] 

Hadoop QA commented on HADOOP-7098:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12468028/hadoop-7098.patch
  against trunk revision 1076314.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 patch.  The patch command could not apply the patch.

Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/299//console

This message is automatically generated.

> tasktracker property not set in conf/hadoop-env.sh
> --------------------------------------------------
>
>                 Key: HADOOP-7098
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7098
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.20.2, 0.21.1, 0.22.0
>            Reporter: Bernd Fondermann
>            Assignee: Bernd Fondermann
>         Attachments: hadoop-7098.patch
>
>
> For all cluster components, except TaskTracker the OPTS environment variable is set like this in hadoop-env.sh:
> export HADOOP_<COMPONENT>_OPTS="-Dcom.sun.management.jmxremote $HADOOP_<COMPONENT>_OPTS"
> The provided patch fixes this.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13319-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 02 21:52:58 2011
Return-Path: <common-issues-return-13319-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60800 invoked from network); 2 Mar 2011 21:52:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Mar 2011 21:52:58 -0000
Received: (qmail 75711 invoked by uid 500); 2 Mar 2011 21:52:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75651 invoked by uid 500); 2 Mar 2011 21:52:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75564 invoked by uid 99); 2 Mar 2011 21:52:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 21:52:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 21:52:56 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E48C94C0BF
	for <common-issues@hadoop.apache.org>; Wed,  2 Mar 2011 21:52:36 +0000 (UTC)
Date: Wed, 2 Mar 2011 21:52:36 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <327620873.9124.1299102756932.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7156:
--------------------------------

    Attachment: hadoop-7156.txt

Here's a heinous workaround for this issue:

When initializing the native library, we check for the problematic nss module - if it's on the system, we're likely to run into this issue, so it adds a lock around the getpwuid_r calls.


Do people think this hack is the right approach? My other thought was to add a new boolean config flag like hadoop.nativeio.workaround.hadoop7156 or some nonsense like that, and if that flag is set, add the monitor lock.


> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13320-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 02 22:03:01 2011
Return-Path: <common-issues-return-13320-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86867 invoked from network); 2 Mar 2011 22:03:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Mar 2011 22:03:00 -0000
Received: (qmail 7539 invoked by uid 500); 2 Mar 2011 22:03:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7495 invoked by uid 500); 2 Mar 2011 22:03:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7487 invoked by uid 99); 2 Mar 2011 22:03:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 22:03:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 22:02:58 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 17BDD4C6AA
	for <common-issues@hadoop.apache.org>; Wed,  2 Mar 2011 22:02:37 +0000 (UTC)
Date: Wed, 2 Mar 2011 22:02:37 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1843437377.9178.1299103357093.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001702#comment-13001702 ] 

Todd Lipcon commented on HADOOP-7156:
-------------------------------------

Another possibility is to look in /etc/redhat-release and match the contents against known-buggy releases.

> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13321-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 02 22:59:58 2011
Return-Path: <common-issues-return-13321-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63204 invoked from network); 2 Mar 2011 22:59:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Mar 2011 22:59:58 -0000
Received: (qmail 70908 invoked by uid 500); 2 Mar 2011 22:59:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70880 invoked by uid 500); 2 Mar 2011 22:59:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70872 invoked by uid 99); 2 Mar 2011 22:59:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 22:59:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 22:59:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C8A4E4C08D
	for <common-issues@hadoop.apache.org>; Wed,  2 Mar 2011 22:59:37 +0000 (UTC)
Date: Wed, 2 Mar 2011 22:59:37 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <312971684.9375.1299106777818.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-6949) Reduces RPC packet size for primitive
 arrays, especially long[], which is used at block reporting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated HADOOP-6949:
-------------------------------

    Status: Open  (was: Patch Available)

> Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6949
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6949
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Navis
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: ObjectWritable.diff, arrayprim.patch, arrayprim_v4.patch, arrayprim_v5.patch
>
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n
> It would not be needed to specify each element types for primitive arrays.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13322-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 02 23:01:59 2011
Return-Path: <common-issues-return-13322-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76111 invoked from network); 2 Mar 2011 23:01:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 2 Mar 2011 23:01:58 -0000
Received: (qmail 85357 invoked by uid 500); 2 Mar 2011 23:01:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85282 invoked by uid 500); 2 Mar 2011 23:01:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85274 invoked by uid 99); 2 Mar 2011 23:01:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 23:01:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Mar 2011 23:01:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 38ACB4C199
	for <common-issues@hadoop.apache.org>; Wed,  2 Mar 2011 23:01:37 +0000 (UTC)
Date: Wed, 2 Mar 2011 23:01:37 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1648686366.9393.1299106897228.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-6949) Reduces RPC packet size for primitive
 arrays, especially long[], which is used at block reporting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated HADOOP-6949:
-------------------------------

    Status: Patch Available  (was: Open)

> Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6949
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6949
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Navis
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: ObjectWritable.diff, arrayprim.patch, arrayprim_v4.patch, arrayprim_v5.patch
>
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n
> It would not be needed to specify each element types for primitive arrays.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13323-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 03 01:55:58 2011
Return-Path: <common-issues-return-13323-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61866 invoked from network); 3 Mar 2011 01:55:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Mar 2011 01:55:57 -0000
Received: (qmail 17922 invoked by uid 500); 3 Mar 2011 01:55:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17889 invoked by uid 500); 3 Mar 2011 01:55:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17880 invoked by uid 99); 3 Mar 2011 01:55:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 01:55:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 01:55:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2F1414C6CD
	for <common-issues@hadoop.apache.org>; Thu,  3 Mar 2011 01:55:37 +0000 (UTC)
Date: Thu, 3 Mar 2011 01:55:37 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1904300350.9852.1299117337189.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001811#comment-13001811 ] 

Allen Wittenauer commented on HADOOP-6255:
------------------------------------------

Going through the doc again (I'll get to the patch later):

* I'm still a little concerned that the deployment document talks about things that are outside the scope of this JIRA.  Even worse, it mentions things that Apache cannot distribute (the LZO code).  This should really get sanitized for Hadoop.

* is include/hdfs.h really the proper location or should it be include/hadoop/hdfs.h?
 

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.100
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.100
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13324-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 03 11:14:58 2011
Return-Path: <common-issues-return-13324-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53175 invoked from network); 3 Mar 2011 11:14:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Mar 2011 11:14:58 -0000
Received: (qmail 33325 invoked by uid 500); 3 Mar 2011 11:14:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33218 invoked by uid 500); 3 Mar 2011 11:14:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33195 invoked by uid 99); 3 Mar 2011 11:14:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 11:14:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 11:14:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 17A5A4D991
	for <common-issues@hadoop.apache.org>; Thu,  3 Mar 2011 11:14:37 +0000 (UTC)
Date: Thu, 3 Mar 2011 11:14:37 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1935772995.10660.1299150877093.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-6754) DefaultCodec.createOutputStream()
 leaks memory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001958#comment-13001958 ] 

Hudson commented on HADOOP-6754:
--------------------------------

Integrated in Hadoop-Common-trunk #619 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/619/])
    HADOOP-6754. DefaultCodec.createOutputStream() leaks memory. Contributed by Aaron Kimball.


> DefaultCodec.createOutputStream() leaks memory
> ----------------------------------------------
>
>                 Key: HADOOP-6754
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6754
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>             Fix For: 0.23.0
>
>         Attachments: CompressionBug.java, HADOOP-6754.2.patch, HADOOP-6754.patch, HADOOP-6754.patch
>
>
> DefaultCodec.createOutputStream() creates a new Compressor instance in each OutputStream. Even if the OutputStream is closed, this leaks memory.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13325-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 03 11:14:59 2011
Return-Path: <common-issues-return-13325-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53188 invoked from network); 3 Mar 2011 11:14:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Mar 2011 11:14:58 -0000
Received: (qmail 33479 invoked by uid 500); 3 Mar 2011 11:14:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33227 invoked by uid 500); 3 Mar 2011 11:14:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33186 invoked by uid 99); 3 Mar 2011 11:14:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 11:14:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 11:14:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E2E654D98D
	for <common-issues@hadoop.apache.org>; Thu,  3 Mar 2011 11:14:36 +0000 (UTC)
Date: Thu, 3 Mar 2011 11:14:36 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <464061047.10656.1299150876925.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <31897278.92301295561324063.JavaMail.jira@thor>
Subject: [jira] Commented: (HADOOP-7114) FsShell should dump all exceptions
 at DEBUG level
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001959#comment-13001959 ] 

Hudson commented on HADOOP-7114:
--------------------------------

Integrated in Hadoop-Common-trunk #619 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/619/])
    HADOOP-7114. FsShell should dump all exceptions at DEBUG level. Contributed by todd.


> FsShell should dump all exceptions at DEBUG level
> -------------------------------------------------
>
>                 Key: HADOOP-7114
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7114
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7114.txt
>
>
> Most of the FsShell commands catch exceptions and then just print out an error like "foo: " + e.getLocalizedMessage(). This is fine when the exception is "user-facing" (eg permissions errors) but in the case of a user hitting a bug you get a useless error message with no stack trace. For example, something "chmod: null" in the case of a NullPointerException bug.
> It would help debug these cases for users and developers if we also logged the exception with full trace at DEBUG level.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13326-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 03 12:15:58 2011
Return-Path: <common-issues-return-13326-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47291 invoked from network); 3 Mar 2011 12:15:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Mar 2011 12:15:58 -0000
Received: (qmail 92665 invoked by uid 500); 3 Mar 2011 12:15:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92615 invoked by uid 500); 3 Mar 2011 12:15:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92607 invoked by uid 99); 3 Mar 2011 12:15:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 12:15:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 12:15:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D73244DAF3
	for <common-issues@hadoop.apache.org>; Thu,  3 Mar 2011 12:15:37 +0000 (UTC)
Date: Thu, 3 Mar 2011 12:15:37 +0000 (UTC)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1772985665.10707.1299154537877.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <10330397.93631295565704195.JavaMail.jira@thor>
Subject: [jira] Updated: (HADOOP-7115) Add a cache for getpwuid_r and
 getpwgid_r calls
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-7115:
--------------------------------

    Attachment: h-7115.1.patch

This patch is a straight port from the 20.100 security branch to trunk for the corresponding commit in the security branch. I will also raise a MR jira to address the MR side of the changes.





> Add a cache for getpwuid_r and getpwgid_r calls
> -----------------------------------------------
>
>                 Key: HADOOP-7115
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7115
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Arun C Murthy
>            Assignee: Devaraj Das
>         Attachments: h-7115.1.patch
>
>
> As discussed in HADOOP-6978, a cache helps a lot.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13327-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 03 16:17:01 2011
Return-Path: <common-issues-return-13327-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54630 invoked from network); 3 Mar 2011 16:17:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Mar 2011 16:17:00 -0000
Received: (qmail 73407 invoked by uid 500); 3 Mar 2011 16:17:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73356 invoked by uid 500); 3 Mar 2011 16:17:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73346 invoked by uid 99); 3 Mar 2011 16:17:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 16:17:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 16:16:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 048B14E553
	for <common-issues@hadoop.apache.org>; Thu,  3 Mar 2011 16:16:37 +0000 (UTC)
Date: Thu, 3 Mar 2011 16:16:37 +0000 (UTC)
From: "Patrick Angeles (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <635282839.11077.1299168997014.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <10697353.42141289538553594.JavaMail.jira@thor>
Subject: [jira] Commented: (HADOOP-7030) new topology mapping
 implementations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002039#comment-13002039 ] 

Patrick Angeles commented on HADOOP-7030:
-----------------------------------------

Hey Tom,

Thanks for the review. Here are some responses:

Could you combine the two types of file, so that if there are three columns the first two are interpreted as a range, otherwise use the first as a single host. Or just support CIDR notation?

- I'd prefer to keep them separate as the first two columns have completely different meanings when using one style (table lookup) over the other (IP-range).

Have you thought about InetAddress to avoid implementing IP address parsing logic? http://guava-libraries.googlecode.com/svn/tags/release08/javadoc/com/google/common/net/InetAddresses.html might be useful (there was talk of introducing Guava recently).

- I have not. Will look into this, although I'd rather keep this patch lightweight and not require the addition of another jar.

RefreshableDNSToSwitchMapping isn't hooked up yet, so perhaps it should go in a follow on JIRA.

- Yes, that is the intent. Those JIRAs would go into MAPREDUCE and HDFS.

The name "TableMapping" is a bit general. How about "FileBasedMapping", or similar?

- I'm willing to listen to suggestions, however I think FileBasedMapping is even more vague :)

The configuration keys should go in CommonConfigurationKeysPublic.

- Since this is a pluggable interface, I should not have to modify existing core code. That's better WRT separation of concerns and componentization. I'm willing to take your suggestion if the general consensus is that I should :)

Primes are not needed in hashCode implementations. For Ip4 Arrays.hashCode(value) is sufficient.

- Ok.

The tests swallow exceptions - there should at least be a comment saying that this is expected. Also, fail() with a message is preferable to assertTrue(false).

- Ok.

The tests should be JUnit 4 style.

- Ok.

> new topology mapping implementations
> ------------------------------------
>
>                 Key: HADOOP-7030
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7030
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.1, 0.20.2, 0.21.0
>            Reporter: Patrick Angeles
>            Assignee: Patrick Angeles
>         Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13328-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 03 16:28:01 2011
Return-Path: <common-issues-return-13328-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95593 invoked from network); 3 Mar 2011 16:28:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Mar 2011 16:28:00 -0000
Received: (qmail 6828 invoked by uid 500); 3 Mar 2011 16:28:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6793 invoked by uid 500); 3 Mar 2011 16:28:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6779 invoked by uid 99); 3 Mar 2011 16:28:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 16:28:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 16:27:58 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3F2A34EA20
	for <common-issues@hadoop.apache.org>; Thu,  3 Mar 2011 16:27:37 +0000 (UTC)
Date: Thu, 3 Mar 2011 16:27:37 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <609138159.11111.1299169657255.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002046#comment-13002046 ] 

Allen Wittenauer commented on HADOOP-7156:
------------------------------------------

Again, we're patching around a particular bug in a particular OS, just as we did in HADOOP-7115.  This trend is very disturbing.  

If we really want to go down this road, then yes, this should be compile time.  But for now, I'd rather just advertise "RHEL 6.0 is broken; don't use it" just like we do for JREs. 

> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13329-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 03 16:41:58 2011
Return-Path: <common-issues-return-13329-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44729 invoked from network); 3 Mar 2011 16:41:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Mar 2011 16:41:57 -0000
Received: (qmail 48538 invoked by uid 500); 3 Mar 2011 16:41:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48480 invoked by uid 500); 3 Mar 2011 16:41:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48472 invoked by uid 99); 3 Mar 2011 16:41:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 16:41:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 16:41:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DC0B04EF3C
	for <common-issues@hadoop.apache.org>; Thu,  3 Mar 2011 16:41:36 +0000 (UTC)
Date: Thu, 3 Mar 2011 16:41:36 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <275843995.11153.1299170496897.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <304321177.482.1298765698936.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7157) Create build hosts configurations
 to prevent degradation of patch validation and snapshot build environment
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002056#comment-13002056 ] 

Allen Wittenauer commented on HADOOP-7157:
------------------------------------------

Why are they being reinstalled and/or in need of configuration updates?  Is something being updated just to be updated or is there a purpose?

> Create build hosts configurations to prevent degradation of patch validation and snapshot build environment
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7157
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7157
>             Project: Hadoop Common
>          Issue Type: Improvement
>         Environment: Apache Hadoop build hosts
>            Reporter: Konstantin Boudnik
>
> Build hosts are being re-jumped, in need for configuration updates, etc. It is hard to maintain the same configuration across a 10+ hosts. A specialized service such as Puppet can be used to maintain build machines software and configurations at a desired level.
> Such configs should be checked into SCM system along with the of sources code.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13330-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 03 16:56:58 2011
Return-Path: <common-issues-return-13330-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98295 invoked from network); 3 Mar 2011 16:56:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Mar 2011 16:56:58 -0000
Received: (qmail 99252 invoked by uid 500); 3 Mar 2011 16:56:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99212 invoked by uid 500); 3 Mar 2011 16:56:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99204 invoked by uid 99); 3 Mar 2011 16:56:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 16:56:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 16:56:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1B2594E670
	for <common-issues@hadoop.apache.org>; Thu,  3 Mar 2011 16:56:37 +0000 (UTC)
Date: Thu, 3 Mar 2011 16:56:37 +0000 (UTC)
From: "T Meyarivan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2042659645.11198.1299171397106.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7160) Configurable initial buffersize for
 getGroupDetails()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Configurable initial buffersize for getGroupDetails()
-----------------------------------------------------

                 Key: HADOOP-7160
                 URL: https://issues.apache.org/jira/browse/HADOOP-7160
             Project: Hadoop Common
          Issue Type: Improvement
          Components: native, security
    Affects Versions: 0.22.0
            Reporter: T Meyarivan


trunk/src/native/src/org/apache/hadoop/security/getGroup.c

"""
int getGroupDetails(gid_t group, char **grpBuf) {
  struct group * grp = NULL;
  size_t currBufferSize = sysconf(_SC_GETGR_R_SIZE_MAX);
  if (currBufferSize < 1024) {
    currBufferSize = 1024;
  }
  *grpBuf = NULL; 
  char *buf = (char*)malloc(sizeof(char) * currBufferSize);

  if (!buf) {
    return ENOMEM;
  }
  int error;
  for (;;) {
    error = getgrgid_r(group, (struct group*)buf,
                       buf + sizeof(struct group),
                       currBufferSize - sizeof(struct group), &grp);
    if(error != ERANGE) {
       break;
    }
    free(buf);
    currBufferSize *= 2;
    buf = malloc(sizeof(char) * currBufferSize);
    if(!buf) {
      return ENOMEM;
    }
...
"""

For large groups, this implies at least 2 queries for the group (number of queries = math.ceil(math.log(response_size/1024, 2)))

In the case of a large cluster with central user/group databases (exposed via LDAP etc), this leads to unnecessary load on the central services. This can be alleviated to a large extent by changing the initial buffer size to a configurable parameter

--

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13331-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 03 17:02:02 2011
Return-Path: <common-issues-return-13331-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13884 invoked from network); 3 Mar 2011 17:02:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Mar 2011 17:02:01 -0000
Received: (qmail 15575 invoked by uid 500); 3 Mar 2011 17:02:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15543 invoked by uid 500); 3 Mar 2011 17:02:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15535 invoked by uid 99); 3 Mar 2011 17:02:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 17:02:01 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 17:01:58 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8D21B4E8A0
	for <common-issues@hadoop.apache.org>; Thu,  3 Mar 2011 17:01:37 +0000 (UTC)
Date: Thu, 3 Mar 2011 17:01:37 +0000 (UTC)
From: "T Meyarivan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1636729989.11220.1299171697574.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2042659645.11198.1299171397106.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7160) Configurable initial buffersize for
 getGroupDetails()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

T Meyarivan updated HADOOP-7160:
--------------------------------

    Description: 
{code:title=trunk/src/native/src/org/apache/hadoop/security/getGroup.c|borderStyle=solid}
int getGroupDetails(gid_t group, char **grpBuf) {
  struct group * grp = NULL;
  size_t currBufferSize = sysconf(_SC_GETGR_R_SIZE_MAX);
  if (currBufferSize < 1024) {
    currBufferSize = 1024;
  }
  *grpBuf = NULL; 
  char *buf = (char*)malloc(sizeof(char) * currBufferSize);

  if (!buf) {
    return ENOMEM;
  }
  int error;
  for (;;) {
    error = getgrgid_r(group, (struct group*)buf,
                       buf + sizeof(struct group),
                       currBufferSize - sizeof(struct group), &grp);
    if(error != ERANGE) {
       break;
    }
    free(buf);
    currBufferSize *= 2;
    buf = malloc(sizeof(char) * currBufferSize);
    if(!buf) {
      return ENOMEM;
    }
...
{code}

For large groups, this implies at least 2 queries for the group (number of queries = math.ceil(math.log(response_size/1024, 2)))

In the case of a large cluster with central user/group databases (exposed via LDAP etc), this leads to unnecessary load on the central services. This can be alleviated to a large extent by changing the initial buffer size to a configurable parameter

--

  was:
trunk/src/native/src/org/apache/hadoop/security/getGroup.c

"""
int getGroupDetails(gid_t group, char **grpBuf) {
  struct group * grp = NULL;
  size_t currBufferSize = sysconf(_SC_GETGR_R_SIZE_MAX);
  if (currBufferSize < 1024) {
    currBufferSize = 1024;
  }
  *grpBuf = NULL; 
  char *buf = (char*)malloc(sizeof(char) * currBufferSize);

  if (!buf) {
    return ENOMEM;
  }
  int error;
  for (;;) {
    error = getgrgid_r(group, (struct group*)buf,
                       buf + sizeof(struct group),
                       currBufferSize - sizeof(struct group), &grp);
    if(error != ERANGE) {
       break;
    }
    free(buf);
    currBufferSize *= 2;
    buf = malloc(sizeof(char) * currBufferSize);
    if(!buf) {
      return ENOMEM;
    }
...
"""

For large groups, this implies at least 2 queries for the group (number of queries = math.ceil(math.log(response_size/1024, 2)))

In the case of a large cluster with central user/group databases (exposed via LDAP etc), this leads to unnecessary load on the central services. This can be alleviated to a large extent by changing the initial buffer size to a configurable parameter

--


> Configurable initial buffersize for getGroupDetails()
> -----------------------------------------------------
>
>                 Key: HADOOP-7160
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7160
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: native, security
>    Affects Versions: 0.22.0
>            Reporter: T Meyarivan
>
> {code:title=trunk/src/native/src/org/apache/hadoop/security/getGroup.c|borderStyle=solid}
> int getGroupDetails(gid_t group, char **grpBuf) {
>   struct group * grp = NULL;
>   size_t currBufferSize = sysconf(_SC_GETGR_R_SIZE_MAX);
>   if (currBufferSize < 1024) {
>     currBufferSize = 1024;
>   }
>   *grpBuf = NULL; 
>   char *buf = (char*)malloc(sizeof(char) * currBufferSize);
>   if (!buf) {
>     return ENOMEM;
>   }
>   int error;
>   for (;;) {
>     error = getgrgid_r(group, (struct group*)buf,
>                        buf + sizeof(struct group),
>                        currBufferSize - sizeof(struct group), &grp);
>     if(error != ERANGE) {
>        break;
>     }
>     free(buf);
>     currBufferSize *= 2;
>     buf = malloc(sizeof(char) * currBufferSize);
>     if(!buf) {
>       return ENOMEM;
>     }
> ...
> {code}
> For large groups, this implies at least 2 queries for the group (number of queries = math.ceil(math.log(response_size/1024, 2)))
> In the case of a large cluster with central user/group databases (exposed via LDAP etc), this leads to unnecessary load on the central services. This can be alleviated to a large extent by changing the initial buffer size to a configurable parameter
> --

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13332-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 03 17:10:00 2011
Return-Path: <common-issues-return-13332-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53966 invoked from network); 3 Mar 2011 17:10:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Mar 2011 17:10:00 -0000
Received: (qmail 38560 invoked by uid 500); 3 Mar 2011 17:10:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38503 invoked by uid 500); 3 Mar 2011 17:10:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38492 invoked by uid 99); 3 Mar 2011 17:10:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 17:10:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 17:09:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 099A94ED14
	for <common-issues@hadoop.apache.org>; Thu,  3 Mar 2011 17:09:37 +0000 (UTC)
Date: Thu, 3 Mar 2011 17:09:37 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <900494991.11255.1299172177035.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2042659645.11198.1299171397106.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7160) Configurable initial buffersize for
 getGroupDetails()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002077#comment-13002077 ] 

Allen Wittenauer commented on HADOOP-7160:
------------------------------------------

What happens if you increase the nscd buffer size?

> Configurable initial buffersize for getGroupDetails()
> -----------------------------------------------------
>
>                 Key: HADOOP-7160
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7160
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: native, security
>    Affects Versions: 0.22.0
>            Reporter: T Meyarivan
>
> {code:title=trunk/src/native/src/org/apache/hadoop/security/getGroup.c|borderStyle=solid}
> int getGroupDetails(gid_t group, char **grpBuf) {
>   struct group * grp = NULL;
>   size_t currBufferSize = sysconf(_SC_GETGR_R_SIZE_MAX);
>   if (currBufferSize < 1024) {
>     currBufferSize = 1024;
>   }
>   *grpBuf = NULL; 
>   char *buf = (char*)malloc(sizeof(char) * currBufferSize);
>   if (!buf) {
>     return ENOMEM;
>   }
>   int error;
>   for (;;) {
>     error = getgrgid_r(group, (struct group*)buf,
>                        buf + sizeof(struct group),
>                        currBufferSize - sizeof(struct group), &grp);
>     if(error != ERANGE) {
>        break;
>     }
>     free(buf);
>     currBufferSize *= 2;
>     buf = malloc(sizeof(char) * currBufferSize);
>     if(!buf) {
>       return ENOMEM;
>     }
> ...
> {code}
> For large groups, this implies at least 2 queries for the group (number of queries = math.ceil(math.log(response_size/1024, 2)))
> In the case of a large cluster with central user/group databases (exposed via LDAP etc), this leads to unnecessary load on the central services. This can be alleviated to a large extent by changing the initial buffer size to a configurable parameter
> --

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13333-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 03 17:23:00 2011
Return-Path: <common-issues-return-13333-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82978 invoked from network); 3 Mar 2011 17:23:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Mar 2011 17:23:00 -0000
Received: (qmail 80858 invoked by uid 500); 3 Mar 2011 17:23:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80815 invoked by uid 500); 3 Mar 2011 17:23:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80798 invoked by uid 99); 3 Mar 2011 17:22:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 17:22:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 17:22:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DA55E4E42A
	for <common-issues@hadoop.apache.org>; Thu,  3 Mar 2011 17:22:36 +0000 (UTC)
Date: Thu, 3 Mar 2011 17:22:36 +0000 (UTC)
From: "T Meyarivan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1972960463.11311.1299172956890.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2042659645.11198.1299171397106.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7160) Configurable initial buffersize for
 getGroupDetails()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002085#comment-13002085 ] 

T Meyarivan commented on HADOOP-7160:
-------------------------------------

If the query is successful, nscd caches it for time X - once nscd caches the query, there is little or no (extra) load on remote servers till it expires

--

> Configurable initial buffersize for getGroupDetails()
> -----------------------------------------------------
>
>                 Key: HADOOP-7160
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7160
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: native, security
>    Affects Versions: 0.22.0
>            Reporter: T Meyarivan
>
> {code:title=trunk/src/native/src/org/apache/hadoop/security/getGroup.c|borderStyle=solid}
> int getGroupDetails(gid_t group, char **grpBuf) {
>   struct group * grp = NULL;
>   size_t currBufferSize = sysconf(_SC_GETGR_R_SIZE_MAX);
>   if (currBufferSize < 1024) {
>     currBufferSize = 1024;
>   }
>   *grpBuf = NULL; 
>   char *buf = (char*)malloc(sizeof(char) * currBufferSize);
>   if (!buf) {
>     return ENOMEM;
>   }
>   int error;
>   for (;;) {
>     error = getgrgid_r(group, (struct group*)buf,
>                        buf + sizeof(struct group),
>                        currBufferSize - sizeof(struct group), &grp);
>     if(error != ERANGE) {
>        break;
>     }
>     free(buf);
>     currBufferSize *= 2;
>     buf = malloc(sizeof(char) * currBufferSize);
>     if(!buf) {
>       return ENOMEM;
>     }
> ...
> {code}
> For large groups, this implies at least 2 queries for the group (number of queries = math.ceil(math.log(response_size/1024, 2)))
> In the case of a large cluster with central user/group databases (exposed via LDAP etc), this leads to unnecessary load on the central services. This can be alleviated to a large extent by changing the initial buffer size to a configurable parameter
> --

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13334-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 03 19:20:58 2011
Return-Path: <common-issues-return-13334-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76700 invoked from network); 3 Mar 2011 19:20:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Mar 2011 19:20:58 -0000
Received: (qmail 66542 invoked by uid 500); 3 Mar 2011 19:20:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66502 invoked by uid 500); 3 Mar 2011 19:20:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66462 invoked by uid 99); 3 Mar 2011 19:20:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 19:20:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 19:20:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 271B64E490
	for <common-issues@hadoop.apache.org>; Thu,  3 Mar 2011 19:20:37 +0000 (UTC)
Date: Thu, 3 Mar 2011 19:20:37 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <197974725.11635.1299180037156.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7161) Remove unnecessary oro dependency
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Remove unnecessary oro dependency
---------------------------------

                 Key: HADOOP-7161
                 URL: https://issues.apache.org/jira/browse/HADOOP-7161
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.23.0
            Reporter: Todd Lipcon
            Priority: Minor


Best I can tell we never use the "oro" dependency, but it's been in ivy.xml forever. Does anyone know any reason we might need it?

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13335-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 03 19:28:58 2011
Return-Path: <common-issues-return-13335-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6634 invoked from network); 3 Mar 2011 19:28:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Mar 2011 19:28:57 -0000
Received: (qmail 98441 invoked by uid 500); 3 Mar 2011 19:28:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98396 invoked by uid 500); 3 Mar 2011 19:28:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98388 invoked by uid 99); 3 Mar 2011 19:28:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 19:28:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 19:28:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0548B4EA1A
	for <common-issues@hadoop.apache.org>; Thu,  3 Mar 2011 19:28:37 +0000 (UTC)
Date: Thu, 3 Mar 2011 19:28:37 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <516483867.11672.1299180517017.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <197974725.11635.1299180037156.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7161) Remove unnecessary oro dependency
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002157#comment-13002157 ] 

Luke Lu commented on HADOOP-7161:
---------------------------------

oro is a transitive dependency from commons-net. Feel free to remove it from ivy.xml though, as it'll get pulled in anyway :)

> Remove unnecessary oro dependency
> ---------------------------------
>
>                 Key: HADOOP-7161
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7161
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Priority: Minor
>
> Best I can tell we never use the "oro" dependency, but it's been in ivy.xml forever. Does anyone know any reason we might need it?

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13336-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 03 20:08:01 2011
Return-Path: <common-issues-return-13336-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60034 invoked from network); 3 Mar 2011 20:08:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Mar 2011 20:08:00 -0000
Received: (qmail 13916 invoked by uid 500); 3 Mar 2011 20:08:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13883 invoked by uid 500); 3 Mar 2011 20:08:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13837 invoked by uid 99); 3 Mar 2011 20:08:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 20:08:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 20:07:58 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 79B1D4EB57
	for <common-issues@hadoop.apache.org>; Thu,  3 Mar 2011 20:07:37 +0000 (UTC)
Date: Thu, 3 Mar 2011 20:07:37 +0000 (UTC)
From: "T Meyarivan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1426823441.11827.1299182857494.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2042659645.11198.1299171397106.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7160) Configurable initial buffersize for
 getGroupDetails()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002177#comment-13002177 ] 

T Meyarivan commented on HADOOP-7160:
-------------------------------------

nscd doesn't cache the results until the query "succeeds" => it takes N queries (the result is discarded N-1 times) 

Cold cache + large job is likely to trigger a flood of queries

--

> Configurable initial buffersize for getGroupDetails()
> -----------------------------------------------------
>
>                 Key: HADOOP-7160
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7160
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: native, security
>    Affects Versions: 0.22.0
>            Reporter: T Meyarivan
>
> {code:title=trunk/src/native/src/org/apache/hadoop/security/getGroup.c|borderStyle=solid}
> int getGroupDetails(gid_t group, char **grpBuf) {
>   struct group * grp = NULL;
>   size_t currBufferSize = sysconf(_SC_GETGR_R_SIZE_MAX);
>   if (currBufferSize < 1024) {
>     currBufferSize = 1024;
>   }
>   *grpBuf = NULL; 
>   char *buf = (char*)malloc(sizeof(char) * currBufferSize);
>   if (!buf) {
>     return ENOMEM;
>   }
>   int error;
>   for (;;) {
>     error = getgrgid_r(group, (struct group*)buf,
>                        buf + sizeof(struct group),
>                        currBufferSize - sizeof(struct group), &grp);
>     if(error != ERANGE) {
>        break;
>     }
>     free(buf);
>     currBufferSize *= 2;
>     buf = malloc(sizeof(char) * currBufferSize);
>     if(!buf) {
>       return ENOMEM;
>     }
> ...
> {code}
> For large groups, this implies at least 2 queries for the group (number of queries = math.ceil(math.log(response_size/1024, 2)))
> In the case of a large cluster with central user/group databases (exposed via LDAP etc), this leads to unnecessary load on the central services. This can be alleviated to a large extent by changing the initial buffer size to a configurable parameter
> --

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13337-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 03 20:23:57 2011
Return-Path: <common-issues-return-13337-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95034 invoked from network); 3 Mar 2011 20:23:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Mar 2011 20:23:57 -0000
Received: (qmail 64216 invoked by uid 500); 3 Mar 2011 20:23:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64176 invoked by uid 500); 3 Mar 2011 20:23:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64168 invoked by uid 99); 3 Mar 2011 20:23:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 20:23:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 20:23:56 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E1C6B4E24A
	for <common-issues@hadoop.apache.org>; Thu,  3 Mar 2011 20:23:36 +0000 (UTC)
Date: Thu, 3 Mar 2011 20:23:36 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <103050926.11866.1299183816920.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2042659645.11198.1299171397106.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7160) Configurable initial buffersize for
 getGroupDetails()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002186#comment-13002186 ] 

Allen Wittenauer commented on HADOOP-7160:
------------------------------------------

Even with tuning of negative-time-to-live, positive-time-to-live, etc?

> Configurable initial buffersize for getGroupDetails()
> -----------------------------------------------------
>
>                 Key: HADOOP-7160
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7160
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: native, security
>    Affects Versions: 0.22.0
>            Reporter: T Meyarivan
>
> {code:title=trunk/src/native/src/org/apache/hadoop/security/getGroup.c|borderStyle=solid}
> int getGroupDetails(gid_t group, char **grpBuf) {
>   struct group * grp = NULL;
>   size_t currBufferSize = sysconf(_SC_GETGR_R_SIZE_MAX);
>   if (currBufferSize < 1024) {
>     currBufferSize = 1024;
>   }
>   *grpBuf = NULL; 
>   char *buf = (char*)malloc(sizeof(char) * currBufferSize);
>   if (!buf) {
>     return ENOMEM;
>   }
>   int error;
>   for (;;) {
>     error = getgrgid_r(group, (struct group*)buf,
>                        buf + sizeof(struct group),
>                        currBufferSize - sizeof(struct group), &grp);
>     if(error != ERANGE) {
>        break;
>     }
>     free(buf);
>     currBufferSize *= 2;
>     buf = malloc(sizeof(char) * currBufferSize);
>     if(!buf) {
>       return ENOMEM;
>     }
> ...
> {code}
> For large groups, this implies at least 2 queries for the group (number of queries = math.ceil(math.log(response_size/1024, 2)))
> In the case of a large cluster with central user/group databases (exposed via LDAP etc), this leads to unnecessary load on the central services. This can be alleviated to a large extent by changing the initial buffer size to a configurable parameter
> --

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13338-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 03 21:39:00 2011
Return-Path: <common-issues-return-13338-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59766 invoked from network); 3 Mar 2011 21:39:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Mar 2011 21:39:00 -0000
Received: (qmail 87554 invoked by uid 500); 3 Mar 2011 21:39:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87511 invoked by uid 500); 3 Mar 2011 21:39:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87420 invoked by uid 99); 3 Mar 2011 21:39:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 21:39:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 21:38:58 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 606234E458
	for <common-issues@hadoop.apache.org>; Thu,  3 Mar 2011 21:38:37 +0000 (UTC)
Date: Thu, 3 Mar 2011 21:38:37 +0000 (UTC)
From: "Patrick Angeles (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <943526599.12099.1299188317389.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <10697353.42141289538553594.JavaMail.jira@thor>
Subject: [jira] Commented: (HADOOP-7030) new topology mapping
 implementations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002252#comment-13002252 ] 

Patrick Angeles commented on HADOOP-7030:
-----------------------------------------

- Could you combine the two types of file, so that if there are three columns the first two are interpreted as a range, otherwise use the first as a single host. Or just support CIDR notation?
- I'd prefer to keep them separate as the first two columns have completely different meanings when using one style (table lookup) over the other (IP-range).

BTW, I don't think CIDR is appropriate here. For the table-based mapping, you can get either hosts or IPs, possibly depending on who (JT or NN) is requesting the rack ID. The docs are unclear here.

In either case, servers within a rack rarely fall in units that are powers of two.




> new topology mapping implementations
> ------------------------------------
>
>                 Key: HADOOP-7030
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7030
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.1, 0.20.2, 0.21.0
>            Reporter: Patrick Angeles
>            Assignee: Patrick Angeles
>         Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13339-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 03 21:45:00 2011
Return-Path: <common-issues-return-13339-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61880 invoked from network); 3 Mar 2011 21:45:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Mar 2011 21:45:00 -0000
Received: (qmail 3115 invoked by uid 500); 3 Mar 2011 21:45:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3064 invoked by uid 500); 3 Mar 2011 21:45:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3056 invoked by uid 99); 3 Mar 2011 21:45:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 21:45:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 21:44:58 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EB03A4E779
	for <common-issues@hadoop.apache.org>; Thu,  3 Mar 2011 21:44:36 +0000 (UTC)
Date: Thu, 3 Mar 2011 21:44:36 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1541538317.12153.1299188676959.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2042659645.11198.1299171397106.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7160) Configurable initial buffersize for
 getGroupDetails()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002257#comment-13002257 ] 

Todd Lipcon commented on HADOOP-7160:
-------------------------------------

so the sysconf result is returning a too-small value as well, I guess?

Rather than making this configurable, it seems we could bump it to something like 64KB - I can't imagine the extra memory usage would harm anyone. How big a buffer do you need for your groups?

> Configurable initial buffersize for getGroupDetails()
> -----------------------------------------------------
>
>                 Key: HADOOP-7160
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7160
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: native, security
>    Affects Versions: 0.22.0
>            Reporter: T Meyarivan
>
> {code:title=trunk/src/native/src/org/apache/hadoop/security/getGroup.c|borderStyle=solid}
> int getGroupDetails(gid_t group, char **grpBuf) {
>   struct group * grp = NULL;
>   size_t currBufferSize = sysconf(_SC_GETGR_R_SIZE_MAX);
>   if (currBufferSize < 1024) {
>     currBufferSize = 1024;
>   }
>   *grpBuf = NULL; 
>   char *buf = (char*)malloc(sizeof(char) * currBufferSize);
>   if (!buf) {
>     return ENOMEM;
>   }
>   int error;
>   for (;;) {
>     error = getgrgid_r(group, (struct group*)buf,
>                        buf + sizeof(struct group),
>                        currBufferSize - sizeof(struct group), &grp);
>     if(error != ERANGE) {
>        break;
>     }
>     free(buf);
>     currBufferSize *= 2;
>     buf = malloc(sizeof(char) * currBufferSize);
>     if(!buf) {
>       return ENOMEM;
>     }
> ...
> {code}
> For large groups, this implies at least 2 queries for the group (number of queries = math.ceil(math.log(response_size/1024, 2)))
> In the case of a large cluster with central user/group databases (exposed via LDAP etc), this leads to unnecessary load on the central services. This can be alleviated to a large extent by changing the initial buffer size to a configurable parameter
> --

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13340-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 03 23:13:01 2011
Return-Path: <common-issues-return-13340-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58490 invoked from network); 3 Mar 2011 23:13:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Mar 2011 23:13:00 -0000
Received: (qmail 41739 invoked by uid 500); 3 Mar 2011 23:13:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41707 invoked by uid 500); 3 Mar 2011 23:13:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41699 invoked by uid 99); 3 Mar 2011 23:13:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 23:13:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Mar 2011 23:12:58 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AC5344E953
	for <common-issues@hadoop.apache.org>; Thu,  3 Mar 2011 23:12:37 +0000 (UTC)
Date: Thu, 3 Mar 2011 23:12:36 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <388132984.12381.1299193956929.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <24749852.181491295940943442.JavaMail.jira@thor>
Subject: [jira] Commented: (HADOOP-7116) raise contrib junit test jvm memory
 size to 512mb
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002308#comment-13002308 ] 

Tom White commented on HADOOP-7116:
-----------------------------------

Looks like this was committed to the 0.20 branch. Can it be closed?

> raise contrib junit test jvm memory size to 512mb
> -------------------------------------------------
>
>                 Key: HADOOP-7116
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7116
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.20.2
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.3
>
>         Attachments: h-7116.patch
>
>
> The streaming tests are failing with out of memory. Raise the memory limit to 512mb.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13341-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 05:08:01 2011
Return-Path: <common-issues-return-13341-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50454 invoked from network); 4 Mar 2011 05:08:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 05:08:01 -0000
Received: (qmail 97332 invoked by uid 500); 4 Mar 2011 05:08:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97287 invoked by uid 500); 4 Mar 2011 05:08:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97261 invoked by uid 99); 4 Mar 2011 05:07:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 05:07:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 05:07:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DDE374FC96
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 05:07:36 +0000 (UTC)
Date: Fri, 4 Mar 2011 05:07:36 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <479929516.13031.1299215256903.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-6370) Contrib project ivy dependencies are
 not included in binary target
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6370:
------------------------------

    Status: Open  (was: Patch Available)

Unfortunately this has fallen out of date. Aaron, would you like to regenerate it? Thanks.

> Contrib project ivy dependencies are not included in binary target
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6370
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6370
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>            Priority: Critical
>         Attachments: HADOOP-6370.2.patch, HADOOP-6370.patch
>
>
> Only Hadoop's own library dependencies are promoted to ${build.dir}/lib; any libraries required by contribs are not redistributed.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13342-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 05:12:01 2011
Return-Path: <common-issues-return-13342-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51148 invoked from network); 4 Mar 2011 05:12:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 05:12:00 -0000
Received: (qmail 8937 invoked by uid 500); 4 Mar 2011 05:12:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8902 invoked by uid 500); 4 Mar 2011 05:12:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8882 invoked by uid 99); 4 Mar 2011 05:12:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 05:12:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 05:11:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D9FEB4FD7F
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 05:11:36 +0000 (UTC)
Date: Fri, 4 Mar 2011 05:11:36 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <541221143.13041.1299215496889.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1213190269.14005.1297638718045.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7139) Allow appending to existing
 SequenceFiles
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-7139:
------------------------------

    Assignee: Mac Yang
      Status: Open  (was: Patch Available)

> Allow appending to existing SequenceFiles
> -----------------------------------------
>
>                 Key: HADOOP-7139
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7139
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.1
>            Reporter: Stephen Rose
>            Assignee: Mac Yang
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7139.patch, HADOOP-7139.patch, HADOOP-7139.patch, HADOOP-7139.patch
>
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>


-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13343-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 05:16:04 2011
Return-Path: <common-issues-return-13343-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54275 invoked from network); 4 Mar 2011 05:16:04 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 05:16:04 -0000
Received: (qmail 25962 invoked by uid 500); 4 Mar 2011 05:16:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25921 invoked by uid 500); 4 Mar 2011 05:16:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25808 invoked by uid 99); 4 Mar 2011 05:16:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 05:16:03 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 05:16:01 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4F0B24FE27
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 05:15:40 +0000 (UTC)
Date: Fri, 4 Mar 2011 05:15:40 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <671515667.13054.1299215740320.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1213190269.14005.1297638718045.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Assigned: (HADOOP-7139) Allow appending to existing
 SequenceFiles
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White reassigned HADOOP-7139:
---------------------------------

    Assignee: Stephen Rose  (was: Mac Yang)

Sorry, I made a mistake assigning this a moment ago when marking it as open (while Todd's feedback is addressed).

> Allow appending to existing SequenceFiles
> -----------------------------------------
>
>                 Key: HADOOP-7139
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7139
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.1
>            Reporter: Stephen Rose
>            Assignee: Stephen Rose
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7139.patch, HADOOP-7139.patch, HADOOP-7139.patch, HADOOP-7139.patch
>
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>


-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13344-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 05:22:02 2011
Return-Path: <common-issues-return-13344-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77301 invoked from network); 4 Mar 2011 05:22:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 05:22:01 -0000
Received: (qmail 40132 invoked by uid 500); 4 Mar 2011 05:22:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40045 invoked by uid 500); 4 Mar 2011 05:22:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39981 invoked by uid 99); 4 Mar 2011 05:22:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 05:22:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 05:21:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E11A14FF26
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 05:21:36 +0000 (UTC)
Date: Fri, 4 Mar 2011 05:21:36 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2068244960.13061.1299216096918.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2257860.286501294766747197.JavaMail.jira@thor>
Subject: [jira] Updated: (HADOOP-7098) tasktracker property not set in
 conf/hadoop-env.sh
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-7098:
------------------------------

    Attachment: HADOOP-7098.patch

Good to know.

Regenerating patch from top level so Hudson can check it.

> tasktracker property not set in conf/hadoop-env.sh
> --------------------------------------------------
>
>                 Key: HADOOP-7098
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7098
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.20.2, 0.21.1, 0.22.0
>            Reporter: Bernd Fondermann
>            Assignee: Bernd Fondermann
>         Attachments: HADOOP-7098.patch, hadoop-7098.patch
>
>
> For all cluster components, except TaskTracker the OPTS environment variable is set like this in hadoop-env.sh:
> export HADOOP_<COMPONENT>_OPTS="-Dcom.sun.management.jmxremote $HADOOP_<COMPONENT>_OPTS"
> The provided patch fixes this.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13345-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 05:34:00 2011
Return-Path: <common-issues-return-13345-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25201 invoked from network); 4 Mar 2011 05:34:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 05:34:00 -0000
Received: (qmail 79754 invoked by uid 500); 4 Mar 2011 05:34:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79723 invoked by uid 500); 4 Mar 2011 05:34:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79703 invoked by uid 99); 4 Mar 2011 05:34:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 05:34:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 05:33:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E0C844E205
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 05:33:36 +0000 (UTC)
Date: Fri, 4 Mar 2011 05:33:36 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <262706744.13075.1299216816917.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2257860.286501294766747197.JavaMail.jira@thor>
Subject: [jira] Commented: (HADOOP-7098) tasktracker property not set in
 conf/hadoop-env.sh
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002457#comment-13002457 ] 

Hadoop QA commented on HADOOP-7098:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12472646/HADOOP-7098.patch
  against trunk revision 1076314.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/300//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/300//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/300//console

This message is automatically generated.

> tasktracker property not set in conf/hadoop-env.sh
> --------------------------------------------------
>
>                 Key: HADOOP-7098
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7098
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.20.2, 0.21.1, 0.22.0
>            Reporter: Bernd Fondermann
>            Assignee: Bernd Fondermann
>         Attachments: HADOOP-7098.patch, hadoop-7098.patch
>
>
> For all cluster components, except TaskTracker the OPTS environment variable is set like this in hadoop-env.sh:
> export HADOOP_<COMPONENT>_OPTS="-Dcom.sun.management.jmxremote $HADOOP_<COMPONENT>_OPTS"
> The provided patch fixes this.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13346-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 05:37:59 2011
Return-Path: <common-issues-return-13346-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51208 invoked from network); 4 Mar 2011 05:37:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 05:37:59 -0000
Received: (qmail 89034 invoked by uid 500); 4 Mar 2011 05:37:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88990 invoked by uid 500); 4 Mar 2011 05:37:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88682 invoked by uid 99); 4 Mar 2011 05:37:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 05:37:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 05:37:56 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E70944E2D5
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 05:37:36 +0000 (UTC)
Date: Fri, 4 Mar 2011 05:37:36 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <53128340.13081.1299217056942.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2257860.286501294766747197.JavaMail.jira@thor>
Subject: [jira] Updated: (HADOOP-7098) tasktracker property not set in
 conf/hadoop-env.sh
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-7098:
------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I've just committed this. Thanks Bernd!

> tasktracker property not set in conf/hadoop-env.sh
> --------------------------------------------------
>
>                 Key: HADOOP-7098
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7098
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.20.2, 0.21.1, 0.22.0
>            Reporter: Bernd Fondermann
>            Assignee: Bernd Fondermann
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7098.patch, hadoop-7098.patch
>
>
> For all cluster components, except TaskTracker the OPTS environment variable is set like this in hadoop-env.sh:
> export HADOOP_<COMPONENT>_OPTS="-Dcom.sun.management.jmxremote $HADOOP_<COMPONENT>_OPTS"
> The provided patch fixes this.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13347-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 05:52:58 2011
Return-Path: <common-issues-return-13347-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99874 invoked from network); 4 Mar 2011 05:52:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 05:52:58 -0000
Received: (qmail 30225 invoked by uid 500); 4 Mar 2011 05:52:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30200 invoked by uid 500); 4 Mar 2011 05:52:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30147 invoked by uid 99); 4 Mar 2011 05:52:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 05:52:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 05:52:56 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E45F94E691
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 05:52:36 +0000 (UTC)
Date: Fri, 4 Mar 2011 05:52:36 +0000 (UTC)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1604211006.13103.1299217956932.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <304321177.482.1298765698936.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7157) Create build hosts configurations
 to prevent degradation of patch validation and snapshot build environment
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002462#comment-13002462 ] 

Konstantin Boudnik commented on HADOOP-7157:
--------------------------------------------

That I don't know, Allen. It has been pointed out that recently some machines were re-flashed (again, I have no idea why). During this op. patch packages got missed which cases builds failing for some days. Thus this JIRA.

> Create build hosts configurations to prevent degradation of patch validation and snapshot build environment
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7157
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7157
>             Project: Hadoop Common
>          Issue Type: Improvement
>         Environment: Apache Hadoop build hosts
>            Reporter: Konstantin Boudnik
>
> Build hosts are being re-jumped, in need for configuration updates, etc. It is hard to maintain the same configuration across a 10+ hosts. A specialized service such as Puppet can be used to maintain build machines software and configurations at a desired level.
> Such configs should be checked into SCM system along with the of sources code.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13348-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 09:29:01 2011
Return-Path: <common-issues-return-13348-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81766 invoked from network); 4 Mar 2011 09:29:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 09:29:01 -0000
Received: (qmail 45664 invoked by uid 500); 4 Mar 2011 09:29:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45590 invoked by uid 500); 4 Mar 2011 09:29:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45543 invoked by uid 99); 4 Mar 2011 09:29:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 09:29:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 09:28:58 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5EE194FEB9
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 09:28:37 +0000 (UTC)
Date: Fri, 4 Mar 2011 09:28:37 +0000 (UTC)
From: "Alexey Diomin (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <978952244.13379.1299230917384.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7162) call method without catching
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

call method without catching
----------------------------

                 Key: HADOOP-7162
                 URL: https://issues.apache.org/jira/browse/HADOOP-7162
             Project: Hadoop Common
          Issue Type: Bug
          Components: util
    Affects Versions: 0.21.0
            Reporter: Alexey Diomin
            Priority: Minor


in file ./src/java/org/apache/hadoop/fs/FsShell.java line 555
call method twice:
1. for init variable
2. for getting data



-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13349-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 09:30:58 2011
Return-Path: <common-issues-return-13349-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87219 invoked from network); 4 Mar 2011 09:30:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 09:30:57 -0000
Received: (qmail 51108 invoked by uid 500); 4 Mar 2011 09:30:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51081 invoked by uid 500); 4 Mar 2011 09:30:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51073 invoked by uid 99); 4 Mar 2011 09:30:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 09:30:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 09:30:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F161D4FFA2
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 09:30:36 +0000 (UTC)
Date: Fri, 4 Mar 2011 09:30:36 +0000 (UTC)
From: "Alexey Diomin (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1887282360.13383.1299231036985.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <978952244.13379.1299230917384.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7162) call method without catching
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexey Diomin updated HADOOP-7162:
----------------------------------

    Attachment: HDFS-7162.path

> call method without catching
> ----------------------------
>
>                 Key: HADOOP-7162
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7162
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Alexey Diomin
>            Priority: Minor
>         Attachments: HDFS-7162.path
>
>
> in file ./src/java/org/apache/hadoop/fs/FsShell.java line 555
> call method twice:
> 1. for init variable
> 2. for getting data

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13350-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 09:30:58 2011
Return-Path: <common-issues-return-13350-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87257 invoked from network); 4 Mar 2011 09:30:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 09:30:57 -0000
Received: (qmail 51341 invoked by uid 500); 4 Mar 2011 09:30:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51293 invoked by uid 500); 4 Mar 2011 09:30:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51144 invoked by uid 99); 4 Mar 2011 09:30:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 09:30:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 09:30:56 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DFE3B4FF9D
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 09:30:36 +0000 (UTC)
Date: Fri, 4 Mar 2011 09:30:36 +0000 (UTC)
From: "Alexey Diomin (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <5109541.13381.1299231036913.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <978952244.13379.1299230917384.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7162) call method without catching
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexey Diomin updated HADOOP-7162:
----------------------------------

    Status: Patch Available  (was: Open)

> call method without catching
> ----------------------------
>
>                 Key: HADOOP-7162
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7162
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Alexey Diomin
>            Priority: Minor
>         Attachments: HDFS-7162.path
>
>
> in file ./src/java/org/apache/hadoop/fs/FsShell.java line 555
> call method twice:
> 1. for init variable
> 2. for getting data

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13351-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 11:16:00 2011
Return-Path: <common-issues-return-13351-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41127 invoked from network); 4 Mar 2011 11:16:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 11:16:00 -0000
Received: (qmail 44385 invoked by uid 500); 4 Mar 2011 11:16:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44326 invoked by uid 500); 4 Mar 2011 11:16:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44318 invoked by uid 99); 4 Mar 2011 11:16:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 11:16:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 11:15:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DA5B24F70D
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 11:15:36 +0000 (UTC)
Date: Fri, 4 Mar 2011 11:15:36 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <439760539.13611.1299237336891.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2257860.286501294766747197.JavaMail.jira@thor>
Subject: [jira] Commented: (HADOOP-7098) tasktracker property not set in
 conf/hadoop-env.sh
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002547#comment-13002547 ] 

Hudson commented on HADOOP-7098:
--------------------------------

Integrated in Hadoop-Common-trunk #620 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/620/])
    HADOOP-7098. Tasktracker property not set in conf/hadoop-env.sh. Contributed by Bernd Fondermann.


> tasktracker property not set in conf/hadoop-env.sh
> --------------------------------------------------
>
>                 Key: HADOOP-7098
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7098
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.20.2, 0.21.1, 0.22.0
>            Reporter: Bernd Fondermann
>            Assignee: Bernd Fondermann
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7098.patch, hadoop-7098.patch
>
>
> For all cluster components, except TaskTracker the OPTS environment variable is set like this in hadoop-env.sh:
> export HADOOP_<COMPONENT>_OPTS="-Dcom.sun.management.jmxremote $HADOOP_<COMPONENT>_OPTS"
> The provided patch fixes this.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13352-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 14:00:02 2011
Return-Path: <common-issues-return-13352-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72578 invoked from network); 4 Mar 2011 14:00:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 14:00:00 -0000
Received: (qmail 93256 invoked by uid 500); 4 Mar 2011 14:00:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93213 invoked by uid 500); 4 Mar 2011 14:00:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93184 invoked by uid 99); 4 Mar 2011 14:00:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 14:00:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 13:59:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0D7534FE13
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 13:59:37 +0000 (UTC)
Date: Fri, 4 Mar 2011 13:59:37 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1287761245.13905.1299247177051.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1846874015.7578.1296749909031.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7131) set() and toString Methods of the
 org.apache.hadoop.io.Text class does not include the root exception, in the
 wrapping RuntimeException.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uma Maheswara Rao G updated HADOOP-7131:
----------------------------------------

    Attachment: HADOOP-7131.patch

> set() and toString Methods of the org.apache.hadoop.io.Text class does not include the root exception, in the wrapping RuntimeException.
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7131
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7131
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Uma Maheswara Rao G
>            Priority: Minor
>         Attachments: HADOOP-7131.patch
>
>
>  In below code snippets, we can include e, instead of e.toString(), so that caller can get complete trace.
> 1) 
>    /** Set to contain the contents of a string.
>    */
>   public void set(String string) {
>     try {
>       ByteBuffer bb = encode(string, true);
>       bytes = bb.array();
>       length = bb.limit();
>     }catch(CharacterCodingException e) {
>       throw new RuntimeException("Should not have happened ",e.toString());
>     }
>   } 
> 2)
>    public String toString() {
>     try {
>       return decode(bytes, 0, length);
>     } catch (CharacterCodingException e) {
>       throw new RuntimeException("Should not have happened ",e.toString());
>     }
>   }

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13353-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 15:26:01 2011
Return-Path: <common-issues-return-13353-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60459 invoked from network); 4 Mar 2011 15:26:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 15:26:00 -0000
Received: (qmail 350 invoked by uid 500); 4 Mar 2011 15:26:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 313 invoked by uid 500); 4 Mar 2011 15:26:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 300 invoked by uid 99); 4 Mar 2011 15:26:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 15:26:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 15:25:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DB69C4F6FD
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 15:25:36 +0000 (UTC)
Date: Fri, 4 Mar 2011 15:25:36 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1976619391.14188.1299252336895.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1846874015.7578.1296749909031.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7131) set() and toString Methods of the
 org.apache.hadoop.io.Text class does not include the root exception, in the
 wrapping RuntimeException.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uma Maheswara Rao G updated HADOOP-7131:
----------------------------------------

    Status: Patch Available  (was: Open)

> set() and toString Methods of the org.apache.hadoop.io.Text class does not include the root exception, in the wrapping RuntimeException.
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7131
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7131
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2, 0.20.1
>            Reporter: Uma Maheswara Rao G
>            Priority: Minor
>         Attachments: HADOOP-7131.patch
>
>
>  In below code snippets, we can include e, instead of e.toString(), so that caller can get complete trace.
> 1) 
>    /** Set to contain the contents of a string.
>    */
>   public void set(String string) {
>     try {
>       ByteBuffer bb = encode(string, true);
>       bytes = bb.array();
>       length = bb.limit();
>     }catch(CharacterCodingException e) {
>       throw new RuntimeException("Should not have happened ",e.toString());
>     }
>   } 
> 2)
>    public String toString() {
>     try {
>       return decode(bytes, 0, length);
>     } catch (CharacterCodingException e) {
>       throw new RuntimeException("Should not have happened ",e.toString());
>     }
>   }

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13354-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 15:31:00 2011
Return-Path: <common-issues-return-13354-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77195 invoked from network); 4 Mar 2011 15:31:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 15:31:00 -0000
Received: (qmail 22424 invoked by uid 500); 4 Mar 2011 15:31:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22395 invoked by uid 500); 4 Mar 2011 15:31:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22386 invoked by uid 99); 4 Mar 2011 15:31:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 15:31:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 15:30:58 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 253204FA02
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 15:30:37 +0000 (UTC)
Date: Fri, 4 Mar 2011 15:30:37 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <571665438.14220.1299252637148.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1846874015.7578.1296749909031.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7131) set() and toString Methods of the
 org.apache.hadoop.io.Text class does not include the root exception, in the
 wrapping RuntimeException.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002667#comment-13002667 ] 

Uma Maheswara Rao G commented on HADOOP-7131:
---------------------------------------------

Here the verification results:
 [exec] +1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
     [exec]                         Please justify why no new tests are needed for this patch.
     [exec]                         Also please list what manual steps were performed to verify this patch.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec] 
     [exec]     +1 system test framework.  The patch passed system test framework compile.


Tests are not required for this changes. that is the reason for -1.

> set() and toString Methods of the org.apache.hadoop.io.Text class does not include the root exception, in the wrapping RuntimeException.
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7131
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7131
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Uma Maheswara Rao G
>            Priority: Minor
>         Attachments: HADOOP-7131.patch
>
>
>  In below code snippets, we can include e, instead of e.toString(), so that caller can get complete trace.
> 1) 
>    /** Set to contain the contents of a string.
>    */
>   public void set(String string) {
>     try {
>       ByteBuffer bb = encode(string, true);
>       bytes = bb.array();
>       length = bb.limit();
>     }catch(CharacterCodingException e) {
>       throw new RuntimeException("Should not have happened ",e.toString());
>     }
>   } 
> 2)
>    public String toString() {
>     try {
>       return decode(bytes, 0, length);
>     } catch (CharacterCodingException e) {
>       throw new RuntimeException("Should not have happened ",e.toString());
>     }
>   }

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13355-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 17:13:57 2011
Return-Path: <common-issues-return-13355-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31643 invoked from network); 4 Mar 2011 17:13:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 17:13:57 -0000
Received: (qmail 52797 invoked by uid 500); 4 Mar 2011 17:13:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52763 invoked by uid 500); 4 Mar 2011 17:13:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52729 invoked by uid 99); 4 Mar 2011 17:13:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 17:13:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 17:13:56 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E24444F66A
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 17:13:36 +0000 (UTC)
Date: Fri, 4 Mar 2011 17:13:36 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1213426933.14624.1299258816923.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <978952244.13379.1299230917384.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7162) call method without catching
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7162:
-------------------------------------------

        Assignee: Alexey Diomin
    Hadoop Flags: [Reviewed]

+1 patch looks good.

> call method without catching
> ----------------------------
>
>                 Key: HADOOP-7162
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7162
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Alexey Diomin
>            Assignee: Alexey Diomin
>            Priority: Minor
>         Attachments: HDFS-7162.path
>
>
> in file ./src/java/org/apache/hadoop/fs/FsShell.java line 555
> call method twice:
> 1. for init variable
> 2. for getting data

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13356-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 17:17:58 2011
Return-Path: <common-issues-return-13356-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40100 invoked from network); 4 Mar 2011 17:17:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 17:17:58 -0000
Received: (qmail 66403 invoked by uid 500); 4 Mar 2011 17:17:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66360 invoked by uid 500); 4 Mar 2011 17:17:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66352 invoked by uid 99); 4 Mar 2011 17:17:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 17:17:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 17:17:56 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DECCB4F843
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 17:17:36 +0000 (UTC)
Date: Fri, 4 Mar 2011 17:17:36 +0000 (UTC)
From: "Joep Rottinghuis (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <919753375.14638.1299259056909.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2042659645.11198.1299171397106.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7160) Configurable initial buffersize for
 getGroupDetails()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002724#comment-13002724 ] 

Joep Rottinghuis commented on HADOOP-7160:
------------------------------------------

Avoiding multiple round-trips seems like a good idea.

If you make it configurable, it would be good to have a way to be able to tell how large the value should be set.
The same mechanism can be used to answer Todd's question.


> Configurable initial buffersize for getGroupDetails()
> -----------------------------------------------------
>
>                 Key: HADOOP-7160
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7160
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: native, security
>    Affects Versions: 0.22.0
>            Reporter: T Meyarivan
>
> {code:title=trunk/src/native/src/org/apache/hadoop/security/getGroup.c|borderStyle=solid}
> int getGroupDetails(gid_t group, char **grpBuf) {
>   struct group * grp = NULL;
>   size_t currBufferSize = sysconf(_SC_GETGR_R_SIZE_MAX);
>   if (currBufferSize < 1024) {
>     currBufferSize = 1024;
>   }
>   *grpBuf = NULL; 
>   char *buf = (char*)malloc(sizeof(char) * currBufferSize);
>   if (!buf) {
>     return ENOMEM;
>   }
>   int error;
>   for (;;) {
>     error = getgrgid_r(group, (struct group*)buf,
>                        buf + sizeof(struct group),
>                        currBufferSize - sizeof(struct group), &grp);
>     if(error != ERANGE) {
>        break;
>     }
>     free(buf);
>     currBufferSize *= 2;
>     buf = malloc(sizeof(char) * currBufferSize);
>     if(!buf) {
>       return ENOMEM;
>     }
> ...
> {code}
> For large groups, this implies at least 2 queries for the group (number of queries = math.ceil(math.log(response_size/1024, 2)))
> In the case of a large cluster with central user/group databases (exposed via LDAP etc), this leads to unnecessary load on the central services. This can be alleviated to a large extent by changing the initial buffer size to a configurable parameter
> --

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13357-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 17:19:58 2011
Return-Path: <common-issues-return-13357-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48801 invoked from network); 4 Mar 2011 17:19:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 17:19:58 -0000
Received: (qmail 72074 invoked by uid 500); 4 Mar 2011 17:19:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72018 invoked by uid 500); 4 Mar 2011 17:19:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72010 invoked by uid 99); 4 Mar 2011 17:19:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 17:19:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 17:19:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0A1B94F913
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 17:19:37 +0000 (UTC)
Date: Fri, 4 Mar 2011 17:19:37 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <560934598.14645.1299259177037.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7163) "java.net.SocketTimeoutException:
 60000 millis timeout" happens a lot
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

"java.net.SocketTimeoutException: 60000 millis timeout" happens a lot
---------------------------------------------------------------------

                 Key: HADOOP-7163
                 URL: https://issues.apache.org/jira/browse/HADOOP-7163
             Project: Hadoop Common
          Issue Type: Bug
          Components: ipc
            Reporter: Owen O'Malley
            Assignee: Devaraj Das
             Fix For: 0.20.100


We don't have retries for the case where the secure SASL connection is getting created from the tasks. There is retry
for TCP connections, but once the TCP connection has been set up, communication at the RPC layer (and that includes
SASL handshake) happens without retries. So for example, a client's "read" can timeout.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13358-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 17:28:00 2011
Return-Path: <common-issues-return-13358-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79077 invoked from network); 4 Mar 2011 17:28:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 17:28:00 -0000
Received: (qmail 99177 invoked by uid 500); 4 Mar 2011 17:28:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99146 invoked by uid 500); 4 Mar 2011 17:28:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99137 invoked by uid 99); 4 Mar 2011 17:28:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 17:28:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 17:27:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D8A494FCA5
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 17:27:36 +0000 (UTC)
Date: Fri, 4 Mar 2011 17:27:36 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2096667661.14656.1299259656883.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Moved: (HADOOP-7164) rmr command is not displaying any error
 message when a path contains wildcard characters and does not exist.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE moved HDFS-1609 to HADOOP-7164:
------------------------------------------------------

          Component/s:     (was: hdfs client)
                       fs
    Affects Version/s:     (was: 0.20.2)
                           (was: 0.20.1)
                       0.20.1
                       0.20.2
                  Key: HADOOP-7164  (was: HDFS-1609)
              Project: Hadoop Common  (was: Hadoop HDFS)

> rmr command is not displaying any error message when a path contains wildcard characters and does not exist.
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7164
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7164
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.2, 0.20.1
>            Reporter: Uma Maheswara Rao G
>            Priority: Minor
>         Attachments: HDFS-1609-src.patch, HDFS-1609-test.patch, HDFS-1609.patch
>
>
> When we give invalid directory path then it will show error message on the console. But if we provide the wildcard expression in invalid directory path then it will not show any error message even there is no pattern match for that path.
> linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin # ./hdfs dfs -rmr /test/test
> rmr: cannot remove /test/test: No such file or directory.
> *linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin # ./hdfs dfs -rmr /test* *
> *linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin #*

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13359-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 17:29:58 2011
Return-Path: <common-issues-return-13359-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79833 invoked from network); 4 Mar 2011 17:29:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 17:29:57 -0000
Received: (qmail 3625 invoked by uid 500); 4 Mar 2011 17:29:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3599 invoked by uid 500); 4 Mar 2011 17:29:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3591 invoked by uid 99); 4 Mar 2011 17:29:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 17:29:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 17:29:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 07A104FDD9
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 17:29:37 +0000 (UTC)
Date: Fri, 4 Mar 2011 17:29:37 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <339251107.14666.1299259777027.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7164) rmr command is not displaying any
 error message when a path contains wildcard characters and does not exist.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7164:
-------------------------------------------

    Hadoop Flags: [Incompatible change]

> tcsh and bash behaves differently for multiple globbing.

If the behavior is not standardized, I suggest to keep the current implementation.

Marking this as "Incompatible Change".

> rmr command is not displaying any error message when a path contains wildcard characters and does not exist.
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7164
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7164
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Uma Maheswara Rao G
>            Priority: Minor
>         Attachments: HDFS-1609-src.patch, HDFS-1609-test.patch, HDFS-1609.patch
>
>
> When we give invalid directory path then it will show error message on the console. But if we provide the wildcard expression in invalid directory path then it will not show any error message even there is no pattern match for that path.
> linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin # ./hdfs dfs -rmr /test/test
> rmr: cannot remove /test/test: No such file or directory.
> *linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin # ./hdfs dfs -rmr /test* *
> *linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin #*

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13360-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 17:31:58 2011
Return-Path: <common-issues-return-13360-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92980 invoked from network); 4 Mar 2011 17:31:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 17:31:58 -0000
Received: (qmail 9807 invoked by uid 500); 4 Mar 2011 17:31:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9746 invoked by uid 500); 4 Mar 2011 17:31:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9719 invoked by uid 99); 4 Mar 2011 17:31:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 17:31:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 17:31:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 46B2B4FEDE
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 17:31:37 +0000 (UTC)
Date: Fri, 4 Mar 2011 17:31:37 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1912492318.14684.1299259897286.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7164) rmr command is not displaying any
 error message when a path contains wildcard characters and does not exist.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002731#comment-13002731 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7164:
------------------------------------------------

I also have moved this to Common.

> rmr command is not displaying any error message when a path contains wildcard characters and does not exist.
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7164
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7164
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Uma Maheswara Rao G
>            Priority: Minor
>         Attachments: HDFS-1609-src.patch, HDFS-1609-test.patch, HDFS-1609.patch
>
>
> When we give invalid directory path then it will show error message on the console. But if we provide the wildcard expression in invalid directory path then it will not show any error message even there is no pattern match for that path.
> linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin # ./hdfs dfs -rmr /test/test
> rmr: cannot remove /test/test: No such file or directory.
> *linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin # ./hdfs dfs -rmr /test* *
> *linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin #*

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13361-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 17:37:57 2011
Return-Path: <common-issues-return-13361-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25559 invoked from network); 4 Mar 2011 17:37:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 17:37:57 -0000
Received: (qmail 27646 invoked by uid 500); 4 Mar 2011 17:37:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27615 invoked by uid 500); 4 Mar 2011 17:37:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27607 invoked by uid 99); 4 Mar 2011 17:37:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 17:37:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 17:37:56 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D61F14F161
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 17:37:36 +0000 (UTC)
Date: Fri, 4 Mar 2011 17:37:36 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1823079031.14696.1299260256873.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <978952244.13379.1299230917384.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7162) call method without catching
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002736#comment-13002736 ] 

Hadoop QA commented on HADOOP-7162:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12472654/HDFS-7162.path
  against trunk revision 1077811.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/302//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/302//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/302//console

This message is automatically generated.

> call method without catching
> ----------------------------
>
>                 Key: HADOOP-7162
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7162
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Alexey Diomin
>            Assignee: Alexey Diomin
>            Priority: Minor
>         Attachments: HDFS-7162.path
>
>
> in file ./src/java/org/apache/hadoop/fs/FsShell.java line 555
> call method twice:
> 1. for init variable
> 2. for getting data

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13362-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 17:44:02 2011
Return-Path: <common-issues-return-13362-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27212 invoked from network); 4 Mar 2011 17:44:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 17:44:02 -0000
Received: (qmail 47244 invoked by uid 500); 4 Mar 2011 17:44:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47184 invoked by uid 500); 4 Mar 2011 17:44:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47175 invoked by uid 99); 4 Mar 2011 17:44:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 17:44:01 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 17:43:58 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AA74E4F37B
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 17:43:37 +0000 (UTC)
Date: Fri, 4 Mar 2011 17:43:37 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1173634415.14710.1299260617694.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1846874015.7578.1296749909031.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7131) set() and toString Methods of the
 org.apache.hadoop.io.Text class does not include the root exception, in the
 wrapping RuntimeException.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002738#comment-13002738 ] 

Hadoop QA commented on HADOOP-7131:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12472667/HADOOP-7131.patch
  against trunk revision 1077811.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/301//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/301//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/301//console

This message is automatically generated.

> set() and toString Methods of the org.apache.hadoop.io.Text class does not include the root exception, in the wrapping RuntimeException.
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7131
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7131
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Uma Maheswara Rao G
>            Priority: Minor
>         Attachments: HADOOP-7131.patch
>
>
>  In below code snippets, we can include e, instead of e.toString(), so that caller can get complete trace.
> 1) 
>    /** Set to contain the contents of a string.
>    */
>   public void set(String string) {
>     try {
>       ByteBuffer bb = encode(string, true);
>       bytes = bb.array();
>       length = bb.limit();
>     }catch(CharacterCodingException e) {
>       throw new RuntimeException("Should not have happened ",e.toString());
>     }
>   } 
> 2)
>    public String toString() {
>     try {
>       return decode(bytes, 0, length);
>     } catch (CharacterCodingException e) {
>       throw new RuntimeException("Should not have happened ",e.toString());
>     }
>   }

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13363-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 21:36:10 2011
Return-Path: <common-issues-return-13363-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27359 invoked from network); 4 Mar 2011 21:36:10 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 21:36:10 -0000
Received: (qmail 41214 invoked by uid 500); 4 Mar 2011 21:36:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41188 invoked by uid 500); 4 Mar 2011 21:36:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41180 invoked by uid 99); 4 Mar 2011 21:36:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 21:36:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 21:36:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E6C72556E2
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 21:35:45 +0000 (UTC)
Date: Fri, 4 Mar 2011 21:35:45 +0000 (UTC)
From: "T Meyarivan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1249524610.111.1299274545941.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2042659645.11198.1299171397106.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7160) Configurable initial buffersize for
 getGroupDetails()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002828#comment-13002828 ] 

T Meyarivan commented on HADOOP-7160:
-------------------------------------

tlipcon@:

Not sure about fixed limits. If it is important to avoid another config parameter, how about adjusting the initial buffer size based on the response size 

--


> Configurable initial buffersize for getGroupDetails()
> -----------------------------------------------------
>
>                 Key: HADOOP-7160
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7160
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: native, security
>    Affects Versions: 0.22.0
>            Reporter: T Meyarivan
>
> {code:title=trunk/src/native/src/org/apache/hadoop/security/getGroup.c|borderStyle=solid}
> int getGroupDetails(gid_t group, char **grpBuf) {
>   struct group * grp = NULL;
>   size_t currBufferSize = sysconf(_SC_GETGR_R_SIZE_MAX);
>   if (currBufferSize < 1024) {
>     currBufferSize = 1024;
>   }
>   *grpBuf = NULL; 
>   char *buf = (char*)malloc(sizeof(char) * currBufferSize);
>   if (!buf) {
>     return ENOMEM;
>   }
>   int error;
>   for (;;) {
>     error = getgrgid_r(group, (struct group*)buf,
>                        buf + sizeof(struct group),
>                        currBufferSize - sizeof(struct group), &grp);
>     if(error != ERANGE) {
>        break;
>     }
>     free(buf);
>     currBufferSize *= 2;
>     buf = malloc(sizeof(char) * currBufferSize);
>     if(!buf) {
>       return ENOMEM;
>     }
> ...
> {code}
> For large groups, this implies at least 2 queries for the group (number of queries = math.ceil(math.log(response_size/1024, 2)))
> In the case of a large cluster with central user/group databases (exposed via LDAP etc), this leads to unnecessary load on the central services. This can be alleviated to a large extent by changing the initial buffer size to a configurable parameter
> --

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13364-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 21:38:14 2011
Return-Path: <common-issues-return-13364-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33301 invoked from network); 4 Mar 2011 21:38:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 21:38:11 -0000
Received: (qmail 46081 invoked by uid 500); 4 Mar 2011 21:38:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46036 invoked by uid 500); 4 Mar 2011 21:38:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46028 invoked by uid 99); 4 Mar 2011 21:38:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 21:38:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 21:38:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E89D655794
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 21:37:47 +0000 (UTC)
Date: Fri, 4 Mar 2011 21:37:47 +0000 (UTC)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <773739797.118.1299274667949.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7165) listLocatedStatus(path, filter) is
 not redefined in FilterFs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

listLocatedStatus(path, filter) is not redefined in FilterFs
------------------------------------------------------------

                 Key: HADOOP-7165
                 URL: https://issues.apache.org/jira/browse/HADOOP-7165
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
            Reporter: Hairong Kuang
            Assignee: Hairong Kuang
             Fix For: 0.23.0


listLocatedStatus(path, filter) is not redefined in FilterFs. So if a job client uses a FilterFs to talk to NameNode, it does not trigger the bulk location optimization.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13365-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 21:46:08 2011
Return-Path: <common-issues-return-13365-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45163 invoked from network); 4 Mar 2011 21:46:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 21:46:07 -0000
Received: (qmail 63279 invoked by uid 500); 4 Mar 2011 21:46:07 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63255 invoked by uid 500); 4 Mar 2011 21:46:07 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63247 invoked by uid 99); 4 Mar 2011 21:46:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 21:46:07 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 21:46:06 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2013555AA6
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 21:45:46 +0000 (UTC)
Date: Fri, 4 Mar 2011 21:45:46 +0000 (UTC)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1249492750.129.1299275146128.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <773739797.118.1299274667949.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7165) listLocatedStatus(path, filter) is
 not redefined in FilterFs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-7165:
----------------------------------

    Attachment: locatedStatusFilter.patch

Here is a straightforward fix.

> listLocatedStatus(path, filter) is not redefined in FilterFs
> ------------------------------------------------------------
>
>                 Key: HADOOP-7165
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7165
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.23.0
>
>         Attachments: locatedStatusFilter.patch
>
>
> listLocatedStatus(path, filter) is not redefined in FilterFs. So if a job client uses a FilterFs to talk to NameNode, it does not trigger the bulk location optimization.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13366-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 21:48:10 2011
Return-Path: <common-issues-return-13366-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62278 invoked from network); 4 Mar 2011 21:48:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 21:48:09 -0000
Received: (qmail 69405 invoked by uid 500); 4 Mar 2011 21:48:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69349 invoked by uid 500); 4 Mar 2011 21:48:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69335 invoked by uid 99); 4 Mar 2011 21:48:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 21:48:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 21:48:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D609A55B34
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 21:47:45 +0000 (UTC)
Date: Fri, 4 Mar 2011 21:47:45 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <132607900.131.1299275265873.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <978952244.13379.1299230917384.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7162) FsShell: call srcFs.listStatus(src)
 twice
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7162:
-------------------------------------------

      Component/s:     (was: util)
                   fs
    Fix Version/s: 0.23.0
                   0.22.0
                   0.21.1
          Summary: FsShell: call srcFs.listStatus(src) twice  (was: call method without catching)

> FsShell: call srcFs.listStatus(src) twice
> -----------------------------------------
>
>                 Key: HADOOP-7162
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7162
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Alexey Diomin
>            Assignee: Alexey Diomin
>            Priority: Minor
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HDFS-7162.path
>
>
> in file ./src/java/org/apache/hadoop/fs/FsShell.java line 555
> call method twice:
> 1. for init variable
> 2. for getting data

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13367-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 21:50:07 2011
Return-Path: <common-issues-return-13367-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78937 invoked from network); 4 Mar 2011 21:50:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 21:50:07 -0000
Received: (qmail 74500 invoked by uid 500); 4 Mar 2011 21:50:07 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74455 invoked by uid 500); 4 Mar 2011 21:50:07 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74445 invoked by uid 99); 4 Mar 2011 21:50:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 21:50:07 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 21:50:06 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 02F8C55C31
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 21:49:46 +0000 (UTC)
Date: Fri, 4 Mar 2011 21:49:46 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <803070126.142.1299275386008.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <978952244.13379.1299230917384.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7162) FsShell: call srcFs.listStatus(src)
 twice
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7162:
-------------------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I have committed this.  Thanks, Alexey!

> FsShell: call srcFs.listStatus(src) twice
> -----------------------------------------
>
>                 Key: HADOOP-7162
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7162
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Alexey Diomin
>            Assignee: Alexey Diomin
>            Priority: Minor
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HDFS-7162.path
>
>
> in file ./src/java/org/apache/hadoop/fs/FsShell.java line 555
> call method twice:
> 1. for init variable
> 2. for getting data

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13368-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 22:05:08 2011
Return-Path: <common-issues-return-13368-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19471 invoked from network); 4 Mar 2011 22:05:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 22:05:08 -0000
Received: (qmail 16301 invoked by uid 500); 4 Mar 2011 22:05:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16268 invoked by uid 500); 4 Mar 2011 22:05:07 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16260 invoked by uid 99); 4 Mar 2011 22:05:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 22:05:07 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 22:05:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7C808552D4
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 22:04:46 +0000 (UTC)
Date: Fri, 4 Mar 2011 22:04:46 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <887072337.181.1299276286506.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <773739797.118.1299274667949.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7165) listLocatedStatus(path, filter) is
 not redefined in FilterFs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002839#comment-13002839 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7165:
------------------------------------------------

+1 patch looks good.

> listLocatedStatus(path, filter) is not redefined in FilterFs
> ------------------------------------------------------------
>
>                 Key: HADOOP-7165
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7165
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.23.0
>
>         Attachments: locatedStatusFilter.patch
>
>
> listLocatedStatus(path, filter) is not redefined in FilterFs. So if a job client uses a FilterFs to talk to NameNode, it does not trigger the bulk location optimization.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13369-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 23:45:11 2011
Return-Path: <common-issues-return-13369-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24662 invoked from network); 4 Mar 2011 23:45:10 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 23:45:10 -0000
Received: (qmail 62567 invoked by uid 500); 4 Mar 2011 23:45:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62459 invoked by uid 500); 4 Mar 2011 23:45:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62330 invoked by uid 99); 4 Mar 2011 23:45:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 23:45:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 23:45:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E1BF455D87
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 23:44:45 +0000 (UTC)
Date: Fri, 4 Mar 2011 23:44:45 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <492129204.336.1299282285921.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002871#comment-13002871 ] 

Todd Lipcon commented on HADOOP-6762:
-------------------------------------

Hi Sam,

We saw the following deadlock which I think is related to this patch:

{noformat}
Thread 50994 (IPC Client (47) connection to XXXXXXXXX:8020 from hdfs):
State: BLOCKED
Blocked count: 7168
Waited count: 7122
Blocked on java.io.DataOutputStream@2e932fec
Blocked by 50828 (sendParams-14)
Stack:
org.apache.hadoop.ipc.Client$Connection.sendPing(Client.java:676)
org.apache.hadoop.ipc.Client$Connection.access$400(Client.java:210)
org.apache.hadoop.ipc.Client$Connection$PingInputStream.handleTimeout(Client.java:340)
org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:370)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:237)
java.io.DataInputStream.readInt(DataInputStream.java:370)
org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:781)
org.apache.hadoop.ipc.Client$Connection.run(Client.java:689)


Thread 50828 (sendParams-14):
State: BLOCKED
Blocked count: 54
Waited count: 8313
Blocked on org.apache.hadoop.ipc.Client$Connection@54aeb3dd
Blocked by 50994 (IPC Client (47) connection to XXXXXX:8020 from hdfs)
Stack:
org.apache.hadoop.ipc.Client$Connection.markClosed(Client.java:809)
org.apache.hadoop.ipc.Client$Connection.access$1200(Client.java:210)
org.apache.hadoop.ipc.Client$Connection$3.run(Client.java:745)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
java.util.concurrent.FutureTask.run(FutureTask.java:138)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:619)
{noformat}

The issue is that in the patch, we have the following inverted lock orders:
sendParam's senderFuture: Connection.out -> Connection (in markClosed)
sendPing: Connection -> Connection.out (explicit sync)

Have you guys seen this issue?

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6762-1.txt, hadoop-6762-10.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt, hadoop-6762-7.txt, hadoop-6762-8.txt, hadoop-6762-9.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13370-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 04 23:55:08 2011
Return-Path: <common-issues-return-13370-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67603 invoked from network); 4 Mar 2011 23:55:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Mar 2011 23:55:07 -0000
Received: (qmail 80258 invoked by uid 500); 4 Mar 2011 23:55:07 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80145 invoked by uid 500); 4 Mar 2011 23:55:07 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80101 invoked by uid 99); 4 Mar 2011 23:55:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 23:55:07 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Mar 2011 23:55:06 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1E66655FA8
	for <common-issues@hadoop.apache.org>; Fri,  4 Mar 2011 23:54:46 +0000 (UTC)
Date: Fri, 4 Mar 2011 23:54:46 +0000 (UTC)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1405826610.356.1299282886121.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002873#comment-13002873 ] 

sam rash commented on HADOOP-6762:
----------------------------------

I haven't noticed this.  In the rev we're running,  inside sendParams(), markClosed() is called outside the synchronized(Connection.this.out), so that lock isn't held...?

(I'm using the original patch I uploaded before we changed it to a Future--it uses a latch)





> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6762-1.txt, hadoop-6762-10.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt, hadoop-6762-7.txt, hadoop-6762-8.txt, hadoop-6762-9.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13371-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 05 00:44:08 2011
Return-Path: <common-issues-return-13371-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34000 invoked from network); 5 Mar 2011 00:44:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Mar 2011 00:44:07 -0000
Received: (qmail 76730 invoked by uid 500); 5 Mar 2011 00:44:07 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76611 invoked by uid 500); 5 Mar 2011 00:44:07 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76602 invoked by uid 99); 5 Mar 2011 00:44:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Mar 2011 00:44:07 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Mar 2011 00:44:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 36CCF55D36
	for <common-issues@hadoop.apache.org>; Sat,  5 Mar 2011 00:43:46 +0000 (UTC)
Date: Sat, 5 Mar 2011 00:43:46 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1822066731.411.1299285826221.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <978952244.13379.1299230917384.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7162) FsShell: call srcFs.listStatus(src)
 twice
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002885#comment-13002885 ] 

Hudson commented on HADOOP-7162:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #517 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/517/])
    

> FsShell: call srcFs.listStatus(src) twice
> -----------------------------------------
>
>                 Key: HADOOP-7162
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7162
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Alexey Diomin
>            Assignee: Alexey Diomin
>            Priority: Minor
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HDFS-7162.path
>
>
> in file ./src/java/org/apache/hadoop/fs/FsShell.java line 555
> call method twice:
> 1. for init variable
> 2. for getting data

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13372-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 05 00:44:08 2011
Return-Path: <common-issues-return-13372-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34018 invoked from network); 5 Mar 2011 00:44:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Mar 2011 00:44:08 -0000
Received: (qmail 76915 invoked by uid 500); 5 Mar 2011 00:44:07 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76824 invoked by uid 500); 5 Mar 2011 00:44:07 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76620 invoked by uid 99); 5 Mar 2011 00:44:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Mar 2011 00:44:07 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Mar 2011 00:44:06 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 12A8255D32
	for <common-issues@hadoop.apache.org>; Sat,  5 Mar 2011 00:43:46 +0000 (UTC)
Date: Sat, 5 Mar 2011 00:43:46 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <665775244.407.1299285826072.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <31897278.92301295561324063.JavaMail.jira@thor>
Subject: [jira] Commented: (HADOOP-7114) FsShell should dump all exceptions
 at DEBUG level
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002886#comment-13002886 ] 

Hudson commented on HADOOP-7114:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #517 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/517/])
    

> FsShell should dump all exceptions at DEBUG level
> -------------------------------------------------
>
>                 Key: HADOOP-7114
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7114
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7114.txt
>
>
> Most of the FsShell commands catch exceptions and then just print out an error like "foo: " + e.getLocalizedMessage(). This is fine when the exception is "user-facing" (eg permissions errors) but in the case of a user hitting a bug you get a useless error message with no stack trace. For example, something "chmod: null" in the case of a NullPointerException bug.
> It would help debug these cases for users and developers if we also logged the exception with full trace at DEBUG level.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13373-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 05 00:44:10 2011
Return-Path: <common-issues-return-13373-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34099 invoked from network); 5 Mar 2011 00:44:10 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Mar 2011 00:44:10 -0000
Received: (qmail 77134 invoked by uid 500); 5 Mar 2011 00:44:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77109 invoked by uid 500); 5 Mar 2011 00:44:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77101 invoked by uid 99); 5 Mar 2011 00:44:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Mar 2011 00:44:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Mar 2011 00:44:07 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CF13155D2B
	for <common-issues@hadoop.apache.org>; Sat,  5 Mar 2011 00:43:45 +0000 (UTC)
Date: Sat, 5 Mar 2011 00:43:45 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1096868553.401.1299285825844.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2257860.286501294766747197.JavaMail.jira@thor>
Subject: [jira] Commented: (HADOOP-7098) tasktracker property not set in
 conf/hadoop-env.sh
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002887#comment-13002887 ] 

Hudson commented on HADOOP-7098:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #517 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/517/])
    

> tasktracker property not set in conf/hadoop-env.sh
> --------------------------------------------------
>
>                 Key: HADOOP-7098
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7098
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.20.2, 0.21.1, 0.22.0
>            Reporter: Bernd Fondermann
>            Assignee: Bernd Fondermann
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7098.patch, hadoop-7098.patch
>
>
> For all cluster components, except TaskTracker the OPTS environment variable is set like this in hadoop-env.sh:
> export HADOOP_<COMPONENT>_OPTS="-Dcom.sun.management.jmxremote $HADOOP_<COMPONENT>_OPTS"
> The provided patch fixes this.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13374-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 05 00:44:11 2011
Return-Path: <common-issues-return-13374-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34138 invoked from network); 5 Mar 2011 00:44:10 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Mar 2011 00:44:10 -0000
Received: (qmail 77378 invoked by uid 500); 5 Mar 2011 00:44:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77326 invoked by uid 500); 5 Mar 2011 00:44:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77318 invoked by uid 99); 5 Mar 2011 00:44:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Mar 2011 00:44:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Mar 2011 00:44:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8F0B155D3B
	for <common-issues@hadoop.apache.org>; Sat,  5 Mar 2011 00:43:46 +0000 (UTC)
Date: Sat, 5 Mar 2011 00:43:46 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1753147427.415.1299285826582.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-6754) DefaultCodec.createOutputStream()
 leaks memory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002884#comment-13002884 ] 

Hudson commented on HADOOP-6754:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #517 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/517/])
    

> DefaultCodec.createOutputStream() leaks memory
> ----------------------------------------------
>
>                 Key: HADOOP-6754
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6754
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>             Fix For: 0.23.0
>
>         Attachments: CompressionBug.java, HADOOP-6754.2.patch, HADOOP-6754.patch, HADOOP-6754.patch
>
>
> DefaultCodec.createOutputStream() creates a new Compressor instance in each OutputStream. Even if the OutputStream is closed, this leaks memory.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13375-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 05 02:27:11 2011
Return-Path: <common-issues-return-13375-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59741 invoked from network); 5 Mar 2011 02:27:09 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Mar 2011 02:27:09 -0000
Received: (qmail 39166 invoked by uid 500); 5 Mar 2011 02:27:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38729 invoked by uid 500); 5 Mar 2011 02:27:07 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37593 invoked by uid 99); 5 Mar 2011 02:26:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Mar 2011 02:26:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Mar 2011 02:11:06 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0178B56979
	for <common-issues@hadoop.apache.org>; Sat,  5 Mar 2011 02:10:46 +0000 (UTC)
Date: Sat, 5 Mar 2011 02:10:46 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <642447675.524.1299291046002.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-6949) Reduces RPC packet size for primitive
 arrays, especially long[], which is used at block reporting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated HADOOP-6949:
-------------------------------

    Attachment: arrayprim_v5.patch

Re-submitting same patch file in hopes that the auto-tester will run.

> Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6949
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6949
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Navis
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: ObjectWritable.diff, arrayprim.patch, arrayprim_v4.patch, arrayprim_v5.patch, arrayprim_v5.patch
>
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n
> It would not be needed to specify each element types for primitive arrays.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13376-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 05 03:01:12 2011
Return-Path: <common-issues-return-13376-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63234 invoked from network); 5 Mar 2011 03:01:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Mar 2011 03:01:11 -0000
Received: (qmail 7774 invoked by uid 500); 5 Mar 2011 03:01:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6632 invoked by uid 500); 5 Mar 2011 03:01:07 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6439 invoked by uid 99); 5 Mar 2011 03:01:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Mar 2011 03:01:04 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Mar 2011 01:45:06 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D76A6566FD
	for <common-issues@hadoop.apache.org>; Sat,  5 Mar 2011 01:44:45 +0000 (UTC)
Date: Sat, 5 Mar 2011 01:44:45 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <189532702.505.1299289485878.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002903#comment-13002903 ] 

Todd Lipcon commented on HADOOP-7156:
-------------------------------------

Apparently no one out there who implements pam modules knows that getpwuid_r is supposed to be reentrant :) A user saw this same issue with VAS:

/lib64/libc.so.6[0x3757c7230f]
/lib64/libc.so.6(cfree+0x4b)[0x3757c7276b]
/opt/quest/lib64/libvas.so.4(vassym_sqlite3FreeX+0xe)[0x2aaaab2bd15e]
/opt/quest/lib64/libvas.so.4[0x2aaaab2ac9b9]
/opt/quest/lib64/libvas.so.4(vassym_sqlite3OsClose+0x93)[0x2aaaab2ad6d3]
/opt/quest/lib64/libvas.so.4(vassym_sqlite3pager_close+0x4d)[0x2aaaab2af1cd]
/opt/quest/lib64/libvas.so.4(vassym_sqlite3BtreeClose+0x15)[0x2aaaab29a235]
/opt/quest/lib64/libvas.so.4(vassym_sqlite3_close+0x15b)[0x2aaaab2ab62b]
/opt/quest/lib64/libvas.so.4(vassql_deinit+0xdf)[0x2aaaab3158de]
/opt/quest/lib64/libvas.so.4[0x2aaaab2f3ef9]
/opt/quest/lib64/libvas.so.4(vascache_deinit+0x30)[0x2aaaab2f438f]
/lib64/libnss_vas3.so.2[0x2aaaab027240]
/lib64/libnss_vas3.so.2(_nss_vas3_getXXent_release_tsd+0x10e)[0x2aaaab0284ff]
/lib64/libnss_vas3.so.2[0x2aaaab02c3c0]
/lib64/libnss_vas3.so.2(_nss_vas3_getpwuid_r+0x36)[0x2aaaab02d242]
/lib64/libc.so.6(getpwuid_r+0xa5)[0x3757c99515]


> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13377-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 05 16:15:08 2011
Return-Path: <common-issues-return-13377-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80121 invoked from network); 5 Mar 2011 16:15:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Mar 2011 16:15:08 -0000
Received: (qmail 99579 invoked by uid 500); 5 Mar 2011 16:15:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99537 invoked by uid 500); 5 Mar 2011 16:15:07 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99525 invoked by uid 99); 5 Mar 2011 16:15:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Mar 2011 16:15:07 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Mar 2011 16:15:06 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D2592569EA
	for <common-issues@hadoop.apache.org>; Sat,  5 Mar 2011 16:14:45 +0000 (UTC)
Date: Sat, 5 Mar 2011 16:14:45 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2043144497.999.1299341685857.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <978952244.13379.1299230917384.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7162) FsShell: call srcFs.listStatus(src)
 twice
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002990#comment-13002990 ] 

Hudson commented on HADOOP-7162:
--------------------------------

Integrated in Hadoop-Common-trunk #621 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/621/])
    HADOOP-7162. Rmove a duplicated call FileSystem.listStatus(..) in FsShell.  Contributed by Alexey Diomin


> FsShell: call srcFs.listStatus(src) twice
> -----------------------------------------
>
>                 Key: HADOOP-7162
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7162
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Alexey Diomin
>            Assignee: Alexey Diomin
>            Priority: Minor
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HDFS-7162.path
>
>
> in file ./src/java/org/apache/hadoop/fs/FsShell.java line 555
> call method twice:
> 1. for init variable
> 2. for getting data

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13378-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Mar 06 10:52:11 2011
Return-Path: <common-issues-return-13378-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30327 invoked from network); 6 Mar 2011 10:52:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Mar 2011 10:52:11 -0000
Received: (qmail 11953 invoked by uid 500); 6 Mar 2011 10:52:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11906 invoked by uid 500); 6 Mar 2011 10:52:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11891 invoked by uid 99); 6 Mar 2011 10:52:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Mar 2011 10:52:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Mar 2011 10:52:08 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5D38E58242
	for <common-issues@hadoop.apache.org>; Sun,  6 Mar 2011 10:51:46 +0000 (UTC)
Date: Sun, 6 Mar 2011 10:51:46 +0000 (UTC)
From: "Yanbo Liang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <749954614.1749.1299408706378.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-6239) Command-line for append
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003128#comment-13003128 ] 

Yanbo Liang commented on HADOOP-6239:
-------------------------------------

Maybe append on a file mostly used by API rather than command-line such as HBase log/transaction files and Facebook Scribe.

> Command-line for append
> -----------------------
>
>                 Key: HADOOP-6239
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6239
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>            Reporter: Koji Noguchi
>
> Once append is implemented in hdfs, it would be nice if users can append files from a command-line. 
> Either have a separate 'append' command or add '-append' option for 'put' and 'cp'
> Like,  
> {noformat}
> % cat mylocalfile  | hadoop dfs -put -append - /user/knoguchi/myfile    ('-' for stdin)
> % hadoop dfs -cp -append myhdfsfile1 myhdfsfile2  
> {noformat}

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-13379-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 03:13:30 2011
Return-Path: <common-issues-return-13379-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36892 invoked from network); 7 Mar 2011 03:13:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 03:13:30 -0000
Received: (qmail 2711 invoked by uid 500); 7 Mar 2011 03:13:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2575 invoked by uid 500); 7 Mar 2011 03:13:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2542 invoked by uid 99); 7 Mar 2011 03:13:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 03:13:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 03:13:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B1BA839ACA2
	for <common-issues@hadoop.apache.org>; Mon,  7 Mar 2011 03:12:59 +0000 (UTC)
Date: Mon, 7 Mar 2011 03:12:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1319909919.263.1299467579724.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1846874015.7578.1296749909031.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Assigned: (HADOOP-7131) set() and toString Methods of the
 org.apache.hadoop.io.Text class does not include the root exception, in the
 wrapping RuntimeException.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon reassigned HADOOP-7131:
-----------------------------------

    Assignee: Uma Maheswara Rao G

> set() and toString Methods of the org.apache.hadoop.io.Text class does not include the root exception, in the wrapping RuntimeException.
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7131
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7131
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>            Priority: Minor
>         Attachments: HADOOP-7131.patch
>
>
>  In below code snippets, we can include e, instead of e.toString(), so that caller can get complete trace.
> 1) 
>    /** Set to contain the contents of a string.
>    */
>   public void set(String string) {
>     try {
>       ByteBuffer bb = encode(string, true);
>       bytes = bb.array();
>       length = bb.limit();
>     }catch(CharacterCodingException e) {
>       throw new RuntimeException("Should not have happened ",e.toString());
>     }
>   } 
> 2)
>    public String toString() {
>     try {
>       return decode(bytes, 0, length);
>     } catch (CharacterCodingException e) {
>       throw new RuntimeException("Should not have happened ",e.toString());
>     }
>   }

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13380-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 03:15:24 2011
Return-Path: <common-issues-return-13380-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38165 invoked from network); 7 Mar 2011 03:15:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 03:15:24 -0000
Received: (qmail 5773 invoked by uid 500); 7 Mar 2011 03:15:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5725 invoked by uid 500); 7 Mar 2011 03:15:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5710 invoked by uid 99); 7 Mar 2011 03:15:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 03:15:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 03:15:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7A17D39AD4D
	for <common-issues@hadoop.apache.org>; Mon,  7 Mar 2011 03:14:59 +0000 (UTC)
Date: Mon, 7 Mar 2011 03:14:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1694176073.267.1299467699496.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1846874015.7578.1296749909031.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7131) set() and toString Methods of the
 org.apache.hadoop.io.Text class does not include the root exception, in the
 wrapping RuntimeException.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003244#comment-13003244 ] 

Todd Lipcon commented on HADOOP-7131:
-------------------------------------

+1

> set() and toString Methods of the org.apache.hadoop.io.Text class does not include the root exception, in the wrapping RuntimeException.
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7131
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7131
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>            Priority: Minor
>         Attachments: HADOOP-7131.patch
>
>
>  In below code snippets, we can include e, instead of e.toString(), so that caller can get complete trace.
> 1) 
>    /** Set to contain the contents of a string.
>    */
>   public void set(String string) {
>     try {
>       ByteBuffer bb = encode(string, true);
>       bytes = bb.array();
>       length = bb.limit();
>     }catch(CharacterCodingException e) {
>       throw new RuntimeException("Should not have happened ",e.toString());
>     }
>   } 
> 2)
>    public String toString() {
>     try {
>       return decode(bytes, 0, length);
>     } catch (CharacterCodingException e) {
>       throw new RuntimeException("Should not have happened ",e.toString());
>     }
>   }

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13381-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 03:17:33 2011
Return-Path: <common-issues-return-13381-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43309 invoked from network); 7 Mar 2011 03:17:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 03:17:33 -0000
Received: (qmail 9696 invoked by uid 500); 7 Mar 2011 03:17:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9663 invoked by uid 500); 7 Mar 2011 03:17:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9635 invoked by uid 99); 7 Mar 2011 03:17:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 03:17:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 03:17:30 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2A13239ADC7
	for <common-issues@hadoop.apache.org>; Mon,  7 Mar 2011 03:17:09 +0000 (UTC)
Date: Mon, 7 Mar 2011 03:17:09 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1499677743.272.1299467829168.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1846874015.7578.1296749909031.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7131) set() and toString Methods of the
 org.apache.hadoop.io.Text class does not include the root exception, in the
 wrapping RuntimeException.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7131:
--------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks!

> set() and toString Methods of the org.apache.hadoop.io.Text class does not include the root exception, in the wrapping RuntimeException.
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7131
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7131
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7131.patch
>
>
>  In below code snippets, we can include e, instead of e.toString(), so that caller can get complete trace.
> 1) 
>    /** Set to contain the contents of a string.
>    */
>   public void set(String string) {
>     try {
>       ByteBuffer bb = encode(string, true);
>       bytes = bb.array();
>       length = bb.limit();
>     }catch(CharacterCodingException e) {
>       throw new RuntimeException("Should not have happened ",e.toString());
>     }
>   } 
> 2)
>    public String toString() {
>     try {
>       return decode(bytes, 0, length);
>     } catch (CharacterCodingException e) {
>       throw new RuntimeException("Should not have happened ",e.toString());
>     }
>   }

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13382-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 03:45:25 2011
Return-Path: <common-issues-return-13382-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30956 invoked from network); 7 Mar 2011 03:45:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 03:45:25 -0000
Received: (qmail 65826 invoked by uid 500); 7 Mar 2011 03:45:25 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65792 invoked by uid 500); 7 Mar 2011 03:45:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65764 invoked by uid 99); 7 Mar 2011 03:45:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 03:45:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 03:45:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7CFC339A3BF
	for <common-issues@hadoop.apache.org>; Mon,  7 Mar 2011 03:44:59 +0000 (UTC)
Date: Mon, 7 Mar 2011 03:44:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1352962807.315.1299469499508.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <824415802.6352.1299019897038.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7159) RPC server should log the client
 hostname when read exception happened
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7159:
--------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

+1. Committed to trunk after running related tests.

> RPC server should log the client hostname when read exception happened
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-7159
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7159
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7159.txt
>
>
> This makes find mismatched clients easier

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13383-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 05:56:36 2011
Return-Path: <common-issues-return-13383-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68905 invoked from network); 7 Mar 2011 05:56:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 05:56:36 -0000
Received: (qmail 1331 invoked by uid 500); 7 Mar 2011 05:56:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 918 invoked by uid 500); 7 Mar 2011 05:56:26 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 798 invoked by uid 99); 7 Mar 2011 05:56:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 05:56:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 05:56:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DE80739AB8B
	for <common-issues@hadoop.apache.org>; Mon,  7 Mar 2011 05:55:59 +0000 (UTC)
Date: Mon, 7 Mar 2011 05:55:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1015038040.391.1299477359906.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003271#comment-13003271 ] 

Todd Lipcon commented on HADOOP-7156:
-------------------------------------

bq. But for now, I'd rather just advertise "RHEL 6.0 is broken; don't use it" just like we do for JREs.

Unfortunately many of us are not in a position to do this - RHEL 6.0 is a must for us, regardless of some bugs it might have. Same with support for Vintela Authentication Services (VAS) which has a similar bug. Asking users to switch their entire OS or auth system is not an option.

I think there are three workable options here from my perspective:

1) Always lock around getpwuid_r. Devaraj is added a cache for this function as part of another JIRA, so it shouldn't be a big performance issue.
2) Add a compile-time macro like LOCK_AROUND_PWUID, which, when set, will add the monitor lock around these calls.
3) Add a runtime Hadoop configuration option like hadoop.workaround.broken.getpwuid, which when enabled adds the lock.

Which, if any, of these seem acceptable to you?

Since we've found that this isn't RHEL6 specific, but in fact occurs with other broken pieces of software as well, I'm leaning towards option #1 or #3.

> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13384-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 10:31:25 2011
Return-Path: <common-issues-return-13384-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47416 invoked from network); 7 Mar 2011 10:31:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 10:31:24 -0000
Received: (qmail 93484 invoked by uid 500); 7 Mar 2011 10:31:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93213 invoked by uid 500); 7 Mar 2011 10:31:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93196 invoked by uid 99); 7 Mar 2011 10:31:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 10:31:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 10:31:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A548D39AED8
	for <common-issues@hadoop.apache.org>; Mon,  7 Mar 2011 10:30:59 +0000 (UTC)
Date: Mon, 7 Mar 2011 10:30:59 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1199370355.677.1299493859673.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7164) rmr command is not displaying any
 error message when a path contains wildcard characters and does not exist.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003313#comment-13003313 ] 

Uma Maheswara Rao G commented on HADOOP-7164:
---------------------------------------------

Fix for this defect is in FsShell.java but the file present in COMMON project. Patch (HDFS-1609-src.patch) need to be updated in COMMON.

After applying this patch in Common, tests will pass.

> rmr command is not displaying any error message when a path contains wildcard characters and does not exist.
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7164
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7164
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Uma Maheswara Rao G
>            Priority: Minor
>         Attachments: HDFS-1609-src.patch, HDFS-1609-test.patch, HDFS-1609.patch
>
>
> When we give invalid directory path then it will show error message on the console. But if we provide the wildcard expression in invalid directory path then it will not show any error message even there is no pattern match for that path.
> linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin # ./hdfs dfs -rmr /test/test
> rmr: cannot remove /test/test: No such file or directory.
> *linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin # ./hdfs dfs -rmr /test* *
> *linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin #*

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13385-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 10:53:23 2011
Return-Path: <common-issues-return-13385-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29845 invoked from network); 7 Mar 2011 10:53:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 10:53:23 -0000
Received: (qmail 53186 invoked by uid 500); 7 Mar 2011 10:53:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53137 invoked by uid 500); 7 Mar 2011 10:53:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53075 invoked by uid 99); 7 Mar 2011 10:53:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 10:53:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 10:53:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7D97C39B971
	for <common-issues@hadoop.apache.org>; Mon,  7 Mar 2011 10:52:59 +0000 (UTC)
Date: Mon, 7 Mar 2011 10:52:59 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1531832144.757.1299495179510.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7164) rmr command is not displaying any
 error message when a path contains wildcard characters and does not exist.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003321#comment-13003321 ] 

Uma Maheswara Rao G commented on HADOOP-7164:
---------------------------------------------

Related Tests are  org.apache.hadoop.hdfs.TestDFSShell.testInvalidPathPatternForCatStatRmr 
org.apache.hadoop.hdfs.TestDFSShell.testInvalidPathPatternForStat  
  This both test cases will pass with HDFS-1609-src.patch.
This is bacause testCode and sourceCode are vailable in HDFS,COMMON respectively.

> rmr command is not displaying any error message when a path contains wildcard characters and does not exist.
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7164
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7164
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Uma Maheswara Rao G
>            Priority: Minor
>         Attachments: HDFS-1609-src.patch, HDFS-1609-test.patch, HDFS-1609.patch
>
>
> When we give invalid directory path then it will show error message on the console. But if we provide the wildcard expression in invalid directory path then it will not show any error message even there is no pattern match for that path.
> linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin # ./hdfs dfs -rmr /test/test
> rmr: cannot remove /test/test: No such file or directory.
> *linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin # ./hdfs dfs -rmr /test* *
> *linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin #*

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13386-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 13:31:22 2011
Return-Path: <common-issues-return-13386-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35953 invoked from network); 7 Mar 2011 13:31:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 13:31:21 -0000
Received: (qmail 27215 invoked by uid 500); 7 Mar 2011 13:31:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27183 invoked by uid 500); 7 Mar 2011 13:31:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27175 invoked by uid 99); 7 Mar 2011 13:31:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 13:31:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 13:31:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 34E5839BA70
	for <common-issues@hadoop.apache.org>; Mon,  7 Mar 2011 13:31:00 +0000 (UTC)
Date: Mon, 7 Mar 2011 13:31:00 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <776745505.908.1299504660213.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1846874015.7578.1296749909031.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7131) set() and toString Methods of the
 org.apache.hadoop.io.Text class does not include the root exception, in the
 wrapping RuntimeException.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003352#comment-13003352 ] 

Hudson commented on HADOOP-7131:
--------------------------------

Integrated in Hadoop-Common-trunk #623 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/623/])
    HADOOP-7131. Exceptions thrown by Text methods should include the causing exception. Contributed by Uma Maheswara Rao G.


> set() and toString Methods of the org.apache.hadoop.io.Text class does not include the root exception, in the wrapping RuntimeException.
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7131
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7131
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7131.patch
>
>
>  In below code snippets, we can include e, instead of e.toString(), so that caller can get complete trace.
> 1) 
>    /** Set to contain the contents of a string.
>    */
>   public void set(String string) {
>     try {
>       ByteBuffer bb = encode(string, true);
>       bytes = bb.array();
>       length = bb.limit();
>     }catch(CharacterCodingException e) {
>       throw new RuntimeException("Should not have happened ",e.toString());
>     }
>   } 
> 2)
>    public String toString() {
>     try {
>       return decode(bytes, 0, length);
>     } catch (CharacterCodingException e) {
>       throw new RuntimeException("Should not have happened ",e.toString());
>     }
>   }

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13387-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 13:31:24 2011
Return-Path: <common-issues-return-13387-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36239 invoked from network); 7 Mar 2011 13:31:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 13:31:24 -0000
Received: (qmail 27611 invoked by uid 500); 7 Mar 2011 13:31:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27576 invoked by uid 500); 7 Mar 2011 13:31:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27568 invoked by uid 99); 7 Mar 2011 13:31:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 13:31:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 13:31:22 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 26CDB39BA6E
	for <common-issues@hadoop.apache.org>; Mon,  7 Mar 2011 13:31:00 +0000 (UTC)
Date: Mon, 7 Mar 2011 13:31:00 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <645894245.906.1299504660155.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <824415802.6352.1299019897038.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7159) RPC server should log the client
 hostname when read exception happened
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003351#comment-13003351 ] 

Hudson commented on HADOOP-7159:
--------------------------------

Integrated in Hadoop-Common-trunk #623 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/623/])
    HADOOP-7159. RPC server should log the client hostname when read exception happened. Contributed by Scott Chen.


> RPC server should log the client hostname when read exception happened
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-7159
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7159
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7159.txt
>
>
> This makes find mismatched clients easier

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13388-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 16:54:24 2011
Return-Path: <common-issues-return-13388-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11452 invoked from network); 7 Mar 2011 16:54:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 16:54:24 -0000
Received: (qmail 77809 invoked by uid 500); 7 Mar 2011 16:54:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77758 invoked by uid 500); 7 Mar 2011 16:54:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77515 invoked by uid 99); 7 Mar 2011 16:54:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 16:54:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 16:54:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7667739BCC2
	for <common-issues@hadoop.apache.org>; Mon,  7 Mar 2011 16:53:59 +0000 (UTC)
Date: Mon, 7 Mar 2011 16:53:59 +0000 (UTC)
From: "Ravi Prakash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <971108340.1387.1299516839481.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-5094) Show dead nodes information in
 dfsadmin -report
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003427#comment-13003427 ] 

Ravi Prakash commented on HADOOP-5094:
--------------------------------------

I'm not sure what the expected behavior is when a node is specified in dfs.exclude and the cluster is started. Maybe this should never be done. But if it IS done. dfsadmin -report shows this

Live datanodes:
....
....
....

Dead datanodes:
report: String index out of range: -1

Is this fine?

The output is as expected when the cluster is started without any dfs.exclude entries, and then one added. (it shows it as a dead node with Decommission Status: Decommissioned) So that is good.

> Show dead nodes information in dfsadmin -report
> -----------------------------------------------
>
>                 Key: HADOOP-5094
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5094
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.18.2
>            Reporter: Jim Huang
>            Assignee: Jakob Homan
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: DfsAdminDeadNode_testCases.html, DfsAdminDeadNode_testCases.html, HADOOP-5094.patch, HADOOP-5094.patch, HADOOP-5094.patch
>
>
> As part of operations responsibility to bring back dead nodes, it will be good to have a quick way to obtain a list of dead data nodes.  
> The current way is to scrape the namenode web UI page and parse that information, but this creates load on the namenode.   
> In search of a less costly way, I noticed dfsadmin -report only reports data nodes with State: "In Service" and "Decommission in progress" get listed.
> Asking for a cheap way to obtain a list of dead nodes.  
> In addition, can the following requests be reviewed for additional enhancement and changes to dfsadmin -report.
> - Consistent formatting output in "Remaining raw bytes:" for the data nodes should have a space between the exact value and the parenthesized value.
> Sample:
> Total raw bytes: 3842232975360 (3.49 TB)
> Remaining raw bytes: 146090593065(136.06 GB)
> Used raw bytes: 3240864964620 (2.95 TB)
> - Include the running version of Hadoop.  
> - What is the meaning of "Total effective bytes"?
> - Display the hostname instead of the IP address for the data node (toggle option?)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13389-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 17:26:25 2011
Return-Path: <common-issues-return-13389-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13167 invoked from network); 7 Mar 2011 17:26:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 17:26:24 -0000
Received: (qmail 52008 invoked by uid 500); 7 Mar 2011 17:26:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51973 invoked by uid 500); 7 Mar 2011 17:26:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51964 invoked by uid 99); 7 Mar 2011 17:26:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 17:26:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 17:26:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 845AD39BAA9
	for <common-issues@hadoop.apache.org>; Mon,  7 Mar 2011 17:25:59 +0000 (UTC)
Date: Mon, 7 Mar 2011 17:25:59 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <6233149.1499.1299518759538.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003453#comment-13003453 ] 

Allen Wittenauer commented on HADOOP-7156:
------------------------------------------

If someone wants to use non-conforming software, that is their problem, not ours.  I'd much rather shame a company into fixing it than pretending that everything is a-ok.  Yes, I realize that is idealistic.

In the case of RHEL 6, it is what? 2 months old?  Like any .0, it is going to have growing pains during its first few months that should get fixed by the vendor.  As for VAS, I'm not really sure I care.

In the case of #1, what is the point of mutexes around a thread-safe function?  Why not just call the unsafe one and avoid the extra performance hit on machines that follow POSIX?

In the case of #3, I'll be OK with this if we also add a huge warning in the task log telling the user that their system isn't POSIX compliant and to call their vendor to get it fixed. :p



> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13390-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 18:15:23 2011
Return-Path: <common-issues-return-13390-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63048 invoked from network); 7 Mar 2011 18:15:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 18:15:23 -0000
Received: (qmail 87141 invoked by uid 500); 7 Mar 2011 18:15:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87107 invoked by uid 500); 7 Mar 2011 18:15:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87099 invoked by uid 99); 7 Mar 2011 18:15:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 18:15:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 18:15:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 85BCC39BECF
	for <common-issues@hadoop.apache.org>; Mon,  7 Mar 2011 18:14:59 +0000 (UTC)
Date: Mon, 7 Mar 2011 18:14:59 +0000 (UTC)
From: "Scott Chen (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <101757560.1642.1299521699543.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <824415802.6352.1299019897038.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7159) RPC server should log the client
 hostname when read exception happened
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003485#comment-13003485 ] 

Scott Chen commented on HADOOP-7159:
------------------------------------

Thanks for the help, Todd :)

> RPC server should log the client hostname when read exception happened
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-7159
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7159
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7159.txt
>
>
> This makes find mismatched clients easier

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13391-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 18:17:25 2011
Return-Path: <common-issues-return-13391-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68165 invoked from network); 7 Mar 2011 18:17:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 18:17:24 -0000
Received: (qmail 92736 invoked by uid 500); 7 Mar 2011 18:17:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92662 invoked by uid 500); 7 Mar 2011 18:17:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92553 invoked by uid 99); 7 Mar 2011 18:17:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 18:17:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 18:17:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C9F6539BFFD
	for <common-issues@hadoop.apache.org>; Mon,  7 Mar 2011 18:16:59 +0000 (UTC)
Date: Mon, 7 Mar 2011 18:16:59 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <136630983.1656.1299521819824.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7164) rmr command is not displaying any
 error message when a path contains wildcard characters and does not exist.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003487#comment-13003487 ] 

Allen Wittenauer commented on HADOOP-7164:
------------------------------------------

Would someone mind confirming that the error messages actually go to stderr?  They cannot go to stdout.

> rmr command is not displaying any error message when a path contains wildcard characters and does not exist.
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7164
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7164
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Uma Maheswara Rao G
>            Priority: Minor
>         Attachments: HDFS-1609-src.patch, HDFS-1609-test.patch, HDFS-1609.patch
>
>
> When we give invalid directory path then it will show error message on the console. But if we provide the wildcard expression in invalid directory path then it will not show any error message even there is no pattern match for that path.
> linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin # ./hdfs dfs -rmr /test/test
> rmr: cannot remove /test/test: No such file or directory.
> *linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin # ./hdfs dfs -rmr /test* *
> *linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin #*

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13392-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 19:34:24 2011
Return-Path: <common-issues-return-13392-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45580 invoked from network); 7 Mar 2011 19:34:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 19:34:23 -0000
Received: (qmail 85309 invoked by uid 500); 7 Mar 2011 19:34:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85278 invoked by uid 500); 7 Mar 2011 19:34:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85270 invoked by uid 99); 7 Mar 2011 19:34:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 19:34:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 19:34:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A2F6F39BEF6
	for <common-issues@hadoop.apache.org>; Mon,  7 Mar 2011 19:33:59 +0000 (UTC)
Date: Mon, 7 Mar 2011 19:33:59 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1779764816.1968.1299526439664.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-6949) Reduces RPC packet size for primitive
 arrays, especially long[], which is used at block reporting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated HADOOP-6949:
-------------------------------

    Status: Open  (was: Patch Available)

> Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6949
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6949
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Navis
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: ObjectWritable.diff, arrayprim.patch, arrayprim_v4.patch
>
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n
> It would not be needed to specify each element types for primitive arrays.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13393-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 19:34:24 2011
Return-Path: <common-issues-return-13393-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45716 invoked from network); 7 Mar 2011 19:34:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 19:34:24 -0000
Received: (qmail 85544 invoked by uid 500); 7 Mar 2011 19:34:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85503 invoked by uid 500); 7 Mar 2011 19:34:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85495 invoked by uid 99); 7 Mar 2011 19:34:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 19:34:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 19:34:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 291A139BF0A
	for <common-issues@hadoop.apache.org>; Mon,  7 Mar 2011 19:34:00 +0000 (UTC)
Date: Mon, 7 Mar 2011 19:34:00 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <103292134.1982.1299526440164.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-6949) Reduces RPC packet size for primitive
 arrays, especially long[], which is used at block reporting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated HADOOP-6949:
-------------------------------

    Attachment:     (was: arrayprim_v5.patch)

> Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6949
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6949
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Navis
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: ObjectWritable.diff, arrayprim.patch, arrayprim_v4.patch
>
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n
> It would not be needed to specify each element types for primitive arrays.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13394-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 19:34:25 2011
Return-Path: <common-issues-return-13394-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45800 invoked from network); 7 Mar 2011 19:34:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 19:34:24 -0000
Received: (qmail 85749 invoked by uid 500); 7 Mar 2011 19:34:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85719 invoked by uid 500); 7 Mar 2011 19:34:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85710 invoked by uid 99); 7 Mar 2011 19:34:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 19:34:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 19:34:22 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CFBDC39BF1C
	for <common-issues@hadoop.apache.org>; Mon,  7 Mar 2011 19:34:00 +0000 (UTC)
Date: Mon, 7 Mar 2011 19:34:00 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <923722278.1998.1299526440847.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-6949) Reduces RPC packet size for primitive
 arrays, especially long[], which is used at block reporting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated HADOOP-6949:
-------------------------------

    Attachment:     (was: arrayprim_v5.patch)

> Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6949
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6949
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Navis
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: ObjectWritable.diff, arrayprim.patch, arrayprim_v4.patch
>
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n
> It would not be needed to specify each element types for primitive arrays.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13395-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 19:36:24 2011
Return-Path: <common-issues-return-13395-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56400 invoked from network); 7 Mar 2011 19:36:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 19:36:24 -0000
Received: (qmail 91751 invoked by uid 500); 7 Mar 2011 19:36:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91696 invoked by uid 500); 7 Mar 2011 19:36:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91682 invoked by uid 99); 7 Mar 2011 19:36:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 19:36:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 19:36:22 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4F4A639B11E
	for <common-issues@hadoop.apache.org>; Mon,  7 Mar 2011 19:36:00 +0000 (UTC)
Date: Mon, 7 Mar 2011 19:36:00 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1432206626.2043.1299526560321.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-6949) Reduces RPC packet size for primitive
 arrays, especially long[], which is used at block reporting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated HADOOP-6949:
-------------------------------

    Attachment: arrayprim_v5.patch

One more time, trying to get auto-test to run.

> Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6949
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6949
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Navis
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: ObjectWritable.diff, arrayprim.patch, arrayprim_v4.patch, arrayprim_v5.patch
>
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n
> It would not be needed to specify each element types for primitive arrays.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13396-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 19:38:26 2011
Return-Path: <common-issues-return-13396-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56855 invoked from network); 7 Mar 2011 19:38:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 19:38:26 -0000
Received: (qmail 97798 invoked by uid 500); 7 Mar 2011 19:38:25 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97728 invoked by uid 500); 7 Mar 2011 19:38:25 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97711 invoked by uid 99); 7 Mar 2011 19:38:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 19:38:25 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 19:38:22 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E73EA39B383
	for <common-issues@hadoop.apache.org>; Mon,  7 Mar 2011 19:38:00 +0000 (UTC)
Date: Mon, 7 Mar 2011 19:38:00 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <189309007.2072.1299526680943.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-6949) Reduces RPC packet size for primitive
 arrays, especially long[], which is used at block reporting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated HADOOP-6949:
-------------------------------

    Affects Version/s: 0.22.0
         Release Note: 
Increments the RPC protocol version in org.apache.hadoop.ipc.Server from 4 to 5.
Introduces a much more efficient wire format to transmit arrays of primitives.  This is done in a way that is 100% backward compatible with previously persisted data, so any stored data will still be readable.  However, data written in the new version cannot be read by a process using the old version of protocol.  Only RPC uses the new format; file writing protocols for application data are unchanged..

  was:
Increments the RPC protocol version in org.apache.hadoop.ipc.Server from 4 to 5.
Introduces a much more efficient wire format to transmit arrays of primitives.  This is done in a way that is 100% backward compatible with previously persisted data, so any stored data will still be readable.  However, data written in the new version cannot be read by a process using the old version of protocol.  Only RPC uses the new format; file writing protocols for application data are unchanged.

               Status: Patch Available  (was: Open)

> Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6949
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6949
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Navis
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: ObjectWritable.diff, arrayprim.patch, arrayprim_v4.patch, arrayprim_v5.patch
>
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n
> It would not be needed to specify each element types for primitive arrays.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13397-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 19:44:26 2011
Return-Path: <common-issues-return-13397-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59979 invoked from network); 7 Mar 2011 19:44:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 19:44:21 -0000
Received: (qmail 18114 invoked by uid 500); 7 Mar 2011 19:44:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18053 invoked by uid 500); 7 Mar 2011 19:44:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17914 invoked by uid 99); 7 Mar 2011 19:44:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 19:44:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 19:44:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B541539BB93
	for <common-issues@hadoop.apache.org>; Mon,  7 Mar 2011 19:43:59 +0000 (UTC)
Date: Mon, 7 Mar 2011 19:43:59 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <573912296.2117.1299527039739.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003537#comment-13003537 ] 

Eric Yang commented on HADOOP-6255:
-----------------------------------

include/hdfs.h is the proper location, if hdfs.h is a public api that dependent project (fuse-dfs) can interface.  I don't know the right answer, but include/hdfs.h works for now.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.100
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.100
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13398-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 21:57:26 2011
Return-Path: <common-issues-return-13398-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2949 invoked from network); 7 Mar 2011 21:57:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 21:57:25 -0000
Received: (qmail 83834 invoked by uid 500); 7 Mar 2011 21:57:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83781 invoked by uid 500); 7 Mar 2011 21:57:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83599 invoked by uid 99); 7 Mar 2011 21:57:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 21:57:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 21:57:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E0B9539C87F
	for <common-issues@hadoop.apache.org>; Mon,  7 Mar 2011 21:56:59 +0000 (UTC)
Date: Mon, 7 Mar 2011 21:56:59 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <645678321.2624.1299535019917.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7166) DaemonFactory should be moved from
 HDFS to common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

DaemonFactory should be moved from HDFS to common
-------------------------------------------------

                 Key: HADOOP-7166
                 URL: https://issues.apache.org/jira/browse/HADOOP-7166
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Jitendra Nath Pandey
            Assignee: Jitendra Nath Pandey


DaemonFactory class is defined in hdfs util. common would be a better place for this class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13399-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 22:11:24 2011
Return-Path: <common-issues-return-13399-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58262 invoked from network); 7 Mar 2011 22:11:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 22:11:23 -0000
Received: (qmail 26270 invoked by uid 500); 7 Mar 2011 22:11:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26246 invoked by uid 500); 7 Mar 2011 22:11:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26210 invoked by uid 99); 7 Mar 2011 22:11:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 22:11:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 22:11:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8624639CE2E
	for <common-issues@hadoop.apache.org>; Mon,  7 Mar 2011 22:10:59 +0000 (UTC)
Date: Mon, 7 Mar 2011 22:10:59 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1640927942.2664.1299535859546.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <645678321.2624.1299535019917.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7166) DaemonFactory should be moved from
 HDFS to common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7166:
-----------------------------------------

    Attachment: HADOOP-7166.1.patch

> DaemonFactory should be moved from HDFS to common
> -------------------------------------------------
>
>                 Key: HADOOP-7166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7166
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-7166.1.patch
>
>
> DaemonFactory class is defined in hdfs util. common would be a better place for this class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13400-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 22:13:21 2011
Return-Path: <common-issues-return-13400-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58948 invoked from network); 7 Mar 2011 22:13:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 22:13:21 -0000
Received: (qmail 29710 invoked by uid 500); 7 Mar 2011 22:13:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29676 invoked by uid 500); 7 Mar 2011 22:13:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29668 invoked by uid 99); 7 Mar 2011 22:13:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 22:13:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 22:13:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6C67839CF06
	for <common-issues@hadoop.apache.org>; Mon,  7 Mar 2011 22:12:59 +0000 (UTC)
Date: Mon, 7 Mar 2011 22:12:59 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1264348902.2667.1299535979440.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <645678321.2624.1299535019917.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7166) DaemonFactory should be moved from
 HDFS to common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7166:
-----------------------------------------

    Status: Patch Available  (was: Open)

Patch submitted for hudson tests.

> DaemonFactory should be moved from HDFS to common
> -------------------------------------------------
>
>                 Key: HADOOP-7166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7166
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-7166.1.patch
>
>
> DaemonFactory class is defined in hdfs util. common would be a better place for this class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13401-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 22:24:28 2011
Return-Path: <common-issues-return-13401-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93209 invoked from network); 7 Mar 2011 22:24:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 22:24:27 -0000
Received: (qmail 60818 invoked by uid 500); 7 Mar 2011 22:24:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60781 invoked by uid 500); 7 Mar 2011 22:24:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60740 invoked by uid 99); 7 Mar 2011 22:24:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 22:24:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 22:24:25 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4C68039C426
	for <common-issues@hadoop.apache.org>; Mon,  7 Mar 2011 22:24:03 +0000 (UTC)
Date: Mon, 7 Mar 2011 22:24:03 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <389161031.2799.1299536643309.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <645678321.2624.1299535019917.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7166) DaemonFactory should be moved from
 HDFS to common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003656#comment-13003656 ] 

Tom White commented on HADOOP-7166:
-----------------------------------

Is the motivation for this for MapReduce to use DaemonFactory?

> DaemonFactory should be moved from HDFS to common
> -------------------------------------------------
>
>                 Key: HADOOP-7166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7166
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-7166.1.patch
>
>
> DaemonFactory class is defined in hdfs util. common would be a better place for this class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13402-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 22:42:22 2011
Return-Path: <common-issues-return-13402-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51521 invoked from network); 7 Mar 2011 22:42:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 22:42:22 -0000
Received: (qmail 3291 invoked by uid 500); 7 Mar 2011 22:42:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3242 invoked by uid 500); 7 Mar 2011 22:42:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3035 invoked by uid 99); 7 Mar 2011 22:42:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 22:42:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 22:42:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6F82139CA91
	for <common-issues@hadoop.apache.org>; Mon,  7 Mar 2011 22:41:59 +0000 (UTC)
Date: Mon, 7 Mar 2011 22:41:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <599378200.2869.1299537719453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7167) Allow using a file to exclude certain
 tests from build
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Allow using a file to exclude certain tests from build
------------------------------------------------------

                 Key: HADOOP-7167
                 URL: https://issues.apache.org/jira/browse/HADOOP-7167
             Project: Hadoop Common
          Issue Type: Improvement
    Affects Versions: 0.23.0
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
            Priority: Minor
             Fix For: 0.23.0


It would be nice to be able to exclude certain tests when running builds. For example, when a test is "known flaky", you may want to exclude it from the main Hudson job, but not actually disable it in the codebase (so that it still runs as part of another Hudson job, for example).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13403-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 22:44:21 2011
Return-Path: <common-issues-return-13403-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51859 invoked from network); 7 Mar 2011 22:44:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 22:44:21 -0000
Received: (qmail 7412 invoked by uid 500); 7 Mar 2011 22:44:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7330 invoked by uid 500); 7 Mar 2011 22:44:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7322 invoked by uid 99); 7 Mar 2011 22:44:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 22:44:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 22:44:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 778A039CB24
	for <common-issues@hadoop.apache.org>; Mon,  7 Mar 2011 22:43:59 +0000 (UTC)
Date: Mon, 7 Mar 2011 22:43:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <61713221.2878.1299537839486.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <599378200.2869.1299537719453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7167) Allow using a file to exclude certain
 tests from build
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7167:
--------------------------------

    Status: Patch Available  (was: Open)

> Allow using a file to exclude certain tests from build
> ------------------------------------------------------
>
>                 Key: HADOOP-7167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7167
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7167.txt
>
>
> It would be nice to be able to exclude certain tests when running builds. For example, when a test is "known flaky", you may want to exclude it from the main Hudson job, but not actually disable it in the codebase (so that it still runs as part of another Hudson job, for example).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13404-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 22:44:23 2011
Return-Path: <common-issues-return-13404-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51903 invoked from network); 7 Mar 2011 22:44:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 22:44:23 -0000
Received: (qmail 7639 invoked by uid 500); 7 Mar 2011 22:44:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7603 invoked by uid 500); 7 Mar 2011 22:44:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7595 invoked by uid 99); 7 Mar 2011 22:44:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 22:44:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 22:44:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6926F39CB22
	for <common-issues@hadoop.apache.org>; Mon,  7 Mar 2011 22:43:59 +0000 (UTC)
Date: Mon, 7 Mar 2011 22:43:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1501282611.2876.1299537839427.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <599378200.2869.1299537719453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7167) Allow using a file to exclude certain
 tests from build
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7167:
--------------------------------

    Attachment: hadoop-7167.txt

> Allow using a file to exclude certain tests from build
> ------------------------------------------------------
>
>                 Key: HADOOP-7167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7167
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7167.txt
>
>
> It would be nice to be able to exclude certain tests when running builds. For example, when a test is "known flaky", you may want to exclude it from the main Hudson job, but not actually disable it in the codebase (so that it still runs as part of another Hudson job, for example).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13405-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 22:49:26 2011
Return-Path: <common-issues-return-13405-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88079 invoked from network); 7 Mar 2011 22:49:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 22:49:25 -0000
Received: (qmail 15806 invoked by uid 500); 7 Mar 2011 22:49:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15743 invoked by uid 500); 7 Mar 2011 22:49:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15561 invoked by uid 99); 7 Mar 2011 22:49:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 22:49:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 22:49:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9789039CC9B
	for <common-issues@hadoop.apache.org>; Mon,  7 Mar 2011 22:48:59 +0000 (UTC)
Date: Mon, 7 Mar 2011 22:48:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1347010045.2884.1299538139617.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7168) Allow using a file to exclude certain
 tests from build
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Allow using a file to exclude certain tests from build
------------------------------------------------------

                 Key: HADOOP-7168
                 URL: https://issues.apache.org/jira/browse/HADOOP-7168
             Project: Hadoop Common
          Issue Type: Improvement
    Affects Versions: 0.23.0
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
            Priority: Minor
             Fix For: 0.23.0
         Attachments: mapreduce-2367.txt

It would be nice to be able to exclude certain tests when running builds. For example, when a test is "known flaky", you may want to exclude it from the main Hudson job, but not actually disable it in the codebase (so that it still runs as part of another Hudson job, for example).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13406-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 22:51:21 2011
Return-Path: <common-issues-return-13406-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95309 invoked from network); 7 Mar 2011 22:51:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 22:51:21 -0000
Received: (qmail 19689 invoked by uid 500); 7 Mar 2011 22:51:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19649 invoked by uid 500); 7 Mar 2011 22:51:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19641 invoked by uid 99); 7 Mar 2011 22:51:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 22:51:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 22:51:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7E6CD39CDAF
	for <common-issues@hadoop.apache.org>; Mon,  7 Mar 2011 22:50:59 +0000 (UTC)
Date: Mon, 7 Mar 2011 22:50:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2111858644.2892.1299538259514.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7169) Allow using a file to exclude certain
 tests from build
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Allow using a file to exclude certain tests from build
------------------------------------------------------

                 Key: HADOOP-7169
                 URL: https://issues.apache.org/jira/browse/HADOOP-7169
             Project: Hadoop Common
          Issue Type: Improvement
    Affects Versions: 0.23.0
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
            Priority: Minor
             Fix For: 0.23.0


It would be nice to be able to exclude certain tests when running builds. For example, when a test is "known flaky", you may want to exclude it from the main Hudson job, but not actually disable it in the codebase (so that it still runs as part of another Hudson job, for example).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13407-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 07 23:47:22 2011
Return-Path: <common-issues-return-13407-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65891 invoked from network); 7 Mar 2011 23:47:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Mar 2011 23:47:21 -0000
Received: (qmail 54902 invoked by uid 500); 7 Mar 2011 23:47:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54771 invoked by uid 500); 7 Mar 2011 23:47:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54627 invoked by uid 99); 7 Mar 2011 23:47:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 23:47:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Mar 2011 23:47:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7A11839CD62
	for <common-issues@hadoop.apache.org>; Mon,  7 Mar 2011 23:46:59 +0000 (UTC)
Date: Mon, 7 Mar 2011 23:46:59 +0000 (UTC)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1044695845.3019.1299541619496.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <599378200.2869.1299537719453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7167) Allow using a file to exclude
 certain tests from build
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003694#comment-13003694 ] 

Konstantin Boudnik commented on HADOOP-7167:
--------------------------------------------

+1 patch looks good. With only exception that we shouldn't overuse this 'facility'

> Allow using a file to exclude certain tests from build
> ------------------------------------------------------
>
>                 Key: HADOOP-7167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7167
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7167.txt
>
>
> It would be nice to be able to exclude certain tests when running builds. For example, when a test is "known flaky", you may want to exclude it from the main Hudson job, but not actually disable it in the codebase (so that it still runs as part of another Hudson job, for example).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13408-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 00:03:21 2011
Return-Path: <common-issues-return-13408-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15850 invoked from network); 8 Mar 2011 00:03:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 00:03:21 -0000
Received: (qmail 91242 invoked by uid 500); 8 Mar 2011 00:03:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91211 invoked by uid 500); 8 Mar 2011 00:03:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91203 invoked by uid 99); 8 Mar 2011 00:03:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 00:03:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 00:03:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 80B6A39C3B7
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 00:02:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 00:02:59 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2142003750.3073.1299542579523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-6912) Guard against NPE when calling
 UGI.isLoginKeytabBased()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6912:
-----------------------------------------

    Status: Patch Available  (was: Open)

> Guard against NPE when calling UGI.isLoginKeytabBased()
> -------------------------------------------------------
>
>                 Key: HADOOP-6912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6912
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6912-01.patch
>
>
> NPE can happen when isLoginKeytabBased() is called before a login is performed. See MAPREDUCE-1992 for an example.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13409-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 00:03:26 2011
Return-Path: <common-issues-return-13409-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16655 invoked from network); 8 Mar 2011 00:03:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 00:03:26 -0000
Received: (qmail 92554 invoked by uid 500); 8 Mar 2011 00:03:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92516 invoked by uid 500); 8 Mar 2011 00:03:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92485 invoked by uid 99); 8 Mar 2011 00:03:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 00:03:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 00:03:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 73B8339C3B4
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 00:02:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 00:02:59 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <628869758.3071.1299542579470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-6912) Guard against NPE when calling
 UGI.isLoginKeytabBased()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6912:
-----------------------------------------

    Status: Open  (was: Patch Available)

> Guard against NPE when calling UGI.isLoginKeytabBased()
> -------------------------------------------------------
>
>                 Key: HADOOP-6912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6912
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6912-01.patch
>
>
> NPE can happen when isLoginKeytabBased() is called before a login is performed. See MAPREDUCE-1992 for an example.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13410-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 00:09:21 2011
Return-Path: <common-issues-return-13410-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46390 invoked from network); 8 Mar 2011 00:09:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 00:09:21 -0000
Received: (qmail 9637 invoked by uid 500); 8 Mar 2011 00:09:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9606 invoked by uid 500); 8 Mar 2011 00:09:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9596 invoked by uid 99); 8 Mar 2011 00:09:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 00:09:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 00:09:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A53D039C534
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 00:08:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 00:08:59 +0000 (UTC)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <438061063.3090.1299542939673.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-6912) Guard against NPE when calling
 UGI.isLoginKeytabBased()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003710#comment-13003710 ] 

Kan Zhang commented on HADOOP-6912:
-----------------------------------

This patch is simple enough that eyeballing should provide confidence. However, one should check again whether the HDFS or MR builds break since the signature of a public method is changing.

> Guard against NPE when calling UGI.isLoginKeytabBased()
> -------------------------------------------------------
>
>                 Key: HADOOP-6912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6912
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6912-01.patch
>
>
> NPE can happen when isLoginKeytabBased() is called before a login is performed. See MAPREDUCE-1992 for an example.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13411-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 00:31:24 2011
Return-Path: <common-issues-return-13411-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9899 invoked from network); 8 Mar 2011 00:31:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 00:31:23 -0000
Received: (qmail 70827 invoked by uid 500); 8 Mar 2011 00:31:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70759 invoked by uid 500); 8 Mar 2011 00:31:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70527 invoked by uid 99); 8 Mar 2011 00:31:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 00:31:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 00:31:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7B01339CCB3
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 00:30:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 00:30:59 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <495334931.3140.1299544259500.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-6912) Guard against NPE when calling
 UGI.isLoginKeytabBased()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003720#comment-13003720 ] 

Jitendra Nath Pandey commented on HADOOP-6912:
----------------------------------------------

+1 for the patch.

> Guard against NPE when calling UGI.isLoginKeytabBased()
> -------------------------------------------------------
>
>                 Key: HADOOP-6912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6912
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6912-01.patch
>
>
> NPE can happen when isLoginKeytabBased() is called before a login is performed. See MAPREDUCE-1992 for an example.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13412-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 00:51:24 2011
Return-Path: <common-issues-return-13412-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97112 invoked from network); 8 Mar 2011 00:51:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 00:51:23 -0000
Received: (qmail 14609 invoked by uid 500); 8 Mar 2011 00:51:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14326 invoked by uid 500); 8 Mar 2011 00:51:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14310 invoked by uid 99); 8 Mar 2011 00:51:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 00:51:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 00:51:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7287039C450
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 00:51:00 +0000 (UTC)
Date: Tue, 8 Mar 2011 00:51:00 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <545918969.3211.1299545460466.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7170) Support UGI in FileContext API
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Support UGI in FileContext API
------------------------------

                 Key: HADOOP-7170
                 URL: https://issues.apache.org/jira/browse/HADOOP-7170
             Project: Hadoop Common
          Issue Type: Bug
          Components: security
            Reporter: Owen O'Malley
            Assignee: Jitendra Nath Pandey
             Fix For: 0.23.0


The FileContext API needs to support UGI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13413-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 00:51:27 2011
Return-Path: <common-issues-return-13413-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97365 invoked from network); 8 Mar 2011 00:51:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 00:51:26 -0000
Received: (qmail 16898 invoked by uid 500); 8 Mar 2011 00:51:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16836 invoked by uid 500); 8 Mar 2011 00:51:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16777 invoked by uid 99); 8 Mar 2011 00:51:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 00:51:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 00:51:22 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8839539C453
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 00:51:00 +0000 (UTC)
Date: Tue, 8 Mar 2011 00:51:00 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <995627871.3214.1299545460554.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7171) Support UGI in FileContext API
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Support UGI in FileContext API
------------------------------

                 Key: HADOOP-7171
                 URL: https://issues.apache.org/jira/browse/HADOOP-7171
             Project: Hadoop Common
          Issue Type: Bug
          Components: security
            Reporter: Owen O'Malley
            Assignee: Jitendra Nath Pandey
             Fix For: 0.23.0


The FileContext API needs to support UGI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13414-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 01:22:23 2011
Return-Path: <common-issues-return-13414-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85333 invoked from network); 8 Mar 2011 01:22:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 01:22:23 -0000
Received: (qmail 95808 invoked by uid 500); 8 Mar 2011 01:22:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95775 invoked by uid 500); 8 Mar 2011 01:22:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95763 invoked by uid 99); 8 Mar 2011 01:22:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 01:22:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 01:22:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6CB4B39CE9A
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 01:21:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 01:21:59 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1976075950.3326.1299547319442.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-6912) Guard against NPE when calling
 UGI.isLoginKeytabBased()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6912:
-----------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I have committed this. Thanks to Kan.

> Guard against NPE when calling UGI.isLoginKeytabBased()
> -------------------------------------------------------
>
>                 Key: HADOOP-6912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6912
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>             Fix For: 0.23.0
>
>         Attachments: c6912-01.patch
>
>
> NPE can happen when isLoginKeytabBased() is called before a login is performed. See MAPREDUCE-1992 for an example.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13415-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 01:22:23 2011
Return-Path: <common-issues-return-13415-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85368 invoked from network); 8 Mar 2011 01:22:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 01:22:23 -0000
Received: (qmail 96085 invoked by uid 500); 8 Mar 2011 01:22:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95984 invoked by uid 500); 8 Mar 2011 01:22:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95878 invoked by uid 99); 8 Mar 2011 01:22:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 01:22:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 01:22:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7C85439CE9D
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 01:21:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 01:21:59 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1430022177.3328.1299547319506.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-6912) Guard against NPE when calling
 UGI.isLoginKeytabBased()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003754#comment-13003754 ] 

Jitendra Nath Pandey commented on HADOOP-6912:
----------------------------------------------

> one should check again whether the HDFS or MR builds break since the signature of a public method is changing.
This method is not being used in HDFS or MR in trunk.

> Guard against NPE when calling UGI.isLoginKeytabBased()
> -------------------------------------------------------
>
>                 Key: HADOOP-6912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6912
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>             Fix For: 0.23.0
>
>         Attachments: c6912-01.patch
>
>
> NPE can happen when isLoginKeytabBased() is called before a login is performed. See MAPREDUCE-1992 for an example.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13416-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 01:24:24 2011
Return-Path: <common-issues-return-13416-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95643 invoked from network); 8 Mar 2011 01:24:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 01:24:24 -0000
Received: (qmail 2128 invoked by uid 500); 8 Mar 2011 01:24:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2092 invoked by uid 500); 8 Mar 2011 01:24:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2084 invoked by uid 99); 8 Mar 2011 01:24:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 01:24:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 01:24:22 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5FBA139CF6E
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 01:24:00 +0000 (UTC)
Date: Tue, 8 Mar 2011 01:24:00 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <178111327.3354.1299547440388.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-6949) Reduces RPC packet size for
 primitive arrays, especially long[], which is used at block reporting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003756#comment-13003756 ] 

Hadoop QA commented on HADOOP-6949:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12472850/arrayprim_v5.patch
  against trunk revision 1078669.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 2 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The applied patch generated 1078 javac compiler warnings (more than the trunk's current 1072 warnings).

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/303//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/303//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/303//console

This message is automatically generated.

> Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6949
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6949
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Navis
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: ObjectWritable.diff, arrayprim.patch, arrayprim_v4.patch, arrayprim_v5.patch
>
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n
> It would not be needed to specify each element types for primitive arrays.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13417-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 01:34:24 2011
Return-Path: <common-issues-return-13417-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34091 invoked from network); 8 Mar 2011 01:34:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 01:34:23 -0000
Received: (qmail 22425 invoked by uid 500); 8 Mar 2011 01:34:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22387 invoked by uid 500); 8 Mar 2011 01:34:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22349 invoked by uid 99); 8 Mar 2011 01:34:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 01:34:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 01:34:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C819D39C239
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 01:34:00 +0000 (UTC)
Date: Tue, 8 Mar 2011 01:34:00 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <266082047.3375.1299548040816.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <599378200.2869.1299537719453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7167) Allow using a file to exclude certain
 tests from build
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7167:
--------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

> Allow using a file to exclude certain tests from build
> ------------------------------------------------------
>
>                 Key: HADOOP-7167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7167
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7167.txt
>
>
> It would be nice to be able to exclude certain tests when running builds. For example, when a test is "known flaky", you may want to exclude it from the main Hudson job, but not actually disable it in the codebase (so that it still runs as part of another Hudson job, for example).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13419-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 02:23:21 2011
Return-Path: <common-issues-return-13419-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82874 invoked from network); 8 Mar 2011 02:23:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 02:23:21 -0000
Received: (qmail 39353 invoked by uid 500); 8 Mar 2011 02:23:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39267 invoked by uid 500); 8 Mar 2011 02:23:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39059 invoked by uid 99); 8 Mar 2011 02:23:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 02:23:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 02:23:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B634339C224
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 02:22:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 02:22:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1539840297.3481.1299550979743.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <130797695.15062.1298594983152.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7154) Should set MALLOC_ARENA_MAX in
 hadoop-env.sh
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7154:
--------------------------------

    Status: Patch Available  (was: Open)

> Should set MALLOC_ARENA_MAX in hadoop-env.sh
> --------------------------------------------
>
>                 Key: HADOOP-7154
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7154
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7154.txt
>
>
> New versions of glibc present in RHEL6 include a new arena allocator design. In several clusters we've seen this new allocator cause huge amounts of virtual memory to be used, since when multiple threads perform allocations, they each get their own memory arena. On a 64-bit system, these arenas are 64M mappings, and the maximum number of arenas is 8 times the number of cores. We've observed a DN process using 14GB of vmem for only 300M of resident set. This causes all kinds of nasty issues for obvious reasons.
> Setting MALLOC_ARENA_MAX to a low number will restrict the number of memory arenas and bound the virtual memory, with no noticeable downside in performance - we've been recommending MALLOC_ARENA_MAX=4. We should set this in hadoop-env.sh to avoid this issue as RHEL6 becomes more and more common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13418-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 02:23:21 2011
Return-Path: <common-issues-return-13418-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82856 invoked from network); 8 Mar 2011 02:23:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 02:23:21 -0000
Received: (qmail 39081 invoked by uid 500); 8 Mar 2011 02:23:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39050 invoked by uid 500); 8 Mar 2011 02:23:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39042 invoked by uid 99); 8 Mar 2011 02:23:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 02:23:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 02:23:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A9A2239C221
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 02:22:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 02:22:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <656553326.3479.1299550979691.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <130797695.15062.1298594983152.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7154) Should set MALLOC_ARENA_MAX in
 hadoop-env.sh
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7154:
--------------------------------

    Attachment: hadoop-7154.txt

Attached patch only does the MALLOC_ARENA_MAX bit. The ulimit fix is tougher since it appears to not be very portable.

> Should set MALLOC_ARENA_MAX in hadoop-env.sh
> --------------------------------------------
>
>                 Key: HADOOP-7154
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7154
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7154.txt
>
>
> New versions of glibc present in RHEL6 include a new arena allocator design. In several clusters we've seen this new allocator cause huge amounts of virtual memory to be used, since when multiple threads perform allocations, they each get their own memory arena. On a 64-bit system, these arenas are 64M mappings, and the maximum number of arenas is 8 times the number of cores. We've observed a DN process using 14GB of vmem for only 300M of resident set. This causes all kinds of nasty issues for obvious reasons.
> Setting MALLOC_ARENA_MAX to a low number will restrict the number of memory arenas and bound the virtual memory, with no noticeable downside in performance - we've been recommending MALLOC_ARENA_MAX=4. We should set this in hadoop-env.sh to avoid this issue as RHEL6 becomes more and more common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13420-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 02:25:23 2011
Return-Path: <common-issues-return-13420-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91053 invoked from network); 8 Mar 2011 02:25:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 02:25:23 -0000
Received: (qmail 43459 invoked by uid 500); 8 Mar 2011 02:25:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43416 invoked by uid 500); 8 Mar 2011 02:25:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43407 invoked by uid 99); 8 Mar 2011 02:25:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 02:25:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 02:25:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6BEB239C2D6
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 02:24:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 02:24:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1819943212.3491.1299551099438.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003787#comment-13003787 ] 

Todd Lipcon commented on HADOOP-7156:
-------------------------------------

I'll assume you're joking about the "huge warning" and implement option #3. Thanks Allen.

> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13421-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 02:55:22 2011
Return-Path: <common-issues-return-13421-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96200 invoked from network); 8 Mar 2011 02:55:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 02:55:22 -0000
Received: (qmail 1439 invoked by uid 500); 8 Mar 2011 02:55:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1266 invoked by uid 500); 8 Mar 2011 02:55:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1079 invoked by uid 99); 8 Mar 2011 02:55:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 02:55:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 02:55:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 95EF439C995
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 02:54:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 02:54:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <951842791.3522.1299552899611.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7156:
--------------------------------

    Attachment: hadoop-7156.txt

Here's a patch which implements the configuration option.

I did not add it to core-default since it's a corner case... does that seem reasonable or should it be documented?

> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt, hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13422-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 02:55:24 2011
Return-Path: <common-issues-return-13422-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96281 invoked from network); 8 Mar 2011 02:55:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 02:55:24 -0000
Received: (qmail 2079 invoked by uid 500); 8 Mar 2011 02:55:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2049 invoked by uid 500); 8 Mar 2011 02:55:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2039 invoked by uid 99); 8 Mar 2011 02:55:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 02:55:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 02:55:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C56A739C99C
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 02:54:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 02:54:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1716097838.3529.1299552899805.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7156:
--------------------------------

    Status: Patch Available  (was: Open)

> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt, hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13423-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 03:27:22 2011
Return-Path: <common-issues-return-13423-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98449 invoked from network); 8 Mar 2011 03:27:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 03:27:21 -0000
Received: (qmail 60606 invoked by uid 500); 8 Mar 2011 03:27:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60580 invoked by uid 500); 8 Mar 2011 03:27:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60563 invoked by uid 99); 8 Mar 2011 03:27:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 03:27:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 03:27:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 85AFA39C0D8
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 03:26:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 03:26:59 +0000 (UTC)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <149314971.3579.1299554819544.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003796#comment-13003796 ] 

Greg Roelofs commented on HADOOP-7156:
--------------------------------------

bq. I did not add it to core-default since it's a corner case... does that seem reasonable or should it be documented?

Please please please...always document things like this. Picture it from the perspective of some new user who hasn't discovered the lists yet (or hasn't been subscribed for very long, and then only on the -user list(s)): they know they're running RHEL-6 and probably will notice a config item that explicitly mentions it (which I'd recommend, along with CentOS and any other known distro versions), but they're much less likely to know what's up with a random glibc crash several months later.

> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt, hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13424-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 06:11:22 2011
Return-Path: <common-issues-return-13424-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38568 invoked from network); 8 Mar 2011 06:11:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 06:11:22 -0000
Received: (qmail 2922 invoked by uid 500); 8 Mar 2011 06:11:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2873 invoked by uid 500); 8 Mar 2011 06:11:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2854 invoked by uid 99); 8 Mar 2011 06:11:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 06:11:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 06:11:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7671439C76D
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 06:10:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 06:10:59 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1357250991.3750.1299564659481.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-6949) Reduces RPC packet size for primitive
 arrays, especially long[], which is used at block reporting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated HADOOP-6949:
-------------------------------

    Attachment: arrayprim_v6.patch

Added @SuppressWarnings("deprecation") annotations to 6 uses of UTF8 class methods (which usages are consistent with usages in ObjectWritable.java).

> Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6949
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6949
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Navis
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: ObjectWritable.diff, arrayprim.patch, arrayprim_v4.patch, arrayprim_v5.patch, arrayprim_v6.patch
>
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n
> It would not be needed to specify each element types for primitive arrays.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13426-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 08:11:43 2011
Return-Path: <common-issues-return-13426-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14456 invoked from network); 8 Mar 2011 08:11:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 08:11:43 -0000
Received: (qmail 95032 invoked by uid 500); 8 Mar 2011 08:11:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94677 invoked by uid 500); 8 Mar 2011 08:11:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94471 invoked by uid 99); 8 Mar 2011 08:11:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 08:11:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 07:35:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 899A739C1EF
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 07:34:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 07:34:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2091456777.3848.1299569699560.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1799619354.3818.1299567419614.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7172) SecureIO should not check owner on
 non-secure clusters that have no native support
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7172:
--------------------------------

    Attachment: hadoop-7172.txt

Here's a patch, but it will end up conflicting once 7115 is committed.

> SecureIO should not check owner on non-secure clusters that have no native support
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7172
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7172
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io, security
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7172.txt
>
>
> The SecureIOUtils.openForRead function currently uses a racy stat/open combo if security is disabled and the native libraries are not available. This ends up shelling out to "ls -ld" which is very very slow. We've seen this cause significant performance regressions on clusters that match this profile.
> Since the racy permissions check doesn't buy us any security anyway, we should just fall back to a normal "open" without any stat() at all, if we can't use the native support to do it efficiently.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13425-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 08:11:48 2011
Return-Path: <common-issues-return-13425-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14492 invoked from network); 8 Mar 2011 08:11:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 08:11:46 -0000
Received: (qmail 94822 invoked by uid 500); 8 Mar 2011 08:11:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94621 invoked by uid 500); 8 Mar 2011 08:11:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94364 invoked by uid 99); 8 Mar 2011 08:11:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 08:11:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 07:35:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9720B39C1F2
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 07:34:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 07:34:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <952656567.3850.1299569699615.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1799619354.3818.1299567419614.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7172) SecureIO should not check owner on
 non-secure clusters that have no native support
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7172:
--------------------------------

    Status: Patch Available  (was: Open)

> SecureIO should not check owner on non-secure clusters that have no native support
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7172
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7172
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io, security
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7172.txt
>
>
> The SecureIOUtils.openForRead function currently uses a racy stat/open combo if security is disabled and the native libraries are not available. This ends up shelling out to "ls -ld" which is very very slow. We've seen this cause significant performance regressions on clusters that match this profile.
> Since the racy permissions check doesn't buy us any security anyway, we should just fall back to a normal "open" without any stat() at all, if we can't use the native support to do it efficiently.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13427-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 08:12:03 2011
Return-Path: <common-issues-return-13427-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15485 invoked from network); 8 Mar 2011 08:11:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 08:11:59 -0000
Received: (qmail 881 invoked by uid 500); 8 Mar 2011 08:11:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 835 invoked by uid 500); 8 Mar 2011 08:11:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 801 invoked by uid 99); 8 Mar 2011 08:11:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 08:11:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 07:35:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A656139C1F4
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 07:34:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 07:34:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1467251144.3852.1299569699678.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1799619354.3818.1299567419614.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Assigned: (HADOOP-7172) SecureIO should not check owner on
 non-secure clusters that have no native support
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon reassigned HADOOP-7172:
-----------------------------------

    Assignee: Todd Lipcon

> SecureIO should not check owner on non-secure clusters that have no native support
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7172
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7172
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io, security
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7172.txt
>
>
> The SecureIOUtils.openForRead function currently uses a racy stat/open combo if security is disabled and the native libraries are not available. This ends up shelling out to "ls -ld" which is very very slow. We've seen this cause significant performance regressions on clusters that match this profile.
> Since the racy permissions check doesn't buy us any security anyway, we should just fall back to a normal "open" without any stat() at all, if we can't use the native support to do it efficiently.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13428-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 08:16:41 2011
Return-Path: <common-issues-return-13428-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19905 invoked from network); 8 Mar 2011 08:16:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 08:16:40 -0000
Received: (qmail 15212 invoked by uid 500); 8 Mar 2011 08:16:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14910 invoked by uid 500); 8 Mar 2011 08:16:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14691 invoked by uid 99); 8 Mar 2011 08:16:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 08:16:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 07:01:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7090039C7D3
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 07:00:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 07:00:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1629313008.3821.1299567659457.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7173) Remove unused fstat() call from
 NativeIO
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Remove unused fstat() call from NativeIO
----------------------------------------

                 Key: HADOOP-7173
                 URL: https://issues.apache.org/jira/browse/HADOOP-7173
             Project: Hadoop Common
          Issue Type: Improvement
    Affects Versions: 0.22.0
            Reporter: Todd Lipcon
             Fix For: 0.22.0


After HADOOP-7115, we don't need the fstat() native call or Stat class anymore. Let's remove them to cut down on unneeded code.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13429-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 08:17:01 2011
Return-Path: <common-issues-return-13429-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21398 invoked from network); 8 Mar 2011 08:16:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 08:16:58 -0000
Received: (qmail 24407 invoked by uid 500); 8 Mar 2011 08:16:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22666 invoked by uid 500); 8 Mar 2011 08:16:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21829 invoked by uid 99); 8 Mar 2011 08:16:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 08:16:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 06:57:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 96BA339C5E8
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 06:56:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 06:56:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1799619354.3818.1299567419614.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7172) SecureIO should not check owner on
 non-secure clusters that have no native support
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

SecureIO should not check owner on non-secure clusters that have no native support
----------------------------------------------------------------------------------

                 Key: HADOOP-7172
                 URL: https://issues.apache.org/jira/browse/HADOOP-7172
             Project: Hadoop Common
          Issue Type: Bug
          Components: io, security
    Affects Versions: 0.22.0
            Reporter: Todd Lipcon
            Priority: Critical
             Fix For: 0.22.0


The SecureIOUtils.openForRead function currently uses a racy stat/open combo if security is disabled and the native libraries are not available. This ends up shelling out to "ls -ld" which is very very slow. We've seen this cause significant performance regressions on clusters that match this profile.

Since the racy permissions check doesn't buy us any security anyway, we should just fall back to a normal "open" without any stat() at all, if we can't use the native support to do it efficiently.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13430-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 08:17:04 2011
Return-Path: <common-issues-return-13430-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21955 invoked from network); 8 Mar 2011 08:17:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 08:17:02 -0000
Received: (qmail 24747 invoked by uid 500); 8 Mar 2011 08:16:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24335 invoked by uid 500); 8 Mar 2011 08:16:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22502 invoked by uid 99); 8 Mar 2011 08:16:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 08:16:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 07:01:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7E56339C7D6
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 07:00:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 07:00:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <569531829.3823.1299567659514.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <10330397.93631295565704195.JavaMail.jira@thor>
Subject: [jira] Commented: (HADOOP-7115) Add a cache for getpwuid_r and
 getpwgid_r calls
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003839#comment-13003839 ] 

Todd Lipcon commented on HADOOP-7115:
-------------------------------------

bq. Do we need fstat and the Stat type at all now? I think not, we might as well remove it.

Actually, just to keep this closer to the 0.20 security branch, why don't we do that part in a separate JIRA. Filed HADOOP-7173.

> Add a cache for getpwuid_r and getpwgid_r calls
> -----------------------------------------------
>
>                 Key: HADOOP-7115
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7115
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Arun C Murthy
>            Assignee: Devaraj Das
>         Attachments: h-7115.1.patch
>
>
> As discussed in HADOOP-6978, a cache helps a lot.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13431-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 08:35:21 2011
Return-Path: <common-issues-return-13431-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5344 invoked from network); 8 Mar 2011 08:35:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 08:35:21 -0000
Received: (qmail 65498 invoked by uid 500); 8 Mar 2011 08:35:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65362 invoked by uid 500); 8 Mar 2011 08:35:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65351 invoked by uid 99); 8 Mar 2011 08:35:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 08:35:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 08:35:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9054139C5CB
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 08:34:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 08:34:59 +0000 (UTC)
From: "Brahma Reddy Battula (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <565981353.3914.1299573299588.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7164) rmr command is not displaying any
 error message when a path contains wildcard characters and does not exist.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003859#comment-13003859 ] 

Brahma Reddy Battula commented on HADOOP-7164:
----------------------------------------------

it will be useful to user if we provide the error message instead of keep quite.


> rmr command is not displaying any error message when a path contains wildcard characters and does not exist.
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7164
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7164
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Uma Maheswara Rao G
>            Priority: Minor
>         Attachments: HDFS-1609-src.patch, HDFS-1609-test.patch, HDFS-1609.patch
>
>
> When we give invalid directory path then it will show error message on the console. But if we provide the wildcard expression in invalid directory path then it will not show any error message even there is no pattern match for that path.
> linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin # ./hdfs dfs -rmr /test/test
> rmr: cannot remove /test/test: No such file or directory.
> *linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin # ./hdfs dfs -rmr /test* *
> *linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin #*

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13432-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 09:16:46 2011
Return-Path: <common-issues-return-13432-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21724 invoked from network); 8 Mar 2011 09:16:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 09:16:46 -0000
Received: (qmail 31948 invoked by uid 500); 8 Mar 2011 09:16:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31912 invoked by uid 500); 8 Mar 2011 09:16:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31883 invoked by uid 99); 8 Mar 2011 09:16:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 09:16:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 06:47:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7D37839C399
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 06:46:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 06:46:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <14795763.3805.1299566819509.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <10330397.93631295565704195.JavaMail.jira@thor>
Subject: [jira] Commented: (HADOOP-7115) Add a cache for getpwuid_r and
 getpwgid_r calls
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003835#comment-13003835 ] 

Todd Lipcon commented on HADOOP-7115:
-------------------------------------

A few comments:
- getUIDforFDOwnerforOwner is a kind of messy name - maybe just getUIDForFDOwner?
- The following should probably be debug level:
{code}
LOG.info("Got UserName " + user + " for UID " + uid + " from the native implementation");
{code}

- ensureInitialized has a formatting issue (cacheTimeout is on the same line as the opening if)
- The log message there should also probably be debug level
- Do we need fstat and the Stat type at all now? I think not, we might as well remove it.
- The change in getGroup.c seems unrelated to this JIRA, right?


> Add a cache for getpwuid_r and getpwgid_r calls
> -----------------------------------------------
>
>                 Key: HADOOP-7115
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7115
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Arun C Murthy
>            Assignee: Devaraj Das
>         Attachments: h-7115.1.patch
>
>
> As discussed in HADOOP-6978, a cache helps a lot.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13433-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 09:58:21 2011
Return-Path: <common-issues-return-13433-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54882 invoked from network); 8 Mar 2011 09:58:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 09:58:21 -0000
Received: (qmail 82523 invoked by uid 500); 8 Mar 2011 09:58:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82499 invoked by uid 500); 8 Mar 2011 09:58:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82491 invoked by uid 99); 8 Mar 2011 09:58:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 09:58:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 09:58:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8946739D696
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 09:57:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 09:57:59 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1627211432.4057.1299578279558.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7174) null is displayed in the console,if
 the src path is invalid while doing copyToLocal operation from commandLine
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

null is displayed in the console,if the src path is invalid while doing copyToLocal operation from commandLine
--------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-7174
                 URL: https://issues.apache.org/jira/browse/HADOOP-7174
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Uma Maheswara Rao G
            Priority: Minor


When we perform copyToLocal operations from commandLine and if src Path is invalid 

srcFS.globStatus(srcpath) will return null. So, when we find the length of resulted value, it will *throw NullPointerException*.

 Since we are handling generic exception , it will display null as the message.


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13434-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 10:00:21 2011
Return-Path: <common-issues-return-13434-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56258 invoked from network); 8 Mar 2011 10:00:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 10:00:21 -0000
Received: (qmail 88611 invoked by uid 500); 8 Mar 2011 10:00:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88586 invoked by uid 500); 8 Mar 2011 10:00:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88578 invoked by uid 99); 8 Mar 2011 10:00:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 10:00:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 10:00:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 737ED39D75F
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 09:59:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 09:59:59 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <300926130.4059.1299578399469.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1627211432.4057.1299578279558.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Assigned: (HADOOP-7174) null is displayed in the console,if
 the src path is invalid while doing copyToLocal operation from commandLine
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uma Maheswara Rao G reassigned HADOOP-7174:
-------------------------------------------

    Assignee: Uma Maheswara Rao G

> null is displayed in the console,if the src path is invalid while doing copyToLocal operation from commandLine
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7174
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7174
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>            Priority: Minor
>
> When we perform copyToLocal operations from commandLine and if src Path is invalid 
> srcFS.globStatus(srcpath) will return null. So, when we find the length of resulted value, it will *throw NullPointerException*.
>  Since we are handling generic exception , it will display null as the message.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13435-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 11:15:21 2011
Return-Path: <common-issues-return-13435-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4710 invoked from network); 8 Mar 2011 11:15:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 11:15:21 -0000
Received: (qmail 81446 invoked by uid 500); 8 Mar 2011 11:15:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81414 invoked by uid 500); 8 Mar 2011 11:15:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81406 invoked by uid 99); 8 Mar 2011 11:15:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 11:15:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 11:15:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8435539CE99
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 11:14:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 11:14:59 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <407951474.4115.1299582899538.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-6912) Guard against NPE when calling
 UGI.isLoginKeytabBased()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003895#comment-13003895 ] 

Hudson commented on HADOOP-6912:
--------------------------------

Integrated in Hadoop-Common-trunk #624 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/624/])
    HADOOP-6912. Guard against NPE when calling UGI.isLoginKeytabBased(). Contributed by Kan Zhang.


> Guard against NPE when calling UGI.isLoginKeytabBased()
> -------------------------------------------------------
>
>                 Key: HADOOP-6912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6912
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>             Fix For: 0.23.0
>
>         Attachments: c6912-01.patch
>
>
> NPE can happen when isLoginKeytabBased() is called before a login is performed. See MAPREDUCE-1992 for an example.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13436-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 11:15:23 2011
Return-Path: <common-issues-return-13436-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4881 invoked from network); 8 Mar 2011 11:15:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 11:15:23 -0000
Received: (qmail 81689 invoked by uid 500); 8 Mar 2011 11:15:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81659 invoked by uid 500); 8 Mar 2011 11:15:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81651 invoked by uid 99); 8 Mar 2011 11:15:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 11:15:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 11:15:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 76C2839CE97
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 11:14:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 11:14:59 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <649770415.4113.1299582899483.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <599378200.2869.1299537719453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7167) Allow using a file to exclude
 certain tests from build
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003894#comment-13003894 ] 

Hudson commented on HADOOP-7167:
--------------------------------

Integrated in Hadoop-Common-trunk #624 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/624/])
    HADOOP-7167. Allow using a file to exclude certain tests from build. Contributed by Todd Lipcon.


> Allow using a file to exclude certain tests from build
> ------------------------------------------------------
>
>                 Key: HADOOP-7167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7167
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7167.txt
>
>
> It would be nice to be able to exclude certain tests when running builds. For example, when a test is "known flaky", you may want to exclude it from the main Hudson job, but not actually disable it in the codebase (so that it still runs as part of another Hudson job, for example).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13437-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 14:21:31 2011
Return-Path: <common-issues-return-13437-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21425 invoked from network); 8 Mar 2011 14:21:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 14:21:29 -0000
Received: (qmail 42430 invoked by uid 500); 8 Mar 2011 14:21:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42397 invoked by uid 500); 8 Mar 2011 14:21:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42383 invoked by uid 99); 8 Mar 2011 14:21:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 14:21:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 14:21:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4D30839DD3C
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 14:21:07 +0000 (UTC)
Date: Tue, 8 Mar 2011 14:21:07 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <885543979.4467.1299594067312.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1627211432.4057.1299578279558.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7174) null is displayed in the console,if
 the src path is invalid while doing copyToLocal operation from commandLine
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uma Maheswara Rao G updated HADOOP-7174:
----------------------------------------

    Attachment: HADOOP-7174.patch

> null is displayed in the console,if the src path is invalid while doing copyToLocal operation from commandLine
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7174
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7174
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>            Priority: Minor
>         Attachments: HADOOP-7174.patch
>
>
> When we perform copyToLocal operations from commandLine and if src Path is invalid 
> srcFS.globStatus(srcpath) will return null. So, when we find the length of resulted value, it will *throw NullPointerException*.
>  Since we are handling generic exception , it will display null as the message.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13438-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 14:47:25 2011
Return-Path: <common-issues-return-13438-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25995 invoked from network); 8 Mar 2011 14:47:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 14:47:25 -0000
Received: (qmail 6072 invoked by uid 500); 8 Mar 2011 14:47:25 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6036 invoked by uid 500); 8 Mar 2011 14:47:25 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6028 invoked by uid 99); 8 Mar 2011 14:47:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 14:47:25 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 14:47:23 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 366B639D8B2
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 14:47:01 +0000 (UTC)
Date: Tue, 8 Mar 2011 14:47:01 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1975849353.4556.1299595621219.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1627211432.4057.1299578279558.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7174) null is displayed in the console,if
 the src path is invalid while doing copyToLocal operation from commandLine
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uma Maheswara Rao G updated HADOOP-7174:
----------------------------------------

    Release Note: Patch provided
          Status: Patch Available  (was: Open)

> null is displayed in the console,if the src path is invalid while doing copyToLocal operation from commandLine
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7174
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7174
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>            Priority: Minor
>         Attachments: HADOOP-7174.patch
>
>
> When we perform copyToLocal operations from commandLine and if src Path is invalid 
> srcFS.globStatus(srcpath) will return null. So, when we find the length of resulted value, it will *throw NullPointerException*.
>  Since we are handling generic exception , it will display null as the message.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13440-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 18:37:10 2011
Return-Path: <common-issues-return-13440-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39014 invoked from network); 8 Mar 2011 18:30:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 18:30:25 -0000
Received: (qmail 7995 invoked by uid 500); 8 Mar 2011 18:30:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7943 invoked by uid 500); 8 Mar 2011 18:30:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7880 invoked by uid 99); 8 Mar 2011 18:30:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 18:30:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 18:30:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C132339D3F4
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 18:29:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 18:29:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1349930333.5607.1299608999786.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1799619354.3818.1299567419614.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7172) SecureIO should not check owner on
 non-secure clusters that have no native support
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7172:
--------------------------------

    Attachment: hadoop-7172.txt

After sleeping on this, I had second thoughts about the original way I did the patch. It doesn't seem good to me that, on an insecure cluster, we have different semantics based on whether the native libraries are available. So, I've changed the condition such that the check is always skipped when security is disabled, regardless of presence of the native libs.

Updated the JavaDoc to indicate that the security check is only done when security is enabled.

> SecureIO should not check owner on non-secure clusters that have no native support
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7172
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7172
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io, security
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7172.txt, hadoop-7172.txt
>
>
> The SecureIOUtils.openForRead function currently uses a racy stat/open combo if security is disabled and the native libraries are not available. This ends up shelling out to "ls -ld" which is very very slow. We've seen this cause significant performance regressions on clusters that match this profile.
> Since the racy permissions check doesn't buy us any security anyway, we should just fall back to a normal "open" without any stat() at all, if we can't use the native support to do it efficiently.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13441-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 18:39:11 2011
Return-Path: <common-issues-return-13441-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52254 invoked from network); 8 Mar 2011 18:32:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 18:32:21 -0000
Received: (qmail 11031 invoked by uid 500); 8 Mar 2011 18:32:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11005 invoked by uid 500); 8 Mar 2011 18:32:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10997 invoked by uid 99); 8 Mar 2011 18:32:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 18:32:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 18:32:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2E6FC39D53F
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 18:32:00 +0000 (UTC)
Date: Tue, 8 Mar 2011 18:32:00 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <203904058.5626.1299609120186.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-6949) Reduces RPC packet size for
 primitive arrays, especially long[], which is used at block reporting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004122#comment-13004122 ] 

Hadoop QA commented on HADOOP-6949:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12472916/arrayprim_v6.patch
  against trunk revision 1079071.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/304//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/304//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/304//console

This message is automatically generated.

> Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6949
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6949
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Navis
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: ObjectWritable.diff, arrayprim.patch, arrayprim_v4.patch, arrayprim_v5.patch, arrayprim_v6.patch
>
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n
> It would not be needed to specify each element types for primitive arrays.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13439-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 18:40:39 2011
Return-Path: <common-issues-return-13439-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59686 invoked from network); 8 Mar 2011 18:34:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 18:34:01 -0000
Received: (qmail 39857 invoked by uid 500); 8 Mar 2011 16:47:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39820 invoked by uid 500); 8 Mar 2011 16:47:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39812 invoked by uid 99); 8 Mar 2011 16:47:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 16:47:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 16:47:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9E99B39D015
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 16:46:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 16:46:59 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <313446781.5207.1299602819646.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1799619354.3818.1299567419614.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7172) SecureIO should not check owner on
 non-secure clusters that have no native support
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004050#comment-13004050 ] 

Eli Collins commented on HADOOP-7172:
-------------------------------------

+1

Change looks good. Please update the javadoc ("@throws IOException if an IO Error occurred, or the user does not match") to qualify this for the new behavior.


> SecureIO should not check owner on non-secure clusters that have no native support
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7172
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7172
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io, security
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7172.txt
>
>
> The SecureIOUtils.openForRead function currently uses a racy stat/open combo if security is disabled and the native libraries are not available. This ends up shelling out to "ls -ld" which is very very slow. We've seen this cause significant performance regressions on clusters that match this profile.
> Since the racy permissions check doesn't buy us any security anyway, we should just fall back to a normal "open" without any stat() at all, if we can't use the native support to do it efficiently.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13442-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 18:55:31 2011
Return-Path: <common-issues-return-13442-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33833 invoked from network); 8 Mar 2011 18:52:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 18:52:21 -0000
Received: (qmail 48784 invoked by uid 500); 8 Mar 2011 18:52:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48737 invoked by uid 500); 8 Mar 2011 18:52:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48729 invoked by uid 99); 8 Mar 2011 18:52:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 18:52:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 18:52:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A05E539DF1D
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 18:51:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 18:51:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <411753473.5737.1299610319653.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1799619354.3818.1299567419614.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7172) SecureIO should not check owner on
 non-secure clusters that have no native support
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7172:
--------------------------------

    Attachment: hadoop-7172.txt

Sorry, forgot to tweak unit tests for the change. This patch should pass both with security on and off

> SecureIO should not check owner on non-secure clusters that have no native support
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7172
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7172
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io, security
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7172.txt, hadoop-7172.txt, hadoop-7172.txt
>
>
> The SecureIOUtils.openForRead function currently uses a racy stat/open combo if security is disabled and the native libraries are not available. This ends up shelling out to "ls -ld" which is very very slow. We've seen this cause significant performance regressions on clusters that match this profile.
> Since the racy permissions check doesn't buy us any security anyway, we should just fall back to a normal "open" without any stat() at all, if we can't use the native support to do it efficiently.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13443-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 19:29:26 2011
Return-Path: <common-issues-return-13443-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60382 invoked from network); 8 Mar 2011 19:29:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 19:29:25 -0000
Received: (qmail 28279 invoked by uid 500); 8 Mar 2011 19:29:25 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28221 invoked by uid 500); 8 Mar 2011 19:29:25 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28213 invoked by uid 99); 8 Mar 2011 19:29:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 19:29:25 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 19:29:22 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 039E239E005
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 19:29:01 +0000 (UTC)
Date: Tue, 8 Mar 2011 19:29:01 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1241314125.5831.1299612541011.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7156:
--------------------------------

    Attachment: hadoop-7156.txt

Updated version includes docs in core-default.xml

Do you agree with the current test case? It will fail with JVM exit on RHEL6 currently. Should we set the workaround variable to true in the test settings?

Another option is to add a bit of code to apply some regexes to /etc/redhat-release and enable by default if running RHEL or CentOS 6.

> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13444-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 19:33:24 2011
Return-Path: <common-issues-return-13444-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86547 invoked from network); 8 Mar 2011 19:33:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 19:33:23 -0000
Received: (qmail 32254 invoked by uid 500); 8 Mar 2011 19:33:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32216 invoked by uid 500); 8 Mar 2011 19:33:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32208 invoked by uid 99); 8 Mar 2011 19:33:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 19:33:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 19:33:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CDA0639E177
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 19:32:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 19:32:59 +0000 (UTC)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1997059073.5860.1299612779838.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-6949) Reduces RPC packet size for
 primitive arrays, especially long[], which is used at block reporting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004150#comment-13004150 ] 

Konstantin Shvachko commented on HADOOP-6949:
---------------------------------------------

I am perfectly fine with dealing with complex types in another jira. If Suresh could review this it should be good to commit.

> Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6949
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6949
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Navis
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: ObjectWritable.diff, arrayprim.patch, arrayprim_v4.patch, arrayprim_v5.patch, arrayprim_v6.patch
>
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n
> It would not be needed to specify each element types for primitive arrays.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13445-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 20:36:21 2011
Return-Path: <common-issues-return-13445-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7953 invoked from network); 8 Mar 2011 20:36:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 20:36:21 -0000
Received: (qmail 46058 invoked by uid 500); 8 Mar 2011 20:36:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46019 invoked by uid 500); 8 Mar 2011 20:36:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46011 invoked by uid 99); 8 Mar 2011 20:36:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 20:36:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 20:36:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E2FCA39D8CE
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 20:35:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 20:35:59 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <53757600.6039.1299616559926.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004195#comment-13004195 ] 

Allen Wittenauer commented on HADOOP-7156:
------------------------------------------

> have broken implementations of 

Let's call it for what it is: POSIX non-compliance.

> Systems known to exhibit this issue include Red Hat Enterprise Linux 6.0,
> CentOS 6.0, and Vintela Authentication Services.

Why is this text in here?  What happens when RHEL 6 gets this fixed? Do we make a new release to pull the text out?  To me, it makes much more sense to include a link to a wiki page where we can keep the information up-to-date.



> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13446-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 20:40:24 2011
Return-Path: <common-issues-return-13446-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8698 invoked from network); 8 Mar 2011 20:40:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 20:40:23 -0000
Received: (qmail 51064 invoked by uid 500); 8 Mar 2011 20:40:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51035 invoked by uid 500); 8 Mar 2011 20:40:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51027 invoked by uid 99); 8 Mar 2011 20:40:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 20:40:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 20:40:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8CD9739DA5A
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 20:39:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 20:39:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1492281362.6049.1299616799573.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004197#comment-13004197 ] 

Todd Lipcon commented on HADOOP-7156:
-------------------------------------

bq. What happens when RHEL 6 gets this fixed

That will be RHEL 6.1, not RHEL 6.0. So this should remain accurate.
Happy to change "Vintela Authentication Services" to "some version of Vintela Authentication Services".

bq. it makes much more sense to include a link to a wiki page where we can keep the information up-to-date

In my experience, we do a really bad job of keeping the wiki up to date. Greg, what do you think?

> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13447-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 20:59:24 2011
Return-Path: <common-issues-return-13447-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57185 invoked from network); 8 Mar 2011 20:59:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 20:59:24 -0000
Received: (qmail 94403 invoked by uid 500); 8 Mar 2011 20:59:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94313 invoked by uid 500); 8 Mar 2011 20:59:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94305 invoked by uid 99); 8 Mar 2011 20:59:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 20:59:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 20:59:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D655139E257
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 20:58:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 20:58:59 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1744068995.6103.1299617939874.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004209#comment-13004209 ] 

Eli Collins commented on HADOOP-7156:
-------------------------------------

The approach and code look good to me. I would leave as is, no need to add autodetection, but we should document it in core-default so users are aware of the issue. 

I think leaving the workaround disabled for the tests is the right thing, this way it fails on a broken system which will help users identify that they need to enable the workaround.

Also, the description seems appropriate to me, this isn't about POSIX compliance, it's just a bug in a library function, that will probably be fixed.

> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13448-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 21:36:26 2011
Return-Path: <common-issues-return-13448-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5888 invoked from network); 8 Mar 2011 21:36:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 21:36:25 -0000
Received: (qmail 66472 invoked by uid 500); 8 Mar 2011 21:36:25 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66443 invoked by uid 500); 8 Mar 2011 21:36:25 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66435 invoked by uid 99); 8 Mar 2011 21:36:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 21:36:25 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 21:36:22 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B253B39EF5A
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 21:36:00 +0000 (UTC)
Date: Tue, 8 Mar 2011 21:36:00 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <661322595.6207.1299620160727.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <130797695.15062.1298594983152.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7154) Should set MALLOC_ARENA_MAX in
 hadoop-env.sh
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004231#comment-13004231 ] 

Tom White commented on HADOOP-7154:
-----------------------------------

+1

> Should set MALLOC_ARENA_MAX in hadoop-env.sh
> --------------------------------------------
>
>                 Key: HADOOP-7154
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7154
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7154.txt
>
>
> New versions of glibc present in RHEL6 include a new arena allocator design. In several clusters we've seen this new allocator cause huge amounts of virtual memory to be used, since when multiple threads perform allocations, they each get their own memory arena. On a 64-bit system, these arenas are 64M mappings, and the maximum number of arenas is 8 times the number of cores. We've observed a DN process using 14GB of vmem for only 300M of resident set. This causes all kinds of nasty issues for obvious reasons.
> Setting MALLOC_ARENA_MAX to a low number will restrict the number of memory arenas and bound the virtual memory, with no noticeable downside in performance - we've been recommending MALLOC_ARENA_MAX=4. We should set this in hadoop-env.sh to avoid this issue as RHEL6 becomes more and more common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13449-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 22:24:25 2011
Return-Path: <common-issues-return-13449-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59395 invoked from network); 8 Mar 2011 22:24:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 22:24:21 -0000
Received: (qmail 66181 invoked by uid 500); 8 Mar 2011 22:24:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66150 invoked by uid 500); 8 Mar 2011 22:24:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66142 invoked by uid 99); 8 Mar 2011 22:24:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 22:24:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 22:24:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8FD7C39E2A2
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 22:23:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 22:23:59 +0000 (UTC)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1701212210.6379.1299623039586.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004258#comment-13004258 ] 

Greg Roelofs commented on HADOOP-7156:
--------------------------------------

Doh!  FF crashed while I was replying, sigh.  Switching to e-mail:

bq. In my experience, we do a really bad job of keeping the wiki up to date. Greg, what do you think?

I agree--we're much better at keeping the code up to date (frequently in
parallel across multiple branches ;-) ) than at keeping the wiki current.

I think the XML config text is fine; you could optionally prefix it with
"As of March 2011, systems known to ..." as a hint to users or future versions
of us to recheck it if significant time has passed.  The comment in NativeIO.c
probably should be modified; perhaps "monitor used for working around a bug
in the sssd security daemon, which was observed in getpwuid_r() on RHEL 6.0,"
or words to that effect.  (Need not be that verbose, of course.)

I also agree with Eli that we can leave the workaround disabled for tests.
It might be worthwhile to add a log message at the start that "this test
may fail (crash) with an invalid free() on some systems; see HADOOP-7156
for details."  Again, feel free to word it however you wish.

Trivial grammo:  "workaround" is a noun; the verb form is "work around"
(similar to layout, backup, setup, cleanup, checkin, cutoff, etc.).  The
various variable names would be more proper if they reflected this (e.g.,
WORK_AROUND_NON_THREADSAFE_CALLS_KEY, workAroundNonThreadSafePasswdCalls
[or workAroundNonThreadsafePasswdCalls, since you're using "threadsafe"
as a single word elsewhere]), but I won't fuss if you leave them as is.


> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13450-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 22:59:24 2011
Return-Path: <common-issues-return-13450-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62361 invoked from network); 8 Mar 2011 22:59:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 22:59:24 -0000
Received: (qmail 32944 invoked by uid 500); 8 Mar 2011 22:59:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32907 invoked by uid 500); 8 Mar 2011 22:59:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32885 invoked by uid 99); 8 Mar 2011 22:59:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 22:59:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 22:59:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B07A839ECDD
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 22:58:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 22:58:59 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1480480034.6441.1299625139719.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004271#comment-13004271 ] 

Allen Wittenauer commented on HADOOP-7156:
------------------------------------------

> That will be RHEL 6.1, not RHEL 6.0. So this should remain accurate.

If I rpm install just a new pam from the Red Hat repo, what version am I running?

> I agree--we're much better at keeping the code up to date (frequently in
> parallel across multiple branches ) than at keeping the wiki current.

Really?  We haven't had a release marked stable in over a year.  (Releases outside Apache do *not* count.)



> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13451-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 23:35:24 2011
Return-Path: <common-issues-return-13451-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17319 invoked from network); 8 Mar 2011 23:35:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 23:35:23 -0000
Received: (qmail 83008 invoked by uid 500); 8 Mar 2011 23:35:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82981 invoked by uid 500); 8 Mar 2011 23:35:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82973 invoked by uid 99); 8 Mar 2011 23:35:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 23:35:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 23:35:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 726E739E4C4
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 23:34:59 +0000 (UTC)
Date: Tue, 8 Mar 2011 23:34:59 +0000 (UTC)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <833161649.6588.1299627299465.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004284#comment-13004284 ] 

Greg Roelofs commented on HADOOP-7156:
--------------------------------------

bq. Really?  We haven't had a release marked stable in over a year.  (Releases outside Apache do *not* count.)

How many of the "currently relevant" wiki pages have been updated in that
year-plus?  (I haven't checked, but the one or two "getting started" ones
I looked at way back when were incomplete at best, and possibly wrong in
places, so I'm guessing the answer is "not many."  That's certainly been
my repeated experience with internal, corporate wikis.)

In my experience, the farther the docs are from the code, the less likely
they are to be current and correct.  That doesn't mean there can't be
additional forms of non-repo docs, but docs within the source-code repo
should be the priority.  Many of us _live_ in the repo, employmentally
speaking.  (New word.  Cool.)


> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13452-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 08 23:52:27 2011
Return-Path: <common-issues-return-13452-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68021 invoked from network); 8 Mar 2011 23:52:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Mar 2011 23:52:27 -0000
Received: (qmail 2742 invoked by uid 500); 8 Mar 2011 23:52:26 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2702 invoked by uid 500); 8 Mar 2011 23:52:26 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2694 invoked by uid 99); 8 Mar 2011 23:52:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 23:52:26 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Mar 2011 23:52:25 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9B2F139E966
	for <common-issues@hadoop.apache.org>; Tue,  8 Mar 2011 23:52:04 +0000 (UTC)
Date: Tue, 8 Mar 2011 23:52:04 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1084234756.6623.1299628324631.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <130797695.15062.1298594983152.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7154) Should set MALLOC_ARENA_MAX in
 hadoop-env.sh
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7154:
--------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

Committed to trunk and branch-22. Thanks Tom for reviewing.

> Should set MALLOC_ARENA_MAX in hadoop-env.sh
> --------------------------------------------
>
>                 Key: HADOOP-7154
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7154
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7154.txt
>
>
> New versions of glibc present in RHEL6 include a new arena allocator design. In several clusters we've seen this new allocator cause huge amounts of virtual memory to be used, since when multiple threads perform allocations, they each get their own memory arena. On a 64-bit system, these arenas are 64M mappings, and the maximum number of arenas is 8 times the number of cores. We've observed a DN process using 14GB of vmem for only 300M of resident set. This causes all kinds of nasty issues for obvious reasons.
> Setting MALLOC_ARENA_MAX to a low number will restrict the number of memory arenas and bound the virtual memory, with no noticeable downside in performance - we've been recommending MALLOC_ARENA_MAX=4. We should set this in hadoop-env.sh to avoid this issue as RHEL6 becomes more and more common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13453-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 09 00:21:52 2011
Return-Path: <common-issues-return-13453-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49041 invoked from network); 9 Mar 2011 00:21:05 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Mar 2011 00:21:05 -0000
Received: (qmail 17560 invoked by uid 500); 9 Mar 2011 00:10:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17524 invoked by uid 500); 9 Mar 2011 00:10:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17494 invoked by uid 99); 9 Mar 2011 00:10:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 00:10:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 00:10:22 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5774539ED58
	for <common-issues@hadoop.apache.org>; Wed,  9 Mar 2011 00:10:01 +0000 (UTC)
Date: Wed, 9 Mar 2011 00:10:01 +0000 (UTC)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2068270548.6685.1299629401355.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1878551188.5886.1297277337604.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch
 the calls in DataStorage to FileUtil.createHardLink(), so we call it once
 per directory instead of once per file
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004298#comment-13004298 ] 

Jakob Homan commented on HADOOP-7133:
-------------------------------------

+1 on this patch, however, even with the backwards compatibility now provided, I'd like to see the HDFS counterpart and commit them both at once.  Once that has been posted and reviewed, I'll go ahead and commit both.

> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7133
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7133
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.20.2
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>             Fix For: 0.22.0
>
>         Attachments: HDFS-1445-trunk.v23_common_1-of-3.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS.  This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS.  It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13454-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 09 00:22:05 2011
Return-Path: <common-issues-return-13454-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49981 invoked from network); 9 Mar 2011 00:21:12 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Mar 2011 00:21:12 -0000
Received: (qmail 19370 invoked by uid 500); 9 Mar 2011 00:20:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19232 invoked by uid 500); 9 Mar 2011 00:20:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19039 invoked by uid 99); 9 Mar 2011 00:16:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 00:16:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 00:16:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 93DF039EEA6
	for <common-issues@hadoop.apache.org>; Wed,  9 Mar 2011 00:15:59 +0000 (UTC)
Date: Wed, 9 Mar 2011 00:15:59 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1630087208.6697.1299629759602.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004300#comment-13004300 ] 

Allen Wittenauer commented on HADOOP-7156:
------------------------------------------

>How many of the "currently relevant" wiki pages have been updated in 
> that year-plus? 

Considering how little has changed since Apache 0.20.0 was released almost two years, not many because there was no need.

This JIRA is a great example of something that will likely be long fixed before an actual Apache release with the fix sees the light of day.  Which is why I'm mostly opposed to it going in, but not enough to -1 it.  I think we'll look like fools when a year down the road we have a reference to a bug that was long fixed, but whatever.

> Many of us live in the repo, employmentally speaking. 

In other words, "Who gives a damn about the people who are not running some form of trunk and are not watching JIRAs religiously."  

> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13455-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 09 01:46:28 2011
Return-Path: <common-issues-return-13455-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60963 invoked from network); 9 Mar 2011 01:46:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Mar 2011 01:46:23 -0000
Received: (qmail 26586 invoked by uid 500); 9 Mar 2011 01:46:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26555 invoked by uid 500); 9 Mar 2011 01:46:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26547 invoked by uid 99); 9 Mar 2011 01:46:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 01:46:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 01:46:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8204C39E340
	for <common-issues@hadoop.apache.org>; Wed,  9 Mar 2011 01:45:59 +0000 (UTC)
Date: Wed, 9 Mar 2011 01:45:59 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1511090042.6857.1299635159529.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1878551188.5886.1297277337604.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch
 the calls in DataStorage to FileUtil.createHardLink(), so we call it once
 per directory instead of once per file
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004326#comment-13004326 ] 

Matt Foley commented on HADOOP-7133:
------------------------------------

Hi Jakob, the HDFS counterpart has been up for some time on HDFS-1445.  However, test-patch can't run against the HDFS part until this COMMON part has been committed.  So perhaps it would be best if you take a quick look at HDFS-1445, then commit this patch so that I can enable test-patch to run against the other part?  Thanks.

> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7133
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7133
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.20.2
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>             Fix For: 0.22.0
>
>         Attachments: HDFS-1445-trunk.v23_common_1-of-3.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS.  This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS.  It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13456-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 09 01:51:21 2011
Return-Path: <common-issues-return-13456-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93923 invoked from network); 9 Mar 2011 01:51:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Mar 2011 01:51:21 -0000
Received: (qmail 31713 invoked by uid 500); 9 Mar 2011 01:51:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31681 invoked by uid 500); 9 Mar 2011 01:51:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31673 invoked by uid 99); 9 Mar 2011 01:51:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 01:51:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 01:51:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7D40239E4D8
	for <common-issues@hadoop.apache.org>; Wed,  9 Mar 2011 01:50:59 +0000 (UTC)
Date: Wed, 9 Mar 2011 01:50:59 +0000 (UTC)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <555131856.6868.1299635459509.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1878551188.5886.1297277337604.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch
 the calls in DataStorage to FileUtil.createHardLink(), so we call it once
 per directory instead of once per file
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004329#comment-13004329 ] 

Jakob Homan commented on HADOOP-7133:
-------------------------------------

bq. I will resubmit part-2, attached to this issue, as soon as part-1 has been approved and committed.
I had interpreted this to mean there would be a new HDFS patch.  Sounds like I'm incorrect and the current patch there is the one you're submitting?

> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7133
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7133
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.20.2
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>             Fix For: 0.22.0
>
>         Attachments: HDFS-1445-trunk.v23_common_1-of-3.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS.  This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS.  It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13457-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 09 02:05:24 2011
Return-Path: <common-issues-return-13457-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41377 invoked from network); 9 Mar 2011 02:05:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Mar 2011 02:05:24 -0000
Received: (qmail 41816 invoked by uid 500); 9 Mar 2011 02:05:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41776 invoked by uid 500); 9 Mar 2011 02:05:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41768 invoked by uid 99); 9 Mar 2011 02:05:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 02:05:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 02:05:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 96B9339E860
	for <common-issues@hadoop.apache.org>; Wed,  9 Mar 2011 02:04:59 +0000 (UTC)
Date: Wed, 9 Mar 2011 02:04:59 +0000 (UTC)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1479480223.6893.1299636299614.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004333#comment-13004333 ] 

Greg Roelofs commented on HADOOP-7156:
--------------------------------------



That wasn't my personal experience (with the setup-related ones), but
admittedly it's a tiny sample.


Dude, I'm still running a 2004-era distro on one of my desktop boxes!
Sure, that's unusual, but ancient servers in production definitely
aren't.  The more you have, the harder it is to validate and move to
a new version.  You've got, what, a few hundred servers to worry about?
We've got a few hundred thousand.  (Not all Hadoop-related, I mean, just
in general.)  I have no idea what the overall mix is, but I was aware of
both RHEL 5.x and 4.x subsets up until fairly recently (the latter also
being 2004-era; the former more like 2007, I think), and I wouldn't be
surprised if there were still *BSD boxes lurking somewhere.  I don't
think there are any RHEL 6.x yet.

So no, OS-related docs don't get stale _that_ quickly, at least not for
some of us. :-)



Not at all.  It's simply a statement that those working most closely with
the code and XML configs--i.e., those who _do_ watch JIRAs and the mailing
lists religiously--are more likely to keep things updated if they're
nearby/noticeable.  We all have limited attention-hours to devote to
maintenance, so anything that makes it easier, faster, or more likely to
happen is a Good Thing.

And I _know_ you're aware that the default state for documentation is
"nonexistent."  Especially where developers are concerned.

(Btw, wasn't there a recent suggestion to automatically pull config keys
and descriptions out of the foo-default.xml files and publish them?  With
something like that in place, the docs you want would fall out of this
and similar patches immediately.  One of my mottos for the last few years
has been, "if it can be automated, it should be automated.")

I guess this has drifted a bit; perhaps further discussion should go to
one of the lists?


> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13458-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 09 02:20:24 2011
Return-Path: <common-issues-return-13458-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74655 invoked from network); 9 Mar 2011 02:20:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Mar 2011 02:20:23 -0000
Received: (qmail 49750 invoked by uid 500); 9 Mar 2011 02:20:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49701 invoked by uid 500); 9 Mar 2011 02:20:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49692 invoked by uid 99); 9 Mar 2011 02:20:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 02:20:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 02:20:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6FAE839F752
	for <common-issues@hadoop.apache.org>; Wed,  9 Mar 2011 02:19:59 +0000 (UTC)
Date: Wed, 9 Mar 2011 02:19:59 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <161461447.6917.1299637199454.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1878551188.5886.1297277337604.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch
 the calls in DataStorage to FileUtil.createHardLink(), so we call it once
 per directory instead of once per file
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004338#comment-13004338 ] 

Matt Foley commented on HADOOP-7133:
------------------------------------

Yes.  s/resubmit/push the "submit patch" button again for/

> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7133
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7133
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.20.2
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>             Fix For: 0.22.0
>
>         Attachments: HDFS-1445-trunk.v23_common_1-of-3.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS.  This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS.  It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13459-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 09 06:04:28 2011
Return-Path: <common-issues-return-13459-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26927 invoked from network); 9 Mar 2011 06:04:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Mar 2011 06:04:27 -0000
Received: (qmail 39951 invoked by uid 500); 9 Mar 2011 06:04:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39419 invoked by uid 500); 9 Mar 2011 06:04:26 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38939 invoked by uid 99); 9 Mar 2011 06:04:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 06:04:26 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 06:04:25 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C75673A05C3
	for <common-issues@hadoop.apache.org>; Wed,  9 Mar 2011 06:04:04 +0000 (UTC)
Date: Wed, 9 Mar 2011 06:04:04 +0000 (UTC)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <419357987.7175.1299650644813.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1878551188.5886.1297277337604.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch
 the calls in DataStorage to FileUtil.createHardLink(), so we call it once
 per directory instead of once per file
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004383#comment-13004383 ] 

Hairong Kuang commented on HADOOP-7133:
---------------------------------------

Matt, this optimization is fantastic. I have ported it to our internal branch. Together with parallel upgrades, it cut the DN upgrade time from 1 hour 40 minutes to 1 minute! Awesome job!

+1. The patch looks good. One minor optimization is that you could use ThreadLocal for shell commands. This will remove the need to create a buffer for each shell command to run.

> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7133
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7133
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.20.2
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>             Fix For: 0.22.0
>
>         Attachments: HDFS-1445-trunk.v23_common_1-of-3.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS.  This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS.  It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13460-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 09 11:14:24 2011
Return-Path: <common-issues-return-13460-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3735 invoked from network); 9 Mar 2011 11:14:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Mar 2011 11:14:24 -0000
Received: (qmail 53710 invoked by uid 500); 9 Mar 2011 11:14:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53653 invoked by uid 500); 9 Mar 2011 11:14:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53645 invoked by uid 99); 9 Mar 2011 11:14:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 11:14:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 11:14:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AE2C93A0CAB
	for <common-issues@hadoop.apache.org>; Wed,  9 Mar 2011 11:13:59 +0000 (UTC)
Date: Wed, 9 Mar 2011 11:13:59 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <979097868.7571.1299669239710.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <130797695.15062.1298594983152.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7154) Should set MALLOC_ARENA_MAX in
 hadoop-env.sh
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004476#comment-13004476 ] 

Hudson commented on HADOOP-7154:
--------------------------------

Integrated in Hadoop-Common-trunk #625 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/625/])
    HADOOP-7154. Should set MALLOC_ARENA_MAX in hadoop-env.sh. Contributed by Todd Lipcon.


> Should set MALLOC_ARENA_MAX in hadoop-env.sh
> --------------------------------------------
>
>                 Key: HADOOP-7154
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7154
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7154.txt
>
>
> New versions of glibc present in RHEL6 include a new arena allocator design. In several clusters we've seen this new allocator cause huge amounts of virtual memory to be used, since when multiple threads perform allocations, they each get their own memory arena. On a 64-bit system, these arenas are 64M mappings, and the maximum number of arenas is 8 times the number of cores. We've observed a DN process using 14GB of vmem for only 300M of resident set. This causes all kinds of nasty issues for obvious reasons.
> Setting MALLOC_ARENA_MAX to a low number will restrict the number of memory arenas and bound the virtual memory, with no noticeable downside in performance - we've been recommending MALLOC_ARENA_MAX=4. We should set this in hadoop-env.sh to avoid this issue as RHEL6 becomes more and more common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13461-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 09 18:05:24 2011
Return-Path: <common-issues-return-13461-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86570 invoked from network); 9 Mar 2011 18:05:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Mar 2011 18:05:24 -0000
Received: (qmail 12432 invoked by uid 500); 9 Mar 2011 18:05:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12368 invoked by uid 500); 9 Mar 2011 18:05:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12360 invoked by uid 99); 9 Mar 2011 18:05:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 18:05:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 18:05:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 92D4F3A1550
	for <common-issues@hadoop.apache.org>; Wed,  9 Mar 2011 18:04:59 +0000 (UTC)
Date: Wed, 9 Mar 2011 18:04:59 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <740083816.8548.1299693899598.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004684#comment-13004684 ] 

Allen Wittenauer commented on HADOOP-7156:
------------------------------------------

Actually, rather than let you dig Yahoo! in deeper, I'm just going to -1 the patch due to the config description on the grounds that it is both too generic and too specific:

* Are we going to update this list if Debian/Scientific Linux/Mandriva/... are found to be non-POSIX compliant?
* If yes, are we going to push a new Apache release when this list is updated?
* What does RHEL 6.0 actually mean?  If I put a new pam rpm that fixes the issue, am I still running RHEL 6.0?

A living document for when to use this as an option is going to be much better over the long haul than a static file.

> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13462-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 09 18:17:24 2011
Return-Path: <common-issues-return-13462-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95275 invoked from network); 9 Mar 2011 18:17:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Mar 2011 18:17:24 -0000
Received: (qmail 37250 invoked by uid 500); 9 Mar 2011 18:17:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37219 invoked by uid 500); 9 Mar 2011 18:17:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37211 invoked by uid 99); 9 Mar 2011 18:17:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 18:17:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 18:17:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AEE5B3A18D1
	for <common-issues@hadoop.apache.org>; Wed,  9 Mar 2011 18:16:59 +0000 (UTC)
Date: Wed, 9 Mar 2011 18:16:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <940183207.8577.1299694619713.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004691#comment-13004691 ] 

Todd Lipcon commented on HADOOP-7156:
-------------------------------------

I will answer all of your questions and hope you will withdraw your -1:

bq.  Are we going to update this list if Debian/Scientific Linux/Mandriva/... are found to be non-POSIX compliant?

I will happily volunteer to +1 and commit a patch that anyone should submit to update the list. Please feel free to do so. The list doesn't claim to be complete, only to provide some examples of systems where it's the case.

bq. If yes, are we going to push a new Apache release when this list is updated?

No, like any other documentation improvement it should wait for the next release. Our definitive documentation lives in the source tree. The fact that some of it is in the wiki doesn't change that.

bq. What does RHEL 6.0 actually mean? If I put a new pam rpm that fixes the issue, am I still running RHEL 6.0?

I would assume RHEL 6.0 means that you are not installing packages that are not part of the RHEL 6.0 release. If you went and installed a random RPM that you built from source or some other vendor, you're no longer running a stock RHEL 6.0. Again, the docs are not meant to be complete. Assumedly if you've updated your pam manually to workaround this issue, you'd know that, and you wouldn't turn on the config option!


And, seriously, with the HADOOP-7115 cache in place, this becomes a really rare race condition. Is it really worth making such a big deal? I'd like to move on to fixing other bugs, please.


> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13463-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 09 18:47:22 2011
Return-Path: <common-issues-return-13463-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11899 invoked from network); 9 Mar 2011 18:47:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Mar 2011 18:47:22 -0000
Received: (qmail 8568 invoked by uid 500); 9 Mar 2011 18:47:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8531 invoked by uid 500); 9 Mar 2011 18:47:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8423 invoked by uid 99); 9 Mar 2011 18:47:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 18:47:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 18:47:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 42FAF3A1875
	for <common-issues@hadoop.apache.org>; Wed,  9 Mar 2011 18:47:00 +0000 (UTC)
Date: Wed, 9 Mar 2011 18:47:00 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <356547615.8708.1299696420271.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004720#comment-13004720 ] 

Allen Wittenauer commented on HADOOP-7156:
------------------------------------------

> No, like any other documentation improvement it should wait for the 
> next release. Our definitive documentation lives in the source tree. 
> The fact that some of it is in the wiki doesn't change that.

I agree to a certain extent.  But listing a moving target from suffering from a problem that exists today that may not exist tomorrow for a release of our software that is still months out is a mistake.  CentOS 6.0 doesn't even exist yet in any official form yet we're declaring it bugged. 

> Assumedly if you've updated your pam manually to workaround this issue, 
> you'd know that, and you wouldn't turn on the config option!

What I am applying the 6-updates repo after a kick? Are packages from https://rhn.redhat.com/errata/rhel-server-6-errata.html not considered part of RHEL 6?  I suspect Red Hat would disagree.

I realize that for the majority of committers don't actually use an Apache release so this doesn't seem like a big deal since you folks release seemingly monthly, if not more frequent.  I also realize that in order for the various commercial interests involved it behooves them to get any and all patches committed to show that they are "contributing".  The reality is that by the time this patch shows up in an actual Apache release, it is bound to be outdated.... and that's where I take offense.


> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13464-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 09 18:59:25 2011
Return-Path: <common-issues-return-13464-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37744 invoked from network); 9 Mar 2011 18:59:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Mar 2011 18:59:24 -0000
Received: (qmail 26079 invoked by uid 500); 9 Mar 2011 18:59:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25953 invoked by uid 500); 9 Mar 2011 18:59:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25800 invoked by uid 99); 9 Mar 2011 18:59:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 18:59:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 18:59:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9ECE63A1D32
	for <common-issues@hadoop.apache.org>; Wed,  9 Mar 2011 18:58:59 +0000 (UTC)
Date: Wed, 9 Mar 2011 18:58:59 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1409102499.8762.1299697139647.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7175) Add isEnabled() to Trash
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Add isEnabled() to Trash
------------------------

                 Key: HADOOP-7175
                 URL: https://issues.apache.org/jira/browse/HADOOP-7175
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
    Affects Versions: 0.22.0
            Reporter: Daryn Sharp
             Fix For: 0.22.0


The moveToTrash method returns false in a number of cases.  It's not possible to discern if false means an error occurred. In particular, it's not possible to know if the trash is disabled vs. an error occurred.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13465-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 09 19:03:22 2011
Return-Path: <common-issues-return-13465-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67428 invoked from network); 9 Mar 2011 19:03:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Mar 2011 19:03:22 -0000
Received: (qmail 42249 invoked by uid 500); 9 Mar 2011 19:03:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42219 invoked by uid 500); 9 Mar 2011 19:03:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42211 invoked by uid 99); 9 Mar 2011 19:03:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 19:03:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 19:03:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9D14B3A1EC3
	for <common-issues@hadoop.apache.org>; Wed,  9 Mar 2011 19:02:59 +0000 (UTC)
Date: Wed, 9 Mar 2011 19:02:59 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1405550983.8768.1299697379640.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1409102499.8762.1299697139647.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7175) Add isEnabled() to Trash
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7175:
--------------------------------

    Status: Patch Available  (was: Open)

> Add isEnabled() to Trash
> ------------------------
>
>                 Key: HADOOP-7175
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7175
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7175.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The moveToTrash method returns false in a number of cases.  It's not possible to discern if false means an error occurred. In particular, it's not possible to know if the trash is disabled vs. an error occurred.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13466-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 09 19:03:24 2011
Return-Path: <common-issues-return-13466-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67769 invoked from network); 9 Mar 2011 19:03:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Mar 2011 19:03:24 -0000
Received: (qmail 43644 invoked by uid 500); 9 Mar 2011 19:03:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43611 invoked by uid 500); 9 Mar 2011 19:03:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43603 invoked by uid 99); 9 Mar 2011 19:03:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 19:03:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 19:03:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 889163A1EC0
	for <common-issues@hadoop.apache.org>; Wed,  9 Mar 2011 19:02:59 +0000 (UTC)
Date: Wed, 9 Mar 2011 19:02:59 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <575836641.8765.1299697379555.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1409102499.8762.1299697139647.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7175) Add isEnabled() to Trash
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7175:
--------------------------------

    Attachment: HADOOP-7175.patch

Adds isEnabled() method and corresponding tests.

> Add isEnabled() to Trash
> ------------------------
>
>                 Key: HADOOP-7175
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7175
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7175.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The moveToTrash method returns false in a number of cases.  It's not possible to discern if false means an error occurred. In particular, it's not possible to know if the trash is disabled vs. an error occurred.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13467-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 09 19:11:25 2011
Return-Path: <common-issues-return-13467-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93479 invoked from network); 9 Mar 2011 19:11:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Mar 2011 19:11:23 -0000
Received: (qmail 65503 invoked by uid 500); 9 Mar 2011 19:11:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65390 invoked by uid 500); 9 Mar 2011 19:11:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65357 invoked by uid 99); 9 Mar 2011 19:11:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 19:11:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 19:11:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AD71C3A1320
	for <common-issues@hadoop.apache.org>; Wed,  9 Mar 2011 19:10:59 +0000 (UTC)
Date: Wed, 9 Mar 2011 19:10:59 +0000 (UTC)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1415942065.8828.1299697859707.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Resolved: (HADOOP-6236) clean-up Trash.moveToTrash() code.
 Should throw exception instead of returning false
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan resolved HADOOP-6236.
---------------------------------

    Resolution: Duplicate

Resolving as duplicate of HADOOP-6345

> clean-up Trash.moveToTrash() code. Should throw exception instead of returning false
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6236
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6236
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Boris Shkolnik
>
> In case of errors it should throw an exception instead of returning false.
> One valid case is when interval is 0 (trash disabled), but that could be moved to a separate method isEnabled().

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13468-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 09 19:11:25 2011
Return-Path: <common-issues-return-13468-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93572 invoked from network); 9 Mar 2011 19:11:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Mar 2011 19:11:24 -0000
Received: (qmail 65943 invoked by uid 500); 9 Mar 2011 19:11:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65868 invoked by uid 500); 9 Mar 2011 19:11:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65663 invoked by uid 99); 9 Mar 2011 19:11:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 19:11:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 19:11:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D66CF3A1326
	for <common-issues@hadoop.apache.org>; Wed,  9 Mar 2011 19:10:59 +0000 (UTC)
Date: Wed, 9 Mar 2011 19:10:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <945141946.8834.1299697859875.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7156:
--------------------------------

    Attachment: hadoop-7156.txt

Made a wiki page and linked to it.

If you don't like this patch or the wiki page, please edit it and contribute a version you find acceptable. Otherwise I will consider this version acceptable and commit tomorrow.

> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13469-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 09 19:44:21 2011
Return-Path: <common-issues-return-13469-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91599 invoked from network); 9 Mar 2011 19:44:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Mar 2011 19:44:21 -0000
Received: (qmail 34379 invoked by uid 500); 9 Mar 2011 19:44:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34344 invoked by uid 500); 9 Mar 2011 19:44:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34336 invoked by uid 99); 9 Mar 2011 19:44:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 19:44:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 19:44:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 671253A1E5F
	for <common-issues@hadoop.apache.org>; Wed,  9 Mar 2011 19:43:59 +0000 (UTC)
Date: Wed, 9 Mar 2011 19:43:59 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1991821639.8902.1299699839419.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7176) Redesign FsShell
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Redesign FsShell
----------------

                 Key: HADOOP-7176
                 URL: https://issues.apache.org/jira/browse/HADOOP-7176
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.22.0
            Reporter: Daryn Sharp
             Fix For: 0.22.0


The FsShell commands are very tightly coupled together which makes it necessarily hard to write new commands.  There is a lot of redundancy between the commands, inconsistencies in handling of paths, handling of errors, and correct return of exit codes.  The FsShell commands should be subclasses of the fs.shell.Command class which is already being used by the -count command, and is used by other commands like dfsadmin.

This will serve as an umbrella bug to track the changes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13470-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 09 20:41:24 2011
Return-Path: <common-issues-return-13470-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88698 invoked from network); 9 Mar 2011 20:41:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Mar 2011 20:41:23 -0000
Received: (qmail 60218 invoked by uid 500); 9 Mar 2011 20:41:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60097 invoked by uid 500); 9 Mar 2011 20:41:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60089 invoked by uid 99); 9 Mar 2011 20:41:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 20:41:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 20:41:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 83A743A1526
	for <common-issues@hadoop.apache.org>; Wed,  9 Mar 2011 20:40:59 +0000 (UTC)
Date: Wed, 9 Mar 2011 20:40:59 +0000 (UTC)
From: "Olga Natkovich (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1549838432.9087.1299703259536.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1991821639.8902.1299699839419.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7176) Redesign FsShell
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004786#comment-13004786 ] 

Olga Natkovich commented on HADOOP-7176:
----------------------------------------

Pig project uses FsShell to export HDFS commands to the users. Things that are important to us:

(1) Backward compatibility
(2) Better error handling. At this point if command fails it seems to always return -1 so we can't tell different problems apart.

> Redesign FsShell
> ----------------
>
>                 Key: HADOOP-7176
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7176
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>             Fix For: 0.22.0
>
>
> The FsShell commands are very tightly coupled together which makes it necessarily hard to write new commands.  There is a lot of redundancy between the commands, inconsistencies in handling of paths, handling of errors, and correct return of exit codes.  The FsShell commands should be subclasses of the fs.shell.Command class which is already being used by the -count command, and is used by other commands like dfsadmin.
> This will serve as an umbrella bug to track the changes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13471-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 09 20:47:24 2011
Return-Path: <common-issues-return-13471-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11155 invoked from network); 9 Mar 2011 20:47:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Mar 2011 20:47:24 -0000
Received: (qmail 77288 invoked by uid 500); 9 Mar 2011 20:47:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77255 invoked by uid 500); 9 Mar 2011 20:47:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77244 invoked by uid 99); 9 Mar 2011 20:47:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 20:47:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 20:47:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B3C293A182B
	for <common-issues@hadoop.apache.org>; Wed,  9 Mar 2011 20:46:59 +0000 (UTC)
Date: Wed, 9 Mar 2011 20:46:59 +0000 (UTC)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <443853259.9105.1299703619733.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1878551188.5886.1297277337604.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch
 the calls in DataStorage to FileUtil.createHardLink(), so we call it once
 per directory instead of once per file
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004790#comment-13004790 ] 

Jakob Homan commented on HADOOP-7133:
-------------------------------------

Hairong: Good suggestion on the threadlocal.  I'm ready to commit this as is.  Would you be ok with doing that optimization in separate JIRA?

> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7133
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7133
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.20.2
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>             Fix For: 0.22.0
>
>         Attachments: HDFS-1445-trunk.v23_common_1-of-3.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS.  This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS.  It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13472-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 09 21:06:24 2011
Return-Path: <common-issues-return-13472-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91755 invoked from network); 9 Mar 2011 21:06:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Mar 2011 21:06:23 -0000
Received: (qmail 19455 invoked by uid 500); 9 Mar 2011 21:06:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19418 invoked by uid 500); 9 Mar 2011 21:06:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19410 invoked by uid 99); 9 Mar 2011 21:06:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 21:06:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 21:06:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A643F3A1F09
	for <common-issues@hadoop.apache.org>; Wed,  9 Mar 2011 21:05:59 +0000 (UTC)
Date: Wed, 9 Mar 2011 21:05:59 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <831096708.9143.1299704759677.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1991821639.8902.1299699839419.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7176) Redesign FsShell
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004797#comment-13004797 ] 

Daryn Sharp commented on HADOOP-7176:
-------------------------------------

Definitively maintaining backwards compatibility.  Let's discuss #2 offline.

> Redesign FsShell
> ----------------
>
>                 Key: HADOOP-7176
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7176
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>             Fix For: 0.22.0
>
>
> The FsShell commands are very tightly coupled together which makes it necessarily hard to write new commands.  There is a lot of redundancy between the commands, inconsistencies in handling of paths, handling of errors, and correct return of exit codes.  The FsShell commands should be subclasses of the fs.shell.Command class which is already being used by the -count command, and is used by other commands like dfsadmin.
> This will serve as an umbrella bug to track the changes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13473-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 09 21:12:25 2011
Return-Path: <common-issues-return-13473-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93765 invoked from network); 9 Mar 2011 21:12:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Mar 2011 21:12:24 -0000
Received: (qmail 32458 invoked by uid 500); 9 Mar 2011 21:12:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32420 invoked by uid 500); 9 Mar 2011 21:12:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32412 invoked by uid 99); 9 Mar 2011 21:12:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 21:12:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 21:12:22 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 653083A1151
	for <common-issues@hadoop.apache.org>; Wed,  9 Mar 2011 21:12:00 +0000 (UTC)
Date: Wed, 9 Mar 2011 21:12:00 +0000 (UTC)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1780407804.9159.1299705120411.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1991821639.8902.1299699839419.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7176) Redesign FsShell
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004799#comment-13004799 ] 

Jakob Homan commented on HADOOP-7176:
-------------------------------------

What do you mean by offline?  Design discussions should be on JIRA.

> Redesign FsShell
> ----------------
>
>                 Key: HADOOP-7176
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7176
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>             Fix For: 0.22.0
>
>
> The FsShell commands are very tightly coupled together which makes it necessarily hard to write new commands.  There is a lot of redundancy between the commands, inconsistencies in handling of paths, handling of errors, and correct return of exit codes.  The FsShell commands should be subclasses of the fs.shell.Command class which is already being used by the -count command, and is used by other commands like dfsadmin.
> This will serve as an umbrella bug to track the changes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13474-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 09 21:20:21 2011
Return-Path: <common-issues-return-13474-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10991 invoked from network); 9 Mar 2011 21:20:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Mar 2011 21:20:21 -0000
Received: (qmail 53481 invoked by uid 500); 9 Mar 2011 21:20:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53444 invoked by uid 500); 9 Mar 2011 21:20:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53435 invoked by uid 99); 9 Mar 2011 21:20:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 21:20:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Mar 2011 21:20:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8718B3A165D
	for <common-issues@hadoop.apache.org>; Wed,  9 Mar 2011 21:19:59 +0000 (UTC)
Date: Wed, 9 Mar 2011 21:19:59 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1440058693.9199.1299705599550.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1991821639.8902.1299699839419.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7176) Redesign FsShell
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004803#comment-13004803 ] 

Daryn Sharp commented on HADOOP-7176:
-------------------------------------

My apologies, we are both yahoos so I figured we could internally discuss and then summarize for approval.

> Redesign FsShell
> ----------------
>
>                 Key: HADOOP-7176
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7176
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>             Fix For: 0.22.0
>
>
> The FsShell commands are very tightly coupled together which makes it necessarily hard to write new commands.  There is a lot of redundancy between the commands, inconsistencies in handling of paths, handling of errors, and correct return of exit codes.  The FsShell commands should be subclasses of the fs.shell.Command class which is already being used by the -count command, and is used by other commands like dfsadmin.
> This will serve as an umbrella bug to track the changes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13475-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 00:34:23 2011
Return-Path: <common-issues-return-13475-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34101 invoked from network); 10 Mar 2011 00:34:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 00:34:23 -0000
Received: (qmail 83931 invoked by uid 500); 10 Mar 2011 00:34:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83858 invoked by uid 500); 10 Mar 2011 00:34:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83641 invoked by uid 99); 10 Mar 2011 00:34:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 00:34:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 00:34:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 637154A8A9
	for <common-issues@hadoop.apache.org>; Thu, 10 Mar 2011 00:34:00 +0000 (UTC)
Date: Thu, 10 Mar 2011 00:34:00 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1359805310.9666.1299717240403.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7177) CodecPool should report which
 compressor it is using
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

CodecPool should report which compressor it is using
----------------------------------------------------

                 Key: HADOOP-7177
                 URL: https://issues.apache.org/jira/browse/HADOOP-7177
             Project: Hadoop Common
          Issue Type: Improvement
          Components: native
            Reporter: Allen Wittenauer
            Priority: Trivial


Certain native compression libraries are overly verbose causing confusion while reading the task logs.  Let's actually say which compressor we got when we report it in the task logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13476-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 00:36:22 2011
Return-Path: <common-issues-return-13476-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49849 invoked from network); 10 Mar 2011 00:36:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 00:36:21 -0000
Received: (qmail 86953 invoked by uid 500); 10 Mar 2011 00:36:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86924 invoked by uid 500); 10 Mar 2011 00:36:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86916 invoked by uid 99); 10 Mar 2011 00:36:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 00:36:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 00:36:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 222054A920
	for <common-issues@hadoop.apache.org>; Thu, 10 Mar 2011 00:36:00 +0000 (UTC)
Date: Thu, 10 Mar 2011 00:36:00 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1205293346.9670.1299717360136.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1359805310.9666.1299717240403.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7177) CodecPool should report which
 compressor it is using
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Allen Wittenauer updated HADOOP-7177:
-------------------------------------

    Attachment: hadoop-7177.txt

> CodecPool should report which compressor it is using
> ----------------------------------------------------
>
>                 Key: HADOOP-7177
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7177
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: native
>            Reporter: Allen Wittenauer
>            Priority: Trivial
>         Attachments: hadoop-7177.txt
>
>
> Certain native compression libraries are overly verbose causing confusion while reading the task logs.  Let's actually say which compressor we got when we report it in the task logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13477-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 00:36:22 2011
Return-Path: <common-issues-return-13477-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49869 invoked from network); 10 Mar 2011 00:36:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 00:36:21 -0000
Received: (qmail 87166 invoked by uid 500); 10 Mar 2011 00:36:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87131 invoked by uid 500); 10 Mar 2011 00:36:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86978 invoked by uid 99); 10 Mar 2011 00:36:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 00:36:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 00:36:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3446C4A924
	for <common-issues@hadoop.apache.org>; Thu, 10 Mar 2011 00:36:00 +0000 (UTC)
Date: Thu, 10 Mar 2011 00:36:00 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1710505014.9672.1299717360211.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1359805310.9666.1299717240403.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7177) CodecPool should report which
 compressor it is using
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Allen Wittenauer updated HADOOP-7177:
-------------------------------------

    Assignee: Allen Wittenauer
      Status: Patch Available  (was: Open)

> CodecPool should report which compressor it is using
> ----------------------------------------------------
>
>                 Key: HADOOP-7177
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7177
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: native
>            Reporter: Allen Wittenauer
>            Assignee: Allen Wittenauer
>            Priority: Trivial
>         Attachments: hadoop-7177.txt
>
>
> Certain native compression libraries are overly verbose causing confusion while reading the task logs.  Let's actually say which compressor we got when we report it in the task logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13478-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 01:58:21 2011
Return-Path: <common-issues-return-13478-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85205 invoked from network); 10 Mar 2011 01:58:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 01:58:21 -0000
Received: (qmail 56796 invoked by uid 500); 10 Mar 2011 01:58:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56759 invoked by uid 500); 10 Mar 2011 01:58:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56751 invoked by uid 99); 10 Mar 2011 01:58:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 01:58:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 01:58:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9E11F3A0989
	for <common-issues@hadoop.apache.org>; Thu, 10 Mar 2011 01:57:59 +0000 (UTC)
Date: Thu, 10 Mar 2011 01:57:59 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <823099050.9889.1299722279644.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <599378200.2869.1299537719453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7167) Allow using a file to exclude
 certain tests from build
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004943#comment-13004943 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7167:
------------------------------------------------

Hi Todd, the exclude file does not work on Cygwin.  Could you take a look?

{noformat}
...
BUILD FAILED
z:\_tsz\hadoop\hdfs\h1\build.xml:745: The following error occurred while executing this line:
z:\_tsz\hadoop\hdfs\h1\build.xml:677: The following error occurred while executing this line:
z:\_tsz\hadoop\hdfs\h1\build.xml:613: Excludesfile z:\dev\null not found.

Total time: 16 seconds
{noformat}


> Allow using a file to exclude certain tests from build
> ------------------------------------------------------
>
>                 Key: HADOOP-7167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7167
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7167.txt
>
>
> It would be nice to be able to exclude certain tests when running builds. For example, when a test is "known flaky", you may want to exclude it from the main Hudson job, but not actually disable it in the codebase (so that it still runs as part of another Hudson job, for example).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13479-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 02:08:21 2011
Return-Path: <common-issues-return-13479-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38058 invoked from network); 10 Mar 2011 02:08:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 02:08:21 -0000
Received: (qmail 64930 invoked by uid 500); 10 Mar 2011 02:08:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64895 invoked by uid 500); 10 Mar 2011 02:08:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64886 invoked by uid 99); 10 Mar 2011 02:08:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 02:08:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 02:08:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 698683A0C21
	for <common-issues@hadoop.apache.org>; Thu, 10 Mar 2011 02:07:59 +0000 (UTC)
Date: Thu, 10 Mar 2011 02:07:59 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2025554241.9900.1299722879428.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1878551188.5886.1297277337604.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch
 the calls in DataStorage to FileUtil.createHardLink(), so we call it once
 per directory instead of once per file
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004945#comment-13004945 ] 

Matt Foley commented on HADOOP-7133:
------------------------------------

Hi Hairong, glad it's working well for you.  Your suggestion to use ThreadLocal buffers is perfect.  It will take me a little time to implement and test, though, so I'd like to go ahead with Jakob's suggestion, if you don't mind, so we can proceed with review of the second part of the patch in HDFS-1445.

> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7133
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7133
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.20.2
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>             Fix For: 0.22.0
>
>         Attachments: HDFS-1445-trunk.v23_common_1-of-3.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS.  This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS.  It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13480-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 03:28:28 2011
Return-Path: <common-issues-return-13480-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71827 invoked from network); 10 Mar 2011 03:28:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 03:28:27 -0000
Received: (qmail 12155 invoked by uid 500); 10 Mar 2011 03:28:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11922 invoked by uid 500); 10 Mar 2011 03:28:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11911 invoked by uid 99); 10 Mar 2011 03:28:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 03:28:25 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 03:28:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 024D43A1FAA
	for <common-issues@hadoop.apache.org>; Thu, 10 Mar 2011 03:28:00 +0000 (UTC)
Date: Thu, 10 Mar 2011 03:28:00 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <868433248.9957.1299727680005.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004956#comment-13004956 ] 

Eli Collins commented on HADOOP-7156:
-------------------------------------

Hey Allen,

If you find the config description "both too generic and too specific" perhaps you could suggest a better one before veto'ing the patch?

Thanks,
Eli

> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13481-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 05:20:22 2011
Return-Path: <common-issues-return-13481-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17832 invoked from network); 10 Mar 2011 05:20:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 05:20:22 -0000
Received: (qmail 84980 invoked by uid 500); 10 Mar 2011 05:20:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84678 invoked by uid 500); 10 Mar 2011 05:20:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84659 invoked by uid 99); 10 Mar 2011 05:20:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 05:20:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 05:20:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BDCCA3A17AA
	for <common-issues@hadoop.apache.org>; Thu, 10 Mar 2011 05:19:59 +0000 (UTC)
Date: Thu, 10 Mar 2011 05:19:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1406105811.10018.1299734399774.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <599378200.2869.1299537719453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Reopened: (HADOOP-7167) Allow using a file to exclude
 certain tests from build
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon reopened HADOOP-7167:
---------------------------------


> Allow using a file to exclude certain tests from build
> ------------------------------------------------------
>
>                 Key: HADOOP-7167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7167
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7167.txt
>
>
> It would be nice to be able to exclude certain tests when running builds. For example, when a test is "known flaky", you may want to exclude it from the main Hudson job, but not actually disable it in the codebase (so that it still runs as part of another Hudson job, for example).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13482-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 05:20:23 2011
Return-Path: <common-issues-return-13482-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17857 invoked from network); 10 Mar 2011 05:20:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 05:20:22 -0000
Received: (qmail 85067 invoked by uid 500); 10 Mar 2011 05:20:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85000 invoked by uid 500); 10 Mar 2011 05:20:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84688 invoked by uid 99); 10 Mar 2011 05:20:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 05:20:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 05:20:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9D9AD3A17A4
	for <common-issues@hadoop.apache.org>; Thu, 10 Mar 2011 05:19:59 +0000 (UTC)
Date: Thu, 10 Mar 2011 05:19:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1804404119.10013.1299734399642.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <599378200.2869.1299537719453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7167) Allow using a file to exclude
 certain tests from build
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004968#comment-13004968 ] 

Todd Lipcon commented on HADOOP-7167:
-------------------------------------

Hey Nicholas. Really sorry about that! Do you know if there is some equivalent to /dev/null in cygwin? ie a file which is always guaranteed to exist and be empty? I wonder what happens if we just pass an empty parameter to the ant excludeFile tag...

If this is getting in your way, feel free to revert and we can commit a new version that works on both platforms.

> Allow using a file to exclude certain tests from build
> ------------------------------------------------------
>
>                 Key: HADOOP-7167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7167
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7167.txt
>
>
> It would be nice to be able to exclude certain tests when running builds. For example, when a test is "known flaky", you may want to exclude it from the main Hudson job, but not actually disable it in the codebase (so that it still runs as part of another Hudson job, for example).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13483-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 10:57:22 2011
Return-Path: <common-issues-return-13483-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20184 invoked from network); 10 Mar 2011 10:57:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 10:57:22 -0000
Received: (qmail 7167 invoked by uid 500); 10 Mar 2011 10:57:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7133 invoked by uid 500); 10 Mar 2011 10:57:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7125 invoked by uid 99); 10 Mar 2011 10:57:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 10:57:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 10:57:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BF79F3A261A
	for <common-issues@hadoop.apache.org>; Thu, 10 Mar 2011 10:56:59 +0000 (UTC)
Date: Thu, 10 Mar 2011 10:56:59 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1986492955.10457.1299754619781.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Moved: (HADOOP-7178) copyToLocal API is creating .crc files
 in local, even after setting verifyChecksum to false at client side.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uma Maheswara Rao G moved HDFS-1733 to HADOOP-7178:
---------------------------------------------------

        Key: HADOOP-7178  (was: HDFS-1733)
    Project: Hadoop Common  (was: Hadoop HDFS)

> copyToLocal API is creating .crc files in local, even after setting verifyChecksum to false at client side.
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7178
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7178
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Uma Maheswara Rao G
>
> When we copy the files from DFS to local, it is creating the .crc file in local filesystem for the verification of checksum even if we disable the checksum verification at client side.
>         When user does not want to do any checksum verification, then what will be the use in creating of these files in local file system.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13484-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 15:04:22 2011
Return-Path: <common-issues-return-13484-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45618 invoked from network); 10 Mar 2011 15:04:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 15:04:21 -0000
Received: (qmail 61301 invoked by uid 500); 10 Mar 2011 15:04:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61261 invoked by uid 500); 10 Mar 2011 15:04:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61253 invoked by uid 99); 10 Mar 2011 15:04:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 15:04:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 15:04:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7A33C3A183B
	for <common-issues@hadoop.apache.org>; Thu, 10 Mar 2011 15:03:59 +0000 (UTC)
Date: Thu, 10 Mar 2011 15:03:59 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <681550845.10789.1299769439497.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1359805310.9666.1299717240403.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7177) CodecPool should report which
 compressor it is using
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Allen Wittenauer updated HADOOP-7177:
-------------------------------------

    Affects Version/s: 0.23.0
                       0.20.2
                       0.21.0
        Fix Version/s: 0.23.0
                       0.22.0
                       0.21.1

> CodecPool should report which compressor it is using
> ----------------------------------------------------
>
>                 Key: HADOOP-7177
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7177
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: native
>    Affects Versions: 0.20.2, 0.21.0, 0.23.0
>            Reporter: Allen Wittenauer
>            Assignee: Allen Wittenauer
>            Priority: Trivial
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: hadoop-7177.txt
>
>
> Certain native compression libraries are overly verbose causing confusion while reading the task logs.  Let's actually say which compressor we got when we report it in the task logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13486-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 15:04:22 2011
Return-Path: <common-issues-return-13486-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45677 invoked from network); 10 Mar 2011 15:04:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 15:04:21 -0000
Received: (qmail 61664 invoked by uid 500); 10 Mar 2011 15:04:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61617 invoked by uid 500); 10 Mar 2011 15:04:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61388 invoked by uid 99); 10 Mar 2011 15:04:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 15:04:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 15:04:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 976433A1846
	for <common-issues@hadoop.apache.org>; Thu, 10 Mar 2011 15:03:59 +0000 (UTC)
Date: Thu, 10 Mar 2011 15:03:59 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1801897752.10793.1299769439616.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1359805310.9666.1299717240403.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7177) CodecPool should report which
 compressor it is using
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Allen Wittenauer updated HADOOP-7177:
-------------------------------------

    Status: Patch Available  (was: Open)

> CodecPool should report which compressor it is using
> ----------------------------------------------------
>
>                 Key: HADOOP-7177
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7177
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: native
>    Affects Versions: 0.21.0, 0.20.2, 0.23.0
>            Reporter: Allen Wittenauer
>            Assignee: Allen Wittenauer
>            Priority: Trivial
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: hadoop-7177.txt
>
>
> Certain native compression libraries are overly verbose causing confusion while reading the task logs.  Let's actually say which compressor we got when we report it in the task logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13485-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 15:04:22 2011
Return-Path: <common-issues-return-13485-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45651 invoked from network); 10 Mar 2011 15:04:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 15:04:21 -0000
Received: (qmail 61443 invoked by uid 500); 10 Mar 2011 15:04:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61370 invoked by uid 500); 10 Mar 2011 15:04:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61262 invoked by uid 99); 10 Mar 2011 15:04:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 15:04:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 15:04:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 89ABD3A1840
	for <common-issues@hadoop.apache.org>; Thu, 10 Mar 2011 15:03:59 +0000 (UTC)
Date: Thu, 10 Mar 2011 15:03:59 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2088361692.10791.1299769439560.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1359805310.9666.1299717240403.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7177) CodecPool should report which
 compressor it is using
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Allen Wittenauer updated HADOOP-7177:
-------------------------------------

    Status: Open  (was: Patch Available)

> CodecPool should report which compressor it is using
> ----------------------------------------------------
>
>                 Key: HADOOP-7177
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7177
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: native
>    Affects Versions: 0.21.0, 0.20.2, 0.23.0
>            Reporter: Allen Wittenauer
>            Assignee: Allen Wittenauer
>            Priority: Trivial
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: hadoop-7177.txt
>
>
> Certain native compression libraries are overly verbose causing confusion while reading the task logs.  Let's actually say which compressor we got when we report it in the task logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13487-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 15:30:24 2011
Return-Path: <common-issues-return-13487-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22488 invoked from network); 10 Mar 2011 15:30:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 15:30:23 -0000
Received: (qmail 15105 invoked by uid 500); 10 Mar 2011 15:30:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15078 invoked by uid 500); 10 Mar 2011 15:30:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15070 invoked by uid 99); 10 Mar 2011 15:30:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 15:30:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 15:30:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8ACEC3A24C9
	for <common-issues@hadoop.apache.org>; Thu, 10 Mar 2011 15:29:59 +0000 (UTC)
Date: Thu, 10 Mar 2011 15:29:59 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1385994617.10843.1299770999565.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <599378200.2869.1299537719453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7167) Allow using a file to exclude
 certain tests from build
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005104#comment-13005104 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7167:
------------------------------------------------

Cygwin actually has {{/dev/null}}.  The following works.

{noformat}
$ ls > /dev/null
{noformat}

Interestingly, {{ls}} shows it only if the full path is given.

{noformat}
$ ls -l /dev/null
crw-rw-rw- 1 tsz Users 1, 3 Nov 30  2006 /dev/null

$ ls -l /dev
total 4
lrwxrwxrwx 1 tsz Users 13 Oct 23  2009 fd -> /proc/self/fd
lrwxrwxrwx 1 tsz Users 15 Oct 23  2009 stderr -> /proc/self/fd/2
lrwxrwxrwx 1 tsz Users 15 Oct 23  2009 stdin -> /proc/self/fd/0
lrwxrwxrwx 1 tsz Users 15 Oct 23  2009 stdout -> /proc/self/fd/1
{noformat}

> Allow using a file to exclude certain tests from build
> ------------------------------------------------------
>
>                 Key: HADOOP-7167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7167
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7167.txt
>
>
> It would be nice to be able to exclude certain tests when running builds. For example, when a test is "known flaky", you may want to exclude it from the main Hudson job, but not actually disable it in the codebase (so that it still runs as part of another Hudson job, for example).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13488-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 20:04:21 2011
Return-Path: <common-issues-return-13488-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65560 invoked from network); 10 Mar 2011 20:04:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 20:04:21 -0000
Received: (qmail 82133 invoked by uid 500); 10 Mar 2011 20:04:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82103 invoked by uid 500); 10 Mar 2011 20:04:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82095 invoked by uid 99); 10 Mar 2011 20:04:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 20:04:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 20:04:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8C5FC3A26B9
	for <common-issues@hadoop.apache.org>; Thu, 10 Mar 2011 20:03:59 +0000 (UTC)
Date: Thu, 10 Mar 2011 20:03:59 +0000 (UTC)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2038449206.11734.1299787439571.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1878551188.5886.1297277337604.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch
 the calls in DataStorage to FileUtil.createHardLink(), so we call it once
 per directory instead of once per file
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005299#comment-13005299 ] 

Hairong Kuang commented on HADOOP-7133:
---------------------------------------

Sure, please go ahead and commit it. I already +1 it. :)

> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7133
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7133
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.20.2
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>             Fix For: 0.22.0
>
>         Attachments: HDFS-1445-trunk.v23_common_1-of-3.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS.  This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS.  It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13489-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 20:11:33 2011
Return-Path: <common-issues-return-13489-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82822 invoked from network); 10 Mar 2011 20:11:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 20:11:33 -0000
Received: (qmail 98283 invoked by uid 500); 10 Mar 2011 20:11:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98191 invoked by uid 500); 10 Mar 2011 20:11:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98117 invoked by uid 99); 10 Mar 2011 20:11:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 20:11:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 20:11:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 534293A2C3A
	for <common-issues@hadoop.apache.org>; Thu, 10 Mar 2011 20:11:05 +0000 (UTC)
Date: Thu, 10 Mar 2011 20:11:05 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <818390768.11760.1299787865337.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7179) Improve HDFS startup scripts
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Improve HDFS startup scripts
----------------------------

                 Key: HADOOP-7179
                 URL: https://issues.apache.org/jira/browse/HADOOP-7179
             Project: Hadoop Common
          Issue Type: Improvement
          Components: scripts
            Reporter: Tanping Wang
            Priority: Minor


Startup scripts should start namenodes, secondary namenodes and datanodes on hosts returned by getConfig (new feature of HDFS Federation).  This jira is Hdfs counterpart of HDFS-1703.  This patch makes changes to hadoop-config.sh and slaves.sh where they belong to COMMON project.  And HDFS-1703 modifies scripts, hdfs, start-hdfs.sh and stop-hdfs.sh where they go to HDFS project.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13490-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 20:13:21 2011
Return-Path: <common-issues-return-13490-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83243 invoked from network); 10 Mar 2011 20:13:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 20:13:21 -0000
Received: (qmail 3971 invoked by uid 500); 10 Mar 2011 20:13:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3944 invoked by uid 500); 10 Mar 2011 20:13:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3935 invoked by uid 99); 10 Mar 2011 20:13:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 20:13:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 20:13:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6CC243A2D33
	for <common-issues@hadoop.apache.org>; Thu, 10 Mar 2011 20:12:59 +0000 (UTC)
Date: Thu, 10 Mar 2011 20:12:59 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1739167197.11766.1299787979441.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <818390768.11760.1299787865337.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7179) Improve HDFS startup scripts
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tanping Wang updated HADOOP-7179:
---------------------------------

    Attachment: HDFS-1703.patch-common

> Improve HDFS startup scripts
> ----------------------------
>
>                 Key: HADOOP-7179
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7179
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Tanping Wang
>            Priority: Minor
>         Attachments: HDFS-1703.patch-common
>
>
> Startup scripts should start namenodes, secondary namenodes and datanodes on hosts returned by getConfig (new feature of HDFS Federation).  This jira is Hdfs counterpart of HDFS-1703.  This patch makes changes to hadoop-config.sh and slaves.sh where they belong to COMMON project.  And HDFS-1703 modifies scripts, hdfs, start-hdfs.sh and stop-hdfs.sh where they go to HDFS project.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13492-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 21:21:36 2011
Return-Path: <common-issues-return-13492-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96345 invoked from network); 10 Mar 2011 21:21:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 21:21:30 -0000
Received: (qmail 12708 invoked by uid 500); 10 Mar 2011 21:21:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12676 invoked by uid 500); 10 Mar 2011 21:21:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12647 invoked by uid 99); 10 Mar 2011 21:21:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 21:21:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 21:21:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 679BC3A37AB
	for <common-issues@hadoop.apache.org>; Thu, 10 Mar 2011 21:20:59 +0000 (UTC)
Date: Thu, 10 Mar 2011 21:20:59 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <920656896.11906.1299792059420.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <818390768.11760.1299787865337.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7179) Improve HDFS startup scripts
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tanping Wang updated HADOOP-7179:
---------------------------------

    Attachment:     (was: HDFS-1703.patch-common)

> Improve HDFS startup scripts
> ----------------------------
>
>                 Key: HADOOP-7179
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7179
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Tanping Wang
>            Priority: Minor
>         Attachments: HADOOP-7179.patch
>
>
> Startup scripts should start namenodes, secondary namenodes and datanodes on hosts returned by getConfig (new feature of HDFS Federation).  This jira is Hdfs counterpart of HDFS-1703.  This patch makes changes to hadoop-config.sh and slaves.sh where they belong to COMMON project.  And HDFS-1703 modifies scripts, hdfs, start-hdfs.sh and stop-hdfs.sh where they go to HDFS project.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13491-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 21:21:36 2011
Return-Path: <common-issues-return-13491-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96342 invoked from network); 10 Mar 2011 21:21:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 21:21:30 -0000
Received: (qmail 12494 invoked by uid 500); 10 Mar 2011 21:21:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12348 invoked by uid 500); 10 Mar 2011 21:21:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12305 invoked by uid 99); 10 Mar 2011 21:21:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 21:21:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 21:21:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6FBEF3A37AC
	for <common-issues@hadoop.apache.org>; Thu, 10 Mar 2011 21:20:59 +0000 (UTC)
Date: Thu, 10 Mar 2011 21:20:59 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <860774437.11907.1299792059454.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <818390768.11760.1299787865337.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7179) Improve HDFS startup scripts
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tanping Wang updated HADOOP-7179:
---------------------------------

    Attachment: HADOOP-7179.patch

> Improve HDFS startup scripts
> ----------------------------
>
>                 Key: HADOOP-7179
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7179
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Tanping Wang
>            Priority: Minor
>         Attachments: HADOOP-7179.patch
>
>
> Startup scripts should start namenodes, secondary namenodes and datanodes on hosts returned by getConfig (new feature of HDFS Federation).  This jira is Hdfs counterpart of HDFS-1703.  This patch makes changes to hadoop-config.sh and slaves.sh where they belong to COMMON project.  And HDFS-1703 modifies scripts, hdfs, start-hdfs.sh and stop-hdfs.sh where they go to HDFS project.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13493-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 22:14:21 2011
Return-Path: <common-issues-return-13493-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75393 invoked from network); 10 Mar 2011 22:14:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 22:14:21 -0000
Received: (qmail 96296 invoked by uid 500); 10 Mar 2011 22:14:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96270 invoked by uid 500); 10 Mar 2011 22:14:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96262 invoked by uid 99); 10 Mar 2011 22:14:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 22:14:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 22:14:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8B4333A2868
	for <common-issues@hadoop.apache.org>; Thu, 10 Mar 2011 22:13:59 +0000 (UTC)
Date: Thu, 10 Mar 2011 22:13:59 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1975706251.11966.1299795239567.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7180) Improve CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Improve CommandFormat
---------------------

                 Key: HADOOP-7180
                 URL: https://issues.apache.org/jira/browse/HADOOP-7180
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Daryn Sharp
             Fix For: 0.22.0


CommandFormat currently takes an array and offset for parsing and returns a list of arguments.  It'd be much more convenient to have it process a list too.  It would also be nice to differentiate between too few and too many args instead of the generic "Illegal number of arguments".  Finally, CommandFormat is completely devoid of tests.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13494-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 22:18:21 2011
Return-Path: <common-issues-return-13494-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84057 invoked from network); 10 Mar 2011 22:18:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 22:18:21 -0000
Received: (qmail 3221 invoked by uid 500); 10 Mar 2011 22:18:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3125 invoked by uid 500); 10 Mar 2011 22:18:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3116 invoked by uid 99); 10 Mar 2011 22:18:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 22:18:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 22:18:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C3EBA3A2A3B
	for <common-issues@hadoop.apache.org>; Thu, 10 Mar 2011 22:17:59 +0000 (UTC)
Date: Thu, 10 Mar 2011 22:17:59 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1837181934.11973.1299795479796.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1975706251.11966.1299795239567.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7180) Improve CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7180:
--------------------------------

    Attachment: HADOOP-7180.patch

Add a slew of tests, add exceptions for too many/few, allowing parsing a list.  Backwards compatibility.

> Improve CommandFormat
> ---------------------
>
>                 Key: HADOOP-7180
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7180
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Daryn Sharp
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7180.patch
>
>
> CommandFormat currently takes an array and offset for parsing and returns a list of arguments.  It'd be much more convenient to have it process a list too.  It would also be nice to differentiate between too few and too many args instead of the generic "Illegal number of arguments".  Finally, CommandFormat is completely devoid of tests.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13495-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 22:20:21 2011
Return-Path: <common-issues-return-13495-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91863 invoked from network); 10 Mar 2011 22:20:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 22:20:21 -0000
Received: (qmail 5252 invoked by uid 500); 10 Mar 2011 22:20:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5225 invoked by uid 500); 10 Mar 2011 22:20:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5217 invoked by uid 99); 10 Mar 2011 22:20:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 22:20:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 22:20:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B807F3A2AD2
	for <common-issues@hadoop.apache.org>; Thu, 10 Mar 2011 22:19:59 +0000 (UTC)
Date: Thu, 10 Mar 2011 22:19:59 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <885576611.11975.1299795599750.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1975706251.11966.1299795239567.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7180) Improve CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7180:
--------------------------------

    Status: Patch Available  (was: Open)

> Improve CommandFormat
> ---------------------
>
>                 Key: HADOOP-7180
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7180
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Daryn Sharp
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7180.patch
>
>
> CommandFormat currently takes an array and offset for parsing and returns a list of arguments.  It'd be much more convenient to have it process a list too.  It would also be nice to differentiate between too few and too many args instead of the generic "Illegal number of arguments".  Finally, CommandFormat is completely devoid of tests.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13496-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 23:24:21 2011
Return-Path: <common-issues-return-13496-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4249 invoked from network); 10 Mar 2011 23:24:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 23:24:21 -0000
Received: (qmail 392 invoked by uid 500); 10 Mar 2011 23:24:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 359 invoked by uid 500); 10 Mar 2011 23:24:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 351 invoked by uid 99); 10 Mar 2011 23:24:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 23:24:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 23:24:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7F8F93A30D5
	for <common-issues@hadoop.apache.org>; Thu, 10 Mar 2011 23:23:59 +0000 (UTC)
Date: Thu, 10 Mar 2011 23:23:59 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1230611280.12168.1299799439519.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <645678321.2624.1299535019917.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7166) DaemonFactory should be moved from
 HDFS to common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005395#comment-13005395 ] 

Jitendra Nath Pandey commented on HADOOP-7166:
----------------------------------------------

>Is the motivation for this for MapReduce to use DaemonFactory?

The motivation is that it is not specific to hdfs and could be used outside hdfs as well, and Daemon.java in common seems to be a logical place to put DaemonFactory. 

> DaemonFactory should be moved from HDFS to common
> -------------------------------------------------
>
>                 Key: HADOOP-7166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7166
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-7166.1.patch
>
>
> DaemonFactory class is defined in hdfs util. common would be a better place for this class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13497-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 23:36:21 2011
Return-Path: <common-issues-return-13497-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59198 invoked from network); 10 Mar 2011 23:36:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 23:36:21 -0000
Received: (qmail 12157 invoked by uid 500); 10 Mar 2011 23:36:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12117 invoked by uid 500); 10 Mar 2011 23:36:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12109 invoked by uid 99); 10 Mar 2011 23:36:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 23:36:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 23:36:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9D9243A342F
	for <common-issues@hadoop.apache.org>; Thu, 10 Mar 2011 23:35:59 +0000 (UTC)
Date: Thu, 10 Mar 2011 23:35:59 +0000 (UTC)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <458008222.12194.1299800159642.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1878551188.5886.1297277337604.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch the
 calls in DataStorage to FileUtil.createHardLink(), so we call it once per
 directory instead of once per file
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-7133:
--------------------------------

       Resolution: Fixed
    Fix Version/s:     (was: 0.22.0)
                   0.23.0
           Status: Resolved  (was: Patch Available)

I've committed this.  Resolving as fixed.  Matt, please open a JIRA for Hairong's suggestion.  Thanks.

> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7133
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7133
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.20.2
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: HDFS-1445-trunk.v23_common_1-of-3.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS.  This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS.  It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13498-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 23:54:22 2011
Return-Path: <common-issues-return-13498-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5443 invoked from network); 10 Mar 2011 23:54:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 23:54:21 -0000
Received: (qmail 30731 invoked by uid 500); 10 Mar 2011 23:54:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30679 invoked by uid 500); 10 Mar 2011 23:54:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30671 invoked by uid 99); 10 Mar 2011 23:54:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 23:54:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 23:54:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1EDFE3A392B
	for <common-issues@hadoop.apache.org>; Thu, 10 Mar 2011 23:54:00 +0000 (UTC)
Date: Thu, 10 Mar 2011 23:54:00 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1605344812.12239.1299801240123.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <130797695.15062.1298594983152.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7154) Should set MALLOC_ARENA_MAX in
 hadoop-env.sh
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005410#comment-13005410 ] 

Hudson commented on HADOOP-7154:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #523 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/523/])
    

> Should set MALLOC_ARENA_MAX in hadoop-env.sh
> --------------------------------------------
>
>                 Key: HADOOP-7154
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7154
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7154.txt
>
>
> New versions of glibc present in RHEL6 include a new arena allocator design. In several clusters we've seen this new allocator cause huge amounts of virtual memory to be used, since when multiple threads perform allocations, they each get their own memory arena. On a 64-bit system, these arenas are 64M mappings, and the maximum number of arenas is 8 times the number of cores. We've observed a DN process using 14GB of vmem for only 300M of resident set. This causes all kinds of nasty issues for obvious reasons.
> Setting MALLOC_ARENA_MAX to a low number will restrict the number of memory arenas and bound the virtual memory, with no noticeable downside in performance - we've been recommending MALLOC_ARENA_MAX=4. We should set this in hadoop-env.sh to avoid this issue as RHEL6 becomes more and more common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13499-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 23:54:24 2011
Return-Path: <common-issues-return-13499-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5516 invoked from network); 10 Mar 2011 23:54:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 23:54:23 -0000
Received: (qmail 31242 invoked by uid 500); 10 Mar 2011 23:54:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31187 invoked by uid 500); 10 Mar 2011 23:54:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31120 invoked by uid 99); 10 Mar 2011 23:54:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 23:54:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 23:54:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B1A333A391D
	for <common-issues@hadoop.apache.org>; Thu, 10 Mar 2011 23:53:59 +0000 (UTC)
Date: Thu, 10 Mar 2011 23:53:59 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <530423876.12225.1299801239724.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <824415802.6352.1299019897038.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7159) RPC server should log the client
 hostname when read exception happened
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005408#comment-13005408 ] 

Hudson commented on HADOOP-7159:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #523 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/523/])
    

> RPC server should log the client hostname when read exception happened
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-7159
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7159
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7159.txt
>
>
> This makes find mismatched clients easier

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13500-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 23:54:24 2011
Return-Path: <common-issues-return-13500-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5553 invoked from network); 10 Mar 2011 23:54:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 23:54:24 -0000
Received: (qmail 31447 invoked by uid 500); 10 Mar 2011 23:54:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31399 invoked by uid 500); 10 Mar 2011 23:54:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31285 invoked by uid 99); 10 Mar 2011 23:54:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 23:54:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 23:54:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CC8E73A3921
	for <common-issues@hadoop.apache.org>; Thu, 10 Mar 2011 23:53:59 +0000 (UTC)
Date: Thu, 10 Mar 2011 23:53:59 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1577538168.12229.1299801239834.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <599378200.2869.1299537719453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7167) Allow using a file to exclude
 certain tests from build
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005409#comment-13005409 ] 

Hudson commented on HADOOP-7167:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #523 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/523/])
    

> Allow using a file to exclude certain tests from build
> ------------------------------------------------------
>
>                 Key: HADOOP-7167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7167
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7167.txt
>
>
> It would be nice to be able to exclude certain tests when running builds. For example, when a test is "known flaky", you may want to exclude it from the main Hudson job, but not actually disable it in the codebase (so that it still runs as part of another Hudson job, for example).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13501-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 23:54:41 2011
Return-Path: <common-issues-return-13501-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5657 invoked from network); 10 Mar 2011 23:54:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 23:54:41 -0000
Received: (qmail 31678 invoked by uid 500); 10 Mar 2011 23:54:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31654 invoked by uid 500); 10 Mar 2011 23:54:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31646 invoked by uid 99); 10 Mar 2011 23:54:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 23:54:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 23:54:40 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5A1CC3A3932
	for <common-issues@hadoop.apache.org>; Thu, 10 Mar 2011 23:54:01 +0000 (UTC)
Date: Thu, 10 Mar 2011 23:54:01 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1937564759.12246.1299801241366.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1846874015.7578.1296749909031.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7131) set() and toString Methods of the
 org.apache.hadoop.io.Text class does not include the root exception, in the
 wrapping RuntimeException.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005412#comment-13005412 ] 

Hudson commented on HADOOP-7131:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #523 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/523/])
    

> set() and toString Methods of the org.apache.hadoop.io.Text class does not include the root exception, in the wrapping RuntimeException.
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7131
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7131
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7131.patch
>
>
>  In below code snippets, we can include e, instead of e.toString(), so that caller can get complete trace.
> 1) 
>    /** Set to contain the contents of a string.
>    */
>   public void set(String string) {
>     try {
>       ByteBuffer bb = encode(string, true);
>       bytes = bb.array();
>       length = bb.limit();
>     }catch(CharacterCodingException e) {
>       throw new RuntimeException("Should not have happened ",e.toString());
>     }
>   } 
> 2)
>    public String toString() {
>     try {
>       return decode(bytes, 0, length);
>     } catch (CharacterCodingException e) {
>       throw new RuntimeException("Should not have happened ",e.toString());
>     }
>   }

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13502-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 23:54:44 2011
Return-Path: <common-issues-return-13502-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5701 invoked from network); 10 Mar 2011 23:54:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 23:54:44 -0000
Received: (qmail 32658 invoked by uid 500); 10 Mar 2011 23:54:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32621 invoked by uid 500); 10 Mar 2011 23:54:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32577 invoked by uid 99); 10 Mar 2011 23:54:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 23:54:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 23:54:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2CE693A392D
	for <common-issues@hadoop.apache.org>; Thu, 10 Mar 2011 23:54:00 +0000 (UTC)
Date: Thu, 10 Mar 2011 23:54:00 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1669855453.12241.1299801240180.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1878551188.5886.1297277337604.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch
 the calls in DataStorage to FileUtil.createHardLink(), so we call it once
 per directory instead of once per file
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005411#comment-13005411 ] 

Hudson commented on HADOOP-7133:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #523 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/523/])
    HADOOP-7133. Batch the calls in DataStorage to FileUtil.createHardLink().  Contributed by Matt Foley.


> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7133
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7133
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.20.2
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: HDFS-1445-trunk.v23_common_1-of-3.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS.  This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS.  It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13503-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 10 23:54:45 2011
Return-Path: <common-issues-return-13503-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5732 invoked from network); 10 Mar 2011 23:54:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Mar 2011 23:54:44 -0000
Received: (qmail 33018 invoked by uid 500); 10 Mar 2011 23:54:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32985 invoked by uid 500); 10 Mar 2011 23:54:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32946 invoked by uid 99); 10 Mar 2011 23:54:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 23:54:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Mar 2011 23:54:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 679893A3934
	for <common-issues@hadoop.apache.org>; Thu, 10 Mar 2011 23:54:01 +0000 (UTC)
Date: Thu, 10 Mar 2011 23:54:01 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1681494637.12248.1299801241421.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-6912) Guard against NPE when calling
 UGI.isLoginKeytabBased()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005413#comment-13005413 ] 

Hudson commented on HADOOP-6912:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #523 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/523/])
    

> Guard against NPE when calling UGI.isLoginKeytabBased()
> -------------------------------------------------------
>
>                 Key: HADOOP-6912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6912
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>             Fix For: 0.23.0
>
>         Attachments: c6912-01.patch
>
>
> NPE can happen when isLoginKeytabBased() is called before a login is performed. See MAPREDUCE-1992 for an example.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13504-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 00:12:22 2011
Return-Path: <common-issues-return-13504-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60842 invoked from network); 11 Mar 2011 00:12:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 00:12:21 -0000
Received: (qmail 50019 invoked by uid 500); 11 Mar 2011 00:12:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49970 invoked by uid 500); 11 Mar 2011 00:12:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49765 invoked by uid 99); 11 Mar 2011 00:12:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 00:12:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 00:12:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C9F3B3A3E3A
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 00:11:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 00:11:59 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <762485406.12281.1299802319824.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <645678321.2624.1299535019917.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7166) DaemonFactory should be moved from
 HDFS to common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005423#comment-13005423 ] 

Suresh Srinivas commented on HADOOP-7166:
-----------------------------------------

Jitendra, when you say DaemonFactory should be moved from HDFS to common, you mean HDFS federation branch right?

> DaemonFactory should be moved from HDFS to common
> -------------------------------------------------
>
>                 Key: HADOOP-7166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7166
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-7166.1.patch
>
>
> DaemonFactory class is defined in hdfs util. common would be a better place for this class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13505-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 00:12:23 2011
Return-Path: <common-issues-return-13505-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60916 invoked from network); 11 Mar 2011 00:12:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 00:12:23 -0000
Received: (qmail 50299 invoked by uid 500); 11 Mar 2011 00:12:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50272 invoked by uid 500); 11 Mar 2011 00:12:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50263 invoked by uid 99); 11 Mar 2011 00:12:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 00:12:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 00:12:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 884773A3E31
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 00:11:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 00:11:59 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1081049168.12272.1299802319554.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <645678321.2624.1299535019917.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7166) DaemonFactory should be moved from
 HDFS to common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005421#comment-13005421 ] 

Suresh Srinivas commented on HADOOP-7166:
-----------------------------------------

+1 for the change.

> DaemonFactory should be moved from HDFS to common
> -------------------------------------------------
>
>                 Key: HADOOP-7166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7166
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-7166.1.patch
>
>
> DaemonFactory class is defined in hdfs util. common would be a better place for this class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13507-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 01:11:21 2011
Return-Path: <common-issues-return-13507-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57067 invoked from network); 11 Mar 2011 01:11:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 01:11:21 -0000
Received: (qmail 93287 invoked by uid 500); 11 Mar 2011 01:11:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93086 invoked by uid 500); 11 Mar 2011 01:11:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93068 invoked by uid 99); 11 Mar 2011 01:11:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 01:11:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 01:11:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C49643A3EFC
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 01:10:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 01:10:59 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <179665913.12425.1299805859802.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <645678321.2624.1299535019917.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7166) DaemonFactory should be moved from
 HDFS to common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7166:
-----------------------------------------

    Hadoop Flags: [Reviewed]
          Status: Patch Available  (was: Open)

> DaemonFactory should be moved from HDFS to common
> -------------------------------------------------
>
>                 Key: HADOOP-7166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7166
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-7166.1.patch
>
>
> DaemonFactory class is defined in hdfs util. common would be a better place for this class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13506-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 01:11:21 2011
Return-Path: <common-issues-return-13506-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57084 invoked from network); 11 Mar 2011 01:11:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 01:11:21 -0000
Received: (qmail 93127 invoked by uid 500); 11 Mar 2011 01:11:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93083 invoked by uid 500); 11 Mar 2011 01:11:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93067 invoked by uid 99); 11 Mar 2011 01:11:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 01:11:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 01:11:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B6A3D3A3EFA
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 01:10:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 01:10:59 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <317060677.12423.1299805859744.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <645678321.2624.1299535019917.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7166) DaemonFactory should be moved from
 HDFS to common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7166:
-----------------------------------------

    Status: Open  (was: Patch Available)

> DaemonFactory should be moved from HDFS to common
> -------------------------------------------------
>
>                 Key: HADOOP-7166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7166
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-7166.1.patch
>
>
> DaemonFactory class is defined in hdfs util. common would be a better place for this class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13508-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 01:11:22 2011
Return-Path: <common-issues-return-13508-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57096 invoked from network); 11 Mar 2011 01:11:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 01:11:21 -0000
Received: (qmail 93514 invoked by uid 500); 11 Mar 2011 01:11:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93478 invoked by uid 500); 11 Mar 2011 01:11:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93125 invoked by uid 99); 11 Mar 2011 01:11:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 01:11:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 01:11:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A6CCA3A3EF7
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 01:10:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 01:10:59 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <223076989.12421.1299805859679.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <645678321.2624.1299535019917.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7166) DaemonFactory should be moved from
 HDFS to common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005442#comment-13005442 ] 

Jitendra Nath Pandey commented on HADOOP-7166:
----------------------------------------------

> you mean HDFS federation branch right?

That is correct. Currently DaemonFactory is committed to the federation branch only. This patch adds it to the common branch and federation branch code will be changed to use the DaemonFactory from common (HDFS-1730).

> DaemonFactory should be moved from HDFS to common
> -------------------------------------------------
>
>                 Key: HADOOP-7166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7166
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-7166.1.patch
>
>
> DaemonFactory class is defined in hdfs util. common would be a better place for this class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13509-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 04:41:23 2011
Return-Path: <common-issues-return-13509-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29766 invoked from network); 11 Mar 2011 04:41:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 04:41:23 -0000
Received: (qmail 99979 invoked by uid 500); 11 Mar 2011 04:41:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99958 invoked by uid 500); 11 Mar 2011 04:41:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99940 invoked by uid 99); 11 Mar 2011 04:41:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 04:41:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 04:41:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7E9F23A3A5F
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 04:41:00 +0000 (UTC)
Date: Fri, 11 Mar 2011 04:41:00 +0000 (UTC)
From: "Joep Rottinghuis (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1073276735.12635.1299818460515.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] Commented: (HADOOP-7106) Re-organize hadoop subversion
 layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005480#comment-13005480 ] 

Joep Rottinghuis commented on HADOOP-7106:
------------------------------------------

What is the plan for the Git repos?
I'm hoping that we can stick a fork in the separate repos and simply maintain everything in the hadoop-common one, as that already has the three projects on the 0.21* branches.
Having three Git repos->three Jenkins jobs and coordinating them is a pain.

> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13510-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 05:11:23 2011
Return-Path: <common-issues-return-13510-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29514 invoked from network); 11 Mar 2011 05:11:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 05:11:22 -0000
Received: (qmail 30460 invoked by uid 500); 11 Mar 2011 05:11:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30034 invoked by uid 500); 11 Mar 2011 05:11:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29881 invoked by uid 99); 11 Mar 2011 05:11:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 05:11:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 05:11:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7C9683A3289
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 05:10:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 05:10:59 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1168956648.12687.1299820259506.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005493#comment-13005493 ] 

Hadoop QA commented on HADOOP-7156:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12473179/hadoop-7156.txt
  against trunk revision 1080396.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/306//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/306//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/306//console

This message is automatically generated.

> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13511-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 07:33:22 2011
Return-Path: <common-issues-return-13511-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95074 invoked from network); 11 Mar 2011 07:33:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 07:33:21 -0000
Received: (qmail 43998 invoked by uid 500); 11 Mar 2011 07:33:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43947 invoked by uid 500); 11 Mar 2011 07:33:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43939 invoked by uid 99); 11 Mar 2011 07:33:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 07:33:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 07:33:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8C5B33A38F8
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 07:32:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 07:32:59 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <939088155.12860.1299828779571.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005535#comment-13005535 ] 

Eli Collins commented on HADOOP-7156:
-------------------------------------

+1  for the latest patch

> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13512-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 11:15:23 2011
Return-Path: <common-issues-return-13512-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91936 invoked from network); 11 Mar 2011 11:15:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 11:15:23 -0000
Received: (qmail 32626 invoked by uid 500); 11 Mar 2011 11:15:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32591 invoked by uid 500); 11 Mar 2011 11:15:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32583 invoked by uid 99); 11 Mar 2011 11:15:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 11:15:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 11:15:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6AF0B3A3384
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 11:14:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 11:14:59 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <499779821.13193.1299842099434.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1878551188.5886.1297277337604.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch
 the calls in DataStorage to FileUtil.createHardLink(), so we call it once
 per directory instead of once per file
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005601#comment-13005601 ] 

Hudson commented on HADOOP-7133:
--------------------------------

Integrated in Hadoop-Common-trunk #627 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/627/])
    HADOOP-7133. Batch the calls in DataStorage to FileUtil.createHardLink().  Contributed by Matt Foley.


> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7133
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7133
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.20.2
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: HDFS-1445-trunk.v23_common_1-of-3.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS.  This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS.  It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13513-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 15:40:21 2011
Return-Path: <common-issues-return-13513-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70804 invoked from network); 11 Mar 2011 15:40:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 15:40:21 -0000
Received: (qmail 96078 invoked by uid 500); 11 Mar 2011 15:40:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96048 invoked by uid 500); 11 Mar 2011 15:40:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96040 invoked by uid 99); 11 Mar 2011 15:40:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 15:40:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 15:40:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 866863A43FA
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 15:39:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 15:39:59 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1098394576.13604.1299857999547.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Assigned: (HADOOP-7178) copyToLocal API is creating .crc
 files in local, even after setting verifyChecksum to false at client side.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uma Maheswara Rao G reassigned HADOOP-7178:
-------------------------------------------

    Assignee: Uma Maheswara Rao G

> copyToLocal API is creating .crc files in local, even after setting verifyChecksum to false at client side.
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7178
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7178
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>
> When we copy the files from DFS to local, it is creating the .crc file in local filesystem for the verification of checksum even if we disable the checksum verification at client side.
>         When user does not want to do any checksum verification, then what will be the use in creating of these files in local file system.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13514-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 15:50:24 2011
Return-Path: <common-issues-return-13514-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17801 invoked from network); 11 Mar 2011 15:50:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 15:50:23 -0000
Received: (qmail 19756 invoked by uid 500); 11 Mar 2011 15:50:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19724 invoked by uid 500); 11 Mar 2011 15:50:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19716 invoked by uid 99); 11 Mar 2011 15:50:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 15:50:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 15:50:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7B4A83A4819
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 15:49:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 15:49:59 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <807446361.13619.1299858599501.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7178) copyToLocal API is creating .crc
 files in local, even after setting verifyChecksum to false at client side.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7178?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D130=
05685#comment-13005685 ]=20

Uma Maheswara Rao G commented on HADOOP-7178:
---------------------------------------------

FileSystem already has the API setVerifyChecksum. As per current implementa=
tion this flag will be set in DistributedFilesystem.Based on this flag, che=
cksum verification will happen. Bydefault LocalFileSystem will be selected =
as  local file system . But LocalFileSystem will create crc files in local =
disk in all the cases because it is CheckSumFileSystem.
But when user explicitly set the verifyCheckSum flag to false, then LocalFi=
leSystem will create the crc files in local disk.
=20
One Option to solve this issue is:
        We can provide an API in FileSystem which will be the getter method=
(getVerifyChecksum) for setVerifyChecksum. DistributedFileSystem can overri=
de this method.

In Filesystem=E2=80=99s copyToLocalFile API, we can use this flag status.
If this flag is disabled, then we can select RawLocalFileSystem as local fi=
le system. This RawLocalFileSystem will not create any of the CRC files.


> copyToLocal API is creating .crc files in local, even after setting verif=
yChecksum to false at client side.
> -------------------------------------------------------------------------=
----------------------------------
>
>                 Key: HADOOP-7178
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7178
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>
> When we copy the files from DFS to local, it is creating the .crc file in=
 local filesystem for the verification of checksum even if we disable the c=
hecksum verification at client side.
>         When user does not want to do any checksum verification, then wha=
t will be the use in creating of these files in local file system.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13515-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 16:38:25 2011
Return-Path: <common-issues-return-13515-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69780 invoked from network); 11 Mar 2011 16:38:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 16:38:23 -0000
Received: (qmail 23829 invoked by uid 500); 11 Mar 2011 16:38:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23738 invoked by uid 500); 11 Mar 2011 16:38:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23720 invoked by uid 99); 11 Mar 2011 16:38:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 16:38:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 16:38:22 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8BEDB3A4A74
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 16:38:01 +0000 (UTC)
Date: Fri, 11 Mar 2011 16:38:01 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1217986099.13764.1299861481570.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1975706251.11966.1299795239567.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7180) Improve CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7180:
-------------------------------------------

    Fix Version/s:     (was: 0.22.0)
           Status: Open  (was: Patch Available)

> Improve CommandFormat
> ---------------------
>
>                 Key: HADOOP-7180
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7180
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Daryn Sharp
>         Attachments: HADOOP-7180.patch
>
>
> CommandFormat currently takes an array and offset for parsing and returns a list of arguments.  It'd be much more convenient to have it process a list too.  It would also be nice to differentiate between too few and too many args instead of the generic "Illegal number of arguments".  Finally, CommandFormat is completely devoid of tests.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13516-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 16:38:25 2011
Return-Path: <common-issues-return-13516-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69792 invoked from network); 11 Mar 2011 16:38:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 16:38:24 -0000
Received: (qmail 23937 invoked by uid 500); 11 Mar 2011 16:38:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23902 invoked by uid 500); 11 Mar 2011 16:38:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23763 invoked by uid 99); 11 Mar 2011 16:38:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 16:38:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 16:38:22 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9CF123A4A77
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 16:38:01 +0000 (UTC)
Date: Fri, 11 Mar 2011 16:38:01 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1640605861.13767.1299861481639.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1975706251.11966.1299795239567.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7180) Improve CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7180:
-------------------------------------------

    Fix Version/s: 0.23.0
         Assignee: Daryn Sharp
           Status: Patch Available  (was: Open)

> Improve CommandFormat
> ---------------------
>
>                 Key: HADOOP-7180
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7180
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7180.patch
>
>
> CommandFormat currently takes an array and offset for parsing and returns a list of arguments.  It'd be much more convenient to have it process a list too.  It would also be nice to differentiate between too few and too many args instead of the generic "Illegal number of arguments".  Finally, CommandFormat is completely devoid of tests.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13517-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 16:38:26 2011
Return-Path: <common-issues-return-13517-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69931 invoked from network); 11 Mar 2011 16:38:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 16:38:25 -0000
Received: (qmail 25154 invoked by uid 500); 11 Mar 2011 16:38:25 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25118 invoked by uid 500); 11 Mar 2011 16:38:25 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25110 invoked by uid 99); 11 Mar 2011 16:38:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 16:38:25 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 16:38:23 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 416F93A4A68
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 16:38:01 +0000 (UTC)
Date: Fri, 11 Mar 2011 16:38:01 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1643860141.13753.1299861481264.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1409102499.8762.1299697139647.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7175) Add isEnabled() to Trash
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7175:
-------------------------------------------

        Fix Version/s:     (was: 0.22.0)
    Affects Version/s:     (was: 0.22.0)
               Status: Open  (was: Patch Available)

> Add isEnabled() to Trash
> ------------------------
>
>                 Key: HADOOP-7175
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7175
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Daryn Sharp
>         Attachments: HADOOP-7175.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The moveToTrash method returns false in a number of cases.  It's not possible to discern if false means an error occurred. In particular, it's not possible to know if the trash is disabled vs. an error occurred.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13518-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 16:38:28 2011
Return-Path: <common-issues-return-13518-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70055 invoked from network); 11 Mar 2011 16:38:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 16:38:27 -0000
Received: (qmail 25409 invoked by uid 500); 11 Mar 2011 16:38:25 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25335 invoked by uid 500); 11 Mar 2011 16:38:25 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25319 invoked by uid 99); 11 Mar 2011 16:38:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 16:38:25 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 16:38:23 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5D03F3A4A6C
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 16:38:01 +0000 (UTC)
Date: Fri, 11 Mar 2011 16:38:01 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1799017579.13757.1299861481377.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1409102499.8762.1299697139647.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7175) Add isEnabled() to Trash
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7175:
-------------------------------------------

    Fix Version/s: 0.23.0
         Assignee: Daryn Sharp
           Status: Patch Available  (was: Open)

> Add isEnabled() to Trash
> ------------------------
>
>                 Key: HADOOP-7175
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7175
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7175.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The moveToTrash method returns false in a number of cases.  It's not possible to discern if false means an error occurred. In particular, it's not possible to know if the trash is disabled vs. an error occurred.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13519-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 18:00:23 2011
Return-Path: <common-issues-return-13519-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20810 invoked from network); 11 Mar 2011 18:00:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 18:00:23 -0000
Received: (qmail 55543 invoked by uid 500); 11 Mar 2011 18:00:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55472 invoked by uid 500); 11 Mar 2011 18:00:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55463 invoked by uid 99); 11 Mar 2011 18:00:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 18:00:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 18:00:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6DEC73A463D
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 17:59:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 17:59:59 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1782554658.13948.1299866399446.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <645678321.2624.1299535019917.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7166) DaemonFactory should be moved from
 HDFS to common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005748#comment-13005748 ] 

Tom White commented on HADOOP-7166:
-----------------------------------

I missed that DaemonFactory was only in the federation branch. I agree that putting it with Daemon (marked @LimitedPrivate) in common makes sense.

> DaemonFactory should be moved from HDFS to common
> -------------------------------------------------
>
>                 Key: HADOOP-7166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7166
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-7166.1.patch
>
>
> DaemonFactory class is defined in hdfs util. common would be a better place for this class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13520-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 18:02:22 2011
Return-Path: <common-issues-return-13520-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37792 invoked from network); 11 Mar 2011 18:02:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 18:02:22 -0000
Received: (qmail 57129 invoked by uid 500); 11 Mar 2011 18:02:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57099 invoked by uid 500); 11 Mar 2011 18:02:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57084 invoked by uid 99); 11 Mar 2011 18:02:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 18:02:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 18:02:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5D2D33A46D4
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 18:02:00 +0000 (UTC)
Date: Fri, 11 Mar 2011 18:02:00 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1381128291.13953.1299866520378.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7181) use ThreadLocal for HardLink command
 buffers
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

use ThreadLocal for HardLink command buffers
--------------------------------------------

                 Key: HADOOP-7181
                 URL: https://issues.apache.org/jira/browse/HADOOP-7181
             Project: Hadoop Common
          Issue Type: Improvement
          Components: util
    Affects Versions: 0.23.0
            Reporter: Matt Foley
            Assignee: Matt Foley
             Fix For: 0.23.0


Referring to HADOOP-7133, Hairong observed that HardLink could be improved if it used ThreadLocal to implement thread-safe re-usable buffers.  We agreed to open it in a new bug so the original patch could proceed.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13521-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 18:16:25 2011
Return-Path: <common-issues-return-13521-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78131 invoked from network); 11 Mar 2011 18:16:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 18:16:24 -0000
Received: (qmail 81148 invoked by uid 500); 11 Mar 2011 18:16:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81089 invoked by uid 500); 11 Mar 2011 18:16:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80873 invoked by uid 99); 11 Mar 2011 18:16:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 18:16:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 18:16:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6F2CE3A4C1F
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 18:15:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 18:15:59 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1070422412.13998.1299867359451.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7182) remove stub implementation of
 HardLink from FileUtil, after fixing any remaining dependencies
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

remove stub implementation of HardLink from FileUtil, after fixing any remaining dependencies
---------------------------------------------------------------------------------------------

                 Key: HADOOP-7182
                 URL: https://issues.apache.org/jira/browse/HADOOP-7182
             Project: Hadoop Common
          Issue Type: Improvement
          Components: util
    Affects Versions: 0.23.0
            Reporter: Matt Foley
            Assignee: Matt Foley
            Priority: Minor
             Fix For: 0.23.0


This is the third part of a refactoring of HardLink:
1. HADOOP-7133 moved HardLink from an inner class of FileUtil to a stand-alone class, but left a stub class in FileUtil to prevent breaking clients in HDFS.
2. HDFS-1445 changes the clients in HDFS to point at the new implementation.
3. After HDFS-1445 is committed, this ticket is to clean up the code by removing the backward-compatibility stub.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13522-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 18:24:22 2011
Return-Path: <common-issues-return-13522-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21175 invoked from network); 11 Mar 2011 18:24:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 18:24:21 -0000
Received: (qmail 96506 invoked by uid 500); 11 Mar 2011 18:24:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96442 invoked by uid 500); 11 Mar 2011 18:24:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96314 invoked by uid 99); 11 Mar 2011 18:24:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 18:24:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 18:24:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BE5C93A4FD9
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 18:23:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 18:23:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1594670430.14027.1299867839776.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7183) WritableComparator.get should not
 cache comparator objects
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

WritableComparator.get should not cache comparator objects
----------------------------------------------------------

                 Key: HADOOP-7183
                 URL: https://issues.apache.org/jira/browse/HADOOP-7183
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.22.0
            Reporter: Todd Lipcon
            Priority: Blocker
             Fix For: 0.22.0


HADOOP-6881 modified WritableComparator.get such that the constructed WritableComparator gets saved back into the static map. This is fine for stateless comparators, but some comparators have per-instance state, and thus this becomes thread-unsafe and causes errors in the shuffle where multiple threads are doing comparisons. An example of a Comparator with per-instance state is WritableComparator itself.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13523-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 18:24:25 2011
Return-Path: <common-issues-return-13523-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21600 invoked from network); 11 Mar 2011 18:24:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 18:24:25 -0000
Received: (qmail 97003 invoked by uid 500); 11 Mar 2011 18:24:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96949 invoked by uid 500); 11 Mar 2011 18:24:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96850 invoked by uid 99); 11 Mar 2011 18:24:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 18:24:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 18:24:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 898913A4FD4
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 18:23:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 18:23:59 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <94757268.14022.1299867839560.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1878551188.5886.1297277337604.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch
 the calls in DataStorage to FileUtil.createHardLink(), so we call it once
 per directory instead of once per file
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005766#comment-13005766 ] 

Matt Foley commented on HADOOP-7133:
------------------------------------

Thanks guys.  HADOOP-7181 opened for Hairong's ThreadLocal suggestion.
HDFS-1445 "submit patch" activated.
HADOOP-7182 opened to clean up the backward compatibility stub from FileUtils.

> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7133
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7133
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.20.2
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: HDFS-1445-trunk.v23_common_1-of-3.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS.  This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS.  It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13524-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 18:46:24 2011
Return-Path: <common-issues-return-13524-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87342 invoked from network); 11 Mar 2011 18:46:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 18:46:23 -0000
Received: (qmail 29647 invoked by uid 500); 11 Mar 2011 18:46:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29618 invoked by uid 500); 11 Mar 2011 18:46:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29610 invoked by uid 99); 11 Mar 2011 18:46:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 18:46:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 18:46:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 95AEB3A470A
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 18:45:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 18:45:59 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <208467215.14057.1299869159609.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005780#comment-13005780 ] 

Allen Wittenauer commented on HADOOP-7156:
------------------------------------------

Removing my -1.

> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13525-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 18:46:24 2011
Return-Path: <common-issues-return-13525-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87378 invoked from network); 11 Mar 2011 18:46:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 18:46:24 -0000
Received: (qmail 29876 invoked by uid 500); 11 Mar 2011 18:46:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29827 invoked by uid 500); 11 Mar 2011 18:46:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29819 invoked by uid 99); 11 Mar 2011 18:46:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 18:46:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 18:46:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A227D3A470C
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 18:45:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 18:45:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <776242083.14059.1299869159660.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7156:
--------------------------------

      Resolution: Fixed
    Release Note: Adds a new configuration hadoop.work.around.non.threadsafe.getpwuid which can be used to enable a mutex around this call to workaround thread-unsafe implementations of getpwuid_r. Users should consult http://wiki.apache.org/hadoop/KnownBrokenPwuidImplementations for a list of such systems.
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13526-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 18:48:21 2011
Return-Path: <common-issues-return-13526-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5134 invoked from network); 11 Mar 2011 18:48:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 18:48:21 -0000
Received: (qmail 30640 invoked by uid 500); 11 Mar 2011 18:48:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30611 invoked by uid 500); 11 Mar 2011 18:48:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30603 invoked by uid 99); 11 Mar 2011 18:48:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 18:48:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 18:48:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6C1883A47B9
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 18:47:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 18:47:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <489980328.14066.1299869279439.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005781#comment-13005781 ] 

Todd Lipcon commented on HADOOP-7156:
-------------------------------------

Thanks. Committed to trunk and 22

> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13527-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 18:56:21 2011
Return-Path: <common-issues-return-13527-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19588 invoked from network); 11 Mar 2011 18:56:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 18:56:21 -0000
Received: (qmail 40846 invoked by uid 500); 11 Mar 2011 18:56:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40815 invoked by uid 500); 11 Mar 2011 18:56:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40807 invoked by uid 99); 11 Mar 2011 18:56:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 18:56:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 18:56:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9FE563A4A72
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 18:55:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 18:55:59 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1450688227.14084.1299869759651.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <599378200.2869.1299537719453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7167) Allow using a file to exclude
 certain tests from build
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005785#comment-13005785 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7167:
------------------------------------------------

Hi Todd, have you got a chance to take a look?  In case you don't have time to work on this, could you revert this, HDFS-1731 and MAPREDUCE-2367?  Many thanks.

> Allow using a file to exclude certain tests from build
> ------------------------------------------------------
>
>                 Key: HADOOP-7167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7167
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7167.txt
>
>
> It would be nice to be able to exclude certain tests when running builds. For example, when a test is "known flaky", you may want to exclude it from the main Hudson job, but not actually disable it in the codebase (so that it still runs as part of another Hudson job, for example).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13528-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 19:04:23 2011
Return-Path: <common-issues-return-13528-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60360 invoked from network); 11 Mar 2011 19:04:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 19:04:22 -0000
Received: (qmail 49700 invoked by uid 500); 11 Mar 2011 19:04:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49630 invoked by uid 500); 11 Mar 2011 19:04:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49398 invoked by uid 99); 11 Mar 2011 19:04:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 19:04:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 19:04:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EF63D3A4D86
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 19:03:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 19:03:59 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <941809286.14122.1299870239977.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005788#comment-13005788 ] 

Hudson commented on HADOOP-7156:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #524 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/524/])
    HADOOP-7156. Workaround for unsafe implementations of getpwuid_r. Contributed by Todd Lipcon.


> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13529-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 19:10:26 2011
Return-Path: <common-issues-return-13529-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73602 invoked from network); 11 Mar 2011 19:10:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 19:10:25 -0000
Received: (qmail 56267 invoked by uid 500); 11 Mar 2011 19:10:25 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56233 invoked by uid 500); 11 Mar 2011 19:10:25 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56225 invoked by uid 99); 11 Mar 2011 19:10:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 19:10:25 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 19:10:25 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2125A3A4FC7
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 19:10:04 +0000 (UTC)
Date: Fri, 11 Mar 2011 19:10:04 +0000 (UTC)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <833104281.14142.1299870604131.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1594670430.14027.1299867839776.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7183) WritableComparator.get should not
 cache comparator objects
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-7183:
---------------------------------

    Fix Version/s: 0.21.1
                   0.20.3

The fix should probably also be merged to the 0.20 and 0.21 branches.

> WritableComparator.get should not cache comparator objects
> ----------------------------------------------------------
>
>                 Key: HADOOP-7183
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7183
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.20.3, 0.21.1, 0.22.0
>
>
> HADOOP-6881 modified WritableComparator.get such that the constructed WritableComparator gets saved back into the static map. This is fine for stateless comparators, but some comparators have per-instance state, and thus this becomes thread-unsafe and causes errors in the shuffle where multiple threads are doing comparisons. An example of a Comparator with per-instance state is WritableComparator itself.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13530-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 19:26:24 2011
Return-Path: <common-issues-return-13530-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21085 invoked from network); 11 Mar 2011 19:26:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 19:26:23 -0000
Received: (qmail 76494 invoked by uid 500); 11 Mar 2011 19:26:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76461 invoked by uid 500); 11 Mar 2011 19:26:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76453 invoked by uid 99); 11 Mar 2011 19:26:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 19:26:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 19:26:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6D4863A43B9
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 19:25:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 19:25:59 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <709466648.14163.1299871559443.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1594670430.14027.1299867839776.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7183) WritableComparator.get should not
 cache comparator objects
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-7183:
------------------------------

    Attachment: HADOOP-7183.patch

The problem is that WritableComparator has a mutable field - DataInputBuffer buffer - which is only used by WritableComparable implementations that *don't* override the optimized binary compare method. IntWritable, Text, etc override this method, so there is no thread safety issue for these.

The remedy is to only register comparators explicitly, i.e. not the "generic" ones, since they may not be thread-safe. This is actually the behaviour that was in place before HADOOP-6881.

I've also updated the javadoc for WritableComparator.define to clarify that it should only be called for thread-safe classes.


> WritableComparator.get should not cache comparator objects
> ----------------------------------------------------------
>
>                 Key: HADOOP-7183
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7183
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.20.3, 0.21.1, 0.22.0
>
>         Attachments: HADOOP-7183.patch
>
>
> HADOOP-6881 modified WritableComparator.get such that the constructed WritableComparator gets saved back into the static map. This is fine for stateless comparators, but some comparators have per-instance state, and thus this becomes thread-unsafe and causes errors in the shuffle where multiple threads are doing comparisons. An example of a Comparator with per-instance state is WritableComparator itself.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13531-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 19:28:23 2011
Return-Path: <common-issues-return-13531-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21426 invoked from network); 11 Mar 2011 19:28:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 19:28:23 -0000
Received: (qmail 79466 invoked by uid 500); 11 Mar 2011 19:28:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79401 invoked by uid 500); 11 Mar 2011 19:28:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79393 invoked by uid 99); 11 Mar 2011 19:28:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 19:28:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 19:28:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7065F3A4443
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 19:27:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 19:27:59 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1750459070.14169.1299871679457.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1594670430.14027.1299867839776.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Assigned: (HADOOP-7183) WritableComparator.get should not
 cache comparator objects
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White reassigned HADOOP-7183:
---------------------------------

    Assignee: Tom White

> WritableComparator.get should not cache comparator objects
> ----------------------------------------------------------
>
>                 Key: HADOOP-7183
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7183
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.20.3, 0.21.1, 0.22.0
>
>         Attachments: HADOOP-7183.patch
>
>
> HADOOP-6881 modified WritableComparator.get such that the constructed WritableComparator gets saved back into the static map. This is fine for stateless comparators, but some comparators have per-instance state, and thus this becomes thread-unsafe and causes errors in the shuffle where multiple threads are doing comparisons. An example of a Comparator with per-instance state is WritableComparator itself.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13532-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 19:30:21 2011
Return-Path: <common-issues-return-13532-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24875 invoked from network); 11 Mar 2011 19:30:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 19:30:21 -0000
Received: (qmail 81522 invoked by uid 500); 11 Mar 2011 19:30:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81487 invoked by uid 500); 11 Mar 2011 19:30:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81478 invoked by uid 99); 11 Mar 2011 19:30:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 19:30:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 19:30:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 66BA93A44E4
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 19:29:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 19:29:59 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1777394536.14174.1299871799417.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <645678321.2624.1299535019917.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7166) DaemonFactory should be moved from
 HDFS to common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7166:
-----------------------------------------

    Attachment: HADOOP-7166.2.patch

Updated patch marking DaemonFactory as LimitedPrivate.

> DaemonFactory should be moved from HDFS to common
> -------------------------------------------------
>
>                 Key: HADOOP-7166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7166
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-7166.1.patch, HADOOP-7166.2.patch
>
>
> DaemonFactory class is defined in hdfs util. common would be a better place for this class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13533-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 19:32:24 2011
Return-Path: <common-issues-return-13533-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44064 invoked from network); 11 Mar 2011 19:32:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 19:32:23 -0000
Received: (qmail 82045 invoked by uid 500); 11 Mar 2011 19:32:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82010 invoked by uid 500); 11 Mar 2011 19:32:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81998 invoked by uid 99); 11 Mar 2011 19:32:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 19:32:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 19:32:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7BA873A45A7
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 19:31:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 19:31:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <477122280.14179.1299871919503.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1594670430.14027.1299867839776.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7183) WritableComparator.get should not
 cache comparator objects
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005797#comment-13005797 ] 

Todd Lipcon commented on HADOOP-7183:
-------------------------------------

+1 pending test results

> WritableComparator.get should not cache comparator objects
> ----------------------------------------------------------
>
>                 Key: HADOOP-7183
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7183
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.20.3, 0.21.1, 0.22.0
>
>         Attachments: HADOOP-7183.patch
>
>
> HADOOP-6881 modified WritableComparator.get such that the constructed WritableComparator gets saved back into the static map. This is fine for stateless comparators, but some comparators have per-instance state, and thus this becomes thread-unsafe and causes errors in the shuffle where multiple threads are doing comparisons. An example of a Comparator with per-instance state is WritableComparator itself.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13534-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 19:34:22 2011
Return-Path: <common-issues-return-13534-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63378 invoked from network); 11 Mar 2011 19:34:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 19:34:21 -0000
Received: (qmail 85398 invoked by uid 500); 11 Mar 2011 19:34:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85369 invoked by uid 500); 11 Mar 2011 19:34:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85361 invoked by uid 99); 11 Mar 2011 19:34:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 19:34:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 19:34:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0D3D83A466D
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 19:34:00 +0000 (UTC)
Date: Fri, 11 Mar 2011 19:34:00 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <155269634.14190.1299872040050.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1594670430.14027.1299867839776.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7183) WritableComparator.get should not
 cache comparator objects
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-7183:
------------------------------

    Status: Patch Available  (was: Open)

> WritableComparator.get should not cache comparator objects
> ----------------------------------------------------------
>
>                 Key: HADOOP-7183
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7183
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.20.3, 0.21.1, 0.22.0
>
>         Attachments: HADOOP-7183.patch
>
>
> HADOOP-6881 modified WritableComparator.get such that the constructed WritableComparator gets saved back into the static map. This is fine for stateless comparators, but some comparators have per-instance state, and thus this becomes thread-unsafe and causes errors in the shuffle where multiple threads are doing comparisons. An example of a Comparator with per-instance state is WritableComparator itself.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13535-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 19:38:24 2011
Return-Path: <common-issues-return-13535-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66053 invoked from network); 11 Mar 2011 19:38:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 19:38:23 -0000
Received: (qmail 91422 invoked by uid 500); 11 Mar 2011 19:38:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91399 invoked by uid 500); 11 Mar 2011 19:38:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91391 invoked by uid 99); 11 Mar 2011 19:38:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 19:38:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 19:38:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 680BD3A479F
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 19:37:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 19:37:59 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1891070034.14196.1299872279422.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <645678321.2624.1299535019917.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7166) DaemonFactory should be moved from
 HDFS to common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005800#comment-13005800 ] 

Jitendra Nath Pandey commented on HADOOP-7166:
----------------------------------------------

ant test was run manually. 

test patch results

     [exec] -1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
     [exec]                         Please justify why no new tests are needed for this patch.
     [exec]                         Also please list what manual steps were performed to verify this patch.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec] 
     [exec]     +1 system test framework.  The patch passed system test framework compile.

No new tests added, because DaemonFactory is a new class with a single method to construct a new Daemon, and it is not used in common. It is used in HDFS and it should be sufficient to test it there.

> DaemonFactory should be moved from HDFS to common
> -------------------------------------------------
>
>                 Key: HADOOP-7166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7166
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-7166.1.patch, HADOOP-7166.2.patch
>
>
> DaemonFactory class is defined in hdfs util. common would be a better place for this class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13536-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 19:40:25 2011
Return-Path: <common-issues-return-13536-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66432 invoked from network); 11 Mar 2011 19:40:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 19:40:25 -0000
Received: (qmail 96369 invoked by uid 500); 11 Mar 2011 19:40:25 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96330 invoked by uid 500); 11 Mar 2011 19:40:25 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96321 invoked by uid 99); 11 Mar 2011 19:40:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 19:40:25 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 19:40:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EEDA03A4857
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 19:39:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 19:39:59 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2124952660.14198.1299872399975.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <645678321.2624.1299535019917.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7166) DaemonFactory should be moved from
 HDFS to common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7166:
-----------------------------------------

    Status: Open  (was: Patch Available)

> DaemonFactory should be moved from HDFS to common
> -------------------------------------------------
>
>                 Key: HADOOP-7166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7166
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-7166.1.patch, HADOOP-7166.2.patch
>
>
> DaemonFactory class is defined in hdfs util. common would be a better place for this class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13537-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 19:40:26 2011
Return-Path: <common-issues-return-13537-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66452 invoked from network); 11 Mar 2011 19:40:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 19:40:25 -0000
Received: (qmail 96607 invoked by uid 500); 11 Mar 2011 19:40:25 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96539 invoked by uid 500); 11 Mar 2011 19:40:25 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96403 invoked by uid 99); 11 Mar 2011 19:40:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 19:40:25 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 19:40:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 080C53A485A
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 19:40:00 +0000 (UTC)
Date: Fri, 11 Mar 2011 19:40:00 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1729449256.14200.1299872400029.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <645678321.2624.1299535019917.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7166) DaemonFactory should be moved from
 HDFS to common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7166:
-----------------------------------------

    Status: Patch Available  (was: Open)

> DaemonFactory should be moved from HDFS to common
> -------------------------------------------------
>
>                 Key: HADOOP-7166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7166
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-7166.1.patch, HADOOP-7166.2.patch
>
>
> DaemonFactory class is defined in hdfs util. common would be a better place for this class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13538-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 19:48:21 2011
Return-Path: <common-issues-return-13538-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97814 invoked from network); 11 Mar 2011 19:48:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 19:48:21 -0000
Received: (qmail 4599 invoked by uid 500); 11 Mar 2011 19:48:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4561 invoked by uid 500); 11 Mar 2011 19:48:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4553 invoked by uid 99); 11 Mar 2011 19:48:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 19:48:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 19:48:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6D7343A4A3C
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 19:47:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 19:47:59 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <986111545.14204.1299872879445.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1409102499.8762.1299697139647.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7175) Add isEnabled() to Trash
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005801#comment-13005801 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7175:
------------------------------------------------

Hi Daryn, there is a tab in the new isEnabled() method.  We don't use tabs in Hadoop.  Could you remove it?  Other then that the patch looks good.

> Add isEnabled() to Trash
> ------------------------
>
>                 Key: HADOOP-7175
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7175
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7175.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The moveToTrash method returns false in a number of cases.  It's not possible to discern if false means an error occurred. In particular, it's not possible to know if the trash is disabled vs. an error occurred.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13539-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 19:50:21 2011
Return-Path: <common-issues-return-13539-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10580 invoked from network); 11 Mar 2011 19:50:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 19:50:21 -0000
Received: (qmail 5607 invoked by uid 500); 11 Mar 2011 19:50:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5581 invoked by uid 500); 11 Mar 2011 19:50:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5573 invoked by uid 99); 11 Mar 2011 19:50:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 19:50:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 19:50:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 280F33A4AEA
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 19:50:00 +0000 (UTC)
Date: Fri, 11 Mar 2011 19:50:00 +0000 (UTC)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1863824612.14207.1299873000160.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1594670430.14027.1299867839776.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7183) WritableComparator.get should not
 cache comparator objects
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005802#comment-13005802 ] 

Doug Cutting commented on HADOOP-7183:
--------------------------------------

+1 patch looks good to me too.

> WritableComparator.get should not cache comparator objects
> ----------------------------------------------------------
>
>                 Key: HADOOP-7183
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7183
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.20.3, 0.21.1, 0.22.0
>
>         Attachments: HADOOP-7183.patch
>
>
> HADOOP-6881 modified WritableComparator.get such that the constructed WritableComparator gets saved back into the static map. This is fine for stateless comparators, but some comparators have per-instance state, and thus this becomes thread-unsafe and causes errors in the shuffle where multiple threads are doing comparisons. An example of a Comparator with per-instance state is WritableComparator itself.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13540-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 19:52:24 2011
Return-Path: <common-issues-return-13540-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11147 invoked from network); 11 Mar 2011 19:52:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 19:52:23 -0000
Received: (qmail 10702 invoked by uid 500); 11 Mar 2011 19:52:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10676 invoked by uid 500); 11 Mar 2011 19:52:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10668 invoked by uid 99); 11 Mar 2011 19:52:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 19:52:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 19:52:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 73D6C3A4BCE
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 19:51:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 19:51:59 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1117020772.14216.1299873119471.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1594670430.14027.1299867839776.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7183) WritableComparator.get should not
 cache comparator objects
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005803#comment-13005803 ] 

Hadoop QA commented on HADOOP-7183:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12473422/HADOOP-7183.patch
  against trunk revision 1080723.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/308//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/308//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/308//console

This message is automatically generated.

> WritableComparator.get should not cache comparator objects
> ----------------------------------------------------------
>
>                 Key: HADOOP-7183
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7183
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.20.3, 0.21.1, 0.22.0
>
>         Attachments: HADOOP-7183.patch
>
>
> HADOOP-6881 modified WritableComparator.get such that the constructed WritableComparator gets saved back into the static map. This is fine for stateless comparators, but some comparators have per-instance state, and thus this becomes thread-unsafe and causes errors in the shuffle where multiple threads are doing comparisons. An example of a Comparator with per-instance state is WritableComparator itself.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13541-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 20:02:21 2011
Return-Path: <common-issues-return-13541-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30864 invoked from network); 11 Mar 2011 20:02:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 20:02:21 -0000
Received: (qmail 14963 invoked by uid 500); 11 Mar 2011 20:02:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14886 invoked by uid 500); 11 Mar 2011 20:02:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14878 invoked by uid 99); 11 Mar 2011 20:02:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 20:02:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 20:02:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 853E83A4FB3
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 20:01:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 20:01:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1155611520.14245.1299873719542.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <599378200.2869.1299537719453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7167) Allow using a file to exclude
 certain tests from build
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005810#comment-13005810 ] 

Todd Lipcon commented on HADOOP-7167:
-------------------------------------

Yes, sorry - I am planning to add an empty file in "src/test/empty-file" and use that as the default instead of /dev/null. Sound good?

> Allow using a file to exclude certain tests from build
> ------------------------------------------------------
>
>                 Key: HADOOP-7167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7167
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7167-cygwin.txt, hadoop-7167.txt
>
>
> It would be nice to be able to exclude certain tests when running builds. For example, when a test is "known flaky", you may want to exclude it from the main Hudson job, but not actually disable it in the codebase (so that it still runs as part of another Hudson job, for example).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13543-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 20:02:22 2011
Return-Path: <common-issues-return-13543-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30989 invoked from network); 11 Mar 2011 20:02:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 20:02:21 -0000
Received: (qmail 15373 invoked by uid 500); 11 Mar 2011 20:02:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15105 invoked by uid 500); 11 Mar 2011 20:02:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14906 invoked by uid 99); 11 Mar 2011 20:02:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 20:02:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 20:02:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 99B6F3A4FB6
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 20:01:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 20:01:59 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1936798502.14248.1299873719626.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1409102499.8762.1299697139647.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7175) Add isEnabled() to Trash
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7175:
--------------------------------

    Attachment: HADOOP-7175-2.patch

Remove tab.

> Add isEnabled() to Trash
> ------------------------
>
>                 Key: HADOOP-7175
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7175
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7175-2.patch, HADOOP-7175.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The moveToTrash method returns false in a number of cases.  It's not possible to discern if false means an error occurred. In particular, it's not possible to know if the trash is disabled vs. an error occurred.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13542-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 20:02:22 2011
Return-Path: <common-issues-return-13542-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30979 invoked from network); 11 Mar 2011 20:02:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 20:02:21 -0000
Received: (qmail 15171 invoked by uid 500); 11 Mar 2011 20:02:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15103 invoked by uid 500); 11 Mar 2011 20:02:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14916 invoked by uid 99); 11 Mar 2011 20:02:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 20:02:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 20:02:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A7CD53A4FB9
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 20:01:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 20:01:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <68886421.14250.1299873719683.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <599378200.2869.1299537719453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7167) Allow using a file to exclude certain
 tests from build
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7167:
--------------------------------

    Attachment: hadoop-7167-cygwin.txt

How does this patch look?

> Allow using a file to exclude certain tests from build
> ------------------------------------------------------
>
>                 Key: HADOOP-7167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7167
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7167-cygwin.txt, hadoop-7167.txt
>
>
> It would be nice to be able to exclude certain tests when running builds. For example, when a test is "known flaky", you may want to exclude it from the main Hudson job, but not actually disable it in the codebase (so that it still runs as part of another Hudson job, for example).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13544-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 21:52:21 2011
Return-Path: <common-issues-return-13544-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98991 invoked from network); 11 Mar 2011 21:52:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 21:52:21 -0000
Received: (qmail 61865 invoked by uid 500); 11 Mar 2011 21:52:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61839 invoked by uid 500); 11 Mar 2011 21:52:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61831 invoked by uid 99); 11 Mar 2011 21:52:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 21:52:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 21:52:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BB2E83A478A
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 21:51:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 21:51:59 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <604024911.14527.1299880319763.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005861#comment-13005861 ] 

Hudson commented on HADOOP-7156:
--------------------------------

Integrated in Hadoop-Common-22-branch #32 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-22-branch/32/])
    HADOOP-7156. Workaround for unsafe implementations of getpwuid_r. Contributed by Todd Lipcon.


> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13545-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 21:52:22 2011
Return-Path: <common-issues-return-13545-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99007 invoked from network); 11 Mar 2011 21:52:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 21:52:21 -0000
Received: (qmail 62005 invoked by uid 500); 11 Mar 2011 21:52:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61930 invoked by uid 500); 11 Mar 2011 21:52:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61840 invoked by uid 99); 11 Mar 2011 21:52:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 21:52:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 21:52:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C86F43A478C
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 21:51:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 21:51:59 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1667119139.14529.1299880319818.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <130797695.15062.1298594983152.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7154) Should set MALLOC_ARENA_MAX in
 hadoop-env.sh
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005862#comment-13005862 ] 

Hudson commented on HADOOP-7154:
--------------------------------

Integrated in Hadoop-Common-22-branch #32 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-22-branch/32/])
    

> Should set MALLOC_ARENA_MAX in hadoop-env.sh
> --------------------------------------------
>
>                 Key: HADOOP-7154
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7154
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7154.txt
>
>
> New versions of glibc present in RHEL6 include a new arena allocator design. In several clusters we've seen this new allocator cause huge amounts of virtual memory to be used, since when multiple threads perform allocations, they each get their own memory arena. On a 64-bit system, these arenas are 64M mappings, and the maximum number of arenas is 8 times the number of cores. We've observed a DN process using 14GB of vmem for only 300M of resident set. This causes all kinds of nasty issues for obvious reasons.
> Setting MALLOC_ARENA_MAX to a low number will restrict the number of memory arenas and bound the virtual memory, with no noticeable downside in performance - we've been recommending MALLOC_ARENA_MAX=4. We should set this in hadoop-env.sh to avoid this issue as RHEL6 becomes more and more common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13546-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 21:52:24 2011
Return-Path: <common-issues-return-13546-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99106 invoked from network); 11 Mar 2011 21:52:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 21:52:24 -0000
Received: (qmail 62675 invoked by uid 500); 11 Mar 2011 21:52:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62616 invoked by uid 500); 11 Mar 2011 21:52:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62608 invoked by uid 99); 11 Mar 2011 21:52:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 21:52:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 21:52:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 26A913A4798
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 21:52:00 +0000 (UTC)
Date: Fri, 11 Mar 2011 21:52:00 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2114051757.14541.1299880320155.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <978952244.13379.1299230917384.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7162) FsShell: call srcFs.listStatus(src)
 twice
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005863#comment-13005863 ] 

Hudson commented on HADOOP-7162:
--------------------------------

Integrated in Hadoop-Common-22-branch #32 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-22-branch/32/])
    

> FsShell: call srcFs.listStatus(src) twice
> -----------------------------------------
>
>                 Key: HADOOP-7162
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7162
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Alexey Diomin
>            Assignee: Alexey Diomin
>            Priority: Minor
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HDFS-7162.path
>
>
> in file ./src/java/org/apache/hadoop/fs/FsShell.java line 555
> call method twice:
> 1. for init variable
> 2. for getting data

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13547-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 21:54:24 2011
Return-Path: <common-issues-return-13547-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99515 invoked from network); 11 Mar 2011 21:54:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 21:54:24 -0000
Received: (qmail 63460 invoked by uid 500); 11 Mar 2011 21:54:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63307 invoked by uid 500); 11 Mar 2011 21:54:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63299 invoked by uid 99); 11 Mar 2011 21:54:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 21:54:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 21:54:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A67E33A488B
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 21:53:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 21:53:59 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <940574892.14547.1299880439678.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <645678321.2624.1299535019917.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7166) DaemonFactory should be moved from
 HDFS to common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005864#comment-13005864 ] 

Suresh Srinivas commented on HADOOP-7166:
-----------------------------------------

Thanks Tom for suggesting the addition of @LimitedPrivate. Patch looks good to me. +1.

> DaemonFactory should be moved from HDFS to common
> -------------------------------------------------
>
>                 Key: HADOOP-7166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7166
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-7166.1.patch, HADOOP-7166.2.patch
>
>
> DaemonFactory class is defined in hdfs util. common would be a better place for this class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13548-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 21:56:21 2011
Return-Path: <common-issues-return-13548-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 371 invoked from network); 11 Mar 2011 21:56:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 21:56:21 -0000
Received: (qmail 64954 invoked by uid 500); 11 Mar 2011 21:56:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64917 invoked by uid 500); 11 Mar 2011 21:56:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64908 invoked by uid 99); 11 Mar 2011 21:56:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 21:56:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 21:56:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7DD9E3A4967
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 21:55:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 21:55:59 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1637434317.14558.1299880559512.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <599378200.2869.1299537719453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7167) Allow using a file to exclude
 certain tests from build
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005865#comment-13005865 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7167:
------------------------------------------------

The patch works if I create {{src/test/empty-file}} manually.
{noformat}
BUILD FAILED
z:\_tsz\hadoop\common\c2\build.xml:827: The following error occurred while executing this line:
z:\_tsz\hadoop\common\c2\build.xml:807: The following error occurred while executing this line:
z:\_tsz\hadoop\common\c2\build.xml:736: Excludesfile z:\_tsz\hadoop\common\c2\src\test\empty-file not found.

Total time: 22 seconds
{noformat}

{noformat}
$ touch src/test/empty-file

...
BUILD SUCCESSFUL
Total time: 1 minute 4 seconds
{noformat}



> Allow using a file to exclude certain tests from build
> ------------------------------------------------------
>
>                 Key: HADOOP-7167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7167
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7167-cygwin.txt, hadoop-7167.txt
>
>
> It would be nice to be able to exclude certain tests when running builds. For example, when a test is "known flaky", you may want to exclude it from the main Hudson job, but not actually disable it in the codebase (so that it still runs as part of another Hudson job, for example).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13549-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 22:14:21 2011
Return-Path: <common-issues-return-13549-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55142 invoked from network); 11 Mar 2011 22:14:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 22:14:21 -0000
Received: (qmail 96526 invoked by uid 500); 11 Mar 2011 22:14:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96489 invoked by uid 500); 11 Mar 2011 22:14:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96478 invoked by uid 99); 11 Mar 2011 22:14:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 22:14:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 22:14:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7AF383A4FF6
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 22:13:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 22:13:59 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <468801590.14616.1299881639500.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <645678321.2624.1299535019917.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7166) DaemonFactory should be moved from
 HDFS to common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005873#comment-13005873 ] 

Hudson commented on HADOOP-7166:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #525 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/525/])
    HADOOP-7166. Add DaemonFactory to common. Contributed by Erik Steffl and jitendra.


> DaemonFactory should be moved from HDFS to common
> -------------------------------------------------
>
>                 Key: HADOOP-7166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7166
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-7166.1.patch, HADOOP-7166.2.patch
>
>
> DaemonFactory class is defined in hdfs util. common would be a better place for this class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13550-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 22:55:24 2011
Return-Path: <common-issues-return-13550-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92038 invoked from network); 11 Mar 2011 22:55:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 22:55:24 -0000
Received: (qmail 51400 invoked by uid 500); 11 Mar 2011 22:55:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51374 invoked by uid 500); 11 Mar 2011 22:55:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51365 invoked by uid 99); 11 Mar 2011 22:55:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 22:55:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 22:55:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 75B303A4AE8
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 22:54:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 22:54:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2024509504.14644.1299884099478.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <599378200.2869.1299537719453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7167) Allow using a file to exclude
 certain tests from build
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005881#comment-13005881 ] 

Todd Lipcon commented on HADOOP-7167:
-------------------------------------

cool, +1 for commit then? (including the new empty-file)

> Allow using a file to exclude certain tests from build
> ------------------------------------------------------
>
>                 Key: HADOOP-7167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7167
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7167-cygwin.txt, hadoop-7167.txt
>
>
> It would be nice to be able to exclude certain tests when running builds. For example, when a test is "known flaky", you may want to exclude it from the main Hudson job, but not actually disable it in the codebase (so that it still runs as part of another Hudson job, for example).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13551-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 23:40:24 2011
Return-Path: <common-issues-return-13551-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41314 invoked from network); 11 Mar 2011 23:40:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 23:40:23 -0000
Received: (qmail 1059 invoked by uid 500); 11 Mar 2011 23:40:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 900 invoked by uid 500); 11 Mar 2011 23:40:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 892 invoked by uid 99); 11 Mar 2011 23:40:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 23:40:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 23:40:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 973003A4685
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 23:39:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 23:39:59 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1548724676.14724.1299886799616.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1359805310.9666.1299717240403.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7177) CodecPool should report which
 compressor it is using
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005902#comment-13005902 ] 

Eli Collins commented on HADOOP-7177:
-------------------------------------

+1

Confirmed TestCodec still passes, given this just changes a log I don't think we need additional tests.

{noformat}
     [exec] 
     [exec] -1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
     [exec]                         Please justify why no new tests are needed for this patch.
     [exec]                         Also please list what manual steps were performed to verify this patch.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec] 
     [exec]     +1 system test framework.  The patch passed system test framework compile.
     [exec] 
{noformat}

> CodecPool should report which compressor it is using
> ----------------------------------------------------
>
>                 Key: HADOOP-7177
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7177
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: native
>    Affects Versions: 0.20.2, 0.21.0, 0.23.0
>            Reporter: Allen Wittenauer
>            Assignee: Allen Wittenauer
>            Priority: Trivial
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: hadoop-7177.txt
>
>
> Certain native compression libraries are overly verbose causing confusion while reading the task logs.  Let's actually say which compressor we got when we report it in the task logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13552-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 23:46:24 2011
Return-Path: <common-issues-return-13552-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55403 invoked from network); 11 Mar 2011 23:46:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 23:46:24 -0000
Received: (qmail 8546 invoked by uid 500); 11 Mar 2011 23:46:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8520 invoked by uid 500); 11 Mar 2011 23:46:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8504 invoked by uid 99); 11 Mar 2011 23:46:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 23:46:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 23:46:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 861D83A47EF
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 23:45:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 23:45:59 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <342830211.14730.1299887159546.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1359805310.9666.1299717240403.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Closed: (HADOOP-7177) CodecPool should report which
 compressor it is using
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins closed HADOOP-7177.
-------------------------------


> CodecPool should report which compressor it is using
> ----------------------------------------------------
>
>                 Key: HADOOP-7177
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7177
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: native
>    Affects Versions: 0.20.2, 0.21.0, 0.23.0
>            Reporter: Allen Wittenauer
>            Assignee: Allen Wittenauer
>            Priority: Trivial
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: hadoop-7177.txt
>
>
> Certain native compression libraries are overly verbose causing confusion while reading the task logs.  Let's actually say which compressor we got when we report it in the task logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13553-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 23:46:24 2011
Return-Path: <common-issues-return-13553-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55424 invoked from network); 11 Mar 2011 23:46:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 23:46:24 -0000
Received: (qmail 8586 invoked by uid 500); 11 Mar 2011 23:46:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8522 invoked by uid 500); 11 Mar 2011 23:46:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8505 invoked by uid 99); 11 Mar 2011 23:46:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 23:46:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 23:46:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 778603A47EC
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 23:45:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 23:45:59 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <461630175.14728.1299887159485.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1359805310.9666.1299717240403.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7177) CodecPool should report which
 compressor it is using
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7177:
--------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I've committed this to trunk and merged to 22 and 21. Thanks Allen!

> CodecPool should report which compressor it is using
> ----------------------------------------------------
>
>                 Key: HADOOP-7177
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7177
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: native
>    Affects Versions: 0.20.2, 0.21.0, 0.23.0
>            Reporter: Allen Wittenauer
>            Assignee: Allen Wittenauer
>            Priority: Trivial
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: hadoop-7177.txt
>
>
> Certain native compression libraries are overly verbose causing confusion while reading the task logs.  Let's actually say which compressor we got when we report it in the task logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13555-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 23:50:21 2011
Return-Path: <common-issues-return-13555-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87105 invoked from network); 11 Mar 2011 23:50:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 23:50:21 -0000
Received: (qmail 14118 invoked by uid 500); 11 Mar 2011 23:50:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13995 invoked by uid 500); 11 Mar 2011 23:50:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13877 invoked by uid 99); 11 Mar 2011 23:50:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 23:50:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 23:50:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A05803A49A5
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 23:49:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 23:49:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1795118329.14744.1299887399653.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <915517661.14739.1299887399476.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7184) Remove deprecated local.cache.size
 from core-default.xml
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7184:
--------------------------------

    Status: Patch Available  (was: Open)

> Remove deprecated local.cache.size from core-default.xml
> --------------------------------------------------------
>
>                 Key: HADOOP-7184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7184
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation, filecache
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7184.txt
>
>
> MAPREDUCE-2379 documents the new name of this parameter (mapreduce.tasktracker.cache.local.size) in mapred-default.xml where it belongs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13554-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 23:50:21 2011
Return-Path: <common-issues-return-13554-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87109 invoked from network); 11 Mar 2011 23:50:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 23:50:21 -0000
Received: (qmail 13813 invoked by uid 500); 11 Mar 2011 23:50:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13783 invoked by uid 500); 11 Mar 2011 23:50:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13774 invoked by uid 99); 11 Mar 2011 23:50:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 23:50:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 23:50:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8C3813A49A3
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 23:49:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 23:49:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <462027254.14742.1299887399570.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <915517661.14739.1299887399476.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7184) Remove deprecated local.cache.size
 from core-default.xml
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7184:
--------------------------------

    Attachment: hadoop-7184.txt

> Remove deprecated local.cache.size from core-default.xml
> --------------------------------------------------------
>
>                 Key: HADOOP-7184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7184
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation, filecache
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7184.txt
>
>
> MAPREDUCE-2379 documents the new name of this parameter (mapreduce.tasktracker.cache.local.size) in mapred-default.xml where it belongs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13556-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 23:50:23 2011
Return-Path: <common-issues-return-13556-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87196 invoked from network); 11 Mar 2011 23:50:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 23:50:23 -0000
Received: (qmail 14536 invoked by uid 500); 11 Mar 2011 23:50:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14485 invoked by uid 500); 11 Mar 2011 23:50:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14477 invoked by uid 99); 11 Mar 2011 23:50:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 23:50:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 23:50:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 750EE3A49A0
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 23:49:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 23:49:59 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <915517661.14739.1299887399476.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7184) Remove deprecated local.cache.size
 from core-default.xml
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Remove deprecated local.cache.size from core-default.xml
--------------------------------------------------------

                 Key: HADOOP-7184
                 URL: https://issues.apache.org/jira/browse/HADOOP-7184
             Project: Hadoop Common
          Issue Type: Bug
          Components: documentation, filecache
    Affects Versions: 0.22.0
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
             Fix For: 0.22.0
         Attachments: hadoop-7184.txt

MAPREDUCE-2379 documents the new name of this parameter (mapreduce.tasktracker.cache.local.size) in mapred-default.xml where it belongs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13557-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 23:54:28 2011
Return-Path: <common-issues-return-13557-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88463 invoked from network); 11 Mar 2011 23:54:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 23:54:27 -0000
Received: (qmail 18627 invoked by uid 500); 11 Mar 2011 23:54:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18600 invoked by uid 500); 11 Mar 2011 23:54:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18592 invoked by uid 99); 11 Mar 2011 23:54:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 23:54:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 23:54:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 697443A4A98
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 23:53:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 23:53:59 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1057921594.14754.1299887639428.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1359805310.9666.1299717240403.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7177) CodecPool should report which
 compressor it is using
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005906#comment-13005906 ] 

Hudson commented on HADOOP-7177:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #526 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/526/])
    HADOOP-7177. CodecPool should report which compressor it is using. Contributed by Allen Wittenauer


> CodecPool should report which compressor it is using
> ----------------------------------------------------
>
>                 Key: HADOOP-7177
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7177
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: native
>    Affects Versions: 0.20.2, 0.21.0, 0.23.0
>            Reporter: Allen Wittenauer
>            Assignee: Allen Wittenauer
>            Priority: Trivial
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: hadoop-7177.txt
>
>
> Certain native compression libraries are overly verbose causing confusion while reading the task logs.  Let's actually say which compressor we got when we report it in the task logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13558-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 11 23:59:23 2011
Return-Path: <common-issues-return-13558-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89798 invoked from network); 11 Mar 2011 23:59:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Mar 2011 23:59:23 -0000
Received: (qmail 22743 invoked by uid 500); 11 Mar 2011 23:59:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22713 invoked by uid 500); 11 Mar 2011 23:59:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22705 invoked by uid 99); 11 Mar 2011 23:59:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 23:59:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Mar 2011 23:59:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 691393A4BAC
	for <common-issues@hadoop.apache.org>; Fri, 11 Mar 2011 23:58:59 +0000 (UTC)
Date: Fri, 11 Mar 2011 23:58:59 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <557025910.14757.1299887939427.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <599378200.2869.1299537719453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7167) Allow using a file to exclude
 certain tests from build
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005907#comment-13005907 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7167:
------------------------------------------------

+1 patch looks good, thanks!

> Allow using a file to exclude certain tests from build
> ------------------------------------------------------
>
>                 Key: HADOOP-7167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7167
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7167-cygwin.txt, hadoop-7167.txt
>
>
> It would be nice to be able to exclude certain tests when running builds. For example, when a test is "known flaky", you may want to exclude it from the main Hudson job, but not actually disable it in the codebase (so that it still runs as part of another Hudson job, for example).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13559-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 12 00:01:21 2011
Return-Path: <common-issues-return-13559-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97838 invoked from network); 12 Mar 2011 00:01:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Mar 2011 00:01:21 -0000
Received: (qmail 23420 invoked by uid 500); 12 Mar 2011 00:01:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23397 invoked by uid 500); 12 Mar 2011 00:01:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23388 invoked by uid 99); 12 Mar 2011 00:01:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 00:01:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 00:01:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B55033A4C7B
	for <common-issues@hadoop.apache.org>; Sat, 12 Mar 2011 00:00:59 +0000 (UTC)
Date: Sat, 12 Mar 2011 00:00:59 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1009938311.14761.1299888059739.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <645678321.2624.1299535019917.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7166) DaemonFactory should be moved from
 HDFS to common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7166:
-----------------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

> DaemonFactory should be moved from HDFS to common
> -------------------------------------------------
>
>                 Key: HADOOP-7166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7166
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-7166.1.patch, HADOOP-7166.2.patch
>
>
> DaemonFactory class is defined in hdfs util. common would be a better place for this class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13560-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 12 00:17:21 2011
Return-Path: <common-issues-return-13560-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50448 invoked from network); 12 Mar 2011 00:17:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Mar 2011 00:17:21 -0000
Received: (qmail 39827 invoked by uid 500); 12 Mar 2011 00:17:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39797 invoked by uid 500); 12 Mar 2011 00:17:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39789 invoked by uid 99); 12 Mar 2011 00:17:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 00:17:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 00:17:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 87BA03A41BC
	for <common-issues@hadoop.apache.org>; Sat, 12 Mar 2011 00:16:59 +0000 (UTC)
Date: Sat, 12 Mar 2011 00:16:59 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <346575510.14814.1299889019552.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Resolved: (HADOOP-6361) Specific Exceptions thrown by
 FileContext and AbstractFileSystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey resolved HADOOP-6361.
------------------------------------------

    Resolution: Duplicate

This has been addressed in HADOOP-6537

> Specific Exceptions thrown by FileContext and AbstractFileSystem
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6361
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6361
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6361.1.patch
>
>
> This jira tracks implementation of the proposal in HDFS-717.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13561-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 12 00:25:23 2011
Return-Path: <common-issues-return-13561-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87947 invoked from network); 12 Mar 2011 00:25:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Mar 2011 00:25:22 -0000
Received: (qmail 46059 invoked by uid 500); 12 Mar 2011 00:25:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46030 invoked by uid 500); 12 Mar 2011 00:25:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46022 invoked by uid 99); 12 Mar 2011 00:25:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 00:25:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 00:25:22 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 268D13A44DC
	for <common-issues@hadoop.apache.org>; Sat, 12 Mar 2011 00:25:01 +0000 (UTC)
Date: Sat, 12 Mar 2011 00:25:01 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2109704952.14835.1299889501154.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <599378200.2869.1299537719453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Resolved: (HADOOP-7167) Allow using a file to exclude
 certain tests from build
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon resolved HADOOP-7167.
---------------------------------

    Resolution: Fixed

Commited fix to trunk. Will go and add patches to the hdfs and mr JIRAs in just a minute.

> Allow using a file to exclude certain tests from build
> ------------------------------------------------------
>
>                 Key: HADOOP-7167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7167
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7167-cygwin.txt, hadoop-7167.txt
>
>
> It would be nice to be able to exclude certain tests when running builds. For example, when a test is "known flaky", you may want to exclude it from the main Hudson job, but not actually disable it in the codebase (so that it still runs as part of another Hudson job, for example).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13562-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 12 00:39:21 2011
Return-Path: <common-issues-return-13562-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42990 invoked from network); 12 Mar 2011 00:39:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Mar 2011 00:39:21 -0000
Received: (qmail 63465 invoked by uid 500); 12 Mar 2011 00:39:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63424 invoked by uid 500); 12 Mar 2011 00:39:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63343 invoked by uid 99); 12 Mar 2011 00:39:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 00:39:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 00:39:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7EACB3A5652
	for <common-issues@hadoop.apache.org>; Sat, 12 Mar 2011 00:38:59 +0000 (UTC)
Date: Sat, 12 Mar 2011 00:38:59 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <869236501.14869.1299890339515.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <995627871.3214.1299545460554.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7171) Support UGI in FileContext API
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005932#comment-13005932 ] 

Jitendra Nath Pandey commented on HADOOP-7171:
----------------------------------------------

Proposed change:
   FileContext object should remember the user (UGI) for which it was created, and any AbstractFileSystem object should be created for that user i.e. within a doAs for that user.



> Support UGI in FileContext API
> ------------------------------
>
>                 Key: HADOOP-7171
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7171
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>
> The FileContext API needs to support UGI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13563-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 12 00:43:21 2011
Return-Path: <common-issues-return-13563-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44612 invoked from network); 12 Mar 2011 00:43:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Mar 2011 00:43:21 -0000
Received: (qmail 67459 invoked by uid 500); 12 Mar 2011 00:43:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67418 invoked by uid 500); 12 Mar 2011 00:43:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67404 invoked by uid 99); 12 Mar 2011 00:43:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 00:43:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 00:43:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 72BA93A573F
	for <common-issues@hadoop.apache.org>; Sat, 12 Mar 2011 00:42:59 +0000 (UTC)
Date: Sat, 12 Mar 2011 00:42:59 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <489378522.14875.1299890579466.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <995627871.3214.1299545460554.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7171) Support UGI in FileContext API
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7171:
-----------------------------------------

    Attachment: HADOOP-7171.2.patch

Patch uploaded.

> Support UGI in FileContext API
> ------------------------------
>
>                 Key: HADOOP-7171
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7171
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7171.2.patch
>
>
> The FileContext API needs to support UGI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13564-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 12 00:43:22 2011
Return-Path: <common-issues-return-13564-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44637 invoked from network); 12 Mar 2011 00:43:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Mar 2011 00:43:21 -0000
Received: (qmail 67737 invoked by uid 500); 12 Mar 2011 00:43:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67636 invoked by uid 500); 12 Mar 2011 00:43:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67600 invoked by uid 99); 12 Mar 2011 00:43:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 00:43:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 00:43:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8EECA3A5743
	for <common-issues@hadoop.apache.org>; Sat, 12 Mar 2011 00:42:59 +0000 (UTC)
Date: Sat, 12 Mar 2011 00:42:59 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1451275095.14879.1299890579582.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <995627871.3214.1299545460554.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7171) Support UGI in FileContext API
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7171:
-----------------------------------------

    Status: Patch Available  (was: Open)

> Support UGI in FileContext API
> ------------------------------
>
>                 Key: HADOOP-7171
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7171
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7171.2.patch
>
>
> The FileContext API needs to support UGI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13565-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 12 00:45:25 2011
Return-Path: <common-issues-return-13565-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48355 invoked from network); 12 Mar 2011 00:45:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Mar 2011 00:45:24 -0000
Received: (qmail 69166 invoked by uid 500); 12 Mar 2011 00:45:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69136 invoked by uid 500); 12 Mar 2011 00:45:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69122 invoked by uid 99); 12 Mar 2011 00:45:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 00:45:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 00:45:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 784CF3A57F2
	for <common-issues@hadoop.apache.org>; Sat, 12 Mar 2011 00:44:59 +0000 (UTC)
Date: Sat, 12 Mar 2011 00:44:59 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <615208354.14886.1299890699489.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <599378200.2869.1299537719453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7167) Allow using a file to exclude
 certain tests from build
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005934#comment-13005934 ] 

Hudson commented on HADOOP-7167:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #527 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/527/])
    HADOOP-7167. Amend previous commit under this JIRA to fix issue on cygwin. Contributed by Todd Lipcon


> Allow using a file to exclude certain tests from build
> ------------------------------------------------------
>
>                 Key: HADOOP-7167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7167
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7167-cygwin.txt, hadoop-7167.txt
>
>
> It would be nice to be able to exclude certain tests when running builds. For example, when a test is "known flaky", you may want to exclude it from the main Hudson job, but not actually disable it in the codebase (so that it still runs as part of another Hudson job, for example).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13566-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 12 09:51:27 2011
Return-Path: <common-issues-return-13566-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99133 invoked from network); 12 Mar 2011 09:51:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Mar 2011 09:51:24 -0000
Received: (qmail 1735 invoked by uid 500); 12 Mar 2011 09:51:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1346 invoked by uid 500); 12 Mar 2011 09:51:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1090 invoked by uid 99); 12 Mar 2011 09:51:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 09:51:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 09:51:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7BC9839D7BA
	for <common-issues@hadoop.apache.org>; Sat, 12 Mar 2011 09:50:59 +0000 (UTC)
Date: Sat, 12 Mar 2011 09:50:59 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1666059415.15299.1299923459503.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1359805310.9666.1299717240403.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7177) CodecPool should report which
 compressor it is using
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006012#comment-13006012 ] 

Hudson commented on HADOOP-7177:
--------------------------------

Integrated in Hadoop-Common-22-branch #33 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-22-branch/33/])
    HADOOP-7177. svn merge -c 1080797 from trunk


> CodecPool should report which compressor it is using
> ----------------------------------------------------
>
>                 Key: HADOOP-7177
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7177
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: native
>    Affects Versions: 0.20.2, 0.21.0, 0.23.0
>            Reporter: Allen Wittenauer
>            Assignee: Allen Wittenauer
>            Priority: Trivial
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: hadoop-7177.txt
>
>
> Certain native compression libraries are overly verbose causing confusion while reading the task logs.  Let's actually say which compressor we got when we report it in the task logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13567-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 12 11:14:21 2011
Return-Path: <common-issues-return-13567-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43002 invoked from network); 12 Mar 2011 11:14:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Mar 2011 11:14:21 -0000
Received: (qmail 41685 invoked by uid 500); 12 Mar 2011 11:14:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41652 invoked by uid 500); 12 Mar 2011 11:14:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41644 invoked by uid 99); 12 Mar 2011 11:14:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 11:14:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 11:14:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9220339ED27
	for <common-issues@hadoop.apache.org>; Sat, 12 Mar 2011 11:13:59 +0000 (UTC)
Date: Sat, 12 Mar 2011 11:13:59 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1104844781.15350.1299928439595.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <645678321.2624.1299535019917.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7166) DaemonFactory should be moved from
 HDFS to common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006026#comment-13006026 ] 

Hudson commented on HADOOP-7166:
--------------------------------

Integrated in Hadoop-Common-trunk #628 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/628/])
    HADOOP-7166. Add DaemonFactory to common. Contributed by Erik Steffl and jitendra.


> DaemonFactory should be moved from HDFS to common
> -------------------------------------------------
>
>                 Key: HADOOP-7166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7166
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-7166.1.patch, HADOOP-7166.2.patch
>
>
> DaemonFactory class is defined in hdfs util. common would be a better place for this class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13568-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 12 11:14:22 2011
Return-Path: <common-issues-return-13568-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43025 invoked from network); 12 Mar 2011 11:14:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Mar 2011 11:14:21 -0000
Received: (qmail 41968 invoked by uid 500); 12 Mar 2011 11:14:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41866 invoked by uid 500); 12 Mar 2011 11:14:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41770 invoked by uid 99); 12 Mar 2011 11:14:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 11:14:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 11:14:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8391939ED24
	for <common-issues@hadoop.apache.org>; Sat, 12 Mar 2011 11:13:59 +0000 (UTC)
Date: Sat, 12 Mar 2011 11:13:59 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <357371727.15348.1299928439535.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1359805310.9666.1299717240403.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7177) CodecPool should report which
 compressor it is using
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006025#comment-13006025 ] 

Hudson commented on HADOOP-7177:
--------------------------------

Integrated in Hadoop-Common-trunk #628 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/628/])
    HADOOP-7177. CodecPool should report which compressor it is using. Contributed by Allen Wittenauer


> CodecPool should report which compressor it is using
> ----------------------------------------------------
>
>                 Key: HADOOP-7177
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7177
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: native
>    Affects Versions: 0.20.2, 0.21.0, 0.23.0
>            Reporter: Allen Wittenauer
>            Assignee: Allen Wittenauer
>            Priority: Trivial
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: hadoop-7177.txt
>
>
> Certain native compression libraries are overly verbose causing confusion while reading the task logs.  Let's actually say which compressor we got when we report it in the task logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13569-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 12 11:14:24 2011
Return-Path: <common-issues-return-13569-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43086 invoked from network); 12 Mar 2011 11:14:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Mar 2011 11:14:24 -0000
Received: (qmail 42123 invoked by uid 500); 12 Mar 2011 11:14:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42079 invoked by uid 500); 12 Mar 2011 11:14:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42071 invoked by uid 99); 12 Mar 2011 11:14:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 11:14:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 11:14:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9860D39ED28
	for <common-issues@hadoop.apache.org>; Sat, 12 Mar 2011 11:13:59 +0000 (UTC)
Date: Sat, 12 Mar 2011 11:13:59 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1982622305.15351.1299928439620.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <599378200.2869.1299537719453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7167) Allow using a file to exclude
 certain tests from build
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006027#comment-13006027 ] 

Hudson commented on HADOOP-7167:
--------------------------------

Integrated in Hadoop-Common-trunk #628 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/628/])
    HADOOP-7167. Amend previous commit under this JIRA to fix issue on cygwin. Contributed by Todd Lipcon


> Allow using a file to exclude certain tests from build
> ------------------------------------------------------
>
>                 Key: HADOOP-7167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7167
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7167-cygwin.txt, hadoop-7167.txt
>
>
> It would be nice to be able to exclude certain tests when running builds. For example, when a test is "known flaky", you may want to exclude it from the main Hudson job, but not actually disable it in the codebase (so that it still runs as part of another Hudson job, for example).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13570-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 12 11:14:25 2011
Return-Path: <common-issues-return-13570-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43131 invoked from network); 12 Mar 2011 11:14:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Mar 2011 11:14:25 -0000
Received: (qmail 42327 invoked by uid 500); 12 Mar 2011 11:14:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42293 invoked by uid 500); 12 Mar 2011 11:14:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42285 invoked by uid 99); 12 Mar 2011 11:14:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 11:14:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 11:14:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B814E39ED2D
	for <common-issues@hadoop.apache.org>; Sat, 12 Mar 2011 11:13:59 +0000 (UTC)
Date: Sat, 12 Mar 2011 11:13:59 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <572513074.15356.1299928439750.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006028#comment-13006028 ] 

Hudson commented on HADOOP-7156:
--------------------------------

Integrated in Hadoop-Common-trunk #628 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/628/])
    HADOOP-7156. Workaround for unsafe implementations of getpwuid_r. Contributed by Todd Lipcon.


> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13571-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 12 11:42:26 2011
Return-Path: <common-issues-return-13571-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34488 invoked from network); 12 Mar 2011 11:42:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Mar 2011 11:42:24 -0000
Received: (qmail 61243 invoked by uid 500); 12 Mar 2011 11:42:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61213 invoked by uid 500); 12 Mar 2011 11:42:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61205 invoked by uid 99); 12 Mar 2011 11:42:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 11:42:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Mar 2011 11:42:22 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8B97C39F715
	for <common-issues@hadoop.apache.org>; Sat, 12 Mar 2011 11:41:59 +0000 (UTC)
Date: Sat, 12 Mar 2011 11:41:59 +0000 (UTC)
From: "Shekhar Gupta (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <171998182.15391.1299930119568.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7185) ERROR streaming.StreamJob: Error
 Launching job : Input path does not exist
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

ERROR streaming.StreamJob: Error Launching job : Input path does not exist
--------------------------------------------------------------------------

                 Key: HADOOP-7185
                 URL: https://issues.apache.org/jira/browse/HADOOP-7185
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.20.2
         Environment: Linux
            Reporter: Shekhar Gupta




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13572-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Mar 13 10:22:22 2011
Return-Path: <common-issues-return-13572-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53551 invoked from network); 13 Mar 2011 10:22:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Mar 2011 10:22:22 -0000
Received: (qmail 40701 invoked by uid 500); 13 Mar 2011 10:22:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40675 invoked by uid 500); 13 Mar 2011 10:22:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40667 invoked by uid 99); 13 Mar 2011 10:22:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Mar 2011 10:22:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Mar 2011 10:22:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 689963A60AD
	for <common-issues@hadoop.apache.org>; Sun, 13 Mar 2011 10:21:59 +0000 (UTC)
Date: Sun, 13 Mar 2011 10:21:59 +0000 (UTC)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2135744443.16040.1300011719424.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <171998182.15391.1299930119568.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7185) ERROR streaming.StreamJob: Error
 Launching job : Input path does not exist
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006166#comment-13006166 ] 

Amareshwari Sriramadasu commented on HADOOP-7185:
-------------------------------------------------

Can you give more details about the problem here? 
>From the description it seems your streaming job failed because of invalid input path, which is a valid failure.

> ERROR streaming.StreamJob: Error Launching job : Input path does not exist
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7185
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7185
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>         Environment: Linux
>            Reporter: Shekhar Gupta
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13573-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Mar 13 18:03:24 2011
Return-Path: <common-issues-return-13573-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12002 invoked from network); 13 Mar 2011 18:03:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Mar 2011 18:03:24 -0000
Received: (qmail 929 invoked by uid 500); 13 Mar 2011 18:03:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 894 invoked by uid 500); 13 Mar 2011 18:03:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 886 invoked by uid 99); 13 Mar 2011 18:03:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Mar 2011 18:03:23 +0000
X-ASF-Spam-Status: No, hits=-1998.3 required=5.0
	tests=ALL_TRUSTED,DEAR_SOMETHING,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Mar 2011 18:03:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 873CC3A782D
	for <common-issues@hadoop.apache.org>; Sun, 13 Mar 2011 18:02:59 +0000 (UTC)
Date: Sun, 13 Mar 2011 18:02:59 +0000 (UTC)
From: "Shekhar Gupta (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2036522478.16304.1300039379550.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <171998182.15391.1299930119568.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7185) ERROR streaming.StreamJob: Error
 Launching job : Input path does not exist
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Shekhar Gupta updated HADOOP-7185:
----------------------------------

    Attachment: rehadoop.zip

Dear Sir

Thank you for your reply on 'jira'.Here's a complete description of the
problem.I  have also attached the conf files in case there's some error
there.

hadoop@shekhar-virtual-machine:~$ /home/hadoop/bin/hadoop jar
contrib/streaming/hadoop-0.20.2-streaming.jar -file /home/hadoop/mapper.py
-mapper /home/hadoop/mapper.py -file /home/hadoop/reducer.py -reducer
/home/hadoop/reducer.py -input /home/hadoop/dfs/data/* -output

/home/hadoop/programs/outputpackageJobJar:
[/tmp/hadoop/hadoop-unjar2244766030749072387/] []
/tmp/streamjob1080103071461565259.jar tmpDir=null
11/02/24 17:06:16 ERROR streaming.StreamJob: Error Launching job : Input
path does not exist:
hdfs://localhost:54310/home/hadoop/Django-1.1.1/examples/data/Think and Grow
Rich!.txt
Input path does not exist:
hdfs://localhost:54310/home/hadoop/Django-1.1.1/examples/data/Thousand
Nights23.txt
Input path does not exist:
hdfs://localhost:54310/home/hadoop/Django-1.1.1/examples/data/Thousand
Nights432].txt
Input path does not exist:
hdfs://localhost:54310/home/hadoop/Django-1.1.1/examples/data/Thousand
Nights65.txt
Input path does not exist:
hdfs://localhost:54310/home/hadoop/Django-1.1.1/examples/data/Thousand
Nights.txt
Streaming Job Failed!

Best Regards


 Shekhar Gupta



On Sun, Mar 13, 2011 at 3:51 PM, Amareshwari Sriramadasu (JIRA) <



> ERROR streaming.StreamJob: Error Launching job : Input path does not exist
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7185
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7185
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>         Environment: Linux
>            Reporter: Shekhar Gupta
>         Attachments: rehadoop.zip
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13574-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Mar 13 20:58:25 2011
Return-Path: <common-issues-return-13574-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56247 invoked from network); 13 Mar 2011 20:58:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Mar 2011 20:58:25 -0000
Received: (qmail 10765 invoked by uid 500); 13 Mar 2011 20:58:25 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10732 invoked by uid 500); 13 Mar 2011 20:58:25 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10724 invoked by uid 99); 13 Mar 2011 20:58:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Mar 2011 20:58:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Mar 2011 20:58:22 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7C3223A7517
	for <common-issues@hadoop.apache.org>; Sun, 13 Mar 2011 20:57:59 +0000 (UTC)
Date: Sun, 13 Mar 2011 20:57:59 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1342904748.16461.1300049879505.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006276#comment-13006276 ] 

Eric Yang commented on HADOOP-6255:
-----------------------------------

Package builder design

For supporting multiple type of packages, this project layout the packaging source code structure like this:

{noformat}
src/packages/rpm
            /deb
            /conf-pseudo
{noformat}

rpm - meta data for creating rpm packages.  SysV init style startup script is also included for start up process in Redhat like environment.

deb - meta data for creating debian packages.  BSD init style startup script is also included for start up process in Ubuntu like environment.

conf-pseudo - Configuration template for demo pseudo cluster setup.  By default both rpm, or deb binary package does not startup the system.  The purpose of conf-pseudo is to create a (rpm/deb) package as demonstration of how to setup a single node cluster and turn on services by configuration.

Software home directory is designed to locate in:

${prefix}/share/${project}

src/packages/update-${project}-env.sh runs in the post installation phase which creates symlinks and making software structure to map to the proposed layout in HADOOP-6255

/etc/default/${project}-evn.sh is symlinked to the project environment script.  Hence, project environment variables are shared across projects.

Project build file can override the package path in the build phase:

Sample build.properties
{noformat}
package.prefix=/usr
package.conf.dir=/etc/${project}
package.log.dir=/var/log/${project}
package.pid.dir=/var/log/${project}
{noformat}

For RPM package, it is possible to override location at installation phase by specifying:

{noformat}
rpm -i ${project}-[version]-[rev].[arch].rpm \
  --relocate /usr=/usr/local/hadoop \
  --relocate /etc/hadoop=/usr/local/etc/hadoop \
  --relocate /var/log/hadoop=/opt/logs/hadoop \
  --relocate /var/run/hadoop=/opt/run/hadoop
{noformat}

The same build structure can be apply to both ant or maven build scripts.  It also expandable to include mac native package installer using this design pattern.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.100
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.100
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13575-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 14 13:13:52 2011
Return-Path: <common-issues-return-13575-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72255 invoked from network); 14 Mar 2011 13:13:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Mar 2011 13:13:52 -0000
Received: (qmail 19129 invoked by uid 500); 14 Mar 2011 13:13:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18961 invoked by uid 500); 14 Mar 2011 13:13:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18952 invoked by uid 99); 14 Mar 2011 13:13:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 13:13:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 13:13:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B7A463A862C
	for <common-issues@hadoop.apache.org>; Mon, 14 Mar 2011 13:13:29 +0000 (UTC)
Date: Mon, 14 Mar 2011 13:13:29 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1430617657.560.1300108409748.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7186)  BackUpNameNode is using 100% CPU and
 not accepting any requests.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

 BackUpNameNode is using 100% CPU and not accepting any requests.
-----------------------------------------------------------------

                 Key: HADOOP-7186
                 URL: https://issues.apache.org/jira/browse/HADOOP-7186
             Project: Hadoop Common
          Issue Type: Bug
          Components: io, ipc
            Reporter: Uma Maheswara Rao G


In our environment, after 3 days long run Backup NameNode is using 100% CPU and not accepting any calls. 
*Thread dump*
"IPC Server Responder" daemon prio=10 tid=0x00007f86c41c6800 nid=0x3b2a runnable [0x00007f86ce579000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

locked <0x00007f86d67e2a20> (a sun.nio.ch.Util$1) 
locked <0x00007f86d67e2a08> (a java.util.Collections$UnmodifiableSet) 
locked <0x00007f86d67e26a8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.apache.hadoop.ipc.Server$Responder.run(Server.java:501) 
Looks like we are running into similar issue like this Jetty one. http://jira.codehaus.org/browse/JETTY-937



--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13576-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 14 13:21:56 2011
Return-Path: <common-issues-return-13576-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94909 invoked from network); 14 Mar 2011 13:21:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Mar 2011 13:21:55 -0000
Received: (qmail 32054 invoked by uid 500); 14 Mar 2011 13:21:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31966 invoked by uid 500); 14 Mar 2011 13:21:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31945 invoked by uid 99); 14 Mar 2011 13:21:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 13:21:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 13:21:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9D15E3A88D3
	for <common-issues@hadoop.apache.org>; Mon, 14 Mar 2011 13:21:29 +0000 (UTC)
Date: Mon, 14 Mar 2011 13:21:29 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1602169021.566.1300108889640.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7178) copyToLocal API is creating .crc
 files in local, even after setting verifyChecksum to false at client side.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uma Maheswara Rao G updated HADOOP-7178:
----------------------------------------

    Attachment: HADOOP-7178_COMMON.patch

> copyToLocal API is creating .crc files in local, even after setting verifyChecksum to false at client side.
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7178
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7178
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7178_COMMON.patch, HADOOP-7178_HDFS.patch
>
>
> When we copy the files from DFS to local, it is creating the .crc file in local filesystem for the verification of checksum even if we disable the checksum verification at client side.
>         When user does not want to do any checksum verification, then what will be the use in creating of these files in local file system.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13577-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 14 13:21:56 2011
Return-Path: <common-issues-return-13577-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94910 invoked from network); 14 Mar 2011 13:21:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Mar 2011 13:21:55 -0000
Received: (qmail 32173 invoked by uid 500); 14 Mar 2011 13:21:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31968 invoked by uid 500); 14 Mar 2011 13:21:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31946 invoked by uid 99); 14 Mar 2011 13:21:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 13:21:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 13:21:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AB01B3A88D5
	for <common-issues@hadoop.apache.org>; Mon, 14 Mar 2011 13:21:29 +0000 (UTC)
Date: Mon, 14 Mar 2011 13:21:29 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1373727997.568.1300108889697.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7178) copyToLocal API is creating .crc
 files in local, even after setting verifyChecksum to false at client side.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uma Maheswara Rao G updated HADOOP-7178:
----------------------------------------

    Attachment: HADOOP-7178_HDFS.patch

> copyToLocal API is creating .crc files in local, even after setting verifyChecksum to false at client side.
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7178
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7178
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7178_COMMON.patch, HADOOP-7178_HDFS.patch
>
>
> When we copy the files from DFS to local, it is creating the .crc file in local filesystem for the verification of checksum even if we disable the checksum verification at client side.
>         When user does not want to do any checksum verification, then what will be the use in creating of these files in local file system.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13578-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 14 13:25:51 2011
Return-Path: <common-issues-return-13578-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13256 invoked from network); 14 Mar 2011 13:25:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Mar 2011 13:25:51 -0000
Received: (qmail 41589 invoked by uid 500); 14 Mar 2011 13:25:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41569 invoked by uid 500); 14 Mar 2011 13:25:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41561 invoked by uid 99); 14 Mar 2011 13:25:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 13:25:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 13:25:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9D8EB3A8ADA
	for <common-issues@hadoop.apache.org>; Mon, 14 Mar 2011 13:25:29 +0000 (UTC)
Date: Mon, 14 Mar 2011 13:25:29 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <459330197.578.1300109129642.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7178) copyToLocal API is creating .crc
 files in local, even after setting verifyChecksum to false at client side.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uma Maheswara Rao G updated HADOOP-7178:
----------------------------------------

    Status: Patch Available  (was: Open)

> copyToLocal API is creating .crc files in local, even after setting verifyChecksum to false at client side.
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7178
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7178
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7178_COMMON.patch, HADOOP-7178_HDFS.patch
>
>
> When we copy the files from DFS to local, it is creating the .crc file in local filesystem for the verification of checksum even if we disable the checksum verification at client side.
>         When user does not want to do any checksum verification, then what will be the use in creating of these files in local file system.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13579-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 14 13:25:54 2011
Return-Path: <common-issues-return-13579-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13465 invoked from network); 14 Mar 2011 13:25:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Mar 2011 13:25:53 -0000
Received: (qmail 41911 invoked by uid 500); 14 Mar 2011 13:25:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41870 invoked by uid 500); 14 Mar 2011 13:25:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41862 invoked by uid 99); 14 Mar 2011 13:25:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 13:25:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 13:25:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8F8353A8AD7
	for <common-issues@hadoop.apache.org>; Mon, 14 Mar 2011 13:25:29 +0000 (UTC)
Date: Mon, 14 Mar 2011 13:25:29 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1115130749.576.1300109129584.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7178) copyToLocal API is creating .crc
 files in local, even after setting verifyChecksum to false at client side.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006401#comment-13006401 ] 

Uma Maheswara Rao G commented on HADOOP-7178:
---------------------------------------------

Implemented the  getVerifyChecksum method in DistributedFileSystem. Provided the patch for the changes in HDFS. So, HADOOP-7178_HDFS.patch need to be updated in HDFS project.

> copyToLocal API is creating .crc files in local, even after setting verifyChecksum to false at client side.
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7178
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7178
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7178_COMMON.patch, HADOOP-7178_HDFS.patch
>
>
> When we copy the files from DFS to local, it is creating the .crc file in local filesystem for the verification of checksum even if we disable the checksum verification at client side.
>         When user does not want to do any checksum verification, then what will be the use in creating of these files in local file system.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13580-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 14 13:46:56 2011
Return-Path: <common-issues-return-13580-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82899 invoked from network); 14 Mar 2011 13:46:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Mar 2011 13:46:56 -0000
Received: (qmail 71291 invoked by uid 500); 14 Mar 2011 13:46:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71237 invoked by uid 500); 14 Mar 2011 13:46:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71086 invoked by uid 99); 14 Mar 2011 13:46:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 13:46:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 13:46:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 008173A8394
	for <common-issues@hadoop.apache.org>; Mon, 14 Mar 2011 13:46:30 +0000 (UTC)
Date: Mon, 14 Mar 2011 13:46:29 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1627534738.611.1300110389998.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7187) Socket Leak in
 org.apache.hadoop.metrics.ganglia.GangliaContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Socket Leak in org.apache.hadoop.metrics.ganglia.GangliaContext
---------------------------------------------------------------

                 Key: HADOOP-7187
                 URL: https://issues.apache.org/jira/browse/HADOOP-7187
             Project: Hadoop Common
          Issue Type: Bug
          Components: metrics
         Environment: Linux
            Reporter: Uma Maheswara Rao G
            Assignee: Uma Maheswara Rao G


Init method is creating DatagramSocket. But this is not closed any where. 


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13581-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 14 15:23:54 2011
Return-Path: <common-issues-return-13581-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4627 invoked from network); 14 Mar 2011 15:23:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Mar 2011 15:23:53 -0000
Received: (qmail 15927 invoked by uid 500); 14 Mar 2011 15:23:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15901 invoked by uid 500); 14 Mar 2011 15:23:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15893 invoked by uid 99); 14 Mar 2011 15:23:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 15:23:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 15:23:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 908143A862A
	for <common-issues@hadoop.apache.org>; Mon, 14 Mar 2011 15:23:29 +0000 (UTC)
Date: Mon, 14 Mar 2011 15:23:29 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1254924605.947.1300116209588.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7188) Chance of Resource Leak in
 org.apache.hadoop.hdfs.server.namenode.FsImage.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Chance of Resource Leak in org.apache.hadoop.hdfs.server.namenode.FsImage.
--------------------------------------------------------------------------

                 Key: HADOOP-7188
                 URL: https://issues.apache.org/jira/browse/HADOOP-7188
             Project: Hadoop Common
          Issue Type: Bug
         Environment: Linux
            Reporter: Uma Maheswara Rao G


In loadFSEdits method,
1) EditLogFileInputStream edits = new EditLogFileInputStream(NNStorage.getStorageFile(sd, NameNodeFile.EDITS));
numEdits = loader.loadFSEdits(edits);
edits.close(); 
2) edits = new EditLogFileInputStream(editsNew);
numEdits += loader.loadFSEdits(edits);
edits.close();

Here if loadFSEdits throws exception then close will not be executed.


 



--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13582-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 14 15:55:53 2011
Return-Path: <common-issues-return-13582-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14631 invoked from network); 14 Mar 2011 15:55:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Mar 2011 15:55:53 -0000
Received: (qmail 74667 invoked by uid 500); 14 Mar 2011 15:55:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74391 invoked by uid 500); 14 Mar 2011 15:55:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74366 invoked by uid 99); 14 Mar 2011 15:55:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 15:55:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 15:55:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 109753A8756
	for <common-issues@hadoop.apache.org>; Mon, 14 Mar 2011 15:55:30 +0000 (UTC)
Date: Mon, 14 Mar 2011 15:55:30 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <162695217.1066.1300118130064.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1991821639.8902.1299699839419.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Assigned: (HADOOP-7176) Redesign FsShell
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp reassigned HADOOP-7176:
-----------------------------------

    Assignee: Daryn Sharp

> Redesign FsShell
> ----------------
>
>                 Key: HADOOP-7176
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7176
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.22.0
>
>
> The FsShell commands are very tightly coupled together which makes it necessarily hard to write new commands.  There is a lot of redundancy between the commands, inconsistencies in handling of paths, handling of errors, and correct return of exit codes.  The FsShell commands should be subclasses of the fs.shell.Command class which is already being used by the -count command, and is used by other commands like dfsadmin.
> This will serve as an umbrella bug to track the changes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13583-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 14 18:33:54 2011
Return-Path: <common-issues-return-13583-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51931 invoked from network); 14 Mar 2011 18:33:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Mar 2011 18:33:54 -0000
Received: (qmail 82879 invoked by uid 500); 14 Mar 2011 18:33:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82833 invoked by uid 500); 14 Mar 2011 18:33:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82825 invoked by uid 99); 14 Mar 2011 18:33:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 18:33:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 18:33:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1012C3A919E
	for <common-issues@hadoop.apache.org>; Mon, 14 Mar 2011 18:33:30 +0000 (UTC)
Date: Mon, 14 Mar 2011 18:33:30 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2102703747.1479.1300127610062.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1409102499.8762.1299697139647.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7175) Add isEnabled() to Trash
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006558#comment-13006558 ] 

Daryn Sharp commented on HADOOP-7175:
-------------------------------------

test-contrib:

test:

compile:

test:

python.pathcheck:
     [echo] 'python.home' is not defined. Please pass -Dpython.home=<Path to Python> to Ant on the command-line.

checkAndRunTests:

BUILD SUCCESSFUL


> Add isEnabled() to Trash
> ------------------------
>
>                 Key: HADOOP-7175
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7175
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7175-2.patch, HADOOP-7175.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The moveToTrash method returns false in a number of cases.  It's not possible to discern if false means an error occurred. In particular, it's not possible to know if the trash is disabled vs. an error occurred.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13584-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 14 18:41:52 2011
Return-Path: <common-issues-return-13584-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70801 invoked from network); 14 Mar 2011 18:41:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Mar 2011 18:41:51 -0000
Received: (qmail 95428 invoked by uid 500); 14 Mar 2011 18:41:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95392 invoked by uid 500); 14 Mar 2011 18:41:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95384 invoked by uid 99); 14 Mar 2011 18:41:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 18:41:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 18:41:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DEF6F3A9436
	for <common-issues@hadoop.apache.org>; Mon, 14 Mar 2011 18:41:29 +0000 (UTC)
Date: Mon, 14 Mar 2011 18:41:29 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1890924886.1499.1300128089909.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Assigned: (HADOOP-6962) FileSystem.mkdirs(Path,
 FSPermission) should use the permission for all of the created directories
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp reassigned HADOOP-6962:
-----------------------------------

    Assignee: Daryn Sharp

> FileSystem.mkdirs(Path, FSPermission) should use the permission for all of the created directories
> --------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6962
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6962
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Owen O'Malley
>            Assignee: Daryn Sharp
>
> Currently, FileSystem.mkdirs only applies the permissions to the last level if it was created. It should be applied to *all* levels that are created.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13585-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 14 19:27:54 2011
Return-Path: <common-issues-return-13585-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19293 invoked from network); 14 Mar 2011 19:27:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Mar 2011 19:27:54 -0000
Received: (qmail 64716 invoked by uid 500); 14 Mar 2011 19:27:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64691 invoked by uid 500); 14 Mar 2011 19:27:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64683 invoked by uid 99); 14 Mar 2011 19:27:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 19:27:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 19:27:52 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E16AB3A84DD
	for <common-issues@hadoop.apache.org>; Mon, 14 Mar 2011 19:27:29 +0000 (UTC)
Date: Mon, 14 Mar 2011 19:27:29 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <298964249.1635.1300130849919.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-6949) Reduces RPC packet size for
 primitive arrays, especially long[], which is used at block reporting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006586#comment-13006586 ] 

Suresh Srinivas commented on HADOOP-6949:
-----------------------------------------

This is an important patch. Thanks for doing it in a backward compatible way and also in a way where both on disk format and RPC format can co-exist.

Comments:
# ArrayPrimitiveWriable
#* Class javadoc is not very clear - not sure what you mean by boxed type not supported by ObjectWritable and why that should be mentioned in the javadoc. I am not sure also how clients can provide external locking; more appropriate would be to say, pass a copy of the array, if concurrent modifications are expected.
#* Comments related to declaredComponentType is not clear. Can you please add description on what componentType and declaredComponentType are and how they are used, bef adding further description.
#* Is PrimitiveArrayWritable a better name?
#* Add @InterfaceAudience.Public and @InterfaceStability.Stable
#* Throw HadoopIllegalArgumentException instead of NullPointerException and IllegalArgumentException
#* Optional: The read write methods could be made static methods in WritableUtils class, so it can be used by others. Also PrimitiveType map, isCanditdatePrimitive()  could also move to helper.
#* Rename isCandidatePrimitive() to isPrimitive()?
#* Is constructor with componentType as argument used any where?
#* Some methods could be private - for example - getDeclaredComponentType(), isDeclaredComponentType(), 
#* Can you use Text instead of UTF8? This avoids deprecation warnings. You could also use WritableUtils to write and read string. This can be done in many other places in the patch.
#* I prefer MalformedInputException instead of ProtocolException (see Text.java for example)
# ObjectWritable.java
#* Can you call set() method while setting instance and declaredClass in #writeObject() method?
#* Not sure about the comment: // This case must come after the "isArray()" case
#* Not sure why this condition is required in #writeObject() - instance.getClass().getName().equals(declaredClass.getName())
# TestArrayWritable
#* In the class javadoc can you add link to PrimitiveArrayWritable?
#* Can you make in, out, bigSet, resultSet and expectedSet final?
#* Should we prefer reading and writing objects using readFields and writeFields - since they are the methods supporting Writable contract?


> Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6949
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6949
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Navis
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: ObjectWritable.diff, arrayprim.patch, arrayprim_v4.patch, arrayprim_v5.patch, arrayprim_v6.patch
>
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n
> It would not be needed to specify each element types for primitive arrays.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13586-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 14 20:57:57 2011
Return-Path: <common-issues-return-13586-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9885 invoked from network); 14 Mar 2011 20:57:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Mar 2011 20:57:56 -0000
Received: (qmail 55342 invoked by uid 500); 14 Mar 2011 20:57:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55301 invoked by uid 500); 14 Mar 2011 20:57:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55258 invoked by uid 99); 14 Mar 2011 20:57:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 20:57:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 20:57:52 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 81CF73A9BC7
	for <common-issues@hadoop.apache.org>; Mon, 14 Mar 2011 20:57:30 +0000 (UTC)
Date: Mon, 14 Mar 2011 20:57:30 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <683476535.2038.1300136250528.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7189) Add ability to enable 'debug'
 property in JAAS configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Add ability to enable 'debug' property in JAAS configuration
------------------------------------------------------------

                 Key: HADOOP-7189
                 URL: https://issues.apache.org/jira/browse/HADOOP-7189
             Project: Hadoop Common
          Issue Type: Improvement
          Components: security
    Affects Versions: 0.23.0
            Reporter: Todd Lipcon
            Priority: Minor


Occasionally users have run into weird "Unable to login" messages. Unfortunately, JAAS obscures the underlying exception message in many cases because it thinks leaking the exception might be insecure in itself. Enabling the "debug" option in the JAAS configuration gets it to dump the underlying issue and makes troubleshooting this kind of issue easier.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13587-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 14 20:59:54 2011
Return-Path: <common-issues-return-13587-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10471 invoked from network); 14 Mar 2011 20:59:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Mar 2011 20:59:54 -0000
Received: (qmail 59002 invoked by uid 500); 14 Mar 2011 20:59:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58964 invoked by uid 500); 14 Mar 2011 20:59:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58956 invoked by uid 99); 14 Mar 2011 20:59:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 20:59:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 20:59:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E991C3A9CF2
	for <common-issues@hadoop.apache.org>; Mon, 14 Mar 2011 20:59:29 +0000 (UTC)
Date: Mon, 14 Mar 2011 20:59:29 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1702989770.2044.1300136369953.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <683476535.2038.1300136250528.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7189) Add ability to enable 'debug'
 property in JAAS configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7189:
--------------------------------

    Attachment: enable-UGI-debug-example.txt

this isn't a patch meant to actually be committed - but just shows the debug setting I'm talking about. This should be made conditional by a Java property or env var.

> Add ability to enable 'debug' property in JAAS configuration
> ------------------------------------------------------------
>
>                 Key: HADOOP-7189
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7189
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Priority: Minor
>         Attachments: enable-UGI-debug-example.txt
>
>
> Occasionally users have run into weird "Unable to login" messages. Unfortunately, JAAS obscures the underlying exception message in many cases because it thinks leaking the exception might be insecure in itself. Enabling the "debug" option in the JAAS configuration gets it to dump the underlying issue and makes troubleshooting this kind of issue easier.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13588-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 14 21:08:52 2011
Return-Path: <common-issues-return-13588-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63593 invoked from network); 14 Mar 2011 21:08:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Mar 2011 21:08:51 -0000
Received: (qmail 77803 invoked by uid 500); 14 Mar 2011 21:08:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77768 invoked by uid 500); 14 Mar 2011 21:08:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77760 invoked by uid 99); 14 Mar 2011 21:08:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 21:08:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 21:08:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3F75C3A9035
	for <common-issues@hadoop.apache.org>; Mon, 14 Mar 2011 21:08:30 +0000 (UTC)
Date: Mon, 14 Mar 2011 21:08:30 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1420232843.2072.1300136910256.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <683476535.2038.1300136250528.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7189) Add ability to enable 'debug'
 property in JAAS configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006635#comment-13006635 ] 

Allen Wittenauer commented on HADOOP-7189:
------------------------------------------

I agree.  This definitely looks like something that should get reformulated so that we could set it for debugging without rebuilding and/or (even worse) restarting the service.

> Add ability to enable 'debug' property in JAAS configuration
> ------------------------------------------------------------
>
>                 Key: HADOOP-7189
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7189
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Priority: Minor
>         Attachments: enable-UGI-debug-example.txt
>
>
> Occasionally users have run into weird "Unable to login" messages. Unfortunately, JAAS obscures the underlying exception message in many cases because it thinks leaking the exception might be insecure in itself. Enabling the "debug" option in the JAAS configuration gets it to dump the underlying issue and makes troubleshooting this kind of issue easier.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13589-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 14 22:21:51 2011
Return-Path: <common-issues-return-13589-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83519 invoked from network); 14 Mar 2011 22:21:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Mar 2011 22:21:51 -0000
Received: (qmail 90818 invoked by uid 500); 14 Mar 2011 22:21:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90784 invoked by uid 500); 14 Mar 2011 22:21:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90776 invoked by uid 99); 14 Mar 2011 22:21:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 22:21:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 22:21:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8869C3A9691
	for <common-issues@hadoop.apache.org>; Mon, 14 Mar 2011 22:21:29 +0000 (UTC)
Date: Mon, 14 Mar 2011 22:21:29 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1005823317.2243.1300141289555.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7190) Put metrics v1 back into the
 hadoop-20-security branch
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Put metrics v1 back into the hadoop-20-security branch
------------------------------------------------------

                 Key: HADOOP-7190
                 URL: https://issues.apache.org/jira/browse/HADOOP-7190
             Project: Hadoop Common
          Issue Type: Bug
          Components: metrics
            Reporter: Owen O'Malley
            Assignee: Owen O'Malley
             Fix For: 0.20.100


The metrics v1 code was removed on the branch. It should be put back and deprecated.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13590-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 14 22:23:54 2011
Return-Path: <common-issues-return-13590-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92733 invoked from network); 14 Mar 2011 22:23:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Mar 2011 22:23:54 -0000
Received: (qmail 94073 invoked by uid 500); 14 Mar 2011 22:23:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94002 invoked by uid 500); 14 Mar 2011 22:23:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93981 invoked by uid 99); 14 Mar 2011 22:23:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 22:23:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 22:23:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9F48D3A9790
	for <common-issues@hadoop.apache.org>; Mon, 14 Mar 2011 22:23:29 +0000 (UTC)
Date: Mon, 14 Mar 2011 22:23:29 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1956241619.2248.1300141409649.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1005823317.2243.1300141289555.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7190) Put metrics v1 back into the
 hadoop-20-security branch
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7190:
----------------------------------

    Attachment: metrics1.patch

> Put metrics v1 back into the hadoop-20-security branch
> ------------------------------------------------------
>
>                 Key: HADOOP-7190
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7190
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.100
>
>         Attachments: metrics1.patch
>
>
> The metrics v1 code was removed on the branch. It should be put back and deprecated.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13591-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 14 22:23:55 2011
Return-Path: <common-issues-return-13591-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92904 invoked from network); 14 Mar 2011 22:23:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Mar 2011 22:23:55 -0000
Received: (qmail 94810 invoked by uid 500); 14 Mar 2011 22:23:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94777 invoked by uid 500); 14 Mar 2011 22:23:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94769 invoked by uid 99); 14 Mar 2011 22:23:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 22:23:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 22:23:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0A0D43A97A2
	for <common-issues@hadoop.apache.org>; Mon, 14 Mar 2011 22:23:30 +0000 (UTC)
Date: Mon, 14 Mar 2011 22:23:30 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <859825523.2261.1300141410037.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1005823317.2243.1300141289555.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7190) Put metrics v1 back into the
 hadoop-20-security branch
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7190:
----------------------------------

    Status: Patch Available  (was: Open)

> Put metrics v1 back into the hadoop-20-security branch
> ------------------------------------------------------
>
>                 Key: HADOOP-7190
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7190
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.100
>
>         Attachments: metrics1.patch
>
>
> The metrics v1 code was removed on the branch. It should be put back and deprecated.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13592-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 14 22:31:54 2011
Return-Path: <common-issues-return-13592-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20081 invoked from network); 14 Mar 2011 22:31:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Mar 2011 22:31:54 -0000
Received: (qmail 12848 invoked by uid 500); 14 Mar 2011 22:31:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12825 invoked by uid 500); 14 Mar 2011 22:31:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12817 invoked by uid 99); 14 Mar 2011 22:31:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 22:31:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 22:31:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B70553A9A33
	for <common-issues@hadoop.apache.org>; Mon, 14 Mar 2011 22:31:29 +0000 (UTC)
Date: Mon, 14 Mar 2011 22:31:29 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <991480301.2282.1300141889746.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1409102499.8762.1299697139647.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7175) Add isEnabled() to Trash
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006681#comment-13006681 ] 

Hadoop QA commented on HADOOP-7175:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12473427/HADOOP-7175-2.patch
  against trunk revision 1080817.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/309//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/309//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/309//console

This message is automatically generated.

> Add isEnabled() to Trash
> ------------------------
>
>                 Key: HADOOP-7175
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7175
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7175-2.patch, HADOOP-7175.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The moveToTrash method returns false in a number of cases.  It's not possible to discern if false means an error occurred. In particular, it's not possible to know if the trash is disabled vs. an error occurred.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13593-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 14 22:37:51 2011
Return-Path: <common-issues-return-13593-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53109 invoked from network); 14 Mar 2011 22:37:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Mar 2011 22:37:51 -0000
Received: (qmail 19119 invoked by uid 500); 14 Mar 2011 22:37:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19079 invoked by uid 500); 14 Mar 2011 22:37:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19071 invoked by uid 99); 14 Mar 2011 22:37:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 22:37:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 22:37:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 86A0A3A9C3E
	for <common-issues@hadoop.apache.org>; Mon, 14 Mar 2011 22:37:29 +0000 (UTC)
Date: Mon, 14 Mar 2011 22:37:29 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <525666833.2286.1300142249548.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1005823317.2243.1300141289555.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7190) Put metrics v1 back into the
 hadoop-20-security branch
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7190:
----------------------------------

    Attachment: metrics1.patch

I missed a directory (logs/metrics). I've also included the interface classification stuff so that it is available.

> Put metrics v1 back into the hadoop-20-security branch
> ------------------------------------------------------
>
>                 Key: HADOOP-7190
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7190
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.100
>
>         Attachments: metrics1.patch, metrics1.patch
>
>
> The metrics v1 code was removed on the branch. It should be put back and deprecated.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13594-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 14 22:45:55 2011
Return-Path: <common-issues-return-13594-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62276 invoked from network); 14 Mar 2011 22:45:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Mar 2011 22:45:55 -0000
Received: (qmail 24064 invoked by uid 500); 14 Mar 2011 22:45:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24030 invoked by uid 500); 14 Mar 2011 22:45:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24022 invoked by uid 99); 14 Mar 2011 22:45:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 22:45:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 22:45:52 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0B5103A9FA5
	for <common-issues@hadoop.apache.org>; Mon, 14 Mar 2011 22:45:31 +0000 (UTC)
Date: Mon, 14 Mar 2011 22:45:31 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <274609777.2314.1300142731043.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1409102499.8762.1299697139647.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7175) Add isEnabled() to Trash
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7175:
-------------------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I have committed this.  Thanks, Daryn!

(Hudson seems not able to pick up COMMON patches automatically.  I have manually started the hudson build.)

> Add isEnabled() to Trash
> ------------------------
>
>                 Key: HADOOP-7175
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7175
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7175-2.patch, HADOOP-7175.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The moveToTrash method returns false in a number of cases.  It's not possible to discern if false means an error occurred. In particular, it's not possible to know if the trash is disabled vs. an error occurred.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13595-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 14 22:49:54 2011
Return-Path: <common-issues-return-13595-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96990 invoked from network); 14 Mar 2011 22:49:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Mar 2011 22:49:54 -0000
Received: (qmail 28053 invoked by uid 500); 14 Mar 2011 22:49:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28009 invoked by uid 500); 14 Mar 2011 22:49:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28001 invoked by uid 99); 14 Mar 2011 22:49:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 22:49:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 22:49:52 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 567B33A90F1
	for <common-issues@hadoop.apache.org>; Mon, 14 Mar 2011 22:49:30 +0000 (UTC)
Date: Mon, 14 Mar 2011 22:49:30 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <551966431.2327.1300142970350.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1005823317.2243.1300141289555.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7190) Put metrics v1 back into the
 hadoop-20-security branch
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7190:
----------------------------------

    Attachment: metrics1.patch

Third try on the various EventCounters.

> Put metrics v1 back into the hadoop-20-security branch
> ------------------------------------------------------
>
>                 Key: HADOOP-7190
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7190
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.100
>
>         Attachments: metrics1.patch, metrics1.patch, metrics1.patch
>
>
> The metrics v1 code was removed on the branch. It should be put back and deprecated.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13596-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 14 23:05:55 2011
Return-Path: <common-issues-return-13596-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52389 invoked from network); 14 Mar 2011 23:05:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Mar 2011 23:05:54 -0000
Received: (qmail 47859 invoked by uid 500); 14 Mar 2011 23:05:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47722 invoked by uid 500); 14 Mar 2011 23:05:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47706 invoked by uid 99); 14 Mar 2011 23:05:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 23:05:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 23:05:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E14A13A9619
	for <common-issues@hadoop.apache.org>; Mon, 14 Mar 2011 23:05:29 +0000 (UTC)
Date: Mon, 14 Mar 2011 23:05:29 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1411350321.2349.1300143929919.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1409102499.8762.1299697139647.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7175) Add isEnabled() to Trash
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006703#comment-13006703 ] 

Hudson commented on HADOOP-7175:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #528 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/528/])
    HADOOP-7175. Add isEnabled() to Trash.  Contributed by Daryn Sharp


> Add isEnabled() to Trash
> ------------------------
>
>                 Key: HADOOP-7175
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7175
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7175-2.patch, HADOOP-7175.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The moveToTrash method returns false in a number of cases.  It's not possible to discern if false means an error occurred. In particular, it's not possible to know if the trash is disabled vs. an error occurred.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13597-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 14 23:45:51 2011
Return-Path: <common-issues-return-13597-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54975 invoked from network); 14 Mar 2011 23:45:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Mar 2011 23:45:51 -0000
Received: (qmail 89877 invoked by uid 500); 14 Mar 2011 23:45:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89843 invoked by uid 500); 14 Mar 2011 23:45:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89835 invoked by uid 99); 14 Mar 2011 23:45:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 23:45:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Mar 2011 23:45:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E484A3A91ED
	for <common-issues@hadoop.apache.org>; Mon, 14 Mar 2011 23:45:29 +0000 (UTC)
Date: Mon, 14 Mar 2011 23:45:29 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1609508724.2457.1300146329932.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1005823317.2243.1300141289555.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7190) Put metrics v1 back into the
 hadoop-20-security branch
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006724#comment-13006724 ] 

Luke Lu commented on HADOOP-7190:
---------------------------------

+1, lgtm.

> Put metrics v1 back into the hadoop-20-security branch
> ------------------------------------------------------
>
>                 Key: HADOOP-7190
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7190
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.203.0
>
>         Attachments: metrics1.patch, metrics1.patch, metrics1.patch
>
>
> The metrics v1 code was removed on the branch. It should be put back and deprecated.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13598-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 15 05:28:55 2011
Return-Path: <common-issues-return-13598-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44907 invoked from network); 15 Mar 2011 05:28:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Mar 2011 05:28:55 -0000
Received: (qmail 77464 invoked by uid 500); 15 Mar 2011 05:28:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77372 invoked by uid 500); 15 Mar 2011 05:28:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77364 invoked by uid 99); 15 Mar 2011 05:28:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 05:28:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 05:28:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8D42C3AA192
	for <common-issues@hadoop.apache.org>; Tue, 15 Mar 2011 05:28:29 +0000 (UTC)
Date: Tue, 15 Mar 2011 05:28:29 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1732043207.2898.1300166909575.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1430617657.560.1300108409748.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Moved: (HADOOP-7191)  BackUpNameNode is using 100% CPU and
 not accepting any requests.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uma Maheswara Rao G moved HDFS-1756 to HADOOP-7191:
---------------------------------------------------

    Component/s:     (was: name-node)
                 ipc
            Key: HADOOP-7191  (was: HDFS-1756)
        Project: Hadoop Common  (was: Hadoop HDFS)

>  BackUpNameNode is using 100% CPU and not accepting any requests.
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7191
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7191
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>            Reporter: Uma Maheswara Rao G
>
> In our environment, after 3 days long run Backup NameNode is using 100% CPU and not accepting any calls. 
> *Thread dump*
> "IPC Server Responder" daemon prio=10 tid=0x00007f86c41c6800 nid=0x3b2a runnable [0x00007f86ce579000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> locked <0x00007f86d67e2a20> (a sun.nio.ch.Util$1) 
> locked <0x00007f86d67e2a08> (a java.util.Collections$UnmodifiableSet) 
> locked <0x00007f86d67e26a8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.hadoop.ipc.Server$Responder.run(Server.java:501) 
> Looks like we are running into similar issue like this Jetty one. http://jira.codehaus.org/browse/JETTY-937

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13599-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 15 07:01:51 2011
Return-Path: <common-issues-return-13599-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59640 invoked from network); 15 Mar 2011 07:01:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Mar 2011 07:01:51 -0000
Received: (qmail 55552 invoked by uid 500); 15 Mar 2011 07:01:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55513 invoked by uid 500); 15 Mar 2011 07:01:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55505 invoked by uid 99); 15 Mar 2011 07:01:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 07:01:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 07:01:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BF3623A94FC
	for <common-issues@hadoop.apache.org>; Tue, 15 Mar 2011 07:01:29 +0000 (UTC)
Date: Tue, 15 Mar 2011 07:01:29 +0000 (UTC)
From: "Harsh J Chouraria (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <510617684.3042.1300172489780.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7192) fs -stat docs aren't updated to
 reflect the format features
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

fs -stat docs aren't updated to reflect the format features
-----------------------------------------------------------

                 Key: HADOOP-7192
                 URL: https://issues.apache.org/jira/browse/HADOOP-7192
             Project: Hadoop Common
          Issue Type: Improvement
          Components: documentation
    Affects Versions: 0.21.0
         Environment: Linux / 0.21
            Reporter: Harsh J Chouraria
            Assignee: Harsh J Chouraria
            Priority: Trivial
             Fix For: 0.23.0


The html docs of the 'fs -stat' command (that is found listed in the File System Shell Guide), does not seem to have the formatting abilities of -stat explained (along with the options).

Like 'fs -help', the docs must also reflect the latest available features.

I shall attach a doc-fix patch shortly.

If anyone has other discrepancies to point out in the web version of the guide, please do so :)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13600-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 15 11:13:51 2011
Return-Path: <common-issues-return-13600-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61792 invoked from network); 15 Mar 2011 11:13:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Mar 2011 11:13:51 -0000
Received: (qmail 32219 invoked by uid 500); 15 Mar 2011 11:13:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32180 invoked by uid 500); 15 Mar 2011 11:13:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32172 invoked by uid 99); 15 Mar 2011 11:13:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 11:13:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 11:13:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CD34B3AAFC8
	for <common-issues@hadoop.apache.org>; Tue, 15 Mar 2011 11:13:29 +0000 (UTC)
Date: Tue, 15 Mar 2011 11:13:29 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1912261929.3368.1300187609837.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1409102499.8762.1299697139647.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7175) Add isEnabled() to Trash
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006872#comment-13006872 ] 

Hudson commented on HADOOP-7175:
--------------------------------

Integrated in Hadoop-Common-trunk #631 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/631/])
    HADOOP-7175. Add isEnabled() to Trash.  Contributed by Daryn Sharp


> Add isEnabled() to Trash
> ------------------------
>
>                 Key: HADOOP-7175
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7175
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7175-2.patch, HADOOP-7175.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> The moveToTrash method returns false in a number of cases.  It's not possible to discern if false means an error occurred. In particular, it's not possible to know if the trash is disabled vs. an error occurred.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13601-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 15 15:12:54 2011
Return-Path: <common-issues-return-13601-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53631 invoked from network); 15 Mar 2011 15:12:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Mar 2011 15:12:53 -0000
Received: (qmail 7189 invoked by uid 500); 15 Mar 2011 15:12:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7052 invoked by uid 500); 15 Mar 2011 15:12:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7044 invoked by uid 99); 15 Mar 2011 15:12:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 15:12:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 15:12:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9C1883AAA79
	for <common-issues@hadoop.apache.org>; Tue, 15 Mar 2011 15:12:29 +0000 (UTC)
Date: Tue, 15 Mar 2011 15:12:29 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1640434528.3799.1300201949636.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1627534738.611.1300110389998.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7187) Socket Leak in
 org.apache.hadoop.metrics.ganglia.GangliaContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uma Maheswara Rao G updated HADOOP-7187:
----------------------------------------

    Attachment: HADOOP-7187.patch

> Socket Leak in org.apache.hadoop.metrics.ganglia.GangliaContext
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7187
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7187
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7187.patch
>
>
> Init method is creating DatagramSocket. But this is not closed any where. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13602-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 15 15:14:54 2011
Return-Path: <common-issues-return-13602-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54250 invoked from network); 15 Mar 2011 15:14:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Mar 2011 15:14:53 -0000
Received: (qmail 10714 invoked by uid 500); 15 Mar 2011 15:14:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10682 invoked by uid 500); 15 Mar 2011 15:14:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10674 invoked by uid 99); 15 Mar 2011 15:14:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 15:14:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 15:14:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9C0E43AAB7C
	for <common-issues@hadoop.apache.org>; Tue, 15 Mar 2011 15:14:29 +0000 (UTC)
Date: Tue, 15 Mar 2011 15:14:29 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2082799532.3805.1300202069635.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1627534738.611.1300110389998.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7187) Socket Leak in
 org.apache.hadoop.metrics.ganglia.GangliaContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uma Maheswara Rao G updated HADOOP-7187:
----------------------------------------

    Status: Patch Available  (was: Open)

> Socket Leak in org.apache.hadoop.metrics.ganglia.GangliaContext
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7187
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7187
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7187.patch
>
>
> Init method is creating DatagramSocket. But this is not closed any where. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13603-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 15 16:25:53 2011
Return-Path: <common-issues-return-13603-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92769 invoked from network); 15 Mar 2011 16:25:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Mar 2011 16:25:53 -0000
Received: (qmail 35971 invoked by uid 500); 15 Mar 2011 16:25:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35944 invoked by uid 500); 15 Mar 2011 16:25:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35936 invoked by uid 99); 15 Mar 2011 16:25:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 16:25:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 16:25:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B0F443AACE6
	for <common-issues@hadoop.apache.org>; Tue, 15 Mar 2011 16:25:29 +0000 (UTC)
Date: Tue, 15 Mar 2011 16:25:29 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1991335107.3976.1300206329721.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1005823317.2243.1300141289555.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7190) Put metrics v1 back into the
 hadoop-20-security branch
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7190:
----------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I just committed this.

> Put metrics v1 back into the hadoop-20-security branch
> ------------------------------------------------------
>
>                 Key: HADOOP-7190
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7190
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.203.0
>
>         Attachments: metrics1.patch, metrics1.patch, metrics1.patch
>
>
> The metrics v1 code was removed on the branch. It should be put back and deprecated.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13604-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 15 18:21:51 2011
Return-Path: <common-issues-return-13604-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58280 invoked from network); 15 Mar 2011 18:21:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Mar 2011 18:21:51 -0000
Received: (qmail 88269 invoked by uid 500); 15 Mar 2011 18:21:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88245 invoked by uid 500); 15 Mar 2011 18:21:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88237 invoked by uid 99); 15 Mar 2011 18:21:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 18:21:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 18:21:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DC82C3AA4B3
	for <common-issues@hadoop.apache.org>; Tue, 15 Mar 2011 18:21:29 +0000 (UTC)
Date: Tue, 15 Mar 2011 18:21:29 +0000 (UTC)
From: "Harsh J Chouraria (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <416799132.4328.1300213289899.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <510617684.3042.1300172489780.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7192) fs -stat docs aren't updated to
 reflect the format features
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J Chouraria updated HADOOP-7192:
--------------------------------------

      Environment: Linux / 0.20  (was: Linux / 0.21)
    Fix Version/s: 0.21.1
                   0.20.3

> fs -stat docs aren't updated to reflect the format features
> -----------------------------------------------------------
>
>                 Key: HADOOP-7192
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7192
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: 0.21.0
>         Environment: Linux / 0.20
>            Reporter: Harsh J Chouraria
>            Assignee: Harsh J Chouraria
>            Priority: Trivial
>              Labels: documentaion
>             Fix For: 0.20.3, 0.21.1, 0.23.0
>
>         Attachments: hadoop.common.fsstatdoc.r1.diff
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> The html docs of the 'fs -stat' command (that is found listed in the File System Shell Guide), does not seem to have the formatting abilities of -stat explained (along with the options).
> Like 'fs -help', the docs must also reflect the latest available features.
> I shall attach a doc-fix patch shortly.
> If anyone has other discrepancies to point out in the web version of the guide, please do so :)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13605-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 15 18:21:54 2011
Return-Path: <common-issues-return-13605-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58496 invoked from network); 15 Mar 2011 18:21:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Mar 2011 18:21:53 -0000
Received: (qmail 88493 invoked by uid 500); 15 Mar 2011 18:21:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88459 invoked by uid 500); 15 Mar 2011 18:21:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88451 invoked by uid 99); 15 Mar 2011 18:21:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 18:21:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 18:21:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A5CEA3AA4AE
	for <common-issues@hadoop.apache.org>; Tue, 15 Mar 2011 18:21:29 +0000 (UTC)
Date: Tue, 15 Mar 2011 18:21:29 +0000 (UTC)
From: "Harsh J Chouraria (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1558378024.4324.1300213289675.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <510617684.3042.1300172489780.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7192) fs -stat docs aren't updated to
 reflect the format features
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J Chouraria updated HADOOP-7192:
--------------------------------------

    Attachment: hadoop.common.fsstatdoc.r1.diff

Doc fixing patch that updates the file system guide.

I couldn't provide a Chinese translation, however (I do not know the language, and do not wish to rely on auto translators ;)).

Requesting someone who knows the Chinese language to extend the patch.

> fs -stat docs aren't updated to reflect the format features
> -----------------------------------------------------------
>
>                 Key: HADOOP-7192
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7192
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: 0.21.0
>         Environment: Linux / 0.21
>            Reporter: Harsh J Chouraria
>            Assignee: Harsh J Chouraria
>            Priority: Trivial
>              Labels: documentaion
>             Fix For: 0.20.3, 0.21.1, 0.23.0
>
>         Attachments: hadoop.common.fsstatdoc.r1.diff
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> The html docs of the 'fs -stat' command (that is found listed in the File System Shell Guide), does not seem to have the formatting abilities of -stat explained (along with the options).
> Like 'fs -help', the docs must also reflect the latest available features.
> I shall attach a doc-fix patch shortly.
> If anyone has other discrepancies to point out in the web version of the guide, please do so :)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13606-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 15 18:31:54 2011
Return-Path: <common-issues-return-13606-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1698 invoked from network); 15 Mar 2011 18:31:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Mar 2011 18:31:54 -0000
Received: (qmail 867 invoked by uid 500); 15 Mar 2011 18:31:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 819 invoked by uid 500); 15 Mar 2011 18:31:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 811 invoked by uid 99); 15 Mar 2011 18:31:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 18:31:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 18:31:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0E21D3AAA0A
	for <common-issues@hadoop.apache.org>; Tue, 15 Mar 2011 18:31:30 +0000 (UTC)
Date: Tue, 15 Mar 2011 18:31:30 +0000 (UTC)
From: "Harsh J Chouraria (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <824854489.4383.1300213890054.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <510617684.3042.1300172489780.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7192) fs -stat docs aren't updated to
 reflect the format features
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J Chouraria updated HADOOP-7192:
--------------------------------------

    Status: Patch Available  (was: Open)

English-only, but marking as PA anyway.

{code} ant docs {code} passes locally.

> fs -stat docs aren't updated to reflect the format features
> -----------------------------------------------------------
>
>                 Key: HADOOP-7192
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7192
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: 0.21.0
>         Environment: Linux / 0.20
>            Reporter: Harsh J Chouraria
>            Assignee: Harsh J Chouraria
>            Priority: Trivial
>              Labels: documentaion
>             Fix For: 0.20.3, 0.21.1, 0.23.0
>
>         Attachments: hadoop.common.fsstatdoc.r1.diff
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> The html docs of the 'fs -stat' command (that is found listed in the File System Shell Guide), does not seem to have the formatting abilities of -stat explained (along with the options).
> Like 'fs -help', the docs must also reflect the latest available features.
> I shall attach a doc-fix patch shortly.
> If anyone has other discrepancies to point out in the web version of the guide, please do so :)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13607-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 15 20:14:54 2011
Return-Path: <common-issues-return-13607-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28191 invoked from network); 15 Mar 2011 20:14:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Mar 2011 20:14:54 -0000
Received: (qmail 33367 invoked by uid 500); 15 Mar 2011 20:14:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33333 invoked by uid 500); 15 Mar 2011 20:14:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33325 invoked by uid 99); 15 Mar 2011 20:14:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 20:14:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 20:14:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C68EA3AA064
	for <common-issues@hadoop.apache.org>; Tue, 15 Mar 2011 20:14:29 +0000 (UTC)
Date: Tue, 15 Mar 2011 20:14:29 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <609291336.4711.1300220069809.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1594670430.14027.1299867839776.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7183) WritableComparator.get should not
 cache comparator objects
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007138#comment-13007138 ] 

Owen O'Malley commented on HADOOP-7183:
---------------------------------------

This behavior wasn't changed by HADOOP-6881, so it has been there for a long time.

> WritableComparator.get should not cache comparator objects
> ----------------------------------------------------------
>
>                 Key: HADOOP-7183
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7183
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.20.3, 0.21.1, 0.22.0
>
>         Attachments: HADOOP-7183.patch
>
>
> HADOOP-6881 modified WritableComparator.get such that the constructed WritableComparator gets saved back into the static map. This is fine for stateless comparators, but some comparators have per-instance state, and thus this becomes thread-unsafe and causes errors in the shuffle where multiple threads are doing comparisons. An example of a Comparator with per-instance state is WritableComparator itself.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13608-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 15 20:30:54 2011
Return-Path: <common-issues-return-13608-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80727 invoked from network); 15 Mar 2011 20:30:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Mar 2011 20:30:53 -0000
Received: (qmail 58620 invoked by uid 500); 15 Mar 2011 20:30:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58583 invoked by uid 500); 15 Mar 2011 20:30:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58575 invoked by uid 99); 15 Mar 2011 20:30:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 20:30:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 20:30:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C16873AAA70
	for <common-issues@hadoop.apache.org>; Tue, 15 Mar 2011 20:30:29 +0000 (UTC)
Date: Tue, 15 Mar 2011 20:30:29 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1062942637.4795.1300221029789.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1594670430.14027.1299867839776.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7183) WritableComparator.get should not
 cache comparator objects
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007153#comment-13007153 ] 

Owen O'Malley commented on HADOOP-7183:
---------------------------------------

I'm not very happy with this fix. The implicit contract is that all comparators in the cache must be thread safe. To special case the WritableComparator case because it isn't thread safe seems like the wrong direction. It seems far less brittle to use a thread local buffer for the WritableComparator.

Have you actually verified that the shuffle's sorts actually do a get per a thread?


> WritableComparator.get should not cache comparator objects
> ----------------------------------------------------------
>
>                 Key: HADOOP-7183
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7183
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.20.3, 0.21.1, 0.22.0
>
>         Attachments: HADOOP-7183.patch
>
>
> HADOOP-6881 modified WritableComparator.get such that the constructed WritableComparator gets saved back into the static map. This is fine for stateless comparators, but some comparators have per-instance state, and thus this becomes thread-unsafe and causes errors in the shuffle where multiple threads are doing comparisons. An example of a Comparator with per-instance state is WritableComparator itself.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13609-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 15 20:50:54 2011
Return-Path: <common-issues-return-13609-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64601 invoked from network); 15 Mar 2011 20:50:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Mar 2011 20:50:54 -0000
Received: (qmail 78396 invoked by uid 500); 15 Mar 2011 20:50:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78354 invoked by uid 500); 15 Mar 2011 20:50:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78324 invoked by uid 99); 15 Mar 2011 20:50:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 20:50:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 20:50:52 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DA1C33AB2DF
	for <common-issues@hadoop.apache.org>; Tue, 15 Mar 2011 20:50:31 +0000 (UTC)
Date: Tue, 15 Mar 2011 20:50:31 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1168401473.4873.1300222231889.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1594670430.14027.1299867839776.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7183) WritableComparator.get should not
 cache comparator objects
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007172#comment-13007172 ] 

Todd Lipcon commented on HADOOP-7183:
-------------------------------------

bq. This behavior wasn't changed by HADOOP-6881

It looks to me like it was. Prior to 6881, if nothing was in the cache, it did not "write back" to the cache in WritableComparator.get(). 6881 changed it to add {{comparators.put}} there.

bq. Have you actually verified that the shuffle's sorts actually do a get per a thread?

Not in the latest shuffle code, but it looked that way to me previously.


Using ThreadLocals seems like a reasonable alternate implementation. But we should still make the contract explicit that cached comparators must be threadsafe (yes, it's fairly obvious, but we seem to have missed it ourselves!)

> WritableComparator.get should not cache comparator objects
> ----------------------------------------------------------
>
>                 Key: HADOOP-7183
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7183
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.20.3, 0.21.1, 0.22.0
>
>         Attachments: HADOOP-7183.patch
>
>
> HADOOP-6881 modified WritableComparator.get such that the constructed WritableComparator gets saved back into the static map. This is fine for stateless comparators, but some comparators have per-instance state, and thus this becomes thread-unsafe and causes errors in the shuffle where multiple threads are doing comparisons. An example of a Comparator with per-instance state is WritableComparator itself.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13610-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 15 21:02:51 2011
Return-Path: <common-issues-return-13610-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88495 invoked from network); 15 Mar 2011 21:02:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Mar 2011 21:02:51 -0000
Received: (qmail 2614 invoked by uid 500); 15 Mar 2011 21:02:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2590 invoked by uid 500); 15 Mar 2011 21:02:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2580 invoked by uid 99); 15 Mar 2011 21:02:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 21:02:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 21:02:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D46DA3AB6FD
	for <common-issues@hadoop.apache.org>; Tue, 15 Mar 2011 21:02:29 +0000 (UTC)
Date: Tue, 15 Mar 2011 21:02:29 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <28524165.4929.1300222949866.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1594670430.14027.1299867839776.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7183) WritableComparator.get should not
 cache comparator objects
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007180#comment-13007180 ] 

Tom White commented on HADOOP-7183:
-----------------------------------

Before HADOOP-6881 the only way to add a WritableComparator to the cache was by explicitly calling WritableComparator.define(). HADOOP-6881 changed WritableComparator.get() to add a generic WritableComparator to the comparators cache when there was none registered for the class. This patch simply restores the previous behaviour.

Making WritableComparator thread-safe by using a thread-local buffer would be a good enhancement but isn't necessary for this fix.

> WritableComparator.get should not cache comparator objects
> ----------------------------------------------------------
>
>                 Key: HADOOP-7183
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7183
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.20.3, 0.21.1, 0.22.0
>
>         Attachments: HADOOP-7183.patch
>
>
> HADOOP-6881 modified WritableComparator.get such that the constructed WritableComparator gets saved back into the static map. This is fine for stateless comparators, but some comparators have per-instance state, and thus this becomes thread-unsafe and causes errors in the shuffle where multiple threads are doing comparisons. An example of a Comparator with per-instance state is WritableComparator itself.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13611-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 15 22:12:54 2011
Return-Path: <common-issues-return-13611-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13931 invoked from network); 15 Mar 2011 22:12:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Mar 2011 22:12:54 -0000
Received: (qmail 33539 invoked by uid 500); 15 Mar 2011 22:12:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33512 invoked by uid 500); 15 Mar 2011 22:12:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33504 invoked by uid 99); 15 Mar 2011 22:12:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 22:12:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 22:12:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9B5A93ABDE5
	for <common-issues@hadoop.apache.org>; Tue, 15 Mar 2011 22:12:29 +0000 (UTC)
Date: Tue, 15 Mar 2011 22:12:29 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <579246913.5238.1300227149632.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1594670430.14027.1299867839776.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7183) WritableComparator.get should not
 cache comparator objects
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007228#comment-13007228 ] 

Owen O'Malley commented on HADOOP-7183:
---------------------------------------

{quote}
6881 changed it to add comparators.put there.
{quote}

Ah, right. Being too efficient. *grin*

{quote}
Making WritableComparator thread-safe by using a thread-local buffer would be a good enhancement but isn't necessary for this fix.
{quote}

This fix is brittle precisely because the expectation is that comparators are stateless. It looks like the shuffle doesn't reuse comparators between threads, but it is much much easier to make them thread-safe than ensure that no one puts in such a use case.

> WritableComparator.get should not cache comparator objects
> ----------------------------------------------------------
>
>                 Key: HADOOP-7183
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7183
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.20.3, 0.21.1, 0.22.0
>
>         Attachments: HADOOP-7183.patch
>
>
> HADOOP-6881 modified WritableComparator.get such that the constructed WritableComparator gets saved back into the static map. This is fine for stateless comparators, but some comparators have per-instance state, and thus this becomes thread-unsafe and causes errors in the shuffle where multiple threads are doing comparisons. An example of a Comparator with per-instance state is WritableComparator itself.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13612-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 15 23:07:51 2011
Return-Path: <common-issues-return-13612-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5083 invoked from network); 15 Mar 2011 23:07:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Mar 2011 23:07:51 -0000
Received: (qmail 99224 invoked by uid 500); 15 Mar 2011 23:07:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99194 invoked by uid 500); 15 Mar 2011 23:07:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99186 invoked by uid 99); 15 Mar 2011 23:07:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 23:07:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 23:07:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9C3813ABEAD
	for <common-issues@hadoop.apache.org>; Tue, 15 Mar 2011 23:07:29 +0000 (UTC)
Date: Tue, 15 Mar 2011 23:07:29 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1942992605.5404.1300230449635.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1975706251.11966.1299795239567.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7180) Improve CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007263#comment-13007263 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7180:
------------------------------------------------

- Please use junit 4 (i.e. {{org.junit.Test}} and other classes {{org.junit.*}} instead of {{junit.framework.TestCase}})
- All public classes and methods (except tests) must have javadoc.
- How about passing {{minPar}}/{{maxPar}} and {{psize}} to {{NotEnoughArgumentsException}}/{{TooManyArgumentsException}} and then shows the numbers in the error messages?
- Minor: how about passing {{pos}} to {{parse(List<String> args)}}, so that we could just return {{parse(Arrays.asList(args), pos)}} in {{parse(String[] args, int pos)}}?

> Improve CommandFormat
> ---------------------
>
>                 Key: HADOOP-7180
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7180
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7180.patch
>
>
> CommandFormat currently takes an array and offset for parsing and returns a list of arguments.  It'd be much more convenient to have it process a list too.  It would also be nice to differentiate between too few and too many args instead of the generic "Illegal number of arguments".  Finally, CommandFormat is completely devoid of tests.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13613-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 15 23:07:54 2011
Return-Path: <common-issues-return-13613-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5128 invoked from network); 15 Mar 2011 23:07:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Mar 2011 23:07:53 -0000
Received: (qmail 99457 invoked by uid 500); 15 Mar 2011 23:07:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99423 invoked by uid 500); 15 Mar 2011 23:07:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99415 invoked by uid 99); 15 Mar 2011 23:07:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 23:07:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Mar 2011 23:07:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A98C33ABEB0
	for <common-issues@hadoop.apache.org>; Tue, 15 Mar 2011 23:07:29 +0000 (UTC)
Date: Tue, 15 Mar 2011 23:07:29 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1374880642.5406.1300230449688.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1975706251.11966.1299795239567.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7180) Improve CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7180:
-------------------------------------------

    Component/s: fs

> Improve CommandFormat
> ---------------------
>
>                 Key: HADOOP-7180
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7180
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7180.patch
>
>
> CommandFormat currently takes an array and offset for parsing and returns a list of arguments.  It'd be much more convenient to have it process a list too.  It would also be nice to differentiate between too few and too many args instead of the generic "Illegal number of arguments".  Finally, CommandFormat is completely devoid of tests.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13614-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 16 05:29:53 2011
Return-Path: <common-issues-return-13614-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93566 invoked from network); 16 Mar 2011 05:29:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Mar 2011 05:29:52 -0000
Received: (qmail 34118 invoked by uid 500); 16 Mar 2011 05:29:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33938 invoked by uid 500); 16 Mar 2011 05:29:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33922 invoked by uid 99); 16 Mar 2011 05:29:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Mar 2011 05:29:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Mar 2011 05:29:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 95F5B3AB996
	for <common-issues@hadoop.apache.org>; Wed, 16 Mar 2011 05:29:29 +0000 (UTC)
Date: Wed, 16 Mar 2011 05:29:29 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1055815913.6001.1300253369611.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7193) Help message is wrong for touchz
 command.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Help message is wrong for touchz command.
-----------------------------------------

                 Key: HADOOP-7193
                 URL: https://issues.apache.org/jira/browse/HADOOP-7193
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs
            Reporter: Uma Maheswara Rao G
            Priority: Minor


Help message for touchz command is
 -touchz <path>: Write a timestamp in yyyy-MM-dd HH:mm:ss format
                in a file at <path>. An error is returned if the file exists with non-zero length.

 Actually current DFS behaviour is that it will not write any time stamp in created file. Just it is creating zero size file.

So better to change the help message to give exact meaning.


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13615-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 16 09:52:52 2011
Return-Path: <common-issues-return-13615-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63117 invoked from network); 16 Mar 2011 09:52:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Mar 2011 09:52:52 -0000
Received: (qmail 55969 invoked by uid 500); 16 Mar 2011 09:52:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55903 invoked by uid 500); 16 Mar 2011 09:52:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55892 invoked by uid 99); 16 Mar 2011 09:52:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Mar 2011 09:52:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Mar 2011 09:52:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8FED43ABAB2
	for <common-issues@hadoop.apache.org>; Wed, 16 Mar 2011 09:52:29 +0000 (UTC)
Date: Wed, 16 Mar 2011 09:52:29 +0000 (UTC)
From: "Devaraj K (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <113773164.6166.1300269149586.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7194) Potential Resource leak in
 IOUtils.java
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Potential Resource leak in IOUtils.java
---------------------------------------

                 Key: HADOOP-7194
                 URL: https://issues.apache.org/jira/browse/HADOOP-7194
             Project: Hadoop Common
          Issue Type: Bug
          Components: io
    Affects Versions: 0.23.0
            Reporter: Devaraj K


{code:title=IOUtils.java|borderStyle=solid}

try {
      copyBytes(in, out, buffSize);
    } finally {
      if(close) {
        out.close();
        in.close();
      }
    }
{code} 
In the above code if any exception throws from the out.close() statement, in.close() statement will not execute and the input stream will not be closed.


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13616-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 16 20:00:52 2011
Return-Path: <common-issues-return-13616-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41309 invoked from network); 16 Mar 2011 20:00:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Mar 2011 20:00:52 -0000
Received: (qmail 32236 invoked by uid 500); 16 Mar 2011 20:00:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32215 invoked by uid 500); 16 Mar 2011 20:00:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32207 invoked by uid 99); 16 Mar 2011 20:00:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Mar 2011 20:00:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Mar 2011 20:00:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D76B53AC0FF
	for <common-issues@hadoop.apache.org>; Wed, 16 Mar 2011 20:00:29 +0000 (UTC)
Date: Wed, 16 Mar 2011 20:00:29 +0000 (UTC)
From: "Harsh J Chouraria (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1520762360.7406.1300305629879.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-1919) Add option to allow Binding Jetty
 to localhost
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007668#comment-13007668 ] 

Harsh J Chouraria commented on HADOOP-1919:
-------------------------------------------

What the OP asks for, specifically, is possible by setting the dfs.http.address to localhost:port 'stead of 0.0.0.0. I think this can be closed with that tip.

As Raghu pointed out, distcp and hftp services will get affected by this, if being called upon external to the machine -- but nothing else apparently. Perhaps /fsck too will fail if used from a machine other than the NameNode's one.

> Add option to allow Binding Jetty to localhost
> ----------------------------------------------
>
>                 Key: HADOOP-1919
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1919
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.14.0
>            Reporter: Thurman Turner
>            Priority: Minor
>
> We would like a configurable option to have Jetty bound to the loopback address of the machine so that the dfs-browser is not accessible from outside the host.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13617-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 16 20:29:54 2011
Return-Path: <common-issues-return-13617-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39200 invoked from network); 16 Mar 2011 20:29:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Mar 2011 20:29:53 -0000
Received: (qmail 89422 invoked by uid 500); 16 Mar 2011 20:29:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89382 invoked by uid 500); 16 Mar 2011 20:29:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89374 invoked by uid 99); 16 Mar 2011 20:29:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Mar 2011 20:29:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Mar 2011 20:29:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 903643ACD09
	for <common-issues@hadoop.apache.org>; Wed, 16 Mar 2011 20:29:29 +0000 (UTC)
Date: Wed, 16 Mar 2011 20:29:29 +0000 (UTC)
From: "James Seigel (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <904650912.7471.1300307369586.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1055815913.6001.1300253369611.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7193) Help message is wrong for touchz
 command.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

James Seigel updated HADOOP-7193:
---------------------------------

    Affects Version/s: 0.21.0
         Release Note: Updated the help for the touchz command.
               Status: Patch Available  (was: Open)

> Help message is wrong for touchz command.
> -----------------------------------------
>
>                 Key: HADOOP-7193
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7193
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Uma Maheswara Rao G
>            Priority: Minor
>
> Help message for touchz command is
>  -touchz <path>: Write a timestamp in yyyy-MM-dd HH:mm:ss format
>                 in a file at <path>. An error is returned if the file exists with non-zero length.
>  Actually current DFS behaviour is that it will not write any time stamp in created file. Just it is creating zero size file.
> So better to change the help message to give exact meaning.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13618-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 16 20:31:00 2011
Return-Path: <common-issues-return-13618-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45512 invoked from network); 16 Mar 2011 20:31:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Mar 2011 20:31:00 -0000
Received: (qmail 89769 invoked by uid 500); 16 Mar 2011 20:30:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89738 invoked by uid 500); 16 Mar 2011 20:30:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89730 invoked by uid 99); 16 Mar 2011 20:30:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Mar 2011 20:30:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Mar 2011 20:30:57 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EB25F3ACDB4
	for <common-issues@hadoop.apache.org>; Wed, 16 Mar 2011 20:30:35 +0000 (UTC)
Date: Wed, 16 Mar 2011 20:30:35 +0000 (UTC)
From: "James Seigel (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <847002224.7477.1300307435960.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1055815913.6001.1300253369611.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7193) Help message is wrong for touchz
 command.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

James Seigel updated HADOOP-7193:
---------------------------------

    Attachment: HADOOP-7193.patch

> Help message is wrong for touchz command.
> -----------------------------------------
>
>                 Key: HADOOP-7193
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7193
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Uma Maheswara Rao G
>            Priority: Minor
>         Attachments: HADOOP-7193.patch
>
>
> Help message for touchz command is
>  -touchz <path>: Write a timestamp in yyyy-MM-dd HH:mm:ss format
>                 in a file at <path>. An error is returned if the file exists with non-zero length.
>  Actually current DFS behaviour is that it will not write any time stamp in created file. Just it is creating zero size file.
> So better to change the help message to give exact meaning.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13619-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 16 22:20:55 2011
Return-Path: <common-issues-return-13619-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11027 invoked from network); 16 Mar 2011 22:20:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Mar 2011 22:20:54 -0000
Received: (qmail 65131 invoked by uid 500); 16 Mar 2011 22:20:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65086 invoked by uid 500); 16 Mar 2011 22:20:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65078 invoked by uid 99); 16 Mar 2011 22:20:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Mar 2011 22:20:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Mar 2011 22:20:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A6D4B3AD13D
	for <common-issues@hadoop.apache.org>; Wed, 16 Mar 2011 22:20:29 +0000 (UTC)
Date: Wed, 16 Mar 2011 22:20:29 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <319221702.7659.1300314029679.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <12359194.193971295977726697.JavaMail.jira@thor>
Subject: [jira] Commented: (HADOOP-7117) Move secondary namenode checkpoint
 configs from core-default.xml to hdfs-default.xml
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007712#comment-13007712 ] 

Hadoop QA commented on HADOOP-7117:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12469585/HADOOP-7117.r1.diff
  against trunk revision 1081598.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/310//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/310//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/310//console

This message is automatically generated.

> Move secondary namenode checkpoint configs from core-default.xml to hdfs-default.xml
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7117
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7117
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Patrick Angeles
>            Assignee: Harsh J Chouraria
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7117.r1.diff
>
>
> The following configs are in core-default.xml, but are really read by the Secondary Namenode. These should be moved to hdfs-default.xml for consistency.
> <property>
> <name>fs.checkpoint.dir</name>
> <value>${hadoop.tmp.dir}/dfs/namesecondary</value>
> <description>Determines where on the local filesystem the DFS secondary
> name node should store the temporary images to merge.
> If this is a comma-delimited list of directories then the image is
> replicated in all of the directories for redundancy.
> </description>
> </property>
> <property>
> <name>fs.checkpoint.edits.dir</name>
> <value>${fs.checkpoint.dir}</value>
> <description>Determines where on the local filesystem the DFS secondary
> name node should store the temporary edits to merge.
> If this is a comma-delimited list of directoires then teh edits is
> replicated in all of the directoires for redundancy.
> Default value is same as fs.checkpoint.dir
> </description>
> </property>
> <property>
> <name>fs.checkpoint.period</name>
> <value>3600</value>
> <description>The number of seconds between two periodic checkpoints.
> </description>
> </property>
> <property>
> <name>fs.checkpoint.size</name>
> <value>67108864</value>
> <description>The size of the current edit log (in bytes) that triggers
> a periodic checkpoint even if the fs.checkpoint.period hasn't expired.
> </description>
> </property>

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13620-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 16 22:44:54 2011
Return-Path: <common-issues-return-13620-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83047 invoked from network); 16 Mar 2011 22:44:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Mar 2011 22:44:54 -0000
Received: (qmail 87285 invoked by uid 500); 16 Mar 2011 22:44:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87240 invoked by uid 500); 16 Mar 2011 22:44:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87232 invoked by uid 99); 16 Mar 2011 22:44:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Mar 2011 22:44:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Mar 2011 22:44:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9595F3AD636
	for <common-issues@hadoop.apache.org>; Wed, 16 Mar 2011 22:44:29 +0000 (UTC)
Date: Wed, 16 Mar 2011 22:44:29 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <618389164.7685.1300315469609.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <12359194.193971295977726697.JavaMail.jira@thor>
Subject: [jira] Updated: (HADOOP-7117) Move secondary namenode checkpoint
 configs from core-default.xml to hdfs-default.xml
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7117:
-------------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.22.0
                   0.21.1
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I have committed this.  Thanks, Harsh!

Also thanks Tanping for reviewing it.

> Move secondary namenode checkpoint configs from core-default.xml to hdfs-default.xml
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7117
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7117
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Patrick Angeles
>            Assignee: Harsh J Chouraria
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HADOOP-7117.r1.diff
>
>
> The following configs are in core-default.xml, but are really read by the Secondary Namenode. These should be moved to hdfs-default.xml for consistency.
> <property>
> <name>fs.checkpoint.dir</name>
> <value>${hadoop.tmp.dir}/dfs/namesecondary</value>
> <description>Determines where on the local filesystem the DFS secondary
> name node should store the temporary images to merge.
> If this is a comma-delimited list of directories then the image is
> replicated in all of the directories for redundancy.
> </description>
> </property>
> <property>
> <name>fs.checkpoint.edits.dir</name>
> <value>${fs.checkpoint.dir}</value>
> <description>Determines where on the local filesystem the DFS secondary
> name node should store the temporary edits to merge.
> If this is a comma-delimited list of directoires then teh edits is
> replicated in all of the directoires for redundancy.
> Default value is same as fs.checkpoint.dir
> </description>
> </property>
> <property>
> <name>fs.checkpoint.period</name>
> <value>3600</value>
> <description>The number of seconds between two periodic checkpoints.
> </description>
> </property>
> <property>
> <name>fs.checkpoint.size</name>
> <value>67108864</value>
> <description>The size of the current edit log (in bytes) that triggers
> a periodic checkpoint even if the fs.checkpoint.period hasn't expired.
> </description>
> </property>

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13621-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 16 23:04:54 2011
Return-Path: <common-issues-return-13621-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72395 invoked from network); 16 Mar 2011 23:04:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Mar 2011 23:04:54 -0000
Received: (qmail 1014 invoked by uid 500); 16 Mar 2011 23:04:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 865 invoked by uid 500); 16 Mar 2011 23:04:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 805 invoked by uid 99); 16 Mar 2011 23:04:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Mar 2011 23:04:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Mar 2011 23:04:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A79203ADAF5
	for <common-issues@hadoop.apache.org>; Wed, 16 Mar 2011 23:04:29 +0000 (UTC)
Date: Wed, 16 Mar 2011 23:04:29 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1200588029.7719.1300316669683.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <12359194.193971295977726697.JavaMail.jira@thor>
Subject: [jira] Commented: (HADOOP-7117) Move secondary namenode checkpoint
 configs from core-default.xml to hdfs-default.xml
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007723#comment-13007723 ] 

Hudson commented on HADOOP-7117:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #529 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/529/])
    HADOOP-7117. Remove fs.checkpoint.* from core-default.xml and replace fs.checkpoint.* with dfs.namenode.checkpoint.* in documentations.  Contributed by Harsh J Chouraria


> Move secondary namenode checkpoint configs from core-default.xml to hdfs-default.xml
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7117
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7117
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Patrick Angeles
>            Assignee: Harsh J Chouraria
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HADOOP-7117.r1.diff
>
>
> The following configs are in core-default.xml, but are really read by the Secondary Namenode. These should be moved to hdfs-default.xml for consistency.
> <property>
> <name>fs.checkpoint.dir</name>
> <value>${hadoop.tmp.dir}/dfs/namesecondary</value>
> <description>Determines where on the local filesystem the DFS secondary
> name node should store the temporary images to merge.
> If this is a comma-delimited list of directories then the image is
> replicated in all of the directories for redundancy.
> </description>
> </property>
> <property>
> <name>fs.checkpoint.edits.dir</name>
> <value>${fs.checkpoint.dir}</value>
> <description>Determines where on the local filesystem the DFS secondary
> name node should store the temporary edits to merge.
> If this is a comma-delimited list of directoires then teh edits is
> replicated in all of the directoires for redundancy.
> Default value is same as fs.checkpoint.dir
> </description>
> </property>
> <property>
> <name>fs.checkpoint.period</name>
> <value>3600</value>
> <description>The number of seconds between two periodic checkpoints.
> </description>
> </property>
> <property>
> <name>fs.checkpoint.size</name>
> <value>67108864</value>
> <description>The size of the current edit log (in bytes) that triggers
> a periodic checkpoint even if the fs.checkpoint.period hasn't expired.
> </description>
> </property>

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13622-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 17 01:40:57 2011
Return-Path: <common-issues-return-13622-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68561 invoked from network); 17 Mar 2011 01:40:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Mar 2011 01:40:57 -0000
Received: (qmail 38162 invoked by uid 500); 17 Mar 2011 01:40:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38130 invoked by uid 500); 17 Mar 2011 01:40:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38088 invoked by uid 99); 17 Mar 2011 01:40:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 01:40:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 01:40:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8F4F03AD19C
	for <common-issues@hadoop.apache.org>; Thu, 17 Mar 2011 01:40:29 +0000 (UTC)
Date: Thu, 17 Mar 2011 01:40:29 +0000 (UTC)
From: "Kang Xiao (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2125066474.8140.1300326029583.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7195) RawLocalFileSystem.rename() should
 not try to do copy
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

RawLocalFileSystem.rename() should not try to do copy
-----------------------------------------------------

                 Key: HADOOP-7195
                 URL: https://issues.apache.org/jira/browse/HADOOP-7195
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
    Affects Versions: 0.21.0, 0.20.2
            Reporter: Kang Xiao


RawLocalFileSystem.rename() try to copy file if fails to call rename of java File. It's really confusing to do copy in a rename interface. For example rename(/a/b/c, /e/f/g) will invoke the copy if /e/f does not exist.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13623-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 17 01:46:54 2011
Return-Path: <common-issues-return-13623-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84687 invoked from network); 17 Mar 2011 01:46:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Mar 2011 01:46:54 -0000
Received: (qmail 44712 invoked by uid 500); 17 Mar 2011 01:46:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44679 invoked by uid 500); 17 Mar 2011 01:46:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44670 invoked by uid 99); 17 Mar 2011 01:46:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 01:46:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 01:46:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C0AC83AD2E5
	for <common-issues@hadoop.apache.org>; Thu, 17 Mar 2011 01:46:29 +0000 (UTC)
Date: Thu, 17 Mar 2011 01:46:29 +0000 (UTC)
From: "Kang Xiao (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <578506932.8146.1300326389785.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2125066474.8140.1300326029583.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7195) RawLocalFileSystem.rename() should
 not try to do copy
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kang Xiao updated HADOOP-7195:
------------------------------

    Attachment: HADOOP-7195.patch

attach file for a sulution.

> RawLocalFileSystem.rename() should not try to do copy
> -----------------------------------------------------
>
>                 Key: HADOOP-7195
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7195
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.2, 0.21.0
>            Reporter: Kang Xiao
>         Attachments: HADOOP-7195.patch
>
>
> RawLocalFileSystem.rename() try to copy file if fails to call rename of java File. It's really confusing to do copy in a rename interface. For example rename(/a/b/c, /e/f/g) will invoke the copy if /e/f does not exist.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13624-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 17 10:08:53 2011
Return-Path: <common-issues-return-13624-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12639 invoked from network); 17 Mar 2011 10:08:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Mar 2011 10:08:52 -0000
Received: (qmail 8387 invoked by uid 500); 17 Mar 2011 10:08:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8216 invoked by uid 500); 17 Mar 2011 10:08:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8115 invoked by uid 99); 17 Mar 2011 10:08:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 10:08:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 10:08:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A12EB3ADB27
	for <common-issues@hadoop.apache.org>; Thu, 17 Mar 2011 10:08:29 +0000 (UTC)
Date: Thu, 17 Mar 2011 10:08:29 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <689968568.8536.1300356509657.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <12359194.193971295977726697.JavaMail.jira@thor>
Subject: [jira] Commented: (HADOOP-7117) Move secondary namenode checkpoint
 configs from core-default.xml to hdfs-default.xml
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007858#comment-13007858 ] 

Hudson commented on HADOOP-7117:
--------------------------------

Integrated in Hadoop-Common-22-branch #34 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-22-branch/34/])
    HADOOP-7117. Remove fs.checkpoint.* from core-default.xml and replace fs.checkpoint.* with dfs.namenode.checkpoint.* in documentations.  Contributed by Harsh J Chouraria


> Move secondary namenode checkpoint configs from core-default.xml to hdfs-default.xml
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7117
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7117
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Patrick Angeles
>            Assignee: Harsh J Chouraria
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HADOOP-7117.r1.diff
>
>
> The following configs are in core-default.xml, but are really read by the Secondary Namenode. These should be moved to hdfs-default.xml for consistency.
> <property>
> <name>fs.checkpoint.dir</name>
> <value>${hadoop.tmp.dir}/dfs/namesecondary</value>
> <description>Determines where on the local filesystem the DFS secondary
> name node should store the temporary images to merge.
> If this is a comma-delimited list of directories then the image is
> replicated in all of the directories for redundancy.
> </description>
> </property>
> <property>
> <name>fs.checkpoint.edits.dir</name>
> <value>${fs.checkpoint.dir}</value>
> <description>Determines where on the local filesystem the DFS secondary
> name node should store the temporary edits to merge.
> If this is a comma-delimited list of directoires then teh edits is
> replicated in all of the directoires for redundancy.
> Default value is same as fs.checkpoint.dir
> </description>
> </property>
> <property>
> <name>fs.checkpoint.period</name>
> <value>3600</value>
> <description>The number of seconds between two periodic checkpoints.
> </description>
> </property>
> <property>
> <name>fs.checkpoint.size</name>
> <value>67108864</value>
> <description>The size of the current edit log (in bytes) that triggers
> a periodic checkpoint even if the fs.checkpoint.period hasn't expired.
> </description>
> </property>

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13625-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 17 10:22:54 2011
Return-Path: <common-issues-return-13625-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38440 invoked from network); 17 Mar 2011 10:22:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Mar 2011 10:22:53 -0000
Received: (qmail 18184 invoked by uid 500); 17 Mar 2011 10:22:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18095 invoked by uid 500); 17 Mar 2011 10:22:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18080 invoked by uid 99); 17 Mar 2011 10:22:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 10:22:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 10:22:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 926423ADF23
	for <common-issues@hadoop.apache.org>; Thu, 17 Mar 2011 10:22:29 +0000 (UTC)
Date: Thu, 17 Mar 2011 10:22:29 +0000 (UTC)
From: "Peter Voss (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1488021550.8544.1300357349596.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7196) CompressionCodecFactory returns
 unconfigured GZipCodec if io.compression.codecs is not set
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

CompressionCodecFactory returns unconfigured GZipCodec if io.compression.codecs is not set
------------------------------------------------------------------------------------------

                 Key: HADOOP-7196
                 URL: https://issues.apache.org/jira/browse/HADOOP-7196
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.20.2
            Reporter: Peter Voss


In case io.compression.codecs property is not set the GZipCodec is added using this code:
{code:java}
List<Class<? extends CompressionCodec>> codecClasses = getCodecClasses(conf);
if (codecClasses == null) {
  addCodec(new GzipCodec());
  addCodec(new DefaultCodec());      
} else {
  Iterator<Class<? extends CompressionCodec>> itr = codecClasses.iterator();
  while (itr.hasNext()) {
    CompressionCodec codec = ReflectionUtils.newInstance(itr.next(), conf);
    addCodec(codec);     
  }
}
{code}
which leaves GzipCodec unconfigured. If it is set via the {{io.compression.codecs}} property it gets configured properly using ReflectionUtils.newInstance(..., conf).

I have seen a lot of NPEs on systems that don't have this property set when using a LineRecordReader (that internally gets the codec from CompressionCodecFactory).

I would suggest to use {{org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec}} as default value for {{io.compression.codecs}}, instead of having another independent code path that deals with the case that this property is not set.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13626-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 17 11:13:54 2011
Return-Path: <common-issues-return-13626-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4931 invoked from network); 17 Mar 2011 11:13:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Mar 2011 11:13:54 -0000
Received: (qmail 70065 invoked by uid 500); 17 Mar 2011 11:13:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70029 invoked by uid 500); 17 Mar 2011 11:13:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70021 invoked by uid 99); 17 Mar 2011 11:13:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 11:13:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 11:13:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8DB903AD563
	for <common-issues@hadoop.apache.org>; Thu, 17 Mar 2011 11:13:29 +0000 (UTC)
Date: Thu, 17 Mar 2011 11:13:29 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <908985549.8654.1300360409576.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <12359194.193971295977726697.JavaMail.jira@thor>
Subject: [jira] Commented: (HADOOP-7117) Move secondary namenode checkpoint
 configs from core-default.xml to hdfs-default.xml
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007891#comment-13007891 ] 

Hudson commented on HADOOP-7117:
--------------------------------

Integrated in Hadoop-Common-trunk #633 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/633/])
    HADOOP-7117. Remove fs.checkpoint.* from core-default.xml and replace fs.checkpoint.* with dfs.namenode.checkpoint.* in documentations.  Contributed by Harsh J Chouraria


> Move secondary namenode checkpoint configs from core-default.xml to hdfs-default.xml
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7117
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7117
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Patrick Angeles
>            Assignee: Harsh J Chouraria
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HADOOP-7117.r1.diff
>
>
> The following configs are in core-default.xml, but are really read by the Secondary Namenode. These should be moved to hdfs-default.xml for consistency.
> <property>
> <name>fs.checkpoint.dir</name>
> <value>${hadoop.tmp.dir}/dfs/namesecondary</value>
> <description>Determines where on the local filesystem the DFS secondary
> name node should store the temporary images to merge.
> If this is a comma-delimited list of directories then the image is
> replicated in all of the directories for redundancy.
> </description>
> </property>
> <property>
> <name>fs.checkpoint.edits.dir</name>
> <value>${fs.checkpoint.dir}</value>
> <description>Determines where on the local filesystem the DFS secondary
> name node should store the temporary edits to merge.
> If this is a comma-delimited list of directoires then teh edits is
> replicated in all of the directoires for redundancy.
> Default value is same as fs.checkpoint.dir
> </description>
> </property>
> <property>
> <name>fs.checkpoint.period</name>
> <value>3600</value>
> <description>The number of seconds between two periodic checkpoints.
> </description>
> </property>
> <property>
> <name>fs.checkpoint.size</name>
> <value>67108864</value>
> <description>The size of the current edit log (in bytes) that triggers
> a periodic checkpoint even if the fs.checkpoint.period hasn't expired.
> </description>
> </property>

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13627-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 17 22:06:54 2011
Return-Path: <common-issues-return-13627-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70776 invoked from network); 17 Mar 2011 22:06:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Mar 2011 22:06:53 -0000
Received: (qmail 48298 invoked by uid 500); 17 Mar 2011 22:06:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48265 invoked by uid 500); 17 Mar 2011 22:06:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48257 invoked by uid 99); 17 Mar 2011 22:06:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 22:06:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 22:06:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 91ECE3AE7D4
	for <common-issues@hadoop.apache.org>; Thu, 17 Mar 2011 22:06:29 +0000 (UTC)
Date: Thu, 17 Mar 2011 22:06:29 +0000 (UTC)
From: "Vrachnis Ilias-Dimitrios (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1311101848.10328.1300399589594.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7197) Support for non-standard ssh port for
 slaves
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Support for non-standard ssh port for slaves
--------------------------------------------

                 Key: HADOOP-7197
                 URL: https://issues.apache.org/jira/browse/HADOOP-7197
             Project: Hadoop Common
          Issue Type: New Feature
          Components: util
    Affects Versions: 0.20.2
         Environment: FreeBSD 8.2-RELEASE
            Reporter: Vrachnis Ilias-Dimitrios
            Priority: Trivial


I was trying to add a slave that ran sshd in a non-standard port (eg. 2222 in stead of 22), when I noticed that there was no way to support another port through the configuration for a single node.

Supporting a different port for all the slaves is possible through the HADOOP_SSH_OPTS variable, but not for a single slave.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13628-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 17 22:08:54 2011
Return-Path: <common-issues-return-13628-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71384 invoked from network); 17 Mar 2011 22:08:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Mar 2011 22:08:53 -0000
Received: (qmail 50297 invoked by uid 500); 17 Mar 2011 22:08:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50160 invoked by uid 500); 17 Mar 2011 22:08:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50152 invoked by uid 99); 17 Mar 2011 22:08:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 22:08:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 22:08:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9A3EA3AE88B
	for <common-issues@hadoop.apache.org>; Thu, 17 Mar 2011 22:08:29 +0000 (UTC)
Date: Thu, 17 Mar 2011 22:08:29 +0000 (UTC)
From: "Vrachnis Ilias-Dimitrios (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <532458103.10333.1300399709628.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1311101848.10328.1300399589594.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7197) Support for non-standard ssh port for
 slaves
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vrachnis Ilias-Dimitrios updated HADOOP-7197:
---------------------------------------------

    Attachment: slaves.sh.patch

Changing the IFS variable to the new line permits the inclusion of ssh flags in the slaves file such as -p <port>

> Support for non-standard ssh port for slaves
> --------------------------------------------
>
>                 Key: HADOOP-7197
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7197
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: util
>    Affects Versions: 0.20.2
>         Environment: FreeBSD 8.2-RELEASE
>            Reporter: Vrachnis Ilias-Dimitrios
>            Priority: Trivial
>              Labels: hadoop
>         Attachments: slaves.sh.patch
>
>
> I was trying to add a slave that ran sshd in a non-standard port (eg. 2222 in stead of 22), when I noticed that there was no way to support another port through the configuration for a single node.
> Supporting a different port for all the slaves is possible through the HADOOP_SSH_OPTS variable, but not for a single slave.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13629-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 17 22:26:55 2011
Return-Path: <common-issues-return-13629-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17492 invoked from network); 17 Mar 2011 22:26:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Mar 2011 22:26:54 -0000
Received: (qmail 71216 invoked by uid 500); 17 Mar 2011 22:26:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71185 invoked by uid 500); 17 Mar 2011 22:26:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71177 invoked by uid 99); 17 Mar 2011 22:26:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 22:26:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 22:26:52 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 70AE03AEE3C
	for <common-issues@hadoop.apache.org>; Thu, 17 Mar 2011 22:26:30 +0000 (UTC)
Date: Thu, 17 Mar 2011 22:26:30 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <958744213.10366.1300400790458.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1311101848.10328.1300399589594.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7197) Support for non-standard ssh port for
 slaves
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7197:
--------------------------------

    Hadoop Flags: [Incompatible change]

> Support for non-standard ssh port for slaves
> --------------------------------------------
>
>                 Key: HADOOP-7197
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7197
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: util
>    Affects Versions: 0.20.2
>         Environment: FreeBSD 8.2-RELEASE
>            Reporter: Vrachnis Ilias-Dimitrios
>            Priority: Trivial
>              Labels: hadoop
>         Attachments: slaves.sh.patch
>
>
> I was trying to add a slave that ran sshd in a non-standard port (eg. 2222 in stead of 22), when I noticed that there was no way to support another port through the configuration for a single node.
> Supporting a different port for all the slaves is possible through the HADOOP_SSH_OPTS variable, but not for a single slave.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13630-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 17 22:49:51 2011
Return-Path: <common-issues-return-13630-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6139 invoked from network); 17 Mar 2011 22:49:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Mar 2011 22:49:51 -0000
Received: (qmail 92832 invoked by uid 500); 17 Mar 2011 22:49:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92788 invoked by uid 500); 17 Mar 2011 22:49:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92780 invoked by uid 99); 17 Mar 2011 22:49:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 22:49:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 22:49:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BA74E3AE465
	for <common-issues@hadoop.apache.org>; Thu, 17 Mar 2011 22:49:29 +0000 (UTC)
Date: Thu, 17 Mar 2011 22:49:29 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <385979528.10392.1300402169760.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1975706251.11966.1299795239567.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7180) Improve CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7180:
--------------------------------

    Attachment: HADOOP-7180-2.patch

Converted to junit 4, added what I think are the missing javadocs(?), added min/max & expected args to parameter exceptions.

I omitted the suggested change to add an index to the list version of parse.  The index is present on the old method to skip over ARGV[0] (the command).  With the forthcoming changes, the command will be consumed from the list before calling the list-based parse.  If it was added, then the sublist/erase would still be required since the list is expected to be  destructively modified.  If you feel strongly about the index, please let me know.

> Improve CommandFormat
> ---------------------
>
>                 Key: HADOOP-7180
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7180
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7180-2.patch, HADOOP-7180.patch
>
>
> CommandFormat currently takes an array and offset for parsing and returns a list of arguments.  It'd be much more convenient to have it process a list too.  It would also be nice to differentiate between too few and too many args instead of the generic "Illegal number of arguments".  Finally, CommandFormat is completely devoid of tests.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13631-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 17 23:00:53 2011
Return-Path: <common-issues-return-13631-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13594 invoked from network); 17 Mar 2011 23:00:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Mar 2011 23:00:52 -0000
Received: (qmail 6890 invoked by uid 500); 17 Mar 2011 23:00:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6839 invoked by uid 500); 17 Mar 2011 23:00:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6791 invoked by uid 99); 17 Mar 2011 23:00:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 23:00:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 23:00:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3B1D53AE803
	for <common-issues@hadoop.apache.org>; Thu, 17 Mar 2011 23:00:30 +0000 (UTC)
Date: Thu, 17 Mar 2011 23:00:30 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1408567439.10413.1300402830238.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1311101848.10328.1300399589594.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7197) Support for non-standard ssh port
 for slaves
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008206#comment-13008206 ] 

Allen Wittenauer commented on HADOOP-7197:
------------------------------------------

You can change ~/.ssh/config to point to a different port per host.

> Support for non-standard ssh port for slaves
> --------------------------------------------
>
>                 Key: HADOOP-7197
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7197
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: util
>    Affects Versions: 0.20.2
>         Environment: FreeBSD 8.2-RELEASE
>            Reporter: Vrachnis Ilias-Dimitrios
>            Priority: Trivial
>              Labels: hadoop
>         Attachments: slaves.sh.patch
>
>
> I was trying to add a slave that ran sshd in a non-standard port (eg. 2222 in stead of 22), when I noticed that there was no way to support another port through the configuration for a single node.
> Supporting a different port for all the slaves is possible through the HADOOP_SSH_OPTS variable, but not for a single slave.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13632-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 17 23:44:53 2011
Return-Path: <common-issues-return-13632-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52641 invoked from network); 17 Mar 2011 23:44:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Mar 2011 23:44:52 -0000
Received: (qmail 48208 invoked by uid 500); 17 Mar 2011 23:44:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48153 invoked by uid 500); 17 Mar 2011 23:44:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47901 invoked by uid 99); 17 Mar 2011 23:44:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 23:44:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 23:44:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B51B53AE43E
	for <common-issues@hadoop.apache.org>; Thu, 17 Mar 2011 23:44:29 +0000 (UTC)
Date: Thu, 17 Mar 2011 23:44:29 +0000 (UTC)
From: "Philip Zeyliger (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1812341271.10485.1300405469738.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7198) Hadoop defaults for web UI ports
 often fall smack in the middle of Linux ephemeral port range
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Hadoop defaults for web UI ports often fall smack in the middle of Linux ephemeral port range
---------------------------------------------------------------------------------------------

                 Key: HADOOP-7198
                 URL: https://issues.apache.org/jira/browse/HADOOP-7198
             Project: Hadoop Common
          Issue Type: Wish
            Reporter: Philip Zeyliger
            Priority: Trivial


It turns out (see http://en.wikipedia.org/wiki/Ephemeral_port and  /proc/sys/net/ipv4/ip_local_port_range) that when you bind to port 0, Linux chooses an ephemeral port.  On my default-ridden Ubuntu Maverick box and on CentOS 5.5, that range is 32768-61000.  So, when HBase binds to 60030 or when mapReduce binds to 50070, there's a small chance that you'll conflict with, say, an FTP session, or with some other Hadoop daemon that's had a listening address configured as :0.

I don't know that there's a practical resolution here, since changing the defaults seems like an ill-fated effort, but if you have any ephemeral port use, you can run into this.  We've now run into it once.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13633-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 17 23:46:56 2011
Return-Path: <common-issues-return-13633-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70017 invoked from network); 17 Mar 2011 23:46:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Mar 2011 23:46:55 -0000
Received: (qmail 51351 invoked by uid 500); 17 Mar 2011 23:46:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51304 invoked by uid 500); 17 Mar 2011 23:46:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51232 invoked by uid 99); 17 Mar 2011 23:46:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 23:46:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 23:46:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F174B3AE50B
	for <common-issues@hadoop.apache.org>; Thu, 17 Mar 2011 23:46:29 +0000 (UTC)
Date: Thu, 17 Mar 2011 23:46:29 +0000 (UTC)
From: =?utf-8?Q?Celina_d=C2=B4_=C3=81vila_Samogin_=28JIRA=29?= <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <941507774.10495.1300405589985.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-6846) Scripts for building Hadoop 0.21.0
 release
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6846?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D130=
08224#comment-13008224 ]=20

Celina d=C2=B4 =C3=81vila Samogin commented on HADOOP-6846:
-------------------------------------------------

I have followed the instructions in the README, I have applied these patche=
s in 0.22.0 common, and everything has worked fine on my notebook. I have e=
dited the tar-munge file and I made =E2=80=8B=E2=80=8Ba few modifications t=
o do it. I've installed the generated tarball for version 0.22.0, hadoop da=
emons (NameNode, SecondaryNameNode, DataNode, JobTracker, TaskTracker) have=
 worked well, and it has worked well on my notebook.

> Scripts for building Hadoop 0.21.0 release
> ------------------------------------------
>
>                 Key: HADOOP-6846
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6846
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: build
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: release-scripts.tar.gz
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13634-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 17 23:50:52 2011
Return-Path: <common-issues-return-13634-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96210 invoked from network); 17 Mar 2011 23:50:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Mar 2011 23:50:51 -0000
Received: (qmail 54190 invoked by uid 500); 17 Mar 2011 23:50:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54149 invoked by uid 500); 17 Mar 2011 23:50:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54141 invoked by uid 99); 17 Mar 2011 23:50:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 23:50:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 23:50:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AD65A3AE6D3
	for <common-issues@hadoop.apache.org>; Thu, 17 Mar 2011 23:50:29 +0000 (UTC)
Date: Thu, 17 Mar 2011 23:50:29 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <752532485.10503.1300405829707.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1812341271.10485.1300405469738.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7198) Hadoop defaults for web UI ports
 often fall smack in the middle of Linux ephemeral port range
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008226#comment-13008226 ] 

Allen Wittenauer commented on HADOOP-7198:
------------------------------------------

We often run into this problem when the local named caching daemon fires up first.  The best solution is to pretty much make sure that Hadoop comes up first or configure the other service to use a limited set of ports.

Any port that is picked is going to conflict with *something* when talking above the 1k port line.

> Hadoop defaults for web UI ports often fall smack in the middle of Linux ephemeral port range
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7198
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7198
>             Project: Hadoop Common
>          Issue Type: Wish
>            Reporter: Philip Zeyliger
>            Priority: Trivial
>
> It turns out (see http://en.wikipedia.org/wiki/Ephemeral_port and  /proc/sys/net/ipv4/ip_local_port_range) that when you bind to port 0, Linux chooses an ephemeral port.  On my default-ridden Ubuntu Maverick box and on CentOS 5.5, that range is 32768-61000.  So, when HBase binds to 60030 or when mapReduce binds to 50070, there's a small chance that you'll conflict with, say, an FTP session, or with some other Hadoop daemon that's had a listening address configured as :0.
> I don't know that there's a practical resolution here, since changing the defaults seems like an ill-fated effort, but if you have any ephemeral port use, you can run into this.  We've now run into it once.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13635-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 17 23:50:52 2011
Return-Path: <common-issues-return-13635-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96225 invoked from network); 17 Mar 2011 23:50:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Mar 2011 23:50:51 -0000
Received: (qmail 54485 invoked by uid 500); 17 Mar 2011 23:50:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54357 invoked by uid 500); 17 Mar 2011 23:50:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54210 invoked by uid 99); 17 Mar 2011 23:50:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 23:50:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 23:50:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 20F9D3AE6E1
	for <common-issues@hadoop.apache.org>; Thu, 17 Mar 2011 23:50:30 +0000 (UTC)
Date: Thu, 17 Mar 2011 23:50:30 +0000 (UTC)
From: =?utf-8?Q?Celina_d=C2=B4_=C3=81vila_Samogin_=28JIRA=29?= <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <530267878.10514.1300405830131.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-6846) Scripts for building Hadoop 0.21.0
 release
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6846?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D130=
08228#comment-13008228 ]=20

Celina d=C2=B4 =C3=81vila Samogin commented on HADOOP-6846:
-------------------------------------------------

I have applied patches from https://issues.apache.org/jira/browse/HADOOP-63=
42.

> Scripts for building Hadoop 0.21.0 release
> ------------------------------------------
>
>                 Key: HADOOP-6846
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6846
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: build
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: release-scripts.tar.gz
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13636-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 17 23:54:54 2011
Return-Path: <common-issues-return-13636-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97138 invoked from network); 17 Mar 2011 23:54:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Mar 2011 23:54:54 -0000
Received: (qmail 57093 invoked by uid 500); 17 Mar 2011 23:54:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56908 invoked by uid 500); 17 Mar 2011 23:54:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56900 invoked by uid 99); 17 Mar 2011 23:54:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 23:54:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Mar 2011 23:54:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F18703AE7A0
	for <common-issues@hadoop.apache.org>; Thu, 17 Mar 2011 23:54:29 +0000 (UTC)
Date: Thu, 17 Mar 2011 23:54:29 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <411176531.10517.1300406069986.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1975706251.11966.1299795239567.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7180) Improve CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008229#comment-13008229 ] 

Hadoop QA commented on HADOOP-7180:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12473957/HADOOP-7180-2.patch
  against trunk revision 1082329.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/311//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/311//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/311//console

This message is automatically generated.

> Improve CommandFormat
> ---------------------
>
>                 Key: HADOOP-7180
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7180
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7180-2.patch, HADOOP-7180.patch
>
>
> CommandFormat currently takes an array and offset for parsing and returns a list of arguments.  It'd be much more convenient to have it process a list too.  It would also be nice to differentiate between too few and too many args instead of the generic "Illegal number of arguments".  Finally, CommandFormat is completely devoid of tests.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13637-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 18 01:48:51 2011
Return-Path: <common-issues-return-13637-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72392 invoked from network); 18 Mar 2011 01:48:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Mar 2011 01:48:51 -0000
Received: (qmail 67447 invoked by uid 500); 18 Mar 2011 01:48:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67407 invoked by uid 500); 18 Mar 2011 01:48:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67399 invoked by uid 99); 18 Mar 2011 01:48:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Mar 2011 01:48:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Mar 2011 01:48:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 931843AE81B
	for <common-issues@hadoop.apache.org>; Fri, 18 Mar 2011 01:48:29 +0000 (UTC)
Date: Fri, 18 Mar 2011 01:48:29 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2013540870.10791.1300412909599.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1975706251.11966.1299795239567.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7180) Improve CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7180:
-------------------------------------------

        Priority: Minor  (was: Major)
    Hadoop Flags: [Reviewed]

+1 patch looks good.

> Improve CommandFormat
> ---------------------
>
>                 Key: HADOOP-7180
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7180
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7180-2.patch, HADOOP-7180.patch
>
>
> CommandFormat currently takes an array and offset for parsing and returns a list of arguments.  It'd be much more convenient to have it process a list too.  It would also be nice to differentiate between too few and too many args instead of the generic "Illegal number of arguments".  Finally, CommandFormat is completely devoid of tests.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13638-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 18 01:54:54 2011
Return-Path: <common-issues-return-13638-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86955 invoked from network); 18 Mar 2011 01:54:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Mar 2011 01:54:53 -0000
Received: (qmail 73066 invoked by uid 500); 18 Mar 2011 01:54:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73036 invoked by uid 500); 18 Mar 2011 01:54:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73028 invoked by uid 99); 18 Mar 2011 01:54:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Mar 2011 01:54:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Mar 2011 01:54:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 833C53AE9F9
	for <common-issues@hadoop.apache.org>; Fri, 18 Mar 2011 01:54:29 +0000 (UTC)
Date: Fri, 18 Mar 2011 01:54:29 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <97141997.10799.1300413269534.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1975706251.11966.1299795239567.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7180) Improve CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7180:
-------------------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I have committed this.  Thanks, Daryn!

> Improve CommandFormat
> ---------------------
>
>                 Key: HADOOP-7180
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7180
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7180-2.patch, HADOOP-7180.patch
>
>
> CommandFormat currently takes an array and offset for parsing and returns a list of arguments.  It'd be much more convenient to have it process a list too.  It would also be nice to differentiate between too few and too many args instead of the generic "Illegal number of arguments".  Finally, CommandFormat is completely devoid of tests.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13639-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 18 02:03:51 2011
Return-Path: <common-issues-return-13639-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14886 invoked from network); 18 Mar 2011 02:03:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Mar 2011 02:03:51 -0000
Received: (qmail 79444 invoked by uid 500); 18 Mar 2011 02:03:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79407 invoked by uid 500); 18 Mar 2011 02:03:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79399 invoked by uid 99); 18 Mar 2011 02:03:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Mar 2011 02:03:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Mar 2011 02:03:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 83E9D3AEBDE
	for <common-issues@hadoop.apache.org>; Fri, 18 Mar 2011 02:03:29 +0000 (UTC)
Date: Fri, 18 Mar 2011 02:03:29 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <651263553.10805.1300413809536.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1975706251.11966.1299795239567.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7180) Improve CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008281#comment-13008281 ] 

Hudson commented on HADOOP-7180:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #530 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/530/])
    Commit the missing file TestCommandFormat.java for HADOOP-7180.
HADOOP-7180. Better support on CommandFormat on the API and exceptions.  Contributed by Daryn Sharp


> Improve CommandFormat
> ---------------------
>
>                 Key: HADOOP-7180
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7180
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7180-2.patch, HADOOP-7180.patch
>
>
> CommandFormat currently takes an array and offset for parsing and returns a list of arguments.  It'd be much more convenient to have it process a list too.  It would also be nice to differentiate between too few and too many args instead of the generic "Illegal number of arguments".  Finally, CommandFormat is completely devoid of tests.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13640-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 18 10:44:52 2011
Return-Path: <common-issues-return-13640-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60489 invoked from network); 18 Mar 2011 10:44:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Mar 2011 10:44:52 -0000
Received: (qmail 99753 invoked by uid 500); 18 Mar 2011 10:44:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99708 invoked by uid 500); 18 Mar 2011 10:44:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99554 invoked by uid 99); 18 Mar 2011 10:44:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Mar 2011 10:44:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Mar 2011 10:44:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9CB103AFAE1
	for <common-issues@hadoop.apache.org>; Fri, 18 Mar 2011 10:44:29 +0000 (UTC)
Date: Fri, 18 Mar 2011 10:44:29 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1154895918.11418.1300445069638.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1055815913.6001.1300253369611.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7193) Help message is wrong for touchz
 command.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uma Maheswara Rao G updated HADOOP-7193:
----------------------------------------

    Attachment: HADOOP-7193.patch

> Help message is wrong for touchz command.
> -----------------------------------------
>
>                 Key: HADOOP-7193
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7193
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Uma Maheswara Rao G
>            Priority: Minor
>         Attachments: HADOOP-7193.patch, HADOOP-7193.patch
>
>
> Help message for touchz command is
>  -touchz <path>: Write a timestamp in yyyy-MM-dd HH:mm:ss format
>                 in a file at <path>. An error is returned if the file exists with non-zero length.
>  Actually current DFS behaviour is that it will not write any time stamp in created file. Just it is creating zero size file.
> So better to change the help message to give exact meaning.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13641-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 18 11:13:51 2011
Return-Path: <common-issues-return-13641-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61346 invoked from network); 18 Mar 2011 11:13:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Mar 2011 11:13:51 -0000
Received: (qmail 31123 invoked by uid 500); 18 Mar 2011 11:13:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31099 invoked by uid 500); 18 Mar 2011 11:13:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31091 invoked by uid 99); 18 Mar 2011 11:13:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Mar 2011 11:13:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Mar 2011 11:13:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 99A253AF3BB
	for <common-issues@hadoop.apache.org>; Fri, 18 Mar 2011 11:13:29 +0000 (UTC)
Date: Fri, 18 Mar 2011 11:13:29 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <634622048.11479.1300446809626.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1975706251.11966.1299795239567.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7180) Improve CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008398#comment-13008398 ] 

Hudson commented on HADOOP-7180:
--------------------------------

Integrated in Hadoop-Common-trunk #634 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/634/])
    Commit the missing file TestCommandFormat.java for HADOOP-7180.
HADOOP-7180. Better support on CommandFormat on the API and exceptions.  Contributed by Daryn Sharp


> Improve CommandFormat
> ---------------------
>
>                 Key: HADOOP-7180
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7180
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7180-2.patch, HADOOP-7180.patch
>
>
> CommandFormat currently takes an array and offset for parsing and returns a list of arguments.  It'd be much more convenient to have it process a list too.  It would also be nice to differentiate between too few and too many args instead of the generic "Illegal number of arguments".  Finally, CommandFormat is completely devoid of tests.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13642-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 18 11:28:52 2011
Return-Path: <common-issues-return-13642-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6677 invoked from network); 18 Mar 2011 11:28:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Mar 2011 11:28:52 -0000
Received: (qmail 44961 invoked by uid 500); 18 Mar 2011 11:28:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44918 invoked by uid 500); 18 Mar 2011 11:28:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44909 invoked by uid 99); 18 Mar 2011 11:28:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Mar 2011 11:28:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Mar 2011 11:28:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AC62F3AF869
	for <common-issues@hadoop.apache.org>; Fri, 18 Mar 2011 11:28:29 +0000 (UTC)
Date: Fri, 18 Mar 2011 11:28:29 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2057547886.11490.1300447709702.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1055815913.6001.1300253369611.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7193) Help message is wrong for touchz
 command.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008400#comment-13008400 ] 

Uma Maheswara Rao G commented on HADOOP-7193:
---------------------------------------------

Thanks James,
Small change in message, HDFS will not write any time stamp in file. Just it will create zero size files and modified/created time as current time.
Updated the appropriate message.

> Help message is wrong for touchz command.
> -----------------------------------------
>
>                 Key: HADOOP-7193
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7193
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Uma Maheswara Rao G
>            Priority: Minor
>         Attachments: HADOOP-7193.patch, HADOOP-7193.patch
>
>
> Help message for touchz command is
>  -touchz <path>: Write a timestamp in yyyy-MM-dd HH:mm:ss format
>                 in a file at <path>. An error is returned if the file exists with non-zero length.
>  Actually current DFS behaviour is that it will not write any time stamp in created file. Just it is creating zero size file.
> So better to change the help message to give exact meaning.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13643-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 18 17:50:56 2011
Return-Path: <common-issues-return-13643-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84434 invoked from network); 18 Mar 2011 17:50:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Mar 2011 17:50:56 -0000
Received: (qmail 36284 invoked by uid 500); 18 Mar 2011 17:50:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36263 invoked by uid 500); 18 Mar 2011 17:50:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36255 invoked by uid 99); 18 Mar 2011 17:50:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Mar 2011 17:50:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Mar 2011 17:50:52 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EC3BC3B094B
	for <common-issues@hadoop.apache.org>; Fri, 18 Mar 2011 17:50:30 +0000 (UTC)
Date: Fri, 18 Mar 2011 17:50:30 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1630998659.12221.1300470630964.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Moved: (HADOOP-7199) fs -put crash that depends on source
 file name
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE moved HDFS-1768 to HADOOP-7199:
------------------------------------------------------

          Component/s:     (was: hdfs client)
                           (was: name-node)
                       fs
    Affects Version/s:     (was: 0.20.2)
                       0.20.2
                  Key: HADOOP-7199  (was: HDFS-1768)
              Project: Hadoop Common  (was: Hadoop HDFS)

> fs -put crash that depends on source file name
> ----------------------------------------------
>
>                 Key: HADOOP-7199
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7199
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.2
>         Environment: Cloudera CDH3B4 in pseudo mode on a Linux 2.6.32-28-generic #55-Ubuntu SMP x86_64 kernel, with Java HotSpot64-Bit Server VM (build 19.1-b02, mixed mode)
>            Reporter: Lars Ailo Bongo
>            Priority: Minor
>
> I have a unit test that includes writing a file to HDFS using copyFromLocalFile. Sometimes the function fails due to a checksum error. Once the issue has occurred "hadoop -put <filename> <anywhere>" also fails as long as the filename is the same as used in the unit test. The error is due to the file content never being sent to the DataNode, hence the file is size zero. 
> The error is not due to the file content. The error does not depend on the HDFS destination name. Restarting the NameNode and DataNode does not resolve the issue. I have not been able to reproduce the error with a simple program. I have also not tested the issue in distributed or standalone mode.
> The only "fix" is to change the source filename.
> Below is error and the NameNode log. There is no entry for this operation in the DataNode log.
> /home/larsab/troilkatt2/test-tmp/data>hadoop fs -put status-test.txt status-test.txt3
> 11/03/18 16:59:54 INFO fs.FSInputChecker: Found checksum error: b[512, 968]=3a646f6e650a323a7365636f6e6453746167653a73746172740a323a7365636f6e6453746167653a646f6e650a323a746869726453746167653a73746172740a323a746869726453746167653a646f6e650a323a74686553696e6b3a73746172740a323a74686553696e6b3a646f6e650a323a54726f696c6b6174743a646f6e650a333a54726f696c6b6174743a73746172740a333a746865536f757263653a73746172740a333a746865536f757263653a646f6e650a333a666972737453746167653a73746172740a333a666972737453746167653a646f6e650a333a7365636f6e6453746167653a73746172740a333a7365636f6e6453746167653a646f6e650a333a746869726453746167653a73746172740a333a746869726453746167653a646f6e650a333a74686553696e6b3a73746172740a333a74686553696e6b3a646f6e650a333a54726f696c6b6174743a646f6e650a343a54726f696c6b6174743a73746172740a343a746865536f757263653a73746172740a343a746865536f757263653a646f6e650a343a666972737453746167653a73746172740a343a666972737453746167653a646f6e650a343a7365636f6e6453746167653a7265636f7665720a
> org.apache.hadoop.fs.ChecksumException: Checksum error: status-test.txt at 512
> 	at org.apache.hadoop.fs.FSInputChecker.verifySum(FSInputChecker.java:277)
> 	at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:241)
> 	at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
> 	at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> 	at java.io.DataInputStream.read(DataInputStream.java:83)
> 	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:49)
> 	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:87)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:224)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:170)
> 	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1283)
> 	at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:134)
> 	at org.apache.hadoop.fs.FsShell.run(FsShell.java:1817)
> 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> 	at org.apache.hadoop.fs.FsShell.main(FsShell.java:1960)
> put: Checksum error: status-test.txt at 512
> NAMENODE
> 2011-03-18 16:59:54,422 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions: 13 Total time for transactions(ms): 1Number of transactions batched in Syncs: 0 Number of syncs: 7 SyncTimes(ms): 220 
> 2011-03-18 16:59:54,444 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=larsab	ip=/127.0.0.1	cmd=create	src=/user/larsab/status-test.txt3	dst=null	perm=larsab:supergroup:rw-r--r--
> 2011-03-18 16:59:54,469 INFO org.apache.hadoop.hdfs.StateChange: Removing lease on  file /user/larsab/status-test.txt3 from client DFSClient_-1004170418
> 2011-03-18 16:59:54,469 INFO org.apache.hadoop.hdfs.StateChange: DIR* NameSystem.completeFile: file /user/larsab/status-test.txt3 is closed by DFSClient_-1004170418

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13644-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 18 18:04:55 2011
Return-Path: <common-issues-return-13644-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30690 invoked from network); 18 Mar 2011 18:04:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Mar 2011 18:04:55 -0000
Received: (qmail 60342 invoked by uid 500); 18 Mar 2011 18:04:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60305 invoked by uid 500); 18 Mar 2011 18:04:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60297 invoked by uid 99); 18 Mar 2011 18:04:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Mar 2011 18:04:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Mar 2011 18:04:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CF4303B0E12
	for <common-issues@hadoop.apache.org>; Fri, 18 Mar 2011 18:04:29 +0000 (UTC)
Date: Fri, 18 Mar 2011 18:04:29 +0000 (UTC)
From: "Lars Ailo Bongo (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1694939217.12276.1300471469845.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7199) Checksum file check
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Lars Ailo Bongo updated HADOOP-7199:
------------------------------------

    Description: 
copyFromLocalFile crashes if a cheksum file exists on the local filesystem and the checksum does not match the file content. This will for example crash "hadoop -fs put ./foo ./foo" with a non-descriptive error.

It is therefore not possible to do:

1. copyToLocalFile(hdfsFile, localFile)       // creates checksum file
2. modify localFile
3. copyFromLocalFile(localFile, hdfsFile)  // uses old checksum

Solution: do not reuse checksum files, or add a parameter  to copyFromLocalFile that specifies that checksum files should not be reused.

  was:
I have a unit test that includes writing a file to HDFS using copyFromLocalFile. Sometimes the function fails due to a checksum error. Once the issue has occurred "hadoop -put <filename> <anywhere>" also fails as long as the filename is the same as used in the unit test. The error is due to the file content never being sent to the DataNode, hence the file is size zero. 

The error is not due to the file content. The error does not depend on the HDFS destination name. Restarting the NameNode and DataNode does not resolve the issue. I have not been able to reproduce the error with a simple program. I have also not tested the issue in distributed or standalone mode.

The only "fix" is to change the source filename.

Below is error and the NameNode log. There is no entry for this operation in the DataNode log.

/home/larsab/troilkatt2/test-tmp/data>hadoop fs -put status-test.txt status-test.txt3
11/03/18 16:59:54 INFO fs.FSInputChecker: Found checksum error: b[512, 968]=3a646f6e650a323a7365636f6e6453746167653a73746172740a323a7365636f6e6453746167653a646f6e650a323a746869726453746167653a73746172740a323a746869726453746167653a646f6e650a323a74686553696e6b3a73746172740a323a74686553696e6b3a646f6e650a323a54726f696c6b6174743a646f6e650a333a54726f696c6b6174743a73746172740a333a746865536f757263653a73746172740a333a746865536f757263653a646f6e650a333a666972737453746167653a73746172740a333a666972737453746167653a646f6e650a333a7365636f6e6453746167653a73746172740a333a7365636f6e6453746167653a646f6e650a333a746869726453746167653a73746172740a333a746869726453746167653a646f6e650a333a74686553696e6b3a73746172740a333a74686553696e6b3a646f6e650a333a54726f696c6b6174743a646f6e650a343a54726f696c6b6174743a73746172740a343a746865536f757263653a73746172740a343a746865536f757263653a646f6e650a343a666972737453746167653a73746172740a343a666972737453746167653a646f6e650a343a7365636f6e6453746167653a7265636f7665720a
org.apache.hadoop.fs.ChecksumException: Checksum error: status-test.txt at 512
	at org.apache.hadoop.fs.FSInputChecker.verifySum(FSInputChecker.java:277)
	at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:241)
	at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:189)
	at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
	at java.io.DataInputStream.read(DataInputStream.java:83)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:49)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:87)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:224)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:170)
	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1283)
	at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:134)
	at org.apache.hadoop.fs.FsShell.run(FsShell.java:1817)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
	at org.apache.hadoop.fs.FsShell.main(FsShell.java:1960)
put: Checksum error: status-test.txt at 512

NAMENODE
2011-03-18 16:59:54,422 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions: 13 Total time for transactions(ms): 1Number of transactions batched in Syncs: 0 Number of syncs: 7 SyncTimes(ms): 220 
2011-03-18 16:59:54,444 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=larsab	ip=/127.0.0.1	cmd=create	src=/user/larsab/status-test.txt3	dst=null	perm=larsab:supergroup:rw-r--r--
2011-03-18 16:59:54,469 INFO org.apache.hadoop.hdfs.StateChange: Removing lease on  file /user/larsab/status-test.txt3 from client DFSClient_-1004170418
2011-03-18 16:59:54,469 INFO org.apache.hadoop.hdfs.StateChange: DIR* NameSystem.completeFile: file /user/larsab/status-test.txt3 is closed by DFSClient_-1004170418

        Summary: Checksum file check   (was: fs -put crash that depends on source file name)

> Checksum file check 
> --------------------
>
>                 Key: HADOOP-7199
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7199
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.2
>         Environment: Cloudera CDH3B4 in pseudo mode on a Linux 2.6.32-28-generic #55-Ubuntu SMP x86_64 kernel, with Java HotSpot64-Bit Server VM (build 19.1-b02, mixed mode)
>            Reporter: Lars Ailo Bongo
>            Priority: Minor
>
> copyFromLocalFile crashes if a cheksum file exists on the local filesystem and the checksum does not match the file content. This will for example crash "hadoop -fs put ./foo ./foo" with a non-descriptive error.
> It is therefore not possible to do:
> 1. copyToLocalFile(hdfsFile, localFile)       // creates checksum file
> 2. modify localFile
> 3. copyFromLocalFile(localFile, hdfsFile)  // uses old checksum
> Solution: do not reuse checksum files, or add a parameter  to copyFromLocalFile that specifies that checksum files should not be reused.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13645-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 18 18:06:54 2011
Return-Path: <common-issues-return-13645-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39809 invoked from network); 18 Mar 2011 18:06:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Mar 2011 18:06:54 -0000
Received: (qmail 61085 invoked by uid 500); 18 Mar 2011 18:06:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61032 invoked by uid 500); 18 Mar 2011 18:06:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61010 invoked by uid 99); 18 Mar 2011 18:06:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Mar 2011 18:06:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Mar 2011 18:06:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D01883B0EF5
	for <common-issues@hadoop.apache.org>; Fri, 18 Mar 2011 18:06:29 +0000 (UTC)
Date: Fri, 18 Mar 2011 18:06:29 +0000 (UTC)
From: "Lars Ailo Bongo (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <80917034.12287.1300471589848.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7199) Crash due to reuse of checksum files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Lars Ailo Bongo updated HADOOP-7199:
------------------------------------

    Summary: Crash due to reuse of checksum files  (was: Checksum file check )

> Crash due to reuse of checksum files
> ------------------------------------
>
>                 Key: HADOOP-7199
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7199
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.2
>         Environment: Cloudera CDH3B4 in pseudo mode on a Linux 2.6.32-28-generic #55-Ubuntu SMP x86_64 kernel, with Java HotSpot64-Bit Server VM (build 19.1-b02, mixed mode)
>            Reporter: Lars Ailo Bongo
>            Priority: Minor
>
> copyFromLocalFile crashes if a cheksum file exists on the local filesystem and the checksum does not match the file content. This will for example crash "hadoop -fs put ./foo ./foo" with a non-descriptive error.
> It is therefore not possible to do:
> 1. copyToLocalFile(hdfsFile, localFile)       // creates checksum file
> 2. modify localFile
> 3. copyFromLocalFile(localFile, hdfsFile)  // uses old checksum
> Solution: do not reuse checksum files, or add a parameter  to copyFromLocalFile that specifies that checksum files should not be reused.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13646-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 18 18:08:57 2011
Return-Path: <common-issues-return-13646-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40310 invoked from network); 18 Mar 2011 18:08:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Mar 2011 18:08:55 -0000
Received: (qmail 66276 invoked by uid 500); 18 Mar 2011 18:08:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66162 invoked by uid 500); 18 Mar 2011 18:08:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66146 invoked by uid 99); 18 Mar 2011 18:08:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Mar 2011 18:08:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Mar 2011 18:08:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C7CEE3B009B
	for <common-issues@hadoop.apache.org>; Fri, 18 Mar 2011 18:08:29 +0000 (UTC)
Date: Fri, 18 Mar 2011 18:08:29 +0000 (UTC)
From: "Lars Ailo Bongo (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1508388708.12292.1300471709814.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7199) Crash due to reuse of checksum
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008553#comment-13008553 ] 

Lars Ailo Bongo commented on HADOOP-7199:
-----------------------------------------

Title and description have been updated.

> Crash due to reuse of checksum files
> ------------------------------------
>
>                 Key: HADOOP-7199
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7199
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.2
>         Environment: Cloudera CDH3B4 in pseudo mode on a Linux 2.6.32-28-generic #55-Ubuntu SMP x86_64 kernel, with Java HotSpot64-Bit Server VM (build 19.1-b02, mixed mode)
>            Reporter: Lars Ailo Bongo
>            Priority: Minor
>
> copyFromLocalFile crashes if a cheksum file exists on the local filesystem and the checksum does not match the file content. This will for example crash "hadoop -fs put ./foo ./foo" with a non-descriptive error.
> It is therefore not possible to do:
> 1. copyToLocalFile(hdfsFile, localFile)       // creates checksum file
> 2. modify localFile
> 3. copyFromLocalFile(localFile, hdfsFile)  // uses old checksum
> Solution: do not reuse checksum files, or add a parameter  to copyFromLocalFile that specifies that checksum files should not be reused.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13647-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 18 19:42:51 2011
Return-Path: <common-issues-return-13647-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31053 invoked from network); 18 Mar 2011 19:42:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Mar 2011 19:42:51 -0000
Received: (qmail 73774 invoked by uid 500); 18 Mar 2011 19:42:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73748 invoked by uid 500); 18 Mar 2011 19:42:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73740 invoked by uid 99); 18 Mar 2011 19:42:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Mar 2011 19:42:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Mar 2011 19:42:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8BC793B0E75
	for <common-issues@hadoop.apache.org>; Fri, 18 Mar 2011 19:42:29 +0000 (UTC)
Date: Fri, 18 Mar 2011 19:42:29 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1769482377.12548.1300477349569.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1070422412.13998.1299867359451.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7182) remove stub implementation of
 HardLink from FileUtil, after fixing any remaining dependencies
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated HADOOP-7182:
-------------------------------

    Attachment: HADOOP-7182-cleanup.patch

Removes stub implementation per spec.

> remove stub implementation of HardLink from FileUtil, after fixing any remaining dependencies
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7182
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7182
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.23.0
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7182-cleanup.patch
>
>
> This is the third part of a refactoring of HardLink:
> 1. HADOOP-7133 moved HardLink from an inner class of FileUtil to a stand-alone class, but left a stub class in FileUtil to prevent breaking clients in HDFS.
> 2. HDFS-1445 changes the clients in HDFS to point at the new implementation.
> 3. After HDFS-1445 is committed, this ticket is to clean up the code by removing the backward-compatibility stub.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13648-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 18 19:44:51 2011
Return-Path: <common-issues-return-13648-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32346 invoked from network); 18 Mar 2011 19:44:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Mar 2011 19:44:51 -0000
Received: (qmail 76080 invoked by uid 500); 18 Mar 2011 19:44:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76044 invoked by uid 500); 18 Mar 2011 19:44:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76035 invoked by uid 99); 18 Mar 2011 19:44:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Mar 2011 19:44:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Mar 2011 19:44:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8A5A93B0F09
	for <common-issues@hadoop.apache.org>; Fri, 18 Mar 2011 19:44:29 +0000 (UTC)
Date: Fri, 18 Mar 2011 19:44:29 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1514128598.12551.1300477469563.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1070422412.13998.1299867359451.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7182) remove stub implementation of
 HardLink from FileUtil, after fixing any remaining dependencies
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated HADOOP-7182:
-------------------------------

    Status: Patch Available  (was: Open)

> remove stub implementation of HardLink from FileUtil, after fixing any remaining dependencies
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7182
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7182
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.23.0
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7182-cleanup.patch
>
>
> This is the third part of a refactoring of HardLink:
> 1. HADOOP-7133 moved HardLink from an inner class of FileUtil to a stand-alone class, but left a stub class in FileUtil to prevent breaking clients in HDFS.
> 2. HDFS-1445 changes the clients in HDFS to point at the new implementation.
> 3. After HDFS-1445 is committed, this ticket is to clean up the code by removing the backward-compatibility stub.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13649-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 18 21:49:54 2011
Return-Path: <common-issues-return-13649-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62659 invoked from network); 18 Mar 2011 21:49:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Mar 2011 21:49:53 -0000
Received: (qmail 17857 invoked by uid 500); 18 Mar 2011 21:49:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17815 invoked by uid 500); 18 Mar 2011 21:49:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17807 invoked by uid 99); 18 Mar 2011 21:49:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Mar 2011 21:49:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Mar 2011 21:49:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9CAAE3B0383
	for <common-issues@hadoop.apache.org>; Fri, 18 Mar 2011 21:49:29 +0000 (UTC)
Date: Fri, 18 Mar 2011 21:49:29 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <181890699.12828.1300484969638.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7200) Hudson should cover fault inject
 compilation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Hudson should cover fault inject compilation 
---------------------------------------------

                 Key: HADOOP-7200
                 URL: https://issues.apache.org/jira/browse/HADOOP-7200
             Project: Hadoop Common
          Issue Type: Task
            Reporter: Eli Collins


Currently Hudson (and test-patch) do not catch changes that create compilation errors in the fault injection tests. We should fix that.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13650-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 19 00:20:52 2011
Return-Path: <common-issues-return-13650-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16669 invoked from network); 19 Mar 2011 00:20:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Mar 2011 00:20:51 -0000
Received: (qmail 48012 invoked by uid 500); 19 Mar 2011 00:20:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47980 invoked by uid 500); 19 Mar 2011 00:20:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47972 invoked by uid 99); 19 Mar 2011 00:20:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 19 Mar 2011 00:20:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 19 Mar 2011 00:20:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9E87F3B0863
	for <common-issues@hadoop.apache.org>; Sat, 19 Mar 2011 00:20:29 +0000 (UTC)
Date: Sat, 19 Mar 2011 00:20:29 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1250121198.13206.1300494029646.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-6949) Reduces RPC packet size for primitive
 arrays, especially long[], which is used at block reporting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated HADOOP-6949:
-------------------------------

    Attachment: arrayprim_v7.patch

# ArrayPrimitiveWriable

    * Class javadoc is not very clear - not sure what you mean by boxed type not supported by ObjectWritable and why that should be mentioned in the javadoc. I am not sure also how clients can provide external locking; more appropriate would be to say, pass a copy of the array, if concurrent modifications are expected.

Javadoc comments simplified.

    * Comments related to declaredComponentType is not clear. Can you please add description on what componentType and declaredComponentType are and how they are used, bef adding further description.

Provided.

    * Is PrimitiveArrayWritable a better name?

By common usage, "PrimitiveArrayWritable" would imply a sub-class of ArrayWritable, which this isn't.  So I thought it best to use a different word order.


    * Add @InterfaceAudience.Public and @InterfaceStability.Stable

Done.

    * Throw HadoopIllegalArgumentException instead of NullPointerException and IllegalArgumentException

Done.

    * Optional: The read write methods could be made static methods in WritableUtils class, so it can be used by others. Also PrimitiveType map, isCanditdatePrimitive() could also move to helper.

After discussion, we agreed there is no benefit in efficiency or code clarity, so no change in this regard.

    * Rename isCandidatePrimitive() to isPrimitive()?

Agreed. Originally, isCandidatePrimitive() provided a layer of indirection so that ArrayPrimitiveWritable didn't have to contract to support ALL primitive types.  However, since it ended up supporting them all, the distinction is irrelevant.
Changed to use Class.isPrimitive().


    * Is constructor with componentType as argument used any where?

No, it is provided to make ArrayPrimitiveWritable more generally useful.


    * Some methods could be private - for example - getDeclaredComponentType(), isDeclaredComponentType(),

No, if the user elects to use the typechecking capabilities of declaredComponentType, he should be able to make both of these queries.


    * Can you use Text instead of UTF8? This avoids deprecation warnings. You could also use WritableUtils to write and read string. This can be done in many other places in the patch.

I do not want to change the usage from that of ObjectWritable.  Essentially all our Writable classes use UTF8.  If we want to change that, please open a different Jira.


    * I prefer MalformedInputException instead of ProtocolException (see Text.java for example)

I agree ProtocolException isn't exactly right.  
Problem with MalformedInputException: it doesn't take a "message" argument, only an integer, so we can't use that.
Changed to just use IOException, since that's what most of the Writable code uses.

==========

# ObjectWritable.java

    * Can you call set() method while setting instance and declaredClass in #writeObject() method?

My understanding is that ObjectWritable.writeObject() is a static method that does not instantiate an ObjectWritable instance.  The variables "instance" and "declaredClass" are both local variables in #writeObject() context.  So calling the non-static "set()" would not be okay.

    * Not sure about the comment: // This case must come after the "isArray()" case

You're right, it's not order-dependent since it uses /if/else if/else/ rather than a sequence of "if"s.
Removed the comment.

    * Not sure why this condition is required in #writeObject() - instance.getClass().getName().equals(declaredClass.getName())

I'm being paranoid.  There can be a range of behaviors if declaredClass isn't exactly the same as instance.getClass(), so I'm refusing to allowCompactArrays if the caller isn't declaring the true type.

==========

# TestArrayPrimitiveWritable

    * In the class javadoc can you add link to PrimitiveArrayWritable?

Done.  Good catch.

    * Can you make in, out, bigSet, resultSet and expectedSet final?

Sure.

    * Should we prefer reading and writing objects using readFields and writeFields - since they are the methods supporting Writable contract?

For "normal" i/o I do use readFields() and write().  Then I test to see that the wire format is as expected, by going around those methods and directly invoking lower-level functionality.  I think this is the only way to get confidence in the wire format encoders and decoders.


> Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6949
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6949
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Navis
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: ObjectWritable.diff, arrayprim.patch, arrayprim_v4.patch, arrayprim_v5.patch, arrayprim_v6.patch, arrayprim_v7.patch
>
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n
> It would not be needed to specify each element types for primitive arrays.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13651-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 19 03:16:51 2011
Return-Path: <common-issues-return-13651-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75618 invoked from network); 19 Mar 2011 03:16:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Mar 2011 03:16:51 -0000
Received: (qmail 60425 invoked by uid 500); 19 Mar 2011 03:16:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60264 invoked by uid 500); 19 Mar 2011 03:16:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60256 invoked by uid 99); 19 Mar 2011 03:16:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 19 Mar 2011 03:16:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 19 Mar 2011 03:16:44 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 91AE53B027B
	for <common-issues@hadoop.apache.org>; Sat, 19 Mar 2011 03:16:22 +0000 (UTC)
Date: Sat, 19 Mar 2011 03:16:22 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1069306489.13356.1300504582593.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <26581586.62081293647746870.JavaMail.jira@thor>
Subject: [jira] Commented: (HADOOP-7080) Documentation dfs.block.size where
 dfs.blocksize has to be used instead.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008720#comment-13008720 ] 

Eli Collins commented on HADOOP-7080:
-------------------------------------

A potential issue here is that the dfs.block.size deprecation is done in HdfsConfiguration, so if somone is using just using the Configuration object the deprecated config won't work.  

> Documentation dfs.block.size where dfs.blocksize has to be used instead.
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-7080
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7080
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>         Environment: Linux (Ubuntu)
>            Reporter: Karl Heinz Marbaise
>            Priority: Minor
>
> The configuration for the block size is named *dfs.block.size* in the [documentation for Release 0.21.0 of Hadoop|http://hadoop.apache.org/common/docs/r0.21.0/cluster_setup.html] but this will not work nor will it be recognized. If i use [*dfs.blocksize*|http://hadoop.apache.org/hdfs/docs/current/api/constant-values.html] instead it will work perfectly.  So this seemed to be a documentation bug.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13652-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 19 21:42:52 2011
Return-Path: <common-issues-return-13652-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72842 invoked from network); 19 Mar 2011 21:42:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Mar 2011 21:42:52 -0000
Received: (qmail 14589 invoked by uid 500); 19 Mar 2011 21:42:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14563 invoked by uid 500); 19 Mar 2011 21:42:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14555 invoked by uid 99); 19 Mar 2011 21:42:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 19 Mar 2011 21:42:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 19 Mar 2011 21:42:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8F3243B196E
	for <common-issues@hadoop.apache.org>; Sat, 19 Mar 2011 21:42:29 +0000 (UTC)
Date: Sat, 19 Mar 2011 21:42:29 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1138256086.13880.1300570949583.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1799619354.3818.1299567419614.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HADOOP-7172) SecureIO should not check owner on
 non-secure clusters that have no native support
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008833#comment-13008833 ] 

Hadoop QA commented on HADOOP-7172:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12473024/hadoop-7172.txt
  against trunk revision 1082788.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/312//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/312//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/312//console

This message is automatically generated.

> SecureIO should not check owner on non-secure clusters that have no native support
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-7172
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7172
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io, security
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7172.txt, hadoop-7172.txt, hadoop-7172.txt
>
>
> The SecureIOUtils.openForRead function currently uses a racy stat/open combo if security is disabled and the native libraries are not available. This ends up shelling out to "ls -ld" which is very very slow. We've seen this cause significant performance regressions on clusters that match this profile.
> Since the racy permissions check doesn't buy us any security anyway, we should just fall back to a normal "open" without any stat() at all, if we can't use the native support to do it efficiently.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13653-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Mar 20 00:52:54 2011
Return-Path: <common-issues-return-13653-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88658 invoked from network); 20 Mar 2011 00:52:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Mar 2011 00:52:54 -0000
Received: (qmail 13656 invoked by uid 500); 20 Mar 2011 00:52:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13617 invoked by uid 500); 20 Mar 2011 00:52:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13609 invoked by uid 99); 20 Mar 2011 00:52:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 Mar 2011 00:52:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 Mar 2011 00:52:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9989E4003FE
	for <common-issues@hadoop.apache.org>; Sun, 20 Mar 2011 00:52:29 +0000 (UTC)
Date: Sun, 20 Mar 2011 00:52:29 +0000 (UTC)
From: "Jonathan Ellis (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1375189809.14004.1300582349625.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Created: (HADOOP-7201) Add Cassandra to list of
 Hadoop-related projects at Apache
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Add Cassandra to list of Hadoop-related projects at Apache
----------------------------------------------------------

                 Key: HADOOP-7201
                 URL: https://issues.apache.org/jira/browse/HADOOP-7201
             Project: Hadoop Common
          Issue Type: Improvement
          Components: documentation
            Reporter: Jonathan Ellis
            Priority: Minor
             Fix For: site


Cassandra provides native read/write support for java Hadoop jobs (with locality), Streaming, Pig, and Hive.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13654-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Mar 20 00:54:54 2011
Return-Path: <common-issues-return-13654-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89256 invoked from network); 20 Mar 2011 00:54:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Mar 2011 00:54:54 -0000
Received: (qmail 15900 invoked by uid 500); 20 Mar 2011 00:54:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15871 invoked by uid 500); 20 Mar 2011 00:54:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15857 invoked by uid 99); 20 Mar 2011 00:54:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 Mar 2011 00:54:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 Mar 2011 00:54:51 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A5DC140046D
	for <common-issues@hadoop.apache.org>; Sun, 20 Mar 2011 00:54:29 +0000 (UTC)
Date: Sun, 20 Mar 2011 00:54:29 +0000 (UTC)
From: "Jonathan Ellis (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1953322961.14009.1300582469676.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1375189809.14004.1300582349625.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (HADOOP-7201) Add Cassandra to list of
 Hadoop-related projects at Apache
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Ellis updated HADOOP-7201:
-----------------------------------

    Attachment: HADOOP-7201.txt

Patch to add Cassandra to list of Hadoop-related projects at Apache

> Add Cassandra to list of Hadoop-related projects at Apache
> ----------------------------------------------------------
>
>                 Key: HADOOP-7201
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7201
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Jonathan Ellis
>            Priority: Minor
>             Fix For: site
>
>         Attachments: HADOOP-7201.txt
>
>
> Cassandra provides native read/write support for java Hadoop jobs (with locality), Streaming, Pig, and Hive.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13655-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Mar 20 20:52:31 2011
Return-Path: <common-issues-return-13655-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82881 invoked from network); 20 Mar 2011 20:52:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Mar 2011 20:52:31 -0000
Received: (qmail 65704 invoked by uid 500); 20 Mar 2011 20:52:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65664 invoked by uid 500); 20 Mar 2011 20:52:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65656 invoked by uid 99); 20 Mar 2011 20:52:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 Mar 2011 20:52:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 Mar 2011 20:52:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 042D43AD3E0
	for <common-issues@hadoop.apache.org>; Sun, 20 Mar 2011 20:52:06 +0000 (UTC)
Date: Sun, 20 Mar 2011 20:52:06 +0000 (UTC)
From: "Ian Holsman (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <263583017.77.1300654326013.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1375189809.14004.1300582349625.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7201) Add Cassandra to list of
 Hadoop-related projects at Apache
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008968#comment-13008968 ] 

Ian Holsman commented on HADOOP-7201:
-------------------------------------

done.
can someone please close this. I don't have the privs.

> Add Cassandra to list of Hadoop-related projects at Apache
> ----------------------------------------------------------
>
>                 Key: HADOOP-7201
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7201
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Jonathan Ellis
>            Priority: Minor
>             Fix For: site
>
>         Attachments: HADOOP-7201.txt
>
>
> Cassandra provides native read/write support for java Hadoop jobs (with locality), Streaming, Pig, and Hive.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13656-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Mar 20 21:24:28 2011
Return-Path: <common-issues-return-13656-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80893 invoked from network); 20 Mar 2011 21:24:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Mar 2011 21:24:28 -0000
Received: (qmail 96969 invoked by uid 500); 20 Mar 2011 21:24:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96898 invoked by uid 500); 20 Mar 2011 21:24:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96888 invoked by uid 99); 20 Mar 2011 21:24:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 Mar 2011 21:24:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 Mar 2011 21:24:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 13DDD3AD93D
	for <common-issues@hadoop.apache.org>; Sun, 20 Mar 2011 21:24:06 +0000 (UTC)
Date: Sun, 20 Mar 2011 21:24:06 +0000 (UTC)
From: "Jonathan Ellis (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1215655756.86.1300656246077.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1375189809.14004.1300582349625.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7201) Add Cassandra to list of
 Hadoop-related projects at Apache
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Ellis resolved HADOOP-7201.
------------------------------------

      Resolution: Fixed
        Assignee: Jonathan Ellis
    Hadoop Flags: [Reviewed]

resolved, thanks!

> Add Cassandra to list of Hadoop-related projects at Apache
> ----------------------------------------------------------
>
>                 Key: HADOOP-7201
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7201
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Jonathan Ellis
>            Assignee: Jonathan Ellis
>            Priority: Minor
>             Fix For: site
>
>         Attachments: HADOOP-7201.txt
>
>
> Cassandra provides native read/write support for java Hadoop jobs (with locality), Streaming, Pig, and Hive.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13657-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 21 18:31:52 2011
Return-Path: <common-issues-return-13657-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72543 invoked from network); 21 Mar 2011 18:31:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Mar 2011 18:31:51 -0000
Received: (qmail 84110 invoked by uid 500); 21 Mar 2011 18:31:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83489 invoked by uid 500); 21 Mar 2011 18:31:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83254 invoked by uid 99); 21 Mar 2011 18:31:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 18:31:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 18:31:48 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C86EE402B97
	for <common-issues@hadoop.apache.org>; Mon, 21 Mar 2011 18:31:11 +0000 (UTC)
Date: Mon, 21 Mar 2011 18:31:11 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1587140683.1515.1300732271817.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7202) Improve Command base class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Improve Command base class
--------------------------

                 Key: HADOOP-7202
                 URL: https://issues.apache.org/jira/browse/HADOOP-7202
             Project: Hadoop Common
          Issue Type: Improvement
    Affects Versions: 0.22.0
            Reporter: Daryn Sharp
            Assignee: Daryn Sharp
             Fix For: 0.23.0


Need to extend the Command base class to allow all command to easily subclass from a code set of code that correctly handles globs and exit codes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13658-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 21 18:39:43 2011
Return-Path: <common-issues-return-13658-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2904 invoked from network); 21 Mar 2011 18:39:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Mar 2011 18:39:43 -0000
Received: (qmail 90431 invoked by uid 500); 21 Mar 2011 18:39:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90403 invoked by uid 500); 21 Mar 2011 18:39:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90395 invoked by uid 99); 21 Mar 2011 18:39:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 18:39:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 18:39:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C94A1402E52
	for <common-issues@hadoop.apache.org>; Mon, 21 Mar 2011 18:39:05 +0000 (UTC)
Date: Mon, 21 Mar 2011 18:39:05 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1665067263.1530.1300732745821.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1587140683.1515.1300732271817.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7202) Improve Command base class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7202:
--------------------------------

    Status: Patch Available  (was: Open)

> Improve Command base class
> --------------------------
>
>                 Key: HADOOP-7202
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7202
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7202.patch
>
>
> Need to extend the Command base class to allow all command to easily subclass from a code set of code that correctly handles globs and exit codes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13659-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 21 18:39:46 2011
Return-Path: <common-issues-return-13659-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2972 invoked from network); 21 Mar 2011 18:39:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Mar 2011 18:39:45 -0000
Received: (qmail 90956 invoked by uid 500); 21 Mar 2011 18:39:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90910 invoked by uid 500); 21 Mar 2011 18:39:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90902 invoked by uid 99); 21 Mar 2011 18:39:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 18:39:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 18:39:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BBE31402E50
	for <common-issues@hadoop.apache.org>; Mon, 21 Mar 2011 18:39:05 +0000 (UTC)
Date: Mon, 21 Mar 2011 18:39:05 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <442631814.1528.1300732745766.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1587140683.1515.1300732271817.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7202) Improve Command base class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7202:
--------------------------------

    Attachment: HADOOP-7202.patch

Add many methods to Command class to facilitate easy overriding for custom behavior.  Add PathData class to encapsulate a path, fs, and filestatus to avoid passing so many args and/or multiple lookups during execution.  Modify Count cmd to conform since it was the simplest one.

(Note: changes in FsShell are temporary, since all the if/then/elses for commands will soon become a generic call to a command object.)

> Improve Command base class
> --------------------------
>
>                 Key: HADOOP-7202
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7202
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7202.patch
>
>
> Need to extend the Command base class to allow all command to easily subclass from a code set of code that correctly handles globs and exit codes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13660-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 21 20:03:48 2011
Return-Path: <common-issues-return-13660-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42687 invoked from network); 21 Mar 2011 20:03:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Mar 2011 20:03:48 -0000
Received: (qmail 14948 invoked by uid 500); 21 Mar 2011 20:03:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14841 invoked by uid 500); 21 Mar 2011 20:03:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14833 invoked by uid 99); 21 Mar 2011 20:03:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 20:03:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 20:03:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C5B2837A4D
	for <common-issues@hadoop.apache.org>; Mon, 21 Mar 2011 20:03:05 +0000 (UTC)
Date: Mon, 21 Mar 2011 20:03:05 +0000 (UTC)
From: "Anastasis Andronidis (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <780376861.1879.1300737785806.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7203) Folder Paths
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Folder Paths
------------

                 Key: HADOOP-7203
                 URL: https://issues.apache.org/jira/browse/HADOOP-7203
             Project: Hadoop Common
          Issue Type: Bug
          Components: build
    Affects Versions: 0.21.0
         Environment: Mac OS X (but the same effect in linux)

$java -version
java version "1.6.0_24"
Java(TM) SE Runtime Environment (build 1.6.0_24-b07-334-10M3326)
Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02-334, mixed mode)
            Reporter: Anastasis Andronidis
            Priority: Minor


I tried to execute: ./bin/hadoop namenode -format in hadoop-0.21.0 inside a folder with path: ~/Desktop/my test/hadoop-0.21.0/
But i kept receiving the following error:

Exception in thread "main" java.lang.NoClassDefFoundError: test/hadoop-0/21/0/bin////logs
Caused by: java.lang.ClassNotFoundException: me.hadoop-0.21.0.bin....logs
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:248)

Tried to change the language just to test what would happen. So I got a same error with a folder name with spaces:

Exception in thread "main" java.lang.NoClassDefFoundError: ??????
Caused by: java.lang.ClassNotFoundException: ??????
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:248)

(You can notice the ???, might this be a java issue?)

I was able to run it only in folder names that didn't include spaces. Both english and greek.

I tried to trace this issue and I found that the problem is in the HADOOP_HOME/hdfs script. On the last line where: exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS $CLASS "$@" tries to be executed, variable HADOOP_OPTS has white spaces which produce the above errors. 

A solution to this problem could be to include double quotes to each path when they are collected and call exec command with eval at the front.

I found a workaround to this issue by adding:

tmp=`echo ${HADOOP_OPTS// /\\\ }`
eval exec "$JAVA" $JAVA_HEAP_MAX $(echo ${tmp//\\ \-/\ \-}) " " $CLASS "$@"

This code basically it added backslashes in front of each space in a path.


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13661-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 21 20:20:50 2011
Return-Path: <common-issues-return-13661-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86216 invoked from network); 21 Mar 2011 20:20:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Mar 2011 20:20:49 -0000
Received: (qmail 47144 invoked by uid 500); 21 Mar 2011 20:20:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47121 invoked by uid 500); 21 Mar 2011 20:20:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47113 invoked by uid 99); 21 Mar 2011 20:20:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 20:20:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 20:20:44 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3FC9E37FF4
	for <common-issues@hadoop.apache.org>; Mon, 21 Mar 2011 20:20:07 +0000 (UTC)
Date: Mon, 21 Mar 2011 20:20:07 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2005010112.1924.1300738807257.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6949) Reduces RPC packet size for
 primitive arrays, especially long[], which is used at block reporting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6949:
------------------------------------

    Status: Open  (was: Patch Available)

> Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6949
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6949
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Navis
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: ObjectWritable.diff, arrayprim.patch, arrayprim_v4.patch, arrayprim_v5.patch, arrayprim_v6.patch, arrayprim_v7.patch
>
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n
> It would not be needed to specify each element types for primitive arrays.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13662-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 21 20:22:47 2011
Return-Path: <common-issues-return-13662-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96699 invoked from network); 21 Mar 2011 20:22:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Mar 2011 20:22:46 -0000
Received: (qmail 50906 invoked by uid 500); 21 Mar 2011 20:22:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50875 invoked by uid 500); 21 Mar 2011 20:22:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50867 invoked by uid 99); 21 Mar 2011 20:22:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 20:22:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 20:22:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C4D85360E0
	for <common-issues@hadoop.apache.org>; Mon, 21 Mar 2011 20:22:05 +0000 (UTC)
Date: Mon, 21 Mar 2011 20:22:05 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1513571876.1938.1300738925802.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6949) Reduces RPC packet size for
 primitive arrays, especially long[], which is used at block reporting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6949:
------------------------------------

    Status: Patch Available  (was: Open)

> Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6949
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6949
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Navis
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: ObjectWritable.diff, arrayprim.patch, arrayprim_v4.patch, arrayprim_v5.patch, arrayprim_v6.patch, arrayprim_v7.patch
>
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n
> It would not be needed to specify each element types for primitive arrays.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13663-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 21 20:42:47 2011
Return-Path: <common-issues-return-13663-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57725 invoked from network); 21 Mar 2011 20:42:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Mar 2011 20:42:47 -0000
Received: (qmail 89218 invoked by uid 500); 21 Mar 2011 20:42:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89153 invoked by uid 500); 21 Mar 2011 20:42:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89145 invoked by uid 99); 21 Mar 2011 20:42:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 20:42:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 20:42:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C61E93681C
	for <common-issues@hadoop.apache.org>; Mon, 21 Mar 2011 20:42:06 +0000 (UTC)
Date: Mon, 21 Mar 2011 20:42:06 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1242547691.2024.1300740126808.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6949) Reduces RPC packet size for
 primitive arrays, especially long[], which is used at block reporting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13009359#comment-13009359 ] 

Hadoop QA commented on HADOOP-6949:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12474048/arrayprim_v7.patch
  against trunk revision 1082788.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/315//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/315//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/315//console

This message is automatically generated.

> Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6949
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6949
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Navis
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: ObjectWritable.diff, arrayprim.patch, arrayprim_v4.patch, arrayprim_v5.patch, arrayprim_v6.patch, arrayprim_v7.patch
>
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n
> It would not be needed to specify each element types for primitive arrays.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13664-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 21 20:44:50 2011
Return-Path: <common-issues-return-13664-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58538 invoked from network); 21 Mar 2011 20:44:50 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Mar 2011 20:44:50 -0000
Received: (qmail 96031 invoked by uid 500); 21 Mar 2011 20:44:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95990 invoked by uid 500); 21 Mar 2011 20:44:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95982 invoked by uid 99); 21 Mar 2011 20:44:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 20:44:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 20:44:44 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 04B2136912
	for <common-issues@hadoop.apache.org>; Mon, 21 Mar 2011 20:44:07 +0000 (UTC)
Date: Mon, 21 Mar 2011 20:44:07 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <434009500.2115.1300740247016.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6949) Reduces RPC packet size for
 primitive arrays, especially long[], which is used at block reporting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13009362#comment-13009362 ] 

Suresh Srinivas commented on HADOOP-6949:
-----------------------------------------

+1 for the patch.

> Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6949
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6949
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Navis
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: ObjectWritable.diff, arrayprim.patch, arrayprim_v4.patch, arrayprim_v5.patch, arrayprim_v6.patch, arrayprim_v7.patch
>
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n
> It would not be needed to specify each element types for primitive arrays.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13665-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 21 21:16:51 2011
Return-Path: <common-issues-return-13665-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65249 invoked from network); 21 Mar 2011 21:16:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Mar 2011 21:16:51 -0000
Received: (qmail 54812 invoked by uid 500); 21 Mar 2011 21:16:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54693 invoked by uid 500); 21 Mar 2011 21:16:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54685 invoked by uid 99); 21 Mar 2011 21:16:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 21:16:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 21:16:48 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1B4D137F7E
	for <common-issues@hadoop.apache.org>; Mon, 21 Mar 2011 21:16:11 +0000 (UTC)
Date: Mon, 21 Mar 2011 21:16:11 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <502896594.2365.1300742171107.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6949) Reduces RPC packet size for
 primitive arrays, especially long[], which is used at block reporting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6949:
------------------------------------

      Resolution: Fixed
    Release Note: 
Increments the RPC protocol version in org.apache.hadoop.ipc.Server from 4 to 5.
Introduces ArrayPrimitiveWritable for a much more efficient wire format to transmit arrays of primitives over RPC. ObjectWritable uses the new writable for array of primitives for RPC and continues to use existing format for on-disk data.

  was:
Increments the RPC protocol version in org.apache.hadoop.ipc.Server from 4 to 5.
Introduces a much more efficient wire format to transmit arrays of primitives.  This is done in a way that is 100% backward compatible with previously persisted data, so any stored data will still be readable.  However, data written in the new version cannot be read by a process using the old version of protocol.  Only RPC uses the new format; file writing protocols for application data are unchanged..

    Hadoop Flags: [Incompatible change, Reviewed]  (was: [Incompatible change])
          Status: Resolved  (was: Patch Available)

I committed the patch. Thank you Matt.

> Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6949
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6949
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Navis
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: ObjectWritable.diff, arrayprim.patch, arrayprim_v4.patch, arrayprim_v5.patch, arrayprim_v6.patch, arrayprim_v7.patch
>
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n
> It would not be needed to specify each element types for primitive arrays.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13666-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 21 21:24:47 2011
Return-Path: <common-issues-return-13666-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5104 invoked from network); 21 Mar 2011 21:24:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Mar 2011 21:24:46 -0000
Received: (qmail 68087 invoked by uid 500); 21 Mar 2011 21:24:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68062 invoked by uid 500); 21 Mar 2011 21:24:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68054 invoked by uid 99); 21 Mar 2011 21:24:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 21:24:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 21:24:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E337B36203
	for <common-issues@hadoop.apache.org>; Mon, 21 Mar 2011 21:24:05 +0000 (UTC)
Date: Mon, 21 Mar 2011 21:24:05 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1485558788.2387.1300742645927.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6949) Reduces RPC packet size for
 primitive arrays, especially long[], which is used at block reporting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13009393#comment-13009393 ] 

Hudson commented on HADOOP-6949:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #531 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/531/])
    HADOOP-6949. Reduce RPC packet size of primitive arrays using ArrayPrimitiveWritable instead of ObjectWritable. Contributed by Matt Foley.


> Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6949
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6949
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Navis
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: ObjectWritable.diff, arrayprim.patch, arrayprim_v4.patch, arrayprim_v5.patch, arrayprim_v6.patch, arrayprim_v7.patch
>
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n
> It would not be needed to specify each element types for primitive arrays.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13667-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 21 22:27:49 2011
Return-Path: <common-issues-return-13667-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6114 invoked from network); 21 Mar 2011 22:27:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Mar 2011 22:27:48 -0000
Received: (qmail 75951 invoked by uid 500); 21 Mar 2011 22:27:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75833 invoked by uid 500); 21 Mar 2011 22:27:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75760 invoked by uid 99); 21 Mar 2011 22:27:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 22:27:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 22:27:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C9DDF41439
	for <common-issues@hadoop.apache.org>; Mon, 21 Mar 2011 22:27:05 +0000 (UTC)
Date: Mon, 21 Mar 2011 22:27:05 +0000 (UTC)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1224526256.2676.1300746425823.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7204) remove local unused fs variable from
 CmdHandler and FsShellPermissions.changePermissions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

remove local unused fs variable from CmdHandler and FsShellPermissions.changePermissions
----------------------------------------------------------------------------------------

                 Key: HADOOP-7204
                 URL: https://issues.apache.org/jira/browse/HADOOP-7204
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Boris Shkolnik
            Assignee: Boris Shkolnik
            Priority: Minor




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13668-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 21 22:29:49 2011
Return-Path: <common-issues-return-13668-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6617 invoked from network); 21 Mar 2011 22:29:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Mar 2011 22:29:49 -0000
Received: (qmail 78394 invoked by uid 500); 21 Mar 2011 22:29:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78339 invoked by uid 500); 21 Mar 2011 22:29:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78266 invoked by uid 99); 21 Mar 2011 22:29:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 22:29:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 22:29:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CA27A414D2
	for <common-issues@hadoop.apache.org>; Mon, 21 Mar 2011 22:29:05 +0000 (UTC)
Date: Mon, 21 Mar 2011 22:29:05 +0000 (UTC)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1835020432.2687.1300746545824.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1224526256.2676.1300746425823.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7204) remove local unused fs variable from
 CmdHandler and FsShellPermissions.changePermissions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-7204:
-----------------------------------

    Attachment: HADOOP-7204.patch

> remove local unused fs variable from CmdHandler and FsShellPermissions.changePermissions
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7204
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7204
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>            Priority: Minor
>         Attachments: HADOOP-7204.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13669-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 22 04:55:52 2011
Return-Path: <common-issues-return-13669-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56774 invoked from network); 22 Mar 2011 04:55:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Mar 2011 04:55:51 -0000
Received: (qmail 97576 invoked by uid 500); 22 Mar 2011 04:55:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97552 invoked by uid 500); 22 Mar 2011 04:55:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97543 invoked by uid 99); 22 Mar 2011 04:55:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 04:55:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 04:55:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B52C141B05
	for <common-issues@hadoop.apache.org>; Tue, 22 Mar 2011 04:55:05 +0000 (UTC)
Date: Tue, 22 Mar 2011 04:55:05 +0000 (UTC)
From: "thron_xv (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1503956360.3231.1300769705738.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6953) start-{dfs,mapred}.sh scripts fail
 if HADOOP_HOME is not set
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13009550#comment-13009550 ] 

thron_xv commented on HADOOP-6953:
----------------------------------

To avoid this issue, you can configure HADOOP_HOME in environment.

For example in Cygwin, just do as below:
  export HADOOP_HOME=/cygdrive/c/hadoop/

> start-{dfs,mapred}.sh scripts fail if HADOOP_HOME is not set
> ------------------------------------------------------------
>
>                 Key: HADOOP-6953
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6953
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.21.0
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.1, 0.22.0
>
>
> If the HADOOP_HOME environment variable is not set then the start and stop scripts for HDFS and MapReduce fail with "Hadoop common not found.". The start-all.sh and stop-all.sh scripts are not affected.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13670-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 22 11:13:45 2011
Return-Path: <common-issues-return-13670-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69898 invoked from network); 22 Mar 2011 11:13:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Mar 2011 11:13:45 -0000
Received: (qmail 12583 invoked by uid 500); 22 Mar 2011 11:13:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12532 invoked by uid 500); 22 Mar 2011 11:13:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12524 invoked by uid 99); 22 Mar 2011 11:13:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 11:13:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 11:13:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1FF774101B
	for <common-issues@hadoop.apache.org>; Tue, 22 Mar 2011 11:13:06 +0000 (UTC)
Date: Tue, 22 Mar 2011 11:13:06 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1580428621.3583.1300792386127.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6949) Reduces RPC packet size for
 primitive arrays, especially long[], which is used at block reporting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13009615#comment-13009615 ] 

Hudson commented on HADOOP-6949:
--------------------------------

Integrated in Hadoop-Common-trunk #638 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/638/])
    HADOOP-6949. Reduce RPC packet size of primitive arrays using ArrayPrimitiveWritable instead of ObjectWritable. Contributed by Matt Foley.


> Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6949
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6949
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Navis
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: ObjectWritable.diff, arrayprim.patch, arrayprim_v4.patch, arrayprim_v5.patch, arrayprim_v6.patch, arrayprim_v7.patch
>
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n
> It would not be needed to specify each element types for primitive arrays.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13671-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 22 16:23:49 2011
Return-Path: <common-issues-return-13671-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75675 invoked from network); 22 Mar 2011 16:23:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Mar 2011 16:23:49 -0000
Received: (qmail 18599 invoked by uid 500); 22 Mar 2011 16:23:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18374 invoked by uid 500); 22 Mar 2011 16:23:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18357 invoked by uid 99); 22 Mar 2011 16:23:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 16:23:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 16:23:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BE8D642E6C
	for <common-issues@hadoop.apache.org>; Tue, 22 Mar 2011 16:23:05 +0000 (UTC)
Date: Tue, 22 Mar 2011 16:23:05 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1908839080.4012.1300810985777.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-5983) Namenode shouldn't read
 mapred-site.xml
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp reassigned HADOOP-5983:
-----------------------------------

    Assignee: Daryn Sharp

> Namenode shouldn't read mapred-site.xml
> ---------------------------------------
>
>                 Key: HADOOP-5983
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5983
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.20.0
>            Reporter: Rajiv Chittajallu
>            Assignee: Daryn Sharp
>
> The name node seem to read mapred-site.xml and fails if it can't parse it.
> 2009-06-05 22:37:15,663 FATAL org.apache.hadoop.conf.Configuration: error parsing conf file: org.xml.sax.SAXParseException: Error attempting to parse XML file (href='/hadoop/conf/local/local-mapred-site.xml').
> 2009-06-05 22:37:15,664 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.lang.RuntimeException: org.xml.sax.SAXParseException: Error attempting to parse XML file (href='/hadoop/conf/local/local-mapred-site.xml').
> In our config,  local-mapred-site.xml is included only in mapred-site.xml which we don't push to the namenode.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13672-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 22 18:16:44 2011
Return-Path: <common-issues-return-13672-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34495 invoked from network); 22 Mar 2011 18:16:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Mar 2011 18:16:43 -0000
Received: (qmail 49604 invoked by uid 500); 22 Mar 2011 18:16:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49579 invoked by uid 500); 22 Mar 2011 18:16:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49571 invoked by uid 99); 22 Mar 2011 18:16:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 18:16:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 18:16:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CB44B42AD0
	for <common-issues@hadoop.apache.org>; Tue, 22 Mar 2011 18:16:05 +0000 (UTC)
Date: Tue, 22 Mar 2011 18:16:05 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <63224446.4256.1300817765829.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1224526256.2676.1300746425823.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7204) remove local unused fs variable
 from CmdHandler and FsShellPermissions.changePermissions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13009754#comment-13009754 ] 

Jitendra Nath Pandey commented on HADOOP-7204:
----------------------------------------------

Patch looks good to me. +1
None of the changed methods are public, however it would be good to compile Hdfs and map-reduce with this change, just to make sure.

> remove local unused fs variable from CmdHandler and FsShellPermissions.changePermissions
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7204
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7204
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>            Priority: Minor
>         Attachments: HADOOP-7204.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13673-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 22 18:34:49 2011
Return-Path: <common-issues-return-13673-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23564 invoked from network); 22 Mar 2011 18:34:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Mar 2011 18:34:47 -0000
Received: (qmail 82630 invoked by uid 500); 22 Mar 2011 18:34:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82573 invoked by uid 500); 22 Mar 2011 18:34:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82509 invoked by uid 99); 22 Mar 2011 18:34:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 18:34:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 18:34:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0BA6F42267
	for <common-issues@hadoop.apache.org>; Tue, 22 Mar 2011 18:34:06 +0000 (UTC)
Date: Tue, 22 Mar 2011 18:34:06 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2109891895.4306.1300818846044.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7205) automatically determine JAVA_HOME on
 OS X
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

automatically determine JAVA_HOME on OS X
-----------------------------------------

                 Key: HADOOP-7205
                 URL: https://issues.apache.org/jira/browse/HADOOP-7205
             Project: Hadoop Common
          Issue Type: Improvement
    Affects Versions: 0.22.0
            Reporter: Daryn Sharp
            Assignee: Daryn Sharp
             Fix For: 0.23.0


OS X provides a java_home command that will return the user's selected jvm.  The hadoop-env.sh should use this command if JAVA_HOME is not set.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13674-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 22 18:36:44 2011
Return-Path: <common-issues-return-13674-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33089 invoked from network); 22 Mar 2011 18:36:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Mar 2011 18:36:43 -0000
Received: (qmail 87478 invoked by uid 500); 22 Mar 2011 18:36:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87435 invoked by uid 500); 22 Mar 2011 18:36:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87427 invoked by uid 99); 22 Mar 2011 18:36:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 18:36:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 18:36:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 42007423BA
	for <common-issues@hadoop.apache.org>; Tue, 22 Mar 2011 18:36:06 +0000 (UTC)
Date: Tue, 22 Mar 2011 18:36:06 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <656437407.4324.1300818966266.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2109891895.4306.1300818846044.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7205) automatically determine JAVA_HOME on
 OS X
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7205:
--------------------------------

    Attachment: HADOOP-7205.patch

> automatically determine JAVA_HOME on OS X
> -----------------------------------------
>
>                 Key: HADOOP-7205
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7205
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7205.patch
>
>
> OS X provides a java_home command that will return the user's selected jvm.  The hadoop-env.sh should use this command if JAVA_HOME is not set.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13675-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 22 18:36:47 2011
Return-Path: <common-issues-return-13675-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33228 invoked from network); 22 Mar 2011 18:36:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Mar 2011 18:36:46 -0000
Received: (qmail 88023 invoked by uid 500); 22 Mar 2011 18:36:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87968 invoked by uid 500); 22 Mar 2011 18:36:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87885 invoked by uid 99); 22 Mar 2011 18:36:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 18:36:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 18:36:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E7022423B0
	for <common-issues@hadoop.apache.org>; Tue, 22 Mar 2011 18:36:05 +0000 (UTC)
Date: Tue, 22 Mar 2011 18:36:05 +0000 (UTC)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1713106700.4314.1300818965942.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1224526256.2676.1300746425823.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7204) remove local unused fs variable from
 CmdHandler and FsShellPermissions.changePermissions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-7204:
-----------------------------------

    Hadoop Flags: [Reviewed]
          Status: Patch Available  (was: Open)

> remove local unused fs variable from CmdHandler and FsShellPermissions.changePermissions
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7204
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7204
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>            Priority: Minor
>         Attachments: HADOOP-7204.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13676-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 22 18:36:47 2011
Return-Path: <common-issues-return-13676-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33263 invoked from network); 22 Mar 2011 18:36:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Mar 2011 18:36:46 -0000
Received: (qmail 88446 invoked by uid 500); 22 Mar 2011 18:36:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88403 invoked by uid 500); 22 Mar 2011 18:36:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88395 invoked by uid 99); 22 Mar 2011 18:36:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 18:36:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 18:36:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D7C85423AD
	for <common-issues@hadoop.apache.org>; Tue, 22 Mar 2011 18:36:05 +0000 (UTC)
Date: Tue, 22 Mar 2011 18:36:05 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <74567979.4312.1300818965880.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2109891895.4306.1300818846044.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7205) automatically determine JAVA_HOME on
 OS X
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7205:
--------------------------------

    Attachment: HADOOP-7205.patch

> automatically determine JAVA_HOME on OS X
> -----------------------------------------
>
>                 Key: HADOOP-7205
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7205
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7205.patch
>
>
> OS X provides a java_home command that will return the user's selected jvm.  The hadoop-env.sh should use this command if JAVA_HOME is not set.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13677-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 22 18:36:48 2011
Return-Path: <common-issues-return-13677-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33333 invoked from network); 22 Mar 2011 18:36:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Mar 2011 18:36:47 -0000
Received: (qmail 88793 invoked by uid 500); 22 Mar 2011 18:36:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88761 invoked by uid 500); 22 Mar 2011 18:36:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88753 invoked by uid 99); 22 Mar 2011 18:36:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 18:36:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 18:36:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 04A2E423B2
	for <common-issues@hadoop.apache.org>; Tue, 22 Mar 2011 18:36:06 +0000 (UTC)
Date: Tue, 22 Mar 2011 18:36:06 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <807806642.4316.1300818966015.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2109891895.4306.1300818846044.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7205) automatically determine JAVA_HOME on
 OS X
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7205:
--------------------------------

    Attachment:     (was: HADOOP-7205.patch)

> automatically determine JAVA_HOME on OS X
> -----------------------------------------
>
>                 Key: HADOOP-7205
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7205
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7205.patch
>
>
> OS X provides a java_home command that will return the user's selected jvm.  The hadoop-env.sh should use this command if JAVA_HOME is not set.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13678-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 22 18:37:08 2011
Return-Path: <common-issues-return-13678-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33865 invoked from network); 22 Mar 2011 18:37:07 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Mar 2011 18:37:07 -0000
Received: (qmail 90902 invoked by uid 500); 22 Mar 2011 18:37:07 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90864 invoked by uid 500); 22 Mar 2011 18:37:07 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90834 invoked by uid 99); 22 Mar 2011 18:37:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 18:37:06 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 18:37:04 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4D09E423BB
	for <common-issues@hadoop.apache.org>; Tue, 22 Mar 2011 18:36:06 +0000 (UTC)
Date: Tue, 22 Mar 2011 18:36:06 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1120004810.4325.1300818966312.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2109891895.4306.1300818846044.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7205) automatically determine JAVA_HOME on
 OS X
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7205:
--------------------------------

    Status: Patch Available  (was: Open)

> automatically determine JAVA_HOME on OS X
> -----------------------------------------
>
>                 Key: HADOOP-7205
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7205
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7205.patch
>
>
> OS X provides a java_home command that will return the user's selected jvm.  The hadoop-env.sh should use this command if JAVA_HOME is not set.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13679-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 22 18:38:46 2011
Return-Path: <common-issues-return-13679-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35432 invoked from network); 22 Mar 2011 18:38:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Mar 2011 18:38:46 -0000
Received: (qmail 822 invoked by uid 500); 22 Mar 2011 18:38:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 788 invoked by uid 500); 22 Mar 2011 18:38:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 780 invoked by uid 99); 22 Mar 2011 18:38:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 18:38:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 18:38:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 363BF424B1
	for <common-issues@hadoop.apache.org>; Tue, 22 Mar 2011 18:38:06 +0000 (UTC)
Date: Tue, 22 Mar 2011 18:38:06 +0000 (UTC)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1843026161.4331.1300819086219.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1224526256.2676.1300746425823.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7204) remove local unused fs variable
 from CmdHandler and FsShellPermissions.changePermissions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13009770#comment-13009770 ] 

Boris Shkolnik commented on HADOOP-7204:
----------------------------------------

compiled hdfs and mr with the change(jar-test). All compiled.

> remove local unused fs variable from CmdHandler and FsShellPermissions.changePermissions
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7204
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7204
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>            Priority: Minor
>         Attachments: HADOOP-7204.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13680-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 22 18:54:44 2011
Return-Path: <common-issues-return-13680-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83264 invoked from network); 22 Mar 2011 18:54:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Mar 2011 18:54:44 -0000
Received: (qmail 32877 invoked by uid 500); 22 Mar 2011 18:54:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32814 invoked by uid 500); 22 Mar 2011 18:54:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32795 invoked by uid 99); 22 Mar 2011 18:54:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 18:54:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 18:54:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 26D4642BBB
	for <common-issues@hadoop.apache.org>; Tue, 22 Mar 2011 18:54:06 +0000 (UTC)
Date: Tue, 22 Mar 2011 18:54:06 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1637655371.4376.1300820046155.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-5983) Namenode shouldn't read
 mapred-site.xml
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-5983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13009783#comment-13009783 ] 

Daryn Sharp commented on HADOOP-5983:
-------------------------------------

I cannot reproduce on trunk.  I am unable to find references to the mapred config in the namenode code.  To verify, I created a mapred-site.xml and local-mapred-site.xml with syntax errors.  The namenode starts w/o a problem.

> Namenode shouldn't read mapred-site.xml
> ---------------------------------------
>
>                 Key: HADOOP-5983
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5983
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.20.0
>            Reporter: Rajiv Chittajallu
>            Assignee: Daryn Sharp
>
> The name node seem to read mapred-site.xml and fails if it can't parse it.
> 2009-06-05 22:37:15,663 FATAL org.apache.hadoop.conf.Configuration: error parsing conf file: org.xml.sax.SAXParseException: Error attempting to parse XML file (href='/hadoop/conf/local/local-mapred-site.xml').
> 2009-06-05 22:37:15,664 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.lang.RuntimeException: org.xml.sax.SAXParseException: Error attempting to parse XML file (href='/hadoop/conf/local/local-mapred-site.xml').
> In our config,  local-mapred-site.xml is included only in mapred-site.xml which we don't push to the namenode.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13681-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 22 18:56:43 2011
Return-Path: <common-issues-return-13681-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84032 invoked from network); 22 Mar 2011 18:56:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Mar 2011 18:56:43 -0000
Received: (qmail 40008 invoked by uid 500); 22 Mar 2011 18:56:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39972 invoked by uid 500); 22 Mar 2011 18:56:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39960 invoked by uid 99); 22 Mar 2011 18:56:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 18:56:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 18:56:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C534D42C93
	for <common-issues@hadoop.apache.org>; Tue, 22 Mar 2011 18:56:05 +0000 (UTC)
Date: Tue, 22 Mar 2011 18:56:05 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <648731709.4384.1300820165804.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-5983) Namenode shouldn't read
 mapred-site.xml
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-5983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp resolved HADOOP-5983.
---------------------------------

    Resolution: Cannot Reproduce

> Namenode shouldn't read mapred-site.xml
> ---------------------------------------
>
>                 Key: HADOOP-5983
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5983
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.20.0
>            Reporter: Rajiv Chittajallu
>            Assignee: Daryn Sharp
>
> The name node seem to read mapred-site.xml and fails if it can't parse it.
> 2009-06-05 22:37:15,663 FATAL org.apache.hadoop.conf.Configuration: error parsing conf file: org.xml.sax.SAXParseException: Error attempting to parse XML file (href='/hadoop/conf/local/local-mapred-site.xml').
> 2009-06-05 22:37:15,664 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.lang.RuntimeException: org.xml.sax.SAXParseException: Error attempting to parse XML file (href='/hadoop/conf/local/local-mapred-site.xml').
> In our config,  local-mapred-site.xml is included only in mapred-site.xml which we don't push to the namenode.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13682-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 22 19:04:50 2011
Return-Path: <common-issues-return-13682-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26908 invoked from network); 22 Mar 2011 19:04:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Mar 2011 19:04:46 -0000
Received: (qmail 55538 invoked by uid 500); 22 Mar 2011 19:04:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55442 invoked by uid 500); 22 Mar 2011 19:04:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55144 invoked by uid 99); 22 Mar 2011 19:04:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 19:04:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 19:04:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B70E44205E
	for <common-issues@hadoop.apache.org>; Tue, 22 Mar 2011 19:04:05 +0000 (UTC)
Date: Tue, 22 Mar 2011 19:04:05 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1011960141.4409.1300820645746.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2109891895.4306.1300818846044.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7205) automatically determine JAVA_HOME
 on OS X
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13009788#comment-13009788 ] 

Allen Wittenauer commented on HADOOP-7205:
------------------------------------------

-1

a) We've been avoiding OS-specific changes in the shells for a very long time, including one to autodetect Java on various Linux distributions.  I see no reason to reverse this now.  I've said it before and I'll say it again:  stuff like this should really go into an installer that modifies hadoop-env.sh, not part of the base executable.

b) Given that Java is moving from Apple controlled to Oracle controlled on OS X, there is a very good chance this command will disappear in a future release.

> automatically determine JAVA_HOME on OS X
> -----------------------------------------
>
>                 Key: HADOOP-7205
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7205
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7205.patch
>
>
> OS X provides a java_home command that will return the user's selected jvm.  The hadoop-env.sh should use this command if JAVA_HOME is not set.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13683-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 22 19:06:44 2011
Return-Path: <common-issues-return-13683-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37957 invoked from network); 22 Mar 2011 19:06:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Mar 2011 19:06:43 -0000
Received: (qmail 57078 invoked by uid 500); 22 Mar 2011 19:06:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57049 invoked by uid 500); 22 Mar 2011 19:06:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57037 invoked by uid 99); 22 Mar 2011 19:06:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 19:06:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 19:06:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C55D84214B
	for <common-issues@hadoop.apache.org>; Tue, 22 Mar 2011 19:06:05 +0000 (UTC)
Date: Tue, 22 Mar 2011 19:06:05 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1272532877.4413.1300820765805.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2109891895.4306.1300818846044.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7205) automatically determine JAVA_HOME on
 OS X
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Allen Wittenauer updated HADOOP-7205:
-------------------------------------

    Priority: Trivial  (was: Major)

> automatically determine JAVA_HOME on OS X
> -----------------------------------------
>
>                 Key: HADOOP-7205
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7205
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7205.patch
>
>
> OS X provides a java_home command that will return the user's selected jvm.  The hadoop-env.sh should use this command if JAVA_HOME is not set.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13684-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 22 20:36:00 2011
Return-Path: <common-issues-return-13684-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23123 invoked from network); 22 Mar 2011 20:36:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Mar 2011 20:36:00 -0000
Received: (qmail 19373 invoked by uid 500); 22 Mar 2011 20:36:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19348 invoked by uid 500); 22 Mar 2011 20:36:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Delivered-To: moderator for common-issues@hadoop.apache.org
Received: (qmail 55012 invoked by uid 99); 22 Mar 2011 12:05:05 -0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
X-Virus-Scanned: OK
Message-ID: <4D88904D.6010006@orkash.com>
Date: Tue, 22 Mar 2011 17:34:29 +0530
From: Manish Yadav <manish.yadav@orkash.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7
MIME-Version: 1.0
To: common-issues@hadoop.apache.org
Subject: Hadoop pipes
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

Hi every body out there
I use wordcount.cpp code in hadoop, in the code of wordcount.cpp in map 
function  i just add too extra line add two lines system("clear"); and 
cout<<"hello world"; just for experimentation purpose.
now the code  is complied successfully ,but when i run it it create 
problems  . It does not produce the desires result.
In my out put i does not found "hello world"or not the screen of my 
system being cleared why it is so? why these statement are  not 
executing(running)?
  what should I do to get my desired result? where  I should  put these 
two line in the code so that they can be executed ?

  I put them in main function too but they are not still responding, 
Please tell me what should  I do to use these two statement ?

From common-issues-return-13685-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 22 20:36:03 2011
Return-Path: <common-issues-return-13685-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23172 invoked from network); 22 Mar 2011 20:36:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Mar 2011 20:36:03 -0000
Received: (qmail 19606 invoked by uid 500); 22 Mar 2011 20:36:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19568 invoked by uid 500); 22 Mar 2011 20:36:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Delivered-To: moderator for common-issues@hadoop.apache.org
Received: (qmail 95037 invoked by uid 99); 22 Mar 2011 12:44:59 -0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
X-Virus-Scanned: OK
Message-ID: <4D8899AD.6000406@orkash.com>
Date: Tue, 22 Mar 2011 18:14:29 +0530
From: Manish Yadav <manish.yadav@orkash.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7
MIME-Version: 1.0
To: common-issues@hadoop.apache.org
Subject: Hadoop configuration
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org

hi everybody
  i configured the hadoop in pseuo distrubution mode on my ubuntu . In 
the start it work goog but now i'm facing a strange problem my name node 
doesnot start . when i use the command bin/start-all.sh it shows msg 
starting namenode  but when i tried to close by the command 
bin/stop-all.sh than it shows
localhost: no tasktracker to stop
no namenode to stop

when i check the log file i didn't understand what it  is here is the 
out put of my log file

************************************************************/
2011-03-11 17:53:34,901 INFO 
org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = ws40-man-lin/192.168.0.133
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 0.20.0
STARTUP_MSG:   build = 
https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.20 -r 
763504; compiled by 'ndaley' on Thu Apr  9 05:18:40 UTC 2009
************************************************************/
2011-03-11 17:53:35,545 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: 
Initializing RPC Metrics with hostName=NameNode, port=8020
2011-03-11 17:53:35,557 INFO 
org.apache.hadoop.hdfs.server.namenode.NameNode: Namenode up at: 
localhost/127.0.0.1:8020
2011-03-11 17:53:35,577 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: 
Initializing JVM Metrics with processName=NameNode, sessionId=null
2011-03-11 17:53:35,579 INFO 
org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics: 
Initializing NameNodeMeterics using context 
object:org.apache.hadoop.metrics.spi.NullContext
2011-03-11 17:53:35,748 INFO 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=hadoop,hadoop
2011-03-11 17:53:35,749 INFO 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup
2011-03-11 17:53:35,749 INFO 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: 
isPermissionEnabled=true
2011-03-11 17:53:35,757 INFO 
org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics: 
Initializing FSNamesystemMetrics using context 
object:org.apache.hadoop.metrics.spi.NullContext
2011-03-11 17:53:35,758 INFO 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered 
FSNamesystemStatusMBean
2011-03-11 17:53:35,825 INFO 
org.apache.hadoop.hdfs.server.common.Storage: Storage directory 
/tmp/hadoop-hadoop/dfs/name does not exist.
2011-03-11 17:53:35,827 ERROR 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem 
initialization failed.
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: 
Directory /tmp/hadoop-hadoop/dfs/name is in an inconsistent state: 
storage directory does not exist or is not accessible.
     at 
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:290)
     at 
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)
     at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:302)
     at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:283)
     at 
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:201)
     at 
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279)
     at 
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:955)
     at 
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:964)
2011-03-11 17:53:35,829 INFO org.apache.hadoop.ipc.Server: Stopping 
server on 8020
2011-03-11 17:53:35,830 ERROR 
org.apache.hadoop.hdfs.server.namenode.NameNode: 
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: 
Directory /tmp/hadoop-hadoop/dfs/name is in an inconsistent state: 
storage directory does not exist or is not accessible.
     at 
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:290)
     at 
org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)
     at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:302)
     at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:283)
     at 
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:201)
     at 
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279)
     at 
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:955)
     at 
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:964)

2011-03-11 17:53:35,830 INFO 
org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ws40-man-lin/192.168.0.133
************************************************************/
please tell me what should i do to run my hadoop correctly?

From common-issues-return-13686-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 22 21:44:45 2011
Return-Path: <common-issues-return-13686-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16377 invoked from network); 22 Mar 2011 21:44:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Mar 2011 21:44:45 -0000
Received: (qmail 13757 invoked by uid 500); 22 Mar 2011 21:44:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13718 invoked by uid 500); 22 Mar 2011 21:44:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13710 invoked by uid 99); 22 Mar 2011 21:44:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 21:44:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 21:44:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C5EAE42239
	for <common-issues@hadoop.apache.org>; Tue, 22 Mar 2011 21:44:05 +0000 (UTC)
Date: Tue, 22 Mar 2011 21:44:05 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <541340500.4740.1300830245807.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2109891895.4306.1300818846044.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7205) automatically determine JAVA_HOME
 on OS X
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13009852#comment-13009852 ] 

Daryn Sharp commented on HADOOP-7205:
-------------------------------------

Point taken, I understand the concern, and I was not aware of this policy.  I was following the motto "when in Rome, do as the Romans do".  It's a standard practice on OS X to use java_home to get the user's selected jvm.  The release notes that announce the deprecation of apple's jvms recommends the use of java_home so it's unlikely to disappear(*).  If the command did disappear, nothing would break with this patch.

Since I'm not aware of history of this policy, may I suggest that the env script sources another env script (only if it exists) that is determined based on uname?  That will compartmentalize os specific logic while retaining certain luxuries expected on each OS.

(*)http://developer.apple.com/library/mac/#releasenotes/Java/JavaSnowLeopardUpdate3LeopardUpdate8RN/NewandNoteworthy/NewandNoteworthy.html



> automatically determine JAVA_HOME on OS X
> -----------------------------------------
>
>                 Key: HADOOP-7205
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7205
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7205.patch
>
>
> OS X provides a java_home command that will return the user's selected jvm.  The hadoop-env.sh should use this command if JAVA_HOME is not set.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13687-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 22 22:36:46 2011
Return-Path: <common-issues-return-13687-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7068 invoked from network); 22 Mar 2011 22:36:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Mar 2011 22:36:46 -0000
Received: (qmail 87361 invoked by uid 500); 22 Mar 2011 22:36:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87326 invoked by uid 500); 22 Mar 2011 22:36:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87318 invoked by uid 99); 22 Mar 2011 22:36:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 22:36:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 22:36:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 00016424FC
	for <common-issues@hadoop.apache.org>; Tue, 22 Mar 2011 22:36:05 +0000 (UTC)
Date: Tue, 22 Mar 2011 22:36:05 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <409088782.4837.1300833365996.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1224526256.2676.1300746425823.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7204) remove local unused fs variable
 from CmdHandler and FsShellPermissions.changePermissions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13009876#comment-13009876 ] 

Hadoop QA commented on HADOOP-7204:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12474243/HADOOP-7204.patch
  against trunk revision 1083957.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/316//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/316//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/316//console

This message is automatically generated.

> remove local unused fs variable from CmdHandler and FsShellPermissions.changePermissions
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7204
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7204
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>            Priority: Minor
>         Attachments: HADOOP-7204.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13688-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 22 22:58:43 2011
Return-Path: <common-issues-return-13688-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55959 invoked from network); 22 Mar 2011 22:58:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Mar 2011 22:58:43 -0000
Received: (qmail 8533 invoked by uid 500); 22 Mar 2011 22:58:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8493 invoked by uid 500); 22 Mar 2011 22:58:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8485 invoked by uid 99); 22 Mar 2011 22:58:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 22:58:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 22:58:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CFA3542E0A
	for <common-issues@hadoop.apache.org>; Tue, 22 Mar 2011 22:58:05 +0000 (UTC)
Date: Tue, 22 Mar 2011 22:58:05 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <327372915.4900.1300834685847.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <995627871.3214.1299545460554.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7171) Support UGI in FileContext API
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7171:
-----------------------------------------

    Status: Open  (was: Patch Available)

> Support UGI in FileContext API
> ------------------------------
>
>                 Key: HADOOP-7171
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7171
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7171.2.patch
>
>
> The FileContext API needs to support UGI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13689-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 22 22:58:43 2011
Return-Path: <common-issues-return-13689-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55981 invoked from network); 22 Mar 2011 22:58:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Mar 2011 22:58:43 -0000
Received: (qmail 8799 invoked by uid 500); 22 Mar 2011 22:58:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8710 invoked by uid 500); 22 Mar 2011 22:58:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8549 invoked by uid 99); 22 Mar 2011 22:58:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 22:58:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 22:58:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DBEA442E0D
	for <common-issues@hadoop.apache.org>; Tue, 22 Mar 2011 22:58:05 +0000 (UTC)
Date: Tue, 22 Mar 2011 22:58:05 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <294378712.4902.1300834685897.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <995627871.3214.1299545460554.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7171) Support UGI in FileContext API
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7171:
-----------------------------------------

    Status: Patch Available  (was: Open)

> Support UGI in FileContext API
> ------------------------------
>
>                 Key: HADOOP-7171
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7171
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7171.2.patch
>
>
> The FileContext API needs to support UGI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13690-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 22 23:46:43 2011
Return-Path: <common-issues-return-13690-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16034 invoked from network); 22 Mar 2011 23:46:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Mar 2011 23:46:43 -0000
Received: (qmail 53019 invoked by uid 500); 22 Mar 2011 23:46:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52992 invoked by uid 500); 22 Mar 2011 23:46:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52984 invoked by uid 99); 22 Mar 2011 23:46:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 23:46:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2011 23:46:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BD77842AA3
	for <common-issues@hadoop.apache.org>; Tue, 22 Mar 2011 23:46:05 +0000 (UTC)
Date: Tue, 22 Mar 2011 23:46:05 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Integrate Snappy compression
----------------------------

                 Key: HADOOP-7206
                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
             Project: Hadoop Common
          Issue Type: New Feature
            Reporter: Eli Collins


Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.

{quote}
Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
{quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13691-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 23 00:14:44 2011
Return-Path: <common-issues-return-13691-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2838 invoked from network); 23 Mar 2011 00:14:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Mar 2011 00:14:44 -0000
Received: (qmail 74591 invoked by uid 500); 23 Mar 2011 00:14:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74554 invoked by uid 500); 23 Mar 2011 00:14:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74546 invoked by uid 99); 23 Mar 2011 00:14:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 00:14:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 00:14:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B4705423CF
	for <common-issues@hadoop.apache.org>; Wed, 23 Mar 2011 00:14:05 +0000 (UTC)
Date: Wed, 23 Mar 2011 00:14:05 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <524968875.5009.1300839245735.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1224526256.2676.1300746425823.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7204) remove local unused fs variable
 from CmdHandler and FsShellPermissions.changePermissions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13009921#comment-13009921 ] 

Hudson commented on HADOOP-7204:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #532 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/532/])
    HADOOP-7204. remove local unused fs variable from CmdHandler and FsShellPermissions.changePermissions


> remove local unused fs variable from CmdHandler and FsShellPermissions.changePermissions
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7204
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7204
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>            Priority: Minor
>         Attachments: HADOOP-7204.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13692-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 23 00:30:45 2011
Return-Path: <common-issues-return-13692-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60767 invoked from network); 23 Mar 2011 00:30:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Mar 2011 00:30:45 -0000
Received: (qmail 89337 invoked by uid 500); 23 Mar 2011 00:30:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89145 invoked by uid 500); 23 Mar 2011 00:30:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89137 invoked by uid 99); 23 Mar 2011 00:30:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 00:30:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 00:30:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DBAB442867
	for <common-issues@hadoop.apache.org>; Wed, 23 Mar 2011 00:30:05 +0000 (UTC)
Date: Wed, 23 Mar 2011 00:30:05 +0000 (UTC)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <896507162.5037.1300840205896.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1224526256.2676.1300746425823.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7204) remove local unused fs variable from
 CmdHandler and FsShellPermissions.changePermissions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-7204:
-----------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

committed to trunk

> remove local unused fs variable from CmdHandler and FsShellPermissions.changePermissions
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7204
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7204
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>            Priority: Minor
>         Attachments: HADOOP-7204.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13693-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 23 00:32:45 2011
Return-Path: <common-issues-return-13693-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73757 invoked from network); 23 Mar 2011 00:32:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Mar 2011 00:32:44 -0000
Received: (qmail 89951 invoked by uid 500); 23 Mar 2011 00:32:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89687 invoked by uid 500); 23 Mar 2011 00:32:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89674 invoked by uid 99); 23 Mar 2011 00:32:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 00:32:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 00:32:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 22EEF428F4
	for <common-issues@hadoop.apache.org>; Wed, 23 Mar 2011 00:32:06 +0000 (UTC)
Date: Wed, 23 Mar 2011 00:32:06 +0000 (UTC)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1723498609.5040.1300840326140.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7207) fs member of FSShell is not really
 needed
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

fs member of FSShell is not really needed
-----------------------------------------

                 Key: HADOOP-7207
                 URL: https://issues.apache.org/jira/browse/HADOOP-7207
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Boris Shkolnik
            Assignee: Boris Shkolnik


FileSystem object should be created on demand when needed in FSShell, instead of always creating one that connects to the default FileSystem.
It will also solve a problem of connecting to non-existant/non-functional default, when it is not really needed for this run.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13694-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 23 01:22:44 2011
Return-Path: <common-issues-return-13694-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38374 invoked from network); 23 Mar 2011 01:22:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Mar 2011 01:22:43 -0000
Received: (qmail 42665 invoked by uid 500); 23 Mar 2011 01:22:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42613 invoked by uid 500); 23 Mar 2011 01:22:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42605 invoked by uid 99); 23 Mar 2011 01:22:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 01:22:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 01:22:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EB08642698
	for <common-issues@hadoop.apache.org>; Wed, 23 Mar 2011 01:22:05 +0000 (UTC)
Date: Wed, 23 Mar 2011 01:22:05 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <606271582.5149.1300843325959.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13009942#comment-13009942 ] 

Todd Lipcon commented on HADOOP-7206:
-------------------------------------

Do you think we should integrate it into Hadoop proper, or start a separate project to implement it as a pluggable codec?

The advantage of doing it as a separate project is that we could provide it for existing Hadoop versions without having to re-release Hadoop. Also allows for quicker bug fixes and granular upgrades when issues are identified.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Eli Collins
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13695-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 23 14:50:46 2011
Return-Path: <common-issues-return-13695-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44301 invoked from network); 23 Mar 2011 14:50:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Mar 2011 14:50:46 -0000
Received: (qmail 4942 invoked by uid 500); 23 Mar 2011 14:50:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4888 invoked by uid 500); 23 Mar 2011 14:50:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4875 invoked by uid 99); 23 Mar 2011 14:50:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 14:50:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 14:50:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BBF87434A2
	for <common-issues@hadoop.apache.org>; Wed, 23 Mar 2011 14:50:05 +0000 (UTC)
Date: Wed, 23 Mar 2011 14:50:05 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1195662636.6130.1300891805766.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7208) equals() and hashCode()
 implementation need to change in StandardSocketFactory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

equals() and hashCode() implementation need to change in StandardSocketFactory
------------------------------------------------------------------------------

                 Key: HADOOP-7208
                 URL: https://issues.apache.org/jira/browse/HADOOP-7208
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Uma Maheswara Rao G


  In Hadoop IPC Client, we are using ClientCache which will maintain the HashMap to keep the Client references.
private Map<SocketFactory, Client> clients =
      new HashMap<SocketFactory, Client>();

 Now let us say, we want use two standard factories with Hadoop. MyStandardSocketFactory (which extends StandardSocketFactory), and StandardSocketFactory. In this case, because of equals and hashcode implementation, MyStandardSocketFactory client can be overridden by StandardSocketFactoryClient


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13696-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 23 18:59:11 2011
Return-Path: <common-issues-return-13696-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50439 invoked from network); 23 Mar 2011 18:59:10 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Mar 2011 18:59:10 -0000
Received: (qmail 64422 invoked by uid 500); 23 Mar 2011 17:12:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64386 invoked by uid 500); 23 Mar 2011 17:12:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64378 invoked by uid 99); 23 Mar 2011 17:12:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 17:12:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 17:12:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EA103463D0
	for <common-issues@hadoop.apache.org>; Wed, 23 Mar 2011 17:12:05 +0000 (UTC)
Date: Wed, 23 Mar 2011 17:12:05 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1348834288.6536.1300900325955.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1627211432.4057.1299578279558.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7174) null is displayed in the
 console,if the src path is invalid while doing copyToLocal operation from
 commandLine
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010228#comment-13010228 ] 

Daryn Sharp commented on HADOOP-7174:
-------------------------------------

Looks good.  Two comments:
1) I'd suggest throwing a more posix error string like srcPath + ": No such file or directory"
2) Should test the exit code of shell.run() to ensure it's non-zero upon failure.  If it happens to return 0, and it's a non-trivial fix, I'm universally addressing it via HADOOP-7176.

> null is displayed in the console,if the src path is invalid while doing copyToLocal operation from commandLine
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7174
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7174
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>            Priority: Minor
>         Attachments: HADOOP-7174.patch
>
>
> When we perform copyToLocal operations from commandLine and if src Path is invalid 
> srcFS.globStatus(srcpath) will return null. So, when we find the length of resulted value, it will *throw NullPointerException*.
>  Since we are handling generic exception , it will display null as the message.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13702-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 23 18:59:12 2011
Return-Path: <common-issues-return-13702-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50423 invoked from network); 23 Mar 2011 18:59:10 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Mar 2011 18:59:10 -0000
Received: (qmail 33800 invoked by uid 500); 23 Mar 2011 18:52:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33778 invoked by uid 500); 23 Mar 2011 18:52:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33770 invoked by uid 99); 23 Mar 2011 18:52:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 18:52:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 18:52:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id ED9454AA25
	for <common-issues@hadoop.apache.org>; Wed, 23 Mar 2011 18:52:06 +0000 (UTC)
Date: Wed, 23 Mar 2011 18:52:06 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1251051248.6803.1300906326969.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010286#comment-13010286 ] 

Allen Wittenauer commented on HADOOP-7206:
------------------------------------------

I'm thinking beyond "just a compression library".  

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Eli Collins
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13699-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 23 19:01:31 2011
Return-Path: <common-issues-return-13699-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62305 invoked from network); 23 Mar 2011 19:01:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Mar 2011 19:01:30 -0000
Received: (qmail 30682 invoked by uid 500); 23 Mar 2011 18:01:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30656 invoked by uid 500); 23 Mar 2011 18:01:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30636 invoked by uid 99); 23 Mar 2011 18:01:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 18:01:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 18:01:44 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 871584A8A8
	for <common-issues@hadoop.apache.org>; Wed, 23 Mar 2011 18:01:06 +0000 (UTC)
Date: Wed, 23 Mar 2011 18:01:06 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1460754890.6680.1300903266550.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010259#comment-13010259 ] 

Allen Wittenauer commented on HADOOP-7206:
------------------------------------------

I don't really care what Cloudera does as a business.  

But Apache's track history on making everything a separate project to release faster has failed.  We'd also be better off if folks would resist the temptation to back port anything and everything. 

We need to draw the line, and it might as well start here.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Eli Collins
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13701-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 23 19:09:33 2011
Return-Path: <common-issues-return-13701-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95822 invoked from network); 23 Mar 2011 19:09:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Mar 2011 19:09:30 -0000
Received: (qmail 50432 invoked by uid 500); 23 Mar 2011 18:09:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50407 invoked by uid 500); 23 Mar 2011 18:09:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50399 invoked by uid 99); 23 Mar 2011 18:09:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 18:09:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 18:09:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B5E204ABB3
	for <common-issues@hadoop.apache.org>; Wed, 23 Mar 2011 18:09:05 +0000 (UTC)
Date: Wed, 23 Mar 2011 18:09:05 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1163516705.6726.1300903745741.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7178) copyToLocal API is creating .crc
 files in local, even after setting verifyChecksum to false at client side.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010266#comment-13010266 ] 

Daryn Sharp commented on HADOOP-7178:
-------------------------------------

I'm not an expert on the filesystem code, but I have a few questions/concerns.

Should the code really be casting to a ChecksumFileSystem w/o first checking instanceof?

Should the default getVerifyChecksum() return instanceof ChecksumFileSystem instead of always true?  That may help prevent changing other filesystem classes which may do not implement setVerifyChecksum.

It looks like more fs classes may need to be changed than just Distributed.  Other filesystem appear to have private booleans for verifyChecksum.  The ChRoot will need to reach into its embedded fs.

> copyToLocal API is creating .crc files in local, even after setting verifyChecksum to false at client side.
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7178
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7178
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7178_COMMON.patch, HADOOP-7178_HDFS.patch
>
>
> When we copy the files from DFS to local, it is creating the .crc file in local filesystem for the verification of checksum even if we disable the checksum verification at client side.
>         When user does not want to do any checksum verification, then what will be the use in creating of these files in local file system.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13698-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 23 19:19:29 2011
Return-Path: <common-issues-return-13698-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17793 invoked from network); 23 Mar 2011 19:19:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Mar 2011 19:19:26 -0000
Received: (qmail 88804 invoked by uid 500); 23 Mar 2011 17:32:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88763 invoked by uid 500); 23 Mar 2011 17:32:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88755 invoked by uid 99); 23 Mar 2011 17:32:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 17:32:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 17:32:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6108849855
	for <common-issues@hadoop.apache.org>; Wed, 23 Mar 2011 17:32:06 +0000 (UTC)
Date: Wed, 23 Mar 2011 17:32:06 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1157601979.6610.1300901526394.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010244#comment-13010244 ] 

Todd Lipcon commented on HADOOP-7206:
-------------------------------------

Hey Allen. When there are real customers (either external customers in my case or internal customers at a lot of the bigger companies using Hadoop) who need a feature, it's not always possible to say no.

Given the choice between putting it in core (and most likely having major users and distros backport) vs putting it in an external library, I prefer the external library. I don't follow your reasoning that putting it in core will speed up an Apache release -- when has adding new code to a project ever made its release cycle faster?

FWIW I have expressed this same opinion even on projects like HBase where releases are frequent and people run trees that are very close to the Apache bits. Keeping projects small makes releases easier,     not harder, and frequent releases means fewer backports.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Eli Collins
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13700-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 23 19:50:29 2011
Return-Path: <common-issues-return-13700-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52388 invoked from network); 23 Mar 2011 19:50:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Mar 2011 19:50:26 -0000
Received: (qmail 36133 invoked by uid 500); 23 Mar 2011 18:03:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36079 invoked by uid 500); 23 Mar 2011 18:03:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36071 invoked by uid 99); 23 Mar 2011 18:03:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 18:03:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 18:03:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6D39E4A9C9
	for <common-issues@hadoop.apache.org>; Wed, 23 Mar 2011 18:03:06 +0000 (UTC)
Date: Wed, 23 Mar 2011 18:03:06 +0000 (UTC)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1332029156.6713.1300903386444.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010260#comment-13010260 ] 

Aaron Kimball commented on HADOOP-7206:
---------------------------------------

+1 on modular separation. Forcing folks to upgrade an entire cluster to allow them to use a compression library in their client MR programs seems like poor design. Compression is pluggable in Hadoop for a reason.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Eli Collins
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13697-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 23 20:03:24 2011
Return-Path: <common-issues-return-13697-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82143 invoked from network); 23 Mar 2011 20:03:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Mar 2011 20:03:23 -0000
Received: (qmail 73176 invoked by uid 500); 23 Mar 2011 17:16:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73148 invoked by uid 500); 23 Mar 2011 17:16:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73140 invoked by uid 99); 23 Mar 2011 17:16:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 17:16:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 17:16:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 01ABA4662A
	for <common-issues@hadoop.apache.org>; Wed, 23 Mar 2011 17:16:06 +0000 (UTC)
Date: Wed, 23 Mar 2011 17:16:06 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1445766659.6553.1300900566003.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010230#comment-13010230 ] 

Allen Wittenauer commented on HADOOP-7206:
------------------------------------------

The disadvantage is that it is yet another reason not to get an Apache release out.  After all, if all the 'good bits' have been back ported, why move forward?

So no, please don't break it out, please don't back port.  Or is the goal to see if we can make it to three years before we get a new, solid release out the door?

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Eli Collins
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13703-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 23 21:05:47 2011
Return-Path: <common-issues-return-13703-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18486 invoked from network); 23 Mar 2011 21:05:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Mar 2011 21:05:46 -0000
Received: (qmail 73269 invoked by uid 500); 23 Mar 2011 21:05:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73239 invoked by uid 500); 23 Mar 2011 21:05:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73230 invoked by uid 99); 23 Mar 2011 21:05:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 21:05:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 21:05:44 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 857994A8F2
	for <common-issues@hadoop.apache.org>; Wed, 23 Mar 2011 21:05:06 +0000 (UTC)
Date: Wed, 23 Mar 2011 21:05:06 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1780116841.7285.1300914306543.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1627211432.4057.1299578279558.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7174) null is displayed in the console,if
 the src path is invalid while doing copyToLocal operation from commandLine
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7174:
-------------------------------------------

    Component/s: fs

> null is displayed in the console,if the src path is invalid while doing copyToLocal operation from commandLine
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7174
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7174
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>            Priority: Minor
>         Attachments: HADOOP-7174.patch
>
>
> When we perform copyToLocal operations from commandLine and if src Path is invalid 
> srcFS.globStatus(srcpath) will return null. So, when we find the length of resulted value, it will *throw NullPointerException*.
>  Since we are handling generic exception , it will display null as the message.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13704-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 23 21:16:46 2011
Return-Path: <common-issues-return-13704-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25444 invoked from network); 23 Mar 2011 21:16:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Mar 2011 21:16:45 -0000
Received: (qmail 90053 invoked by uid 500); 23 Mar 2011 21:16:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90022 invoked by uid 500); 23 Mar 2011 21:16:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90014 invoked by uid 99); 23 Mar 2011 21:16:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 21:16:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 21:16:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C14A54ACAD
	for <common-issues@hadoop.apache.org>; Wed, 23 Mar 2011 21:16:05 +0000 (UTC)
Date: Wed, 23 Mar 2011 21:16:05 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <486040137.7300.1300914965788.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1627534738.611.1300110389998.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7187) Socket Leak in
 org.apache.hadoop.metrics.ganglia.GangliaContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010450#comment-13010450 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7187:
------------------------------------------------

Good catch.
- Could you add {{@Override}} to {{close()}}?
- Please use junit 4 (i.e. {{org.junit.Test}} and other classes org.junit.* instead of {{junit.framework.TestCase}})

> Socket Leak in org.apache.hadoop.metrics.ganglia.GangliaContext
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7187
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7187
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7187.patch
>
>
> Init method is creating DatagramSocket. But this is not closed any where. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13705-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 23 21:28:44 2011
Return-Path: <common-issues-return-13705-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66000 invoked from network); 23 Mar 2011 21:28:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Mar 2011 21:28:44 -0000
Received: (qmail 7143 invoked by uid 500); 23 Mar 2011 21:28:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7106 invoked by uid 500); 23 Mar 2011 21:28:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7098 invoked by uid 99); 23 Mar 2011 21:28:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 21:28:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 21:28:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7555C4A0F7
	for <common-issues@hadoop.apache.org>; Wed, 23 Mar 2011 21:28:06 +0000 (UTC)
Date: Wed, 23 Mar 2011 21:28:06 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1489325550.7321.1300915686477.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1055815913.6001.1300253369611.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7193) Help message is wrong for touchz
 command.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7193:
-------------------------------------------

        Assignee: Uma Maheswara Rao G
    Hadoop Flags: [Reviewed]

+1 patch looks good.

> Help message is wrong for touchz command.
> -----------------------------------------
>
>                 Key: HADOOP-7193
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7193
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>            Priority: Minor
>         Attachments: HADOOP-7193.patch, HADOOP-7193.patch
>
>
> Help message for touchz command is
>  -touchz <path>: Write a timestamp in yyyy-MM-dd HH:mm:ss format
>                 in a file at <path>. An error is returned if the file exists with non-zero length.
>  Actually current DFS behaviour is that it will not write any time stamp in created file. Just it is creating zero size file.
> So better to change the help message to give exact meaning.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13706-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 23 21:36:47 2011
Return-Path: <common-issues-return-13706-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13300 invoked from network); 23 Mar 2011 21:36:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Mar 2011 21:36:43 -0000
Received: (qmail 20880 invoked by uid 500); 23 Mar 2011 21:36:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20811 invoked by uid 500); 23 Mar 2011 21:36:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20724 invoked by uid 99); 23 Mar 2011 21:36:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 21:36:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 21:36:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E78C74A37B
	for <common-issues@hadoop.apache.org>; Wed, 23 Mar 2011 21:36:05 +0000 (UTC)
Date: Wed, 23 Mar 2011 21:36:05 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1898222852.7344.1300916165945.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6962) FileSystem.mkdirs(Path,
 FSPermission) should use the permission for all of the created directories
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010463#comment-13010463 ] 

Daryn Sharp commented on HADOOP-6962:
-------------------------------------

Problem appears to be deeper than the related bugs.  The namenode appears to be the root cause of the incorrect permissions.  The client code is trying to spackle over the issue by changing the final directory's permissions to what they should be.

> FileSystem.mkdirs(Path, FSPermission) should use the permission for all of the created directories
> --------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6962
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6962
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Owen O'Malley
>            Assignee: Daryn Sharp
>
> Currently, FileSystem.mkdirs only applies the permissions to the last level if it was created. It should be applied to *all* levels that are created.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13707-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 23 21:46:43 2011
Return-Path: <common-issues-return-13707-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32770 invoked from network); 23 Mar 2011 21:46:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Mar 2011 21:46:43 -0000
Received: (qmail 32248 invoked by uid 500); 23 Mar 2011 21:46:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32075 invoked by uid 500); 23 Mar 2011 21:46:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32067 invoked by uid 99); 23 Mar 2011 21:46:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 21:46:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 21:46:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CC0034A7D5
	for <common-issues@hadoop.apache.org>; Wed, 23 Mar 2011 21:46:05 +0000 (UTC)
Date: Wed, 23 Mar 2011 21:46:05 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <932561389.7376.1300916765832.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1055815913.6001.1300253369611.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7193) Help message is wrong for touchz
 command.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010470#comment-13010470 ] 

Hadoop QA commented on HADOOP-7193:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12473984/HADOOP-7193.patch
  against trunk revision 1084415.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/317//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/317//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/317//console

This message is automatically generated.

> Help message is wrong for touchz command.
> -----------------------------------------
>
>                 Key: HADOOP-7193
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7193
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>            Priority: Minor
>         Attachments: HADOOP-7193.patch, HADOOP-7193.patch
>
>
> Help message for touchz command is
>  -touchz <path>: Write a timestamp in yyyy-MM-dd HH:mm:ss format
>                 in a file at <path>. An error is returned if the file exists with non-zero length.
>  Actually current DFS behaviour is that it will not write any time stamp in created file. Just it is creating zero size file.
> So better to change the help message to give exact meaning.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13708-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 23 22:03:43 2011
Return-Path: <common-issues-return-13708-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98962 invoked from network); 23 Mar 2011 22:03:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Mar 2011 22:03:43 -0000
Received: (qmail 53727 invoked by uid 500); 23 Mar 2011 22:03:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53691 invoked by uid 500); 23 Mar 2011 22:03:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53683 invoked by uid 99); 23 Mar 2011 22:03:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 22:03:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 22:03:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B59774ACDC
	for <common-issues@hadoop.apache.org>; Wed, 23 Mar 2011 22:03:05 +0000 (UTC)
Date: Wed, 23 Mar 2011 22:03:05 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1189120574.7407.1300917785740.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1055815913.6001.1300253369611.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7193) Help message is wrong for touchz
 command.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7193:
-------------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
                   0.22.0
                   0.21.1
           Status: Resolved  (was: Patch Available)

> Please justify why no new tests are needed for this patch.
This only fixes an help message.

I have committed this.  Thanks, Uma!

> Help message is wrong for touchz command.
> -----------------------------------------
>
>                 Key: HADOOP-7193
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7193
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>            Priority: Minor
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HADOOP-7193.patch, HADOOP-7193.patch
>
>
> Help message for touchz command is
>  -touchz <path>: Write a timestamp in yyyy-MM-dd HH:mm:ss format
>                 in a file at <path>. An error is returned if the file exists with non-zero length.
>  Actually current DFS behaviour is that it will not write any time stamp in created file. Just it is creating zero size file.
> So better to change the help message to give exact meaning.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13709-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 23 22:15:46 2011
Return-Path: <common-issues-return-13709-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20111 invoked from network); 23 Mar 2011 22:15:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 23 Mar 2011 22:15:46 -0000
Received: (qmail 67278 invoked by uid 500); 23 Mar 2011 22:15:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67247 invoked by uid 500); 23 Mar 2011 22:15:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67239 invoked by uid 99); 23 Mar 2011 22:15:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 22:15:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Mar 2011 22:15:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 807F14A099
	for <common-issues@hadoop.apache.org>; Wed, 23 Mar 2011 22:15:06 +0000 (UTC)
Date: Wed, 23 Mar 2011 22:15:06 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2131705302.7444.1300918506523.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1055815913.6001.1300253369611.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7193) Help message is wrong for touchz
 command.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010484#comment-13010484 ] 

Hudson commented on HADOOP-7193:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #533 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/533/])
    HADOOP-7193. Correct the "fs -touchz" command help message.  Contributed by Uma Maheswara Rao G


> Help message is wrong for touchz command.
> -----------------------------------------
>
>                 Key: HADOOP-7193
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7193
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>            Priority: Minor
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HADOOP-7193.patch, HADOOP-7193.patch
>
>
> Help message for touchz command is
>  -touchz <path>: Write a timestamp in yyyy-MM-dd HH:mm:ss format
>                 in a file at <path>. An error is returned if the file exists with non-zero length.
>  Actually current DFS behaviour is that it will not write any time stamp in created file. Just it is creating zero size file.
> So better to change the help message to give exact meaning.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13710-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 06:10:50 2011
Return-Path: <common-issues-return-13710-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40699 invoked from network); 24 Mar 2011 06:10:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 06:10:49 -0000
Received: (qmail 74987 invoked by uid 500); 24 Mar 2011 06:10:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74367 invoked by uid 500); 24 Mar 2011 06:10:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74254 invoked by uid 99); 24 Mar 2011 06:10:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 06:10:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 06:10:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1BC684BDA2
	for <common-issues@hadoop.apache.org>; Thu, 24 Mar 2011 06:10:06 +0000 (UTC)
Date: Thu, 24 Mar 2011 06:10:06 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1492555004.7898.1300947006110.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7178) copyToLocal API is creating .crc
 files in local, even after setting verifyChecksum to false at client side.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010569#comment-13010569 ] 

Uma Maheswara Rao G commented on HADOOP-7178:
---------------------------------------------

Hi Daryn Sharp,
Thanks for your comments,

{noformat} 
Should the code really be casting to a ChecksumFileSystem w/o first checking instanceof?
{noformat} 
  For CopyToLocalFile api, we will select LocalFilesystem as local file system. Here gteLocal() api used to get that local file system.This API  will always return LocalFileSystem which extends CheckSumFileFystem
  So, instanceOf check not required.

{noformat} 
Should the default getVerifyChecksum() return instanceof ChecksumFileSystem instead of always true? That may help prevent changing other filesystem classes which may do not implement setVerifyChecksum.
{noformat}   
  Here By deafult local file system will be LocalFileSystem. This will create crc files. So, I put default as true. If user wanted to disable it, he will provide the implementation for setVerifyChecksum() and getVerifyChecksum() like DistributedFileSystem. 

{noformat}  
It looks like more fs classes may need to be changed than just Distributed. Other filesystem appear to have private booleans for verifyChecksum. 
{noformat}  
  Here we are not forcing implementation classes to implement this methods. If Concrete classes does not want to manage this checkSumVerification flag then default implemenation will work.


> copyToLocal API is creating .crc files in local, even after setting verifyChecksum to false at client side.
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7178
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7178
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7178_COMMON.patch, HADOOP-7178_HDFS.patch
>
>
> When we copy the files from DFS to local, it is creating the .crc file in local filesystem for the verification of checksum even if we disable the checksum verification at client side.
>         When user does not want to do any checksum verification, then what will be the use in creating of these files in local file system.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13711-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 07:16:52 2011
Return-Path: <common-issues-return-13711-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43966 invoked from network); 24 Mar 2011 07:16:52 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 07:16:52 -0000
Received: (qmail 20707 invoked by uid 500); 24 Mar 2011 07:16:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20638 invoked by uid 500); 24 Mar 2011 07:16:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20630 invoked by uid 99); 24 Mar 2011 07:16:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 07:16:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 07:16:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E347C4BD19
	for <common-issues@hadoop.apache.org>; Thu, 24 Mar 2011 07:16:05 +0000 (UTC)
Date: Thu, 24 Mar 2011 07:16:05 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2089562312.7962.1300950965927.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1195662636.6130.1300891805766.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7208) equals() and hashCode()
 implementation need to change in StandardSocketFactory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uma Maheswara Rao G reassigned HADOOP-7208:
-------------------------------------------

    Assignee: Uma Maheswara Rao G

> equals() and hashCode() implementation need to change in StandardSocketFactory
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7208
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7208
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>
>   In Hadoop IPC Client, we are using ClientCache which will maintain the HashMap to keep the Client references.
> private Map<SocketFactory, Client> clients =
>       new HashMap<SocketFactory, Client>();
>  Now let us say, we want use two standard factories with Hadoop. MyStandardSocketFactory (which extends StandardSocketFactory), and StandardSocketFactory. In this case, because of equals and hashcode implementation, MyStandardSocketFactory client can be overridden by StandardSocketFactoryClient

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13712-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 09:58:46 2011
Return-Path: <common-issues-return-13712-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50377 invoked from network); 24 Mar 2011 09:58:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 09:58:46 -0000
Received: (qmail 69015 invoked by uid 500); 24 Mar 2011 09:58:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68989 invoked by uid 500); 24 Mar 2011 09:58:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68976 invoked by uid 99); 24 Mar 2011 09:58:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 09:58:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 09:58:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F2CF14BCFB
	for <common-issues@hadoop.apache.org>; Thu, 24 Mar 2011 09:58:05 +0000 (UTC)
Date: Thu, 24 Mar 2011 09:58:05 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1387967535.8110.1300960685991.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1055815913.6001.1300253369611.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7193) Help message is wrong for touchz
 command.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010617#comment-13010617 ] 

Hudson commented on HADOOP-7193:
--------------------------------

Integrated in Hadoop-Common-22-branch #35 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-22-branch/35/])
    HADOOP-7193. Correct the "fs -touchz" command help message.  Contributed by Uma Maheswara Rao G


> Help message is wrong for touchz command.
> -----------------------------------------
>
>                 Key: HADOOP-7193
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7193
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>            Priority: Minor
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HADOOP-7193.patch, HADOOP-7193.patch
>
>
> Help message for touchz command is
>  -touchz <path>: Write a timestamp in yyyy-MM-dd HH:mm:ss format
>                 in a file at <path>. An error is returned if the file exists with non-zero length.
>  Actually current DFS behaviour is that it will not write any time stamp in created file. Just it is creating zero size file.
> So better to change the help message to give exact meaning.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13713-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 14:35:44 2011
Return-Path: <common-issues-return-13713-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51283 invoked from network); 24 Mar 2011 14:35:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 14:35:44 -0000
Received: (qmail 70070 invoked by uid 500); 24 Mar 2011 14:35:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70037 invoked by uid 500); 24 Mar 2011 14:35:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70029 invoked by uid 99); 24 Mar 2011 14:35:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 14:35:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 14:35:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C95FB4BA3D
	for <common-issues@hadoop.apache.org>; Thu, 24 Mar 2011 14:35:05 +0000 (UTC)
Date: Thu, 24 Mar 2011 14:35:05 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <272403489.8414.1300977305821.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1627211432.4057.1299578279558.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7174) null is displayed in the console,if
 the src path is invalid while doing copyToLocal operation from commandLine
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uma Maheswara Rao G updated HADOOP-7174:
----------------------------------------

    Attachment: HADOOP-7174.1.patch

> null is displayed in the console,if the src path is invalid while doing copyToLocal operation from commandLine
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7174
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7174
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>            Priority: Minor
>         Attachments: HADOOP-7174.1.patch, HADOOP-7174.patch
>
>
> When we perform copyToLocal operations from commandLine and if src Path is invalid 
> srcFS.globStatus(srcpath) will return null. So, when we find the length of resulted value, it will *throw NullPointerException*.
>  Since we are handling generic exception , it will display null as the message.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13714-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 14:35:46 2011
Return-Path: <common-issues-return-13714-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51525 invoked from network); 24 Mar 2011 14:35:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 14:35:46 -0000
Received: (qmail 70282 invoked by uid 500); 24 Mar 2011 14:35:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70251 invoked by uid 500); 24 Mar 2011 14:35:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70243 invoked by uid 99); 24 Mar 2011 14:35:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 14:35:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 14:35:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EC5AF4BA43
	for <common-issues@hadoop.apache.org>; Thu, 24 Mar 2011 14:35:05 +0000 (UTC)
Date: Thu, 24 Mar 2011 14:35:05 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <635582632.8419.1300977305964.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1627534738.611.1300110389998.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7187) Socket Leak in
 org.apache.hadoop.metrics.ganglia.GangliaContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uma Maheswara Rao G updated HADOOP-7187:
----------------------------------------

    Attachment: HADOOP-7187.1.patch

> Socket Leak in org.apache.hadoop.metrics.ganglia.GangliaContext
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7187
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7187
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7187.1.patch, HADOOP-7187.patch
>
>
> Init method is creating DatagramSocket. But this is not closed any where. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13715-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 14:41:46 2011
Return-Path: <common-issues-return-13715-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59859 invoked from network); 24 Mar 2011 14:41:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 14:41:45 -0000
Received: (qmail 79785 invoked by uid 500); 24 Mar 2011 14:41:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79738 invoked by uid 500); 24 Mar 2011 14:41:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79730 invoked by uid 99); 24 Mar 2011 14:41:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 14:41:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 14:41:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B34844BD44
	for <common-issues@hadoop.apache.org>; Thu, 24 Mar 2011 14:41:05 +0000 (UTC)
Date: Thu, 24 Mar 2011 14:41:05 +0000 (UTC)
From: "Devaraj K (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1841161202.8442.1300977665731.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <113773164.6166.1300269149586.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7194) Potential Resource leak in
 IOUtils.java
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj K updated HADOOP-7194:
------------------------------

    Attachment: HADOOP-7194.patch

> Potential Resource leak in IOUtils.java
> ---------------------------------------
>
>                 Key: HADOOP-7194
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7194
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.23.0
>            Reporter: Devaraj K
>         Attachments: HADOOP-7194.patch
>
>
> {code:title=IOUtils.java|borderStyle=solid}
> try {
>       copyBytes(in, out, buffSize);
>     } finally {
>       if(close) {
>         out.close();
>         in.close();
>       }
>     }
> {code} 
> In the above code if any exception throws from the out.close() statement, in.close() statement will not execute and the input stream will not be closed.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13716-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 15:06:44 2011
Return-Path: <common-issues-return-13716-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62181 invoked from network); 24 Mar 2011 15:06:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 15:06:44 -0000
Received: (qmail 40148 invoked by uid 500); 24 Mar 2011 15:06:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40033 invoked by uid 500); 24 Mar 2011 15:06:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40025 invoked by uid 99); 24 Mar 2011 15:06:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 15:06:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 15:06:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5A19B4B6BC
	for <common-issues@hadoop.apache.org>; Thu, 24 Mar 2011 15:06:06 +0000 (UTC)
Date: Thu, 24 Mar 2011 15:06:06 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1110076390.8468.1300979166365.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1627211432.4057.1299578279558.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7174) null is displayed in the
 console,if the src path is invalid while doing copyToLocal operation from
 commandLine
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010695#comment-13010695 ] 

Daryn Sharp commented on HADOOP-7174:
-------------------------------------

+1

> null is displayed in the console,if the src path is invalid while doing copyToLocal operation from commandLine
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7174
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7174
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>            Priority: Minor
>         Attachments: HADOOP-7174.1.patch, HADOOP-7174.patch
>
>
> When we perform copyToLocal operations from commandLine and if src Path is invalid 
> srcFS.globStatus(srcpath) will return null. So, when we find the length of resulted value, it will *throw NullPointerException*.
>  Since we are handling generic exception , it will display null as the message.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13717-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 15:28:45 2011
Return-Path: <common-issues-return-13717-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12430 invoked from network); 24 Mar 2011 15:28:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 15:28:44 -0000
Received: (qmail 91671 invoked by uid 500); 24 Mar 2011 15:28:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91629 invoked by uid 500); 24 Mar 2011 15:28:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91420 invoked by uid 99); 24 Mar 2011 15:28:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 15:28:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 15:28:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C856D4B233
	for <common-issues@hadoop.apache.org>; Thu, 24 Mar 2011 15:28:05 +0000 (UTC)
Date: Thu, 24 Mar 2011 15:28:05 +0000 (UTC)
From: "Rajiv Chittajallu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <900780605.8505.1300980485817.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-5983) Namenode shouldn't read
 mapred-site.xml
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-5983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010707#comment-13010707 ] 

Rajiv Chittajallu commented on HADOOP-5983:
-------------------------------------------

It probably might have been fixed in trunk. 

I still see this on our Y! internal 0.20.2xx branch. But this has been existed for a while and should be the same for a Apache Hadoop 0.20 release.

$ xmllint /conf/hadoop/namenode/mapred-site.xml

/conf/hadoop/namenode/mapred-site.xml:615: parser error : Opening and ending tag mismatch: foobar line 12 and configuration
</configuration>
                ^
/grid/0/gs/conf/hadoop/namenode/mapred-site.xml:616: parser error : Premature end of data in tag configuration line 6


..

2011-03-24 15:16:26,413 FATAL org.apache.hadoop.conf.Configuration: error parsing conf file: org.xml.sax.SAXParseException: The element type "foobar" must be terminated by the matching end-tag "</foobar>".
2011-03-24 15:16:26,413 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.lang.RuntimeException: org.xml.sax.SAXParseException: The element type "foobar" must be terminated by the matching end-tag "</foobar>".
	at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1236)
	at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1092)
	at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1036)
	at org.apache.hadoop.conf.Configuration.get(Configuration.java:414)
	at org.apache.hadoop.conf.Configuration.getLong(Configuration.java:519)
	at org.apache.hadoop.security.Groups.<init>(Groups.java:55)
	at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:137)
	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:180)
	at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:206)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:241)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:434)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1153)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1162)
Caused by: org.xml.sax.SAXParseException: The element type "foobar" must be terminated by the matching end-tag "</foobar>".
	at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:239)
	at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:283)
	at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:180)
	at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1141)
	... 12 more

2011-03-24 15:16:26,414 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: 


> Namenode shouldn't read mapred-site.xml
> ---------------------------------------
>
>                 Key: HADOOP-5983
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5983
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.20.0
>            Reporter: Rajiv Chittajallu
>            Assignee: Daryn Sharp
>
> The name node seem to read mapred-site.xml and fails if it can't parse it.
> 2009-06-05 22:37:15,663 FATAL org.apache.hadoop.conf.Configuration: error parsing conf file: org.xml.sax.SAXParseException: Error attempting to parse XML file (href='/hadoop/conf/local/local-mapred-site.xml').
> 2009-06-05 22:37:15,664 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.lang.RuntimeException: org.xml.sax.SAXParseException: Error attempting to parse XML file (href='/hadoop/conf/local/local-mapred-site.xml').
> In our config,  local-mapred-site.xml is included only in mapred-site.xml which we don't push to the namenode.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13718-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 15:37:43 2011
Return-Path: <common-issues-return-13718-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56834 invoked from network); 24 Mar 2011 15:37:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 15:37:43 -0000
Received: (qmail 9303 invoked by uid 500); 24 Mar 2011 15:37:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9139 invoked by uid 500); 24 Mar 2011 15:37:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9131 invoked by uid 99); 24 Mar 2011 15:37:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 15:37:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 15:37:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D1A054B6C4
	for <common-issues@hadoop.apache.org>; Thu, 24 Mar 2011 15:37:05 +0000 (UTC)
Date: Thu, 24 Mar 2011 15:37:05 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1820538773.8532.1300981025855.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7178) copyToLocal API is creating .crc
 files in local, even after setting verifyChecksum to false at client side.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010713#comment-13010713 ] 

Daryn Sharp commented on HADOOP-7178:
-------------------------------------

The getlocal method is returning a LocalFileSystem, so if "fs" is declared as such, could the cast be removed?

Regarding the getVerifyChecksum(), my concern is it's semantically wrong for all filesystems to claim (by default) that checksum files are supported.  The default should be the truth.  Granted, the implementation is backwards compatible, but future callers may operation on false assumptions of crc files.

Would it be better to default as false in FileSystem, and ChecksumFileSystem overrides for true?  I think this would also retain backwards compatibility?

> copyToLocal API is creating .crc files in local, even after setting verifyChecksum to false at client side.
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7178
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7178
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7178_COMMON.patch, HADOOP-7178_HDFS.patch
>
>
> When we copy the files from DFS to local, it is creating the .crc file in local filesystem for the verification of checksum even if we disable the checksum verification at client side.
>         When user does not want to do any checksum verification, then what will be the use in creating of these files in local file system.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13719-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 15:59:44 2011
Return-Path: <common-issues-return-13719-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4815 invoked from network); 24 Mar 2011 15:59:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 15:59:43 -0000
Received: (qmail 54060 invoked by uid 500); 24 Mar 2011 15:59:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53999 invoked by uid 500); 24 Mar 2011 15:59:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53982 invoked by uid 99); 24 Mar 2011 15:59:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 15:59:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 15:59:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CE5E34B2EF
	for <common-issues@hadoop.apache.org>; Thu, 24 Mar 2011 15:59:05 +0000 (UTC)
Date: Thu, 24 Mar 2011 15:59:05 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2100584279.8591.1300982345841.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-5983) Namenode shouldn't read
 mapred-site.xml
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-5983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010729#comment-13010729 ] 

Daryn Sharp commented on HADOOP-5983:
-------------------------------------

Are you requesting for the bug to be reopened so it can be fixed on an older release?  

An aside: while this is likely a bug since the namenode shouldn't need the mapred conf, it seems odd to push an unneeded file to the namenode, but then not push it's dependencies.  It seems like it should be all-or-nothing operation.  Ie. Push both or neither?

> Namenode shouldn't read mapred-site.xml
> ---------------------------------------
>
>                 Key: HADOOP-5983
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5983
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.20.0
>            Reporter: Rajiv Chittajallu
>            Assignee: Daryn Sharp
>
> The name node seem to read mapred-site.xml and fails if it can't parse it.
> 2009-06-05 22:37:15,663 FATAL org.apache.hadoop.conf.Configuration: error parsing conf file: org.xml.sax.SAXParseException: Error attempting to parse XML file (href='/hadoop/conf/local/local-mapred-site.xml').
> 2009-06-05 22:37:15,664 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.lang.RuntimeException: org.xml.sax.SAXParseException: Error attempting to parse XML file (href='/hadoop/conf/local/local-mapred-site.xml').
> In our config,  local-mapred-site.xml is included only in mapred-site.xml which we don't push to the namenode.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13720-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 16:34:43 2011
Return-Path: <common-issues-return-13720-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44040 invoked from network); 24 Mar 2011 16:34:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 16:34:43 -0000
Received: (qmail 29133 invoked by uid 500); 24 Mar 2011 16:34:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29100 invoked by uid 500); 24 Mar 2011 16:34:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29092 invoked by uid 99); 24 Mar 2011 16:34:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 16:34:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 16:34:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E20D14B691
	for <common-issues@hadoop.apache.org>; Thu, 24 Mar 2011 16:34:05 +0000 (UTC)
Date: Thu, 24 Mar 2011 16:34:05 +0000 (UTC)
From: "Devaraj K (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <856448409.8711.1300984445922.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <113773164.6166.1300269149586.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7194) Potential Resource leak in
 IOUtils.java
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj K updated HADOOP-7194:
------------------------------

    Status: Patch Available  (was: Open)

> Potential Resource leak in IOUtils.java
> ---------------------------------------
>
>                 Key: HADOOP-7194
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7194
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.23.0
>            Reporter: Devaraj K
>         Attachments: HADOOP-7194.patch
>
>
> {code:title=IOUtils.java|borderStyle=solid}
> try {
>       copyBytes(in, out, buffSize);
>     } finally {
>       if(close) {
>         out.close();
>         in.close();
>       }
>     }
> {code} 
> In the above code if any exception throws from the out.close() statement, in.close() statement will not execute and the input stream will not be closed.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13721-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 17:15:43 2011
Return-Path: <common-issues-return-13721-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56436 invoked from network); 24 Mar 2011 17:15:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 17:15:43 -0000
Received: (qmail 91191 invoked by uid 500); 24 Mar 2011 17:15:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91159 invoked by uid 500); 24 Mar 2011 17:15:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91151 invoked by uid 99); 24 Mar 2011 17:15:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 17:15:43 +0000
X-ASF-Spam-Status: No, hits=-1999.7 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD,URIBL_RHS_DOB
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 17:15:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C73184C788
	for <common-issues@hadoop.apache.org>; Thu, 24 Mar 2011 17:15:05 +0000 (UTC)
Date: Thu, 24 Mar 2011 17:15:05 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1774131023.8794.1300986905812.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1587140683.1515.1300732271817.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7202) Improve Command base class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010771#comment-13010771 ] 

Tanping Wang commented on HADOOP-7202:
--------------------------------------

I looked at the patch and think it looks good to me with only a few comments,

1) In PathData.java, class member exists which is a Boolean variable,  its value is determined by checking if FileStatus is null. Since FileStatus is another class member,  by checking whether FileStatus is null, we can already know if the Path exists.  So I think we can eliminate this "exist" member variable?

2) In Command.java#displayError(Exception e)
{code}
    if (errorMessage == null) {
      errorMessage = e.toString();
    }
{code}

StringUtils.stringifyException() is a common utility function which makes a string representation of the exception.  I think using this is better than just toString()
Also,  it would be a good idea to log a debug message using LOG.debug to print out the full exception stack trace, so that debugging is easier.

3) In Command.java#run()
{code}
   * @throws IllegalArgumentException if the option parsing fails
{code}

would it be a good idea to link the function, so it is more clear? i.e.

   * @throws IllegalArgumentException if {@link processOptions(args)} fails

I am confused about the other throws comment
{code}
   * @throws non-IOExceptions such rpc failures
{code}

4) In Command.java#run(String, cmd, LinkedList<String> args)
the exit code of error was -1 in the original code, it is now changed to 1.  Although it does not matter as long as it is non-zero value, why not keep it the same?

5) You are making the methods in PathData.java public, just making it package default by removing the "public" should be sufficient enough in our case?

> Improve Command base class
> --------------------------
>
>                 Key: HADOOP-7202
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7202
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7202.patch
>
>
> Need to extend the Command base class to allow all command to easily subclass from a code set of code that correctly handles globs and exit codes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13722-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 17:17:43 2011
Return-Path: <common-issues-return-13722-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63247 invoked from network); 24 Mar 2011 17:17:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 17:17:43 -0000
Received: (qmail 95054 invoked by uid 500); 24 Mar 2011 17:17:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95012 invoked by uid 500); 24 Mar 2011 17:17:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95004 invoked by uid 99); 24 Mar 2011 17:17:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 17:17:43 +0000
X-ASF-Spam-Status: No, hits=-1999.7 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD,URIBL_RHS_DOB
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 17:17:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CB53B4C86A
	for <common-issues@hadoop.apache.org>; Thu, 24 Mar 2011 17:17:05 +0000 (UTC)
Date: Thu, 24 Mar 2011 17:17:05 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <212888898.8801.1300987025829.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1627211432.4057.1299578279558.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7174) null is displayed in the
 console,if the src path is invalid while doing copyToLocal operation from
 commandLine
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010772#comment-13010772 ] 

Hadoop QA commented on HADOOP-7174:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12474506/HADOOP-7174.1.patch
  against trunk revision 1084769.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 4 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/318//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/318//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/318//console

This message is automatically generated.

> null is displayed in the console,if the src path is invalid while doing copyToLocal operation from commandLine
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7174
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7174
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>            Priority: Minor
>         Attachments: HADOOP-7174.1.patch, HADOOP-7174.patch
>
>
> When we perform copyToLocal operations from commandLine and if src Path is invalid 
> srcFS.globStatus(srcpath) will return null. So, when we find the length of resulted value, it will *throw NullPointerException*.
>  Since we are handling generic exception , it will display null as the message.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13723-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 17:27:43 2011
Return-Path: <common-issues-return-13723-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 412 invoked from network); 24 Mar 2011 17:27:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 17:27:43 -0000
Received: (qmail 13038 invoked by uid 500); 24 Mar 2011 17:27:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12988 invoked by uid 500); 24 Mar 2011 17:27:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12980 invoked by uid 99); 24 Mar 2011 17:27:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 17:27:43 +0000
X-ASF-Spam-Status: No, hits=-1999.7 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD,URIBL_RHS_DOB
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 17:27:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C6D9E4CC80
	for <common-issues@hadoop.apache.org>; Thu, 24 Mar 2011 17:27:05 +0000 (UTC)
Date: Thu, 24 Mar 2011 17:27:05 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1095678097.8836.1300987625809.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1627211432.4057.1299578279558.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7174) null is displayed in the console,if
 the src path is invalid while doing copyToLocal operation from commandLine
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7174:
-------------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
                   0.22.0
                   0.21.1
     Release Note:   (was: Patch provided)
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I have committed this.  Thanks, Uma!

Also, thanks Daryn for reviewing it.

> null is displayed in the console,if the src path is invalid while doing copyToLocal operation from commandLine
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7174
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7174
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>            Priority: Minor
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HADOOP-7174.1.patch, HADOOP-7174.patch
>
>
> When we perform copyToLocal operations from commandLine and if src Path is invalid 
> srcFS.globStatus(srcpath) will return null. So, when we find the length of resulted value, it will *throw NullPointerException*.
>  Since we are handling generic exception , it will display null as the message.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13724-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 17:37:43 2011
Return-Path: <common-issues-return-13724-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45056 invoked from network); 24 Mar 2011 17:37:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 17:37:43 -0000
Received: (qmail 22428 invoked by uid 500); 24 Mar 2011 17:37:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22402 invoked by uid 500); 24 Mar 2011 17:37:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22394 invoked by uid 99); 24 Mar 2011 17:37:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 17:37:43 +0000
X-ASF-Spam-Status: No, hits=-1999.7 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD,URIBL_RHS_DOB
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 17:37:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B115A4C0F2
	for <common-issues@hadoop.apache.org>; Thu, 24 Mar 2011 17:37:05 +0000 (UTC)
Date: Thu, 24 Mar 2011 17:37:05 +0000 (UTC)
From: "Koji Noguchi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1063546936.8872.1300988225722.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1723498609.5040.1300840326140.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7207) fs member of FSShell is not really
 needed
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010787#comment-13010787 ] 

Koji Noguchi commented on HADOOP-7207:
--------------------------------------

Duplicate of HADOOP-5749 ?

> fs member of FSShell is not really needed
> -----------------------------------------
>
>                 Key: HADOOP-7207
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7207
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>
> FileSystem object should be created on demand when needed in FSShell, instead of always creating one that connects to the default FileSystem.
> It will also solve a problem of connecting to non-existant/non-functional default, when it is not really needed for this run.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13725-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 17:43:46 2011
Return-Path: <common-issues-return-13725-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46368 invoked from network); 24 Mar 2011 17:43:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 17:43:45 -0000
Received: (qmail 28385 invoked by uid 500); 24 Mar 2011 17:43:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28355 invoked by uid 500); 24 Mar 2011 17:43:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28347 invoked by uid 99); 24 Mar 2011 17:43:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 17:43:45 +0000
X-ASF-Spam-Status: No, hits=-1999.7 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD,URIBL_RHS_DOB
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 17:43:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0AD584C3B0
	for <common-issues@hadoop.apache.org>; Thu, 24 Mar 2011 17:43:06 +0000 (UTC)
Date: Thu, 24 Mar 2011 17:43:06 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <987716611.8890.1300988586041.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1627211432.4057.1299578279558.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7174) null is displayed in the
 console,if the src path is invalid while doing copyToLocal operation from
 commandLine
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010792#comment-13010792 ] 

Hudson commented on HADOOP-7174:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #534 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/534/])
    HADOOP-7174. Null is displayed in the "fs -copyToLocal" command.  Contributed by Uma Maheswara Rao G


> null is displayed in the console,if the src path is invalid while doing copyToLocal operation from commandLine
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7174
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7174
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>            Priority: Minor
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HADOOP-7174.1.patch, HADOOP-7174.patch
>
>
> When we perform copyToLocal operations from commandLine and if src Path is invalid 
> srcFS.globStatus(srcpath) will return null. So, when we find the length of resulted value, it will *throw NullPointerException*.
>  Since we are handling generic exception , it will display null as the message.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13726-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 18:02:46 2011
Return-Path: <common-issues-return-13726-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16501 invoked from network); 24 Mar 2011 18:02:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 18:02:46 -0000
Received: (qmail 61355 invoked by uid 500); 24 Mar 2011 18:02:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61315 invoked by uid 500); 24 Mar 2011 18:02:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61307 invoked by uid 99); 24 Mar 2011 18:02:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 18:02:45 +0000
X-ASF-Spam-Status: No, hits=-1999.7 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD,URIBL_RHS_DOB
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 18:02:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CAC434CA84
	for <common-issues@hadoop.apache.org>; Thu, 24 Mar 2011 18:02:05 +0000 (UTC)
Date: Thu, 24 Mar 2011 18:02:05 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <585136449.8940.1300989725827.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1627534738.611.1300110389998.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7187) Socket Leak in
 org.apache.hadoop.metrics.ganglia.GangliaContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010804#comment-13010804 ] 

Uma Maheswara Rao G commented on HADOOP-7187:
---------------------------------------------

Hi Nicholas,
 Thanks for your comments,
 I fixed them and submitted the patch.

> Socket Leak in org.apache.hadoop.metrics.ganglia.GangliaContext
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7187
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7187
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7187.1.patch, HADOOP-7187.patch
>
>
> Init method is creating DatagramSocket. But this is not closed any where. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13727-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 18:17:45 2011
Return-Path: <common-issues-return-13727-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55445 invoked from network); 24 Mar 2011 18:17:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 18:17:45 -0000
Received: (qmail 74900 invoked by uid 500); 24 Mar 2011 18:17:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74875 invoked by uid 500); 24 Mar 2011 18:17:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74867 invoked by uid 99); 24 Mar 2011 18:17:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 18:17:45 +0000
X-ASF-Spam-Status: No, hits=-1999.7 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD,URIBL_RHS_DOB
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 18:17:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B2FE24CF63
	for <common-issues@hadoop.apache.org>; Thu, 24 Mar 2011 18:17:05 +0000 (UTC)
Date: Thu, 24 Mar 2011 18:17:05 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1426146929.8972.1300990625729.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1723498609.5040.1300840326140.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7207) fs member of FSShell is not really
 needed
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010813#comment-13010813 ] 

Matt Foley commented on HADOOP-7207:
------------------------------------

Be careful with this one as we move to Federation, because the "-fs" command line option is currently the only way to direct the FSClient at a file system other than the default one, if any.  So if you might want your client to target one of several name services (federated namenodes), you have to use the -fs option or provide an alternative.

> fs member of FSShell is not really needed
> -----------------------------------------
>
>                 Key: HADOOP-7207
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7207
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>
> FileSystem object should be created on demand when needed in FSShell, instead of always creating one that connects to the default FileSystem.
> It will also solve a problem of connecting to non-existant/non-functional default, when it is not really needed for this run.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13728-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 18:33:44 2011
Return-Path: <common-issues-return-13728-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32754 invoked from network); 24 Mar 2011 18:33:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 18:33:43 -0000
Received: (qmail 97283 invoked by uid 500); 24 Mar 2011 18:33:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97251 invoked by uid 500); 24 Mar 2011 18:33:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97242 invoked by uid 99); 24 Mar 2011 18:33:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 18:33:43 +0000
X-ASF-Spam-Status: No, hits=-1999.7 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD,URIBL_RHS_DOB
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 18:33:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DBD744C5FA
	for <common-issues@hadoop.apache.org>; Thu, 24 Mar 2011 18:33:05 +0000 (UTC)
Date: Thu, 24 Mar 2011 18:33:05 +0000 (UTC)
From: "Rajiv Chittajallu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <27426261.9046.1300991585897.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-5983) Namenode shouldn't read
 mapred-site.xml
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-5983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010820#comment-13010820 ] 

Rajiv Chittajallu commented on HADOOP-5983:
-------------------------------------------


This was opened for 0.20 when it was the current release. If its
validated in 0.22, I am ok. 


The dependencies might be more specific to our setup, but since
configuration gets serialized, a user can't have separate parameters for
hdfs and mapred . For eg: fs.default.name when a cluster has multiple
hdfs instances. 



> Namenode shouldn't read mapred-site.xml
> ---------------------------------------
>
>                 Key: HADOOP-5983
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5983
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.20.0
>            Reporter: Rajiv Chittajallu
>            Assignee: Daryn Sharp
>
> The name node seem to read mapred-site.xml and fails if it can't parse it.
> 2009-06-05 22:37:15,663 FATAL org.apache.hadoop.conf.Configuration: error parsing conf file: org.xml.sax.SAXParseException: Error attempting to parse XML file (href='/hadoop/conf/local/local-mapred-site.xml').
> 2009-06-05 22:37:15,664 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.lang.RuntimeException: org.xml.sax.SAXParseException: Error attempting to parse XML file (href='/hadoop/conf/local/local-mapred-site.xml').
> In our config,  local-mapred-site.xml is included only in mapred-site.xml which we don't push to the namenode.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13729-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 19:02:44 2011
Return-Path: <common-issues-return-13729-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17087 invoked from network); 24 Mar 2011 19:02:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 19:02:44 -0000
Received: (qmail 44677 invoked by uid 500); 24 Mar 2011 19:02:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44656 invoked by uid 500); 24 Mar 2011 19:02:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44648 invoked by uid 99); 24 Mar 2011 19:02:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 19:02:44 +0000
X-ASF-Spam-Status: No, hits=-1999.7 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD,URIBL_RHS_DOB
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 19:02:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E7F784C264
	for <common-issues@hadoop.apache.org>; Thu, 24 Mar 2011 19:02:05 +0000 (UTC)
Date: Thu, 24 Mar 2011 19:02:05 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1469732027.9196.1300993325946.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1627534738.611.1300110389998.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7187) Socket Leak in
 org.apache.hadoop.metrics.ganglia.GangliaContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7187:
-------------------------------------------

    Hadoop Flags: [Reviewed]

+1 patch looks good.

> Socket Leak in org.apache.hadoop.metrics.ganglia.GangliaContext
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7187
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7187
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7187.1.patch, HADOOP-7187.patch
>
>
> Init method is creating DatagramSocket. But this is not closed any where. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13730-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 19:18:46 2011
Return-Path: <common-issues-return-13730-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61689 invoked from network); 24 Mar 2011 19:18:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 19:18:45 -0000
Received: (qmail 69670 invoked by uid 500); 24 Mar 2011 19:18:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69626 invoked by uid 500); 24 Mar 2011 19:18:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69618 invoked by uid 99); 24 Mar 2011 19:18:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 19:18:45 +0000
X-ASF-Spam-Status: No, hits=-1999.7 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD,URIBL_RHS_DOB
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 19:18:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D731D4C799
	for <common-issues@hadoop.apache.org>; Thu, 24 Mar 2011 19:18:05 +0000 (UTC)
Date: Thu, 24 Mar 2011 19:18:05 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1979847742.9227.1300994285877.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1627534738.611.1300110389998.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7187) Socket Leak in
 org.apache.hadoop.metrics.ganglia.GangliaContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010852#comment-13010852 ] 

Hadoop QA commented on HADOOP-7187:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12474507/HADOOP-7187.1.patch
  against trunk revision 1085043.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/319//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/319//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/319//console

This message is automatically generated.

> Socket Leak in org.apache.hadoop.metrics.ganglia.GangliaContext
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7187
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7187
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7187.1.patch, HADOOP-7187.patch
>
>
> Init method is creating DatagramSocket. But this is not closed any where. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13731-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 20:45:46 2011
Return-Path: <common-issues-return-13731-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41191 invoked from network); 24 Mar 2011 20:45:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 20:45:45 -0000
Received: (qmail 90787 invoked by uid 500); 24 Mar 2011 20:45:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90758 invoked by uid 500); 24 Mar 2011 20:45:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90750 invoked by uid 99); 24 Mar 2011 20:45:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 20:45:45 +0000
X-ASF-Spam-Status: No, hits=-1999.7 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD,URIBL_RHS_DOB
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 20:45:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C7F7B4C9B9
	for <common-issues@hadoop.apache.org>; Thu, 24 Mar 2011 20:45:05 +0000 (UTC)
Date: Thu, 24 Mar 2011 20:45:05 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <209658622.9507.1300999505815.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1627534738.611.1300110389998.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7187) Socket Leak in
 org.apache.hadoop.metrics.ganglia.GangliaContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7187:
-------------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
                   0.22.0
                   0.21.1
           Status: Resolved  (was: Patch Available)

I have committed this.  Thanks, Uma!

> Socket Leak in org.apache.hadoop.metrics.ganglia.GangliaContext
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7187
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7187
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HADOOP-7187.1.patch, HADOOP-7187.patch
>
>
> Init method is creating DatagramSocket. But this is not closed any where. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13732-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 20:53:46 2011
Return-Path: <common-issues-return-13732-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78197 invoked from network); 24 Mar 2011 20:53:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 20:53:45 -0000
Received: (qmail 7531 invoked by uid 500); 24 Mar 2011 20:53:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7490 invoked by uid 500); 24 Mar 2011 20:53:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7437 invoked by uid 99); 24 Mar 2011 20:53:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 20:53:43 +0000
X-ASF-Spam-Status: No, hits=-1999.7 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD,URIBL_RHS_DOB
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 20:53:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2DC1C4CC9A
	for <common-issues@hadoop.apache.org>; Thu, 24 Mar 2011 20:53:06 +0000 (UTC)
Date: Thu, 24 Mar 2011 20:53:06 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1738871234.9534.1300999986183.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1627534738.611.1300110389998.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7187) Socket Leak in
 org.apache.hadoop.metrics.ganglia.GangliaContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010915#comment-13010915 ] 

Hudson commented on HADOOP-7187:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #535 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/535/])
    HADOOP-7187. Fix socket leak in GangliaContext.  Contributed by Uma Maheswara Rao G


> Socket Leak in org.apache.hadoop.metrics.ganglia.GangliaContext
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7187
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7187
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HADOOP-7187.1.patch, HADOOP-7187.patch
>
>
> Init method is creating DatagramSocket. But this is not closed any where. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13733-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 21:48:46 2011
Return-Path: <common-issues-return-13733-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64111 invoked from network); 24 Mar 2011 21:48:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 21:48:46 -0000
Received: (qmail 6856 invoked by uid 500); 24 Mar 2011 21:48:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6811 invoked by uid 500); 24 Mar 2011 21:48:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6803 invoked by uid 99); 24 Mar 2011 21:48:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 21:48:45 +0000
X-ASF-Spam-Status: No, hits=-1999.7 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD,URIBL_RHS_DOB
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 21:48:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D51854C3CA
	for <common-issues@hadoop.apache.org>; Thu, 24 Mar 2011 21:48:05 +0000 (UTC)
Date: Thu, 24 Mar 2011 21:48:05 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <484037901.9870.1301003285869.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1627534738.611.1300110389998.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7187) Socket Leak in
 org.apache.hadoop.metrics.ganglia.GangliaContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7187:
-------------------------------------------

    Fix Version/s:     (was: 0.21.1)

Reverted it from 0.21 since {{TestGangliaContext}} cannot be compiled.

> Socket Leak in org.apache.hadoop.metrics.ganglia.GangliaContext
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7187
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7187
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: HADOOP-7187.1.patch, HADOOP-7187.patch
>
>
> Init method is creating DatagramSocket. But this is not closed any where. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13734-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 21:50:46 2011
Return-Path: <common-issues-return-13734-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72431 invoked from network); 24 Mar 2011 21:50:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 21:50:46 -0000
Received: (qmail 8910 invoked by uid 500); 24 Mar 2011 21:50:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8880 invoked by uid 500); 24 Mar 2011 21:50:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8872 invoked by uid 99); 24 Mar 2011 21:50:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 21:50:45 +0000
X-ASF-Spam-Status: No, hits=-1999.7 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD,URIBL_RHS_DOB
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 21:50:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3BF204C4BE
	for <common-issues@hadoop.apache.org>; Thu, 24 Mar 2011 21:50:06 +0000 (UTC)
Date: Thu, 24 Mar 2011 21:50:06 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1524283924.9884.1301003406242.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1627534738.611.1300110389998.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7187) Socket Leak in
 org.apache.hadoop.metrics.ganglia.GangliaContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010952#comment-13010952 ] 

Hudson commented on HADOOP-7187:
--------------------------------

Integrated in Hadoop-Common-22-branch #36 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-22-branch/36/])
    HADOOP-7187. Fix socket leak in GangliaContext.  Contributed by Uma Maheswara Rao G


> Socket Leak in org.apache.hadoop.metrics.ganglia.GangliaContext
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7187
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7187
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: HADOOP-7187.1.patch, HADOOP-7187.patch
>
>
> Init method is creating DatagramSocket. But this is not closed any where. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13735-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 24 21:50:46 2011
Return-Path: <common-issues-return-13735-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72449 invoked from network); 24 Mar 2011 21:50:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Mar 2011 21:50:46 -0000
Received: (qmail 9134 invoked by uid 500); 24 Mar 2011 21:50:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9094 invoked by uid 500); 24 Mar 2011 21:50:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9086 invoked by uid 99); 24 Mar 2011 21:50:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 21:50:46 +0000
X-ASF-Spam-Status: No, hits=-1999.7 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD,URIBL_RHS_DOB
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Mar 2011 21:50:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 54E3B4C4C4
	for <common-issues@hadoop.apache.org>; Thu, 24 Mar 2011 21:50:06 +0000 (UTC)
Date: Thu, 24 Mar 2011 21:50:06 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1785402758.9888.1301003406344.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1627211432.4057.1299578279558.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7174) null is displayed in the
 console,if the src path is invalid while doing copyToLocal operation from
 commandLine
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010953#comment-13010953 ] 

Hudson commented on HADOOP-7174:
--------------------------------

Integrated in Hadoop-Common-22-branch #36 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-22-branch/36/])
    HADOOP-7174. Null is displayed in the "fs -copyToLocal" command.  Contributed by Uma Maheswara Rao G


> null is displayed in the console,if the src path is invalid while doing copyToLocal operation from commandLine
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7174
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7174
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>            Priority: Minor
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HADOOP-7174.1.patch, HADOOP-7174.patch
>
>
> When we perform copyToLocal operations from commandLine and if src Path is invalid 
> srcFS.globStatus(srcpath) will return null. So, when we find the length of resulted value, it will *throw NullPointerException*.
>  Since we are handling generic exception , it will display null as the message.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13736-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 25 00:26:43 2011
Return-Path: <common-issues-return-13736-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62944 invoked from network); 25 Mar 2011 00:26:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Mar 2011 00:26:43 -0000
Received: (qmail 89868 invoked by uid 500); 25 Mar 2011 00:26:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89838 invoked by uid 500); 25 Mar 2011 00:26:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89828 invoked by uid 99); 25 Mar 2011 00:26:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 00:26:43 +0000
X-ASF-Spam-Status: No, hits=-1999.7 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD,URIBL_RHS_DOB
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 00:26:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B13EF4C6FA
	for <common-issues@hadoop.apache.org>; Fri, 25 Mar 2011 00:26:05 +0000 (UTC)
Date: Fri, 25 Mar 2011 00:26:05 +0000 (UTC)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1176206702.10337.1301012765722.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Moved] (HADOOP-7209) Extensions to FsShell
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas moved HDFS-1784 to HADOOP-7209:
---------------------------------------------

    Affects Version/s:     (was: 0.20.3)
                       0.20.3
                  Key: HADOOP-7209  (was: HDFS-1784)
              Project: Hadoop Common  (was: Hadoop HDFS)

> Extensions to FsShell
> ---------------------
>
>                 Key: HADOOP-7209
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7209
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.3
>            Reporter: Olga Natkovich
>
> Our project, Pig, exposes FsShell functionality to our end users through a shell command. We want to use this command with no modifications to make sure that whether you work with HDFS through Hadoop or Pig you get identical semantics.
> The main concern that has been recently raised by our users is that there is no way to ignore certain failures that they consider to be benign, for instance, removing a non-existent directory.
> We have 2 asks related to this issue:
> (1) Meaningful error code returned from FsShell (we use java class) so that we can take different actions on different errors
> (2) Unix like ways to tell the command to ignore certain behavior. Here are the commands that we would like to be expanded/implemented:
>    * rm -f
>    * rmdir ---ignore-fail-on-non-empty
>    * mkdir -p 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13737-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 25 00:32:46 2011
Return-Path: <common-issues-return-13737-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82602 invoked from network); 25 Mar 2011 00:32:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Mar 2011 00:32:45 -0000
Received: (qmail 94163 invoked by uid 500); 25 Mar 2011 00:32:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94131 invoked by uid 500); 25 Mar 2011 00:32:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94123 invoked by uid 99); 25 Mar 2011 00:32:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 00:32:45 +0000
X-ASF-Spam-Status: No, hits=-1999.7 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD,URIBL_RHS_DOB
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 00:32:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E275D4C8D0
	for <common-issues@hadoop.apache.org>; Fri, 25 Mar 2011 00:32:05 +0000 (UTC)
Date: Fri, 25 Mar 2011 00:32:05 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1387364966.10342.1301013125924.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-6255:
------------------------------

    Attachment: HADOOP-6255-branch-0.20-security-5.patch

- Move start/stop scripts from $PREFIX/bin to $PREFIX/sbin.
- Added warning message that $HADOOP_HOME is deprecated per Owen's request.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13738-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 25 00:54:45 2011
Return-Path: <common-issues-return-13738-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56312 invoked from network); 25 Mar 2011 00:54:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Mar 2011 00:54:45 -0000
Received: (qmail 20330 invoked by uid 500); 25 Mar 2011 00:54:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20299 invoked by uid 500); 25 Mar 2011 00:54:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20291 invoked by uid 99); 25 Mar 2011 00:54:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 00:54:44 +0000
X-ASF-Spam-Status: No, hits=-1999.7 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD,URIBL_RHS_DOB
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 00:54:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 08B8D4CFE2
	for <common-issues@hadoop.apache.org>; Fri, 25 Mar 2011 00:54:07 +0000 (UTC)
Date: Fri, 25 Mar 2011 00:54:07 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1028059529.10444.1301014447032.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6941) Hadoop 0.21 will not work on non-SUN
 JREs due to use of com.sun.security in
 org/apache/hadoop/security/UserGroupInformation.java
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6941:
--------------------------------

    Fix Version/s: 0.22.0
                   0.21.1

> Hadoop 0.21 will not work on non-SUN JREs due to use of com.sun.security in org/apache/hadoop/security/UserGroupInformation.java
> --------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6941
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6941
>             Project: Hadoop Common
>          Issue Type: Bug
>         Environment: SLES 11, Apache Harmony 6 and SLES 11, IBM Java 6
>            Reporter: Stephen Watt
>            Assignee: Stephen Watt
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HADOOP-6941.patch, hadoop-6941.patch
>
>
> Attempting to format the namenode or attempting to start Hadoop using Apache Harmony or the IBM Java JREs results in the following exception:
> 10/09/07 16:35:05 ERROR namenode.NameNode: java.lang.NoClassDefFoundError: com.sun.security.auth.UnixPrincipal
> 	at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:223)
> 	at java.lang.J9VMInternals.initializeImpl(Native Method)
> 	at java.lang.J9VMInternals.initialize(J9VMInternals.java:200)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setConfigurationParameters(FSNamesystem.java:420)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:391)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1240)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1348)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368)
> Caused by: java.lang.ClassNotFoundException: com.sun.security.auth.UnixPrincipal
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:421)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:652)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:346)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:618)
> 	... 8 more
> This is a negative regression as previous versions of Hadoop worked with these JREs

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13739-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 25 00:54:46 2011
Return-Path: <common-issues-return-13739-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56359 invoked from network); 25 Mar 2011 00:54:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Mar 2011 00:54:46 -0000
Received: (qmail 20562 invoked by uid 500); 25 Mar 2011 00:54:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20528 invoked by uid 500); 25 Mar 2011 00:54:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20520 invoked by uid 99); 25 Mar 2011 00:54:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 00:54:46 +0000
X-ASF-Spam-Status: No, hits=-1999.7 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD,URIBL_RHS_DOB
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 00:54:44 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7B9284CFCD
	for <common-issues@hadoop.apache.org>; Fri, 25 Mar 2011 00:54:06 +0000 (UTC)
Date: Fri, 25 Mar 2011 00:54:06 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <498518309.10427.1301014446502.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6941) Hadoop 0.21 will not work on non-SUN
 JREs due to use of com.sun.security in
 org/apache/hadoop/security/UserGroupInformation.java
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6941:
--------------------------------

    Attachment: hadoop-6941.patch

Updated patch attached. Refactors the initialization code into two functions and uses an annotation on getOsPrincipalClass to remove the warnings.

> Hadoop 0.21 will not work on non-SUN JREs due to use of com.sun.security in org/apache/hadoop/security/UserGroupInformation.java
> --------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6941
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6941
>             Project: Hadoop Common
>          Issue Type: Bug
>         Environment: SLES 11, Apache Harmony 6 and SLES 11, IBM Java 6
>            Reporter: Stephen Watt
>            Assignee: Stephen Watt
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HADOOP-6941.patch, hadoop-6941.patch
>
>
> Attempting to format the namenode or attempting to start Hadoop using Apache Harmony or the IBM Java JREs results in the following exception:
> 10/09/07 16:35:05 ERROR namenode.NameNode: java.lang.NoClassDefFoundError: com.sun.security.auth.UnixPrincipal
> 	at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:223)
> 	at java.lang.J9VMInternals.initializeImpl(Native Method)
> 	at java.lang.J9VMInternals.initialize(J9VMInternals.java:200)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setConfigurationParameters(FSNamesystem.java:420)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:391)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1240)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1348)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368)
> Caused by: java.lang.ClassNotFoundException: com.sun.security.auth.UnixPrincipal
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:421)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:652)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:346)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:618)
> 	... 8 more
> This is a negative regression as previous versions of Hadoop worked with these JREs

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13740-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 25 00:54:47 2011
Return-Path: <common-issues-return-13740-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56404 invoked from network); 25 Mar 2011 00:54:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Mar 2011 00:54:47 -0000
Received: (qmail 20803 invoked by uid 500); 25 Mar 2011 00:54:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20747 invoked by uid 500); 25 Mar 2011 00:54:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20739 invoked by uid 99); 25 Mar 2011 00:54:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 00:54:46 +0000
X-ASF-Spam-Status: No, hits=-1999.7 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD,URIBL_RHS_DOB
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 00:54:44 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id ECA0E4CFDE
	for <common-issues@hadoop.apache.org>; Fri, 25 Mar 2011 00:54:06 +0000 (UTC)
Date: Fri, 25 Mar 2011 00:54:06 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1231694715.10442.1301014446966.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6941) Hadoop 0.21 will not work on non-SUN
 JREs due to use of com.sun.security in
 org/apache/hadoop/security/UserGroupInformation.java
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6941:
--------------------------------

    Affects Version/s:     (was: 0.21.0)
        Fix Version/s:     (was: 0.21.1)
                       0.23.0

> Hadoop 0.21 will not work on non-SUN JREs due to use of com.sun.security in org/apache/hadoop/security/UserGroupInformation.java
> --------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6941
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6941
>             Project: Hadoop Common
>          Issue Type: Bug
>         Environment: SLES 11, Apache Harmony 6 and SLES 11, IBM Java 6
>            Reporter: Stephen Watt
>            Assignee: Stephen Watt
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HADOOP-6941.patch, hadoop-6941.patch
>
>
> Attempting to format the namenode or attempting to start Hadoop using Apache Harmony or the IBM Java JREs results in the following exception:
> 10/09/07 16:35:05 ERROR namenode.NameNode: java.lang.NoClassDefFoundError: com.sun.security.auth.UnixPrincipal
> 	at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:223)
> 	at java.lang.J9VMInternals.initializeImpl(Native Method)
> 	at java.lang.J9VMInternals.initialize(J9VMInternals.java:200)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setConfigurationParameters(FSNamesystem.java:420)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:391)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1240)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1348)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368)
> Caused by: java.lang.ClassNotFoundException: com.sun.security.auth.UnixPrincipal
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:421)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:652)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:346)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:618)
> 	... 8 more
> This is a negative regression as previous versions of Hadoop worked with these JREs

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13741-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 25 11:13:45 2011
Return-Path: <common-issues-return-13741-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29243 invoked from network); 25 Mar 2011 11:13:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Mar 2011 11:13:44 -0000
Received: (qmail 61174 invoked by uid 500); 25 Mar 2011 11:13:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61089 invoked by uid 500); 25 Mar 2011 11:13:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61081 invoked by uid 99); 25 Mar 2011 11:13:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 11:13:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 11:13:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EE6F94D6ED
	for <common-issues@hadoop.apache.org>; Fri, 25 Mar 2011 11:13:05 +0000 (UTC)
Date: Fri, 25 Mar 2011 11:13:05 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <218361125.10941.1301051585973.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1627211432.4057.1299578279558.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7174) null is displayed in the
 console,if the src path is invalid while doing copyToLocal operation from
 commandLine
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011157#comment-13011157 ] 

Hudson commented on HADOOP-7174:
--------------------------------

Integrated in Hadoop-Common-trunk #641 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/641/])
    HADOOP-7174. Null is displayed in the "fs -copyToLocal" command.  Contributed by Uma Maheswara Rao G


> null is displayed in the console,if the src path is invalid while doing copyToLocal operation from commandLine
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7174
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7174
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>            Priority: Minor
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HADOOP-7174.1.patch, HADOOP-7174.patch
>
>
> When we perform copyToLocal operations from commandLine and if src Path is invalid 
> srcFS.globStatus(srcpath) will return null. So, when we find the length of resulted value, it will *throw NullPointerException*.
>  Since we are handling generic exception , it will display null as the message.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13742-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 25 11:13:45 2011
Return-Path: <common-issues-return-13742-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29240 invoked from network); 25 Mar 2011 11:13:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Mar 2011 11:13:44 -0000
Received: (qmail 61244 invoked by uid 500); 25 Mar 2011 11:13:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61100 invoked by uid 500); 25 Mar 2011 11:13:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61091 invoked by uid 99); 25 Mar 2011 11:13:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 11:13:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 11:13:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E07D14D6EB
	for <common-issues@hadoop.apache.org>; Fri, 25 Mar 2011 11:13:05 +0000 (UTC)
Date: Fri, 25 Mar 2011 11:13:05 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <51984340.10939.1301051585916.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1627534738.611.1300110389998.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7187) Socket Leak in
 org.apache.hadoop.metrics.ganglia.GangliaContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011156#comment-13011156 ] 

Hudson commented on HADOOP-7187:
--------------------------------

Integrated in Hadoop-Common-trunk #641 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/641/])
    HADOOP-7187. Fix socket leak in GangliaContext.  Contributed by Uma Maheswara Rao G


> Socket Leak in org.apache.hadoop.metrics.ganglia.GangliaContext
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7187
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7187
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: HADOOP-7187.1.patch, HADOOP-7187.patch
>
>
> Init method is creating DatagramSocket. But this is not closed any where. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13743-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 25 14:03:48 2011
Return-Path: <common-issues-return-13743-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84690 invoked from network); 25 Mar 2011 14:03:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Mar 2011 14:03:47 -0000
Received: (qmail 32384 invoked by uid 500); 25 Mar 2011 14:03:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32348 invoked by uid 500); 25 Mar 2011 14:03:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32279 invoked by uid 99); 25 Mar 2011 14:03:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 14:03:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 14:03:44 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 787D14DDF4
	for <common-issues@hadoop.apache.org>; Fri, 25 Mar 2011 14:03:06 +0000 (UTC)
Date: Fri, 25 Mar 2011 14:03:06 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1995829420.11194.1301061786490.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7209) Extensions to FsShell
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp reassigned HADOOP-7209:
-----------------------------------

    Assignee: Daryn Sharp

> Extensions to FsShell
> ---------------------
>
>                 Key: HADOOP-7209
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7209
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.3
>            Reporter: Olga Natkovich
>            Assignee: Daryn Sharp
>
> Our project, Pig, exposes FsShell functionality to our end users through a shell command. We want to use this command with no modifications to make sure that whether you work with HDFS through Hadoop or Pig you get identical semantics.
> The main concern that has been recently raised by our users is that there is no way to ignore certain failures that they consider to be benign, for instance, removing a non-existent directory.
> We have 2 asks related to this issue:
> (1) Meaningful error code returned from FsShell (we use java class) so that we can take different actions on different errors
> (2) Unix like ways to tell the command to ignore certain behavior. Here are the commands that we would like to be expanded/implemented:
>    * rm -f
>    * rmdir ---ignore-fail-on-non-empty
>    * mkdir -p 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13744-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 25 15:03:46 2011
Return-Path: <common-issues-return-13744-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98841 invoked from network); 25 Mar 2011 15:03:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Mar 2011 15:03:46 -0000
Received: (qmail 66025 invoked by uid 500); 25 Mar 2011 15:03:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65989 invoked by uid 500); 25 Mar 2011 15:03:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65981 invoked by uid 99); 25 Mar 2011 15:03:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 15:03:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 15:03:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0B9764D8B5
	for <common-issues@hadoop.apache.org>; Fri, 25 Mar 2011 15:03:06 +0000 (UTC)
Date: Fri, 25 Mar 2011 15:03:06 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <471610084.11410.1301065386044.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1587140683.1515.1300732271817.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7202) Improve Command base class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7202:
--------------------------------

    Attachment: HADOOP-7202-2.patch

1) I feel that "if (item.exists)" is more intuitive to a reader of external code than "if (item.stat != null)".  I will change it if you feel strongly.

2) Great idea.

3) I went back and added javadocs links to all my new methods. Phew.

4) I'm trying to more closely follow unix conventions.  Also, -1 for invalid usage versus 1 for execution failure, allows external scripts to differentiate the mode of failure.

5) There are other external commands that subclass Command and will need access to this class if updated, so I don't think I can reduce the visibility.

> Improve Command base class
> --------------------------
>
>                 Key: HADOOP-7202
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7202
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7202-2.patch, HADOOP-7202.patch
>
>
> Need to extend the Command base class to allow all command to easily subclass from a code set of code that correctly handles globs and exit codes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13745-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 25 17:54:46 2011
Return-Path: <common-issues-return-13745-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66095 invoked from network); 25 Mar 2011 17:54:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Mar 2011 17:54:46 -0000
Received: (qmail 64540 invoked by uid 500); 25 Mar 2011 17:54:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64511 invoked by uid 500); 25 Mar 2011 17:54:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64503 invoked by uid 99); 25 Mar 2011 17:54:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 17:54:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 17:54:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C99B34DE42
	for <common-issues@hadoop.apache.org>; Fri, 25 Mar 2011 17:54:05 +0000 (UTC)
Date: Fri, 25 Mar 2011 17:54:05 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1125772000.11749.1301075645822.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1627211432.4057.1299578279558.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7174) null is displayed in the
 console,if the src path is invalid while doing copyToLocal operation from
 commandLine
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011310#comment-13011310 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7174:
------------------------------------------------

Hi Uma, some tests in HDFS failed after this; see {{HDFS-1786}}.  Could you fix them as well?

> null is displayed in the console,if the src path is invalid while doing copyToLocal operation from commandLine
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7174
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7174
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>            Priority: Minor
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HADOOP-7174.1.patch, HADOOP-7174.patch
>
>
> When we perform copyToLocal operations from commandLine and if src Path is invalid 
> srcFS.globStatus(srcpath) will return null. So, when we find the length of resulted value, it will *throw NullPointerException*.
>  Since we are handling generic exception , it will display null as the message.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13746-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 25 18:06:43 2011
Return-Path: <common-issues-return-13746-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22114 invoked from network); 25 Mar 2011 18:06:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Mar 2011 18:06:43 -0000
Received: (qmail 78365 invoked by uid 500); 25 Mar 2011 18:06:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78288 invoked by uid 500); 25 Mar 2011 18:06:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78280 invoked by uid 99); 25 Mar 2011 18:06:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 18:06:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 18:06:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CA89E4D37E
	for <common-issues@hadoop.apache.org>; Fri, 25 Mar 2011 18:06:05 +0000 (UTC)
Date: Fri, 25 Mar 2011 18:06:05 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1340370439.11781.1301076365826.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1627211432.4057.1299578279558.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7174) null is displayed in the
 console,if the src path is invalid while doing copyToLocal operation from
 commandLine
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011316#comment-13011316 ] 

Uma Maheswara Rao G commented on HADOOP-7174:
---------------------------------------------

Thanks Nicholas, i addressed this.
I prepared the patch ,  will upload it in HDFS.

> null is displayed in the console,if the src path is invalid while doing copyToLocal operation from commandLine
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7174
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7174
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>            Priority: Minor
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HADOOP-7174.1.patch, HADOOP-7174.patch
>
>
> When we perform copyToLocal operations from commandLine and if src Path is invalid 
> srcFS.globStatus(srcpath) will return null. So, when we find the length of resulted value, it will *throw NullPointerException*.
>  Since we are handling generic exception , it will display null as the message.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13747-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 25 20:51:44 2011
Return-Path: <common-issues-return-13747-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61717 invoked from network); 25 Mar 2011 20:51:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Mar 2011 20:51:43 -0000
Received: (qmail 1165 invoked by uid 500); 25 Mar 2011 20:51:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1129 invoked by uid 500); 25 Mar 2011 20:51:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1121 invoked by uid 99); 25 Mar 2011 20:51:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 20:51:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 20:51:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 294704E051
	for <common-issues@hadoop.apache.org>; Fri, 25 Mar 2011 20:51:06 +0000 (UTC)
Date: Fri, 25 Mar 2011 20:51:06 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1981302789.12287.1301086266165.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <113773164.6166.1300269149586.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7194) Potential Resource leak in
 IOUtils.java
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011411#comment-13011411 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7194:
------------------------------------------------

Good catch!  It seems that closing in the finally-block using {{closeStream(..)}} is sufficient.  The following is not required.
{code}
       if(close) {
         out.close();
+        out = null;
         in.close();
+        in = null;
       }
{code}


> Potential Resource leak in IOUtils.java
> ---------------------------------------
>
>                 Key: HADOOP-7194
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7194
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.23.0
>            Reporter: Devaraj K
>         Attachments: HADOOP-7194.patch
>
>
> {code:title=IOUtils.java|borderStyle=solid}
> try {
>       copyBytes(in, out, buffSize);
>     } finally {
>       if(close) {
>         out.close();
>         in.close();
>       }
>     }
> {code} 
> In the above code if any exception throws from the out.close() statement, in.close() statement will not execute and the input stream will not be closed.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13748-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 25 21:28:44 2011
Return-Path: <common-issues-return-13748-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66979 invoked from network); 25 Mar 2011 21:28:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Mar 2011 21:28:44 -0000
Received: (qmail 58312 invoked by uid 500); 25 Mar 2011 21:28:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58258 invoked by uid 500); 25 Mar 2011 21:28:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58248 invoked by uid 99); 25 Mar 2011 21:28:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 21:28:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 21:28:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4BBF34E978
	for <common-issues@hadoop.apache.org>; Fri, 25 Mar 2011 21:28:06 +0000 (UTC)
Date: Fri, 25 Mar 2011 21:28:06 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <155809398.12391.1301088486306.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6941) Hadoop 0.21 will not work on
 non-SUN JREs due to use of com.sun.security in
 org/apache/hadoop/security/UserGroupInformation.java
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011429#comment-13011429 ] 

Todd Lipcon commented on HADOOP-6941:
-------------------------------------

Looks good to me. Steve, can you confirm this still works on your setup?

> Hadoop 0.21 will not work on non-SUN JREs due to use of com.sun.security in org/apache/hadoop/security/UserGroupInformation.java
> --------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6941
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6941
>             Project: Hadoop Common
>          Issue Type: Bug
>         Environment: SLES 11, Apache Harmony 6 and SLES 11, IBM Java 6
>            Reporter: Stephen Watt
>            Assignee: Stephen Watt
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HADOOP-6941.patch, hadoop-6941.patch
>
>
> Attempting to format the namenode or attempting to start Hadoop using Apache Harmony or the IBM Java JREs results in the following exception:
> 10/09/07 16:35:05 ERROR namenode.NameNode: java.lang.NoClassDefFoundError: com.sun.security.auth.UnixPrincipal
> 	at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:223)
> 	at java.lang.J9VMInternals.initializeImpl(Native Method)
> 	at java.lang.J9VMInternals.initialize(J9VMInternals.java:200)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setConfigurationParameters(FSNamesystem.java:420)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:391)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1240)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1348)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368)
> Caused by: java.lang.ClassNotFoundException: com.sun.security.auth.UnixPrincipal
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:421)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:652)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:346)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:618)
> 	... 8 more
> This is a negative regression as previous versions of Hadoop worked with these JREs

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13749-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 25 22:57:46 2011
Return-Path: <common-issues-return-13749-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53097 invoked from network); 25 Mar 2011 22:57:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Mar 2011 22:57:46 -0000
Received: (qmail 61754 invoked by uid 500); 25 Mar 2011 22:57:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61707 invoked by uid 500); 25 Mar 2011 22:57:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61699 invoked by uid 99); 25 Mar 2011 22:57:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 22:57:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 22:57:44 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B33F64E238
	for <common-issues@hadoop.apache.org>; Fri, 25 Mar 2011 22:57:06 +0000 (UTC)
Date: Fri, 25 Mar 2011 22:57:06 +0000 (UTC)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1466631733.12779.1301093826731.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6949) Reduces RPC packet size for
 primitive arrays, especially long[], which is used at block reporting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011471#comment-13011471 ] 

Konstantin Shvachko commented on HADOOP-6949:
---------------------------------------------

I think we should also commit it to 0.22.

> Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6949
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6949
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Navis
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: ObjectWritable.diff, arrayprim.patch, arrayprim_v4.patch, arrayprim_v5.patch, arrayprim_v6.patch, arrayprim_v7.patch
>
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n
> It would not be needed to specify each element types for primitive arrays.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13750-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 25 22:59:43 2011
Return-Path: <common-issues-return-13750-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53264 invoked from network); 25 Mar 2011 22:59:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Mar 2011 22:59:43 -0000
Received: (qmail 62413 invoked by uid 500); 25 Mar 2011 22:59:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62313 invoked by uid 500); 25 Mar 2011 22:59:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62302 invoked by uid 99); 25 Mar 2011 22:59:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 22:59:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 22:59:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DAB6F4E2D2
	for <common-issues@hadoop.apache.org>; Fri, 25 Mar 2011 22:59:05 +0000 (UTC)
Date: Fri, 25 Mar 2011 22:59:05 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <173142817.12786.1301093945892.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1587140683.1515.1300732271817.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7202) Improve Command base class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011473#comment-13011473 ] 

Tanping Wang commented on HADOOP-7202:
--------------------------------------

Patch looks good to me now.

> Improve Command base class
> --------------------------
>
>                 Key: HADOOP-7202
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7202
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7202-2.patch, HADOOP-7202.patch
>
>
> Need to extend the Command base class to allow all command to easily subclass from a code set of code that correctly handles globs and exit codes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13751-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 25 23:19:44 2011
Return-Path: <common-issues-return-13751-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21372 invoked from network); 25 Mar 2011 23:19:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Mar 2011 23:19:44 -0000
Received: (qmail 81222 invoked by uid 500); 25 Mar 2011 23:19:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81192 invoked by uid 500); 25 Mar 2011 23:19:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81184 invoked by uid 99); 25 Mar 2011 23:19:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 23:19:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 23:19:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4C4344E8DC
	for <common-issues@hadoop.apache.org>; Fri, 25 Mar 2011 23:19:06 +0000 (UTC)
Date: Fri, 25 Mar 2011 23:19:06 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1164009065.12866.1301095146309.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6941) Hadoop 0.21 will not work on non-SUN
 JREs due to use of com.sun.security in
 org/apache/hadoop/security/UserGroupInformation.java
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6941:
--------------------------------

    Hadoop Flags: [Reviewed]
          Status: Patch Available  (was: Open)

> Hadoop 0.21 will not work on non-SUN JREs due to use of com.sun.security in org/apache/hadoop/security/UserGroupInformation.java
> --------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6941
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6941
>             Project: Hadoop Common
>          Issue Type: Bug
>         Environment: SLES 11, Apache Harmony 6 and SLES 11, IBM Java 6
>            Reporter: Stephen Watt
>            Assignee: Stephen Watt
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HADOOP-6941.patch, hadoop-6941.patch
>
>
> Attempting to format the namenode or attempting to start Hadoop using Apache Harmony or the IBM Java JREs results in the following exception:
> 10/09/07 16:35:05 ERROR namenode.NameNode: java.lang.NoClassDefFoundError: com.sun.security.auth.UnixPrincipal
> 	at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:223)
> 	at java.lang.J9VMInternals.initializeImpl(Native Method)
> 	at java.lang.J9VMInternals.initialize(J9VMInternals.java:200)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setConfigurationParameters(FSNamesystem.java:420)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:391)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1240)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1348)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368)
> Caused by: java.lang.ClassNotFoundException: com.sun.security.auth.UnixPrincipal
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:421)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:652)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:346)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:618)
> 	... 8 more
> This is a negative regression as previous versions of Hadoop worked with these JREs

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13752-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 25 23:19:46 2011
Return-Path: <common-issues-return-13752-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21581 invoked from network); 25 Mar 2011 23:19:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Mar 2011 23:19:46 -0000
Received: (qmail 81457 invoked by uid 500); 25 Mar 2011 23:19:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81425 invoked by uid 500); 25 Mar 2011 23:19:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81409 invoked by uid 99); 25 Mar 2011 23:19:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 23:19:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 23:19:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 59BCF4E8DE
	for <common-issues@hadoop.apache.org>; Fri, 25 Mar 2011 23:19:06 +0000 (UTC)
Date: Fri, 25 Mar 2011 23:19:06 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1693709666.12868.1301095146364.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6941) Support non-SUN JREs in
 UserGroupInformation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6941:
--------------------------------

    Summary: Support non-SUN JREs in UserGroupInformation  (was: Hadoop 0.21 will not work on non-SUN JREs due to use of com.sun.security in org/apache/hadoop/security/UserGroupInformation.java)

> Support non-SUN JREs in UserGroupInformation
> --------------------------------------------
>
>                 Key: HADOOP-6941
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6941
>             Project: Hadoop Common
>          Issue Type: Bug
>         Environment: SLES 11, Apache Harmony 6 and SLES 11, IBM Java 6
>            Reporter: Stephen Watt
>            Assignee: Stephen Watt
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HADOOP-6941.patch, hadoop-6941.patch
>
>
> Attempting to format the namenode or attempting to start Hadoop using Apache Harmony or the IBM Java JREs results in the following exception:
> 10/09/07 16:35:05 ERROR namenode.NameNode: java.lang.NoClassDefFoundError: com.sun.security.auth.UnixPrincipal
> 	at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:223)
> 	at java.lang.J9VMInternals.initializeImpl(Native Method)
> 	at java.lang.J9VMInternals.initialize(J9VMInternals.java:200)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setConfigurationParameters(FSNamesystem.java:420)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:391)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1240)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1348)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368)
> Caused by: java.lang.ClassNotFoundException: com.sun.security.auth.UnixPrincipal
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:421)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:652)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:346)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:618)
> 	... 8 more
> This is a negative regression as previous versions of Hadoop worked with these JREs

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13753-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 25 23:27:46 2011
Return-Path: <common-issues-return-13753-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53060 invoked from network); 25 Mar 2011 23:27:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Mar 2011 23:27:46 -0000
Received: (qmail 87079 invoked by uid 500); 25 Mar 2011 23:27:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87041 invoked by uid 500); 25 Mar 2011 23:27:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87033 invoked by uid 99); 25 Mar 2011 23:27:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 23:27:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 23:27:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1DFDC4EB01
	for <common-issues@hadoop.apache.org>; Fri, 25 Mar 2011 23:27:06 +0000 (UTC)
Date: Fri, 25 Mar 2011 23:27:06 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <367709849.12905.1301095626119.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1587140683.1515.1300732271817.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7202) Improve Command base class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011498#comment-13011498 ] 

Tanping Wang commented on HADOOP-7202:
--------------------------------------

+1.

> Improve Command base class
> --------------------------
>
>                 Key: HADOOP-7202
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7202
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7202-2.patch, HADOOP-7202.patch
>
>
> Need to extend the Command base class to allow all command to easily subclass from a code set of code that correctly handles globs and exit codes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13754-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Mar 25 23:45:45 2011
Return-Path: <common-issues-return-13754-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6219 invoked from network); 25 Mar 2011 23:45:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Mar 2011 23:45:45 -0000
Received: (qmail 6093 invoked by uid 500); 25 Mar 2011 23:45:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6063 invoked by uid 500); 25 Mar 2011 23:45:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6055 invoked by uid 99); 25 Mar 2011 23:45:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 23:45:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Mar 2011 23:45:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B61704EE92
	for <common-issues@hadoop.apache.org>; Fri, 25 Mar 2011 23:45:05 +0000 (UTC)
Date: Fri, 25 Mar 2011 23:45:05 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1070507832.12946.1301096705742.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1587140683.1515.1300732271817.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7202) Improve Command base class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011505#comment-13011505 ] 

Hadoop QA commented on HADOOP-7202:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12474619/HADOOP-7202-2.patch
  against trunk revision 1085122.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    -1 javac.  The applied patch generated 1076 javac compiler warnings (more than the trunk's current 1072 warnings).

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/320//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/320//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/320//console

This message is automatically generated.

> Improve Command base class
> --------------------------
>
>                 Key: HADOOP-7202
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7202
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7202-2.patch, HADOOP-7202.patch
>
>
> Need to extend the Command base class to allow all command to easily subclass from a code set of code that correctly handles globs and exit codes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13755-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Mar 26 00:45:46 2011
Return-Path: <common-issues-return-13755-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4234 invoked from network); 26 Mar 2011 00:45:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Mar 2011 00:45:46 -0000
Received: (qmail 52001 invoked by uid 500); 26 Mar 2011 00:45:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51963 invoked by uid 500); 26 Mar 2011 00:45:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51949 invoked by uid 99); 26 Mar 2011 00:45:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Mar 2011 00:45:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Mar 2011 00:45:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 18EB94EA0A
	for <common-issues@hadoop.apache.org>; Sat, 26 Mar 2011 00:45:06 +0000 (UTC)
Date: Sat, 26 Mar 2011 00:45:06 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1996587084.13055.1301100306098.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6949) Reduces RPC packet size for
 primitive arrays, especially long[], which is used at block reporting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011519#comment-13011519 ] 

Matt Foley commented on HADOOP-6949:
------------------------------------

Since this is an incompatible protocol change, please do start whatever discussion is needed in the community.

> Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6949
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6949
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Navis
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: ObjectWritable.diff, arrayprim.patch, arrayprim_v4.patch, arrayprim_v5.patch, arrayprim_v6.patch, arrayprim_v7.patch
>
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n
> It would not be needed to specify each element types for primitive arrays.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13756-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Mar 27 03:33:53 2011
Return-Path: <common-issues-return-13756-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2897 invoked from network); 27 Mar 2011 03:33:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Mar 2011 03:33:53 -0000
Received: (qmail 65123 invoked by uid 500); 27 Mar 2011 03:33:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64961 invoked by uid 500); 27 Mar 2011 03:33:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64948 invoked by uid 99); 27 Mar 2011 03:33:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Mar 2011 03:33:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Mar 2011 03:33:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C8FBB56FE7
	for <common-issues@hadoop.apache.org>; Sun, 27 Mar 2011 03:33:05 +0000 (UTC)
Date: Sun, 27 Mar 2011 03:33:05 +0000 (UTC)
From: "Jingguo Yao (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1250518876.14486.1301196785819.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <15581131.20741291107311011.JavaMail.jira@thor>
Subject: [jira] [Assigned] (HADOOP-7055) Update of commons logging libraries
 causes EventCounter to count logging events incorrectly
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jingguo Yao reassigned HADOOP-7055:
-----------------------------------

    Assignee: Jingguo Yao

> Update of commons logging libraries causes EventCounter to count logging events incorrectly
> -------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7055
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7055
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.21.0, 0.21.1, 0.22.0, 0.23.0
>            Reporter: Jingguo Yao
>            Assignee: Jingguo Yao
>   Original Estimate: 0.5h
>  Remaining Estimate: 0.5h
>
> Hadoop 0.20.2 uses commons logging 1.0.4. EventCounter works correctly with this version of commons logging. Hadoop 0.21.0 uses commons logging 1.1.1 which causes EventCounter to count logging events incorrectly. I have verified it with Hadoop 0.21.0. After start-up of hadoop, I checked jvmmetrics.log after several minutes. In every metrics record, "logError=0, logFatal=0, logInfo=3, logWarn=0" was shown. The following text is an example.
> jvm.metrics: hostName=jingguolin, processName=DataNode, sessionId=, gcCount=3, gcTimeMillis=31, logError=0, logFatal=0, logInfo=3, logWarn=0, maxMemoryM=888.9375, memHeapCommittedM=38.0625, memHeapUsedM=3.6539612, memNonHeapCommittedM=18.25, memNonHeapUsedM=11.335686, threadsBlocked=0, threadsNew=0, threadsRunnable=8, threadsTerminated=0, threadsTimedWaiting=6, threadsWaiting=6
> Then I stopped hadoop and replaced commons logging 1.1.1 with 1.0.4. After the re-start of hadoop, a lot of logging events showed up in jvmmetrics.log.
> I have checked the source code of Log4JLogger for both 1.0.4 (http://svn.apache.org/viewvc/commons/proper/logging/tags/LOGGING_1_0_4/src/java/org/apache/commons/logging/impl/Log4JLogger.java?view=markup) and 1.1.1 (http://svn.apache.org/viewvc/commons/proper/logging/tags/commons-logging-1.1.1/src/java/org/apache/commons/logging/impl/Log4JLogger.java?view=markup). For 1.0.4, Level instances such as Level.INFO are used to construct LoggingEvent. But for 1.1.1, Priority instances such as Priority.INFO are used to construct LoggingEvent. So 1.1.1 version's event.getLevel() always returns Priority instances. EventCounter append method's "==" check always fails between a Level instance and a Priority instance. For "logInfo=3" metrics records produced by commons logging 1.1.1., I think that these 3 logging events are produced by log4j code directly instead of through commons logging API. The following code is EventCounter's append method.
>     public void append(LoggingEvent event) {
>         Level level = event.getLevel();
>         if (level == Level.INFO) {
>             counts.incr(INFO);
>         }
>         else if (level == Level.WARN) {
>             counts.incr(WARN);
>         }
>         else if (level == Level.ERROR) {
>             counts.incr(ERROR);
>         }
>         else if (level == Level.FATAL) {
>             counts.incr(FATAL);
>         }
>     }

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13757-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Mar 27 04:43:46 2011
Return-Path: <common-issues-return-13757-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91390 invoked from network); 27 Mar 2011 04:43:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Mar 2011 04:43:46 -0000
Received: (qmail 79358 invoked by uid 500); 27 Mar 2011 04:43:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79325 invoked by uid 500); 27 Mar 2011 04:43:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79312 invoked by uid 99); 27 Mar 2011 04:43:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Mar 2011 04:43:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Mar 2011 04:43:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B7A2D57802
	for <common-issues@hadoop.apache.org>; Sun, 27 Mar 2011 04:43:05 +0000 (UTC)
Date: Sun, 27 Mar 2011 04:43:05 +0000 (UTC)
From: "Jingguo Yao (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1208479021.14549.1301200985748.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <15581131.20741291107311011.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7055) Update of commons logging
 libraries causes EventCounter to count logging events incorrectly
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011724#comment-13011724 ] 

Jingguo Yao commented on HADOOP-7055:
-------------------------------------

When EventCounter appender is triggered, it only updates its counts field. So equals overhead is not negligible compared to the actual logging which is counts field updating in this case.

But if we think that the overhead incurred by equals is negligible on the overall performance, we can use equals.

> Update of commons logging libraries causes EventCounter to count logging events incorrectly
> -------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7055
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7055
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.21.0, 0.21.1, 0.22.0, 0.23.0
>            Reporter: Jingguo Yao
>            Assignee: Jingguo Yao
>   Original Estimate: 0.5h
>  Remaining Estimate: 0.5h
>
> Hadoop 0.20.2 uses commons logging 1.0.4. EventCounter works correctly with this version of commons logging. Hadoop 0.21.0 uses commons logging 1.1.1 which causes EventCounter to count logging events incorrectly. I have verified it with Hadoop 0.21.0. After start-up of hadoop, I checked jvmmetrics.log after several minutes. In every metrics record, "logError=0, logFatal=0, logInfo=3, logWarn=0" was shown. The following text is an example.
> jvm.metrics: hostName=jingguolin, processName=DataNode, sessionId=, gcCount=3, gcTimeMillis=31, logError=0, logFatal=0, logInfo=3, logWarn=0, maxMemoryM=888.9375, memHeapCommittedM=38.0625, memHeapUsedM=3.6539612, memNonHeapCommittedM=18.25, memNonHeapUsedM=11.335686, threadsBlocked=0, threadsNew=0, threadsRunnable=8, threadsTerminated=0, threadsTimedWaiting=6, threadsWaiting=6
> Then I stopped hadoop and replaced commons logging 1.1.1 with 1.0.4. After the re-start of hadoop, a lot of logging events showed up in jvmmetrics.log.
> I have checked the source code of Log4JLogger for both 1.0.4 (http://svn.apache.org/viewvc/commons/proper/logging/tags/LOGGING_1_0_4/src/java/org/apache/commons/logging/impl/Log4JLogger.java?view=markup) and 1.1.1 (http://svn.apache.org/viewvc/commons/proper/logging/tags/commons-logging-1.1.1/src/java/org/apache/commons/logging/impl/Log4JLogger.java?view=markup). For 1.0.4, Level instances such as Level.INFO are used to construct LoggingEvent. But for 1.1.1, Priority instances such as Priority.INFO are used to construct LoggingEvent. So 1.1.1 version's event.getLevel() always returns Priority instances. EventCounter append method's "==" check always fails between a Level instance and a Priority instance. For "logInfo=3" metrics records produced by commons logging 1.1.1., I think that these 3 logging events are produced by log4j code directly instead of through commons logging API. The following code is EventCounter's append method.
>     public void append(LoggingEvent event) {
>         Level level = event.getLevel();
>         if (level == Level.INFO) {
>             counts.incr(INFO);
>         }
>         else if (level == Level.WARN) {
>             counts.incr(WARN);
>         }
>         else if (level == Level.ERROR) {
>             counts.incr(ERROR);
>         }
>         else if (level == Level.FATAL) {
>             counts.incr(FATAL);
>         }
>     }

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13758-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Mar 27 17:52:45 2011
Return-Path: <common-issues-return-13758-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21246 invoked from network); 27 Mar 2011 17:52:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Mar 2011 17:52:44 -0000
Received: (qmail 9747 invoked by uid 500); 27 Mar 2011 17:52:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9708 invoked by uid 500); 27 Mar 2011 17:52:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9698 invoked by uid 99); 27 Mar 2011 17:52:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Mar 2011 17:52:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Mar 2011 17:52:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B1DCB5794C
	for <common-issues@hadoop.apache.org>; Sun, 27 Mar 2011 17:52:05 +0000 (UTC)
Date: Sun, 27 Mar 2011 17:52:05 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <46167893.15525.1301248325725.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <15581131.20741291107311011.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7055) Update of commons logging
 libraries causes EventCounter to count logging events incorrectly
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011825#comment-13011825 ] 

Luke Lu commented on HADOOP-7055:
---------------------------------

bq. When EventCounter appender is triggered, it only updates its counts field. So equals overhead is not negligible compared to the actual logging which is counts field updating in this case.

When the appender is triggered, a new LoggingEvent object is created and at least one synchronization happened which is probably 100x more expensive then the method call. Note, updating the counts field is a *synchronized* call, which is about 30x more expensive then the equals call. If don't believe it, just benchmark it and make sure you have at least two threads doing the update, so jvm doesn't elide the locks.


> Update of commons logging libraries causes EventCounter to count logging events incorrectly
> -------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7055
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7055
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.21.0, 0.21.1, 0.22.0, 0.23.0
>            Reporter: Jingguo Yao
>            Assignee: Jingguo Yao
>   Original Estimate: 0.5h
>  Remaining Estimate: 0.5h
>
> Hadoop 0.20.2 uses commons logging 1.0.4. EventCounter works correctly with this version of commons logging. Hadoop 0.21.0 uses commons logging 1.1.1 which causes EventCounter to count logging events incorrectly. I have verified it with Hadoop 0.21.0. After start-up of hadoop, I checked jvmmetrics.log after several minutes. In every metrics record, "logError=0, logFatal=0, logInfo=3, logWarn=0" was shown. The following text is an example.
> jvm.metrics: hostName=jingguolin, processName=DataNode, sessionId=, gcCount=3, gcTimeMillis=31, logError=0, logFatal=0, logInfo=3, logWarn=0, maxMemoryM=888.9375, memHeapCommittedM=38.0625, memHeapUsedM=3.6539612, memNonHeapCommittedM=18.25, memNonHeapUsedM=11.335686, threadsBlocked=0, threadsNew=0, threadsRunnable=8, threadsTerminated=0, threadsTimedWaiting=6, threadsWaiting=6
> Then I stopped hadoop and replaced commons logging 1.1.1 with 1.0.4. After the re-start of hadoop, a lot of logging events showed up in jvmmetrics.log.
> I have checked the source code of Log4JLogger for both 1.0.4 (http://svn.apache.org/viewvc/commons/proper/logging/tags/LOGGING_1_0_4/src/java/org/apache/commons/logging/impl/Log4JLogger.java?view=markup) and 1.1.1 (http://svn.apache.org/viewvc/commons/proper/logging/tags/commons-logging-1.1.1/src/java/org/apache/commons/logging/impl/Log4JLogger.java?view=markup). For 1.0.4, Level instances such as Level.INFO are used to construct LoggingEvent. But for 1.1.1, Priority instances such as Priority.INFO are used to construct LoggingEvent. So 1.1.1 version's event.getLevel() always returns Priority instances. EventCounter append method's "==" check always fails between a Level instance and a Priority instance. For "logInfo=3" metrics records produced by commons logging 1.1.1., I think that these 3 logging events are produced by log4j code directly instead of through commons logging API. The following code is EventCounter's append method.
>     public void append(LoggingEvent event) {
>         Level level = event.getLevel();
>         if (level == Level.INFO) {
>             counts.incr(INFO);
>         }
>         else if (level == Level.WARN) {
>             counts.incr(WARN);
>         }
>         else if (level == Level.ERROR) {
>             counts.incr(ERROR);
>         }
>         else if (level == Level.FATAL) {
>             counts.incr(FATAL);
>         }
>     }

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13759-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 28 02:18:44 2011
Return-Path: <common-issues-return-13759-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30899 invoked from network); 28 Mar 2011 02:18:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Mar 2011 02:18:44 -0000
Received: (qmail 92215 invoked by uid 500); 28 Mar 2011 02:18:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92176 invoked by uid 500); 28 Mar 2011 02:18:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92168 invoked by uid 99); 28 Mar 2011 02:18:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 02:18:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 02:18:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BF19C836E8
	for <common-issues@hadoop.apache.org>; Mon, 28 Mar 2011 02:18:06 +0000 (UTC)
Date: Mon, 28 Mar 2011 02:18:06 +0000 (UTC)
From: "Benoit Sigoure (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <360735347.15889.1301278686779.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011893#comment-13011893 ] 

Benoit Sigoure commented on HADOOP-7206:
----------------------------------------

Allen, I agree that situation with back-porting everything in Hadoop has gotten to a ridiculous point.  So while I understand and share your desire to stop back-porting things or splitting things out, you also have to understand the desire of users like us whose business greatly depends on Hadoop/HBase and where we need to move forward quickly.  If we were to wait until Apache releases a version of Hadoop we can use in production with HBase (proper append, no data loss, etc), we'd still be waiting.  So although I don't like the current situation with Hadoop either, I'm glad someone did the grungy work of back-porting things or splitting some things out so we could move forward.

What Todd is proposing is simply a way to make Snappy available quickly to users like us, and we'd be very happy about that.  I think it's in everyone's interest to make this available as soon as possible and not wait for a future Hadoop release.

Note: we're not Cloudera customers, but we use CDH because It Just Works.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Eli Collins
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13760-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 28 05:40:45 2011
Return-Path: <common-issues-return-13760-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99877 invoked from network); 28 Mar 2011 05:40:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Mar 2011 05:40:45 -0000
Received: (qmail 82237 invoked by uid 500); 28 Mar 2011 05:40:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82211 invoked by uid 500); 28 Mar 2011 05:40:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82203 invoked by uid 99); 28 Mar 2011 05:40:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 05:40:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 05:40:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C94C8838D9
	for <common-issues@hadoop.apache.org>; Mon, 28 Mar 2011 05:40:05 +0000 (UTC)
Date: Mon, 28 Mar 2011 05:40:05 +0000 (UTC)
From: "Devaraj K (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <498392844.16024.1301290805820.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <113773164.6166.1300269149586.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7194) Potential Resource leak in
 IOUtils.java
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011916#comment-13011916 ] 

Devaraj K commented on HADOOP-7194:
-----------------------------------

Thanks Nicholas for reviewing.

Closing the streams in the finally-block using closeStream(..) will suppress the exception if out.close()/in.close() throws and will not be thrown back to the caller. 

{code:xml} 

if(close) {
         out.close();
+        out = null;
         in.close();
+        in = null;
       }
{code} 

With the above lines if out.close() throws exception that will be thrown back to the caller and also other streams will be closed in finally-block using closeStream(..).

As per the discussions in the issue MAPREDUCE-2243, this way is followed.


> Potential Resource leak in IOUtils.java
> ---------------------------------------
>
>                 Key: HADOOP-7194
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7194
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.23.0
>            Reporter: Devaraj K
>         Attachments: HADOOP-7194.patch
>
>
> {code:title=IOUtils.java|borderStyle=solid}
> try {
>       copyBytes(in, out, buffSize);
>     } finally {
>       if(close) {
>         out.close();
>         in.close();
>       }
>     }
> {code} 
> In the above code if any exception throws from the out.close() statement, in.close() statement will not execute and the input stream will not be closed.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13761-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 28 08:38:45 2011
Return-Path: <common-issues-return-13761-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82901 invoked from network); 28 Mar 2011 08:38:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Mar 2011 08:38:44 -0000
Received: (qmail 47624 invoked by uid 500); 28 Mar 2011 08:38:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47584 invoked by uid 500); 28 Mar 2011 08:38:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47569 invoked by uid 99); 28 Mar 2011 08:38:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 08:38:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 08:38:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CDB4983B71
	for <common-issues@hadoop.apache.org>; Mon, 28 Mar 2011 08:38:05 +0000 (UTC)
Date: Mon, 28 Mar 2011 08:38:05 +0000 (UTC)
From: "Jingguo Yao (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1369222834.16242.1301301485839.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <15581131.20741291107311011.JavaMail.jira@thor>
Subject: [jira] [Updated] (HADOOP-7055) Update of commons logging libraries
 causes EventCounter to count logging events incorrectly
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jingguo Yao updated HADOOP-7055:
--------------------------------

    Status: Patch Available  (was: Open)

Per Luke's suggestion, use equals instead of == for Level check.

> Update of commons logging libraries causes EventCounter to count logging events incorrectly
> -------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7055
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7055
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.21.0, 0.21.1, 0.22.0, 0.23.0
>            Reporter: Jingguo Yao
>            Assignee: Jingguo Yao
>   Original Estimate: 0.5h
>  Remaining Estimate: 0.5h
>
> Hadoop 0.20.2 uses commons logging 1.0.4. EventCounter works correctly with this version of commons logging. Hadoop 0.21.0 uses commons logging 1.1.1 which causes EventCounter to count logging events incorrectly. I have verified it with Hadoop 0.21.0. After start-up of hadoop, I checked jvmmetrics.log after several minutes. In every metrics record, "logError=0, logFatal=0, logInfo=3, logWarn=0" was shown. The following text is an example.
> jvm.metrics: hostName=jingguolin, processName=DataNode, sessionId=, gcCount=3, gcTimeMillis=31, logError=0, logFatal=0, logInfo=3, logWarn=0, maxMemoryM=888.9375, memHeapCommittedM=38.0625, memHeapUsedM=3.6539612, memNonHeapCommittedM=18.25, memNonHeapUsedM=11.335686, threadsBlocked=0, threadsNew=0, threadsRunnable=8, threadsTerminated=0, threadsTimedWaiting=6, threadsWaiting=6
> Then I stopped hadoop and replaced commons logging 1.1.1 with 1.0.4. After the re-start of hadoop, a lot of logging events showed up in jvmmetrics.log.
> I have checked the source code of Log4JLogger for both 1.0.4 (http://svn.apache.org/viewvc/commons/proper/logging/tags/LOGGING_1_0_4/src/java/org/apache/commons/logging/impl/Log4JLogger.java?view=markup) and 1.1.1 (http://svn.apache.org/viewvc/commons/proper/logging/tags/commons-logging-1.1.1/src/java/org/apache/commons/logging/impl/Log4JLogger.java?view=markup). For 1.0.4, Level instances such as Level.INFO are used to construct LoggingEvent. But for 1.1.1, Priority instances such as Priority.INFO are used to construct LoggingEvent. So 1.1.1 version's event.getLevel() always returns Priority instances. EventCounter append method's "==" check always fails between a Level instance and a Priority instance. For "logInfo=3" metrics records produced by commons logging 1.1.1., I think that these 3 logging events are produced by log4j code directly instead of through commons logging API. The following code is EventCounter's append method.
>     public void append(LoggingEvent event) {
>         Level level = event.getLevel();
>         if (level == Level.INFO) {
>             counts.incr(INFO);
>         }
>         else if (level == Level.WARN) {
>             counts.incr(WARN);
>         }
>         else if (level == Level.ERROR) {
>             counts.incr(ERROR);
>         }
>         else if (level == Level.FATAL) {
>             counts.incr(FATAL);
>         }
>     }

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13762-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 28 08:40:46 2011
Return-Path: <common-issues-return-13762-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83211 invoked from network); 28 Mar 2011 08:40:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Mar 2011 08:40:46 -0000
Received: (qmail 48715 invoked by uid 500); 28 Mar 2011 08:40:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48682 invoked by uid 500); 28 Mar 2011 08:40:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48674 invoked by uid 99); 28 Mar 2011 08:40:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 08:40:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 08:40:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B627083C12
	for <common-issues@hadoop.apache.org>; Mon, 28 Mar 2011 08:40:05 +0000 (UTC)
Date: Mon, 28 Mar 2011 08:40:05 +0000 (UTC)
From: "Jingguo Yao (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1506744836.16246.1301301605742.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <15581131.20741291107311011.JavaMail.jira@thor>
Subject: [jira] [Updated] (HADOOP-7055) Update of commons logging libraries
 causes EventCounter to count logging events incorrectly
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jingguo Yao updated HADOOP-7055:
--------------------------------

    Attachment: HADOOP-7055.patch

> Update of commons logging libraries causes EventCounter to count logging events incorrectly
> -------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7055
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7055
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.21.0, 0.21.1, 0.22.0, 0.23.0
>            Reporter: Jingguo Yao
>            Assignee: Jingguo Yao
>         Attachments: HADOOP-7055.patch
>
>   Original Estimate: 0.5h
>  Remaining Estimate: 0.5h
>
> Hadoop 0.20.2 uses commons logging 1.0.4. EventCounter works correctly with this version of commons logging. Hadoop 0.21.0 uses commons logging 1.1.1 which causes EventCounter to count logging events incorrectly. I have verified it with Hadoop 0.21.0. After start-up of hadoop, I checked jvmmetrics.log after several minutes. In every metrics record, "logError=0, logFatal=0, logInfo=3, logWarn=0" was shown. The following text is an example.
> jvm.metrics: hostName=jingguolin, processName=DataNode, sessionId=, gcCount=3, gcTimeMillis=31, logError=0, logFatal=0, logInfo=3, logWarn=0, maxMemoryM=888.9375, memHeapCommittedM=38.0625, memHeapUsedM=3.6539612, memNonHeapCommittedM=18.25, memNonHeapUsedM=11.335686, threadsBlocked=0, threadsNew=0, threadsRunnable=8, threadsTerminated=0, threadsTimedWaiting=6, threadsWaiting=6
> Then I stopped hadoop and replaced commons logging 1.1.1 with 1.0.4. After the re-start of hadoop, a lot of logging events showed up in jvmmetrics.log.
> I have checked the source code of Log4JLogger for both 1.0.4 (http://svn.apache.org/viewvc/commons/proper/logging/tags/LOGGING_1_0_4/src/java/org/apache/commons/logging/impl/Log4JLogger.java?view=markup) and 1.1.1 (http://svn.apache.org/viewvc/commons/proper/logging/tags/commons-logging-1.1.1/src/java/org/apache/commons/logging/impl/Log4JLogger.java?view=markup). For 1.0.4, Level instances such as Level.INFO are used to construct LoggingEvent. But for 1.1.1, Priority instances such as Priority.INFO are used to construct LoggingEvent. So 1.1.1 version's event.getLevel() always returns Priority instances. EventCounter append method's "==" check always fails between a Level instance and a Priority instance. For "logInfo=3" metrics records produced by commons logging 1.1.1., I think that these 3 logging events are produced by log4j code directly instead of through commons logging API. The following code is EventCounter's append method.
>     public void append(LoggingEvent event) {
>         Level level = event.getLevel();
>         if (level == Level.INFO) {
>             counts.incr(INFO);
>         }
>         else if (level == Level.WARN) {
>             counts.incr(WARN);
>         }
>         else if (level == Level.ERROR) {
>             counts.incr(ERROR);
>         }
>         else if (level == Level.FATAL) {
>             counts.incr(FATAL);
>         }
>     }

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13763-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 28 14:48:46 2011
Return-Path: <common-issues-return-13763-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97420 invoked from network); 28 Mar 2011 14:48:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Mar 2011 14:48:46 -0000
Received: (qmail 82929 invoked by uid 500); 28 Mar 2011 14:48:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82886 invoked by uid 500); 28 Mar 2011 14:48:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82878 invoked by uid 99); 28 Mar 2011 14:48:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 14:48:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 14:48:44 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7C1C583FA1
	for <common-issues@hadoop.apache.org>; Mon, 28 Mar 2011 14:48:06 +0000 (UTC)
Date: Mon, 28 Mar 2011 14:48:06 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <756924939.16800.1301323686504.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1587140683.1515.1300732271817.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7202) Improve Command base class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7202:
--------------------------------

    Attachment: HADOOP-7202-3.patch

Fixed javadoc link warning from mis-capitalized method name.

The javac warnings are due to FsCommand.java stubbing out unnecessary abstract methods which happen to be deprecated.  The deprecations are generating the warnings.

> Improve Command base class
> --------------------------
>
>                 Key: HADOOP-7202
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7202
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7202-2.patch, HADOOP-7202-3.patch, HADOOP-7202.patch
>
>
> Need to extend the Command base class to allow all command to easily subclass from a code set of code that correctly handles globs and exit codes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13764-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 28 16:35:43 2011
Return-Path: <common-issues-return-13764-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47644 invoked from network); 28 Mar 2011 16:35:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Mar 2011 16:35:43 -0000
Received: (qmail 46942 invoked by uid 500); 28 Mar 2011 16:35:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46808 invoked by uid 500); 28 Mar 2011 16:35:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46799 invoked by uid 99); 28 Mar 2011 16:35:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 16:35:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 16:35:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BD09A846BB
	for <common-issues@hadoop.apache.org>; Mon, 28 Mar 2011 16:35:05 +0000 (UTC)
Date: Mon, 28 Mar 2011 16:35:05 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <8299358.16965.1301330105771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7210) Chown command is not working from
 FSShell.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Chown command is not working from FSShell.
------------------------------------------

                 Key: HADOOP-7210
                 URL: https://issues.apache.org/jira/browse/HADOOP-7210
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
         Environment: Linux
            Reporter: Uma Maheswara Rao G
            Assignee: Uma Maheswara Rao G


chown command is not invoking the setOwner on FileSystem.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13765-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 28 16:35:44 2011
Return-Path: <common-issues-return-13765-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47683 invoked from network); 28 Mar 2011 16:35:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Mar 2011 16:35:43 -0000
Received: (qmail 47110 invoked by uid 500); 28 Mar 2011 16:35:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47029 invoked by uid 500); 28 Mar 2011 16:35:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46999 invoked by uid 99); 28 Mar 2011 16:35:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 16:35:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 16:35:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D63E3846BF
	for <common-issues@hadoop.apache.org>; Mon, 28 Mar 2011 16:35:05 +0000 (UTC)
Date: Mon, 28 Mar 2011 16:35:05 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1377782417.16969.1301330105874.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <8299358.16965.1301330105771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7210) Chown command is not working from
 FSShell.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012127#comment-13012127 ] 

Uma Maheswara Rao G commented on HADOOP-7210:
---------------------------------------------

Based on the passed arguments, it is creating the Handlers. when argument is -chown, it is creating ChownHandler. 
{code}
else if (cmd.equals("-chown")) {
      handler = new ChownHandler(argv[startIndex++]);
{code}
But here the problem is, it is not populating owner, group values in this constructor. just it is calling supper. It is not populating the group, owner.
{code}
protected ChownHandler(String cmd) { //for chgrp
      super(cmd);
}
{code}
Since it is not populating, setOwner will not be invoked on fileSystem.So permissions will not change.
{code}
   if (newOwner != null || newGroup != null) {
        try {
          srcFs.setOwner(file.getPath(), newOwner, newGroup);
{code}


> Chown command is not working from FSShell.
> ------------------------------------------
>
>                 Key: HADOOP-7210
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7210
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>
> chown command is not invoking the setOwner on FileSystem.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13766-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 28 16:53:45 2011
Return-Path: <common-issues-return-13766-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4100 invoked from network); 28 Mar 2011 16:53:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Mar 2011 16:53:44 -0000
Received: (qmail 71101 invoked by uid 500); 28 Mar 2011 16:53:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71056 invoked by uid 500); 28 Mar 2011 16:53:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71048 invoked by uid 99); 28 Mar 2011 16:53:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 16:53:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 16:53:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C88FC84C36
	for <common-issues@hadoop.apache.org>; Mon, 28 Mar 2011 16:53:06 +0000 (UTC)
Date: Mon, 28 Mar 2011 16:53:06 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <458485421.16988.1301331186818.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <113773164.6166.1300269149586.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7194) Potential Resource leak in
 IOUtils.java
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012131#comment-13012131 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7194:
------------------------------------------------

> Closing the streams in the finally-block using closeStream(..) will suppress the exception ...

Good point.

+1 then.

> Potential Resource leak in IOUtils.java
> ---------------------------------------
>
>                 Key: HADOOP-7194
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7194
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.23.0
>            Reporter: Devaraj K
>         Attachments: HADOOP-7194.patch
>
>
> {code:title=IOUtils.java|borderStyle=solid}
> try {
>       copyBytes(in, out, buffSize);
>     } finally {
>       if(close) {
>         out.close();
>         in.close();
>       }
>     }
> {code} 
> In the above code if any exception throws from the out.close() statement, in.close() statement will not execute and the input stream will not be closed.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13767-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 28 16:53:47 2011
Return-Path: <common-issues-return-13767-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4185 invoked from network); 28 Mar 2011 16:53:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Mar 2011 16:53:46 -0000
Received: (qmail 71645 invoked by uid 500); 28 Mar 2011 16:53:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71613 invoked by uid 500); 28 Mar 2011 16:53:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71605 invoked by uid 99); 28 Mar 2011 16:53:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 16:53:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 16:53:44 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EC2F484C3B
	for <common-issues@hadoop.apache.org>; Mon, 28 Mar 2011 16:53:06 +0000 (UTC)
Date: Mon, 28 Mar 2011 16:53:06 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <300487413.16993.1301331186964.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <113773164.6166.1300269149586.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7194) Potential Resource leak in
 IOUtils.java
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7194:
-------------------------------------------

        Assignee: Devaraj K
    Hadoop Flags: [Reviewed]

> Potential Resource leak in IOUtils.java
> ---------------------------------------
>
>                 Key: HADOOP-7194
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7194
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.23.0
>            Reporter: Devaraj K
>            Assignee: Devaraj K
>         Attachments: HADOOP-7194.patch
>
>
> {code:title=IOUtils.java|borderStyle=solid}
> try {
>       copyBytes(in, out, buffSize);
>     } finally {
>       if(close) {
>         out.close();
>         in.close();
>       }
>     }
> {code} 
> In the above code if any exception throws from the out.close() statement, in.close() statement will not execute and the input stream will not be closed.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13768-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 28 17:09:46 2011
Return-Path: <common-issues-return-13768-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59462 invoked from network); 28 Mar 2011 17:09:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Mar 2011 17:09:46 -0000
Received: (qmail 91797 invoked by uid 500); 28 Mar 2011 17:09:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91772 invoked by uid 500); 28 Mar 2011 17:09:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91764 invoked by uid 99); 28 Mar 2011 17:09:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 17:09:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 17:09:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0C140841AB
	for <common-issues@hadoop.apache.org>; Mon, 28 Mar 2011 17:09:06 +0000 (UTC)
Date: Mon, 28 Mar 2011 17:09:06 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <508986984.17048.1301332146046.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <113773164.6166.1300269149586.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7194) Potential Resource leak in
 IOUtils.java
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012140#comment-13012140 ] 

Hadoop QA commented on HADOOP-7194:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12474511/HADOOP-7194.patch
  against trunk revision 1085122.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/321//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/321//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/321//console

This message is automatically generated.

> Potential Resource leak in IOUtils.java
> ---------------------------------------
>
>                 Key: HADOOP-7194
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7194
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.23.0
>            Reporter: Devaraj K
>            Assignee: Devaraj K
>         Attachments: HADOOP-7194.patch
>
>
> {code:title=IOUtils.java|borderStyle=solid}
> try {
>       copyBytes(in, out, buffSize);
>     } finally {
>       if(close) {
>         out.close();
>         in.close();
>       }
>     }
> {code} 
> In the above code if any exception throws from the out.close() statement, in.close() statement will not execute and the input stream will not be closed.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13769-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 28 17:15:47 2011
Return-Path: <common-issues-return-13769-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62835 invoked from network); 28 Mar 2011 17:15:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Mar 2011 17:15:46 -0000
Received: (qmail 97269 invoked by uid 500); 28 Mar 2011 17:15:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97217 invoked by uid 500); 28 Mar 2011 17:15:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97209 invoked by uid 99); 28 Mar 2011 17:15:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 17:15:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 17:15:44 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7F95C843FF
	for <common-issues@hadoop.apache.org>; Mon, 28 Mar 2011 17:15:06 +0000 (UTC)
Date: Mon, 28 Mar 2011 17:15:06 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1523893072.17069.1301332506519.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1587140683.1515.1300732271817.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7202) Improve Command base class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012145#comment-13012145 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7202:
------------------------------------------------

Hi Daryn,

- Could you add the annotations to the existing classes?
{code}
@InterfaceAudience.Private
@InterfaceStability.Unstable/Evolving
{code}

- Since they are private unstable interfaces, you may simply remove the methods.  Otherwise, please explain in the javadoc that why these methods are deprecated and what are the alternatives.
> The javac warnings are due to FsCommand.java stubbing out unnecessary abstract methods which happen to be deprecated. ...


- If {{FsCommand}} is not useful now, how about adding it later when it becomes useful?
{code}
+// this class may not look useful now, but it's a placeholder for future
+// functionality to act as a registry for fs commands.  currently it's being
+// used to implement unnecessary abstract methods in the base class
+
+abstract public class FsCommand extends Command {
{code}


> Improve Command base class
> --------------------------
>
>                 Key: HADOOP-7202
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7202
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7202-2.patch, HADOOP-7202-3.patch, HADOOP-7202.patch
>
>
> Need to extend the Command base class to allow all command to easily subclass from a code set of code that correctly handles globs and exit codes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13770-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 28 17:17:45 2011
Return-Path: <common-issues-return-13770-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67341 invoked from network); 28 Mar 2011 17:17:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Mar 2011 17:17:45 -0000
Received: (qmail 98531 invoked by uid 500); 28 Mar 2011 17:17:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98488 invoked by uid 500); 28 Mar 2011 17:17:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98480 invoked by uid 99); 28 Mar 2011 17:17:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 17:17:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 17:17:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AC14384472
	for <common-issues@hadoop.apache.org>; Mon, 28 Mar 2011 17:17:05 +0000 (UTC)
Date: Mon, 28 Mar 2011 17:17:05 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1649410042.17077.1301332625701.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <8299358.16965.1301330105771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7210) Chown command is not working from
 FSShell.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uma Maheswara Rao G updated HADOOP-7210:
----------------------------------------

    Attachment: HADOOP-7210.patch

> Chown command is not working from FSShell.
> ------------------------------------------
>
>                 Key: HADOOP-7210
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7210
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7210.patch
>
>
> chown command is not invoking the setOwner on FileSystem.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13771-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 28 17:40:46 2011
Return-Path: <common-issues-return-13771-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54413 invoked from network); 28 Mar 2011 17:40:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Mar 2011 17:40:45 -0000
Received: (qmail 21337 invoked by uid 500); 28 Mar 2011 17:40:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21294 invoked by uid 500); 28 Mar 2011 17:40:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21286 invoked by uid 99); 28 Mar 2011 17:40:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 17:40:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 17:40:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DC71984C43
	for <common-issues@hadoop.apache.org>; Mon, 28 Mar 2011 17:40:05 +0000 (UTC)
Date: Mon, 28 Mar 2011 17:40:05 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1016709088.17112.1301334005899.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <113773164.6166.1300269149586.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7194) Potential Resource leak in
 IOUtils.java
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7194:
-------------------------------------------

    Affects Version/s:     (was: 0.23.0)

> Potential Resource leak in IOUtils.java
> ---------------------------------------
>
>                 Key: HADOOP-7194
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7194
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Devaraj K
>            Assignee: Devaraj K
>         Attachments: HADOOP-7194.patch
>
>
> {code:title=IOUtils.java|borderStyle=solid}
> try {
>       copyBytes(in, out, buffSize);
>     } finally {
>       if(close) {
>         out.close();
>         in.close();
>       }
>     }
> {code} 
> In the above code if any exception throws from the out.close() statement, in.close() statement will not execute and the input stream will not be closed.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13772-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 28 17:42:46 2011
Return-Path: <common-issues-return-13772-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54851 invoked from network); 28 Mar 2011 17:42:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Mar 2011 17:42:46 -0000
Received: (qmail 21869 invoked by uid 500); 28 Mar 2011 17:42:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21832 invoked by uid 500); 28 Mar 2011 17:42:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21824 invoked by uid 99); 28 Mar 2011 17:42:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 17:42:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 17:42:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EF84784D16
	for <common-issues@hadoop.apache.org>; Mon, 28 Mar 2011 17:42:05 +0000 (UTC)
Date: Mon, 28 Mar 2011 17:42:05 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <632394294.17125.1301334125977.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <8299358.16965.1301330105771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7210) Chown command is not working from
 FSShell.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uma Maheswara Rao G updated HADOOP-7210:
----------------------------------------

    Status: Patch Available  (was: Open)

> Chown command is not working from FSShell.
> ------------------------------------------
>
>                 Key: HADOOP-7210
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7210
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7210.patch
>
>
> chown command is not invoking the setOwner on FileSystem.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13773-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 28 17:42:46 2011
Return-Path: <common-issues-return-13773-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54891 invoked from network); 28 Mar 2011 17:42:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Mar 2011 17:42:46 -0000
Received: (qmail 22084 invoked by uid 500); 28 Mar 2011 17:42:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22047 invoked by uid 500); 28 Mar 2011 17:42:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22039 invoked by uid 99); 28 Mar 2011 17:42:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 17:42:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 17:42:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D158E84D12
	for <common-issues@hadoop.apache.org>; Mon, 28 Mar 2011 17:42:05 +0000 (UTC)
Date: Mon, 28 Mar 2011 17:42:05 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1795064646.17121.1301334125854.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <113773164.6166.1300269149586.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7194) Potential Resource leak in
 IOUtils.java
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7194:
-------------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
                   0.22.0
                   0.21.1
           Status: Resolved  (was: Patch Available)

I have committed this.  Thanks, Devaraj!

> Potential Resource leak in IOUtils.java
> ---------------------------------------
>
>                 Key: HADOOP-7194
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7194
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Devaraj K
>            Assignee: Devaraj K
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HADOOP-7194.patch
>
>
> {code:title=IOUtils.java|borderStyle=solid}
> try {
>       copyBytes(in, out, buffSize);
>     } finally {
>       if(close) {
>         out.close();
>         in.close();
>       }
>     }
> {code} 
> In the above code if any exception throws from the out.close() statement, in.close() statement will not execute and the input stream will not be closed.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13774-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 28 17:54:44 2011
Return-Path: <common-issues-return-13774-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 831 invoked from network); 28 Mar 2011 17:54:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Mar 2011 17:54:43 -0000
Received: (qmail 32407 invoked by uid 500); 28 Mar 2011 17:54:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32361 invoked by uid 500); 28 Mar 2011 17:54:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32353 invoked by uid 99); 28 Mar 2011 17:54:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 17:54:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 17:54:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CC08484047
	for <common-issues@hadoop.apache.org>; Mon, 28 Mar 2011 17:54:05 +0000 (UTC)
Date: Mon, 28 Mar 2011 17:54:05 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1822783219.17146.1301334845832.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <113773164.6166.1300269149586.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7194) Potential Resource leak in
 IOUtils.java
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012156#comment-13012156 ] 

Hudson commented on HADOOP-7194:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #536 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/536/])
    HADOOP-7194. Fix resource leak in IOUtils.copyBytes(..).  Contributed by Devaraj K


> Potential Resource leak in IOUtils.java
> ---------------------------------------
>
>                 Key: HADOOP-7194
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7194
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Devaraj K
>            Assignee: Devaraj K
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HADOOP-7194.patch
>
>
> {code:title=IOUtils.java|borderStyle=solid}
> try {
>       copyBytes(in, out, buffSize);
>     } finally {
>       if(close) {
>         out.close();
>         in.close();
>       }
>     }
> {code} 
> In the above code if any exception throws from the out.close() statement, in.close() statement will not execute and the input stream will not be closed.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13775-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 28 18:45:44 2011
Return-Path: <common-issues-return-13775-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67636 invoked from network); 28 Mar 2011 18:45:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Mar 2011 18:45:43 -0000
Received: (qmail 3588 invoked by uid 500); 28 Mar 2011 18:45:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3520 invoked by uid 500); 28 Mar 2011 18:45:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3512 invoked by uid 99); 28 Mar 2011 18:45:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 18:45:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 18:45:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F398484F08
	for <common-issues@hadoop.apache.org>; Mon, 28 Mar 2011 18:45:05 +0000 (UTC)
Date: Mon, 28 Mar 2011 18:45:05 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1118189672.17250.1301337905994.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1224526256.2676.1300746425823.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7204) remove local unused fs variable
 from CmdHandler and FsShellPermissions.changePermissions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012171#comment-13012171 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7204:
------------------------------------------------

Seems that this caused the recent failure of {{TestHDFSCLI}} (e.g. [build #293|https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/293//testReport/org.apache.hadoop.cli/TestHDFSCLI/testAll/]).  Could you take a look?


> remove local unused fs variable from CmdHandler and FsShellPermissions.changePermissions
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7204
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7204
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>            Priority: Minor
>         Attachments: HADOOP-7204.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13776-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 28 19:13:44 2011
Return-Path: <common-issues-return-13776-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68781 invoked from network); 28 Mar 2011 19:13:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Mar 2011 19:13:43 -0000
Received: (qmail 45724 invoked by uid 500); 28 Mar 2011 19:13:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45693 invoked by uid 500); 28 Mar 2011 19:13:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45685 invoked by uid 99); 28 Mar 2011 19:13:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 19:13:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 19:13:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CF980847D2
	for <common-issues@hadoop.apache.org>; Mon, 28 Mar 2011 19:13:05 +0000 (UTC)
Date: Mon, 28 Mar 2011 19:13:05 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <885549230.17275.1301339585847.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <8299358.16965.1301330105771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7210) Chown command is not working from
 FSShell.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012180#comment-13012180 ] 

Daryn Sharp commented on HADOOP-7210:
-------------------------------------

The root problem is that a prior bug that remove the FileSystem var from the ctors.  This left two ctors with a String signature.  Changing one of the three commands to have a ctor of (String, String) just for the purpose of working around the ambiguity doesn't seem right.

My suggestion would be to split the user/group mode parsing into a separate parseOwnerGroup (or similar) method.  Chown's ctor calls this new method with its own impl, Chgrp overrides the method with it's own impl.

> Chown command is not working from FSShell.
> ------------------------------------------
>
>                 Key: HADOOP-7210
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7210
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7210.patch
>
>
> chown command is not invoking the setOwner on FileSystem.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13777-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 28 19:45:44 2011
Return-Path: <common-issues-return-13777-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68860 invoked from network); 28 Mar 2011 19:45:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Mar 2011 19:45:43 -0000
Received: (qmail 80265 invoked by uid 500); 28 Mar 2011 19:45:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80165 invoked by uid 500); 28 Mar 2011 19:45:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80154 invoked by uid 99); 28 Mar 2011 19:45:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 19:45:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 19:45:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F13DB84320
	for <common-issues@hadoop.apache.org>; Mon, 28 Mar 2011 19:45:05 +0000 (UTC)
Date: Mon, 28 Mar 2011 19:45:05 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1900855498.17331.1301341505984.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1587140683.1515.1300732271817.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7202) Improve Command base class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7202:
--------------------------------

    Attachment: HADOOP-7202-4.patch

Added annotations for audience and stability.    

I am unable to remove the abstract methods that I marked as deprecated (and then stubbed in FsCommand) because other tools are depending on this class.  I'm relatively certain the new apis are firm, but I'd now like to hold off on adding the deprecations until FsShell is completed.

> Improve Command base class
> --------------------------
>
>                 Key: HADOOP-7202
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7202
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7202-2.patch, HADOOP-7202-3.patch, HADOOP-7202-4.patch, HADOOP-7202.patch
>
>
> Need to extend the Command base class to allow all command to easily subclass from a code set of code that correctly handles globs and exit codes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13778-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 28 21:50:46 2011
Return-Path: <common-issues-return-13778-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5511 invoked from network); 28 Mar 2011 21:50:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Mar 2011 21:50:45 -0000
Received: (qmail 45269 invoked by uid 500); 28 Mar 2011 21:50:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45237 invoked by uid 500); 28 Mar 2011 21:50:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45229 invoked by uid 99); 28 Mar 2011 21:50:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 21:50:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 21:50:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B6F6B84AB2
	for <common-issues@hadoop.apache.org>; Mon, 28 Mar 2011 21:50:05 +0000 (UTC)
Date: Mon, 28 Mar 2011 21:50:05 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <444835630.17630.1301349005745.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <113773164.6166.1300269149586.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7194) Potential Resource leak in
 IOUtils.java
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012245#comment-13012245 ] 

Hudson commented on HADOOP-7194:
--------------------------------

Integrated in Hadoop-Common-22-branch #37 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-22-branch/37/])
    HADOOP-7194. Fix resource leak in IOUtils.copyBytes(..).  Contributed by Devaraj K


> Potential Resource leak in IOUtils.java
> ---------------------------------------
>
>                 Key: HADOOP-7194
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7194
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Devaraj K
>            Assignee: Devaraj K
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HADOOP-7194.patch
>
>
> {code:title=IOUtils.java|borderStyle=solid}
> try {
>       copyBytes(in, out, buffSize);
>     } finally {
>       if(close) {
>         out.close();
>         in.close();
>       }
>     }
> {code} 
> In the above code if any exception throws from the out.close() statement, in.close() statement will not execute and the input stream will not be closed.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13779-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 28 21:54:46 2011
Return-Path: <common-issues-return-13779-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6033 invoked from network); 28 Mar 2011 21:54:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Mar 2011 21:54:45 -0000
Received: (qmail 49101 invoked by uid 500); 28 Mar 2011 21:54:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49066 invoked by uid 500); 28 Mar 2011 21:54:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49058 invoked by uid 99); 28 Mar 2011 21:54:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 21:54:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 21:54:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C237484BCB
	for <common-issues@hadoop.apache.org>; Mon, 28 Mar 2011 21:54:05 +0000 (UTC)
Date: Mon, 28 Mar 2011 21:54:05 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2126131872.17638.1301349245792.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7211) Security uses proprietary Sun APIs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Security uses proprietary Sun APIs
----------------------------------

                 Key: HADOOP-7211
                 URL: https://issues.apache.org/jira/browse/HADOOP-7211
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Eli Collins


The security code uses the KrbException, Credentials, and PrincipalName classes from sun.security.krb5 and Krb5Util from sun.security.jgss.krb5. These may disappear in future Java releases. Also Hadoop does not compile using jdks that do not support them, for example with the following IBM JDK.

{noformat}
$ /home/eli/toolchain/java-x86_64-60/bin/java -version
java version "1.6.0"
Java(TM) SE Runtime Environment (build pxa6460sr9fp1-20110208_03(SR9 FP1))
IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64 jvmxa6460sr9-20110203_74623 (JIT enabled, AOT enabled)
J9VM - 20110203_074623
JIT  - r9_20101028_17488ifx3
GC   - 20101027_AA)
JCL  - 20110203_01
{noformat}


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13780-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 28 23:01:44 2011
Return-Path: <common-issues-return-13780-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54547 invoked from network); 28 Mar 2011 23:01:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Mar 2011 23:01:43 -0000
Received: (qmail 17858 invoked by uid 500); 28 Mar 2011 23:01:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17826 invoked by uid 500); 28 Mar 2011 23:01:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17818 invoked by uid 99); 28 Mar 2011 23:01:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 23:01:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 23:01:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C3F7984F98
	for <common-issues@hadoop.apache.org>; Mon, 28 Mar 2011 23:01:05 +0000 (UTC)
Date: Mon, 28 Mar 2011 23:01:05 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <297669161.17734.1301353265784.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1587140683.1515.1300732271817.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7202) Improve Command base class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012264#comment-13012264 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7202:
------------------------------------------------

+1
Thanks for making the changes.  Having {{FsCommand}} makes sense.

> Improve Command base class
> --------------------------
>
>                 Key: HADOOP-7202
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7202
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7202-2.patch, HADOOP-7202-3.patch, HADOOP-7202-4.patch, HADOOP-7202.patch
>
>
> Need to extend the Command base class to allow all command to easily subclass from a code set of code that correctly handles globs and exit codes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13781-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 28 23:17:44 2011
Return-Path: <common-issues-return-13781-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7605 invoked from network); 28 Mar 2011 23:17:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Mar 2011 23:17:43 -0000
Received: (qmail 30305 invoked by uid 500); 28 Mar 2011 23:17:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30258 invoked by uid 500); 28 Mar 2011 23:17:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30118 invoked by uid 99); 28 Mar 2011 23:17:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 23:17:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 23:17:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CADE2843E7
	for <common-issues@hadoop.apache.org>; Mon, 28 Mar 2011 23:17:05 +0000 (UTC)
Date: Mon, 28 Mar 2011 23:17:05 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <213172196.17762.1301354225827.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1587140683.1515.1300732271817.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7202) Improve Command base class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012269#comment-13012269 ] 

Hadoop QA commented on HADOOP-7202:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12474804/HADOOP-7202-4.patch
  against trunk revision 1086309.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/322//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/322//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/322//console

This message is automatically generated.

> Improve Command base class
> --------------------------
>
>                 Key: HADOOP-7202
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7202
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7202-2.patch, HADOOP-7202-3.patch, HADOOP-7202-4.patch, HADOOP-7202.patch
>
>
> Need to extend the Command base class to allow all command to easily subclass from a code set of code that correctly handles globs and exit codes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13782-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Mar 28 23:50:46 2011
Return-Path: <common-issues-return-13782-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36048 invoked from network); 28 Mar 2011 23:50:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Mar 2011 23:50:46 -0000
Received: (qmail 56428 invoked by uid 500); 28 Mar 2011 23:50:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56389 invoked by uid 500); 28 Mar 2011 23:50:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56381 invoked by uid 99); 28 Mar 2011 23:50:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 23:50:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Mar 2011 23:50:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 433B084CF2
	for <common-issues@hadoop.apache.org>; Mon, 28 Mar 2011 23:50:06 +0000 (UTC)
Date: Mon, 28 Mar 2011 23:50:06 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1938837271.17808.1301356206272.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1587140683.1515.1300732271817.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7202) Improve Command base class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7202:
-------------------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I have committed this.  Thanks, Daryn!

Also, thanks Tanping for the review.

> Improve Command base class
> --------------------------
>
>                 Key: HADOOP-7202
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7202
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7202-2.patch, HADOOP-7202-3.patch, HADOOP-7202-4.patch, HADOOP-7202.patch
>
>
> Need to extend the Command base class to allow all command to easily subclass from a code set of code that correctly handles globs and exit codes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13783-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 29 00:04:44 2011
Return-Path: <common-issues-return-13783-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75639 invoked from network); 29 Mar 2011 00:04:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Mar 2011 00:04:43 -0000
Received: (qmail 67154 invoked by uid 500); 29 Mar 2011 00:04:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67120 invoked by uid 500); 29 Mar 2011 00:04:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67112 invoked by uid 99); 29 Mar 2011 00:04:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 00:04:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 00:04:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0D44F84080
	for <common-issues@hadoop.apache.org>; Tue, 29 Mar 2011 00:04:06 +0000 (UTC)
Date: Tue, 29 Mar 2011 00:04:06 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <125802827.17827.1301357046051.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1587140683.1515.1300732271817.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7202) Improve Command base class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012281#comment-13012281 ] 

Hudson commented on HADOOP-7202:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #537 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/537/])
    HADOOP-7202. Improve shell Command base class.  Contributed by Daryn Sharp


> Improve Command base class
> --------------------------
>
>                 Key: HADOOP-7202
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7202
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7202-2.patch, HADOOP-7202-3.patch, HADOOP-7202-4.patch, HADOOP-7202.patch
>
>
> Need to extend the Command base class to allow all command to easily subclass from a code set of code that correctly handles globs and exit codes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13784-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 29 00:36:43 2011
Return-Path: <common-issues-return-13784-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88782 invoked from network); 29 Mar 2011 00:36:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Mar 2011 00:36:43 -0000
Received: (qmail 3228 invoked by uid 500); 29 Mar 2011 00:36:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3173 invoked by uid 500); 29 Mar 2011 00:36:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3165 invoked by uid 99); 29 Mar 2011 00:36:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 00:36:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 00:36:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F115584929
	for <common-issues@hadoop.apache.org>; Tue, 29 Mar 2011 00:36:05 +0000 (UTC)
Date: Tue, 29 Mar 2011 00:36:05 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1488394173.17885.1301358965984.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1587140683.1515.1300732271817.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Reopened] (HADOOP-7202) Improve Command base class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE reopened HADOOP-7202:
--------------------------------------------


HDFS cannot be compiled after this.

> Improve Command base class
> --------------------------
>
>                 Key: HADOOP-7202
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7202
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7202-2.patch, HADOOP-7202-3.patch, HADOOP-7202-4.patch, HADOOP-7202.patch
>
>
> Need to extend the Command base class to allow all command to easily subclass from a code set of code that correctly handles globs and exit codes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13785-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 29 00:42:46 2011
Return-Path: <common-issues-return-13785-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89785 invoked from network); 29 Mar 2011 00:42:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Mar 2011 00:42:46 -0000
Received: (qmail 10467 invoked by uid 500); 29 Mar 2011 00:42:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10438 invoked by uid 500); 29 Mar 2011 00:42:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10430 invoked by uid 99); 29 Mar 2011 00:42:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 00:42:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 00:42:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DDD6784B18
	for <common-issues@hadoop.apache.org>; Tue, 29 Mar 2011 00:42:05 +0000 (UTC)
Date: Tue, 29 Mar 2011 00:42:05 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2028245730.17899.1301359325905.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2126131872.17638.1301349245792.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7211) Security uses proprietary Sun APIs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7211:
--------------------------------

    Component/s: security

> Security uses proprietary Sun APIs
> ----------------------------------
>
>                 Key: HADOOP-7211
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7211
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Eli Collins
>
> The security code uses the KrbException, Credentials, and PrincipalName classes from sun.security.krb5 and Krb5Util from sun.security.jgss.krb5. These may disappear in future Java releases. Also Hadoop does not compile using jdks that do not support them, for example with the following IBM JDK.
> {noformat}
> $ /home/eli/toolchain/java-x86_64-60/bin/java -version
> java version "1.6.0"
> Java(TM) SE Runtime Environment (build pxa6460sr9fp1-20110208_03(SR9 FP1))
> IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64 jvmxa6460sr9-20110203_74623 (JIT enabled, AOT enabled)
> J9VM - 20110203_074623
> JIT  - r9_20101028_17488ifx3
> GC   - 20101027_AA)
> JCL  - 20110203_01
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13786-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 29 00:44:43 2011
Return-Path: <common-issues-return-13786-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90052 invoked from network); 29 Mar 2011 00:44:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Mar 2011 00:44:43 -0000
Received: (qmail 12476 invoked by uid 500); 29 Mar 2011 00:44:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12452 invoked by uid 500); 29 Mar 2011 00:44:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12439 invoked by uid 99); 29 Mar 2011 00:44:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 00:44:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 00:44:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CABC984BC5
	for <common-issues@hadoop.apache.org>; Tue, 29 Mar 2011 00:44:05 +0000 (UTC)
Date: Tue, 29 Mar 2011 00:44:05 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1053147547.17906.1301359445827.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1587140683.1515.1300732271817.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7202) Improve Command base class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012297#comment-13012297 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7202:
------------------------------------------------

I have temporarily reverted this.

> Improve Command base class
> --------------------------
>
>                 Key: HADOOP-7202
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7202
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7202-2.patch, HADOOP-7202-3.patch, HADOOP-7202-4.patch, HADOOP-7202.patch
>
>
> Need to extend the Command base class to allow all command to easily subclass from a code set of code that correctly handles globs and exit codes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13787-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 29 00:54:43 2011
Return-Path: <common-issues-return-13787-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35058 invoked from network); 29 Mar 2011 00:54:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Mar 2011 00:54:43 -0000
Received: (qmail 24573 invoked by uid 500); 29 Mar 2011 00:54:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24542 invoked by uid 500); 29 Mar 2011 00:54:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24534 invoked by uid 99); 29 Mar 2011 00:54:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 00:54:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 00:54:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 015C584E4E
	for <common-issues@hadoop.apache.org>; Tue, 29 Mar 2011 00:54:06 +0000 (UTC)
Date: Tue, 29 Mar 2011 00:54:06 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1814788763.17918.1301360046002.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1587140683.1515.1300732271817.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7202) Improve Command base class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012299#comment-13012299 ] 

Hudson commented on HADOOP-7202:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #538 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/538/])
    Revert HADOOP-7202 since HDFS cannot be compiled.


> Improve Command base class
> --------------------------
>
>                 Key: HADOOP-7202
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7202
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7202-2.patch, HADOOP-7202-3.patch, HADOOP-7202-4.patch, HADOOP-7202.patch
>
>
> Need to extend the Command base class to allow all command to easily subclass from a code set of code that correctly handles globs and exit codes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13789-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 29 04:18:45 2011
Return-Path: <common-issues-return-13789-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71420 invoked from network); 29 Mar 2011 04:18:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Mar 2011 04:18:44 -0000
Received: (qmail 74181 invoked by uid 500); 29 Mar 2011 04:18:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73916 invoked by uid 500); 29 Mar 2011 04:18:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73900 invoked by uid 99); 29 Mar 2011 04:18:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 04:18:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 04:18:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E2EB7857FB
	for <common-issues@hadoop.apache.org>; Tue, 29 Mar 2011 04:18:05 +0000 (UTC)
Date: Tue, 29 Mar 2011 04:18:05 +0000 (UTC)
From: "Devaraj K (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2028196840.18129.1301372285926.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <49961760.7518.1296748349151.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7130) Map Reduce Tasks are continously
 failing, when one among the several harddisks available on the TaskTracker
 fails.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj K reassigned HADOOP-7130:
---------------------------------

    Assignee: Devaraj K

> Map Reduce Tasks are continously failing, when one among the several harddisks available on the TaskTracker fails.
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7130
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7130
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.2, 0.20.3, 0.20-append
>            Reporter: Devaraj K
>            Assignee: Devaraj K
>             Fix For: 0.20.4
>
>         Attachments: HADOOP-7130.patch
>
>
> 1. Pull out one hard disk from Task tracker node (out of 10 disks pull one). Now it is noted that some jobs are failing. 
> However process is continued. 
> 2. Wait for sometime (15 mins) and pull out one disk from another Task tracker. 
> 3. More number of jobs failed now and it can be seen from UI. Process is getting paused.
> The exception can be seen in the job tracker UI for a failed job.
> {code:xml} 
> Error initializing attempt_201010221528_10174_m_000011_0:
> java.io.IOException: Expecting a line not the end of stream
>  at org.apache.hadoop.fs.DF.parseExecResult(DF.java:110)
>  at org.apache.hadoop.util.Shell.runCommand(Shell.java:182)
>  at org.apache.hadoop.util.Shell.run(Shell.java:137)
>  at org.apache.hadoop.fs.DF.getAvailable(DF.java:74)
>  at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:385)
>  at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:134)
>  at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:113)
>  at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:835)
>  at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1790)
>  at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:104)
>  at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1753)
> Error initializing attempt_201010221528_10174_m_000011_1:
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for taskTracker/jobcache/job_201010221528_10174/work
>  at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:454)
>  at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:134)
>  at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:113)
>  at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:835)
>  at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1790)
>  at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:104)
>  at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1753)
> {code} 
> Task Tracker log can be seen here :
> {code:xml} 
> 2010-10-25 16:36:24,215 ERROR mapred.TaskTracker (TaskTracker.java:offerService(1211)) - Caught exception: java.io.IOException: Expecting a line not the end of stream
>         at org.apache.hadoop.fs.DF.parseExecResult(DF.java:110)
>         at org.apache.hadoop.util.Shell.runCommand(Shell.java:182)
>         at org.apache.hadoop.util.Shell.run(Shell.java:137)
>         at org.apache.hadoop.fs.DF.getAvailable(DF.java:74)
>         at org.apache.hadoop.mapred.TaskTracker.getFreeSpace(TaskTracker.java:1586)
>         at org.apache.hadoop.mapred.TaskTracker.transmitHeartBeat(TaskTracker.java:1274)
>         at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1106)
>         at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:1848)
>         at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3022)
> 2010-10-25 16:36:24,216 INFO  mapred.TaskTracker (TaskTracker.java:run(1856)) - Lost connection to JobTracker [/192.168.97.1:9001].  Retrying...
> java.lang.Exception: java.io.IOException: Expecting a line not the end of stream
>         at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1212)
>         at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:1848)
>         at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3022)
> Caused by: java.io.IOException: Expecting a line not the end of stream
>         at org.apache.hadoop.fs.DF.parseExecResult(DF.java:110)
>         at org.apache.hadoop.util.Shell.runCommand(Shell.java:182)
>         at org.apache.hadoop.util.Shell.run(Shell.java:137)
>         at org.apache.hadoop.fs.DF.getAvailable(DF.java:74)
>         at org.apache.hadoop.mapred.TaskTracker.getFreeSpace(TaskTracker.java:1586)
>         at org.apache.hadoop.mapred.TaskTracker.transmitHeartBeat(TaskTracker.java:1274)
>         at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1106)
>         ... 2 more
> 2010-10-25 16:36:29,550 INFO  mapred.TaskTracker (TaskTracker.java:transmitHeartBeat(1256)) - Resending 'status' to '192.168.97.1' with reponseId '18361
> 2010-10-25 16:36:29,550 WARN  mapred.TaskTracker (TaskTracker.java:checkLocalDirs(2982)) - Task Tracker local can not create directory: /hdfsdata/0/mapred/local
> 2010-10-25 16:36:32,656 WARN  mapred.TaskTracker (TaskTracker.java:checkLocalDirs(2982)) - Task Tracker local can not create directory: /hdfsdata/0/mapred/local
> {code} 
> This seems to be fixed in the trunk.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13788-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 29 04:18:44 2011
Return-Path: <common-issues-return-13788-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71388 invoked from network); 29 Mar 2011 04:18:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Mar 2011 04:18:44 -0000
Received: (qmail 74057 invoked by uid 500); 29 Mar 2011 04:18:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73913 invoked by uid 500); 29 Mar 2011 04:18:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73870 invoked by uid 99); 29 Mar 2011 04:18:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 04:18:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 04:18:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F2B8C857FD
	for <common-issues@hadoop.apache.org>; Tue, 29 Mar 2011 04:18:05 +0000 (UTC)
Date: Tue, 29 Mar 2011 04:18:05 +0000 (UTC)
From: "Devaraj K (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <542366838.18131.1301372285991.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1687965.159931294234907446.JavaMail.jira@thor>
Subject: [jira] [Assigned] (HADOOP-7086) Retrying socket connection failure
 times can be made as configurable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj K reassigned HADOOP-7086:
---------------------------------

    Assignee: Devaraj K

> Retrying socket connection failure times can be made as configurable
> --------------------------------------------------------------------
>
>                 Key: HADOOP-7086
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7086
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>    Affects Versions: 0.21.0
>         Environment: NA
>            Reporter: Devaraj K
>            Assignee: Devaraj K
>            Priority: Minor
>         Attachments: common-3899.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> Retrying socket connection failure times are hard coded as 45 and it is giving the retryring message for 45 times as below. 
> 2011-01-04 15:14:30,700 INFO ipc.Client (Client.java:handleConnectionFailure(487)) - Retrying connect to server: /10.18.52.124:50020. Already tried 1 time(s).
> This can be made as configurable and also we can keep the default value as 45. If the user wants to decrease/increase,  they can add this configurable property otherwise it can continue with the default value.
> common\src\java\org\apache\hadoop\ipc\Client.java:
> -----------------------------------------------------------------------
> private synchronized void setupConnection() throws IOException {
>       short ioFailures = 0;
>       short timeoutFailures = 0;
>       while (true) {
>         try {
>           this.socket = socketFactory.createSocket();
>           this.socket.setTcpNoDelay(tcpNoDelay);
>           // connection time out is 20s
>           NetUtils.connect(this.socket, remoteId.getAddress(), 20000);
>           if (rpcTimeout > 0) {
>             pingInterval = rpcTimeout;  // rpcTimeout overwrites pingInterval
>           }
>           this.socket.setSoTimeout(pingInterval);
>           return;
>         } catch (SocketTimeoutException toe) {
>           /*
>            * The max number of retries is 45, which amounts to 20s*45 = 15
>            * minutes retries.
>            */
>           handleConnectionFailure(timeoutFailures++, 45, toe);
>         } catch (IOException ie) {
>           handleConnectionFailure(ioFailures++, maxRetries, ie);
>         }
>       }
>     }

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13790-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 29 09:58:45 2011
Return-Path: <common-issues-return-13790-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88954 invoked from network); 29 Mar 2011 09:58:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Mar 2011 09:58:44 -0000
Received: (qmail 19893 invoked by uid 500); 29 Mar 2011 09:58:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19861 invoked by uid 500); 29 Mar 2011 09:58:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19853 invoked by uid 99); 29 Mar 2011 09:58:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 09:58:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 09:58:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DB1F1858B0
	for <common-issues@hadoop.apache.org>; Tue, 29 Mar 2011 09:58:05 +0000 (UTC)
Date: Tue, 29 Mar 2011 09:58:05 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1194888740.18577.1301392685894.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <8299358.16965.1301330105771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7210) Chown command is not working from
 FSShell.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012415#comment-13012415 ] 

Uma Maheswara Rao G commented on HADOOP-7210:
---------------------------------------------

Hi Daryn,
Thanks for your comments.
I agree with you. But I have one suggestion. We can directly override the run method in Chown and Chgrp directly from CmdHandler. Currently the ChgrpHandler extends the ChownHandler thereby use the same run method.  This might have been done to save some lines of code. But because of this, we are providing additional constructor and additional if checks in the code.

	For example, owner details check is not applicable for chgrp command. The owner will be always null, still we will execute the owner checks for ChgrepHandler also.

{code}
  String newOwner = (owner == null || owner.equals(file.getOwner())) ?
                        null : owner;
  ....
  ....
  if (newOwner != null || newGroup != null) {
{code}

Instead of this, Both Chown and Chgrp handlers can directly extend from CmdHandler provide the run method. 

Please tell me your suggestion on this.


> Chown command is not working from FSShell.
> ------------------------------------------
>
>                 Key: HADOOP-7210
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7210
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7210.patch
>
>
> chown command is not invoking the setOwner on FileSystem.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13791-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 29 11:14:46 2011
Return-Path: <common-issues-return-13791-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36372 invoked from network); 29 Mar 2011 11:14:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Mar 2011 11:14:46 -0000
Received: (qmail 4554 invoked by uid 500); 29 Mar 2011 11:14:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4518 invoked by uid 500); 29 Mar 2011 11:14:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4503 invoked by uid 99); 29 Mar 2011 11:14:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 11:14:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 11:14:44 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 82E1F852E9
	for <common-issues@hadoop.apache.org>; Tue, 29 Mar 2011 11:14:06 +0000 (UTC)
Date: Tue, 29 Mar 2011 11:14:06 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <141695368.18655.1301397246532.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <113773164.6166.1300269149586.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7194) Potential Resource leak in
 IOUtils.java
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012428#comment-13012428 ] 

Hudson commented on HADOOP-7194:
--------------------------------

Integrated in Hadoop-Common-trunk #645 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/645/])
    HADOOP-7194. Fix resource leak in IOUtils.copyBytes(..).  Contributed by Devaraj K


> Potential Resource leak in IOUtils.java
> ---------------------------------------
>
>                 Key: HADOOP-7194
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7194
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Devaraj K
>            Assignee: Devaraj K
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HADOOP-7194.patch
>
>
> {code:title=IOUtils.java|borderStyle=solid}
> try {
>       copyBytes(in, out, buffSize);
>     } finally {
>       if(close) {
>         out.close();
>         in.close();
>       }
>     }
> {code} 
> In the above code if any exception throws from the out.close() statement, in.close() statement will not execute and the input stream will not be closed.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13792-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 29 11:14:47 2011
Return-Path: <common-issues-return-13792-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36394 invoked from network); 29 Mar 2011 11:14:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Mar 2011 11:14:46 -0000
Received: (qmail 4680 invoked by uid 500); 29 Mar 2011 11:14:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4521 invoked by uid 500); 29 Mar 2011 11:14:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4504 invoked by uid 99); 29 Mar 2011 11:14:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 11:14:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 11:14:44 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 89CED852EB
	for <common-issues@hadoop.apache.org>; Tue, 29 Mar 2011 11:14:06 +0000 (UTC)
Date: Tue, 29 Mar 2011 11:14:06 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2062169100.18656.1301397246561.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1587140683.1515.1300732271817.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7202) Improve Command base class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012429#comment-13012429 ] 

Hudson commented on HADOOP-7202:
--------------------------------

Integrated in Hadoop-Common-trunk #645 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/645/])
    Revert HADOOP-7202 since HDFS cannot be compiled.
HADOOP-7202. Improve shell Command base class.  Contributed by Daryn Sharp


> Improve Command base class
> --------------------------
>
>                 Key: HADOOP-7202
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7202
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7202-2.patch, HADOOP-7202-3.patch, HADOOP-7202-4.patch, HADOOP-7202.patch
>
>
> Need to extend the Command base class to allow all command to easily subclass from a code set of code that correctly handles globs and exit codes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13793-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 29 12:53:45 2011
Return-Path: <common-issues-return-13793-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80186 invoked from network); 29 Mar 2011 12:53:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Mar 2011 12:53:45 -0000
Received: (qmail 61075 invoked by uid 500); 29 Mar 2011 12:53:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61042 invoked by uid 500); 29 Mar 2011 12:53:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61034 invoked by uid 99); 29 Mar 2011 12:53:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 12:53:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 12:53:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DF52485C1A
	for <common-issues@hadoop.apache.org>; Tue, 29 Mar 2011 12:53:05 +0000 (UTC)
Date: Tue, 29 Mar 2011 12:53:05 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <39749085.18804.1301403185911.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <8299358.16965.1301330105771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7210) Chown command is not working from
 FSShell.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012463#comment-13012463 ] 

Daryn Sharp commented on HADOOP-7210:
-------------------------------------

Good idea, I think that would work too.  In that case, would chgrp even need to inherit from chown?

> Chown command is not working from FSShell.
> ------------------------------------------
>
>                 Key: HADOOP-7210
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7210
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7210.patch
>
>
> chown command is not invoking the setOwner on FileSystem.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13794-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 29 13:13:44 2011
Return-Path: <common-issues-return-13794-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36768 invoked from network); 29 Mar 2011 13:13:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Mar 2011 13:13:43 -0000
Received: (qmail 91288 invoked by uid 500); 29 Mar 2011 13:13:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91256 invoked by uid 500); 29 Mar 2011 13:13:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91248 invoked by uid 99); 29 Mar 2011 13:13:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 13:13:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 13:13:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DFE4786388
	for <common-issues@hadoop.apache.org>; Tue, 29 Mar 2011 13:13:05 +0000 (UTC)
Date: Tue, 29 Mar 2011 13:13:05 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <968846640.18826.1301404385913.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <8299358.16965.1301330105771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7210) Chown command is not working from
 FSShell.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012469#comment-13012469 ] 

Uma Maheswara Rao G commented on HADOOP-7210:
---------------------------------------------

In this case ChgrpHandler need not extend the ChownHandler 


> Chown command is not working from FSShell.
> ------------------------------------------
>
>                 Key: HADOOP-7210
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7210
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7210.patch
>
>
> chown command is not invoking the setOwner on FileSystem.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13795-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 29 15:51:44 2011
Return-Path: <common-issues-return-13795-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76216 invoked from network); 29 Mar 2011 15:51:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Mar 2011 15:51:43 -0000
Received: (qmail 18386 invoked by uid 500); 29 Mar 2011 15:51:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18348 invoked by uid 500); 29 Mar 2011 15:51:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18340 invoked by uid 99); 29 Mar 2011 15:51:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 15:51:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 15:51:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1762586129
	for <common-issues@hadoop.apache.org>; Tue, 29 Mar 2011 15:51:06 +0000 (UTC)
Date: Tue, 29 Mar 2011 15:51:06 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1203206855.19131.1301413866092.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <8299358.16965.1301330105771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7210) Chown command is not working from
 FSShell.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uma Maheswara Rao G updated HADOOP-7210:
----------------------------------------

    Attachment: HADOOP-7210.1.patch

> Chown command is not working from FSShell.
> ------------------------------------------
>
>                 Key: HADOOP-7210
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7210
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7210.1.patch, HADOOP-7210.patch
>
>
> chown command is not invoking the setOwner on FileSystem.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13796-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 29 15:57:45 2011
Return-Path: <common-issues-return-13796-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78800 invoked from network); 29 Mar 2011 15:57:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Mar 2011 15:57:45 -0000
Received: (qmail 28422 invoked by uid 500); 29 Mar 2011 15:57:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28385 invoked by uid 500); 29 Mar 2011 15:57:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28351 invoked by uid 99); 29 Mar 2011 15:57:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 15:57:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 15:57:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 241D88639A
	for <common-issues@hadoop.apache.org>; Tue, 29 Mar 2011 15:57:06 +0000 (UTC)
Date: Tue, 29 Mar 2011 15:57:06 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <590607924.19157.1301414226142.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <8299358.16965.1301330105771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7210) Chown command is not working from
 FSShell.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012532#comment-13012532 ] 

Daryn Sharp commented on HADOOP-7210:
-------------------------------------

+1 Nice work

> Chown command is not working from FSShell.
> ------------------------------------------
>
>                 Key: HADOOP-7210
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7210
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7210.1.patch, HADOOP-7210.patch
>
>
> chown command is not invoking the setOwner on FileSystem.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13797-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 29 17:53:46 2011
Return-Path: <common-issues-return-13797-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70806 invoked from network); 29 Mar 2011 17:53:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Mar 2011 17:53:46 -0000
Received: (qmail 52033 invoked by uid 500); 29 Mar 2011 17:53:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51986 invoked by uid 500); 29 Mar 2011 17:53:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51978 invoked by uid 99); 29 Mar 2011 17:53:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 17:53:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 17:53:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5ADCB86423
	for <common-issues@hadoop.apache.org>; Tue, 29 Mar 2011 17:53:06 +0000 (UTC)
Date: Tue, 29 Mar 2011 17:53:06 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <755471046.19378.1301421186369.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6949) Reduces RPC packet size for
 primitive arrays, especially long[], which is used at block reporting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012580#comment-13012580 ] 

Matt Foley commented on HADOOP-6949:
------------------------------------

Hi Konstantin, the vote seems to be pretty strongly in favor, so I went ahead and ran some tests of the patch against v0.22.  Confirmed that the patch applies automatically, and hadoop/common compiles and successfully runs core unit tests.

> Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6949
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6949
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Navis
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: ObjectWritable.diff, arrayprim.patch, arrayprim_v4.patch, arrayprim_v5.patch, arrayprim_v6.patch, arrayprim_v7.patch
>
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n
> It would not be needed to specify each element types for primitive arrays.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13798-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 29 17:59:54 2011
Return-Path: <common-issues-return-13798-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73112 invoked from network); 29 Mar 2011 17:59:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Mar 2011 17:59:54 -0000
Received: (qmail 64434 invoked by uid 500); 29 Mar 2011 17:59:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64410 invoked by uid 500); 29 Mar 2011 17:59:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64402 invoked by uid 99); 29 Mar 2011 17:59:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 17:59:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 17:59:44 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CA8748667F
	for <common-issues@hadoop.apache.org>; Tue, 29 Mar 2011 17:59:06 +0000 (UTC)
Date: Tue, 29 Mar 2011 17:59:06 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1433074824.19408.1301421546826.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <8299358.16965.1301330105771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7210) Chown command is not working from
 FSShell.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012584#comment-13012584 ] 

Hadoop QA commented on HADOOP-7210:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12474892/HADOOP-7210.1.patch
  against trunk revision 1086455.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 5 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/323//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/323//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/323//console

This message is automatically generated.

> Chown command is not working from FSShell.
> ------------------------------------------
>
>                 Key: HADOOP-7210
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7210
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7210.1.patch, HADOOP-7210.patch
>
>
> chown command is not invoking the setOwner on FileSystem.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13799-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 29 18:08:43 2011
Return-Path: <common-issues-return-13799-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26954 invoked from network); 29 Mar 2011 18:08:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Mar 2011 18:08:43 -0000
Received: (qmail 81046 invoked by uid 500); 29 Mar 2011 18:08:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81018 invoked by uid 500); 29 Mar 2011 18:08:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81010 invoked by uid 99); 29 Mar 2011 18:08:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 18:08:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 18:08:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CA85C8695D
	for <common-issues@hadoop.apache.org>; Tue, 29 Mar 2011 18:08:05 +0000 (UTC)
Date: Tue, 29 Mar 2011 18:08:05 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <592823699.19423.1301422085826.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <8299358.16965.1301330105771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7210) Chown command is not working from
 FSShell.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012589#comment-13012589 ] 

Todd Lipcon commented on HADOOP-7210:
-------------------------------------

Couple small comments:
- Rather than injecting the "Extn" class into the FS Cache, could you instead set up a new scheme by setting "fs.testfs.impl" to your test class and then calling a command on testfs:///foo? Seems like a cleaner test pattern.
- In ChgrpHandler you don't need to check for group == null anymore, right?

> Chown command is not working from FSShell.
> ------------------------------------------
>
>                 Key: HADOOP-7210
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7210
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7210.1.patch, HADOOP-7210.patch
>
>
> chown command is not invoking the setOwner on FileSystem.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13800-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Mar 29 18:20:46 2011
Return-Path: <common-issues-return-13800-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46406 invoked from network); 29 Mar 2011 18:20:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Mar 2011 18:20:45 -0000
Received: (qmail 14287 invoked by uid 500); 29 Mar 2011 18:20:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14257 invoked by uid 500); 29 Mar 2011 18:20:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14249 invoked by uid 99); 29 Mar 2011 18:20:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 18:20:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Mar 2011 18:20:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1797A86D35
	for <common-issues@hadoop.apache.org>; Tue, 29 Mar 2011 18:20:06 +0000 (UTC)
Date: Tue, 29 Mar 2011 18:20:06 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1261984333.19446.1301422806093.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1627211432.4057.1299578279558.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7174) null is displayed in the console,if
 the src path is invalid while doing copyToLocal operation from commandLine
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7174:
--------------------------------

    Fix Version/s:     (was: 0.21.1)

This wasn't committed to branch-21 -- removing 0.21 fixversion.

> null is displayed in the console,if the src path is invalid while doing copyToLocal operation from commandLine
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7174
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7174
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>            Priority: Minor
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: HADOOP-7174.1.patch, HADOOP-7174.patch
>
>
> When we perform copyToLocal operations from commandLine and if src Path is invalid 
> srcFS.globStatus(srcpath) will return null. So, when we find the length of resulted value, it will *throw NullPointerException*.
>  Since we are handling generic exception , it will display null as the message.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13801-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 30 00:04:45 2011
Return-Path: <common-issues-return-13801-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25190 invoked from network); 30 Mar 2011 00:04:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Mar 2011 00:04:45 -0000
Received: (qmail 85308 invoked by uid 500); 30 Mar 2011 00:04:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84994 invoked by uid 500); 30 Mar 2011 00:04:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84976 invoked by uid 99); 30 Mar 2011 00:04:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Mar 2011 00:04:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Mar 2011 00:04:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BFD2886748
	for <common-issues@hadoop.apache.org>; Wed, 30 Mar 2011 00:04:05 +0000 (UTC)
Date: Wed, 30 Mar 2011 00:04:05 +0000 (UTC)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1235336773.20249.1301443445782.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7212) Reuse connection
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Reuse connection
----------------

                 Key: HADOOP-7212
                 URL: https://issues.apache.org/jira/browse/HADOOP-7212
             Project: Hadoop Common
          Issue Type: Bug
          Components: ipc
            Reporter: Hairong Kuang
            Assignee: Hairong Kuang


One of my recent RPC change introduced a regression. It makes the first RPC to server, getProtocolSignature, and following RPCs not sharing the same connection. If all clients are short lived, this regression would double the number of connections in the cluster.

The cause of the regression is that getProtocolSingature uses VersionProtocol to create a Connection object, and the following RPCs uses its own protocol name like ClientProtocol. Since protocol name is part of Connection object hashcode, this forces the RPC client to create a new Connection object, therefore forcing to create a new TCP/IP connection.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13802-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 30 06:51:51 2011
Return-Path: <common-issues-return-13802-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77136 invoked from network); 30 Mar 2011 06:51:51 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Mar 2011 06:51:51 -0000
Received: (qmail 26769 invoked by uid 500); 30 Mar 2011 06:51:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26668 invoked by uid 500); 30 Mar 2011 06:51:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26646 invoked by uid 99); 30 Mar 2011 06:51:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Mar 2011 06:51:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Mar 2011 06:51:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C90408953C
	for <common-issues@hadoop.apache.org>; Wed, 30 Mar 2011 06:51:05 +0000 (UTC)
Date: Wed, 30 Mar 2011 06:51:05 +0000 (UTC)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2002000154.20818.1301467865819.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1235336773.20249.1301443445782.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7212) Reuse connection
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang resolved HADOOP-7212.
-----------------------------------

    Resolution: Invalid

Just found out that the regression was introduced to only our internal branch not to the Apache trunk. Good!

> Reuse connection
> ----------------
>
>                 Key: HADOOP-7212
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7212
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>
> One of my recent RPC change introduced a regression. It makes the first RPC to server, getProtocolSignature, and following RPCs not sharing the same connection. If all clients are short lived, this regression would double the number of connections in the cluster.
> The cause of the regression is that getProtocolSingature uses VersionProtocol to create a Connection object, and the following RPCs uses its own protocol name like ClientProtocol. Since protocol name is part of Connection object hashcode, this forces the RPC client to create a new Connection object, therefore forcing to create a new TCP/IP connection.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13803-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 30 07:54:47 2011
Return-Path: <common-issues-return-13803-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74642 invoked from network); 30 Mar 2011 07:54:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Mar 2011 07:54:47 -0000
Received: (qmail 371 invoked by uid 500); 30 Mar 2011 07:54:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 310 invoked by uid 500); 30 Mar 2011 07:54:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 294 invoked by uid 99); 30 Mar 2011 07:54:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Mar 2011 07:54:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Mar 2011 07:54:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DB23F89A8B
	for <common-issues@hadoop.apache.org>; Wed, 30 Mar 2011 07:54:05 +0000 (UTC)
Date: Wed, 30 Mar 2011 07:54:05 +0000 (UTC)
From: "Dieter Plaetinck (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1622531359.20885.1301471645894.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7213) document multiple keys per reducer
 oddity in hadoop streaming FAQ
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

document multiple keys per reducer oddity in hadoop streaming FAQ
-----------------------------------------------------------------

                 Key: HADOOP-7213
                 URL: https://issues.apache.org/jira/browse/HADOOP-7213
             Project: Hadoop Common
          Issue Type: Improvement
          Components: documentation
            Reporter: Dieter Plaetinck
            Priority: Minor


Hi,
for a newcomer to hadoop streaming, it comes as a surprise that the reducer receives arbitrary keys, unlike the "real" hadoop where a reducer works on a single key.
An explanation for this is @ http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201103.mbox/browser

I suggest to add this to the FAQ of hadoop streaming

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13804-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 30 12:20:45 2011
Return-Path: <common-issues-return-13804-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25001 invoked from network); 30 Mar 2011 12:20:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Mar 2011 12:20:44 -0000
Received: (qmail 58972 invoked by uid 500); 30 Mar 2011 12:20:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58942 invoked by uid 500); 30 Mar 2011 12:20:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58933 invoked by uid 99); 30 Mar 2011 12:20:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Mar 2011 12:20:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Mar 2011 12:20:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 91D648A9C7
	for <common-issues@hadoop.apache.org>; Wed, 30 Mar 2011 12:20:06 +0000 (UTC)
Date: Wed, 30 Mar 2011 12:20:06 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2119951237.21165.1301487606593.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <8299358.16965.1301330105771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7210) Chown command is not working from
 FSShell.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uma Maheswara Rao G updated HADOOP-7210:
----------------------------------------

    Attachment: HADOOP-7210.2.patch

> Chown command is not working from FSShell.
> ------------------------------------------
>
>                 Key: HADOOP-7210
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7210
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7210.1.patch, HADOOP-7210.2.patch, HADOOP-7210.patch
>
>
> chown command is not invoking the setOwner on FileSystem.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13805-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 30 12:24:44 2011
Return-Path: <common-issues-return-13805-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38340 invoked from network); 30 Mar 2011 12:24:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Mar 2011 12:24:43 -0000
Received: (qmail 65831 invoked by uid 500); 30 Mar 2011 12:24:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65788 invoked by uid 500); 30 Mar 2011 12:24:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65780 invoked by uid 99); 30 Mar 2011 12:24:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Mar 2011 12:24:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Mar 2011 12:24:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E94128AB37
	for <common-issues@hadoop.apache.org>; Wed, 30 Mar 2011 12:24:05 +0000 (UTC)
Date: Wed, 30 Mar 2011 12:24:05 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <41193060.21171.1301487845951.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <8299358.16965.1301330105771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7210) Chown command is not working from
 FSShell.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012927#comment-13012927 ] 

Uma Maheswara Rao G commented on HADOOP-7210:
---------------------------------------------

Thanks Todd for the comments,
I have fixed the comments, please check.

> Chown command is not working from FSShell.
> ------------------------------------------
>
>                 Key: HADOOP-7210
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7210
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7210.1.patch, HADOOP-7210.2.patch, HADOOP-7210.patch
>
>
> chown command is not invoking the setOwner on FileSystem.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13806-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Mar 30 16:20:46 2011
Return-Path: <common-issues-return-13806-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34000 invoked from network); 30 Mar 2011 16:20:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Mar 2011 16:20:45 -0000
Received: (qmail 20496 invoked by uid 500); 30 Mar 2011 16:20:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20466 invoked by uid 500); 30 Mar 2011 16:20:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20458 invoked by uid 99); 30 Mar 2011 16:20:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Mar 2011 16:20:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Mar 2011 16:20:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BBD0E8BDB2
	for <common-issues@hadoop.apache.org>; Wed, 30 Mar 2011 16:20:05 +0000 (UTC)
Date: Wed, 30 Mar 2011 16:20:05 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1243264568.21599.1301502005765.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7214) Hadoop /usr/bin/groups equivalent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Hadoop /usr/bin/groups equivalent
---------------------------------

                 Key: HADOOP-7214
                 URL: https://issues.apache.org/jira/browse/HADOOP-7214
             Project: Hadoop Common
          Issue Type: New Feature
    Affects Versions: 0.23.0
            Reporter: Aaron T. Myers
            Assignee: Aaron T. Myers


Since user -> groups resolution is done on the NN and JT machines, there should be a way for users to determine what groups they're a member of from the NN's and JT's perspective.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13808-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 00:02:46 2011
Return-Path: <common-issues-return-13808-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13230 invoked from network); 31 Mar 2011 00:02:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 00:02:25 -0000
Received: (qmail 77563 invoked by uid 500); 30 Mar 2011 22:15:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77533 invoked by uid 500); 30 Mar 2011 22:15:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77525 invoked by uid 99); 30 Mar 2011 22:15:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Mar 2011 22:15:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Mar 2011 22:15:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B389F8A61F
	for <common-issues@hadoop.apache.org>; Wed, 30 Mar 2011 22:15:05 +0000 (UTC)
Date: Wed, 30 Mar 2011 22:15:05 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <802617055.22414.1301523305731.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <599378200.2869.1299537719453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7167) Allow using a file to exclude
 certain tests from build
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013704#comment-13013704 ] 

Hudson commented on HADOOP-7167:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #539 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/539/])
    Remove redundant entry for HADOOP-7167 in CHANGES.txt


> Allow using a file to exclude certain tests from build
> ------------------------------------------------------
>
>                 Key: HADOOP-7167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7167
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7167-cygwin.txt, hadoop-7167.txt
>
>
> It would be nice to be able to exclude certain tests when running builds. For example, when a test is "known flaky", you may want to exclude it from the main Hudson job, but not actually disable it in the codebase (so that it still runs as part of another Hudson job, for example).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13809-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 00:09:20 2011
Return-Path: <common-issues-return-13809-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19320 invoked from network); 31 Mar 2011 00:03:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 00:03:45 -0000
Received: (qmail 29225 invoked by uid 500); 30 Mar 2011 23:03:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29187 invoked by uid 500); 30 Mar 2011 23:03:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29179 invoked by uid 99); 30 Mar 2011 23:03:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Mar 2011 23:03:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Mar 2011 23:03:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AFEEC8B393
	for <common-issues@hadoop.apache.org>; Wed, 30 Mar 2011 23:03:05 +0000 (UTC)
Date: Wed, 30 Mar 2011 23:03:05 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <843174829.22535.1301526185717.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <8299358.16965.1301330105771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7210) Chown command is not working from
 FSShell.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013721#comment-13013721 ] 

Todd Lipcon commented on HADOOP-7210:
-------------------------------------

+1, looks good to me and verified that it fixes the tests. waiting on the hudson results before commit.

> Chown command is not working from FSShell.
> ------------------------------------------
>
>                 Key: HADOOP-7210
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7210
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7210.1.patch, HADOOP-7210.2.patch, HADOOP-7210.patch
>
>
> chown command is not invoking the setOwner on FileSystem.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13810-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 01:11:45 2011
Return-Path: <common-issues-return-13810-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95766 invoked from network); 31 Mar 2011 01:11:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 01:11:45 -0000
Received: (qmail 22970 invoked by uid 500); 31 Mar 2011 00:11:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22926 invoked by uid 500); 31 Mar 2011 00:11:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22918 invoked by uid 99); 31 Mar 2011 00:11:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 00:11:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 00:11:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BDC798B56A
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 00:11:05 +0000 (UTC)
Date: Thu, 31 Mar 2011 00:11:05 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
-----------------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-7215
                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
             Project: Hadoop Common
          Issue Type: Bug
          Components: security
            Reporter: Suresh Srinivas
            Assignee: Suresh Srinivas
             Fix For: 0.20.203.0, 0.20.204.0, 0.23.0


HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13811-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 01:13:43 2011
Return-Path: <common-issues-return-13811-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96616 invoked from network); 31 Mar 2011 01:13:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 01:13:43 -0000
Received: (qmail 24972 invoked by uid 500); 31 Mar 2011 00:13:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24946 invoked by uid 500); 31 Mar 2011 00:13:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24938 invoked by uid 99); 31 Mar 2011 00:13:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 00:13:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 00:13:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B2B628B5C5
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 00:13:05 +0000 (UTC)
Date: Thu, 31 Mar 2011 00:13:05 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <314338492.22805.1301530385728.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013756#comment-13013756 ] 

Suresh Srinivas commented on HADOOP-7215:
-----------------------------------------

Given that the server validate's the host name of the client's connection with the host name in the client's prinicipal name of format <service>/_HOST@realm, I propose binding the client's socket to the address specified in the principal name.



> RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.20.203.0, 0.20.204.0, 0.23.0
>
>
> HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13812-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 01:15:47 2011
Return-Path: <common-issues-return-13812-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 173 invoked from network); 31 Mar 2011 01:15:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 01:15:46 -0000
Received: (qmail 25667 invoked by uid 500); 31 Mar 2011 00:15:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25631 invoked by uid 500); 31 Mar 2011 00:15:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25623 invoked by uid 99); 31 Mar 2011 00:15:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 00:15:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 00:15:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D44348B643
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 00:15:05 +0000 (UTC)
Date: Thu, 31 Mar 2011 00:15:05 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <66842632.22811.1301530505866.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-7215:
------------------------------------

    Attachment: HADOOP-7215.203.patch

Path for 203 release.

> RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.20.203.0, 0.20.204.0, 0.23.0
>
>         Attachments: HADOOP-7215.203.patch
>
>
> HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13813-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 01:17:56 2011
Return-Path: <common-issues-return-13813-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6454 invoked from network); 31 Mar 2011 01:17:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 01:17:43 -0000
Received: (qmail 26786 invoked by uid 500); 31 Mar 2011 00:17:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26759 invoked by uid 500); 31 Mar 2011 00:17:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26749 invoked by uid 99); 31 Mar 2011 00:17:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 00:17:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 00:17:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B36438B6A6
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 00:17:05 +0000 (UTC)
Date: Thu, 31 Mar 2011 00:17:05 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <553175688.22813.1301530625731.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-7215:
------------------------------------

    Fix Version/s:     (was: 0.20.204.0)

> RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: HADOOP-7215.203.patch
>
>
> HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13807-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 01:17:58 2011
Return-Path: <common-issues-return-13807-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6573 invoked from network); 31 Mar 2011 01:17:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 01:17:45 -0000
Received: (qmail 1412 invoked by uid 500); 30 Mar 2011 21:17:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1387 invoked by uid 500); 30 Mar 2011 21:17:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1379 invoked by uid 99); 30 Mar 2011 21:17:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Mar 2011 21:17:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 Mar 2011 21:17:44 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5E55C8ADAA
	for <common-issues@hadoop.apache.org>; Wed, 30 Mar 2011 21:17:07 +0000 (UTC)
Date: Wed, 30 Mar 2011 21:17:07 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1641971222.22193.1301519827383.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1627211432.4057.1299578279558.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7174) null is displayed in the console,if
 the src path is invalid while doing copyToLocal operation from commandLine
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7174:
-------------------------------------------

    Fix Version/s: 0.21.1

> This wasn't committed to branch-21 - removing 0.21 fixversion.

I have double checked.  This was indeed committed to 0.21.  See also [this comment|https://issues.apache.org/jira/browse/HDFS-1786?focusedCommentId=13012726&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13012726].

> null is displayed in the console,if the src path is invalid while doing copyToLocal operation from commandLine
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7174
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7174
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>            Priority: Minor
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HADOOP-7174.1.patch, HADOOP-7174.patch
>
>
> When we perform copyToLocal operations from commandLine and if src Path is invalid 
> srcFS.globStatus(srcpath) will return null. So, when we find the length of resulted value, it will *throw NullPointerException*.
>  Since we are handling generic exception , it will display null as the message.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13816-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 01:17:59 2011
Return-Path: <common-issues-return-13816-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6655 invoked from network); 31 Mar 2011 01:17:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 01:17:46 -0000
Received: (qmail 71770 invoked by uid 500); 31 Mar 2011 01:17:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71742 invoked by uid 500); 31 Mar 2011 01:17:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71734 invoked by uid 99); 31 Mar 2011 01:17:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 01:17:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 01:17:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C59DA8B3FD
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 01:17:05 +0000 (UTC)
Date: Thu, 31 Mar 2011 01:17:05 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1069023074.22908.1301534225806.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-7215:
------------------------------------

    Attachment:     (was: HADOOP-7215.203.patch)

> RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: HADOOP-7215.trunk.patch
>
>
> HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13817-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 01:21:44 2011
Return-Path: <common-issues-return-13817-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25497 invoked from network); 31 Mar 2011 01:21:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 01:21:44 -0000
Received: (qmail 74991 invoked by uid 500); 31 Mar 2011 01:21:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74963 invoked by uid 500); 31 Mar 2011 01:21:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74952 invoked by uid 99); 31 Mar 2011 01:21:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 01:21:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 01:21:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4510A8B851
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 01:21:06 +0000 (UTC)
Date: Thu, 31 Mar 2011 01:21:06 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <472704440.22917.1301534466280.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013777#comment-13013777 ] 

Suresh Srinivas commented on HADOOP-7215:
-----------------------------------------

I mean HDFS-7104 in my previous comment.

> RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: HADOOP-7215.trunk.patch
>
>
> HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13818-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 01:21:49 2011
Return-Path: <common-issues-return-13818-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25721 invoked from network); 31 Mar 2011 01:21:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 01:21:46 -0000
Received: (qmail 75444 invoked by uid 500); 31 Mar 2011 01:21:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75394 invoked by uid 500); 31 Mar 2011 01:21:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75386 invoked by uid 99); 31 Mar 2011 01:21:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 01:21:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 01:21:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2983F8B83B
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 01:21:06 +0000 (UTC)
Date: Thu, 31 Mar 2011 01:21:06 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1553570407.22914.1301534466166.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013776#comment-13013776 ] 

Suresh Srinivas commented on HADOOP-7215:
-----------------------------------------

HDFS-1704 for some of the protocols such where clientPrincipal is needed, uses the second part as host name for validation. For such protocols, you cannot use a principal name that you mentioned. This does not affect protocols such as ClientProtocol though.

> RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: HADOOP-7215.trunk.patch
>
>
> HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13815-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 01:24:41 2011
Return-Path: <common-issues-return-13815-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38687 invoked from network); 31 Mar 2011 01:24:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 01:24:25 -0000
Received: (qmail 55517 invoked by uid 500); 31 Mar 2011 00:57:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55425 invoked by uid 500); 31 Mar 2011 00:57:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55417 invoked by uid 99); 31 Mar 2011 00:57:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 00:57:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 00:57:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BB4C18BE75
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 00:57:05 +0000 (UTC)
Date: Thu, 31 Mar 2011 00:57:05 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1439316653.22856.1301533025763.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-7215:
------------------------------------

    Attachment: HADOOP-7215.trunk.patch

Patch for the trunk, with minor modifications and tests.

> RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: HADOOP-7215.203.patch, HADOOP-7215.trunk.patch
>
>
> HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13819-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 01:25:46 2011
Return-Path: <common-issues-return-13819-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45235 invoked from network); 31 Mar 2011 01:25:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 01:25:43 -0000
Received: (qmail 79809 invoked by uid 500); 31 Mar 2011 01:25:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79783 invoked by uid 500); 31 Mar 2011 01:25:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79771 invoked by uid 99); 31 Mar 2011 01:25:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 01:25:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 01:25:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CB85B8BD3F
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 01:25:05 +0000 (UTC)
Date: Thu, 31 Mar 2011 01:25:05 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1270320238.22954.1301534705830.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013778#comment-13013778 ] 

Suresh Srinivas commented on HADOOP-7215:
-----------------------------------------

I see what you mean now. We could check if the second part is _HOST in the key and try binding to local address. Let me see if that could be done.

> RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: HADOOP-7215.trunk.patch
>
>
> HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13814-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 01:27:47 2011
Return-Path: <common-issues-return-13814-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49897 invoked from network); 31 Mar 2011 01:27:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 01:27:45 -0000
Received: (qmail 30227 invoked by uid 500); 31 Mar 2011 00:27:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30198 invoked by uid 500); 31 Mar 2011 00:27:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30172 invoked by uid 99); 31 Mar 2011 00:27:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 00:27:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 00:27:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B02058B8F6
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 00:27:05 +0000 (UTC)
Date: Thu, 31 Mar 2011 00:27:05 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1580705103.22823.1301531225718.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013761#comment-13013761 ] 

Todd Lipcon commented on HADOOP-7215:
-------------------------------------

I don't think this is quite right. It works when the IPC client is a server with a principal like hdfs/dn1.foo.com@REALM. But two-part principals are also used for other cases, eg tlipcon/admin@REALM. In this patch, won't that try to bind the local socket of my IPC client to the non-host "admin"?

> RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: HADOOP-7215.203.patch
>
>
> HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13820-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 02:03:46 2011
Return-Path: <common-issues-return-13820-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80797 invoked from network); 31 Mar 2011 02:03:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 02:03:45 -0000
Received: (qmail 11777 invoked by uid 500); 31 Mar 2011 02:03:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11740 invoked by uid 500); 31 Mar 2011 02:03:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11731 invoked by uid 99); 31 Mar 2011 02:03:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 02:03:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 02:03:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B63D98B7C5
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 02:03:05 +0000 (UTC)
Date: Thu, 31 Mar 2011 02:03:05 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <778592929.23013.1301536985743.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <8299358.16965.1301330105771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7210) Chown command is not working from
 FSShell.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013789#comment-13013789 ] 

Hadoop QA commented on HADOOP-7210:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12474960/HADOOP-7210.2.patch
  against trunk revision 1087097.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/324//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/324//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/324//console

This message is automatically generated.

> Chown command is not working from FSShell.
> ------------------------------------------
>
>                 Key: HADOOP-7210
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7210
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7210.1.patch, HADOOP-7210.2.patch, HADOOP-7210.patch
>
>
> chown command is not invoking the setOwner on FileSystem.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13821-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 02:28:43 2011
Return-Path: <common-issues-return-13821-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59314 invoked from network); 31 Mar 2011 02:28:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 02:28:43 -0000
Received: (qmail 26233 invoked by uid 500); 31 Mar 2011 02:28:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26199 invoked by uid 500); 31 Mar 2011 02:28:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26190 invoked by uid 99); 31 Mar 2011 02:28:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 02:28:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 02:28:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BCFDE8BD27
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 02:28:05 +0000 (UTC)
Date: Thu, 31 Mar 2011 02:28:05 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1720159589.23095.1301538485771.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <8299358.16965.1301330105771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7210) Chown command is not working from
 FSShell.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7210:
--------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

Committed to trunk, thanks Uma

> Chown command is not working from FSShell.
> ------------------------------------------
>
>                 Key: HADOOP-7210
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7210
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7210.1.patch, HADOOP-7210.2.patch, HADOOP-7210.patch
>
>
> chown command is not invoking the setOwner on FileSystem.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13822-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 02:44:45 2011
Return-Path: <common-issues-return-13822-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6950 invoked from network); 31 Mar 2011 02:44:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 02:44:45 -0000
Received: (qmail 34782 invoked by uid 500); 31 Mar 2011 02:44:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34758 invoked by uid 500); 31 Mar 2011 02:44:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34739 invoked by uid 99); 31 Mar 2011 02:44:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 02:44:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 02:44:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4686C8B0FB
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 02:44:06 +0000 (UTC)
Date: Thu, 31 Mar 2011 02:44:06 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <767557924.23133.1301539446285.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1687965.159931294234907446.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7086) Retrying socket connection failure
 times can be made as configurable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013799#comment-13013799 ] 

Todd Lipcon commented on HADOOP-7086:
-------------------------------------

Hi Devaraj. I think this is a good idea to make configurable.

- Can you please try to use Mockito rather than make a new TestSocketFactory in the test case? You can do something like:

{code}
SocketFactory mockFactory = Mockito.mock(new SocketFactory());
doThrow(new SocketTimeoutException()).when(mockFactory).createSocket();
Client client = new Client(..., mockFactory);
....
Mockito.verify(mockFactory, Mockito.times(noOfSocketRetries)).createSocket();
{code}

This is a bit more concise I think.

- Also, the patch attached here makes a number of spurious whitespace changes which should be ommitted.
- I don't think the name "ipc.client.connect.max.socket.retries" is very good - it's not clear how it's different from "ipc.client.connect.max.retries" which already exists. Maybe a better name would be "ipc.client.connect.max.retries.on.timeouts"?
- There's a typo in the description of the parameter: "make to establish a server connection". Perhaps "Indicates the number of times a client will retry when encountering a socket timeout while connecting to an IPC server."

> Retrying socket connection failure times can be made as configurable
> --------------------------------------------------------------------
>
>                 Key: HADOOP-7086
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7086
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>    Affects Versions: 0.21.0
>         Environment: NA
>            Reporter: Devaraj K
>            Assignee: Devaraj K
>            Priority: Minor
>         Attachments: common-3899.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> Retrying socket connection failure times are hard coded as 45 and it is giving the retryring message for 45 times as below. 
> 2011-01-04 15:14:30,700 INFO ipc.Client (Client.java:handleConnectionFailure(487)) - Retrying connect to server: /10.18.52.124:50020. Already tried 1 time(s).
> This can be made as configurable and also we can keep the default value as 45. If the user wants to decrease/increase,  they can add this configurable property otherwise it can continue with the default value.
> common\src\java\org\apache\hadoop\ipc\Client.java:
> -----------------------------------------------------------------------
> private synchronized void setupConnection() throws IOException {
>       short ioFailures = 0;
>       short timeoutFailures = 0;
>       while (true) {
>         try {
>           this.socket = socketFactory.createSocket();
>           this.socket.setTcpNoDelay(tcpNoDelay);
>           // connection time out is 20s
>           NetUtils.connect(this.socket, remoteId.getAddress(), 20000);
>           if (rpcTimeout > 0) {
>             pingInterval = rpcTimeout;  // rpcTimeout overwrites pingInterval
>           }
>           this.socket.setSoTimeout(pingInterval);
>           return;
>         } catch (SocketTimeoutException toe) {
>           /*
>            * The max number of retries is 45, which amounts to 20s*45 = 15
>            * minutes retries.
>            */
>           handleConnectionFailure(timeoutFailures++, 45, toe);
>         } catch (IOException ie) {
>           handleConnectionFailure(ioFailures++, maxRetries, ie);
>         }
>       }
>     }

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13823-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 02:44:46 2011
Return-Path: <common-issues-return-13823-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6995 invoked from network); 31 Mar 2011 02:44:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 02:44:46 -0000
Received: (qmail 35012 invoked by uid 500); 31 Mar 2011 02:44:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34976 invoked by uid 500); 31 Mar 2011 02:44:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34968 invoked by uid 99); 31 Mar 2011 02:44:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 02:44:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 02:44:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F3FCA8B0F1
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 02:44:05 +0000 (UTC)
Date: Thu, 31 Mar 2011 02:44:05 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <617171841.23123.1301539445996.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <8299358.16965.1301330105771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7210) Chown command is not working from
 FSShell.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013797#comment-13013797 ] 

Hudson commented on HADOOP-7210:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #540 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/540/])
    HADOOP-7210. Chown command is not working from FSShell. Contributed by Uma Maheswara Rao G


> Chown command is not working from FSShell.
> ------------------------------------------
>
>                 Key: HADOOP-7210
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7210
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7210.1.patch, HADOOP-7210.2.patch, HADOOP-7210.patch
>
>
> chown command is not invoking the setOwner on FileSystem.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13824-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 06:40:48 2011
Return-Path: <common-issues-return-13824-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4727 invoked from network); 31 Mar 2011 06:40:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 06:40:47 -0000
Received: (qmail 91326 invoked by uid 500); 31 Mar 2011 06:40:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91300 invoked by uid 500); 31 Mar 2011 06:40:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91289 invoked by uid 99); 31 Mar 2011 06:40:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 06:40:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 06:40:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B47FB43C6F
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 06:40:05 +0000 (UTC)
Date: Thu, 31 Mar 2011 06:40:05 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <635269266.23604.1301553605735.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013859#comment-13013859 ] 

Suresh Srinivas commented on HADOOP-7215:
-----------------------------------------

Here is a way to do that at the RPC client:
# get part2 from principal name of format <part1>/<part2>@realm and ensure part2 is a host name.
# the address corresponding to this host name belongs to one of the local network interfaces on that host.

If the above two conditions are satisfied, the bind to the socket to that address, before making rpc calls. It should address the following cases:
# Principal name is not of format <part1>/<part2>@realm
# part2 is not a valid host name
# part2 is a valid host name, but not that of the client host

Does this sound reasonable?

> RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: HADOOP-7215.trunk.patch
>
>
> HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13825-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 07:50:49 2011
Return-Path: <common-issues-return-13825-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42327 invoked from network); 31 Mar 2011 07:50:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 07:50:49 -0000
Received: (qmail 52684 invoked by uid 500); 31 Mar 2011 07:50:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52603 invoked by uid 500); 31 Mar 2011 07:50:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52589 invoked by uid 99); 31 Mar 2011 07:50:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 07:50:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 07:50:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DF8DC8B2E6
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 07:50:05 +0000 (UTC)
Date: Thu, 31 Mar 2011 07:50:05 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1490758718.23722.1301557805911.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013872#comment-13013872 ] 

Todd Lipcon commented on HADOOP-7215:
-------------------------------------

Hi Suresh. This seems like a reasonable solution. Maybe we can also check the kerberos info on the protocol interface, and if there's no clientPrincipal set, we can skip all of this? My thinking is that we want to avoid the reverse DNS lookups for the common case of user->cluster communications where no clientPrincipal annotation is present.

> RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: HADOOP-7215.trunk.patch
>
>
> HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13826-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 11:13:45 2011
Return-Path: <common-issues-return-13826-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8859 invoked from network); 31 Mar 2011 11:13:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 11:13:44 -0000
Received: (qmail 2166 invoked by uid 500); 31 Mar 2011 11:13:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2102 invoked by uid 500); 31 Mar 2011 11:13:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2094 invoked by uid 99); 31 Mar 2011 11:13:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 11:13:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 11:13:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EFA5C8BB85
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 11:13:05 +0000 (UTC)
Date: Thu, 31 Mar 2011 11:13:05 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2057181928.24110.1301569985978.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <599378200.2869.1299537719453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7167) Allow using a file to exclude
 certain tests from build
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013924#comment-13013924 ] 

Hudson commented on HADOOP-7167:
--------------------------------

Integrated in Hadoop-Common-trunk #647 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/647/])
    Remove redundant entry for HADOOP-7167 in CHANGES.txt


> Allow using a file to exclude certain tests from build
> ------------------------------------------------------
>
>                 Key: HADOOP-7167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7167
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7167-cygwin.txt, hadoop-7167.txt
>
>
> It would be nice to be able to exclude certain tests when running builds. For example, when a test is "known flaky", you may want to exclude it from the main Hudson job, but not actually disable it in the codebase (so that it still runs as part of another Hudson job, for example).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13827-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 11:13:46 2011
Return-Path: <common-issues-return-13827-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8894 invoked from network); 31 Mar 2011 11:13:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 11:13:46 -0000
Received: (qmail 2362 invoked by uid 500); 31 Mar 2011 11:13:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2327 invoked by uid 500); 31 Mar 2011 11:13:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2318 invoked by uid 99); 31 Mar 2011 11:13:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 11:13:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 11:13:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 095AB8BB87
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 11:13:06 +0000 (UTC)
Date: Thu, 31 Mar 2011 11:13:06 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1325457552.24112.1301569986035.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <8299358.16965.1301330105771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7210) Chown command is not working from
 FSShell.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013925#comment-13013925 ] 

Hudson commented on HADOOP-7210:
--------------------------------

Integrated in Hadoop-Common-trunk #647 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/647/])
    HADOOP-7210. Chown command is not working from FSShell. Contributed by Uma Maheswara Rao G


> Chown command is not working from FSShell.
> ------------------------------------------
>
>                 Key: HADOOP-7210
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7210
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>         Environment: Linux
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7210.1.patch, HADOOP-7210.2.patch, HADOOP-7210.patch
>
>
> chown command is not invoking the setOwner on FileSystem.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13828-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 15:02:45 2011
Return-Path: <common-issues-return-13828-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3381 invoked from network); 31 Mar 2011 15:02:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 15:02:44 -0000
Received: (qmail 88470 invoked by uid 500); 31 Mar 2011 15:02:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88441 invoked by uid 500); 31 Mar 2011 15:02:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87978 invoked by uid 99); 31 Mar 2011 15:02:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 15:02:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 15:02:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0C8128C982
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 15:02:06 +0000 (UTC)
Date: Thu, 31 Mar 2011 15:02:06 +0000 (UTC)
From: "Devaraj K (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1208431188.24452.1301583726048.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1687965.159931294234907446.JavaMail.jira@thor>
Subject: [jira] [Updated] (HADOOP-7086) Retrying socket connection failure
 times can be made as configurable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj K updated HADOOP-7086:
------------------------------

    Attachment: HADOOP-7086.patch

> Retrying socket connection failure times can be made as configurable
> --------------------------------------------------------------------
>
>                 Key: HADOOP-7086
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7086
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>    Affects Versions: 0.21.0
>         Environment: NA
>            Reporter: Devaraj K
>            Assignee: Devaraj K
>            Priority: Minor
>         Attachments: HADOOP-7086.patch, common-3899.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> Retrying socket connection failure times are hard coded as 45 and it is giving the retryring message for 45 times as below. 
> 2011-01-04 15:14:30,700 INFO ipc.Client (Client.java:handleConnectionFailure(487)) - Retrying connect to server: /10.18.52.124:50020. Already tried 1 time(s).
> This can be made as configurable and also we can keep the default value as 45. If the user wants to decrease/increase,  they can add this configurable property otherwise it can continue with the default value.
> common\src\java\org\apache\hadoop\ipc\Client.java:
> -----------------------------------------------------------------------
> private synchronized void setupConnection() throws IOException {
>       short ioFailures = 0;
>       short timeoutFailures = 0;
>       while (true) {
>         try {
>           this.socket = socketFactory.createSocket();
>           this.socket.setTcpNoDelay(tcpNoDelay);
>           // connection time out is 20s
>           NetUtils.connect(this.socket, remoteId.getAddress(), 20000);
>           if (rpcTimeout > 0) {
>             pingInterval = rpcTimeout;  // rpcTimeout overwrites pingInterval
>           }
>           this.socket.setSoTimeout(pingInterval);
>           return;
>         } catch (SocketTimeoutException toe) {
>           /*
>            * The max number of retries is 45, which amounts to 20s*45 = 15
>            * minutes retries.
>            */
>           handleConnectionFailure(timeoutFailures++, 45, toe);
>         } catch (IOException ie) {
>           handleConnectionFailure(ioFailures++, maxRetries, ie);
>         }
>       }
>     }

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13829-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 15:22:45 2011
Return-Path: <common-issues-return-13829-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80699 invoked from network); 31 Mar 2011 15:22:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 15:22:44 -0000
Received: (qmail 49260 invoked by uid 500); 31 Mar 2011 15:22:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49101 invoked by uid 500); 31 Mar 2011 15:22:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49031 invoked by uid 99); 31 Mar 2011 15:22:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 15:22:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 15:22:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D54E38B380
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 15:22:05 +0000 (UTC)
Date: Thu, 31 Mar 2011 15:22:05 +0000 (UTC)
From: "Devaraj K (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1732780269.24567.1301584925870.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1687965.159931294234907446.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7086) Retrying socket connection failure
 times can be made as configurable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014013#comment-13014013 ] 

Devaraj K commented on HADOOP-7086:
-----------------------------------

Thanks Todd for reviewing.

I updated the patch by addressing the above comments. 


> Retrying socket connection failure times can be made as configurable
> --------------------------------------------------------------------
>
>                 Key: HADOOP-7086
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7086
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>    Affects Versions: 0.21.0
>         Environment: NA
>            Reporter: Devaraj K
>            Assignee: Devaraj K
>            Priority: Minor
>         Attachments: HADOOP-7086.patch, common-3899.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> Retrying socket connection failure times are hard coded as 45 and it is giving the retryring message for 45 times as below. 
> 2011-01-04 15:14:30,700 INFO ipc.Client (Client.java:handleConnectionFailure(487)) - Retrying connect to server: /10.18.52.124:50020. Already tried 1 time(s).
> This can be made as configurable and also we can keep the default value as 45. If the user wants to decrease/increase,  they can add this configurable property otherwise it can continue with the default value.
> common\src\java\org\apache\hadoop\ipc\Client.java:
> -----------------------------------------------------------------------
> private synchronized void setupConnection() throws IOException {
>       short ioFailures = 0;
>       short timeoutFailures = 0;
>       while (true) {
>         try {
>           this.socket = socketFactory.createSocket();
>           this.socket.setTcpNoDelay(tcpNoDelay);
>           // connection time out is 20s
>           NetUtils.connect(this.socket, remoteId.getAddress(), 20000);
>           if (rpcTimeout > 0) {
>             pingInterval = rpcTimeout;  // rpcTimeout overwrites pingInterval
>           }
>           this.socket.setSoTimeout(pingInterval);
>           return;
>         } catch (SocketTimeoutException toe) {
>           /*
>            * The max number of retries is 45, which amounts to 20s*45 = 15
>            * minutes retries.
>            */
>           handleConnectionFailure(timeoutFailures++, 45, toe);
>         } catch (IOException ie) {
>           handleConnectionFailure(ioFailures++, maxRetries, ie);
>         }
>       }
>     }

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13830-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 18:26:45 2011
Return-Path: <common-issues-return-13830-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4471 invoked from network); 31 Mar 2011 18:26:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 18:26:44 -0000
Received: (qmail 66473 invoked by uid 500); 31 Mar 2011 18:26:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66385 invoked by uid 500); 31 Mar 2011 18:26:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66377 invoked by uid 99); 31 Mar 2011 18:26:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 18:26:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 18:26:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8818C8C1CE
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 18:26:06 +0000 (UTC)
Date: Thu, 31 Mar 2011 18:26:06 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <816377974.25244.1301595966553.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6941) Support non-SUN JREs in
 UserGroupInformation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6941:
--------------------------------

    Status: Open  (was: Patch Available)

> Support non-SUN JREs in UserGroupInformation
> --------------------------------------------
>
>                 Key: HADOOP-6941
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6941
>             Project: Hadoop Common
>          Issue Type: Bug
>         Environment: SLES 11, Apache Harmony 6 and SLES 11, IBM Java 6
>            Reporter: Stephen Watt
>            Assignee: Stephen Watt
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HADOOP-6941.patch, hadoop-6941.patch
>
>
> Attempting to format the namenode or attempting to start Hadoop using Apache Harmony or the IBM Java JREs results in the following exception:
> 10/09/07 16:35:05 ERROR namenode.NameNode: java.lang.NoClassDefFoundError: com.sun.security.auth.UnixPrincipal
> 	at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:223)
> 	at java.lang.J9VMInternals.initializeImpl(Native Method)
> 	at java.lang.J9VMInternals.initialize(J9VMInternals.java:200)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setConfigurationParameters(FSNamesystem.java:420)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:391)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1240)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1348)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368)
> Caused by: java.lang.ClassNotFoundException: com.sun.security.auth.UnixPrincipal
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:421)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:652)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:346)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:618)
> 	... 8 more
> This is a negative regression as previous versions of Hadoop worked with these JREs

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13831-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 18:47:48 2011
Return-Path: <common-issues-return-13831-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80463 invoked from network); 31 Mar 2011 18:47:48 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 18:47:48 -0000
Received: (qmail 14618 invoked by uid 500); 31 Mar 2011 18:47:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14587 invoked by uid 500); 31 Mar 2011 18:47:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14579 invoked by uid 99); 31 Mar 2011 18:47:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 18:47:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 18:47:45 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EA6DE8CC39
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 18:47:07 +0000 (UTC)
Date: Thu, 31 Mar 2011 18:47:07 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <925849752.25342.1301597227956.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014137#comment-13014137 ] 

Suresh Srinivas commented on HADOOP-7215:
-----------------------------------------

In that case, we have two choices:
# Fail at the client side:
#* If the client principal name does not have <part1>/<part2>@realm format, fail it at the client with appropriate error. 
#* If the format is right, treat part2 as host name. Just try to bind to it and if bind fails, then the failure occurs at the client it self with appropriate error. 

# Fail at the server side:
#* If the client principal name does not have <part1>/<part2>@realm format, bind to any local address for the request.
#* If the format is right, treat part2 as host name. If host name is a valid local address, bind to it else bind to any local address. This request will be rejected by the server.

I am leaning towards (2) because, server is rightly involved in the decision of rejecting the client. It provides a record of this at both the client and the server. This will help debugging on the server side, independent of client.

> RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: HADOOP-7215.trunk.patch
>
>
> HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13832-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 19:26:46 2011
Return-Path: <common-issues-return-13832-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1540 invoked from network); 31 Mar 2011 19:26:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 19:26:45 -0000
Received: (qmail 51289 invoked by uid 500); 31 Mar 2011 19:26:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51259 invoked by uid 500); 31 Mar 2011 19:26:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51251 invoked by uid 99); 31 Mar 2011 19:26:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 19:26:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 19:26:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B45768C7AA
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 19:26:05 +0000 (UTC)
Date: Thu, 31 Mar 2011 19:26:05 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1682830173.25411.1301599565735.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-7215:
------------------------------------

    Attachment:     (was: HADOOP-7215.trunk.patch)

> RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.20.203.0, 0.23.0
>
>
> HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13833-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 19:28:44 2011
Return-Path: <common-issues-return-13833-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1676 invoked from network); 31 Mar 2011 19:28:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 19:28:44 -0000
Received: (qmail 53727 invoked by uid 500); 31 Mar 2011 19:28:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53699 invoked by uid 500); 31 Mar 2011 19:28:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53691 invoked by uid 99); 31 Mar 2011 19:28:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 19:28:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 19:28:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 299128C88D
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 19:28:06 +0000 (UTC)
Date: Thu, 31 Mar 2011 19:28:06 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1279136458.25419.1301599686166.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-7215:
------------------------------------

    Attachment: HADOOP-7215.trunk.patch

Attached patch with changes from option(2) from my previous comment. I also cleaned up NetUtils.java warnings related to javadoc.

> RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: HADOOP-7215.trunk.patch
>
>
> HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13834-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 19:28:46 2011
Return-Path: <common-issues-return-13834-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1765 invoked from network); 31 Mar 2011 19:28:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 19:28:46 -0000
Received: (qmail 54152 invoked by uid 500); 31 Mar 2011 19:28:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54120 invoked by uid 500); 31 Mar 2011 19:28:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54112 invoked by uid 99); 31 Mar 2011 19:28:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 19:28:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 19:28:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4A1408C892
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 19:28:06 +0000 (UTC)
Date: Thu, 31 Mar 2011 19:28:06 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <3966881.25424.1301599686300.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-7215:
------------------------------------

    Status: Patch Available  (was: Open)

> RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: HADOOP-7215.trunk.patch
>
>
> HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13835-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 20:05:44 2011
Return-Path: <common-issues-return-13835-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54494 invoked from network); 31 Mar 2011 20:05:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 20:05:43 -0000
Received: (qmail 12227 invoked by uid 500); 31 Mar 2011 20:05:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12199 invoked by uid 500); 31 Mar 2011 20:05:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12191 invoked by uid 99); 31 Mar 2011 20:05:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 20:05:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 20:05:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B43858C85F
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 20:05:05 +0000 (UTC)
Date: Thu, 31 Mar 2011 20:05:05 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1918520765.25552.1301601905734.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-7215:
------------------------------------

    Attachment: HADOOP-7215.1.trunk.patch

updated patch

> RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: HADOOP-7215.1.trunk.patch, HADOOP-7215.trunk.patch
>
>
> HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13836-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 21:22:08 2011
Return-Path: <common-issues-return-13836-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85037 invoked from network); 31 Mar 2011 20:23:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 20:23:46 -0000
Received: (qmail 54506 invoked by uid 500); 31 Mar 2011 20:23:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54436 invoked by uid 500); 31 Mar 2011 20:23:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54323 invoked by uid 99); 31 Mar 2011 20:23:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 20:23:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 20:23:44 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 678338CE86
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 20:23:06 +0000 (UTC)
Date: Thu, 31 Mar 2011 20:23:06 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <540167034.25581.1301602986420.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014183#comment-13014183 ] 

Hadoop QA commented on HADOOP-7215:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12475153/HADOOP-7215.1.trunk.patch
  against trunk revision 1087159.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.net.TestNetUtils

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/325//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/325//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/325//console

This message is automatically generated.

> RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: HADOOP-7215.1.trunk.patch, HADOOP-7215.trunk.patch
>
>
> HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13837-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 21:27:10 2011
Return-Path: <common-issues-return-13837-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86480 invoked from network); 31 Mar 2011 20:42:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 20:42:25 -0000
Received: (qmail 70627 invoked by uid 500); 31 Mar 2011 20:35:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70602 invoked by uid 500); 31 Mar 2011 20:35:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70594 invoked by uid 99); 31 Mar 2011 20:35:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 20:35:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 20:35:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B29D18C1EC
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 20:35:05 +0000 (UTC)
Date: Thu, 31 Mar 2011 20:35:05 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2108069429.25599.1301603705728.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-7215:
------------------------------------

    Attachment:     (was: HADOOP-7215.trunk.patch)

> RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: HADOOP-7215.1.trunk.patch
>
>
> HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13838-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 21:27:33 2011
Return-Path: <common-issues-return-13838-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86654 invoked from network); 31 Mar 2011 20:45:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 20:45:46 -0000
Received: (qmail 79360 invoked by uid 500); 31 Mar 2011 20:45:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79294 invoked by uid 500); 31 Mar 2011 20:45:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79168 invoked by uid 99); 31 Mar 2011 20:45:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 20:45:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 20:45:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3732A8C602
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 20:45:06 +0000 (UTC)
Date: Thu, 31 Mar 2011 20:45:06 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2037130658.25635.1301604306222.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014193#comment-13014193 ] 

Hadoop QA commented on HADOOP-7215:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12475153/HADOOP-7215.1.trunk.patch
  against trunk revision 1087159.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.net.TestNetUtils

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/326//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/326//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/326//console

This message is automatically generated.

> RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: HADOOP-7215.1.trunk.patch
>
>
> HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13839-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 21:35:29 2011
Return-Path: <common-issues-return-13839-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8721 invoked from network); 31 Mar 2011 21:25:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 21:25:44 -0000
Received: (qmail 38360 invoked by uid 500); 31 Mar 2011 21:25:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38331 invoked by uid 500); 31 Mar 2011 21:25:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38323 invoked by uid 99); 31 Mar 2011 21:25:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 21:25:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 21:25:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F387A8C802
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 21:25:05 +0000 (UTC)
Date: Thu, 31 Mar 2011 21:25:05 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <886508381.25788.1301606705994.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-7215:
------------------------------------

    Attachment: HADOOP-7215.debug.patch

For some reason one of the test I am adding fails in hudson test, but works fine on my machine.

Uploading a debug version of the patch to understand the problem.

> RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: HADOOP-7215.1.trunk.patch, HADOOP-7215.debug.patch
>
>
> HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13840-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 21:43:46 2011
Return-Path: <common-issues-return-13840-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70033 invoked from network); 31 Mar 2011 21:43:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 21:43:45 -0000
Received: (qmail 57842 invoked by uid 500); 31 Mar 2011 21:43:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57806 invoked by uid 500); 31 Mar 2011 21:43:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57798 invoked by uid 99); 31 Mar 2011 21:43:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 21:43:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 21:43:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BD76F8CC20
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 21:43:05 +0000 (UTC)
Date: Thu, 31 Mar 2011 21:43:05 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1805516457.25797.1301607785772.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014224#comment-13014224 ] 

Hadoop QA commented on HADOOP-7215:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12475156/HADOOP-7215.debug.patch
  against trunk revision 1087159.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/327//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/327//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/327//console

This message is automatically generated.

> RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: HADOOP-7215.1.trunk.patch, HADOOP-7215.debug.patch
>
>
> HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13841-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 21:53:45 2011
Return-Path: <common-issues-return-13841-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18774 invoked from network); 31 Mar 2011 21:53:45 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 21:53:45 -0000
Received: (qmail 68054 invoked by uid 500); 31 Mar 2011 21:53:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68018 invoked by uid 500); 31 Mar 2011 21:53:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68010 invoked by uid 99); 31 Mar 2011 21:53:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 21:53:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 21:53:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B8FCC8CF78
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 21:53:05 +0000 (UTC)
Date: Thu, 31 Mar 2011 21:53:05 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <676933154.25816.1301608385754.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-7215:
------------------------------------

    Attachment: HADOOP-7215.2.trunk.patch

With debug log the test did not fail! Trying a new patch again.

> RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: HADOOP-7215.1.trunk.patch, HADOOP-7215.2.trunk.patch, HADOOP-7215.debug.patch
>
>
> HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13842-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 22:13:44 2011
Return-Path: <common-issues-return-13842-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76949 invoked from network); 31 Mar 2011 22:13:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 22:13:44 -0000
Received: (qmail 88958 invoked by uid 500); 31 Mar 2011 22:13:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88850 invoked by uid 500); 31 Mar 2011 22:13:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88774 invoked by uid 99); 31 Mar 2011 22:13:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 22:13:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 22:13:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D68918C509
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 22:13:05 +0000 (UTC)
Date: Thu, 31 Mar 2011 22:13:05 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1849602207.25859.1301609585875.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014237#comment-13014237 ] 

Hadoop QA commented on HADOOP-7215:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12475157/HADOOP-7215.2.trunk.patch
  against trunk revision 1087159.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.net.TestNetUtils

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/328//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/328//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/328//console

This message is automatically generated.

> RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: HADOOP-7215.1.trunk.patch, HADOOP-7215.2.trunk.patch, HADOOP-7215.debug.patch
>
>
> HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13843-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 22:34:44 2011
Return-Path: <common-issues-return-13843-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67463 invoked from network); 31 Mar 2011 22:34:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 22:34:44 -0000
Received: (qmail 13321 invoked by uid 500); 31 Mar 2011 22:34:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13294 invoked by uid 500); 31 Mar 2011 22:34:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13286 invoked by uid 99); 31 Mar 2011 22:34:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 22:34:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 22:34:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C25448CD43
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 22:34:05 +0000 (UTC)
Date: Thu, 31 Mar 2011 22:34:05 +0000 (UTC)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <933976139.25917.1301610845792.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6949) Reduces RPC packet size for
 primitive arrays, especially long[], which is used at block reporting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014249#comment-13014249 ] 

Konstantin Shvachko commented on HADOOP-6949:
---------------------------------------------

The vote was [conducted|http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201103.mbox/%3CAANLkTi=Z6jaYVbFTt3bAQRVsj7CuSKBcK1+ims079ss8@mail.gmail.com%3E] on the dev lists for common, hdfs, and mapreduce.
I counted nine +1s and no -1s. 
We can commit it to 0.22.

> Reduces RPC packet size for primitive arrays, especially long[], which is used at block reporting
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6949
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6949
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Navis
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: ObjectWritable.diff, arrayprim.patch, arrayprim_v4.patch, arrayprim_v5.patch, arrayprim_v6.patch, arrayprim_v7.patch
>
>   Original Estimate: 10m
>  Remaining Estimate: 10m
>
> Current implementation of oah.io.ObjectWritable marshals primitive array types as general object array ; array type string + array length + (element type string + value)*n
> It would not be needed to specify each element types for primitive arrays.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13844-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 22:42:46 2011
Return-Path: <common-issues-return-13844-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72713 invoked from network); 31 Mar 2011 22:42:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 22:42:46 -0000
Received: (qmail 20057 invoked by uid 500); 31 Mar 2011 22:42:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20026 invoked by uid 500); 31 Mar 2011 22:42:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20018 invoked by uid 99); 31 Mar 2011 22:42:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 22:42:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 22:42:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E1E4C8CF85
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 22:42:05 +0000 (UTC)
Date: Thu, 31 Mar 2011 22:42:05 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1931127282.25950.1301611325921.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-7215:
------------------------------------

    Attachment: HADOOP-7215.debug.patch

Patch without debug fails. Trying patch with debug again...

> RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: HADOOP-7215.1.trunk.patch, HADOOP-7215.2.trunk.patch, HADOOP-7215.debug.patch, HADOOP-7215.debug.patch
>
>
> HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13845-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 23:00:44 2011
Return-Path: <common-issues-return-13845-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23323 invoked from network); 31 Mar 2011 23:00:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 23:00:44 -0000
Received: (qmail 33954 invoked by uid 500); 31 Mar 2011 23:00:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33827 invoked by uid 500); 31 Mar 2011 23:00:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33819 invoked by uid 99); 31 Mar 2011 23:00:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 23:00:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 23:00:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DFCC88C571
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 23:00:05 +0000 (UTC)
Date: Thu, 31 Mar 2011 23:00:05 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <115374646.26008.1301612405913.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014265#comment-13014265 ] 

Hadoop QA commented on HADOOP-7215:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12475162/HADOOP-7215.debug.patch
  against trunk revision 1087159.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.net.TestNetUtils

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/329//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/329//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/329//console

This message is automatically generated.

> RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: HADOOP-7215.1.trunk.patch, HADOOP-7215.2.trunk.patch, HADOOP-7215.debug.patch, HADOOP-7215.debug.patch
>
>
> HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13846-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 23:10:44 2011
Return-Path: <common-issues-return-13846-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72677 invoked from network); 31 Mar 2011 23:10:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 23:10:44 -0000
Received: (qmail 39298 invoked by uid 500); 31 Mar 2011 23:10:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39264 invoked by uid 500); 31 Mar 2011 23:10:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39255 invoked by uid 99); 31 Mar 2011 23:10:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 23:10:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 23:10:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B66888C90B
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 23:10:05 +0000 (UTC)
Date: Thu, 31 Mar 2011 23:10:05 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <363371995.26050.1301613005743.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-7215:
------------------------------------

    Attachment: HADOOP-7215.3.trunk.patch

The reason tests are failing is: local host maps to 127.0.1.1 with no interface configured for it.

Looks like the other tests use 127.0.0.1 for local host. Doing the same for the newly added test.

> RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: HADOOP-7215.1.trunk.patch, HADOOP-7215.2.trunk.patch, HADOOP-7215.3.trunk.patch, HADOOP-7215.debug.patch, HADOOP-7215.debug.patch
>
>
> HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-13847-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Mar 31 23:28:43 2011
Return-Path: <common-issues-return-13847-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19588 invoked from network); 31 Mar 2011 23:28:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Mar 2011 23:28:43 -0000
Received: (qmail 66204 invoked by uid 500); 31 Mar 2011 23:28:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66178 invoked by uid 500); 31 Mar 2011 23:28:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66170 invoked by uid 99); 31 Mar 2011 23:28:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 23:28:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Mar 2011 23:28:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B84FB8CE86
	for <common-issues@hadoop.apache.org>; Thu, 31 Mar 2011 23:28:05 +0000 (UTC)
Date: Thu, 31 Mar 2011 23:28:05 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1284838577.26132.1301614085751.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014278#comment-13014278 ] 

Hadoop QA commented on HADOOP-7215:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12475167/HADOOP-7215.3.trunk.patch
  against trunk revision 1087159.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/330//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/330//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/330//console

This message is automatically generated.

> RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: HADOOP-7215.1.trunk.patch, HADOOP-7215.2.trunk.patch, HADOOP-7215.3.trunk.patch, HADOOP-7215.debug.patch, HADOOP-7215.debug.patch
>
>
> HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14354-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun May  1 04:31:53 2011
Return-Path: <common-issues-return-14354-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 13A002504
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun,  1 May 2011 04:31:53 +0000 (UTC)
Received: (qmail 96301 invoked by uid 500); 1 May 2011 04:31:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94838 invoked by uid 500); 1 May 2011 04:31:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94819 invoked by uid 99); 1 May 2011 04:31:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 May 2011 04:31:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 May 2011 04:31:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2A006BB9C1
	for <common-issues@hadoop.apache.org>; Sun,  1 May 2011 04:31:03 +0000 (UTC)
Date: Sun, 1 May 2011 04:31:03 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <836774115.14005.1304224263168.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7254) Programmatically start  processes
 with JMX port open
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Programmatically start  processes with JMX port open 
-----------------------------------------------------

                 Key: HADOOP-7254
                 URL: https://issues.apache.org/jira/browse/HADOOP-7254
             Project: Hadoop Common
          Issue Type: New Feature
          Components: scripts
            Reporter: Tanping Wang
            Priority: Minor
             Fix For: 0.23.0


We propose a programmatic way to start processes with JMX enabled.  This is the counter part of HDFS-1874.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14355-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun May  1 04:33:45 2011
Return-Path: <common-issues-return-14355-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A34972DFD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun,  1 May 2011 04:33:45 +0000 (UTC)
Received: (qmail 96517 invoked by uid 500); 1 May 2011 04:33:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96452 invoked by uid 500); 1 May 2011 04:33:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96444 invoked by uid 99); 1 May 2011 04:33:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 May 2011 04:33:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 May 2011 04:33:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 34805BBA41
	for <common-issues@hadoop.apache.org>; Sun,  1 May 2011 04:33:03 +0000 (UTC)
Date: Sun, 1 May 2011 04:33:03 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1659309625.14009.1304224383211.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <836774115.14005.1304224263168.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7254) Programmatically start  processes
 with JMX port open
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027423#comment-13027423 ] 

Tanping Wang commented on HADOOP-7254:
--------------------------------------

Together with HDFS-1874, we enable hadoop startup scripts to grab JMX port from hdfs-site.xml as one of the properties and start processes with Jmx enabled.

> Programmatically start  processes with JMX port open 
> -----------------------------------------------------
>
>                 Key: HADOOP-7254
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7254
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: scripts
>            Reporter: Tanping Wang
>            Priority: Minor
>             Fix For: 0.23.0
>
>
> We propose a programmatic way to start processes with JMX enabled.  This is the counter part of HDFS-1874.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14356-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun May  1 04:37:46 2011
Return-Path: <common-issues-return-14356-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E6F902E31
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun,  1 May 2011 04:37:46 +0000 (UTC)
Received: (qmail 96932 invoked by uid 500); 1 May 2011 04:37:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96836 invoked by uid 500); 1 May 2011 04:37:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96828 invoked by uid 99); 1 May 2011 04:37:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 May 2011 04:37:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 May 2011 04:37:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1F91DBBACD
	for <common-issues@hadoop.apache.org>; Sun,  1 May 2011 04:37:03 +0000 (UTC)
Date: Sun, 1 May 2011 04:37:03 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2068431797.14012.1304224623125.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <836774115.14005.1304224263168.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7254) Programmatically start  processes
 with JMX port open
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tanping Wang updated HADOOP-7254:
---------------------------------

    Attachment: HADOOP-7254.patch

> Programmatically start  processes with JMX port open 
> -----------------------------------------------------
>
>                 Key: HADOOP-7254
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7254
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: scripts
>            Reporter: Tanping Wang
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7254.patch
>
>
> We propose a programmatic way to start processes with JMX enabled.  This is the counter part of HDFS-1874.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14358-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 17:48:45 2011
Return-Path: <common-issues-return-14358-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9036735FA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  2 May 2011 17:48:45 +0000 (UTC)
Received: (qmail 67916 invoked by uid 500); 2 May 2011 17:48:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67854 invoked by uid 500); 2 May 2011 17:48:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67838 invoked by uid 99); 2 May 2011 17:48:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 17:48:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 17:48:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7C29ABE85E
	for <common-issues@hadoop.apache.org>; Mon,  2 May 2011 17:48:03 +0000 (UTC)
Date: Mon, 2 May 2011 17:48:03 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1821631581.15741.1304358483505.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7227:
-----------------------------------------

    Status: Patch Available  (was: Open)

> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14357-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 17:48:45 2011
Return-Path: <common-issues-return-14357-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8E53135F1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  2 May 2011 17:48:45 +0000 (UTC)
Received: (qmail 67895 invoked by uid 500); 2 May 2011 17:48:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67847 invoked by uid 500); 2 May 2011 17:48:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67831 invoked by uid 99); 2 May 2011 17:48:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 17:48:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 17:48:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 50FC2BE858
	for <common-issues@hadoop.apache.org>; Mon,  2 May 2011 17:48:03 +0000 (UTC)
Date: Mon, 2 May 2011 17:48:03 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <724703962.15735.1304358483328.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7227:
-----------------------------------------

    Status: Open  (was: Patch Available)

> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14359-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 18:30:44 2011
Return-Path: <common-issues-return-14359-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7C5863C33
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  2 May 2011 18:30:44 +0000 (UTC)
Received: (qmail 61654 invoked by uid 500); 2 May 2011 18:30:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61614 invoked by uid 500); 2 May 2011 18:30:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61606 invoked by uid 99); 2 May 2011 18:30:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 18:30:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 18:30:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3C441BE493
	for <common-issues@hadoop.apache.org>; Mon,  2 May 2011 18:30:03 +0000 (UTC)
Date: Mon, 2 May 2011 18:30:03 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <307593864.15803.1304361003243.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7227:
-----------------------------------------

    Status: Open  (was: Patch Available)

> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14360-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 18:34:42 2011
Return-Path: <common-issues-return-14360-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 72D8D3D51
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  2 May 2011 18:34:42 +0000 (UTC)
Received: (qmail 63728 invoked by uid 500); 2 May 2011 18:34:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63699 invoked by uid 500); 2 May 2011 18:34:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63691 invoked by uid 99); 2 May 2011 18:34:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 18:34:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 18:34:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4A2BFBE5FE
	for <common-issues@hadoop.apache.org>; Mon,  2 May 2011 18:34:03 +0000 (UTC)
Date: Mon, 2 May 2011 18:34:03 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <894488730.15813.1304361243300.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7227:
-----------------------------------------

    Attachment: HADOOP-7227.4.patch

> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14361-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 18:34:45 2011
Return-Path: <common-issues-return-14361-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3E1113D7D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  2 May 2011 18:34:45 +0000 (UTC)
Received: (qmail 64364 invoked by uid 500); 2 May 2011 18:34:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64319 invoked by uid 500); 2 May 2011 18:34:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64255 invoked by uid 99); 2 May 2011 18:34:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 18:34:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 18:34:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 75215BE604
	for <common-issues@hadoop.apache.org>; Mon,  2 May 2011 18:34:03 +0000 (UTC)
Date: Mon, 2 May 2011 18:34:03 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1074701532.15819.1304361243476.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7227:
-----------------------------------------

    Status: Patch Available  (was: Open)

> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14362-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 18:52:44 2011
Return-Path: <common-issues-return-14362-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7D6AB358D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  2 May 2011 18:52:44 +0000 (UTC)
Received: (qmail 746 invoked by uid 500); 2 May 2011 18:52:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 713 invoked by uid 500); 2 May 2011 18:52:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 703 invoked by uid 99); 2 May 2011 18:52:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 18:52:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 18:52:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 21B99BEB23
	for <common-issues@hadoop.apache.org>; Mon,  2 May 2011 18:52:03 +0000 (UTC)
Date: Mon, 2 May 2011 18:52:03 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <164716696.15840.1304362323134.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027766#comment-13027766 ] 

Hadoop QA commented on HADOOP-7227:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12477977/HADOOP-7227.4.patch
  against trunk revision 1097322.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 9 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/391//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/391//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/391//console

This message is automatically generated.

> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14363-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 19:10:42 2011
Return-Path: <common-issues-return-14363-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 62A65304B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  2 May 2011 19:10:42 +0000 (UTC)
Received: (qmail 36229 invoked by uid 500); 2 May 2011 19:10:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36193 invoked by uid 500); 2 May 2011 19:10:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36185 invoked by uid 99); 2 May 2011 19:10:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 19:10:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 19:10:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4B0DFBE0C4
	for <common-issues@hadoop.apache.org>; Mon,  2 May 2011 19:10:03 +0000 (UTC)
Date: Mon, 2 May 2011 19:10:03 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <753902427.15901.1304363403304.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027776#comment-13027776 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7227:
------------------------------------------------

+1 patch looks good.

> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14364-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 19:10:42 2011
Return-Path: <common-issues-return-14364-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 888923054
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  2 May 2011 19:10:42 +0000 (UTC)
Received: (qmail 36434 invoked by uid 500); 2 May 2011 19:10:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36367 invoked by uid 500); 2 May 2011 19:10:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36228 invoked by uid 99); 2 May 2011 19:10:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 19:10:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 19:10:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 588ECBE0C6
	for <common-issues@hadoop.apache.org>; Mon,  2 May 2011 19:10:03 +0000 (UTC)
Date: Mon, 2 May 2011 19:10:03 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1553371244.15903.1304363403359.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7227:
-------------------------------------------

     Component/s: ipc
      Issue Type: Improvement  (was: New Feature)
    Hadoop Flags: [Incompatible change, Reviewed]

> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14365-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 20:39:43 2011
Return-Path: <common-issues-return-14365-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 90C233D4D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  2 May 2011 20:39:43 +0000 (UTC)
Received: (qmail 55630 invoked by uid 500); 2 May 2011 20:39:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55575 invoked by uid 500); 2 May 2011 20:39:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55567 invoked by uid 99); 2 May 2011 20:39:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 20:39:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 20:39:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 68B4FBED10
	for <common-issues@hadoop.apache.org>; Mon,  2 May 2011 20:39:04 +0000 (UTC)
Date: Mon, 2 May 2011 20:39:04 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <271292767.16156.1304368744425.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-6255:
------------------------------

    Attachment: HADOOP-6255-branch-0.20-security-7.patch

- Revised patch to remove conf-pseudo packaging.
- Renamed setup-single-node-hadoop.sh to hadoop-setup-single-node.sh
- Symlink hadoop-config.sh to $PREFIX/libexec
- Improved detection of user defined HADOOP_HOME


> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14366-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 20:44:13 2011
Return-Path: <common-issues-return-14366-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9DEA93E72
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  2 May 2011 20:44:13 +0000 (UTC)
Received: (qmail 64442 invoked by uid 500); 2 May 2011 20:44:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64407 invoked by uid 500); 2 May 2011 20:44:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64399 invoked by uid 99); 2 May 2011 20:44:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 20:44:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 20:44:12 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 71378BEEC7
	for <common-issues@hadoop.apache.org>; Mon,  2 May 2011 20:43:04 +0000 (UTC)
Date: Mon, 2 May 2011 20:43:04 +0000 (UTC)
From: "Amr Awadallah (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1514623975.16189.1304368984460.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027831#comment-13027831 ] 

Amr Awadallah commented on HADOOP-6255:
---------------------------------------

I am out of office and will be slower than usual in responding to
emails. If this is urgent then please call my cell phone (or send an
SMS), otherwise I will reply to your email when I get back.

Thanks for your patience,

-- amr


> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14367-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 20:44:15 2011
Return-Path: <common-issues-return-14367-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A01973E7E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  2 May 2011 20:44:15 +0000 (UTC)
Received: (qmail 64705 invoked by uid 500); 2 May 2011 20:44:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64653 invoked by uid 500); 2 May 2011 20:44:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64645 invoked by uid 99); 2 May 2011 20:44:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 20:44:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 20:44:14 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6C7CABEEF1
	for <common-issues@hadoop.apache.org>; Mon,  2 May 2011 20:43:06 +0000 (UTC)
Date: Mon, 2 May 2011 20:43:06 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <753058259.16227.1304368986441.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027833#comment-13027833 ] 

Hadoop QA commented on HADOOP-6255:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12477985/HADOOP-6255-branch-0.20-security-7.patch
  against trunk revision 1097322.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/392//console

This message is automatically generated.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14368-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 20:53:44 2011
Return-Path: <common-issues-return-14368-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F168234AE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  2 May 2011 20:53:44 +0000 (UTC)
Received: (qmail 75253 invoked by uid 500); 2 May 2011 20:53:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75225 invoked by uid 500); 2 May 2011 20:53:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75214 invoked by uid 99); 2 May 2011 20:53:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 20:53:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 20:53:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4773CBE1E4
	for <common-issues@hadoop.apache.org>; Mon,  2 May 2011 20:53:03 +0000 (UTC)
Date: Mon, 2 May 2011 20:53:03 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1790211818.16247.1304369583288.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1243264568.21599.1301502005765.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7214) Hadoop /usr/bin/groups equivalent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7214:
--------------------------------

    Status: Open  (was: Patch Available)

> Hadoop /usr/bin/groups equivalent
> ---------------------------------
>
>                 Key: HADOOP-7214
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7214
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>         Attachments: hadoop-7214.0.txt, hadoop-7214.1.txt, hadoop-7214.2.txt, hadoop-7214.3.txt, hadoop-7214.4.txt, hadoop-7214.5.txt, hadoop-7214.6.txt
>
>
> Since user -> groups resolution is done on the NN and JT machines, there should be a way for users to determine what groups they're a member of from the NN's and JT's perspective.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14369-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 20:59:47 2011
Return-Path: <common-issues-return-14369-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C50CB36D5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  2 May 2011 20:59:47 +0000 (UTC)
Received: (qmail 83484 invoked by uid 500); 2 May 2011 20:59:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83403 invoked by uid 500); 2 May 2011 20:59:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83306 invoked by uid 99); 2 May 2011 20:59:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 20:59:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 20:59:44 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D0273BE4BA
	for <common-issues@hadoop.apache.org>; Mon,  2 May 2011 20:59:05 +0000 (UTC)
Date: Mon, 2 May 2011 20:59:05 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1920717579.16335.1304369945849.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7227:
-----------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
           Status: Resolved  (was: Patch Available)

I have committed this!

> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14370-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 21:13:45 2011
Return-Path: <common-issues-return-14370-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B53903DE4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  2 May 2011 21:13:45 +0000 (UTC)
Received: (qmail 8534 invoked by uid 500); 2 May 2011 21:13:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8397 invoked by uid 500); 2 May 2011 21:13:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8349 invoked by uid 99); 2 May 2011 21:13:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 21:13:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 21:13:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 50FEFBE9B7
	for <common-issues@hadoop.apache.org>; Mon,  2 May 2011 21:13:04 +0000 (UTC)
Date: Mon, 2 May 2011 21:13:04 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <189635228.16406.1304370784328.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027849#comment-13027849 ] 

Hudson commented on HADOOP-7227:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #567 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/567/])
    HADOOP-7227. Remove protocol version check at proxy creation in Hadoop RPC. Contributed by jitendra.


> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14371-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 22:45:42 2011
Return-Path: <common-issues-return-14371-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 69A853E73
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  2 May 2011 22:45:42 +0000 (UTC)
Received: (qmail 32991 invoked by uid 500); 2 May 2011 22:45:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32964 invoked by uid 500); 2 May 2011 22:45:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32956 invoked by uid 99); 2 May 2011 22:45:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 22:45:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 22:45:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 481F0BE664
	for <common-issues@hadoop.apache.org>; Mon,  2 May 2011 22:45:03 +0000 (UTC)
Date: Mon, 2 May 2011 22:45:03 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1149429540.16601.1304376303292.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7227:
-----------------------------------------

    Release Note: 
1. Protocol version check is removed from proxy creation, instead version check is performed at server in every rpc call.
2. This change is backward incompatible because format of the rpc messages is changed to include client version, client method hash and rpc version.
3. rpc version is introduced which should change when the format of rpc messages is changed.

> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14372-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 22:47:42 2011
Return-Path: <common-issues-return-14372-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AE3CD3CB2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  2 May 2011 22:47:42 +0000 (UTC)
Received: (qmail 35452 invoked by uid 500); 2 May 2011 22:47:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35418 invoked by uid 500); 2 May 2011 22:47:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35410 invoked by uid 99); 2 May 2011 22:47:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 22:47:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 22:47:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8DB0ABE72B
	for <common-issues@hadoop.apache.org>; Mon,  2 May 2011 22:47:03 +0000 (UTC)
Date: Mon, 2 May 2011 22:47:03 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1792041693.16619.1304376423577.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <545918969.3211.1299545460466.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7170) Support UGI in FileContext API
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey resolved HADOOP-7170.
------------------------------------------

    Resolution: Duplicate

This is duplicate of HADOOP-7171.

> Support UGI in FileContext API
> ------------------------------
>
>                 Key: HADOOP-7170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7170
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>
> The FileContext API needs to support UGI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14373-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 22:54:44 2011
Return-Path: <common-issues-return-14373-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B12F83FD3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  2 May 2011 22:54:44 +0000 (UTC)
Received: (qmail 44785 invoked by uid 500); 2 May 2011 22:54:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44741 invoked by uid 500); 2 May 2011 22:54:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44733 invoked by uid 99); 2 May 2011 22:54:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 22:54:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 22:54:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A0CE5BE98B
	for <common-issues@hadoop.apache.org>; Mon,  2 May 2011 22:54:03 +0000 (UTC)
Date: Mon, 2 May 2011 22:54:03 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1815350748.16636.1304376843655.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027897#comment-13027897 ] 

Todd Lipcon commented on HADOOP-7227:
-------------------------------------

This patch seems to have broken MR. I'm getting:

    [junit] 11/05/02 15:51:51 ERROR mapred.TaskTracker: Caught exception: java.lang.RuntimeException: java.lang.IllegalAccessException: Class org.apache.hadoop.ipc.WritableRpcEngine$Invocation can not access a member of class org.apache.hadoop.mapred.InterTrackerProtocol with modifiers "public static final"
    [junit]     at org.apache.hadoop.ipc.WritableRpcEngine$Invocation.<init>(WritableRpcEngine.java:85)
    [junit]     at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:249)
    [junit]     at org.apache.hadoop.mapred.$Proxy7.getBuildVersion(Unknown Source)
    [junit]     at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1500)

in a tight loop.

> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14374-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 23:00:46 2011
Return-Path: <common-issues-return-14374-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1124B34F0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  2 May 2011 23:00:46 +0000 (UTC)
Received: (qmail 57310 invoked by uid 500); 2 May 2011 23:00:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57277 invoked by uid 500); 2 May 2011 23:00:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57261 invoked by uid 99); 2 May 2011 23:00:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 23:00:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 23:00:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5ED4BBEB99
	for <common-issues@hadoop.apache.org>; Mon,  2 May 2011 23:00:04 +0000 (UTC)
Date: Mon, 2 May 2011 23:00:04 +0000 (UTC)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <845863001.16682.1304377204384.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027902#comment-13027902 ] 

Konstantin Boudnik commented on HADOOP-6255:
--------------------------------------------

bq. I'm still a little concerned that the deployment document talks about things that are outside the scope of this JIRA. Even worse, it mentions things that Apache cannot distribute (the LZO code). This should really get sanitized for Hadoop.

A valid point indeed. I think having packing outside of the project itself will address such concerns with ease, because packaging can be licensed differently than Hadoop proper.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14375-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 23:00:46 2011
Return-Path: <common-issues-return-14375-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2AA8E34F7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  2 May 2011 23:00:46 +0000 (UTC)
Received: (qmail 57347 invoked by uid 500); 2 May 2011 23:00:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57279 invoked by uid 500); 2 May 2011 23:00:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57262 invoked by uid 99); 2 May 2011 23:00:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 23:00:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 23:00:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8A737BEBA0
	for <common-issues@hadoop.apache.org>; Mon,  2 May 2011 23:00:04 +0000 (UTC)
Date: Mon, 2 May 2011 23:00:04 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <953475594.16688.1304377204564.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027903#comment-13027903 ] 

Todd Lipcon commented on HADOOP-7227:
-------------------------------------

I believe we either need to make all protocols public interfaces, or we need to call setAccessible(true) on the Field instance.

> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14376-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 23:02:42 2011
Return-Path: <common-issues-return-14376-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B78173DC0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  2 May 2011 23:02:42 +0000 (UTC)
Received: (qmail 58414 invoked by uid 500); 2 May 2011 23:02:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58384 invoked by uid 500); 2 May 2011 23:02:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58376 invoked by uid 99); 2 May 2011 23:02:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 23:02:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 23:02:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5F8B7BEC7B
	for <common-issues@hadoop.apache.org>; Mon,  2 May 2011 23:02:03 +0000 (UTC)
Date: Mon, 2 May 2011 23:02:03 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2045225725.16697.1304377323388.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027904#comment-13027904 ] 

Todd Lipcon commented on HADOOP-7227:
-------------------------------------

Making InterTrackerProtocol public got past that issue, but now hitting:

    [junit] java.lang.IllegalAccessError: tried to access class org.apache.hadoop.mapred.HeartbeatResponse from class $Proxy7
    [junit]     at $Proxy7.heartbeat(Unknown Source)
    [junit]     at org.apache.hadoop.mapred.TaskTracker.transmitHeartBeat(TaskTracker.java:1692)
    [junit]     at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1523)
    [junit]     at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:2428)
    [junit]     at org.apache.hadoop.mapred.MiniMRCluster$TaskTrackerRunner.run(MiniMRCluster.java:229)

I am going to temporarily revert this patch from common, since it was insufficiently tested and is breaking the dependent projects.


> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14377-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 23:02:45 2011
Return-Path: <common-issues-return-14377-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7338E3E00
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  2 May 2011 23:02:45 +0000 (UTC)
Received: (qmail 58725 invoked by uid 500); 2 May 2011 23:02:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58618 invoked by uid 500); 2 May 2011 23:02:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58542 invoked by uid 99); 2 May 2011 23:02:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 23:02:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 23:02:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8D217BEC82
	for <common-issues@hadoop.apache.org>; Mon,  2 May 2011 23:02:03 +0000 (UTC)
Date: Mon, 2 May 2011 23:02:03 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <654164627.16704.1304377323575.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Reopened] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon reopened HADOOP-7227:
---------------------------------


> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14378-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 23:06:42 2011
Return-Path: <common-issues-return-14378-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 85FDD367B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  2 May 2011 23:06:42 +0000 (UTC)
Received: (qmail 63764 invoked by uid 500); 2 May 2011 23:06:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63714 invoked by uid 500); 2 May 2011 23:06:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63706 invoked by uid 99); 2 May 2011 23:06:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 23:06:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 23:06:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4C519BED6F
	for <common-issues@hadoop.apache.org>; Mon,  2 May 2011 23:06:03 +0000 (UTC)
Date: Mon, 2 May 2011 23:06:03 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1783166029.16710.1304377563309.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027905#comment-13027905 ] 

Jitendra Nath Pandey commented on HADOOP-7227:
----------------------------------------------

I am running mr tests to look into the issue.

> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14379-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  2 23:24:44 2011
Return-Path: <common-issues-return-14379-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 44A6D3D1A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  2 May 2011 23:24:44 +0000 (UTC)
Received: (qmail 77256 invoked by uid 500); 2 May 2011 23:24:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77178 invoked by uid 500); 2 May 2011 23:24:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77170 invoked by uid 99); 2 May 2011 23:24:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 23:24:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 May 2011 23:24:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3A324BE295
	for <common-issues@hadoop.apache.org>; Mon,  2 May 2011 23:24:03 +0000 (UTC)
Date: Mon, 2 May 2011 23:24:03 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1830219609.16756.1304378643234.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027916#comment-13027916 ] 

Hudson commented on HADOOP-7227:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #568 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/568/])
    Revert HADOOP-7227 from r1098792 since it broke HDFS and MR builds.

( svn merge -c -1098792 )


> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14380-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 01:50:44 2011
Return-Path: <common-issues-return-14380-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DABE436A5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 01:50:43 +0000 (UTC)
Received: (qmail 33020 invoked by uid 500); 3 May 2011 01:50:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32969 invoked by uid 500); 3 May 2011 01:50:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32927 invoked by uid 99); 3 May 2011 01:50:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 01:50:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 01:50:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 576A9BEBA0
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 01:50:03 +0000 (UTC)
Date: Tue, 3 May 2011 01:50:03 +0000 (UTC)
From: "T Jake Luciani (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <76604007.17212.1304387403354.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7255) Performance regression bug caused by
 locking code
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Performance regression bug caused by locking code
-------------------------------------------------

                 Key: HADOOP-7255
                 URL: https://issues.apache.org/jira/browse/HADOOP-7255
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.20.203.0
            Reporter: T Jake Luciani


While testing the 0.20.203 branch we have discovered quite a severe performance regression caused by the following checkin:

http://svn.apache.org/viewvc?view=revision&revision=1079260

This locking seems to cause a pause of a few seconds between tasks, example with DFSIOTest:

*CURRENT CODE*
{noformat}
11/05/02 21:16:28 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
11/05/02 21:16:28 INFO fs.TestDFSIO:            Date & time: Mon May 02 21:16:28 EDT 2011
11/05/02 21:16:28 INFO fs.TestDFSIO:        Number of files: 5
11/05/02 21:16:28 INFO fs.TestDFSIO: Total MBytes processed: 500
11/05/02 21:16:28 INFO fs.TestDFSIO:      Throughput mb/sec: 303.3980582524272
11/05/02 21:16:28 INFO fs.TestDFSIO: Average IO rate mb/sec: 327.69122314453125
11/05/02 21:16:28 INFO fs.TestDFSIO:  IO rate std deviation: 78.95767116969083
11/05/02 21:16:28 INFO fs.TestDFSIO:     Test exec time sec: 19.476
11/05/02 21:16:28 INFO fs.TestDFSIO: 
{noformat}


*WITH PATCH REMOVED*
{noformat}
11/05/02 21:19:03 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
11/05/02 21:19:03 INFO fs.TestDFSIO:            Date & time: Mon May 02 21:19:03 EDT 2011
11/05/02 21:19:03 INFO fs.TestDFSIO:        Number of files: 5
11/05/02 21:19:03 INFO fs.TestDFSIO: Total MBytes processed: 500
11/05/02 21:19:03 INFO fs.TestDFSIO:      Throughput mb/sec: 366.03221083455344
11/05/02 21:19:03 INFO fs.TestDFSIO: Average IO rate mb/sec: 366.35528564453125
11/05/02 21:19:03 INFO fs.TestDFSIO:  IO rate std deviation: 11.020931463080379
11/05/02 21:19:03 INFO fs.TestDFSIO:     Test exec time sec: 3.543
11/05/02 21:19:03 INFO fs.TestDFSIO: 
{noformat}


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14381-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 05:35:46 2011
Return-Path: <common-issues-return-14381-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2C58DD1B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 05:35:46 +0000 (UTC)
Received: (qmail 92955 invoked by uid 500); 3 May 2011 05:35:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92869 invoked by uid 500); 3 May 2011 05:35:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92861 invoked by uid 99); 3 May 2011 05:35:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 05:35:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 05:35:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 901E5BF031
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 05:35:03 +0000 (UTC)
Date: Tue, 3 May 2011 05:35:03 +0000 (UTC)
From: "ramkrishna.s.vasudevan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <9109019.17508.1304400903587.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1720286243.52045.1302604686655.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7221) When Namenode network is
 unplugged, DFSClient operations waits for ever
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028030#comment-13028030 ] 

ramkrishna.s.vasudevan commented on HADOOP-7221:
------------------------------------------------

The problem was whenever network was unplugged the read operation was getting a timedout exception and it was trying again. This continued for almost 15 times and only then some connectionloss exception came and could come out.
By this time it was taking around 45 mins.
Hence we have done something like, configure the parameter 
"max.ping.retries.on.socket.timeout" to a value where you can configure a value after which it should come out after getting a socket time.  So while retrying chk for this configured value and once reached come out.

This problem comes only in unplug scenarios.  So based on the scenario this value can be configured as to when how much time it should chk to get a connection.

> When Namenode network is unplugged, DFSClient operations waits for ever
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-7221
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7221
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>            Reporter: Uma Maheswara Rao G
>
> When NN/DN is shutdown gracefully, the DFSClient operations which are waiting for a response from NN/DN, will throw exception & come out quickly
> But when the NN/DN network is unplugged, the DFSClient operations which are waiting for a response from NN/DN, waits for ever.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14382-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 05:45:47 2011
Return-Path: <common-issues-return-14382-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 25FBA26F0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 05:45:47 +0000 (UTC)
Received: (qmail 99327 invoked by uid 500); 3 May 2011 05:45:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99265 invoked by uid 500); 3 May 2011 05:45:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99175 invoked by uid 99); 3 May 2011 05:45:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 05:45:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 05:45:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 29F86BF4E6
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 05:45:03 +0000 (UTC)
Date: Tue, 3 May 2011 05:45:03 +0000 (UTC)
From: "Bharath Mundlapudi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1278903595.17547.1304401503168.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <76604007.17212.1304387403354.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7255) Performance regression bug caused
 by locking code
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028039#comment-13028039 ] 

Bharath Mundlapudi commented on HADOOP-7255:
--------------------------------------------

Thanks for posting performance data. Can you please let us know the test setup details? Like how big is your cluster?
It is interesting to see high Std Deviation without the patch. Did you try multiple runs?   


> Performance regression bug caused by locking code
> -------------------------------------------------
>
>                 Key: HADOOP-7255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7255
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.203.0
>            Reporter: T Jake Luciani
>
> While testing the 0.20.203 branch we have discovered quite a severe performance regression caused by the following checkin:
> http://svn.apache.org/viewvc?view=revision&revision=1079260
> This locking seems to cause a pause of a few seconds between tasks, example with DFSIOTest:
> *CURRENT CODE*
> {noformat}
> 11/05/02 21:16:28 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
> 11/05/02 21:16:28 INFO fs.TestDFSIO:            Date & time: Mon May 02 21:16:28 EDT 2011
> 11/05/02 21:16:28 INFO fs.TestDFSIO:        Number of files: 5
> 11/05/02 21:16:28 INFO fs.TestDFSIO: Total MBytes processed: 500
> 11/05/02 21:16:28 INFO fs.TestDFSIO:      Throughput mb/sec: 303.3980582524272
> 11/05/02 21:16:28 INFO fs.TestDFSIO: Average IO rate mb/sec: 327.69122314453125
> 11/05/02 21:16:28 INFO fs.TestDFSIO:  IO rate std deviation: 78.95767116969083
> 11/05/02 21:16:28 INFO fs.TestDFSIO:     Test exec time sec: 19.476
> 11/05/02 21:16:28 INFO fs.TestDFSIO: 
> {noformat}
> *WITH PATCH REMOVED*
> {noformat}
> 11/05/02 21:19:03 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
> 11/05/02 21:19:03 INFO fs.TestDFSIO:            Date & time: Mon May 02 21:19:03 EDT 2011
> 11/05/02 21:19:03 INFO fs.TestDFSIO:        Number of files: 5
> 11/05/02 21:19:03 INFO fs.TestDFSIO: Total MBytes processed: 500
> 11/05/02 21:19:03 INFO fs.TestDFSIO:      Throughput mb/sec: 366.03221083455344
> 11/05/02 21:19:03 INFO fs.TestDFSIO: Average IO rate mb/sec: 366.35528564453125
> 11/05/02 21:19:03 INFO fs.TestDFSIO:  IO rate std deviation: 11.020931463080379
> 11/05/02 21:19:03 INFO fs.TestDFSIO:     Test exec time sec: 3.543
> 11/05/02 21:19:03 INFO fs.TestDFSIO: 
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14383-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 05:51:52 2011
Return-Path: <common-issues-return-14383-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A3AE927BB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 05:51:52 +0000 (UTC)
Received: (qmail 3116 invoked by uid 500); 3 May 2011 05:51:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2814 invoked by uid 500); 3 May 2011 05:51:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2494 invoked by uid 99); 3 May 2011 05:51:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 05:51:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 05:51:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 67E61BF7A4
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 05:51:03 +0000 (UTC)
Date: Tue, 3 May 2011 05:51:03 +0000 (UTC)
From: "Bharath Mundlapudi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1002036916.17572.1304401863422.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <76604007.17212.1304387403354.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7255) Performance regression bug caused
 by locking code
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028040#comment-13028040 ] 

Bharath Mundlapudi commented on HADOOP-7255:
--------------------------------------------

Correction: Std deviation is high with patch.

> Performance regression bug caused by locking code
> -------------------------------------------------
>
>                 Key: HADOOP-7255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7255
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.203.0
>            Reporter: T Jake Luciani
>
> While testing the 0.20.203 branch we have discovered quite a severe performance regression caused by the following checkin:
> http://svn.apache.org/viewvc?view=revision&revision=1079260
> This locking seems to cause a pause of a few seconds between tasks, example with DFSIOTest:
> *CURRENT CODE*
> {noformat}
> 11/05/02 21:16:28 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
> 11/05/02 21:16:28 INFO fs.TestDFSIO:            Date & time: Mon May 02 21:16:28 EDT 2011
> 11/05/02 21:16:28 INFO fs.TestDFSIO:        Number of files: 5
> 11/05/02 21:16:28 INFO fs.TestDFSIO: Total MBytes processed: 500
> 11/05/02 21:16:28 INFO fs.TestDFSIO:      Throughput mb/sec: 303.3980582524272
> 11/05/02 21:16:28 INFO fs.TestDFSIO: Average IO rate mb/sec: 327.69122314453125
> 11/05/02 21:16:28 INFO fs.TestDFSIO:  IO rate std deviation: 78.95767116969083
> 11/05/02 21:16:28 INFO fs.TestDFSIO:     Test exec time sec: 19.476
> 11/05/02 21:16:28 INFO fs.TestDFSIO: 
> {noformat}
> *WITH PATCH REMOVED*
> {noformat}
> 11/05/02 21:19:03 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
> 11/05/02 21:19:03 INFO fs.TestDFSIO:            Date & time: Mon May 02 21:19:03 EDT 2011
> 11/05/02 21:19:03 INFO fs.TestDFSIO:        Number of files: 5
> 11/05/02 21:19:03 INFO fs.TestDFSIO: Total MBytes processed: 500
> 11/05/02 21:19:03 INFO fs.TestDFSIO:      Throughput mb/sec: 366.03221083455344
> 11/05/02 21:19:03 INFO fs.TestDFSIO: Average IO rate mb/sec: 366.35528564453125
> 11/05/02 21:19:03 INFO fs.TestDFSIO:  IO rate std deviation: 11.020931463080379
> 11/05/02 21:19:03 INFO fs.TestDFSIO:     Test exec time sec: 3.543
> 11/05/02 21:19:03 INFO fs.TestDFSIO: 
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14384-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 06:33:46 2011
Return-Path: <common-issues-return-14384-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 471C72953
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 06:33:46 +0000 (UTC)
Received: (qmail 31605 invoked by uid 500); 3 May 2011 06:33:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31398 invoked by uid 500); 3 May 2011 06:33:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31315 invoked by uid 99); 3 May 2011 06:33:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 06:33:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 06:33:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3180BBF51B
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 06:33:03 +0000 (UTC)
Date: Tue, 3 May 2011 06:33:03 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <389402084.17703.1304404383199.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7227:
-----------------------------------------

    Attachment: HADOOP-7227.5.patch

New patch calls setAccessible(true) for versionID field. 

> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch, HADOOP-7227.5.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14385-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 06:35:46 2011
Return-Path: <common-issues-return-14385-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B55202C92
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 06:35:46 +0000 (UTC)
Received: (qmail 35251 invoked by uid 500); 3 May 2011 06:35:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35229 invoked by uid 500); 3 May 2011 06:35:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35213 invoked by uid 99); 3 May 2011 06:35:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 06:35:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 06:35:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1CEFCBF5FA
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 06:35:03 +0000 (UTC)
Date: Tue, 3 May 2011 06:35:03 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <85220738.17711.1304404503115.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7227:
-----------------------------------------

    Status: Patch Available  (was: Reopened)

> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch, HADOOP-7227.5.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14386-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 06:58:45 2011
Return-Path: <common-issues-return-14386-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 432042340
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 06:58:45 +0000 (UTC)
Received: (qmail 53708 invoked by uid 500); 3 May 2011 06:58:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53613 invoked by uid 500); 3 May 2011 06:58:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53598 invoked by uid 99); 3 May 2011 06:58:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 06:58:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 06:58:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2C405BFF18
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 06:58:03 +0000 (UTC)
Date: Tue, 3 May 2011 06:58:03 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1816590255.17759.1304405883177.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028073#comment-13028073 ] 

Hadoop QA commented on HADOOP-7227:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478032/HADOOP-7227.5.patch
  against trunk revision 1098840.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 9 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/393//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/393//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/393//console

This message is automatically generated.

> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch, HADOOP-7227.5.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14387-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 11:35:45 2011
Return-Path: <common-issues-return-14387-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9019F21F8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 11:35:45 +0000 (UTC)
Received: (qmail 14251 invoked by uid 500); 3 May 2011 11:35:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14207 invoked by uid 500); 3 May 2011 11:35:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14175 invoked by uid 99); 3 May 2011 11:35:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 11:35:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 11:35:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3807EBF04C
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 11:35:03 +0000 (UTC)
Date: Tue, 3 May 2011 11:35:03 +0000 (UTC)
From: "T Jake Luciani (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1547087401.18257.1304422503226.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <76604007.17212.1304387403354.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7255) Performance regression bug caused
 by locking code
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028162#comment-13028162 ] 

T Jake Luciani commented on HADOOP-7255:
----------------------------------------

For simplicity this is a single node from a fresh checkout.  Yes I did run this multiple times... the stddev varies on read from 78 - 30... writes are fine in terms of stddev.

If you build the test jar you should be able to reproduce locally.

> Performance regression bug caused by locking code
> -------------------------------------------------
>
>                 Key: HADOOP-7255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7255
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.203.0
>            Reporter: T Jake Luciani
>
> While testing the 0.20.203 branch we have discovered quite a severe performance regression caused by the following checkin:
> http://svn.apache.org/viewvc?view=revision&revision=1079260
> This locking seems to cause a pause of a few seconds between tasks, example with DFSIOTest:
> *CURRENT CODE*
> {noformat}
> 11/05/02 21:16:28 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
> 11/05/02 21:16:28 INFO fs.TestDFSIO:            Date & time: Mon May 02 21:16:28 EDT 2011
> 11/05/02 21:16:28 INFO fs.TestDFSIO:        Number of files: 5
> 11/05/02 21:16:28 INFO fs.TestDFSIO: Total MBytes processed: 500
> 11/05/02 21:16:28 INFO fs.TestDFSIO:      Throughput mb/sec: 303.3980582524272
> 11/05/02 21:16:28 INFO fs.TestDFSIO: Average IO rate mb/sec: 327.69122314453125
> 11/05/02 21:16:28 INFO fs.TestDFSIO:  IO rate std deviation: 78.95767116969083
> 11/05/02 21:16:28 INFO fs.TestDFSIO:     Test exec time sec: 19.476
> 11/05/02 21:16:28 INFO fs.TestDFSIO: 
> {noformat}
> *WITH PATCH REMOVED*
> {noformat}
> 11/05/02 21:19:03 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
> 11/05/02 21:19:03 INFO fs.TestDFSIO:            Date & time: Mon May 02 21:19:03 EDT 2011
> 11/05/02 21:19:03 INFO fs.TestDFSIO:        Number of files: 5
> 11/05/02 21:19:03 INFO fs.TestDFSIO: Total MBytes processed: 500
> 11/05/02 21:19:03 INFO fs.TestDFSIO:      Throughput mb/sec: 366.03221083455344
> 11/05/02 21:19:03 INFO fs.TestDFSIO: Average IO rate mb/sec: 366.35528564453125
> 11/05/02 21:19:03 INFO fs.TestDFSIO:  IO rate std deviation: 11.020931463080379
> 11/05/02 21:19:03 INFO fs.TestDFSIO:     Test exec time sec: 3.543
> 11/05/02 21:19:03 INFO fs.TestDFSIO: 
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14388-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 13:31:43 2011
Return-Path: <common-issues-return-14388-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6BAC4927
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 13:31:43 +0000 (UTC)
Received: (qmail 94827 invoked by uid 500); 3 May 2011 13:31:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94285 invoked by uid 500); 3 May 2011 13:31:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94122 invoked by uid 99); 3 May 2011 13:31:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 13:31:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 13:31:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4777BBFC88
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 13:31:03 +0000 (UTC)
Date: Tue, 3 May 2011 13:31:03 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1487106568.18479.1304429463289.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028219#comment-13028219 ] 

Hudson commented on HADOOP-7227:
--------------------------------

Integrated in Hadoop-Common-trunk #677 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/677/])
    Revert HADOOP-7227 from r1098792 since it broke HDFS and MR builds.

( svn merge -c -1098792 )
HADOOP-7227. Remove protocol version check at proxy creation in Hadoop RPC. Contributed by jitendra.


> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch, HADOOP-7227.5.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14389-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 15:20:45 2011
Return-Path: <common-issues-return-14389-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2511E294A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 15:20:45 +0000 (UTC)
Received: (qmail 82807 invoked by uid 500); 3 May 2011 15:20:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82760 invoked by uid 500); 3 May 2011 15:20:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82752 invoked by uid 99); 3 May 2011 15:20:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 15:20:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 15:20:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 940C7BF9DA
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 15:20:03 +0000 (UTC)
Date: Tue, 3 May 2011 15:20:03 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <704905605.18945.1304436003603.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1720286243.52045.1302604686655.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7221) When Namenode network is
 unplugged, DFSClient operations waits for ever
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028262#comment-13028262 ] 

Steve Loughran commented on HADOOP-7221:
----------------------------------------

# which version are you seeing this on?
# when you say unplugged, do you mean the ethernet port of your local machine came unplugged, or the connection to the remote server failed?
# Can you add the stack trace you see in the exceptions, to show where the problem is?

This is an HDFS problem, so re-assigning there


> When Namenode network is unplugged, DFSClient operations waits for ever
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-7221
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7221
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: hdfs client
>            Reporter: Uma Maheswara Rao G
>
> When NN/DN is shutdown gracefully, the DFSClient operations which are waiting for a response from NN/DN, will throw exception & come out quickly
> But when the NN/DN network is unplugged, the DFSClient operations which are waiting for a response from NN/DN, waits for ever.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14390-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 15:20:45 2011
Return-Path: <common-issues-return-14390-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6FF332959
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 15:20:45 +0000 (UTC)
Received: (qmail 83028 invoked by uid 500); 3 May 2011 15:20:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82994 invoked by uid 500); 3 May 2011 15:20:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82979 invoked by uid 99); 3 May 2011 15:20:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 15:20:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 15:20:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A0265BF9DC
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 15:20:03 +0000 (UTC)
Date: Tue, 3 May 2011 15:20:03 +0000 (UTC)
From: "ramkrishna.s.vasudevan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <282223217.18947.1304436003652.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7256) Resource leak during failure
 scenario of closing of resources.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Resource leak during failure scenario of closing of resources. 
---------------------------------------------------------------

                 Key: HADOOP-7256
                 URL: https://issues.apache.org/jira/browse/HADOOP-7256
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.21.0, 0.20.2
            Reporter: ramkrishna.s.vasudevan
            Priority: Minor


Problem Statement:
===============
There are chances of resource leak and stream not getting closed 
Take the case when after copying data we try to close the Input and output stream followed by closing of the socket.
Suppose an exception occurs while closing the input stream(due to runtime exception) then the subsequent operations of closing the output stream and socket may not happen and there is a chance of resource leak.

Scenario 
=======
During long run of map reduce jobs, the copyFromLocalFile() api is getting called.
Here we found some exceptions happening. As a result of this we found the lsof value raising leading to resource leak.

Solution:
=======
While doing a close operation of any resource catch the RuntimeException also rather than catching the IOException alone.

Additionally there are places where we try to close a resource in the catch block.
If this close fails, we just throw and come out of the current flow.
In order to avoid this, we can carry out the close operation in the finally block.

Probable reasons for getting RunTimeExceptions:
=====================================
We have many wrapped stream for writing and reading.
These wrappers are prone to errors.


 



--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14391-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 17:15:44 2011
Return-Path: <common-issues-return-14391-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BAFED2D3C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 17:15:44 +0000 (UTC)
Received: (qmail 82950 invoked by uid 500); 3 May 2011 17:15:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82893 invoked by uid 500); 3 May 2011 17:15:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82885 invoked by uid 99); 3 May 2011 17:15:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 17:15:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 17:15:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5E76DBF874
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 17:15:03 +0000 (UTC)
Date: Tue, 3 May 2011 17:15:03 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <166057275.19192.1304442903383.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028319#comment-13028319 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7227:
------------------------------------------------

+1 the new patch looks good.

> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch, HADOOP-7227.5.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14392-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 17:15:45 2011
Return-Path: <common-issues-return-14392-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E60B22D48
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 17:15:44 +0000 (UTC)
Received: (qmail 83153 invoked by uid 500); 3 May 2011 17:15:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83119 invoked by uid 500); 3 May 2011 17:15:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82996 invoked by uid 99); 3 May 2011 17:15:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 17:15:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 17:15:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6B500BF876
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 17:15:03 +0000 (UTC)
Date: Tue, 3 May 2011 17:15:03 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1414292993.19194.1304442903436.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028320#comment-13028320 ] 

Jitendra Nath Pandey commented on HADOOP-7227:
----------------------------------------------

I ran hdfs tests and mr tests with this patch in common.

hdfs: All tests pass except TestHdfsTrash, TestReplaceDatanodeOnFailure, TestMetaSave which failed for unrelated reasons

mr: All tests passed except TestMiniMRChildTask which also failed for unrelated reason.

> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch, HADOOP-7227.5.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14393-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 17:21:42 2011
Return-Path: <common-issues-return-14393-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 75A072145
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 17:21:42 +0000 (UTC)
Received: (qmail 90216 invoked by uid 500); 3 May 2011 17:21:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90165 invoked by uid 500); 3 May 2011 17:21:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90155 invoked by uid 99); 3 May 2011 17:21:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 17:21:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 17:21:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 24DB0BFA69
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 17:21:03 +0000 (UTC)
Date: Tue, 3 May 2011 17:21:03 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <572111323.19204.1304443263147.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7227:
-----------------------------------------

    Status: Open  (was: Patch Available)

> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch, HADOOP-7227.5.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14394-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 17:23:42 2011
Return-Path: <common-issues-return-14394-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AC6B32186
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 17:23:42 +0000 (UTC)
Received: (qmail 91730 invoked by uid 500); 3 May 2011 17:23:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91700 invoked by uid 500); 3 May 2011 17:23:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91692 invoked by uid 99); 3 May 2011 17:23:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 17:23:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 17:23:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 65BF8BFB3E
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 17:23:03 +0000 (UTC)
Date: Tue, 3 May 2011 17:23:03 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1727467717.19215.1304443383413.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7227:
-----------------------------------------

    Attachment: HADOOP-7227.6.patch

Similar change as previous one in fetchServerMethods in ProtocolProxy as well. 

> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch, HADOOP-7227.5.patch, HADOOP-7227.6.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14395-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 17:25:45 2011
Return-Path: <common-issues-return-14395-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ED1972236
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 17:25:44 +0000 (UTC)
Received: (qmail 97878 invoked by uid 500); 3 May 2011 17:25:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97839 invoked by uid 500); 3 May 2011 17:25:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97831 invoked by uid 99); 3 May 2011 17:25:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 17:25:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 17:25:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7E77ABFC9B
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 17:25:03 +0000 (UTC)
Date: Tue, 3 May 2011 17:25:03 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1351846868.19227.1304443503514.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7227:
-----------------------------------------

    Status: Patch Available  (was: Open)

> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch, HADOOP-7227.5.patch, HADOOP-7227.6.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14396-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 17:49:44 2011
Return-Path: <common-issues-return-14396-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9DDB72327
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 17:49:44 +0000 (UTC)
Received: (qmail 51533 invoked by uid 500); 3 May 2011 17:49:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51494 invoked by uid 500); 3 May 2011 17:49:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51486 invoked by uid 99); 3 May 2011 17:49:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 17:49:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 17:49:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 28E37BF711
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 17:49:03 +0000 (UTC)
Date: Tue, 3 May 2011 17:49:03 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <487222198.19292.1304444943164.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028342#comment-13028342 ] 

Hadoop QA commented on HADOOP-7227:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478070/HADOOP-7227.6.patch
  against trunk revision 1098840.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 9 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/394//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/394//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/394//console

This message is automatically generated.

> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch, HADOOP-7227.5.patch, HADOOP-7227.6.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14397-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 18:01:46 2011
Return-Path: <common-issues-return-14397-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 127E12F3B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 18:01:46 +0000 (UTC)
Received: (qmail 75476 invoked by uid 500); 3 May 2011 18:01:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75437 invoked by uid 500); 3 May 2011 18:01:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75426 invoked by uid 99); 3 May 2011 18:01:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 18:01:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 18:01:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 99415BFD38
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 18:01:04 +0000 (UTC)
Date: Tue, 3 May 2011 18:01:04 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1845876588.19344.1304445664624.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Moved] (HADOOP-7257) A client side mount table to give
 per-application/per-job file system view
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley moved HDFS-1053 to HADOOP-7257:
---------------------------------------------

    Affects Version/s:     (was: 0.23.0)
                       0.23.0
                  Key: HADOOP-7257  (was: HDFS-1053)
              Project: Hadoop Common  (was: Hadoop HDFS)

> A client side mount table to give per-application/per-job file system view
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7257
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7257
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: ViewFs - javdoc2.pdf, ViewFsJavaDoc.pdf, viewFs2.patch, viewFs3.patch, viewfs1.patch
>
>
> This jira proposes a client side mount table to allow application-centric (or job-centric) filesystem views. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14398-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 18:19:43 2011
Return-Path: <common-issues-return-14398-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A8CE92B66
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 18:19:43 +0000 (UTC)
Received: (qmail 9610 invoked by uid 500); 3 May 2011 18:19:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9563 invoked by uid 500); 3 May 2011 18:19:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9530 invoked by uid 99); 3 May 2011 18:19:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 18:19:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 18:19:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 47E0BBF374
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 18:19:03 +0000 (UTC)
Date: Tue, 3 May 2011 18:19:03 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1639623250.19373.1304446743291.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7257) A client side mount table to give
 per-application/per-job file system view
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028356#comment-13028356 ] 

Hadoop QA commented on HADOOP-7257:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478021/viewFs3.patch
  against trunk revision 1098840.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 54 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The applied patch generated 1077 javac compiler warnings (more than the trunk's current 1070 warnings).

    -1 findbugs.  The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings.

    -1 release audit.  The applied patch generated 8 release audit warnings (more than the trunk's current 1 warnings).

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/395//testReport/
Release audit warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/395//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/395//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/395//console

This message is automatically generated.

> A client side mount table to give per-application/per-job file system view
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7257
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7257
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: ViewFs - javdoc2.pdf, ViewFsJavaDoc.pdf, viewFs2.patch, viewFs3.patch, viewfs1.patch
>
>
> This jira proposes a client side mount table to allow application-centric (or job-centric) filesystem views. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14399-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 19:00:42 2011
Return-Path: <common-issues-return-14399-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5D5A42980
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 19:00:42 +0000 (UTC)
Received: (qmail 82829 invoked by uid 500); 3 May 2011 19:00:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82803 invoked by uid 500); 3 May 2011 19:00:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82795 invoked by uid 99); 3 May 2011 19:00:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 19:00:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 19:00:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1D39CBF265
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 19:00:03 +0000 (UTC)
Date: Tue, 3 May 2011 19:00:03 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <64692328.19459.1304449203116.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7258) Gzip codec should not return null
 decompressors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Gzip codec should not return null decompressors
-----------------------------------------------

                 Key: HADOOP-7258
                 URL: https://issues.apache.org/jira/browse/HADOOP-7258
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Owen O'Malley
            Assignee: Owen O'Malley
             Fix For: 0.20.203.0
         Attachments: gzip-decomp.patch

In HADOOP-6315, the gzip codec was changed to return a null codec with the intent to disallow pooling of the decompressors. Rather than break the interface, we can use an annotation to achieve the goal.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14400-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 19:00:46 2011
Return-Path: <common-issues-return-14400-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0256429EB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 19:00:45 +0000 (UTC)
Received: (qmail 84463 invoked by uid 500); 3 May 2011 19:00:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84367 invoked by uid 500); 3 May 2011 19:00:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84316 invoked by uid 99); 3 May 2011 19:00:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 19:00:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 19:00:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2C880BF267
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 19:00:03 +0000 (UTC)
Date: Tue, 3 May 2011 19:00:03 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <734410995.19461.1304449203179.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <64692328.19459.1304449203116.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7258) Gzip codec should not return null
 decompressors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7258:
----------------------------------

    Attachment: gzip-decomp.patch

This patch fixes the problem and the testcases.

> Gzip codec should not return null decompressors
> -----------------------------------------------
>
>                 Key: HADOOP-7258
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7258
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.203.0
>
>         Attachments: gzip-decomp.patch
>
>
> In HADOOP-6315, the gzip codec was changed to return a null codec with the intent to disallow pooling of the decompressors. Rather than break the interface, we can use an annotation to achieve the goal.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14401-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 19:02:42 2011
Return-Path: <common-issues-return-14401-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7D6AD57B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 19:02:42 +0000 (UTC)
Received: (qmail 87828 invoked by uid 500); 3 May 2011 19:02:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87793 invoked by uid 500); 3 May 2011 19:02:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87783 invoked by uid 99); 3 May 2011 19:02:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 19:02:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 19:02:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3F631BF344
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 19:02:03 +0000 (UTC)
Date: Tue, 3 May 2011 19:02:03 +0000 (UTC)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1160942583.19468.1304449323256.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <64692328.19459.1304449203116.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7258) Gzip codec should not return null
 decompressors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028369#comment-13028369 ] 

Chris Douglas commented on HADOOP-7258:
---------------------------------------

+1 Good catch

> Gzip codec should not return null decompressors
> -----------------------------------------------
>
>                 Key: HADOOP-7258
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7258
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.203.0
>
>         Attachments: gzip-decomp.patch
>
>
> In HADOOP-6315, the gzip codec was changed to return a null codec with the intent to disallow pooling of the decompressors. Rather than break the interface, we can use an annotation to achieve the goal.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14402-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 21:14:47 2011
Return-Path: <common-issues-return-14402-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EDC932299
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 21:14:46 +0000 (UTC)
Received: (qmail 19156 invoked by uid 500); 3 May 2011 21:14:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19054 invoked by uid 500); 3 May 2011 21:14:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19036 invoked by uid 99); 3 May 2011 21:14:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 21:14:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 21:14:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 72C0EBF5F3
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 21:14:03 +0000 (UTC)
Date: Tue, 3 May 2011 21:14:03 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <902118445.19780.1304457243466.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <818390768.11760.1299787865337.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7179) Improve HDFS startup scripts
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028430#comment-13028430 ] 

Suresh Srinivas commented on HADOOP-7179:
-----------------------------------------

+1 for the change.

> Improve HDFS startup scripts
> ----------------------------
>
>                 Key: HADOOP-7179
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7179
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Tanping Wang
>            Priority: Minor
>         Attachments: HADOOP-7179.patch
>
>
> Startup scripts should start namenodes, secondary namenodes and datanodes on hosts returned by getConfig (new feature of HDFS Federation).  This jira is Hdfs counterpart of HDFS-1703.  This patch makes changes to hadoop-config.sh and slaves.sh where they belong to COMMON project.  And HDFS-1703 modifies scripts, hdfs, start-hdfs.sh and stop-hdfs.sh where they go to HDFS project.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14403-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 21:21:44 2011
Return-Path: <common-issues-return-14403-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E45FB2863
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 21:21:43 +0000 (UTC)
Received: (qmail 25905 invoked by uid 500); 3 May 2011 21:21:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25833 invoked by uid 500); 3 May 2011 21:21:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25573 invoked by uid 99); 3 May 2011 21:21:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 21:21:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 21:21:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 78F4BBF817
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 21:21:03 +0000 (UTC)
Date: Tue, 3 May 2011 21:21:03 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1586276346.19791.1304457663492.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <818390768.11760.1299787865337.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7179) Improve HDFS startup scripts
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas resolved HADOOP-7179.
-------------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]

I committed the patch.

> Improve HDFS startup scripts
> ----------------------------
>
>                 Key: HADOOP-7179
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7179
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Tanping Wang
>            Priority: Minor
>         Attachments: HADOOP-7179.patch
>
>
> Startup scripts should start namenodes, secondary namenodes and datanodes on hosts returned by getConfig (new feature of HDFS Federation).  This jira is Hdfs counterpart of HDFS-1703.  This patch makes changes to hadoop-config.sh and slaves.sh where they belong to COMMON project.  And HDFS-1703 modifies scripts, hdfs, start-hdfs.sh and stop-hdfs.sh where they go to HDFS project.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14404-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 21:33:43 2011
Return-Path: <common-issues-return-14404-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 218982D1E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 21:33:43 +0000 (UTC)
Received: (qmail 46909 invoked by uid 500); 3 May 2011 21:33:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46814 invoked by uid 500); 3 May 2011 21:33:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46806 invoked by uid 99); 3 May 2011 21:33:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 21:33:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 21:33:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C85B2BFE07
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 21:33:03 +0000 (UTC)
Date: Tue, 3 May 2011 21:33:03 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <805087598.19826.1304458383817.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <818390768.11760.1299787865337.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7179) Improve HDFS startup scripts
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028444#comment-13028444 ] 

Hudson commented on HADOOP-7179:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #569 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/569/])
    HADOOP-7179. Federation: Improve HDFS startup scripts. Contributed by Erik Steffl and Tanping Wang.


> Improve HDFS startup scripts
> ----------------------------
>
>                 Key: HADOOP-7179
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7179
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Tanping Wang
>            Priority: Minor
>         Attachments: HADOOP-7179.patch
>
>
> Startup scripts should start namenodes, secondary namenodes and datanodes on hosts returned by getConfig (new feature of HDFS Federation).  This jira is Hdfs counterpart of HDFS-1703.  This patch makes changes to hadoop-config.sh and slaves.sh where they belong to COMMON project.  And HDFS-1703 modifies scripts, hdfs, start-hdfs.sh and stop-hdfs.sh where they go to HDFS project.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14405-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 21:51:44 2011
Return-Path: <common-issues-return-14405-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BE4F929D7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 21:51:44 +0000 (UTC)
Received: (qmail 65208 invoked by uid 500); 3 May 2011 21:51:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65176 invoked by uid 500); 3 May 2011 21:51:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65166 invoked by uid 99); 3 May 2011 21:51:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 21:51:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 21:51:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4C0FAC043A
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 21:51:03 +0000 (UTC)
Date: Tue, 3 May 2011 21:51:03 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <324434413.19867.1304459463308.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028456#comment-13028456 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7227:
------------------------------------------------

+1

Minor: The following codes repeat twice.  It may deserve putting them in a utility method.  Otherwise, we won't miss the other "versionID" previously.
{code}
+    try {
+      Field versionField = method.getDeclaringClass().getField("versionID");
+      versionField.setAccessible(true);
+      clientVersion = versionField.getLong(method.getDeclaringClass());
+    } catch (NoSuchFieldException ex) {
+      throw new RuntimeException(ex);
+    } catch (IllegalAccessException ex) {
+      throw new RuntimeException(ex);
+    }
{code}


> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch, HADOOP-7227.5.patch, HADOOP-7227.6.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14406-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 22:13:43 2011
Return-Path: <common-issues-return-14406-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EA6AF2AFC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 22:13:42 +0000 (UTC)
Received: (qmail 96610 invoked by uid 500); 3 May 2011 22:13:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96576 invoked by uid 500); 3 May 2011 22:13:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96568 invoked by uid 99); 3 May 2011 22:13:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 22:13:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 22:13:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 45A72C0D00
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 22:13:03 +0000 (UTC)
Date: Tue, 3 May 2011 22:13:03 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2038968232.19928.1304460783281.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028467#comment-13028467 ] 

Jitendra Nath Pandey commented on HADOOP-7227:
----------------------------------------------

> The following codes repeat twice. It may deserve putting them in a utility method.
  That is correct! I would like to add a RPCUtil class and add a method there to get the version. I would prefer to do it in a separate patch as a refactor. Is that ok?

> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch, HADOOP-7227.5.patch, HADOOP-7227.6.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14407-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 22:15:45 2011
Return-Path: <common-issues-return-14407-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0FDC62E13
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 22:15:45 +0000 (UTC)
Received: (qmail 99398 invoked by uid 500); 3 May 2011 22:15:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99365 invoked by uid 500); 3 May 2011 22:15:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99357 invoked by uid 99); 3 May 2011 22:15:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 22:15:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 22:15:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AEDACC0DB7
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 22:15:03 +0000 (UTC)
Date: Tue, 3 May 2011 22:15:03 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <266780624.19944.1304460903713.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028469#comment-13028469 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7227:
------------------------------------------------

Sure.  Let's do it separately.

> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch, HADOOP-7227.5.patch, HADOOP-7227.6.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14408-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 22:15:47 2011
Return-Path: <common-issues-return-14408-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7F25D2E21
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 22:15:45 +0000 (UTC)
Received: (qmail 99633 invoked by uid 500); 3 May 2011 22:15:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99596 invoked by uid 500); 3 May 2011 22:15:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99588 invoked by uid 99); 3 May 2011 22:15:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 22:15:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 22:15:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 851D4C0DB1
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 22:15:03 +0000 (UTC)
Date: Tue, 3 May 2011 22:15:03 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1999687775.19938.1304460903542.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <836774115.14005.1304224263168.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7254) Programmatically start  processes
 with JMX port open
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tanping Wang updated HADOOP-7254:
---------------------------------

    Assignee: Tanping Wang

> Programmatically start  processes with JMX port open 
> -----------------------------------------------------
>
>                 Key: HADOOP-7254
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7254
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: scripts
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7254.patch
>
>
> We propose a programmatic way to start processes with JMX enabled.  This is the counter part of HDFS-1874.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14409-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 22:19:44 2011
Return-Path: <common-issues-return-14409-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A64EE2F97
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 22:19:44 +0000 (UTC)
Received: (qmail 1715 invoked by uid 500); 3 May 2011 22:19:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1684 invoked by uid 500); 3 May 2011 22:19:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1676 invoked by uid 99); 3 May 2011 22:19:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 22:19:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 22:19:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 411B6C0EFC
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 22:19:03 +0000 (UTC)
Date: Tue, 3 May 2011 22:19:03 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <674568242.19953.1304461143263.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <818390768.11760.1299787865337.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7179) Improve HDFS startup scripts
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tanping Wang updated HADOOP-7179:
---------------------------------

    Assignee: Tanping Wang

> Improve HDFS startup scripts
> ----------------------------
>
>                 Key: HADOOP-7179
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7179
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>            Priority: Minor
>         Attachments: HADOOP-7179.patch
>
>
> Startup scripts should start namenodes, secondary namenodes and datanodes on hosts returned by getConfig (new feature of HDFS Federation).  This jira is Hdfs counterpart of HDFS-1703.  This patch makes changes to hadoop-config.sh and slaves.sh where they belong to COMMON project.  And HDFS-1703 modifies scripts, hdfs, start-hdfs.sh and stop-hdfs.sh where they go to HDFS project.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14410-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May  3 22:33:45 2011
Return-Path: <common-issues-return-14410-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 819BB2686
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue,  3 May 2011 22:33:45 +0000 (UTC)
Received: (qmail 25180 invoked by uid 500); 3 May 2011 22:33:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25153 invoked by uid 500); 3 May 2011 22:33:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25145 invoked by uid 99); 3 May 2011 22:33:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 22:33:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 May 2011 22:33:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1CFE3C0413
	for <common-issues@hadoop.apache.org>; Tue,  3 May 2011 22:33:04 +0000 (UTC)
Date: Tue, 3 May 2011 22:33:04 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <440767403.20015.1304461984115.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028482#comment-13028482 ] 

Hudson commented on HADOOP-7227:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #570 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/570/])
    HADOOP-7227. Remove protocol version check at proxy creation in Hadoop RPC. Contributed by jitendra.


> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch, HADOOP-7227.5.patch, HADOOP-7227.6.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14411-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 05:22:49 2011
Return-Path: <common-issues-return-14411-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7974B3707
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 05:22:49 +0000 (UTC)
Received: (qmail 88247 invoked by uid 500); 4 May 2011 05:22:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88101 invoked by uid 500); 4 May 2011 05:22:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88078 invoked by uid 99); 4 May 2011 05:22:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 05:22:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 05:22:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1F55CC04F1
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 05:22:03 +0000 (UTC)
Date: Wed, 4 May 2011 05:22:03 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1233194207.20852.1304486523125.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <64692328.19459.1304449203116.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7258) Gzip codec should not return null
 decompressors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7258:
----------------------------------

    Attachment: fix.patch

I messed up the previous patch by putting the annotation on the constructor rather than the class. I fix the definition of the annotation to only apply to types and put it in the right place.

> Gzip codec should not return null decompressors
> -----------------------------------------------
>
>                 Key: HADOOP-7258
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7258
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.203.0
>
>         Attachments: fix.patch, gzip-decomp.patch
>
>
> In HADOOP-6315, the gzip codec was changed to return a null codec with the intent to disallow pooling of the decompressors. Rather than break the interface, we can use an annotation to achieve the goal.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14412-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 05:40:46 2011
Return-Path: <common-issues-return-14412-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E97033D0E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 05:40:45 +0000 (UTC)
Received: (qmail 96445 invoked by uid 500); 4 May 2011 05:40:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96242 invoked by uid 500); 4 May 2011 05:40:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96230 invoked by uid 99); 4 May 2011 05:40:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 05:40:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 05:40:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 793A1C08E0
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 05:40:03 +0000 (UTC)
Date: Wed, 4 May 2011 05:40:03 +0000 (UTC)
From: "Soren Macbeth (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <990187855.20862.1304487603493.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6945) Inclusion of Old Jackson-JSON
 Breaks tasks using Avro (or any task depending on Jackson JSON)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028616#comment-13028616 ] 

Soren Macbeth commented on HADOOP-6945:
---------------------------------------

+1 It's a pain in the butt to have to strip this out of all the nodes in our cluster. Please either remove it or bump the version to a much more recent one. 

> Inclusion of Old Jackson-JSON Breaks tasks using Avro (or any task depending on Jackson JSON)
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6945
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6945
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Greg Wittel
>
> HADOOP-6184 added the ability to serialize the Configuration to JSON.  However its inclusion of jackson-1.0.1 means that any map-reduce task that depends (directly or indirectly) on Jackson, can only use Jackson 1.0.1 APIs.  The 100% fix is to give the task's classpath priority over the core/common classpath (MAPREDUCE-1700/MAPREDUCE-1938) so that a job can use newer revisions of the library.
> For a nearer term fix, is it possible to upgrade the included Jackson library to a recent version (1.5.x+)?  The APIs are backwards compatible.  alernatively, its possible to eliminate the use of Jackson for this minor feature as the Configuration object could be serialized to JSON without a 3rd party library.
> An ancilliary issue is that the jackson.version referenced in ivy.xml is never specified in the source tree (ivy/libraries.properties).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14413-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 05:42:43 2011
Return-Path: <common-issues-return-14413-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ADA813D21
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 05:42:43 +0000 (UTC)
Received: (qmail 96921 invoked by uid 500); 4 May 2011 05:42:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96764 invoked by uid 500); 4 May 2011 05:42:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96743 invoked by uid 99); 4 May 2011 05:42:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 05:42:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 05:42:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 443C0C09B2
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 05:42:03 +0000 (UTC)
Date: Wed, 4 May 2011 05:42:03 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1878769507.20867.1304487723276.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Closed] (HADOOP-4858) to add appropriate reference to the
 dependent library files in the chukwa/build.xml file
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-4858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley closed HADOOP-4858.
---------------------------------


> to add appropriate reference to the dependent library files in the chukwa/build.xml file 
> -----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4858
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4858
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Giridharan Kesavan
>
> While going through the chukwa/build.xml 's package-hadoop target found that we are trying to copy jsp-api.jar from the hadoop/lib dir but actually the jsp-api. jar resides in chukwa/lib directory. 
> For more details see comment 
> https://issues.apache.org/jira/browse/HADOOP-4709?focusedCommentId=12654489#action_12654489
> -Giri

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14414-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 05:42:45 2011
Return-Path: <common-issues-return-14414-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5CAC53D4B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 05:42:45 +0000 (UTC)
Received: (qmail 98026 invoked by uid 500); 4 May 2011 05:42:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97971 invoked by uid 500); 4 May 2011 05:42:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97737 invoked by uid 99); 4 May 2011 05:42:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 05:42:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 05:42:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 34ABEC09AE
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 05:42:03 +0000 (UTC)
Date: Wed, 4 May 2011 05:42:03 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <549599084.20865.1304487723212.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-4858) to add appropriate reference to the
 dependent library files in the chukwa/build.xml file
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley resolved HADOOP-4858.
-----------------------------------

    Resolution: Not A Problem

Chukwa moved out long ago.

> to add appropriate reference to the dependent library files in the chukwa/build.xml file 
> -----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4858
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4858
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Giridharan Kesavan
>
> While going through the chukwa/build.xml 's package-hadoop target found that we are trying to copy jsp-api.jar from the hadoop/lib dir but actually the jsp-api. jar resides in chukwa/lib directory. 
> For more details see comment 
> https://issues.apache.org/jira/browse/HADOOP-4709?focusedCommentId=12654489#action_12654489
> -Giri

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14415-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 06:12:45 2011
Return-Path: <common-issues-return-14415-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 354EB39B4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 06:12:45 +0000 (UTC)
Received: (qmail 36373 invoked by uid 500); 4 May 2011 06:12:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36235 invoked by uid 500); 4 May 2011 06:12:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36210 invoked by uid 99); 4 May 2011 06:12:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 06:12:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 06:12:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 380B8C021E
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 06:12:03 +0000 (UTC)
Date: Wed, 4 May 2011 06:12:03 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <360710663.20918.1304489523226.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7259) contrib modules should include
 build.properties from parent.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

contrib modules should include build.properties from parent.
------------------------------------------------------------

                 Key: HADOOP-7259
                 URL: https://issues.apache.org/jira/browse/HADOOP-7259
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Owen O'Malley
            Assignee: Owen O'Malley
             Fix For: 0.20.203.0


Current build.properties in the hadoop root directory is not included by the contrib modules.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14416-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 06:16:47 2011
Return-Path: <common-issues-return-14416-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9F4AC3FC3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 06:16:45 +0000 (UTC)
Received: (qmail 47418 invoked by uid 500); 4 May 2011 06:16:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47302 invoked by uid 500); 4 May 2011 06:16:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46578 invoked by uid 99); 4 May 2011 06:16:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 06:16:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 06:16:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1C5B5C0399
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 06:16:03 +0000 (UTC)
Date: Wed, 4 May 2011 06:16:03 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1291166686.20963.1304489763112.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <360710663.20918.1304489523226.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7259) contrib modules should include
 build.properties from parent.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7259:
----------------------------------

    Attachment: contrib.patch

The contrib modules should pick up the build.properties from the enclosing hadoop build.

> contrib modules should include build.properties from parent.
> ------------------------------------------------------------
>
>                 Key: HADOOP-7259
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7259
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.203.0
>
>         Attachments: contrib.patch
>
>
> Current build.properties in the hadoop root directory is not included by the contrib modules.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14417-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 06:30:46 2011
Return-Path: <common-issues-return-14417-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3F7483316
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 06:30:46 +0000 (UTC)
Received: (qmail 58887 invoked by uid 500); 4 May 2011 06:30:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58849 invoked by uid 500); 4 May 2011 06:30:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58673 invoked by uid 99); 4 May 2011 06:30:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 06:30:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 06:30:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7E796C090A
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 06:30:03 +0000 (UTC)
Date: Wed, 4 May 2011 06:30:03 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1696113784.21006.1304490603514.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <360710663.20918.1304489523226.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7259) contrib modules should include
 build.properties from parent.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7259:
----------------------------------

    Component/s: build

> contrib modules should include build.properties from parent.
> ------------------------------------------------------------
>
>                 Key: HADOOP-7259
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7259
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.203.0
>
>         Attachments: contrib.patch
>
>
> Current build.properties in the hadoop root directory is not included by the contrib modules.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14418-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 11:14:44 2011
Return-Path: <common-issues-return-14418-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A7D7B379B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 11:14:44 +0000 (UTC)
Received: (qmail 99403 invoked by uid 500); 4 May 2011 11:14:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99373 invoked by uid 500); 4 May 2011 11:14:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99353 invoked by uid 99); 4 May 2011 11:14:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 11:14:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 11:14:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 63658C16EE
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 11:14:03 +0000 (UTC)
Date: Wed, 4 May 2011 11:14:03 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1985359307.21407.1304507643404.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028710#comment-13028710 ] 

Hudson commented on HADOOP-7227:
--------------------------------

Integrated in Hadoop-Common-trunk #678 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/678/])
    HADOOP-7227. Remove protocol version check at proxy creation in Hadoop RPC. Contributed by jitendra.


> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch, HADOOP-7227.5.patch, HADOOP-7227.6.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14419-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 11:14:44 2011
Return-Path: <common-issues-return-14419-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C0D2137A0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 11:14:44 +0000 (UTC)
Received: (qmail 99491 invoked by uid 500); 4 May 2011 11:14:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99376 invoked by uid 500); 4 May 2011 11:14:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99352 invoked by uid 99); 4 May 2011 11:14:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 11:14:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 11:14:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 32ADEC16E8
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 11:14:03 +0000 (UTC)
Date: Wed, 4 May 2011 11:14:03 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <415745918.21401.1304507643204.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <818390768.11760.1299787865337.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7179) Improve HDFS startup scripts
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028709#comment-13028709 ] 

Hudson commented on HADOOP-7179:
--------------------------------

Integrated in Hadoop-Common-trunk #678 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/678/])
    HADOOP-7179. Federation: Improve HDFS startup scripts. Contributed by Erik Steffl and Tanping Wang.


> Improve HDFS startup scripts
> ----------------------------
>
>                 Key: HADOOP-7179
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7179
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>            Priority: Minor
>         Attachments: HADOOP-7179.patch
>
>
> Startup scripts should start namenodes, secondary namenodes and datanodes on hosts returned by getConfig (new feature of HDFS Federation).  This jira is Hdfs counterpart of HDFS-1703.  This patch makes changes to hadoop-config.sh and slaves.sh where they belong to COMMON project.  And HDFS-1703 modifies scripts, hdfs, start-hdfs.sh and stop-hdfs.sh where they go to HDFS project.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14420-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 13:56:45 2011
Return-Path: <common-issues-return-14420-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4FCA332DE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 13:56:45 +0000 (UTC)
Received: (qmail 74602 invoked by uid 500); 4 May 2011 13:56:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74491 invoked by uid 500); 4 May 2011 13:56:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74483 invoked by uid 99); 4 May 2011 13:56:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 13:56:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 13:56:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 83F9CC1B52
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 13:56:03 +0000 (UTC)
Date: Wed, 4 May 2011 13:56:03 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <294652392.21656.1304517363536.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1991821639.8902.1299699839419.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7176) Redesign FsShell
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028751#comment-13028751 ] 

Daryn Sharp commented on HADOOP-7176:
-------------------------------------

This bug is implicitly fixing bugs in all the commands regarding handling of globs and exit codes of 0 when multiple args and/or globs are supplied and have errors.  If anyone has available time, I'd like to please request reviews for the linked bugs.  Thanks in advance.

> Redesign FsShell
> ----------------
>
>                 Key: HADOOP-7176
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7176
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.22.0
>
>
> The FsShell commands are very tightly coupled together which makes it necessarily hard to write new commands.  There is a lot of redundancy between the commands, inconsistencies in handling of paths, handling of errors, and correct return of exit codes.  The FsShell commands should be subclasses of the fs.shell.Command class which is already being used by the -count command, and is used by other commands like dfsadmin.
> This will serve as an umbrella bug to track the changes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14421-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 14:12:43 2011
Return-Path: <common-issues-return-14421-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CAC623B7B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 14:12:43 +0000 (UTC)
Received: (qmail 8146 invoked by uid 500); 4 May 2011 14:12:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8102 invoked by uid 500); 4 May 2011 14:12:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7886 invoked by uid 99); 4 May 2011 14:12:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 14:12:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 14:12:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 36468C11E9
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 14:12:03 +0000 (UTC)
Date: Wed, 4 May 2011 14:12:03 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1400941320.21693.1304518323219.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6945) Inclusion of Old Jackson-JSON
 Breaks tasks using Avro (or any task depending on Jackson JSON)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028755#comment-13028755 ] 

Steve Loughran commented on HADOOP-6945:
----------------------------------------

given that Jackson is only used to serialize the config, and also that generating valid JSON is fairly straightforward, why not get rid of the jackson dependency at all and just have a json serializer class? jackson (or something like json-lib or gson) could just be used to validate the content in the unit tests.

> Inclusion of Old Jackson-JSON Breaks tasks using Avro (or any task depending on Jackson JSON)
> ---------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6945
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6945
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Greg Wittel
>
> HADOOP-6184 added the ability to serialize the Configuration to JSON.  However its inclusion of jackson-1.0.1 means that any map-reduce task that depends (directly or indirectly) on Jackson, can only use Jackson 1.0.1 APIs.  The 100% fix is to give the task's classpath priority over the core/common classpath (MAPREDUCE-1700/MAPREDUCE-1938) so that a job can use newer revisions of the library.
> For a nearer term fix, is it possible to upgrade the included Jackson library to a recent version (1.5.x+)?  The APIs are backwards compatible.  alernatively, its possible to eliminate the use of Jackson for this minor feature as the Configuration object could be serialized to JSON without a 3rd party library.
> An ancilliary issue is that the jackson.version referenced in ivy.xml is never specified in the source tree (ivy/libraries.properties).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14422-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 16:00:45 2011
Return-Path: <common-issues-return-14422-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0158F3581
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 16:00:45 +0000 (UTC)
Received: (qmail 34734 invoked by uid 500); 4 May 2011 16:00:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34697 invoked by uid 500); 4 May 2011 16:00:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34689 invoked by uid 99); 4 May 2011 16:00:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 16:00:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 16:00:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 38F34C193F
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 16:00:03 +0000 (UTC)
Date: Wed, 4 May 2011 16:00:03 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1876422467.21868.1304524803229.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1912834846.74827.1303423325837.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7236) Refactor FsShell's mkdir
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028794#comment-13028794 ] 

Matt Foley commented on HADOOP-7236:
------------------------------------

It looks generally good, and this refactoring is very good for the FsShell.

However, I think John George was on the right track when he questioned adding mkdirs(), etc., methods to the PathData class.  We seem to already be creating FileContext as a mirror of all FS methods, we shouldn't do the same with PathData.  To me it seems obvious that the invocation of mkdirs(), rightly removed from FSCommand, should be located in the Mkdir class.

Daryl and I talked about various ways to try to "simplify the switch to FileContext".  We concluded (or at least I did -- sorry if I'm putting words in your mouth :-) ) that we were going through contortions to avoid changing twenty single lines in twenty different files at some point in the future; i.e., converting the invocations of FileSystem methods to FileContext methods in each of the implementations of the various FSCommands.  While it would be nice to avoid that future editing, it isn't worth complicating the design, in my opinion.  And isn't it kind of the point of all this that each command (or related subset of commands) can become a separate subclass of FSCommand?  Thanks.

> Refactor FsShell's mkdir
> ------------------------
>
>                 Key: HADOOP-7236
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7236
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7236-2.patch, HADOOP-7236.patch
>
>
> Need to refactor tail to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14423-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 16:18:44 2011
Return-Path: <common-issues-return-14423-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F37EE39AC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 16:18:43 +0000 (UTC)
Received: (qmail 64446 invoked by uid 500); 4 May 2011 16:18:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64392 invoked by uid 500); 4 May 2011 16:18:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64295 invoked by uid 99); 4 May 2011 16:18:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 16:18:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 16:18:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4079DC1FCB
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 16:18:04 +0000 (UTC)
Date: Wed, 4 May 2011 16:18:04 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2083247729.21941.1304525884260.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1912834846.74827.1303423325837.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7236) Refactor FsShell's mkdir
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7236:
--------------------------------

    Attachment: HADOOP-7236-3.patch

I agree, it felt a little awkward.  My motivation was to simplify the move to FileContext or the next new and improved api.  That said, you won me over, I'm fine with switching it back. 

> Refactor FsShell's mkdir
> ------------------------
>
>                 Key: HADOOP-7236
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7236
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7236-2.patch, HADOOP-7236-3.patch, HADOOP-7236.patch
>
>
> Need to refactor tail to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14424-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 16:40:42 2011
Return-Path: <common-issues-return-14424-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CE7823A1F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 16:40:42 +0000 (UTC)
Received: (qmail 11088 invoked by uid 500); 4 May 2011 16:40:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11050 invoked by uid 500); 4 May 2011 16:40:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11042 invoked by uid 99); 4 May 2011 16:40:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 16:40:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 16:40:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 293FFC16ED
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 16:40:03 +0000 (UTC)
Date: Wed, 4 May 2011 16:40:03 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <7782194.22007.1304527203165.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1912834846.74827.1303423325837.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7236) Refactor FsShell's mkdir
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028815#comment-13028815 ] 

Hadoop QA commented on HADOOP-7236:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478175/HADOOP-7236-3.patch
  against trunk revision 1099284.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 7 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/396//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/396//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/396//console

This message is automatically generated.

> Refactor FsShell's mkdir
> ------------------------
>
>                 Key: HADOOP-7236
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7236
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7236-2.patch, HADOOP-7236-3.patch, HADOOP-7236.patch
>
>
> Need to refactor tail to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14425-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 16:40:44 2011
Return-Path: <common-issues-return-14425-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CFFD93A2E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 16:40:44 +0000 (UTC)
Received: (qmail 11346 invoked by uid 500); 4 May 2011 16:40:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11313 invoked by uid 500); 4 May 2011 16:40:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11305 invoked by uid 99); 4 May 2011 16:40:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 16:40:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 16:40:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 397DCC16EF
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 16:40:03 +0000 (UTC)
Date: Wed, 4 May 2011 16:40:03 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1257697462.22009.1304527203232.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <720022722.11722.1304088663379.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7251) Refactor FsShell's getmerge
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028816#comment-13028816 ] 

Matt Foley commented on HADOOP-7251:
------------------------------------

I understand that the lifetime of the Merge object is only a single invocation, but I would prefer to have Merge.delimiter conditionally set to either null or "\n" at the decision point in processOptions(), in addition to being initialized to null by ctor.  It's just to remove concern about re-use, or in case the object ever changes to be re-used.

> Refactor FsShell's getmerge
> ---------------------------
>
>                 Key: HADOOP-7251
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7251
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7251.patch
>
>
> Need to refactor getmerge to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14426-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 16:58:46 2011
Return-Path: <common-issues-return-14426-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4F57D800
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 16:58:46 +0000 (UTC)
Received: (qmail 32528 invoked by uid 500); 4 May 2011 16:58:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32480 invoked by uid 500); 4 May 2011 16:58:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32388 invoked by uid 99); 4 May 2011 16:58:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 16:58:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 16:58:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 496ACC1C60
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 16:58:03 +0000 (UTC)
Date: Wed, 4 May 2011 16:58:03 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2041470620.22033.1304528283297.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <393899407.59444.1302827766144.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7227) Remove protocol version check at
 proxy creation in Hadoop RPC.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7227:
-----------------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

This has been committed!

> Remove protocol version check at proxy creation in Hadoop RPC.
> --------------------------------------------------------------
>
>                 Key: HADOOP-7227
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7227
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7227.2.patch, HADOOP-7227.3.patch, HADOOP-7227.4.patch, HADOOP-7227.5.patch, HADOOP-7227.6.patch
>
>
> Currently when a proxy is created for a protocol, there is a round trip of messages to check the protocol version. The protocol version is not checked in any subsequent rpc which could be a problem if the server restarts with a new protocol version. This issue and also the additional round-trip at proxy creation can be avoided if we add the protocol version in every rpc, and server checks the protocol version for every call.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14427-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 17:14:43 2011
Return-Path: <common-issues-return-14427-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2FD2A3245
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 17:14:43 +0000 (UTC)
Received: (qmail 65329 invoked by uid 500); 4 May 2011 17:14:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65300 invoked by uid 500); 4 May 2011 17:14:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65291 invoked by uid 99); 4 May 2011 17:14:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 17:14:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 17:14:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D00AAC1241
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 17:14:03 +0000 (UTC)
Date: Wed, 4 May 2011 17:14:03 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <386358151.22074.1304529243848.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <720022722.11722.1304088663379.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7251) Refactor FsShell's getmerge
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7251:
--------------------------------

    Attachment: HADOOP-7251-2.patch

Thanks for the comments!

The method parseOptions is now explicitly defining the delimiter to be null or newline in event of re-use -- which isn't currently possible, but is a good idea.

I'm not sure I understand the value of adding a ctor to explicitly set an instance var to null, when the instance var is already initialized to null?  Did I misunderstand?

> Refactor FsShell's getmerge
> ---------------------------
>
>                 Key: HADOOP-7251
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7251
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7251-2.patch, HADOOP-7251.patch
>
>
> Need to refactor getmerge to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14428-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 17:14:43 2011
Return-Path: <common-issues-return-14428-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D4A223257
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 17:14:43 +0000 (UTC)
Received: (qmail 65856 invoked by uid 500); 4 May 2011 17:14:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65819 invoked by uid 500); 4 May 2011 17:14:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65807 invoked by uid 99); 4 May 2011 17:14:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 17:14:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 17:14:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 510FAC124F
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 17:14:04 +0000 (UTC)
Date: Wed, 4 May 2011 17:14:04 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1911524473.22087.1304529244328.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028830#comment-13028830 ] 

Eric Yang commented on HADOOP-6255:
-----------------------------------

{quote} A valid point indeed. I think having packing outside of the project itself will address such concerns with ease, because packaging can be licensed differently than Hadoop proper. {quote}

There can be a LZ0 rpm as add-on, this integration project is extensible.  Hadoop works properly without LZ0.  Hence, the integration jira for Hadoop should focus on stuff on Apache only.  Commercial vendors are aggregating hadoop related technologies to make a proper distribution for commercial use.  People who wants effortless Hadoop setup can choose to pay commercial vendors for their effort and certification.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14429-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 17:38:42 2011
Return-Path: <common-issues-return-14429-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 929913206
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 17:38:42 +0000 (UTC)
Received: (qmail 7587 invoked by uid 500); 4 May 2011 17:38:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7554 invoked by uid 500); 4 May 2011 17:38:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7546 invoked by uid 99); 4 May 2011 17:38:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 17:38:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 17:38:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2D1F6C1C76
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 17:38:03 +0000 (UTC)
Date: Wed, 4 May 2011 17:38:03 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1706452162.22228.1304530683181.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <720022722.11722.1304088663379.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7251) Refactor FsShell's getmerge
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028844#comment-13028844 ] 

Hadoop QA commented on HADOOP-7251:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478182/HADOOP-7251-2.patch
  against trunk revision 1099284.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/397//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/397//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/397//console

This message is automatically generated.

> Refactor FsShell's getmerge
> ---------------------------
>
>                 Key: HADOOP-7251
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7251
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7251-2.patch, HADOOP-7251.patch
>
>
> Need to refactor getmerge to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14430-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 17:42:42 2011
Return-Path: <common-issues-return-14430-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B613B3312
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 17:42:42 +0000 (UTC)
Received: (qmail 16051 invoked by uid 500); 4 May 2011 17:42:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16019 invoked by uid 500); 4 May 2011 17:42:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16011 invoked by uid 99); 4 May 2011 17:42:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 17:42:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 17:42:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3CC63C1EAE
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 17:42:03 +0000 (UTC)
Date: Wed, 4 May 2011 17:42:03 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <59844505.22259.1304530923245.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <360710663.20918.1304489523226.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7259) contrib modules should include
 build.properties from parent.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028848#comment-13028848 ] 

Suresh Srinivas commented on HADOOP-7259:
-----------------------------------------

+1 for the patch.

> contrib modules should include build.properties from parent.
> ------------------------------------------------------------
>
>                 Key: HADOOP-7259
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7259
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.203.0
>
>         Attachments: contrib.patch
>
>
> Current build.properties in the hadoop root directory is not included by the contrib modules.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14431-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 19:33:42 2011
Return-Path: <common-issues-return-14431-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F0E523ADD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 19:33:42 +0000 (UTC)
Received: (qmail 82549 invoked by uid 500); 4 May 2011 19:33:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82495 invoked by uid 500); 4 May 2011 19:33:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82233 invoked by uid 99); 4 May 2011 19:33:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 19:33:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 19:33:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6620CC191C
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 19:33:03 +0000 (UTC)
Date: Wed, 4 May 2011 19:33:03 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1736574369.22575.1304537583414.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7257) A client side mount table to give
 per-application/per-job file system view
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7257:
---------------------------------

    Status: Open  (was: Patch Available)

> A client side mount table to give per-application/per-job file system view
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7257
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7257
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: ViewFs - javdoc2.pdf, ViewFsJavaDoc.pdf, viewFs2.patch, viewFs3.patch, viewfs1.patch
>
>
> This jira proposes a client side mount table to allow application-centric (or job-centric) filesystem views. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14432-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 19:41:45 2011
Return-Path: <common-issues-return-14432-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 78649304A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 19:41:45 +0000 (UTC)
Received: (qmail 92638 invoked by uid 500); 4 May 2011 19:41:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92606 invoked by uid 500); 4 May 2011 19:41:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92598 invoked by uid 99); 4 May 2011 19:41:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 19:41:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 19:41:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DF9D4C1D1A
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 19:41:03 +0000 (UTC)
Date: Wed, 4 May 2011 19:41:03 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <622590916.22625.1304538063912.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7257) A client side mount table to give
 per-application/per-job file system view
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7257:
---------------------------------

    Attachment: viewFs4.patch

Updated patch: Fixed findbug and compiler warnings

> A client side mount table to give per-application/per-job file system view
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7257
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7257
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: ViewFs - javdoc2.pdf, ViewFsJavaDoc.pdf, viewFs2.patch, viewFs3.patch, viewFs4.patch, viewfs1.patch
>
>
> This jira proposes a client side mount table to allow application-centric (or job-centric) filesystem views. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14433-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 19:41:46 2011
Return-Path: <common-issues-return-14433-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4713D3057
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 19:41:46 +0000 (UTC)
Received: (qmail 92887 invoked by uid 500); 4 May 2011 19:41:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92854 invoked by uid 500); 4 May 2011 19:41:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92846 invoked by uid 99); 4 May 2011 19:41:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 19:41:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 19:41:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 84CF2C1D2F
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 19:41:04 +0000 (UTC)
Date: Wed, 4 May 2011 19:41:04 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1664310542.22644.1304538064540.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7257) A client side mount table to give
 per-application/per-job file system view
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7257:
---------------------------------

    Status: Patch Available  (was: Open)

> A client side mount table to give per-application/per-job file system view
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7257
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7257
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: ViewFs - javdoc2.pdf, ViewFsJavaDoc.pdf, viewFs2.patch, viewFs3.patch, viewFs4.patch, viewfs1.patch
>
>
> This jira proposes a client side mount table to allow application-centric (or job-centric) filesystem views. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14434-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 20:00:45 2011
Return-Path: <common-issues-return-14434-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 178353E03
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 20:00:45 +0000 (UTC)
Received: (qmail 21001 invoked by uid 500); 4 May 2011 20:00:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20971 invoked by uid 500); 4 May 2011 20:00:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20958 invoked by uid 99); 4 May 2011 20:00:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 20:00:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 20:00:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2CD49C1324
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 20:00:03 +0000 (UTC)
Date: Wed, 4 May 2011 20:00:03 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1058229005.22687.1304539203180.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7257) A client side mount table to give
 per-application/per-job file system view
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028924#comment-13028924 ] 

Hadoop QA commented on HADOOP-7257:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478200/viewFs4.patch
  against trunk revision 1099284.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 54 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The applied patch generated 1072 javac compiler warnings (more than the trunk's current 1070 warnings).

    -1 findbugs.  The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/398//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/398//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/398//console

This message is automatically generated.

> A client side mount table to give per-application/per-job file system view
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7257
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7257
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: ViewFs - javdoc2.pdf, ViewFsJavaDoc.pdf, viewFs2.patch, viewFs3.patch, viewFs4.patch, viewfs1.patch
>
>
> This jira proposes a client side mount table to allow application-centric (or job-centric) filesystem views. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14435-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 21:27:44 2011
Return-Path: <common-issues-return-14435-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C22443521
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 21:27:44 +0000 (UTC)
Received: (qmail 35724 invoked by uid 500); 4 May 2011 21:27:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35691 invoked by uid 500); 4 May 2011 21:27:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35682 invoked by uid 99); 4 May 2011 21:27:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 21:27:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 21:27:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2780CC12DE
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 21:27:03 +0000 (UTC)
Date: Wed, 4 May 2011 21:27:03 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <174433715.22915.1304544423158.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <444776842.10957.1304053443189.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7250) Refactor FsShell's setrep
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028972#comment-13028972 ] 

Matt Foley commented on HADOOP-7250:
------------------------------------

+1  code reviewed

> Refactor FsShell's setrep
> -------------------------
>
>                 Key: HADOOP-7250
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7250
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7250.patch
>
>
> Need to refactor setrep to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14436-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 21:29:45 2011
Return-Path: <common-issues-return-14436-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 12C2635A2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 21:29:45 +0000 (UTC)
Received: (qmail 36977 invoked by uid 500); 4 May 2011 21:29:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36915 invoked by uid 500); 4 May 2011 21:29:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36907 invoked by uid 99); 4 May 2011 21:29:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 21:29:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 21:29:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A1438C13CC
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 21:29:03 +0000 (UTC)
Date: Wed, 4 May 2011 21:29:03 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1966277966.22932.1304544543657.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1912834846.74827.1303423325837.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7236) Refactor FsShell's mkdir
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028977#comment-13028977 ] 

Matt Foley commented on HADOOP-7236:
------------------------------------

+1 Thanks.

> Refactor FsShell's mkdir
> ------------------------
>
>                 Key: HADOOP-7236
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7236
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7236-2.patch, HADOOP-7236-3.patch, HADOOP-7236.patch
>
>
> Need to refactor tail to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14437-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 21:33:43 2011
Return-Path: <common-issues-return-14437-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 60A5C37BB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 21:33:43 +0000 (UTC)
Received: (qmail 41177 invoked by uid 500); 4 May 2011 21:33:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41148 invoked by uid 500); 4 May 2011 21:33:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41140 invoked by uid 99); 4 May 2011 21:33:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 21:33:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 21:33:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0B549C14F6
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 21:33:04 +0000 (UTC)
Date: Wed, 4 May 2011 21:33:04 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <998353546.22935.1304544784043.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <720022722.11722.1304088663379.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7251) Refactor FsShell's getmerge
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028978#comment-13028978 ] 

Matt Foley commented on HADOOP-7251:
------------------------------------

Never mind about the ctor, I wasn't expressing myself clearly.
This new patch changed exactly what I asked for.
+1.

> Refactor FsShell's getmerge
> ---------------------------
>
>                 Key: HADOOP-7251
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7251
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7251-2.patch, HADOOP-7251.patch
>
>
> Need to refactor getmerge to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14438-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 21:37:42 2011
Return-Path: <common-issues-return-14438-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C9F933B5D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 21:37:42 +0000 (UTC)
Received: (qmail 45665 invoked by uid 500); 4 May 2011 21:37:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45568 invoked by uid 500); 4 May 2011 21:37:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45518 invoked by uid 99); 4 May 2011 21:37:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 21:37:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 21:37:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 670A6C16F3
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 21:37:03 +0000 (UTC)
Date: Wed, 4 May 2011 21:37:03 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1319698382.22958.1304545023419.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1912834846.74827.1303423325837.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7236) Refactor FsShell's mkdir
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7236:
-------------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I have committed this.  Thanks, Daryn!

Also thanks John and Matt for reviewing it.

> Refactor FsShell's mkdir
> ------------------------
>
>                 Key: HADOOP-7236
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7236
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7236-2.patch, HADOOP-7236-3.patch, HADOOP-7236.patch
>
>
> Need to refactor tail to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14439-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 21:55:43 2011
Return-Path: <common-issues-return-14439-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A0CC13628
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 21:55:43 +0000 (UTC)
Received: (qmail 76163 invoked by uid 500); 4 May 2011 21:55:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76135 invoked by uid 500); 4 May 2011 21:55:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76127 invoked by uid 99); 4 May 2011 21:55:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 21:55:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 21:55:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 230BDC1D87
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 21:55:04 +0000 (UTC)
Date: Wed, 4 May 2011 21:55:04 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <784601101.23006.1304546104140.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1912834846.74827.1303423325837.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7236) Refactor FsShell's mkdir
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13028994#comment-13028994 ] 

Hudson commented on HADOOP-7236:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #571 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/571/])
    HADOOP-7236. Refactor the mkdir command to conform to new FsCommand class.  Contributed by Daryn Sharp


> Refactor FsShell's mkdir
> ------------------------
>
>                 Key: HADOOP-7236
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7236
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7236-2.patch, HADOOP-7236-3.patch, HADOOP-7236.patch
>
>
> Need to refactor tail to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14440-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 22:15:42 2011
Return-Path: <common-issues-return-14440-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 71DB53326
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 22:15:42 +0000 (UTC)
Received: (qmail 9370 invoked by uid 500); 4 May 2011 22:15:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9338 invoked by uid 500); 4 May 2011 22:15:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9330 invoked by uid 99); 4 May 2011 22:15:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:15:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:15:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 20EE0C13C7
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 22:15:03 +0000 (UTC)
Date: Wed, 4 May 2011 22:15:03 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1655011592.23071.1304547303131.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <444776842.10957.1304053443189.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7250) Refactor FsShell's setrep
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7250:
--------------------------------

    Attachment: HADOOP-7250-2.patch

Fix minor merge conflicts.

> Refactor FsShell's setrep
> -------------------------
>
>                 Key: HADOOP-7250
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7250
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7250-2.patch, HADOOP-7250.patch
>
>
> Need to refactor setrep to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14441-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 22:34:42 2011
Return-Path: <common-issues-return-14441-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9E00F3DBD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 22:34:42 +0000 (UTC)
Received: (qmail 43495 invoked by uid 500); 4 May 2011 22:34:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43457 invoked by uid 500); 4 May 2011 22:34:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43449 invoked by uid 99); 4 May 2011 22:34:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:34:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:34:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 31DB3C1A4E
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 22:34:03 +0000 (UTC)
Date: Wed, 4 May 2011 22:34:03 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <685568109.23104.1304548443200.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <444776842.10957.1304053443189.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7250) Refactor FsShell's setrep
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029016#comment-13029016 ] 

Hadoop QA commented on HADOOP-7250:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478221/HADOOP-7250-2.patch
  against trunk revision 1099612.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/399//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/399//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/399//console

This message is automatically generated.

> Refactor FsShell's setrep
> -------------------------
>
>                 Key: HADOOP-7250
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7250
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7250-2.patch, HADOOP-7250.patch
>
>
> Need to refactor setrep to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14442-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 22:52:42 2011
Return-Path: <common-issues-return-14442-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BFBAB3417
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 22:52:42 +0000 (UTC)
Received: (qmail 62482 invoked by uid 500); 4 May 2011 22:52:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62444 invoked by uid 500); 4 May 2011 22:52:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62435 invoked by uid 99); 4 May 2011 22:52:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:52:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:52:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 543F2C1F47
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 22:52:03 +0000 (UTC)
Date: Wed, 4 May 2011 22:52:03 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <399396215.23138.1304549523341.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1243264568.21599.1301502005765.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7214) Hadoop /usr/bin/groups equivalent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7214:
-----------------------------------

    Attachment: hadoop-7214.7.patch

Updated patch which switches designs to use a separate protocol.

Note that in this patch I renamed {{RefreshUserMappingsProtocol}} to {{UserGroupMappingProtocol}}. The only other changes in this file are bumping the protocol version ID and adding the {{getGroupsForUser}} method.

> Hadoop /usr/bin/groups equivalent
> ---------------------------------
>
>                 Key: HADOOP-7214
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7214
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>         Attachments: hadoop-7214.0.txt, hadoop-7214.1.txt, hadoop-7214.2.txt, hadoop-7214.3.txt, hadoop-7214.4.txt, hadoop-7214.5.txt, hadoop-7214.6.txt, hadoop-7214.7.patch
>
>
> Since user -> groups resolution is done on the NN and JT machines, there should be a way for users to determine what groups they're a member of from the NN's and JT's perspective.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14443-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 22:56:42 2011
Return-Path: <common-issues-return-14443-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CC86F354B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 22:56:42 +0000 (UTC)
Received: (qmail 69569 invoked by uid 500); 4 May 2011 22:56:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69531 invoked by uid 500); 4 May 2011 22:56:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69516 invoked by uid 99); 4 May 2011 22:56:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:56:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 22:56:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 50E38C1118
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 22:56:03 +0000 (UTC)
Date: Wed, 4 May 2011 22:56:03 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <65067986.23161.1304549763328.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1243264568.21599.1301502005765.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7214) Hadoop /usr/bin/groups equivalent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7214:
-----------------------------------

    Status: Patch Available  (was: Open)

> Hadoop /usr/bin/groups equivalent
> ---------------------------------
>
>                 Key: HADOOP-7214
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7214
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>         Attachments: hadoop-7214.0.txt, hadoop-7214.1.txt, hadoop-7214.2.txt, hadoop-7214.3.txt, hadoop-7214.4.txt, hadoop-7214.5.txt, hadoop-7214.6.txt, hadoop-7214.7.patch
>
>
> Since user -> groups resolution is done on the NN and JT machines, there should be a way for users to determine what groups they're a member of from the NN's and JT's perspective.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14444-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 23:02:45 2011
Return-Path: <common-issues-return-14444-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 38BC33257
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 23:02:45 +0000 (UTC)
Received: (qmail 72976 invoked by uid 500); 4 May 2011 23:02:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72776 invoked by uid 500); 4 May 2011 23:02:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72768 invoked by uid 99); 4 May 2011 23:02:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 23:02:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 23:02:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5485EC1266
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 23:02:03 +0000 (UTC)
Date: Wed, 4 May 2011 23:02:03 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1881795690.23167.1304550123342.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <444776842.10957.1304053443189.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7250) Refactor FsShell's setrep
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7250:
-------------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I have committed this.  Thanks, Daryn!

Also thanks Matt for the review.

> Refactor FsShell's setrep
> -------------------------
>
>                 Key: HADOOP-7250
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7250
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7250-2.patch, HADOOP-7250.patch
>
>
> Need to refactor setrep to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14445-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 23:04:44 2011
Return-Path: <common-issues-return-14445-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DB4723AE1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 23:04:44 +0000 (UTC)
Received: (qmail 75362 invoked by uid 500); 4 May 2011 23:04:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75307 invoked by uid 500); 4 May 2011 23:04:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75299 invoked by uid 99); 4 May 2011 23:04:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 23:04:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 23:04:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 58A9CC133C
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 23:04:03 +0000 (UTC)
Date: Wed, 4 May 2011 23:04:03 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1881620011.23176.1304550243359.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <444776842.10957.1304053443189.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7250) Refactor FsShell's setrep
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029030#comment-13029030 ] 

Hudson commented on HADOOP-7250:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #572 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/572/])
    HADOOP-7250. Refactor the setrep command to conform to new FsCommand class.  Contributed by Daryn Sharp


> Refactor FsShell's setrep
> -------------------------
>
>                 Key: HADOOP-7250
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7250
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7250-2.patch, HADOOP-7250.patch
>
>
> Need to refactor setrep to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14446-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 23:14:45 2011
Return-Path: <common-issues-return-14446-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EE2CD3CB9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 23:14:44 +0000 (UTC)
Received: (qmail 89766 invoked by uid 500); 4 May 2011 23:14:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89733 invoked by uid 500); 4 May 2011 23:14:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89725 invoked by uid 99); 4 May 2011 23:14:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 23:14:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 23:14:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 47D24C1589
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 23:14:03 +0000 (UTC)
Date: Wed, 4 May 2011 23:14:03 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <662529156.23193.1304550843290.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1243264568.21599.1301502005765.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7214) Hadoop /usr/bin/groups equivalent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029035#comment-13029035 ] 

Hadoop QA commented on HADOOP-7214:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478225/hadoop-7214.7.patch
  against trunk revision 1099633.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The applied patch generated 1071 javac compiler warnings (more than the trunk's current 1070 warnings).

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    -1 release audit.  The applied patch generated 2 release audit warnings (more than the trunk's current 1 warnings).

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/400//testReport/
Release audit warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/400//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/400//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/400//console

This message is automatically generated.

> Hadoop /usr/bin/groups equivalent
> ---------------------------------
>
>                 Key: HADOOP-7214
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7214
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>         Attachments: hadoop-7214.0.txt, hadoop-7214.1.txt, hadoop-7214.2.txt, hadoop-7214.3.txt, hadoop-7214.4.txt, hadoop-7214.5.txt, hadoop-7214.6.txt, hadoop-7214.7.patch
>
>
> Since user -> groups resolution is done on the NN and JT machines, there should be a way for users to determine what groups they're a member of from the NN's and JT's perspective.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14447-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 23:16:44 2011
Return-Path: <common-issues-return-14447-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9C686325E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 23:16:44 +0000 (UTC)
Received: (qmail 92388 invoked by uid 500); 4 May 2011 23:16:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92356 invoked by uid 500); 4 May 2011 23:16:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92348 invoked by uid 99); 4 May 2011 23:16:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 23:16:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 23:16:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 35BC3C163E
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 23:16:03 +0000 (UTC)
Date: Wed, 4 May 2011 23:16:03 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1709984454.23203.1304550963217.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <651142754.74841.1303423565756.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7238) Refactor FsShell's cat & text
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029039#comment-13029039 ] 

Matt Foley commented on HADOOP-7238:
------------------------------------

I especially like the re-implementation of printToStdout() as IOUtils.copyBytes().

Didn't read all the nitty-gritty details of the implementation that DIDN'T change, I assume you transcribed them correctly.

+1 code review.

> Refactor FsShell's cat & text
> -----------------------------
>
>                 Key: HADOOP-7238
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7238
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7238.patch
>
>
> Need to refactor cat & text to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14448-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 23:40:42 2011
Return-Path: <common-issues-return-14448-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A392739C4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 23:40:42 +0000 (UTC)
Received: (qmail 17890 invoked by uid 500); 4 May 2011 23:40:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17856 invoked by uid 500); 4 May 2011 23:40:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17848 invoked by uid 99); 4 May 2011 23:40:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 23:40:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 23:40:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 35418C1C72
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 23:40:03 +0000 (UTC)
Date: Wed, 4 May 2011 23:40:03 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <861057822.23235.1304552403215.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1243264568.21599.1301502005765.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7214) Hadoop /usr/bin/groups equivalent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7214:
-----------------------------------

    Attachment: hadoop-7214.8.patch

Patch which adds a missing license header and fixes the single javac warning.

The tests for this feature are in MR and HDFS.

> Hadoop /usr/bin/groups equivalent
> ---------------------------------
>
>                 Key: HADOOP-7214
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7214
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>         Attachments: hadoop-7214.0.txt, hadoop-7214.1.txt, hadoop-7214.2.txt, hadoop-7214.3.txt, hadoop-7214.4.txt, hadoop-7214.5.txt, hadoop-7214.6.txt, hadoop-7214.7.patch, hadoop-7214.8.patch
>
>
> Since user -> groups resolution is done on the NN and JT machines, there should be a way for users to determine what groups they're a member of from the NN's and JT's perspective.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14449-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May  4 23:54:45 2011
Return-Path: <common-issues-return-14449-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0AF623F94
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed,  4 May 2011 23:54:45 +0000 (UTC)
Received: (qmail 28247 invoked by uid 500); 4 May 2011 23:54:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28113 invoked by uid 500); 4 May 2011 23:54:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28105 invoked by uid 99); 4 May 2011 23:54:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 23:54:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 May 2011 23:54:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2A54AC1FAA
	for <common-issues@hadoop.apache.org>; Wed,  4 May 2011 23:54:03 +0000 (UTC)
Date: Wed, 4 May 2011 23:54:03 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1264500345.23249.1304553243170.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1243264568.21599.1301502005765.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7214) Hadoop /usr/bin/groups equivalent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029055#comment-13029055 ] 

Hadoop QA commented on HADOOP-7214:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478229/hadoop-7214.8.patch
  against trunk revision 1099633.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/401//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/401//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/401//console

This message is automatically generated.

> Hadoop /usr/bin/groups equivalent
> ---------------------------------
>
>                 Key: HADOOP-7214
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7214
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>         Attachments: hadoop-7214.0.txt, hadoop-7214.1.txt, hadoop-7214.2.txt, hadoop-7214.3.txt, hadoop-7214.4.txt, hadoop-7214.5.txt, hadoop-7214.6.txt, hadoop-7214.7.patch, hadoop-7214.8.patch
>
>
> Since user -> groups resolution is done on the NN and JT machines, there should be a way for users to determine what groups they're a member of from the NN's and JT's perspective.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14450-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 00:57:45 2011
Return-Path: <common-issues-return-14450-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1F3893C92
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 00:57:45 +0000 (UTC)
Received: (qmail 90499 invoked by uid 500); 5 May 2011 00:57:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90412 invoked by uid 500); 5 May 2011 00:57:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90404 invoked by uid 99); 5 May 2011 00:57:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 00:57:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 00:57:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2519FC1E40
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 00:57:03 +0000 (UTC)
Date: Thu, 5 May 2011 00:57:03 +0000 (UTC)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <464525365.23323.1304557023148.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <304321177.482.1298765698936.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7157) Create build hosts configurations
 to prevent degradation of patch validation and snapshot build environment
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik reassigned HADOOP-7157:
------------------------------------------

    Assignee: Konstantin Boudnik

> Create build hosts configurations to prevent degradation of patch validation and snapshot build environment
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7157
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7157
>             Project: Hadoop Common
>          Issue Type: Improvement
>         Environment: Apache Hadoop build hosts
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>
> Build hosts are being re-jumped, in need for configuration updates, etc. It is hard to maintain the same configuration across a 10+ hosts. A specialized service such as Puppet can be used to maintain build machines software and configurations at a desired level.
> Such configs should be checked into SCM system along with the of sources code.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14451-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 01:03:45 2011
Return-Path: <common-issues-return-14451-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 55B0447B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 01:03:45 +0000 (UTC)
Received: (qmail 95957 invoked by uid 500); 5 May 2011 01:03:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95912 invoked by uid 500); 5 May 2011 01:03:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95904 invoked by uid 99); 5 May 2011 01:03:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 01:03:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 01:03:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 550ABC00DA
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 01:03:03 +0000 (UTC)
Date: Thu, 5 May 2011 01:03:03 +0000 (UTC)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <880764344.23365.1304557383344.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <304321177.482.1298765698936.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7157) Create build hosts configurations to
 prevent degradation of patch validation and snapshot build environment
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-7157:
---------------------------------------

    Attachment: HADOOP-7157.patch

Initial puppet recipe to keep the configuration of build system in check.

> Create build hosts configurations to prevent degradation of patch validation and snapshot build environment
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7157
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7157
>             Project: Hadoop Common
>          Issue Type: Improvement
>         Environment: Apache Hadoop build hosts
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-7157.patch
>
>
> Build hosts are being re-jumped, in need for configuration updates, etc. It is hard to maintain the same configuration across a 10+ hosts. A specialized service such as Puppet can be used to maintain build machines software and configurations at a desired level.
> Such configs should be checked into SCM system along with the of sources code.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14452-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 01:15:44 2011
Return-Path: <common-issues-return-14452-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D6169BDF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 01:15:44 +0000 (UTC)
Received: (qmail 9783 invoked by uid 500); 5 May 2011 01:15:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9745 invoked by uid 500); 5 May 2011 01:15:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9737 invoked by uid 99); 5 May 2011 01:15:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 01:15:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 01:15:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5EF3FC054F
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 01:15:03 +0000 (UTC)
Date: Thu, 5 May 2011 01:15:03 +0000 (UTC)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1625631234.23387.1304558103385.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <304321177.482.1298765698936.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7157) Create build hosts configurations
 to prevent degradation of patch validation and snapshot build environment
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029083#comment-13029083 ] 

Konstantin Boudnik commented on HADOOP-7157:
--------------------------------------------

The following steps are needed to bootstrap puppet and keeping build configuration in check:
 - download puppet [2.6.8|http://puppetlabs.com/downloads/puppet/puppet-2.6.8.tar.gz]. Unpack and install it by running puppet/install.rb
 - run recipe (with superuser privileges): 
{noformat}
puppet apply --debug src/build.env/default-env.puppet
{noformat}
Running this with {{--noop}} will effectively execute dry-run

I propose these steps are to be done say once a day on every build machine.
The set of packages/versions proposed is what we have on Apache Hadoop build machines at the moment.

> Create build hosts configurations to prevent degradation of patch validation and snapshot build environment
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7157
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7157
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>         Environment: Apache Hadoop build hosts
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-7157.patch
>
>
> Build hosts are being re-jumped, in need for configuration updates, etc. It is hard to maintain the same configuration across a 10+ hosts. A specialized service such as Puppet can be used to maintain build machines software and configurations at a desired level.
> Such configs should be checked into SCM system along with the of sources code.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14453-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 01:15:45 2011
Return-Path: <common-issues-return-14453-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 27F24BFB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 01:15:45 +0000 (UTC)
Received: (qmail 10006 invoked by uid 500); 5 May 2011 01:15:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9977 invoked by uid 500); 5 May 2011 01:15:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9969 invoked by uid 99); 5 May 2011 01:15:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 01:15:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 01:15:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9FACEC055C
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 01:15:03 +0000 (UTC)
Date: Thu, 5 May 2011 01:15:03 +0000 (UTC)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1013476244.23396.1304558103651.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <304321177.482.1298765698936.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7157) Create build hosts configurations to
 prevent degradation of patch validation and snapshot build environment
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-7157:
---------------------------------------

    Component/s: build

> Create build hosts configurations to prevent degradation of patch validation and snapshot build environment
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7157
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7157
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>         Environment: Apache Hadoop build hosts
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-7157.patch
>
>
> Build hosts are being re-jumped, in need for configuration updates, etc. It is hard to maintain the same configuration across a 10+ hosts. A specialized service such as Puppet can be used to maintain build machines software and configurations at a desired level.
> Such configs should be checked into SCM system along with the of sources code.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14454-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 01:37:44 2011
Return-Path: <common-issues-return-14454-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A36373E07
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 01:37:44 +0000 (UTC)
Received: (qmail 30987 invoked by uid 500); 5 May 2011 01:37:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30959 invoked by uid 500); 5 May 2011 01:37:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30950 invoked by uid 99); 5 May 2011 01:37:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 01:37:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 01:37:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2FD37C0A0F
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 01:37:03 +0000 (UTC)
Date: Thu, 5 May 2011 01:37:03 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <40024282.23421.1304559423192.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7257) A client side mount table to give
 per-application/per-job file system view
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7257:
---------------------------------

    Attachment: viewFs5.patch

> A client side mount table to give per-application/per-job file system view
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7257
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7257
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: ViewFs - javdoc2.pdf, ViewFsJavaDoc.pdf, viewFs2.patch, viewFs3.patch, viewFs4.patch, viewFs5.patch, viewfs1.patch
>
>
> This jira proposes a client side mount table to allow application-centric (or job-centric) filesystem views. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14455-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 01:39:42 2011
Return-Path: <common-issues-return-14455-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A5DFB3E55
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 01:39:42 +0000 (UTC)
Received: (qmail 32894 invoked by uid 500); 5 May 2011 01:39:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32860 invoked by uid 500); 5 May 2011 01:39:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32852 invoked by uid 99); 5 May 2011 01:39:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 01:39:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 01:39:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 39FB2C0ACB
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 01:39:03 +0000 (UTC)
Date: Thu, 5 May 2011 01:39:03 +0000 (UTC)
From: "Roman Shaposhnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1194699545.23443.1304559543234.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <304321177.482.1298765698936.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7157) Create build hosts configurations
 to prevent degradation of patch validation and snapshot build environment
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029090#comment-13029090 ] 

Roman Shaposhnik commented on HADOOP-7157:
------------------------------------------

In general -- a strong +1 on the approach. H/W tends to go down, being decommissioned and tinkered with so having a reliable way to re-deploy a particular build/test environment is extremely important. Puppet fits the bill perfectly.

Now, a couple of points on Puppet usage:
   1. Not sure you need precise version strings for the kinds of  packages that are specified (and not all of them are specified, actually)
   2. Minor nit: you can simplify an implementation by using an array syntax for resource instantiation
   3. What's your plan on applying proposed recipe? I think it would be nice to have a cron job that would check it out from the apache repo and apply

> Create build hosts configurations to prevent degradation of patch validation and snapshot build environment
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7157
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7157
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>         Environment: Apache Hadoop build hosts
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-7157.patch
>
>
> Build hosts are being re-jumped, in need for configuration updates, etc. It is hard to maintain the same configuration across a 10+ hosts. A specialized service such as Puppet can be used to maintain build machines software and configurations at a desired level.
> Such configs should be checked into SCM system along with the of sources code.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14456-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 01:54:45 2011
Return-Path: <common-issues-return-14456-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7FF703478
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 01:54:45 +0000 (UTC)
Received: (qmail 44756 invoked by uid 500); 5 May 2011 01:54:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44704 invoked by uid 500); 5 May 2011 01:54:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44696 invoked by uid 99); 5 May 2011 01:54:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 01:54:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 01:54:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id ED97EC1043
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 01:54:03 +0000 (UTC)
Date: Thu, 5 May 2011 01:54:03 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1638134187.23489.1304560443969.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7257) A client side mount table to give
 per-application/per-job file system view
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029099#comment-13029099 ] 

Hadoop QA commented on HADOOP-7257:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478238/viewFs5.patch
  against trunk revision 1099633.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 54 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/402//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/402//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/402//console

This message is automatically generated.

> A client side mount table to give per-application/per-job file system view
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7257
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7257
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: ViewFs - javdoc2.pdf, ViewFsJavaDoc.pdf, viewFs2.patch, viewFs3.patch, viewFs4.patch, viewFs5.patch, viewfs1.patch
>
>
> This jira proposes a client side mount table to allow application-centric (or job-centric) filesystem views. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14457-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 05:01:47 2011
Return-Path: <common-issues-return-14457-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 58EA82E54
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 05:01:47 +0000 (UTC)
Received: (qmail 57378 invoked by uid 500); 5 May 2011 05:01:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57291 invoked by uid 500); 5 May 2011 05:01:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57283 invoked by uid 99); 5 May 2011 05:01:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 05:01:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 05:01:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2079EC24E9
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 05:01:03 +0000 (UTC)
Date: Thu, 5 May 2011 05:01:03 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <443243169.23599.1304571663129.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1243264568.21599.1301502005765.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7214) Hadoop /usr/bin/groups equivalent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029125#comment-13029125 ] 

Todd Lipcon commented on HADOOP-7214:
-------------------------------------

- What's the point of the "helper method to get the current user" since it's just wrapping another method directly? Seems silly.
- We need to make sure we commit this at the same time as we commit the MR and HDFS changes, right? (ie this will break the build if we don't?)

> Hadoop /usr/bin/groups equivalent
> ---------------------------------
>
>                 Key: HADOOP-7214
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7214
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>         Attachments: hadoop-7214.0.txt, hadoop-7214.1.txt, hadoop-7214.2.txt, hadoop-7214.3.txt, hadoop-7214.4.txt, hadoop-7214.5.txt, hadoop-7214.6.txt, hadoop-7214.7.patch, hadoop-7214.8.patch
>
>
> Since user -> groups resolution is done on the NN and JT machines, there should be a way for users to determine what groups they're a member of from the NN's and JT's perspective.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14458-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 05:19:44 2011
Return-Path: <common-issues-return-14458-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4A0BC2054
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 05:19:44 +0000 (UTC)
Received: (qmail 65036 invoked by uid 500); 5 May 2011 05:19:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64821 invoked by uid 500); 5 May 2011 05:19:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64812 invoked by uid 99); 5 May 2011 05:19:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 05:19:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 05:19:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 24A22C27C7
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 05:19:03 +0000 (UTC)
Date: Thu, 5 May 2011 05:19:03 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1464984546.23624.1304572743146.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1243264568.21599.1301502005765.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7214) Hadoop /usr/bin/groups equivalent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029130#comment-13029130 ] 

Aaron T. Myers commented on HADOOP-7214:
----------------------------------------

bq. What's the point of the "helper method to get the current user" since it's just wrapping another method directly? Seems silly.

Happy to remove it. It is only a very slight convenience.

bq. We need to make sure we commit this at the same time as we commit the MR and HDFS changes, right? (ie this will break the build if we don't?)

Absolutely. Unless we do what I just suggested in MAPREDUCE-2473, i.e. don't modify the existing {{RefreshUserMappings}} interface, and instead make a separate interface just for fetching.

Let me know if that sounds good to you and I'll go ahead and make those changes.

> Hadoop /usr/bin/groups equivalent
> ---------------------------------
>
>                 Key: HADOOP-7214
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7214
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>         Attachments: hadoop-7214.0.txt, hadoop-7214.1.txt, hadoop-7214.2.txt, hadoop-7214.3.txt, hadoop-7214.4.txt, hadoop-7214.5.txt, hadoop-7214.6.txt, hadoop-7214.7.patch, hadoop-7214.8.patch
>
>
> Since user -> groups resolution is done on the NN and JT machines, there should be a way for users to determine what groups they're a member of from the NN's and JT's perspective.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14459-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 06:23:48 2011
Return-Path: <common-issues-return-14459-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A956821A1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 06:23:48 +0000 (UTC)
Received: (qmail 28483 invoked by uid 500); 5 May 2011 06:23:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28235 invoked by uid 500); 5 May 2011 06:23:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28201 invoked by uid 99); 5 May 2011 06:23:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 06:23:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 06:23:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5E07BC295F
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 06:23:03 +0000 (UTC)
Date: Thu, 5 May 2011 06:23:03 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1432096816.23785.1304576583381.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1243264568.21599.1301502005765.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7214) Hadoop /usr/bin/groups equivalent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7214:
-----------------------------------

    Attachment: hadoop-7214.9.patch

Updated patch addressing Todd's comments.

> Hadoop /usr/bin/groups equivalent
> ---------------------------------
>
>                 Key: HADOOP-7214
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7214
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>         Attachments: hadoop-7214.0.txt, hadoop-7214.1.txt, hadoop-7214.2.txt, hadoop-7214.3.txt, hadoop-7214.4.txt, hadoop-7214.5.txt, hadoop-7214.6.txt, hadoop-7214.7.patch, hadoop-7214.8.patch, hadoop-7214.9.patch
>
>
> Since user -> groups resolution is done on the NN and JT machines, there should be a way for users to determine what groups they're a member of from the NN's and JT's perspective.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14460-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 06:43:46 2011
Return-Path: <common-issues-return-14460-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A60652C28
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 06:43:46 +0000 (UTC)
Received: (qmail 61090 invoked by uid 500); 5 May 2011 06:43:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60974 invoked by uid 500); 5 May 2011 06:43:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60958 invoked by uid 99); 5 May 2011 06:43:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 06:43:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 06:43:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5D8FFC2FCA
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 06:43:03 +0000 (UTC)
Date: Thu, 5 May 2011 06:43:03 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <594914770.23848.1304577783378.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1243264568.21599.1301502005765.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7214) Hadoop /usr/bin/groups equivalent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029176#comment-13029176 ] 

Hadoop QA commented on HADOOP-7214:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478250/hadoop-7214.9.patch
  against trunk revision 1099633.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    -1 release audit.  The applied patch generated 2 release audit warnings (more than the trunk's current 1 warnings).

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/403//testReport/
Release audit warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/403//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/403//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/403//console

This message is automatically generated.

> Hadoop /usr/bin/groups equivalent
> ---------------------------------
>
>                 Key: HADOOP-7214
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7214
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>         Attachments: hadoop-7214.0.txt, hadoop-7214.1.txt, hadoop-7214.2.txt, hadoop-7214.3.txt, hadoop-7214.4.txt, hadoop-7214.5.txt, hadoop-7214.6.txt, hadoop-7214.7.patch, hadoop-7214.8.patch, hadoop-7214.9.patch
>
>
> Since user -> groups resolution is done on the NN and JT machines, there should be a way for users to determine what groups they're a member of from the NN's and JT's perspective.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14461-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 06:49:44 2011
Return-Path: <common-issues-return-14461-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4DA8821DA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 06:49:44 +0000 (UTC)
Received: (qmail 68563 invoked by uid 500); 5 May 2011 06:49:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68405 invoked by uid 500); 5 May 2011 06:49:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68367 invoked by uid 99); 5 May 2011 06:49:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 06:49:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 06:49:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4158DC22BB
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 06:49:03 +0000 (UTC)
Date: Thu, 5 May 2011 06:49:03 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <15136292.23879.1304578143264.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1243264568.21599.1301502005765.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7214) Hadoop /usr/bin/groups equivalent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7214:
-----------------------------------

    Attachment: hadoop-7214.10.patch

Attaching the patch to the right JIRA this time. :P

> Hadoop /usr/bin/groups equivalent
> ---------------------------------
>
>                 Key: HADOOP-7214
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7214
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>         Attachments: hadoop-7214.0.txt, hadoop-7214.1.txt, hadoop-7214.10.patch, hadoop-7214.2.txt, hadoop-7214.3.txt, hadoop-7214.4.txt, hadoop-7214.5.txt, hadoop-7214.6.txt, hadoop-7214.7.patch, hadoop-7214.8.patch, hadoop-7214.9.patch
>
>
> Since user -> groups resolution is done on the NN and JT machines, there should be a way for users to determine what groups they're a member of from the NN's and JT's perspective.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14462-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 07:03:49 2011
Return-Path: <common-issues-return-14462-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 31E652658
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 07:03:49 +0000 (UTC)
Received: (qmail 83862 invoked by uid 500); 5 May 2011 07:03:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83427 invoked by uid 500); 5 May 2011 07:03:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83352 invoked by uid 99); 5 May 2011 07:03:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 07:03:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 07:03:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 53DF7C27D9
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 07:03:03 +0000 (UTC)
Date: Thu, 5 May 2011 07:03:03 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <365101482.23930.1304578983340.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1243264568.21599.1301502005765.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7214) Hadoop /usr/bin/groups equivalent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029197#comment-13029197 ] 

Hadoop QA commented on HADOOP-7214:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478254/hadoop-7214.10.patch
  against trunk revision 1099633.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/404//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/404//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/404//console

This message is automatically generated.

> Hadoop /usr/bin/groups equivalent
> ---------------------------------
>
>                 Key: HADOOP-7214
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7214
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>         Attachments: hadoop-7214.0.txt, hadoop-7214.1.txt, hadoop-7214.10.patch, hadoop-7214.2.txt, hadoop-7214.3.txt, hadoop-7214.4.txt, hadoop-7214.5.txt, hadoop-7214.6.txt, hadoop-7214.7.patch, hadoop-7214.8.patch, hadoop-7214.9.patch
>
>
> Since user -> groups resolution is done on the NN and JT machines, there should be a way for users to determine what groups they're a member of from the NN's and JT's perspective.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14463-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 11:14:44 2011
Return-Path: <common-issues-return-14463-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 06A4DBE6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 11:14:44 +0000 (UTC)
Received: (qmail 85868 invoked by uid 500); 5 May 2011 11:14:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85208 invoked by uid 500); 5 May 2011 11:14:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84910 invoked by uid 99); 5 May 2011 11:14:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 11:14:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 11:14:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4BEDFC2E45
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 11:14:03 +0000 (UTC)
Date: Thu, 5 May 2011 11:14:03 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2029427239.24399.1304594043307.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <444776842.10957.1304053443189.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7250) Refactor FsShell's setrep
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029285#comment-13029285 ] 

Hudson commented on HADOOP-7250:
--------------------------------

Integrated in Hadoop-Common-trunk #679 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/679/])
    HADOOP-7250. Refactor the setrep command to conform to new FsCommand class.  Contributed by Daryn Sharp


> Refactor FsShell's setrep
> -------------------------
>
>                 Key: HADOOP-7250
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7250
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7250-2.patch, HADOOP-7250.patch
>
>
> Need to refactor setrep to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14464-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 11:14:44 2011
Return-Path: <common-issues-return-14464-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CDFE7C04
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 11:14:44 +0000 (UTC)
Received: (qmail 86408 invoked by uid 500); 5 May 2011 11:14:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86384 invoked by uid 500); 5 May 2011 11:14:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86376 invoked by uid 99); 5 May 2011 11:14:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 11:14:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 11:14:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3687EC2E42
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 11:14:03 +0000 (UTC)
Date: Thu, 5 May 2011 11:14:03 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1318714061.24396.1304594043219.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1912834846.74827.1303423325837.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7236) Refactor FsShell's mkdir
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029284#comment-13029284 ] 

Hudson commented on HADOOP-7236:
--------------------------------

Integrated in Hadoop-Common-trunk #679 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/679/])
    HADOOP-7236. Refactor the mkdir command to conform to new FsCommand class.  Contributed by Daryn Sharp


> Refactor FsShell's mkdir
> ------------------------
>
>                 Key: HADOOP-7236
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7236
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7236-2.patch, HADOOP-7236-3.patch, HADOOP-7236.patch
>
>
> Need to refactor tail to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14465-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 14:09:42 2011
Return-Path: <common-issues-return-14465-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DEB9E2CF6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 14:09:42 +0000 (UTC)
Received: (qmail 76812 invoked by uid 500); 5 May 2011 14:09:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76778 invoked by uid 500); 5 May 2011 14:09:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76770 invoked by uid 99); 5 May 2011 14:09:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 14:09:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 14:09:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 27332C2FA7
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 14:09:03 +0000 (UTC)
Date: Thu, 5 May 2011 14:09:03 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1008351639.24627.1304604543156.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <651142754.74841.1303423565756.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7238) Refactor FsShell's cat & text
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029339#comment-13029339 ] 

Daryn Sharp commented on HADOOP-7238:
-------------------------------------

Yes, the file magic checks and the TextRecordInputStream are not changed.  Cut-n-paste.

> Refactor FsShell's cat & text
> -----------------------------
>
>                 Key: HADOOP-7238
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7238
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7238.patch
>
>
> Need to refactor cat & text to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14466-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 15:21:48 2011
Return-Path: <common-issues-return-14466-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C74612BB9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 15:21:48 +0000 (UTC)
Received: (qmail 17382 invoked by uid 500); 5 May 2011 15:21:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17351 invoked by uid 500); 5 May 2011 15:21:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17343 invoked by uid 99); 5 May 2011 15:21:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 15:21:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 15:21:47 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2B6E4C2057
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 15:21:09 +0000 (UTC)
Date: Thu, 5 May 2011 15:21:09 +0000 (UTC)
From: "Thomas Graves (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2022424951.24759.1304608869173.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <285327525.9499.1304014323342.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7248) Have a way to automatically update
 Eclipse .classpath file when new libs are added to the classpath through
 Ivy for 0.20-* based sources
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029367#comment-13029367 ] 

Thomas Graves commented on HADOOP-7248:
---------------------------------------

I've backported this change to the branch-0.20-security.  I'm facing an issue in running the unit tests in eclipse where it can't find the webapps directory now.  I see there was a jira https://issues.apache.org/jira/browse/HADOOP-6461 that went into 22 that worked around the issue.  I'm looking for feedback if people think that is the proper thing to do or if I should have the build.xml copy it into the eclipse classpath or something else?  I'm new to hadoop so am not familiar with all the use cases and processes yet. 

Also does anyone know if all the tests in the 20 branch were running via eclipse? 

> Have a way to automatically update Eclipse .classpath file when new libs are added to the classpath through Ivy for 0.20-* based sources
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7248
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7248
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.3
>            Reporter: Konstantin Boudnik
>            Priority: Minor
>             Fix For: 0.21.0
>
>
> Backport HADOOP-6407 into 0.20 based source trees

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14467-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 16:03:43 2011
Return-Path: <common-issues-return-14467-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B89522D44
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 16:03:43 +0000 (UTC)
Received: (qmail 30136 invoked by uid 500); 5 May 2011 16:03:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30106 invoked by uid 500); 5 May 2011 16:03:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30098 invoked by uid 99); 5 May 2011 16:03:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 16:03:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 16:03:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3A8FBC26EF
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 16:03:04 +0000 (UTC)
Date: Thu, 5 May 2011 16:03:04 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <991076716.25016.1304611384236.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6919) Metrics2: metrics framework
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-6919:
----------------------------

        Fix Version/s: 0.23.0
    Affects Version/s:     (was: 0.20.2)
                       0.22.0
               Status: Patch Available  (was: Open)

> Metrics2: metrics framework
> ---------------------------
>
>                 Key: HADOOP-6919
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6919
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: metrics
>    Affects Versions: 0.22.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-6919-trunk-v1.patch
>
>
> The jira tracks the new metrics framework only changes, i.e., it doesn't track the instrumentation changes (and compatibility issues) to existing code.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14468-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 16:03:46 2011
Return-Path: <common-issues-return-14468-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4B8702D6D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 16:03:46 +0000 (UTC)
Received: (qmail 30643 invoked by uid 500); 5 May 2011 16:03:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30601 invoked by uid 500); 5 May 2011 16:03:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30593 invoked by uid 99); 5 May 2011 16:03:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 16:03:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 16:03:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B20D2C26FE
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 16:03:04 +0000 (UTC)
Date: Thu, 5 May 2011 16:03:04 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <719077590.25029.1304611384726.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6919) Metrics2: metrics framework
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-6919:
----------------------------

    Status: Open  (was: Patch Available)

> Metrics2: metrics framework
> ---------------------------
>
>                 Key: HADOOP-6919
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6919
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: metrics
>    Affects Versions: 0.22.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-6919-trunk-v1.patch
>
>
> The jira tracks the new metrics framework only changes, i.e., it doesn't track the instrumentation changes (and compatibility issues) to existing code.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14469-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 16:05:44 2011
Return-Path: <common-issues-return-14469-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 117A22DBC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 16:05:44 +0000 (UTC)
Received: (qmail 33084 invoked by uid 500); 5 May 2011 16:05:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33047 invoked by uid 500); 5 May 2011 16:05:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33039 invoked by uid 99); 5 May 2011 16:05:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 16:05:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 16:05:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9AED4C27EA
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 16:05:04 +0000 (UTC)
Date: Thu, 5 May 2011 16:05:04 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2139695068.25057.1304611504631.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6919) Metrics2: metrics framework
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-6919:
----------------------------

    Status: Patch Available  (was: Open)

> Metrics2: metrics framework
> ---------------------------
>
>                 Key: HADOOP-6919
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6919
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: metrics
>    Affects Versions: 0.22.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-6919-metrics2-framework-v6.patch, hadoop-6919-trunk-v1.patch
>
>
> The jira tracks the new metrics framework only changes, i.e., it doesn't track the instrumentation changes (and compatibility issues) to existing code.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14470-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 16:05:45 2011
Return-Path: <common-issues-return-14470-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B7A6B2DCA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 16:05:45 +0000 (UTC)
Received: (qmail 33351 invoked by uid 500); 5 May 2011 16:05:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33306 invoked by uid 500); 5 May 2011 16:05:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33298 invoked by uid 99); 5 May 2011 16:05:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 16:05:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 16:05:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 25981C27D5
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 16:05:04 +0000 (UTC)
Date: Thu, 5 May 2011 16:05:04 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <169369933.25044.1304611504149.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6919) Metrics2: metrics framework
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-6919:
----------------------------

    Attachment: hadoop-6919-metrics2-framework-v6.patch

> Metrics2: metrics framework
> ---------------------------
>
>                 Key: HADOOP-6919
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6919
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: metrics
>    Affects Versions: 0.22.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-6919-metrics2-framework-v6.patch, hadoop-6919-trunk-v1.patch
>
>
> The jira tracks the new metrics framework only changes, i.e., it doesn't track the instrumentation changes (and compatibility issues) to existing code.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14471-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 16:09:42 2011
Return-Path: <common-issues-return-14471-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C92F420A4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 16:09:42 +0000 (UTC)
Received: (qmail 38200 invoked by uid 500); 5 May 2011 16:09:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38165 invoked by uid 500); 5 May 2011 16:09:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38148 invoked by uid 99); 5 May 2011 16:09:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 16:09:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 16:09:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4157AC2999
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 16:09:03 +0000 (UTC)
Date: Thu, 5 May 2011 16:09:03 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1559172546.25063.1304611743264.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-6918) Make metrics naming consistent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu resolved HADOOP-6918.
-----------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Release Note: Harmonizing metrics names
     Hadoop Flags: [Incompatible change]

We adopted CapitalizedCamelCase.

> Make metrics naming consistent
> ------------------------------
>
>                 Key: HADOOP-6918
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6918
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: metrics
>    Affects Versions: 0.20.2
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>
> While working HADOOP-6728, I noticed that our metrics naming style is all over the place:
> * Capitalized camel case: e.g., "FilesCreated" in namenode metrics and some rpc metrics
> * uncapitalized camel case: e.g, "threadsBlocked" in jvm metrics and some rpc metrics
> * lowercased underscored: e.g., "bytes_written" in datanode metrics and mapreduce metrics
> Let's make them consistent. How about uncapitalized camel case? My main reason for the camel case: some backends have limits on the name length and underscore is wasteful.
> Once we have a consistent naming style we can do:
> @Metric("Number of INodes created") MutableCounterLong filesCreated;
> instead of the more redundant:
> @Metric({"FilesCreated", "Number of INodes created"}) MutableCounterLong filesCreated;

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14472-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 16:25:45 2011
Return-Path: <common-issues-return-14472-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5B4932BC7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 16:25:45 +0000 (UTC)
Received: (qmail 84424 invoked by uid 500); 5 May 2011 16:25:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84394 invoked by uid 500); 5 May 2011 16:25:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84386 invoked by uid 99); 5 May 2011 16:25:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 16:25:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 16:25:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9F5A6C21CD
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 16:25:03 +0000 (UTC)
Date: Thu, 5 May 2011 16:25:03 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2004895807.25114.1304612703649.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6919) Metrics2: metrics framework
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029414#comment-13029414 ] 

Hadoop QA commented on HADOOP-6919:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478290/hadoop-6919-metrics2-framework-v6.patch
  against trunk revision 1099633.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 36 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/405//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/405//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/405//console

This message is automatically generated.

> Metrics2: metrics framework
> ---------------------------
>
>                 Key: HADOOP-6919
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6919
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: metrics
>    Affects Versions: 0.22.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-6919-metrics2-framework-v6.patch, hadoop-6919-trunk-v1.patch
>
>
> The jira tracks the new metrics framework only changes, i.e., it doesn't track the instrumentation changes (and compatibility issues) to existing code.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14473-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 17:01:44 2011
Return-Path: <common-issues-return-14473-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 71B792C80
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 17:01:44 +0000 (UTC)
Received: (qmail 54277 invoked by uid 500); 5 May 2011 17:01:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54190 invoked by uid 500); 5 May 2011 17:01:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53990 invoked by uid 99); 5 May 2011 17:01:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 17:01:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 17:01:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 49A65C2085
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 17:01:03 +0000 (UTC)
Date: Thu, 5 May 2011 17:01:03 +0000 (UTC)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1983565491.25193.1304614863298.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <304321177.482.1298765698936.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7157) Create build hosts configurations
 to prevent degradation of patch validation and snapshot build environment
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029436#comment-13029436 ] 

Konstantin Boudnik commented on HADOOP-7157:
--------------------------------------------

Thanks for the review. I will fix the patch to make it more compact with resource array and will post it shortly.

As for applying: your #3 is pretty goes pretty much along with what I had in mind by saying "to be done say once a day". Storing it along with rest of Hadoop source base seems to be a logical thing to do too.

> Create build hosts configurations to prevent degradation of patch validation and snapshot build environment
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7157
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7157
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>         Environment: Apache Hadoop build hosts
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-7157.patch
>
>
> Build hosts are being re-jumped, in need for configuration updates, etc. It is hard to maintain the same configuration across a 10+ hosts. A specialized service such as Puppet can be used to maintain build machines software and configurations at a desired level.
> Such configs should be checked into SCM system along with the of sources code.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14474-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 17:05:45 2011
Return-Path: <common-issues-return-14474-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5460B2A42
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 17:05:45 +0000 (UTC)
Received: (qmail 61102 invoked by uid 500); 5 May 2011 17:05:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61064 invoked by uid 500); 5 May 2011 17:05:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61056 invoked by uid 99); 5 May 2011 17:05:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 17:05:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 17:05:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C70DBC2216
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 17:05:03 +0000 (UTC)
Date: Thu, 5 May 2011 17:05:03 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <130482159.25201.1304615103811.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6918) Make metrics naming consistent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029438#comment-13029438 ] 

Eli Collins commented on HADOOP-6918:
-------------------------------------

This is resolved/fixed? This jira fix version 23 and in trunk the metrics names did not change, eg bytes_written is still bytes_written.



> Make metrics naming consistent
> ------------------------------
>
>                 Key: HADOOP-6918
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6918
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: metrics
>    Affects Versions: 0.20.2
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>
> While working HADOOP-6728, I noticed that our metrics naming style is all over the place:
> * Capitalized camel case: e.g., "FilesCreated" in namenode metrics and some rpc metrics
> * uncapitalized camel case: e.g, "threadsBlocked" in jvm metrics and some rpc metrics
> * lowercased underscored: e.g., "bytes_written" in datanode metrics and mapreduce metrics
> Let's make them consistent. How about uncapitalized camel case? My main reason for the camel case: some backends have limits on the name length and underscore is wasteful.
> Once we have a consistent naming style we can do:
> @Metric("Number of INodes created") MutableCounterLong filesCreated;
> instead of the more redundant:
> @Metric({"FilesCreated", "Number of INodes created"}) MutableCounterLong filesCreated;

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14475-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 17:40:42 2011
Return-Path: <common-issues-return-14475-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D32FD2EFA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 17:40:42 +0000 (UTC)
Received: (qmail 32010 invoked by uid 500); 5 May 2011 17:40:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31957 invoked by uid 500); 5 May 2011 17:40:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31949 invoked by uid 99); 5 May 2011 17:40:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 17:40:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 17:40:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 734A9C232D
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 17:40:03 +0000 (UTC)
Date: Thu, 5 May 2011 17:40:03 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <927542926.25333.1304617203469.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6918) Make metrics naming consistent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029457#comment-13029457 ] 

Luke Lu commented on HADOOP-6918:
---------------------------------

Thanks for the comment, Eli, 

The purpose of the jira was to track the proposal/discussion of the changes. I should've commented that the actual changes will be covered by metrics instrumentation patches in HADOOP-6920, HDFS-1117 and MAPREDUCE-1738.

> Make metrics naming consistent
> ------------------------------
>
>                 Key: HADOOP-6918
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6918
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: metrics
>    Affects Versions: 0.20.2
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>
> While working HADOOP-6728, I noticed that our metrics naming style is all over the place:
> * Capitalized camel case: e.g., "FilesCreated" in namenode metrics and some rpc metrics
> * uncapitalized camel case: e.g, "threadsBlocked" in jvm metrics and some rpc metrics
> * lowercased underscored: e.g., "bytes_written" in datanode metrics and mapreduce metrics
> Let's make them consistent. How about uncapitalized camel case? My main reason for the camel case: some backends have limits on the name length and underscore is wasteful.
> Once we have a consistent naming style we can do:
> @Metric("Number of INodes created") MutableCounterLong filesCreated;
> instead of the more redundant:
> @Metric({"FilesCreated", "Number of INodes created"}) MutableCounterLong filesCreated;

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14476-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 17:54:45 2011
Return-Path: <common-issues-return-14476-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 375E4278C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 17:54:45 +0000 (UTC)
Received: (qmail 51091 invoked by uid 500); 5 May 2011 17:54:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51058 invoked by uid 500); 5 May 2011 17:54:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51050 invoked by uid 99); 5 May 2011 17:54:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 17:54:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 17:54:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5B856C2960
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 17:54:03 +0000 (UTC)
Date: Thu, 5 May 2011 17:54:03 +0000 (UTC)
From: "Thomas Graves (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <144382093.25377.1304618043371.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <285327525.9499.1304014323342.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7248) Have a way to automatically update
 Eclipse .classpath file when new libs are added to the classpath through
 Ivy for 0.20-* based sources
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Graves reassigned HADOOP-7248:
-------------------------------------

    Assignee: Thomas Graves

> Have a way to automatically update Eclipse .classpath file when new libs are added to the classpath through Ivy for 0.20-* based sources
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7248
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7248
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.3
>            Reporter: Konstantin Boudnik
>            Assignee: Thomas Graves
>            Priority: Minor
>             Fix For: 0.21.0
>
>
> Backport HADOOP-6407 into 0.20 based source trees

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14477-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 18:16:44 2011
Return-Path: <common-issues-return-14477-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D591946F7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 18:16:44 +0000 (UTC)
Received: (qmail 31978 invoked by uid 500); 5 May 2011 18:16:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31948 invoked by uid 500); 5 May 2011 18:16:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31932 invoked by uid 99); 5 May 2011 18:16:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 18:16:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 18:16:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4D43DC2352
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 18:16:03 +0000 (UTC)
Date: Thu, 5 May 2011 18:16:03 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <979758589.25435.1304619363313.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1243264568.21599.1301502005765.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7214) Hadoop /usr/bin/groups equivalent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029476#comment-13029476 ] 

Todd Lipcon commented on HADOOP-7214:
-------------------------------------

Looks like the JavaDoc still refers to RefreshUserMappingsProtocol where it should refer to GetUserMappingsProtocol.


> Hadoop /usr/bin/groups equivalent
> ---------------------------------
>
>                 Key: HADOOP-7214
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7214
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>         Attachments: hadoop-7214.0.txt, hadoop-7214.1.txt, hadoop-7214.10.patch, hadoop-7214.2.txt, hadoop-7214.3.txt, hadoop-7214.4.txt, hadoop-7214.5.txt, hadoop-7214.6.txt, hadoop-7214.7.patch, hadoop-7214.8.patch, hadoop-7214.9.patch
>
>
> Since user -> groups resolution is done on the NN and JT machines, there should be a way for users to determine what groups they're a member of from the NN's and JT's perspective.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14478-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 18:20:43 2011
Return-Path: <common-issues-return-14478-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 23E3120BC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 18:20:43 +0000 (UTC)
Received: (qmail 43754 invoked by uid 500); 5 May 2011 18:20:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43675 invoked by uid 500); 5 May 2011 18:20:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43667 invoked by uid 99); 5 May 2011 18:20:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 18:20:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 18:20:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 56FF9C255D
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 18:20:03 +0000 (UTC)
Date: Thu, 5 May 2011 18:20:03 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1144595263.25464.1304619603353.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6918) Make metrics naming consistent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029482#comment-13029482 ] 

Eli Collins commented on HADOOP-6918:
-------------------------------------

Thanks. Being consistent makes sense to me. Which jira will have a release note indicating the workaround for users (they'll need to rename according to the new names)? This jira seems like the best one to me since it's also flagged incompatible. 

> Make metrics naming consistent
> ------------------------------
>
>                 Key: HADOOP-6918
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6918
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: metrics
>    Affects Versions: 0.20.2
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>
> While working HADOOP-6728, I noticed that our metrics naming style is all over the place:
> * Capitalized camel case: e.g., "FilesCreated" in namenode metrics and some rpc metrics
> * uncapitalized camel case: e.g, "threadsBlocked" in jvm metrics and some rpc metrics
> * lowercased underscored: e.g., "bytes_written" in datanode metrics and mapreduce metrics
> Let's make them consistent. How about uncapitalized camel case? My main reason for the camel case: some backends have limits on the name length and underscore is wasteful.
> Once we have a consistent naming style we can do:
> @Metric("Number of INodes created") MutableCounterLong filesCreated;
> instead of the more redundant:
> @Metric({"FilesCreated", "Number of INodes created"}) MutableCounterLong filesCreated;

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14479-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 18:41:42 2011
Return-Path: <common-issues-return-14479-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9013F2EC6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 18:41:42 +0000 (UTC)
Received: (qmail 78653 invoked by uid 500); 5 May 2011 18:41:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78623 invoked by uid 500); 5 May 2011 18:41:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78615 invoked by uid 99); 5 May 2011 18:41:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 18:41:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 18:41:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2FAB8C2D09
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 18:41:03 +0000 (UTC)
Date: Thu, 5 May 2011 18:41:03 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <974043849.25504.1304620863191.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <796788985.10688.1304041503305.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7249) Refactor FsShell's
 chmod/chown/chgrp
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029490#comment-13029490 ] 

Matt Foley commented on HADOOP-7249:
------------------------------------

Hi Daryn, generally looks good but in FsShellPermissions.parseOwnerGroup() a tab (instead of spaces) crept in to the added lines just below "group = null;", to wit:
      if (group != null && group.length() == 0) {
-        group = null;
-      }
+          group = null;
+          } <-- problem here, and possibly prev line

> Refactor FsShell's chmod/chown/chgrp
> ------------------------------------
>
>                 Key: HADOOP-7249
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7249
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7249.patch
>
>
> Need to refactor permissions commands to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14480-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 18:43:42 2011
Return-Path: <common-issues-return-14480-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 80178203C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 18:43:42 +0000 (UTC)
Received: (qmail 83059 invoked by uid 500); 5 May 2011 18:43:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83028 invoked by uid 500); 5 May 2011 18:43:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83020 invoked by uid 99); 5 May 2011 18:43:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 18:43:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 18:43:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 27EFCC2DDC
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 18:43:03 +0000 (UTC)
Date: Thu, 5 May 2011 18:43:03 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1748280070.25507.1304620983160.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <796788985.10688.1304041503305.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7249) Refactor FsShell's
 chmod/chown/chgrp
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029491#comment-13029491 ] 

Matt Foley commented on HADOOP-7249:
------------------------------------

{code}
      if (group != null && group.length() == 0) {
-        group = null;
-      }
+          group = null;
+          } <-- problem here, and possibly prev line
{code}


> Refactor FsShell's chmod/chown/chgrp
> ------------------------------------
>
>                 Key: HADOOP-7249
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7249
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7249.patch
>
>
> Need to refactor permissions commands to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14481-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 18:51:42 2011
Return-Path: <common-issues-return-14481-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B04DE2B26
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 18:51:42 +0000 (UTC)
Received: (qmail 10547 invoked by uid 500); 5 May 2011 18:51:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10494 invoked by uid 500); 5 May 2011 18:51:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10486 invoked by uid 99); 5 May 2011 18:51:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 18:51:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 18:51:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3E410C3192
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 18:51:03 +0000 (UTC)
Date: Thu, 5 May 2011 18:51:03 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1830803980.25557.1304621463251.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <796788985.10688.1304041503305.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7249) Refactor FsShell's chmod/chown/chgrp
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7249:
--------------------------------

    Attachment: HADOOP-7249-2.patch

Removed tabs.

> Refactor FsShell's chmod/chown/chgrp
> ------------------------------------
>
>                 Key: HADOOP-7249
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7249
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7249-2.patch, HADOOP-7249.patch
>
>
> Need to refactor permissions commands to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14482-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 19:03:43 2011
Return-Path: <common-issues-return-14482-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 421722271
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 19:03:43 +0000 (UTC)
Received: (qmail 46007 invoked by uid 500); 5 May 2011 19:03:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45952 invoked by uid 500); 5 May 2011 19:03:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45859 invoked by uid 99); 5 May 2011 19:03:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 19:03:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 19:03:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6E788C360C
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 19:03:03 +0000 (UTC)
Date: Thu, 5 May 2011 19:03:03 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <526658133.25588.1304622183449.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <796788985.10688.1304041503305.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7249) Refactor FsShell's
 chmod/chown/chgrp
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029505#comment-13029505 ] 

Hadoop QA commented on HADOOP-7249:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478308/HADOOP-7249-2.patch
  against trunk revision 1099633.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 10 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/406//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/406//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/406//console

This message is automatically generated.

> Refactor FsShell's chmod/chown/chgrp
> ------------------------------------
>
>                 Key: HADOOP-7249
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7249
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7249-2.patch, HADOOP-7249.patch
>
>
> Need to refactor permissions commands to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14483-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 19:13:43 2011
Return-Path: <common-issues-return-14483-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 12751288C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 19:13:43 +0000 (UTC)
Received: (qmail 65087 invoked by uid 500); 5 May 2011 19:13:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65053 invoked by uid 500); 5 May 2011 19:13:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65044 invoked by uid 99); 5 May 2011 19:13:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 19:13:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 19:13:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7CB91C39CF
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 19:13:03 +0000 (UTC)
Date: Thu, 5 May 2011 19:13:03 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <495384625.25619.1304622783507.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1243264568.21599.1301502005765.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7214) Hadoop /usr/bin/groups equivalent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7214:
-----------------------------------

    Attachment: hadoop-7214.11.patch

Thanks for the comments, Todd. I missed adding the file during the splitting of the protocol interfaces.

Updated patch attached.

> Hadoop /usr/bin/groups equivalent
> ---------------------------------
>
>                 Key: HADOOP-7214
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7214
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>         Attachments: hadoop-7214.0.txt, hadoop-7214.1.txt, hadoop-7214.10.patch, hadoop-7214.11.patch, hadoop-7214.2.txt, hadoop-7214.3.txt, hadoop-7214.4.txt, hadoop-7214.5.txt, hadoop-7214.6.txt, hadoop-7214.7.patch, hadoop-7214.8.patch, hadoop-7214.9.patch
>
>
> Since user -> groups resolution is done on the NN and JT machines, there should be a way for users to determine what groups they're a member of from the NN's and JT's perspective.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14484-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 19:33:45 2011
Return-Path: <common-issues-return-14484-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 22192248F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 19:33:45 +0000 (UTC)
Received: (qmail 98230 invoked by uid 500); 5 May 2011 19:33:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98180 invoked by uid 500); 5 May 2011 19:33:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98168 invoked by uid 99); 5 May 2011 19:33:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 19:33:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 19:33:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4D022C21A3
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 19:33:03 +0000 (UTC)
Date: Thu, 5 May 2011 19:33:03 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1723001201.25702.1304623983312.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1243264568.21599.1301502005765.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7214) Hadoop /usr/bin/groups equivalent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029525#comment-13029525 ] 

Hadoop QA commented on HADOOP-7214:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478313/hadoop-7214.11.patch
  against trunk revision 1099633.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 2 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/407//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/407//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/407//console

This message is automatically generated.

> Hadoop /usr/bin/groups equivalent
> ---------------------------------
>
>                 Key: HADOOP-7214
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7214
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>         Attachments: hadoop-7214.0.txt, hadoop-7214.1.txt, hadoop-7214.10.patch, hadoop-7214.11.patch, hadoop-7214.2.txt, hadoop-7214.3.txt, hadoop-7214.4.txt, hadoop-7214.5.txt, hadoop-7214.6.txt, hadoop-7214.7.patch, hadoop-7214.8.patch, hadoop-7214.9.patch
>
>
> Since user -> groups resolution is done on the NN and JT machines, there should be a way for users to determine what groups they're a member of from the NN's and JT's perspective.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14485-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 20:43:45 2011
Return-Path: <common-issues-return-14485-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 34C0D20CA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 20:43:45 +0000 (UTC)
Received: (qmail 16494 invoked by uid 500); 5 May 2011 20:43:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16465 invoked by uid 500); 5 May 2011 20:43:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16455 invoked by uid 99); 5 May 2011 20:43:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 20:43:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 20:43:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 371B9C3BD0
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 20:43:03 +0000 (UTC)
Date: Thu, 5 May 2011 20:43:03 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <210639408.25898.1304628183222.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <796788985.10688.1304041503305.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7249) Refactor FsShell's
 chmod/chown/chgrp
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029554#comment-13029554 ] 

Matt Foley commented on HADOOP-7249:
------------------------------------

+1

> Refactor FsShell's chmod/chown/chgrp
> ------------------------------------
>
>                 Key: HADOOP-7249
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7249
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7249-2.patch, HADOOP-7249.patch
>
>
> Need to refactor permissions commands to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14486-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 21:21:43 2011
Return-Path: <common-issues-return-14486-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 193DD2D14
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 21:21:43 +0000 (UTC)
Received: (qmail 61344 invoked by uid 500); 5 May 2011 21:21:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61304 invoked by uid 500); 5 May 2011 21:21:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61292 invoked by uid 99); 5 May 2011 21:21:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:21:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:21:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 599F1C3848
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 21:21:03 +0000 (UTC)
Date: Thu, 5 May 2011 21:21:03 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <826874576.26062.1304630463364.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2082135242.26059.1304630463280.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7260) update the tlp site with the bylaws
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7260:
----------------------------------

    Attachment: bylaws.patch

> update the tlp site with the bylaws
> -----------------------------------
>
>                 Key: HADOOP-7260
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7260
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: site
>
>         Attachments: bylaws.patch
>
>
> Include the bylaws in the tlp site.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14487-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 21:21:45 2011
Return-Path: <common-issues-return-14487-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 484922D34
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 21:21:45 +0000 (UTC)
Received: (qmail 61783 invoked by uid 500); 5 May 2011 21:21:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61735 invoked by uid 500); 5 May 2011 21:21:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61727 invoked by uid 99); 5 May 2011 21:21:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:21:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:21:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 45458C3845
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 21:21:03 +0000 (UTC)
Date: Thu, 5 May 2011 21:21:03 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2082135242.26059.1304630463280.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7260) update the tlp site with the bylaws
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

update the tlp site with the bylaws
-----------------------------------

                 Key: HADOOP-7260
                 URL: https://issues.apache.org/jira/browse/HADOOP-7260
             Project: Hadoop Common
          Issue Type: Task
          Components: documentation
            Reporter: Owen O'Malley
            Assignee: Owen O'Malley
             Fix For: site
         Attachments: bylaws.patch

Include the bylaws in the tlp site.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14488-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 21:29:44 2011
Return-Path: <common-issues-return-14488-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AF679208D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 21:29:44 +0000 (UTC)
Received: (qmail 72645 invoked by uid 500); 5 May 2011 21:29:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72612 invoked by uid 500); 5 May 2011 21:29:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72603 invoked by uid 99); 5 May 2011 21:29:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:29:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:29:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1DE83C3A91
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 21:29:03 +0000 (UTC)
Date: Thu, 5 May 2011 21:29:03 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1613358262.26067.1304630943119.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2082135242.26059.1304630463280.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7260) update the tlp site with the
 bylaws
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029583#comment-13029583 ] 

Eli Collins commented on HADOOP-7260:
-------------------------------------

+1  lgtm.  I couldn't find the thread with the final bylaws but these look right to me, eg includes the latest amendments made to address Nigel's feedback.

> update the tlp site with the bylaws
> -----------------------------------
>
>                 Key: HADOOP-7260
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7260
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: site
>
>         Attachments: bylaws.patch
>
>
> Include the bylaws in the tlp site.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14489-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 21:33:42 2011
Return-Path: <common-issues-return-14489-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 90CC522F4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 21:33:42 +0000 (UTC)
Received: (qmail 75820 invoked by uid 500); 5 May 2011 21:33:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75788 invoked by uid 500); 5 May 2011 21:33:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75780 invoked by uid 99); 5 May 2011 21:33:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:33:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:33:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 25C7AC3BC0
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 21:33:03 +0000 (UTC)
Date: Thu, 5 May 2011 21:33:03 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1181104140.26071.1304631183151.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2082135242.26059.1304630463280.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7260) update the tlp site with the bylaws
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley resolved HADOOP-7260.
-----------------------------------

    Resolution: Fixed

I just committed this.

> update the tlp site with the bylaws
> -----------------------------------
>
>                 Key: HADOOP-7260
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7260
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: site
>
>         Attachments: bylaws.patch
>
>
> Include the bylaws in the tlp site.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14490-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 21:35:42 2011
Return-Path: <common-issues-return-14490-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CA7282693
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 21:35:42 +0000 (UTC)
Received: (qmail 81465 invoked by uid 500); 5 May 2011 21:35:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81436 invoked by uid 500); 5 May 2011 21:35:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81428 invoked by uid 99); 5 May 2011 21:35:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:35:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:35:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 614FAC3C64
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 21:35:03 +0000 (UTC)
Date: Thu, 5 May 2011 21:35:03 +0000 (UTC)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <485775719.26078.1304631303395.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2082135242.26059.1304630463280.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7260) update the tlp site with the
 bylaws
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029585#comment-13029585 ] 

Nigel Daley commented on HADOOP-7260:
-------------------------------------

+1 lgtm too (assuming broken link is fixed).

> update the tlp site with the bylaws
> -----------------------------------
>
>                 Key: HADOOP-7260
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7260
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: site
>
>         Attachments: bylaws.patch
>
>
> Include the bylaws in the tlp site.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14491-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 21:41:44 2011
Return-Path: <common-issues-return-14491-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 129FA279E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 21:41:44 +0000 (UTC)
Received: (qmail 84988 invoked by uid 500); 5 May 2011 21:41:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84961 invoked by uid 500); 5 May 2011 21:41:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84953 invoked by uid 99); 5 May 2011 21:41:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:41:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:41:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 95F7FC3E87
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 21:41:04 +0000 (UTC)
Date: Thu, 5 May 2011 21:41:04 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1900828118.26109.1304631664611.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-6255:
------------------------------

    Attachment: HADOOP-6255-branch-0.20-security-8.patch

Rebase the patch for 0.20 security branch

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14492-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 21:41:48 2011
Return-Path: <common-issues-return-14492-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 72BBF27ED
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 21:41:48 +0000 (UTC)
Received: (qmail 86078 invoked by uid 500); 5 May 2011 21:41:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86029 invoked by uid 500); 5 May 2011 21:41:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86021 invoked by uid 99); 5 May 2011 21:41:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:41:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:41:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6634AC3E68
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 21:41:03 +0000 (UTC)
Date: Thu, 5 May 2011 21:41:03 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1268665495.26080.1304631663415.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2082135242.26059.1304630463280.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7260) update the tlp site with the
 bylaws
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029586#comment-13029586 ] 

Owen O'Malley commented on HADOOP-7260:
---------------------------------------

I just fixed a minor typo that Nigel pointed out.

> update the tlp site with the bylaws
> -----------------------------------
>
>                 Key: HADOOP-7260
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7260
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: site
>
>         Attachments: bylaws.patch
>
>
> Include the bylaws in the tlp site.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14493-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 21:41:49 2011
Return-Path: <common-issues-return-14493-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B333A27FF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 21:41:49 +0000 (UTC)
Received: (qmail 86333 invoked by uid 500); 5 May 2011 21:41:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86291 invoked by uid 500); 5 May 2011 21:41:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86283 invoked by uid 99); 5 May 2011 21:41:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:41:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:41:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A55A6C3E89
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 21:41:04 +0000 (UTC)
Date: Thu, 5 May 2011 21:41:04 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <605713660.26111.1304631664674.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2082135242.26059.1304630463280.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Closed] (HADOOP-7260) update the tlp site with the bylaws
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley closed HADOOP-7260.
---------------------------------


> update the tlp site with the bylaws
> -----------------------------------
>
>                 Key: HADOOP-7260
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7260
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: site
>
>         Attachments: bylaws.patch
>
>
> Include the bylaws in the tlp site.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14494-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 21:41:50 2011
Return-Path: <common-issues-return-14494-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E44A3280C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 21:41:49 +0000 (UTC)
Received: (qmail 86571 invoked by uid 500); 5 May 2011 21:41:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86519 invoked by uid 500); 5 May 2011 21:41:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86495 invoked by uid 99); 5 May 2011 21:41:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:41:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:41:44 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D35EEC3E90
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 21:41:04 +0000 (UTC)
Date: Thu, 5 May 2011 21:41:04 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1440041838.26118.1304631664862.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-6255:
------------------------------

    Attachment: HADOOP-6255-common-trunk.patch

Rpm/deb packaging for Hadoop common trunk

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14495-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 21:43:45 2011
Return-Path: <common-issues-return-14495-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F2377287A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 21:43:44 +0000 (UTC)
Received: (qmail 87918 invoked by uid 500); 5 May 2011 21:43:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87893 invoked by uid 500); 5 May 2011 21:43:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87885 invoked by uid 99); 5 May 2011 21:43:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:43:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:43:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 53F71C3F5A
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 21:43:03 +0000 (UTC)
Date: Thu, 5 May 2011 21:43:03 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1375501526.26153.1304631783340.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-6255:
------------------------------

    Attachment: HADOOP-6255-hdfs-trunk.patch

Rpm/deb packaging for Hadoop trunk

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14496-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 21:43:46 2011
Return-Path: <common-issues-return-14496-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9012B288C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 21:43:46 +0000 (UTC)
Received: (qmail 88206 invoked by uid 500); 5 May 2011 21:43:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88168 invoked by uid 500); 5 May 2011 21:43:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88160 invoked by uid 99); 5 May 2011 21:43:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:43:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:43:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5BD77C3F79
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 21:43:04 +0000 (UTC)
Date: Thu, 5 May 2011 21:43:04 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1121869318.26181.1304631784372.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029592#comment-13029592 ] 

Hadoop QA commented on HADOOP-6255:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478334/HADOOP-6255-hdfs-trunk.patch
  against trunk revision 1099633.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 14 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/408//console

This message is automatically generated.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14497-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 21:45:45 2011
Return-Path: <common-issues-return-14497-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A21302D49
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 21:45:45 +0000 (UTC)
Received: (qmail 89687 invoked by uid 500); 5 May 2011 21:45:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89647 invoked by uid 500); 5 May 2011 21:45:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89639 invoked by uid 99); 5 May 2011 21:45:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:45:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:45:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3CCD4C3029
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 21:45:03 +0000 (UTC)
Date: Thu, 5 May 2011 21:45:03 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1819517316.26211.1304631903245.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-6255:
------------------------------

    Attachment: HADOOP-6255-mapred-trunk.patch

Rpm/deb packaging for Hadoop Mapred trunk

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14498-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 21:53:45 2011
Return-Path: <common-issues-return-14498-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 69E992E61
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 21:53:45 +0000 (UTC)
Received: (qmail 91856 invoked by uid 500); 5 May 2011 21:53:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91826 invoked by uid 500); 5 May 2011 21:53:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91813 invoked by uid 99); 5 May 2011 21:53:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:53:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:53:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BD479C32C9
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 21:53:03 +0000 (UTC)
Date: Thu, 5 May 2011 21:53:03 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <515626176.26270.1304632383772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029597#comment-13029597 ] 

Hadoop QA commented on HADOOP-6255:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478335/HADOOP-6255-mapred-trunk.patch
  against trunk revision 1099633.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 15 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/409//console

This message is automatically generated.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14499-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 21:55:46 2011
Return-Path: <common-issues-return-14499-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DF3272EE6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 21:55:45 +0000 (UTC)
Received: (qmail 94573 invoked by uid 500); 5 May 2011 21:55:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94542 invoked by uid 500); 5 May 2011 21:55:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94534 invoked by uid 99); 5 May 2011 21:55:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:55:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 21:55:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BC963C33B8
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 21:55:03 +0000 (UTC)
Date: Thu, 5 May 2011 21:55:03 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2093241905.26316.1304632503769.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6918) Make metrics naming consistent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029600#comment-13029600 ] 

Luke Lu commented on HADOOP-6918:
---------------------------------

bq. This jira seems like the best one to me since it's also flagged incompatible.

Make sense, reopen.

As for workaround, metrics sinks can be trivially enhanced to have user defined mappings, which is what we did for production deployment.

> Make metrics naming consistent
> ------------------------------
>
>                 Key: HADOOP-6918
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6918
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: metrics
>    Affects Versions: 0.20.2
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>
> While working HADOOP-6728, I noticed that our metrics naming style is all over the place:
> * Capitalized camel case: e.g., "FilesCreated" in namenode metrics and some rpc metrics
> * uncapitalized camel case: e.g, "threadsBlocked" in jvm metrics and some rpc metrics
> * lowercased underscored: e.g., "bytes_written" in datanode metrics and mapreduce metrics
> Let's make them consistent. How about uncapitalized camel case? My main reason for the camel case: some backends have limits on the name length and underscore is wasteful.
> Once we have a consistent naming style we can do:
> @Metric("Number of INodes created") MutableCounterLong filesCreated;
> instead of the more redundant:
> @Metric({"FilesCreated", "Number of INodes created"}) MutableCounterLong filesCreated;

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14500-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 22:07:44 2011
Return-Path: <common-issues-return-14500-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 90C952CEC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 22:07:44 +0000 (UTC)
Received: (qmail 18846 invoked by uid 500); 5 May 2011 22:07:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18744 invoked by uid 500); 5 May 2011 22:07:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18728 invoked by uid 99); 5 May 2011 22:07:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 22:07:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 22:07:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 366B9C3715
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 22:07:03 +0000 (UTC)
Date: Thu, 5 May 2011 22:07:03 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1065826906.26360.1304633223219.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Reopened] (HADOOP-6918) Make metrics naming consistent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu reopened HADOOP-6918:
-----------------------------


> Make metrics naming consistent
> ------------------------------
>
>                 Key: HADOOP-6918
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6918
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: metrics
>    Affects Versions: 0.20.2
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>
> While working HADOOP-6728, I noticed that our metrics naming style is all over the place:
> * Capitalized camel case: e.g., "FilesCreated" in namenode metrics and some rpc metrics
> * uncapitalized camel case: e.g, "threadsBlocked" in jvm metrics and some rpc metrics
> * lowercased underscored: e.g., "bytes_written" in datanode metrics and mapreduce metrics
> Let's make them consistent. How about uncapitalized camel case? My main reason for the camel case: some backends have limits on the name length and underscore is wasteful.
> Once we have a consistent naming style we can do:
> @Metric("Number of INodes created") MutableCounterLong filesCreated;
> instead of the more redundant:
> @Metric({"FilesCreated", "Number of INodes created"}) MutableCounterLong filesCreated;

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14501-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 22:09:47 2011
Return-Path: <common-issues-return-14501-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 25E032DB6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 22:09:47 +0000 (UTC)
Received: (qmail 23074 invoked by uid 500); 5 May 2011 22:09:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23033 invoked by uid 500); 5 May 2011 22:09:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22996 invoked by uid 99); 5 May 2011 22:09:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 22:09:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 22:09:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 33CA2C377E
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 22:09:03 +0000 (UTC)
Date: Thu, 5 May 2011 22:09:03 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1362418509.26364.1304633343208.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7261) Disable IPV6 for junit tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Disable IPV6 for junit tests
----------------------------

                 Key: HADOOP-7261
                 URL: https://issues.apache.org/jira/browse/HADOOP-7261
             Project: Hadoop Common
          Issue Type: Bug
          Components: test
    Affects Versions: 0.23.0
            Reporter: Suresh Srinivas
            Assignee: Suresh Srinivas
             Fix For: 0.23.0


IPV6 addresses not handles currently in the common library methods. IPV6 can return address as "0:0:0:0:0:0:port". Some utility methods such as NetUtils#createSocketAddress(), NetUtils#normalizeHostName(), NetUtils#getHostNameOfIp() to name a few, do not handle IPV6 address and expect address to be of format host:port.

Until IPV6 is formally supported, I propose disabling IPV6 for junit tests to avoid problems seen in HDFS-1891.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14502-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 22:13:43 2011
Return-Path: <common-issues-return-14502-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E5AF22EC4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 22:13:42 +0000 (UTC)
Received: (qmail 31668 invoked by uid 500); 5 May 2011 22:13:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31638 invoked by uid 500); 5 May 2011 22:13:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31630 invoked by uid 99); 5 May 2011 22:13:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 22:13:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 22:13:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 45487C38A5
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 22:13:03 +0000 (UTC)
Date: Thu, 5 May 2011 22:13:03 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <617317558.26372.1304633583280.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1362418509.26364.1304633343208.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7261) Disable IPV6 for junit tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-7261:
------------------------------------

    Attachment: HADOOP-7261.patch

> Disable IPV6 for junit tests
> ----------------------------
>
>                 Key: HADOOP-7261
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7261
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7261.patch
>
>
> IPV6 addresses not handles currently in the common library methods. IPV6 can return address as "0:0:0:0:0:0:port". Some utility methods such as NetUtils#createSocketAddress(), NetUtils#normalizeHostName(), NetUtils#getHostNameOfIp() to name a few, do not handle IPV6 address and expect address to be of format host:port.
> Until IPV6 is formally supported, I propose disabling IPV6 for junit tests to avoid problems seen in HDFS-1891.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14503-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 22:17:50 2011
Return-Path: <common-issues-return-14503-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4F55D29E5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 22:17:50 +0000 (UTC)
Received: (qmail 33639 invoked by uid 500); 5 May 2011 22:17:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33516 invoked by uid 500); 5 May 2011 22:17:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33508 invoked by uid 99); 5 May 2011 22:17:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 22:17:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 22:17:47 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id ACE1BC39F3
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 22:17:08 +0000 (UTC)
Date: Thu, 5 May 2011 22:17:08 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1832774673.26386.1304633828704.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7257) A client side mount table to give
 per-application/per-job file system view
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029613#comment-13029613 ] 

Jitendra Nath Pandey commented on HADOOP-7257:
----------------------------------------------

+1 for the patch.

> A client side mount table to give per-application/per-job file system view
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7257
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7257
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: ViewFs - javdoc2.pdf, ViewFsJavaDoc.pdf, viewFs2.patch, viewFs3.patch, viewFs4.patch, viewFs5.patch, viewfs1.patch
>
>
> This jira proposes a client side mount table to allow application-centric (or job-centric) filesystem views. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14504-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 22:21:43 2011
Return-Path: <common-issues-return-14504-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0F1872559
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 22:21:43 +0000 (UTC)
Received: (qmail 37633 invoked by uid 500); 5 May 2011 22:21:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37581 invoked by uid 500); 5 May 2011 22:21:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37573 invoked by uid 99); 5 May 2011 22:21:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 22:21:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 22:21:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7F904C3C2A
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 22:21:03 +0000 (UTC)
Date: Thu, 5 May 2011 22:21:03 +0000 (UTC)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1901278751.26423.1304634063518.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1362418509.26364.1304633343208.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7261) Disable IPV6 for junit tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029617#comment-13029617 ] 

Nigel Daley commented on HADOOP-7261:
-------------------------------------

Should we include this in the contrib build.xml too?  Otherwise +1 on the current patch.

> Disable IPV6 for junit tests
> ----------------------------
>
>                 Key: HADOOP-7261
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7261
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7261.patch
>
>
> IPV6 addresses not handles currently in the common library methods. IPV6 can return address as "0:0:0:0:0:0:port". Some utility methods such as NetUtils#createSocketAddress(), NetUtils#normalizeHostName(), NetUtils#getHostNameOfIp() to name a few, do not handle IPV6 address and expect address to be of format host:port.
> Until IPV6 is formally supported, I propose disabling IPV6 for junit tests to avoid problems seen in HDFS-1891.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14505-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 22:37:42 2011
Return-Path: <common-issues-return-14505-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C29172DFB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 22:37:42 +0000 (UTC)
Received: (qmail 61405 invoked by uid 500); 5 May 2011 22:37:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61373 invoked by uid 500); 5 May 2011 22:37:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61365 invoked by uid 99); 5 May 2011 22:37:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 22:37:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 22:37:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 522EAC320C
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 22:37:03 +0000 (UTC)
Date: Thu, 5 May 2011 22:37:03 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1760864585.26482.1304635023333.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1362418509.26364.1304633343208.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7261) Disable IPV6 for junit tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029628#comment-13029628 ] 

Matt Foley commented on HADOOP-7261:
------------------------------------

+1 on this patch

> Disable IPV6 for junit tests
> ----------------------------
>
>                 Key: HADOOP-7261
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7261
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7261.patch
>
>
> IPV6 addresses not handles currently in the common library methods. IPV6 can return address as "0:0:0:0:0:0:port". Some utility methods such as NetUtils#createSocketAddress(), NetUtils#normalizeHostName(), NetUtils#getHostNameOfIp() to name a few, do not handle IPV6 address and expect address to be of format host:port.
> Until IPV6 is formally supported, I propose disabling IPV6 for junit tests to avoid problems seen in HDFS-1891.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14506-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 22:51:43 2011
Return-Path: <common-issues-return-14506-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 082A62661
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 22:51:43 +0000 (UTC)
Received: (qmail 75078 invoked by uid 500); 5 May 2011 22:51:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74954 invoked by uid 500); 5 May 2011 22:51:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74838 invoked by uid 99); 5 May 2011 22:51:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 22:51:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 22:51:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4FCE0C35BB
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 22:51:03 +0000 (UTC)
Date: Thu, 5 May 2011 22:51:03 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <88444274.26521.1304635863323.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1362418509.26364.1304633343208.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7261) Disable IPV6 for junit tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-7261:
------------------------------------

    Status: Patch Available  (was: Open)

> Disable IPV6 for junit tests
> ----------------------------
>
>                 Key: HADOOP-7261
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7261
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7261.patch
>
>
> IPV6 addresses not handles currently in the common library methods. IPV6 can return address as "0:0:0:0:0:0:port". Some utility methods such as NetUtils#createSocketAddress(), NetUtils#normalizeHostName(), NetUtils#getHostNameOfIp() to name a few, do not handle IPV6 address and expect address to be of format host:port.
> Until IPV6 is formally supported, I propose disabling IPV6 for junit tests to avoid problems seen in HDFS-1891.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14507-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May  5 23:03:45 2011
Return-Path: <common-issues-return-14507-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 033602B6F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu,  5 May 2011 23:03:45 +0000 (UTC)
Received: (qmail 89286 invoked by uid 500); 5 May 2011 23:03:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89158 invoked by uid 500); 5 May 2011 23:03:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89132 invoked by uid 99); 5 May 2011 23:03:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 23:03:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 23:03:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 34978C3906
	for <common-issues@hadoop.apache.org>; Thu,  5 May 2011 23:03:03 +0000 (UTC)
Date: Thu, 5 May 2011 23:03:03 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <194387359.26552.1304636583212.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1362418509.26364.1304633343208.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7261) Disable IPV6 for junit tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029641#comment-13029641 ] 

Hadoop QA commented on HADOOP-7261:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478337/HADOOP-7261.patch
  against trunk revision 1099633.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/410//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/410//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/410//console

This message is automatically generated.

> Disable IPV6 for junit tests
> ----------------------------
>
>                 Key: HADOOP-7261
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7261
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7261.patch
>
>
> IPV6 addresses not handles currently in the common library methods. IPV6 can return address as "0:0:0:0:0:0:port". Some utility methods such as NetUtils#createSocketAddress(), NetUtils#normalizeHostName(), NetUtils#getHostNameOfIp() to name a few, do not handle IPV6 address and expect address to be of format host:port.
> Until IPV6 is formally supported, I propose disabling IPV6 for junit tests to avoid problems seen in HDFS-1891.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14508-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 00:08:43 2011
Return-Path: <common-issues-return-14508-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2C2EF2B31
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 00:08:43 +0000 (UTC)
Received: (qmail 56537 invoked by uid 500); 6 May 2011 00:08:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56503 invoked by uid 500); 6 May 2011 00:08:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56495 invoked by uid 99); 6 May 2011 00:08:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 00:08:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 00:08:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 589F6C3659
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 00:08:03 +0000 (UTC)
Date: Fri, 6 May 2011 00:08:03 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <843095077.26651.1304640483359.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-6255:
------------------------------

    Attachment: HADOOP-6255-mapred-trunk-1.patch
                HADOOP-6255-hdfs-trunk-1.patch
                HADOOP-6255-common-trunk-1.patch

Exclude hadoop-env.sh.example from source code, pass in os.arch environment to rpmbuild command.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14509-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 00:12:45 2011
Return-Path: <common-issues-return-14509-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EFE7D2BDB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 00:12:44 +0000 (UTC)
Received: (qmail 61057 invoked by uid 500); 6 May 2011 00:12:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61022 invoked by uid 500); 6 May 2011 00:12:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61014 invoked by uid 99); 6 May 2011 00:12:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 00:12:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 00:12:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4DE85C37B4
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 00:12:03 +0000 (UTC)
Date: Fri, 6 May 2011 00:12:03 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1243982041.26690.1304640723316.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1362418509.26364.1304633343208.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7261) Disable IPV6 for junit tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-7261:
------------------------------------

    Attachment: HADOOP-7261.1.patch

Disabled ipv6 for contrib junit tests. I am not planning to run hudson validation for this. Will commit it once I get +1 for the patch.

> Disable IPV6 for junit tests
> ----------------------------
>
>                 Key: HADOOP-7261
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7261
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7261.1.patch, HADOOP-7261.patch
>
>
> IPV6 addresses not handles currently in the common library methods. IPV6 can return address as "0:0:0:0:0:0:port". Some utility methods such as NetUtils#createSocketAddress(), NetUtils#normalizeHostName(), NetUtils#getHostNameOfIp() to name a few, do not handle IPV6 address and expect address to be of format host:port.
> Until IPV6 is formally supported, I propose disabling IPV6 for junit tests to avoid problems seen in HDFS-1891.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14510-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 00:14:47 2011
Return-Path: <common-issues-return-14510-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DB9092C65
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 00:14:46 +0000 (UTC)
Received: (qmail 63002 invoked by uid 500); 6 May 2011 00:14:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62971 invoked by uid 500); 6 May 2011 00:14:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62963 invoked by uid 99); 6 May 2011 00:14:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 00:14:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 00:14:44 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E68F3C3865
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 00:14:04 +0000 (UTC)
Date: Fri, 6 May 2011 00:14:04 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1198354370.26730.1304640844940.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029667#comment-13029667 ] 

Hadoop QA commented on HADOOP-6255:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478351/HADOOP-6255-mapred-trunk-1.patch
  against trunk revision 1099633.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 15 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/412//console

This message is automatically generated.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14511-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 00:20:43 2011
Return-Path: <common-issues-return-14511-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0E8512273
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 00:20:43 +0000 (UTC)
Received: (qmail 69539 invoked by uid 500); 6 May 2011 00:20:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69479 invoked by uid 500); 6 May 2011 00:20:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69471 invoked by uid 99); 6 May 2011 00:20:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 00:20:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 00:20:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4E8F4C3A3C
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 00:20:03 +0000 (UTC)
Date: Fri, 6 May 2011 00:20:03 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1581162493.26750.1304641203318.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1362418509.26364.1304633343208.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7261) Disable IPV6 for junit tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029670#comment-13029670 ] 

Matt Foley commented on HADOOP-7261:
------------------------------------

Looks right!  Thanks Suresh for doing this, and thanks Nigel for the suggestion.
+1

> Disable IPV6 for junit tests
> ----------------------------
>
>                 Key: HADOOP-7261
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7261
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7261.1.patch, HADOOP-7261.patch
>
>
> IPV6 addresses not handles currently in the common library methods. IPV6 can return address as "0:0:0:0:0:0:port". Some utility methods such as NetUtils#createSocketAddress(), NetUtils#normalizeHostName(), NetUtils#getHostNameOfIp() to name a few, do not handle IPV6 address and expect address to be of format host:port.
> Until IPV6 is formally supported, I propose disabling IPV6 for junit tests to avoid problems seen in HDFS-1891.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14512-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 00:24:45 2011
Return-Path: <common-issues-return-14512-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 25E9F2309
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 00:24:45 +0000 (UTC)
Received: (qmail 71389 invoked by uid 500); 6 May 2011 00:24:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71360 invoked by uid 500); 6 May 2011 00:24:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71352 invoked by uid 99); 6 May 2011 00:24:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 00:24:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 00:24:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 36D49C3B34
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 00:24:03 +0000 (UTC)
Date: Fri, 6 May 2011 00:24:03 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <461314629.26762.1304641443221.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1362418509.26364.1304633343208.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7261) Disable IPV6 for junit tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029673#comment-13029673 ] 

Hadoop QA commented on HADOOP-7261:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478352/HADOOP-7261.1.patch
  against trunk revision 1099633.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/411//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/411//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/411//console

This message is automatically generated.

> Disable IPV6 for junit tests
> ----------------------------
>
>                 Key: HADOOP-7261
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7261
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7261.1.patch, HADOOP-7261.patch
>
>
> IPV6 addresses not handles currently in the common library methods. IPV6 can return address as "0:0:0:0:0:0:port". Some utility methods such as NetUtils#createSocketAddress(), NetUtils#normalizeHostName(), NetUtils#getHostNameOfIp() to name a few, do not handle IPV6 address and expect address to be of format host:port.
> Until IPV6 is formally supported, I propose disabling IPV6 for junit tests to avoid problems seen in HDFS-1891.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14513-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 02:34:44 2011
Return-Path: <common-issues-return-14513-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E2BF2389B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 02:34:43 +0000 (UTC)
Received: (qmail 78555 invoked by uid 500); 6 May 2011 02:34:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78515 invoked by uid 500); 6 May 2011 02:34:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78495 invoked by uid 99); 6 May 2011 02:34:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 02:34:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 02:34:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 393BFC3503
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 02:34:03 +0000 (UTC)
Date: Fri, 6 May 2011 02:34:03 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1118126089.26899.1304649243231.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7257) A client side mount table to give
 per-application/per-job file system view
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029699#comment-13029699 ] 

Hudson commented on HADOOP-7257:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #573 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/573/])
    HADOOP-7257 Client side mount tables (sanjay)


> A client side mount table to give per-application/per-job file system view
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7257
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7257
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: ViewFs - javdoc2.pdf, ViewFsJavaDoc.pdf, viewFs2.patch, viewFs3.patch, viewFs4.patch, viewFs5.patch, viewfs1.patch
>
>
> This jira proposes a client side mount table to allow application-centric (or job-centric) filesystem views. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14514-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 03:26:47 2011
Return-Path: <common-issues-return-14514-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CB6D43BB9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 03:26:47 +0000 (UTC)
Received: (qmail 19897 invoked by uid 500); 6 May 2011 03:26:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19831 invoked by uid 500); 6 May 2011 03:26:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19340 invoked by uid 99); 6 May 2011 03:26:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 03:26:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 03:26:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 55EBFC301F
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 03:26:03 +0000 (UTC)
Date: Fri, 6 May 2011 03:26:03 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1301803643.26993.1304652363348.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <890288429.6872.1303934523413.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7245) FsConfig should use constants in
 CommonConfigurationKeys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7245:
--------------------------------

    Attachment: hadoop-7245-1.patch

Patch attached fixes the compilation errors in the S3 tests.

> FsConfig should use constants in CommonConfigurationKeys
> --------------------------------------------------------
>
>                 Key: HADOOP-7245
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7245
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7245.patch, hadoop-7245-1.patch
>
>
> In particular, FsConfig should use fs.defaultFS instead of the deprecated fs.default.name.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14515-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 03:28:47 2011
Return-Path: <common-issues-return-14515-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 105C03BD4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 03:28:47 +0000 (UTC)
Received: (qmail 20477 invoked by uid 500); 6 May 2011 03:28:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20331 invoked by uid 500); 6 May 2011 03:28:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20296 invoked by uid 99); 6 May 2011 03:28:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 03:28:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 03:28:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 41BB2C310B
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 03:28:03 +0000 (UTC)
Date: Fri, 6 May 2011 03:28:03 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1225189113.26999.1304652483266.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <890288429.6872.1303934523413.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7245) FsConfig should use constants in
 CommonConfigurationKeys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029712#comment-13029712 ] 

Eli Collins commented on HADOOP-7245:
-------------------------------------

Hey Tom,

The change in your patch looks good. Latest patch just updates two tests. I'll commit this if Hudson comes back OK.

Thanks,
Eli

> FsConfig should use constants in CommonConfigurationKeys
> --------------------------------------------------------
>
>                 Key: HADOOP-7245
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7245
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7245.patch, hadoop-7245-1.patch
>
>
> In particular, FsConfig should use fs.defaultFS instead of the deprecated fs.default.name.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14516-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 03:28:47 2011
Return-Path: <common-issues-return-14516-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 35B343BDF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 03:28:47 +0000 (UTC)
Received: (qmail 20673 invoked by uid 500); 6 May 2011 03:28:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20401 invoked by uid 500); 6 May 2011 03:28:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20297 invoked by uid 99); 6 May 2011 03:28:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 03:28:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 03:28:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8DCF2C3117
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 03:28:03 +0000 (UTC)
Date: Fri, 6 May 2011 03:28:03 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <225794797.27009.1304652483577.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <890288429.6872.1303934523413.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7245) FsConfig should use constants in
 CommonConfigurationKeys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7245:
--------------------------------

    Hadoop Flags: [Reviewed]
          Status: Patch Available  (was: Open)

> FsConfig should use constants in CommonConfigurationKeys
> --------------------------------------------------------
>
>                 Key: HADOOP-7245
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7245
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7245.patch, hadoop-7245-1.patch
>
>
> In particular, FsConfig should use fs.defaultFS instead of the deprecated fs.default.name.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14517-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 03:28:48 2011
Return-Path: <common-issues-return-14517-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 376AB3BF4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 03:28:48 +0000 (UTC)
Received: (qmail 20763 invoked by uid 500); 6 May 2011 03:28:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20618 invoked by uid 500); 6 May 2011 03:28:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20298 invoked by uid 99); 6 May 2011 03:28:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 03:28:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 03:28:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6123DC3110
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 03:28:03 +0000 (UTC)
Date: Fri, 6 May 2011 03:28:03 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <662764598.27003.1304652483394.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <890288429.6872.1303934523413.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7245) FsConfig should use constants in
 CommonConfigurationKeys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7245:
--------------------------------

    Status: Open  (was: Patch Available)

> FsConfig should use constants in CommonConfigurationKeys
> --------------------------------------------------------
>
>                 Key: HADOOP-7245
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7245
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7245.patch, hadoop-7245-1.patch
>
>
> In particular, FsConfig should use fs.defaultFS instead of the deprecated fs.default.name.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14518-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 03:34:49 2011
Return-Path: <common-issues-return-14518-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 78F0C30BC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 03:34:49 +0000 (UTC)
Received: (qmail 23441 invoked by uid 500); 6 May 2011 03:34:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23420 invoked by uid 500); 6 May 2011 03:34:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23412 invoked by uid 99); 6 May 2011 03:34:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 03:34:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 03:34:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1D35FC3228
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 03:34:03 +0000 (UTC)
Date: Fri, 6 May 2011 03:34:03 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1008149673.27015.1304652843116.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1716213266.4485.1303865103489.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7244) Documentation change for updated
 configuration keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7244:
--------------------------------

    Attachment: hadoop-7244-1.patch

The change to hod_cluster should go the other way eg from blocksize -> block.size. Attached patch fixes that.  Otherwise looks great.

> Documentation change for updated configuration keys
> ---------------------------------------------------
>
>                 Key: HADOOP-7244
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7244
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7244.patch, hadoop-7244-1.patch
>
>
> Common counterpart of HDFS-671.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14519-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 03:43:49 2011
Return-Path: <common-issues-return-14519-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 25D893153
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 03:43:49 +0000 (UTC)
Received: (qmail 29581 invoked by uid 500); 6 May 2011 03:43:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29421 invoked by uid 500); 6 May 2011 03:43:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29402 invoked by uid 99); 6 May 2011 03:43:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 03:43:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 03:43:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2712AC3436
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 03:43:03 +0000 (UTC)
Date: Fri, 6 May 2011 03:43:03 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <542808712.27029.1304653383156.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <890288429.6872.1303934523413.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7245) FsConfig should use constants in
 CommonConfigurationKeys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029718#comment-13029718 ] 

Hadoop QA commented on HADOOP-7245:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478362/hadoop-7245-1.patch
  against trunk revision 1100026.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/413//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/413//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/413//console

This message is automatically generated.

> FsConfig should use constants in CommonConfigurationKeys
> --------------------------------------------------------
>
>                 Key: HADOOP-7245
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7245
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7245.patch, hadoop-7245-1.patch
>
>
> In particular, FsConfig should use fs.defaultFS instead of the deprecated fs.default.name.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14520-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 03:55:46 2011
Return-Path: <common-issues-return-14520-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 468453610
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 03:55:46 +0000 (UTC)
Received: (qmail 37163 invoked by uid 500); 6 May 2011 03:55:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37127 invoked by uid 500); 6 May 2011 03:55:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37100 invoked by uid 99); 6 May 2011 03:55:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 03:55:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 03:55:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4CF6EC36AC
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 03:55:03 +0000 (UTC)
Date: Fri, 6 May 2011 03:55:03 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <292297660.27039.1304654103311.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1716213266.4485.1303865103489.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7244) Documentation change for updated
 configuration keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029721#comment-13029721 ] 

Hadoop QA commented on HADOOP-7244:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478363/hadoop-7244-1.patch
  against trunk revision 1100026.

    +1 @author.  The patch does not contain any @author tags.

    +0 tests included.  The patch appears to be a documentation patch that doesn't require tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/414//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/414//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/414//console

This message is automatically generated.

> Documentation change for updated configuration keys
> ---------------------------------------------------
>
>                 Key: HADOOP-7244
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7244
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7244.patch, hadoop-7244-1.patch
>
>
> Common counterpart of HDFS-671.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14521-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 04:23:44 2011
Return-Path: <common-issues-return-14521-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8FBB73084
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 04:23:44 +0000 (UTC)
Received: (qmail 57050 invoked by uid 500); 6 May 2011 04:23:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57019 invoked by uid 500); 6 May 2011 04:23:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57003 invoked by uid 99); 6 May 2011 04:23:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 04:23:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 04:23:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 300B4C3CBB
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 04:23:03 +0000 (UTC)
Date: Fri, 6 May 2011 04:23:03 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2096910461.27084.1304655783193.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <890288429.6872.1303934523413.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7245) FsConfig should use constants in
 CommonConfigurationKeys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029730#comment-13029730 ] 

Eli Collins commented on HADOOP-7245:
-------------------------------------

+1

> FsConfig should use constants in CommonConfigurationKeys
> --------------------------------------------------------
>
>                 Key: HADOOP-7245
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7245
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7245.patch, hadoop-7245-1.patch
>
>
> In particular, FsConfig should use fs.defaultFS instead of the deprecated fs.default.name.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14522-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 04:23:45 2011
Return-Path: <common-issues-return-14522-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 91795308D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 04:23:44 +0000 (UTC)
Received: (qmail 57263 invoked by uid 500); 6 May 2011 04:23:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57022 invoked by uid 500); 6 May 2011 04:23:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57002 invoked by uid 99); 6 May 2011 04:23:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 04:23:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 04:23:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 228D8C3CB9
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 04:23:03 +0000 (UTC)
Date: Fri, 6 May 2011 04:23:03 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <803306445.27082.1304655783138.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1716213266.4485.1303865103489.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7244) Documentation change for updated
 configuration keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029729#comment-13029729 ] 

Eli Collins commented on HADOOP-7244:
-------------------------------------

+1

> Documentation change for updated configuration keys
> ---------------------------------------------------
>
>                 Key: HADOOP-7244
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7244
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7244.patch, hadoop-7244-1.patch
>
>
> Common counterpart of HDFS-671.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14523-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 04:31:44 2011
Return-Path: <common-issues-return-14523-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3CE8638A1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 04:31:44 +0000 (UTC)
Received: (qmail 61624 invoked by uid 500); 6 May 2011 04:31:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61410 invoked by uid 500); 6 May 2011 04:31:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61401 invoked by uid 99); 6 May 2011 04:31:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 04:31:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 04:31:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 399AEC3FB2
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 04:31:03 +0000 (UTC)
Date: Fri, 6 May 2011 04:31:03 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <577086366.27110.1304656263232.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1716213266.4485.1303865103489.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7244) Documentation change for updated
 configuration keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7244:
--------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I've committed this to trunk and merged to 22. Thanks Tom!

> Documentation change for updated configuration keys
> ---------------------------------------------------
>
>                 Key: HADOOP-7244
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7244
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7244.patch, hadoop-7244-1.patch
>
>
> Common counterpart of HDFS-671.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14524-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 04:31:44 2011
Return-Path: <common-issues-return-14524-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D1CC938B6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 04:31:44 +0000 (UTC)
Received: (qmail 61660 invoked by uid 500); 6 May 2011 04:31:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61421 invoked by uid 500); 6 May 2011 04:31:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61407 invoked by uid 99); 6 May 2011 04:31:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 04:31:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 04:31:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5B3DEC3FBB
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 04:31:03 +0000 (UTC)
Date: Fri, 6 May 2011 04:31:03 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1926558989.27114.1304656263370.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <890288429.6872.1303934523413.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7245) FsConfig should use constants in
 CommonConfigurationKeys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7245:
--------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I've committed this to trunk and merged to 22. Thanks Tom!

> FsConfig should use constants in CommonConfigurationKeys
> --------------------------------------------------------
>
>                 Key: HADOOP-7245
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7245
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7245.patch, hadoop-7245-1.patch
>
>
> In particular, FsConfig should use fs.defaultFS instead of the deprecated fs.default.name.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14525-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 04:45:47 2011
Return-Path: <common-issues-return-14525-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 884863B86
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 04:45:47 +0000 (UTC)
Received: (qmail 71473 invoked by uid 500); 6 May 2011 04:45:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71444 invoked by uid 500); 6 May 2011 04:45:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71428 invoked by uid 99); 6 May 2011 04:45:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 04:45:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 04:45:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 321A8C3330
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 04:45:03 +0000 (UTC)
Date: Fri, 6 May 2011 04:45:03 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1400511441.27145.1304657103202.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <890288429.6872.1303934523413.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7245) FsConfig should use constants in
 CommonConfigurationKeys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029745#comment-13029745 ] 

Hudson commented on HADOOP-7245:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #574 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/574/])
    HADOOP-7245. FsConfig should use constants in CommonConfigurationKeys. Contributed by Tom White


> FsConfig should use constants in CommonConfigurationKeys
> --------------------------------------------------------
>
>                 Key: HADOOP-7245
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7245
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7245.patch, hadoop-7245-1.patch
>
>
> In particular, FsConfig should use fs.defaultFS instead of the deprecated fs.default.name.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14526-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 04:45:48 2011
Return-Path: <common-issues-return-14526-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 225EC3BB1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 04:45:48 +0000 (UTC)
Received: (qmail 71770 invoked by uid 500); 6 May 2011 04:45:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71446 invoked by uid 500); 6 May 2011 04:45:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71427 invoked by uid 99); 6 May 2011 04:45:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 04:45:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 04:45:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1F551C332C
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 04:45:03 +0000 (UTC)
Date: Fri, 6 May 2011 04:45:03 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1606090802.27143.1304657103125.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1716213266.4485.1303865103489.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7244) Documentation change for updated
 configuration keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029744#comment-13029744 ] 

Hudson commented on HADOOP-7244:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #574 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/574/])
    HADOOP-7244. Documentation change for updated configuration keys. Contributed by Tom White


> Documentation change for updated configuration keys
> ---------------------------------------------------
>
>                 Key: HADOOP-7244
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7244
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7244.patch, hadoop-7244-1.patch
>
>
> Common counterpart of HDFS-671.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14527-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 05:10:47 2011
Return-Path: <common-issues-return-14527-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6077C3165
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 05:10:47 +0000 (UTC)
Received: (qmail 82914 invoked by uid 500); 6 May 2011 05:10:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82710 invoked by uid 500); 6 May 2011 05:10:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82627 invoked by uid 99); 6 May 2011 05:10:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 05:10:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 05:10:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2C8F8C3956
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 05:10:03 +0000 (UTC)
Date: Fri, 6 May 2011 05:10:03 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1031760231.27176.1304658603179.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1594670430.14027.1299867839776.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7183) WritableComparator.get should not
 cache comparator objects
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7183:
--------------------------------

       Resolution: Fixed
    Fix Version/s:     (was: 0.20.3)
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I've committed this to trunk, branch 22, and branch 21. Thanks Tom!

> WritableComparator.get should not cache comparator objects
> ----------------------------------------------------------
>
>                 Key: HADOOP-7183
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7183
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.1, 0.22.0
>
>         Attachments: HADOOP-7183.patch, HADOOP-7183.patch
>
>
> HADOOP-6881 modified WritableComparator.get such that the constructed WritableComparator gets saved back into the static map. This is fine for stateless comparators, but some comparators have per-instance state, and thus this becomes thread-unsafe and causes errors in the shuffle where multiple threads are doing comparisons. An example of a Comparator with per-instance state is WritableComparator itself.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14528-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 05:24:47 2011
Return-Path: <common-issues-return-14528-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A86A636AD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 05:24:47 +0000 (UTC)
Received: (qmail 91043 invoked by uid 500); 6 May 2011 05:24:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90822 invoked by uid 500); 6 May 2011 05:24:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90801 invoked by uid 99); 6 May 2011 05:24:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 05:24:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 05:24:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 52CDEC3CBF
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 05:24:04 +0000 (UTC)
Date: Fri, 6 May 2011 05:24:04 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1941402147.27202.1304659444335.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1594670430.14027.1299867839776.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7183) WritableComparator.get should not
 cache comparator objects
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029757#comment-13029757 ] 

Hudson commented on HADOOP-7183:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #575 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/575/])
    HADOOP-7183. WritableComparator.get should not cache comparator objects. Contributed by Tom White


> WritableComparator.get should not cache comparator objects
> ----------------------------------------------------------
>
>                 Key: HADOOP-7183
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7183
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.1, 0.22.0
>
>         Attachments: HADOOP-7183.patch, HADOOP-7183.patch
>
>
> HADOOP-6881 modified WritableComparator.get such that the constructed WritableComparator gets saved back into the static map. This is fine for stateless comparators, but some comparators have per-instance state, and thus this becomes thread-unsafe and causes errors in the shuffle where multiple threads are doing comparisons. An example of a Comparator with per-instance state is WritableComparator itself.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14529-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 06:24:50 2011
Return-Path: <common-issues-return-14529-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0B12C3F35
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 06:24:50 +0000 (UTC)
Received: (qmail 37221 invoked by uid 500); 6 May 2011 06:24:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37116 invoked by uid 500); 6 May 2011 06:24:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36889 invoked by uid 99); 6 May 2011 06:24:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 06:24:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 06:24:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3913BC3DEF
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 06:24:03 +0000 (UTC)
Date: Fri, 6 May 2011 06:24:03 +0000 (UTC)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1327657029.27279.1304663043230.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7262) Update CS docs with better example
 configs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Update CS docs with better example configs
------------------------------------------

                 Key: HADOOP-7262
                 URL: https://issues.apache.org/jira/browse/HADOOP-7262
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Arun C Murthy
            Assignee: Arun C Murthy
            Priority: Minor
             Fix For: 0.20.204.0


It will be nice to enhance CS docs with real-world example configs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14530-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 07:22:51 2011
Return-Path: <common-issues-return-14530-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 768D137F7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 07:22:51 +0000 (UTC)
Received: (qmail 99603 invoked by uid 500); 6 May 2011 07:22:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99395 invoked by uid 500); 6 May 2011 07:22:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99387 invoked by uid 99); 6 May 2011 07:22:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 07:22:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 07:22:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 27432C35DE
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 07:22:03 +0000 (UTC)
Date: Fri, 6 May 2011 07:22:03 +0000 (UTC)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1433332720.27441.1304666523157.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1327657029.27279.1304663043230.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7262) Update CS docs with better example
 configs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arun C Murthy updated HADOOP-7262:
----------------------------------

    Attachment: capacity_scheduler.pdf
                HADOOP-7262.patch

Patch with a real-world example config and also the generated capacity-scheduler.pdf for review.

> Update CS docs with better example configs
> ------------------------------------------
>
>                 Key: HADOOP-7262
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7262
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>            Priority: Minor
>             Fix For: 0.20.204.0
>
>         Attachments: HADOOP-7262.patch, capacity_scheduler.pdf
>
>
> It will be nice to enhance CS docs with real-world example configs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14531-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 07:30:46 2011
Return-Path: <common-issues-return-14531-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 557413C57
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 07:30:46 +0000 (UTC)
Received: (qmail 8588 invoked by uid 500); 6 May 2011 07:30:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8514 invoked by uid 500); 6 May 2011 07:30:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8340 invoked by uid 99); 6 May 2011 07:30:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 07:30:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 07:30:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7B164C3A34
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 07:30:05 +0000 (UTC)
Date: Fri, 6 May 2011 07:30:05 +0000 (UTC)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <431084003.27479.1304667005500.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6919) Metrics2: metrics framework
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arun C Murthy updated HADOOP-6919:
----------------------------------

      Resolution: Fixed
    Release Note: New metrics2 framework for Hadoop.
          Status: Resolved  (was: Patch Available)

I just committed this. Thanks Luke!

> Metrics2: metrics framework
> ---------------------------
>
>                 Key: HADOOP-6919
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6919
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: metrics
>    Affects Versions: 0.22.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-6919-metrics2-framework-v6.patch, hadoop-6919-trunk-v1.patch
>
>
> The jira tracks the new metrics framework only changes, i.e., it doesn't track the instrumentation changes (and compatibility issues) to existing code.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14532-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 07:47:46 2011
Return-Path: <common-issues-return-14532-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4B1A83358
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 07:47:46 +0000 (UTC)
Received: (qmail 34769 invoked by uid 500); 6 May 2011 07:47:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34741 invoked by uid 500); 6 May 2011 07:47:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34722 invoked by uid 99); 6 May 2011 07:47:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 07:47:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 07:47:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 72C8FC31F7
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 07:47:04 +0000 (UTC)
Date: Fri, 6 May 2011 07:47:04 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <104920240.27532.1304668024466.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6919) Metrics2: metrics framework
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029811#comment-13029811 ] 

Hudson commented on HADOOP-6919:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #576 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/576/])
    HADOOP-6919. New metrics2 framework. Contributed by Luke Lu.


> Metrics2: metrics framework
> ---------------------------
>
>                 Key: HADOOP-6919
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6919
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: metrics
>    Affects Versions: 0.22.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-6919-metrics2-framework-v6.patch, hadoop-6919-trunk-v1.patch
>
>
> The jira tracks the new metrics framework only changes, i.e., it doesn't track the instrumentation changes (and compatibility issues) to existing code.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14533-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 09:44:43 2011
Return-Path: <common-issues-return-14533-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7D23C37F5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 09:44:43 +0000 (UTC)
Received: (qmail 97106 invoked by uid 500); 6 May 2011 09:44:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97007 invoked by uid 500); 6 May 2011 09:44:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96742 invoked by uid 99); 6 May 2011 09:44:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 09:44:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 09:44:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6C44AC3124
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 09:44:03 +0000 (UTC)
Date: Fri, 6 May 2011 09:44:03 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <328220388.27751.1304675043439.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1594670430.14027.1299867839776.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7183) WritableComparator.get should not
 cache comparator objects
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029867#comment-13029867 ] 

Hudson commented on HADOOP-7183:
--------------------------------

Integrated in Hadoop-Common-22-branch #44 (See [https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/44/])
    HADOOP-7183. svn merge -c 1100056 from trunk


> WritableComparator.get should not cache comparator objects
> ----------------------------------------------------------
>
>                 Key: HADOOP-7183
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7183
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.1, 0.22.0
>
>         Attachments: HADOOP-7183.patch, HADOOP-7183.patch
>
>
> HADOOP-6881 modified WritableComparator.get such that the constructed WritableComparator gets saved back into the static map. This is fine for stateless comparators, but some comparators have per-instance state, and thus this becomes thread-unsafe and causes errors in the shuffle where multiple threads are doing comparisons. An example of a Comparator with per-instance state is WritableComparator itself.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14534-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 09:44:45 2011
Return-Path: <common-issues-return-14534-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 442063803
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 09:44:45 +0000 (UTC)
Received: (qmail 97433 invoked by uid 500); 6 May 2011 09:44:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97393 invoked by uid 500); 6 May 2011 09:44:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97385 invoked by uid 99); 6 May 2011 09:44:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 09:44:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 09:44:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3CF5BC311C
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 09:44:03 +0000 (UTC)
Date: Fri, 6 May 2011 09:44:03 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <906537330.27745.1304675043246.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1716213266.4485.1303865103489.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7244) Documentation change for updated
 configuration keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029865#comment-13029865 ] 

Hudson commented on HADOOP-7244:
--------------------------------

Integrated in Hadoop-Common-22-branch #44 (See [https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/44/])
    HADOOP-7244. svn merge -c 1100048 from trunk


> Documentation change for updated configuration keys
> ---------------------------------------------------
>
>                 Key: HADOOP-7244
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7244
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7244.patch, hadoop-7244-1.patch
>
>
> Common counterpart of HDFS-671.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14535-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 09:44:45 2011
Return-Path: <common-issues-return-14535-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ADB133810
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 09:44:45 +0000 (UTC)
Received: (qmail 97662 invoked by uid 500); 6 May 2011 09:44:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97634 invoked by uid 500); 6 May 2011 09:44:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97618 invoked by uid 99); 6 May 2011 09:44:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 09:44:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 09:44:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 55901C3122
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 09:44:03 +0000 (UTC)
Date: Fri, 6 May 2011 09:44:03 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2143256285.27749.1304675043347.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <890288429.6872.1303934523413.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7245) FsConfig should use constants in
 CommonConfigurationKeys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029866#comment-13029866 ] 

Hudson commented on HADOOP-7245:
--------------------------------

Integrated in Hadoop-Common-22-branch #44 (See [https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/44/])
    HADOOP-7245. svn merge -c 1100044 from trunk


> FsConfig should use constants in CommonConfigurationKeys
> --------------------------------------------------------
>
>                 Key: HADOOP-7245
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7245
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7245.patch, hadoop-7245-1.patch
>
>
> In particular, FsConfig should use fs.defaultFS instead of the deprecated fs.default.name.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14536-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 11:16:43 2011
Return-Path: <common-issues-return-14536-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C79DB3188
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 11:16:43 +0000 (UTC)
Received: (qmail 6559 invoked by uid 500); 6 May 2011 11:16:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6536 invoked by uid 500); 6 May 2011 11:16:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6514 invoked by uid 99); 6 May 2011 11:16:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 11:16:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 11:16:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E7EDBC4E6A
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 11:16:03 +0000 (UTC)
Date: Fri, 6 May 2011 11:16:03 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1225991363.27890.1304680563946.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1716213266.4485.1303865103489.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7244) Documentation change for updated
 configuration keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029888#comment-13029888 ] 

Hudson commented on HADOOP-7244:
--------------------------------

Integrated in Hadoop-Common-trunk #680 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/680/])
    HADOOP-7244. Documentation change for updated configuration keys. Contributed by Tom White


> Documentation change for updated configuration keys
> ---------------------------------------------------
>
>                 Key: HADOOP-7244
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7244
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7244.patch, hadoop-7244-1.patch
>
>
> Common counterpart of HDFS-671.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14537-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 11:16:44 2011
Return-Path: <common-issues-return-14537-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 42173319C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 11:16:44 +0000 (UTC)
Received: (qmail 6821 invoked by uid 500); 6 May 2011 11:16:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6780 invoked by uid 500); 6 May 2011 11:16:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6772 invoked by uid 99); 6 May 2011 11:16:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 11:16:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 11:16:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D04A7C4E81
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 11:16:04 +0000 (UTC)
Date: Fri, 6 May 2011 11:16:04 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <324717148.27913.1304680564850.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7257) A client side mount table to give
 per-application/per-job file system view
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029890#comment-13029890 ] 

Hudson commented on HADOOP-7257:
--------------------------------

Integrated in Hadoop-Common-trunk #680 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/680/])
    HADOOP-7257 Client side mount tables (sanjay)


> A client side mount table to give per-application/per-job file system view
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7257
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7257
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: ViewFs - javdoc2.pdf, ViewFsJavaDoc.pdf, viewFs2.patch, viewFs3.patch, viewFs4.patch, viewFs5.patch, viewfs1.patch
>
>
> This jira proposes a client side mount table to allow application-centric (or job-centric) filesystem views. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14538-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 11:16:44 2011
Return-Path: <common-issues-return-14538-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9B61B31B2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 11:16:44 +0000 (UTC)
Received: (qmail 7076 invoked by uid 500); 6 May 2011 11:16:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7027 invoked by uid 500); 6 May 2011 11:16:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7019 invoked by uid 99); 6 May 2011 11:16:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 11:16:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 11:16:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 29842C4E8B
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 11:16:05 +0000 (UTC)
Date: Fri, 6 May 2011 11:16:05 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <914326215.27923.1304680565166.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1594670430.14027.1299867839776.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7183) WritableComparator.get should not
 cache comparator objects
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029891#comment-13029891 ] 

Hudson commented on HADOOP-7183:
--------------------------------

Integrated in Hadoop-Common-trunk #680 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/680/])
    HADOOP-7183. WritableComparator.get should not cache comparator objects. Contributed by Tom White


> WritableComparator.get should not cache comparator objects
> ----------------------------------------------------------
>
>                 Key: HADOOP-7183
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7183
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.1, 0.22.0
>
>         Attachments: HADOOP-7183.patch, HADOOP-7183.patch
>
>
> HADOOP-6881 modified WritableComparator.get such that the constructed WritableComparator gets saved back into the static map. This is fine for stateless comparators, but some comparators have per-instance state, and thus this becomes thread-unsafe and causes errors in the shuffle where multiple threads are doing comparisons. An example of a Comparator with per-instance state is WritableComparator itself.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14539-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 11:16:46 2011
Return-Path: <common-issues-return-14539-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2EFE731E5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 11:16:46 +0000 (UTC)
Received: (qmail 7591 invoked by uid 500); 6 May 2011 11:16:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7539 invoked by uid 500); 6 May 2011 11:16:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7481 invoked by uid 99); 6 May 2011 11:16:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 11:16:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 11:16:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D3BA2C4E68
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 11:16:03 +0000 (UTC)
Date: Fri, 6 May 2011 11:16:03 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1561029591.27888.1304680563864.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6919) Metrics2: metrics framework
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029887#comment-13029887 ] 

Hudson commented on HADOOP-6919:
--------------------------------

Integrated in Hadoop-Common-trunk #680 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/680/])
    HADOOP-6919. New metrics2 framework. Contributed by Luke Lu.


> Metrics2: metrics framework
> ---------------------------
>
>                 Key: HADOOP-6919
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6919
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: metrics
>    Affects Versions: 0.22.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-6919-metrics2-framework-v6.patch, hadoop-6919-trunk-v1.patch
>
>
> The jira tracks the new metrics framework only changes, i.e., it doesn't track the instrumentation changes (and compatibility issues) to existing code.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14540-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 11:16:46 2011
Return-Path: <common-issues-return-14540-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 32A4F31F6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 11:16:46 +0000 (UTC)
Received: (qmail 7623 invoked by uid 500); 6 May 2011 11:16:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7541 invoked by uid 500); 6 May 2011 11:16:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7492 invoked by uid 99); 6 May 2011 11:16:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 11:16:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 11:16:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 053D6C4E6C
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 11:16:04 +0000 (UTC)
Date: Fri, 6 May 2011 11:16:04 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <368606163.27892.1304680564018.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <890288429.6872.1303934523413.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7245) FsConfig should use constants in
 CommonConfigurationKeys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029889#comment-13029889 ] 

Hudson commented on HADOOP-7245:
--------------------------------

Integrated in Hadoop-Common-trunk #680 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/680/])
    HADOOP-7245. FsConfig should use constants in CommonConfigurationKeys. Contributed by Tom White


> FsConfig should use constants in CommonConfigurationKeys
> --------------------------------------------------------
>
>                 Key: HADOOP-7245
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7245
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7245.patch, hadoop-7245-1.patch
>
>
> In particular, FsConfig should use fs.defaultFS instead of the deprecated fs.default.name.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14541-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 16:17:45 2011
Return-Path: <common-issues-return-14541-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D257D376A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 16:17:45 +0000 (UTC)
Received: (qmail 18610 invoked by uid 500); 6 May 2011 16:17:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18571 invoked by uid 500); 6 May 2011 16:17:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18563 invoked by uid 99); 6 May 2011 16:17:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 16:17:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 16:17:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D2B72C4DFE
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 16:17:03 +0000 (UTC)
Date: Fri, 6 May 2011 16:17:03 +0000 (UTC)
From: "Thomas Graves (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1661218055.28421.1304698623859.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <285327525.9499.1304014323342.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7248) Have a way to automatically update
 Eclipse .classpath file when new libs are added to the classpath through
 Ivy for 0.20-* based sources
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Graves updated HADOOP-7248:
----------------------------------

    Attachment: HADOOP-7248-branch-20-security.patch

patch for branch-0.20-security

> Have a way to automatically update Eclipse .classpath file when new libs are added to the classpath through Ivy for 0.20-* based sources
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7248
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7248
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.3
>            Reporter: Konstantin Boudnik
>            Assignee: Thomas Graves
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-7248-branch-20-security.patch
>
>
> Backport HADOOP-6407 into 0.20 based source trees

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14542-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 16:19:45 2011
Return-Path: <common-issues-return-14542-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5ADB23036
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 16:19:45 +0000 (UTC)
Received: (qmail 23902 invoked by uid 500); 6 May 2011 16:19:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23867 invoked by uid 500); 6 May 2011 16:19:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23856 invoked by uid 99); 6 May 2011 16:19:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 16:19:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 16:19:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3BDE4C4EF3
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 16:19:03 +0000 (UTC)
Date: Fri, 6 May 2011 16:19:03 +0000 (UTC)
From: "Thomas Graves (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1815229687.28429.1304698743241.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <285327525.9499.1304014323342.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7248) Have a way to automatically update
 Eclipse .classpath file when new libs are added to the classpath through
 Ivy for 0.20-* based sources
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029999#comment-13029999 ] 

Thomas Graves commented on HADOOP-7248:
---------------------------------------

     [exec] -1 overall.    
     [exec]     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec]     [exec]     +1 tests included.  The patch appears to include 19 new or modified tests.
     [exec]     [exec]     -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.
     [exec]     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec]     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec]     [exec]     -1 Eclipse classpath. The patch causes the Eclipse classpath to differ from the contents of the lib directories.

     [exec]     

The javadoc and eclipse classpath were failing before this patch.  Once this is commited the eclipse classpath check should start showing +1 again.

I manually tested this:
- check out svn tree
- apply patch
- svn rm the empty files
- build eclipse target either from within eclipse or with ant eclipse
- refresh eclipse
- build eclipse project
- everything resolves and you can run unit tests

> Have a way to automatically update Eclipse .classpath file when new libs are added to the classpath through Ivy for 0.20-* based sources
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7248
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7248
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.3
>            Reporter: Konstantin Boudnik
>            Assignee: Thomas Graves
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-7248-branch-20-security.patch
>
>
> Backport HADOOP-6407 into 0.20 based source trees

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14543-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 16:21:44 2011
Return-Path: <common-issues-return-14543-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 11CD6307B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 16:21:44 +0000 (UTC)
Received: (qmail 24760 invoked by uid 500); 6 May 2011 16:21:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24575 invoked by uid 500); 6 May 2011 16:21:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24535 invoked by uid 99); 6 May 2011 16:21:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 16:21:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 16:21:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 07DF9C4FFD
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 16:21:04 +0000 (UTC)
Date: Fri, 6 May 2011 16:21:04 +0000 (UTC)
From: "Thomas Graves (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1371496820.28441.1304698864029.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <285327525.9499.1304014323342.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7248) Have a way to automatically update
 Eclipse .classpath file when new libs are added to the classpath through
 Ivy for 0.20-* based sources
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Graves updated HADOOP-7248:
----------------------------------

    Fix Version/s:     (was: 0.21.0)
                   0.20.205.0
           Status: Patch Available  (was: Open)

> Have a way to automatically update Eclipse .classpath file when new libs are added to the classpath through Ivy for 0.20-* based sources
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7248
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7248
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.3
>            Reporter: Konstantin Boudnik
>            Assignee: Thomas Graves
>            Priority: Minor
>             Fix For: 0.20.205.0
>
>         Attachments: HADOOP-7248-branch-20-security.patch
>
>
> Backport HADOOP-6407 into 0.20 based source trees

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14544-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 16:23:45 2011
Return-Path: <common-issues-return-14544-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 53D783151
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 16:23:45 +0000 (UTC)
Received: (qmail 27048 invoked by uid 500); 6 May 2011 16:23:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27020 invoked by uid 500); 6 May 2011 16:23:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27012 invoked by uid 99); 6 May 2011 16:23:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 16:23:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 16:23:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 97315C409C
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 16:23:03 +0000 (UTC)
Date: Fri, 6 May 2011 16:23:03 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1489376492.28451.1304698983260.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <285327525.9499.1304014323342.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7248) Have a way to automatically update
 Eclipse .classpath file when new libs are added to the classpath through
 Ivy for 0.20-* based sources
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030003#comment-13030003 ] 

Hadoop QA commented on HADOOP-7248:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478446/HADOOP-7248-branch-20-security.patch
  against trunk revision 1100113.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 19 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/415//console

This message is automatically generated.

> Have a way to automatically update Eclipse .classpath file when new libs are added to the classpath through Ivy for 0.20-* based sources
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7248
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7248
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.3
>            Reporter: Konstantin Boudnik
>            Assignee: Thomas Graves
>            Priority: Minor
>             Fix For: 0.20.205.0
>
>         Attachments: HADOOP-7248-branch-20-security.patch
>
>
> Backport HADOOP-6407 into 0.20 based source trees

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14545-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 16:28:43 2011
Return-Path: <common-issues-return-14545-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EC3D232E2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 16:28:42 +0000 (UTC)
Received: (qmail 33772 invoked by uid 500); 6 May 2011 16:28:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33739 invoked by uid 500); 6 May 2011 16:28:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33731 invoked by uid 99); 6 May 2011 16:28:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 16:28:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 16:28:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 43C5DC421C
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 16:28:03 +0000 (UTC)
Date: Fri, 6 May 2011 16:28:03 +0000 (UTC)
From: "Thomas Graves (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1430114315.28461.1304699283274.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <285327525.9499.1304014323342.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7248) Have a way to automatically update
 Eclipse .classpath file when new libs are added to the classpath through
 Ivy for 0.20-* based sources
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030004#comment-13030004 ] 

Thomas Graves commented on HADOOP-7248:
---------------------------------------

build failed because patch is for branch-0.20-security not trunk.

> Have a way to automatically update Eclipse .classpath file when new libs are added to the classpath through Ivy for 0.20-* based sources
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7248
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7248
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.3
>            Reporter: Konstantin Boudnik
>            Assignee: Thomas Graves
>            Priority: Minor
>             Fix For: 0.20.205.0
>
>         Attachments: HADOOP-7248-branch-20-security.patch
>
>
> Backport HADOOP-6407 into 0.20 based source trees

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14546-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 16:32:45 2011
Return-Path: <common-issues-return-14546-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 692963149
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 16:32:45 +0000 (UTC)
Received: (qmail 40521 invoked by uid 500); 6 May 2011 16:32:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40466 invoked by uid 500); 6 May 2011 16:32:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40378 invoked by uid 99); 6 May 2011 16:32:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 16:32:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 16:32:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 76318C4415
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 16:32:03 +0000 (UTC)
Date: Fri, 6 May 2011 16:32:03 +0000 (UTC)
From: "Greg Wittel (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1962550673.28477.1304699523480.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7263) Hadoop Streaming (StreamJob) does
 not delete temporary job/package jar
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Hadoop Streaming (StreamJob) does not delete temporary job/package jar
----------------------------------------------------------------------

                 Key: HADOOP-7263
                 URL: https://issues.apache.org/jira/browse/HADOOP-7263
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.21.0, 0.20.2, 0.20.3, 0.20.4
            Reporter: Greg Wittel
            Priority: Minor


The streaming job driver (org.apache.hadoop.streaming.StreamJob) does not delete the temporary JAR file it generates after a job completes.  Without the fix, /var/tmp fills up  with streaming job jars until they get wiped.

The jar name is stored in the class variable 'jar_'.  The JAR is generated in 'packageJobJar()' and the name stored in jar_.

Fix:  run()/submitAndMonitorJob() should clean up the jar_ file when done.  Or the JAR could be generatd as a tempfile and cleaned up automatically.



--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14547-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 16:46:43 2011
Return-Path: <common-issues-return-14547-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EA5D43722
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 16:46:42 +0000 (UTC)
Received: (qmail 55705 invoked by uid 500); 6 May 2011 16:46:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55669 invoked by uid 500); 6 May 2011 16:46:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55661 invoked by uid 99); 6 May 2011 16:46:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 16:46:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 16:46:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 433B3C4912
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 16:46:03 +0000 (UTC)
Date: Fri, 6 May 2011 16:46:03 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <511078625.28508.1304700363272.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1362418509.26364.1304633343208.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7261) Disable IPV6 for junit tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-7261:
------------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I committed the patch.

> Disable IPV6 for junit tests
> ----------------------------
>
>                 Key: HADOOP-7261
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7261
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7261.1.patch, HADOOP-7261.patch
>
>
> IPV6 addresses not handles currently in the common library methods. IPV6 can return address as "0:0:0:0:0:0:port". Some utility methods such as NetUtils#createSocketAddress(), NetUtils#normalizeHostName(), NetUtils#getHostNameOfIp() to name a few, do not handle IPV6 address and expect address to be of format host:port.
> Until IPV6 is formally supported, I propose disabling IPV6 for junit tests to avoid problems seen in HDFS-1891.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14548-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 18:24:43 2011
Return-Path: <common-issues-return-14548-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 819E93E52
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 18:24:43 +0000 (UTC)
Received: (qmail 93825 invoked by uid 500); 6 May 2011 18:24:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93798 invoked by uid 500); 6 May 2011 18:24:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93788 invoked by uid 99); 6 May 2011 18:24:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 18:24:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 18:24:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BB681C4905
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 18:24:03 +0000 (UTC)
Date: Fri, 6 May 2011 18:24:03 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1330832899.28720.1304706243764.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7264) Bump avro version to at least 1.4.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Bump avro version to at least 1.4.1
-----------------------------------

                 Key: HADOOP-7264
                 URL: https://issues.apache.org/jira/browse/HADOOP-7264
             Project: Hadoop Common
          Issue Type: Improvement
          Components: io
    Affects Versions: 0.21.0
            Reporter: Luke Lu
             Fix For: 0.23.0


Needed by mapreduce 2.0 avro support. Maybe we could jump to Avro 1.5. There is incompatible API changes from 1.3x to 1.4x (Utf8 to CharSequence in user facing APIs) not sure about 1.5x though.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14549-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 18:38:45 2011
Return-Path: <common-issues-return-14549-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 424833512
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 18:38:45 +0000 (UTC)
Received: (qmail 16434 invoked by uid 500); 6 May 2011 18:38:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16400 invoked by uid 500); 6 May 2011 18:38:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16392 invoked by uid 99); 6 May 2011 18:38:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 18:38:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 18:38:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 67C98C4DE1
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 18:38:03 +0000 (UTC)
Date: Fri, 6 May 2011 18:38:03 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <971643595.28765.1304707083421.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1330832899.28720.1304706243764.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7264) Bump avro version to at least
 1.4.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030077#comment-13030077 ] 

Todd Lipcon commented on HADOOP-7264:
-------------------------------------

Is it worth considering maven shade or jarjar for avro? Or in MR2 do we no longer have the issue that the daemon formerly known as the TaskTracker leaks its jar dependencies onto the user classpath?

> Bump avro version to at least 1.4.1
> -----------------------------------
>
>                 Key: HADOOP-7264
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7264
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Luke Lu
>              Labels: avro
>             Fix For: 0.23.0
>
>
> Needed by mapreduce 2.0 avro support. Maybe we could jump to Avro 1.5. There is incompatible API changes from 1.3x to 1.4x (Utf8 to CharSequence in user facing APIs) not sure about 1.5x though.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14550-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 18:56:43 2011
Return-Path: <common-issues-return-14550-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0CA543C02
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 18:56:43 +0000 (UTC)
Received: (qmail 36687 invoked by uid 500); 6 May 2011 18:56:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36660 invoked by uid 500); 6 May 2011 18:56:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36651 invoked by uid 99); 6 May 2011 18:56:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 18:56:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 18:56:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 459B6C42D2
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 18:56:03 +0000 (UTC)
Date: Fri, 6 May 2011 18:56:03 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1251501633.28800.1304708163281.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1330832899.28720.1304706243764.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7264) Bump avro version to at least
 1.4.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030082#comment-13030082 ] 

Luke Lu commented on HADOOP-7264:
---------------------------------

maven shade and jarjar doesn't help in this case for mr2 (it's in the common code in job history), as both client and server needs to be on at least 1.4.1, sans an OSGi containment in the job history server, which would be over-engineering at this point. maven shade/jarjar would be helpful for future client libraries.

BTW, we initially actually used maven shade for various servers, but had to remove it as the added processing time, the annoying warnings and the resulting bulk really don't worth the trouble.

> Bump avro version to at least 1.4.1
> -----------------------------------
>
>                 Key: HADOOP-7264
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7264
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Luke Lu
>              Labels: avro
>             Fix For: 0.23.0
>
>
> Needed by mapreduce 2.0 avro support. Maybe we could jump to Avro 1.5. There is incompatible API changes from 1.3x to 1.4x (Utf8 to CharSequence in user facing APIs) not sure about 1.5x though.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14551-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 19:18:45 2011
Return-Path: <common-issues-return-14551-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 11941348A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 19:18:45 +0000 (UTC)
Received: (qmail 65746 invoked by uid 500); 6 May 2011 19:18:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65699 invoked by uid 500); 6 May 2011 19:18:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65684 invoked by uid 99); 6 May 2011 19:18:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 19:18:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 19:18:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 544EBC4E2F
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 19:18:03 +0000 (UTC)
Date: Fri, 6 May 2011 19:18:03 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1944031344.28880.1304709483342.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1330832899.28720.1304706243764.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7264) Bump avro version to at least
 1.4.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030097#comment-13030097 ] 

Todd Lipcon commented on HADOOP-7264:
-------------------------------------

k, thanks for the perspective. I bring this up because we've heard from a number of users that, when we change the version of dependencies like thrift, avro, or (especially) Jackson, it hurts users who use these same libraries in their jobs.

> Bump avro version to at least 1.4.1
> -----------------------------------
>
>                 Key: HADOOP-7264
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7264
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Luke Lu
>              Labels: avro
>             Fix For: 0.23.0
>
>
> Needed by mapreduce 2.0 avro support. Maybe we could jump to Avro 1.5. There is incompatible API changes from 1.3x to 1.4x (Utf8 to CharSequence in user facing APIs) not sure about 1.5x though.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14552-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 19:18:46 2011
Return-Path: <common-issues-return-14552-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 750BA34B5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 19:18:46 +0000 (UTC)
Received: (qmail 66118 invoked by uid 500); 6 May 2011 19:18:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66057 invoked by uid 500); 6 May 2011 19:18:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65954 invoked by uid 99); 6 May 2011 19:18:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 19:18:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 19:18:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 29818C4E25
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 19:18:03 +0000 (UTC)
Date: Fri, 6 May 2011 19:18:03 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1142652053.28874.1304709483167.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7265) Keep track of relative paths
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Keep track of relative paths
----------------------------

                 Key: HADOOP-7265
                 URL: https://issues.apache.org/jira/browse/HADOOP-7265
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs
    Affects Versions: 0.23.0
            Reporter: Daryn Sharp
            Assignee: Daryn Sharp


As part of the effort to standardize the display of paths, the PathData tracks the exact string used to create a path.  When obtaining a directory's contents, the relative nature of the original path should be preserved.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14553-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 19:37:45 2011
Return-Path: <common-issues-return-14553-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DB96832FB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 19:37:44 +0000 (UTC)
Received: (qmail 95112 invoked by uid 500); 6 May 2011 19:37:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95076 invoked by uid 500); 6 May 2011 19:37:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95068 invoked by uid 99); 6 May 2011 19:37:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 19:37:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 19:37:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3F3E1C44E1
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 19:37:03 +0000 (UTC)
Date: Fri, 6 May 2011 19:37:03 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2076641004.28921.1304710623255.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1330832899.28720.1304706243764.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7264) Bump avro version to at least
 1.4.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030105#comment-13030105 ] 

Luke Lu commented on HADOOP-7264:
---------------------------------

We do want to improve the isolation of dependencies between framework and user code, which I believe is related but really orthogonal to this jira. HADOOP-7150 is trying to achieve that. I believe it's mostly achievable via careful refactor of classpath construction and a little shade/jar/jar in the near term and completely nailed with some like OSGi in the long term.

> Bump avro version to at least 1.4.1
> -----------------------------------
>
>                 Key: HADOOP-7264
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7264
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Luke Lu
>              Labels: avro
>             Fix For: 0.23.0
>
>
> Needed by mapreduce 2.0 avro support. Maybe we could jump to Avro 1.5. There is incompatible API changes from 1.3x to 1.4x (Utf8 to CharSequence in user facing APIs) not sure about 1.5x though.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14554-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 19:41:42 2011
Return-Path: <common-issues-return-14554-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B09A73397
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 19:41:42 +0000 (UTC)
Received: (qmail 2791 invoked by uid 500); 6 May 2011 19:41:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2768 invoked by uid 500); 6 May 2011 19:41:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2760 invoked by uid 99); 6 May 2011 19:41:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 19:41:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 19:41:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 31B5DC45B6
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 19:41:03 +0000 (UTC)
Date: Fri, 6 May 2011 19:41:03 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1605225256.28924.1304710863200.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1142652053.28874.1304709483167.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7265) Keep track of relative paths
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7265:
--------------------------------

    Attachment: HADOOP-7265.patch

> Keep track of relative paths
> ----------------------------
>
>                 Key: HADOOP-7265
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7265
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7265.patch
>
>
> As part of the effort to standardize the display of paths, the PathData tracks the exact string used to create a path.  When obtaining a directory's contents, the relative nature of the original path should be preserved.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14555-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 19:41:44 2011
Return-Path: <common-issues-return-14555-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D7C1633A7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 19:41:44 +0000 (UTC)
Received: (qmail 3057 invoked by uid 500); 6 May 2011 19:41:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3013 invoked by uid 500); 6 May 2011 19:41:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3005 invoked by uid 99); 6 May 2011 19:41:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 19:41:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 19:41:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 451BDC45BA
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 19:41:03 +0000 (UTC)
Date: Fri, 6 May 2011 19:41:03 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1386818731.28926.1304710863279.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1142652053.28874.1304709483167.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7265) Keep track of relative paths
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7265:
--------------------------------

    Status: Patch Available  (was: Open)

> Keep track of relative paths
> ----------------------------
>
>                 Key: HADOOP-7265
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7265
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7265.patch
>
>
> As part of the effort to standardize the display of paths, the PathData tracks the exact string used to create a path.  When obtaining a directory's contents, the relative nature of the original path should be preserved.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14556-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 19:55:42 2011
Return-Path: <common-issues-return-14556-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C1DE93968
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 19:55:42 +0000 (UTC)
Received: (qmail 15338 invoked by uid 500); 6 May 2011 19:55:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15310 invoked by uid 500); 6 May 2011 19:55:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15302 invoked by uid 99); 6 May 2011 19:55:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 19:55:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 19:55:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 304C6C49AA
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 19:55:03 +0000 (UTC)
Date: Fri, 6 May 2011 19:55:03 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1360914186.28966.1304711703193.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1142652053.28874.1304709483167.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7265) Keep track of relative paths
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030112#comment-13030112 ] 

Hadoop QA commented on HADOOP-7265:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478463/HADOOP-7265.patch
  against trunk revision 1100278.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/416//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/416//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/416//console

This message is automatically generated.

> Keep track of relative paths
> ----------------------------
>
>                 Key: HADOOP-7265
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7265
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7265.patch
>
>
> As part of the effort to standardize the display of paths, the PathData tracks the exact string used to create a path.  When obtaining a directory's contents, the relative nature of the original path should be preserved.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14557-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 20:21:43 2011
Return-Path: <common-issues-return-14557-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 22E0735DE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 20:21:43 +0000 (UTC)
Received: (qmail 47230 invoked by uid 500); 6 May 2011 20:21:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47153 invoked by uid 500); 6 May 2011 20:21:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47020 invoked by uid 99); 6 May 2011 20:21:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 20:21:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 20:21:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 67BA6C4085
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 20:21:03 +0000 (UTC)
Date: Fri, 6 May 2011 20:21:03 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <101411461.28994.1304713263421.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <796788985.10688.1304041503305.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7249) Refactor FsShell's chmod/chown/chgrp
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7249:
-------------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I have committed this.  Thanks, Daryn!

Thanks also Matt for reviewing it.

> Refactor FsShell's chmod/chown/chgrp
> ------------------------------------
>
>                 Key: HADOOP-7249
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7249
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7249-2.patch, HADOOP-7249.patch
>
>
> Need to refactor permissions commands to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14558-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 20:35:44 2011
Return-Path: <common-issues-return-14558-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D22C73CA4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 20:35:44 +0000 (UTC)
Received: (qmail 66940 invoked by uid 500); 6 May 2011 20:35:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66901 invoked by uid 500); 6 May 2011 20:35:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66893 invoked by uid 99); 6 May 2011 20:35:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 20:35:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 20:35:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3BDB9C45B6
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 20:35:03 +0000 (UTC)
Date: Fri, 6 May 2011 20:35:03 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1812776221.29036.1304714103241.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <796788985.10688.1304041503305.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7249) Refactor FsShell's
 chmod/chown/chgrp
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030123#comment-13030123 ] 

Hudson commented on HADOOP-7249:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #578 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/578/])
    HADOOP-7249. Refactor the chmod/chown/chgrp command to conform to new FsCommand class.  Contributed by Daryn Sharp


> Refactor FsShell's chmod/chown/chgrp
> ------------------------------------
>
>                 Key: HADOOP-7249
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7249
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7249-2.patch, HADOOP-7249.patch
>
>
> Need to refactor permissions commands to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14559-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 20:35:45 2011
Return-Path: <common-issues-return-14559-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 206283CAE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 20:35:45 +0000 (UTC)
Received: (qmail 67154 invoked by uid 500); 6 May 2011 20:35:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67128 invoked by uid 500); 6 May 2011 20:35:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67119 invoked by uid 99); 6 May 2011 20:35:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 20:35:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 20:35:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5C4B7C45BB
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 20:35:03 +0000 (UTC)
Date: Fri, 6 May 2011 20:35:03 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1151480213.29039.1304714103374.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1362418509.26364.1304633343208.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7261) Disable IPV6 for junit tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030124#comment-13030124 ] 

Hudson commented on HADOOP-7261:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #578 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/578/])
    

> Disable IPV6 for junit tests
> ----------------------------
>
>                 Key: HADOOP-7261
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7261
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7261.1.patch, HADOOP-7261.patch
>
>
> IPV6 addresses not handles currently in the common library methods. IPV6 can return address as "0:0:0:0:0:0:port". Some utility methods such as NetUtils#createSocketAddress(), NetUtils#normalizeHostName(), NetUtils#getHostNameOfIp() to name a few, do not handle IPV6 address and expect address to be of format host:port.
> Until IPV6 is formally supported, I propose disabling IPV6 for junit tests to avoid problems seen in HDFS-1891.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14560-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 20:39:43 2011
Return-Path: <common-issues-return-14560-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 26CED3D96
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 20:39:43 +0000 (UTC)
Received: (qmail 70965 invoked by uid 500); 6 May 2011 20:39:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70934 invoked by uid 500); 6 May 2011 20:39:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70914 invoked by uid 99); 6 May 2011 20:39:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 20:39:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 20:39:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 46E8FC47DF
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 20:39:03 +0000 (UTC)
Date: Fri, 6 May 2011 20:39:03 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1150893561.29067.1304714343287.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7266) Deprecate metrics v1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Deprecate metrics v1
--------------------

                 Key: HADOOP-7266
                 URL: https://issues.apache.org/jira/browse/HADOOP-7266
             Project: Hadoop Common
          Issue Type: Sub-task
            Reporter: Luke Lu
            Assignee: Luke Lu




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14561-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 20:41:47 2011
Return-Path: <common-issues-return-14561-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6A5B83E38
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 20:41:47 +0000 (UTC)
Received: (qmail 75159 invoked by uid 500); 6 May 2011 20:41:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75117 invoked by uid 500); 6 May 2011 20:41:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75025 invoked by uid 99); 6 May 2011 20:41:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 20:41:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 20:41:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7E4DDC48D6
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 20:41:04 +0000 (UTC)
Date: Fri, 6 May 2011 20:41:04 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1660405400.29103.1304714464514.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1150893561.29067.1304714343287.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7266) Deprecate metrics v1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030130#comment-13030130 ] 

Luke Lu commented on HADOOP-7266:
---------------------------------

After all framework metrics have been ported to metrics v2.

> Deprecate metrics v1
> --------------------
>
>                 Key: HADOOP-7266
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7266
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14562-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 20:44:42 2011
Return-Path: <common-issues-return-14562-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D3F1E3FCC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 20:44:42 +0000 (UTC)
Received: (qmail 89582 invoked by uid 500); 6 May 2011 20:44:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89446 invoked by uid 500); 6 May 2011 20:44:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89434 invoked by uid 99); 6 May 2011 20:44:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 20:44:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 20:44:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5504FC43A1
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 20:44:03 +0000 (UTC)
Date: Fri, 6 May 2011 20:44:03 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1687350606.29118.1304714643344.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1150893561.29067.1304714343287.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7266) Deprecate metrics v1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030131#comment-13030131 ] 

Luke Lu commented on HADOOP-7266:
---------------------------------

By framework, I meant non-user/downstream metrics related code in common, hdfs and mapreduce, as per discussion in HADOOP-6728.

> Deprecate metrics v1
> --------------------
>
>                 Key: HADOOP-7266
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7266
>             Project: Hadoop Common
>          Issue Type: Sub-task
>    Affects Versions: 0.21.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14563-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 20:44:44 2011
Return-Path: <common-issues-return-14563-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E93993FD3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 20:44:44 +0000 (UTC)
Received: (qmail 89725 invoked by uid 500); 6 May 2011 20:44:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89692 invoked by uid 500); 6 May 2011 20:44:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89684 invoked by uid 99); 6 May 2011 20:44:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 20:44:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 20:44:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5E251C43A2
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 20:44:03 +0000 (UTC)
Date: Fri, 6 May 2011 20:44:03 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1780076407.29119.1304714643382.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1150893561.29067.1304714343287.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7266) Deprecate metrics v1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-7266:
----------------------------

    Affects Version/s: 0.21.0
        Fix Version/s: 0.23.0

> Deprecate metrics v1
> --------------------
>
>                 Key: HADOOP-7266
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7266
>             Project: Hadoop Common
>          Issue Type: Sub-task
>    Affects Versions: 0.21.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14564-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 20:46:45 2011
Return-Path: <common-issues-return-14564-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4A1563BAC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 20:46:45 +0000 (UTC)
Received: (qmail 93483 invoked by uid 500); 6 May 2011 20:46:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93452 invoked by uid 500); 6 May 2011 20:46:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93424 invoked by uid 99); 6 May 2011 20:46:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 20:46:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 20:46:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C6F52C40CE
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 20:46:04 +0000 (UTC)
Date: Fri, 6 May 2011 20:46:04 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <32744862.29160.1304714764811.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6728) Overhaul metrics framework
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-6728:
----------------------------

    Affects Version/s: 0.21.0
        Fix Version/s: 0.23.0

> Overhaul metrics framework
> --------------------------
>
>                 Key: HADOOP-6728
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6728
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.2, 0.21.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-6728-y20.104.patch, metrics1-flow.png, metrics2-builder-r2.png, metrics2-builder.png, metrics2-flow.png, metrics2-mutable-r2.png, metrics2-mutable.png, metrics2-record-r2.png, metrics2-record.png, metrics2-uml-r2.png, metrics2-uml.png
>
>
> Per discussions with Arun, Chris, Hong and Rajiv, et al, we concluded that the current metrics framework needs an overhaul to:
> * Allow multiple plugins for different monitoring systems simultaneously (see also: HADOOP-6508).
> * Refresh metrics plugin config without server restart.
> ** Including filtering of metrics per plugin.
> * Support metrics schema for plugins.
> The jira will be resolved when core hadoop components (hdfs, mapreduce) are updated to use the new framework . Updates to external components that use the existing metrics framework will be tracked by different issues.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14565-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 20:50:46 2011
Return-Path: <common-issues-return-14565-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 781C83524
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 20:50:46 +0000 (UTC)
Received: (qmail 99460 invoked by uid 500); 6 May 2011 20:50:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99374 invoked by uid 500); 6 May 2011 20:50:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99289 invoked by uid 99); 6 May 2011 20:50:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 20:50:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 20:50:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 35FA1C4282
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 20:50:03 +0000 (UTC)
Date: Fri, 6 May 2011 20:50:03 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <873708488.29176.1304715003217.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7267) Refactor FsShell's rm/rmr/expunge
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Refactor FsShell's rm/rmr/expunge
---------------------------------

                 Key: HADOOP-7267
                 URL: https://issues.apache.org/jira/browse/HADOOP-7267
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs
    Affects Versions: 0.23.0
            Reporter: Daryn Sharp
            Assignee: Daryn Sharp


Refactor to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14566-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 20:58:44 2011
Return-Path: <common-issues-return-14566-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AEAAA363B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 20:58:44 +0000 (UTC)
Received: (qmail 8571 invoked by uid 500); 6 May 2011 20:58:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8540 invoked by uid 500); 6 May 2011 20:58:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8532 invoked by uid 99); 6 May 2011 20:58:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 20:58:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 20:58:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1D283C450C
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 20:58:03 +0000 (UTC)
Date: Fri, 6 May 2011 20:58:03 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <690394661.29192.1304715483115.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <720022722.11722.1304088663379.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7251) Refactor FsShell's getmerge
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7251:
-------------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I have committed this.  Thanks, Daryn!

Thanks also Matt for reviewing it.

> Refactor FsShell's getmerge
> ---------------------------
>
>                 Key: HADOOP-7251
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7251
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7251-2.patch, HADOOP-7251.patch
>
>
> Need to refactor getmerge to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14567-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 21:00:45 2011
Return-Path: <common-issues-return-14567-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E039C3A40
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 21:00:45 +0000 (UTC)
Received: (qmail 10988 invoked by uid 500); 6 May 2011 21:00:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10909 invoked by uid 500); 6 May 2011 21:00:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10894 invoked by uid 99); 6 May 2011 21:00:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 21:00:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 21:00:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7F961C4652
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 21:00:03 +0000 (UTC)
Date: Fri, 6 May 2011 21:00:03 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2074951810.29204.1304715603518.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6920) Metrics2: metrics instrumentation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-6920:
----------------------------

    Attachment: hadoop-6920-metrics2-inst-v4.patch

> Metrics2: metrics instrumentation
> ---------------------------------
>
>                 Key: HADOOP-6920
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6920
>             Project: Hadoop Common
>          Issue Type: Sub-task
>    Affects Versions: 0.20.2
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>         Attachments: hadoop-6920-metrics2-inst-v4.patch
>
>
> The jira tracks the metrics instrumentation for the new framework, i.e., porting jvm and rpc metrics to the new framework.
> This issue (esp. the "incompatible change flag") depends on the outcome of the discussion (proposed in HADOOP-6728) on whether we should support backward compatibility (at some cost.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14568-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 21:00:46 2011
Return-Path: <common-issues-return-14568-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E15DA3A41
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 21:00:45 +0000 (UTC)
Received: (qmail 11008 invoked by uid 500); 6 May 2011 21:00:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10912 invoked by uid 500); 6 May 2011 21:00:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10897 invoked by uid 99); 6 May 2011 21:00:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 21:00:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 21:00:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 89456C4654
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 21:00:03 +0000 (UTC)
Date: Fri, 6 May 2011 21:00:03 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <728127037.29205.1304715603559.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <651142754.74841.1303423565756.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7238) Refactor FsShell's cat & text
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030136#comment-13030136 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7238:
------------------------------------------------

Daryn, the patch does not apply anymore.

> Refactor FsShell's cat & text
> -----------------------------
>
>                 Key: HADOOP-7238
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7238
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7238.patch
>
>
> Need to refactor cat & text to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14569-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 21:02:45 2011
Return-Path: <common-issues-return-14569-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 313BD33B5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 21:02:45 +0000 (UTC)
Received: (qmail 12675 invoked by uid 500); 6 May 2011 21:02:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12641 invoked by uid 500); 6 May 2011 21:02:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12633 invoked by uid 99); 6 May 2011 21:02:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 21:02:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 21:02:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7981CC47A0
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 21:02:03 +0000 (UTC)
Date: Fri, 6 May 2011 21:02:03 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <961850156.29216.1304715723494.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6920) Metrics2: metrics instrumentation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-6920:
----------------------------

        Fix Version/s: 0.23.0
    Affects Version/s:     (was: 0.20.2)
               Status: Patch Available  (was: Open)

This patch also moves the canonical package of EventCounter to org.apache.log.metrics

> Metrics2: metrics instrumentation
> ---------------------------------
>
>                 Key: HADOOP-6920
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6920
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-6920-metrics2-inst-v4.patch
>
>
> The jira tracks the metrics instrumentation for the new framework, i.e., porting jvm and rpc metrics to the new framework.
> This issue (esp. the "incompatible change flag") depends on the outcome of the discussion (proposed in HADOOP-6728) on whether we should support backward compatibility (at some cost.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14570-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 21:04:42 2011
Return-Path: <common-issues-return-14570-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ADE883BB1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 21:04:42 +0000 (UTC)
Received: (qmail 14011 invoked by uid 500); 6 May 2011 21:04:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13954 invoked by uid 500); 6 May 2011 21:04:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13945 invoked by uid 99); 6 May 2011 21:04:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 21:04:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 21:04:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2D857C488C
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 21:04:03 +0000 (UTC)
Date: Fri, 6 May 2011 21:04:03 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <673391250.29224.1304715843182.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1142652053.28874.1304709483167.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7265) Keep track of relative paths
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030139#comment-13030139 ] 

Tanping Wang commented on HADOOP-7265:
--------------------------------------

+1 on the patch.

> Keep track of relative paths
> ----------------------------
>
>                 Key: HADOOP-7265
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7265
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7265.patch
>
>
> As part of the effort to standardize the display of paths, the PathData tracks the exact string used to create a path.  When obtaining a directory's contents, the relative nature of the original path should be preserved.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14571-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 21:06:43 2011
Return-Path: <common-issues-return-14571-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 15B5E3BF0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 21:06:43 +0000 (UTC)
Received: (qmail 14950 invoked by uid 500); 6 May 2011 21:06:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14917 invoked by uid 500); 6 May 2011 21:06:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14898 invoked by uid 99); 6 May 2011 21:06:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 21:06:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 21:06:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6A97FC4916
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 21:06:03 +0000 (UTC)
Date: Fri, 6 May 2011 21:06:03 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1393530997.29227.1304715963433.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6918) Make metrics naming consistent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-6918:
----------------------------

    Affects Version/s: 0.21.0
         Release Note: Harmonizing metrics names to CapitalizedCamelCase  (was: Harmonizing metrics names)

> Make metrics naming consistent
> ------------------------------
>
>                 Key: HADOOP-6918
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6918
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: metrics
>    Affects Versions: 0.20.2, 0.21.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>
> While working HADOOP-6728, I noticed that our metrics naming style is all over the place:
> * Capitalized camel case: e.g., "FilesCreated" in namenode metrics and some rpc metrics
> * uncapitalized camel case: e.g, "threadsBlocked" in jvm metrics and some rpc metrics
> * lowercased underscored: e.g., "bytes_written" in datanode metrics and mapreduce metrics
> Let's make them consistent. How about uncapitalized camel case? My main reason for the camel case: some backends have limits on the name length and underscore is wasteful.
> Once we have a consistent naming style we can do:
> @Metric("Number of INodes created") MutableCounterLong filesCreated;
> instead of the more redundant:
> @Metric({"FilesCreated", "Number of INodes created"}) MutableCounterLong filesCreated;

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14572-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 21:15:18 2011
Return-Path: <common-issues-return-14572-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0DC6B3D9F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 21:15:18 +0000 (UTC)
Received: (qmail 24153 invoked by uid 500); 6 May 2011 21:15:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24128 invoked by uid 500); 6 May 2011 21:15:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24120 invoked by uid 99); 6 May 2011 21:15:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 21:15:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 21:15:16 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 806AEC4B4D
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 21:14:38 +0000 (UTC)
Date: Fri, 6 May 2011 21:14:38 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1710094510.29246.1304716478522.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <720022722.11722.1304088663379.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7251) Refactor FsShell's getmerge
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030144#comment-13030144 ] 

Hudson commented on HADOOP-7251:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #579 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/579/])
    HADOOP-7251. Refactor the getmerge command to conform to new FsCommand class.  Contributed by Daryn Sharp


> Refactor FsShell's getmerge
> ---------------------------
>
>                 Key: HADOOP-7251
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7251
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7251-2.patch, HADOOP-7251.patch
>
>
> Need to refactor getmerge to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14573-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 21:15:18 2011
Return-Path: <common-issues-return-14573-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3D7BD3DAF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 21:15:18 +0000 (UTC)
Received: (qmail 24393 invoked by uid 500); 6 May 2011 21:15:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24354 invoked by uid 500); 6 May 2011 21:15:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24132 invoked by uid 99); 6 May 2011 21:15:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 21:15:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 21:15:16 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6E79FC4B4B
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 21:14:38 +0000 (UTC)
Date: Fri, 6 May 2011 21:14:38 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <952219051.29244.1304716478448.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6920) Metrics2: metrics instrumentation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030143#comment-13030143 ] 

Hadoop QA commented on HADOOP-6920:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478470/hadoop-6920-metrics2-inst-v4.patch
  against trunk revision 1100369.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 11 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    -1 release audit.  The applied patch generated 2 release audit warnings (more than the trunk's current 1 warnings).

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/417//testReport/
Release audit warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/417//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/417//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/417//console

This message is automatically generated.

> Metrics2: metrics instrumentation
> ---------------------------------
>
>                 Key: HADOOP-6920
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6920
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-6920-metrics2-inst-v4.patch
>
>
> The jira tracks the metrics instrumentation for the new framework, i.e., porting jvm and rpc metrics to the new framework.
> This issue (esp. the "incompatible change flag") depends on the outcome of the discussion (proposed in HADOOP-6728) on whether we should support backward compatibility (at some cost.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14574-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 21:40:44 2011
Return-Path: <common-issues-return-14574-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B8E313967
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 21:40:44 +0000 (UTC)
Received: (qmail 49459 invoked by uid 500); 6 May 2011 21:40:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49426 invoked by uid 500); 6 May 2011 21:40:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49418 invoked by uid 99); 6 May 2011 21:40:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 21:40:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 21:40:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 24C3DC4244
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 21:40:03 +0000 (UTC)
Date: Fri, 6 May 2011 21:40:03 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <871190861.29280.1304718003147.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6920) Metrics2: metrics instrumentation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030152#comment-13030152 ] 

Luke Lu commented on HADOOP-6920:
---------------------------------

The RAT warning is from an empty src/java/org/apache/hadoop/fs/shell/Display.java from Nicholas' commit for HADOOP-7251.

> Metrics2: metrics instrumentation
> ---------------------------------
>
>                 Key: HADOOP-6920
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6920
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-6920-metrics2-inst-v4.patch
>
>
> The jira tracks the metrics instrumentation for the new framework, i.e., porting jvm and rpc metrics to the new framework.
> This issue (esp. the "incompatible change flag") depends on the outcome of the discussion (proposed in HADOOP-6728) on whether we should support backward compatibility (at some cost.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14575-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May  6 22:26:42 2011
Return-Path: <common-issues-return-14575-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B8F835F5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri,  6 May 2011 22:26:42 +0000 (UTC)
Received: (qmail 89212 invoked by uid 500); 6 May 2011 22:26:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89157 invoked by uid 500); 6 May 2011 22:26:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89149 invoked by uid 99); 6 May 2011 22:26:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 22:26:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2011 22:26:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2697DC4EE5
	for <common-issues@hadoop.apache.org>; Fri,  6 May 2011 22:26:03 +0000 (UTC)
Date: Fri, 6 May 2011 22:26:03 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1119206474.29342.1304720763154.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <796788985.10688.1304041503305.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7249) Refactor FsShell's
 chmod/chown/chgrp
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030171#comment-13030171 ] 

Hudson commented on HADOOP-7249:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #580 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/580/])
    Remove the empty file accidentally checked it with HADOOP-7249.


> Refactor FsShell's chmod/chown/chgrp
> ------------------------------------
>
>                 Key: HADOOP-7249
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7249
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7249-2.patch, HADOOP-7249.patch
>
>
> Need to refactor permissions commands to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14576-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May  7 00:11:45 2011
Return-Path: <common-issues-return-14576-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 097A23BFE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat,  7 May 2011 00:11:45 +0000 (UTC)
Received: (qmail 59164 invoked by uid 500); 7 May 2011 00:11:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59124 invoked by uid 500); 7 May 2011 00:11:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59116 invoked by uid 99); 7 May 2011 00:11:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 00:11:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 00:11:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 44117C4B24
	for <common-issues@hadoop.apache.org>; Sat,  7 May 2011 00:11:03 +0000 (UTC)
Date: Sat, 7 May 2011 00:11:03 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1736384367.29564.1304727063275.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6918) Make metrics naming consistent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-6918:
----------------------------

    Release Note: 
Harmonizing metrics names to CapitalizedCamelCase.

Examples: bytes_written becomes BytesWritten; threadsBlocked becomes ThreadsBlocked etc.

  was:Harmonizing metrics names to CapitalizedCamelCase


> Make metrics naming consistent
> ------------------------------
>
>                 Key: HADOOP-6918
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6918
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: metrics
>    Affects Versions: 0.20.2, 0.21.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>
> While working HADOOP-6728, I noticed that our metrics naming style is all over the place:
> * Capitalized camel case: e.g., "FilesCreated" in namenode metrics and some rpc metrics
> * uncapitalized camel case: e.g, "threadsBlocked" in jvm metrics and some rpc metrics
> * lowercased underscored: e.g., "bytes_written" in datanode metrics and mapreduce metrics
> Let's make them consistent. How about uncapitalized camel case? My main reason for the camel case: some backends have limits on the name length and underscore is wasteful.
> Once we have a consistent naming style we can do:
> @Metric("Number of INodes created") MutableCounterLong filesCreated;
> instead of the more redundant:
> @Metric({"FilesCreated", "Number of INodes created"}) MutableCounterLong filesCreated;

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14578-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May  7 11:15:44 2011
Return-Path: <common-issues-return-14578-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 15EB4257C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat,  7 May 2011 11:15:44 +0000 (UTC)
Received: (qmail 84886 invoked by uid 500); 7 May 2011 11:15:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84828 invoked by uid 500); 7 May 2011 11:15:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84812 invoked by uid 99); 7 May 2011 11:15:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 11:15:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 11:15:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 62022C5A06
	for <common-issues@hadoop.apache.org>; Sat,  7 May 2011 11:15:03 +0000 (UTC)
Date: Sat, 7 May 2011 11:15:03 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <243299511.30038.1304766903398.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <720022722.11722.1304088663379.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7251) Refactor FsShell's getmerge
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030317#comment-13030317 ] 

Hudson commented on HADOOP-7251:
--------------------------------

Integrated in Hadoop-Common-trunk #681 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/681/])
    HADOOP-7251. Refactor the getmerge command to conform to new FsCommand class.  Contributed by Daryn Sharp


> Refactor FsShell's getmerge
> ---------------------------
>
>                 Key: HADOOP-7251
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7251
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7251-2.patch, HADOOP-7251.patch
>
>
> Need to refactor getmerge to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14579-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May  7 11:15:45 2011
Return-Path: <common-issues-return-14579-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E89A02595
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat,  7 May 2011 11:15:44 +0000 (UTC)
Received: (qmail 85348 invoked by uid 500); 7 May 2011 11:15:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85313 invoked by uid 500); 7 May 2011 11:15:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85305 invoked by uid 99); 7 May 2011 11:15:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 11:15:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 11:15:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 381B9C5A01
	for <common-issues@hadoop.apache.org>; Sat,  7 May 2011 11:15:03 +0000 (UTC)
Date: Sat, 7 May 2011 11:15:03 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1033191605.30033.1304766903226.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <796788985.10688.1304041503305.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7249) Refactor FsShell's
 chmod/chown/chgrp
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030315#comment-13030315 ] 

Hudson commented on HADOOP-7249:
--------------------------------

Integrated in Hadoop-Common-trunk #681 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/681/])
    Remove the empty file accidentally checked it with HADOOP-7249.
HADOOP-7249. Refactor the chmod/chown/chgrp command to conform to new FsCommand class.  Contributed by Daryn Sharp


> Refactor FsShell's chmod/chown/chgrp
> ------------------------------------
>
>                 Key: HADOOP-7249
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7249
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7249-2.patch, HADOOP-7249.patch
>
>
> Need to refactor permissions commands to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14577-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May  7 11:15:45 2011
Return-Path: <common-issues-return-14577-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 19ACC2584
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat,  7 May 2011 11:15:44 +0000 (UTC)
Received: (qmail 84858 invoked by uid 500); 7 May 2011 11:15:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84825 invoked by uid 500); 7 May 2011 11:15:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84809 invoked by uid 99); 7 May 2011 11:15:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 11:15:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 11:15:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5652DC5A05
	for <common-issues@hadoop.apache.org>; Sat,  7 May 2011 11:15:03 +0000 (UTC)
Date: Sat, 7 May 2011 11:15:03 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2008864235.30037.1304766903350.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1362418509.26364.1304633343208.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7261) Disable IPV6 for junit tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030316#comment-13030316 ] 

Hudson commented on HADOOP-7261:
--------------------------------

Integrated in Hadoop-Common-trunk #681 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/681/])
    HADOOP-7261. Disable IPV6 for junit tests. Contributed by Suresh Srinivas.


> Disable IPV6 for junit tests
> ----------------------------
>
>                 Key: HADOOP-7261
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7261
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7261.1.patch, HADOOP-7261.patch
>
>
> IPV6 addresses not handles currently in the common library methods. IPV6 can return address as "0:0:0:0:0:0:port". Some utility methods such as NetUtils#createSocketAddress(), NetUtils#normalizeHostName(), NetUtils#getHostNameOfIp() to name a few, do not handle IPV6 address and expect address to be of format host:port.
> Until IPV6 is formally supported, I propose disabling IPV6 for junit tests to avoid problems seen in HDFS-1891.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14580-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  9 00:35:43 2011
Return-Path: <common-issues-return-14580-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A0FB9348A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  9 May 2011 00:35:43 +0000 (UTC)
Received: (qmail 3222 invoked by uid 500); 9 May 2011 00:35:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3183 invoked by uid 500); 9 May 2011 00:35:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3174 invoked by uid 99); 9 May 2011 00:35:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 00:35:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 00:35:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 219D0C707B
	for <common-issues@hadoop.apache.org>; Mon,  9 May 2011 00:35:03 +0000 (UTC)
Date: Mon, 9 May 2011 00:35:03 +0000 (UTC)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1787564551.31447.1304901303134.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <304321177.482.1298765698936.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7157) Create build hosts configurations to
 prevent degradation of patch validation and snapshot build environment
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-7157:
---------------------------------------

    Attachment: HADOOP-7157.patch

Using resource array instead of decl. list.

> Create build hosts configurations to prevent degradation of patch validation and snapshot build environment
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7157
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7157
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>         Environment: Apache Hadoop build hosts
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-7157.patch, HADOOP-7157.patch
>
>
> Build hosts are being re-jumped, in need for configuration updates, etc. It is hard to maintain the same configuration across a 10+ hosts. A specialized service such as Puppet can be used to maintain build machines software and configurations at a desired level.
> Such configs should be checked into SCM system along with the of sources code.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14581-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  9 05:04:57 2011
Return-Path: <common-issues-return-14581-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 834E84BF3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  9 May 2011 05:04:57 +0000 (UTC)
Received: (qmail 51380 invoked by uid 500); 9 May 2011 05:04:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51140 invoked by uid 500); 9 May 2011 05:04:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51001 invoked by uid 99); 9 May 2011 05:04:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 05:04:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 05:04:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 492D6C7388
	for <common-issues@hadoop.apache.org>; Mon,  9 May 2011 05:04:03 +0000 (UTC)
Date: Mon, 9 May 2011 05:04:03 +0000 (UTC)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1912220915.31712.1304917443296.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7268) FileContext.getLocalFSFileContext()
 behavior needs to be fixed w.r.t tokens
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

FileContext.getLocalFSFileContext() behavior needs to be fixed w.r.t tokens
---------------------------------------------------------------------------

                 Key: HADOOP-7268
                 URL: https://issues.apache.org/jira/browse/HADOOP-7268
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs, security
    Affects Versions: 0.23.0
            Reporter: Devaraj Das
             Fix For: 0.23.0


FileContext.getLocalFSFileContext() instantiates a FileContext object upon the first call to it, and for all subsequent calls returns back that instance (a static localFsSingleton object). With security turned on, this causes some hard-to-debug situations when that fileContext is used for doing HDFS operations. This is because the UserGroupInformation is stored when a FileContext is instantiated. If the process in question wishes to use different UserGroupInformation objects for different file system operations (where the corresponding FileContext objects are obtained via calls to FileContext.getLocalFSFileContext()), it doesn't work.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14582-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  9 13:33:46 2011
Return-Path: <common-issues-return-14582-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 29E7E4887
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  9 May 2011 13:33:46 +0000 (UTC)
Received: (qmail 64398 invoked by uid 500); 9 May 2011 13:33:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64231 invoked by uid 500); 9 May 2011 13:33:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63954 invoked by uid 99); 9 May 2011 13:33:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 13:33:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 13:33:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 42A45C8848
	for <common-issues@hadoop.apache.org>; Mon,  9 May 2011 13:33:03 +0000 (UTC)
Date: Mon, 9 May 2011 13:33:03 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <563672243.32348.1304947983269.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1362418509.26364.1304633343208.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7261) Disable IPV6 for junit tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030726#comment-13030726 ] 

Steve Loughran commented on HADOOP-7261:
----------------------------------------

IPv6 is turned off in the official Hadoop scripts already: HADOOP-6056

> Disable IPV6 for junit tests
> ----------------------------
>
>                 Key: HADOOP-7261
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7261
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7261.1.patch, HADOOP-7261.patch
>
>
> IPV6 addresses not handles currently in the common library methods. IPV6 can return address as "0:0:0:0:0:0:port". Some utility methods such as NetUtils#createSocketAddress(), NetUtils#normalizeHostName(), NetUtils#getHostNameOfIp() to name a few, do not handle IPV6 address and expect address to be of format host:port.
> Until IPV6 is formally supported, I propose disabling IPV6 for junit tests to avoid problems seen in HDFS-1891.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14583-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  9 16:42:43 2011
Return-Path: <common-issues-return-14583-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 11A004FCE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  9 May 2011 16:42:43 +0000 (UTC)
Received: (qmail 91870 invoked by uid 500); 9 May 2011 16:42:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91755 invoked by uid 500); 9 May 2011 16:42:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91708 invoked by uid 99); 9 May 2011 16:42:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 16:42:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 16:42:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 61747C8EAD
	for <common-issues@hadoop.apache.org>; Mon,  9 May 2011 16:42:03 +0000 (UTC)
Date: Mon, 9 May 2011 16:42:03 +0000 (UTC)
From: "Nicholas Telford (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7269) S3 Native should allow customizable
 file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

S3 Native should allow customizable file meta-data (headers)
------------------------------------------------------------

                 Key: HADOOP-7269
                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs/s3
            Reporter: Nicholas Telford
            Priority: Minor


The S3 Native FileSystem currently writes all files with a set of default headers:

 * Content-Type: binary/octet-stream
 * Content-Length: <computed from object size>
 * Content-MD5: <computed from object data>

This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14584-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  9 17:26:43 2011
Return-Path: <common-issues-return-14584-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1C0664CE6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  9 May 2011 17:26:43 +0000 (UTC)
Received: (qmail 63286 invoked by uid 500); 9 May 2011 17:26:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63258 invoked by uid 500); 9 May 2011 17:26:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63250 invoked by uid 99); 9 May 2011 17:26:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 17:26:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 17:26:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 53A65C8F68
	for <common-issues@hadoop.apache.org>; Mon,  9 May 2011 17:26:03 +0000 (UTC)
Date: Mon, 9 May 2011 17:26:03 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1834333561.32850.1304961963339.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1362418509.26364.1304633343208.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7261) Disable IPV6 for junit tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030822#comment-13030822 ] 

Matt Foley commented on HADOOP-7261:
------------------------------------

Yes, please see further https://issues.apache.org/jira/browse/HDFS-1891?focusedCommentId=13029086&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13029086"

These fixes are for the junit and contrib junit tests, which are not adequately covered by the modification to hadoop-config.sh done in HADOOP-6056.

> Disable IPV6 for junit tests
> ----------------------------
>
>                 Key: HADOOP-7261
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7261
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7261.1.patch, HADOOP-7261.patch
>
>
> IPV6 addresses not handles currently in the common library methods. IPV6 can return address as "0:0:0:0:0:0:port". Some utility methods such as NetUtils#createSocketAddress(), NetUtils#normalizeHostName(), NetUtils#getHostNameOfIp() to name a few, do not handle IPV6 address and expect address to be of format host:port.
> Until IPV6 is formally supported, I propose disabling IPV6 for junit tests to avoid problems seen in HDFS-1891.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14585-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  9 17:56:46 2011
Return-Path: <common-issues-return-14585-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9C1794045
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  9 May 2011 17:56:46 +0000 (UTC)
Received: (qmail 14781 invoked by uid 500); 9 May 2011 17:56:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14750 invoked by uid 500); 9 May 2011 17:56:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14742 invoked by uid 99); 9 May 2011 17:56:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 17:56:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 17:56:44 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BBC20C8879
	for <common-issues@hadoop.apache.org>; Mon,  9 May 2011 17:56:04 +0000 (UTC)
Date: Mon, 9 May 2011 17:56:04 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <378830390.32903.1304963764765.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-6255:
------------------------------

    Attachment: HADOOP-6255-common-trunk-2.patch

Fix merge conflicts with most recent svn code.
Polish usage display message for hadoop-setup-hdfs.sh.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14586-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  9 18:14:46 2011
Return-Path: <common-issues-return-14586-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C238E4973
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  9 May 2011 18:14:46 +0000 (UTC)
Received: (qmail 37437 invoked by uid 500); 9 May 2011 18:14:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37406 invoked by uid 500); 9 May 2011 18:14:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37396 invoked by uid 99); 9 May 2011 18:14:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 18:14:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 18:14:44 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7FAD2C8E8B
	for <common-issues@hadoop.apache.org>; Mon,  9 May 2011 18:14:04 +0000 (UTC)
Date: Mon, 9 May 2011 18:14:04 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <527207597.32976.1304964844519.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030837#comment-13030837 ] 

Hadoop QA commented on HADOOP-6255:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478624/HADOOP-6255-common-trunk-2.patch
  against trunk revision 1100400.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 27 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    -1 release audit.  The applied patch generated 3 release audit warnings (more than the trunk's current 1 warnings).

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/418//testReport/
Release audit warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/418//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/418//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/418//console

This message is automatically generated.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14587-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  9 18:24:45 2011
Return-Path: <common-issues-return-14587-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2FA7B4100
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  9 May 2011 18:24:45 +0000 (UTC)
Received: (qmail 51290 invoked by uid 500); 9 May 2011 18:24:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51260 invoked by uid 500); 9 May 2011 18:24:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51252 invoked by uid 99); 9 May 2011 18:24:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 18:24:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 18:24:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3D5A4C8207
	for <common-issues@hadoop.apache.org>; Mon,  9 May 2011 18:24:03 +0000 (UTC)
Date: Mon, 9 May 2011 18:24:03 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2104594010.32996.1304965443247.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1142652053.28874.1304709483167.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7265) Keep track of relative paths
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7265:
-------------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I have committed this.  Thanks Daryn!

Also thanks Tanping for reviewing it.

> Keep track of relative paths
> ----------------------------
>
>                 Key: HADOOP-7265
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7265
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7265.patch
>
>
> As part of the effort to standardize the display of paths, the PathData tracks the exact string used to create a path.  When obtaining a directory's contents, the relative nature of the original path should be preserved.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14588-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  9 18:36:45 2011
Return-Path: <common-issues-return-14588-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1F9B04984
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  9 May 2011 18:36:45 +0000 (UTC)
Received: (qmail 74375 invoked by uid 500); 9 May 2011 18:36:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74345 invoked by uid 500); 9 May 2011 18:36:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74337 invoked by uid 99); 9 May 2011 18:36:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 18:36:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 18:36:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3264CC86BA
	for <common-issues@hadoop.apache.org>; Mon,  9 May 2011 18:36:03 +0000 (UTC)
Date: Mon, 9 May 2011 18:36:03 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1426015920.33018.1304966163203.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1142652053.28874.1304709483167.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7265) Keep track of relative paths
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030847#comment-13030847 ] 

Hudson commented on HADOOP-7265:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #581 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/581/])
    HADOOP-7265. Keep track of relative paths in PathData.  Contributed by Daryn Sharp


> Keep track of relative paths
> ----------------------------
>
>                 Key: HADOOP-7265
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7265
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7265.patch
>
>
> As part of the effort to standardize the display of paths, the PathData tracks the exact string used to create a path.  When obtaining a directory's contents, the relative nature of the original path should be preserved.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14589-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  9 19:15:45 2011
Return-Path: <common-issues-return-14589-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EEF504405
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  9 May 2011 19:15:44 +0000 (UTC)
Received: (qmail 30560 invoked by uid 500); 9 May 2011 19:15:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30522 invoked by uid 500); 9 May 2011 19:15:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30514 invoked by uid 99); 9 May 2011 19:15:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 19:15:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 19:15:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2B86AC8477
	for <common-issues@hadoop.apache.org>; Mon,  9 May 2011 19:15:03 +0000 (UTC)
Date: Mon, 9 May 2011 19:15:03 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <798237261.33129.1304968503174.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <651142754.74841.1303423565756.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7238) Refactor FsShell's cat & text
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7238:
--------------------------------

    Attachment: HADOOP-7238-2.patch

update patch to resolve conflicts in if/then/else chains caused by removal of other commands

> Refactor FsShell's cat & text
> -----------------------------
>
>                 Key: HADOOP-7238
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7238
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7238-2.patch, HADOOP-7238.patch
>
>
> Need to refactor cat & text to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14590-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  9 19:33:43 2011
Return-Path: <common-issues-return-14590-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E7BE14C91
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  9 May 2011 19:33:42 +0000 (UTC)
Received: (qmail 52215 invoked by uid 500); 9 May 2011 19:33:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52186 invoked by uid 500); 9 May 2011 19:33:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52174 invoked by uid 99); 9 May 2011 19:33:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 19:33:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 19:33:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2B3F9C899B
	for <common-issues@hadoop.apache.org>; Mon,  9 May 2011 19:33:03 +0000 (UTC)
Date: Mon, 9 May 2011 19:33:03 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <956153629.33143.1304969583173.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <651142754.74841.1303423565756.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7238) Refactor FsShell's cat & text
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030868#comment-13030868 ] 

Hadoop QA commented on HADOOP-7238:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478631/HADOOP-7238-2.patch
  against trunk revision 1101132.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/419//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/419//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/419//console

This message is automatically generated.

> Refactor FsShell's cat & text
> -----------------------------
>
>                 Key: HADOOP-7238
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7238
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7238-2.patch, HADOOP-7238.patch
>
>
> Need to refactor cat & text to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14591-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  9 20:14:45 2011
Return-Path: <common-issues-return-14591-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0B8EA4525
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  9 May 2011 20:14:45 +0000 (UTC)
Received: (qmail 4243 invoked by uid 500); 9 May 2011 20:14:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4213 invoked by uid 500); 9 May 2011 20:14:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4204 invoked by uid 99); 9 May 2011 20:14:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 20:14:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 20:14:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2DB87C5704
	for <common-issues@hadoop.apache.org>; Mon,  9 May 2011 20:14:03 +0000 (UTC)
Date: Mon, 9 May 2011 20:14:03 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2134590962.33220.1304972043183.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <651142754.74841.1303423565756.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7238) Refactor FsShell's cat & text
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7238:
-------------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I have committed this.  Thanks, Daryn!

Also thanks Matt for reviewing it.

> Refactor FsShell's cat & text
> -----------------------------
>
>                 Key: HADOOP-7238
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7238
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7238-2.patch, HADOOP-7238.patch
>
>
> Need to refactor cat & text to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14592-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  9 20:24:43 2011
Return-Path: <common-issues-return-14592-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 480894D0E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  9 May 2011 20:24:43 +0000 (UTC)
Received: (qmail 25006 invoked by uid 500); 9 May 2011 20:24:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24933 invoked by uid 500); 9 May 2011 20:24:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24803 invoked by uid 99); 9 May 2011 20:24:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 20:24:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 20:24:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4CFA4C5AAC
	for <common-issues@hadoop.apache.org>; Mon,  9 May 2011 20:24:03 +0000 (UTC)
Date: Mon, 9 May 2011 20:24:03 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1026835623.33234.1304972643311.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <651142754.74841.1303423565756.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7238) Refactor FsShell's cat & text
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030886#comment-13030886 ] 

Hudson commented on HADOOP-7238:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #582 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/582/])
    HADOOP-7238. Refactor the cat and text commands to conform to new FsCommand class.  Contributed by Daryn Sharp


> Refactor FsShell's cat & text
> -----------------------------
>
>                 Key: HADOOP-7238
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7238
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7238-2.patch, HADOOP-7238.patch
>
>
> Need to refactor cat & text to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14593-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  9 20:50:45 2011
Return-Path: <common-issues-return-14593-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 196614E1E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  9 May 2011 20:50:45 +0000 (UTC)
Received: (qmail 70960 invoked by uid 500); 9 May 2011 20:50:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70936 invoked by uid 500); 9 May 2011 20:50:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70928 invoked by uid 99); 9 May 2011 20:50:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 20:50:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 20:50:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 34F45C830A
	for <common-issues@hadoop.apache.org>; Mon,  9 May 2011 20:50:03 +0000 (UTC)
Date: Mon, 9 May 2011 20:50:03 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1203422127.33287.1304974203213.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1912220915.31712.1304917443296.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7268) FileContext.getLocalFSFileContext()
 behavior needs to be fixed w.r.t tokens
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey reassigned HADOOP-7268:
--------------------------------------------

    Assignee: Jitendra Nath Pandey

> FileContext.getLocalFSFileContext() behavior needs to be fixed w.r.t tokens
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-7268
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7268
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, security
>    Affects Versions: 0.23.0
>            Reporter: Devaraj Das
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>
> FileContext.getLocalFSFileContext() instantiates a FileContext object upon the first call to it, and for all subsequent calls returns back that instance (a static localFsSingleton object). With security turned on, this causes some hard-to-debug situations when that fileContext is used for doing HDFS operations. This is because the UserGroupInformation is stored when a FileContext is instantiated. If the process in question wishes to use different UserGroupInformation objects for different file system operations (where the corresponding FileContext objects are obtained via calls to FileContext.getLocalFSFileContext()), it doesn't work.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14594-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  9 20:54:45 2011
Return-Path: <common-issues-return-14594-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 39EEF4EA3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  9 May 2011 20:54:45 +0000 (UTC)
Received: (qmail 75290 invoked by uid 500); 9 May 2011 20:54:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75241 invoked by uid 500); 9 May 2011 20:54:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75233 invoked by uid 99); 9 May 2011 20:54:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 20:54:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 20:54:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 31BE2C8486
	for <common-issues@hadoop.apache.org>; Mon,  9 May 2011 20:54:03 +0000 (UTC)
Date: Mon, 9 May 2011 20:54:03 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1219791200.33292.1304974443200.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1912220915.31712.1304917443296.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7268) FileContext.getLocalFSFileContext()
 behavior needs to be fixed w.r.t tokens
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7268:
-----------------------------------------

    Attachment: HADOOP-7268.1.patch

This patch removes the singleton for localfs file context. A new file context object will be created when getLocalFSFileContext() is called. 

> FileContext.getLocalFSFileContext() behavior needs to be fixed w.r.t tokens
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-7268
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7268
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, security
>    Affects Versions: 0.23.0
>            Reporter: Devaraj Das
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7268.1.patch
>
>
> FileContext.getLocalFSFileContext() instantiates a FileContext object upon the first call to it, and for all subsequent calls returns back that instance (a static localFsSingleton object). With security turned on, this causes some hard-to-debug situations when that fileContext is used for doing HDFS operations. This is because the UserGroupInformation is stored when a FileContext is instantiated. If the process in question wishes to use different UserGroupInformation objects for different file system operations (where the corresponding FileContext objects are obtained via calls to FileContext.getLocalFSFileContext()), it doesn't work.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14595-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  9 20:54:45 2011
Return-Path: <common-issues-return-14595-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6C9064EB2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  9 May 2011 20:54:45 +0000 (UTC)
Received: (qmail 75508 invoked by uid 500); 9 May 2011 20:54:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75469 invoked by uid 500); 9 May 2011 20:54:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75355 invoked by uid 99); 9 May 2011 20:54:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 20:54:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 20:54:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5030FC848C
	for <common-issues@hadoop.apache.org>; Mon,  9 May 2011 20:54:03 +0000 (UTC)
Date: Mon, 9 May 2011 20:54:03 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1715973682.33296.1304974443325.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1912220915.31712.1304917443296.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7268) FileContext.getLocalFSFileContext()
 behavior needs to be fixed w.r.t tokens
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7268:
-----------------------------------------

    Status: Patch Available  (was: Open)

> FileContext.getLocalFSFileContext() behavior needs to be fixed w.r.t tokens
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-7268
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7268
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, security
>    Affects Versions: 0.23.0
>            Reporter: Devaraj Das
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7268.1.patch
>
>
> FileContext.getLocalFSFileContext() instantiates a FileContext object upon the first call to it, and for all subsequent calls returns back that instance (a static localFsSingleton object). With security turned on, this causes some hard-to-debug situations when that fileContext is used for doing HDFS operations. This is because the UserGroupInformation is stored when a FileContext is instantiated. If the process in question wishes to use different UserGroupInformation objects for different file system operations (where the corresponding FileContext objects are obtained via calls to FileContext.getLocalFSFileContext()), it doesn't work.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14596-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  9 21:14:45 2011
Return-Path: <common-issues-return-14596-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0A1014984
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  9 May 2011 21:14:45 +0000 (UTC)
Received: (qmail 7631 invoked by uid 500); 9 May 2011 21:14:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7595 invoked by uid 500); 9 May 2011 21:14:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7587 invoked by uid 99); 9 May 2011 21:14:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 21:14:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 21:14:43 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 17B4FC8A39
	for <common-issues@hadoop.apache.org>; Mon,  9 May 2011 21:14:05 +0000 (UTC)
Date: Mon, 9 May 2011 21:14:05 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <746722399.33311.1304975645093.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1912220915.31712.1304917443296.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7268)
 FileContext.getLocalFSFileContext() behavior needs to be fixed w.r.t tokens
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030899#comment-13030899 ] 

Hadoop QA commented on HADOOP-7268:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478634/HADOOP-7268.1.patch
  against trunk revision 1101199.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/420//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/420//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/420//console

This message is automatically generated.

> FileContext.getLocalFSFileContext() behavior needs to be fixed w.r.t tokens
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-7268
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7268
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, security
>    Affects Versions: 0.23.0
>            Reporter: Devaraj Das
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7268.1.patch
>
>
> FileContext.getLocalFSFileContext() instantiates a FileContext object upon the first call to it, and for all subsequent calls returns back that instance (a static localFsSingleton object). With security turned on, this causes some hard-to-debug situations when that fileContext is used for doing HDFS operations. This is because the UserGroupInformation is stored when a FileContext is instantiated. If the process in question wishes to use different UserGroupInformation objects for different file system operations (where the corresponding FileContext objects are obtained via calls to FileContext.getLocalFSFileContext()), it doesn't work.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14597-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  9 21:22:45 2011
Return-Path: <common-issues-return-14597-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 09DAE406C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  9 May 2011 21:22:45 +0000 (UTC)
Received: (qmail 20802 invoked by uid 500); 9 May 2011 21:22:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20755 invoked by uid 500); 9 May 2011 21:22:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20747 invoked by uid 99); 9 May 2011 21:22:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 21:22:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 21:22:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2C38DC8C52
	for <common-issues@hadoop.apache.org>; Mon,  9 May 2011 21:22:03 +0000 (UTC)
Date: Mon, 9 May 2011 21:22:03 +0000 (UTC)
From: "Koji Noguchi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2112834732.33319.1304976123177.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1327657029.27279.1304663043230.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7262) Update CS docs with better example
 configs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030901#comment-13030901 ] 

Koji Noguchi commented on HADOOP-7262:
--------------------------------------

Arun, "mapred.capacity-scheduler.queue.<queue-name>" is repeated many times on the tables in your pdf doc.  We should fill in the right conf names.

> Update CS docs with better example configs
> ------------------------------------------
>
>                 Key: HADOOP-7262
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7262
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>            Priority: Minor
>             Fix For: 0.20.204.0
>
>         Attachments: HADOOP-7262.patch, capacity_scheduler.pdf
>
>
> It will be nice to enhance CS docs with real-world example configs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14598-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May  9 21:40:43 2011
Return-Path: <common-issues-return-14598-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3FF144A7C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon,  9 May 2011 21:40:43 +0000 (UTC)
Received: (qmail 52359 invoked by uid 500); 9 May 2011 21:40:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52309 invoked by uid 500); 9 May 2011 21:40:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52269 invoked by uid 99); 9 May 2011 21:40:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 21:40:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 May 2011 21:40:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3D9BDC82A5
	for <common-issues@hadoop.apache.org>; Mon,  9 May 2011 21:40:03 +0000 (UTC)
Date: Mon, 9 May 2011 21:40:03 +0000 (UTC)
From: "Koji Noguchi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1731762492.33355.1304977203248.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1327657029.27279.1304663043230.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7262) Update CS docs with better example
 configs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030912#comment-13030912 ] 

Koji Noguchi commented on HADOOP-7262:
--------------------------------------

bq. Capacity Guarantees
bq.
Isn't 'guarantee' a strong word to use for the current capacity scheduler?
This is only possible when all of the queues have max-limit equal to queue capacity (since we don't have preemption).

bq. If true, priorities of jobs will be taken into account in scheduling decisions. 
bq.
On our internal cluster, we've never tried this.  It was once pointed out that we still have  priority inversion problems(MAPREDUCE:314) and should not enable this conf.  Was this fixed ?  If not, should we mention that it is not being used/tested yet?



> Update CS docs with better example configs
> ------------------------------------------
>
>                 Key: HADOOP-7262
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7262
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>            Priority: Minor
>             Fix For: 0.20.204.0
>
>         Attachments: HADOOP-7262.patch, capacity_scheduler.pdf
>
>
> It will be nice to enhance CS docs with real-world example configs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14599-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 00:46:42 2011
Return-Path: <common-issues-return-14599-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D83A14EDA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 00:46:42 +0000 (UTC)
Received: (qmail 89963 invoked by uid 500); 10 May 2011 00:46:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89931 invoked by uid 500); 10 May 2011 00:46:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89922 invoked by uid 99); 10 May 2011 00:46:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 00:46:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 00:46:41 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2DE82C9DEE
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 00:46:03 +0000 (UTC)
Date: Tue, 10 May 2011 00:46:03 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1366846970.33629.1304988363184.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7270) Server$Listener.getAddress(..)
 throws NullPointerException
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Server$Listener.getAddress(..) throws NullPointerException
----------------------------------------------------------

                 Key: HADOOP-7270
                 URL: https://issues.apache.org/jira/browse/HADOOP-7270
             Project: Hadoop Common
          Issue Type: Bug
          Components: ipc
            Reporter: Tsz Wo (Nicholas), SZE


In [build #469|https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/469//testReport/org.apache.hadoop.hdfs.server.datanode/TestFiDataTransferProtocol2/pipeline_Fi_29/],
{noformat}
2011-05-09 23:21:35,528 ERROR datanode.DataNode (DataXceiver.java:run(133)) - 127.0.0.1:57149:DataXceiver
java.lang.NullPointerException
	at org.apache.hadoop.ipc.Server$Listener.getAddress(Server.java:518)
	at org.apache.hadoop.ipc.Server.getListenerAddress(Server.java:1662)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.getIpcPort(DataNode.java:1461)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.getDatanodeId(DataNode.java:2747)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiverAspects.ajc$after$org_apache_hadoop_hdfs_server_datanode_BlockReceiverAspects$9$725950a6(BlockReceiverAspects.aj:226)
	at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.close(BlockReceiver.java:230)
	at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:157)
	at org.apache.hadoop.io.IOUtils.closeStream(IOUtils.java:173)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.opWriteBlock(DataXceiver.java:408)
	at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$Receiver.opWriteBlock(DataTransferProtocol.java:414)
	at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$Receiver.processOp(DataTransferProtocol.java:360)
	at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:131)
	at java.lang.Thread.run(Thread.java:662)
{noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14600-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 11:15:45 2011
Return-Path: <common-issues-return-14600-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 33E125C9D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 11:15:44 +0000 (UTC)
Received: (qmail 47769 invoked by uid 500); 10 May 2011 11:15:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47554 invoked by uid 500); 10 May 2011 11:15:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47538 invoked by uid 99); 10 May 2011 11:15:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 11:15:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 11:15:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 538C0C98E9
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 11:15:03 +0000 (UTC)
Date: Tue, 10 May 2011 11:15:03 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1550515234.34402.1305026103339.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1142652053.28874.1304709483167.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7265) Keep track of relative paths
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031123#comment-13031123 ] 

Hudson commented on HADOOP-7265:
--------------------------------

Integrated in Hadoop-Common-trunk #684 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/684/])
    HADOOP-7265. Keep track of relative paths in PathData.  Contributed by Daryn Sharp


> Keep track of relative paths
> ----------------------------
>
>                 Key: HADOOP-7265
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7265
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7265.patch
>
>
> As part of the effort to standardize the display of paths, the PathData tracks the exact string used to create a path.  When obtaining a directory's contents, the relative nature of the original path should be preserved.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14601-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 11:15:46 2011
Return-Path: <common-issues-return-14601-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 343675C9F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 11:15:44 +0000 (UTC)
Received: (qmail 47835 invoked by uid 500); 10 May 2011 11:15:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47562 invoked by uid 500); 10 May 2011 11:15:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47542 invoked by uid 99); 10 May 2011 11:15:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 11:15:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 11:15:42 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3B7C8C98E6
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 11:15:03 +0000 (UTC)
Date: Tue, 10 May 2011 11:15:03 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1131780699.34399.1305026103240.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <651142754.74841.1303423565756.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7238) Refactor FsShell's cat & text
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031122#comment-13031122 ] 

Hudson commented on HADOOP-7238:
--------------------------------

Integrated in Hadoop-Common-trunk #684 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/684/])
    HADOOP-7238. Refactor the cat and text commands to conform to new FsCommand class.  Contributed by Daryn Sharp


> Refactor FsShell's cat & text
> -----------------------------
>
>                 Key: HADOOP-7238
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7238
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7238-2.patch, HADOOP-7238.patch
>
>
> Need to refactor cat & text to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14602-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 16:57:33 2011
Return-Path: <common-issues-return-14602-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E1B4D564C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 16:57:32 +0000 (UTC)
Received: (qmail 32506 invoked by uid 500); 10 May 2011 16:57:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32462 invoked by uid 500); 10 May 2011 16:57:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32454 invoked by uid 99); 10 May 2011 16:57:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 16:57:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 16:57:31 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B858736F12
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 16:56:52 +0000 (UTC)
Date: Tue, 10 May 2011 16:56:52 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1782899901.550.1305046612751.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-6255:
------------------------------

    Attachment: HADOOP-6255-mapred-trunk-2.patch

Update patch to reflect the current state of trunk

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-6255-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14603-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 17:03:31 2011
Return-Path: <common-issues-return-14603-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E774B5789
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 17:03:30 +0000 (UTC)
Received: (qmail 41690 invoked by uid 500); 10 May 2011 17:03:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41664 invoked by uid 500); 10 May 2011 17:03:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41656 invoked by uid 99); 10 May 2011 17:03:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 17:03:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 17:03:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F3062441C3
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 17:02:48 +0000 (UTC)
Date: Tue, 10 May 2011 17:02:48 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1412040970.594.1305046968992.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031255#comment-13031255 ] 

Hadoop QA commented on HADOOP-6255:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478710/HADOOP-6255-mapred-trunk-2.patch
  against trunk revision 1101199.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 15 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/421//console

This message is automatically generated.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-6255-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14604-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 17:15:30 2011
Return-Path: <common-issues-return-14604-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ED6A850B1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 17:15:29 +0000 (UTC)
Received: (qmail 58497 invoked by uid 500); 10 May 2011 17:15:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58449 invoked by uid 500); 10 May 2011 17:15:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58441 invoked by uid 99); 10 May 2011 17:15:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 17:15:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 17:15:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0DAB944578
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 17:14:48 +0000 (UTC)
Date: Tue, 10 May 2011 17:14:48 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <229413150.627.1305047688052.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7257) A client side mount table to give
 per-application/per-job file system view
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7257:
---------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Release Note: viewfs - client-side mount table.
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

> A client side mount table to give per-application/per-job file system view
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7257
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7257
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: ViewFs - javdoc2.pdf, ViewFsJavaDoc.pdf, viewFs2.patch, viewFs3.patch, viewFs4.patch, viewFs5.patch, viewfs1.patch
>
>
> This jira proposes a client side mount table to allow application-centric (or job-centric) filesystem views. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14605-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 17:46:30 2011
Return-Path: <common-issues-return-14605-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 06B955131
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 17:46:30 +0000 (UTC)
Received: (qmail 10777 invoked by uid 500); 10 May 2011 17:46:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10748 invoked by uid 500); 10 May 2011 17:46:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10740 invoked by uid 99); 10 May 2011 17:46:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 17:46:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 17:46:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B962736005
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 17:45:47 +0000 (UTC)
Date: Tue, 10 May 2011 17:45:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1331797428.759.1305049547756.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1912220915.31712.1304917443296.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7268)
 FileContext.getLocalFSFileContext() behavior needs to be fixed w.r.t tokens
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031270#comment-13031270 ] 

Sanjay Radia commented on HADOOP-7268:
--------------------------------------

+1

> FileContext.getLocalFSFileContext() behavior needs to be fixed w.r.t tokens
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-7268
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7268
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, security
>    Affects Versions: 0.23.0
>            Reporter: Devaraj Das
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7268.1.patch
>
>
> FileContext.getLocalFSFileContext() instantiates a FileContext object upon the first call to it, and for all subsequent calls returns back that instance (a static localFsSingleton object). With security turned on, this causes some hard-to-debug situations when that fileContext is used for doing HDFS operations. This is because the UserGroupInformation is stored when a FileContext is instantiated. If the process in question wishes to use different UserGroupInformation objects for different file system operations (where the corresponding FileContext objects are obtained via calls to FileContext.getLocalFSFileContext()), it doesn't work.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14606-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 18:04:29 2011
Return-Path: <common-issues-return-14606-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BC46B5882
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 18:04:29 +0000 (UTC)
Received: (qmail 53655 invoked by uid 500); 10 May 2011 18:04:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53621 invoked by uid 500); 10 May 2011 18:04:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53610 invoked by uid 99); 10 May 2011 18:04:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 18:04:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 18:04:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 62B5A367E9
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 18:03:47 +0000 (UTC)
Date: Tue, 10 May 2011 18:03:47 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <854067926.796.1305050627400.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1912220915.31712.1304917443296.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7268) FileContext.getLocalFSFileContext()
 behavior needs to be fixed w.r.t tokens
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7268:
-----------------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I have committed this.

> FileContext.getLocalFSFileContext() behavior needs to be fixed w.r.t tokens
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-7268
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7268
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, security
>    Affects Versions: 0.23.0
>            Reporter: Devaraj Das
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7268.1.patch
>
>
> FileContext.getLocalFSFileContext() instantiates a FileContext object upon the first call to it, and for all subsequent calls returns back that instance (a static localFsSingleton object). With security turned on, this causes some hard-to-debug situations when that fileContext is used for doing HDFS operations. This is because the UserGroupInformation is stored when a FileContext is instantiated. If the process in question wishes to use different UserGroupInformation objects for different file system operations (where the corresponding FileContext objects are obtained via calls to FileContext.getLocalFSFileContext()), it doesn't work.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14607-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 18:06:30 2011
Return-Path: <common-issues-return-14607-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3689E58E8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 18:06:30 +0000 (UTC)
Received: (qmail 54866 invoked by uid 500); 10 May 2011 18:06:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54827 invoked by uid 500); 10 May 2011 18:06:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54787 invoked by uid 99); 10 May 2011 18:06:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 18:06:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 18:06:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5CD32368C0
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 18:05:47 +0000 (UTC)
Date: Tue, 10 May 2011 18:05:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2068463279.801.1305050747377.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7271) Standardize error messages
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Standardize error messages
--------------------------

                 Key: HADOOP-7271
                 URL: https://issues.apache.org/jira/browse/HADOOP-7271
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs
    Affects Versions: 0.23.0
            Reporter: Daryn Sharp
            Assignee: Daryn Sharp


The FsShell commands have no standard format for the same error message.  For instance, here is a snippet of the variations of just one of many error messages:

cmd: $path: No such file or directory
cmd: cannot stat `$path': No such file or directory
cmd: Can not find listing for $path
cmd: Cannot access $path: No such file or directory.
cmd: No such file or directory `$path'
cmd: File does not exist: $path
cmd: File $path does not exist
... etc ...

These need to be common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14608-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 18:14:27 2011
Return-Path: <common-issues-return-14608-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2762A5B67
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 18:14:27 +0000 (UTC)
Received: (qmail 71138 invoked by uid 500); 10 May 2011 18:14:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71108 invoked by uid 500); 10 May 2011 18:14:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71100 invoked by uid 99); 10 May 2011 18:14:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 18:14:26 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 18:14:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 686E836B85
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 18:13:47 +0000 (UTC)
Date: Tue, 10 May 2011 18:13:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <479801521.811.1305051227424.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2068463279.801.1305050747377.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7271) Standardize error messages
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7271:
--------------------------------

    Status: Patch Available  (was: Open)

> Standardize error messages
> --------------------------
>
>                 Key: HADOOP-7271
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7271
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7271.patch
>
>
> The FsShell commands have no standard format for the same error message.  For instance, here is a snippet of the variations of just one of many error messages:
> cmd: $path: No such file or directory
> cmd: cannot stat `$path': No such file or directory
> cmd: Can not find listing for $path
> cmd: Cannot access $path: No such file or directory.
> cmd: No such file or directory `$path'
> cmd: File does not exist: $path
> cmd: File $path does not exist
> ... etc ...
> These need to be common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14609-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 18:14:29 2011
Return-Path: <common-issues-return-14609-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 590715B74
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 18:14:29 +0000 (UTC)
Received: (qmail 71396 invoked by uid 500); 10 May 2011 18:14:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71356 invoked by uid 500); 10 May 2011 18:14:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71348 invoked by uid 99); 10 May 2011 18:14:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 18:14:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 18:14:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5ADAD36B83
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 18:13:47 +0000 (UTC)
Date: Tue, 10 May 2011 18:13:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <855353749.809.1305051227368.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2068463279.801.1305050747377.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7271) Standardize error messages
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7271:
--------------------------------

    Attachment: HADOOP-7271.patch

Create path exceptions that encapsulate and enforce a standard posix/unix error message.

Change cmds, both redesigned and pending redesign, to use the new exceptions.  The changes made to the pending cmds are to facilitate a redesign of multiple commands w/o constantly breaking hdfs tests each time a command is updated.

Took out some of the TODO items related to standardizing.


> Standardize error messages
> --------------------------
>
>                 Key: HADOOP-7271
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7271
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7271.patch
>
>
> The FsShell commands have no standard format for the same error message.  For instance, here is a snippet of the variations of just one of many error messages:
> cmd: $path: No such file or directory
> cmd: cannot stat `$path': No such file or directory
> cmd: Can not find listing for $path
> cmd: Cannot access $path: No such file or directory.
> cmd: No such file or directory `$path'
> cmd: File does not exist: $path
> cmd: File $path does not exist
> ... etc ...
> These need to be common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14610-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 18:24:27 2011
Return-Path: <common-issues-return-14610-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6519352E5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 18:24:27 +0000 (UTC)
Received: (qmail 95514 invoked by uid 500); 10 May 2011 18:24:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95423 invoked by uid 500); 10 May 2011 18:24:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95415 invoked by uid 99); 10 May 2011 18:24:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 18:24:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 18:24:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7C96844031
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 18:23:47 +0000 (UTC)
Date: Tue, 10 May 2011 18:23:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1830376490.835.1305051827507.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1912220915.31712.1304917443296.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7268)
 FileContext.getLocalFSFileContext() behavior needs to be fixed w.r.t tokens
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031288#comment-13031288 ] 

Hudson commented on HADOOP-7268:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #583 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/583/])
    HADOOP-7268. FileContext.getLocalFSFileContext() behavior needs to be fixed w.r.t tokens. Contributed by jitendra.


> FileContext.getLocalFSFileContext() behavior needs to be fixed w.r.t tokens
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-7268
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7268
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, security
>    Affects Versions: 0.23.0
>            Reporter: Devaraj Das
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7268.1.patch
>
>
> FileContext.getLocalFSFileContext() instantiates a FileContext object upon the first call to it, and for all subsequent calls returns back that instance (a static localFsSingleton object). With security turned on, this causes some hard-to-debug situations when that fileContext is used for doing HDFS operations. This is because the UserGroupInformation is stored when a FileContext is instantiated. If the process in question wishes to use different UserGroupInformation objects for different file system operations (where the corresponding FileContext objects are obtained via calls to FileContext.getLocalFSFileContext()), it doesn't work.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14611-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 18:34:27 2011
Return-Path: <common-issues-return-14611-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A222659B2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 18:34:27 +0000 (UTC)
Received: (qmail 14643 invoked by uid 500); 10 May 2011 18:34:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14617 invoked by uid 500); 10 May 2011 18:34:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14608 invoked by uid 99); 10 May 2011 18:34:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 18:34:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 18:34:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 96F0544376
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 18:33:47 +0000 (UTC)
Date: Tue, 10 May 2011 18:33:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <755433965.849.1305052427615.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2068463279.801.1305050747377.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7271) Standardize error messages
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031290#comment-13031290 ] 

Hadoop QA commented on HADOOP-7271:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478716/HADOOP-7271.patch
  against trunk revision 1101570.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.fs.shell.TestPathExceptions

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/422//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/422//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/422//console

This message is automatically generated.

> Standardize error messages
> --------------------------
>
>                 Key: HADOOP-7271
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7271
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7271.patch
>
>
> The FsShell commands have no standard format for the same error message.  For instance, here is a snippet of the variations of just one of many error messages:
> cmd: $path: No such file or directory
> cmd: cannot stat `$path': No such file or directory
> cmd: Can not find listing for $path
> cmd: Cannot access $path: No such file or directory.
> cmd: No such file or directory `$path'
> cmd: File does not exist: $path
> cmd: File $path does not exist
> ... etc ...
> These need to be common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14612-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 18:48:27 2011
Return-Path: <common-issues-return-14612-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B880450B0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 18:48:27 +0000 (UTC)
Received: (qmail 39524 invoked by uid 500); 10 May 2011 18:48:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39479 invoked by uid 500); 10 May 2011 18:48:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39471 invoked by uid 99); 10 May 2011 18:48:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 18:48:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 18:48:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E263A448D7
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 18:47:47 +0000 (UTC)
Date: Tue, 10 May 2011 18:47:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <101532583.901.1305053267924.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2068463279.801.1305050747377.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7271) Standardize error messages
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7271:
--------------------------------

    Attachment: HADOOP-7271-2.patch

Add the apostrophes needed to make the tests pass.

> Standardize error messages
> --------------------------
>
>                 Key: HADOOP-7271
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7271
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7271-2.patch, HADOOP-7271.patch
>
>
> The FsShell commands have no standard format for the same error message.  For instance, here is a snippet of the variations of just one of many error messages:
> cmd: $path: No such file or directory
> cmd: cannot stat `$path': No such file or directory
> cmd: Can not find listing for $path
> cmd: Cannot access $path: No such file or directory.
> cmd: No such file or directory `$path'
> cmd: File does not exist: $path
> cmd: File $path does not exist
> ... etc ...
> These need to be common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14613-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 19:04:29 2011
Return-Path: <common-issues-return-14613-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B83FE59F3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 19:04:29 +0000 (UTC)
Received: (qmail 72767 invoked by uid 500); 10 May 2011 19:04:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72738 invoked by uid 500); 10 May 2011 19:04:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72730 invoked by uid 99); 10 May 2011 19:04:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 19:04:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 19:04:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7919744FA2
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 19:03:47 +0000 (UTC)
Date: Tue, 10 May 2011 19:03:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <656397007.942.1305054227492.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2068463279.801.1305050747377.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7271) Standardize error messages
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031314#comment-13031314 ] 

Hadoop QA commented on HADOOP-7271:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478720/HADOOP-7271-2.patch
  against trunk revision 1101570.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/423//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/423//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/423//console

This message is automatically generated.

> Standardize error messages
> --------------------------
>
>                 Key: HADOOP-7271
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7271
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7271-2.patch, HADOOP-7271.patch
>
>
> The FsShell commands have no standard format for the same error message.  For instance, here is a snippet of the variations of just one of many error messages:
> cmd: $path: No such file or directory
> cmd: cannot stat `$path': No such file or directory
> cmd: Can not find listing for $path
> cmd: Cannot access $path: No such file or directory.
> cmd: No such file or directory `$path'
> cmd: File does not exist: $path
> cmd: File $path does not exist
> ... etc ...
> These need to be common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14614-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 19:18:28 2011
Return-Path: <common-issues-return-14614-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E605058FE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 19:18:28 +0000 (UTC)
Received: (qmail 89627 invoked by uid 500); 10 May 2011 19:18:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89586 invoked by uid 500); 10 May 2011 19:18:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89578 invoked by uid 99); 10 May 2011 19:18:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 19:18:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 19:18:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0E3EB44419
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 19:17:49 +0000 (UTC)
Date: Tue, 10 May 2011 19:17:49 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <507860065.977.1305055069055.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6920) Metrics2: metrics instrumentation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-6920:
----------------------------

    Status: Patch Available  (was: Open)

> Metrics2: metrics instrumentation
> ---------------------------------
>
>                 Key: HADOOP-6920
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6920
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-6920-metrics2-inst-v4.patch
>
>
> The jira tracks the metrics instrumentation for the new framework, i.e., porting jvm and rpc metrics to the new framework.
> This issue (esp. the "incompatible change flag") depends on the outcome of the discussion (proposed in HADOOP-6728) on whether we should support backward compatibility (at some cost.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14615-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 19:18:30 2011
Return-Path: <common-issues-return-14615-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A79F65932
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 19:18:30 +0000 (UTC)
Received: (qmail 90131 invoked by uid 500); 10 May 2011 19:18:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90102 invoked by uid 500); 10 May 2011 19:18:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90094 invoked by uid 99); 10 May 2011 19:18:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 19:18:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 19:18:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B09444440C
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 19:17:48 +0000 (UTC)
Date: Tue, 10 May 2011 19:17:48 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2128893102.966.1305055068720.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6920) Metrics2: metrics instrumentation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-6920:
----------------------------

    Status: Open  (was: Patch Available)

> Metrics2: metrics instrumentation
> ---------------------------------
>
>                 Key: HADOOP-6920
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6920
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-6920-metrics2-inst-v4.patch
>
>
> The jira tracks the metrics instrumentation for the new framework, i.e., porting jvm and rpc metrics to the new framework.
> This issue (esp. the "incompatible change flag") depends on the outcome of the discussion (proposed in HADOOP-6728) on whether we should support backward compatibility (at some cost.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14616-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 20:17:33 2011
Return-Path: <common-issues-return-14616-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 45D455DBE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 20:17:33 +0000 (UTC)
Received: (qmail 90355 invoked by uid 500); 10 May 2011 20:17:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90323 invoked by uid 500); 10 May 2011 20:17:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90285 invoked by uid 99); 10 May 2011 20:17:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 20:17:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 20:17:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6401344866
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 20:16:47 +0000 (UTC)
Date: Tue, 10 May 2011 20:16:47 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1221827855.1111.1305058607406.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <21922935.186601292622961312.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7068) Ivy resolve force mode should be
 turned off by default
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031347#comment-13031347 ] 

Tom White commented on HADOOP-7068:
-----------------------------------

+1. I agree that this is very confusing and makes it almost impossible to do parallel builds on Jenkins boxes. I'm going to go ahead and commit this, HDFS-1544, and MAPREDUCE-2222, unless there are objections.

> Ivy resolve force mode should be turned off by default
> ------------------------------------------------------
>
>                 Key: HADOOP-7068
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7068
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>         Attachments: hadoop-7068-trunk-v1.patch, hadoop-7068-trunk-v2.patch
>
>
> The problem is introduced by  HADOOP-6486. Which have caused a lot of mysterious artifact issues (unable to downgrade or do parallel dev, without wiping out both m2 and ivy caches etc.) wasting countless hours of dev (many people's) time to track down the issue.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14617-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 20:41:27 2011
Return-Path: <common-issues-return-14617-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 943D651A8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 20:41:27 +0000 (UTC)
Received: (qmail 42287 invoked by uid 500); 10 May 2011 20:41:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42266 invoked by uid 500); 10 May 2011 20:41:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42258 invoked by uid 99); 10 May 2011 20:41:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 20:41:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 20:41:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B5350441C9
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 20:40:47 +0000 (UTC)
Date: Tue, 10 May 2011 20:40:47 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1260170137.1167.1305060047739.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6920) Metrics2: metrics instrumentation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031367#comment-13031367 ] 

Suresh Srinivas commented on HADOOP-6920:
-----------------------------------------

+1 for the change. This patch is also forward port from 203, the metrics related infrastructure change.

> Metrics2: metrics instrumentation
> ---------------------------------
>
>                 Key: HADOOP-6920
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6920
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-6920-metrics2-inst-v4.patch
>
>
> The jira tracks the metrics instrumentation for the new framework, i.e., porting jvm and rpc metrics to the new framework.
> This issue (esp. the "incompatible change flag") depends on the outcome of the discussion (proposed in HADOOP-6728) on whether we should support backward compatibility (at some cost.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14618-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 21:22:27 2011
Return-Path: <common-issues-return-14618-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 413D753F5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 21:22:27 +0000 (UTC)
Received: (qmail 91359 invoked by uid 500); 10 May 2011 21:22:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91330 invoked by uid 500); 10 May 2011 21:22:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91321 invoked by uid 99); 10 May 2011 21:22:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 21:22:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 21:22:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 515D144E42
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 21:21:47 +0000 (UTC)
Date: Tue, 10 May 2011 21:21:47 +0000 (UTC)
From: "John George (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <210442175.1231.1305062507329.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2068463279.801.1305050747377.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7271) Standardize error messages
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031383#comment-13031383 ] 

John George commented on HADOOP-7271:
-------------------------------------

+1. looks good to me.

> Standardize error messages
> --------------------------
>
>                 Key: HADOOP-7271
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7271
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7271-2.patch, HADOOP-7271.patch
>
>
> The FsShell commands have no standard format for the same error message.  For instance, here is a snippet of the variations of just one of many error messages:
> cmd: $path: No such file or directory
> cmd: cannot stat `$path': No such file or directory
> cmd: Can not find listing for $path
> cmd: Cannot access $path: No such file or directory.
> cmd: No such file or directory `$path'
> cmd: File does not exist: $path
> cmd: File $path does not exist
> ... etc ...
> These need to be common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14619-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 21:34:27 2011
Return-Path: <common-issues-return-14619-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 702AC59D9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 21:34:27 +0000 (UTC)
Received: (qmail 7944 invoked by uid 500); 10 May 2011 21:34:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7915 invoked by uid 500); 10 May 2011 21:34:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7907 invoked by uid 99); 10 May 2011 21:34:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 21:34:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 21:34:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9790A4432E
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 21:33:47 +0000 (UTC)
Date: Tue, 10 May 2011 21:33:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2142799800.1261.1305063227617.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2068463279.801.1305050747377.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7271) Standardize error messages
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7271:
-------------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I have committed this.  Thanks, Daryn!

Also thanks John for reviewing it.

> Standardize error messages
> --------------------------
>
>                 Key: HADOOP-7271
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7271
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7271-2.patch, HADOOP-7271.patch
>
>
> The FsShell commands have no standard format for the same error message.  For instance, here is a snippet of the variations of just one of many error messages:
> cmd: $path: No such file or directory
> cmd: cannot stat `$path': No such file or directory
> cmd: Can not find listing for $path
> cmd: Cannot access $path: No such file or directory.
> cmd: No such file or directory `$path'
> cmd: File does not exist: $path
> cmd: File $path does not exist
> ... etc ...
> These need to be common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14620-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 21:46:27 2011
Return-Path: <common-issues-return-14620-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2010D5488
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 21:46:27 +0000 (UTC)
Received: (qmail 25265 invoked by uid 500); 10 May 2011 21:46:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25222 invoked by uid 500); 10 May 2011 21:46:26 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25214 invoked by uid 99); 10 May 2011 21:46:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 21:46:26 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 21:46:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6300844808
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 21:45:47 +0000 (UTC)
Date: Tue, 10 May 2011 21:45:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <463771946.1317.1305063947402.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2068463279.801.1305050747377.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7271) Standardize error messages
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031400#comment-13031400 ] 

Hudson commented on HADOOP-7271:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #584 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/584/])
    HADOOP-7271. Standardize shell command error messages.  Contributed by Daryn Sharp


> Standardize error messages
> --------------------------
>
>                 Key: HADOOP-7271
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7271
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7271-2.patch, HADOOP-7271.patch
>
>
> The FsShell commands have no standard format for the same error message.  For instance, here is a snippet of the variations of just one of many error messages:
> cmd: $path: No such file or directory
> cmd: cannot stat `$path': No such file or directory
> cmd: Can not find listing for $path
> cmd: Cannot access $path: No such file or directory.
> cmd: No such file or directory `$path'
> cmd: File does not exist: $path
> cmd: File $path does not exist
> ... etc ...
> These need to be common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14621-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 21:58:28 2011
Return-Path: <common-issues-return-14621-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 12643522C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 21:58:28 +0000 (UTC)
Received: (qmail 34686 invoked by uid 500); 10 May 2011 21:58:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34659 invoked by uid 500); 10 May 2011 21:58:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34649 invoked by uid 99); 10 May 2011 21:58:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 21:58:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 21:58:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 25DDB44C6A
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 21:57:48 +0000 (UTC)
Date: Tue, 10 May 2011 21:57:48 +0000 (UTC)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <465254807.1357.1305064668151.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <285327525.9499.1304014323342.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7248) Have a way to automatically update
 Eclipse .classpath file when new libs are added to the classpath through
 Ivy for 0.20-* based sources
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-7248:
----------------------------------

    Attachment: C7248-1.patch

This section:
{noformat}
+        <source path="${test.src.dir}/mapred"
+                output="${build.dir.eclipse-test-classes}" />
{noformat}

Didn't work on git, because the src/test/mapred dirs are empty in subversion. I deleted this tree from SVN and updated the patch.

+1 Otherwise. Applied this to a fresh checkout, ran the new target, and all was good.

> Have a way to automatically update Eclipse .classpath file when new libs are added to the classpath through Ivy for 0.20-* based sources
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7248
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7248
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.3
>            Reporter: Konstantin Boudnik
>            Assignee: Thomas Graves
>            Priority: Minor
>             Fix For: 0.20.205.0
>
>         Attachments: C7248-1.patch, HADOOP-7248-branch-20-security.patch
>
>
> Backport HADOOP-6407 into 0.20 based source trees

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14622-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 22:04:29 2011
Return-Path: <common-issues-return-14622-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 92A2356F8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 22:04:29 +0000 (UTC)
Received: (qmail 37646 invoked by uid 500); 10 May 2011 22:04:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37598 invoked by uid 500); 10 May 2011 22:04:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37590 invoked by uid 99); 10 May 2011 22:04:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 22:04:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 22:04:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7100A44DAB
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 22:03:47 +0000 (UTC)
Date: Tue, 10 May 2011 22:03:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <728066274.1366.1305065027459.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <285327525.9499.1304014323342.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7248) Have a way to automatically update
 Eclipse .classpath file when new libs are added to the classpath through
 Ivy for 0.20-* based sources
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031408#comment-13031408 ] 

Hadoop QA commented on HADOOP-7248:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478740/C7248-1.patch
  against trunk revision 1101653.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 19 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/424//console

This message is automatically generated.

> Have a way to automatically update Eclipse .classpath file when new libs are added to the classpath through Ivy for 0.20-* based sources
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7248
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7248
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.3
>            Reporter: Konstantin Boudnik
>            Assignee: Thomas Graves
>            Priority: Minor
>             Fix For: 0.20.205.0
>
>         Attachments: C7248-1.patch, HADOOP-7248-branch-20-security.patch
>
>
> Backport HADOOP-6407 into 0.20 based source trees

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14623-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 22:10:27 2011
Return-Path: <common-issues-return-14623-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 475B8580E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 22:10:27 +0000 (UTC)
Received: (qmail 42852 invoked by uid 500); 10 May 2011 22:10:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42822 invoked by uid 500); 10 May 2011 22:10:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42810 invoked by uid 99); 10 May 2011 22:10:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 22:10:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 22:10:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6A55144F8A
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 22:09:47 +0000 (UTC)
Date: Tue, 10 May 2011 22:09:47 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1105491925.1375.1305065387432.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7272) Remove unnecessary security related
 info logs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Remove unnecessary security related info logs
---------------------------------------------

                 Key: HADOOP-7272
                 URL: https://issues.apache.org/jira/browse/HADOOP-7272
             Project: Hadoop Common
          Issue Type: Improvement
          Components: ipc, security
    Affects Versions: 0.23.0
            Reporter: Suresh Srinivas
            Assignee: Suresh Srinivas
             Fix For: 0.23.0


Two info logs are printed when connection to RPC server is established, is not necessary. On a production cluster, these log lines made up of close to 50% of lines in the namenode log. I propose changing them into debug logs.




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14624-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 22:10:29 2011
Return-Path: <common-issues-return-14624-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B2C3A5840
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 22:10:29 +0000 (UTC)
Received: (qmail 44606 invoked by uid 500); 10 May 2011 22:10:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44572 invoked by uid 500); 10 May 2011 22:10:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44564 invoked by uid 99); 10 May 2011 22:10:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 22:10:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 22:10:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 91FDA44F8F
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 22:09:47 +0000 (UTC)
Date: Tue, 10 May 2011 22:09:47 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1854805184.1380.1305065387594.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1105491925.1375.1305065387432.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7272) Remove unnecessary security
 related info logs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031411#comment-13031411 ] 

Suresh Srinivas commented on HADOOP-7272:
-----------------------------------------

The logs are:
{noformat}
LOG.info("SASL server context established. Negotiated QoP is "
       + saslServer.getNegotiatedProperty(Sasl.QOP));
LOG.info("SASL server successfully authenticated client: " + user);
{noformat}

Of this the second log is also recorded in audit logs.

> Remove unnecessary security related info logs
> ---------------------------------------------
>
>                 Key: HADOOP-7272
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7272
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>    Affects Versions: 0.23.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.23.0
>
>
> Two info logs are printed when connection to RPC server is established, is not necessary. On a production cluster, these log lines made up of close to 50% of lines in the namenode log. I propose changing them into debug logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14625-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 22:12:27 2011
Return-Path: <common-issues-return-14625-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 16E995869
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 22:12:27 +0000 (UTC)
Received: (qmail 46519 invoked by uid 500); 10 May 2011 22:12:26 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46488 invoked by uid 500); 10 May 2011 22:12:26 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46480 invoked by uid 99); 10 May 2011 22:12:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 22:12:26 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 22:12:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 621BB44059
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 22:11:47 +0000 (UTC)
Date: Tue, 10 May 2011 22:11:47 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1388824635.1383.1305065507398.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1105491925.1375.1305065387432.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7272) Remove unnecessary security related
 info logs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-7272:
------------------------------------

    Attachment: HADOOP-7272.patch

> Remove unnecessary security related info logs
> ---------------------------------------------
>
>                 Key: HADOOP-7272
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7272
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>    Affects Versions: 0.23.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7272.patch
>
>
> Two info logs are printed when connection to RPC server is established, is not necessary. On a production cluster, these log lines made up of close to 50% of lines in the namenode log. I propose changing them into debug logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14626-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 22:12:29 2011
Return-Path: <common-issues-return-14626-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D0301587E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 22:12:29 +0000 (UTC)
Received: (qmail 47047 invoked by uid 500); 10 May 2011 22:12:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46987 invoked by uid 500); 10 May 2011 22:12:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46821 invoked by uid 99); 10 May 2011 22:12:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 22:12:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 22:12:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9BDB944061
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 22:11:47 +0000 (UTC)
Date: Tue, 10 May 2011 22:11:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1329338356.1390.1305065507635.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1105491925.1375.1305065387432.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7272) Remove unnecessary security
 related info logs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031413#comment-13031413 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7272:
------------------------------------------------

+1 patch looks good.

> Remove unnecessary security related info logs
> ---------------------------------------------
>
>                 Key: HADOOP-7272
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7272
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>    Affects Versions: 0.23.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7272.patch
>
>
> Two info logs are printed when connection to RPC server is established, is not necessary. On a production cluster, these log lines made up of close to 50% of lines in the namenode log. I propose changing them into debug logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14627-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 22:20:27 2011
Return-Path: <common-issues-return-14627-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BA54F5D4D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 22:20:27 +0000 (UTC)
Received: (qmail 48999 invoked by uid 500); 10 May 2011 22:20:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48962 invoked by uid 500); 10 May 2011 22:20:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48953 invoked by uid 99); 10 May 2011 22:20:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 22:20:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 22:20:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A420A441F3
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 22:19:47 +0000 (UTC)
Date: Tue, 10 May 2011 22:19:47 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <802179983.1407.1305065987668.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6920) Metrics2: metrics instrumentation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6920:
------------------------------------

    Attachment: hadoop-6920.v6.patch

Updates to the patch:
A minor bug fix in DefaultMetricsSystem.java and additional information in the hadoop-metrics2.properties to show how to configure the new metrics2 properties.

> Metrics2: metrics instrumentation
> ---------------------------------
>
>                 Key: HADOOP-6920
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6920
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-6920-metrics2-inst-v4.patch, hadoop-6920.v6.patch
>
>
> The jira tracks the metrics instrumentation for the new framework, i.e., porting jvm and rpc metrics to the new framework.
> This issue (esp. the "incompatible change flag") depends on the outcome of the discussion (proposed in HADOOP-6728) on whether we should support backward compatibility (at some cost.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14628-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 22:22:28 2011
Return-Path: <common-issues-return-14628-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 203325DA5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 22:22:28 +0000 (UTC)
Received: (qmail 50211 invoked by uid 500); 10 May 2011 22:22:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50172 invoked by uid 500); 10 May 2011 22:22:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50164 invoked by uid 99); 10 May 2011 22:22:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 22:22:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 22:22:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0E5494429D
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 22:21:48 +0000 (UTC)
Date: Tue, 10 May 2011 22:21:48 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <875844270.1414.1305066108055.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1105491925.1375.1305065387432.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7272) Remove unnecessary security
 related info logs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031416#comment-13031416 ] 

Suresh Srinivas commented on HADOOP-7272:
-----------------------------------------

Since this is a trivial log change, I compiled the change to make sure the code compiles. Not planning to run hudson job or add tests.

> Remove unnecessary security related info logs
> ---------------------------------------------
>
>                 Key: HADOOP-7272
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7272
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>    Affects Versions: 0.23.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7272.patch
>
>
> Two info logs are printed when connection to RPC server is established, is not necessary. On a production cluster, these log lines made up of close to 50% of lines in the namenode log. I propose changing them into debug logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14629-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 22:26:31 2011
Return-Path: <common-issues-return-14629-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 63BF45F72
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 22:26:31 +0000 (UTC)
Received: (qmail 55131 invoked by uid 500); 10 May 2011 22:26:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55083 invoked by uid 500); 10 May 2011 22:26:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54985 invoked by uid 99); 10 May 2011 22:26:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 22:26:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 22:26:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F12BF44429
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 22:25:47 +0000 (UTC)
Date: Tue, 10 May 2011 22:25:47 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <685824115.1437.1305066347984.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1105491925.1375.1305065387432.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7272) Remove unnecessary security related
 info logs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas resolved HADOOP-7272.
-------------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]

I committed the change.

> Remove unnecessary security related info logs
> ---------------------------------------------
>
>                 Key: HADOOP-7272
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7272
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>    Affects Versions: 0.23.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7272.patch
>
>
> Two info logs are printed when connection to RPC server is established, is not necessary. On a production cluster, these log lines made up of close to 50% of lines in the namenode log. I propose changing them into debug logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14630-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 22:28:27 2011
Return-Path: <common-issues-return-14630-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9AC0E5F9D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 22:28:27 +0000 (UTC)
Received: (qmail 56843 invoked by uid 500); 10 May 2011 22:28:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56811 invoked by uid 500); 10 May 2011 22:28:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56803 invoked by uid 99); 10 May 2011 22:28:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 22:28:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 22:28:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B2043444B6
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 22:27:47 +0000 (UTC)
Date: Tue, 10 May 2011 22:27:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <320706192.1448.1305066467725.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6920) Metrics2: metrics instrumentation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031421#comment-13031421 ] 

Hadoop QA commented on HADOOP-6920:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478742/hadoop-6920.v6.patch
  against trunk revision 1101668.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 12 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/426//console

This message is automatically generated.

> Metrics2: metrics instrumentation
> ---------------------------------
>
>                 Key: HADOOP-6920
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6920
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-6920-metrics2-inst-v4.patch, hadoop-6920.v6.patch
>
>
> The jira tracks the metrics instrumentation for the new framework, i.e., porting jvm and rpc metrics to the new framework.
> This issue (esp. the "incompatible change flag") depends on the outcome of the discussion (proposed in HADOOP-6728) on whether we should support backward compatibility (at some cost.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14631-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 22:32:28 2011
Return-Path: <common-issues-return-14631-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A74655BC7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 22:32:28 +0000 (UTC)
Received: (qmail 62658 invoked by uid 500); 10 May 2011 22:32:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62608 invoked by uid 500); 10 May 2011 22:32:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62600 invoked by uid 99); 10 May 2011 22:32:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 22:32:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 22:32:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C8CEA44716
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 22:31:48 +0000 (UTC)
Date: Tue, 10 May 2011 22:31:48 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <312123642.1492.1305066708818.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6920) Metrics2: metrics instrumentation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031434#comment-13031434 ] 

Hadoop QA commented on HADOOP-6920:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478742/hadoop-6920.v6.patch
  against trunk revision 1101653.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 12 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/425//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/425//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/425//console

This message is automatically generated.

> Metrics2: metrics instrumentation
> ---------------------------------
>
>                 Key: HADOOP-6920
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6920
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-6920-metrics2-inst-v4.patch, hadoop-6920.v6.patch
>
>
> The jira tracks the metrics instrumentation for the new framework, i.e., porting jvm and rpc metrics to the new framework.
> This issue (esp. the "incompatible change flag") depends on the outcome of the discussion (proposed in HADOOP-6728) on whether we should support backward compatibility (at some cost.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14632-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 22:44:50 2011
Return-Path: <common-issues-return-14632-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1DFDC58A9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 22:44:50 +0000 (UTC)
Received: (qmail 80548 invoked by uid 500); 10 May 2011 22:44:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80518 invoked by uid 500); 10 May 2011 22:44:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80510 invoked by uid 99); 10 May 2011 22:44:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 22:44:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 22:44:49 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0A17E44C03
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 22:43:49 +0000 (UTC)
Date: Tue, 10 May 2011 22:43:49 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <140517592.1567.1305067429038.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1105491925.1375.1305065387432.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7272) Remove unnecessary security
 related info logs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031441#comment-13031441 ] 

Hudson commented on HADOOP-7272:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #585 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/585/])
    HADOOP-7272. Remove unnecessary security related info logs. Contributed by Suresh Srinivas.


> Remove unnecessary security related info logs
> ---------------------------------------------
>
>                 Key: HADOOP-7272
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7272
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>    Affects Versions: 0.23.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7272.patch
>
>
> Two info logs are printed when connection to RPC server is established, is not necessary. On a production cluster, these log lines made up of close to 50% of lines in the namenode log. I propose changing them into debug logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14633-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 10 23:32:27 2011
Return-Path: <common-issues-return-14633-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 76E295AB8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 10 May 2011 23:32:27 +0000 (UTC)
Received: (qmail 26245 invoked by uid 500); 10 May 2011 23:32:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26213 invoked by uid 500); 10 May 2011 23:32:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26205 invoked by uid 99); 10 May 2011 23:32:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 23:32:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 23:32:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7B51E44A69
	for <common-issues@hadoop.apache.org>; Tue, 10 May 2011 23:31:47 +0000 (UTC)
Date: Tue, 10 May 2011 23:31:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <215806862.1714.1305070307501.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-6571) /stacks servlet no longer has
 correct content-type
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon resolved HADOOP-6571.
---------------------------------

    Resolution: Duplicate

Fixed by HADOOP-6496 (verified on trunk just now)

> /stacks servlet no longer has correct content-type
> --------------------------------------------------
>
>                 Key: HADOOP-6571
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6571
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>
> This makes /stacks format completely incorrectly. It should have text/plain content type, but instead has text/html.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14634-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 00:16:29 2011
Return-Path: <common-issues-return-14634-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6C3A75884
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 00:16:29 +0000 (UTC)
Received: (qmail 60632 invoked by uid 500); 11 May 2011 00:16:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60589 invoked by uid 500); 11 May 2011 00:16:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60581 invoked by uid 99); 11 May 2011 00:16:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 00:16:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 00:16:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5DA284473F
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 00:15:47 +0000 (UTC)
Date: Wed, 11 May 2011 00:15:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <882303051.1797.1305072947380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6920) Metrics2: metrics instrumentation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031485#comment-13031485 ] 

Hudson commented on HADOOP-6920:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #586 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/586/])
    HADOOP-6920. Metrics instrumentation to move new metrics2 framework. Contributed by Luke Lu.


> Metrics2: metrics instrumentation
> ---------------------------------
>
>                 Key: HADOOP-6920
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6920
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-6920-metrics2-inst-v4.patch, hadoop-6920.v6.patch
>
>
> The jira tracks the metrics instrumentation for the new framework, i.e., porting jvm and rpc metrics to the new framework.
> This issue (esp. the "incompatible change flag") depends on the outcome of the discussion (proposed in HADOOP-6728) on whether we should support backward compatibility (at some cost.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14635-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 00:18:27 2011
Return-Path: <common-issues-return-14635-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3D59251AA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 00:18:27 +0000 (UTC)
Received: (qmail 60957 invoked by uid 500); 11 May 2011 00:18:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60923 invoked by uid 500); 11 May 2011 00:18:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60915 invoked by uid 99); 11 May 2011 00:18:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 00:18:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 00:18:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5E85A447F8
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 00:17:47 +0000 (UTC)
Date: Wed, 11 May 2011 00:17:47 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1147685544.1808.1305073067383.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6920) Metrics2: metrics instrumentation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6920:
------------------------------------

      Resolution: Fixed
    Release Note: 
Metrics names are standardized to use CapitalizedCamelCase. Some examples of this is:
# Metrics names using "_" is changed to new naming scheme. Eg: bytes_written changes to BytesWritten.
# All metrics names start with capitals. Example: threadsBlocked changes to ThreadsBlocked.

    Hadoop Flags: [Incompatible change, Reviewed]
          Status: Resolved  (was: Patch Available)

> Metrics2: metrics instrumentation
> ---------------------------------
>
>                 Key: HADOOP-6920
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6920
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-6920-metrics2-inst-v4.patch, hadoop-6920.v6.patch
>
>
> The jira tracks the metrics instrumentation for the new framework, i.e., porting jvm and rpc metrics to the new framework.
> This issue (esp. the "incompatible change flag") depends on the outcome of the discussion (proposed in HADOOP-6728) on whether we should support backward compatibility (at some cost.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14636-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 00:55:28 2011
Return-Path: <common-issues-return-14636-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E1CBA574F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 00:55:27 +0000 (UTC)
Received: (qmail 87667 invoked by uid 500); 11 May 2011 00:55:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87629 invoked by uid 500); 11 May 2011 00:55:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87620 invoked by uid 99); 11 May 2011 00:55:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 00:55:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 00:55:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0C39E441B1
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 00:54:48 +0000 (UTC)
Date: Wed, 11 May 2011 00:54:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <329847962.1866.1305075288046.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1243264568.21599.1301502005765.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7214) Hadoop /usr/bin/groups equivalent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031493#comment-13031493 ] 

Todd Lipcon commented on HADOOP-7214:
-------------------------------------

Looking good. I think we need to update src/docs/src/documentation/content/xdocs/service_level_auth.xml and also conf/hadoop-policy.xml.template. But let's open another JIRA, since that file is out-of-date for other ACLs as well.

+1 for this patch.

> Hadoop /usr/bin/groups equivalent
> ---------------------------------
>
>                 Key: HADOOP-7214
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7214
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>         Attachments: hadoop-7214.0.txt, hadoop-7214.1.txt, hadoop-7214.10.patch, hadoop-7214.11.patch, hadoop-7214.2.txt, hadoop-7214.3.txt, hadoop-7214.4.txt, hadoop-7214.5.txt, hadoop-7214.6.txt, hadoop-7214.7.patch, hadoop-7214.8.patch, hadoop-7214.9.patch
>
>
> Since user -> groups resolution is done on the NN and JT machines, there should be a way for users to determine what groups they're a member of from the NN's and JT's perspective.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14637-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 01:05:27 2011
Return-Path: <common-issues-return-14637-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AD4875DA4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 01:05:27 +0000 (UTC)
Received: (qmail 99486 invoked by uid 500); 11 May 2011 01:05:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99458 invoked by uid 500); 11 May 2011 01:05:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99450 invoked by uid 99); 11 May 2011 01:05:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 01:05:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 01:05:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 94174443C9
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 01:04:47 +0000 (UTC)
Date: Wed, 11 May 2011 01:04:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1974979068.1902.1305075887603.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1243264568.21599.1301502005765.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7214) Hadoop /usr/bin/groups equivalent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7214:
--------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks, Aaron!

> Hadoop /usr/bin/groups equivalent
> ---------------------------------
>
>                 Key: HADOOP-7214
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7214
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7214.0.txt, hadoop-7214.1.txt, hadoop-7214.10.patch, hadoop-7214.11.patch, hadoop-7214.2.txt, hadoop-7214.3.txt, hadoop-7214.4.txt, hadoop-7214.5.txt, hadoop-7214.6.txt, hadoop-7214.7.patch, hadoop-7214.8.patch, hadoop-7214.9.patch
>
>
> Since user -> groups resolution is done on the NN and JT machines, there should be a way for users to determine what groups they're a member of from the NN's and JT's perspective.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14638-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 01:25:27 2011
Return-Path: <common-issues-return-14638-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5B90D5562
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 01:25:27 +0000 (UTC)
Received: (qmail 17466 invoked by uid 500); 11 May 2011 01:25:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17435 invoked by uid 500); 11 May 2011 01:25:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17427 invoked by uid 99); 11 May 2011 01:25:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 01:25:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 01:25:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5E8ED448E1
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 01:24:47 +0000 (UTC)
Date: Wed, 11 May 2011 01:24:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <669412213.1943.1305077087383.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1243264568.21599.1301502005765.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7214) Hadoop /usr/bin/groups equivalent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031503#comment-13031503 ] 

Hudson commented on HADOOP-7214:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #587 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/587/])
    HADOOP-7214. Add Common functionality necessary to provide an equivalent of /usr/bin/groups for Hadoop. Contributed by Aaron T. Myers.


> Hadoop /usr/bin/groups equivalent
> ---------------------------------
>
>                 Key: HADOOP-7214
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7214
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7214.0.txt, hadoop-7214.1.txt, hadoop-7214.10.patch, hadoop-7214.11.patch, hadoop-7214.2.txt, hadoop-7214.3.txt, hadoop-7214.4.txt, hadoop-7214.5.txt, hadoop-7214.6.txt, hadoop-7214.7.patch, hadoop-7214.8.patch, hadoop-7214.9.patch
>
>
> Since user -> groups resolution is done on the NN and JT machines, there should be a way for users to determine what groups they're a member of from the NN's and JT's perspective.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14639-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 01:55:27 2011
Return-Path: <common-issues-return-14639-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9C00D51FD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 01:55:27 +0000 (UTC)
Received: (qmail 34169 invoked by uid 500); 11 May 2011 01:55:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34136 invoked by uid 500); 11 May 2011 01:55:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34128 invoked by uid 99); 11 May 2011 01:55:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 01:55:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 01:55:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 54324441C7
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 01:54:47 +0000 (UTC)
Date: Wed, 11 May 2011 01:54:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1682376274.1991.1305078887341.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7273) Move JMXGet from hdfs to common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Move JMXGet from hdfs to common
-------------------------------

                 Key: HADOOP-7273
                 URL: https://issues.apache.org/jira/browse/HADOOP-7273
             Project: Hadoop Common
          Issue Type: Improvement
    Affects Versions: 0.21.0
            Reporter: Luke Lu
            Assignee: Luke Lu
             Fix For: 0.23.0


JMXGet is generally useful especially for testing JMX attributes. Let's move it from o.a.h.hdfs.tools to o.a.h.tools.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14640-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 02:44:30 2011
Return-Path: <common-issues-return-14640-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E0DAD4DD2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 02:44:29 +0000 (UTC)
Received: (qmail 63592 invoked by uid 500); 11 May 2011 02:44:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63414 invoked by uid 500); 11 May 2011 02:44:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63406 invoked by uid 99); 11 May 2011 02:44:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 02:44:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 02:44:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6655D44F4C
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 02:43:47 +0000 (UTC)
Date: Wed, 11 May 2011 02:43:47 +0000 (UTC)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <644851451.2025.1305081827415.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <285327525.9499.1304014323342.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7248) Have a way to automatically update
 Eclipse .classpath file when new libs are added to the classpath through
 Ivy for 0.20-* based sources
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-7248:
----------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I took a closer look at the patch, and came across this section:

{noformat}
+  <condition property="ant-eclipse.jar.exists">
+    <available file="${build.dir}/lib/ant-eclipse-1.0-jvm1.2.jar"/>
+  </condition>
+
+  <target name="ant-eclipse-download" unless="ant-eclipse.jar.exists"
+          description="Downloads the ant-eclipse binary.">
+    <get src="http://downloads.sourceforge.net/project/ant-eclipse/ant-eclipse/1.0/ant-eclipse-1.0.bin.tar.bz2"
+         dest="${build.dir}/ant-eclipse-1.0.bin.tar.bz2" usetimestamp="false" />
+
+    <untar src="${build.dir}/ant-eclipse-1.0.bin.tar.bz2"
+           dest="${build.dir}" compression="bzip2">
+      <patternset>
+        <include name="lib/ant-eclipse-1.0-jvm1.2.jar"/>
+      </patternset>
+    </untar>
+    <delete file="${build.dir}/ant-eclipse-1.0.bin.tar.bz2" />
+  </target>
{noformat}

Installing this jar into the maven cache would not only avoid downloading it so often, but it would also make the eclipse target valid when building offline. However, it looks like HADOOP-6407 (and related issues) made the same choice, so this can be a separate issue.

I committed this to the 20-security branch.

> Have a way to automatically update Eclipse .classpath file when new libs are added to the classpath through Ivy for 0.20-* based sources
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7248
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7248
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.3
>            Reporter: Konstantin Boudnik
>            Assignee: Thomas Graves
>            Priority: Minor
>             Fix For: 0.20.205.0
>
>         Attachments: C7248-1.patch, HADOOP-7248-branch-20-security.patch
>
>
> Backport HADOOP-6407 into 0.20 based source trees

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14641-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 03:47:31 2011
Return-Path: <common-issues-return-14641-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8DF7E4F6B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 03:47:31 +0000 (UTC)
Received: (qmail 2238 invoked by uid 500); 11 May 2011 03:47:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2031 invoked by uid 500); 11 May 2011 03:47:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2020 invoked by uid 99); 11 May 2011 03:47:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 03:47:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 03:47:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5968F44CCC
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 03:46:47 +0000 (UTC)
Date: Wed, 11 May 2011 03:46:47 +0000 (UTC)
From: "Ted Yu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <970256549.2061.1305085607363.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6171) "package" task in build.xml should
 copy source with preservelastmodified
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ted Yu updated HADOOP-6171:
---------------------------

    Attachment: HADOOP-6171.patch

> "package" task in build.xml should copy source with preservelastmodified
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6171
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6171
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Todd Lipcon
>              Labels: newbie
>         Attachments: HADOOP-6171.patch
>
>
> In the package task, it copies the source to dist.dir without using preservelastmodified=true. This can cause issues with the autotools configure process for the C++ builds. Namely, it will sometimes think the configure script is out of date with respect to its source files and try to re-bootstrap, which relies on particular versions of autotools on the building computer. This isn't something that should be required for those wanting to build from a distribution tarball.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14642-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 03:53:32 2011
Return-Path: <common-issues-return-14642-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 72DDD447F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 03:53:32 +0000 (UTC)
Received: (qmail 3294 invoked by uid 500); 11 May 2011 03:53:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3175 invoked by uid 500); 11 May 2011 03:53:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3161 invoked by uid 99); 11 May 2011 03:53:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 03:53:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 03:53:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5D2BE44E28
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 03:52:47 +0000 (UTC)
Date: Wed, 11 May 2011 03:52:47 +0000 (UTC)
From: "Ted Yu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1765811320.2063.1305085967377.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6171) "package" task in build.xml should
 copy source with preservelastmodified
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ted Yu updated HADOOP-6171:
---------------------------

    Attachment: HDFS-6171.patch

Here is patch for hdfs subproject.

> "package" task in build.xml should copy source with preservelastmodified
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6171
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6171
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Todd Lipcon
>              Labels: newbie
>         Attachments: HADOOP-6171.patch, HDFS-6171.patch
>
>
> In the package task, it copies the source to dist.dir without using preservelastmodified=true. This can cause issues with the autotools configure process for the C++ builds. Namely, it will sometimes think the configure script is out of date with respect to its source files and try to re-bootstrap, which relies on particular versions of autotools on the building computer. This isn't something that should be required for those wanting to build from a distribution tarball.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14643-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 03:55:30 2011
Return-Path: <common-issues-return-14643-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 65F3744AF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 03:55:30 +0000 (UTC)
Received: (qmail 3749 invoked by uid 500); 11 May 2011 03:55:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3720 invoked by uid 500); 11 May 2011 03:55:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3709 invoked by uid 99); 11 May 2011 03:55:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 03:55:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 03:55:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8A99A44F19
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 03:54:47 +0000 (UTC)
Date: Wed, 11 May 2011 03:54:47 +0000 (UTC)
From: "Ted Yu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <412279545.2070.1305086087564.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6171) "package" task in build.xml should
 copy source with preservelastmodified
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ted Yu updated HADOOP-6171:
---------------------------

    Attachment: HDFS-6171.patch

> "package" task in build.xml should copy source with preservelastmodified
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6171
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6171
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Todd Lipcon
>              Labels: newbie
>         Attachments: HADOOP-6171.patch, HDFS-6171.patch
>
>
> In the package task, it copies the source to dist.dir without using preservelastmodified=true. This can cause issues with the autotools configure process for the C++ builds. Namely, it will sometimes think the configure script is out of date with respect to its source files and try to re-bootstrap, which relies on particular versions of autotools on the building computer. This isn't something that should be required for those wanting to build from a distribution tarball.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14644-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 03:55:32 2011
Return-Path: <common-issues-return-14644-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4F59B44BE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 03:55:32 +0000 (UTC)
Received: (qmail 4005 invoked by uid 500); 11 May 2011 03:55:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3965 invoked by uid 500); 11 May 2011 03:55:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3902 invoked by uid 99); 11 May 2011 03:55:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 03:55:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 03:55:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 83FCC44F18
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 03:54:47 +0000 (UTC)
Date: Wed, 11 May 2011 03:54:47 +0000 (UTC)
From: "Ted Yu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1814914619.2069.1305086087537.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6171) "package" task in build.xml should
 copy source with preservelastmodified
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ted Yu updated HADOOP-6171:
---------------------------

    Attachment:     (was: HDFS-6171.patch)

> "package" task in build.xml should copy source with preservelastmodified
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6171
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6171
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Todd Lipcon
>              Labels: newbie
>         Attachments: HADOOP-6171.patch, HDFS-6171.patch
>
>
> In the package task, it copies the source to dist.dir without using preservelastmodified=true. This can cause issues with the autotools configure process for the C++ builds. Namely, it will sometimes think the configure script is out of date with respect to its source files and try to re-bootstrap, which relies on particular versions of autotools on the building computer. This isn't something that should be required for those wanting to build from a distribution tarball.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14645-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 03:57:28 2011
Return-Path: <common-issues-return-14645-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A0E3544D9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 03:57:28 +0000 (UTC)
Received: (qmail 4945 invoked by uid 500); 11 May 2011 03:57:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4916 invoked by uid 500); 11 May 2011 03:57:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4904 invoked by uid 99); 11 May 2011 03:57:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 03:57:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 03:57:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5880E44FDD
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 03:56:47 +0000 (UTC)
Date: Wed, 11 May 2011 03:56:47 +0000 (UTC)
From: "Ted Yu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1183097533.2073.1305086207359.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6171) "package" task in build.xml should
 copy source with preservelastmodified
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ted Yu updated HADOOP-6171:
---------------------------

    Attachment: MAPRED-6171.patch

> "package" task in build.xml should copy source with preservelastmodified
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6171
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6171
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Todd Lipcon
>              Labels: newbie
>         Attachments: HADOOP-6171.patch, HDFS-6171.patch, MAPRED-6171.patch
>
>
> In the package task, it copies the source to dist.dir without using preservelastmodified=true. This can cause issues with the autotools configure process for the C++ builds. Namely, it will sometimes think the configure script is out of date with respect to its source files and try to re-bootstrap, which relies on particular versions of autotools on the building computer. This isn't something that should be required for those wanting to build from a distribution tarball.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14646-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 04:00:43 2011
Return-Path: <common-issues-return-14646-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F2BD748D2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 04:00:42 +0000 (UTC)
Received: (qmail 6794 invoked by uid 500); 11 May 2011 04:00:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6770 invoked by uid 500); 11 May 2011 04:00:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6762 invoked by uid 99); 11 May 2011 04:00:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 04:00:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 04:00:30 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6B8A4440EE
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 03:59:50 +0000 (UTC)
Date: Wed, 11 May 2011 03:59:50 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1496496939.2081.1305086390434.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <21922935.186601292622961312.JavaMail.jira@thor>
Subject: [jira] [Updated] (HADOOP-7068) Ivy resolve force mode should be
 turned off by default
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-7068:
------------------------------

       Resolution: Fixed
    Fix Version/s: 0.22.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I've just committed this. Thanks, Luke!

> Ivy resolve force mode should be turned off by default
> ------------------------------------------------------
>
>                 Key: HADOOP-7068
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7068
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7068-trunk-v1.patch, hadoop-7068-trunk-v2.patch
>
>
> The problem is introduced by  HADOOP-6486. Which have caused a lot of mysterious artifact issues (unable to downgrade or do parallel dev, without wiping out both m2 and ivy caches etc.) wasting countless hours of dev (many people's) time to track down the issue.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14647-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 04:16:32 2011
Return-Path: <common-issues-return-14647-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3560740A4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 04:16:32 +0000 (UTC)
Received: (qmail 13192 invoked by uid 500); 11 May 2011 04:16:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12992 invoked by uid 500); 11 May 2011 04:16:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12978 invoked by uid 99); 11 May 2011 04:16:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 04:16:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 04:16:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 58E4344516
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 04:15:47 +0000 (UTC)
Date: Wed, 11 May 2011 04:15:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1741005890.2089.1305087347360.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <21922935.186601292622961312.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7068) Ivy resolve force mode should be
 turned off by default
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031530#comment-13031530 ] 

Hudson commented on HADOOP-7068:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #588 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/588/])
    HADOOP-7068. Ivy resolve force mode should be turned off by default. Contributed by Luke Lu.


> Ivy resolve force mode should be turned off by default
> ------------------------------------------------------
>
>                 Key: HADOOP-7068
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7068
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7068-trunk-v1.patch, hadoop-7068-trunk-v2.patch
>
>
> The problem is introduced by  HADOOP-6486. Which have caused a lot of mysterious artifact issues (unable to downgrade or do parallel dev, without wiping out both m2 and ivy caches etc.) wasting countless hours of dev (many people's) time to track down the issue.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14649-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 05:53:32 2011
Return-Path: <common-issues-return-14649-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C97454E18
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 05:53:32 +0000 (UTC)
Received: (qmail 80536 invoked by uid 500); 11 May 2011 05:53:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80288 invoked by uid 500); 11 May 2011 05:53:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80265 invoked by uid 99); 11 May 2011 05:53:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 05:53:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 05:53:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5F9DA45B94
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 05:52:47 +0000 (UTC)
Date: Wed, 11 May 2011 05:52:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1112698860.2202.1305093167388.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6171) "package" task in build.xml should
 copy source with preservelastmodified
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031551#comment-13031551 ] 

Todd Lipcon commented on HADOOP-6171:
-------------------------------------

Hey Ted. Thanks for providing the patches! Unfortunately with the way the project is currently structured, we need to have a separate JIRA for each project (common, hdfs, and mr). Could you open new JIRAs on those projects and "Link" them to this one? You can then upload the individual patches there.

> "package" task in build.xml should copy source with preservelastmodified
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6171
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6171
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Todd Lipcon
>              Labels: newbie
>         Attachments: HADOOP-6171.patch, HDFS-6171.patch, MAPRED-6171.patch
>
>
> In the package task, it copies the source to dist.dir without using preservelastmodified=true. This can cause issues with the autotools configure process for the C++ builds. Namely, it will sometimes think the configure script is out of date with respect to its source files and try to re-bootstrap, which relies on particular versions of autotools on the building computer. This isn't something that should be required for those wanting to build from a distribution tarball.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14648-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 05:53:33 2011
Return-Path: <common-issues-return-14648-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C94974E17
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 05:53:32 +0000 (UTC)
Received: (qmail 80400 invoked by uid 500); 11 May 2011 05:53:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80283 invoked by uid 500); 11 May 2011 05:53:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80266 invoked by uid 99); 11 May 2011 05:53:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 05:53:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 05:53:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6B68A45B96
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 05:52:47 +0000 (UTC)
Date: Wed, 11 May 2011 05:52:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1962419078.2204.1305093167436.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-6171) "package" task in build.xml should
 copy source with preservelastmodified
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon reassigned HADOOP-6171:
-----------------------------------

    Assignee: Ted Yu

> "package" task in build.xml should copy source with preservelastmodified
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6171
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6171
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Todd Lipcon
>            Assignee: Ted Yu
>              Labels: newbie
>         Attachments: HADOOP-6171.patch, HDFS-6171.patch, MAPRED-6171.patch
>
>
> In the package task, it copies the source to dist.dir without using preservelastmodified=true. This can cause issues with the autotools configure process for the C++ builds. Namely, it will sometimes think the configure script is out of date with respect to its source files and try to re-bootstrap, which relies on particular versions of autotools on the building computer. This isn't something that should be required for those wanting to build from a distribution tarball.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14650-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 09:43:28 2011
Return-Path: <common-issues-return-14650-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3B9904EDA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 09:43:28 +0000 (UTC)
Received: (qmail 68764 invoked by uid 500); 11 May 2011 09:43:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68698 invoked by uid 500); 11 May 2011 09:43:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68682 invoked by uid 99); 11 May 2011 09:43:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 09:43:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 09:43:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 51A7A451C0
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 09:42:47 +0000 (UTC)
Date: Wed, 11 May 2011 09:42:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <48925858.2560.1305106967331.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <21922935.186601292622961312.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7068) Ivy resolve force mode should be
 turned off by default
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031617#comment-13031617 ] 

Hudson commented on HADOOP-7068:
--------------------------------

Integrated in Hadoop-Common-22-branch #45 (See [https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/45/])
    Merge -r 1101734:1101735 from trunk to branch-0.22. Fixes: HADOOP-7068


> Ivy resolve force mode should be turned off by default
> ------------------------------------------------------
>
>                 Key: HADOOP-7068
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7068
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7068-trunk-v1.patch, hadoop-7068-trunk-v2.patch
>
>
> The problem is introduced by  HADOOP-6486. Which have caused a lot of mysterious artifact issues (unable to downgrade or do parallel dev, without wiping out both m2 and ivy caches etc.) wasting countless hours of dev (many people's) time to track down the issue.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14652-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 11:15:29 2011
Return-Path: <common-issues-return-14652-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E6F164831
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 11:15:28 +0000 (UTC)
Received: (qmail 95362 invoked by uid 500); 11 May 2011 11:15:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95292 invoked by uid 500); 11 May 2011 11:15:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95275 invoked by uid 99); 11 May 2011 11:15:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 11:15:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 11:15:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 671CA4528A
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 11:14:48 +0000 (UTC)
Date: Wed, 11 May 2011 11:14:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1031285691.2702.1305112488418.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1105491925.1375.1305065387432.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7272) Remove unnecessary security
 related info logs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031641#comment-13031641 ] 

Hudson commented on HADOOP-7272:
--------------------------------

Integrated in Hadoop-Common-trunk #685 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/685/])
    HADOOP-7272. Remove unnecessary security related info logs. Contributed by Suresh Srinivas.


> Remove unnecessary security related info logs
> ---------------------------------------------
>
>                 Key: HADOOP-7272
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7272
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>    Affects Versions: 0.23.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7272.patch
>
>
> Two info logs are printed when connection to RPC server is established, is not necessary. On a production cluster, these log lines made up of close to 50% of lines in the namenode log. I propose changing them into debug logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14653-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 11:15:29 2011
Return-Path: <common-issues-return-14653-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2203D4849
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 11:15:29 +0000 (UTC)
Received: (qmail 95708 invoked by uid 500); 11 May 2011 11:15:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95660 invoked by uid 500); 11 May 2011 11:15:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95609 invoked by uid 99); 11 May 2011 11:15:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 11:15:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 11:15:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0AA004529F
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 11:14:49 +0000 (UTC)
Date: Wed, 11 May 2011 11:14:49 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1825950652.2722.1305112489039.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6920) Metrics2: metrics instrumentation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031644#comment-13031644 ] 

Hudson commented on HADOOP-6920:
--------------------------------

Integrated in Hadoop-Common-trunk #685 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/685/])
    HADOOP-6920. Metrics instrumentation to move new metrics2 framework. Contributed by Luke Lu.


> Metrics2: metrics instrumentation
> ---------------------------------
>
>                 Key: HADOOP-6920
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6920
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-6920-metrics2-inst-v4.patch, hadoop-6920.v6.patch
>
>
> The jira tracks the metrics instrumentation for the new framework, i.e., porting jvm and rpc metrics to the new framework.
> This issue (esp. the "incompatible change flag") depends on the outcome of the discussion (proposed in HADOOP-6728) on whether we should support backward compatibility (at some cost.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14651-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 11:15:29 2011
Return-Path: <common-issues-return-14651-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E6B84482F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 11:15:28 +0000 (UTC)
Received: (qmail 95356 invoked by uid 500); 11 May 2011 11:15:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95289 invoked by uid 500); 11 May 2011 11:15:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95274 invoked by uid 99); 11 May 2011 11:15:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 11:15:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 11:15:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7F87D4528D
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 11:14:48 +0000 (UTC)
Date: Wed, 11 May 2011 11:14:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1767770911.2705.1305112488518.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1912220915.31712.1304917443296.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7268)
 FileContext.getLocalFSFileContext() behavior needs to be fixed w.r.t tokens
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031642#comment-13031642 ] 

Hudson commented on HADOOP-7268:
--------------------------------

Integrated in Hadoop-Common-trunk #685 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/685/])
    HADOOP-7268. FileContext.getLocalFSFileContext() behavior needs to be fixed w.r.t tokens. Contributed by jitendra.


> FileContext.getLocalFSFileContext() behavior needs to be fixed w.r.t tokens
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-7268
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7268
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, security
>    Affects Versions: 0.23.0
>            Reporter: Devaraj Das
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7268.1.patch
>
>
> FileContext.getLocalFSFileContext() instantiates a FileContext object upon the first call to it, and for all subsequent calls returns back that instance (a static localFsSingleton object). With security turned on, this causes some hard-to-debug situations when that fileContext is used for doing HDFS operations. This is because the UserGroupInformation is stored when a FileContext is instantiated. If the process in question wishes to use different UserGroupInformation objects for different file system operations (where the corresponding FileContext objects are obtained via calls to FileContext.getLocalFSFileContext()), it doesn't work.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14654-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 11:15:31 2011
Return-Path: <common-issues-return-14654-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1C5F44869
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 11:15:31 +0000 (UTC)
Received: (qmail 96073 invoked by uid 500); 11 May 2011 11:15:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96041 invoked by uid 500); 11 May 2011 11:15:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96033 invoked by uid 99); 11 May 2011 11:15:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 11:15:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 11:15:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5992E45288
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 11:14:48 +0000 (UTC)
Date: Wed, 11 May 2011 11:14:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <195760776.2700.1305112488363.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2068463279.801.1305050747377.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7271) Standardize error messages
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031640#comment-13031640 ] 

Hudson commented on HADOOP-7271:
--------------------------------

Integrated in Hadoop-Common-trunk #685 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/685/])
    HADOOP-7271. Standardize shell command error messages.  Contributed by Daryn Sharp


> Standardize error messages
> --------------------------
>
>                 Key: HADOOP-7271
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7271
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7271-2.patch, HADOOP-7271.patch
>
>
> The FsShell commands have no standard format for the same error message.  For instance, here is a snippet of the variations of just one of many error messages:
> cmd: $path: No such file or directory
> cmd: cannot stat `$path': No such file or directory
> cmd: Can not find listing for $path
> cmd: Cannot access $path: No such file or directory.
> cmd: No such file or directory `$path'
> cmd: File does not exist: $path
> cmd: File $path does not exist
> ... etc ...
> These need to be common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14655-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 11:15:31 2011
Return-Path: <common-issues-return-14655-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 410504875
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 11:15:31 +0000 (UTC)
Received: (qmail 96241 invoked by uid 500); 11 May 2011 11:15:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96190 invoked by uid 500); 11 May 2011 11:15:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96047 invoked by uid 99); 11 May 2011 11:15:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 11:15:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 11:15:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AD64C45293
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 11:14:48 +0000 (UTC)
Date: Wed, 11 May 2011 11:14:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1534390063.2711.1305112488706.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <21922935.186601292622961312.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7068) Ivy resolve force mode should be
 turned off by default
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031643#comment-13031643 ] 

Hudson commented on HADOOP-7068:
--------------------------------

Integrated in Hadoop-Common-trunk #685 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/685/])
    HADOOP-7068. Ivy resolve force mode should be turned off by default. Contributed by Luke Lu.


> Ivy resolve force mode should be turned off by default
> ------------------------------------------------------
>
>                 Key: HADOOP-7068
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7068
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7068-trunk-v1.patch, hadoop-7068-trunk-v2.patch
>
>
> The problem is introduced by  HADOOP-6486. Which have caused a lot of mysterious artifact issues (unable to downgrade or do parallel dev, without wiping out both m2 and ivy caches etc.) wasting countless hours of dev (many people's) time to track down the issue.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14656-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 11:15:49 2011
Return-Path: <common-issues-return-14656-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 62659491E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 11:15:49 +0000 (UTC)
Received: (qmail 96698 invoked by uid 500); 11 May 2011 11:15:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96668 invoked by uid 500); 11 May 2011 11:15:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96660 invoked by uid 99); 11 May 2011 11:15:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 11:15:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 11:15:48 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1B0A8452A2
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 11:14:49 +0000 (UTC)
Date: Wed, 11 May 2011 11:14:49 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <48645378.2724.1305112489107.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1243264568.21599.1301502005765.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7214) Hadoop /usr/bin/groups equivalent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031645#comment-13031645 ] 

Hudson commented on HADOOP-7214:
--------------------------------

Integrated in Hadoop-Common-trunk #685 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/685/])
    HADOOP-7214. Add Common functionality necessary to provide an equivalent of /usr/bin/groups for Hadoop. Contributed by Aaron T. Myers.


> Hadoop /usr/bin/groups equivalent
> ---------------------------------
>
>                 Key: HADOOP-7214
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7214
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7214.0.txt, hadoop-7214.1.txt, hadoop-7214.10.patch, hadoop-7214.11.patch, hadoop-7214.2.txt, hadoop-7214.3.txt, hadoop-7214.4.txt, hadoop-7214.5.txt, hadoop-7214.6.txt, hadoop-7214.7.patch, hadoop-7214.8.patch, hadoop-7214.9.patch
>
>
> Since user -> groups resolution is done on the NN and JT machines, there should be a way for users to determine what groups they're a member of from the NN's and JT's perspective.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14657-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 14:12:27 2011
Return-Path: <common-issues-return-14657-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 657F146AC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 14:12:27 +0000 (UTC)
Received: (qmail 11163 invoked by uid 500); 11 May 2011 14:12:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11131 invoked by uid 500); 11 May 2011 14:12:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11122 invoked by uid 99); 11 May 2011 14:12:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 14:12:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 14:12:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 830F245912
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 14:11:47 +0000 (UTC)
Date: Wed, 11 May 2011 14:11:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <713216923.3080.1305123107533.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1110985885.74832.1303423445828.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7237) Refactor FsShell's touchz
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7237:
--------------------------------

    Attachment: HADOOP-7237.patch

> Refactor FsShell's touchz
> -------------------------
>
>                 Key: HADOOP-7237
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7237
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7237.patch
>
>
> Need to refactor touchz to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14658-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 14:12:30 2011
Return-Path: <common-issues-return-14658-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5EC5846CF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 14:12:30 +0000 (UTC)
Received: (qmail 11653 invoked by uid 500); 11 May 2011 14:12:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11583 invoked by uid 500); 11 May 2011 14:12:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11491 invoked by uid 99); 11 May 2011 14:12:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 14:12:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 14:12:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 946AD45917
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 14:11:47 +0000 (UTC)
Date: Wed, 11 May 2011 14:11:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1111741355.3082.1305123107604.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1110985885.74832.1303423445828.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7237) Refactor FsShell's touchz
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7237:
--------------------------------

    Status: Patch Available  (was: Open)

> Refactor FsShell's touchz
> -------------------------
>
>                 Key: HADOOP-7237
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7237
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7237.patch
>
>
> Need to refactor touchz to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14659-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 14:24:29 2011
Return-Path: <common-issues-return-14659-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 91B294308
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 14:24:29 +0000 (UTC)
Received: (qmail 45810 invoked by uid 500); 11 May 2011 14:24:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45777 invoked by uid 500); 11 May 2011 14:24:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45769 invoked by uid 99); 11 May 2011 14:24:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 14:24:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 14:24:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5BDE945FDA
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 14:23:47 +0000 (UTC)
Date: Wed, 11 May 2011 14:23:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2115651099.3122.1305123827373.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1110985885.74832.1303423445828.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7237) Refactor FsShell's touchz
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031723#comment-13031723 ] 

Hadoop QA commented on HADOOP-7237:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478822/HADOOP-7237.patch
  against trunk revision 1101735.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 4 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/427//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/427//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/427//console

This message is automatically generated.

> Refactor FsShell's touchz
> -------------------------
>
>                 Key: HADOOP-7237
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7237
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7237.patch
>
>
> Need to refactor touchz to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14660-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 14:58:27 2011
Return-Path: <common-issues-return-14660-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 515BA4809
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 14:58:27 +0000 (UTC)
Received: (qmail 15852 invoked by uid 500); 11 May 2011 14:58:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15817 invoked by uid 500); 11 May 2011 14:58:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15804 invoked by uid 99); 11 May 2011 14:58:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 14:58:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 14:58:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7B8A3451DF
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 14:57:47 +0000 (UTC)
Date: Wed, 11 May 2011 14:57:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1881213413.3225.1305125867502.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <873708488.29176.1304715003217.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7267) Refactor FsShell's rm/rmr/expunge
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7267:
--------------------------------

    Attachment: HADOOP-7267.patch

> Refactor FsShell's rm/rmr/expunge
> ---------------------------------
>
>                 Key: HADOOP-7267
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7267
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7267.patch
>
>
> Refactor to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14661-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 14:58:27 2011
Return-Path: <common-issues-return-14661-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 81FCC4814
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 14:58:27 +0000 (UTC)
Received: (qmail 16082 invoked by uid 500); 11 May 2011 14:58:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16046 invoked by uid 500); 11 May 2011 14:58:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15877 invoked by uid 99); 11 May 2011 14:58:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 14:58:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 14:58:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8B870451E1
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 14:57:47 +0000 (UTC)
Date: Wed, 11 May 2011 14:57:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <797310640.3227.1305125867568.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <873708488.29176.1304715003217.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7267) Refactor FsShell's rm/rmr/expunge
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7267:
--------------------------------

    Status: Patch Available  (was: Open)

> Refactor FsShell's rm/rmr/expunge
> ---------------------------------
>
>                 Key: HADOOP-7267
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7267
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7267.patch
>
>
> Refactor to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14662-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 15:08:27 2011
Return-Path: <common-issues-return-14662-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4DF0D413E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 15:08:27 +0000 (UTC)
Received: (qmail 33443 invoked by uid 500); 11 May 2011 15:08:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33414 invoked by uid 500); 11 May 2011 15:08:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33406 invoked by uid 99); 11 May 2011 15:08:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:08:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:08:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7A27C456E2
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 15:07:47 +0000 (UTC)
Date: Wed, 11 May 2011 15:07:47 +0000 (UTC)
From: "Jonathan Eagles (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <88464990.3250.1305126467497.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7274) CLONE - IOUtils.readFully and
 IOUtils.skipFully have typo in exception creation's message
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

CLONE - IOUtils.readFully and IOUtils.skipFully have typo in exception creation's message
-----------------------------------------------------------------------------------------

                 Key: HADOOP-7274
                 URL: https://issues.apache.org/jira/browse/HADOOP-7274
             Project: Hadoop Common
          Issue Type: Bug
          Components: util
    Affects Versions: 0.22.0, 0.23.0
            Reporter: Jonathan Eagles
            Assignee: Konstantin Boudnik
            Priority: Minor
             Fix For: 0.22.0, 0.23.0


{noformat}
        throw new IOException( "Premeture EOF from inputStream");
{noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14663-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 15:08:27 2011
Return-Path: <common-issues-return-14663-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C2F01414C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 15:08:27 +0000 (UTC)
Received: (qmail 33742 invoked by uid 500); 11 May 2011 15:08:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33657 invoked by uid 500); 11 May 2011 15:08:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33641 invoked by uid 99); 11 May 2011 15:08:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:08:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:08:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BB070456E7
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 15:07:47 +0000 (UTC)
Date: Wed, 11 May 2011 15:07:47 +0000 (UTC)
From: "Jonathan Eagles (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <989084922.3255.1305126467762.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <88464990.3250.1305126467497.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7274) CLONE - IOUtils.readFully and
 IOUtils.skipFully have typo in exception creation's message
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Eagles reassigned HADOOP-7274:
---------------------------------------

    Assignee: Jonathan Eagles  (was: Konstantin Boudnik)

> CLONE - IOUtils.readFully and IOUtils.skipFully have typo in exception creation's message
> -----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7274
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7274
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Jonathan Eagles
>            Assignee: Jonathan Eagles
>            Priority: Minor
>             Fix For: 0.22.0, 0.23.0
>
>
> {noformat}
>         throw new IOException( "Premeture EOF from inputStream");
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14664-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 15:12:29 2011
Return-Path: <common-issues-return-14664-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8FFC842A1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 15:12:29 +0000 (UTC)
Received: (qmail 41123 invoked by uid 500); 11 May 2011 15:12:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41089 invoked by uid 500); 11 May 2011 15:12:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41072 invoked by uid 99); 11 May 2011 15:12:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:12:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:12:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5A4FF45886
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 15:11:47 +0000 (UTC)
Date: Wed, 11 May 2011 15:11:47 +0000 (UTC)
From: "Jonathan Eagles (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1919985336.3267.1305126707366.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <88464990.3250.1305126467497.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7274) CLONE - IOUtils.readFully and
 IOUtils.skipFully have typo in exception creation's message
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Eagles updated HADOOP-7274:
------------------------------------

          Description: 
Same fix as for HADOOP-7057 for the Hadoop security branch

{noformat}
        throw new IOException( "Premeture EOF from inputStream");
{noformat}

  was:
{noformat}
        throw new IOException( "Premeture EOF from inputStream");
{noformat}

    Affects Version/s:     (was: 0.23.0)
                           (was: 0.22.0)
                       0.20.204.0
        Fix Version/s:     (was: 0.23.0)
                           (was: 0.22.0)

> CLONE - IOUtils.readFully and IOUtils.skipFully have typo in exception creation's message
> -----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7274
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7274
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.20.204.0
>            Reporter: Jonathan Eagles
>            Assignee: Jonathan Eagles
>            Priority: Minor
>
> Same fix as for HADOOP-7057 for the Hadoop security branch
> {noformat}
>         throw new IOException( "Premeture EOF from inputStream");
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14665-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 15:14:27 2011
Return-Path: <common-issues-return-14665-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5C72845E8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 15:14:27 +0000 (UTC)
Received: (qmail 50470 invoked by uid 500); 11 May 2011 15:14:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50439 invoked by uid 500); 11 May 2011 15:14:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50431 invoked by uid 99); 11 May 2011 15:14:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:14:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:14:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 687B6459BC
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 15:13:47 +0000 (UTC)
Date: Wed, 11 May 2011 15:13:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1573938455.3271.1305126827424.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <873708488.29176.1304715003217.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7267) Refactor FsShell's rm/rmr/expunge
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031770#comment-13031770 ] 

Hadoop QA commented on HADOOP-7267:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478830/HADOOP-7267.patch
  against trunk revision 1101735.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs.  The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/428//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/428//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/428//console

This message is automatically generated.

> Refactor FsShell's rm/rmr/expunge
> ---------------------------------
>
>                 Key: HADOOP-7267
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7267
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7267.patch
>
>
> Refactor to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14666-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 15:16:45 2011
Return-Path: <common-issues-return-14666-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7321A4C30
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 15:16:45 +0000 (UTC)
Received: (qmail 56808 invoked by uid 500); 11 May 2011 15:16:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56770 invoked by uid 500); 11 May 2011 15:16:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56762 invoked by uid 99); 11 May 2011 15:16:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:16:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:16:44 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9075D45B08
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 15:16:05 +0000 (UTC)
Date: Wed, 11 May 2011 15:16:05 +0000 (UTC)
From: "John George (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <431322288.3277.1305126965588.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1110985885.74832.1303423445828.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7237) Refactor FsShell's touchz
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031772#comment-13031772 ] 

John George commented on HADOOP-7237:
-------------------------------------

+1 looks good.

> Refactor FsShell's touchz
> -------------------------
>
>                 Key: HADOOP-7237
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7237
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7237.patch
>
>
> Need to refactor touchz to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14667-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 15:20:27 2011
Return-Path: <common-issues-return-14667-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4EC664CC4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 15:20:27 +0000 (UTC)
Received: (qmail 69321 invoked by uid 500); 11 May 2011 15:20:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69290 invoked by uid 500); 11 May 2011 15:20:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69275 invoked by uid 99); 11 May 2011 15:20:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:20:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:20:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6B61245CF4
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 15:19:47 +0000 (UTC)
Date: Wed, 11 May 2011 15:19:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1356062114.3289.1305127187436.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <873708488.29176.1304715003217.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7267) Refactor FsShell's rm/rmr/expunge
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031776#comment-13031776 ] 

Daryn Sharp commented on HADOOP-7267:
-------------------------------------

The findbugs warning is not in the code that I modified.  Note that it is in code that will soon be removed.

> Refactor FsShell's rm/rmr/expunge
> ---------------------------------
>
>                 Key: HADOOP-7267
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7267
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7267.patch
>
>
> Refactor to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14668-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 15:32:29 2011
Return-Path: <common-issues-return-14668-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BE9994CFA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 15:32:29 +0000 (UTC)
Received: (qmail 2957 invoked by uid 500); 11 May 2011 15:32:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2917 invoked by uid 500); 11 May 2011 15:32:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2904 invoked by uid 99); 11 May 2011 15:32:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:32:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:32:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A22EF452EC
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 15:31:47 +0000 (UTC)
Date: Wed, 11 May 2011 15:31:47 +0000 (UTC)
From: "Gabriele Kahlout (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1750428105.3327.1305127907661.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6763) Remove verbose logging from the
 Groups class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gabriele Kahlout updated HADOOP-6763:
-------------------------------------

    Affects Version/s: 0.21.0

> Remove verbose logging from the Groups class
> --------------------------------------------
>
>                 Key: HADOOP-6763
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6763
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Owen O'Malley
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6598-BP20-Fix.patch, HADOOP-6598-BP20.patch, HADOOP-6598.patch, HADOOP-6763.patch
>
>
> {quote}
> 2010-02-25 08:30:52,269 INFO  security.Groups (Groups.java:<init>(60)) - Group m
> apping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout
> =300000
> ...
> 2010-02-25 08:30:57,872 INFO  security.Groups (Groups.java:getGroups(76)) - Retu
> rning cached groups for 'oom'
> {quote}
> should both be demoted to debug level.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14669-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 15:34:28 2011
Return-Path: <common-issues-return-14669-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F0CB44706
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 15:34:27 +0000 (UTC)
Received: (qmail 5769 invoked by uid 500); 11 May 2011 15:34:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5724 invoked by uid 500); 11 May 2011 15:34:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5530 invoked by uid 99); 11 May 2011 15:34:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:34:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:34:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 78F604540B
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 15:33:47 +0000 (UTC)
Date: Wed, 11 May 2011 15:33:47 +0000 (UTC)
From: "Gabriele Kahlout (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1136202074.3336.1305128027492.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6763) Remove verbose logging from the
 Groups class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031785#comment-13031785 ] 

Gabriele Kahlout commented on HADOOP-6763:
------------------------------------------

okay, then shouldn't the fix version be set to 0.22?

> Remove verbose logging from the Groups class
> --------------------------------------------
>
>                 Key: HADOOP-6763
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6763
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Owen O'Malley
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6598-BP20-Fix.patch, HADOOP-6598-BP20.patch, HADOOP-6598.patch, HADOOP-6763.patch
>
>
> {quote}
> 2010-02-25 08:30:52,269 INFO  security.Groups (Groups.java:<init>(60)) - Group m
> apping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout
> =300000
> ...
> 2010-02-25 08:30:57,872 INFO  security.Groups (Groups.java:getGroups(76)) - Retu
> rning cached groups for 'oom'
> {quote}
> should both be demoted to debug level.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14670-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 15:34:29 2011
Return-Path: <common-issues-return-14670-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8D10A4714
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 15:34:29 +0000 (UTC)
Received: (qmail 6376 invoked by uid 500); 11 May 2011 15:34:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6341 invoked by uid 500); 11 May 2011 15:34:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6333 invoked by uid 99); 11 May 2011 15:34:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:34:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:34:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 899294540D
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 15:33:47 +0000 (UTC)
Date: Wed, 11 May 2011 15:33:47 +0000 (UTC)
From: "John George (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1498166397.3338.1305128027560.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <873708488.29176.1304715003217.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7267) Refactor FsShell's rm/rmr/expunge
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031786#comment-13031786 ] 

John George commented on HADOOP-7267:
-------------------------------------

+1 The code seems to do what it did before.

> Refactor FsShell's rm/rmr/expunge
> ---------------------------------
>
>                 Key: HADOOP-7267
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7267
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7267.patch
>
>
> Refactor to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14671-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 15:46:29 2011
Return-Path: <common-issues-return-14671-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6DC3844E0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 15:46:29 +0000 (UTC)
Received: (qmail 21340 invoked by uid 500); 11 May 2011 15:46:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21303 invoked by uid 500); 11 May 2011 15:46:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21295 invoked by uid 99); 11 May 2011 15:46:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:46:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:46:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6C6F545AAD
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 15:45:47 +0000 (UTC)
Date: Wed, 11 May 2011 15:45:47 +0000 (UTC)
From: "Jonathan Eagles (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1463086596.3365.1305128747440.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <88464990.3250.1305126467497.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7274) CLONE - IOUtils.readFully and
 IOUtils.skipFully have typo in exception creation's message
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Eagles updated HADOOP-7274:
------------------------------------

    Attachment: HADOOP-7274-0.20-security.patch

> CLONE - IOUtils.readFully and IOUtils.skipFully have typo in exception creation's message
> -----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7274
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7274
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.20.204.0
>            Reporter: Jonathan Eagles
>            Assignee: Jonathan Eagles
>            Priority: Minor
>         Attachments: HADOOP-7274-0.20-security.patch
>
>
> Same fix as for HADOOP-7057 for the Hadoop security branch
> {noformat}
>         throw new IOException( "Premeture EOF from inputStream");
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14672-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 15:46:29 2011
Return-Path: <common-issues-return-14672-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C6D994501
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 15:46:29 +0000 (UTC)
Received: (qmail 21741 invoked by uid 500); 11 May 2011 15:46:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21711 invoked by uid 500); 11 May 2011 15:46:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21663 invoked by uid 99); 11 May 2011 15:46:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:46:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:46:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9520845AB6
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 15:45:47 +0000 (UTC)
Date: Wed, 11 May 2011 15:45:47 +0000 (UTC)
From: "Jonathan Eagles (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1604970339.3371.1305128747607.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <88464990.3250.1305126467497.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7274) CLONE - IOUtils.readFully and
 IOUtils.skipFully have typo in exception creation's message
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Eagles updated HADOOP-7274:
------------------------------------

    Status: Patch Available  (was: Open)

> CLONE - IOUtils.readFully and IOUtils.skipFully have typo in exception creation's message
> -----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7274
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7274
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.20.204.0
>            Reporter: Jonathan Eagles
>            Assignee: Jonathan Eagles
>            Priority: Minor
>         Attachments: HADOOP-7274-0.20-security.patch
>
>
> Same fix as for HADOOP-7057 for the Hadoop security branch
> {noformat}
>         throw new IOException( "Premeture EOF from inputStream");
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14673-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 15:48:29 2011
Return-Path: <common-issues-return-14673-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 754684035
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 15:48:29 +0000 (UTC)
Received: (qmail 26596 invoked by uid 500); 11 May 2011 15:48:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26474 invoked by uid 500); 11 May 2011 15:48:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26466 invoked by uid 99); 11 May 2011 15:48:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:48:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:48:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6B7D145C19
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 15:47:47 +0000 (UTC)
Date: Wed, 11 May 2011 15:47:47 +0000 (UTC)
From: "Jonathan Eagles (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1518782908.3375.1305128867436.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <88464990.3250.1305126467497.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7274) CLONE - IOUtils.readFully and
 IOUtils.skipFully have typo in exception creation's message
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031794#comment-13031794 ] 

Jonathan Eagles commented on HADOOP-7274:
-----------------------------------------

No new tests are needed for this typo-fix issue.

> CLONE - IOUtils.readFully and IOUtils.skipFully have typo in exception creation's message
> -----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7274
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7274
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.20.204.0
>            Reporter: Jonathan Eagles
>            Assignee: Jonathan Eagles
>            Priority: Minor
>         Attachments: HADOOP-7274-0.20-security.patch
>
>
> Same fix as for HADOOP-7057 for the Hadoop security branch
> {noformat}
>         throw new IOException( "Premeture EOF from inputStream");
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14674-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 15:50:29 2011
Return-Path: <common-issues-return-14674-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 75B1F407D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 15:50:29 +0000 (UTC)
Received: (qmail 30313 invoked by uid 500); 11 May 2011 15:50:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30234 invoked by uid 500); 11 May 2011 15:50:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30150 invoked by uid 99); 11 May 2011 15:50:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:50:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:50:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5C18745CC6
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 15:49:47 +0000 (UTC)
Date: Wed, 11 May 2011 15:49:47 +0000 (UTC)
From: "Jonathan Eagles (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1912767830.3385.1305128987373.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <88464990.3250.1305126467497.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7274) CLONE - IOUtils.readFully and
 IOUtils.skipFully have typo in exception creation's message
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031796#comment-13031796 ] 

Jonathan Eagles commented on HADOOP-7274:
-----------------------------------------

No tests were added. Eclipse and javadoc warnings are previously present in the security branch

     [exec] -1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
     [exec]                         Please justify why no tests are needed for this patch.
     [exec] 
     [exec]     -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     -1 Eclipse classpath. The patch causes the Eclipse classpath to differ from the contents of the lib directories.
     [exec] 


> CLONE - IOUtils.readFully and IOUtils.skipFully have typo in exception creation's message
> -----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7274
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7274
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.20.204.0
>            Reporter: Jonathan Eagles
>            Assignee: Jonathan Eagles
>            Priority: Minor
>         Attachments: HADOOP-7274-0.20-security.patch
>
>
> Same fix as for HADOOP-7057 for the Hadoop security branch
> {noformat}
>         throw new IOException( "Premeture EOF from inputStream");
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14675-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 15:54:29 2011
Return-Path: <common-issues-return-14675-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B85C8428B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 15:54:29 +0000 (UTC)
Received: (qmail 35637 invoked by uid 500); 11 May 2011 15:54:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35599 invoked by uid 500); 11 May 2011 15:54:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35591 invoked by uid 99); 11 May 2011 15:54:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:54:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 15:54:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A13ED45F37
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 15:53:47 +0000 (UTC)
Date: Wed, 11 May 2011 15:53:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <635143507.3392.1305129227657.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <88464990.3250.1305126467497.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7274) CLONE - IOUtils.readFully and
 IOUtils.skipFully have typo in exception creation's message
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031798#comment-13031798 ] 

Hadoop QA commented on HADOOP-7274:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478834/HADOOP-7274-0.20-security.patch
  against trunk revision 1101735.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/429//console

This message is automatically generated.

> CLONE - IOUtils.readFully and IOUtils.skipFully have typo in exception creation's message
> -----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7274
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7274
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.20.204.0
>            Reporter: Jonathan Eagles
>            Assignee: Jonathan Eagles
>            Priority: Minor
>         Attachments: HADOOP-7274-0.20-security.patch
>
>
> Same fix as for HADOOP-7057 for the Hadoop security branch
> {noformat}
>         throw new IOException( "Premeture EOF from inputStream");
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14676-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 16:02:29 2011
Return-Path: <common-issues-return-14676-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F0336455B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 16:02:28 +0000 (UTC)
Received: (qmail 44061 invoked by uid 500); 11 May 2011 16:02:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43993 invoked by uid 500); 11 May 2011 16:02:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43963 invoked by uid 99); 11 May 2011 16:02:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 16:02:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 16:02:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 097B645383
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 16:01:48 +0000 (UTC)
Date: Wed, 11 May 2011 16:01:48 +0000 (UTC)
From: "Jonathan Eagles (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <928862306.3422.1305129708035.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <88464990.3250.1305126467497.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7274) CLONE - IOUtils.readFully and
 IOUtils.skipFully have typo in exception creation's message
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031806#comment-13031806 ] 

Jonathan Eagles commented on HADOOP-7274:
-----------------------------------------

The attached patch only applies to the security branch and not the trunk.

> CLONE - IOUtils.readFully and IOUtils.skipFully have typo in exception creation's message
> -----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7274
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7274
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.20.204.0
>            Reporter: Jonathan Eagles
>            Assignee: Jonathan Eagles
>            Priority: Minor
>         Attachments: HADOOP-7274-0.20-security.patch
>
>
> Same fix as for HADOOP-7057 for the Hadoop security branch
> {noformat}
>         throw new IOException( "Premeture EOF from inputStream");
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14677-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 16:10:31 2011
Return-Path: <common-issues-return-14677-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A5A7D4D73
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 16:10:31 +0000 (UTC)
Received: (qmail 63927 invoked by uid 500); 11 May 2011 16:10:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63852 invoked by uid 500); 11 May 2011 16:10:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63834 invoked by uid 99); 11 May 2011 16:10:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 16:10:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 16:10:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8646C457CB
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 16:09:47 +0000 (UTC)
Date: Wed, 11 May 2011 16:09:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1648991564.3455.1305130187546.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7275) Refactor FsShell's stat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Refactor FsShell's stat
-----------------------

                 Key: HADOOP-7275
                 URL: https://issues.apache.org/jira/browse/HADOOP-7275
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs
    Affects Versions: 0.23.0
            Reporter: Daryn Sharp
            Assignee: Daryn Sharp


Refactor to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14678-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 16:20:30 2011
Return-Path: <common-issues-return-14678-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B58A14A57
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 16:20:30 +0000 (UTC)
Received: (qmail 94156 invoked by uid 500); 11 May 2011 16:20:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94095 invoked by uid 500); 11 May 2011 16:20:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93878 invoked by uid 99); 11 May 2011 16:20:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 16:20:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 16:20:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 061D645EB9
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 16:19:48 +0000 (UTC)
Date: Wed, 11 May 2011 16:19:48 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <91100625.3517.1305130788021.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1648991564.3455.1305130187546.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7275) Refactor FsShell's stat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7275:
--------------------------------

    Attachment: HADOOP-7275.patch

> Refactor FsShell's stat
> -----------------------
>
>                 Key: HADOOP-7275
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7275
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7275.patch
>
>
> Refactor to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14679-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 16:20:31 2011
Return-Path: <common-issues-return-14679-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 01AEF4A63
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 16:20:31 +0000 (UTC)
Received: (qmail 94335 invoked by uid 500); 11 May 2011 16:20:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94285 invoked by uid 500); 11 May 2011 16:20:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94187 invoked by uid 99); 11 May 2011 16:20:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 16:20:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 16:20:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 63BC945EBB
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 16:19:48 +0000 (UTC)
Date: Wed, 11 May 2011 16:19:48 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <514910218.3519.1305130788405.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1648991564.3455.1305130187546.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7275) Refactor FsShell's stat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7275:
--------------------------------

    Status: Patch Available  (was: Open)

> Refactor FsShell's stat
> -----------------------
>
>                 Key: HADOOP-7275
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7275
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7275.patch
>
>
> Refactor to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14680-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 16:22:27 2011
Return-Path: <common-issues-return-14680-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9B3874D43
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 16:22:27 +0000 (UTC)
Received: (qmail 98628 invoked by uid 500); 11 May 2011 16:22:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98600 invoked by uid 500); 11 May 2011 16:22:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98592 invoked by uid 99); 11 May 2011 16:22:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 16:22:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 16:22:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6104645F8D
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 16:21:47 +0000 (UTC)
Date: Wed, 11 May 2011 16:21:47 +0000 (UTC)
From: "Trevor Robinson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <532896131.3523.1305130907394.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7276) Hadoop native builds fail on ARM due
 to -m32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Hadoop native builds fail on ARM due to -m32
--------------------------------------------

                 Key: HADOOP-7276
                 URL: https://issues.apache.org/jira/browse/HADOOP-7276
             Project: Hadoop Common
          Issue Type: Bug
          Components: native
    Affects Versions: 0.21.0
         Environment: $ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/arm-linux-gnueabi/gcc/arm-linux-gnueabi/4.5.2/lto-wrapper
Target: arm-linux-gnueabi
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.5.2-8ubuntu4' --with-bugurl=file:///usr/share/doc/gcc-4.5/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.5 --enable-shared --enable-multiarch --with-multiarch-defaults=arm-linux-gnueabi --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib/arm-linux-gnueabi --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.5 --libdir=/usr/lib/arm-linux-gnueabi --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-plugin --enable-gold --enable-ld=default --with-plugin-ld=ld.gold --enable-objc-gc --disable-sjlj-exceptions --with-arch=armv7-a --with-float=softfp --with-fpu=vfpv3-d16 --with-mode=thumb --disable-werror --enable-checking=release --build=arm-linux-gnueabi --host=arm-linux-gnueabi --target=arm-linux-gnueabi
Thread model: posix
gcc version 4.5.2 (Ubuntu/Linaro 4.5.2-8ubuntu4)
$ uname -a
Linux panda0 2.6.38-1002-linaro-omap #3-Ubuntu SMP Fri Apr 15 14:00:54 UTC 2011 armv7l armv7l armv7l GNU/Linux

            Reporter: Trevor Robinson


The native build fails on machine targets where gcc does not support -m32. This is any target other than x86, SPARC, RS/6000, or PowerPC, such as ARM.

$ ant -Dcompile.native=true
...
     [exec] make  all-am
     [exec] make[1]: Entering directory
`/home/trobinson/dev/hadoop-common/build/native/Linux-arm-32'
     [exec] /bin/bash ./libtool  --tag=CC   --mode=compile gcc
-DHAVE_CONFIG_H -I. -I/home/trobinson/dev/hadoop-common/src/native
-I/usr/lib/jvm/java-6-openjdk/include
-I/usr/lib/jvm/java-6-openjdk/include/linux
-I/home/trobinson/dev/hadoop-common/src/native/src
-Isrc/org/apache/hadoop/io/compress/zlib
-Isrc/org/apache/hadoop/security -Isrc/org/apache/hadoop/io/nativeio/
-g -Wall -fPIC -O2 -m32 -g -O2 -MT ZlibCompressor.lo -MD -MP -MF
.deps/ZlibCompressor.Tpo -c -o ZlibCompressor.lo `test -f
'src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c' || echo
'/home/trobinson/dev/hadoop-common/src/native/'`src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
     [exec] libtool: compile:  gcc -DHAVE_CONFIG_H -I.
-I/home/trobinson/dev/hadoop-common/src/native
-I/usr/lib/jvm/java-6-openjdk/include
-I/usr/lib/jvm/java-6-openjdk/include/linux
-I/home/trobinson/dev/hadoop-common/src/native/src
-Isrc/org/apache/hadoop/io/compress/zlib
-Isrc/org/apache/hadoop/security -Isrc/org/apache/hadoop/io/nativeio/
-g -Wall -fPIC -O2 -m32 -g -O2 -MT ZlibCompressor.lo -MD -MP -MF
.deps/ZlibCompressor.Tpo -c
/home/trobinson/dev/hadoop-common/src/native/src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
 -fPIC -DPIC -o .libs/ZlibCompressor.o
     [exec] make[1]: Leaving directory
`/home/trobinson/dev/hadoop-common/build/native/Linux-arm-32'
     [exec] cc1: error: unrecognized command line option "-m32"
     [exec] make[1]: *** [ZlibCompressor.lo] Error 1
     [exec] make: *** [all] Error 2


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14681-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 16:28:27 2011
Return-Path: <common-issues-return-14681-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C2F894FE8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 16:28:27 +0000 (UTC)
Received: (qmail 13607 invoked by uid 500); 11 May 2011 16:28:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13558 invoked by uid 500); 11 May 2011 16:28:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13550 invoked by uid 99); 11 May 2011 16:28:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 16:28:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 16:28:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5EC0245249
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 16:27:47 +0000 (UTC)
Date: Wed, 11 May 2011 16:27:47 +0000 (UTC)
From: "Trevor Robinson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <485512998.3544.1305131267384.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <532896131.3523.1305130907394.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7276) Hadoop native builds fail on ARM due
 to -m32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Trevor Robinson updated HADOOP-7276:
------------------------------------

    Attachment: hadoop-common-arm.patch

This patch introduces an AM_CONDITIONAL called SPECIFY_DATA_MODEL that is disabled when host_cpu starts with "arm" (and is easily extensible for other CPUs). -m$(JVM_DATA_MODEL) is only added to AM_CFLAGS and AM_LDFLAGS when SPECIFY_DATA_MODEL is true.

> Hadoop native builds fail on ARM due to -m32
> --------------------------------------------
>
>                 Key: HADOOP-7276
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7276
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.21.0
>         Environment: $ gcc -v
> Using built-in specs.
> COLLECT_GCC=gcc
> COLLECT_LTO_WRAPPER=/usr/lib/arm-linux-gnueabi/gcc/arm-linux-gnueabi/4.5.2/lto-wrapper
> Target: arm-linux-gnueabi
> Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.5.2-8ubuntu4' --with-bugurl=file:///usr/share/doc/gcc-4.5/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.5 --enable-shared --enable-multiarch --with-multiarch-defaults=arm-linux-gnueabi --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib/arm-linux-gnueabi --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.5 --libdir=/usr/lib/arm-linux-gnueabi --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-plugin --enable-gold --enable-ld=default --with-plugin-ld=ld.gold --enable-objc-gc --disable-sjlj-exceptions --with-arch=armv7-a --with-float=softfp --with-fpu=vfpv3-d16 --with-mode=thumb --disable-werror --enable-checking=release --build=arm-linux-gnueabi --host=arm-linux-gnueabi --target=arm-linux-gnueabi
> Thread model: posix
> gcc version 4.5.2 (Ubuntu/Linaro 4.5.2-8ubuntu4)
> $ uname -a
> Linux panda0 2.6.38-1002-linaro-omap #3-Ubuntu SMP Fri Apr 15 14:00:54 UTC 2011 armv7l armv7l armv7l GNU/Linux
>            Reporter: Trevor Robinson
>         Attachments: hadoop-common-arm.patch
>
>
> The native build fails on machine targets where gcc does not support -m32. This is any target other than x86, SPARC, RS/6000, or PowerPC, such as ARM.
> $ ant -Dcompile.native=true
> ...
>      [exec] make  all-am
>      [exec] make[1]: Entering directory
> `/home/trobinson/dev/hadoop-common/build/native/Linux-arm-32'
>      [exec] /bin/bash ./libtool  --tag=CC   --mode=compile gcc
> -DHAVE_CONFIG_H -I. -I/home/trobinson/dev/hadoop-common/src/native
> -I/usr/lib/jvm/java-6-openjdk/include
> -I/usr/lib/jvm/java-6-openjdk/include/linux
> -I/home/trobinson/dev/hadoop-common/src/native/src
> -Isrc/org/apache/hadoop/io/compress/zlib
> -Isrc/org/apache/hadoop/security -Isrc/org/apache/hadoop/io/nativeio/
> -g -Wall -fPIC -O2 -m32 -g -O2 -MT ZlibCompressor.lo -MD -MP -MF
> .deps/ZlibCompressor.Tpo -c -o ZlibCompressor.lo `test -f
> 'src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c' || echo
> '/home/trobinson/dev/hadoop-common/src/native/'`src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
>      [exec] libtool: compile:  gcc -DHAVE_CONFIG_H -I.
> -I/home/trobinson/dev/hadoop-common/src/native
> -I/usr/lib/jvm/java-6-openjdk/include
> -I/usr/lib/jvm/java-6-openjdk/include/linux
> -I/home/trobinson/dev/hadoop-common/src/native/src
> -Isrc/org/apache/hadoop/io/compress/zlib
> -Isrc/org/apache/hadoop/security -Isrc/org/apache/hadoop/io/nativeio/
> -g -Wall -fPIC -O2 -m32 -g -O2 -MT ZlibCompressor.lo -MD -MP -MF
> .deps/ZlibCompressor.Tpo -c
> /home/trobinson/dev/hadoop-common/src/native/src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
>  -fPIC -DPIC -o .libs/ZlibCompressor.o
>      [exec] make[1]: Leaving directory
> `/home/trobinson/dev/hadoop-common/build/native/Linux-arm-32'
>      [exec] cc1: error: unrecognized command line option "-m32"
>      [exec] make[1]: *** [ZlibCompressor.lo] Error 1
>      [exec] make: *** [all] Error 2

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14682-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 16:34:29 2011
Return-Path: <common-issues-return-14682-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6FFB7475B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 16:34:29 +0000 (UTC)
Received: (qmail 22174 invoked by uid 500); 11 May 2011 16:34:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22130 invoked by uid 500); 11 May 2011 16:34:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22122 invoked by uid 99); 11 May 2011 16:34:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 16:34:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 16:34:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 591DB4545A
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 16:33:47 +0000 (UTC)
Date: Wed, 11 May 2011 16:33:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1797979305.3556.1305131627361.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1648991564.3455.1305130187546.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7275) Refactor FsShell's stat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031834#comment-13031834 ] 

Hadoop QA commented on HADOOP-7275:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478843/HADOOP-7275.patch
  against trunk revision 1101735.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/430//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/430//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/430//console

This message is automatically generated.

> Refactor FsShell's stat
> -----------------------
>
>                 Key: HADOOP-7275
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7275
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7275.patch
>
>
> Refactor to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14683-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 17:24:27 2011
Return-Path: <common-issues-return-14683-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 96F764858
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 17:24:27 +0000 (UTC)
Received: (qmail 4077 invoked by uid 500); 11 May 2011 17:24:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3970 invoked by uid 500); 11 May 2011 17:24:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3962 invoked by uid 99); 11 May 2011 17:24:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 17:24:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 17:24:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 601D2459EB
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 17:23:47 +0000 (UTC)
Date: Wed, 11 May 2011 17:23:47 +0000 (UTC)
From: "Jeffrey Naisbitt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <161449362.3682.1305134627390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7277) Add Eclipse launch tasks for the
 0.20-security branch
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Add Eclipse launch tasks for the 0.20-security branch
-----------------------------------------------------

                 Key: HADOOP-7277
                 URL: https://issues.apache.org/jira/browse/HADOOP-7277
             Project: Hadoop Common
          Issue Type: Improvement
          Components: build
    Affects Versions: 0.20.204.0
            Reporter: Jeffrey Naisbitt
            Assignee: Jeffrey Naisbitt
            Priority: Minor
             Fix For: 0.20.205.0


This is to add the eclipse launchers from HADOOP-5911 to the 0.20 security branch.

Eclipse has a notion of "run configuration", which encapsulates what's needed to run or debug an application. I use this quite a bit to start various Hadoop daemons in debug mode, with breakpoints set, to inspect state and what not.

This is simply configuration, so no tests are provided. After running "ant eclipse" and refreshing your project, you should see entries in the Run Configurations and Debug Configurations for launching the various hadoop daemons from within eclipse. There's a template for testing a specific test, and also templates to run all the tests, the job tracker, and a task tracker. It's likely that some parameters need to be further tweaked to have the same behavior as "ant test", but for most tests, this works.

This also requires a small change to build.xml for the eclipse classpath.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14684-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 17:34:27 2011
Return-Path: <common-issues-return-14684-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CEACA4FF6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 17:34:27 +0000 (UTC)
Received: (qmail 17734 invoked by uid 500); 11 May 2011 17:34:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17696 invoked by uid 500); 11 May 2011 17:34:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17672 invoked by uid 99); 11 May 2011 17:34:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 17:34:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 17:34:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CB7B545ECD
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 17:33:47 +0000 (UTC)
Date: Wed, 11 May 2011 17:33:47 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2017704214.3719.1305135227829.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-6255:
------------------------------

    Attachment: HADOOP-6255-branch-0.20-security-9.patch

Rearrange files to proposed directory structure rather than using symlinks for 0.20security branch.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-6255-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14685-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 17:44:27 2011
Return-Path: <common-issues-return-14685-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7FFEE424C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 17:44:27 +0000 (UTC)
Received: (qmail 32690 invoked by uid 500); 11 May 2011 17:44:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32655 invoked by uid 500); 11 May 2011 17:44:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32644 invoked by uid 99); 11 May 2011 17:44:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 17:44:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 17:44:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9FC064530D
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 17:43:47 +0000 (UTC)
Date: Wed, 11 May 2011 17:43:47 +0000 (UTC)
From: "Jeffrey Naisbitt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1934309629.3774.1305135827650.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <161449362.3682.1305134627390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7277) Add Eclipse launch tasks for the
 0.20-security branch
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jeffrey Naisbitt updated HADOOP-7277:
-------------------------------------

    Attachment: HADOOP-7277-0.20s.patch

Patch for the 0.20-security branch

> Add Eclipse launch tasks for the 0.20-security branch
> -----------------------------------------------------
>
>                 Key: HADOOP-7277
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7277
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.204.0
>            Reporter: Jeffrey Naisbitt
>            Assignee: Jeffrey Naisbitt
>            Priority: Minor
>             Fix For: 0.20.205.0
>
>         Attachments: HADOOP-7277-0.20s.patch
>
>
> This is to add the eclipse launchers from HADOOP-5911 to the 0.20 security branch.
> Eclipse has a notion of "run configuration", which encapsulates what's needed to run or debug an application. I use this quite a bit to start various Hadoop daemons in debug mode, with breakpoints set, to inspect state and what not.
> This is simply configuration, so no tests are provided. After running "ant eclipse" and refreshing your project, you should see entries in the Run Configurations and Debug Configurations for launching the various hadoop daemons from within eclipse. There's a template for testing a specific test, and also templates to run all the tests, the job tracker, and a task tracker. It's likely that some parameters need to be further tweaked to have the same behavior as "ant test", but for most tests, this works.
> This also requires a small change to build.xml for the eclipse classpath.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14686-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 17:44:28 2011
Return-Path: <common-issues-return-14686-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CBC5C425A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 17:44:28 +0000 (UTC)
Received: (qmail 32957 invoked by uid 500); 11 May 2011 17:44:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32921 invoked by uid 500); 11 May 2011 17:44:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32913 invoked by uid 99); 11 May 2011 17:44:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 17:44:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 17:44:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E61E84533B
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 17:43:48 +0000 (UTC)
Date: Wed, 11 May 2011 17:43:48 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <566677411.3805.1305135828939.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031881#comment-13031881 ] 

Hadoop QA commented on HADOOP-6255:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478846/HADOOP-6255-branch-0.20-security-9.patch
  against trunk revision 1101735.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 1 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/431//console

This message is automatically generated.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-6255-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14687-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 17:44:31 2011
Return-Path: <common-issues-return-14687-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3834E426E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 17:44:31 +0000 (UTC)
Received: (qmail 33242 invoked by uid 500); 11 May 2011 17:44:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33193 invoked by uid 500); 11 May 2011 17:44:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33180 invoked by uid 99); 11 May 2011 17:44:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 17:44:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 17:44:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F3CB24533F
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 17:43:48 +0000 (UTC)
Date: Wed, 11 May 2011 17:43:48 +0000 (UTC)
From: "Jeffrey Naisbitt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <672567595.3807.1305135828994.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <161449362.3682.1305134627390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7277) Add Eclipse launch tasks for the
 0.20-security branch
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jeffrey Naisbitt updated HADOOP-7277:
-------------------------------------

    Status: Patch Available  (was: Open)

> Add Eclipse launch tasks for the 0.20-security branch
> -----------------------------------------------------
>
>                 Key: HADOOP-7277
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7277
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.204.0
>            Reporter: Jeffrey Naisbitt
>            Assignee: Jeffrey Naisbitt
>            Priority: Minor
>             Fix For: 0.20.205.0
>
>         Attachments: HADOOP-7277-0.20s.patch
>
>
> This is to add the eclipse launchers from HADOOP-5911 to the 0.20 security branch.
> Eclipse has a notion of "run configuration", which encapsulates what's needed to run or debug an application. I use this quite a bit to start various Hadoop daemons in debug mode, with breakpoints set, to inspect state and what not.
> This is simply configuration, so no tests are provided. After running "ant eclipse" and refreshing your project, you should see entries in the Run Configurations and Debug Configurations for launching the various hadoop daemons from within eclipse. There's a template for testing a specific test, and also templates to run all the tests, the job tracker, and a task tracker. It's likely that some parameters need to be further tweaked to have the same behavior as "ant test", but for most tests, this works.
> This also requires a small change to build.xml for the eclipse classpath.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14688-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 17:52:28 2011
Return-Path: <common-issues-return-14688-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9DBAA4A87
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 17:52:28 +0000 (UTC)
Received: (qmail 42516 invoked by uid 500); 11 May 2011 17:52:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42485 invoked by uid 500); 11 May 2011 17:52:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42476 invoked by uid 99); 11 May 2011 17:52:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 17:52:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 17:52:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 94B8F45708
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 17:51:48 +0000 (UTC)
Date: Wed, 11 May 2011 17:51:48 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1483103966.3855.1305136308605.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031886#comment-13031886 ] 

Eli Collins commented on HADOOP-6255:
-------------------------------------

Patch should be against trunk right?

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-6255-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14689-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 17:54:30 2011
Return-Path: <common-issues-return-14689-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F20594AE4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 17:54:29 +0000 (UTC)
Received: (qmail 44322 invoked by uid 500); 11 May 2011 17:54:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44266 invoked by uid 500); 11 May 2011 17:54:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44258 invoked by uid 99); 11 May 2011 17:54:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 17:54:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 17:54:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B665245827
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 17:53:47 +0000 (UTC)
Date: Wed, 11 May 2011 17:53:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1635990069.3860.1305136427743.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <161449362.3682.1305134627390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7277) Add Eclipse launch tasks for the
 0.20-security branch
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031888#comment-13031888 ] 

Hadoop QA commented on HADOOP-7277:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478849/HADOOP-7277-0.20s.patch
  against trunk revision 1101735.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/432//console

This message is automatically generated.

> Add Eclipse launch tasks for the 0.20-security branch
> -----------------------------------------------------
>
>                 Key: HADOOP-7277
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7277
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.204.0
>            Reporter: Jeffrey Naisbitt
>            Assignee: Jeffrey Naisbitt
>            Priority: Minor
>             Fix For: 0.20.205.0
>
>         Attachments: HADOOP-7277-0.20s.patch
>
>
> This is to add the eclipse launchers from HADOOP-5911 to the 0.20 security branch.
> Eclipse has a notion of "run configuration", which encapsulates what's needed to run or debug an application. I use this quite a bit to start various Hadoop daemons in debug mode, with breakpoints set, to inspect state and what not.
> This is simply configuration, so no tests are provided. After running "ant eclipse" and refreshing your project, you should see entries in the Run Configurations and Debug Configurations for launching the various hadoop daemons from within eclipse. There's a template for testing a specific test, and also templates to run all the tests, the job tracker, and a task tracker. It's likely that some parameters need to be further tweaked to have the same behavior as "ant test", but for most tests, this works.
> This also requires a small change to build.xml for the eclipse classpath.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14690-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 18:02:30 2011
Return-Path: <common-issues-return-14690-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AA80049C2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 18:02:30 +0000 (UTC)
Received: (qmail 57147 invoked by uid 500); 11 May 2011 18:02:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57105 invoked by uid 500); 11 May 2011 18:02:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57097 invoked by uid 99); 11 May 2011 18:02:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 18:02:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 18:02:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4E86245D38
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 18:01:48 +0000 (UTC)
Date: Wed, 11 May 2011 18:01:48 +0000 (UTC)
From: "Jeffrey Naisbitt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <272741462.3945.1305136908318.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <161449362.3682.1305134627390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7277) Add Eclipse launch tasks for the
 0.20-security branch
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031894#comment-13031894 ] 

Jeffrey Naisbitt commented on HADOOP-7277:
------------------------------------------

The patch is for the 0.20-security branch, so the above results should be ignored.  Here are the results from my manual run:
     [exec] -1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 6 new or modified tests.
     [exec] 
     [exec]     -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     -1 Eclipse classpath. The patch causes the Eclipse classpath to differ from the contents of the lib directories.

The javadoc and Eclipse classpath issues are unrelated to these changes.  I'm also not sure why it says 6 new/modified tests are included since none are included in this patch.  None are needed though since it's just eclipse things.


> Add Eclipse launch tasks for the 0.20-security branch
> -----------------------------------------------------
>
>                 Key: HADOOP-7277
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7277
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.204.0
>            Reporter: Jeffrey Naisbitt
>            Assignee: Jeffrey Naisbitt
>            Priority: Minor
>             Fix For: 0.20.205.0
>
>         Attachments: HADOOP-7277-0.20s.patch
>
>
> This is to add the eclipse launchers from HADOOP-5911 to the 0.20 security branch.
> Eclipse has a notion of "run configuration", which encapsulates what's needed to run or debug an application. I use this quite a bit to start various Hadoop daemons in debug mode, with breakpoints set, to inspect state and what not.
> This is simply configuration, so no tests are provided. After running "ant eclipse" and refreshing your project, you should see entries in the Run Configurations and Debug Configurations for launching the various hadoop daemons from within eclipse. There's a template for testing a specific test, and also templates to run all the tests, the job tracker, and a task tracker. It's likely that some parameters need to be further tweaked to have the same behavior as "ant test", but for most tests, this works.
> This also requires a small change to build.xml for the eclipse classpath.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14691-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 18:17:30 2011
Return-Path: <common-issues-return-14691-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 19D70416B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 18:17:30 +0000 (UTC)
Received: (qmail 79894 invoked by uid 500); 11 May 2011 18:17:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79850 invoked by uid 500); 11 May 2011 18:17:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79757 invoked by uid 99); 11 May 2011 18:17:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 18:17:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 18:17:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 78D5F45577
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 18:16:49 +0000 (UTC)
Date: Wed, 11 May 2011 18:16:49 +0000 (UTC)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1894387424.4008.1305137809491.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7278) Automatic Hadoop cluster deployment
 for build validation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Automatic Hadoop cluster deployment for build validation
--------------------------------------------------------

                 Key: HADOOP-7278
                 URL: https://issues.apache.org/jira/browse/HADOOP-7278
             Project: Hadoop Common
          Issue Type: Improvement
          Components: build
    Affects Versions: 0.22.0, 0.23.0
         Environment: Apache Jenkins
            Reporter: Konstantin Boudnik
            Assignee: Konstantin Boudnik




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14692-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 18:25:27 2011
Return-Path: <common-issues-return-14692-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 75A014FE4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 18:25:27 +0000 (UTC)
Received: (qmail 607 invoked by uid 500); 11 May 2011 18:25:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 545 invoked by uid 500); 11 May 2011 18:25:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 537 invoked by uid 99); 11 May 2011 18:25:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 18:25:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 18:25:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6F27E45BD9
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 18:24:47 +0000 (UTC)
Date: Wed, 11 May 2011 18:24:47 +0000 (UTC)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2059519263.4060.1305138287451.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1894387424.4008.1305137809491.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7278) Automatic Hadoop cluster deployment
 for build validation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-7278:
---------------------------------------

    Description: It'd be great to have a way of automatically deploying Hadoop cluster to a set of machine once all components are successfully built (in the form or tar or whatever). Deployed cluster then can be used to run a set of validation jobs to make sure that produced artifacts are viable. 

> Automatic Hadoop cluster deployment for build validation
> --------------------------------------------------------
>
>                 Key: HADOOP-7278
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7278
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0, 0.23.0
>         Environment: Apache Jenkins
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>              Labels: newbie
>
> It'd be great to have a way of automatically deploying Hadoop cluster to a set of machine once all components are successfully built (in the form or tar or whatever). Deployed cluster then can be used to run a set of validation jobs to make sure that produced artifacts are viable. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14693-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 18:25:29 2011
Return-Path: <common-issues-return-14693-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A8D6A400A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 18:25:29 +0000 (UTC)
Received: (qmail 1444 invoked by uid 500); 11 May 2011 18:25:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1408 invoked by uid 500); 11 May 2011 18:25:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1400 invoked by uid 99); 11 May 2011 18:25:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 18:25:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 18:25:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8D82845BDD
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 18:24:47 +0000 (UTC)
Date: Wed, 11 May 2011 18:24:47 +0000 (UTC)
From: "John George (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1447049747.4064.1305138287576.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1648991564.3455.1305130187546.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7275) Refactor FsShell's stat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031921#comment-13031921 ] 

John George commented on HADOOP-7275:
-------------------------------------

+1 The code looks good.

> Refactor FsShell's stat
> -----------------------
>
>                 Key: HADOOP-7275
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7275
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7275.patch
>
>
> Refactor to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14694-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 18:27:27 2011
Return-Path: <common-issues-return-14694-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B3E1D4038
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 18:27:27 +0000 (UTC)
Received: (qmail 4861 invoked by uid 500); 11 May 2011 18:27:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4828 invoked by uid 500); 11 May 2011 18:27:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4820 invoked by uid 99); 11 May 2011 18:27:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 18:27:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 18:27:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BFE2845CD8
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 18:26:47 +0000 (UTC)
Date: Wed, 11 May 2011 18:26:47 +0000 (UTC)
From: "Adrian Cole (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <372749453.4066.1305138407782.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1894387424.4008.1305137809491.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7278) Automatic Hadoop cluster
 deployment for build validation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031923#comment-13031923 ] 

Adrian Cole commented on HADOOP-7278:
-------------------------------------

seems like the ideal case for whirr, right? http://incubator.apache.org/whirr/

> Automatic Hadoop cluster deployment for build validation
> --------------------------------------------------------
>
>                 Key: HADOOP-7278
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7278
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0, 0.23.0
>         Environment: Apache Jenkins
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>              Labels: newbie
>
> It'd be great to have a way of automatically deploying Hadoop cluster to a set of machine once all components are successfully built (in the form or tar or whatever). Deployed cluster then can be used to run a set of validation jobs to make sure that produced artifacts are viable. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14695-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 18:29:29 2011
Return-Path: <common-issues-return-14695-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9AC414069
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 18:29:29 +0000 (UTC)
Received: (qmail 6596 invoked by uid 500); 11 May 2011 18:29:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6568 invoked by uid 500); 11 May 2011 18:29:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6560 invoked by uid 99); 11 May 2011 18:29:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 18:29:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 18:29:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9D74A45E60
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 18:28:49 +0000 (UTC)
Date: Wed, 11 May 2011 18:28:49 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2120909595.4100.1305138529641.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031925#comment-13031925 ] 

Eric Yang commented on HADOOP-6255:
-----------------------------------

> Patch should be against trunk right?

Yes, there are 2 set of patches, one for trunk and one for 0.20 security branch.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-6255-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14696-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 18:29:32 2011
Return-Path: <common-issues-return-14696-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E68CC409E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 18:29:31 +0000 (UTC)
Received: (qmail 7840 invoked by uid 500); 11 May 2011 18:29:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7808 invoked by uid 500); 11 May 2011 18:29:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7800 invoked by uid 99); 11 May 2011 18:29:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 18:29:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 18:29:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CDF9745E6D
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 18:28:49 +0000 (UTC)
Date: Wed, 11 May 2011 18:28:49 +0000 (UTC)
From: "Travis Crawford (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1531104333.4106.1305138529840.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6746) hadoop-daemon.sh does not append to
 log files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Travis Crawford updated HADOOP-6746:
------------------------------------

    Resolution: Won't Fix
        Status: Resolved  (was: Patch Available)

> hadoop-daemon.sh does not append to log files
> ---------------------------------------------
>
>                 Key: HADOOP-6746
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6746
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Travis Crawford
>         Attachments: HADOOP-6746.patch
>
>
> Looking in hadoop-daemon.sh we see the start command redirects logs to the log file by creating a new file. It does not append to the log.
> nohup nice -n $HADOOP_NICENESS "$HADOOP_HOME"/bin/hadoop --config $HADOOP_CONF_DIR $command "$@" > "$log" 2>&1 < /dev/null &
> I believe output should be appended to the log file so logrotate copytruncate works. Currently a large log file cannot be truncated without restarting the running service, which causes issues in production environments.
> The improved line would be exactly the same, except use >> to open the log file in append mode.
> nohup nice -n $HADOOP_NICENESS "$HADOOP_HOME"/bin/hadoop --config $HADOOP_CONF_DIR $command "$@" >> "$log" 2>&1 < /dev/null &

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14697-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 18:38:29 2011
Return-Path: <common-issues-return-14697-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 827E047CD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 18:38:29 +0000 (UTC)
Received: (qmail 23829 invoked by uid 500); 11 May 2011 18:38:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23794 invoked by uid 500); 11 May 2011 18:38:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23784 invoked by uid 99); 11 May 2011 18:38:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 18:38:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 18:38:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 74346452F0
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 18:37:47 +0000 (UTC)
Date: Wed, 11 May 2011 18:37:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1636054751.4136.1305139067472.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1648991564.3455.1305130187546.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7275) Refactor FsShell's stat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7275:
-------------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I have committed this.  Thanks, Daryn!

Also thanks John for reviewing it.

> Refactor FsShell's stat
> -----------------------
>
>                 Key: HADOOP-7275
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7275
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7275.patch
>
>
> Refactor to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14698-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 18:56:29 2011
Return-Path: <common-issues-return-14698-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9435A413F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 18:56:29 +0000 (UTC)
Received: (qmail 54914 invoked by uid 500); 11 May 2011 18:56:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54871 invoked by uid 500); 11 May 2011 18:56:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54863 invoked by uid 99); 11 May 2011 18:56:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 18:56:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 18:56:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 703E245E24
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 18:55:47 +0000 (UTC)
Date: Wed, 11 May 2011 18:55:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1425338217.4200.1305140147456.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1648991564.3455.1305130187546.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7275) Refactor FsShell's stat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031949#comment-13031949 ] 

Hudson commented on HADOOP-7275:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #589 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/589/])
    HADOOP-7275. Refactor the stat commands to conform to new FsCommand class.  Contributed by Daryn Sharp


> Refactor FsShell's stat
> -----------------------
>
>                 Key: HADOOP-7275
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7275
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7275.patch
>
>
> Refactor to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14699-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 19:00:28 2011
Return-Path: <common-issues-return-14699-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C34F84472
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 19:00:28 +0000 (UTC)
Received: (qmail 61470 invoked by uid 500); 11 May 2011 19:00:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61447 invoked by uid 500); 11 May 2011 19:00:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61435 invoked by uid 99); 11 May 2011 19:00:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 19:00:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 19:00:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C66184502F
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 18:59:48 +0000 (UTC)
Date: Wed, 11 May 2011 18:59:48 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2063829655.4212.1305140388808.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1110985885.74832.1303423445828.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7237) Refactor FsShell's touchz
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7237:
--------------------------------

    Attachment: HADOOP-7237-1.patch

Resolve merge conflicts from removal of other commands.

> Refactor FsShell's touchz
> -------------------------
>
>                 Key: HADOOP-7237
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7237
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7237-1.patch, HADOOP-7237.patch
>
>
> Need to refactor touchz to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14700-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 19:10:28 2011
Return-Path: <common-issues-return-14700-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4BF764BC5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 19:10:28 +0000 (UTC)
Received: (qmail 89455 invoked by uid 500); 11 May 2011 19:10:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89411 invoked by uid 500); 11 May 2011 19:10:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89378 invoked by uid 99); 11 May 2011 19:10:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 19:10:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 19:10:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DC32045723
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 19:09:47 +0000 (UTC)
Date: Wed, 11 May 2011 19:09:47 +0000 (UTC)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1577742967.4303.1305140987898.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1894387424.4008.1305137809491.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7278) Automatic Hadoop cluster
 deployment for build validation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031973#comment-13031973 ] 

Konstantin Boudnik commented on HADOOP-7278:
--------------------------------------------

Here's the idea (and the workflow Roman and I had in mind):
- a Jenkins job creates a set of tar-balls for all components (similar to [what is done here|https://builds.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-22-Build/]
- a primitive .deb package (all build machines are Ubuntu based) is created which contains all bits and pieces with required permissions, etc.) and uploaded to a shared location where all build machines can see it
- an event is triggered once the package is ready puppet is executed on a designated set of machines to install the package into /opt/hadoop, generate configs, and start required services
- a validation workload is started by Hudson

This JIRA is going to have a couple of patches:
- .deb creation script
- puppet recipes


> Automatic Hadoop cluster deployment for build validation
> --------------------------------------------------------
>
>                 Key: HADOOP-7278
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7278
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0, 0.23.0
>         Environment: Apache Jenkins
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>              Labels: newbie
>
> It'd be great to have a way of automatically deploying Hadoop cluster to a set of machine once all components are successfully built (in the form or tar or whatever). Deployed cluster then can be used to run a set of validation jobs to make sure that produced artifacts are viable. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14701-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 19:12:29 2011
Return-Path: <common-issues-return-14701-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B22064C85
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 19:12:29 +0000 (UTC)
Received: (qmail 95210 invoked by uid 500); 11 May 2011 19:12:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95163 invoked by uid 500); 11 May 2011 19:12:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95155 invoked by uid 99); 11 May 2011 19:12:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 19:12:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 19:12:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7E7DB45871
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 19:11:47 +0000 (UTC)
Date: Wed, 11 May 2011 19:11:47 +0000 (UTC)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1511272010.4311.1305141107515.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1894387424.4008.1305137809491.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7278) Automatic Hadoop cluster
 deployment for build validation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031974#comment-13031974 ] 

Konstantin Boudnik commented on HADOOP-7278:
--------------------------------------------

Whirr might or mightn't be a better solution for this task, but certainly will be an additional learning curve ;)

> Automatic Hadoop cluster deployment for build validation
> --------------------------------------------------------
>
>                 Key: HADOOP-7278
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7278
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0, 0.23.0
>         Environment: Apache Jenkins
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>              Labels: newbie
>
> It'd be great to have a way of automatically deploying Hadoop cluster to a set of machine once all components are successfully built (in the form or tar or whatever). Deployed cluster then can be used to run a set of validation jobs to make sure that produced artifacts are viable. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14702-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 19:14:29 2011
Return-Path: <common-issues-return-14702-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7A52B4CEE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 19:14:29 +0000 (UTC)
Received: (qmail 1055 invoked by uid 500); 11 May 2011 19:14:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1027 invoked by uid 500); 11 May 2011 19:14:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1019 invoked by uid 99); 11 May 2011 19:14:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 19:14:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 19:14:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5E0CF459BD
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 19:13:47 +0000 (UTC)
Date: Wed, 11 May 2011 19:13:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1030873600.4327.1305141227381.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1110985885.74832.1303423445828.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7237) Refactor FsShell's touchz
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031976#comment-13031976 ] 

Hadoop QA commented on HADOOP-7237:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478860/HADOOP-7237-1.patch
  against trunk revision 1102012.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 4 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/433//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/433//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/433//console

This message is automatically generated.

> Refactor FsShell's touchz
> -------------------------
>
>                 Key: HADOOP-7237
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7237
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7237-1.patch, HADOOP-7237.patch
>
>
> Need to refactor touchz to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14703-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 19:46:29 2011
Return-Path: <common-issues-return-14703-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0B6614959
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 19:46:29 +0000 (UTC)
Received: (qmail 55919 invoked by uid 500); 11 May 2011 19:46:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55874 invoked by uid 500); 11 May 2011 19:46:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55674 invoked by uid 99); 11 May 2011 19:46:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 19:46:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 19:46:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AD99B824F4
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 19:45:47 +0000 (UTC)
Date: Wed, 11 May 2011 19:45:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1566404120.4413.1305143147707.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7279) Static web resources should not be
 duplicated between projects
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Static web resources should not be duplicated between projects
--------------------------------------------------------------

                 Key: HADOOP-7279
                 URL: https://issues.apache.org/jira/browse/HADOOP-7279
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.22.0
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
            Priority: Critical
             Fix For: 0.22.0


Right now, some of the common resources (eg hadoop.css) are duplicated between common and dependent projects, since HDFS and MR can't actually load these resources out of the common "static" webapp.

I'd like to rename common's "static" webapp to "common/static", and host it at a different URL in the dependent projects.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14704-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 19:51:29 2011
Return-Path: <common-issues-return-14704-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 991334614
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 19:51:29 +0000 (UTC)
Received: (qmail 65648 invoked by uid 500); 11 May 2011 19:51:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65616 invoked by uid 500); 11 May 2011 19:51:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65608 invoked by uid 99); 11 May 2011 19:51:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 19:51:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 19:51:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8AFA4826F0
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 19:50:47 +0000 (UTC)
Date: Wed, 11 May 2011 19:50:47 +0000 (UTC)
From: "Jason Rutherglen (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1825469730.4433.1305143447565.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6311) Add support for unix domain
 sockets to JNI libs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031997#comment-13031997 ] 

Jason Rutherglen commented on HADOOP-6311:
------------------------------------------

bq. It may be some work to backport onto 20-append

I think I'll try to backport it to 20-append in order to test and benchmark with the HBASE-3529.  Unless HBase is going to upgrade to a newer Hadoop version soon?

> Add support for unix domain sockets to JNI libs
> -----------------------------------------------
>
>                 Key: HADOOP-6311
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6311
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: native
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: 6311-trunk-inprogress.txt, hadoop-6311.txt
>
>
> For HDFS-347 we need to use unix domain sockets. This JIRA is to include a library in common which adds a o.a.h.net.unix package based on the code from Android (apache 2 license)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14705-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 20:07:27 2011
Return-Path: <common-issues-return-14705-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 890F44F4A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 20:07:27 +0000 (UTC)
Received: (qmail 92016 invoked by uid 500); 11 May 2011 20:07:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91980 invoked by uid 500); 11 May 2011 20:07:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91971 invoked by uid 99); 11 May 2011 20:07:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 20:07:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 20:07:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 84D5E82F3A
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 20:06:47 +0000 (UTC)
Date: Wed, 11 May 2011 20:06:47 +0000 (UTC)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <434615293.4543.1305144407540.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1894387424.4008.1305137809491.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7278) Automatic Hadoop cluster
 deployment for build validation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032021#comment-13032021 ] 

Konstantin Boudnik commented on HADOOP-7278:
--------------------------------------------

Here's some more thoughts and background information about the JIRA:
- there's a number of machines which aren't used by the build system right now and they can be used for on-cluster testing of Hadoop night builds or releases
- .deb packaging seems like a very nice idea because it provides certain consistency, versioning, etc. However, it might be a bit of over-kill for the simple task at hands (thanks Eli for pointing this out;)
- we'd rather skip .deb creation and will do installation of Hadoop right out of the tar-balls into a shared NFS location. Standard Hadoop scripts will be used for start/stop control of the daemons.
- for simplicity sake list of slaves and configuration files will be checked into the SVN (we might want use common or have a separate top-level directory designated for cluster management related issues)

> Automatic Hadoop cluster deployment for build validation
> --------------------------------------------------------
>
>                 Key: HADOOP-7278
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7278
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0, 0.23.0
>         Environment: Apache Jenkins
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>              Labels: newbie
>
> It'd be great to have a way of automatically deploying Hadoop cluster to a set of machine once all components are successfully built (in the form or tar or whatever). Deployed cluster then can be used to run a set of validation jobs to make sure that produced artifacts are viable. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14706-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 20:27:27 2011
Return-Path: <common-issues-return-14706-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AB0C64A60
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 20:27:27 +0000 (UTC)
Received: (qmail 43402 invoked by uid 500); 11 May 2011 20:27:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43372 invoked by uid 500); 11 May 2011 20:27:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43364 invoked by uid 99); 11 May 2011 20:27:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 20:27:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 20:27:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B03E2456EB
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 20:26:47 +0000 (UTC)
Date: Wed, 11 May 2011 20:26:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2099870110.4593.1305145607718.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1110985885.74832.1303423445828.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7237) Refactor FsShell's touchz
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7237:
-------------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I have committed this.  Thanks, Daryn!

Also thanks John for reviewing it.

> Refactor FsShell's touchz
> -------------------------
>
>                 Key: HADOOP-7237
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7237
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7237-1.patch, HADOOP-7237.patch
>
>
> Need to refactor touchz to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14707-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 20:27:29 2011
Return-Path: <common-issues-return-14707-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BB39A4A6F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 20:27:29 +0000 (UTC)
Received: (qmail 43665 invoked by uid 500); 11 May 2011 20:27:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43639 invoked by uid 500); 11 May 2011 20:27:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43631 invoked by uid 99); 11 May 2011 20:27:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 20:27:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 20:27:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 91A2E456E7
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 20:26:47 +0000 (UTC)
Date: Wed, 11 May 2011 20:26:47 +0000 (UTC)
From: "Ted Yu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1586129148.4589.1305145607593.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <683476535.2038.1300136250528.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7189) Add ability to enable 'debug'
 property in JAAS configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ted Yu reassigned HADOOP-7189:
------------------------------

    Assignee: Ted Yu

> Add ability to enable 'debug' property in JAAS configuration
> ------------------------------------------------------------
>
>                 Key: HADOOP-7189
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7189
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Ted Yu
>            Priority: Minor
>              Labels: newbie
>         Attachments: enable-UGI-debug-example.txt
>
>
> Occasionally users have run into weird "Unable to login" messages. Unfortunately, JAAS obscures the underlying exception message in many cases because it thinks leaking the exception might be insecure in itself. Enabling the "debug" option in the JAAS configuration gets it to dump the underlying issue and makes troubleshooting this kind of issue easier.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14708-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 20:36:27 2011
Return-Path: <common-issues-return-14708-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6001C44CD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 20:36:27 +0000 (UTC)
Received: (qmail 63407 invoked by uid 500); 11 May 2011 20:36:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63375 invoked by uid 500); 11 May 2011 20:36:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63367 invoked by uid 99); 11 May 2011 20:36:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 20:36:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 20:36:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 61D8845ABF
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 20:35:47 +0000 (UTC)
Date: Wed, 11 May 2011 20:35:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1297545513.4606.1305146147397.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1110985885.74832.1303423445828.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7237) Refactor FsShell's touchz
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032036#comment-13032036 ] 

Hudson commented on HADOOP-7237:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #590 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/590/])
    HADOOP-7237. Refactor the touchz commands to conform to new FsCommand class.  Contributed by Daryn Sharp


> Refactor FsShell's touchz
> -------------------------
>
>                 Key: HADOOP-7237
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7237
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7237-1.patch, HADOOP-7237.patch
>
>
> Need to refactor touchz to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14709-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 20:38:27 2011
Return-Path: <common-issues-return-14709-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AEEE345E4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 20:38:27 +0000 (UTC)
Received: (qmail 66684 invoked by uid 500); 11 May 2011 20:38:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66627 invoked by uid 500); 11 May 2011 20:38:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66615 invoked by uid 99); 11 May 2011 20:38:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 20:38:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 20:38:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B85E945C34
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 20:37:47 +0000 (UTC)
Date: Wed, 11 May 2011 20:37:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <534273819.4620.1305146267751.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <873708488.29176.1304715003217.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7267) Refactor FsShell's rm/rmr/expunge
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7267:
--------------------------------

    Attachment: HADOOP-7267-2.patch

upmerge to resolve conflict from removal of other commands

> Refactor FsShell's rm/rmr/expunge
> ---------------------------------
>
>                 Key: HADOOP-7267
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7267
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7267-2.patch, HADOOP-7267.patch
>
>
> Refactor to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14710-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 20:44:30 2011
Return-Path: <common-issues-return-14710-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 796664707
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 20:44:30 +0000 (UTC)
Received: (qmail 75426 invoked by uid 500); 11 May 2011 20:44:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75385 invoked by uid 500); 11 May 2011 20:44:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75376 invoked by uid 99); 11 May 2011 20:44:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 20:44:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 20:44:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5B34345F23
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 20:43:47 +0000 (UTC)
Date: Wed, 11 May 2011 20:43:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <44673766.4636.1305146627370.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <873708488.29176.1304715003217.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7267) Refactor FsShell's rm/rmr/expunge
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032044#comment-13032044 ] 

Hadoop QA commented on HADOOP-7267:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478879/HADOOP-7267-2.patch
  against trunk revision 1102068.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/434//console

This message is automatically generated.

> Refactor FsShell's rm/rmr/expunge
> ---------------------------------
>
>                 Key: HADOOP-7267
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7267
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7267-2.patch, HADOOP-7267.patch
>
>
> Refactor to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14711-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 20:50:27 2011
Return-Path: <common-issues-return-14711-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 740034D60
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 20:50:27 +0000 (UTC)
Received: (qmail 87682 invoked by uid 500); 11 May 2011 20:50:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87656 invoked by uid 500); 11 May 2011 20:50:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87648 invoked by uid 99); 11 May 2011 20:50:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 20:50:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 20:50:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6EAB982180
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 20:49:47 +0000 (UTC)
Date: Wed, 11 May 2011 20:49:47 +0000 (UTC)
From: "Gabriele Kahlout (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <672101949.4651.1305146987450.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6763) Remove verbose logging from the
 Groups class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032051#comment-13032051 ] 

Gabriele Kahlout commented on HADOOP-6763:
------------------------------------------

$ svn co -r 941662 http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21/ hadoop
$ ant -f hadoop/build.xml

I still get verbose logging at each command
11/05/11 22:45:39 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000


> Remove verbose logging from the Groups class
> --------------------------------------------
>
>                 Key: HADOOP-6763
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6763
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Owen O'Malley
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6598-BP20-Fix.patch, HADOOP-6598-BP20.patch, HADOOP-6598.patch, HADOOP-6763.patch
>
>
> {quote}
> 2010-02-25 08:30:52,269 INFO  security.Groups (Groups.java:<init>(60)) - Group m
> apping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout
> =300000
> ...
> 2010-02-25 08:30:57,872 INFO  security.Groups (Groups.java:getGroups(76)) - Retu
> rning cached groups for 'oom'
> {quote}
> should both be demoted to debug level.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14712-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 20:55:29 2011
Return-Path: <common-issues-return-14712-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7327D4F57
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 20:55:29 +0000 (UTC)
Received: (qmail 95028 invoked by uid 500); 11 May 2011 20:55:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94960 invoked by uid 500); 11 May 2011 20:55:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94949 invoked by uid 99); 11 May 2011 20:55:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 20:55:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 20:55:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5642682305
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 20:54:47 +0000 (UTC)
Date: Wed, 11 May 2011 20:54:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2069719117.4664.1305147287349.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <873708488.29176.1304715003217.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7267) Refactor FsShell's rm/rmr/expunge
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7267:
--------------------------------

    Attachment: HADOOP-7267-3.patch

upmerge again, third time is a charm...

> Refactor FsShell's rm/rmr/expunge
> ---------------------------------
>
>                 Key: HADOOP-7267
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7267
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7267-2.patch, HADOOP-7267-3.patch, HADOOP-7267.patch
>
>
> Refactor to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14713-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 21:03:30 2011
Return-Path: <common-issues-return-14713-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 48F594791
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 21:03:30 +0000 (UTC)
Received: (qmail 5329 invoked by uid 500); 11 May 2011 21:03:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5300 invoked by uid 500); 11 May 2011 21:03:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5291 invoked by uid 99); 11 May 2011 21:03:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:03:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:03:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5E087825EA
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 21:02:47 +0000 (UTC)
Date: Wed, 11 May 2011 21:02:47 +0000 (UTC)
From: "Ted Yu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <84597352.4689.1305147767381.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <683476535.2038.1300136250528.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7189) Add ability to enable 'debug'
 property in JAAS configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ted Yu updated HADOOP-7189:
---------------------------

    Attachment: HADOOP-7189.txt

This patch is based on Todd's patch.

Introduced an environment variable to control the addition of debug option.

> Add ability to enable 'debug' property in JAAS configuration
> ------------------------------------------------------------
>
>                 Key: HADOOP-7189
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7189
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Ted Yu
>            Priority: Minor
>              Labels: newbie
>         Attachments: HADOOP-7189.txt, enable-UGI-debug-example.txt
>
>
> Occasionally users have run into weird "Unable to login" messages. Unfortunately, JAAS obscures the underlying exception message in many cases because it thinks leaking the exception might be insecure in itself. Enabling the "debug" option in the JAAS configuration gets it to dump the underlying issue and makes troubleshooting this kind of issue easier.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14714-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 21:14:30 2011
Return-Path: <common-issues-return-14714-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 27C434C82
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 21:14:30 +0000 (UTC)
Received: (qmail 21747 invoked by uid 500); 11 May 2011 21:14:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21703 invoked by uid 500); 11 May 2011 21:14:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21695 invoked by uid 99); 11 May 2011 21:14:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:14:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:14:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 010A882B77
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 21:13:48 +0000 (UTC)
Date: Wed, 11 May 2011 21:13:47 +0000 (UTC)
From: "Patrick Hunt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <364647320.4742.1305148428000.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6846) Scripts for building Hadoop 0.22.0
 release
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Patrick Hunt updated HADOOP-6846:
---------------------------------

    Summary: Scripts for building Hadoop 0.22.0 release  (was: Scripts for building Hadoop 0.21.0 release)

> Scripts for building Hadoop 0.22.0 release
> ------------------------------------------
>
>                 Key: HADOOP-6846
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6846
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: build
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: release-scripts.tar.gz
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14715-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 21:14:30 2011
Return-Path: <common-issues-return-14715-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5489A4C92
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 21:14:30 +0000 (UTC)
Received: (qmail 21982 invoked by uid 500); 11 May 2011 21:14:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21908 invoked by uid 500); 11 May 2011 21:14:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21814 invoked by uid 99); 11 May 2011 21:14:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:14:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:14:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0F33582B79
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 21:13:48 +0000 (UTC)
Date: Wed, 11 May 2011 21:13:48 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1986708733.4744.1305148428059.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <873708488.29176.1304715003217.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7267) Refactor FsShell's rm/rmr/expunge
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032065#comment-13032065 ] 

Hadoop QA commented on HADOOP-7267:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478880/HADOOP-7267-3.patch
  against trunk revision 1102068.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs.  The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/435//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/435//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/435//console

This message is automatically generated.

> Refactor FsShell's rm/rmr/expunge
> ---------------------------------
>
>                 Key: HADOOP-7267
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7267
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7267-2.patch, HADOOP-7267-3.patch, HADOOP-7267.patch
>
>
> Refactor to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14716-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 21:16:27 2011
Return-Path: <common-issues-return-14716-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3D9D6410E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 21:16:27 +0000 (UTC)
Received: (qmail 25561 invoked by uid 500); 11 May 2011 21:16:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25526 invoked by uid 500); 11 May 2011 21:16:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25513 invoked by uid 99); 11 May 2011 21:16:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:16:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:16:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B22C282C90
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 21:15:47 +0000 (UTC)
Date: Wed, 11 May 2011 21:15:47 +0000 (UTC)
From: "Patrick Hunt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <719436442.4755.1305148547726.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6846) Scripts for building Hadoop 0.22.0
 release
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Patrick Hunt updated HADOOP-6846:
---------------------------------

    Attachment: HADOOP-6846.patch

I pulled the tar-munge file out of release-scripts.tar.gz and attached it as a patch.

Commit to hadoop/nightly, ensure that the file is executable.

> Scripts for building Hadoop 0.22.0 release
> ------------------------------------------
>
>                 Key: HADOOP-6846
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6846
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: build
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6846.patch, release-scripts.tar.gz
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14717-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 21:16:27 2011
Return-Path: <common-issues-return-14717-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C31384125
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 21:16:27 +0000 (UTC)
Received: (qmail 25809 invoked by uid 500); 11 May 2011 21:16:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25775 invoked by uid 500); 11 May 2011 21:16:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25766 invoked by uid 99); 11 May 2011 21:16:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:16:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:16:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DD0B182C95
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 21:15:47 +0000 (UTC)
Date: Wed, 11 May 2011 21:15:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1498901305.4760.1305148547902.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <683476535.2038.1300136250528.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7189) Add ability to enable 'debug'
 property in JAAS configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032067#comment-13032067 ] 

Todd Lipcon commented on HADOOP-7189:
-------------------------------------

Looks good, though I would suggest using HADOOP_JAAS_DEBUG as the environment variable. Our convention is that env vars are ALL_CAPITAL_LETTERS, reserving dot.separated.words for conf options.

> Add ability to enable 'debug' property in JAAS configuration
> ------------------------------------------------------------
>
>                 Key: HADOOP-7189
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7189
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Ted Yu
>            Priority: Minor
>              Labels: newbie
>         Attachments: HADOOP-7189.txt, enable-UGI-debug-example.txt
>
>
> Occasionally users have run into weird "Unable to login" messages. Unfortunately, JAAS obscures the underlying exception message in many cases because it thinks leaking the exception might be insecure in itself. Enabling the "debug" option in the JAAS configuration gets it to dump the underlying issue and makes troubleshooting this kind of issue easier.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14718-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 21:16:28 2011
Return-Path: <common-issues-return-14718-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2BAEC413E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 21:16:28 +0000 (UTC)
Received: (qmail 26208 invoked by uid 500); 11 May 2011 21:16:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26175 invoked by uid 500); 11 May 2011 21:16:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26167 invoked by uid 99); 11 May 2011 21:16:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:16:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:16:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4A0A882CA2
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 21:15:48 +0000 (UTC)
Date: Wed, 11 May 2011 21:15:48 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1915575342.4773.1305148548300.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <873708488.29176.1304715003217.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7267) Refactor FsShell's rm/rmr/expunge
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7267:
-------------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I have committed this.  Thanks, Daryn!

Also thanks John for reviewing it.

> Refactor FsShell's rm/rmr/expunge
> ---------------------------------
>
>                 Key: HADOOP-7267
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7267
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7267-2.patch, HADOOP-7267-3.patch, HADOOP-7267.patch
>
>
> Refactor to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14719-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 21:18:30 2011
Return-Path: <common-issues-return-14719-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 157674B64
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 21:18:30 +0000 (UTC)
Received: (qmail 31154 invoked by uid 500); 11 May 2011 21:18:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31111 invoked by uid 500); 11 May 2011 21:18:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31103 invoked by uid 99); 11 May 2011 21:18:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:18:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:18:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E7C3082D97
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 21:17:47 +0000 (UTC)
Date: Wed, 11 May 2011 21:17:47 +0000 (UTC)
From: "Andrew Whang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2143050801.4779.1305148667946.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <683476535.2038.1300136250528.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7189) Add ability to enable 'debug'
 property in JAAS configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Whang updated HADOOP-7189:
---------------------------------

    Attachment: HADOOP-7189.patch

identical to HADOOP-7189.txt, but with test case

> Add ability to enable 'debug' property in JAAS configuration
> ------------------------------------------------------------
>
>                 Key: HADOOP-7189
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7189
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Ted Yu
>            Priority: Minor
>              Labels: newbie
>         Attachments: HADOOP-7189.patch, HADOOP-7189.txt, enable-UGI-debug-example.txt
>
>
> Occasionally users have run into weird "Unable to login" messages. Unfortunately, JAAS obscures the underlying exception message in many cases because it thinks leaking the exception might be insecure in itself. Enabling the "debug" option in the JAAS configuration gets it to dump the underlying issue and makes troubleshooting this kind of issue easier.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14720-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 21:20:30 2011
Return-Path: <common-issues-return-14720-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DD2D843E7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 21:20:29 +0000 (UTC)
Received: (qmail 32741 invoked by uid 500); 11 May 2011 21:20:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32716 invoked by uid 500); 11 May 2011 21:20:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32708 invoked by uid 99); 11 May 2011 21:20:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:20:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:20:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B4C7E82E9F
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 21:19:47 +0000 (UTC)
Date: Wed, 11 May 2011 21:19:47 +0000 (UTC)
From: "Patrick Hunt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1514758357.4790.1305148787736.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6846) Scripts for building Hadoop 0.22.0
 release
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Patrick Hunt updated HADOOP-6846:
---------------------------------

    Affects Version/s: 0.22.0
               Status: Patch Available  (was: Open)

Tom White and I (phunt) worked on this as part of the hackathon. It allows a hadoop-0.22.0 tar.gz file to be generated by the jenkins build, which can be seen here:

https://builds.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-22-Build/9/

(hadoop-0.22.0-SNAPSHOT.tar.gz includes core/hdfs/mapred tar.gz's)

After committing this change (hadoop/nightly) edit the build configuration on jenkins to remove this section from "execute shell":

---
# for the moment pull my version
wget --no-check-certificate http://github.com/phunt/hadoop-nightly/raw/master/tar-munge
chmod a+x tar-munge
mv tar-munge ${WORKSPACE}/nightly/tar-munge
---

> Scripts for building Hadoop 0.22.0 release
> ------------------------------------------
>
>                 Key: HADOOP-6846
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6846
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6846.patch, release-scripts.tar.gz
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14721-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 21:22:27 2011
Return-Path: <common-issues-return-14721-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AFC0D44CB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 21:22:27 +0000 (UTC)
Received: (qmail 38682 invoked by uid 500); 11 May 2011 21:22:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38631 invoked by uid 500); 11 May 2011 21:22:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38451 invoked by uid 99); 11 May 2011 21:22:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:22:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:22:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7FBE082F9B
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 21:21:47 +0000 (UTC)
Date: Wed, 11 May 2011 21:21:47 +0000 (UTC)
From: "Patrick Hunt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <368969917.4804.1305148907519.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6846) Scripts for building Hadoop 0.22.0
 release
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032078#comment-13032078 ] 

Patrick Hunt commented on HADOOP-6846:
--------------------------------------

Tom, I just noticed that you did not check "grant for inclusion" on your release-scripts.tar.gz attachment, can you please comment on this jira granting such (given I pulled the patch from that archive). Thanks.

> Scripts for building Hadoop 0.22.0 release
> ------------------------------------------
>
>                 Key: HADOOP-6846
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6846
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6846.patch, release-scripts.tar.gz
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14722-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 21:24:28 2011
Return-Path: <common-issues-return-14722-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 23E63450C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 21:24:28 +0000 (UTC)
Received: (qmail 40179 invoked by uid 500); 11 May 2011 21:24:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40141 invoked by uid 500); 11 May 2011 21:24:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40132 invoked by uid 99); 11 May 2011 21:24:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:24:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:24:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 26CF6820A2
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 21:23:48 +0000 (UTC)
Date: Wed, 11 May 2011 21:23:48 +0000 (UTC)
From: "Ted Yu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <239994899.4823.1305149028155.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <683476535.2038.1300136250528.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7189) Add ability to enable 'debug'
 property in JAAS configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ted Yu updated HADOOP-7189:
---------------------------

    Attachment:     (was: HADOOP-7189.txt)

> Add ability to enable 'debug' property in JAAS configuration
> ------------------------------------------------------------
>
>                 Key: HADOOP-7189
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7189
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Ted Yu
>            Priority: Minor
>              Labels: newbie
>         Attachments: HADOOP-7189.patch, enable-UGI-debug-example.txt
>
>
> Occasionally users have run into weird "Unable to login" messages. Unfortunately, JAAS obscures the underlying exception message in many cases because it thinks leaking the exception might be insecure in itself. Enabling the "debug" option in the JAAS configuration gets it to dump the underlying issue and makes troubleshooting this kind of issue easier.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14723-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 21:26:28 2011
Return-Path: <common-issues-return-14723-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D9FA54584
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 21:26:27 +0000 (UTC)
Received: (qmail 43831 invoked by uid 500); 11 May 2011 21:26:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43793 invoked by uid 500); 11 May 2011 21:26:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43785 invoked by uid 99); 11 May 2011 21:26:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:26:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:26:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E50458219A
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 21:25:47 +0000 (UTC)
Date: Wed, 11 May 2011 21:25:47 +0000 (UTC)
From: "Ted Yu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <182579425.4844.1305149147934.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <683476535.2038.1300136250528.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7189) Add ability to enable 'debug'
 property in JAAS configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ted Yu updated HADOOP-7189:
---------------------------

    Attachment: HADOOP-7189.txt

Changed the name of environment variable according to Todd's suggestion.

> Add ability to enable 'debug' property in JAAS configuration
> ------------------------------------------------------------
>
>                 Key: HADOOP-7189
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7189
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Ted Yu
>            Priority: Minor
>              Labels: newbie
>         Attachments: HADOOP-7189.patch, HADOOP-7189.txt, enable-UGI-debug-example.txt
>
>
> Occasionally users have run into weird "Unable to login" messages. Unfortunately, JAAS obscures the underlying exception message in many cases because it thinks leaking the exception might be insecure in itself. Enabling the "debug" option in the JAAS configuration gets it to dump the underlying issue and makes troubleshooting this kind of issue easier.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14724-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 21:34:28 2011
Return-Path: <common-issues-return-14724-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0A4E34D19
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 21:34:28 +0000 (UTC)
Received: (qmail 60591 invoked by uid 500); 11 May 2011 21:34:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60517 invoked by uid 500); 11 May 2011 21:34:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60284 invoked by uid 99); 11 May 2011 21:34:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:34:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:34:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 70B6582448
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 21:33:47 +0000 (UTC)
Date: Wed, 11 May 2011 21:33:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <413781321.4868.1305149627457.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6846) Scripts for building Hadoop 0.22.0
 release
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032090#comment-13032090 ] 

Hadoop QA commented on HADOOP-6846:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478884/HADOOP-6846.patch
  against trunk revision 1102093.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs.  The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/436//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/436//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/436//console

This message is automatically generated.

> Scripts for building Hadoop 0.22.0 release
> ------------------------------------------
>
>                 Key: HADOOP-6846
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6846
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6846.patch, release-scripts.tar.gz
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14725-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 21:36:27 2011
Return-Path: <common-issues-return-14725-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6D9E24D6D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 21:36:27 +0000 (UTC)
Received: (qmail 62286 invoked by uid 500); 11 May 2011 21:36:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62246 invoked by uid 500); 11 May 2011 21:36:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62238 invoked by uid 99); 11 May 2011 21:36:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:36:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:36:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7A695824F6
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 21:35:47 +0000 (UTC)
Date: Wed, 11 May 2011 21:35:47 +0000 (UTC)
From: "Jason Rutherglen (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1660614550.4882.1305149747498.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6311) Add support for unix domain sockets
 to JNI libs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Rutherglen updated HADOOP-6311:
-------------------------------------

    Affects Version/s: 0.20.0
               Status: Patch Available  (was: Open)

Here's a patch for Hadoop 0.20 append.  There's one test failure as noted below.

ant run-test-core

{quote}
Test org.apache.hadoop.metrics2.impl.TestSinkQueue FAILED
{quote}

> Add support for unix domain sockets to JNI libs
> -----------------------------------------------
>
>                 Key: HADOOP-6311
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6311
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: native
>    Affects Versions: 0.20.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: 6311-trunk-inprogress.txt, hadoop-6311.txt
>
>
> For HDFS-347 we need to use unix domain sockets. This JIRA is to include a library in common which adds a o.a.h.net.unix package based on the code from Android (apache 2 license)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14726-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 21:36:29 2011
Return-Path: <common-issues-return-14726-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 815974D7E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 21:36:29 +0000 (UTC)
Received: (qmail 62578 invoked by uid 500); 11 May 2011 21:36:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62522 invoked by uid 500); 11 May 2011 21:36:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62514 invoked by uid 99); 11 May 2011 21:36:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:36:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:36:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6CFCA824F4
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 21:35:47 +0000 (UTC)
Date: Wed, 11 May 2011 21:35:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <440406233.4880.1305149747443.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <873708488.29176.1304715003217.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7267) Refactor FsShell's rm/rmr/expunge
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032091#comment-13032091 ] 

Hudson commented on HADOOP-7267:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #591 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/591/])
    HADOOP-7267. Refactor the rm/rmr/expunge commands to conform to new FsCommand class.  Contributed by Daryn Sharp


> Refactor FsShell's rm/rmr/expunge
> ---------------------------------
>
>                 Key: HADOOP-7267
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7267
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7267-2.patch, HADOOP-7267-3.patch, HADOOP-7267.patch
>
>
> Refactor to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14727-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 21:40:29 2011
Return-Path: <common-issues-return-14727-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8C94D4E2C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 21:40:29 +0000 (UTC)
Received: (qmail 67938 invoked by uid 500); 11 May 2011 21:40:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67879 invoked by uid 500); 11 May 2011 21:40:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67871 invoked by uid 99); 11 May 2011 21:40:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:40:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:40:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 61FBF825E6
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 21:39:47 +0000 (UTC)
Date: Wed, 11 May 2011 21:39:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <177085104.4897.1305149987397.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1566404120.4413.1305143147707.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7279) Static web resources should not be
 duplicated between projects
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032093#comment-13032093 ] 

Luke Lu commented on HADOOP-7279:
---------------------------------

changing /static would break many downstream dependencies that use common HttpServer. Why can't hdfs and mr use /static for common resources and /hdfs/static and mr/static for their respective resources? Of course separate servlets (DefaultServlet would suffice) need to be registered with HttpServer to serve the latter.

> Static web resources should not be duplicated between projects
> --------------------------------------------------------------
>
>                 Key: HADOOP-7279
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7279
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>
> Right now, some of the common resources (eg hadoop.css) are duplicated between common and dependent projects, since HDFS and MR can't actually load these resources out of the common "static" webapp.
> I'd like to rename common's "static" webapp to "common/static", and host it at a different URL in the dependent projects.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14728-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 21:44:28 2011
Return-Path: <common-issues-return-14728-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0DAA84E9D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 21:44:28 +0000 (UTC)
Received: (qmail 72181 invoked by uid 500); 11 May 2011 21:44:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72146 invoked by uid 500); 11 May 2011 21:44:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72132 invoked by uid 99); 11 May 2011 21:44:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:44:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:44:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E9BF182727
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 21:43:47 +0000 (UTC)
Date: Wed, 11 May 2011 21:43:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1823874475.4923.1305150227954.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6311) Add support for unix domain
 sockets to JNI libs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032096#comment-13032096 ] 

Hadoop QA commented on HADOOP-6311:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12476942/6311-trunk-inprogress.txt
  against trunk revision 1102093.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 2 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/437//console

This message is automatically generated.

> Add support for unix domain sockets to JNI libs
> -----------------------------------------------
>
>                 Key: HADOOP-6311
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6311
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: native
>    Affects Versions: 0.20.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: 6311-trunk-inprogress.txt, hadoop-6311.txt
>
>
> For HDFS-347 we need to use unix domain sockets. This JIRA is to include a library in common which adds a o.a.h.net.unix package based on the code from Android (apache 2 license)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14729-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 21:44:31 2011
Return-Path: <common-issues-return-14729-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 784BB4EB7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 21:44:31 +0000 (UTC)
Received: (qmail 72468 invoked by uid 500); 11 May 2011 21:44:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72440 invoked by uid 500); 11 May 2011 21:44:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72432 invoked by uid 99); 11 May 2011 21:44:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:44:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 21:44:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0156F82729
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 21:43:48 +0000 (UTC)
Date: Wed, 11 May 2011 21:43:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <627663417.4925.1305150228002.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1566404120.4413.1305143147707.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7279) Static web resources should not be
 duplicated between projects
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032097#comment-13032097 ] 

Todd Lipcon commented on HADOOP-7279:
-------------------------------------

What are the downstream dependencies that depend on it? /static is only the hadoop.css file - do we have a lot of stuff depending on that being at the particular URL?

> Static web resources should not be duplicated between projects
> --------------------------------------------------------------
>
>                 Key: HADOOP-7279
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7279
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>
> Right now, some of the common resources (eg hadoop.css) are duplicated between common and dependent projects, since HDFS and MR can't actually load these resources out of the common "static" webapp.
> I'd like to rename common's "static" webapp to "common/static", and host it at a different URL in the dependent projects.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14730-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 22:00:28 2011
Return-Path: <common-issues-return-14730-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F30E149C9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 22:00:27 +0000 (UTC)
Received: (qmail 367 invoked by uid 500); 11 May 2011 22:00:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 328 invoked by uid 500); 11 May 2011 22:00:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 311 invoked by uid 99); 11 May 2011 22:00:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 22:00:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 22:00:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A28A182CDD
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 21:59:47 +0000 (UTC)
Date: Wed, 11 May 2011 21:59:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <352311681.4974.1305151187661.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1566404120.4413.1305143147707.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7279) Static web resources should not be
 duplicated between projects
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032105#comment-13032105 ] 

Luke Lu commented on HADOOP-7279:
---------------------------------

I'm not talking about hadoop.css per se, rather, the default HttpServer serves /static from resource base webapps/static. I _think_ you can achieve what you want without changing the user facing URL (/static).

> Static web resources should not be duplicated between projects
> --------------------------------------------------------------
>
>                 Key: HADOOP-7279
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7279
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>
> Right now, some of the common resources (eg hadoop.css) are duplicated between common and dependent projects, since HDFS and MR can't actually load these resources out of the common "static" webapp.
> I'd like to rename common's "static" webapp to "common/static", and host it at a different URL in the dependent projects.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14731-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 22:14:27 2011
Return-Path: <common-issues-return-14731-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8C4414100
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 22:14:27 +0000 (UTC)
Received: (qmail 23586 invoked by uid 500); 11 May 2011 22:14:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23548 invoked by uid 500); 11 May 2011 22:14:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23540 invoked by uid 99); 11 May 2011 22:14:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 22:14:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 22:14:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C15C982289
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 22:13:47 +0000 (UTC)
Date: Wed, 11 May 2011 22:13:47 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <499651441.5038.1305152027788.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6846) Scripts for building Hadoop 0.22.0
 release
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032117#comment-13032117 ] 

Tom White commented on HADOOP-6846:
-----------------------------------

I grant the patch for inclusion.

> Scripts for building Hadoop 0.22.0 release
> ------------------------------------------
>
>                 Key: HADOOP-6846
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6846
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6846.patch, release-scripts.tar.gz
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14732-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 22:14:31 2011
Return-Path: <common-issues-return-14732-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 37C3D411E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 22:14:31 +0000 (UTC)
Received: (qmail 23964 invoked by uid 500); 11 May 2011 22:14:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23897 invoked by uid 500); 11 May 2011 22:14:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23880 invoked by uid 99); 11 May 2011 22:14:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 22:14:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 22:14:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 445E882298
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 22:13:48 +0000 (UTC)
Date: Wed, 11 May 2011 22:13:48 +0000 (UTC)
From: "Jason Rutherglen (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2122171513.5053.1305152028276.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6311) Add support for unix domain
 sockets to JNI libs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032120#comment-13032120 ] 

Jason Rutherglen commented on HADOOP-6311:
------------------------------------------

Sorry, misspoke, the patch option wasn't given in Jira so there's no file, however the patch was mistakenly for Hadoop trunk.  The port to 0.20-append is a bit more complex.  

> Add support for unix domain sockets to JNI libs
> -----------------------------------------------
>
>                 Key: HADOOP-6311
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6311
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: native
>    Affects Versions: 0.20.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: 6311-trunk-inprogress.txt, hadoop-6311.txt
>
>
> For HDFS-347 we need to use unix domain sockets. This JIRA is to include a library in common which adds a o.a.h.net.unix package based on the code from Android (apache 2 license)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14733-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 22:36:29 2011
Return-Path: <common-issues-return-14733-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E723D430D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 22:36:29 +0000 (UTC)
Received: (qmail 59419 invoked by uid 500); 11 May 2011 22:36:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59355 invoked by uid 500); 11 May 2011 22:36:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59340 invoked by uid 99); 11 May 2011 22:36:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 22:36:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 22:36:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A68E582B4D
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 22:35:47 +0000 (UTC)
Date: Wed, 11 May 2011 22:35:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1388074373.5162.1305153347678.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6846) Scripts for building Hadoop 0.22.0
 release
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032140#comment-13032140 ] 

Hudson commented on HADOOP-6846:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #592 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/592/])
    HADOOP-6846. Scripts for building Hadoop 0.22.0 release.


> Scripts for building Hadoop 0.22.0 release
> ------------------------------------------
>
>                 Key: HADOOP-6846
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6846
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6846.patch, release-scripts.tar.gz
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14734-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 22:47:27 2011
Return-Path: <common-issues-return-14734-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8064546B0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 22:47:27 +0000 (UTC)
Received: (qmail 70103 invoked by uid 500); 11 May 2011 22:47:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69995 invoked by uid 500); 11 May 2011 22:47:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69987 invoked by uid 99); 11 May 2011 22:47:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 22:47:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 22:47:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A6E2782DEF
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 22:46:47 +0000 (UTC)
Date: Wed, 11 May 2011 22:46:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1456162067.5192.1305154007680.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <683476535.2038.1300136250528.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7189) Add ability to enable 'debug'
 property in JAAS configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7189:
--------------------------------

    Affects Version/s:     (was: 0.23.0)
                       0.22.0
         Hadoop Flags: [Reviewed]
               Status: Patch Available  (was: Open)

> Add ability to enable 'debug' property in JAAS configuration
> ------------------------------------------------------------
>
>                 Key: HADOOP-7189
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7189
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Ted Yu
>            Priority: Minor
>              Labels: newbie
>         Attachments: HADOOP-7189.patch, HADOOP-7189.txt, enable-UGI-debug-example.txt
>
>
> Occasionally users have run into weird "Unable to login" messages. Unfortunately, JAAS obscures the underlying exception message in many cases because it thinks leaking the exception might be insecure in itself. Enabling the "debug" option in the JAAS configuration gets it to dump the underlying issue and makes troubleshooting this kind of issue easier.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14735-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 22:47:29 2011
Return-Path: <common-issues-return-14735-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 87F6F46F5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 22:47:29 +0000 (UTC)
Received: (qmail 70291 invoked by uid 500); 11 May 2011 22:47:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70249 invoked by uid 500); 11 May 2011 22:47:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70241 invoked by uid 99); 11 May 2011 22:47:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 22:47:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 22:47:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8277C82DEA
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 22:46:47 +0000 (UTC)
Date: Wed, 11 May 2011 22:46:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1578887688.5187.1305154007531.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <683476535.2038.1300136250528.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7189) Add ability to enable 'debug'
 property in JAAS configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032144#comment-13032144 ] 

Todd Lipcon commented on HADOOP-7189:
-------------------------------------

+1 for Ted's patch. Unfortunately I wasn't able to use the test that Andrew contributed, since this feature happens in a static initialization block and depends on an env var. I manually tested by running the UGI class from the command line with and without the env var set to "true".

> Add ability to enable 'debug' property in JAAS configuration
> ------------------------------------------------------------
>
>                 Key: HADOOP-7189
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7189
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Ted Yu
>            Priority: Minor
>              Labels: newbie
>         Attachments: HADOOP-7189.patch, HADOOP-7189.txt, enable-UGI-debug-example.txt
>
>
> Occasionally users have run into weird "Unable to login" messages. Unfortunately, JAAS obscures the underlying exception message in many cases because it thinks leaking the exception might be insecure in itself. Enabling the "debug" option in the JAAS configuration gets it to dump the underlying issue and makes troubleshooting this kind of issue easier.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14736-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 22:51:27 2011
Return-Path: <common-issues-return-14736-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 457994B40
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 22:51:27 +0000 (UTC)
Received: (qmail 74716 invoked by uid 500); 11 May 2011 22:51:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74678 invoked by uid 500); 11 May 2011 22:51:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74670 invoked by uid 99); 11 May 2011 22:51:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 22:51:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 22:51:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6ACA882F70
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 22:50:47 +0000 (UTC)
Date: Wed, 11 May 2011 22:50:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <835669386.5234.1305154247434.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <683476535.2038.1300136250528.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7189) Add ability to enable 'debug'
 property in JAAS configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7189:
--------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

Committed to 22 and trunk. Thanks, Ted!

> Add ability to enable 'debug' property in JAAS configuration
> ------------------------------------------------------------
>
>                 Key: HADOOP-7189
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7189
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Ted Yu
>            Priority: Minor
>              Labels: newbie
>         Attachments: HADOOP-7189.patch, HADOOP-7189.txt, enable-UGI-debug-example.txt
>
>
> Occasionally users have run into weird "Unable to login" messages. Unfortunately, JAAS obscures the underlying exception message in many cases because it thinks leaking the exception might be insecure in itself. Enabling the "debug" option in the JAAS configuration gets it to dump the underlying issue and makes troubleshooting this kind of issue easier.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14737-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 22:53:27 2011
Return-Path: <common-issues-return-14737-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B51084B8B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 22:53:27 +0000 (UTC)
Received: (qmail 75920 invoked by uid 500); 11 May 2011 22:53:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75895 invoked by uid 500); 11 May 2011 22:53:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75887 invoked by uid 99); 11 May 2011 22:53:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 22:53:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 22:53:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BF89A8200B
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 22:52:47 +0000 (UTC)
Date: Wed, 11 May 2011 22:52:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1667126489.5247.1305154367781.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6846) Scripts for building Hadoop 0.22.0
 release
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032150#comment-13032150 ] 

Hudson commented on HADOOP-6846:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk-Commit #664 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/664/])
    HADOOP-6846. Scripts for building Hadoop 0.22.0 release.


> Scripts for building Hadoop 0.22.0 release
> ------------------------------------------
>
>                 Key: HADOOP-6846
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6846
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6846.patch, release-scripts.tar.gz
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14738-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 23:06:28 2011
Return-Path: <common-issues-return-14738-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 172DA440A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 23:06:28 +0000 (UTC)
Received: (qmail 87493 invoked by uid 500); 11 May 2011 23:06:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87465 invoked by uid 500); 11 May 2011 23:06:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87457 invoked by uid 99); 11 May 2011 23:06:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 23:06:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 23:06:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 077B182502
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 23:05:48 +0000 (UTC)
Date: Wed, 11 May 2011 23:05:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <766882019.5304.1305155148027.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <683476535.2038.1300136250528.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7189) Add ability to enable 'debug'
 property in JAAS configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032164#comment-13032164 ] 

Hudson commented on HADOOP-7189:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #593 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/593/])
    HADOOP-7189. Add ability to enable debug property in JAAS configuration. Contributed by Ted Yu.


> Add ability to enable 'debug' property in JAAS configuration
> ------------------------------------------------------------
>
>                 Key: HADOOP-7189
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7189
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Ted Yu
>            Priority: Minor
>              Labels: newbie
>         Attachments: HADOOP-7189.patch, HADOOP-7189.txt, enable-UGI-debug-example.txt
>
>
> Occasionally users have run into weird "Unable to login" messages. Unfortunately, JAAS obscures the underlying exception message in many cases because it thinks leaking the exception might be insecure in itself. Enabling the "debug" option in the JAAS configuration gets it to dump the underlying issue and makes troubleshooting this kind of issue easier.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14739-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 23:20:29 2011
Return-Path: <common-issues-return-14739-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9C7DA4CFB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 23:20:29 +0000 (UTC)
Received: (qmail 3376 invoked by uid 500); 11 May 2011 23:20:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3282 invoked by uid 500); 11 May 2011 23:20:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3065 invoked by uid 99); 11 May 2011 23:20:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 23:20:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 23:20:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 57C0D8289D
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 23:19:47 +0000 (UTC)
Date: Wed, 11 May 2011 23:19:47 +0000 (UTC)
From: "Forrest Vines (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2105947079.5333.1305155987356.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7280) ChainReducer uses MAPPER_BY_VALUE
 instead of REDUCER_BY_VALUE
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

ChainReducer uses MAPPER_BY_VALUE instead of REDUCER_BY_VALUE
-------------------------------------------------------------

                 Key: HADOOP-7280
                 URL: https://issues.apache.org/jira/browse/HADOOP-7280
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Forrest Vines
            Priority: Minor


On line 293 of o.a.h.mapred.lib.Chain in setReducer(...):

reducerConf.setBoolean(MAPPER_BY_VALUE, byValue);

this should be REDUCER_BY_VALUE.

http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/com.cloudera.hadoop/hadoop-core/0.20.2-737/org/apache/hadoop/mapred/lib/Chain.java#293

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14740-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 23:38:27 2011
Return-Path: <common-issues-return-14740-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CD6EF4773
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 23:38:27 +0000 (UTC)
Received: (qmail 22641 invoked by uid 500); 11 May 2011 23:38:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22589 invoked by uid 500); 11 May 2011 23:38:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22581 invoked by uid 99); 11 May 2011 23:38:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 23:38:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 23:38:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CF2AF82DEA
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 23:37:47 +0000 (UTC)
Date: Wed, 11 May 2011 23:37:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1304102508.5391.1305157067845.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1035897564.5388.1305157067762.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7281) update site to include the
 0.20.203.0 release
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7281:
----------------------------------

      Component/s: documentation
    Fix Version/s: site

> update site to include the 0.20.203.0 release
> ---------------------------------------------
>
>                 Key: HADOOP-7281
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7281
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: site
>
>         Attachments: 203.diff
>
>
> Update site to include the 0.20.203.0 release.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14741-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 23:38:28 2011
Return-Path: <common-issues-return-14741-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 04D41477D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 23:38:28 +0000 (UTC)
Received: (qmail 22868 invoked by uid 500); 11 May 2011 23:38:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22809 invoked by uid 500); 11 May 2011 23:38:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22631 invoked by uid 99); 11 May 2011 23:38:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 23:38:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 23:38:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DC14582DEC
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 23:37:47 +0000 (UTC)
Date: Wed, 11 May 2011 23:37:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <889650419.5393.1305157067898.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1035897564.5388.1305157067762.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7281) update site to include the
 0.20.203.0 release
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7281:
----------------------------------

    Attachment: 203.diff

> update site to include the 0.20.203.0 release
> ---------------------------------------------
>
>                 Key: HADOOP-7281
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7281
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: site
>
>         Attachments: 203.diff
>
>
> Update site to include the 0.20.203.0 release.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14742-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 23:38:31 2011
Return-Path: <common-issues-return-14742-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7AACD47B2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 23:38:31 +0000 (UTC)
Received: (qmail 23498 invoked by uid 500); 11 May 2011 23:38:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23452 invoked by uid 500); 11 May 2011 23:38:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23312 invoked by uid 99); 11 May 2011 23:38:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 23:38:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 23:38:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BAD6E82DE7
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 23:37:47 +0000 (UTC)
Date: Wed, 11 May 2011 23:37:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1035897564.5388.1305157067762.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7281) update site to include the
 0.20.203.0 release
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

update site to include the 0.20.203.0 release
---------------------------------------------

                 Key: HADOOP-7281
                 URL: https://issues.apache.org/jira/browse/HADOOP-7281
             Project: Hadoop Common
          Issue Type: Task
            Reporter: Owen O'Malley
            Assignee: Owen O'Malley
         Attachments: 203.diff

Update site to include the 0.20.203.0 release.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14743-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 23:44:29 2011
Return-Path: <common-issues-return-14743-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DE4EC4884
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 23:44:29 +0000 (UTC)
Received: (qmail 31814 invoked by uid 500); 11 May 2011 23:44:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31747 invoked by uid 500); 11 May 2011 23:44:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31728 invoked by uid 99); 11 May 2011 23:44:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 23:44:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 23:44:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 873A182FD5
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 23:43:47 +0000 (UTC)
Date: Wed, 11 May 2011 23:43:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <316140930.5422.1305157427550.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1035897564.5388.1305157067762.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7281) update site to include the
 0.20.203.0 release
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032182#comment-13032182 ] 

Luke Lu commented on HADOOP-7281:
---------------------------------

+1, lgtm (none generated stuff :)

> update site to include the 0.20.203.0 release
> ---------------------------------------------
>
>                 Key: HADOOP-7281
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7281
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: site
>
>         Attachments: 203.diff
>
>
> Update site to include the 0.20.203.0 release.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14744-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 23:46:29 2011
Return-Path: <common-issues-return-14744-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0D1114371
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 23:46:29 +0000 (UTC)
Received: (qmail 33211 invoked by uid 500); 11 May 2011 23:46:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33171 invoked by uid 500); 11 May 2011 23:46:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33158 invoked by uid 99); 11 May 2011 23:46:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 23:46:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 23:46:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A61B9820D2
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 23:45:47 +0000 (UTC)
Date: Wed, 11 May 2011 23:45:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <37537479.5429.1305157547677.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1035897564.5388.1305157067762.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7281) update site to include the
 0.20.203.0 release
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley resolved HADOOP-7281.
-----------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]

> update site to include the 0.20.203.0 release
> ---------------------------------------------
>
>                 Key: HADOOP-7281
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7281
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: site
>
>         Attachments: 203.diff
>
>
> Update site to include the 0.20.203.0 release.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14745-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 23:46:29 2011
Return-Path: <common-issues-return-14745-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4946343AE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 23:46:29 +0000 (UTC)
Received: (qmail 33628 invoked by uid 500); 11 May 2011 23:46:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33577 invoked by uid 500); 11 May 2011 23:46:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33285 invoked by uid 99); 11 May 2011 23:46:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 23:46:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 23:46:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BD366820D5
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 23:45:47 +0000 (UTC)
Date: Wed, 11 May 2011 23:45:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1963195821.5432.1305157547771.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1035897564.5388.1305157067762.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Closed] (HADOOP-7281) update site to include the 0.20.203.0
 release
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley closed HADOOP-7281.
---------------------------------


> update site to include the 0.20.203.0 release
> ---------------------------------------------
>
>                 Key: HADOOP-7281
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7281
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: site
>
>         Attachments: 203.diff
>
>
> Update site to include the 0.20.203.0 release.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14746-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 11 23:58:29 2011
Return-Path: <common-issues-return-14746-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BD668401C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 11 May 2011 23:58:29 +0000 (UTC)
Received: (qmail 42715 invoked by uid 500); 11 May 2011 23:58:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42686 invoked by uid 500); 11 May 2011 23:58:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42609 invoked by uid 99); 11 May 2011 23:58:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 23:58:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2011 23:58:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7803D823A1
	for <common-issues@hadoop.apache.org>; Wed, 11 May 2011 23:57:47 +0000 (UTC)
Date: Wed, 11 May 2011 23:57:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1107563711.5451.1305158267488.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2105947079.5333.1305155987356.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7280) ChainReducer uses MAPPER_BY_VALUE
 instead of REDUCER_BY_VALUE
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032188#comment-13032188 ] 

Todd Lipcon commented on HADOOP-7280:
-------------------------------------

Hi Forrest. Would you mind attaching an svn patch to fix this issue? Also, I will move this bug to the MAPREDUCE project, since it is MR-specific.

> ChainReducer uses MAPPER_BY_VALUE instead of REDUCER_BY_VALUE
> -------------------------------------------------------------
>
>                 Key: HADOOP-7280
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7280
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Forrest Vines
>            Priority: Minor
>
> On line 293 of o.a.h.mapred.lib.Chain in setReducer(...):
> reducerConf.setBoolean(MAPPER_BY_VALUE, byValue);
> this should be REDUCER_BY_VALUE.
> http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/com.cloudera.hadoop/hadoop-core/0.20.2-737/org/apache/hadoop/mapred/lib/Chain.java#293

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14747-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 00:08:28 2011
Return-Path: <common-issues-return-14747-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5626547B2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 00:08:28 +0000 (UTC)
Received: (qmail 54006 invoked by uid 500); 12 May 2011 00:08:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53971 invoked by uid 500); 12 May 2011 00:08:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53963 invoked by uid 99); 12 May 2011 00:08:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 00:08:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 00:08:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4A01982655
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 00:07:48 +0000 (UTC)
Date: Thu, 12 May 2011 00:07:48 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1904881673.5500.1305158868299.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6763) Remove verbose logging from the
 Groups class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032193#comment-13032193 ] 

Tom White commented on HADOOP-6763:
-----------------------------------

Yes, this should have fix version 0.22.

> Remove verbose logging from the Groups class
> --------------------------------------------
>
>                 Key: HADOOP-6763
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6763
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Owen O'Malley
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6598-BP20-Fix.patch, HADOOP-6598-BP20.patch, HADOOP-6598.patch, HADOOP-6763.patch
>
>
> {quote}
> 2010-02-25 08:30:52,269 INFO  security.Groups (Groups.java:<init>(60)) - Group m
> apping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout
> =300000
> ...
> 2010-02-25 08:30:57,872 INFO  security.Groups (Groups.java:getGroups(76)) - Retu
> rning cached groups for 'oom'
> {quote}
> should both be demoted to debug level.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14748-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 00:11:35 2011
Return-Path: <common-issues-return-14748-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DB86B481B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 00:11:35 +0000 (UTC)
Received: (qmail 55352 invoked by uid 500); 12 May 2011 00:11:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55319 invoked by uid 500); 12 May 2011 00:11:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55307 invoked by uid 99); 12 May 2011 00:11:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 00:11:35 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 00:11:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6C6F382728
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 00:10:48 +0000 (UTC)
Date: Thu, 12 May 2011 00:10:48 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1032573540.5534.1305159048440.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6763) Remove verbose logging from the
 Groups class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6763:
------------------------------

    Fix Version/s: 0.22.0

> Remove verbose logging from the Groups class
> --------------------------------------------
>
>                 Key: HADOOP-6763
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6763
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Owen O'Malley
>            Assignee: Boris Shkolnik
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6598-BP20-Fix.patch, HADOOP-6598-BP20.patch, HADOOP-6598.patch, HADOOP-6763.patch
>
>
> {quote}
> 2010-02-25 08:30:52,269 INFO  security.Groups (Groups.java:<init>(60)) - Group m
> apping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout
> =300000
> ...
> 2010-02-25 08:30:57,872 INFO  security.Groups (Groups.java:getGroups(76)) - Retu
> rning cached groups for 'oom'
> {quote}
> should both be demoted to debug level.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14749-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 01:24:27 2011
Return-Path: <common-issues-return-14749-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 639C14FA0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 01:24:27 +0000 (UTC)
Received: (qmail 34526 invoked by uid 500); 12 May 2011 01:24:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34478 invoked by uid 500); 12 May 2011 01:24:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34470 invoked by uid 99); 12 May 2011 01:24:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 01:24:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 01:24:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5ECFE44DCD
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 01:23:47 +0000 (UTC)
Date: Thu, 12 May 2011 01:23:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <258076528.5755.1305163427384.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6846) Scripts for building Hadoop 0.22.0
 release
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032225#comment-13032225 ] 

Hudson commented on HADOOP-6846:
--------------------------------

Integrated in Hadoop-Mapreduce-22-branch #47 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/47/])
    HADOOP-6846. Scripts for building Hadoop 0.22.0 release.


> Scripts for building Hadoop 0.22.0 release
> ------------------------------------------
>
>                 Key: HADOOP-6846
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6846
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6846.patch, release-scripts.tar.gz
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14750-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 04:34:35 2011
Return-Path: <common-issues-return-14750-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 66AF15E56
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 04:34:35 +0000 (UTC)
Received: (qmail 39791 invoked by uid 500); 12 May 2011 04:34:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39527 invoked by uid 500); 12 May 2011 04:34:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39507 invoked by uid 99); 12 May 2011 04:34:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 04:34:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 04:34:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8515482348
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 04:33:47 +0000 (UTC)
Date: Thu, 12 May 2011 04:33:47 +0000 (UTC)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1698510852.5928.1305174827541.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1894387424.4008.1305137809491.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7278) Automatic Hadoop cluster
 deployment for build validation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032249#comment-13032249 ] 

Nigel Daley commented on HADOOP-7278:
-------------------------------------

To be clear, a single Jenkins build slave will be running the coordination of all this, but the cluster that is stood up will be running on non-slave machines, right?

> Automatic Hadoop cluster deployment for build validation
> --------------------------------------------------------
>
>                 Key: HADOOP-7278
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7278
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0, 0.23.0
>         Environment: Apache Jenkins
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>              Labels: newbie
>
> It'd be great to have a way of automatically deploying Hadoop cluster to a set of machine once all components are successfully built (in the form or tar or whatever). Deployed cluster then can be used to run a set of validation jobs to make sure that produced artifacts are viable. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14751-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 10:57:32 2011
Return-Path: <common-issues-return-14751-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4B8955C75
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 10:57:32 +0000 (UTC)
Received: (qmail 21301 invoked by uid 500); 12 May 2011 10:57:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21128 invoked by uid 500); 12 May 2011 10:57:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21117 invoked by uid 99); 12 May 2011 10:57:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 10:57:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 10:57:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BC9BE828DE
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 10:56:47 +0000 (UTC)
Date: Thu, 12 May 2011 10:56:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1274968774.6476.1305197807769.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6846) Scripts for building Hadoop 0.22.0
 release
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032355#comment-13032355 ] 

Hudson commented on HADOOP-6846:
--------------------------------

Integrated in ZooKeeper-trunk #1180 (See [https://builds.apache.org/hudson/job/ZooKeeper-trunk/1180/])
    HADOOP-6846. Scripts for building Hadoop 0.22.0 release.


> Scripts for building Hadoop 0.22.0 release
> ------------------------------------------
>
>                 Key: HADOOP-6846
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6846
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6846.patch, release-scripts.tar.gz
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14752-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 13:32:31 2011
Return-Path: <common-issues-return-14752-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 72D505445
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 13:32:31 +0000 (UTC)
Received: (qmail 4194 invoked by uid 500); 12 May 2011 13:32:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4161 invoked by uid 500); 12 May 2011 13:32:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4153 invoked by uid 99); 12 May 2011 13:32:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 13:32:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 13:32:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C991A871D5
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 13:31:47 +0000 (UTC)
Date: Thu, 12 May 2011 13:31:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <887707551.6705.1305207107820.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <683476535.2038.1300136250528.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7189) Add ability to enable 'debug'
 property in JAAS configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032404#comment-13032404 ] 

Hudson commented on HADOOP-7189:
--------------------------------

Integrated in Hadoop-Common-trunk #686 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/686/])
    HADOOP-7189. Add ability to enable debug property in JAAS configuration. Contributed by Ted Yu.


> Add ability to enable 'debug' property in JAAS configuration
> ------------------------------------------------------------
>
>                 Key: HADOOP-7189
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7189
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Ted Yu
>            Priority: Minor
>              Labels: newbie
>         Attachments: HADOOP-7189.patch, HADOOP-7189.txt, enable-UGI-debug-example.txt
>
>
> Occasionally users have run into weird "Unable to login" messages. Unfortunately, JAAS obscures the underlying exception message in many cases because it thinks leaking the exception might be insecure in itself. Enabling the "debug" option in the JAAS configuration gets it to dump the underlying issue and makes troubleshooting this kind of issue easier.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14753-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 13:32:31 2011
Return-Path: <common-issues-return-14753-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 987FD5458
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 13:32:31 +0000 (UTC)
Received: (qmail 4303 invoked by uid 500); 12 May 2011 13:32:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4201 invoked by uid 500); 12 May 2011 13:32:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4162 invoked by uid 99); 12 May 2011 13:32:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 13:32:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 13:32:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 02D90871DB
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 13:31:48 +0000 (UTC)
Date: Thu, 12 May 2011 13:31:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1222420961.6711.1305207108008.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1648991564.3455.1305130187546.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7275) Refactor FsShell's stat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032406#comment-13032406 ] 

Hudson commented on HADOOP-7275:
--------------------------------

Integrated in Hadoop-Common-trunk #686 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/686/])
    HADOOP-7275. Refactor the stat commands to conform to new FsCommand class.  Contributed by Daryn Sharp


> Refactor FsShell's stat
> -----------------------
>
>                 Key: HADOOP-7275
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7275
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7275.patch
>
>
> Refactor to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14754-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 13:32:32 2011
Return-Path: <common-issues-return-14754-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 26F6C546B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 13:32:32 +0000 (UTC)
Received: (qmail 4678 invoked by uid 500); 12 May 2011 13:32:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4642 invoked by uid 500); 12 May 2011 13:32:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4634 invoked by uid 99); 12 May 2011 13:32:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 13:32:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 13:32:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1A145871DD
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 13:31:48 +0000 (UTC)
Date: Thu, 12 May 2011 13:31:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <837054086.6713.1305207108103.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6846) Scripts for building Hadoop 0.22.0
 release
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032407#comment-13032407 ] 

Hudson commented on HADOOP-6846:
--------------------------------

Integrated in Hadoop-Common-trunk #686 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/686/])
    HADOOP-6846. Scripts for building Hadoop 0.22.0 release.


> Scripts for building Hadoop 0.22.0 release
> ------------------------------------------
>
>                 Key: HADOOP-6846
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6846
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6846.patch, release-scripts.tar.gz
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14755-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 13:32:33 2011
Return-Path: <common-issues-return-14755-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 90EC754B2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 13:32:33 +0000 (UTC)
Received: (qmail 5350 invoked by uid 500); 12 May 2011 13:32:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5313 invoked by uid 500); 12 May 2011 13:32:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5256 invoked by uid 99); 12 May 2011 13:32:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 13:32:33 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 13:32:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9B047871D0
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 13:31:47 +0000 (UTC)
Date: Thu, 12 May 2011 13:31:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <488384700.6700.1305207107631.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1110985885.74832.1303423445828.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7237) Refactor FsShell's touchz
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032403#comment-13032403 ] 

Hudson commented on HADOOP-7237:
--------------------------------

Integrated in Hadoop-Common-trunk #686 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/686/])
    HADOOP-7237. Refactor the touchz commands to conform to new FsCommand class.  Contributed by Daryn Sharp


> Refactor FsShell's touchz
> -------------------------
>
>                 Key: HADOOP-7237
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7237
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7237-1.patch, HADOOP-7237.patch
>
>
> Need to refactor touchz to conform to new FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14756-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 13:32:33 2011
Return-Path: <common-issues-return-14756-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D37A454C8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 13:32:33 +0000 (UTC)
Received: (qmail 5558 invoked by uid 500); 12 May 2011 13:32:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5512 invoked by uid 500); 12 May 2011 13:32:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5445 invoked by uid 99); 12 May 2011 13:32:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 13:32:33 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 13:32:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E41F1871D8
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 13:31:47 +0000 (UTC)
Date: Thu, 12 May 2011 13:31:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <25631010.6708.1305207107931.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <873708488.29176.1304715003217.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7267) Refactor FsShell's rm/rmr/expunge
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032405#comment-13032405 ] 

Hudson commented on HADOOP-7267:
--------------------------------

Integrated in Hadoop-Common-trunk #686 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/686/])
    HADOOP-7267. Refactor the rm/rmr/expunge commands to conform to new FsCommand class.  Contributed by Daryn Sharp


> Refactor FsShell's rm/rmr/expunge
> ---------------------------------
>
>                 Key: HADOOP-7267
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7267
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7267-2.patch, HADOOP-7267-3.patch, HADOOP-7267.patch
>
>
> Refactor to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14757-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 14:34:31 2011
Return-Path: <common-issues-return-14757-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C9A0B5CF7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 14:34:31 +0000 (UTC)
Received: (qmail 8379 invoked by uid 500); 12 May 2011 14:34:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8329 invoked by uid 500); 12 May 2011 14:34:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8265 invoked by uid 99); 12 May 2011 14:34:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 14:34:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 14:34:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 348ED87B66
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 14:33:48 +0000 (UTC)
Date: Thu, 12 May 2011 14:33:48 +0000 (UTC)
From: "John George (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <903601160.6842.1305210828211.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7282) getRemoteIp could return null in
 cases where the call is ongoing but the ip went away.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

getRemoteIp could return null in cases where the call is ongoing but the ip went away.
--------------------------------------------------------------------------------------

                 Key: HADOOP-7282
                 URL: https://issues.apache.org/jira/browse/HADOOP-7282
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: John George
            Assignee: John George
             Fix For: 0.20.205.0, 0.23.0


getRemoteIp gets the ip from socket instead of the stored ip in Connection object. Thus calls to this function could return null when a client disconnected, but the rpc call is still ongoing...

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14758-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 15:03:33 2011
Return-Path: <common-issues-return-14758-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 03885574F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 15:03:33 +0000 (UTC)
Received: (qmail 46024 invoked by uid 500); 12 May 2011 15:03:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45994 invoked by uid 500); 12 May 2011 15:03:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45982 invoked by uid 99); 12 May 2011 15:03:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 15:03:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 15:03:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 622D287835
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 15:02:47 +0000 (UTC)
Date: Thu, 12 May 2011 15:02:47 +0000 (UTC)
From: "John George (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1972822828.6924.1305212567398.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <903601160.6842.1305210828211.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7282) getRemoteIp could return null in
 cases where the call is ongoing but the ip went away.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

John George updated HADOOP-7282:
--------------------------------

    Attachment: HADOOP-7282.patch
                diffs.txt

> getRemoteIp could return null in cases where the call is ongoing but the ip went away.
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7282
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7282
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: John George
>            Assignee: John George
>             Fix For: 0.20.205.0, 0.23.0
>
>         Attachments: HADOOP-7282.patch, diffs.txt
>
>
> getRemoteIp gets the ip from socket instead of the stored ip in Connection object. Thus calls to this function could return null when a client disconnected, but the rpc call is still ongoing...

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14759-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 15:03:33 2011
Return-Path: <common-issues-return-14759-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 555755764
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 15:03:33 +0000 (UTC)
Received: (qmail 46393 invoked by uid 500); 12 May 2011 15:03:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46345 invoked by uid 500); 12 May 2011 15:03:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46301 invoked by uid 99); 12 May 2011 15:03:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 15:03:33 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 15:03:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7DE9A87839
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 15:02:47 +0000 (UTC)
Date: Thu, 12 May 2011 15:02:47 +0000 (UTC)
From: "John George (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <109984612.6928.1305212567512.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <903601160.6842.1305210828211.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7282) getRemoteIp could return null in
 cases where the call is ongoing but the ip went away.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032446#comment-13032446 ] 

John George commented on HADOOP-7282:
-------------------------------------

Please ignore the "diffs.txt" file - that was accidentally uploaded. Sorry for the confusion .

> getRemoteIp could return null in cases where the call is ongoing but the ip went away.
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7282
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7282
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: John George
>            Assignee: John George
>             Fix For: 0.20.205.0, 0.23.0
>
>         Attachments: HADOOP-7282.patch, diffs.txt
>
>
> getRemoteIp gets the ip from socket instead of the stored ip in Connection object. Thus calls to this function could return null when a client disconnected, but the rpc call is still ongoing...

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14760-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 15:10:31 2011
Return-Path: <common-issues-return-14760-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8F7F85D10
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 15:10:31 +0000 (UTC)
Received: (qmail 58411 invoked by uid 500); 12 May 2011 15:10:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58386 invoked by uid 500); 12 May 2011 15:10:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58378 invoked by uid 99); 12 May 2011 15:10:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 15:10:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 15:10:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 19DD487B4B
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 15:09:48 +0000 (UTC)
Date: Thu, 12 May 2011 15:09:48 +0000 (UTC)
From: "John George (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1314872738.6950.1305212988100.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <903601160.6842.1305210828211.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7282) getRemoteIp could return null in
 cases where the call is ongoing but the ip went away.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

John George updated HADOOP-7282:
--------------------------------

    Fix Version/s:     (was: 0.20.205.0)
           Status: Patch Available  (was: Open)

> getRemoteIp could return null in cases where the call is ongoing but the ip went away.
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7282
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7282
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: John George
>            Assignee: John George
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7282.patch, diffs.txt
>
>
> getRemoteIp gets the ip from socket instead of the stored ip in Connection object. Thus calls to this function could return null when a client disconnected, but the rpc call is still ongoing...

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14761-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 16:24:33 2011
Return-Path: <common-issues-return-14761-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 35D7D5274
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 16:24:33 +0000 (UTC)
Received: (qmail 82428 invoked by uid 500); 12 May 2011 16:24:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82363 invoked by uid 500); 12 May 2011 16:24:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82355 invoked by uid 99); 12 May 2011 16:24:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 16:24:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 16:24:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7695887888
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 16:23:47 +0000 (UTC)
Date: Thu, 12 May 2011 16:23:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1557823194.7157.1305217427482.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <683476535.2038.1300136250528.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7189) Add ability to enable 'debug'
 property in JAAS configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032486#comment-13032486 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7189:
------------------------------------------------

Forgot to run tests and set fix versions?

> Add ability to enable 'debug' property in JAAS configuration
> ------------------------------------------------------------
>
>                 Key: HADOOP-7189
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7189
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Ted Yu
>            Priority: Minor
>              Labels: newbie
>         Attachments: HADOOP-7189.patch, HADOOP-7189.txt, enable-UGI-debug-example.txt
>
>
> Occasionally users have run into weird "Unable to login" messages. Unfortunately, JAAS obscures the underlying exception message in many cases because it thinks leaking the exception might be insecure in itself. Enabling the "debug" option in the JAAS configuration gets it to dump the underlying issue and makes troubleshooting this kind of issue easier.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14762-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 16:58:27 2011
Return-Path: <common-issues-return-14762-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 64F6253A3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 16:58:27 +0000 (UTC)
Received: (qmail 43466 invoked by uid 500); 12 May 2011 16:58:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43433 invoked by uid 500); 12 May 2011 16:58:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43425 invoked by uid 99); 12 May 2011 16:58:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 16:58:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 16:58:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4B94F876FA
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 16:57:47 +0000 (UTC)
Date: Thu, 12 May 2011 16:57:47 +0000 (UTC)
From: "Nicholas Telford (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <785924055.7266.1305219467290.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7269) S3 Native should allow customizable
 file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicholas Telford updated HADOOP-7269:
-------------------------------------

    Attachment: HADOOP-7269-S3-metadata-001.diff

Attached is a patch that adds basic support for key/value metadata when creating files with a FileSystem.

FileSystems that don't support metadata will simply discard any that is provided.

Keys are dependant on the FileSystem being used, in the case of s3native, they're used as HTTP headers and should match the headers available here: http://docs.amazonwebservices.com/AmazonS3/latest/API/

I'm not sure how I feel about this approach. On the one hand, it's fairly abstract and flexible. On the other, it makes the assertion that metadata is always a Map<String, String> which may prove fairly restrictive. Still, I can't think of a better way to do it.

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Priority: Minor
>         Attachments: HADOOP-7269-S3-metadata-001.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14763-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 17:00:32 2011
Return-Path: <common-issues-return-14763-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2911356CC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 17:00:32 +0000 (UTC)
Received: (qmail 48850 invoked by uid 500); 12 May 2011 17:00:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48829 invoked by uid 500); 12 May 2011 17:00:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48821 invoked by uid 99); 12 May 2011 17:00:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 17:00:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 17:00:31 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 17FC287836
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 16:59:52 +0000 (UTC)
Date: Thu, 12 May 2011 16:59:52 +0000 (UTC)
From: "Nicholas Telford (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <154765593.7269.1305219592094.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7269) S3 Native should allow customizable
 file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicholas Telford updated HADOOP-7269:
-------------------------------------

        Assignee: Nicholas Telford
    Release Note: Added support for arbitrary file metadata when creating a new file on a FileSystem that supports metadata.
          Status: Patch Available  (was: Open)

Patch attached that implements the provision of arbitrary FileSystem metadata when creating a file.

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: HADOOP-7269-S3-metadata-001.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14764-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 17:02:28 2011
Return-Path: <common-issues-return-14764-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8116F5FCA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 17:02:28 +0000 (UTC)
Received: (qmail 53416 invoked by uid 500); 12 May 2011 17:02:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53383 invoked by uid 500); 12 May 2011 17:02:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53375 invoked by uid 99); 12 May 2011 17:02:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 17:02:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 17:02:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 70C8587920
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 17:01:48 +0000 (UTC)
Date: Thu, 12 May 2011 17:01:48 +0000 (UTC)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1761724315.7278.1305219708458.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <161449362.3682.1305134627390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7277) Add Eclipse launch tasks for the
 0.20-security branch
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032509#comment-13032509 ] 

Chris Douglas commented on HADOOP-7277:
---------------------------------------

+1

Ran the eclipse target, tried a unit test. I committed this to the 20-security branch. Thanks, Jeff!

> Add Eclipse launch tasks for the 0.20-security branch
> -----------------------------------------------------
>
>                 Key: HADOOP-7277
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7277
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.204.0
>            Reporter: Jeffrey Naisbitt
>            Assignee: Jeffrey Naisbitt
>            Priority: Minor
>             Fix For: 0.20.205.0
>
>         Attachments: HADOOP-7277-0.20s.patch
>
>
> This is to add the eclipse launchers from HADOOP-5911 to the 0.20 security branch.
> Eclipse has a notion of "run configuration", which encapsulates what's needed to run or debug an application. I use this quite a bit to start various Hadoop daemons in debug mode, with breakpoints set, to inspect state and what not.
> This is simply configuration, so no tests are provided. After running "ant eclipse" and refreshing your project, you should see entries in the Run Configurations and Debug Configurations for launching the various hadoop daemons from within eclipse. There's a template for testing a specific test, and also templates to run all the tests, the job tracker, and a task tracker. It's likely that some parameters need to be further tweaked to have the same behavior as "ant test", but for most tests, this works.
> This also requires a small change to build.xml for the eclipse classpath.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14765-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 17:02:31 2011
Return-Path: <common-issues-return-14765-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3CEDD500C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 17:02:31 +0000 (UTC)
Received: (qmail 53683 invoked by uid 500); 12 May 2011 17:02:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53649 invoked by uid 500); 12 May 2011 17:02:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53641 invoked by uid 99); 12 May 2011 17:02:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 17:02:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 17:02:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 843A787923
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 17:01:48 +0000 (UTC)
Date: Thu, 12 May 2011 17:01:48 +0000 (UTC)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1405311599.7281.1305219708538.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <161449362.3682.1305134627390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7277) Add Eclipse launch tasks for the
 0.20-security branch
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-7277:
----------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

> Add Eclipse launch tasks for the 0.20-security branch
> -----------------------------------------------------
>
>                 Key: HADOOP-7277
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7277
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.204.0
>            Reporter: Jeffrey Naisbitt
>            Assignee: Jeffrey Naisbitt
>            Priority: Minor
>             Fix For: 0.20.205.0
>
>         Attachments: HADOOP-7277-0.20s.patch
>
>
> This is to add the eclipse launchers from HADOOP-5911 to the 0.20 security branch.
> Eclipse has a notion of "run configuration", which encapsulates what's needed to run or debug an application. I use this quite a bit to start various Hadoop daemons in debug mode, with breakpoints set, to inspect state and what not.
> This is simply configuration, so no tests are provided. After running "ant eclipse" and refreshing your project, you should see entries in the Run Configurations and Debug Configurations for launching the various hadoop daemons from within eclipse. There's a template for testing a specific test, and also templates to run all the tests, the job tracker, and a task tracker. It's likely that some parameters need to be further tweaked to have the same behavior as "ant test", but for most tests, this works.
> This also requires a small change to build.xml for the eclipse classpath.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14766-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 18:22:31 2011
Return-Path: <common-issues-return-14766-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 69A7F5A41
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 18:22:31 +0000 (UTC)
Received: (qmail 95756 invoked by uid 500); 12 May 2011 18:22:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95690 invoked by uid 500); 12 May 2011 18:22:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95588 invoked by uid 99); 12 May 2011 18:22:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 18:22:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 18:22:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B246B87A78
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 18:21:48 +0000 (UTC)
Date: Thu, 12 May 2011 18:21:48 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2142927778.7596.1305224508727.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6728) Overhaul metrics framework
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-6728:
----------------------------

    Description: 
Per discussions with Arun, Chris, Hong and Rajiv, et al, we concluded that the current metrics framework needs an overhaul to:

* Allow multiple plugins for different monitoring systems simultaneously (see also: HADOOP-6508).
* Refresh metrics plugin config without server restart.
** Including filtering of metrics per plugin.
* Support metrics schema for plugins.

The jira will be resolved when core hadoop components (hdfs, mapreduce) are updated to use the new framework . Updates to external components that use the existing metrics framework will be tracked by different issues.

[The current design wiki|http://wiki.apache.org/hadoop/HADOOP-6728-MetricsV2].

  was:
Per discussions with Arun, Chris, Hong and Rajiv, et al, we concluded that the current metrics framework needs an overhaul to:

* Allow multiple plugins for different monitoring systems simultaneously (see also: HADOOP-6508).
* Refresh metrics plugin config without server restart.
** Including filtering of metrics per plugin.
* Support metrics schema for plugins.

The jira will be resolved when core hadoop components (hdfs, mapreduce) are updated to use the new framework . Updates to external components that use the existing metrics framework will be tracked by different issues.


> Overhaul metrics framework
> --------------------------
>
>                 Key: HADOOP-6728
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6728
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.2, 0.21.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-6728-y20.104.patch, metrics1-flow.png, metrics2-builder-r2.png, metrics2-builder.png, metrics2-flow.png, metrics2-mutable-r2.png, metrics2-mutable.png, metrics2-record-r2.png, metrics2-record.png, metrics2-uml-r2.png, metrics2-uml.png
>
>
> Per discussions with Arun, Chris, Hong and Rajiv, et al, we concluded that the current metrics framework needs an overhaul to:
> * Allow multiple plugins for different monitoring systems simultaneously (see also: HADOOP-6508).
> * Refresh metrics plugin config without server restart.
> ** Including filtering of metrics per plugin.
> * Support metrics schema for plugins.
> The jira will be resolved when core hadoop components (hdfs, mapreduce) are updated to use the new framework . Updates to external components that use the existing metrics framework will be tracked by different issues.
> [The current design wiki|http://wiki.apache.org/hadoop/HADOOP-6728-MetricsV2].

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14767-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 18:40:28 2011
Return-Path: <common-issues-return-14767-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2FA975137
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 18:40:28 +0000 (UTC)
Received: (qmail 22442 invoked by uid 500); 12 May 2011 18:40:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22404 invoked by uid 500); 12 May 2011 18:40:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22351 invoked by uid 99); 12 May 2011 18:40:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 18:40:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 18:40:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C33D78714D
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 18:39:47 +0000 (UTC)
Date: Thu, 12 May 2011 18:39:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1203870342.7638.1305225587796.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-6921) metrics2: metrics plugins
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu resolved HADOOP-6921.
-----------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Release Note: Metrics names are standardized to CapitalizedCamelCase. See release note of HADOOP-6918 and HADOOP-6920.
     Hadoop Flags: [Incompatible change, Reviewed]

> metrics2: metrics plugins
> -------------------------
>
>                 Key: HADOOP-6921
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6921
>             Project: Hadoop Common
>          Issue Type: Sub-task
>    Affects Versions: 0.20.2
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>
> This jira tracks the porting of builtin metrics sink plugins (file, ganglia) for the new metrics framework.
> Whether or not ganglia 3.0/3.1 plugins will be ported depends on the outcome of the discussion (proposed in the parent issue: HADOOP-6728) on backward compatibility (at some cost/limitations of course.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14768-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 18:42:29 2011
Return-Path: <common-issues-return-14768-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 97BAB51B6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 18:42:29 +0000 (UTC)
Received: (qmail 25757 invoked by uid 500); 12 May 2011 18:42:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25726 invoked by uid 500); 12 May 2011 18:42:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25718 invoked by uid 99); 12 May 2011 18:42:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 18:42:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 18:42:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 64A4787249
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 18:41:47 +0000 (UTC)
Date: Thu, 12 May 2011 18:41:47 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <456709527.7650.1305225707408.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7283) Include 32-bit and 64-bit native
 libraries in Jenkins tarball builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Include 32-bit and 64-bit native libraries in Jenkins tarball builds
--------------------------------------------------------------------

                 Key: HADOOP-7283
                 URL: https://issues.apache.org/jira/browse/HADOOP-7283
             Project: Hadoop Common
          Issue Type: Task
          Components: build
            Reporter: Tom White
            Priority: Blocker
             Fix For: 0.22.0


The job at https://builds.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-22-Build/ is building tarballs, but they do not currently include both 32-bit and 64-bit native libraries. We should update/duplicate hadoop-nighly/hudsonBuildHadoopRelease.sh to support post-split builds.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14769-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 18:44:30 2011
Return-Path: <common-issues-return-14769-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8821D5251
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 18:44:30 +0000 (UTC)
Received: (qmail 32919 invoked by uid 500); 12 May 2011 18:44:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32872 invoked by uid 500); 12 May 2011 18:44:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32864 invoked by uid 99); 12 May 2011 18:44:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 18:44:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 18:44:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 634048731A
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 18:43:47 +0000 (UTC)
Date: Thu, 12 May 2011 18:43:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1421291246.7659.1305225827403.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <15581131.20741291107311011.JavaMail.jira@thor>
Subject: [jira] [Updated] (HADOOP-7055) Update of commons logging libraries
 causes EventCounter to count logging events incorrectly
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-7055:
----------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

> Update of commons logging libraries causes EventCounter to count logging events incorrectly
> -------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7055
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7055
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.21.0, 0.21.1, 0.22.0, 0.23.0
>            Reporter: Jingguo Yao
>            Assignee: Jingguo Yao
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7055.patch
>
>   Original Estimate: 0.5h
>  Remaining Estimate: 0.5h
>
> Hadoop 0.20.2 uses commons logging 1.0.4. EventCounter works correctly with this version of commons logging. Hadoop 0.21.0 uses commons logging 1.1.1 which causes EventCounter to count logging events incorrectly. I have verified it with Hadoop 0.21.0. After start-up of hadoop, I checked jvmmetrics.log after several minutes. In every metrics record, "logError=0, logFatal=0, logInfo=3, logWarn=0" was shown. The following text is an example.
> jvm.metrics: hostName=jingguolin, processName=DataNode, sessionId=, gcCount=3, gcTimeMillis=31, logError=0, logFatal=0, logInfo=3, logWarn=0, maxMemoryM=888.9375, memHeapCommittedM=38.0625, memHeapUsedM=3.6539612, memNonHeapCommittedM=18.25, memNonHeapUsedM=11.335686, threadsBlocked=0, threadsNew=0, threadsRunnable=8, threadsTerminated=0, threadsTimedWaiting=6, threadsWaiting=6
> Then I stopped hadoop and replaced commons logging 1.1.1 with 1.0.4. After the re-start of hadoop, a lot of logging events showed up in jvmmetrics.log.
> I have checked the source code of Log4JLogger for both 1.0.4 (http://svn.apache.org/viewvc/commons/proper/logging/tags/LOGGING_1_0_4/src/java/org/apache/commons/logging/impl/Log4JLogger.java?view=markup) and 1.1.1 (http://svn.apache.org/viewvc/commons/proper/logging/tags/commons-logging-1.1.1/src/java/org/apache/commons/logging/impl/Log4JLogger.java?view=markup). For 1.0.4, Level instances such as Level.INFO are used to construct LoggingEvent. But for 1.1.1, Priority instances such as Priority.INFO are used to construct LoggingEvent. So 1.1.1 version's event.getLevel() always returns Priority instances. EventCounter append method's "==" check always fails between a Level instance and a Priority instance. For "logInfo=3" metrics records produced by commons logging 1.1.1., I think that these 3 logging events are produced by log4j code directly instead of through commons logging API. The following code is EventCounter's append method.
>     public void append(LoggingEvent event) {
>         Level level = event.getLevel();
>         if (level == Level.INFO) {
>             counts.incr(INFO);
>         }
>         else if (level == Level.WARN) {
>             counts.incr(WARN);
>         }
>         else if (level == Level.ERROR) {
>             counts.incr(ERROR);
>         }
>         else if (level == Level.FATAL) {
>             counts.incr(FATAL);
>         }
>     }

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14770-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 18:46:27 2011
Return-Path: <common-issues-return-14770-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AC8AD5C4C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 18:46:27 +0000 (UTC)
Received: (qmail 34644 invoked by uid 500); 12 May 2011 18:46:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34598 invoked by uid 500); 12 May 2011 18:46:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34589 invoked by uid 99); 12 May 2011 18:46:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 18:46:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 18:46:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 901B787415
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 18:45:47 +0000 (UTC)
Date: Thu, 12 May 2011 18:45:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <609740634.7669.1305225947587.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-6887) Need a separate metrics per garbage
 collector
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu resolved HADOOP-6887.
-----------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0

> Need a separate metrics per garbage collector
> ---------------------------------------------
>
>                 Key: HADOOP-6887
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6887
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: metrics
>            Reporter: Bharath Mundlapudi
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>
> In addition to current GC metrics which are the sum of all the collectors, Need separate metrics for monitoring young generation and old generation collections per collector w.r.t collection count and collection time. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14771-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 18:54:30 2011
Return-Path: <common-issues-return-14771-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 24A315834
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 18:54:30 +0000 (UTC)
Received: (qmail 46070 invoked by uid 500); 12 May 2011 18:54:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46044 invoked by uid 500); 12 May 2011 18:54:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46036 invoked by uid 99); 12 May 2011 18:54:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 18:54:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 18:54:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 50BE38776D
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 18:53:47 +0000 (UTC)
Date: Thu, 12 May 2011 18:53:47 +0000 (UTC)
From: "Trevor Robinson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <255728245.7700.1305226427327.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <532896131.3523.1305130907394.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7276) Hadoop native builds fail on ARM due
 to -m32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Trevor Robinson updated HADOOP-7276:
------------------------------------

    Status: Patch Available  (was: Open)

> Hadoop native builds fail on ARM due to -m32
> --------------------------------------------
>
>                 Key: HADOOP-7276
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7276
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.21.0
>         Environment: $ gcc -v
> Using built-in specs.
> COLLECT_GCC=gcc
> COLLECT_LTO_WRAPPER=/usr/lib/arm-linux-gnueabi/gcc/arm-linux-gnueabi/4.5.2/lto-wrapper
> Target: arm-linux-gnueabi
> Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.5.2-8ubuntu4' --with-bugurl=file:///usr/share/doc/gcc-4.5/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.5 --enable-shared --enable-multiarch --with-multiarch-defaults=arm-linux-gnueabi --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib/arm-linux-gnueabi --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.5 --libdir=/usr/lib/arm-linux-gnueabi --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-plugin --enable-gold --enable-ld=default --with-plugin-ld=ld.gold --enable-objc-gc --disable-sjlj-exceptions --with-arch=armv7-a --with-float=softfp --with-fpu=vfpv3-d16 --with-mode=thumb --disable-werror --enable-checking=release --build=arm-linux-gnueabi --host=arm-linux-gnueabi --target=arm-linux-gnueabi
> Thread model: posix
> gcc version 4.5.2 (Ubuntu/Linaro 4.5.2-8ubuntu4)
> $ uname -a
> Linux panda0 2.6.38-1002-linaro-omap #3-Ubuntu SMP Fri Apr 15 14:00:54 UTC 2011 armv7l armv7l armv7l GNU/Linux
>            Reporter: Trevor Robinson
>         Attachments: hadoop-common-arm.patch
>
>
> The native build fails on machine targets where gcc does not support -m32. This is any target other than x86, SPARC, RS/6000, or PowerPC, such as ARM.
> $ ant -Dcompile.native=true
> ...
>      [exec] make  all-am
>      [exec] make[1]: Entering directory
> `/home/trobinson/dev/hadoop-common/build/native/Linux-arm-32'
>      [exec] /bin/bash ./libtool  --tag=CC   --mode=compile gcc
> -DHAVE_CONFIG_H -I. -I/home/trobinson/dev/hadoop-common/src/native
> -I/usr/lib/jvm/java-6-openjdk/include
> -I/usr/lib/jvm/java-6-openjdk/include/linux
> -I/home/trobinson/dev/hadoop-common/src/native/src
> -Isrc/org/apache/hadoop/io/compress/zlib
> -Isrc/org/apache/hadoop/security -Isrc/org/apache/hadoop/io/nativeio/
> -g -Wall -fPIC -O2 -m32 -g -O2 -MT ZlibCompressor.lo -MD -MP -MF
> .deps/ZlibCompressor.Tpo -c -o ZlibCompressor.lo `test -f
> 'src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c' || echo
> '/home/trobinson/dev/hadoop-common/src/native/'`src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
>      [exec] libtool: compile:  gcc -DHAVE_CONFIG_H -I.
> -I/home/trobinson/dev/hadoop-common/src/native
> -I/usr/lib/jvm/java-6-openjdk/include
> -I/usr/lib/jvm/java-6-openjdk/include/linux
> -I/home/trobinson/dev/hadoop-common/src/native/src
> -Isrc/org/apache/hadoop/io/compress/zlib
> -Isrc/org/apache/hadoop/security -Isrc/org/apache/hadoop/io/nativeio/
> -g -Wall -fPIC -O2 -m32 -g -O2 -MT ZlibCompressor.lo -MD -MP -MF
> .deps/ZlibCompressor.Tpo -c
> /home/trobinson/dev/hadoop-common/src/native/src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
>  -fPIC -DPIC -o .libs/ZlibCompressor.o
>      [exec] make[1]: Leaving directory
> `/home/trobinson/dev/hadoop-common/build/native/Linux-arm-32'
>      [exec] cc1: error: unrecognized command line option "-m32"
>      [exec] make[1]: *** [ZlibCompressor.lo] Error 1
>      [exec] make: *** [all] Error 2

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14772-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 19:24:27 2011
Return-Path: <common-issues-return-14772-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D16FC5781
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 19:24:27 +0000 (UTC)
Received: (qmail 91744 invoked by uid 500); 12 May 2011 19:24:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91708 invoked by uid 500); 12 May 2011 19:24:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91700 invoked by uid 99); 12 May 2011 19:24:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 19:24:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 19:24:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7023B87481
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 19:23:47 +0000 (UTC)
Date: Thu, 12 May 2011 19:23:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <400449413.7828.1305228227455.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <532896131.3523.1305130907394.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7276) Hadoop native builds fail on ARM
 due to -m32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032601#comment-13032601 ] 

Hadoop QA commented on HADOOP-7276:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478844/hadoop-common-arm.patch
  against trunk revision 1102123.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs.  The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/439//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/439//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/439//console

This message is automatically generated.

> Hadoop native builds fail on ARM due to -m32
> --------------------------------------------
>
>                 Key: HADOOP-7276
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7276
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.21.0
>         Environment: $ gcc -v
> Using built-in specs.
> COLLECT_GCC=gcc
> COLLECT_LTO_WRAPPER=/usr/lib/arm-linux-gnueabi/gcc/arm-linux-gnueabi/4.5.2/lto-wrapper
> Target: arm-linux-gnueabi
> Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.5.2-8ubuntu4' --with-bugurl=file:///usr/share/doc/gcc-4.5/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.5 --enable-shared --enable-multiarch --with-multiarch-defaults=arm-linux-gnueabi --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib/arm-linux-gnueabi --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.5 --libdir=/usr/lib/arm-linux-gnueabi --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-plugin --enable-gold --enable-ld=default --with-plugin-ld=ld.gold --enable-objc-gc --disable-sjlj-exceptions --with-arch=armv7-a --with-float=softfp --with-fpu=vfpv3-d16 --with-mode=thumb --disable-werror --enable-checking=release --build=arm-linux-gnueabi --host=arm-linux-gnueabi --target=arm-linux-gnueabi
> Thread model: posix
> gcc version 4.5.2 (Ubuntu/Linaro 4.5.2-8ubuntu4)
> $ uname -a
> Linux panda0 2.6.38-1002-linaro-omap #3-Ubuntu SMP Fri Apr 15 14:00:54 UTC 2011 armv7l armv7l armv7l GNU/Linux
>            Reporter: Trevor Robinson
>         Attachments: hadoop-common-arm.patch
>
>
> The native build fails on machine targets where gcc does not support -m32. This is any target other than x86, SPARC, RS/6000, or PowerPC, such as ARM.
> $ ant -Dcompile.native=true
> ...
>      [exec] make  all-am
>      [exec] make[1]: Entering directory
> `/home/trobinson/dev/hadoop-common/build/native/Linux-arm-32'
>      [exec] /bin/bash ./libtool  --tag=CC   --mode=compile gcc
> -DHAVE_CONFIG_H -I. -I/home/trobinson/dev/hadoop-common/src/native
> -I/usr/lib/jvm/java-6-openjdk/include
> -I/usr/lib/jvm/java-6-openjdk/include/linux
> -I/home/trobinson/dev/hadoop-common/src/native/src
> -Isrc/org/apache/hadoop/io/compress/zlib
> -Isrc/org/apache/hadoop/security -Isrc/org/apache/hadoop/io/nativeio/
> -g -Wall -fPIC -O2 -m32 -g -O2 -MT ZlibCompressor.lo -MD -MP -MF
> .deps/ZlibCompressor.Tpo -c -o ZlibCompressor.lo `test -f
> 'src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c' || echo
> '/home/trobinson/dev/hadoop-common/src/native/'`src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
>      [exec] libtool: compile:  gcc -DHAVE_CONFIG_H -I.
> -I/home/trobinson/dev/hadoop-common/src/native
> -I/usr/lib/jvm/java-6-openjdk/include
> -I/usr/lib/jvm/java-6-openjdk/include/linux
> -I/home/trobinson/dev/hadoop-common/src/native/src
> -Isrc/org/apache/hadoop/io/compress/zlib
> -Isrc/org/apache/hadoop/security -Isrc/org/apache/hadoop/io/nativeio/
> -g -Wall -fPIC -O2 -m32 -g -O2 -MT ZlibCompressor.lo -MD -MP -MF
> .deps/ZlibCompressor.Tpo -c
> /home/trobinson/dev/hadoop-common/src/native/src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
>  -fPIC -DPIC -o .libs/ZlibCompressor.o
>      [exec] make[1]: Leaving directory
> `/home/trobinson/dev/hadoop-common/build/native/Linux-arm-32'
>      [exec] cc1: error: unrecognized command line option "-m32"
>      [exec] make[1]: *** [ZlibCompressor.lo] Error 1
>      [exec] make: *** [all] Error 2

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14773-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 19:26:27 2011
Return-Path: <common-issues-return-14773-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D988557CB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 19:26:27 +0000 (UTC)
Received: (qmail 93346 invoked by uid 500); 12 May 2011 19:26:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93316 invoked by uid 500); 12 May 2011 19:26:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93308 invoked by uid 99); 12 May 2011 19:26:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 19:26:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 19:26:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AFBB9875BB
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 19:25:47 +0000 (UTC)
Date: Thu, 12 May 2011 19:25:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <43637156.7837.1305228347716.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7269) S3 Native should allow
 customizable file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032602#comment-13032602 ] 

Hadoop QA commented on HADOOP-7269:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12478990/HADOOP-7269-S3-metadata-001.diff
  against trunk revision 1102123.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 9 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The applied patch generated 1071 javac compiler warnings (more than the trunk's current 1068 warnings).

    -1 findbugs.  The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.fs.TestFilterFileSystem

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/440//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/440//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/440//console

This message is automatically generated.

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: HADOOP-7269-S3-metadata-001.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14774-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 19:36:30 2011
Return-Path: <common-issues-return-14774-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E4C185F46
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 19:36:29 +0000 (UTC)
Received: (qmail 9489 invoked by uid 500); 12 May 2011 19:36:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9430 invoked by uid 500); 12 May 2011 19:36:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9422 invoked by uid 99); 12 May 2011 19:36:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 19:36:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 19:36:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B9DAD879D2
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 19:35:47 +0000 (UTC)
Date: Thu, 12 May 2011 19:35:47 +0000 (UTC)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <768915406.7876.1305228947758.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1894387424.4008.1305137809491.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7278) Automatic Hadoop cluster
 deployment for build validation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032610#comment-13032610 ] 

Konstantin Boudnik commented on HADOOP-7278:
--------------------------------------------

Yup. I think we'll be using an existing slave to drive this deployment not adding a new one.
Also, it seems to be a right think to do to chain a number of build to trigger each other e.g.: release build (i.e. the one creating tar balls) triggers deployment, which triggers testing.

> Automatic Hadoop cluster deployment for build validation
> --------------------------------------------------------
>
>                 Key: HADOOP-7278
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7278
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0, 0.23.0
>         Environment: Apache Jenkins
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>              Labels: newbie
>
> It'd be great to have a way of automatically deploying Hadoop cluster to a set of machine once all components are successfully built (in the form or tar or whatever). Deployed cluster then can be used to run a set of validation jobs to make sure that produced artifacts are viable. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14775-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 19:52:29 2011
Return-Path: <common-issues-return-14775-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AB1135681
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 19:52:29 +0000 (UTC)
Received: (qmail 28817 invoked by uid 500); 12 May 2011 19:52:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28776 invoked by uid 500); 12 May 2011 19:52:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28768 invoked by uid 99); 12 May 2011 19:52:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 19:52:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 19:52:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 64A3D87042
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 19:51:47 +0000 (UTC)
Date: Thu, 12 May 2011 19:51:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Trash and shell's rm does not work for viewfs
---------------------------------------------

                 Key: HADOOP-7284
                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Sanjay Radia
            Assignee: Sanjay Radia
             Fix For: 0.23.0




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14776-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 19:54:29 2011
Return-Path: <common-issues-return-14776-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8342E56DC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 19:54:29 +0000 (UTC)
Received: (qmail 33373 invoked by uid 500); 12 May 2011 19:54:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33338 invoked by uid 500); 12 May 2011 19:54:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33330 invoked by uid 99); 12 May 2011 19:54:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 19:54:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 19:54:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 66F5287123
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 19:53:47 +0000 (UTC)
Date: Thu, 12 May 2011 19:53:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1631189769.7924.1305230027418.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032620#comment-13032620 ] 

Sanjay Radia commented on HADOOP-7284:
--------------------------------------

Currently the shell's trashbin works for default file system that is hdfs and when fully qualified hdfs uri is given. 
It does not  work for for viewfs.

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14777-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 19:56:27 2011
Return-Path: <common-issues-return-14777-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 62D585778
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 19:56:27 +0000 (UTC)
Received: (qmail 37352 invoked by uid 500); 12 May 2011 19:56:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37241 invoked by uid 500); 12 May 2011 19:56:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37232 invoked by uid 99); 12 May 2011 19:56:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 19:56:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 19:56:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 66D9F87249
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 19:55:47 +0000 (UTC)
Date: Thu, 12 May 2011 19:55:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <759471379.7927.1305230147417.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7284:
---------------------------------

    Attachment: trash1.patch

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14778-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 19:56:27 2011
Return-Path: <common-issues-return-14778-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D968E5789
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 19:56:27 +0000 (UTC)
Received: (qmail 37557 invoked by uid 500); 12 May 2011 19:56:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37520 invoked by uid 500); 12 May 2011 19:56:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37505 invoked by uid 99); 12 May 2011 19:56:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 19:56:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 19:56:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CAAE587257
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 19:55:47 +0000 (UTC)
Date: Thu, 12 May 2011 19:55:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <452534829.7937.1305230147826.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <683476535.2038.1300136250528.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7189) Add ability to enable 'debug'
 property in JAAS configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7189:
--------------------------------

    Fix Version/s: 0.22.0

Oops, sorry about the fix version. I did check the UGI tests locally before committing - sorry I didn't mention this.

> Add ability to enable 'debug' property in JAAS configuration
> ------------------------------------------------------------
>
>                 Key: HADOOP-7189
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7189
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Ted Yu
>            Priority: Minor
>              Labels: newbie
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7189.patch, HADOOP-7189.txt, enable-UGI-debug-example.txt
>
>
> Occasionally users have run into weird "Unable to login" messages. Unfortunately, JAAS obscures the underlying exception message in many cases because it thinks leaking the exception might be insecure in itself. Enabling the "debug" option in the JAAS configuration gets it to dump the underlying issue and makes troubleshooting this kind of issue easier.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14779-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 19:56:28 2011
Return-Path: <common-issues-return-14779-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 331885798
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 19:56:28 +0000 (UTC)
Received: (qmail 37797 invoked by uid 500); 12 May 2011 19:56:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37759 invoked by uid 500); 12 May 2011 19:56:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37743 invoked by uid 99); 12 May 2011 19:56:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 19:56:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 19:56:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D54B287258
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 19:55:47 +0000 (UTC)
Date: Thu, 12 May 2011 19:55:47 +0000 (UTC)
From: "Jeffrey Naisbitt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1513276976.7938.1305230147870.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <161449362.3682.1305134627390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7277) Add Eclipse launch tasks for the
 0.20-security branch
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032623#comment-13032623 ] 

Jeffrey Naisbitt commented on HADOOP-7277:
------------------------------------------

We should probably also add .launches to the svn:ignore property.  Do you want to do that, Chris?

> Add Eclipse launch tasks for the 0.20-security branch
> -----------------------------------------------------
>
>                 Key: HADOOP-7277
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7277
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.204.0
>            Reporter: Jeffrey Naisbitt
>            Assignee: Jeffrey Naisbitt
>            Priority: Minor
>             Fix For: 0.20.205.0
>
>         Attachments: HADOOP-7277-0.20s.patch
>
>
> This is to add the eclipse launchers from HADOOP-5911 to the 0.20 security branch.
> Eclipse has a notion of "run configuration", which encapsulates what's needed to run or debug an application. I use this quite a bit to start various Hadoop daemons in debug mode, with breakpoints set, to inspect state and what not.
> This is simply configuration, so no tests are provided. After running "ant eclipse" and refreshing your project, you should see entries in the Run Configurations and Debug Configurations for launching the various hadoop daemons from within eclipse. There's a template for testing a specific test, and also templates to run all the tests, the job tracker, and a task tracker. It's likely that some parameters need to be further tweaked to have the same behavior as "ant test", but for most tests, this works.
> This also requires a small change to build.xml for the eclipse classpath.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14780-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 19:56:30 2011
Return-Path: <common-issues-return-14780-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0F28657A5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 19:56:30 +0000 (UTC)
Received: (qmail 38060 invoked by uid 500); 12 May 2011 19:56:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38028 invoked by uid 500); 12 May 2011 19:56:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38020 invoked by uid 99); 12 May 2011 19:56:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 19:56:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 19:56:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8E80487250
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 19:55:47 +0000 (UTC)
Date: Thu, 12 May 2011 19:55:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1375653163.7931.1305230147580.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7284:
---------------------------------

    Status: Patch Available  (was: Open)

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14781-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 20:17:29 2011
Return-Path: <common-issues-return-14781-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D362E5AE1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 20:17:29 +0000 (UTC)
Received: (qmail 75431 invoked by uid 500); 12 May 2011 20:17:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75401 invoked by uid 500); 12 May 2011 20:17:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75393 invoked by uid 99); 12 May 2011 20:17:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 20:17:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 20:17:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7AEF487A6A
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 20:16:47 +0000 (UTC)
Date: Thu, 12 May 2011 20:16:47 +0000 (UTC)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2044285181.8001.1305231407499.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <161449362.3682.1305134627390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7277) Add Eclipse launch tasks for the
 0.20-security branch
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032639#comment-13032639 ] 

Chris Douglas commented on HADOOP-7277:
---------------------------------------

*nod* Good point. Updated .gitignore file and svn:ignore properties for .launches/ and .externalToolBuilders/

> Add Eclipse launch tasks for the 0.20-security branch
> -----------------------------------------------------
>
>                 Key: HADOOP-7277
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7277
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.204.0
>            Reporter: Jeffrey Naisbitt
>            Assignee: Jeffrey Naisbitt
>            Priority: Minor
>             Fix For: 0.20.205.0
>
>         Attachments: HADOOP-7277-0.20s.patch
>
>
> This is to add the eclipse launchers from HADOOP-5911 to the 0.20 security branch.
> Eclipse has a notion of "run configuration", which encapsulates what's needed to run or debug an application. I use this quite a bit to start various Hadoop daemons in debug mode, with breakpoints set, to inspect state and what not.
> This is simply configuration, so no tests are provided. After running "ant eclipse" and refreshing your project, you should see entries in the Run Configurations and Debug Configurations for launching the various hadoop daemons from within eclipse. There's a template for testing a specific test, and also templates to run all the tests, the job tracker, and a task tracker. It's likely that some parameters need to be further tweaked to have the same behavior as "ant test", but for most tests, this works.
> This also requires a small change to build.xml for the eclipse classpath.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14782-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 20:27:30 2011
Return-Path: <common-issues-return-14782-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E4C0E5839
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 20:27:30 +0000 (UTC)
Received: (qmail 87564 invoked by uid 500); 12 May 2011 20:27:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87526 invoked by uid 500); 12 May 2011 20:27:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87518 invoked by uid 99); 12 May 2011 20:27:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 20:27:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 20:27:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B83F287ED6
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 20:26:47 +0000 (UTC)
Date: Thu, 12 May 2011 20:26:47 +0000 (UTC)
From: "Trevor Robinson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1067180573.8056.1305232007751.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <532896131.3523.1305130907394.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7276) Hadoop native builds fail on ARM
 due to -m32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032649#comment-13032649 ] 

Trevor Robinson commented on HADOOP-7276:
-----------------------------------------

No tests included because this change just fixes a build failure. Manually verified that x86-64 builds unchanged (-m64 is properly specified) and that ARM now builds (-m32 is not specified).

Findbugs issue seems to be a configuration issue unrelated to change. From console output:

     [exec] findbugs:
     [exec]     [mkdir] Created dir: /grid/0/hudson/hudson-slave/workspace/PreCommit-HADOOP-Build/trunk/build/test/findbugs
     [exec]  [findbugs] Executing findbugs from ant task
     [exec]  [findbugs] Running FindBugs...
     [exec]  [findbugs] The following classes needed for analysis were missing:
     [exec]  [findbugs]   com.sun.javadoc.Doclet
     [exec]  [findbugs]   com.sun.javadoc.DocErrorReporter
     [exec]  [findbugs]   com.sun.javadoc.AnnotationTypeDoc
     [exec]  [findbugs]   com.sun.javadoc.RootDoc
     [exec]  [findbugs]   com.sun.javadoc.MethodDoc
     [exec]  [findbugs]   com.sun.javadoc.Doc
     [exec]  [findbugs]   com.sun.javadoc.PackageDoc
     [exec]  [findbugs]   com.sun.javadoc.LanguageVersion
     [exec]  [findbugs]   com.sun.javadoc.AnnotationDesc
     [exec]  [findbugs]   com.sun.javadoc.ConstructorDoc
     [exec]  [findbugs]   com.sun.javadoc.FieldDoc
     [exec]  [findbugs]   com.sun.javadoc.ProgramElementDoc
     [exec]  [findbugs]   com.sun.javadoc.ClassDoc
     [exec]  [findbugs]   com.sun.tools.doclets.standard.Standard
     [exec]  [findbugs] Warnings generated: 1
     [exec]  [findbugs] Missing classes: 15
     [exec]  [findbugs] Calculating exit code...
     [exec]  [findbugs] Setting 'missing class' flag (2)
     [exec]  [findbugs] Setting 'bugs found' flag (1)
     [exec]  [findbugs] Exit code set to: 3
     [exec]  [findbugs] Classes needed for analysis were missing
     [exec]  [findbugs] Output saved to /grid/0/hudson/hudson-slave/workspace/PreCommit-HADOOP-Build/trunk/build/test/findbugs/hadoop-findbugs-report.xml
     [exec]  [findbugs] Java Result: 3

Also, the precommit build queue (https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/) seems to be hanging multiple recent jobs at "Recording test results". They're still running at >1hr.

Would a committer please review the change and let me know if I need to resubmit it?

> Hadoop native builds fail on ARM due to -m32
> --------------------------------------------
>
>                 Key: HADOOP-7276
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7276
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.21.0
>         Environment: $ gcc -v
> Using built-in specs.
> COLLECT_GCC=gcc
> COLLECT_LTO_WRAPPER=/usr/lib/arm-linux-gnueabi/gcc/arm-linux-gnueabi/4.5.2/lto-wrapper
> Target: arm-linux-gnueabi
> Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.5.2-8ubuntu4' --with-bugurl=file:///usr/share/doc/gcc-4.5/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.5 --enable-shared --enable-multiarch --with-multiarch-defaults=arm-linux-gnueabi --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib/arm-linux-gnueabi --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.5 --libdir=/usr/lib/arm-linux-gnueabi --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-plugin --enable-gold --enable-ld=default --with-plugin-ld=ld.gold --enable-objc-gc --disable-sjlj-exceptions --with-arch=armv7-a --with-float=softfp --with-fpu=vfpv3-d16 --with-mode=thumb --disable-werror --enable-checking=release --build=arm-linux-gnueabi --host=arm-linux-gnueabi --target=arm-linux-gnueabi
> Thread model: posix
> gcc version 4.5.2 (Ubuntu/Linaro 4.5.2-8ubuntu4)
> $ uname -a
> Linux panda0 2.6.38-1002-linaro-omap #3-Ubuntu SMP Fri Apr 15 14:00:54 UTC 2011 armv7l armv7l armv7l GNU/Linux
>            Reporter: Trevor Robinson
>         Attachments: hadoop-common-arm.patch
>
>
> The native build fails on machine targets where gcc does not support -m32. This is any target other than x86, SPARC, RS/6000, or PowerPC, such as ARM.
> $ ant -Dcompile.native=true
> ...
>      [exec] make  all-am
>      [exec] make[1]: Entering directory
> `/home/trobinson/dev/hadoop-common/build/native/Linux-arm-32'
>      [exec] /bin/bash ./libtool  --tag=CC   --mode=compile gcc
> -DHAVE_CONFIG_H -I. -I/home/trobinson/dev/hadoop-common/src/native
> -I/usr/lib/jvm/java-6-openjdk/include
> -I/usr/lib/jvm/java-6-openjdk/include/linux
> -I/home/trobinson/dev/hadoop-common/src/native/src
> -Isrc/org/apache/hadoop/io/compress/zlib
> -Isrc/org/apache/hadoop/security -Isrc/org/apache/hadoop/io/nativeio/
> -g -Wall -fPIC -O2 -m32 -g -O2 -MT ZlibCompressor.lo -MD -MP -MF
> .deps/ZlibCompressor.Tpo -c -o ZlibCompressor.lo `test -f
> 'src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c' || echo
> '/home/trobinson/dev/hadoop-common/src/native/'`src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
>      [exec] libtool: compile:  gcc -DHAVE_CONFIG_H -I.
> -I/home/trobinson/dev/hadoop-common/src/native
> -I/usr/lib/jvm/java-6-openjdk/include
> -I/usr/lib/jvm/java-6-openjdk/include/linux
> -I/home/trobinson/dev/hadoop-common/src/native/src
> -Isrc/org/apache/hadoop/io/compress/zlib
> -Isrc/org/apache/hadoop/security -Isrc/org/apache/hadoop/io/nativeio/
> -g -Wall -fPIC -O2 -m32 -g -O2 -MT ZlibCompressor.lo -MD -MP -MF
> .deps/ZlibCompressor.Tpo -c
> /home/trobinson/dev/hadoop-common/src/native/src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
>  -fPIC -DPIC -o .libs/ZlibCompressor.o
>      [exec] make[1]: Leaving directory
> `/home/trobinson/dev/hadoop-common/build/native/Linux-arm-32'
>      [exec] cc1: error: unrecognized command line option "-m32"
>      [exec] make[1]: *** [ZlibCompressor.lo] Error 1
>      [exec] make: *** [all] Error 2

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14783-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 20:52:29 2011
Return-Path: <common-issues-return-14783-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C8CC35590
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 20:52:29 +0000 (UTC)
Received: (qmail 17090 invoked by uid 500); 12 May 2011 20:52:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17051 invoked by uid 500); 12 May 2011 20:52:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17042 invoked by uid 99); 12 May 2011 20:52:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 20:52:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 20:52:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6831F87A53
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 20:51:47 +0000 (UTC)
Date: Thu, 12 May 2011 20:51:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <672077683.8145.1305233507423.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032667#comment-13032667 ] 

Hadoop QA commented on HADOOP-7284:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479005/trash1.patch
  against trunk revision 1102123.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs.  The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/441//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/441//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/441//console

This message is automatically generated.

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14784-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 23:51:29 2011
Return-Path: <common-issues-return-14784-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C629753AC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 23:51:29 +0000 (UTC)
Received: (qmail 93955 invoked by uid 500); 12 May 2011 23:51:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93925 invoked by uid 500); 12 May 2011 23:51:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93917 invoked by uid 99); 12 May 2011 23:51:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 23:51:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 23:51:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9167C87873
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 23:50:47 +0000 (UTC)
Date: Thu, 12 May 2011 23:50:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <32042405.8638.1305244247592.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7285) Refactor FsShell's test
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Refactor FsShell's test
-----------------------

                 Key: HADOOP-7285
                 URL: https://issues.apache.org/jira/browse/HADOOP-7285
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs
            Reporter: Daryn Sharp
            Assignee: Daryn Sharp
             Fix For: 0.23.0


Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14785-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 23:51:33 2011
Return-Path: <common-issues-return-14785-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3223453E0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 23:51:33 +0000 (UTC)
Received: (qmail 94422 invoked by uid 500); 12 May 2011 23:51:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94336 invoked by uid 500); 12 May 2011 23:51:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94252 invoked by uid 99); 12 May 2011 23:51:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 23:51:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 23:51:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A427587876
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 23:50:47 +0000 (UTC)
Date: Thu, 12 May 2011 23:50:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1619049648.8641.1305244247668.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7286) Refactor FsShell's du/dus/df
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Refactor FsShell's du/dus/df
----------------------------

                 Key: HADOOP-7286
                 URL: https://issues.apache.org/jira/browse/HADOOP-7286
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs
    Affects Versions: 0.23.0
            Reporter: Daryn Sharp
            Assignee: Daryn Sharp


Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14786-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 23:59:27 2011
Return-Path: <common-issues-return-14786-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7A84B56C9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 23:59:27 +0000 (UTC)
Received: (qmail 6684 invoked by uid 500); 12 May 2011 23:59:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6656 invoked by uid 500); 12 May 2011 23:59:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6648 invoked by uid 99); 12 May 2011 23:59:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 23:59:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 23:59:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5C0A987B52
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 23:58:47 +0000 (UTC)
Date: Thu, 12 May 2011 23:58:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <991205569.8661.1305244727373.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1619049648.8641.1305244247668.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7286) Refactor FsShell's du/dus/df
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7286:
--------------------------------

    Attachment: HADOOP-7286.patch

refactor and make output a bit more conforming to unix standards

> Refactor FsShell's du/dus/df
> ----------------------------
>
>                 Key: HADOOP-7286
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7286
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7286.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14787-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 12 23:59:27 2011
Return-Path: <common-issues-return-14787-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ABF9956D9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 12 May 2011 23:59:27 +0000 (UTC)
Received: (qmail 6910 invoked by uid 500); 12 May 2011 23:59:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6855 invoked by uid 500); 12 May 2011 23:59:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6693 invoked by uid 99); 12 May 2011 23:59:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 23:59:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2011 23:59:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 69B6D87B54
	for <common-issues@hadoop.apache.org>; Thu, 12 May 2011 23:58:47 +0000 (UTC)
Date: Thu, 12 May 2011 23:58:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <303758445.8663.1305244727429.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1619049648.8641.1305244247668.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7286) Refactor FsShell's du/dus/df
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7286:
--------------------------------

    Status: Patch Available  (was: Open)

> Refactor FsShell's du/dus/df
> ----------------------------
>
>                 Key: HADOOP-7286
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7286
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7286.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14788-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 00:31:27 2011
Return-Path: <common-issues-return-14788-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 48B095D62
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 00:31:27 +0000 (UTC)
Received: (qmail 29802 invoked by uid 500); 13 May 2011 00:31:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29773 invoked by uid 500); 13 May 2011 00:31:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29764 invoked by uid 99); 13 May 2011 00:31:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 00:31:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 00:31:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 595FC883A5
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 00:30:47 +0000 (UTC)
Date: Fri, 13 May 2011 00:30:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <187742091.8722.1305246647362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7287) Configuration deprecation mechanism
 doesn't work properly for GenericOptionsParser/Tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Configuration deprecation mechanism doesn't work properly for GenericOptionsParser/Tools
----------------------------------------------------------------------------------------

                 Key: HADOOP-7287
                 URL: https://issues.apache.org/jira/browse/HADOOP-7287
             Project: Hadoop Common
          Issue Type: Bug
          Components: conf
    Affects Versions: 0.22.0
            Reporter: Todd Lipcon
            Priority: Blocker
             Fix For: 0.22.0
         Attachments: hadoop-7287-testcase.txt

For example, you can't use -D options on the "hadoop fs" command line in order to specify the deprecated names of configuration options. The issue is that the ordering is:
- JVM starts
- GenericOptionsParser creates a Configuration object and calls set() for each of the options specified on command line
- DistributedFileSystem or other class eventually instantiates HdfsConfiguration which adds the deprecations
- Some class calls conf.get("new key") and sees the default instead of the version set on the command line

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14789-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 00:31:29 2011
Return-Path: <common-issues-return-14789-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DE36E5DA4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 00:31:28 +0000 (UTC)
Received: (qmail 30101 invoked by uid 500); 13 May 2011 00:31:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30014 invoked by uid 500); 13 May 2011 00:31:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29988 invoked by uid 99); 13 May 2011 00:31:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 00:31:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 00:31:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 764BA883A8
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 00:30:47 +0000 (UTC)
Date: Fri, 13 May 2011 00:30:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <342790462.8725.1305246647481.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <187742091.8722.1305246647362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7287) Configuration deprecation mechanism
 doesn't work properly for GenericOptionsParser/Tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7287:
--------------------------------

    Attachment: hadoop-7287-testcase.txt

Here's a test case which illustrates why this problem happens. You can easily reproduce from the command line by trying something like:

./bin/hadoop fs -Ddfs.block.size=4096 -touchz f1
./bin/hadoop fs -stat "%o" f1

and you'll see that the deprecated option was ignored and there was no warning.

> Configuration deprecation mechanism doesn't work properly for GenericOptionsParser/Tools
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7287
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7287
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7287-testcase.txt
>
>
> For example, you can't use -D options on the "hadoop fs" command line in order to specify the deprecated names of configuration options. The issue is that the ordering is:
> - JVM starts
> - GenericOptionsParser creates a Configuration object and calls set() for each of the options specified on command line
> - DistributedFileSystem or other class eventually instantiates HdfsConfiguration which adds the deprecations
> - Some class calls conf.get("new key") and sees the default instead of the version set on the command line

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14790-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 00:53:27 2011
Return-Path: <common-issues-return-14790-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8F20C5F9C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 00:53:27 +0000 (UTC)
Received: (qmail 50029 invoked by uid 500); 13 May 2011 00:53:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50007 invoked by uid 500); 13 May 2011 00:53:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49999 invoked by uid 99); 13 May 2011 00:53:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 00:53:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 00:53:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6BA7588A0C
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 00:52:47 +0000 (UTC)
Date: Fri, 13 May 2011 00:52:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <141792235.8763.1305247967437.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32042405.8638.1305244247592.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7285) Refactor FsShell's test
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7285:
--------------------------------

    Attachment: HADOOP-7285.patch

> Refactor FsShell's test
> -----------------------
>
>                 Key: HADOOP-7285
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7285
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7285.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14791-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 00:53:29 2011
Return-Path: <common-issues-return-14791-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CCE175FAE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 00:53:29 +0000 (UTC)
Received: (qmail 50331 invoked by uid 500); 13 May 2011 00:53:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50282 invoked by uid 500); 13 May 2011 00:53:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50274 invoked by uid 99); 13 May 2011 00:53:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 00:53:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 00:53:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 92E3988A0F
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 00:52:47 +0000 (UTC)
Date: Fri, 13 May 2011 00:52:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <884610506.8766.1305247967598.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32042405.8638.1305244247592.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7285) Refactor FsShell's test
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7285:
--------------------------------

    Status: Patch Available  (was: Open)

> Refactor FsShell's test
> -----------------------
>
>                 Key: HADOOP-7285
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7285
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7285.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14792-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 09:45:28 2011
Return-Path: <common-issues-return-14792-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C9CCD4D6C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 09:45:28 +0000 (UTC)
Received: (qmail 2945 invoked by uid 500); 13 May 2011 09:45:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2866 invoked by uid 500); 13 May 2011 09:45:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2852 invoked by uid 99); 13 May 2011 09:45:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 09:45:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 09:45:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 796A28899D
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 09:44:47 +0000 (UTC)
Date: Fri, 13 May 2011 09:44:47 +0000 (UTC)
From: "issei yoshida (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1891354442.9530.1305279887493.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

issei yoshida updated HADOOP-7206:
----------------------------------

    Affects Version/s: 0.21.0
         Release Note: 
This patch bring the snappy compression to Hadoop.

It requires the snappy v1.0.2 or higher,
Because it doesn't use C++ interface but C interface and loads the snappy native library dynamically via dlopen.

Its native library is a part of the libhadoop.
               Status: Patch Available  (was: Open)

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14793-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 09:49:31 2011
Return-Path: <common-issues-return-14793-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 579A24081
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 09:49:31 +0000 (UTC)
Received: (qmail 5963 invoked by uid 500); 13 May 2011 09:49:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5820 invoked by uid 500); 13 May 2011 09:49:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5812 invoked by uid 99); 13 May 2011 09:49:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 09:49:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 09:49:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 10D5088B41
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 09:48:49 +0000 (UTC)
Date: Fri, 13 May 2011 09:48:49 +0000 (UTC)
From: "issei yoshida (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <533827770.9611.1305280129065.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032929#comment-13032929 ] 

issei yoshida commented on HADOOP-7206:
---------------------------------------

Hi all,

Thank you for giving permission to relisence regarding hadoop-snappy.

I created the patch that add the snappy compression support to the trunk of the Hadoop SVN repository.

Its native library is a part of the libhadoop and it requires the snappy v1.0.2 or higher.
(Because it doesn't use C++ interface but C interface and load the snappy native library dynamically via dlopen.)

I would like to hear your advice.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>         Attachments: HADOOP-7206.patch
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14794-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 09:49:31 2011
Return-Path: <common-issues-return-14794-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 888A1408A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 09:49:31 +0000 (UTC)
Received: (qmail 6098 invoked by uid 500); 13 May 2011 09:49:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6048 invoked by uid 500); 13 May 2011 09:49:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5974 invoked by uid 99); 13 May 2011 09:49:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 09:49:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 09:49:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0A35C88B40
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 09:48:49 +0000 (UTC)
Date: Fri, 13 May 2011 09:48:49 +0000 (UTC)
From: "issei yoshida (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1075115204.9610.1305280129038.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

issei yoshida updated HADOOP-7206:
----------------------------------

    Attachment: HADOOP-7206.patch

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>         Attachments: HADOOP-7206.patch
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14795-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 10:56:27 2011
Return-Path: <common-issues-return-14795-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8DE7B496C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 10:56:27 +0000 (UTC)
Received: (qmail 32227 invoked by uid 500); 13 May 2011 10:56:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32201 invoked by uid 500); 13 May 2011 10:56:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32193 invoked by uid 99); 13 May 2011 10:56:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 10:56:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 10:56:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8B1B0887FB
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 10:55:47 +0000 (UTC)
Date: Fri, 13 May 2011 10:55:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1459321175.9835.1305284147566.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32042405.8638.1305244247592.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7285) Refactor FsShell's test
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032959#comment-13032959 ] 

Hadoop QA commented on HADOOP-7285:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479046/HADOOP-7285.patch
  against trunk revision 1102123.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs.  The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/443//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/443//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/443//console

This message is automatically generated.

> Refactor FsShell's test
> -----------------------
>
>                 Key: HADOOP-7285
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7285
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7285.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14796-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 10:56:30 2011
Return-Path: <common-issues-return-14796-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 416C24986
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 10:56:30 +0000 (UTC)
Received: (qmail 32685 invoked by uid 500); 13 May 2011 10:56:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32632 invoked by uid 500); 13 May 2011 10:56:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32537 invoked by uid 99); 13 May 2011 10:56:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 10:56:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 10:56:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 72975887F4
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 10:55:47 +0000 (UTC)
Date: Fri, 13 May 2011 10:55:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1568643670.9832.1305284147466.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1619049648.8641.1305244247668.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7286) Refactor FsShell's du/dus/df
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032958#comment-13032958 ] 

Hadoop QA commented on HADOOP-7286:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479039/HADOOP-7286.patch
  against trunk revision 1102123.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 4 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs.  The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/442//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/442//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/442//console

This message is automatically generated.

> Refactor FsShell's du/dus/df
> ----------------------------
>
>                 Key: HADOOP-7286
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7286
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7286.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14797-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 10:58:29 2011
Return-Path: <common-issues-return-14797-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6419C4A20
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 10:58:29 +0000 (UTC)
Received: (qmail 37305 invoked by uid 500); 13 May 2011 10:58:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37269 invoked by uid 500); 13 May 2011 10:58:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37261 invoked by uid 99); 13 May 2011 10:58:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 10:58:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 10:58:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5256588954
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 10:57:49 +0000 (UTC)
Date: Fri, 13 May 2011 10:57:49 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1622700926.9885.1305284269333.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032961#comment-13032961 ] 

Hadoop QA commented on HADOOP-7206:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479073/HADOOP-7206.patch
  against trunk revision 1102123.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs.  The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.io.compress.TestCodec

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/444//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/444//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/444//console

This message is automatically generated.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>         Attachments: HADOOP-7206.patch
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14798-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 11:00:27 2011
Return-Path: <common-issues-return-14798-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A72E34D12
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 11:00:27 +0000 (UTC)
Received: (qmail 38196 invoked by uid 500); 13 May 2011 11:00:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38166 invoked by uid 500); 13 May 2011 11:00:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38158 invoked by uid 99); 13 May 2011 11:00:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 11:00:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 11:00:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B215C889DC
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 10:59:47 +0000 (UTC)
Date: Fri, 13 May 2011 10:59:47 +0000 (UTC)
From: "Nicholas Telford (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <618841109.9890.1305284387725.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7269) S3 Native should allow customizable
 file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicholas Telford updated HADOOP-7269:
-------------------------------------

    Status: In Progress  (was: Patch Available)

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: HADOOP-7269-S3-metadata-001.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14799-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 11:02:27 2011
Return-Path: <common-issues-return-14799-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BFD614732
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 11:02:27 +0000 (UTC)
Received: (qmail 40075 invoked by uid 500); 13 May 2011 11:02:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40046 invoked by uid 500); 13 May 2011 11:02:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40038 invoked by uid 99); 13 May 2011 11:02:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 11:02:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 11:02:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CA0C288A86
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 11:01:47 +0000 (UTC)
Date: Fri, 13 May 2011 11:01:47 +0000 (UTC)
From: "Nicholas Telford (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1440211408.9898.1305284507824.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7269) S3 Native should allow customizable
 file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicholas Telford updated HADOOP-7269:
-------------------------------------

    Status: Patch Available  (was: In Progress)

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: HADOOP-7269-S3-metadata-001.diff, HADOOP-7269-S3-metadata-002.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14800-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 11:02:30 2011
Return-Path: <common-issues-return-14800-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2615B4777
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 11:02:30 +0000 (UTC)
Received: (qmail 40419 invoked by uid 500); 13 May 2011 11:02:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40306 invoked by uid 500); 13 May 2011 11:02:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40298 invoked by uid 99); 13 May 2011 11:02:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 11:02:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 11:02:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9F9C788A80
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 11:01:47 +0000 (UTC)
Date: Fri, 13 May 2011 11:01:47 +0000 (UTC)
From: "Nicholas Telford (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1819460217.9895.1305284507650.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7269) S3 Native should allow customizable
 file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicholas Telford updated HADOOP-7269:
-------------------------------------

    Attachment: HADOOP-7269-S3-metadata-002.diff

FIxed broken unit test that I missed in the first round of the patch.

Fixed unchecked cast. Seems the Jets3t library isn't returning a typed collection as it's docs suggest. Working around this is more costly due to the iteration and type-checking required - is this the right way to solve this issue, or would @SuppressWarnings("unchecked") have been better?

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: HADOOP-7269-S3-metadata-001.diff, HADOOP-7269-S3-metadata-002.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14801-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 11:25:27 2011
Return-Path: <common-issues-return-14801-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 86F3F4C47
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 11:25:27 +0000 (UTC)
Received: (qmail 79108 invoked by uid 500); 13 May 2011 11:25:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79069 invoked by uid 500); 13 May 2011 11:25:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79061 invoked by uid 99); 13 May 2011 11:25:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 11:25:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 11:25:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5C9F6881FA
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 11:24:47 +0000 (UTC)
Date: Fri, 13 May 2011 11:24:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <991263251.9922.1305285887375.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7269) S3 Native should allow
 customizable file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032969#comment-13032969 ] 

Hadoop QA commented on HADOOP-7269:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479088/HADOOP-7269-S3-metadata-002.diff
  against trunk revision 1102123.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 12 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The applied patch generated 1070 javac compiler warnings (more than the trunk's current 1068 warnings).

    -1 findbugs.  The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/445//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/445//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/445//console

This message is automatically generated.

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: HADOOP-7269-S3-metadata-001.diff, HADOOP-7269-S3-metadata-002.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14802-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 12:55:30 2011
Return-Path: <common-issues-return-14802-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6AEAE4E18
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 12:55:30 +0000 (UTC)
Received: (qmail 86425 invoked by uid 500); 13 May 2011 12:55:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86397 invoked by uid 500); 13 May 2011 12:55:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86389 invoked by uid 99); 13 May 2011 12:55:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 12:55:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 12:55:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F28C488464
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 12:54:47 +0000 (UTC)
Date: Fri, 13 May 2011 12:54:47 +0000 (UTC)
From: "Nicholas Telford (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <391702362.10038.1305291287989.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7269) S3 Native should allow
 customizable file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033012#comment-13033012 ] 

Nicholas Telford commented on HADOOP-7269:
------------------------------------------

RE: QA analysis - The findbugs issue is about FsShell.DelayedExceptionThrowing, which my patch hasn't affected at all. Since it's abstract and I can't find anything that uses it, I think that's another issue altogether.

The 2 additional compiler warnings - I can't find them anywhere. If anyone can suggest an effective way to locate these, I'm all ears.

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: HADOOP-7269-S3-metadata-001.diff, HADOOP-7269-S3-metadata-002.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14803-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 14:28:32 2011
Return-Path: <common-issues-return-14803-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1E890481C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 14:28:32 +0000 (UTC)
Received: (qmail 67240 invoked by uid 500); 13 May 2011 14:28:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67183 invoked by uid 500); 13 May 2011 14:28:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67146 invoked by uid 99); 13 May 2011 14:28:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 14:28:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 14:28:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4F4E088D96
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 14:27:47 +0000 (UTC)
Date: Fri, 13 May 2011 14:27:47 +0000 (UTC)
From: "Nicholas Telford (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1115617041.10194.1305296867305.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7288) FsShell.DelayedExceptionThrowing
 generates QA errors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

FsShell.DelayedExceptionThrowing generates QA errors
----------------------------------------------------

                 Key: HADOOP-7288
                 URL: https://issues.apache.org/jira/browse/HADOOP-7288
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
            Reporter: Nicholas Telford
            Assignee: Nicholas Telford
            Priority: Minor


The inner class o.a.h.fs.FsShell.DelayedExceptionThrowing is causing the QA bot to fail all patches due to a findbugs issue (the inner class should be static).

It's an abstract class that is not currently used. It looks as though it's old code that simply needs removing.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14804-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 14:30:27 2011
Return-Path: <common-issues-return-14804-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A843B4A4A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 14:30:27 +0000 (UTC)
Received: (qmail 70772 invoked by uid 500); 13 May 2011 14:30:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70708 invoked by uid 500); 13 May 2011 14:30:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70700 invoked by uid 99); 13 May 2011 14:30:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 14:30:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 14:30:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6D93D88EC3
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 14:29:47 +0000 (UTC)
Date: Fri, 13 May 2011 14:29:47 +0000 (UTC)
From: "Nicholas Telford (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <461660761.10200.1305296987445.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1115617041.10194.1305296867305.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7288) FsShell.DelayedExceptionThrowing
 generates QA errors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicholas Telford updated HADOOP-7288:
-------------------------------------

    Attachment: HADOOP-7288-remove-redundant-class-001.patch

Patch to remove o.a.h.fs.FsShell.DelayedExceptionThrowing class.

> FsShell.DelayedExceptionThrowing generates QA errors
> ----------------------------------------------------
>
>                 Key: HADOOP-7288
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7288
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: HADOOP-7288-remove-redundant-class-001.patch
>
>
> The inner class o.a.h.fs.FsShell.DelayedExceptionThrowing is causing the QA bot to fail all patches due to a findbugs issue (the inner class should be static).
> It's an abstract class that is not currently used. It looks as though it's old code that simply needs removing.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14805-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 15:55:27 2011
Return-Path: <common-issues-return-14805-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C7FAD40B8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 15:55:27 +0000 (UTC)
Received: (qmail 69093 invoked by uid 500); 13 May 2011 15:55:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69062 invoked by uid 500); 13 May 2011 15:55:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69054 invoked by uid 99); 13 May 2011 15:55:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 15:55:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 15:55:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9193489355
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 15:54:47 +0000 (UTC)
Date: Fri, 13 May 2011 15:54:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1906313121.10395.1305302087592.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32042405.8638.1305244247592.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7285) Refactor FsShell's test
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7285:
--------------------------------

    Attachment: HADOOP-7285-2.patch

(Mishap caused by forgetting to mvn-install before running hdfs tests...)

Don't throw FNF exception, just display the error, so correct exit code is returned and the tests in HDFS-1933 will pass.  

Also removed unrelated and unused DelayedExceptionThrowing that is causing a findbugs warning.

> Refactor FsShell's test
> -----------------------
>
>                 Key: HADOOP-7285
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7285
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7285-2.patch, HADOOP-7285.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14806-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 16:15:28 2011
Return-Path: <common-issues-return-14806-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DA1B24CAF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 16:15:27 +0000 (UTC)
Received: (qmail 98690 invoked by uid 500); 13 May 2011 16:15:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98633 invoked by uid 500); 13 May 2011 16:15:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98625 invoked by uid 99); 13 May 2011 16:15:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 16:15:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 16:15:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id ACF3789BCC
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 16:14:47 +0000 (UTC)
Date: Fri, 13 May 2011 16:14:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <326185698.10458.1305303287704.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32042405.8638.1305244247592.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7285) Refactor FsShell's test
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033106#comment-13033106 ] 

Hadoop QA commented on HADOOP-7285:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479132/HADOOP-7285-2.patch
  against trunk revision 1102123.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/446//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/446//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/446//console

This message is automatically generated.

> Refactor FsShell's test
> -----------------------
>
>                 Key: HADOOP-7285
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7285
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7285-2.patch, HADOOP-7285.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14807-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 16:51:28 2011
Return-Path: <common-issues-return-14807-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3207B44E1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 16:51:28 +0000 (UTC)
Received: (qmail 42580 invoked by uid 500); 13 May 2011 16:51:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42551 invoked by uid 500); 13 May 2011 16:51:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42527 invoked by uid 99); 13 May 2011 16:51:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 16:51:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 16:51:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0089589851
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 16:50:48 +0000 (UTC)
Date: Fri, 13 May 2011 16:50:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1437065322.10561.1305305447998.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1619049648.8641.1305244247668.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7286) Refactor FsShell's du/dus/df
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7286:
--------------------------------

    Attachment: HADOOP-7286-2.patch

Fix findbug warning by changing String concat to StringBuilder.  Also removed unrelated and unused method that is generating another findbug warning.

> Refactor FsShell's du/dus/df
> ----------------------------
>
>                 Key: HADOOP-7286
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7286
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7286-2.patch, HADOOP-7286.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14808-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 16:51:28 2011
Return-Path: <common-issues-return-14808-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3688744EF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 16:51:28 +0000 (UTC)
Received: (qmail 42605 invoked by uid 500); 13 May 2011 16:51:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42554 invoked by uid 500); 13 May 2011 16:51:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42528 invoked by uid 99); 13 May 2011 16:51:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 16:51:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 16:51:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EDDDC89850
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 16:50:47 +0000 (UTC)
Date: Fri, 13 May 2011 16:50:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1861357253.10560.1305305447970.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <957076305.12265.1297477317500.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7137) Remove hod contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7137:
--------------------------------

    Fix Version/s:     (was: 0.23.0)
                   0.22.0

+1

lgtm

> Remove hod contrib
> ------------------
>
>                 Key: HADOOP-7137
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7137
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7137.patch, HADOOP-7137.patch
>
>
> As per vote on general@ (http://mail-archives.apache.org/mod_mbox/hadoop-general/201102.mbox/%3CAC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com%3E) I will 
> svn remove common/trunk/src/contrib/hod
> using this Jira.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14809-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 17:05:27 2011
Return-Path: <common-issues-return-14809-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A10EB4CFE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 17:05:27 +0000 (UTC)
Received: (qmail 66528 invoked by uid 500); 13 May 2011 17:05:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66500 invoked by uid 500); 13 May 2011 17:05:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66492 invoked by uid 99); 13 May 2011 17:05:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 17:05:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 17:05:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 66C6589DCC
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 17:04:47 +0000 (UTC)
Date: Fri, 13 May 2011 17:04:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1250250131.10611.1305306287417.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1619049648.8641.1305244247668.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7286) Refactor FsShell's du/dus/df
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033131#comment-13033131 ] 

Hadoop QA commented on HADOOP-7286:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479137/HADOOP-7286-2.patch
  against trunk revision 1102123.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 4 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.metrics2.impl.TestSinkQueue

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/447//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/447//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/447//console

This message is automatically generated.

> Refactor FsShell's du/dus/df
> ----------------------------
>
>                 Key: HADOOP-7286
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7286
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7286-2.patch, HADOOP-7286.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14810-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 17:11:27 2011
Return-Path: <common-issues-return-14810-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9ECEC4E68
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 17:11:27 +0000 (UTC)
Received: (qmail 83881 invoked by uid 500); 13 May 2011 17:11:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83847 invoked by uid 500); 13 May 2011 17:11:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83839 invoked by uid 99); 13 May 2011 17:11:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 17:11:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 17:11:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7013389FAC
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 17:10:47 +0000 (UTC)
Date: Fri, 13 May 2011 17:10:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <417543205.10626.1305306647455.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1619049648.8641.1305244247668.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7286) Refactor FsShell's du/dus/df
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033134#comment-13033134 ] 

Daryn Sharp commented on HADOOP-7286:
-------------------------------------

Metrics test failure is unrelated to this patch.

> Refactor FsShell's du/dus/df
> ----------------------------
>
>                 Key: HADOOP-7286
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7286
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7286-2.patch, HADOOP-7286.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14811-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 17:21:27 2011
Return-Path: <common-issues-return-14811-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 92DA84654
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 17:21:27 +0000 (UTC)
Received: (qmail 92721 invoked by uid 500); 13 May 2011 17:21:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92664 invoked by uid 500); 13 May 2011 17:21:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92651 invoked by uid 99); 13 May 2011 17:21:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 17:21:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 17:21:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 69D3E89345
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 17:20:47 +0000 (UTC)
Date: Fri, 13 May 2011 17:20:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1737110561.10648.1305307247430.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32042405.8638.1305244247592.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7285) Refactor FsShell's test
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033139#comment-13033139 ] 

Aaron T. Myers commented on HADOOP-7285:
----------------------------------------

+1. Patch looks good, Daryn.

One question: why is the {{flag}} instance variable in {{o.a.h.fs.shell.Test}} protected, instead of private?

> Refactor FsShell's test
> -----------------------
>
>                 Key: HADOOP-7285
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7285
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7285-2.patch, HADOOP-7285.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14812-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 17:48:29 2011
Return-Path: <common-issues-return-14812-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EB006458E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 17:48:29 +0000 (UTC)
Received: (qmail 37029 invoked by uid 500); 13 May 2011 17:48:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36982 invoked by uid 500); 13 May 2011 17:48:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36974 invoked by uid 99); 13 May 2011 17:48:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 17:48:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 17:48:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A030589DA2
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 17:47:47 +0000 (UTC)
Date: Fri, 13 May 2011 17:47:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <655456119.10714.1305308867652.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32042405.8638.1305244247592.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7285) Refactor FsShell's test
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033154#comment-13033154 ] 

Daryn Sharp commented on HADOOP-7285:
-------------------------------------

Good question.  I suppose there is no reason.  Shall I change it?

> Refactor FsShell's test
> -----------------------
>
>                 Key: HADOOP-7285
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7285
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7285-2.patch, HADOOP-7285.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14813-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 17:50:29 2011
Return-Path: <common-issues-return-14813-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6D4F44617
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 17:50:29 +0000 (UTC)
Received: (qmail 43366 invoked by uid 500); 13 May 2011 17:50:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43324 invoked by uid 500); 13 May 2011 17:50:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43316 invoked by uid 99); 13 May 2011 17:50:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 17:50:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 17:50:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7050389EA9
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 17:49:47 +0000 (UTC)
Date: Fri, 13 May 2011 17:49:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <928396328.10721.1305308987456.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32042405.8638.1305244247592.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7285) Refactor FsShell's test
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033157#comment-13033157 ] 

Aaron T. Myers commented on HADOOP-7285:
----------------------------------------

Yup. Might as well be private if you don't intend {{Test}} to be subclassed.

> Refactor FsShell's test
> -----------------------
>
>                 Key: HADOOP-7285
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7285
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7285-2.patch, HADOOP-7285.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14814-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 17:56:27 2011
Return-Path: <common-issues-return-14814-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E185E4723
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 17:56:27 +0000 (UTC)
Received: (qmail 49623 invoked by uid 500); 13 May 2011 17:56:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49585 invoked by uid 500); 13 May 2011 17:56:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49577 invoked by uid 99); 13 May 2011 17:56:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 17:56:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 17:56:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9AB6E890C9
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 17:55:47 +0000 (UTC)
Date: Fri, 13 May 2011 17:55:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1232777387.10732.1305309347630.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32042405.8638.1305244247592.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7285) Refactor FsShell's test
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7285:
--------------------------------

    Attachment: HADOOP-7285-3.patch

Change "flag" from protected to private as requested.  Thanks, Aaron!

> Refactor FsShell's test
> -----------------------
>
>                 Key: HADOOP-7285
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7285
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7285-2.patch, HADOOP-7285-3.patch, HADOOP-7285.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14815-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 18:19:39 2011
Return-Path: <common-issues-return-14815-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4F8BB48DC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 18:19:39 +0000 (UTC)
Received: (qmail 73348 invoked by uid 500); 13 May 2011 18:14:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73275 invoked by uid 500); 13 May 2011 18:14:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73246 invoked by uid 99); 13 May 2011 18:14:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 18:14:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 18:14:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8D6C989709
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 18:13:48 +0000 (UTC)
Date: Fri, 13 May 2011 18:13:48 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1450159383.10798.1305310428575.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32042405.8638.1305244247592.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7285) Refactor FsShell's test
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033170#comment-13033170 ] 

Hadoop QA commented on HADOOP-7285:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479147/HADOOP-7285-3.patch
  against trunk revision 1102123.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/448//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/448//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/448//console

This message is automatically generated.

> Refactor FsShell's test
> -----------------------
>
>                 Key: HADOOP-7285
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7285
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7285-2.patch, HADOOP-7285-3.patch, HADOOP-7285.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14816-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 18:22:28 2011
Return-Path: <common-issues-return-14816-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0CF984C5E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 18:22:28 +0000 (UTC)
Received: (qmail 6421 invoked by uid 500); 13 May 2011 18:22:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6399 invoked by uid 500); 13 May 2011 18:22:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6391 invoked by uid 99); 13 May 2011 18:22:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 18:22:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 18:22:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E2C2F89D1E
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 18:21:47 +0000 (UTC)
Date: Fri, 13 May 2011 18:21:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2108583256.10817.1305310907925.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <957076305.12265.1297477317500.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7137) Remove hod contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7137:
--------------------------------

      Resolution: Fixed
    Hadoop Flags: [Incompatible change, Reviewed]  (was: [Incompatible change])
          Status: Resolved  (was: Patch Available)

I've committed this to trunk and branch 22.  Thanks Nigel!

Tested via ant tar and ant package instead of Hudson since most of the change was done via svn remove (the patch only applies to build.xml). 

> Remove hod contrib
> ------------------
>
>                 Key: HADOOP-7137
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7137
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7137.patch, HADOOP-7137.patch
>
>
> As per vote on general@ (http://mail-archives.apache.org/mod_mbox/hadoop-general/201102.mbox/%3CAC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com%3E) I will 
> svn remove common/trunk/src/contrib/hod
> using this Jira.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14817-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 18:22:28 2011
Return-Path: <common-issues-return-14817-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 388F14C6A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 18:22:28 +0000 (UTC)
Received: (qmail 6598 invoked by uid 500); 13 May 2011 18:22:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6552 invoked by uid 500); 13 May 2011 18:22:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6448 invoked by uid 99); 13 May 2011 18:22:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 18:22:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 18:22:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0325589D21
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 18:21:48 +0000 (UTC)
Date: Fri, 13 May 2011 18:21:48 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1127283378.10820.1305310908009.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <227442675.12305.1297477797626.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7138) Remove contrib build and test
 support
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins resolved HADOOP-7138.
---------------------------------

    Resolution: Duplicate

This change is covered by the change in HADOOP-7137.

> Remove contrib build and test support
> -------------------------------------
>
>                 Key: HADOOP-7138
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7138
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Nigel Daley
>             Fix For: 0.23.0
>
>
> With the removal of hod and failmon, these files are no longer needed:
> src/contrib/test/*
> src/contrib/build.xml
> src/contrib/build-contrib.xml
> In addition, the contrib support in the top level build.xml should be removed.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14818-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 18:22:48 2011
Return-Path: <common-issues-return-14818-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 69BC84CBD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 18:22:48 +0000 (UTC)
Received: (qmail 9436 invoked by uid 500); 13 May 2011 18:22:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9370 invoked by uid 500); 13 May 2011 18:22:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9362 invoked by uid 99); 13 May 2011 18:22:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 18:22:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 18:22:47 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1A13989D24
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 18:21:48 +0000 (UTC)
Date: Fri, 13 May 2011 18:21:48 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <83312554.10823.1305310908103.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <227442675.12305.1297477797626.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Closed] (HADOOP-7138) Remove contrib build and test support
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins closed HADOOP-7138.
-------------------------------


> Remove contrib build and test support
> -------------------------------------
>
>                 Key: HADOOP-7138
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7138
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Nigel Daley
>             Fix For: 0.23.0
>
>
> With the removal of hod and failmon, these files are no longer needed:
> src/contrib/test/*
> src/contrib/build.xml
> src/contrib/build-contrib.xml
> In addition, the contrib support in the top level build.xml should be removed.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14819-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 18:24:27 2011
Return-Path: <common-issues-return-14819-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 63A7F4CF1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 18:24:27 +0000 (UTC)
Received: (qmail 10671 invoked by uid 500); 13 May 2011 18:24:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10646 invoked by uid 500); 13 May 2011 18:24:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10638 invoked by uid 99); 13 May 2011 18:24:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 18:24:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 18:24:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 66AE689DF7
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 18:23:47 +0000 (UTC)
Date: Fri, 13 May 2011 18:23:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1780544293.10827.1305311027417.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1619049648.8641.1305244247668.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7286) Refactor FsShell's du/dus/df
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033176#comment-13033176 ] 

Aaron T. Myers commented on HADOOP-7286:
----------------------------------------

Patch looks pretty solid, Daryn. A few comments:

# I think the name "{{Capacity}}" isn't the best. At least in the case of, "{{du}}" and "{{dus}}", there's no capacity information being calculated - just usage info. I suggest renaming to something like "{{FsUsage}}". Thoughts?
# Since all sub-classes of {{o.a.h.fs.shell.Capacity}} need a {{TableBuilder}} object, might as well make this a protected instance variable of {{Capacity}} a la {{humanReadable}}.
# I've always found it silly that we continue to maintain both "{{du}}" and "{{dus}}". Perhaps now would be a good time to deprecate "{{dus}}" in favor of "{{du -s}}" ? Feel free to say this should be done as part of a separate JIRA.

> Refactor FsShell's du/dus/df
> ----------------------------
>
>                 Key: HADOOP-7286
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7286
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7286-2.patch, HADOOP-7286.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14820-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 18:30:29 2011
Return-Path: <common-issues-return-14820-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C22224058
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 18:30:29 +0000 (UTC)
Received: (qmail 18863 invoked by uid 500); 13 May 2011 18:30:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18768 invoked by uid 500); 13 May 2011 18:30:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18760 invoked by uid 99); 13 May 2011 18:30:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 18:30:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 18:30:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9D7CB89FF7
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 18:29:47 +0000 (UTC)
Date: Fri, 13 May 2011 18:29:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1327799579.10843.1305311387641.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1619049648.8641.1305244247668.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7286) Refactor FsShell's du/dus/df
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033182#comment-13033182 ] 

Daryn Sharp commented on HADOOP-7286:
-------------------------------------

All good ideas!  I too dislike all the commands like du/dus that are just permutations of each other, and would love to get rid of them.  How do you propose "dus" is deprecated?  Writing something to stderr?  Or just get rid of it?  I'd favor the latter unless there are objections.

> Refactor FsShell's du/dus/df
> ----------------------------
>
>                 Key: HADOOP-7286
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7286
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7286-2.patch, HADOOP-7286.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14821-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 18:48:29 2011
Return-Path: <common-issues-return-14821-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7F69E4909
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 18:48:29 +0000 (UTC)
Received: (qmail 54346 invoked by uid 500); 13 May 2011 18:48:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54305 invoked by uid 500); 13 May 2011 18:48:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54295 invoked by uid 99); 13 May 2011 18:48:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 18:48:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 18:48:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 562C1897D7
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 18:47:47 +0000 (UTC)
Date: Fri, 13 May 2011 18:47:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <758789398.10913.1305312467349.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32042405.8638.1305244247592.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7285) Refactor FsShell's test
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033199#comment-13033199 ] 

Todd Lipcon commented on HADOOP-7285:
-------------------------------------

+1, looks good to me.

> Refactor FsShell's test
> -----------------------
>
>                 Key: HADOOP-7285
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7285
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7285-2.patch, HADOOP-7285-3.patch, HADOOP-7285.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14822-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 18:50:29 2011
Return-Path: <common-issues-return-14822-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BA2CF4D60
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 18:50:29 +0000 (UTC)
Received: (qmail 57026 invoked by uid 500); 13 May 2011 18:50:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56990 invoked by uid 500); 13 May 2011 18:50:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56982 invoked by uid 99); 13 May 2011 18:50:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 18:50:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 18:50:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A35258990A
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 18:49:47 +0000 (UTC)
Date: Fri, 13 May 2011 18:49:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1262953664.10926.1305312587664.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32042405.8638.1305244247592.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7285) Refactor FsShell's test
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7285:
--------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks, Daryn!

Have we been considering these changes to be incompatible changes, since they change the error output?

> Refactor FsShell's test
> -----------------------
>
>                 Key: HADOOP-7285
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7285
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7285-2.patch, HADOOP-7285-3.patch, HADOOP-7285.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14823-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 18:54:30 2011
Return-Path: <common-issues-return-14823-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 658404E89
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 18:54:30 +0000 (UTC)
Received: (qmail 62925 invoked by uid 500); 13 May 2011 18:54:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62868 invoked by uid 500); 13 May 2011 18:54:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62653 invoked by uid 99); 13 May 2011 18:54:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 18:54:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 18:54:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B3ECA89B43
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 18:53:47 +0000 (UTC)
Date: Fri, 13 May 2011 18:53:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1809166531.10948.1305312827733.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1619049648.8641.1305244247668.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7286) Refactor FsShell's du/dus/df
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033208#comment-13033208 ] 

Todd Lipcon commented on HADOOP-7286:
-------------------------------------

I think we need to leave "dus" for at least one stable version. IMO it should print something like:

WARNING: The "dus" command has been deprecated. Please use "du -s" instead.   <-- on stderr
12345 file:/foo   <-- on stdout



> Refactor FsShell's du/dus/df
> ----------------------------
>
>                 Key: HADOOP-7286
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7286
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7286-2.patch, HADOOP-7286.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14824-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 18:54:32 2011
Return-Path: <common-issues-return-14824-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0736C4EAE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 18:54:32 +0000 (UTC)
Received: (qmail 64455 invoked by uid 500); 13 May 2011 18:54:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64418 invoked by uid 500); 13 May 2011 18:54:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64376 invoked by uid 99); 13 May 2011 18:54:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 18:54:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 18:54:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6E49889B39
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 18:53:47 +0000 (UTC)
Date: Fri, 13 May 2011 18:53:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <539645178.10940.1305312827448.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7289) ivy: test conf should not extend
 common conf
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

ivy: test conf should not extend common conf
--------------------------------------------

                 Key: HADOOP-7289
                 URL: https://issues.apache.org/jira/browse/HADOOP-7289
             Project: Hadoop Common
          Issue Type: Improvement
          Components: build
            Reporter: Tsz Wo (Nicholas), SZE
            Assignee: Tsz Wo (Nicholas), SZE


Otherwise, the same jars will appear in both {{build/ivy/lib/Hadoop-Common/common/}} and {{build/ivy/lib/Hadoop-Common/test/}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14825-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 18:56:29 2011
Return-Path: <common-issues-return-14825-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 89D544F25
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 18:56:29 +0000 (UTC)
Received: (qmail 69338 invoked by uid 500); 13 May 2011 18:56:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69313 invoked by uid 500); 13 May 2011 18:56:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69305 invoked by uid 99); 13 May 2011 18:56:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 18:56:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 18:56:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6D71C89C5A
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 18:55:47 +0000 (UTC)
Date: Fri, 13 May 2011 18:55:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <970560271.10954.1305312947445.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <539645178.10940.1305312827448.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7289) ivy: test conf should not extend
 common conf
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7289:
-------------------------------------------

    Attachment: c7289_20110513.patch

> ivy: test conf should not extend common conf
> --------------------------------------------
>
>                 Key: HADOOP-7289
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7289
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c7289_20110513.patch
>
>
> Otherwise, the same jars will appear in both {{build/ivy/lib/Hadoop-Common/common/}} and {{build/ivy/lib/Hadoop-Common/test/}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14826-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 18:56:30 2011
Return-Path: <common-issues-return-14826-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D05534F3B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 18:56:30 +0000 (UTC)
Received: (qmail 69732 invoked by uid 500); 13 May 2011 18:56:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69668 invoked by uid 500); 13 May 2011 18:56:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69496 invoked by uid 99); 13 May 2011 18:56:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 18:56:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 18:56:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7695D89C5B
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 18:55:47 +0000 (UTC)
Date: Fri, 13 May 2011 18:55:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2068175732.10955.1305312947482.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <539645178.10940.1305312827448.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7289) ivy: test conf should not extend
 common conf
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033211#comment-13033211 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7289:
------------------------------------------------

c7289_20110513.patch: removed the extension and changed mockito to use test conf.

> ivy: test conf should not extend common conf
> --------------------------------------------
>
>                 Key: HADOOP-7289
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7289
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c7289_20110513.patch
>
>
> Otherwise, the same jars will appear in both {{build/ivy/lib/Hadoop-Common/common/}} and {{build/ivy/lib/Hadoop-Common/test/}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14827-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 18:58:27 2011
Return-Path: <common-issues-return-14827-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DFCD1401A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 18:58:27 +0000 (UTC)
Received: (qmail 74173 invoked by uid 500); 13 May 2011 18:58:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74146 invoked by uid 500); 13 May 2011 18:58:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74117 invoked by uid 99); 13 May 2011 18:58:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 18:58:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 18:58:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7ED9889D4B
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 18:57:47 +0000 (UTC)
Date: Fri, 13 May 2011 18:57:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1395249415.10960.1305313067515.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <539645178.10940.1305312827448.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7289) ivy: test conf should not extend
 common conf
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7289:
-------------------------------------------

    Status: Patch Available  (was: Open)

> ivy: test conf should not extend common conf
> --------------------------------------------
>
>                 Key: HADOOP-7289
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7289
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c7289_20110513.patch
>
>
> Otherwise, the same jars will appear in both {{build/ivy/lib/Hadoop-Common/common/}} and {{build/ivy/lib/Hadoop-Common/test/}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14828-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 19:00:30 2011
Return-Path: <common-issues-return-14828-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F0E244332
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 19:00:29 +0000 (UTC)
Received: (qmail 79497 invoked by uid 500); 13 May 2011 19:00:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79441 invoked by uid 500); 13 May 2011 19:00:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79433 invoked by uid 99); 13 May 2011 19:00:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 19:00:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 19:00:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8D23389E26
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 18:59:47 +0000 (UTC)
Date: Fri, 13 May 2011 18:59:47 +0000 (UTC)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2002540982.10966.1305313187574.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1894387424.4008.1305137809491.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7278) Automatic Hadoop cluster deployment
 for build validation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-7278:
---------------------------------------

    Attachment: HADOOP-7278.patch

The patch adds a set of simple configs sufficient to run a 0.22 cluster on a few nodes.
Another part of the patch is a deployment script to be driven by a Hudson job.
This setup works [here|https://builds.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-22-cluster-deploy/]. 
Once a cluster is deployed a set of workloads will be executed to check the [viability of the cluster|https://builds.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-22-cluster-test/].

This is not a part of common source tree strictly speaking. Would it be better to have a separate build_and_deploy repository at the same level of Hadoop component are checked-in into SVN? Thoughts?

> Automatic Hadoop cluster deployment for build validation
> --------------------------------------------------------
>
>                 Key: HADOOP-7278
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7278
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0, 0.23.0
>         Environment: Apache Jenkins
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>              Labels: newbie
>         Attachments: HADOOP-7278.patch
>
>
> It'd be great to have a way of automatically deploying Hadoop cluster to a set of machine once all components are successfully built (in the form or tar or whatever). Deployed cluster then can be used to run a set of validation jobs to make sure that produced artifacts are viable. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14829-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 19:10:29 2011
Return-Path: <common-issues-return-14829-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C552E48BC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 19:10:29 +0000 (UTC)
Received: (qmail 798 invoked by uid 500); 13 May 2011 19:10:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 770 invoked by uid 500); 13 May 2011 19:10:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 762 invoked by uid 99); 13 May 2011 19:10:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 19:10:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 19:10:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A3535891E3
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 19:09:47 +0000 (UTC)
Date: Fri, 13 May 2011 19:09:47 +0000 (UTC)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <147831449.11007.1305313787665.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <88464990.3250.1305126467497.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7274) CLONE - IOUtils.readFully and
 IOUtils.skipFully have typo in exception creation's message
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-7274:
----------------------------------

       Resolution: Fixed
    Fix Version/s: 0.20.205.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

+1

I committed this. Thanks, Jon!

> CLONE - IOUtils.readFully and IOUtils.skipFully have typo in exception creation's message
> -----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7274
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7274
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.20.204.0
>            Reporter: Jonathan Eagles
>            Assignee: Jonathan Eagles
>            Priority: Minor
>             Fix For: 0.20.205.0
>
>         Attachments: HADOOP-7274-0.20-security.patch
>
>
> Same fix as for HADOOP-7057 for the Hadoop security branch
> {noformat}
>         throw new IOException( "Premeture EOF from inputStream");
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14830-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 19:14:30 2011
Return-Path: <common-issues-return-14830-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6446B4996
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 19:14:30 +0000 (UTC)
Received: (qmail 7856 invoked by uid 500); 13 May 2011 19:14:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7781 invoked by uid 500); 13 May 2011 19:14:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7766 invoked by uid 99); 13 May 2011 19:14:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 19:14:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 19:14:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A251789371
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 19:13:47 +0000 (UTC)
Date: Fri, 13 May 2011 19:13:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1946818191.11013.1305314027661.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <539645178.10940.1305312827448.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7289) ivy: test conf should not extend
 common conf
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033224#comment-13033224 ] 

Hadoop QA commented on HADOOP-7289:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479154/c7289_20110513.patch
  against trunk revision 1102861.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.http.TestGlobalFilter
                  org.apache.hadoop.http.TestHttpServerLifecycle
                  org.apache.hadoop.http.TestHttpServerWebapps
                  org.apache.hadoop.http.TestHttpServer
                  org.apache.hadoop.http.TestServletFilter
                  org.apache.hadoop.ipc.TestAvroRpc
                  org.apache.hadoop.log.TestLogLevel

    -1 contrib tests.  The patch failed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/449//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/449//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/449//console

This message is automatically generated.

> ivy: test conf should not extend common conf
> --------------------------------------------
>
>                 Key: HADOOP-7289
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7289
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c7289_20110513.patch
>
>
> Otherwise, the same jars will appear in both {{build/ivy/lib/Hadoop-Common/common/}} and {{build/ivy/lib/Hadoop-Common/test/}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14831-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 19:36:27 2011
Return-Path: <common-issues-return-14831-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E274C475D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 19:36:27 +0000 (UTC)
Received: (qmail 37802 invoked by uid 500); 13 May 2011 19:36:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37775 invoked by uid 500); 13 May 2011 19:36:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37766 invoked by uid 99); 13 May 2011 19:36:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 19:36:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 19:36:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9160D89B69
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 19:35:47 +0000 (UTC)
Date: Fri, 13 May 2011 19:35:47 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <124474506.11064.1305315347592.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <539645178.10940.1305312827448.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7289) ivy: test conf should not extend
 common conf
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033238#comment-13033238 ] 

Eric Yang commented on HADOOP-7289:
-----------------------------------

+1 looks good.

> ivy: test conf should not extend common conf
> --------------------------------------------
>
>                 Key: HADOOP-7289
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7289
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c7289_20110513.patch
>
>
> Otherwise, the same jars will appear in both {{build/ivy/lib/Hadoop-Common/common/}} and {{build/ivy/lib/Hadoop-Common/test/}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14832-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 19:40:29 2011
Return-Path: <common-issues-return-14832-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DD0F24979
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 19:40:28 +0000 (UTC)
Received: (qmail 46285 invoked by uid 500); 13 May 2011 19:40:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46229 invoked by uid 500); 13 May 2011 19:40:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45984 invoked by uid 99); 13 May 2011 19:40:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 19:40:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 19:40:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 61E0689CD6
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 19:39:47 +0000 (UTC)
Date: Fri, 13 May 2011 19:39:47 +0000 (UTC)
From: "Trevor Robinson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <177713183.11069.1305315587397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7290) Unit test failure in
 TestUserGroupInformation.testGetServerSideGroups
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Unit test failure in TestUserGroupInformation.testGetServerSideGroups
---------------------------------------------------------------------

                 Key: HADOOP-7290
                 URL: https://issues.apache.org/jira/browse/HADOOP-7290
             Project: Hadoop Common
          Issue Type: Bug
          Components: test
    Affects Versions: 0.23.0
         Environment: Linux 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:24 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
            Reporter: Trevor Robinson
            Priority: Minor


Testsuite: org.apache.hadoop.security.TestUserGroupInformation
Tests run: 14, Failures: 1, Errors: 0, Time elapsed: 0.278 sec
------------- Standard Output ---------------
trobinson:users guest git
------------- ---------------- ---------------

Testcase: testGetServerSideGroups took 0.051 sec
       FAILED
expected:<g[ues]t> but was:<g[i]t>
junit.framework.AssertionFailedError: expected:<g[ues]t> but was:<g[i]t>
       at org.apache.hadoop.security.TestUserGroupInformation.testGetServerSideGroups(TestUserGroupInformation.java:94)

It seems like the test is assuming that the groups returned by UserGroupInformation.getGroupNames() are in the same order as those returned by executing `id -Gn`. getGroupNames() only documents that the primary group is first, and `man id` doesn't document any ordering, so it seems like the test needs to be reworked to remove that assumption.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14833-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 19:50:30 2011
Return-Path: <common-issues-return-14833-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 11D334FCE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 19:50:30 +0000 (UTC)
Received: (qmail 57957 invoked by uid 500); 13 May 2011 19:50:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57903 invoked by uid 500); 13 May 2011 19:50:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57889 invoked by uid 99); 13 May 2011 19:50:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 19:50:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 19:50:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7B3C289022
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 19:49:47 +0000 (UTC)
Date: Fri, 13 May 2011 19:49:47 +0000 (UTC)
From: "Trevor Robinson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <592767074.11075.1305316187501.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <177713183.11069.1305315587397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7290) Unit test failure in
 TestUserGroupInformation.testGetServerSideGroups
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Trevor Robinson updated HADOOP-7290:
------------------------------------

    Status: Patch Available  (was: Open)

> Unit test failure in TestUserGroupInformation.testGetServerSideGroups
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-7290
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7290
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>         Environment: Linux 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:24 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
>            Reporter: Trevor Robinson
>            Priority: Minor
>              Labels: test
>         Attachments: TestUserGroupInformation-id-order.patch
>
>
> Testsuite: org.apache.hadoop.security.TestUserGroupInformation
> Tests run: 14, Failures: 1, Errors: 0, Time elapsed: 0.278 sec
> ------------- Standard Output ---------------
> trobinson:users guest git
> ------------- ---------------- ---------------
> Testcase: testGetServerSideGroups took 0.051 sec
>        FAILED
> expected:<g[ues]t> but was:<g[i]t>
> junit.framework.AssertionFailedError: expected:<g[ues]t> but was:<g[i]t>
>        at org.apache.hadoop.security.TestUserGroupInformation.testGetServerSideGroups(TestUserGroupInformation.java:94)
> It seems like the test is assuming that the groups returned by UserGroupInformation.getGroupNames() are in the same order as those returned by executing `id -Gn`. getGroupNames() only documents that the primary group is first, and `man id` doesn't document any ordering, so it seems like the test needs to be reworked to remove that assumption.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14834-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 19:50:31 2011
Return-Path: <common-issues-return-14834-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2CC884FDB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 19:50:30 +0000 (UTC)
Received: (qmail 57970 invoked by uid 500); 13 May 2011 19:50:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57906 invoked by uid 500); 13 May 2011 19:50:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57888 invoked by uid 99); 13 May 2011 19:50:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 19:50:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 19:50:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 675E389020
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 19:49:47 +0000 (UTC)
Date: Fri, 13 May 2011 19:49:47 +0000 (UTC)
From: "Trevor Robinson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1724631275.11073.1305316187420.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <177713183.11069.1305315587397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7290) Unit test failure in
 TestUserGroupInformation.testGetServerSideGroups
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Trevor Robinson updated HADOOP-7290:
------------------------------------

    Attachment: TestUserGroupInformation-id-order.patch

Patch puts `id` output in LinkedHashSet instead of ArrayList, then tests against getGroupNames() for size and containment of all values, rather than comparing corresponding array elements.

> Unit test failure in TestUserGroupInformation.testGetServerSideGroups
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-7290
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7290
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>         Environment: Linux 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:24 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
>            Reporter: Trevor Robinson
>            Priority: Minor
>              Labels: test
>         Attachments: TestUserGroupInformation-id-order.patch
>
>
> Testsuite: org.apache.hadoop.security.TestUserGroupInformation
> Tests run: 14, Failures: 1, Errors: 0, Time elapsed: 0.278 sec
> ------------- Standard Output ---------------
> trobinson:users guest git
> ------------- ---------------- ---------------
> Testcase: testGetServerSideGroups took 0.051 sec
>        FAILED
> expected:<g[ues]t> but was:<g[i]t>
> junit.framework.AssertionFailedError: expected:<g[ues]t> but was:<g[i]t>
>        at org.apache.hadoop.security.TestUserGroupInformation.testGetServerSideGroups(TestUserGroupInformation.java:94)
> It seems like the test is assuming that the groups returned by UserGroupInformation.getGroupNames() are in the same order as those returned by executing `id -Gn`. getGroupNames() only documents that the primary group is first, and `man id` doesn't document any ordering, so it seems like the test needs to be reworked to remove that assumption.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14835-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 20:04:28 2011
Return-Path: <common-issues-return-14835-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 223224738
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 20:04:28 +0000 (UTC)
Received: (qmail 75464 invoked by uid 500); 13 May 2011 20:04:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75417 invoked by uid 500); 13 May 2011 20:04:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75328 invoked by uid 99); 13 May 2011 20:04:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 20:04:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 20:04:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 96E5589471
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 20:03:47 +0000 (UTC)
Date: Fri, 13 May 2011 20:03:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1666903543.11117.1305317027614.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <177713183.11069.1305315587397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7290) Unit test failure in
 TestUserGroupInformation.testGetServerSideGroups
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033252#comment-13033252 ] 

Hadoop QA commented on HADOOP-7290:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479160/TestUserGroupInformation-id-order.patch
  against trunk revision 1102861.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    -1 contrib tests.  The patch failed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/450//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/450//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/450//console

This message is automatically generated.

> Unit test failure in TestUserGroupInformation.testGetServerSideGroups
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-7290
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7290
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>         Environment: Linux 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:24 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
>            Reporter: Trevor Robinson
>            Priority: Minor
>              Labels: test
>         Attachments: TestUserGroupInformation-id-order.patch
>
>
> Testsuite: org.apache.hadoop.security.TestUserGroupInformation
> Tests run: 14, Failures: 1, Errors: 0, Time elapsed: 0.278 sec
> ------------- Standard Output ---------------
> trobinson:users guest git
> ------------- ---------------- ---------------
> Testcase: testGetServerSideGroups took 0.051 sec
>        FAILED
> expected:<g[ues]t> but was:<g[i]t>
> junit.framework.AssertionFailedError: expected:<g[ues]t> but was:<g[i]t>
>        at org.apache.hadoop.security.TestUserGroupInformation.testGetServerSideGroups(TestUserGroupInformation.java:94)
> It seems like the test is assuming that the groups returned by UserGroupInformation.getGroupNames() are in the same order as those returned by executing `id -Gn`. getGroupNames() only documents that the primary group is first, and `man id` doesn't document any ordering, so it seems like the test needs to be reworked to remove that assumption.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14836-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 20:18:27 2011
Return-Path: <common-issues-return-14836-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D8705492B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 20:18:27 +0000 (UTC)
Received: (qmail 96600 invoked by uid 500); 13 May 2011 20:18:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96570 invoked by uid 500); 13 May 2011 20:18:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96562 invoked by uid 99); 13 May 2011 20:18:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 20:18:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 20:18:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 682068997B
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 20:17:47 +0000 (UTC)
Date: Fri, 13 May 2011 20:17:47 +0000 (UTC)
From: "Trevor Robinson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1556508098.11151.1305317867423.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <177713183.11069.1305315587397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7290) Unit test failure in
 TestUserGroupInformation.testGetServerSideGroups
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033257#comment-13033257 ] 

Trevor Robinson commented on HADOOP-7290:
-----------------------------------------

Contrib tests unchanged. Trunk/PreCommit configuration broken by revision 1102848?

     [exec] ======================================================================
     [exec] ======================================================================
     [exec]     Running contrib tests.
     [exec] ======================================================================
     [exec] ======================================================================
     [exec] 
     [exec] 
     [exec] /bin/kill -9 12446 
     [exec] kill: No such process
     [exec] /homes/hudson/tools/ant/latest/bin/ant -Dversion=1102861_HADOOP-7290_PATCH-12479160 -Declipse.home=/homes/hudson/tools/eclipse/latest -Dpython.home=/homes/hudson/tools/python/latest -DHadoopPatchProcess= -Dtest.junit.output.format=xml -Dtest.output=no test-contrib
     [exec] Buildfile: build.xml
     [exec] 
     [exec] BUILD FAILED
     [exec] Target "test-contrib" does not exist in the project "Hadoop-Common". 
     [exec] 
     [exec] Total time: 0 seconds

> Unit test failure in TestUserGroupInformation.testGetServerSideGroups
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-7290
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7290
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>         Environment: Linux 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:24 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
>            Reporter: Trevor Robinson
>            Priority: Minor
>              Labels: test
>         Attachments: TestUserGroupInformation-id-order.patch
>
>
> Testsuite: org.apache.hadoop.security.TestUserGroupInformation
> Tests run: 14, Failures: 1, Errors: 0, Time elapsed: 0.278 sec
> ------------- Standard Output ---------------
> trobinson:users guest git
> ------------- ---------------- ---------------
> Testcase: testGetServerSideGroups took 0.051 sec
>        FAILED
> expected:<g[ues]t> but was:<g[i]t>
> junit.framework.AssertionFailedError: expected:<g[ues]t> but was:<g[i]t>
>        at org.apache.hadoop.security.TestUserGroupInformation.testGetServerSideGroups(TestUserGroupInformation.java:94)
> It seems like the test is assuming that the groups returned by UserGroupInformation.getGroupNames() are in the same order as those returned by executing `id -Gn`. getGroupNames() only documents that the primary group is first, and `man id` doesn't document any ordering, so it seems like the test needs to be reworked to remove that assumption.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14837-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 20:20:27 2011
Return-Path: <common-issues-return-14837-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 94182417D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 20:20:27 +0000 (UTC)
Received: (qmail 97545 invoked by uid 500); 13 May 2011 20:20:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97518 invoked by uid 500); 13 May 2011 20:20:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97509 invoked by uid 99); 13 May 2011 20:20:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 20:20:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 20:20:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5A51C899FC
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 20:19:47 +0000 (UTC)
Date: Fri, 13 May 2011 20:19:47 +0000 (UTC)
From: "Trevor Robinson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1648895671.11166.1305317987366.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <957076305.12265.1297477317500.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7137) Remove hod contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033259#comment-13033259 ] 

Trevor Robinson commented on HADOOP-7137:
-----------------------------------------

This commit removed test-contrib which is run by Hudson in PreCommit-HADOOP-Build:

     [exec] ======================================================================
     [exec] ======================================================================
     [exec]     Running contrib tests.
     [exec] ======================================================================
     [exec] ======================================================================
     [exec] 
     [exec] 
     [exec] /bin/kill -9 12446 
     [exec] kill: No such process
     [exec] /homes/hudson/tools/ant/latest/bin/ant -Dversion=1102861_HADOOP-7290_PATCH-12479160 -Declipse.home=/homes/hudson/tools/eclipse/latest -Dpython.home=/homes/hudson/tools/python/latest -DHadoopPatchProcess= -Dtest.junit.output.format=xml -Dtest.output=no test-contrib
     [exec] Buildfile: build.xml
     [exec] 
     [exec] BUILD FAILED
     [exec] Target "test-contrib" does not exist in the project "Hadoop-Common". 
     [exec] 
     [exec] Total time: 0 seconds

> Remove hod contrib
> ------------------
>
>                 Key: HADOOP-7137
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7137
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7137.patch, HADOOP-7137.patch
>
>
> As per vote on general@ (http://mail-archives.apache.org/mod_mbox/hadoop-general/201102.mbox/%3CAC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com%3E) I will 
> svn remove common/trunk/src/contrib/hod
> using this Jira.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14838-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 20:40:31 2011
Return-Path: <common-issues-return-14838-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1B6A84B25
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 20:40:31 +0000 (UTC)
Received: (qmail 22322 invoked by uid 500); 13 May 2011 20:40:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22256 invoked by uid 500); 13 May 2011 20:40:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22240 invoked by uid 99); 13 May 2011 20:40:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 20:40:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 20:40:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5DA6C890F3
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 20:39:47 +0000 (UTC)
Date: Fri, 13 May 2011 20:39:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Update Hudson job not to run test-contrib
-----------------------------------------

                 Key: HADOOP-7291
                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
             Project: Hadoop Common
          Issue Type: Task
            Reporter: Eli Collins


The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14839-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 20:40:31 2011
Return-Path: <common-issues-return-14839-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6ECE04B50
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 20:40:31 +0000 (UTC)
Received: (qmail 22613 invoked by uid 500); 13 May 2011 20:40:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22556 invoked by uid 500); 13 May 2011 20:40:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22371 invoked by uid 99); 13 May 2011 20:40:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 20:40:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 20:40:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7AFDA890F8
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 20:39:47 +0000 (UTC)
Date: Fri, 13 May 2011 20:39:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <558693112.11211.1305319187500.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <177713183.11069.1305315587397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7290) Unit test failure in
 TestUserGroupInformation.testGetServerSideGroups
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033268#comment-13033268 ] 

Eli Collins commented on HADOOP-7290:
-------------------------------------

+1   lgtm

On my system the ordering happens to line up, but you're right, there's no reason it should and it looks like it doesn't on your system.

The Hudson error is HADOOP-7291.


> Unit test failure in TestUserGroupInformation.testGetServerSideGroups
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-7290
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7290
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.23.0
>         Environment: Linux 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:24 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
>            Reporter: Trevor Robinson
>            Priority: Minor
>              Labels: test
>         Attachments: TestUserGroupInformation-id-order.patch
>
>
> Testsuite: org.apache.hadoop.security.TestUserGroupInformation
> Tests run: 14, Failures: 1, Errors: 0, Time elapsed: 0.278 sec
> ------------- Standard Output ---------------
> trobinson:users guest git
> ------------- ---------------- ---------------
> Testcase: testGetServerSideGroups took 0.051 sec
>        FAILED
> expected:<g[ues]t> but was:<g[i]t>
> junit.framework.AssertionFailedError: expected:<g[ues]t> but was:<g[i]t>
>        at org.apache.hadoop.security.TestUserGroupInformation.testGetServerSideGroups(TestUserGroupInformation.java:94)
> It seems like the test is assuming that the groups returned by UserGroupInformation.getGroupNames() are in the same order as those returned by executing `id -Gn`. getGroupNames() only documents that the primary group is first, and `man id` doesn't document any ordering, so it seems like the test needs to be reworked to remove that assumption.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14840-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 20:44:29 2011
Return-Path: <common-issues-return-14840-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1E45E4C37
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 20:44:29 +0000 (UTC)
Received: (qmail 30845 invoked by uid 500); 13 May 2011 20:44:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30798 invoked by uid 500); 13 May 2011 20:44:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30542 invoked by uid 99); 13 May 2011 20:44:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 20:44:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 20:44:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A41758932F
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 20:43:47 +0000 (UTC)
Date: Fri, 13 May 2011 20:43:47 +0000 (UTC)
From: "Trevor Robinson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2137725310.11224.1305319427668.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <177713183.11069.1305315587397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7290) Unit test failure in
 TestUserGroupInformation.testGetServerSideGroups
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Trevor Robinson updated HADOOP-7290:
------------------------------------

    Component/s:     (was: test)
                 security

> Unit test failure in TestUserGroupInformation.testGetServerSideGroups
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-7290
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7290
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.23.0
>         Environment: Linux 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:24 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
>            Reporter: Trevor Robinson
>            Priority: Minor
>              Labels: test
>         Attachments: TestUserGroupInformation-id-order.patch
>
>
> Testsuite: org.apache.hadoop.security.TestUserGroupInformation
> Tests run: 14, Failures: 1, Errors: 0, Time elapsed: 0.278 sec
> ------------- Standard Output ---------------
> trobinson:users guest git
> ------------- ---------------- ---------------
> Testcase: testGetServerSideGroups took 0.051 sec
>        FAILED
> expected:<g[ues]t> but was:<g[i]t>
> junit.framework.AssertionFailedError: expected:<g[ues]t> but was:<g[i]t>
>        at org.apache.hadoop.security.TestUserGroupInformation.testGetServerSideGroups(TestUserGroupInformation.java:94)
> It seems like the test is assuming that the groups returned by UserGroupInformation.getGroupNames() are in the same order as those returned by executing `id -Gn`. getGroupNames() only documents that the primary group is first, and `man id` doesn't document any ordering, so it seems like the test needs to be reworked to remove that assumption.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14841-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 20:44:30 2011
Return-Path: <common-issues-return-14841-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6245B4C50
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 20:44:30 +0000 (UTC)
Received: (qmail 32439 invoked by uid 500); 13 May 2011 20:44:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32406 invoked by uid 500); 13 May 2011 20:44:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32398 invoked by uid 99); 13 May 2011 20:44:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 20:44:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 20:44:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E66E28933B
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 20:43:47 +0000 (UTC)
Date: Fri, 13 May 2011 20:43:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <887589582.11232.1305319427940.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <177713183.11069.1305315587397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7290) Unit test failure in
 TestUserGroupInformation.testGetServerSideGroups
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins reassigned HADOOP-7290:
-----------------------------------

    Assignee: Trevor Robinson

> Unit test failure in TestUserGroupInformation.testGetServerSideGroups
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-7290
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7290
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.23.0
>         Environment: Linux 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:24 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
>            Reporter: Trevor Robinson
>            Assignee: Trevor Robinson
>            Priority: Minor
>              Labels: test
>         Attachments: TestUserGroupInformation-id-order.patch
>
>
> Testsuite: org.apache.hadoop.security.TestUserGroupInformation
> Tests run: 14, Failures: 1, Errors: 0, Time elapsed: 0.278 sec
> ------------- Standard Output ---------------
> trobinson:users guest git
> ------------- ---------------- ---------------
> Testcase: testGetServerSideGroups took 0.051 sec
>        FAILED
> expected:<g[ues]t> but was:<g[i]t>
> junit.framework.AssertionFailedError: expected:<g[ues]t> but was:<g[i]t>
>        at org.apache.hadoop.security.TestUserGroupInformation.testGetServerSideGroups(TestUserGroupInformation.java:94)
> It seems like the test is assuming that the groups returned by UserGroupInformation.getGroupNames() are in the same order as those returned by executing `id -Gn`. getGroupNames() only documents that the primary group is first, and `man id` doesn't document any ordering, so it seems like the test needs to be reworked to remove that assumption.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14842-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 20:46:30 2011
Return-Path: <common-issues-return-14842-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F176E4870
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 20:46:29 +0000 (UTC)
Received: (qmail 35949 invoked by uid 500); 13 May 2011 20:46:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35914 invoked by uid 500); 13 May 2011 20:46:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35828 invoked by uid 99); 13 May 2011 20:46:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 20:46:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 20:46:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 726808944F
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 20:45:47 +0000 (UTC)
Date: Fri, 13 May 2011 20:45:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1450426724.11236.1305319547465.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <177713183.11069.1305315587397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7290) Unit test failure in
 TestUserGroupInformation.testGetServerSideGroups
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7290:
--------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I've committed this. Thanks Trevor!

> Unit test failure in TestUserGroupInformation.testGetServerSideGroups
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-7290
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7290
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.23.0
>         Environment: Linux 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:24 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
>            Reporter: Trevor Robinson
>            Assignee: Trevor Robinson
>            Priority: Minor
>              Labels: test
>         Attachments: TestUserGroupInformation-id-order.patch
>
>
> Testsuite: org.apache.hadoop.security.TestUserGroupInformation
> Tests run: 14, Failures: 1, Errors: 0, Time elapsed: 0.278 sec
> ------------- Standard Output ---------------
> trobinson:users guest git
> ------------- ---------------- ---------------
> Testcase: testGetServerSideGroups took 0.051 sec
>        FAILED
> expected:<g[ues]t> but was:<g[i]t>
> junit.framework.AssertionFailedError: expected:<g[ues]t> but was:<g[i]t>
>        at org.apache.hadoop.security.TestUserGroupInformation.testGetServerSideGroups(TestUserGroupInformation.java:94)
> It seems like the test is assuming that the groups returned by UserGroupInformation.getGroupNames() are in the same order as those returned by executing `id -Gn`. getGroupNames() only documents that the primary group is first, and `man id` doesn't document any ordering, so it seems like the test needs to be reworked to remove that assumption.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14843-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 21:00:33 2011
Return-Path: <common-issues-return-14843-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 48F07478D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 21:00:33 +0000 (UTC)
Received: (qmail 56128 invoked by uid 500); 13 May 2011 21:00:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56045 invoked by uid 500); 13 May 2011 21:00:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56027 invoked by uid 99); 13 May 2011 21:00:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 21:00:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 21:00:31 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E689F89A71
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 20:59:52 +0000 (UTC)
Date: Fri, 13 May 2011 20:59:52 +0000 (UTC)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1236379130.11304.1305320392940.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033283#comment-13033283 ] 

Nigel Daley commented on HADOOP-7291:
-------------------------------------

I've disabled further precommit builds until we fix this.  I had partial change in my workspace for this very issue which I why I wasn't yet committing 7137.  Sorry, should have noted that.

> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14844-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 21:00:34 2011
Return-Path: <common-issues-return-14844-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3034747AE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 21:00:34 +0000 (UTC)
Received: (qmail 56572 invoked by uid 500); 13 May 2011 21:00:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56523 invoked by uid 500); 13 May 2011 21:00:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56515 invoked by uid 99); 13 May 2011 21:00:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 21:00:33 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 21:00:32 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7B91089A83
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 20:59:53 +0000 (UTC)
Date: Fri, 13 May 2011 20:59:53 +0000 (UTC)
From: "John George (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1257258633.11321.1305320393502.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <903601160.6842.1305210828211.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7282) getRemoteIp could return null in
 cases where the call is ongoing but the ip went away.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

John George updated HADOOP-7282:
--------------------------------

    Status: Open  (was: Patch Available)

> getRemoteIp could return null in cases where the call is ongoing but the ip went away.
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7282
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7282
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: John George
>            Assignee: John George
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7282.patch, diffs.txt
>
>
> getRemoteIp gets the ip from socket instead of the stored ip in Connection object. Thus calls to this function could return null when a client disconnected, but the rpc call is still ongoing...

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14845-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 21:00:53 2011
Return-Path: <common-issues-return-14845-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DC0AD496A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 21:00:53 +0000 (UTC)
Received: (qmail 57547 invoked by uid 500); 13 May 2011 21:00:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57501 invoked by uid 500); 13 May 2011 21:00:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57493 invoked by uid 99); 13 May 2011 21:00:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 21:00:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 21:00:52 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8B5D289A85
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 20:59:53 +0000 (UTC)
Date: Fri, 13 May 2011 20:59:53 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1521966324.11323.1305320393567.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <539645178.10940.1305312827448.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7289) ivy: test conf should not extend
 common conf
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033286#comment-13033286 ] 

Todd Lipcon commented on HADOOP-7289:
-------------------------------------

Looks like those http test case failures might be related to this patch?

> ivy: test conf should not extend common conf
> --------------------------------------------
>
>                 Key: HADOOP-7289
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7289
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c7289_20110513.patch
>
>
> Otherwise, the same jars will appear in both {{build/ivy/lib/Hadoop-Common/common/}} and {{build/ivy/lib/Hadoop-Common/test/}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14846-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 21:08:27 2011
Return-Path: <common-issues-return-14846-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8664B4E43
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 21:08:27 +0000 (UTC)
Received: (qmail 80431 invoked by uid 500); 13 May 2011 21:08:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80403 invoked by uid 500); 13 May 2011 21:08:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80391 invoked by uid 99); 13 May 2011 21:08:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 21:08:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 21:08:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 56CC389E12
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 21:07:47 +0000 (UTC)
Date: Fri, 13 May 2011 21:07:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1683786290.11386.1305320867352.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033293#comment-13033293 ] 

Eli Collins commented on HADOOP-7291:
-------------------------------------

Apologies for missing this, I assumed test-contrib was being called by Hudson only via the dependency on the test target. 

> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14847-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 21:12:29 2011
Return-Path: <common-issues-return-14847-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 918D34F7E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 21:12:29 +0000 (UTC)
Received: (qmail 89460 invoked by uid 500); 13 May 2011 21:12:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89428 invoked by uid 500); 13 May 2011 21:12:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89420 invoked by uid 99); 13 May 2011 21:12:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 21:12:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 21:12:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5E1C989F95
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 21:11:47 +0000 (UTC)
Date: Fri, 13 May 2011 21:11:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <928094290.11414.1305321107382.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033298#comment-13033298 ] 

Eli Collins commented on HADOOP-7291:
-------------------------------------

Btw Nigel, if you can point me to the Hudson login I'm happy to update the job command.

> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14848-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 21:16:27 2011
Return-Path: <common-issues-return-14848-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C5322456C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 21:16:27 +0000 (UTC)
Received: (qmail 96186 invoked by uid 500); 13 May 2011 21:16:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96150 invoked by uid 500); 13 May 2011 21:16:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96142 invoked by uid 99); 13 May 2011 21:16:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 21:16:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 21:16:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 96FB289136
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 21:15:47 +0000 (UTC)
Date: Fri, 13 May 2011 21:15:47 +0000 (UTC)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <573823175.11447.1305321347615.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033303#comment-13033303 ] 

Nigel Daley commented on HADOOP-7291:
-------------------------------------

No, test-contrib is called directly in src/test/bin/test-patch.sh

I'll upload an untested patch in a sec.

> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14849-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 21:18:30 2011
Return-Path: <common-issues-return-14849-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 674024F43
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 21:18:30 +0000 (UTC)
Received: (qmail 2199 invoked by uid 500); 13 May 2011 21:18:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2171 invoked by uid 500); 13 May 2011 21:18:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2163 invoked by uid 99); 13 May 2011 21:18:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 21:18:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 21:18:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1897389242
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 21:17:48 +0000 (UTC)
Date: Fri, 13 May 2011 21:17:48 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1578264414.11468.1305321468096.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <539645178.10940.1305312827448.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7289) ivy: test conf should not extend
 common conf
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7289:
-------------------------------------------

    Attachment: c7289_20110513b.patch

The failed tests require slf4j.

c7289_20110513b.patch: fixed also slf4j and junit.

> ivy: test conf should not extend common conf
> --------------------------------------------
>
>                 Key: HADOOP-7289
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7289
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c7289_20110513.patch, c7289_20110513b.patch
>
>
> Otherwise, the same jars will appear in both {{build/ivy/lib/Hadoop-Common/common/}} and {{build/ivy/lib/Hadoop-Common/test/}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14850-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 21:20:27 2011
Return-Path: <common-issues-return-14850-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 95E8746E5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 21:20:27 +0000 (UTC)
Received: (qmail 3424 invoked by uid 500); 13 May 2011 21:20:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3388 invoked by uid 500); 13 May 2011 21:20:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3380 invoked by uid 99); 13 May 2011 21:20:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 21:20:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 21:20:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6989989310
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 21:19:47 +0000 (UTC)
Date: Fri, 13 May 2011 21:19:47 +0000 (UTC)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2024491831.11473.1305321587428.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nigel Daley updated HADOOP-7291:
--------------------------------

    Attachment: HADOOP-7291.patch

This patch includes removing Python dependency since HOD is gone.  It also conditionally run test-contrib if the project has a test-contrib target in build.xml

Remember that the test-patch.sh script is pulled from common into hdfs and mapreduce using svn:externals


> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>         Attachments: HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14851-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 21:27:27 2011
Return-Path: <common-issues-return-14851-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A023D4847
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 21:27:27 +0000 (UTC)
Received: (qmail 16586 invoked by uid 500); 13 May 2011 21:27:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16562 invoked by uid 500); 13 May 2011 21:27:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16554 invoked by uid 99); 13 May 2011 21:27:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 21:27:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 21:27:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 73F5889557
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 21:26:47 +0000 (UTC)
Date: Fri, 13 May 2011 21:26:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1237237551.11495.1305322007471.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033310#comment-13033310 ] 

Eli Collins commented on HADOOP-7291:
-------------------------------------

Ah.   +1  patch lgtm

> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>         Attachments: HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14852-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 21:43:28 2011
Return-Path: <common-issues-return-14852-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 059D940C4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 21:43:28 +0000 (UTC)
Received: (qmail 41249 invoked by uid 500); 13 May 2011 21:43:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41212 invoked by uid 500); 13 May 2011 21:43:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41204 invoked by uid 99); 13 May 2011 21:43:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 21:43:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 21:43:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 588B989BE9
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 21:42:47 +0000 (UTC)
Date: Fri, 13 May 2011 21:42:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1227711425.11580.1305322967359.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7269) S3 Native should allow
 customizable file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033323#comment-13033323 ] 

Todd Lipcon commented on HADOOP-7269:
-------------------------------------

This opens up scope creep that makes me a little bit nervous. If we add the ability to set metadata in create(), then we probably also need the ability to query metadata, modify metadata on existing files, etc, which is all of a sudden a lot of code.

There's a trade-off between least-common-denominator APIs (sucks for everyone) vs support-all-FS-features (sucks for compatibility). I think this might be going too far towards the latter.

Would it make sense to add these functions directly to the S3 filesystems but not to the overall API? Then if a user really needs to code specifically against S3, they can down-cast to S3FS and do what they like.

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: HADOOP-7269-S3-metadata-001.diff, HADOOP-7269-S3-metadata-002.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14853-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 21:51:30 2011
Return-Path: <common-issues-return-14853-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 819324749
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 21:51:30 +0000 (UTC)
Received: (qmail 48438 invoked by uid 500); 13 May 2011 21:51:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48405 invoked by uid 500); 13 May 2011 21:51:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48397 invoked by uid 99); 13 May 2011 21:51:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 21:51:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 21:51:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 91DAB89E9D
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 21:50:47 +0000 (UTC)
Date: Fri, 13 May 2011 21:50:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1500579287.11603.1305323447594.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-1117) DFS Scalability: When the namenode
 is restarted it consumes 80% CPU
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033328#comment-13033328 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-1117:
------------------------------------------------

Why not wait for Hudson/Jenkins?

> DFS Scalability: When the namenode is restarted it consumes 80% CPU
> -------------------------------------------------------------------
>
>                 Key: HADOOP-1117
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1117
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.12.0
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>            Priority: Blocker
>             Fix For: 0.12.1
>
>         Attachments: CpuPendingTransfer3.patch
>
>
> When the namenode is restarted, the datanodes register and each block is inserted into neededReplication. When the namenode exists, safemode it sees starts processing neededReplication. It picks up a block from neededReplication, sees that it has already has the required number of replicas, and continues to the next block in neededReplication. The blocks remain in neededReplication permanentlyhe namenode worker thread to scans this huge list of blocks once every 3 seconds. This consumes plenty of CPU on the namenode.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14854-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 21:53:30 2011
Return-Path: <common-issues-return-14854-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ED8B14778
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 21:53:29 +0000 (UTC)
Received: (qmail 49382 invoked by uid 500); 13 May 2011 21:53:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49342 invoked by uid 500); 13 May 2011 21:53:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49334 invoked by uid 99); 13 May 2011 21:53:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 21:53:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 21:53:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5F7FE89FA9
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 21:52:47 +0000 (UTC)
Date: Fri, 13 May 2011 21:52:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <966467803.11611.1305323567387.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-1117) DFS Scalability: When the namenode
 is restarted it consumes 80% CPU
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033330#comment-13033330 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-1117:
------------------------------------------------

Sorry, wrong issue.

> DFS Scalability: When the namenode is restarted it consumes 80% CPU
> -------------------------------------------------------------------
>
>                 Key: HADOOP-1117
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1117
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.12.0
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>            Priority: Blocker
>             Fix For: 0.12.1
>
>         Attachments: CpuPendingTransfer3.patch
>
>
> When the namenode is restarted, the datanodes register and each block is inserted into neededReplication. When the namenode exists, safemode it sees starts processing neededReplication. It picks up a block from neededReplication, sees that it has already has the required number of replicas, and continues to the next block in neededReplication. The blocks remain in neededReplication permanentlyhe namenode worker thread to scans this huge list of blocks once every 3 seconds. This consumes plenty of CPU on the namenode.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14855-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 22:01:29 2011
Return-Path: <common-issues-return-14855-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A274D4FD8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 22:01:29 +0000 (UTC)
Received: (qmail 56053 invoked by uid 500); 13 May 2011 22:01:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56028 invoked by uid 500); 13 May 2011 22:01:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56020 invoked by uid 99); 13 May 2011 22:01:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:01:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:01:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 57CFD891CE
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 22:00:47 +0000 (UTC)
Date: Fri, 13 May 2011 22:00:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <20192042.11634.1305324047356.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <539645178.10940.1305312827448.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7289) ivy: test conf should not extend
 common conf
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033333#comment-13033333 ] 

Hadoop QA commented on HADOOP-7289:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479171/c7289_20110513b.patch
  against trunk revision 1102893.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.metrics2.impl.TestSinkQueue

    -1 contrib tests.  The patch failed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/453//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/453//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/453//console

This message is automatically generated.

> ivy: test conf should not extend common conf
> --------------------------------------------
>
>                 Key: HADOOP-7289
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7289
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c7289_20110513.patch, c7289_20110513b.patch
>
>
> Otherwise, the same jars will appear in both {{build/ivy/lib/Hadoop-Common/common/}} and {{build/ivy/lib/Hadoop-Common/test/}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14856-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 22:15:28 2011
Return-Path: <common-issues-return-14856-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3561140F8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 22:15:28 +0000 (UTC)
Received: (qmail 68072 invoked by uid 500); 13 May 2011 22:15:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68029 invoked by uid 500); 13 May 2011 22:15:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67910 invoked by uid 99); 13 May 2011 22:15:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:15:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:15:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D2240895FB
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 22:14:47 +0000 (UTC)
Date: Fri, 13 May 2011 22:14:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2021018182.11663.1305324887857.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6728) Overhaul metrics framework
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033341#comment-13033341 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6728:
------------------------------------------------

{{metrics2.impl.TestSinkQueue}} is failing, see [build #453|https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/453//testReport/], [build #447|https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/447//testReport/], etc.

> Overhaul metrics framework
> --------------------------
>
>                 Key: HADOOP-6728
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6728
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.2, 0.21.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-6728-y20.104.patch, metrics1-flow.png, metrics2-builder-r2.png, metrics2-builder.png, metrics2-flow.png, metrics2-mutable-r2.png, metrics2-mutable.png, metrics2-record-r2.png, metrics2-record.png, metrics2-uml-r2.png, metrics2-uml.png
>
>
> Per discussions with Arun, Chris, Hong and Rajiv, et al, we concluded that the current metrics framework needs an overhaul to:
> * Allow multiple plugins for different monitoring systems simultaneously (see also: HADOOP-6508).
> * Refresh metrics plugin config without server restart.
> ** Including filtering of metrics per plugin.
> * Support metrics schema for plugins.
> The jira will be resolved when core hadoop components (hdfs, mapreduce) are updated to use the new framework . Updates to external components that use the existing metrics framework will be tracked by different issues.
> [The current design wiki|http://wiki.apache.org/hadoop/HADOOP-6728-MetricsV2].

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14857-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 22:17:30 2011
Return-Path: <common-issues-return-14857-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3811C495E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 22:17:30 +0000 (UTC)
Received: (qmail 70907 invoked by uid 500); 13 May 2011 22:17:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70881 invoked by uid 500); 13 May 2011 22:17:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70861 invoked by uid 99); 13 May 2011 22:17:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:17:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:17:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D5C2189740
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 22:16:47 +0000 (UTC)
Date: Fri, 13 May 2011 22:16:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1430838266.11712.1305325007872.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <957076305.12265.1297477317500.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7137) Remove hod contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033344#comment-13033344 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7137:
------------------------------------------------

Why not wait for Hudson/Jenkins?

> Remove hod contrib
> ------------------
>
>                 Key: HADOOP-7137
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7137
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7137.patch, HADOOP-7137.patch
>
>
> As per vote on general@ (http://mail-archives.apache.org/mod_mbox/hadoop-general/201102.mbox/%3CAC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com%3E) I will 
> svn remove common/trunk/src/contrib/hod
> using this Jira.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14858-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 22:21:30 2011
Return-Path: <common-issues-return-14858-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 201F646AB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 22:21:30 +0000 (UTC)
Received: (qmail 76886 invoked by uid 500); 13 May 2011 22:21:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76813 invoked by uid 500); 13 May 2011 22:21:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76742 invoked by uid 99); 13 May 2011 22:21:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:21:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:21:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7AEAC898BA
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 22:20:47 +0000 (UTC)
Date: Fri, 13 May 2011 22:20:47 +0000 (UTC)
From: "John George (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <314553038.11731.1305325247500.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <903601160.6842.1305210828211.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7282) getRemoteIp could return null in
 cases where the call is ongoing but the ip went away.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

John George updated HADOOP-7282:
--------------------------------

    Attachment: HADOOP-7282-1.patch

> getRemoteIp could return null in cases where the call is ongoing but the ip went away.
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7282
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7282
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: John George
>            Assignee: John George
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7282-1.patch, HADOOP-7282.patch, diffs.txt
>
>
> getRemoteIp gets the ip from socket instead of the stored ip in Connection object. Thus calls to this function could return null when a client disconnected, but the rpc call is still ongoing...

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14859-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 22:25:28 2011
Return-Path: <common-issues-return-14859-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2370F48B4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 22:25:28 +0000 (UTC)
Received: (qmail 83073 invoked by uid 500); 13 May 2011 22:25:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83043 invoked by uid 500); 13 May 2011 22:25:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83035 invoked by uid 99); 13 May 2011 22:25:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:25:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:25:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8F1A589AF5
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 22:24:47 +0000 (UTC)
Date: Fri, 13 May 2011 22:24:47 +0000 (UTC)
From: "John George (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2012368330.11758.1305325487582.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <903601160.6842.1305210828211.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7282) getRemoteIp could return null in
 cases where the call is ongoing but the ip went away.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

John George updated HADOOP-7282:
--------------------------------

    Status: Patch Available  (was: Open)

> getRemoteIp could return null in cases where the call is ongoing but the ip went away.
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7282
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7282
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: John George
>            Assignee: John George
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7282-1.patch, HADOOP-7282.patch, diffs.txt
>
>
> getRemoteIp gets the ip from socket instead of the stored ip in Connection object. Thus calls to this function could return null when a client disconnected, but the rpc call is still ongoing...

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14860-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 22:25:29 2011
Return-Path: <common-issues-return-14860-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EDAF748DB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 22:25:29 +0000 (UTC)
Received: (qmail 84587 invoked by uid 500); 13 May 2011 22:25:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84547 invoked by uid 500); 13 May 2011 22:25:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84538 invoked by uid 99); 13 May 2011 22:25:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:25:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:25:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7560189AF1
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 22:24:47 +0000 (UTC)
Date: Fri, 13 May 2011 22:24:47 +0000 (UTC)
From: "John George (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1117299334.11754.1305325487477.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <903601160.6842.1305210828211.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7282) getRemoteIp could return null in
 cases where the call is ongoing but the ip went away.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033352#comment-13033352 ] 

John George commented on HADOOP-7282:
-------------------------------------

Ran manual tests to verify that ip address gets printed out correctly even when client exits without completing an rpc. 

Also ran tests to verify that ip gets printed in normal use cases.

No tests attached since it seemed extremely difficult to add tests for the above use-cases.

Results of test-patch. findbugs not because of this patch.
    [exec]
     [exec] -1 overall.
     [exec]
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec]
     [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
     [exec]                         Please justify why no new tests are needed for this patch.
     [exec]                         Also please list what manual steps were performed to verify this patch.
     [exec]
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec]
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec]
     [exec]     -1 findbugs.  The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.
     [exec]
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec]
     [exec]     +1 system test framework.  The patch passed system test framework compile.
     [exec]
     [exec]

~                                

> getRemoteIp could return null in cases where the call is ongoing but the ip went away.
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7282
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7282
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: John George
>            Assignee: John George
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7282-1.patch, HADOOP-7282.patch, diffs.txt
>
>
> getRemoteIp gets the ip from socket instead of the stored ip in Connection object. Thus calls to this function could return null when a client disconnected, but the rpc call is still ongoing...

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14861-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 22:43:27 2011
Return-Path: <common-issues-return-14861-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 98A25417B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 22:43:27 +0000 (UTC)
Received: (qmail 10192 invoked by uid 500); 13 May 2011 22:43:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10142 invoked by uid 500); 13 May 2011 22:43:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10130 invoked by uid 99); 13 May 2011 22:43:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:43:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:43:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6136489FFD
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 22:42:47 +0000 (UTC)
Date: Fri, 13 May 2011 22:42:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2120237073.11802.1305326567394.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1115617041.10194.1305296867305.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7288) FsShell.DelayedExceptionThrowing
 generates QA errors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins resolved HADOOP-7288.
---------------------------------

    Resolution: Fixed

This was fixed in HADOOP-7285.

> FsShell.DelayedExceptionThrowing generates QA errors
> ----------------------------------------------------
>
>                 Key: HADOOP-7288
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7288
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: HADOOP-7288-remove-redundant-class-001.patch
>
>
> The inner class o.a.h.fs.FsShell.DelayedExceptionThrowing is causing the QA bot to fail all patches due to a findbugs issue (the inner class should be static).
> It's an abstract class that is not currently used. It looks as though it's old code that simply needs removing.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14862-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 22:53:30 2011
Return-Path: <common-issues-return-14862-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 338BF48D0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 22:53:30 +0000 (UTC)
Received: (qmail 23648 invoked by uid 500); 13 May 2011 22:53:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23609 invoked by uid 500); 13 May 2011 22:53:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23601 invoked by uid 99); 13 May 2011 22:53:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:53:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:53:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D6A5E892A6
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 22:52:47 +0000 (UTC)
Date: Fri, 13 May 2011 22:52:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1279635768.11870.1305327167875.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7291:
--------------------------------

        Assignee: Nigel Daley
    Hadoop Flags: [Reviewed]

I've committed this to trunk and branch 22. Thanks Nigel!

> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Nigel Daley
>         Attachments: HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14863-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 22:55:27 2011
Return-Path: <common-issues-return-14863-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9CEA24915
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 22:55:27 +0000 (UTC)
Received: (qmail 26071 invoked by uid 500); 13 May 2011 22:55:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26035 invoked by uid 500); 13 May 2011 22:55:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26026 invoked by uid 99); 13 May 2011 22:55:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:55:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:55:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 708BC89387
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 22:54:47 +0000 (UTC)
Date: Fri, 13 May 2011 22:54:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1072820164.11873.1305327287457.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7291:
--------------------------------

    Fix Version/s: 0.22.0

> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Nigel Daley
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14864-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 22:55:28 2011
Return-Path: <common-issues-return-14864-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3A4264931
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 22:55:28 +0000 (UTC)
Received: (qmail 26365 invoked by uid 500); 13 May 2011 22:55:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26290 invoked by uid 500); 13 May 2011 22:55:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26268 invoked by uid 99); 13 May 2011 22:55:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:55:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:55:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AC6C289393
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 22:54:47 +0000 (UTC)
Date: Fri, 13 May 2011 22:54:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1104332365.11882.1305327287702.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins resolved HADOOP-7291.
---------------------------------

    Resolution: Fixed

> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Nigel Daley
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14865-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 22:55:28 2011
Return-Path: <common-issues-return-14865-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3A316492F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 22:55:28 +0000 (UTC)
Received: (qmail 26368 invoked by uid 500); 13 May 2011 22:55:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26314 invoked by uid 500); 13 May 2011 22:55:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26271 invoked by uid 99); 13 May 2011 22:55:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:55:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:55:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9F79E89391
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 22:54:47 +0000 (UTC)
Date: Fri, 13 May 2011 22:54:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1895244652.11880.1305327287649.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <957076305.12265.1297477317500.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7137) Remove hod contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033370#comment-13033370 ] 

Eli Collins commented on HADOOP-7137:
-------------------------------------

I should have. I didn't because most of the change is done via svn remove instead of applying the patch, so I ran the full tarball build, but waiting for Hudson would have caught HADOOP-7291 (the required change to test-patch.sh).

> Remove hod contrib
> ------------------
>
>                 Key: HADOOP-7137
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7137
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7137.patch, HADOOP-7137.patch
>
>
> As per vote on general@ (http://mail-archives.apache.org/mod_mbox/hadoop-general/201102.mbox/%3CAC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com%3E) I will 
> svn remove common/trunk/src/contrib/hod
> using this Jira.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14866-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 22:59:27 2011
Return-Path: <common-issues-return-14866-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E8BD449D5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 22:59:27 +0000 (UTC)
Received: (qmail 31143 invoked by uid 500); 13 May 2011 22:59:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30985 invoked by uid 500); 13 May 2011 22:59:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30977 invoked by uid 99); 13 May 2011 22:59:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:59:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 22:59:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C065B894A1
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 22:58:47 +0000 (UTC)
Date: Fri, 13 May 2011 22:58:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <450658595.11896.1305327527784.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033373#comment-13033373 ] 

Eli Collins commented on HADOOP-7291:
-------------------------------------

Forgot to mention, please enable precommit builds when you get a chance.

> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Nigel Daley
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14867-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 23:33:29 2011
Return-Path: <common-issues-return-14867-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D2CA14B9E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 23:33:29 +0000 (UTC)
Received: (qmail 57636 invoked by uid 500); 13 May 2011 23:33:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57598 invoked by uid 500); 13 May 2011 23:33:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57590 invoked by uid 99); 13 May 2011 23:33:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 23:33:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 23:33:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9176589CA1
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 23:32:47 +0000 (UTC)
Date: Fri, 13 May 2011 23:32:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <866597877.11958.1305329567592.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7292) Metrics 2 TestSinkQueue is racy
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Metrics 2 TestSinkQueue is racy
-------------------------------

                 Key: HADOOP-7292
                 URL: https://issues.apache.org/jira/browse/HADOOP-7292
             Project: Hadoop Common
          Issue Type: Bug
          Components: metrics
    Affects Versions: 0.23.0
            Reporter: Luke Lu
            Assignee: Luke Lu
            Priority: Minor
             Fix For: 0.23.0


The TestSinkQueue is racy (Thread.yield is not enough to guarantee other intended thread getting run), though it's the first time (from HADOOP-7289) I saw it manifested here.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14868-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 23:37:27 2011
Return-Path: <common-issues-return-14868-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7CD234F82
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 23:37:27 +0000 (UTC)
Received: (qmail 59659 invoked by uid 500); 13 May 2011 23:37:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59625 invoked by uid 500); 13 May 2011 23:37:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59617 invoked by uid 99); 13 May 2011 23:37:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 23:37:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 23:37:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 71FB289D68
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 23:36:47 +0000 (UTC)
Date: Fri, 13 May 2011 23:36:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1437948146.11960.1305329807463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6728) Overhaul metrics framework
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033384#comment-13033384 ] 

Luke Lu commented on HADOOP-6728:
---------------------------------

bq. metrics2.impl.TestSinkQueue is failing, see build #453, build #447, etc.

I knew the test is racy (relying on Thread.yield) but it never failed on me until the pre-commit builds for HADOOP-7289 patches. Opened HADOOP-7292 to track the test fix.

> Overhaul metrics framework
> --------------------------
>
>                 Key: HADOOP-6728
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6728
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.2, 0.21.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-6728-y20.104.patch, metrics1-flow.png, metrics2-builder-r2.png, metrics2-builder.png, metrics2-flow.png, metrics2-mutable-r2.png, metrics2-mutable.png, metrics2-record-r2.png, metrics2-record.png, metrics2-uml-r2.png, metrics2-uml.png
>
>
> Per discussions with Arun, Chris, Hong and Rajiv, et al, we concluded that the current metrics framework needs an overhaul to:
> * Allow multiple plugins for different monitoring systems simultaneously (see also: HADOOP-6508).
> * Refresh metrics plugin config without server restart.
> ** Including filtering of metrics per plugin.
> * Support metrics schema for plugins.
> The jira will be resolved when core hadoop components (hdfs, mapreduce) are updated to use the new framework . Updates to external components that use the existing metrics framework will be tracked by different issues.
> [The current design wiki|http://wiki.apache.org/hadoop/HADOOP-6728-MetricsV2].

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14869-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 23:51:28 2011
Return-Path: <common-issues-return-14869-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CFA574669
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 23:51:28 +0000 (UTC)
Received: (qmail 70195 invoked by uid 500); 13 May 2011 23:51:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70165 invoked by uid 500); 13 May 2011 23:51:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70097 invoked by uid 99); 13 May 2011 23:51:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 23:51:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 23:51:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 19543890D5
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 23:50:48 +0000 (UTC)
Date: Fri, 13 May 2011 23:50:48 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <144810310.12034.1305330648100.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-6671:
---------------------------------------

    Attachment: mvn-layout.sh
                hadoop-commons-maven.patch

I spent a few hours playing with hadoop-common mavenization following the ideas in the JIRA.

I've got it to a point that compiles java and native code, runs the tests (with native code if present) and generates the JAR.

I've only had to modify 1 java file, one testcase that had 'build/classes' hardcoded to look for a properties file.

I've had to move directories/files around to follow Maven dir layout.

Still to be done is javadocs, documentation, packaging (that would be an assembly descriptor), wiring jdiff/clover/findbugs, and deployment configuration of artifacts (JARs/SOs).

The end goal will be to generate a TARBALL identical to the one it is being generated today. 

I don't expect all that to be much work.

Once hadoop-commons is done, the same could be done in hadoop-hdfs and hadoop-mapreduce. Also the contrib/ stuff could go into its own Maven module. Finally a root Maven project could be used to wire all the above projects and external dependencies versions would be defined there in a dependencyManagement section.

The good thing is that it is not required to do all at once, we can do common, then hdfs, then mapreduce and finally contrib.

Attached you'll find a script that moves dir/files around to the maven expected locations and a patch containing the Maven pom.xml file, a native-build.xml (Ant file invoked from maven to build native code) and the 1 line change to a testcase.

Instructions to test it:

* checkout hadoop-common trunk
* run the attached script from hadoop-common root dir
* apply the the patch
* Use Maven 3 to to build/test
** -Dcompile.native enables native compilation

IMPORTANT: I couldn't figure this out yet but there is some issue with javah when invoked from Maven/Ant (javah is not being found in LINUX because Maven changes JAVA_HOME to JRE location). TEMPORARY WORKAROUND: softlink JDK/lib/tools.jar in JRE/lib/ext/

Before I continue working on this I want to know if folks would be OK with moving forward with this patch.


> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>         Attachments: build.png, hadoop-commons-maven.patch, mvn-layout.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14870-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 13 23:51:30 2011
Return-Path: <common-issues-return-14870-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 91F494675
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 13 May 2011 23:51:30 +0000 (UTC)
Received: (qmail 70489 invoked by uid 500); 13 May 2011 23:51:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70448 invoked by uid 500); 13 May 2011 23:51:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70439 invoked by uid 99); 13 May 2011 23:51:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 23:51:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 May 2011 23:51:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 545C0890DF
	for <common-issues@hadoop.apache.org>; Fri, 13 May 2011 23:50:48 +0000 (UTC)
Date: Fri, 13 May 2011 23:50:48 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <238554549.12043.1305330648342.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <539645178.10940.1305312827448.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7289) ivy: test conf should not extend
 common conf
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-7289:
------------------------------

    Attachment: HADOOP-7289.patch

Modified Test classpath ordering to include common third party jars first.

> ivy: test conf should not extend common conf
> --------------------------------------------
>
>                 Key: HADOOP-7289
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7289
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: HADOOP-7289.patch, c7289_20110513.patch, c7289_20110513b.patch
>
>
> Otherwise, the same jars will appear in both {{build/ivy/lib/Hadoop-Common/common/}} and {{build/ivy/lib/Hadoop-Common/test/}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14871-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 00:20:28 2011
Return-Path: <common-issues-return-14871-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1816645AA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 14 May 2011 00:20:28 +0000 (UTC)
Received: (qmail 85375 invoked by uid 500); 14 May 2011 00:20:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85319 invoked by uid 500); 14 May 2011 00:20:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85217 invoked by uid 99); 14 May 2011 00:20:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 00:20:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 00:20:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 920AB897B4
	for <common-issues@hadoop.apache.org>; Sat, 14 May 2011 00:19:47 +0000 (UTC)
Date: Sat, 14 May 2011 00:19:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <228261537.12089.1305332387594.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <539645178.10940.1305312827448.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7289) ivy: test conf should not extend
 common conf
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033396#comment-13033396 ] 

Hadoop QA commented on HADOOP-7289:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479191/HADOOP-7289.patch
  against trunk revision 1102914.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/454//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/454//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/454//console

This message is automatically generated.

> ivy: test conf should not extend common conf
> --------------------------------------------
>
>                 Key: HADOOP-7289
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7289
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: HADOOP-7289.patch, c7289_20110513.patch, c7289_20110513b.patch
>
>
> Otherwise, the same jars will appear in both {{build/ivy/lib/Hadoop-Common/common/}} and {{build/ivy/lib/Hadoop-Common/test/}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14872-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 00:32:30 2011
Return-Path: <common-issues-return-14872-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A3D4B47AD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 14 May 2011 00:32:30 +0000 (UTC)
Received: (qmail 94301 invoked by uid 500); 14 May 2011 00:32:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94269 invoked by uid 500); 14 May 2011 00:32:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94261 invoked by uid 99); 14 May 2011 00:32:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 00:32:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 00:32:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 962C7899C7
	for <common-issues@hadoop.apache.org>; Sat, 14 May 2011 00:31:47 +0000 (UTC)
Date: Sat, 14 May 2011 00:31:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <888248017.12100.1305333107612.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033398#comment-13033398 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7291:
------------------------------------------------

In [build #454|https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/454/console], there is a build failed.
{noformat}
     [exec] ======================================================================
     [exec] /grid/0/hudson/hudson-slave/workspace/PreCommit-HADOOP-Build/trunk/src/test/bin/test-patch.sh: line 527: -c: command not found
     [exec] ======================================================================
     [exec]     Running contrib tests.
     [exec] ======================================================================
     [exec] ======================================================================
     [exec] 
     [exec] 
     [exec] /grid/0/hudson/hudson-slave/workspace/PreCommit-HADOOP-Build/trunk/src/test/bin/test-patch.sh: line 533: auxwww: command not found
     [exec] /grid/0/hudson/hudson-slave/workspace/PreCommit-HADOOP-Build/trunk/src/test/bin/test-patch.sh: line 533: HadoopPatchProcess: command not found
     [exec] /homes/hudson/tools/ant/latest/bin/ant -Dversion=  -DHadoopPatchProcess= -Dtest.junit.output.format=xml -Dtest.output=no test-contrib
     [exec] Buildfile: build.xml
     [exec] 
     [exec] BUILD FAILED
     [exec] Target "test-contrib" does not exist in the project "Hadoop-Common". 
     [exec] 
     [exec] Total time: 0 seconds
{noformat}


> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Nigel Daley
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14873-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 00:58:29 2011
Return-Path: <common-issues-return-14873-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 01ECA4484
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 14 May 2011 00:58:29 +0000 (UTC)
Received: (qmail 7135 invoked by uid 500); 14 May 2011 00:58:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7067 invoked by uid 500); 14 May 2011 00:58:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6780 invoked by uid 99); 14 May 2011 00:58:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 00:58:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 00:58:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8071689E99
	for <common-issues@hadoop.apache.org>; Sat, 14 May 2011 00:57:47 +0000 (UTC)
Date: Sat, 14 May 2011 00:57:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <384190727.12136.1305334667522.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Reopened] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins reopened HADOOP-7291:
---------------------------------

      Assignee: Eli Collins  (was: Nigel Daley)

> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14874-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 00:58:29 2011
Return-Path: <common-issues-return-14874-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 361CE4499
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 14 May 2011 00:58:29 +0000 (UTC)
Received: (qmail 7254 invoked by uid 500); 14 May 2011 00:58:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7197 invoked by uid 500); 14 May 2011 00:58:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7054 invoked by uid 99); 14 May 2011 00:58:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 00:58:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 00:58:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A4F1A89E9E
	for <common-issues@hadoop.apache.org>; Sat, 14 May 2011 00:57:47 +0000 (UTC)
Date: Sat, 14 May 2011 00:57:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1336413197.12140.1305334667672.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7291:
--------------------------------

    Attachment: HADOOP-7291-2.patch

Patch attached. Since there's no test-contrib target we can just remove it entirely.

> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14875-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 01:04:27 2011
Return-Path: <common-issues-return-14875-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E807F4A74
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 14 May 2011 01:04:27 +0000 (UTC)
Received: (qmail 11869 invoked by uid 500); 14 May 2011 01:04:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11823 invoked by uid 500); 14 May 2011 01:04:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11815 invoked by uid 99); 14 May 2011 01:04:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 01:04:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 01:04:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 916DC89F7F
	for <common-issues@hadoop.apache.org>; Sat, 14 May 2011 01:03:47 +0000 (UTC)
Date: Sat, 14 May 2011 01:03:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <250584371.12171.1305335027592.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <539645178.10940.1305312827448.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7289) ivy: test conf should not extend
 common conf
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7289:
-------------------------------------------

    Assignee: Eric Yang  (was: Tsz Wo (Nicholas), SZE)

Eric, if test extends master, then it won't download the jars.  Assigning it to you.

> ivy: test conf should not extend common conf
> --------------------------------------------
>
>                 Key: HADOOP-7289
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7289
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Eric Yang
>         Attachments: HADOOP-7289.patch, c7289_20110513.patch, c7289_20110513b.patch
>
>
> Otherwise, the same jars will appear in both {{build/ivy/lib/Hadoop-Common/common/}} and {{build/ivy/lib/Hadoop-Common/test/}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14876-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 01:06:30 2011
Return-Path: <common-issues-return-14876-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 212974B00
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 14 May 2011 01:06:30 +0000 (UTC)
Received: (qmail 13085 invoked by uid 500); 14 May 2011 01:06:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13042 invoked by uid 500); 14 May 2011 01:06:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13034 invoked by uid 99); 14 May 2011 01:06:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 01:06:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 01:06:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C4D8A89FF5
	for <common-issues@hadoop.apache.org>; Sat, 14 May 2011 01:05:47 +0000 (UTC)
Date: Sat, 14 May 2011 01:05:47 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <148439027.12175.1305335147803.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <539645178.10940.1305312827448.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7289) ivy: test conf should not extend
 common conf
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-7289:
------------------------------

    Attachment: HADOOP-7289-1.patch

Patch for extends test from master.

> ivy: test conf should not extend common conf
> --------------------------------------------
>
>                 Key: HADOOP-7289
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7289
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Eric Yang
>         Attachments: HADOOP-7289-1.patch, HADOOP-7289.patch, c7289_20110513.patch, c7289_20110513b.patch
>
>
> Otherwise, the same jars will appear in both {{build/ivy/lib/Hadoop-Common/common/}} and {{build/ivy/lib/Hadoop-Common/test/}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14877-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 01:26:27 2011
Return-Path: <common-issues-return-14877-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CDC1F41A5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 14 May 2011 01:26:27 +0000 (UTC)
Received: (qmail 19656 invoked by uid 500); 14 May 2011 01:26:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19615 invoked by uid 500); 14 May 2011 01:26:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19607 invoked by uid 99); 14 May 2011 01:26:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 01:26:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 01:26:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5F7C18737D
	for <common-issues@hadoop.apache.org>; Sat, 14 May 2011 01:25:47 +0000 (UTC)
Date: Sat, 14 May 2011 01:25:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1369502559.12203.1305336347387.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <539645178.10940.1305312827448.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7289) ivy: test conf should not extend
 common conf
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033412#comment-13033412 ] 

Hadoop QA commented on HADOOP-7289:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479199/HADOOP-7289-1.patch
  against trunk revision 1102914.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/455//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/455//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/455//console

This message is automatically generated.

> ivy: test conf should not extend common conf
> --------------------------------------------
>
>                 Key: HADOOP-7289
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7289
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Eric Yang
>         Attachments: HADOOP-7289-1.patch, HADOOP-7289.patch, c7289_20110513.patch, c7289_20110513b.patch
>
>
> Otherwise, the same jars will appear in both {{build/ivy/lib/Hadoop-Common/common/}} and {{build/ivy/lib/Hadoop-Common/test/}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14878-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 01:34:27 2011
Return-Path: <common-issues-return-14878-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CE72147D4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 14 May 2011 01:34:27 +0000 (UTC)
Received: (qmail 26702 invoked by uid 500); 14 May 2011 01:34:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26664 invoked by uid 500); 14 May 2011 01:34:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26656 invoked by uid 99); 14 May 2011 01:34:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 01:34:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 01:34:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9252287658
	for <common-issues@hadoop.apache.org>; Sat, 14 May 2011 01:33:47 +0000 (UTC)
Date: Sat, 14 May 2011 01:33:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1041215924.12221.1305336827596.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <539645178.10940.1305312827448.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7289) ivy: test conf should not extend
 common conf
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7289:
-------------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

+1 patch looks good.

I have committed this.  Thanks, Eric!

> ivy: test conf should not extend common conf
> --------------------------------------------
>
>                 Key: HADOOP-7289
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7289
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Eric Yang
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7289-1.patch, HADOOP-7289.patch, c7289_20110513.patch, c7289_20110513b.patch
>
>
> Otherwise, the same jars will appear in both {{build/ivy/lib/Hadoop-Common/common/}} and {{build/ivy/lib/Hadoop-Common/test/}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14879-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 01:38:30 2011
Return-Path: <common-issues-return-14879-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E4F97482E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 14 May 2011 01:38:29 +0000 (UTC)
Received: (qmail 28033 invoked by uid 500); 14 May 2011 01:38:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28009 invoked by uid 500); 14 May 2011 01:38:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28001 invoked by uid 99); 14 May 2011 01:38:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 01:38:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 01:38:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6C5AF87724
	for <common-issues@hadoop.apache.org>; Sat, 14 May 2011 01:37:47 +0000 (UTC)
Date: Sat, 14 May 2011 01:37:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <97425691.12233.1305337067440.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033418#comment-13033418 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7291:
------------------------------------------------

Hi Eli, test-patch is also used in HDFS and MapReduce.  So, we cannot remove test-contrib.

> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14880-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 03:10:32 2011
Return-Path: <common-issues-return-14880-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DBE246089
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 14 May 2011 03:10:32 +0000 (UTC)
Received: (qmail 55835 invoked by uid 500); 14 May 2011 03:10:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55729 invoked by uid 500); 14 May 2011 03:10:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55704 invoked by uid 99); 14 May 2011 03:10:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 03:10:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 03:10:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E12C888669
	for <common-issues@hadoop.apache.org>; Sat, 14 May 2011 03:09:47 +0000 (UTC)
Date: Sat, 14 May 2011 03:09:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1819611828.12304.1305342587918.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033430#comment-13033430 ] 

Eli Collins commented on HADOOP-7291:
-------------------------------------

Right, looking at the log more closely ("-c: command not found") it looks like GREP is not defined.  How is test-patch being called? 

> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14881-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 04:19:33 2011
Return-Path: <common-issues-return-14881-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 817CC66FF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 14 May 2011 04:19:33 +0000 (UTC)
Received: (qmail 84572 invoked by uid 500); 14 May 2011 04:19:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84537 invoked by uid 500); 14 May 2011 04:19:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84519 invoked by uid 99); 14 May 2011 04:19:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 04:19:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 04:19:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5BD15B241A
	for <common-issues@hadoop.apache.org>; Sat, 14 May 2011 04:18:48 +0000 (UTC)
Date: Sat, 14 May 2011 04:18:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1468116416.12387.1305346728372.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7284:
--------------------------------

    Status: Open  (was: Patch Available)

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14883-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 04:19:34 2011
Return-Path: <common-issues-return-14883-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4EA266728
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 14 May 2011 04:19:34 +0000 (UTC)
Received: (qmail 84804 invoked by uid 500); 14 May 2011 04:19:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84747 invoked by uid 500); 14 May 2011 04:19:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84480 invoked by uid 99); 14 May 2011 04:19:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 04:19:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 04:19:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 34613B2417
	for <common-issues@hadoop.apache.org>; Sat, 14 May 2011 04:18:48 +0000 (UTC)
Date: Sat, 14 May 2011 04:18:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <338849823.12384.1305346728211.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033442#comment-13033442 ] 

Todd Lipcon commented on HADOOP-7284:
-------------------------------------

Looks like this patch gets rid of the check against trash.isEnabled() -- is that intentional?

Also, it seems this should be straight-forward to unit test.

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14882-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 04:19:34 2011
Return-Path: <common-issues-return-14882-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4A8E86725
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 14 May 2011 04:19:34 +0000 (UTC)
Received: (qmail 84812 invoked by uid 500); 14 May 2011 04:19:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84540 invoked by uid 500); 14 May 2011 04:19:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84484 invoked by uid 99); 14 May 2011 04:19:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 04:19:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 04:19:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B836BB2420
	for <common-issues@hadoop.apache.org>; Sat, 14 May 2011 04:18:48 +0000 (UTC)
Date: Sat, 14 May 2011 04:18:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1585977980.12393.1305346728743.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7178) copyToLocal API is creating .crc
 files in local, even after setting verifyChecksum to false at client side.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7178:
--------------------------------

    Status: Open  (was: Patch Available)

Canceling patch available status for discussion.

> copyToLocal API is creating .crc files in local, even after setting verifyChecksum to false at client side.
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7178
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7178
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7178_COMMON.patch, HADOOP-7178_HDFS.patch
>
>
> When we copy the files from DFS to local, it is creating the .crc file in local filesystem for the verification of checksum even if we disable the checksum verification at client side.
>         When user does not want to do any checksum verification, then what will be the use in creating of these files in local file system.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14884-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 04:31:31 2011
Return-Path: <common-issues-return-14884-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 870946181
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 14 May 2011 04:31:31 +0000 (UTC)
Received: (qmail 90236 invoked by uid 500); 14 May 2011 04:31:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89978 invoked by uid 500); 14 May 2011 04:31:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89967 invoked by uid 99); 14 May 2011 04:31:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 04:31:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 04:31:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5F6DFB266B
	for <common-issues@hadoop.apache.org>; Sat, 14 May 2011 04:30:47 +0000 (UTC)
Date: Sat, 14 May 2011 04:30:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <903096075.12433.1305347447387.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6541) An Interactive Hadoop FS shell
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6541:
--------------------------------

    Status: Open  (was: Patch Available)

> An Interactive Hadoop FS shell
> ------------------------------
>
>                 Key: HADOOP-6541
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6541
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>         Attachments: HADOOP-6541.2.patch, HADOOP-6541.3.patch, HADOOP-6541.4.patch, HADOOP-6541.5.patch, HADOOP-6541.6.patch, HADOOP-6541.patch
>
>
> A shell that allows the user to execute multiple filesystem operations in a single JVM instance at a prompt.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14885-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 04:33:29 2011
Return-Path: <common-issues-return-14885-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 910A66AB4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 14 May 2011 04:33:29 +0000 (UTC)
Received: (qmail 92545 invoked by uid 500); 14 May 2011 04:33:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92472 invoked by uid 500); 14 May 2011 04:33:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92461 invoked by uid 99); 14 May 2011 04:33:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 04:33:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 04:33:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 62CC4B26DB
	for <common-issues@hadoop.apache.org>; Sat, 14 May 2011 04:32:47 +0000 (UTC)
Date: Sat, 14 May 2011 04:32:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1170809196.12454.1305347567401.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <510617684.3042.1300172489780.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7192) fs -stat docs aren't updated to
 reflect the format features
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033451#comment-13033451 ] 

Todd Lipcon commented on HADOOP-7192:
-------------------------------------

+1

> fs -stat docs aren't updated to reflect the format features
> -----------------------------------------------------------
>
>                 Key: HADOOP-7192
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7192
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: 0.21.0
>         Environment: Linux / 0.20
>            Reporter: Harsh J Chouraria
>            Assignee: Harsh J Chouraria
>            Priority: Trivial
>              Labels: documentaion
>             Fix For: 0.20.3, 0.21.1, 0.23.0
>
>         Attachments: hadoop.common.fsstatdoc.r1.diff
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> The html docs of the 'fs -stat' command (that is found listed in the File System Shell Guide), does not seem to have the formatting abilities of -stat explained (along with the options).
> Like 'fs -help', the docs must also reflect the latest available features.
> I shall attach a doc-fix patch shortly.
> If anyone has other discrepancies to point out in the web version of the guide, please do so :)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14886-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 04:35:31 2011
Return-Path: <common-issues-return-14886-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CC2D96F7D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 14 May 2011 04:35:31 +0000 (UTC)
Received: (qmail 93359 invoked by uid 500); 14 May 2011 04:35:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93308 invoked by uid 500); 14 May 2011 04:35:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93285 invoked by uid 99); 14 May 2011 04:35:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 04:35:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 04:35:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9DC1DB2756
	for <common-issues@hadoop.apache.org>; Sat, 14 May 2011 04:34:47 +0000 (UTC)
Date: Sat, 14 May 2011 04:34:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1893140767.12464.1305347687642.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <510617684.3042.1300172489780.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7192) fs -stat docs aren't updated to
 reflect the format features
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7192:
--------------------------------

       Resolution: Fixed
    Fix Version/s:     (was: 0.23.0)
                       (was: 0.21.1)
                       (was: 0.20.3)
                   0.22.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

Committed to 0.22 and trunk. Thanks, Harsh!

(p.s the chinese docs aren't a big deal -- they're known to be out of date)

> fs -stat docs aren't updated to reflect the format features
> -----------------------------------------------------------
>
>                 Key: HADOOP-7192
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7192
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: 0.21.0
>         Environment: Linux / 0.20
>            Reporter: Harsh J Chouraria
>            Assignee: Harsh J Chouraria
>            Priority: Trivial
>              Labels: documentaion
>             Fix For: 0.22.0
>
>         Attachments: hadoop.common.fsstatdoc.r1.diff
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> The html docs of the 'fs -stat' command (that is found listed in the File System Shell Guide), does not seem to have the formatting abilities of -stat explained (along with the options).
> Like 'fs -help', the docs must also reflect the latest available features.
> I shall attach a doc-fix patch shortly.
> If anyone has other discrepancies to point out in the web version of the guide, please do so :)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14887-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 04:39:34 2011
Return-Path: <common-issues-return-14887-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 072E16FA5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 14 May 2011 04:39:34 +0000 (UTC)
Received: (qmail 94044 invoked by uid 500); 14 May 2011 04:39:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94013 invoked by uid 500); 14 May 2011 04:39:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94005 invoked by uid 99); 14 May 2011 04:39:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 04:39:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 04:39:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5F1C9B2808
	for <common-issues@hadoop.apache.org>; Sat, 14 May 2011 04:38:47 +0000 (UTC)
Date: Sat, 14 May 2011 04:38:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <27115803.12474.1305347927386.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6880) Documentation: Chinese (cn) doc is
 old
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6880?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D130=
33454#comment-13033454 ]=20

Todd Lipcon commented on HADOOP-6880:
-------------------------------------

Best I can tell, the Chinese docs haven't been maintained since they were i=
nitially contributed in 2008 (HADOOP-4164).

I would like to propose that we remove them entirely, or add (in Chinese) a=
 big red warning message on the page indicating as much and asking for help=
 translating?

What do folks think?

> Documentation: Chinese (cn) doc is old
> --------------------------------------
>
>                 Key: HADOOP-6880
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6880
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.20.2
>            Reporter: huang jian
>
> Documentation: Chinese (cn) doc is old.
> http://hadoop.apache.org/common/docs/r0.20.2/quickstart.html
> JavaTM 1.6.x, preferably from Sun, must be installed.=20
> Configuration
> Use the following:
> conf/core-site.xml:
> <configuration>
>   <property>
>     <name>fs.default.name</name>
>     <value>hdfs://localhost:9000</value>
>   </property>
> </configuration>
> conf/hdfs-site.xml:
> <configuration>
>   <property>
>     <name>dfs.replication</name>
>     <value>1</value>
>   </property>
> </configuration>
> conf/mapred-site.xml:
> <configuration>
>   <property>
>     <name>mapred.job.tracker</name>
>     <value>localhost:9001</value>
>   </property>
> </configuration>
> http://hadoop.apache.org/common/docs/r0.20.2/cn/quickstart.html
> JavaTM1.5.x=EF=BC=8C=E5=BF=85=E9=A1=BB=E5=AE=89=E8=A3=85=EF=BC=8C=E5=BB=
=BA=E8=AE=AE=E9=80=89=E6=8B=A9Sun=E5=85=AC=E5=8F=B8=E5=8F=91=E8=A1=8C=E7=9A=
=84Java=E7=89=88=E6=9C=AC=E3=80=82=20
> =E9=85=8D=E7=BD=AE
> =E4=BD=BF=E7=94=A8=E5=A6=82=E4=B8=8B=E7=9A=84 conf/hadoop-site.xml:
> <configuration>
>   <property>
>     <name>fs.default.name</name>
>     <value>localhost:9000</value>
>   </property>
>   <property>
>     <name>mapred.job.tracker</name>
>     <value>localhost:9001</value>
>   </property>
>   <property>
>     <name>dfs.replication</name>
>     <value>1</value>
>   </property>
> </configuration>

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14888-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 05:15:31 2011
Return-Path: <common-issues-return-14888-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2FC226B9B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 14 May 2011 05:15:31 +0000 (UTC)
Received: (qmail 8618 invoked by uid 500); 14 May 2011 05:15:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8491 invoked by uid 500); 14 May 2011 05:15:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8478 invoked by uid 99); 14 May 2011 05:15:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 05:15:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 05:15:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 41FD2B2CDC
	for <common-issues@hadoop.apache.org>; Sat, 14 May 2011 05:14:48 +0000 (UTC)
Date: Sat, 14 May 2011 05:14:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <731753847.12510.1305350088266.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6105) Provide a way to automatically
 handle backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033459#comment-13033459 ] 

Todd Lipcon commented on HADOOP-6105:
-------------------------------------

It seems there is a bit of a flaw in the way this JIRA was done. Could those who are still watching this please take a look at HADOOP-7287? Thanks.

> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>            Assignee: V.V.Chaitanya Krishna
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6105-1.patch, HADOOP-6105-10.patch, HADOOP-6105-2.patch, HADOOP-6105-3.patch, HADOOP-6105-4.patch, HADOOP-6105-5.patch, HADOOP-6105-6.patch, HADOOP-6105-7.patch, HADOOP-6105-8.patch, HADOOP-6105-9.patch, HADOOP-6105.patch, HADOOP-6105.patch
>
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14889-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 10:44:29 2011
Return-Path: <common-issues-return-14889-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EF38F623E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 14 May 2011 10:44:28 +0000 (UTC)
Received: (qmail 47274 invoked by uid 500); 14 May 2011 10:44:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47238 invoked by uid 500); 14 May 2011 10:44:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47230 invoked by uid 99); 14 May 2011 10:44:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 10:44:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 10:44:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B24E5C309D
	for <common-issues@hadoop.apache.org>; Sat, 14 May 2011 10:43:47 +0000 (UTC)
Date: Sat, 14 May 2011 10:43:47 +0000 (UTC)
From: "Nicholas Telford (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1992215505.12772.1305369827726.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7269) S3 Native should allow
 customizable file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033499#comment-13033499 ] 

Nicholas Telford commented on HADOOP-7269:
------------------------------------------

That was actually my initial idea, but thought that it'd stand a better chance of being accepted if it was more abstract. I see your point though, it's part of the reason I felt so uneasy about the implementation.

Thanks for the review, I'll refactor the patch for S3 only on Monday.

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: HADOOP-7269-S3-metadata-001.diff, HADOOP-7269-S3-metadata-002.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14890-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 13:18:28 2011
Return-Path: <common-issues-return-14890-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AEB6B6C45
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 14 May 2011 13:18:28 +0000 (UTC)
Received: (qmail 57138 invoked by uid 500); 14 May 2011 13:18:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57082 invoked by uid 500); 14 May 2011 13:18:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57074 invoked by uid 99); 14 May 2011 13:18:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 13:18:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 13:18:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BDEE4B2E6A
	for <common-issues@hadoop.apache.org>; Sat, 14 May 2011 13:17:47 +0000 (UTC)
Date: Sat, 14 May 2011 13:17:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1952849314.12953.1305379067774.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033526#comment-13033526 ] 

Hudson commented on HADOOP-7291:
--------------------------------

Integrated in Hadoop-Mapreduce-22-branch #51 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/51/])
    HADOOP-7291. svn merge -c 1102914 from trunk


> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14891-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 13:30:29 2011
Return-Path: <common-issues-return-14891-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 242506A88
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 14 May 2011 13:30:29 +0000 (UTC)
Received: (qmail 78621 invoked by uid 500); 14 May 2011 13:30:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78591 invoked by uid 500); 14 May 2011 13:30:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78583 invoked by uid 99); 14 May 2011 13:30:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 13:30:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 13:30:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D4DD3CA12C
	for <common-issues@hadoop.apache.org>; Sat, 14 May 2011 13:29:48 +0000 (UTC)
Date: Sat, 14 May 2011 13:29:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <539064270.12992.1305379788868.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <957076305.12265.1297477317500.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7137) Remove hod contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033532#comment-13033532 ] 

Hudson commented on HADOOP-7137:
--------------------------------

Integrated in Hadoop-Common-trunk #688 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/688/])
    HADOOP-7137. Remove hod contrib. Contributed by Nigel Daley


> Remove hod contrib
> ------------------
>
>                 Key: HADOOP-7137
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7137
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7137.patch, HADOOP-7137.patch
>
>
> As per vote on general@ (http://mail-archives.apache.org/mod_mbox/hadoop-general/201102.mbox/%3CAC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com%3E) I will 
> svn remove common/trunk/src/contrib/hod
> using this Jira.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14892-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 13:30:31 2011
Return-Path: <common-issues-return-14892-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DF1026AB7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 14 May 2011 13:30:30 +0000 (UTC)
Received: (qmail 78905 invoked by uid 500); 14 May 2011 13:30:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78870 invoked by uid 500); 14 May 2011 13:30:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78849 invoked by uid 99); 14 May 2011 13:30:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 13:30:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 13:30:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 06CBECA11E
	for <common-issues@hadoop.apache.org>; Sat, 14 May 2011 13:29:48 +0000 (UTC)
Date: Sat, 14 May 2011 13:29:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <849762187.12978.1305379788024.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033529#comment-13033529 ] 

Hudson commented on HADOOP-7291:
--------------------------------

Integrated in Hadoop-Common-trunk #688 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/688/])
    HADOOP-7291. Update Hudson job not to run test-contrib. Contributed by Nigel Daley


> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14893-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 13:30:31 2011
Return-Path: <common-issues-return-14893-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 40DDD6ACE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 14 May 2011 13:30:31 +0000 (UTC)
Received: (qmail 78987 invoked by uid 500); 14 May 2011 13:30:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78924 invoked by uid 500); 14 May 2011 13:30:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78855 invoked by uid 99); 14 May 2011 13:30:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 13:30:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 13:30:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0FC9FCA11F
	for <common-issues@hadoop.apache.org>; Sat, 14 May 2011 13:29:48 +0000 (UTC)
Date: Sat, 14 May 2011 13:29:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1186824359.12979.1305379788061.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <510617684.3042.1300172489780.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7192) fs -stat docs aren't updated to
 reflect the format features
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033530#comment-13033530 ] 

Hudson commented on HADOOP-7192:
--------------------------------

Integrated in Hadoop-Common-trunk #688 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/688/])
    HADOOP-7192. Update fs -stat docs to reflect the format features. Contributed by Harsh J Chouraria.


> fs -stat docs aren't updated to reflect the format features
> -----------------------------------------------------------
>
>                 Key: HADOOP-7192
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7192
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: 0.21.0
>         Environment: Linux / 0.20
>            Reporter: Harsh J Chouraria
>            Assignee: Harsh J Chouraria
>            Priority: Trivial
>              Labels: documentaion
>             Fix For: 0.22.0
>
>         Attachments: hadoop.common.fsstatdoc.r1.diff
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> The html docs of the 'fs -stat' command (that is found listed in the File System Shell Guide), does not seem to have the formatting abilities of -stat explained (along with the options).
> Like 'fs -help', the docs must also reflect the latest available features.
> I shall attach a doc-fix patch shortly.
> If anyone has other discrepancies to point out in the web version of the guide, please do so :)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14895-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 13:30:31 2011
Return-Path: <common-issues-return-14895-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7E66E6AEE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 14 May 2011 13:30:31 +0000 (UTC)
Received: (qmail 79296 invoked by uid 500); 14 May 2011 13:30:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79230 invoked by uid 500); 14 May 2011 13:30:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79103 invoked by uid 99); 14 May 2011 13:30:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 13:30:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 13:30:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A1061CA125
	for <common-issues@hadoop.apache.org>; Sat, 14 May 2011 13:29:48 +0000 (UTC)
Date: Sat, 14 May 2011 13:29:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2063726183.12985.1305379788656.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32042405.8638.1305244247592.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7285) Refactor FsShell's test
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033531#comment-13033531 ] 

Hudson commented on HADOOP-7285:
--------------------------------

Integrated in Hadoop-Common-trunk #688 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/688/])
    HADOOP-7285. Refactor the test command to conform to new FsCommand class. Contributed by Daryn Sharp.


> Refactor FsShell's test
> -----------------------
>
>                 Key: HADOOP-7285
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7285
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7285-2.patch, HADOOP-7285-3.patch, HADOOP-7285.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14894-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 13:30:32 2011
Return-Path: <common-issues-return-14894-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5B6716AD6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 14 May 2011 13:30:31 +0000 (UTC)
Received: (qmail 79049 invoked by uid 500); 14 May 2011 13:30:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78952 invoked by uid 500); 14 May 2011 13:30:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78856 invoked by uid 99); 14 May 2011 13:30:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 13:30:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 13:30:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D2C33CA11A
	for <common-issues@hadoop.apache.org>; Sat, 14 May 2011 13:29:47 +0000 (UTC)
Date: Sat, 14 May 2011 13:29:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <457763966.12974.1305379787859.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <177713183.11069.1305315587397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7290) Unit test failure in
 TestUserGroupInformation.testGetServerSideGroups
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033528#comment-13033528 ] 

Hudson commented on HADOOP-7290:
--------------------------------

Integrated in Hadoop-Common-trunk #688 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/688/])
    HADOOP-7290. Unit test failure in TestUserGroupInformation.testGetServerSideGroups. Contributed by Trevor Robison


> Unit test failure in TestUserGroupInformation.testGetServerSideGroups
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-7290
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7290
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.23.0
>         Environment: Linux 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:24 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
>            Reporter: Trevor Robinson
>            Assignee: Trevor Robinson
>            Priority: Minor
>              Labels: test
>         Attachments: TestUserGroupInformation-id-order.patch
>
>
> Testsuite: org.apache.hadoop.security.TestUserGroupInformation
> Tests run: 14, Failures: 1, Errors: 0, Time elapsed: 0.278 sec
> ------------- Standard Output ---------------
> trobinson:users guest git
> ------------- ---------------- ---------------
> Testcase: testGetServerSideGroups took 0.051 sec
>        FAILED
> expected:<g[ues]t> but was:<g[i]t>
> junit.framework.AssertionFailedError: expected:<g[ues]t> but was:<g[i]t>
>        at org.apache.hadoop.security.TestUserGroupInformation.testGetServerSideGroups(TestUserGroupInformation.java:94)
> It seems like the test is assuming that the groups returned by UserGroupInformation.getGroupNames() are in the same order as those returned by executing `id -Gn`. getGroupNames() only documents that the primary group is first, and `man id` doesn't document any ordering, so it seems like the test needs to be reworked to remove that assumption.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14896-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 13:30:50 2011
Return-Path: <common-issues-return-14896-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 02BC76C74
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 14 May 2011 13:30:50 +0000 (UTC)
Received: (qmail 79990 invoked by uid 500); 14 May 2011 13:30:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79893 invoked by uid 500); 14 May 2011 13:30:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79884 invoked by uid 99); 14 May 2011 13:30:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 13:30:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 13:30:48 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E2214CA12E
	for <common-issues@hadoop.apache.org>; Sat, 14 May 2011 13:29:48 +0000 (UTC)
Date: Sat, 14 May 2011 13:29:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1595173543.12994.1305379788923.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <539645178.10940.1305312827448.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7289) ivy: test conf should not extend
 common conf
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033533#comment-13033533 ] 

Hudson commented on HADOOP-7289:
--------------------------------

Integrated in Hadoop-Common-trunk #688 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/688/])
    HADOOP-7289. In ivy.xml, test conf should not extend common conf.  Contributed by Eric Yang


> ivy: test conf should not extend common conf
> --------------------------------------------
>
>                 Key: HADOOP-7289
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7289
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Eric Yang
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7289-1.patch, HADOOP-7289.patch, c7289_20110513.patch, c7289_20110513b.patch
>
>
> Otherwise, the same jars will appear in both {{build/ivy/lib/Hadoop-Common/common/}} and {{build/ivy/lib/Hadoop-Common/test/}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14897-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 15:39:29 2011
Return-Path: <common-issues-return-14897-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4A4DB6107
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 14 May 2011 15:39:29 +0000 (UTC)
Received: (qmail 85618 invoked by uid 500); 14 May 2011 15:39:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85548 invoked by uid 500); 14 May 2011 15:39:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85472 invoked by uid 99); 14 May 2011 15:39:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 15:39:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 15:39:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 48404CA51F
	for <common-issues@hadoop.apache.org>; Sat, 14 May 2011 15:38:48 +0000 (UTC)
Date: Sat, 14 May 2011 15:38:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <244519365.13108.1305387528292.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033552#comment-13033552 ] 

Hudson commented on HADOOP-7291:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk #679 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/679/])
    HADOOP-7291. Update Hudson job not to run test-contrib. Contributed by Nigel Daley


> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14898-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 15:39:51 2011
Return-Path: <common-issues-return-14898-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E96B161C7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 14 May 2011 15:39:51 +0000 (UTC)
Received: (qmail 87933 invoked by uid 500); 14 May 2011 15:39:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87868 invoked by uid 500); 14 May 2011 15:39:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87763 invoked by uid 99); 14 May 2011 15:39:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 15:39:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 15:39:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E53E6CA588
	for <common-issues@hadoop.apache.org>; Sat, 14 May 2011 15:38:51 +0000 (UTC)
Date: Sat, 14 May 2011 15:38:51 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1206193044.13208.1305387531935.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6846) Scripts for building Hadoop 0.22.0
 release
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033562#comment-13033562 ] 

Hudson commented on HADOOP-6846:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk #679 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/679/])
    

> Scripts for building Hadoop 0.22.0 release
> ------------------------------------------
>
>                 Key: HADOOP-6846
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6846
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6846.patch, release-scripts.tar.gz
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14899-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 14 23:49:28 2011
Return-Path: <common-issues-return-14899-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3E8856382
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 14 May 2011 23:49:28 +0000 (UTC)
Received: (qmail 27046 invoked by uid 500); 14 May 2011 23:49:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27008 invoked by uid 500); 14 May 2011 23:49:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27000 invoked by uid 99); 14 May 2011 23:49:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 23:49:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 May 2011 23:49:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 579FBCA7E7
	for <common-issues@hadoop.apache.org>; Sat, 14 May 2011 23:48:47 +0000 (UTC)
Date: Sat, 14 May 2011 23:48:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1957536738.13486.1305416927355.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-1117) DFS Scalability: When the namenode
 is restarted it consumes 80% CPU
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033619#comment-13033619 ] 

Eli Collins commented on HADOOP-1117:
-------------------------------------

Doesn't this break the rack policy? We should only remove a block from neededReplications when it has sufficient replicas both in terms of replica count and rack count.

{noformat}
+        if (numCurrentReplica >= fileReplication) {
+          neededReplications.remove(block);
+        } else 
{noformat}

> DFS Scalability: When the namenode is restarted it consumes 80% CPU
> -------------------------------------------------------------------
>
>                 Key: HADOOP-1117
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1117
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.12.0
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>            Priority: Blocker
>             Fix For: 0.12.1
>
>         Attachments: CpuPendingTransfer3.patch
>
>
> When the namenode is restarted, the datanodes register and each block is inserted into neededReplication. When the namenode exists, safemode it sees starts processing neededReplication. It picks up a block from neededReplication, sees that it has already has the required number of replicas, and continues to the next block in neededReplication. The blocks remain in neededReplication permanentlyhe namenode worker thread to scans this huge list of blocks once every 3 seconds. This consumes plenty of CPU on the namenode.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14900-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun May 15 17:08:32 2011
Return-Path: <common-issues-return-14900-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 13A584076
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 15 May 2011 17:08:32 +0000 (UTC)
Received: (qmail 30541 invoked by uid 500); 15 May 2011 17:08:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30287 invoked by uid 500); 15 May 2011 17:08:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30276 invoked by uid 99); 15 May 2011 17:08:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 May 2011 17:08:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 May 2011 17:08:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 70B42CBF7A
	for <common-issues@hadoop.apache.org>; Sun, 15 May 2011 17:07:47 +0000 (UTC)
Date: Sun, 15 May 2011 17:07:47 +0000 (UTC)
From: "Robert Henry (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1312765634.14050.1305479267458.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7293) missing spaces in error messages
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

missing spaces in error messages
--------------------------------

                 Key: HADOOP-7293
                 URL: https://issues.apache.org/jira/browse/HADOOP-7293
             Project: Hadoop Common
          Issue Type: Bug
          Components: native
    Affects Versions: 0.20.2
            Reporter: Robert Henry
            Priority: Trivial


Error message(s) are missing spaces.  Here's an example output:
  11/05/15 09:44:10 WARN mapred.JobClient: Error reading task outputhttp://
Generated from this line of source.

./src/mapred/org/apache/hadoop/mapred/JobClient.java:      LOG.warn("Error reading task output" + ioe.getMessage()); 

The 1st arg to LOG.warn should end with a ' '.

There may be other instances of this problem in the source base.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14901-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun May 15 19:14:29 2011
Return-Path: <common-issues-return-14901-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 752874128
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 15 May 2011 19:14:29 +0000 (UTC)
Received: (qmail 14750 invoked by uid 500); 15 May 2011 19:14:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14221 invoked by uid 500); 15 May 2011 19:14:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13980 invoked by uid 99); 15 May 2011 19:14:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 May 2011 19:14:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 May 2011 19:14:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 692D8CB3D7
	for <common-issues@hadoop.apache.org>; Sun, 15 May 2011 19:13:47 +0000 (UTC)
Date: Sun, 15 May 2011 19:13:47 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1266286744.14128.1305486827427.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6865) there will be ant error if ant ran
 without network connected
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033800#comment-13033800 ] 

Steve Loughran commented on HADOOP-6865:
----------------------------------------

Fixes for this
 -store the ivy file in the source tree and use it that way
 -store the SHA1 checksum of the ivy file, and verify that the doenloaded artifact in the ivy cache dir matches it. That's a good security check anyway.

> there will be ant error if ant ran without network connected
> ------------------------------------------------------------
>
>                 Key: HADOOP-6865
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6865
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.20.2
>         Environment: centos 5.4
>            Reporter: Evan Wang
>
> If you run `ant` without network connected, there will be an error below. And even if you connect your network, the error will exist.
> ivy-init-antlib:
>   [typedef] java.util.zip.ZipException: error in opening zip file
>   [typedef]     at java.util.zip.ZipFile.open(Native Method)
>   [typedef]     at java.util.zip.ZipFile.<init>(ZipFile.java:114)
>   [typedef]     at java.util.zip.ZipFile.<init>(ZipFile.java:131)
>   [typedef]     at org.apache.tools.ant.AntClassLoader.getResourceURL(AntClassLoader.java:1028)
>   [typedef]     at org.apache.tools.ant.AntClassLoader$ResourceEnumeration.findNextResource(AntClassLoader.java:147)
>   [typedef]     at org.apache.tools.ant.AntClassLoader$ResourceEnumeration.<init>(AntClassLoader.java:109)
>   [typedef]     at org.apache.tools.ant.AntClassLoader.findResources(AntClassLoader.java:975)
>   [typedef]     at java.lang.ClassLoader.getResources(ClassLoader.java:1016)
>   [typedef]     at org.apache.tools.ant.taskdefs.Definer.resourceToURLs(Definer.java:364)
>   [typedef]     at org.apache.tools.ant.taskdefs.Definer.execute(Definer.java:256)
>   [typedef]     at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288)
>   [typedef]     at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
>   [typedef]     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   [typedef]     at java.lang.reflect.Method.invoke(Method.java:597)
>   [typedef]     at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
>   [typedef]     at org.apache.tools.ant.Task.perform(Task.java:348)
>   [typedef]     at org.apache.tools.ant.Target.execute(Target.java:357)
>   [typedef]     at org.apache.tools.ant.Target.performTasks(Target.java:385)
>   [typedef]     at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1337)
>   [typedef]     at org.apache.tools.ant.Project.executeTarget(Project.java:1306)
>   [typedef]     at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
>   [typedef]     at org.apache.tools.ant.Project.executeTargets(Project.java:1189)
>   [typedef]     at org.apache.tools.ant.Main.runBuild(Main.java:758)
>   [typedef]     at org.apache.tools.ant.Main.startAnt(Main.java:217)
>   [typedef]     at org.apache.tools.ant.launch.Launcher.run(Launcher.java:257)
>   [typedef]     at org.apache.tools.ant.launch.Launcher.main(Launcher.java:104)
>   [typedef] Could not load definitions from resource org/apache/ivy/ant/antlib.xml. It could not be found.
> BUILD FAILED
> /opt/hadoop-0.20.2/build.xml:1644: You need Apache Ivy 2.0 or later from http://ant.apache.org/
>       It could not be loaded from http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.0.0-rc2/ivy-2.0.0-rc2.jar

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14902-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 00:44:30 2011
Return-Path: <common-issues-return-14902-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6B41245E1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 00:44:30 +0000 (UTC)
Received: (qmail 22830 invoked by uid 500); 16 May 2011 00:44:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22786 invoked by uid 500); 16 May 2011 00:44:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22778 invoked by uid 99); 16 May 2011 00:44:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 00:44:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 00:44:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8543FCB8F8
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 00:43:47 +0000 (UTC)
Date: Mon, 16 May 2011 00:43:47 +0000 (UTC)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <847667848.14365.1305506627542.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1894387424.4008.1305137809491.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7278) Automatic Hadoop cluster deployment
 for build validation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-7278:
---------------------------------------

    Status: Patch Available  (was: Open)

Patch is ready for review.

> Automatic Hadoop cluster deployment for build validation
> --------------------------------------------------------
>
>                 Key: HADOOP-7278
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7278
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0, 0.23.0
>         Environment: Apache Jenkins
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>              Labels: newbie
>         Attachments: HADOOP-7278.patch
>
>
> It'd be great to have a way of automatically deploying Hadoop cluster to a set of machine once all components are successfully built (in the form or tar or whatever). Deployed cluster then can be used to run a set of validation jobs to make sure that produced artifacts are viable. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14903-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 15:08:32 2011
Return-Path: <common-issues-return-14903-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 30BF46778
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 15:08:32 +0000 (UTC)
Received: (qmail 80154 invoked by uid 500); 16 May 2011 15:08:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79745 invoked by uid 500); 16 May 2011 15:08:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79731 invoked by uid 99); 16 May 2011 15:08:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 15:08:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 15:08:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 65F3CCC263
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 15:07:47 +0000 (UTC)
Date: Mon, 16 May 2011 15:07:47 +0000 (UTC)
From: "Vitalii Tymchyshyn (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2007918179.15449.1305558467414.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7294) FileUtil uses wrong stat command for
 FreeBSD
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

FileUtil uses wrong stat command for FreeBSD
--------------------------------------------

                 Key: HADOOP-7294
                 URL: https://issues.apache.org/jira/browse/HADOOP-7294
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
    Affects Versions: 0.21.0
         Environment: FreeBSD 8.0-STABLE
            Reporter: Vitalii Tymchyshyn


I get next exception when try to use append:

2011-05-16 17:07:54,648 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.112.0.207:50010, storageID=DS-1047171559-
10.112.0.207-50010-1302796304164, infoPort=50075, ipcPort=50020):DataXceiver
java.io.IOException: Failed to get link count on file /var/data/hdfs/data/current/finalized/subdir26/subdir17/subdir55/blk_-1266943884751786595:
 message=null; error=stat: illegal option -- c; exit value=1
        at org.apache.hadoop.fs.FileUtil.createIOException(FileUtil.java:709)
        at org.apache.hadoop.fs.FileUtil.access$000(FileUtil.java:42)
        at org.apache.hadoop.fs.FileUtil$HardLink.getLinkCount(FileUtil.java:682)
        at org.apache.hadoop.hdfs.server.datanode.ReplicaInfo.unlinkBlock(ReplicaInfo.java:215)
        at org.apache.hadoop.hdfs.server.datanode.FSDataset.append(FSDataset.java:1116)

It seems that FreeBSD is treated like UNIX and so calls 'stat -c%h', while FreeBSD is much more like Mac (since they have same BSD roots):

$ stat --help
stat: illegal option -- -
usage: stat [-FlLnqrsx] [-f format] [-t timefmt] [file ...]

$ stat -f%l a_file
1


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14904-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 15:56:28 2011
Return-Path: <common-issues-return-14904-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1C8ED6217
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 15:56:28 +0000 (UTC)
Received: (qmail 82693 invoked by uid 500); 16 May 2011 15:56:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82660 invoked by uid 500); 16 May 2011 15:56:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82652 invoked by uid 99); 16 May 2011 15:56:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 15:56:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 15:56:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 56D8ACC531
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 15:55:47 +0000 (UTC)
Date: Mon, 16 May 2011 15:55:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <394682192.15518.1305561347352.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <866597877.11958.1305329567592.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7292) Metrics 2 TestSinkQueue is racy
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-7292:
----------------------------

    Attachment: hadoop-7292-test-race-v1.patch

The v1 patch uses proper barrier for thread coordination.

> Metrics 2 TestSinkQueue is racy
> -------------------------------
>
>                 Key: HADOOP-7292
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7292
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.23.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7292-test-race-v1.patch
>
>
> The TestSinkQueue is racy (Thread.yield is not enough to guarantee other intended thread getting run), though it's the first time (from HADOOP-7289) I saw it manifested here.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14905-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 15:58:27 2011
Return-Path: <common-issues-return-14905-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DBB14625C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 15:58:27 +0000 (UTC)
Received: (qmail 85943 invoked by uid 500); 16 May 2011 15:58:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85904 invoked by uid 500); 16 May 2011 15:58:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85896 invoked by uid 99); 16 May 2011 15:58:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 15:58:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 15:58:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 880A9CC60D
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 15:57:47 +0000 (UTC)
Date: Mon, 16 May 2011 15:57:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1340762965.15522.1305561467554.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <866597877.11958.1305329567592.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7292) Metrics 2 TestSinkQueue is racy
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-7292:
----------------------------

    Status: Patch Available  (was: Open)

> Metrics 2 TestSinkQueue is racy
> -------------------------------
>
>                 Key: HADOOP-7292
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7292
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.23.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7292-test-race-v1.patch
>
>
> The TestSinkQueue is racy (Thread.yield is not enough to guarantee other intended thread getting run), though it's the first time (from HADOOP-7289) I saw it manifested here.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14906-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 17:14:28 2011
Return-Path: <common-issues-return-14906-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5B4AF6D1E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 17:14:28 +0000 (UTC)
Received: (qmail 48869 invoked by uid 500); 16 May 2011 17:14:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48717 invoked by uid 500); 16 May 2011 17:14:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48709 invoked by uid 99); 16 May 2011 17:14:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 17:14:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 17:14:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9A673CC375
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 17:13:47 +0000 (UTC)
Date: Mon, 16 May 2011 17:13:47 +0000 (UTC)
From: "Harsh J Chouraria (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <891531383.15685.1305566027629.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2007918179.15449.1305558467414.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7294) FileUtil uses wrong stat command
 for FreeBSD
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034105#comment-13034105 ] 

Harsh J Chouraria commented on HADOOP-7294:
-------------------------------------------

Vitalli,

AFAIK, most of the shell calls were written with Linux in mind, and that platform is what we recommend to use as well (so far). Would you have a solution that would work on all platforms for this? A patch would be great as well, but just tips otherwise would be nice too!

> FileUtil uses wrong stat command for FreeBSD
> --------------------------------------------
>
>                 Key: HADOOP-7294
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7294
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0
>         Environment: FreeBSD 8.0-STABLE
>            Reporter: Vitalii Tymchyshyn
>
> I get next exception when try to use append:
> 2011-05-16 17:07:54,648 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.112.0.207:50010, storageID=DS-1047171559-
> 10.112.0.207-50010-1302796304164, infoPort=50075, ipcPort=50020):DataXceiver
> java.io.IOException: Failed to get link count on file /var/data/hdfs/data/current/finalized/subdir26/subdir17/subdir55/blk_-1266943884751786595:
>  message=null; error=stat: illegal option -- c; exit value=1
>         at org.apache.hadoop.fs.FileUtil.createIOException(FileUtil.java:709)
>         at org.apache.hadoop.fs.FileUtil.access$000(FileUtil.java:42)
>         at org.apache.hadoop.fs.FileUtil$HardLink.getLinkCount(FileUtil.java:682)
>         at org.apache.hadoop.hdfs.server.datanode.ReplicaInfo.unlinkBlock(ReplicaInfo.java:215)
>         at org.apache.hadoop.hdfs.server.datanode.FSDataset.append(FSDataset.java:1116)
> It seems that FreeBSD is treated like UNIX and so calls 'stat -c%h', while FreeBSD is much more like Mac (since they have same BSD roots):
> $ stat --help
> stat: illegal option -- -
> usage: stat [-FlLnqrsx] [-f format] [-t timefmt] [file ...]
> $ stat -f%l a_file
> 1

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14907-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 17:44:28 2011
Return-Path: <common-issues-return-14907-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CE3966D8D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 17:44:28 +0000 (UTC)
Received: (qmail 94863 invoked by uid 500); 16 May 2011 17:44:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94797 invoked by uid 500); 16 May 2011 17:44:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94787 invoked by uid 99); 16 May 2011 17:44:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 17:44:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 17:44:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 619E8CC2EC
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 17:43:48 +0000 (UTC)
Date: Mon, 16 May 2011 17:43:48 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <715152037.15862.1305567828396.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034144#comment-13034144 ] 

Suresh Srinivas commented on HADOOP-7291:
-----------------------------------------

I see hudson pre-commit builds failing with this:
     [exec] Total time: 4 seconds
     [exec] ERROR: usage /grid/0/hudson/hudson-slave/workspace/PreCommit-HDFS-Build/trunk/src/test/bin/test-patch.sh HUDSON <PATCH_DIR> <SUPPORT_DIR> <PS_CMD> <WGET_CMD> <JIRACLI> <SVN_CMD> <GREP_CMD> <PATCH_CMD> <FINDBUGS_HOME> <FORREST_HOME> <ECLIPSE_HOME> <WORKSPACE_BASEDIR> <JIRA_PASSWD> <CURL_CMD> <DEFECT> 

Eli or Nigel, could you please take a look at this? This is urgent, since pre-commit builds cannot be run.

> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14908-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 17:44:30 2011
Return-Path: <common-issues-return-14908-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3E60D6D9C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 17:44:30 +0000 (UTC)
Received: (qmail 95088 invoked by uid 500); 16 May 2011 17:44:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95056 invoked by uid 500); 16 May 2011 17:44:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95048 invoked by uid 99); 16 May 2011 17:44:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 17:44:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 17:44:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 81A28CC2C5
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 17:43:47 +0000 (UTC)
Date: Mon, 16 May 2011 17:43:47 +0000 (UTC)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1782565336.15835.1305567827527.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034142#comment-13034142 ] 

Giridharan Kesavan commented on HADOOP-7291:
--------------------------------------------

https://builds.apache.org/hudson/job/PreCommit-Hdfs-Build/524/console

This is breaking all the test-patch build of hdfs 


> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14909-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 17:48:27 2011
Return-Path: <common-issues-return-14909-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C702F6242
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 17:48:27 +0000 (UTC)
Received: (qmail 245 invoked by uid 500); 16 May 2011 17:48:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 211 invoked by uid 500); 16 May 2011 17:48:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 196 invoked by uid 99); 16 May 2011 17:48:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 17:48:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 17:48:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 57907CC4BD
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 17:47:47 +0000 (UTC)
Date: Mon, 16 May 2011 17:47:47 +0000 (UTC)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1977167381.15871.1305568067355.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034146#comment-13034146 ] 

Giridharan Kesavan commented on HADOOP-7291:
--------------------------------------------

bq. Right, looking at the log more closely ("-c: command not found") it looks like GREP is not defined. How is test-patch being called?

test-patch is called through ant build.xml hudson-test-patch target.
${ANT_HOME}/bin/ant \
        -Dpatch.file=foobar \
        -Dscratch.dir=${WORKSPACE}/patchprocess \
        -Dsupport.dir=/homes/hudson/buildSupport  \
        -Dps.cmd=/bin/ps \
        -Dwget.cmd=/usr/bin/wget \
        -Djiracli.cmd=/homes/hudson/tools/jiracli-1.5/jira \
        -Dsvn.cmd=/usr/bin/svn \
        -Dgrep.cmd=/bin/grep \
        -Dpatch.cmd=/usr/bin/patch \
        -Dfindbugs.home=/homes/hudson/tools/findbugs/latest \
        -Dforrest.home=/homes/hudson/tools/forrest/latest \
        -Declipse.home=/homes/hudson/tools/eclipse/latest \
        -Dpython.home=/homes/hudson/tools/python/latest \
        -Djira.passwd=<> \
        -Dcurl.cmd=/usr/bin/curl \
        -Ddefect=HDFS-${ISSUE_NUM} \
        hudson-test-patch



> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14910-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 17:54:30 2011
Return-Path: <common-issues-return-14910-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F10206425
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 17:54:29 +0000 (UTC)
Received: (qmail 8748 invoked by uid 500); 16 May 2011 17:54:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8719 invoked by uid 500); 16 May 2011 17:54:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8711 invoked by uid 99); 16 May 2011 17:54:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 17:54:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 17:54:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6AEC5CC72D
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 17:53:47 +0000 (UTC)
Date: Mon, 16 May 2011 17:53:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1939893547.15883.1305568427434.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034148#comment-13034148 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7291:
------------------------------------------------

> ... This is urgent, since pre-commit builds cannot be run.

How about we temporarily revert the committed patch?

> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14911-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 17:58:28 2011
Return-Path: <common-issues-return-14911-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 188F86501
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 17:58:28 +0000 (UTC)
Received: (qmail 14224 invoked by uid 500); 16 May 2011 17:58:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14185 invoked by uid 500); 16 May 2011 17:58:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14177 invoked by uid 99); 16 May 2011 17:58:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 17:58:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 17:58:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7320BCC94E
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 17:57:47 +0000 (UTC)
Date: Mon, 16 May 2011 17:57:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <305644102.15898.1305568667468.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034153#comment-13034153 ] 

Matt Foley commented on HADOOP-6671:
------------------------------------

Alejandro, great to see someone working on this!

Would you please add to your "still to be done" list --
* Eclipse integration, especially:
** Eclipse project files generation (hopefully more reliably, automatically, and with less maintenance effort than currently);
** JUnit execution under the Eclipse-native "Java Builder" in the IDE (which works fine today as long as the dependencies and Build Path have been made available to Eclipse in the way it needs - which today usually requires changes to those Eclipse project files).

If this was so obvious you forgot to mention it :-) or if the maven plug-in for Eclipse will just make this happen, that's great, please let me know.  But please do include this in the scope of work, because some of us are very dependent on the Eclipse IDE for our quality of life!  Thanks.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>         Attachments: build.png, hadoop-commons-maven.patch, mvn-layout.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14912-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 18:02:30 2011
Return-Path: <common-issues-return-14912-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 802DD638B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 18:02:30 +0000 (UTC)
Received: (qmail 25386 invoked by uid 500); 16 May 2011 18:02:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25354 invoked by uid 500); 16 May 2011 18:02:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25346 invoked by uid 99); 16 May 2011 18:02:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 18:02:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 18:02:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E9662CCB86
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 18:01:47 +0000 (UTC)
Date: Mon, 16 May 2011 18:01:47 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <272116326.15953.1305568907952.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034156#comment-13034156 ] 

Suresh Srinivas commented on HADOOP-7291:
-----------------------------------------

I have reverted the patch for now.

> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14913-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 18:21:35 2011
Return-Path: <common-issues-return-14913-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B329763AD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 18:21:35 +0000 (UTC)
Received: (qmail 61859 invoked by uid 500); 16 May 2011 18:21:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61832 invoked by uid 500); 16 May 2011 18:21:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61817 invoked by uid 99); 16 May 2011 18:21:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 18:21:35 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 18:21:32 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3F11CCD3B9
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 18:20:52 +0000 (UTC)
Date: Mon, 16 May 2011 18:20:52 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2106674625.16038.1305570052254.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034169#comment-13034169 ] 

Luke Lu commented on HADOOP-6671:
---------------------------------

bq. if the maven plug-in for Eclipse will just make this happen, that's great, please let me know.

Confirmed. mvn eclipse:eclipse should suffice. If you use m2eclipse IDE plugin than the step is not needed either. BTW, NetBeans supports maven projects natively (without any plugins).

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>         Attachments: build.png, hadoop-commons-maven.patch, mvn-layout.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14914-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 19:22:29 2011
Return-Path: <common-issues-return-14914-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 05C1C6465
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 19:22:29 +0000 (UTC)
Received: (qmail 58469 invoked by uid 500); 16 May 2011 19:22:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58432 invoked by uid 500); 16 May 2011 19:22:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58291 invoked by uid 99); 16 May 2011 19:22:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 19:22:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 19:22:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 57E9ECCD9F
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 19:21:48 +0000 (UTC)
Date: Mon, 16 May 2011 19:21:48 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <980162350.16264.1305573708356.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034212#comment-13034212 ] 

Alejandro Abdelnur commented on HADOOP-6671:
--------------------------------------------

Glad to see people are interested in this.

Luke, thanks for the Eclipse info (I don't use Eclipse. I use IntelliJ, IntelliJ integrates nicely with Maven, you just open the POM as a project and you are  done).

JUnit execution should work fine if you first compile the testcases from Maven (running 'mvn test -DskipTests'), I do that trick for IntelliJ. And, if I'm not mistaken you should be able to configure the IDE to this automatically (or an IDE click).



> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>         Attachments: build.png, hadoop-commons-maven.patch, mvn-layout.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14915-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 19:43:28 2011
Return-Path: <common-issues-return-14915-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6C63860E7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 19:43:28 +0000 (UTC)
Received: (qmail 99185 invoked by uid 500); 16 May 2011 19:43:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99127 invoked by uid 500); 16 May 2011 19:43:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99111 invoked by uid 99); 16 May 2011 19:43:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 19:43:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 19:43:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 20B4ACC874
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 19:42:48 +0000 (UTC)
Date: Mon, 16 May 2011 19:42:48 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1539780569.16356.1305574968130.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034233#comment-13034233 ] 

Eli Collins commented on HADOOP-7291:
-------------------------------------

Nigel and I discussed. We need to:
# Apply the HADOOP-7291.patch again
# Update the Hudson job to not pass the python argument 

I'm doing #1, Nigel is doing #2.

Apologies for the disruption guys!

> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14916-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 19:47:27 2011
Return-Path: <common-issues-return-14916-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CEB8761D8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 19:47:27 +0000 (UTC)
Received: (qmail 7511 invoked by uid 500); 16 May 2011 19:47:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7465 invoked by uid 500); 16 May 2011 19:47:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7443 invoked by uid 99); 16 May 2011 19:47:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 19:47:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 19:47:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6D9C9CCB55
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 19:46:47 +0000 (UTC)
Date: Mon, 16 May 2011 19:46:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1229022541.16370.1305575207445.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7295) Create a test-patch script for
 Hudson
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Create a test-patch script for Hudson
-------------------------------------

                 Key: HADOOP-7295
                 URL: https://issues.apache.org/jira/browse/HADOOP-7295
             Project: Hadoop Common
          Issue Type: Task
            Reporter: Eli Collins


We should create a script that Hudson uses to execute test-patch that is in source control so modifications to test-patch.sh arguments can be done w/o updating Hudson.

The script would execute the following, and take just the password as an argument.

{noformat}
${ANT_HOME}/bin/ant \
        -Dpatch.file=foobar \
        -Dscratch.dir=${WORKSPACE}/patchprocess \
        -Dsupport.dir=/homes/hudson/buildSupport  \
        -Dps.cmd=/bin/ps \
        -Dwget.cmd=/usr/bin/wget \
        -Djiracli.cmd=/homes/hudson/tools/jiracli-1.5/jira \
        -Dsvn.cmd=/usr/bin/svn \
        -Dgrep.cmd=/bin/grep \
        -Dpatch.cmd=/usr/bin/patch \
        -Dfindbugs.home=/homes/hudson/tools/findbugs/latest \
        -Dforrest.home=/homes/hudson/tools/forrest/latest \
        -Declipse.home=/homes/hudson/tools/eclipse/latest \
        -Dpython.home=/homes/hudson/tools/python/latest \
        -Djira.passwd=******** \
        -Dcurl.cmd=/usr/bin/curl \
        -Ddefect=HADOOP-${ISSUE_NUM} \
        hudson-test-patch
{noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14917-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 19:57:30 2011
Return-Path: <common-issues-return-14917-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DBA756956
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 19:57:29 +0000 (UTC)
Received: (qmail 27675 invoked by uid 500); 16 May 2011 19:57:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27641 invoked by uid 500); 16 May 2011 19:57:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27633 invoked by uid 99); 16 May 2011 19:57:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 19:57:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 19:57:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 793CECCF39
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 19:56:47 +0000 (UTC)
Date: Mon, 16 May 2011 19:56:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1212620270.16403.1305575807492.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1229022541.16370.1305575207445.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7295) Create a test-patch script for
 Hudson
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034245#comment-13034245 ] 

Owen O'Malley commented on HADOOP-7295:
---------------------------------------

I'm not sure how useful this is. There is way too much embedded context here for it to be useful even in another hudson environment.

> Create a test-patch script for Hudson
> -------------------------------------
>
>                 Key: HADOOP-7295
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7295
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>
> We should create a script that Hudson uses to execute test-patch that is in source control so modifications to test-patch.sh arguments can be done w/o updating Hudson.
> The script would execute the following, and take just the password as an argument.
> {noformat}
> ${ANT_HOME}/bin/ant \
>         -Dpatch.file=foobar \
>         -Dscratch.dir=${WORKSPACE}/patchprocess \
>         -Dsupport.dir=/homes/hudson/buildSupport  \
>         -Dps.cmd=/bin/ps \
>         -Dwget.cmd=/usr/bin/wget \
>         -Djiracli.cmd=/homes/hudson/tools/jiracli-1.5/jira \
>         -Dsvn.cmd=/usr/bin/svn \
>         -Dgrep.cmd=/bin/grep \
>         -Dpatch.cmd=/usr/bin/patch \
>         -Dfindbugs.home=/homes/hudson/tools/findbugs/latest \
>         -Dforrest.home=/homes/hudson/tools/forrest/latest \
>         -Declipse.home=/homes/hudson/tools/eclipse/latest \
>         -Dpython.home=/homes/hudson/tools/python/latest \
>         -Djira.passwd=******** \
>         -Dcurl.cmd=/usr/bin/curl \
>         -Ddefect=HADOOP-${ISSUE_NUM} \
>         hudson-test-patch
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14918-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 20:01:28 2011
Return-Path: <common-issues-return-14918-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3FECF62AF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 20:01:28 +0000 (UTC)
Received: (qmail 35001 invoked by uid 500); 16 May 2011 20:01:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34973 invoked by uid 500); 16 May 2011 20:01:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34955 invoked by uid 99); 16 May 2011 20:01:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 20:01:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 20:01:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9A210CC193
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 20:00:47 +0000 (UTC)
Date: Mon, 16 May 2011 20:00:47 +0000 (UTC)
From: "Vitalii Tymchyshyn (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <452561078.16431.1305576047627.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2007918179.15449.1305558467414.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7294) FileUtil uses wrong stat command
 for FreeBSD
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034251#comment-13034251 ] 

Vitalii Tymchyshyn commented on HADOOP-7294:
--------------------------------------------

Actually there is support for Windows and Mac in the code, so the thing to be changed is to rename Mac to BSD and treat FreeBSD as BSD. In general, I'd prefer it to be configurable.

> FileUtil uses wrong stat command for FreeBSD
> --------------------------------------------
>
>                 Key: HADOOP-7294
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7294
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0
>         Environment: FreeBSD 8.0-STABLE
>            Reporter: Vitalii Tymchyshyn
>
> I get next exception when try to use append:
> 2011-05-16 17:07:54,648 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.112.0.207:50010, storageID=DS-1047171559-
> 10.112.0.207-50010-1302796304164, infoPort=50075, ipcPort=50020):DataXceiver
> java.io.IOException: Failed to get link count on file /var/data/hdfs/data/current/finalized/subdir26/subdir17/subdir55/blk_-1266943884751786595:
>  message=null; error=stat: illegal option -- c; exit value=1
>         at org.apache.hadoop.fs.FileUtil.createIOException(FileUtil.java:709)
>         at org.apache.hadoop.fs.FileUtil.access$000(FileUtil.java:42)
>         at org.apache.hadoop.fs.FileUtil$HardLink.getLinkCount(FileUtil.java:682)
>         at org.apache.hadoop.hdfs.server.datanode.ReplicaInfo.unlinkBlock(ReplicaInfo.java:215)
>         at org.apache.hadoop.hdfs.server.datanode.FSDataset.append(FSDataset.java:1116)
> It seems that FreeBSD is treated like UNIX and so calls 'stat -c%h', while FreeBSD is much more like Mac (since they have same BSD roots):
> $ stat --help
> stat: illegal option -- -
> usage: stat [-FlLnqrsx] [-f format] [-t timefmt] [file ...]
> $ stat -f%l a_file
> 1

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14919-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 20:05:28 2011
Return-Path: <common-issues-return-14919-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F046C60F4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 20:05:27 +0000 (UTC)
Received: (qmail 38387 invoked by uid 500); 16 May 2011 20:05:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38347 invoked by uid 500); 16 May 2011 20:05:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38339 invoked by uid 99); 16 May 2011 20:05:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 20:05:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 20:05:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 85888CC3EB
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 20:04:47 +0000 (UTC)
Date: Mon, 16 May 2011 20:04:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <933340404.16438.1305576287543.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <177713183.11069.1305315587397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7290) Unit test failure in
 TestUserGroupInformation.testGetServerSideGroups
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034254#comment-13034254 ] 

Hudson commented on HADOOP-7290:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #601 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/601/])
    

> Unit test failure in TestUserGroupInformation.testGetServerSideGroups
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-7290
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7290
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.23.0
>         Environment: Linux 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:24 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
>            Reporter: Trevor Robinson
>            Assignee: Trevor Robinson
>            Priority: Minor
>              Labels: test
>         Attachments: TestUserGroupInformation-id-order.patch
>
>
> Testsuite: org.apache.hadoop.security.TestUserGroupInformation
> Tests run: 14, Failures: 1, Errors: 0, Time elapsed: 0.278 sec
> ------------- Standard Output ---------------
> trobinson:users guest git
> ------------- ---------------- ---------------
> Testcase: testGetServerSideGroups took 0.051 sec
>        FAILED
> expected:<g[ues]t> but was:<g[i]t>
> junit.framework.AssertionFailedError: expected:<g[ues]t> but was:<g[i]t>
>        at org.apache.hadoop.security.TestUserGroupInformation.testGetServerSideGroups(TestUserGroupInformation.java:94)
> It seems like the test is assuming that the groups returned by UserGroupInformation.getGroupNames() are in the same order as those returned by executing `id -Gn`. getGroupNames() only documents that the primary group is first, and `man id` doesn't document any ordering, so it seems like the test needs to be reworked to remove that assumption.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14920-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 20:05:28 2011
Return-Path: <common-issues-return-14920-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 61F506101
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 20:05:28 +0000 (UTC)
Received: (qmail 38608 invoked by uid 500); 16 May 2011 20:05:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38584 invoked by uid 500); 16 May 2011 20:05:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38576 invoked by uid 99); 16 May 2011 20:05:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 20:05:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 20:05:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EAC64CC3FC
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 20:04:47 +0000 (UTC)
Date: Mon, 16 May 2011 20:04:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1826467211.16451.1305576287958.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <32042405.8638.1305244247592.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7285) Refactor FsShell's test
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034257#comment-13034257 ] 

Hudson commented on HADOOP-7285:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #601 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/601/])
    

> Refactor FsShell's test
> -----------------------
>
>                 Key: HADOOP-7285
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7285
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7285-2.patch, HADOOP-7285-3.patch, HADOOP-7285.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14921-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 20:05:28 2011
Return-Path: <common-issues-return-14921-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 90625610E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 20:05:28 +0000 (UTC)
Received: (qmail 38799 invoked by uid 500); 16 May 2011 20:05:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38766 invoked by uid 500); 16 May 2011 20:05:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38672 invoked by uid 99); 16 May 2011 20:05:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 20:05:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 20:05:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 05658CC3FE
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 20:04:48 +0000 (UTC)
Date: Mon, 16 May 2011 20:04:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <109430860.16453.1305576288018.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <957076305.12265.1297477317500.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7137) Remove hod contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034258#comment-13034258 ] 

Hudson commented on HADOOP-7137:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #601 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/601/])
    

> Remove hod contrib
> ------------------
>
>                 Key: HADOOP-7137
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7137
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7137.patch, HADOOP-7137.patch
>
>
> As per vote on general@ (http://mail-archives.apache.org/mod_mbox/hadoop-general/201102.mbox/%3CAC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com%3E) I will 
> svn remove common/trunk/src/contrib/hod
> using this Jira.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14922-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 20:05:30 2011
Return-Path: <common-issues-return-14922-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0C09D6122
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 20:05:30 +0000 (UTC)
Received: (qmail 39329 invoked by uid 500); 16 May 2011 20:05:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39288 invoked by uid 500); 16 May 2011 20:05:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39280 invoked by uid 99); 16 May 2011 20:05:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 20:05:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 20:05:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id ABD65CC3F3
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 20:04:47 +0000 (UTC)
Date: Mon, 16 May 2011 20:04:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <916913503.16444.1305576287700.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034255#comment-13034255 ] 

Hudson commented on HADOOP-7291:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #601 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/601/])
    HADOOP-7291. Update Hudson job not to run test-contrib. Contributed by Nigel Daley


> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14923-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 20:05:30 2011
Return-Path: <common-issues-return-14923-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3C6576135
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 20:05:30 +0000 (UTC)
Received: (qmail 39532 invoked by uid 500); 16 May 2011 20:05:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39482 invoked by uid 500); 16 May 2011 20:05:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39422 invoked by uid 99); 16 May 2011 20:05:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 20:05:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 20:05:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C6943CC3F6
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 20:04:47 +0000 (UTC)
Date: Mon, 16 May 2011 20:04:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1845314585.16447.1305576287810.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <510617684.3042.1300172489780.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7192) fs -stat docs aren't updated to
 reflect the format features
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034256#comment-13034256 ] 

Hudson commented on HADOOP-7192:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #601 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/601/])
    

> fs -stat docs aren't updated to reflect the format features
> -----------------------------------------------------------
>
>                 Key: HADOOP-7192
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7192
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: 0.21.0
>         Environment: Linux / 0.20
>            Reporter: Harsh J Chouraria
>            Assignee: Harsh J Chouraria
>            Priority: Trivial
>              Labels: documentaion
>             Fix For: 0.22.0
>
>         Attachments: hadoop.common.fsstatdoc.r1.diff
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> The html docs of the 'fs -stat' command (that is found listed in the File System Shell Guide), does not seem to have the formatting abilities of -stat explained (along with the options).
> Like 'fs -help', the docs must also reflect the latest available features.
> I shall attach a doc-fix patch shortly.
> If anyone has other discrepancies to point out in the web version of the guide, please do so :)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14924-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 20:05:50 2011
Return-Path: <common-issues-return-14924-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 746CE614B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 20:05:50 +0000 (UTC)
Received: (qmail 41160 invoked by uid 500); 16 May 2011 20:05:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41115 invoked by uid 500); 16 May 2011 20:05:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41079 invoked by uid 99); 16 May 2011 20:05:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 20:05:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 20:05:47 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4977ECC409
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 20:04:48 +0000 (UTC)
Date: Mon, 16 May 2011 20:04:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1901831897.16462.1305576288297.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <539645178.10940.1305312827448.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7289) ivy: test conf should not extend
 common conf
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034259#comment-13034259 ] 

Hudson commented on HADOOP-7289:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #601 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/601/])
    

> ivy: test conf should not extend common conf
> --------------------------------------------
>
>                 Key: HADOOP-7289
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7289
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Eric Yang
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7289-1.patch, HADOOP-7289.patch, c7289_20110513.patch, c7289_20110513b.patch
>
>
> Otherwise, the same jars will appear in both {{build/ivy/lib/Hadoop-Common/common/}} and {{build/ivy/lib/Hadoop-Common/test/}}.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14925-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 20:09:30 2011
Return-Path: <common-issues-return-14925-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3BD23624D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 20:09:30 +0000 (UTC)
Received: (qmail 45725 invoked by uid 500); 16 May 2011 20:09:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45683 invoked by uid 500); 16 May 2011 20:09:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45675 invoked by uid 99); 16 May 2011 20:09:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 20:09:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 20:09:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 98BFBCC6D9
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 20:08:47 +0000 (UTC)
Date: Mon, 16 May 2011 20:08:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1403526156.16474.1305576527622.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034261#comment-13034261 ] 

Hudson commented on HADOOP-7291:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk-Commit #675 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/675/])
    HADOOP-7291. Update Hudson job not to run test-contrib. Contributed by Nigel Daley


> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14926-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 20:25:30 2011
Return-Path: <common-issues-return-14926-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F17A96ADA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 20:25:29 +0000 (UTC)
Received: (qmail 72191 invoked by uid 500); 16 May 2011 20:25:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72160 invoked by uid 500); 16 May 2011 20:25:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72129 invoked by uid 99); 16 May 2011 20:25:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 20:25:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 20:25:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6221ECCE50
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 20:24:47 +0000 (UTC)
Date: Mon, 16 May 2011 20:24:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <773602986.16554.1305577487398.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-1117) DFS Scalability: When the namenode
 is restarted it consumes 80% CPU
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034279#comment-13034279 ] 

Matt Foley commented on HADOOP-1117:
------------------------------------

Hey Eli, did you notice the timestamp on this Jira? :-)

Current trunk code is fine.

> DFS Scalability: When the namenode is restarted it consumes 80% CPU
> -------------------------------------------------------------------
>
>                 Key: HADOOP-1117
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1117
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.12.0
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>            Priority: Blocker
>             Fix For: 0.12.1
>
>         Attachments: CpuPendingTransfer3.patch
>
>
> When the namenode is restarted, the datanodes register and each block is inserted into neededReplication. When the namenode exists, safemode it sees starts processing neededReplication. It picks up a block from neededReplication, sees that it has already has the required number of replicas, and continues to the next block in neededReplication. The blocks remain in neededReplication permanentlyhe namenode worker thread to scans this huge list of blocks once every 3 seconds. This consumes plenty of CPU on the namenode.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14927-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 20:33:31 2011
Return-Path: <common-issues-return-14927-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9E3506DB5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 20:33:31 +0000 (UTC)
Received: (qmail 88073 invoked by uid 500); 16 May 2011 20:33:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88014 invoked by uid 500); 16 May 2011 20:33:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87889 invoked by uid 99); 16 May 2011 20:33:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 20:33:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 20:33:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B28E6CC1F8
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 20:32:47 +0000 (UTC)
Date: Mon, 16 May 2011 20:32:47 +0000 (UTC)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1072940974.16581.1305577967728.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2007918179.15449.1305558467414.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7294) FileUtil uses wrong stat command
 for FreeBSD
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034287#comment-13034287 ] 

Jakob Homan commented on HADOOP-7294:
-------------------------------------

In general, we try to follow BSD semantics (http://hadoop.apache.org/hdfs/docs/r0.21.0/hdfs_permissions_guide.html).

> FileUtil uses wrong stat command for FreeBSD
> --------------------------------------------
>
>                 Key: HADOOP-7294
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7294
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0
>         Environment: FreeBSD 8.0-STABLE
>            Reporter: Vitalii Tymchyshyn
>
> I get next exception when try to use append:
> 2011-05-16 17:07:54,648 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.112.0.207:50010, storageID=DS-1047171559-
> 10.112.0.207-50010-1302796304164, infoPort=50075, ipcPort=50020):DataXceiver
> java.io.IOException: Failed to get link count on file /var/data/hdfs/data/current/finalized/subdir26/subdir17/subdir55/blk_-1266943884751786595:
>  message=null; error=stat: illegal option -- c; exit value=1
>         at org.apache.hadoop.fs.FileUtil.createIOException(FileUtil.java:709)
>         at org.apache.hadoop.fs.FileUtil.access$000(FileUtil.java:42)
>         at org.apache.hadoop.fs.FileUtil$HardLink.getLinkCount(FileUtil.java:682)
>         at org.apache.hadoop.hdfs.server.datanode.ReplicaInfo.unlinkBlock(ReplicaInfo.java:215)
>         at org.apache.hadoop.hdfs.server.datanode.FSDataset.append(FSDataset.java:1116)
> It seems that FreeBSD is treated like UNIX and so calls 'stat -c%h', while FreeBSD is much more like Mac (since they have same BSD roots):
> $ stat --help
> stat: illegal option -- -
> usage: stat [-FlLnqrsx] [-f format] [-t timefmt] [file ...]
> $ stat -f%l a_file
> 1

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14928-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 20:39:28 2011
Return-Path: <common-issues-return-14928-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 279226475
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 20:39:28 +0000 (UTC)
Received: (qmail 97266 invoked by uid 500); 16 May 2011 20:39:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97195 invoked by uid 500); 16 May 2011 20:39:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97131 invoked by uid 99); 16 May 2011 20:39:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 20:39:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 20:39:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 90B0CCC5F0
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 20:38:47 +0000 (UTC)
Date: Mon, 16 May 2011 20:38:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1389909957.16601.1305578327589.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1619049648.8641.1305244247668.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7286) Refactor FsShell's du/dus/df
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7286:
--------------------------------

    Attachment: HADOOP-7286-3.patch

Addresses Aaron and Todd's comments.  Added deprecation warnings for dus, rmr, lsr.

> Refactor FsShell's du/dus/df
> ----------------------------
>
>                 Key: HADOOP-7286
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7286
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7286-2.patch, HADOOP-7286-3.patch, HADOOP-7286.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14929-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 21:34:30 2011
Return-Path: <common-issues-return-14929-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CF2A56CCA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 21:34:30 +0000 (UTC)
Received: (qmail 15812 invoked by uid 500); 16 May 2011 21:34:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15708 invoked by uid 500); 16 May 2011 21:34:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15685 invoked by uid 99); 16 May 2011 21:34:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 21:34:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 21:34:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9DD03CC08B
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 21:33:47 +0000 (UTC)
Date: Mon, 16 May 2011 21:33:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1003565032.16776.1305581627643.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-1117) DFS Scalability: When the namenode
 is restarted it consumes 80% CPU
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-1117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034330#comment-13034330 ] 

Eli Collins commented on HADOOP-1117:
-------------------------------------

Ha. Thanks Matt =)

> DFS Scalability: When the namenode is restarted it consumes 80% CPU
> -------------------------------------------------------------------
>
>                 Key: HADOOP-1117
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1117
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.12.0
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>            Priority: Blocker
>             Fix For: 0.12.1
>
>         Attachments: CpuPendingTransfer3.patch
>
>
> When the namenode is restarted, the datanodes register and each block is inserted into neededReplication. When the namenode exists, safemode it sees starts processing neededReplication. It picks up a block from neededReplication, sees that it has already has the required number of replicas, and continues to the next block in neededReplication. The blocks remain in neededReplication permanentlyhe namenode worker thread to scans this huge list of blocks once every 3 seconds. This consumes plenty of CPU on the namenode.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14930-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 21:41:30 2011
Return-Path: <common-issues-return-14930-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D15F7612F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 21:41:30 +0000 (UTC)
Received: (qmail 40345 invoked by uid 500); 16 May 2011 21:41:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40306 invoked by uid 500); 16 May 2011 21:41:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40255 invoked by uid 99); 16 May 2011 21:41:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 21:41:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 21:41:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3D11DCC686
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 21:40:48 +0000 (UTC)
Date: Mon, 16 May 2011 21:40:48 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <201087021.16828.1305582048246.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6929) RPC should have a way to pass
 Security information other than protocol annotations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034345#comment-13034345 ] 

Luke Lu commented on HADOOP-6929:
---------------------------------

+1, lgtm. This is required for mrv2.

> RPC should have a way to pass Security information other than protocol annotations
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6929
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6929
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: Hadoop-6929_v1.patch
>
>
> Currently Hadoop RPC allows protocol annotations as the only way to pass security information. This becomes a problem if protocols are generated and not hand written. For example protocols generated via Avro and passed over Avro tunnel (AvroRpcEngine.java) can't pass the security information.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14931-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 21:51:31 2011
Return-Path: <common-issues-return-14931-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3CAD96AA4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 21:51:31 +0000 (UTC)
Received: (qmail 60851 invoked by uid 500); 16 May 2011 21:51:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60797 invoked by uid 500); 16 May 2011 21:51:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60740 invoked by uid 99); 16 May 2011 21:51:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 21:51:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 21:51:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B4F08CCC9B
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 21:50:47 +0000 (UTC)
Date: Mon, 16 May 2011 21:50:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <680844338.16891.1305582647737.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1619049648.8641.1305244247668.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7286) Refactor FsShell's du/dus/df
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034353#comment-13034353 ] 

Aaron T. Myers commented on HADOOP-7286:
----------------------------------------

Latest patch looks to be almost there. A few nits:

# Comment in {{FsUsage.java}} still references "viewing capacity." You also might as well explicitly say something like "this is the base class for the {{df}} and {{du}} commands.
# I think the {{deprecatedFor(...)}} interface could be clearer. I think "{{boolean isDeprecated(...)}}" and something like "{{String getPreferredCommand(...)}}" would be a lot quicker to understand.

> Refactor FsShell's du/dus/df
> ----------------------------
>
>                 Key: HADOOP-7286
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7286
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7286-2.patch, HADOOP-7286-3.patch, HADOOP-7286.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14932-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 22:02:29 2011
Return-Path: <common-issues-return-14932-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D9FBF61AE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 22:02:28 +0000 (UTC)
Received: (qmail 96239 invoked by uid 500); 16 May 2011 22:02:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96196 invoked by uid 500); 16 May 2011 22:02:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96148 invoked by uid 99); 16 May 2011 22:02:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 22:02:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 22:02:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 191D0CD1EE
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 22:01:48 +0000 (UTC)
Date: Mon, 16 May 2011 22:01:48 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <840135816.16942.1305583308099.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu reassigned HADOOP-7144:
-------------------------------

    Assignee: Luke Lu

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>              Labels: jmx
>             Fix For: 0.23.0
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14933-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 22:14:30 2011
Return-Path: <common-issues-return-14933-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D26C76E5A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 22:14:30 +0000 (UTC)
Received: (qmail 26236 invoked by uid 500); 16 May 2011 22:14:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26183 invoked by uid 500); 16 May 2011 22:14:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26136 invoked by uid 99); 16 May 2011 22:14:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 22:14:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 22:14:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 05691CD6A7
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 22:13:48 +0000 (UTC)
Date: Mon, 16 May 2011 22:13:48 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <612520859.17015.1305584028018.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1619049648.8641.1305244247668.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7286) Refactor FsShell's du/dus/df
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034376#comment-13034376 ] 

Hadoop QA commented on HADOOP-7286:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479380/HADOOP-7286-3.patch
  against trunk revision 1103856.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/456//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/456//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/456//console

This message is automatically generated.

> Refactor FsShell's du/dus/df
> ----------------------------
>
>                 Key: HADOOP-7286
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7286
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7286-2.patch, HADOOP-7286-3.patch, HADOOP-7286.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14934-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 22:18:30 2011
Return-Path: <common-issues-return-14934-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7F824600F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 22:18:30 +0000 (UTC)
Received: (qmail 32389 invoked by uid 500); 16 May 2011 22:18:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32350 invoked by uid 500); 16 May 2011 22:18:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32331 invoked by uid 99); 16 May 2011 22:18:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 22:18:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 22:18:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 900ACCD7D8
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 22:17:47 +0000 (UTC)
Date: Mon, 16 May 2011 22:17:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <388315238.17025.1305584267586.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034378#comment-13034378 ] 

Hudson commented on HADOOP-7291:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk-Commit #676 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/676/])
    MAPREDUCE-2499. MR part of HADOOP-7291. Contributed by Eli Collins


> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14935-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 22:28:28 2011
Return-Path: <common-issues-return-14935-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E230B69EA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 22:28:27 +0000 (UTC)
Received: (qmail 39409 invoked by uid 500); 16 May 2011 22:28:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39379 invoked by uid 500); 16 May 2011 22:28:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39371 invoked by uid 99); 16 May 2011 22:28:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 22:28:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 22:28:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6D8EACDA84
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 22:27:47 +0000 (UTC)
Date: Mon, 16 May 2011 22:27:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1264698948.17047.1305584867445.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7291:
--------------------------------

    Attachment: HADOOP-7291-3.patch

After HDFS-1946 and MR-2499 the common, HDFS and MR precommit jobs run though looks like $GREP is not defined. This is because a new call to runContribTests was introduced in HADOOP-7291.patch which runs before parseArgs. We don't need this call. The attached patch removes it. 



> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291-3.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14936-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 22:32:29 2011
Return-Path: <common-issues-return-14936-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7FA1664C2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 22:32:29 +0000 (UTC)
Received: (qmail 41566 invoked by uid 500); 16 May 2011 22:32:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41541 invoked by uid 500); 16 May 2011 22:32:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41532 invoked by uid 99); 16 May 2011 22:32:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 22:32:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 22:32:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 18302CDB44
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 22:31:49 +0000 (UTC)
Date: Mon, 16 May 2011 22:31:49 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <206624094.17056.1305585109095.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034386#comment-13034386 ] 

Todd Lipcon commented on HADOOP-7291:
-------------------------------------

+1 for HADOOP-7291-3.patch

> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291-3.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14937-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 22:34:30 2011
Return-Path: <common-issues-return-14937-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AF6B96E82
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 22:34:30 +0000 (UTC)
Received: (qmail 47423 invoked by uid 500); 16 May 2011 22:34:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47379 invoked by uid 500); 16 May 2011 22:34:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47366 invoked by uid 99); 16 May 2011 22:34:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 22:34:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 22:34:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8FA31CDC85
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 22:33:47 +0000 (UTC)
Date: Mon, 16 May 2011 22:33:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <603774443.17067.1305585227585.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7291:
--------------------------------

    Status: Patch Available  (was: Reopened)

> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291-3.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14938-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 22:36:28 2011
Return-Path: <common-issues-return-14938-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A3EAA6F79
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 22:36:28 +0000 (UTC)
Received: (qmail 49828 invoked by uid 500); 16 May 2011 22:36:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49801 invoked by uid 500); 16 May 2011 22:36:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49793 invoked by uid 99); 16 May 2011 22:36:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 22:36:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 22:36:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2DDBDCDD46
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 22:35:48 +0000 (UTC)
Date: Mon, 16 May 2011 22:35:48 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1110589074.17078.1305585348184.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034389#comment-13034389 ] 

Hadoop QA commented on HADOOP-7291:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479396/HADOOP-7291-3.patch
  against trunk revision 1103931.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/458//console

This message is automatically generated.

> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291-3.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14939-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 22:38:29 2011
Return-Path: <common-issues-return-14939-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EA8696FFE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 22:38:29 +0000 (UTC)
Received: (qmail 51744 invoked by uid 500); 16 May 2011 22:38:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51710 invoked by uid 500); 16 May 2011 22:38:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51702 invoked by uid 99); 16 May 2011 22:38:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 22:38:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 22:38:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7820FCDDF0
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 22:37:47 +0000 (UTC)
Date: Mon, 16 May 2011 22:37:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <430434228.17085.1305585467488.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1619049648.8641.1305244247668.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7286) Refactor FsShell's du/dus/df
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7286:
--------------------------------

    Attachment: HADOOP-7286-4.patch

Per Aaron, switched back to is/get methods for deprecation queries, and updated comments to be more succinct.

> Refactor FsShell's du/dus/df
> ----------------------------
>
>                 Key: HADOOP-7286
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7286
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7286-2.patch, HADOOP-7286-3.patch, HADOOP-7286-4.patch, HADOOP-7286.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14940-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 22:42:29 2011
Return-Path: <common-issues-return-14940-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 278B660A7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 22:42:29 +0000 (UTC)
Received: (qmail 56047 invoked by uid 500); 16 May 2011 22:42:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55949 invoked by uid 500); 16 May 2011 22:42:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55682 invoked by uid 99); 16 May 2011 22:42:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 22:42:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 22:42:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 99F31CDF7B
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 22:41:47 +0000 (UTC)
Date: Mon, 16 May 2011 22:41:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1324181642.17097.1305585707627.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1619049648.8641.1305244247668.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7286) Refactor FsShell's du/dus/df
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034395#comment-13034395 ] 

Aaron T. Myers commented on HADOOP-7286:
----------------------------------------

+1, the patch looks good to me. Thanks a lot, Daryn.

> Refactor FsShell's du/dus/df
> ----------------------------
>
>                 Key: HADOOP-7286
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7286
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7286-2.patch, HADOOP-7286-3.patch, HADOOP-7286-4.patch, HADOOP-7286.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14941-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 22:44:30 2011
Return-Path: <common-issues-return-14941-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 390346147
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 22:44:30 +0000 (UTC)
Received: (qmail 59824 invoked by uid 500); 16 May 2011 22:44:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59795 invoked by uid 500); 16 May 2011 22:44:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59787 invoked by uid 99); 16 May 2011 22:44:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 22:44:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 22:44:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B79ABCC0E4
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 22:43:47 +0000 (UTC)
Date: Mon, 16 May 2011 22:43:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1776287126.17112.1305585827748.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7291:
--------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

Thanks Todd. I've committed this to trunk and 22. Verified that the precommit Hudson job runs now.

> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291-3.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14942-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 22:54:28 2011
Return-Path: <common-issues-return-14942-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 902D267F1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 22:54:28 +0000 (UTC)
Received: (qmail 71588 invoked by uid 500); 16 May 2011 22:54:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71482 invoked by uid 500); 16 May 2011 22:54:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71374 invoked by uid 99); 16 May 2011 22:54:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 22:54:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 22:54:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D6F22CC4E5
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 22:53:47 +0000 (UTC)
Date: Mon, 16 May 2011 22:53:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <458472779.17142.1305586427877.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034404#comment-13034404 ] 

Hudson commented on HADOOP-7291:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #602 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/602/])
    HADOOP-7291. Remove spurious call to runTestContrib. Contributed by Eli Collins


> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291-3.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14943-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 23:10:35 2011
Return-Path: <common-issues-return-14943-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A74056FEF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 23:10:35 +0000 (UTC)
Received: (qmail 91421 invoked by uid 500); 16 May 2011 23:10:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91378 invoked by uid 500); 16 May 2011 23:10:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91362 invoked by uid 99); 16 May 2011 23:10:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 23:10:35 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 23:10:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C52FACCB4E
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 23:09:47 +0000 (UTC)
Date: Mon, 16 May 2011 23:09:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <454491278.17189.1305587387804.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034414#comment-13034414 ] 

Hudson commented on HADOOP-7291:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk-Commit #677 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/677/])
    HADOOP-7291. Remove spurious call to runTestContrib. Contributed by Eli Collins


> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291-3.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14944-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 16 23:31:28 2011
Return-Path: <common-issues-return-14944-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1EB806F33
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 16 May 2011 23:31:28 +0000 (UTC)
Received: (qmail 11181 invoked by uid 500); 16 May 2011 23:31:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11153 invoked by uid 500); 16 May 2011 23:31:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11144 invoked by uid 99); 16 May 2011 23:31:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 23:31:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 May 2011 23:31:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8F27ACD140
	for <common-issues@hadoop.apache.org>; Mon, 16 May 2011 23:30:47 +0000 (UTC)
Date: Mon, 16 May 2011 23:30:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <122961875.17254.1305588647583.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1619049648.8641.1305244247668.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7286) Refactor FsShell's du/dus/df
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034423#comment-13034423 ] 

Todd Lipcon commented on HADOOP-7286:
-------------------------------------

The patch changes the output in a couple ways that are worth considering:
- It used to print a line saying "Found n items" at the top of the output
- The paths used to be fully qualified but no longer are

example output before patch for "hadoop fs -du /home/todd/tokyo-nosql/":
{noformat}
Found 3 items
3429427  file:/home/todd/tokyo-nosql/tokyo-nosql.pdf
1255650  file:/home/todd/tokyo-nosql/tokyo-nosql-slides-only.pdf
4798464  file:/home/todd/tokyo-nosql/tokyo-nosql.ppt
{noformat}

After:
{noformat}
3429427  /home/todd/tokyo-nosql/tokyo-nosql.pdf
1255650  /home/todd/tokyo-nosql/tokyo-nosql-slides-only.pdf
4798464  /home/todd/tokyo-nosql/tokyo-nosql.ppt
{noformat}

Are these conscious changes that we're OK with?

> Refactor FsShell's du/dus/df
> ----------------------------
>
>                 Key: HADOOP-7286
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7286
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7286-2.patch, HADOOP-7286-3.patch, HADOOP-7286-4.patch, HADOOP-7286.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14945-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 00:05:28 2011
Return-Path: <common-issues-return-14945-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 722686C26
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 00:05:28 +0000 (UTC)
Received: (qmail 41634 invoked by uid 500); 17 May 2011 00:05:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41594 invoked by uid 500); 17 May 2011 00:05:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41585 invoked by uid 99); 17 May 2011 00:05:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 00:05:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 00:05:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 17BF4CD8DD
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 00:04:48 +0000 (UTC)
Date: Tue, 17 May 2011 00:04:48 +0000 (UTC)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <642504288.17346.1305590688093.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034438#comment-13034438 ] 

Konstantin Boudnik commented on HADOOP-6671:
--------------------------------------------

Alejandro, I think {{mvn test-compile}} should do the same.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>         Attachments: build.png, hadoop-commons-maven.patch, mvn-layout.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14946-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 00:13:30 2011
Return-Path: <common-issues-return-14946-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 44A696D7F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 00:13:30 +0000 (UTC)
Received: (qmail 48401 invoked by uid 500); 17 May 2011 00:13:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48356 invoked by uid 500); 17 May 2011 00:13:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48340 invoked by uid 99); 17 May 2011 00:13:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 00:13:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 00:13:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 68BF9CDA73
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 00:12:47 +0000 (UTC)
Date: Tue, 17 May 2011 00:12:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2113194951.17355.1305591167425.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1619049648.8641.1305244247668.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7286) Refactor FsShell's du/dus/df
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034441#comment-13034441 ] 

Daryn Sharp commented on HADOOP-7286:
-------------------------------------

The changes were deliberate on my part.  Here's my thoughts:  FsShell tries to implement posix/unix-like commands, but does so in its own "uniquely" inconsistent fashion which is a surprise to any shell coder.  IMHO, if FsShell wants to be unix-like, then it's output should look unix-like, or at a minimum be self-consistent.

I think du is the only command that is "noticably" different, particularly with regard to the subtraction of the header.  I checked "du" on various systems and didn't find any that printed "Found XXX items".  Going forward, I think du should be able to traverse more than one level deep, at which point it becomes prohibitively expensive for du to sponge memory to batch up all the output just so it can print a total on the first line.

Regarding the path display, it's been a universal change to standardize how paths are displayed since the commands' errors and output used to be wildly inconsistent.  Typically what you give on the cmdline is now what you get back, just like you would expect from a unix-like system.  Although, I can tweak the du output to be absolute uris if necessary.

> Refactor FsShell's du/dus/df
> ----------------------------
>
>                 Key: HADOOP-7286
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7286
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7286-2.patch, HADOOP-7286-3.patch, HADOOP-7286-4.patch, HADOOP-7286.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14947-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 00:19:30 2011
Return-Path: <common-issues-return-14947-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EBF096103
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 00:19:29 +0000 (UTC)
Received: (qmail 55626 invoked by uid 500); 17 May 2011 00:19:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55583 invoked by uid 500); 17 May 2011 00:19:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55575 invoked by uid 99); 17 May 2011 00:19:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 00:19:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 00:19:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 789DBCDC83
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 00:18:47 +0000 (UTC)
Date: Tue, 17 May 2011 00:18:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <55262557.17388.1305591527490.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1619049648.8641.1305244247668.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7286) Refactor FsShell's du/dus/df
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7286:
--------------------------------

    Hadoop Flags: [Incompatible change]

That seems OK to me. I've marked this JIRA as an incompatible change - please add a release note that details the change in format.

> Refactor FsShell's du/dus/df
> ----------------------------
>
>                 Key: HADOOP-7286
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7286
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7286-2.patch, HADOOP-7286-3.patch, HADOOP-7286-4.patch, HADOOP-7286.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14948-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 00:27:27 2011
Return-Path: <common-issues-return-14948-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9B1CE63DA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 00:27:27 +0000 (UTC)
Received: (qmail 65304 invoked by uid 500); 17 May 2011 00:27:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65246 invoked by uid 500); 17 May 2011 00:27:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65237 invoked by uid 99); 17 May 2011 00:27:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 00:27:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 00:27:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 567DBCDEBA
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 00:26:47 +0000 (UTC)
Date: Tue, 17 May 2011 00:26:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <966607170.17400.1305592007351.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1619049648.8641.1305244247668.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7286) Refactor FsShell's du/dus/df
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7286:
--------------------------------

    Release Note: The "Found X items" header has been removed to more closely match unix.  The displayed paths now correspond to the command line arguments instead of always being a fully qualified URI.  Ie. The output will have relative paths if the command line arguments are relative paths.

> Refactor FsShell's du/dus/df
> ----------------------------
>
>                 Key: HADOOP-7286
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7286
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7286-2.patch, HADOOP-7286-3.patch, HADOOP-7286-4.patch, HADOOP-7286.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14949-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 00:33:28 2011
Return-Path: <common-issues-return-14949-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EB63A67D7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 00:33:27 +0000 (UTC)
Received: (qmail 69585 invoked by uid 500); 17 May 2011 00:33:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69563 invoked by uid 500); 17 May 2011 00:33:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69555 invoked by uid 99); 17 May 2011 00:33:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 00:33:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 00:33:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6855ECDFF6
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 00:32:47 +0000 (UTC)
Date: Tue, 17 May 2011 00:32:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <632281247.17404.1305592367423.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1619049648.8641.1305244247668.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7286) Refactor FsShell's du/dus/df
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034451#comment-13034451 ] 

Hadoop QA commented on HADOOP-7286:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479398/HADOOP-7286-4.patch
  against trunk revision 1103931.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/459//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/459//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/459//console

This message is automatically generated.

> Refactor FsShell's du/dus/df
> ----------------------------
>
>                 Key: HADOOP-7286
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7286
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7286-2.patch, HADOOP-7286-3.patch, HADOOP-7286-4.patch, HADOOP-7286.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14950-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 00:37:28 2011
Return-Path: <common-issues-return-14950-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E4A616A3E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 00:37:27 +0000 (UTC)
Received: (qmail 73807 invoked by uid 500); 17 May 2011 00:37:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73773 invoked by uid 500); 17 May 2011 00:37:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73765 invoked by uid 99); 17 May 2011 00:37:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 00:37:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 00:37:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 78982CD11B
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 00:36:47 +0000 (UTC)
Date: Tue, 17 May 2011 00:36:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <901054010.17421.1305592607490.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <866597877.11958.1305329567592.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7292) Metrics 2 TestSinkQueue is racy
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034453#comment-13034453 ] 

Todd Lipcon commented on HADOOP-7292:
-------------------------------------

Hi Luke. I think the CountdownLatch should be a local variable in newSleepingConsumerQueue -- otherwise, the latch gets reused between test functions.

> Metrics 2 TestSinkQueue is racy
> -------------------------------
>
>                 Key: HADOOP-7292
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7292
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.23.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7292-test-race-v1.patch
>
>
> The TestSinkQueue is racy (Thread.yield is not enough to guarantee other intended thread getting run), though it's the first time (from HADOOP-7289) I saw it manifested here.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14951-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 00:43:37 2011
Return-Path: <common-issues-return-14951-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 71F5D6BA4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 00:43:37 +0000 (UTC)
Received: (qmail 81256 invoked by uid 500); 17 May 2011 00:43:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81208 invoked by uid 500); 17 May 2011 00:43:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81167 invoked by uid 99); 17 May 2011 00:43:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 00:43:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 00:43:35 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E22A0CD366
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 00:42:54 +0000 (UTC)
Date: Tue, 17 May 2011 00:42:54 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <52900663.17469.1305592974923.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7269) S3 Native should allow customizable
 file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7269:
--------------------------------

    Status: Open  (was: Patch Available)

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: HADOOP-7269-S3-metadata-001.diff, HADOOP-7269-S3-metadata-002.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14952-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 00:49:29 2011
Return-Path: <common-issues-return-14952-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DE3A960B9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 00:49:28 +0000 (UTC)
Received: (qmail 93156 invoked by uid 500); 17 May 2011 00:49:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93118 invoked by uid 500); 17 May 2011 00:49:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93110 invoked by uid 99); 17 May 2011 00:49:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 00:49:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 00:49:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B01CECD448
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 00:48:47 +0000 (UTC)
Date: Tue, 17 May 2011 00:48:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1358563853.17474.1305593327718.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <866597877.11958.1305329567592.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7292) Metrics 2 TestSinkQueue is racy
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034460#comment-13034460 ] 

Luke Lu commented on HADOOP-7292:
---------------------------------

I was planning to use the latch for other tests as well. The latch will not be reused because junit creates a new instance of the test object for each test method: http://martinfowler.com/bliki/JunitNewInstance.html


> Metrics 2 TestSinkQueue is racy
> -------------------------------
>
>                 Key: HADOOP-7292
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7292
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.23.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7292-test-race-v1.patch
>
>
> The TestSinkQueue is racy (Thread.yield is not enough to guarantee other intended thread getting run), though it's the first time (from HADOOP-7289) I saw it manifested here.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14953-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 00:51:29 2011
Return-Path: <common-issues-return-14953-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D26A860E1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 00:51:29 +0000 (UTC)
Received: (qmail 95176 invoked by uid 500); 17 May 2011 00:51:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95152 invoked by uid 500); 17 May 2011 00:51:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95144 invoked by uid 99); 17 May 2011 00:51:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 00:51:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 00:51:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 56CBACD4A8
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 00:50:47 +0000 (UTC)
Date: Tue, 17 May 2011 00:50:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <684904726.17478.1305593447351.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <866597877.11958.1305329567592.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7292) Metrics 2 TestSinkQueue is racy
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034461#comment-13034461 ] 

Todd Lipcon commented on HADOOP-7292:
-------------------------------------

aha, I guess I learn something new every day :)

Nonetheless since it's only used in that local scope for now, why not minimize it? Makes it easier to understand that it's only triggered from that thread and awaited from the other.

> Metrics 2 TestSinkQueue is racy
> -------------------------------
>
>                 Key: HADOOP-7292
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7292
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.23.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7292-test-race-v1.patch
>
>
> The TestSinkQueue is racy (Thread.yield is not enough to guarantee other intended thread getting run), though it's the first time (from HADOOP-7289) I saw it manifested here.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14954-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 00:55:28 2011
Return-Path: <common-issues-return-14954-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DD978612F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 00:55:27 +0000 (UTC)
Received: (qmail 97330 invoked by uid 500); 17 May 2011 00:55:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97289 invoked by uid 500); 17 May 2011 00:55:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97281 invoked by uid 99); 17 May 2011 00:55:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 00:55:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 00:55:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 75885CD5DF
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 00:54:47 +0000 (UTC)
Date: Tue, 17 May 2011 00:54:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <541311677.17484.1305593687478.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1619049648.8641.1305244247668.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7286) Refactor FsShell's du/dus/df
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7286:
--------------------------------

    Release Note: The "Found X items" header on the output of the "du" command has been removed to more closely match unix. The displayed paths now correspond to the command line arguments instead of always being a fully qualified URI. For example, the output will have relative paths if the command line arguments are relative paths.  (was: The "Found X items" header has been removed to more closely match unix.  The displayed paths now correspond to the command line arguments instead of always being a fully qualified URI.  Ie. The output will have relative paths if the command line arguments are relative paths.)
    Hadoop Flags: [Incompatible change, Reviewed]  (was: [Incompatible change])

+1, committing to trunk. Thanks, Daryn!

> Refactor FsShell's du/dus/df
> ----------------------------
>
>                 Key: HADOOP-7286
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7286
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7286-2.patch, HADOOP-7286-3.patch, HADOOP-7286-4.patch, HADOOP-7286.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14955-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 00:55:30 2011
Return-Path: <common-issues-return-14955-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C47EB6148
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 00:55:30 +0000 (UTC)
Received: (qmail 98008 invoked by uid 500); 17 May 2011 00:55:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97978 invoked by uid 500); 17 May 2011 00:55:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97970 invoked by uid 99); 17 May 2011 00:55:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 00:55:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 00:55:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 50F32CD5E6
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 00:54:48 +0000 (UTC)
Date: Tue, 17 May 2011 00:54:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <272836745.17491.1305593688328.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1619049648.8641.1305244247668.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7286) Refactor FsShell's du/dus/df
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7286:
--------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
           Status: Resolved  (was: Patch Available)

> Refactor FsShell's du/dus/df
> ----------------------------
>
>                 Key: HADOOP-7286
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7286
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7286-2.patch, HADOOP-7286-3.patch, HADOOP-7286-4.patch, HADOOP-7286.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14956-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 01:05:29 2011
Return-Path: <common-issues-return-14956-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 87EEC6667
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 01:05:29 +0000 (UTC)
Received: (qmail 1503 invoked by uid 500); 17 May 2011 01:05:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1415 invoked by uid 500); 17 May 2011 01:05:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1372 invoked by uid 99); 17 May 2011 01:05:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 01:05:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 01:05:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1DCA6CD926
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 01:04:48 +0000 (UTC)
Date: Tue, 17 May 2011 01:04:48 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1737891034.17560.1305594288118.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <866597877.11958.1305329567592.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7292) Metrics 2 TestSinkQueue is racy
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-7292:
----------------------------

    Attachment: hadoop-7292-test-race-v2.patch

Patch v2 make the barrier local for better readability.

> Metrics 2 TestSinkQueue is racy
> -------------------------------
>
>                 Key: HADOOP-7292
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7292
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.23.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7292-test-race-v1.patch, hadoop-7292-test-race-v2.patch
>
>
> The TestSinkQueue is racy (Thread.yield is not enough to guarantee other intended thread getting run), though it's the first time (from HADOOP-7289) I saw it manifested here.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14957-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 01:05:32 2011
Return-Path: <common-issues-return-14957-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 542CA66EA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 01:05:32 +0000 (UTC)
Received: (qmail 3075 invoked by uid 500); 17 May 2011 01:05:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3033 invoked by uid 500); 17 May 2011 01:05:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2927 invoked by uid 99); 17 May 2011 01:05:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 01:05:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 01:05:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A4856CD958
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 01:04:48 +0000 (UTC)
Date: Tue, 17 May 2011 01:04:48 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2083574700.17571.1305594288670.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <866597877.11958.1305329567592.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7292) Metrics 2 TestSinkQueue is racy
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-7292:
----------------------------

    Status: Open  (was: Patch Available)

> Metrics 2 TestSinkQueue is racy
> -------------------------------
>
>                 Key: HADOOP-7292
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7292
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.23.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7292-test-race-v1.patch, hadoop-7292-test-race-v2.patch
>
>
> The TestSinkQueue is racy (Thread.yield is not enough to guarantee other intended thread getting run), though it's the first time (from HADOOP-7289) I saw it manifested here.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14958-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 01:07:27 2011
Return-Path: <common-issues-return-14958-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BCEB4674B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 01:07:27 +0000 (UTC)
Received: (qmail 7232 invoked by uid 500); 17 May 2011 01:07:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7200 invoked by uid 500); 17 May 2011 01:07:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7192 invoked by uid 99); 17 May 2011 01:07:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 01:07:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 01:07:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 60EA6CD9D4
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 01:06:47 +0000 (UTC)
Date: Tue, 17 May 2011 01:06:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1236983986.17574.1305594407393.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <866597877.11958.1305329567592.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7292) Metrics 2 TestSinkQueue is racy
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-7292:
----------------------------

    Status: Patch Available  (was: Open)

> Metrics 2 TestSinkQueue is racy
> -------------------------------
>
>                 Key: HADOOP-7292
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7292
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.23.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7292-test-race-v1.patch, hadoop-7292-test-race-v2.patch
>
>
> The TestSinkQueue is racy (Thread.yield is not enough to guarantee other intended thread getting run), though it's the first time (from HADOOP-7289) I saw it manifested here.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14959-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 01:09:27 2011
Return-Path: <common-issues-return-14959-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E7CBD6799
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 01:09:27 +0000 (UTC)
Received: (qmail 8329 invoked by uid 500); 17 May 2011 01:09:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8293 invoked by uid 500); 17 May 2011 01:09:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8285 invoked by uid 99); 17 May 2011 01:09:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 01:09:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 01:09:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 86E82CDA7C
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 01:08:47 +0000 (UTC)
Date: Tue, 17 May 2011 01:08:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1187897026.17584.1305594527549.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <866597877.11958.1305329567592.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7292) Metrics 2 TestSinkQueue is racy
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7292:
--------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

Committed to trunk.

> Metrics 2 TestSinkQueue is racy
> -------------------------------
>
>                 Key: HADOOP-7292
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7292
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.23.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7292-test-race-v1.patch, hadoop-7292-test-race-v2.patch
>
>
> The TestSinkQueue is racy (Thread.yield is not enough to guarantee other intended thread getting run), though it's the first time (from HADOOP-7289) I saw it manifested here.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14960-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 01:09:30 2011
Return-Path: <common-issues-return-14960-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1574A67A2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 01:09:30 +0000 (UTC)
Received: (qmail 8593 invoked by uid 500); 17 May 2011 01:09:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8549 invoked by uid 500); 17 May 2011 01:09:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8541 invoked by uid 99); 17 May 2011 01:09:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 01:09:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 01:09:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 78F81CDA7A
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 01:08:47 +0000 (UTC)
Date: Tue, 17 May 2011 01:08:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1342812716.17582.1305594527492.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <866597877.11958.1305329567592.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7292) Metrics 2 TestSinkQueue is racy
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034474#comment-13034474 ] 

Todd Lipcon commented on HADOOP-7292:
-------------------------------------

+1. Will commit to trunk momentarily (manually ran the one changed test case)

> Metrics 2 TestSinkQueue is racy
> -------------------------------
>
>                 Key: HADOOP-7292
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7292
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.23.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7292-test-race-v1.patch, hadoop-7292-test-race-v2.patch
>
>
> The TestSinkQueue is racy (Thread.yield is not enough to guarantee other intended thread getting run), though it's the first time (from HADOOP-7289) I saw it manifested here.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14961-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 01:15:27 2011
Return-Path: <common-issues-return-14961-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C2DF46AB0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 01:15:27 +0000 (UTC)
Received: (qmail 16395 invoked by uid 500); 17 May 2011 01:15:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16370 invoked by uid 500); 17 May 2011 01:15:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16362 invoked by uid 99); 17 May 2011 01:15:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 01:15:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 01:15:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6347DCDC47
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 01:14:47 +0000 (UTC)
Date: Tue, 17 May 2011 01:14:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <906014472.17608.1305594887403.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1619049648.8641.1305244247668.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7286) Refactor FsShell's du/dus/df
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034479#comment-13034479 ] 

Hudson commented on HADOOP-7286:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #603 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/603/])
    HADOOP-7286. Refactor the du/dus/df commands to conform to new FsCommand class. Contributed by Daryn Sharp.


> Refactor FsShell's du/dus/df
> ----------------------------
>
>                 Key: HADOOP-7286
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7286
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7286-2.patch, HADOOP-7286-3.patch, HADOOP-7286-4.patch, HADOOP-7286.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14962-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 02:24:31 2011
Return-Path: <common-issues-return-14962-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9C54E41E5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 02:24:31 +0000 (UTC)
Received: (qmail 60575 invoked by uid 500); 17 May 2011 02:24:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60506 invoked by uid 500); 17 May 2011 02:24:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60362 invoked by uid 99); 17 May 2011 02:24:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 02:24:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 02:24:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 691B2CDB86
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 02:23:47 +0000 (UTC)
Date: Tue, 17 May 2011 02:23:47 +0000 (UTC)
From: "Siddharth Seth (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1326019706.17705.1305599027427.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7296) The FsPermission(FsPermission)
 constructor does not use the sticky bit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

The FsPermission(FsPermission) constructor does not use the sticky bit
----------------------------------------------------------------------

                 Key: HADOOP-7296
                 URL: https://issues.apache.org/jira/browse/HADOOP-7296
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
            Reporter: Siddharth Seth
            Priority: Minor


The FsPermission(FsPermission) constructor copies u, g, o from the supplied FsPermission object but ignores the sticky bit.


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14963-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 02:26:27 2011
Return-Path: <common-issues-return-14963-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E0F2E41F0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 02:26:27 +0000 (UTC)
Received: (qmail 62002 invoked by uid 500); 17 May 2011 02:26:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61968 invoked by uid 500); 17 May 2011 02:26:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61960 invoked by uid 99); 17 May 2011 02:26:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 02:26:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 02:26:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6EC4CCDC22
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 02:25:47 +0000 (UTC)
Date: Tue, 17 May 2011 02:25:47 +0000 (UTC)
From: "Siddharth Seth (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <687982308.17710.1305599147450.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1326019706.17705.1305599027427.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7296) The FsPermission(FsPermission)
 constructor does not use the sticky bit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siddharth Seth updated HADOOP-7296:
-----------------------------------

    Attachment: HADOOP7296.patch

Patch.

> The FsPermission(FsPermission) constructor does not use the sticky bit
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-7296
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7296
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Siddharth Seth
>            Priority: Minor
>         Attachments: HADOOP7296.patch
>
>
> The FsPermission(FsPermission) constructor copies u, g, o from the supplied FsPermission object but ignores the sticky bit.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14964-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 02:28:30 2011
Return-Path: <common-issues-return-14964-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1E0D3423C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 02:28:30 +0000 (UTC)
Received: (qmail 63334 invoked by uid 500); 17 May 2011 02:28:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63300 invoked by uid 500); 17 May 2011 02:28:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63291 invoked by uid 99); 17 May 2011 02:28:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 02:28:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 02:28:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A6C0BCDC8E
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 02:27:47 +0000 (UTC)
Date: Tue, 17 May 2011 02:27:47 +0000 (UTC)
From: "Siddharth Seth (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1039923360.17714.1305599267679.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1326019706.17705.1305599027427.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7296) The FsPermission(FsPermission)
 constructor does not use the sticky bit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siddharth Seth updated HADOOP-7296:
-----------------------------------

    Fix Version/s: 0.22.0
           Status: Patch Available  (was: Open)

> The FsPermission(FsPermission) constructor does not use the sticky bit
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-7296
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7296
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Siddharth Seth
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP7296.patch
>
>
> The FsPermission(FsPermission) constructor copies u, g, o from the supplied FsPermission object but ignores the sticky bit.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14965-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 02:30:30 2011
Return-Path: <common-issues-return-14965-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5DFDC4528
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 02:30:30 +0000 (UTC)
Received: (qmail 64550 invoked by uid 500); 17 May 2011 02:30:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64521 invoked by uid 500); 17 May 2011 02:30:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64513 invoked by uid 99); 17 May 2011 02:30:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 02:30:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 02:30:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D8E82CDD50
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 02:29:47 +0000 (UTC)
Date: Tue, 17 May 2011 02:29:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <37886702.17723.1305599387884.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1326019706.17705.1305599027427.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7296) The FsPermission(FsPermission)
 constructor does not use the sticky bit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034505#comment-13034505 ] 

Aaron T. Myers commented on HADOOP-7296:
----------------------------------------

Hi Siddarth,

The patch looks pretty good, though it looks to me like your patch introduces a tab character in {{src/java/org/apache/hadoop/fs/permission/FsPermission.java}}. Our convention is to expand these into spaces. Would you mind uploading a new version without the tab?

Also, what version of Hadoop is this patch for? Please fill in the affects version field as appropriate.

> The FsPermission(FsPermission) constructor does not use the sticky bit
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-7296
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7296
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Siddharth Seth
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP7296.patch
>
>
> The FsPermission(FsPermission) constructor copies u, g, o from the supplied FsPermission object but ignores the sticky bit.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14966-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 02:44:29 2011
Return-Path: <common-issues-return-14966-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8012848E5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 02:44:29 +0000 (UTC)
Received: (qmail 71892 invoked by uid 500); 17 May 2011 02:44:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71720 invoked by uid 500); 17 May 2011 02:44:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71684 invoked by uid 99); 17 May 2011 02:44:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 02:44:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 02:44:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 62D37CDFF1
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 02:43:47 +0000 (UTC)
Date: Tue, 17 May 2011 02:43:47 +0000 (UTC)
From: "Siddharth Seth (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <89297733.17742.1305600227401.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1326019706.17705.1305599027427.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7296) The FsPermission(FsPermission)
 constructor does not use the sticky bit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siddharth Seth updated HADOOP-7296:
-----------------------------------

    Attachment: HADOOP7296_2.patch

Thanks for looking at it. Fixed the tab. This exists in 0.21 and subsequent versions. Will mark that.

> The FsPermission(FsPermission) constructor does not use the sticky bit
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-7296
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7296
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Siddharth Seth
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP7296.patch, HADOOP7296_2.patch
>
>
> The FsPermission(FsPermission) constructor copies u, g, o from the supplied FsPermission object but ignores the sticky bit.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14967-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 02:46:29 2011
Return-Path: <common-issues-return-14967-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 11A3E4495
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 02:46:29 +0000 (UTC)
Received: (qmail 72430 invoked by uid 500); 17 May 2011 02:46:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72243 invoked by uid 500); 17 May 2011 02:46:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72234 invoked by uid 99); 17 May 2011 02:46:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 02:46:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 02:46:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 541C3CD05B
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 02:45:47 +0000 (UTC)
Date: Tue, 17 May 2011 02:45:47 +0000 (UTC)
From: "Siddharth Seth (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1233771820.17751.1305600347341.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1326019706.17705.1305599027427.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7296) The FsPermission(FsPermission)
 constructor does not use the sticky bit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siddharth Seth updated HADOOP-7296:
-----------------------------------

    Affects Version/s: 0.23.0
                       0.22.0
                       0.21.0

> The FsPermission(FsPermission) constructor does not use the sticky bit
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-7296
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7296
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0, 0.23.0
>            Reporter: Siddharth Seth
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP7296.patch, HADOOP7296_2.patch
>
>
> The FsPermission(FsPermission) constructor copies u, g, o from the supplied FsPermission object but ignores the sticky bit.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14968-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 04:36:38 2011
Return-Path: <common-issues-return-14968-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7BF614FCF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 04:36:38 +0000 (UTC)
Received: (qmail 36864 invoked by uid 500); 17 May 2011 04:36:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36748 invoked by uid 500); 17 May 2011 04:36:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36729 invoked by uid 99); 17 May 2011 04:36:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 04:36:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 04:36:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8F5B6CD7EE
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 04:35:47 +0000 (UTC)
Date: Tue, 17 May 2011 04:35:47 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1270466137.17978.1305606947583.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6757) NullPointerException for hadoop
 clients launched from streaming tasks
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034547#comment-13034547 ] 

Tom White commented on HADOOP-6757:
-----------------------------------

I think this may be fixed by MAPREDUCE-2372.

> NullPointerException for hadoop clients launched from streaming tasks
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6757
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6757
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Amar Kamat
>            Assignee: Amar Kamat
>         Attachments: BZ-3620565-v1.0.patch, HADOOP-6757-v1.0.patch
>
>
> TaskRunner sets HADOOP_ROOT_LOGGER to info,TLA while launching the child tasks. TLA implicitly assumes that that task-id information will be made available via the 'hadoop.tasklog.taskid' parameter. 'hadoop.tasklog.taskid' is passed to the child task by the TaskRunner via HADOOP_CLIENT_OPTS. When the streaming task launches a hadoop client (say hadoop job -list), the HADOOP_ROOT_LOGGER of the hadoop client is set to 'info,TLA' but hadoop.tasklog.taskid is not set resulting into NPE.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14969-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 04:40:31 2011
Return-Path: <common-issues-return-14969-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 25D88409A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 04:40:31 +0000 (UTC)
Received: (qmail 40717 invoked by uid 500); 17 May 2011 04:40:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40674 invoked by uid 500); 17 May 2011 04:40:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40665 invoked by uid 99); 17 May 2011 04:40:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 04:40:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 04:40:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5CB50CD920
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 04:39:47 +0000 (UTC)
Date: Tue, 17 May 2011 04:39:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2080004464.17987.1305607187376.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1326019706.17705.1305599027427.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7296) The FsPermission(FsPermission)
 constructor does not use the sticky bit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034550#comment-13034550 ] 

Aaron T. Myers commented on HADOOP-7296:
----------------------------------------

+1, the patch looks good to me.

> The FsPermission(FsPermission) constructor does not use the sticky bit
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-7296
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7296
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0, 0.23.0
>            Reporter: Siddharth Seth
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP7296.patch, HADOOP7296_2.patch
>
>
> The FsPermission(FsPermission) constructor copies u, g, o from the supplied FsPermission object but ignores the sticky bit.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14970-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 05:24:30 2011
Return-Path: <common-issues-return-14970-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 106F1461D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 05:24:30 +0000 (UTC)
Received: (qmail 70492 invoked by uid 500); 17 May 2011 05:24:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70385 invoked by uid 500); 17 May 2011 05:24:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70370 invoked by uid 99); 17 May 2011 05:24:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 05:24:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 05:24:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6E63FCD2FA
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 05:23:47 +0000 (UTC)
Date: Tue, 17 May 2011 05:23:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <137321665.18044.1305609827449.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <903601160.6842.1305210828211.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7282) getRemoteIp could return null in
 cases where the call is ongoing but the ip went away.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034562#comment-13034562 ] 

Hadoop QA commented on HADOOP-7282:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479182/HADOOP-7282-1.patch
  against trunk revision 1103971.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/461//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/461//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/461//console

This message is automatically generated.

> getRemoteIp could return null in cases where the call is ongoing but the ip went away.
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7282
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7282
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: John George
>            Assignee: John George
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7282-1.patch, HADOOP-7282.patch, diffs.txt
>
>
> getRemoteIp gets the ip from socket instead of the stored ip in Connection object. Thus calls to this function could return null when a client disconnected, but the rpc call is still ongoing...

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14971-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 05:36:33 2011
Return-Path: <common-issues-return-14971-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 570814C6B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 05:36:33 +0000 (UTC)
Received: (qmail 74446 invoked by uid 500); 17 May 2011 05:36:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74413 invoked by uid 500); 17 May 2011 05:36:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74390 invoked by uid 99); 17 May 2011 05:36:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 05:36:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 05:36:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5D0CECD5D4
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 05:35:47 +0000 (UTC)
Date: Tue, 17 May 2011 05:35:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <856690449.18051.1305610547378.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1894387424.4008.1305137809491.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7278) Automatic Hadoop cluster
 deployment for build validation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034564#comment-13034564 ] 

Hadoop QA commented on HADOOP-7278:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479155/HADOOP-7278.patch
  against trunk revision 1103971.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/462//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/462//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/462//console

This message is automatically generated.

> Automatic Hadoop cluster deployment for build validation
> --------------------------------------------------------
>
>                 Key: HADOOP-7278
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7278
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0, 0.23.0
>         Environment: Apache Jenkins
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>              Labels: newbie
>         Attachments: HADOOP-7278.patch
>
>
> It'd be great to have a way of automatically deploying Hadoop cluster to a set of machine once all components are successfully built (in the form or tar or whatever). Deployed cluster then can be used to run a set of validation jobs to make sure that produced artifacts are viable. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14972-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 05:38:51 2011
Return-Path: <common-issues-return-14972-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6C07F4CFC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 05:38:51 +0000 (UTC)
Received: (qmail 76292 invoked by uid 500); 17 May 2011 05:38:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76062 invoked by uid 500); 17 May 2011 05:38:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75980 invoked by uid 99); 17 May 2011 05:38:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 05:38:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 05:38:49 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8016ECD972
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 05:37:49 +0000 (UTC)
Date: Tue, 17 May 2011 05:37:49 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1892097529.18103.1305610669521.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034574#comment-13034574 ] 

Hudson commented on HADOOP-7291:
--------------------------------

Integrated in Hadoop-Hdfs-trunk-Commit #658 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/658/])
    

> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291-3.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14973-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 05:39:13 2011
Return-Path: <common-issues-return-14973-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6E2534DB1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 05:39:13 +0000 (UTC)
Received: (qmail 78598 invoked by uid 500); 17 May 2011 05:39:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78214 invoked by uid 500); 17 May 2011 05:39:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78056 invoked by uid 99); 17 May 2011 05:39:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 05:39:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 05:39:09 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 65F78CD9AB
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 05:37:51 +0000 (UTC)
Date: Tue, 17 May 2011 05:37:51 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <335920735.18153.1305610671414.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6846) Scripts for building Hadoop 0.22.0
 release
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034586#comment-13034586 ] 

Hudson commented on HADOOP-6846:
--------------------------------

Integrated in Hadoop-Hdfs-trunk-Commit #658 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/658/])
    

> Scripts for building Hadoop 0.22.0 release
> ------------------------------------------
>
>                 Key: HADOOP-6846
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6846
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6846.patch, release-scripts.tar.gz
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14974-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 05:48:30 2011
Return-Path: <common-issues-return-14974-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 57F87407B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 05:48:30 +0000 (UTC)
Received: (qmail 86979 invoked by uid 500); 17 May 2011 05:48:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86947 invoked by uid 500); 17 May 2011 05:48:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86939 invoked by uid 99); 17 May 2011 05:48:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 05:48:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 05:48:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 893CDCDC58
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 05:47:47 +0000 (UTC)
Date: Tue, 17 May 2011 05:47:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <755942112.18168.1305611267558.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <866597877.11958.1305329567592.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7292) Metrics 2 TestSinkQueue is racy
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034588#comment-13034588 ] 

Hudson commented on HADOOP-7292:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #604 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/604/])
    HADOOP-7292. Fix racy test case TestSinkQueue. Contributed by Luke Lu.


> Metrics 2 TestSinkQueue is racy
> -------------------------------
>
>                 Key: HADOOP-7292
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7292
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.23.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7292-test-race-v1.patch, hadoop-7292-test-race-v2.patch
>
>
> The TestSinkQueue is racy (Thread.yield is not enough to guarantee other intended thread getting run), though it's the first time (from HADOOP-7289) I saw it manifested here.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14975-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 06:08:33 2011
Return-Path: <common-issues-return-14975-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 46F614C08
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 06:08:33 +0000 (UTC)
Received: (qmail 676 invoked by uid 500); 17 May 2011 06:08:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 337 invoked by uid 500); 17 May 2011 06:08:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 327 invoked by uid 99); 17 May 2011 06:08:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 06:08:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 06:08:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C1A88C730A
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 06:07:47 +0000 (UTC)
Date: Tue, 17 May 2011 06:07:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <386472367.18252.1305612467790.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1326019706.17705.1305599027427.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7296) The FsPermission(FsPermission)
 constructor does not use the sticky bit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034595#comment-13034595 ] 

Hadoop QA commented on HADOOP-7296:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479423/HADOOP7296_2.patch
  against trunk revision 1103971.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/463//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/463//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/463//console

This message is automatically generated.

> The FsPermission(FsPermission) constructor does not use the sticky bit
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-7296
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7296
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0, 0.23.0
>            Reporter: Siddharth Seth
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP7296.patch, HADOOP7296_2.patch
>
>
> The FsPermission(FsPermission) constructor copies u, g, o from the supplied FsPermission object but ignores the sticky bit.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14976-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 06:24:35 2011
Return-Path: <common-issues-return-14976-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B241843D1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 06:24:35 +0000 (UTC)
Received: (qmail 16552 invoked by uid 500); 17 May 2011 06:24:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16447 invoked by uid 500); 17 May 2011 06:24:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16435 invoked by uid 99); 17 May 2011 06:24:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 06:24:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 06:24:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8B6F6C785B
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 06:23:49 +0000 (UTC)
Date: Tue, 17 May 2011 06:23:49 +0000 (UTC)
From: "issei yoshida (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1338047981.18317.1305613429567.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034599#comment-13034599 ] 

issei yoshida commented on HADOOP-7206:
---------------------------------------

As a result of talking with Alejandro and Tom, 
we agree that we'll give up the integration into Hadoop core and cancel Hadoop-7206,
because to avoid confusion with users/developers, the code should be in one place either a separate project or integrated into hadoop core.
The hadoop-snappy as an independent project have the flexibility of doing releases and bug fixes independent of Hadoop releases,
so I think I'll continue to develop hadoop-snappy only.

Thanks.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>         Attachments: HADOOP-7206.patch
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14978-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 08:52:28 2011
Return-Path: <common-issues-return-14978-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EC9CA486C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 08:52:28 +0000 (UTC)
Received: (qmail 82390 invoked by uid 500); 17 May 2011 08:52:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82325 invoked by uid 500); 17 May 2011 08:52:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82292 invoked by uid 99); 17 May 2011 08:52:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 08:52:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 08:52:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8CC49CD806
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 08:51:47 +0000 (UTC)
Date: Tue, 17 May 2011 08:51:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <434557255.18499.1305622307573.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7284:
---------------------------------

    Attachment: trash2.patch

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14977-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 08:52:29 2011
Return-Path: <common-issues-return-14977-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EC3A1486A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 08:52:28 +0000 (UTC)
Received: (qmail 82350 invoked by uid 500); 17 May 2011 08:52:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82318 invoked by uid 500); 17 May 2011 08:52:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82283 invoked by uid 99); 17 May 2011 08:52:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 08:52:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 08:52:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 70183CD7FF
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 08:51:47 +0000 (UTC)
Date: Tue, 17 May 2011 08:51:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <356752016.18495.1305622307455.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034632#comment-13034632 ] 

Sanjay Radia commented on HADOOP-7284:
--------------------------------------

trash.isEnabled() was redundant - #moveToTrash() checks that first. #moveToApropriateTrash() calls #moveToTrash().
The unit test ended being small code but harder than it looked (partly because of home dir being /Users and /user in different
systems - we want to the test to work on linux and mac.)
Did some further cleanup of tests and resolved a bug in chRootedFs().
The FileSystem cache causes some weird problems -- one thinks one gets independent FileSystems but not true because of wd.

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14979-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 08:52:30 2011
Return-Path: <common-issues-return-14979-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 60F2C4886
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 08:52:30 +0000 (UTC)
Received: (qmail 82863 invoked by uid 500); 17 May 2011 08:52:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82814 invoked by uid 500); 17 May 2011 08:52:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82806 invoked by uid 99); 17 May 2011 08:52:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 08:52:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 08:52:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 99C07CD808
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 08:51:47 +0000 (UTC)
Date: Tue, 17 May 2011 08:51:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <309808595.18501.1305622307626.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7284:
---------------------------------

    Status: Patch Available  (was: Open)

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14980-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 09:14:30 2011
Return-Path: <common-issues-return-14980-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1B98241AC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 09:14:30 +0000 (UTC)
Received: (qmail 5697 invoked by uid 500); 17 May 2011 09:14:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5653 invoked by uid 500); 17 May 2011 09:14:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5645 invoked by uid 99); 17 May 2011 09:14:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 09:14:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 09:14:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 75C92CD0EB
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 09:13:47 +0000 (UTC)
Date: Tue, 17 May 2011 09:13:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <99554649.18535.1305623627478.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034636#comment-13034636 ] 

Hadoop QA commented on HADOOP-7284:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479439/trash2.patch
  against trunk revision 1103971.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 15 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/464//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/464//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/464//console

This message is automatically generated.

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14981-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 09:44:28 2011
Return-Path: <common-issues-return-14981-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 149344474
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 09:44:28 +0000 (UTC)
Received: (qmail 50313 invoked by uid 500); 17 May 2011 09:44:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50272 invoked by uid 500); 17 May 2011 09:44:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50264 invoked by uid 99); 17 May 2011 09:44:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 09:44:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 09:44:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7E012CDE20
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 09:43:47 +0000 (UTC)
Date: Tue, 17 May 2011 09:43:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1913147996.18584.1305625427512.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <683476535.2038.1300136250528.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7189) Add ability to enable 'debug'
 property in JAAS configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034648#comment-13034648 ] 

Hudson commented on HADOOP-7189:
--------------------------------

Integrated in Hadoop-Common-22-branch #49 (See [https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/49/])
    

> Add ability to enable 'debug' property in JAAS configuration
> ------------------------------------------------------------
>
>                 Key: HADOOP-7189
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7189
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Ted Yu
>            Priority: Minor
>              Labels: newbie
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7189.patch, HADOOP-7189.txt, enable-UGI-debug-example.txt
>
>
> Occasionally users have run into weird "Unable to login" messages. Unfortunately, JAAS obscures the underlying exception message in many cases because it thinks leaking the exception might be insecure in itself. Enabling the "debug" option in the JAAS configuration gets it to dump the underlying issue and makes troubleshooting this kind of issue easier.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14982-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 09:44:29 2011
Return-Path: <common-issues-return-14982-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6F8EC4487
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 09:44:29 +0000 (UTC)
Received: (qmail 50575 invoked by uid 500); 17 May 2011 09:44:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50537 invoked by uid 500); 17 May 2011 09:44:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50528 invoked by uid 99); 17 May 2011 09:44:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 09:44:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 09:44:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DEA42CDE2E
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 09:43:47 +0000 (UTC)
Date: Tue, 17 May 2011 09:43:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1236405800.18596.1305625427908.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <510617684.3042.1300172489780.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7192) fs -stat docs aren't updated to
 reflect the format features
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034650#comment-13034650 ] 

Hudson commented on HADOOP-7192:
--------------------------------

Integrated in Hadoop-Common-22-branch #49 (See [https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/49/])
    

> fs -stat docs aren't updated to reflect the format features
> -----------------------------------------------------------
>
>                 Key: HADOOP-7192
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7192
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>    Affects Versions: 0.21.0
>         Environment: Linux / 0.20
>            Reporter: Harsh J Chouraria
>            Assignee: Harsh J Chouraria
>            Priority: Trivial
>              Labels: documentaion
>             Fix For: 0.22.0
>
>         Attachments: hadoop.common.fsstatdoc.r1.diff
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> The html docs of the 'fs -stat' command (that is found listed in the File System Shell Guide), does not seem to have the formatting abilities of -stat explained (along with the options).
> Like 'fs -help', the docs must also reflect the latest available features.
> I shall attach a doc-fix patch shortly.
> If anyone has other discrepancies to point out in the web version of the guide, please do so :)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14984-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 09:44:33 2011
Return-Path: <common-issues-return-14984-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8AD9544B0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 09:44:33 +0000 (UTC)
Received: (qmail 51641 invoked by uid 500); 17 May 2011 09:44:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51595 invoked by uid 500); 17 May 2011 09:44:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51557 invoked by uid 99); 17 May 2011 09:44:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 09:44:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 09:44:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3D5E1CDE38
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 09:43:48 +0000 (UTC)
Date: Tue, 17 May 2011 09:43:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <205173048.18605.1305625428248.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6846) Scripts for building Hadoop 0.22.0
 release
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034652#comment-13034652 ] 

Hudson commented on HADOOP-6846:
--------------------------------

Integrated in Hadoop-Common-22-branch #49 (See [https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/49/])
    

> Scripts for building Hadoop 0.22.0 release
> ------------------------------------------
>
>                 Key: HADOOP-6846
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6846
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6846.patch, release-scripts.tar.gz
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14983-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 09:44:33 2011
Return-Path: <common-issues-return-14983-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D4E1E44BB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 09:44:33 +0000 (UTC)
Received: (qmail 51539 invoked by uid 500); 17 May 2011 09:44:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51499 invoked by uid 500); 17 May 2011 09:44:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51464 invoked by uid 99); 17 May 2011 09:44:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 09:44:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 09:44:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 29670CDE36
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 09:43:48 +0000 (UTC)
Date: Tue, 17 May 2011 09:43:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <743812508.18603.1305625428166.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <957076305.12265.1297477317500.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7137) Remove hod contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034651#comment-13034651 ] 

Hudson commented on HADOOP-7137:
--------------------------------

Integrated in Hadoop-Common-22-branch #49 (See [https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/49/])
    

> Remove hod contrib
> ------------------
>
>                 Key: HADOOP-7137
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7137
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7137.patch, HADOOP-7137.patch
>
>
> As per vote on general@ (http://mail-archives.apache.org/mod_mbox/hadoop-general/201102.mbox/%3CAC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com%3E) I will 
> svn remove common/trunk/src/contrib/hod
> using this Jira.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14985-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 09:44:34 2011
Return-Path: <common-issues-return-14985-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 81B8C44DB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 09:44:34 +0000 (UTC)
Received: (qmail 52295 invoked by uid 500); 17 May 2011 09:44:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52262 invoked by uid 500); 17 May 2011 09:44:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52221 invoked by uid 99); 17 May 2011 09:44:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 09:44:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 09:44:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C9487CDE2B
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 09:43:47 +0000 (UTC)
Date: Tue, 17 May 2011 09:43:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1108782929.18593.1305625427821.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034649#comment-13034649 ] 

Hudson commented on HADOOP-7291:
--------------------------------

Integrated in Hadoop-Common-22-branch #49 (See [https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/49/])
    HADOOP-7291. svn merge -c 1103931 from trunk


> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291-3.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14986-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 11:14:28 2011
Return-Path: <common-issues-return-14986-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D66AC4A1E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 11:14:28 +0000 (UTC)
Received: (qmail 99763 invoked by uid 500); 17 May 2011 11:14:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99724 invoked by uid 500); 17 May 2011 11:14:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99708 invoked by uid 99); 17 May 2011 11:14:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 11:14:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 11:14:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 58136CD8BC
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 11:13:48 +0000 (UTC)
Date: Tue, 17 May 2011 11:13:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1562014881.18870.1305630828357.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1619049648.8641.1305244247668.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7286) Refactor FsShell's du/dus/df
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034717#comment-13034717 ] 

Hudson commented on HADOOP-7286:
--------------------------------

Integrated in Hadoop-Common-trunk #691 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/691/])
    HADOOP-7286. Refactor the du/dus/df commands to conform to new FsCommand class. Contributed by Daryn Sharp.


> Refactor FsShell's du/dus/df
> ----------------------------
>
>                 Key: HADOOP-7286
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7286
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7286-2.patch, HADOOP-7286-3.patch, HADOOP-7286-4.patch, HADOOP-7286.patch
>
>
> Need to refactor to conform to FsCommand subclass.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14987-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 11:14:29 2011
Return-Path: <common-issues-return-14987-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1298D4A28
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 11:14:29 +0000 (UTC)
Received: (qmail 99917 invoked by uid 500); 17 May 2011 11:14:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99726 invoked by uid 500); 17 May 2011 11:14:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99709 invoked by uid 99); 17 May 2011 11:14:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 11:14:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 11:14:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1D100CD8B2
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 11:13:48 +0000 (UTC)
Date: Tue, 17 May 2011 11:13:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1258061734.18862.1305630828115.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034715#comment-13034715 ] 

Hudson commented on HADOOP-7291:
--------------------------------

Integrated in Hadoop-Common-trunk #691 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/691/])
    HADOOP-7291. Remove spurious call to runTestContrib. Contributed by Eli Collins
HADOOP-7291. Update Hudson job not to run test-contrib. Contributed by Nigel Daley
Reverting the change r1102914 for HADOOP-7291 to fix build issues.


> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291-3.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14988-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 11:14:29 2011
Return-Path: <common-issues-return-14988-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 505334A30
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 11:14:29 +0000 (UTC)
Received: (qmail 217 invoked by uid 500); 17 May 2011 11:14:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 160 invoked by uid 500); 17 May 2011 11:14:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99756 invoked by uid 99); 17 May 2011 11:14:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 11:14:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 11:14:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 39136CD8B6
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 11:13:48 +0000 (UTC)
Date: Tue, 17 May 2011 11:13:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1528579043.18866.1305630828230.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <866597877.11958.1305329567592.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7292) Metrics 2 TestSinkQueue is racy
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034716#comment-13034716 ] 

Hudson commented on HADOOP-7292:
--------------------------------

Integrated in Hadoop-Common-trunk #691 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/691/])
    HADOOP-7292. Fix racy test case TestSinkQueue. Contributed by Luke Lu.


> Metrics 2 TestSinkQueue is racy
> -------------------------------
>
>                 Key: HADOOP-7292
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7292
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.23.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7292-test-race-v1.patch, hadoop-7292-test-race-v2.patch
>
>
> The TestSinkQueue is racy (Thread.yield is not enough to guarantee other intended thread getting run), though it's the first time (from HADOOP-7289) I saw it manifested here.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14989-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 11:20:30 2011
Return-Path: <common-issues-return-14989-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2B1D04130
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 11:20:30 +0000 (UTC)
Received: (qmail 29195 invoked by uid 500); 17 May 2011 11:20:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29159 invoked by uid 500); 17 May 2011 11:20:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29146 invoked by uid 99); 17 May 2011 11:20:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 11:20:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 11:20:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7C8C4CDAEC
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 11:19:47 +0000 (UTC)
Date: Tue, 17 May 2011 11:19:47 +0000 (UTC)
From: "Nicholas Telford (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1226031152.18878.1305631187506.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7269) S3 Native should allow customizable
 file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicholas Telford updated HADOOP-7269:
-------------------------------------

    Attachment: HADOOP-7269-S3-metadata-003.diff

Removed abstraction of metadata at the top-level FileSystem. Metadata is now only supported by the NativeS3FileSystem, and you'll need to cast explicitly in order to use it.

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: HADOOP-7269-S3-metadata-001.diff, HADOOP-7269-S3-metadata-002.diff, HADOOP-7269-S3-metadata-003.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14990-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 11:20:30 2011
Return-Path: <common-issues-return-14990-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 54054413E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 11:20:30 +0000 (UTC)
Received: (qmail 29366 invoked by uid 500); 17 May 2011 11:20:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29299 invoked by uid 500); 17 May 2011 11:20:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29207 invoked by uid 99); 17 May 2011 11:20:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 11:20:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 11:20:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 83011CDAED
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 11:19:47 +0000 (UTC)
Date: Tue, 17 May 2011 11:19:47 +0000 (UTC)
From: "Nicholas Telford (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <799978699.18879.1305631187533.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7269) S3 Native should allow customizable
 file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicholas Telford updated HADOOP-7269:
-------------------------------------

    Release Note: Added support for arbitrary file metadata when creating a new file on the NativeS3FileSystem.  (was: Added support for arbitrary file metadata when creating a new file on a FileSystem that supports metadata.)
          Status: Patch Available  (was: Open)

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: HADOOP-7269-S3-metadata-001.diff, HADOOP-7269-S3-metadata-002.diff, HADOOP-7269-S3-metadata-003.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14991-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 11:33:31 2011
Return-Path: <common-issues-return-14991-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 70D334655
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 11:33:31 +0000 (UTC)
Received: (qmail 45876 invoked by uid 500); 17 May 2011 11:33:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45841 invoked by uid 500); 17 May 2011 11:33:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45833 invoked by uid 99); 17 May 2011 11:33:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 11:33:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 11:33:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D0C17CD041
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 11:32:48 +0000 (UTC)
Date: Tue, 17 May 2011 11:32:48 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2101837345.18901.1305631968851.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7269) S3 Native should allow
 customizable file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034725#comment-13034725 ] 

Hadoop QA commented on HADOOP-7269:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479451/HADOOP-7269-S3-metadata-003.diff
  against trunk revision 1103971.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 9 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The applied patch generated 1070 javac compiler warnings (more than the trunk's current 1068 warnings).

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/465//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/465//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/465//console

This message is automatically generated.

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: HADOOP-7269-S3-metadata-001.diff, HADOOP-7269-S3-metadata-002.diff, HADOOP-7269-S3-metadata-003.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14992-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 12:14:29 2011
Return-Path: <common-issues-return-14992-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 837944CAF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 12:14:29 +0000 (UTC)
Received: (qmail 7212 invoked by uid 500); 17 May 2011 12:14:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7153 invoked by uid 500); 17 May 2011 12:14:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6975 invoked by uid 99); 17 May 2011 12:14:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 12:14:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 12:14:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 64CADCDF3D
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 12:13:47 +0000 (UTC)
Date: Tue, 17 May 2011 12:13:47 +0000 (UTC)
From: "ramkrishna.s.vasudevan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1562691557.18939.1305634427409.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <282223217.18947.1304436003652.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7256) Resource leak during failure
 scenario of closing of resources.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ramkrishna.s.vasudevan updated HADOOP-7256:
-------------------------------------------

    Description: 
Problem Statement:
===============
There are chances of resource leak and stream not getting closed 
Take the case when after copying data we try to close the Input and output stream followed by closing of the socket.
Suppose an exception occurs while closing the input stream(due to runtime exception) then the subsequent operations of closing the output stream and socket may not happen and there is a chance of resource leak.

Scenario 
=======
During long run of map reduce jobs, the copyFromLocalFile() api is getting called.
Here we found some exceptions happening. As a result of this we found the lsof value raising leading to resource leak.

Solution:
=======
While doing a close operation of any resource catch the RuntimeException also rather than catching the IOException alone.

Additionally there are places where we try to close a resource in the catch block.
If this close fails, we just throw and come out of the current flow.
In order to avoid this, we can carry out the close operation in the finally block.

Probable reasons for getting RunTimeExceptions:
=====================================
We may get runtime exception from customised hadoop streams like FSDataOutputStream.close() . So better to handle RunTimeExceptions also.


 



  was:
Problem Statement:
===============
There are chances of resource leak and stream not getting closed 
Take the case when after copying data we try to close the Input and output stream followed by closing of the socket.
Suppose an exception occurs while closing the input stream(due to runtime exception) then the subsequent operations of closing the output stream and socket may not happen and there is a chance of resource leak.

Scenario 
=======
During long run of map reduce jobs, the copyFromLocalFile() api is getting called.
Here we found some exceptions happening. As a result of this we found the lsof value raising leading to resource leak.

Solution:
=======
While doing a close operation of any resource catch the RuntimeException also rather than catching the IOException alone.

Additionally there are places where we try to close a resource in the catch block.
If this close fails, we just throw and come out of the current flow.
In order to avoid this, we can carry out the close operation in the finally block.

Probable reasons for getting RunTimeExceptions:
=====================================
We have many wrapped stream for writing and reading.
These wrappers are prone to errors.


 




> Resource leak during failure scenario of closing of resources. 
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7256
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7256
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2, 0.21.0
>            Reporter: ramkrishna.s.vasudevan
>            Priority: Minor
>   Original Estimate: 8h
>  Remaining Estimate: 8h
>
> Problem Statement:
> ===============
> There are chances of resource leak and stream not getting closed 
> Take the case when after copying data we try to close the Input and output stream followed by closing of the socket.
> Suppose an exception occurs while closing the input stream(due to runtime exception) then the subsequent operations of closing the output stream and socket may not happen and there is a chance of resource leak.
> Scenario 
> =======
> During long run of map reduce jobs, the copyFromLocalFile() api is getting called.
> Here we found some exceptions happening. As a result of this we found the lsof value raising leading to resource leak.
> Solution:
> =======
> While doing a close operation of any resource catch the RuntimeException also rather than catching the IOException alone.
> Additionally there are places where we try to close a resource in the catch block.
> If this close fails, we just throw and come out of the current flow.
> In order to avoid this, we can carry out the close operation in the finally block.
> Probable reasons for getting RunTimeExceptions:
> =====================================
> We may get runtime exception from customised hadoop streams like FSDataOutputStream.close() . So better to handle RunTimeExceptions also.
>  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14993-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 13:15:28 2011
Return-Path: <common-issues-return-14993-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 484564597
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 13:15:28 +0000 (UTC)
Received: (qmail 20283 invoked by uid 500); 17 May 2011 13:15:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20195 invoked by uid 500); 17 May 2011 13:15:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20186 invoked by uid 99); 17 May 2011 13:15:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 13:15:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 13:15:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B159ACD7C5
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 13:14:47 +0000 (UTC)
Date: Tue, 17 May 2011 13:14:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <317037825.19028.1305638087723.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034747#comment-13034747 ] 

Hudson commented on HADOOP-7291:
--------------------------------

Integrated in Hadoop-Mapreduce-22-branch #52 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/52/])
    HADOOP-7291. svn merge -c 1103931 from trunk


> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291-3.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14994-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 15:20:28 2011
Return-Path: <common-issues-return-14994-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0F90146FD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 15:20:28 +0000 (UTC)
Received: (qmail 99599 invoked by uid 500); 17 May 2011 15:20:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99563 invoked by uid 500); 17 May 2011 15:20:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99554 invoked by uid 99); 17 May 2011 15:20:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 15:20:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 15:20:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7BEC4CD198
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 15:19:47 +0000 (UTC)
Date: Tue, 17 May 2011 15:19:47 +0000 (UTC)
From: "arnaud p (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1250812071.19292.1305645587504.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7297) Error in the documentation regarding
 Checkpoint/Backup Node
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Error in the documentation regarding Checkpoint/Backup Node
-----------------------------------------------------------

                 Key: HADOOP-7297
                 URL: https://issues.apache.org/jira/browse/HADOOP-7297
             Project: Hadoop Common
          Issue Type: Bug
          Components: documentation
    Affects Versions: 0.20.203.0
            Reporter: arnaud p
            Priority: Trivial


On http://hadoop.apache.org/common/docs/r0.20.203.0/hdfs_user_guide.html#Checkpoint+Node: the command bin/hdfs namenode -checkpoint required to launch the backup/checkpoint node does not exist.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14995-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 15:41:31 2011
Return-Path: <common-issues-return-14995-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3D4904D05
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 15:41:31 +0000 (UTC)
Received: (qmail 73171 invoked by uid 500); 17 May 2011 15:41:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73053 invoked by uid 500); 17 May 2011 15:41:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73021 invoked by uid 99); 17 May 2011 15:41:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 15:41:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 15:41:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AB6BACDDB7
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 15:40:47 +0000 (UTC)
Date: Tue, 17 May 2011 15:40:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1307298103.19384.1305646847699.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034822#comment-13034822 ] 

Hudson commented on HADOOP-7291:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk #682 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/682/])
    HADOOP-7291. Remove spurious call to runTestContrib. Contributed by Eli Collins
MAPREDUCE-2499. MR part of HADOOP-7291. Contributed by Eli Collins
HADOOP-7291. Update Hudson job not to run test-contrib. Contributed by Nigel Daley
Reverting the change r1102914 for HADOOP-7291 to fix build issues.


> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291-3.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14996-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 15:47:30 2011
Return-Path: <common-issues-return-14996-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 404914B31
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 15:47:30 +0000 (UTC)
Received: (qmail 88343 invoked by uid 500); 17 May 2011 15:47:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88286 invoked by uid 500); 17 May 2011 15:47:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88278 invoked by uid 99); 17 May 2011 15:47:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 15:47:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 15:47:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AA14DCE129
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 15:46:49 +0000 (UTC)
Date: Tue, 17 May 2011 15:46:49 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1619732914.19485.1305647209693.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1240879157.4951.1300837565772.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7206) Integrate Snappy compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034829#comment-13034829 ] 

Tom White commented on HADOOP-7206:
-----------------------------------

I actually think that, long-term, Snappy compression belongs in Hadoop, along with the other compression codecs. The hadoop-snappy project is a useful stopgap until we get regular releases going again, and it allows other projects like HBase to use Snappy in the meantime.

> Integrate Snappy compression
> ----------------------------
>
>                 Key: HADOOP-7206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7206
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Eli Collins
>         Attachments: HADOOP-7206.patch
>
>
> Google release Zippy as an open source (APLv2) project called Snappy (http://code.google.com/p/snappy). This tracks integrating it into Hadoop.
> {quote}
> Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14997-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 16:16:28 2011
Return-Path: <common-issues-return-14997-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 14B234CF7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 16:16:28 +0000 (UTC)
Received: (qmail 49309 invoked by uid 500); 17 May 2011 16:16:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49268 invoked by uid 500); 17 May 2011 16:16:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49260 invoked by uid 99); 17 May 2011 16:16:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 16:16:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 16:16:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 933FBCEB30
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 16:15:47 +0000 (UTC)
Date: Tue, 17 May 2011 16:15:47 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1434765246.19531.1305648947599.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1326019706.17705.1305599027427.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7296) The FsPermission(FsPermission)
 constructor does not use the sticky bit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-7296:
------------------------------

      Resolution: Fixed
        Assignee: Siddharth Seth
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I've just committed this. Thanks, Siddharth!

> The FsPermission(FsPermission) constructor does not use the sticky bit
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-7296
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7296
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0, 0.23.0
>            Reporter: Siddharth Seth
>            Assignee: Siddharth Seth
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP7296.patch, HADOOP7296_2.patch
>
>
> The FsPermission(FsPermission) constructor copies u, g, o from the supplied FsPermission object but ignores the sticky bit.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14998-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 16:34:30 2011
Return-Path: <common-issues-return-14998-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5EC2747FF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 16:34:30 +0000 (UTC)
Received: (qmail 79823 invoked by uid 500); 17 May 2011 16:34:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79705 invoked by uid 500); 17 May 2011 16:34:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79696 invoked by uid 99); 17 May 2011 16:34:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 16:34:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 16:34:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C1D27CD351
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 16:33:47 +0000 (UTC)
Date: Tue, 17 May 2011 16:33:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <824661406.19623.1305650027790.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1326019706.17705.1305599027427.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7296) The FsPermission(FsPermission)
 constructor does not use the sticky bit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034862#comment-13034862 ] 

Hudson commented on HADOOP-7296:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #605 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/605/])
    HADOOP-7296. The FsPermission(FsPermission) constructor does not use the sticky bit. Contributed by Siddharth Seth


> The FsPermission(FsPermission) constructor does not use the sticky bit
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-7296
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7296
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0, 0.23.0
>            Reporter: Siddharth Seth
>            Assignee: Siddharth Seth
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP7296.patch, HADOOP7296_2.patch
>
>
> The FsPermission(FsPermission) constructor copies u, g, o from the supplied FsPermission object but ignores the sticky bit.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-14999-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 17:39:27 2011
Return-Path: <common-issues-return-14999-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D869D487D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 17:39:27 +0000 (UTC)
Received: (qmail 15553 invoked by uid 500); 17 May 2011 17:39:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15519 invoked by uid 500); 17 May 2011 17:39:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15511 invoked by uid 99); 17 May 2011 17:39:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 17:39:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 17:39:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 58619CEE16
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 17:38:47 +0000 (UTC)
Date: Tue, 17 May 2011 17:38:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <199428749.19856.1305653927358.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <903601160.6842.1305210828211.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7282) getRemoteIp could return null in
 cases where the call is ongoing but the ip went away.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7282:
-------------------------------------------

     Component/s: ipc
    Hadoop Flags: [Reviewed]

+1 patch looks good.

> getRemoteIp could return null in cases where the call is ongoing but the ip went away.
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7282
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7282
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>            Reporter: John George
>            Assignee: John George
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7282-1.patch, HADOOP-7282.patch, diffs.txt
>
>
> getRemoteIp gets the ip from socket instead of the stored ip in Connection object. Thus calls to this function could return null when a client disconnected, but the rpc call is still ongoing...

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15000-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 17:45:30 2011
Return-Path: <common-issues-return-15000-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 344BC4DCC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 17:45:30 +0000 (UTC)
Received: (qmail 25115 invoked by uid 500); 17 May 2011 17:45:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25083 invoked by uid 500); 17 May 2011 17:45:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25075 invoked by uid 99); 17 May 2011 17:45:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 17:45:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 17:45:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 586BACE017
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 17:44:47 +0000 (UTC)
Date: Tue, 17 May 2011 17:44:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <908680369.19867.1305654287358.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <903601160.6842.1305210828211.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7282) getRemoteIp could return null in
 cases where the call is ongoing but the ip went away.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7282:
-------------------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I have committed this.  Thanks, John!

> getRemoteIp could return null in cases where the call is ongoing but the ip went away.
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7282
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7282
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>            Reporter: John George
>            Assignee: John George
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7282-1.patch, HADOOP-7282.patch, diffs.txt
>
>
> getRemoteIp gets the ip from socket instead of the stored ip in Connection object. Thus calls to this function could return null when a client disconnected, but the rpc call is still ongoing...

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15001-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 18:04:30 2011
Return-Path: <common-issues-return-15001-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 40CF14AD8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 18:04:30 +0000 (UTC)
Received: (qmail 61328 invoked by uid 500); 17 May 2011 18:04:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61297 invoked by uid 500); 17 May 2011 18:04:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61283 invoked by uid 99); 17 May 2011 18:04:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 18:04:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 18:04:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7EDE7CE71E
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 18:03:47 +0000 (UTC)
Date: Tue, 17 May 2011 18:03:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <588371746.19922.1305655427516.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <903601160.6842.1305210828211.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7282) getRemoteIp could return null in
 cases where the call is ongoing but the ip went away.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13034923#comment-13034923 ] 

Hudson commented on HADOOP-7282:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #606 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/606/])
    HADOOP-7282. ipc.Server.getRemoteIp() may return null.  Contributed by John George


> getRemoteIp could return null in cases where the call is ongoing but the ip went away.
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7282
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7282
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>            Reporter: John George
>            Assignee: John George
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7282-1.patch, HADOOP-7282.patch, diffs.txt
>
>
> getRemoteIp gets the ip from socket instead of the stored ip in Connection object. Thus calls to this function could return null when a client disconnected, but the rpc call is still ongoing...

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15002-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 19:58:29 2011
Return-Path: <common-issues-return-15002-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EF35F49B6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 19:58:29 +0000 (UTC)
Received: (qmail 10401 invoked by uid 500); 17 May 2011 19:58:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10361 invoked by uid 500); 17 May 2011 19:58:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10353 invoked by uid 99); 17 May 2011 19:58:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 19:58:29 +0000
X-ASF-Spam-Status: No, hits=-1998.0 required=5.0
	tests=ALL_TRUSTED,FB_GET_MEDS,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 19:58:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 581DFCE4D1
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 19:57:47 +0000 (UTC)
Date: Tue, 17 May 2011 19:57:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1102218377.20250.1305662267357.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035009#comment-13035009 ] 

Todd Lipcon commented on HADOOP-7284:
-------------------------------------

Hey Sanjay. I'm trying to follow this patch. Is the change to getHomeDirectory in ViewFileSystem just for the sake of the tests?

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15003-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 21:42:30 2011
Return-Path: <common-issues-return-15003-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 595114133
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 21:42:30 +0000 (UTC)
Received: (qmail 16242 invoked by uid 500); 17 May 2011 21:42:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16197 invoked by uid 500); 17 May 2011 21:42:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16189 invoked by uid 99); 17 May 2011 21:42:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 21:42:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 21:42:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 70A58CE0AF
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 21:41:47 +0000 (UTC)
Date: Tue, 17 May 2011 21:41:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1381606410.20720.1305668507458.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035080#comment-13035080 ] 

Sanjay Radia commented on HADOOP-7284:
--------------------------------------

Good question. Yes and no.
If you recall, when I did FileContext and AbstractFileSystem, the home directory came from the server side. Our current default of /user/<uname> is used for FileSystem since it has no notion of a server defaults. 
Enter Viewfs -- there is no notion of a server side. Also a viewfs could have mount points to different file systems; so I can't pick the ss defaults for home directory from any one of them. 
This hit home when i tried to run the tests because the mac and linux and hdfs have different home directory pathname.
SO the test exposed a real problem in the system design by the fact that our tests run on different platforms and hence different file systems. 

I haven't figured out what is the cleanest way to fix this (but I give a suggestion below that I will file as a jira).
The intermediate solution is that the person configuring the view file system will create a mount point for  /user or /Users etc  depending on what he wants as the default home dir. So I simply coded a rule in viewfilesystem impl,  for now,  to see if it has mount point of /user or /Users etc, till we figure out a better solution.

Here is proposal on which I am going to shortly file a jira: let the viewfs config indicate the home directory. In a sense 
it is like a SS default. Should this simply be a key in the config or something else (like pick it up from a hdfs?)

I could have made the tests work by simply not using a mount point and test path starting with  "/user" or "/Users" but then it does not test our most common usecase. BTW I plan to add a test to HDFS also.
"

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15004-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 17 21:44:30 2011
Return-Path: <common-issues-return-15004-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 391554175
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 17 May 2011 21:44:30 +0000 (UTC)
Received: (qmail 18363 invoked by uid 500); 17 May 2011 21:44:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18326 invoked by uid 500); 17 May 2011 21:44:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18318 invoked by uid 99); 17 May 2011 21:44:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 21:44:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 May 2011 21:44:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6B68CCE1DF
	for <common-issues@hadoop.apache.org>; Tue, 17 May 2011 21:43:47 +0000 (UTC)
Date: Tue, 17 May 2011 21:43:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1659602871.20725.1305668627436.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1326019706.17705.1305599027427.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7296) The FsPermission(FsPermission)
 constructor does not use the sticky bit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035081#comment-13035081 ] 

Hudson commented on HADOOP-7296:
--------------------------------

Integrated in Hadoop-Common-22-branch #50 (See [https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/50/])
    Merge -r 1104373:1104374 from trunk to branch-0.22. Fixes: HADOOP-7296

tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1104377
Files : 
* /hadoop/common/branches/branch-0.22/src/test/core/org/apache/hadoop/fs/permission/TestFsPermission.java
* /hadoop/common/branches/branch-0.22/CHANGES.txt
* /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/fs/permission/FsPermission.java


> The FsPermission(FsPermission) constructor does not use the sticky bit
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-7296
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7296
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0, 0.23.0
>            Reporter: Siddharth Seth
>            Assignee: Siddharth Seth
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP7296.patch, HADOOP7296_2.patch
>
>
> The FsPermission(FsPermission) constructor copies u, g, o from the supplied FsPermission object but ignores the sticky bit.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15005-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 05:22:32 2011
Return-Path: <common-issues-return-15005-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8E5A76A6A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 05:22:32 +0000 (UTC)
Received: (qmail 62604 invoked by uid 500); 18 May 2011 05:22:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62397 invoked by uid 500); 18 May 2011 05:22:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62360 invoked by uid 99); 18 May 2011 05:22:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 05:22:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 05:22:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6FB2CCE2E6
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 05:21:47 +0000 (UTC)
Date: Wed, 18 May 2011 05:21:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1484131492.21509.1305696107453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7298) Add test utility for writing
 multi-threaded tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Add test utility for writing multi-threaded tests
-------------------------------------------------

                 Key: HADOOP-7298
                 URL: https://issues.apache.org/jira/browse/HADOOP-7298
             Project: Hadoop Common
          Issue Type: Test
          Components: test
    Affects Versions: 0.22.0
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
             Fix For: 0.22.0


A lot of our tests spawn off multiple threads in order to check various synchronization issues, etc. It's often tedious to write these kinds of tests because you have to manually propagate exceptions back to the main thread, etc.

In HBase we have developed a testing utility which makes writing these kinds of tests much easier. I'd like to copy that utility into Hadoop so we can use it here as well.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15006-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 05:25:32 2011
Return-Path: <common-issues-return-15006-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 091C56AEC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 05:25:32 +0000 (UTC)
Received: (qmail 65747 invoked by uid 500); 18 May 2011 05:25:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65716 invoked by uid 500); 18 May 2011 05:25:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65703 invoked by uid 99); 18 May 2011 05:25:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 05:25:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 05:25:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7D5CDCE454
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 05:24:47 +0000 (UTC)
Date: Wed, 18 May 2011 05:24:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1381772830.21513.1305696287509.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1484131492.21509.1305696107453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7298) Add test utility for writing
 multi-threaded tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035202#comment-13035202 ] 

Todd Lipcon commented on HADOOP-7298:
-------------------------------------

The HBase class lives here:
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/MultithreadedTestUtil.java

For an example usage, see testLABThreading in the following test:
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java

> Add test utility for writing multi-threaded tests
> -------------------------------------------------
>
>                 Key: HADOOP-7298
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7298
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>
> A lot of our tests spawn off multiple threads in order to check various synchronization issues, etc. It's often tedious to write these kinds of tests because you have to manually propagate exceptions back to the main thread, etc.
> In HBase we have developed a testing utility which makes writing these kinds of tests much easier. I'd like to copy that utility into Hadoop so we can use it here as well.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15007-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 05:46:29 2011
Return-Path: <common-issues-return-15007-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B4CB0692A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 05:46:29 +0000 (UTC)
Received: (qmail 82106 invoked by uid 500); 18 May 2011 05:46:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81858 invoked by uid 500); 18 May 2011 05:46:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81839 invoked by uid 99); 18 May 2011 05:46:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 05:46:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 05:46:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8D234CEE0D
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 05:45:47 +0000 (UTC)
Date: Wed, 18 May 2011 05:45:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2113872824.21549.1305697547574.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1484131492.21509.1305696107453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7298) Add test utility for writing
 multi-threaded tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7298:
--------------------------------

    Attachment: hadoop-7298.txt

Cleaned up the hbase utility a bit and added test cases for the test utility.

I think this will make a nice building block to build some stress/fuzz tests for the namenode for HDFS-988. But, no need to commit it until such a test is ready.

> Add test utility for writing multi-threaded tests
> -------------------------------------------------
>
>                 Key: HADOOP-7298
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7298
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7298.txt
>
>
> A lot of our tests spawn off multiple threads in order to check various synchronization issues, etc. It's often tedious to write these kinds of tests because you have to manually propagate exceptions back to the main thread, etc.
> In HBase we have developed a testing utility which makes writing these kinds of tests much easier. I'd like to copy that utility into Hadoop so we can use it here as well.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15008-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 07:21:42 2011
Return-Path: <common-issues-return-15008-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 41A9C6060
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 07:21:42 +0000 (UTC)
Received: (qmail 67473 invoked by uid 500); 18 May 2011 07:21:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67253 invoked by uid 500); 18 May 2011 07:21:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67207 invoked by uid 99); 18 May 2011 07:21:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 07:21:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 07:21:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DE8AFCFAB8
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 07:20:47 +0000 (UTC)
Date: Wed, 18 May 2011 07:20:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1095760346.21644.1305703247908.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7284:
---------------------------------

    Status: Open  (was: Patch Available)

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15009-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 07:21:42 2011
Return-Path: <common-issues-return-15009-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 54EFD6062
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 07:21:42 +0000 (UTC)
Received: (qmail 67479 invoked by uid 500); 18 May 2011 07:21:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67281 invoked by uid 500); 18 May 2011 07:21:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67208 invoked by uid 99); 18 May 2011 07:21:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 07:21:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 07:21:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EE56BCFABA
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 07:20:47 +0000 (UTC)
Date: Wed, 18 May 2011 07:20:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <532212030.21646.1305703247973.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7284:
---------------------------------

    Status: Patch Available  (was: Open)

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15010-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 07:21:42 2011
Return-Path: <common-issues-return-15010-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5D39D6066
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 07:21:42 +0000 (UTC)
Received: (qmail 67566 invoked by uid 500); 18 May 2011 07:21:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67373 invoked by uid 500); 18 May 2011 07:21:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67202 invoked by uid 99); 18 May 2011 07:21:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 07:21:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 07:21:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B8942CFAB4
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 07:20:47 +0000 (UTC)
Date: Wed, 18 May 2011 07:20:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1941584114.21640.1305703247753.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7284:
---------------------------------

    Attachment: trash3.patch

Minor fix - removal of a warning,

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15011-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 07:33:34 2011
Return-Path: <common-issues-return-15011-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0387F63E6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 07:33:34 +0000 (UTC)
Received: (qmail 77232 invoked by uid 500); 18 May 2011 07:33:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77079 invoked by uid 500); 18 May 2011 07:33:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77040 invoked by uid 99); 18 May 2011 07:33:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 07:33:33 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 07:33:32 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C58FCCFE44
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 07:32:52 +0000 (UTC)
Date: Wed, 18 May 2011 07:32:52 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <535597598.21656.1305703972805.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035226#comment-13035226 ] 

Hadoop QA commented on HADOOP-7284:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479557/trash3.patch
  against trunk revision 1104426.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 15 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/466//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/466//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/466//console

This message is automatically generated.

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15012-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 08:36:31 2011
Return-Path: <common-issues-return-15012-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7A77663E5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 08:36:31 +0000 (UTC)
Received: (qmail 31070 invoked by uid 500); 18 May 2011 08:36:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31023 invoked by uid 500); 18 May 2011 08:36:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31015 invoked by uid 99); 18 May 2011 08:36:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 08:36:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 08:36:30 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8AB65CF511
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 08:35:50 +0000 (UTC)
Date: Wed, 18 May 2011 08:35:50 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1459392622.21718.1305707750564.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1195662636.6130.1300891805766.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7208) equals() and hashCode()
 implementation need to change in StandardSocketFactory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035235#comment-13035235 ] 

Uma Maheswara Rao G commented on HADOOP-7208:
---------------------------------------------

Sorry for the delay in replying and patch preparation.

We may need this change if we use our custom StandardSocketFactory

In the equals method if we change the implementation like 
return obj.getClass().getName().equals(this.getClass().getName());

It gives a findbug error.

Is it fine ? is there any way to get approval for the same.

> equals() and hashCode() implementation need to change in StandardSocketFactory
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7208
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7208
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>
>   In Hadoop IPC Client, we are using ClientCache which will maintain the HashMap to keep the Client references.
> private Map<SocketFactory, Client> clients =
>       new HashMap<SocketFactory, Client>();
>  Now let us say, we want use two standard factories with Hadoop. MyStandardSocketFactory (which extends StandardSocketFactory), and StandardSocketFactory. In this case, because of equals and hashcode implementation, MyStandardSocketFactory client can be overridden by StandardSocketFactoryClient

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15013-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 09:38:28 2011
Return-Path: <common-issues-return-15013-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6A3426392
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 09:38:28 +0000 (UTC)
Received: (qmail 86510 invoked by uid 500); 18 May 2011 09:38:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86478 invoked by uid 500); 18 May 2011 09:38:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86470 invoked by uid 99); 18 May 2011 09:38:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 09:38:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 09:38:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AA606CF873
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 09:37:47 +0000 (UTC)
Date: Wed, 18 May 2011 09:37:47 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1418311642.21786.1305711467694.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1195662636.6130.1300891805766.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7208) equals() and hashCode()
 implementation need to change in StandardSocketFactory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035249#comment-13035249 ] 

Steve Loughran commented on HADOOP-7208:
----------------------------------------

checking by classname is wrong because if you have >1 classloader you can have multiple classes with the name -even singleton classes.

Better to use this.getClass().equals(that.getClass())

For a real review you really need to attach the patch


> equals() and hashCode() implementation need to change in StandardSocketFactory
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7208
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7208
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>
>   In Hadoop IPC Client, we are using ClientCache which will maintain the HashMap to keep the Client references.
> private Map<SocketFactory, Client> clients =
>       new HashMap<SocketFactory, Client>();
>  Now let us say, we want use two standard factories with Hadoop. MyStandardSocketFactory (which extends StandardSocketFactory), and StandardSocketFactory. In this case, because of equals and hashcode implementation, MyStandardSocketFactory client can be overridden by StandardSocketFactoryClient

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15014-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 10:22:28 2011
Return-Path: <common-issues-return-15014-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 42DA96216
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 10:22:28 +0000 (UTC)
Received: (qmail 63138 invoked by uid 500); 18 May 2011 10:22:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63106 invoked by uid 500); 18 May 2011 10:22:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63096 invoked by uid 99); 18 May 2011 10:22:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 10:22:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 10:22:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6A74BCF84E
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 10:21:47 +0000 (UTC)
Date: Wed, 18 May 2011 10:21:47 +0000 (UTC)
From: "Vitalii Tymchyshyn (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <572963958.21860.1305714107432.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7299) FSOutputSummer do not flushBuffer on
 OutputStream.flush
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

FSOutputSummer do not flushBuffer on OutputStream.flush
-------------------------------------------------------

                 Key: HADOOP-7299
                 URL: https://issues.apache.org/jira/browse/HADOOP-7299
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
    Affects Versions: 0.21.0
            Reporter: Vitalii Tymchyshyn


When working with "file://" FileSystem (e.g. in tests), ChecksumFileSystem is used.
I am calling next operations:
create
write
hflush
open
seek
read

This results in:

java.lang.IllegalStateException: java.io.IOException: Cannot seek after EOF
	at com.dbl.util.grid.apache.hdfs.FileSystemStorageHolder.get(FileSystemStorageHolder.java:52)
	at com.dbl.util.grid.apache.hdfs.FileSystemStorageTest.testRecursiveSave(FileSystemStorageTest.java:31)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:59)
	at org.junit.internal.runners.MethodRoadie.runTestMethod(MethodRoadie.java:98)
	at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:79)
	at org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:87)
	at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:77)
	at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:42)
	at org.junit.internal.runners.JUnit4ClassRunner.invokeTestMethod(JUnit4ClassRunner.java:88)
	at org.junit.internal.runners.JUnit4ClassRunner.runMethods(JUnit4ClassRunner.java:51)
	at org.junit.internal.runners.JUnit4ClassRunner$1.run(JUnit4ClassRunner.java:44)
	at org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:27)
	at org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:37)
	at org.junit.internal.runners.JUnit4ClassRunner.run(JUnit4ClassRunner.java:42)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:130)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:97)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:192)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:60)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:115)
Caused by: java.io.IOException: Cannot seek after EOF
	at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.seek(ChecksumFileSystem.java:296)
	at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:42)
	at com.dbl.util.grid.apache.hdfs.FileSystemStorageHolder.get(FileSystemStorageHolder.java:49)
	... 26 more


It seems that FSOutputSummer does not implements flush OutputStream method - so hflush gets ignored.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15015-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 11:14:28 2011
Return-Path: <common-issues-return-15015-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9F8566EA0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 11:14:28 +0000 (UTC)
Received: (qmail 52620 invoked by uid 500); 18 May 2011 11:14:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52561 invoked by uid 500); 18 May 2011 11:14:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52553 invoked by uid 99); 18 May 2011 11:14:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 11:14:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 11:14:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DDC16CEE1F
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 11:13:47 +0000 (UTC)
Date: Wed, 18 May 2011 11:13:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1545486769.21979.1305717227904.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1326019706.17705.1305599027427.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7296) The FsPermission(FsPermission)
 constructor does not use the sticky bit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035293#comment-13035293 ] 

Hudson commented on HADOOP-7296:
--------------------------------

Integrated in Hadoop-Common-trunk #692 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/692/])
    HADOOP-7296. The FsPermission(FsPermission) constructor does not use the sticky bit. Contributed by Siddharth Seth

tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1104374
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/permission/FsPermission.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/permission/TestFsPermission.java


> The FsPermission(FsPermission) constructor does not use the sticky bit
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-7296
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7296
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0, 0.23.0
>            Reporter: Siddharth Seth
>            Assignee: Siddharth Seth
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP7296.patch, HADOOP7296_2.patch
>
>
> The FsPermission(FsPermission) constructor copies u, g, o from the supplied FsPermission object but ignores the sticky bit.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15016-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 11:14:30 2011
Return-Path: <common-issues-return-15016-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9746B6EBC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 11:14:30 +0000 (UTC)
Received: (qmail 53211 invoked by uid 500); 18 May 2011 11:14:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53179 invoked by uid 500); 18 May 2011 11:14:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53171 invoked by uid 99); 18 May 2011 11:14:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 11:14:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 11:14:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B0405CEE14
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 11:13:47 +0000 (UTC)
Date: Wed, 18 May 2011 11:13:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1630415618.21973.1305717227718.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <903601160.6842.1305210828211.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7282) getRemoteIp could return null in
 cases where the call is ongoing but the ip went away.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035292#comment-13035292 ] 

Hudson commented on HADOOP-7282:
--------------------------------

Integrated in Hadoop-Common-trunk #692 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/692/])
    HADOOP-7282. ipc.Server.getRemoteIp() may return null.  Contributed by John George

szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1104426
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/ipc/Server.java


> getRemoteIp could return null in cases where the call is ongoing but the ip went away.
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7282
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7282
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>            Reporter: John George
>            Assignee: John George
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7282-1.patch, HADOOP-7282.patch, diffs.txt
>
>
> getRemoteIp gets the ip from socket instead of the stored ip in Connection object. Thus calls to this function could return null when a client disconnected, but the rpc call is still ongoing...

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15017-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 11:34:29 2011
Return-Path: <common-issues-return-15017-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8CC836D93
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 11:34:29 +0000 (UTC)
Received: (qmail 83715 invoked by uid 500); 18 May 2011 11:34:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83655 invoked by uid 500); 18 May 2011 11:34:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83647 invoked by uid 99); 18 May 2011 11:34:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 11:34:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 11:34:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D19F8CF618
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 11:33:48 +0000 (UTC)
Date: Wed, 18 May 2011 11:33:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <769112774.22011.1305718428855.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035300#comment-13035300 ] 

Hudson commented on HADOOP-7291:
--------------------------------

Integrated in Hadoop-Hdfs-22-branch #49 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-22-branch/49/])
    

> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291-3.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15018-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 11:34:52 2011
Return-Path: <common-issues-return-15018-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CDCED6E06
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 11:34:52 +0000 (UTC)
Received: (qmail 85259 invoked by uid 500); 18 May 2011 11:34:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85209 invoked by uid 500); 18 May 2011 11:34:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85193 invoked by uid 99); 18 May 2011 11:34:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 11:34:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 11:34:50 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4559DCF654
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 11:33:51 +0000 (UTC)
Date: Wed, 18 May 2011 11:33:51 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1085297702.22058.1305718431280.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6846) Scripts for building Hadoop 0.22.0
 release
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035309#comment-13035309 ] 

Hudson commented on HADOOP-6846:
--------------------------------

Integrated in Hadoop-Hdfs-22-branch #49 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-22-branch/49/])
    

> Scripts for building Hadoop 0.22.0 release
> ------------------------------------------
>
>                 Key: HADOOP-6846
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6846
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6846.patch, release-scripts.tar.gz
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15019-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 12:42:30 2011
Return-Path: <common-issues-return-15019-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5DE006EF3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 12:42:30 +0000 (UTC)
Received: (qmail 13917 invoked by uid 500); 18 May 2011 12:42:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13881 invoked by uid 500); 18 May 2011 12:42:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13873 invoked by uid 99); 18 May 2011 12:42:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 12:42:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 12:42:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6C512CF1DC
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 12:41:47 +0000 (UTC)
Date: Wed, 18 May 2011 12:41:47 +0000 (UTC)
From: "Niels Basjes (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <733356005.22201.1305722507440.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6238437.286071293099061858.JavaMail.jira@thor>
Subject: [jira] [Updated] (HADOOP-7076) Splittable Gzip
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Niels Basjes updated HADOOP-7076:
---------------------------------

    Attachment: HADOOP-7076-2011-05-18.patch

This patch has no code changes compared to the previous one. Only the Javadoc has been improved to provide additional suggestions for optimal usage of this patch.

> Splittable Gzip
> ---------------
>
>                 Key: HADOOP-7076
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7076
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: Niels Basjes
>            Assignee: Niels Basjes
>         Attachments: HADOOP-7076-2011-01-26.patch, HADOOP-7076-2011-01-29.patch, HADOOP-7076-2011-02-05.patch, HADOOP-7076-2011-02-06.patch, HADOOP-7076-2011-05-18.patch, HADOOP-7076.patch
>
>
> Files compressed with the gzip codec are not splittable due to the nature of the codec.
> This limits the options you have scaling out when reading large gzipped input files.
> Given the fact that gunzipping a 1GiB file usually takes only 2 minutes I figured that for some use cases wasting some resources may result in a shorter job time under certain conditions.
> So reading the entire input file from the start for each split (wasting resources!!) may lead to additional scalability.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15020-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 13:04:30 2011
Return-Path: <common-issues-return-15020-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 30E9E6038
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 13:04:30 +0000 (UTC)
Received: (qmail 62881 invoked by uid 500); 18 May 2011 13:04:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62857 invoked by uid 500); 18 May 2011 13:04:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62849 invoked by uid 99); 18 May 2011 13:04:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 13:04:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 13:04:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5FBA8CF9A4
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 13:03:47 +0000 (UTC)
Date: Wed, 18 May 2011 13:03:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <185548080.22241.1305723827388.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6238437.286071293099061858.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7076) Splittable Gzip
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035355#comment-13035355 ] 

Hadoop QA commented on HADOOP-7076:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479577/HADOOP-7076-2011-05-18.patch
  against trunk revision 1104426.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 5 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/467//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/467//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/467//console

This message is automatically generated.

> Splittable Gzip
> ---------------
>
>                 Key: HADOOP-7076
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7076
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: Niels Basjes
>            Assignee: Niels Basjes
>         Attachments: HADOOP-7076-2011-01-26.patch, HADOOP-7076-2011-01-29.patch, HADOOP-7076-2011-02-05.patch, HADOOP-7076-2011-02-06.patch, HADOOP-7076-2011-05-18.patch, HADOOP-7076.patch
>
>
> Files compressed with the gzip codec are not splittable due to the nature of the codec.
> This limits the options you have scaling out when reading large gzipped input files.
> Given the fact that gunzipping a 1GiB file usually takes only 2 minutes I figured that for some use cases wasting some resources may result in a shorter job time under certain conditions.
> So reading the entire input file from the start for each split (wasting resources!!) may lead to additional scalability.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15021-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 14:32:31 2011
Return-Path: <common-issues-return-15021-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 30FBC6FAE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 14:32:31 +0000 (UTC)
Received: (qmail 1506 invoked by uid 500); 18 May 2011 14:32:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1469 invoked by uid 500); 18 May 2011 14:32:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1461 invoked by uid 99); 18 May 2011 14:32:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 14:32:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 14:32:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2D8C3CFF71
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 14:31:48 +0000 (UTC)
Date: Wed, 18 May 2011 14:31:48 +0000 (UTC)
From: "ramkrishna.s.vasudevan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <528970851.22443.1305729108182.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-5342) DataNodes do not start up because
 InconsistentFSStateException on just part of the disks in use
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035405#comment-13035405 ] 

ramkrishna.s.vasudevan commented on HADOOP-5342:
------------------------------------------------

I would like to suggest 
Pls correct me if am wrong

namespace id is getting updated immediately after one of the disks of the dfs.data.dir got updated. 
instead update the namespace id after parsing all the dfs.data.dir storage directories 

> DataNodes do not start up because InconsistentFSStateException on just part of the disks in use
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5342
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5342
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.18.2
>            Reporter: Christian Kunz
>            Assignee: Hairong Kuang
>            Priority: Critical
>
> After restarting a cluster (including rebooting) the dfs got corrupted because many DataNodes did not start up, running into the following exception:
> 2009-02-26 22:33:53,774 ERROR org.apache.hadoop.dfs.DataNode: org.apache.hadoop.dfs.InconsistentFSStateException: Directory xxx  is in an inconsistent state: version file in current directory is missing.
> 	at org.apache.hadoop.dfs.Storage$StorageDirectory.analyzeStorage(Storage.java:326)
> 	at org.apache.hadoop.dfs.DataStorage.recoverTransitionRead(DataStorage.java:105)
> 	at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:306)
> 	at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:223)
> 	at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:3030)
> 	at org.apache.hadoop.dfs.DataNode.instantiateDataNode(DataNode.java:2985)
> 	at org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:2993)
> 	at org.apache.hadoop.dfs.DataNode.main(DataNode.java:3115)
> This happens when using multiple disks with at least one previously marked as read-only, such that the storage version became out-dated, but after reboot it was mounted read-write, resulting in the DataNode not starting because of out-dated version.
> This is a big headache. If a DataNode has multiple disks of which at least one has the correct storage version then out-dated versions should not bring down the DataNode.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15022-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 15:21:28 2011
Return-Path: <common-issues-return-15022-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 74CD563BF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 15:21:28 +0000 (UTC)
Received: (qmail 90919 invoked by uid 500); 18 May 2011 15:21:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90880 invoked by uid 500); 18 May 2011 15:21:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90810 invoked by uid 99); 18 May 2011 15:21:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 15:21:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 15:21:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AD682CF578
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 15:20:47 +0000 (UTC)
Date: Wed, 18 May 2011 15:20:47 +0000 (UTC)
From: "ramkrishna.s.vasudevan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1078508724.22597.1305732047706.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <282223217.18947.1304436003652.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7256) Resource leak during failure
 scenario of closing of resources.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ramkrishna.s.vasudevan updated HADOOP-7256:
-------------------------------------------

    Attachment: HADOOP-7256-patch-1.patch

> Resource leak during failure scenario of closing of resources. 
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7256
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7256
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2, 0.21.0
>            Reporter: ramkrishna.s.vasudevan
>            Priority: Minor
>         Attachments: HADOOP-7256-patch-1.patch
>
>   Original Estimate: 8h
>  Remaining Estimate: 8h
>
> Problem Statement:
> ===============
> There are chances of resource leak and stream not getting closed 
> Take the case when after copying data we try to close the Input and output stream followed by closing of the socket.
> Suppose an exception occurs while closing the input stream(due to runtime exception) then the subsequent operations of closing the output stream and socket may not happen and there is a chance of resource leak.
> Scenario 
> =======
> During long run of map reduce jobs, the copyFromLocalFile() api is getting called.
> Here we found some exceptions happening. As a result of this we found the lsof value raising leading to resource leak.
> Solution:
> =======
> While doing a close operation of any resource catch the RuntimeException also rather than catching the IOException alone.
> Additionally there are places where we try to close a resource in the catch block.
> If this close fails, we just throw and come out of the current flow.
> In order to avoid this, we can carry out the close operation in the finally block.
> Probable reasons for getting RunTimeExceptions:
> =====================================
> We may get runtime exception from customised hadoop streams like FSDataOutputStream.close() . So better to handle RunTimeExceptions also.
>  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15023-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 15:29:28 2011
Return-Path: <common-issues-return-15023-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 405046692
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 15:29:28 +0000 (UTC)
Received: (qmail 4567 invoked by uid 500); 18 May 2011 15:29:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4474 invoked by uid 500); 18 May 2011 15:29:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4466 invoked by uid 99); 18 May 2011 15:29:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 15:29:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 15:29:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6040ACF97E
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 15:28:47 +0000 (UTC)
Date: Wed, 18 May 2011 15:28:47 +0000 (UTC)
From: "ramkrishna.s.vasudevan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <177203204.22625.1305732527390.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <282223217.18947.1304436003652.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7256) Resource leak during failure
 scenario of closing of resources.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ramkrishna.s.vasudevan updated HADOOP-7256:
-------------------------------------------

    Fix Version/s: 0.23.0
     Hadoop Flags: [Incompatible change, Reviewed]  (was: [Incompatible change])
           Status: Patch Available  (was: Open)

> Resource leak during failure scenario of closing of resources. 
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7256
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7256
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0, 0.20.2
>            Reporter: ramkrishna.s.vasudevan
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7256-patch-1.patch
>
>   Original Estimate: 8h
>  Remaining Estimate: 8h
>
> Problem Statement:
> ===============
> There are chances of resource leak and stream not getting closed 
> Take the case when after copying data we try to close the Input and output stream followed by closing of the socket.
> Suppose an exception occurs while closing the input stream(due to runtime exception) then the subsequent operations of closing the output stream and socket may not happen and there is a chance of resource leak.
> Scenario 
> =======
> During long run of map reduce jobs, the copyFromLocalFile() api is getting called.
> Here we found some exceptions happening. As a result of this we found the lsof value raising leading to resource leak.
> Solution:
> =======
> While doing a close operation of any resource catch the RuntimeException also rather than catching the IOException alone.
> Additionally there are places where we try to close a resource in the catch block.
> If this close fails, we just throw and come out of the current flow.
> In order to avoid this, we can carry out the close operation in the finally block.
> Probable reasons for getting RunTimeExceptions:
> =====================================
> We may get runtime exception from customised hadoop streams like FSDataOutputStream.close() . So better to handle RunTimeExceptions also.
>  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15024-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 15:43:30 2011
Return-Path: <common-issues-return-15024-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E25EC6F30
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 15:43:30 +0000 (UTC)
Received: (qmail 34282 invoked by uid 500); 18 May 2011 15:43:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34249 invoked by uid 500); 18 May 2011 15:43:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34241 invoked by uid 99); 18 May 2011 15:43:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 15:43:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 15:43:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5F7C9CFFCC
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 15:42:47 +0000 (UTC)
Date: Wed, 18 May 2011 15:42:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1430437559.22668.1305733367387.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <282223217.18947.1304436003652.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7256) Resource leak during failure
 scenario of closing of resources.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035447#comment-13035447 ] 

Hadoop QA commented on HADOOP-7256:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479607/HADOOP-7256-patch-1.patch
  against trunk revision 1104426.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/468//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/468//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/468//console

This message is automatically generated.

> Resource leak during failure scenario of closing of resources. 
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7256
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7256
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2, 0.21.0
>            Reporter: ramkrishna.s.vasudevan
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7256-patch-1.patch
>
>   Original Estimate: 8h
>  Remaining Estimate: 8h
>
> Problem Statement:
> ===============
> There are chances of resource leak and stream not getting closed 
> Take the case when after copying data we try to close the Input and output stream followed by closing of the socket.
> Suppose an exception occurs while closing the input stream(due to runtime exception) then the subsequent operations of closing the output stream and socket may not happen and there is a chance of resource leak.
> Scenario 
> =======
> During long run of map reduce jobs, the copyFromLocalFile() api is getting called.
> Here we found some exceptions happening. As a result of this we found the lsof value raising leading to resource leak.
> Solution:
> =======
> While doing a close operation of any resource catch the RuntimeException also rather than catching the IOException alone.
> Additionally there are places where we try to close a resource in the catch block.
> If this close fails, we just throw and come out of the current flow.
> In order to avoid this, we can carry out the close operation in the finally block.
> Probable reasons for getting RunTimeExceptions:
> =====================================
> We may get runtime exception from customised hadoop streams like FSDataOutputStream.close() . So better to handle RunTimeExceptions also.
>  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15025-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 17:37:29 2011
Return-Path: <common-issues-return-15025-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F012664F1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 17:37:28 +0000 (UTC)
Received: (qmail 35623 invoked by uid 500); 18 May 2011 17:37:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35591 invoked by uid 500); 18 May 2011 17:37:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35580 invoked by uid 99); 18 May 2011 17:37:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 17:37:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 17:37:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2600CCFC89
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 17:36:48 +0000 (UTC)
Date: Wed, 18 May 2011 17:36:48 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <537424235.22918.1305740208152.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <187742091.8722.1305246647362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7287) Configuration deprecation mechanism
 doesn't work properly for GenericOptionsParser/Tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7287:
-----------------------------------

    Affects Version/s: 0.23.0
        Fix Version/s: 0.23.0

> Configuration deprecation mechanism doesn't work properly for GenericOptionsParser/Tools
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7287
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7287
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Aaron T. Myers
>            Priority: Blocker
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: hadoop-7287-testcase.txt
>
>
> For example, you can't use -D options on the "hadoop fs" command line in order to specify the deprecated names of configuration options. The issue is that the ordering is:
> - JVM starts
> - GenericOptionsParser creates a Configuration object and calls set() for each of the options specified on command line
> - DistributedFileSystem or other class eventually instantiates HdfsConfiguration which adds the deprecations
> - Some class calls conf.get("new key") and sees the default instead of the version set on the command line

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15026-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 17:37:31 2011
Return-Path: <common-issues-return-15026-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2422D6506
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 17:37:31 +0000 (UTC)
Received: (qmail 35952 invoked by uid 500); 18 May 2011 17:37:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35921 invoked by uid 500); 18 May 2011 17:37:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35892 invoked by uid 99); 18 May 2011 17:37:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 17:37:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 17:37:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C874FCFC7F
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 17:36:47 +0000 (UTC)
Date: Wed, 18 May 2011 17:36:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1460248525.22908.1305740207817.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <187742091.8722.1305246647362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7287) Configuration deprecation mechanism
 doesn't work properly for GenericOptionsParser/Tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers reassigned HADOOP-7287:
--------------------------------------

    Assignee: Aaron T. Myers

> Configuration deprecation mechanism doesn't work properly for GenericOptionsParser/Tools
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7287
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7287
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Aaron T. Myers
>            Priority: Blocker
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: hadoop-7287-testcase.txt
>
>
> For example, you can't use -D options on the "hadoop fs" command line in order to specify the deprecated names of configuration options. The issue is that the ordering is:
> - JVM starts
> - GenericOptionsParser creates a Configuration object and calls set() for each of the options specified on command line
> - DistributedFileSystem or other class eventually instantiates HdfsConfiguration which adds the deprecations
> - Some class calls conf.get("new key") and sees the default instead of the version set on the command line

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15027-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 17:37:31 2011
Return-Path: <common-issues-return-15027-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3D404650D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 17:37:31 +0000 (UTC)
Received: (qmail 36005 invoked by uid 500); 18 May 2011 17:37:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35924 invoked by uid 500); 18 May 2011 17:37:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35893 invoked by uid 99); 18 May 2011 17:37:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 17:37:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 17:37:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A60D8CFC7A
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 17:36:47 +0000 (UTC)
Date: Wed, 18 May 2011 17:36:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <64862320.22903.1305740207676.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <3255382.248681296161204958.JavaMail.jira@thor>
Subject: [jira] [Assigned] (HADOOP-7121) Exceptions while serializing IPC
 call response are not handled well
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon reassigned HADOOP-7121:
-----------------------------------

    Assignee:     (was: Todd Lipcon)

> Exceptions while serializing IPC call response are not handled well
> -------------------------------------------------------------------
>
>                 Key: HADOOP-7121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7121
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Priority: Critical
>              Labels: newbie
>             Fix For: 0.22.0
>
>
> We had a situation where for some reason the serialization of an RPC call's response was throwing OOME. When this happens, the exception is not caught, and the call never gets a response - the client just hangs. Additionally, the OOME propagated all the way to the top of the IPC handler and caused the handler. Plus, the Handler upon exit only logged to stdout and not to the log4j logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15028-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 17:51:30 2011
Return-Path: <common-issues-return-15028-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 902B86F23
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 17:51:30 +0000 (UTC)
Received: (qmail 68228 invoked by uid 500); 18 May 2011 17:51:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68202 invoked by uid 500); 18 May 2011 17:51:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68186 invoked by uid 99); 18 May 2011 17:51:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 17:51:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 17:51:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C7257CF2D7
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 17:50:47 +0000 (UTC)
Date: Wed, 18 May 2011 17:50:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <371640206.22983.1305741047812.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6605:
--------------------------------

          Description: The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.  (was: JAVA_HOME can be set in hadoop-env.sh and hadoop-config.sh should respect this. hadoop-config will source hadoop-env after it determines a config directory. If JAVA_HOME still isn't set, it will try to guess one from a list of Linux usual locations. If one still isn't set it will quit with a verbose error message.)
             Priority: Minor  (was: Major)
    Affects Version/s:     (was: 0.20.2)
        Fix Version/s: 0.22.0
             Assignee: Eli Collins  (was: Chad Metcalf)
              Summary: Add JAVA_HOME detection to hadoop-config  (was: Improve JAVA_HOME detection and handling in hadoop-config)

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15029-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 17:55:28 2011
Return-Path: <common-issues-return-15029-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1D055611B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 17:55:28 +0000 (UTC)
Received: (qmail 79639 invoked by uid 500); 18 May 2011 17:55:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79607 invoked by uid 500); 18 May 2011 17:55:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79591 invoked by uid 99); 18 May 2011 17:55:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 17:55:27 +0000
X-ASF-Spam-Status: No, hits=-1998.0 required=5.0
	tests=ALL_TRUSTED,FB_GET_MEDS,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 17:55:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 68729CF50C
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 17:54:47 +0000 (UTC)
Date: Wed, 18 May 2011 17:54:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1419221698.23009.1305741287424.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7300) Configuration methods that return
 collections are inconsistent about mutability
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Configuration methods that return collections are inconsistent about mutability
-------------------------------------------------------------------------------

                 Key: HADOOP-7300
                 URL: https://issues.apache.org/jira/browse/HADOOP-7300
             Project: Hadoop Common
          Issue Type: Bug
          Components: conf
    Affects Versions: 0.22.0
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
             Fix For: 0.22.0
         Attachments: hadoop-7300.txt

In particular, getTrimmedStringCollection seems to return an immutable collection, whereas getStringCollection returns a mutable one.

IMO we should always return mutable collections since these methods by definition are doing copies.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15031-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 17:55:29 2011
Return-Path: <common-issues-return-15031-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A2C8C6138
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 17:55:29 +0000 (UTC)
Received: (qmail 80236 invoked by uid 500); 18 May 2011 17:55:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80166 invoked by uid 500); 18 May 2011 17:55:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80007 invoked by uid 99); 18 May 2011 17:55:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 17:55:28 +0000
X-ASF-Spam-Status: No, hits=-1998.0 required=5.0
	tests=ALL_TRUSTED,FB_GET_MEDS,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 17:55:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D7453CF51B
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 17:54:47 +0000 (UTC)
Date: Wed, 18 May 2011 17:54:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <875275556.23023.1305741287878.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1419221698.23009.1305741287424.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7300) Configuration methods that return
 collections are inconsistent about mutability
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7300:
--------------------------------

    Status: Patch Available  (was: Open)

> Configuration methods that return collections are inconsistent about mutability
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-7300
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7300
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7300.txt
>
>
> In particular, getTrimmedStringCollection seems to return an immutable collection, whereas getStringCollection returns a mutable one.
> IMO we should always return mutable collections since these methods by definition are doing copies.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15030-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 17:55:30 2011
Return-Path: <common-issues-return-15030-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 440CD6155
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 17:55:30 +0000 (UTC)
Received: (qmail 80176 invoked by uid 500); 18 May 2011 17:55:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80111 invoked by uid 500); 18 May 2011 17:55:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79943 invoked by uid 99); 18 May 2011 17:55:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 17:55:28 +0000
X-ASF-Spam-Status: No, hits=-1998.0 required=5.0
	tests=ALL_TRUSTED,FB_GET_MEDS,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 17:55:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C58D0CF519
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 17:54:47 +0000 (UTC)
Date: Wed, 18 May 2011 17:54:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <770000035.23021.1305741287806.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1419221698.23009.1305741287424.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7300) Configuration methods that return
 collections are inconsistent about mutability
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7300:
--------------------------------

    Attachment: hadoop-7300.txt

> Configuration methods that return collections are inconsistent about mutability
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-7300
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7300
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7300.txt
>
>
> In particular, getTrimmedStringCollection seems to return an immutable collection, whereas getStringCollection returns a mutable one.
> IMO we should always return mutable collections since these methods by definition are doing copies.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15032-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 17:55:30 2011
Return-Path: <common-issues-return-15032-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C2D82616D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 17:55:30 +0000 (UTC)
Received: (qmail 80371 invoked by uid 500); 18 May 2011 17:55:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80327 invoked by uid 500); 18 May 2011 17:55:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80247 invoked by uid 99); 18 May 2011 17:55:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 17:55:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 17:55:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8686FCF50F
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 17:54:47 +0000 (UTC)
Date: Wed, 18 May 2011 17:54:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1941619250.23012.1305741287547.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6605:
--------------------------------

    Attachment: hadoop-6605-1.patch

Patch attached. Implements the behavior in the (updated) description. 

It detects Java for a number of different version of java (Linux platform default, Sun Java)on a couple different host types (Linux, Mac, tar install). It looks for Sun Java first since Hadoop currently requires Sun java. This list can be updated for other host types (eg add /usr/local for Solaris).

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15033-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 17:59:28 2011
Return-Path: <common-issues-return-15033-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9C5B76316
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 17:59:28 +0000 (UTC)
Received: (qmail 85966 invoked by uid 500); 18 May 2011 17:59:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85938 invoked by uid 500); 18 May 2011 17:59:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85930 invoked by uid 99); 18 May 2011 17:59:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 17:59:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 17:59:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DDC1DCF730
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 17:58:47 +0000 (UTC)
Date: Wed, 18 May 2011 17:58:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1645019806.23050.1305741527905.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035524#comment-13035524 ] 

Eli Collins commented on HADOOP-6605:
-------------------------------------

Forgot to mention testing, since the unit tests don't cover this. I verified with this change that:
# JAVA_HOME if set in conf/hadoop-env.sh takes precedence over the term env and this detection
# JAVA_HOME if set by the term env takes precedence over this detection
# JAVA_HOME is now detected if installed w/o being set explicitly, and errs out if none is detected

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15034-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 18:01:33 2011
Return-Path: <common-issues-return-15034-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8F5E76B58
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 18:01:33 +0000 (UTC)
Received: (qmail 90195 invoked by uid 500); 18 May 2011 18:01:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90148 invoked by uid 500); 18 May 2011 18:01:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90103 invoked by uid 99); 18 May 2011 18:01:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:01:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:01:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id ACDFDCF8A7
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 18:00:48 +0000 (UTC)
Date: Wed, 18 May 2011 18:00:48 +0000 (UTC)
From: "Jonathan Hsieh (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <600939426.23052.1305741648704.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7301) FSDataInputStream should expose a
 getWrappedStream method
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

FSDataInputStream should expose a getWrappedStream method
---------------------------------------------------------

                 Key: HADOOP-7301
                 URL: https://issues.apache.org/jira/browse/HADOOP-7301
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Jonathan Hsieh


Ideally FSDataInputStream should expose a getWrappedStream method similarly to how FSDataOutputStream exposes a getWrappedStream method.  Exposing this is useful for verifying correctness in tests cases.  This FSDataInputStream type is the class that the o.a.h.fs.FileSystem.open call returns.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15035-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 18:11:30 2011
Return-Path: <common-issues-return-15035-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 388D96BFD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 18:11:30 +0000 (UTC)
Received: (qmail 12367 invoked by uid 500); 18 May 2011 18:11:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12333 invoked by uid 500); 18 May 2011 18:11:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12321 invoked by uid 99); 18 May 2011 18:11:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:11:30 +0000
X-ASF-Spam-Status: No, hits=-1998.0 required=5.0
	tests=ALL_TRUSTED,FB_GET_MEDS,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:11:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 746BCCFCB3
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 18:10:47 +0000 (UTC)
Date: Wed, 18 May 2011 18:10:47 +0000 (UTC)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <71593733.23105.1305742247473.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1419221698.23009.1305741287424.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7300) Configuration methods that return
 collections are inconsistent about mutability
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035535#comment-13035535 ] 

Chris Douglas commented on HADOOP-7300:
---------------------------------------

+1

> Configuration methods that return collections are inconsistent about mutability
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-7300
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7300
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7300.txt
>
>
> In particular, getTrimmedStringCollection seems to return an immutable collection, whereas getStringCollection returns a mutable one.
> IMO we should always return mutable collections since these methods by definition are doing copies.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15036-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 18:15:30 2011
Return-Path: <common-issues-return-15036-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 561326E76
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 18:15:30 +0000 (UTC)
Received: (qmail 21862 invoked by uid 500); 18 May 2011 18:15:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21832 invoked by uid 500); 18 May 2011 18:15:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21824 invoked by uid 99); 18 May 2011 18:15:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:15:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:15:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A3C64CFE4C
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 18:14:47 +0000 (UTC)
Date: Wed, 18 May 2011 18:14:47 +0000 (UTC)
From: "Jonathan Hsieh (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1057171311.23120.1305742487667.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <600939426.23052.1305741648704.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7301) FSDataInputStream should expose a
 getWrappedStream method
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Hsieh updated HADOOP-7301:
-----------------------------------

    Attachment: hadoop-7301.patch

> FSDataInputStream should expose a getWrappedStream method
> ---------------------------------------------------------
>
>                 Key: HADOOP-7301
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7301
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jonathan Hsieh
>         Attachments: hadoop-7301.patch
>
>
> Ideally FSDataInputStream should expose a getWrappedStream method similarly to how FSDataOutputStream exposes a getWrappedStream method.  Exposing this is useful for verifying correctness in tests cases.  This FSDataInputStream type is the class that the o.a.h.fs.FileSystem.open call returns.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15037-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 18:15:30 2011
Return-Path: <common-issues-return-15037-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7E8426E84
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 18:15:30 +0000 (UTC)
Received: (qmail 22069 invoked by uid 500); 18 May 2011 18:15:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22019 invoked by uid 500); 18 May 2011 18:15:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21909 invoked by uid 99); 18 May 2011 18:15:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:15:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:15:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 90D1ACFE49
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 18:14:47 +0000 (UTC)
Date: Wed, 18 May 2011 18:14:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1977164801.23117.1305742487589.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035537#comment-13035537 ] 

Eli Collins commented on HADOOP-6605:
-------------------------------------

Allen raised some good questions earlier...

bq. Why doesn't this try the java that is the path? 

I assume you mean that if java is in the path it could infer JAVA_HOME by looking at the parent of the containing directory. That's reasonable, think it would make sense to do this in addition to looking in standard locations (since we do care about the java we're looking for). If you feel strongly I can incorporate that into this change.

bq. Why doesn't this verify that $JAVA_HOME/bin/java actually works?

I think it's OK to assume that the java installed in java home works.

bq. Shouldn't it verify that it found JDK6 or better?

That would be a good improvement, it also needs to check for Sun-compatible Java since Hadoop currently depends on Sun-specific classes. Let's handle this in a separate jira.

bq. Why is the error message RH-based specific? Heck, why have the "please download" message at all?

You're right, it shouldn't be. The latest patch is just a sligh modification to the current error message.

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15038-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 18:17:29 2011
Return-Path: <common-issues-return-15038-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 94026666D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 18:17:29 +0000 (UTC)
Received: (qmail 25416 invoked by uid 500); 18 May 2011 18:17:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25373 invoked by uid 500); 18 May 2011 18:17:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25365 invoked by uid 99); 18 May 2011 18:17:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:17:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:17:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D8386CFF5C
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 18:16:48 +0000 (UTC)
Date: Wed, 18 May 2011 18:16:48 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1490420586.23145.1305742608882.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <187742091.8722.1305246647362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7287) Configuration deprecation mechanism
 doesn't work properly for GenericOptionsParser/Tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7287:
-----------------------------------

    Attachment: hadoop-7287-trunk.0.patch

Patch for trunk which addresses the issue.

> Configuration deprecation mechanism doesn't work properly for GenericOptionsParser/Tools
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7287
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7287
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Aaron T. Myers
>            Priority: Blocker
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: hadoop-7287-testcase.txt, hadoop-7287-trunk.0.patch
>
>
> For example, you can't use -D options on the "hadoop fs" command line in order to specify the deprecated names of configuration options. The issue is that the ordering is:
> - JVM starts
> - GenericOptionsParser creates a Configuration object and calls set() for each of the options specified on command line
> - DistributedFileSystem or other class eventually instantiates HdfsConfiguration which adds the deprecations
> - Some class calls conf.get("new key") and sees the default instead of the version set on the command line

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15040-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 18:17:30 2011
Return-Path: <common-issues-return-15040-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 307B7668D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 18:17:30 +0000 (UTC)
Received: (qmail 25709 invoked by uid 500); 18 May 2011 18:17:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25629 invoked by uid 500); 18 May 2011 18:17:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25611 invoked by uid 99); 18 May 2011 18:17:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:17:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:17:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3E50ACFF64
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 18:16:49 +0000 (UTC)
Date: Wed, 18 May 2011 18:16:49 +0000 (UTC)
From: "Jonathan Hsieh (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1001885976.23151.1305742609252.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <600939426.23052.1305741648704.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7301) FSDataInputStream should expose a
 getWrappedStream method
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Hsieh updated HADOOP-7301:
-----------------------------------

    Fix Version/s: 0.23.0

> FSDataInputStream should expose a getWrappedStream method
> ---------------------------------------------------------
>
>                 Key: HADOOP-7301
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7301
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jonathan Hsieh
>            Assignee: Jonathan Hsieh
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7301.patch
>
>
> Ideally FSDataInputStream should expose a getWrappedStream method similarly to how FSDataOutputStream exposes a getWrappedStream method.  Exposing this is useful for verifying correctness in tests cases.  This FSDataInputStream type is the class that the o.a.h.fs.FileSystem.open call returns.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15039-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 18:17:30 2011
Return-Path: <common-issues-return-15039-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2FCF3668B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 18:17:30 +0000 (UTC)
Received: (qmail 25694 invoked by uid 500); 18 May 2011 18:17:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25627 invoked by uid 500); 18 May 2011 18:17:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25612 invoked by uid 99); 18 May 2011 18:17:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:17:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:17:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 37848CFF63
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 18:16:49 +0000 (UTC)
Date: Wed, 18 May 2011 18:16:49 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1711964526.23150.1305742609224.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <187742091.8722.1305246647362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7287) Configuration deprecation mechanism
 doesn't work properly for GenericOptionsParser/Tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7287:
-----------------------------------

    Status: Patch Available  (was: Open)

> Configuration deprecation mechanism doesn't work properly for GenericOptionsParser/Tools
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7287
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7287
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Aaron T. Myers
>            Priority: Blocker
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: hadoop-7287-testcase.txt, hadoop-7287-trunk.0.patch
>
>
> For example, you can't use -D options on the "hadoop fs" command line in order to specify the deprecated names of configuration options. The issue is that the ordering is:
> - JVM starts
> - GenericOptionsParser creates a Configuration object and calls set() for each of the options specified on command line
> - DistributedFileSystem or other class eventually instantiates HdfsConfiguration which adds the deprecations
> - Some class calls conf.get("new key") and sees the default instead of the version set on the command line

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15041-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 18:17:31 2011
Return-Path: <common-issues-return-15041-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5EB9866BF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 18:17:31 +0000 (UTC)
Received: (qmail 26157 invoked by uid 500); 18 May 2011 18:17:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26123 invoked by uid 500); 18 May 2011 18:17:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26115 invoked by uid 99); 18 May 2011 18:17:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:17:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:17:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AE2E6CFF4E
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 18:16:48 +0000 (UTC)
Date: Wed, 18 May 2011 18:16:48 +0000 (UTC)
From: "Jonathan Hsieh (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <319470067.23140.1305742608710.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <600939426.23052.1305741648704.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7301) FSDataInputStream should expose a
 getWrappedStream method
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Hsieh reassigned HADOOP-7301:
--------------------------------------

    Assignee: Jonathan Hsieh

> FSDataInputStream should expose a getWrappedStream method
> ---------------------------------------------------------
>
>                 Key: HADOOP-7301
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7301
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jonathan Hsieh
>            Assignee: Jonathan Hsieh
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7301.patch
>
>
> Ideally FSDataInputStream should expose a getWrappedStream method similarly to how FSDataOutputStream exposes a getWrappedStream method.  Exposing this is useful for verifying correctness in tests cases.  This FSDataInputStream type is the class that the o.a.h.fs.FileSystem.open call returns.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15042-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 18:23:28 2011
Return-Path: <common-issues-return-15042-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 42965637C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 18:23:28 +0000 (UTC)
Received: (qmail 32789 invoked by uid 500); 18 May 2011 18:23:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32758 invoked by uid 500); 18 May 2011 18:23:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32749 invoked by uid 99); 18 May 2011 18:23:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:23:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:23:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9091CCF223
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 18:22:47 +0000 (UTC)
Date: Wed, 18 May 2011 18:22:47 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1593391.23183.1305742967588.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7269) S3 Native should allow
 customizable file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035545#comment-13035545 ] 

Tom White commented on HADOOP-7269:
-----------------------------------

This looks good. Can you run Jets3tNativeS3FileSystemContractTest to check that it works against real S3? You can set credentials in src/test/core-site.xml.

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: HADOOP-7269-S3-metadata-001.diff, HADOOP-7269-S3-metadata-002.diff, HADOOP-7269-S3-metadata-003.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15043-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 18:27:28 2011
Return-Path: <common-issues-return-15043-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 398D766D9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 18:27:28 +0000 (UTC)
Received: (qmail 41406 invoked by uid 500); 18 May 2011 18:27:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41370 invoked by uid 500); 18 May 2011 18:27:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41362 invoked by uid 99); 18 May 2011 18:27:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:27:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:27:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 85E16CF41A
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 18:26:47 +0000 (UTC)
Date: Wed, 18 May 2011 18:26:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <686872308.23199.1305743207545.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <187742091.8722.1305246647362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7287) Configuration deprecation
 mechanism doesn't work properly for GenericOptionsParser/Tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035549#comment-13035549 ] 

Todd Lipcon commented on HADOOP-7287:
-------------------------------------

This seems fairly reasonable. I think since this is such a central part of the code, we should make sure to run the HDFS and MR tests against the patched Common jar. Would you mind doing that and posting results?

> Configuration deprecation mechanism doesn't work properly for GenericOptionsParser/Tools
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7287
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7287
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Aaron T. Myers
>            Priority: Blocker
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: hadoop-7287-testcase.txt, hadoop-7287-trunk.0.patch
>
>
> For example, you can't use -D options on the "hadoop fs" command line in order to specify the deprecated names of configuration options. The issue is that the ordering is:
> - JVM starts
> - GenericOptionsParser creates a Configuration object and calls set() for each of the options specified on command line
> - DistributedFileSystem or other class eventually instantiates HdfsConfiguration which adds the deprecations
> - Some class calls conf.get("new key") and sees the default instead of the version set on the command line

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15044-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 18:27:30 2011
Return-Path: <common-issues-return-15044-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 52C4B66EA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 18:27:30 +0000 (UTC)
Received: (qmail 41654 invoked by uid 500); 18 May 2011 18:27:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41620 invoked by uid 500); 18 May 2011 18:27:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41612 invoked by uid 99); 18 May 2011 18:27:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:27:30 +0000
X-ASF-Spam-Status: No, hits=-1998.0 required=5.0
	tests=ALL_TRUSTED,FB_GET_MEDS,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:27:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5D129CF410
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 18:26:47 +0000 (UTC)
Date: Wed, 18 May 2011 18:26:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <635738882.23193.1305743207377.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1419221698.23009.1305741287424.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7300) Configuration methods that return
 collections are inconsistent about mutability
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035548#comment-13035548 ] 

Hadoop QA commented on HADOOP-7300:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479625/hadoop-7300.txt
  against trunk revision 1104426.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/469//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/469//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/469//console

This message is automatically generated.

> Configuration methods that return collections are inconsistent about mutability
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-7300
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7300
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7300.txt
>
>
> In particular, getTrimmedStringCollection seems to return an immutable collection, whereas getStringCollection returns a mutable one.
> IMO we should always return mutable collections since these methods by definition are doing copies.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15045-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 18:33:28 2011
Return-Path: <common-issues-return-15045-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0C34268B8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 18:33:28 +0000 (UTC)
Received: (qmail 50859 invoked by uid 500); 18 May 2011 18:33:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50709 invoked by uid 500); 18 May 2011 18:33:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50701 invoked by uid 99); 18 May 2011 18:33:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:33:27 +0000
X-ASF-Spam-Status: No, hits=-1998.0 required=5.0
	tests=ALL_TRUSTED,FB_GET_MEDS,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:33:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 64FADCF74C
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 18:32:47 +0000 (UTC)
Date: Wed, 18 May 2011 18:32:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <280634606.23240.1305743567410.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1419221698.23009.1305741287424.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7300) Configuration methods that return
 collections are inconsistent about mutability
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7300:
--------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

Committed to trunk and 0.22. Thanks for the quick review, Chris!

> Configuration methods that return collections are inconsistent about mutability
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-7300
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7300
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7300.txt
>
>
> In particular, getTrimmedStringCollection seems to return an immutable collection, whereas getStringCollection returns a mutable one.
> IMO we should always return mutable collections since these methods by definition are doing copies.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15046-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 18:33:30 2011
Return-Path: <common-issues-return-15046-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 51E20691A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 18:33:30 +0000 (UTC)
Received: (qmail 52275 invoked by uid 500); 18 May 2011 18:33:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52230 invoked by uid 500); 18 May 2011 18:33:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52216 invoked by uid 99); 18 May 2011 18:33:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:33:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:33:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8AA97CF750
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 18:32:47 +0000 (UTC)
Date: Wed, 18 May 2011 18:32:47 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <137836695.23244.1305743567564.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035557#comment-13035557 ] 

Tom White commented on HADOOP-7215:
-----------------------------------

Can this be closed? Looks like it's been committed.

> RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>            Priority: Blocker
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: HADOOP-7215.1.trunk.patch, HADOOP-7215.2.trunk.patch, HADOOP-7215.2xx.patch, HADOOP-7215.3.trunk.patch, HADOOP-7215.debug.patch, HADOOP-7215.debug.patch
>
>
> HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15047-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 18:37:35 2011
Return-Path: <common-issues-return-15047-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B99106EF6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 18:37:35 +0000 (UTC)
Received: (qmail 64071 invoked by uid 500); 18 May 2011 18:37:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64034 invoked by uid 500); 18 May 2011 18:37:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64026 invoked by uid 99); 18 May 2011 18:37:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:37:35 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:37:33 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E46D8CF98E
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 18:36:52 +0000 (UTC)
Date: Wed, 18 May 2011 18:36:52 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1232343467.23266.1305743812931.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6846) Scripts for building Hadoop 0.22.0
 release
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6846:
------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

> Scripts for building Hadoop 0.22.0 release
> ------------------------------------------
>
>                 Key: HADOOP-6846
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6846
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6846.patch, release-scripts.tar.gz
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15048-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 18:41:30 2011
Return-Path: <common-issues-return-15048-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 951546163
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 18:41:30 +0000 (UTC)
Received: (qmail 73541 invoked by uid 500); 18 May 2011 18:41:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73498 invoked by uid 500); 18 May 2011 18:41:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73490 invoked by uid 99); 18 May 2011 18:41:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:41:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:41:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C3F0ACFC2A
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 18:40:47 +0000 (UTC)
Date: Wed, 18 May 2011 18:40:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <555449440.23326.1305744047799.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2109891895.4306.1300818846044.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7205) automatically determine JAVA_HOME on
 OS X
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7205:
--------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

This dupes HADOOP-6605, which handles java detection via looking for specific, standard java home's for various versions of Java (which may be used across OS types) rather than use an os-specific detection method (java_home in OSX). 

Agree with Allen that we avoid OS-specific changes when possible. There are places which care about host-specific and java-specific issues and these may require some host-specific/java-specific changes.

> automatically determine JAVA_HOME on OS X
> -----------------------------------------
>
>                 Key: HADOOP-7205
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7205
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7205.patch
>
>
> OS X provides a java_home command that will return the user's selected jvm.  The hadoop-env.sh should use this command if JAVA_HOME is not set.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15049-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 18:41:30 2011
Return-Path: <common-issues-return-15049-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CBCE76171
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 18:41:30 +0000 (UTC)
Received: (qmail 73772 invoked by uid 500); 18 May 2011 18:41:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73725 invoked by uid 500); 18 May 2011 18:41:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73584 invoked by uid 99); 18 May 2011 18:41:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:41:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:41:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D3AE8CFC2C
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 18:40:47 +0000 (UTC)
Date: Wed, 18 May 2011 18:40:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1638444830.23328.1305744047863.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2109891895.4306.1300818846044.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Closed] (HADOOP-7205) automatically determine JAVA_HOME on
 OS X
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins closed HADOOP-7205.
-------------------------------


> automatically determine JAVA_HOME on OS X
> -----------------------------------------
>
>                 Key: HADOOP-7205
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7205
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7205.patch
>
>
> OS X provides a java_home command that will return the user's selected jvm.  The hadoop-env.sh should use this command if JAVA_HOME is not set.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15050-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 18:43:30 2011
Return-Path: <common-issues-return-15050-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C57CB61E2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 18:43:30 +0000 (UTC)
Received: (qmail 81462 invoked by uid 500); 18 May 2011 18:43:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81425 invoked by uid 500); 18 May 2011 18:43:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81410 invoked by uid 99); 18 May 2011 18:43:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:43:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:43:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AFE8FCFDD6
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 18:42:47 +0000 (UTC)
Date: Wed, 18 May 2011 18:42:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1067055985.23343.1305744167717.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <187742091.8722.1305246647362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7287) Configuration deprecation
 mechanism doesn't work properly for GenericOptionsParser/Tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035572#comment-13035572 ] 

Hadoop QA commented on HADOOP-7287:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479630/hadoop-7287-trunk.0.patch
  against trunk revision 1124368.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/470//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/470//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/470//console

This message is automatically generated.

> Configuration deprecation mechanism doesn't work properly for GenericOptionsParser/Tools
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7287
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7287
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Aaron T. Myers
>            Priority: Blocker
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: hadoop-7287-testcase.txt, hadoop-7287-trunk.0.patch
>
>
> For example, you can't use -D options on the "hadoop fs" command line in order to specify the deprecated names of configuration options. The issue is that the ordering is:
> - JVM starts
> - GenericOptionsParser creates a Configuration object and calls set() for each of the options specified on command line
> - DistributedFileSystem or other class eventually instantiates HdfsConfiguration which adds the deprecations
> - Some class calls conf.get("new key") and sees the default instead of the version set on the command line

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15051-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 18:45:28 2011
Return-Path: <common-issues-return-15051-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 39C8F6614
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 18:45:28 +0000 (UTC)
Received: (qmail 86580 invoked by uid 500); 18 May 2011 18:45:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86544 invoked by uid 500); 18 May 2011 18:45:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86535 invoked by uid 99); 18 May 2011 18:45:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:45:27 +0000
X-ASF-Spam-Status: No, hits=-1998.0 required=5.0
	tests=ALL_TRUSTED,FB_GET_MEDS,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:45:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 607CCCFE9C
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 18:44:47 +0000 (UTC)
Date: Wed, 18 May 2011 18:44:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <476313231.23349.1305744287391.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1419221698.23009.1305741287424.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7300) Configuration methods that return
 collections are inconsistent about mutability
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035573#comment-13035573 ] 

Hudson commented on HADOOP-7300:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #607 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/607/])
    HADOOP-7300. Configuration methods that return collections are inconsistent about mutability. Contributed by Todd Lipcon.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1124368
Files : 
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/conf/TestConfiguration.java
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/conf/Configuration.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/util/StringUtils.java


> Configuration methods that return collections are inconsistent about mutability
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-7300
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7300
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7300.txt
>
>
> In particular, getTrimmedStringCollection seems to return an immutable collection, whereas getStringCollection returns a mutable one.
> IMO we should always return mutable collections since these methods by definition are doing copies.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15052-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 18:49:30 2011
Return-Path: <common-issues-return-15052-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 53225696D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 18:49:30 +0000 (UTC)
Received: (qmail 94481 invoked by uid 500); 18 May 2011 18:49:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94441 invoked by uid 500); 18 May 2011 18:49:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94431 invoked by uid 99); 18 May 2011 18:49:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:49:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:49:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 80A05CFFD3
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 18:48:47 +0000 (UTC)
Date: Wed, 18 May 2011 18:48:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <443059034.23356.1305744527523.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <187742091.8722.1305246647362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7287) Configuration deprecation
 mechanism doesn't work properly for GenericOptionsParser/Tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035575#comment-13035575 ] 

Aaron T. Myers commented on HADOOP-7287:
----------------------------------------

bq. I think since this is such a central part of the code, we should make sure to run the HDFS and MR tests against the patched Common jar. Would you mind doing that and posting results?

Happy to. This is presently underway.

> Configuration deprecation mechanism doesn't work properly for GenericOptionsParser/Tools
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7287
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7287
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Aaron T. Myers
>            Priority: Blocker
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: hadoop-7287-testcase.txt, hadoop-7287-trunk.0.patch
>
>
> For example, you can't use -D options on the "hadoop fs" command line in order to specify the deprecated names of configuration options. The issue is that the ordering is:
> - JVM starts
> - GenericOptionsParser creates a Configuration object and calls set() for each of the options specified on command line
> - DistributedFileSystem or other class eventually instantiates HdfsConfiguration which adds the deprecations
> - Some class calls conf.get("new key") and sees the default instead of the version set on the command line

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15053-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 18:53:31 2011
Return-Path: <common-issues-return-15053-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2354F6C36
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 18:53:31 +0000 (UTC)
Received: (qmail 4905 invoked by uid 500); 18 May 2011 18:53:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4861 invoked by uid 500); 18 May 2011 18:53:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4824 invoked by uid 99); 18 May 2011 18:53:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:53:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 18:53:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 89F02CF1B2
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 18:52:47 +0000 (UTC)
Date: Wed, 18 May 2011 18:52:47 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1378165525.23378.1305744767561.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035578#comment-13035578 ] 

Allen Wittenauer commented on HADOOP-6605:
------------------------------------------

Why this shouldn't be in a one-time setup script rather than executed on *every* hadoop invocation?  Are we going to bastardized all of the scripts to deal with all of the other OS inconsistencies?

I'm definitely -1 at this point.

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15054-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 19:23:30 2011
Return-Path: <common-issues-return-15054-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3DF296F43
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 19:23:30 +0000 (UTC)
Received: (qmail 55984 invoked by uid 500); 18 May 2011 19:23:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55957 invoked by uid 500); 18 May 2011 19:23:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55949 invoked by uid 99); 18 May 2011 19:23:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 19:23:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 19:23:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 76CD9CFEA3
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 19:22:47 +0000 (UTC)
Date: Wed, 18 May 2011 19:22:47 +0000 (UTC)
From: "Jonathan Hsieh (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1845541919.23475.1305746567483.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <600939426.23052.1305741648704.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7301) FSDataInputStream should expose a
 getWrappedStream method
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Hsieh updated HADOOP-7301:
-----------------------------------

    Status: Patch Available  (was: Open)

> FSDataInputStream should expose a getWrappedStream method
> ---------------------------------------------------------
>
>                 Key: HADOOP-7301
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7301
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jonathan Hsieh
>            Assignee: Jonathan Hsieh
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7301.patch
>
>
> Ideally FSDataInputStream should expose a getWrappedStream method similarly to how FSDataOutputStream exposes a getWrappedStream method.  Exposing this is useful for verifying correctness in tests cases.  This FSDataInputStream type is the class that the o.a.h.fs.FileSystem.open call returns.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15055-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 19:27:30 2011
Return-Path: <common-issues-return-15055-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 972ED6016
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 19:27:30 +0000 (UTC)
Received: (qmail 62016 invoked by uid 500); 18 May 2011 19:27:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61978 invoked by uid 500); 18 May 2011 19:27:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61966 invoked by uid 99); 18 May 2011 19:27:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 19:27:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 19:27:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C72C2CF134
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 19:26:47 +0000 (UTC)
Date: Wed, 18 May 2011 19:26:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1614608559.23510.1305746807812.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035606#comment-13035606 ] 

Eli Collins commented on HADOOP-6605:
-------------------------------------

bq. Why this shouldn't be in a one-time setup script rather than executed on every hadoop invocation?

Because the java installation may change between invocations, eg you may eg install java between invocations of the hadoop command, or you may remove one java installation and then install a new one and you don't want the old one (which is no longer present) cached anywhere. Eg

# exec hadoop  # fails due to lack of java
# yum install java-xyz
# exec hadoop  # now picks up java

This detection only kicks in if JAVA_HOME is not explicitly set elsewhere, if you want to always use a particular version of java, just set JAVA_HOME.

Any remaining technical objection?

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15056-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 19:31:28 2011
Return-Path: <common-issues-return-15056-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7E9E66870
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 19:31:28 +0000 (UTC)
Received: (qmail 68150 invoked by uid 500); 18 May 2011 19:31:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68107 invoked by uid 500); 18 May 2011 19:31:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68099 invoked by uid 99); 18 May 2011 19:31:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 19:31:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 19:31:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8CB4FCF2D6
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 19:30:47 +0000 (UTC)
Date: Wed, 18 May 2011 19:30:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1192415109.23517.1305747047573.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035609#comment-13035609 ] 

Daryn Sharp commented on HADOOP-6605:
-------------------------------------

-1: Only in that I don't like the logic as it would apply to OS X, but I do really like the concept.  Minor point is /Library/Java/Home was deprecated in 10.5 in favor of /usr/libexec/java_home.  Ignoring that, if for some reason I had a copy of java in any of the myriad of directories listed prior to /Library/Java/Home, then this patch will pick that (wrong) version of java, instead of the one I set in "Java Preferences".



> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15057-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 19:43:30 2011
Return-Path: <common-issues-return-15057-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2651C6B24
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 19:43:30 +0000 (UTC)
Received: (qmail 92832 invoked by uid 500); 18 May 2011 19:43:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92800 invoked by uid 500); 18 May 2011 19:43:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92792 invoked by uid 99); 18 May 2011 19:43:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 19:43:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 19:43:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 64F47CF733
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 19:42:47 +0000 (UTC)
Date: Wed, 18 May 2011 19:42:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1564799358.23539.1305747767410.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <600939426.23052.1305741648704.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7301) FSDataInputStream should expose a
 getWrappedStream method
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035614#comment-13035614 ] 

Hadoop QA commented on HADOOP-7301:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479628/hadoop-7301.patch
  against trunk revision 1124368.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/471//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/471//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/471//console

This message is automatically generated.

> FSDataInputStream should expose a getWrappedStream method
> ---------------------------------------------------------
>
>                 Key: HADOOP-7301
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7301
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jonathan Hsieh
>            Assignee: Jonathan Hsieh
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7301.patch
>
>
> Ideally FSDataInputStream should expose a getWrappedStream method similarly to how FSDataOutputStream exposes a getWrappedStream method.  Exposing this is useful for verifying correctness in tests cases.  This FSDataInputStream type is the class that the o.a.h.fs.FileSystem.open call returns.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15058-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 19:51:28 2011
Return-Path: <common-issues-return-15058-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8E56161B5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 19:51:28 +0000 (UTC)
Received: (qmail 4406 invoked by uid 500); 18 May 2011 19:51:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4380 invoked by uid 500); 18 May 2011 19:51:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4372 invoked by uid 99); 18 May 2011 19:51:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 19:51:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 19:51:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9BD51CFA7D
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 19:50:47 +0000 (UTC)
Date: Wed, 18 May 2011 19:50:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <82011012.23571.1305748247635.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035619#comment-13035619 ] 

Eli Collins commented on HADOOP-6605:
-------------------------------------

Thanks for the feedback Daryn.  It looks like /usr/libexec/java_home is the right method for newer Mac systems as the paths it returns is not very generic (in the way the others are), eg has particular dot version numbers in the name. How about, if /usr/libexec/java_home is present, the path it returns is put in the list before /Library/Java/Home? This should work for both older and newer versions of OS X. I could remove /Library/Java/Home from the current version of the patch and we could do that as a change that applies atop this one (I don't have a Mac) or I could make this change as part of this patch and you could try it for me?

(As an aside, I'm surprised -1s are coming out on rev 1 of a patch, we normally we go back and forth on the review before veto'ing a patch, ie if there are technical objections that are not addressed in subsequent versions. I will try to address any technical objections raised).

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15059-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 20:05:28 2011
Return-Path: <common-issues-return-15059-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 39E856AE2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 20:05:28 +0000 (UTC)
Received: (qmail 23680 invoked by uid 500); 18 May 2011 20:05:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23655 invoked by uid 500); 18 May 2011 20:05:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23647 invoked by uid 99); 18 May 2011 20:05:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 20:05:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 20:05:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8A72FCF0BA
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 20:04:47 +0000 (UTC)
Date: Wed, 18 May 2011 20:04:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <128632698.23611.1305749087563.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <600939426.23052.1305741648704.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7301) FSDataInputStream should expose a
 getWrappedStream method
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035626#comment-13035626 ] 

Eli Collins commented on HADOOP-7301:
-------------------------------------

Let's add a comment similar to the one FSDataOutputStream#getWrappredStream. Otherwise looks good.

{noformat}
   // Returns the underlying output stream. This is used by unit tests.
{noformat}


> FSDataInputStream should expose a getWrappedStream method
> ---------------------------------------------------------
>
>                 Key: HADOOP-7301
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7301
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jonathan Hsieh
>            Assignee: Jonathan Hsieh
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7301.patch
>
>
> Ideally FSDataInputStream should expose a getWrappedStream method similarly to how FSDataOutputStream exposes a getWrappedStream method.  Exposing this is useful for verifying correctness in tests cases.  This FSDataInputStream type is the class that the o.a.h.fs.FileSystem.open call returns.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15060-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 20:09:28 2011
Return-Path: <common-issues-return-15060-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 96A626B60
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 20:09:28 +0000 (UTC)
Received: (qmail 25600 invoked by uid 500); 18 May 2011 20:09:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25575 invoked by uid 500); 18 May 2011 20:09:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25567 invoked by uid 99); 18 May 2011 20:09:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 20:09:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 20:09:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 921DDCF2B7
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 20:08:47 +0000 (UTC)
Date: Wed, 18 May 2011 20:08:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <960437402.23641.1305749327595.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-6832) Provide a web server plugin that
 uses a static user for the web UI
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon reassigned HADOOP-6832:
-----------------------------------

    Assignee: Todd Lipcon  (was: Owen O'Malley)

I will update Owen's patch

> Provide a web server plugin that uses a static user for the web UI
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6832
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6832
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Todd Lipcon
>         Attachments: h-6382.patch, static-web-user.patch
>
>
> We need a simple plugin that uses a static user for clusters with security that don't want to authenticate users on the web UI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15061-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 20:11:28 2011
Return-Path: <common-issues-return-15061-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C9EE86C81
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 20:11:28 +0000 (UTC)
Received: (qmail 35968 invoked by uid 500); 18 May 2011 20:11:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35911 invoked by uid 500); 18 May 2011 20:11:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35702 invoked by uid 99); 18 May 2011 20:11:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 20:11:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 20:11:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DE295CF46F
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 20:10:47 +0000 (UTC)
Date: Wed, 18 May 2011 20:10:47 +0000 (UTC)
From: "Jonathan Hsieh (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <937581138.23651.1305749447906.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <600939426.23052.1305741648704.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7301) FSDataInputStream should expose a
 getWrappedStream method
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Hsieh updated HADOOP-7301:
-----------------------------------

    Attachment: hadoop-7301.2.patch

updated to add requested comment.

> FSDataInputStream should expose a getWrappedStream method
> ---------------------------------------------------------
>
>                 Key: HADOOP-7301
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7301
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jonathan Hsieh
>            Assignee: Jonathan Hsieh
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7301.2.patch, hadoop-7301.patch
>
>
> Ideally FSDataInputStream should expose a getWrappedStream method similarly to how FSDataOutputStream exposes a getWrappedStream method.  Exposing this is useful for verifying correctness in tests cases.  This FSDataInputStream type is the class that the o.a.h.fs.FileSystem.open call returns.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15062-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 20:19:31 2011
Return-Path: <common-issues-return-15062-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6470C61C9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 20:19:31 +0000 (UTC)
Received: (qmail 63695 invoked by uid 500); 18 May 2011 20:19:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63532 invoked by uid 500); 18 May 2011 20:19:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63256 invoked by uid 99); 18 May 2011 20:19:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 20:19:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 20:19:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 82314CF9C1
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 20:18:47 +0000 (UTC)
Date: Wed, 18 May 2011 20:18:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <121020826.23681.1305749927530.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <600939426.23052.1305741648704.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7301) FSDataInputStream should expose a
 getWrappedStream method
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035639#comment-13035639 ] 

Eli Collins commented on HADOOP-7301:
-------------------------------------

+1

> FSDataInputStream should expose a getWrappedStream method
> ---------------------------------------------------------
>
>                 Key: HADOOP-7301
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7301
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jonathan Hsieh
>            Assignee: Jonathan Hsieh
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7301.2.patch, hadoop-7301.patch
>
>
> Ideally FSDataInputStream should expose a getWrappedStream method similarly to how FSDataOutputStream exposes a getWrappedStream method.  Exposing this is useful for verifying correctness in tests cases.  This FSDataInputStream type is the class that the o.a.h.fs.FileSystem.open call returns.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15063-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 20:19:31 2011
Return-Path: <common-issues-return-15063-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7153261DD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 20:19:31 +0000 (UTC)
Received: (qmail 63733 invoked by uid 500); 18 May 2011 20:19:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63646 invoked by uid 500); 18 May 2011 20:19:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63425 invoked by uid 99); 18 May 2011 20:19:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 20:19:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 20:19:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AB9E4CF9C7
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 20:18:47 +0000 (UTC)
Date: Wed, 18 May 2011 20:18:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <294975823.23686.1305749927699.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <600939426.23052.1305741648704.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7301) FSDataInputStream should expose a
 getWrappedStream method
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7301:
--------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I've committed this. Thanks Jon!

> FSDataInputStream should expose a getWrappedStream method
> ---------------------------------------------------------
>
>                 Key: HADOOP-7301
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7301
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jonathan Hsieh
>            Assignee: Jonathan Hsieh
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7301.2.patch, hadoop-7301.patch
>
>
> Ideally FSDataInputStream should expose a getWrappedStream method similarly to how FSDataOutputStream exposes a getWrappedStream method.  Exposing this is useful for verifying correctness in tests cases.  This FSDataInputStream type is the class that the o.a.h.fs.FileSystem.open call returns.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15064-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 20:23:31 2011
Return-Path: <common-issues-return-15064-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D2ADA6B0D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 20:23:31 +0000 (UTC)
Received: (qmail 83086 invoked by uid 500); 18 May 2011 20:23:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83033 invoked by uid 500); 18 May 2011 20:23:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82999 invoked by uid 99); 18 May 2011 20:23:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 20:23:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 20:23:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D8910CFBD3
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 20:22:47 +0000 (UTC)
Date: Wed, 18 May 2011 20:22:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1284150474.23711.1305750167883.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <600939426.23052.1305741648704.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7301) FSDataInputStream should expose a
 getWrappedStream method
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035645#comment-13035645 ] 

Hadoop QA commented on HADOOP-7301:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479641/hadoop-7301.2.patch
  against trunk revision 1124368.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/472//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/472//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/472//console

This message is automatically generated.

> FSDataInputStream should expose a getWrappedStream method
> ---------------------------------------------------------
>
>                 Key: HADOOP-7301
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7301
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jonathan Hsieh
>            Assignee: Jonathan Hsieh
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7301.2.patch, hadoop-7301.patch
>
>
> Ideally FSDataInputStream should expose a getWrappedStream method similarly to how FSDataOutputStream exposes a getWrappedStream method.  Exposing this is useful for verifying correctness in tests cases.  This FSDataInputStream type is the class that the o.a.h.fs.FileSystem.open call returns.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15065-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 20:35:28 2011
Return-Path: <common-issues-return-15065-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7F35D6191
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 20:35:28 +0000 (UTC)
Received: (qmail 94876 invoked by uid 500); 18 May 2011 20:35:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94838 invoked by uid 500); 18 May 2011 20:35:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94830 invoked by uid 99); 18 May 2011 20:35:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 20:35:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 20:35:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C1B1CD0148
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 20:34:47 +0000 (UTC)
Date: Wed, 18 May 2011 20:34:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <237053536.23743.1305750887790.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-6832) Provide a web server plugin that
 uses a static user for the web UI
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley reassigned HADOOP-6832:
-------------------------------------

    Assignee: Owen O'Malley  (was: Todd Lipcon)

> Provide a web server plugin that uses a static user for the web UI
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6832
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6832
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: h-6382.patch, static-web-user.patch
>
>
> We need a simple plugin that uses a static user for clusters with security that don't want to authenticate users on the web UI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15066-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 20:35:28 2011
Return-Path: <common-issues-return-15066-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ABA74619F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 20:35:28 +0000 (UTC)
Received: (qmail 95108 invoked by uid 500); 18 May 2011 20:35:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95071 invoked by uid 500); 18 May 2011 20:35:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94921 invoked by uid 99); 18 May 2011 20:35:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 20:35:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 20:35:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D6ACED014B
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 20:34:47 +0000 (UTC)
Date: Wed, 18 May 2011 20:34:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <959392417.23746.1305750887876.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6832) Provide a web server plugin that
 uses a static user for the web UI
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035650#comment-13035650 ] 

Owen O'Malley commented on HADOOP-6832:
---------------------------------------

I forgot about this one. I'll go ahead and do it.

> Provide a web server plugin that uses a static user for the web UI
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6832
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6832
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: h-6382.patch, static-web-user.patch
>
>
> We need a simple plugin that uses a static user for clusters with security that don't want to authenticate users on the web UI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15067-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 20:35:30 2011
Return-Path: <common-issues-return-15067-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A198C61CC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 20:35:30 +0000 (UTC)
Received: (qmail 95779 invoked by uid 500); 18 May 2011 20:35:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95747 invoked by uid 500); 18 May 2011 20:35:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95735 invoked by uid 99); 18 May 2011 20:35:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 20:35:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 20:35:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 98B28D0142
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 20:34:47 +0000 (UTC)
Date: Wed, 18 May 2011 20:34:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <796130917.23739.1305750887622.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <600939426.23052.1305741648704.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7301) FSDataInputStream should expose a
 getWrappedStream method
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035649#comment-13035649 ] 

Hudson commented on HADOOP-7301:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #608 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/608/])
    HADOOP-7301. FSDataInputStream should expose a getWrappedStream method. Contributed by Jonathan Hsieh

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1124406
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/FSDataInputStream.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/FSMainOperationsBaseTest.java


> FSDataInputStream should expose a getWrappedStream method
> ---------------------------------------------------------
>
>                 Key: HADOOP-7301
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7301
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jonathan Hsieh
>            Assignee: Jonathan Hsieh
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7301.2.patch, hadoop-7301.patch
>
>
> Ideally FSDataInputStream should expose a getWrappedStream method similarly to how FSDataOutputStream exposes a getWrappedStream method.  Exposing this is useful for verifying correctness in tests cases.  This FSDataInputStream type is the class that the o.a.h.fs.FileSystem.open call returns.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15068-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 20:39:30 2011
Return-Path: <common-issues-return-15068-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B8A076623
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 20:39:30 +0000 (UTC)
Received: (qmail 7095 invoked by uid 500); 18 May 2011 20:39:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7054 invoked by uid 500); 18 May 2011 20:39:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7046 invoked by uid 99); 18 May 2011 20:39:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 20:39:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 20:39:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 83490D02E9
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 20:38:47 +0000 (UTC)
Date: Wed, 18 May 2011 20:38:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <582937045.23758.1305751127534.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6832) Provide a web server plugin that
 uses a static user for the web UI
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035652#comment-13035652 ] 

Owen O'Malley commented on HADOOP-6832:
---------------------------------------

Actually, looking back, since it is already committed to 0.20.203.0, let's go ahead and commit this to trunk and file a new jira. Does that make sense Todd?

> Provide a web server plugin that uses a static user for the web UI
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6832
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6832
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: h-6382.patch, static-web-user.patch
>
>
> We need a simple plugin that uses a static user for clusters with security that don't want to authenticate users on the web UI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15069-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 20:43:28 2011
Return-Path: <common-issues-return-15069-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4BA4B6732
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 20:43:28 +0000 (UTC)
Received: (qmail 10264 invoked by uid 500); 18 May 2011 20:43:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10191 invoked by uid 500); 18 May 2011 20:43:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10183 invoked by uid 99); 18 May 2011 20:43:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 20:43:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 20:43:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 99DF1D04AC
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 20:42:47 +0000 (UTC)
Date: Wed, 18 May 2011 20:42:47 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <226726444.23789.1305751367627.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035654#comment-13035654 ] 

Allen Wittenauer commented on HADOOP-6605:
------------------------------------------

java_home is not a stable interface.  It cannot be trusted to last, especially given the question marks that hang over Java on OS X.

I'm -1 for a variety of reasons:

a) we're getting into platform specific code without a good reason
b) this could have significant performance ramifications as we add more and more paths to support more and more OSes
c) there is a config option to specifically eliminate the need to do this

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15070-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 20:51:30 2011
Return-Path: <common-issues-return-15070-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 948F66F17
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 20:51:30 +0000 (UTC)
Received: (qmail 22505 invoked by uid 500); 18 May 2011 20:51:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22467 invoked by uid 500); 18 May 2011 20:51:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22459 invoked by uid 99); 18 May 2011 20:51:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 20:51:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 20:51:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8E0FCD076D
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 20:50:47 +0000 (UTC)
Date: Wed, 18 May 2011 20:50:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <121278988.23830.1305751847578.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035659#comment-13035659 ] 

Eli Collins commented on HADOOP-6605:
-------------------------------------

bq. a) we're getting into platform specific code without a good reason

The motivation is that users don't shouldn't have to manually set an environment variable that we can detect automatically. Multiple users have requested this. Why is this not a good reason? We make improvements both for new users (who want an easier out of the box experience) and experienced admins (who will manually configure HADOOP_HOME).

bq. b) this could have significant performance ramifications as we add more and more paths to support more and more OSes

How so?  The new code *only* executes if HADOOP_HOME is not already set, and the code just checks for the existence of bin/java in a set of directories, it bails as soon as it finds one. This is not an expensive operation.

bq. c) there is a config option to specifically eliminate the need to do this

The existence of a configuration parameter does not mean we're not allowed to try to detect a reasonable default if it's not set. We also automatically set a default MAX_HEAP_SIZE and let users override it, this is not different.

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15071-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 20:53:30 2011
Return-Path: <common-issues-return-15071-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8A81B6FBF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 20:53:30 +0000 (UTC)
Received: (qmail 25603 invoked by uid 500); 18 May 2011 20:53:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25565 invoked by uid 500); 18 May 2011 20:53:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25557 invoked by uid 99); 18 May 2011 20:53:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 20:53:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 20:53:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C8AB4D0855
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 20:52:47 +0000 (UTC)
Date: Wed, 18 May 2011 20:52:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <748905583.23844.1305751967818.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6832) Provide a web server plugin that
 uses a static user for the web UI
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035662#comment-13035662 ] 

Todd Lipcon commented on HADOOP-6832:
-------------------------------------

Actually nearly complete with the new patch for trunk. Since this was committed to 203 even though it hadn't been +1ed here, I don't see why that should hold this back from being fixed.

> Provide a web server plugin that uses a static user for the web UI
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6832
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6832
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: h-6382.patch, static-web-user.patch
>
>
> We need a simple plugin that uses a static user for clusters with security that don't want to authenticate users on the web UI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15072-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 21:07:28 2011
Return-Path: <common-issues-return-15072-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 73A4369C4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 21:07:28 +0000 (UTC)
Received: (qmail 47553 invoked by uid 500); 18 May 2011 21:07:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47527 invoked by uid 500); 18 May 2011 21:07:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47518 invoked by uid 99); 18 May 2011 21:07:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 21:07:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 21:07:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9F819D0D06
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 21:06:47 +0000 (UTC)
Date: Wed, 18 May 2011 21:06:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <442145780.23882.1305752807650.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035669#comment-13035669 ] 

Daryn Sharp commented on HADOOP-6605:
-------------------------------------

(My apologies on the neg.  I thought no number meant "I have general concerns but it's ok if someone else +1s", whereas -1 meant "I think there's something really wrong I want addressed" as in the path handling.)

Personally, I don't think it's too messy if hadoop knows how to make a few reasonable guesses, based on conventions, for a few common OSes if JAVA_HOME isn't already set.  The tiny performance impact is only taken by those that chose to not set JAVA_HOME.

Something like the following could be used to restrict the search path for the appropriate os.  

{code}
case "$(uname -s)" in
  Darwin)
    candidates=($(/usr/libexec/java_home) /Library/Java/Home)
    ;;
  Windows)
    candidates=(...no idea...)
    ;;
  Linux)
    candidates=(/usr/java/default /usr/lib/jvm/default-java <maybe something "alternatives" based> ...)
    ...
    ;;
esac
{code}


> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15073-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 21:07:30 2011
Return-Path: <common-issues-return-15073-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BB9A069E0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 21:07:30 +0000 (UTC)
Received: (qmail 48004 invoked by uid 500); 18 May 2011 21:07:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47980 invoked by uid 500); 18 May 2011 21:07:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47972 invoked by uid 99); 18 May 2011 21:07:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 21:07:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 21:07:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 034B6D0D11
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 21:06:48 +0000 (UTC)
Date: Wed, 18 May 2011 21:06:48 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <393793425.23893.1305752808010.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6962) FileSystem.mkdirs(Path,
 FSPermission) should use the permission for all of the created directories
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035671#comment-13035671 ] 

Sanjay Radia commented on HADOOP-6962:
--------------------------------------

This breaks compatibility. That's a big deal.

BTW are the semantics different for localfs and hdfs?

> FileSystem.mkdirs(Path, FSPermission) should use the permission for all of the created directories
> --------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6962
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6962
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Owen O'Malley
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-6962.patch
>
>
> Currently, FileSystem.mkdirs only applies the permissions to the last level if it was created. It should be applied to *all* levels that are created.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15074-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 21:17:37 2011
Return-Path: <common-issues-return-15074-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CBB366975
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 21:17:37 +0000 (UTC)
Received: (qmail 70122 invoked by uid 500); 18 May 2011 21:17:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70077 invoked by uid 500); 18 May 2011 21:17:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70047 invoked by uid 99); 18 May 2011 21:17:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 21:17:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 21:17:34 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8AD28CF161
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 21:16:54 +0000 (UTC)
Date: Wed, 18 May 2011 21:16:54 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <11194223.23953.1305753414565.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035679#comment-13035679 ] 

Daryn Sharp commented on HADOOP-6605:
-------------------------------------

bq. java_home is not a stable interface. It cannot be trusted to last, especially given the question marks that hang over Java on OS X.

This is not true. Apple engineers tell devs on their java-dev list to use java_home.  The following technote says to use java_home.  The utility was added in part so future non-system provided jvms don't monkey with system controlled directories.

http://developer.apple.com/library/mac/#qa/qa1170/_index.html

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15075-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 21:23:30 2011
Return-Path: <common-issues-return-15075-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A70A866DB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 21:23:30 +0000 (UTC)
Received: (qmail 80806 invoked by uid 500); 18 May 2011 21:23:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80759 invoked by uid 500); 18 May 2011 21:23:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80751 invoked by uid 99); 18 May 2011 21:23:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 21:23:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 21:23:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id ABC19CF4DD
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 21:22:47 +0000 (UTC)
Date: Wed, 18 May 2011 21:22:47 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1546407342.23991.1305753767700.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035686#comment-13035686 ] 

Allen Wittenauer commented on HADOOP-6605:
------------------------------------------

I'm sure that would remain true if Apple was still going to supply the JVM.  Given that the responsibility has shifted to Oracle, all bets are off.

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15076-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 21:23:30 2011
Return-Path: <common-issues-return-15076-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D9A7866EC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 21:23:30 +0000 (UTC)
Received: (qmail 81036 invoked by uid 500); 18 May 2011 21:23:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80987 invoked by uid 500); 18 May 2011 21:23:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80966 invoked by uid 99); 18 May 2011 21:23:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 21:23:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 21:23:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DBBCACF4E5
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 21:22:47 +0000 (UTC)
Date: Wed, 18 May 2011 21:22:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1834833382.23997.1305753767896.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <600939426.23052.1305741648704.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7301) FSDataInputStream should expose a
 getWrappedStream method
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035687#comment-13035687 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7301:
------------------------------------------------

Hi, new public method needs javadoc.

> FSDataInputStream should expose a getWrappedStream method
> ---------------------------------------------------------
>
>                 Key: HADOOP-7301
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7301
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jonathan Hsieh
>            Assignee: Jonathan Hsieh
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7301.2.patch, hadoop-7301.patch
>
>
> Ideally FSDataInputStream should expose a getWrappedStream method similarly to how FSDataOutputStream exposes a getWrappedStream method.  Exposing this is useful for verifying correctness in tests cases.  This FSDataInputStream type is the class that the o.a.h.fs.FileSystem.open call returns.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15077-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 21:27:30 2011
Return-Path: <common-issues-return-15077-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 58F21698F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 21:27:30 +0000 (UTC)
Received: (qmail 94846 invoked by uid 500); 18 May 2011 21:27:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94807 invoked by uid 500); 18 May 2011 21:27:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94799 invoked by uid 99); 18 May 2011 21:27:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 21:27:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 21:27:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AA268CF77E
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 21:26:47 +0000 (UTC)
Date: Wed, 18 May 2011 21:26:47 +0000 (UTC)
From: "Jonathan Hsieh (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <611716361.24038.1305754007693.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <600939426.23052.1305741648704.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7301) FSDataInputStream should expose a
 getWrappedStream method
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035694#comment-13035694 ] 

Jonathan Hsieh commented on HADOOP-7301:
----------------------------------------

Nicholas, 

The patch seems to be committed now.  Let me know how you would like me to proceed: (new patch? incremental patch? new jira?) 

Jon.

> FSDataInputStream should expose a getWrappedStream method
> ---------------------------------------------------------
>
>                 Key: HADOOP-7301
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7301
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jonathan Hsieh
>            Assignee: Jonathan Hsieh
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7301.2.patch, hadoop-7301.patch
>
>
> Ideally FSDataInputStream should expose a getWrappedStream method similarly to how FSDataOutputStream exposes a getWrappedStream method.  Exposing this is useful for verifying correctness in tests cases.  This FSDataInputStream type is the class that the o.a.h.fs.FileSystem.open call returns.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15078-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 21:31:30 2011
Return-Path: <common-issues-return-15078-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A2EA16236
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 21:31:30 +0000 (UTC)
Received: (qmail 8558 invoked by uid 500); 18 May 2011 21:31:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8498 invoked by uid 500); 18 May 2011 21:31:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8485 invoked by uid 99); 18 May 2011 21:31:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 21:31:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 21:31:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B89BDCF9F9
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 21:30:47 +0000 (UTC)
Date: Wed, 18 May 2011 21:30:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1810925148.24071.1305754247752.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6962) FileSystem.mkdirs(Path,
 FSPermission) should use the permission for all of the created directories
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035697#comment-13035697 ] 

Daryn Sharp commented on HADOOP-6962:
-------------------------------------

Hadoop's mkdirs used to follow posix and create dirs with correct permissions, but the semantics were broken sometime after 0.20.9.  Yes, currently the semantics differ for localfs (follows posix) and hdfs (does not).  This patch restores posix compliance for hdfs mkdirs.


> FileSystem.mkdirs(Path, FSPermission) should use the permission for all of the created directories
> --------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6962
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6962
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Owen O'Malley
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-6962.patch
>
>
> Currently, FileSystem.mkdirs only applies the permissions to the last level if it was created. It should be applied to *all* levels that are created.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15079-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 21:37:29 2011
Return-Path: <common-issues-return-15079-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4537A6273
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 21:37:29 +0000 (UTC)
Received: (qmail 27874 invoked by uid 500); 18 May 2011 21:37:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27833 invoked by uid 500); 18 May 2011 21:37:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27819 invoked by uid 99); 18 May 2011 21:37:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 21:37:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 21:37:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7A2E4CFE2F
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 21:36:48 +0000 (UTC)
Date: Wed, 18 May 2011 21:36:48 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1062156855.24116.1305754608497.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <600939426.23052.1305741648704.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7301) FSDataInputStream should expose a
 getWrappedStream method
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035702#comment-13035702 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7301:
------------------------------------------------

Hi Jon, either reverting the committed patch and reopening this, or fixing it in a new jira is fine.  Thanks for the fast response.

> FSDataInputStream should expose a getWrappedStream method
> ---------------------------------------------------------
>
>                 Key: HADOOP-7301
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7301
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jonathan Hsieh
>            Assignee: Jonathan Hsieh
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7301.2.patch, hadoop-7301.patch
>
>
> Ideally FSDataInputStream should expose a getWrappedStream method similarly to how FSDataOutputStream exposes a getWrappedStream method.  Exposing this is useful for verifying correctness in tests cases.  This FSDataInputStream type is the class that the o.a.h.fs.FileSystem.open call returns.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15080-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 21:43:28 2011
Return-Path: <common-issues-return-15080-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 98D4D642A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 21:43:28 +0000 (UTC)
Received: (qmail 40036 invoked by uid 500); 18 May 2011 21:43:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40012 invoked by uid 500); 18 May 2011 21:43:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40004 invoked by uid 99); 18 May 2011 21:43:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 21:43:28 +0000
X-ASF-Spam-Status: No, hits=-1998.0 required=5.0
	tests=ALL_TRUSTED,FB_GET_MEDS,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 21:43:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 89509D01E9
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 21:42:47 +0000 (UTC)
Date: Wed, 18 May 2011 21:42:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1999246873.24177.1305754967559.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1419221698.23009.1305741287424.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7300) Configuration methods that return
 collections are inconsistent about mutability
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035708#comment-13035708 ] 

Hudson commented on HADOOP-7300:
--------------------------------

Integrated in Hadoop-Common-22-branch #51 (See [https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/51/])
    HADOOP-7300. Configuration methods that return collections are inconsistent about mutability. Contributed by Todd Lipcon.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1124370
Files : 
* /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/util/StringUtils.java
* /hadoop/common/branches/branch-0.22/CHANGES.txt
* /hadoop/common/branches/branch-0.22/src/test/core/org/apache/hadoop/conf/TestConfiguration.java
* /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/conf/Configuration.java


> Configuration methods that return collections are inconsistent about mutability
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-7300
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7300
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7300.txt
>
>
> In particular, getTrimmedStringCollection seems to return an immutable collection, whereas getStringCollection returns a mutable one.
> IMO we should always return mutable collections since these methods by definition are doing copies.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15081-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 21:45:30 2011
Return-Path: <common-issues-return-15081-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 229EB692F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 21:45:30 +0000 (UTC)
Received: (qmail 44052 invoked by uid 500); 18 May 2011 21:45:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44023 invoked by uid 500); 18 May 2011 21:45:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44011 invoked by uid 99); 18 May 2011 21:45:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 21:45:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 21:45:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6573DD0285
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 21:44:47 +0000 (UTC)
Date: Wed, 18 May 2011 21:44:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <989471619.24181.1305755087411.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6832) Provide a web server plugin that
 uses a static user for the web UI
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6832:
----------------------------------

    Attachment: h-6832.patch

Adds the configuration variable and makes it part of the default list.

> Provide a web server plugin that uses a static user for the web UI
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6832
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6832
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: h-6382.patch, h-6832.patch, static-web-user.patch
>
>
> We need a simple plugin that uses a static user for clusters with security that don't want to authenticate users on the web UI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15082-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 21:49:28 2011
Return-Path: <common-issues-return-15082-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2228E6B77
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 21:49:28 +0000 (UTC)
Received: (qmail 50898 invoked by uid 500); 18 May 2011 21:49:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50872 invoked by uid 500); 18 May 2011 21:49:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50856 invoked by uid 99); 18 May 2011 21:49:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 21:49:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 21:49:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 767B4D05BE
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 21:48:47 +0000 (UTC)
Date: Wed, 18 May 2011 21:48:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <787799661.24190.1305755327481.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6832) Provide a web server plugin that
 uses a static user for the web UI
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6832:
----------------------------------

    Attachment: h-6832.patch

Forgot to use the default user.

> Provide a web server plugin that uses a static user for the web UI
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6832
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6832
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: h-6382.patch, h-6832.patch, h-6832.patch, hadoop-6832.txt, static-web-user.patch
>
>
> We need a simple plugin that uses a static user for clusters with security that don't want to authenticate users on the web UI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15083-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 21:49:30 2011
Return-Path: <common-issues-return-15083-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C75226BA3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 21:49:30 +0000 (UTC)
Received: (qmail 51616 invoked by uid 500); 18 May 2011 21:49:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51569 invoked by uid 500); 18 May 2011 21:49:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51557 invoked by uid 99); 18 May 2011 21:49:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 21:49:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 21:49:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BF512D05C9
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 21:48:47 +0000 (UTC)
Date: Wed, 18 May 2011 21:48:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <759197849.24200.1305755327777.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6832) Provide a web server plugin that
 uses a static user for the web UI
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6832:
--------------------------------

    Attachment: hadoop-6832.txt

Patch similar to Owen's but with some improvements:
- doesn't rely on static variables to configure the static user
- adds unit tests
- documents the new configuration

> Provide a web server plugin that uses a static user for the web UI
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6832
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6832
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: h-6382.patch, h-6832.patch, h-6832.patch, hadoop-6832.txt, static-web-user.patch
>
>
> We need a simple plugin that uses a static user for clusters with security that don't want to authenticate users on the web UI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15084-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 22:07:28 2011
Return-Path: <common-issues-return-15084-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 91B1068F2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 22:07:28 +0000 (UTC)
Received: (qmail 88563 invoked by uid 500); 18 May 2011 22:07:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88539 invoked by uid 500); 18 May 2011 22:07:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88529 invoked by uid 99); 18 May 2011 22:07:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 22:07:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 22:07:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CA665CF517
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 22:06:47 +0000 (UTC)
Date: Wed, 18 May 2011 22:06:47 +0000 (UTC)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <495008496.24288.1305756407825.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <957076305.12265.1297477317500.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7137) Remove hod contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035778#comment-13035778 ] 

Nigel Daley commented on HADOOP-7137:
-------------------------------------

Just removed contrib/hod from Jira and add to the Attic wiki: http://wiki.apache.org/hadoop/Attic

> Remove hod contrib
> ------------------
>
>                 Key: HADOOP-7137
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7137
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7137.patch, HADOOP-7137.patch
>
>
> As per vote on general@ (http://mail-archives.apache.org/mod_mbox/hadoop-general/201102.mbox/%3CAC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com%3E) I will 
> svn remove common/trunk/src/contrib/hod
> using this Jira.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15085-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 22:15:30 2011
Return-Path: <common-issues-return-15085-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2F7816B8E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 22:15:30 +0000 (UTC)
Received: (qmail 98219 invoked by uid 500); 18 May 2011 22:15:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98187 invoked by uid 500); 18 May 2011 22:15:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98179 invoked by uid 99); 18 May 2011 22:15:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 22:15:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 22:15:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6D202CF952
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 22:14:47 +0000 (UTC)
Date: Wed, 18 May 2011 22:14:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <368080761.24327.1305756887443.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <187742091.8722.1305246647362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7287) Configuration deprecation
 mechanism doesn't work properly for GenericOptionsParser/Tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035786#comment-13035786 ] 

Aaron T. Myers commented on HADOOP-7287:
----------------------------------------

HDFS tests which failed:

org.apache.hadoop.tools.TestJMXGet
org.apache.hadoop.hdfs.server.namenode.TestBlocksWithNotEnoughRacks
org.apache.hadoop.hdfs.TestDecommission
org.apache.hadoop.hdfs.TestDatanodeBlockScanner
org.apache.hadoop.hdfs.TestHDFSTrash

All of which, I believe, are known to be flaky or failing. I was able to run TestBlocksWithNotEnoughRacks, TestDecommission, and TestDatanodeBlockScanner individually, and they passed.

Running the M/R tests now.

> Configuration deprecation mechanism doesn't work properly for GenericOptionsParser/Tools
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7287
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7287
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Aaron T. Myers
>            Priority: Blocker
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: hadoop-7287-testcase.txt, hadoop-7287-trunk.0.patch
>
>
> For example, you can't use -D options on the "hadoop fs" command line in order to specify the deprecated names of configuration options. The issue is that the ordering is:
> - JVM starts
> - GenericOptionsParser creates a Configuration object and calls set() for each of the options specified on command line
> - DistributedFileSystem or other class eventually instantiates HdfsConfiguration which adds the deprecations
> - Some class calls conf.get("new key") and sees the default instead of the version set on the command line

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15086-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 22:23:30 2011
Return-Path: <common-issues-return-15086-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3876A621A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 22:23:30 +0000 (UTC)
Received: (qmail 10367 invoked by uid 500); 18 May 2011 22:23:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10335 invoked by uid 500); 18 May 2011 22:23:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10327 invoked by uid 99); 18 May 2011 22:23:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 22:23:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 22:23:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7722ECFE07
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 22:22:47 +0000 (UTC)
Date: Wed, 18 May 2011 22:22:47 +0000 (UTC)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <746180913.24396.1305757367484.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <957076305.12265.1297477317500.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7137) Remove hod contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035797#comment-13035797 ] 

Nigel Daley commented on HADOOP-7137:
-------------------------------------

Oops, I mean I closed as won't fix all the hod Jira's.  I didn't actually removed the Jira component.

> Remove hod contrib
> ------------------
>
>                 Key: HADOOP-7137
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7137
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7137.patch, HADOOP-7137.patch
>
>
> As per vote on general@ (http://mail-archives.apache.org/mod_mbox/hadoop-general/201102.mbox/%3CAC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com%3E) I will 
> svn remove common/trunk/src/contrib/hod
> using this Jira.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15087-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 22:27:30 2011
Return-Path: <common-issues-return-15087-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A4A3762B4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 22:27:30 +0000 (UTC)
Received: (qmail 13531 invoked by uid 500); 18 May 2011 22:27:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13488 invoked by uid 500); 18 May 2011 22:27:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13480 invoked by uid 99); 18 May 2011 22:27:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 22:27:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 22:27:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AE8E8D002D
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 22:26:47 +0000 (UTC)
Date: Wed, 18 May 2011 22:26:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <135654339.24421.1305757607712.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6832) Provide a web server plugin that
 uses a static user for the web UI
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6832:
----------------------------------

    Attachment: h-6832.patch

thanks for the test case, todd.

incorporated todd's suggestions, although i'm not wild about doing the username lookup for each request.

The default user is already in the code and doesn't need to be in the configuration file too.

Using a non-unix userid means the chance for false positives is much lower. Besides, dr. who is fun.

> Provide a web server plugin that uses a static user for the web UI
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6832
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6832
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: h-6382.patch, h-6832.patch, h-6832.patch, h-6832.patch, hadoop-6832.txt, static-web-user.patch
>
>
> We need a simple plugin that uses a static user for clusters with security that don't want to authenticate users on the web UI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15088-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 22:29:28 2011
Return-Path: <common-issues-return-15088-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5746C6549
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 22:29:28 +0000 (UTC)
Received: (qmail 19840 invoked by uid 500); 18 May 2011 22:29:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19813 invoked by uid 500); 18 May 2011 22:29:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19805 invoked by uid 99); 18 May 2011 22:29:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 22:29:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 22:29:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 87852D0161
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 22:28:47 +0000 (UTC)
Date: Wed, 18 May 2011 22:28:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1055691373.24444.1305757727552.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6832) Provide a web server plugin that
 uses a static user for the web UI
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035804#comment-13035804 ] 

Hadoop QA commented on HADOOP-6832:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479664/hadoop-6832.txt
  against trunk revision 1124406.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 2 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/473//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/473//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/473//console

This message is automatically generated.

> Provide a web server plugin that uses a static user for the web UI
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6832
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6832
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: h-6382.patch, h-6832.patch, h-6832.patch, h-6832.patch, hadoop-6832.txt, static-web-user.patch
>
>
> We need a simple plugin that uses a static user for clusters with security that don't want to authenticate users on the web UI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15089-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 22:31:31 2011
Return-Path: <common-issues-return-15089-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A91416D07
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 22:31:31 +0000 (UTC)
Received: (qmail 25057 invoked by uid 500); 18 May 2011 22:31:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25018 invoked by uid 500); 18 May 2011 22:31:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25006 invoked by uid 99); 18 May 2011 22:31:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 22:31:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 22:31:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7CBD8D01FA
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 22:30:48 +0000 (UTC)
Date: Wed, 18 May 2011 22:30:48 +0000 (UTC)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <748675750.24459.1305757848507.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <957076305.12265.1297477317500.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7137) Remove hod contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nigel Daley updated HADOOP-7137:
--------------------------------

    Attachment: HADOOP-7137-additional.patch

A patch to also remove HOD english docs.

> Remove hod contrib
> ------------------
>
>                 Key: HADOOP-7137
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7137
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7137-additional.patch, HADOOP-7137.patch, HADOOP-7137.patch
>
>
> As per vote on general@ (http://mail-archives.apache.org/mod_mbox/hadoop-general/201102.mbox/%3CAC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com%3E) I will 
> svn remove common/trunk/src/contrib/hod
> using this Jira.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15090-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 22:33:28 2011
Return-Path: <common-issues-return-15090-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 66BD16603
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 22:33:28 +0000 (UTC)
Received: (qmail 26450 invoked by uid 500); 18 May 2011 22:33:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26412 invoked by uid 500); 18 May 2011 22:33:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26404 invoked by uid 99); 18 May 2011 22:33:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 22:33:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 22:33:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8E151D0261
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 22:32:47 +0000 (UTC)
Date: Wed, 18 May 2011 22:32:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2056810736.24467.1305757967578.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6832) Provide a web server plugin that
 uses a static user for the web UI
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035808#comment-13035808 ] 

Todd Lipcon commented on HADOOP-6832:
-------------------------------------

bq. incorporated todd's suggestions, although i'm not wild about doing the username lookup for each request.
What do you mean by "username lookup" here? The conf lookup only happens when the HttpServer starts. The "new User()" construction does happen for every request, but that's hardly heavy lifting (eg we do the same thing on every RPC)

bq. The default user is already in the code and doesn't need to be in the configuration file too.
The same could be said for every other default throughout all of common, HDFS, and MR, no? This is our status quo.

bq. Using a non-unix userid means the chance for false positives is much lower. Besides, dr. who is fun.
Apparently most of our very-confused customers who've filed tickets about this don't agree. To quote one operator:
{noformat}
One thing I'd like to add is the fact that DrWho.Tardis are the default
user.group for Hadoop is horrible and confusing. For my $0.02 on this as
a sys-admin I want a useful error message that doesn't make me go digging
though the Hadoop Jira for 15 minutes
{noformat}

> Provide a web server plugin that uses a static user for the web UI
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6832
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6832
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: h-6382.patch, h-6832.patch, h-6832.patch, h-6832.patch, hadoop-6832.txt, static-web-user.patch
>
>
> We need a simple plugin that uses a static user for clusters with security that don't want to authenticate users on the web UI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15091-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 22:37:30 2011
Return-Path: <common-issues-return-15091-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 785736B7E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 22:37:30 +0000 (UTC)
Received: (qmail 35310 invoked by uid 500); 18 May 2011 22:37:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35261 invoked by uid 500); 18 May 2011 22:37:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35253 invoked by uid 99); 18 May 2011 22:37:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 22:37:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 22:37:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B217BD037E
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 22:36:47 +0000 (UTC)
Date: Wed, 18 May 2011 22:36:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <87781171.24477.1305758207726.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6962) FileSystem.mkdirs(Path,
 FSPermission) should use the permission for all of the created directories
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035810#comment-13035810 ] 

Eli Collins commented on HADOOP-6962:
-------------------------------------

bq. the semantics were broken sometime after 0.20.9

What is 0.20.9?

> FileSystem.mkdirs(Path, FSPermission) should use the permission for all of the created directories
> --------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6962
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6962
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Owen O'Malley
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-6962.patch
>
>
> Currently, FileSystem.mkdirs only applies the permissions to the last level if it was created. It should be applied to *all* levels that are created.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15092-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 22:41:30 2011
Return-Path: <common-issues-return-15092-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B250A6C39
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 22:41:30 +0000 (UTC)
Received: (qmail 42330 invoked by uid 500); 18 May 2011 22:41:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42298 invoked by uid 500); 18 May 2011 22:41:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42288 invoked by uid 99); 18 May 2011 22:41:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 22:41:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 22:41:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E4F87D04CE
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 22:40:47 +0000 (UTC)
Date: Wed, 18 May 2011 22:40:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1919470452.24499.1305758447934.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <957076305.12265.1297477317500.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7137) Remove hod contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035815#comment-13035815 ] 

Eli Collins commented on HADOOP-7137:
-------------------------------------

+1

I built and verified the generated docs.

> Remove hod contrib
> ------------------
>
>                 Key: HADOOP-7137
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7137
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7137-additional.patch, HADOOP-7137.patch, HADOOP-7137.patch
>
>
> As per vote on general@ (http://mail-archives.apache.org/mod_mbox/hadoop-general/201102.mbox/%3CAC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com%3E) I will 
> svn remove common/trunk/src/contrib/hod
> using this Jira.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15093-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 22:45:28 2011
Return-Path: <common-issues-return-15093-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8362D6FDC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 22:45:28 +0000 (UTC)
Received: (qmail 46591 invoked by uid 500); 18 May 2011 22:45:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46545 invoked by uid 500); 18 May 2011 22:45:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46532 invoked by uid 99); 18 May 2011 22:45:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 22:45:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 22:45:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B4A3DD0628
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 22:44:47 +0000 (UTC)
Date: Wed, 18 May 2011 22:44:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <441095169.24523.1305758687736.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <957076305.12265.1297477317500.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Closed] (HADOOP-7137) Remove hod contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins closed HADOOP-7137.
-------------------------------


I've committed this to branch 22 and trunk. Thanks Nigel!

> Remove hod contrib
> ------------------
>
>                 Key: HADOOP-7137
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7137
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7137-additional.patch, HADOOP-7137.patch, HADOOP-7137.patch
>
>
> As per vote on general@ (http://mail-archives.apache.org/mod_mbox/hadoop-general/201102.mbox/%3CAC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com%3E) I will 
> svn remove common/trunk/src/contrib/hod
> using this Jira.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15094-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 23:01:28 2011
Return-Path: <common-issues-return-15094-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B17596DAB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 23:01:28 +0000 (UTC)
Received: (qmail 68502 invoked by uid 500); 18 May 2011 23:01:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68434 invoked by uid 500); 18 May 2011 23:01:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68374 invoked by uid 99); 18 May 2011 23:01:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 23:01:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 23:01:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CE1AFD0C10
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 23:00:47 +0000 (UTC)
Date: Wed, 18 May 2011 23:00:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1916735165.24590.1305759647840.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6832) Provide a web server plugin that
 uses a static user for the web UI
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035829#comment-13035829 ] 

Owen O'Malley commented on HADOOP-6832:
---------------------------------------

{quote}
What do you mean by "username lookup" here?
{quote}
{code}
+      this.username = conf.getInitParameter(USERNAME_KEY);
+      this.user = new User(username);
{code}
I agree it isn't huge, which is why I left it in, but it also isn't adding any value.

{quote}
The same could be said for every other default throughout all of common, HDFS, and MR, no?
{quote}

No, actually. We do it in a lot of places, but it leads to lots of confusion. Furthermore, this isn't a framework default, but a plugin default. I don't think it is appropriate to put into the default configuration file.

{quote}
Apparently most of our very-confused customers who've filed tickets about this don't agree. 
{quote}

The issue isn't the default value of "dr.who". Having a magic value of "webuser" will have *exactly* the same problem with trying to figure out where it is coming from. Furthermore, if they *actually* have a real *webuser* account, it could create problems. I guess we could use "default.web.user" or something. 


> Provide a web server plugin that uses a static user for the web UI
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6832
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6832
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: h-6382.patch, h-6832.patch, h-6832.patch, h-6832.patch, hadoop-6832.txt, static-web-user.patch
>
>
> We need a simple plugin that uses a static user for clusters with security that don't want to authenticate users on the web UI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15095-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 23:07:28 2011
Return-Path: <common-issues-return-15095-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B27536CC2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 23:07:28 +0000 (UTC)
Received: (qmail 76348 invoked by uid 500); 18 May 2011 23:07:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76310 invoked by uid 500); 18 May 2011 23:07:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76302 invoked by uid 99); 18 May 2011 23:07:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 23:07:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 23:07:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 169D6D0F50
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 23:06:48 +0000 (UTC)
Date: Wed, 18 May 2011 23:06:48 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1162391771.24670.1305760008089.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6832) Provide a web server plugin that
 uses a static user for the web UI
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6832:
----------------------------------

    Status: Patch Available  (was: Open)

> Provide a web server plugin that uses a static user for the web UI
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6832
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6832
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: h-6382.patch, h-6832.patch, h-6832.patch, h-6832.patch, hadoop-6832.txt, static-web-user.patch
>
>
> We need a simple plugin that uses a static user for clusters with security that don't want to authenticate users on the web UI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15096-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 23:07:30 2011
Return-Path: <common-issues-return-15096-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A26646CE5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 23:07:30 +0000 (UTC)
Received: (qmail 76970 invoked by uid 500); 18 May 2011 23:07:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76920 invoked by uid 500); 18 May 2011 23:07:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76836 invoked by uid 99); 18 May 2011 23:07:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 23:07:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 23:07:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B4F76D0F46
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 23:06:47 +0000 (UTC)
Date: Wed, 18 May 2011 23:06:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <624032516.24660.1305760007738.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6832) Provide a web server plugin that
 uses a static user for the web UI
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6832:
----------------------------------

    Status: Open  (was: Patch Available)

> Provide a web server plugin that uses a static user for the web UI
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6832
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6832
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: h-6382.patch, h-6832.patch, h-6832.patch, h-6832.patch, hadoop-6832.txt, static-web-user.patch
>
>
> We need a simple plugin that uses a static user for clusters with security that don't want to authenticate users on the web UI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15097-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 23:41:30 2011
Return-Path: <common-issues-return-15097-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 435686F4D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 23:41:30 +0000 (UTC)
Received: (qmail 26780 invoked by uid 500); 18 May 2011 23:41:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26746 invoked by uid 500); 18 May 2011 23:41:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26738 invoked by uid 99); 18 May 2011 23:41:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 23:41:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 23:41:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4D918D095C
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 23:40:47 +0000 (UTC)
Date: Wed, 18 May 2011 23:40:47 +0000 (UTC)
From: "Ari Rabkin (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2117896899.24772.1305762047299.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7302) webinterface.private.actions should
 not be in common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

webinterface.private.actions should not be in common
----------------------------------------------------

                 Key: HADOOP-7302
                 URL: https://issues.apache.org/jira/browse/HADOOP-7302
             Project: Hadoop Common
          Issue Type: Bug
          Components: documentation
    Affects Versions: 0.23.0
            Reporter: Ari Rabkin
            Assignee: Ari Rabkin
             Fix For: 0.23.0


The comment in -defaults says that it applies to both NN and JT. This is wrong. This option is only referenced via the JobTracker. 
I propose to delete it here, and file a second issue in MAPREDUCE for renaming the option (deprecating the existing name.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15098-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 23:43:30 2011
Return-Path: <common-issues-return-15098-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 148506F8C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 23:43:30 +0000 (UTC)
Received: (qmail 28576 invoked by uid 500); 18 May 2011 23:43:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28545 invoked by uid 500); 18 May 2011 23:43:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28537 invoked by uid 99); 18 May 2011 23:43:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 23:43:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 23:43:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 501B4D09ED
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 23:42:47 +0000 (UTC)
Date: Wed, 18 May 2011 23:42:47 +0000 (UTC)
From: "Ari Rabkin (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1477077700.24774.1305762167309.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2117896899.24772.1305762047299.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7302) webinterface.private.actions should
 not be in common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ari Rabkin updated HADOOP-7302:
-------------------------------

    Description: 
The comment in -defaults says that webinterface.private.actions applies to both NN and JT. This is wrong. This option is only referenced via the JobTracker. 

I propose to delete it here, and file a second issue in MAPREDUCE for renaming the option (deprecating the existing name.)

  was:
The comment in -defaults says that it applies to both NN and JT. This is wrong. This option is only referenced via the JobTracker. 
I propose to delete it here, and file a second issue in MAPREDUCE for renaming the option (deprecating the existing name.)


> webinterface.private.actions should not be in common
> ----------------------------------------------------
>
>                 Key: HADOOP-7302
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7302
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.23.0
>            Reporter: Ari Rabkin
>            Assignee: Ari Rabkin
>             Fix For: 0.23.0
>
>
> The comment in -defaults says that webinterface.private.actions applies to both NN and JT. This is wrong. This option is only referenced via the JobTracker. 
> I propose to delete it here, and file a second issue in MAPREDUCE for renaming the option (deprecating the existing name.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15099-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 23:49:30 2011
Return-Path: <common-issues-return-15099-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 61A2765CC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 23:49:30 +0000 (UTC)
Received: (qmail 37150 invoked by uid 500); 18 May 2011 23:49:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37124 invoked by uid 500); 18 May 2011 23:49:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37111 invoked by uid 99); 18 May 2011 23:49:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 23:49:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 23:49:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 647CFD0CAB
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 23:48:47 +0000 (UTC)
Date: Wed, 18 May 2011 23:48:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2133591247.24822.1305762527408.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6605:
--------------------------------

    Attachment: hadoop-6605-2.patch

Updated patch attached. Incorporates Daryn's suggestion: builds the candidates array based on the OS type (Solaris, OSX and Linux).

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch, hadoop-6605-2.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15100-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 23:53:27 2011
Return-Path: <common-issues-return-15100-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EC410665F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 23:53:27 +0000 (UTC)
Received: (qmail 42709 invoked by uid 500); 18 May 2011 23:53:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42683 invoked by uid 500); 18 May 2011 23:53:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42674 invoked by uid 99); 18 May 2011 23:53:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 23:53:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 23:53:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 53A48D0D7C
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 23:52:47 +0000 (UTC)
Date: Wed, 18 May 2011 23:52:47 +0000 (UTC)
From: "Ari Rabkin (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1840560349.24836.1305762767339.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2117896899.24772.1305762047299.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7302) webinterface.private.actions should
 not be in common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ari Rabkin updated HADOOP-7302:
-------------------------------

    Attachment: HADOOP-7302.patch

> webinterface.private.actions should not be in common
> ----------------------------------------------------
>
>                 Key: HADOOP-7302
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7302
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.23.0
>            Reporter: Ari Rabkin
>            Assignee: Ari Rabkin
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7302.patch
>
>
> The comment in -defaults says that webinterface.private.actions applies to both NN and JT. This is wrong. This option is only referenced via the JobTracker. 
> I propose to delete it here, and file a second issue in MAPREDUCE for renaming the option (deprecating the existing name.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15101-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 23:55:28 2011
Return-Path: <common-issues-return-15101-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 30B8766C9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 23:55:28 +0000 (UTC)
Received: (qmail 44025 invoked by uid 500); 18 May 2011 23:55:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43989 invoked by uid 500); 18 May 2011 23:55:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43981 invoked by uid 99); 18 May 2011 23:55:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 23:55:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 23:55:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 726ACD0E30
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 23:54:47 +0000 (UTC)
Date: Wed, 18 May 2011 23:54:47 +0000 (UTC)
From: "Ari Rabkin (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1634536089.24838.1305762887465.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2117896899.24772.1305762047299.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7302) webinterface.private.actions should
 not be in common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ari Rabkin updated HADOOP-7302:
-------------------------------

    Release Note: option webinterface.private.actions has been renamed to mapreduce.jobtracker.webinterface.trusted
          Status: Patch Available  (was: Open)

Note: no change to code, hence no tests.

> webinterface.private.actions should not be in common
> ----------------------------------------------------
>
>                 Key: HADOOP-7302
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7302
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.23.0
>            Reporter: Ari Rabkin
>            Assignee: Ari Rabkin
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7302.patch
>
>
> The comment in -defaults says that webinterface.private.actions applies to both NN and JT. This is wrong. This option is only referenced via the JobTracker. 
> I propose to delete it here, and file a second issue in MAPREDUCE for renaming the option (deprecating the existing name.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15102-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 23:55:30 2011
Return-Path: <common-issues-return-15102-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7831366D8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 23:55:30 +0000 (UTC)
Received: (qmail 44308 invoked by uid 500); 18 May 2011 23:55:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44246 invoked by uid 500); 18 May 2011 23:55:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44238 invoked by uid 99); 18 May 2011 23:55:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 23:55:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 23:55:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A53EFD0E37
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 23:54:47 +0000 (UTC)
Date: Wed, 18 May 2011 23:54:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2138318482.24845.1305762887673.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6962) FileSystem.mkdirs(Path,
 FSPermission) should use the permission for all of the created directories
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035867#comment-13035867 ] 

Sanjay Radia commented on HADOOP-6962:
--------------------------------------

20.09 is internal yahoo release -- at least on the internal release the semantics appeared to have changed.  The internal Y! release does not matter to the larger Hadoop community except that he may have helped detect a problem.
Main questions are:
* Did the semantics change and if so  accidentally or on purpose.
* What is the right semantics?
* Are we breaking compatibility if we change the behavior.

I have always disliked the recursive mkdirs.

> FileSystem.mkdirs(Path, FSPermission) should use the permission for all of the created directories
> --------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6962
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6962
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Owen O'Malley
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-6962.patch
>
>
> Currently, FileSystem.mkdirs only applies the permissions to the last level if it was created. It should be applied to *all* levels that are created.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15103-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 18 23:59:31 2011
Return-Path: <common-issues-return-15103-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9C2D167C6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 18 May 2011 23:59:31 +0000 (UTC)
Received: (qmail 49656 invoked by uid 500); 18 May 2011 23:59:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49633 invoked by uid 500); 18 May 2011 23:59:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49625 invoked by uid 99); 18 May 2011 23:59:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 23:59:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 23:59:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4DE8BD0F69
	for <common-issues@hadoop.apache.org>; Wed, 18 May 2011 23:58:48 +0000 (UTC)
Date: Wed, 18 May 2011 23:58:48 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1300358620.24880.1305763128316.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6832) Provide a web server plugin that
 uses a static user for the web UI
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035873#comment-13035873 ] 

Hadoop QA commented on HADOOP-6832:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479679/h-6832.patch
  against trunk revision 1124456.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/474//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/474//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/474//console

This message is automatically generated.

> Provide a web server plugin that uses a static user for the web UI
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6832
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6832
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: h-6382.patch, h-6832.patch, h-6832.patch, h-6832.patch, hadoop-6832.txt, static-web-user.patch
>
>
> We need a simple plugin that uses a static user for clusters with security that don't want to authenticate users on the web UI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15104-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 00:10:28 2011
Return-Path: <common-issues-return-15104-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 87AD56F64
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 00:10:28 +0000 (UTC)
Received: (qmail 62745 invoked by uid 500); 19 May 2011 00:10:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62705 invoked by uid 500); 19 May 2011 00:10:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62697 invoked by uid 99); 19 May 2011 00:10:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 00:10:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 00:10:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DA7A2D03E1
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 00:09:47 +0000 (UTC)
Date: Thu, 19 May 2011 00:09:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1892687593.24913.1305763787891.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6962) FileSystem.mkdirs(Path,
 FSPermission) should use the permission for all of the created directories
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035877#comment-13035877 ] 

Sanjay Radia commented on HADOOP-6962:
--------------------------------------

Daryn. There are 2 mkdirs methods.
* The *static* FileSystem#mkdirs(fs, dir, permission) is very explicitly
creating the dirs with the default and changing the permissions of the new leaf. This is 
not accidental.
* the *non static* Filesystem#mkdirs is an abstract method. Raw local file system implements mkdirs as
{code}
  public boolean mkdirs(Path f, FsPermission permission) throws IOException {
    boolean b = mkdirs(f);
    if(b) {
      setPermission(f, permission);
    }
    return b;
  }
{code}


> FileSystem.mkdirs(Path, FSPermission) should use the permission for all of the created directories
> --------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6962
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6962
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Owen O'Malley
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-6962.patch
>
>
> Currently, FileSystem.mkdirs only applies the permissions to the last level if it was created. It should be applied to *all* levels that are created.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15105-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 00:10:32 2011
Return-Path: <common-issues-return-15105-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 15F1E6F8F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 00:10:32 +0000 (UTC)
Received: (qmail 63396 invoked by uid 500); 19 May 2011 00:10:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63355 invoked by uid 500); 19 May 2011 00:10:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63347 invoked by uid 99); 19 May 2011 00:10:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 00:10:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 00:10:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A50E4D03E8
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 00:09:48 +0000 (UTC)
Date: Thu, 19 May 2011 00:09:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <916126225.24920.1305763788670.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <957076305.12265.1297477317500.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7137) Remove hod contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035878#comment-13035878 ] 

Hudson commented on HADOOP-7137:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #609 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/609/])
    HADOOP-7137. Remove hod contrib docs. Contributed by Nigel Daley

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1124456
Files : 
* /hadoop/common/trunk/src/docs/src/documentation/content/xdocs/site.xml
* /hadoop/common/trunk/src/docs/src/documentation/content/xdocs/hod_scheduler.xml


> Remove hod contrib
> ------------------
>
>                 Key: HADOOP-7137
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7137
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7137-additional.patch, HADOOP-7137.patch, HADOOP-7137.patch
>
>
> As per vote on general@ (http://mail-archives.apache.org/mod_mbox/hadoop-general/201102.mbox/%3CAC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com%3E) I will 
> svn remove common/trunk/src/contrib/hod
> using this Jira.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15106-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 00:18:28 2011
Return-Path: <common-issues-return-15106-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 367B36F94
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 00:18:28 +0000 (UTC)
Received: (qmail 73590 invoked by uid 500); 19 May 2011 00:18:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73555 invoked by uid 500); 19 May 2011 00:18:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73547 invoked by uid 99); 19 May 2011 00:18:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 00:18:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 00:18:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 93A1FD064F
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 00:17:47 +0000 (UTC)
Date: Thu, 19 May 2011 00:17:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <709182728.24947.1305764267601.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <187742091.8722.1305246647362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7287) Configuration deprecation
 mechanism doesn't work properly for GenericOptionsParser/Tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035882#comment-13035882 ] 

Aaron T. Myers commented on HADOOP-7287:
----------------------------------------

All of the M/R core tests passed.

> Configuration deprecation mechanism doesn't work properly for GenericOptionsParser/Tools
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7287
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7287
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Aaron T. Myers
>            Priority: Blocker
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: hadoop-7287-testcase.txt, hadoop-7287-trunk.0.patch
>
>
> For example, you can't use -D options on the "hadoop fs" command line in order to specify the deprecated names of configuration options. The issue is that the ordering is:
> - JVM starts
> - GenericOptionsParser creates a Configuration object and calls set() for each of the options specified on command line
> - DistributedFileSystem or other class eventually instantiates HdfsConfiguration which adds the deprecations
> - Some class calls conf.get("new key") and sees the default instead of the version set on the command line

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15107-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 01:51:28 2011
Return-Path: <common-issues-return-15107-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5DD1D61EA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 01:51:28 +0000 (UTC)
Received: (qmail 64006 invoked by uid 500); 19 May 2011 01:51:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63968 invoked by uid 500); 19 May 2011 01:51:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63960 invoked by uid 99); 19 May 2011 01:51:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 01:51:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 01:51:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7D72FD0F49
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 01:50:47 +0000 (UTC)
Date: Thu, 19 May 2011 01:50:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1427281566.25234.1305769847510.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2117896899.24772.1305762047299.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7302) webinterface.private.actions
 should not be in common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035926#comment-13035926 ] 

Hadoop QA commented on HADOOP-7302:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479691/HADOOP-7302.patch
  against trunk revision 1124456.

    +1 @author.  The patch does not contain any @author tags.

    +0 tests included.  The patch appears to be a documentation patch that doesn't require tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/475//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/475//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/475//console

This message is automatically generated.

> webinterface.private.actions should not be in common
> ----------------------------------------------------
>
>                 Key: HADOOP-7302
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7302
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.23.0
>            Reporter: Ari Rabkin
>            Assignee: Ari Rabkin
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7302.patch
>
>
> The comment in -defaults says that webinterface.private.actions applies to both NN and JT. This is wrong. This option is only referenced via the JobTracker. 
> I propose to delete it here, and file a second issue in MAPREDUCE for renaming the option (deprecating the existing name.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15108-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 03:03:35 2011
Return-Path: <common-issues-return-15108-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1BFF84349
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 03:03:35 +0000 (UTC)
Received: (qmail 22973 invoked by uid 500); 19 May 2011 03:03:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22948 invoked by uid 500); 19 May 2011 03:03:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22924 invoked by uid 99); 19 May 2011 03:03:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 03:03:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 03:03:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 68175D02E6
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 03:02:47 +0000 (UTC)
Date: Thu, 19 May 2011 03:02:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1192476522.25400.1305774167423.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <187742091.8722.1305246647362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7287) Configuration deprecation mechanism
 doesn't work properly for GenericOptionsParser/Tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7287:
--------------------------------

    Attachment: hadoop-7287-still-broken.txt

Though this fixes the test case, it doesn't deal with the case where the "new" key is being set as a default in one of the -default.xml files.

This patch includes another test case that shows the issue.

> Configuration deprecation mechanism doesn't work properly for GenericOptionsParser/Tools
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7287
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7287
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Aaron T. Myers
>            Priority: Blocker
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: hadoop-7287-still-broken.txt, hadoop-7287-testcase.txt, hadoop-7287-trunk.0.patch
>
>
> For example, you can't use -D options on the "hadoop fs" command line in order to specify the deprecated names of configuration options. The issue is that the ordering is:
> - JVM starts
> - GenericOptionsParser creates a Configuration object and calls set() for each of the options specified on command line
> - DistributedFileSystem or other class eventually instantiates HdfsConfiguration which adds the deprecations
> - Some class calls conf.get("new key") and sees the default instead of the version set on the command line

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15109-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 03:03:35 2011
Return-Path: <common-issues-return-15109-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4C6704350
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 03:03:35 +0000 (UTC)
Received: (qmail 23072 invoked by uid 500); 19 May 2011 03:03:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22951 invoked by uid 500); 19 May 2011 03:03:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22927 invoked by uid 99); 19 May 2011 03:03:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 03:03:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 03:03:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A52FFD02EC
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 03:02:47 +0000 (UTC)
Date: Thu, 19 May 2011 03:02:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <654578426.25405.1305774167673.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <187742091.8722.1305246647362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7287) Configuration deprecation mechanism
 doesn't work properly for GenericOptionsParser/Tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7287:
--------------------------------

    Status: Open  (was: Patch Available)

> Configuration deprecation mechanism doesn't work properly for GenericOptionsParser/Tools
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7287
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7287
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Aaron T. Myers
>            Priority: Blocker
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: hadoop-7287-still-broken.txt, hadoop-7287-testcase.txt, hadoop-7287-trunk.0.patch
>
>
> For example, you can't use -D options on the "hadoop fs" command line in order to specify the deprecated names of configuration options. The issue is that the ordering is:
> - JVM starts
> - GenericOptionsParser creates a Configuration object and calls set() for each of the options specified on command line
> - DistributedFileSystem or other class eventually instantiates HdfsConfiguration which adds the deprecations
> - Some class calls conf.get("new key") and sees the default instead of the version set on the command line

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15110-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 04:04:30 2011
Return-Path: <common-issues-return-15110-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 315894EDC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 04:04:30 +0000 (UTC)
Received: (qmail 60644 invoked by uid 500); 19 May 2011 04:04:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60392 invoked by uid 500); 19 May 2011 04:04:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60324 invoked by uid 99); 19 May 2011 04:04:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 04:04:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 04:04:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 88E96D0E8F
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 04:03:47 +0000 (UTC)
Date: Thu, 19 May 2011 04:03:47 +0000 (UTC)
From: "James Warren (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <467412339.25463.1305777827557.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7303) 'docs' target incompatible with
 apache forrest 0.9
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

'docs' target incompatible with apache forrest 0.9
--------------------------------------------------

                 Key: HADOOP-7303
                 URL: https://issues.apache.org/jira/browse/HADOOP-7303
             Project: Hadoop Common
          Issue Type: Bug
          Components: build
         Environment: MacOS 10.6 / Forrest 0.9 / Ant 1.8.2 / JDK 1.6
            Reporter: James Warren


The ant target "cn-docs" fails when using forrest 0.9.  I did confirm that the build is fine using 0.8, and that the docs for HDFS and MAPREDUCE work with 0.9.  I did not try using JDK5 due to HADOOP-7075.

Will attach the stacktrace to this ticket - and I would appreciate confirmation of the bug as today was my first day using forrest.



--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15111-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 04:06:31 2011
Return-Path: <common-issues-return-15111-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 23E6F4F1C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 04:06:31 +0000 (UTC)
Received: (qmail 62359 invoked by uid 500); 19 May 2011 04:06:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62290 invoked by uid 500); 19 May 2011 04:06:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62273 invoked by uid 99); 19 May 2011 04:06:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 04:06:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 04:06:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5A8C6D0EFF
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 04:05:47 +0000 (UTC)
Date: Thu, 19 May 2011 04:05:47 +0000 (UTC)
From: "James Warren (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <883633528.25466.1305777947367.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <467412339.25463.1305777827557.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7303) 'docs' target incompatible with
 apache forrest 0.9
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

James Warren updated HADOOP-7303:
---------------------------------

    Attachment: hadoop-common-forrest-0.9.log

> 'docs' target incompatible with apache forrest 0.9
> --------------------------------------------------
>
>                 Key: HADOOP-7303
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7303
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>         Environment: MacOS 10.6 / Forrest 0.9 / Ant 1.8.2 / JDK 1.6
>            Reporter: James Warren
>         Attachments: hadoop-common-forrest-0.9.log
>
>
> The ant target "cn-docs" fails when using forrest 0.9.  I did confirm that the build is fine using 0.8, and that the docs for HDFS and MAPREDUCE work with 0.9.  I did not try using JDK5 due to HADOOP-7075.
> Will attach the stacktrace to this ticket - and I would appreciate confirmation of the bug as today was my first day using forrest.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15112-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 04:16:34 2011
Return-Path: <common-issues-return-15112-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9CB1F44D8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 04:16:34 +0000 (UTC)
Received: (qmail 65385 invoked by uid 500); 19 May 2011 04:16:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65363 invoked by uid 500); 19 May 2011 04:16:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65355 invoked by uid 99); 19 May 2011 04:16:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 04:16:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 04:16:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B45FBD00CB
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 04:15:47 +0000 (UTC)
Date: Thu, 19 May 2011 04:15:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <217053580.25474.1305778547735.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <467412339.25463.1305777827557.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7303) 'docs' target incompatible with
 apache forrest 0.9
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035973#comment-13035973 ] 

Todd Lipcon commented on HADOOP-7303:
-------------------------------------

hrm... no idea there - maybe worth filing a forrest JIRA and linking the two?

> 'docs' target incompatible with apache forrest 0.9
> --------------------------------------------------
>
>                 Key: HADOOP-7303
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7303
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>         Environment: MacOS 10.6 / Forrest 0.9 / Ant 1.8.2 / JDK 1.6
>            Reporter: James Warren
>         Attachments: hadoop-common-forrest-0.9.log
>
>
> The ant target "cn-docs" fails when using forrest 0.9.  I did confirm that the build is fine using 0.8, and that the docs for HDFS and MAPREDUCE work with 0.9.  I did not try using JDK5 due to HADOOP-7075.
> Will attach the stacktrace to this ticket - and I would appreciate confirmation of the bug as today was my first day using forrest.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15113-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 08:03:36 2011
Return-Path: <common-issues-return-15113-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3E57944BC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 08:03:36 +0000 (UTC)
Received: (qmail 38655 invoked by uid 500); 19 May 2011 08:03:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38609 invoked by uid 500); 19 May 2011 08:03:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38569 invoked by uid 99); 19 May 2011 08:03:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 08:03:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 08:03:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9086DD02ED
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 08:02:47 +0000 (UTC)
Date: Thu, 19 May 2011 08:02:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <307701012.26013.1305792167588.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <187742091.8722.1305246647362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7287) Configuration deprecation mechanism
 doesn't work properly for GenericOptionsParser/Tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7287:
-----------------------------------

    Attachment: hadoop-7287-trunk.1.patch

Great catch, Todd, and thanks for providing the test case. I've attached an updated patch which addresses this issue.

> Configuration deprecation mechanism doesn't work properly for GenericOptionsParser/Tools
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7287
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7287
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Aaron T. Myers
>            Priority: Blocker
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: hadoop-7287-still-broken.txt, hadoop-7287-testcase.txt, hadoop-7287-trunk.0.patch, hadoop-7287-trunk.1.patch
>
>
> For example, you can't use -D options on the "hadoop fs" command line in order to specify the deprecated names of configuration options. The issue is that the ordering is:
> - JVM starts
> - GenericOptionsParser creates a Configuration object and calls set() for each of the options specified on command line
> - DistributedFileSystem or other class eventually instantiates HdfsConfiguration which adds the deprecations
> - Some class calls conf.get("new key") and sees the default instead of the version set on the command line

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15114-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 08:27:29 2011
Return-Path: <common-issues-return-15114-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4BBA346CF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 08:27:29 +0000 (UTC)
Received: (qmail 68841 invoked by uid 500); 19 May 2011 08:27:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68681 invoked by uid 500); 19 May 2011 08:27:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68619 invoked by uid 99); 19 May 2011 08:27:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 08:27:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 08:27:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9AAE6D0CB1
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 08:26:47 +0000 (UTC)
Date: Thu, 19 May 2011 08:26:47 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1734226577.26045.1305793607629.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7269) S3 Native should allow
 customizable file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036062#comment-13036062 ] 

Suresh Srinivas commented on HADOOP-7269:
-----------------------------------------

> Todd has said - Would it make sense to add these functions directly to the S3 filesystems but not to the overall API?
I still see FileSystem.java change in the new patch.

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: HADOOP-7269-S3-metadata-001.diff, HADOOP-7269-S3-metadata-002.diff, HADOOP-7269-S3-metadata-003.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15115-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 08:44:28 2011
Return-Path: <common-issues-return-15115-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A266D40D9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 08:44:28 +0000 (UTC)
Received: (qmail 87698 invoked by uid 500); 19 May 2011 08:44:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87656 invoked by uid 500); 19 May 2011 08:44:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87426 invoked by uid 99); 19 May 2011 08:44:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 08:44:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 08:44:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 81A0DD04A8
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 08:43:47 +0000 (UTC)
Date: Thu, 19 May 2011 08:43:47 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1773866358.26074.1305794627527.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1484131492.21509.1305696107453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7298) Add test utility for writing
 multi-threaded tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036068#comment-13036068 ] 

Suresh Srinivas commented on HADOOP-7298:
-----------------------------------------

Todd, given that this is a utility based on which tests are going to be written, can you add javadoc and possibly examples or brief description on how to use it.

> Add test utility for writing multi-threaded tests
> -------------------------------------------------
>
>                 Key: HADOOP-7298
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7298
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7298.txt
>
>
> A lot of our tests spawn off multiple threads in order to check various synchronization issues, etc. It's often tedious to write these kinds of tests because you have to manually propagate exceptions back to the main thread, etc.
> In HBase we have developed a testing utility which makes writing these kinds of tests much easier. I'd like to copy that utility into Hadoop so we can use it here as well.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15116-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 09:12:28 2011
Return-Path: <common-issues-return-15116-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 556384652
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 09:12:28 +0000 (UTC)
Received: (qmail 42581 invoked by uid 500); 19 May 2011 09:12:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42550 invoked by uid 500); 19 May 2011 09:12:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42541 invoked by uid 99); 19 May 2011 09:12:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 09:12:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 09:12:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8A75BD00F3
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 09:11:47 +0000 (UTC)
Date: Thu, 19 May 2011 09:11:47 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <397574138.26107.1305796307564.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1250812071.19292.1305645587504.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7297) Error in the documentation
 regarding Checkpoint/Backup Node
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036072#comment-13036072 ] 

Suresh Srinivas commented on HADOOP-7297:
-----------------------------------------

namenode -checkpoint and backup nodes were added in 0.21 and they are not in 0.20.203 release.

> Error in the documentation regarding Checkpoint/Backup Node
> -----------------------------------------------------------
>
>                 Key: HADOOP-7297
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7297
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.20.203.0
>            Reporter: arnaud p
>            Priority: Trivial
>
> On http://hadoop.apache.org/common/docs/r0.20.203.0/hdfs_user_guide.html#Checkpoint+Node: the command bin/hdfs namenode -checkpoint required to launch the backup/checkpoint node does not exist.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15117-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 09:12:28 2011
Return-Path: <common-issues-return-15117-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A0DC3465E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 09:12:28 +0000 (UTC)
Received: (qmail 42831 invoked by uid 500); 19 May 2011 09:12:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42784 invoked by uid 500); 19 May 2011 09:12:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42776 invoked by uid 99); 19 May 2011 09:12:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 09:12:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 09:12:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CA195D00FA
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 09:11:47 +0000 (UTC)
Date: Thu, 19 May 2011 09:11:47 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <254790679.26113.1305796307824.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1250812071.19292.1305645587504.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7297) Error in the documentation
 regarding Checkpoint/Backup Node
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas resolved HADOOP-7297.
-------------------------------------

    Resolution: Not A Problem

I do not think this is a problem for 0.20.203.

> Error in the documentation regarding Checkpoint/Backup Node
> -----------------------------------------------------------
>
>                 Key: HADOOP-7297
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7297
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.20.203.0
>            Reporter: arnaud p
>            Priority: Trivial
>
> On http://hadoop.apache.org/common/docs/r0.20.203.0/hdfs_user_guide.html#Checkpoint+Node: the command bin/hdfs namenode -checkpoint required to launch the backup/checkpoint node does not exist.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15118-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 09:20:30 2011
Return-Path: <common-issues-return-15118-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 360F64191
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 09:20:30 +0000 (UTC)
Received: (qmail 65786 invoked by uid 500); 19 May 2011 09:20:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65753 invoked by uid 500); 19 May 2011 09:20:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65743 invoked by uid 99); 19 May 2011 09:20:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 09:20:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 09:20:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5ABB6D0580
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 09:19:47 +0000 (UTC)
Date: Thu, 19 May 2011 09:19:47 +0000 (UTC)
From: "Harsh J Chouraria (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1677465598.26135.1305796787368.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1250812071.19292.1305645587504.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7297) Error in the documentation
 regarding Checkpoint/Backup Node
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036081#comment-13036081 ] 

Harsh J Chouraria commented on HADOOP-7297:
-------------------------------------------

Suresh - They are present in the docs, so one would think they exist in 203 as well. The docs surely ought to be fixed if these features are not present.

(This since OP had posted a link that's 203 doc specific)

> Error in the documentation regarding Checkpoint/Backup Node
> -----------------------------------------------------------
>
>                 Key: HADOOP-7297
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7297
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.20.203.0
>            Reporter: arnaud p
>            Priority: Trivial
>
> On http://hadoop.apache.org/common/docs/r0.20.203.0/hdfs_user_guide.html#Checkpoint+Node: the command bin/hdfs namenode -checkpoint required to launch the backup/checkpoint node does not exist.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15119-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 09:44:30 2011
Return-Path: <common-issues-return-15119-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A5A3B4F22
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 09:44:30 +0000 (UTC)
Received: (qmail 14436 invoked by uid 500); 19 May 2011 09:44:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14408 invoked by uid 500); 19 May 2011 09:44:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14400 invoked by uid 99); 19 May 2011 09:44:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 09:44:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 09:44:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B82BDD10FE
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 09:43:47 +0000 (UTC)
Date: Thu, 19 May 2011 09:43:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <99460319.26200.1305798227751.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <957076305.12265.1297477317500.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7137) Remove hod contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036095#comment-13036095 ] 

Hudson commented on HADOOP-7137:
--------------------------------

Integrated in Hadoop-Common-22-branch #52 (See [https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/52/])
    HADOOP-7137. svn merge -c 1124456 from trunk

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1124457
Files : 
* /hadoop/common/branches/branch-0.22/src/docs
* /hadoop/common/branches/branch-0.22/src/docs/src/documentation/content/xdocs/hod_scheduler.xml
* /hadoop/common/branches/branch-0.22
* /hadoop/common/branches/branch-0.22/src/test/core
* /hadoop/common/branches/branch-0.22/src/test/core/org/apache/hadoop/io/TestSequenceFile.java
* /hadoop/common/branches/branch-0.22/CHANGES.txt
* /hadoop/common/branches/branch-0.22/src/java
* /hadoop/common/branches/branch-0.22/src/docs/src/documentation/content/xdocs/site.xml


> Remove hod contrib
> ------------------
>
>                 Key: HADOOP-7137
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7137
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7137-additional.patch, HADOOP-7137.patch, HADOOP-7137.patch
>
>
> As per vote on general@ (http://mail-archives.apache.org/mod_mbox/hadoop-general/201102.mbox/%3CAC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com%3E) I will 
> svn remove common/trunk/src/contrib/hod
> using this Jira.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15120-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 10:49:28 2011
Return-Path: <common-issues-return-15120-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5D9524AC7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 10:49:28 +0000 (UTC)
Received: (qmail 98017 invoked by uid 500); 19 May 2011 10:49:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97979 invoked by uid 500); 19 May 2011 10:49:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97968 invoked by uid 99); 19 May 2011 10:49:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 10:49:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 10:49:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8883BD04B3
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 10:48:47 +0000 (UTC)
Date: Thu, 19 May 2011 10:48:47 +0000 (UTC)
From: "ramkrishna.s.vasudevan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1738535981.26274.1305802127556.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7304) BackUpNameNode is using 100% CPU and
 not accepting any requests.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

BackUpNameNode is using 100% CPU and not accepting any requests. 
-----------------------------------------------------------------

                 Key: HADOOP-7304
                 URL: https://issues.apache.org/jira/browse/HADOOP-7304
             Project: Hadoop Common
          Issue Type: Bug
          Components: ipc
    Affects Versions: 0.21.0
            Reporter: ramkrishna.s.vasudevan
             Fix For: 0.20-append


In our environment, Backup NameNode is using 100% CPU and not accepting any calls in 3days long run.
Thread dump 
"IPC Server Responder" daemon prio=10 tid=0x00007f86c41c6800 nid=0x3b2a runnable [0x00007f86ce579000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) 
locked <0x00007f86d67e2a20> (a sun.nio.ch.Util$1) 
locked <0x00007f86d67e2a08> (a java.util.Collections$UnmodifiableSet) 
locked <0x00007f86d67e26a8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.apache.hadoop.ipc.Server$Responder.run(Server.java:501) 
Looks like same issue occurred in jetty also. http://jira.codehaus.org/browse/JETTY-937 




 



--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15121-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 10:49:31 2011
Return-Path: <common-issues-return-15121-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B23FF4B1B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 10:49:31 +0000 (UTC)
Received: (qmail 99994 invoked by uid 500); 19 May 2011 10:49:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99958 invoked by uid 500); 19 May 2011 10:49:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99918 invoked by uid 99); 19 May 2011 10:49:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 10:49:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 10:49:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A8CE2D04BC
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 10:48:47 +0000 (UTC)
Date: Thu, 19 May 2011 10:48:47 +0000 (UTC)
From: "ramkrishna.s.vasudevan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1587045213.26279.1305802127688.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1738535981.26274.1305802127556.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7304) BackUpNameNode is using 100% CPU and
 not accepting any requests.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ramkrishna.s.vasudevan updated HADOOP-7304:
-------------------------------------------

    Affects Version/s: 0.20-append

> BackUpNameNode is using 100% CPU and not accepting any requests. 
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7304
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7304
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.20-append, 0.21.0
>            Reporter: ramkrishna.s.vasudevan
>             Fix For: 0.20-append
>
>
> In our environment, Backup NameNode is using 100% CPU and not accepting any calls in 3days long run.
> Thread dump 
> "IPC Server Responder" daemon prio=10 tid=0x00007f86c41c6800 nid=0x3b2a runnable [0x00007f86ce579000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) 
> locked <0x00007f86d67e2a20> (a sun.nio.ch.Util$1) 
> locked <0x00007f86d67e2a08> (a java.util.Collections$UnmodifiableSet) 
> locked <0x00007f86d67e26a8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.hadoop.ipc.Server$Responder.run(Server.java:501) 
> Looks like same issue occurred in jetty also. http://jira.codehaus.org/browse/JETTY-937 
>  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15122-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 11:09:30 2011
Return-Path: <common-issues-return-15122-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 316864436
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 11:09:30 +0000 (UTC)
Received: (qmail 25440 invoked by uid 500); 19 May 2011 11:09:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25407 invoked by uid 500); 19 May 2011 11:09:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25399 invoked by uid 99); 19 May 2011 11:09:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 11:09:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 11:09:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 59D86D0B20
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 11:08:47 +0000 (UTC)
Date: Thu, 19 May 2011 11:08:47 +0000 (UTC)
From: "ramkrishna.s.vasudevan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <597712050.26292.1305803327364.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1738535981.26274.1305802127556.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7304) BackUpNameNode is using 100% CPU and
 not accepting any requests.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ramkrishna.s.vasudevan updated HADOOP-7304:
-------------------------------------------

    Affects Version/s:     (was: 0.20-append)
                           (was: 0.21.0)
                       0.23.0
        Fix Version/s:     (was: 0.20-append)

> BackUpNameNode is using 100% CPU and not accepting any requests. 
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7304
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7304
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.23.0
>            Reporter: ramkrishna.s.vasudevan
>
> In our environment, Backup NameNode is using 100% CPU and not accepting any calls in 3days long run.
> Thread dump 
> "IPC Server Responder" daemon prio=10 tid=0x00007f86c41c6800 nid=0x3b2a runnable [0x00007f86ce579000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) 
> locked <0x00007f86d67e2a20> (a sun.nio.ch.Util$1) 
> locked <0x00007f86d67e2a08> (a java.util.Collections$UnmodifiableSet) 
> locked <0x00007f86d67e26a8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.hadoop.ipc.Server$Responder.run(Server.java:501) 
> Looks like same issue occurred in jetty also. http://jira.codehaus.org/browse/JETTY-937 
>  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15123-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 12:11:32 2011
Return-Path: <common-issues-return-15123-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 56A794C4A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 12:11:32 +0000 (UTC)
Received: (qmail 27442 invoked by uid 500); 19 May 2011 12:11:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27354 invoked by uid 500); 19 May 2011 12:11:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27331 invoked by uid 99); 19 May 2011 12:11:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 12:11:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 12:11:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9CB7FD107B
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 12:10:47 +0000 (UTC)
Date: Thu, 19 May 2011 12:10:47 +0000 (UTC)
From: "Niels Basjes (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1008788632.26385.1305807047638.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7305) Eclipse project files are incomplete
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Eclipse project files are incomplete
------------------------------------

                 Key: HADOOP-7305
                 URL: https://issues.apache.org/jira/browse/HADOOP-7305
             Project: Hadoop Common
          Issue Type: Improvement
          Components: build
            Reporter: Niels Basjes
            Assignee: Niels Basjes
            Priority: Minor


After a fresh checkout of hadoop-common I do 'ant compile eclipse'.
I open eclipse, set ANT_HOME and build the project. 
At that point the following error appears:
{quote}
The type com.sun.javadoc.RootDoc cannot be resolved. It is indirectly referenced from required .class files	ExcludePrivateAnnotationsJDiffDoclet.java	/common/src/java/org/apache/hadoop/classification/tools	line 1	Java Problem
{quote}

The solution is to add the "tools.jar" from the JDK to the buildpath/classpath.
This should be fixed in the build.xml.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15124-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 12:13:29 2011
Return-Path: <common-issues-return-15124-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 149AD4D0E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 12:13:29 +0000 (UTC)
Received: (qmail 35184 invoked by uid 500); 19 May 2011 12:13:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35137 invoked by uid 500); 19 May 2011 12:13:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34834 invoked by uid 99); 19 May 2011 12:13:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 12:13:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 12:13:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C9295D11BC
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 12:12:47 +0000 (UTC)
Date: Thu, 19 May 2011 12:12:47 +0000 (UTC)
From: "Niels Basjes (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <922742944.26392.1305807167820.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1008788632.26385.1305807047638.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7305) Eclipse project files are incomplete
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Niels Basjes updated HADOOP-7305:
---------------------------------

    Attachment: HADOOP-7305-2011-05-19.patch

Add tools.jar to the classpath for the eclipse target in ant.

> Eclipse project files are incomplete
> ------------------------------------
>
>                 Key: HADOOP-7305
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7305
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Niels Basjes
>            Assignee: Niels Basjes
>            Priority: Minor
>         Attachments: HADOOP-7305-2011-05-19.patch
>
>
> After a fresh checkout of hadoop-common I do 'ant compile eclipse'.
> I open eclipse, set ANT_HOME and build the project. 
> At that point the following error appears:
> {quote}
> The type com.sun.javadoc.RootDoc cannot be resolved. It is indirectly referenced from required .class files	ExcludePrivateAnnotationsJDiffDoclet.java	/common/src/java/org/apache/hadoop/classification/tools	line 1	Java Problem
> {quote}
> The solution is to add the "tools.jar" from the JDK to the buildpath/classpath.
> This should be fixed in the build.xml.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15125-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 12:13:31 2011
Return-Path: <common-issues-return-15125-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 16AF54D59
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 12:13:31 +0000 (UTC)
Received: (qmail 36859 invoked by uid 500); 19 May 2011 12:13:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36827 invoked by uid 500); 19 May 2011 12:13:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36819 invoked by uid 99); 19 May 2011 12:13:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 12:13:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 12:13:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1A158D11C4
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 12:12:48 +0000 (UTC)
Date: Thu, 19 May 2011 12:12:48 +0000 (UTC)
From: "Niels Basjes (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <903659163.26400.1305807168103.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1008788632.26385.1305807047638.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7305) Eclipse project files are incomplete
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Niels Basjes updated HADOOP-7305:
---------------------------------

    Release Note: Added missing library during creation of the eclipse project files.
          Status: Patch Available  (was: Open)

> Eclipse project files are incomplete
> ------------------------------------
>
>                 Key: HADOOP-7305
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7305
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Niels Basjes
>            Assignee: Niels Basjes
>            Priority: Minor
>         Attachments: HADOOP-7305-2011-05-19.patch
>
>
> After a fresh checkout of hadoop-common I do 'ant compile eclipse'.
> I open eclipse, set ANT_HOME and build the project. 
> At that point the following error appears:
> {quote}
> The type com.sun.javadoc.RootDoc cannot be resolved. It is indirectly referenced from required .class files	ExcludePrivateAnnotationsJDiffDoclet.java	/common/src/java/org/apache/hadoop/classification/tools	line 1	Java Problem
> {quote}
> The solution is to add the "tools.jar" from the JDK to the buildpath/classpath.
> This should be fixed in the build.xml.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15126-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 12:33:30 2011
Return-Path: <common-issues-return-15126-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3984F4ADD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 12:33:30 +0000 (UTC)
Received: (qmail 66597 invoked by uid 500); 19 May 2011 12:33:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66473 invoked by uid 500); 19 May 2011 12:33:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66465 invoked by uid 99); 19 May 2011 12:33:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 12:33:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 12:33:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5817BD1848
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 12:32:47 +0000 (UTC)
Date: Thu, 19 May 2011 12:32:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1272668368.26448.1305808367357.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1008788632.26385.1305807047638.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7305) Eclipse project files are
 incomplete
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036149#comment-13036149 ] 

Hadoop QA commented on HADOOP-7305:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479760/HADOOP-7305-2011-05-19.patch
  against trunk revision 1124456.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/476//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/476//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/476//console

This message is automatically generated.

> Eclipse project files are incomplete
> ------------------------------------
>
>                 Key: HADOOP-7305
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7305
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Niels Basjes
>            Assignee: Niels Basjes
>            Priority: Minor
>         Attachments: HADOOP-7305-2011-05-19.patch
>
>
> After a fresh checkout of hadoop-common I do 'ant compile eclipse'.
> I open eclipse, set ANT_HOME and build the project. 
> At that point the following error appears:
> {quote}
> The type com.sun.javadoc.RootDoc cannot be resolved. It is indirectly referenced from required .class files	ExcludePrivateAnnotationsJDiffDoclet.java	/common/src/java/org/apache/hadoop/classification/tools	line 1	Java Problem
> {quote}
> The solution is to add the "tools.jar" from the JDK to the buildpath/classpath.
> This should be fixed in the build.xml.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15127-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 12:37:28 2011
Return-Path: <common-issues-return-15127-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 44AF34038
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 12:37:28 +0000 (UTC)
Received: (qmail 76390 invoked by uid 500); 19 May 2011 12:37:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76220 invoked by uid 500); 19 May 2011 12:37:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76212 invoked by uid 99); 19 May 2011 12:37:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 12:37:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 12:37:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6FDF5D1A1F
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 12:36:47 +0000 (UTC)
Date: Thu, 19 May 2011 12:36:47 +0000 (UTC)
From: "Niels Basjes (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <187374403.26454.1305808607454.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1008788632.26385.1305807047638.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7305) Eclipse project files are
 incomplete
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036151#comment-13036151 ] 

Niels Basjes commented on HADOOP-7305:
--------------------------------------

Regarding the "
   -1 tests included.  The patch doesn't appear to include any new or modified tests.
                       Please justify why no new tests are needed for this patch.
                       Also please list what manual steps were performed to verify this patch.
"

To test this: Do what I described in the issue and the mentioned error should not occur.

> Eclipse project files are incomplete
> ------------------------------------
>
>                 Key: HADOOP-7305
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7305
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Niels Basjes
>            Assignee: Niels Basjes
>            Priority: Minor
>         Attachments: HADOOP-7305-2011-05-19.patch
>
>
> After a fresh checkout of hadoop-common I do 'ant compile eclipse'.
> I open eclipse, set ANT_HOME and build the project. 
> At that point the following error appears:
> {quote}
> The type com.sun.javadoc.RootDoc cannot be resolved. It is indirectly referenced from required .class files	ExcludePrivateAnnotationsJDiffDoclet.java	/common/src/java/org/apache/hadoop/classification/tools	line 1	Java Problem
> {quote}
> The solution is to add the "tools.jar" from the JDK to the buildpath/classpath.
> This should be fixed in the build.xml.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15128-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 13:32:28 2011
Return-Path: <common-issues-return-15128-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 745B74E61
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 13:32:28 +0000 (UTC)
Received: (qmail 95425 invoked by uid 500); 19 May 2011 13:32:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95391 invoked by uid 500); 19 May 2011 13:32:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95383 invoked by uid 99); 19 May 2011 13:32:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 13:32:28 +0000
X-ASF-Spam-Status: No, hits=-1998.0 required=5.0
	tests=ALL_TRUSTED,FB_GET_MEDS,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 13:32:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9C9CBD12B9
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 13:31:47 +0000 (UTC)
Date: Thu, 19 May 2011 13:31:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <366104012.26546.1305811907638.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1419221698.23009.1305741287424.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7300) Configuration methods that return
 collections are inconsistent about mutability
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036179#comment-13036179 ] 

Hudson commented on HADOOP-7300:
--------------------------------

Integrated in Hadoop-Common-trunk #693 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/693/])
    HADOOP-7300. Configuration methods that return collections are inconsistent about mutability. Contributed by Todd Lipcon.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1124368
Files : 
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/conf/TestConfiguration.java
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/conf/Configuration.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/util/StringUtils.java


> Configuration methods that return collections are inconsistent about mutability
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-7300
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7300
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7300.txt
>
>
> In particular, getTrimmedStringCollection seems to return an immutable collection, whereas getStringCollection returns a mutable one.
> IMO we should always return mutable collections since these methods by definition are doing copies.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15129-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 13:32:30 2011
Return-Path: <common-issues-return-15129-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6AA934E95
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 13:32:30 +0000 (UTC)
Received: (qmail 95679 invoked by uid 500); 19 May 2011 13:32:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95650 invoked by uid 500); 19 May 2011 13:32:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95642 invoked by uid 99); 19 May 2011 13:32:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 13:32:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 13:32:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 89461D12B7
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 13:31:47 +0000 (UTC)
Date: Thu, 19 May 2011 13:31:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <826611719.26544.1305811907559.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <600939426.23052.1305741648704.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7301) FSDataInputStream should expose a
 getWrappedStream method
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036178#comment-13036178 ] 

Hudson commented on HADOOP-7301:
--------------------------------

Integrated in Hadoop-Common-trunk #693 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/693/])
    HADOOP-7301. FSDataInputStream should expose a getWrappedStream method. Contributed by Jonathan Hsieh

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1124406
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/FSDataInputStream.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/FSMainOperationsBaseTest.java


> FSDataInputStream should expose a getWrappedStream method
> ---------------------------------------------------------
>
>                 Key: HADOOP-7301
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7301
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jonathan Hsieh
>            Assignee: Jonathan Hsieh
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7301.2.patch, hadoop-7301.patch
>
>
> Ideally FSDataInputStream should expose a getWrappedStream method similarly to how FSDataOutputStream exposes a getWrappedStream method.  Exposing this is useful for verifying correctness in tests cases.  This FSDataInputStream type is the class that the o.a.h.fs.FileSystem.open call returns.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15130-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 13:32:31 2011
Return-Path: <common-issues-return-15130-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EA8744EB0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 13:32:30 +0000 (UTC)
Received: (qmail 95935 invoked by uid 500); 19 May 2011 13:32:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95884 invoked by uid 500); 19 May 2011 13:32:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95876 invoked by uid 99); 19 May 2011 13:32:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 13:32:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 13:32:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AAF30D12BB
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 13:31:47 +0000 (UTC)
Date: Thu, 19 May 2011 13:31:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2080523371.26548.1305811907696.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <957076305.12265.1297477317500.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7137) Remove hod contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036180#comment-13036180 ] 

Hudson commented on HADOOP-7137:
--------------------------------

Integrated in Hadoop-Common-trunk #693 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/693/])
    HADOOP-7137. Remove hod contrib docs. Contributed by Nigel Daley

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1124456
Files : 
* /hadoop/common/trunk/src/docs/src/documentation/content/xdocs/site.xml
* /hadoop/common/trunk/src/docs/src/documentation/content/xdocs/hod_scheduler.xml


> Remove hod contrib
> ------------------
>
>                 Key: HADOOP-7137
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7137
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7137-additional.patch, HADOOP-7137.patch, HADOOP-7137.patch
>
>
> As per vote on general@ (http://mail-archives.apache.org/mod_mbox/hadoop-general/201102.mbox/%3CAC35A7EF-1D68-4055-8D47-EDA2FCF8C2F6@mac.com%3E) I will 
> svn remove common/trunk/src/contrib/hod
> using this Jira.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15131-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 14:16:33 2011
Return-Path: <common-issues-return-15131-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 885604DAF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 14:16:31 +0000 (UTC)
Received: (qmail 29044 invoked by uid 500); 19 May 2011 14:16:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28998 invoked by uid 500); 19 May 2011 14:16:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28951 invoked by uid 99); 19 May 2011 14:16:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 14:16:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 14:16:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BE9F9D15DA
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 14:15:47 +0000 (UTC)
Date: Thu, 19 May 2011 14:15:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <25051130.26686.1305814547777.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036202#comment-13036202 ] 

Daryn Sharp commented on HADOOP-6605:
-------------------------------------

Love it.  Comments:

SunOS - why not jdk1.6.* to get latest patch version, or jdk*.*?  Using the last item in the glob expansion or search in reverse order should generally get the latest version.

Linux - Same as SunOS.  Also, why aren't /usr/java/default and /usr/lib/jvm/default-java checked first?

Overall, maybe globs aren't a great idea since it may lead to a lot more paths being added which I think is Eli's concern.  Would it make more sense to only support using the "standard" mechanisms of an OS if that OS provides one?  In which case the functionality is to make hadoop a good citizen of that OS, and not to make pseudo-intelligent guesses?

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch, hadoop-6605-2.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15132-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 14:18:31 2011
Return-Path: <common-issues-return-15132-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B5BBA477D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 14:18:31 +0000 (UTC)
Received: (qmail 33765 invoked by uid 500); 19 May 2011 14:18:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33710 invoked by uid 500); 19 May 2011 14:18:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33652 invoked by uid 99); 19 May 2011 14:18:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 14:18:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 14:18:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B3A6ED173B
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 14:17:47 +0000 (UTC)
Date: Thu, 19 May 2011 14:17:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <986514989.26699.1305814667732.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036205#comment-13036205 ] 

Daryn Sharp commented on HADOOP-6605:
-------------------------------------

Correction: Eli's concern = Allen's concern.

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch, hadoop-6605-2.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15133-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 15:31:30 2011
Return-Path: <common-issues-return-15133-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4ECFB482C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 15:31:30 +0000 (UTC)
Received: (qmail 70679 invoked by uid 500); 19 May 2011 15:31:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70648 invoked by uid 500); 19 May 2011 15:31:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70639 invoked by uid 99); 19 May 2011 15:31:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 15:31:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 15:31:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 654E5D171F
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 15:30:47 +0000 (UTC)
Date: Thu, 19 May 2011 15:30:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <447201594.26940.1305819047411.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6962) FileSystem.mkdirs(Path,
 FSPermission) should use the permission for all of the created directories
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036252#comment-13036252 ] 

Daryn Sharp commented on HADOOP-6962:
-------------------------------------

Apologies, my comments were more about the patch on the linked HDFS-1869.  The FileSystem api for creating directories is somewhat confusing.  Here's my reading of the code:

For localfs:
1. *public boolean mkdirs(Path f)* uses the umask for dirs0..N.
2. *public boolean mkdirs(Path f, FsPermission permission)* uses umask for dir0..N-1, given perms for dirN.

For hdfs:  
1. *public boolean mkdirs(Path f)* uses the parent dir's permission for dir0..N-1, 0777 for dirN.
2. *public boolean mkdirs(Path f, FsPermission permission)* uses the parent dir's permission for dir0..N-1, given permission for dirN

In both cases, *public +static+ boolean mkdirs(FileSystem fs, Path dir, FsPermission permission)* calls #1 + setPermission which has the same effect as #2, hence this change.

The linked HDFS bug creates dir0..N with the given permissions which restores how it used to work, and restores posix compliance.  However, it now differs from localfs, but localfs is in contradiction to FileSystem:

{code}
  /**
   * Make the given file and all non-existent parents into
   * directories. Has the semantics of Unix 'mkdir -p'.
   * Existence of the directory hierarchy is not an error.
   */
  public abstract boolean mkdirs(Path f, FsPermission permission
      ) throws IOException;
{code}

> FileSystem.mkdirs(Path, FSPermission) should use the permission for all of the created directories
> --------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6962
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6962
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Owen O'Malley
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-6962.patch
>
>
> Currently, FileSystem.mkdirs only applies the permissions to the last level if it was created. It should be applied to *all* levels that are created.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15134-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 16:05:29 2011
Return-Path: <common-issues-return-15134-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A88034B6C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 16:05:29 +0000 (UTC)
Received: (qmail 46167 invoked by uid 500); 19 May 2011 16:05:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46139 invoked by uid 500); 19 May 2011 16:05:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46122 invoked by uid 99); 19 May 2011 16:05:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 16:05:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 16:05:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 96A90D1367
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 16:04:47 +0000 (UTC)
Date: Thu, 19 May 2011 16:04:47 +0000 (UTC)
From: "Harsh J Chouraria (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <136728320.27026.1305821087613.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1250812071.19292.1305645587504.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Reopened] (HADOOP-7297) Error in the documentation
 regarding Checkpoint/Backup Node
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J Chouraria reopened HADOOP-7297:
---------------------------------------


Reopening since the issue of docs is valid. There are CN and BN node docs on the tagged svn rev: http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.203.0/src/docs/src/documentation/content/xdocs/hdfs_user_guide.xml

> Error in the documentation regarding Checkpoint/Backup Node
> -----------------------------------------------------------
>
>                 Key: HADOOP-7297
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7297
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.20.203.0
>            Reporter: arnaud p
>            Priority: Trivial
>
> On http://hadoop.apache.org/common/docs/r0.20.203.0/hdfs_user_guide.html#Checkpoint+Node: the command bin/hdfs namenode -checkpoint required to launch the backup/checkpoint node does not exist.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15135-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 16:05:30 2011
Return-Path: <common-issues-return-15135-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B09404BA1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 16:05:30 +0000 (UTC)
Received: (qmail 48007 invoked by uid 500); 19 May 2011 16:05:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47978 invoked by uid 500); 19 May 2011 16:05:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47970 invoked by uid 99); 19 May 2011 16:05:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 16:05:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 16:05:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A3FBAD136B
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 16:04:47 +0000 (UTC)
Date: Thu, 19 May 2011 16:04:47 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1695075500.27028.1305821087668.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036268#comment-13036268 ] 

Allen Wittenauer commented on HADOOP-6605:
------------------------------------------

FYI: I'm not removing my -1 as long as this remains in the main hadoop bin code path.

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch, hadoop-6605-2.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15136-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 16:27:31 2011
Return-Path: <common-issues-return-15136-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DC15145D2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 16:27:30 +0000 (UTC)
Received: (qmail 87340 invoked by uid 500); 19 May 2011 16:27:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87296 invoked by uid 500); 19 May 2011 16:27:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87288 invoked by uid 99); 19 May 2011 16:27:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 16:27:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 16:27:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5EA46D1B13
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 16:26:47 +0000 (UTC)
Date: Thu, 19 May 2011 16:26:47 +0000 (UTC)
From: "Harsh J Chouraria (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1182406264.27131.1305822407384.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1250812071.19292.1305645587504.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7297) Error in the documentation
 regarding Checkpoint/Backup Node
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036280#comment-13036280 ] 

Harsh J Chouraria commented on HADOOP-7297:
-------------------------------------------

(What I mean is the bad docs ought to be removed, not the inclusion of CN/BN).

> Error in the documentation regarding Checkpoint/Backup Node
> -----------------------------------------------------------
>
>                 Key: HADOOP-7297
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7297
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.20.203.0
>            Reporter: arnaud p
>            Priority: Trivial
>
> On http://hadoop.apache.org/common/docs/r0.20.203.0/hdfs_user_guide.html#Checkpoint+Node: the command bin/hdfs namenode -checkpoint required to launch the backup/checkpoint node does not exist.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15137-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 17:53:30 2011
Return-Path: <common-issues-return-15137-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CC5B347F7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 17:53:30 +0000 (UTC)
Received: (qmail 7337 invoked by uid 500); 19 May 2011 17:53:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7276 invoked by uid 500); 19 May 2011 17:53:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7232 invoked by uid 99); 19 May 2011 17:53:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 17:53:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 17:53:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id ADEDAD1B0C
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 17:52:47 +0000 (UTC)
Date: Thu, 19 May 2011 17:52:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <580585038.27366.1305827567709.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7306) Start metrics system even if config
 files are missing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Start metrics system even if config files are missing
-----------------------------------------------------

                 Key: HADOOP-7306
                 URL: https://issues.apache.org/jira/browse/HADOOP-7306
             Project: Hadoop Common
          Issue Type: Improvement
          Components: metrics
    Affects Versions: 0.23.0
            Reporter: Luke Lu
            Assignee: Luke Lu
             Fix For: 0.23.0


Per experience and discussion with HDFS-1922, it seems preferable to treat missing metrics config file as empty/default config, which is more compatible with metrics v1 behavior (the MBeans are always registered.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15138-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 18:01:29 2011
Return-Path: <common-issues-return-15138-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4FE8F4183
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 18:01:29 +0000 (UTC)
Received: (qmail 21137 invoked by uid 500); 19 May 2011 18:01:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21105 invoked by uid 500); 19 May 2011 18:01:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21097 invoked by uid 99); 19 May 2011 18:01:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 18:01:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 18:01:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7B12DD1E3D
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 18:00:48 +0000 (UTC)
Date: Thu, 19 May 2011 18:00:48 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <64213023.27398.1305828048299.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <580585038.27366.1305827567709.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7306) Start metrics system even if config
 files are missing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-7306:
----------------------------

    Attachment: hadoop-7306-missing-conf-v1.patch

> Start metrics system even if config files are missing
> -----------------------------------------------------
>
>                 Key: HADOOP-7306
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7306
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: metrics
>    Affects Versions: 0.23.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7306-missing-conf-v1.patch
>
>
> Per experience and discussion with HDFS-1922, it seems preferable to treat missing metrics config file as empty/default config, which is more compatible with metrics v1 behavior (the MBeans are always registered.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15139-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 18:03:32 2011
Return-Path: <common-issues-return-15139-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8D4664BE7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 18:03:32 +0000 (UTC)
Received: (qmail 25732 invoked by uid 500); 19 May 2011 18:03:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25635 invoked by uid 500); 19 May 2011 18:03:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25581 invoked by uid 99); 19 May 2011 18:03:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 18:03:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 18:03:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B595ED1FA5
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 18:02:48 +0000 (UTC)
Date: Thu, 19 May 2011 18:02:48 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2113839943.27420.1305828168740.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <580585038.27366.1305827567709.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7306) Start metrics system even if config
 files are missing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-7306:
----------------------------

    Status: Patch Available  (was: Open)

patch v1 also fix a minor bug for metrics system shutdown in mini-cluster mode.

> Start metrics system even if config files are missing
> -----------------------------------------------------
>
>                 Key: HADOOP-7306
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7306
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: metrics
>    Affects Versions: 0.23.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7306-missing-conf-v1.patch
>
>
> Per experience and discussion with HDFS-1922, it seems preferable to treat missing metrics config file as empty/default config, which is more compatible with metrics v1 behavior (the MBeans are always registered.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15140-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 18:33:28 2011
Return-Path: <common-issues-return-15140-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A96A84139
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 18:33:28 +0000 (UTC)
Received: (qmail 69395 invoked by uid 500); 19 May 2011 18:33:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69363 invoked by uid 500); 19 May 2011 18:33:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69354 invoked by uid 99); 19 May 2011 18:33:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 18:33:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 18:33:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D854BD1DE5
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 18:32:47 +0000 (UTC)
Date: Thu, 19 May 2011 18:32:47 +0000 (UTC)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <87037440.27579.1305829967882.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6832) Provide a web server plugin that
 uses a static user for the web UI
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-6832:
----------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

+1 on the merged {{h-6832.patch}}.

I committed this. Thanks Owen and Todd!

Using {{default.web.user}} would be more descriptive, but at least searching for "hadoop dr.who" yields relevant, whinging results.

> Provide a web server plugin that uses a static user for the web UI
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6832
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6832
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: h-6382.patch, h-6832.patch, h-6832.patch, h-6832.patch, hadoop-6832.txt, static-web-user.patch
>
>
> We need a simple plugin that uses a static user for clusters with security that don't want to authenticate users on the web UI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15141-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 18:37:28 2011
Return-Path: <common-issues-return-15141-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 613BF4645
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 18:37:28 +0000 (UTC)
Received: (qmail 73971 invoked by uid 500); 19 May 2011 18:37:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73938 invoked by uid 500); 19 May 2011 18:37:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73930 invoked by uid 99); 19 May 2011 18:37:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 18:37:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 18:37:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8AAD9D1F64
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 18:36:47 +0000 (UTC)
Date: Thu, 19 May 2011 18:36:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <273487876.27592.1305830207564.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6832) Provide a web server plugin that
 uses a static user for the web UI
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036356#comment-13036356 ] 

Todd Lipcon commented on HADOOP-6832:
-------------------------------------

I am still strongly against "dr who". I understand it's some kind of tradition to put Dr Who references throughout our code base. Unfortunately the users aren't in on the joke. I will open another JIRA to remove these references.

> Provide a web server plugin that uses a static user for the web UI
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6832
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6832
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: h-6382.patch, h-6832.patch, h-6832.patch, h-6832.patch, hadoop-6832.txt, static-web-user.patch
>
>
> We need a simple plugin that uses a static user for clusters with security that don't want to authenticate users on the web UI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15142-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 18:37:32 2011
Return-Path: <common-issues-return-15142-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1880B4674
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 18:37:32 +0000 (UTC)
Received: (qmail 74809 invoked by uid 500); 19 May 2011 18:37:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74747 invoked by uid 500); 19 May 2011 18:37:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74657 invoked by uid 99); 19 May 2011 18:37:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 18:37:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 18:37:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B11AFD1F6A
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 18:36:47 +0000 (UTC)
Date: Thu, 19 May 2011 18:36:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <889607684.27597.1305830207722.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7307) Remove Dr Who references
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Remove Dr Who references
------------------------

                 Key: HADOOP-7307
                 URL: https://issues.apache.org/jira/browse/HADOOP-7307
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.22.0
            Reporter: Todd Lipcon
             Fix For: 0.22.0


I've had feedback from a number of different operators that the "dr who" references that occasionally come out of Hadoop are annoying and confusing. We should replace them with more descriptive error messages, usernames, etc.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15143-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 18:41:31 2011
Return-Path: <common-issues-return-15143-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4B2834B00
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 18:41:31 +0000 (UTC)
Received: (qmail 87751 invoked by uid 500); 19 May 2011 18:41:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87697 invoked by uid 500); 19 May 2011 18:41:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87680 invoked by uid 99); 19 May 2011 18:41:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 18:41:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 18:41:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 20DBDD1186
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 18:40:48 +0000 (UTC)
Date: Thu, 19 May 2011 18:40:48 +0000 (UTC)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1633474446.27609.1305830448131.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6832) Provide a web server plugin that
 uses a static user for the web UI
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036358#comment-13036358 ] 

Chris Douglas commented on HADOOP-6832:
---------------------------------------

*nod* Discussing that in a separate thread makes sense.

> Provide a web server plugin that uses a static user for the web UI
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6832
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6832
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: h-6382.patch, h-6832.patch, h-6832.patch, h-6832.patch, hadoop-6832.txt, static-web-user.patch
>
>
> We need a simple plugin that uses a static user for clusters with security that don't want to authenticate users on the web UI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15144-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 18:45:28 2011
Return-Path: <common-issues-return-15144-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 448504EB2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 18:45:28 +0000 (UTC)
Received: (qmail 92590 invoked by uid 500); 19 May 2011 18:45:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92559 invoked by uid 500); 19 May 2011 18:45:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92551 invoked by uid 99); 19 May 2011 18:45:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 18:45:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 18:45:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6FCBAD132A
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 18:44:47 +0000 (UTC)
Date: Thu, 19 May 2011 18:44:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <202727988.27625.1305830687454.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1008788632.26385.1305807047638.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7305) Eclipse project files are
 incomplete
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036361#comment-13036361 ] 

Todd Lipcon commented on HADOOP-7305:
-------------------------------------

+1

> Eclipse project files are incomplete
> ------------------------------------
>
>                 Key: HADOOP-7305
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7305
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Niels Basjes
>            Assignee: Niels Basjes
>            Priority: Minor
>         Attachments: HADOOP-7305-2011-05-19.patch
>
>
> After a fresh checkout of hadoop-common I do 'ant compile eclipse'.
> I open eclipse, set ANT_HOME and build the project. 
> At that point the following error appears:
> {quote}
> The type com.sun.javadoc.RootDoc cannot be resolved. It is indirectly referenced from required .class files	ExcludePrivateAnnotationsJDiffDoclet.java	/common/src/java/org/apache/hadoop/classification/tools	line 1	Java Problem
> {quote}
> The solution is to add the "tools.jar" from the JDK to the buildpath/classpath.
> This should be fixed in the build.xml.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15145-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 18:49:33 2011
Return-Path: <common-issues-return-15145-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BFFBB42B9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 18:49:33 +0000 (UTC)
Received: (qmail 7953 invoked by uid 500); 19 May 2011 18:49:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7924 invoked by uid 500); 19 May 2011 18:49:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7916 invoked by uid 99); 19 May 2011 18:49:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 18:49:33 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 18:49:31 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CE903D14E6
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 18:48:50 +0000 (UTC)
Date: Thu, 19 May 2011 18:48:50 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1813524902.27636.1305830930842.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1008788632.26385.1305807047638.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7305) Eclipse project files are incomplete
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7305:
--------------------------------

       Resolution: Fixed
    Fix Version/s: 0.22.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

Verified that "ant eclipse" generated a .classpath file with the correct tools.jar. Committed to 0.22 and trunk. Thanks, Niels!

> Eclipse project files are incomplete
> ------------------------------------
>
>                 Key: HADOOP-7305
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7305
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Niels Basjes
>            Assignee: Niels Basjes
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7305-2011-05-19.patch
>
>
> After a fresh checkout of hadoop-common I do 'ant compile eclipse'.
> I open eclipse, set ANT_HOME and build the project. 
> At that point the following error appears:
> {quote}
> The type com.sun.javadoc.RootDoc cannot be resolved. It is indirectly referenced from required .class files	ExcludePrivateAnnotationsJDiffDoclet.java	/common/src/java/org/apache/hadoop/classification/tools	line 1	Java Problem
> {quote}
> The solution is to add the "tools.jar" from the JDK to the buildpath/classpath.
> This should be fixed in the build.xml.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15146-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 18:51:30 2011
Return-Path: <common-issues-return-15146-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D9F7D4332
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 18:51:30 +0000 (UTC)
Received: (qmail 10538 invoked by uid 500); 19 May 2011 18:51:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10509 invoked by uid 500); 19 May 2011 18:51:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10501 invoked by uid 99); 19 May 2011 18:51:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 18:51:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 18:51:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1090CD15E9
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 18:50:48 +0000 (UTC)
Date: Thu, 19 May 2011 18:50:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <420986534.27640.1305831048064.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <580585038.27366.1305827567709.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7306) Start metrics system even if
 config files are missing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036365#comment-13036365 ] 

Todd Lipcon commented on HADOOP-7306:
-------------------------------------

+1 pending Hudson results. Thanks, Luke

> Start metrics system even if config files are missing
> -----------------------------------------------------
>
>                 Key: HADOOP-7306
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7306
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: metrics
>    Affects Versions: 0.23.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7306-missing-conf-v1.patch
>
>
> Per experience and discussion with HDFS-1922, it seems preferable to treat missing metrics config file as empty/default config, which is more compatible with metrics v1 behavior (the MBeans are always registered.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15147-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 18:55:28 2011
Return-Path: <common-issues-return-15147-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3182A43E7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 18:55:28 +0000 (UTC)
Received: (qmail 15877 invoked by uid 500); 19 May 2011 18:55:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15839 invoked by uid 500); 19 May 2011 18:55:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15831 invoked by uid 99); 19 May 2011 18:55:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 18:55:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 18:55:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 818E6D1850
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 18:54:47 +0000 (UTC)
Date: Thu, 19 May 2011 18:54:47 +0000 (UTC)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <318493213.27658.1305831287527.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <889607684.27597.1305830207722.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7307) Remove Dr Who references
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036368#comment-13036368 ] 

Jakob Homan commented on HADOOP-7307:
-------------------------------------

I don't think it's worth removing them, although correcting them would be a good idea.  The Doctor is his name, not his title.  

> Remove Dr Who references
> ------------------------
>
>                 Key: HADOOP-7307
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7307
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>             Fix For: 0.22.0
>
>
> I've had feedback from a number of different operators that the "dr who" references that occasionally come out of Hadoop are annoying and confusing. We should replace them with more descriptive error messages, usernames, etc.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15148-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 19:30:29 2011
Return-Path: <common-issues-return-15148-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DC8D54D18
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 19:30:28 +0000 (UTC)
Received: (qmail 79416 invoked by uid 500); 19 May 2011 19:30:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79388 invoked by uid 500); 19 May 2011 19:30:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79380 invoked by uid 99); 19 May 2011 19:30:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 19:30:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 19:30:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E32C8D18AC
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 19:29:47 +0000 (UTC)
Date: Thu, 19 May 2011 19:29:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1995734346.27798.1305833387927.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1008788632.26385.1305807047638.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7305) Eclipse project files are
 incomplete
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036394#comment-13036394 ] 

Hudson commented on HADOOP-7305:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #610 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/610/])
    HADOOP-7305. Eclipse project classpath should include tools.jar from JDK. Contributed by Niels Basjes.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1125051
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/build.xml


> Eclipse project files are incomplete
> ------------------------------------
>
>                 Key: HADOOP-7305
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7305
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Niels Basjes
>            Assignee: Niels Basjes
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7305-2011-05-19.patch
>
>
> After a fresh checkout of hadoop-common I do 'ant compile eclipse'.
> I open eclipse, set ANT_HOME and build the project. 
> At that point the following error appears:
> {quote}
> The type com.sun.javadoc.RootDoc cannot be resolved. It is indirectly referenced from required .class files	ExcludePrivateAnnotationsJDiffDoclet.java	/common/src/java/org/apache/hadoop/classification/tools	line 1	Java Problem
> {quote}
> The solution is to add the "tools.jar" from the JDK to the buildpath/classpath.
> This should be fixed in the build.xml.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15149-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 19:30:30 2011
Return-Path: <common-issues-return-15149-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BDC6F4D67
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 19:30:30 +0000 (UTC)
Received: (qmail 80126 invoked by uid 500); 19 May 2011 19:30:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80094 invoked by uid 500); 19 May 2011 19:30:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80086 invoked by uid 99); 19 May 2011 19:30:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 19:30:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 19:30:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id ADEBED18A7
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 19:29:47 +0000 (UTC)
Date: Thu, 19 May 2011 19:29:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1727244861.27794.1305833387709.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6832) Provide a web server plugin that
 uses a static user for the web UI
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036393#comment-13036393 ] 

Hudson commented on HADOOP-6832:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #610 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/610/])
    HADOOP-6832. Add an authentication plugin using a configurable static user
for the web UI. Contributed by Owen O'Malley and Todd Lipcon

cdouglas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1125043
Files : 
* /hadoop/common/trunk/src/java/org/apache/hadoop/http/lib
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/http/lib/package.html
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/http/lib
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/http/lib/TestStaticUserWebFilter.java
* /hadoop/common/trunk/src/java/core-default.xml
* /hadoop/common/trunk/src/java/org/apache/hadoop/http/lib/StaticUserWebFilter.java


> Provide a web server plugin that uses a static user for the web UI
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6832
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6832
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: h-6382.patch, h-6832.patch, h-6832.patch, h-6832.patch, hadoop-6832.txt, static-web-user.patch
>
>
> We need a simple plugin that uses a static user for clusters with security that don't want to authenticate users on the web UI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15150-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 19:40:28 2011
Return-Path: <common-issues-return-15150-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6A56C4389
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 19:40:28 +0000 (UTC)
Received: (qmail 88956 invoked by uid 500); 19 May 2011 19:40:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88908 invoked by uid 500); 19 May 2011 19:40:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88892 invoked by uid 99); 19 May 2011 19:40:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 19:40:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 19:40:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 668F0D1C69
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 19:39:47 +0000 (UTC)
Date: Thu, 19 May 2011 19:39:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1035186083.27823.1305833987416.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036401#comment-13036401 ] 

Todd Lipcon commented on HADOOP-7284:
-------------------------------------

Hi Sanjay. I think adding a configuration for viewfs for the home directory pattern would be the best route. eg a C-style format string like "/user/%s/" could be used.

If you want to commit this as is to fix the bug and do that in a separate JIRA, that's fine. +1 or happy to re-review if you want to do the config option in this one.

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15151-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 20:05:28 2011
Return-Path: <common-issues-return-15151-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 472074929
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 20:05:28 +0000 (UTC)
Received: (qmail 28746 invoked by uid 500); 19 May 2011 20:05:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28716 invoked by uid 500); 19 May 2011 20:05:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28704 invoked by uid 99); 19 May 2011 20:05:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 20:05:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 20:05:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 611D2D1728
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 20:04:47 +0000 (UTC)
Date: Thu, 19 May 2011 20:04:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2007932227.27902.1305835487394.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7269) S3 Native should allow
 customizable file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036412#comment-13036412 ] 

Todd Lipcon commented on HADOOP-7269:
-------------------------------------

hmm, Suresh, are you looking at HADOOP-7269-S3-metadata-003.diff? I don't see changes to FileSystem.java

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: HADOOP-7269-S3-metadata-001.diff, HADOOP-7269-S3-metadata-002.diff, HADOOP-7269-S3-metadata-003.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15152-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 20:23:28 2011
Return-Path: <common-issues-return-15152-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 099C14756
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 20:23:28 +0000 (UTC)
Received: (qmail 57882 invoked by uid 500); 19 May 2011 20:23:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57858 invoked by uid 500); 19 May 2011 20:23:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57850 invoked by uid 99); 19 May 2011 20:23:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 20:23:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 20:23:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 61E5DD1005
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 20:22:47 +0000 (UTC)
Date: Thu, 19 May 2011 20:22:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <665646418.27971.1305836567397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7308) Remove unused TaskLogAppender
 configurations from log4j.properties
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Remove unused TaskLogAppender configurations from log4j.properties
------------------------------------------------------------------

                 Key: HADOOP-7308
                 URL: https://issues.apache.org/jira/browse/HADOOP-7308
             Project: Hadoop Common
          Issue Type: Improvement
          Components: conf
    Affects Versions: 0.22.0
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
             Fix For: 0.22.0
         Attachments: hadoop-7308.txt

MAPREDUCE-2372 improved TaskLogAppender to no longer need as much "wiring" in log4j.properties. There are also some old properties in there that are no longer used (eg logsRetainHours and noKeepSplits).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15153-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 20:23:30 2011
Return-Path: <common-issues-return-15153-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 66563476C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 20:23:30 +0000 (UTC)
Received: (qmail 58137 invoked by uid 500); 19 May 2011 20:23:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58107 invoked by uid 500); 19 May 2011 20:23:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58099 invoked by uid 99); 19 May 2011 20:23:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 20:23:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 20:23:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 840D1D1008
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 20:22:47 +0000 (UTC)
Date: Thu, 19 May 2011 20:22:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1138455782.27974.1305836567537.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <665646418.27971.1305836567397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7308) Remove unused TaskLogAppender
 configurations from log4j.properties
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7308:
--------------------------------

    Attachment: hadoop-7308.txt

> Remove unused TaskLogAppender configurations from log4j.properties
> ------------------------------------------------------------------
>
>                 Key: HADOOP-7308
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7308
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7308.txt
>
>
> MAPREDUCE-2372 improved TaskLogAppender to no longer need as much "wiring" in log4j.properties. There are also some old properties in there that are no longer used (eg logsRetainHours and noKeepSplits).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15154-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 20:23:32 2011
Return-Path: <common-issues-return-15154-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 224774796
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 20:23:32 +0000 (UTC)
Received: (qmail 58544 invoked by uid 500); 19 May 2011 20:23:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58482 invoked by uid 500); 19 May 2011 20:23:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58336 invoked by uid 99); 19 May 2011 20:23:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 20:23:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 20:23:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9A108D100A
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 20:22:47 +0000 (UTC)
Date: Thu, 19 May 2011 20:22:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1097799572.27976.1305836567627.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <665646418.27971.1305836567397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7308) Remove unused TaskLogAppender
 configurations from log4j.properties
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7308:
--------------------------------

    Status: Patch Available  (was: Open)

> Remove unused TaskLogAppender configurations from log4j.properties
> ------------------------------------------------------------------
>
>                 Key: HADOOP-7308
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7308
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7308.txt
>
>
> MAPREDUCE-2372 improved TaskLogAppender to no longer need as much "wiring" in log4j.properties. There are also some old properties in there that are no longer used (eg logsRetainHours and noKeepSplits).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15155-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 20:27:29 2011
Return-Path: <common-issues-return-15155-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 309B44AB5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 20:27:29 +0000 (UTC)
Received: (qmail 67904 invoked by uid 500); 19 May 2011 20:27:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67871 invoked by uid 500); 19 May 2011 20:27:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67863 invoked by uid 99); 19 May 2011 20:27:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 20:27:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 20:27:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 42EBED12F7
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 20:26:48 +0000 (UTC)
Date: Thu, 19 May 2011 20:26:48 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1418067763.28013.1305836808270.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036432#comment-13036432 ] 

Sanjay Radia commented on HADOOP-7284:
--------------------------------------

Thanks. I will file a separate jira because it is not clear to me if the config should be for only viewfs or a more general config variable that applies to all file systems. 
Further, Doug and I debated this on HDFS-593; so i misspoke that we had added SS config - the patch did not go through since Doug didn't think it was important enough, although he agreed that it wasn't a client side property.

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15156-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 20:27:32 2011
Return-Path: <common-issues-return-15156-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CEA654B05
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 20:27:32 +0000 (UTC)
Received: (qmail 68678 invoked by uid 500); 19 May 2011 20:27:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68624 invoked by uid 500); 19 May 2011 20:27:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68507 invoked by uid 99); 19 May 2011 20:27:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 20:27:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 20:27:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 58FBFD12FC
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 20:26:48 +0000 (UTC)
Date: Thu, 19 May 2011 20:26:48 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1169466475.28015.1305836808361.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7309) improve trademark symbol usage and
 add trademark footer
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

improve trademark symbol usage and add trademark footer
-------------------------------------------------------

                 Key: HADOOP-7309
                 URL: https://issues.apache.org/jira/browse/HADOOP-7309
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Owen O'Malley


We only need to mention trademarks on the first usage, add a footer specifying the trademarks, and update the pdf files.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15157-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 20:27:33 2011
Return-Path: <common-issues-return-15157-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 213684B16
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 20:27:33 +0000 (UTC)
Received: (qmail 68803 invoked by uid 500); 19 May 2011 20:27:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68745 invoked by uid 500); 19 May 2011 20:27:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68595 invoked by uid 99); 19 May 2011 20:27:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 20:27:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 20:27:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6D12AD12FF
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 20:26:48 +0000 (UTC)
Date: Thu, 19 May 2011 20:26:48 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1371522151.28018.1305836808443.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1169466475.28015.1305836808361.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7309) improve trademark symbol usage and
 add trademark footer
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley reassigned HADOOP-7309:
-------------------------------------

    Assignee: Owen O'Malley

> improve trademark symbol usage and add trademark footer
> -------------------------------------------------------
>
>                 Key: HADOOP-7309
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7309
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>
> We only need to mention trademarks on the first usage, add a footer specifying the trademarks, and update the pdf files.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15158-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 20:29:28 2011
Return-Path: <common-issues-return-15158-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4F3DC4C45
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 20:29:28 +0000 (UTC)
Received: (qmail 71538 invoked by uid 500); 19 May 2011 20:29:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71506 invoked by uid 500); 19 May 2011 20:29:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71492 invoked by uid 99); 19 May 2011 20:29:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 20:29:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 20:29:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5CE9ED1409
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 20:28:47 +0000 (UTC)
Date: Thu, 19 May 2011 20:28:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <832964438.28020.1305836927377.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1169466475.28015.1305836808361.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7309) improve trademark symbol usage and
 add trademark footer
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7309:
----------------------------------

    Attachment: site.patch

> improve trademark symbol usage and add trademark footer
> -------------------------------------------------------
>
>                 Key: HADOOP-7309
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7309
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: site.patch
>
>
> We only need to mention trademarks on the first usage, add a footer specifying the trademarks, and update the pdf files.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15159-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 20:33:31 2011
Return-Path: <common-issues-return-15159-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E72134FAD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 20:33:30 +0000 (UTC)
Received: (qmail 83534 invoked by uid 500); 19 May 2011 20:33:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83471 invoked by uid 500); 19 May 2011 20:33:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83450 invoked by uid 99); 19 May 2011 20:33:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 20:33:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 20:33:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8E9E1D1637
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 20:32:47 +0000 (UTC)
Date: Thu, 19 May 2011 20:32:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1566570321.28035.1305837167581.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <187742091.8722.1305246647362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7287) Configuration deprecation
 mechanism doesn't work properly for GenericOptionsParser/Tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036435#comment-13036435 ] 

Aaron T. Myers commented on HADOOP-7287:
----------------------------------------

Just finished running the full test suite. All of the M/R tests passed. The following HDFS tests failed, all of which are known to be failing or flaky.

org.apache.hadoop.tools.TestJMXGet
org.apache.hadoop.hdfs.TestFileConcurrentReader
org.apache.hadoop.hdfs.TestDFSStorageStateRecovery

> Configuration deprecation mechanism doesn't work properly for GenericOptionsParser/Tools
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7287
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7287
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Aaron T. Myers
>            Priority: Blocker
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: hadoop-7287-still-broken.txt, hadoop-7287-testcase.txt, hadoop-7287-trunk.0.patch, hadoop-7287-trunk.1.patch
>
>
> For example, you can't use -D options on the "hadoop fs" command line in order to specify the deprecated names of configuration options. The issue is that the ordering is:
> - JVM starts
> - GenericOptionsParser creates a Configuration object and calls set() for each of the options specified on command line
> - DistributedFileSystem or other class eventually instantiates HdfsConfiguration which adds the deprecations
> - Some class calls conf.get("new key") and sees the default instead of the version set on the command line

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15160-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 20:41:30 2011
Return-Path: <common-issues-return-15160-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8172447A2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 20:41:30 +0000 (UTC)
Received: (qmail 99597 invoked by uid 500); 19 May 2011 20:41:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99569 invoked by uid 500); 19 May 2011 20:41:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99560 invoked by uid 99); 19 May 2011 20:41:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 20:41:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 20:41:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9A94ED18EF
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 20:40:47 +0000 (UTC)
Date: Thu, 19 May 2011 20:40:47 +0000 (UTC)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2103866252.28052.1305837647629.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1169466475.28015.1305836808361.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7309) improve trademark symbol usage and
 add trademark footer
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036440#comment-13036440 ] 

Chris Douglas commented on HADOOP-7309:
---------------------------------------

+1

Built on my box, verified footer generated in both html and pdf docs.

> improve trademark symbol usage and add trademark footer
> -------------------------------------------------------
>
>                 Key: HADOOP-7309
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7309
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: site.patch
>
>
> We only need to mention trademarks on the first usage, add a footer specifying the trademarks, and update the pdf files.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15161-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 21:02:30 2011
Return-Path: <common-issues-return-15161-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6360940D2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 21:02:30 +0000 (UTC)
Received: (qmail 33255 invoked by uid 500); 19 May 2011 21:02:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33176 invoked by uid 500); 19 May 2011 21:02:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33038 invoked by uid 99); 19 May 2011 21:02:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:02:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:02:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C685FD1333
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 21:01:47 +0000 (UTC)
Date: Thu, 19 May 2011 21:01:47 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <22523821.28152.1305838907809.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1105491925.1375.1305065387432.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7272) Remove unnecessary security related
 info logs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-7272:
------------------------------------

    Attachment: HADOOP-7272.rel205.patch

Patch for 0.20.security branch - to become part of 0.20.205 release.

> Remove unnecessary security related info logs
> ---------------------------------------------
>
>                 Key: HADOOP-7272
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7272
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>    Affects Versions: 0.23.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7272.patch, HADOOP-7272.rel205.patch
>
>
> Two info logs are printed when connection to RPC server is established, is not necessary. On a production cluster, these log lines made up of close to 50% of lines in the namenode log. I propose changing them into debug logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15162-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 21:02:32 2011
Return-Path: <common-issues-return-15162-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A132C411A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 21:02:32 +0000 (UTC)
Received: (qmail 33676 invoked by uid 500); 19 May 2011 21:02:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33315 invoked by uid 500); 19 May 2011 21:02:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33134 invoked by uid 99); 19 May 2011 21:02:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:02:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:02:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EC696D1352
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 21:01:47 +0000 (UTC)
Date: Thu, 19 May 2011 21:01:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <51041940.28155.1305838907965.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7310) Trash location needs to be revisited
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Trash location needs to be revisited
------------------------------------

                 Key: HADOOP-7310
                 URL: https://issues.apache.org/jira/browse/HADOOP-7310
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Sanjay Radia
            Assignee: Sanjay Radia
             Fix For: 0.23.0




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15163-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 21:04:30 2011
Return-Path: <common-issues-return-15163-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8948C4AD6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 21:04:30 +0000 (UTC)
Received: (qmail 38303 invoked by uid 500); 19 May 2011 21:04:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38272 invoked by uid 500); 19 May 2011 21:04:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38264 invoked by uid 99); 19 May 2011 21:04:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:04:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:04:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A53B6D14B3
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 21:03:47 +0000 (UTC)
Date: Thu, 19 May 2011 21:03:47 +0000 (UTC)
From: "Nicholas Telford (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1061413532.28162.1305839027673.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7269) S3 Native should allow
 customizable file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036458#comment-13036458 ] 

Nicholas Telford commented on HADOOP-7269:
------------------------------------------

I double checked the patch - there aren't any changes to FileSystem.java

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: HADOOP-7269-S3-metadata-001.diff, HADOOP-7269-S3-metadata-002.diff, HADOOP-7269-S3-metadata-003.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15164-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 21:15:31 2011
Return-Path: <common-issues-return-15164-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DF6F64512
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 21:15:31 +0000 (UTC)
Received: (qmail 59993 invoked by uid 500); 19 May 2011 21:15:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59958 invoked by uid 500); 19 May 2011 21:15:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59950 invoked by uid 99); 19 May 2011 21:15:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:15:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:15:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4FF59D1B04
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 21:14:48 +0000 (UTC)
Date: Thu, 19 May 2011 21:14:48 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <387003054.28213.1305839688324.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1169466475.28015.1305836808361.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7309) improve trademark symbol usage and
 add trademark footer
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley resolved HADOOP-7309.
-----------------------------------

    Resolution: Fixed

committed.

> improve trademark symbol usage and add trademark footer
> -------------------------------------------------------
>
>                 Key: HADOOP-7309
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7309
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: site.patch
>
>
> We only need to mention trademarks on the first usage, add a footer specifying the trademarks, and update the pdf files.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15165-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 21:15:36 2011
Return-Path: <common-issues-return-15165-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 87B8F4578
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 21:15:34 +0000 (UTC)
Received: (qmail 60543 invoked by uid 500); 19 May 2011 21:15:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60487 invoked by uid 500); 19 May 2011 21:15:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60433 invoked by uid 99); 19 May 2011 21:15:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:15:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:15:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 84840D1B0E
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 21:14:48 +0000 (UTC)
Date: Thu, 19 May 2011 21:14:48 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1352898241.28220.1305839688539.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1169466475.28015.1305836808361.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Closed] (HADOOP-7309) improve trademark symbol usage and
 add trademark footer
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley closed HADOOP-7309.
---------------------------------


> improve trademark symbol usage and add trademark footer
> -------------------------------------------------------
>
>                 Key: HADOOP-7309
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7309
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: site.patch
>
>
> We only need to mention trademarks on the first usage, add a footer specifying the trademarks, and update the pdf files.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15166-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 21:19:28 2011
Return-Path: <common-issues-return-15166-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6FB68478D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 21:19:28 +0000 (UTC)
Received: (qmail 74571 invoked by uid 500); 19 May 2011 21:19:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74536 invoked by uid 500); 19 May 2011 21:19:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74528 invoked by uid 99); 19 May 2011 21:19:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:19:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:19:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 89C47D1DDD
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 21:18:47 +0000 (UTC)
Date: Thu, 19 May 2011 21:18:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1321633523.28241.1305839927560.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <51041940.28155.1305838907965.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7310) Trash location needs to be
 revisited
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036475#comment-13036475 ] 

Sanjay Radia commented on HADOOP-7310:
--------------------------------------

Trash is located at /user/<userName>/.Trash
Its location needs to be revisted:
# Does not work well with Quotas. For example, /user/joe and /project/xx may have quotas. When files are deleted from /project/xx they move to /user/joe/.Trash which causes problems since the user quota is usually much less than that of project-xx.
# If customer has a special  data hdfs cluster where there are no home directories, then one is forced to have home directories on this data cluster just to accommodate Trash; this  can be confusing or may be used as a way get around quotas. 
# With federation, one is likely to have a NN for user accounts and other NNs for /projects, /data  etc. The issues listed for  point 2 apply here.


I propose two solutions:
* A.  Put trash in /Trash/<userName>; generally no quotas will be configured for /Trash since it is cleaned up periodically. This addresses all the three issues raised above. This solution however breaks compatibility. Note Trash applies ONLY to rm's performed via the shell's -rm command.
* B. The trash location is configured as SS configuration variable. 

I prefer solution A, since it is one less configuration variable. Do folks see *real* compatibility problem here?

> Trash location needs to be revisited
> ------------------------------------
>
>                 Key: HADOOP-7310
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7310
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15167-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 21:25:30 2011
Return-Path: <common-issues-return-15167-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D1F7F4D4D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 21:25:30 +0000 (UTC)
Received: (qmail 85927 invoked by uid 500); 19 May 2011 21:25:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85894 invoked by uid 500); 19 May 2011 21:25:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85881 invoked by uid 99); 19 May 2011 21:25:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:25:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:25:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7B6DBD11D0
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 21:24:48 +0000 (UTC)
Date: Thu, 19 May 2011 21:24:48 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2130487242.28272.1305840288502.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036480#comment-13036480 ] 

Eli Collins commented on HADOOP-6605:
-------------------------------------

bq. FYI: I'm not removing my -1 as long as this remains in the main hadoop bin code path.

Allen - what is your technical objection? There is not a performance impact to hadoop-config.sh, and even if there were, hadoop-config.sh is not a measurable part of hadoop command execution.

{noformat}
common1 (trunk)$ export JAVA_HOME=/home/eli/toolchain/jdk1.6.0_24-x64
common1 (trunk)$ time for i in {1..10}; do . bin/hadoop-config.sh ; done

real	0m0.817s
user	0m0.500s
sys	0m0.120s
common1 (trunk)$ unset JAVA_HOME
common1 (trunk)$ time for i in {1..10}; do . bin/hadoop-config.sh ; done

real	0m0.806s
user	0m0.550s
sys	0m0.060s
{noformat}

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch, hadoop-6605-2.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15168-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 21:25:34 2011
Return-Path: <common-issues-return-15168-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D296F4DD5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 21:25:34 +0000 (UTC)
Received: (qmail 88350 invoked by uid 500); 19 May 2011 21:25:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88314 invoked by uid 500); 19 May 2011 21:25:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88279 invoked by uid 99); 19 May 2011 21:25:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:25:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:25:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BEABAD11DA
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 21:24:48 +0000 (UTC)
Date: Thu, 19 May 2011 21:24:48 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <367985444.28281.1305840288777.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1105491925.1375.1305065387432.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7272) Remove unnecessary security
 related info logs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036483#comment-13036483 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7272:
------------------------------------------------

LOG.info should be LOG.debug in the 205 patch.


> Remove unnecessary security related info logs
> ---------------------------------------------
>
>                 Key: HADOOP-7272
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7272
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>    Affects Versions: 0.23.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7272.patch, HADOOP-7272.rel205.patch
>
>
> Two info logs are printed when connection to RPC server is established, is not necessary. On a production cluster, these log lines made up of close to 50% of lines in the namenode log. I propose changing them into debug logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15169-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 21:27:29 2011
Return-Path: <common-issues-return-15169-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1EA734E9C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 21:27:29 +0000 (UTC)
Received: (qmail 94258 invoked by uid 500); 19 May 2011 21:27:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94198 invoked by uid 500); 19 May 2011 21:27:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94101 invoked by uid 99); 19 May 2011 21:27:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:27:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:27:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DC507D1356
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 21:26:47 +0000 (UTC)
Date: Thu, 19 May 2011 21:26:47 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <349233579.28291.1305840407899.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1105491925.1375.1305065387432.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7272) Remove unnecessary security related
 info logs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-7272:
------------------------------------

    Attachment: HADOOP-7272.rel205.patch

New patch addresses the comment.

> Remove unnecessary security related info logs
> ---------------------------------------------
>
>                 Key: HADOOP-7272
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7272
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>    Affects Versions: 0.23.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7272.patch, HADOOP-7272.rel205.patch, HADOOP-7272.rel205.patch
>
>
> Two info logs are printed when connection to RPC server is established, is not necessary. On a production cluster, these log lines made up of close to 50% of lines in the namenode log. I propose changing them into debug logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15170-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 21:27:31 2011
Return-Path: <common-issues-return-15170-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3982B4EC2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 21:27:31 +0000 (UTC)
Received: (qmail 94963 invoked by uid 500); 19 May 2011 21:27:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94918 invoked by uid 500); 19 May 2011 21:27:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94910 invoked by uid 99); 19 May 2011 21:27:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:27:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:27:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 395A4D1361
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 21:26:48 +0000 (UTC)
Date: Thu, 19 May 2011 21:26:48 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <955503386.28301.1305840408231.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <889607684.27597.1305830207722.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7307) Remove Dr Who references
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036487#comment-13036487 ] 

Owen O'Malley commented on HADOOP-7307:
---------------------------------------

I don't like this proposed change either. The result is easy to google (see http://lmgtfy.com/?q=hadoop+drwho ) and is more interesting that a utilitarian "default.web.user" or some such.

> Remove Dr Who references
> ------------------------
>
>                 Key: HADOOP-7307
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7307
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>             Fix For: 0.22.0
>
>
> I've had feedback from a number of different operators that the "dr who" references that occasionally come out of Hadoop are annoying and confusing. We should replace them with more descriptive error messages, usernames, etc.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15171-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 21:32:29 2011
Return-Path: <common-issues-return-15171-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F24AB4C21
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 21:32:28 +0000 (UTC)
Received: (qmail 99686 invoked by uid 500); 19 May 2011 21:32:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99656 invoked by uid 500); 19 May 2011 21:32:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99584 invoked by uid 99); 19 May 2011 21:32:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:32:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:32:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 07CD1D160F
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 21:31:48 +0000 (UTC)
Date: Thu, 19 May 2011 21:31:48 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <650203133.28347.1305840708028.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1105491925.1375.1305065387432.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7272) Remove unnecessary security
 related info logs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036494#comment-13036494 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7272:
------------------------------------------------

+1 the new 205 patch looks good.

> Remove unnecessary security related info logs
> ---------------------------------------------
>
>                 Key: HADOOP-7272
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7272
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>    Affects Versions: 0.23.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7272.patch, HADOOP-7272.rel205.patch, HADOOP-7272.rel205.patch
>
>
> Two info logs are printed when connection to RPC server is established, is not necessary. On a production cluster, these log lines made up of close to 50% of lines in the namenode log. I propose changing them into debug logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15172-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 21:34:31 2011
Return-Path: <common-issues-return-15172-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ECC36468F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 21:34:30 +0000 (UTC)
Received: (qmail 4469 invoked by uid 500); 19 May 2011 21:34:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4421 invoked by uid 500); 19 May 2011 21:34:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4401 invoked by uid 99); 19 May 2011 21:34:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:34:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:34:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AE9D5D175E
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 21:33:47 +0000 (UTC)
Date: Thu, 19 May 2011 21:33:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1474873918.28357.1305840827712.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <889607684.27597.1305830207722.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7307) Remove Dr Who references
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036495#comment-13036495 ] 

Todd Lipcon commented on HADOOP-7307:
-------------------------------------

Some of our poor users don't have internet access in the buildings/datacenters where their clusters reside. (I don't know how they keep busy while waiting on their MR jobs without twitter... but somehow they manage).

So, a self-documenting "utilitarian" solution is preferable to an "interesting" one that has to be googled.

> Remove Dr Who references
> ------------------------
>
>                 Key: HADOOP-7307
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7307
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>             Fix For: 0.22.0
>
>
> I've had feedback from a number of different operators that the "dr who" references that occasionally come out of Hadoop are annoying and confusing. We should replace them with more descriptive error messages, usernames, etc.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15173-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 21:44:30 2011
Return-Path: <common-issues-return-15173-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B707E49EB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 21:44:30 +0000 (UTC)
Received: (qmail 24065 invoked by uid 500); 19 May 2011 21:44:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23975 invoked by uid 500); 19 May 2011 21:44:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23967 invoked by uid 99); 19 May 2011 21:44:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:44:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:44:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A7AF7D1CE8
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 21:43:47 +0000 (UTC)
Date: Thu, 19 May 2011 21:43:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1193575182.28438.1305841427683.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1008788632.26385.1305807047638.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7305) Eclipse project files are
 incomplete
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036509#comment-13036509 ] 

Hudson commented on HADOOP-7305:
--------------------------------

Integrated in Hadoop-Common-22-branch #53 (See [https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/53/])
    HADOOP-7305. Eclipse project classpath should include tools.jar from JDK. Contributed by Niels Basjes.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1125050
Files : 
* /hadoop/common/branches/branch-0.22/CHANGES.txt
* /hadoop/common/branches/branch-0.22/build.xml


> Eclipse project files are incomplete
> ------------------------------------
>
>                 Key: HADOOP-7305
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7305
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Niels Basjes
>            Assignee: Niels Basjes
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7305-2011-05-19.patch
>
>
> After a fresh checkout of hadoop-common I do 'ant compile eclipse'.
> I open eclipse, set ANT_HOME and build the project. 
> At that point the following error appears:
> {quote}
> The type com.sun.javadoc.RootDoc cannot be resolved. It is indirectly referenced from required .class files	ExcludePrivateAnnotationsJDiffDoclet.java	/common/src/java/org/apache/hadoop/classification/tools	line 1	Java Problem
> {quote}
> The solution is to add the "tools.jar" from the JDK to the buildpath/classpath.
> This should be fixed in the build.xml.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15174-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 21:46:30 2011
Return-Path: <common-issues-return-15174-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 81FF44526
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 21:46:30 +0000 (UTC)
Received: (qmail 25388 invoked by uid 500); 19 May 2011 21:46:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25343 invoked by uid 500); 19 May 2011 21:46:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25335 invoked by uid 99); 19 May 2011 21:46:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:46:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 21:46:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A0BF2D1E12
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 21:45:47 +0000 (UTC)
Date: Thu, 19 May 2011 21:45:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1132150553.28453.1305841547655.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036511#comment-13036511 ] 

Eli Collins commented on HADOOP-6605:
-------------------------------------

In the above the ". bin/hadoop-config.sh" should use "bash" so it tries to re-detect JAVA_HOME w/ each invocation, however the difference in run time is still tiny.

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch, hadoop-6605-2.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15175-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 22:02:29 2011
Return-Path: <common-issues-return-15175-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9E28C4F25
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 22:02:29 +0000 (UTC)
Received: (qmail 41729 invoked by uid 500); 19 May 2011 22:02:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41705 invoked by uid 500); 19 May 2011 22:02:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41696 invoked by uid 99); 19 May 2011 22:02:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 22:02:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 22:02:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B724ED14F3
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 22:01:48 +0000 (UTC)
Date: Thu, 19 May 2011 22:01:48 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1312733255.28522.1305842508746.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-6255:
------------------------------

    Attachment: HADOOP-6255-common-trunk-4.patch

Store config templates in $PREFIX/share/hadoop/templates, and change related script to use the new location.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk-4.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-6255-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15176-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 22:11:56 2011
Return-Path: <common-issues-return-15176-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C0BB14A3F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 22:11:56 +0000 (UTC)
Received: (qmail 51577 invoked by uid 500); 19 May 2011 22:11:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51504 invoked by uid 500); 19 May 2011 22:11:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51473 invoked by uid 99); 19 May 2011 22:11:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 22:11:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 22:04:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A2601D1627
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 22:03:47 +0000 (UTC)
Date: Thu, 19 May 2011 22:03:47 +0000 (UTC)
From: "Eugene Koontz (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <852549278.28545.1305842627661.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <889607684.27597.1305830207722.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7307) Remove Dr Who references
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036523#comment-13036523 ] 

Eugene Koontz commented on HADOOP-7307:
---------------------------------------

A bit ironically, the first link that google returns has the following:

"Not the most helpful username to pick for that scenario, to say the least."

> Remove Dr Who references
> ------------------------
>
>                 Key: HADOOP-7307
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7307
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>             Fix For: 0.22.0
>
>
> I've had feedback from a number of different operators that the "dr who" references that occasionally come out of Hadoop are annoying and confusing. We should replace them with more descriptive error messages, usernames, etc.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15177-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 22:22:28 2011
Return-Path: <common-issues-return-15177-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8863A415D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 22:22:28 +0000 (UTC)
Received: (qmail 59814 invoked by uid 500); 19 May 2011 22:22:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59786 invoked by uid 500); 19 May 2011 22:22:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59778 invoked by uid 99); 19 May 2011 22:22:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 22:22:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 22:22:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 76721D1E93
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 22:21:47 +0000 (UTC)
Date: Thu, 19 May 2011 22:21:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1854804070.28608.1305843707481.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036532#comment-13036532 ] 

Owen O'Malley commented on HADOOP-6255:
---------------------------------------

For trunk, the default should be $PREFIX/var/log and $PREFIX/var/run.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk-4.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-6255-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15178-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 22:24:28 2011
Return-Path: <common-issues-return-15178-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 63257419A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 22:24:28 +0000 (UTC)
Received: (qmail 61191 invoked by uid 500); 19 May 2011 22:24:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61164 invoked by uid 500); 19 May 2011 22:24:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61156 invoked by uid 99); 19 May 2011 22:24:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 22:24:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 22:24:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 793A7D1F80
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 22:23:47 +0000 (UTC)
Date: Thu, 19 May 2011 22:23:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1634949548.28642.1305843827493.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036533#comment-13036533 ] 

Owen O'Malley commented on HADOOP-6255:
---------------------------------------

It also looks like you picked up a generated file (configure).

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk-4.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-6255-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15179-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 22:40:30 2011
Return-Path: <common-issues-return-15179-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8BA7B4AD8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 22:40:30 +0000 (UTC)
Received: (qmail 85352 invoked by uid 500); 19 May 2011 22:40:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85310 invoked by uid 500); 19 May 2011 22:40:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85301 invoked by uid 99); 19 May 2011 22:40:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 22:40:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 22:40:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B4839D2507
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 22:39:47 +0000 (UTC)
Date: Thu, 19 May 2011 22:39:47 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <454487436.28749.1305844787735.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-6255:
------------------------------

    Attachment: HADOOP-6255-common-trunk-5.patch

Set default log and pid directory to $PREFIX/var/log and $PREFIX/var/run respectively for hadoop-env.sh

Removed incorrect change to src/test/c++/runAs/configure.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk-4.patch, HADOOP-6255-common-trunk-5.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-6255-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15180-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 22:50:30 2011
Return-Path: <common-issues-return-15180-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 54262434D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 22:50:30 +0000 (UTC)
Received: (qmail 98321 invoked by uid 500); 19 May 2011 22:50:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98271 invoked by uid 500); 19 May 2011 22:50:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98263 invoked by uid 99); 19 May 2011 22:50:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 22:50:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 22:50:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 705ADD2918
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 22:49:47 +0000 (UTC)
Date: Thu, 19 May 2011 22:49:47 +0000 (UTC)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1145070693.28819.1305845387456.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7311) Port remaining metrics v1 from trunk
 to branch-0.20-security
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Port remaining metrics v1 from trunk to branch-0.20-security
------------------------------------------------------------

                 Key: HADOOP-7311
                 URL: https://issues.apache.org/jira/browse/HADOOP-7311
             Project: Hadoop Common
          Issue Type: Bug
          Components: metrics
    Affects Versions: 0.20.203.0
            Reporter: Konstantin Shvachko
            Assignee: Konstantin Shvachko
             Fix For: 0.20.204.0


HADOOP-7190 added metrics packages/classes. This is a port from trunk to make them actually work for the security branch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15181-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 22:56:30 2011
Return-Path: <common-issues-return-15181-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 80CEE4515
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 22:56:30 +0000 (UTC)
Received: (qmail 3732 invoked by uid 500); 19 May 2011 22:56:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3570 invoked by uid 500); 19 May 2011 22:56:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3562 invoked by uid 99); 19 May 2011 22:56:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 22:56:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 22:56:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AB23DD2A8B
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 22:55:47 +0000 (UTC)
Date: Thu, 19 May 2011 22:55:47 +0000 (UTC)
From: "Cesar Delgado (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1772838797.28835.1305845747698.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <889607684.27597.1305830207722.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7307) Remove Dr Who references
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036553#comment-13036553 ] 

Cesar Delgado commented on HADOOP-7307:
---------------------------------------

As an operations person this was horrible the first time it came up.  Seeing a user that is not supposed to be in your system and was not created by anyone can cause a huge panic and waste a lot of time.  The error should be thrown with the correct user and group (not Tardis) for the process on the system (ie. *NIX user and group)

Errors should be clear and concise.  I'm ok with looking at manuals for extra information but, like Todd said, "just google it" is not an option in most data-centers.

> Remove Dr Who references
> ------------------------
>
>                 Key: HADOOP-7307
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7307
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>             Fix For: 0.22.0
>
>
> I've had feedback from a number of different operators that the "dr who" references that occasionally come out of Hadoop are annoying and confusing. We should replace them with more descriptive error messages, usernames, etc.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15182-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 23:04:28 2011
Return-Path: <common-issues-return-15182-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8CF1B4B7B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 23:04:28 +0000 (UTC)
Received: (qmail 9720 invoked by uid 500); 19 May 2011 23:04:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9691 invoked by uid 500); 19 May 2011 23:04:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9678 invoked by uid 99); 19 May 2011 23:04:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 23:04:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 23:04:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 659A0D2C81
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 23:03:47 +0000 (UTC)
Date: Thu, 19 May 2011 23:03:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <367540856.28871.1305846227412.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036559#comment-13036559 ] 

Eli Collins commented on HADOOP-6605:
-------------------------------------

Thanks for reviewing Daryn!
 
bq. SunOS - why not jdk1.6.* to get latest patch version, or jdk*.*? Using the last item in the glob expansion or search in reverse order should generally get the latest version.

>From googling it looked like "jdk1.6.0" was a stable path on Solaris, ie no glob necessary. Maybe someone who uses Solaris can verify, I'm fine punting Solaris to a future change as well.

bq. Linux - Same as SunOS. Also, why aren't /usr/java/default and /usr/lib/jvm/default-java checked first?

The code looks for Sun Java 6 first because Hadoop depends on Sun Java 6, ie we don't necessarily want the system default Java.

bq. Overall, maybe globs aren't a great idea since it may lead to a lot more paths being added which I think is Allen's concern. 

These globs should not result in a lot more paths being added. Eg how many paths would you expect "/usr/java/jdk1.6*" to match on most systems? Probably none, a couple would be a lot right? Even if these globs matched 20 paths per above it shouldn't impact performance.

bq. Would it make more sense to only support using the "standard" mechanisms of an OS if that OS provides one? In which case the functionality is to make hadoop a good citizen of that OS, and not to make pseudo-intelligent guesses?

There is no standard path across Linux distributions for Sun Java 6, these globs match Sun Java 6 on popular Linux distributions, that's how the list was generated.

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch, hadoop-6605-2.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15183-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 23:06:29 2011
Return-Path: <common-issues-return-15183-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EA6F54C81
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 23:06:29 +0000 (UTC)
Received: (qmail 10975 invoked by uid 500); 19 May 2011 23:06:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10943 invoked by uid 500); 19 May 2011 23:06:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10935 invoked by uid 99); 19 May 2011 23:06:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 23:06:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 23:06:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 513CCD2CFC
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 23:05:47 +0000 (UTC)
Date: Thu, 19 May 2011 23:05:47 +0000 (UTC)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2059142414.28880.1305846347329.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1145070693.28819.1305845387456.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7311) Port remaining metrics v1 from trunk
 to branch-0.20-security
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Shvachko updated HADOOP-7311:
----------------------------------------

    Attachment: metrics-0.20-security.patch

Could somebody please review this.

> Port remaining metrics v1 from trunk to branch-0.20-security
> ------------------------------------------------------------
>
>                 Key: HADOOP-7311
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7311
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.203.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: 0.20.204.0
>
>         Attachments: metrics-0.20-security.patch
>
>
> HADOOP-7190 added metrics packages/classes. This is a port from trunk to make them actually work for the security branch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15184-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 23:14:30 2011
Return-Path: <common-issues-return-15184-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8F16A4DFF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 23:14:30 +0000 (UTC)
Received: (qmail 20169 invoked by uid 500); 19 May 2011 23:14:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20136 invoked by uid 500); 19 May 2011 23:14:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20128 invoked by uid 99); 19 May 2011 23:14:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 23:14:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 23:14:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B0D63D2EF1
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 23:13:47 +0000 (UTC)
Date: Thu, 19 May 2011 23:13:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1653939372.28898.1305846827721.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <889607684.27597.1305830207722.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7307) Remove Dr Who references
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036563#comment-13036563 ] 

Owen O'Malley commented on HADOOP-7307:
---------------------------------------

Cesar, this isn't about changing an error message. Todd is suggesting changing the default name. You would have had *exactly* the same problem with "webuser" except it would be far worse panic.

> Remove Dr Who references
> ------------------------
>
>                 Key: HADOOP-7307
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7307
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>             Fix For: 0.22.0
>
>
> I've had feedback from a number of different operators that the "dr who" references that occasionally come out of Hadoop are annoying and confusing. We should replace them with more descriptive error messages, usernames, etc.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15185-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 23:20:30 2011
Return-Path: <common-issues-return-15185-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8027B463E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 23:20:30 +0000 (UTC)
Received: (qmail 27874 invoked by uid 500); 19 May 2011 23:20:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27849 invoked by uid 500); 19 May 2011 23:20:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27841 invoked by uid 99); 19 May 2011 23:20:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 23:20:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 23:20:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D13ADD11A8
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 23:19:47 +0000 (UTC)
Date: Thu, 19 May 2011 23:19:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2145166079.28929.1305847187853.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <889607684.27597.1305830207722.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7307) Remove Dr Who references
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036570#comment-13036570 ] 

Todd Lipcon commented on HADOOP-7307:
-------------------------------------

I think if I saw messages in my audit log that "webuser" or "hdfswebui" was accessing files, I'd pretty quickly figure out where they were coming from. Seeing messages from "dr.who" in the audit log would scare me.

> Remove Dr Who references
> ------------------------
>
>                 Key: HADOOP-7307
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7307
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>             Fix For: 0.22.0
>
>
> I've had feedback from a number of different operators that the "dr who" references that occasionally come out of Hadoop are annoying and confusing. We should replace them with more descriptive error messages, usernames, etc.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15186-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 19 23:51:31 2011
Return-Path: <common-issues-return-15186-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9F0194890
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 19 May 2011 23:51:31 +0000 (UTC)
Received: (qmail 63384 invoked by uid 500); 19 May 2011 23:51:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63353 invoked by uid 500); 19 May 2011 23:51:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63345 invoked by uid 99); 19 May 2011 23:51:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 23:51:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 May 2011 23:51:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B7CF5D1B92
	for <common-issues@hadoop.apache.org>; Thu, 19 May 2011 23:50:48 +0000 (UTC)
Date: Thu, 19 May 2011 23:50:48 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <281161099.28997.1305849048749.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1145070693.28819.1305845387456.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7311) Port remaining metrics v1 from
 trunk to branch-0.20-security
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036581#comment-13036581 ] 

Luke Lu commented on HADOOP-7311:
---------------------------------

Trunk is already on metrics 2. HADOOP-7190 added back the old package to make it compatible with legacy tools. 0.20-2xx release is on metrics 2 since the beginning and is in production for many months.

> Port remaining metrics v1 from trunk to branch-0.20-security
> ------------------------------------------------------------
>
>                 Key: HADOOP-7311
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7311
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.203.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: 0.20.204.0
>
>         Attachments: metrics-0.20-security.patch
>
>
> HADOOP-7190 added metrics packages/classes. This is a port from trunk to make them actually work for the security branch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15187-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 00:03:32 2011
Return-Path: <common-issues-return-15187-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 554354328
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 00:03:32 +0000 (UTC)
Received: (qmail 83008 invoked by uid 500); 20 May 2011 00:03:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82957 invoked by uid 500); 20 May 2011 00:03:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82835 invoked by uid 99); 20 May 2011 00:03:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 00:03:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 00:03:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 48BF8D2090
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 00:02:49 +0000 (UTC)
Date: Fri, 20 May 2011 00:02:49 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <677175669.29028.1305849769294.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <187742091.8722.1305246647362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7287) Configuration deprecation
 mechanism doesn't work properly for GenericOptionsParser/Tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036587#comment-13036587 ] 

Hadoop QA commented on HADOOP-7287:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479736/hadoop-7287-trunk.1.patch
  against trunk revision 1125051.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/484//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/484//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/484//console

This message is automatically generated.

> Configuration deprecation mechanism doesn't work properly for GenericOptionsParser/Tools
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7287
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7287
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Aaron T. Myers
>            Priority: Blocker
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: hadoop-7287-still-broken.txt, hadoop-7287-testcase.txt, hadoop-7287-trunk.0.patch, hadoop-7287-trunk.1.patch
>
>
> For example, you can't use -D options on the "hadoop fs" command line in order to specify the deprecated names of configuration options. The issue is that the ordering is:
> - JVM starts
> - GenericOptionsParser creates a Configuration object and calls set() for each of the options specified on command line
> - DistributedFileSystem or other class eventually instantiates HdfsConfiguration which adds the deprecations
> - Some class calls conf.get("new key") and sees the default instead of the version set on the command line

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15188-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 00:05:30 2011
Return-Path: <common-issues-return-15188-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2478C4776
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 00:05:30 +0000 (UTC)
Received: (qmail 83845 invoked by uid 500); 20 May 2011 00:05:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83716 invoked by uid 500); 20 May 2011 00:05:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83529 invoked by uid 99); 20 May 2011 00:05:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 00:05:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 00:05:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5D0D1D20F6
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 00:04:47 +0000 (UTC)
Date: Fri, 20 May 2011 00:04:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1093348183.29034.1305849887377.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7312) core-default.xml lists configuration
 version as 0.21
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

core-default.xml lists configuration version as 0.21
----------------------------------------------------

                 Key: HADOOP-7312
                 URL: https://issues.apache.org/jira/browse/HADOOP-7312
             Project: Hadoop Common
          Issue Type: Bug
          Components: conf
    Affects Versions: 0.22.0
            Reporter: Todd Lipcon
            Priority: Minor
             Fix For: 0.22.0


This key was added in HADOOP-6233, though appears unused. I suppose it's somewhat useful to try to diagnose if someone has old versions of core-default.xml on the classpath.

Either way it should probably be updated to say 0.22 in the branch and 0.23 in trunk.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15189-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 00:09:28 2011
Return-Path: <common-issues-return-15189-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B28E548DF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 00:09:28 +0000 (UTC)
Received: (qmail 89701 invoked by uid 500); 20 May 2011 00:09:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89661 invoked by uid 500); 20 May 2011 00:09:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89640 invoked by uid 99); 20 May 2011 00:09:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 00:09:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 00:09:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9FF6CD2248
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 00:08:47 +0000 (UTC)
Date: Fri, 20 May 2011 00:08:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1624586812.29037.1305850127652.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1145070693.28819.1305845387456.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7311) Port remaining metrics v1 from
 trunk to branch-0.20-security
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036589#comment-13036589 ] 

Luke Lu commented on HADOOP-7311:
---------------------------------

I wonder what you're trying to achieve. The patch completely breaks metrics2 instrumentation (i.e. people already on metrics2 will not see the metrics, especially JMX metrics support in JT and TT).

If you just want to make Ganglia work, porting Ganglia plugin to metrics2 is a much smaller patch.

> Port remaining metrics v1 from trunk to branch-0.20-security
> ------------------------------------------------------------
>
>                 Key: HADOOP-7311
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7311
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.203.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: 0.20.204.0
>
>         Attachments: metrics-0.20-security.patch
>
>
> HADOOP-7190 added metrics packages/classes. This is a port from trunk to make them actually work for the security branch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15190-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 00:15:31 2011
Return-Path: <common-issues-return-15190-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 377784B84
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 00:15:31 +0000 (UTC)
Received: (qmail 92771 invoked by uid 500); 20 May 2011 00:15:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92738 invoked by uid 500); 20 May 2011 00:15:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92730 invoked by uid 99); 20 May 2011 00:15:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 00:15:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 00:15:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3B5BED243D
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 00:14:50 +0000 (UTC)
Date: Fri, 20 May 2011 00:14:50 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1154349099.29094.1305850490239.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036594#comment-13036594 ] 

Hadoop QA commented on HADOOP-6255:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479854/HADOOP-6255-common-trunk-5.patch
  against trunk revision 1125051.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 27 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    -1 release audit.  The applied patch generated 3 release audit warnings (more than the trunk's current 1 warnings).

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/485//testReport/
Release audit warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/485//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/485//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/485//console

This message is automatically generated.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk-4.patch, HADOOP-6255-common-trunk-5.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-6255-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15191-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 00:27:28 2011
Return-Path: <common-issues-return-15191-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6A2994428
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 00:27:28 +0000 (UTC)
Received: (qmail 15353 invoked by uid 500); 20 May 2011 00:27:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15322 invoked by uid 500); 20 May 2011 00:27:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15313 invoked by uid 99); 20 May 2011 00:27:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 00:27:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 00:27:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7D1F2D2796
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 00:26:47 +0000 (UTC)
Date: Fri, 20 May 2011 00:26:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2101164540.29120.1305851207509.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <665646418.27971.1305836567397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7308) Remove unused TaskLogAppender
 configurations from log4j.properties
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036598#comment-13036598 ] 

Hadoop QA commented on HADOOP-7308:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479825/hadoop-7308.txt
  against trunk revision 1125051.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/486//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/486//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/486//console

This message is automatically generated.

> Remove unused TaskLogAppender configurations from log4j.properties
> ------------------------------------------------------------------
>
>                 Key: HADOOP-7308
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7308
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7308.txt
>
>
> MAPREDUCE-2372 improved TaskLogAppender to no longer need as much "wiring" in log4j.properties. There are also some old properties in there that are no longer used (eg logsRetainHours and noKeepSplits).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15192-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 00:27:29 2011
Return-Path: <common-issues-return-15192-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 00A204446
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 00:27:29 +0000 (UTC)
Received: (qmail 15659 invoked by uid 500); 20 May 2011 00:27:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15582 invoked by uid 500); 20 May 2011 00:27:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15548 invoked by uid 99); 20 May 2011 00:27:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 00:27:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 00:27:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B0DBED279C
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 00:26:47 +0000 (UTC)
Date: Fri, 20 May 2011 00:26:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <42598014.29126.1305851207721.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <580585038.27366.1305827567709.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7306) Start metrics system even if
 config files are missing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036600#comment-13036600 ] 

Hadoop QA commented on HADOOP-7306:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479806/hadoop-7306-missing-conf-v1.patch
  against trunk revision 1125051.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/487//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/487//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/487//console

This message is automatically generated.

> Start metrics system even if config files are missing
> -----------------------------------------------------
>
>                 Key: HADOOP-7306
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7306
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: metrics
>    Affects Versions: 0.23.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7306-missing-conf-v1.patch
>
>
> Per experience and discussion with HDFS-1922, it seems preferable to treat missing metrics config file as empty/default config, which is more compatible with metrics v1 behavior (the MBeans are always registered.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15193-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 00:29:28 2011
Return-Path: <common-issues-return-15193-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7C41B4496
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 00:29:28 +0000 (UTC)
Received: (qmail 16694 invoked by uid 500); 20 May 2011 00:29:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16664 invoked by uid 500); 20 May 2011 00:29:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16656 invoked by uid 99); 20 May 2011 00:29:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 00:29:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 00:29:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A15A0D283C
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 00:28:47 +0000 (UTC)
Date: Fri, 20 May 2011 00:28:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1173108669.29149.1305851327657.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <889607684.27597.1305830207722.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7307) Remove Dr Who references
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036602#comment-13036602 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7307:
------------------------------------------------

> ... The result is easy to google (see http://lmgtfy.com/?q=hadoop+drwho ) ...

I had removed "Dr Who" once in HADOOP-2833, which is the second search result.  However, it managed to come back.  Now, people are scared -- it becomes the phantom of the cluster.  Just kidding.  :)

> Remove Dr Who references
> ------------------------
>
>                 Key: HADOOP-7307
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7307
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>             Fix For: 0.22.0
>
>
> I've had feedback from a number of different operators that the "dr who" references that occasionally come out of Hadoop are annoying and confusing. We should replace them with more descriptive error messages, usernames, etc.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15194-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 00:35:30 2011
Return-Path: <common-issues-return-15194-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6493E4B36
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 00:35:30 +0000 (UTC)
Received: (qmail 24942 invoked by uid 500); 20 May 2011 00:35:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24917 invoked by uid 500); 20 May 2011 00:35:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24908 invoked by uid 99); 20 May 2011 00:35:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 00:35:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 00:35:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9B143D298B
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 00:34:47 +0000 (UTC)
Date: Fri, 20 May 2011 00:34:47 +0000 (UTC)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1863194420.29179.1305851687630.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <889607684.27597.1305830207722.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7307) Remove Dr Who references
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036606#comment-13036606 ] 

Jakob Homan commented on HADOOP-7307:
-------------------------------------

bq. However, it managed to come back.
The Doctor can regenerate 13 times (http://bit.ly/kh3sDj).  As such, it'll take 11 more jiras before this effort succeeds.

> Remove Dr Who references
> ------------------------
>
>                 Key: HADOOP-7307
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7307
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>             Fix For: 0.22.0
>
>
> I've had feedback from a number of different operators that the "dr who" references that occasionally come out of Hadoop are annoying and confusing. We should replace them with more descriptive error messages, usernames, etc.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15195-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 01:10:31 2011
Return-Path: <common-issues-return-15195-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CBC7E4DF8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 01:10:31 +0000 (UTC)
Received: (qmail 53651 invoked by uid 500); 20 May 2011 01:10:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53618 invoked by uid 500); 20 May 2011 01:10:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53608 invoked by uid 99); 20 May 2011 01:10:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 01:10:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 01:10:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6DFE0D2008
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 01:09:48 +0000 (UTC)
Date: Fri, 20 May 2011 01:09:48 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1942245897.29267.1305853788447.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-6671:
---------------------------------------

    Attachment: HADOOP-6671.patch
                mvn-layout.sh

How to apply the patch (I'm using GIT):

* Checkout hadoop-common trunk
* run the mvn-layout.sh script
* git 'add -u' and 'git add' all new files dirs (git will recognize all the moving of files are renames)
* commit all changes as first commit (a separate JIRA could be created for this if you find it necessary)
* apply the HADOOP-6671.patch
* test all works as indicated below
* commit

It requires Maven 3

Typical Maven goals:

* mvn clean   
* mvn compile
* mvn test
* mvn package

The last one generates the TARBALL. As with the Ant build, FORREST_HOME must be defined.

Options:

* -Dtest=<TestClass> to run a single test
* -DskipTests to skip running the tests (package by default runs tests)
* -Dcompile.native to compile the native code

This patch does the minimal code changes (one line in a test class where 'build/' was hardcoded) and the fixFont script for chinese documentation.

There are some testcases that are writing files in the current directory instead under the build directory. Those should be fixed as part of a follow up issue (thus concentrating this one in just the mavenization).

Native code compilation and Documentation generation are done using 2 auxiliary ant scripts invoked from Maven POM.

As a follow up, when Mavenizing HDFS and MAPREDUCE we can introduce a parent POM file where all dependencies versions are commonly defined (same version for all).

A couple of JAR files are missing from the final TARBALL, I assume they were included in the Ant build due to a ivy scope mistake (jdiff, aspectj-tools)

JDIFF/Clover/FindBugs/Cobertura/etc/etc can be easily integrated in the Maven POM. But again, I think this should be done incrementally on top of this.



> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>         Attachments: HADOOP-6671.patch, build.png, hadoop-commons-maven.patch, mvn-layout.sh, mvn-layout.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15196-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 01:10:32 2011
Return-Path: <common-issues-return-15196-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3A9134E07
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 01:10:32 +0000 (UTC)
Received: (qmail 53893 invoked by uid 500); 20 May 2011 01:10:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53856 invoked by uid 500); 20 May 2011 01:10:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53847 invoked by uid 99); 20 May 2011 01:10:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 01:10:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 01:10:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 563ABD203B
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 01:09:49 +0000 (UTC)
Date: Fri, 20 May 2011 01:09:49 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2053156835.29290.1305853789350.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-6671:
---------------------------------------

    Status: Patch Available  (was: Open)

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>         Attachments: HADOOP-6671.patch, build.png, hadoop-commons-maven.patch, mvn-layout.sh, mvn-layout.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15197-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 01:14:30 2011
Return-Path: <common-issues-return-15197-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 81470405D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 01:14:30 +0000 (UTC)
Received: (qmail 58884 invoked by uid 500); 20 May 2011 01:14:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58859 invoked by uid 500); 20 May 2011 01:14:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58851 invoked by uid 99); 20 May 2011 01:14:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 01:14:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 01:14:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0A40AD214A
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 01:13:49 +0000 (UTC)
Date: Fri, 20 May 2011 01:13:49 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <762590996.29312.1305854029038.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036616#comment-13036616 ] 

Hadoop QA commented on HADOOP-6671:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479868/HADOOP-6671.patch
  against trunk revision 1125051.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 18 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/488//console

This message is automatically generated.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>         Attachments: HADOOP-6671.patch, build.png, hadoop-commons-maven.patch, mvn-layout.sh, mvn-layout.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15198-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 02:00:29 2011
Return-Path: <common-issues-return-15198-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A71C54237
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 02:00:29 +0000 (UTC)
Received: (qmail 86169 invoked by uid 500); 20 May 2011 02:00:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86137 invoked by uid 500); 20 May 2011 02:00:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86129 invoked by uid 99); 20 May 2011 02:00:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 02:00:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 02:00:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C3045D2ACE
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 01:59:48 +0000 (UTC)
Date: Fri, 20 May 2011 01:59:48 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <868302516.29389.1305856788795.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-6255:
------------------------------

    Attachment: HADOOP-6255-common-trunk-6.patch

* Bugfix for debian package name
* Use absolute path for setup scripts in case $HADOOP_PREFIX/bin/hadoop is not in user PATH.
* Fix typo in patch 5.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk-4.patch, HADOOP-6255-common-trunk-5.patch, HADOOP-6255-common-trunk-6.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-6255-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15199-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 02:30:28 2011
Return-Path: <common-issues-return-15199-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4F92262B9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 02:30:28 +0000 (UTC)
Received: (qmail 1185 invoked by uid 500); 20 May 2011 02:30:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1160 invoked by uid 500); 20 May 2011 02:30:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1152 invoked by uid 99); 20 May 2011 02:30:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 02:30:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 02:30:26 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5CBFCD2138
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 02:29:47 +0000 (UTC)
Date: Fri, 20 May 2011 02:29:47 +0000 (UTC)
From: "E. Sammer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1810461116.29429.1305858587376.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7313) Remove all framework use of
 Configuration default args
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Remove all framework use of Configuration default args
------------------------------------------------------

                 Key: HADOOP-7313
                 URL: https://issues.apache.org/jira/browse/HADOOP-7313
             Project: Hadoop Common
          Issue Type: Improvement
          Components: conf
    Affects Versions: 0.21.0
            Reporter: E. Sammer


The Hadoop code base should not use the form of Configuration#get*() that accepts a default.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15200-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 02:30:32 2011
Return-Path: <common-issues-return-15200-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E46B86328
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 02:30:31 +0000 (UTC)
Received: (qmail 1950 invoked by uid 500); 20 May 2011 02:30:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1914 invoked by uid 500); 20 May 2011 02:30:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1865 invoked by uid 99); 20 May 2011 02:30:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 02:30:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 02:30:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C1966D213C
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 02:29:47 +0000 (UTC)
Date: Fri, 20 May 2011 02:29:47 +0000 (UTC)
From: "E. Sammer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1541567534.29431.1305858587789.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1810461116.29429.1305858587376.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7313) Remove all framework use of
 Configuration default args
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036632#comment-13036632 ] 

E. Sammer commented on HADOOP-7313:
-----------------------------------

Many places in the code base use the form of Configuration#get*() that accepts defaults. This makes life much easier for developers in that we can be sure we don't get a NullPointerException on return, but it creates a situation where *-default.xml does not match the defaults supplied in code. This presents potentially odd behavior. Ideally, an option missing from the proper *-default.xml file is a build / release error rather than something that silently results in an undocumented value.

Ari Rabkin did some research into this[1] and identified some of these inconsistencies.

[1] http://www.cs.berkeley.edu/~asrabkin/hadoop_opts/cdh3u0-options.html (While this says cdh3u0, I believe it still applies to Apache Hadoop trunk).

> Remove all framework use of Configuration default args
> ------------------------------------------------------
>
>                 Key: HADOOP-7313
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7313
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>    Affects Versions: 0.21.0
>            Reporter: E. Sammer
>
> The Hadoop code base should not use the form of Configuration#get*() that accepts a default.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15201-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 03:05:33 2011
Return-Path: <common-issues-return-15201-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D1EA761DD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 03:05:33 +0000 (UTC)
Received: (qmail 27110 invoked by uid 500); 20 May 2011 03:05:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27082 invoked by uid 500); 20 May 2011 03:05:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27070 invoked by uid 99); 20 May 2011 03:05:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 03:05:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 03:05:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3B84AD2875
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 03:04:49 +0000 (UTC)
Date: Fri, 20 May 2011 03:04:49 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <641650530.29489.1305860689240.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036640#comment-13036640 ] 

Todd Lipcon commented on HADOOP-6671:
-------------------------------------

Cool stuff, Alejandro! A few things I noticed while trying this out locally:

- nit: The project name is "Hadoop Common" rather than "Hadoop Commons"
- I'm getting errors for our usage of the com.sun.javadoc stuff - eg:
{code}
/home/todd/git/hadoop-common/src/main/java/org/apache/hadoop/classification/tools/ExcludePrivateAnnotationsJDiffDoclet.java:[42,16] cannot access com.sun.javadoc.Doclet
class file for com.sun.javadoc.Doclet not found
    return JDiff.start(RootDocProcessor.process(root));
/home/todd/git/hadoop-common/src/main/java/org/apache/hadoop/classification/tools/RootDocProcessor.java:[55,12] cannot find symbol
{code}
Is this the bit where I need to add a softlink? Any workaround for this we can do local to the project?

- "mvn test" seems to be dumping all the log to stdout. Is there a way to get it to log just to the files like ant used to?
- is there an equivalent of our "bin-package" target which makes a tarball but doesn't build the forrest docs?
- the assembly doesn't seem to build if I choose not to compile native:
{code}

[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] An Ant BuildException has occured: /home/todd/git/hadoop-common/target/native/lib does not exist.
{code}


> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>         Attachments: HADOOP-6671.patch, build.png, hadoop-commons-maven.patch, mvn-layout.sh, mvn-layout.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15202-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 03:24:32 2011
Return-Path: <common-issues-return-15202-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EACAC6929
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 03:24:31 +0000 (UTC)
Received: (qmail 37059 invoked by uid 500); 20 May 2011 03:24:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36847 invoked by uid 500); 20 May 2011 03:24:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36839 invoked by uid 99); 20 May 2011 03:24:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 03:24:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 03:24:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 57DA9D2C5C
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 03:23:47 +0000 (UTC)
Date: Fri, 20 May 2011 03:23:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1293410178.29518.1305861827356.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <580585038.27366.1305827567709.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7306) Start metrics system even if config
 files are missing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7306:
--------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

Committed to trunk, thanks Luke!

> Start metrics system even if config files are missing
> -----------------------------------------------------
>
>                 Key: HADOOP-7306
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7306
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: metrics
>    Affects Versions: 0.23.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7306-missing-conf-v1.patch
>
>
> Per experience and discussion with HDFS-1922, it seems preferable to treat missing metrics config file as empty/default config, which is more compatible with metrics v1 behavior (the MBeans are always registered.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15203-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 03:38:31 2011
Return-Path: <common-issues-return-15203-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 580796E9D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 03:38:31 +0000 (UTC)
Received: (qmail 44664 invoked by uid 500); 20 May 2011 03:38:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44524 invoked by uid 500); 20 May 2011 03:38:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44511 invoked by uid 99); 20 May 2011 03:38:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 03:38:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 03:38:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8F215D2F9E
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 03:37:47 +0000 (UTC)
Date: Fri, 20 May 2011 03:37:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <809017996.29546.1305862667583.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2117896899.24772.1305762047299.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7302) webinterface.private.actions should
 not be in common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7302:
--------------------------------

       Resolution: Fixed
    Fix Version/s:     (was: 0.23.0)
                   0.22.0
     Release Note: Option webinterface.private.actions has been renamed to mapreduce.jobtracker.webinterface.trusted and should be specified in mapred-site.xml instead of core-site.xml  (was: option webinterface.private.actions has been renamed to mapreduce.jobtracker.webinterface.trusted)
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

Committed to 22 and trunk (I elected to commit to 22 since it is low risk cleanup)

> webinterface.private.actions should not be in common
> ----------------------------------------------------
>
>                 Key: HADOOP-7302
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7302
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.23.0
>            Reporter: Ari Rabkin
>            Assignee: Ari Rabkin
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7302.patch
>
>
> The comment in -defaults says that webinterface.private.actions applies to both NN and JT. This is wrong. This option is only referenced via the JobTracker. 
> I propose to delete it here, and file a second issue in MAPREDUCE for renaming the option (deprecating the existing name.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15204-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 03:45:32 2011
Return-Path: <common-issues-return-15204-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 737A162BD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 03:45:32 +0000 (UTC)
Received: (qmail 47879 invoked by uid 500); 20 May 2011 03:45:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47716 invoked by uid 500); 20 May 2011 03:45:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47705 invoked by uid 99); 20 May 2011 03:45:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 03:45:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 03:45:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8A8B6D212E
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 03:44:47 +0000 (UTC)
Date: Fri, 20 May 2011 03:44:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1040833875.29556.1305863087548.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <580585038.27366.1305827567709.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7306) Start metrics system even if
 config files are missing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036653#comment-13036653 ] 

Hudson commented on HADOOP-7306:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #612 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/612/])
    HADOOP-7306. Start metrics system even if config files are missing. Contributed by Luke Lu.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1125216
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/metrics2/impl/TestMetricsConfig.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/metrics2/impl/MetricsConfig.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java


> Start metrics system even if config files are missing
> -----------------------------------------------------
>
>                 Key: HADOOP-7306
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7306
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: metrics
>    Affects Versions: 0.23.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7306-missing-conf-v1.patch
>
>
> Per experience and discussion with HDFS-1922, it seems preferable to treat missing metrics config file as empty/default config, which is more compatible with metrics v1 behavior (the MBeans are always registered.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15205-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 03:59:33 2011
Return-Path: <common-issues-return-15205-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 79EEA6429
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 03:59:33 +0000 (UTC)
Received: (qmail 52648 invoked by uid 500); 20 May 2011 03:59:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52615 invoked by uid 500); 20 May 2011 03:59:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52589 invoked by uid 99); 20 May 2011 03:59:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 03:59:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 03:59:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6F662D24E8
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 03:58:47 +0000 (UTC)
Date: Fri, 20 May 2011 03:58:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1972722743.29575.1305863927453.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2117896899.24772.1305762047299.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7302) webinterface.private.actions
 should not be in common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036657#comment-13036657 ] 

Hudson commented on HADOOP-7302:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #613 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/613/])
    HADOOP-7302. webinterface.private.actions should be renamed and moved to the MapReduce project. Contributed by Ari Rabkin.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1125221
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/core-default.xml


> webinterface.private.actions should not be in common
> ----------------------------------------------------
>
>                 Key: HADOOP-7302
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7302
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.23.0
>            Reporter: Ari Rabkin
>            Assignee: Ari Rabkin
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7302.patch
>
>
> The comment in -defaults says that webinterface.private.actions applies to both NN and JT. This is wrong. This option is only referenced via the JobTracker. 
> I propose to delete it here, and file a second issue in MAPREDUCE for renaming the option (deprecating the existing name.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15206-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 03:59:34 2011
Return-Path: <common-issues-return-15206-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7C9EA642B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 03:59:33 +0000 (UTC)
Received: (qmail 52851 invoked by uid 500); 20 May 2011 03:59:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52620 invoked by uid 500); 20 May 2011 03:59:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52602 invoked by uid 99); 20 May 2011 03:59:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 03:59:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 03:59:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8C4B6D24EC
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 03:58:47 +0000 (UTC)
Date: Fri, 20 May 2011 03:58:47 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <698659632.29579.1305863927571.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <456709527.7650.1305225707408.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7283) Include 32-bit and 64-bit native
 libraries in Jenkins tarball builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White resolved HADOOP-7283.
-------------------------------

    Resolution: Fixed
      Assignee: Tom White

The build at https://builds.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-22-Build/ is now producing tarballs with the correct native libraries.

> Include 32-bit and 64-bit native libraries in Jenkins tarball builds
> --------------------------------------------------------------------
>
>                 Key: HADOOP-7283
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7283
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: build
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.22.0
>
>
> The job at https://builds.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-22-Build/ is building tarballs, but they do not currently include both 32-bit and 64-bit native libraries. We should update/duplicate hadoop-nighly/hudsonBuildHadoopRelease.sh to support post-split builds.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15207-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 04:43:38 2011
Return-Path: <common-issues-return-15207-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 186EA664E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 04:43:38 +0000 (UTC)
Received: (qmail 75912 invoked by uid 500); 20 May 2011 04:43:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75336 invoked by uid 500); 20 May 2011 04:43:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75300 invoked by uid 99); 20 May 2011 04:43:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 04:43:35 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 04:43:32 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CE576D2FE6
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 04:42:51 +0000 (UTC)
Date: Fri, 20 May 2011 04:42:51 +0000 (UTC)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <196830359.29647.1305866571841.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1145070693.28819.1305845387456.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7311) Port remaining metrics v1 from
 trunk to branch-0.20-security
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036667#comment-13036667 ] 

Konstantin Shvachko commented on HADOOP-7311:
---------------------------------------------

I am not doing anything with metrix2 except trying not to break it.
My motivation is to make the original metrics work, which doesn't for me in 0.20-security branch.

> The patch completely breaks metrics2 instrumentation

Are you saying metrics2 are broken in 0.22?

> Port remaining metrics v1 from trunk to branch-0.20-security
> ------------------------------------------------------------
>
>                 Key: HADOOP-7311
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7311
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.203.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: 0.20.204.0
>
>         Attachments: metrics-0.20-security.patch
>
>
> HADOOP-7190 added metrics packages/classes. This is a port from trunk to make them actually work for the security branch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15208-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 06:29:29 2011
Return-Path: <common-issues-return-15208-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AAE676323
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 06:29:29 +0000 (UTC)
Received: (qmail 42472 invoked by uid 500); 20 May 2011 06:29:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42439 invoked by uid 500); 20 May 2011 06:29:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42430 invoked by uid 99); 20 May 2011 06:29:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 06:29:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 06:29:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 65D54D2C28
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 06:28:47 +0000 (UTC)
Date: Fri, 20 May 2011 06:28:47 +0000 (UTC)
From: "Harsh J Chouraria (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1151338537.29793.1305872927413.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1093348183.29034.1305849887377.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7312) core-default.xml lists
 configuration version as 0.21
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J Chouraria reassigned HADOOP-7312:
-----------------------------------------

    Assignee: Harsh J Chouraria

> core-default.xml lists configuration version as 0.21
> ----------------------------------------------------
>
>                 Key: HADOOP-7312
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7312
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Harsh J Chouraria
>            Priority: Minor
>             Fix For: 0.22.0
>
>
> This key was added in HADOOP-6233, though appears unused. I suppose it's somewhat useful to try to diagnose if someone has old versions of core-default.xml on the classpath.
> Either way it should probably be updated to say 0.22 in the branch and 0.23 in trunk.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15209-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 06:43:31 2011
Return-Path: <common-issues-return-15209-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7C909694E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 06:43:31 +0000 (UTC)
Received: (qmail 50814 invoked by uid 500); 20 May 2011 06:43:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50786 invoked by uid 500); 20 May 2011 06:43:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50760 invoked by uid 99); 20 May 2011 06:43:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 06:43:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 06:43:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 630CDD215D
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 06:42:47 +0000 (UTC)
Date: Fri, 20 May 2011 06:42:47 +0000 (UTC)
From: "Harsh J Chouraria (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1667705469.29807.1305873767402.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1093348183.29034.1305849887377.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7312) core-default.xml lists configuration
 version as 0.21
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J Chouraria updated HADOOP-7312:
--------------------------------------

    Attachment: HADOOP-7312-23.r1.diff
                HADOOP-7312-22.r1.diff

The version string also carries the complete version number (0.21.0, for instance) instead of just the major.minor -- perhaps its a good idea to switch to just two ones for avoiding maintaining over time?

> core-default.xml lists configuration version as 0.21
> ----------------------------------------------------
>
>                 Key: HADOOP-7312
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7312
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Harsh J Chouraria
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7312-22.r1.diff, HADOOP-7312-23.r1.diff
>
>
> This key was added in HADOOP-6233, though appears unused. I suppose it's somewhat useful to try to diagnose if someone has old versions of core-default.xml on the classpath.
> Either way it should probably be updated to say 0.22 in the branch and 0.23 in trunk.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15210-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 12:55:29 2011
Return-Path: <common-issues-return-15210-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8FBCE6FDD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 12:55:29 +0000 (UTC)
Received: (qmail 44104 invoked by uid 500); 20 May 2011 12:55:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44080 invoked by uid 500); 20 May 2011 12:55:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44070 invoked by uid 99); 20 May 2011 12:55:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 12:55:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 12:55:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C878DD3466
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 12:54:47 +0000 (UTC)
Date: Fri, 20 May 2011 12:54:47 +0000 (UTC)
From: "Matt Smith (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <813413793.30334.1305896087817.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <889607684.27597.1305830207722.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7307) Remove Dr Who references
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036802#comment-13036802 ] 

Matt Smith commented on HADOOP-7307:
------------------------------------

You gave me hope and then took it away.
Basically, Todd...
Run.

> Remove Dr Who references
> ------------------------
>
>                 Key: HADOOP-7307
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7307
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>             Fix For: 0.22.0
>
>
> I've had feedback from a number of different operators that the "dr who" references that occasionally come out of Hadoop are annoying and confusing. We should replace them with more descriptive error messages, usernames, etc.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15211-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 13:33:31 2011
Return-Path: <common-issues-return-15211-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1C4DB67A4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 13:33:31 +0000 (UTC)
Received: (qmail 8609 invoked by uid 500); 20 May 2011 13:33:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8569 invoked by uid 500); 20 May 2011 13:33:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8557 invoked by uid 99); 20 May 2011 13:33:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 13:33:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 13:33:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B927ED31BE
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 13:32:47 +0000 (UTC)
Date: Fri, 20 May 2011 13:32:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1710210271.30396.1305898367755.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2117896899.24772.1305762047299.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7302) webinterface.private.actions
 should not be in common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036820#comment-13036820 ] 

Hudson commented on HADOOP-7302:
--------------------------------

Integrated in Hadoop-Common-trunk #694 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/694/])
    HADOOP-7302. webinterface.private.actions should be renamed and moved to the MapReduce project. Contributed by Ari Rabkin.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1125221
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/core-default.xml


> webinterface.private.actions should not be in common
> ----------------------------------------------------
>
>                 Key: HADOOP-7302
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7302
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.23.0
>            Reporter: Ari Rabkin
>            Assignee: Ari Rabkin
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7302.patch
>
>
> The comment in -defaults says that webinterface.private.actions applies to both NN and JT. This is wrong. This option is only referenced via the JobTracker. 
> I propose to delete it here, and file a second issue in MAPREDUCE for renaming the option (deprecating the existing name.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15212-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 13:33:31 2011
Return-Path: <common-issues-return-15212-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6BC5467BB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 13:33:31 +0000 (UTC)
Received: (qmail 8844 invoked by uid 500); 20 May 2011 13:33:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8811 invoked by uid 500); 20 May 2011 13:33:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8803 invoked by uid 99); 20 May 2011 13:33:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 13:33:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 13:33:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 23373D31C6
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 13:32:48 +0000 (UTC)
Date: Fri, 20 May 2011 13:32:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <875542021.30403.1305898368141.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6832) Provide a web server plugin that
 uses a static user for the web UI
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036821#comment-13036821 ] 

Hudson commented on HADOOP-6832:
--------------------------------

Integrated in Hadoop-Common-trunk #694 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/694/])
    HADOOP-6832. Add an authentication plugin using a configurable static user
for the web UI. Contributed by Owen O'Malley and Todd Lipcon

cdouglas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1125043
Files : 
* /hadoop/common/trunk/src/java/org/apache/hadoop/http/lib
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/http/lib/package.html
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/http/lib
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/http/lib/TestStaticUserWebFilter.java
* /hadoop/common/trunk/src/java/core-default.xml
* /hadoop/common/trunk/src/java/org/apache/hadoop/http/lib/StaticUserWebFilter.java


> Provide a web server plugin that uses a static user for the web UI
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6832
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6832
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: h-6382.patch, h-6832.patch, h-6832.patch, h-6832.patch, hadoop-6832.txt, static-web-user.patch
>
>
> We need a simple plugin that uses a static user for clusters with security that don't want to authenticate users on the web UI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15213-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 13:33:31 2011
Return-Path: <common-issues-return-15213-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9901867CA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 13:33:31 +0000 (UTC)
Received: (qmail 9077 invoked by uid 500); 20 May 2011 13:33:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8994 invoked by uid 500); 20 May 2011 13:33:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8916 invoked by uid 99); 20 May 2011 13:33:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 13:33:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 13:33:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3DE81D31CA
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 13:32:48 +0000 (UTC)
Date: Fri, 20 May 2011 13:32:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1874743663.30406.1305898368250.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <580585038.27366.1305827567709.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7306) Start metrics system even if
 config files are missing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036822#comment-13036822 ] 

Hudson commented on HADOOP-7306:
--------------------------------

Integrated in Hadoop-Common-trunk #694 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/694/])
    HADOOP-7306. Start metrics system even if config files are missing. Contributed by Luke Lu.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1125216
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/metrics2/impl/TestMetricsConfig.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/metrics2/impl/MetricsConfig.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java


> Start metrics system even if config files are missing
> -----------------------------------------------------
>
>                 Key: HADOOP-7306
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7306
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: metrics
>    Affects Versions: 0.23.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7306-missing-conf-v1.patch
>
>
> Per experience and discussion with HDFS-1922, it seems preferable to treat missing metrics config file as empty/default config, which is more compatible with metrics v1 behavior (the MBeans are always registered.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15214-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 13:33:32 2011
Return-Path: <common-issues-return-15214-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A3E3867EB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 13:33:32 +0000 (UTC)
Received: (qmail 9359 invoked by uid 500); 20 May 2011 13:33:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9317 invoked by uid 500); 20 May 2011 13:33:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9309 invoked by uid 99); 20 May 2011 13:33:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 13:33:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 13:33:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 51015D31CE
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 13:32:48 +0000 (UTC)
Date: Fri, 20 May 2011 13:32:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1645911889.30408.1305898368328.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1008788632.26385.1305807047638.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7305) Eclipse project files are
 incomplete
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036823#comment-13036823 ] 

Hudson commented on HADOOP-7305:
--------------------------------

Integrated in Hadoop-Common-trunk #694 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/694/])
    HADOOP-7305. Eclipse project classpath should include tools.jar from JDK. Contributed by Niels Basjes.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1125051
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/build.xml


> Eclipse project files are incomplete
> ------------------------------------
>
>                 Key: HADOOP-7305
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7305
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Niels Basjes
>            Assignee: Niels Basjes
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7305-2011-05-19.patch
>
>
> After a fresh checkout of hadoop-common I do 'ant compile eclipse'.
> I open eclipse, set ANT_HOME and build the project. 
> At that point the following error appears:
> {quote}
> The type com.sun.javadoc.RootDoc cannot be resolved. It is indirectly referenced from required .class files	ExcludePrivateAnnotationsJDiffDoclet.java	/common/src/java/org/apache/hadoop/classification/tools	line 1	Java Problem
> {quote}
> The solution is to add the "tools.jar" from the JDK to the buildpath/classpath.
> This should be fixed in the build.xml.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15215-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 14:23:30 2011
Return-Path: <common-issues-return-15215-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C13766B9D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 14:23:30 +0000 (UTC)
Received: (qmail 77323 invoked by uid 500); 20 May 2011 14:23:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77280 invoked by uid 500); 20 May 2011 14:23:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77272 invoked by uid 99); 20 May 2011 14:23:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 14:23:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 14:23:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BF11DD349E
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 14:22:47 +0000 (UTC)
Date: Fri, 20 May 2011 14:22:47 +0000 (UTC)
From: "Jeffrey Naisbitt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <467948518.30516.1305901367779.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7314) Add support for throwing
 UnknownHostException when a host doesn't resolve
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Add support for throwing UnknownHostException when a host doesn't resolve
-------------------------------------------------------------------------

                 Key: HADOOP-7314
                 URL: https://issues.apache.org/jira/browse/HADOOP-7314
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Jeffrey Naisbitt
            Assignee: Jeffrey Naisbitt


As part of MAPREDUCE-2489, we need support for having the resolve methods (for DNS mapping) throw UnknownHostExceptions.  (Currently, they hide the exception).  Since the existing 'resolve' method is ultimately used by several other locations/components, I propose we add a new 'resolveValidHosts' method.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15216-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 14:25:30 2011
Return-Path: <common-issues-return-15216-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4908E6C39
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 14:25:30 +0000 (UTC)
Received: (qmail 78928 invoked by uid 500); 20 May 2011 14:25:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78886 invoked by uid 500); 20 May 2011 14:25:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78876 invoked by uid 99); 20 May 2011 14:25:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 14:25:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 14:25:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 60B6AD35C2
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 14:24:47 +0000 (UTC)
Date: Fri, 20 May 2011 14:24:47 +0000 (UTC)
From: "Jeffrey Naisbitt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <506083476.30519.1305901487392.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <467948518.30516.1305901367779.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7314) Add support for throwing
 UnknownHostException when a host doesn't resolve
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jeffrey Naisbitt updated HADOOP-7314:
-------------------------------------

    Attachment: HADOOP-7314.patch

> Add support for throwing UnknownHostException when a host doesn't resolve
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-7314
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7314
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jeffrey Naisbitt
>            Assignee: Jeffrey Naisbitt
>         Attachments: HADOOP-7314.patch
>
>
> As part of MAPREDUCE-2489, we need support for having the resolve methods (for DNS mapping) throw UnknownHostExceptions.  (Currently, they hide the exception).  Since the existing 'resolve' method is ultimately used by several other locations/components, I propose we add a new 'resolveValidHosts' method.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15217-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 14:27:30 2011
Return-Path: <common-issues-return-15217-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A23456CD5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 14:27:30 +0000 (UTC)
Received: (qmail 81108 invoked by uid 500); 20 May 2011 14:27:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81061 invoked by uid 500); 20 May 2011 14:27:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81049 invoked by uid 99); 20 May 2011 14:27:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 14:27:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 14:27:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 78B43D36FC
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 14:26:47 +0000 (UTC)
Date: Fri, 20 May 2011 14:26:47 +0000 (UTC)
From: "Jeffrey Naisbitt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <516823503.30521.1305901607490.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <467948518.30516.1305901367779.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7314) Add support for throwing
 UnknownHostException when a host doesn't resolve
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jeffrey Naisbitt updated HADOOP-7314:
-------------------------------------

    Status: Patch Available  (was: Open)

> Add support for throwing UnknownHostException when a host doesn't resolve
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-7314
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7314
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jeffrey Naisbitt
>            Assignee: Jeffrey Naisbitt
>         Attachments: HADOOP-7314.patch
>
>
> As part of MAPREDUCE-2489, we need support for having the resolve methods (for DNS mapping) throw UnknownHostExceptions.  (Currently, they hide the exception).  Since the existing 'resolve' method is ultimately used by several other locations/components, I propose we add a new 'resolveValidHosts' method.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15218-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 14:29:28 2011
Return-Path: <common-issues-return-15218-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 63E726D6A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 14:29:28 +0000 (UTC)
Received: (qmail 86693 invoked by uid 500); 20 May 2011 14:29:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86663 invoked by uid 500); 20 May 2011 14:29:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86653 invoked by uid 99); 20 May 2011 14:29:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 14:29:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 14:29:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5D72AD379C
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 14:28:47 +0000 (UTC)
Date: Fri, 20 May 2011 14:28:47 +0000 (UTC)
From: "Robert Joseph Evans (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <891600564.30544.1305901727379.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Joseph Evans updated HADOOP-7144:
----------------------------------------

    Attachment: HADOOP-7411-trunk-alpha.patch

I have attached an initial patch pulling in JMXProxyServlet.java from Tomcat 7.0.14.
I pulled over two classes  and part of a third.  

org.apache.tomcat.util.ExceptionUtils - only the package name changed.

org.apache.tomcat.util.modeler.Registry - two methods from this class were moved into JMXProxyServlet getType and convertValue and modified to use mBeanServer from JMXProxyServlet instead of the MBeanServer from Registry

org.apache.catalina.manager.JMXProxyServlet - pacakge name changed.  Registry was removed and replaced with a direct call to get MBeanServer, and two private methods.

There are no tests and no documentation changes right now (Which is why I marked it alpha).  I wanted feedback on it first.

1) is the package OK org.apache.hadoop.jmx?
2) is the URL OK /jmx?
3) where would be an appropriate place to document this?
4) is this even the right thing to do?  I am not sure how standard the format is.  I need to dig into it a bit more to really understand it, because all I have done is a simple port.

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>              Labels: jmx
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7411-trunk-alpha.patch
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15219-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 14:33:31 2011
Return-Path: <common-issues-return-15219-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 649C26E07
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 14:33:31 +0000 (UTC)
Received: (qmail 94365 invoked by uid 500); 20 May 2011 14:33:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94327 invoked by uid 500); 20 May 2011 14:33:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94314 invoked by uid 99); 20 May 2011 14:33:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 14:33:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 14:33:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8C1E1D390A
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 14:32:47 +0000 (UTC)
Date: Fri, 20 May 2011 14:32:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1422465933.30552.1305901967570.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036849#comment-13036849 ] 

Daryn Sharp commented on HADOOP-6605:
-------------------------------------

(btw, I previously attempted to imply my neg was withdrawn, but I just want to make it unambiguously clear.  I really like the idea as evidenced by my jira dupped to this one)

bq. From googling it looked like "jdk1.6.0" was a stable path on Solaris, ie no glob necessary. Maybe someone who uses Solaris can verify, I'm fine punting Solaris to a future change as well.

My only point/concern is that if a jdk1.6.1. or jdk1.6.2 is installed that this implementation would still default to 1.6.0.  What if a 1.6.1, but not 1.6.0, is installed?  Or if only jdk1.7.* is installed and perhaps hadoop works fine with either java 6 or 7?  The detection is of little use to the user, however I don't know the Solaris conventions.  Maybe they have a way to set a "default" java?  Perhaps a symlink somewhere?  I have no strong feelings, and punting would be fine, or maybe a Solaris user could chime in.

bq. These globs should not result in a lot more paths being added. Eg how many paths would you expect "/usr/java/jdk1.6*" to match on most systems? Probably none, a couple would be a lot right? Even if these globs matched 20 paths per above it shouldn't impact performance.

Don't worry, I have no performance concerns.  I fully agree a few stat calls is minuscule in the overall execution.  My mild concern is the maintainability introduced by having to update paths for newer versions of java. However, being a stickler for consistency, I have a little more concern about how updating the paths introduces inconsistent behavior in how the jdk is selected.  Then again, perhaps Linux folks are accustomed to inconsistency? :)

bq. bq. Linux - Same as SunOS. Also, why aren't /usr/java/default and /usr/lib/jvm/default-java checked first?

bq. The code looks for Sun Java 6 first because Hadoop depends on Sun Java 6, ie we don't necessarily want the system default Java.

Only for the sake of discussion: If the goal is be a "good citizen" of the host os, then picking the system default is sufficient.  This approach should eliminate the need to periodically update the script to pick the newer "preferred" java versions, and remove the inconsistency in how the java version is selected between hadoop releases.  It would parallel the Mac/Darwin detection, ie. use the system's default method (if there is one), and if the user wants a different version then they must either change the system default or explicitly set JAVA_HOME.

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch, hadoop-6605-2.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15220-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 14:59:29 2011
Return-Path: <common-issues-return-15220-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0EBC16F35
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 14:59:29 +0000 (UTC)
Received: (qmail 32278 invoked by uid 500); 20 May 2011 14:59:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32235 invoked by uid 500); 20 May 2011 14:59:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32227 invoked by uid 99); 20 May 2011 14:59:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 14:59:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 14:59:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 15612D32EA
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 14:58:48 +0000 (UTC)
Date: Fri, 20 May 2011 14:58:48 +0000 (UTC)
From: "Robert Joseph Evans (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <790465403.30607.1305903528084.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036861#comment-13036861 ] 

Robert Joseph Evans commented on HADOOP-7144:
---------------------------------------------

Just a quick note, I read through the code for the JMXProxy and it is a plain text format, not JSON or XML.

It only supports GET but that can be be modified with a ?set&attr=<KEY>&val=<VALUE> to set an attribute; ?get&attr=<KEY> to get a specific attribute; or ?qry=<QUERY> to output all attributes who's names match the query using the mBeanServer.queryNames method.  the default is ?qry=*:* if nothing else if given.

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>              Labels: jmx
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7411-trunk-alpha.patch
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15221-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 15:31:28 2011
Return-Path: <common-issues-return-15221-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6B7D16C99
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 15:31:28 +0000 (UTC)
Received: (qmail 85304 invoked by uid 500); 20 May 2011 15:31:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85195 invoked by uid 500); 20 May 2011 15:31:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85187 invoked by uid 99); 20 May 2011 15:31:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 15:31:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 15:31:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7D661D3094
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 15:30:47 +0000 (UTC)
Date: Fri, 20 May 2011 15:30:47 +0000 (UTC)
From: "Harsh J Chouraria (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <909186496.30746.1305905447510.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1093348183.29034.1305849887377.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7312) core-default.xml lists configuration
 version as 0.21
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J Chouraria updated HADOOP-7312:
--------------------------------------

    Status: Patch Available  (was: Open)

> core-default.xml lists configuration version as 0.21
> ----------------------------------------------------
>
>                 Key: HADOOP-7312
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7312
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Harsh J Chouraria
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7312-22.r1.diff, HADOOP-7312-23.r1.diff
>
>
> This key was added in HADOOP-6233, though appears unused. I suppose it's somewhat useful to try to diagnose if someone has old versions of core-default.xml on the classpath.
> Either way it should probably be updated to say 0.22 in the branch and 0.23 in trunk.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15222-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 15:45:30 2011
Return-Path: <common-issues-return-15222-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 62D716FCA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 15:45:30 +0000 (UTC)
Received: (qmail 12866 invoked by uid 500); 20 May 2011 15:45:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12838 invoked by uid 500); 20 May 2011 15:45:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12830 invoked by uid 99); 20 May 2011 15:45:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 15:45:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 15:45:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 62E67D37A9
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 15:44:47 +0000 (UTC)
Date: Fri, 20 May 2011 15:44:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1688903594.30856.1305906287401.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1093348183.29034.1305849887377.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7312) core-default.xml lists
 configuration version as 0.21
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036899#comment-13036899 ] 

Hadoop QA commented on HADOOP-7312:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479881/HADOOP-7312-23.r1.diff
  against trunk revision 1125221.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/491//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/491//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/491//console

This message is automatically generated.

> core-default.xml lists configuration version as 0.21
> ----------------------------------------------------
>
>                 Key: HADOOP-7312
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7312
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Harsh J Chouraria
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7312-22.r1.diff, HADOOP-7312-23.r1.diff
>
>
> This key was added in HADOOP-6233, though appears unused. I suppose it's somewhat useful to try to diagnose if someone has old versions of core-default.xml on the classpath.
> Either way it should probably be updated to say 0.22 in the branch and 0.23 in trunk.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15223-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 17:38:30 2011
Return-Path: <common-issues-return-15223-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B8291673A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 17:38:30 +0000 (UTC)
Received: (qmail 78423 invoked by uid 500); 20 May 2011 17:38:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78358 invoked by uid 500); 20 May 2011 17:38:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78214 invoked by uid 99); 20 May 2011 17:38:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 17:38:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 17:38:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 94FDAD3721
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 17:37:47 +0000 (UTC)
Date: Fri, 20 May 2011 17:37:47 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <844152790.31163.1305913067607.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-6255:
------------------------------

    Attachment: HADOOP-6255-branch-0.20-security-10.patch

Fixed 0.20 security branch to use absolute path for setup scripts.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-10.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk-4.patch, HADOOP-6255-common-trunk-5.patch, HADOOP-6255-common-trunk-6.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-6255-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15224-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 17:44:30 2011
Return-Path: <common-issues-return-15224-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4574D6B24
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 17:44:30 +0000 (UTC)
Received: (qmail 93416 invoked by uid 500); 20 May 2011 17:44:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93383 invoked by uid 500); 20 May 2011 17:44:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93375 invoked by uid 99); 20 May 2011 17:44:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 17:44:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 17:44:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2AF02D39D8
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 17:43:49 +0000 (UTC)
Date: Fri, 20 May 2011 17:43:49 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1005561839.31231.1305913429172.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036950#comment-13036950 ] 

Hadoop QA commented on HADOOP-6255:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479934/HADOOP-6255-branch-0.20-security-10.patch
  against trunk revision 1125221.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 1 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/492//console

This message is automatically generated.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-10.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk-4.patch, HADOOP-6255-common-trunk-5.patch, HADOOP-6255-common-trunk-6.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-6255-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15225-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 17:44:32 2011
Return-Path: <common-issues-return-15225-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 645916B38
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 17:44:32 +0000 (UTC)
Received: (qmail 93694 invoked by uid 500); 20 May 2011 17:44:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93656 invoked by uid 500); 20 May 2011 17:44:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93648 invoked by uid 99); 20 May 2011 17:44:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 17:44:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 17:44:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4181CD39DB
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 17:43:49 +0000 (UTC)
Date: Fri, 20 May 2011 17:43:49 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <740076828.31234.1305913429265.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036951#comment-13036951 ] 

Allen Wittenauer commented on HADOOP-6605:
------------------------------------------

My technical objection is that there is a high likelihood of getting this wrong.

For example, we should have a check to make sure that the java we find is actually 1.6.x.  We need a check to make sure that the java we find actually works properly (i.e., I believe it was _18 or so that caused us issues).  We have to make the determination if we want to allow heterogenous JREs.    What if /usr/java/x isn't a Sun Java?  

The wildcard generates a whole spate of problems.  Do we always pick the oldest version?  Do we reverse the regex and grab the newest?  No matter what we do there, this is going to cause major problems with those that want to do staged rollouts of new JREs.

No, the "but you can always set it" is not an excuse for allowing surprising behavior.

...

Again:  This is way too much work to shove into the interactive hadoop commands.  Until this moves out into a one-time setup, my binding -1 will remain.

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch, hadoop-6605-2.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15226-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 17:46:28 2011
Return-Path: <common-issues-return-15226-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BADC265E1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 17:46:28 +0000 (UTC)
Received: (qmail 99769 invoked by uid 500); 20 May 2011 17:46:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99733 invoked by uid 500); 20 May 2011 17:46:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99723 invoked by uid 99); 20 May 2011 17:46:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 17:46:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 17:46:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BE7B8D3B56
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 17:45:47 +0000 (UTC)
Date: Fri, 20 May 2011 17:45:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1678631154.31253.1305913547777.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1145070693.28819.1305845387456.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7311) Port remaining metrics v1 from
 trunk to branch-0.20-security
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036953#comment-13036953 ] 

Luke Lu commented on HADOOP-7311:
---------------------------------

bq. I am not doing anything with metrix2 except trying not to break it.

The patch completely removes the instrumentation and tests of metrics v2 in RPC and HDFS.

bq. My motivation is to make the original metrics work, which doesn't for me in 0.20-security branch.

Can you tell me what's missing for you in metrics v2, except for Gangalia? I can help you porting...

bq. Are you saying metrics2 are broken in 0.22?

metrics2 is never in 0.22. It's in 0.20.2xx and trunk (targeting 0.23)

Last September, after we deployed 0.20.200 internally, I asked the community about the evolution paths of metrics on [the mailing list|http://goo.gl/Rjb1], and the people who care to answer [chose #2|http://goo.gl/NLLs], which was exactly what we did for 0.20.203 and trunk.

Your goal now seems to be #3, but your patch doesn't do that. I anticipated possibility of #3 or #3a and put all metrics instrumentation behind interfaces in 0.20.2xx, so it's at least possible without changing the instrumentation code (besides the updater or metrics source implementation). As I pointed out in the original post in the mailing lists, there are down sides for #3, I'd really like to know the reasons for that. I can certainly help you implementing #3, if you can convince the community that it's really needed for 0.20.2xx releases.


> Port remaining metrics v1 from trunk to branch-0.20-security
> ------------------------------------------------------------
>
>                 Key: HADOOP-7311
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7311
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.203.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: 0.20.204.0
>
>         Attachments: metrics-0.20-security.patch
>
>
> HADOOP-7190 added metrics packages/classes. This is a port from trunk to make them actually work for the security branch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15227-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 18:00:29 2011
Return-Path: <common-issues-return-15227-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6AC026BF9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 18:00:29 +0000 (UTC)
Received: (qmail 21478 invoked by uid 500); 20 May 2011 18:00:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21453 invoked by uid 500); 20 May 2011 18:00:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21445 invoked by uid 99); 20 May 2011 18:00:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 18:00:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 18:00:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 61B4FD30D9
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 17:59:48 +0000 (UTC)
Date: Fri, 20 May 2011 17:59:48 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <953809656.31308.1305914388396.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036963#comment-13036963 ] 

Luke Lu commented on HADOOP-7144:
---------------------------------

bq. is the package OK org.apache.hadoop.jmx? is the URL OK /jmx?

lgtm.

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>              Labels: jmx
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7411-trunk-alpha.patch
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15228-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 18:00:31 2011
Return-Path: <common-issues-return-15228-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E78E96C40
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 18:00:30 +0000 (UTC)
Received: (qmail 22361 invoked by uid 500); 20 May 2011 18:00:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22330 invoked by uid 500); 20 May 2011 18:00:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22322 invoked by uid 99); 20 May 2011 18:00:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 18:00:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 18:00:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B5824D30C9
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 17:59:47 +0000 (UTC)
Date: Fri, 20 May 2011 17:59:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1634978174.31294.1305914387740.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036962#comment-13036962 ] 

Luke Lu commented on HADOOP-7144:
---------------------------------

bq. is the package OK org.apache.hadoop.jmx? is the URL OK /jmx?
bq. where would be an appropriate place to document this?

package-info.java for javadoc would be a start.

bq. Is this even the right thing to do? I am not sure how standard the format is. I need to dig into it a bit more to really understand it, because all I have done is a simple port.

We need read-only access and have it return JSON, something like this:

{noformat}
  [ 
    { 
      "bean": "Hadoop:service=...",
      "attrs": {
        "attr1": "value1",
        ...
        "attrn": "valuen"
      }
    },
    ...
  ]
{noformat}

We also need a way to return a pre-configured set of attributes.

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>              Labels: jmx
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7411-trunk-alpha.patch
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15229-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 19:13:31 2011
Return-Path: <common-issues-return-15229-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F19C26A0F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 19:13:30 +0000 (UTC)
Received: (qmail 46129 invoked by uid 500); 20 May 2011 19:13:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46081 invoked by uid 500); 20 May 2011 19:13:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46063 invoked by uid 99); 20 May 2011 19:13:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 19:13:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 19:13:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0645DD3E6D
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 19:12:48 +0000 (UTC)
Date: Fri, 20 May 2011 19:12:48 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2126435900.31532.1305918768022.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7269) S3 Native should allow
 customizable file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037010#comment-13037010 ] 

Suresh Srinivas commented on HADOOP-7269:
-----------------------------------------

Sorry, I think I was looking at *001.diff. 

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: HADOOP-7269-S3-metadata-001.diff, HADOOP-7269-S3-metadata-002.diff, HADOOP-7269-S3-metadata-003.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15230-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 19:27:29 2011
Return-Path: <common-issues-return-15230-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 327986254
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 19:27:29 +0000 (UTC)
Received: (qmail 64475 invoked by uid 500); 20 May 2011 19:27:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64441 invoked by uid 500); 20 May 2011 19:27:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64431 invoked by uid 99); 20 May 2011 19:27:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 19:27:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 19:27:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3928CD33E2
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 19:26:48 +0000 (UTC)
Date: Fri, 20 May 2011 19:26:48 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1393491250.31596.1305919608231.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-6671:
---------------------------------------

    Attachment: HADOOP-6671b.patch

Todd,

Thanks for the fast response and detailed review.

I've taken care of all your comments except the one regarding the tools.jar (still trying to figure out, the Ant <javah> is not finding tools.jar (for some reason JRE is being used, thus my current hack is still needed).

The following maven options can be used to alter the build/tests:

* -DskipTests
* -DskipDocs
* -DcompileNative
* -Dtest=<TESTCLASS>



> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>         Attachments: HADOOP-6671.patch, HADOOP-6671b.patch, build.png, hadoop-commons-maven.patch, mvn-layout.sh, mvn-layout.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15231-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 19:42:31 2011
Return-Path: <common-issues-return-15231-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8A5E46AEA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 19:42:31 +0000 (UTC)
Received: (qmail 87522 invoked by uid 500); 20 May 2011 19:42:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87481 invoked by uid 500); 20 May 2011 19:42:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87446 invoked by uid 99); 20 May 2011 19:42:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 19:42:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 19:42:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 124A6D3A71
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 19:41:48 +0000 (UTC)
Date: Fri, 20 May 2011 19:41:48 +0000 (UTC)
From: "stack (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <396511573.31658.1305920508071.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7315) Add dynamic config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Add dynamic config
------------------

                 Key: HADOOP-7315
                 URL: https://issues.apache.org/jira/browse/HADOOP-7315
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: stack


I'm sure this issue exists already, at least as part of the discussion around making online schema edits possible, but no hard this having its own issue.  Ted started a conversation on this topic up on dev and Todd suggested we lookd at how Hadoop did it over in HADOOP-7001

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15232-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 20:47:28 2011
Return-Path: <common-issues-return-15232-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 791E861E3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 20:47:28 +0000 (UTC)
Received: (qmail 13302 invoked by uid 500); 20 May 2011 20:47:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13242 invoked by uid 500); 20 May 2011 20:47:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13234 invoked by uid 99); 20 May 2011 20:47:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 20:47:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 20:47:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9193AD321A
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 20:46:47 +0000 (UTC)
Date: Fri, 20 May 2011 20:46:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <795544138.31863.1305924407593.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1659614701.18791.1297804257612.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7146) RPC server leaks file descriptors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7146:
--------------------------------

    Attachment: hadoop-7146.txt

Couldn't find a way to do the test with MXBeans. This patch just uses /proc to see how many FDs are open, and uses junit's Assume support so it won't run on systems where that's not available.

> RPC server leaks file descriptors
> ---------------------------------
>
>                 Key: HADOOP-7146
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7146
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: fd-leak.txt, hadoop-7146.txt
>
>
> Both the Listener and Responder thread call Selector.open but don't have a matching .close(). This causes a leak of anonymous pipes. Not a big deal because people rarely close and re-open servers, but worth fixing.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15233-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 20:48:31 2011
Return-Path: <common-issues-return-15233-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 47D1667F0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 20:48:31 +0000 (UTC)
Received: (qmail 14057 invoked by uid 500); 20 May 2011 20:48:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14027 invoked by uid 500); 20 May 2011 20:48:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14019 invoked by uid 99); 20 May 2011 20:48:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 20:48:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 20:48:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 27EF4D3337
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 20:47:48 +0000 (UTC)
Date: Fri, 20 May 2011 20:47:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1293400528.31873.1305924468160.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1659614701.18791.1297804257612.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7146) RPC server leaks file descriptors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7146:
--------------------------------

    Fix Version/s: 0.22.0
           Status: Patch Available  (was: Open)

> RPC server leaks file descriptors
> ---------------------------------
>
>                 Key: HADOOP-7146
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7146
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: fd-leak.txt, hadoop-7146.txt
>
>
> Both the Listener and Responder thread call Selector.open but don't have a matching .close(). This causes a leak of anonymous pipes. Not a big deal because people rarely close and re-open servers, but worth fixing.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15234-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 20:52:31 2011
Return-Path: <common-issues-return-15234-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5483A6A7E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 20:52:31 +0000 (UTC)
Received: (qmail 18785 invoked by uid 500); 20 May 2011 20:52:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18751 invoked by uid 500); 20 May 2011 20:52:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18739 invoked by uid 99); 20 May 2011 20:52:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 20:52:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 20:52:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B8B49D35B1
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 20:51:47 +0000 (UTC)
Date: Fri, 20 May 2011 20:51:47 +0000 (UTC)
From: "Robert Joseph Evans (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <350711713.31900.1305924707753.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037077#comment-13037077 ] 

Robert Joseph Evans commented on HADOOP-7144:
---------------------------------------------

Ok because we are going to go with JSON, and not use JMXProxyServlet more or less unchanged, I think I will take the core of it, but I will probably rename it so there is no confusion with the Tomcat one.

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>              Labels: jmx
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7411-trunk-alpha.patch
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15235-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 21:35:30 2011
Return-Path: <common-issues-return-15235-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D4212672A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 21:35:30 +0000 (UTC)
Received: (qmail 89395 invoked by uid 500); 20 May 2011 21:35:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89364 invoked by uid 500); 20 May 2011 21:35:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89356 invoked by uid 99); 20 May 2011 21:35:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 21:35:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 21:35:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AFDF5D447D
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 21:34:47 +0000 (UTC)
Date: Fri, 20 May 2011 21:34:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1576731972.32004.1305927287390.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1659614701.18791.1297804257612.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7146) RPC server leaks file descriptors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037099#comment-13037099 ] 

Aaron T. Myers commented on HADOOP-7146:
----------------------------------------

+1, patch looks great, Todd.

I also ran {{TestDFSStorageStateRecovery}} with this patch applied in the course of debugging HDFS-1884, and can confirm that this patch fixes that test as well.

> RPC server leaks file descriptors
> ---------------------------------
>
>                 Key: HADOOP-7146
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7146
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: fd-leak.txt, hadoop-7146.txt
>
>
> Both the Listener and Responder thread call Selector.open but don't have a matching .close(). This causes a leak of anonymous pipes. Not a big deal because people rarely close and re-open servers, but worth fixing.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15236-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 22:01:32 2011
Return-Path: <common-issues-return-15236-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E29076D4F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 22:01:31 +0000 (UTC)
Received: (qmail 21804 invoked by uid 500); 20 May 2011 22:01:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21766 invoked by uid 500); 20 May 2011 22:01:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21753 invoked by uid 99); 20 May 2011 22:01:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 22:01:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 22:01:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DC9B7D4A8C
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 22:00:48 +0000 (UTC)
Date: Fri, 20 May 2011 22:00:48 +0000 (UTC)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1878512825.32072.1305928848900.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037110#comment-13037110 ] 

Chris Douglas commented on HADOOP-6605:
---------------------------------------

This struck me as an obvious win, but I'm starting to see Allen's point. Incomplete detection may save some users a couple minutes, but it will cause weird bugs in other systems. The globbing/non-Sun JVM cases are especially salient. The benefit is clear to anyone who's set this variable dozens (if not hundreds) of times, but the potential cost is more subtle. As automatic behavior, this is trying to be too clever for a nominal benefit.

That said, the MacOS tool seems designed to accomplish _exactly_ this, and I'd have no problem if it were to show up in the script. If it stops being a reliable way of setting {{JAVA_HOME}}, we'll just fix it. If there are similar tools in Solaris/Linux (doesn't {{/etc/alternatives}} support similar functionality?), we can make use of them.

> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch, hadoop-6605-2.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15237-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 23:32:28 2011
Return-Path: <common-issues-return-15237-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9E399633A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 23:32:28 +0000 (UTC)
Received: (qmail 16067 invoked by uid 500); 20 May 2011 23:32:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16006 invoked by uid 500); 20 May 2011 23:32:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15998 invoked by uid 99); 20 May 2011 23:32:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 23:32:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 23:32:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8955ED4FF9
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 23:31:47 +0000 (UTC)
Date: Fri, 20 May 2011 23:31:47 +0000 (UTC)
From: "Jonathan Hsieh (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <531829213.32234.1305934307559.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7316) Add javadoc comments to
 FSDataInputStream.getWrappedStream public method.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Add javadoc comments to FSDataInputStream.getWrappedStream public method.
-------------------------------------------------------------------------

                 Key: HADOOP-7316
                 URL: https://issues.apache.org/jira/browse/HADOOP-7316
             Project: Hadoop Common
          Issue Type: New Feature
            Reporter: Jonathan Hsieh
             Fix For: 0.23.0


This is a method made public for testing.  In comments in HADOOP-7301 after commit, adding javadoc comments was requested.  This is a follow up jira to address it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15238-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 23:46:28 2011
Return-Path: <common-issues-return-15238-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 773F36F66
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 23:46:28 +0000 (UTC)
Received: (qmail 37426 invoked by uid 500); 20 May 2011 23:46:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37402 invoked by uid 500); 20 May 2011 23:46:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37394 invoked by uid 99); 20 May 2011 23:46:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 23:46:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 23:46:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8858ED433D
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 23:45:47 +0000 (UTC)
Date: Fri, 20 May 2011 23:45:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1173363105.32261.1305935147555.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <187742091.8722.1305246647362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7287) Configuration deprecation
 mechanism doesn't work properly for GenericOptionsParser/Tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037162#comment-13037162 ] 

Todd Lipcon commented on HADOOP-7287:
-------------------------------------

I tried the manual test case described above in this JIRA -- using {{./bin/hadoop fs -Ddfs.blocksize=4096 -touchz f}} against a local HDFS cluster. The config does not appear to take effect. So, apparently the test cases above are not good enough representations of what really happens for this use case.

> Configuration deprecation mechanism doesn't work properly for GenericOptionsParser/Tools
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7287
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7287
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Aaron T. Myers
>            Priority: Blocker
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: hadoop-7287-still-broken.txt, hadoop-7287-testcase.txt, hadoop-7287-trunk.0.patch, hadoop-7287-trunk.1.patch
>
>
> For example, you can't use -D options on the "hadoop fs" command line in order to specify the deprecated names of configuration options. The issue is that the ordering is:
> - JVM starts
> - GenericOptionsParser creates a Configuration object and calls set() for each of the options specified on command line
> - DistributedFileSystem or other class eventually instantiates HdfsConfiguration which adds the deprecations
> - Some class calls conf.get("new key") and sees the default instead of the version set on the command line

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15239-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 20 23:50:29 2011
Return-Path: <common-issues-return-15239-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 340536F12
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 20 May 2011 23:50:29 +0000 (UTC)
Received: (qmail 47090 invoked by uid 500); 20 May 2011 23:50:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47036 invoked by uid 500); 20 May 2011 23:50:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46800 invoked by uid 99); 20 May 2011 23:50:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 23:50:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 May 2011 23:50:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BCE3FD445B
	for <common-issues@hadoop.apache.org>; Fri, 20 May 2011 23:49:47 +0000 (UTC)
Date: Fri, 20 May 2011 23:49:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1931082294.32276.1305935387770.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7317) RPC.stopProxy doesn't actually close
 proxy
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

RPC.stopProxy doesn't actually close proxy
------------------------------------------

                 Key: HADOOP-7317
                 URL: https://issues.apache.org/jira/browse/HADOOP-7317
             Project: Hadoop Common
          Issue Type: Bug
          Components: ipc
    Affects Versions: 0.22.0
            Reporter: Todd Lipcon
             Fix For: 0.22.0


Discovered while investigating HDFS-1965, it turns out that the reference-counting done in WritableRpcEngine.ClientCache doesn't map one-to-one with open TCP connections. This means that it's easy to accidentally leave TCP connections open longer than expected so long as the client has any other connections open at all.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15240-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 21 00:22:30 2011
Return-Path: <common-issues-return-15240-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C7A256FD0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 21 May 2011 00:22:30 +0000 (UTC)
Received: (qmail 78157 invoked by uid 500); 21 May 2011 00:22:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78048 invoked by uid 500); 21 May 2011 00:22:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78039 invoked by uid 99); 21 May 2011 00:22:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 May 2011 00:22:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 May 2011 00:22:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C2772D4F24
	for <common-issues@hadoop.apache.org>; Sat, 21 May 2011 00:21:47 +0000 (UTC)
Date: Sat, 21 May 2011 00:21:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <563441287.32435.1305937307793.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <187742091.8722.1305246647362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7287) Configuration deprecation
 mechanism doesn't work properly for GenericOptionsParser/Tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037193#comment-13037193 ] 

Todd Lipcon commented on HADOOP-7287:
-------------------------------------

With HDFS-1932 committed the above manual test now passes.

But, one more issue I think before we can commit this: in this test case it doesn't print any warning about use of the deprecated flag. Would that be easy to address or is it better to do in another jira?

> Configuration deprecation mechanism doesn't work properly for GenericOptionsParser/Tools
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7287
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7287
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Aaron T. Myers
>            Priority: Blocker
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: hadoop-7287-still-broken.txt, hadoop-7287-testcase.txt, hadoop-7287-trunk.0.patch, hadoop-7287-trunk.1.patch
>
>
> For example, you can't use -D options on the "hadoop fs" command line in order to specify the deprecated names of configuration options. The issue is that the ordering is:
> - JVM starts
> - GenericOptionsParser creates a Configuration object and calls set() for each of the options specified on command line
> - DistributedFileSystem or other class eventually instantiates HdfsConfiguration which adds the deprecations
> - Some class calls conf.get("new key") and sees the default instead of the version set on the command line

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15241-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 21 00:50:31 2011
Return-Path: <common-issues-return-15241-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2F67962D1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 21 May 2011 00:50:31 +0000 (UTC)
Received: (qmail 97701 invoked by uid 500); 21 May 2011 00:50:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97666 invoked by uid 500); 21 May 2011 00:50:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97652 invoked by uid 99); 21 May 2011 00:50:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 May 2011 00:50:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 May 2011 00:50:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0F189D4661
	for <common-issues@hadoop.apache.org>; Sat, 21 May 2011 00:49:48 +0000 (UTC)
Date: Sat, 21 May 2011 00:49:48 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2106254479.32564.1305938988058.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <187742091.8722.1305246647362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7287) Configuration deprecation mechanism
 doesn't work properly for GenericOptionsParser/Tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7287:
-----------------------------------

    Attachment: hadoop-7287-trunk.2.patch

Updated patch which prints a warning message when the deprecated key is accessed.

> Configuration deprecation mechanism doesn't work properly for GenericOptionsParser/Tools
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7287
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7287
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Aaron T. Myers
>            Priority: Blocker
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: hadoop-7287-still-broken.txt, hadoop-7287-testcase.txt, hadoop-7287-trunk.0.patch, hadoop-7287-trunk.1.patch, hadoop-7287-trunk.2.patch
>
>
> For example, you can't use -D options on the "hadoop fs" command line in order to specify the deprecated names of configuration options. The issue is that the ordering is:
> - JVM starts
> - GenericOptionsParser creates a Configuration object and calls set() for each of the options specified on command line
> - DistributedFileSystem or other class eventually instantiates HdfsConfiguration which adds the deprecations
> - Some class calls conf.get("new key") and sees the default instead of the version set on the command line

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15242-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 21 01:54:30 2011
Return-Path: <common-issues-return-15242-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CBFFC6EED
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 21 May 2011 01:54:30 +0000 (UTC)
Received: (qmail 30595 invoked by uid 500); 21 May 2011 01:54:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30544 invoked by uid 500); 21 May 2011 01:54:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30536 invoked by uid 99); 21 May 2011 01:54:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 May 2011 01:54:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 May 2011 01:54:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7588DD40F1
	for <common-issues@hadoop.apache.org>; Sat, 21 May 2011 01:53:47 +0000 (UTC)
Date: Sat, 21 May 2011 01:53:47 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <594900281.32665.1305942827478.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1185800002.15672.1298625818523.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7156) getpwuid_r is not thread-safe on
 RHEL6
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037223#comment-13037223 ] 

Allen Wittenauer commented on HADOOP-7156:
------------------------------------------

Red Hat has released a fix for this as part of RHEL 6.1: http://rhn.redhat.com/errata/RHSA-2011-0560.html .

> getpwuid_r is not thread-safe on RHEL6
> --------------------------------------
>
>                 Key: HADOOP-7156
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7156
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>         Environment: RHEL 6.0 "Santiago"
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt, hadoop-7156.txt
>
>
> Due to the following bug in SSSD, functions like getpwuid_r are not thread-safe in RHEL 6.0 if sssd is specified in /etc/nsswitch.conf (as it is by default):
> https://fedorahosted.org/sssd/ticket/640
> This causes many fetch failures in the case that the native libraries are available, since the SecureIO functions call getpwuid_r as part of fstat. By enabling -Xcheck:jni I get the following trace on JVM crash:
> *** glibc detected *** /mnt/toolchain/JDK6u20-64bit/bin/java: free(): invalid pointer: 0x0000003575741d23 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3575675676]
> /lib64/libnss_sss.so.2(_nss_sss_getpwuid_r+0x11b)[0x7fe716cb42cb]
> /lib64/libc.so.6(getpwuid_r+0xdd)[0x35756a5dfd]

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15243-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 21 12:41:42 2011
Return-Path: <common-issues-return-15243-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6428649EE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 21 May 2011 12:41:42 +0000 (UTC)
Received: (qmail 31814 invoked by uid 500); 21 May 2011 12:41:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31228 invoked by uid 500); 21 May 2011 12:41:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30932 invoked by uid 99); 21 May 2011 12:41:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 May 2011 12:41:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 May 2011 12:41:39 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7ABBAD5BFE
	for <common-issues@hadoop.apache.org>; Sat, 21 May 2011 12:40:05 +0000 (UTC)
Date: Sat, 21 May 2011 12:40:05 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1864133464.33353.1305981605498.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6846) Scripts for building Hadoop 0.22.0
 release
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037362#comment-13037362 ] 

Hudson commented on HADOOP-6846:
--------------------------------

Integrated in Hadoop-Hdfs-trunk #673 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/673/])
    

> Scripts for building Hadoop 0.22.0 release
> ------------------------------------------
>
>                 Key: HADOOP-6846
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6846
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6846.patch, release-scripts.tar.gz
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15244-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 21 12:42:00 2011
Return-Path: <common-issues-return-15244-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 44DD64A5C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 21 May 2011 12:42:00 +0000 (UTC)
Received: (qmail 33604 invoked by uid 500); 21 May 2011 12:42:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33569 invoked by uid 500); 21 May 2011 12:42:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33561 invoked by uid 99); 21 May 2011 12:42:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 May 2011 12:42:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 May 2011 12:41:59 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4543FD5C32
	for <common-issues@hadoop.apache.org>; Sat, 21 May 2011 12:40:07 +0000 (UTC)
Date: Sat, 21 May 2011 12:40:07 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <79119287.33400.1305981607280.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <291388771.11207.1305319187380.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7291) Update Hudson job not to run
 test-contrib
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037371#comment-13037371 ] 

Hudson commented on HADOOP-7291:
--------------------------------

Integrated in Hadoop-Hdfs-trunk #673 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/673/])
    

> Update Hudson job not to run test-contrib
> -----------------------------------------
>
>                 Key: HADOOP-7291
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7291
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7291-2.patch, HADOOP-7291-3.patch, HADOOP-7291.patch
>
>
> The test-contrib target was removed in HADOOP-7137, which causes the Hudson job to fail. The build file doesn't execute test-contrib so I suspect the Hudson job needs to be updated to not call ant with the test-contrib target.  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15245-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 21 18:47:29 2011
Return-Path: <common-issues-return-15245-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E13D24ACC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 21 May 2011 18:47:28 +0000 (UTC)
Received: (qmail 97794 invoked by uid 500); 21 May 2011 18:47:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97692 invoked by uid 500); 21 May 2011 18:47:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97683 invoked by uid 99); 21 May 2011 18:47:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 May 2011 18:47:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 May 2011 18:47:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BBF7BD57B0
	for <common-issues@hadoop.apache.org>; Sat, 21 May 2011 18:46:47 +0000 (UTC)
Date: Sat, 21 May 2011 18:46:47 +0000 (UTC)
From: "Bharath Mundlapudi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1686068557.33696.1306003607766.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6605) Add JAVA_HOME detection to
 hadoop-config
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037435#comment-13037435 ] 

Bharath Mundlapudi commented on HADOOP-6605:
--------------------------------------------

Most of the time, i do side-by-side jdk installations to see which jdk is better. Many times, default jdk location is not preferred, Esp. as Allen mentioned sometimes /usr/bin/java isn't Sun Java.

IMO, having .hadooprc file might be useful thing. Again, this file should be part of the bundle. This is kind of creating virtual environment for hadoop installation. This will solve multiple version problem also.  

Sometimes, it is hard to know which jdk binary hadoop is using, part of the reason is one can set JAVA_HOME at multiple locations like shell, hadoop-env.sh etc Having consistent location for setting this also helpful. One has to do ps cmd to figure out which location java is running from.
 
Also, having a helper script like checkhadoopenv.sh should let the user know its current virtual environment. This will be useful for debugging purpose.

I think, we should have one consistent way to set environment for hadoop this avoids user confusion.

Main point: set all the environment variables required for hadoop in .hadooprc file defaulting to certain values. User can edit this file depending on his/her case may be.

Defaulting env can be a tricky thing if user didn't set in .hadooprc. If this happens, we should let user know that he/she should update .hadooprc file for the missing env variables.

This file (.hadooprc) is user editable. We should document and make it the only place users can tinker their environment. This avoids most of the confusion. Do you agree?




> Add JAVA_HOME detection to hadoop-config
> ----------------------------------------
>
>                 Key: HADOOP-6605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6605
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chad Metcalf
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6605.patch, hadoop-6605-1.patch, hadoop-6605-2.patch
>
>
> The commands that source hadoop-config.sh currently bail with an error if JAVA_HOME is not set. Let's detect JAVA_HOME (from a list of locations on various OS types) if JAVA_HOME is not already set by hadoop-env.sh or the environment. This way users don't have to manually configure it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15246-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 21 19:31:31 2011
Return-Path: <common-issues-return-15246-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EE0354A9D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 21 May 2011 19:31:30 +0000 (UTC)
Received: (qmail 25101 invoked by uid 500); 21 May 2011 19:31:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25069 invoked by uid 500); 21 May 2011 19:31:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25061 invoked by uid 99); 21 May 2011 19:31:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 May 2011 19:31:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 May 2011 19:31:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B1E62D5DE0
	for <common-issues@hadoop.apache.org>; Sat, 21 May 2011 19:30:47 +0000 (UTC)
Date: Sat, 21 May 2011 19:30:47 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1852460677.33713.1306006247725.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <889607684.27597.1305830207722.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7307) Remove Dr Who references
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037437#comment-13037437 ] 

Steve Loughran commented on HADOOP-7307:
----------------------------------------

Todd, your mention of the lack of internet access and their objections to references to Dr Who appearing in their logs tells me that your customers are the Daleks, whose Datacenter-7 is a 150 Exabyte facility on the dark side of the moon. I know this as we had to deal with the hardware issues of heat dissipation in servers deployed in a vacuum, and the logistics of a facility designed to be maintained by three-wheeled lifeforms which lack the ability to handle hot-swap disks or the little clips on CAT-5 cables. 

I propose we meet the needs of these valued customers and also the Cybermen by making the name of the Time Lord configurable. They can set {{hadoop.timelord=The Master}} to their hadoop-site file, and be happy.

Personally, I'd focus on the more pressing issue, namely that the Faster Than Light network backbone really confuses the Job Tracker, as it is getting completion notifications coming before the work has actually been scheduled...



> Remove Dr Who references
> ------------------------
>
>                 Key: HADOOP-7307
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7307
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>             Fix For: 0.22.0
>
>
> I've had feedback from a number of different operators that the "dr who" references that occasionally come out of Hadoop are annoying and confusing. We should replace them with more descriptive error messages, usernames, etc.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15247-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 21 21:43:28 2011
Return-Path: <common-issues-return-15247-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1F59A422B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 21 May 2011 21:43:28 +0000 (UTC)
Received: (qmail 16129 invoked by uid 500); 21 May 2011 21:43:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16085 invoked by uid 500); 21 May 2011 21:43:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16077 invoked by uid 99); 21 May 2011 21:43:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 May 2011 21:43:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 May 2011 21:43:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6580ED6FF3
	for <common-issues@hadoop.apache.org>; Sat, 21 May 2011 21:42:47 +0000 (UTC)
Date: Sat, 21 May 2011 21:42:47 +0000 (UTC)
From: "Bharath Mundlapudi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1834598058.33825.1306014167412.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <51041940.28155.1305838907965.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7310) Trash location needs to be
 revisited
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037460#comment-13037460 ] 

Bharath Mundlapudi commented on HADOOP-7310:
--------------------------------------------

+1 to solution A.

IMO, Trash is a system wide abstract. Placing /trash/username is cleaner and permissions are honored like USERA can't delete/peek USERB's trash content.


> Trash location needs to be revisited
> ------------------------------------
>
>                 Key: HADOOP-7310
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7310
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15249-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun May 22 05:15:36 2011
Return-Path: <common-issues-return-15249-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1F73A6BE8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 22 May 2011 05:15:36 +0000 (UTC)
Received: (qmail 97875 invoked by uid 500); 22 May 2011 05:15:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97591 invoked by uid 500); 22 May 2011 05:15:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96490 invoked by uid 99); 22 May 2011 05:15:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 05:15:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 05:15:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 919B4D6F8B
	for <common-issues@hadoop.apache.org>; Sun, 22 May 2011 05:14:47 +0000 (UTC)
Date: Sun, 22 May 2011 05:14:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1212537303.34070.1306041287593.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7318) MD5Hash factory should reset the
 digester it returns
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

MD5Hash factory should reset the digester it returns
----------------------------------------------------

                 Key: HADOOP-7318
                 URL: https://issues.apache.org/jira/browse/HADOOP-7318
             Project: Hadoop Common
          Issue Type: Bug
          Components: io
    Affects Versions: 0.22.0
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
            Priority: Critical
             Fix For: 0.22.0
         Attachments: hadoop-7318.txt

Currently the getDigest() method in MD5Hash does not reset the digester it returns. Since it's a thread-local, this means that a previous aborted usage of the same digester could leave some state around. For example, if the secondary namenode receives an IOException while transfering the image, and does another image transfer with the same thread, it will think it has received an invalid digest.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15248-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun May 22 05:15:36 2011
Return-Path: <common-issues-return-15248-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 57E1D6BF9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 22 May 2011 05:15:36 +0000 (UTC)
Received: (qmail 97949 invoked by uid 500); 22 May 2011 05:15:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97341 invoked by uid 500); 22 May 2011 05:15:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96343 invoked by uid 99); 22 May 2011 05:15:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 05:15:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 05:15:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AB1C8D6F8F
	for <common-issues@hadoop.apache.org>; Sun, 22 May 2011 05:14:47 +0000 (UTC)
Date: Sun, 22 May 2011 05:14:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1088369686.34073.1306041287697.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1212537303.34070.1306041287593.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7318) MD5Hash factory should reset the
 digester it returns
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7318:
--------------------------------

    Attachment: hadoop-7318.txt

> MD5Hash factory should reset the digester it returns
> ----------------------------------------------------
>
>                 Key: HADOOP-7318
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7318
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7318.txt
>
>
> Currently the getDigest() method in MD5Hash does not reset the digester it returns. Since it's a thread-local, this means that a previous aborted usage of the same digester could leave some state around. For example, if the secondary namenode receives an IOException while transfering the image, and does another image transfer with the same thread, it will think it has received an invalid digest.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15250-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun May 22 05:15:36 2011
Return-Path: <common-issues-return-15250-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AB8226C0B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 22 May 2011 05:15:36 +0000 (UTC)
Received: (qmail 97933 invoked by uid 500); 22 May 2011 05:15:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97597 invoked by uid 500); 22 May 2011 05:15:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96396 invoked by uid 99); 22 May 2011 05:15:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 05:15:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 05:15:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BE2E4D6F91
	for <common-issues@hadoop.apache.org>; Sun, 22 May 2011 05:14:47 +0000 (UTC)
Date: Sun, 22 May 2011 05:14:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <383352387.34075.1306041287775.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1212537303.34070.1306041287593.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7318) MD5Hash factory should reset the
 digester it returns
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7318:
--------------------------------

    Status: Patch Available  (was: Open)

> MD5Hash factory should reset the digester it returns
> ----------------------------------------------------
>
>                 Key: HADOOP-7318
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7318
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7318.txt
>
>
> Currently the getDigest() method in MD5Hash does not reset the digester it returns. Since it's a thread-local, this means that a previous aborted usage of the same digester could leave some state around. For example, if the secondary namenode receives an IOException while transfering the image, and does another image transfer with the same thread, it will think it has received an invalid digest.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15251-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun May 22 05:17:30 2011
Return-Path: <common-issues-return-15251-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3A2B463A3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 22 May 2011 05:17:30 +0000 (UTC)
Received: (qmail 99946 invoked by uid 500); 22 May 2011 05:17:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99837 invoked by uid 500); 22 May 2011 05:17:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99827 invoked by uid 99); 22 May 2011 05:17:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 05:17:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 05:17:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 64F8DD6007
	for <common-issues@hadoop.apache.org>; Sun, 22 May 2011 05:16:47 +0000 (UTC)
Date: Sun, 22 May 2011 05:16:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1998997027.34081.1306041407410.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1212537303.34070.1306041287593.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7318) MD5Hash factory should reset the
 digester it returns
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7318:
--------------------------------

    Attachment: hadoop-7318.txt

oops, fixed test case to use constant throughout.

> MD5Hash factory should reset the digester it returns
> ----------------------------------------------------
>
>                 Key: HADOOP-7318
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7318
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7318.txt, hadoop-7318.txt
>
>
> Currently the getDigest() method in MD5Hash does not reset the digester it returns. Since it's a thread-local, this means that a previous aborted usage of the same digester could leave some state around. For example, if the secondary namenode receives an IOException while transfering the image, and does another image transfer with the same thread, it will think it has received an invalid digest.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15252-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun May 22 05:53:32 2011
Return-Path: <common-issues-return-15252-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1E5FD69AB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 22 May 2011 05:53:32 +0000 (UTC)
Received: (qmail 7066 invoked by uid 500); 22 May 2011 05:53:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6829 invoked by uid 500); 22 May 2011 05:53:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6817 invoked by uid 99); 22 May 2011 05:53:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 05:53:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 05:53:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B6F05D645B
	for <common-issues@hadoop.apache.org>; Sun, 22 May 2011 05:52:47 +0000 (UTC)
Date: Sun, 22 May 2011 05:52:47 +0000 (UTC)
From: "Jake Mannix (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <521209480.34101.1306043567746.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <889607684.27597.1305830207722.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7307) Remove Dr Who references
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037511#comment-13037511 ] 

Jake Mannix commented on HADOOP-7307:
-------------------------------------

-1, this change would incompatibly break all productions systems where the cluster admin has nagios scripts monitoring log files for "dr.who".  

Please wait until 1.0 for this change.

> Remove Dr Who references
> ------------------------
>
>                 Key: HADOOP-7307
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7307
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>             Fix For: 0.22.0
>
>
> I've had feedback from a number of different operators that the "dr who" references that occasionally come out of Hadoop are annoying and confusing. We should replace them with more descriptive error messages, usernames, etc.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15253-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun May 22 06:47:33 2011
Return-Path: <common-issues-return-15253-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0122369C6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 22 May 2011 06:47:33 +0000 (UTC)
Received: (qmail 19971 invoked by uid 500); 22 May 2011 06:47:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19905 invoked by uid 500); 22 May 2011 06:47:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18941 invoked by uid 99); 22 May 2011 06:47:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 06:47:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 06:47:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6771CD6BF3
	for <common-issues@hadoop.apache.org>; Sun, 22 May 2011 06:46:47 +0000 (UTC)
Date: Sun, 22 May 2011 06:46:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1082683490.34139.1306046807420.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <187742091.8722.1305246647362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7287) Configuration deprecation mechanism
 doesn't work properly for GenericOptionsParser/Tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7287:
-----------------------------------

    Status: Patch Available  (was: Open)

> Configuration deprecation mechanism doesn't work properly for GenericOptionsParser/Tools
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7287
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7287
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Aaron T. Myers
>            Priority: Blocker
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: hadoop-7287-still-broken.txt, hadoop-7287-testcase.txt, hadoop-7287-trunk.0.patch, hadoop-7287-trunk.1.patch, hadoop-7287-trunk.2.patch
>
>
> For example, you can't use -D options on the "hadoop fs" command line in order to specify the deprecated names of configuration options. The issue is that the ordering is:
> - JVM starts
> - GenericOptionsParser creates a Configuration object and calls set() for each of the options specified on command line
> - DistributedFileSystem or other class eventually instantiates HdfsConfiguration which adds the deprecations
> - Some class calls conf.get("new key") and sees the default instead of the version set on the command line

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15254-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun May 22 07:06:52 2011
Return-Path: <common-issues-return-15254-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1488963A3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 22 May 2011 07:06:52 +0000 (UTC)
Received: (qmail 27069 invoked by uid 500); 22 May 2011 07:06:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26775 invoked by uid 500); 22 May 2011 07:06:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26724 invoked by uid 99); 22 May 2011 07:05:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 07:05:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 07:05:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 797D8D6F5D
	for <common-issues@hadoop.apache.org>; Sun, 22 May 2011 07:04:47 +0000 (UTC)
Date: Sun, 22 May 2011 07:04:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1481989289.34157.1306047887494.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <187742091.8722.1305246647362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7287) Configuration deprecation
 mechanism doesn't work properly for GenericOptionsParser/Tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037517#comment-13037517 ] 

Hadoop QA commented on HADOOP-7287:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479976/hadoop-7287-trunk.2.patch
  against trunk revision 1125221.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/496//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/496//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/496//console

This message is automatically generated.

> Configuration deprecation mechanism doesn't work properly for GenericOptionsParser/Tools
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7287
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7287
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Aaron T. Myers
>            Priority: Blocker
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: hadoop-7287-still-broken.txt, hadoop-7287-testcase.txt, hadoop-7287-trunk.0.patch, hadoop-7287-trunk.1.patch, hadoop-7287-trunk.2.patch
>
>
> For example, you can't use -D options on the "hadoop fs" command line in order to specify the deprecated names of configuration options. The issue is that the ordering is:
> - JVM starts
> - GenericOptionsParser creates a Configuration object and calls set() for each of the options specified on command line
> - DistributedFileSystem or other class eventually instantiates HdfsConfiguration which adds the deprecations
> - Some class calls conf.get("new key") and sees the default instead of the version set on the command line

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15255-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun May 22 07:29:31 2011
Return-Path: <common-issues-return-15255-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D7A126BD7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 22 May 2011 07:29:31 +0000 (UTC)
Received: (qmail 48869 invoked by uid 500); 22 May 2011 07:29:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48808 invoked by uid 500); 22 May 2011 07:29:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48315 invoked by uid 99); 22 May 2011 07:29:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 07:29:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 07:29:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6B02AD63A1
	for <common-issues@hadoop.apache.org>; Sun, 22 May 2011 07:28:47 +0000 (UTC)
Date: Sun, 22 May 2011 07:28:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <998819878.34172.1306049327435.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1212537303.34070.1306041287593.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7318) MD5Hash factory should reset the
 digester it returns
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037523#comment-13037523 ] 

Eli Collins commented on HADOOP-7318:
-------------------------------------

+1   good catch

> MD5Hash factory should reset the digester it returns
> ----------------------------------------------------
>
>                 Key: HADOOP-7318
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7318
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7318.txt, hadoop-7318.txt
>
>
> Currently the getDigest() method in MD5Hash does not reset the digester it returns. Since it's a thread-local, this means that a previous aborted usage of the same digester could leave some state around. For example, if the secondary namenode receives an IOException while transfering the image, and does another image transfer with the same thread, it will think it has received an invalid digest.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15256-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun May 22 08:32:30 2011
Return-Path: <common-issues-return-15256-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 68D256CE6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 22 May 2011 08:32:30 +0000 (UTC)
Received: (qmail 67868 invoked by uid 500); 22 May 2011 08:32:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67816 invoked by uid 500); 22 May 2011 08:32:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67808 invoked by uid 99); 22 May 2011 08:32:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 08:32:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 08:32:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 559E1D6EB1
	for <common-issues@hadoop.apache.org>; Sun, 22 May 2011 08:31:47 +0000 (UTC)
Date: Sun, 22 May 2011 08:31:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2031438916.34213.1306053107347.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <51041940.28155.1305838907965.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7310) Trash location needs to be
 revisited
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037533#comment-13037533 ] 

Eli Collins commented on HADOOP-7310:
-------------------------------------

Solution A sounds good to me as well. This also has the nice property that you can set a single quota on all trash.

With federation do you expect that trash will live on the same NN as home directories? If home directories span multiple NNs (probably uncommon) then it seems like you would want trash to as well, and this is more complex for admins when Trash is not a child of the user directory.

In terms of compatibility, post this change, I assume /user/joe/.Trash be considered a regular directory, and admins can move the contents to /Trash/joe as part of the upgrade.

> Trash location needs to be revisited
> ------------------------------------
>
>                 Key: HADOOP-7310
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7310
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15257-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun May 22 09:34:29 2011
Return-Path: <common-issues-return-15257-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E3C266DAD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 22 May 2011 09:34:28 +0000 (UTC)
Received: (qmail 98243 invoked by uid 500); 22 May 2011 09:34:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98215 invoked by uid 500); 22 May 2011 09:34:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98207 invoked by uid 99); 22 May 2011 09:34:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 09:34:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 09:34:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BA790D69A5
	for <common-issues@hadoop.apache.org>; Sun, 22 May 2011 09:33:47 +0000 (UTC)
Date: Sun, 22 May 2011 09:33:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2013629184.34255.1306056827760.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <531829213.32234.1305934307559.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7316) Add public javadocs to
 FSDataInputStream and FSDataOutputStream
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7316:
--------------------------------

    Assignee: Eli Collins
     Summary: Add public javadocs to FSDataInputStream and FSDataOutputStream  (was: Add javadoc comments to FSDataInputStream.getWrappedStream public method.)

Might as well do this for all public methods in both FSDataInputStream and FSDataOutputStream.

> Add public javadocs to FSDataInputStream and FSDataOutputStream
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7316
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7316
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Jonathan Hsieh
>            Assignee: Eli Collins
>             Fix For: 0.23.0
>
>
> This is a method made public for testing.  In comments in HADOOP-7301 after commit, adding javadoc comments was requested.  This is a follow up jira to address it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15258-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun May 22 09:38:29 2011
Return-Path: <common-issues-return-15258-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1EB266DCE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 22 May 2011 09:38:29 +0000 (UTC)
Received: (qmail 98787 invoked by uid 500); 22 May 2011 09:38:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98753 invoked by uid 500); 22 May 2011 09:38:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98745 invoked by uid 99); 22 May 2011 09:38:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 09:38:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 09:38:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 83F72D6AAB
	for <common-issues@hadoop.apache.org>; Sun, 22 May 2011 09:37:47 +0000 (UTC)
Date: Sun, 22 May 2011 09:37:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <731792007.34262.1306057067537.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <531829213.32234.1305934307559.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7316) Add public javadocs to
 FSDataInputStream and FSDataOutputStream
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7316:
--------------------------------

    Attachment: hadoop-7316-1.patch

Patch attached. Also fixes a couple minor javadoc errors in FsCommand and Command.

> Add public javadocs to FSDataInputStream and FSDataOutputStream
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7316
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7316
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Jonathan Hsieh
>            Assignee: Eli Collins
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7316-1.patch
>
>
> This is a method made public for testing.  In comments in HADOOP-7301 after commit, adding javadoc comments was requested.  This is a follow up jira to address it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15259-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun May 22 09:38:30 2011
Return-Path: <common-issues-return-15259-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8BA136DE3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 22 May 2011 09:38:30 +0000 (UTC)
Received: (qmail 99103 invoked by uid 500); 22 May 2011 09:38:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99071 invoked by uid 500); 22 May 2011 09:38:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99060 invoked by uid 99); 22 May 2011 09:38:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 09:38:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 09:38:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9318AD6AAE
	for <common-issues@hadoop.apache.org>; Sun, 22 May 2011 09:37:47 +0000 (UTC)
Date: Sun, 22 May 2011 09:37:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2108030996.34264.1306057067599.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <531829213.32234.1305934307559.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7316) Add public javadocs to
 FSDataInputStream and FSDataOutputStream
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7316:
--------------------------------

    Status: Patch Available  (was: Open)

> Add public javadocs to FSDataInputStream and FSDataOutputStream
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7316
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7316
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Jonathan Hsieh
>            Assignee: Eli Collins
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7316-1.patch
>
>
> This is a method made public for testing.  In comments in HADOOP-7301 after commit, adding javadoc comments was requested.  This is a follow up jira to address it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15260-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun May 22 16:28:31 2011
Return-Path: <common-issues-return-15260-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2158E604C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 22 May 2011 16:28:31 +0000 (UTC)
Received: (qmail 68504 invoked by uid 500); 22 May 2011 16:28:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68459 invoked by uid 500); 22 May 2011 16:28:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68451 invoked by uid 99); 22 May 2011 16:28:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 16:28:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 16:28:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 04EDFD7F32
	for <common-issues@hadoop.apache.org>; Sun, 22 May 2011 16:27:48 +0000 (UTC)
Date: Sun, 22 May 2011 16:27:48 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <798906530.34565.1306081668016.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <531829213.32234.1305934307559.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7316) Add public javadocs to
 FSDataInputStream and FSDataOutputStream
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-7316:
----------------------------------

    Status: Open  (was: Patch Available)

The newly visible method should be annotated with interface audience private.

Why was it necessary to make this method public instead of package visible? All testing should be able to use that.

> Add public javadocs to FSDataInputStream and FSDataOutputStream
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7316
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7316
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Jonathan Hsieh
>            Assignee: Eli Collins
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7316-1.patch
>
>
> This is a method made public for testing.  In comments in HADOOP-7301 after commit, adding javadoc comments was requested.  This is a follow up jira to address it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15261-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun May 22 18:57:30 2011
Return-Path: <common-issues-return-15261-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5B2E46BA8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 22 May 2011 18:57:30 +0000 (UTC)
Received: (qmail 67073 invoked by uid 500); 22 May 2011 18:57:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67031 invoked by uid 500); 22 May 2011 18:57:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67023 invoked by uid 99); 22 May 2011 18:57:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 18:57:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 18:57:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5DC8CD7BE9
	for <common-issues@hadoop.apache.org>; Sun, 22 May 2011 18:56:47 +0000 (UTC)
Date: Sun, 22 May 2011 18:56:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1702529409.34652.1306090607380.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <51041940.28155.1305838907965.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7310) Trash location needs to be
 revisited
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037644#comment-13037644 ] 

Aaron T. Myers commented on HADOOP-7310:
----------------------------------------

Could we perhaps just make the trash prefix configurable in the same vain as {{mapreduce.jobtracker.staging.root.dir}}? I'm thinking some configurable path onto which "${user.name}/.Trash" would be appended. This way admins who wanted to could set the prefix to "{{/user}}", and thereby retain the current location of the trash directories, and admins who wanted a single directory for the trash at the root could set it to "{{/Trash}}", much like Sanjay's proposed solution A above.

> Trash location needs to be revisited
> ------------------------------------
>
>                 Key: HADOOP-7310
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7310
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15262-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun May 22 20:51:28 2011
Return-Path: <common-issues-return-15262-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8C87A659A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 22 May 2011 20:51:28 +0000 (UTC)
Received: (qmail 4073 invoked by uid 500); 22 May 2011 20:51:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4043 invoked by uid 500); 22 May 2011 20:51:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4035 invoked by uid 99); 22 May 2011 20:51:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 20:51:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 May 2011 20:51:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 994F4D70D4
	for <common-issues@hadoop.apache.org>; Sun, 22 May 2011 20:50:47 +0000 (UTC)
Date: Sun, 22 May 2011 20:50:47 +0000 (UTC)
From: "Ted Yu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1251822328.34737.1306097447624.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7319) Coarse-grained dynamic configuration
 changes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Coarse-grained dynamic configuration changes
--------------------------------------------

                 Key: HADOOP-7319
                 URL: https://issues.apache.org/jira/browse/HADOOP-7319
             Project: Hadoop Common
          Issue Type: Improvement
          Components: conf
            Reporter: Ted Yu


HADOOP-7001 introduced mechanism for performing dynamic configuration changes where reconfigureProperty()/reconfigurePropertyImpl() only notifies single property change.

Normally, components which use ReconfigurableBase would involve several related properties whose update should be done atomically.

This JIRA provides coarse-grained dynamic configuration changes with the following benefits:
1. consistency updating related properties dynamically
2. reduction of lock contention when multiple properties are changed in proximity

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15263-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 00:59:29 2011
Return-Path: <common-issues-return-15263-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A44B268DD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 00:59:29 +0000 (UTC)
Received: (qmail 6929 invoked by uid 500); 23 May 2011 00:59:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6902 invoked by uid 500); 23 May 2011 00:59:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6894 invoked by uid 99); 23 May 2011 00:59:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 00:59:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 00:59:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 824D3D8263
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 00:58:48 +0000 (UTC)
Date: Mon, 23 May 2011 00:58:48 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1824798075.34861.1306112328530.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1659614701.18791.1297804257612.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7146) RPC server leaks file descriptors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037686#comment-13037686 ] 

Hadoop QA commented on HADOOP-7146:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12479955/hadoop-7146.txt
  against trunk revision 1125221.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/501//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/501//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/501//console

This message is automatically generated.

> RPC server leaks file descriptors
> ---------------------------------
>
>                 Key: HADOOP-7146
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7146
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: fd-leak.txt, hadoop-7146.txt
>
>
> Both the Listener and Responder thread call Selector.open but don't have a matching .close(). This causes a leak of anonymous pipes. Not a big deal because people rarely close and re-open servers, but worth fixing.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15264-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 00:59:29 2011
Return-Path: <common-issues-return-15264-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D32F468F0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 00:59:29 +0000 (UTC)
Received: (qmail 7219 invoked by uid 500); 23 May 2011 00:59:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7131 invoked by uid 500); 23 May 2011 00:59:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6969 invoked by uid 99); 23 May 2011 00:59:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 00:59:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 00:59:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8DBECD8265
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 00:58:48 +0000 (UTC)
Date: Mon, 23 May 2011 00:58:48 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1274333081.34863.1306112328577.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1212537303.34070.1306041287593.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7318) MD5Hash factory should reset the
 digester it returns
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037687#comment-13037687 ] 

Hadoop QA commented on HADOOP-7318:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480018/hadoop-7318.txt
  against trunk revision 1125221.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/503//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/503//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/503//console

This message is automatically generated.

> MD5Hash factory should reset the digester it returns
> ----------------------------------------------------
>
>                 Key: HADOOP-7318
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7318
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7318.txt, hadoop-7318.txt
>
>
> Currently the getDigest() method in MD5Hash does not reset the digester it returns. Since it's a thread-local, this means that a previous aborted usage of the same digester could leave some state around. For example, if the secondary namenode receives an IOException while transfering the image, and does another image transfer with the same thread, it will think it has received an invalid digest.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15265-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 01:31:28 2011
Return-Path: <common-issues-return-15265-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5A5696D60
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 01:31:28 +0000 (UTC)
Received: (qmail 37195 invoked by uid 500); 23 May 2011 01:31:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37148 invoked by uid 500); 23 May 2011 01:31:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37140 invoked by uid 99); 23 May 2011 01:31:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 01:31:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 01:31:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6024FD8864
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 01:30:47 +0000 (UTC)
Date: Mon, 23 May 2011 01:30:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1517848555.34925.1306114247390.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1212537303.34070.1306041287593.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7318) MD5Hash factory should reset the
 digester it returns
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7318:
--------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I've committed this to trunk and branch 22. Thanks Todd!

> MD5Hash factory should reset the digester it returns
> ----------------------------------------------------
>
>                 Key: HADOOP-7318
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7318
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7318.txt, hadoop-7318.txt
>
>
> Currently the getDigest() method in MD5Hash does not reset the digester it returns. Since it's a thread-local, this means that a previous aborted usage of the same digester could leave some state around. For example, if the secondary namenode receives an IOException while transfering the image, and does another image transfer with the same thread, it will think it has received an invalid digest.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15266-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 02:35:29 2011
Return-Path: <common-issues-return-15266-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 46A394A8E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 02:35:29 +0000 (UTC)
Received: (qmail 75244 invoked by uid 500); 23 May 2011 02:35:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75168 invoked by uid 500); 23 May 2011 02:35:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75160 invoked by uid 99); 23 May 2011 02:35:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 02:35:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 02:35:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 60297D83F1
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 02:34:47 +0000 (UTC)
Date: Mon, 23 May 2011 02:34:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <765057651.34961.1306118087390.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1212537303.34070.1306041287593.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7318) MD5Hash factory should reset the
 digester it returns
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037700#comment-13037700 ] 

Hudson commented on HADOOP-7318:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #615 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/615/])
    

> MD5Hash factory should reset the digester it returns
> ----------------------------------------------------
>
>                 Key: HADOOP-7318
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7318
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7318.txt, hadoop-7318.txt
>
>
> Currently the getDigest() method in MD5Hash does not reset the digester it returns. Since it's a thread-local, this means that a previous aborted usage of the same digester could leave some state around. For example, if the secondary namenode receives an IOException while transfering the image, and does another image transfer with the same thread, it will think it has received an invalid digest.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15267-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 03:11:30 2011
Return-Path: <common-issues-return-15267-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E8EA54F04
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 03:11:30 +0000 (UTC)
Received: (qmail 88470 invoked by uid 500); 23 May 2011 03:11:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88264 invoked by uid 500); 23 May 2011 03:11:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88245 invoked by uid 99); 23 May 2011 03:11:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 03:11:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 03:11:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BBBB2D8B90
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 03:10:48 +0000 (UTC)
Date: Mon, 23 May 2011 03:10:48 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <135529210.35067.1306120248765.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <889607684.27597.1305830207722.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7307) Remove Dr Who references
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037705#comment-13037705 ] 

Allen Wittenauer commented on HADOOP-7307:
------------------------------------------

Does this actually still happen in trunk or is there an edge-case where the username doesn't get resolved?

> Remove Dr Who references
> ------------------------
>
>                 Key: HADOOP-7307
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7307
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>             Fix For: 0.22.0
>
>
> I've had feedback from a number of different operators that the "dr who" references that occasionally come out of Hadoop are annoying and confusing. We should replace them with more descriptive error messages, usernames, etc.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15268-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 05:15:34 2011
Return-Path: <common-issues-return-15268-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 94B1E4B33
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 05:15:34 +0000 (UTC)
Received: (qmail 51860 invoked by uid 500); 23 May 2011 05:15:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51818 invoked by uid 500); 23 May 2011 05:15:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51763 invoked by uid 99); 23 May 2011 05:15:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 05:15:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 05:15:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 56479D860C
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 05:14:47 +0000 (UTC)
Date: Mon, 23 May 2011 05:14:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <708223499.35202.1306127687350.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1659614701.18791.1297804257612.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7146) RPC server leaks file descriptors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037737#comment-13037737 ] 

Matt Foley commented on HADOOP-7146:
------------------------------------

Todd, kudos for finding this and how to fix it, and also to Aaron for nailing the TestDFSStorageStateRecovery problem.

I'd like to suggest a more structured fix:
1. Move the "readSelector = Selector.open()" statement from Listener() to the Reader() ctor, and change the ctor to have no arguments.
2. In Reader.run(), wrap the whole method in a try/finally context, with your five-line fix in the "finally" clause, and make it conditional on (readSelector != null).
3. In Responder.run(), similarly wrap the whole method in a try/finally context, with your five-line fix in the "finally" clause, and make it conditional on (writeSelector != null).

I would have liked to recommend putting the "readSelector = Selector.open()" [Reader] and "writeSelector = Selector.open()" [Responder] statements at the beginning of their respective run() methods, so the try/finally context would really be structured right, but I'm not sure its okay in the Responder context -- it looks like some of its methods may invoke writeSelector without the run() being running.

> RPC server leaks file descriptors
> ---------------------------------
>
>                 Key: HADOOP-7146
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7146
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: fd-leak.txt, hadoop-7146.txt
>
>
> Both the Listener and Responder thread call Selector.open but don't have a matching .close(). This causes a leak of anonymous pipes. Not a big deal because people rarely close and re-open servers, but worth fixing.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15269-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 09:43:29 2011
Return-Path: <common-issues-return-15269-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 970E647F9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 09:43:29 +0000 (UTC)
Received: (qmail 43893 invoked by uid 500); 23 May 2011 09:43:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43868 invoked by uid 500); 23 May 2011 09:43:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43860 invoked by uid 99); 23 May 2011 09:43:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 09:43:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 09:43:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7CB08D8687
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 09:42:47 +0000 (UTC)
Date: Mon, 23 May 2011 09:42:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <139208930.35610.1306143767507.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1212537303.34070.1306041287593.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7318) MD5Hash factory should reset the
 digester it returns
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037821#comment-13037821 ] 

Hudson commented on HADOOP-7318:
--------------------------------

Integrated in Hadoop-Common-22-branch #55 (See [https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/55/])
    HADOOP-7318. svn merge -c 1126287 from trunk

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1126288
Files : 
* /hadoop/common/branches/branch-0.22/src/test/core/org/apache/hadoop/io/TestMD5Hash.java
* /hadoop/common/branches/branch-0.22/src/docs
* /hadoop/common/branches/branch-0.22
* /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/io/MD5Hash.java
* /hadoop/common/branches/branch-0.22/src/test/core
* /hadoop/common/branches/branch-0.22/src/test/core/org/apache/hadoop/io/TestSequenceFile.java
* /hadoop/common/branches/branch-0.22/CHANGES.txt
* /hadoop/common/branches/branch-0.22/src/java


> MD5Hash factory should reset the digester it returns
> ----------------------------------------------------
>
>                 Key: HADOOP-7318
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7318
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7318.txt, hadoop-7318.txt
>
>
> Currently the getDigest() method in MD5Hash does not reset the digester it returns. Since it's a thread-local, this means that a previous aborted usage of the same digester could leave some state around. For example, if the secondary namenode receives an IOException while transfering the image, and does another image transfer with the same thread, it will think it has received an invalid digest.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15270-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 09:43:30 2011
Return-Path: <common-issues-return-15270-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CAE124803
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 09:43:30 +0000 (UTC)
Received: (qmail 44158 invoked by uid 500); 23 May 2011 09:43:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44122 invoked by uid 500); 23 May 2011 09:43:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44114 invoked by uid 99); 23 May 2011 09:43:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 09:43:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 09:43:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6EDFBD8685
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 09:42:47 +0000 (UTC)
Date: Mon, 23 May 2011 09:42:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <459327566.35608.1306143767450.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <2117896899.24772.1305762047299.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7302) webinterface.private.actions
 should not be in common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037820#comment-13037820 ] 

Hudson commented on HADOOP-7302:
--------------------------------

Integrated in Hadoop-Common-22-branch #55 (See [https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/55/])
    

> webinterface.private.actions should not be in common
> ----------------------------------------------------
>
>                 Key: HADOOP-7302
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7302
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.23.0
>            Reporter: Ari Rabkin
>            Assignee: Ari Rabkin
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7302.patch
>
>
> The comment in -defaults says that webinterface.private.actions applies to both NN and JT. This is wrong. This option is only referenced via the JobTracker. 
> I propose to delete it here, and file a second issue in MAPREDUCE for renaming the option (deprecating the existing name.)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15271-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 12:31:31 2011
Return-Path: <common-issues-return-15271-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3F0F84FA0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 12:31:31 +0000 (UTC)
Received: (qmail 43520 invoked by uid 500); 23 May 2011 12:31:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43481 invoked by uid 500); 23 May 2011 12:31:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43473 invoked by uid 99); 23 May 2011 12:31:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 12:31:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 12:31:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BCD76D9205
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 12:30:47 +0000 (UTC)
Date: Mon, 23 May 2011 12:30:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1408573578.35806.1306153847770.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1212537303.34070.1306041287593.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7318) MD5Hash factory should reset the
 digester it returns
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037909#comment-13037909 ] 

Hudson commented on HADOOP-7318:
--------------------------------

Integrated in Hadoop-Common-trunk #697 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/697/])
    HADOOP-7318. MD5Hash factory should reset the digester it returns. Contributed by Todd Lipcon

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1126287
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/io/TestMD5Hash.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/io/MD5Hash.java


> MD5Hash factory should reset the digester it returns
> ----------------------------------------------------
>
>                 Key: HADOOP-7318
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7318
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7318.txt, hadoop-7318.txt
>
>
> Currently the getDigest() method in MD5Hash does not reset the digester it returns. Since it's a thread-local, this means that a previous aborted usage of the same digester could leave some state around. For example, if the secondary namenode receives an IOException while transfering the image, and does another image transfer with the same thread, it will think it has received an invalid digest.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15272-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 14:47:34 2011
Return-Path: <common-issues-return-15272-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 26BFF4D03
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 14:47:34 +0000 (UTC)
Received: (qmail 19369 invoked by uid 500); 23 May 2011 14:47:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19335 invoked by uid 500); 23 May 2011 14:47:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19327 invoked by uid 99); 23 May 2011 14:47:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 14:47:33 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 14:47:33 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1CC30D9F31
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 14:46:53 +0000 (UTC)
Date: Mon, 23 May 2011 14:46:53 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1455428770.36108.1306162013114.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1195662636.6130.1300891805766.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7208) equals() and hashCode()
 implementation need to change in StandardSocketFactory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uma Maheswara Rao G updated HADOOP-7208:
----------------------------------------

    Attachment: HADOOP-7208.patch

> equals() and hashCode() implementation need to change in StandardSocketFactory
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7208
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7208
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HADOOP-7208.patch
>
>
>   In Hadoop IPC Client, we are using ClientCache which will maintain the HashMap to keep the Client references.
> private Map<SocketFactory, Client> clients =
>       new HashMap<SocketFactory, Client>();
>  Now let us say, we want use two standard factories with Hadoop. MyStandardSocketFactory (which extends StandardSocketFactory), and StandardSocketFactory. In this case, because of equals and hashcode implementation, MyStandardSocketFactory client can be overridden by StandardSocketFactoryClient

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15273-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 14:51:28 2011
Return-Path: <common-issues-return-15273-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4D3834462
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 14:51:28 +0000 (UTC)
Received: (qmail 30239 invoked by uid 500); 23 May 2011 14:51:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30201 invoked by uid 500); 23 May 2011 14:51:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30193 invoked by uid 99); 23 May 2011 14:51:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 14:51:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 14:51:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 64DCCD9195
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 14:50:47 +0000 (UTC)
Date: Mon, 23 May 2011 14:50:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1828723839.36128.1306162247409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7320) Refactor FsShell's copy & move
 commands
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Refactor FsShell's copy & move commands
---------------------------------------

                 Key: HADOOP-7320
                 URL: https://issues.apache.org/jira/browse/HADOOP-7320
             Project: Hadoop Common
          Issue Type: Improvement
    Affects Versions: 0.23.0
            Reporter: Daryn Sharp
            Assignee: Daryn Sharp


Need to refactor the move and copy commands to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15274-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 15:02:30 2011
Return-Path: <common-issues-return-15274-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 44B0F42D3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 15:02:30 +0000 (UTC)
Received: (qmail 57807 invoked by uid 500); 23 May 2011 15:02:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57771 invoked by uid 500); 23 May 2011 15:02:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57759 invoked by uid 99); 23 May 2011 15:02:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 15:02:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 15:02:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 368DAD9603
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 15:01:49 +0000 (UTC)
Date: Mon, 23 May 2011 15:01:49 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <333526277.36147.1306162909220.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1195662636.6130.1300891805766.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7208) equals() and hashCode()
 implementation need to change in StandardSocketFactory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uma Maheswara Rao G updated HADOOP-7208:
----------------------------------------

        Fix Version/s: 0.23.0
    Affects Version/s: 0.23.0
         Hadoop Flags: [Reviewed]
               Status: Patch Available  (was: Open)

> equals() and hashCode() implementation need to change in StandardSocketFactory
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7208
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7208
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7208.patch
>
>
>   In Hadoop IPC Client, we are using ClientCache which will maintain the HashMap to keep the Client references.
> private Map<SocketFactory, Client> clients =
>       new HashMap<SocketFactory, Client>();
>  Now let us say, we want use two standard factories with Hadoop. MyStandardSocketFactory (which extends StandardSocketFactory), and StandardSocketFactory. In this case, because of equals and hashcode implementation, MyStandardSocketFactory client can be overridden by StandardSocketFactoryClient

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15275-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 15:24:29 2011
Return-Path: <common-issues-return-15275-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CD8374BC1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 15:24:29 +0000 (UTC)
Received: (qmail 96199 invoked by uid 500); 23 May 2011 15:24:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96131 invoked by uid 500); 23 May 2011 15:24:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96057 invoked by uid 99); 23 May 2011 15:24:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 15:24:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 15:24:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 99578D9EF3
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 15:23:47 +0000 (UTC)
Date: Mon, 23 May 2011 15:23:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1672571552.36183.1306164227624.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1195662636.6130.1300891805766.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7208) equals() and hashCode()
 implementation need to change in StandardSocketFactory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037989#comment-13037989 ] 

Hadoop QA commented on HADOOP-7208:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480117/HADOOP-7208.patch
  against trunk revision 1126287.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/504//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/504//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/504//console

This message is automatically generated.

> equals() and hashCode() implementation need to change in StandardSocketFactory
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7208
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7208
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7208.patch
>
>
>   In Hadoop IPC Client, we are using ClientCache which will maintain the HashMap to keep the Client references.
> private Map<SocketFactory, Client> clients =
>       new HashMap<SocketFactory, Client>();
>  Now let us say, we want use two standard factories with Hadoop. MyStandardSocketFactory (which extends StandardSocketFactory), and StandardSocketFactory. In this case, because of equals and hashcode implementation, MyStandardSocketFactory client can be overridden by StandardSocketFactoryClient

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15276-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 15:52:30 2011
Return-Path: <common-issues-return-15276-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0B25544B9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 15:52:30 +0000 (UTC)
Received: (qmail 51462 invoked by uid 500); 23 May 2011 15:52:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51416 invoked by uid 500); 23 May 2011 15:52:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51182 invoked by uid 99); 23 May 2011 15:52:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 15:52:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 15:52:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A6CF5D9AA2
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 15:51:47 +0000 (UTC)
Date: Mon, 23 May 2011 15:51:47 +0000 (UTC)
From: "jiraposter@reviews.apache.org (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1838691526.36239.1306165907679.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <467948518.30516.1305901367779.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7314) Add support for throwing
 UnknownHostException when a host doesn't resolve
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038002#comment-13038002 ] 

jiraposter@reviews.apache.org commented on HADOOP-7314:
-------------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/775/
-----------------------------------------------------------

Review request for hadoop-common and hadoop-mapreduce.


Summary
-------

As part of MAPREDUCE-2489, we need support for having the resolve methods (for DNS mapping) throw UnknownHostExceptions. (Currently, they hide the exception). Since the existing 'resolve' method is ultimately used by several other locations/components, I propose we add a new 'resolveValidHosts' method.


This addresses bug HADOOP-7314.
    https://issues.apache.org/jira/browse/HADOOP-7314


Diffs
-----

  trunk/src/java/org/apache/hadoop/net/DNSToSwitchMapping.java 1125067 
  trunk/src/java/org/apache/hadoop/net/ScriptBasedMapping.java 1125067 
  trunk/src/test/core/org/apache/hadoop/net/StaticMapping.java 1125067 
  trunk/src/test/core/org/apache/hadoop/net/TestScriptBasedMapping.java 1125067 
  trunk/src/java/org/apache/hadoop/net/CachedDNSToSwitchMapping.java 1125067 

Diff: https://reviews.apache.org/r/775/diff


Testing
-------


Thanks,

Jeffrey



> Add support for throwing UnknownHostException when a host doesn't resolve
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-7314
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7314
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jeffrey Naisbitt
>            Assignee: Jeffrey Naisbitt
>         Attachments: HADOOP-7314.patch
>
>
> As part of MAPREDUCE-2489, we need support for having the resolve methods (for DNS mapping) throw UnknownHostExceptions.  (Currently, they hide the exception).  Since the existing 'resolve' method is ultimately used by several other locations/components, I propose we add a new 'resolveValidHosts' method.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15277-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 16:40:28 2011
Return-Path: <common-issues-return-15277-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A259E4BDC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 16:40:28 +0000 (UTC)
Received: (qmail 38851 invoked by uid 500); 23 May 2011 16:40:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38824 invoked by uid 500); 23 May 2011 16:40:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38816 invoked by uid 99); 23 May 2011 16:40:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 16:40:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 16:40:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8DCEED9FE9
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 16:39:47 +0000 (UTC)
Date: Mon, 23 May 2011 16:39:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <192274156.36401.1306168787577.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7284:
---------------------------------

    Status: Open  (was: Patch Available)

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15278-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 16:40:28 2011
Return-Path: <common-issues-return-15278-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CDD124BED
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 16:40:28 +0000 (UTC)
Received: (qmail 39061 invoked by uid 500); 23 May 2011 16:40:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39009 invoked by uid 500); 23 May 2011 16:40:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38951 invoked by uid 99); 23 May 2011 16:40:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 16:40:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 16:40:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B06D3D9FED
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 16:39:47 +0000 (UTC)
Date: Mon, 23 May 2011 16:39:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <31932799.36405.1306168787719.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7284:
---------------------------------

    Status: Patch Available  (was: Open)

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15279-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 16:40:30 2011
Return-Path: <common-issues-return-15279-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E08034BFD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 16:40:30 +0000 (UTC)
Received: (qmail 39360 invoked by uid 500); 23 May 2011 16:40:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39331 invoked by uid 500); 23 May 2011 16:40:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39310 invoked by uid 99); 23 May 2011 16:40:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 16:40:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 16:40:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 72012D9FE5
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 16:39:47 +0000 (UTC)
Date: Mon, 23 May 2011 16:39:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1911286635.36397.1306168787463.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7284:
---------------------------------

    Attachment: trash4.patch

I had forgot to do "git add .."; the new tests are now part of the patch.

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15280-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 16:55:29 2011
Return-Path: <common-issues-return-15280-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F23EE46CF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 16:55:28 +0000 (UTC)
Received: (qmail 73671 invoked by uid 500); 23 May 2011 16:55:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73615 invoked by uid 500); 23 May 2011 16:55:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73559 invoked by uid 99); 23 May 2011 16:55:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 16:55:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 16:55:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A4600D9739
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 16:54:47 +0000 (UTC)
Date: Mon, 23 May 2011 16:54:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1184135421.36448.1306169687669.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038040#comment-13038040 ] 

Hadoop QA commented on HADOOP-7284:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480126/trash4.patch
  against trunk revision 1126287.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 17 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    -1 release audit.  The applied patch generated 2 release audit warnings (more than the trunk's current 1 warnings).

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.fs.viewfs.TestViewFsTrash

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/505//testReport/
Release audit warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/505//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/505//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/505//console

This message is automatically generated.

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15281-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 16:55:29 2011
Return-Path: <common-issues-return-15281-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A859046E1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 16:55:29 +0000 (UTC)
Received: (qmail 74027 invoked by uid 500); 23 May 2011 16:55:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73978 invoked by uid 500); 23 May 2011 16:55:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73962 invoked by uid 99); 23 May 2011 16:55:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 16:55:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 16:55:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 45562D9754
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 16:54:48 +0000 (UTC)
Date: Mon, 23 May 2011 16:54:48 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <813155626.36466.1306169688280.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <531829213.32234.1305934307559.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7316) Add public javadocs to
 FSDataInputStream and FSDataOutputStream
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7316:
-------------------------------------------

    Component/s: documentation
     Issue Type: Improvement  (was: New Feature)

> Add public javadocs to FSDataInputStream and FSDataOutputStream
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7316
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7316
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Jonathan Hsieh
>            Assignee: Eli Collins
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7316-1.patch
>
>
> This is a method made public for testing.  In comments in HADOOP-7301 after commit, adding javadoc comments was requested.  This is a follow up jira to address it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15282-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 17:12:28 2011
Return-Path: <common-issues-return-15282-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6E2E04163
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 17:12:28 +0000 (UTC)
Received: (qmail 93308 invoked by uid 500); 23 May 2011 17:12:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93280 invoked by uid 500); 23 May 2011 17:12:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93272 invoked by uid 99); 23 May 2011 17:12:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 17:12:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 17:12:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5C785D9EE2
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 17:11:47 +0000 (UTC)
Date: Mon, 23 May 2011 17:11:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <989015970.36506.1306170707375.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <531829213.32234.1305934307559.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7316) Add public javadocs to
 FSDataInputStream and FSDataOutputStream
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7316:
--------------------------------

    Attachment: hadoop-7316-2.patch

Thanks Owen.  Updated patch attached. 

I marked both FSDataInputStream and FSDataOutputStream#getWrappedStream LimitedPrivate to HDFS.

The addition of FSDataInputStream#getWrappedStream mirrors the same method in FSDataOutputStream. Both live in common but are just used for HDFS tests, which is why they are not package visible. Could also expose these via a test  adapter class but I think the annotation is sufficient.

> Add public javadocs to FSDataInputStream and FSDataOutputStream
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7316
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7316
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Jonathan Hsieh
>            Assignee: Eli Collins
>             Fix For: 0.23.0
>
>         Attachments: hadoop-7316-1.patch, hadoop-7316-2.patch
>
>
> This is a method made public for testing.  In comments in HADOOP-7301 after commit, adding javadoc comments was requested.  This is a follow up jira to address it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15283-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 17:24:29 2011
Return-Path: <common-issues-return-15283-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 541934CEA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 17:24:29 +0000 (UTC)
Received: (qmail 20682 invoked by uid 500); 23 May 2011 17:24:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20561 invoked by uid 500); 23 May 2011 17:24:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20548 invoked by uid 99); 23 May 2011 17:24:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 17:24:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 17:24:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 30434D962A
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 17:23:48 +0000 (UTC)
Date: Mon, 23 May 2011 17:23:48 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <147164333.36596.1306171428194.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-6671:
---------------------------------------

    Attachment: HADOOP-6671c.patch

HADOOP-6671c.patch uses the maven native:javah mojo to generate the javah stuff.

This solves the need tools.jar issue.

I've also moved all the native autoreconf/configure/make to happen under the BUILD directory (target/). I've done this before for the documentation generation. This  makes all files generated by the build to happen under target/

Still jdiff invocation is missing

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>         Attachments: HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, build.png, hadoop-commons-maven.patch, mvn-layout.sh, mvn-layout.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15284-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 17:34:31 2011
Return-Path: <common-issues-return-15284-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EBEC54515
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 17:34:30 +0000 (UTC)
Received: (qmail 35147 invoked by uid 500); 23 May 2011 17:34:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35118 invoked by uid 500); 23 May 2011 17:34:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35110 invoked by uid 99); 23 May 2011 17:34:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 17:34:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 17:34:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A25B2D9C60
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 17:33:47 +0000 (UTC)
Date: Mon, 23 May 2011 17:33:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <187366143.36661.1306172027661.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038082#comment-13038082 ] 

Hadoop QA commented on HADOOP-6671:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480134/HADOOP-6671c.patch
  against trunk revision 1126287.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 18 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/506//console

This message is automatically generated.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>         Attachments: HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, build.png, hadoop-commons-maven.patch, mvn-layout.sh, mvn-layout.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15285-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 18:47:31 2011
Return-Path: <common-issues-return-15285-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0C5D642D9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 18:47:31 +0000 (UTC)
Received: (qmail 73023 invoked by uid 500); 23 May 2011 18:47:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72992 invoked by uid 500); 23 May 2011 18:47:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72984 invoked by uid 99); 23 May 2011 18:47:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 18:47:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 18:47:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 54591D9DB6
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 18:46:47 +0000 (UTC)
Date: Mon, 23 May 2011 18:46:47 +0000 (UTC)
From: "Roko Kruze (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1472612857.36898.1306176407342.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7321) The link to the documentation on the
 hadoop common page is broken
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

The link to the documentation on the hadoop common page is broken
-----------------------------------------------------------------

                 Key: HADOOP-7321
                 URL: https://issues.apache.org/jira/browse/HADOOP-7321
             Project: Hadoop Common
          Issue Type: Bug
          Components: documentation
            Reporter: Roko Kruze


Currently the following link is broken for the documentation: http://hadoop.apache.org/common/docs/stable/

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15286-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 19:03:28 2011
Return-Path: <common-issues-return-15286-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E19D44EFE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 19:03:28 +0000 (UTC)
Received: (qmail 6963 invoked by uid 500); 23 May 2011 19:03:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6906 invoked by uid 500); 23 May 2011 19:03:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6898 invoked by uid 99); 23 May 2011 19:03:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 19:03:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 19:03:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D4D31D9578
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 19:02:47 +0000 (UTC)
Date: Mon, 23 May 2011 19:02:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1728938569.36954.1306177367868.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-6508) Incorrect values for metrics with
 CompositeContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu resolved HADOOP-6508.
-----------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0

> Incorrect values for metrics with CompositeContext
> --------------------------------------------------
>
>                 Key: HADOOP-6508
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6508
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.0
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>         Attachments: CompositeContext-solution.png, CompositeContext.png
>
>
> In our clusters, when we use CompositeContext with two contexts, second context gets wrong values.
> This problem is consistent on 500 (and above) node cluster.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15287-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 19:54:28 2011
Return-Path: <common-issues-return-15287-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8AFB34867
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 19:54:28 +0000 (UTC)
Received: (qmail 1304 invoked by uid 500); 23 May 2011 19:54:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1273 invoked by uid 500); 23 May 2011 19:54:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1265 invoked by uid 99); 23 May 2011 19:54:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 19:54:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 19:54:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 76AFED9B68
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 19:53:47 +0000 (UTC)
Date: Mon, 23 May 2011 19:53:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1050959151.37070.1306180427483.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1828723839.36128.1306162247409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7320) Refactor FsShell's copy & move
 commands
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7320:
--------------------------------

    Attachment: HADOOP-7320.patch

> Refactor FsShell's copy & move commands
> ---------------------------------------
>
>                 Key: HADOOP-7320
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7320
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7320.patch
>
>
> Need to refactor the move and copy commands to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15288-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 19:54:29 2011
Return-Path: <common-issues-return-15288-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 181A9487E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 19:54:29 +0000 (UTC)
Received: (qmail 1588 invoked by uid 500); 23 May 2011 19:54:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1551 invoked by uid 500); 23 May 2011 19:54:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1519 invoked by uid 99); 23 May 2011 19:54:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 19:54:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 19:54:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E09EED9B7A
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 19:53:47 +0000 (UTC)
Date: Mon, 23 May 2011 19:53:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1013737498.37086.1306180427917.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1828723839.36128.1306162247409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7320) Refactor FsShell's copy & move
 commands
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7320:
--------------------------------

    Status: Patch Available  (was: Open)

> Refactor FsShell's copy & move commands
> ---------------------------------------
>
>                 Key: HADOOP-7320
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7320
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7320.patch
>
>
> Need to refactor the move and copy commands to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15289-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 20:14:28 2011
Return-Path: <common-issues-return-15289-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B7B724CB0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 20:14:28 +0000 (UTC)
Received: (qmail 41452 invoked by uid 500); 23 May 2011 20:14:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41425 invoked by uid 500); 23 May 2011 20:14:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41417 invoked by uid 99); 23 May 2011 20:14:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 20:14:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 20:14:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BDEB2DA4E8
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 20:13:47 +0000 (UTC)
Date: Mon, 23 May 2011 20:13:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1911748846.37160.1306181627774.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1828723839.36128.1306162247409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7320) Refactor FsShell's copy & move
 commands
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038182#comment-13038182 ] 

Hadoop QA commented on HADOOP-7320:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480149/HADOOP-7320.patch
  against trunk revision 1126287.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 7 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.fs.TestFsShellReturnCode

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/507//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/507//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/507//console

This message is automatically generated.

> Refactor FsShell's copy & move commands
> ---------------------------------------
>
>                 Key: HADOOP-7320
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7320
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7320.patch
>
>
> Need to refactor the move and copy commands to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15290-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 20:34:32 2011
Return-Path: <common-issues-return-15290-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1CD5542A1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 20:34:32 +0000 (UTC)
Received: (qmail 74041 invoked by uid 500); 23 May 2011 20:34:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73994 invoked by uid 500); 23 May 2011 20:34:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73943 invoked by uid 99); 23 May 2011 20:34:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 20:34:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 20:34:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A9C2DDAC8B
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 20:33:47 +0000 (UTC)
Date: Mon, 23 May 2011 20:33:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <143464855.37251.1306182827692.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <187742091.8722.1305246647362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7287) Configuration deprecation mechanism
 doesn't work properly for GenericOptionsParser/Tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7287:
--------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

Committed to trunk and branch 22. Thanks, Aaron!

> Configuration deprecation mechanism doesn't work properly for GenericOptionsParser/Tools
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7287
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7287
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Aaron T. Myers
>            Priority: Blocker
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: hadoop-7287-still-broken.txt, hadoop-7287-testcase.txt, hadoop-7287-trunk.0.patch, hadoop-7287-trunk.1.patch, hadoop-7287-trunk.2.patch
>
>
> For example, you can't use -D options on the "hadoop fs" command line in order to specify the deprecated names of configuration options. The issue is that the ordering is:
> - JVM starts
> - GenericOptionsParser creates a Configuration object and calls set() for each of the options specified on command line
> - DistributedFileSystem or other class eventually instantiates HdfsConfiguration which adds the deprecations
> - Some class calls conf.get("new key") and sees the default instead of the version set on the command line

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15291-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 20:54:31 2011
Return-Path: <common-issues-return-15291-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 31C9949BD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 20:54:31 +0000 (UTC)
Received: (qmail 21170 invoked by uid 500); 23 May 2011 20:54:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21137 invoked by uid 500); 23 May 2011 20:54:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21111 invoked by uid 99); 23 May 2011 20:54:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 20:54:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 20:54:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E267CDA4CC
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 20:53:47 +0000 (UTC)
Date: Mon, 23 May 2011 20:53:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <573778160.37365.1306184027924.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <187742091.8722.1305246647362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7287) Configuration deprecation
 mechanism doesn't work properly for GenericOptionsParser/Tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038220#comment-13038220 ] 

Hudson commented on HADOOP-7287:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #616 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/616/])
    HADOOP-7287. Configuration deprecation mechanism doesn't work properly for GenericOptionsParser and Tools. Contributed by Aaron T. Myers.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1126719
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/conf/Configuration.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/conf/TestConfigurationDeprecation.java
* /hadoop/common/trunk/src/test/test-fake-default.xml


> Configuration deprecation mechanism doesn't work properly for GenericOptionsParser/Tools
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7287
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7287
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Aaron T. Myers
>            Priority: Blocker
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: hadoop-7287-still-broken.txt, hadoop-7287-testcase.txt, hadoop-7287-trunk.0.patch, hadoop-7287-trunk.1.patch, hadoop-7287-trunk.2.patch
>
>
> For example, you can't use -D options on the "hadoop fs" command line in order to specify the deprecated names of configuration options. The issue is that the ordering is:
> - JVM starts
> - GenericOptionsParser creates a Configuration object and calls set() for each of the options specified on command line
> - DistributedFileSystem or other class eventually instantiates HdfsConfiguration which adds the deprecations
> - Some class calls conf.get("new key") and sees the default instead of the version set on the command line

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15292-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 21:32:30 2011
Return-Path: <common-issues-return-15292-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A739F40EB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 21:32:30 +0000 (UTC)
Received: (qmail 10140 invoked by uid 500); 23 May 2011 21:32:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10109 invoked by uid 500); 23 May 2011 21:32:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10101 invoked by uid 99); 23 May 2011 21:32:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 21:32:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 21:32:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 831FEDA1BE
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 21:31:47 +0000 (UTC)
Date: Mon, 23 May 2011 21:31:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1462758028.37472.1306186307533.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1828723839.36128.1306162247409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7320) Refactor FsShell's copy & move
 commands
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7320:
--------------------------------

    Attachment: HADOOP-7320-2.patch

Fix the test that failed

> Refactor FsShell's copy & move commands
> ---------------------------------------
>
>                 Key: HADOOP-7320
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7320
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7320-2.patch, HADOOP-7320.patch
>
>
> Need to refactor the move and copy commands to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15293-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 21:36:28 2011
Return-Path: <common-issues-return-15293-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CF6CE4A65
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 21:36:28 +0000 (UTC)
Received: (qmail 14701 invoked by uid 500); 23 May 2011 21:36:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14598 invoked by uid 500); 23 May 2011 21:36:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14543 invoked by uid 99); 23 May 2011 21:36:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 21:36:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 21:36:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BB223DA387
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 21:35:47 +0000 (UTC)
Date: Mon, 23 May 2011 21:35:47 +0000 (UTC)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <769883795.37494.1306186547763.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <51041940.28155.1305838907965.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7310) Trash location needs to be
 revisited
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038247#comment-13038247 ] 

Chris Douglas commented on HADOOP-7310:
---------------------------------------

bq. Could we perhaps just make the trash prefix configurable in the same vain as mapreduce.jobtracker.staging.root.dir

This is solution B.

> Trash location needs to be revisited
> ------------------------------------
>
>                 Key: HADOOP-7310
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7310
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15294-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 21:40:28 2011
Return-Path: <common-issues-return-15294-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B86B54B21
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 21:40:28 +0000 (UTC)
Received: (qmail 17994 invoked by uid 500); 23 May 2011 21:40:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17970 invoked by uid 500); 23 May 2011 21:40:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17962 invoked by uid 99); 23 May 2011 21:40:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 21:40:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 21:40:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7C641DA478
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 21:39:47 +0000 (UTC)
Date: Mon, 23 May 2011 21:39:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1695574627.37499.1306186787506.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038249#comment-13038249 ] 

Todd Lipcon commented on HADOOP-7284:
-------------------------------------

- Looks like new files need license headers.
- there's an empty tearDown method
- is TestViewFsTrash.TestLFS used?

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15295-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 21:40:28 2011
Return-Path: <common-issues-return-15295-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DFECD4B2C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 21:40:28 +0000 (UTC)
Received: (qmail 18251 invoked by uid 500); 23 May 2011 21:40:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18179 invoked by uid 500); 23 May 2011 21:40:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18063 invoked by uid 99); 23 May 2011 21:40:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 21:40:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 21:40:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8A554DA47A
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 21:39:47 +0000 (UTC)
Date: Mon, 23 May 2011 21:39:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <144427131.37501.1306186787563.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <51041940.28155.1305838907965.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7310) Trash location needs to be
 revisited
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038250#comment-13038250 ] 

Todd Lipcon commented on HADOOP-7310:
-------------------------------------

If we go with a config, can we use a format string like "/user/%s/.Trash"? In my experience, the semantics of mapreduce.jobtracker.staging.root.dir have confused many users.

> Trash location needs to be revisited
> ------------------------------------
>
>                 Key: HADOOP-7310
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7310
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15296-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 21:43:39 2011
Return-Path: <common-issues-return-15296-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4D5CD4BED
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 21:43:39 +0000 (UTC)
Received: (qmail 24089 invoked by uid 500); 23 May 2011 21:43:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24048 invoked by uid 500); 23 May 2011 21:43:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24040 invoked by uid 99); 23 May 2011 21:43:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 21:43:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 21:43:36 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 11DC0DA5FF
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 21:42:56 +0000 (UTC)
Date: Mon, 23 May 2011 21:42:56 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <144701422.37543.1306186976068.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6929) RPC should have a way to pass
 Security information other than protocol annotations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038254#comment-13038254 ] 

Owen O'Malley commented on HADOOP-6929:
---------------------------------------

This isn't the right approach. In particular, you don't want to put class names in configuration and certainly don't want the new SecurityContext to *replace* the current one. We want to use the annotations if they exist and fall back on other mechanisms when they don't.

{code}
public abstract class SecurityInfo {
  public abstract KerberofInfo getKerberosInfo(Class<?> protocol);
  public abstract TokenInfo getTokenInfo(Class<?> protocol);
}

public class SecurityUtil {
  private static ServiceLoader<SecurityInfo> securityInfoProviders =
    new ServiceLoader<SecurityInfo>(SecurityInfo.class);
  public static KerberosInfo getKerberosInfo(Class<?> protocol) {
    for(SecurityInfo provider: securityInfoProviders) {
      Class<?> result = provider.getKerberosInfo(protocol);
      if (result != null) return result;
    }
    return null;
  }
  public static TokenInfo getTokenInfo(Class<?> protocol) {...
  }
}
{code}

The Hadoop jar can register the AnnotatedSecurityInfo as the default. If we wish to implement more than one in the default jar, we can define a StandardSecurityInfo that first checks AnnotatedSecurityInfo and then falls back to the second one.

> RPC should have a way to pass Security information other than protocol annotations
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6929
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6929
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: Hadoop-6929_v1.patch
>
>
> Currently Hadoop RPC allows protocol annotations as the only way to pass security information. This becomes a problem if protocols are generated and not hand written. For example protocols generated via Avro and passed over Avro tunnel (AvroRpcEngine.java) can't pass the security information.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15297-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 21:43:39 2011
Return-Path: <common-issues-return-15297-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 80C6E4BF8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 21:43:39 +0000 (UTC)
Received: (qmail 24332 invoked by uid 500); 23 May 2011 21:43:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24281 invoked by uid 500); 23 May 2011 21:43:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24169 invoked by uid 99); 23 May 2011 21:43:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 21:43:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 21:43:36 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 285D9DA60F
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 21:42:56 +0000 (UTC)
Date: Mon, 23 May 2011 21:42:56 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2128265590.37545.1306186976161.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <187742091.8722.1305246647362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7287) Configuration deprecation
 mechanism doesn't work properly for GenericOptionsParser/Tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038255#comment-13038255 ] 

Hudson commented on HADOOP-7287:
--------------------------------

Integrated in Hadoop-Common-22-branch #56 (See [https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/56/])
    HADOOP-7287. Configuration deprecation mechanism doesn't work properly for GenericOptionsParser and Tools. Contributed by Aaron T. Myers.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1126724
Files : 
* /hadoop/common/branches/branch-0.22/src/test/core/org/apache/hadoop/conf/TestConfigurationDeprecation.java
* /hadoop/common/branches/branch-0.22/src/test/test-fake-default.xml
* /hadoop/common/branches/branch-0.22/CHANGES.txt
* /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/conf/Configuration.java


> Configuration deprecation mechanism doesn't work properly for GenericOptionsParser/Tools
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7287
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7287
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Aaron T. Myers
>            Priority: Blocker
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: hadoop-7287-still-broken.txt, hadoop-7287-testcase.txt, hadoop-7287-trunk.0.patch, hadoop-7287-trunk.1.patch, hadoop-7287-trunk.2.patch
>
>
> For example, you can't use -D options on the "hadoop fs" command line in order to specify the deprecated names of configuration options. The issue is that the ordering is:
> - JVM starts
> - GenericOptionsParser creates a Configuration object and calls set() for each of the options specified on command line
> - DistributedFileSystem or other class eventually instantiates HdfsConfiguration which adds the deprecations
> - Some class calls conf.get("new key") and sees the default instead of the version set on the command line

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15298-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 21:43:39 2011
Return-Path: <common-issues-return-15298-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BB1504C0F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 21:43:39 +0000 (UTC)
Received: (qmail 24560 invoked by uid 500); 23 May 2011 21:43:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24521 invoked by uid 500); 23 May 2011 21:43:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24479 invoked by uid 99); 23 May 2011 21:43:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 21:43:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 21:43:37 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5B9DADA617
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 21:42:56 +0000 (UTC)
Date: Mon, 23 May 2011 21:42:56 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <314002947.37550.1306186976371.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1828723839.36128.1306162247409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7320) Refactor FsShell's copy & move
 commands
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038256#comment-13038256 ] 

Hadoop QA commented on HADOOP-7320:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480166/HADOOP-7320-2.patch
  against trunk revision 1126719.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 10 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/508//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/508//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/508//console

This message is automatically generated.

> Refactor FsShell's copy & move commands
> ---------------------------------------
>
>                 Key: HADOOP-7320
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7320
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7320-2.patch, HADOOP-7320.patch
>
>
> Need to refactor the move and copy commands to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15299-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 21:45:30 2011
Return-Path: <common-issues-return-15299-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7F1624FE2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 21:45:30 +0000 (UTC)
Received: (qmail 27044 invoked by uid 500); 23 May 2011 21:45:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27014 invoked by uid 500); 23 May 2011 21:45:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27006 invoked by uid 99); 23 May 2011 21:45:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 21:45:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 21:45:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5D874DA6B8
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 21:44:47 +0000 (UTC)
Date: Mon, 23 May 2011 21:44:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2019813364.37554.1306187087379.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <51041940.28155.1305838907965.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7310) Trash location needs to be
 revisited
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038257#comment-13038257 ] 

Aaron T. Myers commented on HADOOP-7310:
----------------------------------------

bq. This is solution B.

Good point. :P

Please interpret my comment, then, as pointing out that making this change optionally backward compatible is pretty trivial, so why not? Some admins will be unhappy about this incompatible change, while few if any will be unhappy about an extra configuration variable they need not concern themselves with.

Also, +1 to Todd's suggestion of a format string versus straight prefix.

> Trash location needs to be revisited
> ------------------------------------
>
>                 Key: HADOOP-7310
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7310
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15300-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 21:47:30 2011
Return-Path: <common-issues-return-15300-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ACB364F4B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 21:47:30 +0000 (UTC)
Received: (qmail 30454 invoked by uid 500); 23 May 2011 21:47:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30425 invoked by uid 500); 23 May 2011 21:47:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30417 invoked by uid 99); 23 May 2011 21:47:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 21:47:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 21:47:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7D0CCDA807
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 21:46:47 +0000 (UTC)
Date: Mon, 23 May 2011 21:46:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <73001203.37562.1306187207508.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1195662636.6130.1300891805766.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7208) equals() and hashCode()
 implementation need to change in StandardSocketFactory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038259#comment-13038259 ] 

Todd Lipcon commented on HADOOP-7208:
-------------------------------------

- please use spaces, not tabs for indentation
- should probably remove the comment "dummy hash code"
- in tests, please use assertEquals instead of assertTrue with an == condition. Same for the false assertion
- to illustrate the bug, does DummySocketFactory need to override anything at all? I think it could just be an empty subclass. Also, the attached javadoc doesn't represent what it does

> equals() and hashCode() implementation need to change in StandardSocketFactory
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7208
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7208
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7208.patch
>
>
>   In Hadoop IPC Client, we are using ClientCache which will maintain the HashMap to keep the Client references.
> private Map<SocketFactory, Client> clients =
>       new HashMap<SocketFactory, Client>();
>  Now let us say, we want use two standard factories with Hadoop. MyStandardSocketFactory (which extends StandardSocketFactory), and StandardSocketFactory. In this case, because of equals and hashcode implementation, MyStandardSocketFactory client can be overridden by StandardSocketFactoryClient

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15301-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 21:58:28 2011
Return-Path: <common-issues-return-15301-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 61F9E4B4C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 21:58:28 +0000 (UTC)
Received: (qmail 60224 invoked by uid 500); 23 May 2011 21:58:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60184 invoked by uid 500); 23 May 2011 21:58:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60176 invoked by uid 99); 23 May 2011 21:58:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 21:58:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 21:58:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 699CEDAB6E
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 21:57:47 +0000 (UTC)
Date: Mon, 23 May 2011 21:57:47 +0000 (UTC)
From: "Bharath Mundlapudi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1067116917.37592.1306187867429.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7322) Adding a util method in FileUtil for
 directory listing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Adding a util method in FileUtil for directory listing
------------------------------------------------------

                 Key: HADOOP-7322
                 URL: https://issues.apache.org/jira/browse/HADOOP-7322
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Bharath Mundlapudi
            Assignee: Bharath Mundlapudi
            Priority: Minor
             Fix For: 0.23.0


While testing Disk Fail Inplace, we encountered lots of NPE from Dir.listFiles API. This API can return null when Dir is not directory or disk is bad. I am proposing to have a File Util which can be used consistently across to deal with disk issues. This util api will do the following:

1. When error happens it will throw IOException
2. Else it will return empty list or list of files.

Signature:
File[] FileUtil.listFiles(File dir) throws IOException {}

This way we no need to write wrapper code every where. Also, API is consistent with the signature.
 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15302-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 21:58:32 2011
Return-Path: <common-issues-return-15302-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 87AB04B85
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 21:58:32 +0000 (UTC)
Received: (qmail 60872 invoked by uid 500); 23 May 2011 21:58:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60811 invoked by uid 500); 23 May 2011 21:58:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60647 invoked by uid 99); 23 May 2011 21:58:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 21:58:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 21:58:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7A6BDDAB70
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 21:57:47 +0000 (UTC)
Date: Mon, 23 May 2011 21:57:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2045123227.37594.1306187867498.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <889607684.27597.1305830207722.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7307) Remove Dr Who references
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038265#comment-13038265 ] 

Owen O'Malley commented on HADOOP-7307:
---------------------------------------

Allen, it only happens in the web ui when there isn't a better auth module plugged in.

> Remove Dr Who references
> ------------------------
>
>                 Key: HADOOP-7307
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7307
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>             Fix For: 0.22.0
>
>
> I've had feedback from a number of different operators that the "dr who" references that occasionally come out of Hadoop are annoying and confusing. We should replace them with more descriptive error messages, usernames, etc.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15303-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 22:02:30 2011
Return-Path: <common-issues-return-15303-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AFC83482B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 22:02:30 +0000 (UTC)
Received: (qmail 67545 invoked by uid 500); 23 May 2011 22:02:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67513 invoked by uid 500); 23 May 2011 22:02:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67505 invoked by uid 99); 23 May 2011 22:02:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 22:02:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 22:02:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7DE7DDACA9
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 22:01:47 +0000 (UTC)
Date: Mon, 23 May 2011 22:01:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1906682438.37613.1306188107512.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1828723839.36128.1306162247409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7320) Refactor FsShell's copy & move
 commands
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038267#comment-13038267 ] 

Todd Lipcon commented on HADOOP-7320:
-------------------------------------

- Can you explain the changes with regard to RawLocalFileSystem vs LocalFileSystem in TestFsShellReturnCode? I'm not sure I follow what's going on here.
- typo "pathson" -> "paths on" for Rename
- maybe rename the outer "Copy" class to "CopyCommands" since it's just a container? Also, should it really be extending FsCommand since it is not itself a command? (I realize these are left over from HADOOP-7251... can address in a separate JIRA if you prefer)

> Refactor FsShell's copy & move commands
> ---------------------------------------
>
>                 Key: HADOOP-7320
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7320
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7320-2.patch, HADOOP-7320.patch
>
>
> Need to refactor the move and copy commands to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15304-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 22:24:29 2011
Return-Path: <common-issues-return-15304-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 234604FD1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 22:24:29 +0000 (UTC)
Received: (qmail 3392 invoked by uid 500); 23 May 2011 22:24:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3362 invoked by uid 500); 23 May 2011 22:24:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3353 invoked by uid 99); 23 May 2011 22:24:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 22:24:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 22:24:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1643FDA3AD
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 22:23:48 +0000 (UTC)
Date: Mon, 23 May 2011 22:23:48 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <714303163.37672.1306189428087.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6929) RPC should have a way to pass
 Security information other than protocol annotations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038279#comment-13038279 ] 

Luke Lu commented on HADOOP-6929:
---------------------------------

bq. In particular, you don't want to put class names in configuration

This is a pervasive anti-pattern used in hadoop all over the place, HADOOP-7150 is supposed to address that.

bq. and certainly don't want the new SecurityContext to replace the current one.

Agreed. This is a major flaw of the current patch, though the flawed mechanism is still workable if the new security info implements the fallback mechanism.

bq. private static ServiceLoader<SecurityInfo> securityInfoProviders = new ServiceLoader<SecurityInfo>(SecurityInfo.class);

The usage should be:
{code}
ServiceLoader<SecurityInfo> securityInfoProviders = ServiceLoader.load(SecurityInfo.class);
{code}

> RPC should have a way to pass Security information other than protocol annotations
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6929
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6929
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: Hadoop-6929_v1.patch
>
>
> Currently Hadoop RPC allows protocol annotations as the only way to pass security information. This becomes a problem if protocols are generated and not hand written. For example protocols generated via Avro and passed over Avro tunnel (AvroRpcEngine.java) can't pass the security information.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15305-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 22:26:28 2011
Return-Path: <common-issues-return-15305-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6AACA412E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 22:26:28 +0000 (UTC)
Received: (qmail 6812 invoked by uid 500); 23 May 2011 22:26:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6779 invoked by uid 500); 23 May 2011 22:26:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6770 invoked by uid 99); 23 May 2011 22:26:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 22:26:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 22:26:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6C161DA45B
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 22:25:47 +0000 (UTC)
Date: Mon, 23 May 2011 22:25:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <739161311.37680.1306189547439.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1828723839.36128.1306162247409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7320) Refactor FsShell's copy & move
 commands
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038280#comment-13038280 ] 

Daryn Sharp commented on HADOOP-7320:
-------------------------------------

bq. Can you explain the changes with regard to RawLocalFileSystem vs LocalFileSystem in TestFsShellReturnCode? I'm not sure I follow what's going on here.

It was confusing for me too.  The property fs.file.impl is supposed to be a LocalFileSystem, but the test was defining it as a RawLocalFileSystem.  Calls to FileSystem.getLocal() instantiate the fs.file.impl class and then cast it to a LocalFileSystem (runtime kaboom).  Using the wrong class accidentally worked until this patch.

I'll address the last 2 bullet points on this jira.

> Refactor FsShell's copy & move commands
> ---------------------------------------
>
>                 Key: HADOOP-7320
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7320
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7320-2.patch, HADOOP-7320.patch
>
>
> Need to refactor the move and copy commands to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15306-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 22:40:30 2011
Return-Path: <common-issues-return-15306-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 065134A85
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 22:40:30 +0000 (UTC)
Received: (qmail 21609 invoked by uid 500); 23 May 2011 22:40:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21543 invoked by uid 500); 23 May 2011 22:40:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21535 invoked by uid 99); 23 May 2011 22:40:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 22:40:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 22:40:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EBC9DDA91F
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 22:39:48 +0000 (UTC)
Date: Mon, 23 May 2011 22:39:48 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <367060043.37743.1306190388962.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-6255:
------------------------------

    Attachment: HADOOP-6255-common-trunk-7.patch

Change configuration directory from $PREFIX/conf to $PREFIX/etc/hadoop per Owen's recommendation.  For RPM/deb, it will use /etc/hadoop as default, and create symlink for $PREFIX/etc/hadoop point to /etc/hadoop.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-10.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk-4.patch, HADOOP-6255-common-trunk-5.patch, HADOOP-6255-common-trunk-6.patch, HADOOP-6255-common-trunk-7.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-6255-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15307-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 22:42:32 2011
Return-Path: <common-issues-return-15307-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C87D04B53
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 22:42:32 +0000 (UTC)
Received: (qmail 30414 invoked by uid 500); 23 May 2011 22:42:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30373 invoked by uid 500); 23 May 2011 22:42:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30365 invoked by uid 99); 23 May 2011 22:42:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 22:42:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 22:42:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C7EA7DAA47
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 22:41:48 +0000 (UTC)
Date: Mon, 23 May 2011 22:41:48 +0000 (UTC)
From: "Amr Awadallah (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1126777029.37782.1306190508815.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038290#comment-13038290 ] 

Amr Awadallah commented on HADOOP-6255:
---------------------------------------

I am out of office on an international business trip this week. I will
be slower than usual in responding to emails. If this is urgent then
please call my cell phone (or send an SMS), otherwise I will reply to
your email when I get back.

Thanks for your patience,

-- amr


> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-10.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk-4.patch, HADOOP-6255-common-trunk-5.patch, HADOOP-6255-common-trunk-6.patch, HADOOP-6255-common-trunk-7.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-6255-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15308-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 22:46:32 2011
Return-Path: <common-issues-return-15308-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1003A4822
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 22:46:32 +0000 (UTC)
Received: (qmail 38581 invoked by uid 500); 23 May 2011 22:46:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38549 invoked by uid 500); 23 May 2011 22:46:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38541 invoked by uid 99); 23 May 2011 22:46:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 22:46:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 22:46:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D14F3DABF6
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 22:45:48 +0000 (UTC)
Date: Mon, 23 May 2011 22:45:48 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1340967858.37828.1306190748854.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6255:
--------------------------------

    Comment: was deleted

(was: I am out of office on an international business trip this week. I will
be slower than usual in responding to emails. If this is urgent then
please call my cell phone (or send an SMS), otherwise I will reply to
your email when I get back.

Thanks for your patience,

-- amr
)

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-10.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk-4.patch, HADOOP-6255-common-trunk-5.patch, HADOOP-6255-common-trunk-6.patch, HADOOP-6255-common-trunk-7.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-6255-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15309-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 22:52:28 2011
Return-Path: <common-issues-return-15309-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E7B5C46D1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 22:52:28 +0000 (UTC)
Received: (qmail 45922 invoked by uid 500); 23 May 2011 22:52:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45893 invoked by uid 500); 23 May 2011 22:52:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45881 invoked by uid 99); 23 May 2011 22:52:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 22:52:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 22:52:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id ED0DADAE25
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 22:51:47 +0000 (UTC)
Date: Mon, 23 May 2011 22:51:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <725078979.37843.1306191107967.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <360710663.20918.1304489523226.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7259) contrib modules should include
 build.properties from parent.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley resolved HADOOP-7259.
-----------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]

I just committed this.

> contrib modules should include build.properties from parent.
> ------------------------------------------------------------
>
>                 Key: HADOOP-7259
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7259
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: contrib.patch
>
>
> Current build.properties in the hadoop root directory is not included by the contrib modules.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15310-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 22:54:31 2011
Return-Path: <common-issues-return-15310-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D73E147D9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 22:54:31 +0000 (UTC)
Received: (qmail 53964 invoked by uid 500); 23 May 2011 22:54:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53924 invoked by uid 500); 23 May 2011 22:54:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53914 invoked by uid 99); 23 May 2011 22:54:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 22:54:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 22:54:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0C7FDDAF88
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 22:53:48 +0000 (UTC)
Date: Mon, 23 May 2011 22:53:48 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <252913023.37856.1306191228048.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038300#comment-13038300 ] 

Hadoop QA commented on HADOOP-6255:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480177/HADOOP-6255-common-trunk-7.patch
  against trunk revision 1126719.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 27 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    -1 release audit.  The applied patch generated 3 release audit warnings (more than the trunk's current 1 warnings).

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/509//testReport/
Release audit warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/509//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/509//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/509//console

This message is automatically generated.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-10.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk-4.patch, HADOOP-6255-common-trunk-5.patch, HADOOP-6255-common-trunk-6.patch, HADOOP-6255-common-trunk-7.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-6255-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15311-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 22:56:28 2011
Return-Path: <common-issues-return-15311-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 58467480E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 22:56:28 +0000 (UTC)
Received: (qmail 54658 invoked by uid 500); 23 May 2011 22:56:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54618 invoked by uid 500); 23 May 2011 22:56:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54609 invoked by uid 99); 23 May 2011 22:56:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 22:56:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 22:56:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 60CD6DA034
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 22:55:47 +0000 (UTC)
Date: Mon, 23 May 2011 22:55:47 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <654871262.37888.1306191347393.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7323) Add capability to resolve
 compression codec based on codec name
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Add capability to resolve compression codec based on codec name
---------------------------------------------------------------

                 Key: HADOOP-7323
                 URL: https://issues.apache.org/jira/browse/HADOOP-7323
             Project: Hadoop Common
          Issue Type: Improvement
          Components: io
    Affects Versions: 0.21.0
            Reporter: Alejandro Abdelnur
            Assignee: Alejandro Abdelnur
             Fix For: 0.22.0


When setting up a compression codec in an MR job the full class name of the codec must be used.

To ease usability, compression codecs should be resolved by their codec name (ie 'gzip', 'deflate', 'zlib', 'bzip2') instead their full codec class name.

Besides easy of use for Hadoop users who would use the codec alias instead the full codec class name, it could simplify how HBase resolves loads the codecs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15312-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 23:06:28 2011
Return-Path: <common-issues-return-15312-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B96564045
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 23:06:28 +0000 (UTC)
Received: (qmail 63870 invoked by uid 500); 23 May 2011 23:06:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63812 invoked by uid 500); 23 May 2011 23:06:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63711 invoked by uid 99); 23 May 2011 23:06:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 23:06:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 23:06:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 84DDFDA34A
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 23:05:47 +0000 (UTC)
Date: Mon, 23 May 2011 23:05:47 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <108701884.37911.1306191947540.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <298707236.22802.1301530265774.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7215) RPC clients must connect over a
 network interface corresponding to the host name in the client's kerberos
 principal key
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-7215:
------------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

The patch is committed.

> RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7215
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>            Priority: Blocker
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: HADOOP-7215.1.trunk.patch, HADOOP-7215.2.trunk.patch, HADOOP-7215.2xx.patch, HADOOP-7215.3.trunk.patch, HADOOP-7215.debug.patch, HADOOP-7215.debug.patch
>
>
> HDFS-7104 introduced a change where RPC server matches client's hostname with the hostname specified in the client's Kerberos principal name. RPC client binds the socket to a random local address, which might not match the hostname specified in the principal name. This results authorization failure of the client at the server.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15313-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 23:14:28 2011
Return-Path: <common-issues-return-15313-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B7A7341D3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 23:14:28 +0000 (UTC)
Received: (qmail 73023 invoked by uid 500); 23 May 2011 23:14:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72996 invoked by uid 500); 23 May 2011 23:14:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72987 invoked by uid 99); 23 May 2011 23:14:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 23:14:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 23:14:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A3962DA50C
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 23:13:47 +0000 (UTC)
Date: Mon, 23 May 2011 23:13:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <982284402.37932.1306192427666.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1828723839.36128.1306162247409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7320) Refactor FsShell's copy & move
 commands
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7320:
--------------------------------

    Attachment: HADOOP-7320-3.patch

* Fix "pathson" misspelling.
* Rename Copy to CopyCommands, ditto for Move.
* Removed "extends" from Copy/MoveCommands.  It was a leftover from when I had common methods in the class, prior to moving them to CommandWithDestination.

> Refactor FsShell's copy & move commands
> ---------------------------------------
>
>                 Key: HADOOP-7320
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7320
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7320-2.patch, HADOOP-7320-3.patch, HADOOP-7320.patch
>
>
> Need to refactor the move and copy commands to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15314-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 23:20:28 2011
Return-Path: <common-issues-return-15314-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AAECB48BE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 23:20:28 +0000 (UTC)
Received: (qmail 78139 invoked by uid 500); 23 May 2011 23:20:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78112 invoked by uid 500); 23 May 2011 23:20:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78104 invoked by uid 99); 23 May 2011 23:20:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 23:20:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 23:20:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8236EDA6D4
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 23:19:47 +0000 (UTC)
Date: Mon, 23 May 2011 23:19:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1082485394.37949.1306192787530.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <360710663.20918.1304489523226.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7259) contrib modules should include
 build.properties from parent.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038314#comment-13038314 ] 

Hudson commented on HADOOP-7259:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk-Commit #697 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/697/])
    HADOOP-7259. Contrib modules should include the build.properties from
the enclosing hadoop directory. (omalley)

omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1126801
Files : 
* /hadoop/mapreduce/trunk/CHANGES.txt
* /hadoop/mapreduce/trunk/src/contrib/build-contrib.xml


> contrib modules should include build.properties from parent.
> ------------------------------------------------------------
>
>                 Key: HADOOP-7259
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7259
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: contrib.patch
>
>
> Current build.properties in the hadoop root directory is not included by the contrib modules.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15315-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 23:24:31 2011
Return-Path: <common-issues-return-15315-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4730F493B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 23:24:31 +0000 (UTC)
Received: (qmail 81409 invoked by uid 500); 23 May 2011 23:24:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81362 invoked by uid 500); 23 May 2011 23:24:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81354 invoked by uid 99); 23 May 2011 23:24:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 23:24:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 23:24:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 64303DA7FA
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 23:23:47 +0000 (UTC)
Date: Mon, 23 May 2011 23:23:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2096537703.37957.1306193027407.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1828723839.36128.1306162247409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7320) Refactor FsShell's copy & move
 commands
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038316#comment-13038316 ] 

Hadoop QA commented on HADOOP-7320:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480183/HADOOP-7320-3.patch
  against trunk revision 1126719.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 10 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/510//console

This message is automatically generated.

> Refactor FsShell's copy & move commands
> ---------------------------------------
>
>                 Key: HADOOP-7320
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7320
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7320-2.patch, HADOOP-7320-3.patch, HADOOP-7320.patch
>
>
> Need to refactor the move and copy commands to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15316-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 23:26:30 2011
Return-Path: <common-issues-return-15316-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CD676495E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 23:26:30 +0000 (UTC)
Received: (qmail 83320 invoked by uid 500); 23 May 2011 23:26:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83287 invoked by uid 500); 23 May 2011 23:26:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83279 invoked by uid 99); 23 May 2011 23:26:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 23:26:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 23:26:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 849A3DA84B
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 23:25:47 +0000 (UTC)
Date: Mon, 23 May 2011 23:25:47 +0000 (UTC)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1245077827.37960.1306193147539.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1145070693.28819.1305845387456.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7311) Port remaining metrics v1 from
 trunk to branch-0.20-security
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038318#comment-13038318 ] 

Konstantin Shvachko commented on HADOOP-7311:
---------------------------------------------

Luke, I don't know how you can make decisions based only on one comment. I don't think Gary (the only one who cared) would be happy with the solution, when he sees it, as he was asking for "a phased transition" to metrics2 for HBase. I sure missed your email.

Let's see how we can use metrics2 instead of the original metrics, before we attempt option #3, which is make both metrics frameworks work.

Could you please elaborate or post links on how to get metrics2 run. We use Ganglia.

> Port remaining metrics v1 from trunk to branch-0.20-security
> ------------------------------------------------------------
>
>                 Key: HADOOP-7311
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7311
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.203.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: 0.20.204.0
>
>         Attachments: metrics-0.20-security.patch
>
>
> HADOOP-7190 added metrics packages/classes. This is a port from trunk to make them actually work for the security branch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15317-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 23:28:31 2011
Return-Path: <common-issues-return-15317-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E6885497B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 23:28:30 +0000 (UTC)
Received: (qmail 83771 invoked by uid 500); 23 May 2011 23:28:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83739 invoked by uid 500); 23 May 2011 23:28:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83729 invoked by uid 99); 23 May 2011 23:28:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 23:28:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 23:28:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 57259DA8AF
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 23:27:47 +0000 (UTC)
Date: Mon, 23 May 2011 23:27:47 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <775122501.37965.1306193267353.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <654871262.37888.1306191347393.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7323) Add capability to resolve
 compression codec based on codec name
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-7323:
---------------------------------------

    Attachment: HADOOP-7323.patch

The codec alias is taken from the Codec short class name (classname without package) after removing the 'Codec' postfix if present, else the full short class name is used as alias.

The patch adds to the CompressionCodecFactory.getCodecByClassName()method an alias lookup fallback if the codec class has not been found (this preserves existing behavior).

The alias lookup is case-insensitive.


> Add capability to resolve compression codec based on codec name
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7323
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7323
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7323.patch
>
>
> When setting up a compression codec in an MR job the full class name of the codec must be used.
> To ease usability, compression codecs should be resolved by their codec name (ie 'gzip', 'deflate', 'zlib', 'bzip2') instead their full codec class name.
> Besides easy of use for Hadoop users who would use the codec alias instead the full codec class name, it could simplify how HBase resolves loads the codecs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15318-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 23:40:31 2011
Return-Path: <common-issues-return-15318-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CD9034FBA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 23:40:31 +0000 (UTC)
Received: (qmail 88102 invoked by uid 500); 23 May 2011 23:40:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88047 invoked by uid 500); 23 May 2011 23:40:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88032 invoked by uid 99); 23 May 2011 23:40:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 23:40:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 23:40:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 77935DAB70
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 23:39:47 +0000 (UTC)
Date: Mon, 23 May 2011 23:39:47 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1727435655.37974.1306193987486.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-6255:
------------------------------

    Attachment: HADOOP-6255-branch-0.20-security-11.patch

Same $PREFIX/etc/hadoop configuration change as trunk but for 0.20 security branch.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-10.patch, HADOOP-6255-branch-0.20-security-11.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk-4.patch, HADOOP-6255-common-trunk-5.patch, HADOOP-6255-common-trunk-6.patch, HADOOP-6255-common-trunk-7.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-6255-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15319-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 23:44:31 2011
Return-Path: <common-issues-return-15319-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E83144005
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 23:44:30 +0000 (UTC)
Received: (qmail 89875 invoked by uid 500); 23 May 2011 23:44:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89824 invoked by uid 500); 23 May 2011 23:44:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89723 invoked by uid 99); 23 May 2011 23:44:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 23:44:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 23:44:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A6A32DACC0
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 23:43:48 +0000 (UTC)
Date: Mon, 23 May 2011 23:43:48 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <557060708.38037.1306194228679.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038323#comment-13038323 ] 

Hadoop QA commented on HADOOP-6255:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480186/HADOOP-6255-branch-0.20-security-11.patch
  against trunk revision 1126719.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 1 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/511//console

This message is automatically generated.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-10.patch, HADOOP-6255-branch-0.20-security-11.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk-4.patch, HADOOP-6255-common-trunk-5.patch, HADOOP-6255-common-trunk-6.patch, HADOOP-6255-common-trunk-7.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-6255-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15320-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 23 23:51:28 2011
Return-Path: <common-issues-return-15320-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 633A04665
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 23 May 2011 23:51:28 +0000 (UTC)
Received: (qmail 92830 invoked by uid 500); 23 May 2011 23:51:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92802 invoked by uid 500); 23 May 2011 23:51:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92794 invoked by uid 99); 23 May 2011 23:51:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 23:51:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2011 23:51:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5615CDAE1B
	for <common-issues@hadoop.apache.org>; Mon, 23 May 2011 23:50:47 +0000 (UTC)
Date: Mon, 23 May 2011 23:50:47 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <16042182.38051.1306194647349.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <654871262.37888.1306191347393.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7323) Add capability to resolve
 compression codec based on codec name
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038326#comment-13038326 ] 

Tom White commented on HADOOP-7323:
-----------------------------------

This looks good. The only nit I noticed is that the javadoc on codecsByName in CompressionCodecFactory is incorrect.

> Add capability to resolve compression codec based on codec name
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7323
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7323
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7323.patch
>
>
> When setting up a compression codec in an MR job the full class name of the codec must be used.
> To ease usability, compression codecs should be resolved by their codec name (ie 'gzip', 'deflate', 'zlib', 'bzip2') instead their full codec class name.
> Besides easy of use for Hadoop users who would use the codec alias instead the full codec class name, it could simplify how HBase resolves loads the codecs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15321-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 00:17:31 2011
Return-Path: <common-issues-return-15321-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 06D1F49F6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 00:17:31 +0000 (UTC)
Received: (qmail 13963 invoked by uid 500); 24 May 2011 00:17:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13926 invoked by uid 500); 24 May 2011 00:17:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13918 invoked by uid 99); 24 May 2011 00:17:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 00:17:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 00:17:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5D615DA393
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 00:16:47 +0000 (UTC)
Date: Tue, 24 May 2011 00:16:47 +0000 (UTC)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <991224987.38069.1306196207379.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1145070693.28819.1305845387456.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7311) Port remaining metrics v1 from
 trunk to branch-0.20-security
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038331#comment-13038331 ] 

Konstantin Shvachko commented on HADOOP-7311:
---------------------------------------------

I should've mentioned, I haven't been following metrics closely, so any help from you, Luke, is greatly appreciated.

> Port remaining metrics v1 from trunk to branch-0.20-security
> ------------------------------------------------------------
>
>                 Key: HADOOP-7311
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7311
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.203.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: 0.20.204.0
>
>         Attachments: metrics-0.20-security.patch
>
>
> HADOOP-7190 added metrics packages/classes. This is a port from trunk to make them actually work for the security branch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15322-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 00:19:28 2011
Return-Path: <common-issues-return-15322-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A090443C6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 00:19:28 +0000 (UTC)
Received: (qmail 16456 invoked by uid 500); 24 May 2011 00:19:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16426 invoked by uid 500); 24 May 2011 00:19:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16418 invoked by uid 99); 24 May 2011 00:19:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 00:19:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 00:19:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5D2E8DA406
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 00:18:47 +0000 (UTC)
Date: Tue, 24 May 2011 00:18:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2349015.38074.1306196327378.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1145070693.28819.1305845387456.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7311) Port remaining metrics v1 from
 trunk to branch-0.20-security
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038332#comment-13038332 ] 

Luke Lu commented on HADOOP-7311:
---------------------------------

bq. Luke, I don't know how you can make decisions based only on one comment.

Well, it was approved by Y dev/ops, of course :). The post was sent to both general and common-dev. People have many weeks to voice their opinions. The main consumers of the metrics system are ops and perf guys. I guess that they don't have strong opinions on how it should work as long as it can be easily made to work. I'm sure that you heard of lazy consensus.

bq. I don't think Gary (the only one who cared) would be happy with the solution, when he sees it, as he was asking for "a phased transition" to metrics2 for HBase.

#2 the OP clearly outlined the pros and cons of the approach. HBase will continue sending their specific metrics with metrics1 and Hadoop (common, hdfs, mapreduce) will send hadoop framework metrics with metrics2. i.e., in the transition period, ops would need to configure both hadoop-metrics.properties for HBase and hadoop-metrics2.properties for Hadoop. It's a working solution for the transition period.

bq. Could you please elaborate or post links on how to get metrics2 run. We use Ganglia.

The syntax to configure a consumer/sink for metrics (in the hadoop-metrics2.properties):
{noformat}
[service-name].sink.[sink-name].[option]=value
{noformat}
Some examples are included in the trunk conf/hadoop-metrics2.properties for common and hdfs, as well as the [configuration section|http://wiki.apache.org/hadoop/HADOOP-6728-MetricsV2#Configuration] of the [design doc|http://wiki.apache.org/hadoop/HADOOP-6728-MetricsV2].

An example of metrics2 config using ganglia (3.0.x or 3.1.x?) would be:
{noformat}
*.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.Ganglia31
*.period=10

namenode.sink.ganglia.address=localhost:8649
jobtracker.sink.ganglia.address=localhost:8649
tasktracker.sink.ganglia.address=localhost:8649
{noformat}

Of course, the org.apache.hadoop.metrics2.sink.ganglia.* classes don't exist yet. We'll have to implement them (sink examples can be found in the FileSink or the EchoPlugin in the design doc.).


> Port remaining metrics v1 from trunk to branch-0.20-security
> ------------------------------------------------------------
>
>                 Key: HADOOP-7311
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7311
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.203.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: 0.20.204.0
>
>         Attachments: metrics-0.20-security.patch
>
>
> HADOOP-7190 added metrics packages/classes. This is a port from trunk to make them actually work for the security branch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15323-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 00:23:30 2011
Return-Path: <common-issues-return-15323-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A08FC4AC2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 00:23:30 +0000 (UTC)
Received: (qmail 20161 invoked by uid 500); 24 May 2011 00:23:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20122 invoked by uid 500); 24 May 2011 00:23:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20114 invoked by uid 99); 24 May 2011 00:23:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 00:23:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 00:23:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 648E2DA4CA
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 00:22:47 +0000 (UTC)
Date: Tue, 24 May 2011 00:22:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1956630411.38079.1306196567408.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1145070693.28819.1305845387456.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7311) Port remaining metrics v1 from
 trunk to branch-0.20-security
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038333#comment-13038333 ] 

Luke Lu commented on HADOOP-7311:
---------------------------------

bq. I should've mentioned, I haven't been following metrics closely, so any help from you, Luke, is greatly appreciated.

No problem. I'm committed to make the transition as painless as possible :)

> Port remaining metrics v1 from trunk to branch-0.20-security
> ------------------------------------------------------------
>
>                 Key: HADOOP-7311
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7311
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.203.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: 0.20.204.0
>
>         Attachments: metrics-0.20-security.patch
>
>
> HADOOP-7190 added metrics packages/classes. This is a port from trunk to make them actually work for the security branch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15324-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 00:47:30 2011
Return-Path: <common-issues-return-15324-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5C71249B5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 00:47:30 +0000 (UTC)
Received: (qmail 41816 invoked by uid 500); 24 May 2011 00:47:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41789 invoked by uid 500); 24 May 2011 00:47:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41774 invoked by uid 99); 24 May 2011 00:47:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 00:47:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 00:47:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6FA51DA944
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 00:46:47 +0000 (UTC)
Date: Tue, 24 May 2011 00:46:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <557825167.38099.1306198007453.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7324) Ganglia plugins for metrics v2
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Ganglia plugins for metrics v2
------------------------------

                 Key: HADOOP-7324
                 URL: https://issues.apache.org/jira/browse/HADOOP-7324
             Project: Hadoop Common
          Issue Type: New Feature
          Components: metrics
    Affects Versions: 0.20.203.0, 0.23.0
            Reporter: Luke Lu
             Fix For: 0.20.204.0, 0.23.0


Although, all metrics in metrics v2 are exposed via the standard JMX mechanisms, there are a few users who prefer using Ganglia to collect metrics.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15325-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 01:23:28 2011
Return-Path: <common-issues-return-15325-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AC4B04694
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 01:23:28 +0000 (UTC)
Received: (qmail 88782 invoked by uid 500); 24 May 2011 01:23:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88756 invoked by uid 500); 24 May 2011 01:23:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88748 invoked by uid 99); 24 May 2011 01:23:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 01:23:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 01:23:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8495EDA108
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 01:22:47 +0000 (UTC)
Date: Tue, 24 May 2011 01:22:47 +0000 (UTC)
From: "Brock Noland (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1387497904.38167.1306200167539.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1160101136.38164.1306200167436.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7325) hadoop command - do not accept class
 names starting with a hyphen
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Brock Noland updated HADOOP-7325:
---------------------------------

    Attachment: hadoop-illegal-class-name-0.patch

Attached implements requested improvement.

> hadoop command - do not accept class names starting with a hyphen
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7325
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Brock Noland
>            Priority: Minor
>         Attachments: hadoop-illegal-class-name-0.patch
>
>
> If this is committed I will look at patches for hdfs and mapred.
> When teaching a good portion of the students in every single class execute:
> {code}
> $ hadoop -fs ls /
> {code}
> The -fs is passed directly to the JVM and the JVM fails to start:
> {code}
> $ ./bin/hadoop -fs ls /
> Unrecognized option: -fs
> Could not create the Java virtual machine.
> {code}
> Which is confusing and typically requires explanation. The attached patch improves that behavior:
> {code}
> $ ./bin/hadoop -fs ls /
> Error: No command named `-fs' was found. Perhaps you meant `hadoop fs'
> {code}
> The only risk I can see is if someone is abusing the implementation of hadoop command doing something like so:
> {code}
> $ ./bin/hadoop -Xmx1g  org.apache.hadoop.util.RunJar
> RunJar jarFile [mainClass] args...
> {code}
> The hadoop command does not appear to advertise allowing JVM options before the classname.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15326-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 01:23:30 2011
Return-Path: <common-issues-return-15326-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AEBC246B0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 01:23:30 +0000 (UTC)
Received: (qmail 89435 invoked by uid 500); 24 May 2011 01:23:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89397 invoked by uid 500); 24 May 2011 01:23:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89389 invoked by uid 99); 24 May 2011 01:23:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 01:23:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 01:23:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6B6BCDA105
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 01:22:47 +0000 (UTC)
Date: Tue, 24 May 2011 01:22:47 +0000 (UTC)
From: "Brock Noland (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1160101136.38164.1306200167436.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7325) hadoop command - do not accept class
 names starting with a hyphen
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

hadoop command - do not accept class names starting with a hyphen
-----------------------------------------------------------------

                 Key: HADOOP-7325
                 URL: https://issues.apache.org/jira/browse/HADOOP-7325
             Project: Hadoop Common
          Issue Type: Improvement
          Components: scripts
            Reporter: Brock Noland
            Priority: Minor
         Attachments: hadoop-illegal-class-name-0.patch

If this is committed I will look at patches for hdfs and mapred.

When teaching a good portion of the students in every single class execute:

{code}
$ hadoop -fs ls /
{code}

The -fs is passed directly to the JVM and the JVM fails to start:

{code}
$ ./bin/hadoop -fs ls /
Unrecognized option: -fs
Could not create the Java virtual machine.
{code}

Which is confusing and typically requires explanation. The attached patch improves that behavior:

{code}
$ ./bin/hadoop -fs ls /
Error: No command named `-fs' was found. Perhaps you meant `hadoop fs'
{code}

The only risk I can see is if someone is abusing the implementation of hadoop command doing something like so:

{code}
$ ./bin/hadoop -Xmx1g  org.apache.hadoop.util.RunJar
RunJar jarFile [mainClass] args...
{code}

The hadoop command does not appear to advertise allowing JVM options before the classname.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15327-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 03:10:36 2011
Return-Path: <common-issues-return-15327-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DDD336EC6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 03:10:35 +0000 (UTC)
Received: (qmail 60912 invoked by uid 500); 24 May 2011 03:10:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60704 invoked by uid 500); 24 May 2011 03:10:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60685 invoked by uid 99); 24 May 2011 03:10:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 03:10:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 03:10:30 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D4080DA8C2
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 03:09:49 +0000 (UTC)
Date: Tue, 24 May 2011 03:09:49 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1216005333.38264.1306206589865.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <889607684.27597.1305830207722.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7307) Remove Dr Who references
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038365#comment-13038365 ] 

Allen Wittenauer commented on HADOOP-7307:
------------------------------------------

Hmm. That's sort of surprising.  I would have expected the user name to get resolved to the user that is running the datanode when the built-in auth was being used.  (Doesn't it default to the username of the user running the Java process everywhere else?  Why would it be different for the HDFS web ui?)  But that's also bad for what are hopefully obvious reasons though.

FWIW, in Apache 0.20.2 and prior, we had dfs.web.ugi which looks like it defaulted to "webuser,webugi".  

I'm inclined to think that we likely need to bring this setting back and default it to either webuser or DrWho.  It sounds like the default changed in 0.21, so this is a tricky one.

> Remove Dr Who references
> ------------------------
>
>                 Key: HADOOP-7307
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7307
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>             Fix For: 0.22.0
>
>
> I've had feedback from a number of different operators that the "dr who" references that occasionally come out of Hadoop are annoying and confusing. We should replace them with more descriptive error messages, usernames, etc.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15328-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 08:16:35 2011
Return-Path: <common-issues-return-15328-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1B3B36087
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 08:16:35 +0000 (UTC)
Received: (qmail 80745 invoked by uid 500); 24 May 2011 08:16:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80207 invoked by uid 500); 24 May 2011 08:16:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79456 invoked by uid 99); 24 May 2011 08:16:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 08:16:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 08:16:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8693CDBD1C
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 08:15:48 +0000 (UTC)
Date: Tue, 24 May 2011 08:15:48 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <246913857.38570.1306224948547.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Moved] (HADOOP-7326) TestTrash infinite loops if run on a
 home directory with stuff in .Trash
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley moved HDFS-1988 to HADOOP-7326:
------------------------------------------

          Component/s:     (was: test)
                       test
        Fix Version/s:     (was: 0.22.0)
    Affects Version/s:     (was: 0.22.0)
                       0.23.0
                       0.22.0
                  Key: HADOOP-7326  (was: HDFS-1988)
              Project: Hadoop Common  (was: Hadoop HDFS)

> TestTrash infinite loops if run on a home directory with stuff in .Trash
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-7326
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7326
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Matt Foley
>
> Seems to have started failing recently in many commit builds as well as the last two nightly builds of 22:
> https://builds.apache.org/hudson/job/Hadoop-Hdfs-22-branch/51/testReport/org.apache.hadoop.hdfs/TestHDFSTrash/testTrashEmptier/

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15329-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 08:22:31 2011
Return-Path: <common-issues-return-15329-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 88C226184
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 08:22:31 +0000 (UTC)
Received: (qmail 91502 invoked by uid 500); 24 May 2011 08:22:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91458 invoked by uid 500); 24 May 2011 08:22:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91449 invoked by uid 99); 24 May 2011 08:22:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 08:22:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 08:22:30 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4660ADBF9C
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 08:21:50 +0000 (UTC)
Date: Tue, 24 May 2011 08:21:50 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2042948202.38587.1306225310284.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7326) TestTrash.testTrashEmptier()
 infinite loops if run on a home directory with stuff in .Trash
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated HADOOP-7326:
-------------------------------

    Description: This bug was discovered while investigating HDFS-1967, intermittent failure of TestHDFSTrash, which calls TestTrash.  If the user id running the test has a ~/.Trash directory that already has 4 or more files in it, a loop in TestTrash.testTrashEmptier() will never terminate, because it is waiting for the count of objects to increase from zero to exactly three (and it creates one object right away).  (was: Seems to have started failing recently in many commit builds as well as the last two nightly builds of 22:
https://builds.apache.org/hudson/job/Hadoop-Hdfs-22-branch/51/testReport/org.apache.hadoop.hdfs/TestHDFSTrash/testTrashEmptier/)
       Assignee: Matt Foley
        Summary: TestTrash.testTrashEmptier() infinite loops if run on a home directory with stuff in .Trash  (was: TestTrash infinite loops if run on a home directory with stuff in .Trash)

> TestTrash.testTrashEmptier() infinite loops if run on a home directory with stuff in .Trash
> -------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7326
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7326
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>
> This bug was discovered while investigating HDFS-1967, intermittent failure of TestHDFSTrash, which calls TestTrash.  If the user id running the test has a ~/.Trash directory that already has 4 or more files in it, a loop in TestTrash.testTrashEmptier() will never terminate, because it is waiting for the count of objects to increase from zero to exactly three (and it creates one object right away).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15330-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 08:32:29 2011
Return-Path: <common-issues-return-15330-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E0FA1616D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 08:32:29 +0000 (UTC)
Received: (qmail 6313 invoked by uid 500); 24 May 2011 08:32:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6280 invoked by uid 500); 24 May 2011 08:32:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6251 invoked by uid 99); 24 May 2011 08:32:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 08:32:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 08:32:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BD1ADDB45F
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 08:31:48 +0000 (UTC)
Date: Tue, 24 May 2011 08:31:48 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <164637914.38616.1306225908771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7327) FileSystem.listStatus() throws
 NullPointerException instead of IOException upon access permission failure
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

FileSystem.listStatus() throws NullPointerException instead of IOException upon access permission failure
---------------------------------------------------------------------------------------------------------

                 Key: HADOOP-7327
                 URL: https://issues.apache.org/jira/browse/HADOOP-7327
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
    Affects Versions: 0.22.0, 0.23.0
            Reporter: Matt Foley
            Assignee: Matt Foley


Many processes that call listStatus() expect to handle IOException, but instead are getting runtime error NullPointerException, if the directory being scanned is visible but no-access to the running user id.  For example, if directory foo is drwxr-xr-x, and subdirectory foo/bar is drwx------, then trying to do listStatus(Path(foo/bar)) will cause a NullPointerException.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15331-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 08:43:30 2011
Return-Path: <common-issues-return-15331-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B5B886EBE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 08:43:30 +0000 (UTC)
Received: (qmail 19408 invoked by uid 500); 24 May 2011 08:43:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19348 invoked by uid 500); 24 May 2011 08:43:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19340 invoked by uid 99); 24 May 2011 08:43:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 08:43:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 08:43:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7D43CDB86C
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 08:42:47 +0000 (UTC)
Date: Tue, 24 May 2011 08:42:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <101847024.38638.1306226567509.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <164637914.38616.1306225908771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7327) FileSystem.listStatus() throws
 NullPointerException instead of IOException upon access permission failure
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated HADOOP-7327:
-------------------------------

    Status: Patch Available  (was: Open)

> FileSystem.listStatus() throws NullPointerException instead of IOException upon access permission failure
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7327
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7327
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>         Attachments: hadoop-7327-1.patch
>
>
> Many processes that call listStatus() expect to handle IOException, but instead are getting runtime error NullPointerException, if the directory being scanned is visible but no-access to the running user id.  For example, if directory foo is drwxr-xr-x, and subdirectory foo/bar is drwx------, then trying to do listStatus(Path(foo/bar)) will cause a NullPointerException.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15332-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 08:43:31 2011
Return-Path: <common-issues-return-15332-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E51AF6EC9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 08:43:30 +0000 (UTC)
Received: (qmail 19608 invoked by uid 500); 24 May 2011 08:43:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19578 invoked by uid 500); 24 May 2011 08:43:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19444 invoked by uid 99); 24 May 2011 08:43:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 08:43:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 08:43:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 63FC2DB869
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 08:42:47 +0000 (UTC)
Date: Tue, 24 May 2011 08:42:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <504043202.38636.1306226567406.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <164637914.38616.1306225908771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7327) FileSystem.listStatus() throws
 NullPointerException instead of IOException upon access permission failure
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated HADOOP-7327:
-------------------------------

    Attachment: hadoop-7327-1.patch

The problem is that the underlying implementation relies on File.list(), which returns null upon access error, rather than throwing IOException.  (The underlying implementation can be hard to track down because of layering of FilterFileSystems, but for an example see RawLocalFileSystem.listStatus(Path).)

We can't change that behavior, because it is defined by the Java io library.  But at our FileSystem class level, it is appropriate to check for this condition at the point where FileSystem.listStatus() is about to generate a NullPointerException, and throw an IOException instead.

> FileSystem.listStatus() throws NullPointerException instead of IOException upon access permission failure
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7327
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7327
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>         Attachments: hadoop-7327-1.patch
>
>
> Many processes that call listStatus() expect to handle IOException, but instead are getting runtime error NullPointerException, if the directory being scanned is visible but no-access to the running user id.  For example, if directory foo is drwxr-xr-x, and subdirectory foo/bar is drwx------, then trying to do listStatus(Path(foo/bar)) will cause a NullPointerException.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15333-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 08:53:30 2011
Return-Path: <common-issues-return-15333-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9527C6655
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 08:53:30 +0000 (UTC)
Received: (qmail 34965 invoked by uid 500); 24 May 2011 08:53:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34936 invoked by uid 500); 24 May 2011 08:53:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34928 invoked by uid 99); 24 May 2011 08:53:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 08:53:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 08:53:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 630B3DBCE5
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 08:52:47 +0000 (UTC)
Date: Tue, 24 May 2011 08:52:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <706384435.38654.1306227167402.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7326) TestTrash.testTrashEmptier()
 infinite loops if run on a home directory with stuff in .Trash
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated HADOOP-7326:
-------------------------------

    Attachment: hadoop-7326-TestTrash-1.patch

The problem is that testTrashEmptier() naively counts all the objects in .Trash, but it needs to filter for only Trash checkpoint subdirectory objects, the same way Trash.expunge() does.

This patch fixes the problem, and also cleans up some logging in the Trash class, to make it more useful in interpreting such situations.

> TestTrash.testTrashEmptier() infinite loops if run on a home directory with stuff in .Trash
> -------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7326
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7326
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>         Attachments: hadoop-7326-TestTrash-1.patch
>
>
> This bug was discovered while investigating HDFS-1967, intermittent failure of TestHDFSTrash, which calls TestTrash.  If the user id running the test has a ~/.Trash directory that already has 4 or more files in it, a loop in TestTrash.testTrashEmptier() will never terminate, because it is waiting for the count of objects to increase from zero to exactly three (and it creates one object right away).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15334-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 09:05:29 2011
Return-Path: <common-issues-return-15334-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 422EC6D5D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 09:05:29 +0000 (UTC)
Received: (qmail 43965 invoked by uid 500); 24 May 2011 09:05:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43903 invoked by uid 500); 24 May 2011 09:05:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43636 invoked by uid 99); 24 May 2011 09:05:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 09:05:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 09:05:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7E45DDB272
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 09:04:47 +0000 (UTC)
Date: Tue, 24 May 2011 09:04:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1493528378.38689.1306227887514.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <164637914.38616.1306225908771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7327) FileSystem.listStatus() throws
 NullPointerException instead of IOException upon access permission failure
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038462#comment-13038462 ] 

Hadoop QA commented on HADOOP-7327:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480218/hadoop-7327-1.patch
  against trunk revision 1126719.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:


    -1 system test framework.  The patch failed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/512//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/512//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/512//console

This message is automatically generated.

> FileSystem.listStatus() throws NullPointerException instead of IOException upon access permission failure
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7327
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7327
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>         Attachments: hadoop-7327-1.patch
>
>
> Many processes that call listStatus() expect to handle IOException, but instead are getting runtime error NullPointerException, if the directory being scanned is visible but no-access to the running user id.  For example, if directory foo is drwxr-xr-x, and subdirectory foo/bar is drwx------, then trying to do listStatus(Path(foo/bar)) will cause a NullPointerException.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15335-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 10:15:32 2011
Return-Path: <common-issues-return-15335-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7E9ED6BE8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 10:15:32 +0000 (UTC)
Received: (qmail 43221 invoked by uid 500); 24 May 2011 10:15:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43178 invoked by uid 500); 24 May 2011 10:15:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43157 invoked by uid 99); 24 May 2011 10:15:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 10:15:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 10:15:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 71E3ADBF1D
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 10:14:47 +0000 (UTC)
Date: Tue, 24 May 2011 10:14:47 +0000 (UTC)
From: "Harsh J Chouraria (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Give more information about a missing Serializer class
------------------------------------------------------

                 Key: HADOOP-7328
                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
             Project: Hadoop Common
          Issue Type: Improvement
          Components: io
    Affects Versions: 0.20.2
            Reporter: Harsh J Chouraria
            Assignee: Harsh J Chouraria
             Fix For: 0.23.0


When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15336-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 11:01:33 2011
Return-Path: <common-issues-return-15336-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 860DD6C10
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 11:01:33 +0000 (UTC)
Received: (qmail 9086 invoked by uid 500); 24 May 2011 11:01:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8973 invoked by uid 500); 24 May 2011 11:01:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8814 invoked by uid 99); 24 May 2011 11:01:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 11:01:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 11:01:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 68221DB00D
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 11:00:48 +0000 (UTC)
Date: Tue, 24 May 2011 11:00:48 +0000 (UTC)
From: "Harsh J Chouraria (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <770306368.38868.1306234848423.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J Chouraria updated HADOOP-7328:
--------------------------------------

    Attachment: HADOOP-7328.r1.diff

Patch that adds a FATAL log when a null is about to be received for a requested serializer.

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J Chouraria
>            Assignee: Harsh J Chouraria
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15337-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 11:01:36 2011
Return-Path: <common-issues-return-15337-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0697B6C80
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 11:01:36 +0000 (UTC)
Received: (qmail 10475 invoked by uid 500); 24 May 2011 11:01:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10436 invoked by uid 500); 24 May 2011 11:01:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10334 invoked by uid 99); 24 May 2011 11:01:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 11:01:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 11:01:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8B461DB010
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 11:00:48 +0000 (UTC)
Date: Tue, 24 May 2011 11:00:48 +0000 (UTC)
From: "Harsh J Chouraria (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <819320487.38871.1306234848567.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J Chouraria updated HADOOP-7328:
--------------------------------------

    Status: Patch Available  (was: Open)

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J Chouraria
>            Assignee: Harsh J Chouraria
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15338-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 11:15:28 2011
Return-Path: <common-issues-return-15338-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9592B6C74
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 11:15:28 +0000 (UTC)
Received: (qmail 33735 invoked by uid 500); 24 May 2011 11:15:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33701 invoked by uid 500); 24 May 2011 11:15:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33692 invoked by uid 99); 24 May 2011 11:15:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 11:15:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 11:15:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 748BDDB46E
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 11:14:47 +0000 (UTC)
Date: Tue, 24 May 2011 11:14:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <966968865.38887.1306235687474.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <187742091.8722.1305246647362.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7287) Configuration deprecation
 mechanism doesn't work properly for GenericOptionsParser/Tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038505#comment-13038505 ] 

Hudson commented on HADOOP-7287:
--------------------------------

Integrated in Hadoop-Common-trunk #698 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/698/])
    HADOOP-7287. Configuration deprecation mechanism doesn't work properly for GenericOptionsParser and Tools. Contributed by Aaron T. Myers.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1126719
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/conf/Configuration.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/conf/TestConfigurationDeprecation.java
* /hadoop/common/trunk/src/test/test-fake-default.xml


> Configuration deprecation mechanism doesn't work properly for GenericOptionsParser/Tools
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7287
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7287
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Aaron T. Myers
>            Priority: Blocker
>             Fix For: 0.22.0, 0.23.0
>
>         Attachments: hadoop-7287-still-broken.txt, hadoop-7287-testcase.txt, hadoop-7287-trunk.0.patch, hadoop-7287-trunk.1.patch, hadoop-7287-trunk.2.patch
>
>
> For example, you can't use -D options on the "hadoop fs" command line in order to specify the deprecated names of configuration options. The issue is that the ordering is:
> - JVM starts
> - GenericOptionsParser creates a Configuration object and calls set() for each of the options specified on command line
> - DistributedFileSystem or other class eventually instantiates HdfsConfiguration which adds the deprecations
> - Some class calls conf.get("new key") and sees the default instead of the version set on the command line

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15339-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 11:15:28 2011
Return-Path: <common-issues-return-15339-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C55E96C81
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 11:15:28 +0000 (UTC)
Received: (qmail 34014 invoked by uid 500); 24 May 2011 11:15:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33931 invoked by uid 500); 24 May 2011 11:15:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33791 invoked by uid 99); 24 May 2011 11:15:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 11:15:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 11:15:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 860C3DB471
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 11:14:47 +0000 (UTC)
Date: Tue, 24 May 2011 11:14:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1527859869.38889.1306235687545.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038506#comment-13038506 ] 

Hadoop QA commented on HADOOP-7328:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480231/HADOOP-7328.r1.diff
  against trunk revision 1126719.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:


    -1 system test framework.  The patch failed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/513//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/513//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/513//console

This message is automatically generated.

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J Chouraria
>            Assignee: Harsh J Chouraria
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15340-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 11:25:28 2011
Return-Path: <common-issues-return-15340-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 82811620D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 11:25:28 +0000 (UTC)
Received: (qmail 48405 invoked by uid 500); 24 May 2011 11:25:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48376 invoked by uid 500); 24 May 2011 11:25:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48367 invoked by uid 99); 24 May 2011 11:25:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 11:25:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 11:25:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 74A43DB778
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 11:24:47 +0000 (UTC)
Date: Tue, 24 May 2011 11:24:47 +0000 (UTC)
From: "Harsh J Chouraria (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1940533552.38901.1306236287474.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038509#comment-13038509 ] 

Harsh J Chouraria commented on HADOOP-7328:
-------------------------------------------

Justifications: A LOG addition, does not require a test case addition/change.

The other seemingly irrelevant change is a checkstyle fix, the line was using a tab char.

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J Chouraria
>            Assignee: Harsh J Chouraria
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15341-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 12:38:31 2011
Return-Path: <common-issues-return-15341-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 13AE56716
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 12:38:31 +0000 (UTC)
Received: (qmail 69667 invoked by uid 500); 24 May 2011 12:38:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69643 invoked by uid 500); 24 May 2011 12:38:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69635 invoked by uid 99); 24 May 2011 12:38:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 12:38:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 12:38:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B131BDB23C
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 12:37:47 +0000 (UTC)
Date: Tue, 24 May 2011 12:37:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1665568468.38996.1306240667722.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <360710663.20918.1304489523226.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7259) contrib modules should include
 build.properties from parent.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038527#comment-13038527 ] 

Hudson commented on HADOOP-7259:
--------------------------------

Integrated in Hadoop-Hdfs-trunk #676 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/676/])
    HADOOP-7259. Contrib modules should include the build.properties from
the enclosing hadoop directory. (omalley)

omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1126795
Files : 
* /hadoop/hdfs/trunk/src/contrib/build-contrib.xml
* /hadoop/hdfs/trunk/CHANGES.txt


> contrib modules should include build.properties from parent.
> ------------------------------------------------------------
>
>                 Key: HADOOP-7259
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7259
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: contrib.patch
>
>
> Current build.properties in the hadoop root directory is not included by the contrib modules.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15342-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 14:06:28 2011
Return-Path: <common-issues-return-15342-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9302A68B9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 14:06:28 +0000 (UTC)
Received: (qmail 25769 invoked by uid 500); 24 May 2011 14:06:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25743 invoked by uid 500); 24 May 2011 14:06:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25735 invoked by uid 99); 24 May 2011 14:06:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 14:06:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 14:06:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 820BADB679
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 14:05:47 +0000 (UTC)
Date: Tue, 24 May 2011 14:05:47 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <865079863.39181.1306245947529.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1195662636.6130.1300891805766.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7208) equals() and hashCode()
 implementation need to change in StandardSocketFactory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uma Maheswara Rao G updated HADOOP-7208:
----------------------------------------

    Attachment: HADOOP-7208-2.patch

> equals() and hashCode() implementation need to change in StandardSocketFactory
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7208
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7208
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7208-2.patch, HADOOP-7208.patch
>
>
>   In Hadoop IPC Client, we are using ClientCache which will maintain the HashMap to keep the Client references.
> private Map<SocketFactory, Client> clients =
>       new HashMap<SocketFactory, Client>();
>  Now let us say, we want use two standard factories with Hadoop. MyStandardSocketFactory (which extends StandardSocketFactory), and StandardSocketFactory. In this case, because of equals and hashcode implementation, MyStandardSocketFactory client can be overridden by StandardSocketFactoryClient

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15343-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 14:08:28 2011
Return-Path: <common-issues-return-15343-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BFD6D692F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 14:08:28 +0000 (UTC)
Received: (qmail 27738 invoked by uid 500); 24 May 2011 14:08:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27710 invoked by uid 500); 24 May 2011 14:08:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27698 invoked by uid 99); 24 May 2011 14:08:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 14:08:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 14:08:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9EAF6DB780
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 14:07:47 +0000 (UTC)
Date: Tue, 24 May 2011 14:07:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1472928761.39188.1306246067646.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1828723839.36128.1306162247409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7320) Refactor FsShell's copy & move
 commands
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038569#comment-13038569 ] 

Daryn Sharp commented on HADOOP-7320:
-------------------------------------

The patch problem is due to renaming Copy to CopyCommands per review comments.  Subversion is generating a patch that it can't reapply.  Namely that the patch deletes Copy, but the hunks for CopyCommands are relative to the Copy command.

Should I regenerate the patch by svn removing and adding the renamed file?  Since that will lose history (it's a very new file so maybe it doesn't matter), is there another way to generate the patch?

> Refactor FsShell's copy & move commands
> ---------------------------------------
>
>                 Key: HADOOP-7320
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7320
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7320-2.patch, HADOOP-7320-3.patch, HADOOP-7320.patch
>
>
> Need to refactor the move and copy commands to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15344-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 14:18:28 2011
Return-Path: <common-issues-return-15344-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 868946ABE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 14:18:28 +0000 (UTC)
Received: (qmail 49411 invoked by uid 500); 24 May 2011 14:18:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49292 invoked by uid 500); 24 May 2011 14:18:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49284 invoked by uid 99); 24 May 2011 14:18:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 14:18:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 14:18:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6EF65DBE3A
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 14:17:47 +0000 (UTC)
Date: Tue, 24 May 2011 14:17:47 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1856569606.39247.1306246667451.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1195662636.6130.1300891805766.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7208) equals() and hashCode()
 implementation need to change in StandardSocketFactory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uma Maheswara Rao G updated HADOOP-7208:
----------------------------------------

    Status: Open  (was: Patch Available)

> equals() and hashCode() implementation need to change in StandardSocketFactory
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7208
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7208
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7208-2.patch, HADOOP-7208.patch
>
>
>   In Hadoop IPC Client, we are using ClientCache which will maintain the HashMap to keep the Client references.
> private Map<SocketFactory, Client> clients =
>       new HashMap<SocketFactory, Client>();
>  Now let us say, we want use two standard factories with Hadoop. MyStandardSocketFactory (which extends StandardSocketFactory), and StandardSocketFactory. In this case, because of equals and hashcode implementation, MyStandardSocketFactory client can be overridden by StandardSocketFactoryClient

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15345-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 14:18:28 2011
Return-Path: <common-issues-return-15345-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B5CFC6ACC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 14:18:28 +0000 (UTC)
Received: (qmail 49552 invoked by uid 500); 24 May 2011 14:18:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49522 invoked by uid 500); 24 May 2011 14:18:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49364 invoked by uid 99); 24 May 2011 14:18:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 14:18:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 14:18:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8132DDBE3C
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 14:17:47 +0000 (UTC)
Date: Tue, 24 May 2011 14:17:47 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1088923721.39249.1306246667526.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1195662636.6130.1300891805766.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7208) equals() and hashCode()
 implementation need to change in StandardSocketFactory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uma Maheswara Rao G updated HADOOP-7208:
----------------------------------------

    Status: Patch Available  (was: Open)

> equals() and hashCode() implementation need to change in StandardSocketFactory
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7208
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7208
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7208-2.patch, HADOOP-7208.patch
>
>
>   In Hadoop IPC Client, we are using ClientCache which will maintain the HashMap to keep the Client references.
> private Map<SocketFactory, Client> clients =
>       new HashMap<SocketFactory, Client>();
>  Now let us say, we want use two standard factories with Hadoop. MyStandardSocketFactory (which extends StandardSocketFactory), and StandardSocketFactory. In this case, because of equals and hashcode implementation, MyStandardSocketFactory client can be overridden by StandardSocketFactoryClient

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15346-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 14:54:30 2011
Return-Path: <common-issues-return-15346-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 201256B46
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 14:54:30 +0000 (UTC)
Received: (qmail 28769 invoked by uid 500); 24 May 2011 14:54:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28720 invoked by uid 500); 24 May 2011 14:54:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28658 invoked by uid 99); 24 May 2011 14:54:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 14:54:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 14:54:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 94F56DB269
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 14:53:48 +0000 (UTC)
Date: Tue, 24 May 2011 14:53:48 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1330323586.39358.1306248828606.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1195662636.6130.1300891805766.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7208) equals() and hashCode()
 implementation need to change in StandardSocketFactory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038599#comment-13038599 ] 

Hadoop QA commented on HADOOP-7208:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480249/HADOOP-7208-2.patch
  against trunk revision 1126719.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/514//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/514//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/514//console

This message is automatically generated.

> equals() and hashCode() implementation need to change in StandardSocketFactory
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7208
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7208
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7208-2.patch, HADOOP-7208.patch
>
>
>   In Hadoop IPC Client, we are using ClientCache which will maintain the HashMap to keep the Client references.
> private Map<SocketFactory, Client> clients =
>       new HashMap<SocketFactory, Client>();
>  Now let us say, we want use two standard factories with Hadoop. MyStandardSocketFactory (which extends StandardSocketFactory), and StandardSocketFactory. In this case, because of equals and hashcode implementation, MyStandardSocketFactory client can be overridden by StandardSocketFactoryClient

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15347-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 15:33:31 2011
Return-Path: <common-issues-return-15347-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7128F65D9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 15:33:31 +0000 (UTC)
Received: (qmail 30094 invoked by uid 500); 24 May 2011 15:33:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30040 invoked by uid 500); 24 May 2011 15:33:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30029 invoked by uid 99); 24 May 2011 15:33:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 15:33:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 15:33:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0418CDC6DB
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 15:32:48 +0000 (UTC)
Date: Tue, 24 May 2011 15:32:48 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <684141003.39475.1306251168013.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6541) An Interactive Hadoop FS shell
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6541:
--------------------------------

    Comment: was deleted

(was: I will be out of office Apr 18-22, 2011.  If you have anything urgent,
please contact my manager Omer Trajman omer@cloudera.com.

-- 
--
Alex Kozlov
Solutions Architect
Cloudera, Inc
<http://www.cloudera.com/company/press-center/hadoop-world-nyc/>
)

> An Interactive Hadoop FS shell
> ------------------------------
>
>                 Key: HADOOP-6541
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6541
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>         Attachments: HADOOP-6541.2.patch, HADOOP-6541.3.patch, HADOOP-6541.4.patch, HADOOP-6541.5.patch, HADOOP-6541.6.patch, HADOOP-6541.patch
>
>
> A shell that allows the user to execute multiple filesystem operations in a single JVM instance at a prompt.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15348-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 15:45:31 2011
Return-Path: <common-issues-return-15348-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B14DB6E97
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 15:45:31 +0000 (UTC)
Received: (qmail 56479 invoked by uid 500); 24 May 2011 15:45:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56438 invoked by uid 500); 24 May 2011 15:45:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56430 invoked by uid 99); 24 May 2011 15:45:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 15:45:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 15:45:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4A68FDCC93
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 15:44:48 +0000 (UTC)
Date: Tue, 24 May 2011 15:44:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <420270233.39525.1306251888301.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <360710663.20918.1304489523226.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7259) contrib modules should include
 build.properties from parent.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038636#comment-13038636 ] 

Hudson commented on HADOOP-7259:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk #689 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/689/])
    HADOOP-7259. Contrib modules should include the build.properties from
the enclosing hadoop directory. (omalley)

omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1126801
Files : 
* /hadoop/mapreduce/trunk/CHANGES.txt
* /hadoop/mapreduce/trunk/src/contrib/build-contrib.xml


> contrib modules should include build.properties from parent.
> ------------------------------------------------------------
>
>                 Key: HADOOP-7259
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7259
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: contrib.patch
>
>
> Current build.properties in the hadoop root directory is not included by the contrib modules.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15349-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 17:34:30 2011
Return-Path: <common-issues-return-15349-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 52D1E60CB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 17:34:30 +0000 (UTC)
Received: (qmail 75795 invoked by uid 500); 24 May 2011 17:34:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75757 invoked by uid 500); 24 May 2011 17:34:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75748 invoked by uid 99); 24 May 2011 17:34:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 17:34:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 17:34:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2BBBEDC5B3
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 17:33:49 +0000 (UTC)
Date: Tue, 24 May 2011 17:33:49 +0000 (UTC)
From: "Ismael Juma (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1736248395.39771.1306258429176.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6349) Implement FastLZCodec for
 fastlz/lzo algorithm
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038681#comment-13038681 ] 

Ismael Juma commented on HADOOP-6349:
-------------------------------------

What is the status of this? Is anyone still working on it?

> Implement FastLZCodec for fastlz/lzo algorithm
> ----------------------------------------------
>
>                 Key: HADOOP-6349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6349
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: William Kinney
>         Attachments: HADOOP-6349-TestFastLZCodec.patch, HADOOP-6349.patch, TestCodecPerformance.java, TestCodecPerformance.java, hadoop-6349-1.patch, hadoop-6349-2.patch, hadoop-6349-3.patch, hadoop-6349-4.patch, testCodecPerfResults.tsv
>
>
> Per  [HADOOP-4874|http://issues.apache.org/jira/browse/HADOOP-4874], FastLZ is a good (speed, license) alternative to LZO. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15350-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 17:56:29 2011
Return-Path: <common-issues-return-15350-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C892C6BCC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 17:56:29 +0000 (UTC)
Received: (qmail 17310 invoked by uid 500); 24 May 2011 17:56:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17261 invoked by uid 500); 24 May 2011 17:56:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17253 invoked by uid 99); 24 May 2011 17:56:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 17:56:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 17:56:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9A934DCCD5
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 17:55:48 +0000 (UTC)
Date: Tue, 24 May 2011 17:55:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <4374891.39800.1306259748630.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6349) Implement FastLZCodec for
 fastlz/lzo algorithm
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038687#comment-13038687 ] 

Todd Lipcon commented on HADOOP-6349:
-------------------------------------

I think this has largely been superceded by the recent work on Snappy

> Implement FastLZCodec for fastlz/lzo algorithm
> ----------------------------------------------
>
>                 Key: HADOOP-6349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6349
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: William Kinney
>         Attachments: HADOOP-6349-TestFastLZCodec.patch, HADOOP-6349.patch, TestCodecPerformance.java, TestCodecPerformance.java, hadoop-6349-1.patch, hadoop-6349-2.patch, hadoop-6349-3.patch, hadoop-6349-4.patch, testCodecPerfResults.tsv
>
>
> Per  [HADOOP-4874|http://issues.apache.org/jira/browse/HADOOP-4874], FastLZ is a good (speed, license) alternative to LZO. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15351-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 18:00:29 2011
Return-Path: <common-issues-return-15351-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A62B96FAF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 18:00:29 +0000 (UTC)
Received: (qmail 23538 invoked by uid 500); 24 May 2011 18:00:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23507 invoked by uid 500); 24 May 2011 18:00:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23367 invoked by uid 99); 24 May 2011 18:00:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:00:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:00:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 47F9ADCE96
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 17:59:48 +0000 (UTC)
Date: Tue, 24 May 2011 17:59:48 +0000 (UTC)
From: "Ismael Juma (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1718609519.39852.1306259988291.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6349) Implement FastLZCodec for
 fastlz/lzo algorithm
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038691#comment-13038691 ] 

Ismael Juma commented on HADOOP-6349:
-------------------------------------

Thanks, that's good to know.

> Implement FastLZCodec for fastlz/lzo algorithm
> ----------------------------------------------
>
>                 Key: HADOOP-6349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6349
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: William Kinney
>         Attachments: HADOOP-6349-TestFastLZCodec.patch, HADOOP-6349.patch, TestCodecPerformance.java, TestCodecPerformance.java, hadoop-6349-1.patch, hadoop-6349-2.patch, hadoop-6349-3.patch, hadoop-6349-4.patch, testCodecPerfResults.tsv
>
>
> Per  [HADOOP-4874|http://issues.apache.org/jira/browse/HADOOP-4874], FastLZ is a good (speed, license) alternative to LZO. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15352-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 18:13:28 2011
Return-Path: <common-issues-return-15352-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BD445641D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 18:13:28 +0000 (UTC)
Received: (qmail 43969 invoked by uid 500); 24 May 2011 18:13:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43941 invoked by uid 500); 24 May 2011 18:13:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43933 invoked by uid 99); 24 May 2011 18:13:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:13:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:13:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A477ADC2F4
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 18:12:47 +0000 (UTC)
Date: Tue, 24 May 2011 18:12:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <902068360.39911.1306260767670.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1828723839.36128.1306162247409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7320) Refactor FsShell's copy & move
 commands
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7320:
--------------------------------

    Attachment: HADOOP-7320-rename.patch

This patch must be applied first to allow the real patch to rename Copy.java to CopyCommands.java.

> Refactor FsShell's copy & move commands
> ---------------------------------------
>
>                 Key: HADOOP-7320
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7320
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7320-2.patch, HADOOP-7320-3.patch, HADOOP-7320-rename.patch, HADOOP-7320.patch
>
>
> Need to refactor the move and copy commands to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15353-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 18:25:31 2011
Return-Path: <common-issues-return-15353-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1B1146BA8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 18:25:31 +0000 (UTC)
Received: (qmail 66425 invoked by uid 500); 24 May 2011 18:25:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66386 invoked by uid 500); 24 May 2011 18:25:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66378 invoked by uid 99); 24 May 2011 18:25:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:25:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:25:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 79EC4DC7CD
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 18:24:47 +0000 (UTC)
Date: Tue, 24 May 2011 18:24:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1537828972.39944.1306261487496.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1828723839.36128.1306162247409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7320) Refactor FsShell's copy & move
 commands
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038701#comment-13038701 ] 

Hadoop QA commented on HADOOP-7320:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480293/HADOOP-7320-rename.patch
  against trunk revision 1126719.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The patch appears to cause tar ant target to fail.

    -1 findbugs.  The patch appears to cause Findbugs (version 1.3.9) to fail.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:


    -1 system test framework.  The patch failed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/515//testReport/
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/515//console

This message is automatically generated.

> Refactor FsShell's copy & move commands
> ---------------------------------------
>
>                 Key: HADOOP-7320
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7320
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7320-2.patch, HADOOP-7320-3.patch, HADOOP-7320-rename.patch, HADOOP-7320.patch
>
>
> Need to refactor the move and copy commands to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15354-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 18:31:32 2011
Return-Path: <common-issues-return-15354-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7CABD64D5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 18:31:32 +0000 (UTC)
Received: (qmail 74346 invoked by uid 500); 24 May 2011 18:31:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74314 invoked by uid 500); 24 May 2011 18:31:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74306 invoked by uid 99); 24 May 2011 18:31:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:31:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:31:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 617B5DC9BA
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 18:30:48 +0000 (UTC)
Date: Tue, 24 May 2011 18:30:48 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <399132680.39974.1306261848394.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-6671:
---------------------------------------

    Attachment: mvn-layout2.sh
                HADOOP-6671d.patch

Integrated checkstyle and findbugs, added option to skipDocs.

Only thing missing is JDiff integration (which it seems a dead project and the Maven plugin does not seem to work properly).

Is JDIFF required? Any other alternative?

Maven goals:

* clean
* compile (-DcompileNative)
* test (-DcompileNative, -Dtest=<TESTCLASS>, -DskipTests)
* package (-DcompileNative, -DskipDocs, -DskipTests | -Dtest=<TESTCLASS>)
* checkstyle:checkstyle
* findbugs:findbugs



> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>         Attachments: HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15355-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 18:33:31 2011
Return-Path: <common-issues-return-15355-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5BCAF6F18
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 18:33:31 +0000 (UTC)
Received: (qmail 77606 invoked by uid 500); 24 May 2011 18:33:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77469 invoked by uid 500); 24 May 2011 18:33:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77457 invoked by uid 99); 24 May 2011 18:33:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:33:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:33:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 65B11DCA5F
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 18:32:47 +0000 (UTC)
Date: Tue, 24 May 2011 18:32:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1944447565.39977.1306261967413.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038705#comment-13038705 ] 

Hadoop QA commented on HADOOP-6671:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480296/mvn-layout2.sh
  against trunk revision 1126719.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 11 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/516//console

This message is automatically generated.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>         Attachments: HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15356-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 18:35:29 2011
Return-Path: <common-issues-return-15356-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 132B16310
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 18:35:29 +0000 (UTC)
Received: (qmail 79691 invoked by uid 500); 24 May 2011 18:35:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79532 invoked by uid 500); 24 May 2011 18:35:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79524 invoked by uid 99); 24 May 2011 18:35:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:35:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:35:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 15EBCDCB5B
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 18:34:48 +0000 (UTC)
Date: Tue, 24 May 2011 18:34:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <211255432.40024.1306262088083.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038706#comment-13038706 ] 

Todd Lipcon commented on HADOOP-6671:
-------------------------------------

Hey Alejandro. Yep, we do use jdiff in order to detect incompatible changes between releases. Tom is one of the experts there.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>         Attachments: HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15357-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 18:37:28 2011
Return-Path: <common-issues-return-15357-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 652E46369
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 18:37:28 +0000 (UTC)
Received: (qmail 80948 invoked by uid 500); 24 May 2011 18:37:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80903 invoked by uid 500); 24 May 2011 18:37:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80895 invoked by uid 99); 24 May 2011 18:37:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:37:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:37:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5B446DCBFF
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 18:36:47 +0000 (UTC)
Date: Tue, 24 May 2011 18:36:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1167897674.40028.1306262207370.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038708#comment-13038708 ] 

Todd Lipcon commented on HADOOP-7328:
-------------------------------------

Maybe instead of a new log, we could have the tasks throw a more useful exception?

Also, would it be possible to detect this issue at submission time? That's much more user friendly than making the user look through failed task logs.

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J Chouraria
>            Assignee: Harsh J Chouraria
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15358-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 18:39:30 2011
Return-Path: <common-issues-return-15358-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D4E5B63DB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 18:39:30 +0000 (UTC)
Received: (qmail 83387 invoked by uid 500); 24 May 2011 18:39:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83200 invoked by uid 500); 24 May 2011 18:39:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83192 invoked by uid 99); 24 May 2011 18:39:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:39:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:39:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 921C4DCC7D
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 18:38:47 +0000 (UTC)
Date: Tue, 24 May 2011 18:38:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1386606355.40032.1306262327595.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <164637914.38616.1306225908771.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7327) FileSystem.listStatus() throws
 NullPointerException instead of IOException upon access permission failure
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038709#comment-13038709 ] 

Todd Lipcon commented on HADOOP-7327:
-------------------------------------

This looks good, but unfortunately the Hudson build slave seems to be in bad shape (disk full). I'm working on getting that fixed. Would it be possible to run the tests locally in the meantime?

Also, can you add a unit test? Should be easy to mkdir/chown a directory to reproduce the issue.

> FileSystem.listStatus() throws NullPointerException instead of IOException upon access permission failure
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7327
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7327
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>         Attachments: hadoop-7327-1.patch
>
>
> Many processes that call listStatus() expect to handle IOException, but instead are getting runtime error NullPointerException, if the directory being scanned is visible but no-access to the running user id.  For example, if directory foo is drwxr-xr-x, and subdirectory foo/bar is drwx------, then trying to do listStatus(Path(foo/bar)) will cause a NullPointerException.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15359-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 18:41:28 2011
Return-Path: <common-issues-return-15359-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A529E6418
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 18:41:28 +0000 (UTC)
Received: (qmail 85760 invoked by uid 500); 24 May 2011 18:41:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85736 invoked by uid 500); 24 May 2011 18:41:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85728 invoked by uid 99); 24 May 2011 18:41:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:41:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:41:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 888F4DCD28
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 18:40:47 +0000 (UTC)
Date: Tue, 24 May 2011 18:40:47 +0000 (UTC)
From: "Harsh J Chouraria (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1772490679.40041.1306262447556.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Harsh J Chouraria updated HADOOP-7328:
--------------------------------------

    Status: Open  (was: Patch Available)

I thought of the throws XYZException way, but that'd require an incompatible change wouldn't it?

I think it should be possible to detect the issue at sub-time, but I'll have to try that with some tests. Let me get back to you with something in that direction.

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J Chouraria
>            Assignee: Harsh J Chouraria
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15360-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 18:45:28 2011
Return-Path: <common-issues-return-15360-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9A408687A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 18:45:28 +0000 (UTC)
Received: (qmail 97516 invoked by uid 500); 24 May 2011 18:45:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97436 invoked by uid 500); 24 May 2011 18:45:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97428 invoked by uid 99); 24 May 2011 18:45:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:45:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:45:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 768A3DCF27
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 18:44:47 +0000 (UTC)
Date: Tue, 24 May 2011 18:44:47 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1385664767.40049.1306262687482.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <654871262.37888.1306191347393.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7323) Add capability to resolve
 compression codec based on codec name
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-7323:
---------------------------------------

    Attachment: HADOOP-7323b.patch

Thanks Tom.

I've updated the Javadocs.

I've also improved the alias resolution to resolve to the short class name, both with 'codec' ending and without 'codec' ending.

This means that the aliases for GzipCodec are 'gzip' and 'gzipcodec' (both case insensitive)

> Add capability to resolve compression codec based on codec name
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7323
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7323
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7323.patch, HADOOP-7323b.patch
>
>
> When setting up a compression codec in an MR job the full class name of the codec must be used.
> To ease usability, compression codecs should be resolved by their codec name (ie 'gzip', 'deflate', 'zlib', 'bzip2') instead their full codec class name.
> Besides easy of use for Hadoop users who would use the codec alias instead the full codec class name, it could simplify how HBase resolves loads the codecs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15361-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 18:45:29 2011
Return-Path: <common-issues-return-15361-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E3DE96893
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 18:45:28 +0000 (UTC)
Received: (qmail 97726 invoked by uid 500); 24 May 2011 18:45:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97667 invoked by uid 500); 24 May 2011 18:45:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97659 invoked by uid 99); 24 May 2011 18:45:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:45:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:45:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BD15CDCF29
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 18:44:47 +0000 (UTC)
Date: Tue, 24 May 2011 18:44:47 +0000 (UTC)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1200649117.40051.1306262687548.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <654871262.37888.1306191347393.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7323) Add capability to resolve
 compression codec based on codec name
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-7323:
---------------------------------------

    Status: Patch Available  (was: Open)

> Add capability to resolve compression codec based on codec name
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7323
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7323
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7323.patch, HADOOP-7323b.patch
>
>
> When setting up a compression codec in an MR job the full class name of the codec must be used.
> To ease usability, compression codecs should be resolved by their codec name (ie 'gzip', 'deflate', 'zlib', 'bzip2') instead their full codec class name.
> Besides easy of use for Hadoop users who would use the codec alias instead the full codec class name, it could simplify how HBase resolves loads the codecs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15362-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 18:51:29 2011
Return-Path: <common-issues-return-15362-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DD86E6B8C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 18:51:28 +0000 (UTC)
Received: (qmail 5679 invoked by uid 500); 24 May 2011 18:51:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5652 invoked by uid 500); 24 May 2011 18:51:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5643 invoked by uid 99); 24 May 2011 18:51:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:51:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:51:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B8496DC157
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 18:50:47 +0000 (UTC)
Date: Tue, 24 May 2011 18:50:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <670535761.40066.1306263047751.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <860231771.38800.1306232087463.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7328) Give more information about a
 missing Serializer class
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038714#comment-13038714 ] 

Todd Lipcon commented on HADOOP-7328:
-------------------------------------

I don't think it's an incompatible change to replace what's currently a NullPointerException with something more descriptive. If the method signature can't throw an IOException, at least a RuntimeException with a nice error message would be an improvement. (To be clear, I'm not talking about changing the serialization factory, but rather the point in MR where the NPE gets thrown because of the null serializer)

> Give more information about a missing Serializer class
> ------------------------------------------------------
>
>                 Key: HADOOP-7328
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7328
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Harsh J Chouraria
>            Assignee: Harsh J Chouraria
>              Labels: io, serialization
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7328.r1.diff
>
>
> When you have a key/value class that's non Writable and you forget to attach io.serializers for the same, an NPE is thrown by the tasks with no information on why or what's missing and what led to it. I think a better exception can be thrown by SerializationFactory instead of an NPE when a class is not found accepted by any of the loaded ones.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15363-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 18:51:31 2011
Return-Path: <common-issues-return-15363-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 257F06BF2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 18:51:31 +0000 (UTC)
Received: (qmail 7056 invoked by uid 500); 24 May 2011 18:51:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7002 invoked by uid 500); 24 May 2011 18:51:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6970 invoked by uid 99); 24 May 2011 18:51:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:51:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:51:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8D165DC153
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 18:50:47 +0000 (UTC)
Date: Tue, 24 May 2011 18:50:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1423962105.40062.1306263047574.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <64692328.19459.1304449203116.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7258) Gzip codec should not return null
 decompressors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley resolved HADOOP-7258.
-----------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]

I just committed this.

> Gzip codec should not return null decompressors
> -----------------------------------------------
>
>                 Key: HADOOP-7258
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7258
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: fix.patch, gzip-decomp.patch
>
>
> In HADOOP-6315, the gzip codec was changed to return a null codec with the intent to disallow pooling of the decompressors. Rather than break the interface, we can use an annotation to achieve the goal.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15364-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 18:53:30 2011
Return-Path: <common-issues-return-15364-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C6FFE6D31
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 18:53:30 +0000 (UTC)
Received: (qmail 12581 invoked by uid 500); 24 May 2011 18:53:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12550 invoked by uid 500); 24 May 2011 18:53:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12542 invoked by uid 99); 24 May 2011 18:53:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:53:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:53:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 84A01DC262
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 18:52:47 +0000 (UTC)
Date: Tue, 24 May 2011 18:52:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1112509341.40079.1306263167539.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1659614701.18791.1297804257612.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7146) RPC server leaks file descriptors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7146:
--------------------------------

    Status: Open  (was: Patch Available)

> RPC server leaks file descriptors
> ---------------------------------
>
>                 Key: HADOOP-7146
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7146
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: fd-leak.txt, hadoop-7146.txt
>
>
> Both the Listener and Responder thread call Selector.open but don't have a matching .close(). This causes a leak of anonymous pipes. Not a big deal because people rarely close and re-open servers, but worth fixing.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15365-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 18:59:30 2011
Return-Path: <common-issues-return-15365-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 11F556EAF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 18:59:30 +0000 (UTC)
Received: (qmail 18535 invoked by uid 500); 24 May 2011 18:59:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18509 invoked by uid 500); 24 May 2011 18:59:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18501 invoked by uid 99); 24 May 2011 18:59:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:59:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 18:59:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D7CCFDC4FF
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 18:58:48 +0000 (UTC)
Date: Tue, 24 May 2011 18:58:48 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1115941752.40121.1306263528880.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038720#comment-13038720 ] 

Tom White commented on HADOOP-6671:
-----------------------------------

We currently publish jdiff documentation as a part of a release (e.g. http://hadoop.apache.org/common/docs/r0.20.2/jdiff/changes.html). There's also HADOOP-7035 which refines this to publish changes categorized by API stability and compatibility (there is an example at http://people.apache.org/~tomwhite/HADOOP-7035/common/).

HADOOP-7035 will include documenting the process for generating jdiff for a release, so I don't think that we need to get it integrated in Maven as a part of this issue. (If needed at a later point we could hook it into Maven by calling out to the script.) Does that sound reasonable?

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>         Attachments: HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15366-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 19:03:30 2011
Return-Path: <common-issues-return-15366-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C61EA60C7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 19:03:30 +0000 (UTC)
Received: (qmail 24755 invoked by uid 500); 24 May 2011 19:03:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24729 invoked by uid 500); 24 May 2011 19:03:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24721 invoked by uid 99); 24 May 2011 19:03:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 19:03:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 19:03:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 694BADC694
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 19:02:47 +0000 (UTC)
Date: Tue, 24 May 2011 19:02:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <290878101.40132.1306263767427.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <654871262.37888.1306191347393.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7323) Add capability to resolve
 compression codec based on codec name
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038724#comment-13038724 ] 

Hadoop QA commented on HADOOP-7323:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480300/HADOOP-7323b.patch
  against trunk revision 1127215.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/517//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/517//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/517//console

This message is automatically generated.

> Add capability to resolve compression codec based on codec name
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7323
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7323
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7323.patch, HADOOP-7323b.patch
>
>
> When setting up a compression codec in an MR job the full class name of the codec must be used.
> To ease usability, compression codecs should be resolved by their codec name (ie 'gzip', 'deflate', 'zlib', 'bzip2') instead their full codec class name.
> Besides easy of use for Hadoop users who would use the codec alias instead the full codec class name, it could simplify how HBase resolves loads the codecs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15367-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 19:05:31 2011
Return-Path: <common-issues-return-15367-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1638B64D6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 19:05:31 +0000 (UTC)
Received: (qmail 26707 invoked by uid 500); 24 May 2011 19:05:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26674 invoked by uid 500); 24 May 2011 19:05:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26666 invoked by uid 99); 24 May 2011 19:05:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 19:05:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 19:05:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B768DDC756
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 19:04:47 +0000 (UTC)
Date: Tue, 24 May 2011 19:04:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1033307530.40140.1306263887748.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <64692328.19459.1304449203116.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7258) Gzip codec should not return null
 decompressors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038725#comment-13038725 ] 

Hudson commented on HADOOP-7258:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #617 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/617/])
    HADOOP-7258. The Gzip codec should not return null decompressors. (omalley)

omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1127215
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/io/compress/DoNotPool.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/io/compress/CodecPool.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/io/compress/TestCodec.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipDecompressor.java


> Gzip codec should not return null decompressors
> -----------------------------------------------
>
>                 Key: HADOOP-7258
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7258
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: fix.patch, gzip-decomp.patch
>
>
> In HADOOP-6315, the gzip codec was changed to return a null codec with the intent to disallow pooling of the decompressors. Rather than break the interface, we can use an annotation to achieve the goal.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15368-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 19:14:32 2011
Return-Path: <common-issues-return-15368-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B9CB168BC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 19:14:32 +0000 (UTC)
Received: (qmail 41850 invoked by uid 500); 24 May 2011 19:14:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41747 invoked by uid 500); 24 May 2011 19:14:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41732 invoked by uid 99); 24 May 2011 19:14:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 19:14:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 19:14:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B8B03DCA67
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 19:13:47 +0000 (UTC)
Date: Tue, 24 May 2011 19:13:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1555700467.40158.1306264427753.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-6988) Add support for reading multiple
 hadoop delegation token files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers resolved HADOOP-6988.
------------------------------------

    Resolution: Won't Fix

Looks like we won't be reaching consensus on this issue. Closing out as it has long ago gone stale. Thanks for the thoughtful comments, everyone.

> Add support for reading multiple hadoop delegation token files
> --------------------------------------------------------------
>
>                 Key: HADOOP-6988
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6988
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>         Attachments: hadoop-6988.0.txt, hadoop-6988.1.txt
>
>
> It would be nice if there were a way to specify multiple delegation token files via the HADOOP_TOKEN_FILE_LOCATION environment variable and the "mapreduce.job.credentials.binary" configuration value. I suggest a colon-separated list of paths, each of which is read as a separate delegation token file.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15369-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 19:16:28 2011
Return-Path: <common-issues-return-15369-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 975D66E87
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 19:16:28 +0000 (UTC)
Received: (qmail 47729 invoked by uid 500); 24 May 2011 19:16:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47698 invoked by uid 500); 24 May 2011 19:16:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47690 invoked by uid 99); 24 May 2011 19:16:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 19:16:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 19:16:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 83BB2DCB37
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 19:15:47 +0000 (UTC)
Date: Tue, 24 May 2011 19:15:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <39417520.40191.1306264547536.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1659614701.18791.1297804257612.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7146) RPC server leaks file descriptors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7146:
--------------------------------

    Attachment: hadoop-7146.txt

I took all of your suggestions, except I didn't add null checks - the variables in question are never reassigned to null, so there's no way they could be, as far as I can tell.

> RPC server leaks file descriptors
> ---------------------------------
>
>                 Key: HADOOP-7146
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7146
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: fd-leak.txt, hadoop-7146.txt, hadoop-7146.txt
>
>
> Both the Listener and Responder thread call Selector.open but don't have a matching .close(). This causes a leak of anonymous pipes. Not a big deal because people rarely close and re-open servers, but worth fixing.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15370-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 19:16:31 2011
Return-Path: <common-issues-return-15370-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0E4AB6EC3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 19:16:31 +0000 (UTC)
Received: (qmail 47984 invoked by uid 500); 24 May 2011 19:16:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47947 invoked by uid 500); 24 May 2011 19:16:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47939 invoked by uid 99); 24 May 2011 19:16:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 19:16:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 19:16:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C39C0DCB3F
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 19:15:47 +0000 (UTC)
Date: Tue, 24 May 2011 19:15:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <215879726.40198.1306264547798.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1659614701.18791.1297804257612.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7146) RPC server leaks file descriptors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7146:
--------------------------------

    Status: Patch Available  (was: Open)

> RPC server leaks file descriptors
> ---------------------------------
>
>                 Key: HADOOP-7146
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7146
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: fd-leak.txt, hadoop-7146.txt, hadoop-7146.txt
>
>
> Both the Listener and Responder thread call Selector.open but don't have a matching .close(). This causes a leak of anonymous pipes. Not a big deal because people rarely close and re-open servers, but worth fixing.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15371-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 19:25:31 2011
Return-Path: <common-issues-return-15371-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 100BE633C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 19:25:31 +0000 (UTC)
Received: (qmail 70890 invoked by uid 500); 24 May 2011 19:25:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70861 invoked by uid 500); 24 May 2011 19:25:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70752 invoked by uid 99); 24 May 2011 19:25:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 19:25:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 19:25:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 765B7DCF3F
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 19:24:47 +0000 (UTC)
Date: Tue, 24 May 2011 19:24:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <753999512.40230.1306265087481.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7326) TestTrash.testTrashEmptier()
 infinite loops if run on a home directory with stuff in .Trash
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038742#comment-13038742 ] 

Todd Lipcon commented on HADOOP-7326:
-------------------------------------

- Rather than making CHECKPOINT public, can you add a new method like Trash.isCheckpointFileName(String)?
- It's still not clear why TestHDFSTrash was putting stuff in the local filesystem's .Trash folder. There are two things wrong there - 1) TestHDFSTrash is supposed to test HDFS, not local, and 2) it seems to try to set up the .Trash dir inside build/ so that different builds dont interact poorly.

> TestTrash.testTrashEmptier() infinite loops if run on a home directory with stuff in .Trash
> -------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7326
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7326
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>         Attachments: hadoop-7326-TestTrash-1.patch
>
>
> This bug was discovered while investigating HDFS-1967, intermittent failure of TestHDFSTrash, which calls TestTrash.  If the user id running the test has a ~/.Trash directory that already has 4 or more files in it, a loop in TestTrash.testTrashEmptier() will never terminate, because it is waiting for the count of objects to increase from zero to exactly three (and it creates one object right away).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15372-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 19:39:29 2011
Return-Path: <common-issues-return-15372-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3C9BE6D7A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 19:39:29 +0000 (UTC)
Received: (qmail 85701 invoked by uid 500); 24 May 2011 19:39:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85659 invoked by uid 500); 24 May 2011 19:39:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85651 invoked by uid 99); 24 May 2011 19:39:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 19:39:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 19:39:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 005F6DC3D5
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 19:38:48 +0000 (UTC)
Date: Tue, 24 May 2011 19:38:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <110476382.40255.1306265927998.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1659614701.18791.1297804257612.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7146) RPC server leaks file descriptors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038747#comment-13038747 ] 

Hadoop QA commented on HADOOP-7146:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480303/hadoop-7146.txt
  against trunk revision 1127215.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/518//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/518//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/518//console

This message is automatically generated.

> RPC server leaks file descriptors
> ---------------------------------
>
>                 Key: HADOOP-7146
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7146
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: fd-leak.txt, hadoop-7146.txt, hadoop-7146.txt
>
>
> Both the Listener and Responder thread call Selector.open but don't have a matching .close(). This causes a leak of anonymous pipes. Not a big deal because people rarely close and re-open servers, but worth fixing.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15373-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 20:30:31 2011
Return-Path: <common-issues-return-15373-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5DC006BB6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 20:30:31 +0000 (UTC)
Received: (qmail 63469 invoked by uid 500); 24 May 2011 20:30:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63361 invoked by uid 500); 24 May 2011 20:30:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63348 invoked by uid 99); 24 May 2011 20:30:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 20:30:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 20:30:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4C9BDDC54F
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 20:29:48 +0000 (UTC)
Date: Tue, 24 May 2011 20:29:48 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <815266661.40361.1306268988310.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038763#comment-13038763 ] 

Owen O'Malley commented on HADOOP-6671:
---------------------------------------

I don't think we should commit this until the jdiff is done.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>         Attachments: HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15374-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 20:36:29 2011
Return-Path: <common-issues-return-15374-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0F06D6195
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 20:36:29 +0000 (UTC)
Received: (qmail 86891 invoked by uid 500); 24 May 2011 20:36:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86851 invoked by uid 500); 24 May 2011 20:36:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86843 invoked by uid 99); 24 May 2011 20:36:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 20:36:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 20:36:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 60675DC85B
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 20:35:48 +0000 (UTC)
Date: Tue, 24 May 2011 20:35:48 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1081328090.40398.1306269348390.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7284:
---------------------------------

    Status: Open  (was: Patch Available)

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15375-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 20:36:31 2011
Return-Path: <common-issues-return-15375-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0D59B61A7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 20:36:31 +0000 (UTC)
Received: (qmail 87188 invoked by uid 500); 24 May 2011 20:36:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87153 invoked by uid 500); 24 May 2011 20:36:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87141 invoked by uid 99); 24 May 2011 20:36:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 20:36:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 20:36:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 84CB7DC83B
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 20:35:47 +0000 (UTC)
Date: Tue, 24 May 2011 20:35:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1778013909.40371.1306269347538.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038767#comment-13038767 ] 

Owen O'Malley commented on HADOOP-6671:
---------------------------------------

The current patch doesn't apply. It is trying to modify maven/checkstyle.xml.


> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>         Attachments: HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15376-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 20:38:28 2011
Return-Path: <common-issues-return-15376-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9B5EF6209
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 20:38:28 +0000 (UTC)
Received: (qmail 89492 invoked by uid 500); 24 May 2011 20:38:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89445 invoked by uid 500); 24 May 2011 20:38:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89435 invoked by uid 99); 24 May 2011 20:38:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 20:38:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 20:38:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D2F48DC991
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 20:37:47 +0000 (UTC)
Date: Tue, 24 May 2011 20:37:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1301372736.40415.1306269467861.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7284:
---------------------------------

    Status: Patch Available  (was: Open)

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15377-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 20:38:30 2011
Return-Path: <common-issues-return-15377-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BE0926222
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 20:38:30 +0000 (UTC)
Received: (qmail 89848 invoked by uid 500); 24 May 2011 20:38:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89808 invoked by uid 500); 24 May 2011 20:38:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89796 invoked by uid 99); 24 May 2011 20:38:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 20:38:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 20:38:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A045EDC98A
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 20:37:47 +0000 (UTC)
Date: Tue, 24 May 2011 20:37:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1222860111.40408.1306269467653.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7284:
---------------------------------

    Attachment: trash5.patch

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15378-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 20:40:31 2011
Return-Path: <common-issues-return-15378-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E0B8162D4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 20:40:30 +0000 (UTC)
Received: (qmail 93505 invoked by uid 500); 24 May 2011 20:40:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93453 invoked by uid 500); 24 May 2011 20:40:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93438 invoked by uid 99); 24 May 2011 20:40:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 20:40:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 20:40:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BCC66DCA2F
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 20:39:47 +0000 (UTC)
Date: Tue, 24 May 2011 20:39:47 +0000 (UTC)
From: "Bharath Mundlapudi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <905267277.40422.1306269587769.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1067116917.37592.1306187867429.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7322) Adding a util method in FileUtil for
 directory listing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bharath Mundlapudi updated HADOOP-7322:
---------------------------------------

    Attachment: HADOOP-7322-1.patch

> Adding a util method in FileUtil for directory listing
> ------------------------------------------------------
>
>                 Key: HADOOP-7322
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7322
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7322-1.patch
>
>
> While testing Disk Fail Inplace, we encountered lots of NPE from Dir.listFiles API. This API can return null when Dir is not directory or disk is bad. I am proposing to have a File Util which can be used consistently across to deal with disk issues. This util api will do the following:
> 1. When error happens it will throw IOException
> 2. Else it will return empty list or list of files.
> Signature:
> File[] FileUtil.listFiles(File dir) throws IOException {}
> This way we no need to write wrapper code every where. Also, API is consistent with the signature.
>  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15379-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 21:01:28 2011
Return-Path: <common-issues-return-15379-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AA5CD681C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 21:01:28 +0000 (UTC)
Received: (qmail 33476 invoked by uid 500); 24 May 2011 21:01:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33413 invoked by uid 500); 24 May 2011 21:01:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33405 invoked by uid 99); 24 May 2011 21:01:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 21:01:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 21:01:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C304BDC23F
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 21:00:47 +0000 (UTC)
Date: Tue, 24 May 2011 21:00:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1263165421.40455.1306270847795.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038779#comment-13038779 ] 

Hadoop QA commented on HADOOP-7284:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480312/trash5.patch
  against trunk revision 1127215.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 17 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.fs.viewfs.TestViewFsTrash

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/519//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/519//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/519//console

This message is automatically generated.

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15380-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 21:21:28 2011
Return-Path: <common-issues-return-15380-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7B8F86474
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 21:21:28 +0000 (UTC)
Received: (qmail 70461 invoked by uid 500); 24 May 2011 21:21:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70414 invoked by uid 500); 24 May 2011 21:21:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70406 invoked by uid 99); 24 May 2011 21:21:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 21:21:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 21:21:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 90F80DCB75
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 21:20:47 +0000 (UTC)
Date: Tue, 24 May 2011 21:20:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1286444982.40527.1306272047590.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1008788632.26385.1305807047638.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7305) Eclipse project files are
 incomplete
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038796#comment-13038796 ] 

Daryn Sharp commented on HADOOP-7305:
-------------------------------------

Explicitly including tools.jar breaks OS X because it uses a classes.jar instead of tools.jar.

My understanding is that a JDK will implicitly load tools.jar/classes.jar, whereas a JRE will not.  The patch is adding $JAVA_HOME/../lib/tools.jar to the classpath.  The use of ".." leads me to believe the submitter set JAVA_HOME to a JRE, and is using this patch to "break out" and load a JDK library.

If my understanding is correct, would someone please revert this patch?  Otherwise, could the classpath modification use a conditional so as to not break OS X?

> Eclipse project files are incomplete
> ------------------------------------
>
>                 Key: HADOOP-7305
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7305
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Niels Basjes
>            Assignee: Niels Basjes
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7305-2011-05-19.patch
>
>
> After a fresh checkout of hadoop-common I do 'ant compile eclipse'.
> I open eclipse, set ANT_HOME and build the project. 
> At that point the following error appears:
> {quote}
> The type com.sun.javadoc.RootDoc cannot be resolved. It is indirectly referenced from required .class files	ExcludePrivateAnnotationsJDiffDoclet.java	/common/src/java/org/apache/hadoop/classification/tools	line 1	Java Problem
> {quote}
> The solution is to add the "tools.jar" from the JDK to the buildpath/classpath.
> This should be fixed in the build.xml.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15381-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 21:37:29 2011
Return-Path: <common-issues-return-15381-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9445664DF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 21:37:29 +0000 (UTC)
Received: (qmail 7900 invoked by uid 500); 24 May 2011 21:37:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7773 invoked by uid 500); 24 May 2011 21:37:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7757 invoked by uid 99); 24 May 2011 21:37:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 21:37:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 21:37:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8197ADC9DA
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 21:36:47 +0000 (UTC)
Date: Tue, 24 May 2011 21:36:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <494721186.40603.1306273007527.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7326) TestTrash.testTrashEmptier()
 infinite loops if run on a home directory with stuff in .Trash
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038815#comment-13038815 ] 

Matt Foley commented on HADOOP-7326:
------------------------------------

Hi Todd,
bq. Rather than making CHECKPOINT public, can you add a new method like Trash.isCheckpointFileName(String)?

I'd prefer not to create this new API, for the following reasons:
* CHECKPOINT isn't public, it's protected, which is common practice in the Hadoop codebase for exposing internal stuff to unit tests.  
* It's a static final constant, so there's no harm in exposing it.  
* I only want to test the basename, not the whole path, so claiming "isCheckpointFileName" is larger scope than needed.  
* There's no other demand for a test for "isCheckpointBasename", as shown by the fact that it doesn't already exist.

bq. It's still not clear why TestHDFSTrash was putting stuff in the local filesystem's .Trash folder. There are two things wrong there - 1) TestHDFSTrash is supposed to test HDFS, not local...

Completely agree, so I've copied your comment into HDFS-1967, which deals with TestHDFSTrash.  If you can help me understand what TestHDFSTrash thinks it's doing, and whether it accomplishes it, I'd appreciate it.  (I think it's trying to do exactly equivalent tests on Local FileSystem with TestTrash and HDFS FileSystem with TestHDFSTrash, but it doesn't seem to be succeeding.)

bq. ...and 2) it seems [it should] try to set up the .Trash dir inside build/ so that different builds dont interact poorly.

Yah, this is a weird one.  I'm told that Trash is a fairly old feature, and was deliberately set up so that on local filesystems it would use the local canonical Trash.  Not what I would have chosen, but changing would require not just a change to the unit test, but a large change to the semantics of Trash.

Please let me know if you can +1 this patch on the basis of these responses.  Thanks.

> TestTrash.testTrashEmptier() infinite loops if run on a home directory with stuff in .Trash
> -------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7326
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7326
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>         Attachments: hadoop-7326-TestTrash-1.patch
>
>
> This bug was discovered while investigating HDFS-1967, intermittent failure of TestHDFSTrash, which calls TestTrash.  If the user id running the test has a ~/.Trash directory that already has 4 or more files in it, a loop in TestTrash.testTrashEmptier() will never terminate, because it is waiting for the count of objects to increase from zero to exactly three (and it creates one object right away).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15382-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 21:45:31 2011
Return-Path: <common-issues-return-15382-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1CC386D53
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 21:45:31 +0000 (UTC)
Received: (qmail 30638 invoked by uid 500); 24 May 2011 21:45:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30600 invoked by uid 500); 24 May 2011 21:45:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30587 invoked by uid 99); 24 May 2011 21:45:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 21:45:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 21:45:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B6789DC1A3
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 21:44:47 +0000 (UTC)
Date: Tue, 24 May 2011 21:44:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1306867000.40651.1306273487744.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6671:
----------------------------------

    Attachment: mvn-layout2.sh

This version uses svn mv to modify the working directory.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>         Attachments: HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15383-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 21:47:29 2011
Return-Path: <common-issues-return-15383-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 55E976BAE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 21:47:29 +0000 (UTC)
Received: (qmail 32820 invoked by uid 500); 24 May 2011 21:47:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32783 invoked by uid 500); 24 May 2011 21:47:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32750 invoked by uid 99); 24 May 2011 21:47:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 21:47:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 21:47:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8F402DC323
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 21:46:47 +0000 (UTC)
Date: Tue, 24 May 2011 21:46:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <101675095.40678.1306273607583.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038828#comment-13038828 ] 

Owen O'Malley commented on HADOOP-6671:
---------------------------------------

You should modify the script to also delete the files and directories that are no longer needed using svn rm.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>         Attachments: HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15384-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 21:53:29 2011
Return-Path: <common-issues-return-15384-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8729E6FDF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 21:53:29 +0000 (UTC)
Received: (qmail 39116 invoked by uid 500); 24 May 2011 21:53:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39064 invoked by uid 500); 24 May 2011 21:53:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39055 invoked by uid 99); 24 May 2011 21:53:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 21:53:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 21:53:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 985B7DC68C
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 21:52:48 +0000 (UTC)
Date: Tue, 24 May 2011 21:52:48 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1227102234.40753.1306273968620.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038834#comment-13038834 ] 

Hadoop QA commented on HADOOP-6671:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480329/mvn-layout2.sh
  against trunk revision 1127215.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 11 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/520//console

This message is automatically generated.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>         Attachments: HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15385-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 22:03:30 2011
Return-Path: <common-issues-return-15385-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8DFA562CD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 22:03:30 +0000 (UTC)
Received: (qmail 52205 invoked by uid 500); 24 May 2011 22:03:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52167 invoked by uid 500); 24 May 2011 22:03:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52156 invoked by uid 99); 24 May 2011 22:03:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 22:03:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 22:03:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 77442DCB77
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 22:02:47 +0000 (UTC)
Date: Tue, 24 May 2011 22:02:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <678927083.40820.1306274567485.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <360710663.20918.1304489523226.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7259) contrib modules should include
 build.properties from parent.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038846#comment-13038846 ] 

Hudson commented on HADOOP-7259:
--------------------------------

Integrated in Hadoop-Hdfs-trunk-Commit #681 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/681/])
    

> contrib modules should include build.properties from parent.
> ------------------------------------------------------------
>
>                 Key: HADOOP-7259
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7259
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: contrib.patch
>
>
> Current build.properties in the hadoop root directory is not included by the contrib modules.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15386-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 22:14:28 2011
Return-Path: <common-issues-return-15386-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 765C269E2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 22:14:28 +0000 (UTC)
Received: (qmail 67594 invoked by uid 500); 24 May 2011 22:14:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67568 invoked by uid 500); 24 May 2011 22:14:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67560 invoked by uid 99); 24 May 2011 22:14:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 22:14:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 22:14:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 851CBDCF59
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 22:13:47 +0000 (UTC)
Date: Tue, 24 May 2011 22:13:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1151992895.40839.1306275227541.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7326) TestTrash.testTrashEmptier()
 infinite loops if run on a home directory with stuff in .Trash
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038851#comment-13038851 ] 

Matt Foley commented on HADOOP-7326:
------------------------------------

Minor goof: the last LOG.warn() statement uses "+ e" but should use ", e"

> TestTrash.testTrashEmptier() infinite loops if run on a home directory with stuff in .Trash
> -------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7326
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7326
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>         Attachments: hadoop-7326-TestTrash-1.patch
>
>
> This bug was discovered while investigating HDFS-1967, intermittent failure of TestHDFSTrash, which calls TestTrash.  If the user id running the test has a ~/.Trash directory that already has 4 or more files in it, a loop in TestTrash.testTrashEmptier() will never terminate, because it is waiting for the count of objects to increase from zero to exactly three (and it creates one object right away).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15387-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 22:32:30 2011
Return-Path: <common-issues-return-15387-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AF1656EBA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 22:32:30 +0000 (UTC)
Received: (qmail 90876 invoked by uid 500); 24 May 2011 22:32:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90847 invoked by uid 500); 24 May 2011 22:32:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90839 invoked by uid 99); 24 May 2011 22:32:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 22:32:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 22:32:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8D4E7DC66B
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 22:31:47 +0000 (UTC)
Date: Tue, 24 May 2011 22:31:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <584726501.40914.1306276307575.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1659614701.18791.1297804257612.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7146) RPC server leaks file descriptors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038865#comment-13038865 ] 

Matt Foley commented on HADOOP-7146:
------------------------------------

bq. I didn't add null checks - the variables in question are never reassigned to null, so there's no way they could be, as far as I can tell.

Quite right. To safeguard that, could you please make those two variables "final"?

Minor: In Reader.doRunLoop(), the LOG.Info() could use ", e" instead of "+ StringUtils.stringifyException(e)"

+1 if those small changes are okay with you.

> RPC server leaks file descriptors
> ---------------------------------
>
>                 Key: HADOOP-7146
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7146
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: fd-leak.txt, hadoop-7146.txt, hadoop-7146.txt
>
>
> Both the Listener and Responder thread call Selector.open but don't have a matching .close(). This causes a leak of anonymous pipes. Not a big deal because people rarely close and re-open servers, but worth fixing.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15388-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 22:36:30 2011
Return-Path: <common-issues-return-15388-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9B85468A4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 22:36:30 +0000 (UTC)
Received: (qmail 94584 invoked by uid 500); 24 May 2011 22:36:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94553 invoked by uid 500); 24 May 2011 22:36:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94544 invoked by uid 99); 24 May 2011 22:36:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 22:36:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 22:36:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 86C0DDC832
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 22:35:47 +0000 (UTC)
Date: Tue, 24 May 2011 22:35:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2135158207.40935.1306276547548.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7284:
---------------------------------

    Status: Open  (was: Patch Available)

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15389-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 22:36:32 2011
Return-Path: <common-issues-return-15389-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8F83F68D2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 22:36:32 +0000 (UTC)
Received: (qmail 94927 invoked by uid 500); 24 May 2011 22:36:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94819 invoked by uid 500); 24 May 2011 22:36:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94801 invoked by uid 99); 24 May 2011 22:36:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 22:36:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 22:36:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8E427DC834
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 22:35:47 +0000 (UTC)
Date: Tue, 24 May 2011 22:35:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1152243679.40936.1306276547579.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7284:
---------------------------------

    Attachment: trash6.patch

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15390-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 23:14:30 2011
Return-Path: <common-issues-return-15390-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A71E766E1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 23:14:30 +0000 (UTC)
Received: (qmail 24071 invoked by uid 500); 24 May 2011 23:14:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24044 invoked by uid 500); 24 May 2011 23:14:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24034 invoked by uid 99); 24 May 2011 23:14:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 23:14:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 23:14:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8C970DC26A
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 23:13:47 +0000 (UTC)
Date: Tue, 24 May 2011 23:13:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <547658332.40988.1306278827572.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1067116917.37592.1306187867429.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7322) Adding a util method in FileUtil
 for directory listing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038878#comment-13038878 ] 

Matt Foley commented on HADOOP-7322:
------------------------------------

Hi Bharath,
1. suggest javadoc comment for FileUtil.listFiles() to read:
+   * A wrapper for {@link File#listFiles()}. This java.io API returns null 
+   * when dir is not directory or for any I/O error. Instead of having to
+   * null check everywhere File#listFiles() is used, we will add this utility 
+   * api to get around this problem, for the majority of cases where we prefer
+   * an IOException to be thrown.

2. Not sure about the change to RunJar.main().  Isn't jar extraction usually pretty forgiving?  It is currently written to skip any directory it can't read.  Can you please give an argument for why that's wrong?

3. In the last line of testlistFiles(), to assure that the referenced directory can't possibly exist, why not use the same name as the directory you just deleted in the previous line?  Maybe after asserting:
Assert.assertTrue("Failed to delete test dir", !newDir.exists());

> Adding a util method in FileUtil for directory listing
> ------------------------------------------------------
>
>                 Key: HADOOP-7322
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7322
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7322-1.patch
>
>
> While testing Disk Fail Inplace, we encountered lots of NPE from Dir.listFiles API. This API can return null when Dir is not directory or disk is bad. I am proposing to have a File Util which can be used consistently across to deal with disk issues. This util api will do the following:
> 1. When error happens it will throw IOException
> 2. Else it will return empty list or list of files.
> Signature:
> File[] FileUtil.listFiles(File dir) throws IOException {}
> This way we no need to write wrapper code every where. Also, API is consistent with the signature.
>  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15391-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 23:19:28 2011
Return-Path: <common-issues-return-15391-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 681F16850
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 23:19:28 +0000 (UTC)
Received: (qmail 28490 invoked by uid 500); 24 May 2011 23:19:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28466 invoked by uid 500); 24 May 2011 23:19:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28458 invoked by uid 99); 24 May 2011 23:19:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 23:19:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 23:19:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5DD50DC3A7
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 23:18:47 +0000 (UTC)
Date: Tue, 24 May 2011 23:18:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1890735514.40995.1306279127381.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7284:
---------------------------------

    Status: Patch Available  (was: Open)

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15392-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 24 23:33:28 2011
Return-Path: <common-issues-return-15392-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 86FF16E3D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 24 May 2011 23:33:28 +0000 (UTC)
Received: (qmail 40394 invoked by uid 500); 24 May 2011 23:33:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40350 invoked by uid 500); 24 May 2011 23:33:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40341 invoked by uid 99); 24 May 2011 23:33:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 23:33:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 May 2011 23:33:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 94B12DC74C
	for <common-issues@hadoop.apache.org>; Tue, 24 May 2011 23:32:47 +0000 (UTC)
Date: Tue, 24 May 2011 23:32:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <483742036.41009.1306279967605.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038881#comment-13038881 ] 

Hadoop QA commented on HADOOP-7284:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480338/trash6.patch
  against trunk revision 1127215.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 17 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.fs.viewfs.TestViewFsTrash

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/521//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/521//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/521//console

This message is automatically generated.

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15393-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 00:33:30 2011
Return-Path: <common-issues-return-15393-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AEE0C6521
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 00:33:30 +0000 (UTC)
Received: (qmail 4971 invoked by uid 500); 25 May 2011 00:33:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4945 invoked by uid 500); 25 May 2011 00:33:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4937 invoked by uid 99); 25 May 2011 00:33:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 00:33:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 00:33:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 83859DC435
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 00:32:47 +0000 (UTC)
Date: Wed, 25 May 2011 00:32:47 +0000 (UTC)
From: "Bharath Mundlapudi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1932253256.41056.1306283567535.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1067116917.37592.1306187867429.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7322) Adding a util method in FileUtil for
 directory listing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bharath Mundlapudi updated HADOOP-7322:
---------------------------------------

    Attachment: HADOOP-7322-2.patch

Thanks for the comments, Matt. 

Regarding 2: Yes, we can be forgiving on this case. I added this because of eliminating null check. 

Attaching a patch which addresses all these comments. 

> Adding a util method in FileUtil for directory listing
> ------------------------------------------------------
>
>                 Key: HADOOP-7322
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7322
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7322-1.patch, HADOOP-7322-2.patch
>
>
> While testing Disk Fail Inplace, we encountered lots of NPE from Dir.listFiles API. This API can return null when Dir is not directory or disk is bad. I am proposing to have a File Util which can be used consistently across to deal with disk issues. This util api will do the following:
> 1. When error happens it will throw IOException
> 2. Else it will return empty list or list of files.
> Signature:
> File[] FileUtil.listFiles(File dir) throws IOException {}
> This way we no need to write wrapper code every where. Also, API is consistent with the signature.
>  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15394-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 01:23:28 2011
Return-Path: <common-issues-return-15394-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 823AA6AA1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 01:23:28 +0000 (UTC)
Received: (qmail 53812 invoked by uid 500); 25 May 2011 01:23:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53784 invoked by uid 500); 25 May 2011 01:23:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53776 invoked by uid 99); 25 May 2011 01:23:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 01:23:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 01:23:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5914CDD21A
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 01:22:47 +0000 (UTC)
Date: Wed, 25 May 2011 01:22:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <495082955.41159.1306286567361.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1067116917.37592.1306187867429.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7322) Adding a util method in FileUtil
 for directory listing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038908#comment-13038908 ] 

Matt Foley commented on HADOOP-7322:
------------------------------------

Thanks, Bharath.  Unfortunately I have one more item.  Looking more at testlistFiles(), there are many calls that could cause an IOException, but only the last one should be an allowed IOException.  Therefore, I think we have to remove the "expected=IOException.class" from the @Test annotation, and instead use the idiom:
{code}
try {
  files = FileUtil.listFiles(newDir);
  fail("IOException expected on listFiles() for non-existent dir " + newDir.getString());
} catch (IOException ioe) {
  //expected
}
{code}


> Adding a util method in FileUtil for directory listing
> ------------------------------------------------------
>
>                 Key: HADOOP-7322
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7322
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7322-1.patch, HADOOP-7322-2.patch
>
>
> While testing Disk Fail Inplace, we encountered lots of NPE from Dir.listFiles API. This API can return null when Dir is not directory or disk is bad. I am proposing to have a File Util which can be used consistently across to deal with disk issues. This util api will do the following:
> 1. When error happens it will throw IOException
> 2. Else it will return empty list or list of files.
> Signature:
> File[] FileUtil.listFiles(File dir) throws IOException {}
> This way we no need to write wrapper code every where. Also, API is consistent with the signature.
>  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15395-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 06:24:35 2011
Return-Path: <common-issues-return-15395-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C586C4239
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 06:24:35 +0000 (UTC)
Received: (qmail 48744 invoked by uid 500); 25 May 2011 06:24:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48527 invoked by uid 500); 25 May 2011 06:24:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48511 invoked by uid 99); 25 May 2011 06:24:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 06:24:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 06:24:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 67AE8DD169
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 06:23:47 +0000 (UTC)
Date: Wed, 25 May 2011 06:23:47 +0000 (UTC)
From: "XieXianshan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1164170325.41524.1306304627421.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7329) incomplete help message  is
 displayed for df -h option
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

incomplete help message  is displayed for df -h option
------------------------------------------------------

                 Key: HADOOP-7329
                 URL: https://issues.apache.org/jira/browse/HADOOP-7329
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs
    Affects Versions: 0.23.0
            Reporter: XieXianshan
            Priority: Minor
             Fix For: 0.23.0


The help message for the command "hdfs dfs -help df" is displayed like this:
"-df [<path> ...]:    Shows the capacity, free and used space of the filesystem.
        If the filesystem has multiple partitions, and no path to a
        particular partition is specified, then the status of the root
        partitions will be shown."
and the information about df -h option is missed,despite the fact that df -h option is implemented.

Therefore,the expected message should be displayed like this:
"-df [-h] [<path> ...]:    Shows the capacity, free and used space of the filesystem.
        If the filesystem has multiple partitions, and no path to a
        particular partition is specified, then the status of the root
        partitions will be shown.
          -h   Formats the sizes of files in a human-readable fashion
               rather than a number of bytes."

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15396-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 06:38:34 2011
Return-Path: <common-issues-return-15396-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 123D44994
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 06:38:34 +0000 (UTC)
Received: (qmail 60532 invoked by uid 500); 25 May 2011 06:38:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59975 invoked by uid 500); 25 May 2011 06:38:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59896 invoked by uid 99); 25 May 2011 06:38:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 06:38:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 06:38:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6194CDD5BB
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 06:37:47 +0000 (UTC)
Date: Wed, 25 May 2011 06:37:47 +0000 (UTC)
From: "XieXianshan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <183232839.41536.1306305467396.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1164170325.41524.1306304627421.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7329) incomplete help message  is
 displayed for df -h option
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

XieXianshan updated HADOOP-7329:
--------------------------------

    Attachment: HADOOP-7329

fix the help message for df -h option.

> incomplete help message  is displayed for df -h option
> ------------------------------------------------------
>
>                 Key: HADOOP-7329
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7329
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7329
>
>
> The help message for the command "hdfs dfs -help df" is displayed like this:
> "-df [<path> ...]:    Shows the capacity, free and used space of the filesystem.
>         If the filesystem has multiple partitions, and no path to a
>         particular partition is specified, then the status of the root
>         partitions will be shown."
> and the information about df -h option is missed,despite the fact that df -h option is implemented.
> Therefore,the expected message should be displayed like this:
> "-df [-h] [<path> ...]:    Shows the capacity, free and used space of the filesystem.
>         If the filesystem has multiple partitions, and no path to a
>         particular partition is specified, then the status of the root
>         partitions will be shown.
>           -h   Formats the sizes of files in a human-readable fashion
>                rather than a number of bytes."

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15397-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 06:40:31 2011
Return-Path: <common-issues-return-15397-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4CC5549C1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 06:40:31 +0000 (UTC)
Received: (qmail 61737 invoked by uid 500); 25 May 2011 06:40:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61712 invoked by uid 500); 25 May 2011 06:40:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61704 invoked by uid 99); 25 May 2011 06:40:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 06:40:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 06:40:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5A318DD684
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 06:39:47 +0000 (UTC)
Date: Wed, 25 May 2011 06:39:47 +0000 (UTC)
From: "XieXianshan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <903607703.41541.1306305587365.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1164170325.41524.1306304627421.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7329) incomplete help message  is
 displayed for df -h option
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

XieXianshan updated HADOOP-7329:
--------------------------------

    Status: Patch Available  (was: Open)

Fix the help message for df -h option.

> incomplete help message  is displayed for df -h option
> ------------------------------------------------------
>
>                 Key: HADOOP-7329
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7329
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7329
>
>
> The help message for the command "hdfs dfs -help df" is displayed like this:
> "-df [<path> ...]:    Shows the capacity, free and used space of the filesystem.
>         If the filesystem has multiple partitions, and no path to a
>         particular partition is specified, then the status of the root
>         partitions will be shown."
> and the information about df -h option is missed,despite the fact that df -h option is implemented.
> Therefore,the expected message should be displayed like this:
> "-df [-h] [<path> ...]:    Shows the capacity, free and used space of the filesystem.
>         If the filesystem has multiple partitions, and no path to a
>         particular partition is specified, then the status of the root
>         partitions will be shown.
>           -h   Formats the sizes of files in a human-readable fashion
>                rather than a number of bytes."

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15398-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 07:01:33 2011
Return-Path: <common-issues-return-15398-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F3D464B2F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 07:01:32 +0000 (UTC)
Received: (qmail 83734 invoked by uid 500); 25 May 2011 07:01:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83532 invoked by uid 500); 25 May 2011 07:01:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83524 invoked by uid 99); 25 May 2011 07:01:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 07:01:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 07:01:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 87488DDE36
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 07:00:47 +0000 (UTC)
Date: Wed, 25 May 2011 07:00:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1447018837.41611.1306306847550.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1164170325.41524.1306304627421.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7329) incomplete help message  is
 displayed for df -h option
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038990#comment-13038990 ] 

Hadoop QA commented on HADOOP-7329:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480372/HADOOP-7329
  against trunk revision 1127215.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/522//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/522//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/522//console

This message is automatically generated.

> incomplete help message  is displayed for df -h option
> ------------------------------------------------------
>
>                 Key: HADOOP-7329
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7329
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7329
>
>
> The help message for the command "hdfs dfs -help df" is displayed like this:
> "-df [<path> ...]:    Shows the capacity, free and used space of the filesystem.
>         If the filesystem has multiple partitions, and no path to a
>         particular partition is specified, then the status of the root
>         partitions will be shown."
> and the information about df -h option is missed,despite the fact that df -h option is implemented.
> Therefore,the expected message should be displayed like this:
> "-df [-h] [<path> ...]:    Shows the capacity, free and used space of the filesystem.
>         If the filesystem has multiple partitions, and no path to a
>         particular partition is specified, then the status of the root
>         partitions will be shown.
>           -h   Formats the sizes of files in a human-readable fashion
>                rather than a number of bytes."

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15399-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 07:07:37 2011
Return-Path: <common-issues-return-15399-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 125774ADA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 07:07:37 +0000 (UTC)
Received: (qmail 91017 invoked by uid 500); 25 May 2011 07:07:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90988 invoked by uid 500); 25 May 2011 07:07:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90968 invoked by uid 99); 25 May 2011 07:07:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 07:07:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 07:07:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5C281DD017
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 07:06:47 +0000 (UTC)
Date: Wed, 25 May 2011 07:06:47 +0000 (UTC)
From: "Harsh J Chouraria (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1133222311.41620.1306307207373.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1164170325.41524.1306304627421.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7329) incomplete help message  is
 displayed for df -h option
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038992#comment-13038992 ] 

Harsh J Chouraria commented on HADOOP-7329:
-------------------------------------------

Patch LGTM :)

> incomplete help message  is displayed for df -h option
> ------------------------------------------------------
>
>                 Key: HADOOP-7329
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7329
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7329
>
>
> The help message for the command "hdfs dfs -help df" is displayed like this:
> "-df [<path> ...]:    Shows the capacity, free and used space of the filesystem.
>         If the filesystem has multiple partitions, and no path to a
>         particular partition is specified, then the status of the root
>         partitions will be shown."
> and the information about df -h option is missed,despite the fact that df -h option is implemented.
> Therefore,the expected message should be displayed like this:
> "-df [-h] [<path> ...]:    Shows the capacity, free and used space of the filesystem.
>         If the filesystem has multiple partitions, and no path to a
>         particular partition is specified, then the status of the root
>         partitions will be shown.
>           -h   Formats the sizes of files in a human-readable fashion
>                rather than a number of bytes."

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15400-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 07:35:33 2011
Return-Path: <common-issues-return-15400-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D37124AC4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 07:35:33 +0000 (UTC)
Received: (qmail 16790 invoked by uid 500); 25 May 2011 07:35:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16545 invoked by uid 500); 25 May 2011 07:35:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16528 invoked by uid 99); 25 May 2011 07:35:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 07:35:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 07:35:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 85C00DD92E
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 07:34:47 +0000 (UTC)
Date: Wed, 25 May 2011 07:34:47 +0000 (UTC)
From: "Bharath Mundlapudi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <103123414.41644.1306308887544.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1067116917.37592.1306187867429.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7322) Adding a util method in FileUtil for
 directory listing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bharath Mundlapudi updated HADOOP-7322:
---------------------------------------

    Attachment: HADOOP-7322-3.patch

Not a problem. I have updated the testcase.

> Adding a util method in FileUtil for directory listing
> ------------------------------------------------------
>
>                 Key: HADOOP-7322
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7322
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7322-1.patch, HADOOP-7322-2.patch, HADOOP-7322-3.patch
>
>
> While testing Disk Fail Inplace, we encountered lots of NPE from Dir.listFiles API. This API can return null when Dir is not directory or disk is bad. I am proposing to have a File Util which can be used consistently across to deal with disk issues. This util api will do the following:
> 1. When error happens it will throw IOException
> 2. Else it will return empty list or list of files.
> Signature:
> File[] FileUtil.listFiles(File dir) throws IOException {}
> This way we no need to write wrapper code every where. Also, API is consistent with the signature.
>  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15401-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 07:35:34 2011
Return-Path: <common-issues-return-15401-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DE63D4AC5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 07:35:33 +0000 (UTC)
Received: (qmail 16789 invoked by uid 500); 25 May 2011 07:35:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16547 invoked by uid 500); 25 May 2011 07:35:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16536 invoked by uid 99); 25 May 2011 07:35:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 07:35:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 07:35:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9AB69DD930
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 07:34:47 +0000 (UTC)
Date: Wed, 25 May 2011 07:34:47 +0000 (UTC)
From: "Bharath Mundlapudi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1682192711.41646.1306308887630.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1067116917.37592.1306187867429.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7322) Adding a util method in FileUtil for
 directory listing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bharath Mundlapudi updated HADOOP-7322:
---------------------------------------

    Status: Patch Available  (was: Open)

> Adding a util method in FileUtil for directory listing
> ------------------------------------------------------
>
>                 Key: HADOOP-7322
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7322
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7322-1.patch, HADOOP-7322-2.patch, HADOOP-7322-3.patch
>
>
> While testing Disk Fail Inplace, we encountered lots of NPE from Dir.listFiles API. This API can return null when Dir is not directory or disk is bad. I am proposing to have a File Util which can be used consistently across to deal with disk issues. This util api will do the following:
> 1. When error happens it will throw IOException
> 2. Else it will return empty list or list of files.
> Signature:
> File[] FileUtil.listFiles(File dir) throws IOException {}
> This way we no need to write wrapper code every where. Also, API is consistent with the signature.
>  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15402-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 08:00:29 2011
Return-Path: <common-issues-return-15402-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EDBAE4857
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 08:00:29 +0000 (UTC)
Received: (qmail 60766 invoked by uid 500); 25 May 2011 08:00:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60615 invoked by uid 500); 25 May 2011 08:00:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60405 invoked by uid 99); 25 May 2011 08:00:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 08:00:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 08:00:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7A489DD35B
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 07:59:47 +0000 (UTC)
Date: Wed, 25 May 2011 07:59:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1751023592.41697.1306310387497.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1067116917.37592.1306187867429.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7322) Adding a util method in FileUtil
 for directory listing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039009#comment-13039009 ] 

Hadoop QA commented on HADOOP-7322:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480373/HADOOP-7322-3.patch
  against trunk revision 1127215.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/523//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/523//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/523//console

This message is automatically generated.

> Adding a util method in FileUtil for directory listing
> ------------------------------------------------------
>
>                 Key: HADOOP-7322
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7322
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7322-1.patch, HADOOP-7322-2.patch, HADOOP-7322-3.patch
>
>
> While testing Disk Fail Inplace, we encountered lots of NPE from Dir.listFiles API. This API can return null when Dir is not directory or disk is bad. I am proposing to have a File Util which can be used consistently across to deal with disk issues. This util api will do the following:
> 1. When error happens it will throw IOException
> 2. Else it will return empty list or list of files.
> Signature:
> File[] FileUtil.listFiles(File dir) throws IOException {}
> This way we no need to write wrapper code every where. Also, API is consistent with the signature.
>  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15403-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 08:38:29 2011
Return-Path: <common-issues-return-15403-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 265564444
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 08:38:29 +0000 (UTC)
Received: (qmail 23738 invoked by uid 500); 25 May 2011 08:38:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23694 invoked by uid 500); 25 May 2011 08:38:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23681 invoked by uid 99); 25 May 2011 08:38:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 08:38:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 08:38:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 1CC5FDD535
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 08:37:48 +0000 (UTC)
Date: Wed, 25 May 2011 08:37:48 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <960913349.41918.1306312668114.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7284:
---------------------------------

    Attachment: trash7.patch

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15404-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 08:38:29 2011
Return-Path: <common-issues-return-15404-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4BED6444F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 08:38:29 +0000 (UTC)
Received: (qmail 23854 invoked by uid 500); 25 May 2011 08:38:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23704 invoked by uid 500); 25 May 2011 08:38:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23695 invoked by uid 99); 25 May 2011 08:38:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 08:38:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 08:38:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 3E9BDDD539
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 08:37:48 +0000 (UTC)
Date: Wed, 25 May 2011 08:37:48 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <613865644.41922.1306312668253.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7284:
---------------------------------

    Status: Patch Available  (was: Open)

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15405-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 08:38:31 2011
Return-Path: <common-issues-return-15405-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 515D14465
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 08:38:31 +0000 (UTC)
Received: (qmail 24372 invoked by uid 500); 25 May 2011 08:38:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24346 invoked by uid 500); 25 May 2011 08:38:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24338 invoked by uid 99); 25 May 2011 08:38:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 08:38:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 08:38:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0FAC3DD533
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 08:37:48 +0000 (UTC)
Date: Wed, 25 May 2011 08:37:48 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1819459722.41916.1306312668060.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7284:
---------------------------------

    Status: Open  (was: Patch Available)

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15406-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 10:20:30 2011
Return-Path: <common-issues-return-15406-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C18A94142
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 10:20:30 +0000 (UTC)
Received: (qmail 37530 invoked by uid 500); 25 May 2011 10:20:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37501 invoked by uid 500); 25 May 2011 10:20:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37493 invoked by uid 99); 25 May 2011 10:20:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 10:20:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 10:20:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6E20ADD034
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 10:19:47 +0000 (UTC)
Date: Wed, 25 May 2011 10:19:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2037941060.42061.1306318787447.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039059#comment-13039059 ] 

Hadoop QA commented on HADOOP-7284:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480383/trash7.patch
  against trunk revision 1127215.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 17 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/524//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/524//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/524//console

This message is automatically generated.

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15407-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 11:16:28 2011
Return-Path: <common-issues-return-15407-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 88F954FDA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 11:16:28 +0000 (UTC)
Received: (qmail 1441 invoked by uid 500); 25 May 2011 11:16:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1412 invoked by uid 500); 25 May 2011 11:16:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1401 invoked by uid 99); 25 May 2011 11:16:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 11:16:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 11:16:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5B0EEDD5D6
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 11:15:47 +0000 (UTC)
Date: Wed, 25 May 2011 11:15:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1756668841.42109.1306322147369.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <64692328.19459.1304449203116.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7258) Gzip codec should not return null
 decompressors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039074#comment-13039074 ] 

Hudson commented on HADOOP-7258:
--------------------------------

Integrated in Hadoop-Common-trunk #699 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/699/])
    HADOOP-7258. The Gzip codec should not return null decompressors. (omalley)

omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1127215
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/io/compress/DoNotPool.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/io/compress/CodecPool.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/io/compress/TestCodec.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipDecompressor.java


> Gzip codec should not return null decompressors
> -----------------------------------------------
>
>                 Key: HADOOP-7258
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7258
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.203.0, 0.23.0
>
>         Attachments: fix.patch, gzip-decomp.patch
>
>
> In HADOOP-6315, the gzip codec was changed to return a null codec with the intent to disallow pooling of the decompressors. Rather than break the interface, we can use an annotation to achieve the goal.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15408-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 16:22:32 2011
Return-Path: <common-issues-return-15408-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8A9E546A7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 16:22:32 +0000 (UTC)
Received: (qmail 12577 invoked by uid 500); 25 May 2011 16:22:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12353 invoked by uid 500); 25 May 2011 16:22:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12218 invoked by uid 99); 25 May 2011 16:22:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 16:22:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 16:22:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9E939DE773
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 16:21:47 +0000 (UTC)
Date: Wed, 25 May 2011 16:21:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <714147725.42596.1306340507646.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1659614701.18791.1297804257612.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7146) RPC server leaks file descriptors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7146:
--------------------------------

    Attachment: hadoop-7146.txt

New patch addressing Matt's latest comments.

I did change the log messages, though they weren't new from this patch. I agree that it's better not to use stringify, though.

> RPC server leaks file descriptors
> ---------------------------------
>
>                 Key: HADOOP-7146
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7146
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: fd-leak.txt, hadoop-7146.txt, hadoop-7146.txt, hadoop-7146.txt
>
>
> Both the Listener and Responder thread call Selector.open but don't have a matching .close(). This causes a leak of anonymous pipes. Not a big deal because people rarely close and re-open servers, but worth fixing.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15409-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 16:32:29 2011
Return-Path: <common-issues-return-15409-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 39F1D43E7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 16:32:29 +0000 (UTC)
Received: (qmail 27162 invoked by uid 500); 25 May 2011 16:32:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27123 invoked by uid 500); 25 May 2011 16:32:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27115 invoked by uid 99); 25 May 2011 16:32:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 16:32:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 16:32:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0C5D0DEAD2
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 16:31:48 +0000 (UTC)
Date: Wed, 25 May 2011 16:31:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1648411209.42627.1306341108047.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1164170325.41524.1306304627421.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7329) incomplete help message  is
 displayed for df -h option
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039189#comment-13039189 ] 

Todd Lipcon commented on HADOOP-7329:
-------------------------------------

+1. I don't think we need tests for this - we don't currently have any tests on help output.

I manually verified the output of "hadoop fs -help" with the patch and it looks great. Thanks.

> incomplete help message  is displayed for df -h option
> ------------------------------------------------------
>
>                 Key: HADOOP-7329
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7329
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7329
>
>
> The help message for the command "hdfs dfs -help df" is displayed like this:
> "-df [<path> ...]:    Shows the capacity, free and used space of the filesystem.
>         If the filesystem has multiple partitions, and no path to a
>         particular partition is specified, then the status of the root
>         partitions will be shown."
> and the information about df -h option is missed,despite the fact that df -h option is implemented.
> Therefore,the expected message should be displayed like this:
> "-df [-h] [<path> ...]:    Shows the capacity, free and used space of the filesystem.
>         If the filesystem has multiple partitions, and no path to a
>         particular partition is specified, then the status of the root
>         partitions will be shown.
>           -h   Formats the sizes of files in a human-readable fashion
>                rather than a number of bytes."

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15410-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 16:34:31 2011
Return-Path: <common-issues-return-15410-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 005324DEC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 16:34:30 +0000 (UTC)
Received: (qmail 31504 invoked by uid 500); 25 May 2011 16:34:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31416 invoked by uid 500); 25 May 2011 16:34:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31311 invoked by uid 99); 25 May 2011 16:34:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 16:34:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 16:34:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8CED2DEBDB
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 16:33:47 +0000 (UTC)
Date: Wed, 25 May 2011 16:33:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <946467755.42634.1306341227574.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1164170325.41524.1306304627421.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7329) incomplete help message  is
 displayed for df -h option
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7329:
--------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks XieXianshan.

> incomplete help message  is displayed for df -h option
> ------------------------------------------------------
>
>                 Key: HADOOP-7329
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7329
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7329
>
>
> The help message for the command "hdfs dfs -help df" is displayed like this:
> "-df [<path> ...]:    Shows the capacity, free and used space of the filesystem.
>         If the filesystem has multiple partitions, and no path to a
>         particular partition is specified, then the status of the root
>         partitions will be shown."
> and the information about df -h option is missed,despite the fact that df -h option is implemented.
> Therefore,the expected message should be displayed like this:
> "-df [-h] [<path> ...]:    Shows the capacity, free and used space of the filesystem.
>         If the filesystem has multiple partitions, and no path to a
>         particular partition is specified, then the status of the root
>         partitions will be shown.
>           -h   Formats the sizes of files in a human-readable fashion
>                rather than a number of bytes."

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15411-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 16:34:32 2011
Return-Path: <common-issues-return-15411-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2776F4E01
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 16:34:31 +0000 (UTC)
Received: (qmail 31677 invoked by uid 500); 25 May 2011 16:34:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31619 invoked by uid 500); 25 May 2011 16:34:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31418 invoked by uid 99); 25 May 2011 16:34:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 16:34:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 16:34:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id ADB5BDEBE4
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 16:33:47 +0000 (UTC)
Date: Wed, 25 May 2011 16:33:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1486146470.42638.1306341227708.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1164170325.41524.1306304627421.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7329) incomplete help message  is
 displayed for df -h option
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon reassigned HADOOP-7329:
-----------------------------------

    Assignee: XieXianshan

> incomplete help message  is displayed for df -h option
> ------------------------------------------------------
>
>                 Key: HADOOP-7329
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7329
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Assignee: XieXianshan
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7329
>
>
> The help message for the command "hdfs dfs -help df" is displayed like this:
> "-df [<path> ...]:    Shows the capacity, free and used space of the filesystem.
>         If the filesystem has multiple partitions, and no path to a
>         particular partition is specified, then the status of the root
>         partitions will be shown."
> and the information about df -h option is missed,despite the fact that df -h option is implemented.
> Therefore,the expected message should be displayed like this:
> "-df [-h] [<path> ...]:    Shows the capacity, free and used space of the filesystem.
>         If the filesystem has multiple partitions, and no path to a
>         particular partition is specified, then the status of the root
>         partitions will be shown.
>           -h   Formats the sizes of files in a human-readable fashion
>                rather than a number of bytes."

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15412-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 16:42:30 2011
Return-Path: <common-issues-return-15412-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2A994420E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 16:42:30 +0000 (UTC)
Received: (qmail 41044 invoked by uid 500); 25 May 2011 16:42:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40795 invoked by uid 500); 25 May 2011 16:42:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40768 invoked by uid 99); 25 May 2011 16:42:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 16:42:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 16:42:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8DBDFDEE9D
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 16:41:47 +0000 (UTC)
Date: Wed, 25 May 2011 16:41:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <104532185.42652.1306341707577.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1659614701.18791.1297804257612.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7146) RPC server leaks file descriptors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039193#comment-13039193 ] 

Hadoop QA commented on HADOOP-7146:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480424/hadoop-7146.txt
  against trunk revision 1127215.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/525//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/525//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/525//console

This message is automatically generated.

> RPC server leaks file descriptors
> ---------------------------------
>
>                 Key: HADOOP-7146
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7146
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: fd-leak.txt, hadoop-7146.txt, hadoop-7146.txt, hadoop-7146.txt
>
>
> Both the Listener and Responder thread call Selector.open but don't have a matching .close(). This causes a leak of anonymous pipes. Not a big deal because people rarely close and re-open servers, but worth fixing.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15413-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 16:51:29 2011
Return-Path: <common-issues-return-15413-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0F5A6491C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 16:51:29 +0000 (UTC)
Received: (qmail 50425 invoked by uid 500); 25 May 2011 16:51:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50383 invoked by uid 500); 25 May 2011 16:51:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50370 invoked by uid 99); 25 May 2011 16:51:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 16:51:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 16:51:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DEF76DE28C
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 16:50:47 +0000 (UTC)
Date: Wed, 25 May 2011 16:50:47 +0000 (UTC)
From: "Robert Joseph Evans (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1662908194.42684.1306342247909.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Joseph Evans reassigned HADOOP-7144:
-------------------------------------------

    Assignee: Robert Joseph Evans  (was: Luke Lu)

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Robert Joseph Evans
>              Labels: jmx
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7411-trunk-alpha.patch
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15414-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 16:57:29 2011
Return-Path: <common-issues-return-15414-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1771A4C80
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 16:57:29 +0000 (UTC)
Received: (qmail 59983 invoked by uid 500); 25 May 2011 16:57:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59946 invoked by uid 500); 25 May 2011 16:57:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59938 invoked by uid 99); 25 May 2011 16:57:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 16:57:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 16:57:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B5EDCDE4E9
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 16:56:47 +0000 (UTC)
Date: Wed, 25 May 2011 16:56:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <108001323.42703.1306342607741.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1164170325.41524.1306304627421.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7329) incomplete help message  is
 displayed for df -h option
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039203#comment-13039203 ] 

Hudson commented on HADOOP-7329:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #618 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/618/])
    HADOOP-7329. Improve help message for "df" to include "-h" flag. Contributed by Xie Xianshan.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1127576
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/FsUsage.java


> incomplete help message  is displayed for df -h option
> ------------------------------------------------------
>
>                 Key: HADOOP-7329
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7329
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Assignee: XieXianshan
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7329
>
>
> The help message for the command "hdfs dfs -help df" is displayed like this:
> "-df [<path> ...]:    Shows the capacity, free and used space of the filesystem.
>         If the filesystem has multiple partitions, and no path to a
>         particular partition is specified, then the status of the root
>         partitions will be shown."
> and the information about df -h option is missed,despite the fact that df -h option is implemented.
> Therefore,the expected message should be displayed like this:
> "-df [-h] [<path> ...]:    Shows the capacity, free and used space of the filesystem.
>         If the filesystem has multiple partitions, and no path to a
>         particular partition is specified, then the status of the root
>         partitions will be shown.
>           -h   Formats the sizes of files in a human-readable fashion
>                rather than a number of bytes."

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15415-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 17:30:31 2011
Return-Path: <common-issues-return-15415-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4F3B045E0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 17:30:31 +0000 (UTC)
Received: (qmail 15997 invoked by uid 500); 25 May 2011 17:30:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15949 invoked by uid 500); 25 May 2011 17:30:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15941 invoked by uid 99); 25 May 2011 17:30:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 17:30:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 17:30:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D93EEDE1FF
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 17:29:47 +0000 (UTC)
Date: Wed, 25 May 2011 17:29:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <967450658.42792.1306344587885.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1828723839.36128.1306162247409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7320) Refactor FsShell's copy & move
 commands
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039224#comment-13039224 ] 

Todd Lipcon commented on HADOOP-7320:
-------------------------------------

+1. Hudson doesn't know how to deal with the renamed file in the test-patch process, but I re-ran tests locally.

> Refactor FsShell's copy & move commands
> ---------------------------------------
>
>                 Key: HADOOP-7320
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7320
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7320-2.patch, HADOOP-7320-3.patch, HADOOP-7320-rename.patch, HADOOP-7320.patch
>
>
> Need to refactor the move and copy commands to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15416-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 17:30:33 2011
Return-Path: <common-issues-return-15416-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0A6484610
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 17:30:33 +0000 (UTC)
Received: (qmail 16284 invoked by uid 500); 25 May 2011 17:30:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16196 invoked by uid 500); 25 May 2011 17:30:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16094 invoked by uid 99); 25 May 2011 17:30:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 17:30:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 17:30:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 159E4DE208
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 17:29:48 +0000 (UTC)
Date: Wed, 25 May 2011 17:29:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <110907153.42798.1306344588085.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1828723839.36128.1306162247409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7320) Refactor FsShell's copy & move
 commands
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7320:
--------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks, Daryn!

> Refactor FsShell's copy & move commands
> ---------------------------------------
>
>                 Key: HADOOP-7320
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7320
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7320-2.patch, HADOOP-7320-3.patch, HADOOP-7320-rename.patch, HADOOP-7320.patch
>
>
> Need to refactor the move and copy commands to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15417-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 17:46:31 2011
Return-Path: <common-issues-return-15417-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EFBA2490B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 17:46:30 +0000 (UTC)
Received: (qmail 44403 invoked by uid 500); 25 May 2011 17:46:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44375 invoked by uid 500); 25 May 2011 17:46:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44367 invoked by uid 99); 25 May 2011 17:46:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 17:46:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 17:46:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9AC2CDE7C3
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 17:45:47 +0000 (UTC)
Date: Wed, 25 May 2011 17:45:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <900695874.42831.1306345547630.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1828723839.36128.1306162247409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7320) Refactor FsShell's copy & move
 commands
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039232#comment-13039232 ] 

Hudson commented on HADOOP-7320:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #619 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/619/])
    HADOOP-7320. Refactor the copy and move commands to conform to new FsCommand class. Contributed by Daryn Sharp.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1127591
Files : 
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/FsCommand.java
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/PathData.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/cli/testConf.xml
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/CopyCommands.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/CommandWithDestination.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/MoveCommands.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/PathExceptions.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Ls.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Copy.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/FsShell.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/TestFsShellReturnCode.java


> Refactor FsShell's copy & move commands
> ---------------------------------------
>
>                 Key: HADOOP-7320
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7320
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7320-2.patch, HADOOP-7320-3.patch, HADOOP-7320-rename.patch, HADOOP-7320.patch
>
>
> Need to refactor the move and copy commands to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15418-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 18:02:31 2011
Return-Path: <common-issues-return-15418-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 06F31464B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 18:02:31 +0000 (UTC)
Received: (qmail 75152 invoked by uid 500); 25 May 2011 18:02:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75112 invoked by uid 500); 25 May 2011 18:02:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75104 invoked by uid 99); 25 May 2011 18:02:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 18:02:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 18:02:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CF4ADDEEF2
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 18:01:49 +0000 (UTC)
Date: Wed, 25 May 2011 18:01:49 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <123741536.42906.1306346509845.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039245#comment-13039245 ] 

Owen O'Malley commented on HADOOP-6255:
---------------------------------------

The hadoop-setup-single-node script is creating the config files in the cwd instead of the config directory.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-6255-branch-0.20-security-10.patch, HADOOP-6255-branch-0.20-security-11.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk-4.patch, HADOOP-6255-common-trunk-5.patch, HADOOP-6255-common-trunk-6.patch, HADOOP-6255-common-trunk-7.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-6255-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15419-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 19:26:30 2011
Return-Path: <common-issues-return-15419-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9E00947DB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 19:26:30 +0000 (UTC)
Received: (qmail 97103 invoked by uid 500); 25 May 2011 19:26:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97075 invoked by uid 500); 25 May 2011 19:26:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97067 invoked by uid 99); 25 May 2011 19:26:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 19:26:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 19:26:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5E750DEFE4
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 19:25:47 +0000 (UTC)
Date: Wed, 25 May 2011 19:25:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1531712102.43070.1306351547383.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7284:
---------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

Committed the patch

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15420-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 19:46:29 2011
Return-Path: <common-issues-return-15420-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 02ECB4B68
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 19:46:29 +0000 (UTC)
Received: (qmail 24112 invoked by uid 500); 25 May 2011 19:46:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24001 invoked by uid 500); 25 May 2011 19:46:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23993 invoked by uid 99); 25 May 2011 19:46:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 19:46:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 19:46:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 90C79DE6A8
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 19:45:47 +0000 (UTC)
Date: Wed, 25 May 2011 19:45:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <493331462.43094.1306352747589.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039302#comment-13039302 ] 

Hudson commented on HADOOP-7284:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #620 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/620/])
    HADOOP-7284 Trash and shell's rm does not work for viewfs (Sanjay Radia)

sradia : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1127642
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/viewfs/ChRootedFileSystem.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Delete.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/ViewFileSystemTestSetup.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/TestTrash.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/Trash.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/TestChRootedFs.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/TestChRootedFileSystem.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/TestViewFsTrash.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/TestFSMainOperationsLocalFileSystem.java


> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15421-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 20:09:30 2011
Return-Path: <common-issues-return-15421-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A988744DC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 20:09:30 +0000 (UTC)
Received: (qmail 59406 invoked by uid 500); 25 May 2011 20:09:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59368 invoked by uid 500); 25 May 2011 20:09:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59360 invoked by uid 99); 25 May 2011 20:09:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 20:09:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 20:09:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 775F5DEE7C
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 20:08:47 +0000 (UTC)
Date: Wed, 25 May 2011 20:08:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <840909694.43123.1306354127485.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039310#comment-13039310 ] 

Todd Lipcon commented on HADOOP-7284:
-------------------------------------

Was the latest revision reviewed? There are lots of diffs between trash3.patch which I had +1ed and trash7.patch which was committed (eg the use of "/homes" as a default home directory, which is very Yahoo specific)

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15422-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 20:44:28 2011
Return-Path: <common-issues-return-15422-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E45FC487D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 20:44:28 +0000 (UTC)
Received: (qmail 6490 invoked by uid 500); 25 May 2011 20:44:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6462 invoked by uid 500); 25 May 2011 20:44:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6454 invoked by uid 99); 25 May 2011 20:44:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 20:44:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 20:44:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9079BDE98D
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 20:43:47 +0000 (UTC)
Date: Wed, 25 May 2011 20:43:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <145488850.43184.1306356227588.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039321#comment-13039321 ] 

Todd Lipcon commented on HADOOP-7284:
-------------------------------------

This patch seems to also have broken the HDFS build. I am going to temporarily revert it.

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15423-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 20:46:30 2011
Return-Path: <common-issues-return-15423-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7B74643C4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 20:46:30 +0000 (UTC)
Received: (qmail 9431 invoked by uid 500); 25 May 2011 20:46:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9401 invoked by uid 500); 25 May 2011 20:46:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9376 invoked by uid 99); 25 May 2011 20:46:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 20:46:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 20:46:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 81426DEA7A
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 20:45:47 +0000 (UTC)
Date: Wed, 25 May 2011 20:45:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1753848736.43192.1306356347526.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Reopened] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon reopened HADOOP-7284:
---------------------------------


> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15424-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 21:06:28 2011
Return-Path: <common-issues-return-15424-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D17A44904
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 21:06:28 +0000 (UTC)
Received: (qmail 39753 invoked by uid 500); 25 May 2011 21:06:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39715 invoked by uid 500); 25 May 2011 21:06:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39707 invoked by uid 99); 25 May 2011 21:06:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 21:06:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 21:06:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A0236DE334
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 21:05:47 +0000 (UTC)
Date: Wed, 25 May 2011 21:05:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1063645027.43235.1306357547652.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039333#comment-13039333 ] 

Hudson commented on HADOOP-7284:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #621 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/621/])
    Revert HADOOP-7284 (r1127642) since it broke the HDFS build.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1127679
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/viewfs/ChRootedFileSystem.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Delete.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/ViewFileSystemTestSetup.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/TestTrash.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/Trash.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/TestChRootedFs.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/TestChRootedFileSystem.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/TestViewFsTrash.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/TestFSMainOperationsLocalFileSystem.java


> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15425-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 21:26:28 2011
Return-Path: <common-issues-return-15425-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D7D394874
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 21:26:28 +0000 (UTC)
Received: (qmail 69292 invoked by uid 500); 25 May 2011 21:26:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69256 invoked by uid 500); 25 May 2011 21:26:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69247 invoked by uid 99); 25 May 2011 21:26:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 21:26:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 21:26:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B38CDDEC46
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 21:25:47 +0000 (UTC)
Date: Wed, 25 May 2011 21:25:47 +0000 (UTC)
From: "Robert Joseph Evans (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <942611595.43291.1306358747732.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Joseph Evans updated HADOOP-7144:
----------------------------------------

    Status: Patch Available  (was: Open)

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Robert Joseph Evans
>              Labels: jmx
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7411-0.20.20X-V1.patch, HADOOP-7411-trunk-V1.patch, HADOOP-7411-trunk-alpha.patch, jmx.json
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15426-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 21:26:31 2011
Return-Path: <common-issues-return-15426-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2DB844885
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 21:26:31 +0000 (UTC)
Received: (qmail 69558 invoked by uid 500); 25 May 2011 21:26:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69525 invoked by uid 500); 25 May 2011 21:26:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69512 invoked by uid 99); 25 May 2011 21:26:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 21:26:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 21:26:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 711DDDEC3B
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 21:25:47 +0000 (UTC)
Date: Wed, 25 May 2011 21:25:47 +0000 (UTC)
From: "Robert Joseph Evans (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1356385802.43283.1306358747460.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Joseph Evans updated HADOOP-7144:
----------------------------------------

    Attachment: jmx.json
                HADOOP-7411-trunk-V1.patch
                HADOOP-7411-0.20.20X-V1.patch

This is version 1 of the patch.  It includes javadocs, tests, and the servlet.

The servlet is placed under /jmx and returns data in a JSON format.  Example output is attached too.

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Robert Joseph Evans
>              Labels: jmx
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7411-0.20.20X-V1.patch, HADOOP-7411-trunk-V1.patch, HADOOP-7411-trunk-alpha.patch, jmx.json
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15427-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 21:58:30 2011
Return-Path: <common-issues-return-15427-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AA884499F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 21:58:30 +0000 (UTC)
Received: (qmail 96786 invoked by uid 500); 25 May 2011 21:58:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96756 invoked by uid 500); 25 May 2011 21:58:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96748 invoked by uid 99); 25 May 2011 21:58:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 21:58:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 21:58:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5C739DF655
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 21:57:47 +0000 (UTC)
Date: Wed, 25 May 2011 21:57:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <111266951.43369.1306360667375.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1067116917.37592.1306187867429.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7322) Adding a util method in FileUtil for
 directory listing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matt Foley updated HADOOP-7322:
-------------------------------

      Resolution: Fixed
    Release Note: Use of this new utility method avoids null result from File.listFiles(), and consequent NPEs.
          Status: Resolved  (was: Patch Available)

+1.  Committed to trunk.  Thanks, Bharath!

> Adding a util method in FileUtil for directory listing
> ------------------------------------------------------
>
>                 Key: HADOOP-7322
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7322
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7322-1.patch, HADOOP-7322-2.patch, HADOOP-7322-3.patch
>
>
> While testing Disk Fail Inplace, we encountered lots of NPE from Dir.listFiles API. This API can return null when Dir is not directory or disk is bad. I am proposing to have a File Util which can be used consistently across to deal with disk issues. This util api will do the following:
> 1. When error happens it will throw IOException
> 2. Else it will return empty list or list of files.
> Signature:
> File[] FileUtil.listFiles(File dir) throws IOException {}
> This way we no need to write wrapper code every where. Also, API is consistent with the signature.
>  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15428-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 22:06:30 2011
Return-Path: <common-issues-return-15428-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ECE53405A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 22:06:29 +0000 (UTC)
Received: (qmail 3342 invoked by uid 500); 25 May 2011 22:06:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3303 invoked by uid 500); 25 May 2011 22:06:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3292 invoked by uid 99); 25 May 2011 22:06:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 22:06:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 22:06:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 89842DF893
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 22:05:48 +0000 (UTC)
Date: Wed, 25 May 2011 22:05:48 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <123147266.43383.1306361148560.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6255?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-6255:
------------------------------

    Attachment: HADOOP-6255-common-trunk-8.patch

hadoop-setup-single-node.sh utilize hadoop-setup-conf.sh to generate config=
uration.  haddop-setup-conf.sh is a utility for admin to generate config in=
 cwd, and admin can push out config files to remote nodes with admin's own =
utilities like rsync or scp.  However, the single-node setup was meant to s=
etup automatically with minimal config.  It is nicer to have config file ge=
nerated to HADOOP_CONF_DIR without storing a copy in cwd when hadoop-setup-=
single-node.sh invokes hadoop-setup-conf.sh. =20

The patch is revised to avoid writing to cwd, if automatic setup is choosen=
.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-625=
5-branch-0.20-security-10.patch, HADOOP-6255-branch-0.20-security-11.patch,=
 HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security=
-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20=
-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-br=
anch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOO=
P-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-security.patch=
, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-2.patch, HADOO=
P-6255-common-trunk-4.patch, HADOOP-6255-common-trunk-5.patch, HADOOP-6255-=
common-trunk-6.patch, HADOOP-6255-common-trunk-7.patch, HADOOP-6255-common-=
trunk-8.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.pat=
ch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-=
6255-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patc=
h, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15429-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 22:10:30 2011
Return-Path: <common-issues-return-15429-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8EE4A40DE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 22:10:30 +0000 (UTC)
Received: (qmail 6248 invoked by uid 500); 25 May 2011 22:10:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6198 invoked by uid 500); 25 May 2011 22:10:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6190 invoked by uid 99); 25 May 2011 22:10:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 22:10:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 22:10:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5FF5FDF9D6
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 22:09:47 +0000 (UTC)
Date: Wed, 25 May 2011 22:09:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1937737372.43422.1306361387389.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7330) The metrics source mbean
 implementation should return the attribute value instead of the object
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

The metrics source mbean implementation should return the attribute value instead of the object
-----------------------------------------------------------------------------------------------

                 Key: HADOOP-7330
                 URL: https://issues.apache.org/jira/browse/HADOOP-7330
             Project: Hadoop Common
          Issue Type: Bug
          Components: metrics
    Affects Versions: 0.20.203.0
            Reporter: Luke Lu
            Assignee: Luke Lu
             Fix For: 0.20.203.1


The MetricsSourceAdapter#getAttribute in 0.20.203 is returning the attribute object instead of the value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15430-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 22:12:29 2011
Return-Path: <common-issues-return-15430-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 851054179
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 22:12:29 +0000 (UTC)
Received: (qmail 10755 invoked by uid 500); 25 May 2011 22:12:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10636 invoked by uid 500); 25 May 2011 22:12:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10628 invoked by uid 99); 25 May 2011 22:12:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 22:12:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 22:12:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 36B4BDFAE6
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 22:11:48 +0000 (UTC)
Date: Wed, 25 May 2011 22:11:48 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1058945627.43443.1306361508220.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039372#comment-13039372 ] 

Luke Lu commented on HADOOP-7144:
---------------------------------

+1, lgtm, let's see hudson/jenkins' +1. Thanks for the json output (looks like it's 203?) as well, I spotted a bug (HADOOP-7330) the metrics system JMX code that I thought I fixed but didn't go into 203.

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Robert Joseph Evans
>              Labels: jmx
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7411-0.20.20X-V1.patch, HADOOP-7411-trunk-V1.patch, HADOOP-7411-trunk-alpha.patch, jmx.json
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15431-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 22:16:29 2011
Return-Path: <common-issues-return-15431-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0E0A646B4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 22:16:29 +0000 (UTC)
Received: (qmail 16203 invoked by uid 500); 25 May 2011 22:16:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16175 invoked by uid 500); 25 May 2011 22:16:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16165 invoked by uid 99); 25 May 2011 22:16:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 22:16:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 22:16:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 92024DFC19
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 22:15:47 +0000 (UTC)
Date: Wed, 25 May 2011 22:15:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1219740832.43458.1306361747594.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1067116917.37592.1306187867429.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7322) Adding a util method in FileUtil
 for directory listing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039376#comment-13039376 ] 

Hudson commented on HADOOP-7322:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #622 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/622/])
    HADOOP-7322. Adding a util method in FileUtil for directory listing, avoid NPEs on File.listFiles(). Contributed by Bharath Mundlapudi.

mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1127697
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/RawLocalFileSystem.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/FileUtil.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/TestFileUtil.java


> Adding a util method in FileUtil for directory listing
> ------------------------------------------------------
>
>                 Key: HADOOP-7322
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7322
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7322-1.patch, HADOOP-7322-2.patch, HADOOP-7322-3.patch
>
>
> While testing Disk Fail Inplace, we encountered lots of NPE from Dir.listFiles API. This API can return null when Dir is not directory or disk is bad. I am proposing to have a File Util which can be used consistently across to deal with disk issues. This util api will do the following:
> 1. When error happens it will throw IOException
> 2. Else it will return empty list or list of files.
> Signature:
> File[] FileUtil.listFiles(File dir) throws IOException {}
> This way we no need to write wrapper code every where. Also, API is consistent with the signature.
>  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15432-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 22:18:29 2011
Return-Path: <common-issues-return-15432-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E3CBF4068
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 22:18:28 +0000 (UTC)
Received: (qmail 19263 invoked by uid 500); 25 May 2011 22:18:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19087 invoked by uid 500); 25 May 2011 22:18:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19079 invoked by uid 99); 25 May 2011 22:18:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 22:18:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 22:18:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A4251DFCA3
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 22:17:47 +0000 (UTC)
Date: Wed, 25 May 2011 22:17:47 +0000 (UTC)
From: "Robert Joseph Evans (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1158450599.43469.1306361867668.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039380#comment-13039380 ] 

Robert Joseph Evans commented on HADOOP-7144:
---------------------------------------------

The JSON is from a task tracker on 20 security trunk, with the name of the box replaced with foo

http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security
Revision: 1127672

It is targeted for 0.20.205

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Robert Joseph Evans
>              Labels: jmx
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7411-0.20.20X-V1.patch, HADOOP-7411-trunk-V1.patch, HADOOP-7411-trunk-alpha.patch, jmx.json
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15433-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 22:20:30 2011
Return-Path: <common-issues-return-15433-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C2CD348A4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 22:20:30 +0000 (UTC)
Received: (qmail 22294 invoked by uid 500); 25 May 2011 22:20:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22269 invoked by uid 500); 25 May 2011 22:20:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22257 invoked by uid 99); 25 May 2011 22:20:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 22:20:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 22:20:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5DF28DFD4A
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 22:19:47 +0000 (UTC)
Date: Wed, 25 May 2011 22:19:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <905561758.43479.1306361987381.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1937737372.43422.1306361387389.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7330) The metrics source mbean
 implementation should return the attribute value instead of the object
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-7330:
----------------------------

    Attachment: hadoop-7330-jmx-v1.patch

This is only for the branch-0.20-security* branches. yahoo-merge and trunk do not have this bug.

> The metrics source mbean implementation should return the attribute value instead of the object
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7330
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7330
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.203.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.20.203.1
>
>         Attachments: hadoop-7330-jmx-v1.patch
>
>
> The MetricsSourceAdapter#getAttribute in 0.20.203 is returning the attribute object instead of the value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15434-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 22:22:28 2011
Return-Path: <common-issues-return-15434-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CAF2E4914
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 22:22:28 +0000 (UTC)
Received: (qmail 28284 invoked by uid 500); 25 May 2011 22:22:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28234 invoked by uid 500); 25 May 2011 22:22:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28223 invoked by uid 99); 25 May 2011 22:22:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 22:22:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 22:22:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A1840DFE3B
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 22:21:47 +0000 (UTC)
Date: Wed, 25 May 2011 22:21:47 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1252934613.43491.1306362107658.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <654871262.37888.1306191347393.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7323) Add capability to resolve
 compression codec based on codec name
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039384#comment-13039384 ] 

Tom White commented on HADOOP-7323:
-----------------------------------

+1

> Add capability to resolve compression codec based on codec name
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7323
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7323
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7323.patch, HADOOP-7323b.patch
>
>
> When setting up a compression codec in an MR job the full class name of the codec must be used.
> To ease usability, compression codecs should be resolved by their codec name (ie 'gzip', 'deflate', 'zlib', 'bzip2') instead their full codec class name.
> Besides easy of use for Hadoop users who would use the codec alias instead the full codec class name, it could simplify how HBase resolves loads the codecs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15435-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 22:24:30 2011
Return-Path: <common-issues-return-15435-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BE6A1493F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 22:24:30 +0000 (UTC)
Received: (qmail 29585 invoked by uid 500); 25 May 2011 22:24:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29557 invoked by uid 500); 25 May 2011 22:24:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29549 invoked by uid 99); 25 May 2011 22:24:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 22:24:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 22:24:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5E03CDFE99
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 22:23:47 +0000 (UTC)
Date: Wed, 25 May 2011 22:23:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1395794000.43493.1306362227381.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039385#comment-13039385 ] 

Luke Lu commented on HADOOP-7144:
---------------------------------

bq. The JSON is from a task tracker on 20 security trunk, with the name of the box replaced with foo

Thanks for the precise answer. I meant whether the output is from 0.20.20x.

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Robert Joseph Evans
>              Labels: jmx
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7411-0.20.20X-V1.patch, HADOOP-7411-trunk-V1.patch, HADOOP-7411-trunk-alpha.patch, jmx.json
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15436-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 22:49:30 2011
Return-Path: <common-issues-return-15436-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 104D74889
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 22:49:30 +0000 (UTC)
Received: (qmail 65190 invoked by uid 500); 25 May 2011 22:49:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65129 invoked by uid 500); 25 May 2011 22:49:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65012 invoked by uid 99); 25 May 2011 22:49:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 22:49:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 22:49:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AD572DF6A8
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 22:48:48 +0000 (UTC)
Date: Wed, 25 May 2011 22:48:48 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <886461764.43576.1306363728706.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6255?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-6255:
------------------------------

    Attachment: HADOOP-6255-common-trunk-9.patch

Enhancement to hadoop-setup-single-node.sh:

- Delete datanode directory, if namenode format is chosen to avoid incompat=
ible version of datanode directory left over.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-625=
5-branch-0.20-security-10.patch, HADOOP-6255-branch-0.20-security-11.patch,=
 HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security=
-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20=
-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-br=
anch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOO=
P-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-security.patch=
, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-2.patch, HADOO=
P-6255-common-trunk-4.patch, HADOOP-6255-common-trunk-5.patch, HADOOP-6255-=
common-trunk-6.patch, HADOOP-6255-common-trunk-7.patch, HADOOP-6255-common-=
trunk-8.patch, HADOOP-6255-common-trunk-9.patch, HADOOP-6255-common-trunk.p=
atch, HADOOP-6255-hdfs-trunk-1.patch, HADOOP-6255-hdfs-trunk.patch, HADOOP-=
6255-mapred-trunk-1.patch, HADOOP-6255-mapred-trunk-2.patch, HADOOP-6255-ma=
pred-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15437-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 23:39:28 2011
Return-Path: <common-issues-return-15437-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AC3D241B1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 23:39:28 +0000 (UTC)
Received: (qmail 12880 invoked by uid 500); 25 May 2011 23:39:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12856 invoked by uid 500); 25 May 2011 23:39:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12847 invoked by uid 99); 25 May 2011 23:39:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 23:39:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 23:39:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9016BDE3E5
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 23:38:47 +0000 (UTC)
Date: Wed, 25 May 2011 23:38:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <494045772.43671.1306366727586.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039420#comment-13039420 ] 

Sanjay Radia commented on HADOOP-7284:
--------------------------------------

Sorry for breaking the HDFS build - will check this.
Todd after you last review I merely fixed the bug when running the test on hudson
 -- the /homes is NOT a Yahoo thing but Linux default for Hudson machines.
 You had agreed that I should file a jira to specify how a home dir is
specified for the different implementations of file system on various platforms.  

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15438-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 23:39:30 2011
Return-Path: <common-issues-return-15438-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7798B41CE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 23:39:30 +0000 (UTC)
Received: (qmail 13543 invoked by uid 500); 25 May 2011 23:39:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13475 invoked by uid 500); 25 May 2011 23:39:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13295 invoked by uid 99); 25 May 2011 23:39:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 23:39:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 23:39:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E2EC7DE41C
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 23:38:48 +0000 (UTC)
Date: Wed, 25 May 2011 23:38:48 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1465370992.43706.1306366728926.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6255?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-6255:
------------------------------

    Attachment: HADOOP-6255-common-trunk-10.patch

Reduce the safemode threshold-pct to 1.0f, and safemode.extension to 3 for =
default template to improve initial setup user experience of using hadoop-s=
etup-single-node.sh.


> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-625=
5-branch-0.20-security-10.patch, HADOOP-6255-branch-0.20-security-11.patch,=
 HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security=
-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20=
-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-br=
anch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOO=
P-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-security.patch=
, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-10.patch, HADO=
OP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk-4.patch, HADOOP-6255=
-common-trunk-5.patch, HADOOP-6255-common-trunk-6.patch, HADOOP-6255-common=
-trunk-7.patch, HADOOP-6255-common-trunk-8.patch, HADOOP-6255-common-trunk-=
9.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch, HA=
DOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-6255-m=
apred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, dep=
loyment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15439-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Wed May 25 23:51:28 2011
Return-Path: <common-issues-return-15439-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A587049C5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Wed, 25 May 2011 23:51:28 +0000 (UTC)
Received: (qmail 26578 invoked by uid 500); 25 May 2011 23:51:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26537 invoked by uid 500); 25 May 2011 23:51:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26441 invoked by uid 99); 25 May 2011 23:51:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 23:51:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2011 23:51:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 757DDDE8FD
	for <common-issues@hadoop.apache.org>; Wed, 25 May 2011 23:50:47 +0000 (UTC)
Date: Wed, 25 May 2011 23:50:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <246694272.43748.1306367447477.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039431#comment-13039431 ] 

Todd Lipcon commented on HADOOP-7284:
-------------------------------------

bq. the /homes is NOT a Yahoo thing but Linux default for Hudson machines.

Only for the Hudson machines in *.yahoo.net ;-) I've never seen /homes/ anywhere else.


Also, I thought this test was supposed to set up a LocalFileSystem subclass (TestLFS) that responded with a directory inside build/ as a home dir?

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15440-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 00:01:40 2011
Return-Path: <common-issues-return-15440-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 84EC5461A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 00:01:40 +0000 (UTC)
Received: (qmail 32117 invoked by uid 500); 26 May 2011 00:01:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32081 invoked by uid 500); 26 May 2011 00:01:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32072 invoked by uid 99); 26 May 2011 00:01:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 00:01:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 00:01:39 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 56AADDEC40
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 00:00:59 +0000 (UTC)
Date: Thu, 26 May 2011 00:00:59 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <885656412.43773.1306368059351.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <889607684.27597.1305830207722.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7307) Remove Dr Who references
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039435#comment-13039435 ] 

Allen Wittenauer commented on HADOOP-7307:
------------------------------------------

BTW, since we're playing with 0.20.203 on our test box:

JT UI runs as hadoop.
NN UI appears to use dfs.web.ugi, as we're blowing errors in the NN log for the (unknown our systems) 'webuser' account.

Are we *sure* this isn't already fixed in trunk? 

> Remove Dr Who references
> ------------------------
>
>                 Key: HADOOP-7307
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7307
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>             Fix For: 0.22.0
>
>
> I've had feedback from a number of different operators that the "dr who" references that occasionally come out of Hadoop are annoying and confusing. We should replace them with more descriptive error messages, usernames, etc.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15441-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 00:07:30 2011
Return-Path: <common-issues-return-15441-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E5259434A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 00:07:30 +0000 (UTC)
Received: (qmail 36528 invoked by uid 500); 26 May 2011 00:07:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36494 invoked by uid 500); 26 May 2011 00:07:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36479 invoked by uid 99); 26 May 2011 00:07:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 00:07:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 00:07:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 79F76DEE01
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 00:06:47 +0000 (UTC)
Date: Thu, 26 May 2011 00:06:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1828440147.43779.1306368407496.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <889607684.27597.1305830207722.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7307) Remove Dr Who references
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039437#comment-13039437 ] 

Todd Lipcon commented on HADOOP-7307:
-------------------------------------

If you're running without security on, it continues to use dfs.web.ugi (default webuser). If you turn security on, it will use the new static user plugin, which defaults to DrWho. Makes sense, right?

> Remove Dr Who references
> ------------------------
>
>                 Key: HADOOP-7307
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7307
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>             Fix For: 0.22.0
>
>
> I've had feedback from a number of different operators that the "dr who" references that occasionally come out of Hadoop are annoying and confusing. We should replace them with more descriptive error messages, usernames, etc.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15442-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 00:09:30 2011
Return-Path: <common-issues-return-15442-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BBF4343B8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 00:09:30 +0000 (UTC)
Received: (qmail 39368 invoked by uid 500); 26 May 2011 00:09:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39334 invoked by uid 500); 26 May 2011 00:09:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39325 invoked by uid 99); 26 May 2011 00:09:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 00:09:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 00:09:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 84BC9DEEB1
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 00:08:47 +0000 (UTC)
Date: Thu, 26 May 2011 00:08:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <655348775.43797.1306368527540.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7326) TestTrash.testTrashEmptier()
 infinite loops if run on a home directory with stuff in .Trash
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039438#comment-13039438 ] 

Matt Foley commented on HADOOP-7326:
------------------------------------

In HADOOP-7284, Todd observes that TestTrash is "supposed to set up a LocalFileSystem subclass (TestLFS) that responded with a directory inside build/ as a home dir".  Seems like that would be a good way to get around this silliness with shared .Trash directory.  It looks like the current implementation is deficient.  I'll look into fixing it.

> TestTrash.testTrashEmptier() infinite loops if run on a home directory with stuff in .Trash
> -------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7326
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7326
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.22.0, 0.23.0
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>         Attachments: hadoop-7326-TestTrash-1.patch
>
>
> This bug was discovered while investigating HDFS-1967, intermittent failure of TestHDFSTrash, which calls TestTrash.  If the user id running the test has a ~/.Trash directory that already has 4 or more files in it, a loop in TestTrash.testTrashEmptier() will never terminate, because it is waiting for the count of objects to increase from zero to exactly three (and it creates one object right away).

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15443-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 00:11:30 2011
Return-Path: <common-issues-return-15443-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8A6034454
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 00:11:30 +0000 (UTC)
Received: (qmail 43530 invoked by uid 500); 26 May 2011 00:11:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43497 invoked by uid 500); 26 May 2011 00:11:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43489 invoked by uid 99); 26 May 2011 00:11:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 00:11:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 00:11:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 610B8DEF22
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 00:10:47 +0000 (UTC)
Date: Thu, 26 May 2011 00:10:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <909012996.43802.1306368647394.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039439#comment-13039439 ] 

Matt Foley commented on HADOOP-7284:
------------------------------------

In HADOOP-7326, we've noted that the TestLFS implementation seems deficient - but a really good idea if it worked!

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15444-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 00:13:29 2011
Return-Path: <common-issues-return-15444-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4CA9844E5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 00:13:29 +0000 (UTC)
Received: (qmail 45341 invoked by uid 500); 26 May 2011 00:13:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45279 invoked by uid 500); 26 May 2011 00:13:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45032 invoked by uid 99); 26 May 2011 00:13:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 00:13:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 00:13:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C539EDF03C
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 00:12:47 +0000 (UTC)
Date: Thu, 26 May 2011 00:12:47 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <982898781.43811.1306368767804.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <889607684.27597.1305830207722.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7307) Remove Dr Who references
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039440#comment-13039440 ] 

Allen Wittenauer commented on HADOOP-7307:
------------------------------------------

Ah ok.  So we have a plug-in that basically ignores an (already existing) config setting that is practically custom built for this purpose.

+1 to honor it in the plug-in.

The JIRA title/description should probably flushed out to specify this is the actual problem.

> Remove Dr Who references
> ------------------------
>
>                 Key: HADOOP-7307
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7307
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>             Fix For: 0.22.0
>
>
> I've had feedback from a number of different operators that the "dr who" references that occasionally come out of Hadoop are annoying and confusing. We should replace them with more descriptive error messages, usernames, etc.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15445-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 01:01:36 2011
Return-Path: <common-issues-return-15445-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D24CF463B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 01:01:36 +0000 (UTC)
Received: (qmail 79785 invoked by uid 500); 26 May 2011 01:01:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79753 invoked by uid 500); 26 May 2011 01:01:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79745 invoked by uid 99); 26 May 2011 01:01:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 01:01:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 01:01:35 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A9FE1DFC1F
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 01:00:55 +0000 (UTC)
Date: Thu, 26 May 2011 01:00:55 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1635619820.43903.1306371655692.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1659614701.18791.1297804257612.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7146) RPC server leaks file descriptors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039461#comment-13039461 ] 

Matt Foley commented on HADOOP-7146:
------------------------------------

+1.  Looks great!  Thanks, Todd.

> RPC server leaks file descriptors
> ---------------------------------
>
>                 Key: HADOOP-7146
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7146
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: fd-leak.txt, hadoop-7146.txt, hadoop-7146.txt, hadoop-7146.txt
>
>
> Both the Listener and Responder thread call Selector.open but don't have a matching .close(). This causes a leak of anonymous pipes. Not a big deal because people rarely close and re-open servers, but worth fixing.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15446-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 01:05:30 2011
Return-Path: <common-issues-return-15446-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D1897442E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 01:05:30 +0000 (UTC)
Received: (qmail 93095 invoked by uid 500); 26 May 2011 01:05:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93059 invoked by uid 500); 26 May 2011 01:05:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93051 invoked by uid 99); 26 May 2011 01:05:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 01:05:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 01:05:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 93B0CDFDB1
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 01:04:47 +0000 (UTC)
Date: Thu, 26 May 2011 01:04:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1158109903.43954.1306371887601.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039463#comment-13039463 ] 

Sanjay Radia commented on HADOOP-7284:
--------------------------------------

> Also, I thought this test was supposed to set up a LocalFileSystem subclass (TestLFS) that responded with a directory inside build/ as a home dir?

I don't recall saying that.
The homedir needs to be set for ViewFs anyway - my patch sets it up via set of rules. 
A jira is needed to discuss what is the correct way to set home dir: server side or config for  HDFS, config for viewfs or something more consistent.

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15447-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 01:19:28 2011
Return-Path: <common-issues-return-15447-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C769046D5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 01:19:28 +0000 (UTC)
Received: (qmail 2343 invoked by uid 500); 26 May 2011 01:19:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2312 invoked by uid 500); 26 May 2011 01:19:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2302 invoked by uid 99); 26 May 2011 01:19:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 01:19:28 +0000
X-ASF-Spam-Status: No, hits=-1998.0 required=5.0
	tests=ALL_TRUSTED,FB_GET_MEDS,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 01:19:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7A896DF0AD
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 01:18:47 +0000 (UTC)
Date: Thu, 26 May 2011 01:18:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <859937697.44003.1306372727498.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039466#comment-13039466 ] 

Todd Lipcon commented on HADOOP-7284:
-------------------------------------

bq. I don't recall saying that.

I understood this part of the patch to indicate that:

{code}
+  static class TestLFS extends LocalFileSystem {
+    Path home;
+    TestLFS() throws IOException {
+      this(new Path(FileSystemTestHelper.TEST_ROOT_DIR));
+    }
+    TestLFS(Path home) throws IOException {
+      super();
+      this.home = home;
+    }
+    public Path getHomeDirectory() {
+      return home;
+    }
+  }
{code}

bq. The homedir needs to be set for ViewFs anyway - my patch sets it up via set of rules. 

right -- but the inclusion of /homes/ and not /home is very Yahoo-centric. The standard linux/bsd heirarchy puts home dirs in /home, not /homes.

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15448-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 01:29:30 2011
Return-Path: <common-issues-return-15448-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AD2EC4C3E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 01:29:30 +0000 (UTC)
Received: (qmail 8546 invoked by uid 500); 26 May 2011 01:29:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8521 invoked by uid 500); 26 May 2011 01:29:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8513 invoked by uid 99); 26 May 2011 01:29:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 01:29:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 01:29:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7314CDF328
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 01:28:47 +0000 (UTC)
Date: Thu, 26 May 2011 01:28:47 +0000 (UTC)
From: "Matt Foley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <995648418.44024.1306373327468.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039470#comment-13039470 ] 

Matt Foley commented on HADOOP-7284:
------------------------------------

My comment re TestLFS was of course directed at the version in TestTrash, not the new TestViewFsTrash proposed in the patch, which I haven't reviewed.

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15449-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 01:41:30 2011
Return-Path: <common-issues-return-15449-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EE3EC42AE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 01:41:30 +0000 (UTC)
Received: (qmail 16848 invoked by uid 500); 26 May 2011 01:41:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16789 invoked by uid 500); 26 May 2011 01:41:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16489 invoked by uid 99); 26 May 2011 01:41:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 01:41:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 01:41:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 739F6DF57E
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 01:40:47 +0000 (UTC)
Date: Thu, 26 May 2011 01:40:47 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <504299282.44039.1306374047470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7331) Makes hadoop-daemon.sh to return 1
 if daemon processes did not get started
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Makes hadoop-daemon.sh to return 1 if daemon processes did not get started
--------------------------------------------------------------------------

                 Key: HADOOP-7331
                 URL: https://issues.apache.org/jira/browse/HADOOP-7331
             Project: Hadoop Common
          Issue Type: Improvement
          Components: scripts
    Affects Versions: 0.23.0
            Reporter: Tanping Wang
            Assignee: Tanping Wang
            Priority: Trivial
             Fix For: 0.23.0


Makes hadoop-daemon.sh to return 1 if daemon processes did not get started.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15450-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 01:49:29 2011
Return-Path: <common-issues-return-15450-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7415649D4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 01:49:29 +0000 (UTC)
Received: (qmail 20149 invoked by uid 500); 26 May 2011 01:49:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20079 invoked by uid 500); 26 May 2011 01:49:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20071 invoked by uid 99); 26 May 2011 01:49:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 01:49:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 01:49:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 28C18DF767
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 01:48:48 +0000 (UTC)
Date: Thu, 26 May 2011 01:48:48 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2053987489.44054.1306374528163.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039474#comment-13039474 ] 

Sanjay Radia commented on HADOOP-7284:
--------------------------------------

Todd, 
I originally copied  LFS from the base test but then realized that there is more fundamental issue with homedir. One needs to be able to configure home dir 
based on whatever your local convention is. (HDFS-593 identified this issue a long time ago and a patch was available but Doug felt it wasn't high priority).

Turns out if I had used LFS in this Jira,  it would have masked the real problem.
The  design, implementation of trash and its tests has many issues.
Also getting the test to work on Hudson is a valid use case (nothing to do with Yahoo).
We both agreed that a jira is needed to address the homedir issue properly.
If I remove /homes then it will not work on Hudson unless I mask it out with LFS. 
Please propose an alternate solution. I will also think of an alternate solution. 




> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15451-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 01:53:28 2011
Return-Path: <common-issues-return-15451-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 844194C94
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 01:53:28 +0000 (UTC)
Received: (qmail 25207 invoked by uid 500); 26 May 2011 01:53:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25167 invoked by uid 500); 26 May 2011 01:53:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25159 invoked by uid 99); 26 May 2011 01:53:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 01:53:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 01:53:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 688E4DF853
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 01:52:47 +0000 (UTC)
Date: Thu, 26 May 2011 01:52:47 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1540010814.44057.1306374767425.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <504299282.44039.1306374047470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7331) Makes hadoop-daemon.sh to return 1
 if daemon processes did not get started
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tanping Wang updated HADOOP-7331:
---------------------------------

    Attachment: HADOOP-7331.patch

> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7331
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7331
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7331.patch
>
>
> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15452-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 02:03:28 2011
Return-Path: <common-issues-return-15452-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 83E7868CB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 02:03:28 +0000 (UTC)
Received: (qmail 32776 invoked by uid 500); 26 May 2011 02:03:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32749 invoked by uid 500); 26 May 2011 02:03:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32741 invoked by uid 99); 26 May 2011 02:03:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 02:03:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 02:03:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 63F8DDFB49
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 02:02:47 +0000 (UTC)
Date: Thu, 26 May 2011 02:02:47 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1286645112.44096.1306375367406.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <504299282.44039.1306374047470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7331) Makes hadoop-daemon.sh to return 1
 if daemon processes did not get started
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tanping Wang updated HADOOP-7331:
---------------------------------

    Attachment: HADOOP-7331.patch

> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7331
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7331
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7331.patch
>
>
> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15453-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 02:03:31 2011
Return-Path: <common-issues-return-15453-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 52D8C6912
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 02:03:31 +0000 (UTC)
Received: (qmail 33048 invoked by uid 500); 26 May 2011 02:03:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33009 invoked by uid 500); 26 May 2011 02:03:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33001 invoked by uid 99); 26 May 2011 02:03:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 02:03:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 02:03:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5D7E3DFB48
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 02:02:47 +0000 (UTC)
Date: Thu, 26 May 2011 02:02:47 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <619393103.44095.1306375367379.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <504299282.44039.1306374047470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7331) Makes hadoop-daemon.sh to return 1
 if daemon processes did not get started
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tanping Wang updated HADOOP-7331:
---------------------------------

    Attachment:     (was: HADOOP-7331.patch)

> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7331
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7331
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7331.patch
>
>
> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15454-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 02:07:31 2011
Return-Path: <common-issues-return-15454-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 51D1C6E44
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 02:07:31 +0000 (UTC)
Received: (qmail 35675 invoked by uid 500); 26 May 2011 02:07:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35551 invoked by uid 500); 26 May 2011 02:07:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35492 invoked by uid 99); 26 May 2011 02:07:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 02:07:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 02:07:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0D9B1DFC50
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 02:06:48 +0000 (UTC)
Date: Thu, 26 May 2011 02:06:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <791472242.44117.1306375608052.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <504299282.44039.1306374047470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7331) Makes hadoop-daemon.sh to return 1
 if daemon processes did not get started
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039479#comment-13039479 ] 

Todd Lipcon commented on HADOOP-7331:
-------------------------------------

I think we need something a lot more involved to really do a useful job of this. The issue is that most of our daemons start and then exit a few seconds after startup (eg once the JVM has loaded and it has had a chance to do some basic initialization). For example, with an invalid config or bind address, hadoop-daemon.sh will still return 0 in this case.

Regarding the patch itself: no need to use wc -l on the output of {{ps}}. Instead, something like:
{code}
if ! ps -p $! > /dev/null ; then
  exit 1
fi
{code}
since ps has a non-zero exit code if the process is running.

> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7331
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7331
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7331.patch
>
>
> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15455-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 02:13:31 2011
Return-Path: <common-issues-return-15455-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 79AF26F30
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 02:13:31 +0000 (UTC)
Received: (qmail 42995 invoked by uid 500); 26 May 2011 02:13:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42957 invoked by uid 500); 26 May 2011 02:13:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42948 invoked by uid 99); 26 May 2011 02:13:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 02:13:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 02:13:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C7C1FDFF0A
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 02:12:47 +0000 (UTC)
Date: Thu, 26 May 2011 02:12:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1096736268.44158.1306375967814.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039482#comment-13039482 ] 

Todd Lipcon commented on HADOOP-7284:
-------------------------------------

I agree it's nice to work on Hudson -- but I want to confine that to the test code. Unfortunately as is, it will work on the Apache Hudson but maybe not on other people's Hudsons or local boxes, right?

Would the client-side conf be a lot of work? It seems like it would avoid all of this complication, simplify the tests, and let it pass on everyone's boxes?

Maybe we could look at the $HOME environment variable to figure out the path?

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15456-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 02:41:32 2011
Return-Path: <common-issues-return-15456-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EB6736ADF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 02:41:31 +0000 (UTC)
Received: (qmail 59445 invoked by uid 500); 26 May 2011 02:41:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59385 invoked by uid 500); 26 May 2011 02:41:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59369 invoked by uid 99); 26 May 2011 02:41:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 02:41:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 02:41:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5B365DF623
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 02:40:47 +0000 (UTC)
Date: Thu, 26 May 2011 02:40:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1847550073.44182.1306377647370.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039489#comment-13039489 ] 

Sanjay Radia commented on HADOOP-7284:
--------------------------------------

The reason I didn't do the config option is that the user will have to write code to
determine what value to set based on the environment in which he runs. (I wrote the config option code but then realized that it was not the right solution and didn't want to make a new config parameter public and then be forced to pull it out later.)

The $HOME may not work since the $HOME on my desktop is not the right one for HDFS to which  I am submitting the job. My admin has set up a viewfs config that matches the
hdfs cluster in which my job will run. The $HOME of the submitting machine is not
relevant.

This is a fairly subtle problem - I am not sure which solution is right  otherwise I would have put it in. The solution in this patch is:  the viewfilesystem impl
has a set of heuristics that make it work for some common conventions: /user /Users, /home, /homes. It is an okay solution but clearner not general enough to work in all environments. In the short term it will work till we figure out a cleaner solution. 

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15457-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 05:16:30 2011
Return-Path: <common-issues-return-15457-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B1E7F6BE3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 05:16:30 +0000 (UTC)
Received: (qmail 46149 invoked by uid 500); 26 May 2011 05:16:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46126 invoked by uid 500); 26 May 2011 05:16:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46117 invoked by uid 99); 26 May 2011 05:16:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 05:16:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 05:16:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5E435DF472
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 05:15:47 +0000 (UTC)
Date: Thu, 26 May 2011 05:15:47 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1718801008.44308.1306386947382.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <504299282.44039.1306374047470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7331) Makes hadoop-daemon.sh to return 1
 if daemon processes did not get started
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tanping Wang updated HADOOP-7331:
---------------------------------

    Attachment: HDFS-7331.2.patch

Thanks for the comment!  I agree that we need a more complex mechanism in order for hadoop-daemon.sh process to really know that daemons are not able to start due to various problems that may happen.  Suppose now we want to restrict our problem set to deal with the issues like invalid config or unable to bind address in which cases the daemon should die rather quickly after starting.  What about having a simple solution for now: we put the process to sleep for a few seconds before checking pid?  (Made changes based the feedback.)

> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7331
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7331
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7331.patch, HDFS-7331.2.patch
>
>
> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15458-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 05:18:31 2011
Return-Path: <common-issues-return-15458-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5A8A565BB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 05:18:31 +0000 (UTC)
Received: (qmail 49865 invoked by uid 500); 26 May 2011 05:18:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49834 invoked by uid 500); 26 May 2011 05:18:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49813 invoked by uid 99); 26 May 2011 05:18:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 05:18:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 05:18:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C9423DF542
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 05:17:47 +0000 (UTC)
Date: Thu, 26 May 2011 05:17:47 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <923292489.44316.1306387067820.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <504299282.44039.1306374047470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7331) Makes hadoop-daemon.sh to return 1
 if daemon processes did not get started
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tanping Wang updated HADOOP-7331:
---------------------------------

    Attachment: HADOOP-7331.2.patch

> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7331
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7331
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7331.2.patch, HADOOP-7331.patch
>
>
> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15459-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 05:18:31 2011
Return-Path: <common-issues-return-15459-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B405165D1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 05:18:31 +0000 (UTC)
Received: (qmail 50119 invoked by uid 500); 26 May 2011 05:18:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49836 invoked by uid 500); 26 May 2011 05:18:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49812 invoked by uid 99); 26 May 2011 05:18:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 05:18:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 05:18:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AF1E5DF540
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 05:17:47 +0000 (UTC)
Date: Thu, 26 May 2011 05:17:47 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <967663515.44314.1306387067713.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <504299282.44039.1306374047470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7331) Makes hadoop-daemon.sh to return 1
 if daemon processes did not get started
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tanping Wang updated HADOOP-7331:
---------------------------------

    Attachment:     (was: HDFS-7331.2.patch)

> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7331
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7331
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7331.2.patch, HADOOP-7331.patch
>
>
> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15460-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 05:24:30 2011
Return-Path: <common-issues-return-15460-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BE3826D31
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 05:24:30 +0000 (UTC)
Received: (qmail 56629 invoked by uid 500); 26 May 2011 05:24:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56602 invoked by uid 500); 26 May 2011 05:24:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56594 invoked by uid 99); 26 May 2011 05:24:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 05:24:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 05:24:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9400DDF72F
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 05:23:47 +0000 (UTC)
Date: Thu, 26 May 2011 05:23:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <954130393.44326.1306387427602.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1008788632.26385.1305807047638.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7305) Eclipse project files are
 incomplete
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039513#comment-13039513 ] 

Eli Collins commented on HADOOP-7305:
-------------------------------------

Hey Niels - can you chime in wrt Daryn's request?

> Eclipse project files are incomplete
> ------------------------------------
>
>                 Key: HADOOP-7305
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7305
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Niels Basjes
>            Assignee: Niels Basjes
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7305-2011-05-19.patch
>
>
> After a fresh checkout of hadoop-common I do 'ant compile eclipse'.
> I open eclipse, set ANT_HOME and build the project. 
> At that point the following error appears:
> {quote}
> The type com.sun.javadoc.RootDoc cannot be resolved. It is indirectly referenced from required .class files	ExcludePrivateAnnotationsJDiffDoclet.java	/common/src/java/org/apache/hadoop/classification/tools	line 1	Java Problem
> {quote}
> The solution is to add the "tools.jar" from the JDK to the buildpath/classpath.
> This should be fixed in the build.xml.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15461-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 05:30:29 2011
Return-Path: <common-issues-return-15461-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C6C9860AF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 05:30:29 +0000 (UTC)
Received: (qmail 63394 invoked by uid 500); 26 May 2011 05:30:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63355 invoked by uid 500); 26 May 2011 05:30:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63344 invoked by uid 99); 26 May 2011 05:30:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 05:30:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 05:30:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7A923DF8E2
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 05:29:47 +0000 (UTC)
Date: Thu, 26 May 2011 05:29:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1692434703.44338.1306387787498.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <504299282.44039.1306374047470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7331) Makes hadoop-daemon.sh to return 1
 if daemon processes did not get started
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039515#comment-13039515 ] 

Todd Lipcon commented on HADOOP-7331:
-------------------------------------

Yep, I think sleeping is a good idea.

Assuming this has been manually tested, +1 for this simple fix before doing something more complicated.

> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7331
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7331
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7331.2.patch, HADOOP-7331.patch
>
>
> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15462-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 07:52:36 2011
Return-Path: <common-issues-return-15462-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B103764D0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 07:52:36 +0000 (UTC)
Received: (qmail 34177 invoked by uid 500); 26 May 2011 07:52:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33885 invoked by uid 500); 26 May 2011 07:52:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33854 invoked by uid 99); 26 May 2011 07:52:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 07:52:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 07:52:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 64966DF2BD
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 07:51:47 +0000 (UTC)
Date: Thu, 26 May 2011 07:51:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <503549136.44645.1306396307408.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7332) Deadlock in IPC
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Deadlock in IPC
---------------

                 Key: HADOOP-7332
                 URL: https://issues.apache.org/jira/browse/HADOOP-7332
             Project: Hadoop Common
          Issue Type: Bug
          Components: ipc
    Affects Versions: 0.22.0
            Reporter: Todd Lipcon
             Fix For: 0.22.0


Saw this during a run of TestIPC on 0.22 branch:

    [junit] Java stack information for the threads listed above:
    [junit] ===================================================
    [junit] "IPC Client (47) connection to /0:0:0:0:0:0:0:0:48853 from an unknown user":
    [junit]     at org.apache.hadoop.ipc.Client$ParallelResults.callComplete(Client.java:879)
    [junit]     - waiting to lock <0x00000000f599ef88> (a org.apache.hadoop.ipc.Client$ParallelResults)
    [junit]     at org.apache.hadoop.ipc.Client$ParallelCall.callComplete(Client.java:862)
    [junit]     at org.apache.hadoop.ipc.Client$Call.setException(Client.java:185)
    [junit]     - locked <0x00000000f59e2818> (a org.apache.hadoop.ipc.Client$ParallelCall)
    [junit]     at org.apache.hadoop.ipc.Client$Connection.cleanupCalls(Client.java:843)
    [junit]     at org.apache.hadoop.ipc.Client$Connection.close(Client.java:832)
    [junit]     - locked <0x00000000f59d8a90> (a org.apache.hadoop.ipc.Client$Connection)
    [junit]     at org.apache.hadoop.ipc.Client$Connection.run(Client.java:708)
    [junit] "Thread-242":
    [junit]     at org.apache.hadoop.ipc.Client$Connection.markClosed(Client.java:788)
    [junit]     - waiting to lock <0x00000000f59d8a90> (a org.apache.hadoop.ipc.Client$Connection)
    [junit]     at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:742)
    [junit]     at org.apache.hadoop.ipc.Client.call(Client.java:1109)
    [junit]     - locked <0x00000000f599ef88> (a org.apache.hadoop.ipc.Client$ParallelResults)
    [junit]     at org.apache.hadoop.ipc.TestIPC$ParallelCaller.run(TestIPC.java:135)


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15463-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 07:56:28 2011
Return-Path: <common-issues-return-15463-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E2A896691
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 07:56:28 +0000 (UTC)
Received: (qmail 40494 invoked by uid 500); 26 May 2011 07:56:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40468 invoked by uid 500); 26 May 2011 07:56:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40447 invoked by uid 99); 26 May 2011 07:56:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 07:56:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 07:56:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BF5A0DF534
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 07:55:47 +0000 (UTC)
Date: Thu, 26 May 2011 07:55:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <645760220.44665.1306396547780.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1659614701.18791.1297804257612.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7146) RPC server leaks file descriptors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7146:
--------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

Committed to 22 and trunk. Thanks Matt and Aaron

> RPC server leaks file descriptors
> ---------------------------------
>
>                 Key: HADOOP-7146
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7146
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: fd-leak.txt, hadoop-7146.txt, hadoop-7146.txt, hadoop-7146.txt
>
>
> Both the Listener and Responder thread call Selector.open but don't have a matching .close(). This causes a leak of anonymous pipes. Not a big deal because people rarely close and re-open servers, but worth fixing.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15464-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 09:11:29 2011
Return-Path: <common-issues-return-15464-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0466C6973
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 09:11:29 +0000 (UTC)
Received: (qmail 30764 invoked by uid 500); 26 May 2011 09:11:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30723 invoked by uid 500); 26 May 2011 09:11:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30709 invoked by uid 99); 26 May 2011 09:11:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 09:11:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 09:11:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9AC5EE052E
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 09:10:47 +0000 (UTC)
Date: Thu, 26 May 2011 09:10:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <755015128.44828.1306401047630.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039600#comment-13039600 ] 

Hadoop QA commented on HADOOP-7144:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480460/jmx.json
  against trunk revision 1127811.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/526//console

This message is automatically generated.

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Robert Joseph Evans
>              Labels: jmx
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7411-0.20.20X-V1.patch, HADOOP-7411-trunk-V1.patch, HADOOP-7411-trunk-alpha.patch, jmx.json
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15465-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 09:23:30 2011
Return-Path: <common-issues-return-15465-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B08526FE3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 09:23:30 +0000 (UTC)
Received: (qmail 47785 invoked by uid 500); 26 May 2011 09:23:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47697 invoked by uid 500); 26 May 2011 09:23:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47683 invoked by uid 99); 26 May 2011 09:23:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 09:23:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 09:23:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4552AE08ED
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 09:22:49 +0000 (UTC)
Date: Thu, 26 May 2011 09:22:49 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1397408236.44871.1306401769280.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D130=
39605#comment-13039605 ]=20

Hadoop QA commented on HADOOP-6255:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment=20
  http://issues.apache.org/jira/secure/attachment/12480476/HADOOP-6255-comm=
on-trunk-10.patch
  against trunk revision 1127811.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 27 new or modified tes=
ts.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of java=
c compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.=
3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number=
 of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compi=
le.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/5=
27//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Bu=
ild/527//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build=
/527//console

This message is automatically generated.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-625=
5-branch-0.20-security-10.patch, HADOOP-6255-branch-0.20-security-11.patch,=
 HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.20-security=
-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-branch-0.20=
-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HADOOP-6255-br=
anch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.patch, HADOO=
P-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-security.patch=
, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-10.patch, HADO=
OP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk-4.patch, HADOOP-6255=
-common-trunk-5.patch, HADOOP-6255-common-trunk-6.patch, HADOOP-6255-common=
-trunk-7.patch, HADOOP-6255-common-trunk-8.patch, HADOOP-6255-common-trunk-=
9.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch, HA=
DOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-6255-m=
apred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, dep=
loyment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15466-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 10:48:30 2011
Return-Path: <common-issues-return-15466-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9CE586E06
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 10:48:30 +0000 (UTC)
Received: (qmail 66552 invoked by uid 500); 26 May 2011 10:48:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66511 invoked by uid 500); 26 May 2011 10:48:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66503 invoked by uid 99); 26 May 2011 10:48:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 10:48:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 10:48:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 632BEE042D
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 10:47:47 +0000 (UTC)
Date: Thu, 26 May 2011 10:47:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1603308083.45025.1306406867402.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1659614701.18791.1297804257612.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7146) RPC server leaks file descriptors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039638#comment-13039638 ] 

Hudson commented on HADOOP-7146:
--------------------------------

Integrated in Hadoop-Common-22-branch #57 (See [https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/57/])
    HADOOP-7146. RPC server leaks file descriptors. Contributed by Todd Lipcon.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1127815
Files : 
* /hadoop/common/branches/branch-0.22/src/java/org/apache/hadoop/ipc/Server.java
* /hadoop/common/branches/branch-0.22/src/test/core/org/apache/hadoop/ipc/TestIPC.java
* /hadoop/common/branches/branch-0.22/CHANGES.txt


> RPC server leaks file descriptors
> ---------------------------------
>
>                 Key: HADOOP-7146
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7146
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: fd-leak.txt, hadoop-7146.txt, hadoop-7146.txt, hadoop-7146.txt
>
>
> Both the Listener and Responder thread call Selector.open but don't have a matching .close(). This causes a leak of anonymous pipes. Not a big deal because people rarely close and re-open servers, but worth fixing.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15467-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 10:56:31 2011
Return-Path: <common-issues-return-15467-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E181B606B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 10:56:30 +0000 (UTC)
Received: (qmail 78973 invoked by uid 500); 26 May 2011 10:56:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78872 invoked by uid 500); 26 May 2011 10:56:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78864 invoked by uid 99); 26 May 2011 10:56:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 10:56:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 10:56:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9D170E069A
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 10:55:47 +0000 (UTC)
Date: Thu, 26 May 2011 10:55:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <599082826.45047.1306407347639.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1659614701.18791.1297804257612.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7146) RPC server leaks file descriptors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039640#comment-13039640 ] 

Hudson commented on HADOOP-7146:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #623 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/623/])
    HADOOP-7146. RPC server leaks file descriptors. Contributed by Todd Lipcon.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1127811
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/ipc/Server.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/ipc/TestIPC.java


> RPC server leaks file descriptors
> ---------------------------------
>
>                 Key: HADOOP-7146
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7146
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: fd-leak.txt, hadoop-7146.txt, hadoop-7146.txt, hadoop-7146.txt
>
>
> Both the Listener and Responder thread call Selector.open but don't have a matching .close(). This causes a leak of anonymous pipes. Not a big deal because people rarely close and re-open servers, but worth fixing.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15468-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 12:39:29 2011
Return-Path: <common-issues-return-15468-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6D10863DD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 12:39:29 +0000 (UTC)
Received: (qmail 64362 invoked by uid 500); 26 May 2011 12:39:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64323 invoked by uid 500); 26 May 2011 12:39:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64315 invoked by uid 99); 26 May 2011 12:39:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 12:39:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 12:39:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 47F06E0AAF
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 12:38:48 +0000 (UTC)
Date: Thu, 26 May 2011 12:38:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <934475441.45205.1306413528291.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1659614701.18791.1297804257612.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7146) RPC server leaks file descriptors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039680#comment-13039680 ] 

Hudson commented on HADOOP-7146:
--------------------------------

Integrated in Hadoop-Common-trunk #700 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/700/])
    HADOOP-7146. RPC server leaks file descriptors. Contributed by Todd Lipcon.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1127811
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/ipc/Server.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/ipc/TestIPC.java


> RPC server leaks file descriptors
> ---------------------------------
>
>                 Key: HADOOP-7146
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7146
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: fd-leak.txt, hadoop-7146.txt, hadoop-7146.txt, hadoop-7146.txt
>
>
> Both the Listener and Responder thread call Selector.open but don't have a matching .close(). This causes a leak of anonymous pipes. Not a big deal because people rarely close and re-open servers, but worth fixing.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15469-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 12:39:30 2011
Return-Path: <common-issues-return-15469-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E564F63EF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 12:39:29 +0000 (UTC)
Received: (qmail 64601 invoked by uid 500); 26 May 2011 12:39:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64564 invoked by uid 500); 26 May 2011 12:39:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64556 invoked by uid 99); 26 May 2011 12:39:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 12:39:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 12:39:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A398FE0ABC
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 12:38:48 +0000 (UTC)
Date: Thu, 26 May 2011 12:38:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2119035056.45216.1306413528667.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1828723839.36128.1306162247409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7320) Refactor FsShell's copy & move
 commands
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039681#comment-13039681 ] 

Hudson commented on HADOOP-7320:
--------------------------------

Integrated in Hadoop-Common-trunk #700 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/700/])
    HADOOP-7320. Refactor the copy and move commands to conform to new FsCommand class. Contributed by Daryn Sharp.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1127591
Files : 
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/FsCommand.java
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/PathData.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/cli/testConf.xml
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/CopyCommands.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/CommandWithDestination.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/MoveCommands.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/PathExceptions.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Ls.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Copy.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/FsShell.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/TestFsShellReturnCode.java


> Refactor FsShell's copy & move commands
> ---------------------------------------
>
>                 Key: HADOOP-7320
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7320
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7320-2.patch, HADOOP-7320-3.patch, HADOOP-7320-rename.patch, HADOOP-7320.patch
>
>
> Need to refactor the move and copy commands to conform to the FsCommand class.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15470-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 12:39:30 2011
Return-Path: <common-issues-return-15470-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4B20E63F8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 12:39:30 +0000 (UTC)
Received: (qmail 64859 invoked by uid 500); 26 May 2011 12:39:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64817 invoked by uid 500); 26 May 2011 12:39:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64809 invoked by uid 99); 26 May 2011 12:39:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 12:39:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 12:39:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CBB82E0AC2
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 12:38:48 +0000 (UTC)
Date: Thu, 26 May 2011 12:38:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1642097636.45221.1306413528831.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039682#comment-13039682 ] 

Hudson commented on HADOOP-7284:
--------------------------------

Integrated in Hadoop-Common-trunk #700 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/700/])
    Revert HADOOP-7284 (r1127642) since it broke the HDFS build.
HADOOP-7284 Trash and shell's rm does not work for viewfs (Sanjay Radia)

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1127679
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/viewfs/ChRootedFileSystem.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Delete.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/ViewFileSystemTestSetup.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/TestTrash.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/Trash.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/TestChRootedFs.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/TestChRootedFileSystem.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/TestViewFsTrash.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/TestFSMainOperationsLocalFileSystem.java

sradia : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1127642
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/viewfs/ChRootedFileSystem.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/Delete.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/ViewFileSystemTestSetup.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/TestTrash.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/Trash.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/TestChRootedFs.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/TestChRootedFileSystem.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/TestViewFsTrash.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/viewfs/TestFSMainOperationsLocalFileSystem.java


> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15471-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 12:39:32 2011
Return-Path: <common-issues-return-15471-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A43376425
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 12:39:32 +0000 (UTC)
Received: (qmail 65657 invoked by uid 500); 26 May 2011 12:39:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65609 invoked by uid 500); 26 May 2011 12:39:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65600 invoked by uid 99); 26 May 2011 12:39:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 12:39:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 12:39:30 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 05D9DE0ACC
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 12:38:49 +0000 (UTC)
Date: Thu, 26 May 2011 12:38:49 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <155338093.45228.1306413529021.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1164170325.41524.1306304627421.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7329) incomplete help message  is
 displayed for df -h option
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039684#comment-13039684 ] 

Hudson commented on HADOOP-7329:
--------------------------------

Integrated in Hadoop-Common-trunk #700 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/700/])
    HADOOP-7329. Improve help message for "df" to include "-h" flag. Contributed by Xie Xianshan.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1127576
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/shell/FsUsage.java


> incomplete help message  is displayed for df -h option
> ------------------------------------------------------
>
>                 Key: HADOOP-7329
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7329
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Assignee: XieXianshan
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7329
>
>
> The help message for the command "hdfs dfs -help df" is displayed like this:
> "-df [<path> ...]:    Shows the capacity, free and used space of the filesystem.
>         If the filesystem has multiple partitions, and no path to a
>         particular partition is specified, then the status of the root
>         partitions will be shown."
> and the information about df -h option is missed,despite the fact that df -h option is implemented.
> Therefore,the expected message should be displayed like this:
> "-df [-h] [<path> ...]:    Shows the capacity, free and used space of the filesystem.
>         If the filesystem has multiple partitions, and no path to a
>         particular partition is specified, then the status of the root
>         partitions will be shown.
>           -h   Formats the sizes of files in a human-readable fashion
>                rather than a number of bytes."

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15472-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 12:39:33 2011
Return-Path: <common-issues-return-15472-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E35436430
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 12:39:32 +0000 (UTC)
Received: (qmail 65784 invoked by uid 500); 26 May 2011 12:39:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65741 invoked by uid 500); 26 May 2011 12:39:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65686 invoked by uid 99); 26 May 2011 12:39:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 12:39:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 12:39:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id ED768E0AC9
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 12:38:48 +0000 (UTC)
Date: Thu, 26 May 2011 12:38:48 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <199230293.45226.1306413528969.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1067116917.37592.1306187867429.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7322) Adding a util method in FileUtil
 for directory listing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039683#comment-13039683 ] 

Hudson commented on HADOOP-7322:
--------------------------------

Integrated in Hadoop-Common-trunk #700 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/700/])
    HADOOP-7322. Adding a util method in FileUtil for directory listing, avoid NPEs on File.listFiles(). Contributed by Bharath Mundlapudi.

mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1127697
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/RawLocalFileSystem.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/fs/FileUtil.java
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/TestFileUtil.java


> Adding a util method in FileUtil for directory listing
> ------------------------------------------------------
>
>                 Key: HADOOP-7322
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7322
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7322-1.patch, HADOOP-7322-2.patch, HADOOP-7322-3.patch
>
>
> While testing Disk Fail Inplace, we encountered lots of NPE from Dir.listFiles API. This API can return null when Dir is not directory or disk is bad. I am proposing to have a File Util which can be used consistently across to deal with disk issues. This util api will do the following:
> 1. When error happens it will throw IOException
> 2. Else it will return empty list or list of files.
> Signature:
> File[] FileUtil.listFiles(File dir) throws IOException {}
> This way we no need to write wrapper code every where. Also, API is consistent with the signature.
>  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15473-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 14:49:30 2011
Return-Path: <common-issues-return-15473-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D6A776615
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 14:49:30 +0000 (UTC)
Received: (qmail 87445 invoked by uid 500); 26 May 2011 14:49:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87413 invoked by uid 500); 26 May 2011 14:49:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87405 invoked by uid 99); 26 May 2011 14:49:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 14:49:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 14:49:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5BF09E0D9B
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 14:48:47 +0000 (UTC)
Date: Thu, 26 May 2011 14:48:47 +0000 (UTC)
From: "Robert Joseph Evans (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <568525803.45431.1306421327373.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039724#comment-13039724 ] 

Robert Joseph Evans commented on HADOOP-7144:
---------------------------------------------

Ahh, Really Jenkins tried to use the JSON as a patch?  Oh well the following are the manual results for trunk
     [exec] +1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 4 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec] 
     [exec]     +1 system test framework.  The patch passed system test framework compile.
     [exec] 


> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Robert Joseph Evans
>              Labels: jmx
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7411-0.20.20X-V1.patch, HADOOP-7411-trunk-V1.patch, HADOOP-7411-trunk-alpha.patch, jmx.json
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15474-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 15:05:31 2011
Return-Path: <common-issues-return-15474-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0B762654F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 15:05:31 +0000 (UTC)
Received: (qmail 20043 invoked by uid 500); 26 May 2011 15:05:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20005 invoked by uid 500); 26 May 2011 15:05:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19991 invoked by uid 99); 26 May 2011 15:05:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 15:05:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 15:05:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7A968E04C3
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 15:04:47 +0000 (UTC)
Date: Thu, 26 May 2011 15:04:47 +0000 (UTC)
From: "Eric Caspole (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1654473161.45482.1306422287498.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7333) Performance improvement in
 PureJavaCrc32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Performance improvement in PureJavaCrc32
----------------------------------------

                 Key: HADOOP-7333
                 URL: https://issues.apache.org/jira/browse/HADOOP-7333
             Project: Hadoop Common
          Issue Type: Improvement
          Components: util
    Affects Versions: 0.21.0
         Environment: Linux x64
            Reporter: Eric Caspole
            Priority: Minor


I would like to propose a small patch to 

  org.apache.hadoop.util.PureJavaCrc32.update(byte[] b, int off, int len)

Currently the method stores the intermediate result back into the data member "crc." I noticed this method gets
inlined into DataChecksum.update() and that method appears as one of the hotter methods in a simple hprof profile collected while running terasort and gridmix.

If the code is modified to save the temporary result into a local and just once store the final result back into the data member, it results in slightly more efficient hotspot codegen.

I tested this change using the the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32 on a variety of linux x64 AMD and Intel multi-socket and multi-core systems I have available to test.

The patch removes several stores of the intermediate result to memory yielding a 0%-10% speedup in the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32.
 
If you use a debug hotspot JVM with -XX:+PrintOptoAssembly, you can see the intermediate stores such as:

414     movq    R9, [rsp + #24] # spill
419     movl    [R9 + #12 (8-bit)], RDX # int ! Field PureJavaCrc32.crc
41d     xorl    R10, RDX        # int

The patch results in just one final store of the fully computed value.


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15475-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 15:07:29 2011
Return-Path: <common-issues-return-15475-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E92656619
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 15:07:28 +0000 (UTC)
Received: (qmail 22622 invoked by uid 500); 26 May 2011 15:07:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22589 invoked by uid 500); 26 May 2011 15:07:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22581 invoked by uid 99); 26 May 2011 15:07:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 15:07:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 15:07:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B110FE0608
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 15:06:47 +0000 (UTC)
Date: Thu, 26 May 2011 15:06:47 +0000 (UTC)
From: "Eric Caspole (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <542285436.45490.1306422407721.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1654473161.45482.1306422287498.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7333) Performance improvement in
 PureJavaCrc32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Caspole updated HADOOP-7333:
---------------------------------

    Attachment: HADOOP-7333.patch

> Performance improvement in PureJavaCrc32
> ----------------------------------------
>
>                 Key: HADOOP-7333
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7333
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>         Environment: Linux x64
>            Reporter: Eric Caspole
>            Priority: Minor
>         Attachments: HADOOP-7333.patch
>
>
> I would like to propose a small patch to 
>   org.apache.hadoop.util.PureJavaCrc32.update(byte[] b, int off, int len)
> Currently the method stores the intermediate result back into the data member "crc." I noticed this method gets
> inlined into DataChecksum.update() and that method appears as one of the hotter methods in a simple hprof profile collected while running terasort and gridmix.
> If the code is modified to save the temporary result into a local and just once store the final result back into the data member, it results in slightly more efficient hotspot codegen.
> I tested this change using the the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32 on a variety of linux x64 AMD and Intel multi-socket and multi-core systems I have available to test.
> The patch removes several stores of the intermediate result to memory yielding a 0%-10% speedup in the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32.
>  
> If you use a debug hotspot JVM with -XX:+PrintOptoAssembly, you can see the intermediate stores such as:
> 414     movq    R9, [rsp + #24] # spill
> 419     movl    [R9 + #12 (8-bit)], RDX # int ! Field PureJavaCrc32.crc
> 41d     xorl    R10, RDX        # int
> The patch results in just one final store of the fully computed value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15476-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 16:22:29 2011
Return-Path: <common-issues-return-15476-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DFFF26F84
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 16:22:28 +0000 (UTC)
Received: (qmail 56867 invoked by uid 500); 26 May 2011 16:22:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56839 invoked by uid 500); 26 May 2011 16:22:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56831 invoked by uid 99); 26 May 2011 16:22:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 16:22:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 16:22:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C20DBE03FE
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 16:21:47 +0000 (UTC)
Date: Thu, 26 May 2011 16:21:47 +0000 (UTC)
From: "Robert Joseph Evans (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1457725625.45652.1306426907791.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039758#comment-13039758 ] 

Robert Joseph Evans commented on HADOOP-7144:
---------------------------------------------

Hey Luke, what will it take to get HADOOP-7330 into 0.20.20X?  I looked at the patch and it looks straight forward?  It would be nice to have the JSON output consistent between releases. 

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Robert Joseph Evans
>              Labels: jmx
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7411-0.20.20X-V1.patch, HADOOP-7411-trunk-V1.patch, HADOOP-7411-trunk-alpha.patch, jmx.json
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15477-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 16:48:37 2011
Return-Path: <common-issues-return-15477-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0B40E6363
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 16:48:37 +0000 (UTC)
Received: (qmail 84487 invoked by uid 500); 26 May 2011 16:48:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84460 invoked by uid 500); 26 May 2011 16:48:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84452 invoked by uid 99); 26 May 2011 16:48:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 16:48:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 16:48:35 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CC45BE0F10
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 16:47:55 +0000 (UTC)
Date: Thu, 26 May 2011 16:47:55 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1425457432.45758.1306428475833.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039773#comment-13039773 ] 

Luke Lu commented on HADOOP-7144:
---------------------------------

bq. what will it take to get HADOOP-7330 into 0.20.20X?

+1 on the jira and we'll nag one of the committers to push the changes to these branches :)

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Robert Joseph Evans
>              Labels: jmx
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7411-0.20.20X-V1.patch, HADOOP-7411-trunk-V1.patch, HADOOP-7411-trunk-alpha.patch, jmx.json
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15478-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 16:52:31 2011
Return-Path: <common-issues-return-15478-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 129E864D4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 16:52:31 +0000 (UTC)
Received: (qmail 90408 invoked by uid 500); 26 May 2011 16:52:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90372 invoked by uid 500); 26 May 2011 16:52:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90364 invoked by uid 99); 26 May 2011 16:52:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 16:52:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 16:52:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A6C2CE00A2
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 16:51:47 +0000 (UTC)
Date: Thu, 26 May 2011 16:51:47 +0000 (UTC)
From: "Robert Joseph Evans (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <500265503.45787.1306428707679.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1937737372.43422.1306361387389.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7330) The metrics source mbean
 implementation should return the attribute value instead of the object
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039777#comment-13039777 ] 

Robert Joseph Evans commented on HADOOP-7330:
---------------------------------------------

Comment +1 on the patch.  It looks good, some tests would be nice, but it is a single line change that really doesn't impact much.  Plus it makes 0.20.20X consistent with trunk.

> The metrics source mbean implementation should return the attribute value instead of the object
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7330
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7330
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.203.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.20.203.1
>
>         Attachments: hadoop-7330-jmx-v1.patch
>
>
> The MetricsSourceAdapter#getAttribute in 0.20.203 is returning the attribute object instead of the value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15479-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 16:58:29 2011
Return-Path: <common-issues-return-15479-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E70A466B1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 16:58:28 +0000 (UTC)
Received: (qmail 4465 invoked by uid 500); 26 May 2011 16:58:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4433 invoked by uid 500); 26 May 2011 16:58:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4421 invoked by uid 99); 26 May 2011 16:58:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 16:58:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 16:58:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A5E07E02AF
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 16:57:47 +0000 (UTC)
Date: Thu, 26 May 2011 16:57:47 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <234427785.45794.1306429067676.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6255?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-6255:
------------------------------

    Attachment: HADOOP-6255-common-trunk-11.patch

Replace setup script sudo command with su for external management software =
to invoke the script without tty.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-625=
5-branch-0.20-security-10.patch, HADOOP-6255-branch-0.20-security-11.patch,=
 HADOOP-6255-branch-0.20-security-12.patch, HADOOP-6255-branch-0.20-securit=
y-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.2=
0-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-b=
ranch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADO=
OP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security-9.pa=
tch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk-1.pat=
ch, HADOOP-6255-common-trunk-10.patch, HADOOP-6255-common-trunk-11.patch, H=
ADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk-4.patch, HADOOP-6=
255-common-trunk-5.patch, HADOOP-6255-common-trunk-6.patch, HADOOP-6255-com=
mon-trunk-7.patch, HADOOP-6255-common-trunk-8.patch, HADOOP-6255-common-tru=
nk-9.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch,=
 HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-625=
5-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, =
deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15480-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 16:58:30 2011
Return-Path: <common-issues-return-15480-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 55AB066CF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 16:58:30 +0000 (UTC)
Received: (qmail 4915 invoked by uid 500); 26 May 2011 16:58:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4856 invoked by uid 500); 26 May 2011 16:58:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4789 invoked by uid 99); 26 May 2011 16:58:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 16:58:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 16:58:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 00EBEE02DB
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 16:57:49 +0000 (UTC)
Date: Thu, 26 May 2011 16:57:48 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1679270735.45829.1306429069000.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6255?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-6255:
------------------------------

    Attachment: HADOOP-6255-branch-0.20-security-12.patch

Replace setup script sudo command with su for external management software =
to invoke the script without tty, same patch for 0.20 security branch.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-625=
5-branch-0.20-security-10.patch, HADOOP-6255-branch-0.20-security-11.patch,=
 HADOOP-6255-branch-0.20-security-12.patch, HADOOP-6255-branch-0.20-securit=
y-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.2=
0-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-b=
ranch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADO=
OP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security-9.pa=
tch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk-1.pat=
ch, HADOOP-6255-common-trunk-10.patch, HADOOP-6255-common-trunk-11.patch, H=
ADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk-4.patch, HADOOP-6=
255-common-trunk-5.patch, HADOOP-6255-common-trunk-6.patch, HADOOP-6255-com=
mon-trunk-7.patch, HADOOP-6255-common-trunk-8.patch, HADOOP-6255-common-tru=
nk-9.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch,=
 HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-625=
5-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, =
deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15481-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 17:00:29 2011
Return-Path: <common-issues-return-15481-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 158386A54
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 17:00:29 +0000 (UTC)
Received: (qmail 8370 invoked by uid 500); 26 May 2011 17:00:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8326 invoked by uid 500); 26 May 2011 17:00:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8318 invoked by uid 99); 26 May 2011 17:00:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 17:00:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 17:00:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CE849E0442
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 16:59:47 +0000 (UTC)
Date: Thu, 26 May 2011 16:59:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2107855344.45871.1306429187842.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1654473161.45482.1306422287498.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7333) Performance improvement in
 PureJavaCrc32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7333:
--------------------------------

    Status: Patch Available  (was: Open)

Hey Eric. Nice work! I'm hitting the "submit patch" button so that the QA bot will run on this. I'll also try to get a chance to reproduce your performance test results.

> Performance improvement in PureJavaCrc32
> ----------------------------------------
>
>                 Key: HADOOP-7333
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7333
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>         Environment: Linux x64
>            Reporter: Eric Caspole
>            Priority: Minor
>         Attachments: HADOOP-7333.patch
>
>
> I would like to propose a small patch to 
>   org.apache.hadoop.util.PureJavaCrc32.update(byte[] b, int off, int len)
> Currently the method stores the intermediate result back into the data member "crc." I noticed this method gets
> inlined into DataChecksum.update() and that method appears as one of the hotter methods in a simple hprof profile collected while running terasort and gridmix.
> If the code is modified to save the temporary result into a local and just once store the final result back into the data member, it results in slightly more efficient hotspot codegen.
> I tested this change using the the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32 on a variety of linux x64 AMD and Intel multi-socket and multi-core systems I have available to test.
> The patch removes several stores of the intermediate result to memory yielding a 0%-10% speedup in the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32.
>  
> If you use a debug hotspot JVM with -XX:+PrintOptoAssembly, you can see the intermediate stores such as:
> 414     movq    R9, [rsp + #24] # spill
> 419     movl    [R9 + #12 (8-bit)], RDX # int ! Field PureJavaCrc32.crc
> 41d     xorl    R10, RDX        # int
> The patch results in just one final store of the fully computed value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15482-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 17:00:31 2011
Return-Path: <common-issues-return-15482-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 60EDE6A6C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 17:00:29 +0000 (UTC)
Received: (qmail 8597 invoked by uid 500); 26 May 2011 17:00:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8571 invoked by uid 500); 26 May 2011 17:00:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8402 invoked by uid 99); 26 May 2011 17:00:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 17:00:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 17:00:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E0CC0E0444
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 16:59:47 +0000 (UTC)
Date: Thu, 26 May 2011 16:59:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2027813462.45873.1306429187917.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1654473161.45482.1306422287498.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7333) Performance improvement in
 PureJavaCrc32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon reassigned HADOOP-7333:
-----------------------------------

    Assignee: Eric Caspole

> Performance improvement in PureJavaCrc32
> ----------------------------------------
>
>                 Key: HADOOP-7333
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7333
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>         Environment: Linux x64
>            Reporter: Eric Caspole
>            Assignee: Eric Caspole
>            Priority: Minor
>         Attachments: HADOOP-7333.patch
>
>
> I would like to propose a small patch to 
>   org.apache.hadoop.util.PureJavaCrc32.update(byte[] b, int off, int len)
> Currently the method stores the intermediate result back into the data member "crc." I noticed this method gets
> inlined into DataChecksum.update() and that method appears as one of the hotter methods in a simple hprof profile collected while running terasort and gridmix.
> If the code is modified to save the temporary result into a local and just once store the final result back into the data member, it results in slightly more efficient hotspot codegen.
> I tested this change using the the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32 on a variety of linux x64 AMD and Intel multi-socket and multi-core systems I have available to test.
> The patch removes several stores of the intermediate result to memory yielding a 0%-10% speedup in the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32.
>  
> If you use a debug hotspot JVM with -XX:+PrintOptoAssembly, you can see the intermediate stores such as:
> 414     movq    R9, [rsp + #24] # spill
> 419     movl    [R9 + #12 (8-bit)], RDX # int ! Field PureJavaCrc32.crc
> 41d     xorl    R10, RDX        # int
> The patch results in just one final store of the fully computed value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15483-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 17:04:31 2011
Return-Path: <common-issues-return-15483-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 631946F60
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 17:04:31 +0000 (UTC)
Received: (qmail 20831 invoked by uid 500); 26 May 2011 17:04:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20783 invoked by uid 500); 26 May 2011 17:04:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20770 invoked by uid 99); 26 May 2011 17:04:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 17:04:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 17:04:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B911BE0605
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 17:03:47 +0000 (UTC)
Date: Thu, 26 May 2011 17:03:47 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2141829326.45892.1306429427755.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1937737372.43422.1306361387389.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7330) The metrics source mbean
 implementation should return the attribute value instead of the object
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039785#comment-13039785 ] 

Luke Lu commented on HADOOP-7330:
---------------------------------

bq. some tests would be nice.

Agreed. It'd be awesome if you can take up HADOOP-7273, which would help us write tests for JMX stuff outside HDFS.

> The metrics source mbean implementation should return the attribute value instead of the object
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7330
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7330
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.203.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.20.203.1
>
>         Attachments: hadoop-7330-jmx-v1.patch
>
>
> The MetricsSourceAdapter#getAttribute in 0.20.203 is returning the attribute object instead of the value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15484-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 17:04:33 2011
Return-Path: <common-issues-return-15484-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DFE9E6F8A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 17:04:32 +0000 (UTC)
Received: (qmail 21093 invoked by uid 500); 26 May 2011 17:04:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21058 invoked by uid 500); 26 May 2011 17:04:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21050 invoked by uid 99); 26 May 2011 17:04:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 17:04:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 17:04:30 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 273EDE0659
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 17:03:49 +0000 (UTC)
Date: Thu, 26 May 2011 17:03:49 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1820128409.45924.1306429429157.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D130=
39786#comment-13039786 ]=20

Hadoop QA commented on HADOOP-6255:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment=20
  http://issues.apache.org/jira/secure/attachment/12480556/HADOOP-6255-bran=
ch-0.20-security-12.patch
  against trunk revision 1127811.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 1 new or modified test=
s.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build=
/529//console

This message is automatically generated.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-625=
5-branch-0.20-security-10.patch, HADOOP-6255-branch-0.20-security-11.patch,=
 HADOOP-6255-branch-0.20-security-12.patch, HADOOP-6255-branch-0.20-securit=
y-2.patch, HADOOP-6255-branch-0.20-security-3.patch, HADOOP-6255-branch-0.2=
0-security-4.patch, HADOOP-6255-branch-0.20-security-5.patch, HADOOP-6255-b=
ranch-0.20-security-6.patch, HADOOP-6255-branch-0.20-security-7.patch, HADO=
OP-6255-branch-0.20-security-8.patch, HADOOP-6255-branch-0.20-security-9.pa=
tch, HADOOP-6255-branch-0.20-security.patch, HADOOP-6255-common-trunk-1.pat=
ch, HADOOP-6255-common-trunk-10.patch, HADOOP-6255-common-trunk-11.patch, H=
ADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk-4.patch, HADOOP-6=
255-common-trunk-5.patch, HADOOP-6255-common-trunk-6.patch, HADOOP-6255-com=
mon-trunk-7.patch, HADOOP-6255-common-trunk-8.patch, HADOOP-6255-common-tru=
nk-9.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.patch,=
 HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-625=
5-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patch, =
deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15485-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 17:22:29 2011
Return-Path: <common-issues-return-15485-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 01BE767A4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 17:22:29 +0000 (UTC)
Received: (qmail 51381 invoked by uid 500); 26 May 2011 17:22:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51355 invoked by uid 500); 26 May 2011 17:22:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51347 invoked by uid 99); 26 May 2011 17:22:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 17:22:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 17:22:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 96B8AE0FA4
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 17:21:47 +0000 (UTC)
Date: Thu, 26 May 2011 17:21:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1983759717.45975.1306430507614.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1654473161.45482.1306422287498.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7333) Performance improvement in
 PureJavaCrc32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039798#comment-13039798 ] 

Hadoop QA commented on HADOOP-7333:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480549/HADOOP-7333.patch
  against trunk revision 1127811.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/528//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/528//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/528//console

This message is automatically generated.

> Performance improvement in PureJavaCrc32
> ----------------------------------------
>
>                 Key: HADOOP-7333
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7333
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>         Environment: Linux x64
>            Reporter: Eric Caspole
>            Assignee: Eric Caspole
>            Priority: Minor
>         Attachments: HADOOP-7333.patch
>
>
> I would like to propose a small patch to 
>   org.apache.hadoop.util.PureJavaCrc32.update(byte[] b, int off, int len)
> Currently the method stores the intermediate result back into the data member "crc." I noticed this method gets
> inlined into DataChecksum.update() and that method appears as one of the hotter methods in a simple hprof profile collected while running terasort and gridmix.
> If the code is modified to save the temporary result into a local and just once store the final result back into the data member, it results in slightly more efficient hotspot codegen.
> I tested this change using the the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32 on a variety of linux x64 AMD and Intel multi-socket and multi-core systems I have available to test.
> The patch removes several stores of the intermediate result to memory yielding a 0%-10% speedup in the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32.
>  
> If you use a debug hotspot JVM with -XX:+PrintOptoAssembly, you can see the intermediate stores such as:
> 414     movq    R9, [rsp + #24] # spill
> 419     movl    [R9 + #12 (8-bit)], RDX # int ! Field PureJavaCrc32.crc
> 41d     xorl    R10, RDX        # int
> The patch results in just one final store of the fully computed value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15486-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 17:24:31 2011
Return-Path: <common-issues-return-15486-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 09C15689A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 17:24:31 +0000 (UTC)
Received: (qmail 58250 invoked by uid 500); 26 May 2011 17:24:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58216 invoked by uid 500); 26 May 2011 17:24:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58195 invoked by uid 99); 26 May 2011 17:24:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 17:24:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 17:24:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 927C9E00B5
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 17:23:47 +0000 (UTC)
Date: Thu, 26 May 2011 17:23:47 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1551128637.45989.1306430627596.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <504299282.44039.1306374047470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7331) Makes hadoop-daemon.sh to return 1
 if daemon processes did not get started
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039801#comment-13039801 ] 

Tanping Wang commented on HADOOP-7331:
--------------------------------------

This was manually tested.  Three test cases were covered:
# make binding address troublesome -- by changing the config file to bind two name node service address to the same.  After starting name node, hadoop-daemon returned 1.  And error message were logged in log files.  
# make config file invalid - by renaming hdfs-site.xml to something else.  After starting name node, hadoop-daemon returned 1.  And error message were logged in log, i.e. "NameNode: java.lang.IllegalArgumentException: Invalid URI for NameNode address.."
# make some user errors - by not providing the correct options to hadoop-daemon.sh e.g.
hadoop-daemon.sh --conf... --script hdfs 
made hdfs script does not exist.  hadoop-dameon.sh returned 1 and error was output through std out.


> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7331
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7331
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7331.2.patch, HADOOP-7331.patch
>
>
> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15487-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 17:28:29 2011
Return-Path: <common-issues-return-15487-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3C89C6904
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 17:28:29 +0000 (UTC)
Received: (qmail 60838 invoked by uid 500); 26 May 2011 17:28:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60798 invoked by uid 500); 26 May 2011 17:28:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60789 invoked by uid 99); 26 May 2011 17:28:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 17:28:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 17:28:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C0B67E02EB
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 17:27:47 +0000 (UTC)
Date: Thu, 26 May 2011 17:27:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <628088806.46017.1306430867786.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1654473161.45482.1306422287498.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7333) Performance improvement in
 PureJavaCrc32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039807#comment-13039807 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7333:
------------------------------------------------

Eric, good observation!  I just have tried your patch and got the following results.  Could you also post your results?  You may simply copy and paste the text output on the JIRA.

- With patch:
java.version = 1.6.0_10
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_10-b33
java.vm.version = 11.0-b15
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = amd64
os.name = Linux
os.version = 2.6.9-55.ELsmp

Performance Table (The unit is MB/sec)
|| Num Bytes ||    CRC32 || PureJavaCrc32 ||
|          1 |     7.673 |        105.282 |
|          2 |    14.646 |        131.429 |
|          4 |    28.523 |        115.807 |
|          8 |    51.734 |        281.044 |
|         16 |    87.154 |        296.732 |
|         32 |   131.368 |        338.979 |
|         64 |   181.210 |        363.733 |
|        128 |   219.660 |        376.653 |
|        256 |   247.728 |        383.204 |
|        512 |   263.704 |        387.144 |
|       1024 |   271.574 |        387.665 |
|       2048 |   276.787 |        389.429 |
|       4096 |   279.420 |        390.218 |
|       8192 |   280.808 |        390.630 |
|      16384 |   280.895 |        388.867 |
|      32768 |   280.641 |        386.111 |
|      65536 |   280.908 |        386.008 |
|     131072 |   281.049 |        386.089 |
|     262144 |   281.117 |        386.124 |
|     524288 |   281.133 |        386.130 |
|    1048576 |   281.193 |        386.013 |
|    2097152 |   281.208 |        385.508 |
|    4194304 |   280.439 |        384.409 |
|    8388608 |   279.023 |        382.025 |
|   16777216 |   278.610 |        381.445 |

- Without patch:
java.version = 1.6.0_10
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_10-b33
java.vm.version = 11.0-b15
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = amd64
os.name = Linux
os.version = 2.6.9-55.ELsmp

Performance Table (The unit is MB/sec)
|| Num Bytes ||    CRC32 || PureJavaCrc32 ||
|          1 |     7.812 |         78.613 |
|          2 |    15.149 |        135.446 |
|          4 |    28.214 |        144.649 |
|          8 |    50.598 |        292.826 |
|         16 |    87.112 |        294.373 |
|         32 |   132.665 |        354.003 |
|         64 |   179.303 |        387.926 |
|        128 |   218.238 |        408.221 |
|        256 |   246.192 |        418.157 |
|        512 |   262.853 |        424.399 |
|       1024 |   271.822 |        425.695 |
|       2048 |   276.041 |        428.904 |
|       4096 |   279.137 |        428.804 |
|       8192 |   280.552 |        429.013 |
|      16384 |   280.489 |        429.608 |
|      32768 |   280.647 |        426.743 |
|      65536 |   281.974 |        427.366 |
|     131072 |   282.104 |        427.439 |
|     262144 |   282.079 |        427.408 |
|     524288 |   282.252 |        427.355 |
|    1048576 |   282.310 |        427.171 |
|    2097152 |   282.136 |        426.867 |
|    4194304 |   280.107 |        425.458 |
|    8388608 |   280.020 |        422.325 |
|   16777216 |   279.599 |        421.762 |


> Performance improvement in PureJavaCrc32
> ----------------------------------------
>
>                 Key: HADOOP-7333
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7333
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>         Environment: Linux x64
>            Reporter: Eric Caspole
>            Assignee: Eric Caspole
>            Priority: Minor
>         Attachments: HADOOP-7333.patch
>
>
> I would like to propose a small patch to 
>   org.apache.hadoop.util.PureJavaCrc32.update(byte[] b, int off, int len)
> Currently the method stores the intermediate result back into the data member "crc." I noticed this method gets
> inlined into DataChecksum.update() and that method appears as one of the hotter methods in a simple hprof profile collected while running terasort and gridmix.
> If the code is modified to save the temporary result into a local and just once store the final result back into the data member, it results in slightly more efficient hotspot codegen.
> I tested this change using the the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32 on a variety of linux x64 AMD and Intel multi-socket and multi-core systems I have available to test.
> The patch removes several stores of the intermediate result to memory yielding a 0%-10% speedup in the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32.
>  
> If you use a debug hotspot JVM with -XX:+PrintOptoAssembly, you can see the intermediate stores such as:
> 414     movq    R9, [rsp + #24] # spill
> 419     movl    [R9 + #12 (8-bit)], RDX # int ! Field PureJavaCrc32.crc
> 41d     xorl    R10, RDX        # int
> The patch results in just one final store of the fully computed value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15488-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 17:28:32 2011
Return-Path: <common-issues-return-15488-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 20B46690F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 17:28:32 +0000 (UTC)
Received: (qmail 61217 invoked by uid 500); 26 May 2011 17:28:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61185 invoked by uid 500); 26 May 2011 17:28:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61177 invoked by uid 99); 26 May 2011 17:28:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 17:28:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 17:28:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B3287E02E8
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 17:27:47 +0000 (UTC)
Date: Thu, 26 May 2011 17:27:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <533611289.46015.1306430867730.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1654473161.45482.1306422287498.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7333) Performance improvement in
 PureJavaCrc32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039806#comment-13039806 ] 

Todd Lipcon commented on HADOOP-7333:
-------------------------------------

I was able to reproduce a similar performance gain on one of our test boxes (Xeon E5540 @ 2.53ghz)

Without patch:

|| Num Bytes ||    CRC32 || PureJavaCrc32 ||
|          1 |    15.644 |        167.530 |
|          2 |    28.458 |        167.751 |
|          4 |    56.510 |        269.064 |
|          8 |    94.573 |        440.632 |
|         16 |   144.602 |        458.159 |
|         32 |   197.831 |        557.829 |
|         64 |   246.404 |        618.492 |
|        128 |   275.926 |        630.191 |
|        256 |   294.701 |        646.540 |
|        512 |   306.519 |        660.747 |
|       1024 |   308.296 |        666.046 |
|       2048 |   315.532 |        675.444 |
|       4096 |   314.267 |        666.920 |
|       8192 |   314.343 |        669.222 |
|      16384 |   314.655 |        674.295 |
|      32768 |   314.940 |        665.232 |
|      65536 |   314.991 |        670.965 |
|     131072 |   314.950 |        678.721 |
|     262144 |   318.098 |        665.080 |
|     524288 |   317.110 |        661.923 |
|    1048576 |   315.832 |        672.078 |
|    2097152 |   313.902 |        669.421 |
|    4194304 |   315.350 |        670.617 |
|    8388608 |   316.019 |        673.716 |
|   16777216 |   316.324 |        657.439 |

With patch:

|| Num Bytes ||    CRC32 || PureJavaCrc32 ||
|          1 |    15.801 |        150.324 |
|          2 |    29.117 |        159.762 |
|          4 |    60.321 |        261.519 |
|          8 |   100.779 |        423.568 |
|         16 |   153.917 |        458.597 |
|         32 |   208.510 |        564.227 |
|         64 |   249.382 |        644.973 |
|        128 |   281.078 |        714.154 |
|        256 |   298.333 |        737.455 |
|        512 |   308.425 |        750.980 |
|       1024 |   313.149 |        759.572 |
|       2048 |   315.489 |        764.234 |
|       4096 |   316.663 |        767.175 |
|       8192 |   317.716 |        768.538 |
|      16384 |   315.930 |        758.694 |
|      32768 |   317.824 |        763.277 |
|      65536 |   317.246 |        764.298 |
|     131072 |   317.927 |        765.582 |
|     262144 |   317.217 |        750.438 |
|     524288 |   318.005 |        759.085 |
|    1048576 |   318.100 |        763.027 |
|    2097152 |   318.050 |        762.100 |
|    4194304 |   317.742 |        757.370 |
|    8388608 |   316.887 |        753.327 |
|   16777216 |   315.039 |        750.667 |

Our most common use of CRC32 in DFS is with 512 byte chunk size, where we see about a 20% speedup in this test run. For big chunks, looks like about 15% - this is closer to the MR spill checksumming case.

So, +1 pending Hudson results.

> Performance improvement in PureJavaCrc32
> ----------------------------------------
>
>                 Key: HADOOP-7333
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7333
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>         Environment: Linux x64
>            Reporter: Eric Caspole
>            Assignee: Eric Caspole
>            Priority: Minor
>         Attachments: HADOOP-7333.patch
>
>
> I would like to propose a small patch to 
>   org.apache.hadoop.util.PureJavaCrc32.update(byte[] b, int off, int len)
> Currently the method stores the intermediate result back into the data member "crc." I noticed this method gets
> inlined into DataChecksum.update() and that method appears as one of the hotter methods in a simple hprof profile collected while running terasort and gridmix.
> If the code is modified to save the temporary result into a local and just once store the final result back into the data member, it results in slightly more efficient hotspot codegen.
> I tested this change using the the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32 on a variety of linux x64 AMD and Intel multi-socket and multi-core systems I have available to test.
> The patch removes several stores of the intermediate result to memory yielding a 0%-10% speedup in the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32.
>  
> If you use a debug hotspot JVM with -XX:+PrintOptoAssembly, you can see the intermediate stores such as:
> 414     movq    R9, [rsp + #24] # spill
> 419     movl    [R9 + #12 (8-bit)], RDX # int ! Field PureJavaCrc32.crc
> 41d     xorl    R10, RDX        # int
> The patch results in just one final store of the fully computed value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15489-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 17:31:29 2011
Return-Path: <common-issues-return-15489-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B46066FF8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 17:31:29 +0000 (UTC)
Received: (qmail 63405 invoked by uid 500); 26 May 2011 17:31:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63376 invoked by uid 500); 26 May 2011 17:31:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63368 invoked by uid 99); 26 May 2011 17:31:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 17:31:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 17:31:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4808BE03F4
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 17:30:48 +0000 (UTC)
Date: Thu, 26 May 2011 17:30:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1087602784.46035.1306431048291.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1654473161.45482.1306422287498.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7333) Performance improvement in
 PureJavaCrc32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039809#comment-13039809 ] 

Todd Lipcon commented on HADOOP-7333:
-------------------------------------

Looks like Nicholas and I were doing the same thing :) For the record, here's my JVM info:
java.version = 1.6.0_20
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_20-b02
java.vm.version = 16.3-b01
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = amd64
os.name = Linux
os.version = 2.6.18-194.11.1.el5


Nicholas - you good for us to commit this?

> Performance improvement in PureJavaCrc32
> ----------------------------------------
>
>                 Key: HADOOP-7333
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7333
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>         Environment: Linux x64
>            Reporter: Eric Caspole
>            Assignee: Eric Caspole
>            Priority: Minor
>         Attachments: HADOOP-7333.patch
>
>
> I would like to propose a small patch to 
>   org.apache.hadoop.util.PureJavaCrc32.update(byte[] b, int off, int len)
> Currently the method stores the intermediate result back into the data member "crc." I noticed this method gets
> inlined into DataChecksum.update() and that method appears as one of the hotter methods in a simple hprof profile collected while running terasort and gridmix.
> If the code is modified to save the temporary result into a local and just once store the final result back into the data member, it results in slightly more efficient hotspot codegen.
> I tested this change using the the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32 on a variety of linux x64 AMD and Intel multi-socket and multi-core systems I have available to test.
> The patch removes several stores of the intermediate result to memory yielding a 0%-10% speedup in the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32.
>  
> If you use a debug hotspot JVM with -XX:+PrintOptoAssembly, you can see the intermediate stores such as:
> 414     movq    R9, [rsp + #24] # spill
> 419     movl    [R9 + #12 (8-bit)], RDX # int ! Field PureJavaCrc32.crc
> 41d     xorl    R10, RDX        # int
> The patch results in just one final store of the fully computed value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15490-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 17:39:30 2011
Return-Path: <common-issues-return-15490-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B1A136070
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 17:39:30 +0000 (UTC)
Received: (qmail 79778 invoked by uid 500); 26 May 2011 17:39:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79731 invoked by uid 500); 26 May 2011 17:39:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79723 invoked by uid 99); 26 May 2011 17:39:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 17:39:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 17:39:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 818E5E0733
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 17:38:47 +0000 (UTC)
Date: Thu, 26 May 2011 17:38:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <918107472.46052.1306431527527.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <504299282.44039.1306374047470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7331) Makes hadoop-daemon.sh to return 1
 if daemon processes did not get started
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7331:
--------------------------------

    Status: Patch Available  (was: Open)

> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7331
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7331
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7331.2.patch, HADOOP-7331.patch
>
>
> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15491-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 17:43:29 2011
Return-Path: <common-issues-return-15491-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E861C60FA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 17:43:28 +0000 (UTC)
Received: (qmail 84087 invoked by uid 500); 26 May 2011 17:43:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84056 invoked by uid 500); 26 May 2011 17:43:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84047 invoked by uid 99); 26 May 2011 17:43:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 17:43:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 17:43:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6D0D5E08F1
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 17:42:47 +0000 (UTC)
Date: Thu, 26 May 2011 17:42:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <149715124.46067.1306431767443.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7269) S3 Native should allow
 customizable file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039815#comment-13039815 ] 

Todd Lipcon commented on HADOOP-7269:
-------------------------------------

Hi Nicholas. Have you had a chance to run the test case that Tom suggested?

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: HADOOP-7269-S3-metadata-001.diff, HADOOP-7269-S3-metadata-002.diff, HADOOP-7269-S3-metadata-003.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15492-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 17:45:29 2011
Return-Path: <common-issues-return-15492-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A693A64B7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 17:45:29 +0000 (UTC)
Received: (qmail 86265 invoked by uid 500); 26 May 2011 17:45:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86203 invoked by uid 500); 26 May 2011 17:45:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86191 invoked by uid 99); 26 May 2011 17:45:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 17:45:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 17:45:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4BD51E0AFA
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 17:44:48 +0000 (UTC)
Date: Thu, 26 May 2011 17:44:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1002966774.46087.1306431888307.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1093348183.29034.1305849887377.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7312) core-default.xml lists configuration
 version as 0.21
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7312:
--------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

+1. Committed to trunk and 22. Thanks, Harsh!

> core-default.xml lists configuration version as 0.21
> ----------------------------------------------------
>
>                 Key: HADOOP-7312
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7312
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Harsh J Chouraria
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7312-22.r1.diff, HADOOP-7312-23.r1.diff
>
>
> This key was added in HADOOP-6233, though appears unused. I suppose it's somewhat useful to try to diagnose if someone has old versions of core-default.xml on the classpath.
> Either way it should probably be updated to say 0.22 in the branch and 0.23 in trunk.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15493-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 18:00:31 2011
Return-Path: <common-issues-return-15493-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 43A2C6C75
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 18:00:31 +0000 (UTC)
Received: (qmail 14317 invoked by uid 500); 26 May 2011 18:00:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14243 invoked by uid 500); 26 May 2011 18:00:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14235 invoked by uid 99); 26 May 2011 18:00:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 18:00:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 18:00:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BE8D7E117C
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 17:59:47 +0000 (UTC)
Date: Thu, 26 May 2011 17:59:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <266455737.46150.1306432787776.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <504299282.44039.1306374047470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7331) Makes hadoop-daemon.sh to return 1
 if daemon processes did not get started
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039833#comment-13039833 ] 

Hadoop QA commented on HADOOP-7331:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480505/HADOOP-7331.2.patch
  against trunk revision 1127811.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/530//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/530//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/530//console

This message is automatically generated.

> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7331
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7331
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7331.2.patch, HADOOP-7331.patch
>
>
> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15494-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 18:06:30 2011
Return-Path: <common-issues-return-15494-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D1CCE6FE2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 18:06:30 +0000 (UTC)
Received: (qmail 28179 invoked by uid 500); 26 May 2011 18:06:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28144 invoked by uid 500); 26 May 2011 18:06:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28136 invoked by uid 99); 26 May 2011 18:06:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 18:06:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 18:06:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 94FE5E13D5
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 18:05:49 +0000 (UTC)
Date: Thu, 26 May 2011 18:05:49 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1350430221.46167.1306433149606.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1093348183.29034.1305849887377.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7312) core-default.xml lists
 configuration version as 0.21
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039838#comment-13039838 ] 

Hudson commented on HADOOP-7312:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #624 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/624/])
    HADOOP-7312. Update value of hadoop.common.configuration.version. Contributed by Harsh J Chouraria.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1128003
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/core-default.xml


> core-default.xml lists configuration version as 0.21
> ----------------------------------------------------
>
>                 Key: HADOOP-7312
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7312
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Harsh J Chouraria
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7312-22.r1.diff, HADOOP-7312-23.r1.diff
>
>
> This key was added in HADOOP-6233, though appears unused. I suppose it's somewhat useful to try to diagnose if someone has old versions of core-default.xml on the classpath.
> Either way it should probably be updated to say 0.22 in the branch and 0.23 in trunk.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15495-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 18:06:31 2011
Return-Path: <common-issues-return-15495-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4FC166FEE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 18:06:31 +0000 (UTC)
Received: (qmail 28410 invoked by uid 500); 26 May 2011 18:06:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28384 invoked by uid 500); 26 May 2011 18:06:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28376 invoked by uid 99); 26 May 2011 18:06:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 18:06:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 18:06:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EEB28E13DE
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 18:05:49 +0000 (UTC)
Date: Thu, 26 May 2011 18:05:49 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1217518578.46176.1306433149974.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <282223217.18947.1304436003652.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7256) Resource leak during failure
 scenario of closing of resources.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039840#comment-13039840 ] 

Todd Lipcon commented on HADOOP-7256:
-------------------------------------

- looks like there are some hard tabs, please configure your editor to use spaces per our coding guidelines
- in the newly added LOG.debug line, please include some text. eg LOG.debug("Ignoring error closing socket", ignored);
- in the test case, please add a fail("Didn't throw") line after the IOUtils.copyBytes call. That will make sure that the exception is properly propagated and not swallowed.
- Probably better to separate the test in two -- one for the case where the inputstream throws, and another for the case when the output stream throws.

> Resource leak during failure scenario of closing of resources. 
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7256
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7256
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2, 0.21.0
>            Reporter: ramkrishna.s.vasudevan
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7256-patch-1.patch
>
>   Original Estimate: 8h
>  Remaining Estimate: 8h
>
> Problem Statement:
> ===============
> There are chances of resource leak and stream not getting closed 
> Take the case when after copying data we try to close the Input and output stream followed by closing of the socket.
> Suppose an exception occurs while closing the input stream(due to runtime exception) then the subsequent operations of closing the output stream and socket may not happen and there is a chance of resource leak.
> Scenario 
> =======
> During long run of map reduce jobs, the copyFromLocalFile() api is getting called.
> Here we found some exceptions happening. As a result of this we found the lsof value raising leading to resource leak.
> Solution:
> =======
> While doing a close operation of any resource catch the RuntimeException also rather than catching the IOException alone.
> Additionally there are places where we try to close a resource in the catch block.
> If this close fails, we just throw and come out of the current flow.
> In order to avoid this, we can carry out the close operation in the finally block.
> Probable reasons for getting RunTimeExceptions:
> =====================================
> We may get runtime exception from customised hadoop streams like FSDataOutputStream.close() . So better to handle RunTimeExceptions also.
>  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15496-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 18:10:28 2011
Return-Path: <common-issues-return-15496-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B14146113
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 18:10:28 +0000 (UTC)
Received: (qmail 34928 invoked by uid 500); 26 May 2011 18:10:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34903 invoked by uid 500); 26 May 2011 18:10:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34895 invoked by uid 99); 26 May 2011 18:10:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 18:10:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 18:10:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7B114E156B
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 18:09:47 +0000 (UTC)
Date: Thu, 26 May 2011 18:09:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <366381527.46198.1306433387500.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <282223217.18947.1304436003652.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7256) Resource leak during failure
 scenario of closing of resources.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7256:
--------------------------------

    Hadoop Flags:   (was: [Reviewed, Incompatible change])

- How is this an incompatible change?
- Please don't click the "reviewed" checkbox until a committer has reviewed.

> Resource leak during failure scenario of closing of resources. 
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7256
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7256
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2, 0.21.0
>            Reporter: ramkrishna.s.vasudevan
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7256-patch-1.patch
>
>   Original Estimate: 8h
>  Remaining Estimate: 8h
>
> Problem Statement:
> ===============
> There are chances of resource leak and stream not getting closed 
> Take the case when after copying data we try to close the Input and output stream followed by closing of the socket.
> Suppose an exception occurs while closing the input stream(due to runtime exception) then the subsequent operations of closing the output stream and socket may not happen and there is a chance of resource leak.
> Scenario 
> =======
> During long run of map reduce jobs, the copyFromLocalFile() api is getting called.
> Here we found some exceptions happening. As a result of this we found the lsof value raising leading to resource leak.
> Solution:
> =======
> While doing a close operation of any resource catch the RuntimeException also rather than catching the IOException alone.
> Additionally there are places where we try to close a resource in the catch block.
> If this close fails, we just throw and come out of the current flow.
> In order to avoid this, we can carry out the close operation in the finally block.
> Probable reasons for getting RunTimeExceptions:
> =====================================
> We may get runtime exception from customised hadoop streams like FSDataOutputStream.close() . So better to handle RunTimeExceptions also.
>  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15497-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 18:10:31 2011
Return-Path: <common-issues-return-15497-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 55E2E6121
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 18:10:31 +0000 (UTC)
Received: (qmail 35219 invoked by uid 500); 26 May 2011 18:10:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35166 invoked by uid 500); 26 May 2011 18:10:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35150 invoked by uid 99); 26 May 2011 18:10:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 18:10:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 18:10:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 962AEE156E
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 18:09:47 +0000 (UTC)
Date: Thu, 26 May 2011 18:09:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1621657814.46201.1306433387611.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <282223217.18947.1304436003652.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7256) Resource leak during failure
 scenario of closing of resources.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7256:
--------------------------------

    Status: Open  (was: Patch Available)

> Resource leak during failure scenario of closing of resources. 
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7256
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7256
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0, 0.20.2
>            Reporter: ramkrishna.s.vasudevan
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7256-patch-1.patch
>
>   Original Estimate: 8h
>  Remaining Estimate: 8h
>
> Problem Statement:
> ===============
> There are chances of resource leak and stream not getting closed 
> Take the case when after copying data we try to close the Input and output stream followed by closing of the socket.
> Suppose an exception occurs while closing the input stream(due to runtime exception) then the subsequent operations of closing the output stream and socket may not happen and there is a chance of resource leak.
> Scenario 
> =======
> During long run of map reduce jobs, the copyFromLocalFile() api is getting called.
> Here we found some exceptions happening. As a result of this we found the lsof value raising leading to resource leak.
> Solution:
> =======
> While doing a close operation of any resource catch the RuntimeException also rather than catching the IOException alone.
> Additionally there are places where we try to close a resource in the catch block.
> If this close fails, we just throw and come out of the current flow.
> In order to avoid this, we can carry out the close operation in the finally block.
> Probable reasons for getting RunTimeExceptions:
> =====================================
> We may get runtime exception from customised hadoop streams like FSDataOutputStream.close() . So better to handle RunTimeExceptions also.
>  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15498-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 18:44:31 2011
Return-Path: <common-issues-return-15498-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6484C6099
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 18:44:31 +0000 (UTC)
Received: (qmail 71177 invoked by uid 500); 26 May 2011 18:44:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71147 invoked by uid 500); 26 May 2011 18:44:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71131 invoked by uid 99); 26 May 2011 18:44:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 18:44:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 18:44:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 58460E1FE9
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 18:43:47 +0000 (UTC)
Date: Thu, 26 May 2011 18:43:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1260350704.46296.1306435427358.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7334) test-patch should check for hard
 tabs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

test-patch should check for hard tabs
-------------------------------------

                 Key: HADOOP-7334
                 URL: https://issues.apache.org/jira/browse/HADOOP-7334
             Project: Hadoop Common
          Issue Type: Improvement
          Components: build, test
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
            Priority: Minor


Our coding guidelines say that hard tabs are disallowed in the Hadoop code, but they sometimes sneak in (there are about 280 in the common codebase at the moment).

We should run a simple check for this in the test-patch process so it's harder for them to sneak in.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15499-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 18:46:29 2011
Return-Path: <common-issues-return-15499-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1F0C46B22
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 18:46:29 +0000 (UTC)
Received: (qmail 74050 invoked by uid 500); 26 May 2011 18:46:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74024 invoked by uid 500); 26 May 2011 18:46:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74016 invoked by uid 99); 26 May 2011 18:46:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 18:46:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 18:46:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id EB9DCE00B2
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 18:45:47 +0000 (UTC)
Date: Thu, 26 May 2011 18:45:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <674363634.46298.1306435547961.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1260350704.46296.1306435427358.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7334) test-patch should check for hard
 tabs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7334:
--------------------------------

    Attachment: hadoop-7334.txt

I tested this on a patch that had some tabs, and it correctly identified them.

> test-patch should check for hard tabs
> -------------------------------------
>
>                 Key: HADOOP-7334
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7334
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7334.txt
>
>
> Our coding guidelines say that hard tabs are disallowed in the Hadoop code, but they sometimes sneak in (there are about 280 in the common codebase at the moment).
> We should run a simple check for this in the test-patch process so it's harder for them to sneak in.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15500-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 19:05:31 2011
Return-Path: <common-issues-return-15500-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EAC236E5F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 19:05:30 +0000 (UTC)
Received: (qmail 8784 invoked by uid 500); 26 May 2011 19:05:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8737 invoked by uid 500); 26 May 2011 19:05:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8729 invoked by uid 99); 26 May 2011 19:05:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 19:05:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 19:05:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A60C7E08E5
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 19:04:47 +0000 (UTC)
Date: Thu, 26 May 2011 19:04:47 +0000 (UTC)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1686696685.46345.1306436687674.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1260350704.46296.1306435427358.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7334) test-patch should check for hard
 tabs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039863#comment-13039863 ] 

Nigel Daley commented on HADOOP-7334:
-------------------------------------

+1.  Looks like it only checks for tabs on + lines.

> test-patch should check for hard tabs
> -------------------------------------
>
>                 Key: HADOOP-7334
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7334
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7334.txt
>
>
> Our coding guidelines say that hard tabs are disallowed in the Hadoop code, but they sometimes sneak in (there are about 280 in the common codebase at the moment).
> We should run a simple check for this in the test-patch process so it's harder for them to sneak in.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15501-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 19:25:31 2011
Return-Path: <common-issues-return-15501-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 65CA16A35
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 19:25:31 +0000 (UTC)
Received: (qmail 36609 invoked by uid 500); 26 May 2011 19:25:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36566 invoked by uid 500); 26 May 2011 19:25:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36558 invoked by uid 99); 26 May 2011 19:25:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 19:25:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 19:25:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 17CC8E114C
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 19:24:48 +0000 (UTC)
Date: Thu, 26 May 2011 19:24:48 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <529623865.46415.1306437888064.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039873#comment-13039873 ] 

Sanjay Radia commented on HADOOP-7284:
--------------------------------------

Given that this Jira is stuck I will change it to use a config value in the viewfs config.
I am not sure if this is the right solution and in the short term i preferred my original solution till we figure out the right one. 
But there are more important things HDFS needs so I rather compromise than debate.
I am testing the updated patch and will post it shortly.

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15502-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 20:06:29 2011
Return-Path: <common-issues-return-15502-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0E94C6147
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 20:06:29 +0000 (UTC)
Received: (qmail 4036 invoked by uid 500); 26 May 2011 20:06:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3999 invoked by uid 500); 26 May 2011 20:06:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3991 invoked by uid 99); 26 May 2011 20:06:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 20:06:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 20:06:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 60A49E1DC5
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 20:05:47 +0000 (UTC)
Date: Thu, 26 May 2011 20:05:47 +0000 (UTC)
From: "Eric Caspole (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1601120505.46530.1306440347392.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1654473161.45482.1306422287498.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7333) Performance improvement in
 PureJavaCrc32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039890#comment-13039890 ] 

Eric Caspole commented on HADOOP-7333:
--------------------------------------

Here is an example of what I got:

baseline:

java.version = 1.6.0_25
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_25-b06
java.vm.version = 20.0-b11
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = amd64
os.name = Linux
os.version = 2.6.35-22-generic

Performance Table (The unit is MB/sec)
|| Num Bytes ||    CRC32 || PureJavaCrc32 ||
|          1 |    11.312 |         83.741 |
|          2 |    21.210 |        160.665 |
|          4 |    42.377 |        195.647 |
|          8 |    75.407 |        264.901 |
|         16 |   124.451 |        291.424 |
|         32 |   183.565 |        376.901 |
|         64 |   245.988 |        424.204 |
|        128 |   281.473 |        436.240 |
|        256 |   317.394 |        458.765 |
|        512 |   335.792 |        479.419 |
|       1024 |   347.709 |        473.562 |
|       2048 |   351.318 |        476.762 |
|       4096 |   351.745 |        485.425 |
|       8192 |   357.670 |        479.614 |
|      16384 |   355.637 |        481.792 |
|      32768 |   357.449 |        483.500 |
|      65536 |   358.486 |        472.664 |
|     131072 |   355.985 |        483.490 |
|     262144 |   361.117 |        483.599 |
|     524288 |   357.867 |        468.159 |
|    1048576 |   355.371 |        476.476 |
|    2097152 |   355.238 |        476.458 |
|    4194304 |   354.549 |        471.960 |
|    8388608 |   349.739 |        465.464 |
|   16777216 |   347.075 |        461.695 |


with patch:

java.version = 1.6.0_25
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_25-b06
java.vm.version = 20.0-b11
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = amd64
os.name = Linux
os.version = 2.6.35-22-generic

Performance Table (The unit is MB/sec)
|| Num Bytes ||    CRC32 || PureJavaCrc32 ||
|          1 |    11.388 |         70.238 |
|          2 |    21.377 |        140.269 |
|          4 |    42.950 |        195.840 |
|          8 |    76.818 |        316.527 |
|         16 |   126.218 |        336.181 |
|         32 |   187.139 |        407.078 |
|         64 |   246.038 |        450.022 |
|        128 |   283.940 |        463.666 |
|        256 |   315.997 |        484.709 |
|        512 |   337.407 |        492.866 |
|       1024 |   346.651 |        497.357 |
|       2048 |   349.376 |        507.337 |
|       4096 |   356.193 |        497.322 |
|       8192 |   356.303 |        506.458 |
|      16384 |   353.121 |        503.858 |
|      32768 |   351.595 |        503.809 |
|      65536 |   358.412 |        500.009 |
|     131072 |   356.675 |        503.672 |
|     262144 |   356.723 |        501.896 |
|     524288 |   357.432 |        497.297 |
|    1048576 |   349.544 |        500.216 |
|    2097152 |   350.197 |        500.098 |
|    4194304 |   350.040 |        497.357 |
|    8388608 |   348.890 |        477.051 |
|   16777216 |   344.792 |        484.409 |


> Performance improvement in PureJavaCrc32
> ----------------------------------------
>
>                 Key: HADOOP-7333
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7333
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>         Environment: Linux x64
>            Reporter: Eric Caspole
>            Assignee: Eric Caspole
>            Priority: Minor
>         Attachments: HADOOP-7333.patch
>
>
> I would like to propose a small patch to 
>   org.apache.hadoop.util.PureJavaCrc32.update(byte[] b, int off, int len)
> Currently the method stores the intermediate result back into the data member "crc." I noticed this method gets
> inlined into DataChecksum.update() and that method appears as one of the hotter methods in a simple hprof profile collected while running terasort and gridmix.
> If the code is modified to save the temporary result into a local and just once store the final result back into the data member, it results in slightly more efficient hotspot codegen.
> I tested this change using the the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32 on a variety of linux x64 AMD and Intel multi-socket and multi-core systems I have available to test.
> The patch removes several stores of the intermediate result to memory yielding a 0%-10% speedup in the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32.
>  
> If you use a debug hotspot JVM with -XX:+PrintOptoAssembly, you can see the intermediate stores such as:
> 414     movq    R9, [rsp + #24] # spill
> 419     movl    [R9 + #12 (8-bit)], RDX # int ! Field PureJavaCrc32.crc
> 41d     xorl    R10, RDX        # int
> The patch results in just one final store of the fully computed value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15503-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 20:10:34 2011
Return-Path: <common-issues-return-15503-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 20E4B6389
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 20:10:34 +0000 (UTC)
Received: (qmail 16486 invoked by uid 500); 26 May 2011 20:10:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16416 invoked by uid 500); 26 May 2011 20:10:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16346 invoked by uid 99); 26 May 2011 20:10:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 20:10:33 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 20:10:31 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5E4FEE1FCE
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 20:09:50 +0000 (UTC)
Date: Thu, 26 May 2011 20:09:50 +0000 (UTC)
From: "Jonathan Eagles (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1795806506.46579.1306440590382.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1260350704.46296.1306435427358.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7334) test-patch should check for hard
 tabs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039892#comment-13039892 ] 

Jonathan Eagles commented on HADOOP-7334:
-----------------------------------------

Todd, I wondering if we should remove the '-c' on the first grep. I'm in favor of coding style checking. Do you think we have a need to do something like checkstyle in the future?

> test-patch should check for hard tabs
> -------------------------------------
>
>                 Key: HADOOP-7334
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7334
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7334.txt
>
>
> Our coding guidelines say that hard tabs are disallowed in the Hadoop code, but they sometimes sneak in (there are about 280 in the common codebase at the moment).
> We should run a simple check for this in the test-patch process so it's harder for them to sneak in.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15504-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 20:10:34 2011
Return-Path: <common-issues-return-15504-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7C74C63A3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 20:10:34 +0000 (UTC)
Received: (qmail 16786 invoked by uid 500); 26 May 2011 20:10:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16717 invoked by uid 500); 26 May 2011 20:10:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16530 invoked by uid 99); 26 May 2011 20:10:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 20:10:33 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 20:10:31 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8BF7DE1FD3
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 20:09:50 +0000 (UTC)
Date: Thu, 26 May 2011 20:09:50 +0000 (UTC)
From: "Jonathan Eagles (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1722215629.46584.1306440590570.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1260350704.46296.1306435427358.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7334) test-patch should check for hard
 tabs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039894#comment-13039894 ] 

Jonathan Eagles commented on HADOOP-7334:
-----------------------------------------

Other than what I said above, I think this patch looks great.

> test-patch should check for hard tabs
> -------------------------------------
>
>                 Key: HADOOP-7334
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7334
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7334.txt
>
>
> Our coding guidelines say that hard tabs are disallowed in the Hadoop code, but they sometimes sneak in (there are about 280 in the common codebase at the moment).
> We should run a simple check for this in the test-patch process so it's harder for them to sneak in.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15505-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 20:23:28 2011
Return-Path: <common-issues-return-15505-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A9F396B10
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 20:23:28 +0000 (UTC)
Received: (qmail 34018 invoked by uid 500); 26 May 2011 20:23:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33958 invoked by uid 500); 26 May 2011 20:23:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33826 invoked by uid 99); 26 May 2011 20:23:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 20:23:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 20:23:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6DB6EE150E
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 20:22:47 +0000 (UTC)
Date: Thu, 26 May 2011 20:22:47 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1276197862.46631.1306441367446.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <504299282.44039.1306374047470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7331) Makes hadoop-daemon.sh to return 1
 if daemon processes did not get started
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039904#comment-13039904 ] 

Tanping Wang commented on HADOOP-7331:
--------------------------------------

No unit tests included since this is scripts.  Manually tested.

> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7331
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7331
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7331.2.patch, HADOOP-7331.patch
>
>
> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15506-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 20:29:29 2011
Return-Path: <common-issues-return-15506-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 597DB6BE4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 20:29:29 +0000 (UTC)
Received: (qmail 41205 invoked by uid 500); 26 May 2011 20:29:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41164 invoked by uid 500); 26 May 2011 20:29:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40935 invoked by uid 99); 26 May 2011 20:29:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 20:29:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 20:29:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8C608E16A4
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 20:28:47 +0000 (UTC)
Date: Thu, 26 May 2011 20:28:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <625681177.46645.1306441727571.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1654473161.45482.1306422287498.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7333) Performance improvement in
 PureJavaCrc32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039905#comment-13039905 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7333:
------------------------------------------------

> Looks like Nicholas and I were doing the same thing :) ...

Todd, this reminds me the fun when we were doing HADOOP-6166 and HADOOP-6148.

Eric, thanks for joining us.

> So, +1 pending Hudson results.

+1 patch looks good.  Let me run some more tests before committing this.

> Performance improvement in PureJavaCrc32
> ----------------------------------------
>
>                 Key: HADOOP-7333
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7333
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>         Environment: Linux x64
>            Reporter: Eric Caspole
>            Assignee: Eric Caspole
>            Priority: Minor
>         Attachments: HADOOP-7333.patch
>
>
> I would like to propose a small patch to 
>   org.apache.hadoop.util.PureJavaCrc32.update(byte[] b, int off, int len)
> Currently the method stores the intermediate result back into the data member "crc." I noticed this method gets
> inlined into DataChecksum.update() and that method appears as one of the hotter methods in a simple hprof profile collected while running terasort and gridmix.
> If the code is modified to save the temporary result into a local and just once store the final result back into the data member, it results in slightly more efficient hotspot codegen.
> I tested this change using the the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32 on a variety of linux x64 AMD and Intel multi-socket and multi-core systems I have available to test.
> The patch removes several stores of the intermediate result to memory yielding a 0%-10% speedup in the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32.
>  
> If you use a debug hotspot JVM with -XX:+PrintOptoAssembly, you can see the intermediate stores such as:
> 414     movq    R9, [rsp + #24] # spill
> 419     movl    [R9 + #12 (8-bit)], RDX # int ! Field PureJavaCrc32.crc
> 41d     xorl    R10, RDX        # int
> The patch results in just one final store of the fully computed value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15507-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 21:44:31 2011
Return-Path: <common-issues-return-15507-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 012D4637B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 21:44:31 +0000 (UTC)
Received: (qmail 47520 invoked by uid 500); 26 May 2011 21:44:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47471 invoked by uid 500); 26 May 2011 21:44:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47463 invoked by uid 99); 26 May 2011 21:44:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 21:44:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 21:44:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AA31CE12B2
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 21:43:47 +0000 (UTC)
Date: Thu, 26 May 2011 21:43:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <939431288.46789.1306446227694.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1093348183.29034.1305849887377.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7312) core-default.xml lists
 configuration version as 0.21
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039950#comment-13039950 ] 

Hudson commented on HADOOP-7312:
--------------------------------

Integrated in Hadoop-Common-22-branch #58 (See [https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/58/])
    HADOOP-7312. Update value of hadoop.common.configuration.version. Contributed by Harsh J Chouraria.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1128004
Files : 
* /hadoop/common/branches/branch-0.22/src/java/core-default.xml
* /hadoop/common/branches/branch-0.22/CHANGES.txt


> core-default.xml lists configuration version as 0.21
> ----------------------------------------------------
>
>                 Key: HADOOP-7312
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7312
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Harsh J Chouraria
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7312-22.r1.diff, HADOOP-7312-23.r1.diff
>
>
> This key was added in HADOOP-6233, though appears unused. I suppose it's somewhat useful to try to diagnose if someone has old versions of core-default.xml on the classpath.
> Either way it should probably be updated to say 0.22 in the branch and 0.23 in trunk.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15508-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 21:52:29 2011
Return-Path: <common-issues-return-15508-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D0E8E68F0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 21:52:29 +0000 (UTC)
Received: (qmail 55178 invoked by uid 500); 26 May 2011 21:52:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55137 invoked by uid 500); 26 May 2011 21:52:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55129 invoked by uid 99); 26 May 2011 21:52:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 21:52:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_FRT_LITTLE,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 21:52:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E0341E14DD
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 21:51:47 +0000 (UTC)
Date: Thu, 26 May 2011 21:51:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <31886023.46803.1306446707915.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1654473161.45482.1306422287498.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7333) Performance improvement in
 PureJavaCrc32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039952#comment-13039952 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7333:
------------------------------------------------

Tried 6 cases: three different machines, two JVMs (the latest 1.6.0_25-b06 and an earlier version) for each machine.  The patch always improves the performance on the latest JVM (1.6.0_25-b06) but degrades the performance on some earlier JVMs.

In the table below, *CRC32* is {{java.util.zip.CRC32}}, *PureJavaCrc32* is the current code in trunk and *H7333* is the current code with the patch .

-----
h3. 1.1) Linux 2.6.9-55.ELsmp - Java 1.6.0_10
java.version = 1.6.0_10
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_10-b33
java.vm.version = 11.0-b15
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = amd64
os.name = Linux
os.version = 2.6.9-55.ELsmp

Performance Table (The unit is MB/sec)
|| Num Bytes ||    CRC32 || PureJavaCrc32 ||    H7333 ||
|          1 |     7.842 |         72.924 |    82.086 |
|          2 |    15.270 |        106.023 |   121.040 |
|          4 |    27.828 |        103.828 |   119.558 |
|          8 |    52.146 |        263.903 |   253.742 |
|         16 |    85.452 |        283.530 |   280.151 |
|         32 |   130.481 |        343.833 |   325.426 |
|         64 |   177.955 |        380.677 |   354.599 |
|        128 |   217.593 |        404.632 |   372.172 |
|        256 |   243.913 |        416.707 |   379.768 |
|        512 |   261.980 |        423.663 |   384.451 |
|       1024 |   271.463 |        425.220 |   386.562 |
|       2048 |   276.924 |        429.696 |   389.164 |
|       4096 |   279.536 |        430.665 |   390.104 |
|       8192 |   280.603 |        431.043 |   390.557 |
|      16384 |   281.350 |        431.056 |   389.888 |
|      32768 |   281.492 |        427.595 |   387.046 |
|      65536 |   281.838 |        426.990 |   385.032 |
|     131072 |   281.953 |        427.164 |   386.727 |
|     262144 |   282.085 |        427.328 |   387.082 |
|     524288 |   282.137 |        427.503 |   387.093 |
|    1048576 |   282.076 |        427.465 |   387.000 |
|    2097152 |   282.106 |        427.219 |   386.776 |
|    4194304 |   281.215 |        425.642 |   385.575 |
|    8388608 |   280.048 |        422.767 |   383.008 |
|   16777216 |   279.671 |        422.120 |   382.612 |

h3. 1.2) Linux 2.6.9-55.ELsmp - Java 1.6.0_25-b06
java.version = 1.6.0_25
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_25-b06
java.vm.version = 20.0-b11
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = amd64
os.name = Linux
os.version = 2.6.9-55.ELsmp

Performance Table (The unit is MB/sec)
|| Num Bytes ||    CRC32 || PureJavaCrc32 ||    H7333 ||
|          1 |     7.408 |        105.707 |    91.593 |
|          2 |    14.466 |        166.277 |   148.577 |
|          4 |    27.889 |        195.146 |   202.841 |
|          8 |    50.039 |        248.775 |   248.598 |
|         16 |    84.972 |        329.594 |   335.569 |
|         32 |   130.063 |        394.096 |   403.355 |
|         64 |   178.457 |        436.276 |   448.967 |
|        128 |   217.633 |        461.647 |   475.962 |
|        256 |   245.799 |        474.164 |   490.574 |
|        512 |   262.378 |        482.075 |   498.107 |
|       1024 |   270.899 |        482.538 |   498.268 |
|       2048 |   276.426 |        486.091 |   502.043 |
|       4096 |   279.105 |        487.801 |   504.114 |
|       8192 |   278.792 |        488.647 |   504.989 |
|      16384 |   281.229 |        488.611 |   505.019 |
|      32768 |   281.292 |        485.426 |   501.475 |
|      65536 |   281.670 |        485.282 |   499.889 |
|     131072 |   280.752 |        483.555 |   499.573 |
|     262144 |   280.870 |        483.588 |   499.616 |
|     524288 |   280.863 |        483.364 |   499.629 |
|    1048576 |   280.913 |        483.524 |   499.278 |
|    2097152 |   280.552 |        483.234 |   499.226 |
|    4194304 |   280.172 |        481.426 |   497.313 |
|    8388608 |   278.749 |        477.359 |   492.912 |
|   16777216 |   278.458 |        476.276 |   492.101 |

h3. 2.1) Linux 2.6.18-53.1.13.el5 - Java 1.6.0_05-b13
java.version = 1.6.0_05
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_05-b13
java.vm.version = 10.0-b19
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = i386
os.name = Linux
os.version = 2.6.18-53.1.13.el5

Performance Table (The unit is MB/sec)
|| Num Bytes ||    CRC32 || PureJavaCrc32 ||    H7333 ||
|          1 |     9.705 |        111.072 |   111.346 |
|          2 |    18.016 |        138.367 |   137.494 |
|          4 |    34.638 |        213.342 |   226.139 |
|          8 |    62.939 |        221.194 |   239.386 |
|         16 |   107.019 |        289.123 |   307.963 |
|         32 |   164.449 |        368.337 |   378.862 |
|         64 |   224.498 |        425.263 |   423.380 |
|        128 |   275.259 |        463.062 |   449.381 |
|        256 |   309.731 |        485.426 |   465.417 |
|        512 |   331.076 |        497.375 |   473.646 |
|       1024 |   342.002 |        499.592 |   475.978 |
|       2048 |   349.365 |        504.120 |   479.042 |
|       4096 |   352.857 |        506.511 |   480.351 |
|       8192 |   353.812 |        507.538 |   474.044 |
|      16384 |   351.864 |        507.212 |   480.870 |
|      32768 |   351.206 |        499.854 |   479.526 |
|      65536 |   352.413 |        492.054 |   480.082 |
|     131072 |   348.713 |        499.692 |   480.194 |
|     262144 |   353.375 |        491.986 |   480.235 |
|     524288 |   353.703 |        499.682 |   480.275 |
|    1048576 |   353.387 |        499.606 |   480.259 |
|    2097152 |   352.382 |        491.653 |   480.062 |
|    4194304 |   352.810 |        499.002 |   479.758 |
|    8388608 |   351.093 |        493.473 |   475.056 |
|   16777216 |   350.968 |        492.484 |   473.960 |

h3. 2.2) Linux 2.6.18-53.1.13.el5 - Java 1.6.0_25-b06

java.version = 1.6.0_25
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_25-b06
java.vm.version = 20.0-b11
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = amd64
os.name = Linux
os.version = 2.6.18-53.1.13.el5

Performance Table (The unit is MB/sec)
|| Num Bytes ||    CRC32 || PureJavaCrc32 ||    H7333 ||
|          1 |     8.623 |        131.418 |   108.732 |
|          2 |    17.125 |        195.779 |   179.235 |
|          4 |    31.958 |        239.181 |   250.013 |
|          8 |    56.425 |        296.820 |   306.358 |
|         16 |    96.881 |        397.425 |   406.786 |
|         32 |   154.939 |        475.686 |   499.760 |
|         64 |   211.171 |        526.751 |   556.665 |
|        128 |   267.582 |        559.062 |   591.984 |
|        256 |   305.220 |        575.348 |   613.347 |
|        512 |   326.476 |        583.960 |   623.138 |
|       1024 |   338.115 |        585.560 |   624.430 |
|       2048 |   345.509 |        589.541 |   628.801 |
|       4096 |   349.223 |        591.533 |   630.996 |
|       8192 |   351.142 |        592.521 |   632.064 |
|      16384 |   352.057 |        592.771 |   628.888 |
|      32768 |   351.449 |        588.814 |   627.688 |
|      65536 |   352.759 |        588.454 |   627.405 |
|     131072 |   352.914 |        588.611 |   627.559 |
|     262144 |   352.735 |        588.665 |   627.592 |
|     524288 |   353.095 |        588.660 |   627.592 |
|    1048576 |   353.145 |        588.571 |   627.485 |
|    2097152 |   353.140 |        588.262 |   627.276 |
|    4194304 |   352.952 |        587.857 |   626.714 |
|    8388608 |   350.475 |        581.068 |   618.994 |
|   16777216 |   349.872 |        579.330 |   617.299 |

h3. 3.1) Windows XP - Java 1.6.0_23-b05
java.version = 1.6.0_23
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_23-b05
java.vm.version = 19.0-b09
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) Client VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = x86
os.name = Windows XP
os.version = 5.1

Performance Table (The unit is MB/sec)
|| Num Bytes ||    CRC32 || PureJavaCrc32 ||    H7333 ||
|          1 |     4.276 |         62.669 |    64.448 |
|          2 |     7.978 |        102.239 |   109.401 |
|          4 |    16.730 |        132.354 |   147.217 |
|          8 |    31.439 |        182.390 |   195.586 |
|         16 |    54.836 |        219.618 |   234.732 |
|         32 |    87.627 |        238.995 |   270.211 |
|         64 |   129.170 |        253.900 |   304.123 |
|        128 |   170.240 |        278.521 |   321.418 |
|        256 |   211.477 |        286.423 |   341.665 |
|        512 |   265.165 |        304.382 |   279.309 |
|       1024 |   282.990 |        249.336 |   350.560 |
|       2048 |   243.438 |        307.147 |   353.229 |
|       4096 |   294.114 |        309.688 |   281.155 |
|       8192 |   301.212 |        280.119 |   334.366 |
|      16384 |   243.790 |        290.962 |   344.765 |
|      32768 |   292.988 |        295.948 |   272.812 |
|      65536 |   290.493 |        282.761 |   348.932 |
|     131072 |   250.337 |        301.592 |   352.376 |
|     262144 |   302.263 |        307.579 |   285.689 |
|     524288 |   304.144 |        306.469 |   331.565 |
|    1048576 |   289.816 |        205.993 |   350.036 |
|    2097152 |   295.914 |        289.658 |   285.215 |
|    4194304 |   270.211 |        300.434 |   295.595 |
|    8388608 |   236.343 |        276.725 |   339.176 |
|   16777216 |   291.456 |        295.043 |   311.783 |

h3. 3.2) Windows XP - Java 1.6.0_25-b06
java.version = 1.6.0_25
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_25-b06
java.vm.version = 20.0-b11
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) Client VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = x86
os.name = Windows XP
os.version = 5.1

Performance Table (The unit is MB/sec)
|| Num Bytes ||    CRC32 || PureJavaCrc32 ||    H7333 ||
|          1 |     3.868 |         63.751 |    68.806 |
|          2 |     7.819 |         98.380 |   115.517 |
|          4 |    15.446 |        124.773 |   164.977 |
|          8 |    29.635 |        174.449 |   237.747 |
|         16 |    55.030 |        196.089 |   290.992 |
|         32 |    94.131 |        244.999 |   325.735 |
|         64 |   143.858 |        266.674 |   346.240 |
|        128 |   192.723 |        330.168 |   281.406 |
|        256 |   237.439 |        332.342 |   362.283 |
|        512 |   216.150 |        333.393 |   364.037 |
|       1024 |   273.192 |        337.913 |   282.942 |
|       2048 |   293.097 |        336.928 |   368.440 |
|       4096 |   300.675 |        260.551 |   368.051 |
|       8192 |   304.412 |        338.468 |   368.042 |
|      16384 |   232.549 |        338.616 |   367.385 |
|      32768 |   305.303 |        334.487 |   286.756 |
|      65536 |   306.435 |        331.820 |   356.131 |
|     131072 |   274.592 |        292.947 |   363.439 |
|     262144 |   304.008 |        336.345 |   285.965 |
|     524288 |   306.601 |        328.582 |   367.409 |
|    1048576 |   302.045 |        251.833 |   367.058 |
|    2097152 |   296.679 |        333.299 |   362.986 |
|    4194304 |   243.700 |        332.293 |   344.360 |
|    8388608 |   302.936 |        332.315 |   303.883 |
|   16777216 |   301.516 |        329.897 |   362.755 |



> Performance improvement in PureJavaCrc32
> ----------------------------------------
>
>                 Key: HADOOP-7333
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7333
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>         Environment: Linux x64
>            Reporter: Eric Caspole
>            Assignee: Eric Caspole
>            Priority: Minor
>         Attachments: HADOOP-7333.patch
>
>
> I would like to propose a small patch to 
>   org.apache.hadoop.util.PureJavaCrc32.update(byte[] b, int off, int len)
> Currently the method stores the intermediate result back into the data member "crc." I noticed this method gets
> inlined into DataChecksum.update() and that method appears as one of the hotter methods in a simple hprof profile collected while running terasort and gridmix.
> If the code is modified to save the temporary result into a local and just once store the final result back into the data member, it results in slightly more efficient hotspot codegen.
> I tested this change using the the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32 on a variety of linux x64 AMD and Intel multi-socket and multi-core systems I have available to test.
> The patch removes several stores of the intermediate result to memory yielding a 0%-10% speedup in the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32.
>  
> If you use a debug hotspot JVM with -XX:+PrintOptoAssembly, you can see the intermediate stores such as:
> 414     movq    R9, [rsp + #24] # spill
> 419     movl    [R9 + #12 (8-bit)], RDX # int ! Field PureJavaCrc32.crc
> 41d     xorl    R10, RDX        # int
> The patch results in just one final store of the fully computed value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15509-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 21:56:28 2011
Return-Path: <common-issues-return-15509-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9478A69BA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 21:56:28 +0000 (UTC)
Received: (qmail 62277 invoked by uid 500); 26 May 2011 21:56:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62247 invoked by uid 500); 26 May 2011 21:56:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62239 invoked by uid 99); 26 May 2011 21:56:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 21:56:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 21:56:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5C015E1682
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 21:55:47 +0000 (UTC)
Date: Thu, 26 May 2011 21:55:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1117117735.46822.1306446947373.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1654473161.45482.1306422287498.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7333) Performance improvement in
 PureJavaCrc32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7333:
-------------------------------------------

    Attachment: c7333_20110526.patch

c7333_20110526.patch: the codes I used for testing.  See if anyone wants to try it.

> Performance improvement in PureJavaCrc32
> ----------------------------------------
>
>                 Key: HADOOP-7333
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7333
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>         Environment: Linux x64
>            Reporter: Eric Caspole
>            Assignee: Eric Caspole
>            Priority: Minor
>         Attachments: HADOOP-7333.patch, c7333_20110526.patch
>
>
> I would like to propose a small patch to 
>   org.apache.hadoop.util.PureJavaCrc32.update(byte[] b, int off, int len)
> Currently the method stores the intermediate result back into the data member "crc." I noticed this method gets
> inlined into DataChecksum.update() and that method appears as one of the hotter methods in a simple hprof profile collected while running terasort and gridmix.
> If the code is modified to save the temporary result into a local and just once store the final result back into the data member, it results in slightly more efficient hotspot codegen.
> I tested this change using the the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32 on a variety of linux x64 AMD and Intel multi-socket and multi-core systems I have available to test.
> The patch removes several stores of the intermediate result to memory yielding a 0%-10% speedup in the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32.
>  
> If you use a debug hotspot JVM with -XX:+PrintOptoAssembly, you can see the intermediate stores such as:
> 414     movq    R9, [rsp + #24] # spill
> 419     movl    [R9 + #12 (8-bit)], RDX # int ! Field PureJavaCrc32.crc
> 41d     xorl    R10, RDX        # int
> The patch results in just one final store of the fully computed value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15510-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 22:09:31 2011
Return-Path: <common-issues-return-15510-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 99F85605D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 22:09:31 +0000 (UTC)
Received: (qmail 74215 invoked by uid 500); 26 May 2011 22:09:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74174 invoked by uid 500); 26 May 2011 22:09:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74166 invoked by uid 99); 26 May 2011 22:09:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 22:09:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 22:09:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 63597E1A8C
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 22:08:48 +0000 (UTC)
Date: Thu, 26 May 2011 22:08:48 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <743418768.46858.1306447728403.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1654473161.45482.1306422287498.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7333) Performance improvement in
 PureJavaCrc32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039960#comment-13039960 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7333:
------------------------------------------------

> Nicholas - you good for us to commit this?

Let's wait til for a day.  We should commit it unless something new is found.

> Performance improvement in PureJavaCrc32
> ----------------------------------------
>
>                 Key: HADOOP-7333
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7333
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>         Environment: Linux x64
>            Reporter: Eric Caspole
>            Assignee: Eric Caspole
>            Priority: Minor
>         Attachments: HADOOP-7333.patch, c7333_20110526.patch
>
>
> I would like to propose a small patch to 
>   org.apache.hadoop.util.PureJavaCrc32.update(byte[] b, int off, int len)
> Currently the method stores the intermediate result back into the data member "crc." I noticed this method gets
> inlined into DataChecksum.update() and that method appears as one of the hotter methods in a simple hprof profile collected while running terasort and gridmix.
> If the code is modified to save the temporary result into a local and just once store the final result back into the data member, it results in slightly more efficient hotspot codegen.
> I tested this change using the the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32 on a variety of linux x64 AMD and Intel multi-socket and multi-core systems I have available to test.
> The patch removes several stores of the intermediate result to memory yielding a 0%-10% speedup in the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32.
>  
> If you use a debug hotspot JVM with -XX:+PrintOptoAssembly, you can see the intermediate stores such as:
> 414     movq    R9, [rsp + #24] # spill
> 419     movl    [R9 + #12 (8-bit)], RDX # int ! Field PureJavaCrc32.crc
> 41d     xorl    R10, RDX        # int
> The patch results in just one final store of the fully computed value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15511-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 22:15:31 2011
Return-Path: <common-issues-return-15511-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7F066629E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 22:15:31 +0000 (UTC)
Received: (qmail 87922 invoked by uid 500); 26 May 2011 22:15:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87896 invoked by uid 500); 26 May 2011 22:15:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87888 invoked by uid 99); 26 May 2011 22:15:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 22:15:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 22:15:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 918F2E1C21
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 22:14:47 +0000 (UTC)
Date: Thu, 26 May 2011 22:14:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <883564157.46876.1306448087592.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1654473161.45482.1306422287498.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7333) Performance improvement in
 PureJavaCrc32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039964#comment-13039964 ] 

Hadoop QA commented on HADOOP-7333:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480587/c7333_20110526.patch
  against trunk revision 1128003.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/531//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/531//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/531//console

This message is automatically generated.

> Performance improvement in PureJavaCrc32
> ----------------------------------------
>
>                 Key: HADOOP-7333
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7333
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>         Environment: Linux x64
>            Reporter: Eric Caspole
>            Assignee: Eric Caspole
>            Priority: Minor
>         Attachments: HADOOP-7333.patch, c7333_20110526.patch
>
>
> I would like to propose a small patch to 
>   org.apache.hadoop.util.PureJavaCrc32.update(byte[] b, int off, int len)
> Currently the method stores the intermediate result back into the data member "crc." I noticed this method gets
> inlined into DataChecksum.update() and that method appears as one of the hotter methods in a simple hprof profile collected while running terasort and gridmix.
> If the code is modified to save the temporary result into a local and just once store the final result back into the data member, it results in slightly more efficient hotspot codegen.
> I tested this change using the the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32 on a variety of linux x64 AMD and Intel multi-socket and multi-core systems I have available to test.
> The patch removes several stores of the intermediate result to memory yielding a 0%-10% speedup in the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32.
>  
> If you use a debug hotspot JVM with -XX:+PrintOptoAssembly, you can see the intermediate stores such as:
> 414     movq    R9, [rsp + #24] # spill
> 419     movl    [R9 + #12 (8-bit)], RDX # int ! Field PureJavaCrc32.crc
> 41d     xorl    R10, RDX        # int
> The patch results in just one final store of the fully computed value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15512-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 22:29:30 2011
Return-Path: <common-issues-return-15512-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 00125683D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 22:29:29 +0000 (UTC)
Received: (qmail 97702 invoked by uid 500); 26 May 2011 22:29:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97668 invoked by uid 500); 26 May 2011 22:29:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97660 invoked by uid 99); 26 May 2011 22:29:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 22:29:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 22:29:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A5E66E1FC0
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 22:28:48 +0000 (UTC)
Date: Thu, 26 May 2011 22:28:48 +0000 (UTC)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <726980622.46928.1306448928676.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6904) A baby step towards inter-version
 RPC communications
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039967#comment-13039967 ] 

Luke Lu commented on HADOOP-6904:
---------------------------------

bq. VersionedProtocol keeps the old API: getProtocolVersion for backwards compatibility purpose.

The patch added a new method getProtocolSignature to the VersionedProtocol interface, which breaks (compile and runtime) all existing code that uses VersionedProtocol.

Looking at the code added to mapreduce protocols, it seems that the default getProtocolSignature impl doesn't need the interface change i.e., the API breakage doesn't seem necessary in default cases. For people who want to map a protocol to a different impl than the default. An additional static ProtocolSignature.bind(String protocol, Type impl) method can be used to establish the mapping at class init time.

> A baby step towards inter-version RPC communications
> ----------------------------------------------------
>
>                 Key: HADOOP-6904
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6904
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.23.0
>
>         Attachments: majorMinorVersion.patch, majorMinorVersion1.patch, rpcCompatible-trunk.patch, rpcCompatible-trunk1.patch, rpcCompatible-trunk10.patch, rpcCompatible-trunk11.patch, rpcCompatible-trunk2.patch, rpcCompatible-trunk4.patch, rpcCompatible-trunk5.patch, rpcCompatible-trunk6.patch, rpcCompatible-trunk7.patch, rpcCompatible-trunk8.patch, rpcCompatible-trunk9.patch, rpcVersion.patch, rpcVersion1.patch
>
>
> Currently RPC communications in Hadoop is very strict. If a client has a different version from that of the server, a VersionMismatched exception is thrown and the client can not connect to the server. This force us to update both client and server all at once if a RPC protocol is changed. But sometime different versions do not mean the client & server are not compatible. It would be nice if we could relax this restriction and allows us to support inter-version communications.
> My idea is that DfsClient catches VersionMismatched exception when it connects to NameNode. It then checks if the client & the server is compatible. If yes, it sets the NameNode version in the dfs client and allows the client to continue talking to NameNode. Otherwise, rethrow the VersionMismatch exception.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15513-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 22:29:32 2011
Return-Path: <common-issues-return-15513-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E00EE685A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 22:29:31 +0000 (UTC)
Received: (qmail 97984 invoked by uid 500); 26 May 2011 22:29:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97945 invoked by uid 500); 26 May 2011 22:29:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97937 invoked by uid 99); 26 May 2011 22:29:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 22:29:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 22:29:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 96B0DE1FBE
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 22:28:48 +0000 (UTC)
Date: Thu, 26 May 2011 22:28:48 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1884034450.46926.1306448928613.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6255?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-6255:
------------------------------

    Attachment: HADOOP-6255-branch-0.20-security-13.patch

Minor bugfix in hadoop-setup-single-node.sh to work on correctly in Debian.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-625=
5-branch-0.20-security-10.patch, HADOOP-6255-branch-0.20-security-11.patch,=
 HADOOP-6255-branch-0.20-security-12.patch, HADOOP-6255-branch-0.20-securit=
y-13.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.=
20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-=
branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HAD=
OOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.p=
atch, HADOOP-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-sec=
urity.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-10.=
patch, HADOOP-6255-common-trunk-11.patch, HADOOP-6255-common-trunk-2.patch,=
 HADOOP-6255-common-trunk-4.patch, HADOOP-6255-common-trunk-5.patch, HADOOP=
-6255-common-trunk-6.patch, HADOOP-6255-common-trunk-7.patch, HADOOP-6255-c=
ommon-trunk-8.patch, HADOOP-6255-common-trunk-9.patch, HADOOP-6255-common-t=
runk.patch, HADOOP-6255-hdfs-trunk-1.patch, HADOOP-6255-hdfs-trunk.patch, H=
ADOOP-6255-mapred-trunk-1.patch, HADOOP-6255-mapred-trunk-2.patch, HADOOP-6=
255-mapred-trunk.patch, HADOOP-6255.patch, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15514-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 22:31:33 2011
Return-Path: <common-issues-return-15514-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 33A8C6002
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 22:31:33 +0000 (UTC)
Received: (qmail 3042 invoked by uid 500); 26 May 2011 22:31:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3003 invoked by uid 500); 26 May 2011 22:31:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2968 invoked by uid 99); 26 May 2011 22:31:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 22:31:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 22:31:30 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7EDC6E1087
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 22:30:49 +0000 (UTC)
Date: Thu, 26 May 2011 22:30:49 +0000 (UTC)
From: "Eric Yang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1565923507.46982.1306449049516.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6255?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Yang updated HADOOP-6255:
------------------------------

    Attachment: HADOOP-6255-common-trunk-12.patch

Minor bug fixes:

* Removed hadoop-config.sh from /usr/sbin
* export environment to su -c in hadoop-setup-single-node.sh and hadoop-cre=
ate-user.sh
* Removed *.debian and *.redhat from /usr/sbin
* Renamed hadoop to hadoop-common, and hadoop-mapred to hadoop-mapreduce

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-625=
5-branch-0.20-security-10.patch, HADOOP-6255-branch-0.20-security-11.patch,=
 HADOOP-6255-branch-0.20-security-12.patch, HADOOP-6255-branch-0.20-securit=
y-13.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.=
20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-=
branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HAD=
OOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.p=
atch, HADOOP-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-sec=
urity.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-10.=
patch, HADOOP-6255-common-trunk-11.patch, HADOOP-6255-common-trunk-12.patch=
, HADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk-4.patch, HADOO=
P-6255-common-trunk-5.patch, HADOOP-6255-common-trunk-6.patch, HADOOP-6255-=
common-trunk-7.patch, HADOOP-6255-common-trunk-8.patch, HADOOP-6255-common-=
trunk-9.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.pat=
ch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-=
6255-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patc=
h, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15515-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 22:45:29 2011
Return-Path: <common-issues-return-15515-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 131B462B2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 22:45:29 +0000 (UTC)
Received: (qmail 12661 invoked by uid 500); 26 May 2011 22:45:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12618 invoked by uid 500); 26 May 2011 22:45:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12541 invoked by uid 99); 26 May 2011 22:45:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 22:45:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 22:45:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A8058E1464
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 22:44:47 +0000 (UTC)
Date: Thu, 26 May 2011 22:44:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1482218930.47011.1306449887684.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D130=
39976#comment-13039976 ]=20

Hadoop QA commented on HADOOP-6255:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment=20
  http://issues.apache.org/jira/secure/attachment/12480593/HADOOP-6255-comm=
on-trunk-12.patch
  against trunk revision 1128003.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 27 new or modified tes=
ts.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of java=
c compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.=
3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number=
 of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compi=
le.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/5=
32//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Bu=
ild/532//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build=
/532//console

This message is automatically generated.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.203.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-625=
5-branch-0.20-security-10.patch, HADOOP-6255-branch-0.20-security-11.patch,=
 HADOOP-6255-branch-0.20-security-12.patch, HADOOP-6255-branch-0.20-securit=
y-13.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.=
20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-=
branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HAD=
OOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.p=
atch, HADOOP-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-sec=
urity.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-10.=
patch, HADOOP-6255-common-trunk-11.patch, HADOOP-6255-common-trunk-12.patch=
, HADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk-4.patch, HADOO=
P-6255-common-trunk-5.patch, HADOOP-6255-common-trunk-6.patch, HADOOP-6255-=
common-trunk-7.patch, HADOOP-6255-common-trunk-8.patch, HADOOP-6255-common-=
trunk-9.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.pat=
ch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-=
6255-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patc=
h, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15516-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 22:51:28 2011
Return-Path: <common-issues-return-15516-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5E3436591
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 22:51:28 +0000 (UTC)
Received: (qmail 20126 invoked by uid 500); 26 May 2011 22:51:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20035 invoked by uid 500); 26 May 2011 22:51:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20027 invoked by uid 99); 26 May 2011 22:51:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 22:51:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 22:51:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5D73EE1689
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 22:50:47 +0000 (UTC)
Date: Thu, 26 May 2011 22:50:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2009754766.47057.1306450247379.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1937737372.43422.1306361387389.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Resolved] (HADOOP-7330) The metrics source mbean
 implementation should return the attribute value instead of the object
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley resolved HADOOP-7330.
-----------------------------------

    Resolution: Fixed

I just committed this. Thanks, Luke!

> The metrics source mbean implementation should return the attribute value instead of the object
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7330
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7330
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.203.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.20.203.1
>
>         Attachments: hadoop-7330-jmx-v1.patch
>
>
> The MetricsSourceAdapter#getAttribute in 0.20.203 is returning the attribute object instead of the value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15517-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Thu May 26 23:10:29 2011
Return-Path: <common-issues-return-15517-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 82A8C6CA7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Thu, 26 May 2011 23:10:29 +0000 (UTC)
Received: (qmail 39673 invoked by uid 500); 26 May 2011 23:10:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39636 invoked by uid 500); 26 May 2011 23:10:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39625 invoked by uid 99); 26 May 2011 23:10:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 23:10:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 May 2011 23:10:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 34292E1AF5
	for <common-issues@hadoop.apache.org>; Thu, 26 May 2011 23:09:48 +0000 (UTC)
Date: Thu, 26 May 2011 23:09:48 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <489520204.47099.1306451388210.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6671) To use maven for hadoop common
 builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039986#comment-13039986 ] 

Tom White commented on HADOOP-6671:
-----------------------------------

The Jenkins scripts in http://svn.apache.org/repos/asf/hadoop/nightly/ will need updating too. To test the changes we could have a branch containing the Mavenized tree (i.e. with the patch from this issue applied), and a copy of the Jenkins nightly build job that uses a Maven version of the nightly script. For patch submission we can test the script manually rather than hooking it up to Jenkins across the board. We'd only commit this change when we are happy that the Jenkins jobs are working properly.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>         Attachments: HADOOP-6671.patch, HADOOP-6671b.patch, HADOOP-6671c.patch, HADOOP-6671d.patch, build.png, hadoop-commons-maven.patch, mvn-layout.sh, mvn-layout.sh, mvn-layout2.sh, mvn-layout2.sh
>
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15518-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 02:02:29 2011
Return-Path: <common-issues-return-15518-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 579C64ADA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 02:02:29 +0000 (UTC)
Received: (qmail 55588 invoked by uid 500); 27 May 2011 02:02:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55549 invoked by uid 500); 27 May 2011 02:02:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55541 invoked by uid 99); 27 May 2011 02:02:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 02:02:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 02:02:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CE07DE11D6
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 02:01:47 +0000 (UTC)
Date: Fri, 27 May 2011 02:01:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1410253917.47440.1306461707840.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1260350704.46296.1306435427358.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7334) test-patch should check for hard
 tabs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7334:
--------------------------------

    Attachment: hadoop-7334.txt

You're absolutely right about the first -c.. I was testing with a patch that had only one tab so I didn't notice :)

Something like checkstyle probably makes sense at some point, but right now we have so many violations that it would be a big change. Spaces vs tabs is one thing everyone agrees on and where there's never any question of "aesthetics" :) So let's start here and maybe some day ease in some checkstyle.

> test-patch should check for hard tabs
> -------------------------------------
>
>                 Key: HADOOP-7334
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7334
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7334.txt, hadoop-7334.txt
>
>
> Our coding guidelines say that hard tabs are disallowed in the Hadoop code, but they sometimes sneak in (there are about 280 in the common codebase at the moment).
> We should run a simple check for this in the test-patch process so it's harder for them to sneak in.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15519-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 02:04:31 2011
Return-Path: <common-issues-return-15519-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2F19B4409
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 02:04:31 +0000 (UTC)
Received: (qmail 56015 invoked by uid 500); 27 May 2011 02:04:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55987 invoked by uid 500); 27 May 2011 02:04:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55978 invoked by uid 99); 27 May 2011 02:04:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 02:04:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 02:04:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 584B9E1240
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 02:03:47 +0000 (UTC)
Date: Fri, 27 May 2011 02:03:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <288244867.47442.1306461827358.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1654473161.45482.1306422287498.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7333) Performance improvement in
 PureJavaCrc32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040028#comment-13040028 ] 

Todd Lipcon commented on HADOOP-7333:
-------------------------------------

Nice work checking this out on prior JVMs. I wonder if you use -XX:+AggressiveOpts on the earlier JVM if it would have the same performance increases?

> Performance improvement in PureJavaCrc32
> ----------------------------------------
>
>                 Key: HADOOP-7333
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7333
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>         Environment: Linux x64
>            Reporter: Eric Caspole
>            Assignee: Eric Caspole
>            Priority: Minor
>         Attachments: HADOOP-7333.patch, c7333_20110526.patch
>
>
> I would like to propose a small patch to 
>   org.apache.hadoop.util.PureJavaCrc32.update(byte[] b, int off, int len)
> Currently the method stores the intermediate result back into the data member "crc." I noticed this method gets
> inlined into DataChecksum.update() and that method appears as one of the hotter methods in a simple hprof profile collected while running terasort and gridmix.
> If the code is modified to save the temporary result into a local and just once store the final result back into the data member, it results in slightly more efficient hotspot codegen.
> I tested this change using the the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32 on a variety of linux x64 AMD and Intel multi-socket and multi-core systems I have available to test.
> The patch removes several stores of the intermediate result to memory yielding a 0%-10% speedup in the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32.
>  
> If you use a debug hotspot JVM with -XX:+PrintOptoAssembly, you can see the intermediate stores such as:
> 414     movq    R9, [rsp + #24] # spill
> 419     movl    [R9 + #12 (8-bit)], RDX # int ! Field PureJavaCrc32.crc
> 41d     xorl    R10, RDX        # int
> The patch results in just one final store of the fully computed value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15520-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 02:08:30 2011
Return-Path: <common-issues-return-15520-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8DEB7446B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 02:08:30 +0000 (UTC)
Received: (qmail 58081 invoked by uid 500); 27 May 2011 02:08:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58049 invoked by uid 500); 27 May 2011 02:08:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58041 invoked by uid 99); 27 May 2011 02:08:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 02:08:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 02:08:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5BF6FE139D
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 02:07:47 +0000 (UTC)
Date: Fri, 27 May 2011 02:07:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <770884625.47452.1306462067373.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1260350704.46296.1306435427358.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7334) test-patch should check for hard
 tabs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040029#comment-13040029 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7334:
------------------------------------------------

Nice work!  We should have this long time ago.

BTW, do you want to also check 80 characters per line?  But it is arguably if it is the rule we want to enforce.

> test-patch should check for hard tabs
> -------------------------------------
>
>                 Key: HADOOP-7334
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7334
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7334.txt, hadoop-7334.txt
>
>
> Our coding guidelines say that hard tabs are disallowed in the Hadoop code, but they sometimes sneak in (there are about 280 in the common codebase at the moment).
> We should run a simple check for this in the test-patch process so it's harder for them to sneak in.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15521-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 02:16:29 2011
Return-Path: <common-issues-return-15521-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 03726498C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 02:16:29 +0000 (UTC)
Received: (qmail 62533 invoked by uid 500); 27 May 2011 02:16:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62497 invoked by uid 500); 27 May 2011 02:16:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62488 invoked by uid 99); 27 May 2011 02:16:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 02:16:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 02:16:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8DA7AE159F
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 02:15:47 +0000 (UTC)
Date: Fri, 27 May 2011 02:15:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2119411263.47466.1306462547576.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1654473161.45482.1306422287498.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7333) Performance improvement in
 PureJavaCrc32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040032#comment-13040032 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7333:
------------------------------------------------

> ... I wonder if you use -XX:+AggressiveOpts on the earlier JVM if it would have the same performance increases?

By comparing the following with (2.1), it seems both columns have some better numbers but the patch still degrades the performance.

-----
h3. 2.3) Linux 2.6.18-53.1.13.el5 - Java 1.6.0_05-b13 with -XX:+AggressiveOpts
$ java -XX:+AggressiveOpts -cp hadoop-common-0.23.0-SNAPSHOT.jar:hadoop-common-test-0.23.0-SNAPSHOT.jar  org.apache.hadoop.util.TestPureJavaCrc32\$PerformanceTest
java.version = 1.6.0_05
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_05-b13
java.vm.version = 10.0-b19
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = i386
os.name = Linux
os.version = 2.6.18-53.1.13.el5

Performance Table (The unit is MB/sec)
|| Num Bytes ||    CRC32 || PureJavaCrc32 ||    H7333 ||
|          1 |     9.699 |        111.729 |   108.432 |
|          2 |    18.007 |        139.071 |   138.762 |
|          4 |    35.060 |        217.921 |   223.770 |
|          8 |    64.308 |        220.957 |   241.875 |
|         16 |   108.495 |        290.035 |   308.596 |
|         32 |   166.368 |        365.532 |   379.159 |
|         64 |   226.203 |        424.410 |   423.369 |
|        128 |   275.945 |        463.763 |   449.295 |
|        256 |   311.077 |        485.665 |   465.597 |
|        512 |   331.712 |        498.074 |   473.644 |
|       1024 |   340.871 |        504.308 |   475.610 |
|       2048 |   346.745 |        507.910 |   478.512 |
|       4096 |   351.109 |        508.981 |   480.016 |
|       8192 |   352.892 |        509.891 |   480.836 |
|      16384 |   350.394 |        508.394 |   480.766 |
|      32768 |   350.816 |        500.573 |   479.659 |
|      65536 |   348.351 |        500.105 |   473.086 |
|     131072 |   352.889 |        499.988 |   480.385 |
|     262144 |   352.796 |        499.779 |   480.421 |
|     524288 |   353.542 |        499.895 |   480.432 |
|    1048576 |   353.364 |        499.754 |   480.367 |
|    2097152 |   352.447 |        499.452 |   480.190 |
|    4194304 |   352.423 |        499.187 |   479.956 |
|    8388608 |   351.037 |        493.194 |   475.243 |
|   16777216 |   350.760 |        492.260 |   474.271 |

> Performance improvement in PureJavaCrc32
> ----------------------------------------
>
>                 Key: HADOOP-7333
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7333
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>         Environment: Linux x64
>            Reporter: Eric Caspole
>            Assignee: Eric Caspole
>            Priority: Minor
>         Attachments: HADOOP-7333.patch, c7333_20110526.patch
>
>
> I would like to propose a small patch to 
>   org.apache.hadoop.util.PureJavaCrc32.update(byte[] b, int off, int len)
> Currently the method stores the intermediate result back into the data member "crc." I noticed this method gets
> inlined into DataChecksum.update() and that method appears as one of the hotter methods in a simple hprof profile collected while running terasort and gridmix.
> If the code is modified to save the temporary result into a local and just once store the final result back into the data member, it results in slightly more efficient hotspot codegen.
> I tested this change using the the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32 on a variety of linux x64 AMD and Intel multi-socket and multi-core systems I have available to test.
> The patch removes several stores of the intermediate result to memory yielding a 0%-10% speedup in the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32.
>  
> If you use a debug hotspot JVM with -XX:+PrintOptoAssembly, you can see the intermediate stores such as:
> 414     movq    R9, [rsp + #24] # spill
> 419     movl    [R9 + #12 (8-bit)], RDX # int ! Field PureJavaCrc32.crc
> 41d     xorl    R10, RDX        # int
> The patch results in just one final store of the fully computed value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15522-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 02:16:31 2011
Return-Path: <common-issues-return-15522-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 00F4A49BE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 02:16:31 +0000 (UTC)
Received: (qmail 62793 invoked by uid 500); 27 May 2011 02:16:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62757 invoked by uid 500); 27 May 2011 02:16:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62749 invoked by uid 99); 27 May 2011 02:16:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 02:16:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 02:16:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 79B3CE159D
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 02:15:47 +0000 (UTC)
Date: Fri, 27 May 2011 02:15:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1193788051.47464.1306462547495.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1260350704.46296.1306435427358.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7334) test-patch should check for hard
 tabs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040031#comment-13040031 ] 

Todd Lipcon commented on HADOOP-7334:
-------------------------------------

IMO 80 chars per line is one of those things that should be kept in general, but if the occasional line flows over it's not a big deal. There's some times when it's much less readable to try to keep to it.

But this is where we get into religious wars ;-) I'm a 100-per-line devotee myself!

> test-patch should check for hard tabs
> -------------------------------------
>
>                 Key: HADOOP-7334
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7334
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7334.txt, hadoop-7334.txt
>
>
> Our coding guidelines say that hard tabs are disallowed in the Hadoop code, but they sometimes sneak in (there are about 280 in the common codebase at the moment).
> We should run a simple check for this in the test-patch process so it's harder for them to sneak in.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15523-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 02:20:28 2011
Return-Path: <common-issues-return-15523-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C27594AC1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 02:20:28 +0000 (UTC)
Received: (qmail 65836 invoked by uid 500); 27 May 2011 02:20:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65804 invoked by uid 500); 27 May 2011 02:20:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65796 invoked by uid 99); 27 May 2011 02:20:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 02:20:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 02:20:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 83927E16A9
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 02:19:47 +0000 (UTC)
Date: Fri, 27 May 2011 02:19:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1614742603.47477.1306462787535.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1260350704.46296.1306435427358.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7334) test-patch should check for hard
 tabs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040034#comment-13040034 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-7334:
------------------------------------------------

Provided that most monitors are wide-screen nowadays, the 80-per-line rule is probably out-dated.

> test-patch should check for hard tabs
> -------------------------------------
>
>                 Key: HADOOP-7334
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7334
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7334.txt, hadoop-7334.txt
>
>
> Our coding guidelines say that hard tabs are disallowed in the Hadoop code, but they sometimes sneak in (there are about 280 in the common codebase at the moment).
> We should run a simple check for this in the test-patch process so it's harder for them to sneak in.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15524-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 02:28:30 2011
Return-Path: <common-issues-return-15524-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C47D64C1D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 02:28:30 +0000 (UTC)
Received: (qmail 81589 invoked by uid 500); 27 May 2011 02:28:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81560 invoked by uid 500); 27 May 2011 02:28:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81552 invoked by uid 99); 27 May 2011 02:28:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 02:28:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 02:28:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 83D15E188B
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 02:27:47 +0000 (UTC)
Date: Fri, 27 May 2011 02:27:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <369440082.47490.1306463267536.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1654473161.45482.1306422287498.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7333) Performance improvement in
 PureJavaCrc32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040036#comment-13040036 ] 

Todd Lipcon commented on HADOOP-7333:
-------------------------------------

So, I guess the question is: are we OK with trading off a 5-10% performance hit on older JVMs for a 10-20% performance gain on newer ones?

I think yes.

> Performance improvement in PureJavaCrc32
> ----------------------------------------
>
>                 Key: HADOOP-7333
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7333
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>         Environment: Linux x64
>            Reporter: Eric Caspole
>            Assignee: Eric Caspole
>            Priority: Minor
>         Attachments: HADOOP-7333.patch, c7333_20110526.patch
>
>
> I would like to propose a small patch to 
>   org.apache.hadoop.util.PureJavaCrc32.update(byte[] b, int off, int len)
> Currently the method stores the intermediate result back into the data member "crc." I noticed this method gets
> inlined into DataChecksum.update() and that method appears as one of the hotter methods in a simple hprof profile collected while running terasort and gridmix.
> If the code is modified to save the temporary result into a local and just once store the final result back into the data member, it results in slightly more efficient hotspot codegen.
> I tested this change using the the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32 on a variety of linux x64 AMD and Intel multi-socket and multi-core systems I have available to test.
> The patch removes several stores of the intermediate result to memory yielding a 0%-10% speedup in the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32.
>  
> If you use a debug hotspot JVM with -XX:+PrintOptoAssembly, you can see the intermediate stores such as:
> 414     movq    R9, [rsp + #24] # spill
> 419     movl    [R9 + #12 (8-bit)], RDX # int ! Field PureJavaCrc32.crc
> 41d     xorl    R10, RDX        # int
> The patch results in just one final store of the fully computed value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15525-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 03:45:34 2011
Return-Path: <common-issues-return-15525-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B531C42FE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 03:45:34 +0000 (UTC)
Received: (qmail 35271 invoked by uid 500); 27 May 2011 03:45:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35054 invoked by uid 500); 27 May 2011 03:45:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35034 invoked by uid 99); 27 May 2011 03:45:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 03:45:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 03:45:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A5BC8E1ABB
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 03:44:47 +0000 (UTC)
Date: Fri, 27 May 2011 03:44:47 +0000 (UTC)
From: "ramkrishna.s.vasudevan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <85369923.47686.1306467887675.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <282223217.18947.1304436003652.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7256) Resource leak during failure
 scenario of closing of resources.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040044#comment-13040044 ] 

ramkrishna.s.vasudevan commented on HADOOP-7256:
------------------------------------------------

Hi Todd,

Thanks for the comments.  When I had submitted this issue my IDE had the wrong formatter. Now I have changed them.

I will fix all your comments and resubmit it and also will take care this formatting and check box problem doesnt come here then.

> Resource leak during failure scenario of closing of resources. 
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7256
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7256
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2, 0.21.0
>            Reporter: ramkrishna.s.vasudevan
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7256-patch-1.patch
>
>   Original Estimate: 8h
>  Remaining Estimate: 8h
>
> Problem Statement:
> ===============
> There are chances of resource leak and stream not getting closed 
> Take the case when after copying data we try to close the Input and output stream followed by closing of the socket.
> Suppose an exception occurs while closing the input stream(due to runtime exception) then the subsequent operations of closing the output stream and socket may not happen and there is a chance of resource leak.
> Scenario 
> =======
> During long run of map reduce jobs, the copyFromLocalFile() api is getting called.
> Here we found some exceptions happening. As a result of this we found the lsof value raising leading to resource leak.
> Solution:
> =======
> While doing a close operation of any resource catch the RuntimeException also rather than catching the IOException alone.
> Additionally there are places where we try to close a resource in the catch block.
> If this close fails, we just throw and come out of the current flow.
> In order to avoid this, we can carry out the close operation in the finally block.
> Probable reasons for getting RunTimeExceptions:
> =====================================
> We may get runtime exception from customised hadoop streams like FSDataOutputStream.close() . So better to handle RunTimeExceptions also.
>  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15526-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 04:27:33 2011
Return-Path: <common-issues-return-15526-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1106842CE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 04:27:33 +0000 (UTC)
Received: (qmail 59251 invoked by uid 500); 27 May 2011 04:27:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59015 invoked by uid 500); 27 May 2011 04:27:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58999 invoked by uid 99); 27 May 2011 04:27:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 04:27:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 04:27:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 91F24E1461
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 04:26:47 +0000 (UTC)
Date: Fri, 27 May 2011 04:26:47 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1834072493.47727.1306470407592.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1260350704.46296.1306435427358.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7334) test-patch should check for hard
 tabs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040050#comment-13040050 ] 

Tom White commented on HADOOP-7334:
-----------------------------------

> Do you think we have a need to do something like checkstyle in the future?

We have long had a checkstyle target in ant, and plots of checkstyle violations over time (see https://builds.apache.org/job/Hadoop-Common-trunk/), but unfortunately we haven't enforced the rule that the number may not increase (unlike javadoc or findbugs warnings for example).

Another way of implementing this JIRA would be to enable such a rule (perhaps with a weaker set of checkstyle rules than the current set, e.g. drop the 80-per-line rule).

> test-patch should check for hard tabs
> -------------------------------------
>
>                 Key: HADOOP-7334
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7334
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7334.txt, hadoop-7334.txt
>
>
> Our coding guidelines say that hard tabs are disallowed in the Hadoop code, but they sometimes sneak in (there are about 280 in the common codebase at the moment).
> We should run a simple check for this in the test-patch process so it's harder for them to sneak in.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15527-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 06:19:33 2011
Return-Path: <common-issues-return-15527-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2B55B4A3F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 06:19:33 +0000 (UTC)
Received: (qmail 40505 invoked by uid 500); 27 May 2011 06:19:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40465 invoked by uid 500); 27 May 2011 06:19:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40341 invoked by uid 99); 27 May 2011 06:19:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 06:19:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 06:19:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8E155E2210
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 06:18:47 +0000 (UTC)
Date: Fri, 27 May 2011 06:18:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1144203721.47874.1306477127578.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7284:
---------------------------------

    Attachment: trash8.patch

Updated patch.
Uses videwfs's config to set homedir.

Note I have retained the signature of
trashShell()'s original signature and added
an overloaded trashShell so that the 
tests in the hdfs project do not need to be changed.

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch, trash8.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15528-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 06:21:32 2011
Return-Path: <common-issues-return-15528-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7129B4C71
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 06:21:32 +0000 (UTC)
Received: (qmail 42155 invoked by uid 500); 27 May 2011 06:21:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41739 invoked by uid 500); 27 May 2011 06:21:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41727 invoked by uid 99); 27 May 2011 06:21:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 06:21:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 06:21:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5885FE2286
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 06:20:47 +0000 (UTC)
Date: Fri, 27 May 2011 06:20:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <622191463.47879.1306477247359.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7284:
---------------------------------

    Status: Patch Available  (was: Reopened)

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch, trash8.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15529-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 06:41:32 2011
Return-Path: <common-issues-return-15529-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EBF634444
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 06:41:31 +0000 (UTC)
Received: (qmail 55642 invoked by uid 500); 27 May 2011 06:41:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55527 invoked by uid 500); 27 May 2011 06:41:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55505 invoked by uid 99); 27 May 2011 06:41:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 06:41:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 06:41:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5B6A6E285E
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 06:40:47 +0000 (UTC)
Date: Fri, 27 May 2011 06:40:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <673579404.47930.1306478447365.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040086#comment-13040086 ] 

Hadoop QA commented on HADOOP-7284:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480629/trash8.patch
  against trunk revision 1128003.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 17 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.fs.viewfs.TestViewFsTrash

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/533//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/533//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/533//console

This message is automatically generated.

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch, trash8.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15530-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 06:43:32 2011
Return-Path: <common-issues-return-15530-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 411BE44AC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 06:43:32 +0000 (UTC)
Received: (qmail 56632 invoked by uid 500); 27 May 2011 06:43:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56585 invoked by uid 500); 27 May 2011 06:43:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56562 invoked by uid 99); 27 May 2011 06:43:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 06:43:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 06:43:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 089D2E2992
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 06:42:48 +0000 (UTC)
Date: Fri, 27 May 2011 06:42:48 +0000 (UTC)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1076600742.47948.1306478568031.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1260350704.46296.1306435427358.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7334) test-patch should check for hard
 tabs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040088#comment-13040088 ] 

Nigel Daley commented on HADOOP-7334:
-------------------------------------

Tom, that's a good idea.  Enable checkstyle with just the tab check and then we can grow the checks over time.  Start small and iterate.

> test-patch should check for hard tabs
> -------------------------------------
>
>                 Key: HADOOP-7334
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7334
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-7334.txt, hadoop-7334.txt
>
>
> Our coding guidelines say that hard tabs are disallowed in the Hadoop code, but they sometimes sneak in (there are about 280 in the common codebase at the moment).
> We should run a simple check for this in the test-patch process so it's harder for them to sneak in.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15532-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 08:02:34 2011
Return-Path: <common-issues-return-15532-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E80794B33
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 08:02:34 +0000 (UTC)
Received: (qmail 53103 invoked by uid 500); 27 May 2011 08:02:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52961 invoked by uid 500); 27 May 2011 08:02:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51521 invoked by uid 99); 27 May 2011 08:02:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 08:02:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 08:02:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A6807E2169
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 08:01:48 +0000 (UTC)
Date: Fri, 27 May 2011 08:01:48 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1053603540.48091.1306483308678.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7284:
---------------------------------

    Attachment: trash9.patch

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch, trash8.patch, trash9.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15533-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 08:02:35 2011
Return-Path: <common-issues-return-15533-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EDB364B49
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 08:02:34 +0000 (UTC)
Received: (qmail 53107 invoked by uid 500); 27 May 2011 08:02:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52984 invoked by uid 500); 27 May 2011 08:02:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52950 invoked by uid 99); 27 May 2011 08:02:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 08:02:33 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 08:02:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 09C41E2173
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 08:01:49 +0000 (UTC)
Date: Fri, 27 May 2011 08:01:49 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1930010661.48100.1306483309036.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7284:
---------------------------------

    Status: Patch Available  (was: Open)

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch, trash8.patch, trash9.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15531-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 08:02:35 2011
Return-Path: <common-issues-return-15531-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EF2FA4B4D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 08:02:34 +0000 (UTC)
Received: (qmail 53101 invoked by uid 500); 27 May 2011 08:02:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52737 invoked by uid 500); 27 May 2011 08:02:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51522 invoked by uid 99); 27 May 2011 08:02:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 08:02:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 08:02:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D5DB3E216F
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 08:01:48 +0000 (UTC)
Date: Fri, 27 May 2011 08:01:48 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <716146991.48096.1306483308873.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7284:
---------------------------------

    Status: Open  (was: Patch Available)

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch, trash8.patch, trash9.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15534-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 08:15:31 2011
Return-Path: <common-issues-return-15534-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0AFF447C7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 08:15:31 +0000 (UTC)
Received: (qmail 76144 invoked by uid 500); 27 May 2011 08:15:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75997 invoked by uid 500); 27 May 2011 08:15:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75983 invoked by uid 99); 27 May 2011 08:15:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 08:15:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 08:15:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 52AFCE26B2
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 08:14:47 +0000 (UTC)
Date: Fri, 27 May 2011 08:14:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1788442843.48137.1306484087335.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040123#comment-13040123 ] 

Hadoop QA commented on HADOOP-7284:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480632/trash9.patch
  against trunk revision 1128003.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 17 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.fs.viewfs.TestViewFsTrash

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/534//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/534//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/534//console

This message is automatically generated.

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch, trash8.patch, trash9.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15535-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 08:42:33 2011
Return-Path: <common-issues-return-15535-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E22B34721
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 08:42:32 +0000 (UTC)
Received: (qmail 23499 invoked by uid 500); 27 May 2011 08:42:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23415 invoked by uid 500); 27 May 2011 08:42:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23263 invoked by uid 99); 27 May 2011 08:42:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 08:42:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 08:42:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0AC7FE20C9
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 08:41:48 +0000 (UTC)
Date: Fri, 27 May 2011 08:41:48 +0000 (UTC)
From: "ramkrishna.s.vasudevan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1459910516.48207.1306485708040.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1738535981.26274.1305802127556.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7304) BackUpNameNode is using 100% CPU and
 not accepting any requests.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ramkrishna.s.vasudevan updated HADOOP-7304:
-------------------------------------------

    Hadoop Flags:   (was: [Reviewed])

> BackUpNameNode is using 100% CPU and not accepting any requests. 
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7304
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7304
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.23.0
>            Reporter: ramkrishna.s.vasudevan
>
> In our environment, Backup NameNode is using 100% CPU and not accepting any calls in 3days long run.
> Thread dump 
> "IPC Server Responder" daemon prio=10 tid=0x00007f86c41c6800 nid=0x3b2a runnable [0x00007f86ce579000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) 
> locked <0x00007f86d67e2a20> (a sun.nio.ch.Util$1) 
> locked <0x00007f86d67e2a08> (a java.util.Collections$UnmodifiableSet) 
> locked <0x00007f86d67e26a8> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at org.apache.hadoop.ipc.Server$Responder.run(Server.java:501) 
> Looks like same issue occurred in jetty also. http://jira.codehaus.org/browse/JETTY-937 
>  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15536-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 09:52:31 2011
Return-Path: <common-issues-return-15536-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 08BF34BD8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 09:52:31 +0000 (UTC)
Received: (qmail 24143 invoked by uid 500); 27 May 2011 09:52:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24114 invoked by uid 500); 27 May 2011 09:52:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24106 invoked by uid 99); 27 May 2011 09:52:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 09:52:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 09:52:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 94567E2895
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 09:51:47 +0000 (UTC)
Date: Fri, 27 May 2011 09:51:47 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1392842184.48304.1306489907603.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <504299282.44039.1306374047470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7331) Makes hadoop-daemon.sh to return 1
 if daemon processes did not get started
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040157#comment-13040157 ] 

Steve Loughran commented on HADOOP-7331:
----------------------------------------

I understand why there aren't any tests for this, but worry about the sleep 3; statement: why choose three seconds? It seems one of those only-works-on-some-machines sttings. 

> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7331
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7331
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7331.2.patch, HADOOP-7331.patch
>
>
> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15537-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 10:17:30 2011
Return-Path: <common-issues-return-15537-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CFF1D4383
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 10:17:30 +0000 (UTC)
Received: (qmail 62826 invoked by uid 500); 27 May 2011 10:17:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62679 invoked by uid 500); 27 May 2011 10:17:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62671 invoked by uid 99); 27 May 2011 10:17:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 10:17:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 10:17:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 92D09E2154
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 10:16:47 +0000 (UTC)
Date: Fri, 27 May 2011 10:16:47 +0000 (UTC)
From: "Nicholas Telford (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1302656896.48352.1306491407597.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7269) S3 Native should allow
 customizable file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040165#comment-13040165 ] 

Nicholas Telford commented on HADOOP-7269:
------------------------------------------

Totally missed that. As requested, tested:
{noformat}
$ ant -Dtestcase=Jets3tNativeS3FileSystemContractTest test
...
  [junit] Running org.apache.hadoop.fs.s3native.Jets3tNativeS3FileSystemContractTest
  [junit] Tests run: 37, Failures: 0, Errors: 0, Time elapsed: 155.312 sec
...
BUILD SUCCESSFUL
Total time: 3 minutes 2 seconds
{noformat}

I ran this before and after applying the patch, the results were the same.

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: HADOOP-7269-S3-metadata-001.diff, HADOOP-7269-S3-metadata-002.diff, HADOOP-7269-S3-metadata-003.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15538-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 10:50:29 2011
Return-Path: <common-issues-return-15538-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3600B41A9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 10:50:29 +0000 (UTC)
Received: (qmail 6664 invoked by uid 500); 27 May 2011 10:50:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6636 invoked by uid 500); 27 May 2011 10:50:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6627 invoked by uid 99); 27 May 2011 10:50:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 10:50:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 10:50:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F140AE2D74
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 10:49:47 +0000 (UTC)
Date: Fri, 27 May 2011 10:49:47 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1448025795.48412.1306493387985.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6941) Support non-SUN JREs in
 UserGroupInformation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040178#comment-13040178 ] 

Steve Loughran commented on HADOOP-6941:
----------------------------------------

SteveW is now ex-IBM, so probably can't check this. If the patch doesnt break sun JVM, someone with the IBM JVM will have to test it -even if they can't build it on that JVM.

> Support non-SUN JREs in UserGroupInformation
> --------------------------------------------
>
>                 Key: HADOOP-6941
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6941
>             Project: Hadoop Common
>          Issue Type: Bug
>         Environment: SLES 11, Apache Harmony 6 and SLES 11, IBM Java 6
>            Reporter: Stephen Watt
>            Assignee: Stephen Watt
>             Fix For: 0.21.1, 0.22.0, 0.23.0
>
>         Attachments: HADOOP-6941.patch, hadoop-6941.patch
>
>
> Attempting to format the namenode or attempting to start Hadoop using Apache Harmony or the IBM Java JREs results in the following exception:
> 10/09/07 16:35:05 ERROR namenode.NameNode: java.lang.NoClassDefFoundError: com.sun.security.auth.UnixPrincipal
> 	at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:223)
> 	at java.lang.J9VMInternals.initializeImpl(Native Method)
> 	at java.lang.J9VMInternals.initialize(J9VMInternals.java:200)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setConfigurationParameters(FSNamesystem.java:420)
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:391)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1240)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1348)
> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368)
> Caused by: java.lang.ClassNotFoundException: com.sun.security.auth.UnixPrincipal
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:421)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:652)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:346)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:618)
> 	... 8 more
> This is a negative regression as previous versions of Hadoop worked with these JREs

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15539-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 11:16:28 2011
Return-Path: <common-issues-return-15539-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B606C4276
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 11:16:28 +0000 (UTC)
Received: (qmail 45956 invoked by uid 500); 27 May 2011 11:16:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45887 invoked by uid 500); 27 May 2011 11:16:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45877 invoked by uid 99); 27 May 2011 11:16:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 11:16:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 11:16:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7E799E264C
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 11:15:47 +0000 (UTC)
Date: Fri, 27 May 2011 11:15:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1573064541.48453.1306494947514.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1093348183.29034.1305849887377.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7312) core-default.xml lists
 configuration version as 0.21
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040187#comment-13040187 ] 

Hudson commented on HADOOP-7312:
--------------------------------

Integrated in Hadoop-Common-trunk #701 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/701/])
    HADOOP-7312. Update value of hadoop.common.configuration.version. Contributed by Harsh J Chouraria.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1128003
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/core-default.xml


> core-default.xml lists configuration version as 0.21
> ----------------------------------------------------
>
>                 Key: HADOOP-7312
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7312
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Harsh J Chouraria
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7312-22.r1.diff, HADOOP-7312-23.r1.diff
>
>
> This key was added in HADOOP-6233, though appears unused. I suppose it's somewhat useful to try to diagnose if someone has old versions of core-default.xml on the classpath.
> Either way it should probably be updated to say 0.22 in the branch and 0.23 in trunk.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15540-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 14:27:28 2011
Return-Path: <common-issues-return-15540-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E6C324137
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 14:27:28 +0000 (UTC)
Received: (qmail 12115 invoked by uid 500); 27 May 2011 14:27:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12026 invoked by uid 500); 27 May 2011 14:27:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12018 invoked by uid 99); 27 May 2011 14:27:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 14:27:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 14:27:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A3C47E201F
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 14:26:47 +0000 (UTC)
Date: Fri, 27 May 2011 14:26:47 +0000 (UTC)
From: "Eric Caspole (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2142804085.48758.1306506407667.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1654473161.45482.1306422287498.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7333) Performance improvement in
 PureJavaCrc32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040253#comment-13040253 ] 

Eric Caspole commented on HADOOP-7333:
--------------------------------------

I only tried the patch on 6u23 and 6u25. There have been many hotspot compiler optimizations introduced especially since 6u21 and my general feeling is that people who are still running 6u10 etc are more worried about stability in their exact environment vs performance.

> Performance improvement in PureJavaCrc32
> ----------------------------------------
>
>                 Key: HADOOP-7333
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7333
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>         Environment: Linux x64
>            Reporter: Eric Caspole
>            Assignee: Eric Caspole
>            Priority: Minor
>         Attachments: HADOOP-7333.patch, c7333_20110526.patch
>
>
> I would like to propose a small patch to 
>   org.apache.hadoop.util.PureJavaCrc32.update(byte[] b, int off, int len)
> Currently the method stores the intermediate result back into the data member "crc." I noticed this method gets
> inlined into DataChecksum.update() and that method appears as one of the hotter methods in a simple hprof profile collected while running terasort and gridmix.
> If the code is modified to save the temporary result into a local and just once store the final result back into the data member, it results in slightly more efficient hotspot codegen.
> I tested this change using the the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32 on a variety of linux x64 AMD and Intel multi-socket and multi-core systems I have available to test.
> The patch removes several stores of the intermediate result to memory yielding a 0%-10% speedup in the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32.
>  
> If you use a debug hotspot JVM with -XX:+PrintOptoAssembly, you can see the intermediate stores such as:
> 414     movq    R9, [rsp + #24] # spill
> 419     movl    [R9 + #12 (8-bit)], RDX # int ! Field PureJavaCrc32.crc
> 41d     xorl    R10, RDX        # int
> The patch results in just one final store of the fully computed value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15541-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 16:54:29 2011
Return-Path: <common-issues-return-15541-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DE1FD4C7C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 16:54:28 +0000 (UTC)
Received: (qmail 33686 invoked by uid 500); 27 May 2011 16:54:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33654 invoked by uid 500); 27 May 2011 16:54:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33645 invoked by uid 99); 27 May 2011 16:54:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 16:54:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 16:54:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8C88BE3AA6
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 16:53:47 +0000 (UTC)
Date: Fri, 27 May 2011 16:53:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1430888302.49062.1306515227572.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6255?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6255:
----------------------------------

       Resolution: Fixed
    Fix Version/s:     (was: 0.20.203.0)
                   0.23.0
                   0.20.204.0
     Hadoop Flags: [Incompatible change, Reviewed]
           Status: Resolved  (was: Patch Available)

I just committed this. Thanks, Eric!

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.204.0, 0.23.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-625=
5-branch-0.20-security-10.patch, HADOOP-6255-branch-0.20-security-11.patch,=
 HADOOP-6255-branch-0.20-security-12.patch, HADOOP-6255-branch-0.20-securit=
y-13.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.=
20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-=
branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HAD=
OOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.p=
atch, HADOOP-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-sec=
urity.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-10.=
patch, HADOOP-6255-common-trunk-11.patch, HADOOP-6255-common-trunk-12.patch=
, HADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk-4.patch, HADOO=
P-6255-common-trunk-5.patch, HADOOP-6255-common-trunk-6.patch, HADOOP-6255-=
common-trunk-7.patch, HADOOP-6255-common-trunk-8.patch, HADOOP-6255-common-=
trunk-9.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.pat=
ch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-=
6255-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patc=
h, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15542-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 16:56:31 2011
Return-Path: <common-issues-return-15542-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DA7B14D0C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 16:56:31 +0000 (UTC)
Received: (qmail 38120 invoked by uid 500); 27 May 2011 16:56:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38092 invoked by uid 500); 27 May 2011 16:56:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38084 invoked by uid 99); 27 May 2011 16:56:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 16:56:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 16:56:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 73129E3B69
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 16:55:47 +0000 (UTC)
Date: Fri, 27 May 2011 16:55:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <290744278.49093.1306515347468.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D130=
40319#comment-13040319 ]=20

Hudson commented on HADOOP-6255:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #625 (See [https://builds.apache.o=
rg/hudson/job/Hadoop-Common-trunk-Commit/625/])
    HADOOP-6255. Create RPM and Debian packages for common. Changes deploym=
ent
layout to be consistent across the binary tgz, rpm, and deb. Adds setup
scripts for easy one node cluster configuration and user creation.
(Eric Yang via omalley)

omalley : http://svn.apache.org/viewcvs.cgi/?root=3DApache-SVN&view=3Drev&r=
ev=3D1128385
Files :=20
* /hadoop/common/trunk/src/packages/deb/hadoop.control/postinst
* /hadoop/common/trunk/src/docs/cn/src/documentation/content/xdocs/mapred_t=
utorial.xml
* /hadoop/common/trunk/bin/stop-all.sh
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/packages/rpm/init.d/hadoop-tasktracker
* /hadoop/common/trunk/src/packages/hadoop-setup-single-node.sh
* /hadoop/common/trunk/src/test/system/c++/runAs/runAs.c
* /hadoop/common/trunk/bin/hadoop-daemon.sh
* /hadoop/common/trunk/src/packages/rpm/spec/hadoop.spec
* /hadoop/common/trunk/src/packages/deb/hadoop.control/control
* /hadoop/common/trunk/src/packages/deb/hadoop.control/prerm
* /hadoop/common/trunk/bin/hadoop-config.sh
* /hadoop/common/trunk/src/packages/templates
* /hadoop/common/trunk/src/native/Makefile.am
* /hadoop/common/trunk/src/docs/src/documentation/content/xdocs/cluster_set=
up.xml
* /hadoop/common/trunk/src/packages/deb/init.d/hadoop-namenode
* /hadoop/common/trunk/src/packages/templates/conf
* /hadoop/common/trunk/src/packages/deb
* /hadoop/common/trunk/src/packages/deb/hadoop.control/preinst
* /hadoop/common/trunk/ivy/libraries.properties
* /hadoop/common/trunk/src/docs/cn/src/documentation/content/xdocs/cluster_=
setup.xml
* /hadoop/common/trunk/bin/hadoop
* /hadoop/common/trunk/src/packages/deb/init.d/hadoop-tasktracker
* /hadoop/common/trunk/src/test/system/c++/runAs/configure
* /hadoop/common/trunk/src/native/src/org/apache/hadoop/io/compress/zlib/Ma=
kefile.am
* /hadoop/common/trunk/bin/slaves.sh
* /hadoop/common/trunk/src/packages/deb/init.d/hadoop-jobtracker
* /hadoop/common/trunk/src/packages
* /hadoop/common/trunk/src/packages/templates/conf/core-site.xml
* /hadoop/common/trunk/src/packages/rpm/init.d/hadoop-namenode
* /hadoop/common/trunk/src/docs/cn/src/documentation/content/xdocs/hod_admi=
n_guide.xml
* /hadoop/common/trunk/bin/rcc
* /hadoop/common/trunk/src/packages/deb/init.d
* /hadoop/common/trunk/src/docs/src/documentation/content/xdocs/site.xml
* /hadoop/common/trunk/src/docs/src/documentation/content/xdocs/single_node=
_setup.xml
* /hadoop/common/trunk/src/native/packageNativeHadoop.sh
* /hadoop/common/trunk/src/packages/deb/init.d/hadoop-datanode
* /hadoop/common/trunk/src/test/system/c++/runAs/runAs.h.in
* /hadoop/common/trunk/src/packages/update-hadoop-env.sh
* /hadoop/common/trunk/src/packages/rpm/init.d/hadoop-jobtracker
* /hadoop/common/trunk/src/packages/rpm/spec
* /hadoop/common/trunk/src/docs/src/documentation/content/xdocs/commands_ma=
nual.xml
* /hadoop/common/trunk/src/docs/cn/src/documentation/content/xdocs/quicksta=
rt.xml
* /hadoop/common/trunk/bin/hadoop-daemons.sh
* /hadoop/common/trunk/conf/hadoop-env.sh.template
* /hadoop/common/trunk/ivy.xml
* /hadoop/common/trunk/build.xml
* /hadoop/common/trunk/src/packages/hadoop-setup-hdfs.sh
* /hadoop/common/trunk/src/packages/deb/hadoop.control
* /hadoop/common/trunk/src/docs/cn/src/documentation/content/xdocs/commands=
_manual.xml
* /hadoop/common/trunk/src/packages/deb/hadoop.control/postrm
* /hadoop/common/trunk/src/docs/src/documentation/content/xdocs/deployment_=
layout.xml
* /hadoop/common/trunk/src/test/system/java/org/apache/hadoop/test/system/p=
rocess/HadoopDaemonRemoteCluster.java
* /hadoop/common/trunk/src/packages/rpm/init.d
* /hadoop/common/trunk/bin/start-all.sh
* /hadoop/common/trunk/src/test/system/c++/runAs/configure.ac
* /hadoop/common/trunk/src/packages/hadoop-setup-conf.sh
* /hadoop/common/trunk/src/packages/deb/hadoop.control/conffile
* /hadoop/common/trunk/src/docs/cn/src/documentation/content/xdocs/streamin=
g.xml
* /hadoop/common/trunk/src/packages/hadoop-create-user.sh
* /hadoop/common/trunk/src/packages/rpm
* /hadoop/common/trunk/src/native/lib/Makefile.am
* /hadoop/common/trunk/src/packages/rpm/init.d/hadoop-datanode


> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.204.0, 0.23.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-625=
5-branch-0.20-security-10.patch, HADOOP-6255-branch-0.20-security-11.patch,=
 HADOOP-6255-branch-0.20-security-12.patch, HADOOP-6255-branch-0.20-securit=
y-13.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.=
20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-=
branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HAD=
OOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.p=
atch, HADOOP-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-sec=
urity.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-10.=
patch, HADOOP-6255-common-trunk-11.patch, HADOOP-6255-common-trunk-12.patch=
, HADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk-4.patch, HADOO=
P-6255-common-trunk-5.patch, HADOOP-6255-common-trunk-6.patch, HADOOP-6255-=
common-trunk-7.patch, HADOOP-6255-common-trunk-8.patch, HADOOP-6255-common-=
trunk-9.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.pat=
ch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-=
6255-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patc=
h, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15543-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 17:04:29 2011
Return-Path: <common-issues-return-15543-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 03303452F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 17:04:29 +0000 (UTC)
Received: (qmail 50081 invoked by uid 500); 27 May 2011 17:04:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50051 invoked by uid 500); 27 May 2011 17:04:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50039 invoked by uid 99); 27 May 2011 17:04:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 17:04:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 17:04:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9ED80E3E98
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 17:03:47 +0000 (UTC)
Date: Fri, 27 May 2011 17:03:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <243847304.49148.1306515827647.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D130=
40326#comment-13040326 ]=20

Tsz Wo (Nicholas), SZE commented on HADOOP-6255:
------------------------------------------------

Great job!  We finally have RPM in Apache Hadoop.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.204.0, 0.23.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-625=
5-branch-0.20-security-10.patch, HADOOP-6255-branch-0.20-security-11.patch,=
 HADOOP-6255-branch-0.20-security-12.patch, HADOOP-6255-branch-0.20-securit=
y-13.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.=
20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-=
branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HAD=
OOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.p=
atch, HADOOP-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-sec=
urity.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-10.=
patch, HADOOP-6255-common-trunk-11.patch, HADOOP-6255-common-trunk-12.patch=
, HADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk-4.patch, HADOO=
P-6255-common-trunk-5.patch, HADOOP-6255-common-trunk-6.patch, HADOOP-6255-=
common-trunk-7.patch, HADOOP-6255-common-trunk-8.patch, HADOOP-6255-common-=
trunk-9.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.pat=
ch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-=
6255-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patc=
h, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15544-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 17:04:32 2011
Return-Path: <common-issues-return-15544-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4C2934579
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 17:04:32 +0000 (UTC)
Received: (qmail 51798 invoked by uid 500); 27 May 2011 17:04:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51766 invoked by uid 500); 27 May 2011 17:04:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51758 invoked by uid 99); 27 May 2011 17:04:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 17:04:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 17:04:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DE9F0E3EBA
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 17:03:48 +0000 (UTC)
Date: Fri, 27 May 2011 17:03:48 +0000 (UTC)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1538852217.49180.1306515828908.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1654473161.45482.1306422287498.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7333) Performance improvement in
 PureJavaCrc32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040327#comment-13040327 ] 

Scott Carey commented on HADOOP-7333:
-------------------------------------

In particular, hotspot had some changes in how it analyzes loops (loop predicates) and how it looks at register allocation as a result.  I'm guessing that those changes are related.

It is now as fast as the fastest C implementations on the latest JVM.

I am a little disappointed that hotspot couldn't do this itself, but I suppose it makes sense to push changes to a member variable's memory address often because it can be read simultaneously.

> Performance improvement in PureJavaCrc32
> ----------------------------------------
>
>                 Key: HADOOP-7333
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7333
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>         Environment: Linux x64
>            Reporter: Eric Caspole
>            Assignee: Eric Caspole
>            Priority: Minor
>         Attachments: HADOOP-7333.patch, c7333_20110526.patch
>
>
> I would like to propose a small patch to 
>   org.apache.hadoop.util.PureJavaCrc32.update(byte[] b, int off, int len)
> Currently the method stores the intermediate result back into the data member "crc." I noticed this method gets
> inlined into DataChecksum.update() and that method appears as one of the hotter methods in a simple hprof profile collected while running terasort and gridmix.
> If the code is modified to save the temporary result into a local and just once store the final result back into the data member, it results in slightly more efficient hotspot codegen.
> I tested this change using the the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32 on a variety of linux x64 AMD and Intel multi-socket and multi-core systems I have available to test.
> The patch removes several stores of the intermediate result to memory yielding a 0%-10% speedup in the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32.
>  
> If you use a debug hotspot JVM with -XX:+PrintOptoAssembly, you can see the intermediate stores such as:
> 414     movq    R9, [rsp + #24] # spill
> 419     movl    [R9 + #12 (8-bit)], RDX # int ! Field PureJavaCrc32.crc
> 41d     xorl    R10, RDX        # int
> The patch results in just one final store of the fully computed value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15545-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 18:49:28 2011
Return-Path: <common-issues-return-15545-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 79348494B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 18:49:28 +0000 (UTC)
Received: (qmail 13232 invoked by uid 500); 27 May 2011 18:49:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13204 invoked by uid 500); 27 May 2011 18:49:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13196 invoked by uid 99); 27 May 2011 18:49:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 18:49:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 18:49:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5610DE3316
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 18:48:47 +0000 (UTC)
Date: Fri, 27 May 2011 18:48:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <162067005.49500.1306522127349.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1160101136.38164.1306200167436.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7325) hadoop command - do not accept
 class names starting with a hyphen
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon reassigned HADOOP-7325:
-----------------------------------

    Assignee: Brock Noland

> hadoop command - do not accept class names starting with a hyphen
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7325
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Brock Noland
>            Assignee: Brock Noland
>            Priority: Minor
>         Attachments: hadoop-illegal-class-name-0.patch
>
>
> If this is committed I will look at patches for hdfs and mapred.
> When teaching a good portion of the students in every single class execute:
> {code}
> $ hadoop -fs ls /
> {code}
> The -fs is passed directly to the JVM and the JVM fails to start:
> {code}
> $ ./bin/hadoop -fs ls /
> Unrecognized option: -fs
> Could not create the Java virtual machine.
> {code}
> Which is confusing and typically requires explanation. The attached patch improves that behavior:
> {code}
> $ ./bin/hadoop -fs ls /
> Error: No command named `-fs' was found. Perhaps you meant `hadoop fs'
> {code}
> The only risk I can see is if someone is abusing the implementation of hadoop command doing something like so:
> {code}
> $ ./bin/hadoop -Xmx1g  org.apache.hadoop.util.RunJar
> RunJar jarFile [mainClass] args...
> {code}
> The hadoop command does not appear to advertise allowing JVM options before the classname.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15546-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 18:51:29 2011
Return-Path: <common-issues-return-15546-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1559449C7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 18:51:29 +0000 (UTC)
Received: (qmail 17113 invoked by uid 500); 27 May 2011 18:51:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17086 invoked by uid 500); 27 May 2011 18:51:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17078 invoked by uid 99); 27 May 2011 18:51:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 18:51:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 18:51:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DD81DE3442
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 18:50:47 +0000 (UTC)
Date: Fri, 27 May 2011 18:50:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1010309377.49516.1306522247904.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1160101136.38164.1306200167436.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7325) hadoop command - do not accept class
 names starting with a hyphen
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-7325:
-----------------------------------

    Status: Patch Available  (was: Open)

Hitting 'submit patch' so Hudson will run the pre-commit hooks.

> hadoop command - do not accept class names starting with a hyphen
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7325
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Brock Noland
>            Assignee: Brock Noland
>            Priority: Minor
>         Attachments: hadoop-illegal-class-name-0.patch
>
>
> If this is committed I will look at patches for hdfs and mapred.
> When teaching a good portion of the students in every single class execute:
> {code}
> $ hadoop -fs ls /
> {code}
> The -fs is passed directly to the JVM and the JVM fails to start:
> {code}
> $ ./bin/hadoop -fs ls /
> Unrecognized option: -fs
> Could not create the Java virtual machine.
> {code}
> Which is confusing and typically requires explanation. The attached patch improves that behavior:
> {code}
> $ ./bin/hadoop -fs ls /
> Error: No command named `-fs' was found. Perhaps you meant `hadoop fs'
> {code}
> The only risk I can see is if someone is abusing the implementation of hadoop command doing something like so:
> {code}
> $ ./bin/hadoop -Xmx1g  org.apache.hadoop.util.RunJar
> RunJar jarFile [mainClass] args...
> {code}
> The hadoop command does not appear to advertise allowing JVM options before the classname.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15547-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 18:59:32 2011
Return-Path: <common-issues-return-15547-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E83604C74
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 18:59:31 +0000 (UTC)
Received: (qmail 29600 invoked by uid 500); 27 May 2011 18:59:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29576 invoked by uid 500); 27 May 2011 18:59:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29568 invoked by uid 99); 27 May 2011 18:59:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 18:59:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 18:59:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9FD41E375D
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 18:58:48 +0000 (UTC)
Date: Fri, 27 May 2011 18:58:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <209432576.49556.1306522728651.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1654473161.45482.1306422287498.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7333) Performance improvement in
 PureJavaCrc32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040387#comment-13040387 ] 

Todd Lipcon commented on HADOOP-7333:
-------------------------------------

Sounds like we're all on board. I'll commit momentarily.

> Performance improvement in PureJavaCrc32
> ----------------------------------------
>
>                 Key: HADOOP-7333
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7333
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>         Environment: Linux x64
>            Reporter: Eric Caspole
>            Assignee: Eric Caspole
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7333.patch, c7333_20110526.patch
>
>
> I would like to propose a small patch to 
>   org.apache.hadoop.util.PureJavaCrc32.update(byte[] b, int off, int len)
> Currently the method stores the intermediate result back into the data member "crc." I noticed this method gets
> inlined into DataChecksum.update() and that method appears as one of the hotter methods in a simple hprof profile collected while running terasort and gridmix.
> If the code is modified to save the temporary result into a local and just once store the final result back into the data member, it results in slightly more efficient hotspot codegen.
> I tested this change using the the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32 on a variety of linux x64 AMD and Intel multi-socket and multi-core systems I have available to test.
> The patch removes several stores of the intermediate result to memory yielding a 0%-10% speedup in the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32.
>  
> If you use a debug hotspot JVM with -XX:+PrintOptoAssembly, you can see the intermediate stores such as:
> 414     movq    R9, [rsp + #24] # spill
> 419     movl    [R9 + #12 (8-bit)], RDX # int ! Field PureJavaCrc32.crc
> 41d     xorl    R10, RDX        # int
> The patch results in just one final store of the fully computed value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15548-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 18:59:32 2011
Return-Path: <common-issues-return-15548-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 259E54C82
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 18:59:32 +0000 (UTC)
Received: (qmail 29837 invoked by uid 500); 27 May 2011 18:59:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29810 invoked by uid 500); 27 May 2011 18:59:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29751 invoked by uid 99); 27 May 2011 18:59:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 18:59:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 18:59:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B043BE375F
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 18:58:48 +0000 (UTC)
Date: Fri, 27 May 2011 18:58:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1744112265.49558.1306522728718.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1654473161.45482.1306422287498.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7333) Performance improvement in
 PureJavaCrc32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7333:
--------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

Committed to trunk. Thanks, Eric!

> Performance improvement in PureJavaCrc32
> ----------------------------------------
>
>                 Key: HADOOP-7333
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7333
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>         Environment: Linux x64
>            Reporter: Eric Caspole
>            Assignee: Eric Caspole
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7333.patch, c7333_20110526.patch
>
>
> I would like to propose a small patch to 
>   org.apache.hadoop.util.PureJavaCrc32.update(byte[] b, int off, int len)
> Currently the method stores the intermediate result back into the data member "crc." I noticed this method gets
> inlined into DataChecksum.update() and that method appears as one of the hotter methods in a simple hprof profile collected while running terasort and gridmix.
> If the code is modified to save the temporary result into a local and just once store the final result back into the data member, it results in slightly more efficient hotspot codegen.
> I tested this change using the the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32 on a variety of linux x64 AMD and Intel multi-socket and multi-core systems I have available to test.
> The patch removes several stores of the intermediate result to memory yielding a 0%-10% speedup in the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32.
>  
> If you use a debug hotspot JVM with -XX:+PrintOptoAssembly, you can see the intermediate stores such as:
> 414     movq    R9, [rsp + #24] # spill
> 419     movl    [R9 + #12 (8-bit)], RDX # int ! Field PureJavaCrc32.crc
> 41d     xorl    R10, RDX        # int
> The patch results in just one final store of the fully computed value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15549-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 19:05:31 2011
Return-Path: <common-issues-return-15549-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3439042DA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 19:05:31 +0000 (UTC)
Received: (qmail 32985 invoked by uid 500); 27 May 2011 19:05:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32957 invoked by uid 500); 27 May 2011 19:05:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32948 invoked by uid 99); 27 May 2011 19:05:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 19:05:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 19:05:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 82DC9E393A
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 19:04:47 +0000 (UTC)
Date: Fri, 27 May 2011 19:04:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1936308365.49580.1306523087532.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1160101136.38164.1306200167436.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7325) hadoop command - do not accept
 class names starting with a hyphen
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040390#comment-13040390 ] 

Hadoop QA commented on HADOOP-7325:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480193/hadoop-illegal-class-name-0.patch
  against trunk revision 1128385.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/535//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/535//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/535//console

This message is automatically generated.

> hadoop command - do not accept class names starting with a hyphen
> -----------------------------------------------------------------
>
>                 Key: HADOOP-7325
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7325
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Brock Noland
>            Assignee: Brock Noland
>            Priority: Minor
>         Attachments: hadoop-illegal-class-name-0.patch
>
>
> If this is committed I will look at patches for hdfs and mapred.
> When teaching a good portion of the students in every single class execute:
> {code}
> $ hadoop -fs ls /
> {code}
> The -fs is passed directly to the JVM and the JVM fails to start:
> {code}
> $ ./bin/hadoop -fs ls /
> Unrecognized option: -fs
> Could not create the Java virtual machine.
> {code}
> Which is confusing and typically requires explanation. The attached patch improves that behavior:
> {code}
> $ ./bin/hadoop -fs ls /
> Error: No command named `-fs' was found. Perhaps you meant `hadoop fs'
> {code}
> The only risk I can see is if someone is abusing the implementation of hadoop command doing something like so:
> {code}
> $ ./bin/hadoop -Xmx1g  org.apache.hadoop.util.RunJar
> RunJar jarFile [mainClass] args...
> {code}
> The hadoop command does not appear to advertise allowing JVM options before the classname.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15550-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 19:15:31 2011
Return-Path: <common-issues-return-15550-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B851F45FF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 19:15:31 +0000 (UTC)
Received: (qmail 37517 invoked by uid 500); 27 May 2011 19:15:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37468 invoked by uid 500); 27 May 2011 19:15:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37406 invoked by uid 99); 27 May 2011 19:15:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 19:15:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 19:15:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F3A94E3C82
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 19:14:47 +0000 (UTC)
Date: Fri, 27 May 2011 19:14:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1010720816.49615.1306523687994.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1654473161.45482.1306422287498.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7333) Performance improvement in
 PureJavaCrc32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040397#comment-13040397 ] 

Hudson commented on HADOOP-7333:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #626 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/626/])
    HADOOP-7333. Performance improvement in PureJavaCrc32. Contributed by Eric Caspole.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1128425
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/util/PureJavaCrc32.java


> Performance improvement in PureJavaCrc32
> ----------------------------------------
>
>                 Key: HADOOP-7333
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7333
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>         Environment: Linux x64
>            Reporter: Eric Caspole
>            Assignee: Eric Caspole
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7333.patch, c7333_20110526.patch
>
>
> I would like to propose a small patch to 
>   org.apache.hadoop.util.PureJavaCrc32.update(byte[] b, int off, int len)
> Currently the method stores the intermediate result back into the data member "crc." I noticed this method gets
> inlined into DataChecksum.update() and that method appears as one of the hotter methods in a simple hprof profile collected while running terasort and gridmix.
> If the code is modified to save the temporary result into a local and just once store the final result back into the data member, it results in slightly more efficient hotspot codegen.
> I tested this change using the the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32 on a variety of linux x64 AMD and Intel multi-socket and multi-core systems I have available to test.
> The patch removes several stores of the intermediate result to memory yielding a 0%-10% speedup in the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32.
>  
> If you use a debug hotspot JVM with -XX:+PrintOptoAssembly, you can see the intermediate stores such as:
> 414     movq    R9, [rsp + #24] # spill
> 419     movl    [R9 + #12 (8-bit)], RDX # int ! Field PureJavaCrc32.crc
> 41d     xorl    R10, RDX        # int
> The patch results in just one final store of the fully computed value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15551-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 21:08:31 2011
Return-Path: <common-issues-return-15551-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 00A654AF1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 21:08:31 +0000 (UTC)
Received: (qmail 45013 invoked by uid 500); 27 May 2011 21:08:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44980 invoked by uid 500); 27 May 2011 21:08:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44968 invoked by uid 99); 27 May 2011 21:08:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 21:08:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 21:08:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5B838E3E33
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 21:07:47 +0000 (UTC)
Date: Fri, 27 May 2011 21:07:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <314496369.49888.1306530467371.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7335) Force entropy to come from non-true
 random for tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Force entropy to come from non-true random for tests
----------------------------------------------------

                 Key: HADOOP-7335
                 URL: https://issues.apache.org/jira/browse/HADOOP-7335
             Project: Hadoop Common
          Issue Type: Improvement
          Components: build, test
    Affects Versions: 0.22.0
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
            Priority: Minor


Passing the system property {{-Djava.security.egd=file:///dev/urandom}} forces the JVM to seed its PRNG from non-true random (/dev/urandom) instead of the true random (/dev/random). This makes the tests run faster, since without it they often hang waiting for entropy while Jetty is initializing.

We should turn this on for the test targets by default, so developers/hudson boxes don't have to make this change system-wide or use workarounds like rngtools.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15552-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 21:44:28 2011
Return-Path: <common-issues-return-15552-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F16744F64
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 21:44:28 +0000 (UTC)
Received: (qmail 90727 invoked by uid 500); 27 May 2011 21:44:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90695 invoked by uid 500); 27 May 2011 21:44:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90684 invoked by uid 99); 27 May 2011 21:44:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 21:44:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 21:44:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6DCF3E3AF6
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 21:43:47 +0000 (UTC)
Date: Fri, 27 May 2011 21:43:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1586355296.49985.1306532627446.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <532896131.3523.1305130907394.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Assigned] (HADOOP-7276) Hadoop native builds fail on ARM
 due to -m32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins reassigned HADOOP-7276:
-----------------------------------

    Assignee: Trevor Robinson

+1 looks good. I also verified the x64 native build still works.  

> Hadoop native builds fail on ARM due to -m32
> --------------------------------------------
>
>                 Key: HADOOP-7276
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7276
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.21.0
>         Environment: $ gcc -v
> Using built-in specs.
> COLLECT_GCC=gcc
> COLLECT_LTO_WRAPPER=/usr/lib/arm-linux-gnueabi/gcc/arm-linux-gnueabi/4.5.2/lto-wrapper
> Target: arm-linux-gnueabi
> Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.5.2-8ubuntu4' --with-bugurl=file:///usr/share/doc/gcc-4.5/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.5 --enable-shared --enable-multiarch --with-multiarch-defaults=arm-linux-gnueabi --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib/arm-linux-gnueabi --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.5 --libdir=/usr/lib/arm-linux-gnueabi --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-plugin --enable-gold --enable-ld=default --with-plugin-ld=ld.gold --enable-objc-gc --disable-sjlj-exceptions --with-arch=armv7-a --with-float=softfp --with-fpu=vfpv3-d16 --with-mode=thumb --disable-werror --enable-checking=release --build=arm-linux-gnueabi --host=arm-linux-gnueabi --target=arm-linux-gnueabi
> Thread model: posix
> gcc version 4.5.2 (Ubuntu/Linaro 4.5.2-8ubuntu4)
> $ uname -a
> Linux panda0 2.6.38-1002-linaro-omap #3-Ubuntu SMP Fri Apr 15 14:00:54 UTC 2011 armv7l armv7l armv7l GNU/Linux
>            Reporter: Trevor Robinson
>            Assignee: Trevor Robinson
>         Attachments: hadoop-common-arm.patch
>
>
> The native build fails on machine targets where gcc does not support -m32. This is any target other than x86, SPARC, RS/6000, or PowerPC, such as ARM.
> $ ant -Dcompile.native=true
> ...
>      [exec] make  all-am
>      [exec] make[1]: Entering directory
> `/home/trobinson/dev/hadoop-common/build/native/Linux-arm-32'
>      [exec] /bin/bash ./libtool  --tag=CC   --mode=compile gcc
> -DHAVE_CONFIG_H -I. -I/home/trobinson/dev/hadoop-common/src/native
> -I/usr/lib/jvm/java-6-openjdk/include
> -I/usr/lib/jvm/java-6-openjdk/include/linux
> -I/home/trobinson/dev/hadoop-common/src/native/src
> -Isrc/org/apache/hadoop/io/compress/zlib
> -Isrc/org/apache/hadoop/security -Isrc/org/apache/hadoop/io/nativeio/
> -g -Wall -fPIC -O2 -m32 -g -O2 -MT ZlibCompressor.lo -MD -MP -MF
> .deps/ZlibCompressor.Tpo -c -o ZlibCompressor.lo `test -f
> 'src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c' || echo
> '/home/trobinson/dev/hadoop-common/src/native/'`src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
>      [exec] libtool: compile:  gcc -DHAVE_CONFIG_H -I.
> -I/home/trobinson/dev/hadoop-common/src/native
> -I/usr/lib/jvm/java-6-openjdk/include
> -I/usr/lib/jvm/java-6-openjdk/include/linux
> -I/home/trobinson/dev/hadoop-common/src/native/src
> -Isrc/org/apache/hadoop/io/compress/zlib
> -Isrc/org/apache/hadoop/security -Isrc/org/apache/hadoop/io/nativeio/
> -g -Wall -fPIC -O2 -m32 -g -O2 -MT ZlibCompressor.lo -MD -MP -MF
> .deps/ZlibCompressor.Tpo -c
> /home/trobinson/dev/hadoop-common/src/native/src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
>  -fPIC -DPIC -o .libs/ZlibCompressor.o
>      [exec] make[1]: Leaving directory
> `/home/trobinson/dev/hadoop-common/build/native/Linux-arm-32'
>      [exec] cc1: error: unrecognized command line option "-m32"
>      [exec] make[1]: *** [ZlibCompressor.lo] Error 1
>      [exec] make: *** [all] Error 2

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15553-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 21:44:29 2011
Return-Path: <common-issues-return-15553-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1E6BF4F6C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 21:44:29 +0000 (UTC)
Received: (qmail 90993 invoked by uid 500); 27 May 2011 21:44:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90928 invoked by uid 500); 27 May 2011 21:44:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90813 invoked by uid 99); 27 May 2011 21:44:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 21:44:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 21:44:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8A41FE3AFB
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 21:43:47 +0000 (UTC)
Date: Fri, 27 May 2011 21:43:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1802634104.49989.1306532627563.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <532896131.3523.1305130907394.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7276) Hadoop native builds fail on ARM
 due to -m32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040460#comment-13040460 ] 

Eli Collins commented on HADOOP-7276:
-------------------------------------

I don't see the findbugs warning locally. Don't think a test is needed since this to get the code compiling.

{noformat}
     [exec] 
     [exec] -1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
     [exec]                         Please justify why no new tests are needed for this patch.
     [exec]                         Also please list what manual steps were performed to verify this patch.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec] 
     [exec]     +1 system test framework.  The patch passed system test framework compile.
     [exec] 
{noformat}

> Hadoop native builds fail on ARM due to -m32
> --------------------------------------------
>
>                 Key: HADOOP-7276
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7276
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.21.0
>         Environment: $ gcc -v
> Using built-in specs.
> COLLECT_GCC=gcc
> COLLECT_LTO_WRAPPER=/usr/lib/arm-linux-gnueabi/gcc/arm-linux-gnueabi/4.5.2/lto-wrapper
> Target: arm-linux-gnueabi
> Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.5.2-8ubuntu4' --with-bugurl=file:///usr/share/doc/gcc-4.5/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.5 --enable-shared --enable-multiarch --with-multiarch-defaults=arm-linux-gnueabi --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib/arm-linux-gnueabi --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.5 --libdir=/usr/lib/arm-linux-gnueabi --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-plugin --enable-gold --enable-ld=default --with-plugin-ld=ld.gold --enable-objc-gc --disable-sjlj-exceptions --with-arch=armv7-a --with-float=softfp --with-fpu=vfpv3-d16 --with-mode=thumb --disable-werror --enable-checking=release --build=arm-linux-gnueabi --host=arm-linux-gnueabi --target=arm-linux-gnueabi
> Thread model: posix
> gcc version 4.5.2 (Ubuntu/Linaro 4.5.2-8ubuntu4)
> $ uname -a
> Linux panda0 2.6.38-1002-linaro-omap #3-Ubuntu SMP Fri Apr 15 14:00:54 UTC 2011 armv7l armv7l armv7l GNU/Linux
>            Reporter: Trevor Robinson
>            Assignee: Trevor Robinson
>         Attachments: hadoop-common-arm.patch
>
>
> The native build fails on machine targets where gcc does not support -m32. This is any target other than x86, SPARC, RS/6000, or PowerPC, such as ARM.
> $ ant -Dcompile.native=true
> ...
>      [exec] make  all-am
>      [exec] make[1]: Entering directory
> `/home/trobinson/dev/hadoop-common/build/native/Linux-arm-32'
>      [exec] /bin/bash ./libtool  --tag=CC   --mode=compile gcc
> -DHAVE_CONFIG_H -I. -I/home/trobinson/dev/hadoop-common/src/native
> -I/usr/lib/jvm/java-6-openjdk/include
> -I/usr/lib/jvm/java-6-openjdk/include/linux
> -I/home/trobinson/dev/hadoop-common/src/native/src
> -Isrc/org/apache/hadoop/io/compress/zlib
> -Isrc/org/apache/hadoop/security -Isrc/org/apache/hadoop/io/nativeio/
> -g -Wall -fPIC -O2 -m32 -g -O2 -MT ZlibCompressor.lo -MD -MP -MF
> .deps/ZlibCompressor.Tpo -c -o ZlibCompressor.lo `test -f
> 'src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c' || echo
> '/home/trobinson/dev/hadoop-common/src/native/'`src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
>      [exec] libtool: compile:  gcc -DHAVE_CONFIG_H -I.
> -I/home/trobinson/dev/hadoop-common/src/native
> -I/usr/lib/jvm/java-6-openjdk/include
> -I/usr/lib/jvm/java-6-openjdk/include/linux
> -I/home/trobinson/dev/hadoop-common/src/native/src
> -Isrc/org/apache/hadoop/io/compress/zlib
> -Isrc/org/apache/hadoop/security -Isrc/org/apache/hadoop/io/nativeio/
> -g -Wall -fPIC -O2 -m32 -g -O2 -MT ZlibCompressor.lo -MD -MP -MF
> .deps/ZlibCompressor.Tpo -c
> /home/trobinson/dev/hadoop-common/src/native/src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
>  -fPIC -DPIC -o .libs/ZlibCompressor.o
>      [exec] make[1]: Leaving directory
> `/home/trobinson/dev/hadoop-common/build/native/Linux-arm-32'
>      [exec] cc1: error: unrecognized command line option "-m32"
>      [exec] make[1]: *** [ZlibCompressor.lo] Error 1
>      [exec] make: *** [all] Error 2

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15554-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 21:48:31 2011
Return-Path: <common-issues-return-15554-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9B3EB45F9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 21:48:31 +0000 (UTC)
Received: (qmail 95694 invoked by uid 500); 27 May 2011 21:48:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95658 invoked by uid 500); 27 May 2011 21:48:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95650 invoked by uid 99); 27 May 2011 21:48:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 21:48:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 21:48:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9117CE3C48
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 21:47:47 +0000 (UTC)
Date: Fri, 27 May 2011 21:47:47 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <215166959.49995.1306532867591.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <532896131.3523.1305130907394.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7276) Hadoop native builds fail on ARM due
 to -m32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-7276:
--------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I've committed this to trunk. Thanks Trevor!

> Hadoop native builds fail on ARM due to -m32
> --------------------------------------------
>
>                 Key: HADOOP-7276
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7276
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.21.0
>         Environment: $ gcc -v
> Using built-in specs.
> COLLECT_GCC=gcc
> COLLECT_LTO_WRAPPER=/usr/lib/arm-linux-gnueabi/gcc/arm-linux-gnueabi/4.5.2/lto-wrapper
> Target: arm-linux-gnueabi
> Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.5.2-8ubuntu4' --with-bugurl=file:///usr/share/doc/gcc-4.5/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.5 --enable-shared --enable-multiarch --with-multiarch-defaults=arm-linux-gnueabi --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib/arm-linux-gnueabi --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.5 --libdir=/usr/lib/arm-linux-gnueabi --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-plugin --enable-gold --enable-ld=default --with-plugin-ld=ld.gold --enable-objc-gc --disable-sjlj-exceptions --with-arch=armv7-a --with-float=softfp --with-fpu=vfpv3-d16 --with-mode=thumb --disable-werror --enable-checking=release --build=arm-linux-gnueabi --host=arm-linux-gnueabi --target=arm-linux-gnueabi
> Thread model: posix
> gcc version 4.5.2 (Ubuntu/Linaro 4.5.2-8ubuntu4)
> $ uname -a
> Linux panda0 2.6.38-1002-linaro-omap #3-Ubuntu SMP Fri Apr 15 14:00:54 UTC 2011 armv7l armv7l armv7l GNU/Linux
>            Reporter: Trevor Robinson
>            Assignee: Trevor Robinson
>             Fix For: 0.23.0
>
>         Attachments: hadoop-common-arm.patch
>
>
> The native build fails on machine targets where gcc does not support -m32. This is any target other than x86, SPARC, RS/6000, or PowerPC, such as ARM.
> $ ant -Dcompile.native=true
> ...
>      [exec] make  all-am
>      [exec] make[1]: Entering directory
> `/home/trobinson/dev/hadoop-common/build/native/Linux-arm-32'
>      [exec] /bin/bash ./libtool  --tag=CC   --mode=compile gcc
> -DHAVE_CONFIG_H -I. -I/home/trobinson/dev/hadoop-common/src/native
> -I/usr/lib/jvm/java-6-openjdk/include
> -I/usr/lib/jvm/java-6-openjdk/include/linux
> -I/home/trobinson/dev/hadoop-common/src/native/src
> -Isrc/org/apache/hadoop/io/compress/zlib
> -Isrc/org/apache/hadoop/security -Isrc/org/apache/hadoop/io/nativeio/
> -g -Wall -fPIC -O2 -m32 -g -O2 -MT ZlibCompressor.lo -MD -MP -MF
> .deps/ZlibCompressor.Tpo -c -o ZlibCompressor.lo `test -f
> 'src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c' || echo
> '/home/trobinson/dev/hadoop-common/src/native/'`src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
>      [exec] libtool: compile:  gcc -DHAVE_CONFIG_H -I.
> -I/home/trobinson/dev/hadoop-common/src/native
> -I/usr/lib/jvm/java-6-openjdk/include
> -I/usr/lib/jvm/java-6-openjdk/include/linux
> -I/home/trobinson/dev/hadoop-common/src/native/src
> -Isrc/org/apache/hadoop/io/compress/zlib
> -Isrc/org/apache/hadoop/security -Isrc/org/apache/hadoop/io/nativeio/
> -g -Wall -fPIC -O2 -m32 -g -O2 -MT ZlibCompressor.lo -MD -MP -MF
> .deps/ZlibCompressor.Tpo -c
> /home/trobinson/dev/hadoop-common/src/native/src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
>  -fPIC -DPIC -o .libs/ZlibCompressor.o
>      [exec] make[1]: Leaving directory
> `/home/trobinson/dev/hadoop-common/build/native/Linux-arm-32'
>      [exec] cc1: error: unrecognized command line option "-m32"
>      [exec] make[1]: *** [ZlibCompressor.lo] Error 1
>      [exec] make: *** [all] Error 2

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15555-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 22:06:29 2011
Return-Path: <common-issues-return-15555-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9CFB44E93
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 22:06:29 +0000 (UTC)
Received: (qmail 10856 invoked by uid 500); 27 May 2011 22:06:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10780 invoked by uid 500); 27 May 2011 22:06:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10726 invoked by uid 99); 27 May 2011 22:06:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 22:06:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 22:06:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id CA643E313E
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 22:05:47 +0000 (UTC)
Date: Fri, 27 May 2011 22:05:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1765227103.50045.1306533947825.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <532896131.3523.1305130907394.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7276) Hadoop native builds fail on ARM
 due to -m32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040467#comment-13040467 ] 

Hudson commented on HADOOP-7276:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #627 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/627/])
    HADOOP-7276. Hadoop native builds fail on ARM due to -m32. Contributed by Trevor Robinson

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1128475
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/native/configure.ac
* /hadoop/common/trunk/src/native/Makefile.am
* /hadoop/common/trunk/src/native/src/org/apache/hadoop/io/compress/zlib/Makefile.am
* /hadoop/common/trunk/src/native/lib/Makefile.am


> Hadoop native builds fail on ARM due to -m32
> --------------------------------------------
>
>                 Key: HADOOP-7276
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7276
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.21.0
>         Environment: $ gcc -v
> Using built-in specs.
> COLLECT_GCC=gcc
> COLLECT_LTO_WRAPPER=/usr/lib/arm-linux-gnueabi/gcc/arm-linux-gnueabi/4.5.2/lto-wrapper
> Target: arm-linux-gnueabi
> Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.5.2-8ubuntu4' --with-bugurl=file:///usr/share/doc/gcc-4.5/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.5 --enable-shared --enable-multiarch --with-multiarch-defaults=arm-linux-gnueabi --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib/arm-linux-gnueabi --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.5 --libdir=/usr/lib/arm-linux-gnueabi --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-plugin --enable-gold --enable-ld=default --with-plugin-ld=ld.gold --enable-objc-gc --disable-sjlj-exceptions --with-arch=armv7-a --with-float=softfp --with-fpu=vfpv3-d16 --with-mode=thumb --disable-werror --enable-checking=release --build=arm-linux-gnueabi --host=arm-linux-gnueabi --target=arm-linux-gnueabi
> Thread model: posix
> gcc version 4.5.2 (Ubuntu/Linaro 4.5.2-8ubuntu4)
> $ uname -a
> Linux panda0 2.6.38-1002-linaro-omap #3-Ubuntu SMP Fri Apr 15 14:00:54 UTC 2011 armv7l armv7l armv7l GNU/Linux
>            Reporter: Trevor Robinson
>            Assignee: Trevor Robinson
>             Fix For: 0.23.0
>
>         Attachments: hadoop-common-arm.patch
>
>
> The native build fails on machine targets where gcc does not support -m32. This is any target other than x86, SPARC, RS/6000, or PowerPC, such as ARM.
> $ ant -Dcompile.native=true
> ...
>      [exec] make  all-am
>      [exec] make[1]: Entering directory
> `/home/trobinson/dev/hadoop-common/build/native/Linux-arm-32'
>      [exec] /bin/bash ./libtool  --tag=CC   --mode=compile gcc
> -DHAVE_CONFIG_H -I. -I/home/trobinson/dev/hadoop-common/src/native
> -I/usr/lib/jvm/java-6-openjdk/include
> -I/usr/lib/jvm/java-6-openjdk/include/linux
> -I/home/trobinson/dev/hadoop-common/src/native/src
> -Isrc/org/apache/hadoop/io/compress/zlib
> -Isrc/org/apache/hadoop/security -Isrc/org/apache/hadoop/io/nativeio/
> -g -Wall -fPIC -O2 -m32 -g -O2 -MT ZlibCompressor.lo -MD -MP -MF
> .deps/ZlibCompressor.Tpo -c -o ZlibCompressor.lo `test -f
> 'src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c' || echo
> '/home/trobinson/dev/hadoop-common/src/native/'`src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
>      [exec] libtool: compile:  gcc -DHAVE_CONFIG_H -I.
> -I/home/trobinson/dev/hadoop-common/src/native
> -I/usr/lib/jvm/java-6-openjdk/include
> -I/usr/lib/jvm/java-6-openjdk/include/linux
> -I/home/trobinson/dev/hadoop-common/src/native/src
> -Isrc/org/apache/hadoop/io/compress/zlib
> -Isrc/org/apache/hadoop/security -Isrc/org/apache/hadoop/io/nativeio/
> -g -Wall -fPIC -O2 -m32 -g -O2 -MT ZlibCompressor.lo -MD -MP -MF
> .deps/ZlibCompressor.Tpo -c
> /home/trobinson/dev/hadoop-common/src/native/src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
>  -fPIC -DPIC -o .libs/ZlibCompressor.o
>      [exec] make[1]: Leaving directory
> `/home/trobinson/dev/hadoop-common/build/native/Linux-arm-32'
>      [exec] cc1: error: unrecognized command line option "-m32"
>      [exec] make[1]: *** [ZlibCompressor.lo] Error 1
>      [exec] make: *** [all] Error 2

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15556-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 22:31:31 2011
Return-Path: <common-issues-return-15556-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8E0A24F3E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 22:31:31 +0000 (UTC)
Received: (qmail 33430 invoked by uid 500); 27 May 2011 22:31:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33383 invoked by uid 500); 27 May 2011 22:31:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33375 invoked by uid 99); 27 May 2011 22:31:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 22:31:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 22:31:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 61D0FE3AA8
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 22:30:48 +0000 (UTC)
Date: Fri, 27 May 2011 22:30:48 +0000 (UTC)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <572891901.50112.1306535448397.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Updated] (HADOOP-7106) Re-organize hadoop subversion layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nigel Daley updated HADOOP-7106:
--------------------------------

    Attachment: HADOOP-7106-auth.patch

New auth patch that correct paths now that everything is going under common.

> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15557-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 22:52:32 2011
Return-Path: <common-issues-return-15557-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 18AF346FB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 22:52:32 +0000 (UTC)
Received: (qmail 45725 invoked by uid 500); 27 May 2011 22:52:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45669 invoked by uid 500); 27 May 2011 22:52:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45567 invoked by uid 99); 27 May 2011 22:52:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 22:52:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 22:52:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 667FFE3092
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 22:51:47 +0000 (UTC)
Date: Fri, 27 May 2011 22:51:47 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1435375533.50176.1306536707416.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7336) TestFileContextResolveAfs will fail
 with default test.build.data property.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

TestFileContextResolveAfs will fail with default test.build.data property.
--------------------------------------------------------------------------

                 Key: HADOOP-7336
                 URL: https://issues.apache.org/jira/browse/HADOOP-7336
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.23.0
            Reporter: Jitendra Nath Pandey
            Assignee: Jitendra Nath Pandey
            Priority: Minor
             Fix For: 0.23.0


In TestFileContextResolveAfs if test.build.data property is not set and default is used, the test case will try to create that in the root directory and that will fail. /tmp should be used as default as in many other test cases. Normally, test.build.data will be set and this issue should not occur.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15558-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 23:02:28 2011
Return-Path: <common-issues-return-15558-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AC67D443E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 23:02:28 +0000 (UTC)
Received: (qmail 56369 invoked by uid 500); 27 May 2011 23:02:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56343 invoked by uid 500); 27 May 2011 23:02:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56335 invoked by uid 99); 27 May 2011 23:02:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 23:02:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 23:02:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 85D13E330E
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 23:01:47 +0000 (UTC)
Date: Fri, 27 May 2011 23:01:47 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1736647063.50200.1306537307545.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1435375533.50176.1306536707416.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7336) TestFileContextResolveAfs will fail
 with default test.build.data property.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7336:
-----------------------------------------

    Status: Patch Available  (was: Open)

> TestFileContextResolveAfs will fail with default test.build.data property.
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7336
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7336
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7336.2.patch
>
>
> In TestFileContextResolveAfs if test.build.data property is not set and default is used, the test case will try to create that in the root directory and that will fail. /tmp should be used as default as in many other test cases. Normally, test.build.data will be set and this issue should not occur.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15559-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 23:02:30 2011
Return-Path: <common-issues-return-15559-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B670A447D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 23:02:30 +0000 (UTC)
Received: (qmail 56629 invoked by uid 500); 27 May 2011 23:02:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56596 invoked by uid 500); 27 May 2011 23:02:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56588 invoked by uid 99); 27 May 2011 23:02:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 23:02:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 23:02:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 78D18E330A
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 23:01:47 +0000 (UTC)
Date: Fri, 27 May 2011 23:01:47 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1053116384.50198.1306537307491.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1435375533.50176.1306536707416.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7336) TestFileContextResolveAfs will fail
 with default test.build.data property.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7336:
-----------------------------------------

    Attachment: HADOOP-7336.2.patch

> TestFileContextResolveAfs will fail with default test.build.data property.
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7336
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7336
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7336.2.patch
>
>
> In TestFileContextResolveAfs if test.build.data property is not set and default is used, the test case will try to create that in the root directory and that will fail. /tmp should be used as default as in many other test cases. Normally, test.build.data will be set and this issue should not occur.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15560-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 23:20:29 2011
Return-Path: <common-issues-return-15560-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DD7A0472E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 23:20:28 +0000 (UTC)
Received: (qmail 78552 invoked by uid 500); 27 May 2011 23:20:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78527 invoked by uid 500); 27 May 2011 23:20:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78519 invoked by uid 99); 27 May 2011 23:20:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 23:20:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 23:20:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 855A3E36B0
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 23:19:47 +0000 (UTC)
Date: Fri, 27 May 2011 23:19:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1528936420.50221.1306538387542.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1435375533.50176.1306536707416.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7336) TestFileContextResolveAfs will
 fail with default test.build.data property.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040495#comment-13040495 ] 

Hadoop QA commented on HADOOP-7336:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480716/HADOOP-7336.2.patch
  against trunk revision 1128475.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/536//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/536//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/536//console

This message is automatically generated.

> TestFileContextResolveAfs will fail with default test.build.data property.
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7336
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7336
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7336.2.patch
>
>
> In TestFileContextResolveAfs if test.build.data property is not set and default is used, the test case will try to create that in the root directory and that will fail. /tmp should be used as default as in many other test cases. Normally, test.build.data will be set and this issue should not occur.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15561-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 23:49:29 2011
Return-Path: <common-issues-return-15561-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EB73C48FD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 23:49:28 +0000 (UTC)
Received: (qmail 8071 invoked by uid 500); 27 May 2011 23:49:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8037 invoked by uid 500); 27 May 2011 23:49:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8029 invoked by uid 99); 27 May 2011 23:49:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 23:49:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 23:49:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 63AE3E3D9E
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 23:48:47 +0000 (UTC)
Date: Fri, 27 May 2011 23:48:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1956994843.50267.1306540127404.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7337) Annotate PureJavaCrc32 as a public
 API
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Annotate PureJavaCrc32 as a public API
--------------------------------------

                 Key: HADOOP-7337
                 URL: https://issues.apache.org/jira/browse/HADOOP-7337
             Project: Hadoop Common
          Issue Type: Improvement
          Components: util
            Reporter: Tsz Wo (Nicholas), SZE
            Assignee: Tsz Wo (Nicholas), SZE
            Priority: Minor


The API of PureJavaCrc32 is stable.  It is incorrect to annotate it as private unstable.


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15562-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 23:51:28 2011
Return-Path: <common-issues-return-15562-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DB6324932
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 23:51:28 +0000 (UTC)
Received: (qmail 9882 invoked by uid 500); 27 May 2011 23:51:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9844 invoked by uid 500); 27 May 2011 23:51:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9834 invoked by uid 99); 27 May 2011 23:51:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 23:51:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 23:51:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C1D02E3E81
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 23:50:47 +0000 (UTC)
Date: Fri, 27 May 2011 23:50:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <480999673.50281.1306540247790.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1956994843.50267.1306540127404.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7337) Annotate PureJavaCrc32 as a public
 API
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7337:
-------------------------------------------

    Attachment: c7337_20110527.patch

c7337_20110527.patch: changed the annotation to public stable.

> Annotate PureJavaCrc32 as a public API
> --------------------------------------
>
>                 Key: HADOOP-7337
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7337
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>            Priority: Minor
>         Attachments: c7337_20110527.patch
>
>
> The API of PureJavaCrc32 is stable.  It is incorrect to annotate it as private unstable.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15563-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 23:51:29 2011
Return-Path: <common-issues-return-15563-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 13FE14948
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 23:51:29 +0000 (UTC)
Received: (qmail 10138 invoked by uid 500); 27 May 2011 23:51:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10082 invoked by uid 500); 27 May 2011 23:51:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9919 invoked by uid 99); 27 May 2011 23:51:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 23:51:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 23:51:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D1965E3E83
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 23:50:47 +0000 (UTC)
Date: Fri, 27 May 2011 23:50:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <180402026.50283.1306540247855.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1956994843.50267.1306540127404.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7337) Annotate PureJavaCrc32 as a public
 API
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7337:
-------------------------------------------

    Status: Patch Available  (was: Open)

> Annotate PureJavaCrc32 as a public API
> --------------------------------------
>
>                 Key: HADOOP-7337
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7337
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>            Priority: Minor
>         Attachments: c7337_20110527.patch
>
>
> The API of PureJavaCrc32 is stable.  It is incorrect to annotate it as private unstable.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15564-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Fri May 27 23:53:28 2011
Return-Path: <common-issues-return-15564-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AA4F14993
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Fri, 27 May 2011 23:53:28 +0000 (UTC)
Received: (qmail 11762 invoked by uid 500); 27 May 2011 23:53:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11736 invoked by uid 500); 27 May 2011 23:53:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11725 invoked by uid 99); 27 May 2011 23:53:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 23:53:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 May 2011 23:53:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5577DE3F0C
	for <common-issues@hadoop.apache.org>; Fri, 27 May 2011 23:52:47 +0000 (UTC)
Date: Fri, 27 May 2011 23:52:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1651213885.50286.1306540367346.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1956994843.50267.1306540127404.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7337) Annotate PureJavaCrc32 as a public
 API
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040506#comment-13040506 ] 

Todd Lipcon commented on HADOOP-7337:
-------------------------------------

+1

> Annotate PureJavaCrc32 as a public API
> --------------------------------------
>
>                 Key: HADOOP-7337
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7337
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>            Priority: Minor
>         Attachments: c7337_20110527.patch
>
>
> The API of PureJavaCrc32 is stable.  It is incorrect to annotate it as private unstable.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15565-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 28 00:01:35 2011
Return-Path: <common-issues-return-15565-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id AE77E4457
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 28 May 2011 00:01:35 +0000 (UTC)
Received: (qmail 14780 invoked by uid 500); 28 May 2011 00:01:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14730 invoked by uid 500); 28 May 2011 00:01:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14707 invoked by uid 99); 28 May 2011 00:01:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 May 2011 00:01:35 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 May 2011 00:01:34 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 06D8CE3196
	for <common-issues@hadoop.apache.org>; Sat, 28 May 2011 00:00:54 +0000 (UTC)
Date: Sat, 28 May 2011 00:00:54 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <118923021.50328.1306540854025.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7338) LocalDirAllocator improvements for
 MR-2178
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

LocalDirAllocator improvements for MR-2178
------------------------------------------

                 Key: HADOOP-7338
                 URL: https://issues.apache.org/jira/browse/HADOOP-7338
             Project: Hadoop Common
          Issue Type: Improvement
    Affects Versions: 0.22.0
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
             Fix For: 0.22.0
         Attachments: hadoop-7338.txt

These are some changes that come from the following yahoo-merge commit, necessary for MAPREDUCE-2178 to build.

    commit 446e41f2424ddc866770cda677450c18feeb597f
    Author: Chris Douglas <cdouglas@apache.org>
    Date:   Sun Oct 3 02:00:15 2010 -0700
    
        Apply y201 to trunk
    
    git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/yahoo-merge@1079151 13f79535-47bb-0310-9956-ffa450edef68

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15566-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 28 00:03:28 2011
Return-Path: <common-issues-return-15566-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8CDEF4DA6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 28 May 2011 00:03:28 +0000 (UTC)
Received: (qmail 17039 invoked by uid 500); 28 May 2011 00:03:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17001 invoked by uid 500); 28 May 2011 00:03:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16993 invoked by uid 99); 28 May 2011 00:03:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 May 2011 00:03:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 May 2011 00:03:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5F3C3E324A
	for <common-issues@hadoop.apache.org>; Sat, 28 May 2011 00:02:47 +0000 (UTC)
Date: Sat, 28 May 2011 00:02:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1808502900.50337.1306540967386.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <118923021.50328.1306540854025.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7338) LocalDirAllocator improvements for
 MR-2178
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7338:
--------------------------------

    Attachment: hadoop-7338.txt

Attaching patch from yahoo-merge. This probably needs a couple unit tests.

> LocalDirAllocator improvements for MR-2178
> ------------------------------------------
>
>                 Key: HADOOP-7338
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7338
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-7338.txt
>
>
> These are some changes that come from the following yahoo-merge commit, necessary for MAPREDUCE-2178 to build.
>     commit 446e41f2424ddc866770cda677450c18feeb597f
>     Author: Chris Douglas <cdouglas@apache.org>
>     Date:   Sun Oct 3 02:00:15 2010 -0700
>     
>         Apply y201 to trunk
>     
>     git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/yahoo-merge@1079151 13f79535-47bb-0310-9956-ffa450edef68

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15567-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 28 00:05:29 2011
Return-Path: <common-issues-return-15567-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E9C034171
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 28 May 2011 00:05:28 +0000 (UTC)
Received: (qmail 17780 invoked by uid 500); 28 May 2011 00:05:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17687 invoked by uid 500); 28 May 2011 00:05:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17679 invoked by uid 99); 28 May 2011 00:05:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 May 2011 00:05:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 May 2011 00:05:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 94E99E333E
	for <common-issues@hadoop.apache.org>; Sat, 28 May 2011 00:04:47 +0000 (UTC)
Date: Sat, 28 May 2011 00:04:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <615720184.50347.1306541087606.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1956994843.50267.1306540127404.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7337) Annotate PureJavaCrc32 as a public
 API
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040514#comment-13040514 ] 

Hadoop QA commented on HADOOP-7337:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480721/c7337_20110527.patch
  against trunk revision 1128475.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/537//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/537//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/537//console

This message is automatically generated.

> Annotate PureJavaCrc32 as a public API
> --------------------------------------
>
>                 Key: HADOOP-7337
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7337
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>            Priority: Minor
>         Attachments: c7337_20110527.patch
>
>
> The API of PureJavaCrc32 is stable.  It is incorrect to annotate it as private unstable.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15568-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 28 04:53:42 2011
Return-Path: <common-issues-return-15568-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 511FE64A5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 28 May 2011 04:53:42 +0000 (UTC)
Received: (qmail 3703 invoked by uid 500); 28 May 2011 04:53:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3489 invoked by uid 500); 28 May 2011 04:53:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3480 invoked by uid 99); 28 May 2011 04:53:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 May 2011 04:53:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 May 2011 04:53:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 720E2E481E
	for <common-issues@hadoop.apache.org>; Sat, 28 May 2011 04:52:47 +0000 (UTC)
Date: Sat, 28 May 2011 04:52:47 +0000 (UTC)
From: "Otis Gospodnetic (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1350909266.50575.1306558367463.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1682376274.1991.1305078887341.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7273) Move JMXGet from hdfs to common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040547#comment-13040547 ] 

Otis Gospodnetic commented on HADOOP-7273:
------------------------------------------

Is it Hadoop-dependent?  Why not spin it out all the way to make it available to others?


> Move JMXGet from hdfs to common
> -------------------------------
>
>                 Key: HADOOP-7273
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7273
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.21.0
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.23.0
>
>
> JMXGet is generally useful especially for testing JMX attributes. Let's move it from o.a.h.hdfs.tools to o.a.h.tools.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15569-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 28 11:16:32 2011
Return-Path: <common-issues-return-15569-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 236586BAB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 28 May 2011 11:16:32 +0000 (UTC)
Received: (qmail 31418 invoked by uid 500); 28 May 2011 11:16:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31373 invoked by uid 500); 28 May 2011 11:16:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31364 invoked by uid 99); 28 May 2011 11:16:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 May 2011 11:16:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 May 2011 11:16:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 0E55FE4A08
	for <common-issues@hadoop.apache.org>; Sat, 28 May 2011 11:15:49 +0000 (UTC)
Date: Sat, 28 May 2011 11:15:49 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1824584104.50742.1306581349055.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6255) Create an rpm integration project
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D130=
40575#comment-13040575 ]=20

Hudson commented on HADOOP-6255:
--------------------------------

Integrated in Hadoop-Common-trunk #702 (See [https://builds.apache.org/huds=
on/job/Hadoop-Common-trunk/702/])
    HADOOP-6255. Create RPM and Debian packages for common. Changes deploym=
ent
layout to be consistent across the binary tgz, rpm, and deb. Adds setup
scripts for easy one node cluster configuration and user creation.
(Eric Yang via omalley)

omalley : http://svn.apache.org/viewcvs.cgi/?root=3DApache-SVN&view=3Drev&r=
ev=3D1128385
Files :=20
* /hadoop/common/trunk/src/packages/deb/hadoop.control/postinst
* /hadoop/common/trunk/src/docs/cn/src/documentation/content/xdocs/mapred_t=
utorial.xml
* /hadoop/common/trunk/bin/stop-all.sh
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/packages/rpm/init.d/hadoop-tasktracker
* /hadoop/common/trunk/src/packages/hadoop-setup-single-node.sh
* /hadoop/common/trunk/src/test/system/c++/runAs/runAs.c
* /hadoop/common/trunk/bin/hadoop-daemon.sh
* /hadoop/common/trunk/src/packages/rpm/spec/hadoop.spec
* /hadoop/common/trunk/src/packages/deb/hadoop.control/control
* /hadoop/common/trunk/src/packages/deb/hadoop.control/prerm
* /hadoop/common/trunk/bin/hadoop-config.sh
* /hadoop/common/trunk/src/packages/templates
* /hadoop/common/trunk/src/native/Makefile.am
* /hadoop/common/trunk/src/docs/src/documentation/content/xdocs/cluster_set=
up.xml
* /hadoop/common/trunk/src/packages/deb/init.d/hadoop-namenode
* /hadoop/common/trunk/src/packages/templates/conf
* /hadoop/common/trunk/src/packages/deb
* /hadoop/common/trunk/src/packages/deb/hadoop.control/preinst
* /hadoop/common/trunk/ivy/libraries.properties
* /hadoop/common/trunk/src/docs/cn/src/documentation/content/xdocs/cluster_=
setup.xml
* /hadoop/common/trunk/bin/hadoop
* /hadoop/common/trunk/src/packages/deb/init.d/hadoop-tasktracker
* /hadoop/common/trunk/src/test/system/c++/runAs/configure
* /hadoop/common/trunk/src/native/src/org/apache/hadoop/io/compress/zlib/Ma=
kefile.am
* /hadoop/common/trunk/bin/slaves.sh
* /hadoop/common/trunk/src/packages/deb/init.d/hadoop-jobtracker
* /hadoop/common/trunk/src/packages
* /hadoop/common/trunk/src/packages/templates/conf/core-site.xml
* /hadoop/common/trunk/src/packages/rpm/init.d/hadoop-namenode
* /hadoop/common/trunk/src/docs/cn/src/documentation/content/xdocs/hod_admi=
n_guide.xml
* /hadoop/common/trunk/bin/rcc
* /hadoop/common/trunk/src/packages/deb/init.d
* /hadoop/common/trunk/src/docs/src/documentation/content/xdocs/site.xml
* /hadoop/common/trunk/src/docs/src/documentation/content/xdocs/single_node=
_setup.xml
* /hadoop/common/trunk/src/native/packageNativeHadoop.sh
* /hadoop/common/trunk/src/packages/deb/init.d/hadoop-datanode
* /hadoop/common/trunk/src/test/system/c++/runAs/runAs.h.in
* /hadoop/common/trunk/src/packages/update-hadoop-env.sh
* /hadoop/common/trunk/src/packages/rpm/init.d/hadoop-jobtracker
* /hadoop/common/trunk/src/packages/rpm/spec
* /hadoop/common/trunk/src/docs/src/documentation/content/xdocs/commands_ma=
nual.xml
* /hadoop/common/trunk/src/docs/cn/src/documentation/content/xdocs/quicksta=
rt.xml
* /hadoop/common/trunk/bin/hadoop-daemons.sh
* /hadoop/common/trunk/conf/hadoop-env.sh.template
* /hadoop/common/trunk/ivy.xml
* /hadoop/common/trunk/build.xml
* /hadoop/common/trunk/src/packages/hadoop-setup-hdfs.sh
* /hadoop/common/trunk/src/packages/deb/hadoop.control
* /hadoop/common/trunk/src/docs/cn/src/documentation/content/xdocs/commands=
_manual.xml
* /hadoop/common/trunk/src/packages/deb/hadoop.control/postrm
* /hadoop/common/trunk/src/docs/src/documentation/content/xdocs/deployment_=
layout.xml
* /hadoop/common/trunk/src/test/system/java/org/apache/hadoop/test/system/p=
rocess/HadoopDaemonRemoteCluster.java
* /hadoop/common/trunk/src/packages/rpm/init.d
* /hadoop/common/trunk/bin/start-all.sh
* /hadoop/common/trunk/src/test/system/c++/runAs/configure.ac
* /hadoop/common/trunk/src/packages/hadoop-setup-conf.sh
* /hadoop/common/trunk/src/packages/deb/hadoop.control/conffile
* /hadoop/common/trunk/src/docs/cn/src/documentation/content/xdocs/streamin=
g.xml
* /hadoop/common/trunk/src/packages/hadoop-create-user.sh
* /hadoop/common/trunk/src/packages/rpm
* /hadoop/common/trunk/src/native/lib/Makefile.am
* /hadoop/common/trunk/src/packages/rpm/init.d/hadoop-datanode


> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.20.203.0
>            Reporter: Owen O'Malley
>            Assignee: Eric Yang
>             Fix For: 0.20.204.0, 0.23.0
>
>         Attachments: HADOOP-6255-branch-0.20-security-1.patch, HADOOP-625=
5-branch-0.20-security-10.patch, HADOOP-6255-branch-0.20-security-11.patch,=
 HADOOP-6255-branch-0.20-security-12.patch, HADOOP-6255-branch-0.20-securit=
y-13.patch, HADOOP-6255-branch-0.20-security-2.patch, HADOOP-6255-branch-0.=
20-security-3.patch, HADOOP-6255-branch-0.20-security-4.patch, HADOOP-6255-=
branch-0.20-security-5.patch, HADOOP-6255-branch-0.20-security-6.patch, HAD=
OOP-6255-branch-0.20-security-7.patch, HADOOP-6255-branch-0.20-security-8.p=
atch, HADOOP-6255-branch-0.20-security-9.patch, HADOOP-6255-branch-0.20-sec=
urity.patch, HADOOP-6255-common-trunk-1.patch, HADOOP-6255-common-trunk-10.=
patch, HADOOP-6255-common-trunk-11.patch, HADOOP-6255-common-trunk-12.patch=
, HADOOP-6255-common-trunk-2.patch, HADOOP-6255-common-trunk-4.patch, HADOO=
P-6255-common-trunk-5.patch, HADOOP-6255-common-trunk-6.patch, HADOOP-6255-=
common-trunk-7.patch, HADOOP-6255-common-trunk-8.patch, HADOOP-6255-common-=
trunk-9.patch, HADOOP-6255-common-trunk.patch, HADOOP-6255-hdfs-trunk-1.pat=
ch, HADOOP-6255-hdfs-trunk.patch, HADOOP-6255-mapred-trunk-1.patch, HADOOP-=
6255-mapred-trunk-2.patch, HADOOP-6255-mapred-trunk.patch, HADOOP-6255.patc=
h, deployment.pdf, deployment.tex
>
>
> We should be able to create RPMs for Hadoop releases.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15570-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 28 11:16:32 2011
Return-Path: <common-issues-return-15570-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 347676BBD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 28 May 2011 11:16:32 +0000 (UTC)
Received: (qmail 31506 invoked by uid 500); 28 May 2011 11:16:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31387 invoked by uid 500); 28 May 2011 11:16:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31374 invoked by uid 99); 28 May 2011 11:16:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 May 2011 11:16:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 May 2011 11:16:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8F197E4A14
	for <common-issues@hadoop.apache.org>; Sat, 28 May 2011 11:15:49 +0000 (UTC)
Date: Sat, 28 May 2011 11:15:49 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1787864975.50754.1306581349583.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <532896131.3523.1305130907394.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7276) Hadoop native builds fail on ARM
 due to -m32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040577#comment-13040577 ] 

Hudson commented on HADOOP-7276:
--------------------------------

Integrated in Hadoop-Common-trunk #702 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/702/])
    HADOOP-7276. Hadoop native builds fail on ARM due to -m32. Contributed by Trevor Robinson

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1128475
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/native/configure.ac
* /hadoop/common/trunk/src/native/Makefile.am
* /hadoop/common/trunk/src/native/src/org/apache/hadoop/io/compress/zlib/Makefile.am
* /hadoop/common/trunk/src/native/lib/Makefile.am


> Hadoop native builds fail on ARM due to -m32
> --------------------------------------------
>
>                 Key: HADOOP-7276
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7276
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.21.0
>         Environment: $ gcc -v
> Using built-in specs.
> COLLECT_GCC=gcc
> COLLECT_LTO_WRAPPER=/usr/lib/arm-linux-gnueabi/gcc/arm-linux-gnueabi/4.5.2/lto-wrapper
> Target: arm-linux-gnueabi
> Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.5.2-8ubuntu4' --with-bugurl=file:///usr/share/doc/gcc-4.5/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.5 --enable-shared --enable-multiarch --with-multiarch-defaults=arm-linux-gnueabi --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib/arm-linux-gnueabi --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.5 --libdir=/usr/lib/arm-linux-gnueabi --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-plugin --enable-gold --enable-ld=default --with-plugin-ld=ld.gold --enable-objc-gc --disable-sjlj-exceptions --with-arch=armv7-a --with-float=softfp --with-fpu=vfpv3-d16 --with-mode=thumb --disable-werror --enable-checking=release --build=arm-linux-gnueabi --host=arm-linux-gnueabi --target=arm-linux-gnueabi
> Thread model: posix
> gcc version 4.5.2 (Ubuntu/Linaro 4.5.2-8ubuntu4)
> $ uname -a
> Linux panda0 2.6.38-1002-linaro-omap #3-Ubuntu SMP Fri Apr 15 14:00:54 UTC 2011 armv7l armv7l armv7l GNU/Linux
>            Reporter: Trevor Robinson
>            Assignee: Trevor Robinson
>             Fix For: 0.23.0
>
>         Attachments: hadoop-common-arm.patch
>
>
> The native build fails on machine targets where gcc does not support -m32. This is any target other than x86, SPARC, RS/6000, or PowerPC, such as ARM.
> $ ant -Dcompile.native=true
> ...
>      [exec] make  all-am
>      [exec] make[1]: Entering directory
> `/home/trobinson/dev/hadoop-common/build/native/Linux-arm-32'
>      [exec] /bin/bash ./libtool  --tag=CC   --mode=compile gcc
> -DHAVE_CONFIG_H -I. -I/home/trobinson/dev/hadoop-common/src/native
> -I/usr/lib/jvm/java-6-openjdk/include
> -I/usr/lib/jvm/java-6-openjdk/include/linux
> -I/home/trobinson/dev/hadoop-common/src/native/src
> -Isrc/org/apache/hadoop/io/compress/zlib
> -Isrc/org/apache/hadoop/security -Isrc/org/apache/hadoop/io/nativeio/
> -g -Wall -fPIC -O2 -m32 -g -O2 -MT ZlibCompressor.lo -MD -MP -MF
> .deps/ZlibCompressor.Tpo -c -o ZlibCompressor.lo `test -f
> 'src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c' || echo
> '/home/trobinson/dev/hadoop-common/src/native/'`src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
>      [exec] libtool: compile:  gcc -DHAVE_CONFIG_H -I.
> -I/home/trobinson/dev/hadoop-common/src/native
> -I/usr/lib/jvm/java-6-openjdk/include
> -I/usr/lib/jvm/java-6-openjdk/include/linux
> -I/home/trobinson/dev/hadoop-common/src/native/src
> -Isrc/org/apache/hadoop/io/compress/zlib
> -Isrc/org/apache/hadoop/security -Isrc/org/apache/hadoop/io/nativeio/
> -g -Wall -fPIC -O2 -m32 -g -O2 -MT ZlibCompressor.lo -MD -MP -MF
> .deps/ZlibCompressor.Tpo -c
> /home/trobinson/dev/hadoop-common/src/native/src/org/apache/hadoop/io/compress/zlib/ZlibCompressor.c
>  -fPIC -DPIC -o .libs/ZlibCompressor.o
>      [exec] make[1]: Leaving directory
> `/home/trobinson/dev/hadoop-common/build/native/Linux-arm-32'
>      [exec] cc1: error: unrecognized command line option "-m32"
>      [exec] make[1]: *** [ZlibCompressor.lo] Error 1
>      [exec] make: *** [all] Error 2

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15571-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 28 11:16:33 2011
Return-Path: <common-issues-return-15571-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0DCAB6BCF
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 28 May 2011 11:16:33 +0000 (UTC)
Received: (qmail 31882 invoked by uid 500); 28 May 2011 11:16:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31855 invoked by uid 500); 28 May 2011 11:16:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31847 invoked by uid 99); 28 May 2011 11:16:32 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 May 2011 11:16:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 May 2011 11:16:30 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 78A45E4A12
	for <common-issues@hadoop.apache.org>; Sat, 28 May 2011 11:15:49 +0000 (UTC)
Date: Sat, 28 May 2011 11:15:49 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <118885136.50752.1306581349490.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1654473161.45482.1306422287498.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7333) Performance improvement in
 PureJavaCrc32
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040576#comment-13040576 ] 

Hudson commented on HADOOP-7333:
--------------------------------

Integrated in Hadoop-Common-trunk #702 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/702/])
    HADOOP-7333. Performance improvement in PureJavaCrc32. Contributed by Eric Caspole.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1128425
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/util/PureJavaCrc32.java


> Performance improvement in PureJavaCrc32
> ----------------------------------------
>
>                 Key: HADOOP-7333
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7333
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>         Environment: Linux x64
>            Reporter: Eric Caspole
>            Assignee: Eric Caspole
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7333.patch, c7333_20110526.patch
>
>
> I would like to propose a small patch to 
>   org.apache.hadoop.util.PureJavaCrc32.update(byte[] b, int off, int len)
> Currently the method stores the intermediate result back into the data member "crc." I noticed this method gets
> inlined into DataChecksum.update() and that method appears as one of the hotter methods in a simple hprof profile collected while running terasort and gridmix.
> If the code is modified to save the temporary result into a local and just once store the final result back into the data member, it results in slightly more efficient hotspot codegen.
> I tested this change using the the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32 on a variety of linux x64 AMD and Intel multi-socket and multi-core systems I have available to test.
> The patch removes several stores of the intermediate result to memory yielding a 0%-10% speedup in the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32.
>  
> If you use a debug hotspot JVM with -XX:+PrintOptoAssembly, you can see the intermediate stores such as:
> 414     movq    R9, [rsp + #24] # spill
> 419     movl    [R9 + #12 (8-bit)], RDX # int ! Field PureJavaCrc32.crc
> 41d     xorl    R10, RDX        # int
> The patch results in just one final store of the fully computed value.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15572-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 28 22:35:28 2011
Return-Path: <common-issues-return-15572-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ADCBF63F2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 28 May 2011 22:35:28 +0000 (UTC)
Received: (qmail 2249 invoked by uid 500); 28 May 2011 22:35:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2223 invoked by uid 500); 28 May 2011 22:35:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2215 invoked by uid 99); 28 May 2011 22:35:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 May 2011 22:35:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 May 2011 22:35:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 531B9E5697
	for <common-issues@hadoop.apache.org>; Sat, 28 May 2011 22:34:47 +0000 (UTC)
Date: Sat, 28 May 2011 22:34:47 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <676652622.51268.1306622087336.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1956994843.50267.1306540127404.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7337) Annotate PureJavaCrc32 as a public
 API
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-7337:
-------------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.23.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

No new tests added since this is an annotation and javadoc change.

Thanks Todd for the review.

I have committed this.

> Annotate PureJavaCrc32 as a public API
> --------------------------------------
>
>                 Key: HADOOP-7337
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7337
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: c7337_20110527.patch
>
>
> The API of PureJavaCrc32 is stable.  It is incorrect to annotate it as private unstable.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15573-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sat May 28 22:45:28 2011
Return-Path: <common-issues-return-15573-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A15F46795
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sat, 28 May 2011 22:45:28 +0000 (UTC)
Received: (qmail 4979 invoked by uid 500); 28 May 2011 22:45:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4944 invoked by uid 500); 28 May 2011 22:45:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4936 invoked by uid 99); 28 May 2011 22:45:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 May 2011 22:45:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 May 2011 22:45:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 589A2E585B
	for <common-issues@hadoop.apache.org>; Sat, 28 May 2011 22:44:47 +0000 (UTC)
Date: Sat, 28 May 2011 22:44:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1014636354.51288.1306622687359.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1956994843.50267.1306540127404.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7337) Annotate PureJavaCrc32 as a public
 API
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040688#comment-13040688 ] 

Hudson commented on HADOOP-7337:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #628 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/628/])
    HADOOP-7337. Change PureJavaCrc32 annotations to public stable.

szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1128789
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/util/PureJavaCrc32.java


> Annotate PureJavaCrc32 as a public API
> --------------------------------------
>
>                 Key: HADOOP-7337
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7337
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: c7337_20110527.patch
>
>
> The API of PureJavaCrc32 is stable.  It is incorrect to annotate it as private unstable.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15574-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun May 29 01:57:31 2011
Return-Path: <common-issues-return-15574-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 189996303
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 29 May 2011 01:57:31 +0000 (UTC)
Received: (qmail 81235 invoked by uid 500); 29 May 2011 01:57:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81167 invoked by uid 500); 29 May 2011 01:57:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81159 invoked by uid 99); 29 May 2011 01:57:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 May 2011 01:57:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 May 2011 01:57:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D4F35E5594
	for <common-issues@hadoop.apache.org>; Sun, 29 May 2011 01:56:47 +0000 (UTC)
Date: Sun, 29 May 2011 01:56:47 +0000 (UTC)
From: "Kevin Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1942225372.51490.1306634207869.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6767) Patch for running Hadoop on
 Windows without Cygwin
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040721#comment-13040721 ] 

Kevin Wang commented on HADOOP-6767:
------------------------------------

Hi Guys,

I am pretty new to this site. I am wondering when this patch can be accepted?

Thanks,

Kevin

> Patch for running Hadoop on Windows without Cygwin
> --------------------------------------------------
>
>                 Key: HADOOP-6767
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6767
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, conf, scripts
>    Affects Versions: 0.22.0
>         Environment: Windows XP, 2003, 7, 2008
>            Reporter: Volodymyr Orlov
>              Labels: blockdecompressorstream
>         Attachments: HADOOP-6767.patch, Hadoop-0.20.2-patched.zip, hadoop-6767-r1.tar.gz
>
>
> Proposed patch from Codeminders adds a possibility to run Hadoop on Windows without Cygwin. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15575-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun May 29 02:01:29 2011
Return-Path: <common-issues-return-15575-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2C0AC43B8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 29 May 2011 02:01:29 +0000 (UTC)
Received: (qmail 82467 invoked by uid 500); 29 May 2011 02:01:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82436 invoked by uid 500); 29 May 2011 02:01:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82428 invoked by uid 99); 29 May 2011 02:01:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 May 2011 02:01:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 May 2011 02:01:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 08965E5681
	for <common-issues@hadoop.apache.org>; Sun, 29 May 2011 02:00:48 +0000 (UTC)
Date: Sun, 29 May 2011 02:00:48 +0000 (UTC)
From: "Kevin Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1023851985.51506.1306634448032.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6767) Patch for running Hadoop on
 Windows without Cygwin
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040722#comment-13040722 ] 

Kevin Wang commented on HADOOP-6767:
------------------------------------

Hi Volodymyr,

I am wondering whether you have plan to migrate the change to JNI and Apache Daemon? 

Thanks,

Kevin



> Patch for running Hadoop on Windows without Cygwin
> --------------------------------------------------
>
>                 Key: HADOOP-6767
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6767
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, conf, scripts
>    Affects Versions: 0.22.0
>         Environment: Windows XP, 2003, 7, 2008
>            Reporter: Volodymyr Orlov
>              Labels: blockdecompressorstream
>         Attachments: HADOOP-6767.patch, Hadoop-0.20.2-patched.zip, hadoop-6767-r1.tar.gz
>
>
> Proposed patch from Codeminders adds a possibility to run Hadoop on Windows without Cygwin. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15576-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun May 29 03:58:31 2011
Return-Path: <common-issues-return-15576-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B80534DBE
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 29 May 2011 03:58:31 +0000 (UTC)
Received: (qmail 28515 invoked by uid 500); 29 May 2011 03:58:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28489 invoked by uid 500); 29 May 2011 03:58:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28480 invoked by uid 99); 29 May 2011 03:58:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 May 2011 03:58:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 May 2011 03:58:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F2E48E65C2
	for <common-issues@hadoop.apache.org>; Sun, 29 May 2011 03:57:47 +0000 (UTC)
Date: Sun, 29 May 2011 03:57:47 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <703410915.51559.1306641467991.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <504299282.44039.1306374047470.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7331) Makes hadoop-daemon.sh to return 1
 if daemon processes did not get started
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040735#comment-13040735 ] 

Tanping Wang commented on HADOOP-7331:
--------------------------------------

The idea is to simply check if the PID still exists shortly after the JVM of the daemon process is created.  As discussed early, we are simplifying the solution to deal with common scenarios that may cause JVM to die shortly after the JVM is created, such as invalid configuration files.  And yes, this is machine specific.  We do not have data to show how much exact time the JVM may die on top of commodity hardware.  This is to provide the operator a quick feedback if JVM may not have been started if some straight-forward mis-configuration has happened.  And regarding to why choose 3 seconds??  If you notice that there is also
{code} 
sleep 1; head "$log"
{code}
in the same question can be asked regarding to why choose 1 second.  I simply want to give a few seconds before checking the existence of JVM after its starts.  And I can certainly make this sleep time configurable for the operators to adjust.

(BTW, what are you referring to by "one of those"?)

> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7331
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7331
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.23.0
>            Reporter: Tanping Wang
>            Assignee: Tanping Wang
>            Priority: Trivial
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7331.2.patch, HADOOP-7331.patch
>
>
> Makes hadoop-daemon.sh to return 1 if daemon processes did not get started.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15577-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun May 29 11:15:32 2011
Return-Path: <common-issues-return-15577-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E69404009
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 29 May 2011 11:15:31 +0000 (UTC)
Received: (qmail 99007 invoked by uid 500); 29 May 2011 11:15:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98976 invoked by uid 500); 29 May 2011 11:15:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98967 invoked by uid 99); 29 May 2011 11:15:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 May 2011 11:15:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 May 2011 11:15:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7F922E6A12
	for <common-issues@hadoop.apache.org>; Sun, 29 May 2011 11:14:47 +0000 (UTC)
Date: Sun, 29 May 2011 11:14:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <973348227.51816.1306667687519.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1956994843.50267.1306540127404.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7337) Annotate PureJavaCrc32 as a public
 API
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040784#comment-13040784 ] 

Hudson commented on HADOOP-7337:
--------------------------------

Integrated in Hadoop-Common-trunk #703 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk/703/])
    HADOOP-7337. Change PureJavaCrc32 annotations to public stable.

szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1128789
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/java/org/apache/hadoop/util/PureJavaCrc32.java


> Annotate PureJavaCrc32 as a public API
> --------------------------------------
>
>                 Key: HADOOP-7337
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7337
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: c7337_20110527.patch
>
>
> The API of PureJavaCrc32 is stable.  It is incorrect to annotate it as private unstable.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15578-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun May 29 11:29:31 2011
Return-Path: <common-issues-return-15578-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 659FC459D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 29 May 2011 11:29:31 +0000 (UTC)
Received: (qmail 6134 invoked by uid 500); 29 May 2011 11:29:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6084 invoked by uid 500); 29 May 2011 11:29:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6076 invoked by uid 99); 29 May 2011 11:29:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 May 2011 11:29:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 May 2011 11:29:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2287FE6DA3
	for <common-issues@hadoop.apache.org>; Sun, 29 May 2011 11:28:48 +0000 (UTC)
Date: Sun, 29 May 2011 11:28:48 +0000 (UTC)
From: "Volodymyr Orlov (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <584053657.51842.1306668528138.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6767) Patch for running Hadoop on
 Windows without Cygwin
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040788#comment-13040788 ] 

Volodymyr Orlov commented on HADOOP-6767:
-----------------------------------------

Hi Kevin,

Despite the fact that I haven't done anything regarding this patch I am still planning to continue my work on it. The problem that I have faced with is that I won't have an opportunity to test my patch in a real deployment scenario as soon as I will finish it. In company, where I am wokring now we are not deploying Hadoop on Windows machines any more. That's why I can't see any real application for my possible contribution in this area.
If you are interested in Hadoop on Windows, and you have Windows machines and intention to deploy your Hadoop cluster on them, we could combine our efforts and finish this patch. Here is my email address - vorl@codeminders.com. Please feel free to contact me if you are interested in my proposal.

Vladimir

> Patch for running Hadoop on Windows without Cygwin
> --------------------------------------------------
>
>                 Key: HADOOP-6767
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6767
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, conf, scripts
>    Affects Versions: 0.22.0
>         Environment: Windows XP, 2003, 7, 2008
>            Reporter: Volodymyr Orlov
>              Labels: blockdecompressorstream
>         Attachments: HADOOP-6767.patch, Hadoop-0.20.2-patched.zip, hadoop-6767-r1.tar.gz
>
>
> Proposed patch from Codeminders adds a possibility to run Hadoop on Windows without Cygwin. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15579-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun May 29 22:27:29 2011
Return-Path: <common-issues-return-15579-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A74194C8A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 29 May 2011 22:27:29 +0000 (UTC)
Received: (qmail 7175 invoked by uid 500); 29 May 2011 22:27:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7143 invoked by uid 500); 29 May 2011 22:27:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7132 invoked by uid 99); 29 May 2011 22:27:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 May 2011 22:27:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 May 2011 22:27:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 2D602D9922
	for <common-issues@hadoop.apache.org>; Sun, 29 May 2011 22:26:48 +0000 (UTC)
Date: Sun, 29 May 2011 22:26:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <625942760.52355.1306708008182.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7106) Re-organize hadoop subversion
 layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040891#comment-13040891 ] 

Todd Lipcon commented on HADOOP-7106:
-------------------------------------

Hey Nigel. I'm working on testing this against a local mirror of the ASF svn repository.

I noticed that the svn:externals settings at the bottom are inconsistent - the src/test ones all use https:// URLs but the site/ ones use http://. Is that on purpose?

> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15580-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Sun May 29 23:03:30 2011
Return-Path: <common-issues-return-15580-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 992304A8A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Sun, 29 May 2011 23:03:30 +0000 (UTC)
Received: (qmail 12779 invoked by uid 500); 29 May 2011 23:03:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12752 invoked by uid 500); 29 May 2011 23:03:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12744 invoked by uid 99); 29 May 2011 23:03:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 May 2011 23:03:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 May 2011 23:03:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5F88AD9DBD
	for <common-issues@hadoop.apache.org>; Sun, 29 May 2011 23:02:47 +0000 (UTC)
Date: Sun, 29 May 2011 23:02:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <97603347.52365.1306710167388.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1435375533.50176.1306536707416.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7336) TestFileContextResolveAfs will
 fail with default test.build.data property.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040894#comment-13040894 ] 

Todd Lipcon commented on HADOOP-7336:
-------------------------------------

+1

> TestFileContextResolveAfs will fail with default test.build.data property.
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7336
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7336
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7336.2.patch
>
>
> In TestFileContextResolveAfs if test.build.data property is not set and default is used, the test case will try to create that in the root directory and that will fail. /tmp should be used as default as in many other test cases. Normally, test.build.data will be set and this issue should not occur.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15581-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 30 01:47:28 2011
Return-Path: <common-issues-return-15581-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B9743432E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 30 May 2011 01:47:28 +0000 (UTC)
Received: (qmail 69992 invoked by uid 500); 30 May 2011 01:47:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69961 invoked by uid 500); 30 May 2011 01:47:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69953 invoked by uid 99); 30 May 2011 01:47:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 01:47:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 01:47:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 68220E8FC0
	for <common-issues@hadoop.apache.org>; Mon, 30 May 2011 01:46:47 +0000 (UTC)
Date: Mon, 30 May 2011 01:46:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1260388168.52549.1306720007423.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Updated] (HADOOP-7106) Re-organize hadoop subversion layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7106:
--------------------------------

    Attachment: HADOOP-7106-git.sh
                HADOOP-7106.sh

> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15582-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 30 02:02:30 2011
Return-Path: <common-issues-return-15582-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1A8D962A5
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 30 May 2011 02:02:30 +0000 (UTC)
Received: (qmail 79601 invoked by uid 500); 30 May 2011 02:02:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79476 invoked by uid 500); 30 May 2011 02:02:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79406 invoked by uid 99); 30 May 2011 02:02:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 02:02:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 02:02:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8566FE8232
	for <common-issues@hadoop.apache.org>; Mon, 30 May 2011 02:01:48 +0000 (UTC)
Date: Mon, 30 May 2011 02:01:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1905605821.52599.1306720908543.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7106) Re-organize hadoop subversion
 layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040971#comment-13040971 ] 

Todd Lipcon commented on HADOOP-7106:
-------------------------------------

This afternoon I performed the following tests:

- I set up a local mirror from last month's SVN dump, and then used the ASF's git mirror scripts to create a local git mirror as well.
- Modified Nigel's HADOOP-7106.sh script with following changes:
-- parameterized svn location (the version of svn on my rhel box didn't support the --depth argument)
-- parameterized SVN root, so I could point it at my local mirror
-- fixed the svn externals links to point at hadoop/common/trunk/common instead of hadoop/trunk/common
- Ran Nigel's script and verified that trunk and branch-0.22 had the correct layout
- Committed it to my local svn mirror
- ran the "update-mirror.sh" git mirror script. This took 20 minutes or so as it pulled in all of the history from the new branches. It might take longer upstream.

At this point, the git mirror showed a single commit in trunk that moved all of the files inside common/ and added hdfs/* and mapreduce/* as new files (rather than detecting any kind of merge). This is what I expected

I tried a few things at this point, but ran into some limitations of git: namely, that git won't detect renames that happen as part of a merge commit. So, I took the following angle of attack in a local repo:
- fetch trunk from hadoop-hdfs.git (this is the last commit before 7106 is committed -- since 7106 removed the svn directory, the hdfs.git repo basically got frozen at this point)
- add a new commit where I mv everything inside an hdfs/ directory
- do the same thing for mapreduce
- do the same thing for common (with the commit right before HADOOP-7106
- create a new commit for "trunk" which has the above three branches as parents, and the same log message:

{noformat}
commit 34f047ed9e435be5932d53165477064144f5961c
Merge: 898037a 2ec2d49 0619a1a
Author: Todd Lipcon <todd@cloudera.com>
Date:   Sun May 29 18:35:31 2011 -0700

    HADOOP-7106. Re-organize layout
    
    git-svn-id: file:///data/1/todd/asf-load/hadoop/common/trunk@1098499 13f79535-47bb-0310-9956-ffa450edef68
{noformat}

- verified that commands like "git log -M --follow hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java" properly follow the history through the merge
- force push this new 'trunk' back into hadoop-common.git:refs/remotes/trunk

Next I made some svn commits in the merged repo and verified that the 'update-mirror.sh' script pulled them in on top of the merge with no problems.

A script to perform the above sequence is attached here as HADOOP-7106-git.sh.

In order to do this on ASF, I'll need to have access to the box that does the git mirroring. I guess I need to talk to Infra people to get that.

> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15583-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 30 03:29:32 2011
Return-Path: <common-issues-return-15583-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EBD1B6859
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 30 May 2011 03:29:31 +0000 (UTC)
Received: (qmail 24319 invoked by uid 500); 30 May 2011 03:29:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24195 invoked by uid 500); 30 May 2011 03:29:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24183 invoked by uid 99); 30 May 2011 03:29:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 03:29:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 03:29:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7C53FE8EDE
	for <common-issues@hadoop.apache.org>; Mon, 30 May 2011 03:28:47 +0000 (UTC)
Date: Mon, 30 May 2011 03:28:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1164026560.52642.1306726127506.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1195662636.6130.1300891805766.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7208) equals() and hashCode()
 implementation need to change in StandardSocketFactory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040983#comment-13040983 ] 

Todd Lipcon commented on HADOOP-7208:
-------------------------------------

- There are still hard tabs in this patch.
- there is no contract that hashCode() has to differ if two objects are unequal -- so your unit test there is not valid
- for the implementation of hashcode, might as well use getClass().hashCode() -- you don't need to get the class name


> equals() and hashCode() implementation need to change in StandardSocketFactory
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7208
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7208
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7208-2.patch, HADOOP-7208.patch
>
>
>   In Hadoop IPC Client, we are using ClientCache which will maintain the HashMap to keep the Client references.
> private Map<SocketFactory, Client> clients =
>       new HashMap<SocketFactory, Client>();
>  Now let us say, we want use two standard factories with Hadoop. MyStandardSocketFactory (which extends StandardSocketFactory), and StandardSocketFactory. In this case, because of equals and hashcode implementation, MyStandardSocketFactory client can be overridden by StandardSocketFactoryClient

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15584-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 30 03:29:33 2011
Return-Path: <common-issues-return-15584-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 2CB956866
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 30 May 2011 03:29:33 +0000 (UTC)
Received: (qmail 24504 invoked by uid 500); 30 May 2011 03:29:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24437 invoked by uid 500); 30 May 2011 03:29:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24213 invoked by uid 99); 30 May 2011 03:29:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 03:29:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 03:29:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9FE18E8EE2
	for <common-issues@hadoop.apache.org>; Mon, 30 May 2011 03:28:47 +0000 (UTC)
Date: Mon, 30 May 2011 03:28:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <657986535.52646.1306726127651.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1195662636.6130.1300891805766.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7208) equals() and hashCode()
 implementation need to change in StandardSocketFactory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7208:
--------------------------------

    Status: Open  (was: Patch Available)

> equals() and hashCode() implementation need to change in StandardSocketFactory
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7208
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7208
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7208-2.patch, HADOOP-7208.patch
>
>
>   In Hadoop IPC Client, we are using ClientCache which will maintain the HashMap to keep the Client references.
> private Map<SocketFactory, Client> clients =
>       new HashMap<SocketFactory, Client>();
>  Now let us say, we want use two standard factories with Hadoop. MyStandardSocketFactory (which extends StandardSocketFactory), and StandardSocketFactory. In this case, because of equals and hashcode implementation, MyStandardSocketFactory client can be overridden by StandardSocketFactoryClient

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15585-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 30 04:01:34 2011
Return-Path: <common-issues-return-15585-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EDFC86CFD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 30 May 2011 04:01:33 +0000 (UTC)
Received: (qmail 46642 invoked by uid 500); 30 May 2011 04:01:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46421 invoked by uid 500); 30 May 2011 04:01:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46400 invoked by uid 99); 30 May 2011 04:01:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 04:01:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 04:01:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DE4CAE850A
	for <common-issues@hadoop.apache.org>; Mon, 30 May 2011 04:00:47 +0000 (UTC)
Date: Mon, 30 May 2011 04:00:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <210230351.52692.1306728047907.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1117582463.32721.1304959323395.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7269) S3 Native should allow
 customizable file meta-data (headers)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040993#comment-13040993 ] 

Todd Lipcon commented on HADOOP-7269:
-------------------------------------

Hi Nicholas. Sorry I missed this on the first pass -- it looks like your test case stores a file with metadata, but you don't do any check to make sure that it can be retrieved again with retrieveMetadata. Would you mind adding that?

> S3 Native should allow customizable file meta-data (headers)
> ------------------------------------------------------------
>
>                 Key: HADOOP-7269
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7269
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>         Attachments: HADOOP-7269-S3-metadata-001.diff, HADOOP-7269-S3-metadata-002.diff, HADOOP-7269-S3-metadata-003.diff
>
>
> The S3 Native FileSystem currently writes all files with a set of default headers:
>  * Content-Type: binary/octet-stream
>  * Content-Length: <computed from object size>
>  * Content-MD5: <computed from object data>
> This is a good start, however many applications would benefit from the ability to customize (for example) the Content-Type and Expires headers for the file. Ideally the implementation should be abstract enough to customize all of the available S3 headers and provide a facility for other FileSystems to specify optional file metadata.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15586-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 30 05:27:35 2011
Return-Path: <common-issues-return-15586-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1035265FC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 30 May 2011 05:27:35 +0000 (UTC)
Received: (qmail 81459 invoked by uid 500); 30 May 2011 05:27:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81236 invoked by uid 500); 30 May 2011 05:27:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81221 invoked by uid 99); 30 May 2011 05:27:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 05:27:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 05:27:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 601DEE865A
	for <common-issues@hadoop.apache.org>; Mon, 30 May 2011 05:26:47 +0000 (UTC)
Date: Mon, 30 May 2011 05:26:47 +0000 (UTC)
From: "Min Zhou (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1992675736.52742.1306733207390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7339) Introduce a buffered checksum for
 avoiding frequently calls on Checksum.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Introduce a buffered checksum for avoiding frequently calls on Checksum.update()
--------------------------------------------------------------------------------

                 Key: HADOOP-7339
                 URL: https://issues.apache.org/jira/browse/HADOOP-7339
             Project: Hadoop Common
          Issue Type: Improvement
          Components: util
            Reporter: Min Zhou


We found that PureJavaCRC32/CRC32.update() is the TOP 1 of the methods consuming CPU in a map side, and in reduce side, it cost a lots of CPU too.

IFileOutputStream would frequently call Checksum.update() during writing a record. It's very common a MR key/value less than 512 bytes. Checksum.update() would be called every time writing a key/value. 

Test case: terasort 100MB. 
Checksum.update() calls has be reduced from 4030348 to 28069.  This method is not a hotspot anymore.




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15587-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 30 05:35:34 2011
Return-Path: <common-issues-return-15587-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E47B96B2C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 30 May 2011 05:35:33 +0000 (UTC)
Received: (qmail 85062 invoked by uid 500); 30 May 2011 05:35:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84873 invoked by uid 500); 30 May 2011 05:35:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84865 invoked by uid 99); 30 May 2011 05:35:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 05:35:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 05:35:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 544A4E8830
	for <common-issues@hadoop.apache.org>; Mon, 30 May 2011 05:34:47 +0000 (UTC)
Date: Mon, 30 May 2011 05:34:47 +0000 (UTC)
From: "Min Zhou (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1203876943.52751.1306733687341.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1992675736.52742.1306733207390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7339) Introduce a buffered checksum for
 avoiding frequently calls on Checksum.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Min Zhou updated HADOOP-7339:
-----------------------------

    Fix Version/s: 0.22.0
           Status: Patch Available  (was: Open)

submit a patch for this issue

> Introduce a buffered checksum for avoiding frequently calls on Checksum.update()
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-7339
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7339
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Min Zhou
>             Fix For: 0.22.0
>
>
> We found that PureJavaCRC32/CRC32.update() is the TOP 1 of the methods consuming CPU in a map side, and in reduce side, it cost a lots of CPU too.
> IFileOutputStream would frequently call Checksum.update() during writing a record. It's very common a MR key/value less than 512 bytes. Checksum.update() would be called every time writing a key/value. 
> Test case: terasort 100MB. 
> Checksum.update() calls has be reduced from 4030348 to 28069.  This method is not a hotspot anymore.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15588-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 30 05:37:32 2011
Return-Path: <common-issues-return-15588-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1F3DB6B44
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 30 May 2011 05:37:32 +0000 (UTC)
Received: (qmail 85272 invoked by uid 500); 30 May 2011 05:37:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85236 invoked by uid 500); 30 May 2011 05:37:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85228 invoked by uid 99); 30 May 2011 05:37:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 05:37:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 05:37:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 86BBEE88B8
	for <common-issues@hadoop.apache.org>; Mon, 30 May 2011 05:36:47 +0000 (UTC)
Date: Mon, 30 May 2011 05:36:47 +0000 (UTC)
From: "Min Zhou (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1458416602.52753.1306733807548.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1992675736.52742.1306733207390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7339) Introduce a buffered checksum for
 avoiding frequently calls on Checksum.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Min Zhou updated HADOOP-7339:
-----------------------------

    Attachment: HADOOP-7339-v1.diff

submit a patch for this issue

> Introduce a buffered checksum for avoiding frequently calls on Checksum.update()
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-7339
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7339
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Min Zhou
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7339-v1.diff
>
>
> We found that PureJavaCRC32/CRC32.update() is the TOP 1 of the methods consuming CPU in a map side, and in reduce side, it cost a lots of CPU too.
> IFileOutputStream would frequently call Checksum.update() during writing a record. It's very common a MR key/value less than 512 bytes. Checksum.update() would be called every time writing a key/value. 
> Test case: terasort 100MB. 
> Checksum.update() calls has be reduced from 4030348 to 28069.  This method is not a hotspot anymore.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15589-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 30 05:43:30 2011
Return-Path: <common-issues-return-15589-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0AB696B5D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 30 May 2011 05:43:30 +0000 (UTC)
Received: (qmail 85844 invoked by uid 500); 30 May 2011 05:43:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85732 invoked by uid 500); 30 May 2011 05:43:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85723 invoked by uid 99); 30 May 2011 05:43:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 05:43:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 05:43:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 65931E89E0
	for <common-issues@hadoop.apache.org>; Mon, 30 May 2011 05:42:47 +0000 (UTC)
Date: Mon, 30 May 2011 05:42:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <640983355.52757.1306734167412.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1992675736.52742.1306733207390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7339) Introduce a buffered checksum for
 avoiding frequently calls on Checksum.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041006#comment-13041006 ] 

Hadoop QA commented on HADOOP-7339:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480818/HADOOP-7339-v1.diff
  against trunk revision 1128789.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/538//console

This message is automatically generated.

> Introduce a buffered checksum for avoiding frequently calls on Checksum.update()
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-7339
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7339
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Min Zhou
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7339-v1.diff
>
>
> We found that PureJavaCRC32/CRC32.update() is the TOP 1 of the methods consuming CPU in a map side, and in reduce side, it cost a lots of CPU too.
> IFileOutputStream would frequently call Checksum.update() during writing a record. It's very common a MR key/value less than 512 bytes. Checksum.update() would be called every time writing a key/value. 
> Test case: terasort 100MB. 
> Checksum.update() calls has be reduced from 4030348 to 28069.  This method is not a hotspot anymore.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15590-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 30 06:03:33 2011
Return-Path: <common-issues-return-15590-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 176DB615F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 30 May 2011 06:03:33 +0000 (UTC)
Received: (qmail 96064 invoked by uid 500); 30 May 2011 06:03:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95943 invoked by uid 500); 30 May 2011 06:03:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95935 invoked by uid 99); 30 May 2011 06:03:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 06:03:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 06:03:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5B2A1E8E65
	for <common-issues@hadoop.apache.org>; Mon, 30 May 2011 06:02:47 +0000 (UTC)
Date: Mon, 30 May 2011 06:02:47 +0000 (UTC)
From: "Min Zhou (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <852334824.52781.1306735367370.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1992675736.52742.1306733207390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7339) Introduce a buffered checksum for
 avoiding frequently calls on Checksum.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Min Zhou updated HADOOP-7339:
-----------------------------

    Attachment: HADOOP-7339-v2.diff

fixed svn conflicts.

> Introduce a buffered checksum for avoiding frequently calls on Checksum.update()
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-7339
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7339
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Min Zhou
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7339-v1.diff, HADOOP-7339-v2.diff
>
>
> We found that PureJavaCRC32/CRC32.update() is the TOP 1 of the methods consuming CPU in a map side, and in reduce side, it cost a lots of CPU too.
> IFileOutputStream would frequently call Checksum.update() during writing a record. It's very common a MR key/value less than 512 bytes. Checksum.update() would be called every time writing a key/value. 
> Test case: terasort 100MB. 
> Checksum.update() calls has be reduced from 4030348 to 28069.  This method is not a hotspot anymore.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15591-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 30 06:07:30 2011
Return-Path: <common-issues-return-15591-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0BECF664A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 30 May 2011 06:07:30 +0000 (UTC)
Received: (qmail 99640 invoked by uid 500); 30 May 2011 06:07:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99615 invoked by uid 500); 30 May 2011 06:07:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99607 invoked by uid 99); 30 May 2011 06:07:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 06:07:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 06:07:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 583FEE8F40
	for <common-issues@hadoop.apache.org>; Mon, 30 May 2011 06:06:47 +0000 (UTC)
Date: Mon, 30 May 2011 06:06:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <545185438.52783.1306735607358.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1992675736.52742.1306733207390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7339) Introduce a buffered checksum for
 avoiding frequently calls on Checksum.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041010#comment-13041010 ] 

Todd Lipcon commented on HADOOP-7339:
-------------------------------------

I understand how this could make a big difference using the builtin CRC32. But, with PureJavaCrc32, there shouldn't be any particularly large advantage to buffering like this.

Did you see a speedup in terasort performance? Just seeing fewer calls doesn't necessarily mean it's an improvement.

> Introduce a buffered checksum for avoiding frequently calls on Checksum.update()
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-7339
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7339
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Min Zhou
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7339-v1.diff, HADOOP-7339-v2.diff
>
>
> We found that PureJavaCRC32/CRC32.update() is the TOP 1 of the methods consuming CPU in a map side, and in reduce side, it cost a lots of CPU too.
> IFileOutputStream would frequently call Checksum.update() during writing a record. It's very common a MR key/value less than 512 bytes. Checksum.update() would be called every time writing a key/value. 
> Test case: terasort 100MB. 
> Checksum.update() calls has be reduced from 4030348 to 28069.  This method is not a hotspot anymore.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15592-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 30 06:09:31 2011
Return-Path: <common-issues-return-15592-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 4AF9F6669
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 30 May 2011 06:09:31 +0000 (UTC)
Received: (qmail 125 invoked by uid 500); 30 May 2011 06:09:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99997 invoked by uid 500); 30 May 2011 06:09:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99985 invoked by uid 99); 30 May 2011 06:09:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 06:09:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 06:09:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5CFE0E8FE1
	for <common-issues@hadoop.apache.org>; Mon, 30 May 2011 06:08:47 +0000 (UTC)
Date: Mon, 30 May 2011 06:08:47 +0000 (UTC)
From: "Min Zhou (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <475994145.52785.1306735727377.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1992675736.52742.1306733207390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7339) Introduce a buffered checksum for
 avoiding frequently calls on Checksum.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041011#comment-13041011 ] 

Min Zhou commented on HADOOP-7339:
----------------------------------

Orginal source file in svn has many DOS characters which would caused a failure when appling this patch.

> Introduce a buffered checksum for avoiding frequently calls on Checksum.update()
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-7339
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7339
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Min Zhou
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7339-v1.diff, HADOOP-7339-v2.diff
>
>
> We found that PureJavaCRC32/CRC32.update() is the TOP 1 of the methods consuming CPU in a map side, and in reduce side, it cost a lots of CPU too.
> IFileOutputStream would frequently call Checksum.update() during writing a record. It's very common a MR key/value less than 512 bytes. Checksum.update() would be called every time writing a key/value. 
> Test case: terasort 100MB. 
> Checksum.update() calls has be reduced from 4030348 to 28069.  This method is not a hotspot anymore.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15593-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 30 06:25:36 2011
Return-Path: <common-issues-return-15593-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8697E6BB6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 30 May 2011 06:25:36 +0000 (UTC)
Received: (qmail 9598 invoked by uid 500); 30 May 2011 06:25:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9411 invoked by uid 500); 30 May 2011 06:25:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9387 invoked by uid 99); 30 May 2011 06:25:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 06:25:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 06:25:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 55759E8447
	for <common-issues@hadoop.apache.org>; Mon, 30 May 2011 06:24:47 +0000 (UTC)
Date: Mon, 30 May 2011 06:24:47 +0000 (UTC)
From: "Min Zhou (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1979466764.52793.1306736687346.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1992675736.52742.1306733207390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7339) Introduce a buffered checksum for
 avoiding frequently calls on Checksum.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041012#comment-13041012 ] 

Min Zhou commented on HADOOP-7339:
----------------------------------

@Todd

I don't think this patch can speed up terasort. This method is not a bottleneck of a MR job. Random seeks during the shuffle stage should be the bottle bottleneck. 
This patch helps reducing CPU consuming, which i think can also be ragarded as an improvement. 

> Introduce a buffered checksum for avoiding frequently calls on Checksum.update()
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-7339
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7339
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Min Zhou
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7339-v1.diff, HADOOP-7339-v2.diff
>
>
> We found that PureJavaCRC32/CRC32.update() is the TOP 1 of the methods consuming CPU in a map side, and in reduce side, it cost a lots of CPU too.
> IFileOutputStream would frequently call Checksum.update() during writing a record. It's very common a MR key/value less than 512 bytes. Checksum.update() would be called every time writing a key/value. 
> Test case: terasort 100MB. 
> Checksum.update() calls has be reduced from 4030348 to 28069.  This method is not a hotspot anymore.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15594-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 30 06:31:33 2011
Return-Path: <common-issues-return-15594-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E86B1640E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 30 May 2011 06:31:33 +0000 (UTC)
Received: (qmail 11848 invoked by uid 500); 30 May 2011 06:31:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11638 invoked by uid 500); 30 May 2011 06:31:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11627 invoked by uid 99); 30 May 2011 06:31:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 06:31:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 06:31:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id AE0FDE8633
	for <common-issues@hadoop.apache.org>; Mon, 30 May 2011 06:30:47 +0000 (UTC)
Date: Mon, 30 May 2011 06:30:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <85338105.52804.1306737047709.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1992675736.52742.1306733207390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7339) Introduce a buffered checksum for
 avoiding frequently calls on Checksum.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041014#comment-13041014 ] 

Hadoop QA commented on HADOOP-7339:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480819/HADOOP-7339-v2.diff
  against trunk revision 1128789.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/539//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/539//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/539//console

This message is automatically generated.

> Introduce a buffered checksum for avoiding frequently calls on Checksum.update()
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-7339
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7339
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Min Zhou
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7339-v1.diff, HADOOP-7339-v2.diff
>
>
> We found that PureJavaCRC32/CRC32.update() is the TOP 1 of the methods consuming CPU in a map side, and in reduce side, it cost a lots of CPU too.
> IFileOutputStream would frequently call Checksum.update() during writing a record. It's very common a MR key/value less than 512 bytes. Checksum.update() would be called every time writing a key/value. 
> Test case: terasort 100MB. 
> Checksum.update() calls has be reduced from 4030348 to 28069.  This method is not a hotspot anymore.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15595-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 30 12:17:32 2011
Return-Path: <common-issues-return-15595-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1E3F26606
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 30 May 2011 12:17:32 +0000 (UTC)
Received: (qmail 7533 invoked by uid 500); 30 May 2011 12:17:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7479 invoked by uid 500); 30 May 2011 12:17:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7471 invoked by uid 99); 30 May 2011 12:17:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 12:17:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 12:17:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 910E7E865E
	for <common-issues@hadoop.apache.org>; Mon, 30 May 2011 12:16:47 +0000 (UTC)
Date: Mon, 30 May 2011 12:16:47 +0000 (UTC)
From: "HariSree (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <589599883.53275.1306757807590.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-702) DFS Upgrade Proposal
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041105#comment-13041105 ] 

HariSree commented on HADOOP-702:
---------------------------------

What is the expectation in the following scenario : 

Namenode has 3 name dirs configured ..
1) Namenode upgrade starts - Upgrade fails after 1st directory is upgraded  (2nd and 3rd dir is left unchanged ..)
2) Namenode starts 
3) Namenode shutdown and rollbacked

Are the new changes done meant to be visible ? With current implementation , after a rollback new changes are visible .. Is this expected ??

As per my observation , since Namenode is saving the latest image dir(the upgraded 1st dir since checkpointtime is incremented during upgrade for this dir) will be loaded and saved to all dirs during loadfsimage  .. 

But if a ROLLBACK is done , the 1st dir will be rolled back (the older copy becomes current and its checkpointtime is now LESS than other dirs ..) and others left behind since they dont contain previous .. Now during loadfsimage , the 2nd dir will be selected since it has the highest checkpoint time and saved to all dirs (including 1st ) .. Now due to this , the new changes b/w UPGRADE and ROLLBACK present in 2nd dir gets reflected even after ROLLBACK ..

Is this expected ? I ask this since  : 

1) this is not the case with a SUCCESSFULL Upgrade/Rollback (New changes lost after rollback).. & 
2) FSStateTransition7.htm says { r0. if previous directory does not exist then fail rollback } , but it continues in the current implementation .. 


> DFS Upgrade Proposal
> --------------------
>
>                 Key: HADOOP-702
>                 URL: https://issues.apache.org/jira/browse/HADOOP-702
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: 0.13.0
>
>         Attachments: DFSUpgradeProposal3.html, FSStateTransition6.htm, FSStateTransition7.htm, FSStateTransitionApr03.patch, Manual-TestCases-HdfsConversion.txt, TestPlan-HdfsUpgrade.html, TestPlan-HdfsUpgrade.html, TestPlan-HdfsUpgrade.html
>
>
> Currently the DFS cluster upgrade procedure is manual.
> http://wiki.apache.org/lucene-hadoop/Hadoop_Upgrade
> It is rather complicated and does not guarantee data recoverability in case of software errors or administrator mistakes.
> This is a description of utilities that make the upgrade process almost automatic and minimize chance of loosing or corrupting data.
> Please see the attached html file for details.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15596-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 30 16:30:29 2011
Return-Path: <common-issues-return-15596-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CA69D6D21
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 30 May 2011 16:30:29 +0000 (UTC)
Received: (qmail 71449 invoked by uid 500); 30 May 2011 16:30:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71388 invoked by uid 500); 30 May 2011 16:30:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71380 invoked by uid 99); 30 May 2011 16:30:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 16:30:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 16:30:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5E8D1E9971
	for <common-issues@hadoop.apache.org>; Mon, 30 May 2011 16:29:47 +0000 (UTC)
Date: Mon, 30 May 2011 16:29:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <832868218.53721.1306772987383.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1992675736.52742.1306733207390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7339) Introduce a buffered checksum for
 avoiding frequently calls on Checksum.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041190#comment-13041190 ] 

Todd Lipcon commented on HADOOP-7339:
-------------------------------------

Hi Min. Maybe you could make a microbenchmark which uses IFile.Writer to write a few GB to a path in /dev/shm? This should cut IO out of the equation and give us an idea of the CPU reduction.

> Introduce a buffered checksum for avoiding frequently calls on Checksum.update()
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-7339
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7339
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Min Zhou
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7339-v1.diff, HADOOP-7339-v2.diff
>
>
> We found that PureJavaCRC32/CRC32.update() is the TOP 1 of the methods consuming CPU in a map side, and in reduce side, it cost a lots of CPU too.
> IFileOutputStream would frequently call Checksum.update() during writing a record. It's very common a MR key/value less than 512 bytes. Checksum.update() would be called every time writing a key/value. 
> Test case: terasort 100MB. 
> Checksum.update() calls has be reduced from 4030348 to 28069.  This method is not a hotspot anymore.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15597-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Mon May 30 19:22:29 2011
Return-Path: <common-issues-return-15597-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0925969F2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Mon, 30 May 2011 19:22:29 +0000 (UTC)
Received: (qmail 20570 invoked by uid 500); 30 May 2011 19:22:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20539 invoked by uid 500); 30 May 2011 19:22:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20531 invoked by uid 99); 30 May 2011 19:22:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 19:22:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 May 2011 19:22:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 973F7E997B
	for <common-issues@hadoop.apache.org>; Mon, 30 May 2011 19:21:47 +0000 (UTC)
Date: Mon, 30 May 2011 19:21:47 +0000 (UTC)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1848775783.53995.1306783307616.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1992675736.52742.1306733207390.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7339) Introduce a buffered checksum for
 avoiding frequently calls on Checksum.update()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041242#comment-13041242 ] 

Hong Tang commented on HADOOP-7339:
-----------------------------------

@min, you may minimize the impact of shuffle seek overhead by limiting the amount of input data, and choosing a small reducer number. The effect of CPU saturation may be more pronounced if you set the # of map slots > # of cores, and you use GZIP for intermediate data compression.

> Introduce a buffered checksum for avoiding frequently calls on Checksum.update()
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-7339
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7339
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Min Zhou
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7339-v1.diff, HADOOP-7339-v2.diff
>
>
> We found that PureJavaCRC32/CRC32.update() is the TOP 1 of the methods consuming CPU in a map side, and in reduce side, it cost a lots of CPU too.
> IFileOutputStream would frequently call Checksum.update() during writing a record. It's very common a MR key/value less than 512 bytes. Checksum.update() would be called every time writing a key/value. 
> Test case: terasort 100MB. 
> Checksum.update() calls has be reduced from 4030348 to 28069.  This method is not a hotspot anymore.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15598-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 07:27:37 2011
Return-Path: <common-issues-return-15598-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DDC834B48
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 07:27:37 +0000 (UTC)
Received: (qmail 66054 invoked by uid 500); 31 May 2011 07:27:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66026 invoked by uid 500); 31 May 2011 07:27:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66016 invoked by uid 99); 31 May 2011 07:27:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 07:27:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 07:27:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BE2E4EA43C
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 07:26:47 +0000 (UTC)
Date: Tue, 31 May 2011 07:26:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1586184196.55471.1306826807775.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7284:
---------------------------------

    Status: Open  (was: Patch Available)

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch, trash8.patch, trash9.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15599-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 07:29:33 2011
Return-Path: <common-issues-return-15599-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 267754B65
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 07:29:33 +0000 (UTC)
Received: (qmail 66511 invoked by uid 500); 31 May 2011 07:29:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66461 invoked by uid 500); 31 May 2011 07:29:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66445 invoked by uid 99); 31 May 2011 07:29:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 07:29:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 07:29:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id C646BEA575
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 07:28:47 +0000 (UTC)
Date: Tue, 31 May 2011 07:28:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <493199464.55480.1306826927809.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7284:
---------------------------------

    Status: Patch Available  (was: Open)

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash10.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch, trash8.patch, trash9.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15600-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 07:29:34 2011
Return-Path: <common-issues-return-15600-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 137F64B7D
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 07:29:34 +0000 (UTC)
Received: (qmail 66788 invoked by uid 500); 31 May 2011 07:29:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66478 invoked by uid 500); 31 May 2011 07:29:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66466 invoked by uid 99); 31 May 2011 07:29:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 07:29:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 07:29:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A463DEA571
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 07:28:47 +0000 (UTC)
Date: Tue, 31 May 2011 07:28:47 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2014952348.55476.1306826927670.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-7284:
---------------------------------

    Attachment: trash10.patch

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash10.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch, trash8.patch, trash9.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15601-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 07:45:37 2011
Return-Path: <common-issues-return-15601-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E906D4951
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 07:45:36 +0000 (UTC)
Received: (qmail 83563 invoked by uid 500); 31 May 2011 07:45:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82969 invoked by uid 500); 31 May 2011 07:45:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82945 invoked by uid 99); 31 May 2011 07:45:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 07:45:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 07:45:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 7C470EAC05
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 07:44:47 +0000 (UTC)
Date: Tue, 31 May 2011 07:44:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <148651276.55509.1306827887505.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041470#comment-13041470 ] 

Hadoop QA commented on HADOOP-7284:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480908/trash10.patch
  against trunk revision 1128789.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 17 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/540//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/540//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/540//console

This message is automatically generated.

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash10.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch, trash8.patch, trash9.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15602-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 10:09:32 2011
Return-Path: <common-issues-return-15602-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 269654F44
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 10:09:32 +0000 (UTC)
Received: (qmail 74350 invoked by uid 500); 31 May 2011 10:09:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74311 invoked by uid 500); 31 May 2011 10:09:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74303 invoked by uid 99); 31 May 2011 10:09:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 10:09:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 10:09:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 61D5FEB069
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 10:08:47 +0000 (UTC)
Date: Tue, 31 May 2011 10:08:47 +0000 (UTC)
From: "XieXianshan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1517249960.55703.1306836527397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6387) FsShell -getmerge source file
 pattern is broken
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

XieXianshan updated HADOOP-6387:
--------------------------------

    Attachment: HADOOP-6387.patch

Make "source file pattern" work.

> FsShell -getmerge source file pattern is broken
> -----------------------------------------------
>
>                 Key: HADOOP-6387
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6387
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Eli Collins
>            Priority: Minor
>         Attachments: HADOOP-6387.patch
>
>
> The FsShell -getmerge command doesn't work if the "source file pattern" matches files. See below. If the current behavior is intended then we should update the help documentation and java docs to match, but it would be nice if the user could specify a set of files in a directory rather than just directories.
> {code}
> $ hadoop fs -help getmerge
> -getmerge <src> <localdst>:  Get all the files in the directories that 
> 		match the source file pattern and merge and sort them to only
> 		one file on local fs. <src> is kept.
> $ hadoop fs -ls
> Found 3 items
> -rw-r--r--   1 eli supergroup          2 2009-11-23 17:39 /user/eli/1.txt
> -rw-r--r--   1 eli supergroup          2 2009-11-23 17:39 /user/eli/2.txt
> -rw-r--r--   1 eli supergroup          2 2009-11-23 17:39 /user/eli/3.txt
> $ hadoop fs -getmerge /user/eli/*.txt sorted.txt
> $ cat sorted.txt
> cat: sorted.txt: No such file or directory
> $ hadoop fs -getmerge /user/eli/* sorted.txt
> $ cat sorted.txt
> cat: sorted.txt: No such file or directory
> $ hadoop fs -getmerge /user/* sorted.txt
> $ cat sorted.txt 
> 1
> 2
> 3
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15603-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 10:16:29 2011
Return-Path: <common-issues-return-15603-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EB68C447C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 10:16:28 +0000 (UTC)
Received: (qmail 83214 invoked by uid 500); 31 May 2011 10:16:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83170 invoked by uid 500); 31 May 2011 10:16:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83162 invoked by uid 99); 31 May 2011 10:16:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 10:16:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 10:16:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B450DEB2A7
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 10:15:47 +0000 (UTC)
Date: Tue, 31 May 2011 10:15:47 +0000 (UTC)
From: "XieXianshan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1821829737.55729.1306836947735.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-6387) FsShell -getmerge source file
 pattern is broken
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

XieXianshan updated HADOOP-6387:
--------------------------------

        Fix Version/s: 0.23.0
             Assignee: XieXianshan
    Affects Version/s: 0.23.0
               Status: Patch Available  (was: Open)

this patch make "source file pattern" work.

> FsShell -getmerge source file pattern is broken
> -----------------------------------------------
>
>                 Key: HADOOP-6387
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6387
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Eli Collins
>            Assignee: XieXianshan
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-6387.patch
>
>
> The FsShell -getmerge command doesn't work if the "source file pattern" matches files. See below. If the current behavior is intended then we should update the help documentation and java docs to match, but it would be nice if the user could specify a set of files in a directory rather than just directories.
> {code}
> $ hadoop fs -help getmerge
> -getmerge <src> <localdst>:  Get all the files in the directories that 
> 		match the source file pattern and merge and sort them to only
> 		one file on local fs. <src> is kept.
> $ hadoop fs -ls
> Found 3 items
> -rw-r--r--   1 eli supergroup          2 2009-11-23 17:39 /user/eli/1.txt
> -rw-r--r--   1 eli supergroup          2 2009-11-23 17:39 /user/eli/2.txt
> -rw-r--r--   1 eli supergroup          2 2009-11-23 17:39 /user/eli/3.txt
> $ hadoop fs -getmerge /user/eli/*.txt sorted.txt
> $ cat sorted.txt
> cat: sorted.txt: No such file or directory
> $ hadoop fs -getmerge /user/eli/* sorted.txt
> $ cat sorted.txt
> cat: sorted.txt: No such file or directory
> $ hadoop fs -getmerge /user/* sorted.txt
> $ cat sorted.txt 
> 1
> 2
> 3
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15604-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 10:28:28 2011
Return-Path: <common-issues-return-15604-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DD23E4A00
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 10:28:28 +0000 (UTC)
Received: (qmail 1541 invoked by uid 500); 31 May 2011 10:28:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1508 invoked by uid 500); 31 May 2011 10:28:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1500 invoked by uid 99); 31 May 2011 10:28:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 10:28:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 10:28:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 611FCEB5CE
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 10:27:47 +0000 (UTC)
Date: Tue, 31 May 2011 10:27:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <944188212.55742.1306837667394.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6387) FsShell -getmerge source file
 pattern is broken
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041516#comment-13041516 ] 

Hadoop QA commented on HADOOP-6387:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480923/HADOOP-6387.patch
  against trunk revision 1128789.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The patch appears to cause tar ant target to fail.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:


    -1 system test framework.  The patch failed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/541//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/541//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/541//console

This message is automatically generated.

> FsShell -getmerge source file pattern is broken
> -----------------------------------------------
>
>                 Key: HADOOP-6387
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6387
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Eli Collins
>            Assignee: XieXianshan
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-6387.patch
>
>
> The FsShell -getmerge command doesn't work if the "source file pattern" matches files. See below. If the current behavior is intended then we should update the help documentation and java docs to match, but it would be nice if the user could specify a set of files in a directory rather than just directories.
> {code}
> $ hadoop fs -help getmerge
> -getmerge <src> <localdst>:  Get all the files in the directories that 
> 		match the source file pattern and merge and sort them to only
> 		one file on local fs. <src> is kept.
> $ hadoop fs -ls
> Found 3 items
> -rw-r--r--   1 eli supergroup          2 2009-11-23 17:39 /user/eli/1.txt
> -rw-r--r--   1 eli supergroup          2 2009-11-23 17:39 /user/eli/2.txt
> -rw-r--r--   1 eli supergroup          2 2009-11-23 17:39 /user/eli/3.txt
> $ hadoop fs -getmerge /user/eli/*.txt sorted.txt
> $ cat sorted.txt
> cat: sorted.txt: No such file or directory
> $ hadoop fs -getmerge /user/eli/* sorted.txt
> $ cat sorted.txt
> cat: sorted.txt: No such file or directory
> $ hadoop fs -getmerge /user/* sorted.txt
> $ cat sorted.txt 
> 1
> 2
> 3
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15605-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 10:30:29 2011
Return-Path: <common-issues-return-15605-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9784B4C56
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 10:30:29 +0000 (UTC)
Received: (qmail 2012 invoked by uid 500); 31 May 2011 10:30:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1980 invoked by uid 500); 31 May 2011 10:30:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1970 invoked by uid 99); 31 May 2011 10:30:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 10:30:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 10:30:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6EF1FEB668
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 10:29:48 +0000 (UTC)
Date: Tue, 31 May 2011 10:29:48 +0000 (UTC)
From: "XieXianshan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <53044194.55753.1306837788451.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7340) incomplete help message is displayed
 for getmerge [addnl] option
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

incomplete help message is displayed for getmerge [addnl] option
----------------------------------------------------------------

                 Key: HADOOP-7340
                 URL: https://issues.apache.org/jira/browse/HADOOP-7340
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs
    Affects Versions: 0.23.0
            Reporter: XieXianshan
            Assignee: XieXianshan
             Fix For: 0.23.0


The help message for the command "hdfs dfs -help getmerge" is displayed like this:
"-getmerge <src> <localdst> [addnl]:	Get all the files in the directories that
		match the source file pattern and merge and sort them to only
		one file on local fs. <src> is kept."
and the information about [addnl] option is missed,despite the fact that [addnl] option is implemented.

Therefore,the expected message should be displayed like this:
"-getmerge <src> <localdst> [addnl]:	Get all the files in the directories that
		match the source file pattern and merge and sort them to only
		one file on local fs. <src> is kept.
		  addnl   Optionally addnl can be set to enable adding a newline 
		          character at the end of each file."

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15606-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 10:32:30 2011
Return-Path: <common-issues-return-15606-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 99DAE4647
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 10:32:30 +0000 (UTC)
Received: (qmail 4157 invoked by uid 500); 31 May 2011 10:32:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4115 invoked by uid 500); 31 May 2011 10:32:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4107 invoked by uid 99); 31 May 2011 10:32:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 10:32:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 10:32:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 56671EB6EC
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 10:31:47 +0000 (UTC)
Date: Tue, 31 May 2011 10:31:47 +0000 (UTC)
From: "XieXianshan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <460389824.55755.1306837907350.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <53044194.55753.1306837788451.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7340) incomplete help message is displayed
 for getmerge [addnl] option
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

XieXianshan updated HADOOP-7340:
--------------------------------

    Attachment: HADOOP-7340

> incomplete help message is displayed for getmerge [addnl] option
> ----------------------------------------------------------------
>
>                 Key: HADOOP-7340
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7340
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Assignee: XieXianshan
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7340
>
>
> The help message for the command "hdfs dfs -help getmerge" is displayed like this:
> "-getmerge <src> <localdst> [addnl]:	Get all the files in the directories that
> 		match the source file pattern and merge and sort them to only
> 		one file on local fs. <src> is kept."
> and the information about [addnl] option is missed,despite the fact that [addnl] option is implemented.
> Therefore,the expected message should be displayed like this:
> "-getmerge <src> <localdst> [addnl]:	Get all the files in the directories that
> 		match the source file pattern and merge and sort them to only
> 		one file on local fs. <src> is kept.
> 		  addnl   Optionally addnl can be set to enable adding a newline 
> 		          character at the end of each file."

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15607-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 10:36:28 2011
Return-Path: <common-issues-return-15607-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BC3F7433A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 10:36:28 +0000 (UTC)
Received: (qmail 8963 invoked by uid 500); 31 May 2011 10:36:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8918 invoked by uid 500); 31 May 2011 10:36:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8910 invoked by uid 99); 31 May 2011 10:36:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 10:36:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 10:36:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 71131EB806
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 10:35:47 +0000 (UTC)
Date: Tue, 31 May 2011 10:35:47 +0000 (UTC)
From: "XieXianshan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1316420138.55759.1306838147459.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <53044194.55753.1306837788451.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7340) incomplete help message is displayed
 for getmerge [addnl] option
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

XieXianshan updated HADOOP-7340:
--------------------------------

    Status: Patch Available  (was: Open)

Improve help message for fsShell "getmerge".

> incomplete help message is displayed for getmerge [addnl] option
> ----------------------------------------------------------------
>
>                 Key: HADOOP-7340
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7340
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Assignee: XieXianshan
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7340
>
>
> The help message for the command "hdfs dfs -help getmerge" is displayed like this:
> "-getmerge <src> <localdst> [addnl]:	Get all the files in the directories that
> 		match the source file pattern and merge and sort them to only
> 		one file on local fs. <src> is kept."
> and the information about [addnl] option is missed,despite the fact that [addnl] option is implemented.
> Therefore,the expected message should be displayed like this:
> "-getmerge <src> <localdst> [addnl]:	Get all the files in the directories that
> 		match the source file pattern and merge and sort them to only
> 		one file on local fs. <src> is kept.
> 		  addnl   Optionally addnl can be set to enable adding a newline 
> 		          character at the end of each file."

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15608-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 10:55:28 2011
Return-Path: <common-issues-return-15608-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C5AF141A8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 10:55:28 +0000 (UTC)
Received: (qmail 34027 invoked by uid 500); 31 May 2011 10:55:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33999 invoked by uid 500); 31 May 2011 10:55:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33989 invoked by uid 99); 31 May 2011 10:55:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 10:55:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 10:55:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 560CAEBDD3
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 10:54:47 +0000 (UTC)
Date: Tue, 31 May 2011 10:54:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1897195668.55780.1306839287348.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <53044194.55753.1306837788451.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7340) incomplete help message is
 displayed for getmerge [addnl] option
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041522#comment-13041522 ] 

Hadoop QA commented on HADOOP-7340:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480925/HADOOP-7340
  against trunk revision 1128789.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/542//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/542//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/542//console

This message is automatically generated.

> incomplete help message is displayed for getmerge [addnl] option
> ----------------------------------------------------------------
>
>                 Key: HADOOP-7340
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7340
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Assignee: XieXianshan
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7340
>
>
> The help message for the command "hdfs dfs -help getmerge" is displayed like this:
> "-getmerge <src> <localdst> [addnl]:	Get all the files in the directories that
> 		match the source file pattern and merge and sort them to only
> 		one file on local fs. <src> is kept."
> and the information about [addnl] option is missed,despite the fact that [addnl] option is implemented.
> Therefore,the expected message should be displayed like this:
> "-getmerge <src> <localdst> [addnl]:	Get all the files in the directories that
> 		match the source file pattern and merge and sort them to only
> 		one file on local fs. <src> is kept.
> 		  addnl   Optionally addnl can be set to enable adding a newline 
> 		          character at the end of each file."

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15609-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 11:16:32 2011
Return-Path: <common-issues-return-15609-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 13A43426C
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 11:16:32 +0000 (UTC)
Received: (qmail 54932 invoked by uid 500); 31 May 2011 11:16:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54899 invoked by uid 500); 31 May 2011 11:16:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54891 invoked by uid 99); 31 May 2011 11:16:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 11:16:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 11:16:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 66F7CEB419
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 11:15:47 +0000 (UTC)
Date: Tue, 31 May 2011 11:15:47 +0000 (UTC)
From: "Harsh J Chouraria (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <755509231.55802.1306840547418.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <53044194.55753.1306837788451.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7340) incomplete help message is
 displayed for getmerge [addnl] option
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041528#comment-13041528 ] 

Harsh J Chouraria commented on HADOOP-7340:
-------------------------------------------

I think the text can still be improved. For instance, reading that line still doesn't help me (who's never used 'addnl' before) understand _what_ it ought to be set to. Should I type {{-getmerge path1 path2 addnl}} or should I type {{-getmerge path1 path2 '\n'}}?

> incomplete help message is displayed for getmerge [addnl] option
> ----------------------------------------------------------------
>
>                 Key: HADOOP-7340
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7340
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Assignee: XieXianshan
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7340
>
>
> The help message for the command "hdfs dfs -help getmerge" is displayed like this:
> "-getmerge <src> <localdst> [addnl]:	Get all the files in the directories that
> 		match the source file pattern and merge and sort them to only
> 		one file on local fs. <src> is kept."
> and the information about [addnl] option is missed,despite the fact that [addnl] option is implemented.
> Therefore,the expected message should be displayed like this:
> "-getmerge <src> <localdst> [addnl]:	Get all the files in the directories that
> 		match the source file pattern and merge and sort them to only
> 		one file on local fs. <src> is kept.
> 		  addnl   Optionally addnl can be set to enable adding a newline 
> 		          character at the end of each file."

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15610-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 15:04:32 2011
Return-Path: <common-issues-return-15610-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 29B034834
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 15:04:32 +0000 (UTC)
Received: (qmail 81192 invoked by uid 500); 31 May 2011 15:04:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81146 invoked by uid 500); 31 May 2011 15:04:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81138 invoked by uid 99); 31 May 2011 15:04:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 15:04:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 15:04:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 70748EB3F4
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 15:03:47 +0000 (UTC)
Date: Tue, 31 May 2011 15:03:47 +0000 (UTC)
From: "ramkrishna.s.vasudevan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <413224117.56199.1306854227457.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <282223217.18947.1304436003652.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7256) Resource leak during failure
 scenario of closing of resources.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ramkrishna.s.vasudevan updated HADOOP-7256:
-------------------------------------------

    Attachment: HADOOP-7256-patch-2.patch

> Resource leak during failure scenario of closing of resources. 
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7256
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7256
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2, 0.21.0
>            Reporter: ramkrishna.s.vasudevan
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7256-patch-1.patch, HADOOP-7256-patch-2.patch
>
>   Original Estimate: 8h
>  Remaining Estimate: 8h
>
> Problem Statement:
> ===============
> There are chances of resource leak and stream not getting closed 
> Take the case when after copying data we try to close the Input and output stream followed by closing of the socket.
> Suppose an exception occurs while closing the input stream(due to runtime exception) then the subsequent operations of closing the output stream and socket may not happen and there is a chance of resource leak.
> Scenario 
> =======
> During long run of map reduce jobs, the copyFromLocalFile() api is getting called.
> Here we found some exceptions happening. As a result of this we found the lsof value raising leading to resource leak.
> Solution:
> =======
> While doing a close operation of any resource catch the RuntimeException also rather than catching the IOException alone.
> Additionally there are places where we try to close a resource in the catch block.
> If this close fails, we just throw and come out of the current flow.
> In order to avoid this, we can carry out the close operation in the finally block.
> Probable reasons for getting RunTimeExceptions:
> =====================================
> We may get runtime exception from customised hadoop streams like FSDataOutputStream.close() . So better to handle RunTimeExceptions also.
>  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15611-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 15:05:37 2011
Return-Path: <common-issues-return-15611-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 43A6548A2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 15:05:37 +0000 (UTC)
Received: (qmail 83934 invoked by uid 500); 31 May 2011 15:05:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83899 invoked by uid 500); 31 May 2011 15:05:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83890 invoked by uid 99); 31 May 2011 15:05:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 15:05:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 15:05:32 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9AD4BEB4B1
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 15:04:52 +0000 (UTC)
Date: Tue, 31 May 2011 15:04:52 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <51189024.56205.1306854292631.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1195662636.6130.1300891805766.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7208) equals() and hashCode()
 implementation need to change in StandardSocketFactory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uma Maheswara Rao G updated HADOOP-7208:
----------------------------------------

    Attachment: HADOOP-7208-3.patch

> equals() and hashCode() implementation need to change in StandardSocketFactory
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7208
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7208
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7208-2.patch, HADOOP-7208-3.patch, HADOOP-7208.patch
>
>
>   In Hadoop IPC Client, we are using ClientCache which will maintain the HashMap to keep the Client references.
> private Map<SocketFactory, Client> clients =
>       new HashMap<SocketFactory, Client>();
>  Now let us say, we want use two standard factories with Hadoop. MyStandardSocketFactory (which extends StandardSocketFactory), and StandardSocketFactory. In this case, because of equals and hashcode implementation, MyStandardSocketFactory client can be overridden by StandardSocketFactoryClient

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15612-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 15:13:34 2011
Return-Path: <common-issues-return-15612-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B719B4B9B
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 15:13:34 +0000 (UTC)
Received: (qmail 97293 invoked by uid 500); 31 May 2011 15:13:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97250 invoked by uid 500); 31 May 2011 15:13:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97233 invoked by uid 99); 31 May 2011 15:13:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 15:13:34 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 15:13:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id A4290EB931
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 15:12:47 +0000 (UTC)
Date: Tue, 31 May 2011 15:12:47 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1677743433.56226.1306854767669.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <18917644.179531294294848031.JavaMail.jira@thor>
Subject: [jira] [Updated] (HADOOP-7090) Possible resource leaks in hadoop
 core code
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uma Maheswara Rao G updated HADOOP-7090:
----------------------------------------

    Attachment: HADOOP-7090.0.patch

> Possible resource leaks in hadoop core code
> -------------------------------------------
>
>                 Key: HADOOP-7090
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7090
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Gokul
>         Attachments: HADOOP-7090.0.patch
>
>
> It is always a good practice to close the IO streams in a finally block.. 
> For example, look at the following piece of code in the Writer class of BloomMapFile 
> {code:title=BloomMapFile .java|borderStyle=solid}
>     public synchronized void close() throws IOException {
>       super.close();
>       DataOutputStream out = fs.create(new Path(dir, BLOOM_FILE_NAME), true);
>       bloomFilter.write(out);
>       out.flush();
>       out.close();
>     }
> {code} 
> If an exception occurs during fs.create or on any other line,  out.close() will not be executed..
> The following can reduce the scope of resorce leaks..
> {code:title=BloomMapFile .java|borderStyle=solid}
>     public synchronized void close() throws IOException {
>       super.close();
>       DataOutputStream out = null;
>       try{
>           out = fs.create(new Path(dir, BLOOM_FILE_NAME), true);
>           bloomFilter.write(out);
>           out.flush();
>       }finally{
> 	 IOUtils.closeStream(out);
>     }
> {code} 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15613-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 15:21:32 2011
Return-Path: <common-issues-return-15613-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 27A074606
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 15:21:32 +0000 (UTC)
Received: (qmail 12617 invoked by uid 500); 31 May 2011 15:21:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12460 invoked by uid 500); 31 May 2011 15:21:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12452 invoked by uid 99); 31 May 2011 15:21:31 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 15:21:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 15:21:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6F4B2EBD81
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 15:20:47 +0000 (UTC)
Date: Tue, 31 May 2011 15:20:47 +0000 (UTC)
From: "ramkrishna.s.vasudevan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1404035813.56249.1306855247452.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <282223217.18947.1304436003652.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7256) Resource leak during failure
 scenario of closing of resources.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ramkrishna.s.vasudevan updated HADOOP-7256:
-------------------------------------------

    Status: Patch Available  (was: Open)

> Resource leak during failure scenario of closing of resources. 
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7256
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7256
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0, 0.20.2
>            Reporter: ramkrishna.s.vasudevan
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7256-patch-1.patch, HADOOP-7256-patch-2.patch
>
>   Original Estimate: 8h
>  Remaining Estimate: 8h
>
> Problem Statement:
> ===============
> There are chances of resource leak and stream not getting closed 
> Take the case when after copying data we try to close the Input and output stream followed by closing of the socket.
> Suppose an exception occurs while closing the input stream(due to runtime exception) then the subsequent operations of closing the output stream and socket may not happen and there is a chance of resource leak.
> Scenario 
> =======
> During long run of map reduce jobs, the copyFromLocalFile() api is getting called.
> Here we found some exceptions happening. As a result of this we found the lsof value raising leading to resource leak.
> Solution:
> =======
> While doing a close operation of any resource catch the RuntimeException also rather than catching the IOException alone.
> Additionally there are places where we try to close a resource in the catch block.
> If this close fails, we just throw and come out of the current flow.
> In order to avoid this, we can carry out the close operation in the finally block.
> Probable reasons for getting RunTimeExceptions:
> =====================================
> We may get runtime exception from customised hadoop streams like FSDataOutputStream.close() . So better to handle RunTimeExceptions also.
>  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15614-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 15:23:32 2011
Return-Path: <common-issues-return-15614-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5EF50478A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 15:23:32 +0000 (UTC)
Received: (qmail 18620 invoked by uid 500); 31 May 2011 15:23:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18589 invoked by uid 500); 31 May 2011 15:23:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18581 invoked by uid 99); 31 May 2011 15:23:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 15:23:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 15:23:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B8E4BEBF38
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 15:22:47 +0000 (UTC)
Date: Tue, 31 May 2011 15:22:47 +0000 (UTC)
From: "ramkrishna.s.vasudevan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <83399774.56262.1306855367754.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <18917644.179531294294848031.JavaMail.jira@thor>
Subject: [jira] [Updated] (HADOOP-7090) Possible resource leaks in hadoop
 core code
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ramkrishna.s.vasudevan updated HADOOP-7090:
-------------------------------------------

    Affects Version/s: 0.23.0
               Status: Patch Available  (was: Open)

> Possible resource leaks in hadoop core code
> -------------------------------------------
>
>                 Key: HADOOP-7090
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7090
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0, 0.23.0
>            Reporter: Gokul
>         Attachments: HADOOP-7090.0.patch
>
>
> It is always a good practice to close the IO streams in a finally block.. 
> For example, look at the following piece of code in the Writer class of BloomMapFile 
> {code:title=BloomMapFile .java|borderStyle=solid}
>     public synchronized void close() throws IOException {
>       super.close();
>       DataOutputStream out = fs.create(new Path(dir, BLOOM_FILE_NAME), true);
>       bloomFilter.write(out);
>       out.flush();
>       out.close();
>     }
> {code} 
> If an exception occurs during fs.create or on any other line,  out.close() will not be executed..
> The following can reduce the scope of resorce leaks..
> {code:title=BloomMapFile .java|borderStyle=solid}
>     public synchronized void close() throws IOException {
>       super.close();
>       DataOutputStream out = null;
>       try{
>           out = fs.create(new Path(dir, BLOOM_FILE_NAME), true);
>           bloomFilter.write(out);
>           out.flush();
>       }finally{
> 	 IOUtils.closeStream(out);
>     }
> {code} 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15615-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 15:23:33 2011
Return-Path: <common-issues-return-15615-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 615F647BA
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 15:23:33 +0000 (UTC)
Received: (qmail 19280 invoked by uid 500); 31 May 2011 15:23:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19243 invoked by uid 500); 31 May 2011 15:23:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19149 invoked by uid 99); 31 May 2011 15:23:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 15:23:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 15:23:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 633FFEBF48
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 15:22:48 +0000 (UTC)
Date: Tue, 31 May 2011 15:22:48 +0000 (UTC)
From: "ramkrishna.s.vasudevan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <145176474.56275.1306855368403.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1195662636.6130.1300891805766.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7208) equals() and hashCode()
 implementation need to change in StandardSocketFactory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ramkrishna.s.vasudevan updated HADOOP-7208:
-------------------------------------------

    Status: Patch Available  (was: Open)

> equals() and hashCode() implementation need to change in StandardSocketFactory
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7208
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7208
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7208-2.patch, HADOOP-7208-3.patch, HADOOP-7208.patch
>
>
>   In Hadoop IPC Client, we are using ClientCache which will maintain the HashMap to keep the Client references.
> private Map<SocketFactory, Client> clients =
>       new HashMap<SocketFactory, Client>();
>  Now let us say, we want use two standard factories with Hadoop. MyStandardSocketFactory (which extends StandardSocketFactory), and StandardSocketFactory. In this case, because of equals and hashcode implementation, MyStandardSocketFactory client can be overridden by StandardSocketFactoryClient

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15616-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 15:23:56 2011
Return-Path: <common-issues-return-15616-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BA55F47F0
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 15:23:56 +0000 (UTC)
Received: (qmail 20467 invoked by uid 500); 31 May 2011 15:23:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20445 invoked by uid 500); 31 May 2011 15:23:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20437 invoked by uid 99); 31 May 2011 15:23:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 15:23:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 15:23:52 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8AC59EBF4F
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 15:22:48 +0000 (UTC)
Date: Tue, 31 May 2011 15:22:48 +0000 (UTC)
From: "ramkrishna.s.vasudevan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2126480424.56280.1306855368565.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1195662636.6130.1300891805766.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7208) equals() and hashCode()
 implementation need to change in StandardSocketFactory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ramkrishna.s.vasudevan updated HADOOP-7208:
-------------------------------------------

    Hadoop Flags:   (was: [Reviewed])

> equals() and hashCode() implementation need to change in StandardSocketFactory
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7208
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7208
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7208-2.patch, HADOOP-7208-3.patch, HADOOP-7208.patch
>
>
>   In Hadoop IPC Client, we are using ClientCache which will maintain the HashMap to keep the Client references.
> private Map<SocketFactory, Client> clients =
>       new HashMap<SocketFactory, Client>();
>  Now let us say, we want use two standard factories with Hadoop. MyStandardSocketFactory (which extends StandardSocketFactory), and StandardSocketFactory. In this case, because of equals and hashcode implementation, MyStandardSocketFactory client can be overridden by StandardSocketFactoryClient

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15617-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 15:35:34 2011
Return-Path: <common-issues-return-15617-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7E1C144CB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 15:35:34 +0000 (UTC)
Received: (qmail 53285 invoked by uid 500); 31 May 2011 15:35:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53253 invoked by uid 500); 31 May 2011 15:35:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53244 invoked by uid 99); 31 May 2011 15:35:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 15:35:34 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 15:35:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id D47ADEB6B5
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 15:34:49 +0000 (UTC)
Date: Tue, 31 May 2011 15:34:49 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <642891181.56356.1306856089867.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-6387) FsShell -getmerge source file
 pattern is broken
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041629#comment-13041629 ] 

Daryn Sharp commented on HADOOP-6387:
-------------------------------------

Using checkDest adds new logic that I don't think is warranted:  The dest is supposed to be a file per the usage of copyMerge.  With this change, if the given dest is a directory then the output is the dest + basename of the first source directory. I feel that's unexpected and surprisingly behavior, so I'd like to see that reverted.

There's an easier way to get at the full arg list higher in the call stack: processArguments().  Hooking in there and eliminate processPath(s) since those are intended to process each path individually.  In processArguments, make a call to super (don't copy-n-paste like in the patch), and then call FileUtil's copymerge.  Implement processPathArgument with a null body to short-out the calls to the now removed processPath(s).

(Just an aside: I've been tempted to eliminate the call to FileUtils and just use processPath to append to the dst file)

> FsShell -getmerge source file pattern is broken
> -----------------------------------------------
>
>                 Key: HADOOP-6387
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6387
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Eli Collins
>            Assignee: XieXianshan
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-6387.patch
>
>
> The FsShell -getmerge command doesn't work if the "source file pattern" matches files. See below. If the current behavior is intended then we should update the help documentation and java docs to match, but it would be nice if the user could specify a set of files in a directory rather than just directories.
> {code}
> $ hadoop fs -help getmerge
> -getmerge <src> <localdst>:  Get all the files in the directories that 
> 		match the source file pattern and merge and sort them to only
> 		one file on local fs. <src> is kept.
> $ hadoop fs -ls
> Found 3 items
> -rw-r--r--   1 eli supergroup          2 2009-11-23 17:39 /user/eli/1.txt
> -rw-r--r--   1 eli supergroup          2 2009-11-23 17:39 /user/eli/2.txt
> -rw-r--r--   1 eli supergroup          2 2009-11-23 17:39 /user/eli/3.txt
> $ hadoop fs -getmerge /user/eli/*.txt sorted.txt
> $ cat sorted.txt
> cat: sorted.txt: No such file or directory
> $ hadoop fs -getmerge /user/eli/* sorted.txt
> $ cat sorted.txt
> cat: sorted.txt: No such file or directory
> $ hadoop fs -getmerge /user/* sorted.txt
> $ cat sorted.txt 
> 1
> 2
> 3
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15618-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 15:35:35 2011
Return-Path: <common-issues-return-15618-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1851544D8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 15:35:35 +0000 (UTC)
Received: (qmail 53539 invoked by uid 500); 31 May 2011 15:35:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53505 invoked by uid 500); 31 May 2011 15:35:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53487 invoked by uid 99); 31 May 2011 15:35:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 15:35:34 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 15:35:30 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 10F7AEB6BA
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 15:34:50 +0000 (UTC)
Date: Tue, 31 May 2011 15:34:50 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <983583704.56361.1306856090065.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <282223217.18947.1304436003652.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7256) Resource leak during failure
 scenario of closing of resources.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041630#comment-13041630 ] 

Hadoop QA commented on HADOOP-7256:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480943/HADOOP-7256-patch-2.patch
  against trunk revision 1128789.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/543//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/543//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/543//console

This message is automatically generated.

> Resource leak during failure scenario of closing of resources. 
> ---------------------------------------------------------------
>
>                 Key: HADOOP-7256
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7256
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2, 0.21.0
>            Reporter: ramkrishna.s.vasudevan
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7256-patch-1.patch, HADOOP-7256-patch-2.patch
>
>   Original Estimate: 8h
>  Remaining Estimate: 8h
>
> Problem Statement:
> ===============
> There are chances of resource leak and stream not getting closed 
> Take the case when after copying data we try to close the Input and output stream followed by closing of the socket.
> Suppose an exception occurs while closing the input stream(due to runtime exception) then the subsequent operations of closing the output stream and socket may not happen and there is a chance of resource leak.
> Scenario 
> =======
> During long run of map reduce jobs, the copyFromLocalFile() api is getting called.
> Here we found some exceptions happening. As a result of this we found the lsof value raising leading to resource leak.
> Solution:
> =======
> While doing a close operation of any resource catch the RuntimeException also rather than catching the IOException alone.
> Additionally there are places where we try to close a resource in the catch block.
> If this close fails, we just throw and come out of the current flow.
> In order to avoid this, we can carry out the close operation in the finally block.
> Probable reasons for getting RunTimeExceptions:
> =====================================
> We may get runtime exception from customised hadoop streams like FSDataOutputStream.close() . So better to handle RunTimeExceptions also.
>  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15619-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 15:35:35 2011
Return-Path: <common-issues-return-15619-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B703344E9
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 15:35:35 +0000 (UTC)
Received: (qmail 53807 invoked by uid 500); 31 May 2011 15:35:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53768 invoked by uid 500); 31 May 2011 15:35:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53760 invoked by uid 99); 31 May 2011 15:35:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 15:35:35 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 15:35:31 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id F09FFEB6C1
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 15:34:50 +0000 (UTC)
Date: Tue, 31 May 2011 15:34:50 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2076075170.56366.1306856090979.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <18917644.179531294294848031.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7090) Possible resource leaks in hadoop
 core code
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041631#comment-13041631 ] 

Hadoop QA commented on HADOOP-7090:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480947/HADOOP-7090.0.patch
  against trunk revision 1128789.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/544//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/544//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/544//console

This message is automatically generated.

> Possible resource leaks in hadoop core code
> -------------------------------------------
>
>                 Key: HADOOP-7090
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7090
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0, 0.23.0
>            Reporter: Gokul
>         Attachments: HADOOP-7090.0.patch
>
>
> It is always a good practice to close the IO streams in a finally block.. 
> For example, look at the following piece of code in the Writer class of BloomMapFile 
> {code:title=BloomMapFile .java|borderStyle=solid}
>     public synchronized void close() throws IOException {
>       super.close();
>       DataOutputStream out = fs.create(new Path(dir, BLOOM_FILE_NAME), true);
>       bloomFilter.write(out);
>       out.flush();
>       out.close();
>     }
> {code} 
> If an exception occurs during fs.create or on any other line,  out.close() will not be executed..
> The following can reduce the scope of resorce leaks..
> {code:title=BloomMapFile .java|borderStyle=solid}
>     public synchronized void close() throws IOException {
>       super.close();
>       DataOutputStream out = null;
>       try{
>           out = fs.create(new Path(dir, BLOOM_FILE_NAME), true);
>           bloomFilter.write(out);
>           out.flush();
>       }finally{
> 	 IOUtils.closeStream(out);
>     }
> {code} 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15620-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 15:45:32 2011
Return-Path: <common-issues-return-15620-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 3869F4B0A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 15:45:32 +0000 (UTC)
Received: (qmail 73119 invoked by uid 500); 31 May 2011 15:45:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73066 invoked by uid 500); 31 May 2011 15:45:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73058 invoked by uid 99); 31 May 2011 15:45:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 15:45:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 15:45:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8695FEBB0A
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 15:44:47 +0000 (UTC)
Date: Tue, 31 May 2011 15:44:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1120596033.56381.1306856687548.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1195662636.6130.1300891805766.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7208) equals() and hashCode()
 implementation need to change in StandardSocketFactory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041635#comment-13041635 ] 

Hadoop QA commented on HADOOP-7208:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480944/HADOOP-7208-3.patch
  against trunk revision 1128789.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/545//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/545//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/545//console

This message is automatically generated.

> equals() and hashCode() implementation need to change in StandardSocketFactory
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7208
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7208
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7208-2.patch, HADOOP-7208-3.patch, HADOOP-7208.patch
>
>
>   In Hadoop IPC Client, we are using ClientCache which will maintain the HashMap to keep the Client references.
> private Map<SocketFactory, Client> clients =
>       new HashMap<SocketFactory, Client>();
>  Now let us say, we want use two standard factories with Hadoop. MyStandardSocketFactory (which extends StandardSocketFactory), and StandardSocketFactory. In this case, because of equals and hashcode implementation, MyStandardSocketFactory client can be overridden by StandardSocketFactoryClient

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15621-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 16:25:34 2011
Return-Path: <common-issues-return-15621-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 6000742D6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 16:25:34 +0000 (UTC)
Received: (qmail 50770 invoked by uid 500); 31 May 2011 16:25:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50734 invoked by uid 500); 31 May 2011 16:25:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50726 invoked by uid 99); 31 May 2011 16:25:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 16:25:34 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 16:25:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 74EE7EBC76
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 16:24:47 +0000 (UTC)
Date: Tue, 31 May 2011 16:24:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1339894883.56464.1306859087475.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <53044194.55753.1306837788451.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7340) incomplete help message is
 displayed for getmerge [addnl] option
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041651#comment-13041651 ] 

Todd Lipcon commented on HADOOP-7340:
-------------------------------------

apparently [addnl] should be either "true" or "false"... but like the TODO says in CopyCommands.java, we should deprecate that and make it a flag like -addnl

> incomplete help message is displayed for getmerge [addnl] option
> ----------------------------------------------------------------
>
>                 Key: HADOOP-7340
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7340
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: XieXianshan
>            Assignee: XieXianshan
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7340
>
>
> The help message for the command "hdfs dfs -help getmerge" is displayed like this:
> "-getmerge <src> <localdst> [addnl]:	Get all the files in the directories that
> 		match the source file pattern and merge and sort them to only
> 		one file on local fs. <src> is kept."
> and the information about [addnl] option is missed,despite the fact that [addnl] option is implemented.
> Therefore,the expected message should be displayed like this:
> "-getmerge <src> <localdst> [addnl]:	Get all the files in the directories that
> 		match the source file pattern and merge and sort them to only
> 		one file on local fs. <src> is kept.
> 		  addnl   Optionally addnl can be set to enable adding a newline 
> 		          character at the end of each file."

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15622-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 16:49:34 2011
Return-Path: <common-issues-return-15622-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 324144446
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 16:49:34 +0000 (UTC)
Received: (qmail 96431 invoked by uid 500); 31 May 2011 16:49:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96391 invoked by uid 500); 31 May 2011 16:49:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96382 invoked by uid 99); 31 May 2011 16:49:33 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 16:49:33 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 16:49:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 61BBBEB72E
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 16:48:47 +0000 (UTC)
Date: Tue, 31 May 2011 16:48:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1470673003.56543.1306860527397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7341) Fix option parsing in CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Fix option parsing in CommandFormat
-----------------------------------

                 Key: HADOOP-7341
                 URL: https://issues.apache.org/jira/browse/HADOOP-7341
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
    Affects Versions: 0.23.0
            Reporter: Daryn Sharp
            Assignee: Daryn Sharp


CommandFormat currently allows options in any location within the args.  This is not the intended behavior for FsShell commands.  Prior to the redesign, the commands used to expect option processing to stop at the first non-option.

CommandFormat was an existing class prior the redesign, but it was only used by "count" to find the -q flag.  All commands were converted to using this class, thus inherited the unintended behavior.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15623-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 16:58:32 2011
Return-Path: <common-issues-return-15623-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 461E345B4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 16:58:32 +0000 (UTC)
Received: (qmail 6113 invoked by uid 500); 31 May 2011 16:58:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6089 invoked by uid 500); 31 May 2011 16:58:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6079 invoked by uid 99); 31 May 2011 16:58:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 16:58:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 16:58:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 75051EBA2C
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 16:57:47 +0000 (UTC)
Date: Tue, 31 May 2011 16:57:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1822961924.56560.1306861067475.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7284:
--------------------------------

    Attachment: trash11.patch

as I was reviewing trash10.patch I did some trivial cleanup: fixed up a couple of typos, renamed "homedir" to "homedirPrefix", and relocated the new variable definition up to the top of the classes along with the rest of 
them.

Sanjay, if these edits look OK to you, then +1.

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash10.patch, trash11.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch, trash8.patch, trash9.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15624-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 17:09:34 2011
Return-Path: <common-issues-return-15624-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 710EB4FB4
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 17:09:34 +0000 (UTC)
Received: (qmail 26261 invoked by uid 500); 31 May 2011 17:09:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26233 invoked by uid 500); 31 May 2011 17:09:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26225 invoked by uid 99); 31 May 2011 17:09:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 17:09:34 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 17:09:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8C899EBEFD
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 17:08:47 +0000 (UTC)
Date: Tue, 31 May 2011 17:08:47 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <257527138.56610.1306861727572.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1470673003.56543.1306860527397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7341) Fix option parsing in
 CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041678#comment-13041678 ] 

Aaron T. Myers commented on HADOOP-7341:
----------------------------------------

Now seems like a great time to scrap {{CommandFormat}} altogether, in favor of the capabilities provided by Commons CLI, a dependency which Hadoop already includes for use in the {{GenericOptionsParser}}.

Thoughts?

> Fix option parsing in CommandFormat
> -----------------------------------
>
>                 Key: HADOOP-7341
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7341
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>
> CommandFormat currently allows options in any location within the args.  This is not the intended behavior for FsShell commands.  Prior to the redesign, the commands used to expect option processing to stop at the first non-option.
> CommandFormat was an existing class prior the redesign, but it was only used by "count" to find the -q flag.  All commands were converted to using this class, thus inherited the unintended behavior.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15625-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 17:15:34 2011
Return-Path: <common-issues-return-15625-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B6AAC43DD
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 17:15:34 +0000 (UTC)
Received: (qmail 36528 invoked by uid 500); 31 May 2011 17:15:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36484 invoked by uid 500); 31 May 2011 17:15:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36337 invoked by uid 99); 31 May 2011 17:15:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 17:15:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 17:15:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 78C4DEB18C
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 17:14:47 +0000 (UTC)
Date: Tue, 31 May 2011 17:14:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <573038587.56628.1306862087491.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1946167265.7915.1305229907409.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7284) Trash and shell's rm does not work
 for viewfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041681#comment-13041681 ] 

Hadoop QA commented on HADOOP-7284:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480962/trash11.patch
  against trunk revision 1128789.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 18 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/546//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/546//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/546//console

This message is automatically generated.

> Trash and shell's rm does not work for viewfs
> ---------------------------------------------
>
>                 Key: HADOOP-7284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7284
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.23.0
>
>         Attachments: trash1.patch, trash10.patch, trash11.patch, trash2.patch, trash3.patch, trash4.patch, trash5.patch, trash6.patch, trash7.patch, trash8.patch, trash9.patch
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15626-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 18:08:31 2011
Return-Path: <common-issues-return-15626-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 377194A35
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 18:08:31 +0000 (UTC)
Received: (qmail 29472 invoked by uid 500); 31 May 2011 18:08:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29443 invoked by uid 500); 31 May 2011 18:08:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29435 invoked by uid 99); 31 May 2011 18:08:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 18:08:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 18:08:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id DF3F1ECAC7
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 18:07:47 +0000 (UTC)
Date: Tue, 31 May 2011 18:07:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1896224663.56882.1306865267910.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1470673003.56543.1306860527397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7341) Fix option parsing in
 CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041719#comment-13041719 ] 

Todd Lipcon commented on HADOOP-7341:
-------------------------------------

hmm -- do you think it's correct behavior that we only support flags at the front of the args? Linux "ls" for example is fine with "ls / -l"

> Fix option parsing in CommandFormat
> -----------------------------------
>
>                 Key: HADOOP-7341
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7341
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>
> CommandFormat currently allows options in any location within the args.  This is not the intended behavior for FsShell commands.  Prior to the redesign, the commands used to expect option processing to stop at the first non-option.
> CommandFormat was an existing class prior the redesign, but it was only used by "count" to find the -q flag.  All commands were converted to using this class, thus inherited the unintended behavior.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15627-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 18:43:30 2011
Return-Path: <common-issues-return-15627-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7855A48A7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 18:43:30 +0000 (UTC)
Received: (qmail 16330 invoked by uid 500); 31 May 2011 18:43:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16292 invoked by uid 500); 31 May 2011 18:43:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16284 invoked by uid 99); 31 May 2011 18:43:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 18:43:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 18:43:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 61F3EEB708
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 18:42:47 +0000 (UTC)
Date: Tue, 31 May 2011 18:42:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1870506554.56972.1306867367397.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1195662636.6130.1300891805766.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7208) equals() and hashCode()
 implementation need to change in StandardSocketFactory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7208:
--------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

+1. Committed to trunk. Thanks, Uma

> equals() and hashCode() implementation need to change in StandardSocketFactory
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7208
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7208
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7208-2.patch, HADOOP-7208-3.patch, HADOOP-7208.patch
>
>
>   In Hadoop IPC Client, we are using ClientCache which will maintain the HashMap to keep the Client references.
> private Map<SocketFactory, Client> clients =
>       new HashMap<SocketFactory, Client>();
>  Now let us say, we want use two standard factories with Hadoop. MyStandardSocketFactory (which extends StandardSocketFactory), and StandardSocketFactory. In this case, because of equals and hashcode implementation, MyStandardSocketFactory client can be overridden by StandardSocketFactoryClient

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15628-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 19:00:34 2011
Return-Path: <common-issues-return-15628-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 8FC1347D6
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 19:00:34 +0000 (UTC)
Received: (qmail 60635 invoked by uid 500); 31 May 2011 19:00:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60608 invoked by uid 500); 31 May 2011 19:00:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60600 invoked by uid 99); 31 May 2011 19:00:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 19:00:34 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 19:00:32 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 381FAEBF82
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 18:59:51 +0000 (UTC)
Date: Tue, 31 May 2011 18:59:51 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <97534241.57040.1306868391219.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1470673003.56543.1306860527397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7341) Fix option parsing in
 CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041752#comment-13041752 ] 

Daryn Sharp commented on HADOOP-7341:
-------------------------------------

Aaron:
I completely agree the parser needs to be changed, but I don't currently have the time available.  My goal here is a very small/quick change to revert how every FsShell command (except count) to how it used to work.  If you're ok with that, I think another jira should switch the parser.

Todd:
I suppose it comes down to whether hadoop should favor posix or gnu extensions?

{noformat}
$ mkdir a b

$ ls a -l b
a:
total 0
b:
total 0

$ POSIXLY_CORRECT=1 ls a -l b
ls: -l: No such file or directory
a:
b:
{noformat}

I was just planning to restore the prior behavior.

> Fix option parsing in CommandFormat
> -----------------------------------
>
>                 Key: HADOOP-7341
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7341
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>
> CommandFormat currently allows options in any location within the args.  This is not the intended behavior for FsShell commands.  Prior to the redesign, the commands used to expect option processing to stop at the first non-option.
> CommandFormat was an existing class prior the redesign, but it was only used by "count" to find the -q flag.  All commands were converted to using this class, thus inherited the unintended behavior.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15629-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 19:06:30 2011
Return-Path: <common-issues-return-15629-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BE1984D25
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 19:06:30 +0000 (UTC)
Received: (qmail 71756 invoked by uid 500); 31 May 2011 19:06:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71726 invoked by uid 500); 31 May 2011 19:06:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71718 invoked by uid 99); 31 May 2011 19:06:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 19:06:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 19:06:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 8D3B9EC2A0
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 19:05:47 +0000 (UTC)
Date: Tue, 31 May 2011 19:05:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <371115798.57068.1306868747575.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1195662636.6130.1300891805766.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7208) equals() and hashCode()
 implementation need to change in StandardSocketFactory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041758#comment-13041758 ] 

Hudson commented on HADOOP-7208:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #629 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/629/])
    HADOOP-7208. Fix implementation of equals() and hashCode() in StandardSocketFactory. Contributed by Uma Maheswara Rao G.

todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1129840
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/ipc/TestSocketFactory.java
* /hadoop/common/trunk/src/java/org/apache/hadoop/net/StandardSocketFactory.java


> equals() and hashCode() implementation need to change in StandardSocketFactory
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-7208
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7208
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7208-2.patch, HADOOP-7208-3.patch, HADOOP-7208.patch
>
>
>   In Hadoop IPC Client, we are using ClientCache which will maintain the HashMap to keep the Client references.
> private Map<SocketFactory, Client> clients =
>       new HashMap<SocketFactory, Client>();
>  Now let us say, we want use two standard factories with Hadoop. MyStandardSocketFactory (which extends StandardSocketFactory), and StandardSocketFactory. In this case, because of equals and hashcode implementation, MyStandardSocketFactory client can be overridden by StandardSocketFactoryClient

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15630-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 19:20:29 2011
Return-Path: <common-issues-return-15630-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D0FA147D7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 19:20:29 +0000 (UTC)
Received: (qmail 8381 invoked by uid 500); 31 May 2011 19:20:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8338 invoked by uid 500); 31 May 2011 19:20:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8323 invoked by uid 99); 31 May 2011 19:20:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 19:20:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 19:20:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 61FC1EC886
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 19:19:48 +0000 (UTC)
Date: Tue, 31 May 2011 19:19:48 +0000 (UTC)
From: "Bharath Mundlapudi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <397648942.57126.1306869588398.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7342) Add an utility API in FileUtil for
 JDK File.list
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Add an utility API in FileUtil for JDK File.list
------------------------------------------------

                 Key: HADOOP-7342
                 URL: https://issues.apache.org/jira/browse/HADOOP-7342
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.23.0
            Reporter: Bharath Mundlapudi
            Assignee: Bharath Mundlapudi
            Priority: Minor
             Fix For: 0.23.0


Java File.list API can return null when disk is bad or directory is not a directory. This utility API in FileUtil will throw an exception when this happens rather than returning null. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15631-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 19:22:30 2011
Return-Path: <common-issues-return-15631-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ABAFB49DC
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 19:22:30 +0000 (UTC)
Received: (qmail 18108 invoked by uid 500); 31 May 2011 19:22:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18076 invoked by uid 500); 31 May 2011 19:22:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18012 invoked by uid 99); 31 May 2011 19:22:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 19:22:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 19:22:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 90DF4EC989
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 19:21:47 +0000 (UTC)
Date: Tue, 31 May 2011 19:21:47 +0000 (UTC)
From: "Bharath Mundlapudi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1203219351.57135.1306869707590.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <397648942.57126.1306869588398.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7342) Add an utility API in FileUtil for
 JDK File.list
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bharath Mundlapudi updated HADOOP-7342:
---------------------------------------

    Attachment: HADOOP-7342-1.patch

Attaching a patch for this.

> Add an utility API in FileUtil for JDK File.list
> ------------------------------------------------
>
>                 Key: HADOOP-7342
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7342
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7342-1.patch
>
>
> Java File.list API can return null when disk is bad or directory is not a directory. This utility API in FileUtil will throw an exception when this happens rather than returning null. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15632-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 20:01:28 2011
Return-Path: <common-issues-return-15632-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id BA4C54EC7
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 20:01:28 +0000 (UTC)
Received: (qmail 53776 invoked by uid 500); 31 May 2011 20:01:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53740 invoked by uid 500); 31 May 2011 20:01:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53732 invoked by uid 99); 31 May 2011 20:01:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 20:01:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 20:01:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 799B0EC80A
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 20:00:47 +0000 (UTC)
Date: Tue, 31 May 2011 20:00:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <142891067.57235.1306872047494.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1470673003.56543.1306860527397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7341) Fix option parsing in
 CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041791#comment-13041791 ] 

Todd Lipcon commented on HADOOP-7341:
-------------------------------------

ah, ok. I'm fine with just restoring the old behavior.

> Fix option parsing in CommandFormat
> -----------------------------------
>
>                 Key: HADOOP-7341
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7341
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>
> CommandFormat currently allows options in any location within the args.  This is not the intended behavior for FsShell commands.  Prior to the redesign, the commands used to expect option processing to stop at the first non-option.
> CommandFormat was an existing class prior the redesign, but it was only used by "count" to find the -q flag.  All commands were converted to using this class, thus inherited the unintended behavior.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15633-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 20:45:28 2011
Return-Path: <common-issues-return-15633-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D302B4075
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 20:45:28 +0000 (UTC)
Received: (qmail 59015 invoked by uid 500); 31 May 2011 20:45:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58974 invoked by uid 500); 31 May 2011 20:45:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58966 invoked by uid 99); 31 May 2011 20:45:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 20:45:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 20:45:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 81614EC6E8
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 20:44:47 +0000 (UTC)
Date: Tue, 31 May 2011 20:44:47 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <2058617390.57316.1306874687526.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1435375533.50176.1306536707416.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7336) TestFileContextResolveAfs will
 fail with default test.build.data property.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041813#comment-13041813 ] 

Hudson commented on HADOOP-7336:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #630 (See [https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/630/])
    HADOOP-7336. TestFileContextResolveAfs will fail with default test.build.data property.

jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1129905
Files : 
* /hadoop/common/trunk/CHANGES.txt
* /hadoop/common/trunk/src/test/core/org/apache/hadoop/fs/TestFileContextResolveAfs.java


> TestFileContextResolveAfs will fail with default test.build.data property.
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7336
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7336
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7336.2.patch
>
>
> In TestFileContextResolveAfs if test.build.data property is not set and default is used, the test case will try to create that in the root directory and that will fail. /tmp should be used as default as in many other test cases. Normally, test.build.data will be set and this issue should not occur.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15634-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 20:47:29 2011
Return-Path: <common-issues-return-15634-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1AC634F80
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 20:47:29 +0000 (UTC)
Received: (qmail 61110 invoked by uid 500); 31 May 2011 20:47:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61071 invoked by uid 500); 31 May 2011 20:47:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61063 invoked by uid 99); 31 May 2011 20:47:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 20:47:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 20:47:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id BAD90EC83B
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 20:46:47 +0000 (UTC)
Date: Tue, 31 May 2011 20:46:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1133733464.57320.1306874807761.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1470673003.56543.1306860527397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7341) Fix option parsing in CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7341:
--------------------------------

    Attachment: HADOOP-7341.patch

Stop at first non-option, like before.
Also stop at opt/arg demarcation "--".
Remove unnecessary & unused first param of null.

> Fix option parsing in CommandFormat
> -----------------------------------
>
>                 Key: HADOOP-7341
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7341
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7341.patch
>
>
> CommandFormat currently allows options in any location within the args.  This is not the intended behavior for FsShell commands.  Prior to the redesign, the commands used to expect option processing to stop at the first non-option.
> CommandFormat was an existing class prior the redesign, but it was only used by "count" to find the -q flag.  All commands were converted to using this class, thus inherited the unintended behavior.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15635-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 20:47:31 2011
Return-Path: <common-issues-return-15635-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 631824FD8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 20:47:31 +0000 (UTC)
Received: (qmail 61534 invoked by uid 500); 31 May 2011 20:47:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61455 invoked by uid 500); 31 May 2011 20:47:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61435 invoked by uid 99); 31 May 2011 20:47:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 20:47:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 20:47:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id E1732EC83F
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 20:46:47 +0000 (UTC)
Date: Tue, 31 May 2011 20:46:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1510539831.57324.1306874807920.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1470673003.56543.1306860527397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7341) Fix option parsing in CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7341:
--------------------------------

    Status: Patch Available  (was: Open)

> Fix option parsing in CommandFormat
> -----------------------------------
>
>                 Key: HADOOP-7341
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7341
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7341.patch
>
>
> CommandFormat currently allows options in any location within the args.  This is not the intended behavior for FsShell commands.  Prior to the redesign, the commands used to expect option processing to stop at the first non-option.
> CommandFormat was an existing class prior the redesign, but it was only used by "count" to find the -q flag.  All commands were converted to using this class, thus inherited the unintended behavior.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15636-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 20:51:29 2011
Return-Path: <common-issues-return-15636-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E89A744E2
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 20:51:28 +0000 (UTC)
Received: (qmail 68093 invoked by uid 500); 31 May 2011 20:51:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68047 invoked by uid 500); 31 May 2011 20:51:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68039 invoked by uid 99); 31 May 2011 20:51:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 20:51:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 20:51:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 84179ECA34
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 20:50:47 +0000 (UTC)
Date: Tue, 31 May 2011 20:50:47 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1357458239.57346.1306875047537.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1435375533.50176.1306536707416.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7336) TestFileContextResolveAfs will fail
 with default test.build.data property.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-7336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-7336:
-----------------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

> TestFileContextResolveAfs will fail with default test.build.data property.
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-7336
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7336
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.23.0
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>            Priority: Minor
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7336.2.patch
>
>
> In TestFileContextResolveAfs if test.build.data property is not set and default is used, the test case will try to create that in the root directory and that will fail. /tmp should be used as default as in many other test cases. Normally, test.build.data will be set and this issue should not occur.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15637-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 21:05:30 2011
Return-Path: <common-issues-return-15637-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 69AA84E4A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 21:05:30 +0000 (UTC)
Received: (qmail 89372 invoked by uid 500); 31 May 2011 21:05:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89154 invoked by uid 500); 31 May 2011 21:05:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89138 invoked by uid 99); 31 May 2011 21:05:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 21:05:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 21:05:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id ADC49ECEB4
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 21:04:47 +0000 (UTC)
Date: Tue, 31 May 2011 21:04:47 +0000 (UTC)
From: "Thomas Graves (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1738050873.57393.1306875887708.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7343) backport HADOOP-7008 and HADOOP-7042
 to branch-0.20-security
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

backport HADOOP-7008 and HADOOP-7042 to branch-0.20-security
------------------------------------------------------------

                 Key: HADOOP-7343
                 URL: https://issues.apache.org/jira/browse/HADOOP-7343
             Project: Hadoop Common
          Issue Type: Improvement
          Components: test
    Affects Versions: 0.20.204.0
            Reporter: Thomas Graves
            Assignee: Thomas Graves
            Priority: Minor


backport HADOOP-7008 and HADOOP-7042 to branch-0.20-security so that we can enable test-patch.sh to have a configured number of acceptable findbugs and javadoc warnings

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15638-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 21:05:30 2011
Return-Path: <common-issues-return-15638-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id D02A84E77
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 21:05:30 +0000 (UTC)
Received: (qmail 90707 invoked by uid 500); 31 May 2011 21:05:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90679 invoked by uid 500); 31 May 2011 21:05:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90671 invoked by uid 99); 31 May 2011 21:05:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 21:05:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 21:05:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 76223ECEAD
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 21:04:47 +0000 (UTC)
Date: Tue, 31 May 2011 21:04:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <350642054.57386.1306875887480.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1470673003.56543.1306860527397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7341) Fix option parsing in
 CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041829#comment-13041829 ] 

Hadoop QA commented on HADOOP-7341:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12480991/HADOOP-7341.patch
  against trunk revision 1129905.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The applied patch generated 1072 javac compiler warnings (more than the trunk's current 1068 warnings).

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/547//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/547//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/547//console

This message is automatically generated.

> Fix option parsing in CommandFormat
> -----------------------------------
>
>                 Key: HADOOP-7341
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7341
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7341.patch
>
>
> CommandFormat currently allows options in any location within the args.  This is not the intended behavior for FsShell commands.  Prior to the redesign, the commands used to expect option processing to stop at the first non-option.
> CommandFormat was an existing class prior the redesign, but it was only used by "count" to find the -q flag.  All commands were converted to using this class, thus inherited the unintended behavior.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15639-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 21:25:31 2011
Return-Path: <common-issues-return-15639-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 1CB554B82
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 21:25:31 +0000 (UTC)
Received: (qmail 33788 invoked by uid 500); 31 May 2011 21:25:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33757 invoked by uid 500); 31 May 2011 21:25:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33736 invoked by uid 99); 31 May 2011 21:25:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 21:25:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 21:25:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 647CBEC615
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 21:24:47 +0000 (UTC)
Date: Tue, 31 May 2011 21:24:47 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <737703557.57434.1306877087408.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041841#comment-13041841 ] 

Owen O'Malley commented on HADOOP-7144:
---------------------------------------

I don't think the exception handling is appropriate. Why do you need to catch them at all?

You certainly don't want to silently swallow Errors or mix their handling with IOExceptions.

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Robert Joseph Evans
>              Labels: jmx
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7411-0.20.20X-V1.patch, HADOOP-7411-trunk-V1.patch, HADOOP-7411-trunk-alpha.patch, jmx.json
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15640-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 21:29:31 2011
Return-Path: <common-issues-return-15640-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DF3784CC3
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 21:29:30 +0000 (UTC)
Received: (qmail 45254 invoked by uid 500); 31 May 2011 21:29:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45205 invoked by uid 500); 31 May 2011 21:29:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45197 invoked by uid 99); 31 May 2011 21:29:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 21:29:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 21:29:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 82F40EC7EE
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 21:28:47 +0000 (UTC)
Date: Tue, 31 May 2011 21:28:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1403090349.57452.1306877327533.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7106) Re-organize hadoop subversion
 layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041843#comment-13041843 ] 

Todd Lipcon commented on HADOOP-7106:
-------------------------------------

Hi Nigel. I ran through my tests again with a clean mirror... one more thing came up: for branches like MR-279 and HDFS-1073, it seems like we want to create new merged branches as well - for MR279 it probably should be the "yahoo-merge" branches, and for HDFS-1073 it should be trunk.

Do you agree?

I don't think any of the other branches are "live".

> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15641-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 21:49:28 2011
Return-Path: <common-issues-return-15641-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 995594CB8
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 21:49:28 +0000 (UTC)
Received: (qmail 74566 invoked by uid 500); 31 May 2011 21:49:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74534 invoked by uid 500); 31 May 2011 21:49:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74526 invoked by uid 99); 31 May 2011 21:49:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 21:49:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 21:49:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 64986ECE4A
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 21:48:47 +0000 (UTC)
Date: Tue, 31 May 2011 21:48:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <864548259.57518.1306878527408.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (HADOOP-7344) globStatus doesn't grok groupings
 with a slash
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

globStatus doesn't grok groupings with a slash
----------------------------------------------

                 Key: HADOOP-7344
                 URL: https://issues.apache.org/jira/browse/HADOOP-7344
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
    Affects Versions: 0.23.0
            Reporter: Daryn Sharp


If a glob contains a grouping with a single item that contains a slash, ex. "{a/b}", then globStatus throws {{"Illegal file pattern: Unclosed group near index 2"}} -- regardless of whether the path exists.  However, if the glob set contains more than one item, ex. "{a/b,c}", then it throws a {{NullPointerException}} from {{FileSystem.java:1277}}.

{code}
1276: FileStatus[] files = globStatusInternal(new Path(filePattern), filter);
1277: for (FileStatus file : files) {
1278:   results.add(file);
1279: }
{code}

The method {{globStatusInternal}} can return null, so the iterator fails with the NPE.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15642-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 22:08:30 2011
Return-Path: <common-issues-return-15642-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 0212D4722
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 22:08:30 +0000 (UTC)
Received: (qmail 99533 invoked by uid 500); 31 May 2011 22:08:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99472 invoked by uid 500); 31 May 2011 22:08:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99464 invoked by uid 99); 31 May 2011 22:08:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 22:08:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 22:08:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 91B2AEC332
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 22:07:48 +0000 (UTC)
Date: Tue, 31 May 2011 22:07:48 +0000 (UTC)
From: "Jukka Zitting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1094225103.57572.1306879668593.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7106) Re-organize hadoop subversion
 layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041857#comment-13041857 ] 

Jukka Zitting commented on HADOOP-7106:
---------------------------------------

As Todd already tested, the move as planned should work just fine from the perspective of the git.apache.org mirror. The HADOOP-7106-git.sh script looks nice. Todd, I'll sync up with you directly on how to best make this happen for git.apache.org.

I suppose we should remove the hadoop-hdfs.git and hadoop-mapreduce.git mirrors once the svn tree has been reorganized.

Do you want to also move the resulting shared svn structure one level up from hadoop/common/{trunk,branches,tags} to hadoop/{trunk,branches,tags} (and presumably rename the hadoop-common.git mirror to hadoop.git)? If yes, we'll need to figure out how to best do this without breaking history in the git mirror since our current move-svn-project.sh script can only handle cases where the entire svn structure is moved in a single svn move (like "svn move ^/incubator/project ^/project"). One possible solution (given help from svn admins) could be:

{code:sh}
$ svn move ^/hadoop ^/hadoop-tmp
$ svn move ^/hadoop-tmp/common ^/hadoop
$ svn move ^/hadoop-tmp/logos ^/hadoop/logos
$ svn move ^/hadoop-tmp/nightly ^/hadoop/nightly
$ svn delete ^/hadoop-tmp
{code}

Though I'm not sure how that would look like to existing svn checkouts. Another alternative is to simply move the trunk, branches and tags separately, and try to patch the git mirror to properly follow such svn history. Todd, you think you could come up with a script to do that. See the existing move-svn-project.sh script [1] for an example of the kind of history patching we now do for more regular svn moves.

[1] https://svn.apache.org/repos/infra/infrastructure/trunk/projects/git/bin/move-svn-project.sh

> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15643-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 22:14:29 2011
Return-Path: <common-issues-return-15643-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id DBFBC4976
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 22:14:29 +0000 (UTC)
Received: (qmail 9374 invoked by uid 500); 31 May 2011 22:14:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9337 invoked by uid 500); 31 May 2011 22:14:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9197 invoked by uid 99); 31 May 2011 22:14:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 22:14:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 22:14:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 4E99EEC599
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 22:13:48 +0000 (UTC)
Date: Tue, 31 May 2011 22:13:48 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <260270348.57602.1306880028318.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7106) Re-organize hadoop subversion
 layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041865#comment-13041865 ] 

Owen O'Malley commented on HADOOP-7106:
---------------------------------------

Jukka,
   I'd proposed not changing the svn url for common precisely to avoid those breaks. *smile*

> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15644-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 22:18:30 2011
Return-Path: <common-issues-return-15644-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B632947AB
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 22:18:30 +0000 (UTC)
Received: (qmail 14523 invoked by uid 500); 31 May 2011 22:18:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14496 invoked by uid 500); 31 May 2011 22:18:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14488 invoked by uid 99); 31 May 2011 22:18:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 22:18:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 22:18:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 5D486EC6A2
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 22:17:47 +0000 (UTC)
Date: Tue, 31 May 2011 22:17:47 +0000 (UTC)
From: "Robert Joseph Evans (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1403190025.57626.1306880267378.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1728703995.16605.1297739397760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7144) Expose JMX with something like
 JMXProxyServlet
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041866#comment-13041866 ] 

Robert Joseph Evans commented on HADOOP-7144:
---------------------------------------------

It is there because that is what tomcat is doing inside their JMXProxy Servlet which I modified, and didn't take the time to understand all of the possible exceptions that could be thrown in each of the cases to clean it up.

Part of the reason for ignoring the exceptions is because JMX can execute arbitrary code inside the JMXBeans, which could throw all kinds of exceptions.  Ideally we would like the code to be robust so that if some beans are causing issues that we can still output data for the rest of the beans.  But I will look at what it takes to clean up the exception handling.

> Expose JMX with something like JMXProxyServlet 
> -----------------------------------------------
>
>                 Key: HADOOP-7144
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7144
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Luke Lu
>            Assignee: Robert Joseph Evans
>              Labels: jmx
>             Fix For: 0.23.0
>
>         Attachments: HADOOP-7411-0.20.20X-V1.patch, HADOOP-7411-trunk-V1.patch, HADOOP-7411-trunk-alpha.patch, jmx.json
>
>
> Much of the Hadoop metrics and status info is available via JMX, especially since 0.20.100, and 0.22+ (HDFS-1318, HADOOP-6728 etc.) For operations staff not familiar JMX setup, especially JMX with SSL and firewall tunnelling, the usage can be daunting. Using a JMXProxyServlet (a la Tomcat) to translate JMX attributes into JSON output would make a lot of non-Java admins happy.
> We could probably use Tomcat's JMXProxyServlet code directly, if it's already output some standard format (JSON or XML etc.) The code is simple enough to port over and can probably integrate with the common HttpServer as one of the default servelet (maybe /jmx) for the pluggable security.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15645-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 22:28:31 2011
Return-Path: <common-issues-return-15645-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 401E7426E
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 22:28:31 +0000 (UTC)
Received: (qmail 26419 invoked by uid 500); 31 May 2011 22:28:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26373 invoked by uid 500); 31 May 2011 22:28:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26360 invoked by uid 99); 31 May 2011 22:28:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 22:28:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 22:28:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id B7443ECA5A
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 22:27:47 +0000 (UTC)
Date: Tue, 31 May 2011 22:27:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <325261561.57659.1306880867747.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Updated] (HADOOP-7106) Re-organize hadoop subversion layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-7106:
--------------------------------

    Attachment: HADOOP-7106.sh

Another revision of the shell script that makes combined branches for HDFS-1073 out of (trunk,trunk,HDFS-1073) and MR-279 out of (yahoo-merge,MR-279,yahoo-merge).

Will also need to update the git fixer script for this.

> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15646-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 22:30:29 2011
Return-Path: <common-issues-return-15646-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 926BE4560
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 22:30:29 +0000 (UTC)
Received: (qmail 27910 invoked by uid 500); 31 May 2011 22:30:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27881 invoked by uid 500); 31 May 2011 22:30:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27873 invoked by uid 99); 31 May 2011 22:30:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 22:30:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 22:30:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 545EBECB26
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 22:29:48 +0000 (UTC)
Date: Tue, 31 May 2011 22:29:48 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1171877622.57706.1306880988342.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7106) Re-organize hadoop subversion
 layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041871#comment-13041871 ] 

Todd Lipcon commented on HADOOP-7106:
-------------------------------------

I agree with Owen - let's postpone the "shifting up a level" at this point. The extra layer of path heirarchy is just a minor annoyance. Maybe we can tackle it some other time -- but at this point I'm feeling pretty comfortable about the current plan and don't want to have to go through a bunch more testing.

Jukka: do you do IRC? I'm usually idling in #asfinfra as tlipcon

> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15647-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 22:30:32 2011
Return-Path: <common-issues-return-15647-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 321EC45B1
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 22:30:32 +0000 (UTC)
Received: (qmail 28226 invoked by uid 500); 31 May 2011 22:30:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28161 invoked by uid 500); 31 May 2011 22:30:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28146 invoked by uid 99); 31 May 2011 22:30:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 22:30:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 22:30:29 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 33EB2ECB22
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 22:29:48 +0000 (UTC)
Date: Tue, 31 May 2011 22:29:48 +0000 (UTC)
From: "Jukka Zitting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <878862694.57702.1306880988209.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <6014584.371831295027265635.JavaMail.jira@thor>
Subject: [jira] [Commented] (HADOOP-7106) Re-organize hadoop subversion
 layout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041870#comment-13041870 ] 

Jukka Zitting commented on HADOOP-7106:
---------------------------------------

Right! Sorry for overlooking your comment, Owen. If keeping the "common" bit in the svn URL is not a problem, then I'd recommend keeping it there to avoid the extra complexity.

> Re-organize hadoop subversion layout
> ------------------------------------
>
>                 Key: HADOOP-7106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7106
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Nigel Daley
>            Assignee: Nigel Daley
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-auth.patch, HADOOP-7106-git.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh, HADOOP-7106.sh
>
>
> As discussed on general@ at http://tinyurl.com/4q6lhxm

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15648-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 22:52:31 2011
Return-Path: <common-issues-return-15648-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F2B71479F
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 22:52:30 +0000 (UTC)
Received: (qmail 51210 invoked by uid 500); 31 May 2011 22:52:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51137 invoked by uid 500); 31 May 2011 22:52:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51129 invoked by uid 99); 31 May 2011 22:52:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 22:52:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 22:52:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 9A718EC1E1
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 22:51:47 +0000 (UTC)
Date: Tue, 31 May 2011 22:51:47 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <558167422.57769.1306882307629.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1470673003.56543.1306860527397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HADOOP-7341) Fix option parsing in CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HADOOP-7341:
--------------------------------

    Attachment: HADOOP-7341-2.patch

clean up a few warnings

> Fix option parsing in CommandFormat
> -----------------------------------
>
>                 Key: HADOOP-7341
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7341
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7341-2.patch, HADOOP-7341.patch
>
>
> CommandFormat currently allows options in any location within the args.  This is not the intended behavior for FsShell commands.  Prior to the redesign, the commands used to expect option processing to stop at the first non-option.
> CommandFormat was an existing class prior the redesign, but it was only used by "count" to find the -q flag.  All commands were converted to using this class, thus inherited the unintended behavior.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15649-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 23:03:30 2011
Return-Path: <common-issues-return-15649-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id B42C04B52
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 23:03:30 +0000 (UTC)
Received: (qmail 63209 invoked by uid 500); 31 May 2011 23:03:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63164 invoked by uid 500); 31 May 2011 23:03:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63155 invoked by uid 99); 31 May 2011 23:03:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 23:03:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 23:03:28 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 6089DEC4D1
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 23:02:47 +0000 (UTC)
Date: Tue, 31 May 2011 23:02:47 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <1184738979.57792.1306882967392.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <3255382.248681296161204958.JavaMail.jira@thor>
Subject: [jira] [Assigned] (HADOOP-7121) Exceptions while serializing IPC
 call response are not handled well
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-7121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon reassigned HADOOP-7121:
-----------------------------------

    Assignee: Todd Lipcon

> Exceptions while serializing IPC call response are not handled well
> -------------------------------------------------------------------
>
>                 Key: HADOOP-7121
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7121
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>              Labels: newbie
>             Fix For: 0.22.0
>
>
> We had a situation where for some reason the serialization of an RPC call's response was throwing OOME. When this happens, the exception is not caught, and the call never gets a response - the client just hangs. Additionally, the OOME propagated all the way to the top of the IPC handler and caused the handler. Plus, the Handler upon exit only logged to stdout and not to the log4j logs.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-15650-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org  Tue May 31 23:05:28 2011
Return-Path: <common-issues-return-15650-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C6BD3404A
	for <apmail-hadoop-common-issues-archive@minotaur.apache.org>; Tue, 31 May 2011 23:05:28 +0000 (UTC)
Received: (qmail 63883 invoked by uid 500); 31 May 2011 23:05:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63859 invoked by uid 500); 31 May 2011 23:05:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63851 invoked by uid 99); 31 May 2011 23:05:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 23:05:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2011 23:05:27 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116])
	by hel.zones.apache.org (Postfix) with ESMTP id 79549EC5F3
	for <common-issues@hadoop.apache.org>; Tue, 31 May 2011 23:04:47 +0000 (UTC)
Date: Tue, 31 May 2011 23:04:47 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <84557190.57802.1306883087493.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: <1470673003.56543.1306860527397.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HADOOP-7341) Fix option parsing in
 CommandFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041891#comment-13041891 ] 

Hadoop QA commented on HADOOP-7341:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12481006/HADOOP-7341-2.patch
  against trunk revision 1129905.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/548//testReport/
Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/548//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/548//console

This message is automatically generated.

> Fix option parsing in CommandFormat
> -----------------------------------
>
>                 Key: HADOOP-7341
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7341
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HADOOP-7341-2.patch, HADOOP-7341.patch
>
>
> CommandFormat currently allows options in any location within the args.  This is not the intended behavior for FsShell commands.  Prior to the redesign, the commands used to expect option processing to stop at the first non-option.
> CommandFormat was an existing class prior the redesign, but it was only used by "count" to find the -q flag.  All commands were converted to using this class, thus inherited the unintended behavior.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From common-issues-return-7099-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 00:20:49 2010
Return-Path: <common-issues-return-7099-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23610 invoked from network); 1 Apr 2010 00:20:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 00:20:49 -0000
Received: (qmail 69276 invoked by uid 500); 1 Apr 2010 00:20:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69174 invoked by uid 500); 1 Apr 2010 00:20:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69117 invoked by uid 99); 1 Apr 2010 00:20:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 00:20:48 +0000
X-ASF-Spam-Status: No, hits=-1189.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 00:20:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 51186234C4C5
	for <common-issues@hadoop.apache.org>; Thu,  1 Apr 2010 00:20:27 +0000 (UTC)
Message-ID: <446984836.621421270081227331.JavaMail.jira@brutus.apache.org>
Date: Thu, 1 Apr 2010 00:20:27 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6669) zlib.compress.level  ignored for
 DefaultCodec initialization
In-Reply-To: <1208299192.618051270074867352.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852191#action_12852191 ] 

Hadoop QA commented on HADOOP-6669:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12440413/HADOOP-6669-0.patch
  against trunk revision 927979.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/436/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/436/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/436/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/436/console

This message is automatically generated.

> zlib.compress.level  ignored for DefaultCodec initialization 
> -------------------------------------------------------------
>
>                 Key: HADOOP-6669
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6669
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Koji Noguchi
>            Assignee: Koji Noguchi
>            Priority: Minor
>         Attachments: HADOOP-6669-0.patch
>
>
> HADOOP-5879 added a compression level for codecs, but DefaultCodec seems to ignore this conf value when initialized.
> This is only when codec is first created.  reinit() probably sets it right. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7100-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 05:43:54 2010
Return-Path: <common-issues-return-7100-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40537 invoked from network); 1 Apr 2010 05:43:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 05:43:54 -0000
Received: (qmail 27885 invoked by uid 500); 1 Apr 2010 05:43:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27773 invoked by uid 500); 1 Apr 2010 05:43:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27765 invoked by uid 99); 1 Apr 2010 05:43:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 05:43:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 05:43:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 957F2234C4C5
	for <common-issues@hadoop.apache.org>; Thu,  1 Apr 2010 05:43:27 +0000 (UTC)
Message-ID: <1968610034.625451270100607611.JavaMail.jira@brutus.apache.org>
Date: Thu, 1 Apr 2010 05:43:27 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6658) Exclude Private elements from
 generated Javadoc
In-Reply-To: <995349254.452871269404067173.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852256#action_12852256 ] 

Hadoop QA commented on HADOOP-6658:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12439822/HADOOP-6658.patch
  against trunk revision 927979.

    +1 @author.  The patch does not contain any @author tags.

    +0 tests included.  The patch appears to be a documentation patch that doesn't require tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    -1 release audit.  The applied patch generated 2 release audit warnings (more than the trunk's current 1 warnings).

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/437/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/437/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/437/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/437/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/437/console

This message is automatically generated.

> Exclude Private elements from generated Javadoc
> -----------------------------------------------
>
>                 Key: HADOOP-6658
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6658
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-6658.patch, HADOOP-6658.patch
>
>
> Packages, classes and methods that are marked with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation should not appear in the public Javadoc. Developer Javadoc generated using the "javadoc-dev" ant target should continue to generate Javadoc for all program elements.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7101-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 06:07:53 2010
Return-Path: <common-issues-return-7101-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44448 invoked from network); 1 Apr 2010 06:07:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 06:07:52 -0000
Received: (qmail 41252 invoked by uid 500); 1 Apr 2010 06:07:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40939 invoked by uid 500); 1 Apr 2010 06:07:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40915 invoked by uid 99); 1 Apr 2010 06:07:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 06:07:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 06:07:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 5CDDD234C4CC
	for <common-issues@hadoop.apache.org>; Thu,  1 Apr 2010 06:07:27 +0000 (UTC)
Message-ID: <931278055.625871270102047379.JavaMail.jira@brutus.apache.org>
Date: Thu, 1 Apr 2010 06:07:27 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6670) UserGroupInformation doesn't support
 use in hash tables
In-Reply-To: <242202762.620161270078590059.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6670:
----------------------------------

    Attachment: fs-close.patch

This patch fixes the problem and the testcases. I'm still getting a test case failure in the dist cache one.

> UserGroupInformation doesn't support use in hash tables
> -------------------------------------------------------
>
>                 Key: HADOOP-6670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6670
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: fs-close.patch
>
>
> The UserGroupInformation objects are mutable, but they are used as keys in hash tables. This leads to serious problems in the FileSystem cache and RPC connection cache. We need to change the hashCode to be the identity hash code of the Subject and change equals to use == between the Subjects.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7102-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 09:26:54 2010
Return-Path: <common-issues-return-7102-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86520 invoked from network); 1 Apr 2010 09:26:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 09:26:54 -0000
Received: (qmail 26974 invoked by uid 500); 1 Apr 2010 09:26:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26883 invoked by uid 500); 1 Apr 2010 09:26:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26630 invoked by uid 99); 1 Apr 2010 09:26:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 09:26:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 09:26:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 6CCE4234C4C9
	for <common-issues@hadoop.apache.org>; Thu,  1 Apr 2010 09:26:27 +0000 (UTC)
Message-ID: <497521518.628591270113987444.JavaMail.jira@brutus.apache.org>
Date: Thu, 1 Apr 2010 09:26:27 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6640) FileSystem.get() does RPC retries
 within a static synchronized block
In-Reply-To: <1353980096.327641268867247264.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852301#action_12852301 ] 

Hadoop QA commented on HADOOP-6640:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12440376/getFS.patch
  against trunk revision 927979.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/438/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/438/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/438/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/438/console

This message is automatically generated.

> FileSystem.get() does RPC retries within a static synchronized block
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6640
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6640
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>         Environment: all
>            Reporter: Alejandro Abdelnur
>            Assignee: Hairong Kuang
>            Priority: Critical
>         Attachments: getFS.patch
>
>
> If using FileSystem.get() in a multithreaded environment, and one get() locks because the NN URI is too slow or not responding and retries are in progress, all other get() (for the diffferent users, NN) are blocked.
> the synchronized block in in the static instance of Cache inner class.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7103-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 11:10:52 2010
Return-Path: <common-issues-return-7103-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5711 invoked from network); 1 Apr 2010 11:10:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 11:10:52 -0000
Received: (qmail 29900 invoked by uid 500); 1 Apr 2010 11:10:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29862 invoked by uid 500); 1 Apr 2010 11:10:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29763 invoked by uid 99); 1 Apr 2010 11:10:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 11:10:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 11:10:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 2F150234C4B2
	for <common-issues@hadoop.apache.org>; Thu,  1 Apr 2010 11:10:27 +0000 (UTC)
Message-ID: <859246128.630391270120227191.JavaMail.jira@brutus.apache.org>
Date: Thu, 1 Apr 2010 11:10:27 +0000 (UTC)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6671) To use maven for hadoop common builds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

To use maven for hadoop common builds
-------------------------------------

                 Key: HADOOP-6671
                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
             Project: Hadoop Common
          Issue Type: Improvement
          Components: build
    Affects Versions: 0.22.0
            Reporter: Giridharan Kesavan


We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
Drawbacks with the current approach:
* Use ivy for dependency management with ivy.xml
* Use maven-ant-task for artifact publishing to the maven repository
* pom files are not generated dynamically 

To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.

I would like to have a branch created to work on mavenizing  hadoop common.


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7104-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 12:06:51 2010
Return-Path: <common-issues-return-7104-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22799 invoked from network); 1 Apr 2010 12:06:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 12:06:51 -0000
Received: (qmail 38260 invoked by uid 500); 1 Apr 2010 12:06:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38149 invoked by uid 500); 1 Apr 2010 12:06:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38135 invoked by uid 99); 1 Apr 2010 12:06:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 12:06:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 12:06:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 2B4E2234C4B2
	for <common-issues@hadoop.apache.org>; Thu,  1 Apr 2010 12:06:27 +0000 (UTC)
Message-ID: <603433340.633261270123587176.JavaMail.jira@brutus.apache.org>
Date: Thu, 1 Apr 2010 12:06:27 +0000 (UTC)
From: "Xiao Kang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6672) BytesWritable.write(buf) use much
 more CPU in writeInt() then write(buf)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

BytesWritable.write(buf) use much more CPU in writeInt() then write(buf)
------------------------------------------------------------------------

                 Key: HADOOP-6672
                 URL: https://issues.apache.org/jira/browse/HADOOP-6672
             Project: Hadoop Common
          Issue Type: Improvement
          Components: io
    Affects Versions: 0.20.2
            Reporter: Xiao Kang


BytesWritable.write() use nearly 4 times of CPU in wirteInt() as write buffer. It may be optimized.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7105-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 12:12:48 2010
Return-Path: <common-issues-return-7105-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23549 invoked from network); 1 Apr 2010 12:12:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 12:12:48 -0000
Received: (qmail 46102 invoked by uid 500); 1 Apr 2010 12:12:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46057 invoked by uid 500); 1 Apr 2010 12:12:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46040 invoked by uid 99); 1 Apr 2010 12:12:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 12:12:48 +0000
X-ASF-Spam-Status: No, hits=-1194.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 12:12:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3C198234C4B2
	for <common-issues@hadoop.apache.org>; Thu,  1 Apr 2010 12:12:27 +0000 (UTC)
Message-ID: <1427827168.633331270123947244.JavaMail.jira@brutus.apache.org>
Date: Thu, 1 Apr 2010 12:12:27 +0000 (UTC)
From: "Xiao Kang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6672) BytesWritable.write(buf) use much
 more CPU in writeInt() then write(buf)
In-Reply-To: <603433340.633261270123587176.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xiao Kang updated HADOOP-6672:
------------------------------

    Attachment: screenshot-1.jpg

A YJP screenshot attached.

comparing as following:

BytesWritable.write(DataOutput)  30%, 308k times
   DataOutputStream.writeInt()       23%, 308k times
       $Buffer.write()                             21%, 1235k times
   DataOutputStream.write()            6%, 308k times
       $Buffer.write()                             5%, 308k times

> BytesWritable.write(buf) use much more CPU in writeInt() then write(buf)
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6672
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6672
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Xiao Kang
>         Attachments: screenshot-1.jpg
>
>
> BytesWritable.write() use nearly 4 times of CPU in wirteInt() as write buffer. It may be optimized.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7106-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 14:24:49 2010
Return-Path: <common-issues-return-7106-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72587 invoked from network); 1 Apr 2010 14:24:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 14:24:48 -0000
Received: (qmail 45035 invoked by uid 500); 1 Apr 2010 14:24:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44809 invoked by uid 500); 1 Apr 2010 14:24:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44795 invoked by uid 99); 1 Apr 2010 14:24:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 14:24:48 +0000
X-ASF-Spam-Status: No, hits=-1195.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 14:24:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 378AA234C4C8
	for <common-issues@hadoop.apache.org>; Thu,  1 Apr 2010 14:24:27 +0000 (UTC)
Message-ID: <903886190.635511270131867226.JavaMail.jira@brutus.apache.org>
Date: Thu, 1 Apr 2010 14:24:27 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6671) To use maven for hadoop common
 builds
In-Reply-To: <859246128.630391270120227191.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852399#action_12852399 ] 

Allen Wittenauer commented on HADOOP-6671:
------------------------------------------

My initial thoughts is that every time someone tinkers with either ivy or maven or the build process in general, it always seems to get worse and worse for those of us that build on non-Internet connected machines. This sounds like another movement towards more pain.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7107-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 15:05:55 2010
Return-Path: <common-issues-return-7107-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85842 invoked from network); 1 Apr 2010 15:05:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 15:05:49 -0000
Received: (qmail 96459 invoked by uid 500); 1 Apr 2010 15:05:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96432 invoked by uid 500); 1 Apr 2010 15:05:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96424 invoked by uid 99); 1 Apr 2010 15:05:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 15:05:49 +0000
X-ASF-Spam-Status: No, hits=-1196.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 15:05:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1650E234C4C7
	for <common-issues@hadoop.apache.org>; Thu,  1 Apr 2010 15:05:28 +0000 (UTC)
Message-ID: <1419166118.635971270134328090.JavaMail.jira@brutus.apache.org>
Date: Thu, 1 Apr 2010 15:05:28 +0000 (UTC)
From: "Patrick Angeles (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6671) To use maven for hadoop common
 builds
In-Reply-To: <859246128.630391270120227191.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852409#action_12852409 ] 

Patrick Angeles commented on HADOOP-6671:
-----------------------------------------

Giri,

I like the idea, and I'll pitch in however I can.

@Allen

You can very easily set up an internal proxy for Maven. I'm assuming your build machines have some kind of network access to get to the source code...

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7108-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 15:15:57 2010
Return-Path: <common-issues-return-7108-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91993 invoked from network); 1 Apr 2010 15:15:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 15:15:57 -0000
Received: (qmail 8257 invoked by uid 500); 1 Apr 2010 15:15:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8227 invoked by uid 500); 1 Apr 2010 15:15:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8219 invoked by uid 99); 1 Apr 2010 15:15:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 15:15:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 15:15:55 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C69B5234C4C6
	for <common-issues@hadoop.apache.org>; Thu,  1 Apr 2010 15:15:33 +0000 (UTC)
Message-ID: <181748217.636131270134933812.JavaMail.jira@brutus.apache.org>
Date: Thu, 1 Apr 2010 15:15:33 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6671) To use maven for hadoop common
 builds
In-Reply-To: <859246128.630391270120227191.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852413#action_12852413 ] 

Allen Wittenauer commented on HADOOP-6671:
------------------------------------------

No, they don't.  

I bulid on my desktop mac, let ivy download all of its stuff, then scp .ivy2 and the other stuff that ivy pulls in to my real build machine.  So your assumption is false.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7109-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 15:21:51 2010
Return-Path: <common-issues-return-7109-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98014 invoked from network); 1 Apr 2010 15:21:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 15:21:51 -0000
Received: (qmail 23846 invoked by uid 500); 1 Apr 2010 15:21:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23811 invoked by uid 500); 1 Apr 2010 15:21:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23803 invoked by uid 99); 1 Apr 2010 15:21:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 15:21:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 15:21:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4B176234C4C6
	for <common-issues@hadoop.apache.org>; Thu,  1 Apr 2010 15:21:27 +0000 (UTC)
Message-ID: <1850719995.636261270135287306.JavaMail.jira@brutus.apache.org>
Date: Thu, 1 Apr 2010 15:21:27 +0000 (UTC)
From: "Patrick Angeles (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6671) To use maven for hadoop common
 builds
In-Reply-To: <859246128.630391270120227191.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852416#action_12852416 ] 

Patrick Angeles commented on HADOOP-6671:
-----------------------------------------

How do you scp without network access?

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7110-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 15:25:48 2010
Return-Path: <common-issues-return-7110-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99272 invoked from network); 1 Apr 2010 15:25:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 15:25:48 -0000
Received: (qmail 26132 invoked by uid 500); 1 Apr 2010 15:25:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26105 invoked by uid 500); 1 Apr 2010 15:25:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26097 invoked by uid 99); 1 Apr 2010 15:25:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 15:25:48 +0000
X-ASF-Spam-Status: No, hits=-1196.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 15:25:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 32459234C4C7
	for <common-issues@hadoop.apache.org>; Thu,  1 Apr 2010 15:25:27 +0000 (UTC)
Message-ID: <864816868.636371270135527205.JavaMail.jira@brutus.apache.org>
Date: Thu, 1 Apr 2010 15:25:27 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6671) To use maven for hadoop common
 builds
In-Reply-To: <859246128.630391270120227191.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852419#action_12852419 ] 

Allen Wittenauer commented on HADOOP-6671:
------------------------------------------

They don't have access to the Internet.  They do have internal network access. And, no, I'm not willing to setup a service to build code.  I'm sure Maven is a great idea if you have a ton of resources (mainly time and effort).  I don't.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7111-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 15:52:51 2010
Return-Path: <common-issues-return-7111-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8698 invoked from network); 1 Apr 2010 15:52:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 15:52:51 -0000
Received: (qmail 71822 invoked by uid 500); 1 Apr 2010 15:52:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71780 invoked by uid 500); 1 Apr 2010 15:52:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71772 invoked by uid 99); 1 Apr 2010 15:52:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 15:52:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 15:52:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 37F44234C4C7
	for <common-issues@hadoop.apache.org>; Thu,  1 Apr 2010 15:52:27 +0000 (UTC)
Message-ID: <107979518.636661270137147228.JavaMail.jira@brutus.apache.org>
Date: Thu, 1 Apr 2010 15:52:27 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6671) To use maven for hadoop common
 builds
In-Reply-To: <859246128.630391270120227191.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852423#action_12852423 ] 

Steve Loughran commented on HADOOP-6671:
----------------------------------------

I've had bad experiences with M2 in the past; these colour my opinions. I don't know how much maven2 has improved since then. 

What I do have to deal with on a regular basis, even today, is people who write POM files who get the dependencies correct for their own build and test, but which screw up everyone else downstream. Recent logging JARs are an example. Accordingly, I view a POM file as an artifact for downstream users that you have to get right, not just some internal thing, as we can do today with ivy and ant files.

This means saying "we should move to maven to eliminate having ivy.xml and POM files" isn't a good enough reason for me. If it improves testing, build times,-even to reduce ivy and ant xml maintenance costs, then yes , but not just "because you can".

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7112-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 16:02:51 2010
Return-Path: <common-issues-return-7112-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11408 invoked from network); 1 Apr 2010 16:02:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 16:02:51 -0000
Received: (qmail 83132 invoked by uid 500); 1 Apr 2010 16:02:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83100 invoked by uid 500); 1 Apr 2010 16:02:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83077 invoked by uid 99); 1 Apr 2010 16:02:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 16:02:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 16:02:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4A818234C4C6
	for <common-issues@hadoop.apache.org>; Thu,  1 Apr 2010 16:02:27 +0000 (UTC)
Message-ID: <1453163482.636841270137747304.JavaMail.jira@brutus.apache.org>
Date: Thu, 1 Apr 2010 16:02:27 +0000 (UTC)
From: "Koji Noguchi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6669) zlib.compress.level  ignored for
 DefaultCodec initialization
In-Reply-To: <1208299192.618051270074867352.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852428#action_12852428 ] 

Koji Noguchi commented on HADOOP-6669:
--------------------------------------

Hm. Didn't know that native library is not used in hudson testing.

My unit test was testing
1) gzipcodec-native
2) defaultcodec-native
3) defaultcodec-non-native

(1) and (2) are skipped in hudson testing.

Hudson testReport showing
bq. 2010-04-01 00:15:52,882 WARN  compress.TestCodec (TestCodec.java:testCodecInitWithCompressionLevel(373)) - testCodecInitWithCompressionLevel for native skipped: native libs not loaded

Testing manually, (2) and (3) failed without the patch and succeeded with the patch.


> zlib.compress.level  ignored for DefaultCodec initialization 
> -------------------------------------------------------------
>
>                 Key: HADOOP-6669
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6669
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Koji Noguchi
>            Assignee: Koji Noguchi
>            Priority: Minor
>         Attachments: HADOOP-6669-0.patch
>
>
> HADOOP-5879 added a compression level for codecs, but DefaultCodec seems to ignore this conf value when initialized.
> This is only when codec is first created.  reinit() probably sets it right. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7113-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 16:14:48 2010
Return-Path: <common-issues-return-7113-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14586 invoked from network); 1 Apr 2010 16:14:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 16:14:48 -0000
Received: (qmail 96161 invoked by uid 500); 1 Apr 2010 16:14:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96119 invoked by uid 500); 1 Apr 2010 16:14:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96111 invoked by uid 99); 1 Apr 2010 16:14:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 16:14:48 +0000
X-ASF-Spam-Status: No, hits=-1196.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 16:14:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3E95C234C4C8
	for <common-issues@hadoop.apache.org>; Thu,  1 Apr 2010 16:14:27 +0000 (UTC)
Message-ID: <1215461248.636991270138467255.JavaMail.jira@brutus.apache.org>
Date: Thu, 1 Apr 2010 16:14:27 +0000 (UTC)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6671) To use maven for hadoop common
 builds
In-Reply-To: <859246128.630391270120227191.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852432#action_12852432 ] 

Giridharan Kesavan commented on HADOOP-6671:
--------------------------------------------

@Allen 
you can use mvn to do offline builds by passing -o argument 

@Steve 
Apart from maintaining ivy.xml and pom.xml, by having the pom file it reduces the build scripts maintenance. 
As the pom file has got standards for doing any kind of build stuff and its easy to maintain. 


> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7114-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 16:58:50 2010
Return-Path: <common-issues-return-7114-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26383 invoked from network); 1 Apr 2010 16:58:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 16:58:50 -0000
Received: (qmail 55867 invoked by uid 500); 1 Apr 2010 16:58:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55833 invoked by uid 500); 1 Apr 2010 16:58:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55802 invoked by uid 99); 1 Apr 2010 16:58:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 16:58:49 +0000
X-ASF-Spam-Status: No, hits=-1196.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 16:58:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3534F234C4C5
	for <common-issues@hadoop.apache.org>; Thu,  1 Apr 2010 16:58:28 +0000 (UTC)
Message-ID: <1070436824.637781270141108212.JavaMail.jira@brutus.apache.org>
Date: Thu, 1 Apr 2010 16:58:28 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6658) Exclude Private elements from
 generated Javadoc
In-Reply-To: <995349254.452871269404067173.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6658:
------------------------------

    Status: Open  (was: Patch Available)

> Exclude Private elements from generated Javadoc
> -----------------------------------------------
>
>                 Key: HADOOP-6658
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6658
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-6658.patch, HADOOP-6658.patch
>
>
> Packages, classes and methods that are marked with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation should not appear in the public Javadoc. Developer Javadoc generated using the "javadoc-dev" ant target should continue to generate Javadoc for all program elements.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7115-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 18:21:49 2010
Return-Path: <common-issues-return-7115-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47049 invoked from network); 1 Apr 2010 18:21:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 18:21:48 -0000
Received: (qmail 1544 invoked by uid 500); 1 Apr 2010 18:21:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1501 invoked by uid 500); 1 Apr 2010 18:21:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1344 invoked by uid 99); 1 Apr 2010 18:21:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 18:21:48 +0000
X-ASF-Spam-Status: No, hits=-1197.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 18:21:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3F291234C4CC
	for <common-issues@hadoop.apache.org>; Thu,  1 Apr 2010 18:21:27 +0000 (UTC)
Message-ID: <728382734.639531270146087257.JavaMail.jira@brutus.apache.org>
Date: Thu, 1 Apr 2010 18:21:27 +0000 (UTC)
From: "Lee Tucker (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6671) To use maven for hadoop common
 builds
In-Reply-To: <859246128.630391270120227191.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852490#action_12852490 ] 

Lee Tucker commented on HADOOP-6671:
------------------------------------

What it does do is provide a plugin architecture for a whole range of tools instead of having to roll each one independently into the build.xml.   It provides project structure and actually improves inter-project communication by managing transitive build dependencies.    Given that we already rely on net access (and maven repositories) to pull things through Ivy for dependencies, I'm not exactly sure how Maven makes that any worse.   You had to fill a cache somehow already.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7116-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 19:59:48 2010
Return-Path: <common-issues-return-7116-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82553 invoked from network); 1 Apr 2010 19:59:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 19:59:48 -0000
Received: (qmail 28305 invoked by uid 500); 1 Apr 2010 19:59:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28275 invoked by uid 500); 1 Apr 2010 19:59:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28260 invoked by uid 99); 1 Apr 2010 19:59:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 19:59:48 +0000
X-ASF-Spam-Status: No, hits=-1198.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 19:59:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 35F3A234C4B2
	for <common-issues@hadoop.apache.org>; Thu,  1 Apr 2010 19:59:27 +0000 (UTC)
Message-ID: <1956865453.641391270151967220.JavaMail.jira@brutus.apache.org>
Date: Thu, 1 Apr 2010 19:59:27 +0000 (UTC)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6673) nightly builds have incorrect
 VersionInfo
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

nightly builds have incorrect VersionInfo
-----------------------------------------

                 Key: HADOOP-6673
                 URL: https://issues.apache.org/jira/browse/HADOOP-6673
             Project: Hadoop Common
          Issue Type: Bug
          Components: build
            Reporter: Aaron Kimball


Snapshots of Hadoop trunk downloaded from Ivy have VersionInfo.getVersion() returning "Unknown"

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7117-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 20:16:51 2010
Return-Path: <common-issues-return-7117-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88236 invoked from network); 1 Apr 2010 20:16:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 20:16:51 -0000
Received: (qmail 68712 invoked by uid 500); 1 Apr 2010 20:16:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68680 invoked by uid 500); 1 Apr 2010 20:16:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68661 invoked by uid 99); 1 Apr 2010 20:16:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 20:16:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 20:16:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 5DB8B234C4E1
	for <common-issues@hadoop.apache.org>; Thu,  1 Apr 2010 20:16:27 +0000 (UTC)
Message-ID: <1519562878.641851270152987383.JavaMail.jira@brutus.apache.org>
Date: Thu, 1 Apr 2010 20:16:27 +0000 (UTC)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6659) Switch RPC to use Avro
In-Reply-To: <387448273.476441269469290584.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852530#action_12852530 ] 

Sanjay Radia commented on HADOOP-6659:
--------------------------------------

A design document as Jeff suggested would be useful. 

I assume that the goal is still to make the RPC completely pulggable. i.e Till the AVRO path is stable, the default config uses the Hadoop  RPC with the old serialization and that there is no -ve performance impact on the 
old code paths. 

> Switch RPC to use Avro
> ----------------------
>
>                 Key: HADOOP-6659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6659
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Doug Cutting
>
> This is an umbrella issue for moving HDFS and MapReduce RPC to use Avro.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7118-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 20:50:51 2010
Return-Path: <common-issues-return-7118-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94541 invoked from network); 1 Apr 2010 20:50:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 20:50:51 -0000
Received: (qmail 15688 invoked by uid 500); 1 Apr 2010 20:50:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15629 invoked by uid 500); 1 Apr 2010 20:50:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15621 invoked by uid 99); 1 Apr 2010 20:50:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 20:50:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 20:50:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id AEECE234C4E5
	for <common-issues@hadoop.apache.org>; Thu,  1 Apr 2010 20:50:27 +0000 (UTC)
Message-ID: <404415091.642751270155027715.JavaMail.jira@brutus.apache.org>
Date: Thu, 1 Apr 2010 20:50:27 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6670) UserGroupInformation doesn't support
 use in hash tables
In-Reply-To: <242202762.620161270078590059.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6670:
----------------------------------

    Attachment: fs-close.patch

This patch changes UGI.hashCode to be the identity hash of the Subject and equals to check == between the subjects.

> UserGroupInformation doesn't support use in hash tables
> -------------------------------------------------------
>
>                 Key: HADOOP-6670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6670
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: fs-close.patch, fs-close.patch
>
>
> The UserGroupInformation objects are mutable, but they are used as keys in hash tables. This leads to serious problems in the FileSystem cache and RPC connection cache. We need to change the hashCode to be the identity hash code of the Subject and change equals to use == between the Subjects.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7119-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 20:50:52 2010
Return-Path: <common-issues-return-7119-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94673 invoked from network); 1 Apr 2010 20:50:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 20:50:51 -0000
Received: (qmail 15845 invoked by uid 500); 1 Apr 2010 20:50:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15806 invoked by uid 500); 1 Apr 2010 20:50:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15798 invoked by uid 99); 1 Apr 2010 20:50:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 20:50:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 20:50:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id A9718234C4E1
	for <common-issues@hadoop.apache.org>; Thu,  1 Apr 2010 20:50:27 +0000 (UTC)
Message-ID: <1418757389.642711270155027693.JavaMail.jira@brutus.apache.org>
Date: Thu, 1 Apr 2010 20:50:27 +0000 (UTC)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6659) Switch RPC to use Avro
In-Reply-To: <387448273.476441269469290584.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852541#action_12852541 ] 

Doug Cutting commented on HADOOP-6659:
--------------------------------------

> A design document as Jeff suggested would be useful.

What's unspecified in current issues?  If you like I can collate the comments on the various issues linked here into a single document if that would make them more readable for you.

> I assume that the goal is still to make the RPC completely pluggable. i.e 'til the AVRO path is stable, the default config uses the Hadoop RPC with the old serialization and that there is no -ve performance impact on the
old code paths.

RPC is already pluggable w/o negative performance impact.  That was done in HADOOP-6422.

I don't expect that switching to Avro serialization for RPCs will affect performance, but that should certainly be tested before we make Avro serialization the default.  Switching the transport is more likely to affect performance, but that switch can be made separately and after switching serializations.

So I see something like the following steps:
 - HDFS
 -- get hdfs tests to pass using Avro RPC serialization
 -- test hdfs performance using Avro RPC serialization
 -- switch HDFS to use Avro RPC serialization by default 
 -- design, implement and switch HDFS to use IDL-driven Avro RPC (HDFS-1069)
 - MapReduce
 -- get mapreduce tests to pass using Avro RPC serialization
 -- test mapreduce performance using Avro RPC serialization
 -- switch Mapreduce to use Avro RPC serialization by default
 -- design, implement and switch MapReduce to use IDL-driven Avro RPC (HDFS-1069)
 - Transport
  -- Design and develop an interoperable, secure, high-performance Avro transport (AVRO-341)
  -- port HDFS and MapReduce to use this optionally
  -- test HDFS and mapreduce with this new transport
  -- switch HDFS and Mapreduce to use new transport by default

Would it be useful to file a Jira issue for each of these?

> Switch RPC to use Avro
> ----------------------
>
>                 Key: HADOOP-6659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6659
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Doug Cutting
>
> This is an umbrella issue for moving HDFS and MapReduce RPC to use Avro.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7120-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 21:12:48 2010
Return-Path: <common-issues-return-7120-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 649 invoked from network); 1 Apr 2010 21:12:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 21:12:48 -0000
Received: (qmail 40671 invoked by uid 500); 1 Apr 2010 21:12:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40644 invoked by uid 500); 1 Apr 2010 21:12:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40636 invoked by uid 99); 1 Apr 2010 21:12:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 21:12:48 +0000
X-ASF-Spam-Status: No, hits=-1198.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 21:12:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 39401234C4CB
	for <common-issues@hadoop.apache.org>; Thu,  1 Apr 2010 21:12:27 +0000 (UTC)
Message-ID: <1870935114.643041270156347233.JavaMail.jira@brutus.apache.org>
Date: Thu, 1 Apr 2010 21:12:27 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6640) FileSystem.get() does RPC retries
 within a static synchronized block
In-Reply-To: <1353980096.327641268867247264.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852549#action_12852549 ] 

Hudson commented on HADOOP-6640:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #213 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/213/])
    . FileSystem.get() does RPC retries within a static synchronized block. Contributed by Hairong Kuang.


> FileSystem.get() does RPC retries within a static synchronized block
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6640
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6640
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>         Environment: all
>            Reporter: Alejandro Abdelnur
>            Assignee: Hairong Kuang
>            Priority: Critical
>         Attachments: getFS.patch
>
>
> If using FileSystem.get() in a multithreaded environment, and one get() locks because the NN URI is too slow or not responding and retries are in progress, all other get() (for the diffferent users, NN) are blocked.
> the synchronized block in in the static instance of Cache inner class.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7121-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 21:36:48 2010
Return-Path: <common-issues-return-7121-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5741 invoked from network); 1 Apr 2010 21:36:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 21:36:48 -0000
Received: (qmail 71481 invoked by uid 500); 1 Apr 2010 21:36:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71452 invoked by uid 500); 1 Apr 2010 21:36:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71420 invoked by uid 99); 1 Apr 2010 21:36:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 21:36:48 +0000
X-ASF-Spam-Status: No, hits=-1198.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 21:36:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 490CC234C4D4
	for <common-issues@hadoop.apache.org>; Thu,  1 Apr 2010 21:36:27 +0000 (UTC)
Message-ID: <1610879694.643521270157787298.JavaMail.jira@brutus.apache.org>
Date: Thu, 1 Apr 2010 21:36:27 +0000 (UTC)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6640) FileSystem.get() does RPC retries
 within a static synchronized block
In-Reply-To: <1353980096.327641268867247264.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6640:
----------------------------------

       Resolution: Fixed
    Fix Version/s: 0.22.0
           Status: Resolved  (was: Patch Available)

I've just committed this!

> FileSystem.get() does RPC retries within a static synchronized block
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6640
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6640
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>         Environment: all
>            Reporter: Alejandro Abdelnur
>            Assignee: Hairong Kuang
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: getFS.patch
>
>
> If using FileSystem.get() in a multithreaded environment, and one get() locks because the NN URI is too slow or not responding and retries are in progress, all other get() (for the diffferent users, NN) are blocked.
> the synchronized block in in the static instance of Cache inner class.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7122-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 21:54:48 2010
Return-Path: <common-issues-return-7122-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11447 invoked from network); 1 Apr 2010 21:54:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 21:54:48 -0000
Received: (qmail 97233 invoked by uid 500); 1 Apr 2010 21:54:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97190 invoked by uid 500); 1 Apr 2010 21:54:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97174 invoked by uid 99); 1 Apr 2010 21:54:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 21:54:48 +0000
X-ASF-Spam-Status: No, hits=-1198.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 21:54:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 6FF01234C4CB
	for <common-issues@hadoop.apache.org>; Thu,  1 Apr 2010 21:54:27 +0000 (UTC)
Message-ID: <341005357.644031270158867457.JavaMail.jira@brutus.apache.org>
Date: Thu, 1 Apr 2010 21:54:27 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6671) To use maven for hadoop common
 builds
In-Reply-To: <859246128.630391270120227191.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852569#action_12852569 ] 

Allen Wittenauer commented on HADOOP-6671:
------------------------------------------

Owen and Lee have told me that offline builds will have basically the same amount of pain that they do now.  I'd be happier with less pain, but same is acceptable.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7123-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 22:24:52 2010
Return-Path: <common-issues-return-7123-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25968 invoked from network); 1 Apr 2010 22:24:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 22:24:51 -0000
Received: (qmail 29207 invoked by uid 500); 1 Apr 2010 22:24:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29152 invoked by uid 500); 1 Apr 2010 22:24:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29144 invoked by uid 99); 1 Apr 2010 22:24:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 22:24:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 22:24:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 66525234C4C9
	for <common-issues@hadoop.apache.org>; Thu,  1 Apr 2010 22:24:27 +0000 (UTC)
Message-ID: <366054150.644541270160667418.JavaMail.jira@brutus.apache.org>
Date: Thu, 1 Apr 2010 22:24:27 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6668:
------------------------------

    Attachment: HADOOP-6668.patch

I have had a first attempt at applying annotations to the common codebase. They are summarized in this table; consult the patch for finer-grained detail.

||Package org.apache.hadoop.|| Visibility & Stability || Notes ||
|classification|public stable| Should we mark them evolving?|
|conf|public stable| |
|fs|public stable|filesystem implementations are public stable, but any internal classes are private unstable|
|fs.shell|private unstable| |
|http|limited-private unstable| |
|io|public stable| |
|io.compress|public stable| |
|io.compress.bzip2|private unstable| |
|io.compress.zlib|private unstable| |
|io.file.tfile|public evolving| |
|io.retry|limited-private evolving| |
|io.serializer*|public evolving| |
|ipc*|limited-private evolving| |
|metrics*|public evolving| |
|net|mixture of public evolving and limited-private unstable| |
|record*|deprecated| |
|security*|mixture of public/limited-private evolving?| |
|util|public evolving|GenericOptionsParser, ReflectionUtils, Progressable, Tool, ToolRunner are public stable|
|util.bloom|public stable| |
|util.hash|public stable| |

I haven't annotated the security APIs yet - if anyone has suggestions for these, I'd be grateful.

I'm wondering whether to mark deprecated elements as 'evolving' (if they were deprecated in 0.19 or earlier), so that they can be removed sooner than the next major release. We wouldn't necessarily do this, but it might be wise to have the option.

Here are some guidelines that I used in applying the annotations. Selecting the annotations to apply is a judgement call, so I'd appreciate feedback both on the general rules and the particular annotations I have applied.
* Public user APIs should be marked as 'public stable'. These appear in public Javadoc.
* APIs that are exposed to the user but not coded against directly should be marked as 'public evolving' (e.g. the metrics API, which is used in configuration).  These appear in public Javadoc.
* Non-user facing APIs that are used within Hadoop subprojects should be marked as 'limited-private evolving' (e.g. RPC) or 'limited-private unstable'. These appear in developer Javadoc.
* Anything that is private implementation code should be marked as 'private unstable'. These appear in developer Javadoc.
* If an API is not annotated then it defaults to the usual Java visibility and Apache Hadoop stability (http://wiki.apache.org/hadoop/Roadmap) rules.


> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7124-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 22:30:52 2010
Return-Path: <common-issues-return-7124-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26633 invoked from network); 1 Apr 2010 22:30:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 22:30:52 -0000
Received: (qmail 34415 invoked by uid 500); 1 Apr 2010 22:30:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34380 invoked by uid 500); 1 Apr 2010 22:30:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34347 invoked by uid 99); 1 Apr 2010 22:30:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 22:30:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 22:30:49 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E4827234C4C5
	for <common-issues@hadoop.apache.org>; Thu,  1 Apr 2010 22:30:28 +0000 (UTC)
Message-ID: <2054761731.644751270161028935.JavaMail.jira@brutus.apache.org>
Date: Thu, 1 Apr 2010 22:30:28 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6668:
------------------------------

    Status: Patch Available  (was: Open)

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7125-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 22:54:51 2010
Return-Path: <common-issues-return-7125-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30096 invoked from network); 1 Apr 2010 22:54:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 22:54:51 -0000
Received: (qmail 55468 invoked by uid 500); 1 Apr 2010 22:54:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55440 invoked by uid 500); 1 Apr 2010 22:54:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55432 invoked by uid 99); 1 Apr 2010 22:54:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 22:54:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 22:54:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 6DFB2234C4CC
	for <common-issues@hadoop.apache.org>; Thu,  1 Apr 2010 22:54:27 +0000 (UTC)
Message-ID: <1897051467.645171270162467449.JavaMail.jira@brutus.apache.org>
Date: Thu, 1 Apr 2010 22:54:27 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852590#action_12852590 ] 

Tom White commented on HADOOP-6668:
-----------------------------------

With this patch (and HADOOP-6658) the public Javadoc would look like this: http://people.apache.org/~tomwhite/HADOOP-6668/docs/api/

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7126-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 23:10:51 2010
Return-Path: <common-issues-return-7126-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34678 invoked from network); 1 Apr 2010 23:10:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 23:10:50 -0000
Received: (qmail 69723 invoked by uid 500); 1 Apr 2010 23:10:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69684 invoked by uid 500); 1 Apr 2010 23:10:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69676 invoked by uid 99); 1 Apr 2010 23:10:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 23:10:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 23:10:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3D7AC234C4C6
	for <common-issues@hadoop.apache.org>; Thu,  1 Apr 2010 23:10:27 +0000 (UTC)
Message-ID: <1367642977.645421270163427250.JavaMail.jira@brutus.apache.org>
Date: Thu, 1 Apr 2010 23:10:27 +0000 (UTC)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852594#action_12852594 ] 

Doug Cutting commented on HADOOP-6668:
--------------------------------------

The io.serializer.avro package should probably be public evolving.  Perhaps all of io.serialization should be?

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7127-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 01 23:16:48 2010
Return-Path: <common-issues-return-7127-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35369 invoked from network); 1 Apr 2010 23:16:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Apr 2010 23:16:48 -0000
Received: (qmail 73376 invoked by uid 500); 1 Apr 2010 23:16:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73291 invoked by uid 500); 1 Apr 2010 23:16:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73283 invoked by uid 99); 1 Apr 2010 23:16:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 23:16:48 +0000
X-ASF-Spam-Status: No, hits=-1199.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Apr 2010 23:16:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3728B234C4BE
	for <common-issues@hadoop.apache.org>; Thu,  1 Apr 2010 23:16:27 +0000 (UTC)
Message-ID: <375873020.645441270163787224.JavaMail.jira@brutus.apache.org>
Date: Thu, 1 Apr 2010 23:16:27 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6674) Performance Improvement in Secure RPC
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Performance Improvement in Secure RPC
-------------------------------------

                 Key: HADOOP-6674
                 URL: https://issues.apache.org/jira/browse/HADOOP-6674
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Jitendra Nath Pandey
            Assignee: Jitendra Nath Pandey


This jira introduces two performance improvements in Sasl RPC
1. Setting of Sasl 'quality of protection' to authentication only.
2. Addition of BufferedOutputStream underneath SaslOutputStream.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7128-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 02 00:25:51 2010
Return-Path: <common-issues-return-7128-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45543 invoked from network); 2 Apr 2010 00:25:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Apr 2010 00:25:51 -0000
Received: (qmail 13549 invoked by uid 500); 2 Apr 2010 00:25:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13520 invoked by uid 500); 2 Apr 2010 00:25:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13512 invoked by uid 99); 2 Apr 2010 00:25:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 00:25:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 00:25:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 38786234C4C7
	for <common-issues@hadoop.apache.org>; Fri,  2 Apr 2010 00:25:27 +0000 (UTC)
Message-ID: <1945611333.647171270167927230.JavaMail.jira@brutus.apache.org>
Date: Fri, 2 Apr 2010 00:25:27 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852618#action_12852618 ] 

Hadoop QA commented on HADOOP-6668:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12440548/HADOOP-6668.patch
  against trunk revision 930096.

    +1 @author.  The patch does not contain any @author tags.

    +0 tests included.  The patch appears to be a documentation patch that doesn't require tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/439/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/439/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/439/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/439/console

This message is automatically generated.

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7129-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 02 00:31:49 2010
Return-Path: <common-issues-return-7129-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52750 invoked from network); 2 Apr 2010 00:31:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Apr 2010 00:31:49 -0000
Received: (qmail 16215 invoked by uid 500); 2 Apr 2010 00:31:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16189 invoked by uid 500); 2 Apr 2010 00:31:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16181 invoked by uid 99); 2 Apr 2010 00:31:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 00:31:49 +0000
X-ASF-Spam-Status: No, hits=-1200.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 00:31:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1D6D1234C4C6
	for <common-issues@hadoop.apache.org>; Fri,  2 Apr 2010 00:31:28 +0000 (UTC)
Message-ID: <933986032.647271270168288119.JavaMail.jira@brutus.apache.org>
Date: Fri, 2 Apr 2010 00:31:28 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852620#action_12852620 ] 

Tom White commented on HADOOP-6668:
-----------------------------------

> The io.serializer.avro package should probably be public evolving. Perhaps all of io.serialization should be?

I agree - I marked all of the serializer classes as public evolving in the patch.

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7130-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 02 00:45:50 2010
Return-Path: <common-issues-return-7130-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54877 invoked from network); 2 Apr 2010 00:45:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Apr 2010 00:45:50 -0000
Received: (qmail 21024 invoked by uid 500); 2 Apr 2010 00:45:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20988 invoked by uid 500); 2 Apr 2010 00:45:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20980 invoked by uid 99); 2 Apr 2010 00:45:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 00:45:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 00:45:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3E50E234C4C5
	for <common-issues@hadoop.apache.org>; Fri,  2 Apr 2010 00:45:27 +0000 (UTC)
Message-ID: <486892632.647511270169127254.JavaMail.jira@brutus.apache.org>
Date: Fri, 2 Apr 2010 00:45:27 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6674) Performance Improvement in Secure RPC
In-Reply-To: <375873020.645441270163787224.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6674:
-----------------------------------------

    Attachment: HADOOP-6674-y20.1.patch

Patch for 20 branch. 

> Performance Improvement in Secure RPC
> -------------------------------------
>
>                 Key: HADOOP-6674
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6674
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6674-y20.1.patch
>
>
> This jira introduces two performance improvements in Sasl RPC
> 1. Setting of Sasl 'quality of protection' to authentication only.
> 2. Addition of BufferedOutputStream underneath SaslOutputStream.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7131-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 02 01:29:51 2010
Return-Path: <common-issues-return-7131-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61175 invoked from network); 2 Apr 2010 01:29:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Apr 2010 01:29:51 -0000
Received: (qmail 38914 invoked by uid 500); 2 Apr 2010 01:29:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38815 invoked by uid 500); 2 Apr 2010 01:29:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38807 invoked by uid 99); 2 Apr 2010 01:29:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 01:29:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 01:29:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 8A05F234C4E3
	for <common-issues@hadoop.apache.org>; Fri,  2 Apr 2010 01:29:27 +0000 (UTC)
Message-ID: <2108375054.648301270171767564.JavaMail.jira@brutus.apache.org>
Date: Fri, 2 Apr 2010 01:29:27 +0000 (UTC)
From: "Alex Newman (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5670) Hadoop configurations should be
 read from a distributed system
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852633#action_12852633 ] 

Alex Newman commented on HADOOP-5670:
-------------------------------------

Howdy, 

This is really exciting as I feel currently  that configs are a hard problem. Currently it can be difficult to get transparency and flexibility depending on the configs scope. Specifically, where do I audit if a user sets a particular job paramater, or an admin sets a paramater for a particular system. It seems that this jira is about setting up scope on a cluster wide basis. LDAP and Zookeeper could allow configs across clusters. 

This is something that gasp vms logicals got right IMO in that it is one interface to solve all of these problems. I think one giant key value store is a good first step, but realistically you need different permissions based on if you are making a change intra cluster, inter cluster, for one box, for one user or for one job. Being able to go back and see what the hierarchy of those values, and having them being centrally managed would be really neat. Just my two cents, I am just spoiled from my slow big machines.

> Hadoop configurations should be read from a distributed system
> --------------------------------------------------------------
>
>                 Key: HADOOP-5670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5670
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: conf
>            Reporter: Allen Wittenauer
>
> Rather than distributing the hadoop configuration files to every data node, compute node, etc, Hadoop should be able to read configuration information (dynamically!) from LDAP, ZooKeeper, whatever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7135-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 02 13:36:53 2010
Return-Path: <common-issues-return-7135-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8637 invoked from network); 2 Apr 2010 13:36:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Apr 2010 13:36:50 -0000
Received: (qmail 5200 invoked by uid 500); 2 Apr 2010 13:36:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5074 invoked by uid 500); 2 Apr 2010 13:36:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4870 invoked by uid 99); 2 Apr 2010 13:36:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 13:36:49 +0000
X-ASF-Spam-Status: No, hits=-1204.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 13:36:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3A57B234C4BE
	for <common-issues@hadoop.apache.org>; Fri,  2 Apr 2010 13:36:27 +0000 (UTC)
Message-ID: <958823413.659461270215387238.JavaMail.jira@brutus.apache.org>
Date: Fri, 2 Apr 2010 13:36:27 +0000 (UTC)
From: "Xiao Kang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6672) BytesWritable.write(buf) use much
 more CPU in writeInt() then write(buf)
In-Reply-To: <603433340.633261270123587176.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xiao Kang updated HADOOP-6672:
------------------------------

    Attachment: BytesWritable.java.patch

Patch attached.

Transfer int to a four bytes buffer ints and use write(ints, 0, 4) to reduce invocation count of write. 

> BytesWritable.write(buf) use much more CPU in writeInt() then write(buf)
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6672
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6672
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Xiao Kang
>         Attachments: BytesWritable.java.patch, screenshot-1.jpg
>
>
> BytesWritable.write() use nearly 4 times of CPU in wirteInt() as write buffer. It may be optimized.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7133-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 02 14:01:32 2010
Return-Path: <common-issues-return-7133-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36017 invoked from network); 2 Apr 2010 14:01:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Apr 2010 14:01:32 -0000
Received: (qmail 15709 invoked by uid 500); 2 Apr 2010 08:34:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15656 invoked by uid 500); 2 Apr 2010 08:34:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15647 invoked by uid 99); 2 Apr 2010 08:34:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 08:34:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 08:34:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 458FA234C4C8
	for <common-issues@hadoop.apache.org>; Fri,  2 Apr 2010 08:34:27 +0000 (UTC)
Message-ID: <2074106225.654861270197267284.JavaMail.jira@brutus.apache.org>
Date: Fri, 2 Apr 2010 08:34:27 +0000 (UTC)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4196) Possible performance enhancement in
 Hadoop compress module
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852751#action_12852751 ] 

Hong Tang commented on HADOOP-4196:
-----------------------------------

Xiao, you may create a sub-task from this jira that deals with the first optimization.

The remaining optimizations are less obvious, and it would be useful to do some profiling after the first optimization is done.

> Possible performance enhancement in Hadoop compress module
> ----------------------------------------------------------
>
>                 Key: HADOOP-4196
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4196
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.18.0
>            Reporter: Hong Tang
>
> There are several less performant implementation issues with the current Hadoop compression module. Generally, the opportunities all come from the fact that the granuarities of I/O operations from the CompressionStream and DecompressionStream are not controllable by the users, and thus users are forced to attach BufferedInputStream or BufferedOutputStream to both ends of the CompressionStream and DecompressionStream:
> - ZlibCompressor: always returns false from needInput() after setInput(), and thus lead to a native call deflateBytesDirect() for almost every write() operation from CompressorStream(). This becomes problematic when applications call write() on the CompressorStream with small write sizes (e.g. one byte at a time). It is better to follow similar code path in LzoCompressor and append to internal uncompressed data buffer.
> - CompressorStream: whenever the compressor produces some compressed data, it will directly issue write() calls to the down stream. Could be improved by keep appending to the byte[] until it is full (or half full) before writing to the down stream. Otherwise, applications have to use a BufferedOutputStream as the down stream in case the output sizes from CompressorStream is too small. This generally causes double buffering.
> - BlockCompressorStream: similar issue as described above.
> - BlockDecompressorStream: getCompressedData() reads only one compressed chunk at a time. Could be better to read a full buffer, and then obtain compressed chunk from buffer (similar to DecompressStream is doing, but admittedly a bit more complicated).
> In generally, the following could be some guideline of Compressor/Decompressor and CompressorStream/DecompressorStream design/implementation that can give users some performance guarantee:
> - Compressor and Decompressor keep two DirectByteBuffer, the size of which should be tuned to be optimal with regard to the specific compression/decompression algorithm. Ensure always call Compressor.compress() will a full (or near full) uncompressed data DirectBuffer.
> - CompressorStream and DecompressorStream maintains a byte[] to read data from the down stream. The size of the byte[] should be user customizable (add a bufferSize parameter to CompressionCodec's createInputStream and createOutputStream interface). Ensure that I/O from the down stream at or near the granularity of the size of the byte[]. So applications can simply rely on the buffering inside CompressorStream and DecompressorStream (for the case of LZO: BlockCompressorStream and BlockDecompressorStream).
> A more radical change would be to let the downward InputStream to directly deposit data to a ByteBuffer or the downard OutputStream to accept input data from ByteBuffer. We may call it ByteBufferInputStream and ByteBufferOutputStream. The CompressorStream and DecompressorStream may simply test whether the down stream indeed implements such interfaces and bypass its own byte[] buffer if true.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7136-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 02 14:06:48 2010
Return-Path: <common-issues-return-7136-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41283 invoked from network); 2 Apr 2010 14:06:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Apr 2010 14:06:48 -0000
Received: (qmail 38852 invoked by uid 500); 2 Apr 2010 14:06:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38830 invoked by uid 500); 2 Apr 2010 14:06:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38820 invoked by uid 99); 2 Apr 2010 14:06:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 14:06:48 +0000
X-ASF-Spam-Status: No, hits=-1204.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 14:06:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 60763234C4C6
	for <common-issues@hadoop.apache.org>; Fri,  2 Apr 2010 14:06:27 +0000 (UTC)
Message-ID: <1586542796.659711270217187394.JavaMail.jira@brutus.apache.org>
Date: Fri, 2 Apr 2010 14:06:27 +0000 (UTC)
From: "Xiao Kang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6672) BytesWritable.write(buf) use much
 more CPU in writeInt() then write(buf)
In-Reply-To: <603433340.633261270123587176.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xiao Kang updated HADOOP-6672:
------------------------------

    Attachment: screenshot-2.jpg

screenshot-2 after aplying the patch

> BytesWritable.write(buf) use much more CPU in writeInt() then write(buf)
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6672
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6672
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Xiao Kang
>         Attachments: BytesWritable.java.patch, screenshot-1.jpg, screenshot-2.jpg
>
>
> BytesWritable.write() use nearly 4 times of CPU in wirteInt() as write buffer. It may be optimized.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7134-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 02 14:08:52 2010
Return-Path: <common-issues-return-7134-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43180 invoked from network); 2 Apr 2010 14:08:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Apr 2010 14:08:52 -0000
Received: (qmail 64500 invoked by uid 500); 2 Apr 2010 11:18:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63665 invoked by uid 500); 2 Apr 2010 11:18:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63649 invoked by uid 99); 2 Apr 2010 11:18:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 11:18:49 +0000
X-ASF-Spam-Status: No, hits=-1203.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 11:18:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 80B6F234C4CB
	for <common-issues@hadoop.apache.org>; Fri,  2 Apr 2010 11:18:27 +0000 (UTC)
Message-ID: <1983161889.656501270207107526.JavaMail.jira@brutus.apache.org>
Date: Fri, 2 Apr 2010 11:18:27 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6640) FileSystem.get() does RPC retries
 within a static synchronized block
In-Reply-To: <1353980096.327641268867247264.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852789#action_12852789 ] 

Hudson commented on HADOOP-6640:
--------------------------------

Integrated in Hadoop-Common-trunk #294 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/294/])
    . FileSystem.get() does RPC retries within a static synchronized block. Contributed by Hairong Kuang.


> FileSystem.get() does RPC retries within a static synchronized block
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6640
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6640
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>         Environment: all
>            Reporter: Alejandro Abdelnur
>            Assignee: Hairong Kuang
>            Priority: Critical
>             Fix For: 0.22.0
>
>         Attachments: getFS.patch
>
>
> If using FileSystem.get() in a multithreaded environment, and one get() locks because the NN URI is too slow or not responding and retries are in progress, all other get() (for the diffferent users, NN) are blocked.
> the synchronized block in in the static instance of Cache inner class.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7137-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 02 14:14:49 2010
Return-Path: <common-issues-return-7137-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48493 invoked from network); 2 Apr 2010 14:14:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Apr 2010 14:14:48 -0000
Received: (qmail 49474 invoked by uid 500); 2 Apr 2010 14:14:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49427 invoked by uid 500); 2 Apr 2010 14:14:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49419 invoked by uid 99); 2 Apr 2010 14:14:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 14:14:48 +0000
X-ASF-Spam-Status: No, hits=-1204.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 14:14:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 837A9234C4C6
	for <common-issues@hadoop.apache.org>; Fri,  2 Apr 2010 14:14:27 +0000 (UTC)
Message-ID: <857731477.659991270217667537.JavaMail.jira@brutus.apache.org>
Date: Fri, 2 Apr 2010 14:14:27 +0000 (UTC)
From: "Xiao Kang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6672) BytesWritable.write(buf) use much
 more CPU in writeInt() then write(buf)
In-Reply-To: <603433340.633261270123587176.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852829#action_12852829 ] 

Xiao Kang commented on HADOOP-6672:
-----------------------------------

Compring two screenshots as following:

1. In screenshot-2 BytesWritable.write() spends more time on DataOutputStream.write()
2. In screenshot-1 BytesWritable.write() invoked 308k times in 10,780ms, whereas in screenshot-2 invoked 804k times in 11,010ms. So the time of a single invocation of BytesWritable.write() is reduced nearly 60%.

> BytesWritable.write(buf) use much more CPU in writeInt() then write(buf)
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6672
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6672
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Xiao Kang
>         Attachments: BytesWritable.java.patch, screenshot-1.jpg, screenshot-2.jpg
>
>
> BytesWritable.write() use nearly 4 times of CPU in wirteInt() as write buffer. It may be optimized.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7138-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 02 14:26:48 2010
Return-Path: <common-issues-return-7138-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56787 invoked from network); 2 Apr 2010 14:26:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Apr 2010 14:26:48 -0000
Received: (qmail 66840 invoked by uid 500); 2 Apr 2010 14:26:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66807 invoked by uid 500); 2 Apr 2010 14:26:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66799 invoked by uid 99); 2 Apr 2010 14:26:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 14:26:48 +0000
X-ASF-Spam-Status: No, hits=-1204.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 14:26:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 37C39234C4C5
	for <common-issues@hadoop.apache.org>; Fri,  2 Apr 2010 14:26:27 +0000 (UTC)
Message-ID: <977451137.660411270218387227.JavaMail.jira@brutus.apache.org>
Date: Fri, 2 Apr 2010 14:26:27 +0000 (UTC)
From: "Xiao Kang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6672) BytesWritable.write(buf) use much
 more CPU in writeInt() then write(buf)
In-Reply-To: <603433340.633261270123587176.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852833#action_12852833 ] 

Xiao Kang commented on HADOOP-6672:
-----------------------------------

Sun JDK's implementation of writeInt() writeLong() is not the same in class DataOutputStream. writeInt() wirtes one byte four times whereas wirteLong() writes a eight bytes buffer only one time. The underlying org.apache.hadoop.mapred.MapTask$MapOutputBuffer.write() spends too much time to get lock. So it's much efficient to implement  writeInt() as writeLong() to decrease the invocation counts of MapOutputBuffer.write().

    public final void writeInt(int v) throws IOException {
        out.write((v >>> 24) & 0xFF);
        out.write((v >>> 16) & 0xFF);
        out.write((v >>>  8) & 0xFF);
        out.write((v >>>  0) & 0xFF);
        incCount(4);
    }

    private byte writeBuffer[] = new byte[8];

    public final void writeLong(long v) throws IOException {
        writeBuffer[0] = (byte)(v >>> 56);
        writeBuffer[1] = (byte)(v >>> 48);
        writeBuffer[2] = (byte)(v >>> 40);
        writeBuffer[3] = (byte)(v >>> 32);
        writeBuffer[4] = (byte)(v >>> 24);
        writeBuffer[5] = (byte)(v >>> 16);
        writeBuffer[6] = (byte)(v >>>  8);
        writeBuffer[7] = (byte)(v >>>  0);
        out.write(writeBuffer, 0, 8);
	incCount(8);
    }

> BytesWritable.write(buf) use much more CPU in writeInt() then write(buf)
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6672
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6672
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Xiao Kang
>         Attachments: BytesWritable.java.patch, screenshot-1.jpg, screenshot-2.jpg
>
>
> BytesWritable.write() use nearly 4 times of CPU in wirteInt() as write buffer. It may be optimized.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7132-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 02 14:30:50 2010
Return-Path: <common-issues-return-7132-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58819 invoked from network); 2 Apr 2010 14:30:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Apr 2010 14:30:50 -0000
Received: (qmail 96813 invoked by uid 500); 2 Apr 2010 05:30:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96788 invoked by uid 500); 2 Apr 2010 05:30:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96763 invoked by uid 99); 2 Apr 2010 05:30:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 05:30:49 +0000
X-ASF-Spam-Status: No, hits=-1202.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 05:30:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DD6A7234C4D5
	for <common-issues@hadoop.apache.org>; Fri,  2 Apr 2010 05:30:27 +0000 (UTC)
Message-ID: <472358324.652881270186227906.JavaMail.jira@brutus.apache.org>
Date: Fri, 2 Apr 2010 05:30:27 +0000 (UTC)
From: "Xiao Kang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4196) Possible performance enhancement in
 Hadoop compress module
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-4196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852715#action_12852715 ] 

Xiao Kang commented on HADOOP-4196:
-----------------------------------

Thanks Hong Tang for noticing duplication of another jira HADOOP-6662. 

Since the first performance enhancement suggestion is clear and easy to implement, maybe we can resolve it seperately. Close HADOOP-6662 and move the patch to this or dicuss in HADOOP-6662.

> Possible performance enhancement in Hadoop compress module
> ----------------------------------------------------------
>
>                 Key: HADOOP-4196
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4196
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.18.0
>            Reporter: Hong Tang
>
> There are several less performant implementation issues with the current Hadoop compression module. Generally, the opportunities all come from the fact that the granuarities of I/O operations from the CompressionStream and DecompressionStream are not controllable by the users, and thus users are forced to attach BufferedInputStream or BufferedOutputStream to both ends of the CompressionStream and DecompressionStream:
> - ZlibCompressor: always returns false from needInput() after setInput(), and thus lead to a native call deflateBytesDirect() for almost every write() operation from CompressorStream(). This becomes problematic when applications call write() on the CompressorStream with small write sizes (e.g. one byte at a time). It is better to follow similar code path in LzoCompressor and append to internal uncompressed data buffer.
> - CompressorStream: whenever the compressor produces some compressed data, it will directly issue write() calls to the down stream. Could be improved by keep appending to the byte[] until it is full (or half full) before writing to the down stream. Otherwise, applications have to use a BufferedOutputStream as the down stream in case the output sizes from CompressorStream is too small. This generally causes double buffering.
> - BlockCompressorStream: similar issue as described above.
> - BlockDecompressorStream: getCompressedData() reads only one compressed chunk at a time. Could be better to read a full buffer, and then obtain compressed chunk from buffer (similar to DecompressStream is doing, but admittedly a bit more complicated).
> In generally, the following could be some guideline of Compressor/Decompressor and CompressorStream/DecompressorStream design/implementation that can give users some performance guarantee:
> - Compressor and Decompressor keep two DirectByteBuffer, the size of which should be tuned to be optimal with regard to the specific compression/decompression algorithm. Ensure always call Compressor.compress() will a full (or near full) uncompressed data DirectBuffer.
> - CompressorStream and DecompressorStream maintains a byte[] to read data from the down stream. The size of the byte[] should be user customizable (add a bufferSize parameter to CompressionCodec's createInputStream and createOutputStream interface). Ensure that I/O from the down stream at or near the granularity of the size of the byte[]. So applications can simply rely on the buffering inside CompressorStream and DecompressorStream (for the case of LZO: BlockCompressorStream and BlockDecompressorStream).
> A more radical change would be to let the downward InputStream to directly deposit data to a ByteBuffer or the downard OutputStream to accept input data from ByteBuffer. We may call it ByteBufferInputStream and ByteBufferOutputStream. The CompressorStream and DecompressorStream may simply test whether the down stream indeed implements such interfaces and bypass its own byte[] buffer if true.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7139-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 02 16:52:48 2010
Return-Path: <common-issues-return-7139-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3980 invoked from network); 2 Apr 2010 16:52:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Apr 2010 16:52:48 -0000
Received: (qmail 34308 invoked by uid 500); 2 Apr 2010 16:52:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34280 invoked by uid 500); 2 Apr 2010 16:52:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34272 invoked by uid 99); 2 Apr 2010 16:52:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 16:52:48 +0000
X-ASF-Spam-Status: No, hits=-1205.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 16:52:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 68726234C4BE
	for <common-issues@hadoop.apache.org>; Fri,  2 Apr 2010 16:52:27 +0000 (UTC)
Message-ID: <108314726.661851270227147426.JavaMail.jira@brutus.apache.org>
Date: Fri, 2 Apr 2010 16:52:27 +0000 (UTC)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852865#action_12852865 ] 

Doug Cutting commented on HADOOP-6668:
--------------------------------------

> I marked all of the serializer classes as public evolving in the patch.

Great!

It would be good if the evolving status was visible in javadoc, similar to deprecation.  Should I file an issue for that?  I'd be happy to help out with it, if you like.

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7140-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 02 17:26:49 2010
Return-Path: <common-issues-return-7140-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11504 invoked from network); 2 Apr 2010 17:26:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Apr 2010 17:26:49 -0000
Received: (qmail 71388 invoked by uid 500); 2 Apr 2010 17:26:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71352 invoked by uid 500); 2 Apr 2010 17:26:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71343 invoked by uid 99); 2 Apr 2010 17:26:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 17:26:49 +0000
X-ASF-Spam-Status: No, hits=-1205.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 17:26:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id ABB69234C4C9
	for <common-issues@hadoop.apache.org>; Fri,  2 Apr 2010 17:26:27 +0000 (UTC)
Message-ID: <1196091803.662411270229187702.JavaMail.jira@brutus.apache.org>
Date: Fri, 2 Apr 2010 17:26:27 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852876#action_12852876 ] 

Tom White commented on HADOOP-6668:
-----------------------------------

The annotations do appear in javadoc, see http://people.apache.org/~tomwhite/HADOOP-6668/docs/api/org/apache/hadoop/io/serializer/SerializationBase.html, for example.

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7141-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 02 17:36:48 2010
Return-Path: <common-issues-return-7141-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13467 invoked from network); 2 Apr 2010 17:36:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Apr 2010 17:36:48 -0000
Received: (qmail 86514 invoked by uid 500); 2 Apr 2010 17:36:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86379 invoked by uid 500); 2 Apr 2010 17:36:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86363 invoked by uid 99); 2 Apr 2010 17:36:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 17:36:48 +0000
X-ASF-Spam-Status: No, hits=-1206.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 17:36:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 41834234C4C7
	for <common-issues@hadoop.apache.org>; Fri,  2 Apr 2010 17:36:27 +0000 (UTC)
Message-ID: <2052319870.662641270229787267.JavaMail.jira@brutus.apache.org>
Date: Fri, 2 Apr 2010 17:36:27 +0000 (UTC)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852883#action_12852883 ] 

Doug Cutting commented on HADOOP-6668:
--------------------------------------

> The annotations do appear in javadoc [ ... ]

I think they should be more prominent, like deprecation.  A bolded "Evolving" as the first word of the javadoc comment, that appears on the package summary page, like "Deprecated" would be ideal.  We want folks to be very aware that these APIs may change, so that, when they do, they won't scream at us for breaking things.

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7142-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 02 17:58:48 2010
Return-Path: <common-issues-return-7142-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17717 invoked from network); 2 Apr 2010 17:58:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Apr 2010 17:58:48 -0000
Received: (qmail 11943 invoked by uid 500); 2 Apr 2010 17:58:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11906 invoked by uid 500); 2 Apr 2010 17:58:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11898 invoked by uid 99); 2 Apr 2010 17:58:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 17:58:48 +0000
X-ASF-Spam-Status: No, hits=-1206.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 17:58:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 8BBBD234C4C5
	for <common-issues@hadoop.apache.org>; Fri,  2 Apr 2010 17:58:27 +0000 (UTC)
Message-ID: <1909424524.663221270231107571.JavaMail.jira@brutus.apache.org>
Date: Fri, 2 Apr 2010 17:58:27 +0000 (UTC)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6672) BytesWritable.write(buf) use much
 more CPU in writeInt() then write(buf)
In-Reply-To: <603433340.633261270123587176.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852892#action_12852892 ] 

Hong Tang commented on HADOOP-6672:
-----------------------------------

Which version of hadoop did you use when you obtain the profiling data? In trunk, MapOUtputBuffer.Buffer.write(byte[], int, int) is not synchronized. 

> BytesWritable.write(buf) use much more CPU in writeInt() then write(buf)
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6672
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6672
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Xiao Kang
>         Attachments: BytesWritable.java.patch, screenshot-1.jpg, screenshot-2.jpg
>
>
> BytesWritable.write() use nearly 4 times of CPU in wirteInt() as write buffer. It may be optimized.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7143-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 02 18:00:51 2010
Return-Path: <common-issues-return-7143-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18415 invoked from network); 2 Apr 2010 18:00:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Apr 2010 18:00:51 -0000
Received: (qmail 16110 invoked by uid 500); 2 Apr 2010 18:00:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16008 invoked by uid 500); 2 Apr 2010 18:00:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16000 invoked by uid 99); 2 Apr 2010 18:00:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 18:00:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 18:00:49 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1C2A1234C1EF
	for <common-issues@hadoop.apache.org>; Fri,  2 Apr 2010 18:00:28 +0000 (UTC)
Message-ID: <358839696.663261270231228114.JavaMail.jira@brutus.apache.org>
Date: Fri, 2 Apr 2010 18:00:28 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6675) Highlight Evolving elements in
 Javadoc
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Highlight Evolving elements in Javadoc
--------------------------------------

                 Key: HADOOP-6675
                 URL: https://issues.apache.org/jira/browse/HADOOP-6675
             Project: Hadoop Common
          Issue Type: Improvement
          Components: documentation
            Reporter: Tom White


It would be nice if we could mark Evolving (and Unstable) program elements in bold as the first word of the Javadoc comment (like "Deprecated").


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7144-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 02 18:02:51 2010
Return-Path: <common-issues-return-7144-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18822 invoked from network); 2 Apr 2010 18:02:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Apr 2010 18:02:50 -0000
Received: (qmail 18096 invoked by uid 500); 2 Apr 2010 18:02:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18055 invoked by uid 500); 2 Apr 2010 18:02:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18047 invoked by uid 99); 2 Apr 2010 18:02:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 18:02:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 18:02:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3C688234C4BE
	for <common-issues@hadoop.apache.org>; Fri,  2 Apr 2010 18:02:27 +0000 (UTC)
Message-ID: <1959781876.663341270231347246.JavaMail.jira@brutus.apache.org>
Date: Fri, 2 Apr 2010 18:02:27 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852893#action_12852893 ] 

Tom White commented on HADOOP-6668:
-----------------------------------

This is harder to achieve. The standard doclet is not every extensible, so we'd probably have to copy then modify it; see http://java.sun.com/j2se/javadoc/faq/index.html#sourcedoclet. I've created HADOOP-6675 to track this.

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7145-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 02 18:08:50 2010
Return-Path: <common-issues-return-7145-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20360 invoked from network); 2 Apr 2010 18:08:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Apr 2010 18:08:50 -0000
Received: (qmail 25955 invoked by uid 500); 2 Apr 2010 18:08:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25918 invoked by uid 500); 2 Apr 2010 18:08:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25910 invoked by uid 99); 2 Apr 2010 18:08:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 18:08:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 18:08:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 2CD8A234C1EF
	for <common-issues@hadoop.apache.org>; Fri,  2 Apr 2010 18:08:27 +0000 (UTC)
Message-ID: <1601380660.663491270231707182.JavaMail.jira@brutus.apache.org>
Date: Fri, 2 Apr 2010 18:08:27 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6676) Create a javac plugin that produces
 warnings for programs using Evolving APIs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Create a javac plugin that produces warnings for programs using Evolving APIs
-----------------------------------------------------------------------------

                 Key: HADOOP-6676
                 URL: https://issues.apache.org/jira/browse/HADOOP-6676
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Tom White


Such a feature would be useful to enforce MAPREDUCE-1637. It might also be useful for users who wish to check their programs don't use Evolving APIs. 

It could be implemented using an annotation processor: http://java.sun.com/javase/6/docs/technotes/tools/solaris/javac.html#processing

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7146-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 02 18:26:51 2010
Return-Path: <common-issues-return-7146-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30096 invoked from network); 2 Apr 2010 18:26:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Apr 2010 18:26:51 -0000
Received: (qmail 44347 invoked by uid 500); 2 Apr 2010 18:26:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44321 invoked by uid 500); 2 Apr 2010 18:26:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44313 invoked by uid 99); 2 Apr 2010 18:26:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 18:26:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 18:26:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 5051D234C4D8
	for <common-issues@hadoop.apache.org>; Fri,  2 Apr 2010 18:26:27 +0000 (UTC)
Message-ID: <1125956429.663861270232787328.JavaMail.jira@brutus.apache.org>
Date: Fri, 2 Apr 2010 18:26:27 +0000 (UTC)
From: "Patrick Angeles (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6382) publish hadoop jars to apache mvn
 repo.
In-Reply-To: <1682027337.1258523559848.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852900#action_12852900 ] 

Patrick Angeles commented on HADOOP-6382:
-----------------------------------------

I'm curious if anyone's managed to deploy to the apache repo using HTTP.

It says here that the wagon-http provider doesn't support deployment:

http://maven.apache.org/wagon/wagon-providers/wagon-http/

> publish hadoop jars to apache mvn repo.
> ---------------------------------------
>
>                 Key: HADOOP-6382
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6382
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>    Affects Versions: 0.20.0
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>             Fix For: 0.20.3
>
>         Attachments: h20-v3tov5.patch, HADOOP-6382-findbugs.patch, HADOOP-6382-findbugs.patch, hadoop-6382-v1.patch, hadoop-6382-v2.patch, hadoop-6382-v3.patch, hadoop-6382-v4.patch, hadoop-6382-v5.patch, hadoop-6382.patch, HADOOP-6382.script.patch
>
>
> use maven ant task to publish hadoop 20 jars to the apache maven repo

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7147-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 02 19:54:49 2010
Return-Path: <common-issues-return-7147-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49198 invoked from network); 2 Apr 2010 19:54:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Apr 2010 19:54:49 -0000
Received: (qmail 39686 invoked by uid 500); 2 Apr 2010 19:54:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39656 invoked by uid 500); 2 Apr 2010 19:54:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39648 invoked by uid 99); 2 Apr 2010 19:54:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 19:54:48 +0000
X-ASF-Spam-Status: No, hits=-1207.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 19:54:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CD98B234C4D5
	for <common-issues@hadoop.apache.org>; Fri,  2 Apr 2010 19:54:27 +0000 (UTC)
Message-ID: <1283784957.665271270238067841.JavaMail.jira@brutus.apache.org>
Date: Fri, 2 Apr 2010 19:54:27 +0000 (UTC)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6382) publish hadoop jars to apache mvn
 repo.
In-Reply-To: <1682027337.1258523559848.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852935#action_12852935 ] 

Giridharan Kesavan commented on HADOOP-6382:
--------------------------------------------

wagon plugin supports both http and https but we can only publish to the apache nexus repo using https.

> publish hadoop jars to apache mvn repo.
> ---------------------------------------
>
>                 Key: HADOOP-6382
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6382
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>    Affects Versions: 0.20.0
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>             Fix For: 0.20.3
>
>         Attachments: h20-v3tov5.patch, HADOOP-6382-findbugs.patch, HADOOP-6382-findbugs.patch, hadoop-6382-v1.patch, hadoop-6382-v2.patch, hadoop-6382-v3.patch, hadoop-6382-v4.patch, hadoop-6382-v5.patch, hadoop-6382.patch, HADOOP-6382.script.patch
>
>
> use maven ant task to publish hadoop 20 jars to the apache maven repo

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7148-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 02 20:25:48 2010
Return-Path: <common-issues-return-7148-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53685 invoked from network); 2 Apr 2010 20:25:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Apr 2010 20:25:48 -0000
Received: (qmail 59280 invoked by uid 500); 2 Apr 2010 20:25:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59244 invoked by uid 500); 2 Apr 2010 20:25:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59224 invoked by uid 99); 2 Apr 2010 20:25:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 20:25:48 +0000
X-ASF-Spam-Status: No, hits=-1207.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 20:25:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 42D63234C1EF
	for <common-issues@hadoop.apache.org>; Fri,  2 Apr 2010 20:25:27 +0000 (UTC)
Message-ID: <680532766.665461270239927272.JavaMail.jira@brutus.apache.org>
Date: Fri, 2 Apr 2010 20:25:27 +0000 (UTC)
From: "Alan Gates (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6677) InterfaceAudience.LimitedPrivate
 should take a string not an enum
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

InterfaceAudience.LimitedPrivate should take a string not an enum
-----------------------------------------------------------------

                 Key: HADOOP-6677
                 URL: https://issues.apache.org/jira/browse/HADOOP-6677
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.21.0
            Reporter: Alan Gates
            Priority: Minor


Trying to keep the list of all possible components sharing limited private interfaces up to date is painful.  As other subprojects beyond HDFS and MR use this interface it will only get worse (see PIG-1311).  If it is converted to a string other subprojects can use it without requiring patches to common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7149-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 02 20:27:50 2010
Return-Path: <common-issues-return-7149-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53987 invoked from network); 2 Apr 2010 20:27:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Apr 2010 20:27:50 -0000
Received: (qmail 61225 invoked by uid 500); 2 Apr 2010 20:27:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61140 invoked by uid 500); 2 Apr 2010 20:27:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61132 invoked by uid 99); 2 Apr 2010 20:27:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 20:27:50 +0000
X-ASF-Spam-Status: No, hits=-1207.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Apr 2010 20:27:49 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id AA3A1234C4CB
	for <common-issues@hadoop.apache.org>; Fri,  2 Apr 2010 20:27:28 +0000 (UTC)
Message-ID: <945441826.665601270240048696.JavaMail.jira@brutus.apache.org>
Date: Fri, 2 Apr 2010 20:27:28 +0000 (UTC)
From: "E. Sammer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6671) To use maven for hadoop common
 builds
In-Reply-To: <859246128.630391270120227191.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852939#action_12852939 ] 

E. Sammer commented on HADOOP-6671:
-----------------------------------

So much of the build system today "acts as" maven it might as well be maven. Having some consistency in the way the individual projects are built would be extremely nice. Maven is as painful as every other build tool we've seen but at least its pain comes in a consistent, known, quantity. The tooling support tends to be better due to the predictable nature of the the build process. The surrounding infrastructure (such as Hudson) is already mvn aware. The main problem with mvn tends to be errant dependencies in poms but this can be avoided (see the SpringSource Ivy / Maven repositories for an example of *very* well maintained metadata - http://www.springsource.com/repository/app/). +1

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7150-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 03 00:14:51 2010
Return-Path: <common-issues-return-7150-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94794 invoked from network); 3 Apr 2010 00:14:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Apr 2010 00:14:51 -0000
Received: (qmail 71223 invoked by uid 500); 3 Apr 2010 00:14:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71186 invoked by uid 500); 3 Apr 2010 00:14:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71178 invoked by uid 99); 3 Apr 2010 00:14:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Apr 2010 00:14:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Apr 2010 00:14:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id AE573234C1EF
	for <common-issues@hadoop.apache.org>; Sat,  3 Apr 2010 00:14:27 +0000 (UTC)
Message-ID: <360625036.668801270253667713.JavaMail.jira@brutus.apache.org>
Date: Sat, 3 Apr 2010 00:14:27 +0000 (UTC)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6678) Propose some changes to FileContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Propose some changes to FileContext
-----------------------------------

                 Key: HADOOP-6678
                 URL: https://issues.apache.org/jira/browse/HADOOP-6678
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs
            Reporter: Hairong Kuang
             Fix For: 0.21.0, 0.22.0


# Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.
# Remove methods isFile(Path), isDirectory(Path), and exists.
All these methods are implemented by calling getFileStatus(Path).But most users are not aware of this. They would write code as below: 
{code}
  FileContext fc = ..;
  if (fc.exists(path)) {
    if (fc.isFile(path)) {
     ...
    } else {
    ...
    }
  }
{code}
The above code adds unnecessary getFileInfo RPC to NameNode. In our production clusters, we often see that the number of getFileStatus calls is multiple times of the open calls. If we remove isFile, isDirectory, and exists from FileContext, users have to explicitly call getFileStatus first, it is more likely that they will write more efficient code as follow:
{code}
  FileContext fc = ...;
  FileStatus fstatus = fc.getFileStatus(path);
  if (fstatus.isFile() {
    ...
  } else {
    ...
  }
{code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7151-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 03 00:20:49 2010
Return-Path: <common-issues-return-7151-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96127 invoked from network); 3 Apr 2010 00:20:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Apr 2010 00:20:49 -0000
Received: (qmail 78544 invoked by uid 500); 3 Apr 2010 00:20:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78510 invoked by uid 500); 3 Apr 2010 00:20:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78502 invoked by uid 99); 3 Apr 2010 00:20:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Apr 2010 00:20:48 +0000
X-ASF-Spam-Status: No, hits=-1209.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Apr 2010 00:20:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id AE45F234C48C
	for <common-issues@hadoop.apache.org>; Sat,  3 Apr 2010 00:20:27 +0000 (UTC)
Message-ID: <1285385225.668931270254027712.JavaMail.jira@brutus.apache.org>
Date: Sat, 3 Apr 2010 00:20:27 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6678) Propose some changes to FileContext
In-Reply-To: <360625036.668801270253667713.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853019#action_12853019 ] 

Eli Collins commented on HADOOP-6678:
-------------------------------------

+1

HADOOP-6585 (Add FileStatus#isDirectory and isFile)

(Replace usage of FileStatus#isDir())

> Propose some changes to FileContext
> -----------------------------------
>
>                 Key: HADOOP-6678
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6678
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Hairong Kuang
>             Fix For: 0.21.0, 0.22.0
>
>
> # Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.
> # Remove methods isFile(Path), isDirectory(Path), and exists.
> All these methods are implemented by calling getFileStatus(Path).But most users are not aware of this. They would write code as below: 
> {code}
>   FileContext fc = ..;
>   if (fc.exists(path)) {
>     if (fc.isFile(path)) {
>      ...
>     } else {
>     ...
>     }
>   }
> {code}
> The above code adds unnecessary getFileInfo RPC to NameNode. In our production clusters, we often see that the number of getFileStatus calls is multiple times of the open calls. If we remove isFile, isDirectory, and exists from FileContext, users have to explicitly call getFileStatus first, it is more likely that they will write more efficient code as follow:
> {code}
>   FileContext fc = ...;
>   FileStatus fstatus = fc.getFileStatus(path);
>   if (fstatus.isFile() {
>     ...
>   } else {
>     ...
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7152-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 03 00:22:48 2010
Return-Path: <common-issues-return-7152-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96336 invoked from network); 3 Apr 2010 00:22:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Apr 2010 00:22:48 -0000
Received: (qmail 78932 invoked by uid 500); 3 Apr 2010 00:22:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78897 invoked by uid 500); 3 Apr 2010 00:22:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78888 invoked by uid 99); 3 Apr 2010 00:22:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Apr 2010 00:22:48 +0000
X-ASF-Spam-Status: No, hits=-1209.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Apr 2010 00:22:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 7B05F234C4BE
	for <common-issues@hadoop.apache.org>; Sat,  3 Apr 2010 00:22:27 +0000 (UTC)
Message-ID: <275660690.669001270254147502.JavaMail.jira@brutus.apache.org>
Date: Sat, 3 Apr 2010 00:22:27 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6678) Propose some changes to FileContext
In-Reply-To: <360625036.668801270253667713.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853022#action_12853022 ] 

Eli Collins commented on HADOOP-6678:
-------------------------------------

Accidentally submitted the last comment..

+1 

I cold do part 2 in lieu of HADOOP-6585 (Add FileStatus#isDirectory and isFile) and HDFS-995, MR-1535 (Replace usage of FileStatus#isDir()) which I just started working on today.

Thanks,
Eli




> Propose some changes to FileContext
> -----------------------------------
>
>                 Key: HADOOP-6678
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6678
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Hairong Kuang
>             Fix For: 0.21.0, 0.22.0
>
>
> # Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.
> # Remove methods isFile(Path), isDirectory(Path), and exists.
> All these methods are implemented by calling getFileStatus(Path).But most users are not aware of this. They would write code as below: 
> {code}
>   FileContext fc = ..;
>   if (fc.exists(path)) {
>     if (fc.isFile(path)) {
>      ...
>     } else {
>     ...
>     }
>   }
> {code}
> The above code adds unnecessary getFileInfo RPC to NameNode. In our production clusters, we often see that the number of getFileStatus calls is multiple times of the open calls. If we remove isFile, isDirectory, and exists from FileContext, users have to explicitly call getFileStatus first, it is more likely that they will write more efficient code as follow:
> {code}
>   FileContext fc = ...;
>   FileStatus fstatus = fc.getFileStatus(path);
>   if (fstatus.isFile() {
>     ...
>   } else {
>     ...
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7153-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 03 00:24:51 2010
Return-Path: <common-issues-return-7153-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97017 invoked from network); 3 Apr 2010 00:24:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Apr 2010 00:24:51 -0000
Received: (qmail 82047 invoked by uid 500); 3 Apr 2010 00:24:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82018 invoked by uid 500); 3 Apr 2010 00:24:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82010 invoked by uid 99); 3 Apr 2010 00:24:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Apr 2010 00:24:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Apr 2010 00:24:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C30BD234C4C5
	for <common-issues@hadoop.apache.org>; Sat,  3 Apr 2010 00:24:27 +0000 (UTC)
Message-ID: <1854111176.669061270254267798.JavaMail.jira@brutus.apache.org>
Date: Sat, 3 Apr 2010 00:24:27 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6678) Propose some changes to FileContext
In-Reply-To: <360625036.668801270253667713.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853024#action_12853024 ] 

Todd Lipcon commented on HADOOP-6678:
-------------------------------------

The other option for #2 is a stat cache, no? Of course we're trading off one set of problems for another, but I think for a lot of use cases a 3 second stat cache (a la linux nfs default) is totally reasonable, so long as it's easily disabled.

> Propose some changes to FileContext
> -----------------------------------
>
>                 Key: HADOOP-6678
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6678
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Hairong Kuang
>             Fix For: 0.21.0, 0.22.0
>
>
> # Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.
> # Remove methods isFile(Path), isDirectory(Path), and exists.
> All these methods are implemented by calling getFileStatus(Path).But most users are not aware of this. They would write code as below: 
> {code}
>   FileContext fc = ..;
>   if (fc.exists(path)) {
>     if (fc.isFile(path)) {
>      ...
>     } else {
>     ...
>     }
>   }
> {code}
> The above code adds unnecessary getFileInfo RPC to NameNode. In our production clusters, we often see that the number of getFileStatus calls is multiple times of the open calls. If we remove isFile, isDirectory, and exists from FileContext, users have to explicitly call getFileStatus first, it is more likely that they will write more efficient code as follow:
> {code}
>   FileContext fc = ...;
>   FileStatus fstatus = fc.getFileStatus(path);
>   if (fstatus.isFile() {
>     ...
>   } else {
>     ...
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7154-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 03 00:48:48 2010
Return-Path: <common-issues-return-7154-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8899 invoked from network); 3 Apr 2010 00:48:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Apr 2010 00:48:48 -0000
Received: (qmail 95389 invoked by uid 500); 3 Apr 2010 00:48:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95349 invoked by uid 500); 3 Apr 2010 00:48:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95341 invoked by uid 99); 3 Apr 2010 00:48:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Apr 2010 00:48:48 +0000
X-ASF-Spam-Status: No, hits=-1209.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Apr 2010 00:48:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 735C1234C4C5
	for <common-issues@hadoop.apache.org>; Sat,  3 Apr 2010 00:48:27 +0000 (UTC)
Message-ID: <118906139.669551270255707471.JavaMail.jira@brutus.apache.org>
Date: Sat, 3 Apr 2010 00:48:27 +0000 (UTC)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6539) Hadoop 20 Doc Backport
In-Reply-To: <1135704534.52341265323767904.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-6539:
----------------------------------

    Attachment: C6539-2-y20s.patch

> Hadoop 20 Doc Backport
> ----------------------
>
>                 Key: HADOOP-6539
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6539
>             Project: Hadoop Common
>          Issue Type: Task
>    Affects Versions: 0.20.0
>            Reporter: Corinne Chandel
>            Assignee: Corinne Chandel
>         Attachments: C6539-1-y20.patch, C6539-2-y20.patch, C6539-2-y20s.patch, hadoop-6539-2.patch, HADOOP-6539-y20.patch, hadoop-logo-2.gif
>
>
> Hadoop 20 Doc Backport 
> Updates to Hadoop 20 docs (internal).
> After commit, docs should be reviewed.
> (1) Delete these files
> > hdfs_shell.xml
> > hod_admin_guide.xml
> > hod_config_guide.xml
> > hod_user_guide.xml
> > quickstart.xml
> (2) Add these files
> > file_system_shell.xml  (replaces hdfs_shell.xml)
> > hod_scheduler.xml   (replaces/combines the 3 hod guides)
> > single_node_setup.xml   (replaces quickstart.xml)
> (3) Add logo file
> > hadoop-logo-2.gif 
> (4) Apply patch
> > apply to Hadoop 20 (internal) - ALL files include edits (some minor, some major).
> > DO NOT APPLY TO APACHE

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7155-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 03 15:03:52 2010
Return-Path: <common-issues-return-7155-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36520 invoked from network); 3 Apr 2010 15:03:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Apr 2010 15:03:52 -0000
Received: (qmail 49646 invoked by uid 500); 3 Apr 2010 15:03:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49626 invoked by uid 500); 3 Apr 2010 15:03:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49618 invoked by uid 99); 3 Apr 2010 15:03:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Apr 2010 15:03:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Apr 2010 15:03:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 8A81F234C4C6
	for <common-issues@hadoop.apache.org>; Sat,  3 Apr 2010 15:03:27 +0000 (UTC)
Message-ID: <1444792364.674481270307007566.JavaMail.jira@brutus.apache.org>
Date: Sat, 3 Apr 2010 15:03:27 +0000 (UTC)
From: "Lars Francke (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6671) To use maven for hadoop common
 builds
In-Reply-To: <859246128.630391270120227191.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853146#action_12853146 ] 

Lars Francke commented on HADOOP-6671:
--------------------------------------

I don't have a lot of insight into the Hadoop Common project itself but I've done a lot of the work on the recent HBase transition together with Paul Smith and Kay Kay and would like to offer my help if needed.

There are basically two ways to go: Either shuffle around a lot of directories to conform to the standard Maven layout or to override the Maven defaults and keep the current layout. Both ways are actually very painful but I prefer the first one. While that means a lot of work and a lot of swearing it also means that you'll get a consistent layout (with most other Maven projects) and the configuration is a lot easier. Some tools tend(ed) to not work properly with the non standard layout (strictly speaking this are bugs in the tools/plugins). You basically have to do a lot of work up front but after that it shouldn't be too hard to maintain.

The directory moving/renaming unfortunately tends not to work too well with Subversion and branches (or I was doing it wrong) so I don't know how big the benefit would be to start a new one for doing this.

What Paul Smith has done in HBASE-2099 is to provide a script that contained all the necessary commands (svn mv...) to finish the move (there are a couple more changes in other tickets) and a patch containing the .pom files. This has been much improved since and we've learned a lot from it. We are in fact still tweaking it and will change the pom structure once again to rely on the common Apache parent pom in addition to a few smaller fixes that are still outstanding.

I'd propose a similar structure as HBase now has: A pom in the main directory with common information (packaging "pom") and then two modules ("core" and "contrib"). So the src/contrib directory would move to the top-level as would the docs directory (btw: I've never used Forrest with Maven but if Ant can do it Maven should be able to do it too). The final tarball as currently (trunk) created by ant tar should be easily reproducible by Maven (I couldn't find where most of the conribs ended up, are those currently missung from the .tar?)

Paul Smith has indicated interest in this and I'd be interested and willing to help too if needed. But if you've already got someone to do the work I'm not going to complain :) It'd just be a shame to duplicate work here as the moving around of stuff in SVN is painful work if that's the way you decide to go.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7156-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 03 16:23:49 2010
Return-Path: <common-issues-return-7156-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43739 invoked from network); 3 Apr 2010 16:23:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Apr 2010 16:23:49 -0000
Received: (qmail 85927 invoked by uid 500); 3 Apr 2010 16:23:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85870 invoked by uid 500); 3 Apr 2010 16:23:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85862 invoked by uid 99); 3 Apr 2010 16:23:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Apr 2010 16:23:48 +0000
X-ASF-Spam-Status: No, hits=-1212.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Apr 2010 16:23:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3C111234C48D
	for <common-issues@hadoop.apache.org>; Sat,  3 Apr 2010 16:23:27 +0000 (UTC)
Message-ID: <794188356.674691270311807245.JavaMail.jira@brutus.apache.org>
Date: Sat, 3 Apr 2010 16:23:27 +0000 (UTC)
From: "Xiao Kang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6663) BlockDecompressorStream get EOF
 exception when decompressing the file compressed from empty file
In-Reply-To: <1619883340.543171269850767204.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xiao Kang updated HADOOP-6663:
------------------------------

    Attachment: BlockDecompressorStream.java.patch

New patch attached, including test case.

> BlockDecompressorStream get EOF exception when decompressing the file compressed from empty file
> ------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6663
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6663
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Xiao Kang
>         Attachments: BlockDecompressorStream.java.patch, BlockDecompressorStream.patch
>
>
> An empty file can be compressed using BlockDecompressorStream, which is for block-based compressiong algorithm such as LZO. However, when decompressing the compressed file, BlockDecompressorStream get EOF exception.
> Here is a typical exception stack:
> java.io.EOFException
> at org.apache.hadoop.io.compress.BlockDecompressorStream.rawReadInt(BlockDecompressorStream.java:125)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:96)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:82)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at java.io.InputStream.read(InputStream.java:85)
> at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:39)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:186)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:170)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:18)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
> at org.apache.hadoop.mapred.Child.main(Child.java:196)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7157-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 03 16:31:51 2010
Return-Path: <common-issues-return-7157-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44357 invoked from network); 3 Apr 2010 16:31:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Apr 2010 16:31:51 -0000
Received: (qmail 92049 invoked by uid 500); 3 Apr 2010 16:31:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92026 invoked by uid 500); 3 Apr 2010 16:31:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92012 invoked by uid 99); 3 Apr 2010 16:31:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Apr 2010 16:31:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Apr 2010 16:31:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3685E234C4BE
	for <common-issues@hadoop.apache.org>; Sat,  3 Apr 2010 16:31:27 +0000 (UTC)
Message-ID: <404958215.674831270312287222.JavaMail.jira@brutus.apache.org>
Date: Sat, 3 Apr 2010 16:31:27 +0000 (UTC)
From: "Xiao Kang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6672) BytesWritable.write(buf) use much
 more CPU in writeInt() then write(buf)
In-Reply-To: <603433340.633261270123587176.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853155#action_12853155 ] 

Xiao Kang commented on HADOOP-6672:
-----------------------------------

The profiling data is got from a internal version created from hadoop 0.19 trunk.

MapOutputBuffer.Buffer.write(byte[], int, int) is also not a synchronized method in the profiling version. However, it trys to get a ReentrantLock(MapOUtputBuffer.Buffer.write(byte[], int, int)) before witting data to buffer and this is the performance issue.

> BytesWritable.write(buf) use much more CPU in writeInt() then write(buf)
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6672
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6672
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Xiao Kang
>         Attachments: BytesWritable.java.patch, screenshot-1.jpg, screenshot-2.jpg
>
>
> BytesWritable.write() use nearly 4 times of CPU in wirteInt() as write buffer. It may be optimized.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7158-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 03 20:19:52 2010
Return-Path: <common-issues-return-7158-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76100 invoked from network); 3 Apr 2010 20:19:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Apr 2010 20:19:52 -0000
Received: (qmail 19039 invoked by uid 500); 3 Apr 2010 20:19:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19009 invoked by uid 500); 3 Apr 2010 20:19:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19001 invoked by uid 99); 3 Apr 2010 20:19:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Apr 2010 20:19:52 +0000
X-ASF-Spam-Status: No, hits=-1212.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Apr 2010 20:19:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 37A28234C48D
	for <common-issues@hadoop.apache.org>; Sat,  3 Apr 2010 20:19:27 +0000 (UTC)
Message-ID: <1386936572.676501270325967226.JavaMail.jira@brutus.apache.org>
Date: Sat, 3 Apr 2010 20:19:27 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6663) BlockDecompressorStream get EOF
 exception when decompressing the file compressed from empty file
In-Reply-To: <1619883340.543171269850767204.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853192#action_12853192 ] 

Todd Lipcon commented on HADOOP-6663:
-------------------------------------

+1, I've seen this issue in production as well. Fix and test case look good, except please add the apache header to the test case, and preferably update the test case to JUnit 4 style

> BlockDecompressorStream get EOF exception when decompressing the file compressed from empty file
> ------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6663
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6663
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Xiao Kang
>         Attachments: BlockDecompressorStream.java.patch, BlockDecompressorStream.patch
>
>
> An empty file can be compressed using BlockDecompressorStream, which is for block-based compressiong algorithm such as LZO. However, when decompressing the compressed file, BlockDecompressorStream get EOF exception.
> Here is a typical exception stack:
> java.io.EOFException
> at org.apache.hadoop.io.compress.BlockDecompressorStream.rawReadInt(BlockDecompressorStream.java:125)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:96)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:82)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at java.io.InputStream.read(InputStream.java:85)
> at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:39)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:186)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:170)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:18)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
> at org.apache.hadoop.mapred.Child.main(Child.java:196)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7159-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 04 06:40:53 2010
Return-Path: <common-issues-return-7159-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46767 invoked from network); 4 Apr 2010 06:40:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Apr 2010 06:40:52 -0000
Received: (qmail 61927 invoked by uid 500); 4 Apr 2010 06:40:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61360 invoked by uid 500); 4 Apr 2010 06:40:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61087 invoked by uid 99); 4 Apr 2010 06:40:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 04 Apr 2010 06:40:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 04 Apr 2010 06:40:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 361D9234C48C
	for <common-issues@hadoop.apache.org>; Sun,  4 Apr 2010 06:40:27 +0000 (UTC)
Message-ID: <18346259.678881270363227220.JavaMail.jira@brutus.apache.org>
Date: Sun, 4 Apr 2010 06:40:27 +0000 (UTC)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6679) add retry logic to
 NetUtils.createSocketAddr() when hostname resolution fails
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

add retry logic to NetUtils.createSocketAddr() when hostname resolution fails
-----------------------------------------------------------------------------

                 Key: HADOOP-6679
                 URL: https://issues.apache.org/jira/browse/HADOOP-6679
             Project: Hadoop Common
          Issue Type: Improvement
    Affects Versions: 0.20.0
            Reporter: sam rash
            Priority: Minor


transient DNS errors can cause the InetSocketAddress created in NetUtils.createSocketAddr() to be unresolved when the hostname is resolvable.  Add a single retry after a short sleep using a simple InetSocketAddressFactory class

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7160-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 04 06:46:52 2010
Return-Path: <common-issues-return-7160-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47246 invoked from network); 4 Apr 2010 06:46:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Apr 2010 06:46:52 -0000
Received: (qmail 62672 invoked by uid 500); 4 Apr 2010 06:46:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62569 invoked by uid 500); 4 Apr 2010 06:46:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62561 invoked by uid 99); 4 Apr 2010 06:46:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 04 Apr 2010 06:46:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 04 Apr 2010 06:46:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 2FD71234C1F2
	for <common-issues@hadoop.apache.org>; Sun,  4 Apr 2010 06:46:27 +0000 (UTC)
Message-ID: <1008925993.678911270363587194.JavaMail.jira@brutus.apache.org>
Date: Sun, 4 Apr 2010 06:46:27 +0000 (UTC)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6679) add retry logic to
 NetUtils.createSocketAddr() when hostname resolution fails
In-Reply-To: <18346259.678881270363227220.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

sam rash updated HADOOP-6679:
-----------------------------

    Attachment: HADOOP-6679-patch-1.txt

patch includes factory to create a resolved InetSocketAddress and updates NetUtil.createSocketAddr() to use it

includes basic unit test

> add retry logic to NetUtils.createSocketAddr() when hostname resolution fails
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-6679
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6679
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.0
>            Reporter: sam rash
>            Priority: Minor
>         Attachments: HADOOP-6679-patch-1.txt
>
>
> transient DNS errors can cause the InetSocketAddress created in NetUtils.createSocketAddr() to be unresolved when the hostname is resolvable.  Add a single retry after a short sleep using a simple InetSocketAddressFactory class

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7161-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 04 16:53:49 2010
Return-Path: <common-issues-return-7161-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13436 invoked from network); 4 Apr 2010 16:53:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Apr 2010 16:53:49 -0000
Received: (qmail 71323 invoked by uid 500); 4 Apr 2010 16:53:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71272 invoked by uid 500); 4 Apr 2010 16:53:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71264 invoked by uid 99); 4 Apr 2010 16:53:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 04 Apr 2010 16:53:48 +0000
X-ASF-Spam-Status: No, hits=-1214.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 04 Apr 2010 16:53:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3632C234C495
	for <common-issues@hadoop.apache.org>; Sun,  4 Apr 2010 16:53:27 +0000 (UTC)
Message-ID: <1464787205.680851270400007221.JavaMail.jira@brutus.apache.org>
Date: Sun, 4 Apr 2010 16:53:27 +0000 (UTC)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6672) BytesWritable.write(buf) use much
 more CPU in writeInt() then write(buf)
In-Reply-To: <603433340.633261270123587176.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853267#action_12853267 ] 

Hong Tang commented on HADOOP-6672:
-----------------------------------

I see. This sounds like a duplicate of HADOOP-5664. Would you please profile trunk and confirm that the problem persists?

> BytesWritable.write(buf) use much more CPU in writeInt() then write(buf)
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6672
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6672
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Xiao Kang
>         Attachments: BytesWritable.java.patch, screenshot-1.jpg, screenshot-2.jpg
>
>
> BytesWritable.write() use nearly 4 times of CPU in wirteInt() as write buffer. It may be optimized.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7162-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 05 04:21:54 2010
Return-Path: <common-issues-return-7162-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12529 invoked from network); 5 Apr 2010 04:21:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Apr 2010 04:21:54 -0000
Received: (qmail 91136 invoked by uid 500); 5 Apr 2010 04:21:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90642 invoked by uid 500); 5 Apr 2010 04:21:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90627 invoked by uid 99); 5 Apr 2010 04:21:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 04:21:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 04:21:49 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CB5E9234C48C
	for <common-issues@hadoop.apache.org>; Mon,  5 Apr 2010 04:21:27 +0000 (UTC)
Message-ID: <1679478823.682611270441287832.JavaMail.jira@brutus.apache.org>
Date: Mon, 5 Apr 2010 04:21:27 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6677) InterfaceAudience.LimitedPrivate
 should take a string not an enum
In-Reply-To: <680532766.665461270239927272.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853300#action_12853300 ] 

Tom White commented on HADOOP-6677:
-----------------------------------

+1 However, I wonder if we could go further and not list projects at all. It seems odd to mandate a pre-determined audience for a particular API. To me, it seems better to say "this is a developer-level interface" (stability is independent), which may be used by different projects. A Private API, by contrast, is one that may not be used outside the project.

With this simplification, the LimitedPrivate annotation would be interpreted as being intended for developers rather than users of Hadoop. This would allow any project to use the API, including subprojects, contrib projects, or projects that are not a part of Hadoop.

For example, by marking the MapReduce interfaces that schedulers need limited-private the scheduler implementations could be hosted anywhere - as contrib modules as they are today, or as an independent project - it shouldn't matter. Futhermore, by making the Java visibility public, the scheduler implementations would not have to be in the same package as the interfaces, as they are today. Users would not see these interfaces, since they are not in public javadoc.


> InterfaceAudience.LimitedPrivate should take a string not an enum
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6677
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6677
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Alan Gates
>            Priority: Minor
>
> Trying to keep the list of all possible components sharing limited private interfaces up to date is painful.  As other subprojects beyond HDFS and MR use this interface it will only get worse (see PIG-1311).  If it is converted to a string other subprojects can use it without requiring patches to common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7163-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 05 14:43:50 2010
Return-Path: <common-issues-return-7163-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4325 invoked from network); 5 Apr 2010 14:43:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Apr 2010 14:43:50 -0000
Received: (qmail 4765 invoked by uid 500); 5 Apr 2010 14:43:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4154 invoked by uid 500); 5 Apr 2010 14:43:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4143 invoked by uid 99); 5 Apr 2010 14:43:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 14:43:48 +0000
X-ASF-Spam-Status: No, hits=-1216.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 14:43:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4CCAD234C4AE
	for <common-issues@hadoop.apache.org>; Mon,  5 Apr 2010 14:43:27 +0000 (UTC)
Message-ID: <1629624146.687721270478607313.JavaMail.jira@brutus.apache.org>
Date: Mon, 5 Apr 2010 14:43:27 +0000 (UTC)
From: "Andrew Klochkov (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6680) hadoop-cloud push command invokes
 proxy creation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

hadoop-cloud push command invokes proxy creation
------------------------------------------------

                 Key: HADOOP-6680
                 URL: https://issues.apache.org/jira/browse/HADOOP-6680
             Project: Hadoop Common
          Issue Type: Bug
          Components: contrib/cloud
            Reporter: Andrew Klochkov


There's a typo in src/contrib/cloud/src/py/hadoop/cloud/cli.py: when "push" command is invoked, the script actually invokes proxy creation instead of copying files. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7164-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 05 14:47:49 2010
Return-Path: <common-issues-return-7164-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5963 invoked from network); 5 Apr 2010 14:47:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Apr 2010 14:47:49 -0000
Received: (qmail 9432 invoked by uid 500); 5 Apr 2010 14:47:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9404 invoked by uid 500); 5 Apr 2010 14:47:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9396 invoked by uid 99); 5 Apr 2010 14:47:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 14:47:48 +0000
X-ASF-Spam-Status: No, hits=-1217.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 14:47:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 52065234C48D
	for <common-issues@hadoop.apache.org>; Mon,  5 Apr 2010 14:47:27 +0000 (UTC)
Message-ID: <307203174.687841270478847334.JavaMail.jira@brutus.apache.org>
Date: Mon, 5 Apr 2010 14:47:27 +0000 (UTC)
From: "Andrew Klochkov (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6680) hadoop-cloud push command invokes
 proxy creation
In-Reply-To: <1629624146.687721270478607313.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Klochkov updated HADOOP-6680:
------------------------------------

    Release Note: Fix "push" command in contrib/cloud scripts
          Status: Patch Available  (was: Open)

> hadoop-cloud push command invokes proxy creation
> ------------------------------------------------
>
>                 Key: HADOOP-6680
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6680
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: contrib/cloud
>            Reporter: Andrew Klochkov
>         Attachments: HADOOP-6680.patch
>
>
> There's a typo in src/contrib/cloud/src/py/hadoop/cloud/cli.py: when "push" command is invoked, the script actually invokes proxy creation instead of copying files. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7165-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 05 14:47:51 2010
Return-Path: <common-issues-return-7165-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6011 invoked from network); 5 Apr 2010 14:47:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Apr 2010 14:47:51 -0000
Received: (qmail 9571 invoked by uid 500); 5 Apr 2010 14:47:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9533 invoked by uid 500); 5 Apr 2010 14:47:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9525 invoked by uid 99); 5 Apr 2010 14:47:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 14:47:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 14:47:49 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 555A8234C4A8
	for <common-issues@hadoop.apache.org>; Mon,  5 Apr 2010 14:47:27 +0000 (UTC)
Message-ID: <606454420.687861270478847348.JavaMail.jira@brutus.apache.org>
Date: Mon, 5 Apr 2010 14:47:27 +0000 (UTC)
From: "Andrew Klochkov (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6680) hadoop-cloud push command invokes
 proxy creation
In-Reply-To: <1629624146.687721270478607313.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Klochkov updated HADOOP-6680:
------------------------------------

    Attachment: HADOOP-6680.patch

> hadoop-cloud push command invokes proxy creation
> ------------------------------------------------
>
>                 Key: HADOOP-6680
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6680
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: contrib/cloud
>            Reporter: Andrew Klochkov
>         Attachments: HADOOP-6680.patch
>
>
> There's a typo in src/contrib/cloud/src/py/hadoop/cloud/cli.py: when "push" command is invoked, the script actually invokes proxy creation instead of copying files. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7166-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 05 14:59:51 2010
Return-Path: <common-issues-return-7166-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8326 invoked from network); 5 Apr 2010 14:59:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Apr 2010 14:59:51 -0000
Received: (qmail 15460 invoked by uid 500); 5 Apr 2010 14:59:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15434 invoked by uid 500); 5 Apr 2010 14:59:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15417 invoked by uid 99); 5 Apr 2010 14:59:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 14:59:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 14:59:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 40CB0234C48D
	for <common-issues@hadoop.apache.org>; Mon,  5 Apr 2010 14:59:27 +0000 (UTC)
Message-ID: <315596123.688091270479567264.JavaMail.jira@brutus.apache.org>
Date: Mon, 5 Apr 2010 14:59:27 +0000 (UTC)
From: "Andrew Klochkov (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6681) Fill AWS credentials when configuring
 Hadoop on EC2 instances
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Fill AWS credentials when configuring Hadoop on EC2 instances
-------------------------------------------------------------

                 Key: HADOOP-6681
                 URL: https://issues.apache.org/jira/browse/HADOOP-6681
             Project: Hadoop Common
          Issue Type: Improvement
          Components: contrib/cloud
            Reporter: Andrew Klochkov
         Attachments: HADOOP-6681.patch

There's a function "configure_hadoop" in the hadoop-ec2-init-remote.sh script used to configure EC2 nodes for Hadoop. The function actually uses AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, but they are never passed to it. It can be fixed in service.py by passing those variables.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7167-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 05 14:59:52 2010
Return-Path: <common-issues-return-7167-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8384 invoked from network); 5 Apr 2010 14:59:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Apr 2010 14:59:52 -0000
Received: (qmail 15669 invoked by uid 500); 5 Apr 2010 14:59:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15624 invoked by uid 500); 5 Apr 2010 14:59:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15433 invoked by uid 99); 5 Apr 2010 14:59:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 14:59:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 14:59:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 491C3234C4AD
	for <common-issues@hadoop.apache.org>; Mon,  5 Apr 2010 14:59:27 +0000 (UTC)
Message-ID: <580154867.688141270479567298.JavaMail.jira@brutus.apache.org>
Date: Mon, 5 Apr 2010 14:59:27 +0000 (UTC)
From: "Andrew Klochkov (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6681) Fill AWS credentials when configuring
 Hadoop on EC2 instances
In-Reply-To: <315596123.688091270479567264.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Klochkov updated HADOOP-6681:
------------------------------------

    Attachment: HADOOP-6681.patch

> Fill AWS credentials when configuring Hadoop on EC2 instances
> -------------------------------------------------------------
>
>                 Key: HADOOP-6681
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6681
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: contrib/cloud
>            Reporter: Andrew Klochkov
>         Attachments: HADOOP-6681.patch
>
>
> There's a function "configure_hadoop" in the hadoop-ec2-init-remote.sh script used to configure EC2 nodes for Hadoop. The function actually uses AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, but they are never passed to it. It can be fixed in service.py by passing those variables.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7168-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 05 15:01:52 2010
Return-Path: <common-issues-return-7168-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13174 invoked from network); 5 Apr 2010 15:01:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Apr 2010 15:01:52 -0000
Received: (qmail 25133 invoked by uid 500); 5 Apr 2010 15:01:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25088 invoked by uid 500); 5 Apr 2010 15:01:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25044 invoked by uid 99); 5 Apr 2010 15:01:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 15:01:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 15:01:49 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 2DC88234C4AC
	for <common-issues@hadoop.apache.org>; Mon,  5 Apr 2010 15:01:28 +0000 (UTC)
Message-ID: <147317554.688201270479688186.JavaMail.jira@brutus.apache.org>
Date: Mon, 5 Apr 2010 15:01:28 +0000 (UTC)
From: "Andrew Klochkov (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6681) Fill AWS credentials when configuring
 Hadoop on EC2 instances
In-Reply-To: <315596123.688091270479567264.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Klochkov updated HADOOP-6681:
------------------------------------

    Release Note: Fill AWS credentials when configuring Hadoop EC2 instances
          Status: Patch Available  (was: Open)

> Fill AWS credentials when configuring Hadoop on EC2 instances
> -------------------------------------------------------------
>
>                 Key: HADOOP-6681
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6681
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: contrib/cloud
>            Reporter: Andrew Klochkov
>         Attachments: HADOOP-6681.patch
>
>
> There's a function "configure_hadoop" in the hadoop-ec2-init-remote.sh script used to configure EC2 nodes for Hadoop. The function actually uses AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, but they are never passed to it. It can be fixed in service.py by passing those variables.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7169-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 05 16:20:51 2010
Return-Path: <common-issues-return-7169-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33775 invoked from network); 5 Apr 2010 16:20:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Apr 2010 16:20:51 -0000
Received: (qmail 21770 invoked by uid 500); 5 Apr 2010 16:20:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21732 invoked by uid 500); 5 Apr 2010 16:20:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21724 invoked by uid 99); 5 Apr 2010 16:20:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 16:20:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 16:20:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EB810234C4AE
	for <common-issues@hadoop.apache.org>; Mon,  5 Apr 2010 16:20:27 +0000 (UTC)
Message-ID: <1098759203.690341270484427963.JavaMail.jira@brutus.apache.org>
Date: Mon, 5 Apr 2010 16:20:27 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6658) Exclude Private elements from
 generated Javadoc
In-Reply-To: <995349254.452871269404067173.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6658:
------------------------------

    Status: Patch Available  (was: Open)

> Exclude Private elements from generated Javadoc
> -----------------------------------------------
>
>                 Key: HADOOP-6658
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6658
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch
>
>
> Packages, classes and methods that are marked with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation should not appear in the public Javadoc. Developer Javadoc generated using the "javadoc-dev" ant target should continue to generate Javadoc for all program elements.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7170-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 05 16:20:51 2010
Return-Path: <common-issues-return-7170-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33802 invoked from network); 5 Apr 2010 16:20:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Apr 2010 16:20:51 -0000
Received: (qmail 21904 invoked by uid 500); 5 Apr 2010 16:20:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21860 invoked by uid 500); 5 Apr 2010 16:20:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21775 invoked by uid 99); 5 Apr 2010 16:20:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 16:20:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 16:20:49 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E43BC234C4A8
	for <common-issues@hadoop.apache.org>; Mon,  5 Apr 2010 16:20:27 +0000 (UTC)
Message-ID: <827840899.690301270484427934.JavaMail.jira@brutus.apache.org>
Date: Mon, 5 Apr 2010 16:20:27 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6658) Exclude Private elements from
 generated Javadoc
In-Reply-To: <995349254.452871269404067173.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6658:
------------------------------

    Attachment: HADOOP-6658.patch

New patch which excludes annotations that are marked Private or LimitedPrivate. Also removes Javadoc and RAT warnings.

> Exclude Private elements from generated Javadoc
> -----------------------------------------------
>
>                 Key: HADOOP-6658
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6658
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch
>
>
> Packages, classes and methods that are marked with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation should not appear in the public Javadoc. Developer Javadoc generated using the "javadoc-dev" ant target should continue to generate Javadoc for all program elements.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7171-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 05 16:30:54 2010
Return-Path: <common-issues-return-7171-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34937 invoked from network); 5 Apr 2010 16:30:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Apr 2010 16:30:54 -0000
Received: (qmail 26192 invoked by uid 500); 5 Apr 2010 16:30:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26162 invoked by uid 500); 5 Apr 2010 16:30:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26154 invoked by uid 99); 5 Apr 2010 16:30:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 16:30:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 16:30:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 51570234C4B3
	for <common-issues@hadoop.apache.org>; Mon,  5 Apr 2010 16:30:27 +0000 (UTC)
Message-ID: <60952063.690571270485027332.JavaMail.jira@brutus.apache.org>
Date: Mon, 5 Apr 2010 16:30:27 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6670) UserGroupInformation doesn't support
 use in hash tables
In-Reply-To: <242202762.620161270078590059.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6670:
----------------------------------

    Attachment: fs-close.patch

This is the updated patch for y20. Not for inclusion.

> UserGroupInformation doesn't support use in hash tables
> -------------------------------------------------------
>
>                 Key: HADOOP-6670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6670
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: fs-close.patch, fs-close.patch, fs-close.patch
>
>
> The UserGroupInformation objects are mutable, but they are used as keys in hash tables. This leads to serious problems in the FileSystem cache and RPC connection cache. We need to change the hashCode to be the identity hash code of the Subject and change equals to use == between the Subjects.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7172-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 05 17:57:55 2010
Return-Path: <common-issues-return-7172-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54431 invoked from network); 5 Apr 2010 17:57:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Apr 2010 17:57:55 -0000
Received: (qmail 5126 invoked by uid 500); 5 Apr 2010 17:57:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5096 invoked by uid 500); 5 Apr 2010 17:57:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5088 invoked by uid 99); 5 Apr 2010 17:57:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 17:57:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 17:57:52 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 45F12234C495
	for <common-issues@hadoop.apache.org>; Mon,  5 Apr 2010 17:57:31 +0000 (UTC)
Message-ID: <1182425019.692791270490251285.JavaMail.jira@brutus.apache.org>
Date: Mon, 5 Apr 2010 17:57:31 +0000 (UTC)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6678) Propose some changes to FileContext
In-Reply-To: <360625036.668801270253667713.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853465#action_12853465 ] 

Hairong Kuang commented on HADOOP-6678:
---------------------------------------

We thought about client side stat cache too. Actually right before I filed the jira, we went through the two options one more time. Although I am excited about the cache solution, we feel that the proposed solution for #2 is less controversy, straightforward, and also exposes more information to for users to write more efficient code. 

> Propose some changes to FileContext
> -----------------------------------
>
>                 Key: HADOOP-6678
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6678
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Hairong Kuang
>             Fix For: 0.21.0, 0.22.0
>
>
> # Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.
> # Remove methods isFile(Path), isDirectory(Path), and exists.
> All these methods are implemented by calling getFileStatus(Path).But most users are not aware of this. They would write code as below: 
> {code}
>   FileContext fc = ..;
>   if (fc.exists(path)) {
>     if (fc.isFile(path)) {
>      ...
>     } else {
>     ...
>     }
>   }
> {code}
> The above code adds unnecessary getFileInfo RPC to NameNode. In our production clusters, we often see that the number of getFileStatus calls is multiple times of the open calls. If we remove isFile, isDirectory, and exists from FileContext, users have to explicitly call getFileStatus first, it is more likely that they will write more efficient code as follow:
> {code}
>   FileContext fc = ...;
>   FileStatus fstatus = fc.getFileStatus(path);
>   if (fstatus.isFile() {
>     ...
>   } else {
>     ...
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7173-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 05 18:01:49 2010
Return-Path: <common-issues-return-7173-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55796 invoked from network); 5 Apr 2010 18:01:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Apr 2010 18:01:49 -0000
Received: (qmail 9653 invoked by uid 500); 5 Apr 2010 18:01:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9626 invoked by uid 500); 5 Apr 2010 18:01:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9618 invoked by uid 99); 5 Apr 2010 18:01:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 18:01:49 +0000
X-ASF-Spam-Status: No, hits=-1218.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 18:01:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 154C0234C4AD
	for <common-issues@hadoop.apache.org>; Mon,  5 Apr 2010 18:01:28 +0000 (UTC)
Message-ID: <816889941.693011270490488086.JavaMail.jira@brutus.apache.org>
Date: Mon, 5 Apr 2010 18:01:28 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6678) Propose some changes to FileContext
In-Reply-To: <360625036.668801270253667713.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853468#action_12853468 ] 

Todd Lipcon commented on HADOOP-6678:
-------------------------------------

OK, sounds good. Thanks for the info, Hairong.

> Propose some changes to FileContext
> -----------------------------------
>
>                 Key: HADOOP-6678
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6678
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Hairong Kuang
>             Fix For: 0.21.0, 0.22.0
>
>
> # Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.
> # Remove methods isFile(Path), isDirectory(Path), and exists.
> All these methods are implemented by calling getFileStatus(Path).But most users are not aware of this. They would write code as below: 
> {code}
>   FileContext fc = ..;
>   if (fc.exists(path)) {
>     if (fc.isFile(path)) {
>      ...
>     } else {
>     ...
>     }
>   }
> {code}
> The above code adds unnecessary getFileInfo RPC to NameNode. In our production clusters, we often see that the number of getFileStatus calls is multiple times of the open calls. If we remove isFile, isDirectory, and exists from FileContext, users have to explicitly call getFileStatus first, it is more likely that they will write more efficient code as follow:
> {code}
>   FileContext fc = ...;
>   FileStatus fstatus = fc.getFileStatus(path);
>   if (fstatus.isFile() {
>     ...
>   } else {
>     ...
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7174-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 05 18:40:51 2010
Return-Path: <common-issues-return-7174-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69883 invoked from network); 5 Apr 2010 18:40:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Apr 2010 18:40:50 -0000
Received: (qmail 59231 invoked by uid 500); 5 Apr 2010 18:40:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59163 invoked by uid 500); 5 Apr 2010 18:40:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59155 invoked by uid 99); 5 Apr 2010 18:40:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 18:40:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 18:40:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 82565234C4B5
	for <common-issues@hadoop.apache.org>; Mon,  5 Apr 2010 18:40:27 +0000 (UTC)
Message-ID: <2110561584.694021270492827532.JavaMail.jira@brutus.apache.org>
Date: Mon, 5 Apr 2010 18:40:27 +0000 (UTC)
From: "Alan Gates (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6677) InterfaceAudience.LimitedPrivate
 should take a string not an enum
In-Reply-To: <680532766.665461270239927272.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853484#action_12853484 ] 

Alan Gates commented on HADOOP-6677:
------------------------------------

The thing I like about having a component name associated with the limited private interface is it tells developers who depends on that interface.  So if they want to change it they know who to talk to.  But I don't see it as critical.  If others feel it's better removed, I'm ok with that.

> InterfaceAudience.LimitedPrivate should take a string not an enum
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6677
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6677
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Alan Gates
>            Priority: Minor
>
> Trying to keep the list of all possible components sharing limited private interfaces up to date is painful.  As other subprojects beyond HDFS and MR use this interface it will only get worse (see PIG-1311).  If it is converted to a string other subprojects can use it without requiring patches to common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7175-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 05 19:40:49 2010
Return-Path: <common-issues-return-7175-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86547 invoked from network); 5 Apr 2010 19:40:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Apr 2010 19:40:49 -0000
Received: (qmail 43821 invoked by uid 500); 5 Apr 2010 19:40:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43754 invoked by uid 500); 5 Apr 2010 19:40:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43746 invoked by uid 99); 5 Apr 2010 19:40:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 19:40:49 +0000
X-ASF-Spam-Status: No, hits=-1219.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 19:40:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 55AD8234C4AC
	for <common-issues@hadoop.apache.org>; Mon,  5 Apr 2010 19:40:28 +0000 (UTC)
Message-ID: <45301982.695231270496428350.JavaMail.jira@brutus.apache.org>
Date: Mon, 5 Apr 2010 19:40:28 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6680) hadoop-cloud push command invokes
 proxy creation
In-Reply-To: <1629624146.687721270478607313.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853503#action_12853503 ] 

Hadoop QA commented on HADOOP-6680:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12440756/HADOOP-6680.patch
  against trunk revision 930096.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/440/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/440/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/440/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/440/console

This message is automatically generated.

> hadoop-cloud push command invokes proxy creation
> ------------------------------------------------
>
>                 Key: HADOOP-6680
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6680
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: contrib/cloud
>            Reporter: Andrew Klochkov
>         Attachments: HADOOP-6680.patch
>
>
> There's a typo in src/contrib/cloud/src/py/hadoop/cloud/cli.py: when "push" command is invoked, the script actually invokes proxy creation instead of copying files. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7176-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 05 19:42:52 2010
Return-Path: <common-issues-return-7176-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86805 invoked from network); 5 Apr 2010 19:42:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Apr 2010 19:42:51 -0000
Received: (qmail 44167 invoked by uid 500); 5 Apr 2010 19:42:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44131 invoked by uid 500); 5 Apr 2010 19:42:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44123 invoked by uid 99); 5 Apr 2010 19:42:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 19:42:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 19:42:49 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DFDA5234C4AD
	for <common-issues@hadoop.apache.org>; Mon,  5 Apr 2010 19:42:27 +0000 (UTC)
Message-ID: <1154910324.695301270496547916.JavaMail.jira@brutus.apache.org>
Date: Mon, 5 Apr 2010 19:42:27 +0000 (UTC)
From: "lu cheng (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5881) Simplify configuration related to
 task-memory-monitoring and memory-based scheduling
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853504#action_12853504 ] 

lu cheng commented on HADOOP-5881:
----------------------------------

Hi guys,
If set mapred.cluster.max.map.memory.mb  and mapred.cluster.max.reduce.memory.mb in the mapred-site.xml, for example, to be 3072, when I submit a grep job I get:
10/04/05 15:23:09 INFO mapred.FileInputFormat: Total input paths to process : 14
org.apache.hadoop.ipc.RemoteException: java.io.IOException: job_201004051501_0001(-1 memForMapTasks -1 memForReduceTasks): Invalid job requirements.
        at org.apache.hadoop.mapred.JobTracker.checkMemoryRequirements(JobTracker.java:3864)
        at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3014)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)

        at org.apache.hadoop.ipc.Client.call(Client.java:740)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
        at org.apache.hadoop.mapred.$Proxy1.submitJob(Unknown Source)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:800)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1249)
        at org.apache.hadoop.examples.Grep.run(Grep.java:69)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.examples.Grep.main(Grep.java:93)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
        at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)


Also, If you set  mapred.cluster.map.memory.mb and mapred.cluster.reduce.memory.mb to be 512, the system always prompts you that :
INFO mapred.JobClient: Task Id : attempt_201004051259_0001_r_000000_0, Status : FAILED
TaskTree [pid=32315,tipID=attempt_201004051259_0001_r_000000_0] is running beyond memory-limits. Current usage : 556195840bytes. Limit : -1048576bytes. Killing task.


The version I deploy is 0.20.2.

It seems in some place the code use bytes, and in some place it uses MB.

Can someone fix this problem?


> Simplify configuration related to task-memory-monitoring and memory-based scheduling
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5881
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5881
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Vinod K V
>            Assignee: Vinod K V
>             Fix For: 0.20.1
>
>         Attachments: HADOOP-5881-20090526-branch-20.1.txt, HADOOP-5881-20090526.1.txt
>
>
> The configuration we have now is pretty complicated. Besides everything else, the mechanism of not specifying parameters separately for maps and reduces leads to problems like HADOOP-5811. This JIRA should address simplifying things and at the same time solving these problems.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7177-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 05 19:50:51 2010
Return-Path: <common-issues-return-7177-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87614 invoked from network); 5 Apr 2010 19:50:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Apr 2010 19:50:50 -0000
Received: (qmail 48338 invoked by uid 500); 5 Apr 2010 19:50:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48313 invoked by uid 500); 5 Apr 2010 19:50:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48305 invoked by uid 99); 5 Apr 2010 19:50:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 19:50:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 19:50:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 6CA51234C4A8
	for <common-issues@hadoop.apache.org>; Mon,  5 Apr 2010 19:50:27 +0000 (UTC)
Message-ID: <1734716398.695431270497027443.JavaMail.jira@brutus.apache.org>
Date: Mon, 5 Apr 2010 19:50:27 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6677) InterfaceAudience.LimitedPrivate
 should take a string not an enum
In-Reply-To: <680532766.665461270239927272.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853507#action_12853507 ] 

Tom White commented on HADOOP-6677:
-----------------------------------

Perhaps it could be optional - if no components are listed, then that means that any component may use it.

> InterfaceAudience.LimitedPrivate should take a string not an enum
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6677
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6677
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Alan Gates
>            Priority: Minor
>
> Trying to keep the list of all possible components sharing limited private interfaces up to date is painful.  As other subprojects beyond HDFS and MR use this interface it will only get worse (see PIG-1311).  If it is converted to a string other subprojects can use it without requiring patches to common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7178-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 05 19:58:51 2010
Return-Path: <common-issues-return-7178-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89250 invoked from network); 5 Apr 2010 19:58:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Apr 2010 19:58:51 -0000
Received: (qmail 59045 invoked by uid 500); 5 Apr 2010 19:58:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59022 invoked by uid 500); 5 Apr 2010 19:58:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59014 invoked by uid 99); 5 Apr 2010 19:58:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 19:58:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 19:58:49 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C7340234C48D
	for <common-issues@hadoop.apache.org>; Mon,  5 Apr 2010 19:58:27 +0000 (UTC)
Message-ID: <914979899.695841270497507814.JavaMail.jira@brutus.apache.org>
Date: Mon, 5 Apr 2010 19:58:27 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6681) Fill AWS credentials when
 configuring Hadoop on EC2 instances
In-Reply-To: <315596123.688091270479567264.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853518#action_12853518 ] 

Hadoop QA commented on HADOOP-6681:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12440759/HADOOP-6681.patch
  against trunk revision 930096.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/441/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/441/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/441/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/441/console

This message is automatically generated.

> Fill AWS credentials when configuring Hadoop on EC2 instances
> -------------------------------------------------------------
>
>                 Key: HADOOP-6681
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6681
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: contrib/cloud
>            Reporter: Andrew Klochkov
>         Attachments: HADOOP-6681.patch
>
>
> There's a function "configure_hadoop" in the hadoop-ec2-init-remote.sh script used to configure EC2 nodes for Hadoop. The function actually uses AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, but they are never passed to it. It can be fixed in service.py by passing those variables.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7179-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 05 20:15:48 2010
Return-Path: <common-issues-return-7179-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96597 invoked from network); 5 Apr 2010 20:15:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Apr 2010 20:15:48 -0000
Received: (qmail 88487 invoked by uid 500); 5 Apr 2010 20:15:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88456 invoked by uid 500); 5 Apr 2010 20:15:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88448 invoked by uid 99); 5 Apr 2010 20:15:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 20:15:48 +0000
X-ASF-Spam-Status: No, hits=-1220.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 20:15:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4DADC234C4AC
	for <common-issues@hadoop.apache.org>; Mon,  5 Apr 2010 20:15:27 +0000 (UTC)
Message-ID: <298546830.696081270498527317.JavaMail.jira@brutus.apache.org>
Date: Mon, 5 Apr 2010 20:15:27 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6658) Exclude Private elements from
 generated Javadoc
In-Reply-To: <995349254.452871269404067173.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853526#action_12853526 ] 

Hadoop QA commented on HADOOP-6658:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12440770/HADOOP-6658.patch
  against trunk revision 930096.

    +1 @author.  The patch does not contain any @author tags.

    +0 tests included.  The patch appears to be a documentation patch that doesn't require tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/442/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/442/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/442/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/442/console

This message is automatically generated.

> Exclude Private elements from generated Javadoc
> -----------------------------------------------
>
>                 Key: HADOOP-6658
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6658
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch
>
>
> Packages, classes and methods that are marked with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation should not appear in the public Javadoc. Developer Javadoc generated using the "javadoc-dev" ant target should continue to generate Javadoc for all program elements.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7180-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 05 20:28:51 2010
Return-Path: <common-issues-return-7180-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2397 invoked from network); 5 Apr 2010 20:28:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Apr 2010 20:28:51 -0000
Received: (qmail 8468 invoked by uid 500); 5 Apr 2010 20:28:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8443 invoked by uid 500); 5 Apr 2010 20:28:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8433 invoked by uid 99); 5 Apr 2010 20:28:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 20:28:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 20:28:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D7ED8234C4C7
	for <common-issues@hadoop.apache.org>; Mon,  5 Apr 2010 20:28:27 +0000 (UTC)
Message-ID: <1596794454.696661270499307883.JavaMail.jira@brutus.apache.org>
Date: Mon, 5 Apr 2010 20:28:27 +0000 (UTC)
From: "Patrick Hunt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5670) Hadoop configurations should be
 read from a distributed system
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853532#action_12853532 ] 

Patrick Hunt commented on HADOOP-5670:
--------------------------------------

Here's a really interesting paper from Akamai related to Quorum use for configuration:
"ACMS: The Akamai Configuration Management System"
http://www.usenix.org/event/nsdi05/tech/full_papers/sherman/sherman_html/index.html

ZooKeeper has a REST based interface that might work well if client's (hadoop) primary interest is to poll for configuration
during bootstrap. Not that one has to use REST, if the ZK client interface were used directly then one could register
for dynamic changes via ZK watches. Additionally the hadoop nodes could take advantage of other ZK features to implement
group membership,(nodes register themselves, and their state, with ZK, allowing simple monitoring
to be implemented), Leader election, etc...

> Hadoop configurations should be read from a distributed system
> --------------------------------------------------------------
>
>                 Key: HADOOP-5670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5670
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: conf
>            Reporter: Allen Wittenauer
>
> Rather than distributing the hadoop configuration files to every data node, compute node, etc, Hadoop should be able to read configuration information (dynamically!) from LDAP, ZooKeeper, whatever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7181-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 05 22:00:48 2010
Return-Path: <common-issues-return-7181-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27503 invoked from network); 5 Apr 2010 22:00:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Apr 2010 22:00:48 -0000
Received: (qmail 24867 invoked by uid 500); 5 Apr 2010 22:00:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24761 invoked by uid 500); 5 Apr 2010 22:00:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24753 invoked by uid 99); 5 Apr 2010 22:00:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 22:00:48 +0000
X-ASF-Spam-Status: No, hits=-1220.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 22:00:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 40B6B234C4AE
	for <common-issues@hadoop.apache.org>; Mon,  5 Apr 2010 22:00:27 +0000 (UTC)
Message-ID: <1580699864.697991270504827264.JavaMail.jira@brutus.apache.org>
Date: Mon, 5 Apr 2010 22:00:27 +0000 (UTC)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6678) Propose some changes to FileContext
In-Reply-To: <360625036.668801270253667713.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853558#action_12853558 ] 

Hairong Kuang commented on HADOOP-6678:
---------------------------------------

> I cold do part 2 in lieu of HADOOP-6585 (Add FileStatus#isDirectory and isFile) and HDFS-995, MR-1535.
Eli, thanks for volunteering to do part 2. Your help would be great!

> Propose some changes to FileContext
> -----------------------------------
>
>                 Key: HADOOP-6678
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6678
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Hairong Kuang
>             Fix For: 0.21.0, 0.22.0
>
>
> # Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.
> # Remove methods isFile(Path), isDirectory(Path), and exists.
> All these methods are implemented by calling getFileStatus(Path).But most users are not aware of this. They would write code as below: 
> {code}
>   FileContext fc = ..;
>   if (fc.exists(path)) {
>     if (fc.isFile(path)) {
>      ...
>     } else {
>     ...
>     }
>   }
> {code}
> The above code adds unnecessary getFileInfo RPC to NameNode. In our production clusters, we often see that the number of getFileStatus calls is multiple times of the open calls. If we remove isFile, isDirectory, and exists from FileContext, users have to explicitly call getFileStatus first, it is more likely that they will write more efficient code as follow:
> {code}
>   FileContext fc = ...;
>   FileStatus fstatus = fc.getFileStatus(path);
>   if (fstatus.isFile() {
>     ...
>   } else {
>     ...
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7182-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 05 23:54:50 2010
Return-Path: <common-issues-return-7182-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47015 invoked from network); 5 Apr 2010 23:54:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Apr 2010 23:54:50 -0000
Received: (qmail 10788 invoked by uid 500); 5 Apr 2010 23:54:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10750 invoked by uid 500); 5 Apr 2010 23:54:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10742 invoked by uid 99); 5 Apr 2010 23:54:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 23:54:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Apr 2010 23:54:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 38960234C4A8
	for <common-issues@hadoop.apache.org>; Mon,  5 Apr 2010 23:54:27 +0000 (UTC)
Message-ID: <1523422048.700531270511667229.JavaMail.jira@brutus.apache.org>
Date: Mon, 5 Apr 2010 23:54:27 +0000 (UTC)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6682) NetUtils:normalizeHostName does not
 process hostnames starting with [a-f] correctly
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
-----------------------------------------------------------------------------------

                 Key: HADOOP-6682
                 URL: https://issues.apache.org/jira/browse/HADOOP-6682
             Project: Hadoop Common
          Issue Type: Bug
          Components: io
            Reporter: Jakob Homan


  public static String normalizeHostName(String name) {
    if (Character.digit(name.charAt(0), 16) != -1) {
      return name;

This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address.  This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f].  Such names will not be converted to an ip address, but be returned unchanged.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7183-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 01:20:49 2010
Return-Path: <common-issues-return-7183-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67887 invoked from network); 6 Apr 2010 01:20:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 01:20:49 -0000
Received: (qmail 58011 invoked by uid 500); 6 Apr 2010 01:20:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57916 invoked by uid 500); 6 Apr 2010 01:20:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57907 invoked by uid 99); 6 Apr 2010 01:20:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 01:20:49 +0000
X-ASF-Spam-Status: No, hits=-1222.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 01:20:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 217D8234C4B4
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 01:20:28 +0000 (UTC)
Message-ID: <1093171124.703071270516828136.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 01:20:28 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6421) Symbolic links
In-Reply-To: <789869504.1260323298357.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853634#action_12853634 ] 

Hudson commented on HADOOP-6421:
--------------------------------

Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #146 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/146/])
    

> Symbolic links
> --------------
>
>                 Key: HADOOP-6421
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6421
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>         Attachments: symlink-25-common.patch, symlink-26-common.patch, symlink-26-common.patch, symlink24-common.patch, symlink27-common.patch, symlink28-common.patch, symlink29-common.patch, symlink29-common.patch, symlink29-common.patch, symlink30-common.patch, symlink31-common.patch, symlink32-common.patch, symlink33-common.patch, symlink34-common.patch, symlink35-common.patch, symlink36-common.patch, symlink37-common.patch, symlink38-common.patch, symlink39-common.patch
>
>
> Here's a jira for the common parts of HDFS-245, mostly changes to FileContext and AbstractFileSystem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7184-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 01:21:14 2010
Return-Path: <common-issues-return-7184-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68579 invoked from network); 6 Apr 2010 01:21:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 01:21:14 -0000
Received: (qmail 60725 invoked by uid 500); 6 Apr 2010 01:21:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60686 invoked by uid 500); 6 Apr 2010 01:21:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60655 invoked by uid 99); 6 Apr 2010 01:21:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 01:21:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 01:21:11 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 611E7234C4EF
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 01:20:30 +0000 (UTC)
Message-ID: <324936148.704191270516830396.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 01:20:30 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6569) FsShell#cat should avoid calling
 unecessary getFileStatus before opening a file to read
In-Reply-To: <1333327371.313901266350847909.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853647#action_12853647 ] 

Hudson commented on HADOOP-6569:
--------------------------------

Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #146 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/146/])
    

> FsShell#cat should avoid calling unecessary getFileStatus before opening a file to read
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6569
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6569
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: optimizeCat-yahoo.patch, optimizeCat-yahoo1.patch, optimizeCat-yahoo2.patch, optimizeCat.patch, optimizeCat.patch
>
>
> Since FileSystem#open throws a FileNotFoundException when the file to be read does not exist, there is no need to check if the file is a directory or not before open. In case of HDFS, this could reduce one getFileInfo RPC to NameNode.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7185-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 01:21:37 2010
Return-Path: <common-issues-return-7185-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69512 invoked from network); 6 Apr 2010 01:21:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 01:21:36 -0000
Received: (qmail 62721 invoked by uid 500); 6 Apr 2010 01:21:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62683 invoked by uid 500); 6 Apr 2010 01:21:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62634 invoked by uid 99); 6 Apr 2010 01:21:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 01:21:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 01:21:32 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 26DBE234C4D7
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 01:20:32 +0000 (UTC)
Message-ID: <464878401.705251270516832158.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 01:20:32 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6579) A utility for reading and writing
 tokens into a URL safe string.
In-Reply-To: <843427920.414981266707188674.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853665#action_12853665 ] 

Hudson commented on HADOOP-6579:
--------------------------------

Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #146 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/146/])
    

> A utility for reading and writing tokens into a URL safe string.
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6579
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6579
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.22.0
>
>         Attachments: c-6579-hdfs.patch, c-6579-mr.patch, c-6579.patch, c-6579.patch, c-6579.patch, c-6579.patch, c-6579.patch
>
>
> We need to include HDFS delegation tokens in the URLs while browsing the file system. Therefore, we need a url-safe way to encode and decode them.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7186-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 01:56:48 2010
Return-Path: <common-issues-return-7186-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73709 invoked from network); 6 Apr 2010 01:56:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 01:56:48 -0000
Received: (qmail 87226 invoked by uid 500); 6 Apr 2010 01:56:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87177 invoked by uid 500); 6 Apr 2010 01:56:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87169 invoked by uid 99); 6 Apr 2010 01:56:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 01:56:48 +0000
X-ASF-Spam-Status: No, hits=-1223.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 01:56:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 45B7F234C48D
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 01:56:27 +0000 (UTC)
Message-ID: <2121624364.706511270518987284.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 01:56:27 +0000 (UTC)
From: "Xiao Kang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6683) the first optimization:
 ZlibCompressor does not fully utilize the buffer
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

the first optimization: ZlibCompressor does not fully utilize the buffer
------------------------------------------------------------------------

                 Key: HADOOP-6683
                 URL: https://issues.apache.org/jira/browse/HADOOP-6683
             Project: Hadoop Common
          Issue Type: Sub-task
          Components: io
    Affects Versions: 0.20.2
            Reporter: Xiao Kang


Thanks for Hong Tang's advice.

Sub task created for the first optimization. HADOOP-6662 closed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7187-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 01:58:48 2010
Return-Path: <common-issues-return-7187-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73969 invoked from network); 6 Apr 2010 01:58:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 01:58:48 -0000
Received: (qmail 89028 invoked by uid 500); 6 Apr 2010 01:58:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88993 invoked by uid 500); 6 Apr 2010 01:58:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88985 invoked by uid 99); 6 Apr 2010 01:58:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 01:58:48 +0000
X-ASF-Spam-Status: No, hits=-1223.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 01:58:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4361A234C4AD
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 01:58:27 +0000 (UTC)
Message-ID: <746365244.706581270519107275.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 01:58:27 +0000 (UTC)
From: "Xiao Kang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4196) Possible performance enhancement in
 Hadoop compress module
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-4196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853692#action_12853692 ] 

Xiao Kang commented on HADOOP-4196:
-----------------------------------

Thank you. Sub task HADOOP-6683 created.

> Possible performance enhancement in Hadoop compress module
> ----------------------------------------------------------
>
>                 Key: HADOOP-4196
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4196
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.18.0
>            Reporter: Hong Tang
>
> There are several less performant implementation issues with the current Hadoop compression module. Generally, the opportunities all come from the fact that the granuarities of I/O operations from the CompressionStream and DecompressionStream are not controllable by the users, and thus users are forced to attach BufferedInputStream or BufferedOutputStream to both ends of the CompressionStream and DecompressionStream:
> - ZlibCompressor: always returns false from needInput() after setInput(), and thus lead to a native call deflateBytesDirect() for almost every write() operation from CompressorStream(). This becomes problematic when applications call write() on the CompressorStream with small write sizes (e.g. one byte at a time). It is better to follow similar code path in LzoCompressor and append to internal uncompressed data buffer.
> - CompressorStream: whenever the compressor produces some compressed data, it will directly issue write() calls to the down stream. Could be improved by keep appending to the byte[] until it is full (or half full) before writing to the down stream. Otherwise, applications have to use a BufferedOutputStream as the down stream in case the output sizes from CompressorStream is too small. This generally causes double buffering.
> - BlockCompressorStream: similar issue as described above.
> - BlockDecompressorStream: getCompressedData() reads only one compressed chunk at a time. Could be better to read a full buffer, and then obtain compressed chunk from buffer (similar to DecompressStream is doing, but admittedly a bit more complicated).
> In generally, the following could be some guideline of Compressor/Decompressor and CompressorStream/DecompressorStream design/implementation that can give users some performance guarantee:
> - Compressor and Decompressor keep two DirectByteBuffer, the size of which should be tuned to be optimal with regard to the specific compression/decompression algorithm. Ensure always call Compressor.compress() will a full (or near full) uncompressed data DirectBuffer.
> - CompressorStream and DecompressorStream maintains a byte[] to read data from the down stream. The size of the byte[] should be user customizable (add a bufferSize parameter to CompressionCodec's createInputStream and createOutputStream interface). Ensure that I/O from the down stream at or near the granularity of the size of the byte[]. So applications can simply rely on the buffering inside CompressorStream and DecompressorStream (for the case of LZO: BlockCompressorStream and BlockDecompressorStream).
> A more radical change would be to let the downward InputStream to directly deposit data to a ByteBuffer or the downard OutputStream to accept input data from ByteBuffer. We may call it ByteBufferInputStream and ByteBufferOutputStream. The CompressorStream and DecompressorStream may simply test whether the down stream indeed implements such interfaces and bypass its own byte[] buffer if true.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7188-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 02:12:55 2010
Return-Path: <common-issues-return-7188-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76034 invoked from network); 6 Apr 2010 02:12:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 02:12:55 -0000
Received: (qmail 97728 invoked by uid 500); 6 Apr 2010 02:12:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97636 invoked by uid 500); 6 Apr 2010 02:12:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97628 invoked by uid 99); 6 Apr 2010 02:12:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 02:12:55 +0000
X-ASF-Spam-Status: No, hits=-1223.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 02:12:55 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 97686234C48C
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 02:12:34 +0000 (UTC)
Message-ID: <1588736317.706971270519954619.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 02:12:34 +0000 (UTC)
From: "Xiao Kang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6683) the first optimization:
 ZlibCompressor does not fully utilize the buffer
In-Reply-To: <2121624364.706511270518987284.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xiao Kang updated HADOOP-6683:
------------------------------

    Attachment: ZlibCompressor.java.patch

Patch attached.

> the first optimization: ZlibCompressor does not fully utilize the buffer
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6683
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6683
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Xiao Kang
>         Attachments: ZlibCompressor.java.patch
>
>
> Thanks for Hong Tang's advice.
> Sub task created for the first optimization. HADOOP-6662 closed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7189-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 02:12:58 2010
Return-Path: <common-issues-return-7189-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76125 invoked from network); 6 Apr 2010 02:12:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 02:12:58 -0000
Received: (qmail 97867 invoked by uid 500); 6 Apr 2010 02:12:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97824 invoked by uid 500); 6 Apr 2010 02:12:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97816 invoked by uid 99); 6 Apr 2010 02:12:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 02:12:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 02:12:55 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id F29CD234C4B2
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 02:12:34 +0000 (UTC)
Message-ID: <1874355203.707041270519954992.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 02:12:34 +0000 (UTC)
From: "Xiao Kang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6662) hadoop zlib compression does not
 fully utilize the buffer
In-Reply-To: <814636202.542801269849567653.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xiao Kang updated HADOOP-6662:
------------------------------

    Resolution: Duplicate
        Status: Resolved  (was: Patch Available)

Duplicated with HADOOP-6683.

> hadoop zlib compression does not fully utilize the buffer
> ---------------------------------------------------------
>
>                 Key: HADOOP-6662
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6662
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Xiao Kang
>         Attachments: ZlibCompressor.patch
>
>
> org.apache.hadoop.io.compress.ZlibCompressonr does not fully utilize its buffer. 
> Its needesInput() return false when there is any data in its buffer (64K by default). The performance will greately degrade since an JNI call will be invoded at each time the write() method of CompressonStream is called. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7190-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 02:28:48 2010
Return-Path: <common-issues-return-7190-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77808 invoked from network); 6 Apr 2010 02:28:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 02:28:48 -0000
Received: (qmail 8210 invoked by uid 500); 6 Apr 2010 02:28:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8186 invoked by uid 500); 6 Apr 2010 02:28:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8178 invoked by uid 99); 6 Apr 2010 02:28:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 02:28:48 +0000
X-ASF-Spam-Status: No, hits=-1223.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 02:28:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 40B9F234C4A8
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 02:28:27 +0000 (UTC)
Message-ID: <1692488264.707211270520907264.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 02:28:27 +0000 (UTC)
From: "Xiao Kang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6663) BlockDecompressorStream get EOF
 exception when decompressing the file compressed from empty file
In-Reply-To: <1619883340.543171269850767204.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xiao Kang updated HADOOP-6663:
------------------------------

    Attachment: BlockDecompressorStream.java.patch

Thank you for your advice. 

New patch attached with apache header and JUnit 4 style test case.

> BlockDecompressorStream get EOF exception when decompressing the file compressed from empty file
> ------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6663
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6663
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Xiao Kang
>         Attachments: BlockDecompressorStream.java.patch, BlockDecompressorStream.java.patch, BlockDecompressorStream.patch
>
>
> An empty file can be compressed using BlockDecompressorStream, which is for block-based compressiong algorithm such as LZO. However, when decompressing the compressed file, BlockDecompressorStream get EOF exception.
> Here is a typical exception stack:
> java.io.EOFException
> at org.apache.hadoop.io.compress.BlockDecompressorStream.rawReadInt(BlockDecompressorStream.java:125)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:96)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:82)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at java.io.InputStream.read(InputStream.java:85)
> at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:39)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:186)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:170)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:18)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
> at org.apache.hadoop.mapred.Child.main(Child.java:196)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7191-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 03:32:51 2010
Return-Path: <common-issues-return-7191-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85731 invoked from network); 6 Apr 2010 03:32:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 03:32:51 -0000
Received: (qmail 46513 invoked by uid 500); 6 Apr 2010 03:32:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46402 invoked by uid 500); 6 Apr 2010 03:32:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46394 invoked by uid 99); 6 Apr 2010 03:32:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 03:32:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 03:32:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 52C3A234C4B2
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 03:32:27 +0000 (UTC)
Message-ID: <763120834.708171270524747338.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 03:32:27 +0000 (UTC)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5881) Simplify configuration related to
 task-memory-monitoring and memory-based scheduling
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853717#action_12853717 ] 

Vinod K V commented on HADOOP-5881:
-----------------------------------

Are you using *all* the configuration items properly as documented [here|https://issues.apache.org/jira/browse/HADOOP-5881?focusedCommentId=12712919&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12712919] ? You should be setting all of them with appropriate values.

The first error you mentioned seems to happening because you only configured the max limits but not the cluster wide slot size and the per-job configuration.

No idea about the second problem. If you give more details, we can see.

We have been using this feature with the mentioned configuration items so far without any issues, so I don't think there is any problem of using bytes vs MB at all.

> Simplify configuration related to task-memory-monitoring and memory-based scheduling
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5881
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5881
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Vinod K V
>            Assignee: Vinod K V
>             Fix For: 0.20.1
>
>         Attachments: HADOOP-5881-20090526-branch-20.1.txt, HADOOP-5881-20090526.1.txt
>
>
> The configuration we have now is pretty complicated. Besides everything else, the mechanism of not specifying parameters separately for maps and reduces leads to problems like HADOOP-5811. This JIRA should address simplifying things and at the same time solving these problems.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7192-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 04:42:49 2010
Return-Path: <common-issues-return-7192-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93176 invoked from network); 6 Apr 2010 04:42:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 04:42:49 -0000
Received: (qmail 87298 invoked by uid 500); 6 Apr 2010 04:42:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87213 invoked by uid 500); 6 Apr 2010 04:42:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87205 invoked by uid 99); 6 Apr 2010 04:42:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 04:42:48 +0000
X-ASF-Spam-Status: No, hits=-1224.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 04:42:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 5C1E3234C4AC
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 04:42:27 +0000 (UTC)
Message-ID: <2087770439.709191270528947376.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 04:42:27 +0000 (UTC)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6682) NetUtils:normalizeHostName does not
 process hostnames starting with [a-f] correctly
In-Reply-To: <1523422048.700531270511667229.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853739#action_12853739 ] 

Hong Tang commented on HADOOP-6682:
-----------------------------------

Similar issue came up in MAPREDUCE-1222. The following code might be useful:

{noformat}
+  static Pattern IP_PATTERN;
+  
+  static {
+    // 0-255
+    String IPV4BK1 = "(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)";
+    // .b.c.d - where b/c/d are 0-255, and optionally adding two more
+    // backslashes before each period
+    String IPV4BKN = "(?:\\\\?\\." + IPV4BK1 + "){3}";
+    String IPV4_PATTERN = IPV4BK1 + IPV4BKN;
+    
+    // first hexadecimal number
+    String IPV6BK1 = "(?:[0-9a-fA-F]{1,4})";
+    // remaining 7 hexadecimal numbers, each preceded with ":".
+    String IPV6BKN = "(?::" + IPV6BK1 + "){7}";
+    String IPV6_PATTERN = IPV6BK1 + IPV6BKN;
+
+    IP_PATTERN = Pattern.compile(
+        "^(?:" + IPV4_PATTERN + "|" + IPV6_PATTERN + ")$");
+  }
+
+ 
+  static boolean isIPAddress(String hostname) {
+    return IP_PATTERN.matcher(hostname).matches();
+  }
{noformat}

> NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6682
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6682
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Jakob Homan
>
>   public static String normalizeHostName(String name) {
>     if (Character.digit(name.charAt(0), 16) != -1) {
>       return name;
> This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address.  This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f].  Such names will not be converted to an ip address, but be returned unchanged.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7193-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 04:44:49 2010
Return-Path: <common-issues-return-7193-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93442 invoked from network); 6 Apr 2010 04:44:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 04:44:48 -0000
Received: (qmail 87664 invoked by uid 500); 6 Apr 2010 04:44:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87641 invoked by uid 500); 6 Apr 2010 04:44:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87633 invoked by uid 99); 6 Apr 2010 04:44:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 04:44:48 +0000
X-ASF-Spam-Status: No, hits=-1224.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 04:44:47 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4CC7B234C4AC
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 04:44:27 +0000 (UTC)
Message-ID: <581308153.709241270529067313.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 04:44:27 +0000 (UTC)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6682) NetUtils:normalizeHostName does not
 process hostnames starting with [a-f] correctly
In-Reply-To: <1523422048.700531270511667229.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853740#action_12853740 ] 

Hong Tang commented on HADOOP-6682:
-----------------------------------

The regex IPV4BKN needs some change because in job history dots (.) are escaped with double backslashes.

> NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6682
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6682
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Jakob Homan
>
>   public static String normalizeHostName(String name) {
>     if (Character.digit(name.charAt(0), 16) != -1) {
>       return name;
> This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address.  This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f].  Such names will not be converted to an ip address, but be returned unchanged.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7194-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 08:25:58 2010
Return-Path: <common-issues-return-7194-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35024 invoked from network); 6 Apr 2010 08:25:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 08:25:58 -0000
Received: (qmail 44282 invoked by uid 500); 6 Apr 2010 08:25:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44181 invoked by uid 500); 6 Apr 2010 08:25:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44171 invoked by uid 99); 6 Apr 2010 08:25:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 08:25:55 +0000
X-ASF-Spam-Status: No, hits=-1225.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 08:25:54 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 223BD234C4CC
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 08:25:34 +0000 (UTC)
Message-ID: <1387308538.3001270542334139.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 08:25:34 +0000 (UTC)
From: "Andrey Kuzmin (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5670) Hadoop configurations should be
 read from a distributed system
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853801#action_12853801 ] 

Andrey Kuzmin commented on HADOOP-5670:
---------------------------------------

> nodes register themselves, and their state, with ZK, allowing simple monitoring to be implemented
I would add (dynamic) membership-aware scheduling. It's typically far from straightforward for a scheduler to account for nodes (re)joining  group unless there is an explicit group registration mechanism, with join/leave events dispatched to scheduler and other interested parties.

> Hadoop configurations should be read from a distributed system
> --------------------------------------------------------------
>
>                 Key: HADOOP-5670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5670
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: conf
>            Reporter: Allen Wittenauer
>
> Rather than distributing the hadoop configuration files to every data node, compute node, etc, Hadoop should be able to read configuration information (dynamically!) from LDAP, ZooKeeper, whatever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7195-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 10:50:01 2010
Return-Path: <common-issues-return-7195-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63112 invoked from network); 6 Apr 2010 10:50:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 10:50:01 -0000
Received: (qmail 74561 invoked by uid 500); 6 Apr 2010 10:50:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74403 invoked by uid 500); 6 Apr 2010 10:49:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74071 invoked by uid 99); 6 Apr 2010 10:49:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 10:49:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 10:49:55 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 77E54234C4B5
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 10:49:34 +0000 (UTC)
Message-ID: <1448622083.5321270550974490.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 10:49:34 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6421) Symbolic links
In-Reply-To: <789869504.1260323298357.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853849#action_12853849 ] 

Hudson commented on HADOOP-6421:
--------------------------------

Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #302 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/302/])
    

> Symbolic links
> --------------
>
>                 Key: HADOOP-6421
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6421
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>         Attachments: symlink-25-common.patch, symlink-26-common.patch, symlink-26-common.patch, symlink24-common.patch, symlink27-common.patch, symlink28-common.patch, symlink29-common.patch, symlink29-common.patch, symlink29-common.patch, symlink30-common.patch, symlink31-common.patch, symlink32-common.patch, symlink33-common.patch, symlink34-common.patch, symlink35-common.patch, symlink36-common.patch, symlink37-common.patch, symlink38-common.patch, symlink39-common.patch
>
>
> Here's a jira for the common parts of HDFS-245, mostly changes to FileContext and AbstractFileSystem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7196-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 10:50:19 2010
Return-Path: <common-issues-return-7196-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63455 invoked from network); 6 Apr 2010 10:50:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 10:50:19 -0000
Received: (qmail 75262 invoked by uid 500); 6 Apr 2010 10:50:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75093 invoked by uid 500); 6 Apr 2010 10:50:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74987 invoked by uid 99); 6 Apr 2010 10:50:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 10:50:18 +0000
X-ASF-Spam-Status: No, hits=-1226.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 10:50:17 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EE89D29A0013
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 10:49:35 +0000 (UTC)
Message-ID: <213853279.6291270550975976.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 10:49:35 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6569) FsShell#cat should avoid calling
 unecessary getFileStatus before opening a file to read
In-Reply-To: <1333327371.313901266350847909.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853859#action_12853859 ] 

Hudson commented on HADOOP-6569:
--------------------------------

Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #302 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/302/])
    

> FsShell#cat should avoid calling unecessary getFileStatus before opening a file to read
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6569
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6569
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: optimizeCat-yahoo.patch, optimizeCat-yahoo1.patch, optimizeCat-yahoo2.patch, optimizeCat.patch, optimizeCat.patch
>
>
> Since FileSystem#open throws a FileNotFoundException when the file to be read does not exist, there is no need to check if the file is a directory or not before open. In case of HDFS, this could reduce one getFileInfo RPC to NameNode.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7197-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 10:50:40 2010
Return-Path: <common-issues-return-7197-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63850 invoked from network); 6 Apr 2010 10:50:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 10:50:40 -0000
Received: (qmail 76165 invoked by uid 500); 6 Apr 2010 10:50:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76122 invoked by uid 500); 6 Apr 2010 10:50:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76090 invoked by uid 99); 6 Apr 2010 10:50:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 10:50:39 +0000
X-ASF-Spam-Status: No, hits=-1227.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 10:50:37 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 9E58129A002D
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 10:49:37 +0000 (UTC)
Message-ID: <1949404944.7151270550977647.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 10:49:37 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6579) A utility for reading and writing
 tokens into a URL safe string.
In-Reply-To: <843427920.414981266707188674.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853874#action_12853874 ] 

Hudson commented on HADOOP-6579:
--------------------------------

Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #302 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/302/])
    

> A utility for reading and writing tokens into a URL safe string.
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6579
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6579
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.22.0
>
>         Attachments: c-6579-hdfs.patch, c-6579-mr.patch, c-6579.patch, c-6579.patch, c-6579.patch, c-6579.patch, c-6579.patch
>
>
> We need to include HDFS delegation tokens in the URLs while browsing the file system. Therefore, we need a url-safe way to encode and decode them.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7198-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 13:18:56 2010
Return-Path: <common-issues-return-7198-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6435 invoked from network); 6 Apr 2010 13:18:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 13:18:56 -0000
Received: (qmail 30624 invoked by uid 500); 6 Apr 2010 13:18:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30469 invoked by uid 500); 6 Apr 2010 13:18:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30100 invoked by uid 99); 6 Apr 2010 13:18:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 13:18:54 +0000
X-ASF-Spam-Status: No, hits=-1228.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 13:18:53 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 88EDA234C4A8
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 13:18:33 +0000 (UTC)
Message-ID: <1258928011.9761270559913559.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 13:18:33 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6679) add retry logic to
 NetUtils.createSocketAddr() when hostname resolution fails
In-Reply-To: <18346259.678881270363227220.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853925#action_12853925 ] 

Steve Loughran commented on HADOOP-6679:
----------------------------------------

# This is a git patch, not something hudson will handle, try --no-prefix, I think.

# What JVM options are you running to disable DNS caching, both successful and unsuccessful. Without that, retry in code may not be useful.

# If we are changing how ports are opened, I'd quite like to see a string parameter saying what the port is to be used for, perhaps with some logging. Sometimes it can be tricky trying to find out why some apparenlty random port is open, and if it was all logged (at -debug, with a stack trace), some things may be easier to deal with. 

> add retry logic to NetUtils.createSocketAddr() when hostname resolution fails
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-6679
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6679
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.0
>            Reporter: sam rash
>            Priority: Minor
>         Attachments: HADOOP-6679-patch-1.txt
>
>
> transient DNS errors can cause the InetSocketAddress created in NetUtils.createSocketAddr() to be unresolved when the hostname is resolvable.  Add a single retry after a short sleep using a simple InetSocketAddressFactory class

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7199-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 13:46:58 2010
Return-Path: <common-issues-return-7199-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13228 invoked from network); 6 Apr 2010 13:46:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 13:46:57 -0000
Received: (qmail 63669 invoked by uid 500); 6 Apr 2010 13:46:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63459 invoked by uid 500); 6 Apr 2010 13:46:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63354 invoked by uid 99); 6 Apr 2010 13:46:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 13:46:56 +0000
X-ASF-Spam-Status: No, hits=-1228.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 13:46:55 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 863EE234C4B2
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 13:46:34 +0000 (UTC)
Message-ID: <1775798644.10141270561594548.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 13:46:34 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6421) Symbolic links
In-Reply-To: <789869504.1260323298357.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853933#action_12853933 ] 

Hudson commented on HADOOP-6421:
--------------------------------

Integrated in Hadoop-Hdfs-trunk #275 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/275/])
    

> Symbolic links
> --------------
>
>                 Key: HADOOP-6421
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6421
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>         Attachments: symlink-25-common.patch, symlink-26-common.patch, symlink-26-common.patch, symlink24-common.patch, symlink27-common.patch, symlink28-common.patch, symlink29-common.patch, symlink29-common.patch, symlink29-common.patch, symlink30-common.patch, symlink31-common.patch, symlink32-common.patch, symlink33-common.patch, symlink34-common.patch, symlink35-common.patch, symlink36-common.patch, symlink37-common.patch, symlink38-common.patch, symlink39-common.patch
>
>
> Here's a jira for the common parts of HDFS-245, mostly changes to FileContext and AbstractFileSystem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7200-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 13:47:20 2010
Return-Path: <common-issues-return-7200-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13871 invoked from network); 6 Apr 2010 13:47:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 13:47:20 -0000
Received: (qmail 65429 invoked by uid 500); 6 Apr 2010 13:47:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65327 invoked by uid 500); 6 Apr 2010 13:47:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65310 invoked by uid 99); 6 Apr 2010 13:47:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 13:47:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 13:47:17 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 800FD234C4CB
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 13:46:36 +0000 (UTC)
Message-ID: <812155180.11261270561596523.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 13:46:36 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6569) FsShell#cat should avoid calling
 unecessary getFileStatus before opening a file to read
In-Reply-To: <1333327371.313901266350847909.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853946#action_12853946 ] 

Hudson commented on HADOOP-6569:
--------------------------------

Integrated in Hadoop-Hdfs-trunk #275 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/275/])
    

> FsShell#cat should avoid calling unecessary getFileStatus before opening a file to read
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6569
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6569
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: optimizeCat-yahoo.patch, optimizeCat-yahoo1.patch, optimizeCat-yahoo2.patch, optimizeCat.patch, optimizeCat.patch
>
>
> Since FileSystem#open throws a FileNotFoundException when the file to be read does not exist, there is no need to check if the file is a directory or not before open. In case of HDFS, this could reduce one getFileInfo RPC to NameNode.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7201-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 13:47:44 2010
Return-Path: <common-issues-return-7201-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14948 invoked from network); 6 Apr 2010 13:47:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 13:47:43 -0000
Received: (qmail 67761 invoked by uid 500); 6 Apr 2010 13:47:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67669 invoked by uid 500); 6 Apr 2010 13:47:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67594 invoked by uid 99); 6 Apr 2010 13:47:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 13:47:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 13:47:39 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C8646234C51F
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 13:46:38 +0000 (UTC)
Message-ID: <404607948.12471270561598820.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 13:46:38 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6579) A utility for reading and writing
 tokens into a URL safe string.
In-Reply-To: <843427920.414981266707188674.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853966#action_12853966 ] 

Hudson commented on HADOOP-6579:
--------------------------------

Integrated in Hadoop-Hdfs-trunk #275 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/275/])
    

> A utility for reading and writing tokens into a URL safe string.
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6579
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6579
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.22.0
>
>         Attachments: c-6579-hdfs.patch, c-6579-mr.patch, c-6579.patch, c-6579.patch, c-6579.patch, c-6579.patch, c-6579.patch
>
>
> We need to include HDFS delegation tokens in the URLs while browsing the file system. Therefore, we need a url-safe way to encode and decode them.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7202-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 13:56:55 2010
Return-Path: <common-issues-return-7202-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18671 invoked from network); 6 Apr 2010 13:56:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 13:56:55 -0000
Received: (qmail 80757 invoked by uid 500); 6 Apr 2010 13:56:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80709 invoked by uid 500); 6 Apr 2010 13:56:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80701 invoked by uid 99); 6 Apr 2010 13:56:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 13:56:54 +0000
X-ASF-Spam-Status: No, hits=-1229.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 13:56:53 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id AD842234C4AB
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 13:56:33 +0000 (UTC)
Message-ID: <1339639092.13171270562193709.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 13:56:33 +0000 (UTC)
From: "Xiao Kang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6663) BlockDecompressorStream get EOF
 exception when decompressing the file compressed from empty file
In-Reply-To: <1619883340.543171269850767204.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xiao Kang updated HADOOP-6663:
------------------------------

    Release Note: Fix EOF exception in BlockDecompressorStream when decompressing previous compressed empty file
          Status: Patch Available  (was: Open)

> BlockDecompressorStream get EOF exception when decompressing the file compressed from empty file
> ------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6663
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6663
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Xiao Kang
>         Attachments: BlockDecompressorStream.java.patch, BlockDecompressorStream.java.patch, BlockDecompressorStream.patch
>
>
> An empty file can be compressed using BlockDecompressorStream, which is for block-based compressiong algorithm such as LZO. However, when decompressing the compressed file, BlockDecompressorStream get EOF exception.
> Here is a typical exception stack:
> java.io.EOFException
> at org.apache.hadoop.io.compress.BlockDecompressorStream.rawReadInt(BlockDecompressorStream.java:125)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:96)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:82)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at java.io.InputStream.read(InputStream.java:85)
> at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:39)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:186)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:170)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:18)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
> at org.apache.hadoop.mapred.Child.main(Child.java:196)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7203-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 14:21:05 2010
Return-Path: <common-issues-return-7203-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24933 invoked from network); 6 Apr 2010 14:21:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 14:21:05 -0000
Received: (qmail 14489 invoked by uid 500); 6 Apr 2010 14:21:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14428 invoked by uid 500); 6 Apr 2010 14:21:04 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14268 invoked by uid 99); 6 Apr 2010 14:21:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 14:21:04 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 14:21:01 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 7CD67234C1F2
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 14:20:40 +0000 (UTC)
Message-ID: <919316322.13621270563640510.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 14:20:40 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6663) BlockDecompressorStream get EOF
 exception when decompressing the file compressed from empty file
In-Reply-To: <1619883340.543171269850767204.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12853991#action_12853991 ] 

Hadoop QA commented on HADOOP-6663:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12440828/BlockDecompressorStream.java.patch
  against trunk revision 930096.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/443/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/443/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/443/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/443/console

This message is automatically generated.

> BlockDecompressorStream get EOF exception when decompressing the file compressed from empty file
> ------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6663
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6663
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Xiao Kang
>         Attachments: BlockDecompressorStream.java.patch, BlockDecompressorStream.java.patch, BlockDecompressorStream.patch
>
>
> An empty file can be compressed using BlockDecompressorStream, which is for block-based compressiong algorithm such as LZO. However, when decompressing the compressed file, BlockDecompressorStream get EOF exception.
> Here is a typical exception stack:
> java.io.EOFException
> at org.apache.hadoop.io.compress.BlockDecompressorStream.rawReadInt(BlockDecompressorStream.java:125)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:96)
> at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:82)
> at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
> at java.io.InputStream.read(InputStream.java:85)
> at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:134)
> at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:39)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:186)
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:170)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:18)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
> at org.apache.hadoop.mapred.Child.main(Child.java:196)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7204-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 15:41:55 2010
Return-Path: <common-issues-return-7204-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45405 invoked from network); 6 Apr 2010 15:41:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 15:41:55 -0000
Received: (qmail 67219 invoked by uid 500); 6 Apr 2010 15:41:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67145 invoked by uid 500); 6 Apr 2010 15:41:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67099 invoked by uid 99); 6 Apr 2010 15:41:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 15:41:55 +0000
X-ASF-Spam-Status: No, hits=-1230.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 15:41:54 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E6C15234C4C9
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 15:41:33 +0000 (UTC)
Message-ID: <1658399483.16001270568493944.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 15:41:33 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5670) Hadoop configurations should be
 read from a distributed system
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854023#action_12854023 ] 

Allen Wittenauer commented on HADOOP-5670:
------------------------------------------

Considering that the task tracker talks to the job tracker (where the scheduler runs), the dynamic add/join is already there.

I want to re-iterate something here:  Think about the client, not just the server!  Chances are good the clients are NOT owned by the same entity that runs the Hadoop grid.  Building something that is strictly usable by the server is not going to be that much in terms of bang-for-the-buck, considering any reasonable configuration management tool could do that. 

... and given the various private convos I've had about this, I'm still not convinced ZK is a reasonable answer.  Sure you can probably do more with it, but that comes at a cost, especially on the operations side where there is (highly likely) already a DS in place.  It just isn't practical to require ZK for this.

> Hadoop configurations should be read from a distributed system
> --------------------------------------------------------------
>
>                 Key: HADOOP-5670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5670
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: conf
>            Reporter: Allen Wittenauer
>
> Rather than distributing the hadoop configuration files to every data node, compute node, etc, Hadoop should be able to read configuration information (dynamically!) from LDAP, ZooKeeper, whatever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7205-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 16:01:59 2010
Return-Path: <common-issues-return-7205-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52178 invoked from network); 6 Apr 2010 16:01:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 16:01:59 -0000
Received: (qmail 11540 invoked by uid 500); 6 Apr 2010 16:01:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11510 invoked by uid 500); 6 Apr 2010 16:01:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11502 invoked by uid 99); 6 Apr 2010 16:01:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 16:01:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 16:01:55 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 467CC234C4CD
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 16:01:34 +0000 (UTC)
Message-ID: <1562177850.16741270569694287.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 16:01:34 +0000 (UTC)
From: "Patrick Hunt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5670) Hadoop configurations should be
 read from a distributed system
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854033#action_12854033 ] 

Patrick Hunt commented on HADOOP-5670:
--------------------------------------

Allen and I had a conversation about this a while ago and he made some very good points for storing certain data
in a DS vs ZK. In particular user data makes alot of sense imo to be stored in DS. Data. Keep in mind that
ZK is all about coordination, not "data storage". We don't support search for example, which is a significant 
feature in most DSs. Also integration with legacy systems (your existing user database) is also a feature
of most DSs that ZK does not have. While ZK could do these things, a typical
DS will do them for you out of the box, and make your admin's lives easier in the sense that they already
have experience with this.

At the same time things like coordination are best served by ZK. Keeping track of which nodes are allocated
to which functions, the status of processes and coordinating operations between them, the load and activity 
of processes (nodes), Leader election within a highly reliable/available service, distributed locks and work
queues, etc...

Take a look at LinkedIn's Norbert for an example of one instantiation of something like this: http://bit.ly/6OQhwe

> Hadoop configurations should be read from a distributed system
> --------------------------------------------------------------
>
>                 Key: HADOOP-5670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5670
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: conf
>            Reporter: Allen Wittenauer
>
> Rather than distributing the hadoop configuration files to every data node, compute node, etc, Hadoop should be able to read configuration information (dynamically!) from LDAP, ZooKeeper, whatever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7206-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 16:31:56 2010
Return-Path: <common-issues-return-7206-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57727 invoked from network); 6 Apr 2010 16:31:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 16:31:56 -0000
Received: (qmail 38031 invoked by uid 500); 6 Apr 2010 16:31:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37986 invoked by uid 500); 6 Apr 2010 16:31:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37978 invoked by uid 99); 6 Apr 2010 16:31:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 16:31:56 +0000
X-ASF-Spam-Status: No, hits=-1230.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 16:31:55 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 73D97234C4CB
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 16:31:35 +0000 (UTC)
Message-ID: <1301946846.17801270571495473.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 16:31:35 +0000 (UTC)
From: "Alex Newman (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5670) Hadoop configurations should be
 read from a distributed system
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854046#action_12854046 ] 

Alex Newman commented on HADOOP-5670:
-------------------------------------

So I think their exists a couple of different conversations going on here. I suppose if all we wish is to replace basically the -D options passed into hadoop, then I am very openminded to a ldap based scheme as a plugin. I would be very happy if it included some scoping so that configuration could be provided on a intra cluster, inter cluster,  host ,user or job level. Guidelines should be provided as how not to disrupt existing ldap users. 

However their exists numerous other things which fall into the cluster configuration in which ldap may not be the best choice. 

In regards to operations, I think deployment complexity is an argument against ldap. Even if an organization were willing to store such configuration in their existing ldap infrastructure, which I am skeptical of in many cases, it would require organizations not using ldap to run additional infrastructure to solve a problem which could be solved by something already provided by the hadoop infrastructure. I would feel as though if we saw many users deploying a isolated ldap installation just for hadoop that we would have made people's life harder not easier.

Allen, sorry to be a pain , but could you be a bit more pedantic about what  you mean by configurations ? Do you think a hybrid system may make sense here? 

> Hadoop configurations should be read from a distributed system
> --------------------------------------------------------------
>
>                 Key: HADOOP-5670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5670
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: conf
>            Reporter: Allen Wittenauer
>
> Rather than distributing the hadoop configuration files to every data node, compute node, etc, Hadoop should be able to read configuration information (dynamically!) from LDAP, ZooKeeper, whatever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7207-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 16:49:57 2010
Return-Path: <common-issues-return-7207-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60575 invoked from network); 6 Apr 2010 16:49:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 16:49:57 -0000
Received: (qmail 53812 invoked by uid 500); 6 Apr 2010 16:49:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53706 invoked by uid 500); 6 Apr 2010 16:49:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53698 invoked by uid 99); 6 Apr 2010 16:49:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 16:49:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 16:49:54 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 79F40234C4A8
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 16:49:33 +0000 (UTC)
Message-ID: <1513256943.18131270572573498.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 16:49:33 +0000 (UTC)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6683) the first optimization:
 ZlibCompressor does not fully utilize the buffer
In-Reply-To: <2121624364.706511270518987284.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854054#action_12854054 ] 

Todd Lipcon commented on HADOOP-6683:
-------------------------------------

Hi Xiao,

Do you have any benchmarks on this? Would be interesting to see.

-Todd

> the first optimization: ZlibCompressor does not fully utilize the buffer
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6683
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6683
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Xiao Kang
>         Attachments: ZlibCompressor.java.patch
>
>
> Thanks for Hong Tang's advice.
> Sub task created for the first optimization. HADOOP-6662 closed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7208-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 17:07:55 2010
Return-Path: <common-issues-return-7208-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68010 invoked from network); 6 Apr 2010 17:07:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 17:07:55 -0000
Received: (qmail 82832 invoked by uid 500); 6 Apr 2010 17:07:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82801 invoked by uid 500); 6 Apr 2010 17:07:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82793 invoked by uid 99); 6 Apr 2010 17:07:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 17:07:55 +0000
X-ASF-Spam-Status: No, hits=-1230.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 17:07:54 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 2B1FA234C48C
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 17:07:34 +0000 (UTC)
Message-ID: <595744630.18631270573654175.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 17:07:34 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6680) hadoop-cloud push command invokes
 proxy creation
In-Reply-To: <1629624146.687721270478607313.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6680:
------------------------------

       Resolution: Fixed
    Fix Version/s: 0.22.0
         Assignee: Andrew Klochkov
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I've just committed this. Thanks Andrew!

> hadoop-cloud push command invokes proxy creation
> ------------------------------------------------
>
>                 Key: HADOOP-6680
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6680
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: contrib/cloud
>            Reporter: Andrew Klochkov
>            Assignee: Andrew Klochkov
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6680.patch
>
>
> There's a typo in src/contrib/cloud/src/py/hadoop/cloud/cli.py: when "push" command is invoked, the script actually invokes proxy creation instead of copying files. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7209-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 17:15:55 2010
Return-Path: <common-issues-return-7209-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70318 invoked from network); 6 Apr 2010 17:15:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 17:15:55 -0000
Received: (qmail 92443 invoked by uid 500); 6 Apr 2010 17:15:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92405 invoked by uid 500); 6 Apr 2010 17:15:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92397 invoked by uid 99); 6 Apr 2010 17:15:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 17:15:54 +0000
X-ASF-Spam-Status: No, hits=-1230.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 17:15:54 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 11C52234C48C
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 17:15:34 +0000 (UTC)
Message-ID: <1865717367.18751270574134071.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 17:15:34 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6684) Create a wrapper for TFile that
 handles generic serialization
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Create a wrapper for TFile that handles generic serialization
-------------------------------------------------------------

                 Key: HADOOP-6684
                 URL: https://issues.apache.org/jira/browse/HADOOP-6684
             Project: Hadoop Common
          Issue Type: New Feature
            Reporter: Owen O'Malley
            Assignee: Owen O'Malley


It would be good to have an OFile class that reads and writes serialized objects from TFiles.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7210-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 17:21:54 2010
Return-Path: <common-issues-return-7210-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71718 invoked from network); 6 Apr 2010 17:21:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 17:21:54 -0000
Received: (qmail 99461 invoked by uid 500); 6 Apr 2010 17:21:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99424 invoked by uid 500); 6 Apr 2010 17:21:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99416 invoked by uid 99); 6 Apr 2010 17:21:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 17:21:54 +0000
X-ASF-Spam-Status: No, hits=-1231.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 17:21:53 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 7EEEB234C4AC
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 17:21:33 +0000 (UTC)
Message-ID: <1988527879.18861270574493519.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 17:21:33 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6680) hadoop-cloud push command invokes
 proxy creation
In-Reply-To: <1629624146.687721270478607313.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854072#action_12854072 ] 

Hudson commented on HADOOP-6680:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #214 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/214/])
    . hadoop-cloud push command invokes proxy creation. Contributed by Andrew Klochkov.


> hadoop-cloud push command invokes proxy creation
> ------------------------------------------------
>
>                 Key: HADOOP-6680
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6680
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: contrib/cloud
>            Reporter: Andrew Klochkov
>            Assignee: Andrew Klochkov
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6680.patch
>
>
> There's a typo in src/contrib/cloud/src/py/hadoop/cloud/cli.py: when "push" command is invoked, the script actually invokes proxy creation instead of copying files. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7211-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 17:25:58 2010
Return-Path: <common-issues-return-7211-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73078 invoked from network); 6 Apr 2010 17:25:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 17:25:58 -0000
Received: (qmail 4921 invoked by uid 500); 6 Apr 2010 17:25:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4862 invoked by uid 500); 6 Apr 2010 17:25:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4617 invoked by uid 99); 6 Apr 2010 17:25:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 17:25:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 17:25:54 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 9A195234C48C
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 17:25:33 +0000 (UTC)
Message-ID: <328977938.18921270574733630.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 17:25:33 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6685) Change the generic serialization
 framework API to use serialization-specific bytes instead of
 Map<String,String> for configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Change the generic serialization framework API to use serialization-specific bytes instead of Map<String,String> for configuration
----------------------------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-6685
                 URL: https://issues.apache.org/jira/browse/HADOOP-6685
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Owen O'Malley
            Assignee: Owen O'Malley


Currently, the generic serialization framework uses Map<String,String> for the serialization specific configuration. Since this data is really internal to the specific serialization, I think we should change it to be an opaque binary blob. This will simplify the interface for defining specific serializations for different contexts (MAPREDUCE-1462). It will also move us toward having serialized objects for Mappers, Reducers, etc (MAPREDUCE-1183).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7212-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 17:53:56 2010
Return-Path: <common-issues-return-7212-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80360 invoked from network); 6 Apr 2010 17:53:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 17:53:56 -0000
Received: (qmail 59088 invoked by uid 500); 6 Apr 2010 17:53:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59060 invoked by uid 500); 6 Apr 2010 17:53:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59052 invoked by uid 99); 6 Apr 2010 17:53:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 17:53:56 +0000
X-ASF-Spam-Status: No, hits=-1231.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 17:53:54 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DEE46234C4D8
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 17:53:34 +0000 (UTC)
Message-ID: <1371339222.19951270576414912.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 17:53:34 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5670) Hadoop configurations should be
 read from a distributed system
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854083#action_12854083 ] 

Allen Wittenauer commented on HADOOP-5670:
------------------------------------------

bq. it would require organizations not using ldap to run additional infrastructure to solve a problem which could be solved by something already provided by the hadoop infrastructure. I would feel as though if we saw many users deploying a isolated ldap installation just for hadoop that we would have made people's life harder not easier.

If you s,LDAP,ZK,g above, you'll find it is a better fit.  The cold, hard reality is that LDAP is everywhere, ZK is not.  There are some key features in ZK that are/were missing  in order for it to fit here (with the good news being that those gaps are slowly closing).  But the truth of the matter is that LDAP is a well understood technology by most IT departments and ZK is not.  (Sidenote: it would be interesting to know how well used the security components of ZK are...)

Also, I don't think you should think of ZK as part of the Hadoop infrastructure.  It is a sub project (and therefore part of the ecosystem), but you can run Hadoop without using ZK and many many many people do, including Yahoo! for years. 

bq. Allen, sorry to be a pain , but could you be a bit more pedantic about what you mean by configurations ? Do you think a hybrid system may make sense here? 

I'll try and write/diagram something up with a concrete proposal as to how I think this should be done, based on conversations I've had with Owen and others over the years.  You'll find I'm thinking way beyond just a simple 10 node grid. :)


> Hadoop configurations should be read from a distributed system
> --------------------------------------------------------------
>
>                 Key: HADOOP-5670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5670
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: conf
>            Reporter: Allen Wittenauer
>
> Rather than distributing the hadoop configuration files to every data node, compute node, etc, Hadoop should be able to read configuration information (dynamically!) from LDAP, ZooKeeper, whatever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7213-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 18:15:58 2010
Return-Path: <common-issues-return-7213-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85304 invoked from network); 6 Apr 2010 18:15:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 18:15:58 -0000
Received: (qmail 85884 invoked by uid 500); 6 Apr 2010 18:15:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85821 invoked by uid 500); 6 Apr 2010 18:15:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85673 invoked by uid 99); 6 Apr 2010 18:15:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 18:15:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 18:15:54 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id AE67C234C4B9
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 18:15:33 +0000 (UTC)
Message-ID: <1957316386.20691270577733713.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 18:15:33 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6681) Fill AWS credentials when
 configuring Hadoop on EC2 instances
In-Reply-To: <315596123.688091270479567264.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854098#action_12854098 ] 

Tom White commented on HADOOP-6681:
-----------------------------------

Passing these credentials automatically may not be what the user wants, and may in fact be a security liability (since the credentials are passed to the cluster, which may be shared with other users). You can achieve the same result explicitly with the following command line arguments:

{code}
--env AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \ 
--env AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
{code}

Does this solve your issue?

> Fill AWS credentials when configuring Hadoop on EC2 instances
> -------------------------------------------------------------
>
>                 Key: HADOOP-6681
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6681
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: contrib/cloud
>            Reporter: Andrew Klochkov
>         Attachments: HADOOP-6681.patch
>
>
> There's a function "configure_hadoop" in the hadoop-ec2-init-remote.sh script used to configure EC2 nodes for Hadoop. The function actually uses AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, but they are never passed to it. It can be fixed in service.py by passing those variables.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7214-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 18:19:55 2010
Return-Path: <common-issues-return-7214-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86235 invoked from network); 6 Apr 2010 18:19:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 18:19:55 -0000
Received: (qmail 93612 invoked by uid 500); 6 Apr 2010 18:19:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93577 invoked by uid 500); 6 Apr 2010 18:19:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93560 invoked by uid 99); 6 Apr 2010 18:19:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 18:19:54 +0000
X-ASF-Spam-Status: No, hits=-1231.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 18:19:53 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 7F673234C4A8
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 18:19:33 +0000 (UTC)
Message-ID: <592319741.20791270577973520.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 18:19:33 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6658) Exclude Private elements from
 generated Javadoc
In-Reply-To: <995349254.452871269404067173.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854100#action_12854100 ] 

Tom White commented on HADOOP-6658:
-----------------------------------

Test failures are unrelated. I think this is ready to be committed.

> Exclude Private elements from generated Javadoc
> -----------------------------------------------
>
>                 Key: HADOOP-6658
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6658
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch
>
>
> Packages, classes and methods that are marked with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation should not appear in the public Javadoc. Developer Javadoc generated using the "javadoc-dev" ant target should continue to generate Javadoc for all program elements.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7215-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 18:19:57 2010
Return-Path: <common-issues-return-7215-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86309 invoked from network); 6 Apr 2010 18:19:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 18:19:57 -0000
Received: (qmail 93982 invoked by uid 500); 6 Apr 2010 18:19:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93942 invoked by uid 500); 6 Apr 2010 18:19:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93934 invoked by uid 99); 6 Apr 2010 18:19:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 18:19:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 18:19:54 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 8A16D234C4B3
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 18:19:33 +0000 (UTC)
Message-ID: <1188523401.20861270577973564.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 18:19:33 +0000 (UTC)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6679) add retry logic to
 NetUtils.createSocketAddr() when hostname resolution fails
In-Reply-To: <18346259.678881270363227220.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854102#action_12854102 ] 

sam rash commented on HADOOP-6679:
----------------------------------

1. ah yes, I see the --no-prefix option.  I'll update and post the patch

2. my understanding is the jvm caches on successful resolution, not on failure, so a retry should be useful
http://java.sun.com/javase/6/docs/technotes/guides/net/properties.html#nct
Do I misunderstand the docs?

3. you mean you want a port/service description passed in here?  

  public static InetSocketAddress createWithResolveRetry(
    String hostname,
    int port,
   String portDescription
  ) {
....

and I assume the NetUtils.createSocketAddr() would now have to take this?  It's current signature is

  public static InetSocketAddress createSocketAddr(String target,
                                                   int defaultPort) {

can you clarify what you mean by this?




> add retry logic to NetUtils.createSocketAddr() when hostname resolution fails
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-6679
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6679
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.0
>            Reporter: sam rash
>            Priority: Minor
>         Attachments: HADOOP-6679-patch-1.txt
>
>
> transient DNS errors can cause the InetSocketAddress created in NetUtils.createSocketAddr() to be unresolved when the hostname is resolvable.  Add a single retry after a short sleep using a simple InetSocketAddressFactory class

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7216-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 18:24:01 2010
Return-Path: <common-issues-return-7216-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87205 invoked from network); 6 Apr 2010 18:24:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 18:24:01 -0000
Received: (qmail 97673 invoked by uid 500); 6 Apr 2010 18:24:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97646 invoked by uid 500); 6 Apr 2010 18:24:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97638 invoked by uid 99); 6 Apr 2010 18:24:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 18:24:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 18:23:57 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0F0BA234C4D0
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 18:23:36 +0000 (UTC)
Message-ID: <297419593.21351270578216060.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 18:23:36 +0000 (UTC)
From: "Alex Newman (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5670) Hadoop configurations should be
 read from a distributed system
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854105#action_12854105 ] 

Alex Newman commented on HADOOP-5670:
-------------------------------------

Allen, A more concrete proposal would be great, I am just trying to understand which part of the problem we are trying to solve. It's good to see people putting some real elbow great in making hadoop more palatable to ops/system as they are a very important stakeholder. I do think that requiring sysadmins to understand the details of zk, just to do configuration stuff would be kindof a show stopper for acceptance. Also having cluster and host wide information in ldap definitely seems like a win in the portability department. The only real worry is that we end up increasing the deployment complexity or develope things in such a way that people have to deploy an array of ldap servers within each cluster. Considering I have never worked at any large enterprise  which uses directory services besides notes or active directory, I think that this could seriously be a burden if we do it wrong.

> Hadoop configurations should be read from a distributed system
> --------------------------------------------------------------
>
>                 Key: HADOOP-5670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5670
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: conf
>            Reporter: Allen Wittenauer
>
> Rather than distributing the hadoop configuration files to every data node, compute node, etc, Hadoop should be able to read configuration information (dynamically!) from LDAP, ZooKeeper, whatever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7217-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 18:25:55 2010
Return-Path: <common-issues-return-7217-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87886 invoked from network); 6 Apr 2010 18:25:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 18:25:54 -0000
Received: (qmail 1564 invoked by uid 500); 6 Apr 2010 18:25:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1523 invoked by uid 500); 6 Apr 2010 18:25:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1515 invoked by uid 99); 6 Apr 2010 18:25:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 18:25:54 +0000
X-ASF-Spam-Status: No, hits=-1231.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 18:25:53 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D99C0234C4AC
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 18:25:33 +0000 (UTC)
Message-ID: <556437840.21421270578333890.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 18:25:33 +0000 (UTC)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6679) add retry logic to
 NetUtils.createSocketAddr() when hostname resolution fails
In-Reply-To: <18346259.678881270363227220.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

sam rash updated HADOOP-6679:
-----------------------------

    Attachment: HADOOP-6679-patch-2.txt

this patch applies when i run

patch -p0 < HADOOP-6679-patch-2.txt

from the root of a hadoop checkout

please advise if I need to generate it in some other way


> add retry logic to NetUtils.createSocketAddr() when hostname resolution fails
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-6679
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6679
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.0
>            Reporter: sam rash
>            Priority: Minor
>         Attachments: HADOOP-6679-patch-1.txt, HADOOP-6679-patch-2.txt
>
>
> transient DNS errors can cause the InetSocketAddress created in NetUtils.createSocketAddr() to be unresolved when the hostname is resolvable.  Add a single retry after a short sleep using a simple InetSocketAddressFactory class

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7218-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 18:29:57 2010
Return-Path: <common-issues-return-7218-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94571 invoked from network); 6 Apr 2010 18:29:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 18:29:57 -0000
Received: (qmail 3818 invoked by uid 500); 6 Apr 2010 18:29:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3712 invoked by uid 500); 6 Apr 2010 18:29:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3704 invoked by uid 99); 6 Apr 2010 18:29:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 18:29:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 18:29:54 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D7AA9234C4CC
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 18:29:33 +0000 (UTC)
Message-ID: <343130937.21811270578573882.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 18:29:33 +0000 (UTC)
From: "Patrick Hunt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5670) Hadoop configurations should be
 read from a distributed system
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854108#action_12854108 ] 

Patrick Hunt commented on HADOOP-5670:
--------------------------------------

bq. would be interesting to know how well used the security components of ZK are

ZK has plugable auth with acls on the znodes. Yahoo uses this internally for multi-tenant ZK clusters, the auth
in this case is Yahoo specific implementation (uses certificates). We have met with the Yahoo Hadoop Security
team, we don't have a kerberos plugin for ZK auth yet but we have discussed it.

bq. you can run Hadoop without using ZK

that's true for MR/HDFS today, it may not be so in future.

HBase requires ZK today, so if you're using HBase you're already using ZK. ;-)

bq. LDAP is a well understood technology by most IT departments and ZK is not

I agree with this, in general most IT departments are going to be much more familiar with technologies such
as LDAP, MySQL, Exchange, etc... that have been around for a while. This is a big plus. ZK is still very new
relative to these mature technologies. (although the same could be said for Hadoop itself).


> Hadoop configurations should be read from a distributed system
> --------------------------------------------------------------
>
>                 Key: HADOOP-5670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5670
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: conf
>            Reporter: Allen Wittenauer
>
> Rather than distributing the hadoop configuration files to every data node, compute node, etc, Hadoop should be able to read configuration information (dynamically!) from LDAP, ZooKeeper, whatever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7219-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 18:55:57 2010
Return-Path: <common-issues-return-7219-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 189 invoked from network); 6 Apr 2010 18:55:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 18:55:57 -0000
Received: (qmail 26093 invoked by uid 500); 6 Apr 2010 18:55:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26064 invoked by uid 500); 6 Apr 2010 18:55:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26056 invoked by uid 99); 6 Apr 2010 18:55:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 18:55:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 18:55:55 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EF9B7234C4CC
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 18:55:33 +0000 (UTC)
Message-ID: <37723026.22931270580133980.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 18:55:33 +0000 (UTC)
From: "Mahadev konar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5670) Hadoop configurations should be
 read from a distributed system
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854121#action_12854121 ] 

Mahadev konar commented on HADOOP-5670:
---------------------------------------

Just my thought - I think ZooKeeper might become a part of Hadoop Deployment in somewhere near term (for High  Availability) or some other part of Hadoop. With HBase already deploying ZooKeeper as part of there stack, it might need a little more thinking in just outrightly rejecting the idea of ZooKeeper because of unfamiliarity to ops.




> Hadoop configurations should be read from a distributed system
> --------------------------------------------------------------
>
>                 Key: HADOOP-5670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5670
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: conf
>            Reporter: Allen Wittenauer
>
> Rather than distributing the hadoop configuration files to every data node, compute node, etc, Hadoop should be able to read configuration information (dynamically!) from LDAP, ZooKeeper, whatever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7220-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 19:05:55 2010
Return-Path: <common-issues-return-7220-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2858 invoked from network); 6 Apr 2010 19:05:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 19:05:55 -0000
Received: (qmail 34467 invoked by uid 500); 6 Apr 2010 19:05:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34430 invoked by uid 500); 6 Apr 2010 19:05:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34418 invoked by uid 99); 6 Apr 2010 19:05:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 19:05:55 +0000
X-ASF-Spam-Status: No, hits=-1232.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 19:05:54 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4D869234C4CA
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 19:05:34 +0000 (UTC)
Message-ID: <1562551390.23411270580734316.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 19:05:34 +0000 (UTC)
From: "Alex Newman (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5670) Hadoop configurations should be
 read from a distributed system
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854125#action_12854125 ] 

Alex Newman commented on HADOOP-5670:
-------------------------------------

I would say it this way:

- What's the right fit
- What role could ldap play
- What are the specific goals we are trying to solve
- How do we make ops/syseng guys comfortable with our decisions.
- How do we reduce operational complexity
- How do we reduce deployment complexity


> Hadoop configurations should be read from a distributed system
> --------------------------------------------------------------
>
>                 Key: HADOOP-5670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5670
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: conf
>            Reporter: Allen Wittenauer
>
> Rather than distributing the hadoop configuration files to every data node, compute node, etc, Hadoop should be able to read configuration information (dynamically!) from LDAP, ZooKeeper, whatever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7221-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 21:53:55 2010
Return-Path: <common-issues-return-7221-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37974 invoked from network); 6 Apr 2010 21:53:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 21:53:54 -0000
Received: (qmail 39275 invoked by uid 500); 6 Apr 2010 21:53:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39244 invoked by uid 500); 6 Apr 2010 21:53:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39236 invoked by uid 99); 6 Apr 2010 21:53:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 21:53:54 +0000
X-ASF-Spam-Status: No, hits=-1233.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 21:53:53 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 8FF02234C4AF
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 21:53:33 +0000 (UTC)
Message-ID: <214074546.27691270590813588.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 21:53:33 +0000 (UTC)
From: "David Ciemiewicz (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6335) Support reading of concatenated gzip
 and bzip2 files
In-Reply-To: <557853755.1256648579470.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

David Ciemiewicz updated HADOOP-6335:
-------------------------------------

    Summary: Support reading of concatenated gzip and bzip2 files  (was: Support reading of concatenated gzip files)

bzip2 also supports concatenation of individually compressed files.

bzcat has absolutely no problem reading all of the data in a single concatenated file.

Unfortuantely, M/R loses records.  My Hadoop Streaming and Pig jobs saw only 2% of the data in the concatenated file.  That's a 98% data loss.  Not very good.  There should be no data loss in reading these concatenated files.

> Support reading of concatenated gzip and bzip2 files
> ----------------------------------------------------
>
>                 Key: HADOOP-6335
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6335
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>
> GzipCodec.GzipInputStream needs to support reading of concatenated gzip files.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7222-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 22:03:57 2010
Return-Path: <common-issues-return-7222-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40195 invoked from network); 6 Apr 2010 22:03:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 22:03:56 -0000
Received: (qmail 50901 invoked by uid 500); 6 Apr 2010 22:03:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50873 invoked by uid 500); 6 Apr 2010 22:03:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50865 invoked by uid 99); 6 Apr 2010 22:03:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 22:03:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 22:03:54 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 87ABB234C48C
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 22:03:33 +0000 (UTC)
Message-ID: <70650712.27801270591413554.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 22:03:33 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6686) Remove redundant exception class name
 in unwrapped exceptions thrown at the RPC client
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Remove redundant exception class name in unwrapped exceptions thrown at the RPC client
--------------------------------------------------------------------------------------

                 Key: HADOOP-6686
                 URL: https://issues.apache.org/jira/browse/HADOOP-6686
             Project: Hadoop Common
          Issue Type: Improvement
         Environment: At RPC client, when the exception thrown by the server is unwrapped, the exception message includes redundant exception class name. This redundant information should be removed.
            Reporter: Suresh Srinivas
            Assignee: Suresh Srinivas
             Fix For: 0.22.0




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7223-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 22:07:57 2010
Return-Path: <common-issues-return-7223-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41621 invoked from network); 6 Apr 2010 22:07:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 22:07:57 -0000
Received: (qmail 54819 invoked by uid 500); 6 Apr 2010 22:07:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54789 invoked by uid 500); 6 Apr 2010 22:07:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54781 invoked by uid 99); 6 Apr 2010 22:07:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 22:07:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 22:07:54 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 796FA234C48C
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 22:07:33 +0000 (UTC)
Message-ID: <825076237.27831270591653496.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 22:07:33 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6686) Remove redundant exception class
 name in unwrapped exceptions thrown at the RPC client
In-Reply-To: <70650712.27801270591413554.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854217#action_12854217 ] 

Suresh Srinivas commented on HADOOP-6686:
-----------------------------------------

RPC server sends exception class name and printed stack trace in response. Proposed change:

# if the exception is unwrapped, with the message set to the first line of printed stack trace from response. The message is "<Server exception class name>: <server exception message>"
#* While unwrapping, I propose removing the redundant "<exception class name>: " from the message. Other than HDFSCli test I cannot think of a reason why an application depends on the error string to include exception name, especially given that the exception type that is thrown has that information already.
#. if the exception is not unwrapped, RemoteException is thrown as it is, with printed stack trace as the message.
#* This behavior will be retained. The exception name here along with the stack trace (albeit the server side) is useful for debugging.

> Remove redundant exception class name in unwrapped exceptions thrown at the RPC client
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6686
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6686
>             Project: Hadoop Common
>          Issue Type: Improvement
>         Environment: At RPC client, when the exception thrown by the server is unwrapped, the exception message includes redundant exception class name. This redundant information should be removed.
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7224-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 22:13:59 2010
Return-Path: <common-issues-return-7224-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42393 invoked from network); 6 Apr 2010 22:13:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 22:13:59 -0000
Received: (qmail 61383 invoked by uid 500); 6 Apr 2010 22:13:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61328 invoked by uid 500); 6 Apr 2010 22:13:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61319 invoked by uid 99); 6 Apr 2010 22:13:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 22:13:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 22:13:56 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D46E3234C48D
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 22:13:35 +0000 (UTC)
Message-ID: <1945721081.27911270592015868.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 22:13:35 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6686) Remove redundant exception class name
 in unwrapped exceptions thrown at the RPC client
In-Reply-To: <70650712.27801270591413554.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6686:
------------------------------------

    Attachment: HADOOP-6686.patch

> Remove redundant exception class name in unwrapped exceptions thrown at the RPC client
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6686
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6686
>             Project: Hadoop Common
>          Issue Type: Improvement
>         Environment: At RPC client, when the exception thrown by the server is unwrapped, the exception message includes redundant exception class name. This redundant information should be removed.
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6686.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7225-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 22:17:57 2010
Return-Path: <common-issues-return-7225-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43032 invoked from network); 6 Apr 2010 22:17:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 22:17:56 -0000
Received: (qmail 66179 invoked by uid 500); 6 Apr 2010 22:17:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66126 invoked by uid 500); 6 Apr 2010 22:17:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65877 invoked by uid 99); 6 Apr 2010 22:17:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 22:17:55 +0000
X-ASF-Spam-Status: No, hits=-1233.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 22:17:54 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 517F9234C48C
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 22:17:34 +0000 (UTC)
Message-ID: <191053771.27981270592254332.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 22:17:34 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6687) user object in the subject in UGI
  should be reused in case of a relogin.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

user object in the subject in UGI  should be reused in case of a relogin.
-------------------------------------------------------------------------

                 Key: HADOOP-6687
                 URL: https://issues.apache.org/jira/browse/HADOOP-6687
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Jitendra Nath Pandey


 In case of a relogin, a new user object is created in the subject. This causes the subject to lose some state for example authentication method. Instead, a new user object should not be added to the subject if one is already present.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7226-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 22:18:18 2010
Return-Path: <common-issues-return-7226-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43263 invoked from network); 6 Apr 2010 22:18:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 22:18:18 -0000
Received: (qmail 67453 invoked by uid 500); 6 Apr 2010 22:18:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67405 invoked by uid 500); 6 Apr 2010 22:18:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67397 invoked by uid 99); 6 Apr 2010 22:18:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 22:18:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 22:18:16 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 69FE5234C4B4
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 22:17:34 +0000 (UTC)
Message-ID: <1569201423.28091270592254433.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 22:17:34 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6675) Highlight Evolving elements in
 Javadoc
In-Reply-To: <358839696.663261270231228114.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854221#action_12854221 ] 

Tom White commented on HADOOP-6675:
-----------------------------------

I'm not sure we can use an annotation processor for this, since (by my reading) they process annotations that appear in the source being compiled, rather than annotations that are _referenced_ by the source being compiled.

Anyone got any suggestions for how this might be implemented?


> Highlight Evolving elements in Javadoc
> --------------------------------------
>
>                 Key: HADOOP-6675
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6675
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Tom White
>
> It would be nice if we could mark Evolving (and Unstable) program elements in bold as the first word of the Javadoc comment (like "Deprecated").

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7227-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 22:23:54 2010
Return-Path: <common-issues-return-7227-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44329 invoked from network); 6 Apr 2010 22:23:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 22:23:54 -0000
Received: (qmail 77120 invoked by uid 500); 6 Apr 2010 22:23:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77085 invoked by uid 500); 6 Apr 2010 22:23:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77077 invoked by uid 99); 6 Apr 2010 22:23:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 22:23:54 +0000
X-ASF-Spam-Status: No, hits=-1233.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 22:23:53 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 91184234C4AF
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 22:23:33 +0000 (UTC)
Message-ID: <897635840.28251270592613592.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 22:23:33 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6676) Create a javac plugin that produces
 warnings for programs using Evolving APIs
In-Reply-To: <1601380660.663491270231707182.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854224#action_12854224 ] 

Tom White commented on HADOOP-6676:
-----------------------------------

I'm not sure we can use an annotation processor for this, since (by my reading) they process annotations that appear in the source being compiled, rather than annotations that are _referenced_ by the source being compiled.

Anyone got any suggestions for how this might be implemented?

> Create a javac plugin that produces warnings for programs using Evolving APIs
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-6676
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6676
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Tom White
>
> Such a feature would be useful to enforce MAPREDUCE-1637. It might also be useful for users who wish to check their programs don't use Evolving APIs. 
> It could be implemented using an annotation processor: http://java.sun.com/javase/6/docs/technotes/tools/solaris/javac.html#processing

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7228-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 22:23:57 2010
Return-Path: <common-issues-return-7228-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44416 invoked from network); 6 Apr 2010 22:23:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 22:23:57 -0000
Received: (qmail 77314 invoked by uid 500); 6 Apr 2010 22:23:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77250 invoked by uid 500); 6 Apr 2010 22:23:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77218 invoked by uid 99); 6 Apr 2010 22:23:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 22:23:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 22:23:54 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 977B8234C4B3
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 22:23:33 +0000 (UTC)
Message-ID: <1273818309.28281270592613619.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 22:23:33 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6675) Highlight Evolving elements in
 Javadoc
In-Reply-To: <358839696.663261270231228114.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6675:
------------------------------

    Comment: was deleted

(was: I'm not sure we can use an annotation processor for this, since (by my reading) they process annotations that appear in the source being compiled, rather than annotations that are _referenced_ by the source being compiled.

Anyone got any suggestions for how this might be implemented?
)

> Highlight Evolving elements in Javadoc
> --------------------------------------
>
>                 Key: HADOOP-6675
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6675
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Tom White
>
> It would be nice if we could mark Evolving (and Unstable) program elements in bold as the first word of the Javadoc comment (like "Deprecated").

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7229-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 23:16:55 2010
Return-Path: <common-issues-return-7229-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53758 invoked from network); 6 Apr 2010 23:16:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 23:16:55 -0000
Received: (qmail 36512 invoked by uid 500); 6 Apr 2010 23:16:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36456 invoked by uid 500); 6 Apr 2010 23:16:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36442 invoked by uid 99); 6 Apr 2010 23:16:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 23:16:55 +0000
X-ASF-Spam-Status: No, hits=-1234.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 23:16:54 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E676B234C4AD
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 23:16:33 +0000 (UTC)
Message-ID: <1415657441.29551270595793943.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 23:16:33 +0000 (UTC)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6658) Exclude Private elements from
 generated Javadoc
In-Reply-To: <995349254.452871269404067173.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854243#action_12854243 ] 

Doug Cutting commented on HADOOP-6658:
--------------------------------------

+1 Looks good to me.  I also saw that you okayed this use of LGPL'd JDiff on legal-discuss@.


> Exclude Private elements from generated Javadoc
> -----------------------------------------------
>
>                 Key: HADOOP-6658
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6658
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch
>
>
> Packages, classes and methods that are marked with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation should not appear in the public Javadoc. Developer Javadoc generated using the "javadoc-dev" ant target should continue to generate Javadoc for all program elements.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7230-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 06 23:24:55 2010
Return-Path: <common-issues-return-7230-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56123 invoked from network); 6 Apr 2010 23:24:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Apr 2010 23:24:55 -0000
Received: (qmail 43847 invoked by uid 500); 6 Apr 2010 23:24:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43812 invoked by uid 500); 6 Apr 2010 23:24:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43791 invoked by uid 99); 6 Apr 2010 23:24:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 23:24:55 +0000
X-ASF-Spam-Status: No, hits=-1234.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 23:24:54 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 9F043234C4A8
	for <common-issues@hadoop.apache.org>; Tue,  6 Apr 2010 23:24:33 +0000 (UTC)
Message-ID: <198668007.29941270596273650.JavaMail.jira@brutus.apache.org>
Date: Tue, 6 Apr 2010 23:24:33 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6658) Exclude Private elements from
 generated Javadoc
In-Reply-To: <995349254.452871269404067173.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854248#action_12854248 ] 

Tom White commented on HADOOP-6658:
-----------------------------------

Yes, here's the relevant [thread|http://mail-archives.apache.org/mod_mbox/www-legal-discuss/201004.mbox/<l2zb23e01b51004011458s5019eb71s2b3d86583fa34ca3@mail.gmail.com>]

> Exclude Private elements from generated Javadoc
> -----------------------------------------------
>
>                 Key: HADOOP-6658
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6658
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch
>
>
> Packages, classes and methods that are marked with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation should not appear in the public Javadoc. Developer Javadoc generated using the "javadoc-dev" ant target should continue to generate Javadoc for all program elements.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7231-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 00:34:54 2010
Return-Path: <common-issues-return-7231-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73781 invoked from network); 7 Apr 2010 00:34:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 00:34:54 -0000
Received: (qmail 90605 invoked by uid 500); 7 Apr 2010 00:34:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90580 invoked by uid 500); 7 Apr 2010 00:34:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90572 invoked by uid 99); 7 Apr 2010 00:34:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 00:34:54 +0000
X-ASF-Spam-Status: No, hits=-1235.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 00:34:53 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 84912234C48C
	for <common-issues@hadoop.apache.org>; Wed,  7 Apr 2010 00:34:33 +0000 (UTC)
Message-ID: <915957858.31901270600473542.JavaMail.jira@brutus.apache.org>
Date: Wed, 7 Apr 2010 00:34:33 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6687) user object in the subject in UGI
  should be reused in case of a relogin.
In-Reply-To: <191053771.27981270592254332.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6687:
-----------------------------------------

    Attachment: HADOOP-6687-y20.2.patch

Patch for hadoop-20 added.

> user object in the subject in UGI  should be reused in case of a relogin.
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6687
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6687
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jitendra Nath Pandey
>         Attachments: HADOOP-6687-y20.2.patch
>
>
>  In case of a relogin, a new user object is created in the subject. This causes the subject to lose some state for example authentication method. Instead, a new user object should not be added to the subject if one is already present.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7232-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 00:40:55 2010
Return-Path: <common-issues-return-7232-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74816 invoked from network); 7 Apr 2010 00:40:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 00:40:55 -0000
Received: (qmail 93688 invoked by uid 500); 7 Apr 2010 00:40:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93651 invoked by uid 500); 7 Apr 2010 00:40:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93643 invoked by uid 99); 7 Apr 2010 00:40:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 00:40:55 +0000
X-ASF-Spam-Status: No, hits=-1235.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 00:40:54 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 10563234C48D
	for <common-issues@hadoop.apache.org>; Wed,  7 Apr 2010 00:40:34 +0000 (UTC)
Message-ID: <139504475.32091270600834065.JavaMail.jira@brutus.apache.org>
Date: Wed, 7 Apr 2010 00:40:34 +0000 (UTC)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6686) Remove redundant exception class
 name in unwrapped exceptions thrown at the RPC client
In-Reply-To: <70650712.27801270591413554.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854281#action_12854281 ] 

Doug Cutting commented on HADOOP-6686:
--------------------------------------

This is definitely an improvement, but I wonder if it might be better yet if RemoteException didn't include the exception name in its message.  It might include it in toString(), as is standard, but not in getMessage().  RemoteException has a field named "className" that already stores the remote exception name.  It also redundantly includes the exception name in the exceptions message, since that's created from the value of Throwable#printStackTrace().  Rather RemoteException might parseout just the message (the first line, after the class name) and use that as its message and separately store the remote stack trace in a field, like the class name.  Then, even if the exception is not unwrapped, its message and toString() would still match that of the original exception.

> Remove redundant exception class name in unwrapped exceptions thrown at the RPC client
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6686
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6686
>             Project: Hadoop Common
>          Issue Type: Improvement
>         Environment: At RPC client, when the exception thrown by the server is unwrapped, the exception message includes redundant exception class name. This redundant information should be removed.
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6686.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7233-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 03:32:55 2010
Return-Path: <common-issues-return-7233-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2226 invoked from network); 7 Apr 2010 03:32:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 03:32:55 -0000
Received: (qmail 13329 invoked by uid 500); 7 Apr 2010 03:32:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13227 invoked by uid 500); 7 Apr 2010 03:32:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13215 invoked by uid 99); 7 Apr 2010 03:32:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 03:32:54 +0000
X-ASF-Spam-Status: No, hits=-1235.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 03:32:53 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id A4293234C4BC
	for <common-issues@hadoop.apache.org>; Wed,  7 Apr 2010 03:32:33 +0000 (UTC)
Message-ID: <1931402982.34641270611153671.JavaMail.jira@brutus.apache.org>
Date: Wed, 7 Apr 2010 03:32:33 +0000 (UTC)
From: "Xiao Kang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6683) the first optimization:
 ZlibCompressor does not fully utilize the buffer
In-Reply-To: <2121624364.706511270518987284.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854316#action_12854316 ] 

Xiao Kang commented on HADOOP-6683:
-----------------------------------

A comparision test was performed on a 1.8GB web log file. The result is as follows:

|| read file buffer size || write to compress stream buffer size || old time(secs) || new time(secs) || decrease % ||
|65536|	100 |67|	49|	26.8%|
|65536|	200| 56.5|	46.5|	17.7%|
|65536|	400| 51.5|	45|	12.6%|
|65536|	800| 48.5|	44.5|	8.2%|
|65536|	1024|	46.8|	44.2|	9.8%|
|65536|	4096|	45|	43.5|	3.3%|
|65536|	65536|	44.6|	43.2|	3.1%|


Is there any standard benchmark for compression suitable for this case?

> the first optimization: ZlibCompressor does not fully utilize the buffer
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6683
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6683
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Xiao Kang
>         Attachments: ZlibCompressor.java.patch
>
>
> Thanks for Hong Tang's advice.
> Sub task created for the first optimization. HADOOP-6662 closed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7234-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 03:32:58 2010
Return-Path: <common-issues-return-7234-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2288 invoked from network); 7 Apr 2010 03:32:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 03:32:57 -0000
Received: (qmail 13424 invoked by uid 500); 7 Apr 2010 03:32:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13384 invoked by uid 500); 7 Apr 2010 03:32:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13375 invoked by uid 99); 7 Apr 2010 03:32:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 03:32:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 03:32:54 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 9E9FC234C4B8
	for <common-issues@hadoop.apache.org>; Wed,  7 Apr 2010 03:32:33 +0000 (UTC)
Message-ID: <1997280535.34601270611153648.JavaMail.jira@brutus.apache.org>
Date: Wed, 7 Apr 2010 03:32:33 +0000 (UTC)
From: "Paul Smith (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6671) To use maven for hadoop common
 builds
In-Reply-To: <859246128.630391270120227191.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854315#action_12854315 ] 

Paul Smith commented on HADOOP-6671:
------------------------------------

Lars beat me to the jira comment, so I'll just say "Yeah, what Lars said".

Happy to put my hand up to help, rather than a branch, I'd say a simply script like what was in HBASE-2099 is simple to work with for a reviewer, it outlines the migration steps needed rather than some hideous patch to review.

In regards to AllanW's comments on sync'ing things around, that is still possible, rather than sync'ng the .ivy directory it's just ensuring the ~/.m2/repository directories are in sync, and then working in Maven's offline mode if that server doesn't have internet connectivity.

Yes, one has to get the POM right, and that does come with experience, so perhaps if Lars and I can help here to get it off in the right direction that can ease any potential pain.    IntelliJ and Eclipse Maven support is now 1st-class citizens really, Ivy less so.    For future modularization, the Maven migration will pay off, splitting out code into nice modular chunks becomes much less work keeping the build system in sync.

Anyway, happy to help out here too.

> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7235-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 03:36:55 2010
Return-Path: <common-issues-return-7235-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2806 invoked from network); 7 Apr 2010 03:36:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 03:36:55 -0000
Received: (qmail 15945 invoked by uid 500); 7 Apr 2010 03:36:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15856 invoked by uid 500); 7 Apr 2010 03:36:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15605 invoked by uid 99); 7 Apr 2010 03:36:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 03:36:54 +0000
X-ASF-Spam-Status: No, hits=-1236.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 03:36:53 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 7F240234C4AB
	for <common-issues@hadoop.apache.org>; Wed,  7 Apr 2010 03:36:33 +0000 (UTC)
Message-ID: <1924885043.34701270611393519.JavaMail.jira@brutus.apache.org>
Date: Wed, 7 Apr 2010 03:36:33 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6677) InterfaceAudience.LimitedPrivate
 should take a string not an enum
In-Reply-To: <680532766.665461270239927272.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6677:
------------------------------

    Attachment: HADOOP-6677.patch

Here's a patch to change the enum to a String. 

> InterfaceAudience.LimitedPrivate should take a string not an enum
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6677
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6677
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Alan Gates
>            Priority: Minor
>         Attachments: HADOOP-6677.patch
>
>
> Trying to keep the list of all possible components sharing limited private interfaces up to date is painful.  As other subprojects beyond HDFS and MR use this interface it will only get worse (see PIG-1311).  If it is converted to a string other subprojects can use it without requiring patches to common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7236-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 03:38:58 2010
Return-Path: <common-issues-return-7236-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2973 invoked from network); 7 Apr 2010 03:38:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 03:38:57 -0000
Received: (qmail 16290 invoked by uid 500); 7 Apr 2010 03:38:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16217 invoked by uid 500); 7 Apr 2010 03:38:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16204 invoked by uid 99); 7 Apr 2010 03:38:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 03:38:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 03:38:54 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 9C594234C4AB
	for <common-issues@hadoop.apache.org>; Wed,  7 Apr 2010 03:38:33 +0000 (UTC)
Message-ID: <1919804992.34741270611513639.JavaMail.jira@brutus.apache.org>
Date: Wed, 7 Apr 2010 03:38:33 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6677) InterfaceAudience.LimitedPrivate
 should take a string not an enum
In-Reply-To: <680532766.665461270239927272.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6677:
------------------------------

    Assignee: Tom White
      Status: Patch Available  (was: Open)

> InterfaceAudience.LimitedPrivate should take a string not an enum
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6677
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6677
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Alan Gates
>            Assignee: Tom White
>            Priority: Minor
>         Attachments: HADOOP-6677.patch
>
>
> Trying to keep the list of all possible components sharing limited private interfaces up to date is painful.  As other subprojects beyond HDFS and MR use this interface it will only get worse (see PIG-1311).  If it is converted to a string other subprojects can use it without requiring patches to common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7237-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 03:48:56 2010
Return-Path: <common-issues-return-7237-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4452 invoked from network); 7 Apr 2010 03:48:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 03:48:56 -0000
Received: (qmail 21398 invoked by uid 500); 7 Apr 2010 03:48:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21362 invoked by uid 500); 7 Apr 2010 03:48:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21345 invoked by uid 99); 7 Apr 2010 03:48:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 03:48:54 +0000
X-ASF-Spam-Status: No, hits=-1236.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 03:48:53 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 9D5B7234C4B2
	for <common-issues@hadoop.apache.org>; Wed,  7 Apr 2010 03:48:33 +0000 (UTC)
Message-ID: <1833622004.34881270612113643.JavaMail.jira@brutus.apache.org>
Date: Wed, 7 Apr 2010 03:48:33 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6668:
------------------------------

    Issue Type: Sub-task  (was: Improvement)
        Parent: HADOOP-5064

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7238-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 03:50:57 2010
Return-Path: <common-issues-return-7238-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4593 invoked from network); 7 Apr 2010 03:50:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 03:50:57 -0000
Received: (qmail 23759 invoked by uid 500); 7 Apr 2010 03:50:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23571 invoked by uid 500); 7 Apr 2010 03:50:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23553 invoked by uid 99); 7 Apr 2010 03:50:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 03:50:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 03:50:54 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 90CF5234C4AE
	for <common-issues@hadoop.apache.org>; Wed,  7 Apr 2010 03:50:33 +0000 (UTC)
Message-ID: <45613504.34951270612233592.JavaMail.jira@brutus.apache.org>
Date: Wed, 7 Apr 2010 03:50:33 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6675) Highlight Evolving elements in
 Javadoc
In-Reply-To: <358839696.663261270231228114.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6675:
------------------------------

    Issue Type: Sub-task  (was: Improvement)
        Parent: HADOOP-5064

> Highlight Evolving elements in Javadoc
> --------------------------------------
>
>                 Key: HADOOP-6675
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6675
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>
> It would be nice if we could mark Evolving (and Unstable) program elements in bold as the first word of the Javadoc comment (like "Deprecated").

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7239-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 03:52:58 2010
Return-Path: <common-issues-return-7239-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4908 invoked from network); 7 Apr 2010 03:52:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 03:52:58 -0000
Received: (qmail 25827 invoked by uid 500); 7 Apr 2010 03:52:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25731 invoked by uid 500); 7 Apr 2010 03:52:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25723 invoked by uid 99); 7 Apr 2010 03:52:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 03:52:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 03:52:54 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 8EB39234C4AC
	for <common-issues@hadoop.apache.org>; Wed,  7 Apr 2010 03:52:33 +0000 (UTC)
Message-ID: <1737934621.35001270612353583.JavaMail.jira@brutus.apache.org>
Date: Wed, 7 Apr 2010 03:52:33 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6658) Exclude Private elements from
 generated Javadoc
In-Reply-To: <995349254.452871269404067173.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6658:
------------------------------

         Priority: Blocker  (was: Major)
    Fix Version/s: 0.21.0

> Exclude Private elements from generated Javadoc
> -----------------------------------------------
>
>                 Key: HADOOP-6658
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6658
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch
>
>
> Packages, classes and methods that are marked with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation should not appear in the public Javadoc. Developer Javadoc generated using the "javadoc-dev" ant target should continue to generate Javadoc for all program elements.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7240-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 03:54:56 2010
Return-Path: <common-issues-return-7240-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5089 invoked from network); 7 Apr 2010 03:54:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 03:54:56 -0000
Received: (qmail 27272 invoked by uid 500); 7 Apr 2010 03:54:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27208 invoked by uid 500); 7 Apr 2010 03:54:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27188 invoked by uid 99); 7 Apr 2010 03:54:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 03:54:55 +0000
X-ASF-Spam-Status: No, hits=-1236.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 03:54:53 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 95C98234C4AE
	for <common-issues@hadoop.apache.org>; Wed,  7 Apr 2010 03:54:33 +0000 (UTC)
Message-ID: <1652417773.35071270612473612.JavaMail.jira@brutus.apache.org>
Date: Wed, 7 Apr 2010 03:54:33 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6668:
------------------------------

         Priority: Blocker  (was: Major)
    Fix Version/s: 0.21.0

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7241-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 03:56:55 2010
Return-Path: <common-issues-return-7241-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5293 invoked from network); 7 Apr 2010 03:56:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 03:56:55 -0000
Received: (qmail 27990 invoked by uid 500); 7 Apr 2010 03:56:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27915 invoked by uid 500); 7 Apr 2010 03:56:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27905 invoked by uid 99); 7 Apr 2010 03:56:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 03:56:54 +0000
X-ASF-Spam-Status: No, hits=-1236.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 03:56:53 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 90B42234C4AE
	for <common-issues@hadoop.apache.org>; Wed,  7 Apr 2010 03:56:33 +0000 (UTC)
Message-ID: <2121682951.35141270612593591.JavaMail.jira@brutus.apache.org>
Date: Wed, 7 Apr 2010 03:56:33 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6677) InterfaceAudience.LimitedPrivate
 should take a string not an enum
In-Reply-To: <680532766.665461270239927272.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854322#action_12854322 ] 

Hadoop QA commented on HADOOP-6677:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12440988/HADOOP-6677.patch
  against trunk revision 931226.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/444/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/444/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/444/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/444/console

This message is automatically generated.

> InterfaceAudience.LimitedPrivate should take a string not an enum
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6677
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6677
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Alan Gates
>            Assignee: Tom White
>            Priority: Minor
>         Attachments: HADOOP-6677.patch
>
>
> Trying to keep the list of all possible components sharing limited private interfaces up to date is painful.  As other subprojects beyond HDFS and MR use this interface it will only get worse (see PIG-1311).  If it is converted to a string other subprojects can use it without requiring patches to common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7242-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 03:58:59 2010
Return-Path: <common-issues-return-7242-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5610 invoked from network); 7 Apr 2010 03:58:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 03:58:58 -0000
Received: (qmail 29180 invoked by uid 500); 7 Apr 2010 03:58:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29074 invoked by uid 500); 7 Apr 2010 03:58:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29066 invoked by uid 99); 7 Apr 2010 03:58:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 03:58:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 03:58:55 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DE618234C4AE
	for <common-issues@hadoop.apache.org>; Wed,  7 Apr 2010 03:58:34 +0000 (UTC)
Message-ID: <1114440895.35211270612714910.JavaMail.jira@brutus.apache.org>
Date: Wed, 7 Apr 2010 03:58:34 +0000 (UTC)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6677) InterfaceAudience.LimitedPrivate
 should take a string not an enum
In-Reply-To: <680532766.665461270239927272.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854323#action_12854323 ] 

Jakob Homan commented on HADOOP-6677:
-------------------------------------

This would be a pretty big change to the classification scheme that was argued out quite exhaustively in HADOOP-5073; there should be time for people to re-hash if they wish before anything is committed.  Nicholas had made the point there that having an enum avoids problems with typos and such, but this is a minor benefit.  If we're removing the limitation of who should use LimitedPrivate, and who can depend on it, then it's not really limited private anymore?

> InterfaceAudience.LimitedPrivate should take a string not an enum
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6677
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6677
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Alan Gates
>            Assignee: Tom White
>            Priority: Minor
>         Attachments: HADOOP-6677.patch
>
>
> Trying to keep the list of all possible components sharing limited private interfaces up to date is painful.  As other subprojects beyond HDFS and MR use this interface it will only get worse (see PIG-1311).  If it is converted to a string other subprojects can use it without requiring patches to common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7243-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 04:00:58 2010
Return-Path: <common-issues-return-7243-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6439 invoked from network); 7 Apr 2010 04:00:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 04:00:58 -0000
Received: (qmail 30981 invoked by uid 500); 7 Apr 2010 04:00:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30490 invoked by uid 500); 7 Apr 2010 04:00:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30399 invoked by uid 99); 7 Apr 2010 04:00:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 04:00:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 04:00:55 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 8C6CE234C4AF
	for <common-issues@hadoop.apache.org>; Wed,  7 Apr 2010 04:00:33 +0000 (UTC)
Message-ID: <781891790.35291270612833573.JavaMail.jira@brutus.apache.org>
Date: Wed, 7 Apr 2010 04:00:33 +0000 (UTC)
From: "Xiao Kang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6687) user object in the subject in UGI
  should be reused in case of a relogin.
In-Reply-To: <191053771.27981270592254332.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854325#action_12854325 ] 

Xiao Kang commented on HADOOP-6687:
-----------------------------------

Which revision is the patch against? Since no such method "public boolean commit() throws LoginException" found in http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20/src/core/org/apache/hadoop/security/UserGroupInformation.java.

> user object in the subject in UGI  should be reused in case of a relogin.
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6687
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6687
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jitendra Nath Pandey
>         Attachments: HADOOP-6687-y20.2.patch
>
>
>  In case of a relogin, a new user object is created in the subject. This causes the subject to lose some state for example authentication method. Instead, a new user object should not be added to the subject if one is already present.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7244-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 04:55:58 2010
Return-Path: <common-issues-return-7244-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12514 invoked from network); 7 Apr 2010 04:55:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 04:55:58 -0000
Received: (qmail 69316 invoked by uid 500); 7 Apr 2010 04:55:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69154 invoked by uid 500); 7 Apr 2010 04:55:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69136 invoked by uid 99); 7 Apr 2010 04:55:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 04:55:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 04:55:55 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 830AA234C4B2
	for <common-issues@hadoop.apache.org>; Wed,  7 Apr 2010 04:55:34 +0000 (UTC)
Message-ID: <697071094.36141270616134535.JavaMail.jira@brutus.apache.org>
Date: Wed, 7 Apr 2010 04:55:34 +0000 (UTC)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6674) Performance Improvement in Secure
 RPC
In-Reply-To: <375873020.645441270163787224.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854344#action_12854344 ] 

Owen O'Malley commented on HADOOP-6674:
---------------------------------------

This looks good.

> Performance Improvement in Secure RPC
> -------------------------------------
>
>                 Key: HADOOP-6674
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6674
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6674-y20.1.patch
>
>
> This jira introduces two performance improvements in Sasl RPC
> 1. Setting of Sasl 'quality of protection' to authentication only.
> 2. Addition of BufferedOutputStream underneath SaslOutputStream.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7245-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 07:59:56 2010
Return-Path: <common-issues-return-7245-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46166 invoked from network); 7 Apr 2010 07:59:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 07:59:56 -0000
Received: (qmail 25366 invoked by uid 500); 7 Apr 2010 07:59:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25254 invoked by uid 500); 7 Apr 2010 07:59:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25237 invoked by uid 99); 7 Apr 2010 07:59:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 07:59:55 +0000
X-ASF-Spam-Status: No, hits=-1237.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 07:59:54 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 78FA3234C4AB
	for <common-issues@hadoop.apache.org>; Wed,  7 Apr 2010 07:59:33 +0000 (UTC)
Message-ID: <1780197215.37411270627173494.JavaMail.jira@brutus.apache.org>
Date: Wed, 7 Apr 2010 07:59:33 +0000 (UTC)
From: "Danny Leshem (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6688) FileSystem.delete(...)
 implementations should not throw FileNotFoundException
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

FileSystem.delete(...) implementations should not throw FileNotFoundException
-----------------------------------------------------------------------------

                 Key: HADOOP-6688
                 URL: https://issues.apache.org/jira/browse/HADOOP-6688
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs, fs/s3
    Affects Versions: 0.20.2
         Environment: Amazon EC2/S3
            Reporter: Danny Leshem
            Priority: Blocker
             Fix For: 0.20.3, 0.21.0, 0.22.0


S3FileSystem.delete(Path path, boolean recursive) may fail and throw a FileNotFoundException if a directory is being deleted while at the same time some of its files are deleted in the background.

This is definitely not the expected behavior of a delete method. If one of the to-be-deleted files is found missing, the method should not fail and simply continue. This is true for the general contract of FileSystem.delete, and also for its various implementations: RawLocalFileSystem (and specifically FileUtil.fullyDelete) exhibits the same problem.

The fix is to silently catch and ignore FileNotFoundExceptions in delete loops. This can very easily be unit-tested, at least for RawLocalFileSystem.


The reason this issue bothers me is that the cleanup part of a long (Mahout) MR job inconsistently fails for me, and I think this is the root problem. The log shows:
{code}
java.io.FileNotFoundException: s3://S3-BUCKET/tmp/0008E25BF7554CA9/2521362836721872/DistributedMatrix.times.outputVector/_temporary/_attempt_201004061215_0092_r_000002_0/part-00002: No such file or directory.
	at org.apache.hadoop.fs.s3.S3FileSystem.getFileStatus(S3FileSystem.java:334)
	at org.apache.hadoop.fs.s3.S3FileSystem.listStatus(S3FileSystem.java:193)
	at org.apache.hadoop.fs.s3.S3FileSystem.delete(S3FileSystem.java:303)
	at org.apache.hadoop.fs.s3.S3FileSystem.delete(S3FileSystem.java:312)
	at org.apache.hadoop.mapred.FileOutputCommitter.cleanupJob(FileOutputCommitter.java:64)
	at org.apache.hadoop.mapred.OutputCommitter.cleanupJob(OutputCommitter.java:135)
	at org.apache.hadoop.mapred.Task.runJobCleanupTask(Task.java:826)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:292)
	at org.apache.hadoop.mapred.Child.main(Child.java:170)
{code}
(similar errors are displayed for ReduceTask.run)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7246-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 11:16:00 2010
Return-Path: <common-issues-return-7246-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80062 invoked from network); 7 Apr 2010 11:16:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 11:16:00 -0000
Received: (qmail 5866 invoked by uid 500); 7 Apr 2010 11:16:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5756 invoked by uid 500); 7 Apr 2010 11:15:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5738 invoked by uid 99); 7 Apr 2010 11:15:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 11:15:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 11:15:54 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 7B846234C4AB
	for <common-issues@hadoop.apache.org>; Wed,  7 Apr 2010 11:15:33 +0000 (UTC)
Message-ID: <615928254.40091270638933504.JavaMail.jira@brutus.apache.org>
Date: Wed, 7 Apr 2010 11:15:33 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6680) hadoop-cloud push command invokes
 proxy creation
In-Reply-To: <1629624146.687721270478607313.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854421#action_12854421 ] 

Hudson commented on HADOOP-6680:
--------------------------------

Integrated in Hadoop-Common-trunk #296 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/296/])
    . hadoop-cloud push command invokes proxy creation. Contributed by Andrew Klochkov.


> hadoop-cloud push command invokes proxy creation
> ------------------------------------------------
>
>                 Key: HADOOP-6680
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6680
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: contrib/cloud
>            Reporter: Andrew Klochkov
>            Assignee: Andrew Klochkov
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6680.patch
>
>
> There's a typo in src/contrib/cloud/src/py/hadoop/cloud/cli.py: when "push" command is invoked, the script actually invokes proxy creation instead of copying files. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7247-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 16:20:57 2010
Return-Path: <common-issues-return-7247-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62900 invoked from network); 7 Apr 2010 16:20:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 16:20:56 -0000
Received: (qmail 79375 invoked by uid 500); 7 Apr 2010 16:20:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78798 invoked by uid 500); 7 Apr 2010 16:20:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78322 invoked by uid 99); 7 Apr 2010 16:20:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 16:20:55 +0000
X-ASF-Spam-Status: No, hits=-1240.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 16:20:54 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 15878234C4BB
	for <common-issues@hadoop.apache.org>; Wed,  7 Apr 2010 16:20:34 +0000 (UTC)
Message-ID: <861198162.46241270657234087.JavaMail.jira@brutus.apache.org>
Date: Wed, 7 Apr 2010 16:20:34 +0000 (UTC)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6671) To use maven for hadoop common
 builds
In-Reply-To: <859246128.630391270120227191.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854577#action_12854577 ] 

Doug Cutting commented on HADOOP-6671:
--------------------------------------

> The directory moving/renaming unfortunately tends not to work too well with Subversion and branches (or I was doing it wrong) so I don't know how big the benefit would be to start a new one for doing this.

Yes, if the branch is at all long-lived it will require lots of merges to keep it up to date, and such merges will not be easy.

In my experience tree reorganizations are easier to develop as:
 - a shell script that contains a sequence of 'svn mv' commands
 - a patch file to be applied after the script has run

For example, AVRO-163 was developed this way.


> To use maven for hadoop common builds
> -------------------------------------
>
>                 Key: HADOOP-6671
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6671
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Giridharan Kesavan
>
> We are now able to publish hadoop artifacts to the maven repo successfully [ Hadoop-6382]
> Drawbacks with the current approach:
> * Use ivy for dependency management with ivy.xml
> * Use maven-ant-task for artifact publishing to the maven repository
> * pom files are not generated dynamically 
> To address this I propose we use maven to build hadoop-common, which would help us to manage dependencies, publish artifacts and have one single xml file(POM) for dependency management and artifact publishing.
> I would like to have a branch created to work on mavenizing  hadoop common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7248-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 16:58:55 2010
Return-Path: <common-issues-return-7248-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74230 invoked from network); 7 Apr 2010 16:58:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 16:58:55 -0000
Received: (qmail 44055 invoked by uid 500); 7 Apr 2010 16:58:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44029 invoked by uid 500); 7 Apr 2010 16:58:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44020 invoked by uid 99); 7 Apr 2010 16:58:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 16:58:55 +0000
X-ASF-Spam-Status: No, hits=-1241.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 16:58:53 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 89436234C4AB
	for <common-issues@hadoop.apache.org>; Wed,  7 Apr 2010 16:58:33 +0000 (UTC)
Message-ID: <1171478926.47021270659513561.JavaMail.jira@brutus.apache.org>
Date: Wed, 7 Apr 2010 16:58:33 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6686) Remove redundant exception class
 name in unwrapped exceptions thrown at the RPC client
In-Reply-To: <70650712.27801270591413554.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854596#action_12854596 ] 

Suresh Srinivas commented on HADOOP-6686:
-----------------------------------------

The problem with that approach is, if stack trace is printed for RemoteException (not sure if any does toString() for the exception), then the exception name printed will be RemoteException. The wrapped exception name is lost!

> Remove redundant exception class name in unwrapped exceptions thrown at the RPC client
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6686
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6686
>             Project: Hadoop Common
>          Issue Type: Improvement
>         Environment: At RPC client, when the exception thrown by the server is unwrapped, the exception message includes redundant exception class name. This redundant information should be removed.
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6686.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7249-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 17:11:17 2010
Return-Path: <common-issues-return-7249-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77210 invoked from network); 7 Apr 2010 17:11:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 17:11:17 -0000
Received: (qmail 70783 invoked by uid 500); 7 Apr 2010 17:11:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70759 invoked by uid 500); 7 Apr 2010 17:11:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70751 invoked by uid 99); 7 Apr 2010 17:11:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 17:11:17 +0000
X-ASF-Spam-Status: No, hits=-1241.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 17:11:16 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 222AF234C4B9
	for <common-issues@hadoop.apache.org>; Wed,  7 Apr 2010 17:10:41 +0000 (UTC)
Message-ID: <201612114.47571270660241139.JavaMail.jira@brutus.apache.org>
Date: Wed, 7 Apr 2010 17:10:41 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6686) Remove redundant exception class
 name in unwrapped exceptions thrown at the RPC client
In-Reply-To: <70650712.27801270591413554.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854603#action_12854603 ] 

Suresh Srinivas commented on HADOOP-6686:
-----------------------------------------

After running a quick test, looks like in Throwable#printStackTrace(), the first line printed is Throwable#toString(). This means the change you suggested can be made without losing the wrapped exception info, if we add toString() to RemoteException to print wrapped exception name.

> Remove redundant exception class name in unwrapped exceptions thrown at the RPC client
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6686
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6686
>             Project: Hadoop Common
>          Issue Type: Improvement
>         Environment: At RPC client, when the exception thrown by the server is unwrapped, the exception message includes redundant exception class name. This redundant information should be removed.
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6686.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7250-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 17:18:58 2010
Return-Path: <common-issues-return-7250-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79399 invoked from network); 7 Apr 2010 17:18:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 17:18:58 -0000
Received: (qmail 80498 invoked by uid 500); 7 Apr 2010 17:18:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80456 invoked by uid 500); 7 Apr 2010 17:18:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80414 invoked by uid 99); 7 Apr 2010 17:18:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 17:18:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 17:18:55 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id B9062234C4AC
	for <common-issues@hadoop.apache.org>; Wed,  7 Apr 2010 17:18:34 +0000 (UTC)
Message-ID: <1987122042.47681270660714756.JavaMail.jira@brutus.apache.org>
Date: Wed, 7 Apr 2010 17:18:34 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6689) Add directory renaming test to
 FileContextMainOperationsBaseTest
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Add directory renaming test to FileContextMainOperationsBaseTest
----------------------------------------------------------------

                 Key: HADOOP-6689
                 URL: https://issues.apache.org/jira/browse/HADOOP-6689
             Project: Hadoop Common
          Issue Type: Test
          Components: fs, test
            Reporter: Eli Collins
            Assignee: Eli Collins
            Priority: Minor


I noticed FileContextMainOperationsBaseTest does not have a test that renames an empty directory to an empty directory (and shows that this fails without the overwrite option). 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7251-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 17:20:54 2010
Return-Path: <common-issues-return-7251-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79611 invoked from network); 7 Apr 2010 17:20:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 17:20:54 -0000
Received: (qmail 80921 invoked by uid 500); 7 Apr 2010 17:20:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80888 invoked by uid 500); 7 Apr 2010 17:20:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80880 invoked by uid 99); 7 Apr 2010 17:20:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 17:20:54 +0000
X-ASF-Spam-Status: No, hits=-1241.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 17:20:53 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 7281D234C4A9
	for <common-issues@hadoop.apache.org>; Wed,  7 Apr 2010 17:20:33 +0000 (UTC)
Message-ID: <674251801.47741270660833467.JavaMail.jira@brutus.apache.org>
Date: Wed, 7 Apr 2010 17:20:33 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6689) Add directory renaming test to
 FileContextMainOperationsBaseTest
In-Reply-To: <1987122042.47681270660714756.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6689:
--------------------------------

    Attachment: hadoop-6689-1.patch

Adds testRenameDirectoryAsEmptyDirectory and fixes some comments in testRenameDirectoryAsNonEmptyDirectory. Tested on HDFS and locally via  TestHDFSFileContextMainOperations and TestHDFSFileContextMainOperations. 


> Add directory renaming test to FileContextMainOperationsBaseTest
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6689
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6689
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>         Attachments: hadoop-6689-1.patch
>
>
> I noticed FileContextMainOperationsBaseTest does not have a test that renames an empty directory to an empty directory (and shows that this fails without the overwrite option). 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7252-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 17:22:55 2010
Return-Path: <common-issues-return-7252-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80466 invoked from network); 7 Apr 2010 17:22:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 17:22:54 -0000
Received: (qmail 86940 invoked by uid 500); 7 Apr 2010 17:22:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86855 invoked by uid 500); 7 Apr 2010 17:22:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86847 invoked by uid 99); 7 Apr 2010 17:22:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 17:22:54 +0000
X-ASF-Spam-Status: No, hits=-1241.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 17:22:53 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id B9CEF234C4B1
	for <common-issues@hadoop.apache.org>; Wed,  7 Apr 2010 17:22:33 +0000 (UTC)
Message-ID: <670545847.47811270660953760.JavaMail.jira@brutus.apache.org>
Date: Wed, 7 Apr 2010 17:22:33 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6689) Add directory renaming test to
 FileContextMainOperationsBaseTest
In-Reply-To: <1987122042.47681270660714756.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6689:
--------------------------------

    Status: Patch Available  (was: Open)

> Add directory renaming test to FileContextMainOperationsBaseTest
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6689
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6689
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>         Attachments: hadoop-6689-1.patch
>
>
> I noticed FileContextMainOperationsBaseTest does not have a test that renames an empty directory to an empty directory (and shows that this fails without the overwrite option). 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7253-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 18:52:58 2010
Return-Path: <common-issues-return-7253-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6376 invoked from network); 7 Apr 2010 18:52:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 18:52:57 -0000
Received: (qmail 22726 invoked by uid 500); 7 Apr 2010 17:52:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22684 invoked by uid 500); 7 Apr 2010 17:52:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22676 invoked by uid 99); 7 Apr 2010 17:52:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 17:52:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 17:52:54 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 78696234C4B2
	for <common-issues@hadoop.apache.org>; Wed,  7 Apr 2010 17:52:33 +0000 (UTC)
Message-ID: <1739215520.48381270662753492.JavaMail.jira@brutus.apache.org>
Date: Wed, 7 Apr 2010 17:52:33 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6689) Add directory renaming test to
 FileContextMainOperationsBaseTest
In-Reply-To: <1987122042.47681270660714756.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854618#action_12854618 ] 

Hadoop QA commented on HADOOP-6689:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12441059/hadoop-6689-1.patch
  against trunk revision 931226.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/445/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/445/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/445/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/445/console

This message is automatically generated.

> Add directory renaming test to FileContextMainOperationsBaseTest
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6689
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6689
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>         Attachments: hadoop-6689-1.patch
>
>
> I noticed FileContextMainOperationsBaseTest does not have a test that renames an empty directory to an empty directory (and shows that this fails without the overwrite option). 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7257-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 18:55:34 2010
Return-Path: <common-issues-return-7257-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7503 invoked from network); 7 Apr 2010 18:55:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 18:55:34 -0000
Received: (qmail 67970 invoked by uid 500); 7 Apr 2010 18:28:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67931 invoked by uid 500); 7 Apr 2010 18:28:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67923 invoked by uid 99); 7 Apr 2010 18:28:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 18:28:54 +0000
X-ASF-Spam-Status: No, hits=-1242.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 18:28:53 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id A24DD234C4A9
	for <common-issues@hadoop.apache.org>; Wed,  7 Apr 2010 18:28:33 +0000 (UTC)
Message-ID: <1118540616.49151270664913663.JavaMail.jira@brutus.apache.org>
Date: Wed, 7 Apr 2010 18:28:33 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6689) Add directory renaming test to
 FileContextMainOperationsBaseTest
In-Reply-To: <1987122042.47681270660714756.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854638#action_12854638 ] 

Hadoop QA commented on HADOOP-6689:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12441066/hadoop-6689-2.patch
  against trunk revision 931226.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/446/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/446/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/446/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/446/console

This message is automatically generated.

> Add directory renaming test to FileContextMainOperationsBaseTest
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6689
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6689
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>         Attachments: hadoop-6689-1.patch, hadoop-6689-2.patch
>
>
> I noticed FileContextMainOperationsBaseTest does not have a test that renames an empty directory to an empty directory (and shows that this fails without the overwrite option). 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7258-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 19:04:59 2010
Return-Path: <common-issues-return-7258-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13769 invoked from network); 7 Apr 2010 19:04:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 19:04:57 -0000
Received: (qmail 7101 invoked by uid 500); 7 Apr 2010 19:04:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7052 invoked by uid 500); 7 Apr 2010 19:04:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7028 invoked by uid 99); 7 Apr 2010 19:04:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 19:04:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 19:04:54 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 9B382234C4AE
	for <common-issues@hadoop.apache.org>; Wed,  7 Apr 2010 19:04:33 +0000 (UTC)
Message-ID: <1203106676.49571270667073635.JavaMail.jira@brutus.apache.org>
Date: Wed, 7 Apr 2010 19:04:33 +0000 (UTC)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6686) Remove redundant exception class
 name in unwrapped exceptions thrown at the RPC client
In-Reply-To: <70650712.27801270591413554.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854647#action_12854647 ] 

Doug Cutting commented on HADOOP-6686:
--------------------------------------

> add toString() to RemoteException to print wrapped exception name

+1 I think that's a good approach.


> Remove redundant exception class name in unwrapped exceptions thrown at the RPC client
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6686
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6686
>             Project: Hadoop Common
>          Issue Type: Improvement
>         Environment: At RPC client, when the exception thrown by the server is unwrapped, the exception message includes redundant exception class name. This redundant information should be removed.
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6686.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7256-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 19:10:54 2010
Return-Path: <common-issues-return-7256-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15988 invoked from network); 7 Apr 2010 19:10:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 19:10:54 -0000
Received: (qmail 40346 invoked by uid 500); 7 Apr 2010 18:10:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40320 invoked by uid 500); 7 Apr 2010 18:10:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40236 invoked by uid 99); 7 Apr 2010 18:10:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 18:10:54 +0000
X-ASF-Spam-Status: No, hits=-1241.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 18:10:53 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 9EEC4234C4AE
	for <common-issues@hadoop.apache.org>; Wed,  7 Apr 2010 18:10:33 +0000 (UTC)
Message-ID: <60338526.48841270663833650.JavaMail.jira@brutus.apache.org>
Date: Wed, 7 Apr 2010 18:10:33 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6689) Add directory renaming test to
 FileContextMainOperationsBaseTest
In-Reply-To: <1987122042.47681270660714756.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6689:
--------------------------------

    Status: Patch Available  (was: Open)

> Add directory renaming test to FileContextMainOperationsBaseTest
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6689
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6689
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>         Attachments: hadoop-6689-1.patch, hadoop-6689-2.patch
>
>
> I noticed FileContextMainOperationsBaseTest does not have a test that renames an empty directory to an empty directory (and shows that this fails without the overwrite option). 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7255-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 19:10:54 2010
Return-Path: <common-issues-return-7255-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16012 invoked from network); 7 Apr 2010 19:10:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 19:10:54 -0000
Received: (qmail 40259 invoked by uid 500); 7 Apr 2010 18:10:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40191 invoked by uid 500); 7 Apr 2010 18:10:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40133 invoked by uid 99); 7 Apr 2010 18:10:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 18:10:54 +0000
X-ASF-Spam-Status: No, hits=-1241.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 18:10:53 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 95DFF234C4A9
	for <common-issues@hadoop.apache.org>; Wed,  7 Apr 2010 18:10:33 +0000 (UTC)
Message-ID: <1995300475.48801270663833612.JavaMail.jira@brutus.apache.org>
Date: Wed, 7 Apr 2010 18:10:33 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6689) Add directory renaming test to
 FileContextMainOperationsBaseTest
In-Reply-To: <1987122042.47681270660714756.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6689:
--------------------------------

    Attachment: hadoop-6689-2.patch

Right patch this time.

> Add directory renaming test to FileContextMainOperationsBaseTest
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6689
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6689
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>         Attachments: hadoop-6689-1.patch, hadoop-6689-2.patch
>
>
> I noticed FileContextMainOperationsBaseTest does not have a test that renames an empty directory to an empty directory (and shows that this fails without the overwrite option). 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7254-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 19:10:55 2010
Return-Path: <common-issues-return-7254-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16013 invoked from network); 7 Apr 2010 19:10:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 19:10:54 -0000
Received: (qmail 40114 invoked by uid 500); 7 Apr 2010 18:10:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40062 invoked by uid 500); 7 Apr 2010 18:10:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40054 invoked by uid 99); 7 Apr 2010 18:10:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 18:10:54 +0000
X-ASF-Spam-Status: No, hits=-1241.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 18:10:53 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 9BCF3234C4AC
	for <common-issues@hadoop.apache.org>; Wed,  7 Apr 2010 18:10:33 +0000 (UTC)
Message-ID: <183977128.48821270663833637.JavaMail.jira@brutus.apache.org>
Date: Wed, 7 Apr 2010 18:10:33 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6689) Add directory renaming test to
 FileContextMainOperationsBaseTest
In-Reply-To: <1987122042.47681270660714756.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6689:
--------------------------------

    Status: Open  (was: Patch Available)

> Add directory renaming test to FileContextMainOperationsBaseTest
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6689
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6689
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>         Attachments: hadoop-6689-1.patch, hadoop-6689-2.patch
>
>
> I noticed FileContextMainOperationsBaseTest does not have a test that renames an empty directory to an empty directory (and shows that this fails without the overwrite option). 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7259-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 07 23:21:58 2010
Return-Path: <common-issues-return-7259-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82205 invoked from network); 7 Apr 2010 23:21:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Apr 2010 23:21:58 -0000
Received: (qmail 20445 invoked by uid 500); 7 Apr 2010 23:21:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20413 invoked by uid 500); 7 Apr 2010 23:21:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20405 invoked by uid 99); 7 Apr 2010 23:21:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 23:21:58 +0000
X-ASF-Spam-Status: No, hits=-1243.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Apr 2010 23:21:57 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 14AD8234C4AE
	for <common-issues@hadoop.apache.org>; Wed,  7 Apr 2010 23:21:37 +0000 (UTC)
Message-ID: <1663642933.61270682497083.JavaMail.jira@brutus.apache.org>
Date: Wed, 7 Apr 2010 23:21:37 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6686) Remove redundant exception class
 name in unwrapped exceptions thrown at the RPC client
In-Reply-To: <70650712.27801270591413554.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854721#action_12854721 ] 

Suresh Srinivas commented on HADOOP-6686:
-----------------------------------------

With the change RemoteException has stack trace as exception message without the exception name. However unwrapped exceptions have only the first line from the stack trace without the exception name. Should we  change the unwrapped exception to also include the stack trace? I think it will will be good for debugging.

> Remove redundant exception class name in unwrapped exceptions thrown at the RPC client
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6686
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6686
>             Project: Hadoop Common
>          Issue Type: Improvement
>         Environment: At RPC client, when the exception thrown by the server is unwrapped, the exception message includes redundant exception class name. This redundant information should be removed.
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6686.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7260-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 02:18:00 2010
Return-Path: <common-issues-return-7260-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12879 invoked from network); 8 Apr 2010 02:18:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 02:18:00 -0000
Received: (qmail 51878 invoked by uid 500); 8 Apr 2010 02:18:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51851 invoked by uid 500); 8 Apr 2010 02:18:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51837 invoked by uid 99); 8 Apr 2010 02:17:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 02:17:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 02:17:57 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D539529A0014
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 02:17:36 +0000 (UTC)
Message-ID: <294275186.4281270693056872.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 02:17:36 +0000 (UTC)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6690) FilterFileSystem doesn't overwrite
 setTimes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

FilterFileSystem doesn't overwrite setTimes
-------------------------------------------

                 Key: HADOOP-6690
                 URL: https://issues.apache.org/jira/browse/HADOOP-6690
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.20.3, 0.21.0, 0.22.0
            Reporter: Rodrigo Schmidt
             Fix For: 0.22.0


FilterFileSystem seems to be a little outdated and it doesn't implement a few methods (setTimes being the most important one):

- setTimes(Path, long, long)
- copyFromLocalFile(boolean,boolean, Path, Path)
- copyFromLocalFile(boolean, boolean, Path[], Path)
- getUsed()
- deleteOnExit()

I'm not sure if all of these methods should be wrapped in FilterFileSystem, but given its purpose, I would say the more the better. It would be great to have other people's opinions about this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7261-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 02:21:57 2010
Return-Path: <common-issues-return-7261-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13242 invoked from network); 8 Apr 2010 02:21:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 02:21:57 -0000
Received: (qmail 53766 invoked by uid 500); 8 Apr 2010 02:21:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53732 invoked by uid 500); 8 Apr 2010 02:21:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53723 invoked by uid 99); 8 Apr 2010 02:21:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 02:21:57 +0000
X-ASF-Spam-Status: No, hits=-1245.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 02:21:56 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id A708D29A0012
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 02:21:36 +0000 (UTC)
Message-ID: <816143913.4311270693296682.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 02:21:36 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6563:
--------------------------------

    Attachment: hadoop-6563-1.patch

Attached patch adds test coverage for intermediate symlinks in paths, and fixes a bug these tests uncovered. 

It also adds a lot of rename unit tests so that the full set of behavior with symlinks, and it conforms to the posix behavior (modulo the overwrite option which is not present in {{rename(2)}}). Here's the current FileContext#rename behavior, renaming *source* to *dest* on HDFS, augmented with symlinks:

* If *source* does not exist then a FileNotFoundException is thrown.
* If *dest* does not exist then *source* is renamed *dest*. If *source* is a symlink then the symlink itself is renamed, not the link target.
* If *dest* exists and is a non-empty directory then an IOException is thrown.
* If *dest* exists and the OVERWRITE option is not given then an IOException is thrown.
* If *dest* exists and the OVERWRITE option is given then the behavior depends on the type of *source* and *dest*:

||Source||    Dest||    Result||
|fileA  |    fileB|     Renames *fileA* to *fileB*|
|       |     dirB|     Fails: both need to be file or directory|
|       |    linkB|     *fileA* is renamed *linkB* (regardless of what *linkB* points to)|
|dirA   |    fileB|     Fails: both need to be file or directory|
|       |     dirB|     Renames *dirA* to *dirB* (*dirB* is empty)|
|       |    linkB|     Fails: both need to be file or directory|
|linkA  |    fileB|     Renames *linkA* to *fileB*|
|       |     dirB|     Fails: *dirB* is a directory|
|       |    linkB|     Renames *linkA* to *linkB* (regardless of what *linkB* points to)|


> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>         Attachments: hadoop-6563-1.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7262-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 02:23:58 2010
Return-Path: <common-issues-return-7262-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13608 invoked from network); 8 Apr 2010 02:23:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 02:23:57 -0000
Received: (qmail 55200 invoked by uid 500); 8 Apr 2010 02:23:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55168 invoked by uid 500); 8 Apr 2010 02:23:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54989 invoked by uid 99); 8 Apr 2010 02:23:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 02:23:57 +0000
X-ASF-Spam-Status: No, hits=-1245.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 02:23:56 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 9864629A0012
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 02:23:36 +0000 (UTC)
Message-ID: <2115262619.4351270693416623.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 02:23:36 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6563:
--------------------------------

    Status: Patch Available  (was: Open)

> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>         Attachments: hadoop-6563-1.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7263-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 02:49:01 2010
Return-Path: <common-issues-return-7263-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16703 invoked from network); 8 Apr 2010 02:49:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 02:49:00 -0000
Received: (qmail 70944 invoked by uid 500); 8 Apr 2010 02:49:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70880 invoked by uid 500); 8 Apr 2010 02:49:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70837 invoked by uid 99); 8 Apr 2010 02:49:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 02:49:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 02:48:57 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id AF00A29A0014
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 02:48:36 +0000 (UTC)
Message-ID: <2000578866.4721270694916715.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 02:48:36 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854777#action_12854777 ] 

Hadoop QA commented on HADOOP-6563:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12441102/hadoop-6563-1.patch
  against trunk revision 931226.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/447/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/447/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/447/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/447/console

This message is automatically generated.

> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>         Attachments: hadoop-6563-1.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7264-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 04:09:01 2010
Return-Path: <common-issues-return-7264-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33560 invoked from network); 8 Apr 2010 04:09:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 04:09:00 -0000
Received: (qmail 42298 invoked by uid 500); 8 Apr 2010 04:09:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41963 invoked by uid 500); 8 Apr 2010 04:09:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41946 invoked by uid 99); 8 Apr 2010 04:08:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 04:08:59 +0000
X-ASF-Spam-Status: No, hits=-1245.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 04:08:58 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C2483234C4AE
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 04:08:38 +0000 (UTC)
Message-ID: <531383320.5441270699718794.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 04:08:38 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6678) Propose some changes to FileContext
In-Reply-To: <360625036.668801270253667713.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6678:
--------------------------------

    Attachment: hadoop-6678-1.patch

Here's a patch for part 2, removes FileContext#isFile, isDirectory, and exists. 

Notes:
* We no longer call fixRelativePart on the path passed to getFileStatus since that method already calls fixRelativePart on the argument (so this redundant call is now gone)
* deleteOnExit now throws the FileNotFoundException like delete rather than return false 
* Removed UnresolvedLinkException from the throws clause of the public FileContext APIs since they'll never throw this exception (FileContext resolves links). 
* Patch applies to trunk plus HADOOP-6689 and HADOOP-6563

> Propose some changes to FileContext
> -----------------------------------
>
>                 Key: HADOOP-6678
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6678
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Hairong Kuang
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6678-1.patch
>
>
> # Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.
> # Remove methods isFile(Path), isDirectory(Path), and exists.
> All these methods are implemented by calling getFileStatus(Path).But most users are not aware of this. They would write code as below: 
> {code}
>   FileContext fc = ..;
>   if (fc.exists(path)) {
>     if (fc.isFile(path)) {
>      ...
>     } else {
>     ...
>     }
>   }
> {code}
> The above code adds unnecessary getFileInfo RPC to NameNode. In our production clusters, we often see that the number of getFileStatus calls is multiple times of the open calls. If we remove isFile, isDirectory, and exists from FileContext, users have to explicitly call getFileStatus first, it is more likely that they will write more efficient code as follow:
> {code}
>   FileContext fc = ...;
>   FileStatus fstatus = fc.getFileStatus(path);
>   if (fstatus.isFile() {
>     ...
>   } else {
>     ...
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7265-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 04:14:59 2010
Return-Path: <common-issues-return-7265-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34336 invoked from network); 8 Apr 2010 04:14:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 04:14:59 -0000
Received: (qmail 46239 invoked by uid 500); 8 Apr 2010 04:14:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46131 invoked by uid 500); 8 Apr 2010 04:14:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46118 invoked by uid 99); 8 Apr 2010 04:14:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 04:14:57 +0000
X-ASF-Spam-Status: No, hits=-1245.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 04:14:56 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id BE0AF29A0016
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 04:14:36 +0000 (UTC)
Message-ID: <1449178439.5621270700076777.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 04:14:36 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6678) Propose some changes to FileContext
In-Reply-To: <360625036.668801270253667713.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6678:
--------------------------------

    Status: Patch Available  (was: Open)

> Propose some changes to FileContext
> -----------------------------------
>
>                 Key: HADOOP-6678
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6678
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Hairong Kuang
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6678-1.patch
>
>
> # Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.
> # Remove methods isFile(Path), isDirectory(Path), and exists.
> All these methods are implemented by calling getFileStatus(Path).But most users are not aware of this. They would write code as below: 
> {code}
>   FileContext fc = ..;
>   if (fc.exists(path)) {
>     if (fc.isFile(path)) {
>      ...
>     } else {
>     ...
>     }
>   }
> {code}
> The above code adds unnecessary getFileInfo RPC to NameNode. In our production clusters, we often see that the number of getFileStatus calls is multiple times of the open calls. If we remove isFile, isDirectory, and exists from FileContext, users have to explicitly call getFileStatus first, it is more likely that they will write more efficient code as follow:
> {code}
>   FileContext fc = ...;
>   FileStatus fstatus = fc.getFileStatus(path);
>   if (fstatus.isFile() {
>     ...
>   } else {
>     ...
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7266-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 04:15:01 2010
Return-Path: <common-issues-return-7266-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34398 invoked from network); 8 Apr 2010 04:15:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 04:15:00 -0000
Received: (qmail 46324 invoked by uid 500); 8 Apr 2010 04:15:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46277 invoked by uid 500); 8 Apr 2010 04:15:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46268 invoked by uid 99); 8 Apr 2010 04:15:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 04:15:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 04:14:57 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id B175A234C4B2
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 04:14:36 +0000 (UTC)
Message-ID: <1055289006.5541270700076725.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 04:14:36 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6678) Propose some changes to FileContext
In-Reply-To: <360625036.668801270253667713.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854789#action_12854789 ] 

Eli Collins commented on HADOOP-6678:
-------------------------------------

Forgot to mention, this patch also adds FileStatus#isFile, currently it just returns !isDir so that it's clear this change is mostly just changing syntax. Further FileStatus changes will be made in HADOOP-6585.

> Propose some changes to FileContext
> -----------------------------------
>
>                 Key: HADOOP-6678
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6678
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Hairong Kuang
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6678-1.patch
>
>
> # Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.
> # Remove methods isFile(Path), isDirectory(Path), and exists.
> All these methods are implemented by calling getFileStatus(Path).But most users are not aware of this. They would write code as below: 
> {code}
>   FileContext fc = ..;
>   if (fc.exists(path)) {
>     if (fc.isFile(path)) {
>      ...
>     } else {
>     ...
>     }
>   }
> {code}
> The above code adds unnecessary getFileInfo RPC to NameNode. In our production clusters, we often see that the number of getFileStatus calls is multiple times of the open calls. If we remove isFile, isDirectory, and exists from FileContext, users have to explicitly call getFileStatus first, it is more likely that they will write more efficient code as follow:
> {code}
>   FileContext fc = ...;
>   FileStatus fstatus = fc.getFileStatus(path);
>   if (fstatus.isFile() {
>     ...
>   } else {
>     ...
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7267-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 04:27:10 2010
Return-Path: <common-issues-return-7267-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35667 invoked from network); 8 Apr 2010 04:27:10 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 04:27:10 -0000
Received: (qmail 52658 invoked by uid 500); 8 Apr 2010 04:27:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52273 invoked by uid 500); 8 Apr 2010 04:27:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52259 invoked by uid 99); 8 Apr 2010 04:27:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 04:27:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 04:27:06 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4B251234C4AC
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 04:26:45 +0000 (UTC)
Message-ID: <304516670.5721270700805306.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 04:26:45 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6678) Propose some changes to FileContext
In-Reply-To: <360625036.668801270253667713.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854792#action_12854792 ] 

Hadoop QA commented on HADOOP-6678:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12441111/hadoop-6678-1.patch
  against trunk revision 931226.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 21 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/448/console

This message is automatically generated.

> Propose some changes to FileContext
> -----------------------------------
>
>                 Key: HADOOP-6678
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6678
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Hairong Kuang
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6678-1.patch
>
>
> # Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.
> # Remove methods isFile(Path), isDirectory(Path), and exists.
> All these methods are implemented by calling getFileStatus(Path).But most users are not aware of this. They would write code as below: 
> {code}
>   FileContext fc = ..;
>   if (fc.exists(path)) {
>     if (fc.isFile(path)) {
>      ...
>     } else {
>     ...
>     }
>   }
> {code}
> The above code adds unnecessary getFileInfo RPC to NameNode. In our production clusters, we often see that the number of getFileStatus calls is multiple times of the open calls. If we remove isFile, isDirectory, and exists from FileContext, users have to explicitly call getFileStatus first, it is more likely that they will write more efficient code as follow:
> {code}
>   FileContext fc = ...;
>   FileStatus fstatus = fc.getFileStatus(path);
>   if (fstatus.isFile() {
>     ...
>   } else {
>     ...
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7268-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 04:34:59 2010
Return-Path: <common-issues-return-7268-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36113 invoked from network); 8 Apr 2010 04:34:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 04:34:58 -0000
Received: (qmail 57832 invoked by uid 500); 8 Apr 2010 04:34:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57736 invoked by uid 500); 8 Apr 2010 04:34:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57727 invoked by uid 99); 8 Apr 2010 04:34:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 04:34:57 +0000
X-ASF-Spam-Status: No, hits=-1245.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 04:34:57 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id BD005234C4AE
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 04:34:36 +0000 (UTC)
Message-ID: <1607173243.5851270701276772.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 04:34:36 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6678) Propose some changes to FileContext
In-Reply-To: <360625036.668801270253667713.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854795#action_12854795 ] 

Eli Collins commented on HADOOP-6678:
-------------------------------------

Right, the patch requires HADOOP-6689 and HADOOP-6563, sorry for the noise.

> Propose some changes to FileContext
> -----------------------------------
>
>                 Key: HADOOP-6678
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6678
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Hairong Kuang
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6678-1.patch
>
>
> # Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.
> # Remove methods isFile(Path), isDirectory(Path), and exists.
> All these methods are implemented by calling getFileStatus(Path).But most users are not aware of this. They would write code as below: 
> {code}
>   FileContext fc = ..;
>   if (fc.exists(path)) {
>     if (fc.isFile(path)) {
>      ...
>     } else {
>     ...
>     }
>   }
> {code}
> The above code adds unnecessary getFileInfo RPC to NameNode. In our production clusters, we often see that the number of getFileStatus calls is multiple times of the open calls. If we remove isFile, isDirectory, and exists from FileContext, users have to explicitly call getFileStatus first, it is more likely that they will write more efficient code as follow:
> {code}
>   FileContext fc = ...;
>   FileStatus fstatus = fc.getFileStatus(path);
>   if (fstatus.isFile() {
>     ...
>   } else {
>     ...
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7269-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 04:36:58 2010
Return-Path: <common-issues-return-7269-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36280 invoked from network); 8 Apr 2010 04:36:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 04:36:58 -0000
Received: (qmail 59228 invoked by uid 500); 8 Apr 2010 04:36:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59128 invoked by uid 500); 8 Apr 2010 04:36:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59120 invoked by uid 99); 8 Apr 2010 04:36:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 04:36:57 +0000
X-ASF-Spam-Status: No, hits=-1245.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 04:36:56 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C2188234C4AE
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 04:36:36 +0000 (UTC)
Message-ID: <291768451.5911270701396793.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 04:36:36 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6690) FilterFileSystem doesn't overwrite
 setTimes
In-Reply-To: <294275186.4281270693056872.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854796#action_12854796 ] 

Eli Collins commented on HADOOP-6690:
-------------------------------------

Can you use FilterFs? It has setTimes.

> FilterFileSystem doesn't overwrite setTimes
> -------------------------------------------
>
>                 Key: HADOOP-6690
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6690
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.3, 0.21.0, 0.22.0
>            Reporter: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>
> FilterFileSystem seems to be a little outdated and it doesn't implement a few methods (setTimes being the most important one):
> - setTimes(Path, long, long)
> - copyFromLocalFile(boolean,boolean, Path, Path)
> - copyFromLocalFile(boolean, boolean, Path[], Path)
> - getUsed()
> - deleteOnExit()
> I'm not sure if all of these methods should be wrapped in FilterFileSystem, but given its purpose, I would say the more the better. It would be great to have other people's opinions about this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7270-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 05:02:59 2010
Return-Path: <common-issues-return-7270-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39236 invoked from network); 8 Apr 2010 05:02:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 05:02:58 -0000
Received: (qmail 81317 invoked by uid 500); 8 Apr 2010 05:02:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81267 invoked by uid 500); 8 Apr 2010 05:02:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81259 invoked by uid 99); 8 Apr 2010 05:02:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 05:02:57 +0000
X-ASF-Spam-Status: No, hits=-1245.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 05:02:57 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id B9F5D234C4B3
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 05:02:36 +0000 (UTC)
Message-ID: <1901914082.6321270702956760.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 05:02:36 +0000 (UTC)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6690) FilterFileSystem doesn't overwrite
 setTimes
In-Reply-To: <294275186.4281270693056872.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854810#action_12854810 ] 

Rodrigo Schmidt commented on HADOOP-6690:
-----------------------------------------

Is FilterFs supposed to suppress FilterFileSystem? When was it introduced? 

> FilterFileSystem doesn't overwrite setTimes
> -------------------------------------------
>
>                 Key: HADOOP-6690
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6690
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.3, 0.21.0, 0.22.0
>            Reporter: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>
> FilterFileSystem seems to be a little outdated and it doesn't implement a few methods (setTimes being the most important one):
> - setTimes(Path, long, long)
> - copyFromLocalFile(boolean,boolean, Path, Path)
> - copyFromLocalFile(boolean, boolean, Path[], Path)
> - getUsed()
> - deleteOnExit()
> I'm not sure if all of these methods should be wrapped in FilterFileSystem, but given its purpose, I would say the more the better. It would be great to have other people's opinions about this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7271-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 06:31:02 2010
Return-Path: <common-issues-return-7271-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50826 invoked from network); 8 Apr 2010 06:31:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 06:31:02 -0000
Received: (qmail 30979 invoked by uid 500); 8 Apr 2010 06:31:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30845 invoked by uid 500); 8 Apr 2010 06:31:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30827 invoked by uid 99); 8 Apr 2010 06:31:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 06:31:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 06:30:58 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 7E42A234C4BB
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 06:30:37 +0000 (UTC)
Message-ID: <405516146.7131270708237515.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 06:30:37 +0000 (UTC)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6690) FilterFileSystem doesn't overwrite
 setTimes
In-Reply-To: <294275186.4281270693056872.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt reassigned HADOOP-6690:
---------------------------------------

    Assignee: Rodrigo Schmidt

> FilterFileSystem doesn't overwrite setTimes
> -------------------------------------------
>
>                 Key: HADOOP-6690
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6690
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.3, 0.21.0, 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>
> FilterFileSystem seems to be a little outdated and it doesn't implement a few methods (setTimes being the most important one):
> - setTimes(Path, long, long)
> - copyFromLocalFile(boolean,boolean, Path, Path)
> - copyFromLocalFile(boolean, boolean, Path[], Path)
> - getUsed()
> - deleteOnExit()
> I'm not sure if all of these methods should be wrapped in FilterFileSystem, but given its purpose, I would say the more the better. It would be great to have other people's opinions about this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7272-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 06:34:59 2010
Return-Path: <common-issues-return-7272-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51661 invoked from network); 8 Apr 2010 06:34:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 06:34:59 -0000
Received: (qmail 33663 invoked by uid 500); 8 Apr 2010 06:34:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33149 invoked by uid 500); 8 Apr 2010 06:34:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33136 invoked by uid 99); 8 Apr 2010 06:34:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 06:34:57 +0000
X-ASF-Spam-Status: No, hits=-1246.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 06:34:57 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id BFF03234C4B3
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 06:34:36 +0000 (UTC)
Message-ID: <149282565.7241270708476785.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 06:34:36 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6690) FilterFileSystem doesn't overwrite
 setTimes
In-Reply-To: <294275186.4281270693056872.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854831#action_12854831 ] 

Eli Collins commented on HADOOP-6690:
-------------------------------------

FilterFs is the AbstractFileSystem version of FilterFileSystem. If you're doing work on trunk you should be writing against the new APIs (or at least make sure they stay on par), if you're making a change against branch 20 then you'll want FilterFileSystem. I'm assuming you want setTimes to work on LocalFileSystem?

> FilterFileSystem doesn't overwrite setTimes
> -------------------------------------------
>
>                 Key: HADOOP-6690
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6690
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.3, 0.21.0, 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>
> FilterFileSystem seems to be a little outdated and it doesn't implement a few methods (setTimes being the most important one):
> - setTimes(Path, long, long)
> - copyFromLocalFile(boolean,boolean, Path, Path)
> - copyFromLocalFile(boolean, boolean, Path[], Path)
> - getUsed()
> - deleteOnExit()
> I'm not sure if all of these methods should be wrapped in FilterFileSystem, but given its purpose, I would say the more the better. It would be great to have other people's opinions about this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7273-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 06:40:59 2010
Return-Path: <common-issues-return-7273-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58612 invoked from network); 8 Apr 2010 06:40:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 06:40:59 -0000
Received: (qmail 36011 invoked by uid 500); 8 Apr 2010 06:40:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35919 invoked by uid 500); 8 Apr 2010 06:40:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35911 invoked by uid 99); 8 Apr 2010 06:40:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 06:40:58 +0000
X-ASF-Spam-Status: No, hits=-1246.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 06:40:57 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 41CD3234C4B1
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 06:40:37 +0000 (UTC)
Message-ID: <1554778221.7331270708837268.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 06:40:37 +0000 (UTC)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6690) FilterFileSystem doesn't overwrite
 setTimes
In-Reply-To: <294275186.4281270693056872.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854834#action_12854834 ] 

Rodrigo Schmidt commented on HADOOP-6690:
-----------------------------------------

I noticed the problem when using contrib/raid, which currently relies on FilterFileSystem (because it was originally implemented on hadoop 0.20). Maybe we should change the trunk version to work with FilterFs.

However, this workaround doesn't make FilterFileSystem more correct. We should either fix it or deprecate it so that people don't feel compelled to use it in the future. What do you think?

> FilterFileSystem doesn't overwrite setTimes
> -------------------------------------------
>
>                 Key: HADOOP-6690
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6690
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.3, 0.21.0, 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>
> FilterFileSystem seems to be a little outdated and it doesn't implement a few methods (setTimes being the most important one):
> - setTimes(Path, long, long)
> - copyFromLocalFile(boolean,boolean, Path, Path)
> - copyFromLocalFile(boolean, boolean, Path[], Path)
> - getUsed()
> - deleteOnExit()
> I'm not sure if all of these methods should be wrapped in FilterFileSystem, but given its purpose, I would say the more the better. It would be great to have other people's opinions about this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7274-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 07:12:01 2010
Return-Path: <common-issues-return-7274-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63281 invoked from network); 8 Apr 2010 07:12:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 07:12:01 -0000
Received: (qmail 64011 invoked by uid 500); 8 Apr 2010 07:12:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63902 invoked by uid 500); 8 Apr 2010 07:12:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63776 invoked by uid 99); 8 Apr 2010 07:12:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 07:12:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 07:11:57 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DA938234C4AE
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 07:11:36 +0000 (UTC)
Message-ID: <1852004665.7631270710696894.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 07:11:36 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6690) FilterFileSystem doesn't overwrite
 setTimes
In-Reply-To: <294275186.4281270693056872.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854841#action_12854841 ] 

Eli Collins commented on HADOOP-6690:
-------------------------------------

Having FilterFileSystem override setTimes (and other APIs) is very reasonable. Just pointing out the new APIs in case you were doing trunk work. We'll want to move contrib/raid off FileSystem as part of HADOOP-6446.

> FilterFileSystem doesn't overwrite setTimes
> -------------------------------------------
>
>                 Key: HADOOP-6690
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6690
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.3, 0.21.0, 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>
> FilterFileSystem seems to be a little outdated and it doesn't implement a few methods (setTimes being the most important one):
> - setTimes(Path, long, long)
> - copyFromLocalFile(boolean,boolean, Path, Path)
> - copyFromLocalFile(boolean, boolean, Path[], Path)
> - getUsed()
> - deleteOnExit()
> I'm not sure if all of these methods should be wrapped in FilterFileSystem, but given its purpose, I would say the more the better. It would be great to have other people's opinions about this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7275-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 07:24:02 2010
Return-Path: <common-issues-return-7275-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65210 invoked from network); 8 Apr 2010 07:24:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 07:24:01 -0000
Received: (qmail 72658 invoked by uid 500); 8 Apr 2010 07:24:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72547 invoked by uid 500); 8 Apr 2010 07:24:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72536 invoked by uid 99); 8 Apr 2010 07:24:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 07:24:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 07:23:58 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0155D234C4AE
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 07:23:37 +0000 (UTC)
Message-ID: <1301382618.7711270711417004.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 07:23:37 +0000 (UTC)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6690) FilterFileSystem doesn't overwrite
 setTimes
In-Reply-To: <294275186.4281270693056872.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12854843#action_12854843 ] 

Rodrigo Schmidt commented on HADOOP-6690:
-----------------------------------------

Sure! Thanks for the hint, BTW. 

With respect to HADOOP-6446, I should be able to help with Raid/Har/DistCp as soon as I'm done with some critical raid patches.

> FilterFileSystem doesn't overwrite setTimes
> -------------------------------------------
>
>                 Key: HADOOP-6690
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6690
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.3, 0.21.0, 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>
> FilterFileSystem seems to be a little outdated and it doesn't implement a few methods (setTimes being the most important one):
> - setTimes(Path, long, long)
> - copyFromLocalFile(boolean,boolean, Path, Path)
> - copyFromLocalFile(boolean, boolean, Path[], Path)
> - getUsed()
> - deleteOnExit()
> I'm not sure if all of these methods should be wrapped in FilterFileSystem, but given its purpose, I would say the more the better. It would be great to have other people's opinions about this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7276-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 18:48:02 2010
Return-Path: <common-issues-return-7276-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36228 invoked from network); 8 Apr 2010 18:48:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 18:48:00 -0000
Received: (qmail 95370 invoked by uid 500); 8 Apr 2010 18:47:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95002 invoked by uid 500); 8 Apr 2010 18:47:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94639 invoked by uid 99); 8 Apr 2010 18:47:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 18:47:58 +0000
X-ASF-Spam-Status: No, hits=-1250.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 18:47:57 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1C275234C4AB
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 18:47:37 +0000 (UTC)
Message-ID: <252386459.18441270752457114.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 18:47:37 +0000 (UTC)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6691) TestFileSystemCaching sometimes hang
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

TestFileSystemCaching sometimes hang
------------------------------------

                 Key: HADOOP-6691
                 URL: https://issues.apache.org/jira/browse/HADOOP-6691
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.22.0
            Reporter: Hairong Kuang
            Assignee: Hairong Kuang
             Fix For: 0.22.0


TestFileSystemCaching#testCacheEnabledWithInitializeForeverFS() sometimes hangs if InitializeForeverFileSystem initializes first.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7277-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 18:51:58 2010
Return-Path: <common-issues-return-7277-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38698 invoked from network); 8 Apr 2010 18:51:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 18:51:58 -0000
Received: (qmail 3976 invoked by uid 500); 8 Apr 2010 18:51:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3866 invoked by uid 500); 8 Apr 2010 18:51:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3856 invoked by uid 99); 8 Apr 2010 18:51:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 18:51:58 +0000
X-ASF-Spam-Status: No, hits=-1250.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 18:51:57 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id B4705234C4A9
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 18:51:36 +0000 (UTC)
Message-ID: <393372529.18621270752696738.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 18:51:36 +0000 (UTC)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6691) TestFileSystemCaching sometimes hang
In-Reply-To: <252386459.18441270752457114.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6691:
----------------------------------

    Status: Patch Available  (was: Open)

> TestFileSystemCaching sometimes hang
> ------------------------------------
>
>                 Key: HADOOP-6691
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6691
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: cachingHang.patch
>
>
> TestFileSystemCaching#testCacheEnabledWithInitializeForeverFS() sometimes hangs if InitializeForeverFileSystem initializes first.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7278-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 18:51:59 2010
Return-Path: <common-issues-return-7278-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38732 invoked from network); 8 Apr 2010 18:51:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 18:51:58 -0000
Received: (qmail 4112 invoked by uid 500); 8 Apr 2010 18:51:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4001 invoked by uid 500); 8 Apr 2010 18:51:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3990 invoked by uid 99); 8 Apr 2010 18:51:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 18:51:58 +0000
X-ASF-Spam-Status: No, hits=-1250.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 18:51:57 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id B12AB234C1EF
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 18:51:36 +0000 (UTC)
Message-ID: <962547499.18601270752696724.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 18:51:36 +0000 (UTC)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6691) TestFileSystemCaching sometimes hang
In-Reply-To: <252386459.18441270752457114.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6691:
----------------------------------

    Attachment: cachingHang.patch

This patch uses Semaphore to fix the synchronization problem.

> TestFileSystemCaching sometimes hang
> ------------------------------------
>
>                 Key: HADOOP-6691
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6691
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: cachingHang.patch
>
>
> TestFileSystemCaching#testCacheEnabledWithInitializeForeverFS() sometimes hangs if InitializeForeverFileSystem initializes first.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7279-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 19:01:59 2010
Return-Path: <common-issues-return-7279-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41442 invoked from network); 8 Apr 2010 19:01:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 19:01:59 -0000
Received: (qmail 28195 invoked by uid 500); 8 Apr 2010 19:01:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28124 invoked by uid 500); 8 Apr 2010 19:01:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28109 invoked by uid 99); 8 Apr 2010 19:01:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 19:01:58 +0000
X-ASF-Spam-Status: No, hits=-1250.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 19:01:57 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 99F24234C4AC
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 19:01:37 +0000 (UTC)
Message-ID: <344287310.19001270753297629.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 19:01:37 +0000 (UTC)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6691) TestFileSystemCaching sometimes hang
In-Reply-To: <252386459.18441270752457114.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6691:
-------------------------------------------

     Component/s: test
    Hadoop Flags: [Reviewed]

+1 patch looks good.

> TestFileSystemCaching sometimes hang
> ------------------------------------
>
>                 Key: HADOOP-6691
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6691
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: cachingHang.patch
>
>
> TestFileSystemCaching#testCacheEnabledWithInitializeForeverFS() sometimes hangs if InitializeForeverFileSystem initializes first.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7280-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 19:15:58 2010
Return-Path: <common-issues-return-7280-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45135 invoked from network); 8 Apr 2010 19:15:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 19:15:57 -0000
Received: (qmail 49272 invoked by uid 500); 8 Apr 2010 19:15:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49244 invoked by uid 500); 8 Apr 2010 19:15:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49236 invoked by uid 99); 8 Apr 2010 19:15:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 19:15:57 +0000
X-ASF-Spam-Status: No, hits=-1250.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 19:15:56 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id B58CB234C4A9
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 19:15:36 +0000 (UTC)
Message-ID: <444801227.19571270754136742.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 19:15:36 +0000 (UTC)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6687) user object in the subject in UGI
  should be reused in case of a relogin.
In-Reply-To: <191053771.27981270592254332.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855101#action_12855101 ] 

Jitendra Nath Pandey commented on HADOOP-6687:
----------------------------------------------

The patch is against yahoo-hadoop-20, which has the security code. I should have clarified that. 
It will be forward ported to trunk.

> user object in the subject in UGI  should be reused in case of a relogin.
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6687
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6687
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jitendra Nath Pandey
>         Attachments: HADOOP-6687-y20.2.patch
>
>
>  In case of a relogin, a new user object is created in the subject. This causes the subject to lose some state for example authentication method. Instead, a new user object should not be added to the subject if one is already present.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7281-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 19:18:02 2010
Return-Path: <common-issues-return-7281-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45659 invoked from network); 8 Apr 2010 19:18:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 19:18:01 -0000
Received: (qmail 57379 invoked by uid 500); 8 Apr 2010 19:18:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57329 invoked by uid 500); 8 Apr 2010 19:18:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57321 invoked by uid 99); 8 Apr 2010 19:18:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 19:18:01 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 19:17:59 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C364C234C4AE
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 19:17:36 +0000 (UTC)
Message-ID: <20253537.19651270754256799.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 19:17:36 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6691) TestFileSystemCaching sometimes
 hang
In-Reply-To: <252386459.18441270752457114.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855105#action_12855105 ] 

Hadoop QA commented on HADOOP-6691:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12441194/cachingHang.patch
  against trunk revision 931226.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/449/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/449/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/449/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/449/console

This message is automatically generated.

> TestFileSystemCaching sometimes hang
> ------------------------------------
>
>                 Key: HADOOP-6691
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6691
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: cachingHang.patch
>
>
> TestFileSystemCaching#testCacheEnabledWithInitializeForeverFS() sometimes hangs if InitializeForeverFileSystem initializes first.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7282-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 21:33:00 2010
Return-Path: <common-issues-return-7282-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75172 invoked from network); 8 Apr 2010 21:33:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 21:33:00 -0000
Received: (qmail 40040 invoked by uid 500); 8 Apr 2010 21:33:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39957 invoked by uid 500); 8 Apr 2010 21:33:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39947 invoked by uid 99); 8 Apr 2010 21:33:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 21:33:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 21:32:57 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 986F6234C1EF
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 21:32:36 +0000 (UTC)
Message-ID: <1287285294.21951270762356623.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 21:32:36 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6691) TestFileSystemCaching sometimes
 hang
In-Reply-To: <252386459.18441270752457114.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855169#action_12855169 ] 

Hudson commented on HADOOP-6691:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #215 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/215/])
    HADOOP-6691. TestFileSystemCaching sometimes hangs. Contributed by Hairong Kuang.


> TestFileSystemCaching sometimes hang
> ------------------------------------
>
>                 Key: HADOOP-6691
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6691
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: cachingHang.patch
>
>
> TestFileSystemCaching#testCacheEnabledWithInitializeForeverFS() sometimes hangs if InitializeForeverFileSystem initializes first.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7283-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 21:35:00 2010
Return-Path: <common-issues-return-7283-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75718 invoked from network); 8 Apr 2010 21:35:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 21:35:00 -0000
Received: (qmail 41088 invoked by uid 500); 8 Apr 2010 21:35:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41040 invoked by uid 500); 8 Apr 2010 21:35:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41032 invoked by uid 99); 8 Apr 2010 21:35:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 21:35:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 21:34:58 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D0D6A234C1EF
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 21:34:36 +0000 (UTC)
Message-ID: <1104971507.22061270762476854.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 21:34:36 +0000 (UTC)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6691) TestFileSystemCaching sometimes hang
In-Reply-To: <252386459.18441270752457114.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6691:
----------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I've just committed this!

> TestFileSystemCaching sometimes hang
> ------------------------------------
>
>                 Key: HADOOP-6691
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6691
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: cachingHang.patch
>
>
> TestFileSystemCaching#testCacheEnabledWithInitializeForeverFS() sometimes hangs if InitializeForeverFileSystem initializes first.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7284-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 21:37:04 2010
Return-Path: <common-issues-return-7284-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76428 invoked from network); 8 Apr 2010 21:37:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 21:37:04 -0000
Received: (qmail 44515 invoked by uid 500); 8 Apr 2010 21:37:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44478 invoked by uid 500); 8 Apr 2010 21:37:02 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44456 invoked by uid 99); 8 Apr 2010 21:37:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 21:37:02 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 21:36:59 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id B5F7F234C4AD
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 21:36:36 +0000 (UTC)
Message-ID: <253397745.22171270762596744.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 21:36:36 +0000 (UTC)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6692) Propose change to
 FileContext#listStatus
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Propose change to FileContext#listStatus
----------------------------------------

                 Key: HADOOP-6692
                 URL: https://issues.apache.org/jira/browse/HADOOP-6692
             Project: Hadoop Common
          Issue Type: Sub-task
            Reporter: Hairong Kuang
            Assignee: Hairong Kuang


Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7285-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 21:46:58 2010
Return-Path: <common-issues-return-7285-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77730 invoked from network); 8 Apr 2010 21:46:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 21:46:57 -0000
Received: (qmail 57573 invoked by uid 500); 8 Apr 2010 21:46:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57547 invoked by uid 500); 8 Apr 2010 21:46:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57538 invoked by uid 99); 8 Apr 2010 21:46:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 21:46:57 +0000
X-ASF-Spam-Status: No, hits=-1251.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 21:46:56 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id ABCFD234C48D
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 21:46:36 +0000 (UTC)
Message-ID: <1813599937.22271270763196702.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 21:46:36 +0000 (UTC)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6692) Propose change to
 FileContext#listStatus
In-Reply-To: <253397745.22171270762596744.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6692:
----------------------------------

    Attachment: listStatusItor.patch

This patch:
# add Iterator<FileStatus> listStatus(Path) to FileContext;
# move FileStatus[] listStatus(Path) to FileContext#Util;
# add Iterator<FileStatus> listStatusItor(Path) to AbstractFileSystem which provides a default implementation by using FileStatus[] listStatus(Path);
# add unit test to test Iterator<FileStatus> listStatus(Path).

> Propose change to FileContext#listStatus
> ----------------------------------------
>
>                 Key: HADOOP-6692
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6692
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>         Attachments: listStatusItor.patch
>
>
> Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7286-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 21:47:02 2010
Return-Path: <common-issues-return-7286-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77796 invoked from network); 8 Apr 2010 21:47:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 21:47:01 -0000
Received: (qmail 57775 invoked by uid 500); 8 Apr 2010 21:47:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57695 invoked by uid 500); 8 Apr 2010 21:47:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57678 invoked by uid 99); 8 Apr 2010 21:47:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 21:47:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 21:46:57 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id AEA35234C4AB
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 21:46:36 +0000 (UTC)
Message-ID: <1138587062.22291270763196714.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 21:46:36 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6693) Add metrics to track kerberos login
 activity
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Add metrics to track kerberos login activity
--------------------------------------------

                 Key: HADOOP-6693
                 URL: https://issues.apache.org/jira/browse/HADOOP-6693
             Project: Hadoop Common
          Issue Type: Improvement
    Affects Versions: 0.22.0
            Reporter: Suresh Srinivas
            Assignee: Suresh Srinivas
             Fix For: 0.22.0


Need metrics to track kerberos login activity such as login rate and the time taken for login.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7287-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 21:50:58 2010
Return-Path: <common-issues-return-7287-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78767 invoked from network); 8 Apr 2010 21:50:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 21:50:58 -0000
Received: (qmail 68858 invoked by uid 500); 8 Apr 2010 21:50:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68799 invoked by uid 500); 8 Apr 2010 21:50:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68791 invoked by uid 99); 8 Apr 2010 21:50:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 21:50:58 +0000
X-ASF-Spam-Status: No, hits=-1251.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 21:50:57 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DD5F3234C4AC
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 21:50:36 +0000 (UTC)
Message-ID: <428715501.22391270763436905.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 21:50:36 +0000 (UTC)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6678) Propose some changes to FileContext
In-Reply-To: <360625036.668801270253667713.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855178#action_12855178 ] 

Hairong Kuang commented on HADOOP-6678:
---------------------------------------

Thanks Eli for working on part 2. I will review it.

Since Eli submitted the patch to part 2 to this jira, I created HADOOP-6692 for the listStatus change.

> Propose some changes to FileContext
> -----------------------------------
>
>                 Key: HADOOP-6678
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6678
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Hairong Kuang
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6678-1.patch
>
>
> # Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.
> # Remove methods isFile(Path), isDirectory(Path), and exists.
> All these methods are implemented by calling getFileStatus(Path).But most users are not aware of this. They would write code as below: 
> {code}
>   FileContext fc = ..;
>   if (fc.exists(path)) {
>     if (fc.isFile(path)) {
>      ...
>     } else {
>     ...
>     }
>   }
> {code}
> The above code adds unnecessary getFileInfo RPC to NameNode. In our production clusters, we often see that the number of getFileStatus calls is multiple times of the open calls. If we remove isFile, isDirectory, and exists from FileContext, users have to explicitly call getFileStatus first, it is more likely that they will write more efficient code as follow:
> {code}
>   FileContext fc = ...;
>   FileStatus fstatus = fc.getFileStatus(path);
>   if (fstatus.isFile() {
>     ...
>   } else {
>     ...
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7288-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 22:17:00 2010
Return-Path: <common-issues-return-7288-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85177 invoked from network); 8 Apr 2010 22:17:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 22:17:00 -0000
Received: (qmail 8091 invoked by uid 500); 8 Apr 2010 22:17:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8063 invoked by uid 500); 8 Apr 2010 22:17:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8055 invoked by uid 99); 8 Apr 2010 22:17:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 22:17:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 22:16:57 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id A319D234C4AB
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 22:16:36 +0000 (UTC)
Message-ID: <2028330217.22821270764996667.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 22:16:36 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6693) Add metrics to track kerberos login
 activity
In-Reply-To: <1138587062.22291270763196714.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6693:
------------------------------------

    Attachment: HADOOP-6693.patch

Patch adds new metrics "login" of type MetricTimeVaryingRate under metrics context "ugi" and metrics record "ugi".

> Add metrics to track kerberos login activity
> --------------------------------------------
>
>                 Key: HADOOP-6693
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6693
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6693.patch
>
>
> Need metrics to track kerberos login activity such as login rate and the time taken for login.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7289-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 22:37:01 2010
Return-Path: <common-issues-return-7289-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87611 invoked from network); 8 Apr 2010 22:37:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 22:37:00 -0000
Received: (qmail 25641 invoked by uid 500); 8 Apr 2010 22:37:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25612 invoked by uid 500); 8 Apr 2010 22:37:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25604 invoked by uid 99); 8 Apr 2010 22:37:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 22:37:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 22:36:58 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id A7D2C234C48D
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 22:36:36 +0000 (UTC)
Message-ID: <1342138473.23161270766196686.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 22:36:36 +0000 (UTC)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6585) Add FileStatus#isDirectory and isFile
In-Reply-To: <649415366.425221266809667869.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6585:
--------------------------------

    Attachment: hadoop-6585-1.patch

Patch attached. The change is fairly straightforward since almost all uses of isDir are called on FileStatus object retrieve using getFileStatus. Since getFileStatus fully resolves links the resulting status is either a file or a directory so !isDir can simply be replaced with isFile. Since FileSystem does not support symlinks isDir is equivalent to isDirectory, and isFile is equivalent to !isDir so those changes are straightforward as well.  This patch applies against trunk with the patches for HADOOP-6563, and HADOOP-6678 applied.

> Add FileStatus#isDirectory and isFile
> -------------------------------------
>
>                 Key: HADOOP-6585
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6585
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6585-1.patch
>
>
> Per Sanjay's suggestion in HADOOP-6421 let's deprecate FileStatus#isDir() and add isDirectory() and isFile() to compliment isSymlink. Currently clients assume !isDir() implies a file, which is no longer true with symlinks. I'll file a separate jira to change the various uses of !isDir() to be isFile() or isFile() or isSymlink() as appropriate.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7290-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 23:33:00 2010
Return-Path: <common-issues-return-7290-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95887 invoked from network); 8 Apr 2010 23:33:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 23:33:00 -0000
Received: (qmail 67979 invoked by uid 500); 8 Apr 2010 23:33:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67947 invoked by uid 500); 8 Apr 2010 23:33:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67939 invoked by uid 99); 8 Apr 2010 23:33:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 23:33:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 23:32:57 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EF48D234C4A9
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 23:32:36 +0000 (UTC)
Message-ID: <2134993227.24211270769556979.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 23:32:36 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6658) Exclude Private elements from
 generated Javadoc
In-Reply-To: <995349254.452871269404067173.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6658:
------------------------------

    Attachment: HADOOP-6658.patch

New version of the doclets that takes a option to specify the stability: -unstable (the default), -evolving, -stable. For example, to exclude evolving APIs you would specify -stable. This is useful for using JDiff to examine compatibility differences.

> Exclude Private elements from generated Javadoc
> -----------------------------------------------
>
>                 Key: HADOOP-6658
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6658
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch
>
>
> Packages, classes and methods that are marked with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation should not appear in the public Javadoc. Developer Javadoc generated using the "javadoc-dev" ant target should continue to generate Javadoc for all program elements.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7291-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 23:42:58 2010
Return-Path: <common-issues-return-7291-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96836 invoked from network); 8 Apr 2010 23:42:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 23:42:57 -0000
Received: (qmail 71029 invoked by uid 500); 8 Apr 2010 23:42:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70979 invoked by uid 500); 8 Apr 2010 23:42:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70971 invoked by uid 99); 8 Apr 2010 23:42:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 23:42:57 +0000
X-ASF-Spam-Status: No, hits=-1252.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 23:42:56 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id A940D234C4AC
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 23:42:36 +0000 (UTC)
Message-ID: <407286493.24481270770156692.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 23:42:36 +0000 (UTC)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6678) Propose some changes to FileContext
In-Reply-To: <360625036.668801270253667713.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855211#action_12855211 ] 

Hairong Kuang commented on HADOOP-6678:
---------------------------------------

+1 on Eli's patch.

> Propose some changes to FileContext
> -----------------------------------
>
>                 Key: HADOOP-6678
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6678
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Hairong Kuang
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6678-1.patch
>
>
> # Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.
> # Remove methods isFile(Path), isDirectory(Path), and exists.
> All these methods are implemented by calling getFileStatus(Path).But most users are not aware of this. They would write code as below: 
> {code}
>   FileContext fc = ..;
>   if (fc.exists(path)) {
>     if (fc.isFile(path)) {
>      ...
>     } else {
>     ...
>     }
>   }
> {code}
> The above code adds unnecessary getFileInfo RPC to NameNode. In our production clusters, we often see that the number of getFileStatus calls is multiple times of the open calls. If we remove isFile, isDirectory, and exists from FileContext, users have to explicitly call getFileStatus first, it is more likely that they will write more efficient code as follow:
> {code}
>   FileContext fc = ...;
>   FileStatus fstatus = fc.getFileStatus(path);
>   if (fstatus.isFile() {
>     ...
>   } else {
>     ...
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7292-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 23:43:00 2010
Return-Path: <common-issues-return-7292-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96886 invoked from network); 8 Apr 2010 23:43:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 23:43:00 -0000
Received: (qmail 71141 invoked by uid 500); 8 Apr 2010 23:43:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71111 invoked by uid 500); 8 Apr 2010 23:43:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71103 invoked by uid 99); 8 Apr 2010 23:43:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 23:43:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 23:42:57 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C1ABC234C4B5
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 23:42:36 +0000 (UTC)
Message-ID: <1085347214.24541270770156792.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 23:42:36 +0000 (UTC)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6678) Propose some changes to FileContext
In-Reply-To: <360625036.668801270253667713.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang reassigned HADOOP-6678:
-------------------------------------

    Assignee: Eli Collins

> Propose some changes to FileContext
> -----------------------------------
>
>                 Key: HADOOP-6678
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6678
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Hairong Kuang
>            Assignee: Eli Collins
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6678-1.patch
>
>
> # Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.
> # Remove methods isFile(Path), isDirectory(Path), and exists.
> All these methods are implemented by calling getFileStatus(Path).But most users are not aware of this. They would write code as below: 
> {code}
>   FileContext fc = ..;
>   if (fc.exists(path)) {
>     if (fc.isFile(path)) {
>      ...
>     } else {
>     ...
>     }
>   }
> {code}
> The above code adds unnecessary getFileInfo RPC to NameNode. In our production clusters, we often see that the number of getFileStatus calls is multiple times of the open calls. If we remove isFile, isDirectory, and exists from FileContext, users have to explicitly call getFileStatus first, it is more likely that they will write more efficient code as follow:
> {code}
>   FileContext fc = ...;
>   FileStatus fstatus = fc.getFileStatus(path);
>   if (fstatus.isFile() {
>     ...
>   } else {
>     ...
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7293-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 23:44:58 2010
Return-Path: <common-issues-return-7293-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97222 invoked from network); 8 Apr 2010 23:44:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 23:44:57 -0000
Received: (qmail 71917 invoked by uid 500); 8 Apr 2010 23:44:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71855 invoked by uid 500); 8 Apr 2010 23:44:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71847 invoked by uid 99); 8 Apr 2010 23:44:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 23:44:57 +0000
X-ASF-Spam-Status: No, hits=-1252.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 23:44:56 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id ACA48234C4AE
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 23:44:36 +0000 (UTC)
Message-ID: <2093025351.24621270770276706.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 23:44:36 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6658) Exclude Private elements from
 generated Javadoc
In-Reply-To: <995349254.452871269404067173.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6658:
------------------------------

    Status: Patch Available  (was: Open)

> Exclude Private elements from generated Javadoc
> -----------------------------------------------
>
>                 Key: HADOOP-6658
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6658
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch
>
>
> Packages, classes and methods that are marked with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation should not appear in the public Javadoc. Developer Javadoc generated using the "javadoc-dev" ant target should continue to generate Javadoc for all program elements.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7294-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 23:44:58 2010
Return-Path: <common-issues-return-7294-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97234 invoked from network); 8 Apr 2010 23:44:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 23:44:57 -0000
Received: (qmail 72064 invoked by uid 500); 8 Apr 2010 23:44:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71985 invoked by uid 500); 8 Apr 2010 23:44:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71967 invoked by uid 99); 8 Apr 2010 23:44:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 23:44:57 +0000
X-ASF-Spam-Status: No, hits=-1252.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 23:44:56 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id A7256234C4A9
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 23:44:36 +0000 (UTC)
Message-ID: <1244144472.24581270770276683.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 23:44:36 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6658) Exclude Private elements from
 generated Javadoc
In-Reply-To: <995349254.452871269404067173.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6658:
------------------------------

    Status: Open  (was: Patch Available)

> Exclude Private elements from generated Javadoc
> -----------------------------------------------
>
>                 Key: HADOOP-6658
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6658
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch
>
>
> Packages, classes and methods that are marked with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation should not appear in the public Javadoc. Developer Javadoc generated using the "javadoc-dev" ant target should continue to generate Javadoc for all program elements.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7295-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 08 23:49:00 2010
Return-Path: <common-issues-return-7295-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97631 invoked from network); 8 Apr 2010 23:49:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Apr 2010 23:49:00 -0000
Received: (qmail 73537 invoked by uid 500); 8 Apr 2010 23:49:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73500 invoked by uid 500); 8 Apr 2010 23:49:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73492 invoked by uid 99); 8 Apr 2010 23:49:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 23:49:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Apr 2010 23:48:57 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 9D321234C1EF
	for <common-issues@hadoop.apache.org>; Thu,  8 Apr 2010 23:48:36 +0000 (UTC)
Message-ID: <1694183112.24671270770516642.JavaMail.jira@brutus.apache.org>
Date: Thu, 8 Apr 2010 23:48:36 +0000 (UTC)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6694) Implement listStatus that returns an
 Iterator of FileStatus
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Implement listStatus that returns an Iterator of FileStatus 
------------------------------------------------------------

                 Key: HADOOP-6694
                 URL: https://issues.apache.org/jira/browse/HADOOP-6694
             Project: Hadoop Common
          Issue Type: New Feature
    Affects Versions: 0.22.0
            Reporter: Hairong Kuang
            Assignee: Hairong Kuang
             Fix For: 0.22.0


This jira is for an efficient implementation of the method Iterator<FileStatus> listStatus(Path) in HDFS.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7296-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 00:04:58 2010
Return-Path: <common-issues-return-7296-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 481 invoked from network); 9 Apr 2010 00:04:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 00:04:58 -0000
Received: (qmail 87018 invoked by uid 500); 9 Apr 2010 00:04:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86985 invoked by uid 500); 9 Apr 2010 00:04:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86977 invoked by uid 99); 9 Apr 2010 00:04:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 00:04:58 +0000
X-ASF-Spam-Status: No, hits=-1252.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 00:04:57 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 2E5B2234C4C2
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 00:04:37 +0000 (UTC)
Message-ID: <1875694320.25121270771477189.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 00:04:37 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6668:
------------------------------

    Status: Patch Available  (was: Open)

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7297-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 00:05:01 2010
Return-Path: <common-issues-return-7297-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 584 invoked from network); 9 Apr 2010 00:05:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 00:05:00 -0000
Received: (qmail 87220 invoked by uid 500); 9 Apr 2010 00:05:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87188 invoked by uid 500); 9 Apr 2010 00:05:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87167 invoked by uid 99); 9 Apr 2010 00:05:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 00:05:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 00:04:58 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 23EBE234C4BB
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 00:04:37 +0000 (UTC)
Message-ID: <1515433683.25051270771477146.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 00:04:37 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6668:
------------------------------

    Status: Open  (was: Patch Available)

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7298-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 00:05:02 2010
Return-Path: <common-issues-return-7298-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 723 invoked from network); 9 Apr 2010 00:05:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 00:05:02 -0000
Received: (qmail 87355 invoked by uid 500); 9 Apr 2010 00:05:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87323 invoked by uid 500); 9 Apr 2010 00:05:02 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87315 invoked by uid 99); 9 Apr 2010 00:05:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 00:05:02 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 00:04:58 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0A07F234C4AD
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 00:04:37 +0000 (UTC)
Message-ID: <1438450118.24941270771477040.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 00:04:37 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6668:
------------------------------

    Attachment: HADOOP-6668.patch

New patch with a few more annotations.

With this patch and a similar backport to the 0.20 branch, the differences with the 0.20 (generated by HADOOP-6658 and a modified JDiff that doesn't show compatible changes) look like this: http://people.apache.org/~tomwhite/HADOOP-6668/docs/jdiff/changes.html. 

These are a mixture of compatible changes (that look incompatible to JDiff, e.g. the ones in org.apache.hadoop.io.compress), or elements that were deprecated in 0.20 and have been removed in trunk, e.g. FileSystem#delete(Path). Do we need to restore these deprecated methods in trunk for compatibilities sake in the forthcoming 0.21 release?

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7299-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 00:09:00 2010
Return-Path: <common-issues-return-7299-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1194 invoked from network); 9 Apr 2010 00:09:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 00:09:00 -0000
Received: (qmail 89179 invoked by uid 500); 9 Apr 2010 00:09:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89146 invoked by uid 500); 9 Apr 2010 00:09:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89138 invoked by uid 99); 9 Apr 2010 00:09:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 00:09:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 00:08:58 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id AED2C234C4AE
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 00:08:36 +0000 (UTC)
Message-ID: <1797423329.25201270771716715.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 00:08:36 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6658) Exclude Private elements from
 generated Javadoc
In-Reply-To: <995349254.452871269404067173.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855220#action_12855220 ] 

Hadoop QA commented on HADOOP-6658:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12441232/HADOOP-6658.patch
  against trunk revision 932115.

    +1 @author.  The patch does not contain any @author tags.

    +0 tests included.  The patch appears to be a documentation patch that doesn't require tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/450/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/450/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/450/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/450/console

This message is automatically generated.

> Exclude Private elements from generated Javadoc
> -----------------------------------------------
>
>                 Key: HADOOP-6658
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6658
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch
>
>
> Packages, classes and methods that are marked with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation should not appear in the public Javadoc. Developer Javadoc generated using the "javadoc-dev" ant target should continue to generate Javadoc for all program elements.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7300-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 00:27:01 2010
Return-Path: <common-issues-return-7300-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7793 invoked from network); 9 Apr 2010 00:27:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 00:27:00 -0000
Received: (qmail 4531 invoked by uid 500); 9 Apr 2010 00:27:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4500 invoked by uid 500); 9 Apr 2010 00:27:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4487 invoked by uid 99); 9 Apr 2010 00:27:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 00:27:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 00:26:58 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D7CA6234C4AE
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 00:26:36 +0000 (UTC)
Message-ID: <721456456.25481270772796882.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 00:26:36 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855223#action_12855223 ] 

Hadoop QA commented on HADOOP-6668:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12441236/HADOOP-6668.patch
  against trunk revision 932115.

    +1 @author.  The patch does not contain any @author tags.

    +0 tests included.  The patch appears to be a documentation patch that doesn't require tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/451/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/451/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/451/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/451/console

This message is automatically generated.

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7301-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 02:17:15 2010
Return-Path: <common-issues-return-7301-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28538 invoked from network); 9 Apr 2010 02:17:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 02:17:14 -0000
Received: (qmail 94467 invoked by uid 500); 9 Apr 2010 02:17:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94428 invoked by uid 500); 9 Apr 2010 02:17:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94420 invoked by uid 99); 9 Apr 2010 02:17:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 02:17:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 02:17:12 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D4951234C4AD
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 02:16:50 +0000 (UTC)
Message-ID: <1143585732.201270779410869.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 02:16:50 +0000 (UTC)
From: "Xiao Kang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6683) the first optimization:
 ZlibCompressor does not fully utilize the buffer
In-Reply-To: <2121624364.706511270518987284.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xiao Kang updated HADOOP-6683:
------------------------------

    Release Note: Improve the buffer utilization of ZlibCompressor to avoid invoking a JNI per write request.
          Status: Patch Available  (was: Open)

> the first optimization: ZlibCompressor does not fully utilize the buffer
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6683
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6683
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Xiao Kang
>         Attachments: ZlibCompressor.java.patch
>
>
> Thanks for Hong Tang's advice.
> Sub task created for the first optimization. HADOOP-6662 closed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7302-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 02:40:15 2010
Return-Path: <common-issues-return-7302-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33050 invoked from network); 9 Apr 2010 02:40:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 02:40:15 -0000
Received: (qmail 11863 invoked by uid 500); 9 Apr 2010 02:40:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11812 invoked by uid 500); 9 Apr 2010 02:40:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11804 invoked by uid 99); 9 Apr 2010 02:40:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 02:40:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 02:40:12 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CF9D5234C4AD
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 02:39:50 +0000 (UTC)
Message-ID: <1635961066.351270780790848.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 02:39:50 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6683) the first optimization:
 ZlibCompressor does not fully utilize the buffer
In-Reply-To: <2121624364.706511270518987284.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855255#action_12855255 ] 

Hadoop QA commented on HADOOP-6683:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12440826/ZlibCompressor.java.patch
  against trunk revision 932115.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/452/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/452/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/452/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/452/console

This message is automatically generated.

> the first optimization: ZlibCompressor does not fully utilize the buffer
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6683
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6683
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Xiao Kang
>         Attachments: ZlibCompressor.java.patch
>
>
> Thanks for Hong Tang's advice.
> Sub task created for the first optimization. HADOOP-6662 closed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7303-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 04:46:13 2010
Return-Path: <common-issues-return-7303-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48723 invoked from network); 9 Apr 2010 04:46:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 04:46:13 -0000
Received: (qmail 73403 invoked by uid 500); 9 Apr 2010 04:46:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72937 invoked by uid 500); 9 Apr 2010 04:46:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72618 invoked by uid 99); 9 Apr 2010 04:46:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 04:46:11 +0000
X-ASF-Spam-Status: No, hits=-1253.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 04:46:11 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C6A4329A0011
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 04:45:50 +0000 (UTC)
Message-ID: <679324412.961270788350812.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 04:45:50 +0000 (UTC)
From: "Ted Yu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6695) Job Tracker should be able to refresh
 classpath without restarting
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Job Tracker should be able to refresh classpath without restarting
------------------------------------------------------------------

                 Key: HADOOP-6695
                 URL: https://issues.apache.org/jira/browse/HADOOP-6695
             Project: Hadoop Common
          Issue Type: Improvement
          Components: conf
    Affects Versions: 0.20.1
            Reporter: Ted Yu


When I tried to run HBase export in cluster mode, I copied zookeeper-3.2.1.jar to hadoop lib directory.
But initial attempts failed with java.lang.ClassNotFoundException because job tracker wasn't restarted.

It is desirable for job tracker to pick up new classpath without restarting which is sometimes hard to do in production environment.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7304-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 06:52:21 2010
Return-Path: <common-issues-return-7304-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70660 invoked from network); 9 Apr 2010 06:52:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 06:52:21 -0000
Received: (qmail 59476 invoked by uid 500); 9 Apr 2010 06:52:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59361 invoked by uid 500); 9 Apr 2010 06:52:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59353 invoked by uid 99); 9 Apr 2010 06:52:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 06:52:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 06:52:16 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 23D4B29A0011
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 06:51:55 +0000 (UTC)
Message-ID: <267539718.2781270795915145.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 06:51:55 +0000 (UTC)
From: "eric baldeschwieler (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6659) Switch RPC to use Avro
In-Reply-To: <387448273.476441269469290584.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855298#action_12855298 ] 

eric baldeschwieler commented on HADOOP-6659:
---------------------------------------------

Hi Doug,

-1

The RPC system is fundamental to hadoop's stability.  Until your proposal has been formally documented, including an IDL, which I think is critical to the stated goal of backward compatibility, I'm not going to be comfortable seeing it committed.  Beyond understanding the design, you need a testing plan and test results to review.  Without this step, you will be throwing a huge tax onto others in the community.  We have the downside of all this change and have to work hard to debug your work, yet we see no benefit.

If you want to perform radical surgery on Hadoop, you need to convince the community that taking your work is worth the risk you are imposing on us.  Bugs in Hadoop can take down businesses.  In yahoo's case, they can cost us millions of dollars in lost revenue.

Perhaps you can proceed as you suggest on a branch where you will not disrupt other work or impose an unfair testing burden on the project?

E14


> Switch RPC to use Avro
> ----------------------
>
>                 Key: HADOOP-6659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6659
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Doug Cutting
>
> This is an umbrella issue for moving HDFS and MapReduce RPC to use Avro.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7305-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 08:32:15 2010
Return-Path: <common-issues-return-7305-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87086 invoked from network); 9 Apr 2010 08:32:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 08:32:15 -0000
Received: (qmail 73834 invoked by uid 500); 9 Apr 2010 08:32:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73705 invoked by uid 500); 9 Apr 2010 08:32:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73697 invoked by uid 99); 9 Apr 2010 08:32:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 08:32:12 +0000
X-ASF-Spam-Status: No, hits=-1254.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 08:32:11 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 7412029A0013
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 08:31:51 +0000 (UTC)
Message-ID: <1833832220.3641270801911474.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 08:31:51 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-3262) make Hadoop compile under Apache
 Harmony
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-3262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855317#action_12855317 ] 

Steve Loughran commented on HADOOP-3262:
----------------------------------------

Note that although the original bug HARMONY-5807 is fixed, until Harmony moves to Java 6, Hadoop will not compile under it.

> make Hadoop compile under Apache Harmony
> ----------------------------------------
>
>                 Key: HADOOP-3262
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3262
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Doug Cutting
>         Attachments: HADOOP-3262.patch
>
>
> Some small changes are required to get Hadoop Core to compile with Apache Harmony.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7306-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 11:28:16 2010
Return-Path: <common-issues-return-7306-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16321 invoked from network); 9 Apr 2010 11:28:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 11:28:16 -0000
Received: (qmail 39783 invoked by uid 500); 9 Apr 2010 11:28:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39673 invoked by uid 500); 9 Apr 2010 11:28:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39665 invoked by uid 99); 9 Apr 2010 11:28:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 11:28:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 11:28:11 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DAE3F29A0014
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 11:27:50 +0000 (UTC)
Message-ID: <921469300.5171270812470895.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 11:27:50 +0000 (UTC)
From: "Xiao Kang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6683) the first optimization:
 ZlibCompressor does not fully utilize the buffer
In-Reply-To: <2121624364.706511270518987284.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855353#action_12855353 ] 

Xiao Kang commented on HADOOP-6683:
-----------------------------------

This patch does not add any new function and he test case src/test/org/apache/hadoop/core/io/compress/TestCodec.java has covered this patch. 

> the first optimization: ZlibCompressor does not fully utilize the buffer
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6683
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6683
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Xiao Kang
>         Attachments: ZlibCompressor.java.patch
>
>
> Thanks for Hong Tang's advice.
> Sub task created for the first optimization. HADOOP-6662 closed. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7307-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 11:42:16 2010
Return-Path: <common-issues-return-7307-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18425 invoked from network); 9 Apr 2010 11:42:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 11:42:15 -0000
Received: (qmail 49309 invoked by uid 500); 9 Apr 2010 11:42:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49247 invoked by uid 500); 9 Apr 2010 11:42:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49236 invoked by uid 99); 9 Apr 2010 11:42:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 11:42:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 11:42:12 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C919E234C1EF
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 11:41:51 +0000 (UTC)
Message-ID: <654841781.5281270813311822.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 11:41:51 +0000 (UTC)
From: "Andrew Klochkov (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6681) Fill AWS credentials when
 configuring Hadoop on EC2 instances
In-Reply-To: <315596123.688091270479567264.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855356#action_12855356 ] 

Andrew Klochkov commented on HADOOP-6681:
-----------------------------------------

Yes, it does solve the issue. 

> Fill AWS credentials when configuring Hadoop on EC2 instances
> -------------------------------------------------------------
>
>                 Key: HADOOP-6681
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6681
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: contrib/cloud
>            Reporter: Andrew Klochkov
>         Attachments: HADOOP-6681.patch
>
>
> There's a function "configure_hadoop" in the hadoop-ec2-init-remote.sh script used to configure EC2 nodes for Hadoop. The function actually uses AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, but they are never passed to it. It can be fixed in service.py by passing those variables.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7308-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 11:50:15 2010
Return-Path: <common-issues-return-7308-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19742 invoked from network); 9 Apr 2010 11:50:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 11:50:15 -0000
Received: (qmail 58762 invoked by uid 500); 9 Apr 2010 11:50:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58708 invoked by uid 500); 9 Apr 2010 11:50:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58677 invoked by uid 99); 9 Apr 2010 11:50:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 11:50:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 11:50:12 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 2A3D729A0011
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 11:49:51 +0000 (UTC)
Message-ID: <992554534.5321270813791171.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 11:49:51 +0000 (UTC)
From: "Andrew Klochkov (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6696) Use contrib/cloud scripts for
 hardware cluster deployment
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Use contrib/cloud scripts for hardware cluster deployment
---------------------------------------------------------

                 Key: HADOOP-6696
                 URL: https://issues.apache.org/jira/browse/HADOOP-6696
             Project: Hadoop Common
          Issue Type: Improvement
          Components: contrib/cloud
            Reporter: Andrew Klochkov
            Assignee: Andrew Klochkov


contrib/cloud scripts can be used as an easy way to deploy Hadoop on a hardware cluster, by implementing a "fixed" provider. Here "fixed" means it uses a preconfigured set of hosts to deploy to. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7309-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 12:06:14 2010
Return-Path: <common-issues-return-7309-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22517 invoked from network); 9 Apr 2010 12:06:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 12:06:13 -0000
Received: (qmail 73556 invoked by uid 500); 9 Apr 2010 12:06:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73519 invoked by uid 500); 9 Apr 2010 12:06:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73511 invoked by uid 99); 9 Apr 2010 12:06:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 12:06:13 +0000
X-ASF-Spam-Status: No, hits=-1255.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 12:06:12 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 8B8D729A0013
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 12:05:51 +0000 (UTC)
Message-ID: <1906345555.5561270814751570.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 12:05:51 +0000 (UTC)
From: "Andrew Klochkov (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6696) Use contrib/cloud scripts for
 hardware cluster deployment
In-Reply-To: <992554534.5321270813791171.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Klochkov updated HADOOP-6696:
------------------------------------

    Attachment: HADOOP-6696.patch

Attached a patch. 

Some notes on this implementation:

1. It enables Hadoop service deployment only, it doesn't work for zookeeper. 

2. Configuration is done in $HOME/.hadoop-cloud/clusters.cfg. 

Sample config is below. It's set to run jobtracker, namenode and secondary namenode on 172.16.128.203, and run 2 slaves on 172.16.128.206 and 172.16.128.239.  Parameter NUM_SLAVES of the launch-cluster command becomes meaningless in case of fixed provider. README is not modified.

[fixed-cluster]
cloud_provider=fixed
image_id=
instance_type=
key_name=
availability_zone=
public_key=/keys/location/hadoop.pem
private_key=/keys/location/hadoop.pem
ssh_options=-i %(private_key)s -o StrictHostKeyChecking=no
nodes.nn_snn_jt=172.16.128.203
nodes.dn_tt=172.16.128.206, 172.16.128.239
env=MAX_MAP_TASKS=2
env=MAX_REDUCE_TASKS=1

3. Files hadoop-ec2-init-remote.sh and hadoop-fixed-init-remote.sh in "data" dir are almost identical. Obviously it makes sense to extract common code in a separate scripts, and later use it by other providers too.

> Use contrib/cloud scripts for hardware cluster deployment
> ---------------------------------------------------------
>
>                 Key: HADOOP-6696
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6696
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: contrib/cloud
>            Reporter: Andrew Klochkov
>            Assignee: Andrew Klochkov
>         Attachments: HADOOP-6696.patch
>
>
> contrib/cloud scripts can be used as an easy way to deploy Hadoop on a hardware cluster, by implementing a "fixed" provider. Here "fixed" means it uses a preconfigured set of hosts to deploy to. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7310-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 12:06:14 2010
Return-Path: <common-issues-return-7310-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22576 invoked from network); 9 Apr 2010 12:06:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 12:06:14 -0000
Received: (qmail 73675 invoked by uid 500); 9 Apr 2010 12:06:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73651 invoked by uid 500); 9 Apr 2010 12:06:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73643 invoked by uid 99); 9 Apr 2010 12:06:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 12:06:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 12:06:12 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 8E91F29A0015
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 12:05:51 +0000 (UTC)
Message-ID: <15390055.5581270814751583.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 12:05:51 +0000 (UTC)
From: "Andrew Klochkov (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6696) Use contrib/cloud scripts for
 hardware cluster deployment
In-Reply-To: <992554534.5321270813791171.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Klochkov updated HADOOP-6696:
------------------------------------

    Status: Patch Available  (was: Open)

> Use contrib/cloud scripts for hardware cluster deployment
> ---------------------------------------------------------
>
>                 Key: HADOOP-6696
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6696
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: contrib/cloud
>            Reporter: Andrew Klochkov
>            Assignee: Andrew Klochkov
>         Attachments: HADOOP-6696.patch
>
>
> contrib/cloud scripts can be used as an easy way to deploy Hadoop on a hardware cluster, by implementing a "fixed" provider. Here "fixed" means it uses a preconfigured set of hosts to deploy to. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7311-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 12:16:15 2010
Return-Path: <common-issues-return-7311-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24524 invoked from network); 9 Apr 2010 12:16:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 12:16:15 -0000
Received: (qmail 87841 invoked by uid 500); 9 Apr 2010 12:16:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87793 invoked by uid 500); 9 Apr 2010 12:16:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87671 invoked by uid 99); 9 Apr 2010 12:16:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 12:16:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 12:16:12 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0E1E429A0013
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 12:15:51 +0000 (UTC)
Message-ID: <1691420732.5731270815351046.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 12:15:51 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6696) Use contrib/cloud scripts for
 hardware cluster deployment
In-Reply-To: <992554534.5321270813791171.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855365#action_12855365 ] 

Hadoop QA commented on HADOOP-6696:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12441291/HADOOP-6696.patch
  against trunk revision 932115.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 patch.  The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/453/console

This message is automatically generated.

> Use contrib/cloud scripts for hardware cluster deployment
> ---------------------------------------------------------
>
>                 Key: HADOOP-6696
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6696
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: contrib/cloud
>            Reporter: Andrew Klochkov
>            Assignee: Andrew Klochkov
>         Attachments: HADOOP-6696.patch
>
>
> contrib/cloud scripts can be used as an easy way to deploy Hadoop on a hardware cluster, by implementing a "fixed" provider. Here "fixed" means it uses a preconfigured set of hosts to deploy to. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7312-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 12:42:15 2010
Return-Path: <common-issues-return-7312-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36676 invoked from network); 9 Apr 2010 12:42:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 12:42:14 -0000
Received: (qmail 7149 invoked by uid 500); 9 Apr 2010 12:42:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7117 invoked by uid 500); 9 Apr 2010 12:42:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7105 invoked by uid 99); 9 Apr 2010 12:42:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 12:42:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 12:42:12 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C4108234C1EF
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 12:41:50 +0000 (UTC)
Message-ID: <1338746519.5911270816910801.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 12:41:50 +0000 (UTC)
From: "Andrew Klochkov (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6696) Use contrib/cloud scripts for
 hardware cluster deployment
In-Reply-To: <992554534.5321270813791171.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Klochkov updated HADOOP-6696:
------------------------------------

    Status: Open  (was: Patch Available)

> Use contrib/cloud scripts for hardware cluster deployment
> ---------------------------------------------------------
>
>                 Key: HADOOP-6696
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6696
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: contrib/cloud
>            Reporter: Andrew Klochkov
>            Assignee: Andrew Klochkov
>         Attachments: HADOOP-6696.patch
>
>
> contrib/cloud scripts can be used as an easy way to deploy Hadoop on a hardware cluster, by implementing a "fixed" provider. Here "fixed" means it uses a preconfigured set of hosts to deploy to. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7313-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 12:44:12 2010
Return-Path: <common-issues-return-7313-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37193 invoked from network); 9 Apr 2010 12:44:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 12:44:12 -0000
Received: (qmail 14455 invoked by uid 500); 9 Apr 2010 12:44:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14429 invoked by uid 500); 9 Apr 2010 12:44:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14421 invoked by uid 99); 9 Apr 2010 12:44:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 12:44:11 +0000
X-ASF-Spam-Status: No, hits=-1255.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 12:44:11 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DB804234C1EF
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 12:43:50 +0000 (UTC)
Message-ID: <1255161641.5931270817030897.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 12:43:50 +0000 (UTC)
From: "Andrew Klochkov (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6696) Use contrib/cloud scripts for
 hardware cluster deployment
In-Reply-To: <992554534.5321270813791171.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Klochkov updated HADOOP-6696:
------------------------------------

    Attachment: HADOOP-6696.patch

First patch was incorrect. Attaching a correct one.

> Use contrib/cloud scripts for hardware cluster deployment
> ---------------------------------------------------------
>
>                 Key: HADOOP-6696
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6696
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: contrib/cloud
>            Reporter: Andrew Klochkov
>            Assignee: Andrew Klochkov
>         Attachments: HADOOP-6696.patch, HADOOP-6696.patch
>
>
> contrib/cloud scripts can be used as an easy way to deploy Hadoop on a hardware cluster, by implementing a "fixed" provider. Here "fixed" means it uses a preconfigured set of hosts to deploy to. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7314-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 13:04:14 2010
Return-Path: <common-issues-return-7314-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43156 invoked from network); 9 Apr 2010 13:04:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 13:04:14 -0000
Received: (qmail 54458 invoked by uid 500); 9 Apr 2010 13:04:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54423 invoked by uid 500); 9 Apr 2010 13:04:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54415 invoked by uid 99); 9 Apr 2010 13:04:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 13:04:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 13:04:12 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id F2E2C234C1EF
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 13:03:50 +0000 (UTC)
Message-ID: <1561889313.6181270818230993.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 13:03:50 +0000 (UTC)
From: "Andrew Klochkov (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6696) Use contrib/cloud scripts for
 hardware cluster deployment
In-Reply-To: <992554534.5321270813791171.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Klochkov updated HADOOP-6696:
------------------------------------

    Status: Patch Available  (was: Open)

> Use contrib/cloud scripts for hardware cluster deployment
> ---------------------------------------------------------
>
>                 Key: HADOOP-6696
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6696
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: contrib/cloud
>            Reporter: Andrew Klochkov
>            Assignee: Andrew Klochkov
>         Attachments: HADOOP-6696.patch, HADOOP-6696.patch
>
>
> contrib/cloud scripts can be used as an easy way to deploy Hadoop on a hardware cluster, by implementing a "fixed" provider. Here "fixed" means it uses a preconfigured set of hosts to deploy to. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7315-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 13:30:13 2010
Return-Path: <common-issues-return-7315-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48459 invoked from network); 9 Apr 2010 13:30:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 13:30:13 -0000
Received: (qmail 79425 invoked by uid 500); 9 Apr 2010 13:30:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79338 invoked by uid 500); 9 Apr 2010 13:30:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79232 invoked by uid 99); 9 Apr 2010 13:30:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 13:30:12 +0000
X-ASF-Spam-Status: No, hits=-1255.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 13:30:11 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C0466234C1EF
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 13:29:50 +0000 (UTC)
Message-ID: <1548946120.6441270819790786.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 13:29:50 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6696) Use contrib/cloud scripts for
 hardware cluster deployment
In-Reply-To: <992554534.5321270813791171.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855383#action_12855383 ] 

Hadoop QA commented on HADOOP-6696:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12441292/HADOOP-6696.patch
  against trunk revision 932115.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/454/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/454/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/454/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/454/console

This message is automatically generated.

> Use contrib/cloud scripts for hardware cluster deployment
> ---------------------------------------------------------
>
>                 Key: HADOOP-6696
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6696
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: contrib/cloud
>            Reporter: Andrew Klochkov
>            Assignee: Andrew Klochkov
>         Attachments: HADOOP-6696.patch, HADOOP-6696.patch
>
>
> contrib/cloud scripts can be used as an easy way to deploy Hadoop on a hardware cluster, by implementing a "fixed" provider. Here "fixed" means it uses a preconfigured set of hosts to deploy to. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7316-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 15:24:19 2010
Return-Path: <common-issues-return-7316-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78752 invoked from network); 9 Apr 2010 15:24:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 15:24:14 -0000
Received: (qmail 43922 invoked by uid 500); 9 Apr 2010 15:24:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43887 invoked by uid 500); 9 Apr 2010 15:24:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43879 invoked by uid 99); 9 Apr 2010 15:24:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 15:24:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 15:24:12 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E61AF29A0014
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 15:23:50 +0000 (UTC)
Message-ID: <1757698896.8521270826630941.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 15:23:50 +0000 (UTC)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6681) Fill AWS credentials when configuring
 Hadoop on EC2 instances
In-Reply-To: <315596123.688091270479567264.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6681:
------------------------------

    Resolution: Won't Fix
        Status: Resolved  (was: Patch Available)

Thanks Andrew.

> Fill AWS credentials when configuring Hadoop on EC2 instances
> -------------------------------------------------------------
>
>                 Key: HADOOP-6681
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6681
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: contrib/cloud
>            Reporter: Andrew Klochkov
>         Attachments: HADOOP-6681.patch
>
>
> There's a function "configure_hadoop" in the hadoop-ec2-init-remote.sh script used to configure EC2 nodes for Hadoop. The function actually uses AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, but they are never passed to it. It can be fixed in service.py by passing those variables.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7317-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 16:29:13 2010
Return-Path: <common-issues-return-7317-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96844 invoked from network); 9 Apr 2010 16:29:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 16:29:13 -0000
Received: (qmail 44543 invoked by uid 500); 9 Apr 2010 16:29:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44514 invoked by uid 500); 9 Apr 2010 16:29:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44506 invoked by uid 99); 9 Apr 2010 16:29:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 16:29:12 +0000
X-ASF-Spam-Status: No, hits=-1256.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 16:29:11 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 2C77D234C4C8
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 16:28:51 +0000 (UTC)
Message-ID: <363819536.9771270830531181.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 16:28:51 +0000 (UTC)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6659) Switch RPC to use Avro
In-Reply-To: <387448273.476441269469290584.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855460#action_12855460 ] 

Doug Cutting commented on HADOOP-6659:
--------------------------------------

Eric, this work is proceeding using a "branch in place" strategy, since long-lived branches are onerous to mange.  The intent is to be largely equivalent to a branch, in that existing code will not be disturbed.  The first step was to make the RPC functionality we wish to replace pluggable.  That was committed last September in HADOOP-6170.  Now we can develop and test the new RPC implementation without disturbing the existing implementation.  There is no intent to switch to the new implementation until all concerns about it are resolved.  Some minor, non-functional changes have and will be made to the existing implementation to aid IDL inference, i.e. adding some annotations and renaming some private methods in (HDFS-1077).

> Until your proposal has been formally documented, including an IDL, which I think is critical to the stated goal of backward compatibility, I'm not going to be comfortable seeing it committed.

This issue does not attempt to fully implement cross-version wire-compatiblity.  Nor does it attempt to move Hadoop to IDL-driven RPC.  Rather it proposes to move Hadoop to a system that we can use to implement wire-compatibility and IDL-driven RPC.

Until we switch, we cannot dictate the IDL, the IDL is derived from the existing Java interfaces used to define RPC protocols.  Once we've switched to the new Avro-based implementation, then we can elect to, protocol-by-protocol, client-by-client and server-by-server, switch to an IDL-driven implementation.  (An IDL derived by the current Java interfaces is attached to HDFS-1069.  This is in JSON form, but Avro has an alternate IDL syntax that, by the time we consider switching, would be used instead.)

Changes to the protocol often imply changes to client and server logic.  This issue does not propose to alter client and server logic, but rather to track and retain the existing logic and protocols.

Before we make a release that we claim will support wire-compatibility we should certainly review our protocols carefully.  But first we need to switch an RPC system that permits clients and servers to use different protocol versions and that supports IDL-driven protocols.  That's the primary focus of this issue.  Once we switch, our wire-compatibility problems will not be automatically solved, rather we'll then have tools in place to address them.

> Beyond understanding the design, you need a testing plan and test results to review.

Yes, I agree, this should be fully tested to everyone's satisfaction before we make any switch.

> Switch RPC to use Avro
> ----------------------
>
>                 Key: HADOOP-6659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6659
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Doug Cutting
>
> This is an umbrella issue for moving HDFS and MapReduce RPC to use Avro.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7318-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 16:39:14 2010
Return-Path: <common-issues-return-7318-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98848 invoked from network); 9 Apr 2010 16:39:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 16:39:14 -0000
Received: (qmail 61271 invoked by uid 500); 9 Apr 2010 16:39:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61244 invoked by uid 500); 9 Apr 2010 16:39:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61236 invoked by uid 99); 9 Apr 2010 16:39:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 16:39:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 16:39:12 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 343EC29A0011
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 16:38:51 +0000 (UTC)
Message-ID: <314161499.9971270831131213.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 16:38:51 +0000 (UTC)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6697) script to that would let us checkout
 code from different repos
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

script to that would let us checkout code from different repos
--------------------------------------------------------------

                 Key: HADOOP-6697
                 URL: https://issues.apache.org/jira/browse/HADOOP-6697
             Project: Hadoop Common
          Issue Type: Test
          Components: build
            Reporter: Giridharan Kesavan


To write a shell script that would let us checkout code from two different repository , where we cant use svn:externals.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7319-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 17:03:14 2010
Return-Path: <common-issues-return-7319-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8441 invoked from network); 9 Apr 2010 17:03:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 17:03:14 -0000
Received: (qmail 7367 invoked by uid 500); 9 Apr 2010 17:03:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7329 invoked by uid 500); 9 Apr 2010 17:03:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7321 invoked by uid 99); 9 Apr 2010 17:03:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 17:03:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 17:03:11 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DF17429A0014
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 17:02:50 +0000 (UTC)
Message-ID: <2092997274.11291270832570912.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 17:02:50 +0000 (UTC)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6692) Propose change to
 FileContext#listStatus
In-Reply-To: <253397745.22171270762596744.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6692:
----------------------------------

        Fix Version/s: 0.22.0
    Affects Version/s: 0.22.0
               Status: Patch Available  (was: Open)

> Propose change to FileContext#listStatus
> ----------------------------------------
>
>                 Key: HADOOP-6692
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6692
>             Project: Hadoop Common
>          Issue Type: Sub-task
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listStatusItor.patch
>
>
> Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7320-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 17:20:12 2010
Return-Path: <common-issues-return-7320-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13801 invoked from network); 9 Apr 2010 17:20:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 17:20:12 -0000
Received: (qmail 32659 invoked by uid 500); 9 Apr 2010 17:20:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32621 invoked by uid 500); 9 Apr 2010 17:20:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32613 invoked by uid 99); 9 Apr 2010 17:20:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 17:20:12 +0000
X-ASF-Spam-Status: No, hits=-1256.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 17:20:11 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C868729A0014
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 17:19:50 +0000 (UTC)
Message-ID: <1473580671.11571270833590819.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 17:19:50 +0000 (UTC)
From: "Jeff Bowles (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6697) script to that would let us checkout
 code from different repos
In-Reply-To: <314161499.9971270831131213.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jeff Bowles updated HADOOP-6697:
--------------------------------

    Release Note: A simple piece of shell-script plumbing, so that you can invoke a single-script to take a specific action (loading a secondary GIT or SVN repository prior to running a build, possibly to load your build-scripts from a second-source that has different IP restrictions).
          Status: Patch Available  (was: Open)

> script to that would let us checkout code from different repos
> --------------------------------------------------------------
>
>                 Key: HADOOP-6697
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6697
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: build
>            Reporter: Giridharan Kesavan
>
> To write a shell script that would let us checkout code from two different repository , where we cant use svn:externals.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7321-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 17:22:14 2010
Return-Path: <common-issues-return-7321-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14363 invoked from network); 9 Apr 2010 17:22:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 17:22:14 -0000
Received: (qmail 34888 invoked by uid 500); 9 Apr 2010 17:22:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34851 invoked by uid 500); 9 Apr 2010 17:22:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34843 invoked by uid 99); 9 Apr 2010 17:22:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 17:22:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 17:22:11 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CCFE029A0012
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 17:21:50 +0000 (UTC)
Message-ID: <345116030.11601270833710838.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 17:21:50 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6693) Add metrics to track kerberos login
 activity
In-Reply-To: <1138587062.22291270763196714.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855478#action_12855478 ] 

Suresh Srinivas commented on HADOOP-6693:
-----------------------------------------

Note that the new metrics is aggregated across all the nodes in the cluster, since no distinction made in metrics record name about the node where the metrics was reported from.

> Add metrics to track kerberos login activity
> --------------------------------------------
>
>                 Key: HADOOP-6693
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6693
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6693.patch
>
>
> Need metrics to track kerberos login activity such as login rate and the time taken for login.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7322-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 17:30:16 2010
Return-Path: <common-issues-return-7322-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16589 invoked from network); 9 Apr 2010 17:30:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 17:30:16 -0000
Received: (qmail 43796 invoked by uid 500); 9 Apr 2010 17:30:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43752 invoked by uid 500); 9 Apr 2010 17:30:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43744 invoked by uid 99); 9 Apr 2010 17:30:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 17:30:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 17:30:12 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E9B45234C4BC
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 17:29:50 +0000 (UTC)
Message-ID: <2104668537.11881270834190956.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 17:29:50 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6692) Propose change to
 FileContext#listStatus
In-Reply-To: <253397745.22171270762596744.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855482#action_12855482 ] 

Hadoop QA commented on HADOOP-6692:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12441211/listStatusItor.patch
  against trunk revision 932115.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 9 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/455/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/455/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/455/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/455/console

This message is automatically generated.

> Propose change to FileContext#listStatus
> ----------------------------------------
>
>                 Key: HADOOP-6692
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6692
>             Project: Hadoop Common
>          Issue Type: Sub-task
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listStatusItor.patch
>
>
> Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7323-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 17:32:14 2010
Return-Path: <common-issues-return-7323-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16968 invoked from network); 9 Apr 2010 17:32:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 17:32:14 -0000
Received: (qmail 44641 invoked by uid 500); 9 Apr 2010 17:32:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44606 invoked by uid 500); 9 Apr 2010 17:32:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44598 invoked by uid 99); 9 Apr 2010 17:32:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 17:32:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 17:32:12 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DB332234C4AE
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 17:31:50 +0000 (UTC)
Message-ID: <1259330561.11961270834310897.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 17:31:50 +0000 (UTC)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4322) Input/Output Format for TFile
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855483#action_12855483 ] 

Vinod K V commented on HADOOP-4322:
-----------------------------------


When this patch is tested internally, we found some problem - the job gives out negative "input byes" count when using this outputformat. I don't have any context about this, but just updating this bug with the analysis by Rahul..

{code}

ObjectFileRecordReader has getPos() method implementation , this method is giving incorrect values for offset.

Code flow in the framework is like below.
=====
      beforePos = getPos();
      //call to user's record reader 'next() method.
      afterPos = getPos();

//then for counter we do the following:
inputByteCounter.increment(afterPos - beforePos);//this is the counter which is 
                                                 //in question 
=====

(ObjectFileRecordReader's getPos() method ) afterPos < beforePos , this is resulting in the -ve increment to the counter.
{code}

So this patch shouldn't be committed as is without a relook.

> Input/Output Format for TFile
> -----------------------------
>
>                 Key: HADOOP-4322
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4322
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Amir Youssefi
>            Assignee: Amir Youssefi
>         Attachments: ObjectFileInputOutputFormat_1.patch
>
>
> Input/Output Format for TFile

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7324-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 17:34:14 2010
Return-Path: <common-issues-return-7324-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17191 invoked from network); 9 Apr 2010 17:34:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 17:34:14 -0000
Received: (qmail 48307 invoked by uid 500); 9 Apr 2010 17:34:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48217 invoked by uid 500); 9 Apr 2010 17:34:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48148 invoked by uid 99); 9 Apr 2010 17:34:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 17:34:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 17:34:12 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 270F0234C4AD
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 17:33:51 +0000 (UTC)
Message-ID: <1649880560.12051270834431158.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 17:33:51 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6697) script to that would let us
 checkout code from different repos
In-Reply-To: <314161499.9971270831131213.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855484#action_12855484 ] 

Hadoop QA commented on HADOOP-6697:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org
  against trunk revision 932115.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 patch.  The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/456/console

This message is automatically generated.

> script to that would let us checkout code from different repos
> --------------------------------------------------------------
>
>                 Key: HADOOP-6697
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6697
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: build
>            Reporter: Giridharan Kesavan
>
> To write a shell script that would let us checkout code from two different repository , where we cant use svn:externals.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7325-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 17:42:12 2010
Return-Path: <common-issues-return-7325-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18534 invoked from network); 9 Apr 2010 17:42:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 17:42:12 -0000
Received: (qmail 61491 invoked by uid 500); 9 Apr 2010 17:42:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61449 invoked by uid 500); 9 Apr 2010 17:42:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61441 invoked by uid 99); 9 Apr 2010 17:42:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 17:42:11 +0000
X-ASF-Spam-Status: No, hits=-1257.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 17:42:11 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id F0888234C4AB
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 17:41:50 +0000 (UTC)
Message-ID: <999266844.12291270834910984.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 17:41:50 +0000 (UTC)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6697) script to that would let us
 checkout code from different repos
In-Reply-To: <314161499.9971270831131213.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855487#action_12855487 ] 

Doug Cutting commented on HADOOP-6697:
--------------------------------------

Can you explain this more?

> script to that would let us checkout code from different repos
> --------------------------------------------------------------
>
>                 Key: HADOOP-6697
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6697
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: build
>            Reporter: Giridharan Kesavan
>
> To write a shell script that would let us checkout code from two different repository , where we cant use svn:externals.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7326-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 17:58:15 2010
Return-Path: <common-issues-return-7326-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23276 invoked from network); 9 Apr 2010 17:58:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 17:58:15 -0000
Received: (qmail 96615 invoked by uid 500); 9 Apr 2010 17:58:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96575 invoked by uid 500); 9 Apr 2010 17:58:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96567 invoked by uid 99); 9 Apr 2010 17:58:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 17:58:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 17:58:12 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D08AB234C4AC
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 17:57:50 +0000 (UTC)
Message-ID: <1988987109.12681270835870853.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 17:57:50 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6686) Remove redundant exception class name
 in unwrapped exceptions thrown at the RPC client
In-Reply-To: <70650712.27801270591413554.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6686:
------------------------------------

    Attachment: HADOOP-6686.1.patch

Updated patch:
# RemoteException message is server side stack trace minus redundant exception name prefix.
# Unwrapped exceptions from remote exception has same message as RemoteException, instead of picking only the first line from the server stack trace. The unwrapped message is same as that of RemoteException message.

> Remove redundant exception class name in unwrapped exceptions thrown at the RPC client
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6686
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6686
>             Project: Hadoop Common
>          Issue Type: Improvement
>         Environment: At RPC client, when the exception thrown by the server is unwrapped, the exception message includes redundant exception class name. This redundant information should be removed.
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6686.1.patch, HADOOP-6686.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7327-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 09 18:28:12 2010
Return-Path: <common-issues-return-7327-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36380 invoked from network); 9 Apr 2010 18:28:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Apr 2010 18:28:12 -0000
Received: (qmail 35357 invoked by uid 500); 9 Apr 2010 18:28:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35255 invoked by uid 500); 9 Apr 2010 18:28:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35234 invoked by uid 99); 9 Apr 2010 18:28:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 18:28:12 +0000
X-ASF-Spam-Status: No, hits=-1257.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Apr 2010 18:28:11 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DD94C234C4AC
	for <common-issues@hadoop.apache.org>; Fri,  9 Apr 2010 18:27:50 +0000 (UTC)
Message-ID: <597761108.13231270837670906.JavaMail.jira@brutus.apache.org>
Date: Fri, 9 Apr 2010 18:27:50 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6692) Propose change to
 FileContext#listStatus
In-Reply-To: <253397745.22171270762596744.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855508#action_12855508 ] 

Suresh Srinivas commented on HADOOP-6692:
-----------------------------------------

Hairong, this is a good change.

In order to simplify transition to FileContext, should we retain the old FileContext.listStatus() method as it is and add iterative list status as listStatusIterator()? It also captures the semantics quite well.

Minor:
# FileContext.listStatus() that returns iterator -  add information about throwing UnsupportedOperationException and NoSuchElementException in the RuntimeExceptions section on attempt to delete an entry from iterator.
# FileContext.listStatus() that retuns array -  has invalid description in @return.  Also add a note to use the itertor version if the need is to iterate over the list, with a @link to the other method
# AFS.listStatusItertor() - method javadoc should point @link to the right method in FileContext.
# AFS.listStatus() - method javadoc should point @link to the right method in FileContext. Current link is invalid and is missing # prefix to listStatus().


> Propose change to FileContext#listStatus
> ----------------------------------------
>
>                 Key: HADOOP-6692
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6692
>             Project: Hadoop Common
>          Issue Type: Sub-task
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listStatusItor.patch
>
>
> Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7328-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 10 23:29:04 2010
Return-Path: <common-issues-return-7328-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16004 invoked from network); 10 Apr 2010 23:29:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Apr 2010 23:29:03 -0000
Received: (qmail 98512 invoked by uid 500); 10 Apr 2010 23:29:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98482 invoked by uid 500); 10 Apr 2010 23:29:02 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98474 invoked by uid 99); 10 Apr 2010 23:29:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Apr 2010 23:29:02 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Apr 2010 23:29:00 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3ANMnlK000830
	for <common-issues@hadoop.apache.org>; Sat, 10 Apr 2010 19:22:49 -0400 (EDT)
Message-ID: <15787287.5791270941769321.JavaMail.jira@thor>
Date: Sat, 10 Apr 2010 19:22:49 -0400 (EDT)
From: "eric baldeschwieler (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6659) Switch RPC to use Avro
In-Reply-To: <387448273.476441269469290584.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855655#action_12855655 ] 

eric baldeschwieler commented on HADOOP-6659:
---------------------------------------------

Hi Doug,

Thanks, I think I understand your "branch in place" proposal.  I think you need to understand my position.  I'm responsible for making sure the hadoop code base is stable enough that we can run our production work on it.  If we move incrementally as you propose, my customers will be exposed to a lot of risk and disruption for no return. Yahoo QA staff and developers will be pulled off what they are working on to complete this project.  That is not how we want to allocate our resources.

That is why I'm rejecting your current proposal and suggesting that you instead do more work on an actual branch, including getting the IDL based RPC scheme coded.  Once you build something that has actual return for my customers, than we will be happy to help with the final debug and tuning, but first I would like to see you complete the project.  Only then do I want to see something that effects every single RPC call incorporated into the code base.

I understand that this is not how you would like to proceed and I understand that this is not historically how you would have proceeded with a project like this in Hadoop.  But, as Hadoop evolves and the number of business built on it increase, the way we approach radical change in the code base needs to change as well.  I assert that to proceed with something as radical as this, you need the consensus of Hadoop committers behind you.  You do not have it.

Please understand that I am supportive of your desire to incorporate an AVRO based RPC into Hadoop.  I just feel that you need to take a more cautious approach.  Otherwise you tax the entire community and I don't support that.

Thanks!

> Switch RPC to use Avro
> ----------------------
>
>                 Key: HADOOP-6659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6659
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Doug Cutting
>
> This is an umbrella issue for moving HDFS and MapReduce RPC to use Avro.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7329-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 11 07:16:53 2010
Return-Path: <common-issues-return-7329-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49077 invoked from network); 11 Apr 2010 07:16:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Apr 2010 07:16:53 -0000
Received: (qmail 24606 invoked by uid 500); 11 Apr 2010 07:16:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24491 invoked by uid 500); 11 Apr 2010 07:16:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24483 invoked by uid 99); 11 Apr 2010 07:16:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 11 Apr 2010 07:16:50 +0000
X-ASF-Spam-Status: No, hits=-1261.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 11 Apr 2010 07:16:49 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3B7Afwo000863
	for <common-issues@hadoop.apache.org>; Sun, 11 Apr 2010 03:10:41 -0400 (EDT)
Message-ID: <21261745.7641270969841016.JavaMail.jira@thor>
Date: Sun, 11 Apr 2010 03:10:41 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6698) Revert the io.serialization package
 to 0.20.2's api
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Revert the io.serialization package to 0.20.2's api
---------------------------------------------------

                 Key: HADOOP-6698
                 URL: https://issues.apache.org/jira/browse/HADOOP-6698
             Project: Hadoop Common
          Issue Type: Bug
          Components: io
            Reporter: Owen O'Malley
            Priority: Blocker
             Fix For: 0.21.0


I have a lot of concern about the usability of the new generic serialization framework. Toward that end, I've filed a jira for improving it. There is resistance to pushing a new API into 0.21 at the last moment, so we should back out the changes rather than introducing a new api in 0.21 and deprecating it in 0.22.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7330-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 11 13:22:52 2010
Return-Path: <common-issues-return-7330-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78962 invoked from network); 11 Apr 2010 13:22:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Apr 2010 13:22:52 -0000
Received: (qmail 49599 invoked by uid 500); 11 Apr 2010 13:22:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49559 invoked by uid 500); 11 Apr 2010 13:22:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49551 invoked by uid 99); 11 Apr 2010 13:22:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 11 Apr 2010 13:22:51 +0000
X-ASF-Spam-Status: No, hits=-1261.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 11 Apr 2010 13:22:50 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3BDGfZP003962
	for <common-issues@hadoop.apache.org>; Sun, 11 Apr 2010 09:16:41 -0400 (EDT)
Message-ID: <4713826.9511270991801071.JavaMail.jira@thor>
Date: Sun, 11 Apr 2010 09:16:41 -0400 (EDT)
From: "Gianmarco De Francisci Morales (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6699) Make MapFile interface consistent
 with SequenceFile
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Make MapFile interface consistent with SequenceFile
---------------------------------------------------

                 Key: HADOOP-6699
                 URL: https://issues.apache.org/jira/browse/HADOOP-6699
             Project: Hadoop Common
          Issue Type: Improvement
          Components: io
    Affects Versions: 0.20.2, 0.20.1
            Reporter: Gianmarco De Francisci Morales
            Priority: Minor


The constructor for the SequenceFile.Reader/Writer classes accept a Path object as the specification of the file to read.
MapFile related classes instead want a String object that represents the MapFile directory.

I propose to make them consistent by replacing the String parameter with a Path one in MapFile related classes.


-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7331-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 11 16:58:55 2010
Return-Path: <common-issues-return-7331-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10908 invoked from network); 11 Apr 2010 16:58:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Apr 2010 16:58:55 -0000
Received: (qmail 63874 invoked by uid 500); 11 Apr 2010 16:58:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63824 invoked by uid 500); 11 Apr 2010 16:58:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63816 invoked by uid 99); 11 Apr 2010 16:58:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 11 Apr 2010 16:58:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 11 Apr 2010 16:58:52 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3BGqgwh012041
	for <common-issues@hadoop.apache.org>; Sun, 11 Apr 2010 12:52:43 -0400 (EDT)
Message-ID: <25584078.11261271004762892.JavaMail.jira@thor>
Date: Sun, 11 Apr 2010 12:52:42 -0400 (EDT)
From: "Ted Yu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6695) Job Tracker should be able to
 refresh classpath without restarting
In-Reply-To: <679324412.961270788350812.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855751#action_12855751 ] 

Ted Yu commented on HADOOP-6695:
--------------------------------

One possibility is for JobClient to pass its effective CLASSPATH, e.g. through (a new) mapred.job.classpath JobConf property, to JobTracker. JobTracker examines this classpath against its classpath to find new jars and put them into DistributedCache.

The benefit is that when user adds new jars to HADOOP_CLASSPATH which get picked up by JobClient, JobTracker is able to honor the addition.


> Job Tracker should be able to refresh classpath without restarting
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6695
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6695
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>    Affects Versions: 0.20.1
>            Reporter: Ted Yu
>
> When I tried to run HBase export in cluster mode, I copied zookeeper-3.2.1.jar to hadoop lib directory.
> But initial attempts failed with java.lang.ClassNotFoundException because job tracker wasn't restarted.
> It is desirable for job tracker to pick up new classpath without restarting which is sometimes hard to do in production environment.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7332-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 06:36:05 2010
Return-Path: <common-issues-return-7332-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40515 invoked from network); 12 Apr 2010 06:36:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 06:36:05 -0000
Received: (qmail 21712 invoked by uid 500); 12 Apr 2010 06:36:05 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21596 invoked by uid 500); 12 Apr 2010 06:36:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21588 invoked by uid 99); 12 Apr 2010 06:36:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 06:36:02 +0000
X-ASF-Spam-Status: No, hits=-1264.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 06:36:02 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3C6Zfeg023038
	for <common-issues@hadoop.apache.org>; Mon, 12 Apr 2010 02:35:41 -0400 (EDT)
Message-ID: <13117554.17981271054141421.JavaMail.jira@thor>
Date: Mon, 12 Apr 2010 02:35:41 -0400 (EDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Moved: (HADOOP-6700) Hadoop commands guide should include
 examples
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amareshwari Sriramadasu moved MAPREDUCE-1691 to HADOOP-6700:
------------------------------------------------------------

          Project: Hadoop Common  (was: Hadoop Map/Reduce)
              Key: HADOOP-6700  (was: MAPREDUCE-1691)
    Fix Version/s: 0.22.0
                       (was: 0.22.0)
      Component/s: documentation
                       (was: documentation)

> Hadoop commands guide should include examples
> ---------------------------------------------
>
>                 Key: HADOOP-6700
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6700
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: documentation
>            Reporter: Amareshwari Sriramadasu
>             Fix For: 0.22.0
>
>
> Currently, The Hadoop command guide (http://hadoop.apache.org/common/docs/r0.20.0/commands_manual.html) just lists all the available command line options, with a description. It should include examples for each command for more clarity.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7333-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 16:11:16 2010
Return-Path: <common-issues-return-7333-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95358 invoked from network); 12 Apr 2010 16:11:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 16:11:16 -0000
Received: (qmail 75053 invoked by uid 500); 12 Apr 2010 16:11:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75019 invoked by uid 500); 12 Apr 2010 16:11:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75011 invoked by uid 99); 12 Apr 2010 16:11:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 16:11:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 16:11:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3CGAoCZ027988
	for <common-issues@hadoop.apache.org>; Mon, 12 Apr 2010 12:10:51 -0400 (EDT)
Message-ID: <1792291.26381271088650982.JavaMail.jira@thor>
Date: Mon, 12 Apr 2010 12:10:50 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6659) Switch RPC to use Avro
In-Reply-To: <387448273.476441269469290584.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856045#action_12856045 ] 

Doug Cutting commented on HADOOP-6659:
--------------------------------------

Eric, the fundamental change that adds "risk" has already been committed to trunk, in HADOOP-6422, permitting one to swap in a different RPC implementation via the configuration.  After that, until we make the switch, no changes should be made that affect runtime behavior.  So if you have a problem with that change, please have one of your committers veto and revert it and I will stop working on this.

> Switch RPC to use Avro
> ----------------------
>
>                 Key: HADOOP-6659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6659
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Doug Cutting
>
> This is an umbrella issue for moving HDFS and MapReduce RPC to use Avro.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7334-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 18:32:18 2010
Return-Path: <common-issues-return-7334-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65134 invoked from network); 12 Apr 2010 18:32:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 18:32:17 -0000
Received: (qmail 13056 invoked by uid 500); 12 Apr 2010 18:32:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13010 invoked by uid 500); 12 Apr 2010 18:32:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13002 invoked by uid 99); 12 Apr 2010 18:32:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 18:32:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 18:32:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3CIVqsl029671
	for <common-issues@hadoop.apache.org>; Mon, 12 Apr 2010 14:31:52 -0400 (EDT)
Message-ID: <29784448.2801271097111992.JavaMail.jira@thor>
Date: Mon, 12 Apr 2010 14:31:51 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6701)  Incorrect exit codes for "dfs
 -chown", "dfs -chgrp"
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

 Incorrect exit codes for "dfs -chown", "dfs -chgrp"
----------------------------------------------------

                 Key: HADOOP-6701
                 URL: https://issues.apache.org/jira/browse/HADOOP-6701
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
    Affects Versions: 0.20.2, 0.20.1, 0.20.0, 0.19.1
            Reporter: Ravi Phulari
            Assignee: Ravi Phulari
            Priority: Minor
             Fix For: 0.20.3, 0.21.0, 0.22.0


ravi@localhost:~$ hadoop dfs -chgrp abcd /; echo $?
chgrp: changing ownership of
'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
0

ravi@localhost:~$ hadoop dfs -chown  abcd /; echo $?
chown: changing ownership of
'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
0

ravi@localhost:~$ hadoop dfs -chmod 755 /DOESNTEXIST; echo $?
chmod: could not get status for '/DOESNTEXIST': File does not exist: /DOESNTEXIST
0

-

Exit codes for both of the above invocations should be non-zero to indicate that the command failed.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7335-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 18:32:19 2010
Return-Path: <common-issues-return-7335-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65212 invoked from network); 12 Apr 2010 18:32:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 18:32:19 -0000
Received: (qmail 13201 invoked by uid 500); 12 Apr 2010 18:32:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13168 invoked by uid 500); 12 Apr 2010 18:32:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13160 invoked by uid 99); 12 Apr 2010 18:32:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 18:32:19 +0000
X-ASF-Spam-Status: No, hits=-1269.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 18:32:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3CIVvfe029779
	for <common-issues@hadoop.apache.org>; Mon, 12 Apr 2010 14:31:57 -0400 (EDT)
Message-ID: <9112463.3161271097117915.JavaMail.jira@thor>
Date: Mon, 12 Apr 2010 14:31:57 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6659) Switch RPC to use Avro
In-Reply-To: <387448273.476441269469290584.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856107#action_12856107 ] 

Sanjay Radia commented on HADOOP-6659:
--------------------------------------

While Avro reflection will make it easier to get AVRO into Hadoop's wire protocol,  I  believe that  AVRO IDL-driven protocols are necessary for wire compatibility. Why?
* A protocol needs to be designed to be compatible. Serialization technologies like PB and Avro allows one to add/delete fields easily; I like that, but it can be  misleading. I don't think we designed the current Hadoop protocols  carefully with an eye towards compatibility.
** For example, as part of HDFS-1052 we would like to extend a the blockId to have an additional field.
Avro would help us do that very easily, but it would not work unless the client side treats the blockid  as an opaque object that is NOT deserialized and simply passed unintepreted to the DNs. I think there are many such examples.
* RMI and Hadoop RPC make it too easy to pass any Java object that one is using internally across the wire. Avro using reflection will continue that.   One needs to examine every object that is being passed across the wire decide if it necessary and what its type should be.
* PB and Avro are very powerful and useful tools - unfortunately a Reflection based approach make badly designed protocols appear to be  good because they give you the impression that your protocol is magically compatible; and it mostly does, but it can miss corner cases and encourages creating messy protocol that exposes too many types.

Hence I propose that we take every single Hadoop protocol and design it for compatibility using Avro IDL. (HDFS-1069, MAPREDUCE-1689)   Based on what I have read, I believe Doug is agreeing with the above.
Switching to IDL however, is not a pluggable change - hence this part needs to be done in a branch. 


The question I have been struggling with is: what are benefits of the reflection scheme since I assert that the resulting protocols *cannot* be declared as "the hadoop wire-compatible protocols". The only benefit is that the reflection based protocols can be used for  cross-language access to hadoop. 



> Switch RPC to use Avro
> ----------------------
>
>                 Key: HADOOP-6659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6659
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Doug Cutting
>
> This is an umbrella issue for moving HDFS and MapReduce RPC to use Avro.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7336-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 18:36:12 2010
Return-Path: <common-issues-return-7336-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66077 invoked from network); 12 Apr 2010 18:36:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 18:36:12 -0000
Received: (qmail 18381 invoked by uid 500); 12 Apr 2010 18:36:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18353 invoked by uid 500); 12 Apr 2010 18:36:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18345 invoked by uid 99); 12 Apr 2010 18:36:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 18:36:12 +0000
X-ASF-Spam-Status: No, hits=-1269.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 18:36:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3CIZoH3029810
	for <common-issues@hadoop.apache.org>; Mon, 12 Apr 2010 14:35:50 -0400 (EDT)
Message-ID: <1795078.3241271097350791.JavaMail.jira@thor>
Date: Mon, 12 Apr 2010 14:35:50 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6701)  Incorrect exit codes for "dfs
 -chown", "dfs -chgrp"
In-Reply-To: <29784448.2801271097111992.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6701:
---------------------------------

    Status: Patch Available  (was: Open)

>  Incorrect exit codes for "dfs -chown", "dfs -chgrp"
> ----------------------------------------------------
>
>                 Key: HADOOP-6701
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6701
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.2, 0.20.1, 0.20.0, 0.19.1
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>            Priority: Minor
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6701.patch
>
>
> ravi@localhost:~$ hadoop dfs -chgrp abcd /; echo $?
> chgrp: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chown  abcd /; echo $?
> chown: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chmod 755 /DOESNTEXIST; echo $?
> chmod: could not get status for '/DOESNTEXIST': File does not exist: /DOESNTEXIST
> 0
> -
> Exit codes for both of the above invocations should be non-zero to indicate that the command failed.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7337-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 18:36:14 2010
Return-Path: <common-issues-return-7337-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66117 invoked from network); 12 Apr 2010 18:36:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 18:36:14 -0000
Received: (qmail 18512 invoked by uid 500); 12 Apr 2010 18:36:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18489 invoked by uid 500); 12 Apr 2010 18:36:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18481 invoked by uid 99); 12 Apr 2010 18:36:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 18:36:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 18:36:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3CIZn8w029798
	for <common-issues@hadoop.apache.org>; Mon, 12 Apr 2010 14:35:49 -0400 (EDT)
Message-ID: <31751202.3201271097349768.JavaMail.jira@thor>
Date: Mon, 12 Apr 2010 14:35:49 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6701)  Incorrect exit codes for "dfs
 -chown", "dfs -chgrp"
In-Reply-To: <29784448.2801271097111992.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6701:
---------------------------------

    Attachment: HADOOP-6701.patch

Attaching simple patch for fixing exit codes for -ch(mod/grp/own)  commands.

>  Incorrect exit codes for "dfs -chown", "dfs -chgrp"
> ----------------------------------------------------
>
>                 Key: HADOOP-6701
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6701
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.19.1, 0.20.0, 0.20.1, 0.20.2
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>            Priority: Minor
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6701.patch
>
>
> ravi@localhost:~$ hadoop dfs -chgrp abcd /; echo $?
> chgrp: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chown  abcd /; echo $?
> chown: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chmod 755 /DOESNTEXIST; echo $?
> chmod: could not get status for '/DOESNTEXIST': File does not exist: /DOESNTEXIST
> 0
> -
> Exit codes for both of the above invocations should be non-zero to indicate that the command failed.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7338-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 18:38:21 2010
Return-Path: <common-issues-return-7338-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67399 invoked from network); 12 Apr 2010 18:38:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 18:38:21 -0000
Received: (qmail 22489 invoked by uid 500); 12 Apr 2010 18:38:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22449 invoked by uid 500); 12 Apr 2010 18:38:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22441 invoked by uid 99); 12 Apr 2010 18:38:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 18:38:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 18:38:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3CIbvBP029943
	for <common-issues@hadoop.apache.org>; Mon, 12 Apr 2010 14:37:57 -0400 (EDT)
Message-ID: <18158216.3611271097476998.JavaMail.jira@thor>
Date: Mon, 12 Apr 2010 14:37:56 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6659) Switch RPC to use Avro
In-Reply-To: <387448273.476441269469290584.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856111#action_12856111 ] 

Sanjay Radia commented on HADOOP-6659:
--------------------------------------

Doug says:
> switch HDFS [and MR] to use Avro RPC serialization by default.
Given that the resulting protocols are not the official wire compatible protocol, why change the default till we have completed the move to  Avro IDL?

> Switch RPC to use Avro
> ----------------------
>
>                 Key: HADOOP-6659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6659
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Doug Cutting
>
> This is an umbrella issue for moving HDFS and MapReduce RPC to use Avro.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7339-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 18:50:13 2010
Return-Path: <common-issues-return-7339-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69987 invoked from network); 12 Apr 2010 18:50:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 18:50:13 -0000
Received: (qmail 46041 invoked by uid 500); 12 Apr 2010 18:50:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45993 invoked by uid 500); 12 Apr 2010 18:50:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45985 invoked by uid 99); 12 Apr 2010 18:50:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 18:50:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 18:50:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3CInnAs000059
	for <common-issues@hadoop.apache.org>; Mon, 12 Apr 2010 14:49:49 -0400 (EDT)
Message-ID: <33216045.3911271098189600.JavaMail.jira@thor>
Date: Mon, 12 Apr 2010 14:49:49 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6701)  Incorrect exit codes for "dfs
 -chown", "dfs -chgrp"
In-Reply-To: <29784448.2801271097111992.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6701:
---------------------------------

    Attachment: HADOOP-6701-trunk.patch

Attaching patch for trunk.

>  Incorrect exit codes for "dfs -chown", "dfs -chgrp"
> ----------------------------------------------------
>
>                 Key: HADOOP-6701
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6701
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.19.1, 0.20.0, 0.20.1, 0.20.2
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>            Priority: Minor
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6701-trunk.patch, HADOOP-6701.patch
>
>
> ravi@localhost:~$ hadoop dfs -chgrp abcd /; echo $?
> chgrp: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chown  abcd /; echo $?
> chown: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chmod 755 /DOESNTEXIST; echo $?
> chmod: could not get status for '/DOESNTEXIST': File does not exist: /DOESNTEXIST
> 0
> -
> Exit codes for both of the above invocations should be non-zero to indicate that the command failed.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7340-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 18:56:18 2010
Return-Path: <common-issues-return-7340-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70799 invoked from network); 12 Apr 2010 18:56:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 18:56:17 -0000
Received: (qmail 51890 invoked by uid 500); 12 Apr 2010 18:56:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51853 invoked by uid 500); 12 Apr 2010 18:56:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51845 invoked by uid 99); 12 Apr 2010 18:56:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 18:56:17 +0000
X-ASF-Spam-Status: No, hits=-1269.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 18:56:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3CItu5N000212
	for <common-issues@hadoop.apache.org>; Mon, 12 Apr 2010 14:55:56 -0400 (EDT)
Message-ID: <12858218.4381271098556261.JavaMail.jira@thor>
Date: Mon, 12 Apr 2010 14:55:56 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6659) Switch RPC to use Avro
In-Reply-To: <387448273.476441269469290584.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856123#action_12856123 ] 

Doug Cutting commented on HADOOP-6659:
--------------------------------------

> AVRO IDL-driven protocols are necessary for wire compatibility.

I agree.  As I said above, before we declare that we support wire-compatibility we should perform a careful audit of our RPC protocols.  With the approach I've suggested, this can largely be pursued in parallel.  Until we switch we can examine the reflected protocol and work to improve (or at least develop a list of proposed improvements) the Java interfaces.

> what are benefits of the reflection scheme

By not branching we can continue development of HDFS and mapreduce in parallel while we develop and test a new RPC serialization and transport.  As mentioned above, once we've switched to Avro-serialization using reflect, then we can begin, protocol-by-protocol, switching each to an IDL-based approach.  Each protocol can be addressed in a separate issue with no massive branch required.  We might, e.g., prioritize client-facing protocols first, so that we can support wire compatibility of clients before we support rolling cluster upgrades.  We can even separate updating clients from updating servers.  Once we've completed the transition to an IDL-driven system, then we can, protocol-by-protocol, method-by-method, work to improve the IDL to the point where we're willing to declare our support of wire-compatibility.  At no point is trunk broken or are large areas blocked from changes and fixes.


> Switch RPC to use Avro
> ----------------------
>
>                 Key: HADOOP-6659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6659
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Doug Cutting
>
> This is an umbrella issue for moving HDFS and MapReduce RPC to use Avro.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7341-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 19:20:14 2010
Return-Path: <common-issues-return-7341-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76794 invoked from network); 12 Apr 2010 19:20:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 19:20:14 -0000
Received: (qmail 85984 invoked by uid 500); 12 Apr 2010 19:20:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85947 invoked by uid 500); 12 Apr 2010 19:20:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85939 invoked by uid 99); 12 Apr 2010 19:20:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 19:20:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 19:20:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3CJJndi000385
	for <common-issues@hadoop.apache.org>; Mon, 12 Apr 2010 15:19:49 -0400 (EDT)
Message-ID: <19279931.4771271099989836.JavaMail.jira@thor>
Date: Mon, 12 Apr 2010 15:19:49 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6688) FileSystem.delete(...)
 implementations should not throw FileNotFoundException
In-Reply-To: <1780197215.37411270627173494.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856131#action_12856131 ] 

Tom White commented on HADOOP-6688:
-----------------------------------

Could this be a consistency issue, like HADOOP-6208? 

I'm not sure a blanket ignore of FileNotFoundExceptions is quite right: if you call delete with a path that doesn't exist should it not throw FileNotFoundException? I can see that if a file in a subdirectory of the path being deleted doesn't exist then that should not result in a FileNotFoundException. You might want to check if the new FileContext API exhibits the same problem, so it can be considered for the contract there too.

I don't think this is a blocker. As a workaround you could create a wrapped FileOutputCommitter that catches and ignores FileNotFoundException.

> FileSystem.delete(...) implementations should not throw FileNotFoundException
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-6688
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6688
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, fs/s3
>    Affects Versions: 0.20.2
>         Environment: Amazon EC2/S3
>            Reporter: Danny Leshem
>            Priority: Blocker
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>
> S3FileSystem.delete(Path path, boolean recursive) may fail and throw a FileNotFoundException if a directory is being deleted while at the same time some of its files are deleted in the background.
> This is definitely not the expected behavior of a delete method. If one of the to-be-deleted files is found missing, the method should not fail and simply continue. This is true for the general contract of FileSystem.delete, and also for its various implementations: RawLocalFileSystem (and specifically FileUtil.fullyDelete) exhibits the same problem.
> The fix is to silently catch and ignore FileNotFoundExceptions in delete loops. This can very easily be unit-tested, at least for RawLocalFileSystem.
> The reason this issue bothers me is that the cleanup part of a long (Mahout) MR job inconsistently fails for me, and I think this is the root problem. The log shows:
> {code}
> java.io.FileNotFoundException: s3://S3-BUCKET/tmp/0008E25BF7554CA9/2521362836721872/DistributedMatrix.times.outputVector/_temporary/_attempt_201004061215_0092_r_000002_0/part-00002: No such file or directory.
> 	at org.apache.hadoop.fs.s3.S3FileSystem.getFileStatus(S3FileSystem.java:334)
> 	at org.apache.hadoop.fs.s3.S3FileSystem.listStatus(S3FileSystem.java:193)
> 	at org.apache.hadoop.fs.s3.S3FileSystem.delete(S3FileSystem.java:303)
> 	at org.apache.hadoop.fs.s3.S3FileSystem.delete(S3FileSystem.java:312)
> 	at org.apache.hadoop.mapred.FileOutputCommitter.cleanupJob(FileOutputCommitter.java:64)
> 	at org.apache.hadoop.mapred.OutputCommitter.cleanupJob(OutputCommitter.java:135)
> 	at org.apache.hadoop.mapred.Task.runJobCleanupTask(Task.java:826)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:292)
> 	at org.apache.hadoop.mapred.Child.main(Child.java:170)
> {code}
> (similar errors are displayed for ReduceTask.run)

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7342-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 19:44:13 2010
Return-Path: <common-issues-return-7342-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84395 invoked from network); 12 Apr 2010 19:44:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 19:44:11 -0000
Received: (qmail 6801 invoked by uid 500); 12 Apr 2010 19:44:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6762 invoked by uid 500); 12 Apr 2010 19:44:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6753 invoked by uid 99); 12 Apr 2010 19:44:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 19:44:11 +0000
X-ASF-Spam-Status: No, hits=-1270.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 19:44:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3CJhnr0000509
	for <common-issues@hadoop.apache.org>; Mon, 12 Apr 2010 15:43:49 -0400 (EDT)
Message-ID: <33456055.4991271101429708.JavaMail.jira@thor>
Date: Mon, 12 Apr 2010 15:43:49 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6686) Remove redundant exception class
 name in unwrapped exceptions thrown at the RPC client
In-Reply-To: <70650712.27801270591413554.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856137#action_12856137 ] 

Doug Cutting commented on HADOOP-6686:
--------------------------------------

+1 This looks good to me.

> Remove redundant exception class name in unwrapped exceptions thrown at the RPC client
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6686
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6686
>             Project: Hadoop Common
>          Issue Type: Improvement
>         Environment: At RPC client, when the exception thrown by the server is unwrapped, the exception message includes redundant exception class name. This redundant information should be removed.
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6686.1.patch, HADOOP-6686.patch
>
>


-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7343-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 20:00:14 2010
Return-Path: <common-issues-return-7343-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90849 invoked from network); 12 Apr 2010 20:00:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 20:00:13 -0000
Received: (qmail 26311 invoked by uid 500); 12 Apr 2010 20:00:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26276 invoked by uid 500); 12 Apr 2010 20:00:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26268 invoked by uid 99); 12 Apr 2010 20:00:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 20:00:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 20:00:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3CJxnPa000775
	for <common-issues@hadoop.apache.org>; Mon, 12 Apr 2010 15:59:49 -0400 (EDT)
Message-ID: <27199217.5521271102389444.JavaMail.jira@thor>
Date: Mon, 12 Apr 2010 15:59:49 -0400 (EDT)
From: "Danny Leshem (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6688) FileSystem.delete(...)
 implementations should not throw FileNotFoundException
In-Reply-To: <1780197215.37411270627173494.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856145#action_12856145 ] 

Danny Leshem commented on HADOOP-6688:
--------------------------------------

Tom, this is a classic issue of coding "for the programmer" or "against the programmer".

What is the expected effect of calling FileSystem.delete(someFile)? A reasonable answer would be "someFile no longer exists", and the method should throw an exception only if that expected effect somehow did not happen, meaning the file is there.

For those unlikely users who care whether the file was actually deleted or wasn't there to begin with, you can have FileSystem.delete() return a boolean to convey this information. This is common practice in many file system APIs.

I have no idea as to the root cause of the file not existing. It might be a similar consistency issue to the one you mentioned - I have not looked into it. Since fixing FileSystem.delete is rather easy (and the change is very contained), I think it's a better solution than working around the issue.

> FileSystem.delete(...) implementations should not throw FileNotFoundException
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-6688
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6688
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, fs/s3
>    Affects Versions: 0.20.2
>         Environment: Amazon EC2/S3
>            Reporter: Danny Leshem
>            Priority: Blocker
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>
> S3FileSystem.delete(Path path, boolean recursive) may fail and throw a FileNotFoundException if a directory is being deleted while at the same time some of its files are deleted in the background.
> This is definitely not the expected behavior of a delete method. If one of the to-be-deleted files is found missing, the method should not fail and simply continue. This is true for the general contract of FileSystem.delete, and also for its various implementations: RawLocalFileSystem (and specifically FileUtil.fullyDelete) exhibits the same problem.
> The fix is to silently catch and ignore FileNotFoundExceptions in delete loops. This can very easily be unit-tested, at least for RawLocalFileSystem.
> The reason this issue bothers me is that the cleanup part of a long (Mahout) MR job inconsistently fails for me, and I think this is the root problem. The log shows:
> {code}
> java.io.FileNotFoundException: s3://S3-BUCKET/tmp/0008E25BF7554CA9/2521362836721872/DistributedMatrix.times.outputVector/_temporary/_attempt_201004061215_0092_r_000002_0/part-00002: No such file or directory.
> 	at org.apache.hadoop.fs.s3.S3FileSystem.getFileStatus(S3FileSystem.java:334)
> 	at org.apache.hadoop.fs.s3.S3FileSystem.listStatus(S3FileSystem.java:193)
> 	at org.apache.hadoop.fs.s3.S3FileSystem.delete(S3FileSystem.java:303)
> 	at org.apache.hadoop.fs.s3.S3FileSystem.delete(S3FileSystem.java:312)
> 	at org.apache.hadoop.mapred.FileOutputCommitter.cleanupJob(FileOutputCommitter.java:64)
> 	at org.apache.hadoop.mapred.OutputCommitter.cleanupJob(OutputCommitter.java:135)
> 	at org.apache.hadoop.mapred.Task.runJobCleanupTask(Task.java:826)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:292)
> 	at org.apache.hadoop.mapred.Child.main(Child.java:170)
> {code}
> (similar errors are displayed for ReduceTask.run)

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7344-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 20:40:14 2010
Return-Path: <common-issues-return-7344-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1244 invoked from network); 12 Apr 2010 20:40:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 20:40:14 -0000
Received: (qmail 73274 invoked by uid 500); 12 Apr 2010 20:40:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73244 invoked by uid 500); 12 Apr 2010 20:40:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73236 invoked by uid 99); 12 Apr 2010 20:40:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 20:40:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 20:40:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3CKdo93001151
	for <common-issues@hadoop.apache.org>; Mon, 12 Apr 2010 16:39:50 -0400 (EDT)
Message-ID: <27405518.6311271104790390.JavaMail.jira@thor>
Date: Mon, 12 Apr 2010 16:39:50 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6688) FileSystem.delete(...)
 implementations should not throw FileNotFoundException
In-Reply-To: <1780197215.37411270627173494.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856158#action_12856158 ] 

Tom White commented on HADOOP-6688:
-----------------------------------

I think this is related to the discussion in HADOOP-6631 about having a 'force' option for delete. I didn't mean to suggest that this issue shouldn't be fixed, I was merely pointing out a workaround that might solve your problem in the meantime, while the filesystem semantics are discussed in more detail.

> FileSystem.delete(...) implementations should not throw FileNotFoundException
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-6688
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6688
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, fs/s3
>    Affects Versions: 0.20.2
>         Environment: Amazon EC2/S3
>            Reporter: Danny Leshem
>            Priority: Blocker
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>
> S3FileSystem.delete(Path path, boolean recursive) may fail and throw a FileNotFoundException if a directory is being deleted while at the same time some of its files are deleted in the background.
> This is definitely not the expected behavior of a delete method. If one of the to-be-deleted files is found missing, the method should not fail and simply continue. This is true for the general contract of FileSystem.delete, and also for its various implementations: RawLocalFileSystem (and specifically FileUtil.fullyDelete) exhibits the same problem.
> The fix is to silently catch and ignore FileNotFoundExceptions in delete loops. This can very easily be unit-tested, at least for RawLocalFileSystem.
> The reason this issue bothers me is that the cleanup part of a long (Mahout) MR job inconsistently fails for me, and I think this is the root problem. The log shows:
> {code}
> java.io.FileNotFoundException: s3://S3-BUCKET/tmp/0008E25BF7554CA9/2521362836721872/DistributedMatrix.times.outputVector/_temporary/_attempt_201004061215_0092_r_000002_0/part-00002: No such file or directory.
> 	at org.apache.hadoop.fs.s3.S3FileSystem.getFileStatus(S3FileSystem.java:334)
> 	at org.apache.hadoop.fs.s3.S3FileSystem.listStatus(S3FileSystem.java:193)
> 	at org.apache.hadoop.fs.s3.S3FileSystem.delete(S3FileSystem.java:303)
> 	at org.apache.hadoop.fs.s3.S3FileSystem.delete(S3FileSystem.java:312)
> 	at org.apache.hadoop.mapred.FileOutputCommitter.cleanupJob(FileOutputCommitter.java:64)
> 	at org.apache.hadoop.mapred.OutputCommitter.cleanupJob(OutputCommitter.java:135)
> 	at org.apache.hadoop.mapred.Task.runJobCleanupTask(Task.java:826)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:292)
> 	at org.apache.hadoop.mapred.Child.main(Child.java:170)
> {code}
> (similar errors are displayed for ReduceTask.run)

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7345-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 20:57:15 2010
Return-Path: <common-issues-return-7345-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7351 invoked from network); 12 Apr 2010 20:57:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 20:57:14 -0000
Received: (qmail 5088 invoked by uid 500); 12 Apr 2010 20:57:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5039 invoked by uid 500); 12 Apr 2010 20:57:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4917 invoked by uid 99); 12 Apr 2010 20:57:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 20:57:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 20:57:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3CKunhN001549
	for <common-issues@hadoop.apache.org>; Mon, 12 Apr 2010 16:56:49 -0400 (EDT)
Message-ID: <17396326.6551271105809673.JavaMail.jira@thor>
Date: Mon, 12 Apr 2010 16:56:49 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6686) Remove redundant exception class
 name in unwrapped exceptions thrown at the RPC client
In-Reply-To: <70650712.27801270591413554.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856164#action_12856164 ] 

Suresh Srinivas commented on HADOOP-6686:
-----------------------------------------

Doug, could you please review and +1 HDFS-1083 as well. Committing both these changes together will keep the window for which HDFS test fails small.

> Remove redundant exception class name in unwrapped exceptions thrown at the RPC client
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6686
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6686
>             Project: Hadoop Common
>          Issue Type: Improvement
>         Environment: At RPC client, when the exception thrown by the server is unwrapped, the exception message includes redundant exception class name. This redundant information should be removed.
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6686.1.patch, HADOOP-6686.patch
>
>


-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7346-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 20:59:17 2010
Return-Path: <common-issues-return-7346-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9425 invoked from network); 12 Apr 2010 20:59:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 20:59:16 -0000
Received: (qmail 11932 invoked by uid 500); 12 Apr 2010 20:59:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11896 invoked by uid 500); 12 Apr 2010 20:59:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11870 invoked by uid 99); 12 Apr 2010 20:59:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 20:59:13 +0000
X-ASF-Spam-Status: No, hits=-1270.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 20:59:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3CKwpnJ001592
	for <common-issues@hadoop.apache.org>; Mon, 12 Apr 2010 16:58:52 -0400 (EDT)
Message-ID: <18743668.6651271105931804.JavaMail.jira@thor>
Date: Mon, 12 Apr 2010 16:58:51 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6507) Hadoop Common Docs - delete 3 doc
 files that do not belong under Common
In-Reply-To: <1315292172.20371264211241498.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White resolved HADOOP-6507.
-------------------------------

    Hadoop Flags: [Reviewed]
      Resolution: Fixed

I've just committed this. Thanks Corinne!

> Hadoop Common Docs - delete 3 doc files that do not belong under Common
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-6507
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6507
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>    Affects Versions: 0.21.0
>            Reporter: Corinne Chandel
>            Assignee: Corinne Chandel
>            Priority: Blocker
>             Fix For: 0.21.0
>
>
> Delete these 3 files from Hadoop Common TRUNK
> \src\docs\src\documentation\content\xdocs\streaming.xml  (this file already in (and belongs in) Hadoop MAPREDUCE TRUNK)
> \src\docs\src\documentation\content\xdocs\libhdfs.xml        (this file already in (and belongs in) Hadoop HDFS TRUNK)
> \src\docs\src\documentation\content\xdocs\hdfs_permissions_guide.xml  (this file already in (and belongs in) Hadoop HDFS TRUNK)

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7347-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 21:13:11 2010
Return-Path: <common-issues-return-7347-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18868 invoked from network); 12 Apr 2010 21:13:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 21:13:11 -0000
Received: (qmail 32348 invoked by uid 500); 12 Apr 2010 21:13:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32319 invoked by uid 500); 12 Apr 2010 21:13:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32311 invoked by uid 99); 12 Apr 2010 21:13:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 21:13:11 +0000
X-ASF-Spam-Status: No, hits=-1270.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 21:13:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3CLCn8M001733
	for <common-issues@hadoop.apache.org>; Mon, 12 Apr 2010 17:12:50 -0400 (EDT)
Message-ID: <12471585.7061271106769898.JavaMail.jira@thor>
Date: Mon, 12 Apr 2010 17:12:49 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6507) Hadoop Common Docs - delete 3 doc
 files that do not belong under Common
In-Reply-To: <1315292172.20371264211241498.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856176#action_12856176 ] 

Hudson commented on HADOOP-6507:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #216 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/216/])
    HADOOP-6507. Hadoop Common Docs - delete 3 doc files that do not belong under Common. Contributed by Corinne Chandel.


> Hadoop Common Docs - delete 3 doc files that do not belong under Common
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-6507
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6507
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>    Affects Versions: 0.21.0
>            Reporter: Corinne Chandel
>            Assignee: Corinne Chandel
>            Priority: Blocker
>             Fix For: 0.21.0
>
>
> Delete these 3 files from Hadoop Common TRUNK
> \src\docs\src\documentation\content\xdocs\streaming.xml  (this file already in (and belongs in) Hadoop MAPREDUCE TRUNK)
> \src\docs\src\documentation\content\xdocs\libhdfs.xml        (this file already in (and belongs in) Hadoop HDFS TRUNK)
> \src\docs\src\documentation\content\xdocs\hdfs_permissions_guide.xml  (this file already in (and belongs in) Hadoop HDFS TRUNK)

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7348-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 21:37:15 2010
Return-Path: <common-issues-return-7348-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45829 invoked from network); 12 Apr 2010 21:37:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 21:37:15 -0000
Received: (qmail 73851 invoked by uid 500); 12 Apr 2010 21:37:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73824 invoked by uid 500); 12 Apr 2010 21:37:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73816 invoked by uid 99); 12 Apr 2010 21:37:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 21:37:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 21:37:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3CLanLD001910
	for <common-issues@hadoop.apache.org>; Mon, 12 Apr 2010 17:36:50 -0400 (EDT)
Message-ID: <13060465.7551271108209697.JavaMail.jira@thor>
Date: Mon, 12 Apr 2010 17:36:49 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6701)  Incorrect exit codes for "dfs
 -chown", "dfs -chgrp"
In-Reply-To: <29784448.2801271097111992.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856188#action_12856188 ] 

Konstantin Boudnik commented on HADOOP-6701:
--------------------------------------------

Looks like the patch is incomplete because it the case with {{chmod}}. The other two seem intact to me. 

>  Incorrect exit codes for "dfs -chown", "dfs -chgrp"
> ----------------------------------------------------
>
>                 Key: HADOOP-6701
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6701
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.19.1, 0.20.0, 0.20.1, 0.20.2
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>            Priority: Minor
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6701-trunk.patch, HADOOP-6701.patch
>
>
> ravi@localhost:~$ hadoop dfs -chgrp abcd /; echo $?
> chgrp: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chown  abcd /; echo $?
> chown: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chmod 755 /DOESNTEXIST; echo $?
> chmod: could not get status for '/DOESNTEXIST': File does not exist: /DOESNTEXIST
> 0
> -
> Exit codes for both of the above invocations should be non-zero to indicate that the command failed.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7349-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 21:39:13 2010
Return-Path: <common-issues-return-7349-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47109 invoked from network); 12 Apr 2010 21:39:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 21:39:13 -0000
Received: (qmail 74682 invoked by uid 500); 12 Apr 2010 21:39:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74633 invoked by uid 500); 12 Apr 2010 21:39:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74625 invoked by uid 99); 12 Apr 2010 21:39:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 21:39:13 +0000
X-ASF-Spam-Status: No, hits=-1270.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 21:39:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3CLcqJ5001970
	for <common-issues@hadoop.apache.org>; Mon, 12 Apr 2010 17:38:52 -0400 (EDT)
Message-ID: <5769839.7731271108332131.JavaMail.jira@thor>
Date: Mon, 12 Apr 2010 17:38:52 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6422) permit RPC protocols to be
 implemented by Avro
In-Reply-To: <931659516.1260389418441.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856190#action_12856190 ] 

Doug Cutting commented on HADOOP-6422:
--------------------------------------

For the record, I mis-identified the primary commit for this issue.  The revision was 889889.

http://svn.apache.org/viewvc?view=revision&revision=889889

> permit RPC protocols to be implemented by Avro
> ----------------------------------------------
>
>                 Key: HADOOP-6422
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6422
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: ipc
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6422.patch, HADOOP-6422.patch, HADOOP-6422.patch, HADOOP-6422.patch, HADOOP-6422.patch
>
>
> To more easily permit Hadoop to evolve to use Avro RPC, I propose to change RPC to use different implementations for clients and servers based on the configuration.  This is not intended as an end-user configuration: only a single RPC implementation will be supported in a given release, but rather a tool to permit us to more easily develop and test new RPC implementations.  As such, the configuration parameters used would not be documented.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7350-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 22:11:11 2010
Return-Path: <common-issues-return-7350-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63572 invoked from network); 12 Apr 2010 22:11:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 22:11:11 -0000
Received: (qmail 10045 invoked by uid 500); 12 Apr 2010 22:11:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10008 invoked by uid 500); 12 Apr 2010 22:11:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9994 invoked by uid 99); 12 Apr 2010 22:11:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 22:11:11 +0000
X-ASF-Spam-Status: No, hits=-1270.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 22:11:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3CMAn2I002174
	for <common-issues@hadoop.apache.org>; Mon, 12 Apr 2010 18:10:49 -0400 (EDT)
Message-ID: <13294811.8221271110249495.JavaMail.jira@thor>
Date: Mon, 12 Apr 2010 18:10:49 -0400 (EDT)
From: "Gaurav Jain (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5759) IllegalArgumentException when
 CombineFileInputFormat is used as job InputFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-5759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856201#action_12856201 ] 

Gaurav Jain commented on HADOOP-5759:
-------------------------------------

I am trying to use CombineFile* classes to use in Zebra. After this patch, it works for me, so far.

However, I was wondering if somebody has done any performance analysis on CombineFile*    vs    InputFile* ( data locality   vs  ( non ) data-locality )

If so, can I have access to it? 

> IllegalArgumentException when CombineFileInputFormat is used as job InputFormat
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-5759
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5759
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.1
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.20.2, 0.21.0
>
>         Attachments: patch-5759-1.txt, patch-5759-2.txt, patch-5759.txt
>
>
> As per my understanding, CombineFileInputFormat is creating splits with rackname as split location. 
> When I use CombineFileInputFormat as the InputFormat for job, job initialization fails with following exception :
> 2009-04-28 14:10:40,162 ERROR mapred.EagerTaskInitializationListener (EagerTaskInitializationListener.java:run(83)) - Job initialization failed:
> java.lang.IllegalArgumentException: Network location name contains /: /default-rack
>   at org.apache.hadoop.net.NodeBase.set(NodeBase.java:76)
>   at org.apache.hadoop.net.NodeBase.<init>(NodeBase.java:57)
>   at org.apache.hadoop.mapred.JobTracker.addHostToNodeMapping(JobTracker.java:2342)
>   at org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2336)
>   at org.apache.hadoop.mapred.JobInProgress.createCache(JobInProgress.java:344)
>   at org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:441)
>   at org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:81)
>   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
>   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>   at java.lang.Thread.run(Thread.java:619)
> When I changed CombineFileInputFormat to pass just rackname (without '/'), JT wrongly resolves  the node as /default-rack/<rack-name>.
> Solution is to pass hostnames holding the block(on the rack),  instead of rackname.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7351-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 12 23:41:15 2010
Return-Path: <common-issues-return-7351-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14768 invoked from network); 12 Apr 2010 23:41:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Apr 2010 23:41:15 -0000
Received: (qmail 7360 invoked by uid 500); 12 Apr 2010 23:41:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7316 invoked by uid 500); 12 Apr 2010 23:41:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7308 invoked by uid 99); 12 Apr 2010 23:41:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 23:41:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Apr 2010 23:41:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3CNepwV003011
	for <common-issues@hadoop.apache.org>; Mon, 12 Apr 2010 19:40:51 -0400 (EDT)
Message-ID: <16144898.9811271115651031.JavaMail.jira@thor>
Date: Mon, 12 Apr 2010 19:40:51 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6678) Propose some changes to FileContext
In-Reply-To: <360625036.668801270253667713.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856225#action_12856225 ] 

Eli Collins commented on HADOOP-6678:
-------------------------------------

Thanks for the review Hairong. Could you check out HDFS-1089 as well?  It's simple, just removes uses of FileContext#isFile, isDirectory and exists from HDFS, which needs to be checked in before this patch. 

> Propose some changes to FileContext
> -----------------------------------
>
>                 Key: HADOOP-6678
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6678
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Hairong Kuang
>            Assignee: Eli Collins
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6678-1.patch
>
>
> # Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.
> # Remove methods isFile(Path), isDirectory(Path), and exists.
> All these methods are implemented by calling getFileStatus(Path).But most users are not aware of this. They would write code as below: 
> {code}
>   FileContext fc = ..;
>   if (fc.exists(path)) {
>     if (fc.isFile(path)) {
>      ...
>     } else {
>     ...
>     }
>   }
> {code}
> The above code adds unnecessary getFileInfo RPC to NameNode. In our production clusters, we often see that the number of getFileStatus calls is multiple times of the open calls. If we remove isFile, isDirectory, and exists from FileContext, users have to explicitly call getFileStatus first, it is more likely that they will write more efficient code as follow:
> {code}
>   FileContext fc = ...;
>   FileStatus fstatus = fc.getFileStatus(path);
>   if (fstatus.isFile() {
>     ...
>   } else {
>     ...
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7352-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 00:05:14 2010
Return-Path: <common-issues-return-7352-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36963 invoked from network); 13 Apr 2010 00:05:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 00:05:13 -0000
Received: (qmail 24275 invoked by uid 500); 13 Apr 2010 00:05:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24251 invoked by uid 500); 13 Apr 2010 00:05:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24243 invoked by uid 99); 13 Apr 2010 00:05:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 00:05:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 00:05:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3D04nt2003273
	for <common-issues@hadoop.apache.org>; Mon, 12 Apr 2010 20:04:50 -0400 (EDT)
Message-ID: <9314864.10251271117089716.JavaMail.jira@thor>
Date: Mon, 12 Apr 2010 20:04:49 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6677) InterfaceAudience.LimitedPrivate
 should take a string not an enum
In-Reply-To: <680532766.665461270239927272.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856234#action_12856234 ] 

Tom White commented on HADOOP-6677:
-----------------------------------

> This would be a pretty big change to the classification scheme that was argued out quite exhaustively in HADOOP-5073; there should be time for people to re-hash if they wish before anything is committed. 

I agree that people should have a chance to discuss this change. As the annotations begin to get more traction, it seems reasonable to be able to change them to make them better fit how they are actually being used.

> If we're removing the limitation of who should use LimitedPrivate, and who can depend on it, then it's not really limited private anymore?

The code that is annotated LimitedPrivate gets to declare who may use it, so in that sense it is still limited private, no?


> InterfaceAudience.LimitedPrivate should take a string not an enum
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6677
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6677
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Alan Gates
>            Assignee: Tom White
>            Priority: Minor
>         Attachments: HADOOP-6677.patch
>
>
> Trying to keep the list of all possible components sharing limited private interfaces up to date is painful.  As other subprojects beyond HDFS and MR use this interface it will only get worse (see PIG-1311).  If it is converted to a string other subprojects can use it without requiring patches to common.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7353-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 00:07:11 2010
Return-Path: <common-issues-return-7353-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37735 invoked from network); 13 Apr 2010 00:07:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 00:07:11 -0000
Received: (qmail 24885 invoked by uid 500); 13 Apr 2010 00:07:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24851 invoked by uid 500); 13 Apr 2010 00:07:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24796 invoked by uid 99); 13 Apr 2010 00:07:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 00:07:11 +0000
X-ASF-Spam-Status: No, hits=-1271.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 00:07:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3D06oBX003310
	for <common-issues@hadoop.apache.org>; Mon, 12 Apr 2010 20:06:50 -0400 (EDT)
Message-ID: <26652222.10331271117210074.JavaMail.jira@thor>
Date: Mon, 12 Apr 2010 20:06:50 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6580) UGI should contain authentication
 method.
In-Reply-To: <1786976638.415521266712107907.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6580:
-----------------------------------------

    Status: Open  (was: Patch Available)

> UGI should contain authentication method.
> -----------------------------------------
>
>                 Key: HADOOP-6580
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6580
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: c-6580.patch, HADOOP-6580-0_20.5.patch, HADOOP-6580.1.patch, HADOOP-6580.2.patch, HADOOP-6580.3.patch, HADOOP-6580.4.patch, HADOOP-6580.5.patch, HADOOP-6580.6.patch, HADOOP-6580.7.patch, HADOOP-6580.8.patch, HADOOP-6580.9.patch
>
>
> The UserGroupInformation should contain authentication method in its subject. This will be used in HDFS to issue delegation tokens only to kerberos authenticated clients.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7354-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 00:07:14 2010
Return-Path: <common-issues-return-7354-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37791 invoked from network); 13 Apr 2010 00:07:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 00:07:13 -0000
Received: (qmail 25023 invoked by uid 500); 13 Apr 2010 00:07:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24984 invoked by uid 500); 13 Apr 2010 00:07:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24976 invoked by uid 99); 13 Apr 2010 00:07:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 00:07:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 00:07:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3D06njW003298
	for <common-issues@hadoop.apache.org>; Mon, 12 Apr 2010 20:06:49 -0400 (EDT)
Message-ID: <6690077.10291271117209475.JavaMail.jira@thor>
Date: Mon, 12 Apr 2010 20:06:49 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6580) UGI should contain authentication
 method.
In-Reply-To: <1786976638.415521266712107907.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6580:
-----------------------------------------

    Attachment: HADOOP-6580.9.patch

Patch updated to merge with latest trunk.

> UGI should contain authentication method.
> -----------------------------------------
>
>                 Key: HADOOP-6580
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6580
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: c-6580.patch, HADOOP-6580-0_20.5.patch, HADOOP-6580.1.patch, HADOOP-6580.2.patch, HADOOP-6580.3.patch, HADOOP-6580.4.patch, HADOOP-6580.5.patch, HADOOP-6580.6.patch, HADOOP-6580.7.patch, HADOOP-6580.8.patch, HADOOP-6580.9.patch
>
>
> The UserGroupInformation should contain authentication method in its subject. This will be used in HDFS to issue delegation tokens only to kerberos authenticated clients.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7355-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 00:09:12 2010
Return-Path: <common-issues-return-7355-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39511 invoked from network); 13 Apr 2010 00:09:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 00:09:11 -0000
Received: (qmail 26073 invoked by uid 500); 13 Apr 2010 00:09:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26029 invoked by uid 500); 13 Apr 2010 00:09:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26021 invoked by uid 99); 13 Apr 2010 00:09:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 00:09:11 +0000
X-ASF-Spam-Status: No, hits=-1271.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 00:09:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3D08okb003335
	for <common-issues@hadoop.apache.org>; Mon, 12 Apr 2010 20:08:50 -0400 (EDT)
Message-ID: <4566518.10391271117330510.JavaMail.jira@thor>
Date: Mon, 12 Apr 2010 20:08:50 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6580) UGI should contain authentication
 method.
In-Reply-To: <1786976638.415521266712107907.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6580:
-----------------------------------------

    Status: Patch Available  (was: Open)

HADOOP-6580.9.patch  is submitted for hudson tests.

> UGI should contain authentication method.
> -----------------------------------------
>
>                 Key: HADOOP-6580
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6580
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: c-6580.patch, HADOOP-6580-0_20.5.patch, HADOOP-6580.1.patch, HADOOP-6580.2.patch, HADOOP-6580.3.patch, HADOOP-6580.4.patch, HADOOP-6580.5.patch, HADOOP-6580.6.patch, HADOOP-6580.7.patch, HADOOP-6580.8.patch, HADOOP-6580.9.patch
>
>
> The UserGroupInformation should contain authentication method in its subject. This will be used in HDFS to issue delegation tokens only to kerberos authenticated clients.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7356-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 00:24:11 2010
Return-Path: <common-issues-return-7356-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47422 invoked from network); 13 Apr 2010 00:24:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 00:24:11 -0000
Received: (qmail 31575 invoked by uid 500); 13 Apr 2010 00:24:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31533 invoked by uid 500); 13 Apr 2010 00:24:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31525 invoked by uid 99); 13 Apr 2010 00:24:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 00:24:11 +0000
X-ASF-Spam-Status: No, hits=-1271.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 00:24:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3D0NocE003519
	for <common-issues@hadoop.apache.org>; Mon, 12 Apr 2010 20:23:50 -0400 (EDT)
Message-ID: <14275127.10811271118230075.JavaMail.jira@thor>
Date: Mon, 12 Apr 2010 20:23:50 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6692) Propose change to
 FileContext#listStatus
In-Reply-To: <253397745.22171270762596744.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856242#action_12856242 ] 

Eli Collins commented on HADOOP-6692:
-------------------------------------

Hey Hairong,

The patch looks good to me. Only one minor edit in addition to Suresh's comments: FileContext#listStatus will never throw UnresolvedLinkException so you can remove that from the throws clause and javadoc.

Thanks,
Eli

> Propose change to FileContext#listStatus
> ----------------------------------------
>
>                 Key: HADOOP-6692
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6692
>             Project: Hadoop Common
>          Issue Type: Sub-task
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listStatusItor.patch
>
>
> Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7357-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 03:00:17 2010
Return-Path: <common-issues-return-7357-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13119 invoked from network); 13 Apr 2010 03:00:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 03:00:17 -0000
Received: (qmail 29120 invoked by uid 500); 13 Apr 2010 03:00:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29011 invoked by uid 500); 13 Apr 2010 03:00:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29003 invoked by uid 99); 13 Apr 2010 03:00:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 03:00:16 +0000
X-ASF-Spam-Status: No, hits=-1272.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 03:00:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3D2xsKl005290
	for <common-issues@hadoop.apache.org>; Mon, 12 Apr 2010 22:59:55 -0400 (EDT)
Message-ID: <6436930.13721271127594879.JavaMail.jira@thor>
Date: Mon, 12 Apr 2010 22:59:54 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6659) Switch RPC to use Avro
In-Reply-To: <387448273.476441269469290584.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856284#action_12856284 ] 

Sanjay Radia commented on HADOOP-6659:
--------------------------------------

My viewpoint is in partial agreement with Doug and Eric:
# Implement the pluggable RPC in trunk and it needs to be well tested (agree with Doug)
# Implement the AVRO IDL based protocols in a branch(es). (agree with Eric).
# Only after step 2 do we declare the new protocols to be wire compatible in future. (we are all in agreement here)
# After step 1, leave the default to be current RPC - but any customer can easily change this by a config variable. (Disagree with Doug).

There is low risk in #1 as it is pluggable. 
Moving to Avro IDL will change many data types in our servers and cannot be pluggable. This needs to be well tested (correctness and performance) and cannot cross releases;  hence it is best done in a branch(es). HDFS and MR can be separate branches; however I don't think we can split the HDFS protocols as they use many common data types (block-id, block-token, etc). 


> Switch RPC to use Avro
> ----------------------
>
>                 Key: HADOOP-6659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6659
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Doug Cutting
>
> This is an umbrella issue for moving HDFS and MapReduce RPC to use Avro.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7358-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 05:03:19 2010
Return-Path: <common-issues-return-7358-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28266 invoked from network); 13 Apr 2010 05:03:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 05:03:19 -0000
Received: (qmail 91873 invoked by uid 500); 13 Apr 2010 05:03:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91763 invoked by uid 500); 13 Apr 2010 05:03:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91755 invoked by uid 99); 13 Apr 2010 05:03:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 05:03:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 05:03:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3D52smM006496
	for <common-issues@hadoop.apache.org>; Tue, 13 Apr 2010 01:02:54 -0400 (EDT)
Message-ID: <16356799.15441271134974402.JavaMail.jira@thor>
Date: Tue, 13 Apr 2010 01:02:54 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6659) Switch RPC to use Avro
In-Reply-To: <387448273.476441269469290584.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856303#action_12856303 ] 

Doug Cutting commented on HADOOP-6659:
--------------------------------------

Sanjay, I'm pleased to see we have so few differences.  I leave for a one week vacation tomorrow morning, and look forward to working with you more on this when I return.

> Switch RPC to use Avro
> ----------------------
>
>                 Key: HADOOP-6659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6659
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Doug Cutting
>
> This is an umbrella issue for moving HDFS and MapReduce RPC to use Avro.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7359-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 05:15:17 2010
Return-Path: <common-issues-return-7359-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30295 invoked from network); 13 Apr 2010 05:15:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 05:15:16 -0000
Received: (qmail 94895 invoked by uid 500); 13 Apr 2010 05:15:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94835 invoked by uid 500); 13 Apr 2010 05:15:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94824 invoked by uid 99); 13 Apr 2010 05:15:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 05:15:16 +0000
X-ASF-Spam-Status: No, hits=-1273.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 05:15:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3D5EsAw006655
	for <common-issues@hadoop.apache.org>; Tue, 13 Apr 2010 01:14:55 -0400 (EDT)
Message-ID: <25341396.15821271135694902.JavaMail.jira@thor>
Date: Tue, 13 Apr 2010 01:14:54 -0400 (EDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6659) Switch RPC to use Avro
In-Reply-To: <387448273.476441269469290584.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856304#action_12856304 ] 

Arun C Murthy commented on HADOOP-6659:
---------------------------------------

I think Sanjay's proposal is reasonable, and glad to see we are close to consensus on this important issue.

> Switch RPC to use Avro
> ----------------------
>
>                 Key: HADOOP-6659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6659
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Doug Cutting
>
> This is an umbrella issue for moving HDFS and MapReduce RPC to use Avro.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7360-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 13:37:19 2010
Return-Path: <common-issues-return-7360-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38010 invoked from network); 13 Apr 2010 13:37:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 13:37:19 -0000
Received: (qmail 49145 invoked by uid 500); 13 Apr 2010 13:37:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49040 invoked by uid 500); 13 Apr 2010 13:37:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49032 invoked by uid 99); 13 Apr 2010 13:37:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 13:37:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 13:37:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3DDar3j011237
	for <common-issues@hadoop.apache.org>; Tue, 13 Apr 2010 09:36:54 -0400 (EDT)
Message-ID: <16631073.24081271165813863.JavaMail.jira@thor>
Date: Tue, 13 Apr 2010 09:36:53 -0400 (EDT)
From: "Greg Wilkins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6528) Jetty returns -1 resulting in
 Hadoop masters / slaves to fail during startup.
In-Reply-To: <2116312423.157761264936054577.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856423#action_12856423 ] 

Greg Wilkins commented on HADOOP-6528:
--------------------------------------

I'm still pondering this one... and still can't think of an explanation other than something in hadoop is calling close.

I've attached a patch to http://jira.codehaus.org/browse/JETTY-748 that changes the way localPort is handled:
Instead of calling socket.getLocalPort() when connector.getLocalPort() is called, this patch calls socket.getLocalPort() in the open method and holds the result in a volatile int. If close is called, the int is set to -2, so this will tell us if the port has not been opened, or has been closed before it was used. 

This patch is not yet in a release, but if somebody has a setup where this problem occurs, then  we can produce a snapshot build for you to test against.

 



> Jetty returns -1 resulting in Hadoop masters / slaves to fail during startup.
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-6528
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6528
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hemanth Yamijala
>         Attachments: jetty-server-failure.log
>
>
> A recent test failure on Hudson seems to indicate that Jetty's Server.getConnectors()[0].getLocalPort() is returning -1 in the HttpServer.getPort() method. When this happens, Hadoop masters / slaves that use Jetty fail to startup correctly.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7361-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 17:33:22 2010
Return-Path: <common-issues-return-7361-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39718 invoked from network); 13 Apr 2010 16:44:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 16:44:02 -0000
Received: (qmail 1900 invoked by uid 500); 13 Apr 2010 16:44:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1767 invoked by uid 500); 13 Apr 2010 16:44:02 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1734 invoked by uid 99); 13 Apr 2010 16:44:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 16:44:02 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 16:43:59 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3DGhbAu015927
	for <common-issues@hadoop.apache.org>; Tue, 13 Apr 2010 12:43:38 -0400 (EDT)
Message-ID: <11723633.34461271177017775.JavaMail.jira@thor>
Date: Tue, 13 Apr 2010 12:43:37 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-1722) Make streaming to handle non-utf8
 byte array
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856491#action_12856491 ] 

Hudson commented on HADOOP-1722:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk #285 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/285/])
    MAPREDUCE-889. binary communication formats added to Streaming by HADOOP-1722 should be documented. Contributed by Klaas Bosteels.


> Make streaming to handle non-utf8 byte array
> --------------------------------------------
>
>                 Key: HADOOP-1722
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1722
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Runping Qi
>            Assignee: Klaas Bosteels
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-1722-branch-0.18.patch, HADOOP-1722-branch-0.19.patch, HADOOP-1722-v0.20.1.patch, HADOOP-1722-v2.patch, HADOOP-1722-v3.patch, HADOOP-1722-v4.patch, HADOOP-1722-v4.patch, HADOOP-1722-v5.patch, HADOOP-1722-v6.patch, HADOOP-1722.patch
>
>
> Right now, the streaming framework expects the output sof the steam process (mapper or reducer) are line 
> oriented UTF-8 text. This limit makes it impossible to use those programs whose outputs may be non-UTF-8
>  (international encoding, or maybe even binary data). Streaming can overcome this limit by introducing a simple
> encoding protocol. For example, it can allow the mapper/reducer to hexencode its keys/values, 
> the framework decodes them in the Java side.
> This way, as long as the mapper/reducer executables follow this encoding protocol, 
> they can output arabitary bytearray and the streaming framework can handle them.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7362-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 18:19:01 2010
Return-Path: <common-issues-return-7362-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66160 invoked from network); 13 Apr 2010 16:52:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 16:52:26 -0000
Received: (qmail 62573 invoked by uid 500); 13 Apr 2010 16:52:26 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62459 invoked by uid 500); 13 Apr 2010 16:52:26 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62449 invoked by uid 99); 13 Apr 2010 16:52:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 16:52:26 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 16:52:23 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3DGq1lZ016965
	for <common-issues@hadoop.apache.org>; Tue, 13 Apr 2010 12:52:02 -0400 (EDT)
Message-ID: <1967560.37611271177521152.JavaMail.jira@thor>
Date: Tue, 13 Apr 2010 12:52:01 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6689) Add directory renaming test to
 FileContextMainOperationsBaseTest
In-Reply-To: <1987122042.47681270660714756.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856492#action_12856492 ] 

Suresh Srinivas commented on HADOOP-6689:
-----------------------------------------

+1 for the patch.

> Add directory renaming test to FileContextMainOperationsBaseTest
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6689
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6689
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>         Attachments: hadoop-6689-1.patch, hadoop-6689-2.patch
>
>
> I noticed FileContextMainOperationsBaseTest does not have a test that renames an empty directory to an empty directory (and shows that this fails without the overwrite option). 

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7363-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 18:39:37 2010
Return-Path: <common-issues-return-7363-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17937 invoked from network); 13 Apr 2010 17:02:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 17:02:39 -0000
Received: (qmail 87100 invoked by uid 500); 13 Apr 2010 17:02:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87075 invoked by uid 500); 13 Apr 2010 17:02:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87067 invoked by uid 99); 13 Apr 2010 17:02:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 17:02:39 +0000
X-ASF-Spam-Status: No, hits=-1277.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 17:02:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3DH2Is1018287
	for <common-issues@hadoop.apache.org>; Tue, 13 Apr 2010 13:02:18 -0400 (EDT)
Message-ID: <6754793.41601271178138265.JavaMail.jira@thor>
Date: Tue, 13 Apr 2010 13:02:18 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6689) Add directory renaming test to
 FileContextMainOperationsBaseTest
In-Reply-To: <1987122042.47681270660714756.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6689:
------------------------------------

         Hadoop Flags: [Reviewed]
        Fix Version/s: 0.22.0
    Affects Version/s: 0.22.0

> Add directory renaming test to FileContextMainOperationsBaseTest
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6689
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6689
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6689-1.patch, hadoop-6689-2.patch
>
>
> I noticed FileContextMainOperationsBaseTest does not have a test that renames an empty directory to an empty directory (and shows that this fails without the overwrite option). 

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7364-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 18:41:36 2010
Return-Path: <common-issues-return-7364-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25633 invoked from network); 13 Apr 2010 17:04:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 17:04:22 -0000
Received: (qmail 92496 invoked by uid 500); 13 Apr 2010 17:04:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92466 invoked by uid 500); 13 Apr 2010 17:04:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92458 invoked by uid 99); 13 Apr 2010 17:04:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 17:04:22 +0000
X-ASF-Spam-Status: No, hits=-1277.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 17:04:21 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3DH41KK018408
	for <common-issues@hadoop.apache.org>; Tue, 13 Apr 2010 13:04:01 -0400 (EDT)
Message-ID: <20409538.41921271178241368.JavaMail.jira@thor>
Date: Tue, 13 Apr 2010 13:04:01 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6689) Add directory renaming test to
 FileContextMainOperationsBaseTest
In-Reply-To: <1987122042.47681270660714756.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6689:
------------------------------------

        Status: Resolved  (was: Patch Available)
    Resolution: Fixed

I committed the patch. Thank you Eli.

> Add directory renaming test to FileContextMainOperationsBaseTest
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6689
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6689
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6689-1.patch, hadoop-6689-2.patch
>
>
> I noticed FileContextMainOperationsBaseTest does not have a test that renames an empty directory to an empty directory (and shows that this fails without the overwrite option). 

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7365-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 18:52:09 2010
Return-Path: <common-issues-return-7365-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61536 invoked from network); 13 Apr 2010 17:10:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 17:10:43 -0000
Received: (qmail 10016 invoked by uid 500); 13 Apr 2010 17:10:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9990 invoked by uid 500); 13 Apr 2010 17:10:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9982 invoked by uid 99); 13 Apr 2010 17:10:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 17:10:43 +0000
X-ASF-Spam-Status: No, hits=-1277.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 17:10:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3DHAMXi019306
	for <common-issues@hadoop.apache.org>; Tue, 13 Apr 2010 13:10:22 -0400 (EDT)
Message-ID: <33296145.44351271178622255.JavaMail.jira@thor>
Date: Tue, 13 Apr 2010 13:10:22 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6689) Add directory renaming test to
 FileContextMainOperationsBaseTest
In-Reply-To: <1987122042.47681270660714756.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856503#action_12856503 ] 

Hudson commented on HADOOP-6689:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #217 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/217/])
    HADOOP-6689. Add directory renaming test to existing FileContext tests. Contributed by Eli Collins.


> Add directory renaming test to FileContextMainOperationsBaseTest
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6689
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6689
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6689-1.patch, hadoop-6689-2.patch
>
>
> I noticed FileContextMainOperationsBaseTest does not have a test that renames an empty directory to an empty directory (and shows that this fails without the overwrite option). 

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7366-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 18:55:03 2010
Return-Path: <common-issues-return-7366-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89977 invoked from network); 13 Apr 2010 17:15:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 17:15:37 -0000
Received: (qmail 28572 invoked by uid 500); 13 Apr 2010 17:15:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28552 invoked by uid 500); 13 Apr 2010 17:15:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28544 invoked by uid 99); 13 Apr 2010 17:15:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 17:15:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 17:15:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3DHFDQH019814
	for <common-issues@hadoop.apache.org>; Tue, 13 Apr 2010 13:15:13 -0400 (EDT)
Message-ID: <426140.45751271178913376.JavaMail.jira@thor>
Date: Tue, 13 Apr 2010 13:15:13 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6686) Remove redundant exception class name
 in unwrapped exceptions thrown at the RPC client
In-Reply-To: <70650712.27801270591413554.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6686:
------------------------------------

    Status: Patch Available  (was: Open)

> Remove redundant exception class name in unwrapped exceptions thrown at the RPC client
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6686
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6686
>             Project: Hadoop Common
>          Issue Type: Improvement
>         Environment: At RPC client, when the exception thrown by the server is unwrapped, the exception message includes redundant exception class name. This redundant information should be removed.
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6686.1.patch, HADOOP-6686.patch
>
>


-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7367-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 19:24:15 2010
Return-Path: <common-issues-return-7367-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25382 invoked from network); 13 Apr 2010 17:41:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 17:41:49 -0000
Received: (qmail 69329 invoked by uid 500); 13 Apr 2010 17:41:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69293 invoked by uid 500); 13 Apr 2010 17:41:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69285 invoked by uid 99); 13 Apr 2010 17:41:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 17:41:49 +0000
X-ASF-Spam-Status: No, hits=-1278.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 17:41:48 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3DHfSRM023245
	for <common-issues@hadoop.apache.org>; Tue, 13 Apr 2010 13:41:28 -0400 (EDT)
Message-ID: <27363908.51701271180488589.JavaMail.jira@thor>
Date: Tue, 13 Apr 2010 13:41:28 -0400 (EDT)
From: "eric baldeschwieler (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6659) Switch RPC to use Avro
In-Reply-To: <387448273.476441269469290584.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856512#action_12856512 ] 

eric baldeschwieler commented on HADOOP-6659:
---------------------------------------------

If we can get to agreement that the current RPC remains the default and that IDL work happens in a branch and that only after we have a reasonably complete backwards compat solution do we change to AVRO RPC by default, then I think we are good.

My concern is not with the already committed plugability, but with the change of the default RPC and incremental change over of the protocols.  Once that work starts, we are committed to finishing it and we can not do that in the timeframe of the next release IMO.  I'd love to be proven wrong in a branch...

Would we keep the plugable API once we make the transition?  Seems like plugable support of an IDL driven RPC will be hard / not valuable.  Thoughts?

> Switch RPC to use Avro
> ----------------------
>
>                 Key: HADOOP-6659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6659
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Doug Cutting
>
> This is an umbrella issue for moving HDFS and MapReduce RPC to use Avro.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7368-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 19:27:04 2010
Return-Path: <common-issues-return-7368-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48919 invoked from network); 13 Apr 2010 17:47:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 17:47:42 -0000
Received: (qmail 81280 invoked by uid 500); 13 Apr 2010 17:47:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81247 invoked by uid 500); 13 Apr 2010 17:47:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81239 invoked by uid 99); 13 Apr 2010 17:47:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 17:47:42 +0000
X-ASF-Spam-Status: No, hits=-1278.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 17:47:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3DHlKlX024363
	for <common-issues@hadoop.apache.org>; Tue, 13 Apr 2010 13:47:21 -0400 (EDT)
Message-ID: <23089091.52971271180840969.JavaMail.jira@thor>
Date: Tue, 13 Apr 2010 13:47:20 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6439) Shuffle deadlocks on wrong number
 of maps
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856513#action_12856513 ] 

Tom White commented on HADOOP-6439:
-----------------------------------

Is this ready to be committed?

> Shuffle deadlocks on wrong number of maps
> -----------------------------------------
>
>                 Key: HADOOP-6439
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6439
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Owen O'Malley
>            Assignee: V.V.Chaitanya Krishna
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6439-1.patch, HADOOP-6439-2.patch, HADOOP-6439-3.patch, HADOOP-6439-4.patch, HADOOP-6439-5.patch, HADOOP-6439-6.patch, mr-1252.patch
>
>
> The new shuffle assumes that the number of maps is correct. The new JobSubmitter sets the old value. Something misfires in the middle causing:
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is deprecated. Instead, use mapreduce.job.splitfile
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
> But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the job.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7369-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 20:56:16 2010
Return-Path: <common-issues-return-7369-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64604 invoked from network); 13 Apr 2010 20:56:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 20:56:15 -0000
Received: (qmail 82351 invoked by uid 500); 13 Apr 2010 20:56:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82319 invoked by uid 500); 13 Apr 2010 20:56:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82311 invoked by uid 99); 13 Apr 2010 20:56:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 20:56:15 +0000
X-ASF-Spam-Status: No, hits=-1280.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 20:56:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3DKtsNC009013
	for <common-issues@hadoop.apache.org>; Tue, 13 Apr 2010 16:55:54 -0400 (EDT)
Message-ID: <21947133.72601271192154792.JavaMail.jira@thor>
Date: Tue, 13 Apr 2010 16:55:54 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6701)  Incorrect exit codes for "dfs
 -chown", "dfs -chgrp"
In-Reply-To: <29784448.2801271097111992.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6701:
---------------------------------

    Attachment:     (was: HADOOP-6701-trunk.patch)

>  Incorrect exit codes for "dfs -chown", "dfs -chgrp"
> ----------------------------------------------------
>
>                 Key: HADOOP-6701
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6701
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.19.1, 0.20.0, 0.20.1, 0.20.2
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>            Priority: Minor
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>
> ravi@localhost:~$ hadoop dfs -chgrp abcd /; echo $?
> chgrp: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chown  abcd /; echo $?
> chown: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chmod 755 /DOESNTEXIST; echo $?
> chmod: could not get status for '/DOESNTEXIST': File does not exist: /DOESNTEXIST
> 0
> -
> Exit codes for both of the above invocations should be non-zero to indicate that the command failed.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7370-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 20:56:18 2010
Return-Path: <common-issues-return-7370-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64683 invoked from network); 13 Apr 2010 20:56:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 20:56:18 -0000
Received: (qmail 82505 invoked by uid 500); 13 Apr 2010 20:56:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82472 invoked by uid 500); 13 Apr 2010 20:56:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82464 invoked by uid 99); 13 Apr 2010 20:56:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 20:56:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 20:56:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3DKtsAO009004
	for <common-issues@hadoop.apache.org>; Tue, 13 Apr 2010 16:55:54 -0400 (EDT)
Message-ID: <22557697.72571271192154154.JavaMail.jira@thor>
Date: Tue, 13 Apr 2010 16:55:54 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6701)  Incorrect exit codes for "dfs
 -chown", "dfs -chgrp"
In-Reply-To: <29784448.2801271097111992.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6701:
---------------------------------

    Attachment:     (was: HADOOP-6701.patch)

>  Incorrect exit codes for "dfs -chown", "dfs -chgrp"
> ----------------------------------------------------
>
>                 Key: HADOOP-6701
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6701
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.19.1, 0.20.0, 0.20.1, 0.20.2
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>            Priority: Minor
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>
> ravi@localhost:~$ hadoop dfs -chgrp abcd /; echo $?
> chgrp: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chown  abcd /; echo $?
> chown: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chmod 755 /DOESNTEXIST; echo $?
> chmod: could not get status for '/DOESNTEXIST': File does not exist: /DOESNTEXIST
> 0
> -
> Exit codes for both of the above invocations should be non-zero to indicate that the command failed.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7371-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 21:04:16 2010
Return-Path: <common-issues-return-7371-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75334 invoked from network); 13 Apr 2010 21:04:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 21:04:16 -0000
Received: (qmail 2471 invoked by uid 500); 13 Apr 2010 21:04:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2442 invoked by uid 500); 13 Apr 2010 21:04:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2431 invoked by uid 99); 13 Apr 2010 21:04:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 21:04:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 21:04:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3DL3qKP009188
	for <common-issues@hadoop.apache.org>; Tue, 13 Apr 2010 17:03:52 -0400 (EDT)
Message-ID: <3004506.72901271192632679.JavaMail.jira@thor>
Date: Tue, 13 Apr 2010 17:03:52 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6701)  Incorrect exit codes for "dfs
 -chown", "dfs -chgrp"
In-Reply-To: <29784448.2801271097111992.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6701:
---------------------------------

    Attachment: HADOOP-6701.patch

Attaching patch to fix exit codes for {{fs -ch(own/mod/grp)}}.  Patch includes unit tests for theses issues.

This patch does not fix bug for incorrect exit code when wild card input is given to the {{fs -ch(own/mod/grp)}}. 
There is bug in {{FsShell.java}} which causes incorrect exit codes when wildcard inputs are given to -ch*  commands.

{code:title=Bar.java|borderStyle=solid}
 for (int i=startIndex; i<args.length; i++) {
      Path srcPath = new Path(args[i]);
      FileSystem srcFs = srcPath.getFileSystem(getConf());
      Path[] paths = FileUtil.stat2Paths(srcFs.globStatus(srcPath), srcPath);
      for(Path path : paths) {
        try {
{code}

I will open new Jira to fix this issue.

>  Incorrect exit codes for "dfs -chown", "dfs -chgrp"
> ----------------------------------------------------
>
>                 Key: HADOOP-6701
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6701
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.19.1, 0.20.0, 0.20.1, 0.20.2
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>            Priority: Minor
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6701.patch
>
>
> ravi@localhost:~$ hadoop dfs -chgrp abcd /; echo $?
> chgrp: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chown  abcd /; echo $?
> chown: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chmod 755 /DOESNTEXIST; echo $?
> chmod: could not get status for '/DOESNTEXIST': File does not exist: /DOESNTEXIST
> 0
> -
> Exit codes for both of the above invocations should be non-zero to indicate that the command failed.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7372-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 21:10:28 2010
Return-Path: <common-issues-return-7372-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81223 invoked from network); 13 Apr 2010 21:10:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 21:10:27 -0000
Received: (qmail 12815 invoked by uid 500); 13 Apr 2010 21:10:26 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12745 invoked by uid 500); 13 Apr 2010 21:10:26 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12612 invoked by uid 99); 13 Apr 2010 21:10:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 21:10:26 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 21:10:23 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3DLA175009358
	for <common-issues@hadoop.apache.org>; Tue, 13 Apr 2010 17:10:02 -0400 (EDT)
Message-ID: <30954625.73221271193001806.JavaMail.jira@thor>
Date: Tue, 13 Apr 2010 17:10:01 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6702) Incorrect exit codes for "dfs
 -chown", "dfs -chgrp"  when input is given in wildcard format.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Incorrect exit codes for "dfs -chown", "dfs -chgrp"  when input is given in wildcard format.
--------------------------------------------------------------------------------------------

                 Key: HADOOP-6702
                 URL: https://issues.apache.org/jira/browse/HADOOP-6702
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
    Affects Versions: 0.20.2, 0.20.1, 0.20.0, 0.19.1
            Reporter: Ravi Phulari
            Assignee: Ravi Phulari
            Priority: Minor
             Fix For: 0.20.3, 0.21.0, 0.22.0


Currently incorrect exit codes  are given for "dfs -chown", "dfs -chgrp"  when input is given in wildcard format.

This bug is due to missing update of errors count in {{FsShell.java}}.

{code:title=FsShell.java|borderStyle=solid}
int runCmdHandler(CmdHandler handler, String[] args,
                                   int startIndex, boolean recursive) 
                                   throws IOException {
    int errors = 0;
    
    for (int i=startIndex; i<args.length; i++) {
      Path srcPath = new Path(args[i]);
      FileSystem srcFs = srcPath.getFileSystem(getConf());
      Path[] paths = FileUtil.stat2Paths(srcFs.globStatus(srcPath), srcPath);
      for(Path path : paths) {
        try {
          FileStatus file = srcFs.getFileStatus(path);
          if (file == null) {
            System.err.println(handler.getName() + 
                               ": could not get status for '" + path + "'");
            errors++;
          } else {
            errors += runCmdHandler(handler, file, srcFs, recursive);
          }
        } catch (IOException e) {
          String msg = (e.getMessage() != null ? e.getLocalizedMessage() :
            (e.getCause().getMessage() != null ? 
                e.getCause().getLocalizedMessage() : "null"));
          System.err.println(handler.getName() + ": could not get status for '"
                                        + path + "': " + msg.split("\n")[0]);
          errors++;
        }
      }
    }
 {code}

If there are no files on HDFS matching to wildcard input then  {{srcFs.globStatus(srcpath)}} returns 0. 
{{ Path[] paths = FileUtil.stat2Paths(srcFs.globStatus(srcPath), srcPath);}}

Resulting no increment in {{errors}} and command exits with 0 even though file/directory does not exist.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7373-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 21:36:15 2010
Return-Path: <common-issues-return-7373-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35424 invoked from network); 13 Apr 2010 21:36:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 21:36:12 -0000
Received: (qmail 45606 invoked by uid 500); 13 Apr 2010 21:36:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45582 invoked by uid 500); 13 Apr 2010 21:36:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45574 invoked by uid 99); 13 Apr 2010 21:36:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 21:36:12 +0000
X-ASF-Spam-Status: No, hits=-1281.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 21:36:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3DLZpF9009952
	for <common-issues@hadoop.apache.org>; Tue, 13 Apr 2010 17:35:51 -0400 (EDT)
Message-ID: <23186149.74901271194550994.JavaMail.jira@thor>
Date: Tue, 13 Apr 2010 17:35:50 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6703) Prevent renaming a file, symlink or
 directory to itself
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Prevent renaming a file, symlink or directory to itself
-------------------------------------------------------

                 Key: HADOOP-6703
                 URL: https://issues.apache.org/jira/browse/HADOOP-6703
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.22.0
            Reporter: Eli Collins
            Assignee: Eli Collins
            Priority: Minor
             Fix For: 0.22.0


Per HDFS-1088 let's throw a FileAlreadyExistsException if renaming a file, symlink or directory to itself, or a symlink to the file it links to. 


-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7374-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 21:50:15 2010
Return-Path: <common-issues-return-7374-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68246 invoked from network); 13 Apr 2010 21:50:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 21:50:13 -0000
Received: (qmail 63656 invoked by uid 500); 13 Apr 2010 21:50:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63623 invoked by uid 500); 13 Apr 2010 21:50:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63615 invoked by uid 99); 13 Apr 2010 21:50:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 21:50:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 21:50:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3DLnnOi010391
	for <common-issues@hadoop.apache.org>; Tue, 13 Apr 2010 17:49:49 -0400 (EDT)
Message-ID: <17194546.76151271195389942.JavaMail.jira@thor>
Date: Tue, 13 Apr 2010 17:49:49 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6703) Prevent renaming a file, symlink or
 directory to itself
In-Reply-To: <23186149.74901271194550994.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6703:
--------------------------------

    Attachment: hadoop-6703-1.patch

Patch attached.

> Prevent renaming a file, symlink or directory to itself
> -------------------------------------------------------
>
>                 Key: HADOOP-6703
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6703
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6703-1.patch
>
>
> Per HDFS-1088 let's throw a FileAlreadyExistsException if renaming a file, symlink or directory to itself, or a symlink to the file it links to. 

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7375-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 22:12:13 2010
Return-Path: <common-issues-return-7375-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23722 invoked from network); 13 Apr 2010 22:12:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 22:12:12 -0000
Received: (qmail 89944 invoked by uid 500); 13 Apr 2010 22:12:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89918 invoked by uid 500); 13 Apr 2010 22:12:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89907 invoked by uid 99); 13 Apr 2010 22:12:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 22:12:12 +0000
X-ASF-Spam-Status: No, hits=-1281.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 22:12:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3DMBpDf012902
	for <common-issues@hadoop.apache.org>; Tue, 13 Apr 2010 18:11:51 -0400 (EDT)
Message-ID: <31038474.77701271196711062.JavaMail.jira@thor>
Date: Tue, 13 Apr 2010 18:11:51 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6563:
--------------------------------

    Status: In Progress  (was: Patch Available)

> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>         Attachments: hadoop-6563-1.patch, hadoop-6563-2.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7376-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 22:12:14 2010
Return-Path: <common-issues-return-7376-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23747 invoked from network); 13 Apr 2010 22:12:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 22:12:12 -0000
Received: (qmail 90105 invoked by uid 500); 13 Apr 2010 22:12:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90051 invoked by uid 500); 13 Apr 2010 22:12:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90043 invoked by uid 99); 13 Apr 2010 22:12:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 22:12:12 +0000
X-ASF-Spam-Status: No, hits=-1281.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 22:12:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3DMBpx8012914
	for <common-issues@hadoop.apache.org>; Tue, 13 Apr 2010 18:11:51 -0400 (EDT)
Message-ID: <30805710.77741271196711420.JavaMail.jira@thor>
Date: Tue, 13 Apr 2010 18:11:51 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6563:
--------------------------------

        Fix Version/s: 0.22.0
    Affects Version/s: 0.22.0

> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6563-1.patch, hadoop-6563-2.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7377-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 22:12:18 2010
Return-Path: <common-issues-return-7377-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23958 invoked from network); 13 Apr 2010 22:12:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 22:12:16 -0000
Received: (qmail 90285 invoked by uid 500); 13 Apr 2010 22:12:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90195 invoked by uid 500); 13 Apr 2010 22:12:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90181 invoked by uid 99); 13 Apr 2010 22:12:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 22:12:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 22:12:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3DMBoBk012889
	for <common-issues@hadoop.apache.org>; Tue, 13 Apr 2010 18:11:50 -0400 (EDT)
Message-ID: <30245278.77661271196710164.JavaMail.jira@thor>
Date: Tue, 13 Apr 2010 18:11:50 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6563:
--------------------------------

    Attachment: hadoop-6563-2.patch

Fixes the javadoc warning.

> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6563-1.patch, hadoop-6563-2.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7378-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 22:12:20 2010
Return-Path: <common-issues-return-7378-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24010 invoked from network); 13 Apr 2010 22:12:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 22:12:16 -0000
Received: (qmail 90966 invoked by uid 500); 13 Apr 2010 22:12:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90922 invoked by uid 500); 13 Apr 2010 22:12:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90874 invoked by uid 99); 13 Apr 2010 22:12:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 22:12:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 22:12:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3DMBpn2012908
	for <common-issues@hadoop.apache.org>; Tue, 13 Apr 2010 18:11:51 -0400 (EDT)
Message-ID: <5465276.77721271196711249.JavaMail.jira@thor>
Date: Tue, 13 Apr 2010 18:11:51 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6563:
--------------------------------

    Status: Patch Available  (was: In Progress)

> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>         Attachments: hadoop-6563-1.patch, hadoop-6563-2.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7379-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 22:20:14 2010
Return-Path: <common-issues-return-7379-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42202 invoked from network); 13 Apr 2010 22:20:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 22:20:13 -0000
Received: (qmail 1857 invoked by uid 500); 13 Apr 2010 22:20:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1829 invoked by uid 500); 13 Apr 2010 22:20:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1821 invoked by uid 99); 13 Apr 2010 22:20:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 22:20:12 +0000
X-ASF-Spam-Status: No, hits=-1281.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 22:20:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3DMJp2Q013865
	for <common-issues@hadoop.apache.org>; Tue, 13 Apr 2010 18:19:51 -0400 (EDT)
Message-ID: <19417201.78041271197191331.JavaMail.jira@thor>
Date: Tue, 13 Apr 2010 18:19:51 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6580) UGI should contain authentication
 method.
In-Reply-To: <1786976638.415521266712107907.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856655#action_12856655 ] 

Jitendra Nath Pandey commented on HADOOP-6580:
----------------------------------------------

test-patch results

     [exec] +1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 6 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.


> UGI should contain authentication method.
> -----------------------------------------
>
>                 Key: HADOOP-6580
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6580
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: c-6580.patch, HADOOP-6580-0_20.5.patch, HADOOP-6580.1.patch, HADOOP-6580.2.patch, HADOOP-6580.3.patch, HADOOP-6580.4.patch, HADOOP-6580.5.patch, HADOOP-6580.6.patch, HADOOP-6580.7.patch, HADOOP-6580.8.patch, HADOOP-6580.9.patch
>
>
> The UserGroupInformation should contain authentication method in its subject. This will be used in HDFS to issue delegation tokens only to kerberos authenticated clients.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7380-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 22:20:19 2010
Return-Path: <common-issues-return-7380-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42390 invoked from network); 13 Apr 2010 22:20:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 22:20:17 -0000
Received: (qmail 2112 invoked by uid 500); 13 Apr 2010 22:20:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2083 invoked by uid 500); 13 Apr 2010 22:20:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2075 invoked by uid 99); 13 Apr 2010 22:20:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 22:20:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 22:20:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3DMJrOX013892
	for <common-issues@hadoop.apache.org>; Tue, 13 Apr 2010 18:19:53 -0400 (EDT)
Message-ID: <25450725.78111271197193769.JavaMail.jira@thor>
Date: Tue, 13 Apr 2010 18:19:53 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6580) UGI should contain authentication
 method.
In-Reply-To: <1786976638.415521266712107907.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856657#action_12856657 ] 

Jitendra Nath Pandey commented on HADOOP-6580:
----------------------------------------------

ant test also ran successfully.

> UGI should contain authentication method.
> -----------------------------------------
>
>                 Key: HADOOP-6580
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6580
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: c-6580.patch, HADOOP-6580-0_20.5.patch, HADOOP-6580.1.patch, HADOOP-6580.2.patch, HADOOP-6580.3.patch, HADOOP-6580.4.patch, HADOOP-6580.5.patch, HADOOP-6580.6.patch, HADOOP-6580.7.patch, HADOOP-6580.8.patch, HADOOP-6580.9.patch
>
>
> The UserGroupInformation should contain authentication method in its subject. This will be used in HDFS to issue delegation tokens only to kerberos authenticated clients.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7381-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 23:18:12 2010
Return-Path: <common-issues-return-7381-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97375 invoked from network); 13 Apr 2010 23:18:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 23:18:12 -0000
Received: (qmail 58862 invoked by uid 500); 13 Apr 2010 23:18:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58822 invoked by uid 500); 13 Apr 2010 23:18:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58814 invoked by uid 99); 13 Apr 2010 23:18:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 23:18:12 +0000
X-ASF-Spam-Status: No, hits=-1282.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 23:18:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3DNHoHl018127
	for <common-issues@hadoop.apache.org>; Tue, 13 Apr 2010 19:17:50 -0400 (EDT)
Message-ID: <18417308.80631271200670196.JavaMail.jira@thor>
Date: Tue, 13 Apr 2010 19:17:50 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6580) UGI should contain authentication
 method.
In-Reply-To: <1786976638.415521266712107907.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6580:
-----------------------------------

        Status: Resolved  (was: Patch Available)
    Resolution: Fixed

committed this. Thanks Jitendra.

> UGI should contain authentication method.
> -----------------------------------------
>
>                 Key: HADOOP-6580
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6580
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: c-6580.patch, HADOOP-6580-0_20.5.patch, HADOOP-6580.1.patch, HADOOP-6580.2.patch, HADOOP-6580.3.patch, HADOOP-6580.4.patch, HADOOP-6580.5.patch, HADOOP-6580.6.patch, HADOOP-6580.7.patch, HADOOP-6580.8.patch, HADOOP-6580.9.patch
>
>
> The UserGroupInformation should contain authentication method in its subject. This will be used in HDFS to issue delegation tokens only to kerberos authenticated clients.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7382-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 13 23:22:17 2010
Return-Path: <common-issues-return-7382-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1671 invoked from network); 13 Apr 2010 23:22:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Apr 2010 23:22:17 -0000
Received: (qmail 63092 invoked by uid 500); 13 Apr 2010 23:22:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63026 invoked by uid 500); 13 Apr 2010 23:22:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63014 invoked by uid 99); 13 Apr 2010 23:22:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 23:22:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Apr 2010 23:22:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3DNLqhr018195
	for <common-issues@hadoop.apache.org>; Tue, 13 Apr 2010 19:21:53 -0400 (EDT)
Message-ID: <9190855.80771271200912913.JavaMail.jira@thor>
Date: Tue, 13 Apr 2010 19:21:52 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6580) UGI should contain authentication
 method.
In-Reply-To: <1786976638.415521266712107907.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856683#action_12856683 ] 

Hudson commented on HADOOP-6580:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #218 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/218/])
    HADOOP-6580. UGI should contain authentication method.


> UGI should contain authentication method.
> -----------------------------------------
>
>                 Key: HADOOP-6580
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6580
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: c-6580.patch, HADOOP-6580-0_20.5.patch, HADOOP-6580.1.patch, HADOOP-6580.2.patch, HADOOP-6580.3.patch, HADOOP-6580.4.patch, HADOOP-6580.5.patch, HADOOP-6580.6.patch, HADOOP-6580.7.patch, HADOOP-6580.8.patch, HADOOP-6580.9.patch
>
>
> The UserGroupInformation should contain authentication method in its subject. This will be used in HDFS to issue delegation tokens only to kerberos authenticated clients.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7383-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 03:06:15 2010
Return-Path: <common-issues-return-7383-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18989 invoked from network); 14 Apr 2010 03:06:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 03:06:14 -0000
Received: (qmail 22925 invoked by uid 500); 14 Apr 2010 03:06:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22814 invoked by uid 500); 14 Apr 2010 03:06:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22806 invoked by uid 99); 14 Apr 2010 03:06:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 03:06:13 +0000
X-ASF-Spam-Status: No, hits=-1283.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 03:06:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3E35qfm012770
	for <common-issues@hadoop.apache.org>; Tue, 13 Apr 2010 23:05:52 -0400 (EDT)
Message-ID: <9346772.90011271214352565.JavaMail.jira@thor>
Date: Tue, 13 Apr 2010 23:05:52 -0400 (EDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6439) Shuffle deadlocks on wrong number
 of maps
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856732#action_12856732 ] 

V.V.Chaitanya Krishna commented on HADOOP-6439:
-----------------------------------------------

The patch applies even after a long time. *smile*

Running through hudson again.

> Shuffle deadlocks on wrong number of maps
> -----------------------------------------
>
>                 Key: HADOOP-6439
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6439
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Owen O'Malley
>            Assignee: V.V.Chaitanya Krishna
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6439-1.patch, HADOOP-6439-2.patch, HADOOP-6439-3.patch, HADOOP-6439-4.patch, HADOOP-6439-5.patch, HADOOP-6439-6.patch, mr-1252.patch
>
>
> The new shuffle assumes that the number of maps is correct. The new JobSubmitter sets the old value. Something misfires in the middle causing:
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is deprecated. Instead, use mapreduce.job.splitfile
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
> But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the job.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7384-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 03:08:14 2010
Return-Path: <common-issues-return-7384-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19177 invoked from network); 14 Apr 2010 03:08:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 03:08:14 -0000
Received: (qmail 23146 invoked by uid 500); 14 Apr 2010 03:08:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23080 invoked by uid 500); 14 Apr 2010 03:08:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23071 invoked by uid 99); 14 Apr 2010 03:08:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 03:08:13 +0000
X-ASF-Spam-Status: No, hits=-1283.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 03:08:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3E37q6b020031
	for <common-issues@hadoop.apache.org>; Tue, 13 Apr 2010 23:07:52 -0400 (EDT)
Message-ID: <29898453.90251271214472549.JavaMail.jira@thor>
Date: Tue, 13 Apr 2010 23:07:52 -0400 (EDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6439) Shuffle deadlocks on wrong number of
 maps
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna updated HADOOP-6439:
------------------------------------------

    Attachment: HADOOP-6439-6.patch

Re-submitting the same patch for Hudson.

> Shuffle deadlocks on wrong number of maps
> -----------------------------------------
>
>                 Key: HADOOP-6439
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6439
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Owen O'Malley
>            Assignee: V.V.Chaitanya Krishna
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6439-1.patch, HADOOP-6439-2.patch, HADOOP-6439-3.patch, HADOOP-6439-4.patch, HADOOP-6439-5.patch, HADOOP-6439-6.patch, HADOOP-6439-6.patch, mr-1252.patch
>
>
> The new shuffle assumes that the number of maps is correct. The new JobSubmitter sets the old value. Something misfires in the middle causing:
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is deprecated. Instead, use mapreduce.job.splitfile
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
> But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the job.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7385-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 03:08:15 2010
Return-Path: <common-issues-return-7385-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19232 invoked from network); 14 Apr 2010 03:08:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 03:08:15 -0000
Received: (qmail 23270 invoked by uid 500); 14 Apr 2010 03:08:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23220 invoked by uid 500); 14 Apr 2010 03:08:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23212 invoked by uid 99); 14 Apr 2010 03:08:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 03:08:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 03:08:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3E37pia019992
	for <common-issues@hadoop.apache.org>; Tue, 13 Apr 2010 23:07:51 -0400 (EDT)
Message-ID: <17970817.90121271214471313.JavaMail.jira@thor>
Date: Tue, 13 Apr 2010 23:07:51 -0400 (EDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6439) Shuffle deadlocks on wrong number of
 maps
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna updated HADOOP-6439:
------------------------------------------

    Status: Open  (was: Patch Available)

> Shuffle deadlocks on wrong number of maps
> -----------------------------------------
>
>                 Key: HADOOP-6439
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6439
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Owen O'Malley
>            Assignee: V.V.Chaitanya Krishna
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6439-1.patch, HADOOP-6439-2.patch, HADOOP-6439-3.patch, HADOOP-6439-4.patch, HADOOP-6439-5.patch, HADOOP-6439-6.patch, HADOOP-6439-6.patch, mr-1252.patch
>
>
> The new shuffle assumes that the number of maps is correct. The new JobSubmitter sets the old value. Something misfires in the middle causing:
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is deprecated. Instead, use mapreduce.job.splitfile
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
> But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the job.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7386-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 03:08:16 2010
Return-Path: <common-issues-return-7386-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19249 invoked from network); 14 Apr 2010 03:08:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 03:08:15 -0000
Received: (qmail 23399 invoked by uid 500); 14 Apr 2010 03:08:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23352 invoked by uid 500); 14 Apr 2010 03:08:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23344 invoked by uid 99); 14 Apr 2010 03:08:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 03:08:15 +0000
X-ASF-Spam-Status: No, hits=-1283.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 03:08:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3E37skU020070
	for <common-issues@hadoop.apache.org>; Tue, 13 Apr 2010 23:07:54 -0400 (EDT)
Message-ID: <30160009.90381271214474283.JavaMail.jira@thor>
Date: Tue, 13 Apr 2010 23:07:54 -0400 (EDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6439) Shuffle deadlocks on wrong number of
 maps
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna updated HADOOP-6439:
------------------------------------------

    Status: Patch Available  (was: Open)

> Shuffle deadlocks on wrong number of maps
> -----------------------------------------
>
>                 Key: HADOOP-6439
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6439
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Owen O'Malley
>            Assignee: V.V.Chaitanya Krishna
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6439-1.patch, HADOOP-6439-2.patch, HADOOP-6439-3.patch, HADOOP-6439-4.patch, HADOOP-6439-5.patch, HADOOP-6439-6.patch, HADOOP-6439-6.patch, mr-1252.patch
>
>
> The new shuffle assumes that the number of maps is correct. The new JobSubmitter sets the old value. Something misfires in the middle causing:
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is deprecated. Instead, use mapreduce.job.splitfile
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
> But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the job.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7387-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 06:01:17 2010
Return-Path: <common-issues-return-7387-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59759 invoked from network); 14 Apr 2010 06:01:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 06:01:16 -0000
Received: (qmail 27316 invoked by uid 500); 14 Apr 2010 06:01:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27200 invoked by uid 500); 14 Apr 2010 06:01:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27192 invoked by uid 99); 14 Apr 2010 06:01:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 06:01:15 +0000
X-ASF-Spam-Status: No, hits=-1284.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 06:01:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3E60suB022400
	for <common-issues@hadoop.apache.org>; Wed, 14 Apr 2010 02:00:54 -0400 (EDT)
Message-ID: <12424868.95681271224854158.JavaMail.jira@thor>
Date: Wed, 14 Apr 2010 02:00:54 -0400 (EDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6439) Shuffle deadlocks on wrong number of
 maps
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna updated HADOOP-6439:
------------------------------------------

    Status: Open  (was: Patch Available)

> Shuffle deadlocks on wrong number of maps
> -----------------------------------------
>
>                 Key: HADOOP-6439
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6439
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Owen O'Malley
>            Assignee: V.V.Chaitanya Krishna
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6439-1.patch, HADOOP-6439-2.patch, HADOOP-6439-3.patch, HADOOP-6439-4.patch, HADOOP-6439-5.patch, HADOOP-6439-6.patch, HADOOP-6439-6.patch, mr-1252.patch
>
>
> The new shuffle assumes that the number of maps is correct. The new JobSubmitter sets the old value. Something misfires in the middle causing:
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is deprecated. Instead, use mapreduce.job.splitfile
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
> But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the job.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7388-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 06:01:19 2010
Return-Path: <common-issues-return-7388-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59819 invoked from network); 14 Apr 2010 06:01:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 06:01:19 -0000
Received: (qmail 27386 invoked by uid 500); 14 Apr 2010 06:01:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27354 invoked by uid 500); 14 Apr 2010 06:01:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27346 invoked by uid 99); 14 Apr 2010 06:01:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 06:01:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 06:01:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3E60tCv022433
	for <common-issues@hadoop.apache.org>; Wed, 14 Apr 2010 02:00:55 -0400 (EDT)
Message-ID: <9779708.95791271224855465.JavaMail.jira@thor>
Date: Wed, 14 Apr 2010 02:00:55 -0400 (EDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6439) Shuffle deadlocks on wrong number of
 maps
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna updated HADOOP-6439:
------------------------------------------

    Status: Patch Available  (was: Open)

> Shuffle deadlocks on wrong number of maps
> -----------------------------------------
>
>                 Key: HADOOP-6439
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6439
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Owen O'Malley
>            Assignee: V.V.Chaitanya Krishna
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6439-1.patch, HADOOP-6439-2.patch, HADOOP-6439-3.patch, HADOOP-6439-4.patch, HADOOP-6439-5.patch, HADOOP-6439-6.patch, HADOOP-6439-6.patch, mr-1252.patch
>
>
> The new shuffle assumes that the number of maps is correct. The new JobSubmitter sets the old value. Something misfires in the middle causing:
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is deprecated. Instead, use mapreduce.job.splitfile
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
> But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the job.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7389-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 08:27:13 2010
Return-Path: <common-issues-return-7389-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43976 invoked from network); 14 Apr 2010 08:27:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 08:27:13 -0000
Received: (qmail 65720 invoked by uid 500); 14 Apr 2010 08:27:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65608 invoked by uid 500); 14 Apr 2010 08:27:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65599 invoked by uid 99); 14 Apr 2010 08:27:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 08:27:12 +0000
X-ASF-Spam-Status: No, hits=-1284.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 08:27:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3E8Qp3l026361
	for <common-issues@hadoop.apache.org>; Wed, 14 Apr 2010 04:26:51 -0400 (EDT)
Message-ID: <32604308.104881271233611247.JavaMail.jira@thor>
Date: Wed, 14 Apr 2010 04:26:51 -0400 (EDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6439) Shuffle deadlocks on wrong number
 of maps
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856796#action_12856796 ] 

V.V.Chaitanya Krishna commented on HADOOP-6439:
-----------------------------------------------

The Hudson result doesn't seem to be posting here. The result is as below:
{noformat}
  [exec] BUILD SUCCESSFUL
     [exec] Total time: 7 seconds
     [exec] 
     [exec] 
     [exec] 
     [exec] 
     [exec] +1 overall.  Here are the results of testing the latest attachment 
     [exec]   http://issues.apache.org/jira/secure/attachment/12441685/HADOOP-6439-6.patch
     [exec]   against trunk revision 933810.
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec] 
     [exec]     +1 core tests.  The patch passed core unit tests.
     [exec] 
     [exec]     +1 contrib tests.  The patch passed contrib unit tests.
     [exec] 
     [exec] Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/462/testReport/
     [exec] Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/462/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
     [exec] Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/462/artifact/trunk/build/test/checkstyle-errors.html
     [exec] Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/462/console
     [exec] 

{noformat}

> Shuffle deadlocks on wrong number of maps
> -----------------------------------------
>
>                 Key: HADOOP-6439
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6439
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Owen O'Malley
>            Assignee: V.V.Chaitanya Krishna
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6439-1.patch, HADOOP-6439-2.patch, HADOOP-6439-3.patch, HADOOP-6439-4.patch, HADOOP-6439-5.patch, HADOOP-6439-6.patch, HADOOP-6439-6.patch, mr-1252.patch
>
>
> The new shuffle assumes that the number of maps is correct. The new JobSubmitter sets the old value. Something misfires in the middle causing:
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is deprecated. Instead, use mapreduce.job.splitfile
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
> But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the job.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7390-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 15:18:13 2010
Return-Path: <common-issues-return-7390-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19372 invoked from network); 14 Apr 2010 15:18:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 15:18:13 -0000
Received: (qmail 87375 invoked by uid 500); 14 Apr 2010 15:18:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87350 invoked by uid 500); 14 Apr 2010 15:18:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87342 invoked by uid 99); 14 Apr 2010 15:18:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 15:18:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 15:18:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3EFHmR0006224
	for <common-issues@hadoop.apache.org>; Wed, 14 Apr 2010 11:17:49 -0400 (EDT)
Message-ID: <6991209.115951271258268758.JavaMail.jira@thor>
Date: Wed, 14 Apr 2010 11:17:48 -0400 (EDT)
From: "Neil Bliss (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6704) add support for Parascale filesystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

add support for Parascale filesystem
------------------------------------

                 Key: HADOOP-6704
                 URL: https://issues.apache.org/jira/browse/HADOOP-6704
             Project: Hadoop Common
          Issue Type: New Feature
          Components: fs
    Affects Versions: 0.20.2
            Reporter: Neil Bliss


Parascale has developed an org.apache.hadoop.fs implementation that allows users to use Hadoop on Parascale storage clusters.  We'd like to contribute this work to the community. Should this be placed under contrib, or integrated into the org.apache.hadoop.fs space?

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7391-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 15:46:19 2010
Return-Path: <common-issues-return-7391-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42837 invoked from network); 14 Apr 2010 15:46:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 15:46:16 -0000
Received: (qmail 59787 invoked by uid 500); 14 Apr 2010 15:46:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59749 invoked by uid 500); 14 Apr 2010 15:46:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59716 invoked by uid 99); 14 Apr 2010 15:46:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 15:46:15 +0000
X-ASF-Spam-Status: No, hits=-1287.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 15:46:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3EFjsLW006795
	for <common-issues@hadoop.apache.org>; Wed, 14 Apr 2010 11:45:54 -0400 (EDT)
Message-ID: <13986423.116831271259954541.JavaMail.jira@thor>
Date: Wed, 14 Apr 2010 11:45:54 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6704) add support for Parascale
 filesystem
In-Reply-To: <6991209.115951271258268758.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856935#action_12856935 ] 

Eli Collins commented on HADOOP-6704:
-------------------------------------

Seems like it should go in o.a.h.fs with the other non-hdfs file systems if it's going to be actively maintained. Will need some unit tests to help prevent people from breaking it since hudson doesn't have a parascale backend.

> add support for Parascale filesystem
> ------------------------------------
>
>                 Key: HADOOP-6704
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6704
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.20.2
>            Reporter: Neil Bliss
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Parascale has developed an org.apache.hadoop.fs implementation that allows users to use Hadoop on Parascale storage clusters.  We'd like to contribute this work to the community. Should this be placed under contrib, or integrated into the org.apache.hadoop.fs space?

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7392-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 15:48:12 2010
Return-Path: <common-issues-return-7392-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44500 invoked from network); 14 Apr 2010 15:48:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 15:48:12 -0000
Received: (qmail 64948 invoked by uid 500); 14 Apr 2010 15:48:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64896 invoked by uid 500); 14 Apr 2010 15:48:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64824 invoked by uid 99); 14 Apr 2010 15:48:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 15:48:12 +0000
X-ASF-Spam-Status: No, hits=-1287.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 15:48:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3EFloVA006842
	for <common-issues@hadoop.apache.org>; Wed, 14 Apr 2010 11:47:51 -0400 (EDT)
Message-ID: <4088062.116921271260070920.JavaMail.jira@thor>
Date: Wed, 14 Apr 2010 11:47:50 -0400 (EDT)
From: "Neil Bliss (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6704) add support for Parascale
 filesystem
In-Reply-To: <6991209.115951271258268758.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856937#action_12856937 ] 

Neil Bliss commented on HADOOP-6704:
------------------------------------

ok, patch forthcoming shortly.

> add support for Parascale filesystem
> ------------------------------------
>
>                 Key: HADOOP-6704
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6704
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.20.2
>            Reporter: Neil Bliss
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Parascale has developed an org.apache.hadoop.fs implementation that allows users to use Hadoop on Parascale storage clusters.  We'd like to contribute this work to the community. Should this be placed under contrib, or integrated into the org.apache.hadoop.fs space?

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7393-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 23:11:14 2010
Return-Path: <common-issues-return-7393-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16041 invoked from network); 14 Apr 2010 23:11:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 23:11:14 -0000
Received: (qmail 90196 invoked by uid 500); 14 Apr 2010 23:11:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90104 invoked by uid 500); 14 Apr 2010 23:11:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90088 invoked by uid 99); 14 Apr 2010 23:11:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 23:11:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 23:11:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3ENAmM4012275
	for <common-issues@hadoop.apache.org>; Wed, 14 Apr 2010 19:10:49 -0400 (EDT)
Message-ID: <5861257.127621271286648655.JavaMail.jira@thor>
Date: Wed, 14 Apr 2010 19:10:48 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6705) jiracli fails to upload test-patch
 comments to jira
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

jiracli fails to upload test-patch comments to jira
---------------------------------------------------

                 Key: HADOOP-6705
                 URL: https://issues.apache.org/jira/browse/HADOOP-6705
             Project: Hadoop Common
          Issue Type: Test
            Reporter: Giridharan Kesavan
            Assignee: Giridharan Kesavan


     [exec] ======================================================================
     [exec]     Adding comment to Jira.
     [exec] ======================================================================
     [exec] ======================================================================
     [exec] 
     [exec] 
     [exec] Failed to connect to: http://issues.apache.org/jira/rpc/soap/jirasoapservice-v2?wsdl
     [exec] Failed to connect to: http://issues.apache.org/jira/rpc/soap/jirasoapservice-v2?wsdl
     [exec] Failed to connect to: http://issues.apache.org/jira/rpc/soap/jirasoapservice-v2?wsdl
     [exec]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current


-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7394-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 23:31:11 2010
Return-Path: <common-issues-return-7394-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30971 invoked from network); 14 Apr 2010 23:31:10 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 23:31:10 -0000
Received: (qmail 7619 invoked by uid 500); 14 Apr 2010 23:31:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7507 invoked by uid 500); 14 Apr 2010 23:31:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7499 invoked by uid 99); 14 Apr 2010 23:31:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 23:31:10 +0000
X-ASF-Spam-Status: No, hits=-1290.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 23:31:09 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3ENUm7t012452
	for <common-issues@hadoop.apache.org>; Wed, 14 Apr 2010 19:30:49 -0400 (EDT)
Message-ID: <1788317.127991271287848814.JavaMail.jira@thor>
Date: Wed, 14 Apr 2010 19:30:48 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6705) jiracli fails to upload test-patch
 comments to jira
In-Reply-To: <5861257.127621271286648655.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6705:
---------------------------------------

    Attachment: HADOOP-6705.PATCH

I ve configured a different version of jira cli that works with the current jira version. This patch would help the test-patch script to 
work with the new jiracli


> jiracli fails to upload test-patch comments to jira
> ---------------------------------------------------
>
>                 Key: HADOOP-6705
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6705
>             Project: Hadoop Common
>          Issue Type: Test
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>         Attachments: HADOOP-6705.PATCH
>
>
>      [exec] ======================================================================
>      [exec]     Adding comment to Jira.
>      [exec] ======================================================================
>      [exec] ======================================================================
>      [exec] 
>      [exec] 
>      [exec] Failed to connect to: http://issues.apache.org/jira/rpc/soap/jirasoapservice-v2?wsdl
>      [exec] Failed to connect to: http://issues.apache.org/jira/rpc/soap/jirasoapservice-v2?wsdl
>      [exec] Failed to connect to: http://issues.apache.org/jira/rpc/soap/jirasoapservice-v2?wsdl
>      [exec]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7395-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 14 23:33:12 2010
Return-Path: <common-issues-return-7395-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33861 invoked from network); 14 Apr 2010 23:33:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Apr 2010 23:33:12 -0000
Received: (qmail 12469 invoked by uid 500); 14 Apr 2010 23:33:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12439 invoked by uid 500); 14 Apr 2010 23:33:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12431 invoked by uid 99); 14 Apr 2010 23:33:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 23:33:12 +0000
X-ASF-Spam-Status: No, hits=-1290.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 23:33:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3ENWoEE012468
	for <common-issues@hadoop.apache.org>; Wed, 14 Apr 2010 19:32:50 -0400 (EDT)
Message-ID: <14933500.128021271287970555.JavaMail.jira@thor>
Date: Wed, 14 Apr 2010 19:32:50 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6705) jiracli fails to upload test-patch
 comments to jira
In-Reply-To: <5861257.127621271286648655.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857147#action_12857147 ] 

Giridharan Kesavan commented on HADOOP-6705:
--------------------------------------------

this patch cant be tested with test-patch. This is not going to change the current build system except for that it would be using a different version of jiracli.

> jiracli fails to upload test-patch comments to jira
> ---------------------------------------------------
>
>                 Key: HADOOP-6705
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6705
>             Project: Hadoop Common
>          Issue Type: Test
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>         Attachments: HADOOP-6705.PATCH
>
>
>      [exec] ======================================================================
>      [exec]     Adding comment to Jira.
>      [exec] ======================================================================
>      [exec] ======================================================================
>      [exec] 
>      [exec] 
>      [exec] Failed to connect to: http://issues.apache.org/jira/rpc/soap/jirasoapservice-v2?wsdl
>      [exec] Failed to connect to: http://issues.apache.org/jira/rpc/soap/jirasoapservice-v2?wsdl
>      [exec] Failed to connect to: http://issues.apache.org/jira/rpc/soap/jirasoapservice-v2?wsdl
>      [exec]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7396-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 00:34:25 2010
Return-Path: <common-issues-return-7396-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75437 invoked from network); 15 Apr 2010 00:34:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 00:34:25 -0000
Received: (qmail 52878 invoked by uid 500); 15 Apr 2010 00:34:25 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52844 invoked by uid 500); 15 Apr 2010 00:34:25 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52834 invoked by uid 99); 15 Apr 2010 00:34:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 00:34:25 +0000
X-ASF-Spam-Status: No, hits=-1290.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 00:34:24 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3F0Y3fh012913
	for <common-issues@hadoop.apache.org>; Wed, 14 Apr 2010 20:34:03 -0400 (EDT)
Message-ID: <19329091.128961271291643642.JavaMail.jira@thor>
Date: Wed, 14 Apr 2010 20:34:03 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Relogin behavior for RPC clients could be improved
--------------------------------------------------

                 Key: HADOOP-6706
                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
             Project: Hadoop Common
          Issue Type: Bug
          Components: security
    Affects Versions: 0.22.0
            Reporter: Devaraj Das
            Assignee: Devaraj Das
             Fix For: 0.22.0


Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7397-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 00:42:14 2010
Return-Path: <common-issues-return-7397-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79478 invoked from network); 15 Apr 2010 00:42:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 00:42:13 -0000
Received: (qmail 58797 invoked by uid 500); 15 Apr 2010 00:42:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58759 invoked by uid 500); 15 Apr 2010 00:42:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58751 invoked by uid 99); 15 Apr 2010 00:42:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 00:42:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 00:42:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3F0fn1J013085
	for <common-issues@hadoop.apache.org>; Wed, 14 Apr 2010 20:41:50 -0400 (EDT)
Message-ID: <23627497.129461271292109748.JavaMail.jira@thor>
Date: Wed, 14 Apr 2010 20:41:49 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6706:
--------------------------------

    Attachment: 6706.bp20.patch

Attaching a patch for the Y20 branch. "Exception" is caught and a relogin is attempted. We were debating whether to have relogin done outside the exception block before doing a saslConnect. But even in that case, we need to be able to handle exceptions since some other thread might be currently using the subject credentials that we are just about to destroy (in relogin) and that thread should retry the connection after a relogin. In any case the exception handling is required.

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706.bp20.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7398-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 02:04:16 2010
Return-Path: <common-issues-return-7398-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 826 invoked from network); 15 Apr 2010 02:04:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 02:04:15 -0000
Received: (qmail 7248 invoked by uid 500); 15 Apr 2010 02:04:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7219 invoked by uid 500); 15 Apr 2010 02:04:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7211 invoked by uid 99); 15 Apr 2010 02:04:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 02:04:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 02:04:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3F23pYB013994
	for <common-issues@hadoop.apache.org>; Wed, 14 Apr 2010 22:03:51 -0400 (EDT)
Message-ID: <31930877.131071271297031100.JavaMail.jira@thor>
Date: Wed, 14 Apr 2010 22:03:51 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6369) add Grid Engine support to HOD
In-Reply-To: <1501134231.1257939999839.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857179#action_12857179 ] 

Hadoop QA commented on HADOOP-6369:
-----------------------------------

+1 @author.  The patch does not contain any @author tags.
    
              +1 tests included.  The patch appears to include 3 new or modified tests.
    
              +1 javadoc.  The javadoc tool did not generate any warning messages.
    
              +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
    
          +1 findbugs.  The patch does not introduce any new Findbugs warnings.
  
         +1 release audit.  The applied patch does not increase the total number of release audit warnings.
   
              +1 core tests.  The patch passed core unit tests.
   
              +1 contrib tests.  The patch passed contrib unit tests.
    
          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/42/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/42/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/42/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/42/console

> add Grid Engine support to HOD
> ------------------------------
>
>                 Key: HADOOP-6369
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6369
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: contrib/hod
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: Simone Leo
>         Attachments: HADOOP-6369.patch, hod_ge.patch
>
>
> At [CRS4, Distributed Computing Group|http://dc.crs4.it], we developed and tested a simple patch that adds Grid Engine support to HOD. Since porting HOD to SGE is listed in the [Project Suggestions page|http://wiki.apache.org/hadoop/ProjectSuggestions], we decided to share the patch with the community. After patching, HOD passes all unit tests at our site, and we tested the SGE driver on a production cluster consisting of about 400 nodes. The patch should preserve Torque functionality, although we were not able to test this on production.
> *NOTE*: the patch is for the official hadoop-0.20.1 HOD release. It should be easy, however, to generate patches for other versions.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7399-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 03:22:12 2010
Return-Path: <common-issues-return-7399-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38000 invoked from network); 15 Apr 2010 03:22:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 03:22:12 -0000
Received: (qmail 64676 invoked by uid 500); 15 Apr 2010 03:22:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64556 invoked by uid 500); 15 Apr 2010 03:22:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64542 invoked by uid 99); 15 Apr 2010 03:22:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 03:22:11 +0000
X-ASF-Spam-Status: No, hits=-1290.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 03:22:09 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3F3LniE014481
	for <common-issues@hadoop.apache.org>; Wed, 14 Apr 2010 23:21:49 -0400 (EDT)
Message-ID: <1959997.131731271301709476.JavaMail.jira@thor>
Date: Wed, 14 Apr 2010 23:21:49 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6705) jiracli fails to upload test-patch
 comments to jira
In-Reply-To: <5861257.127621271286648655.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857184#action_12857184 ] 

Hudson commented on HADOOP-6705:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #220 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/220/])
    HADOOP-6705. Fix to work with 1.5 version of jiracli


> jiracli fails to upload test-patch comments to jira
> ---------------------------------------------------
>
>                 Key: HADOOP-6705
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6705
>             Project: Hadoop Common
>          Issue Type: Test
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>         Attachments: HADOOP-6705.PATCH
>
>
>      [exec] ======================================================================
>      [exec]     Adding comment to Jira.
>      [exec] ======================================================================
>      [exec] ======================================================================
>      [exec] 
>      [exec] 
>      [exec] Failed to connect to: http://issues.apache.org/jira/rpc/soap/jirasoapservice-v2?wsdl
>      [exec] Failed to connect to: http://issues.apache.org/jira/rpc/soap/jirasoapservice-v2?wsdl
>      [exec] Failed to connect to: http://issues.apache.org/jira/rpc/soap/jirasoapservice-v2?wsdl
>      [exec]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7400-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 03:46:18 2010
Return-Path: <common-issues-return-7400-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42957 invoked from network); 15 Apr 2010 03:46:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 03:46:18 -0000
Received: (qmail 74720 invoked by uid 500); 15 Apr 2010 03:46:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74618 invoked by uid 500); 15 Apr 2010 03:46:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74610 invoked by uid 99); 15 Apr 2010 03:46:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 03:46:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 03:46:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3F3jrH8014606
	for <common-issues@hadoop.apache.org>; Wed, 14 Apr 2010 23:45:53 -0400 (EDT)
Message-ID: <17485232.131921271303153426.JavaMail.jira@thor>
Date: Wed, 14 Apr 2010 23:45:53 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6369) add Grid Engine support to HOD
In-Reply-To: <1501134231.1257939999839.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857186#action_12857186 ] 

Hadoop QA commented on HADOOP-6369:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12424727/HADOOP-6369.patch
  against trunk revision 934273.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/43/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/43/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/43/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/43/console

This message is automatically generated.

> add Grid Engine support to HOD
> ------------------------------
>
>                 Key: HADOOP-6369
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6369
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: contrib/hod
>    Affects Versions: 0.20.1, 0.21.0, 0.22.0
>            Reporter: Simone Leo
>         Attachments: HADOOP-6369.patch, hod_ge.patch
>
>
> At [CRS4, Distributed Computing Group|http://dc.crs4.it], we developed and tested a simple patch that adds Grid Engine support to HOD. Since porting HOD to SGE is listed in the [Project Suggestions page|http://wiki.apache.org/hadoop/ProjectSuggestions], we decided to share the patch with the community. After patching, HOD passes all unit tests at our site, and we tested the SGE driver on a production cluster consisting of about 400 nodes. The patch should preserve Torque functionality, although we were not able to test this on production.
> *NOTE*: the patch is for the official hadoop-0.20.1 HOD release. It should be easy, however, to generate patches for other versions.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7401-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 06:49:12 2010
Return-Path: <common-issues-return-7401-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11256 invoked from network); 15 Apr 2010 06:49:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 06:49:12 -0000
Received: (qmail 13028 invoked by uid 500); 15 Apr 2010 06:49:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12914 invoked by uid 500); 15 Apr 2010 06:49:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12903 invoked by uid 99); 15 Apr 2010 06:49:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 06:49:11 +0000
X-ASF-Spam-Status: No, hits=-1291.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 06:49:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3F6mnQv016705
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 02:48:49 -0400 (EDT)
Message-ID: <6671890.134011271314129577.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 02:48:49 -0400 (EDT)
From: "Ruyue Ma (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6707) BlockDecompressorStream.resetState()
 has a bug, which often causes the block decompressing not to work.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

BlockDecompressorStream.resetState() has a bug, which often causes the block decompressing not to work.
-------------------------------------------------------------------------------------------------------

                 Key: HADOOP-6707
                 URL: https://issues.apache.org/jira/browse/HADOOP-6707
             Project: Hadoop Common
          Issue Type: Bug
          Components: io
            Reporter: Ruyue Ma
            Assignee: Ruyue Ma
             Fix For: 0.21.0


BlockDecompressorStream.resetState() has a bug.

The method of resetState() doesn't reinit the two member variables(int originalBlockSize; int noUncompressedBytes) of Class BlockDecompressorStream.  

This bug will cause the block decompressing not to work.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7402-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 15:51:16 2010
Return-Path: <common-issues-return-7402-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66646 invoked from network); 15 Apr 2010 15:51:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 15:51:16 -0000
Received: (qmail 18423 invoked by uid 500); 15 Apr 2010 15:51:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18259 invoked by uid 500); 15 Apr 2010 15:51:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18248 invoked by uid 99); 15 Apr 2010 15:51:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 15:51:14 +0000
X-ASF-Spam-Status: No, hits=-1294.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 15:51:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3FFoqZe022716
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 11:50:52 -0400 (EDT)
Message-ID: <25947545.142741271346652661.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 11:50:52 -0400 (EDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6704) add support for Parascale
 filesystem
In-Reply-To: <6991209.115951271258268758.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857361#action_12857361 ] 

Steve Loughran commented on HADOOP-6704:
----------------------------------------

# This is a new feature, so the patch should be against SVN_HEAD, not an older version of Hadoop. 
# The newer version of Hadoop is moving to some new FS APIs; you might want to consider working with them.
# HDFS-708 has discussed the issue of stress testing filesystems; this new filesystem back end could be a use case, if you are willing to participate.
# One issue with all third party filesystems is regression testing: they don't get enough of it. If there is any way to make this easier \-and that could include you running a local version of Hudson to grab SVN_HEAD of Hadoop and testing MR jobs over your filestore, then end users will be grateful. 
# Involvement in testing forthcoming releases is equally important, as is ongoing maintenance. It is really hard for an OSS project to test/maintain code that works with other peoples infrastructure, and motivation can be trouble too, so you have to be willing to stay involved -otherwise the code just gradually stops working. 



> add support for Parascale filesystem
> ------------------------------------
>
>                 Key: HADOOP-6704
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6704
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.20.2
>            Reporter: Neil Bliss
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Parascale has developed an org.apache.hadoop.fs implementation that allows users to use Hadoop on Parascale storage clusters.  We'd like to contribute this work to the community. Should this be placed under contrib, or integrated into the org.apache.hadoop.fs space?

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7403-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 16:37:16 2010
Return-Path: <common-issues-return-7403-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79616 invoked from network); 15 Apr 2010 16:37:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 16:37:16 -0000
Received: (qmail 95230 invoked by uid 500); 15 Apr 2010 16:37:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95170 invoked by uid 500); 15 Apr 2010 16:37:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95162 invoked by uid 99); 15 Apr 2010 16:37:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 16:37:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 16:37:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3FGapvG023323
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 12:36:51 -0400 (EDT)
Message-ID: <25695093.144231271349411628.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 12:36:51 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6705) jiracli fails to upload test-patch
 comments to jira
In-Reply-To: <5861257.127621271286648655.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857393#action_12857393 ] 

Hudson commented on HADOOP-6705:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk #287 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/287/])
    HADOOP-6705. Fix to work with 1.5 version of jiracli


> jiracli fails to upload test-patch comments to jira
> ---------------------------------------------------
>
>                 Key: HADOOP-6705
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6705
>             Project: Hadoop Common
>          Issue Type: Test
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>         Attachments: HADOOP-6705.PATCH
>
>
>      [exec] ======================================================================
>      [exec]     Adding comment to Jira.
>      [exec] ======================================================================
>      [exec] ======================================================================
>      [exec] 
>      [exec] 
>      [exec] Failed to connect to: http://issues.apache.org/jira/rpc/soap/jirasoapservice-v2?wsdl
>      [exec] Failed to connect to: http://issues.apache.org/jira/rpc/soap/jirasoapservice-v2?wsdl
>      [exec] Failed to connect to: http://issues.apache.org/jira/rpc/soap/jirasoapservice-v2?wsdl
>      [exec]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7404-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 21:59:13 2010
Return-Path: <common-issues-return-7404-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77322 invoked from network); 15 Apr 2010 21:59:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 21:59:13 -0000
Received: (qmail 51683 invoked by uid 500); 15 Apr 2010 21:59:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51633 invoked by uid 500); 15 Apr 2010 21:59:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51624 invoked by uid 99); 15 Apr 2010 21:59:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 21:59:12 +0000
X-ASF-Spam-Status: No, hits=-1296.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 21:59:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3FLwp1j027368
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 17:58:51 -0400 (EDT)
Message-ID: <1679892.152781271368731460.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 17:58:51 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6692) Add FileContext#listStatus that
 returns an iterator
In-Reply-To: <253397745.22171270762596744.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6692:
--------------------------------

    Summary: Add FileContext#listStatus that returns an iterator  (was: Propose change to FileContext#listStatus)

> Add FileContext#listStatus that returns an iterator
> ---------------------------------------------------
>
>                 Key: HADOOP-6692
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6692
>             Project: Hadoop Common
>          Issue Type: Sub-task
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listStatusItor.patch
>
>
> Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7405-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 21:59:15 2010
Return-Path: <common-issues-return-7405-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77377 invoked from network); 15 Apr 2010 21:59:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 21:59:15 -0000
Received: (qmail 51839 invoked by uid 500); 15 Apr 2010 21:59:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51780 invoked by uid 500); 15 Apr 2010 21:59:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51767 invoked by uid 99); 15 Apr 2010 21:59:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 21:59:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 21:59:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3FLwoUT027353
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 17:58:50 -0400 (EDT)
Message-ID: <31341801.152731271368730600.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 17:58:50 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6678) Remove FileContext#isFile,
 isDirectory and exists
In-Reply-To: <360625036.668801270253667713.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6678:
--------------------------------

    Summary: Remove FileContext#isFile, isDirectory and exists  (was: Propose some changes to FileContext)

> Remove FileContext#isFile, isDirectory and exists
> -------------------------------------------------
>
>                 Key: HADOOP-6678
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6678
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Hairong Kuang
>            Assignee: Eli Collins
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6678-1.patch
>
>
> # Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.
> # Remove methods isFile(Path), isDirectory(Path), and exists.
> All these methods are implemented by calling getFileStatus(Path).But most users are not aware of this. They would write code as below: 
> {code}
>   FileContext fc = ..;
>   if (fc.exists(path)) {
>     if (fc.isFile(path)) {
>      ...
>     } else {
>     ...
>     }
>   }
> {code}
> The above code adds unnecessary getFileInfo RPC to NameNode. In our production clusters, we often see that the number of getFileStatus calls is multiple times of the open calls. If we remove isFile, isDirectory, and exists from FileContext, users have to explicitly call getFileStatus first, it is more likely that they will write more efficient code as follow:
> {code}
>   FileContext fc = ...;
>   FileStatus fstatus = fc.getFileStatus(path);
>   if (fstatus.isFile() {
>     ...
>   } else {
>     ...
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7406-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 22:11:12 2010
Return-Path: <common-issues-return-7406-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35189 invoked from network); 15 Apr 2010 22:11:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 22:11:11 -0000
Received: (qmail 68357 invoked by uid 500); 15 Apr 2010 22:11:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68248 invoked by uid 500); 15 Apr 2010 22:11:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68227 invoked by uid 99); 15 Apr 2010 22:11:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 22:11:11 +0000
X-ASF-Spam-Status: No, hits=-1296.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 22:11:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3FMAn1k027483
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 18:10:50 -0400 (EDT)
Message-ID: <8355863.153081271369449816.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 18:10:49 -0400 (EDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6657) Common portion of MAPREDUCE-1545
In-Reply-To: <74469463.448561269390627214.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857566#action_12857566 ] 

Steve Loughran commented on HADOOP-6657:
----------------------------------------

I've just checked in the change, but JIRA isn't giving me permission to work on the issue, to mark it as fixed.

> Common portion of MAPREDUCE-1545
> --------------------------------
>
>                 Key: HADOOP-6657
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6657
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6657-trunk-v1.patch, hadoop-6657-trunk-v2.patch, hadoop-6657-trunk-v3.patch
>
>
> Common portion of the MAPREDUCE-1545. 

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7407-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 22:21:13 2010
Return-Path: <common-issues-return-7407-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80372 invoked from network); 15 Apr 2010 22:21:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 22:21:13 -0000
Received: (qmail 78626 invoked by uid 500); 15 Apr 2010 22:21:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78598 invoked by uid 500); 15 Apr 2010 22:21:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78590 invoked by uid 99); 15 Apr 2010 22:21:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 22:21:13 +0000
X-ASF-Spam-Status: No, hits=-1297.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 22:21:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3FMKqYQ027607
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 18:20:52 -0400 (EDT)
Message-ID: <23756292.153431271370052193.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 18:20:52 -0400 (EDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6708) New file format for very large
 records
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

New file format for very large records
--------------------------------------

                 Key: HADOOP-6708
                 URL: https://issues.apache.org/jira/browse/HADOOP-6708
             Project: Hadoop Common
          Issue Type: New Feature
          Components: io
            Reporter: Aaron Kimball
            Assignee: Aaron Kimball


A file format that handles multi-gigabyte records efficiently, with lazy disk access

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7408-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 22:23:21 2010
Return-Path: <common-issues-return-7408-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85219 invoked from network); 15 Apr 2010 22:23:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 22:23:20 -0000
Received: (qmail 81713 invoked by uid 500); 15 Apr 2010 22:23:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81666 invoked by uid 500); 15 Apr 2010 22:23:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81658 invoked by uid 99); 15 Apr 2010 22:23:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 22:23:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 22:23:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3FMMuD1027735
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 18:22:56 -0400 (EDT)
Message-ID: <13709801.153851271370176868.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 18:22:56 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6657) Common portion of MAPREDUCE-1545
In-Reply-To: <74469463.448561269390627214.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857576#action_12857576 ] 

Hudson commented on HADOOP-6657:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #221 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/221/])
    HADOOP-6657. Add a capitalization method to StringUtils for MAPREDUCE-1545


> Common portion of MAPREDUCE-1545
> --------------------------------
>
>                 Key: HADOOP-6657
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6657
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6657-trunk-v1.patch, hadoop-6657-trunk-v2.patch, hadoop-6657-trunk-v3.patch
>
>
> Common portion of the MAPREDUCE-1545. 

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7409-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 22:23:22 2010
Return-Path: <common-issues-return-7409-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85304 invoked from network); 15 Apr 2010 22:23:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 22:23:21 -0000
Received: (qmail 81944 invoked by uid 500); 15 Apr 2010 22:23:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81909 invoked by uid 500); 15 Apr 2010 22:23:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81901 invoked by uid 99); 15 Apr 2010 22:23:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 22:23:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 22:23:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3FMMwtc027774
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 18:22:58 -0400 (EDT)
Message-ID: <15840001.153981271370178769.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 18:22:58 -0400 (EDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6708) New file format for very large
 records
In-Reply-To: <23756292.153431271370052193.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron Kimball updated HADOOP-6708:
----------------------------------

    Attachment: lobfile.pdf

> New file format for very large records
> --------------------------------------
>
>                 Key: HADOOP-6708
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6708
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>         Attachments: lobfile.pdf
>
>
> A file format that handles multi-gigabyte records efficiently, with lazy disk access

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7410-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 22:23:22 2010
Return-Path: <common-issues-return-7410-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85331 invoked from network); 15 Apr 2010 22:23:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 22:23:21 -0000
Received: (qmail 82071 invoked by uid 500); 15 Apr 2010 22:23:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82043 invoked by uid 500); 15 Apr 2010 22:23:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81991 invoked by uid 99); 15 Apr 2010 22:23:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 22:23:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 22:23:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3FMMwBW027768
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 18:22:58 -0400 (EDT)
Message-ID: <27206252.153961271370178521.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 18:22:58 -0400 (EDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6708) New file format for very large
 records
In-Reply-To: <23756292.153431271370052193.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857578#action_12857578 ] 

Aaron Kimball commented on HADOOP-6708:
---------------------------------------

In working on Sqoop, I need to import records which may be several gigabytes in size each. I require a file format that allows me to store these records in an efficient, grouped fashion.

Users may then want to open a file containing many such records and partially-read individual records, but still access subsequent records efficiently. 

I'm attaching to this issue a proposal for a _LobFile_ format which will store these large objects. (The basis for this work surrounds import of BLOB and CLOB-typed columns.) The specification proposal analyzes available file formats and my understanding of why they aren't appropriate here.

> New file format for very large records
> --------------------------------------
>
>                 Key: HADOOP-6708
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6708
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>
> A file format that handles multi-gigabyte records efficiently, with lazy disk access

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7411-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 22:25:14 2010
Return-Path: <common-issues-return-7411-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89545 invoked from network); 15 Apr 2010 22:25:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 22:25:14 -0000
Received: (qmail 86416 invoked by uid 500); 15 Apr 2010 22:25:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86372 invoked by uid 500); 15 Apr 2010 22:25:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86364 invoked by uid 99); 15 Apr 2010 22:25:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 22:25:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 22:25:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3FMOoRM027800
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 18:24:50 -0400 (EDT)
Message-ID: <19178489.154041271370290333.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 18:24:50 -0400 (EDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6708) New file format for very large
 records
In-Reply-To: <23756292.153431271370052193.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857580#action_12857580 ] 

Aaron Kimball commented on HADOOP-6708:
---------------------------------------

Please review the attached proposal. This will serve as a basis to foment discussion; hopefully we can arrive at a conclusion as to how to best implement this need. Some still-open points are listed at the end of the document.


> New file format for very large records
> --------------------------------------
>
>                 Key: HADOOP-6708
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6708
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>         Attachments: lobfile.pdf
>
>
> A file format that handles multi-gigabyte records efficiently, with lazy disk access

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7412-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 23:11:15 2010
Return-Path: <common-issues-return-7412-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77994 invoked from network); 15 Apr 2010 23:11:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 23:11:14 -0000
Received: (qmail 26854 invoked by uid 500); 15 Apr 2010 23:11:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26773 invoked by uid 500); 15 Apr 2010 23:11:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26765 invoked by uid 99); 15 Apr 2010 23:11:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 23:11:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 23:11:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3FNAoha028047
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 19:10:51 -0400 (EDT)
Message-ID: <9122855.154341271373050801.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 19:10:50 -0400 (EDT)
From: "Gaurav Jain (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4565) MultiFileInputSplit can use data
 locality information to create splits
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857586#action_12857586 ] 

Gaurav Jain commented on HADOOP-4565:
-------------------------------------


Did you happen to hit the following issue after this jiras changes?

https://issues.apache.org/jira/browse/HDFS-347

Looks like I am hitting this issue. I am running some more benchmarks to trace down the details.

> MultiFileInputSplit can use data locality information to create splits
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-4565
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4565
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>             Fix For: 0.20.0
>
>         Attachments: CombineMultiFile.patch, CombineMultiFile2.patch, CombineMultiFile3.patch, CombineMultiFile4.patch, CombineMultiFile5.patch, CombineMultiFile7.patch, CombineMultiFile8.patch, CombineMultiFile9.patch, TestCombine.txt
>
>
> The MultiFileInputFormat takes a set of paths and creates splits based on file sizes. Each splits contains a few files an each split are roughly equal in size. It would be efficient if we can extend this InputFormat to create splits such each all the blocks in one split and either node-local or rack-local.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7413-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 23:17:13 2010
Return-Path: <common-issues-return-7413-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80022 invoked from network); 15 Apr 2010 23:17:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 23:17:13 -0000
Received: (qmail 31457 invoked by uid 500); 15 Apr 2010 23:17:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31426 invoked by uid 500); 15 Apr 2010 23:17:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31418 invoked by uid 99); 15 Apr 2010 23:17:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 23:17:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 23:17:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3FNGnnd028098
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 19:16:49 -0400 (EDT)
Message-ID: <28677223.154481271373409459.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 19:16:49 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6708) New file format for very large
 records
In-Reply-To: <23756292.153431271370052193.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6708?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D128=
57588#action_12857588 ]=20

Hong Tang commented on HADOOP-6708:
-----------------------------------

bq. Shortcomings of TFile:=20

bq. =E2=80=A2 Length =EF=AC=81elds are encoded as integers, not longs. This=
 does not support records > 2 GB.=20

This is an intentional restriction. All integers are in VInt/VLong format w=
hich is fully wire compatible. You can easily make a case to request such l=
imit be lifted.

bq. =E2=80=A2 The expected length of the value must be precisely-known or w=
ritten as -1. I have records where I have a minimum bound on their size (mo=
re precisely: Ahead of time, I know their length in characters but not in b=
ytes), but not necessarily a more precise byte count value. A goal of this =
format is the ability to ef=EF=AC=81ciently partially-read a character stre=
am-oriented record, then skip quickly to the next record. The format I prop=
ose will allow you to specify that a record is at least a certain length. T=
hat way you can ef=EF=AC=81ciently skip most of a record and then search fo=
r a synchronization boundary after that. (Since I expect most characters to=
 be encoded in a single byte, but not necessarily all of them.)=20

Even if you do not know the length of the record you write (namely specifyi=
ng -1 during writing), you can still efficiently skip a record (even after =
partially consuming some bytes of the record). Isn't it sufficient for your=
 case? Searching for a synchronization boundary is very inefficient than le=
ngth-prefixed encoding.


> New file format for very large records
> --------------------------------------
>
>                 Key: HADOOP-6708
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6708
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>         Attachments: lobfile.pdf
>
>
> A file format that handles multi-gigabyte records efficiently, with lazy =
disk access

--=20
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: htt=
ps://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

From common-issues-return-7414-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 23:41:11 2010
Return-Path: <common-issues-return-7414-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85406 invoked from network); 15 Apr 2010 23:41:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 23:41:11 -0000
Received: (qmail 60151 invoked by uid 500); 15 Apr 2010 23:41:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60109 invoked by uid 500); 15 Apr 2010 23:41:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60101 invoked by uid 99); 15 Apr 2010 23:41:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 23:41:11 +0000
X-ASF-Spam-Status: No, hits=-1297.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 23:41:09 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3FNenMu028259
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 19:40:49 -0400 (EDT)
Message-ID: <11903376.154751271374849451.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 19:40:49 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6709) Re-instate deprecated FileSystem
 methods that were removed after 0.20
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Re-instate deprecated FileSystem methods that were removed after 0.20
---------------------------------------------------------------------

                 Key: HADOOP-6709
                 URL: https://issues.apache.org/jira/browse/HADOOP-6709
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs
            Reporter: Tom White
            Assignee: Tom White
            Priority: Blocker
             Fix For: 0.21.0


To make the FileSystem API in the 0.21 release (which should be treated as a minor release) compatible with 0.20 the deprecated methods removed in HADOOP-4779 need to be re-instated. They should still be marked as deprecated, however.


-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7415-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 23:57:11 2010
Return-Path: <common-issues-return-7415-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89019 invoked from network); 15 Apr 2010 23:57:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 23:57:11 -0000
Received: (qmail 70612 invoked by uid 500); 15 Apr 2010 23:57:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70583 invoked by uid 500); 15 Apr 2010 23:57:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70575 invoked by uid 99); 15 Apr 2010 23:57:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 23:57:10 +0000
X-ASF-Spam-Status: No, hits=-1297.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 23:57:09 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3FNunS9028373
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 19:56:49 -0400 (EDT)
Message-ID: <8947165.155011271375809341.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 19:56:49 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6710) Symbolic umask for file creation is
 not consistent with linux
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Symbolic umask for file creation is not consistent with linux
-------------------------------------------------------------

                 Key: HADOOP-6710
                 URL: https://issues.apache.org/jira/browse/HADOOP-6710
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Suresh Srinivas


Currently both octal and symbolic umask are used to reset the file creation mode bits. This is not consistent with the behavior defined in posix. Making it consistent would avoid confusion to the HDFS users.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7416-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 23:59:13 2010
Return-Path: <common-issues-return-7416-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89631 invoked from network); 15 Apr 2010 23:59:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 23:59:13 -0000
Received: (qmail 74916 invoked by uid 500); 15 Apr 2010 23:59:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74876 invoked by uid 500); 15 Apr 2010 23:59:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74868 invoked by uid 99); 15 Apr 2010 23:59:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 23:59:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 23:59:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3FNwn8g028393
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 19:58:49 -0400 (EDT)
Message-ID: <30232174.155051271375929165.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 19:58:49 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6710) Symbolic umask for file creation is
 not consistent with linux
In-Reply-To: <8947165.155011271375809341.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857600#action_12857600 ] 

Suresh Srinivas commented on HADOOP-6710:
-----------------------------------------

Snippet from the posix specification http://www.opengroup.org/onlinepubs/000095399/utilities/umask.html:
For a symbolic_mode value, the new value of the file mode creation mask shall be the logical complement of the file permission bits portion of the file mode specified by the symbolic_mode  string.

In a symbolic_mode value, the permissions op characters '+' and '-' shall be interpreted relative to the current file mode creation mask; '+' shall cause the bits for the indicated permissions to be cleared in the mask; '-' shall cause the bits for the indicated permissions to be set in the mask.

The interpretation of mode values that specify file mode bits other than the file permission bits is unspecified.

In the octal integer form of mode, the specified bits are set in the file mode creation mask.

The file mode creation mask shall be set to the resulting numeric value.

> Symbolic umask for file creation is not consistent with linux
> -------------------------------------------------------------
>
>                 Key: HADOOP-6710
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6710
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Suresh Srinivas
>
> Currently both octal and symbolic umask are used to reset the file creation mode bits. This is not consistent with the behavior defined in posix. Making it consistent would avoid confusion to the HDFS users.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7417-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 15 23:59:15 2010
Return-Path: <common-issues-return-7417-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89676 invoked from network); 15 Apr 2010 23:59:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Apr 2010 23:59:15 -0000
Received: (qmail 75046 invoked by uid 500); 15 Apr 2010 23:59:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75011 invoked by uid 500); 15 Apr 2010 23:59:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75003 invoked by uid 99); 15 Apr 2010 23:59:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 23:59:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Apr 2010 23:59:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3FNwpNH028432
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 19:58:51 -0400 (EDT)
Message-ID: <20779928.155181271375931795.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 19:58:51 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6710) Symbolic umask for file creation is
 not consistent with posix
In-Reply-To: <8947165.155011271375809341.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6710:
------------------------------------

              Summary: Symbolic umask for file creation is not consistent with posix  (was: Symbolic umask for file creation is not consistent with linux)
    Affects Version/s: 0.21.0
                       0.22.0

> Symbolic umask for file creation is not consistent with posix
> -------------------------------------------------------------
>
>                 Key: HADOOP-6710
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6710
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Suresh Srinivas
>
> Currently both octal and symbolic umask are used to reset the file creation mode bits. This is not consistent with the behavior defined in posix. Making it consistent would avoid confusion to the HDFS users.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7418-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 00:11:12 2010
Return-Path: <common-issues-return-7418-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93397 invoked from network); 16 Apr 2010 00:11:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 00:11:12 -0000
Received: (qmail 82142 invoked by uid 500); 16 Apr 2010 00:11:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82118 invoked by uid 500); 16 Apr 2010 00:11:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82076 invoked by uid 99); 16 Apr 2010 00:11:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 00:11:12 +0000
X-ASF-Spam-Status: No, hits=-1297.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 00:11:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3G0AoXd028556
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 20:10:51 -0400 (EDT)
Message-ID: <32032177.155531271376650748.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 20:10:50 -0400 (EDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6708) New file format for very large
 records
In-Reply-To: <23756292.153431271370052193.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6708?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D128=
57609#action_12857609 ]=20

Aaron Kimball commented on HADOOP-6708:
---------------------------------------

Hong,

bq. * Length =EF=AC=81elds are encoded as integers, not longs. This does no=
t support records > 2 GB.
bq. This is an intentional restriction. All integers are in VInt/VLong form=
at which is fully wire compatible. You can easily make a case to request su=
ch limit be lifted.

So does this mean that the API for TFile could be changed without complicat=
ion to accept/return {{long}} values? I read the TFile spec and it points o=
ut in several different locations the 2 GB value limit. By reading that, it=
 sounds as though other aspects of TFile may break based on the assumed int=
eger size there.

bq. Even if you do not know the length of the record you write (namely spec=
ifying -1 during writing), you can still efficiently skip a record (even af=
ter partially consuming some bytes of the record). Isn't it sufficient for =
your case? Searching for a synchronization boundary is very inefficient tha=
n length-prefixed encoding.

Data comes to me from JDBC through an InputStream or a Reader that I am not=
 sure how long it is. I read from that InputStream/Reader and write its con=
tents into an OutputStream/Writer that dumps into a file (LobFile). In the =
case where I have a character-based Reader, I know how many characters I ha=
ve, which is a lower bound on the number of bytes, but not exact.  So my pl=
an was to seek ahead by that much, then search for the boundary. Assuming m=
ost characters are one byte, the search will be pretty quick.

How does TFile support length skipping if you don't pre-declare the lengths=
?


> New file format for very large records
> --------------------------------------
>
>                 Key: HADOOP-6708
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6708
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>         Attachments: lobfile.pdf
>
>
> A file format that handles multi-gigabyte records efficiently, with lazy =
disk access

--=20
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: htt=
ps://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

From common-issues-return-7419-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 00:27:13 2010
Return-Path: <common-issues-return-7419-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4779 invoked from network); 16 Apr 2010 00:27:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 00:27:13 -0000
Received: (qmail 95375 invoked by uid 500); 16 Apr 2010 00:27:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95341 invoked by uid 500); 16 Apr 2010 00:27:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95333 invoked by uid 99); 16 Apr 2010 00:27:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 00:27:12 +0000
X-ASF-Spam-Status: No, hits=-1297.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 00:27:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3G0Qpov028877
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 20:26:51 -0400 (EDT)
Message-ID: <26425966.155701271377611333.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 20:26:51 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6521) FsPermission:SetUMask not updated to
 use new-style umask setting.
In-Reply-To: <797058971.135641264793314607.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6521:
------------------------------------

    Attachment: hadoop-6521.patch

Changes in the patch:
# Symbolic and octal umask mode was added configurable with param name "dfs.umaskmode". It deprecates the old param "dfs.umask". Previously for backward compatibility new deprecation mechanism introduced in 0.21 was used. This mechanism is useful when a config param name changes, to map the old config name to the new. However in case of umask, the semantics of the param also changes. Old param expects decimal value and the new param  expects symbolic or octal value. The deprecation mechanism in addition to mapping old to new names, must also translated the old value to new value. Given the lack of this capability, umask will no longer use config deprecation mechanism.

# Following changes from 0.20 version of the patch is carried forward, with the exception of test changes. Test changes are needed in HDFS. I will created a separate jira for that.
{quote}
    1.  To ensure backward compatibility, when applications use the old key "dfs.umask" and the server default has "dfs.umaskmode", old key overrides the new key. That is first the old key is used for getting umask and then the new key.
   2. FsPermission.setUMask(Configuration conf, FsPermission umask) currently has a bug. When setting the "dfs.umaskmode" param in the configuration, it should convert the umask decimal value to octal.
{quote}


> FsPermission:SetUMask not updated to use new-style umask setting.
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6521
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6521
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Jakob Homan
>            Assignee: Suresh Srinivas
>         Attachments: hadoop-6521.patch, hadoop-6521.rel20.1.patch, hadoop-6521.rel20.patch, hadoop-6521.rel20.patch, hadoop-6521.rel20.patch
>
>
> FsPermission:
> {code}
> 221   /** Set the user file creation mask (umask) */
> 222   public static void setUMask(Configuration conf, FsPermission umask) {                                    
> 223     conf.setInt(UMASK_LABEL, umask.toShort());
> 224   }
> {code}
> Needs to be updated to not use a decimal value. This is a bug introduced by HADOOP-6234.  

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7420-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 00:33:51 2010
Return-Path: <common-issues-return-7420-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8778 invoked from network); 16 Apr 2010 00:33:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 00:33:50 -0000
Received: (qmail 99770 invoked by uid 500); 16 Apr 2010 00:33:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99741 invoked by uid 500); 16 Apr 2010 00:33:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99733 invoked by uid 99); 16 Apr 2010 00:33:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 00:33:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 00:33:48 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3G0XQZd000426
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 20:33:26 -0400 (EDT)
Message-ID: <11650152.81271378006338.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 20:33:26 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6708) New file format for very large
 records
In-Reply-To: <23756292.153431271370052193.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857620#action_12857620 ] 

Hong Tang commented on HADOOP-6708:
-----------------------------------

bq. So does this mean that the API for TFile could be changed without complication to accept/return long values?

yes, if by "complication" you mean wire compatibility. You may also need to remove the checks for length violations in various places. Note that there is still a key length restriction of 64KB.

bq. How does TFile support length skipping if you don't pre-declare the lengths?

it uses chunk encoding. The whole value stream is encoded with a chain of chunks. We use an internal buffer to accumulate small writes and once full, flush out the buffer as one chunk. Each chunk is length prefixed. All but the terminal chunk have their lengths written out as negative integers. The chunk size is controlled by parameter "tfile.io.chunk.size" (default to 1MB). When skipping, it skips chunk by chunk.


> New file format for very large records
> --------------------------------------
>
>                 Key: HADOOP-6708
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6708
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>         Attachments: lobfile.pdf
>
>
> A file format that handles multi-gigabyte records efficiently, with lazy disk access

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7421-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 00:39:51 2010
Return-Path: <common-issues-return-7421-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11684 invoked from network); 16 Apr 2010 00:39:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 00:39:51 -0000
Received: (qmail 2998 invoked by uid 500); 16 Apr 2010 00:39:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2968 invoked by uid 500); 16 Apr 2010 00:39:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2959 invoked by uid 99); 16 Apr 2010 00:39:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 00:39:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 00:39:48 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3G0dRb8008644
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 20:39:27 -0400 (EDT)
Message-ID: <23833185.161271378367151.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 20:39:27 -0400 (EDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6708) New file format for very large
 records
In-Reply-To: <23756292.153431271370052193.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857621#action_12857621 ] 

Aaron Kimball commented on HADOOP-6708:
---------------------------------------

I don't think that works in this scenario. Suppose I have a record that is 8 GB long; I read the first kilobyte or two out of the record, then intend to discard the rest and start with the next record.

If we have a chunk size of 1 MB, then skipping through this will require 8000 seeks, or ~64000 milliseconds. Even upping this chunk size drastically to 100 MB means a 640 ms delay.


> New file format for very large records
> --------------------------------------
>
>                 Key: HADOOP-6708
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6708
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>         Attachments: lobfile.pdf
>
>
> A file format that handles multi-gigabyte records efficiently, with lazy disk access

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7422-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 00:43:50 2010
Return-Path: <common-issues-return-7422-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12207 invoked from network); 16 Apr 2010 00:43:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 00:43:49 -0000
Received: (qmail 5196 invoked by uid 500); 16 Apr 2010 00:43:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5154 invoked by uid 500); 16 Apr 2010 00:43:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5146 invoked by uid 99); 16 Apr 2010 00:43:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 00:43:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 00:43:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3G0hPqH012503
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 20:43:25 -0400 (EDT)
Message-ID: <20866272.181271378605113.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 20:43:25 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6709) Re-instate deprecated FileSystem
 methods that were removed after 0.20
In-Reply-To: <11903376.154751271374849451.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6709:
------------------------------

    Attachment: HADOOP-6709.patch

Patch to re-instate the removed methods. No extra tests.

> Re-instate deprecated FileSystem methods that were removed after 0.20
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6709
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6709
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6709.patch
>
>
> To make the FileSystem API in the 0.21 release (which should be treated as a minor release) compatible with 0.20 the deprecated methods removed in HADOOP-4779 need to be re-instated. They should still be marked as deprecated, however.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7423-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 00:53:48 2010
Return-Path: <common-issues-return-7423-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15356 invoked from network); 16 Apr 2010 00:53:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 00:53:48 -0000
Received: (qmail 10420 invoked by uid 500); 16 Apr 2010 00:53:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10368 invoked by uid 500); 16 Apr 2010 00:53:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10360 invoked by uid 99); 16 Apr 2010 00:53:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 00:53:48 +0000
X-ASF-Spam-Status: No, hits=-1297.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 00:53:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3G0rRjB025584
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 20:53:27 -0400 (EDT)
Message-ID: <26218954.391271379207293.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 20:53:27 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6708) New file format for very large
 records
In-Reply-To: <23756292.153431271370052193.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857627#action_12857627 ] 

Hong Tang commented on HADOOP-6708:
-----------------------------------

bq. I don't think that works in this scenario. Suppose I have a record that is 8 GB long; I read the first kilobyte or two out of the record, then intend to discard the rest and start with the next record.

Your analysis is almost right. However, there is one optimization could be done to support this: TFile does block compression, and an 8GB record is likely to exceed the TFIle block size after compression (unless you have something like all zeros). So it would be the last record in the block. And we can speed up the skipping of the last record in a block by positioning the cursor to the beginning of the next block without chunk decoding. On the other hand, if your 8GB actually compresses very well within one TFile block (256KB by default), then it is really a sequential read of 256KB from HDFS.

>From your description, it seems that you do not plan to use compression, which sounds a bit surprising to me...



> New file format for very large records
> --------------------------------------
>
>                 Key: HADOOP-6708
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6708
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>         Attachments: lobfile.pdf
>
>
> A file format that handles multi-gigabyte records efficiently, with lazy disk access

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7424-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 01:21:50 2010
Return-Path: <common-issues-return-7424-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22066 invoked from network); 16 Apr 2010 01:21:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 01:21:50 -0000
Received: (qmail 32806 invoked by uid 500); 16 Apr 2010 01:21:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32756 invoked by uid 500); 16 Apr 2010 01:21:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32748 invoked by uid 99); 16 Apr 2010 01:21:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 01:21:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 01:21:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3G1LPCP000826
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 21:21:26 -0400 (EDT)
Message-ID: <11754477.861271380885955.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 21:21:25 -0400 (EDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6708) New file format for very large
 records
In-Reply-To: <23756292.153431271370052193.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857635#action_12857635 ] 

Aaron Kimball commented on HADOOP-6708:
---------------------------------------

I'm not sure what you mean by this optimization. Can you please explain further?

What's the relationship between "blocks" and "chunks" in a TFile? It sounds like a record can span multiple chunks. Is a record fully contained in a block? If it compresses an 8 GB record down to, say, 2 GB, will that still require skipping chunk-wise through the compressed data?

I do plan on using compression. Given the very large record lengths I'm designing for, I expect that it's acceptable to compress each record individually. The current writeup doesn't propose how to handle compression elegantly. But I'm leaning toward writing out a table of compressed record lengths at the end of the file.

> New file format for very large records
> --------------------------------------
>
>                 Key: HADOOP-6708
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6708
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>         Attachments: lobfile.pdf
>
>
> A file format that handles multi-gigabyte records efficiently, with lazy disk access

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7425-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 01:23:50 2010
Return-Path: <common-issues-return-7425-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22337 invoked from network); 16 Apr 2010 01:23:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 01:23:50 -0000
Received: (qmail 33440 invoked by uid 500); 16 Apr 2010 01:23:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33408 invoked by uid 500); 16 Apr 2010 01:23:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33400 invoked by uid 99); 16 Apr 2010 01:23:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 01:23:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 01:23:48 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3G1NQuG000864
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 21:23:26 -0400 (EDT)
Message-ID: <5738194.961271381006602.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 21:23:26 -0400 (EDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6708) New file format for very large
 records
In-Reply-To: <23756292.153431271370052193.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857637#action_12857637 ] 

Aaron Kimball commented on HADOOP-6708:
---------------------------------------

Also how does TFile handle splits and resynchronizing? It doesn't seem like there's an InputFormat for it.


> New file format for very large records
> --------------------------------------
>
>                 Key: HADOOP-6708
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6708
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>         Attachments: lobfile.pdf
>
>
> A file format that handles multi-gigabyte records efficiently, with lazy disk access

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7426-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 01:37:48 2010
Return-Path: <common-issues-return-7426-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23332 invoked from network); 16 Apr 2010 01:37:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 01:37:48 -0000
Received: (qmail 36052 invoked by uid 500); 16 Apr 2010 01:37:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36017 invoked by uid 500); 16 Apr 2010 01:37:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36007 invoked by uid 99); 16 Apr 2010 01:37:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 01:37:47 +0000
X-ASF-Spam-Status: No, hits=-1297.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 01:37:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3G1bPMt000933
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 21:37:26 -0400 (EDT)
Message-ID: <20841992.1101271381845967.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 21:37:25 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6708) New file format for very large
 records
In-Reply-To: <23756292.153431271370052193.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857639#action_12857639 ] 

Hong Tang commented on HADOOP-6708:
-----------------------------------

bq. What's the relationship between "blocks" and "chunks" in a TFile?
A TFile contains zero or more compressed blocks. Each block contains sequences of key, value, key, value. Each value can contain 1 to more chunks. A block has a minimum size of 256KB. Whenever we accumulate enough data that exceeds the minimum block size, we "close" the current block and starts a new block. All blocks have their offsets and lengths recorded in some index section.

bq. Is a record fully contained in a block?
Yes.

bq.  If it compresses an 8 GB record down to, say, 2 GB, will that still require skipping chunk-wise through the compressed data?
No, because it would be the last record in that block. With my suggested optimization, it would be an O(1) operation to skip that record.

bq. Also how does TFile handle splits and resynchronizing? It doesn't seem like there's an InputFormat for it. 
Writing an input format for it is pretty easy, I believe Owen has a prototype of OFile on top of TFile on his laptop. :) Generally, you would extend from FileInputFormat, and your record reader would be backed up by a TFile.Reader.Scanner created by TFile.Reader.createScannerByByteRange(long offset, long length). Internally, this method would move the bytes range to the boundary of TFile compression blocks (through the block index it maintains). 

> New file format for very large records
> --------------------------------------
>
>                 Key: HADOOP-6708
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6708
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>         Attachments: lobfile.pdf
>
>
> A file format that handles multi-gigabyte records efficiently, with lazy disk access

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7427-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 01:57:51 2010
Return-Path: <common-issues-return-7427-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26964 invoked from network); 16 Apr 2010 01:57:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 01:57:51 -0000
Received: (qmail 49361 invoked by uid 500); 16 Apr 2010 01:57:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49249 invoked by uid 500); 16 Apr 2010 01:57:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49240 invoked by uid 99); 16 Apr 2010 01:57:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 01:57:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 01:57:48 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3G1vQ0C001083
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 21:57:27 -0400 (EDT)
Message-ID: <25374898.1431271383046934.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 21:57:26 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6668:
------------------------------

    Attachment: HADOOP-6668.patch

New patch which makes classes in io.compress Evolving (not Stable) since several classes made incompatible changes to constructors between 0.20 and trunk (to be 0.21) by throwing IOException (in 0.20 they don't throw an exception). Similarly util.GenericOptionsParser is now Evolving since its constructors now throw IOException.

I've marked the security classes as Evolving too. It's likely some could be marked as private or limited-private, but I didn't know which ones.

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7428-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 01:57:52 2010
Return-Path: <common-issues-return-7428-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26994 invoked from network); 16 Apr 2010 01:57:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 01:57:51 -0000
Received: (qmail 49488 invoked by uid 500); 16 Apr 2010 01:57:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49388 invoked by uid 500); 16 Apr 2010 01:57:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49380 invoked by uid 99); 16 Apr 2010 01:57:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 01:57:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 01:57:49 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3G1vRBR001104
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 21:57:27 -0400 (EDT)
Message-ID: <11753629.1501271383047957.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 21:57:27 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6668:
------------------------------

    Status: Open  (was: Patch Available)

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7429-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 01:57:53 2010
Return-Path: <common-issues-return-7429-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27074 invoked from network); 16 Apr 2010 01:57:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 01:57:53 -0000
Received: (qmail 49564 invoked by uid 500); 16 Apr 2010 01:57:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49529 invoked by uid 500); 16 Apr 2010 01:57:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49517 invoked by uid 99); 16 Apr 2010 01:57:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 01:57:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 01:57:49 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3G1vSCA001125
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 21:57:28 -0400 (EDT)
Message-ID: <20540659.1571271383048605.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 21:57:28 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6668:
------------------------------

    Status: Patch Available  (was: Open)

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7430-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 02:15:52 2010
Return-Path: <common-issues-return-7430-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31110 invoked from network); 16 Apr 2010 02:15:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 02:15:52 -0000
Received: (qmail 67647 invoked by uid 500); 16 Apr 2010 02:15:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67626 invoked by uid 500); 16 Apr 2010 02:15:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67618 invoked by uid 99); 16 Apr 2010 02:15:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 02:15:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 02:15:49 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3G2FPLT001249
	for <common-issues@hadoop.apache.org>; Thu, 15 Apr 2010 22:15:28 -0400 (EDT)
Message-ID: <5668055.1791271384125882.JavaMail.jira@thor>
Date: Thu, 15 Apr 2010 22:15:25 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857651#action_12857651 ] 

Hadoop QA commented on HADOOP-6668:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12441908/HADOOP-6668.patch
  against trunk revision 934619.

    +1 @author.  The patch does not contain any @author tags.

    +0 tests included.  The patch appears to be a documentation patch that doesn't require tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/463/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/463/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/463/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/463/console

This message is automatically generated.

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7431-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 06:33:48 2010
Return-Path: <common-issues-return-7431-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18735 invoked from network); 16 Apr 2010 06:33:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 06:33:48 -0000
Received: (qmail 8328 invoked by uid 500); 16 Apr 2010 06:33:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8200 invoked by uid 500); 16 Apr 2010 06:33:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8191 invoked by uid 99); 16 Apr 2010 06:33:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 06:33:46 +0000
X-ASF-Spam-Status: No, hits=-1298.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 06:33:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3G6XOJa003054
	for <common-issues@hadoop.apache.org>; Fri, 16 Apr 2010 02:33:25 -0400 (EDT)
Message-ID: <26101198.3701271399604546.JavaMail.jira@thor>
Date: Fri, 16 Apr 2010 02:33:24 -0400 (EDT)
From: "Amar Kamat (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6711) Configuration should support list of
 values
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Configuration should support list of values
-------------------------------------------

                 Key: HADOOP-6711
                 URL: https://issues.apache.org/jira/browse/HADOOP-6711
             Project: Hadoop Common
          Issue Type: New Feature
          Components: conf
            Reporter: Amar Kamat


Configuration supports 2 operations namely _set()_ and _get()_. It would be nice to have an inbuild support for lists where there can be multiple values (i.e list of values) assigned to one key. A workaround could be
{code}
// Assume Key be the parameter key and newValue be the value to be added/appended
Configuration c = new Configuration();
String value = c.get(Key);
value = value + " " + newValue
c.set(Key, value);
{code}

One common usecase is that in a production enviroment, some user facing params (e.g mapred.child.java.opts) are set to default values (say for performance reasons). Users themselves might want to *add* to this list. Doing a set would overwrite the previous values. The above mentioned hack is doable via code but not via command line. Hence we need the framework to support lists.


-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7432-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 15:51:09 2010
Return-Path: <common-issues-return-7432-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6673 invoked from network); 16 Apr 2010 15:51:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 15:51:08 -0000
Received: (qmail 44361 invoked by uid 500); 16 Apr 2010 15:51:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44324 invoked by uid 500); 16 Apr 2010 15:51:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44315 invoked by uid 99); 16 Apr 2010 15:51:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 15:51:07 +0000
X-ASF-Spam-Status: No, hits=-1300.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 15:51:06 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3GFokno004966
	for <common-issues@hadoop.apache.org>; Fri, 16 Apr 2010 11:50:46 -0400 (EDT)
Message-ID: <28312042.12461271433046220.JavaMail.jira@thor>
Date: Fri, 16 Apr 2010 11:50:46 -0400 (EDT)
From: "Koji Noguchi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6702) Incorrect exit codes for "dfs
 -chown", "dfs -chgrp"  when input is given in wildcard format.
In-Reply-To: <30954625.73221271193001806.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857862#action_12857862 ] 

Koji Noguchi commented on HADOOP-6702:
--------------------------------------

Incorrect return code for wildcard in hadoop is not limited to chown/chgrp.  
It's everywhere.
For example, 
In 'ls', this is how unix performs, 

{noformat}
% ls nonexist*
ls: No match.
% echo $?
1
% ls nonexist* file*
fileA
% echo $?
0
% ls file* nonexist* 
fileA
% echo $?
0
{noformat} 
*It returns 0 as long as one of the globing matches*.
and in hadoop 'ls'
{noformat}

% hadoop dfs -ls file\* nonexist\*
Found 1 items
-rw-------   3 knoguchi users          7 2010-04-08 15:57 /user/knoguchi/fileA
ls: Cannot access nonexist*: No such file or directory.
% echo $?
255

% hadoop dfs -ls nonexist\* file\* 
ls: Cannot access nonexist*: No such file or directory.
Found 1 items
-rw-------   3 knoguchi users          7 2010-04-08 15:57 /user/knoguchi/fileA
% echo $?
0
% 
{noformat}

hadoop 'ls' simply returns the result of the last globbing.

This behavior is also inconsistent from command to command. 
Picking three hadoop commands. chgrp/ls/du. 

|| command || single globbing  || multiple globbing ||
| chown/chgrp/etc | X (returns 0 even if globing returns empty) | X (returns 0 even if all the globbing returns empty) |
| ls | O | X (returns last globbing result) | 
| du | O | X (returns non-zero even if one of the globbing fail) |  

Suggested fix in this Jira would simply change the behavior of 'chown/chgrp'  to 'du'-like which means multiple globbing will still be incorrect.


> Incorrect exit codes for "dfs -chown", "dfs -chgrp"  when input is given in wildcard format.
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6702
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6702
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.19.1, 0.20.0, 0.20.1, 0.20.2
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>            Priority: Minor
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>
> Currently incorrect exit codes  are given for "dfs -chown", "dfs -chgrp"  when input is given in wildcard format.
> This bug is due to missing update of errors count in {{FsShell.java}}.
> {code:title=FsShell.java|borderStyle=solid}
> int runCmdHandler(CmdHandler handler, String[] args,
>                                    int startIndex, boolean recursive) 
>                                    throws IOException {
>     int errors = 0;
>     
>     for (int i=startIndex; i<args.length; i++) {
>       Path srcPath = new Path(args[i]);
>       FileSystem srcFs = srcPath.getFileSystem(getConf());
>       Path[] paths = FileUtil.stat2Paths(srcFs.globStatus(srcPath), srcPath);
>       for(Path path : paths) {
>         try {
>           FileStatus file = srcFs.getFileStatus(path);
>           if (file == null) {
>             System.err.println(handler.getName() + 
>                                ": could not get status for '" + path + "'");
>             errors++;
>           } else {
>             errors += runCmdHandler(handler, file, srcFs, recursive);
>           }
>         } catch (IOException e) {
>           String msg = (e.getMessage() != null ? e.getLocalizedMessage() :
>             (e.getCause().getMessage() != null ? 
>                 e.getCause().getLocalizedMessage() : "null"));
>           System.err.println(handler.getName() + ": could not get status for '"
>                                         + path + "': " + msg.split("\n")[0]);
>           errors++;
>         }
>       }
>     }
>  {code}
> If there are no files on HDFS matching to wildcard input then  {{srcFs.globStatus(srcpath)}} returns 0. 
> {{ Path[] paths = FileUtil.stat2Paths(srcFs.globStatus(srcPath), srcPath);}}
> Resulting no increment in {{errors}} and command exits with 0 even though file/directory does not exist.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7433-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 18:02:50 2010
Return-Path: <common-issues-return-7433-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41227 invoked from network); 16 Apr 2010 18:02:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 18:02:50 -0000
Received: (qmail 87718 invoked by uid 500); 16 Apr 2010 18:02:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87667 invoked by uid 500); 16 Apr 2010 18:02:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87659 invoked by uid 99); 16 Apr 2010 18:02:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 18:02:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 18:02:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3GI2Px1005986
	for <common-issues@hadoop.apache.org>; Fri, 16 Apr 2010 14:02:26 -0400 (EDT)
Message-ID: <3326377.14401271440945812.JavaMail.jira@thor>
Date: Fri, 16 Apr 2010 14:02:25 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6710) Symbolic umask for file creation is
 not consistent with posix
In-Reply-To: <8947165.155011271375809341.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6710:
------------------------------------

    Attachment: hadoop-6710.patch

Attached patch:
# Adds support for posix compliant symbolic umask support with extensive tests
# Current symbolic umask of format "u=,g=,o=" is not parsed correctly. This should be same as octal umask 777. Changing the regex to support this.

> Symbolic umask for file creation is not consistent with posix
> -------------------------------------------------------------
>
>                 Key: HADOOP-6710
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6710
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Suresh Srinivas
>         Attachments: hadoop-6710.patch
>
>
> Currently both octal and symbolic umask are used to reset the file creation mode bits. This is not consistent with the behavior defined in posix. Making it consistent would avoid confusion to the HDFS users.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7434-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 18:06:50 2010
Return-Path: <common-issues-return-7434-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42300 invoked from network); 16 Apr 2010 18:06:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 18:06:50 -0000
Received: (qmail 93407 invoked by uid 500); 16 Apr 2010 18:06:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93383 invoked by uid 500); 16 Apr 2010 18:06:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93375 invoked by uid 99); 16 Apr 2010 18:06:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 18:06:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 18:06:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3GI6Qvq006011
	for <common-issues@hadoop.apache.org>; Fri, 16 Apr 2010 14:06:26 -0400 (EDT)
Message-ID: <12392505.14451271441186078.JavaMail.jira@thor>
Date: Fri, 16 Apr 2010 14:06:26 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6711) Configuration should support list
 of values
In-Reply-To: <26101198.3701271399604546.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857913#action_12857913 ] 

Hong Tang commented on HADOOP-6711:
-----------------------------------

A word of caution: the proposed implementation using string concatenation is inefficient when a user wants to add a lot of items on the list. I have been hit by this before when using FileInputFormat.addInputPath.

> Configuration should support list of values
> -------------------------------------------
>
>                 Key: HADOOP-6711
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6711
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: conf
>            Reporter: Amar Kamat
>
> Configuration supports 2 operations namely _set()_ and _get()_. It would be nice to have an inbuild support for lists where there can be multiple values (i.e list of values) assigned to one key. A workaround could be
> {code}
> // Assume Key be the parameter key and newValue be the value to be added/appended
> Configuration c = new Configuration();
> String value = c.get(Key);
> value = value + " " + newValue
> c.set(Key, value);
> {code}
> One common usecase is that in a production enviroment, some user facing params (e.g mapred.child.java.opts) are set to default values (say for performance reasons). Users themselves might want to *add* to this list. Doing a set would overwrite the previous values. The above mentioned hack is doable via code but not via command line. Hence we need the framework to support lists.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7435-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 18:08:49 2010
Return-Path: <common-issues-return-7435-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42734 invoked from network); 16 Apr 2010 18:08:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 18:08:49 -0000
Received: (qmail 96409 invoked by uid 500); 16 Apr 2010 18:08:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96387 invoked by uid 500); 16 Apr 2010 18:08:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96379 invoked by uid 99); 16 Apr 2010 18:08:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 18:08:48 +0000
X-ASF-Spam-Status: No, hits=-1301.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 18:08:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3GI8Qwn006036
	for <common-issues@hadoop.apache.org>; Fri, 16 Apr 2010 14:08:27 -0400 (EDT)
Message-ID: <3768320.14511271441306936.JavaMail.jira@thor>
Date: Fri, 16 Apr 2010 14:08:26 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857915#action_12857915 ] 

Jitendra Nath Pandey commented on HADOOP-6706:
----------------------------------------------

patch looks good.

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706.bp20.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7436-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 18:38:48 2010
Return-Path: <common-issues-return-7436-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56978 invoked from network); 16 Apr 2010 18:38:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 18:38:48 -0000
Received: (qmail 36977 invoked by uid 500); 16 Apr 2010 18:38:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36887 invoked by uid 500); 16 Apr 2010 18:38:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36879 invoked by uid 99); 16 Apr 2010 18:38:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 18:38:48 +0000
X-ASF-Spam-Status: No, hits=-1301.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 18:38:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3GIcRGV006294
	for <common-issues@hadoop.apache.org>; Fri, 16 Apr 2010 14:38:27 -0400 (EDT)
Message-ID: <26322324.15141271443107229.JavaMail.jira@thor>
Date: Fri, 16 Apr 2010 14:38:27 -0400 (EDT)
From: "David Rosenstrauch (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6372) MurmurHash does not yield the same
 results as the reference C++ implementation when size % 4 >= 2
In-Reply-To: <1460990402.1258114539644.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857927#action_12857927 ] 

David Rosenstrauch commented on HADOOP-6372:
--------------------------------------------

I just ran into this problem as well.  Are there any plans to release a fix?

> MurmurHash does not yield the same results as the reference C++ implementation when size % 4 >= 2
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6372
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6372
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.20.1
>            Reporter: olivier gillet
>            Priority: Trivial
>         Attachments: HADOOP-6372.patch, murmur.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Last rounds of MurmurHash are done in reverse order. data[length - 3], data[length - 2] and data[length - 1] in the block processing the remaining bytes should be data[len_m +2], data[len_m + 1], data[len_m].

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7437-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 18:42:48 2010
Return-Path: <common-issues-return-7437-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57990 invoked from network); 16 Apr 2010 18:42:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 18:42:47 -0000
Received: (qmail 43929 invoked by uid 500); 16 Apr 2010 18:42:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43910 invoked by uid 500); 16 Apr 2010 18:42:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43902 invoked by uid 99); 16 Apr 2010 18:42:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 18:42:47 +0000
X-ASF-Spam-Status: No, hits=-1301.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 18:42:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3GIgPnU006329
	for <common-issues@hadoop.apache.org>; Fri, 16 Apr 2010 14:42:26 -0400 (EDT)
Message-ID: <11208919.15231271443345906.JavaMail.jira@thor>
Date: Fri, 16 Apr 2010 14:42:25 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6703) Prevent renaming a file, symlink or
 directory to itself
In-Reply-To: <23186149.74901271194550994.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6703:
--------------------------------

    Attachment: hadoop-6703-2.patch

Per HDFS-1088, minor update to report file names in the exceptions.

> Prevent renaming a file, symlink or directory to itself
> -------------------------------------------------------
>
>                 Key: HADOOP-6703
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6703
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6703-1.patch, hadoop-6703-2.patch
>
>
> Per HDFS-1088 let's throw a FileAlreadyExistsException if renaming a file, symlink or directory to itself, or a symlink to the file it links to. 

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7438-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 19:12:51 2010
Return-Path: <common-issues-return-7438-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64728 invoked from network); 16 Apr 2010 19:12:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 19:12:50 -0000
Received: (qmail 81642 invoked by uid 500); 16 Apr 2010 19:12:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81621 invoked by uid 500); 16 Apr 2010 19:12:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81613 invoked by uid 99); 16 Apr 2010 19:12:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 19:12:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 19:12:48 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3GJCQ7w006606
	for <common-issues@hadoop.apache.org>; Fri, 16 Apr 2010 15:12:27 -0400 (EDT)
Message-ID: <30255840.15791271445146948.JavaMail.jira@thor>
Date: Fri, 16 Apr 2010 15:12:26 -0400 (EDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6708) New file format for very large
 records
In-Reply-To: <23756292.153431271370052193.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857937#action_12857937 ] 

Aaron Kimball commented on HADOOP-6708:
---------------------------------------

Hong,

This sounds like TFile could be adapted to my needs. Thanks for helping explain all of that thoroughly. Is this block offset/length index already maintained? It sounds like the skip-to-the-next-block optimization is not already implemented. Do you know what are the next steps required to make that happen?

The other thing that needs to happen is an API to support long-valued lengths. Should I submit a patch that modifies the existing method signatures? Or provide additional methods (e.g., as in http://commons.apache.org/io/api-1.4/org/apache/commons/io/input/CountingInputStream.html)

> New file format for very large records
> --------------------------------------
>
>                 Key: HADOOP-6708
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6708
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>         Attachments: lobfile.pdf
>
>
> A file format that handles multi-gigabyte records efficiently, with lazy disk access

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7439-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 19:50:47 2010
Return-Path: <common-issues-return-7439-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79979 invoked from network); 16 Apr 2010 19:50:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 19:50:46 -0000
Received: (qmail 21251 invoked by uid 500); 16 Apr 2010 19:50:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21218 invoked by uid 500); 16 Apr 2010 19:50:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21210 invoked by uid 99); 16 Apr 2010 19:50:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 19:50:46 +0000
X-ASF-Spam-Status: No, hits=-1302.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 19:50:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3GJoP4t006964
	for <common-issues@hadoop.apache.org>; Fri, 16 Apr 2010 15:50:25 -0400 (EDT)
Message-ID: <22512726.16491271447425209.JavaMail.jira@thor>
Date: Fri, 16 Apr 2010 15:50:25 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857954#action_12857954 ] 

Devaraj Das commented on HADOOP-6706:
-------------------------------------

One note on the patch is that this is a stop-gap solution (and is lame, in that it banks on exception handling to do relogins). The ideal solution is to use the Refreshable interface and in that case we don't have to rely on exceptions being thrown before attempting relogins. However, as Owen noted in http://tinyurl.com/y6putxk, the patch that uses the Refreshable interface doesn't seem to be working as expected.

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706.bp20.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7440-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 20:12:46 2010
Return-Path: <common-issues-return-7440-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95726 invoked from network); 16 Apr 2010 20:12:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 20:12:46 -0000
Received: (qmail 42980 invoked by uid 500); 16 Apr 2010 20:12:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42940 invoked by uid 500); 16 Apr 2010 20:12:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42932 invoked by uid 99); 16 Apr 2010 20:12:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 20:12:46 +0000
X-ASF-Spam-Status: No, hits=-1302.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 20:12:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3GKCPt0007063
	for <common-issues@hadoop.apache.org>; Fri, 16 Apr 2010 16:12:25 -0400 (EDT)
Message-ID: <11053553.16651271448745166.JavaMail.jira@thor>
Date: Fri, 16 Apr 2010 16:12:25 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6705) jiracli fails to upload test-patch
 comments to jira
In-Reply-To: <5861257.127621271286648655.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan resolved HADOOP-6705.
----------------------------------------

    Fix Version/s: 0.22.0
       Resolution: Fixed

> jiracli fails to upload test-patch comments to jira
> ---------------------------------------------------
>
>                 Key: HADOOP-6705
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6705
>             Project: Hadoop Common
>          Issue Type: Test
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6705.PATCH
>
>
>      [exec] ======================================================================
>      [exec]     Adding comment to Jira.
>      [exec] ======================================================================
>      [exec] ======================================================================
>      [exec] 
>      [exec] 
>      [exec] Failed to connect to: http://issues.apache.org/jira/rpc/soap/jirasoapservice-v2?wsdl
>      [exec] Failed to connect to: http://issues.apache.org/jira/rpc/soap/jirasoapservice-v2?wsdl
>      [exec] Failed to connect to: http://issues.apache.org/jira/rpc/soap/jirasoapservice-v2?wsdl
>      [exec]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7441-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 20:14:46 2010
Return-Path: <common-issues-return-7441-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96690 invoked from network); 16 Apr 2010 20:14:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 20:14:46 -0000
Received: (qmail 45499 invoked by uid 500); 16 Apr 2010 20:14:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45464 invoked by uid 500); 16 Apr 2010 20:14:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45456 invoked by uid 99); 16 Apr 2010 20:14:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 20:14:46 +0000
X-ASF-Spam-Status: No, hits=-1302.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 20:14:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3GKEO2s007082
	for <common-issues@hadoop.apache.org>; Fri, 16 Apr 2010 16:14:25 -0400 (EDT)
Message-ID: <11673989.16681271448864839.JavaMail.jira@thor>
Date: Fri, 16 Apr 2010 16:14:24 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6705) jiracli fails to upload test-patch
 comments to jira
In-Reply-To: <5861257.127621271286648655.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6705:
---------------------------------------

    Component/s: build

Addin in the right component for jira

> jiracli fails to upload test-patch comments to jira
> ---------------------------------------------------
>
>                 Key: HADOOP-6705
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6705
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: build
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6705.PATCH
>
>
>      [exec] ======================================================================
>      [exec]     Adding comment to Jira.
>      [exec] ======================================================================
>      [exec] ======================================================================
>      [exec] 
>      [exec] 
>      [exec] Failed to connect to: http://issues.apache.org/jira/rpc/soap/jirasoapservice-v2?wsdl
>      [exec] Failed to connect to: http://issues.apache.org/jira/rpc/soap/jirasoapservice-v2?wsdl
>      [exec] Failed to connect to: http://issues.apache.org/jira/rpc/soap/jirasoapservice-v2?wsdl
>      [exec]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7442-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 20:47:51 2010
Return-Path: <common-issues-return-7442-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13766 invoked from network); 16 Apr 2010 20:47:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 20:47:50 -0000
Received: (qmail 611 invoked by uid 500); 16 Apr 2010 20:47:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 585 invoked by uid 500); 16 Apr 2010 20:47:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 577 invoked by uid 99); 16 Apr 2010 20:47:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 20:47:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 20:47:48 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3GKlQCo007322
	for <common-issues@hadoop.apache.org>; Fri, 16 Apr 2010 16:47:26 -0400 (EDT)
Message-ID: <7132674.17211271450846709.JavaMail.jira@thor>
Date: Fri, 16 Apr 2010 16:47:26 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6703) Prevent renaming a file, symlink or
 directory to itself
In-Reply-To: <23186149.74901271194550994.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857966#action_12857966 ] 

Suresh Srinivas commented on HADOOP-6703:
-----------------------------------------

+1 for the patch.

> Prevent renaming a file, symlink or directory to itself
> -------------------------------------------------------
>
>                 Key: HADOOP-6703
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6703
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6703-1.patch, hadoop-6703-2.patch
>
>
> Per HDFS-1088 let's throw a FileAlreadyExistsException if renaming a file, symlink or directory to itself, or a symlink to the file it links to. 

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7443-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 21:27:48 2010
Return-Path: <common-issues-return-7443-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32574 invoked from network); 16 Apr 2010 21:27:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 21:27:48 -0000
Received: (qmail 39224 invoked by uid 500); 16 Apr 2010 21:27:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39194 invoked by uid 500); 16 Apr 2010 21:27:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39186 invoked by uid 99); 16 Apr 2010 21:27:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 21:27:48 +0000
X-ASF-Spam-Status: No, hits=-1302.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 21:27:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3GLRQN2007618
	for <common-issues@hadoop.apache.org>; Fri, 16 Apr 2010 17:27:26 -0400 (EDT)
Message-ID: <12748021.17721271453246766.JavaMail.jira@thor>
Date: Fri, 16 Apr 2010 17:27:26 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6711) Configuration should support list
 of values
In-Reply-To: <26101198.3701271399604546.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858010#action_12858010 ] 

Jitendra Nath Pandey commented on HADOOP-6711:
----------------------------------------------

Configuration has a getStringCollection API : public Collection<String> getStringCollection(String name)

There is no such set method though.

> Configuration should support list of values
> -------------------------------------------
>
>                 Key: HADOOP-6711
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6711
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: conf
>            Reporter: Amar Kamat
>
> Configuration supports 2 operations namely _set()_ and _get()_. It would be nice to have an inbuild support for lists where there can be multiple values (i.e list of values) assigned to one key. A workaround could be
> {code}
> // Assume Key be the parameter key and newValue be the value to be added/appended
> Configuration c = new Configuration();
> String value = c.get(Key);
> value = value + " " + newValue
> c.set(Key, value);
> {code}
> One common usecase is that in a production enviroment, some user facing params (e.g mapred.child.java.opts) are set to default values (say for performance reasons). Users themselves might want to *add* to this list. Doing a set would overwrite the previous values. The above mentioned hack is doable via code but not via command line. Hence we need the framework to support lists.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7444-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 21:35:51 2010
Return-Path: <common-issues-return-7444-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43983 invoked from network); 16 Apr 2010 21:35:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 21:35:51 -0000
Received: (qmail 43305 invoked by uid 500); 16 Apr 2010 21:35:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43277 invoked by uid 500); 16 Apr 2010 21:35:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43269 invoked by uid 99); 16 Apr 2010 21:35:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 21:35:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 21:35:48 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3GLZR0S007674
	for <common-issues@hadoop.apache.org>; Fri, 16 Apr 2010 17:35:27 -0400 (EDT)
Message-ID: <1111229.17831271453727002.JavaMail.jira@thor>
Date: Fri, 16 Apr 2010 17:35:27 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6708) New file format for very large
 records
In-Reply-To: <23756292.153431271370052193.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858011#action_12858011 ] 

Hong Tang commented on HADOOP-6708:
-----------------------------------

bq. Is this block offset/length index already maintained?
Yes.

bq. It sounds like the skip-to-the-next-block optimization is not already implemented. Do you know what are the next steps required to make that happen?
Correct. There is no immediate plan for me to work on this. The implementation could be quite simple but I'll need a bit more time to refresh my mind of the code structure. Also, would you be able to help test this?

bq. The other thing that needs to happen is an API to support long-valued lengths. Should I submit a patch that modifies the existing method signatures? 
I think modifying the existing method signature is fine (not clear why we need CountingInputStream in TFile, users should be able to create one on top of the input stream we provide, right?).

> New file format for very large records
> --------------------------------------
>
>                 Key: HADOOP-6708
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6708
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>         Attachments: lobfile.pdf
>
>
> A file format that handles multi-gigabyte records efficiently, with lazy disk access

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7445-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 21:43:48 2010
Return-Path: <common-issues-return-7445-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75436 invoked from network); 16 Apr 2010 21:43:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 21:43:48 -0000
Received: (qmail 45929 invoked by uid 500); 16 Apr 2010 21:43:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45901 invoked by uid 500); 16 Apr 2010 21:43:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45893 invoked by uid 99); 16 Apr 2010 21:43:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 21:43:47 +0000
X-ASF-Spam-Status: No, hits=-1302.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 21:43:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3GLhQ2s007777
	for <common-issues@hadoop.apache.org>; Fri, 16 Apr 2010 17:43:26 -0400 (EDT)
Message-ID: <3120465.18111271454206301.JavaMail.jira@thor>
Date: Fri, 16 Apr 2010 17:43:26 -0400 (EDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6708) New file format for very large
 records
In-Reply-To: <23756292.153431271370052193.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858015#action_12858015 ] 

Aaron Kimball commented on HADOOP-6708:
---------------------------------------

I'm definitely able to help test. I can do some manual testing of large records / performance after there's a patch available.

I wasn't suggesting that we use CountingInputStream in there; I was mostly using that as an example of having dual interfaces to retrieve the same value -- one as an integer ({{getCount()}}), and the other as a long ({{getByteCount()}}). This is not my preference; I'd prefer to modify the existing methods. But it is an incompatible change, so I figured I'd check first.



> New file format for very large records
> --------------------------------------
>
>                 Key: HADOOP-6708
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6708
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>         Attachments: lobfile.pdf
>
>
> A file format that handles multi-gigabyte records efficiently, with lazy disk access

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7446-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 21:47:50 2010
Return-Path: <common-issues-return-7446-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77254 invoked from network); 16 Apr 2010 21:47:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 21:47:50 -0000
Received: (qmail 49958 invoked by uid 500); 16 Apr 2010 21:47:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49931 invoked by uid 500); 16 Apr 2010 21:47:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49923 invoked by uid 99); 16 Apr 2010 21:47:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 21:47:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 21:47:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3GLlQ0K007829
	for <common-issues@hadoop.apache.org>; Fri, 16 Apr 2010 17:47:26 -0400 (EDT)
Message-ID: <28013434.18251271454446322.JavaMail.jira@thor>
Date: Fri, 16 Apr 2010 17:47:26 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6708) New file format for very large
 records
In-Reply-To: <23756292.153431271370052193.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858017#action_12858017 ] 

Tom White commented on HADOOP-6708:
-----------------------------------

FWIW I've marked the TFile interfaces as "Evolving" in HADOOP-6668, which would be consistent with making an incompatible change for the next release.

> New file format for very large records
> --------------------------------------
>
>                 Key: HADOOP-6708
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6708
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>         Attachments: lobfile.pdf
>
>
> A file format that handles multi-gigabyte records efficiently, with lazy disk access

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7447-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 22:05:50 2010
Return-Path: <common-issues-return-7447-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85566 invoked from network); 16 Apr 2010 22:05:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 22:05:50 -0000
Received: (qmail 68899 invoked by uid 500); 16 Apr 2010 22:05:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68839 invoked by uid 500); 16 Apr 2010 22:05:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68826 invoked by uid 99); 16 Apr 2010 22:05:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 22:05:50 +0000
X-ASF-Spam-Status: No, hits=-1302.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 22:05:49 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3GM5Tok007975
	for <common-issues@hadoop.apache.org>; Fri, 16 Apr 2010 18:05:29 -0400 (EDT)
Message-ID: <19068671.18561271455529383.JavaMail.jira@thor>
Date: Fri, 16 Apr 2010 18:05:29 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6701)  Incorrect exit codes for "dfs
 -chown", "dfs -chgrp"
In-Reply-To: <29784448.2801271097111992.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6701:
---------------------------------

    Attachment:     (was: HADOOP-6701.patch)

>  Incorrect exit codes for "dfs -chown", "dfs -chgrp"
> ----------------------------------------------------
>
>                 Key: HADOOP-6701
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6701
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.19.1, 0.20.0, 0.20.1, 0.20.2
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>            Priority: Minor
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>
> ravi@localhost:~$ hadoop dfs -chgrp abcd /; echo $?
> chgrp: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chown  abcd /; echo $?
> chown: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chmod 755 /DOESNTEXIST; echo $?
> chmod: could not get status for '/DOESNTEXIST': File does not exist: /DOESNTEXIST
> 0
> -
> Exit codes for both of the above invocations should be non-zero to indicate that the command failed.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7448-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 22:07:52 2010
Return-Path: <common-issues-return-7448-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86636 invoked from network); 16 Apr 2010 22:07:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 22:07:51 -0000
Received: (qmail 72670 invoked by uid 500); 16 Apr 2010 22:07:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72642 invoked by uid 500); 16 Apr 2010 22:07:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72634 invoked by uid 99); 16 Apr 2010 22:07:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 22:07:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 22:07:49 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3GM7RTh008040
	for <common-issues@hadoop.apache.org>; Fri, 16 Apr 2010 18:07:27 -0400 (EDT)
Message-ID: <26757374.18621271455647479.JavaMail.jira@thor>
Date: Fri, 16 Apr 2010 18:07:27 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6701)  Incorrect exit codes for "dfs
 -chown", "dfs -chgrp"
In-Reply-To: <29784448.2801271097111992.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6701:
---------------------------------

    Attachment: HADOOP-6701-20s.patch

Attaching patch for hadoop 20.s .

>  Incorrect exit codes for "dfs -chown", "dfs -chgrp"
> ----------------------------------------------------
>
>                 Key: HADOOP-6701
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6701
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.19.1, 0.20.0, 0.20.1, 0.20.2
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>            Priority: Minor
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6701-20s.patch, HADOOP-6701-trunk.patch
>
>
> ravi@localhost:~$ hadoop dfs -chgrp abcd /; echo $?
> chgrp: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chown  abcd /; echo $?
> chown: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chmod 755 /DOESNTEXIST; echo $?
> chmod: could not get status for '/DOESNTEXIST': File does not exist: /DOESNTEXIST
> 0
> -
> Exit codes for both of the above invocations should be non-zero to indicate that the command failed.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7449-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 22:07:55 2010
Return-Path: <common-issues-return-7449-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86740 invoked from network); 16 Apr 2010 22:07:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 22:07:55 -0000
Received: (qmail 72902 invoked by uid 500); 16 Apr 2010 22:07:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72871 invoked by uid 500); 16 Apr 2010 22:07:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72863 invoked by uid 99); 16 Apr 2010 22:07:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 22:07:55 +0000
X-ASF-Spam-Status: No, hits=-1302.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 22:07:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3GM7XRf008101
	for <common-issues@hadoop.apache.org>; Fri, 16 Apr 2010 18:07:33 -0400 (EDT)
Message-ID: <11715354.18821271455653935.JavaMail.jira@thor>
Date: Fri, 16 Apr 2010 18:07:33 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6701)  Incorrect exit codes for "dfs
 -chown", "dfs -chgrp"
In-Reply-To: <29784448.2801271097111992.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6701:
---------------------------------

    Attachment: HADOOP-6701-trunk.patch

Patch for trunk . 

>  Incorrect exit codes for "dfs -chown", "dfs -chgrp"
> ----------------------------------------------------
>
>                 Key: HADOOP-6701
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6701
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.19.1, 0.20.0, 0.20.1, 0.20.2
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>            Priority: Minor
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6701-20s.patch, HADOOP-6701-trunk.patch
>
>
> ravi@localhost:~$ hadoop dfs -chgrp abcd /; echo $?
> chgrp: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chown  abcd /; echo $?
> chown: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chmod 755 /DOESNTEXIST; echo $?
> chmod: could not get status for '/DOESNTEXIST': File does not exist: /DOESNTEXIST
> 0
> -
> Exit codes for both of the above invocations should be non-zero to indicate that the command failed.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7450-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 22:15:51 2010
Return-Path: <common-issues-return-7450-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90164 invoked from network); 16 Apr 2010 22:15:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 22:15:51 -0000
Received: (qmail 83476 invoked by uid 500); 16 Apr 2010 22:15:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83441 invoked by uid 500); 16 Apr 2010 22:15:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83433 invoked by uid 99); 16 Apr 2010 22:15:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 22:15:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 22:15:48 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3GMFQkw008256
	for <common-issues@hadoop.apache.org>; Fri, 16 Apr 2010 18:15:26 -0400 (EDT)
Message-ID: <21450939.19121271456126588.JavaMail.jira@thor>
Date: Fri, 16 Apr 2010 18:15:26 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858033#action_12858033 ] 

Suresh Srinivas commented on HADOOP-6563:
-----------------------------------------

Eli, does this functionality comply with rename specification of posix? It is available at http://www.opengroup.org/onlinepubs/000095399/functions/rename.html.


> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6563-1.patch, hadoop-6563-2.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7451-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 16 23:51:51 2010
Return-Path: <common-issues-return-7451-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45409 invoked from network); 16 Apr 2010 23:51:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Apr 2010 23:51:51 -0000
Received: (qmail 35006 invoked by uid 500); 16 Apr 2010 23:51:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34966 invoked by uid 500); 16 Apr 2010 23:51:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34958 invoked by uid 99); 16 Apr 2010 23:51:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 23:51:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Apr 2010 23:51:49 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3GNpRBC008884
	for <common-issues@hadoop.apache.org>; Fri, 16 Apr 2010 19:51:27 -0400 (EDT)
Message-ID: <2429504.20581271461887379.JavaMail.jira@thor>
Date: Fri, 16 Apr 2010 19:51:27 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858054#action_12858054 ] 

Eli Collins commented on HADOOP-6563:
-------------------------------------

Yes.  See the initial comment for the caveat "conforms to the posix behavior (modulo the overwrite option which is not present in rename(2))".

> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6563-1.patch, hadoop-6563-2.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7453-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 18 16:21:48 2010
Return-Path: <common-issues-return-7453-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22921 invoked from network); 18 Apr 2010 16:21:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Apr 2010 16:21:48 -0000
Received: (qmail 34801 invoked by uid 500); 18 Apr 2010 16:21:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34653 invoked by uid 500); 18 Apr 2010 16:21:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34635 invoked by uid 99); 18 Apr 2010 16:21:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 18 Apr 2010 16:21:46 +0000
X-ASF-Spam-Status: No, hits=-1307.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 18 Apr 2010 16:21:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3IGLPNf002439
	for <common-issues@hadoop.apache.org>; Sun, 18 Apr 2010 12:21:25 -0400 (EDT)
Message-ID: <21174558.32921271607685888.JavaMail.jira@thor>
Date: Sun, 18 Apr 2010 12:21:25 -0400 (EDT)
From: "John Plevyak (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6712) librecordio support for xerces 3,
 eliminate compiler warnings and the (optional) ability to compile in the
 source directory
In-Reply-To: <13573953.32891271607685237.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

John Plevyak updated HADOOP-6712:
---------------------------------

    Attachment: librecordio-jp-v1.patch

> librecordio support for xerces 3, eliminate compiler warnings and the (optional) ability to compile in the source directory
> ---------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6712
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6712
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: record
>         Environment: 64-bit linux w/gcc 4.4.3 w/xerces 3
>            Reporter: John Plevyak
>         Attachments: librecordio-jp-v1.patch
>
>
> I don't know if this code is current supported, but since it is in the tree here are some fixes:
> 1. support for xerces 3.X as well as 2.X
>     the patch checks XERCES_VERSION_MAJOR and I have tested on 3.X but before
>    committing, someone should retest on 2.X
> 2. gcc 4.4.3 on 64-bit complains about using %lld with int64_t.  Casting
>     to 'long long int' solves the issue
> 3. since there is currently no ant target, check if LIBRECORDIO_BUILD_DIR
>     is undefined and if so assume '.' to support compiling in the source directory
>     This should not effect "normal" compilation if/when an ant target is created.
> patch attached

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7452-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 18 16:21:48 2010
Return-Path: <common-issues-return-7452-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22918 invoked from network); 18 Apr 2010 16:21:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Apr 2010 16:21:48 -0000
Received: (qmail 34705 invoked by uid 500); 18 Apr 2010 16:21:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34634 invoked by uid 500); 18 Apr 2010 16:21:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34626 invoked by uid 99); 18 Apr 2010 16:21:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 18 Apr 2010 16:21:46 +0000
X-ASF-Spam-Status: No, hits=-1307.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 18 Apr 2010 16:21:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3IGLP4S002430
	for <common-issues@hadoop.apache.org>; Sun, 18 Apr 2010 12:21:25 -0400 (EDT)
Message-ID: <13573953.32891271607685237.JavaMail.jira@thor>
Date: Sun, 18 Apr 2010 12:21:25 -0400 (EDT)
From: "John Plevyak (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6712) librecordio support for xerces 3,
 eliminate compiler warnings and the (optional) ability to compile in the
 source directory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

librecordio support for xerces 3, eliminate compiler warnings and the (optional) ability to compile in the source directory
---------------------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-6712
                 URL: https://issues.apache.org/jira/browse/HADOOP-6712
             Project: Hadoop Common
          Issue Type: Bug
          Components: record
         Environment: 64-bit linux w/gcc 4.4.3 w/xerces 3
            Reporter: John Plevyak
         Attachments: librecordio-jp-v1.patch

I don't know if this code is current supported, but since it is in the tree here are some fixes:

1. support for xerces 3.X as well as 2.X
    the patch checks XERCES_VERSION_MAJOR and I have tested on 3.X but before
   committing, someone should retest on 2.X

2. gcc 4.4.3 on 64-bit complains about using %lld with int64_t.  Casting
    to 'long long int' solves the issue

3. since there is currently no ant target, check if LIBRECORDIO_BUILD_DIR
    is undefined and if so assume '.' to support compiling in the source directory
    This should not effect "normal" compilation if/when an ant target is created.

patch attached

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

From common-issues-return-7454-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 19 05:55:14 2010
Return-Path: <common-issues-return-7454-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30194 invoked from network); 19 Apr 2010 05:55:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Apr 2010 05:55:14 -0000
Received: (qmail 78734 invoked by uid 500); 19 Apr 2010 05:55:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78608 invoked by uid 500); 19 Apr 2010 05:55:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78600 invoked by uid 99); 19 Apr 2010 05:55:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 05:55:11 +0000
X-ASF-Spam-Status: No, hits=-1308.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 05:55:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3J5sn6M008605
	for <common-issues@hadoop.apache.org>; Mon, 19 Apr 2010 01:54:50 -0400 (EDT)
Message-ID: <20387500.4981271656489920.JavaMail.jira@thor>
Date: Mon, 19 Apr 2010 01:54:49 -0400 (EDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6713) The RPC server Listener thread is a
 scalability bottleneck
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

The RPC server Listener thread is a scalability bottleneck
----------------------------------------------------------

                 Key: HADOOP-6713
                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
             Project: Hadoop Common
          Issue Type: Improvement
          Components: ipc
    Affects Versions: 0.20.2, 0.20.1, 0.20.0
            Reporter: dhruba borthakur
            Assignee: Dmytro Molkov


The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7455-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 19 06:12:17 2010
Return-Path: <common-issues-return-7455-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38913 invoked from network); 19 Apr 2010 06:12:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Apr 2010 06:12:17 -0000
Received: (qmail 92131 invoked by uid 500); 19 Apr 2010 06:12:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92108 invoked by uid 500); 19 Apr 2010 06:12:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92093 invoked by uid 99); 19 Apr 2010 06:12:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 06:12:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 06:12:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3J6Bp9J008694
	for <common-issues@hadoop.apache.org>; Mon, 19 Apr 2010 02:11:51 -0400 (EDT)
Message-ID: <28549148.5131271657510999.JavaMail.jira@thor>
Date: Mon, 19 Apr 2010 02:11:50 -0400 (EDT)
From: "Christian Kunz (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6713) The RPC server Listener thread is a
 scalability bottleneck
In-Reply-To: <20387500.4981271656489920.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858391#action_12858391 ] 

Christian Kunz commented on HADOOP-6713:
----------------------------------------

That would be a great improvement for our clusters as well. Probably for any cluster with a lot of clients not necessarily being map-reduce applications.

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.20.0, 0.20.1, 0.20.2
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7456-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 19 06:12:17 2010
Return-Path: <common-issues-return-7456-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38915 invoked from network); 19 Apr 2010 06:12:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Apr 2010 06:12:17 -0000
Received: (qmail 92156 invoked by uid 500); 19 Apr 2010 06:12:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92111 invoked by uid 500); 19 Apr 2010 06:12:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92092 invoked by uid 99); 19 Apr 2010 06:12:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 06:12:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 06:12:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3J6Bok8008685
	for <common-issues@hadoop.apache.org>; Mon, 19 Apr 2010 02:11:50 -0400 (EDT)
Message-ID: <3985483.5101271657510471.JavaMail.jira@thor>
Date: Mon, 19 Apr 2010 02:11:50 -0400 (EDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6713) The RPC server Listener thread is a
 scalability bottleneck
In-Reply-To: <20387500.4981271656489920.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858390#action_12858390 ] 

dhruba borthakur commented on HADOOP-6713:
------------------------------------------

My proposal is that the Server.Listener.run() thread only executes doAccept(). Also there are a bunch of open Selector objects (instead of just one Selector object in the current code).  Each of these Selector objects are handled by its own thread.

The Server.Listener.run() invoke doAccept() that inserts a SocketChannel into a randomly selected Selector object. The thread that is monitoring this Selector object will then invoke doRead() and process the RPC.

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.20.0, 0.20.1, 0.20.2
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7457-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 19 06:26:25 2010
Return-Path: <common-issues-return-7457-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52664 invoked from network); 19 Apr 2010 06:26:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Apr 2010 06:26:25 -0000
Received: (qmail 668 invoked by uid 500); 19 Apr 2010 06:26:25 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99479 invoked by uid 500); 19 Apr 2010 06:26:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99463 invoked by uid 99); 19 Apr 2010 06:26:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 06:26:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 06:26:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3J6Po33008745
	for <common-issues@hadoop.apache.org>; Mon, 19 Apr 2010 02:25:51 -0400 (EDT)
Message-ID: <26053823.5171271658350596.JavaMail.jira@thor>
Date: Mon, 19 Apr 2010 02:25:50 -0400 (EDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6713) The RPC server Listener thread is a
 scalability bottleneck
In-Reply-To: <20387500.4981271656489920.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858392#action_12858392 ] 

dhruba borthakur commented on HADOOP-6713:
------------------------------------------

Hi Christian, I agree with you. We have around 40K clients hitting the namenode and the NN  ( inspite of running NN on a newest and greatest Nehalem) has one CPU completely maxed out by the Server.Listener thread.

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.20.0, 0.20.1, 0.20.2
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7458-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 19 15:59:20 2010
Return-Path: <common-issues-return-7458-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78163 invoked from network); 19 Apr 2010 15:59:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Apr 2010 15:59:20 -0000
Received: (qmail 92123 invoked by uid 500); 19 Apr 2010 15:59:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92092 invoked by uid 500); 19 Apr 2010 15:59:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92073 invoked by uid 99); 19 Apr 2010 15:59:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 15:59:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 15:59:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3JFwnqk001810
	for <common-issues@hadoop.apache.org>; Mon, 19 Apr 2010 11:58:49 -0400 (EDT)
Message-ID: <19191606.14561271692729693.JavaMail.jira@thor>
Date: Mon, 19 Apr 2010 11:58:49 -0400 (EDT)
From: "Patrick Angeles (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6714) FsShell 'hadoop fs -text' does not
 support compression codecs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

FsShell 'hadoop fs -text' does not support compression codecs
-------------------------------------------------------------

                 Key: HADOOP-6714
                 URL: https://issues.apache.org/jira/browse/HADOOP-6714
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Patrick Angeles


Currently, 'hadoop fs -text myfile' looks at the first few magic bytes of a file to determine whether it is gzip compressed or a sequence file. This means 'fs -text' cannot properly decode .deflate or .bz2 files (or other codecs specified via configuration).

It should be fairly straightforward to add support for other codecs by checking the file extension against the CompressionCodecFactory to retrieve an appropriate Codec.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7459-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 19 16:12:13 2010
Return-Path: <common-issues-return-7459-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90745 invoked from network); 19 Apr 2010 16:12:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Apr 2010 16:12:13 -0000
Received: (qmail 23323 invoked by uid 500); 19 Apr 2010 16:12:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23263 invoked by uid 500); 19 Apr 2010 16:12:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23255 invoked by uid 99); 19 Apr 2010 16:12:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 16:12:12 +0000
X-ASF-Spam-Status: No, hits=-1311.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 16:12:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3JGBpoW001934
	for <common-issues@hadoop.apache.org>; Mon, 19 Apr 2010 12:11:51 -0400 (EDT)
Message-ID: <24892727.14801271693511099.JavaMail.jira@thor>
Date: Mon, 19 Apr 2010 12:11:51 -0400 (EDT)
From: "Edward Capriolo (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Reopened: (HADOOP-6664) fs.inmemory.size.mb not listed in
 conf. Cluster setup page gives wrong advice.
In-Reply-To: <1859905315.553281269879147219.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Edward Capriolo reopened HADOOP-6664:
-------------------------------------


In the future, can you please not close a ticket before I even have a chance to reply.

1) the generated documentation on the site is wrong. 
2) the generated xml files in the src directory are putting variables in the wrong files.

People who are not 'In the know' will put configuration variables in the wrong file and not get the effect they desire. 

> fs.inmemory.size.mb not listed in conf. Cluster setup page gives wrong advice.
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6664
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6664
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: conf, documentation
>    Affects Versions: 0.20.2
>            Reporter: Edward Capriolo
>
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> fs.inmemory.size.mb does not appear in any xml file
> {noformat}
> grep "fs.inmemory.size.mb" ./mapred/mapred-default.xml 
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./hdfs/hdfs-default.xml 
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./core/core-default.xml 
> {noformat}
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> Documentation error:
> Real-World Cluster Configurations
> {noformat}
> conf/core-site.xml  	io.sort.factor  	100  	More streams merged at once while sorting files.
> conf/core-site.xml 	io.sort.mb 	200 	Higher memory-limit while sorting data.
> {noformat}
> core --- io.sort.factor					-- should be mapred
> core --- io.sort.mb					-- should be mapred

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7460-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 19 17:24:12 2010
Return-Path: <common-issues-return-7460-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17608 invoked from network); 19 Apr 2010 17:24:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Apr 2010 17:24:11 -0000
Received: (qmail 55370 invoked by uid 500); 19 Apr 2010 17:24:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55258 invoked by uid 500); 19 Apr 2010 17:24:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55250 invoked by uid 99); 19 Apr 2010 17:24:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 17:24:11 +0000
X-ASF-Spam-Status: No, hits=-1312.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 17:24:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3JHNnOh002766
	for <common-issues@hadoop.apache.org>; Mon, 19 Apr 2010 13:23:49 -0400 (EDT)
Message-ID: <4191593.16391271697829807.JavaMail.jira@thor>
Date: Mon, 19 Apr 2010 13:23:49 -0400 (EDT)
From: "Patrick Angeles (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6714) FsShell 'hadoop fs -text' does not
 support compression codecs
In-Reply-To: <19191606.14561271692729693.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Patrick Angeles updated HADOOP-6714:
------------------------------------

    Attachment: HADOOP-6714.patch

Attaching simple patch to resolve a file decompressor on 'hadoop fs -text' before continuing with prior behavior.

> FsShell 'hadoop fs -text' does not support compression codecs
> -------------------------------------------------------------
>
>                 Key: HADOOP-6714
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6714
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Patrick Angeles
>         Attachments: HADOOP-6714.patch
>
>
> Currently, 'hadoop fs -text myfile' looks at the first few magic bytes of a file to determine whether it is gzip compressed or a sequence file. This means 'fs -text' cannot properly decode .deflate or .bz2 files (or other codecs specified via configuration).
> It should be fairly straightforward to add support for other codecs by checking the file extension against the CompressionCodecFactory to retrieve an appropriate Codec.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7461-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 19 17:24:14 2010
Return-Path: <common-issues-return-7461-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17653 invoked from network); 19 Apr 2010 17:24:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Apr 2010 17:24:14 -0000
Received: (qmail 55428 invoked by uid 500); 19 Apr 2010 17:24:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55395 invoked by uid 500); 19 Apr 2010 17:24:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55387 invoked by uid 99); 19 Apr 2010 17:24:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 17:24:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 17:24:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3JHNoXf002772
	for <common-issues@hadoop.apache.org>; Mon, 19 Apr 2010 13:23:50 -0400 (EDT)
Message-ID: <2214550.16411271697830117.JavaMail.jira@thor>
Date: Mon, 19 Apr 2010 13:23:50 -0400 (EDT)
From: "Patrick Angeles (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6714) FsShell 'hadoop fs -text' does not
 support compression codecs
In-Reply-To: <19191606.14561271692729693.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Patrick Angeles updated HADOOP-6714:
------------------------------------

    Status: Patch Available  (was: Open)

> FsShell 'hadoop fs -text' does not support compression codecs
> -------------------------------------------------------------
>
>                 Key: HADOOP-6714
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6714
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Patrick Angeles
>         Attachments: HADOOP-6714.patch
>
>
> Currently, 'hadoop fs -text myfile' looks at the first few magic bytes of a file to determine whether it is gzip compressed or a sequence file. This means 'fs -text' cannot properly decode .deflate or .bz2 files (or other codecs specified via configuration).
> It should be fairly straightforward to add support for other codecs by checking the file extension against the CompressionCodecFactory to retrieve an appropriate Codec.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7462-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 19 17:50:15 2010
Return-Path: <common-issues-return-7462-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26467 invoked from network); 19 Apr 2010 17:50:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Apr 2010 17:50:15 -0000
Received: (qmail 4915 invoked by uid 500); 19 Apr 2010 17:50:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4802 invoked by uid 500); 19 Apr 2010 17:50:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4793 invoked by uid 99); 19 Apr 2010 17:50:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 17:50:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Apr 2010 17:50:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3JHnoo8003172
	for <common-issues@hadoop.apache.org>; Mon, 19 Apr 2010 13:49:50 -0400 (EDT)
Message-ID: <9907722.16991271699390496.JavaMail.jira@thor>
Date: Mon, 19 Apr 2010 13:49:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6714) FsShell 'hadoop fs -text' does not
 support compression codecs
In-Reply-To: <19191606.14561271692729693.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858612#action_12858612 ] 

Hadoop QA commented on HADOOP-6714:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12442200/HADOOP-6714.patch
  against trunk revision 934619.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/464/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/464/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/464/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/464/console

This message is automatically generated.

> FsShell 'hadoop fs -text' does not support compression codecs
> -------------------------------------------------------------
>
>                 Key: HADOOP-6714
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6714
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Patrick Angeles
>         Attachments: HADOOP-6714.patch
>
>
> Currently, 'hadoop fs -text myfile' looks at the first few magic bytes of a file to determine whether it is gzip compressed or a sequence file. This means 'fs -text' cannot properly decode .deflate or .bz2 files (or other codecs specified via configuration).
> It should be fairly straightforward to add support for other codecs by checking the file extension against the CompressionCodecFactory to retrieve an appropriate Codec.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7463-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 09:40:19 2010
Return-Path: <common-issues-return-7463-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34530 invoked from network); 20 Apr 2010 09:40:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 09:40:18 -0000
Received: (qmail 44835 invoked by uid 500); 20 Apr 2010 09:40:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43978 invoked by uid 500); 20 Apr 2010 09:40:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43691 invoked by uid 99); 20 Apr 2010 09:40:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 09:40:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 09:40:09 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3K9dmeT008745
	for <common-issues@hadoop.apache.org>; Tue, 20 Apr 2010 09:39:48 GMT
Message-ID: <22838944.80011271756388297.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 05:39:48 -0400 (EDT)
From: "Neil Bliss (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6704) add support for Parascale filesystem
In-Reply-To: <6991209.115951271258268758.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Neil Bliss updated HADOOP-6704:
-------------------------------

    Attachment: HADOOP-6704.0.20.2.patch

Here's a patch against the 0.20.2 tag.  A patch against trunk will be forthcoming shortly.

I'll take a look at the stress testing Jira and see if it's something we can help with.

> add support for Parascale filesystem
> ------------------------------------
>
>                 Key: HADOOP-6704
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6704
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.20.2
>            Reporter: Neil Bliss
>         Attachments: HADOOP-6704.0.20.2.patch
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Parascale has developed an org.apache.hadoop.fs implementation that allows users to use Hadoop on Parascale storage clusters.  We'd like to contribute this work to the community. Should this be placed under contrib, or integrated into the org.apache.hadoop.fs space?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7464-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 09:40:19 2010
Return-Path: <common-issues-return-7464-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34538 invoked from network); 20 Apr 2010 09:40:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 09:40:19 -0000
Received: (qmail 44938 invoked by uid 500); 20 Apr 2010 09:40:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44837 invoked by uid 500); 20 Apr 2010 09:40:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44160 invoked by uid 99); 20 Apr 2010 09:40:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 09:40:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 09:40:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3K9dqOP008808
	for <common-issues@hadoop.apache.org>; Tue, 20 Apr 2010 09:39:52 GMT
Message-ID: <17929090.80181271756392446.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 05:39:52 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6706:
--------------------------------

    Attachment: 6706.bp20.1.patch

Attaching a patch that fixes a bug in the earlier version of the Y20s patch. Not for commit.

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706.bp20.1.patch, 6706.bp20.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7465-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 09:41:17 2010
Return-Path: <common-issues-return-7465-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36565 invoked from network); 20 Apr 2010 09:41:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 09:41:16 -0000
Received: (qmail 52503 invoked by uid 500); 20 Apr 2010 09:41:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51552 invoked by uid 500); 20 Apr 2010 09:41:05 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51222 invoked by uid 99); 20 Apr 2010 09:41:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 09:41:02 +0000
X-ASF-Spam-Status: No, hits=-1315.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 09:41:01 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3K9efwo009488
	for <common-issues@hadoop.apache.org>; Tue, 20 Apr 2010 09:40:41 GMT
Message-ID: <7375715.82441271756441695.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 05:40:41 -0400 (EDT)
From: "Ravi Gummadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6715) AccessControlList.toString() returns
 empty string when we set acl to "*"
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

AccessControlList.toString() returns empty string when we set acl to "*"
------------------------------------------------------------------------

                 Key: HADOOP-6715
                 URL: https://issues.apache.org/jira/browse/HADOOP-6715
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Ravi Gummadi
            Assignee: Ravi Gummadi


AccessControlList.toString() returns empty string when we set the acl to "*" and also when we set it to empty(i.e. " "). This is causing wrong values for ACLs shown on jobdetails.jsp and jobdetailshistory.jsp web pages when acls are set to "*".

I think AccessControlList.toString() needs to be changed to return "*" when we set the acl to "*".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7466-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 09:41:38 2010
Return-Path: <common-issues-return-7466-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37860 invoked from network); 20 Apr 2010 09:41:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 09:41:33 -0000
Received: (qmail 59976 invoked by uid 500); 20 Apr 2010 09:41:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58659 invoked by uid 500); 20 Apr 2010 09:41:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58622 invoked by uid 99); 20 Apr 2010 09:41:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 09:41:27 +0000
X-ASF-Spam-Status: No, hits=-1315.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 09:41:26 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3K9f6A0009847
	for <common-issues@hadoop.apache.org>; Tue, 20 Apr 2010 09:41:06 GMT
Message-ID: <22495579.83621271756466349.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 05:41:06 -0400 (EDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6439) Shuffle deadlocks on wrong number of
 maps
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala updated HADOOP-6439:
-------------------------------------

    Status: Open  (was: Patch Available)

I looked at the patch as a sanity check and it seems fine. There were couple of formatting problems in the patch which I will fix and upload a new one soon. Canceling the current patch to take care of that.

> Shuffle deadlocks on wrong number of maps
> -----------------------------------------
>
>                 Key: HADOOP-6439
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6439
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Owen O'Malley
>            Assignee: V.V.Chaitanya Krishna
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6439-1.patch, HADOOP-6439-2.patch, HADOOP-6439-3.patch, HADOOP-6439-4.patch, HADOOP-6439-5.patch, HADOOP-6439-6.patch, HADOOP-6439-6.patch, mr-1252.patch
>
>
> The new shuffle assumes that the number of maps is correct. The new JobSubmitter sets the old value. Something misfires in the middle causing:
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is deprecated. Instead, use mapreduce.job.splitfile
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
> But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the job.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7467-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 09:46:23 2010
Return-Path: <common-issues-return-7467-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39874 invoked from network); 20 Apr 2010 09:46:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 09:46:23 -0000
Received: (qmail 68361 invoked by uid 500); 20 Apr 2010 09:46:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68327 invoked by uid 500); 20 Apr 2010 09:46:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68215 invoked by uid 99); 20 Apr 2010 09:46:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 09:46:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 09:46:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3K9jtux009952
	for <common-issues@hadoop.apache.org>; Tue, 20 Apr 2010 09:45:55 GMT
Message-ID: <16692809.83851271756755821.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 05:45:55 -0400 (EDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6439) Shuffle deadlocks on wrong number of
 maps
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala updated HADOOP-6439:
-------------------------------------

    Attachment: HADOOP-6439-7.patch

Patch fixing minor formatting problems; no other changes.

> Shuffle deadlocks on wrong number of maps
> -----------------------------------------
>
>                 Key: HADOOP-6439
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6439
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Owen O'Malley
>            Assignee: V.V.Chaitanya Krishna
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6439-1.patch, HADOOP-6439-2.patch, HADOOP-6439-3.patch, HADOOP-6439-4.patch, HADOOP-6439-5.patch, HADOOP-6439-6.patch, HADOOP-6439-6.patch, HADOOP-6439-7.patch, mr-1252.patch
>
>
> The new shuffle assumes that the number of maps is correct. The new JobSubmitter sets the old value. Something misfires in the middle causing:
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is deprecated. Instead, use mapreduce.job.splitfile
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
> But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the job.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7468-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 09:48:16 2010
Return-Path: <common-issues-return-7468-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40078 invoked from network); 20 Apr 2010 09:48:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 09:48:16 -0000
Received: (qmail 70122 invoked by uid 500); 20 Apr 2010 09:48:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70098 invoked by uid 500); 20 Apr 2010 09:48:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70090 invoked by uid 99); 20 Apr 2010 09:48:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 09:48:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 09:48:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3K9lqMV009990
	for <common-issues@hadoop.apache.org>; Tue, 20 Apr 2010 09:47:52 GMT
Message-ID: <3272037.83961271756872164.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 05:47:52 -0400 (EDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6439) Shuffle deadlocks on wrong number of
 maps
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala updated HADOOP-6439:
-------------------------------------

    Status: Patch Available  (was: Open)

Running through Hudson. Chaitanya, can you please (one last time hopefully, *smile*), test a core jar built with this patch with mapreduce and HDFS and then I can commit this ?

> Shuffle deadlocks on wrong number of maps
> -----------------------------------------
>
>                 Key: HADOOP-6439
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6439
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Owen O'Malley
>            Assignee: V.V.Chaitanya Krishna
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6439-1.patch, HADOOP-6439-2.patch, HADOOP-6439-3.patch, HADOOP-6439-4.patch, HADOOP-6439-5.patch, HADOOP-6439-6.patch, HADOOP-6439-6.patch, HADOOP-6439-7.patch, mr-1252.patch
>
>
> The new shuffle assumes that the number of maps is correct. The new JobSubmitter sets the old value. Something misfires in the middle causing:
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is deprecated. Instead, use mapreduce.job.splitfile
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
> But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the job.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7469-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 09:57:16 2010
Return-Path: <common-issues-return-7469-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43625 invoked from network); 20 Apr 2010 09:57:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 09:57:16 -0000
Received: (qmail 80518 invoked by uid 500); 20 Apr 2010 09:57:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80406 invoked by uid 500); 20 Apr 2010 09:57:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80393 invoked by uid 99); 20 Apr 2010 09:57:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 09:57:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 09:57:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3K9upMl010215
	for <common-issues@hadoop.apache.org>; Tue, 20 Apr 2010 09:56:51 GMT
Message-ID: <33363576.84001271757411543.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 05:56:51 -0400 (EDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6715) AccessControlList.toString()
 returns empty string when we set acl to "*"
In-Reply-To: <7375715.82441271756441695.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858798#action_12858798 ] 

Hemanth Yamijala commented on HADOOP-6715:
------------------------------------------

Maybe something a little more user-friendly ? Like ALL USERS or something like that ? There is a similar, but more pertinent, display issue with empty string. There, we display an empty string, which in effect shows up as nothing. It would be nice to show something like NONE.

> AccessControlList.toString() returns empty string when we set acl to "*"
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6715
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6715
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>
> AccessControlList.toString() returns empty string when we set the acl to "*" and also when we set it to empty(i.e. " "). This is causing wrong values for ACLs shown on jobdetails.jsp and jobdetailshistory.jsp web pages when acls are set to "*".
> I think AccessControlList.toString() needs to be changed to return "*" when we set the acl to "*".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7470-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 13:11:15 2010
Return-Path: <common-issues-return-7470-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95931 invoked from network); 20 Apr 2010 13:11:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 13:11:15 -0000
Received: (qmail 60603 invoked by uid 500); 20 Apr 2010 13:11:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60419 invoked by uid 500); 20 Apr 2010 13:11:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60408 invoked by uid 99); 20 Apr 2010 13:11:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 13:11:14 +0000
X-ASF-Spam-Status: No, hits=-1316.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 13:11:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3KDAqOZ012275
	for <common-issues@hadoop.apache.org>; Tue, 20 Apr 2010 13:10:52 GMT
Message-ID: <3519259.87341271769052618.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 09:10:52 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6439) Shuffle deadlocks on wrong number
 of maps
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858866#action_12858866 ] 

Hadoop QA commented on HADOOP-6439:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12442271/HADOOP-6439-7.patch
  against trunk revision 934619.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/465/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/465/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/465/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/465/console

This message is automatically generated.

> Shuffle deadlocks on wrong number of maps
> -----------------------------------------
>
>                 Key: HADOOP-6439
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6439
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Owen O'Malley
>            Assignee: V.V.Chaitanya Krishna
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6439-1.patch, HADOOP-6439-2.patch, HADOOP-6439-3.patch, HADOOP-6439-4.patch, HADOOP-6439-5.patch, HADOOP-6439-6.patch, HADOOP-6439-6.patch, HADOOP-6439-7.patch, mr-1252.patch
>
>
> The new shuffle assumes that the number of maps is correct. The new JobSubmitter sets the old value. Something misfires in the middle causing:
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is deprecated. Instead, use mapreduce.job.splitfile
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
> But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the job.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7471-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 13:43:15 2010
Return-Path: <common-issues-return-7471-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6540 invoked from network); 20 Apr 2010 13:43:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 13:43:15 -0000
Received: (qmail 2593 invoked by uid 500); 20 Apr 2010 13:43:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2521 invoked by uid 500); 20 Apr 2010 13:43:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2512 invoked by uid 99); 20 Apr 2010 13:43:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 13:43:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 13:43:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3KDgoKj012567
	for <common-issues@hadoop.apache.org>; Tue, 20 Apr 2010 13:42:50 GMT
Message-ID: <22410054.87931271770970243.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 09:42:50 -0400 (EDT)
From: "Ravi Gummadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6715) AccessControlList.toString()
 returns empty string when we set acl to "*"
In-Reply-To: <7375715.82441271756441695.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858884#action_12858884 ] 

Ravi Gummadi commented on HADOOP-6715:
--------------------------------------

Right. Would look good, if .toString() is used for displaying only.
But if .toString() is used to save it as a String and later set it to a key in some Configuration object, then "ALL USERS" will not be considered as "*" and will cause an issue ?

> AccessControlList.toString() returns empty string when we set acl to "*"
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6715
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6715
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>
> AccessControlList.toString() returns empty string when we set the acl to "*" and also when we set it to empty(i.e. " "). This is causing wrong values for ACLs shown on jobdetails.jsp and jobdetailshistory.jsp web pages when acls are set to "*".
> I think AccessControlList.toString() needs to be changed to return "*" when we set the acl to "*".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7472-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 13:45:12 2010
Return-Path: <common-issues-return-7472-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7199 invoked from network); 20 Apr 2010 13:45:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 13:45:12 -0000
Received: (qmail 5945 invoked by uid 500); 20 Apr 2010 13:45:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5909 invoked by uid 500); 20 Apr 2010 13:45:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5901 invoked by uid 99); 20 Apr 2010 13:45:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 13:45:12 +0000
X-ASF-Spam-Status: No, hits=-1316.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 13:45:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3KDioHt012603
	for <common-issues@hadoop.apache.org>; Tue, 20 Apr 2010 13:44:51 GMT
Message-ID: <18786686.88011271771090949.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 09:44:50 -0400 (EDT)
From: "Ravi Gummadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6715) AccessControlList.toString() returns
 empty string when we set acl to "*"
In-Reply-To: <7375715.82441271756441695.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated HADOOP-6715:
---------------------------------

    Description: 
AccessControlList.toString() returns empty string when we set the acl to "\*" and also when we set it to empty(i.e. " "). This is causing wrong values for ACLs shown on jobdetails.jsp and jobdetailshistory.jsp web pages when acls are set to "\*".

I think AccessControlList.toString() needs to be changed to return "\*" when we set the acl to "\*".

  was:
AccessControlList.toString() returns empty string when we set the acl to "*" and also when we set it to empty(i.e. " "). This is causing wrong values for ACLs shown on jobdetails.jsp and jobdetailshistory.jsp web pages when acls are set to "*".

I think AccessControlList.toString() needs to be changed to return "*" when we set the acl to "*".


> AccessControlList.toString() returns empty string when we set acl to "*"
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6715
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6715
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>
> AccessControlList.toString() returns empty string when we set the acl to "\*" and also when we set it to empty(i.e. " "). This is causing wrong values for ACLs shown on jobdetails.jsp and jobdetailshistory.jsp web pages when acls are set to "\*".
> I think AccessControlList.toString() needs to be changed to return "\*" when we set the acl to "\*".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7473-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 14:35:15 2010
Return-Path: <common-issues-return-7473-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29629 invoked from network); 20 Apr 2010 14:35:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 14:35:14 -0000
Received: (qmail 96722 invoked by uid 500); 20 Apr 2010 14:35:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96671 invoked by uid 500); 20 Apr 2010 14:35:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96663 invoked by uid 99); 20 Apr 2010 14:35:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 14:35:14 +0000
X-ASF-Spam-Status: No, hits=-1317.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 14:35:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3KEYpSK013257
	for <common-issues@hadoop.apache.org>; Tue, 20 Apr 2010 14:34:52 GMT
Message-ID: <22754322.88951271774091754.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 10:34:51 -0400 (EDT)
From: "Neil Bliss (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6704) add support for Parascale
 filesystem
In-Reply-To: <6991209.115951271258268758.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858905#action_12858905 ] 

Neil Bliss commented on HADOOP-6704:
------------------------------------

that patch still contains unit test failures.  I will fix it and respin the patch.

> add support for Parascale filesystem
> ------------------------------------
>
>                 Key: HADOOP-6704
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6704
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.20.2
>            Reporter: Neil Bliss
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Parascale has developed an org.apache.hadoop.fs implementation that allows users to use Hadoop on Parascale storage clusters.  We'd like to contribute this work to the community. Should this be placed under contrib, or integrated into the org.apache.hadoop.fs space?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7474-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 14:35:20 2010
Return-Path: <common-issues-return-7474-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29889 invoked from network); 20 Apr 2010 14:35:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 14:35:18 -0000
Received: (qmail 97215 invoked by uid 500); 20 Apr 2010 14:35:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97182 invoked by uid 500); 20 Apr 2010 14:35:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97174 invoked by uid 99); 20 Apr 2010 14:35:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 14:35:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 14:35:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3KEYs4w013281
	for <common-issues@hadoop.apache.org>; Tue, 20 Apr 2010 14:34:54 GMT
Message-ID: <8866948.89031271774094520.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 10:34:54 -0400 (EDT)
From: "Neil Bliss (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6704) add support for Parascale filesystem
In-Reply-To: <6991209.115951271258268758.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Neil Bliss updated HADOOP-6704:
-------------------------------

    Attachment:     (was: HADOOP-6704.0.20.2.patch)

> add support for Parascale filesystem
> ------------------------------------
>
>                 Key: HADOOP-6704
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6704
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.20.2
>            Reporter: Neil Bliss
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Parascale has developed an org.apache.hadoop.fs implementation that allows users to use Hadoop on Parascale storage clusters.  We'd like to contribute this work to the community. Should this be placed under contrib, or integrated into the org.apache.hadoop.fs space?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7475-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 15:31:13 2010
Return-Path: <common-issues-return-7475-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58703 invoked from network); 20 Apr 2010 15:31:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 15:31:13 -0000
Received: (qmail 683 invoked by uid 500); 20 Apr 2010 15:31:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 628 invoked by uid 500); 20 Apr 2010 15:31:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 619 invoked by uid 99); 20 Apr 2010 15:31:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 15:31:12 +0000
X-ASF-Spam-Status: No, hits=-1317.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 15:31:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3KFUpjU014070
	for <common-issues@hadoop.apache.org>; Tue, 20 Apr 2010 15:30:51 GMT
Message-ID: <5124415.90511271777451177.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 11:30:51 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6690) FilterFileSystem doesn't overwrite
 setTimes
In-Reply-To: <294275186.4281270693056872.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6690:
------------------------------------

    Attachment: HADOOP-6690.0.patch

Uploading patch

> FilterFileSystem doesn't overwrite setTimes
> -------------------------------------------
>
>                 Key: HADOOP-6690
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6690
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.3, 0.21.0, 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6690.0.patch
>
>
> FilterFileSystem seems to be a little outdated and it doesn't implement a few methods (setTimes being the most important one):
> - setTimes(Path, long, long)
> - copyFromLocalFile(boolean,boolean, Path, Path)
> - copyFromLocalFile(boolean, boolean, Path[], Path)
> - getUsed()
> - deleteOnExit()
> I'm not sure if all of these methods should be wrapped in FilterFileSystem, but given its purpose, I would say the more the better. It would be great to have other people's opinions about this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7476-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 15:33:16 2010
Return-Path: <common-issues-return-7476-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59710 invoked from network); 20 Apr 2010 15:33:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 15:33:16 -0000
Received: (qmail 4091 invoked by uid 500); 20 Apr 2010 15:33:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4067 invoked by uid 500); 20 Apr 2010 15:33:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4059 invoked by uid 99); 20 Apr 2010 15:33:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 15:33:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 15:33:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3KFWqGP014095
	for <common-issues@hadoop.apache.org>; Tue, 20 Apr 2010 15:32:52 GMT
Message-ID: <20193989.90581271777572301.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 11:32:52 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6690) FilterFileSystem doesn't overwrite
 setTimes
In-Reply-To: <294275186.4281270693056872.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6690:
------------------------------------

               Status: Patch Available  (was: Open)
    Affects Version/s:     (was: 0.21.0)
                           (was: 0.20.3)

> FilterFileSystem doesn't overwrite setTimes
> -------------------------------------------
>
>                 Key: HADOOP-6690
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6690
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6690.0.patch
>
>
> FilterFileSystem seems to be a little outdated and it doesn't implement a few methods (setTimes being the most important one):
> - setTimes(Path, long, long)
> - copyFromLocalFile(boolean,boolean, Path, Path)
> - copyFromLocalFile(boolean, boolean, Path[], Path)
> - getUsed()
> - deleteOnExit()
> I'm not sure if all of these methods should be wrapped in FilterFileSystem, but given its purpose, I would say the more the better. It would be great to have other people's opinions about this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7477-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 18:55:13 2010
Return-Path: <common-issues-return-7477-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71910 invoked from network); 20 Apr 2010 18:55:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 18:55:13 -0000
Received: (qmail 58135 invoked by uid 500); 20 Apr 2010 18:55:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57832 invoked by uid 500); 20 Apr 2010 18:55:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57708 invoked by uid 99); 20 Apr 2010 18:55:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 18:55:12 +0000
X-ASF-Spam-Status: No, hits=-1318.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 18:55:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3KIsocI016187
	for <common-issues@hadoop.apache.org>; Tue, 20 Apr 2010 18:54:51 GMT
Message-ID: <19456845.94831271789690859.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 14:54:50 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6716) System won't start in non-secure mode
 when kerb5.conf (edu.mit.kerberos on Mac) is not present
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

System won't start in non-secure mode when kerb5.conf (edu.mit.kerberos on Mac) is not present
----------------------------------------------------------------------------------------------

                 Key: HADOOP-6716
                 URL: https://issues.apache.org/jira/browse/HADOOP-6716
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Boris Shkolnik




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7478-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 19:09:13 2010
Return-Path: <common-issues-return-7478-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79317 invoked from network); 20 Apr 2010 19:09:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 19:09:12 -0000
Received: (qmail 79811 invoked by uid 500); 20 Apr 2010 19:09:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79705 invoked by uid 500); 20 Apr 2010 19:09:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79697 invoked by uid 99); 20 Apr 2010 19:09:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 19:09:12 +0000
X-ASF-Spam-Status: No, hits=-1318.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 19:09:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3KJ8o5U016278
	for <common-issues@hadoop.apache.org>; Tue, 20 Apr 2010 19:08:50 GMT
Message-ID: <785035.94951271790530341.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 15:08:50 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6664) fs.inmemory.size.mb not listed in
 conf. Cluster setup page gives wrong advice.
In-Reply-To: <1859905315.553281269879147219.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859024#action_12859024 ] 

Chris Douglas commented on HADOOP-6664:
---------------------------------------

Sorry for closing the issue prematurely, but I'm still unclear on what this issue is about. It sounded like you were saying that {{io.sort.factor}} and {{io.sort.mb}} belong in mapred-default.xml rather than core-default.xml, which I thought I'd answered by noting that these parameters are also used in o.a.h.io.SequenceFile (which is in core, not mapred). Given that {{fs.inmemory.size.mb}} is unused, that it doesn't appear in the default configs is also correct.

bq. the generated documentation on the site is wrong. 
bq. the generated xml files in the src directory are putting variables in the wrong files.

How? Can you either explain what is "wrong" or post a patch correcting the error?

> fs.inmemory.size.mb not listed in conf. Cluster setup page gives wrong advice.
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6664
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6664
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: conf, documentation
>    Affects Versions: 0.20.2
>            Reporter: Edward Capriolo
>
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> fs.inmemory.size.mb does not appear in any xml file
> {noformat}
> grep "fs.inmemory.size.mb" ./mapred/mapred-default.xml 
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./hdfs/hdfs-default.xml 
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./core/core-default.xml 
> {noformat}
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> Documentation error:
> Real-World Cluster Configurations
> {noformat}
> conf/core-site.xml  	io.sort.factor  	100  	More streams merged at once while sorting files.
> conf/core-site.xml 	io.sort.mb 	200 	Higher memory-limit while sorting data.
> {noformat}
> core --- io.sort.factor					-- should be mapred
> core --- io.sort.mb					-- should be mapred

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7479-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 19:25:16 2010
Return-Path: <common-issues-return-7479-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87133 invoked from network); 20 Apr 2010 19:25:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 19:25:15 -0000
Received: (qmail 94894 invoked by uid 500); 20 Apr 2010 19:25:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94844 invoked by uid 500); 20 Apr 2010 19:25:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94836 invoked by uid 99); 20 Apr 2010 19:25:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 19:25:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 19:25:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3KJOoMX016441
	for <common-issues@hadoop.apache.org>; Tue, 20 Apr 2010 19:24:51 GMT
Message-ID: <208551.95351271791490779.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 15:24:50 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6690) FilterFileSystem doesn't overwrite
 setTimes
In-Reply-To: <294275186.4281270693056872.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859031#action_12859031 ] 

Eli Collins commented on HADOOP-6690:
-------------------------------------

+1  Patch looks good.   Mind adding TestFilterFs that's a FilterFs equivalent to TestFilterFileSystem? Should be an pretty easy port.

> FilterFileSystem doesn't overwrite setTimes
> -------------------------------------------
>
>                 Key: HADOOP-6690
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6690
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6690.0.patch
>
>
> FilterFileSystem seems to be a little outdated and it doesn't implement a few methods (setTimes being the most important one):
> - setTimes(Path, long, long)
> - copyFromLocalFile(boolean,boolean, Path, Path)
> - copyFromLocalFile(boolean, boolean, Path[], Path)
> - getUsed()
> - deleteOnExit()
> I'm not sure if all of these methods should be wrapped in FilterFileSystem, but given its purpose, I would say the more the better. It would be great to have other people's opinions about this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7480-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 19:31:14 2010
Return-Path: <common-issues-return-7480-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91662 invoked from network); 20 Apr 2010 19:31:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 19:31:14 -0000
Received: (qmail 6171 invoked by uid 500); 20 Apr 2010 19:31:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5967 invoked by uid 500); 20 Apr 2010 19:31:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5955 invoked by uid 99); 20 Apr 2010 19:31:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 19:31:13 +0000
X-ASF-Spam-Status: No, hits=-1318.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 19:31:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3KJUq3a016511
	for <common-issues@hadoop.apache.org>; Tue, 20 Apr 2010 19:30:52 GMT
Message-ID: <5222107.95511271791852375.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 15:30:52 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6717) Log levels in o.a.h.security.Groups
 too high
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Log levels in o.a.h.security.Groups too high
--------------------------------------------

                 Key: HADOOP-6717
                 URL: https://issues.apache.org/jira/browse/HADOOP-6717
             Project: Hadoop Common
          Issue Type: Improvement
          Components: security
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
            Priority: Trivial


The info logs in Groups.java for every getGroups call is causing my unrelated HDFS unit test to run out of memory since it logs so darn much.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7481-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 19:33:12 2010
Return-Path: <common-issues-return-7481-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93280 invoked from network); 20 Apr 2010 19:33:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 19:33:12 -0000
Received: (qmail 11130 invoked by uid 500); 20 Apr 2010 19:33:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11104 invoked by uid 500); 20 Apr 2010 19:33:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11096 invoked by uid 99); 20 Apr 2010 19:33:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 19:33:12 +0000
X-ASF-Spam-Status: No, hits=-1319.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 19:33:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3KJWomT016529
	for <common-issues@hadoop.apache.org>; Tue, 20 Apr 2010 19:32:50 GMT
Message-ID: <24357208.95571271791970778.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 15:32:50 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6717) Log levels in o.a.h.security.Groups
 too high
In-Reply-To: <5222107.95511271791852375.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6717:
--------------------------------

    Status: Patch Available  (was: Open)

> Log levels in o.a.h.security.Groups too high
> --------------------------------------------
>
>                 Key: HADOOP-6717
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6717
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Trivial
>         Attachments: hadoop-6717.txt
>
>
> The info logs in Groups.java for every getGroups call is causing my unrelated HDFS unit test to run out of memory since it logs so darn much.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7482-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 19:33:14 2010
Return-Path: <common-issues-return-7482-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93336 invoked from network); 20 Apr 2010 19:33:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 19:33:14 -0000
Received: (qmail 11268 invoked by uid 500); 20 Apr 2010 19:33:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11245 invoked by uid 500); 20 Apr 2010 19:33:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11237 invoked by uid 99); 20 Apr 2010 19:33:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 19:33:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 19:33:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3KJWoaj016523
	for <common-issues@hadoop.apache.org>; Tue, 20 Apr 2010 19:32:50 GMT
Message-ID: <2113857.95551271791970169.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 15:32:50 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6717) Log levels in o.a.h.security.Groups
 too high
In-Reply-To: <5222107.95511271791852375.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6717:
--------------------------------

    Attachment: hadoop-6717.txt

> Log levels in o.a.h.security.Groups too high
> --------------------------------------------
>
>                 Key: HADOOP-6717
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6717
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Trivial
>         Attachments: hadoop-6717.txt
>
>
> The info logs in Groups.java for every getGroups call is causing my unrelated HDFS unit test to run out of memory since it logs so darn much.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7483-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 20:03:13 2010
Return-Path: <common-issues-return-7483-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13357 invoked from network); 20 Apr 2010 20:03:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 20:03:13 -0000
Received: (qmail 59058 invoked by uid 500); 20 Apr 2010 20:03:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58970 invoked by uid 500); 20 Apr 2010 20:03:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58962 invoked by uid 99); 20 Apr 2010 20:03:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 20:03:12 +0000
X-ASF-Spam-Status: No, hits=-1319.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 20:03:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3KK2ojO016811
	for <common-issues@hadoop.apache.org>; Tue, 20 Apr 2010 20:02:50 GMT
Message-ID: <32873051.96001271793770473.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 16:02:50 -0400 (EDT)
From: "Edward Capriolo (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6664) fs.inmemory.size.mb not listed in
 conf. Cluster setup page gives wrong advice.
In-Reply-To: <1859905315.553281269879147219.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859044#action_12859044 ] 

Edward Capriolo commented on HADOOP-6664:
-----------------------------------------

If I understandard correctly the docs for current are based on current stable 0.20.2.  Current stable does not use fs.inmemory.size.mb.

http://hadoop.apache.org/common/docs/current/cluster_setup.html. Under real world configurations 

{noformat}
conf/core-site.xml  	fs.inmemory.size.mb  	200  	 Larger amount of memory allocated for the in-memory file-system used to merge map-outputs at the reduces. 
{noformat}

As to "io.sort.factor and io.sort.mb"

They both appear in mapred-default.xml
{noformat}
[edward@ec src]$ grep -R "io.sort.factor" */*.xml
mapred/mapred-default.xml:  <name>io.sort.factor</name>
{noformat}

They should be in core-default.xml (only), or in both core-default.xml and mapred-default.conf.

Think about the end user. An end user might read a blog that states, "io.sort.factor is a magic tune set this to XXXX for awesome performance". Which file should end user put this variable in?

{noformat}
grep -R "io.sort.factor" */*.xml    
mapred/mapred-default.xml:  <name>io.sort.factor</name>
{noformat}

End user thinks, "Since I found this variable in mapred-default.xml it makese sense that I should override it in mapred-site.xml" 

The user puts the variable in the wrong place, because end user has no (easy) way of knowing that SequenceFile uses io.sort.factor or io.sort.mb. Does that make sense?


> fs.inmemory.size.mb not listed in conf. Cluster setup page gives wrong advice.
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6664
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6664
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: conf, documentation
>    Affects Versions: 0.20.2
>            Reporter: Edward Capriolo
>
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> fs.inmemory.size.mb does not appear in any xml file
> {noformat}
> grep "fs.inmemory.size.mb" ./mapred/mapred-default.xml 
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./hdfs/hdfs-default.xml 
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./core/core-default.xml 
> {noformat}
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> Documentation error:
> Real-World Cluster Configurations
> {noformat}
> conf/core-site.xml  	io.sort.factor  	100  	More streams merged at once while sorting files.
> conf/core-site.xml 	io.sort.mb 	200 	Higher memory-limit while sorting data.
> {noformat}
> core --- io.sort.factor					-- should be mapred
> core --- io.sort.mb					-- should be mapred

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7484-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 22:02:24 2010
Return-Path: <common-issues-return-7484-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78979 invoked from network); 20 Apr 2010 22:02:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 22:02:24 -0000
Received: (qmail 24351 invoked by uid 500); 20 Apr 2010 22:02:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24303 invoked by uid 500); 20 Apr 2010 22:02:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24295 invoked by uid 99); 20 Apr 2010 22:02:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 22:02:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 22:02:21 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3KM1xVl017944
	for <common-issues@hadoop.apache.org>; Tue, 20 Apr 2010 22:02:00 GMT
Message-ID: <13545372.98221271800919730.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 18:01:59 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6690) FilterFileSystem doesn't overwrite
 setTimes
In-Reply-To: <294275186.4281270693056872.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859089#action_12859089 ] 

Rodrigo Schmidt commented on HADOOP-6690:
-----------------------------------------

Sure! I can do that. I'll update the patch later today.

> FilterFileSystem doesn't overwrite setTimes
> -------------------------------------------
>
>                 Key: HADOOP-6690
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6690
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6690.0.patch
>
>
> FilterFileSystem seems to be a little outdated and it doesn't implement a few methods (setTimes being the most important one):
> - setTimes(Path, long, long)
> - copyFromLocalFile(boolean,boolean, Path, Path)
> - copyFromLocalFile(boolean, boolean, Path[], Path)
> - getUsed()
> - deleteOnExit()
> I'm not sure if all of these methods should be wrapped in FilterFileSystem, but given its purpose, I would say the more the better. It would be great to have other people's opinions about this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7485-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 22:44:15 2010
Return-Path: <common-issues-return-7485-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 871 invoked from network); 20 Apr 2010 22:44:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 22:44:15 -0000
Received: (qmail 70026 invoked by uid 500); 20 Apr 2010 22:44:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69932 invoked by uid 500); 20 Apr 2010 22:44:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69922 invoked by uid 99); 20 Apr 2010 22:44:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 22:44:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 22:44:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3KMhnPi018317
	for <common-issues@hadoop.apache.org>; Tue, 20 Apr 2010 22:43:50 GMT
Message-ID: <723292.98981271803429721.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 18:43:49 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6717) Log levels in o.a.h.security.Groups
 too high
In-Reply-To: <5222107.95511271791852375.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859104#action_12859104 ] 

Hadoop QA commented on HADOOP-6717:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12442343/hadoop-6717.txt
  against trunk revision 934619.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/44/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/44/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/44/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/44/console

This message is automatically generated.

> Log levels in o.a.h.security.Groups too high
> --------------------------------------------
>
>                 Key: HADOOP-6717
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6717
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Trivial
>         Attachments: hadoop-6717.txt
>
>
> The info logs in Groups.java for every getGroups call is causing my unrelated HDFS unit test to run out of memory since it logs so darn much.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7486-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 22:46:11 2010
Return-Path: <common-issues-return-7486-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1739 invoked from network); 20 Apr 2010 22:46:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 22:46:11 -0000
Received: (qmail 70711 invoked by uid 500); 20 Apr 2010 22:46:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70607 invoked by uid 500); 20 Apr 2010 22:46:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70594 invoked by uid 99); 20 Apr 2010 22:46:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 22:46:11 +0000
X-ASF-Spam-Status: No, hits=-1319.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 22:46:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3KMjoO0018335
	for <common-issues@hadoop.apache.org>; Tue, 20 Apr 2010 22:45:50 GMT
Message-ID: <12149021.99011271803550088.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 18:45:50 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6717) Log levels in o.a.h.security.Groups
 too high
In-Reply-To: <5222107.95511271791852375.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859105#action_12859105 ] 

Todd Lipcon commented on HADOOP-6717:
-------------------------------------

bq. -1 tests included. The patch doesn't appear to include any new or modified tests.

This is a trivial change, nothing to test.

> Log levels in o.a.h.security.Groups too high
> --------------------------------------------
>
>                 Key: HADOOP-6717
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6717
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Trivial
>         Attachments: hadoop-6717.txt
>
>
> The info logs in Groups.java for every getGroups call is causing my unrelated HDFS unit test to run out of memory since it logs so darn much.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7487-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 22:50:15 2010
Return-Path: <common-issues-return-7487-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3623 invoked from network); 20 Apr 2010 22:50:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 22:50:14 -0000
Received: (qmail 75107 invoked by uid 500); 20 Apr 2010 22:50:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75008 invoked by uid 500); 20 Apr 2010 22:50:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74998 invoked by uid 99); 20 Apr 2010 22:50:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 22:50:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 22:50:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3KMnolr018370
	for <common-issues@hadoop.apache.org>; Tue, 20 Apr 2010 22:49:50 GMT
Message-ID: <8429307.99101271803790256.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 18:49:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6690) FilterFileSystem doesn't overwrite
 setTimes
In-Reply-To: <294275186.4281270693056872.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859107#action_12859107 ] 

Hadoop QA commented on HADOOP-6690:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12442315/HADOOP-6690.0.patch
  against trunk revision 934619.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The applied patch generated 1028 javac compiler warnings (more than the trunk's current 1024 warnings).

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/466/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/466/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/466/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/466/console

This message is automatically generated.

> FilterFileSystem doesn't overwrite setTimes
> -------------------------------------------
>
>                 Key: HADOOP-6690
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6690
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6690.0.patch
>
>
> FilterFileSystem seems to be a little outdated and it doesn't implement a few methods (setTimes being the most important one):
> - setTimes(Path, long, long)
> - copyFromLocalFile(boolean,boolean, Path, Path)
> - copyFromLocalFile(boolean, boolean, Path[], Path)
> - getUsed()
> - deleteOnExit()
> I'm not sure if all of these methods should be wrapped in FilterFileSystem, but given its purpose, I would say the more the better. It would be great to have other people's opinions about this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7488-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 22:58:12 2010
Return-Path: <common-issues-return-7488-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7804 invoked from network); 20 Apr 2010 22:58:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 22:58:11 -0000
Received: (qmail 80478 invoked by uid 500); 20 Apr 2010 22:58:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80439 invoked by uid 500); 20 Apr 2010 22:58:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80431 invoked by uid 99); 20 Apr 2010 22:58:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 22:58:11 +0000
X-ASF-Spam-Status: No, hits=-1320.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 22:58:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3KMvoO1018493
	for <common-issues@hadoop.apache.org>; Tue, 20 Apr 2010 22:57:50 GMT
Message-ID: <11572587.99461271804270474.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 18:57:50 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6690) FilterFileSystem doesn't overwrite
 setTimes
In-Reply-To: <294275186.4281270693056872.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859113#action_12859113 ] 

Rodrigo Schmidt commented on HADOOP-6690:
-----------------------------------------

The warnings are generated because the methods of the DontCheck class are used by reflection. I'll silence these warnings in my next patch, but I'm not sure if Hadoop QA won't complain about that.

> FilterFileSystem doesn't overwrite setTimes
> -------------------------------------------
>
>                 Key: HADOOP-6690
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6690
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6690.0.patch
>
>
> FilterFileSystem seems to be a little outdated and it doesn't implement a few methods (setTimes being the most important one):
> - setTimes(Path, long, long)
> - copyFromLocalFile(boolean,boolean, Path, Path)
> - copyFromLocalFile(boolean, boolean, Path[], Path)
> - getUsed()
> - deleteOnExit()
> I'm not sure if all of these methods should be wrapped in FilterFileSystem, but given its purpose, I would say the more the better. It would be great to have other people's opinions about this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7489-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 20 23:36:14 2010
Return-Path: <common-issues-return-7489-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27939 invoked from network); 20 Apr 2010 23:36:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Apr 2010 23:36:14 -0000
Received: (qmail 18037 invoked by uid 500); 20 Apr 2010 23:36:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17982 invoked by uid 500); 20 Apr 2010 23:36:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17967 invoked by uid 99); 20 Apr 2010 23:36:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 23:36:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 23:36:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3KNZntV018934
	for <common-issues@hadoop.apache.org>; Tue, 20 Apr 2010 23:35:49 GMT
Message-ID: <15471232.100441271806549356.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 19:35:49 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6718) Client does not close connection when
 an exception happens during SASL negotiation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Client does not close connection when an exception happens during SASL negotiation
----------------------------------------------------------------------------------

                 Key: HADOOP-6718
                 URL: https://issues.apache.org/jira/browse/HADOOP-6718
             Project: Hadoop Common
          Issue Type: Bug
          Components: ipc
    Affects Versions: 0.22.0
            Reporter: Devaraj Das
            Assignee: Devaraj Das
             Fix For: 0.22.0


setupSaslConnection in the RPC client might fail to successfully set up a sasl connection (e.g. if the principal is wrongly configured). It throws an exception back to the caller (setupIOstreams). setupIOstreams marks the connection as closed but it doesn't really close the socket. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7490-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 00:25:13 2010
Return-Path: <common-issues-return-7490-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69751 invoked from network); 21 Apr 2010 00:25:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 00:25:12 -0000
Received: (qmail 68907 invoked by uid 500); 21 Apr 2010 00:25:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68858 invoked by uid 500); 21 Apr 2010 00:25:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68850 invoked by uid 99); 21 Apr 2010 00:25:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 00:25:11 +0000
X-ASF-Spam-Status: No, hits=-1319.6 required=10.0
	tests=ALL_TRUSTED,AWL,FB_GET_MEDS
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 00:25:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L0OoHp019721
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 00:24:50 GMT
Message-ID: <7676854.101531271809490191.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 20:24:50 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6690) FilterFileSystem doesn't overwrite
 setTimes
In-Reply-To: <294275186.4281270693056872.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859141#action_12859141 ] 

Rodrigo Schmidt commented on HADOOP-6690:
-----------------------------------------

Eli, while I was creating TestFilterFs, I realized there was a number of methods that were not being implemented. Some of them are quite suspicious like getUri(), getStatistics(), getHomeDirectory(), ... Can you please double check if FilterFs is fine without these methods. If it's not the case, it would be better to create a separate JIRA for that.

Here is the complete list:

{code}
{
    public FSDataInputStream open(final Path f) { return null; }
    public void checkPath(Path path) { }
    public Statistics getStatistics() { return null; }
    public URI getUri() { return null; }
    public Path getHomeDirectory() { return null; }
    public void checkScheme(URI uri, String supportedScheme) { }
    public String getUriPath(final Path p) { return null; }
    public void renameInternal(final Path src, final Path dst, boolean overwrite) { }
    public FsStatus getFsStatus(final Path f) { return null; }
}
{code}

> FilterFileSystem doesn't overwrite setTimes
> -------------------------------------------
>
>                 Key: HADOOP-6690
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6690
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6690.0.patch
>
>
> FilterFileSystem seems to be a little outdated and it doesn't implement a few methods (setTimes being the most important one):
> - setTimes(Path, long, long)
> - copyFromLocalFile(boolean,boolean, Path, Path)
> - copyFromLocalFile(boolean, boolean, Path[], Path)
> - getUsed()
> - deleteOnExit()
> I'm not sure if all of these methods should be wrapped in FilterFileSystem, but given its purpose, I would say the more the better. It would be great to have other people's opinions about this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7491-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 00:51:13 2010
Return-Path: <common-issues-return-7491-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87526 invoked from network); 21 Apr 2010 00:51:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 00:51:12 -0000
Received: (qmail 87353 invoked by uid 500); 21 Apr 2010 00:51:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87320 invoked by uid 500); 21 Apr 2010 00:51:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87312 invoked by uid 99); 21 Apr 2010 00:51:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 00:51:12 +0000
X-ASF-Spam-Status: No, hits=-1320.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 00:51:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L0opMK020092
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 00:50:51 GMT
Message-ID: <14007865.102351271811051578.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 20:50:51 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6690) FilterFileSystem doesn't overwrite
 setTimes
In-Reply-To: <294275186.4281270693056872.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6690:
------------------------------------

    Attachment:     (was: HADOOP-6690.1.patch)

> FilterFileSystem doesn't overwrite setTimes
> -------------------------------------------
>
>                 Key: HADOOP-6690
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6690
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6690.0.patch, HADOOP-6690.1.patch
>
>
> FilterFileSystem seems to be a little outdated and it doesn't implement a few methods (setTimes being the most important one):
> - setTimes(Path, long, long)
> - copyFromLocalFile(boolean,boolean, Path, Path)
> - copyFromLocalFile(boolean, boolean, Path[], Path)
> - getUsed()
> - deleteOnExit()
> I'm not sure if all of these methods should be wrapped in FilterFileSystem, but given its purpose, I would say the more the better. It would be great to have other people's opinions about this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7492-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 00:51:15 2010
Return-Path: <common-issues-return-7492-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87588 invoked from network); 21 Apr 2010 00:51:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 00:51:15 -0000
Received: (qmail 87476 invoked by uid 500); 21 Apr 2010 00:51:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87454 invoked by uid 500); 21 Apr 2010 00:51:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87446 invoked by uid 99); 21 Apr 2010 00:51:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 00:51:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 00:51:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L0ooeu020073
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 00:50:51 GMT
Message-ID: <25290002.102291271811050757.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 20:50:50 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6690) FilterFileSystem doesn't overwrite
 setTimes
In-Reply-To: <294275186.4281270693056872.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6690:
------------------------------------

    Attachment: HADOOP-6690.1.patch

New patch without any warnings.

> FilterFileSystem doesn't overwrite setTimes
> -------------------------------------------
>
>                 Key: HADOOP-6690
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6690
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6690.0.patch, HADOOP-6690.1.patch
>
>
> FilterFileSystem seems to be a little outdated and it doesn't implement a few methods (setTimes being the most important one):
> - setTimes(Path, long, long)
> - copyFromLocalFile(boolean,boolean, Path, Path)
> - copyFromLocalFile(boolean, boolean, Path[], Path)
> - getUsed()
> - deleteOnExit()
> I'm not sure if all of these methods should be wrapped in FilterFileSystem, but given its purpose, I would say the more the better. It would be great to have other people's opinions about this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7493-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 00:51:15 2010
Return-Path: <common-issues-return-7493-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87619 invoked from network); 21 Apr 2010 00:51:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 00:51:15 -0000
Received: (qmail 87624 invoked by uid 500); 21 Apr 2010 00:51:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87588 invoked by uid 500); 21 Apr 2010 00:51:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87580 invoked by uid 99); 21 Apr 2010 00:51:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 00:51:15 +0000
X-ASF-Spam-Status: No, hits=-1320.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 00:51:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L0osci020128
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 00:50:54 GMT
Message-ID: <32513304.102471271811054469.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 20:50:54 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6690) FilterFileSystem doesn't overwrite
 setTimes
In-Reply-To: <294275186.4281270693056872.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6690:
------------------------------------

    Attachment: HADOOP-6690.1.patch

New patch without any warnings.

> FilterFileSystem doesn't overwrite setTimes
> -------------------------------------------
>
>                 Key: HADOOP-6690
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6690
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6690.0.patch, HADOOP-6690.1.patch
>
>
> FilterFileSystem seems to be a little outdated and it doesn't implement a few methods (setTimes being the most important one):
> - setTimes(Path, long, long)
> - copyFromLocalFile(boolean,boolean, Path, Path)
> - copyFromLocalFile(boolean, boolean, Path[], Path)
> - getUsed()
> - deleteOnExit()
> I'm not sure if all of these methods should be wrapped in FilterFileSystem, but given its purpose, I would say the more the better. It would be great to have other people's opinions about this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7494-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 00:51:16 2010
Return-Path: <common-issues-return-7494-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87683 invoked from network); 21 Apr 2010 00:51:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 00:51:16 -0000
Received: (qmail 87755 invoked by uid 500); 21 Apr 2010 00:51:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87723 invoked by uid 500); 21 Apr 2010 00:51:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87715 invoked by uid 99); 21 Apr 2010 00:51:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 00:51:16 +0000
X-ASF-Spam-Status: No, hits=-1320.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 00:51:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L0otaa020164
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 00:50:55 GMT
Message-ID: <13581695.102591271811055618.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 20:50:55 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6690) FilterFileSystem doesn't overwrite
 setTimes
In-Reply-To: <294275186.4281270693056872.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6690:
------------------------------------

    Status: Patch Available  (was: Open)

> FilterFileSystem doesn't overwrite setTimes
> -------------------------------------------
>
>                 Key: HADOOP-6690
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6690
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6690.0.patch, HADOOP-6690.1.patch
>
>
> FilterFileSystem seems to be a little outdated and it doesn't implement a few methods (setTimes being the most important one):
> - setTimes(Path, long, long)
> - copyFromLocalFile(boolean,boolean, Path, Path)
> - copyFromLocalFile(boolean, boolean, Path[], Path)
> - getUsed()
> - deleteOnExit()
> I'm not sure if all of these methods should be wrapped in FilterFileSystem, but given its purpose, I would say the more the better. It would be great to have other people's opinions about this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7495-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 00:51:18 2010
Return-Path: <common-issues-return-7495-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87744 invoked from network); 21 Apr 2010 00:51:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 00:51:18 -0000
Received: (qmail 87899 invoked by uid 500); 21 Apr 2010 00:51:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87867 invoked by uid 500); 21 Apr 2010 00:51:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87859 invoked by uid 99); 21 Apr 2010 00:51:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 00:51:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 00:51:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L0otum020146
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 00:50:55 GMT
Message-ID: <25098635.102531271811055034.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 20:50:55 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6690) FilterFileSystem doesn't overwrite
 setTimes
In-Reply-To: <294275186.4281270693056872.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6690:
------------------------------------

    Status: Open  (was: Patch Available)

> FilterFileSystem doesn't overwrite setTimes
> -------------------------------------------
>
>                 Key: HADOOP-6690
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6690
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6690.0.patch, HADOOP-6690.1.patch
>
>
> FilterFileSystem seems to be a little outdated and it doesn't implement a few methods (setTimes being the most important one):
> - setTimes(Path, long, long)
> - copyFromLocalFile(boolean,boolean, Path, Path)
> - copyFromLocalFile(boolean, boolean, Path[], Path)
> - getUsed()
> - deleteOnExit()
> I'm not sure if all of these methods should be wrapped in FilterFileSystem, but given its purpose, I would say the more the better. It would be great to have other people's opinions about this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7496-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 00:56:14 2010
Return-Path: <common-issues-return-7496-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89940 invoked from network); 21 Apr 2010 00:56:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 00:56:14 -0000
Received: (qmail 90642 invoked by uid 500); 21 Apr 2010 00:56:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90617 invoked by uid 500); 21 Apr 2010 00:56:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90609 invoked by uid 99); 21 Apr 2010 00:56:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 00:56:14 +0000
X-ASF-Spam-Status: No, hits=-1998.0 required=10.0
	tests=ALL_TRUSTED,FB_GET_MEDS
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 00:56:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L0to2b020190
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 00:55:50 GMT
Message-ID: <18354396.102631271811350896.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 20:55:50 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6719) Missing methods on FilterFs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Missing methods on FilterFs
---------------------------

                 Key: HADOOP-6719
                 URL: https://issues.apache.org/jira/browse/HADOOP-6719
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.22.0
            Reporter: Rodrigo Schmidt
            Assignee: Rodrigo Schmidt
             Fix For: 0.22.0


The following methods are not declared on FilterFs(). Some of them are fine, but others like getUri(), getStatistics(), getHomeDirectory() look like they should be there.

Here is the complete list:

{code}
{
    public FSDataInputStream open(final Path f) { return null; }
    public void checkPath(Path path) { }
    public Statistics getStatistics() { return null; }
    public URI getUri() { return null; }
    public Path getHomeDirectory() { return null; }
    public void checkScheme(URI uri, String supportedScheme) { }
    public String getUriPath(final Path p) { return null; }
    public void renameInternal(final Path src, final Path dst, boolean overwrite) { }
    public FsStatus getFsStatus(final Path f) { return null; }
}
{code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7497-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 00:58:15 2010
Return-Path: <common-issues-return-7497-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91076 invoked from network); 21 Apr 2010 00:58:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 00:58:14 -0000
Received: (qmail 91589 invoked by uid 500); 21 Apr 2010 00:58:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91545 invoked by uid 500); 21 Apr 2010 00:58:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91535 invoked by uid 99); 21 Apr 2010 00:58:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 00:58:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 00:58:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L0voAO020213
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 00:57:50 GMT
Message-ID: <19670103.102701271811470629.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 20:57:50 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6690) FilterFileSystem doesn't overwrite
 setTimes
In-Reply-To: <294275186.4281270693056872.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859153#action_12859153 ] 

Rodrigo Schmidt commented on HADOOP-6690:
-----------------------------------------

I was talking to Dhruba and we thought it would be better to create a new issue to discuss the methods missing on FilterFs() and its unit test. I just created HADOOP-6719 for that.

> FilterFileSystem doesn't overwrite setTimes
> -------------------------------------------
>
>                 Key: HADOOP-6690
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6690
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6690.0.patch, HADOOP-6690.1.patch
>
>
> FilterFileSystem seems to be a little outdated and it doesn't implement a few methods (setTimes being the most important one):
> - setTimes(Path, long, long)
> - copyFromLocalFile(boolean,boolean, Path, Path)
> - copyFromLocalFile(boolean, boolean, Path[], Path)
> - getUsed()
> - deleteOnExit()
> I'm not sure if all of these methods should be wrapped in FilterFileSystem, but given its purpose, I would say the more the better. It would be great to have other people's opinions about this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7498-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 01:00:12 2010
Return-Path: <common-issues-return-7498-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92113 invoked from network); 21 Apr 2010 01:00:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 01:00:12 -0000
Received: (qmail 92598 invoked by uid 500); 21 Apr 2010 01:00:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92576 invoked by uid 500); 21 Apr 2010 01:00:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92568 invoked by uid 99); 21 Apr 2010 01:00:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 01:00:12 +0000
X-ASF-Spam-Status: No, hits=-1319.8 required=10.0
	tests=ALL_TRUSTED,AWL,FB_GET_MEDS
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 01:00:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L0xpKI020275
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 00:59:51 GMT
Message-ID: <1286125.102781271811591231.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 20:59:51 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6719) Missing methods on FilterFs
In-Reply-To: <18354396.102631271811350896.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6719:
------------------------------------

    Status: Patch Available  (was: Open)

> Missing methods on FilterFs
> ---------------------------
>
>                 Key: HADOOP-6719
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6719
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6719.0.patch
>
>
> The following methods are not declared on FilterFs(). Some of them are fine, but others like getUri(), getStatistics(), getHomeDirectory() look like they should be there.
> Here is the complete list:
> {code}
> {
>     public FSDataInputStream open(final Path f) { return null; }
>     public void checkPath(Path path) { }
>     public Statistics getStatistics() { return null; }
>     public URI getUri() { return null; }
>     public Path getHomeDirectory() { return null; }
>     public void checkScheme(URI uri, String supportedScheme) { }
>     public String getUriPath(final Path p) { return null; }
>     public void renameInternal(final Path src, final Path dst, boolean overwrite) { }
>     public FsStatus getFsStatus(final Path f) { return null; }
> }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7499-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 01:00:15 2010
Return-Path: <common-issues-return-7499-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92171 invoked from network); 21 Apr 2010 01:00:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 01:00:14 -0000
Received: (qmail 92740 invoked by uid 500); 21 Apr 2010 01:00:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92711 invoked by uid 500); 21 Apr 2010 01:00:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92703 invoked by uid 99); 21 Apr 2010 01:00:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 01:00:14 +0000
X-ASF-Spam-Status: No, hits=-1998.0 required=10.0
	tests=ALL_TRUSTED,FB_GET_MEDS
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 01:00:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L0xoqJ020263
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 00:59:50 GMT
Message-ID: <13460370.102741271811590587.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 20:59:50 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6719) Missing methods on FilterFs
In-Reply-To: <18354396.102631271811350896.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6719:
------------------------------------

    Attachment: HADOOP-6719.0.patch

Unit test that passes with the current implementation of FilterFs, but might have to be modified if we decide some of the missing methods should be there.

> Missing methods on FilterFs
> ---------------------------
>
>                 Key: HADOOP-6719
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6719
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6719.0.patch
>
>
> The following methods are not declared on FilterFs(). Some of them are fine, but others like getUri(), getStatistics(), getHomeDirectory() look like they should be there.
> Here is the complete list:
> {code}
> {
>     public FSDataInputStream open(final Path f) { return null; }
>     public void checkPath(Path path) { }
>     public Statistics getStatistics() { return null; }
>     public URI getUri() { return null; }
>     public Path getHomeDirectory() { return null; }
>     public void checkScheme(URI uri, String supportedScheme) { }
>     public String getUriPath(final Path p) { return null; }
>     public void renameInternal(final Path src, final Path dst, boolean overwrite) { }
>     public FsStatus getFsStatus(final Path f) { return null; }
> }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7500-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 01:14:12 2010
Return-Path: <common-issues-return-7500-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98092 invoked from network); 21 Apr 2010 01:14:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 01:14:12 -0000
Received: (qmail 97352 invoked by uid 500); 21 Apr 2010 01:14:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97296 invoked by uid 500); 21 Apr 2010 01:14:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97288 invoked by uid 99); 21 Apr 2010 01:14:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 01:14:11 +0000
X-ASF-Spam-Status: No, hits=-1319.9 required=10.0
	tests=ALL_TRUSTED,AWL,FB_GET_MEDS
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 01:14:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L1Dp43020412
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 01:13:51 GMT
Message-ID: <20253075.102971271812431060.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 21:13:51 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6719) Missing methods on FilterFs
In-Reply-To: <18354396.102631271811350896.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859158#action_12859158 ] 

Hadoop QA commented on HADOOP-6719:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12442378/HADOOP-6719.0.patch
  against trunk revision 934619.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/45/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/45/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/45/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/45/console

This message is automatically generated.

> Missing methods on FilterFs
> ---------------------------
>
>                 Key: HADOOP-6719
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6719
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6719.0.patch
>
>
> The following methods are not declared on FilterFs(). Some of them are fine, but others like getUri(), getStatistics(), getHomeDirectory() look like they should be there.
> Here is the complete list:
> {code}
> {
>     public FSDataInputStream open(final Path f) { return null; }
>     public void checkPath(Path path) { }
>     public Statistics getStatistics() { return null; }
>     public URI getUri() { return null; }
>     public Path getHomeDirectory() { return null; }
>     public void checkScheme(URI uri, String supportedScheme) { }
>     public String getUriPath(final Path p) { return null; }
>     public void renameInternal(final Path src, final Path dst, boolean overwrite) { }
>     public FsStatus getFsStatus(final Path f) { return null; }
> }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7501-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 02:24:15 2010
Return-Path: <common-issues-return-7501-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32146 invoked from network); 21 Apr 2010 02:24:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 02:24:15 -0000
Received: (qmail 44986 invoked by uid 500); 21 Apr 2010 02:24:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44950 invoked by uid 500); 21 Apr 2010 02:24:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44942 invoked by uid 99); 21 Apr 2010 02:24:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 02:24:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 02:24:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L2NoMq020806
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 02:23:51 GMT
Message-ID: <27289628.103571271816630898.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 22:23:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6690) FilterFileSystem doesn't overwrite
 setTimes
In-Reply-To: <294275186.4281270693056872.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859169#action_12859169 ] 

Hadoop QA commented on HADOOP-6690:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12442377/HADOOP-6690.1.patch
  against trunk revision 934619.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The applied patch generated 1028 javac compiler warnings (more than the trunk's current 1024 warnings).

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/467/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/467/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/467/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/467/console

This message is automatically generated.

> FilterFileSystem doesn't overwrite setTimes
> -------------------------------------------
>
>                 Key: HADOOP-6690
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6690
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6690.0.patch, HADOOP-6690.1.patch
>
>
> FilterFileSystem seems to be a little outdated and it doesn't implement a few methods (setTimes being the most important one):
> - setTimes(Path, long, long)
> - copyFromLocalFile(boolean,boolean, Path, Path)
> - copyFromLocalFile(boolean, boolean, Path[], Path)
> - getUsed()
> - deleteOnExit()
> I'm not sure if all of these methods should be wrapped in FilterFileSystem, but given its purpose, I would say the more the better. It would be great to have other people's opinions about this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7502-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 03:08:17 2010
Return-Path: <common-issues-return-7502-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46872 invoked from network); 21 Apr 2010 03:08:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 03:08:17 -0000
Received: (qmail 67344 invoked by uid 500); 21 Apr 2010 03:08:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67218 invoked by uid 500); 21 Apr 2010 03:08:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67209 invoked by uid 99); 21 Apr 2010 03:08:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 03:08:14 +0000
X-ASF-Spam-Status: No, hits=-1321.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 03:08:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L37q7C021006
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 03:07:53 GMT
Message-ID: <21009261.103861271819272790.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 23:07:52 -0400 (EDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6439) Shuffle deadlocks on wrong number
 of maps
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859175#action_12859175 ] 

V.V.Chaitanya Krishna commented on HADOOP-6439:
-----------------------------------------------

Ran tests of mapreduce and hdfs with the core jar built with this patch.
All tests passed except for TestTrackerDistributedCacheManager in mapreduce and TestFiHFlush in hdfs. 
These tests failed even without this patch being applied.

> Shuffle deadlocks on wrong number of maps
> -----------------------------------------
>
>                 Key: HADOOP-6439
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6439
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Owen O'Malley
>            Assignee: V.V.Chaitanya Krishna
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6439-1.patch, HADOOP-6439-2.patch, HADOOP-6439-3.patch, HADOOP-6439-4.patch, HADOOP-6439-5.patch, HADOOP-6439-6.patch, HADOOP-6439-6.patch, HADOOP-6439-7.patch, mr-1252.patch
>
>
> The new shuffle assumes that the number of maps is correct. The new JobSubmitter sets the old value. Something misfires in the middle causing:
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is deprecated. Instead, use mapreduce.job.splitfile
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
> But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the job.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7503-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 03:14:14 2010
Return-Path: <common-issues-return-7503-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47583 invoked from network); 21 Apr 2010 03:14:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 03:14:13 -0000
Received: (qmail 70844 invoked by uid 500); 21 Apr 2010 03:14:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70727 invoked by uid 500); 21 Apr 2010 03:14:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70717 invoked by uid 99); 21 Apr 2010 03:14:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 03:14:11 +0000
X-ASF-Spam-Status: No, hits=-1320.3 required=10.0
	tests=ALL_TRUSTED,AWL,FB_GET_MEDS
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 03:14:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L3Doj0021193
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 03:13:50 GMT
Message-ID: <18667817.104441271819630411.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 23:13:50 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6719) Missing methods on FilterFs
In-Reply-To: <18354396.102631271811350896.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859178#action_12859178 ] 

Eli Collins commented on HADOOP-6719:
-------------------------------------

My understanding is that FilterFs should override all AFS methods, the FilterFs javadoc concurs: "The class <code>FilterFs</code> itself simply overrides all methods of <code>AbstractFileSystem</code> with versions that pass all requests to the contained file system." It currently gets away with not overriding these because its subclasses, ChecksumFs and LocalFs, don't need to override them. 


> Missing methods on FilterFs
> ---------------------------
>
>                 Key: HADOOP-6719
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6719
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6719.0.patch
>
>
> The following methods are not declared on FilterFs(). Some of them are fine, but others like getUri(), getStatistics(), getHomeDirectory() look like they should be there.
> Here is the complete list:
> {code}
> {
>     public FSDataInputStream open(final Path f) { return null; }
>     public void checkPath(Path path) { }
>     public Statistics getStatistics() { return null; }
>     public URI getUri() { return null; }
>     public Path getHomeDirectory() { return null; }
>     public void checkScheme(URI uri, String supportedScheme) { }
>     public String getUriPath(final Path p) { return null; }
>     public void renameInternal(final Path src, final Path dst, boolean overwrite) { }
>     public FsStatus getFsStatus(final Path f) { return null; }
> }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7504-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 03:14:14 2010
Return-Path: <common-issues-return-7504-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47589 invoked from network); 21 Apr 2010 03:14:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 03:14:13 -0000
Received: (qmail 70865 invoked by uid 500); 21 Apr 2010 03:14:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70736 invoked by uid 500); 21 Apr 2010 03:14:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70726 invoked by uid 99); 21 Apr 2010 03:14:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 03:14:12 +0000
X-ASF-Spam-Status: No, hits=-1320.3 required=10.0
	tests=ALL_TRUSTED,AWL,FB_GET_MEDS
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 03:14:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L3Dpe4021205
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 03:13:51 GMT
Message-ID: <30123209.104481271819631196.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 23:13:51 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6719) Missing methods on FilterFs
In-Reply-To: <18354396.102631271811350896.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859179#action_12859179 ] 

Eli Collins commented on HADOOP-6719:
-------------------------------------

Forgot to mention in the patch DontCheck should be removed  and the necessary methods overridden in FilterFs.

> Missing methods on FilterFs
> ---------------------------
>
>                 Key: HADOOP-6719
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6719
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6719.0.patch
>
>
> The following methods are not declared on FilterFs(). Some of them are fine, but others like getUri(), getStatistics(), getHomeDirectory() look like they should be there.
> Here is the complete list:
> {code}
> {
>     public FSDataInputStream open(final Path f) { return null; }
>     public void checkPath(Path path) { }
>     public Statistics getStatistics() { return null; }
>     public URI getUri() { return null; }
>     public Path getHomeDirectory() { return null; }
>     public void checkScheme(URI uri, String supportedScheme) { }
>     public String getUriPath(final Path p) { return null; }
>     public void renameInternal(final Path src, final Path dst, boolean overwrite) { }
>     public FsStatus getFsStatus(final Path f) { return null; }
> }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7505-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 03:49:14 2010
Return-Path: <common-issues-return-7505-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52013 invoked from network); 21 Apr 2010 03:49:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 03:49:13 -0000
Received: (qmail 81416 invoked by uid 500); 21 Apr 2010 03:49:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81306 invoked by uid 500); 21 Apr 2010 03:49:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81297 invoked by uid 99); 21 Apr 2010 03:49:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 03:49:11 +0000
X-ASF-Spam-Status: No, hits=-1321.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 03:49:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L3mnFn021323
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 03:48:50 GMT
Message-ID: <24069909.104601271821729836.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 23:48:49 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6716) System won't start in non-secure
 mode when kerb5.conf (edu.mit.kerberos on Mac) is not present
In-Reply-To: <19456845.94831271789690859.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859182#action_12859182 ] 

Konstantin Boudnik commented on HADOOP-6716:
--------------------------------------------

I believe it is caused by the fact that Kerberos needs a config file to merely instantiate its {{Config}} class. Very unfortunate.

> System won't start in non-secure mode when kerb5.conf (edu.mit.kerberos on Mac) is not present
> ----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6716
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6716
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7506-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 03:53:15 2010
Return-Path: <common-issues-return-7506-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52182 invoked from network); 21 Apr 2010 03:53:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 03:53:14 -0000
Received: (qmail 81883 invoked by uid 500); 21 Apr 2010 03:53:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81765 invoked by uid 500); 21 Apr 2010 03:53:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81757 invoked by uid 99); 21 Apr 2010 03:53:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 03:53:12 +0000
X-ASF-Spam-Status: No, hits=-1321.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 03:53:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L3qpin021364
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 03:52:51 GMT
Message-ID: <18211355.104701271821971172.JavaMail.jira@thor>
Date: Tue, 20 Apr 2010 23:52:51 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6690) FilterFileSystem doesn't overwrite
 setTimes
In-Reply-To: <294275186.4281270693056872.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6690:
------------------------------------

    Status: Open  (was: Patch Available)

> FilterFileSystem doesn't overwrite setTimes
> -------------------------------------------
>
>                 Key: HADOOP-6690
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6690
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6690.0.patch, HADOOP-6690.1.patch
>
>
> FilterFileSystem seems to be a little outdated and it doesn't implement a few methods (setTimes being the most important one):
> - setTimes(Path, long, long)
> - copyFromLocalFile(boolean,boolean, Path, Path)
> - copyFromLocalFile(boolean, boolean, Path[], Path)
> - getUsed()
> - deleteOnExit()
> I'm not sure if all of these methods should be wrapped in FilterFileSystem, but given its purpose, I would say the more the better. It would be great to have other people's opinions about this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7507-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 04:15:15 2010
Return-Path: <common-issues-return-7507-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55252 invoked from network); 21 Apr 2010 04:15:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 04:15:15 -0000
Received: (qmail 93105 invoked by uid 500); 21 Apr 2010 04:15:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92937 invoked by uid 500); 21 Apr 2010 04:15:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92916 invoked by uid 99); 21 Apr 2010 04:15:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 04:15:12 +0000
X-ASF-Spam-Status: No, hits=-1321.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 04:15:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L4EpF1021994
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 04:14:51 GMT
Message-ID: <6823505.105151271823291064.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 00:14:51 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6658) Exclude Private elements from
 generated Javadoc
In-Reply-To: <995349254.452871269404067173.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6658:
------------------------------

    Status: Open  (was: Patch Available)

> Exclude Private elements from generated Javadoc
> -----------------------------------------------
>
>                 Key: HADOOP-6658
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6658
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch
>
>
> Packages, classes and methods that are marked with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation should not appear in the public Javadoc. Developer Javadoc generated using the "javadoc-dev" ant target should continue to generate Javadoc for all program elements.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7508-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 04:15:15 2010
Return-Path: <common-issues-return-7508-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55266 invoked from network); 21 Apr 2010 04:15:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 04:15:15 -0000
Received: (qmail 93130 invoked by uid 500); 21 Apr 2010 04:15:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92960 invoked by uid 500); 21 Apr 2010 04:15:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92951 invoked by uid 99); 21 Apr 2010 04:15:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 04:15:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 04:15:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L4EoqN021982
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 04:14:50 GMT
Message-ID: <23905821.105111271823290448.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 00:14:50 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6658) Exclude Private elements from
 generated Javadoc
In-Reply-To: <995349254.452871269404067173.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6658:
------------------------------

    Attachment: HADOOP-6658.patch

Test failures were due to a xerces jar being pulled onto the classpath for jdiff. It's not needed for compilation, or for the targets that use jdiff, since they use the xerces jar bundled with jdiff. 

> Exclude Private elements from generated Javadoc
> -----------------------------------------------
>
>                 Key: HADOOP-6658
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6658
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch
>
>
> Packages, classes and methods that are marked with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation should not appear in the public Javadoc. Developer Javadoc generated using the "javadoc-dev" ant target should continue to generate Javadoc for all program elements.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7509-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 04:15:16 2010
Return-Path: <common-issues-return-7509-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55291 invoked from network); 21 Apr 2010 04:15:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 04:15:15 -0000
Received: (qmail 93260 invoked by uid 500); 21 Apr 2010 04:15:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93216 invoked by uid 500); 21 Apr 2010 04:15:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93107 invoked by uid 99); 21 Apr 2010 04:15:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 04:15:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 04:15:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L4Ep0H022006
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 04:14:51 GMT
Message-ID: <9812506.105191271823291407.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 00:14:51 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6658) Exclude Private elements from
 generated Javadoc
In-Reply-To: <995349254.452871269404067173.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6658:
------------------------------

    Status: Patch Available  (was: Open)

> Exclude Private elements from generated Javadoc
> -----------------------------------------------
>
>                 Key: HADOOP-6658
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6658
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch
>
>
> Packages, classes and methods that are marked with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation should not appear in the public Javadoc. Developer Javadoc generated using the "javadoc-dev" ant target should continue to generate Javadoc for all program elements.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7511-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 04:37:15 2010
Return-Path: <common-issues-return-7511-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57442 invoked from network); 21 Apr 2010 04:37:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 04:37:15 -0000
Received: (qmail 3786 invoked by uid 500); 21 Apr 2010 04:37:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3563 invoked by uid 500); 21 Apr 2010 04:37:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3547 invoked by uid 99); 21 Apr 2010 04:37:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 04:37:12 +0000
X-ASF-Spam-Status: No, hits=-1321.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 04:37:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L4apEj022126
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 04:36:51 GMT
Message-ID: <25775877.105401271824611102.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 00:36:51 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6690) FilterFileSystem doesn't overwrite
 setTimes
In-Reply-To: <294275186.4281270693056872.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6690:
------------------------------------

    Status: Patch Available  (was: Open)

> FilterFileSystem doesn't overwrite setTimes
> -------------------------------------------
>
>                 Key: HADOOP-6690
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6690
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6690.0.patch, HADOOP-6690.1.patch, HADOOP-6690.2.patch
>
>
> FilterFileSystem seems to be a little outdated and it doesn't implement a few methods (setTimes being the most important one):
> - setTimes(Path, long, long)
> - copyFromLocalFile(boolean,boolean, Path, Path)
> - copyFromLocalFile(boolean, boolean, Path[], Path)
> - getUsed()
> - deleteOnExit()
> I'm not sure if all of these methods should be wrapped in FilterFileSystem, but given its purpose, I would say the more the better. It would be great to have other people's opinions about this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7510-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 04:37:16 2010
Return-Path: <common-issues-return-7510-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57468 invoked from network); 21 Apr 2010 04:37:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 04:37:15 -0000
Received: (qmail 3787 invoked by uid 500); 21 Apr 2010 04:37:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3560 invoked by uid 500); 21 Apr 2010 04:37:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3543 invoked by uid 99); 21 Apr 2010 04:37:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 04:37:11 +0000
X-ASF-Spam-Status: No, hits=-1321.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 04:37:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L4aoaY022107
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 04:36:50 GMT
Message-ID: <8175298.105341271824610365.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 00:36:50 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6690) FilterFileSystem doesn't overwrite
 setTimes
In-Reply-To: <294275186.4281270693056872.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6690:
------------------------------------

    Attachment: HADOOP-6690.2.patch

Found a couple of more warnings.

> FilterFileSystem doesn't overwrite setTimes
> -------------------------------------------
>
>                 Key: HADOOP-6690
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6690
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6690.0.patch, HADOOP-6690.1.patch, HADOOP-6690.2.patch
>
>
> FilterFileSystem seems to be a little outdated and it doesn't implement a few methods (setTimes being the most important one):
> - setTimes(Path, long, long)
> - copyFromLocalFile(boolean,boolean, Path, Path)
> - copyFromLocalFile(boolean, boolean, Path[], Path)
> - getUsed()
> - deleteOnExit()
> I'm not sure if all of these methods should be wrapped in FilterFileSystem, but given its purpose, I would say the more the better. It would be great to have other people's opinions about this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7512-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 04:45:15 2010
Return-Path: <common-issues-return-7512-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58355 invoked from network); 21 Apr 2010 04:45:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 04:45:15 -0000
Received: (qmail 8392 invoked by uid 500); 21 Apr 2010 04:45:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8278 invoked by uid 500); 21 Apr 2010 04:45:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8268 invoked by uid 99); 21 Apr 2010 04:45:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 04:45:14 +0000
X-ASF-Spam-Status: No, hits=-1321.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 04:45:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L4iqaa022186
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 04:44:52 GMT
Message-ID: <3562419.105531271825092286.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 00:44:52 -0400 (EDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6439) Shuffle deadlocks on wrong number
 of maps
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859191#action_12859191 ] 

Hemanth Yamijala commented on HADOOP-6439:
------------------------------------------

Chaitanya, thanks for running the tests. The patch for trunk is ready for commit. However, I tried to see if the patch applies to branch 0.21 as well, since this is a blocker for that version. The patch failed to apply, unfortunately. Can you please upload a patch for that as well ?

> Shuffle deadlocks on wrong number of maps
> -----------------------------------------
>
>                 Key: HADOOP-6439
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6439
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Owen O'Malley
>            Assignee: V.V.Chaitanya Krishna
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6439-1.patch, HADOOP-6439-2.patch, HADOOP-6439-3.patch, HADOOP-6439-4.patch, HADOOP-6439-5.patch, HADOOP-6439-6.patch, HADOOP-6439-6.patch, HADOOP-6439-7.patch, mr-1252.patch
>
>
> The new shuffle assumes that the number of maps is correct. The new JobSubmitter sets the old value. Something misfires in the middle causing:
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is deprecated. Instead, use mapreduce.job.splitfile
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
> But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the job.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7513-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 04:53:17 2010
Return-Path: <common-issues-return-7513-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59468 invoked from network); 21 Apr 2010 04:53:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 04:53:16 -0000
Received: (qmail 15120 invoked by uid 500); 21 Apr 2010 04:53:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15011 invoked by uid 500); 21 Apr 2010 04:53:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15003 invoked by uid 99); 21 Apr 2010 04:53:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 04:53:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 04:53:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L4qpv2022373
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 04:52:51 GMT
Message-ID: <1915793.106111271825571451.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 00:52:51 -0400 (EDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6439) Shuffle deadlocks on wrong number
 of maps
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859195#action_12859195 ] 

Amareshwari Sriramadasu commented on HADOOP-6439:
-------------------------------------------------

bq. However, I tried to see if the patch applies to branch 0.21 as well, since this is a blocker for that version. The patch failed to apply, unfortunately. Can you please upload a patch for that as well ? 
@Hemanth, I don't think we need to update patches for branch 0.21, as per discussion in general mailing list, branch 0.21 will be cut from trunk itself. 
@Tom, please correct me if I'm wrong.

> Shuffle deadlocks on wrong number of maps
> -----------------------------------------
>
>                 Key: HADOOP-6439
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6439
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Owen O'Malley
>            Assignee: V.V.Chaitanya Krishna
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6439-1.patch, HADOOP-6439-2.patch, HADOOP-6439-3.patch, HADOOP-6439-4.patch, HADOOP-6439-5.patch, HADOOP-6439-6.patch, HADOOP-6439-6.patch, HADOOP-6439-7.patch, mr-1252.patch
>
>
> The new shuffle assumes that the number of maps is correct. The new JobSubmitter sets the old value. Something misfires in the middle causing:
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is deprecated. Instead, use mapreduce.job.splitfile
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
> But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the job.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7514-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 04:55:12 2010
Return-Path: <common-issues-return-7514-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59948 invoked from network); 21 Apr 2010 04:55:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 04:55:12 -0000
Received: (qmail 17427 invoked by uid 500); 21 Apr 2010 04:55:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17355 invoked by uid 500); 21 Apr 2010 04:55:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17347 invoked by uid 99); 21 Apr 2010 04:55:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 04:55:11 +0000
X-ASF-Spam-Status: No, hits=-1321.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 04:55:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L4soAo022414
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 04:54:50 GMT
Message-ID: <5029427.106171271825690031.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 00:54:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6690) FilterFileSystem doesn't overwrite
 setTimes
In-Reply-To: <294275186.4281270693056872.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859196#action_12859196 ] 

Hadoop QA commented on HADOOP-6690:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12442393/HADOOP-6690.2.patch
  against trunk revision 934619.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/46/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/46/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/46/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/46/console

This message is automatically generated.

> FilterFileSystem doesn't overwrite setTimes
> -------------------------------------------
>
>                 Key: HADOOP-6690
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6690
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6690.0.patch, HADOOP-6690.1.patch, HADOOP-6690.2.patch
>
>
> FilterFileSystem seems to be a little outdated and it doesn't implement a few methods (setTimes being the most important one):
> - setTimes(Path, long, long)
> - copyFromLocalFile(boolean,boolean, Path, Path)
> - copyFromLocalFile(boolean, boolean, Path[], Path)
> - getUsed()
> - deleteOnExit()
> I'm not sure if all of these methods should be wrapped in FilterFileSystem, but given its purpose, I would say the more the better. It would be great to have other people's opinions about this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7515-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 04:59:16 2010
Return-Path: <common-issues-return-7515-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61437 invoked from network); 21 Apr 2010 04:59:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 04:59:12 -0000
Received: (qmail 20282 invoked by uid 500); 21 Apr 2010 04:59:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20219 invoked by uid 500); 21 Apr 2010 04:59:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20211 invoked by uid 99); 21 Apr 2010 04:59:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 04:59:12 +0000
X-ASF-Spam-Status: No, hits=-1321.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 04:59:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L4woKr022439
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 04:58:50 GMT
Message-ID: <4412474.106231271825930708.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 00:58:50 -0400 (EDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6690) FilterFileSystem doesn't overwrite
 setTimes
In-Reply-To: <294275186.4281270693056872.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859197#action_12859197 ] 

dhruba borthakur commented on HADOOP-6690:
------------------------------------------

+1 thanks, I will commit this patch.

> FilterFileSystem doesn't overwrite setTimes
> -------------------------------------------
>
>                 Key: HADOOP-6690
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6690
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6690.0.patch, HADOOP-6690.1.patch, HADOOP-6690.2.patch
>
>
> FilterFileSystem seems to be a little outdated and it doesn't implement a few methods (setTimes being the most important one):
> - setTimes(Path, long, long)
> - copyFromLocalFile(boolean,boolean, Path, Path)
> - copyFromLocalFile(boolean, boolean, Path[], Path)
> - getUsed()
> - deleteOnExit()
> I'm not sure if all of these methods should be wrapped in FilterFileSystem, but given its purpose, I would say the more the better. It would be great to have other people's opinions about this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7516-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 04:59:18 2010
Return-Path: <common-issues-return-7516-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61785 invoked from network); 21 Apr 2010 04:59:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 04:59:16 -0000
Received: (qmail 20420 invoked by uid 500); 21 Apr 2010 04:59:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20375 invoked by uid 500); 21 Apr 2010 04:59:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20367 invoked by uid 99); 21 Apr 2010 04:59:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 04:59:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 04:59:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L4wqFD022472
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 04:58:52 GMT
Message-ID: <457280.106341271825932765.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 00:58:52 -0400 (EDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6439) Shuffle deadlocks on wrong number
 of maps
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859198#action_12859198 ] 

Hemanth Yamijala commented on HADOOP-6439:
------------------------------------------

bq. @Hemanth, I don't think we need to update patches for branch 0.21, as per discussion in general mailing list, branch 0.21 will be cut from trunk itself.

Ah. If that's so, then we are good to go. I will wait for Tom's / anyone else's confirmation on this.

> Shuffle deadlocks on wrong number of maps
> -----------------------------------------
>
>                 Key: HADOOP-6439
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6439
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Owen O'Malley
>            Assignee: V.V.Chaitanya Krishna
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6439-1.patch, HADOOP-6439-2.patch, HADOOP-6439-3.patch, HADOOP-6439-4.patch, HADOOP-6439-5.patch, HADOOP-6439-6.patch, HADOOP-6439-6.patch, HADOOP-6439-7.patch, mr-1252.patch
>
>
> The new shuffle assumes that the number of maps is correct. The new JobSubmitter sets the old value. Something misfires in the middle causing:
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is deprecated. Instead, use mapreduce.job.splitfile
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
> But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the job.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7518-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 05:03:18 2010
Return-Path: <common-issues-return-7518-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62613 invoked from network); 21 Apr 2010 05:03:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 05:03:16 -0000
Received: (qmail 22224 invoked by uid 500); 21 Apr 2010 05:03:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22081 invoked by uid 500); 21 Apr 2010 05:03:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22056 invoked by uid 99); 21 Apr 2010 05:03:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 05:03:15 +0000
X-ASF-Spam-Status: No, hits=-1320.8 required=10.0
	tests=ALL_TRUSTED,AWL,FB_GET_MEDS
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 05:03:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L52r0h022578
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 05:02:53 GMT
Message-ID: <10919803.106621271826173060.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 01:02:53 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6719) Missing methods on FilterFs
In-Reply-To: <18354396.102631271811350896.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6719:
------------------------------------

    Attachment: HADOOP-6719.1.patch

Following Eli's suggestions.

> Missing methods on FilterFs
> ---------------------------
>
>                 Key: HADOOP-6719
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6719
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6719.0.patch, HADOOP-6719.1.patch
>
>
> The following methods are not declared on FilterFs(). Some of them are fine, but others like getUri(), getStatistics(), getHomeDirectory() look like they should be there.
> Here is the complete list:
> {code}
> {
>     public FSDataInputStream open(final Path f) { return null; }
>     public void checkPath(Path path) { }
>     public Statistics getStatistics() { return null; }
>     public URI getUri() { return null; }
>     public Path getHomeDirectory() { return null; }
>     public void checkScheme(URI uri, String supportedScheme) { }
>     public String getUriPath(final Path p) { return null; }
>     public void renameInternal(final Path src, final Path dst, boolean overwrite) { }
>     public FsStatus getFsStatus(final Path f) { return null; }
> }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7517-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 05:03:19 2010
Return-Path: <common-issues-return-7517-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62597 invoked from network); 21 Apr 2010 05:03:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 05:03:16 -0000
Received: (qmail 22222 invoked by uid 500); 21 Apr 2010 05:03:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22071 invoked by uid 500); 21 Apr 2010 05:03:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22054 invoked by uid 99); 21 Apr 2010 05:03:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 05:03:14 +0000
X-ASF-Spam-Status: No, hits=-1320.8 required=10.0
	tests=ALL_TRUSTED,AWL,FB_GET_MEDS
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 05:03:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L52r5O022590
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 05:02:53 GMT
Message-ID: <4483014.106661271826173398.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 01:02:53 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6719) Missing methods on FilterFs
In-Reply-To: <18354396.102631271811350896.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6719:
------------------------------------

    Status: Patch Available  (was: Open)

> Missing methods on FilterFs
> ---------------------------
>
>                 Key: HADOOP-6719
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6719
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6719.0.patch, HADOOP-6719.1.patch
>
>
> The following methods are not declared on FilterFs(). Some of them are fine, but others like getUri(), getStatistics(), getHomeDirectory() look like they should be there.
> Here is the complete list:
> {code}
> {
>     public FSDataInputStream open(final Path f) { return null; }
>     public void checkPath(Path path) { }
>     public Statistics getStatistics() { return null; }
>     public URI getUri() { return null; }
>     public Path getHomeDirectory() { return null; }
>     public void checkScheme(URI uri, String supportedScheme) { }
>     public String getUriPath(final Path p) { return null; }
>     public void renameInternal(final Path src, final Path dst, boolean overwrite) { }
>     public FsStatus getFsStatus(final Path f) { return null; }
> }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7519-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 05:03:20 2010
Return-Path: <common-issues-return-7519-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62614 invoked from network); 21 Apr 2010 05:03:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 05:03:16 -0000
Received: (qmail 22362 invoked by uid 500); 21 Apr 2010 05:03:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22084 invoked by uid 500); 21 Apr 2010 05:03:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22072 invoked by uid 99); 21 Apr 2010 05:03:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 05:03:15 +0000
X-ASF-Spam-Status: No, hits=-1998.0 required=10.0
	tests=ALL_TRUSTED,FB_GET_MEDS
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 05:03:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L52oXl022554
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 05:02:51 GMT
Message-ID: <26434377.106541271826170755.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 01:02:50 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6719) Missing methods on FilterFs
In-Reply-To: <18354396.102631271811350896.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6719:
------------------------------------

    Status: Open  (was: Patch Available)

> Missing methods on FilterFs
> ---------------------------
>
>                 Key: HADOOP-6719
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6719
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6719.0.patch, HADOOP-6719.1.patch
>
>
> The following methods are not declared on FilterFs(). Some of them are fine, but others like getUri(), getStatistics(), getHomeDirectory() look like they should be there.
> Here is the complete list:
> {code}
> {
>     public FSDataInputStream open(final Path f) { return null; }
>     public void checkPath(Path path) { }
>     public Statistics getStatistics() { return null; }
>     public URI getUri() { return null; }
>     public Path getHomeDirectory() { return null; }
>     public void checkScheme(URI uri, String supportedScheme) { }
>     public String getUriPath(final Path p) { return null; }
>     public void renameInternal(final Path src, final Path dst, boolean overwrite) { }
>     public FsStatus getFsStatus(final Path f) { return null; }
> }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7520-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 05:33:21 2010
Return-Path: <common-issues-return-7520-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70464 invoked from network); 21 Apr 2010 05:33:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 05:33:21 -0000
Received: (qmail 35136 invoked by uid 500); 21 Apr 2010 05:33:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35018 invoked by uid 500); 21 Apr 2010 05:33:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35010 invoked by uid 99); 21 Apr 2010 05:33:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 05:33:19 +0000
X-ASF-Spam-Status: No, hits=-1998.0 required=10.0
	tests=ALL_TRUSTED,FB_GET_MEDS
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 05:33:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L5WsTV022770
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 05:32:55 GMT
Message-ID: <9766024.106991271827974611.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 01:32:54 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6719) Missing methods on FilterFs
In-Reply-To: <18354396.102631271811350896.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859208#action_12859208 ] 

Hadoop QA commented on HADOOP-6719:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12442395/HADOOP-6719.1.patch
  against trunk revision 934619.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/47/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/47/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/47/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/47/console

This message is automatically generated.

> Missing methods on FilterFs
> ---------------------------
>
>                 Key: HADOOP-6719
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6719
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6719.0.patch, HADOOP-6719.1.patch
>
>
> The following methods are not declared on FilterFs(). Some of them are fine, but others like getUri(), getStatistics(), getHomeDirectory() look like they should be there.
> Here is the complete list:
> {code}
> {
>     public FSDataInputStream open(final Path f) { return null; }
>     public void checkPath(Path path) { }
>     public Statistics getStatistics() { return null; }
>     public URI getUri() { return null; }
>     public Path getHomeDirectory() { return null; }
>     public void checkScheme(URI uri, String supportedScheme) { }
>     public String getUriPath(final Path p) { return null; }
>     public void renameInternal(final Path src, final Path dst, boolean overwrite) { }
>     public FsStatus getFsStatus(final Path f) { return null; }
> }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7521-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 05:57:18 2010
Return-Path: <common-issues-return-7521-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82112 invoked from network); 21 Apr 2010 05:57:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 05:57:18 -0000
Received: (qmail 50682 invoked by uid 500); 21 Apr 2010 05:57:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50481 invoked by uid 500); 21 Apr 2010 05:57:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50459 invoked by uid 99); 21 Apr 2010 05:57:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 05:57:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 05:57:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L5uob1023027
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 05:56:50 GMT
Message-ID: <32356063.107651271829410416.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 01:56:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6658) Exclude Private elements from
 generated Javadoc
In-Reply-To: <995349254.452871269404067173.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859217#action_12859217 ] 

Hadoop QA commented on HADOOP-6658:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12442392/HADOOP-6658.patch
  against trunk revision 934619.

    +1 @author.  The patch does not contain any @author tags.

    +0 tests included.  The patch appears to be a documentation patch that doesn't require tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/468/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/468/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/468/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/468/console

This message is automatically generated.

> Exclude Private elements from generated Javadoc
> -----------------------------------------------
>
>                 Key: HADOOP-6658
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6658
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch
>
>
> Packages, classes and methods that are marked with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation should not appear in the public Javadoc. Developer Javadoc generated using the "javadoc-dev" ant target should continue to generate Javadoc for all program elements.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7522-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 06:31:20 2010
Return-Path: <common-issues-return-7522-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5139 invoked from network); 21 Apr 2010 06:31:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 06:31:19 -0000
Received: (qmail 73701 invoked by uid 500); 21 Apr 2010 06:31:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73519 invoked by uid 500); 21 Apr 2010 06:31:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73493 invoked by uid 99); 21 Apr 2010 06:31:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 06:31:15 +0000
X-ASF-Spam-Status: No, hits=-1998.0 required=10.0
	tests=ALL_TRUSTED,FB_GET_MEDS
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 06:31:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L6UqWr023479
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 06:30:52 GMT
Message-ID: <12884800.108831271831452171.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 02:30:52 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6719) Missing methods on FilterFs
In-Reply-To: <18354396.102631271811350896.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859233#action_12859233 ] 

Rodrigo Schmidt commented on HADOOP-6719:
-----------------------------------------

Implementing checkScheme inside FilterFs has broken 128 unit tests. I'm taking it out and trying again.

> Missing methods on FilterFs
> ---------------------------
>
>                 Key: HADOOP-6719
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6719
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6719.0.patch, HADOOP-6719.1.patch
>
>
> The following methods are not declared on FilterFs(). Some of them are fine, but others like getUri(), getStatistics(), getHomeDirectory() look like they should be there.
> Here is the complete list:
> {code}
> {
>     public FSDataInputStream open(final Path f) { return null; }
>     public void checkPath(Path path) { }
>     public Statistics getStatistics() { return null; }
>     public URI getUri() { return null; }
>     public Path getHomeDirectory() { return null; }
>     public void checkScheme(URI uri, String supportedScheme) { }
>     public String getUriPath(final Path p) { return null; }
>     public void renameInternal(final Path src, final Path dst, boolean overwrite) { }
>     public FsStatus getFsStatus(final Path f) { return null; }
> }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7523-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 06:33:13 2010
Return-Path: <common-issues-return-7523-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5984 invoked from network); 21 Apr 2010 06:33:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 06:33:13 -0000
Received: (qmail 75062 invoked by uid 500); 21 Apr 2010 06:33:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75010 invoked by uid 500); 21 Apr 2010 06:33:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75002 invoked by uid 99); 21 Apr 2010 06:33:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 06:33:12 +0000
X-ASF-Spam-Status: No, hits=-1321.4 required=10.0
	tests=ALL_TRUSTED,AWL,FB_GET_MEDS
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 06:33:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L6Wokd023516
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 06:32:50 GMT
Message-ID: <7613064.108941271831570533.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 02:32:50 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6719) Missing methods on FilterFs
In-Reply-To: <18354396.102631271811350896.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6719:
------------------------------------

    Status: Open  (was: Patch Available)

> Missing methods on FilterFs
> ---------------------------
>
>                 Key: HADOOP-6719
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6719
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6719.0.patch, HADOOP-6719.1.patch, HADOOP-6719.2.patch
>
>
> The following methods are not declared on FilterFs(). Some of them are fine, but others like getUri(), getStatistics(), getHomeDirectory() look like they should be there.
> Here is the complete list:
> {code}
> {
>     public FSDataInputStream open(final Path f) { return null; }
>     public void checkPath(Path path) { }
>     public Statistics getStatistics() { return null; }
>     public URI getUri() { return null; }
>     public Path getHomeDirectory() { return null; }
>     public void checkScheme(URI uri, String supportedScheme) { }
>     public String getUriPath(final Path p) { return null; }
>     public void renameInternal(final Path src, final Path dst, boolean overwrite) { }
>     public FsStatus getFsStatus(final Path f) { return null; }
> }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7524-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 06:33:15 2010
Return-Path: <common-issues-return-7524-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6035 invoked from network); 21 Apr 2010 06:33:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 06:33:15 -0000
Received: (qmail 75176 invoked by uid 500); 21 Apr 2010 06:33:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75145 invoked by uid 500); 21 Apr 2010 06:33:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75137 invoked by uid 99); 21 Apr 2010 06:33:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 06:33:15 +0000
X-ASF-Spam-Status: No, hits=-1998.0 required=10.0
	tests=ALL_TRUSTED,FB_GET_MEDS
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 06:33:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L6Wpee023528
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 06:32:51 GMT
Message-ID: <5116574.108981271831571291.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 02:32:51 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6719) Missing methods on FilterFs
In-Reply-To: <18354396.102631271811350896.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6719:
------------------------------------

    Attachment: HADOOP-6719.2.patch

> Missing methods on FilterFs
> ---------------------------
>
>                 Key: HADOOP-6719
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6719
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6719.0.patch, HADOOP-6719.1.patch, HADOOP-6719.2.patch
>
>
> The following methods are not declared on FilterFs(). Some of them are fine, but others like getUri(), getStatistics(), getHomeDirectory() look like they should be there.
> Here is the complete list:
> {code}
> {
>     public FSDataInputStream open(final Path f) { return null; }
>     public void checkPath(Path path) { }
>     public Statistics getStatistics() { return null; }
>     public URI getUri() { return null; }
>     public Path getHomeDirectory() { return null; }
>     public void checkScheme(URI uri, String supportedScheme) { }
>     public String getUriPath(final Path p) { return null; }
>     public void renameInternal(final Path src, final Path dst, boolean overwrite) { }
>     public FsStatus getFsStatus(final Path f) { return null; }
> }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7525-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 06:33:16 2010
Return-Path: <common-issues-return-7525-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6075 invoked from network); 21 Apr 2010 06:33:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 06:33:15 -0000
Received: (qmail 75316 invoked by uid 500); 21 Apr 2010 06:33:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75278 invoked by uid 500); 21 Apr 2010 06:33:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75264 invoked by uid 99); 21 Apr 2010 06:33:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 06:33:15 +0000
X-ASF-Spam-Status: No, hits=-1998.0 required=10.0
	tests=ALL_TRUSTED,FB_GET_MEDS
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 06:33:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L6Wpfx023540
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 06:32:51 GMT
Message-ID: <28130136.109021271831571658.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 02:32:51 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6719) Missing methods on FilterFs
In-Reply-To: <18354396.102631271811350896.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6719:
------------------------------------

    Status: Patch Available  (was: Open)

Took checkScheme() out of FilterFs

> Missing methods on FilterFs
> ---------------------------
>
>                 Key: HADOOP-6719
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6719
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6719.0.patch, HADOOP-6719.1.patch, HADOOP-6719.2.patch
>
>
> The following methods are not declared on FilterFs(). Some of them are fine, but others like getUri(), getStatistics(), getHomeDirectory() look like they should be there.
> Here is the complete list:
> {code}
> {
>     public FSDataInputStream open(final Path f) { return null; }
>     public void checkPath(Path path) { }
>     public Statistics getStatistics() { return null; }
>     public URI getUri() { return null; }
>     public Path getHomeDirectory() { return null; }
>     public void checkScheme(URI uri, String supportedScheme) { }
>     public String getUriPath(final Path p) { return null; }
>     public void renameInternal(final Path src, final Path dst, boolean overwrite) { }
>     public FsStatus getFsStatus(final Path f) { return null; }
> }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7526-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 09:59:14 2010
Return-Path: <common-issues-return-7526-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63560 invoked from network); 21 Apr 2010 09:59:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 09:59:14 -0000
Received: (qmail 42010 invoked by uid 500); 21 Apr 2010 09:59:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41880 invoked by uid 500); 21 Apr 2010 09:59:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41871 invoked by uid 99); 21 Apr 2010 09:59:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 09:59:11 +0000
X-ASF-Spam-Status: No, hits=-1322.3 required=10.0
	tests=ALL_TRUSTED,AWL,FB_GET_MEDS
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 09:59:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3L9wohS025465
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 09:58:50 GMT
Message-ID: <31867653.111491271843930521.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 05:58:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6719) Missing methods on FilterFs
In-Reply-To: <18354396.102631271811350896.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859284#action_12859284 ] 

Hadoop QA commented on HADOOP-6719:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12442401/HADOOP-6719.2.patch
  against trunk revision 934619.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The patch appears to cause tar ant target to fail.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    -1 contrib tests.  The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/469/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/469/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/469/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/469/console

This message is automatically generated.

> Missing methods on FilterFs
> ---------------------------
>
>                 Key: HADOOP-6719
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6719
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6719.0.patch, HADOOP-6719.1.patch, HADOOP-6719.2.patch
>
>
> The following methods are not declared on FilterFs(). Some of them are fine, but others like getUri(), getStatistics(), getHomeDirectory() look like they should be there.
> Here is the complete list:
> {code}
> {
>     public FSDataInputStream open(final Path f) { return null; }
>     public void checkPath(Path path) { }
>     public Statistics getStatistics() { return null; }
>     public URI getUri() { return null; }
>     public Path getHomeDirectory() { return null; }
>     public void checkScheme(URI uri, String supportedScheme) { }
>     public String getUriPath(final Path p) { return null; }
>     public void renameInternal(final Path src, final Path dst, boolean overwrite) { }
>     public FsStatus getFsStatus(final Path f) { return null; }
> }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7527-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 11:09:15 2010
Return-Path: <common-issues-return-7527-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83733 invoked from network); 21 Apr 2010 11:09:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 11:09:15 -0000
Received: (qmail 64985 invoked by uid 500); 21 Apr 2010 11:09:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64853 invoked by uid 500); 21 Apr 2010 11:09:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64841 invoked by uid 99); 21 Apr 2010 11:09:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 11:09:11 +0000
X-ASF-Spam-Status: No, hits=-1323.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 11:09:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3LB8oSv028894
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 11:08:50 GMT
Message-ID: <13949680.112501271848130766.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 07:08:50 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6715) AccessControlList.toString() returns
 empty string when we set acl to "*"
In-Reply-To: <7375715.82441271756441695.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V updated HADOOP-6715:
------------------------------

    Component/s: security
                 util

I propose we separate the display functionality from {{toString()}} as the latter can potentially be used for other purposes as Ravi mentioned above. So we will
 - fix {{toString()}} addressing the bug related to the wild-card '*'
 - add a new API say {{AccessControlList.getDisplayString()}}. This should return "ALL USERS" for '*' and "NONE" for ' ' (empty space character) and the output of {{toString}} for every other value of ACL.

Thoughts?

> AccessControlList.toString() returns empty string when we set acl to "*"
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6715
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6715
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security, util
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>
> AccessControlList.toString() returns empty string when we set the acl to "\*" and also when we set it to empty(i.e. " "). This is causing wrong values for ACLs shown on jobdetails.jsp and jobdetailshistory.jsp web pages when acls are set to "\*".
> I think AccessControlList.toString() needs to be changed to return "\*" when we set the acl to "\*".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7528-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 11:24:17 2010
Return-Path: <common-issues-return-7528-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86981 invoked from network); 21 Apr 2010 11:24:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 11:24:17 -0000
Received: (qmail 86883 invoked by uid 500); 21 Apr 2010 11:24:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86393 invoked by uid 500); 21 Apr 2010 11:24:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86385 invoked by uid 99); 21 Apr 2010 11:24:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 11:24:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 11:24:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3LBNppl029037
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 11:23:51 GMT
Message-ID: <15654921.112641271849031712.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 07:23:51 -0400 (EDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6715) AccessControlList.toString()
 returns empty string when we set acl to "*"
In-Reply-To: <7375715.82441271756441695.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859303#action_12859303 ] 

Hemanth Yamijala commented on HADOOP-6715:
------------------------------------------

Can one of you help me understand what Ravi means by "save it as a String and later set it to a key in some Configuration object" ? It seems like we need to store the ACL objects in some map, and possibly these need to be reconstructed from a serialized representation (like for task log access, maybe ?) and we are using the key of the map as the String that is thus serialized.

If that's the case, can we serialize the ACL using some representation that stores the actual name and value as separate fields rather than a toString representation, use a hashCode / equals on the ACL object to build a key based on these fields, and use toString for display purposes. This seems more canonical to me (inline with what toString is typically used for).

> AccessControlList.toString() returns empty string when we set acl to "*"
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6715
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6715
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security, util
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>
> AccessControlList.toString() returns empty string when we set the acl to "\*" and also when we set it to empty(i.e. " "). This is causing wrong values for ACLs shown on jobdetails.jsp and jobdetailshistory.jsp web pages when acls are set to "\*".
> I think AccessControlList.toString() needs to be changed to return "\*" when we set the acl to "\*".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7529-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 13:18:19 2010
Return-Path: <common-issues-return-7529-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24637 invoked from network); 21 Apr 2010 13:18:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 13:18:19 -0000
Received: (qmail 72520 invoked by uid 500); 21 Apr 2010 13:18:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72455 invoked by uid 500); 21 Apr 2010 13:18:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72447 invoked by uid 99); 21 Apr 2010 13:18:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 13:18:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 13:18:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3LDHskf000316
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 13:17:54 GMT
Message-ID: <11090946.114501271855874117.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 09:17:54 -0400 (EDT)
From: "Danny Leshem (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6688) FileSystem.delete(...)
 implementations should not throw FileNotFoundException
In-Reply-To: <1780197215.37411270627173494.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859341#action_12859341 ] 

Danny Leshem commented on HADOOP-6688:
--------------------------------------

While going over the code to suggest a patch, I found out this was partially fixed as part of HADOOP-6201: S3FileSystem.delete now catches FileNotFoundException and simply returns false, as opposed to propagating the exception as described in the issue.

The reason I'm calling this a partial fix is that such recursive directory delete will stop (returning false) the moment the above issue happens. The proper behavior, in my opinion, is to 1) continue deleting files 2) return false only if the top-most directory could not be found.

> FileSystem.delete(...) implementations should not throw FileNotFoundException
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-6688
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6688
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, fs/s3
>    Affects Versions: 0.20.2
>         Environment: Amazon EC2/S3
>            Reporter: Danny Leshem
>            Priority: Blocker
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>
> S3FileSystem.delete(Path path, boolean recursive) may fail and throw a FileNotFoundException if a directory is being deleted while at the same time some of its files are deleted in the background.
> This is definitely not the expected behavior of a delete method. If one of the to-be-deleted files is found missing, the method should not fail and simply continue. This is true for the general contract of FileSystem.delete, and also for its various implementations: RawLocalFileSystem (and specifically FileUtil.fullyDelete) exhibits the same problem.
> The fix is to silently catch and ignore FileNotFoundExceptions in delete loops. This can very easily be unit-tested, at least for RawLocalFileSystem.
> The reason this issue bothers me is that the cleanup part of a long (Mahout) MR job inconsistently fails for me, and I think this is the root problem. The log shows:
> {code}
> java.io.FileNotFoundException: s3://S3-BUCKET/tmp/0008E25BF7554CA9/2521362836721872/DistributedMatrix.times.outputVector/_temporary/_attempt_201004061215_0092_r_000002_0/part-00002: No such file or directory.
> 	at org.apache.hadoop.fs.s3.S3FileSystem.getFileStatus(S3FileSystem.java:334)
> 	at org.apache.hadoop.fs.s3.S3FileSystem.listStatus(S3FileSystem.java:193)
> 	at org.apache.hadoop.fs.s3.S3FileSystem.delete(S3FileSystem.java:303)
> 	at org.apache.hadoop.fs.s3.S3FileSystem.delete(S3FileSystem.java:312)
> 	at org.apache.hadoop.mapred.FileOutputCommitter.cleanupJob(FileOutputCommitter.java:64)
> 	at org.apache.hadoop.mapred.OutputCommitter.cleanupJob(OutputCommitter.java:135)
> 	at org.apache.hadoop.mapred.Task.runJobCleanupTask(Task.java:826)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:292)
> 	at org.apache.hadoop.mapred.Child.main(Child.java:170)
> {code}
> (similar errors are displayed for ReduceTask.run)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7530-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 13:46:13 2010
Return-Path: <common-issues-return-7530-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31852 invoked from network); 21 Apr 2010 13:46:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 13:46:13 -0000
Received: (qmail 32212 invoked by uid 500); 21 Apr 2010 13:46:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32106 invoked by uid 500); 21 Apr 2010 13:46:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32098 invoked by uid 99); 21 Apr 2010 13:46:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 13:46:12 +0000
X-ASF-Spam-Status: No, hits=-1324.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 13:46:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3LDjoQW000625
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 13:45:50 GMT
Message-ID: <10840354.115101271857550552.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 09:45:50 -0400 (EDT)
From: "Danny Leshem (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6688) FileSystem.delete(...)
 implementations should not throw FileNotFoundException
In-Reply-To: <1780197215.37411270627173494.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Danny Leshem updated HADOOP-6688:
---------------------------------

    Priority: Minor  (was: Blocker)

> FileSystem.delete(...) implementations should not throw FileNotFoundException
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-6688
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6688
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, fs/s3
>    Affects Versions: 0.20.2
>         Environment: Amazon EC2/S3
>            Reporter: Danny Leshem
>            Priority: Minor
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>
> S3FileSystem.delete(Path path, boolean recursive) may fail and throw a FileNotFoundException if a directory is being deleted while at the same time some of its files are deleted in the background.
> This is definitely not the expected behavior of a delete method. If one of the to-be-deleted files is found missing, the method should not fail and simply continue. This is true for the general contract of FileSystem.delete, and also for its various implementations: RawLocalFileSystem (and specifically FileUtil.fullyDelete) exhibits the same problem.
> The fix is to silently catch and ignore FileNotFoundExceptions in delete loops. This can very easily be unit-tested, at least for RawLocalFileSystem.
> The reason this issue bothers me is that the cleanup part of a long (Mahout) MR job inconsistently fails for me, and I think this is the root problem. The log shows:
> {code}
> java.io.FileNotFoundException: s3://S3-BUCKET/tmp/0008E25BF7554CA9/2521362836721872/DistributedMatrix.times.outputVector/_temporary/_attempt_201004061215_0092_r_000002_0/part-00002: No such file or directory.
> 	at org.apache.hadoop.fs.s3.S3FileSystem.getFileStatus(S3FileSystem.java:334)
> 	at org.apache.hadoop.fs.s3.S3FileSystem.listStatus(S3FileSystem.java:193)
> 	at org.apache.hadoop.fs.s3.S3FileSystem.delete(S3FileSystem.java:303)
> 	at org.apache.hadoop.fs.s3.S3FileSystem.delete(S3FileSystem.java:312)
> 	at org.apache.hadoop.mapred.FileOutputCommitter.cleanupJob(FileOutputCommitter.java:64)
> 	at org.apache.hadoop.mapred.OutputCommitter.cleanupJob(OutputCommitter.java:135)
> 	at org.apache.hadoop.mapred.Task.runJobCleanupTask(Task.java:826)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:292)
> 	at org.apache.hadoop.mapred.Child.main(Child.java:170)
> {code}
> (similar errors are displayed for ReduceTask.run)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7531-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 16:02:23 2010
Return-Path: <common-issues-return-7531-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89060 invoked from network); 21 Apr 2010 16:02:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 16:02:23 -0000
Received: (qmail 3893 invoked by uid 500); 21 Apr 2010 16:02:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3872 invoked by uid 500); 21 Apr 2010 16:02:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3859 invoked by uid 99); 21 Apr 2010 16:02:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 16:02:21 +0000
X-ASF-Spam-Status: No, hits=-1324.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 16:02:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3LG20c4002849
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 16:02:00 GMT
Message-ID: <8756315.117431271865720189.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 12:02:00 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6439) Shuffle deadlocks on wrong number
 of maps
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859407#action_12859407 ] 

Tom White commented on HADOOP-6439:
-----------------------------------

Hemanth/Amareshwari

That's right - I'm going to cut a new 0.21 branch from trunk on April 30, so this patch only needs to be committed to trunk. Thanks!

Tom

> Shuffle deadlocks on wrong number of maps
> -----------------------------------------
>
>                 Key: HADOOP-6439
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6439
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Owen O'Malley
>            Assignee: V.V.Chaitanya Krishna
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6439-1.patch, HADOOP-6439-2.patch, HADOOP-6439-3.patch, HADOOP-6439-4.patch, HADOOP-6439-5.patch, HADOOP-6439-6.patch, HADOOP-6439-6.patch, HADOOP-6439-7.patch, mr-1252.patch
>
>
> The new shuffle assumes that the number of maps is correct. The new JobSubmitter sets the old value. Something misfires in the middle causing:
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is deprecated. Instead, use mapreduce.job.splitfile
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
> But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the job.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7532-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 16:40:13 2010
Return-Path: <common-issues-return-7532-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13581 invoked from network); 21 Apr 2010 16:40:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 16:40:13 -0000
Received: (qmail 51717 invoked by uid 500); 21 Apr 2010 16:40:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51649 invoked by uid 500); 21 Apr 2010 16:40:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51640 invoked by uid 99); 21 Apr 2010 16:40:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 16:40:12 +0000
X-ASF-Spam-Status: No, hits=-1325.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 16:40:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3LGdpa0003141
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 16:39:51 GMT
Message-ID: <24609601.117901271867991151.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 12:39:51 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6658) Exclude Private elements from
 generated Javadoc
In-Reply-To: <995349254.452871269404067173.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6658:
------------------------------

    Attachment: HADOOP-6658.patch

Tests now pass, but annotation processing doesn't seem to be working with the new classes. This patch excludes them from annotation processing, since we don't need fault injection for these tools.

> Exclude Private elements from generated Javadoc
> -----------------------------------------------
>
>                 Key: HADOOP-6658
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6658
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch
>
>
> Packages, classes and methods that are marked with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation should not appear in the public Javadoc. Developer Javadoc generated using the "javadoc-dev" ant target should continue to generate Javadoc for all program elements.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7533-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 16:42:12 2010
Return-Path: <common-issues-return-7533-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14556 invoked from network); 21 Apr 2010 16:42:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 16:42:12 -0000
Received: (qmail 53860 invoked by uid 500); 21 Apr 2010 16:42:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53800 invoked by uid 500); 21 Apr 2010 16:42:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53786 invoked by uid 99); 21 Apr 2010 16:42:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 16:42:12 +0000
X-ASF-Spam-Status: No, hits=-1325.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 16:42:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3LGfomF003153
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 16:41:51 GMT
Message-ID: <25370135.117941271868110369.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 12:41:50 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6658) Exclude Private elements from
 generated Javadoc
In-Reply-To: <995349254.452871269404067173.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6658:
------------------------------

    Status: Open  (was: Patch Available)

> Exclude Private elements from generated Javadoc
> -----------------------------------------------
>
>                 Key: HADOOP-6658
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6658
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch
>
>
> Packages, classes and methods that are marked with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation should not appear in the public Javadoc. Developer Javadoc generated using the "javadoc-dev" ant target should continue to generate Javadoc for all program elements.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7534-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 16:42:13 2010
Return-Path: <common-issues-return-7534-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14666 invoked from network); 21 Apr 2010 16:42:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 16:42:13 -0000
Received: (qmail 54328 invoked by uid 500); 21 Apr 2010 16:42:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54294 invoked by uid 500); 21 Apr 2010 16:42:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54286 invoked by uid 99); 21 Apr 2010 16:42:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 16:42:13 +0000
X-ASF-Spam-Status: No, hits=-1325.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 16:42:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3LGfqrS003171
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 16:41:52 GMT
Message-ID: <20270726.118001271868112534.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 12:41:52 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6658) Exclude Private elements from
 generated Javadoc
In-Reply-To: <995349254.452871269404067173.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6658:
------------------------------

    Status: Patch Available  (was: Open)

> Exclude Private elements from generated Javadoc
> -----------------------------------------------
>
>                 Key: HADOOP-6658
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6658
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch
>
>
> Packages, classes and methods that are marked with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation should not appear in the public Javadoc. Developer Javadoc generated using the "javadoc-dev" ant target should continue to generate Javadoc for all program elements.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7535-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 17:00:16 2010
Return-Path: <common-issues-return-7535-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24212 invoked from network); 21 Apr 2010 17:00:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 17:00:15 -0000
Received: (qmail 77441 invoked by uid 500); 21 Apr 2010 17:00:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77417 invoked by uid 500); 21 Apr 2010 17:00:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77409 invoked by uid 99); 21 Apr 2010 17:00:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 17:00:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 17:00:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3LGxpl6003326
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 16:59:52 GMT
Message-ID: <22332736.118281271869191787.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 12:59:51 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6709) Re-instate deprecated FileSystem
 methods that were removed after 0.20
In-Reply-To: <11903376.154751271374849451.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6709:
------------------------------

    Status: Patch Available  (was: Open)

> Re-instate deprecated FileSystem methods that were removed after 0.20
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6709
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6709
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6709.patch
>
>
> To make the FileSystem API in the 0.21 release (which should be treated as a minor release) compatible with 0.20 the deprecated methods removed in HADOOP-4779 need to be re-instated. They should still be marked as deprecated, however.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7536-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 17:10:13 2010
Return-Path: <common-issues-return-7536-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31300 invoked from network); 21 Apr 2010 17:10:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 17:10:13 -0000
Received: (qmail 91789 invoked by uid 500); 21 Apr 2010 17:10:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91752 invoked by uid 500); 21 Apr 2010 17:10:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91744 invoked by uid 99); 21 Apr 2010 17:10:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 17:10:13 +0000
X-ASF-Spam-Status: No, hits=-1325.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 17:10:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3LH9piI003398
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 17:09:52 GMT
Message-ID: <6089356.118421271869791842.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 13:09:51 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6698) Revert the io.serialization package
 to 0.20.2's api
In-Reply-To: <21261745.7641270969841016.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859432#action_12859432 ] 

Tom White commented on HADOOP-6698:
-----------------------------------

We need to revert HADOOP-6165 to achieve this.

> Revert the io.serialization package to 0.20.2's api
> ---------------------------------------------------
>
>                 Key: HADOOP-6698
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6698
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Owen O'Malley
>            Priority: Blocker
>             Fix For: 0.21.0
>
>
> I have a lot of concern about the usability of the new generic serialization framework. Toward that end, I've filed a jira for improving it. There is resistance to pushing a new API into 0.21 at the last moment, so we should back out the changes rather than introducing a new api in 0.21 and deprecating it in 0.22.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7537-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 17:10:14 2010
Return-Path: <common-issues-return-7537-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31360 invoked from network); 21 Apr 2010 17:10:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 17:10:14 -0000
Received: (qmail 91931 invoked by uid 500); 21 Apr 2010 17:10:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91897 invoked by uid 500); 21 Apr 2010 17:10:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91889 invoked by uid 99); 21 Apr 2010 17:10:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 17:10:14 +0000
X-ASF-Spam-Status: No, hits=-1325.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 17:10:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3LH9qn2003410
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 17:09:52 GMT
Message-ID: <10433631.118461271869792616.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 13:09:52 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6658) Exclude Private elements from
 generated Javadoc
In-Reply-To: <995349254.452871269404067173.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859433#action_12859433 ] 

Hadoop QA commented on HADOOP-6658:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12442450/HADOOP-6658.patch
  against trunk revision 934619.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/470/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/470/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/470/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/470/console

This message is automatically generated.

> Exclude Private elements from generated Javadoc
> -----------------------------------------------
>
>                 Key: HADOOP-6658
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6658
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch
>
>
> Packages, classes and methods that are marked with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation should not appear in the public Javadoc. Developer Javadoc generated using the "javadoc-dev" ant target should continue to generate Javadoc for all program elements.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7538-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 17:18:14 2010
Return-Path: <common-issues-return-7538-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33277 invoked from network); 21 Apr 2010 17:18:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 17:18:14 -0000
Received: (qmail 5259 invoked by uid 500); 21 Apr 2010 17:18:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5132 invoked by uid 500); 21 Apr 2010 17:18:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5124 invoked by uid 99); 21 Apr 2010 17:18:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 17:18:13 +0000
X-ASF-Spam-Status: No, hits=-1325.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 17:18:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3LHHpmf003450
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 17:17:52 GMT
Message-ID: <3141183.118561271870271817.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 13:17:51 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6709) Re-instate deprecated FileSystem
 methods that were removed after 0.20
In-Reply-To: <11903376.154751271374849451.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859436#action_12859436 ] 

Hadoop QA commented on HADOOP-6709:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12441900/HADOOP-6709.patch
  against trunk revision 934619.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/48/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/48/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/48/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/48/console

This message is automatically generated.

> Re-instate deprecated FileSystem methods that were removed after 0.20
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6709
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6709
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6709.patch
>
>
> To make the FileSystem API in the 0.21 release (which should be treated as a minor release) compatible with 0.20 the deprecated methods removed in HADOOP-4779 need to be re-instated. They should still be marked as deprecated, however.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7539-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 17:48:15 2010
Return-Path: <common-issues-return-7539-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48720 invoked from network); 21 Apr 2010 17:48:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 17:48:15 -0000
Received: (qmail 71272 invoked by uid 500); 21 Apr 2010 17:48:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71222 invoked by uid 500); 21 Apr 2010 17:48:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71214 invoked by uid 99); 21 Apr 2010 17:48:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 17:48:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 17:48:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3LHloEW003736
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 17:47:50 GMT
Message-ID: <394180.119291271872070855.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 13:47:50 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6703) Prevent renaming a file, symlink or
 directory to itself
In-Reply-To: <23186149.74901271194550994.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859446#action_12859446 ] 

Suresh Srinivas commented on HADOOP-6703:
-----------------------------------------

Eli, this patch does not apply to trunk. Can you please resubmit the patch.

> Prevent renaming a file, symlink or directory to itself
> -------------------------------------------------------
>
>                 Key: HADOOP-6703
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6703
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6703-1.patch, hadoop-6703-2.patch
>
>
> Per HDFS-1088 let's throw a FileAlreadyExistsException if renaming a file, symlink or directory to itself, or a symlink to the file it links to. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7540-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 18:35:16 2010
Return-Path: <common-issues-return-7540-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86545 invoked from network); 21 Apr 2010 18:35:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 18:35:16 -0000
Received: (qmail 82269 invoked by uid 500); 21 Apr 2010 18:35:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82057 invoked by uid 500); 21 Apr 2010 18:35:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82045 invoked by uid 99); 21 Apr 2010 18:35:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 18:35:15 +0000
X-ASF-Spam-Status: No, hits=-1326.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 18:35:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3LIYqH9004673
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 18:34:53 GMT
Message-ID: <19093190.121691271874892828.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 14:34:52 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6708) New file format for very large
 records
In-Reply-To: <23756292.153431271370052193.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6708?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D128=
59474#action_12859474 ]=20

Doug Cutting commented on HADOOP-6708:
--------------------------------------

> Shortcomings of Avro File Format:
> =E2=80=A2 Data is expected to have a schema

But that schema can be just "bytes".

> =E2=80=A2 No lazy reading API (yet?)

True.  Would this be hard to add?

Also, is it important to add this to Common, or should it rather belong in =
Sqoop?

> New file format for very large records
> --------------------------------------
>
>                 Key: HADOOP-6708
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6708
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>         Attachments: lobfile.pdf
>
>
> A file format that handles multi-gigabyte records efficiently, with lazy =
disk access

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7541-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 19:40:19 2010
Return-Path: <common-issues-return-7541-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33567 invoked from network); 21 Apr 2010 19:40:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 19:40:18 -0000
Received: (qmail 2179 invoked by uid 500); 21 Apr 2010 19:40:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2100 invoked by uid 500); 21 Apr 2010 19:40:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2092 invoked by uid 99); 21 Apr 2010 19:40:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 19:40:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 19:40:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3LJdqAa005258
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 19:39:54 GMT
Message-ID: <867538.122891271878792724.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 15:39:52 -0400 (EDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6439) Shuffle deadlocks on wrong number of
 maps
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala updated HADOOP-6439:
-------------------------------------

           Status: Resolved  (was: Patch Available)
     Hadoop Flags: [Reviewed]
    Fix Version/s:     (was: 0.21.0)
       Resolution: Fixed

I just committed this. Thanks, Chaitanya ! (Also marked the fix for version to 0.22, based on Tom's response).

> Shuffle deadlocks on wrong number of maps
> -----------------------------------------
>
>                 Key: HADOOP-6439
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6439
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Owen O'Malley
>            Assignee: V.V.Chaitanya Krishna
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6439-1.patch, HADOOP-6439-2.patch, HADOOP-6439-3.patch, HADOOP-6439-4.patch, HADOOP-6439-5.patch, HADOOP-6439-6.patch, HADOOP-6439-6.patch, HADOOP-6439-7.patch, mr-1252.patch
>
>
> The new shuffle assumes that the number of maps is correct. The new JobSubmitter sets the old value. Something misfires in the middle causing:
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is deprecated. Instead, use mapreduce.job.splitfile
> 09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
> But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the job.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7542-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 19:50:13 2010
Return-Path: <common-issues-return-7542-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39760 invoked from network); 21 Apr 2010 19:50:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 19:50:12 -0000
Received: (qmail 10232 invoked by uid 500); 21 Apr 2010 19:50:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10129 invoked by uid 500); 21 Apr 2010 19:50:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10120 invoked by uid 99); 21 Apr 2010 19:50:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 19:50:12 +0000
X-ASF-Spam-Status: No, hits=-1326.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 19:50:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3LJnprE005324
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 19:49:51 GMT
Message-ID: <25932594.123041271879391214.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 15:49:51 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6658) Exclude Private elements from
 generated Javadoc
In-Reply-To: <995349254.452871269404067173.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859502#action_12859502 ] 

Tom White commented on HADOOP-6658:
-----------------------------------

I'd like to commit this soon, since it makes it easier to check API compatibility changes in HADOOP-6668 and MAPREDUCE-1623.

> Exclude Private elements from generated Javadoc
> -----------------------------------------------
>
>                 Key: HADOOP-6658
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6658
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch
>
>
> Packages, classes and methods that are marked with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation should not appear in the public Javadoc. Developer Javadoc generated using the "javadoc-dev" ant target should continue to generate Javadoc for all program elements.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7543-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 20:06:14 2010
Return-Path: <common-issues-return-7543-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48632 invoked from network); 21 Apr 2010 20:06:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 20:06:14 -0000
Received: (qmail 28261 invoked by uid 500); 21 Apr 2010 20:06:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28223 invoked by uid 500); 21 Apr 2010 20:06:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28215 invoked by uid 99); 21 Apr 2010 20:06:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 20:06:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 20:06:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3LK5oZl005444
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 20:05:50 GMT
Message-ID: <18252373.123271271880350772.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 16:05:50 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6676) Create a javac plugin that produces
 warnings for programs using Evolving APIs
In-Reply-To: <1601380660.663491270231707182.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859507#action_12859507 ] 

Tom White commented on HADOOP-6676:
-----------------------------------

Another approach would be to write an Ant task, or Maven mojo to enforce this. This is the approach taken by Kohsuke Kawaguchi for Hudson: http://weblogs.java.net/blog/kohsuke/archive/2010/04/09/potd-custom-access-modifier, http://github.com/kohsuke/access-modifier.

> Create a javac plugin that produces warnings for programs using Evolving APIs
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-6676
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6676
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Tom White
>
> Such a feature would be useful to enforce MAPREDUCE-1637. It might also be useful for users who wish to check their programs don't use Evolving APIs. 
> It could be implemented using an annotation processor: http://java.sun.com/javase/6/docs/technotes/tools/solaris/javac.html#processing

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7544-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 21:25:13 2010
Return-Path: <common-issues-return-7544-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94992 invoked from network); 21 Apr 2010 21:25:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 21:25:12 -0000
Received: (qmail 39552 invoked by uid 500); 21 Apr 2010 21:25:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39487 invoked by uid 500); 21 Apr 2010 21:25:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39479 invoked by uid 99); 21 Apr 2010 21:25:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 21:25:11 +0000
X-ASF-Spam-Status: No, hits=-1325.9 required=10.0
	tests=ALL_TRUSTED,AWL,FB_GET_MEDS
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 21:25:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3LLOouQ006027
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 21:24:50 GMT
Message-ID: <19074730.124271271885090214.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 17:24:50 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6719) Missing methods on FilterFs
In-Reply-To: <18354396.102631271811350896.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6719:
------------------------------------

    Status: Open  (was: Patch Available)

> Missing methods on FilterFs
> ---------------------------
>
>                 Key: HADOOP-6719
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6719
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6719.0.patch, HADOOP-6719.1.patch, HADOOP-6719.2.patch
>
>
> The following methods are not declared on FilterFs(). Some of them are fine, but others like getUri(), getStatistics(), getHomeDirectory() look like they should be there.
> Here is the complete list:
> {code}
> {
>     public FSDataInputStream open(final Path f) { return null; }
>     public void checkPath(Path path) { }
>     public Statistics getStatistics() { return null; }
>     public URI getUri() { return null; }
>     public Path getHomeDirectory() { return null; }
>     public void checkScheme(URI uri, String supportedScheme) { }
>     public String getUriPath(final Path p) { return null; }
>     public void renameInternal(final Path src, final Path dst, boolean overwrite) { }
>     public FsStatus getFsStatus(final Path f) { return null; }
> }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7545-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 21:27:13 2010
Return-Path: <common-issues-return-7545-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96080 invoked from network); 21 Apr 2010 21:27:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 21:27:13 -0000
Received: (qmail 41136 invoked by uid 500); 21 Apr 2010 21:27:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41025 invoked by uid 500); 21 Apr 2010 21:27:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41017 invoked by uid 99); 21 Apr 2010 21:27:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 21:27:12 +0000
X-ASF-Spam-Status: No, hits=-1325.9 required=10.0
	tests=ALL_TRUSTED,AWL,FB_GET_MEDS
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 21:27:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3LLQpS0006057
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 21:26:51 GMT
Message-ID: <28500938.124371271885211841.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 17:26:51 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6719) Missing methods on FilterFs
In-Reply-To: <18354396.102631271811350896.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6719:
------------------------------------

    Status: Patch Available  (was: Open)

> Missing methods on FilterFs
> ---------------------------
>
>                 Key: HADOOP-6719
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6719
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6719.0.patch, HADOOP-6719.1.patch, HADOOP-6719.2.patch, HADOOP-6719.3.patch
>
>
> The following methods are not declared on FilterFs(). Some of them are fine, but others like getUri(), getStatistics(), getHomeDirectory() look like they should be there.
> Here is the complete list:
> {code}
> {
>     public FSDataInputStream open(final Path f) { return null; }
>     public void checkPath(Path path) { }
>     public Statistics getStatistics() { return null; }
>     public URI getUri() { return null; }
>     public Path getHomeDirectory() { return null; }
>     public void checkScheme(URI uri, String supportedScheme) { }
>     public String getUriPath(final Path p) { return null; }
>     public void renameInternal(final Path src, final Path dst, boolean overwrite) { }
>     public FsStatus getFsStatus(final Path f) { return null; }
> }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7546-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 21:27:15 2010
Return-Path: <common-issues-return-7546-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96137 invoked from network); 21 Apr 2010 21:27:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 21:27:15 -0000
Received: (qmail 41204 invoked by uid 500); 21 Apr 2010 21:27:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41167 invoked by uid 500); 21 Apr 2010 21:27:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41159 invoked by uid 99); 21 Apr 2010 21:27:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 21:27:14 +0000
X-ASF-Spam-Status: No, hits=-1998.0 required=10.0
	tests=ALL_TRUSTED,FB_GET_MEDS
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 21:27:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3LLQocc006039
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 21:26:50 GMT
Message-ID: <3685864.124311271885210502.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 17:26:50 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6719) Missing methods on FilterFs
In-Reply-To: <18354396.102631271811350896.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6719:
------------------------------------

    Attachment: HADOOP-6719.3.patch

Eclipse had added a bad import to the file. Submitting a new patch.

> Missing methods on FilterFs
> ---------------------------
>
>                 Key: HADOOP-6719
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6719
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6719.0.patch, HADOOP-6719.1.patch, HADOOP-6719.2.patch, HADOOP-6719.3.patch
>
>
> The following methods are not declared on FilterFs(). Some of them are fine, but others like getUri(), getStatistics(), getHomeDirectory() look like they should be there.
> Here is the complete list:
> {code}
> {
>     public FSDataInputStream open(final Path f) { return null; }
>     public void checkPath(Path path) { }
>     public Statistics getStatistics() { return null; }
>     public URI getUri() { return null; }
>     public Path getHomeDirectory() { return null; }
>     public void checkScheme(URI uri, String supportedScheme) { }
>     public String getUriPath(final Path p) { return null; }
>     public void renameInternal(final Path src, final Path dst, boolean overwrite) { }
>     public FsStatus getFsStatus(final Path f) { return null; }
> }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7547-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 21:53:12 2010
Return-Path: <common-issues-return-7547-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13216 invoked from network); 21 Apr 2010 21:53:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 21:53:12 -0000
Received: (qmail 78215 invoked by uid 500); 21 Apr 2010 21:53:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78186 invoked by uid 500); 21 Apr 2010 21:53:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78178 invoked by uid 99); 21 Apr 2010 21:53:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 21:53:11 +0000
X-ASF-Spam-Status: No, hits=-1326.0 required=10.0
	tests=ALL_TRUSTED,AWL,FB_GET_MEDS
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 21:53:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3LLqoNN006332
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 21:52:50 GMT
Message-ID: <32158828.124931271886770141.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 17:52:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6719) Missing methods on FilterFs
In-Reply-To: <18354396.102631271811350896.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859540#action_12859540 ] 

Hadoop QA commented on HADOOP-6719:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12442481/HADOOP-6719.3.patch
  against trunk revision 936463.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/471/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/471/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/471/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/471/console

This message is automatically generated.

> Missing methods on FilterFs
> ---------------------------
>
>                 Key: HADOOP-6719
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6719
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6719.0.patch, HADOOP-6719.1.patch, HADOOP-6719.2.patch, HADOOP-6719.3.patch
>
>
> The following methods are not declared on FilterFs(). Some of them are fine, but others like getUri(), getStatistics(), getHomeDirectory() look like they should be there.
> Here is the complete list:
> {code}
> {
>     public FSDataInputStream open(final Path f) { return null; }
>     public void checkPath(Path path) { }
>     public Statistics getStatistics() { return null; }
>     public URI getUri() { return null; }
>     public Path getHomeDirectory() { return null; }
>     public void checkScheme(URI uri, String supportedScheme) { }
>     public String getUriPath(final Path p) { return null; }
>     public void renameInternal(final Path src, final Path dst, boolean overwrite) { }
>     public FsStatus getFsStatus(final Path f) { return null; }
> }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7548-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 22:15:13 2010
Return-Path: <common-issues-return-7548-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25550 invoked from network); 21 Apr 2010 22:15:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 22:15:13 -0000
Received: (qmail 3314 invoked by uid 500); 21 Apr 2010 22:15:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3249 invoked by uid 500); 21 Apr 2010 22:15:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3241 invoked by uid 99); 21 Apr 2010 22:15:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 22:15:13 +0000
X-ASF-Spam-Status: No, hits=-1327.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 22:15:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3LMEpu9006611
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 22:14:52 GMT
Message-ID: <21252162.125531271888091709.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 18:14:51 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6685) Change the generic serialization
 framework API to use serialization-specific bytes instead of
 Map<String,String> for configuration
In-Reply-To: <328977938.18921270574733630.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859550#action_12859550 ] 

Doug Cutting commented on HADOOP-6685:
--------------------------------------

Different serializations currently use common keys to share parts of their configuration data.  For example, many name a class that implements the serialization.  Several name a class that represents the instances.  The code that accesses these properties is shared, either through a common base class or a utility class.

Binary blobs can also be considerably more fragile than a Map<String,String>.  If we wish the versions of clients and servers to be able to vary, then blobs stored in configurations must be version-tolerant, i.e., written and read with a fairly sophisticated serialization system.

> Change the generic serialization framework API to use serialization-specific bytes instead of Map<String,String> for configuration
> ----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6685
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6685
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>
> Currently, the generic serialization framework uses Map<String,String> for the serialization specific configuration. Since this data is really internal to the specific serialization, I think we should change it to be an opaque binary blob. This will simplify the interface for defining specific serializations for different contexts (MAPREDUCE-1462). It will also move us toward having serialized objects for Mappers, Reducers, etc (MAPREDUCE-1183).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7549-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 22:21:14 2010
Return-Path: <common-issues-return-7549-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28537 invoked from network); 21 Apr 2010 22:21:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 22:21:13 -0000
Received: (qmail 10116 invoked by uid 500); 21 Apr 2010 22:21:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10046 invoked by uid 500); 21 Apr 2010 22:21:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10038 invoked by uid 99); 21 Apr 2010 22:21:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 22:21:13 +0000
X-ASF-Spam-Status: No, hits=-1327.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 22:21:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3LMKqnj006747
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 22:20:52 GMT
Message-ID: <10828799.125911271888452063.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 18:20:52 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6698) Revert the io.serialization package
 to 0.20.2's api
In-Reply-To: <21261745.7641270969841016.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859555#action_12859555 ] 

Doug Cutting commented on HADOOP-6698:
--------------------------------------

This issue is fundamentally about backing out changes related to HADOOP-1126, since you've rejected that patch.  In that light, HADOOP-6420, HADOOP-6438 and HADOOP-6443 should perhaps also be backed out.

> Revert the io.serialization package to 0.20.2's api
> ---------------------------------------------------
>
>                 Key: HADOOP-6698
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6698
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Owen O'Malley
>            Priority: Blocker
>             Fix For: 0.21.0
>
>
> I have a lot of concern about the usability of the new generic serialization framework. Toward that end, I've filed a jira for improving it. There is resistance to pushing a new API into 0.21 at the last moment, so we should back out the changes rather than introducing a new api in 0.21 and deprecating it in 0.22.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7550-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 22:21:15 2010
Return-Path: <common-issues-return-7550-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28585 invoked from network); 21 Apr 2010 22:21:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 22:21:14 -0000
Received: (qmail 10226 invoked by uid 500); 21 Apr 2010 22:21:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10183 invoked by uid 500); 21 Apr 2010 22:21:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10175 invoked by uid 99); 21 Apr 2010 22:21:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 22:21:14 +0000
X-ASF-Spam-Status: No, hits=-1327.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 22:21:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3LMKrSA006780
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 22:20:53 GMT
Message-ID: <2216479.126021271888453547.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 18:20:53 -0400 (EDT)
From: "Dmytro Molkov (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6713) The RPC server Listener thread is a
 scalability bottleneck
In-Reply-To: <20387500.4981271656489920.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dmytro Molkov updated HADOOP-6713:
----------------------------------

    Attachment: HADOOP-6713.patch

Please have a look. We did the same change on hadoop-0.20 and ran it for a day under heavy write and read load on the test cluster.
The unittest included just reruns the TestRCP with multiple number of reader threads

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.20.0, 0.20.1, 0.20.2
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7551-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 22:29:13 2010
Return-Path: <common-issues-return-7551-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34413 invoked from network); 21 Apr 2010 22:29:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 22:29:13 -0000
Received: (qmail 21844 invoked by uid 500); 21 Apr 2010 22:29:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21734 invoked by uid 500); 21 Apr 2010 22:29:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21723 invoked by uid 99); 21 Apr 2010 22:29:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 22:29:13 +0000
X-ASF-Spam-Status: No, hits=-1326.4 required=10.0
	tests=ALL_TRUSTED,AWL,FB_GET_MEDS
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 22:29:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3LMSoe3006864
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 22:28:51 GMT
Message-ID: <29127622.126151271888930649.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 18:28:50 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6719) Missing methods on FilterFs
In-Reply-To: <18354396.102631271811350896.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859559#action_12859559 ] 

Eli Collins commented on HADOOP-6719:
-------------------------------------

+1 Looks good.

> Missing methods on FilterFs
> ---------------------------
>
>                 Key: HADOOP-6719
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6719
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6719.0.patch, HADOOP-6719.1.patch, HADOOP-6719.2.patch, HADOOP-6719.3.patch
>
>
> The following methods are not declared on FilterFs(). Some of them are fine, but others like getUri(), getStatistics(), getHomeDirectory() look like they should be there.
> Here is the complete list:
> {code}
> {
>     public FSDataInputStream open(final Path f) { return null; }
>     public void checkPath(Path path) { }
>     public Statistics getStatistics() { return null; }
>     public URI getUri() { return null; }
>     public Path getHomeDirectory() { return null; }
>     public void checkScheme(URI uri, String supportedScheme) { }
>     public String getUriPath(final Path p) { return null; }
>     public void renameInternal(final Path src, final Path dst, boolean overwrite) { }
>     public FsStatus getFsStatus(final Path f) { return null; }
> }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7552-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 22:29:17 2010
Return-Path: <common-issues-return-7552-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34664 invoked from network); 21 Apr 2010 22:29:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 22:29:15 -0000
Received: (qmail 22336 invoked by uid 500); 21 Apr 2010 22:29:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22308 invoked by uid 500); 21 Apr 2010 22:29:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22300 invoked by uid 99); 21 Apr 2010 22:29:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 22:29:15 +0000
X-ASF-Spam-Status: No, hits=-1327.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 22:29:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3LMSrox006912
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 22:28:53 GMT
Message-ID: <22402204.126311271888933638.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 18:28:53 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6698) Revert the io.serialization package
 to 0.20.2's api
In-Reply-To: <21261745.7641270969841016.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859562#action_12859562 ] 

Tom White commented on HADOOP-6698:
-----------------------------------

Doug, I think you mean MAPREDUCE-1126 (not HADOOP-1126). Also HADOOP-6323 needs backing out.

> Revert the io.serialization package to 0.20.2's api
> ---------------------------------------------------
>
>                 Key: HADOOP-6698
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6698
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Owen O'Malley
>            Priority: Blocker
>             Fix For: 0.21.0
>
>
> I have a lot of concern about the usability of the new generic serialization framework. Toward that end, I've filed a jira for improving it. There is resistance to pushing a new API into 0.21 at the last moment, so we should back out the changes rather than introducing a new api in 0.21 and deprecating it in 0.22.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7553-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 21 23:03:14 2010
Return-Path: <common-issues-return-7553-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51880 invoked from network); 21 Apr 2010 23:03:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Apr 2010 23:03:14 -0000
Received: (qmail 62340 invoked by uid 500); 21 Apr 2010 23:03:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62311 invoked by uid 500); 21 Apr 2010 23:03:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62302 invoked by uid 99); 21 Apr 2010 23:03:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 23:03:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Apr 2010 23:03:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3LN2o4O007255
	for <common-issues@hadoop.apache.org>; Wed, 21 Apr 2010 23:02:50 GMT
Message-ID: <13963626.126891271890970356.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 19:02:50 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6716) System won't start in non-secure mode
 when kerb5.conf (edu.mit.kerberos on Mac) is not present
In-Reply-To: <19456845.94831271789690859.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6716:
-----------------------------------

    Attachment: HADOOP-6716-BP20-3.patch

for previous version, not for commit

> System won't start in non-secure mode when kerb5.conf (edu.mit.kerberos on Mac) is not present
> ----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6716
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6716
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>         Attachments: HADOOP-6716-BP20-3.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7554-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 00:02:21 2010
Return-Path: <common-issues-return-7554-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81236 invoked from network); 22 Apr 2010 00:02:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 00:02:21 -0000
Received: (qmail 6255 invoked by uid 500); 22 Apr 2010 00:02:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6225 invoked by uid 500); 22 Apr 2010 00:02:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6217 invoked by uid 99); 22 Apr 2010 00:02:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 00:02:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 00:02:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3M01vAI007659
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 00:01:57 GMT
Message-ID: <19139700.127831271894516991.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 20:01:56 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6698) Revert the io.serialization package
 to 0.20.2's api
In-Reply-To: <21261745.7641270969841016.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6698:
------------------------------

    Attachment: HADOOP-6698.patch

Here's a patch which reverts all these changes.

> Revert the io.serialization package to 0.20.2's api
> ---------------------------------------------------
>
>                 Key: HADOOP-6698
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6698
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Owen O'Malley
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6698.patch
>
>
> I have a lot of concern about the usability of the new generic serialization framework. Toward that end, I've filed a jira for improving it. There is resistance to pushing a new API into 0.21 at the last moment, so we should back out the changes rather than introducing a new api in 0.21 and deprecating it in 0.22.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7555-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 00:02:22 2010
Return-Path: <common-issues-return-7555-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81278 invoked from network); 22 Apr 2010 00:02:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 00:02:22 -0000
Received: (qmail 6385 invoked by uid 500); 22 Apr 2010 00:02:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6358 invoked by uid 500); 22 Apr 2010 00:02:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6350 invoked by uid 99); 22 Apr 2010 00:02:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 00:02:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 00:02:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3M01wi9007692
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 00:01:58 GMT
Message-ID: <25808015.127941271894518253.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 20:01:58 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6698) Revert the io.serialization package
 to 0.20.2's api
In-Reply-To: <21261745.7641270969841016.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6698:
------------------------------

    Status: Patch Available  (was: Open)

> Revert the io.serialization package to 0.20.2's api
> ---------------------------------------------------
>
>                 Key: HADOOP-6698
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6698
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Owen O'Malley
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6698.patch
>
>
> I have a lot of concern about the usability of the new generic serialization framework. Toward that end, I've filed a jira for improving it. There is resistance to pushing a new API into 0.21 at the last moment, so we should back out the changes rather than introducing a new api in 0.21 and deprecating it in 0.22.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7556-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 00:08:16 2010
Return-Path: <common-issues-return-7556-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84307 invoked from network); 22 Apr 2010 00:08:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 00:08:16 -0000
Received: (qmail 10505 invoked by uid 500); 22 Apr 2010 00:08:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10467 invoked by uid 500); 22 Apr 2010 00:08:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10459 invoked by uid 99); 22 Apr 2010 00:08:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 00:08:15 +0000
X-ASF-Spam-Status: No, hits=-1327.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 00:08:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3M07sN7007775
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 00:07:54 GMT
Message-ID: <18309610.128141271894874307.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 20:07:54 -0400 (EDT)
From: "Dmytro Molkov (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6713) The RPC server Listener thread is a
 scalability bottleneck
In-Reply-To: <20387500.4981271656489920.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dmytro Molkov updated HADOOP-6713:
----------------------------------

               Status: Patch Available  (was: Open)
    Affects Version/s: 0.21.0
                           (was: 0.20.0)
                           (was: 0.20.1)
                           (was: 0.20.2)

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7557-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 00:28:15 2010
Return-Path: <common-issues-return-7557-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3614 invoked from network); 22 Apr 2010 00:28:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 00:28:13 -0000
Received: (qmail 28931 invoked by uid 500); 22 Apr 2010 00:28:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28893 invoked by uid 500); 22 Apr 2010 00:28:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28885 invoked by uid 99); 22 Apr 2010 00:28:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 00:28:12 +0000
X-ASF-Spam-Status: No, hits=-1328.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 00:28:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3M0RpfM007970
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 00:27:51 GMT
Message-ID: <17221086.128401271896071834.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 20:27:51 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6716) System won't start in non-secure
 mode when kerb5.conf (edu.mit.kerberos on Mac) is not present
In-Reply-To: <19456845.94831271789690859.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik reassigned HADOOP-6716:
--------------------------------------

    Assignee: Boris Shkolnik

> System won't start in non-secure mode when kerb5.conf (edu.mit.kerberos on Mac) is not present
> ----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6716
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6716
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6716-BP20-3.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7558-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 00:28:18 2010
Return-Path: <common-issues-return-7558-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4090 invoked from network); 22 Apr 2010 00:28:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 00:28:18 -0000
Received: (qmail 29449 invoked by uid 500); 22 Apr 2010 00:28:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29411 invoked by uid 500); 22 Apr 2010 00:28:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29403 invoked by uid 99); 22 Apr 2010 00:28:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 00:28:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 00:28:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3M0Rsxr008009
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 00:27:54 GMT
Message-ID: <28124216.128531271896074072.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 20:27:54 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6713) The RPC server Listener thread is a
 scalability bottleneck
In-Reply-To: <20387500.4981271656489920.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859599#action_12859599 ] 

Hadoop QA commented on HADOOP-6713:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12442483/HADOOP-6713.patch
  against trunk revision 936463.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/49/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/49/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/49/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/49/console

This message is automatically generated.

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7559-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 00:30:18 2010
Return-Path: <common-issues-return-7559-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5147 invoked from network); 22 Apr 2010 00:30:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 00:30:18 -0000
Received: (qmail 31013 invoked by uid 500); 22 Apr 2010 00:30:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30949 invoked by uid 500); 22 Apr 2010 00:30:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30941 invoked by uid 99); 22 Apr 2010 00:30:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 00:30:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 00:30:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3M0TsmV008078
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 00:29:54 GMT
Message-ID: <19620632.128671271896194065.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 20:29:54 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6698) Revert the io.serialization package
 to 0.20.2's api
In-Reply-To: <21261745.7641270969841016.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859601#action_12859601 ] 

Hadoop QA commented on HADOOP-6698:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12442493/HADOOP-6698.patch
  against trunk revision 936463.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 15 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs.  The patch appears to introduce 2 new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/472/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/472/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/472/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/472/console

This message is automatically generated.

> Revert the io.serialization package to 0.20.2's api
> ---------------------------------------------------
>
>                 Key: HADOOP-6698
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6698
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Owen O'Malley
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6698.patch
>
>
> I have a lot of concern about the usability of the new generic serialization framework. Toward that end, I've filed a jira for improving it. There is resistance to pushing a new API into 0.21 at the last moment, so we should back out the changes rather than introducing a new api in 0.21 and deprecating it in 0.22.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7560-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 01:16:12 2010
Return-Path: <common-issues-return-7560-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16296 invoked from network); 22 Apr 2010 01:16:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 01:16:12 -0000
Received: (qmail 70064 invoked by uid 500); 22 Apr 2010 01:16:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69762 invoked by uid 500); 22 Apr 2010 01:16:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69754 invoked by uid 99); 22 Apr 2010 01:16:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 01:16:11 +0000
X-ASF-Spam-Status: No, hits=-1328.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 01:16:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3M1Fo5a008938
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 01:15:50 GMT
Message-ID: <10283915.129251271898950060.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 21:15:50 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6716) System won't start in non-secure
 mode when kerb5.conf (edu.mit.kerberos on Mac) is not present
In-Reply-To: <19456845.94831271789690859.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859616#action_12859616 ] 

Konstantin Boudnik commented on HADOOP-6716:
--------------------------------------------

+ 1 patch looks good. One small nit:
{noformat}
+        throw new IllegalArgumentException("Can't get Kerberos configuration",ke);
{noformat}
put a whitespace after comma. However, it might be considered like a formatting change from the previous state of the code. So, it's up to you.

> System won't start in non-secure mode when kerb5.conf (edu.mit.kerberos on Mac) is not present
> ----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6716
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6716
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6716-BP20-3.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7561-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 03:11:18 2010
Return-Path: <common-issues-return-7561-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32812 invoked from network); 22 Apr 2010 03:11:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 03:11:18 -0000
Received: (qmail 44813 invoked by uid 500); 22 Apr 2010 03:11:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44697 invoked by uid 500); 22 Apr 2010 03:11:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44689 invoked by uid 99); 22 Apr 2010 03:11:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 03:11:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 03:11:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3M3AoWp009752
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 03:10:51 GMT
Message-ID: <16084720.130061271905850584.JavaMail.jira@thor>
Date: Wed, 21 Apr 2010 23:10:50 -0400 (EDT)
From: "Jeff Hammerbacher (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6714) FsShell 'hadoop fs -text' does not
 support compression codecs
In-Reply-To: <19191606.14561271692729693.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859633#action_12859633 ] 

Jeff Hammerbacher commented on HADOOP-6714:
-------------------------------------------

Sorry, linked an issue in the wrong window and can't figure out how to delete the link. This issue is in no way related to HDFS-1051.

> FsShell 'hadoop fs -text' does not support compression codecs
> -------------------------------------------------------------
>
>                 Key: HADOOP-6714
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6714
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Patrick Angeles
>         Attachments: HADOOP-6714.patch
>
>
> Currently, 'hadoop fs -text myfile' looks at the first few magic bytes of a file to determine whether it is gzip compressed or a sequence file. This means 'fs -text' cannot properly decode .deflate or .bz2 files (or other codecs specified via configuration).
> It should be fairly straightforward to add support for other codecs by checking the file extension against the CompressionCodecFactory to retrieve an appropriate Codec.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7562-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 06:27:17 2010
Return-Path: <common-issues-return-7562-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61564 invoked from network); 22 Apr 2010 06:27:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 06:27:17 -0000
Received: (qmail 44477 invoked by uid 500); 22 Apr 2010 06:27:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44359 invoked by uid 500); 22 Apr 2010 06:27:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44346 invoked by uid 99); 22 Apr 2010 06:27:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 06:27:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 06:27:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3M6QoDu011386
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 06:26:50 GMT
Message-ID: <23283278.132111271917610036.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 02:26:50 -0400 (EDT)
From: "Ravi Gummadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6715) AccessControlList.toString()
 returns empty string when we set acl to "*"
In-Reply-To: <7375715.82441271756441695.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859680#action_12859680 ] 

Ravi Gummadi commented on HADOOP-6715:
--------------------------------------

Right Hemanth. People wouldn't expect toString() to give serialized string of object. We can have our own messages in it.
Will modify AccessControlList.toString() itself to give "ALL USERS" and "NONE" for "\*" and empty acl cases.

> AccessControlList.toString() returns empty string when we set acl to "*"
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6715
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6715
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security, util
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>
> AccessControlList.toString() returns empty string when we set the acl to "\*" and also when we set it to empty(i.e. " "). This is causing wrong values for ACLs shown on jobdetails.jsp and jobdetailshistory.jsp web pages when acls are set to "\*".
> I think AccessControlList.toString() needs to be changed to return "\*" when we set the acl to "\*".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7563-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 09:30:17 2010
Return-Path: <common-issues-return-7563-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1316 invoked from network); 22 Apr 2010 09:30:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 09:30:17 -0000
Received: (qmail 78034 invoked by uid 500); 22 Apr 2010 09:30:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77090 invoked by uid 500); 22 Apr 2010 09:30:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76961 invoked by uid 99); 22 Apr 2010 09:30:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 09:30:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 09:30:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3M9ToL9013553
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 09:29:50 GMT
Message-ID: <20025730.135761271928590041.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 05:29:50 -0400 (EDT)
From: "Subramaniam Krishnan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6720)  'Killed' jobs and 'Failed' jobs
 should be displayed seperately in JT UI
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

 'Killed' jobs and 'Failed' jobs should be displayed seperately in JT UI
------------------------------------------------------------------------

                 Key: HADOOP-6720
                 URL: https://issues.apache.org/jira/browse/HADOOP-6720
             Project: Hadoop Common
          Issue Type: New Feature
         Environment: all
            Reporter: Subramaniam Krishnan
            Assignee: Subramaniam Krishnan
            Priority: Critical
             Fix For: 0.19.0


It is not possible, via APIs, to know if a job has been killed, only that has not been successful.


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7564-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 09:34:16 2010
Return-Path: <common-issues-return-7564-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2413 invoked from network); 22 Apr 2010 09:34:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 09:34:16 -0000
Received: (qmail 83737 invoked by uid 500); 22 Apr 2010 09:34:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83646 invoked by uid 500); 22 Apr 2010 09:34:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83637 invoked by uid 99); 22 Apr 2010 09:34:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 09:34:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 09:34:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3M9XncC013585
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 09:33:50 GMT
Message-ID: <11455475.135791271928829851.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 05:33:49 -0400 (EDT)
From: "Subramaniam Krishnan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6720)  'Killed' jobs and 'Failed' jobs
 should be displayed seperately in JobTracker UI
In-Reply-To: <20025730.135761271928590041.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Subramaniam Krishnan updated HADOOP-6720:
-----------------------------------------

              Summary:  'Killed' jobs and 'Failed' jobs should be displayed seperately in JobTracker UI  (was:  'Killed' jobs and 'Failed' jobs should be displayed seperately in JT UI)
         Hadoop Flags:   (was: [Reviewed])
           Issue Type: Bug  (was: New Feature)
             Assignee:     (was: Subramaniam Krishnan)
        Fix Version/s: 0.20.2
                           (was: 0.19.0)
    Affects Version/s: 0.20.1
             Priority: Major  (was: Critical)
          Description: 
The JobTracker UI shows both Failed/Killed Jobs as Failed. The Killed job status has been separated from Failed as part of HADOOP-3924, so the UI needs to be updated to reflect the same.


  was:
It is not possible, via APIs, to know if a job has been killed, only that has not been successful.



>  'Killed' jobs and 'Failed' jobs should be displayed seperately in JobTracker UI
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6720
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6720
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.1
>         Environment: all
>            Reporter: Subramaniam Krishnan
>             Fix For: 0.20.2
>
>
> The JobTracker UI shows both Failed/Killed Jobs as Failed. The Killed job status has been separated from Failed as part of HADOOP-3924, so the UI needs to be updated to reflect the same.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7565-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 15:40:19 2010
Return-Path: <common-issues-return-7565-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20911 invoked from network); 22 Apr 2010 15:40:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 15:40:18 -0000
Received: (qmail 67620 invoked by uid 500); 22 Apr 2010 15:40:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67413 invoked by uid 500); 22 Apr 2010 15:40:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67077 invoked by uid 99); 22 Apr 2010 15:40:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 15:40:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 15:40:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3MFdrrq016694
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 15:39:53 GMT
Message-ID: <30086789.139961271950793498.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 11:39:53 -0400 (EDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6713) The RPC server Listener thread is a
 scalability bottleneck
In-Reply-To: <20387500.4981271656489920.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859848#action_12859848 ] 

dhruba borthakur commented on HADOOP-6713:
------------------------------------------

+1 code looks good.

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7566-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 17:04:15 2010
Return-Path: <common-issues-return-7566-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54941 invoked from network); 22 Apr 2010 17:04:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 17:04:12 -0000
Received: (qmail 48197 invoked by uid 500); 22 Apr 2010 17:04:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47739 invoked by uid 500); 22 Apr 2010 17:04:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47729 invoked by uid 99); 22 Apr 2010 17:04:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 17:04:11 +0000
X-ASF-Spam-Status: No, hits=-1332.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 17:04:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3MH3oPY017772
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 17:03:51 GMT
Message-ID: <29523020.142331271955830599.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 13:03:50 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6716) System won't start in non-secure
 mode when kerb5.conf (edu.mit.kerberos on Mac) is not present
In-Reply-To: <19456845.94831271789690859.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859901#action_12859901 ] 

Boris Shkolnik commented on HADOOP-6716:
----------------------------------------

will do in trunk commit.

> System won't start in non-secure mode when kerb5.conf (edu.mit.kerberos on Mac) is not present
> ----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6716
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6716
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6716-BP20-3.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7567-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 17:10:16 2010
Return-Path: <common-issues-return-7567-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58222 invoked from network); 22 Apr 2010 17:10:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 17:10:16 -0000
Received: (qmail 60885 invoked by uid 500); 22 Apr 2010 17:10:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60856 invoked by uid 500); 22 Apr 2010 17:10:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60848 invoked by uid 99); 22 Apr 2010 17:10:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 17:10:15 +0000
X-ASF-Spam-Status: No, hits=-1332.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 17:10:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3MH9sBb017873
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 17:09:54 GMT
Message-ID: <13322819.142611271956194604.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 13:09:54 -0400 (EDT)
From: "Koji Noguchi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6713) The RPC server Listener thread is a
 scalability bottleneck
In-Reply-To: <20387500.4981271656489920.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859905#action_12859905 ] 

Koji Noguchi commented on HADOOP-6713:
--------------------------------------

Nice. Any performance numbers ?

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7568-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 18:18:17 2010
Return-Path: <common-issues-return-7568-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99885 invoked from network); 22 Apr 2010 18:18:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 18:18:17 -0000
Received: (qmail 68617 invoked by uid 500); 22 Apr 2010 18:18:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68591 invoked by uid 500); 22 Apr 2010 18:18:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68583 invoked by uid 99); 22 Apr 2010 18:18:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 18:18:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 18:18:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3MIHqws018667
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 18:17:53 GMT
Message-ID: <14958301.144581271960272442.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 14:17:52 -0400 (EDT)
From: "Dmytro Molkov (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6713) The RPC server Listener thread is a
 scalability bottleneck
In-Reply-To: <20387500.4981271656489920.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859935#action_12859935 ] 

Dmytro Molkov commented on HADOOP-6713:
---------------------------------------

We were running performance test on our test cluster.
The test itself is creating a tree of directories with files on the leafs in a depths first search fashion: there is a root and we create N directories in the root directory for the test, each mapper then starts in one of those directories and creates its own subtree with files on the leafs.

Then there is a read job that for each mapper does ls on the directory, chooses random element in ls, if it is a directory then repeat if it is a file then do read on the file. The files are 4K in size so the read time is small and we are mostly hitting the namenode with this job.

We were running the branch that had this fix and it also had read write locks for namenode instead of synchronized sections.

The version without fixes could only get namenode to use 175% cpu. With fixes in place we were using 750% cpu for read only load (when the second job was running on its own and 550% for read-write load when two jobs were running in parallel. 

In the read-write mode the ration of reads to writes was 8:1 (800 read clients vs 100 write clients).

We are not putting the read-write locks in production in this iteration, seems we feel like we need to do more testing on it. As soon as I have some results for the branch with this fix only I will post my findings here.

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7569-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 18:48:12 2010
Return-Path: <common-issues-return-7569-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23050 invoked from network); 22 Apr 2010 18:48:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 18:48:12 -0000
Received: (qmail 22595 invoked by uid 500); 22 Apr 2010 18:48:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22564 invoked by uid 500); 22 Apr 2010 18:48:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22556 invoked by uid 99); 22 Apr 2010 18:48:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 18:48:11 +0000
X-ASF-Spam-Status: No, hits=-1333.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 18:48:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3MIlpif018980
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 18:47:51 GMT
Message-ID: <13254833.145371271962071069.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 14:47:51 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6721) Create a new logo for Hadoop security
 related uses
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Create a new logo for Hadoop security related uses
--------------------------------------------------

                 Key: HADOOP-6721
                 URL: https://issues.apache.org/jira/browse/HADOOP-6721
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Owen O'Malley
            Assignee: Owen O'Malley


We've created a new logo for Hadoop security topics that we wanted to share with the community.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7570-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 19:24:15 2010
Return-Path: <common-issues-return-7570-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45024 invoked from network); 22 Apr 2010 19:24:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 19:24:14 -0000
Received: (qmail 99440 invoked by uid 500); 22 Apr 2010 19:24:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99392 invoked by uid 500); 22 Apr 2010 19:24:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99381 invoked by uid 99); 22 Apr 2010 19:24:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 19:24:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 19:24:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3MJNoIB019333
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 19:23:50 GMT
Message-ID: <32027091.146181271964230347.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 15:23:50 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6674) Performance Improvement in Secure RPC
In-Reply-To: <375873020.645441270163787224.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6674:
--------------------------------

    Attachment: HADOOP-6674-y20.1.bugfix.patch

Attaching a patch fixing a bug. This patch is on top of the earlier patch - http://tinyurl.com/2beueg9.

> Performance Improvement in Secure RPC
> -------------------------------------
>
>                 Key: HADOOP-6674
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6674
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.patch
>
>
> This jira introduces two performance improvements in Sasl RPC
> 1. Setting of Sasl 'quality of protection' to authentication only.
> 2. Addition of BufferedOutputStream underneath SaslOutputStream.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7571-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 19:36:11 2010
Return-Path: <common-issues-return-7571-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50798 invoked from network); 22 Apr 2010 19:36:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 19:36:11 -0000
Received: (qmail 15521 invoked by uid 500); 22 Apr 2010 19:36:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15490 invoked by uid 500); 22 Apr 2010 19:36:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15481 invoked by uid 99); 22 Apr 2010 19:36:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 19:36:11 +0000
X-ASF-Spam-Status: No, hits=-1333.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 19:36:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3MJZnLL019417
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 19:35:49 GMT
Message-ID: <22005334.146291271964949915.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 15:35:49 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6718) Client does not close connection when
 an exception happens during SASL negotiation
In-Reply-To: <15471232.100441271806549356.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6718:
--------------------------------

    Attachment: 6718-bp20.patch

Attaching a patch for Y20S. Not for commit.

> Client does not close connection when an exception happens during SASL negotiation
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6718
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6718
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6718-bp20.patch
>
>
> setupSaslConnection in the RPC client might fail to successfully set up a sasl connection (e.g. if the principal is wrongly configured). It throws an exception back to the caller (setupIOstreams). setupIOstreams marks the connection as closed but it doesn't really close the socket. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7572-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 20:20:25 2010
Return-Path: <common-issues-return-7572-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74536 invoked from network); 22 Apr 2010 20:20:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 20:20:25 -0000
Received: (qmail 92218 invoked by uid 500); 22 Apr 2010 20:20:25 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92176 invoked by uid 500); 22 Apr 2010 20:20:25 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92168 invoked by uid 99); 22 Apr 2010 20:20:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 20:20:25 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 20:20:22 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3MKK0a2019962
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 20:20:00 GMT
Message-ID: <33423659.147481271967600460.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 16:20:00 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6659) Switch RPC to use Avro
In-Reply-To: <387448273.476441269469290584.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859992#action_12859992 ] 

Doug Cutting commented on HADOOP-6659:
--------------------------------------

Sanjay> Moving to Avro IDL will change many data types in our servers and cannot be pluggable.
Sanjay> I don't think we can split the HDFS protocols as they use many common data types (block-id, block-token, etc). 

I agree that it may not be easy to do this incrementally, but I think it would be much easier than trying to do it in a branch.  For example, existing datatypes like Block that are used in many protocols could be made to implement Avro's SpecificRecord interface so that both IDL and reflect-based code can use them.  Then we could more easily consider porting protocols one-by-one.  Once we've ported all protocols that use Block, then we could, as a single patch, replace it with an IDL-generated version.

Eric> If we can get to agreement that the current RPC remains the default and that IDL work happens in a branch and that only after we have a reasonably complete backwards compat solution do we change to AVRO RPC by default, then I think we are good.

If Avro reflect-based RPC passes all tests, including performance, reliability, etc., why wouldn't we switch to it?  Even without using an IDL, this would let client and server versions differ.  We certainly don't want to encourage independent 3rd party implementations of protocols until we're happy that the protocols are what we intend to support long-term, but I don't yet see a reason not to switch once Avro-based RPC is functionally equivalent.

Eric> My concern is not with the already committed plugability, but with the change of the default RPC and incremental change over of the protocols. Once that work starts, we are committed to finishing it and we can not do that in the timeframe of the next release IMO.

I don't follow.  If we proceed incrementally, thoroughly testing changes before committing them, then we should be able to release at any time, no?

Eric> Would we keep the plugable API once we make the transition?

I see no reason to keep it.

Folks are welcome to start IDL-based branch(es) if they prefer to operate that way.  In that case, I will cease work on support for an incremental approach, as it would be redundant.  I fear that, since such branches would be long-lived that they'll prove very painful to maintain, especially if they make substantial changes to protocols.  We should not prohibit trunk changes to the HDFS and MapReduce protocols.

> Switch RPC to use Avro
> ----------------------
>
>                 Key: HADOOP-6659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6659
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Doug Cutting
>
> This is an umbrella issue for moving HDFS and MapReduce RPC to use Avro.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7573-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 20:20:26 2010
Return-Path: <common-issues-return-7573-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74608 invoked from network); 22 Apr 2010 20:20:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 20:20:26 -0000
Received: (qmail 92371 invoked by uid 500); 22 Apr 2010 20:20:26 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92344 invoked by uid 500); 22 Apr 2010 20:20:26 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92336 invoked by uid 99); 22 Apr 2010 20:20:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 20:20:26 +0000
X-ASF-Spam-Status: No, hits=-1333.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 20:20:25 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3MKK5Ta020056
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 20:20:05 GMT
Message-ID: <13154941.147791271967605388.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 16:20:05 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6698) Revert the io.serialization package
 to 0.20.2's api
In-Reply-To: <21261745.7641270969841016.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6698:
------------------------------

    Status: Open  (was: Patch Available)

> Revert the io.serialization package to 0.20.2's api
> ---------------------------------------------------
>
>                 Key: HADOOP-6698
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6698
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Owen O'Malley
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6698.patch, HADOOP-6698.patch
>
>
> I have a lot of concern about the usability of the new generic serialization framework. Toward that end, I've filed a jira for improving it. There is resistance to pushing a new API into 0.21 at the last moment, so we should back out the changes rather than introducing a new api in 0.21 and deprecating it in 0.22.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7574-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 20:20:28 2010
Return-Path: <common-issues-return-7574-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74719 invoked from network); 22 Apr 2010 20:20:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 20:20:27 -0000
Received: (qmail 92610 invoked by uid 500); 22 Apr 2010 20:20:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92585 invoked by uid 500); 22 Apr 2010 20:20:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92577 invoked by uid 99); 22 Apr 2010 20:20:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 20:20:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 20:20:25 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3MKK3hJ020023
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 20:20:03 GMT
Message-ID: <15837320.147681271967603921.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 16:20:03 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6698) Revert the io.serialization package
 to 0.20.2's api
In-Reply-To: <21261745.7641270969841016.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6698:
------------------------------

    Attachment: HADOOP-6698.patch

> Revert the io.serialization package to 0.20.2's api
> ---------------------------------------------------
>
>                 Key: HADOOP-6698
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6698
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Owen O'Malley
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6698.patch, HADOOP-6698.patch
>
>
> I have a lot of concern about the usability of the new generic serialization framework. Toward that end, I've filed a jira for improving it. There is resistance to pushing a new API into 0.21 at the last moment, so we should back out the changes rather than introducing a new api in 0.21 and deprecating it in 0.22.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7575-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 20:20:28 2010
Return-Path: <common-issues-return-7575-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74735 invoked from network); 22 Apr 2010 20:20:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 20:20:28 -0000
Received: (qmail 92743 invoked by uid 500); 22 Apr 2010 20:20:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92718 invoked by uid 500); 22 Apr 2010 20:20:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92710 invoked by uid 99); 22 Apr 2010 20:20:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 20:20:27 +0000
X-ASF-Spam-Status: No, hits=-1333.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 20:20:26 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3MKK6tU020089
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 20:20:06 GMT
Message-ID: <9814883.147901271967606639.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 16:20:06 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6698) Revert the io.serialization package
 to 0.20.2's api
In-Reply-To: <21261745.7641270969841016.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6698:
------------------------------

    Status: Patch Available  (was: Open)

> Revert the io.serialization package to 0.20.2's api
> ---------------------------------------------------
>
>                 Key: HADOOP-6698
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6698
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Owen O'Malley
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6698.patch, HADOOP-6698.patch
>
>
> I have a lot of concern about the usability of the new generic serialization framework. Toward that end, I've filed a jira for improving it. There is resistance to pushing a new API into 0.21 at the last moment, so we should back out the changes rather than introducing a new api in 0.21 and deprecating it in 0.22.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7576-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 20:37:19 2010
Return-Path: <common-issues-return-7576-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84057 invoked from network); 22 Apr 2010 20:37:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 20:37:17 -0000
Received: (qmail 19305 invoked by uid 500); 22 Apr 2010 20:37:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19283 invoked by uid 500); 22 Apr 2010 20:37:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19275 invoked by uid 99); 22 Apr 2010 20:37:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 20:37:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 20:37:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3MKarSF020306
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 20:36:53 GMT
Message-ID: <10337309.148251271968613581.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 16:36:53 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6698) Revert the io.serialization package
 to 0.20.2's api
In-Reply-To: <21261745.7641270969841016.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12859998#action_12859998 ] 

Hadoop QA commented on HADOOP-6698:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12442620/HADOOP-6698.patch
  against trunk revision 936463.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 15 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/473/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/473/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/473/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/473/console

This message is automatically generated.

> Revert the io.serialization package to 0.20.2's api
> ---------------------------------------------------
>
>                 Key: HADOOP-6698
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6698
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Owen O'Malley
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6698.patch, HADOOP-6698.patch
>
>
> I have a lot of concern about the usability of the new generic serialization framework. Toward that end, I've filed a jira for improving it. There is resistance to pushing a new API into 0.21 at the last moment, so we should back out the changes rather than introducing a new api in 0.21 and deprecating it in 0.22.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7578-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 20:51:16 2010
Return-Path: <common-issues-return-7578-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92185 invoked from network); 22 Apr 2010 20:51:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 20:51:14 -0000
Received: (qmail 44860 invoked by uid 500); 22 Apr 2010 20:51:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44822 invoked by uid 500); 22 Apr 2010 20:51:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44791 invoked by uid 99); 22 Apr 2010 20:51:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 20:51:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 20:51:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3MKootq020418
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 20:50:50 GMT
Message-ID: <25286480.148431271969450271.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 16:50:50 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6658) Exclude Private elements from
 generated Javadoc
In-Reply-To: <995349254.452871269404067173.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6658:
------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

I've just committed this.

> Exclude Private elements from generated Javadoc
> -----------------------------------------------
>
>                 Key: HADOOP-6658
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6658
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch, HADOOP-6658.patch
>
>
> Packages, classes and methods that are marked with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation should not appear in the public Javadoc. Developer Javadoc generated using the "javadoc-dev" ant target should continue to generate Javadoc for all program elements.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7577-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 20:51:17 2010
Return-Path: <common-issues-return-7577-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92175 invoked from network); 22 Apr 2010 20:51:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 20:51:14 -0000
Received: (qmail 44443 invoked by uid 500); 22 Apr 2010 20:51:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44396 invoked by uid 500); 22 Apr 2010 20:51:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44225 invoked by uid 99); 22 Apr 2010 20:51:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 20:51:13 +0000
X-ASF-Spam-Status: No, hits=-1334.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 20:51:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3MKoqi4020430
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 20:50:52 GMT
Message-ID: <28929584.148471271969452644.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 16:50:52 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6718) Client does not close connection
 when an exception happens during SASL negotiation
In-Reply-To: <15471232.100441271806549356.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860005#action_12860005 ] 

Jakob Homan commented on HADOOP-6718:
-------------------------------------

Seems reasonable. +1.

> Client does not close connection when an exception happens during SASL negotiation
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6718
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6718
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6718-bp20.patch
>
>
> setupSaslConnection in the RPC client might fail to successfully set up a sasl connection (e.g. if the principal is wrongly configured). It throws an exception back to the caller (setupIOstreams). setupIOstreams marks the connection as closed but it doesn't really close the socket. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7579-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 21:15:13 2010
Return-Path: <common-issues-return-7579-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5586 invoked from network); 22 Apr 2010 21:15:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 21:15:13 -0000
Received: (qmail 78579 invoked by uid 500); 22 Apr 2010 21:15:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78547 invoked by uid 500); 22 Apr 2010 21:15:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78539 invoked by uid 99); 22 Apr 2010 21:15:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 21:15:13 +0000
X-ASF-Spam-Status: No, hits=-1334.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 21:15:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3MLEpMk020707
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 21:14:51 GMT
Message-ID: <33482592.148961271970891438.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 17:14:51 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860017#action_12860017 ] 

Tom White commented on HADOOP-6668:
-----------------------------------

To help evaluate this work, I've patched JDiff to add an {{-incompatible}} option, which only shows incompatible differences. (In fact it shows some false positives too. That is, changes that are in fact compatible.) Using this patched JDiff (patch is at https://sourceforge.net/tracker/?func=detail&aid=2990626&group_id=37160&atid=419055#), I have produced a diff between 0.20 and trunk with both the patch for this issue, and HADOOP-6709 applied (I backported the annotations from HADOOP-5073, and the patch for this issue to 0.20):

http://people.apache.org/~tomwhite/HADOOP-6668/docs/jdiff/changes.html

There are no significant differences here, so I believe that this patch is ready to be committed. (Note that the changes in org.apache.hadoop.io will disappear once HADOOP-6698 is in.)

I'd appreciate it if someone could review this change. Thanks.

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7580-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 21:19:16 2010
Return-Path: <common-issues-return-7580-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7472 invoked from network); 22 Apr 2010 21:19:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 21:19:16 -0000
Received: (qmail 85889 invoked by uid 500); 22 Apr 2010 21:19:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85857 invoked by uid 500); 22 Apr 2010 21:19:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85849 invoked by uid 99); 22 Apr 2010 21:19:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 21:19:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 21:19:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3MLIqYH020767
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 21:18:52 GMT
Message-ID: <9795980.149101271971132311.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 17:18:52 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6721) Create a new logo for Hadoop security
 related uses
In-Reply-To: <13254833.145371271962071069.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6721:
----------------------------------

    Attachment: hadoop-security-logo.jpg

> Create a new logo for Hadoop security related uses
> --------------------------------------------------
>
>                 Key: HADOOP-6721
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6721
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: hadoop-security-logo.jpg
>
>
> We've created a new logo for Hadoop security topics that we wanted to share with the community.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7581-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 21:25:13 2010
Return-Path: <common-issues-return-7581-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10384 invoked from network); 22 Apr 2010 21:25:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 21:25:13 -0000
Received: (qmail 92591 invoked by uid 500); 22 Apr 2010 21:25:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92484 invoked by uid 500); 22 Apr 2010 21:25:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92476 invoked by uid 99); 22 Apr 2010 21:25:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 21:25:13 +0000
X-ASF-Spam-Status: No, hits=-1334.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 21:25:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3MLOpbW020851
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 21:24:51 GMT
Message-ID: <17651772.149321271971491537.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 17:24:51 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6721) Create a new logo for Hadoop security
 related uses
In-Reply-To: <13254833.145371271962071069.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6721:
----------------------------------

    Attachment: hadoop-helmet.jpg

Hairong's daughter drew up an armored elephant that is cute.

> Create a new logo for Hadoop security related uses
> --------------------------------------------------
>
>                 Key: HADOOP-6721
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6721
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: hadoop-helmet.jpg, hadoop-security-logo.jpg
>
>
> We've created a new logo for Hadoop security topics that we wanted to share with the community.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7582-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 21:33:15 2010
Return-Path: <common-issues-return-7582-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14185 invoked from network); 22 Apr 2010 21:33:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 21:33:15 -0000
Received: (qmail 3798 invoked by uid 500); 22 Apr 2010 21:33:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3689 invoked by uid 500); 22 Apr 2010 21:33:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3681 invoked by uid 99); 22 Apr 2010 21:33:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 21:33:15 +0000
X-ASF-Spam-Status: No, hits=-1334.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 21:33:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3MLWrFA020957
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 21:32:54 GMT
Message-ID: <12617116.149581271971973974.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 17:32:53 -0400 (EDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6708) New file format for very large
 records
In-Reply-To: <23756292.153431271370052193.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860026#action_12860026 ] 

Aaron Kimball commented on HADOOP-6708:
---------------------------------------

| But that schema can be just "bytes".

Of course. And Sqoop would use such a file in this manner. But in building a feature into Avro's file format, would it be possible to include a {{getRecordAsByteStream()}} / {{getRecordAsCharStream()}} that makes sense in the context of a file where many underlying schemata don't necessarily make sense in a byte-wise form?

| True. Would this be hard to add?

:smile: You'd be in a better position than I to comment on that.

As for the common/sqoop question: I have written prototype code that provides this file format in Sqoop itself, but I haven't pushed it out yet. If it's infeasible to add this to Hadoop common, then I'll continue to polish that prototype code and just include it directly in Sqoop. However in discussion with other engineers, it's come up that such a very-large-record format may have broader applications than just Sqoop. Furthermore, people will want to use inputformats, etc., that operate over these records. Folks could link against Sqoop's jar to get to these file formats and InputFormat classes, but that's One More Dependency that they may not want to manage. Given that these records are just byte or character streams, it doesn't seem necessary to restrict it to just Sqoop. Also the ability to expand the scope of an existing format to encapsulate these records could lower maintenance costs over time for clients who are storing data in this format.


> New file format for very large records
> --------------------------------------
>
>                 Key: HADOOP-6708
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6708
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>         Attachments: lobfile.pdf
>
>
> A file format that handles multi-gigabyte records efficiently, with lazy disk access

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7584-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 21:33:16 2010
Return-Path: <common-issues-return-7584-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14244 invoked from network); 22 Apr 2010 21:33:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 21:33:16 -0000
Received: (qmail 3993 invoked by uid 500); 22 Apr 2010 21:33:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3956 invoked by uid 500); 22 Apr 2010 21:33:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3869 invoked by uid 99); 22 Apr 2010 21:33:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 21:33:15 +0000
X-ASF-Spam-Status: No, hits=-1334.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 21:33:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3MLWsw7020979
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 21:32:54 GMT
Message-ID: <33218351.149651271971974710.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 17:32:54 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6721) Create a new logo for Hadoop
 security related uses
In-Reply-To: <13254833.145371271962071069.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860027#action_12860027 ] 

Owen O'Malley commented on HADOOP-6721:
---------------------------------------

I'll start a thread on general to call for an approval vote from the PMC. Clearly, I'm proposing hadoop-security-logo.jpg and not Hairong's daughter's picture.

> Create a new logo for Hadoop security related uses
> --------------------------------------------------
>
>                 Key: HADOOP-6721
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6721
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: hadoop-helmet.jpg, hadoop-security-logo.jpg
>
>
> We've created a new logo for Hadoop security topics that we wanted to share with the community.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7583-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 21:33:16 2010
Return-Path: <common-issues-return-7583-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14223 invoked from network); 22 Apr 2010 21:33:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 21:33:15 -0000
Received: (qmail 3873 invoked by uid 500); 22 Apr 2010 21:33:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3823 invoked by uid 500); 22 Apr 2010 21:33:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3815 invoked by uid 99); 22 Apr 2010 21:33:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 21:33:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 21:33:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3MLWpU5020915
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 21:32:52 GMT
Message-ID: <16330637.149441271971971645.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 17:32:51 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6721) Create a new logo for Hadoop
 security related uses
In-Reply-To: <13254833.145371271962071069.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860025#action_12860025 ] 

Eli Collins commented on HADOOP-6721:
-------------------------------------

+1

> Create a new logo for Hadoop security related uses
> --------------------------------------------------
>
>                 Key: HADOOP-6721
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6721
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: hadoop-helmet.jpg, hadoop-security-logo.jpg
>
>
> We've created a new logo for Hadoop security topics that we wanted to share with the community.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7585-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 21:37:13 2010
Return-Path: <common-issues-return-7585-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16332 invoked from network); 22 Apr 2010 21:37:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 21:37:13 -0000
Received: (qmail 12255 invoked by uid 500); 22 Apr 2010 21:37:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12229 invoked by uid 500); 22 Apr 2010 21:37:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12221 invoked by uid 99); 22 Apr 2010 21:37:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 21:37:13 +0000
X-ASF-Spam-Status: No, hits=-1334.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 21:37:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3MLapIm021029
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 21:36:51 GMT
Message-ID: <19808990.149731271972211466.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 17:36:51 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6721) Create a new logo for Hadoop
 security related uses
In-Reply-To: <13254833.145371271962071069.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860028#action_12860028 ] 

Allen Wittenauer commented on HADOOP-6721:
------------------------------------------

I dunno, Owen.  The hadoop-security-logo seems awfully polished to properly represent Hadoop. ;)

> Create a new logo for Hadoop security related uses
> --------------------------------------------------
>
>                 Key: HADOOP-6721
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6721
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: hadoop-helmet.jpg, hadoop-security-logo.jpg
>
>
> We've created a new logo for Hadoop security topics that we wanted to share with the community.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7586-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 21:43:16 2010
Return-Path: <common-issues-return-7586-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19403 invoked from network); 22 Apr 2010 21:43:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 21:43:16 -0000
Received: (qmail 21904 invoked by uid 500); 22 Apr 2010 21:43:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21866 invoked by uid 500); 22 Apr 2010 21:43:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21858 invoked by uid 99); 22 Apr 2010 21:43:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 21:43:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 21:43:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3MLgqBp021094
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 21:42:52 GMT
Message-ID: <18773525.149891271972572277.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 17:42:52 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6721) Create a new logo for Hadoop
 security related uses
In-Reply-To: <13254833.145371271962071069.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860033#action_12860033 ] 

Doug Cutting commented on HADOOP-6721:
--------------------------------------

Where would this be used?  Do you expect there to be a separate secure releases for Hadoop?  As Hadoop becomes secure, wouldn't we simply continue to use the existing logo?

> Create a new logo for Hadoop security related uses
> --------------------------------------------------
>
>                 Key: HADOOP-6721
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6721
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: hadoop-helmet.jpg, hadoop-security-logo.jpg
>
>
> We've created a new logo for Hadoop security topics that we wanted to share with the community.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7587-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 22:43:12 2010
Return-Path: <common-issues-return-7587-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51244 invoked from network); 22 Apr 2010 22:43:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 22:43:11 -0000
Received: (qmail 77408 invoked by uid 500); 22 Apr 2010 22:43:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77379 invoked by uid 500); 22 Apr 2010 22:43:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77371 invoked by uid 99); 22 Apr 2010 22:43:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 22:43:11 +0000
X-ASF-Spam-Status: No, hits=-1334.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 22:43:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3MMgopD026537
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 22:42:50 GMT
Message-ID: <24753137.150651271976170499.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 18:42:50 -0400 (EDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6690) FilterFileSystem doesn't overwrite
 setTimes
In-Reply-To: <294275186.4281270693056872.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-6690:
-------------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

I just committed this. Thanks Rodrigo!

> FilterFileSystem doesn't overwrite setTimes
> -------------------------------------------
>
>                 Key: HADOOP-6690
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6690
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6690.0.patch, HADOOP-6690.1.patch, HADOOP-6690.2.patch
>
>
> FilterFileSystem seems to be a little outdated and it doesn't implement a few methods (setTimes being the most important one):
> - setTimes(Path, long, long)
> - copyFromLocalFile(boolean,boolean, Path, Path)
> - copyFromLocalFile(boolean, boolean, Path[], Path)
> - getUsed()
> - deleteOnExit()
> I'm not sure if all of these methods should be wrapped in FilterFileSystem, but given its purpose, I would say the more the better. It would be great to have other people's opinions about this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7588-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 22:49:14 2010
Return-Path: <common-issues-return-7588-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55000 invoked from network); 22 Apr 2010 22:49:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 22:49:14 -0000
Received: (qmail 86808 invoked by uid 500); 22 Apr 2010 22:49:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86773 invoked by uid 500); 22 Apr 2010 22:49:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86765 invoked by uid 99); 22 Apr 2010 22:49:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 22:49:14 +0000
X-ASF-Spam-Status: No, hits=-1333.5 required=10.0
	tests=ALL_TRUSTED,AWL,FB_GET_MEDS
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 22:49:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3MMmrPJ026858
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 22:48:53 GMT
Message-ID: <22041265.150871271976533634.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 18:48:53 -0400 (EDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6719) Missing methods on FilterFs
In-Reply-To: <18354396.102631271811350896.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-6719:
-------------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

I just committed this. Thanks Rodrigo!

> Missing methods on FilterFs
> ---------------------------
>
>                 Key: HADOOP-6719
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6719
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6719.0.patch, HADOOP-6719.1.patch, HADOOP-6719.2.patch, HADOOP-6719.3.patch
>
>
> The following methods are not declared on FilterFs(). Some of them are fine, but others like getUri(), getStatistics(), getHomeDirectory() look like they should be there.
> Here is the complete list:
> {code}
> {
>     public FSDataInputStream open(final Path f) { return null; }
>     public void checkPath(Path path) { }
>     public Statistics getStatistics() { return null; }
>     public URI getUri() { return null; }
>     public Path getHomeDirectory() { return null; }
>     public void checkScheme(URI uri, String supportedScheme) { }
>     public String getUriPath(final Path p) { return null; }
>     public void renameInternal(final Path src, final Path dst, boolean overwrite) { }
>     public FsStatus getFsStatus(final Path f) { return null; }
> }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7589-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 22:49:17 2010
Return-Path: <common-issues-return-7589-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55122 invoked from network); 22 Apr 2010 22:49:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 22:49:17 -0000
Received: (qmail 87081 invoked by uid 500); 22 Apr 2010 22:49:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87032 invoked by uid 500); 22 Apr 2010 22:49:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87024 invoked by uid 99); 22 Apr 2010 22:49:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 22:49:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 22:49:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3MMmrZw026846
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 22:48:53 GMT
Message-ID: <17611730.150831271976533242.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 18:48:53 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6674) Performance Improvement in Secure
 RPC
In-Reply-To: <375873020.645441270163787224.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860051#action_12860051 ] 

Owen O'Malley commented on HADOOP-6674:
---------------------------------------

Two comments:
  1. Don't bother with useSasl and just use useWrap. useWrap implies useSasl.
  2. I don't think you correctly handle the case of pings when useWrap is true.

> Performance Improvement in Secure RPC
> -------------------------------------
>
>                 Key: HADOOP-6674
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6674
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.patch
>
>
> This jira introduces two performance improvements in Sasl RPC
> 1. Setting of Sasl 'quality of protection' to authentication only.
> 2. Addition of BufferedOutputStream underneath SaslOutputStream.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7590-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 22 22:55:16 2010
Return-Path: <common-issues-return-7590-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58719 invoked from network); 22 Apr 2010 22:55:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Apr 2010 22:55:15 -0000
Received: (qmail 92029 invoked by uid 500); 22 Apr 2010 22:55:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92006 invoked by uid 500); 22 Apr 2010 22:55:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91998 invoked by uid 99); 22 Apr 2010 22:55:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 22:55:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Apr 2010 22:55:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3MMsp5f015272
	for <common-issues@hadoop.apache.org>; Thu, 22 Apr 2010 22:54:51 GMT
Message-ID: <6608937.151151271976891684.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 18:54:51 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6708) New file format for very large
 records
In-Reply-To: <23756292.153431271370052193.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860056#action_12860056 ] 

Doug Cutting commented on HADOOP-6708:
--------------------------------------

> would it be possible to include a getRecordAsByteStream() / getRecordAsCharStream()

What might be simpler is to implement a DatumReader<InputStream> that only works when the schema is "bytes" and whose #read() implementation returns a clone of BinaryDecoder#getInputStream().  You'd want to use a "direct" binary decoder and pass it an input stream implementation that you know how to clone.  Then you could use the vanilla DataFileReader, seek to entries and read them.



> New file format for very large records
> --------------------------------------
>
>                 Key: HADOOP-6708
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6708
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>         Attachments: lobfile.pdf
>
>
> A file format that handles multi-gigabyte records efficiently, with lazy disk access

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7591-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 01:10:14 2010
Return-Path: <common-issues-return-7591-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94831 invoked from network); 23 Apr 2010 01:10:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 01:10:14 -0000
Received: (qmail 261 invoked by uid 500); 23 Apr 2010 01:10:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 234 invoked by uid 500); 23 Apr 2010 01:10:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 226 invoked by uid 99); 23 Apr 2010 01:10:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 01:10:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 01:10:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3N19oNg003785
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 01:09:50 GMT
Message-ID: <12583703.152991271984990100.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 21:09:50 -0400 (EDT)
From: "Edward J. Yoon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-3207) Add .externalToolBuilders to
 svn:ignore list
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-3207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Edward J. Yoon resolved HADOOP-3207.
------------------------------------

    Resolution: Won't Fix

won't fix

> Add .externalToolBuilders to svn:ignore list
> --------------------------------------------
>
>                 Key: HADOOP-3207
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3207
>             Project: Hadoop Common
>          Issue Type: Task
>            Reporter: Edward J. Yoon
>
> Could you, please, .externalToolBuilders to svn:ignores? 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7592-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 01:40:13 2010
Return-Path: <common-issues-return-7592-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5462 invoked from network); 23 Apr 2010 01:40:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 01:40:13 -0000
Received: (qmail 12492 invoked by uid 500); 23 Apr 2010 01:40:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12455 invoked by uid 500); 23 Apr 2010 01:40:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12447 invoked by uid 99); 23 Apr 2010 01:40:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 01:40:13 +0000
X-ASF-Spam-Status: No, hits=-1335.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 01:40:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3N1dq65004050
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 01:39:52 GMT
Message-ID: <5204273.153441271986792009.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 21:39:52 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6674) Performance Improvement in Secure RPC
In-Reply-To: <375873020.645441270163787224.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6674:
--------------------------------

    Attachment: HADOOP-6674-y20.1.bugfix.patch

Attaching a slightly modified version of the earlier patch.

> Performance Improvement in Secure RPC
> -------------------------------------
>
>                 Key: HADOOP-6674
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6674
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.patch
>
>
> This jira introduces two performance improvements in Sasl RPC
> 1. Setting of Sasl 'quality of protection' to authentication only.
> 2. Addition of BufferedOutputStream underneath SaslOutputStream.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7593-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 01:42:12 2010
Return-Path: <common-issues-return-7593-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7181 invoked from network); 23 Apr 2010 01:42:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 01:42:12 -0000
Received: (qmail 15335 invoked by uid 500); 23 Apr 2010 01:42:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15306 invoked by uid 500); 23 Apr 2010 01:42:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15298 invoked by uid 99); 23 Apr 2010 01:42:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 01:42:12 +0000
X-ASF-Spam-Status: No, hits=-1335.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 01:42:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3N1fp04004071
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 01:41:51 GMT
Message-ID: <2070758.153511271986911071.JavaMail.jira@thor>
Date: Thu, 22 Apr 2010 21:41:51 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6674) Performance Improvement in Secure
 RPC
In-Reply-To: <375873020.645441270163787224.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860098#action_12860098 ] 

Devaraj Das commented on HADOOP-6674:
-------------------------------------

In response to Owen's comment (2), the case of pings when useWrap is true is handled within processUnwrappedData (since the Pings are also wrapped and sent from the client)

> Performance Improvement in Secure RPC
> -------------------------------------
>
>                 Key: HADOOP-6674
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6674
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.patch
>
>
> This jira introduces two performance improvements in Sasl RPC
> 1. Setting of Sasl 'quality of protection' to authentication only.
> 2. Addition of BufferedOutputStream underneath SaslOutputStream.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7594-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 07:14:16 2010
Return-Path: <common-issues-return-7594-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82606 invoked from network); 23 Apr 2010 07:14:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 07:14:16 -0000
Received: (qmail 57850 invoked by uid 500); 23 Apr 2010 07:14:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57725 invoked by uid 500); 23 Apr 2010 07:14:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57716 invoked by uid 99); 23 Apr 2010 07:14:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 07:14:12 +0000
X-ASF-Spam-Status: No, hits=-1336.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 07:14:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3N7DpXk006998
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 07:13:51 GMT
Message-ID: <21303456.156191272006831455.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 03:13:51 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6686) Remove redundant exception class name
 in unwrapped exceptions thrown at the RPC client
In-Reply-To: <70650712.27801270591413554.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6686:
------------------------------------

    Status: Open  (was: Patch Available)

> Remove redundant exception class name in unwrapped exceptions thrown at the RPC client
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6686
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6686
>             Project: Hadoop Common
>          Issue Type: Improvement
>         Environment: At RPC client, when the exception thrown by the server is unwrapped, the exception message includes redundant exception class name. This redundant information should be removed.
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6686.1.patch, HADOOP-6686.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7595-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 07:16:13 2010
Return-Path: <common-issues-return-7595-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83176 invoked from network); 23 Apr 2010 07:16:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 07:16:13 -0000
Received: (qmail 58929 invoked by uid 500); 23 Apr 2010 07:16:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58824 invoked by uid 500); 23 Apr 2010 07:16:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58816 invoked by uid 99); 23 Apr 2010 07:16:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 07:16:11 +0000
X-ASF-Spam-Status: No, hits=-1336.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 07:16:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3N7Fo6S007021
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 07:15:50 GMT
Message-ID: <26701754.156221272006950204.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 03:15:50 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6686) Remove redundant exception class name
 in unwrapped exceptions thrown at the RPC client
In-Reply-To: <70650712.27801270591413554.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6686:
------------------------------------

    Status: Patch Available  (was: Open)

> Remove redundant exception class name in unwrapped exceptions thrown at the RPC client
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6686
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6686
>             Project: Hadoop Common
>          Issue Type: Improvement
>         Environment: At RPC client, when the exception thrown by the server is unwrapped, the exception message includes redundant exception class name. This redundant information should be removed.
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6686.1.patch, HADOOP-6686.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7596-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 07:38:14 2010
Return-Path: <common-issues-return-7596-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87024 invoked from network); 23 Apr 2010 07:38:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 07:38:14 -0000
Received: (qmail 78319 invoked by uid 500); 23 Apr 2010 07:38:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78188 invoked by uid 500); 23 Apr 2010 07:38:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78179 invoked by uid 99); 23 Apr 2010 07:38:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 07:38:11 +0000
X-ASF-Spam-Status: No, hits=-1336.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 07:38:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3N7bnQT007175
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 07:37:50 GMT
Message-ID: <6335111.156461272008269810.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 03:37:49 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6686) Remove redundant exception class
 name in unwrapped exceptions thrown at the RPC client
In-Reply-To: <70650712.27801270591413554.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860159#action_12860159 ] 

Hadoop QA commented on HADOOP-6686:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12441321/HADOOP-6686.1.patch
  against trunk revision 937093.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/474/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/474/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/474/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/474/console

This message is automatically generated.

> Remove redundant exception class name in unwrapped exceptions thrown at the RPC client
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6686
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6686
>             Project: Hadoop Common
>          Issue Type: Improvement
>         Environment: At RPC client, when the exception thrown by the server is unwrapped, the exception message includes redundant exception class name. This redundant information should be removed.
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6686.1.patch, HADOOP-6686.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7597-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 08:10:18 2010
Return-Path: <common-issues-return-7597-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92592 invoked from network); 23 Apr 2010 08:10:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 08:10:18 -0000
Received: (qmail 10715 invoked by uid 500); 23 Apr 2010 08:10:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10604 invoked by uid 500); 23 Apr 2010 08:10:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10384 invoked by uid 99); 23 Apr 2010 08:10:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 08:10:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 08:10:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3N89pSt007430
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 08:09:51 GMT
Message-ID: <17944576.156691272010191201.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 04:09:51 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6686) Remove redundant exception class name
 in unwrapped exceptions thrown at the RPC client
In-Reply-To: <70650712.27801270591413554.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6686:
------------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Incompatible change, Reviewed]
    Release Note: The exceptions thrown by the RPC client no longer carries a redundant exception class name in exception message. 
      Resolution: Fixed

Tests are not included in this patch, since it is covered in HDFS-1083. I committed this patch.

> Remove redundant exception class name in unwrapped exceptions thrown at the RPC client
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6686
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6686
>             Project: Hadoop Common
>          Issue Type: Improvement
>         Environment: At RPC client, when the exception thrown by the server is unwrapped, the exception message includes redundant exception class name. This redundant information should be removed.
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6686.1.patch, HADOOP-6686.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7598-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 11:25:23 2010
Return-Path: <common-issues-return-7598-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36268 invoked from network); 23 Apr 2010 11:25:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 11:25:23 -0000
Received: (qmail 99725 invoked by uid 500); 23 Apr 2010 11:25:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99609 invoked by uid 500); 23 Apr 2010 11:25:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99600 invoked by uid 99); 23 Apr 2010 11:25:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 11:25:22 +0000
X-ASF-Spam-Status: No, hits=-1337.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 11:25:21 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3NBP0tW009329
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 11:25:01 GMT
Message-ID: <3847588.159671272021900884.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 07:25:00 -0400 (EDT)
From: "eric baldeschwieler (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6659) Switch RPC to use Avro
In-Reply-To: <387448273.476441269469290584.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860220#action_12860220 ] 

eric baldeschwieler commented on HADOOP-6659:
---------------------------------------------

Hi Doug,

I do not support an incremental approach of protocol porting. The idea  
of supporting parallel impementations of our current hdfs protocols is  
mind boggling, as you've told me several times during Hadoop's  
development.

All of this change would be a huge tax on folks trying to stabilize  
Hadoop for release.

I agree doing this in a branch would be a huge job. This project is a  
huge job. That is my concern. It's not fair on the community to  
implicitly sign us all up to finish something of this scale. If you  
are signing up to do the project, great!  We will help you close once  
you can demonstrate you are close to the finish line.

I don't support throwing the whole community into a state where we  
need to finish this to ship good releases before anyone understands  
how we are going to finish it or who will finish it or how much work  
it will be.

Thanks for taking the time to understand my concerns!

---
E14 - via iPhone

On Apr 22, 2010, at 1:21 PM, "Doug Cutting (JIRA)" <jira@apache.org>  



> Switch RPC to use Avro
> ----------------------
>
>                 Key: HADOOP-6659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6659
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Doug Cutting
>
> This is an umbrella issue for moving HDFS and MapReduce RPC to use Avro.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7599-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 15:25:25 2010
Return-Path: <common-issues-return-7599-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99820 invoked from network); 23 Apr 2010 15:25:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 15:25:24 -0000
Received: (qmail 14547 invoked by uid 500); 23 Apr 2010 15:25:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14494 invoked by uid 500); 23 Apr 2010 15:25:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14486 invoked by uid 99); 23 Apr 2010 15:25:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 15:25:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 15:25:21 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3NFOxDm011244
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 15:24:59 GMT
Message-ID: <11889170.162261272036299545.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 11:24:59 -0400 (EDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6659) Switch RPC to use Avro
In-Reply-To: <387448273.476441269469290584.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860274#action_12860274 ] 

Arun C Murthy commented on HADOOP-6659:
---------------------------------------

Doug, the concern I have is that doing an incremental approach i.e. protocol-by-protocol, exposes us to the risk that a late-breaking bug will leave the projects (HDFS/Map-Reduce) in a state where we can't move forward or back since we have too many changes. I'm sure you see this.

Having said that I do understand it's harder to this in a branch. However, you will find a lot of support to ease the pain - I would be willing to volunteer to help on the Map-Reduce IDL work in the branch, I'm sure we will find others willing to pitch in for HDFS etc. That way we could get the system working end-to-end with Avro, test at scale and merge switch swiftly. Does that sounds reasonable?

> Switch RPC to use Avro
> ----------------------
>
>                 Key: HADOOP-6659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6659
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Doug Cutting
>
> This is an umbrella issue for moving HDFS and MapReduce RPC to use Avro.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7601-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 17:11:57 2010
Return-Path: <common-issues-return-7601-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2857 invoked from network); 23 Apr 2010 17:11:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 17:11:57 -0000
Received: (qmail 80609 invoked by uid 500); 23 Apr 2010 17:05:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80575 invoked by uid 500); 23 Apr 2010 17:05:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80567 invoked by uid 99); 23 Apr 2010 17:05:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 17:05:17 +0000
X-ASF-Spam-Status: No, hits=-1338.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 17:05:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3NH4uHw012329
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 17:04:56 GMT
Message-ID: <26998253.164551272042296914.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 13:04:56 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6659) Switch RPC to use Avro
In-Reply-To: <387448273.476441269469290584.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860316#action_12860316 ] 

Doug Cutting commented on HADOOP-6659:
--------------------------------------

> I would be willing to volunteer to help on the Map-Reduce IDL work in the branch

I am happy to kibitz on a branch.  But I would only embrace a branched strategy if I felt I could dedicate myself full-time to that one effort together with similarly dedicated colleagues so that the branch time could be minimized.  Unfortunately, I cannot dedicate myself full-time to such an endeavor.  If others can, then they might pursue this, but I cannot in good faith, but would still be happy to assist how I can and will certainly not be an obstacle to such a plan.

> Switch RPC to use Avro
> ----------------------
>
>                 Key: HADOOP-6659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6659
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Doug Cutting
>
> This is an umbrella issue for moving HDFS and MapReduce RPC to use Avro.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7600-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 17:26:00 2010
Return-Path: <common-issues-return-7600-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15351 invoked from network); 23 Apr 2010 17:25:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 17:25:59 -0000
Received: (qmail 68838 invoked by uid 500); 23 Apr 2010 16:59:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68780 invoked by uid 500); 23 Apr 2010 16:59:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68771 invoked by uid 99); 23 Apr 2010 16:59:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 16:59:18 +0000
X-ASF-Spam-Status: No, hits=-1338.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 16:59:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3NGwuOJ012149
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 16:58:57 GMT
Message-ID: <5636395.164091272041936805.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 12:58:56 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6659) Switch RPC to use Avro
In-Reply-To: <387448273.476441269469290584.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860312#action_12860312 ] 

Doug Cutting commented on HADOOP-6659:
--------------------------------------

> exposes us to the risk that a late-breaking bug will leave the projects (HDFS/Map-Reduce) in a state where we can't move forward or back

We should test each change thoroughly before we commit it.  We should not commit changes we feel are excessively risky and insufficiently tested.  I do not propose committing any changes that are not agreed to be safe.

Changing the serialization used for RPC is not a change in semantics, it's a change in syntax.  As such, this change alone should not introduce any deep bugs.  It may introduce NullPointerException-like bugs, but these are generally easy to fix.  It should not introduce any new synchronization or other logic bugs.  Thus I think the chance of a deep bug that goes undetected and will take a great investment to fix is minimal.

If others disagree and wish to develop this as a branch they are welcome to do so, but I do not think we should freeze changes to protocols in trunk while they work in this branch.

> Switch RPC to use Avro
> ----------------------
>
>                 Key: HADOOP-6659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6659
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Doug Cutting
>
> This is an umbrella issue for moving HDFS and MapReduce RPC to use Avro.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7602-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 18:09:12 2010
Return-Path: <common-issues-return-7602-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40328 invoked from network); 23 Apr 2010 18:09:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 18:09:12 -0000
Received: (qmail 58928 invoked by uid 500); 23 Apr 2010 18:09:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58892 invoked by uid 500); 23 Apr 2010 18:09:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58884 invoked by uid 99); 23 Apr 2010 18:09:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 18:09:12 +0000
X-ASF-Spam-Status: No, hits=-1339.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 18:09:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3NI8ofu012779
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 18:08:50 GMT
Message-ID: <3767164.165361272046130535.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 14:08:50 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6710) Symbolic umask for file creation is
 not consistent with posix
In-Reply-To: <8947165.155011271375809341.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6710:
--------------------------------

    Hadoop Flags: [Reviewed]

+1. Looks excellent.

> Symbolic umask for file creation is not consistent with posix
> -------------------------------------------------------------
>
>                 Key: HADOOP-6710
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6710
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Suresh Srinivas
>         Attachments: hadoop-6710.patch
>
>
> Currently both octal and symbolic umask are used to reset the file creation mode bits. This is not consistent with the behavior defined in posix. Making it consistent would avoid confusion to the HDFS users.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7603-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 18:13:12 2010
Return-Path: <common-issues-return-7603-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42083 invoked from network); 23 Apr 2010 18:13:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 18:13:12 -0000
Received: (qmail 67915 invoked by uid 500); 23 Apr 2010 18:13:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67875 invoked by uid 500); 23 Apr 2010 18:13:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67866 invoked by uid 99); 23 Apr 2010 18:13:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 18:13:12 +0000
X-ASF-Spam-Status: No, hits=-1339.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 18:13:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3NICp4Y012839
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 18:12:51 GMT
Message-ID: <7062082.165511272046371574.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 14:12:51 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6521) FsPermission:SetUMask not updated to
 use new-style umask setting.
In-Reply-To: <797058971.135641264793314607.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6521:
--------------------------------

    Hadoop Flags: [Reviewed]

If we're - correctly - not using the deprecatedKeyWasSet method, and this was the only place it was used and it's not yet been released, might we just as well remove it?  Otherwise it's an unused, potentially buggy API.  Otherwise, +1.

> FsPermission:SetUMask not updated to use new-style umask setting.
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6521
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6521
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Jakob Homan
>            Assignee: Suresh Srinivas
>         Attachments: hadoop-6521.patch, hadoop-6521.rel20.1.patch, hadoop-6521.rel20.patch, hadoop-6521.rel20.patch, hadoop-6521.rel20.patch
>
>
> FsPermission:
> {code}
> 221   /** Set the user file creation mask (umask) */
> 222   public static void setUMask(Configuration conf, FsPermission umask) {                                    
> 223     conf.setInt(UMASK_LABEL, umask.toShort());
> 224   }
> {code}
> Needs to be updated to not use a decimal value. This is a bug introduced by HADOOP-6234.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7604-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 19:29:12 2010
Return-Path: <common-issues-return-7604-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64596 invoked from network); 23 Apr 2010 19:29:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 19:29:11 -0000
Received: (qmail 44951 invoked by uid 500); 23 Apr 2010 19:29:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44915 invoked by uid 500); 23 Apr 2010 19:29:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44907 invoked by uid 99); 23 Apr 2010 19:29:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 19:29:11 +0000
X-ASF-Spam-Status: No, hits=-1339.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 19:29:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3NJSoAf013359
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 19:28:50 GMT
Message-ID: <10358602.166501272050930194.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 15:28:50 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6701)  Incorrect exit codes for "dfs
 -chown", "dfs -chgrp"
In-Reply-To: <29784448.2801271097111992.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860356#action_12860356 ] 

Ravi Phulari commented on HADOOP-6701:
--------------------------------------

This fix will change the way {{ch(mod/own/grp)}} returns exit codes.  

Failure to execute chmod, chown, chgrp commands will return non zero exit code with an error message. 

>  Incorrect exit codes for "dfs -chown", "dfs -chgrp"
> ----------------------------------------------------
>
>                 Key: HADOOP-6701
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6701
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.19.1, 0.20.0, 0.20.1, 0.20.2
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>            Priority: Minor
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6701-20s.patch, HADOOP-6701-trunk.patch
>
>
> ravi@localhost:~$ hadoop dfs -chgrp abcd /; echo $?
> chgrp: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chown  abcd /; echo $?
> chown: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chmod 755 /DOESNTEXIST; echo $?
> chmod: could not get status for '/DOESNTEXIST': File does not exist: /DOESNTEXIST
> 0
> -
> Exit codes for both of the above invocations should be non-zero to indicate that the command failed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7605-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 20:54:15 2010
Return-Path: <common-issues-return-7605-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96206 invoked from network); 23 Apr 2010 20:54:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 20:54:14 -0000
Received: (qmail 62135 invoked by uid 500); 23 Apr 2010 20:54:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62092 invoked by uid 500); 23 Apr 2010 20:54:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62060 invoked by uid 99); 23 Apr 2010 20:54:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 20:54:13 +0000
X-ASF-Spam-Status: No, hits=-1339.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 20:54:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3NKrpk8013759
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 20:53:52 GMT
Message-ID: <18050830.167121272056031914.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 16:53:51 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6710) Symbolic umask for file creation is
 not consistent with posix
In-Reply-To: <8947165.155011271375809341.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6710:
------------------------------------

    Status: Patch Available  (was: Open)

> Symbolic umask for file creation is not consistent with posix
> -------------------------------------------------------------
>
>                 Key: HADOOP-6710
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6710
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Suresh Srinivas
>         Attachments: hadoop-6710.patch
>
>
> Currently both octal and symbolic umask are used to reset the file creation mode bits. This is not consistent with the behavior defined in posix. Making it consistent would avoid confusion to the HDFS users.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7606-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 21:10:14 2010
Return-Path: <common-issues-return-7606-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3072 invoked from network); 23 Apr 2010 21:10:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 21:10:14 -0000
Received: (qmail 84884 invoked by uid 500); 23 Apr 2010 21:10:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84798 invoked by uid 500); 23 Apr 2010 21:10:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84790 invoked by uid 99); 23 Apr 2010 21:10:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 21:10:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 21:10:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3NL9owg013880
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 21:09:50 GMT
Message-ID: <20764211.167321272056990295.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 17:09:50 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6701)  Incorrect exit codes for "dfs
 -chown", "dfs -chgrp"
In-Reply-To: <29784448.2801271097111992.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6701:
---------------------------------

    Attachment:     (was: HADOOP-6701-20s.patch)

>  Incorrect exit codes for "dfs -chown", "dfs -chgrp"
> ----------------------------------------------------
>
>                 Key: HADOOP-6701
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6701
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.19.1, 0.20.0, 0.20.1, 0.20.2
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>            Priority: Minor
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6701-trunk.patch
>
>
> ravi@localhost:~$ hadoop dfs -chgrp abcd /; echo $?
> chgrp: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chown  abcd /; echo $?
> chown: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chmod 755 /DOESNTEXIST; echo $?
> chmod: could not get status for '/DOESNTEXIST': File does not exist: /DOESNTEXIST
> 0
> -
> Exit codes for both of the above invocations should be non-zero to indicate that the command failed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7607-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 21:12:11 2010
Return-Path: <common-issues-return-7607-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3758 invoked from network); 23 Apr 2010 21:12:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 21:12:11 -0000
Received: (qmail 85847 invoked by uid 500); 23 Apr 2010 21:12:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85795 invoked by uid 500); 23 Apr 2010 21:12:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85784 invoked by uid 99); 23 Apr 2010 21:12:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 21:12:11 +0000
X-ASF-Spam-Status: No, hits=-1339.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 21:12:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3NLBnOY013895
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 21:11:50 GMT
Message-ID: <13145260.167351272057109857.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 17:11:49 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6701)  Incorrect exit codes for "dfs
 -chown", "dfs -chgrp"
In-Reply-To: <29784448.2801271097111992.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6701:
---------------------------------

    Attachment:     (was: HADOOP-6701-trunk.patch)

>  Incorrect exit codes for "dfs -chown", "dfs -chgrp"
> ----------------------------------------------------
>
>                 Key: HADOOP-6701
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6701
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.19.1, 0.20.0, 0.20.1, 0.20.2
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>            Priority: Minor
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>
> ravi@localhost:~$ hadoop dfs -chgrp abcd /; echo $?
> chgrp: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chown  abcd /; echo $?
> chown: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chmod 755 /DOESNTEXIST; echo $?
> chmod: could not get status for '/DOESNTEXIST': File does not exist: /DOESNTEXIST
> 0
> -
> Exit codes for both of the above invocations should be non-zero to indicate that the command failed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7608-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 21:18:13 2010
Return-Path: <common-issues-return-7608-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5323 invoked from network); 23 Apr 2010 21:18:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 21:18:13 -0000
Received: (qmail 92849 invoked by uid 500); 23 Apr 2010 21:18:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92820 invoked by uid 500); 23 Apr 2010 21:18:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92812 invoked by uid 99); 23 Apr 2010 21:18:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 21:18:13 +0000
X-ASF-Spam-Status: No, hits=-1339.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 21:18:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3NLHqJM013968
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 21:17:52 GMT
Message-ID: <30409191.167551272057472053.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 17:17:52 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6701)  Incorrect exit codes for "dfs
 -chown", "dfs -chgrp"
In-Reply-To: <29784448.2801271097111992.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6701:
---------------------------------

    Attachment: HADOOP-6701.patch

Patch for 0.20.

>  Incorrect exit codes for "dfs -chown", "dfs -chgrp"
> ----------------------------------------------------
>
>                 Key: HADOOP-6701
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6701
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.19.1, 0.20.0, 0.20.1, 0.20.2
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>            Priority: Minor
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6701.patch
>
>
> ravi@localhost:~$ hadoop dfs -chgrp abcd /; echo $?
> chgrp: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chown  abcd /; echo $?
> chown: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chmod 755 /DOESNTEXIST; echo $?
> chmod: could not get status for '/DOESNTEXIST': File does not exist: /DOESNTEXIST
> 0
> -
> Exit codes for both of the above invocations should be non-zero to indicate that the command failed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7609-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 21:22:14 2010
Return-Path: <common-issues-return-7609-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6316 invoked from network); 23 Apr 2010 21:22:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 21:22:14 -0000
Received: (qmail 1035 invoked by uid 500); 23 Apr 2010 21:22:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 999 invoked by uid 500); 23 Apr 2010 21:22:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 991 invoked by uid 99); 23 Apr 2010 21:22:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 21:22:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 21:22:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3NLLoM3013995
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 21:21:50 GMT
Message-ID: <31815083.167611272057710545.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 17:21:50 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6674) Performance Improvement in Secure
 RPC
In-Reply-To: <375873020.645441270163787224.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860379#action_12860379 ] 

Owen O'Malley commented on HADOOP-6674:
---------------------------------------

The new patch looks good.

> Performance Improvement in Secure RPC
> -------------------------------------
>
>                 Key: HADOOP-6674
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6674
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.patch
>
>
> This jira introduces two performance improvements in Sasl RPC
> 1. Setting of Sasl 'quality of protection' to authentication only.
> 2. Addition of BufferedOutputStream underneath SaslOutputStream.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7610-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 21:28:15 2010
Return-Path: <common-issues-return-7610-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7362 invoked from network); 23 Apr 2010 21:28:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 21:28:14 -0000
Received: (qmail 10239 invoked by uid 500); 23 Apr 2010 21:28:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10204 invoked by uid 500); 23 Apr 2010 21:28:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10196 invoked by uid 99); 23 Apr 2010 21:28:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 21:28:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 21:28:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3NLRo0S014036
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 21:27:51 GMT
Message-ID: <4908349.167711272058070731.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 17:27:50 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6521) FsPermission:SetUMask not updated to
 use new-style umask setting.
In-Reply-To: <797058971.135641264793314607.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6521:
------------------------------------

    Attachment: hadoop-6521.patch

Removed deprecateKeyWasSet() method and related tests.

> FsPermission:SetUMask not updated to use new-style umask setting.
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6521
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6521
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Jakob Homan
>            Assignee: Suresh Srinivas
>         Attachments: hadoop-6521.patch, hadoop-6521.patch, hadoop-6521.rel20.1.patch, hadoop-6521.rel20.patch, hadoop-6521.rel20.patch, hadoop-6521.rel20.patch
>
>
> FsPermission:
> {code}
> 221   /** Set the user file creation mask (umask) */
> 222   public static void setUMask(Configuration conf, FsPermission umask) {                                    
> 223     conf.setInt(UMASK_LABEL, umask.toShort());
> 224   }
> {code}
> Needs to be updated to not use a decimal value. This is a bug introduced by HADOOP-6234.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7611-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 21:28:15 2010
Return-Path: <common-issues-return-7611-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7408 invoked from network); 23 Apr 2010 21:28:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 21:28:15 -0000
Received: (qmail 10386 invoked by uid 500); 23 Apr 2010 21:28:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10338 invoked by uid 500); 23 Apr 2010 21:28:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10330 invoked by uid 99); 23 Apr 2010 21:28:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 21:28:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 21:28:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3NLRpuQ014054
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 21:27:51 GMT
Message-ID: <23667449.167771272058071703.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 17:27:51 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6521) FsPermission:SetUMask not updated to
 use new-style umask setting.
In-Reply-To: <797058971.135641264793314607.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6521:
------------------------------------

    Status: Patch Available  (was: Open)

> FsPermission:SetUMask not updated to use new-style umask setting.
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6521
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6521
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Jakob Homan
>            Assignee: Suresh Srinivas
>         Attachments: hadoop-6521.patch, hadoop-6521.patch, hadoop-6521.rel20.1.patch, hadoop-6521.rel20.patch, hadoop-6521.rel20.patch, hadoop-6521.rel20.patch
>
>
> FsPermission:
> {code}
> 221   /** Set the user file creation mask (umask) */
> 222   public static void setUMask(Configuration conf, FsPermission umask) {                                    
> 223     conf.setInt(UMASK_LABEL, umask.toShort());
> 224   }
> {code}
> Needs to be updated to not use a decimal value. This is a bug introduced by HADOOP-6234.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7612-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 21:30:15 2010
Return-Path: <common-issues-return-7612-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7994 invoked from network); 23 Apr 2010 21:30:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 21:30:15 -0000
Received: (qmail 12293 invoked by uid 500); 23 Apr 2010 21:30:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12266 invoked by uid 500); 23 Apr 2010 21:30:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12258 invoked by uid 99); 23 Apr 2010 21:30:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 21:30:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 21:30:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3NLTpwi014081
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 21:29:51 GMT
Message-ID: <27600861.167841272058191359.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 17:29:51 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6508) Incorrect values for metrics with
 CompositeContext
In-Reply-To: <1389520400.1361264392034628.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu reassigned HADOOP-6508:
-------------------------------

    Assignee: Luke Lu

> Incorrect values for metrics with CompositeContext
> --------------------------------------------------
>
>                 Key: HADOOP-6508
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6508
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.0
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: CompositeContext-solution.png, CompositeContext.png
>
>
> In our clusters, when we use CompositeContext with two contexts, second context gets wrong values.
> This problem is consistent on 500 (and above) node cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7613-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 21:44:15 2010
Return-Path: <common-issues-return-7613-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10679 invoked from network); 23 Apr 2010 21:44:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 21:44:15 -0000
Received: (qmail 20542 invoked by uid 500); 23 Apr 2010 21:44:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20458 invoked by uid 500); 23 Apr 2010 21:44:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20450 invoked by uid 99); 23 Apr 2010 21:44:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 21:44:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 21:44:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3NLhppS014220
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 21:43:51 GMT
Message-ID: <3132565.168191272059031250.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 17:43:51 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6521) FsPermission:SetUMask not updated
 to use new-style umask setting.
In-Reply-To: <797058971.135641264793314607.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860389#action_12860389 ] 

Hadoop QA commented on HADOOP-6521:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12442708/hadoop-6521.patch
  against trunk revision 937183.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/50/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/50/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/50/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/50/console

This message is automatically generated.

> FsPermission:SetUMask not updated to use new-style umask setting.
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6521
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6521
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Jakob Homan
>            Assignee: Suresh Srinivas
>         Attachments: hadoop-6521.patch, hadoop-6521.patch, hadoop-6521.rel20.1.patch, hadoop-6521.rel20.patch, hadoop-6521.rel20.patch, hadoop-6521.rel20.patch
>
>
> FsPermission:
> {code}
> 221   /** Set the user file creation mask (umask) */
> 222   public static void setUMask(Configuration conf, FsPermission umask) {                                    
> 223     conf.setInt(UMASK_LABEL, umask.toShort());
> 224   }
> {code}
> Needs to be updated to not use a decimal value. This is a bug introduced by HADOOP-6234.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7614-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 22:00:12 2010
Return-Path: <common-issues-return-7614-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14407 invoked from network); 23 Apr 2010 22:00:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 22:00:12 -0000
Received: (qmail 37945 invoked by uid 500); 23 Apr 2010 22:00:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37914 invoked by uid 500); 23 Apr 2010 22:00:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37883 invoked by uid 99); 23 Apr 2010 22:00:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 22:00:11 +0000
X-ASF-Spam-Status: No, hits=-1339.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 22:00:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3NLxols014429
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 21:59:50 GMT
Message-ID: <2234490.168681272059990720.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 17:59:50 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6701)  Incorrect exit codes for "dfs
 -chown", "dfs -chgrp"
In-Reply-To: <29784448.2801271097111992.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860395#action_12860395 ] 

Suresh Srinivas commented on HADOOP-6701:
-----------------------------------------

+1 for the patch.

>  Incorrect exit codes for "dfs -chown", "dfs -chgrp"
> ----------------------------------------------------
>
>                 Key: HADOOP-6701
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6701
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.19.1, 0.20.0, 0.20.1, 0.20.2
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>            Priority: Minor
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6701.patch
>
>
> ravi@localhost:~$ hadoop dfs -chgrp abcd /; echo $?
> chgrp: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chown  abcd /; echo $?
> chown: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chmod 755 /DOESNTEXIST; echo $?
> chmod: could not get status for '/DOESNTEXIST': File does not exist: /DOESNTEXIST
> 0
> -
> Exit codes for both of the above invocations should be non-zero to indicate that the command failed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7615-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 22:08:12 2010
Return-Path: <common-issues-return-7615-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15937 invoked from network); 23 Apr 2010 22:08:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 22:08:12 -0000
Received: (qmail 45583 invoked by uid 500); 23 Apr 2010 22:08:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45555 invoked by uid 500); 23 Apr 2010 22:08:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45545 invoked by uid 99); 23 Apr 2010 22:08:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 22:08:12 +0000
X-ASF-Spam-Status: No, hits=-1339.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 22:08:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3NM7pxo014470
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 22:07:51 GMT
Message-ID: <25096925.168771272060471145.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 18:07:51 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6692) Add FileContext#listStatus that
 returns an iterator
In-Reply-To: <253397745.22171270762596744.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6692:
----------------------------------

    Attachment: listStatusItor1.patch

Thank Suresh and Eli for review my patch. Those are good feedback. This patch addresses most of the comments except for
> FileContext.listStatus() that returns iterator - add information about throwing UnsupportedOperationException and NoSuchElementException in the RuntimeExceptions section on attempt to delete an entry from iterator.

Those exceptions are not thrown in listStatus method. Those may be thrown when iterating happens.

> AFS.listStatusItertor() - method javadoc should point @link to the right method in FileContext.
It points to the right method. It happens that in FileContext the method that returns an iterator is named as listStatus. 

> Add FileContext#listStatus that returns an iterator
> ---------------------------------------------------
>
>                 Key: HADOOP-6692
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6692
>             Project: Hadoop Common
>          Issue Type: Sub-task
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listStatusItor.patch, listStatusItor1.patch
>
>
> Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7616-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 22:08:15 2010
Return-Path: <common-issues-return-7616-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15992 invoked from network); 23 Apr 2010 22:08:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 22:08:15 -0000
Received: (qmail 45747 invoked by uid 500); 23 Apr 2010 22:08:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45697 invoked by uid 500); 23 Apr 2010 22:08:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45689 invoked by uid 99); 23 Apr 2010 22:08:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 22:08:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 22:08:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3NM7qwh014488
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 22:07:52 GMT
Message-ID: <32279467.168831272060472103.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 18:07:52 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6521) FsPermission:SetUMask not updated to
 use new-style umask setting.
In-Reply-To: <797058971.135641264793314607.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6521:
------------------------------------

    Status: Open  (was: Patch Available)

> FsPermission:SetUMask not updated to use new-style umask setting.
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6521
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6521
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Jakob Homan
>            Assignee: Suresh Srinivas
>         Attachments: hadoop-6521.patch, hadoop-6521.patch, hadoop-6521.rel20.1.patch, hadoop-6521.rel20.patch, hadoop-6521.rel20.patch, hadoop-6521.rel20.patch
>
>
> FsPermission:
> {code}
> 221   /** Set the user file creation mask (umask) */
> 222   public static void setUMask(Configuration conf, FsPermission umask) {                                    
> 223     conf.setInt(UMASK_LABEL, umask.toShort());
> 224   }
> {code}
> Needs to be updated to not use a decimal value. This is a bug introduced by HADOOP-6234.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7617-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 22:10:15 2010
Return-Path: <common-issues-return-7617-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16239 invoked from network); 23 Apr 2010 22:10:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 22:10:15 -0000
Received: (qmail 47337 invoked by uid 500); 23 Apr 2010 22:10:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47295 invoked by uid 500); 23 Apr 2010 22:10:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47287 invoked by uid 99); 23 Apr 2010 22:10:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 22:10:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 22:10:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3NM9oiK014518
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 22:09:51 GMT
Message-ID: <9889262.168911272060590911.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 18:09:50 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6521) FsPermission:SetUMask not updated to
 use new-style umask setting.
In-Reply-To: <797058971.135641264793314607.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6521:
------------------------------------

    Attachment: hadoop-6521.1.patch

Attaching the right patch.

> FsPermission:SetUMask not updated to use new-style umask setting.
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6521
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6521
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Jakob Homan
>            Assignee: Suresh Srinivas
>         Attachments: hadoop-6521.1.patch, hadoop-6521.patch, hadoop-6521.patch, hadoop-6521.rel20.1.patch, hadoop-6521.rel20.patch, hadoop-6521.rel20.patch, hadoop-6521.rel20.patch
>
>
> FsPermission:
> {code}
> 221   /** Set the user file creation mask (umask) */
> 222   public static void setUMask(Configuration conf, FsPermission umask) {                                    
> 223     conf.setInt(UMASK_LABEL, umask.toShort());
> 224   }
> {code}
> Needs to be updated to not use a decimal value. This is a bug introduced by HADOOP-6234.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7618-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 22:10:15 2010
Return-Path: <common-issues-return-7618-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16281 invoked from network); 23 Apr 2010 22:10:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 22:10:15 -0000
Received: (qmail 47466 invoked by uid 500); 23 Apr 2010 22:10:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47428 invoked by uid 500); 23 Apr 2010 22:10:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47420 invoked by uid 99); 23 Apr 2010 22:10:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 22:10:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 22:10:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3NM9pe8014536
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 22:09:51 GMT
Message-ID: <9109602.168971272060591846.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 18:09:51 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6521) FsPermission:SetUMask not updated to
 use new-style umask setting.
In-Reply-To: <797058971.135641264793314607.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6521:
------------------------------------

    Status: Patch Available  (was: Open)

> FsPermission:SetUMask not updated to use new-style umask setting.
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6521
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6521
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Jakob Homan
>            Assignee: Suresh Srinivas
>         Attachments: hadoop-6521.1.patch, hadoop-6521.patch, hadoop-6521.patch, hadoop-6521.rel20.1.patch, hadoop-6521.rel20.patch, hadoop-6521.rel20.patch, hadoop-6521.rel20.patch
>
>
> FsPermission:
> {code}
> 221   /** Set the user file creation mask (umask) */
> 222   public static void setUMask(Configuration conf, FsPermission umask) {                                    
> 223     conf.setInt(UMASK_LABEL, umask.toShort());
> 224   }
> {code}
> Needs to be updated to not use a decimal value. This is a bug introduced by HADOOP-6234.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7619-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 22:19:29 2010
Return-Path: <common-issues-return-7619-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17048 invoked from network); 23 Apr 2010 22:19:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 22:19:28 -0000
Received: (qmail 49869 invoked by uid 500); 23 Apr 2010 22:19:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49833 invoked by uid 500); 23 Apr 2010 22:19:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49825 invoked by uid 99); 23 Apr 2010 22:19:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 22:19:28 +0000
X-ASF-Spam-Status: No, hits=-1339.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 22:19:27 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3NMJ77l014568
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 22:19:07 GMT
Message-ID: <25991305.169041272061147307.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 18:19:07 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6710) Symbolic umask for file creation is
 not consistent with posix
In-Reply-To: <8947165.155011271375809341.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860402#action_12860402 ] 

Hadoop QA commented on HADOOP-6710:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12441987/hadoop-6710.patch
  against trunk revision 937183.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/475/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/475/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/475/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/475/console

This message is automatically generated.

> Symbolic umask for file creation is not consistent with posix
> -------------------------------------------------------------
>
>                 Key: HADOOP-6710
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6710
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Suresh Srinivas
>         Attachments: hadoop-6710.patch
>
>
> Currently both octal and symbolic umask are used to reset the file creation mode bits. This is not consistent with the behavior defined in posix. Making it consistent would avoid confusion to the HDFS users.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7620-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 22:25:15 2010
Return-Path: <common-issues-return-7620-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18025 invoked from network); 23 Apr 2010 22:25:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 22:25:15 -0000
Received: (qmail 55318 invoked by uid 500); 23 Apr 2010 22:25:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55279 invoked by uid 500); 23 Apr 2010 22:25:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55265 invoked by uid 99); 23 Apr 2010 22:25:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 22:25:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 22:25:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3NMOpKa014625
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 22:24:51 GMT
Message-ID: <28803728.169121272061491153.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 18:24:51 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6521) FsPermission:SetUMask not updated
 to use new-style umask setting.
In-Reply-To: <797058971.135641264793314607.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860404#action_12860404 ] 

Hadoop QA commented on HADOOP-6521:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12442715/hadoop-6521.1.patch
  against trunk revision 937183.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/51/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/51/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/51/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/51/console

This message is automatically generated.

> FsPermission:SetUMask not updated to use new-style umask setting.
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6521
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6521
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Jakob Homan
>            Assignee: Suresh Srinivas
>         Attachments: hadoop-6521.1.patch, hadoop-6521.patch, hadoop-6521.patch, hadoop-6521.rel20.1.patch, hadoop-6521.rel20.patch, hadoop-6521.rel20.patch, hadoop-6521.rel20.patch
>
>
> FsPermission:
> {code}
> 221   /** Set the user file creation mask (umask) */
> 222   public static void setUMask(Configuration conf, FsPermission umask) {                                    
> 223     conf.setInt(UMASK_LABEL, umask.toShort());
> 224   }
> {code}
> Needs to be updated to not use a decimal value. This is a bug introduced by HADOOP-6234.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7621-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 22:55:15 2010
Return-Path: <common-issues-return-7621-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23368 invoked from network); 23 Apr 2010 22:55:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 22:55:15 -0000
Received: (qmail 73087 invoked by uid 500); 23 Apr 2010 22:55:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73057 invoked by uid 500); 23 Apr 2010 22:55:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73049 invoked by uid 99); 23 Apr 2010 22:55:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 22:55:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 22:55:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3NMsp5Q014762
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 22:54:51 GMT
Message-ID: <22298101.169331272063291509.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 18:54:51 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6701)  Incorrect exit codes for "dfs
 -chown", "dfs -chgrp"
In-Reply-To: <29784448.2801271097111992.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860410#action_12860410 ] 

Ravi Phulari commented on HADOOP-6701:
--------------------------------------

*This patch changes exit codes returned from  chmod, chgrp and chown operations*

*  Old Behavior  of {{chmod,chown,chgrp}} operation.
  ** If operation is successful then - exit code 0 and no error message.
  ** If operation is unsuccessful then - exit code 0 and error message printed.
{noformat} 
*If files does not exist and with globbed input*
[rphulari@statepick-lm]> bin/hadoop fs -chmod 777 st*; echo $?
0

*If files does not exist without globbed input*
[rphulari@statepick-lm]> bin/hadoop fs -chmod 777 st; echo $?
chmod: could not get status for 'st': File does not exist: st
0

*If user does not have permission to change file-permission*
ravi@localhost:~$ hadoop dfs -chgrp abcd /; echo $?
chgrp: changing ownership of
'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
0

*If user does not have permission to change file-permission*
ravi@localhost:~$ hadoop dfs -chown abcd /; echo $?
chown: changing ownership of
'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
0

*If user has permission to change file-permission and file does not exist*
ravi@localhost:~$ hadoop dfs -chmod 755 /DOESNTEXIST; echo $?
chmod: could not get status for '/DOESNTEXIST': File does not exist: /DOESNTEXIST
0
{noformat} 

*  New Behavior  of {{chmod,chown,chgrp}} operation.
  ** If operation is successful then - exit code 0 is returned and no error message.
  ** If operation is unsuccessful then - exit code 1 is returned and error message printed.

{noformat} 
*If file does not exist*
[rphulari@lm]> bin/hadoop fs -chown rphulari:users est; echo $?
chown: could not get status for 'est': File does not exist: est
1

*If files does not exist and with globbed input*
[rphulari@statepick-lm]> bin/hadoop fs -chown rphulari:users est\*; echo $?
chown: could not get status for 'est*'
1

*If file exist and operation is successful*
[rphulari@statepick-lm]> bin/hadoop fs -chmod 700 test; echo $?
0


*If file/dir with glob pattern exits and operation is successful*
[rphulari@statepick-lm]> bin/hadoop fs -chmod 777 test*; echo $?
0
[rphulari@statepick-lm]> bin/hadoop fs -ls 
Found 2 items
-rw-rw-rw-   1 admin    supergroup          0 2010-04-15 13:54 /user/rphulari/t
drwxrwxrwx   - rphulari userss              0 2010-03-30 23:39 /user/rphulari/test

{noformat} 

*Unchanged behavior - need to be fixed in different Jira*
Because following bug is due to Permissions code we will change it in separate Jira .

{noformat} 
*If user does not have permission to change file-permission*
ravi@localhost:~$ hadoop dfs -chown abcd /; echo $?
chown: changing ownership of
'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
0
{noformat} 


>  Incorrect exit codes for "dfs -chown", "dfs -chgrp"
> ----------------------------------------------------
>
>                 Key: HADOOP-6701
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6701
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.19.1, 0.20.0, 0.20.1, 0.20.2
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>            Priority: Minor
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6701.patch
>
>
> ravi@localhost:~$ hadoop dfs -chgrp abcd /; echo $?
> chgrp: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chown  abcd /; echo $?
> chown: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chmod 755 /DOESNTEXIST; echo $?
> chmod: could not get status for '/DOESNTEXIST': File does not exist: /DOESNTEXIST
> 0
> -
> Exit codes for both of the above invocations should be non-zero to indicate that the command failed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7622-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 22:57:14 2010
Return-Path: <common-issues-return-7622-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23643 invoked from network); 23 Apr 2010 22:57:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 22:57:14 -0000
Received: (qmail 75493 invoked by uid 500); 23 Apr 2010 22:57:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75386 invoked by uid 500); 23 Apr 2010 22:57:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75376 invoked by uid 99); 23 Apr 2010 22:57:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 22:57:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 22:57:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3NMuoNk014787
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 22:56:50 GMT
Message-ID: <31803128.169361272063410068.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 18:56:50 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6701)  Incorrect exit codes for "dfs
 -chown", "dfs -chgrp"
In-Reply-To: <29784448.2801271097111992.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6701:
---------------------------------

    Attachment:     (was: HADOOP-6701.patch)

>  Incorrect exit codes for "dfs -chown", "dfs -chgrp"
> ----------------------------------------------------
>
>                 Key: HADOOP-6701
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6701
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.19.1, 0.20.0, 0.20.1, 0.20.2
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>            Priority: Minor
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>
> ravi@localhost:~$ hadoop dfs -chgrp abcd /; echo $?
> chgrp: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chown  abcd /; echo $?
> chown: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chmod 755 /DOESNTEXIST; echo $?
> chmod: could not get status for '/DOESNTEXIST': File does not exist: /DOESNTEXIST
> 0
> -
> Exit codes for both of the above invocations should be non-zero to indicate that the command failed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7623-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 23:03:14 2010
Return-Path: <common-issues-return-7623-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25153 invoked from network); 23 Apr 2010 23:03:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 23:03:13 -0000
Received: (qmail 80280 invoked by uid 500); 23 Apr 2010 23:03:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80260 invoked by uid 500); 23 Apr 2010 23:03:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80252 invoked by uid 99); 23 Apr 2010 23:03:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 23:03:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 23:03:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3NN2oPf014814
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 23:02:50 GMT
Message-ID: <3700062.169411272063770182.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 19:02:50 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6701)  Incorrect exit codes for "dfs
 -chown", "dfs -chgrp"
In-Reply-To: <29784448.2801271097111992.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6701:
---------------------------------

    Attachment: HADOOP-6701.v20s.patch

>  Incorrect exit codes for "dfs -chown", "dfs -chgrp"
> ----------------------------------------------------
>
>                 Key: HADOOP-6701
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6701
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.19.1, 0.20.0, 0.20.1, 0.20.2
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>            Priority: Minor
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6701.v20s.patch
>
>
> ravi@localhost:~$ hadoop dfs -chgrp abcd /; echo $?
> chgrp: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chown  abcd /; echo $?
> chown: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chmod 755 /DOESNTEXIST; echo $?
> chmod: could not get status for '/DOESNTEXIST': File does not exist: /DOESNTEXIST
> 0
> -
> Exit codes for both of the above invocations should be non-zero to indicate that the command failed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7624-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 23:11:12 2010
Return-Path: <common-issues-return-7624-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26138 invoked from network); 23 Apr 2010 23:11:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 23:11:12 -0000
Received: (qmail 85289 invoked by uid 500); 23 Apr 2010 23:11:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85266 invoked by uid 500); 23 Apr 2010 23:11:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85258 invoked by uid 99); 23 Apr 2010 23:11:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 23:11:12 +0000
X-ASF-Spam-Status: No, hits=-1339.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 23:11:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3NNAo7x014843
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 23:10:51 GMT
Message-ID: <5807281.169461272064250635.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 19:10:50 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6692) Add FileContext#listStatus that
 returns an iterator
In-Reply-To: <253397745.22171270762596744.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6692:
----------------------------------

    Status: Open  (was: Patch Available)

> Add FileContext#listStatus that returns an iterator
> ---------------------------------------------------
>
>                 Key: HADOOP-6692
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6692
>             Project: Hadoop Common
>          Issue Type: Sub-task
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listStatusItor.patch, listStatusItor1.patch
>
>
> Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7625-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 23:13:14 2010
Return-Path: <common-issues-return-7625-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26350 invoked from network); 23 Apr 2010 23:13:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 23:13:13 -0000
Received: (qmail 85706 invoked by uid 500); 23 Apr 2010 23:13:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85668 invoked by uid 500); 23 Apr 2010 23:13:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85660 invoked by uid 99); 23 Apr 2010 23:13:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 23:13:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 23:13:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3NNCo5f014858
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 23:12:50 GMT
Message-ID: <14595053.169511272064370223.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 19:12:50 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6692) Add FileContext#listStatus that
 returns an iterator
In-Reply-To: <253397745.22171270762596744.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6692:
----------------------------------

    Status: Patch Available  (was: Open)

> Add FileContext#listStatus that returns an iterator
> ---------------------------------------------------
>
>                 Key: HADOOP-6692
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6692
>             Project: Hadoop Common
>          Issue Type: Sub-task
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listStatusItor.patch, listStatusItor1.patch
>
>
> Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7626-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 23 23:39:14 2010
Return-Path: <common-issues-return-7626-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31182 invoked from network); 23 Apr 2010 23:39:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Apr 2010 23:39:14 -0000
Received: (qmail 2693 invoked by uid 500); 23 Apr 2010 23:39:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2586 invoked by uid 500); 23 Apr 2010 23:39:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2578 invoked by uid 99); 23 Apr 2010 23:39:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 23:39:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 23:39:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3NNcoAh015006
	for <common-issues@hadoop.apache.org>; Fri, 23 Apr 2010 23:38:50 GMT
Message-ID: <6113794.169831272065930239.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 19:38:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6692) Add FileContext#listStatus that
 returns an iterator
In-Reply-To: <253397745.22171270762596744.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860420#action_12860420 ] 

Hadoop QA commented on HADOOP-6692:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12442712/listStatusItor1.patch
  against trunk revision 937183.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 9 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/476/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/476/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/476/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/476/console

This message is automatically generated.

> Add FileContext#listStatus that returns an iterator
> ---------------------------------------------------
>
>                 Key: HADOOP-6692
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6692
>             Project: Hadoop Common
>          Issue Type: Sub-task
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listStatusItor.patch, listStatusItor1.patch
>
>
> Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7627-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 24 00:01:19 2010
Return-Path: <common-issues-return-7627-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36273 invoked from network); 24 Apr 2010 00:01:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Apr 2010 00:01:16 -0000
Received: (qmail 14855 invoked by uid 500); 24 Apr 2010 00:01:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14827 invoked by uid 500); 24 Apr 2010 00:01:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14819 invoked by uid 99); 24 Apr 2010 00:01:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Apr 2010 00:01:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Apr 2010 00:01:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3O00qFM015169
	for <common-issues@hadoop.apache.org>; Sat, 24 Apr 2010 00:00:52 GMT
Message-ID: <16521354.170171272067252691.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 20:00:52 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6692) Add FileContext#listStatus that
 returns an iterator
In-Reply-To: <253397745.22171270762596744.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860428#action_12860428 ] 

Suresh Srinivas commented on HADOOP-6692:
-----------------------------------------

+1 for the patch.

> Add FileContext#listStatus that returns an iterator
> ---------------------------------------------------
>
>                 Key: HADOOP-6692
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6692
>             Project: Hadoop Common
>          Issue Type: Sub-task
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listStatusItor.patch, listStatusItor1.patch
>
>
> Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7628-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 24 00:03:13 2010
Return-Path: <common-issues-return-7628-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37049 invoked from network); 24 Apr 2010 00:03:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Apr 2010 00:03:12 -0000
Received: (qmail 16694 invoked by uid 500); 24 Apr 2010 00:03:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16643 invoked by uid 500); 24 Apr 2010 00:03:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16635 invoked by uid 99); 24 Apr 2010 00:03:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Apr 2010 00:03:12 +0000
X-ASF-Spam-Status: No, hits=-1340.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Apr 2010 00:03:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3O02pXH015211
	for <common-issues@hadoop.apache.org>; Sat, 24 Apr 2010 00:02:51 GMT
Message-ID: <1908495.170311272067371799.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 20:02:51 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6521) FsPermission:SetUMask not updated to
 use new-style umask setting.
In-Reply-To: <797058971.135641264793314607.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6521:
------------------------------------

           Status: Resolved  (was: Patch Available)
    Fix Version/s: 0.22.0
       Resolution: Fixed

I committed this change.

> FsPermission:SetUMask not updated to use new-style umask setting.
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6521
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6521
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Jakob Homan
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6521.1.patch, hadoop-6521.patch, hadoop-6521.patch, hadoop-6521.rel20.1.patch, hadoop-6521.rel20.patch, hadoop-6521.rel20.patch, hadoop-6521.rel20.patch
>
>
> FsPermission:
> {code}
> 221   /** Set the user file creation mask (umask) */
> 222   public static void setUMask(Configuration conf, FsPermission umask) {                                    
> 223     conf.setInt(UMASK_LABEL, umask.toShort());
> 224   }
> {code}
> Needs to be updated to not use a decimal value. This is a bug introduced by HADOOP-6234.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7629-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 24 00:15:11 2010
Return-Path: <common-issues-return-7629-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38563 invoked from network); 24 Apr 2010 00:15:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Apr 2010 00:15:11 -0000
Received: (qmail 27857 invoked by uid 500); 24 Apr 2010 00:15:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27790 invoked by uid 500); 24 Apr 2010 00:15:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27782 invoked by uid 99); 24 Apr 2010 00:15:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Apr 2010 00:15:11 +0000
X-ASF-Spam-Status: No, hits=-1340.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Apr 2010 00:15:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3O0EoiU015291
	for <common-issues@hadoop.apache.org>; Sat, 24 Apr 2010 00:14:50 GMT
Message-ID: <22891557.170501272068090266.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 20:14:50 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6657) Common portion of MAPREDUCE-1545
In-Reply-To: <74469463.448561269390627214.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-6657:
----------------------------

        Status: Resolved  (was: Patch Available)
    Resolution: Fixed

Thanks! Steve.

> Common portion of MAPREDUCE-1545
> --------------------------------
>
>                 Key: HADOOP-6657
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6657
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6657-trunk-v1.patch, hadoop-6657-trunk-v2.patch, hadoop-6657-trunk-v3.patch
>
>
> Common portion of the MAPREDUCE-1545. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7630-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 24 02:06:12 2010
Return-Path: <common-issues-return-7630-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65510 invoked from network); 24 Apr 2010 02:06:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Apr 2010 02:06:11 -0000
Received: (qmail 70018 invoked by uid 500); 24 Apr 2010 02:06:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69990 invoked by uid 500); 24 Apr 2010 02:06:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69981 invoked by uid 99); 24 Apr 2010 02:06:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Apr 2010 02:06:11 +0000
X-ASF-Spam-Status: No, hits=-1340.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Apr 2010 02:06:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3O25ou4015847
	for <common-issues@hadoop.apache.org>; Sat, 24 Apr 2010 02:05:50 GMT
Message-ID: <16807848.171581272074750214.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 22:05:50 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6710) Symbolic umask for file creation is
 not consistent with posix
In-Reply-To: <8947165.155011271375809341.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6710:
------------------------------------

        Status: Resolved  (was: Patch Available)
      Assignee: Suresh Srinivas
    Resolution: Fixed

Committed the patch.

> Symbolic umask for file creation is not consistent with posix
> -------------------------------------------------------------
>
>                 Key: HADOOP-6710
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6710
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>         Attachments: hadoop-6710.patch
>
>
> Currently both octal and symbolic umask are used to reset the file creation mode bits. This is not consistent with the behavior defined in posix. Making it consistent would avoid confusion to the HDFS users.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7631-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 24 02:32:25 2010
Return-Path: <common-issues-return-7631-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68947 invoked from network); 24 Apr 2010 02:32:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Apr 2010 02:32:25 -0000
Received: (qmail 76344 invoked by uid 500); 24 Apr 2010 02:32:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76304 invoked by uid 500); 24 Apr 2010 02:32:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76296 invoked by uid 99); 24 Apr 2010 02:32:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Apr 2010 02:32:24 +0000
X-ASF-Spam-Status: No, hits=-1340.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Apr 2010 02:32:23 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3O2W3rQ015960
	for <common-issues@hadoop.apache.org>; Sat, 24 Apr 2010 02:32:03 GMT
Message-ID: <20065512.171841272076323119.JavaMail.jira@thor>
Date: Fri, 23 Apr 2010 22:32:03 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6599) Split RPC metrics into summary and
 detailed metrics
In-Reply-To: <230085983.539401267138948106.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860466#action_12860466 ] 

Suresh Srinivas commented on HADOOP-6599:
-----------------------------------------

Currently RPC metrics has the following information:

Context name *rpc* and record name *metrics* has the following:
||Type||Name of the metrics||Description||
|MetricsTimeVaryingLong|ReceivedBytes|Number of bytes in RPC requests received in reporting
period|
|MetricsTimeVaryingLong|SentBytes|Number of bytes sent RPC response in reporting period|
|MetricsTimeVaryingRate|RpcQueueTime|Time a request spent in queue and rate at which RPC requests are received|
|rpcProcessingTime|RpcProcessingTime|Number of RPC request processed and time for processing it|
|MetricsIntValue|NumOpenConnections|Number of open connections to the RPC server|
|MetricsIntValue|callQueueLen|Length of the RPC queue|
|MetricsTimeVaryingInt |rpcAuthenticationFailures|Number of authentication failures at the RPC layer|
|MetricsTimeVaryingInt |rpcAuthenticationSuccesses|Number of authentication success at the RPC layer|
|MetricsTimeVaryingInt |rpcAuthorizationFailures|Number of authorization failure at the RPC layer|

|MetricsTimeVaryingInt |rpcAuthorizationSuccesses|Number of authorization success at the RPC layer|


Context name *rpc* and record name *detailed-metrics* has metrics for each RPC method of type and name set to RPC method name.



> Split RPC metrics into summary and detailed metrics
> ---------------------------------------------------
>
>                 Key: HADOOP-6599
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6599
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>         Attachments: hadoop-6599.1.patch, hadoop-6599.2.patch, hadoop-6599.2.patch, hadoop-6599.patch, hadoop-6599.patch, hadoop-6599.rel20.patch
>
>
> Currently RPC metrics tracks items that provides summary of RPC usage along with more detailed per method statistics that tracks number of method calls made, time spent on that call. Combining both summary and detailed together results in large metrics report which metrics collection systems may not handle. Proposal is to split the metrics into summary and detailed metrics.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7632-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 24 05:12:15 2010
Return-Path: <common-issues-return-7632-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6195 invoked from network); 24 Apr 2010 05:12:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Apr 2010 05:12:14 -0000
Received: (qmail 22344 invoked by uid 500); 24 Apr 2010 05:12:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22222 invoked by uid 500); 24 Apr 2010 05:12:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22207 invoked by uid 99); 24 Apr 2010 05:12:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Apr 2010 05:12:13 +0000
X-ASF-Spam-Status: No, hits=-1341.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Apr 2010 05:12:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3O5BqIO017009
	for <common-issues@hadoop.apache.org>; Sat, 24 Apr 2010 05:11:52 GMT
Message-ID: <16311048.172771272085912088.JavaMail.jira@thor>
Date: Sat, 24 Apr 2010 01:11:52 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6692) Add FileContext#listStatus that
 returns an iterator
In-Reply-To: <253397745.22171270762596744.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6692:
----------------------------------

    Attachment: listStatusItor2.patch

This patch in addition made a change to TestFilterFs to fix the failed test.

> Add FileContext#listStatus that returns an iterator
> ---------------------------------------------------
>
>                 Key: HADOOP-6692
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6692
>             Project: Hadoop Common
>          Issue Type: Sub-task
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listStatusItor.patch, listStatusItor1.patch, listStatusItor2.patch
>
>
> Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7633-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 24 05:12:15 2010
Return-Path: <common-issues-return-7633-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6224 invoked from network); 24 Apr 2010 05:12:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Apr 2010 05:12:15 -0000
Received: (qmail 22395 invoked by uid 500); 24 Apr 2010 05:12:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22275 invoked by uid 500); 24 Apr 2010 05:12:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22224 invoked by uid 99); 24 Apr 2010 05:12:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Apr 2010 05:12:14 +0000
X-ASF-Spam-Status: No, hits=-1341.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Apr 2010 05:12:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3O5BrJK017039
	for <common-issues@hadoop.apache.org>; Sat, 24 Apr 2010 05:11:53 GMT
Message-ID: <14001529.172871272085913264.JavaMail.jira@thor>
Date: Sat, 24 Apr 2010 01:11:53 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6692) Add FileContext#listStatus that
 returns an iterator
In-Reply-To: <253397745.22171270762596744.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6692:
----------------------------------

    Status: Patch Available  (was: Open)

> Add FileContext#listStatus that returns an iterator
> ---------------------------------------------------
>
>                 Key: HADOOP-6692
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6692
>             Project: Hadoop Common
>          Issue Type: Sub-task
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listStatusItor.patch, listStatusItor1.patch, listStatusItor2.patch
>
>
> Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7634-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 24 05:12:16 2010
Return-Path: <common-issues-return-7634-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6286 invoked from network); 24 Apr 2010 05:12:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Apr 2010 05:12:16 -0000
Received: (qmail 22525 invoked by uid 500); 24 Apr 2010 05:12:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22500 invoked by uid 500); 24 Apr 2010 05:12:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22491 invoked by uid 99); 24 Apr 2010 05:12:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Apr 2010 05:12:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Apr 2010 05:12:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3O5BqEj017024
	for <common-issues@hadoop.apache.org>; Sat, 24 Apr 2010 05:11:52 GMT
Message-ID: <9805562.172821272085912796.JavaMail.jira@thor>
Date: Sat, 24 Apr 2010 01:11:52 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6692) Add FileContext#listStatus that
 returns an iterator
In-Reply-To: <253397745.22171270762596744.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6692:
----------------------------------

    Status: Open  (was: Patch Available)

> Add FileContext#listStatus that returns an iterator
> ---------------------------------------------------
>
>                 Key: HADOOP-6692
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6692
>             Project: Hadoop Common
>          Issue Type: Sub-task
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listStatusItor.patch, listStatusItor1.patch, listStatusItor2.patch
>
>
> Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7635-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 24 05:40:12 2010
Return-Path: <common-issues-return-7635-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8695 invoked from network); 24 Apr 2010 05:40:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Apr 2010 05:40:12 -0000
Received: (qmail 32619 invoked by uid 500); 24 Apr 2010 05:40:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32570 invoked by uid 500); 24 Apr 2010 05:40:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32562 invoked by uid 99); 24 Apr 2010 05:40:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Apr 2010 05:40:11 +0000
X-ASF-Spam-Status: No, hits=-1341.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Apr 2010 05:40:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3O5do0u017116
	for <common-issues@hadoop.apache.org>; Sat, 24 Apr 2010 05:39:50 GMT
Message-ID: <32742506.172921272087590373.JavaMail.jira@thor>
Date: Sat, 24 Apr 2010 01:39:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6692) Add FileContext#listStatus that
 returns an iterator
In-Reply-To: <253397745.22171270762596744.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860486#action_12860486 ] 

Hadoop QA commented on HADOOP-6692:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12442734/listStatusItor2.patch
  against trunk revision 937577.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 12 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/477/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/477/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/477/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/477/console

This message is automatically generated.

> Add FileContext#listStatus that returns an iterator
> ---------------------------------------------------
>
>                 Key: HADOOP-6692
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6692
>             Project: Hadoop Common
>          Issue Type: Sub-task
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listStatusItor.patch, listStatusItor1.patch, listStatusItor2.patch
>
>
> Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7636-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 24 07:32:15 2010
Return-Path: <common-issues-return-7636-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27319 invoked from network); 24 Apr 2010 07:32:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Apr 2010 07:32:14 -0000
Received: (qmail 62312 invoked by uid 500); 24 Apr 2010 07:32:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62097 invoked by uid 500); 24 Apr 2010 07:32:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62069 invoked by uid 99); 24 Apr 2010 07:32:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Apr 2010 07:32:12 +0000
X-ASF-Spam-Status: No, hits=-1341.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Apr 2010 07:32:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3O7VoS9017607
	for <common-issues@hadoop.apache.org>; Sat, 24 Apr 2010 07:31:51 GMT
Message-ID: <9520812.173691272094310976.JavaMail.jira@thor>
Date: Sat, 24 Apr 2010 03:31:50 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6521) FsPermission:SetUMask not updated to
 use new-style umask setting.
In-Reply-To: <797058971.135641264793314607.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6521:
------------------------------------

           Issue Type: Test  (was: Bug)
    Affects Version/s: 0.21.0
                       0.22.0

> FsPermission:SetUMask not updated to use new-style umask setting.
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6521
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6521
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Jakob Homan
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6521.1.patch, hadoop-6521.patch, hadoop-6521.patch, hadoop-6521.rel20.1.patch, hadoop-6521.rel20.patch, hadoop-6521.rel20.patch, hadoop-6521.rel20.patch
>
>
> FsPermission:
> {code}
> 221   /** Set the user file creation mask (umask) */
> 222   public static void setUMask(Configuration conf, FsPermission umask) {                                    
> 223     conf.setInt(UMASK_LABEL, umask.toShort());
> 224   }
> {code}
> Needs to be updated to not use a decimal value. This is a bug introduced by HADOOP-6234.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7637-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 24 07:32:15 2010
Return-Path: <common-issues-return-7637-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27328 invoked from network); 24 Apr 2010 07:32:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Apr 2010 07:32:14 -0000
Received: (qmail 62330 invoked by uid 500); 24 Apr 2010 07:32:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62192 invoked by uid 500); 24 Apr 2010 07:32:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62071 invoked by uid 99); 24 Apr 2010 07:32:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Apr 2010 07:32:12 +0000
X-ASF-Spam-Status: No, hits=-1341.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Apr 2010 07:32:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3O7VpPg017625
	for <common-issues@hadoop.apache.org>; Sat, 24 Apr 2010 07:31:51 GMT
Message-ID: <19635654.173751272094311796.JavaMail.jira@thor>
Date: Sat, 24 Apr 2010 03:31:51 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6521) FsPermission:SetUMask not updated to
 use new-style umask setting.
In-Reply-To: <797058971.135641264793314607.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6521:
------------------------------------

    Issue Type: Bug  (was: Test)

> FsPermission:SetUMask not updated to use new-style umask setting.
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6521
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6521
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Jakob Homan
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6521.1.patch, hadoop-6521.patch, hadoop-6521.patch, hadoop-6521.rel20.1.patch, hadoop-6521.rel20.patch, hadoop-6521.rel20.patch, hadoop-6521.rel20.patch
>
>
> FsPermission:
> {code}
> 221   /** Set the user file creation mask (umask) */
> 222   public static void setUMask(Configuration conf, FsPermission umask) {                                    
> 223     conf.setInt(UMASK_LABEL, umask.toShort());
> 224   }
> {code}
> Needs to be updated to not use a decimal value. This is a bug introduced by HADOOP-6234.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7638-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 24 21:34:17 2010
Return-Path: <common-issues-return-7638-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48746 invoked from network); 24 Apr 2010 21:34:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Apr 2010 21:34:16 -0000
Received: (qmail 6647 invoked by uid 500); 24 Apr 2010 21:34:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6581 invoked by uid 500); 24 Apr 2010 21:34:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6568 invoked by uid 99); 24 Apr 2010 21:34:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Apr 2010 21:34:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Apr 2010 21:34:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3OLXp1q020696
	for <common-issues@hadoop.apache.org>; Sat, 24 Apr 2010 21:33:51 GMT
Message-ID: <27568424.177581272144831632.JavaMail.jira@thor>
Date: Sat, 24 Apr 2010 17:33:51 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6698) Revert the io.serialization package
 to 0.20.2's api
In-Reply-To: <21261745.7641270969841016.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860590#action_12860590 ] 

Chris Douglas commented on HADOOP-6698:
---------------------------------------

bq. This issue is fundamentally about backing out changes related to HADOOP-1126, since you've rejected that patch. In that light, HADOOP-6420, HADOOP-6438 and HADOOP-6443 should perhaps also be backed out.

HADOOP-6420 was completed with MAPREDUCE-1126 as a use case, but it's an independent feature. Does it need to be backed out? Also, HADOOP-6438 was closed as invalid, not committed, so fortunately there's nothing to be backed out, there.

> Revert the io.serialization package to 0.20.2's api
> ---------------------------------------------------
>
>                 Key: HADOOP-6698
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6698
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Owen O'Malley
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6698.patch, HADOOP-6698.patch
>
>
> I have a lot of concern about the usability of the new generic serialization framework. Toward that end, I've filed a jira for improving it. There is resistance to pushing a new API into 0.21 at the last moment, so we should back out the changes rather than introducing a new api in 0.21 and deprecating it in 0.22.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7639-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Apr 24 23:42:12 2010
Return-Path: <common-issues-return-7639-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10760 invoked from network); 24 Apr 2010 23:42:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Apr 2010 23:42:12 -0000
Received: (qmail 69662 invoked by uid 500); 24 Apr 2010 23:42:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69632 invoked by uid 500); 24 Apr 2010 23:42:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69624 invoked by uid 99); 24 Apr 2010 23:42:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Apr 2010 23:42:12 +0000
X-ASF-Spam-Status: No, hits=-1342.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Apr 2010 23:42:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3ONfokI022307
	for <common-issues@hadoop.apache.org>; Sat, 24 Apr 2010 23:41:51 GMT
Message-ID: <10234488.178701272152510902.JavaMail.jira@thor>
Date: Sat, 24 Apr 2010 19:41:50 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6722) NetUtils.connect should check that it
 hasn't connected a socket to itself
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

NetUtils.connect should check that it hasn't connected a socket to itself
-------------------------------------------------------------------------

                 Key: HADOOP-6722
                 URL: https://issues.apache.org/jira/browse/HADOOP-6722
             Project: Hadoop Common
          Issue Type: Bug
          Components: util
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon


I had no idea this was possible, but it turns out that a TCP connection will be established in the rare case that the local side of the socket binds to the ephemeral port that you later try to connect to. This can present itself in very very rare occasion when an RPC client is trying to connect to a daemon running on the same node, but that daemon is down. To see what I'm talking about, run "while true ; do telnet localhost 60020 ; done" on a multicore box and wait several minutes.

This can be easily detected in NetUtils.connect by making sure the local address/port is not equal to the remote address/port.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7640-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 25 00:07:12 2010
Return-Path: <common-issues-return-7640-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20946 invoked from network); 25 Apr 2010 00:07:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Apr 2010 00:07:12 -0000
Received: (qmail 76464 invoked by uid 500); 25 Apr 2010 00:07:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76400 invoked by uid 500); 25 Apr 2010 00:07:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76390 invoked by uid 99); 25 Apr 2010 00:07:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 00:07:12 +0000
X-ASF-Spam-Status: No, hits=-1343.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 00:07:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3P06ocf022395
	for <common-issues@hadoop.apache.org>; Sun, 25 Apr 2010 00:06:51 GMT
Message-ID: <10039627.178831272154010542.JavaMail.jira@thor>
Date: Sat, 24 Apr 2010 20:06:50 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6670) UserGroupInformation doesn't support
 use in hash tables
In-Reply-To: <242202762.620161270078590059.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6670:
--------------------------------

    Attachment: 6670-gridmix-test.bugfix.patch

Fixes a testcase that got affected by the UGI changes in hashcode/equals methods.

> UserGroupInformation doesn't support use in hash tables
> -------------------------------------------------------
>
>                 Key: HADOOP-6670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6670
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 6670-gridmix-test.bugfix.patch, fs-close.patch, fs-close.patch, fs-close.patch
>
>
> The UserGroupInformation objects are mutable, but they are used as keys in hash tables. This leads to serious problems in the FileSystem cache and RPC connection cache. We need to change the hashCode to be the identity hash code of the Subject and change equals to use == between the Subjects.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7641-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 25 00:31:11 2010
Return-Path: <common-issues-return-7641-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36407 invoked from network); 25 Apr 2010 00:31:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Apr 2010 00:31:11 -0000
Received: (qmail 83986 invoked by uid 500); 25 Apr 2010 00:31:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83959 invoked by uid 500); 25 Apr 2010 00:31:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83951 invoked by uid 99); 25 Apr 2010 00:31:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 00:31:11 +0000
X-ASF-Spam-Status: No, hits=-1343.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 00:31:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3P0Un1l022439
	for <common-issues@hadoop.apache.org>; Sun, 25 Apr 2010 00:30:50 GMT
Message-ID: <7630396.178871272155449941.JavaMail.jira@thor>
Date: Sat, 24 Apr 2010 20:30:49 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6722) NetUtils.connect should check that it
 hasn't connected a socket to itself
In-Reply-To: <10234488.178701272152510902.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6722:
--------------------------------

    Attachment: hadoop-6722.txt

Here's a patch which detects this situation and throws a ConnectException.

The test case manufactures the problem by binding to an ephemeral port and then trying to connect to itself.

(fwiw, I actually did run into this issue while testing hbase under failure injection scenarios - a client tried to open RPC to the hbase server, but got itself instead, and was very unhappy)

> NetUtils.connect should check that it hasn't connected a socket to itself
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6722
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6722
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-6722.txt
>
>
> I had no idea this was possible, but it turns out that a TCP connection will be established in the rare case that the local side of the socket binds to the ephemeral port that you later try to connect to. This can present itself in very very rare occasion when an RPC client is trying to connect to a daemon running on the same node, but that daemon is down. To see what I'm talking about, run "while true ; do telnet localhost 60020 ; done" on a multicore box and wait several minutes.
> This can be easily detected in NetUtils.connect by making sure the local address/port is not equal to the remote address/port.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7642-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 25 00:31:14 2010
Return-Path: <common-issues-return-7642-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36462 invoked from network); 25 Apr 2010 00:31:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Apr 2010 00:31:14 -0000
Received: (qmail 84132 invoked by uid 500); 25 Apr 2010 00:31:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84093 invoked by uid 500); 25 Apr 2010 00:31:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84085 invoked by uid 99); 25 Apr 2010 00:31:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 00:31:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 00:31:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3P0UogN022445
	for <common-issues@hadoop.apache.org>; Sun, 25 Apr 2010 00:30:50 GMT
Message-ID: <13832946.178891272155450322.JavaMail.jira@thor>
Date: Sat, 24 Apr 2010 20:30:50 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6722) NetUtils.connect should check that it
 hasn't connected a socket to itself
In-Reply-To: <10234488.178701272152510902.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6722:
--------------------------------

    Status: Patch Available  (was: Open)

> NetUtils.connect should check that it hasn't connected a socket to itself
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6722
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6722
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-6722.txt
>
>
> I had no idea this was possible, but it turns out that a TCP connection will be established in the rare case that the local side of the socket binds to the ephemeral port that you later try to connect to. This can present itself in very very rare occasion when an RPC client is trying to connect to a daemon running on the same node, but that daemon is down. To see what I'm talking about, run "while true ; do telnet localhost 60020 ; done" on a multicore box and wait several minutes.
> This can be easily detected in NetUtils.connect by making sure the local address/port is not equal to the remote address/port.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7643-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 25 01:30:12 2010
Return-Path: <common-issues-return-7643-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53385 invoked from network); 25 Apr 2010 01:30:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Apr 2010 01:30:12 -0000
Received: (qmail 97303 invoked by uid 500); 25 Apr 2010 01:30:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97268 invoked by uid 500); 25 Apr 2010 01:30:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97260 invoked by uid 99); 25 Apr 2010 01:30:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 01:30:12 +0000
X-ASF-Spam-Status: No, hits=-1343.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 01:30:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3P1Tohm022580
	for <common-issues@hadoop.apache.org>; Sun, 25 Apr 2010 01:29:50 GMT
Message-ID: <4417161.179021272158990520.JavaMail.jira@thor>
Date: Sat, 24 Apr 2010 21:29:50 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6722) NetUtils.connect should check that
 it hasn't connected a socket to itself
In-Reply-To: <10234488.178701272152510902.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860624#action_12860624 ] 

Hong Tang commented on HADOOP-6722:
-----------------------------------

I was bit by this "feature" of TCP in my past life too. :)

> NetUtils.connect should check that it hasn't connected a socket to itself
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6722
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6722
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-6722.txt
>
>
> I had no idea this was possible, but it turns out that a TCP connection will be established in the rare case that the local side of the socket binds to the ephemeral port that you later try to connect to. This can present itself in very very rare occasion when an RPC client is trying to connect to a daemon running on the same node, but that daemon is down. To see what I'm talking about, run "while true ; do telnet localhost 60020 ; done" on a multicore box and wait several minutes.
> This can be easily detected in NetUtils.connect by making sure the local address/port is not equal to the remote address/port.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7644-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 25 02:12:12 2010
Return-Path: <common-issues-return-7644-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57339 invoked from network); 25 Apr 2010 02:12:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Apr 2010 02:12:12 -0000
Received: (qmail 7757 invoked by uid 500); 25 Apr 2010 02:12:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7728 invoked by uid 500); 25 Apr 2010 02:12:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7720 invoked by uid 99); 25 Apr 2010 02:12:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 02:12:11 +0000
X-ASF-Spam-Status: No, hits=-1343.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 02:12:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3P2Bond022795
	for <common-issues@hadoop.apache.org>; Sun, 25 Apr 2010 02:11:50 GMT
Message-ID: <21334961.179401272161510466.JavaMail.jira@thor>
Date: Sat, 24 Apr 2010 22:11:50 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6722) NetUtils.connect should check that
 it hasn't connected a socket to itself
In-Reply-To: <10234488.178701272152510902.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860631#action_12860631 ] 

Allen Wittenauer commented on HADOOP-6722:
------------------------------------------

FWIW, I can't duplicate this on Solaris or Mac OS X, both of which are based upon BSD sockets.  So I'm guessing this is a bug in the Linux TCP/IP stack.

> NetUtils.connect should check that it hasn't connected a socket to itself
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6722
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6722
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-6722.txt
>
>
> I had no idea this was possible, but it turns out that a TCP connection will be established in the rare case that the local side of the socket binds to the ephemeral port that you later try to connect to. This can present itself in very very rare occasion when an RPC client is trying to connect to a daemon running on the same node, but that daemon is down. To see what I'm talking about, run "while true ; do telnet localhost 60020 ; done" on a multicore box and wait several minutes.
> This can be easily detected in NetUtils.connect by making sure the local address/port is not equal to the remote address/port.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7645-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 25 02:24:12 2010
Return-Path: <common-issues-return-7645-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57900 invoked from network); 25 Apr 2010 02:24:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Apr 2010 02:24:12 -0000
Received: (qmail 8314 invoked by uid 500); 25 Apr 2010 02:24:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8278 invoked by uid 500); 25 Apr 2010 02:24:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8270 invoked by uid 99); 25 Apr 2010 02:24:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 02:24:12 +0000
X-ASF-Spam-Status: No, hits=-1343.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 02:24:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3P2No7j022822
	for <common-issues@hadoop.apache.org>; Sun, 25 Apr 2010 02:23:51 GMT
Message-ID: <1380569.179441272162230848.JavaMail.jira@thor>
Date: Sat, 24 Apr 2010 22:23:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6722) NetUtils.connect should check that
 it hasn't connected a socket to itself
In-Reply-To: <10234488.178701272152510902.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860632#action_12860632 ] 

Hadoop QA commented on HADOOP-6722:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12442761/hadoop-6722.txt
  against trunk revision 937577.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 2 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/478/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/478/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/478/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/478/console

This message is automatically generated.

> NetUtils.connect should check that it hasn't connected a socket to itself
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6722
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6722
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-6722.txt
>
>
> I had no idea this was possible, but it turns out that a TCP connection will be established in the rare case that the local side of the socket binds to the ephemeral port that you later try to connect to. This can present itself in very very rare occasion when an RPC client is trying to connect to a daemon running on the same node, but that daemon is down. To see what I'm talking about, run "while true ; do telnet localhost 60020 ; done" on a multicore box and wait several minutes.
> This can be easily detected in NetUtils.connect by making sure the local address/port is not equal to the remote address/port.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7646-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 25 02:28:13 2010
Return-Path: <common-issues-return-7646-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58077 invoked from network); 25 Apr 2010 02:28:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Apr 2010 02:28:12 -0000
Received: (qmail 8536 invoked by uid 500); 25 Apr 2010 02:28:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8495 invoked by uid 500); 25 Apr 2010 02:28:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8486 invoked by uid 99); 25 Apr 2010 02:28:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 02:28:12 +0000
X-ASF-Spam-Status: No, hits=-1343.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 02:28:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3P2RpAK022840
	for <common-issues@hadoop.apache.org>; Sun, 25 Apr 2010 02:27:51 GMT
Message-ID: <32230337.179481272162471140.JavaMail.jira@thor>
Date: Sat, 24 Apr 2010 22:27:51 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6722) NetUtils.connect should check that
 it hasn't connected a socket to itself
In-Reply-To: <10234488.178701272152510902.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860633#action_12860633 ] 

Hong Tang commented on HADOOP-6722:
-----------------------------------

http://www.rampa.sk/static/tcpLoopConnect.html

This is not a bug, but the usefulness of this feature is certainly questionable.

> NetUtils.connect should check that it hasn't connected a socket to itself
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6722
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6722
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-6722.txt
>
>
> I had no idea this was possible, but it turns out that a TCP connection will be established in the rare case that the local side of the socket binds to the ephemeral port that you later try to connect to. This can present itself in very very rare occasion when an RPC client is trying to connect to a daemon running on the same node, but that daemon is down. To see what I'm talking about, run "while true ; do telnet localhost 60020 ; done" on a multicore box and wait several minutes.
> This can be easily detected in NetUtils.connect by making sure the local address/port is not equal to the remote address/port.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7647-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 25 02:32:30 2010
Return-Path: <common-issues-return-7647-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58226 invoked from network); 25 Apr 2010 02:32:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Apr 2010 02:32:30 -0000
Received: (qmail 10436 invoked by uid 500); 25 Apr 2010 02:32:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10346 invoked by uid 500); 25 Apr 2010 02:32:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10338 invoked by uid 99); 25 Apr 2010 02:32:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 02:32:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 02:32:24 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3P2W2MU022858
	for <common-issues@hadoop.apache.org>; Sun, 25 Apr 2010 02:32:02 GMT
Message-ID: <32735426.179521272162722554.JavaMail.jira@thor>
Date: Sat, 24 Apr 2010 22:32:02 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6722) NetUtils.connect should check that
 it hasn't connected a socket to itself
In-Reply-To: <10234488.178701272152510902.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860634#action_12860634 ] 

Hong Tang commented on HADOOP-6722:
-----------------------------------

Also, it seems somebody experienced this problem on freebsd too. http://osdir.com/ml/freebsd.devel.hackers/2002-05/msg00209.html

> NetUtils.connect should check that it hasn't connected a socket to itself
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6722
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6722
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-6722.txt
>
>
> I had no idea this was possible, but it turns out that a TCP connection will be established in the rare case that the local side of the socket binds to the ephemeral port that you later try to connect to. This can present itself in very very rare occasion when an RPC client is trying to connect to a daemon running on the same node, but that daemon is down. To see what I'm talking about, run "while true ; do telnet localhost 60020 ; done" on a multicore box and wait several minutes.
> This can be easily detected in NetUtils.connect by making sure the local address/port is not equal to the remote address/port.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7648-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 25 18:52:16 2010
Return-Path: <common-issues-return-7648-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25954 invoked from network); 25 Apr 2010 18:52:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Apr 2010 18:52:16 -0000
Received: (qmail 94584 invoked by uid 500); 25 Apr 2010 18:52:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94056 invoked by uid 500); 25 Apr 2010 18:52:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94042 invoked by uid 99); 25 Apr 2010 18:52:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 18:52:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 18:52:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3PIpp0F027168
	for <common-issues@hadoop.apache.org>; Sun, 25 Apr 2010 18:51:51 GMT
Message-ID: <23386803.184291272221511446.JavaMail.jira@thor>
Date: Sun, 25 Apr 2010 14:51:51 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6635) Install or deploy source jars to
 maven repo
In-Reply-To: <1995629254.314101268836107275.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6635:
--------------------------------

    Hadoop Flags: [Reviewed]

+1.  I manually tested by applying all three of the patches, generating the source jar files, verifying their contents and starting a Maven-based project in Eclipse that depended on the local versions and verifying that Eclipse pulled in the source jars as necessary.  This will be a big help.

> Install or deploy source jars to maven repo
> -------------------------------------------
>
>                 Key: HADOOP-6635
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6635
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Patrick Angeles
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6635-branch-20.patch, HADOOP-6635.patch
>
>
> Publishing source jars to the local m2 cache or a public maven repo is extremely handy for Hadoop users that want to be able to inspect the Hadoop source from within their IDE.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7649-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 25 19:08:15 2010
Return-Path: <common-issues-return-7649-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33897 invoked from network); 25 Apr 2010 19:08:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Apr 2010 19:08:15 -0000
Received: (qmail 7172 invoked by uid 500); 25 Apr 2010 19:08:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7063 invoked by uid 500); 25 Apr 2010 19:08:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7055 invoked by uid 99); 25 Apr 2010 19:08:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 19:08:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 19:08:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3PJ7pWA027306
	for <common-issues@hadoop.apache.org>; Sun, 25 Apr 2010 19:07:51 GMT
Message-ID: <3318692.184581272222470994.JavaMail.jira@thor>
Date: Sun, 25 Apr 2010 15:07:50 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6635) Install or deploy source jars to
 maven repo
In-Reply-To: <1995629254.314101268836107275.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6635:
--------------------------------

        Status: Resolved  (was: Patch Available)
      Assignee: Patrick Angeles
    Resolution: Fixed

I've committed this. Thanks Patrick!  Resolving as fixed.

> Install or deploy source jars to maven repo
> -------------------------------------------
>
>                 Key: HADOOP-6635
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6635
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>            Reporter: Patrick Angeles
>            Assignee: Patrick Angeles
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6635-branch-20.patch, HADOOP-6635.patch
>
>
> Publishing source jars to the local m2 cache or a public maven repo is extremely handy for Hadoop users that want to be able to inspect the Hadoop source from within their IDE.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7650-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 25 19:26:11 2010
Return-Path: <common-issues-return-7650-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42099 invoked from network); 25 Apr 2010 19:26:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Apr 2010 19:26:11 -0000
Received: (qmail 12780 invoked by uid 500); 25 Apr 2010 19:26:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12756 invoked by uid 500); 25 Apr 2010 19:26:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12748 invoked by uid 99); 25 Apr 2010 19:26:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 19:26:11 +0000
X-ASF-Spam-Status: No, hits=-1344.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 19:26:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3PJPngZ027385
	for <common-issues@hadoop.apache.org>; Sun, 25 Apr 2010 19:25:50 GMT
Message-ID: <33033218.184741272223549956.JavaMail.jira@thor>
Date: Sun, 25 Apr 2010 15:25:49 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6723) unchecked exceptions thrown in IPC
 Connection orphan clients
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

unchecked exceptions thrown in IPC Connection orphan clients
------------------------------------------------------------

                 Key: HADOOP-6723
                 URL: https://issues.apache.org/jira/browse/HADOOP-6723
             Project: Hadoop Common
          Issue Type: Bug
          Components: ipc
    Affects Versions: 0.20.2
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
            Priority: Critical
             Fix For: 0.21.0, 0.22.0


If the server sends back some malformed data, for example,  receiveResponse() can end up with an incorrect call ID. Then, when it tries to find it in the calls map, it will end up with null and throw NPE in receiveResponse. This isn't caught anywhere, so the original IPC client ends up hanging forever instead of catching an exception. Another example is if the writable implementation itself throws an unchecked exception or OOME.

We should catch Throwable in Connection.run() and shut down the connection if we catch one.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7651-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 25 19:30:15 2010
Return-Path: <common-issues-return-7651-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43696 invoked from network); 25 Apr 2010 19:30:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Apr 2010 19:30:15 -0000
Received: (qmail 14631 invoked by uid 500); 25 Apr 2010 19:30:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14601 invoked by uid 500); 25 Apr 2010 19:30:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14593 invoked by uid 99); 25 Apr 2010 19:30:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 19:30:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 19:30:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3PJTpaP027408
	for <common-issues@hadoop.apache.org>; Sun, 25 Apr 2010 19:29:51 GMT
Message-ID: <31654329.184791272223791041.JavaMail.jira@thor>
Date: Sun, 25 Apr 2010 15:29:51 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6722) NetUtils.connect should check that
 it hasn't connected a socket to itself
In-Reply-To: <10234488.178701272152510902.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860742#action_12860742 ] 

Todd Lipcon commented on HADOOP-6722:
-------------------------------------

Hudson bot isn't commenting automatically anymore, but results here:
hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/478/

   [exec] +1 overall.  Here are the results of testing the latest attachment 
     [exec]   http://issues.apache.org/jira/secure/attachment/12442761/hadoop-6722.txt
     [exec]   against trunk revision 937577.
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 2 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec] 
     [exec]     +1 core tests.  The patch passed core unit tests.
     [exec] 
     [exec]     +1 contrib tests.  The patch passed contrib unit tests.

> NetUtils.connect should check that it hasn't connected a socket to itself
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6722
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6722
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-6722.txt
>
>
> I had no idea this was possible, but it turns out that a TCP connection will be established in the rare case that the local side of the socket binds to the ephemeral port that you later try to connect to. This can present itself in very very rare occasion when an RPC client is trying to connect to a daemon running on the same node, but that daemon is down. To see what I'm talking about, run "while true ; do telnet localhost 60020 ; done" on a multicore box and wait several minutes.
> This can be easily detected in NetUtils.connect by making sure the local address/port is not equal to the remote address/port.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7652-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 25 19:32:14 2010
Return-Path: <common-issues-return-7652-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44538 invoked from network); 25 Apr 2010 19:32:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Apr 2010 19:32:14 -0000
Received: (qmail 16340 invoked by uid 500); 25 Apr 2010 19:32:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16310 invoked by uid 500); 25 Apr 2010 19:32:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16302 invoked by uid 99); 25 Apr 2010 19:32:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 19:32:13 +0000
X-ASF-Spam-Status: No, hits=-1344.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 19:32:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3PJVqX9027429
	for <common-issues@hadoop.apache.org>; Sun, 25 Apr 2010 19:31:52 GMT
Message-ID: <6145239.184861272223912226.JavaMail.jira@thor>
Date: Sun, 25 Apr 2010 15:31:52 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6722) NetUtils.connect should check that
 it hasn't connected a socket to itself
In-Reply-To: <10234488.178701272152510902.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860743#action_12860743 ] 

Todd Lipcon commented on HADOOP-6722:
-------------------------------------

btw, Allen: I was able to reproduce on OSX. It just takes a while sometimes.

> NetUtils.connect should check that it hasn't connected a socket to itself
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6722
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6722
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-6722.txt
>
>
> I had no idea this was possible, but it turns out that a TCP connection will be established in the rare case that the local side of the socket binds to the ephemeral port that you later try to connect to. This can present itself in very very rare occasion when an RPC client is trying to connect to a daemon running on the same node, but that daemon is down. To see what I'm talking about, run "while true ; do telnet localhost 60020 ; done" on a multicore box and wait several minutes.
> This can be easily detected in NetUtils.connect by making sure the local address/port is not equal to the remote address/port.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7653-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 25 21:26:12 2010
Return-Path: <common-issues-return-7653-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97696 invoked from network); 25 Apr 2010 21:26:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Apr 2010 21:26:12 -0000
Received: (qmail 74675 invoked by uid 500); 25 Apr 2010 21:26:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74645 invoked by uid 500); 25 Apr 2010 21:26:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74637 invoked by uid 99); 25 Apr 2010 21:26:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 21:26:12 +0000
X-ASF-Spam-Status: No, hits=-1345.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 21:26:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3PLPo79028018
	for <common-issues@hadoop.apache.org>; Sun, 25 Apr 2010 21:25:51 GMT
Message-ID: <10656671.186071272230750735.JavaMail.jira@thor>
Date: Sun, 25 Apr 2010 17:25:50 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6717) Log levels in o.a.h.security.Groups
 too high
In-Reply-To: <5222107.95511271791852375.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6717:
--------------------------------

           Status: Resolved  (was: Patch Available)
     Hadoop Flags: [Reviewed]
    Fix Version/s: 0.22.0
       Resolution: Fixed

+1.  I've committed this.  Thanks Todd.  Resolving as fixed.

> Log levels in o.a.h.security.Groups too high
> --------------------------------------------
>
>                 Key: HADOOP-6717
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6717
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Trivial
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6717.txt
>
>
> The info logs in Groups.java for every getGroups call is causing my unrelated HDFS unit test to run out of memory since it logs so darn much.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7654-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 25 21:32:12 2010
Return-Path: <common-issues-return-7654-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 727 invoked from network); 25 Apr 2010 21:32:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Apr 2010 21:32:11 -0000
Received: (qmail 77172 invoked by uid 500); 25 Apr 2010 21:32:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77130 invoked by uid 500); 25 Apr 2010 21:32:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77122 invoked by uid 99); 25 Apr 2010 21:32:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 21:32:11 +0000
X-ASF-Spam-Status: No, hits=-1345.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 21:32:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3PLVoSV028069
	for <common-issues@hadoop.apache.org>; Sun, 25 Apr 2010 21:31:50 GMT
Message-ID: <25405417.186201272231110495.JavaMail.jira@thor>
Date: Sun, 25 Apr 2010 17:31:50 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6598) Remove verbose logging from the
 Groups class
In-Reply-To: <1329445528.527921267115907870.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860766#action_12860766 ] 

Jakob Homan commented on HADOOP-6598:
-------------------------------------

Looks like HADOOP-6717 partially duplicated this work and has been committed.  We should update the patch.

> Remove verbose logging from the Groups class
> --------------------------------------------
>
>                 Key: HADOOP-6598
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6598
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6598-BP20-Fix.patch, HADOOP-6598-BP20.patch, HADOOP-6598.patch
>
>
> {quote}
> 2010-02-25 08:30:52,269 INFO  security.Groups (Groups.java:<init>(60)) - Group m
> apping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout
> =300000
> ...
> 2010-02-25 08:30:57,872 INFO  security.Groups (Groups.java:getGroups(76)) - Retu
> rning cached groups for 'oom'
> {quote}
> should both be demoted to debug level.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7655-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 25 21:32:15 2010
Return-Path: <common-issues-return-7655-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 804 invoked from network); 25 Apr 2010 21:32:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Apr 2010 21:32:15 -0000
Received: (qmail 77323 invoked by uid 500); 25 Apr 2010 21:32:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77275 invoked by uid 500); 25 Apr 2010 21:32:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77267 invoked by uid 99); 25 Apr 2010 21:32:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 21:32:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 21:32:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3PLVpnX028078
	for <common-issues@hadoop.apache.org>; Sun, 25 Apr 2010 21:31:51 GMT
Message-ID: <25454349.186231272231111478.JavaMail.jira@thor>
Date: Sun, 25 Apr 2010 17:31:51 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6598) Remove verbose logging from the
 Groups class
In-Reply-To: <1329445528.527921267115907870.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6598:
--------------------------------

    Status: Open  (was: Patch Available)

Canceling patch.

> Remove verbose logging from the Groups class
> --------------------------------------------
>
>                 Key: HADOOP-6598
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6598
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6598-BP20-Fix.patch, HADOOP-6598-BP20.patch, HADOOP-6598.patch
>
>
> {quote}
> 2010-02-25 08:30:52,269 INFO  security.Groups (Groups.java:<init>(60)) - Group m
> apping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout
> =300000
> ...
> 2010-02-25 08:30:57,872 INFO  security.Groups (Groups.java:getGroups(76)) - Retu
> rning cached groups for 'oom'
> {quote}
> should both be demoted to debug level.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7656-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 25 21:38:15 2010
Return-Path: <common-issues-return-7656-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3086 invoked from network); 25 Apr 2010 21:38:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Apr 2010 21:38:15 -0000
Received: (qmail 79395 invoked by uid 500); 25 Apr 2010 21:38:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79337 invoked by uid 500); 25 Apr 2010 21:38:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79329 invoked by uid 99); 25 Apr 2010 21:38:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 21:38:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 21:38:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3PLboGu028121
	for <common-issues@hadoop.apache.org>; Sun, 25 Apr 2010 21:37:51 GMT
Message-ID: <25037371.186331272231470954.JavaMail.jira@thor>
Date: Sun, 25 Apr 2010 17:37:50 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6677) InterfaceAudience.LimitedPrivate
 should take a string not an enum
In-Reply-To: <680532766.665461270239927272.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860769#action_12860769 ] 

Jakob Homan commented on HADOOP-6677:
-------------------------------------

I'm OK with the switch to strings; it'll make the annotation more useful to other projects, as Alan said, and it's not like we have any real control over someone else using the APIs. My concern is more about the possibility of making it optional; For LimitedPrivate, in Hadoop at least, we should always be clear about who we're including. Making the list of targets optional would make LimitedPrivate just a confusing version of Private.

> InterfaceAudience.LimitedPrivate should take a string not an enum
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6677
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6677
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Alan Gates
>            Assignee: Tom White
>            Priority: Minor
>         Attachments: HADOOP-6677.patch
>
>
> Trying to keep the list of all possible components sharing limited private interfaces up to date is painful.  As other subprojects beyond HDFS and MR use this interface it will only get worse (see PIG-1311).  If it is converted to a string other subprojects can use it without requiring patches to common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7657-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 25 21:40:15 2010
Return-Path: <common-issues-return-7657-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4001 invoked from network); 25 Apr 2010 21:40:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Apr 2010 21:40:15 -0000
Received: (qmail 80246 invoked by uid 500); 25 Apr 2010 21:40:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80209 invoked by uid 500); 25 Apr 2010 21:40:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80201 invoked by uid 99); 25 Apr 2010 21:40:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 21:40:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Apr 2010 21:40:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3PLdpI9028153
	for <common-issues@hadoop.apache.org>; Sun, 25 Apr 2010 21:39:51 GMT
Message-ID: <29841784.186411272231591178.JavaMail.jira@thor>
Date: Sun, 25 Apr 2010 17:39:51 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6677) InterfaceAudience.LimitedPrivate
 should take a string not an enum
In-Reply-To: <680532766.665461270239927272.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860770#action_12860770 ] 

Jakob Homan commented on HADOOP-6677:
-------------------------------------

So basically, +1 on this patch.  Any changes to the semantics of LiimitedPrivate should be hashed out in another JIRA.

> InterfaceAudience.LimitedPrivate should take a string not an enum
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6677
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6677
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Alan Gates
>            Assignee: Tom White
>            Priority: Minor
>         Attachments: HADOOP-6677.patch
>
>
> Trying to keep the list of all possible components sharing limited private interfaces up to date is painful.  As other subprojects beyond HDFS and MR use this interface it will only get worse (see PIG-1311).  If it is converted to a string other subprojects can use it without requiring patches to common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7658-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Apr 25 23:52:03 2010
Return-Path: <common-issues-return-7658-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59262 invoked from network); 25 Apr 2010 23:52:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Apr 2010 23:52:03 -0000
Received: (qmail 17907 invoked by uid 500); 25 Apr 2010 23:52:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17798 invoked by uid 500); 25 Apr 2010 23:52:02 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Delivered-To: moderator for common-issues@hadoop.apache.org
Received: (qmail 94419 invoked by uid 99); 25 Apr 2010 22:18:32 -0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_HELO_PASS,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of wangq9@cs.rpi.edu designates 128.113.126.12 as permitted sender)
X-Hash: SCtCte|13fe458c8947e3787fd06a3bb5137ff759f170ba|cd1d9aa67da38fb18e385b461386aafa
X-SMTP-From: accepted <wangq9@cs.rpi.edu> harvestman.cs.rpi.edu [128.213.6.26] (harvestman.cs.rpi.edu)
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=cs.rpi.edu; h=
	message-id:date:subject:from:to:mime-version:content-type
	:content-transfer-encoding; s=default; i=
	128.213.6.26@cs.rpi.edu; t=1272233882; x=1272838682; l=2098; bh=
	OS1XXAZtRgLx7QaYqEDq/1QHD/Y=; b=qZUfOBaNKO7RU2Y9J7qSwEJiK53Osrpp
	DpKy8cubHDg0JsBcVRO+nAVLVws2/24OYGbAFcwtEV5qO93q5boiXmei2N2uPlme
	JIJSWhq42ZwZJfrsYsKxg3YGFDmn7lARk124fPYioFfhUOitT52g9UFOe8sB2Mrr
	51bf037dq6E=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=cs.rpi.edu; h=message-id
	:date:subject:from:to:mime-version:content-type
	:content-transfer-encoding; q=dns; s=default; b=f/GCLJSvpo9XXOQ+
	70MoCrYxGqjvbsammWWdv35UHyfQDox0qWT6FTYD2YTUeNe2Imkf1uIqjpcvuhWo
	xy6hnI7Yz8cYAUiLDdo4thsVll6SW1QCw2dOjcimoZ8/AnZwqLGnngI/Td0vHQIK
	gYI6A5OcdtGDcwG+970Tzm2uOGU=
X-Spam-Report: Spam Report from newman.cs.rpi.edu (SA:3.2.5):
	-1.8 ALL_TRUSTED            Passed through trusted hosts only via SMTP
	-0.9 BAYES_00               BODY: Bayesian spam probability is 0 to 1% [score: 0.0000]
	0.0 AWL                    AWL: From: address is in the auto white-list
X-Spam-Info: -2.7, local; ALL_TRUSTED,AWL,BAYES_00
X-Spam-Scanned-By: newman.cs.rpi.edu using SpamAssassin 3.2.5
X-Virus-Scanned-By: newman.cs.rpi.edu
X-Authentication-Warning: harvestman.cs.rpi.edu: httpd set sender to wangq9@cs.rpi.edu using -f
Message-ID: <53326.74.70.176.43.1272233879.squirrel@webmail.cs.rpi.edu>
Date: Sun, 25 Apr 2010 18:17:59 -0400 (EDT)
Subject: Cannot lock storage, no datanode
From: "Qingling Wang" <wangq9@cs.rpi.edu>
To: common-issues@hadoop.apache.org
User-Agent: SquirrelMail/1.4.10a
MIME-Version: 1.0
Content-Type: text/plain;charset=iso-8859-1
Content-Transfer-Encoding: 8bit
X-Priority: 3 (Normal)
Importance: Normal
X-Scanned-By: MIMEDefang 2.67 on 128.113.126.12
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

I'm using Hadoop-0.20.2. I got 3 machines here, one is master and the
other two are slaves. When I started all things using bin/start-all.sh,
one of the salves cannot start datanode, like this:
jps
tasktracker

And here is the datanode log file of this slave
STARTUP_MSG: Starting DataNode
STARTUP_MSG:   host = juno.mri.cs.rpi.edu/128.213.60.20
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 0.20.2
STARTUP_MSG:   build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2010-04-23 17:46:17,525 INFO org.apache.hadoop.hdfs.server.common.Storage:
Cannot lock storage /projects/wcl-nobackup/hadoop/hdfs/data. The directory
is already locked.
2010-04-23 17:46:17,629 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
Cannot lock storage /projects/wcl-nobackup/hadoop/hdfs/data. The directory
is already locked.
        at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:510)
        at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:363)
        at
org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:112)
        at
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:298)
        at
org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:216)
        at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1283)
        at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1238)
        at
org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1246)
        at
org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1368)

2010-04-23 17:46:17,630 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at juno.mri.cs.rpi.edu/128.213.60.20

Could anyone help with this? Thanks.

Hayden

From common-issues-return-7659-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 02:00:12 2010
Return-Path: <common-issues-return-7659-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26157 invoked from network); 26 Apr 2010 02:00:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 02:00:12 -0000
Received: (qmail 66381 invoked by uid 500); 26 Apr 2010 02:00:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66335 invoked by uid 500); 26 Apr 2010 02:00:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66327 invoked by uid 99); 26 Apr 2010 02:00:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 02:00:12 +0000
X-ASF-Spam-Status: No, hits=-1345.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 02:00:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3Q1xod4022156
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 01:59:51 GMT
Message-ID: <1785112.186921272247190863.JavaMail.jira@thor>
Date: Sun, 25 Apr 2010 21:59:50 -0400 (EDT)
From: "Rajiv Chittajallu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6599) Split RPC metrics into summary and
 detailed metrics
In-Reply-To: <230085983.539401267138948106.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860780#action_12860780 ] 

Rajiv Chittajallu commented on HADOOP-6599:
-------------------------------------------

This should probably split at the context level. The current metrics context implementation in hadoop, you can only limit reporting at the context level.

> Split RPC metrics into summary and detailed metrics
> ---------------------------------------------------
>
>                 Key: HADOOP-6599
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6599
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>         Attachments: hadoop-6599.1.patch, hadoop-6599.2.patch, hadoop-6599.2.patch, hadoop-6599.patch, hadoop-6599.patch, hadoop-6599.rel20.patch
>
>
> Currently RPC metrics tracks items that provides summary of RPC usage along with more detailed per method statistics that tracks number of method calls made, time spent on that call. Combining both summary and detailed together results in large metrics report which metrics collection systems may not handle. Proposal is to split the metrics into summary and detailed metrics.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7660-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 02:18:14 2010
Return-Path: <common-issues-return-7660-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33792 invoked from network); 26 Apr 2010 02:18:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 02:18:14 -0000
Received: (qmail 78806 invoked by uid 500); 26 Apr 2010 02:18:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78763 invoked by uid 500); 26 Apr 2010 02:18:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78755 invoked by uid 99); 26 Apr 2010 02:18:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 02:18:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 02:18:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3Q2HoMg003700
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 02:17:50 GMT
Message-ID: <4486259.186941272248270321.JavaMail.jira@thor>
Date: Sun, 25 Apr 2010 22:17:50 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6724) IPC doesn't properly handle IOEs
 thrown by socket factory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

IPC doesn't properly handle IOEs thrown by socket factory
---------------------------------------------------------

                 Key: HADOOP-6724
                 URL: https://issues.apache.org/jira/browse/HADOOP-6724
             Project: Hadoop Common
          Issue Type: Bug
          Components: ipc
    Affects Versions: 0.20.3, 0.21.0, 0.22.0
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon


If the socket factory throws an IOE inside setupIOStreams, then handleConnectionFailure will be called with socket still null, and thus generate an NPE on socket.close(). This ends up orphaning clients, etc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7661-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 02:20:13 2010
Return-Path: <common-issues-return-7661-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34553 invoked from network); 26 Apr 2010 02:20:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 02:20:13 -0000
Received: (qmail 79552 invoked by uid 500); 26 Apr 2010 02:20:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79514 invoked by uid 500); 26 Apr 2010 02:20:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79506 invoked by uid 99); 26 Apr 2010 02:20:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 02:20:12 +0000
X-ASF-Spam-Status: No, hits=-1345.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 02:20:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3Q2Jp3C006294
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 02:19:51 GMT
Message-ID: <11361609.187011272248391582.JavaMail.jira@thor>
Date: Sun, 25 Apr 2010 22:19:51 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6724) IPC doesn't properly handle IOEs
 thrown by socket factory
In-Reply-To: <4486259.186941272248270321.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6724:
--------------------------------

    Attachment: hadoop-6724.txt

> IPC doesn't properly handle IOEs thrown by socket factory
> ---------------------------------------------------------
>
>                 Key: HADOOP-6724
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6724
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.20.3, 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-6724.txt
>
>
> If the socket factory throws an IOE inside setupIOStreams, then handleConnectionFailure will be called with socket still null, and thus generate an NPE on socket.close(). This ends up orphaning clients, etc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7662-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 02:20:16 2010
Return-Path: <common-issues-return-7662-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34740 invoked from network); 26 Apr 2010 02:20:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 02:20:16 -0000
Received: (qmail 79966 invoked by uid 500); 26 Apr 2010 02:20:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79934 invoked by uid 500); 26 Apr 2010 02:20:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79926 invoked by uid 99); 26 Apr 2010 02:20:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 02:20:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 02:20:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3Q2Jqsg006325
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 02:19:52 GMT
Message-ID: <28074545.187111272248392609.JavaMail.jira@thor>
Date: Sun, 25 Apr 2010 22:19:52 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6724) IPC doesn't properly handle IOEs
 thrown by socket factory
In-Reply-To: <4486259.186941272248270321.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6724:
--------------------------------

    Status: Patch Available  (was: Open)

> IPC doesn't properly handle IOEs thrown by socket factory
> ---------------------------------------------------------
>
>                 Key: HADOOP-6724
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6724
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.20.3, 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-6724.txt
>
>
> If the socket factory throws an IOE inside setupIOStreams, then handleConnectionFailure will be called with socket still null, and thus generate an NPE on socket.close(). This ends up orphaning clients, etc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7663-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 02:37:16 2010
Return-Path: <common-issues-return-7663-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41081 invoked from network); 26 Apr 2010 02:37:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 02:37:16 -0000
Received: (qmail 84096 invoked by uid 500); 26 Apr 2010 02:37:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84000 invoked by uid 500); 26 Apr 2010 02:37:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83992 invoked by uid 99); 26 Apr 2010 02:37:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 02:37:15 +0000
X-ASF-Spam-Status: No, hits=-1345.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 02:37:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3Q2ani1008436
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 02:36:50 GMT
Message-ID: <29774751.187151272249409872.JavaMail.jira@thor>
Date: Sun, 25 Apr 2010 22:36:49 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6523) add an expiration date field to
 Token class.
In-Reply-To: <2119481605.137151264795714568.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan reassigned HADOOP-6523:
-----------------------------------

    Assignee: Boris Shkolnik

> add an expiration date field to Token class. 
> ---------------------------------------------
>
>                 Key: HADOOP-6523
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6523
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6523.patch
>
>
> Needed by MAPREDUCE-1430

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7664-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 02:41:15 2010
Return-Path: <common-issues-return-7664-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42481 invoked from network); 26 Apr 2010 02:41:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 02:41:14 -0000
Received: (qmail 84382 invoked by uid 500); 26 Apr 2010 02:41:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84286 invoked by uid 500); 26 Apr 2010 02:41:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84278 invoked by uid 99); 26 Apr 2010 02:41:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 02:41:13 +0000
X-ASF-Spam-Status: No, hits=-1345.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 02:41:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3Q2eqqH010127
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 02:40:52 GMT
Message-ID: <29783386.187171272249652361.JavaMail.jira@thor>
Date: Sun, 25 Apr 2010 22:40:52 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6724) IPC doesn't properly handle IOEs
 thrown by socket factory
In-Reply-To: <4486259.186941272248270321.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860785#action_12860785 ] 

Hadoop QA commented on HADOOP-6724:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12442801/hadoop-6724.txt
  against trunk revision 937881.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/479/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/479/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/479/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/479/console

This message is automatically generated.

> IPC doesn't properly handle IOEs thrown by socket factory
> ---------------------------------------------------------
>
>                 Key: HADOOP-6724
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6724
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.20.3, 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-6724.txt
>
>
> If the socket factory throws an IOE inside setupIOStreams, then handleConnectionFailure will be called with socket still null, and thus generate an NPE on socket.close(). This ends up orphaning clients, etc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7665-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 02:45:12 2010
Return-Path: <common-issues-return-7665-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44221 invoked from network); 26 Apr 2010 02:45:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 02:45:12 -0000
Received: (qmail 86250 invoked by uid 500); 26 Apr 2010 02:45:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86167 invoked by uid 500); 26 Apr 2010 02:45:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86159 invoked by uid 99); 26 Apr 2010 02:45:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 02:45:11 +0000
X-ASF-Spam-Status: No, hits=-1345.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 02:45:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3Q2inJU010187
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 02:44:50 GMT
Message-ID: <500439.187271272249889862.JavaMail.jira@thor>
Date: Sun, 25 Apr 2010 22:44:49 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6723) unchecked exceptions thrown in IPC
 Connection orphan clients
In-Reply-To: <33033218.184741272223549956.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6723:
--------------------------------

    Attachment: hadoop-6723.txt

Patch simply catches the exception and closes down the connection. The included test case times out without this patch, since the call waits forever.

> unchecked exceptions thrown in IPC Connection orphan clients
> ------------------------------------------------------------
>
>                 Key: HADOOP-6723
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6723
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.20.2
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6723.txt
>
>
> If the server sends back some malformed data, for example,  receiveResponse() can end up with an incorrect call ID. Then, when it tries to find it in the calls map, it will end up with null and throw NPE in receiveResponse. This isn't caught anywhere, so the original IPC client ends up hanging forever instead of catching an exception. Another example is if the writable implementation itself throws an unchecked exception or OOME.
> We should catch Throwable in Connection.run() and shut down the connection if we catch one.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7666-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 02:45:14 2010
Return-Path: <common-issues-return-7666-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44276 invoked from network); 26 Apr 2010 02:45:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 02:45:14 -0000
Received: (qmail 86344 invoked by uid 500); 26 Apr 2010 02:45:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86302 invoked by uid 500); 26 Apr 2010 02:45:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86294 invoked by uid 99); 26 Apr 2010 02:45:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 02:45:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 02:45:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3Q2ioq5010193
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 02:44:50 GMT
Message-ID: <25129167.187291272249890467.JavaMail.jira@thor>
Date: Sun, 25 Apr 2010 22:44:50 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6723) unchecked exceptions thrown in IPC
 Connection orphan clients
In-Reply-To: <33033218.184741272223549956.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6723:
--------------------------------

    Status: Patch Available  (was: Open)

> unchecked exceptions thrown in IPC Connection orphan clients
> ------------------------------------------------------------
>
>                 Key: HADOOP-6723
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6723
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.20.2
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6723.txt
>
>
> If the server sends back some malformed data, for example,  receiveResponse() can end up with an incorrect call ID. Then, when it tries to find it in the calls map, it will end up with null and throw NPE in receiveResponse. This isn't caught anywhere, so the original IPC client ends up hanging forever instead of catching an exception. Another example is if the writable implementation itself throws an unchecked exception or OOME.
> We should catch Throwable in Connection.run() and shut down the connection if we catch one.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7667-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 04:09:56 2010
Return-Path: <common-issues-return-7667-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88341 invoked from network); 26 Apr 2010 04:09:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 04:09:56 -0000
Received: (qmail 13861 invoked by uid 500); 26 Apr 2010 04:09:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13814 invoked by uid 500); 26 Apr 2010 04:09:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13806 invoked by uid 99); 26 Apr 2010 04:09:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 04:09:55 +0000
X-ASF-Spam-Status: No, hits=-1345.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 04:09:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3Q49Y2p004148
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 04:09:34 GMT
Message-ID: <29763440.421272254974013.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 00:09:34 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6677) InterfaceAudience.LimitedPrivate
 should take a string not an enum
In-Reply-To: <680532766.665461270239927272.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860796#action_12860796 ] 

Tom White commented on HADOOP-6677:
-----------------------------------

I agree we can discuss a change in semantics elsewhere. I'll commit this tomorrow (Monday) as it will be good to get this change in before they appear in a release, and we start applying the annotations more widely in HADOOP-6668 and MAPREDUCE-1623.

> InterfaceAudience.LimitedPrivate should take a string not an enum
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6677
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6677
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Alan Gates
>            Assignee: Tom White
>            Priority: Minor
>         Attachments: HADOOP-6677.patch
>
>
> Trying to keep the list of all possible components sharing limited private interfaces up to date is painful.  As other subprojects beyond HDFS and MR use this interface it will only get worse (see PIG-1311).  If it is converted to a string other subprojects can use it without requiring patches to common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7668-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 04:16:58 2010
Return-Path: <common-issues-return-7668-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91616 invoked from network); 26 Apr 2010 04:16:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 04:16:58 -0000
Received: (qmail 15126 invoked by uid 500); 26 Apr 2010 04:16:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15022 invoked by uid 500); 26 Apr 2010 04:16:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15014 invoked by uid 99); 26 Apr 2010 04:16:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 04:16:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 04:16:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3Q4GXt3004203
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 04:16:34 GMT
Message-ID: <12187537.551272255393907.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 00:16:33 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6698) Revert the io.serialization package
 to 0.20.2's api
In-Reply-To: <21261745.7641270969841016.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860798#action_12860798 ] 

Tom White commented on HADOOP-6698:
-----------------------------------

The patch I submitted backs out HADOOP-6420 - should I regenerate leaving it in? 

> Revert the io.serialization package to 0.20.2's api
> ---------------------------------------------------
>
>                 Key: HADOOP-6698
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6698
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Owen O'Malley
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6698.patch, HADOOP-6698.patch
>
>
> I have a lot of concern about the usability of the new generic serialization framework. Toward that end, I've filed a jira for improving it. There is resistance to pushing a new API into 0.21 at the last moment, so we should back out the changes rather than introducing a new api in 0.21 and deprecating it in 0.22.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7669-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 04:26:55 2010
Return-Path: <common-issues-return-7669-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95379 invoked from network); 26 Apr 2010 04:26:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 04:26:54 -0000
Received: (qmail 16939 invoked by uid 500); 26 Apr 2010 04:26:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16829 invoked by uid 500); 26 Apr 2010 04:26:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16814 invoked by uid 99); 26 Apr 2010 04:26:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 04:26:53 +0000
X-ASF-Spam-Status: No, hits=-1345.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 04:26:52 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3Q4QWdL004266
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 04:26:32 GMT
Message-ID: <9084581.651272255992465.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 00:26:32 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6725) Evaluate HtmlUnit for unit and
 regression testing webpages
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Evaluate HtmlUnit for unit and regression testing webpages
----------------------------------------------------------

                 Key: HADOOP-6725
                 URL: https://issues.apache.org/jira/browse/HADOOP-6725
             Project: Hadoop Common
          Issue Type: Test
          Components: test
            Reporter: Jakob Homan
            Priority: Minor


HtmlUnit (http://htmlunit.sourceforge.net/) looks like it may be a good tool to help unit testing and evaluating our various webpages throughout the project.  Currently this is done only occasionally in the code (usually falls to being a manual test during release cycles), and when it is done, usually the code to parse the webpage, etc. is re-written each time.  The framework is Apache licensed, so including it won't be an issue.  If it's found to be useful, new JIRAs for HDFS and MR should be opened.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7670-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 05:34:00 2010
Return-Path: <common-issues-return-7670-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24199 invoked from network); 26 Apr 2010 05:34:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 05:34:00 -0000
Received: (qmail 42197 invoked by uid 500); 26 Apr 2010 05:34:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42084 invoked by uid 500); 26 Apr 2010 05:33:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42076 invoked by uid 99); 26 Apr 2010 05:33:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 05:33:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 05:33:57 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3Q5XYEQ004830
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 05:33:34 GMT
Message-ID: <3836635.1891272260014356.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 01:33:34 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6698) Revert the io.serialization package
 to 0.20.2's api
In-Reply-To: <21261745.7641270969841016.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860817#action_12860817 ] 

Chris Douglas commented on HADOOP-6698:
---------------------------------------

bq. The patch I submitted backs out HADOOP-6420 - should I regenerate leaving it in?

Reverting it isn't part of this issue, but I'm +0 on retaining it. Is there another use case, other than the patch in MAPREDUCE-1126?

> Revert the io.serialization package to 0.20.2's api
> ---------------------------------------------------
>
>                 Key: HADOOP-6698
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6698
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Owen O'Malley
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6698.patch, HADOOP-6698.patch
>
>
> I have a lot of concern about the usability of the new generic serialization framework. Toward that end, I've filed a jira for improving it. There is resistance to pushing a new API into 0.21 at the last moment, so we should back out the changes rather than introducing a new api in 0.21 and deprecating it in 0.22.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7671-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 06:27:57 2010
Return-Path: <common-issues-return-7671-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54150 invoked from network); 26 Apr 2010 06:27:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 06:27:57 -0000
Received: (qmail 71893 invoked by uid 500); 26 Apr 2010 06:27:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71777 invoked by uid 500); 26 Apr 2010 06:27:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71769 invoked by uid 99); 26 Apr 2010 06:27:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 06:27:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 06:27:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3Q6RWRI005240
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 06:27:32 GMT
Message-ID: <10870082.2661272263252439.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 02:27:32 -0400 (EDT)
From: "Ravi Gummadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5647) TestJobHistory fails if /tmp/_logs is
 not writable to. Testcase should not depend on /tmp
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated HADOOP-5647:
---------------------------------

    Release Note: Removed dependency of testcase on /tmp and made it to use test.build.data directory instead.

> TestJobHistory fails if /tmp/_logs is not writable to. Testcase should not depend on /tmp
> -----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5647
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5647
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5647.patch, yhadoop20-5647.patch
>
>
> TestJobHistory sets /tmp as hadoop.job.history.user.location to check if the history file is created in that directory or not. If /tmp/_logs is already created by some other user, this test will fail because of not having write permission.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7672-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 10:19:01 2010
Return-Path: <common-issues-return-7672-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66516 invoked from network); 26 Apr 2010 10:19:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 10:19:01 -0000
Received: (qmail 64045 invoked by uid 500); 26 Apr 2010 10:19:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62918 invoked by uid 500); 26 Apr 2010 10:18:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62894 invoked by uid 99); 26 Apr 2010 10:18:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 10:18:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 10:18:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3QAIX9s007378
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 10:18:33 GMT
Message-ID: <11909011.6021272277113014.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 06:18:33 -0400 (EDT)
From: "Johannes Zillmann (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6726) can't control maxRetries in case of
 SocketTimeoutException
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

can't control maxRetries in case of SocketTimeoutException
----------------------------------------------------------

                 Key: HADOOP-6726
                 URL: https://issues.apache.org/jira/browse/HADOOP-6726
             Project: Hadoop Common
          Issue Type: Improvement
          Components: ipc
    Affects Versions: 0.20.2
            Reporter: Johannes Zillmann


One can set _ipc.client.connect.max.retries_ for _org.apache.hadoop.ipc.Client_.
This comes to effect on IOExceptions but not on SocketTimeoutException.
Client$Connection:307:
{code:java}
          } catch (SocketTimeoutException toe) {
            /* The max number of retries is 45,
             * which amounts to 20s*45 = 15 minutes retries.
             */
            handleConnectionFailure(timeoutFailures++, 45, toe);
          } catch (IOException ie) {
            handleConnectionFailure(ioFailures++, maxRetries, ie);
          }
{code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7673-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 10:20:57 2010
Return-Path: <common-issues-return-7673-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67451 invoked from network); 26 Apr 2010 10:20:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 10:20:56 -0000
Received: (qmail 67204 invoked by uid 500); 26 Apr 2010 10:20:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67104 invoked by uid 500); 26 Apr 2010 10:20:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67094 invoked by uid 99); 26 Apr 2010 10:20:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 10:20:55 +0000
X-ASF-Spam-Status: No, hits=-1347.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 10:20:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3QAKXLL007412
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 10:20:34 GMT
Message-ID: <101218.6081272277233759.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 06:20:33 -0400 (EDT)
From: "Johannes Zillmann (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6726) can't control maxRetries in case of
 SocketTimeoutException
In-Reply-To: <11909011.6021272277113014.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860883#action_12860883 ] 

Johannes Zillmann commented on HADOOP-6726:
-------------------------------------------

It would be fine if _ipc.client.connect.max.retries_ would also be applied to SocketTimeoutException or another config value would be introduced!

> can't control maxRetries in case of SocketTimeoutException
> ----------------------------------------------------------
>
>                 Key: HADOOP-6726
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6726
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.20.2
>            Reporter: Johannes Zillmann
>
> One can set _ipc.client.connect.max.retries_ for _org.apache.hadoop.ipc.Client_.
> This comes to effect on IOExceptions but not on SocketTimeoutException.
> Client$Connection:307:
> {code:java}
>           } catch (SocketTimeoutException toe) {
>             /* The max number of retries is 45,
>              * which amounts to 20s*45 = 15 minutes retries.
>              */
>             handleConnectionFailure(timeoutFailures++, 45, toe);
>           } catch (IOException ie) {
>             handleConnectionFailure(ioFailures++, maxRetries, ie);
>           }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7674-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 11:21:02 2010
Return-Path: <common-issues-return-7674-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96672 invoked from network); 26 Apr 2010 11:21:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 11:21:01 -0000
Received: (qmail 20793 invoked by uid 500); 26 Apr 2010 11:21:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20683 invoked by uid 500); 26 Apr 2010 11:21:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20675 invoked by uid 99); 26 Apr 2010 11:21:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 11:21:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 11:20:58 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3QBKauq007873
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 11:20:36 GMT
Message-ID: <9817259.6681272280836439.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 07:20:36 -0400 (EDT)
From: "Thomas Koch (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4675) Current Ganglia metrics
 implementation is incompatible with Ganglia 3.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860896#action_12860896 ] 

Thomas Koch commented on HADOOP-4675:
-------------------------------------

It would be awesome, if somebody could provide a patch for 0.20, then the upcomming Debian Squezze will get this patch and work together with the version of Ganglia in Debian Squeeze. I'm not enough into any of both to work on this.

> Current Ganglia metrics implementation is incompatible with Ganglia 3.1
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-4675
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4675
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: metrics
>    Affects Versions: 0.21.0
>            Reporter: Brian Bockelman
>            Assignee: Brian Bockelman
>             Fix For: 0.21.0
>
>         Attachments: hadoop-4675-2.patch, hadoop-4675-3.patch, HADOOP-4675-4.patch, HADOOP-4675-v5.patch, HADOOP-4675-v6.patch, HADOOP-4675-v7.patch, HADOOP-4675-v8.patch, HADOOP-4675-v9.patch, hadoop-4675.patch
>
>
> Ganglia changed its wire protocol in the 3.1.x series; the current implementation only works for 3.0.x.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7675-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 18:45:59 2010
Return-Path: <common-issues-return-7675-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42384 invoked from network); 26 Apr 2010 18:45:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 18:45:59 -0000
Received: (qmail 24110 invoked by uid 500); 26 Apr 2010 18:45:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24087 invoked by uid 500); 26 Apr 2010 18:45:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24079 invoked by uid 99); 26 Apr 2010 18:45:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 18:45:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 18:45:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3QIjWJv012170
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 18:45:33 GMT
Message-ID: <24529574.14631272307532975.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 14:45:32 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6725) Evaluate HtmlUnit for unit and
 regression testing webpages
In-Reply-To: <9084581.651272255992465.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861040#action_12861040 ] 

Konstantin Boudnik commented on HADOOP-6725:
--------------------------------------------

Seems like what we need is [JSPUnit|http://www.jboss.org/jsfunit/] for our front-end UI is auto-generated by JSP. JSPUnit runs on top of HtmlUnit framework though.

> Evaluate HtmlUnit for unit and regression testing webpages
> ----------------------------------------------------------
>
>                 Key: HADOOP-6725
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6725
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>            Reporter: Jakob Homan
>            Priority: Minor
>
> HtmlUnit (http://htmlunit.sourceforge.net/) looks like it may be a good tool to help unit testing and evaluating our various webpages throughout the project.  Currently this is done only occasionally in the code (usually falls to being a manual test during release cycles), and when it is done, usually the code to parse the webpage, etc. is re-written each time.  The framework is Apache licensed, so including it won't be an issue.  If it's found to be useful, new JIRAs for HDFS and MR should be opened.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7676-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 18:53:10 2010
Return-Path: <common-issues-return-7676-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46348 invoked from network); 26 Apr 2010 18:53:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 18:53:09 -0000
Received: (qmail 30739 invoked by uid 500); 26 Apr 2010 18:53:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30637 invoked by uid 500); 26 Apr 2010 18:53:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30568 invoked by uid 99); 26 Apr 2010 18:53:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 18:53:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 18:53:07 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3QIqk5X012243
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 18:52:46 GMT
Message-ID: <16242805.14831272307966014.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 14:52:46 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6725) Evaluate HtmlUnit for unit and
 regression testing webpages
In-Reply-To: <9084581.651272255992465.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861046#action_12861046 ] 

Jakob Homan commented on HADOOP-6725:
-------------------------------------

That works too.

> Evaluate HtmlUnit for unit and regression testing webpages
> ----------------------------------------------------------
>
>                 Key: HADOOP-6725
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6725
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>            Reporter: Jakob Homan
>            Priority: Minor
>
> HtmlUnit (http://htmlunit.sourceforge.net/) looks like it may be a good tool to help unit testing and evaluating our various webpages throughout the project.  Currently this is done only occasionally in the code (usually falls to being a manual test during release cycles), and when it is done, usually the code to parse the webpage, etc. is re-written each time.  The framework is Apache licensed, so including it won't be an issue.  If it's found to be useful, new JIRAs for HDFS and MR should be opened.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7677-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 20:36:56 2010
Return-Path: <common-issues-return-7677-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13992 invoked from network); 26 Apr 2010 20:36:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 20:36:56 -0000
Received: (qmail 91474 invoked by uid 500); 26 Apr 2010 20:36:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91453 invoked by uid 500); 26 Apr 2010 20:36:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91445 invoked by uid 99); 26 Apr 2010 20:36:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 20:36:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 20:36:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3QKaW5A013071
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 20:36:32 GMT
Message-ID: <8566780.16531272314192201.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 16:36:32 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6723) unchecked exceptions thrown in IPC
 Connection orphan clients
In-Reply-To: <33033218.184741272223549956.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861085#action_12861085 ] 

Todd Lipcon commented on HADOOP-6723:
-------------------------------------

Hudson didn't post.

http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/480/

 [exec] +1 overall.  Here are the results of testing the latest attachment 
     [exec]   http://issues.apache.org/jira/secure/attachment/12442802/hadoop-6723.txt
     [exec]   against trunk revision 937881.
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec] 
     [exec]     +1 core tests.  The patch passed core unit tests.
     [exec] 
     [exec]     +1 contrib tests.  The patch passed contrib unit tests.

> unchecked exceptions thrown in IPC Connection orphan clients
> ------------------------------------------------------------
>
>                 Key: HADOOP-6723
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6723
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.20.2
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6723.txt
>
>
> If the server sends back some malformed data, for example,  receiveResponse() can end up with an incorrect call ID. Then, when it tries to find it in the calls map, it will end up with null and throw NPE in receiveResponse. This isn't caught anywhere, so the original IPC client ends up hanging forever instead of catching an exception. Another example is if the writable implementation itself throws an unchecked exception or OOME.
> We should catch Throwable in Connection.run() and shut down the connection if we catch one.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7678-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 20:50:55 2010
Return-Path: <common-issues-return-7678-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22147 invoked from network); 26 Apr 2010 20:50:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 20:50:55 -0000
Received: (qmail 14317 invoked by uid 500); 26 Apr 2010 20:50:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14211 invoked by uid 500); 26 Apr 2010 20:50:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14203 invoked by uid 99); 26 Apr 2010 20:50:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 20:50:55 +0000
X-ASF-Spam-Status: No, hits=-1350.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 20:50:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3QKoXFk013343
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 20:50:34 GMT
Message-ID: <28218978.17101272315033769.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 16:50:33 -0400 (EDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6461) webapps aren't located correctly
 post-split
In-Reply-To: <428473464.1261365498148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steve Loughran updated HADOOP-6461:
-----------------------------------

       Priority: Blocker  (was: Major)
    Component/s: util

uprating the severity as without this the webapps may not work, it all depends on the classpath order.

> webapps aren't located correctly post-split
> -------------------------------------------
>
>                 Key: HADOOP-6461
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6461
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Steve Loughran
>            Priority: Blocker
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6461.patch, HADOOP-6461.patch, HADOOP-6461.patch, hadoop-6461.patch, hadoop-6461.patch, hadoop-6461.patch, HADOOP-6461.txt, hadoop-6461.txt, hadoop-6461.txt
>
>
> Post-split, when one builds common it creates an empty build/webapps dir. If you then try to launch the NN for example using HADOOP_HDFS_HOME=/path/to/hdfs hdfs namenode, HttpServer.getWebAppsPath locates the empty common webapps dir, and the NN web UI fails to load.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7679-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 20:56:57 2010
Return-Path: <common-issues-return-7679-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27877 invoked from network); 26 Apr 2010 20:56:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 20:56:57 -0000
Received: (qmail 20288 invoked by uid 500); 26 Apr 2010 20:56:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20247 invoked by uid 500); 26 Apr 2010 20:56:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20238 invoked by uid 99); 26 Apr 2010 20:56:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 20:56:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 20:56:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3QKuXju013460
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 20:56:33 GMT
Message-ID: <25321906.17361272315393692.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 16:56:33 -0400 (EDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6194) Add service base class and tests to
 hadoop-common/util
In-Reply-To: <165334509.1250255474864.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steve Loughran updated HADOOP-6194:
-----------------------------------

    Attachment: HADOOP-6194.patch

This is the most recent version of the lifecycle/service base class for hadoop-common

# Named {{HadoopService}} so that the Service in the security package doesn't cause confusion
# the protected methods are {{serviceStart}} and {{serviceClose{{
# Anything purely of interest to those nodes that have workers is in the abstract subclass {{HadoopServiceWithWorkers}}. Currently there's just one abstract method to return the no of workers, though I'd like there to be a standard way to get the list of  workers and ports and last reported time, which we could then feed to management tools and a standard servlet that would report this up. That is future work, these base classes just set things up.



> Add service base class and tests to hadoop-common/util
> ------------------------------------------------------
>
>                 Key: HADOOP-6194
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6194
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6194-1.patch, HADOOP-6194.patch
>
>
> For a service base class, we need to add to common
> # The Service class
> # A mock Service class (it's not in test, its in src/, because it helps in functional tests downstream)
> # Unit tests
> This is the issue to cover this work

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7680-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 21:02:55 2010
Return-Path: <common-issues-return-7680-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31444 invoked from network); 26 Apr 2010 21:02:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 21:02:55 -0000
Received: (qmail 29598 invoked by uid 500); 26 Apr 2010 21:02:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29565 invoked by uid 500); 26 Apr 2010 21:02:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29557 invoked by uid 99); 26 Apr 2010 21:02:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 21:02:55 +0000
X-ASF-Spam-Status: No, hits=-1350.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 21:02:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3QL2Xf7013523
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 21:02:33 GMT
Message-ID: <14904601.17511272315753494.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 17:02:33 -0400 (EDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6194) Add service base class and tests to
 hadoop-common/util
In-Reply-To: <165334509.1250255474864.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steve Loughran updated HADOOP-6194:
-----------------------------------

           Status: Patch Available  (was: Open)
    Fix Version/s: 0.22.0
                       (was: 0.21.0)

> Add service base class and tests to hadoop-common/util
> ------------------------------------------------------
>
>                 Key: HADOOP-6194
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6194
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6194-1.patch, HADOOP-6194.patch
>
>
> For a service base class, we need to add to common
> # The Service class
> # A mock Service class (it's not in test, its in src/, because it helps in functional tests downstream)
> # Unit tests
> This is the issue to cover this work

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7681-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 21:03:04 2010
Return-Path: <common-issues-return-7681-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31611 invoked from network); 26 Apr 2010 21:03:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 21:03:04 -0000
Received: (qmail 30424 invoked by uid 500); 26 Apr 2010 21:03:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30384 invoked by uid 500); 26 Apr 2010 21:03:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30376 invoked by uid 99); 26 Apr 2010 21:03:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 21:03:03 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 21:03:01 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3QL2e2T013586
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 21:02:40 GMT
Message-ID: <27469060.17721272315760135.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 17:02:40 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6713) The RPC server Listener thread is a
 scalability bottleneck
In-Reply-To: <20387500.4981271656489920.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861108#action_12861108 ] 

Hairong Kuang commented on HADOOP-6713:
---------------------------------------

This is a great idea! Separating "accept" from "read" should also greatly reduce the Connection reset errors observed at the client when NameNode is busy. Dhruba asked me to review this patch. So here are a few comments:

1. Please remove the System.out.println or change it to be a log statement;
2. Listener#run() should remove doRead() else branch;
3. Now that accept is done is a separate thread, doAccept() should accept as many as possible (not limit to up to 10 as in the trunk). Another option is to use a blocking accept channel.
4. Optional: the synchronization between listener thread & read thread is very interesting. It took me a while to figure out that it works. But it seems to me that the code is hard to understand and maintain. Another option is that each reader thread maintains a queue of pending registration channels. After choosing a reader, a listener thread simply adds an accepted channel into its pending queue and then wakes up the reader thread. Each reader thread  registers all the pending channels before select().

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7682-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 21:25:56 2010
Return-Path: <common-issues-return-7682-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45242 invoked from network); 26 Apr 2010 21:25:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 21:25:56 -0000
Received: (qmail 55202 invoked by uid 500); 26 Apr 2010 21:25:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55153 invoked by uid 500); 26 Apr 2010 21:25:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55145 invoked by uid 99); 26 Apr 2010 21:25:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 21:25:56 +0000
X-ASF-Spam-Status: No, hits=-1350.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 21:25:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3QLPYSF013839
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 21:25:34 GMT
Message-ID: <18756309.18371272317134591.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 17:25:34 -0400 (EDT)
From: "Dmytro Molkov (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6713) The RPC server Listener thread is a
 scalability bottleneck
In-Reply-To: <20387500.4981271656489920.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dmytro Molkov updated HADOOP-6713:
----------------------------------

    Attachment: HADOOP-6713.2.patch

modifying the patch to address Hairong's comments:

1. Removed the System.out
2. Removed doRead from listener#run
3. Changed the code to accept all the connections in the queue

4. I wrote a comment on startAdd method to explain the synchronization a little. The reason I went with this decision is to keep the semantics of doAccept. After the iteration with certain channel that channel is registered with the selector and is ready to be read. Otherwise it possibly could affect other parts of the code, when the connection is accepted (doAccept iteration for the channel is over) but connectionList is not modified yet. And the connection is only created when the channel is registered with selector.

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: HADOOP-6713.2.patch, HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7683-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 21:26:01 2010
Return-Path: <common-issues-return-7683-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45322 invoked from network); 26 Apr 2010 21:26:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 21:26:00 -0000
Received: (qmail 55366 invoked by uid 500); 26 Apr 2010 21:26:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55334 invoked by uid 500); 26 Apr 2010 21:26:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55326 invoked by uid 99); 26 Apr 2010 21:26:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 21:26:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 21:25:58 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3QLPb8F013890
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 21:25:37 GMT
Message-ID: <21543601.18541272317137036.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 17:25:37 -0400 (EDT)
From: "Dmytro Molkov (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6713) The RPC server Listener thread is a
 scalability bottleneck
In-Reply-To: <20387500.4981271656489920.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dmytro Molkov updated HADOOP-6713:
----------------------------------

    Status: Open  (was: Patch Available)

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: HADOOP-6713.2.patch, HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7684-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 21:27:57 2010
Return-Path: <common-issues-return-7684-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47022 invoked from network); 26 Apr 2010 21:27:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 21:27:56 -0000
Received: (qmail 57348 invoked by uid 500); 26 Apr 2010 21:27:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57311 invoked by uid 500); 26 Apr 2010 21:27:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57303 invoked by uid 99); 26 Apr 2010 21:27:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 21:27:56 +0000
X-ASF-Spam-Status: No, hits=-1350.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 21:27:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3QLRZNO013943
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 21:27:35 GMT
Message-ID: <30511846.18711272317255191.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 17:27:35 -0400 (EDT)
From: "Dmytro Molkov (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6713) The RPC server Listener thread is a
 scalability bottleneck
In-Reply-To: <20387500.4981271656489920.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dmytro Molkov updated HADOOP-6713:
----------------------------------

    Status: Patch Available  (was: Open)

Resubmitting for Hudson

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: HADOOP-6713.2.patch, HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7685-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 21:44:00 2010
Return-Path: <common-issues-return-7685-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55143 invoked from network); 26 Apr 2010 21:44:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 21:44:00 -0000
Received: (qmail 69649 invoked by uid 500); 26 Apr 2010 21:44:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69598 invoked by uid 500); 26 Apr 2010 21:44:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69590 invoked by uid 99); 26 Apr 2010 21:44:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 21:44:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 21:43:58 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3QLhaDU014150
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 21:43:36 GMT
Message-ID: <12022251.19231272318216558.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 17:43:36 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6713) The RPC server Listener thread is a
 scalability bottleneck
In-Reply-To: <20387500.4981271656489920.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861130#action_12861130 ] 

Hadoop QA commented on HADOOP-6713:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12442902/HADOOP-6713.2.patch
  against trunk revision 938136.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/52/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/52/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/52/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/52/console

This message is automatically generated.

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: HADOOP-6713.2.patch, HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7686-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 21:46:04 2010
Return-Path: <common-issues-return-7686-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56153 invoked from network); 26 Apr 2010 21:46:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 21:46:03 -0000
Received: (qmail 73598 invoked by uid 500); 26 Apr 2010 21:46:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73562 invoked by uid 500); 26 Apr 2010 21:46:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73554 invoked by uid 99); 26 Apr 2010 21:46:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 21:46:03 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 21:46:01 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3QLjdto014192
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 21:45:39 GMT
Message-ID: <11060154.19301272318339010.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 17:45:39 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6194) Add service base class and tests to
 hadoop-common/util
In-Reply-To: <165334509.1250255474864.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861131#action_12861131 ] 

Hadoop QA commented on HADOOP-6194:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12442895/HADOOP-6194.patch
  against trunk revision 938136.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    -1 javac.  The applied patch generated 1026 javac compiler warnings (more than the trunk's current 1024 warnings).

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/481/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/481/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/481/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/481/console

This message is automatically generated.

> Add service base class and tests to hadoop-common/util
> ------------------------------------------------------
>
>                 Key: HADOOP-6194
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6194
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6194-1.patch, HADOOP-6194.patch
>
>
> For a service base class, we need to add to common
> # The Service class
> # A mock Service class (it's not in test, its in src/, because it helps in functional tests downstream)
> # Unit tests
> This is the issue to cover this work

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7687-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 21:46:04 2010
Return-Path: <common-issues-return-7687-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56189 invoked from network); 26 Apr 2010 21:46:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 21:46:04 -0000
Received: (qmail 73725 invoked by uid 500); 26 Apr 2010 21:46:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73695 invoked by uid 500); 26 Apr 2010 21:46:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73687 invoked by uid 99); 26 Apr 2010 21:46:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 21:46:03 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 21:46:01 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3QLjeL4014204
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 21:45:40 GMT
Message-ID: <31761926.19341272318340443.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 17:45:40 -0400 (EDT)
From: "Christian Kunz (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6726) can't control maxRetries in case of
 SocketTimeoutException
In-Reply-To: <11909011.6021272277113014.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861132#action_12861132 ] 

Christian Kunz commented on HADOOP-6726:
----------------------------------------

I completely agree. A client's retry behavior should not be hard-coded into a library. Imagine a high number of clients overwhelming the namenode with requests -- currently there is no easy way for clients to implement  a decent backoff scheme such that the namenode can recover. 

> can't control maxRetries in case of SocketTimeoutException
> ----------------------------------------------------------
>
>                 Key: HADOOP-6726
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6726
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.20.2
>            Reporter: Johannes Zillmann
>
> One can set _ipc.client.connect.max.retries_ for _org.apache.hadoop.ipc.Client_.
> This comes to effect on IOExceptions but not on SocketTimeoutException.
> Client$Connection:307:
> {code:java}
>           } catch (SocketTimeoutException toe) {
>             /* The max number of retries is 45,
>              * which amounts to 20s*45 = 15 minutes retries.
>              */
>             handleConnectionFailure(timeoutFailures++, 45, toe);
>           } catch (IOException ie) {
>             handleConnectionFailure(ioFailures++, maxRetries, ie);
>           }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7688-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 22:45:54 2010
Return-Path: <common-issues-return-7688-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87839 invoked from network); 26 Apr 2010 22:45:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 22:45:53 -0000
Received: (qmail 16320 invoked by uid 500); 26 Apr 2010 22:45:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16281 invoked by uid 500); 26 Apr 2010 22:45:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16272 invoked by uid 99); 26 Apr 2010 22:45:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 22:45:53 +0000
X-ASF-Spam-Status: No, hits=-1350.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 22:45:52 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3QMjWkp007647
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 22:45:32 GMT
Message-ID: <7896991.20651272321932479.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 18:45:32 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6692) Add FileContext#listStatus that
 returns an iterator
In-Reply-To: <253397745.22171270762596744.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6692:
----------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

I've committed this!

> Add FileContext#listStatus that returns an iterator
> ---------------------------------------------------
>
>                 Key: HADOOP-6692
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6692
>             Project: Hadoop Common
>          Issue Type: Sub-task
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listStatusItor.patch, listStatusItor1.patch, listStatusItor2.patch
>
>
> Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7689-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 22:47:57 2010
Return-Path: <common-issues-return-7689-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89213 invoked from network); 26 Apr 2010 22:47:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 22:47:56 -0000
Received: (qmail 19444 invoked by uid 500); 26 Apr 2010 22:47:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19417 invoked by uid 500); 26 Apr 2010 22:47:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19409 invoked by uid 99); 26 Apr 2010 22:47:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 22:47:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 22:47:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3QMlWWu007781
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 22:47:33 GMT
Message-ID: <1043036.20721272322052831.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 18:47:32 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6692) Add FileContext#listStatus that
 returns an iterator
In-Reply-To: <253397745.22171270762596744.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6692:
----------------------------------

    Release Note: This issue adds Iterator<FileStatus> listStatus(Path) to FileContext, moves FileStatus[] listStatus(Path) to FileContext#Util, and adds Iterator<FileStatus> listStatusItor(Path) to AbstractFileSystem which provides a default implementation by using FileStatus[] listStatus(Path).
     Component/s: fs

> Add FileContext#listStatus that returns an iterator
> ---------------------------------------------------
>
>                 Key: HADOOP-6692
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6692
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listStatusItor.patch, listStatusItor1.patch, listStatusItor2.patch
>
>
> Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7690-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 23:15:55 2010
Return-Path: <common-issues-return-7690-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5351 invoked from network); 26 Apr 2010 23:15:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 23:15:54 -0000
Received: (qmail 44292 invoked by uid 500); 26 Apr 2010 23:15:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44198 invoked by uid 500); 26 Apr 2010 23:15:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44190 invoked by uid 99); 26 Apr 2010 23:15:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 23:15:54 +0000
X-ASF-Spam-Status: No, hits=-1351.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 23:15:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3QNFXhN025082
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 23:15:33 GMT
Message-ID: <1959824.21091272323733712.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 19:15:33 -0400 (EDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6708) New file format for very large
 records
In-Reply-To: <23756292.153431271370052193.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861161#action_12861161 ] 

Aaron Kimball commented on HADOOP-6708:
---------------------------------------

Possibly. But then we'd have to wait for the next Avro release as well, and then ensure that this Avro version works with the available Hadoop releases, which would introduce further dependency-management complications. The point of doing this large object format work here is for broader compatibility. Until Avro compatibility in general is improved in Hadoop (e.g., resolution of MAPREDUCE-815), it doesn't seem like this directly facilitates this goal.

> New file format for very large records
> --------------------------------------
>
>                 Key: HADOOP-6708
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6708
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>         Attachments: lobfile.pdf
>
>
> A file format that handles multi-gigabyte records efficiently, with lazy disk access

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7691-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Apr 26 23:35:55 2010
Return-Path: <common-issues-return-7691-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13636 invoked from network); 26 Apr 2010 23:35:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Apr 2010 23:35:55 -0000
Received: (qmail 65699 invoked by uid 500); 26 Apr 2010 23:35:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65668 invoked by uid 500); 26 Apr 2010 23:35:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65660 invoked by uid 99); 26 Apr 2010 23:35:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 23:35:55 +0000
X-ASF-Spam-Status: No, hits=-1351.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Apr 2010 23:35:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3QNZXqo003126
	for <common-issues@hadoop.apache.org>; Mon, 26 Apr 2010 23:35:33 GMT
Message-ID: <2161375.21431272324933481.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 19:35:33 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6678) Remove FileContext#isFile,
 isDirectory and exists
In-Reply-To: <360625036.668801270253667713.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861169#action_12861169 ] 

Hairong Kuang commented on HADOOP-6678:
---------------------------------------

Hi Eli, sorry for the delay for getting back to this issue. I was on vacation & worked on a cluster emergency after I came back.

But I got some time to think more about this proposed FileContext change. I am quite concerned about removing FileContext#exists. This is a useful often-used method that's not easy for a user to write. Do you think if it is a good idea that we still keep this method but put it in FileContext#Util with a comment saying that "be cautious about using this method." and list a few scenarios that could avoid calling this method? 

> Remove FileContext#isFile, isDirectory and exists
> -------------------------------------------------
>
>                 Key: HADOOP-6678
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6678
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Hairong Kuang
>            Assignee: Eli Collins
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6678-1.patch
>
>
> # Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.
> # Remove methods isFile(Path), isDirectory(Path), and exists.
> All these methods are implemented by calling getFileStatus(Path).But most users are not aware of this. They would write code as below: 
> {code}
>   FileContext fc = ..;
>   if (fc.exists(path)) {
>     if (fc.isFile(path)) {
>      ...
>     } else {
>     ...
>     }
>   }
> {code}
> The above code adds unnecessary getFileInfo RPC to NameNode. In our production clusters, we often see that the number of getFileStatus calls is multiple times of the open calls. If we remove isFile, isDirectory, and exists from FileContext, users have to explicitly call getFileStatus first, it is more likely that they will write more efficient code as follow:
> {code}
>   FileContext fc = ...;
>   FileStatus fstatus = fc.getFileStatus(path);
>   if (fstatus.isFile() {
>     ...
>   } else {
>     ...
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7692-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 01:47:55 2010
Return-Path: <common-issues-return-7692-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81919 invoked from network); 27 Apr 2010 01:47:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 01:47:55 -0000
Received: (qmail 37202 invoked by uid 500); 27 Apr 2010 01:47:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37169 invoked by uid 500); 27 Apr 2010 01:47:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37160 invoked by uid 99); 27 Apr 2010 01:47:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 01:47:55 +0000
X-ASF-Spam-Status: No, hits=-1351.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 01:47:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3R1lXfq005640
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 01:47:33 GMT
Message-ID: <17037022.22901272332853694.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 21:47:33 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6526) Need mapping from long principal
 names to local OS user names
In-Reply-To: <786866304.151051264870594555.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861202#action_12861202 ] 

Konstantin Boudnik commented on HADOOP-6526:
--------------------------------------------

Latest patch introduces {{src/test/krb5.conf}} which is needed by a couple of tests only. The use of this configuration file for some tests is enabled by the property java.security.krb5.conf. Kerberos has a bug in the implementation of the logic around this property (see http://bugs.sun.com/view_bug.do?bug_id=6857795)

This badly affects any tests running from under ant environment (i.e. Herriot tests (HADOOP-6332)) and on another hand isn't sufficient for Eclipse environment.


> Need mapping from long principal names to local OS user names
> -------------------------------------------------------------
>
>                 Key: HADOOP-6526
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6526
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: c-6526-y20.patch, c-6526-y20.patch, c-6526.patch, HADOOP-6526-y20.2.patch, HADOOP-6526-y20.4.patch
>
>
> We need a configurable mapping from full user names (eg. omalley@APACHE.ORG) to local user names (eg. omalley). For many organizations it is sufficient to just use the prefix, however, in the case of shared clusters there may be duplicated prefixes. A configurable mapping will let administrators resolve the issue.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7693-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 01:49:56 2010
Return-Path: <common-issues-return-7693-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82726 invoked from network); 27 Apr 2010 01:49:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 01:49:56 -0000
Received: (qmail 39300 invoked by uid 500); 27 Apr 2010 01:49:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39268 invoked by uid 500); 27 Apr 2010 01:49:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39260 invoked by uid 99); 27 Apr 2010 01:49:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 01:49:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 01:49:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3R1nWYM008567
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 01:49:32 GMT
Message-ID: <13539320.22991272332972499.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 21:49:32 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6526) Need mapping from long principal
 names to local OS user names
In-Reply-To: <786866304.151051264870594555.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6526:
---------------------------------------

    Attachment: 3595485.patch

Another issue is that the setting affects _all_ tests. This is especially bad for tests which are running on an actual cluster but from the source workspace i.e. Herriot tests. This settings forces default realm to be set to APACHE.ORG which is non-sensical in environments with different realm names.

A better way is to set this property directly in the functional tests requiring this config file. Other tests shouldn't be affected.

This is dirty hack to workaround the problem, although we shouldn't be modifying the whole build just because of a couple of tests requiring a custom config file.



> Need mapping from long principal names to local OS user names
> -------------------------------------------------------------
>
>                 Key: HADOOP-6526
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6526
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 3595485.patch, c-6526-y20.patch, c-6526-y20.patch, c-6526.patch, HADOOP-6526-y20.2.patch, HADOOP-6526-y20.4.patch
>
>
> We need a configurable mapping from full user names (eg. omalley@APACHE.ORG) to local user names (eg. omalley). For many organizations it is sufficient to just use the prefix, however, in the case of shared clusters there may be duplicated prefixes. A configurable mapping will let administrators resolve the issue.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7695-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 03:48:14 2010
Return-Path: <common-issues-return-7695-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28644 invoked from network); 27 Apr 2010 03:48:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 03:48:13 -0000
Received: (qmail 94296 invoked by uid 500); 27 Apr 2010 03:48:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94059 invoked by uid 500); 27 Apr 2010 03:48:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94050 invoked by uid 99); 27 Apr 2010 03:48:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 03:48:12 +0000
X-ASF-Spam-Status: No, hits=-1352.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 03:48:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3R3lpqh020783
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 03:47:51 GMT
Message-ID: <31640475.23961272340071829.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 23:47:51 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6677) InterfaceAudience.LimitedPrivate
 should take a string not an enum
In-Reply-To: <680532766.665461270239927272.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6677:
------------------------------

    Status: Open  (was: Patch Available)

> InterfaceAudience.LimitedPrivate should take a string not an enum
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6677
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6677
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Alan Gates
>            Assignee: Tom White
>            Priority: Minor
>         Attachments: HADOOP-6677.patch, HADOOP-6677.patch
>
>
> Trying to keep the list of all possible components sharing limited private interfaces up to date is painful.  As other subprojects beyond HDFS and MR use this interface it will only get worse (see PIG-1311).  If it is converted to a string other subprojects can use it without requiring patches to common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7694-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 03:48:14 2010
Return-Path: <common-issues-return-7694-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28623 invoked from network); 27 Apr 2010 03:48:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 03:48:13 -0000
Received: (qmail 94183 invoked by uid 500); 27 Apr 2010 03:48:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94049 invoked by uid 500); 27 Apr 2010 03:48:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94032 invoked by uid 99); 27 Apr 2010 03:48:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 03:48:12 +0000
X-ASF-Spam-Status: No, hits=-1352.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 03:48:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3R3lYRm020762
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 03:47:34 GMT
Message-ID: <12835587.23891272340054549.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 23:47:34 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6677) InterfaceAudience.LimitedPrivate
 should take a string not an enum
In-Reply-To: <680532766.665461270239927272.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6677:
------------------------------

    Attachment: HADOOP-6677.patch

Updated patch for trunk.

> InterfaceAudience.LimitedPrivate should take a string not an enum
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6677
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6677
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Alan Gates
>            Assignee: Tom White
>            Priority: Minor
>         Attachments: HADOOP-6677.patch, HADOOP-6677.patch
>
>
> Trying to keep the list of all possible components sharing limited private interfaces up to date is painful.  As other subprojects beyond HDFS and MR use this interface it will only get worse (see PIG-1311).  If it is converted to a string other subprojects can use it without requiring patches to common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7696-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 03:48:17 2010
Return-Path: <common-issues-return-7696-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28754 invoked from network); 27 Apr 2010 03:48:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 03:48:17 -0000
Received: (qmail 94357 invoked by uid 500); 27 Apr 2010 03:48:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94323 invoked by uid 500); 27 Apr 2010 03:48:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94315 invoked by uid 99); 27 Apr 2010 03:48:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 03:48:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 03:48:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3R3lrOI020804
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 03:47:53 GMT
Message-ID: <14851832.24031272340073091.JavaMail.jira@thor>
Date: Mon, 26 Apr 2010 23:47:53 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6677) InterfaceAudience.LimitedPrivate
 should take a string not an enum
In-Reply-To: <680532766.665461270239927272.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6677:
------------------------------

    Status: Patch Available  (was: Open)

> InterfaceAudience.LimitedPrivate should take a string not an enum
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6677
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6677
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Alan Gates
>            Assignee: Tom White
>            Priority: Minor
>         Attachments: HADOOP-6677.patch, HADOOP-6677.patch
>
>
> Trying to keep the list of all possible components sharing limited private interfaces up to date is painful.  As other subprojects beyond HDFS and MR use this interface it will only get worse (see PIG-1311).  If it is converted to a string other subprojects can use it without requiring patches to common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7697-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 04:13:58 2010
Return-Path: <common-issues-return-7697-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39072 invoked from network); 27 Apr 2010 04:13:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 04:13:58 -0000
Received: (qmail 817 invoked by uid 500); 27 Apr 2010 04:13:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 649 invoked by uid 500); 27 Apr 2010 04:13:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 632 invoked by uid 99); 27 Apr 2010 04:13:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 04:13:56 +0000
X-ASF-Spam-Status: No, hits=-1352.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 04:13:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3R4DZ9u021550
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 04:13:35 GMT
Message-ID: <15262957.24351272341615579.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 00:13:35 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6698) Revert the io.serialization package
 to 0.20.2's api
In-Reply-To: <21261745.7641270969841016.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6698:
------------------------------

    Attachment: HADOOP-6698.patch

Here's a new patch which doesn't revert HADOOP-6420 (we can always revert if separately, if need be).

> Revert the io.serialization package to 0.20.2's api
> ---------------------------------------------------
>
>                 Key: HADOOP-6698
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6698
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Owen O'Malley
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6698.patch, HADOOP-6698.patch, HADOOP-6698.patch
>
>
> I have a lot of concern about the usability of the new generic serialization framework. Toward that end, I've filed a jira for improving it. There is resistance to pushing a new API into 0.21 at the last moment, so we should back out the changes rather than introducing a new api in 0.21 and deprecating it in 0.22.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7698-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 04:14:00 2010
Return-Path: <common-issues-return-7698-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39141 invoked from network); 27 Apr 2010 04:13:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 04:13:59 -0000
Received: (qmail 924 invoked by uid 500); 27 Apr 2010 04:13:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 859 invoked by uid 500); 27 Apr 2010 04:13:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 851 invoked by uid 99); 27 Apr 2010 04:13:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 04:13:59 +0000
X-ASF-Spam-Status: No, hits=-1352.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 04:13:58 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3R4DcU5021616
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 04:13:38 GMT
Message-ID: <19950495.24571272341618412.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 00:13:38 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6698) Revert the io.serialization package
 to 0.20.2's api
In-Reply-To: <21261745.7641270969841016.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6698:
------------------------------

    Status: Patch Available  (was: Open)

> Revert the io.serialization package to 0.20.2's api
> ---------------------------------------------------
>
>                 Key: HADOOP-6698
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6698
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Owen O'Malley
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6698.patch, HADOOP-6698.patch, HADOOP-6698.patch
>
>
> I have a lot of concern about the usability of the new generic serialization framework. Toward that end, I've filed a jira for improving it. There is resistance to pushing a new API into 0.21 at the last moment, so we should back out the changes rather than introducing a new api in 0.21 and deprecating it in 0.22.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7699-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 04:14:01 2010
Return-Path: <common-issues-return-7699-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39209 invoked from network); 27 Apr 2010 04:14:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 04:14:00 -0000
Received: (qmail 1069 invoked by uid 500); 27 Apr 2010 04:14:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1026 invoked by uid 500); 27 Apr 2010 04:14:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1018 invoked by uid 99); 27 Apr 2010 04:14:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 04:14:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 04:13:58 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3R4Db9n021583
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 04:13:37 GMT
Message-ID: <20238804.24461272341617024.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 00:13:37 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6698) Revert the io.serialization package
 to 0.20.2's api
In-Reply-To: <21261745.7641270969841016.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6698:
------------------------------

      Status: Open  (was: Patch Available)
    Assignee: Tom White

> Revert the io.serialization package to 0.20.2's api
> ---------------------------------------------------
>
>                 Key: HADOOP-6698
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6698
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Owen O'Malley
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6698.patch, HADOOP-6698.patch, HADOOP-6698.patch
>
>
> I have a lot of concern about the usability of the new generic serialization framework. Toward that end, I've filed a jira for improving it. There is resistance to pushing a new API into 0.21 at the last moment, so we should back out the changes rather than introducing a new api in 0.21 and deprecating it in 0.22.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7700-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 04:31:56 2010
Return-Path: <common-issues-return-7700-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48284 invoked from network); 27 Apr 2010 04:31:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 04:31:56 -0000
Received: (qmail 8232 invoked by uid 500); 27 Apr 2010 04:31:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8114 invoked by uid 500); 27 Apr 2010 04:31:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8106 invoked by uid 99); 27 Apr 2010 04:31:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 04:31:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 04:31:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3R4VVRo021843
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 04:31:32 GMT
Message-ID: <25342261.25111272342691786.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 00:31:31 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6724) IPC doesn't properly handle IOEs
 thrown by socket factory
In-Reply-To: <4486259.186941272248270321.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861235#action_12861235 ] 

Tom White commented on HADOOP-6724:
-----------------------------------

+1 (Nice Mockito test.)

> IPC doesn't properly handle IOEs thrown by socket factory
> ---------------------------------------------------------
>
>                 Key: HADOOP-6724
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6724
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.20.3, 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-6724.txt
>
>
> If the socket factory throws an IOE inside setupIOStreams, then handleConnectionFailure will be called with socket still null, and thus generate an NPE on socket.close(). This ends up orphaning clients, etc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7701-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 04:35:56 2010
Return-Path: <common-issues-return-7701-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50079 invoked from network); 27 Apr 2010 04:35:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 04:35:55 -0000
Received: (qmail 9421 invoked by uid 500); 27 Apr 2010 04:35:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9302 invoked by uid 500); 27 Apr 2010 04:35:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9290 invoked by uid 99); 27 Apr 2010 04:35:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 04:35:54 +0000
X-ASF-Spam-Status: No, hits=-1352.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 04:35:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3R4ZWr3021877
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 04:35:33 GMT
Message-ID: <5128259.25151272342932881.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 00:35:32 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6722) NetUtils.connect should check that
 it hasn't connected a socket to itself
In-Reply-To: <10234488.178701272152510902.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861236#action_12861236 ] 

Tom White commented on HADOOP-6722:
-----------------------------------

+1

> NetUtils.connect should check that it hasn't connected a socket to itself
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6722
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6722
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-6722.txt
>
>
> I had no idea this was possible, but it turns out that a TCP connection will be established in the rare case that the local side of the socket binds to the ephemeral port that you later try to connect to. This can present itself in very very rare occasion when an RPC client is trying to connect to a daemon running on the same node, but that daemon is down. To see what I'm talking about, run "while true ; do telnet localhost 60020 ; done" on a multicore box and wait several minutes.
> This can be easily detected in NetUtils.connect by making sure the local address/port is not equal to the remote address/port.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7702-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 04:35:58 2010
Return-Path: <common-issues-return-7702-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50160 invoked from network); 27 Apr 2010 04:35:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 04:35:58 -0000
Received: (qmail 9508 invoked by uid 500); 27 Apr 2010 04:35:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9477 invoked by uid 500); 27 Apr 2010 04:35:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9462 invoked by uid 99); 27 Apr 2010 04:35:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 04:35:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 04:35:56 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3R4ZY69021910
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 04:35:34 GMT
Message-ID: <5791448.25261272342934621.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 00:35:34 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6698) Revert the io.serialization package
 to 0.20.2's api
In-Reply-To: <21261745.7641270969841016.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861237#action_12861237 ] 

Hadoop QA commented on HADOOP-6698:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12442922/HADOOP-6698.patch
  against trunk revision 938264.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 12 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/53/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/53/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/53/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/53/console

This message is automatically generated.

> Revert the io.serialization package to 0.20.2's api
> ---------------------------------------------------
>
>                 Key: HADOOP-6698
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6698
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Owen O'Malley
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6698.patch, HADOOP-6698.patch, HADOOP-6698.patch
>
>
> I have a lot of concern about the usability of the new generic serialization framework. Toward that end, I've filed a jira for improving it. There is resistance to pushing a new API into 0.21 at the last moment, so we should back out the changes rather than introducing a new api in 0.21 and deprecating it in 0.22.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7703-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 04:39:55 2010
Return-Path: <common-issues-return-7703-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51667 invoked from network); 27 Apr 2010 04:39:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 04:39:55 -0000
Received: (qmail 10415 invoked by uid 500); 27 Apr 2010 04:39:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10302 invoked by uid 500); 27 Apr 2010 04:39:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10274 invoked by uid 99); 27 Apr 2010 04:39:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 04:39:54 +0000
X-ASF-Spam-Status: No, hits=-1352.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 04:39:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3R4dWan021948
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 04:39:33 GMT
Message-ID: <33427005.25321272343172894.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 00:39:32 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6667) RPC.waitForProxy should retry through
 NoRouteToHostException
In-Reply-To: <1055367164.614271270066767200.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6667:
------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

I've just committed this. Thanks Todd!

> RPC.waitForProxy should retry through NoRouteToHostException
> ------------------------------------------------------------
>
>                 Key: HADOOP-6667
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6667
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-6667.txt
>
>
> RPC.waitForProxy already loops through ConnectExceptions, but NoRouteToHostException is not a subclass of ConnectException. In the case that the NN is on a VIP, the No Route To Host error is reasonably common during a failover, so we should retry through it just the same as the other connection errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7704-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 04:47:54 2010
Return-Path: <common-issues-return-7704-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55012 invoked from network); 27 Apr 2010 04:47:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 04:47:54 -0000
Received: (qmail 12805 invoked by uid 500); 27 Apr 2010 04:47:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12694 invoked by uid 500); 27 Apr 2010 04:47:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12686 invoked by uid 99); 27 Apr 2010 04:47:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 04:47:53 +0000
X-ASF-Spam-Status: No, hits=-1352.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 04:47:52 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3R4lVVG022009
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 04:47:32 GMT
Message-ID: <3522478.25441272343651857.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 00:47:31 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6623) Add StringUtils.split for non-escaped
 single-character separator
In-Reply-To: <621076782.115561267860687191.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6623:
------------------------------

    Status: Open  (was: Patch Available)

Looks good. Since this is an optimization, we should really have a benchmark showing a speed improvement.

> Add StringUtils.split for non-escaped single-character separator
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6623
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6623
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-6623.txt
>
>
> This is for HDFS-1028 but useful generally. String.split("/") for example is way slower than an implementation that is specific to only single-character separators.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7705-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 04:57:55 2010
Return-Path: <common-issues-return-7705-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59328 invoked from network); 27 Apr 2010 04:57:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 04:57:55 -0000
Received: (qmail 21482 invoked by uid 500); 27 Apr 2010 04:57:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21381 invoked by uid 500); 27 Apr 2010 04:57:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21369 invoked by uid 99); 27 Apr 2010 04:57:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 04:57:54 +0000
X-ASF-Spam-Status: No, hits=-1352.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 04:57:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3R4vWlA022068
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 04:57:33 GMT
Message-ID: <4754926.25521272344252778.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 00:57:32 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6714) FsShell 'hadoop fs -text' does not
 support compression codecs
In-Reply-To: <19191606.14561271692729693.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White reassigned HADOOP-6714:
---------------------------------

    Assignee: Patrick Angeles

> FsShell 'hadoop fs -text' does not support compression codecs
> -------------------------------------------------------------
>
>                 Key: HADOOP-6714
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6714
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Patrick Angeles
>            Assignee: Patrick Angeles
>         Attachments: HADOOP-6714.patch
>
>
> Currently, 'hadoop fs -text myfile' looks at the first few magic bytes of a file to determine whether it is gzip compressed or a sequence file. This means 'fs -text' cannot properly decode .deflate or .bz2 files (or other codecs specified via configuration).
> It should be fairly straightforward to add support for other codecs by checking the file extension against the CompressionCodecFactory to retrieve an appropriate Codec.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7706-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 07:09:58 2010
Return-Path: <common-issues-return-7706-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31632 invoked from network); 27 Apr 2010 07:09:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 07:09:58 -0000
Received: (qmail 7338 invoked by uid 500); 27 Apr 2010 07:09:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7218 invoked by uid 500); 27 Apr 2010 07:09:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7210 invoked by uid 99); 27 Apr 2010 07:09:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 07:09:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 07:09:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3R79Xae023234
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 07:09:33 GMT
Message-ID: <29131110.27361272352173284.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 03:09:33 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6677) InterfaceAudience.LimitedPrivate
 should take a string not an enum
In-Reply-To: <680532766.665461270239927272.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861271#action_12861271 ] 

Hadoop QA commented on HADOOP-6677:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12442921/HADOOP-6677.patch
  against trunk revision 938322.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/482/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/482/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/482/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/482/console

This message is automatically generated.

> InterfaceAudience.LimitedPrivate should take a string not an enum
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6677
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6677
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Alan Gates
>            Assignee: Tom White
>            Priority: Minor
>         Attachments: HADOOP-6677.patch, HADOOP-6677.patch
>
>
> Trying to keep the list of all possible components sharing limited private interfaces up to date is painful.  As other subprojects beyond HDFS and MR use this interface it will only get worse (see PIG-1311).  If it is converted to a string other subprojects can use it without requiring patches to common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7707-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 14:10:59 2010
Return-Path: <common-issues-return-7707-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45170 invoked from network); 27 Apr 2010 14:10:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 14:10:58 -0000
Received: (qmail 208 invoked by uid 500); 27 Apr 2010 14:10:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99852 invoked by uid 500); 27 Apr 2010 14:10:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99844 invoked by uid 99); 27 Apr 2010 14:10:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 14:10:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 14:10:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3REAXbM002467
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 14:10:33 GMT
Message-ID: <8291744.33771272377433440.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 10:10:33 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6194) Add service base class and tests to
 hadoop-common/util
In-Reply-To: <165334509.1250255474864.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861404#action_12861404 ] 

Tom White commented on HADOOP-6194:
-----------------------------------

Some comments on the latest patch:

* Naming: maybe "LifecycleService" instead of "HadoopService"? "MockService" could be "MockLifecycleService".
* Also, just "State" instead of "ServiceState", so the full form would be "LifecycleService.State". Some methods already refer to "State", while others refer to "ServiceState", so this change would help make this more regular.
* Javadoc on line 44 of HadoopService is malformed.
* It's unclear what the unit test is testing - the code in HadoopService, or the code in MockService, or both? In general, mocks are used as doubles for testing logic in the real class. Can the code be changed to make the distinction clearer?
* As Chris pointed out, it would be good to know that a real service daemon works with this base code.


> Add service base class and tests to hadoop-common/util
> ------------------------------------------------------
>
>                 Key: HADOOP-6194
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6194
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6194-1.patch, HADOOP-6194.patch
>
>
> For a service base class, we need to add to common
> # The Service class
> # A mock Service class (it's not in test, its in src/, because it helps in functional tests downstream)
> # Unit tests
> This is the issue to cover this work

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7708-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 16:36:55 2010
Return-Path: <common-issues-return-7708-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94401 invoked from network); 27 Apr 2010 16:36:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 16:36:54 -0000
Received: (qmail 11431 invoked by uid 500); 27 Apr 2010 16:36:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11334 invoked by uid 500); 27 Apr 2010 16:36:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11326 invoked by uid 99); 27 Apr 2010 16:36:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 16:36:54 +0000
X-ASF-Spam-Status: No, hits=-1355.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 16:36:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RGaWRJ004379
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 16:36:32 GMT
Message-ID: <8330645.36881272386192489.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 12:36:32 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6623) Add StringUtils.split for non-escaped
 single-character separator
In-Reply-To: <621076782.115561267860687191.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6623:
--------------------------------

    Attachment: hadoop-6623.txt

Attached is an updated patch which adds a main() function to TestStringUtils which acts as a benchmark. Benchmark results;

Java impl #4:1339ms
Java impl #5:1095ms
Java impl #6:1257ms
Java impl #7:1386ms
Java impl #8:1470ms
Java impl #9:1467ms
StringUtils impl #4:274ms
StringUtils impl #5:274ms
StringUtils impl #6:274ms
StringUtils impl #7:277ms
StringUtils impl #8:289ms
StringUtils impl #9:291ms

If I double the number of separators in the test string to 10, results are:

Java impl #4:1407ms
Java impl #5:1411ms
Java impl #6:1449ms
Java impl #7:1443ms
Java impl #8:1641ms
Java impl #9:1409ms
StringUtils impl #4:347ms
StringUtils impl #5:347ms
StringUtils impl #6:346ms
StringUtils impl #7:347ms
StringUtils impl #8:355ms
StringUtils impl #9:346ms


> Add StringUtils.split for non-escaped single-character separator
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6623
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6623
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-6623.txt, hadoop-6623.txt
>
>
> This is for HDFS-1028 but useful generally. String.split("/") for example is way slower than an implementation that is specific to only single-character separators.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7709-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 16:38:57 2010
Return-Path: <common-issues-return-7709-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94610 invoked from network); 27 Apr 2010 16:38:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 16:38:57 -0000
Received: (qmail 12124 invoked by uid 500); 27 Apr 2010 16:38:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12017 invoked by uid 500); 27 Apr 2010 16:38:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12009 invoked by uid 99); 27 Apr 2010 16:38:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 16:38:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 16:38:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RGcWMs008357
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 16:38:33 GMT
Message-ID: <9374912.36911272386312800.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 12:38:32 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6623) Add StringUtils.split for non-escaped
 single-character separator
In-Reply-To: <621076782.115561267860687191.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6623:
--------------------------------

    Status: Patch Available  (was: Open)

> Add StringUtils.split for non-escaped single-character separator
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6623
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6623
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-6623.txt, hadoop-6623.txt
>
>
> This is for HDFS-1028 but useful generally. String.split("/") for example is way slower than an implementation that is specific to only single-character separators.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7710-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 16:55:55 2010
Return-Path: <common-issues-return-7710-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98097 invoked from network); 27 Apr 2010 16:55:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 16:55:55 -0000
Received: (qmail 38810 invoked by uid 500); 27 Apr 2010 16:55:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38778 invoked by uid 500); 27 Apr 2010 16:55:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38770 invoked by uid 99); 27 Apr 2010 16:55:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 16:55:54 +0000
X-ASF-Spam-Status: No, hits=-1355.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 16:55:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RGtWHS027998
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 16:55:34 GMT
Message-ID: <8165324.37131272387332709.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 12:55:32 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6677) InterfaceAudience.LimitedPrivate
 should take a string not an enum
In-Reply-To: <680532766.665461270239927272.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6677:
------------------------------

           Status: Resolved  (was: Patch Available)
     Hadoop Flags: [Reviewed]
    Fix Version/s: 0.22.0
       Resolution: Fixed

I've just committed this.

> InterfaceAudience.LimitedPrivate should take a string not an enum
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6677
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6677
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Alan Gates
>            Assignee: Tom White
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6677.patch, HADOOP-6677.patch
>
>
> Trying to keep the list of all possible components sharing limited private interfaces up to date is painful.  As other subprojects beyond HDFS and MR use this interface it will only get worse (see PIG-1311).  If it is converted to a string other subprojects can use it without requiring patches to common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7711-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 17:20:54 2010
Return-Path: <common-issues-return-7711-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3900 invoked from network); 27 Apr 2010 17:20:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 17:20:54 -0000
Received: (qmail 71860 invoked by uid 500); 27 Apr 2010 17:20:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71828 invoked by uid 500); 27 Apr 2010 17:20:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71820 invoked by uid 99); 27 Apr 2010 17:20:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 17:20:53 +0000
X-ASF-Spam-Status: No, hits=-1355.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 17:20:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RHKWKr018503
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 17:20:32 GMT
Message-ID: <30704813.37611272388832445.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 13:20:32 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6600) mechanism for authorization check for
 inter-server protocols
In-Reply-To: <337583123.549081267171588168.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6600:
-----------------------------------

    Attachment: HADOOP-6600-6.patch

merged with the latest in trunk

> mechanism for authorization check for inter-server protocols
> ------------------------------------------------------------
>
>                 Key: HADOOP-6600
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6600
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6600-1-BP20.patch, HADOOP-6600-1.patch, HADOOP-6600-3-BP20.patch, HADOOP-6600-4-BP20.patch, HADOOP-6600-5.patch, HADOOP-6600-6.patch, HADOOP-6600-BP20-fix.patch, HADOOP-6600.patch
>
>
> allow configuration of authorization for based on protocol (inter-server).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7712-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 17:20:57 2010
Return-Path: <common-issues-return-7712-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3945 invoked from network); 27 Apr 2010 17:20:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 17:20:57 -0000
Received: (qmail 72007 invoked by uid 500); 27 Apr 2010 17:20:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71980 invoked by uid 500); 27 Apr 2010 17:20:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71972 invoked by uid 99); 27 Apr 2010 17:20:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 17:20:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 17:20:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RHKXQp018512
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 17:20:33 GMT
Message-ID: <14423344.37641272388833509.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 13:20:33 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6600) mechanism for authorization check for
 inter-server protocols
In-Reply-To: <337583123.549081267171588168.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6600:
-----------------------------------

    Status: Patch Available  (was: Open)

> mechanism for authorization check for inter-server protocols
> ------------------------------------------------------------
>
>                 Key: HADOOP-6600
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6600
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6600-1-BP20.patch, HADOOP-6600-1.patch, HADOOP-6600-3-BP20.patch, HADOOP-6600-4-BP20.patch, HADOOP-6600-5.patch, HADOOP-6600-6.patch, HADOOP-6600-BP20-fix.patch, HADOOP-6600.patch
>
>
> allow configuration of authorization for based on protocol (inter-server).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7713-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 17:41:00 2010
Return-Path: <common-issues-return-7713-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8091 invoked from network); 27 Apr 2010 17:40:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 17:40:59 -0000
Received: (qmail 94620 invoked by uid 500); 27 Apr 2010 17:40:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94590 invoked by uid 500); 27 Apr 2010 17:40:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94582 invoked by uid 99); 27 Apr 2010 17:40:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 17:40:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 17:40:57 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RHeZ0C018736
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 17:40:36 GMT
Message-ID: <4179769.37951272390035863.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 13:40:35 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6713) The RPC server Listener thread is a
 scalability bottleneck
In-Reply-To: <20387500.4981271656489920.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861466#action_12861466 ] 

Hairong Kuang commented on HADOOP-6713:
---------------------------------------

+1

Note that this patch assigns read channels to readers in a round-robin fashion. Ideally it would be nicer if all readers share a set of selected keys and ready-to-read channels are dynamically assigned to each reader. This would enforce better balance among readers. But I think we could have this improvement at a later time.

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: HADOOP-6713.2.patch, HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7714-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 17:44:57 2010
Return-Path: <common-issues-return-7714-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9014 invoked from network); 27 Apr 2010 17:44:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 17:44:56 -0000
Received: (qmail 1933 invoked by uid 500); 27 Apr 2010 17:44:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1907 invoked by uid 500); 27 Apr 2010 17:44:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1899 invoked by uid 99); 27 Apr 2010 17:44:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 17:44:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 17:44:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RHiWUJ018769
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 17:44:32 GMT
Message-ID: <29523211.38021272390272274.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 13:44:32 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6600) mechanism for authorization check
 for inter-server protocols
In-Reply-To: <337583123.549081267171588168.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861468#action_12861468 ] 

Hadoop QA commented on HADOOP-6600:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12442975/HADOOP-6600-6.patch
  against trunk revision 938563.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/54/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/54/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/54/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/54/console

This message is automatically generated.

> mechanism for authorization check for inter-server protocols
> ------------------------------------------------------------
>
>                 Key: HADOOP-6600
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6600
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6600-1-BP20.patch, HADOOP-6600-1.patch, HADOOP-6600-3-BP20.patch, HADOOP-6600-4-BP20.patch, HADOOP-6600-5.patch, HADOOP-6600-6.patch, HADOOP-6600-BP20-fix.patch, HADOOP-6600.patch
>
>
> allow configuration of authorization for based on protocol (inter-server).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7715-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 18:16:58 2010
Return-Path: <common-issues-return-7715-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24825 invoked from network); 27 Apr 2010 18:16:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 18:16:58 -0000
Received: (qmail 53258 invoked by uid 500); 27 Apr 2010 18:16:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53231 invoked by uid 500); 27 Apr 2010 18:16:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53191 invoked by uid 99); 27 Apr 2010 18:16:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 18:16:57 +0000
X-ASF-Spam-Status: No, hits=-1355.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 18:16:57 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RIGatE019141
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 18:16:36 GMT
Message-ID: <9996549.38671272392196697.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 14:16:36 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6713) The RPC server Listener thread is a
 scalability bottleneck
In-Reply-To: <20387500.4981271656489920.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6713:
----------------------------------

           Status: Resolved  (was: Patch Available)
     Hadoop Flags: [Reviewed]
    Fix Version/s: 0.22.0
       Resolution: Fixed

I've just committed this. Thanks, Dmytro!

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6713.2.patch, HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7716-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 18:41:57 2010
Return-Path: <common-issues-return-7716-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44634 invoked from network); 27 Apr 2010 18:41:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 18:41:56 -0000
Received: (qmail 95426 invoked by uid 500); 27 Apr 2010 18:41:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95395 invoked by uid 500); 27 Apr 2010 18:41:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95387 invoked by uid 99); 27 Apr 2010 18:41:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 18:41:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 18:41:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RIfWss019446
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 18:41:32 GMT
Message-ID: <25091072.38951272393692243.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 14:41:32 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6623) Add StringUtils.split for
 non-escaped single-character separator
In-Reply-To: <621076782.115561267860687191.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861488#action_12861488 ] 

Hadoop QA commented on HADOOP-6623:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12442970/hadoop-6623.txt
  against trunk revision 938590.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/483/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/483/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/483/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/483/console

This message is automatically generated.

> Add StringUtils.split for non-escaped single-character separator
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6623
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6623
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-6623.txt, hadoop-6623.txt
>
>
> This is for HDFS-1028 but useful generally. String.split("/") for example is way slower than an implementation that is specific to only single-character separators.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7717-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 18:43:56 2010
Return-Path: <common-issues-return-7717-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45695 invoked from network); 27 Apr 2010 18:43:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 18:43:56 -0000
Received: (qmail 97896 invoked by uid 500); 27 Apr 2010 18:43:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97869 invoked by uid 500); 27 Apr 2010 18:43:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97860 invoked by uid 99); 27 Apr 2010 18:43:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 18:43:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 18:43:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RIhWhd019519
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 18:43:32 GMT
Message-ID: <8589517.39021272393812574.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 14:43:32 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6701)  Incorrect exit codes for "dfs
 -chown", "dfs -chgrp"
In-Reply-To: <29784448.2801271097111992.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6701:
---------------------------------

    Attachment:     (was: HADOOP-6701.v20s.patch)

>  Incorrect exit codes for "dfs -chown", "dfs -chgrp"
> ----------------------------------------------------
>
>                 Key: HADOOP-6701
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6701
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.19.1, 0.20.0, 0.20.1, 0.20.2
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>            Priority: Minor
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>
> ravi@localhost:~$ hadoop dfs -chgrp abcd /; echo $?
> chgrp: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chown  abcd /; echo $?
> chown: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chmod 755 /DOESNTEXIST; echo $?
> chmod: could not get status for '/DOESNTEXIST': File does not exist: /DOESNTEXIST
> 0
> -
> Exit codes for both of the above invocations should be non-zero to indicate that the command failed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7718-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 18:47:57 2010
Return-Path: <common-issues-return-7718-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48111 invoked from network); 27 Apr 2010 18:47:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 18:47:57 -0000
Received: (qmail 3981 invoked by uid 500); 27 Apr 2010 18:47:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3944 invoked by uid 500); 27 Apr 2010 18:47:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3936 invoked by uid 99); 27 Apr 2010 18:47:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 18:47:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 18:47:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RIlW7q019609
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 18:47:33 GMT
Message-ID: <3943820.39131272394052688.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 14:47:32 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6701)  Incorrect exit codes for "dfs
 -chown", "dfs -chgrp"
In-Reply-To: <29784448.2801271097111992.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6701:
---------------------------------

    Attachment: HADOOP-6701-v20.patch

>  Incorrect exit codes for "dfs -chown", "dfs -chgrp"
> ----------------------------------------------------
>
>                 Key: HADOOP-6701
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6701
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.19.1, 0.20.0, 0.20.1, 0.20.2
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>            Priority: Minor
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6701-trunk.patch, HADOOP-6701-v20.patch
>
>
> ravi@localhost:~$ hadoop dfs -chgrp abcd /; echo $?
> chgrp: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chown  abcd /; echo $?
> chown: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chmod 755 /DOESNTEXIST; echo $?
> chmod: could not get status for '/DOESNTEXIST': File does not exist: /DOESNTEXIST
> 0
> -
> Exit codes for both of the above invocations should be non-zero to indicate that the command failed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7719-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 18:47:58 2010
Return-Path: <common-issues-return-7719-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48130 invoked from network); 27 Apr 2010 18:47:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 18:47:57 -0000
Received: (qmail 4149 invoked by uid 500); 27 Apr 2010 18:47:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4084 invoked by uid 500); 27 Apr 2010 18:47:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4076 invoked by uid 99); 27 Apr 2010 18:47:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 18:47:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 18:47:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RIlXg2019618
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 18:47:33 GMT
Message-ID: <23225552.39161272394053866.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 14:47:33 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6701)  Incorrect exit codes for "dfs
 -chown", "dfs -chgrp"
In-Reply-To: <29784448.2801271097111992.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6701:
---------------------------------

    Attachment: HADOOP-6701-trunk.patch

>  Incorrect exit codes for "dfs -chown", "dfs -chgrp"
> ----------------------------------------------------
>
>                 Key: HADOOP-6701
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6701
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.19.1, 0.20.0, 0.20.1, 0.20.2
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>            Priority: Minor
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6701-trunk.patch, HADOOP-6701-v20.patch
>
>
> ravi@localhost:~$ hadoop dfs -chgrp abcd /; echo $?
> chgrp: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chown  abcd /; echo $?
> chown: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chmod 755 /DOESNTEXIST; echo $?
> chmod: could not get status for '/DOESNTEXIST': File does not exist: /DOESNTEXIST
> 0
> -
> Exit codes for both of the above invocations should be non-zero to indicate that the command failed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7720-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 20:48:54 2010
Return-Path: <common-issues-return-7720-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74204 invoked from network); 27 Apr 2010 20:48:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 20:48:54 -0000
Received: (qmail 50608 invoked by uid 500); 27 Apr 2010 20:48:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50585 invoked by uid 500); 27 Apr 2010 20:48:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50577 invoked by uid 99); 27 Apr 2010 20:48:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 20:48:54 +0000
X-ASF-Spam-Status: No, hits=-1356.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 20:48:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RKmW2Q021487
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 20:48:32 GMT
Message-ID: <19260242.41221272401312673.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 16:48:32 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6693) Add metrics to track kerberos login
 activity
In-Reply-To: <1138587062.22291270763196714.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861531#action_12861531 ] 

Owen O'Malley commented on HADOOP-6693:
---------------------------------------

To figure out the time take for login, we also need the number of logins to divide by. I'd suggest adding the number of successes and failures for keytab logins.

> Add metrics to track kerberos login activity
> --------------------------------------------
>
>                 Key: HADOOP-6693
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6693
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6693.patch
>
>
> Need metrics to track kerberos login activity such as login rate and the time taken for login.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7721-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 22:03:57 2010
Return-Path: <common-issues-return-7721-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18374 invoked from network); 27 Apr 2010 22:03:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 22:03:57 -0000
Received: (qmail 60415 invoked by uid 500); 27 Apr 2010 22:03:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60375 invoked by uid 500); 27 Apr 2010 22:03:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60339 invoked by uid 99); 27 Apr 2010 22:03:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:03:55 +0000
X-ASF-Spam-Status: No, hits=-1357.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:03:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RM3XHc022928
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 22:03:34 GMT
Message-ID: <8376663.43661272405813571.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 18:03:33 -0400 (EDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6194) Add service base class and tests to
 hadoop-common/util
In-Reply-To: <165334509.1250255474864.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steve Loughran updated HADOOP-6194:
-----------------------------------

    Attachment: HADOOP-6194.patch

This is an updated patch which

* uses the name LifecycleService for the class, LifecycleServiceWithWorkers for the subclass intended for services with workers.
* added much more javadocs and even some more test cases to the test

I'll stick up the HDFS services tomorrow; I will synchronise and run all the tests. The big changes there are primarily in adding more shutdown logic to remove all threads; this could be teased out to make the lifecycle patch look smaller. 

I have a patch for the JT and TT too; the only troublespot there is in startup, where I believe it can block forever waiting for the filesystem to go live in startup. There's also MAPREDUCE-437. These are separate issues from how you start and stop the things, just needed to make sure that when you start or stop them they clean up and don't ever block waiting for some external dependency like a functional filesystem.


> Add service base class and tests to hadoop-common/util
> ------------------------------------------------------
>
>                 Key: HADOOP-6194
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6194
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6194-1.patch, HADOOP-6194.patch, HADOOP-6194.patch
>
>
> For a service base class, we need to add to common
> # The Service class
> # A mock Service class (it's not in test, its in src/, because it helps in functional tests downstream)
> # Unit tests
> This is the issue to cover this work

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7722-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 22:03:57 2010
Return-Path: <common-issues-return-7722-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18415 invoked from network); 27 Apr 2010 22:03:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 22:03:57 -0000
Received: (qmail 61526 invoked by uid 500); 27 Apr 2010 22:03:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61492 invoked by uid 500); 27 Apr 2010 22:03:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61457 invoked by uid 99); 27 Apr 2010 22:03:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:03:56 +0000
X-ASF-Spam-Status: No, hits=-1357.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:03:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RM3ZHc022952
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 22:03:35 GMT
Message-ID: <6219512.43741272405815855.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 18:03:35 -0400 (EDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6194) Add service base class and tests to
 hadoop-common/util
In-Reply-To: <165334509.1250255474864.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steve Loughran updated HADOOP-6194:
-----------------------------------

    Status: Open  (was: Patch Available)

> Add service base class and tests to hadoop-common/util
> ------------------------------------------------------
>
>                 Key: HADOOP-6194
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6194
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6194-1.patch, HADOOP-6194.patch, HADOOP-6194.patch
>
>
> For a service base class, we need to add to common
> # The Service class
> # A mock Service class (it's not in test, its in src/, because it helps in functional tests downstream)
> # Unit tests
> This is the issue to cover this work

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7723-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 22:07:56 2010
Return-Path: <common-issues-return-7723-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20371 invoked from network); 27 Apr 2010 22:07:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 22:07:56 -0000
Received: (qmail 68116 invoked by uid 500); 27 Apr 2010 22:07:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68042 invoked by uid 500); 27 Apr 2010 22:07:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68034 invoked by uid 99); 27 Apr 2010 22:07:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:07:56 +0000
X-ASF-Spam-Status: No, hits=-1357.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:07:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RM7ZbP023058
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 22:07:35 GMT
Message-ID: <31382041.43921272406055031.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 18:07:35 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6563:
--------------------------------

    Status: Patch Available  (was: Open)

Kicking hudson.

> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6563-1.patch, hadoop-6563-2.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7724-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 22:07:58 2010
Return-Path: <common-issues-return-7724-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20480 invoked from network); 27 Apr 2010 22:07:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 22:07:57 -0000
Received: (qmail 68345 invoked by uid 500); 27 Apr 2010 22:07:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68298 invoked by uid 500); 27 Apr 2010 22:07:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68290 invoked by uid 99); 27 Apr 2010 22:07:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:07:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:07:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RM7XKT023045
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 22:07:34 GMT
Message-ID: <4232508.43881272406053723.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 18:07:33 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6563:
--------------------------------

    Status: Open  (was: Patch Available)

> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6563-1.patch, hadoop-6563-2.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7725-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 22:10:00 2010
Return-Path: <common-issues-return-7725-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21487 invoked from network); 27 Apr 2010 22:09:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 22:09:57 -0000
Received: (qmail 69380 invoked by uid 500); 27 Apr 2010 22:09:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69345 invoked by uid 500); 27 Apr 2010 22:09:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69337 invoked by uid 99); 27 Apr 2010 22:09:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:09:57 +0000
X-ASF-Spam-Status: No, hits=-1357.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:09:56 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RM9ahl023164
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 22:09:36 GMT
Message-ID: <4612573.44181272406176320.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 18:09:36 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6668:
------------------------------

    Status: Patch Available  (was: Open)

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7726-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 22:10:00 2010
Return-Path: <common-issues-return-7726-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21519 invoked from network); 27 Apr 2010 22:09:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 22:09:58 -0000
Received: (qmail 69519 invoked by uid 500); 27 Apr 2010 22:09:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69482 invoked by uid 500); 27 Apr 2010 22:09:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69474 invoked by uid 99); 27 Apr 2010 22:09:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:09:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:09:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RM9Ye1023119
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 22:09:34 GMT
Message-ID: <16842392.44041272406174104.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 18:09:34 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6668:
------------------------------

    Attachment: HADOOP-6668.patch

New patch with changes for HADOOP-6677.

Using this patch I've generated [javadoc|http://people.apache.org/~tomwhite/HADOOP-6668/docs/api/] and [JDiff with 0.20|http://people.apache.org/~tomwhite/HADOOP-6668/docs/jdiff/changes.html].


> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7727-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 22:10:02 2010
Return-Path: <common-issues-return-7727-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21598 invoked from network); 27 Apr 2010 22:09:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 22:09:59 -0000
Received: (qmail 69658 invoked by uid 500); 27 Apr 2010 22:09:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69624 invoked by uid 500); 27 Apr 2010 22:09:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69616 invoked by uid 99); 27 Apr 2010 22:09:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:09:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:09:56 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RM9ZJC023142
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 22:09:35 GMT
Message-ID: <11770232.44111272406175314.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 18:09:35 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6668:
------------------------------

    Status: Open  (was: Patch Available)

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7728-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 22:11:58 2010
Return-Path: <common-issues-return-7728-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23032 invoked from network); 27 Apr 2010 22:11:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 22:11:58 -0000
Received: (qmail 73092 invoked by uid 500); 27 Apr 2010 22:11:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73060 invoked by uid 500); 27 Apr 2010 22:11:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73052 invoked by uid 99); 27 Apr 2010 22:11:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:11:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:11:56 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RMBYot023230
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 22:11:34 GMT
Message-ID: <1652137.44331272406294367.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 18:11:34 -0400 (EDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6194) Add service base class and tests to
 hadoop-common/util
In-Reply-To: <165334509.1250255474864.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steve Loughran updated HADOOP-6194:
-----------------------------------

    Status: Patch Available  (was: Open)

> Add service base class and tests to hadoop-common/util
> ------------------------------------------------------
>
>                 Key: HADOOP-6194
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6194
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6194-1.patch, HADOOP-6194.patch, HADOOP-6194.patch
>
>
> For a service base class, we need to add to common
> # The Service class
> # A mock Service class (it's not in test, its in src/, because it helps in functional tests downstream)
> # Unit tests
> This is the issue to cover this work

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7729-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 22:23:55 2010
Return-Path: <common-issues-return-7729-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28016 invoked from network); 27 Apr 2010 22:23:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 22:23:54 -0000
Received: (qmail 80363 invoked by uid 500); 27 Apr 2010 22:23:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80320 invoked by uid 500); 27 Apr 2010 22:23:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80312 invoked by uid 99); 27 Apr 2010 22:23:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:23:54 +0000
X-ASF-Spam-Status: No, hits=-1357.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:23:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RMNXI9023444
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 22:23:33 GMT
Message-ID: <18057927.44711272407013101.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 18:23:33 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861587#action_12861587 ] 

Hadoop QA commented on HADOOP-6668:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12443010/HADOOP-6668.patch
  against trunk revision 938590.

    +1 @author.  The patch does not contain any @author tags.

    +0 tests included.  The patch appears to be a documentation patch that doesn't require tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/55/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/55/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/55/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/55/console

This message is automatically generated.

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7730-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 22:29:53 2010
Return-Path: <common-issues-return-7730-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31030 invoked from network); 27 Apr 2010 22:29:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 22:29:53 -0000
Received: (qmail 84523 invoked by uid 500); 27 Apr 2010 22:29:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84499 invoked by uid 500); 27 Apr 2010 22:29:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84491 invoked by uid 99); 27 Apr 2010 22:29:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:29:53 +0000
X-ASF-Spam-Status: No, hits=-1357.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:29:52 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RMTWFj023546
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 22:29:32 GMT
Message-ID: <16855474.44841272407372240.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 18:29:32 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6693) Add metrics to track kerberos login
 activity
In-Reply-To: <1138587062.22291270763196714.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861590#action_12861590 ] 

Suresh Srinivas commented on HADOOP-6693:
-----------------------------------------

Owen, MetricsTimeVaryingRate exposes two pieces of information: Number of operations and average time taken for the operation. Does it cover your requirement?

I will change the patch to include two metrics - one for tracking login success and another for login failure.

> Add metrics to track kerberos login activity
> --------------------------------------------
>
>                 Key: HADOOP-6693
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6693
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6693.patch
>
>
> Need metrics to track kerberos login activity such as login rate and the time taken for login.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7731-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 22:31:54 2010
Return-Path: <common-issues-return-7731-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31775 invoked from network); 27 Apr 2010 22:31:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 22:31:53 -0000
Received: (qmail 85172 invoked by uid 500); 27 Apr 2010 22:31:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85131 invoked by uid 500); 27 Apr 2010 22:31:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85123 invoked by uid 99); 27 Apr 2010 22:31:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:31:53 +0000
X-ASF-Spam-Status: No, hits=-1357.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:31:52 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RMVWbi023577
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 22:31:32 GMT
Message-ID: <5921865.44871272407492430.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 18:31:32 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6693) Add metrics to track kerberos login
 activity
In-Reply-To: <1138587062.22291270763196714.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861591#action_12861591 ] 

Suresh Srinivas commented on HADOOP-6693:
-----------------------------------------

Also should we add metrics for logout?

> Add metrics to track kerberos login activity
> --------------------------------------------
>
>                 Key: HADOOP-6693
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6693
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6693.patch
>
>
> Need metrics to track kerberos login activity such as login rate and the time taken for login.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7732-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 22:41:54 2010
Return-Path: <common-issues-return-7732-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36215 invoked from network); 27 Apr 2010 22:41:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 22:41:53 -0000
Received: (qmail 92634 invoked by uid 500); 27 Apr 2010 22:41:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92592 invoked by uid 500); 27 Apr 2010 22:41:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92584 invoked by uid 99); 27 Apr 2010 22:41:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:41:53 +0000
X-ASF-Spam-Status: No, hits=-1357.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:41:52 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RMfWGL023770
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 22:41:32 GMT
Message-ID: <27108051.45131272408092596.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 18:41:32 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6693) Add metrics to track kerberos login
 activity
In-Reply-To: <1138587062.22291270763196714.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6693:
------------------------------------

    Attachment: HADOOP-6693.1.patch

New patch adds MetricsTimeVaryingRate metrics for login success and failure.

> Add metrics to track kerberos login activity
> --------------------------------------------
>
>                 Key: HADOOP-6693
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6693
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6693.1.patch, HADOOP-6693.patch
>
>
> Need metrics to track kerberos login activity such as login rate and the time taken for login.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7733-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 22:45:59 2010
Return-Path: <common-issues-return-7733-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38521 invoked from network); 27 Apr 2010 22:45:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 22:45:58 -0000
Received: (qmail 97075 invoked by uid 500); 27 Apr 2010 22:45:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97049 invoked by uid 500); 27 Apr 2010 22:45:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97011 invoked by uid 99); 27 Apr 2010 22:45:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:45:58 +0000
X-ASF-Spam-Status: No, hits=-1357.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:45:57 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RMjbBh023865
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 22:45:37 GMT
Message-ID: <17543651.45321272408337430.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 18:45:37 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861602#action_12861602 ] 

Hadoop QA commented on HADOOP-6563:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12441659/hadoop-6563-2.patch
  against trunk revision 938590.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/484/console

This message is automatically generated.

> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6563-1.patch, hadoop-6563-2.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7734-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 22:47:56 2010
Return-Path: <common-issues-return-7734-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39528 invoked from network); 27 Apr 2010 22:47:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 22:47:56 -0000
Received: (qmail 1128 invoked by uid 500); 27 Apr 2010 22:47:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1094 invoked by uid 500); 27 Apr 2010 22:47:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1078 invoked by uid 99); 27 Apr 2010 22:47:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:47:56 +0000
X-ASF-Spam-Status: No, hits=-1357.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:47:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RMlZ2e023918
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 22:47:35 GMT
Message-ID: <29547932.45441272408455082.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 18:47:35 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6703) Prevent renaming a file, symlink or
 directory to itself
In-Reply-To: <23186149.74901271194550994.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6703:
--------------------------------

    Attachment: hadoop-6703-3.patch

Patch attached removed the dependency on HADOOP-6563 and merges with trunk. This one just needs to go in with HDFS-1100 which has a +1. 

> Prevent renaming a file, symlink or directory to itself
> -------------------------------------------------------
>
>                 Key: HADOOP-6703
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6703
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6703-1.patch, hadoop-6703-2.patch, hadoop-6703-3.patch
>
>
> Per HDFS-1088 let's throw a FileAlreadyExistsException if renaming a file, symlink or directory to itself, or a symlink to the file it links to. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7735-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 22:49:57 2010
Return-Path: <common-issues-return-7735-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40480 invoked from network); 27 Apr 2010 22:49:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 22:49:56 -0000
Received: (qmail 2015 invoked by uid 500); 27 Apr 2010 22:49:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1985 invoked by uid 500); 27 Apr 2010 22:49:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1977 invoked by uid 99); 27 Apr 2010 22:49:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:49:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:49:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RMnXOO023971
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 22:49:33 GMT
Message-ID: <13960683.45531272408573306.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 18:49:33 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6703) Prevent renaming a file, symlink or
 directory to itself
In-Reply-To: <23186149.74901271194550994.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6703:
--------------------------------

    Status: Patch Available  (was: Open)

> Prevent renaming a file, symlink or directory to itself
> -------------------------------------------------------
>
>                 Key: HADOOP-6703
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6703
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6703-1.patch, hadoop-6703-2.patch, hadoop-6703-3.patch
>
>
> Per HDFS-1088 let's throw a FileAlreadyExistsException if renaming a file, symlink or directory to itself, or a symlink to the file it links to. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7736-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 22:59:57 2010
Return-Path: <common-issues-return-7736-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44832 invoked from network); 27 Apr 2010 22:59:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 22:59:57 -0000
Received: (qmail 15534 invoked by uid 500); 27 Apr 2010 22:59:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15422 invoked by uid 500); 27 Apr 2010 22:59:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15409 invoked by uid 99); 27 Apr 2010 22:59:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:59:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 22:59:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RMxXhv024129
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 22:59:33 GMT
Message-ID: <7658105.45771272409173127.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 18:59:33 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6194) Add service base class and tests to
 hadoop-common/util
In-Reply-To: <165334509.1250255474864.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861609#action_12861609 ] 

Hadoop QA commented on HADOOP-6194:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12443007/HADOOP-6194.patch
  against trunk revision 938590.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The applied patch generated 1026 javac compiler warnings (more than the trunk's current 1024 warnings).

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/56/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/56/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/56/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/56/console

This message is automatically generated.

> Add service base class and tests to hadoop-common/util
> ------------------------------------------------------
>
>                 Key: HADOOP-6194
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6194
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6194-1.patch, HADOOP-6194.patch, HADOOP-6194.patch
>
>
> For a service base class, we need to add to common
> # The Service class
> # A mock Service class (it's not in test, its in src/, because it helps in functional tests downstream)
> # Unit tests
> This is the issue to cover this work

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7737-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 23:25:58 2010
Return-Path: <common-issues-return-7737-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56015 invoked from network); 27 Apr 2010 23:25:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 23:25:57 -0000
Received: (qmail 30889 invoked by uid 500); 27 Apr 2010 23:25:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30846 invoked by uid 500); 27 Apr 2010 23:25:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30838 invoked by uid 99); 27 Apr 2010 23:25:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 23:25:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 23:25:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RNPXj0024388
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 23:25:34 GMT
Message-ID: <6452536.46081272410733849.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 19:25:33 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6563:
--------------------------------

    Attachment: hadoop-6563-3.patch

Attached patch merges with trunk.

> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6563-1.patch, hadoop-6563-2.patch, hadoop-6563-3.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7738-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 23:25:58 2010
Return-Path: <common-issues-return-7738-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56025 invoked from network); 27 Apr 2010 23:25:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 23:25:57 -0000
Received: (qmail 31041 invoked by uid 500); 27 Apr 2010 23:25:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30981 invoked by uid 500); 27 Apr 2010 23:25:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30970 invoked by uid 99); 27 Apr 2010 23:25:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 23:25:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 23:25:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RNPYff024394
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 23:25:34 GMT
Message-ID: <458587.46101272410734299.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 19:25:34 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6563:
--------------------------------

    Status: Open  (was: Patch Available)

> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6563-1.patch, hadoop-6563-2.patch, hadoop-6563-3.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7739-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 23:25:58 2010
Return-Path: <common-issues-return-7739-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56075 invoked from network); 27 Apr 2010 23:25:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 23:25:58 -0000
Received: (qmail 31153 invoked by uid 500); 27 Apr 2010 23:25:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31116 invoked by uid 500); 27 Apr 2010 23:25:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31108 invoked by uid 99); 27 Apr 2010 23:25:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 23:25:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 23:25:56 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RNPYOs024406
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 23:25:34 GMT
Message-ID: <31685420.46141272410734694.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 19:25:34 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6563:
--------------------------------

    Status: Patch Available  (was: Open)

> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6563-1.patch, hadoop-6563-2.patch, hadoop-6563-3.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7740-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 23:40:00 2010
Return-Path: <common-issues-return-7740-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61442 invoked from network); 27 Apr 2010 23:40:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 23:40:00 -0000
Received: (qmail 35071 invoked by uid 500); 27 Apr 2010 23:39:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34961 invoked by uid 500); 27 Apr 2010 23:39:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34917 invoked by uid 99); 27 Apr 2010 23:39:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 23:39:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 23:39:57 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RNdZap024582
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 23:39:35 GMT
Message-ID: <27804597.46491272411575818.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 19:39:35 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861620#action_12861620 ] 

Hadoop QA commented on HADOOP-6563:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12443020/hadoop-6563-3.patch
  against trunk revision 938590.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The patch appears to cause tar ant target to fail.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    -1 contrib tests.  The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/57/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/57/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/57/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/57/console

This message is automatically generated.

> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6563-1.patch, hadoop-6563-2.patch, hadoop-6563-3.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7741-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 23:43:57 2010
Return-Path: <common-issues-return-7741-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63062 invoked from network); 27 Apr 2010 23:43:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 23:43:56 -0000
Received: (qmail 39216 invoked by uid 500); 27 Apr 2010 23:43:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39191 invoked by uid 500); 27 Apr 2010 23:43:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39183 invoked by uid 99); 27 Apr 2010 23:43:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 23:43:56 +0000
X-ASF-Spam-Status: No, hits=-1357.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 23:43:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RNhZOO024622
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 23:43:35 GMT
Message-ID: <6218727.46591272411815222.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 19:43:35 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6727) Remove UnresolvedLinkException from
 public FileContext APIs
In-Reply-To: <27900682.46541272411814280.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6727:
--------------------------------

    Status: Patch Available  (was: Open)

> Remove UnresolvedLinkException from public FileContext APIs
> -----------------------------------------------------------
>
>                 Key: HADOOP-6727
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6727
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6727-1.patch
>
>
> HADOOP-6537 added UnresolvedLinkException to the throws clause and java docs of FileContext public APIs. FileContext fully resolves symbolic links, UnresolvedLinkException exception is only used internally by FSLinkResolver. I'll attach a patch which updates the APIs and javadoc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7742-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 23:43:59 2010
Return-Path: <common-issues-return-7742-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63171 invoked from network); 27 Apr 2010 23:43:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 23:43:58 -0000
Received: (qmail 39506 invoked by uid 500); 27 Apr 2010 23:43:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39468 invoked by uid 500); 27 Apr 2010 23:43:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39460 invoked by uid 99); 27 Apr 2010 23:43:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 23:43:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 23:43:56 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RNhYFO024607
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 23:43:34 GMT
Message-ID: <27900682.46541272411814280.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 19:43:34 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6727) Remove UnresolvedLinkException from
 public FileContext APIs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Remove UnresolvedLinkException from public FileContext APIs
-----------------------------------------------------------

                 Key: HADOOP-6727
                 URL: https://issues.apache.org/jira/browse/HADOOP-6727
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
    Affects Versions: 0.21.0, 0.22.0
            Reporter: Eli Collins
            Assignee: Eli Collins
            Priority: Minor
             Fix For: 0.21.0, 0.22.0
         Attachments: hadoop-6727-1.patch

HADOOP-6537 added UnresolvedLinkException to the throws clause and java docs of FileContext public APIs. FileContext fully resolves symbolic links, UnresolvedLinkException exception is only used internally by FSLinkResolver. I'll attach a patch which updates the APIs and javadoc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7743-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Apr 27 23:43:59 2010
Return-Path: <common-issues-return-7743-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63208 invoked from network); 27 Apr 2010 23:43:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Apr 2010 23:43:58 -0000
Received: (qmail 39656 invoked by uid 500); 27 Apr 2010 23:43:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39612 invoked by uid 500); 27 Apr 2010 23:43:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39604 invoked by uid 99); 27 Apr 2010 23:43:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 23:43:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Apr 2010 23:43:56 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3RNhZAv024616
	for <common-issues@hadoop.apache.org>; Tue, 27 Apr 2010 23:43:35 GMT
Message-ID: <663125.46571272411815033.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 19:43:35 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6727) Remove UnresolvedLinkException from
 public FileContext APIs
In-Reply-To: <27900682.46541272411814280.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6727:
--------------------------------

    Attachment: hadoop-6727-1.patch

> Remove UnresolvedLinkException from public FileContext APIs
> -----------------------------------------------------------
>
>                 Key: HADOOP-6727
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6727
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6727-1.patch
>
>
> HADOOP-6537 added UnresolvedLinkException to the throws clause and java docs of FileContext public APIs. FileContext fully resolves symbolic links, UnresolvedLinkException exception is only used internally by FSLinkResolver. I'll attach a patch which updates the APIs and javadoc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7744-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 00:01:57 2010
Return-Path: <common-issues-return-7744-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71574 invoked from network); 28 Apr 2010 00:01:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 00:01:56 -0000
Received: (qmail 54208 invoked by uid 500); 28 Apr 2010 00:01:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54148 invoked by uid 500); 28 Apr 2010 00:01:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54098 invoked by uid 99); 28 Apr 2010 00:01:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 00:01:56 +0000
X-ASF-Spam-Status: No, hits=-1357.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 00:01:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3S01YTh024871
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 00:01:34 GMT
Message-ID: <32623682.46961272412894896.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 20:01:34 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6563:
--------------------------------

    Status: Patch Available  (was: Open)

> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6563-1.patch, hadoop-6563-2.patch, hadoop-6563-3.patch, hadoop-6563-4.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7745-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 00:01:57 2010
Return-Path: <common-issues-return-7745-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71612 invoked from network); 28 Apr 2010 00:01:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 00:01:57 -0000
Received: (qmail 54899 invoked by uid 500); 28 Apr 2010 00:01:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54873 invoked by uid 500); 28 Apr 2010 00:01:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54865 invoked by uid 99); 28 Apr 2010 00:01:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 00:01:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 00:01:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3S01Xi4024850
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 00:01:33 GMT
Message-ID: <9277001.46891272412893022.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 20:01:33 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6563:
--------------------------------

    Status: Open  (was: Patch Available)

> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6563-1.patch, hadoop-6563-2.patch, hadoop-6563-3.patch, hadoop-6563-4.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7746-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 00:01:58 2010
Return-Path: <common-issues-return-7746-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71736 invoked from network); 28 Apr 2010 00:01:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 00:01:58 -0000
Received: (qmail 55044 invoked by uid 500); 28 Apr 2010 00:01:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55008 invoked by uid 500); 28 Apr 2010 00:01:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55000 invoked by uid 99); 28 Apr 2010 00:01:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 00:01:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 00:01:56 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3S01Yu6024865
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 00:01:34 GMT
Message-ID: <100100.46941272412894692.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 20:01:34 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6563:
--------------------------------

    Attachment: hadoop-6563-4.patch

Right patch this time - sorry for the noise.

> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6563-1.patch, hadoop-6563-2.patch, hadoop-6563-3.patch, hadoop-6563-4.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7747-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 00:05:53 2010
Return-Path: <common-issues-return-7747-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73630 invoked from network); 28 Apr 2010 00:05:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 00:05:53 -0000
Received: (qmail 57316 invoked by uid 500); 28 Apr 2010 00:05:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57284 invoked by uid 500); 28 Apr 2010 00:05:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57272 invoked by uid 99); 28 Apr 2010 00:05:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 00:05:53 +0000
X-ASF-Spam-Status: No, hits=-1357.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 00:05:52 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3S05VPR024903
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 00:05:32 GMT
Message-ID: <30130416.46981272413131757.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 20:05:31 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6727) Remove UnresolvedLinkException from
 public FileContext APIs
In-Reply-To: <27900682.46541272411814280.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861625#action_12861625 ] 

Hadoop QA commented on HADOOP-6727:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12443023/hadoop-6727-1.patch
  against trunk revision 938590.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/58/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/58/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/58/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/58/console

This message is automatically generated.

> Remove UnresolvedLinkException from public FileContext APIs
> -----------------------------------------------------------
>
>                 Key: HADOOP-6727
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6727
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6727-1.patch
>
>
> HADOOP-6537 added UnresolvedLinkException to the throws clause and java docs of FileContext public APIs. FileContext fully resolves symbolic links, UnresolvedLinkException exception is only used internally by FSLinkResolver. I'll attach a patch which updates the APIs and javadoc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7748-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 00:20:56 2010
Return-Path: <common-issues-return-7748-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80751 invoked from network); 28 Apr 2010 00:20:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 00:20:56 -0000
Received: (qmail 64760 invoked by uid 500); 28 Apr 2010 00:20:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64734 invoked by uid 500); 28 Apr 2010 00:20:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64726 invoked by uid 99); 28 Apr 2010 00:20:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 00:20:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 00:20:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3S0KWb9025011
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 00:20:32 GMT
Message-ID: <1606532.47041272414032076.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 20:20:32 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861627#action_12861627 ] 

Hadoop QA commented on HADOOP-6563:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12443025/hadoop-6563-4.patch
  against trunk revision 938590.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/59/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/59/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/59/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/59/console

This message is automatically generated.

> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6563-1.patch, hadoop-6563-2.patch, hadoop-6563-3.patch, hadoop-6563-4.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7749-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 02:07:56 2010
Return-Path: <common-issues-return-7749-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39220 invoked from network); 28 Apr 2010 02:07:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 02:07:56 -0000
Received: (qmail 42622 invoked by uid 500); 28 Apr 2010 02:07:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42593 invoked by uid 500); 28 Apr 2010 02:07:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42585 invoked by uid 99); 28 Apr 2010 02:07:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 02:07:56 +0000
X-ASF-Spam-Status: No, hits=-1358.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 02:07:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3S27YTn027396
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 02:07:34 GMT
Message-ID: <22280907.48461272420454687.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 22:07:34 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6678) Remove FileContext#isFile,
 isDirectory and exists
In-Reply-To: <360625036.668801270253667713.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861645#action_12861645 ] 

Eli Collins commented on HADOOP-6678:
-------------------------------------

Hey Hairong,  no worries - looking at the implementation of FileContext#exists I think it can be simplified:

{code}
  public boolean exists(final Path f) throws AccessControlException,  IOException {
    try {
      return getFileStatus(f) != null;            
    } catch (FileNotFoundException e) {
      return false;
    } catch (UnsupportedFileSystemException e) {
      return false;
    } catch (UnresolvedLinkException e) {
      return false;
    }
  }
{code}

# AFS#getFileStatus should never return null, eg see Hdfs#getFileStatus. Per the spec if should throw a FileNotFoundException if f does not exist.
# Seems like the client should have to deal with UnsupportedFileSystemException rather than have FC#exists swallow it
# UnresolvedLinkException will never be thrown (see HADOOP-6727)

Therefore I think we can write FC#exists as

{code}
  public boolean exists(final Path f) throws AccessControlException, UnsupportedFileSystemException,  IOException {
    try {
      FileStatus fs =  getFileStatus(f);
      assert fs != null;
      return true;
    } catch (FileNotFoundException e) {
      return false;
    }
  }
{code}

Since this is still not a one-liner putting it in FileContext#util seems reasonable to me.  I'll send out new patches.



> Remove FileContext#isFile, isDirectory and exists
> -------------------------------------------------
>
>                 Key: HADOOP-6678
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6678
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Hairong Kuang
>            Assignee: Eli Collins
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6678-1.patch
>
>
> # Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.
> # Remove methods isFile(Path), isDirectory(Path), and exists.
> All these methods are implemented by calling getFileStatus(Path).But most users are not aware of this. They would write code as below: 
> {code}
>   FileContext fc = ..;
>   if (fc.exists(path)) {
>     if (fc.isFile(path)) {
>      ...
>     } else {
>     ...
>     }
>   }
> {code}
> The above code adds unnecessary getFileInfo RPC to NameNode. In our production clusters, we often see that the number of getFileStatus calls is multiple times of the open calls. If we remove isFile, isDirectory, and exists from FileContext, users have to explicitly call getFileStatus first, it is more likely that they will write more efficient code as follow:
> {code}
>   FileContext fc = ...;
>   FileStatus fstatus = fc.getFileStatus(path);
>   if (fstatus.isFile() {
>     ...
>   } else {
>     ...
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7750-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 02:11:54 2010
Return-Path: <common-issues-return-7750-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40833 invoked from network); 28 Apr 2010 02:11:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 02:11:53 -0000
Received: (qmail 45375 invoked by uid 500); 28 Apr 2010 02:11:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45263 invoked by uid 500); 28 Apr 2010 02:11:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45255 invoked by uid 99); 28 Apr 2010 02:11:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 02:11:53 +0000
X-ASF-Spam-Status: No, hits=-1358.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 02:11:52 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3S2BWUm027433
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 02:11:32 GMT
Message-ID: <17506266.48501272420692262.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 22:11:32 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6728) Overhaul metric framework
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Overhaul metric framework
-------------------------

                 Key: HADOOP-6728
                 URL: https://issues.apache.org/jira/browse/HADOOP-6728
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Luke Lu
             Fix For: 0.20.3


Per discussions with Arun, Chris, Hong and Rajiv, et al, we concluded that the current metrics framework needs an overhaul to:

* Allow multiple plugins for different monitoring systems simultaneously (see also: HADOOP-6508).
* Refresh metrics plugin config without server restart.
** Including filtering of metrics per plugin.
* Support metrics schema for plugins.

The jira will be resolved when core hadoop components (hdfs, mapreduce) are updated to use the new framework . Updates to external components that use the existing metrics framework will be tracked by different issues.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7751-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 02:13:53 2010
Return-Path: <common-issues-return-7751-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41731 invoked from network); 28 Apr 2010 02:13:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 02:13:53 -0000
Received: (qmail 49234 invoked by uid 500); 28 Apr 2010 02:13:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49208 invoked by uid 500); 28 Apr 2010 02:13:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49200 invoked by uid 99); 28 Apr 2010 02:13:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 02:13:53 +0000
X-ASF-Spam-Status: No, hits=-1358.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 02:13:52 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3S2DVFY027445
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 02:13:32 GMT
Message-ID: <7703070.48531272420811844.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 22:13:31 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6727) Remove UnresolvedLinkException from
 public FileContext APIs
In-Reply-To: <27900682.46541272411814280.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861646#action_12861646 ] 

Eli Collins commented on HADOOP-6727:
-------------------------------------

Just updating throws clauses and java docs, hence the lack of tests.

> Remove UnresolvedLinkException from public FileContext APIs
> -----------------------------------------------------------
>
>                 Key: HADOOP-6727
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6727
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6727-1.patch
>
>
> HADOOP-6537 added UnresolvedLinkException to the throws clause and java docs of FileContext public APIs. FileContext fully resolves symbolic links, UnresolvedLinkException exception is only used internally by FSLinkResolver. I'll attach a patch which updates the APIs and javadoc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7752-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 02:40:57 2010
Return-Path: <common-issues-return-7752-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53563 invoked from network); 28 Apr 2010 02:40:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 02:40:57 -0000
Received: (qmail 63340 invoked by uid 500); 28 Apr 2010 02:40:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63313 invoked by uid 500); 28 Apr 2010 02:40:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63305 invoked by uid 99); 28 Apr 2010 02:40:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 02:40:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 02:40:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3S2eW54027706
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 02:40:33 GMT
Message-ID: <18861639.48801272422432910.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 22:40:32 -0400 (EDT)
From: "Ted Yu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6729) serializer.JavaSerialization should
 be added to io.serializations by default
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

serializer.JavaSerialization should be added to io.serializations by default
----------------------------------------------------------------------------

                 Key: HADOOP-6729
                 URL: https://issues.apache.org/jira/browse/HADOOP-6729
             Project: Hadoop Common
          Issue Type: Improvement
          Components: conf
    Affects Versions: 0.20.2
            Reporter: Ted Yu


org.apache.hadoop.io.serializer.JavaSerialization isn't included in io.serializations by default.

When a class which implements the Serializable interface is used, user would see the following without serializer.JavaSerialization:

java.lang.NullPointerException
   at
org.apache.hadoop.io.serializer.SerializationFactory.getSerializer(SerializationFactory.java:73)
   at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:759)
   at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:487)
   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:575)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
   at org.apache.hadoop.mapred.Child.main(Child.java:170)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7753-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 03:17:20 2010
Return-Path: <common-issues-return-7753-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63052 invoked from network); 28 Apr 2010 03:17:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 03:17:20 -0000
Received: (qmail 76571 invoked by uid 500); 28 Apr 2010 03:17:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76448 invoked by uid 500); 28 Apr 2010 03:17:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76439 invoked by uid 99); 28 Apr 2010 03:17:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 03:17:18 +0000
X-ASF-Spam-Status: No, hits=-1358.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 03:17:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3S3GvDd018534
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 03:16:57 GMT
Message-ID: <1341397.49011272424617203.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 23:16:57 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6703) Prevent renaming a file, symlink or
 directory to itself
In-Reply-To: <23186149.74901271194550994.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861656#action_12861656 ] 

Hadoop QA commented on HADOOP-6703:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12443017/hadoop-6703-3.patch
  against trunk revision 938590.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/485/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/485/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/485/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/485/console

This message is automatically generated.

> Prevent renaming a file, symlink or directory to itself
> -------------------------------------------------------
>
>                 Key: HADOOP-6703
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6703
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6703-1.patch, hadoop-6703-2.patch, hadoop-6703-3.patch
>
>
> Per HDFS-1088 let's throw a FileAlreadyExistsException if renaming a file, symlink or directory to itself, or a symlink to the file it links to. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7754-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 03:22:55 2010
Return-Path: <common-issues-return-7754-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63424 invoked from network); 28 Apr 2010 03:22:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 03:22:54 -0000
Received: (qmail 78856 invoked by uid 500); 28 Apr 2010 03:22:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78760 invoked by uid 500); 28 Apr 2010 03:22:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78752 invoked by uid 99); 28 Apr 2010 03:22:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 03:22:53 +0000
X-ASF-Spam-Status: No, hits=-1358.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 03:22:52 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3S3MVd3019644
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 03:22:32 GMT
Message-ID: <28673486.49031272424951920.JavaMail.jira@thor>
Date: Tue, 27 Apr 2010 23:22:31 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6729) serializer.JavaSerialization should
 be added to io.serializations by default
In-Reply-To: <18861639.48801272422432910.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861657#action_12861657 ] 

Tom White commented on HADOOP-6729:
-----------------------------------

The JavaSerialization was written as an experimental serialization, to prove the abstraction. I'm not sure it should be enabled by default since it's not efficient, hasn't been tested at scale (as far as I know), and we should encourage users to use other serializations like Writables or Avro.

In any case, we could improve the error message if the type is Serializable, or print a warning.

> serializer.JavaSerialization should be added to io.serializations by default
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-6729
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6729
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>    Affects Versions: 0.20.2
>            Reporter: Ted Yu
>
> org.apache.hadoop.io.serializer.JavaSerialization isn't included in io.serializations by default.
> When a class which implements the Serializable interface is used, user would see the following without serializer.JavaSerialization:
> java.lang.NullPointerException
>    at
> org.apache.hadoop.io.serializer.SerializationFactory.getSerializer(SerializationFactory.java:73)
>    at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:759)
>    at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:487)
>    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:575)
>    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>    at org.apache.hadoop.mapred.Child.main(Child.java:170)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7755-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 05:34:55 2010
Return-Path: <common-issues-return-7755-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1148 invoked from network); 28 Apr 2010 05:34:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 05:34:55 -0000
Received: (qmail 31632 invoked by uid 500); 28 Apr 2010 05:34:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31510 invoked by uid 500); 28 Apr 2010 05:34:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31497 invoked by uid 99); 28 Apr 2010 05:34:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 05:34:54 +0000
X-ASF-Spam-Status: No, hits=-1358.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 05:34:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3S5YXK1002859
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 05:34:33 GMT
Message-ID: <26804697.50211272432873072.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 01:34:33 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6701)  Incorrect exit codes for "dfs
 -chown", "dfs -chgrp"
In-Reply-To: <29784448.2801271097111992.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6701:
------------------------------------

    Attachment: HADOOP-6701-trunk.patch

Refreshing the patch for hudson to pickup.

>  Incorrect exit codes for "dfs -chown", "dfs -chgrp"
> ----------------------------------------------------
>
>                 Key: HADOOP-6701
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6701
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.19.1, 0.20.0, 0.20.1, 0.20.2
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>            Priority: Minor
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6701-trunk.patch, HADOOP-6701-trunk.patch, HADOOP-6701-v20.patch
>
>
> ravi@localhost:~$ hadoop dfs -chgrp abcd /; echo $?
> chgrp: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chown  abcd /; echo $?
> chown: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chmod 755 /DOESNTEXIST; echo $?
> chmod: could not get status for '/DOESNTEXIST': File does not exist: /DOESNTEXIST
> 0
> -
> Exit codes for both of the above invocations should be non-zero to indicate that the command failed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7756-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 05:34:57 2010
Return-Path: <common-issues-return-7756-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1202 invoked from network); 28 Apr 2010 05:34:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 05:34:57 -0000
Received: (qmail 31685 invoked by uid 500); 28 Apr 2010 05:34:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31657 invoked by uid 500); 28 Apr 2010 05:34:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31649 invoked by uid 99); 28 Apr 2010 05:34:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 05:34:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 05:34:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3S5YXQw002868
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 05:34:33 GMT
Message-ID: <5954593.50241272432873757.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 01:34:33 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6701)  Incorrect exit codes for "dfs
 -chown", "dfs -chgrp"
In-Reply-To: <29784448.2801271097111992.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6701:
------------------------------------

    Status: Open  (was: Patch Available)

>  Incorrect exit codes for "dfs -chown", "dfs -chgrp"
> ----------------------------------------------------
>
>                 Key: HADOOP-6701
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6701
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.2, 0.20.1, 0.20.0, 0.19.1
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>            Priority: Minor
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6701-trunk.patch, HADOOP-6701-trunk.patch, HADOOP-6701-v20.patch
>
>
> ravi@localhost:~$ hadoop dfs -chgrp abcd /; echo $?
> chgrp: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chown  abcd /; echo $?
> chown: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chmod 755 /DOESNTEXIST; echo $?
> chmod: could not get status for '/DOESNTEXIST': File does not exist: /DOESNTEXIST
> 0
> -
> Exit codes for both of the above invocations should be non-zero to indicate that the command failed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7757-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 05:36:55 2010
Return-Path: <common-issues-return-7757-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2273 invoked from network); 28 Apr 2010 05:36:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 05:36:55 -0000
Received: (qmail 33019 invoked by uid 500); 28 Apr 2010 05:36:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32907 invoked by uid 500); 28 Apr 2010 05:36:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32899 invoked by uid 99); 28 Apr 2010 05:36:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 05:36:54 +0000
X-ASF-Spam-Status: No, hits=-1358.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 05:36:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3S5aWLF002890
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 05:36:33 GMT
Message-ID: <9064030.50271272432992699.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 01:36:32 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6701)  Incorrect exit codes for "dfs
 -chown", "dfs -chgrp"
In-Reply-To: <29784448.2801271097111992.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6701:
------------------------------------

    Status: Patch Available  (was: Open)

>  Incorrect exit codes for "dfs -chown", "dfs -chgrp"
> ----------------------------------------------------
>
>                 Key: HADOOP-6701
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6701
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.2, 0.20.1, 0.20.0, 0.19.1
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>            Priority: Minor
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6701-trunk.patch, HADOOP-6701-trunk.patch, HADOOP-6701-v20.patch
>
>
> ravi@localhost:~$ hadoop dfs -chgrp abcd /; echo $?
> chgrp: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chown  abcd /; echo $?
> chown: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chmod 755 /DOESNTEXIST; echo $?
> chmod: could not get status for '/DOESNTEXIST': File does not exist: /DOESNTEXIST
> 0
> -
> Exit codes for both of the above invocations should be non-zero to indicate that the command failed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7758-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 05:43:01 2010
Return-Path: <common-issues-return-7758-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4795 invoked from network); 28 Apr 2010 05:43:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 05:43:01 -0000
Received: (qmail 38561 invoked by uid 500); 28 Apr 2010 05:43:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38467 invoked by uid 500); 28 Apr 2010 05:43:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38459 invoked by uid 99); 28 Apr 2010 05:43:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 05:43:00 +0000
X-ASF-Spam-Status: No, hits=-1358.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 05:42:59 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3S5gcKD002973
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 05:42:39 GMT
Message-ID: <18056048.50391272433358979.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 01:42:38 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6703) Prevent renaming a file, symlink or
 directory to itself
In-Reply-To: <23186149.74901271194550994.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861679#action_12861679 ] 

Suresh Srinivas commented on HADOOP-6703:
-----------------------------------------

+1 for the patch.

> Prevent renaming a file, symlink or directory to itself
> -------------------------------------------------------
>
>                 Key: HADOOP-6703
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6703
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6703-1.patch, hadoop-6703-2.patch, hadoop-6703-3.patch
>
>
> Per HDFS-1088 let's throw a FileAlreadyExistsException if renaming a file, symlink or directory to itself, or a symlink to the file it links to. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7759-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 05:48:59 2010
Return-Path: <common-issues-return-7759-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7445 invoked from network); 28 Apr 2010 05:48:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 05:48:57 -0000
Received: (qmail 43567 invoked by uid 500); 28 Apr 2010 05:48:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43474 invoked by uid 500); 28 Apr 2010 05:48:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43466 invoked by uid 99); 28 Apr 2010 05:48:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 05:48:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 05:48:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3S5mWcZ003052
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 05:48:32 GMT
Message-ID: <9926802.50441272433712344.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 01:48:32 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6703) Prevent renaming a file, symlink or
 directory to itself
In-Reply-To: <23186149.74901271194550994.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6703:
------------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

I committed the patch. Thank you Eli.

> Prevent renaming a file, symlink or directory to itself
> -------------------------------------------------------
>
>                 Key: HADOOP-6703
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6703
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6703-1.patch, hadoop-6703-2.patch, hadoop-6703-3.patch
>
>
> Per HDFS-1088 let's throw a FileAlreadyExistsException if renaming a file, symlink or directory to itself, or a symlink to the file it links to. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7760-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 06:00:59 2010
Return-Path: <common-issues-return-7760-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12042 invoked from network); 28 Apr 2010 06:00:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 06:00:59 -0000
Received: (qmail 51918 invoked by uid 500); 28 Apr 2010 06:00:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51816 invoked by uid 500); 28 Apr 2010 06:00:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51807 invoked by uid 99); 28 Apr 2010 06:00:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 06:00:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 06:00:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3S60WKN003230
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 06:00:32 GMT
Message-ID: <27687348.50491272434432767.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 02:00:32 -0400 (EDT)
From: "Ravi Gummadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6631) FileUtil.fullyDelete() should
 continue to delete other files despite failure at any level.
In-Reply-To: <615835698.224041268388087233.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated HADOOP-6631:
---------------------------------

    Attachment: HADOOP-6631.patch

Attaching patch for trunk.

> FileUtil.fullyDelete() should continue to delete other files despite failure at any level.
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6631
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6631
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, util
>            Reporter: Vinod K V
>            Assignee: Ravi Gummadi
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6631.patch
>
>
> Ravi commented about this on HADOOP-6536. Paraphrasing...
> Currently FileUtil.fullyDelete(myDir) comes out stopping deletion of other files/directories if it is unable to delete a file/dir(say because of not having permissions to delete that file/dir) anywhere under myDir. This is because we return from method if the recursive call "if(!fullyDelete()) {return false;}" fails at any level of recursion.
> Shouldn't it continue with deletion of other files/dirs continuing in the for loop instead of returning false here ?
> I guess fullyDelete() should delete as many files as possible(similar to 'rm -rf').

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7761-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 06:10:55 2010
Return-Path: <common-issues-return-7761-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14608 invoked from network); 28 Apr 2010 06:10:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 06:10:55 -0000
Received: (qmail 58848 invoked by uid 500); 28 Apr 2010 06:10:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58759 invoked by uid 500); 28 Apr 2010 06:10:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58748 invoked by uid 99); 28 Apr 2010 06:10:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 06:10:54 +0000
X-ASF-Spam-Status: No, hits=-1358.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 06:10:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3S6AWKE003414
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 06:10:32 GMT
Message-ID: <1639458.50661272435032580.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 02:10:32 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861685#action_12861685 ] 

Suresh Srinivas commented on HADOOP-6563:
-----------------------------------------

Modified the table you have a little bit. Hopefully it has more clarity:
||Source||Dest||Result||
|fileA|fileB|fileB is removed. fileA is renamed fileB|
| |dirB|Fails: both need to be file or directory|
| |linkB|linkB is removed (regardles of what it points to). fileA is renamed linkB|
|dirA|fileB|Fails: both need to be file or directory|
| |dirB|dirB is deleted(if empty). dirA is renamed to dirB|
| |linkB|Fails: if Source is a directory, then Dest must be a directory|
|linkA|fileB|fileB is deleted. linkA renamed to fileB|
| |dirB|if Dest is a directory then Source must be a directory|
| |linkB|linkB is removed (regardless of what it points to. linkA is renamed to linkB|

Comments:
# typo danling
# testRenameSymlinkToFileItLinksTo - when rename fails, should the test check for exception? Is this part of another jira?
# Why is creating dangling links allowed?
# Tests are hard to follow without additional descprition in the test methods what is being tested and what is expected. It took me a while to follow the tests.



> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6563-1.patch, hadoop-6563-2.patch, hadoop-6563-3.patch, hadoop-6563-4.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7762-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 06:29:01 2010
Return-Path: <common-issues-return-7762-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28218 invoked from network); 28 Apr 2010 06:29:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 06:29:01 -0000
Received: (qmail 71667 invoked by uid 500); 28 Apr 2010 06:29:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71542 invoked by uid 500); 28 Apr 2010 06:28:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71532 invoked by uid 99); 28 Apr 2010 06:28:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 06:28:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 06:28:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3S6SXI0003634
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 06:28:34 GMT
Message-ID: <2368186.50821272436113332.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 02:28:33 -0400 (EDT)
From: "Ravi Gummadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6536) FileUtil.fullyDelete(dir) behavior is
 not defined when we pass a symlink as the argument
In-Reply-To: <508958946.48471265176519280.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated HADOOP-6536:
---------------------------------

    Attachment: HADOOP-6536.patch

Attaching patch for trunk. This patch is on top of HADOOP-6631.

> FileUtil.fullyDelete(dir) behavior is not defined when we pass a symlink as the argument
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6536
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6536
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Ravi Gummadi
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6536.patch
>
>
> FileUtil.fullyDelete(dir) deletes contents of sym-linked directory when we pass a symlink. If this is the behavior, it should be documented as so. 
> Or it should be changed not to delete the contents of the sym-linked directory.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7763-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 08:02:01 2010
Return-Path: <common-issues-return-7763-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71515 invoked from network); 28 Apr 2010 08:02:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 08:02:01 -0000
Received: (qmail 59222 invoked by uid 500); 28 Apr 2010 08:02:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58979 invoked by uid 500); 28 Apr 2010 08:01:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58963 invoked by uid 99); 28 Apr 2010 08:01:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 08:01:57 +0000
X-ASF-Spam-Status: No, hits=-1359.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 08:01:56 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3S81alO005191
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 08:01:36 GMT
Message-ID: <11761180.52441272441696331.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 04:01:36 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6634) AccessControlList uses full-principal
 names to verify acls causing queue-acls to fail
In-Reply-To: <136802909.310821268816187241.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V updated HADOOP-6634:
------------------------------

    Status: Patch Available  (was: Open)

Running it through Hudson.

> AccessControlList uses full-principal names to verify acls causing queue-acls to fail
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6634
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6634
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Vinod K V
>            Assignee: Vinod K V
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6634-20100317-ydist.1.txt, HADOOP-6634-20100428.txt
>
>
> ACL configuration so far was using short user-names. With the changed {{UserGroupInformation}}, short names are different from the long fully-qualified names. {{AccessControlList}} continues to use {{UserGroupInformation.getUserName()}} for verifying access control. Because of this, queue acls fail for a user "user@domain.org" even though the short name "user" is part of the acl configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7764-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 08:02:02 2010
Return-Path: <common-issues-return-7764-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71548 invoked from network); 28 Apr 2010 08:02:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 08:02:01 -0000
Received: (qmail 59348 invoked by uid 500); 28 Apr 2010 08:02:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59305 invoked by uid 500); 28 Apr 2010 08:02:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58983 invoked by uid 99); 28 Apr 2010 08:01:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 08:01:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 08:01:57 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3S81ZWe005167
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 08:01:35 GMT
Message-ID: <13000090.52361272441695151.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 04:01:35 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6634) AccessControlList uses full-principal
 names to verify acls causing queue-acls to fail
In-Reply-To: <136802909.310821268816187241.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V updated HADOOP-6634:
------------------------------

    Attachment: HADOOP-6634-20100428.txt

Patch for trunk. The testcase change should serve as both verifying the patch as well as a regression test.

> AccessControlList uses full-principal names to verify acls causing queue-acls to fail
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6634
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6634
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Vinod K V
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6634-20100317-ydist.1.txt, HADOOP-6634-20100428.txt
>
>
> ACL configuration so far was using short user-names. With the changed {{UserGroupInformation}}, short names are different from the long fully-qualified names. {{AccessControlList}} continues to use {{UserGroupInformation.getUserName()}} for verifying access control. Because of this, queue acls fail for a user "user@domain.org" even though the short name "user" is part of the acl configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7765-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 08:02:02 2010
Return-Path: <common-issues-return-7765-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71572 invoked from network); 28 Apr 2010 08:02:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 08:02:01 -0000
Received: (qmail 59442 invoked by uid 500); 28 Apr 2010 08:02:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59318 invoked by uid 500); 28 Apr 2010 08:02:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58982 invoked by uid 99); 28 Apr 2010 08:01:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 08:01:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 08:01:57 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3S81ZXc005179
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 08:01:35 GMT
Message-ID: <11368997.52401272441695713.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 04:01:35 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6634) AccessControlList uses
 full-principal names to verify acls causing queue-acls to fail
In-Reply-To: <136802909.310821268816187241.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V reassigned HADOOP-6634:
---------------------------------

    Assignee: Vinod K V

> AccessControlList uses full-principal names to verify acls causing queue-acls to fail
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6634
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6634
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Vinod K V
>            Assignee: Vinod K V
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6634-20100317-ydist.1.txt, HADOOP-6634-20100428.txt
>
>
> ACL configuration so far was using short user-names. With the changed {{UserGroupInformation}}, short names are different from the long fully-qualified names. {{AccessControlList}} continues to use {{UserGroupInformation.getUserName()}} for verifying access control. Because of this, queue acls fail for a user "user@domain.org" even though the short name "user" is part of the acl configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7766-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 08:25:56 2010
Return-Path: <common-issues-return-7766-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83598 invoked from network); 28 Apr 2010 08:25:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 08:25:55 -0000
Received: (qmail 96710 invoked by uid 500); 28 Apr 2010 08:25:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96579 invoked by uid 500); 28 Apr 2010 08:25:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96473 invoked by uid 99); 28 Apr 2010 08:25:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 08:25:55 +0000
X-ASF-Spam-Status: No, hits=-1359.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 08:25:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3S8PW9A005549
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 08:25:33 GMT
Message-ID: <31291762.52831272443132973.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 04:25:32 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6634) AccessControlList uses
 full-principal names to verify acls causing queue-acls to fail
In-Reply-To: <136802909.310821268816187241.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861721#action_12861721 ] 

Hadoop QA commented on HADOOP-6634:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12443050/HADOOP-6634-20100428.txt
  against trunk revision 938788.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/60/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/60/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/60/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/60/console

This message is automatically generated.

> AccessControlList uses full-principal names to verify acls causing queue-acls to fail
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6634
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6634
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Vinod K V
>            Assignee: Vinod K V
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6634-20100317-ydist.1.txt, HADOOP-6634-20100428.txt
>
>
> ACL configuration so far was using short user-names. With the changed {{UserGroupInformation}}, short names are different from the long fully-qualified names. {{AccessControlList}} continues to use {{UserGroupInformation.getUserName()}} for verifying access control. Because of this, queue acls fail for a user "user@domain.org" even though the short name "user" is part of the acl configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7767-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 08:50:00 2010
Return-Path: <common-issues-return-7767-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95992 invoked from network); 28 Apr 2010 08:50:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 08:50:00 -0000
Received: (qmail 12643 invoked by uid 500); 28 Apr 2010 08:50:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12593 invoked by uid 500); 28 Apr 2010 08:49:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12585 invoked by uid 99); 28 Apr 2010 08:49:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 08:49:59 +0000
X-ASF-Spam-Status: No, hits=-1359.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 08:49:58 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3S8ncfb005972
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 08:49:38 GMT
Message-ID: <15341575.53281272444578393.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 04:49:38 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6701)  Incorrect exit codes for "dfs
 -chown", "dfs -chgrp"
In-Reply-To: <29784448.2801271097111992.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861730#action_12861730 ] 

Hadoop QA commented on HADOOP-6701:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12443038/HADOOP-6701-trunk.patch
  against trunk revision 938788.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/486/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/486/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/486/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/486/console

This message is automatically generated.

>  Incorrect exit codes for "dfs -chown", "dfs -chgrp"
> ----------------------------------------------------
>
>                 Key: HADOOP-6701
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6701
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.19.1, 0.20.0, 0.20.1, 0.20.2
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>            Priority: Minor
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6701-trunk.patch, HADOOP-6701-trunk.patch, HADOOP-6701-v20.patch
>
>
> ravi@localhost:~$ hadoop dfs -chgrp abcd /; echo $?
> chgrp: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chown  abcd /; echo $?
> chown: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chmod 755 /DOESNTEXIST; echo $?
> chmod: could not get status for '/DOESNTEXIST': File does not exist: /DOESNTEXIST
> 0
> -
> Exit codes for both of the above invocations should be non-zero to indicate that the command failed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7768-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 08:53:59 2010
Return-Path: <common-issues-return-7768-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97922 invoked from network); 28 Apr 2010 08:53:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 08:53:59 -0000
Received: (qmail 19410 invoked by uid 500); 28 Apr 2010 08:53:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19369 invoked by uid 500); 28 Apr 2010 08:53:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19361 invoked by uid 99); 28 Apr 2010 08:53:59 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 08:53:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 08:53:56 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3S8rYIH006054
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 08:53:34 GMT
Message-ID: <13676591.53341272444814663.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 04:53:34 -0400 (EDT)
From: "Ravi Gummadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6634) AccessControlList uses
 full-principal names to verify acls causing queue-acls to fail
In-Reply-To: <136802909.310821268816187241.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861731#action_12861731 ] 

Ravi Gummadi commented on HADOOP-6634:
--------------------------------------

Patch for trunk looks good.
+1

> AccessControlList uses full-principal names to verify acls causing queue-acls to fail
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6634
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6634
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Vinod K V
>            Assignee: Vinod K V
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6634-20100317-ydist.1.txt, HADOOP-6634-20100428.txt
>
>
> ACL configuration so far was using short user-names. With the changed {{UserGroupInformation}}, short names are different from the long fully-qualified names. {{AccessControlList}} continues to use {{UserGroupInformation.getUserName()}} for verifying access control. Because of this, queue acls fail for a user "user@domain.org" even though the short name "user" is part of the acl configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7769-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 10:23:00 2010
Return-Path: <common-issues-return-7769-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45196 invoked from network); 28 Apr 2010 10:23:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 10:23:00 -0000
Received: (qmail 19864 invoked by uid 500); 28 Apr 2010 10:22:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19738 invoked by uid 500); 28 Apr 2010 10:22:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19577 invoked by uid 99); 28 Apr 2010 10:22:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 10:22:56 +0000
X-ASF-Spam-Status: No, hits=-1359.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 10:22:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SAMZXP007829
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 10:22:35 GMT
Message-ID: <25848725.54941272450155089.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 06:22:35 -0400 (EDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6725) Evaluate HtmlUnit for unit and
 regression testing webpages
In-Reply-To: <9084581.651272255992465.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861754#action_12861754 ] 

Steve Loughran commented on HADOOP-6725:
----------------------------------------

I am +1 for this and anything else which tests the webapps. HtmlUnit is the best in-java web app test framework I know of; I use it on my code. Right now we have some tests for a servlet or two, but nothing looks at the JSP/webapp content, so things like HADOOP-6461 aren't being picked up

# HtmlUnit is looking at moving to the ASF; support from the Hadoop team would be welcome
# JSF unit is separate, it's testing Java Server Faces. HtmlUnit will happily test JSP and pages with javascript in them
# For ease of testing, HTML elements need IDs. HADOOP-5388 covers this. The tags and the tests can go hand in hand. 
# I can help with setting up the test framework, try and get Marc Guillemot to review the design too.

I would really benefit from some HtmlUnit tests that could be run against a live cluster rather than just a miniDFS and miniMR cluster; the tests could be something standalone/redistributable that could be given the URL of a NN, DN, JT or TT and scan through all the expected pages. Having sequences to submit jobs and other actions then becomes further work.


> Evaluate HtmlUnit for unit and regression testing webpages
> ----------------------------------------------------------
>
>                 Key: HADOOP-6725
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6725
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>            Reporter: Jakob Homan
>            Priority: Minor
>
> HtmlUnit (http://htmlunit.sourceforge.net/) looks like it may be a good tool to help unit testing and evaluating our various webpages throughout the project.  Currently this is done only occasionally in the code (usually falls to being a manual test during release cycles), and when it is done, usually the code to parse the webpage, etc. is re-written each time.  The framework is Apache licensed, so including it won't be an issue.  If it's found to be useful, new JIRAs for HDFS and MR should be opened.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7770-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 15:50:04 2010
Return-Path: <common-issues-return-7770-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68707 invoked from network); 28 Apr 2010 15:50:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 15:50:04 -0000
Received: (qmail 45045 invoked by uid 500); 28 Apr 2010 15:50:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45023 invoked by uid 500); 28 Apr 2010 15:50:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45015 invoked by uid 99); 28 Apr 2010 15:50:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 15:50:03 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 15:50:00 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SFnc2d013735
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 15:49:39 GMT
Message-ID: <15604563.59261272469778644.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 11:49:38 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6721) Create a new logo for Hadoop
 security related uses
In-Reply-To: <13254833.145371271962071069.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley resolved HADOOP-6721.
-----------------------------------

    Fix Version/s: site
       Resolution: Fixed

I just committed this to hadoop/logos.

> Create a new logo for Hadoop security related uses
> --------------------------------------------------
>
>                 Key: HADOOP-6721
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6721
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: site
>
>         Attachments: hadoop-helmet.jpg, hadoop-security-logo.jpg
>
>
> We've created a new logo for Hadoop security topics that we wanted to share with the community.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7771-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 17:22:13 2010
Return-Path: <common-issues-return-7771-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95897 invoked from network); 28 Apr 2010 17:22:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 17:22:12 -0000
Received: (qmail 70583 invoked by uid 500); 28 Apr 2010 17:22:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70553 invoked by uid 500); 28 Apr 2010 17:22:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70545 invoked by uid 99); 28 Apr 2010 17:22:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 17:22:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 17:22:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SHLmrs015332
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 17:21:48 GMT
Message-ID: <21490757.631272475308647.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 13:21:48 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6729) serializer.JavaSerialization should
 be added to io.serializations by default
In-Reply-To: <18861639.48801272422432910.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861847#action_12861847 ] 

Tom White commented on HADOOP-6729:
-----------------------------------

One inefficiency of JavaSerialization is the fact that it stores the classname with every record. This is actually worse than normal Java serialization, which uses backreferences to classnames to make the resulting stream more compact. This optimization is disabled in Hadoop (see JavaSerializationSerializer#serialize()) because records are reordered in the shuffle, which would break back references.

Another inefficiency is that JavaSerialization creates a new object every time the deserialize() is called. In the context of large scale data processing, where there may be billions of records, this is very expensive, which is why Writables and Avro reuse instances.

> serializer.JavaSerialization should be added to io.serializations by default
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-6729
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6729
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>    Affects Versions: 0.20.2
>            Reporter: Ted Yu
>
> org.apache.hadoop.io.serializer.JavaSerialization isn't included in io.serializations by default.
> When a class which implements the Serializable interface is used, user would see the following without serializer.JavaSerialization:
> java.lang.NullPointerException
>    at
> org.apache.hadoop.io.serializer.SerializationFactory.getSerializer(SerializationFactory.java:73)
>    at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:759)
>    at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:487)
>    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:575)
>    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>    at org.apache.hadoop.mapred.Child.main(Child.java:170)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7772-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 17:32:14 2010
Return-Path: <common-issues-return-7772-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98755 invoked from network); 28 Apr 2010 17:32:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 17:32:14 -0000
Received: (qmail 85932 invoked by uid 500); 28 Apr 2010 17:32:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85821 invoked by uid 500); 28 Apr 2010 17:32:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85813 invoked by uid 99); 28 Apr 2010 17:32:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 17:32:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 17:32:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SHVoAA015570
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 17:31:50 GMT
Message-ID: <29196702.1081272475910560.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 13:31:50 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6725) Evaluate HtmlUnit for unit and
 regression testing webpages
In-Reply-To: <9084581.651272255992465.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861856#action_12861856 ] 

Konstantin Boudnik commented on HADOOP-6725:
--------------------------------------------

As soon as HADOOP-6332 is in the trunk (I really hope to complete the forward port work before the end of May) we can start using Herriot framework to perform such tests.

> Evaluate HtmlUnit for unit and regression testing webpages
> ----------------------------------------------------------
>
>                 Key: HADOOP-6725
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6725
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>            Reporter: Jakob Homan
>            Priority: Minor
>
> HtmlUnit (http://htmlunit.sourceforge.net/) looks like it may be a good tool to help unit testing and evaluating our various webpages throughout the project.  Currently this is done only occasionally in the code (usually falls to being a manual test during release cycles), and when it is done, usually the code to parse the webpage, etc. is re-written each time.  The framework is Apache licensed, so including it won't be an issue.  If it's found to be useful, new JIRAs for HDFS and MR should be opened.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7773-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 17:32:16 2010
Return-Path: <common-issues-return-7773-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98808 invoked from network); 28 Apr 2010 17:32:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 17:32:15 -0000
Received: (qmail 86075 invoked by uid 500); 28 Apr 2010 17:32:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85961 invoked by uid 500); 28 Apr 2010 17:32:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85953 invoked by uid 99); 28 Apr 2010 17:32:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 17:32:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 17:32:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SHVpq0015594
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 17:31:51 GMT
Message-ID: <22717423.1161272475911658.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 13:31:51 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6725) Evaluate HtmlUnit for unit and
 regression testing webpages
In-Reply-To: <9084581.651272255992465.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861857#action_12861857 ] 

Konstantin Boudnik commented on HADOOP-6725:
--------------------------------------------

I meant 'using HtmlUnit' framework.

> Evaluate HtmlUnit for unit and regression testing webpages
> ----------------------------------------------------------
>
>                 Key: HADOOP-6725
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6725
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>            Reporter: Jakob Homan
>            Priority: Minor
>
> HtmlUnit (http://htmlunit.sourceforge.net/) looks like it may be a good tool to help unit testing and evaluating our various webpages throughout the project.  Currently this is done only occasionally in the code (usually falls to being a manual test during release cycles), and when it is done, usually the code to parse the webpage, etc. is re-written each time.  The framework is Apache licensed, so including it won't be an issue.  If it's found to be useful, new JIRAs for HDFS and MR should be opened.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7774-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 17:34:18 2010
Return-Path: <common-issues-return-7774-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99824 invoked from network); 28 Apr 2010 17:34:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 17:34:15 -0000
Received: (qmail 93206 invoked by uid 500); 28 Apr 2010 17:34:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93175 invoked by uid 500); 28 Apr 2010 17:34:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93167 invoked by uid 99); 28 Apr 2010 17:34:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 17:34:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 17:34:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SHXoMt015625
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 17:33:50 GMT
Message-ID: <21118885.1231272476030556.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 13:33:50 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6701)  Incorrect exit codes for "dfs
 -chown", "dfs -chgrp"
In-Reply-To: <29784448.2801271097111992.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6701:
------------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Incompatible change, Reviewed]
    Release Note: Commands chmod, chown and chgrp now returns non zero exit code and an error message on failure instead of returning zero.
      Resolution: Fixed

I committed the patch. Thank you Ravi.

>  Incorrect exit codes for "dfs -chown", "dfs -chgrp"
> ----------------------------------------------------
>
>                 Key: HADOOP-6701
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6701
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.19.1, 0.20.0, 0.20.1, 0.20.2
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>            Priority: Minor
>             Fix For: 0.20.3, 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6701-trunk.patch, HADOOP-6701-trunk.patch, HADOOP-6701-v20.patch
>
>
> ravi@localhost:~$ hadoop dfs -chgrp abcd /; echo $?
> chgrp: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chown  abcd /; echo $?
> chown: changing ownership of
> 'hdfs://localhost/':org.apache.hadoop.security.AccessControlException: Permission denied
> 0
> ravi@localhost:~$ hadoop dfs -chmod 755 /DOESNTEXIST; echo $?
> chmod: could not get status for '/DOESNTEXIST': File does not exist: /DOESNTEXIST
> 0
> -
> Exit codes for both of the above invocations should be non-zero to indicate that the command failed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7775-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 17:38:13 2010
Return-Path: <common-issues-return-7775-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 747 invoked from network); 28 Apr 2010 17:38:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 17:38:12 -0000
Received: (qmail 99216 invoked by uid 500); 28 Apr 2010 17:38:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99184 invoked by uid 500); 28 Apr 2010 17:38:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99176 invoked by uid 99); 28 Apr 2010 17:38:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 17:38:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 17:38:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SHbm8W015668
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 17:37:48 GMT
Message-ID: <23276844.1271272476268763.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 13:37:48 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6724) IPC doesn't properly handle IOEs
 thrown by socket factory
In-Reply-To: <4486259.186941272248270321.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6724:
------------------------------

           Status: Resolved  (was: Patch Available)
     Hadoop Flags: [Reviewed]
    Fix Version/s: 0.22.0
       Resolution: Fixed

I've just committed this. Thanks Todd!

> IPC doesn't properly handle IOEs thrown by socket factory
> ---------------------------------------------------------
>
>                 Key: HADOOP-6724
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6724
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.20.3, 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6724.txt
>
>
> If the socket factory throws an IOE inside setupIOStreams, then handleConnectionFailure will be called with socket still null, and thus generate an NPE on socket.close(). This ends up orphaning clients, etc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7776-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 17:40:15 2010
Return-Path: <common-issues-return-7776-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2406 invoked from network); 28 Apr 2010 17:40:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 17:40:15 -0000
Received: (qmail 3955 invoked by uid 500); 28 Apr 2010 17:40:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3919 invoked by uid 500); 28 Apr 2010 17:40:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3911 invoked by uid 99); 28 Apr 2010 17:40:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 17:40:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 17:40:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SHdoTp015707
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 17:39:51 GMT
Message-ID: <15310065.1331272476390931.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 13:39:50 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6678) Remove FileContext#isFile,
 isDirectory and exists
In-Reply-To: <360625036.668801270253667713.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861863#action_12861863 ] 

Hairong Kuang commented on HADOOP-6678:
---------------------------------------

> I think we can write FC#exists as..
+1. Thank you, Eli!

> Remove FileContext#isFile, isDirectory and exists
> -------------------------------------------------
>
>                 Key: HADOOP-6678
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6678
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Hairong Kuang
>            Assignee: Eli Collins
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6678-1.patch
>
>
> # Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.
> # Remove methods isFile(Path), isDirectory(Path), and exists.
> All these methods are implemented by calling getFileStatus(Path).But most users are not aware of this. They would write code as below: 
> {code}
>   FileContext fc = ..;
>   if (fc.exists(path)) {
>     if (fc.isFile(path)) {
>      ...
>     } else {
>     ...
>     }
>   }
> {code}
> The above code adds unnecessary getFileInfo RPC to NameNode. In our production clusters, we often see that the number of getFileStatus calls is multiple times of the open calls. If we remove isFile, isDirectory, and exists from FileContext, users have to explicitly call getFileStatus first, it is more likely that they will write more efficient code as follow:
> {code}
>   FileContext fc = ...;
>   FileStatus fstatus = fc.getFileStatus(path);
>   if (fstatus.isFile() {
>     ...
>   } else {
>     ...
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7777-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 17:48:15 2010
Return-Path: <common-issues-return-7777-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6987 invoked from network); 28 Apr 2010 17:48:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 17:48:15 -0000
Received: (qmail 15297 invoked by uid 500); 28 Apr 2010 17:48:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15265 invoked by uid 500); 28 Apr 2010 17:48:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15257 invoked by uid 99); 28 Apr 2010 17:48:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 17:48:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 17:48:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SHlpN9015846
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 17:47:51 GMT
Message-ID: <2551419.1541272476871253.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 13:47:51 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6722) NetUtils.connect should check that it
 hasn't connected a socket to itself
In-Reply-To: <10234488.178701272152510902.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6722:
------------------------------

           Status: Resolved  (was: Patch Available)
     Hadoop Flags: [Reviewed]
    Fix Version/s: 0.22.0
       Resolution: Fixed

I've just committed this. Thanks Todd!

> NetUtils.connect should check that it hasn't connected a socket to itself
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6722
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6722
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6722.txt
>
>
> I had no idea this was possible, but it turns out that a TCP connection will be established in the rare case that the local side of the socket binds to the ephemeral port that you later try to connect to. This can present itself in very very rare occasion when an RPC client is trying to connect to a daemon running on the same node, but that daemon is down. To see what I'm talking about, run "while true ; do telnet localhost 60020 ; done" on a multicore box and wait several minutes.
> This can be easily detected in NetUtils.connect by making sure the local address/port is not equal to the remote address/port.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7778-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 17:52:13 2010
Return-Path: <common-issues-return-7778-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10398 invoked from network); 28 Apr 2010 17:52:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 17:52:13 -0000
Received: (qmail 27911 invoked by uid 500); 28 Apr 2010 17:52:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27885 invoked by uid 500); 28 Apr 2010 17:52:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27877 invoked by uid 99); 28 Apr 2010 17:52:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 17:52:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 17:52:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SHpnva015923
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 17:51:49 GMT
Message-ID: <5690651.1611272477108997.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 13:51:48 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6723) unchecked exceptions thrown in IPC
 Connection orphan clients
In-Reply-To: <33033218.184741272223549956.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861868#action_12861868 ] 

Tom White commented on HADOOP-6723:
-----------------------------------

+1

> unchecked exceptions thrown in IPC Connection orphan clients
> ------------------------------------------------------------
>
>                 Key: HADOOP-6723
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6723
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.20.2
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6723.txt
>
>
> If the server sends back some malformed data, for example,  receiveResponse() can end up with an incorrect call ID. Then, when it tries to find it in the calls map, it will end up with null and throw NPE in receiveResponse. This isn't caught anywhere, so the original IPC client ends up hanging forever instead of catching an exception. Another example is if the writable implementation itself throws an unchecked exception or OOME.
> We should catch Throwable in Connection.run() and shut down the connection if we catch one.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7779-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 18:21:16 2010
Return-Path: <common-issues-return-7779-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27095 invoked from network); 28 Apr 2010 18:21:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 18:21:16 -0000
Received: (qmail 78258 invoked by uid 500); 28 Apr 2010 18:21:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78153 invoked by uid 500); 28 Apr 2010 18:21:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78144 invoked by uid 99); 28 Apr 2010 18:21:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 18:21:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 18:21:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SIKq9v016610
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 18:20:52 GMT
Message-ID: <12184905.2671272478852187.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 14:20:52 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6728) Overhaul metrics framework
In-Reply-To: <17506266.48501272420692262.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-6728:
----------------------------

              Summary: Overhaul metrics framework  (was: Overhaul metric framework)
             Assignee: Luke Lu
        Fix Version/s: 0.22.0
                           (was: 0.20.3)
    Affects Version/s: 0.20.2

> Overhaul metrics framework
> --------------------------
>
>                 Key: HADOOP-6728
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6728
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.2
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>
> Per discussions with Arun, Chris, Hong and Rajiv, et al, we concluded that the current metrics framework needs an overhaul to:
> * Allow multiple plugins for different monitoring systems simultaneously (see also: HADOOP-6508).
> * Refresh metrics plugin config without server restart.
> ** Including filtering of metrics per plugin.
> * Support metrics schema for plugins.
> The jira will be resolved when core hadoop components (hdfs, mapreduce) are updated to use the new framework . Updates to external components that use the existing metrics framework will be tracked by different issues.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7780-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 18:52:11 2010
Return-Path: <common-issues-return-7780-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48836 invoked from network); 28 Apr 2010 18:52:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 18:52:11 -0000
Received: (qmail 8078 invoked by uid 500); 28 Apr 2010 18:52:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8044 invoked by uid 500); 28 Apr 2010 18:52:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8036 invoked by uid 99); 28 Apr 2010 18:52:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 18:52:11 +0000
X-ASF-Spam-Status: No, hits=-1361.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 18:52:09 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SIpnwn017470
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 18:51:49 GMT
Message-ID: <8913331.3881272480709335.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 14:51:49 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6730) FileContext#copy needs unit tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

FileContext#copy needs unit tests
---------------------------------

                 Key: HADOOP-6730
                 URL: https://issues.apache.org/jira/browse/HADOOP-6730
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
    Affects Versions: 0.22.0
            Reporter: Eli Collins
             Fix For: 0.22.0


I modifed FileContext#checkDest in HADOOP-6678 and noticed it's only caller FileContext#copy has no callers itself. It needs unit tests.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7781-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 19:14:14 2010
Return-Path: <common-issues-return-7781-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59706 invoked from network); 28 Apr 2010 19:14:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 19:14:14 -0000
Received: (qmail 32263 invoked by uid 500); 28 Apr 2010 19:14:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32196 invoked by uid 500); 28 Apr 2010 19:14:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32133 invoked by uid 99); 28 Apr 2010 19:14:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 19:14:11 +0000
X-ASF-Spam-Status: No, hits=-1361.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 19:14:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SJDnv1021213
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 19:13:49 GMT
Message-ID: <4671784.4281272482029710.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 15:13:49 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6678) Remove FileContext#isFile,
 isDirectory and exists
In-Reply-To: <360625036.668801270253667713.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6678:
--------------------------------

    Attachment: hadoop-6678-2.patch

Thanks Hairong. Patch attached.
* Removes FC#isFile and FC#isDirectory, and uses thereof.
* Refactors FC#exists per previous comment and moves to Util
* Refactors FC#checkDest. Note that it is now passed overwrite instead of false. It (and it's callers) needs unit tests. Filed HADOOP-6730 for that.
* Adds Path#getName tests to TestPath to sanity check my understanding of FC#checkDest.
* Modifies test classes to use the new helper methods

Still needs to be applied with HDFS-1089 (easy 4 line patch). ant test-core passes.

> Remove FileContext#isFile, isDirectory and exists
> -------------------------------------------------
>
>                 Key: HADOOP-6678
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6678
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Hairong Kuang
>            Assignee: Eli Collins
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6678-1.patch, hadoop-6678-2.patch
>
>
> # Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.
> # Remove methods isFile(Path), isDirectory(Path), and exists.
> All these methods are implemented by calling getFileStatus(Path).But most users are not aware of this. They would write code as below: 
> {code}
>   FileContext fc = ..;
>   if (fc.exists(path)) {
>     if (fc.isFile(path)) {
>      ...
>     } else {
>     ...
>     }
>   }
> {code}
> The above code adds unnecessary getFileInfo RPC to NameNode. In our production clusters, we often see that the number of getFileStatus calls is multiple times of the open calls. If we remove isFile, isDirectory, and exists from FileContext, users have to explicitly call getFileStatus first, it is more likely that they will write more efficient code as follow:
> {code}
>   FileContext fc = ...;
>   FileStatus fstatus = fc.getFileStatus(path);
>   if (fstatus.isFile() {
>     ...
>   } else {
>     ...
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7782-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 21:02:11 2010
Return-Path: <common-issues-return-7782-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15807 invoked from network); 28 Apr 2010 21:02:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 21:02:11 -0000
Received: (qmail 72870 invoked by uid 500); 28 Apr 2010 21:02:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72843 invoked by uid 500); 28 Apr 2010 21:02:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72833 invoked by uid 99); 28 Apr 2010 21:02:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 21:02:11 +0000
X-ASF-Spam-Status: No, hits=-1362.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 21:02:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SL1okb014217
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 21:01:50 GMT
Message-ID: <1021566.6631272488510169.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 17:01:50 -0400 (EDT)
From: "Scott Chen (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6515) Make maximum number of http threads
 configurable
In-Reply-To: <1919319763.85831264619136665.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Scott Chen updated HADOOP-6515:
-------------------------------

    Attachment: HADOOP-6515-v3.1.patch

> Make maximum number of http threads configurable
> ------------------------------------------------
>
>                 Key: HADOOP-6515
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6515
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>         Attachments: HADOOP-6515-v2.patch, HADOOP-6515-v3.1.patch, HADOOP-6515-v3.patch, HADOOP-6515.patch
>
>
> We found that http server threads may use considerable amount of resource in NameNode and JobTracker.
> It would be good if the number of threads is allowed to be configured to a smaller number.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7783-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 21:02:12 2010
Return-Path: <common-issues-return-7783-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15854 invoked from network); 28 Apr 2010 21:02:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 21:02:12 -0000
Received: (qmail 73028 invoked by uid 500); 28 Apr 2010 21:02:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72979 invoked by uid 500); 28 Apr 2010 21:02:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72971 invoked by uid 99); 28 Apr 2010 21:02:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 21:02:12 +0000
X-ASF-Spam-Status: No, hits=-1362.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 21:02:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SL1pse014239
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 21:01:51 GMT
Message-ID: <23463341.6701272488511146.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 17:01:51 -0400 (EDT)
From: "Scott Chen (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6515) Make maximum number of http threads
 configurable
In-Reply-To: <1919319763.85831264619136665.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Scott Chen updated HADOOP-6515:
-------------------------------

    Status: Open  (was: Patch Available)

> Make maximum number of http threads configurable
> ------------------------------------------------
>
>                 Key: HADOOP-6515
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6515
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>         Attachments: HADOOP-6515-v2.patch, HADOOP-6515-v3.1.patch, HADOOP-6515-v3.patch, HADOOP-6515.patch
>
>
> We found that http server threads may use considerable amount of resource in NameNode and JobTracker.
> It would be good if the number of threads is allowed to be configured to a smaller number.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7784-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 21:02:16 2010
Return-Path: <common-issues-return-7784-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15956 invoked from network); 28 Apr 2010 21:02:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 21:02:16 -0000
Received: (qmail 73283 invoked by uid 500); 28 Apr 2010 21:02:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73251 invoked by uid 500); 28 Apr 2010 21:02:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73243 invoked by uid 99); 28 Apr 2010 21:02:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 21:02:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 21:02:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SL1rlp014270
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 21:01:53 GMT
Message-ID: <9236384.6801272488513130.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 17:01:53 -0400 (EDT)
From: "Scott Chen (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6515) Make maximum number of http threads
 configurable
In-Reply-To: <1919319763.85831264619136665.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Scott Chen updated HADOOP-6515:
-------------------------------

    Status: Patch Available  (was: Open)

> Make maximum number of http threads configurable
> ------------------------------------------------
>
>                 Key: HADOOP-6515
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6515
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>         Attachments: HADOOP-6515-v2.patch, HADOOP-6515-v3.1.patch, HADOOP-6515-v3.patch, HADOOP-6515.patch
>
>
> We found that http server threads may use considerable amount of resource in NameNode and JobTracker.
> It would be good if the number of threads is allowed to be configured to a smaller number.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7785-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 21:04:14 2010
Return-Path: <common-issues-return-7785-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17125 invoked from network); 28 Apr 2010 21:04:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 21:04:14 -0000
Received: (qmail 73980 invoked by uid 500); 28 Apr 2010 21:04:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73904 invoked by uid 500); 28 Apr 2010 21:04:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73896 invoked by uid 99); 28 Apr 2010 21:04:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 21:04:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 21:04:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SL3oJ1016312
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 21:03:51 GMT
Message-ID: <16304949.6901272488630309.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 17:03:50 -0400 (EDT)
From: "Scott Chen (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6515) Make maximum number of http threads
 configurable
In-Reply-To: <1919319763.85831264619136665.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861939#action_12861939 ] 

Scott Chen commented on HADOOP-6515:
------------------------------------

It has been a while so the patch doesn't apply. I have update it.
@Dhruba, I have removed the sleep in the test. Could you help me review this patch again? Thanks.

> Make maximum number of http threads configurable
> ------------------------------------------------
>
>                 Key: HADOOP-6515
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6515
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>         Attachments: HADOOP-6515-v2.patch, HADOOP-6515-v3.1.patch, HADOOP-6515-v3.patch, HADOOP-6515.patch
>
>
> We found that http server threads may use considerable amount of resource in NameNode and JobTracker.
> It would be good if the number of threads is allowed to be configured to a smaller number.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7786-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 21:26:14 2010
Return-Path: <common-issues-return-7786-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29791 invoked from network); 28 Apr 2010 21:26:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 21:26:13 -0000
Received: (qmail 87531 invoked by uid 500); 28 Apr 2010 21:26:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87494 invoked by uid 500); 28 Apr 2010 21:26:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87486 invoked by uid 99); 28 Apr 2010 21:26:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 21:26:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 21:26:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SLPnDj001451
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 21:25:49 GMT
Message-ID: <23678751.7091272489949674.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 17:25:49 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6678) Remove FileContext#isFile,
 isDirectory and exists
In-Reply-To: <360625036.668801270253667713.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861942#action_12861942 ] 

Hairong Kuang commented on HADOOP-6678:
---------------------------------------

+1. The patch looks good to me.

> Remove FileContext#isFile, isDirectory and exists
> -------------------------------------------------
>
>                 Key: HADOOP-6678
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6678
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Hairong Kuang
>            Assignee: Eli Collins
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6678-1.patch, hadoop-6678-2.patch
>
>
> # Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.
> # Remove methods isFile(Path), isDirectory(Path), and exists.
> All these methods are implemented by calling getFileStatus(Path).But most users are not aware of this. They would write code as below: 
> {code}
>   FileContext fc = ..;
>   if (fc.exists(path)) {
>     if (fc.isFile(path)) {
>      ...
>     } else {
>     ...
>     }
>   }
> {code}
> The above code adds unnecessary getFileInfo RPC to NameNode. In our production clusters, we often see that the number of getFileStatus calls is multiple times of the open calls. If we remove isFile, isDirectory, and exists from FileContext, users have to explicitly call getFileStatus first, it is more likely that they will write more efficient code as follow:
> {code}
>   FileContext fc = ...;
>   FileStatus fstatus = fc.getFileStatus(path);
>   if (fstatus.isFile() {
>     ...
>   } else {
>     ...
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7787-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 21:30:11 2010
Return-Path: <common-issues-return-7787-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32030 invoked from network); 28 Apr 2010 21:30:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 21:30:11 -0000
Received: (qmail 90636 invoked by uid 500); 28 Apr 2010 21:30:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90570 invoked by uid 500); 28 Apr 2010 21:30:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90562 invoked by uid 99); 28 Apr 2010 21:30:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 21:30:11 +0000
X-ASF-Spam-Status: No, hits=-1362.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 21:30:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SLTokp001492
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 21:29:50 GMT
Message-ID: <7445672.7141272490190062.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 17:29:50 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6687) user object in the subject in UGI
  should be reused in case of a relogin.
In-Reply-To: <191053771.27981270592254332.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey reassigned HADOOP-6687:
--------------------------------------------

    Assignee: Jitendra Nath Pandey

> user object in the subject in UGI  should be reused in case of a relogin.
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6687
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6687
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6687-y20.2.patch
>
>
>  In case of a relogin, a new user object is created in the subject. This causes the subject to lose some state for example authentication method. Instead, a new user object should not be added to the subject if one is already present.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7788-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 21:38:13 2010
Return-Path: <common-issues-return-7788-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36761 invoked from network); 28 Apr 2010 21:38:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 21:38:12 -0000
Received: (qmail 102 invoked by uid 500); 28 Apr 2010 21:38:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99885 invoked by uid 500); 28 Apr 2010 21:38:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99873 invoked by uid 99); 28 Apr 2010 21:38:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 21:38:12 +0000
X-ASF-Spam-Status: No, hits=-1362.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 21:38:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SLbpVL001843
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 21:37:51 GMT
Message-ID: <2049080.8131272490670987.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 17:37:50 -0400 (EDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6194) Add service base class and tests to
 hadoop-common/util
In-Reply-To: <165334509.1250255474864.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861950#action_12861950 ] 

Steve Loughran commented on HADOOP-6194:
----------------------------------------

I've submitted a patch to the HDFS services under HDFS-326 which is in sync with this to show it works

Now, how do you find out javac warnings that don't appear to get printed to the hudson console?

> Add service base class and tests to hadoop-common/util
> ------------------------------------------------------
>
>                 Key: HADOOP-6194
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6194
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6194-1.patch, HADOOP-6194.patch, HADOOP-6194.patch
>
>
> For a service base class, we need to add to common
> # The Service class
> # A mock Service class (it's not in test, its in src/, because it helps in functional tests downstream)
> # Unit tests
> This is the issue to cover this work

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7789-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 22:24:15 2010
Return-Path: <common-issues-return-7789-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61575 invoked from network); 28 Apr 2010 22:24:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 22:24:15 -0000
Received: (qmail 40933 invoked by uid 500); 28 Apr 2010 22:24:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40903 invoked by uid 500); 28 Apr 2010 22:24:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40895 invoked by uid 99); 28 Apr 2010 22:24:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 22:24:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 22:24:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SMNpTA005546
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 22:23:51 GMT
Message-ID: <2800020.9731272493431396.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 18:23:51 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6731) Octal umask with less than 3 digits
 is not interpreted correctly
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Octal umask with less than 3 digits is not interpreted correctly
----------------------------------------------------------------

                 Key: HADOOP-6731
                 URL: https://issues.apache.org/jira/browse/HADOOP-6731
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.21.0, 0.22.0
            Reporter: Suresh Srinivas
            Assignee: Suresh Srinivas
             Fix For: 0.22.0


Octal umask 7 is interpreted as 700 instead of 007. Octal umask 77 is interpreted as 770 instead of 077.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7790-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 22:24:16 2010
Return-Path: <common-issues-return-7790-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61632 invoked from network); 28 Apr 2010 22:24:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 22:24:15 -0000
Received: (qmail 41072 invoked by uid 500); 28 Apr 2010 22:24:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41038 invoked by uid 500); 28 Apr 2010 22:24:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41030 invoked by uid 99); 28 Apr 2010 22:24:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 22:24:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 22:24:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SMNpf5005569
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 22:23:51 GMT
Message-ID: <33393941.9761272493431748.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 18:23:51 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6731) Octal umask with less than 3 digits
 is not interpreted correctly
In-Reply-To: <2800020.9731272493431396.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6731:
------------------------------------

    Component/s: fs

> Octal umask with less than 3 digits is not interpreted correctly
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6731
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6731
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>
> Octal umask 7 is interpreted as 700 instead of 007. Octal umask 77 is interpreted as 770 instead of 077.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7791-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 22:28:11 2010
Return-Path: <common-issues-return-7791-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63331 invoked from network); 28 Apr 2010 22:28:10 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 22:28:10 -0000
Received: (qmail 46104 invoked by uid 500); 28 Apr 2010 22:28:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46080 invoked by uid 500); 28 Apr 2010 22:28:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46068 invoked by uid 99); 28 Apr 2010 22:28:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 22:28:10 +0000
X-ASF-Spam-Status: No, hits=-1363.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 22:28:09 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SMRnF8010857
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 22:27:49 GMT
Message-ID: <7125616.9841272493669162.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 18:27:49 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6731) Octal umask with less than 3 digits
 is not handled
In-Reply-To: <2800020.9731272493431396.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6731:
------------------------------------

        Summary: Octal umask with less than 3 digits is not handled  (was: Octal umask with less than 3 digits is not interpreted correctly)
    Description: Umasks with less than three digits should be accepted as valid umask. Umask with single digit, say 7, should be handled as 007. Similarly umask with two digits, say 77, should be handled as 077.  (was: Octal umask 7 is interpreted as 700 instead of 007. Octal umask 77 is interpreted as 770 instead of 077.)

> Octal umask with less than 3 digits is not handled
> --------------------------------------------------
>
>                 Key: HADOOP-6731
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6731
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>
> Umasks with less than three digits should be accepted as valid umask. Umask with single digit, say 7, should be handled as 007. Similarly umask with two digits, say 77, should be handled as 077.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7792-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 22:30:13 2010
Return-Path: <common-issues-return-7792-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64215 invoked from network); 28 Apr 2010 22:30:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 22:30:13 -0000
Received: (qmail 47878 invoked by uid 500); 28 Apr 2010 22:30:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47845 invoked by uid 500); 28 Apr 2010 22:30:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47837 invoked by uid 99); 28 Apr 2010 22:30:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 22:30:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 22:30:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SMTneR015659
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 22:29:49 GMT
Message-ID: <6251785.9871272493789315.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 18:29:49 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6731) Octal umask with less than 3 digits
 is not handled
In-Reply-To: <2800020.9731272493431396.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6731:
------------------------------------

    Attachment: HADOOP-6713.patch

Attached patch:
# Handle single digit and two digit octal umasks
# Interprets them and applies them correctly as in the description.

> Octal umask with less than 3 digits is not handled
> --------------------------------------------------
>
>                 Key: HADOOP-6731
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6731
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6713.patch
>
>
> Umasks with less than three digits should be accepted as valid umask. Umask with single digit, say 7, should be handled as 007. Similarly umask with two digits, say 77, should be handled as 077.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7793-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 22:34:12 2010
Return-Path: <common-issues-return-7793-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66049 invoked from network); 28 Apr 2010 22:34:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 22:34:11 -0000
Received: (qmail 52422 invoked by uid 500); 28 Apr 2010 22:34:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52372 invoked by uid 500); 28 Apr 2010 22:34:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52363 invoked by uid 99); 28 Apr 2010 22:34:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 22:34:11 +0000
X-ASF-Spam-Status: No, hits=-1363.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 22:34:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SMXoNi025466
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 22:33:50 GMT
Message-ID: <29436901.9911272494030076.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 18:33:50 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6731) Octal umask with less than 3 digits
 is not handled
In-Reply-To: <2800020.9731272493431396.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6731:
------------------------------------

    Attachment: HADOOP-6731.patch

Renaming the patch with correct jira number.

> Octal umask with less than 3 digits is not handled
> --------------------------------------------------
>
>                 Key: HADOOP-6731
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6731
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6731.patch
>
>
> Umasks with less than three digits should be accepted as valid umask. Umask with single digit, say 7, should be handled as 007. Similarly umask with two digits, say 77, should be handled as 077.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7794-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 22:34:12 2010
Return-Path: <common-issues-return-7794-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66057 invoked from network); 28 Apr 2010 22:34:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 22:34:11 -0000
Received: (qmail 52581 invoked by uid 500); 28 Apr 2010 22:34:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52508 invoked by uid 500); 28 Apr 2010 22:34:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52470 invoked by uid 99); 28 Apr 2010 22:34:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 22:34:11 +0000
X-ASF-Spam-Status: No, hits=-1363.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 22:34:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SMXosI025489
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 22:33:50 GMT
Message-ID: <12812802.9931272494030506.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 18:33:50 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6731) Octal umask with less than 3 digits
 is not handled
In-Reply-To: <2800020.9731272493431396.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6731:
------------------------------------

    Attachment:     (was: HADOOP-6713.patch)

> Octal umask with less than 3 digits is not handled
> --------------------------------------------------
>
>                 Key: HADOOP-6731
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6731
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6731.patch
>
>
> Umasks with less than three digits should be accepted as valid umask. Umask with single digit, say 7, should be handled as 007. Similarly umask with two digits, say 77, should be handled as 077.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7795-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 22:42:12 2010
Return-Path: <common-issues-return-7795-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70417 invoked from network); 28 Apr 2010 22:42:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 22:42:12 -0000
Received: (qmail 60566 invoked by uid 500); 28 Apr 2010 22:42:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60503 invoked by uid 500); 28 Apr 2010 22:42:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60495 invoked by uid 99); 28 Apr 2010 22:42:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 22:42:12 +0000
X-ASF-Spam-Status: No, hits=-1363.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 22:42:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SMfpEJ024898
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 22:41:51 GMT
Message-ID: <28006824.10081272494511349.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 18:41:51 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6678) Remove FileContext#isFile,
 isDirectory and exists
In-Reply-To: <360625036.668801270253667713.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6678:
----------------------------------

          Status: Patch Available  (was: Open)
    Hadoop Flags: [Incompatible change, Reviewed]  (was: [Incompatible change])

> Remove FileContext#isFile, isDirectory and exists
> -------------------------------------------------
>
>                 Key: HADOOP-6678
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6678
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Hairong Kuang
>            Assignee: Eli Collins
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6678-1.patch, hadoop-6678-2.patch
>
>
> # Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.
> # Remove methods isFile(Path), isDirectory(Path), and exists.
> All these methods are implemented by calling getFileStatus(Path).But most users are not aware of this. They would write code as below: 
> {code}
>   FileContext fc = ..;
>   if (fc.exists(path)) {
>     if (fc.isFile(path)) {
>      ...
>     } else {
>     ...
>     }
>   }
> {code}
> The above code adds unnecessary getFileInfo RPC to NameNode. In our production clusters, we often see that the number of getFileStatus calls is multiple times of the open calls. If we remove isFile, isDirectory, and exists from FileContext, users have to explicitly call getFileStatus first, it is more likely that they will write more efficient code as follow:
> {code}
>   FileContext fc = ...;
>   FileStatus fstatus = fc.getFileStatus(path);
>   if (fstatus.isFile() {
>     ...
>   } else {
>     ...
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7796-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 22:42:14 2010
Return-Path: <common-issues-return-7796-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70476 invoked from network); 28 Apr 2010 22:42:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 22:42:14 -0000
Received: (qmail 60678 invoked by uid 500); 28 Apr 2010 22:42:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60640 invoked by uid 500); 28 Apr 2010 22:42:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60632 invoked by uid 99); 28 Apr 2010 22:42:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 22:42:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 22:42:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SMfoWt024819
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 22:41:50 GMT
Message-ID: <27422764.10021272494510170.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 18:41:50 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6678) Remove FileContext#isFile,
 isDirectory and exists
In-Reply-To: <360625036.668801270253667713.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6678:
----------------------------------

    Status: Open  (was: Patch Available)

> Remove FileContext#isFile, isDirectory and exists
> -------------------------------------------------
>
>                 Key: HADOOP-6678
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6678
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Hairong Kuang
>            Assignee: Eli Collins
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6678-1.patch, hadoop-6678-2.patch
>
>
> # Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.
> # Remove methods isFile(Path), isDirectory(Path), and exists.
> All these methods are implemented by calling getFileStatus(Path).But most users are not aware of this. They would write code as below: 
> {code}
>   FileContext fc = ..;
>   if (fc.exists(path)) {
>     if (fc.isFile(path)) {
>      ...
>     } else {
>     ...
>     }
>   }
> {code}
> The above code adds unnecessary getFileInfo RPC to NameNode. In our production clusters, we often see that the number of getFileStatus calls is multiple times of the open calls. If we remove isFile, isDirectory, and exists from FileContext, users have to explicitly call getFileStatus first, it is more likely that they will write more efficient code as follow:
> {code}
>   FileContext fc = ...;
>   FileStatus fstatus = fc.getFileStatus(path);
>   if (fstatus.isFile() {
>     ...
>   } else {
>     ...
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7797-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 22:50:17 2010
Return-Path: <common-issues-return-7797-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75686 invoked from network); 28 Apr 2010 22:50:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 22:50:16 -0000
Received: (qmail 67316 invoked by uid 500); 28 Apr 2010 22:50:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67267 invoked by uid 500); 28 Apr 2010 22:50:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67259 invoked by uid 99); 28 Apr 2010 22:50:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 22:50:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 22:50:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SMnqet017091
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 22:49:52 GMT
Message-ID: <17254952.10281272494992361.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 18:49:52 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6698) Revert the io.serialization package
 to 0.20.2's api
In-Reply-To: <21261745.7641270969841016.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861977#action_12861977 ] 

Doug Cutting commented on HADOOP-6698:
--------------------------------------

> HADOOP-6420 was completed with MAPREDUCE-1126 as a use case, but it's an independent feature. Does it need to be backed out?

If it has no other use cases, it should be backed out, no?  If someone comes up with a use case, we can add it back, but, since it was added with MAPREDUCE-1126 as its only use case, and MAPREDUCE-1126 has stalled, I'd vote to back it out.

> Revert the io.serialization package to 0.20.2's api
> ---------------------------------------------------
>
>                 Key: HADOOP-6698
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6698
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Owen O'Malley
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6698.patch, HADOOP-6698.patch, HADOOP-6698.patch
>
>
> I have a lot of concern about the usability of the new generic serialization framework. Toward that end, I've filed a jira for improving it. There is resistance to pushing a new API into 0.21 at the last moment, so we should back out the changes rather than introducing a new api in 0.21 and deprecating it in 0.22.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7798-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 22:52:15 2010
Return-Path: <common-issues-return-7798-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76669 invoked from network); 28 Apr 2010 22:52:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 22:52:14 -0000
Received: (qmail 68710 invoked by uid 500); 28 Apr 2010 22:52:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68685 invoked by uid 500); 28 Apr 2010 22:52:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68677 invoked by uid 99); 28 Apr 2010 22:52:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 22:52:14 +0000
X-ASF-Spam-Status: No, hits=-1363.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 22:52:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SMpr8q020542
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 22:51:53 GMT
Message-ID: <14299726.10561272495113374.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 18:51:53 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6698) Revert the io.serialization package
 to 0.20.2's api
In-Reply-To: <21261745.7641270969841016.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861982#action_12861982 ] 

Doug Cutting commented on HADOOP-6698:
--------------------------------------

I should have been clearer: I'd vote to remove HADOOP-6420 as a part of this issue, since it's only current use case is to work with the serialization APIs that are being reverted in this issue.

> Revert the io.serialization package to 0.20.2's api
> ---------------------------------------------------
>
>                 Key: HADOOP-6698
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6698
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Owen O'Malley
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6698.patch, HADOOP-6698.patch, HADOOP-6698.patch
>
>
> I have a lot of concern about the usability of the new generic serialization framework. Toward that end, I've filed a jira for improving it. There is resistance to pushing a new API into 0.21 at the last moment, so we should back out the changes rather than introducing a new api in 0.21 and deprecating it in 0.22.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7799-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 22:52:15 2010
Return-Path: <common-issues-return-7799-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76697 invoked from network); 28 Apr 2010 22:52:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 22:52:14 -0000
Received: (qmail 68852 invoked by uid 500); 28 Apr 2010 22:52:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68820 invoked by uid 500); 28 Apr 2010 22:52:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68812 invoked by uid 99); 28 Apr 2010 22:52:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 22:52:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 22:52:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SMpo5A020415
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 22:51:50 GMT
Message-ID: <16216929.10421272495110621.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 18:51:50 -0400 (EDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6194) Add service base class and tests to
 hadoop-common/util
In-Reply-To: <165334509.1250255474864.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron Kimball updated HADOOP-6194:
----------------------------------

    Attachment: find-new-warnings

Steve, I've attached a bash script that does some awk + diff magic on the trunk javac warnings and patch javac warnings output to attempt to discover the new ones. It does a reasonable job (at the very least, it should tell you which files to look at).

> Add service base class and tests to hadoop-common/util
> ------------------------------------------------------
>
>                 Key: HADOOP-6194
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6194
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>             Fix For: 0.22.0
>
>         Attachments: find-new-warnings, HADOOP-6194-1.patch, HADOOP-6194.patch, HADOOP-6194.patch
>
>
> For a service base class, we need to add to common
> # The Service class
> # A mock Service class (it's not in test, its in src/, because it helps in functional tests downstream)
> # Unit tests
> This is the issue to cover this work

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7800-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 22:58:11 2010
Return-Path: <common-issues-return-7800-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79802 invoked from network); 28 Apr 2010 22:58:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 22:58:11 -0000
Received: (qmail 74868 invoked by uid 500); 28 Apr 2010 22:58:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74836 invoked by uid 500); 28 Apr 2010 22:58:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74828 invoked by uid 99); 28 Apr 2010 22:58:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 22:58:10 +0000
X-ASF-Spam-Status: No, hits=-1363.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 22:58:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SMvnnQ002175
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 22:57:49 GMT
Message-ID: <23406023.10631272495469609.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 18:57:49 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6710) Symbolic umask for file creation is
 not consistent with posix
In-Reply-To: <8947165.155011271375809341.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6710:
------------------------------------

    Attachment: hadoop-6710.rel20.patch

20 version of the patch.

> Symbolic umask for file creation is not consistent with posix
> -------------------------------------------------------------
>
>                 Key: HADOOP-6710
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6710
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>         Attachments: hadoop-6710.patch, hadoop-6710.rel20.patch
>
>
> Currently both octal and symbolic umask are used to reset the file creation mode bits. This is not consistent with the behavior defined in posix. Making it consistent would avoid confusion to the HDFS users.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7801-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 23:06:14 2010
Return-Path: <common-issues-return-7801-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84143 invoked from network); 28 Apr 2010 23:06:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 23:06:14 -0000
Received: (qmail 78978 invoked by uid 500); 28 Apr 2010 23:06:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78933 invoked by uid 500); 28 Apr 2010 23:06:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78925 invoked by uid 99); 28 Apr 2010 23:06:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 23:06:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 23:06:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SN5oba008124
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 23:05:50 GMT
Message-ID: <10352364.10791272495950440.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 19:05:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6678) Remove FileContext#isFile,
 isDirectory and exists
In-Reply-To: <360625036.668801270253667713.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861989#action_12861989 ] 

Hadoop QA commented on HADOOP-6678:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12443114/hadoop-6678-2.patch
  against trunk revision 939026.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 24 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/61/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/61/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/61/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/61/console

This message is automatically generated.

> Remove FileContext#isFile, isDirectory and exists
> -------------------------------------------------
>
>                 Key: HADOOP-6678
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6678
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Hairong Kuang
>            Assignee: Eli Collins
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6678-1.patch, hadoop-6678-2.patch
>
>
> # Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.
> # Remove methods isFile(Path), isDirectory(Path), and exists.
> All these methods are implemented by calling getFileStatus(Path).But most users are not aware of this. They would write code as below: 
> {code}
>   FileContext fc = ..;
>   if (fc.exists(path)) {
>     if (fc.isFile(path)) {
>      ...
>     } else {
>     ...
>     }
>   }
> {code}
> The above code adds unnecessary getFileInfo RPC to NameNode. In our production clusters, we often see that the number of getFileStatus calls is multiple times of the open calls. If we remove isFile, isDirectory, and exists from FileContext, users have to explicitly call getFileStatus first, it is more likely that they will write more efficient code as follow:
> {code}
>   FileContext fc = ...;
>   FileStatus fstatus = fc.getFileStatus(path);
>   if (fstatus.isFile() {
>     ...
>   } else {
>     ...
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7802-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Apr 28 23:46:18 2010
Return-Path: <common-issues-return-7802-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2882 invoked from network); 28 Apr 2010 23:46:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Apr 2010 23:46:18 -0000
Received: (qmail 6318 invoked by uid 500); 28 Apr 2010 23:46:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6260 invoked by uid 500); 28 Apr 2010 23:46:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6252 invoked by uid 99); 28 Apr 2010 23:46:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 23:46:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Apr 2010 23:46:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3SNjrHH017260
	for <common-issues@hadoop.apache.org>; Wed, 28 Apr 2010 23:45:54 GMT
Message-ID: <24818683.11321272498353794.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 19:45:53 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6698) Revert the io.serialization package
 to 0.20.2's api
In-Reply-To: <21261745.7641270969841016.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861999#action_12861999 ] 

Tom White commented on HADOOP-6698:
-----------------------------------

OK, I can remove HADOOP-6420 here. I'll commit this soon unless there are any objections.

> Revert the io.serialization package to 0.20.2's api
> ---------------------------------------------------
>
>                 Key: HADOOP-6698
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6698
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Owen O'Malley
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6698.patch, HADOOP-6698.patch, HADOOP-6698.patch
>
>
> I have a lot of concern about the usability of the new generic serialization framework. Toward that end, I've filed a jira for improving it. There is resistance to pushing a new API into 0.21 at the last moment, so we should back out the changes rather than introducing a new api in 0.21 and deprecating it in 0.22.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7803-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 00:02:13 2010
Return-Path: <common-issues-return-7803-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10843 invoked from network); 29 Apr 2010 00:02:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 00:02:13 -0000
Received: (qmail 18630 invoked by uid 500); 29 Apr 2010 00:02:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18598 invoked by uid 500); 29 Apr 2010 00:02:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18590 invoked by uid 99); 29 Apr 2010 00:02:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 00:02:13 +0000
X-ASF-Spam-Status: No, hits=-1363.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 00:02:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3T01p0X019918
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 00:01:52 GMT
Message-ID: <33129608.11541272499311823.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 20:01:51 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6678) Remove FileContext#isFile,
 isDirectory and exists
In-Reply-To: <360625036.668801270253667713.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6678:
----------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]  (was: [Incompatible change, Reviewed])
      Resolution: Fixed

I've just committed this. Thanks, Eli!

> Remove FileContext#isFile, isDirectory and exists
> -------------------------------------------------
>
>                 Key: HADOOP-6678
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6678
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Hairong Kuang
>            Assignee: Eli Collins
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6678-1.patch, hadoop-6678-2.patch
>
>
> # Add a method  Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.
> # Remove methods isFile(Path), isDirectory(Path), and exists.
> All these methods are implemented by calling getFileStatus(Path).But most users are not aware of this. They would write code as below: 
> {code}
>   FileContext fc = ..;
>   if (fc.exists(path)) {
>     if (fc.isFile(path)) {
>      ...
>     } else {
>     ...
>     }
>   }
> {code}
> The above code adds unnecessary getFileInfo RPC to NameNode. In our production clusters, we often see that the number of getFileStatus calls is multiple times of the open calls. If we remove isFile, isDirectory, and exists from FileContext, users have to explicitly call getFileStatus first, it is more likely that they will write more efficient code as follow:
> {code}
>   FileContext fc = ...;
>   FileStatus fstatus = fc.getFileStatus(path);
>   if (fstatus.isFile() {
>     ...
>   } else {
>     ...
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7804-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 00:08:11 2010
Return-Path: <common-issues-return-7804-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14161 invoked from network); 29 Apr 2010 00:08:10 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 00:08:10 -0000
Received: (qmail 25884 invoked by uid 500); 29 Apr 2010 00:08:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25850 invoked by uid 500); 29 Apr 2010 00:08:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25842 invoked by uid 99); 29 Apr 2010 00:08:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 00:08:10 +0000
X-ASF-Spam-Status: No, hits=-1363.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 00:08:09 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3T07nZe025467
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 00:07:49 GMT
Message-ID: <15529177.11731272499669322.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 20:07:49 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862007#action_12862007 ] 

Suresh Srinivas commented on HADOOP-6563:
-----------------------------------------

Ignore my comment 3, since symlinks should allow creation of dangling links.

> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6563-1.patch, hadoop-6563-2.patch, hadoop-6563-3.patch, hadoop-6563-4.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7805-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 00:30:14 2010
Return-Path: <common-issues-return-7805-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32403 invoked from network); 29 Apr 2010 00:30:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 00:30:14 -0000
Received: (qmail 56239 invoked by uid 500); 29 Apr 2010 00:30:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56207 invoked by uid 500); 29 Apr 2010 00:30:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56198 invoked by uid 99); 29 Apr 2010 00:30:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 00:30:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 00:30:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3T0Toix006784
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 00:29:50 GMT
Message-ID: <21883365.12751272500990461.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 20:29:50 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6731) Octal umask with less than 3 digits
 is not handled
In-Reply-To: <2800020.9731272493431396.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862027#action_12862027 ] 

Eli Collins commented on HADOOP-6731:
-------------------------------------

+1  Looks good to me.

> Octal umask with less than 3 digits is not handled
> --------------------------------------------------
>
>                 Key: HADOOP-6731
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6731
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6731.patch
>
>
> Umasks with less than three digits should be accepted as valid umask. Umask with single digit, say 7, should be handled as 007. Similarly umask with two digits, say 77, should be handled as 077.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7806-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 00:38:13 2010
Return-Path: <common-issues-return-7806-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35668 invoked from network); 29 Apr 2010 00:38:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 00:38:13 -0000
Received: (qmail 59852 invoked by uid 500); 29 Apr 2010 00:38:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59796 invoked by uid 500); 29 Apr 2010 00:38:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59788 invoked by uid 99); 29 Apr 2010 00:38:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 00:38:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 00:38:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3T0boKZ004175
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 00:37:50 GMT
Message-ID: <29620962.12831272501470064.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 20:37:50 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6732) Improve FsShell's heap consumption by
 switching to listStatus that returns an iterator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Improve FsShell's heap consumption by switching to listStatus that returns an iterator
--------------------------------------------------------------------------------------

                 Key: HADOOP-6732
                 URL: https://issues.apache.org/jira/browse/HADOOP-6732
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Hairong Kuang
             Fix For: 0.22.0


When listing a large directory from the command line using the default heap configuration, FsShell often runs out of memory. This is because all stats of the entries under the directory need to be in memory before printing them. The new API listStatus that returns an iterator of FileStatus, which implemented in HDFS-1091, no longer requires that all entries are fetched first. Thus switching to this new API will greatly improve the use of heap space.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7807-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 00:48:13 2010
Return-Path: <common-issues-return-7807-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42696 invoked from network); 29 Apr 2010 00:48:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 00:48:12 -0000
Received: (qmail 70515 invoked by uid 500); 29 Apr 2010 00:48:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70471 invoked by uid 500); 29 Apr 2010 00:48:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70463 invoked by uid 99); 29 Apr 2010 00:48:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 00:48:12 +0000
X-ASF-Spam-Status: No, hits=-1363.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 00:48:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3T0lp9p019686
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 00:47:51 GMT
Message-ID: <20072337.13341272502071348.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 20:47:51 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6515) Make maximum number of http threads
 configurable
In-Reply-To: <1919319763.85831264619136665.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862034#action_12862034 ] 

Hadoop QA commented on HADOOP-6515:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12443121/HADOOP-6515-v3.1.patch
  against trunk revision 939140.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 4 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/487/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/487/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/487/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/487/console

This message is automatically generated.

> Make maximum number of http threads configurable
> ------------------------------------------------
>
>                 Key: HADOOP-6515
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6515
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>         Attachments: HADOOP-6515-v2.patch, HADOOP-6515-v3.1.patch, HADOOP-6515-v3.patch, HADOOP-6515.patch
>
>
> We found that http server threads may use considerable amount of resource in NameNode and JobTracker.
> It would be good if the number of threads is allowed to be configured to a smaller number.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7808-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 01:09:12 2010
Return-Path: <common-issues-return-7808-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46866 invoked from network); 29 Apr 2010 01:09:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 01:09:12 -0000
Received: (qmail 82909 invoked by uid 500); 29 Apr 2010 01:09:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82884 invoked by uid 500); 29 Apr 2010 01:09:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82876 invoked by uid 99); 29 Apr 2010 01:09:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 01:09:12 +0000
X-ASF-Spam-Status: No, hits=-1363.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 01:09:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3T18oN9020005
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 01:08:51 GMT
Message-ID: <19107557.13681272503330896.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 21:08:50 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6526) Need mapping from long principal
 names to local OS user names
In-Reply-To: <786866304.151051264870594555.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862041#action_12862041 ] 

Konstantin Boudnik commented on HADOOP-6526:
--------------------------------------------

[The patch|https://issues.apache.org/jira/secure/attachment/12442917/3595485.patch] is done on top of the last {{HADOOP-6526-y20.4.patch}} and isn't for commit.

> Need mapping from long principal names to local OS user names
> -------------------------------------------------------------
>
>                 Key: HADOOP-6526
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6526
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 3595485.patch, c-6526-y20.patch, c-6526-y20.patch, c-6526.patch, HADOOP-6526-y20.2.patch, HADOOP-6526-y20.4.patch
>
>
> We need a configurable mapping from full user names (eg. omalley@APACHE.ORG) to local user names (eg. omalley). For many organizations it is sufficient to just use the prefix, however, in the case of shared clusters there may be duplicated prefixes. A configurable mapping will let administrators resolve the issue.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7809-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 01:13:14 2010
Return-Path: <common-issues-return-7809-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47180 invoked from network); 29 Apr 2010 01:13:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 01:13:13 -0000
Received: (qmail 85295 invoked by uid 500); 29 Apr 2010 01:13:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85267 invoked by uid 500); 29 Apr 2010 01:13:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85259 invoked by uid 99); 29 Apr 2010 01:13:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 01:13:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 01:13:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3T1Cne8020057
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 01:12:50 GMT
Message-ID: <4696179.13771272503569659.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 21:12:49 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6731) Octal umask with less than 3 digits
 is not handled
In-Reply-To: <2800020.9731272493431396.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862043#action_12862043 ] 

Jakob Homan commented on HADOOP-6731:
-------------------------------------

If this change goes through, we should update the documentation in FsShell.java (in common):
{noformat}      "\tMODE\tMode is same as mode used for chmod shell command.\n" +
      "\t\tOnly letters recognized are 'rwxXt'. E.g. +t,a+r,g-w,+rwx,o=r\n\n" +
      "\tOCTALMODE Mode specifed in 3 or 4 digits. If 4 digits, the first may\n" +
      "\tbe 1 or 0 to turn the sticky bit on or off, respectively.  Unlike " +
      "\tshell command, it is not possible to specify only part of the mode\n" +{noformat}
Eli - it's always good to consider the other components as well!

> Octal umask with less than 3 digits is not handled
> --------------------------------------------------
>
>                 Key: HADOOP-6731
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6731
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6731.patch
>
>
> Umasks with less than three digits should be accepted as valid umask. Umask with single digit, say 7, should be handled as 007. Similarly umask with two digits, say 77, should be handled as 077.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7810-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 03:33:13 2010
Return-Path: <common-issues-return-7810-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65453 invoked from network); 29 Apr 2010 03:33:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 03:33:13 -0000
Received: (qmail 67219 invoked by uid 500); 29 Apr 2010 03:33:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67103 invoked by uid 500); 29 Apr 2010 03:33:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67095 invoked by uid 99); 29 Apr 2010 03:33:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 03:33:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 03:33:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3T3Wmf6027501
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 03:32:48 GMT
Message-ID: <2563190.15351272511968608.JavaMail.jira@thor>
Date: Wed, 28 Apr 2010 23:32:48 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6733) Create a test for FileSystem API
 compatibility between releases
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Create a test for FileSystem API compatibility between releases
---------------------------------------------------------------

                 Key: HADOOP-6733
                 URL: https://issues.apache.org/jira/browse/HADOOP-6733
             Project: Hadoop Common
          Issue Type: Sub-task
          Components: fs
            Reporter: Tom White
            Priority: Blocker
             Fix For: 0.21.0


We should have an automated test for checking that programs written against an old version of the FileSystem API still run with a newer version. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7811-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 04:34:11 2010
Return-Path: <common-issues-return-7811-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70689 invoked from network); 29 Apr 2010 04:34:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 04:34:11 -0000
Received: (qmail 91572 invoked by uid 500); 29 Apr 2010 04:34:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91445 invoked by uid 500); 29 Apr 2010 04:34:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91436 invoked by uid 99); 29 Apr 2010 04:34:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 04:34:10 +0000
X-ASF-Spam-Status: No, hits=-1364.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 04:34:09 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3T4XnFO022924
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 04:33:49 GMT
Message-ID: <23398578.16061272515629443.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 00:33:49 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6563:
--------------------------------

    Status: Patch Available  (was: Open)

> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6563-1.patch, hadoop-6563-2.patch, hadoop-6563-3.patch, hadoop-6563-4.patch, hadoop-6563-5.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7812-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 04:34:13 2010
Return-Path: <common-issues-return-7812-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70732 invoked from network); 29 Apr 2010 04:34:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 04:34:12 -0000
Received: (qmail 91624 invoked by uid 500); 29 Apr 2010 04:34:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91588 invoked by uid 500); 29 Apr 2010 04:34:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91580 invoked by uid 99); 29 Apr 2010 04:34:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 04:34:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 04:34:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3T4XmWS022912
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 04:33:49 GMT
Message-ID: <1185465.16021272515628759.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 00:33:48 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6563:
--------------------------------

    Attachment: hadoop-6563-5.patch

The updated table is more clear, thanks. Modified tests like testRenameSymlinkToFileItLinksTo check the particular exception thrown (like the tests that cover renaming a file, directory, etc to itself). Added additional comments to the tests and fixed the typo. Patch attached.


> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6563-1.patch, hadoop-6563-2.patch, hadoop-6563-3.patch, hadoop-6563-4.patch, hadoop-6563-5.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7813-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 04:34:13 2010
Return-Path: <common-issues-return-7813-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70744 invoked from network); 29 Apr 2010 04:34:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 04:34:13 -0000
Received: (qmail 91765 invoked by uid 500); 29 Apr 2010 04:34:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91723 invoked by uid 500); 29 Apr 2010 04:34:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91615 invoked by uid 99); 29 Apr 2010 04:34:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 04:34:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 04:34:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3T4Xn6r022918
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 04:33:49 GMT
Message-ID: <20334088.16041272515629233.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 00:33:49 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6563:
--------------------------------

    Status: Open  (was: Patch Available)

> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6563-1.patch, hadoop-6563-2.patch, hadoop-6563-3.patch, hadoop-6563-4.patch, hadoop-6563-5.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7814-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 04:48:13 2010
Return-Path: <common-issues-return-7814-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73002 invoked from network); 29 Apr 2010 04:48:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 04:48:13 -0000
Received: (qmail 2959 invoked by uid 500); 29 Apr 2010 04:48:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2934 invoked by uid 500); 29 Apr 2010 04:48:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2923 invoked by uid 99); 29 Apr 2010 04:48:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 04:48:12 +0000
X-ASF-Spam-Status: No, hits=-1364.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 04:48:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3T4loba023131
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 04:47:50 GMT
Message-ID: <20524089.16611272516470209.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 00:47:50 -0400 (EDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6515) Make maximum number of http threads
 configurable
In-Reply-To: <1919319763.85831264619136665.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862094#action_12862094 ] 

dhruba borthakur commented on HADOOP-6515:
------------------------------------------

+1 patch looks good.

> Make maximum number of http threads configurable
> ------------------------------------------------
>
>                 Key: HADOOP-6515
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6515
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>         Attachments: HADOOP-6515-v2.patch, HADOOP-6515-v3.1.patch, HADOOP-6515-v3.patch, HADOOP-6515.patch
>
>
> We found that http server threads may use considerable amount of resource in NameNode and JobTracker.
> It would be good if the number of threads is allowed to be configured to a smaller number.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7815-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 09:34:22 2010
Return-Path: <common-issues-return-7815-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28574 invoked from network); 29 Apr 2010 09:34:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 09:34:22 -0000
Received: (qmail 93051 invoked by uid 500); 29 Apr 2010 09:34:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93029 invoked by uid 500); 29 Apr 2010 09:34:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93021 invoked by uid 99); 29 Apr 2010 09:34:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 09:34:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 09:34:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3T9Xto3025460
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 09:33:55 GMT
Message-ID: <33320469.2111272533635600.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 05:33:55 -0400 (EDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6634) AccessControlList uses full-principal
 names to verify acls causing queue-acls to fail
In-Reply-To: <136802909.310821268816187241.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sharad Agarwal updated HADOOP-6634:
-----------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

I just committed this. Thanks Vinod.

> AccessControlList uses full-principal names to verify acls causing queue-acls to fail
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6634
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6634
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Vinod K V
>            Assignee: Vinod K V
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6634-20100317-ydist.1.txt, HADOOP-6634-20100428.txt
>
>
> ACL configuration so far was using short user-names. With the changed {{UserGroupInformation}}, short names are different from the long fully-qualified names. {{AccessControlList}} continues to use {{UserGroupInformation.getUserName()}} for verifying access control. Because of this, queue acls fail for a user "user@domain.org" even though the short name "user" is part of the acl configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7816-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 09:40:21 2010
Return-Path: <common-issues-return-7816-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29737 invoked from network); 29 Apr 2010 09:40:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 09:40:20 -0000
Received: (qmail 5210 invoked by uid 500); 29 Apr 2010 09:40:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4912 invoked by uid 500); 29 Apr 2010 09:40:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4901 invoked by uid 99); 29 Apr 2010 09:40:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 09:40:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 09:40:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3T9dsb0025489
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 09:39:54 GMT
Message-ID: <23151929.2131272533994276.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 05:39:54 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6715) AccessControlList.toString()
 returns empty string when we set acl to "*"
In-Reply-To: <7375715.82441271756441695.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862144#action_12862144 ] 

Vinod K V commented on HADOOP-6715:
-----------------------------------

Um.. 'ALL', 'USERS', 'NONE' can themselves be users/groups and will lead to confusions, however corner-cased they might be.

How about something simpler like the following? Essentially, I'm trying to make the space character stand out by using special chars '[' and ']' which are unlikely to end up in user/group names.
{code}
Job-ACLs: [*]
Job-ACLs: [ ]
Job-ACLs: [ group1]
Job-ACLs: [user1,user2 group1,group2]
{code}

> AccessControlList.toString() returns empty string when we set acl to "*"
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6715
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6715
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security, util
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>
> AccessControlList.toString() returns empty string when we set the acl to "\*" and also when we set it to empty(i.e. " "). This is causing wrong values for ACLs shown on jobdetails.jsp and jobdetailshistory.jsp web pages when acls are set to "\*".
> I think AccessControlList.toString() needs to be changed to return "\*" when we set the acl to "\*".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7817-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 09:44:20 2010
Return-Path: <common-issues-return-7817-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31433 invoked from network); 29 Apr 2010 09:44:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 09:44:20 -0000
Received: (qmail 9901 invoked by uid 500); 29 Apr 2010 09:44:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9824 invoked by uid 500); 29 Apr 2010 09:44:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9816 invoked by uid 99); 29 Apr 2010 09:44:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 09:44:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 09:44:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3T9hsVw025502
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 09:43:54 GMT
Message-ID: <33395135.2151272534234285.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 05:43:54 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6715) AccessControlList.toString()
 returns empty string when we set acl to "*"
In-Reply-To: <7375715.82441271756441695.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862145#action_12862145 ] 

Vinod K V commented on HADOOP-6715:
-----------------------------------

In fact more spaces between user-list and group-list makes things clearer, I think
{code}
Job-ACLs: [*]
Job-ACLs: [             ]
Job-ACLs: [             group1]
Job-ACLs: [user1,user2             group1,group2]
{code}

> AccessControlList.toString() returns empty string when we set acl to "*"
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6715
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6715
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security, util
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>
> AccessControlList.toString() returns empty string when we set the acl to "\*" and also when we set it to empty(i.e. " "). This is causing wrong values for ACLs shown on jobdetails.jsp and jobdetailshistory.jsp web pages when acls are set to "\*".
> I think AccessControlList.toString() needs to be changed to return "\*" when we set the acl to "\*".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7818-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 11:00:21 2010
Return-Path: <common-issues-return-7818-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49440 invoked from network); 29 Apr 2010 11:00:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 11:00:20 -0000
Received: (qmail 14410 invoked by uid 500); 29 Apr 2010 11:00:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14279 invoked by uid 500); 29 Apr 2010 11:00:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14262 invoked by uid 99); 29 Apr 2010 11:00:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 11:00:16 +0000
X-ASF-Spam-Status: No, hits=-1365.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 11:00:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TAxs8M027733
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 10:59:54 GMT
Message-ID: <30509879.3061272538794368.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 06:59:54 -0400 (EDT)
From: "Joni Niemi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6734) Jets3tNativeFileSystemStore wrongly
 calls S3Service.createBucket during initialisation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Jets3tNativeFileSystemStore wrongly calls S3Service.createBucket during initialisation
--------------------------------------------------------------------------------------

                 Key: HADOOP-6734
                 URL: https://issues.apache.org/jira/browse/HADOOP-6734
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs/s3
    Affects Versions: 0.18.3
         Environment: S3 Europe
            Reporter: Joni Niemi


Reason: If a bucket is created with CreateBucketConfiguration specified, s3service.createBucket will fail. 
Symptoms: If a bucket has CreateBucketConfiguration, Jets3tNativeFileSystemStore will fail with BucketAlreadyOwnedByYou Error.

A detailed descrioption from a blog (http://john.keyes.ie/boto-create_bucket-bucketalreadyownedbyyou-error/)

{quote}This evening I encountered a problem with it though. When the bucket did not exist, the method behaved as expected. When the bucket did exist though I received the following error response:

{code}<?xml version="1.0" encoding="UTF-8"?>
<Error>
    <Code>BucketAlreadyOwnedByYou</Code>
    <Message>
        Your previous request to create the named 
        bucket succeeded and you already own it.
    </Message>
    ...
</Error>{code}

This problem only manifests itself with buckets that are hosted in the EU. If the bucket is created in the US then the create_bucket method behaves as described.

    For buckets created with a <CreateBucketConfiguration>, you will receive an error if you attempt to recreate the same bucket.

To create a bucket in the EU, the bucket is created with a CreateBucketConfiguration specified. I now use the following code to avoid the problem and it works for both US and EU buckets.

{code}
def get_bucket():
    try:
        bucket = conn.get_bucket('xxx', validate=True)
    except S3ResponseError, e:
        if e.code == "NoSuchBucket":
            bucket = conn.create_bucket('xxx', location='EU')
        else:
            raise e
    return bucket
{code}
{quote}



-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7819-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 11:02:27 2010
Return-Path: <common-issues-return-7819-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49962 invoked from network); 29 Apr 2010 11:02:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 11:02:27 -0000
Received: (qmail 16717 invoked by uid 500); 29 Apr 2010 11:02:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16598 invoked by uid 500); 29 Apr 2010 11:02:26 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16590 invoked by uid 99); 29 Apr 2010 11:02:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 11:02:25 +0000
X-ASF-Spam-Status: No, hits=-1365.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 11:02:24 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TB24VT027821
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 11:02:04 GMT
Message-ID: <23276038.3221272538924298.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 07:02:04 -0400 (EDT)
From: "Marc Guillemot (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6725) Evaluate HtmlUnit for unit and
 regression testing webpages
In-Reply-To: <9084581.651272255992465.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862156#action_12862156 ] 

Marc Guillemot commented on HADOOP-6725:
----------------------------------------

@Konstantin: this is not JSPUnit but JSFUnit. Do you use JSP or JSF?

What kind of failures did you have in the past that should not occur again when tested (for instance with HtmlUnit)?

> Evaluate HtmlUnit for unit and regression testing webpages
> ----------------------------------------------------------
>
>                 Key: HADOOP-6725
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6725
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>            Reporter: Jakob Homan
>            Priority: Minor
>
> HtmlUnit (http://htmlunit.sourceforge.net/) looks like it may be a good tool to help unit testing and evaluating our various webpages throughout the project.  Currently this is done only occasionally in the code (usually falls to being a manual test during release cycles), and when it is done, usually the code to parse the webpage, etc. is re-written each time.  The framework is Apache licensed, so including it won't be an issue.  If it's found to be useful, new JIRAs for HDFS and MR should be opened.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7820-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 14:11:20 2010
Return-Path: <common-issues-return-7820-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16087 invoked from network); 29 Apr 2010 14:11:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 14:11:20 -0000
Received: (qmail 26694 invoked by uid 500); 29 Apr 2010 14:11:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26672 invoked by uid 500); 29 Apr 2010 14:11:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26644 invoked by uid 99); 29 Apr 2010 14:11:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 14:11:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 14:11:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TEAtKp020372
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 14:10:55 GMT
Message-ID: <12960288.5831272550255386.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 10:10:55 -0400 (EDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6725) Evaluate HtmlUnit for unit and
 regression testing webpages
In-Reply-To: <9084581.651272255992465.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862205#action_12862205 ] 

Steve Loughran commented on HADOOP-6725:
----------------------------------------

right now we have the situation in which pretty much none of the web UI is tested. API, yes, HTTP, no. There is a cute test that fetches some XML from a servlet and hands off the work to Xerces, but otherwise, nothing. This is unfortunate, as HADOOP-6461 shows -it is not clear that on SVN_TRUNK, the webapps are working.

downstream, some people may be checking the web. My own deployment code takes a list of pages and runs through them, which is how I know about HADOOP-6461. 

What I would like to see test-wise is
# Something to run through all the standard HTML, JSP pages, artwork, check for their presence.
# Maybe: grab dependencies (stylesheets &c)
# Fetch the logs

The more advanced stuff is to run through some real operations: 

# create files via the client API, check you you can browse them 
# submit a job on the command line, and track it via the web UI 

Security: 
# Make unauthenticated calls and expect rejection.
# Have something log <script> text, fetch it from the logs, check the angle-brackets get filtered
# Check cookies are marked httponly once that gets adopted.
# anything else we can think of.

> Evaluate HtmlUnit for unit and regression testing webpages
> ----------------------------------------------------------
>
>                 Key: HADOOP-6725
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6725
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>            Reporter: Jakob Homan
>            Priority: Minor
>
> HtmlUnit (http://htmlunit.sourceforge.net/) looks like it may be a good tool to help unit testing and evaluating our various webpages throughout the project.  Currently this is done only occasionally in the code (usually falls to being a manual test during release cycles), and when it is done, usually the code to parse the webpage, etc. is re-written each time.  The framework is Apache licensed, so including it won't be an issue.  If it's found to be useful, new JIRAs for HDFS and MR should be opened.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7821-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 16:30:20 2010
Return-Path: <common-issues-return-7821-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74775 invoked from network); 29 Apr 2010 16:30:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 16:30:16 -0000
Received: (qmail 64349 invoked by uid 500); 29 Apr 2010 16:30:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64296 invoked by uid 500); 29 Apr 2010 16:30:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64288 invoked by uid 99); 29 Apr 2010 16:30:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 16:30:16 +0000
X-ASF-Spam-Status: No, hits=-1366.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 16:30:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TGTsOA006780
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 16:29:54 GMT
Message-ID: <24272228.8031272558594323.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 12:29:54 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6734) Jets3tNativeFileSystemStore wrongly
 calls S3Service.createBucket during initialisation
In-Reply-To: <30509879.3061272538794368.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862249#action_12862249 ] 

Tom White commented on HADOOP-6734:
-----------------------------------

HADOOP-4422 removed bucket creation from S3 filesystems. Does this fix the problem for you?

> Jets3tNativeFileSystemStore wrongly calls S3Service.createBucket during initialisation
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6734
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6734
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 0.18.3
>         Environment: S3 Europe
>            Reporter: Joni Niemi
>
> Reason: If a bucket is created with CreateBucketConfiguration specified, s3service.createBucket will fail. 
> Symptoms: If a bucket has CreateBucketConfiguration, Jets3tNativeFileSystemStore will fail with BucketAlreadyOwnedByYou Error.
> A detailed descrioption from a blog (http://john.keyes.ie/boto-create_bucket-bucketalreadyownedbyyou-error/)
> {quote}This evening I encountered a problem with it though. When the bucket did not exist, the method behaved as expected. When the bucket did exist though I received the following error response:
> {code}<?xml version="1.0" encoding="UTF-8"?>
> <Error>
>     <Code>BucketAlreadyOwnedByYou</Code>
>     <Message>
>         Your previous request to create the named 
>         bucket succeeded and you already own it.
>     </Message>
>     ...
> </Error>{code}
> This problem only manifests itself with buckets that are hosted in the EU. If the bucket is created in the US then the create_bucket method behaves as described.
>     For buckets created with a <CreateBucketConfiguration>, you will receive an error if you attempt to recreate the same bucket.
> To create a bucket in the EU, the bucket is created with a CreateBucketConfiguration specified. I now use the following code to avoid the problem and it works for both US and EU buckets.
> {code}
> def get_bucket():
>     try:
>         bucket = conn.get_bucket('xxx', validate=True)
>     except S3ResponseError, e:
>         if e.code == "NoSuchBucket":
>             bucket = conn.create_bucket('xxx', location='EU')
>         else:
>             raise e
>     return bucket
> {code}
> {quote}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7822-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 16:58:22 2010
Return-Path: <common-issues-return-7822-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90006 invoked from network); 29 Apr 2010 16:58:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 16:58:21 -0000
Received: (qmail 12309 invoked by uid 500); 29 Apr 2010 16:58:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12265 invoked by uid 500); 29 Apr 2010 16:58:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12257 invoked by uid 99); 29 Apr 2010 16:58:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 16:58:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 16:58:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TGvvWP007191
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 16:57:57 GMT
Message-ID: <6274237.8301272560277105.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 12:57:57 -0400 (EDT)
From: "Neil Bliss (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6704) add support for Parascale filesystem
In-Reply-To: <6991209.115951271258268758.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Neil Bliss updated HADOOP-6704:
-------------------------------

    Attachment: HADOOP-6704.0.20.2.patch
                HADOOP-6704.patch

Patches to -0.20.2 and -trunk adding support for Parascale storage as the DFS.

> add support for Parascale filesystem
> ------------------------------------
>
>                 Key: HADOOP-6704
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6704
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.20.2
>            Reporter: Neil Bliss
>         Attachments: HADOOP-6704.0.20.2.patch, HADOOP-6704.patch
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Parascale has developed an org.apache.hadoop.fs implementation that allows users to use Hadoop on Parascale storage clusters.  We'd like to contribute this work to the community. Should this be placed under contrib, or integrated into the org.apache.hadoop.fs space?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7823-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 18:10:17 2010
Return-Path: <common-issues-return-7823-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33442 invoked from network); 29 Apr 2010 18:10:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 18:10:17 -0000
Received: (qmail 21386 invoked by uid 500); 29 Apr 2010 18:10:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21361 invoked by uid 500); 29 Apr 2010 18:10:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21353 invoked by uid 99); 29 Apr 2010 18:10:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 18:10:17 +0000
X-ASF-Spam-Status: No, hits=-1367.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 18:10:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TI9uux016089
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 18:09:56 GMT
Message-ID: <5619617.9951272564596045.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 14:09:56 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6709) Re-instate deprecated FileSystem
 methods that were removed after 0.20
In-Reply-To: <11903376.154751271374849451.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862297#action_12862297 ] 

Eli Collins commented on HADOOP-6709:
-------------------------------------

+1   Looks good to me.   Nit: please put @Deprecated for getBlockSize and getLength on their own line to be consistent with the other methods.

> Re-instate deprecated FileSystem methods that were removed after 0.20
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6709
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6709
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6709.patch
>
>
> To make the FileSystem API in the 0.21 release (which should be treated as a minor release) compatible with 0.20 the deprecated methods removed in HADOOP-4779 need to be re-instated. They should still be marked as deprecated, however.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7824-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 18:12:22 2010
Return-Path: <common-issues-return-7824-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34648 invoked from network); 29 Apr 2010 18:12:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 18:12:21 -0000
Received: (qmail 26662 invoked by uid 500); 29 Apr 2010 18:12:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26623 invoked by uid 500); 29 Apr 2010 18:12:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26615 invoked by uid 99); 29 Apr 2010 18:12:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 18:12:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 18:12:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TIBwkZ016219
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 18:11:58 GMT
Message-ID: <1446587.10381272564718187.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 14:11:58 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6698) Revert the io.serialization package
 to 0.20.2's api
In-Reply-To: <21261745.7641270969841016.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6698:
------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

I've just committed this (https://issues.apache.org/jira/secure/attachment/12442620/HADOOP-6698.patch).

> Revert the io.serialization package to 0.20.2's api
> ---------------------------------------------------
>
>                 Key: HADOOP-6698
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6698
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Owen O'Malley
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6698.patch, HADOOP-6698.patch, HADOOP-6698.patch
>
>
> I have a lot of concern about the usability of the new generic serialization framework. Toward that end, I've filed a jira for improving it. There is resistance to pushing a new API into 0.21 at the last moment, so we should back out the changes rather than introducing a new api in 0.21 and deprecating it in 0.22.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7825-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 18:18:19 2010
Return-Path: <common-issues-return-7825-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38351 invoked from network); 29 Apr 2010 18:18:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 18:18:19 -0000
Received: (qmail 42495 invoked by uid 500); 29 Apr 2010 18:18:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42443 invoked by uid 500); 29 Apr 2010 18:18:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42313 invoked by uid 99); 29 Apr 2010 18:18:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 18:18:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 18:18:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TIHt4L016374
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 18:17:55 GMT
Message-ID: <4538291.10821272565075079.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 14:17:55 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6733) Create a test for FileSystem API
 compatibility between releases
In-Reply-To: <2563190.15351272511968608.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862310#action_12862310 ] 

Konstantin Boudnik commented on HADOOP-6733:
--------------------------------------------

I believe the effort needs to go beyond FileSystem API.

> Create a test for FileSystem API compatibility between releases
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6733
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6733
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs
>            Reporter: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>
> We should have an automated test for checking that programs written against an old version of the FileSystem API still run with a newer version. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7826-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 18:22:19 2010
Return-Path: <common-issues-return-7826-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41024 invoked from network); 29 Apr 2010 18:22:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 18:22:19 -0000
Received: (qmail 55295 invoked by uid 500); 29 Apr 2010 18:22:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55266 invoked by uid 500); 29 Apr 2010 18:22:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55258 invoked by uid 99); 29 Apr 2010 18:22:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 18:22:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 18:22:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TILtXc016615
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 18:21:55 GMT
Message-ID: <6461171.11161272565315073.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 14:21:55 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6733) Create a test for FileSystem API
 compatibility between releases
In-Reply-To: <2563190.15351272511968608.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862314#action_12862314 ] 

Tom White commented on HADOOP-6733:
-----------------------------------

I totally agree, but we need to start somewhere. My hope is to have some basic sanity checks for compatibility which we can release 0.21 with. Over time I would hope that the API tests are built out to cover more of the codebase.

> Create a test for FileSystem API compatibility between releases
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6733
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6733
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs
>            Reporter: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>
> We should have an automated test for checking that programs written against an old version of the FileSystem API still run with a newer version. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7827-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 18:26:18 2010
Return-Path: <common-issues-return-7827-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49277 invoked from network); 29 Apr 2010 18:26:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 18:26:18 -0000
Received: (qmail 61985 invoked by uid 500); 29 Apr 2010 18:26:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61960 invoked by uid 500); 29 Apr 2010 18:26:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61952 invoked by uid 99); 29 Apr 2010 18:26:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 18:26:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 18:26:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TIPrt9023264
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 18:25:54 GMT
Message-ID: <11566000.11451272565553980.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 14:25:53 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6731) Octal umask with less than 3 digits
 is not handled
In-Reply-To: <2800020.9731272493431396.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862321#action_12862321 ] 

Suresh Srinivas commented on HADOOP-6731:
-----------------------------------------

Currently UmaskParser does not parse sticky bit. So do you see any concern in allowing less than 3 digit umask?

> Octal umask with less than 3 digits is not handled
> --------------------------------------------------
>
>                 Key: HADOOP-6731
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6731
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6731.patch
>
>
> Umasks with less than three digits should be accepted as valid umask. Umask with single digit, say 7, should be handled as 007. Similarly umask with two digits, say 77, should be handled as 077.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7828-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 18:33:15 2010
Return-Path: <common-issues-return-7828-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54535 invoked from network); 29 Apr 2010 18:33:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 18:33:15 -0000
Received: (qmail 72078 invoked by uid 500); 29 Apr 2010 18:33:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72040 invoked by uid 500); 29 Apr 2010 18:33:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72032 invoked by uid 99); 29 Apr 2010 18:33:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 18:33:14 +0000
X-ASF-Spam-Status: No, hits=-1367.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 18:33:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TIWrW0007042
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 18:32:53 GMT
Message-ID: <3098461.12511272565973557.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 14:32:53 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6731) Octal umask with less than 3 digits
 is not handled
In-Reply-To: <2800020.9731272493431396.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862329#action_12862329 ] 

Suresh Srinivas commented on HADOOP-6731:
-----------------------------------------

On HDFS we do not support set-user-ID-on-execution-bit. On linux all the following is support for chmod:
7, 77, 777, 1777. Should we make chmod parser also consistent with this? I feel we should either change both creation umask and chmod umask or leave both of them as it is. Any thoughts?

> Octal umask with less than 3 digits is not handled
> --------------------------------------------------
>
>                 Key: HADOOP-6731
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6731
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6731.patch
>
>
> Umasks with less than three digits should be accepted as valid umask. Umask with single digit, say 7, should be handled as 007. Similarly umask with two digits, say 77, should be handled as 077.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7829-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 18:37:18 2010
Return-Path: <common-issues-return-7829-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56753 invoked from network); 29 Apr 2010 18:37:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 18:37:17 -0000
Received: (qmail 80529 invoked by uid 500); 29 Apr 2010 18:37:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80500 invoked by uid 500); 29 Apr 2010 18:37:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80492 invoked by uid 99); 29 Apr 2010 18:37:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 18:37:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 18:37:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TIarTk005423
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 18:36:54 GMT
Message-ID: <11810338.12551272566213752.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 14:36:53 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6731) Octal umask with less than 3 digits
 is not handled
In-Reply-To: <2800020.9731272493431396.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862331#action_12862331 ] 

Jakob Homan commented on HADOOP-6731:
-------------------------------------

Yep, we should fix it for both umask and permissions to be consistent. If so, this is a good change to make. 

> Octal umask with less than 3 digits is not handled
> --------------------------------------------------
>
>                 Key: HADOOP-6731
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6731
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6731.patch
>
>
> Umasks with less than three digits should be accepted as valid umask. Umask with single digit, say 7, should be handled as 007. Similarly umask with two digits, say 77, should be handled as 077.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7830-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 18:39:19 2010
Return-Path: <common-issues-return-7830-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57997 invoked from network); 29 Apr 2010 18:39:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 18:39:18 -0000
Received: (qmail 81446 invoked by uid 500); 29 Apr 2010 18:39:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81422 invoked by uid 500); 29 Apr 2010 18:39:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81414 invoked by uid 99); 29 Apr 2010 18:39:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 18:39:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 18:39:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TIctel010902
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 18:38:55 GMT
Message-ID: <15901641.12621272566335478.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 14:38:55 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6710) Symbolic umask for file creation is
 not consistent with posix
In-Reply-To: <8947165.155011271375809341.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862333#action_12862333 ] 

Jakob Homan commented on HADOOP-6710:
-------------------------------------

+1 on 20 version.

> Symbolic umask for file creation is not consistent with posix
> -------------------------------------------------------------
>
>                 Key: HADOOP-6710
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6710
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>         Attachments: hadoop-6710.patch, hadoop-6710.rel20.patch
>
>
> Currently both octal and symbolic umask are used to reset the file creation mode bits. This is not consistent with the behavior defined in posix. Making it consistent would avoid confusion to the HDFS users.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7831-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 18:39:19 2010
Return-Path: <common-issues-return-7831-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58053 invoked from network); 29 Apr 2010 18:39:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 18:39:19 -0000
Received: (qmail 81589 invoked by uid 500); 29 Apr 2010 18:39:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81558 invoked by uid 500); 29 Apr 2010 18:39:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81550 invoked by uid 99); 29 Apr 2010 18:39:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 18:39:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 18:39:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TIcuOc010911
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 18:38:56 GMT
Message-ID: <3309148.12651272566336246.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 14:38:56 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6693) Add metrics to track kerberos login
 activity
In-Reply-To: <1138587062.22291270763196714.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862334#action_12862334 ] 

Owen O'Malley commented on HADOOP-6693:
---------------------------------------

+1

> Add metrics to track kerberos login activity
> --------------------------------------------
>
>                 Key: HADOOP-6693
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6693
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6693.1.patch, HADOOP-6693.patch
>
>
> Need metrics to track kerberos login activity such as login rate and the time taken for login.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7832-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 18:57:21 2010
Return-Path: <common-issues-return-7832-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67262 invoked from network); 29 Apr 2010 18:57:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 18:57:18 -0000
Received: (qmail 1022 invoked by uid 500); 29 Apr 2010 18:57:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 985 invoked by uid 500); 29 Apr 2010 18:57:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 977 invoked by uid 99); 29 Apr 2010 18:57:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 18:57:18 +0000
X-ASF-Spam-Status: No, hits=-1367.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 18:57:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TIuu4Y011297
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 18:56:57 GMT
Message-ID: <27249672.13261272567416156.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 14:56:56 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6668:
------------------------------

    Attachment: HADOOP-6668.patch

The old patch went stale - here's a new one.

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7833-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 19:31:19 2010
Return-Path: <common-issues-return-7833-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85071 invoked from network); 29 Apr 2010 19:31:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 19:31:18 -0000
Received: (qmail 33754 invoked by uid 500); 29 Apr 2010 19:31:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33715 invoked by uid 500); 29 Apr 2010 19:31:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33707 invoked by uid 99); 29 Apr 2010 19:31:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 19:31:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 19:31:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TJUsUD011649
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 19:30:55 GMT
Message-ID: <24861647.14461272569454475.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 15:30:54 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6730) FileContext#copy needs unit tests
In-Reply-To: <8913331.3881272480709335.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari reassigned HADOOP-6730:
------------------------------------

    Assignee: Ravi Phulari

> FileContext#copy needs unit tests
> ---------------------------------
>
>                 Key: HADOOP-6730
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6730
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Ravi Phulari
>             Fix For: 0.22.0
>
>
> I modifed FileContext#checkDest in HADOOP-6678 and noticed it's only caller FileContext#copy has no callers itself. It needs unit tests.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7834-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 19:35:21 2010
Return-Path: <common-issues-return-7834-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87207 invoked from network); 29 Apr 2010 19:35:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 19:35:21 -0000
Received: (qmail 37302 invoked by uid 500); 29 Apr 2010 19:35:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37271 invoked by uid 500); 29 Apr 2010 19:35:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37263 invoked by uid 99); 29 Apr 2010 19:35:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 19:35:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 19:35:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TJYvlv005366
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 19:34:57 GMT
Message-ID: <32774354.14531272569697134.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 15:34:57 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6730) Bug in FileContext#copy
In-Reply-To: <8913331.3881272480709335.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6730:
---------------------------------

        Summary: Bug in FileContext#copy  (was: FileContext#copy needs unit tests)
    Description: 
Thanks to Eli, He noticed that there is no test for FileContext#Copy operation. 

On further investigation with the help of Sanjay we found that there is bug in FileContext#checkDest.

{noformat}
  FileStatus dstFs = getFileStatus(dst);
    try {
      if (dstFs.isDir()) {
        if (null == srcNa
{noformat}


 *FileStatus dstFs = getFileStatus(dst);* should be in try...catch block.

{noformat}
    try {
       FileStatus dstFs = getFileStatus(dst);
       if (dstFs.isDir()) {
          if (null == srcNa
{noformat}




  was:I modifed FileContext#checkDest in HADOOP-6678 and noticed it's only caller FileContext#copy has no callers itself. It needs unit tests.


Adding information about bug in FileContext#checkDest. 

Will upload patch soon with bug fix and Tests for copy.

> Bug in FileContext#copy
> -----------------------
>
>                 Key: HADOOP-6730
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6730
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Ravi Phulari
>             Fix For: 0.22.0
>
>
> Thanks to Eli, He noticed that there is no test for FileContext#Copy operation. 
> On further investigation with the help of Sanjay we found that there is bug in FileContext#checkDest.
> {noformat}
>   FileStatus dstFs = getFileStatus(dst);
>     try {
>       if (dstFs.isDir()) {
>         if (null == srcNa
> {noformat}
>  *FileStatus dstFs = getFileStatus(dst);* should be in try...catch block.
> {noformat}
>     try {
>        FileStatus dstFs = getFileStatus(dst);
>        if (dstFs.isDir()) {
>           if (null == srcNa
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7835-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 19:45:20 2010
Return-Path: <common-issues-return-7835-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91961 invoked from network); 29 Apr 2010 19:45:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 19:45:20 -0000
Received: (qmail 47722 invoked by uid 500); 29 Apr 2010 19:45:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47683 invoked by uid 500); 29 Apr 2010 19:45:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47675 invoked by uid 99); 29 Apr 2010 19:45:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 19:45:20 +0000
X-ASF-Spam-Status: No, hits=-1368.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 19:45:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TJix9p002863
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 19:44:59 GMT
Message-ID: <27000115.14891272570299193.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 15:44:59 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6668:
------------------------------

    Status: Open  (was: Patch Available)

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7836-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 19:49:19 2010
Return-Path: <common-issues-return-7836-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94303 invoked from network); 29 Apr 2010 19:49:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 19:49:19 -0000
Received: (qmail 53946 invoked by uid 500); 29 Apr 2010 19:49:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53877 invoked by uid 500); 29 Apr 2010 19:49:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53869 invoked by uid 99); 29 Apr 2010 19:49:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 19:49:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 19:49:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TJmtfT007860
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 19:48:55 GMT
Message-ID: <18440680.14941272570535203.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 15:48:55 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6730) Bug in FileContext#copy
In-Reply-To: <8913331.3881272480709335.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862371#action_12862371 ] 

Eli Collins commented on HADOOP-6730:
-------------------------------------

The call to getFileStatus is outside the try block to make sure FileNotFoundException is thrown per the interface no?

> Bug in FileContext#copy
> -----------------------
>
>                 Key: HADOOP-6730
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6730
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Ravi Phulari
>             Fix For: 0.22.0
>
>
> Thanks to Eli, He noticed that there is no test for FileContext#Copy operation. 
> On further investigation with the help of Sanjay we found that there is bug in FileContext#checkDest.
> {noformat}
>   FileStatus dstFs = getFileStatus(dst);
>     try {
>       if (dstFs.isDir()) {
>         if (null == srcNa
> {noformat}
>  *FileStatus dstFs = getFileStatus(dst);* should be in try...catch block.
> {noformat}
>     try {
>        FileStatus dstFs = getFileStatus(dst);
>        if (dstFs.isDir()) {
>           if (null == srcNa
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7837-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 19:51:28 2010
Return-Path: <common-issues-return-7837-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96959 invoked from network); 29 Apr 2010 19:51:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 19:51:21 -0000
Received: (qmail 58750 invoked by uid 500); 29 Apr 2010 19:51:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58719 invoked by uid 500); 29 Apr 2010 19:51:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58707 invoked by uid 99); 29 Apr 2010 19:51:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 19:51:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 19:51:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TJovcU010991
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 19:50:57 GMT
Message-ID: <8856626.15141272570657222.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 15:50:57 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6668:
------------------------------

    Status: Patch Available  (was: Open)

Here's the [javadoc|http://people.apache.org/~tomwhite/HADOOP-6668/docs/api/] and [JDiff with 0.20|http://people.apache.org/~tomwhite/HADOOP-6668/docs/jdiff/changes.html] generated with this patch applied.

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7838-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 19:58:18 2010
Return-Path: <common-issues-return-7838-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1637 invoked from network); 29 Apr 2010 19:58:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 19:58:18 -0000
Received: (qmail 64124 invoked by uid 500); 29 Apr 2010 19:58:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64045 invoked by uid 500); 29 Apr 2010 19:58:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64037 invoked by uid 99); 29 Apr 2010 19:58:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 19:58:17 +0000
X-ASF-Spam-Status: No, hits=-1368.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 19:58:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TJvuek011135
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 19:57:56 GMT
Message-ID: <6005215.15251272571076373.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 15:57:56 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6735) Remove falut injection compilation
 from default ant compilation and ant test-core.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Remove falut injection compilation from default ant compilation and ant test-core.
----------------------------------------------------------------------------------

                 Key: HADOOP-6735
                 URL: https://issues.apache.org/jira/browse/HADOOP-6735
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Ravi Phulari


Compiling fault-injection code while running ant tests increases test time by considerable amount of time.  It would be great if by default fi code is not compiled every time ant tests are run.
We should add flag to run fault injection code on demand.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7839-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 19:58:19 2010
Return-Path: <common-issues-return-7839-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1706 invoked from network); 29 Apr 2010 19:58:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 19:58:19 -0000
Received: (qmail 64222 invoked by uid 500); 29 Apr 2010 19:58:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64182 invoked by uid 500); 29 Apr 2010 19:58:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64174 invoked by uid 99); 29 Apr 2010 19:58:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 19:58:19 +0000
X-ASF-Spam-Status: No, hits=-1368.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 19:58:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TJvvfD011153
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 19:57:57 GMT
Message-ID: <3263960.15311272571077962.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 15:57:57 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6709) Re-instate deprecated FileSystem
 methods that were removed after 0.20
In-Reply-To: <11903376.154751271374849451.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6709:
------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

I've just committed this. (I fixed the nit.)

> Re-instate deprecated FileSystem methods that were removed after 0.20
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6709
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6709
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6709.patch
>
>
> To make the FileSystem API in the 0.21 release (which should be treated as a minor release) compatible with 0.20 the deprecated methods removed in HADOOP-4779 need to be re-instated. They should still be marked as deprecated, however.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7840-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 19:58:23 2010
Return-Path: <common-issues-return-7840-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1926 invoked from network); 29 Apr 2010 19:58:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 19:58:22 -0000
Received: (qmail 65613 invoked by uid 500); 29 Apr 2010 19:58:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65577 invoked by uid 500); 29 Apr 2010 19:58:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65569 invoked by uid 99); 29 Apr 2010 19:58:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 19:58:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 19:58:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TJvxBG011170
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 19:57:59 GMT
Message-ID: <11964248.15361272571079287.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 15:57:59 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6735) Remove fault injection compilation
 from default ant compilation and ant test-core.
In-Reply-To: <6005215.15251272571076373.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6735:
---------------------------------

    Summary: Remove fault injection compilation from default ant compilation and ant test-core.  (was: Remove falut injection compilation from default ant compilation and ant test-core.)

> Remove fault injection compilation from default ant compilation and ant test-core.
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6735
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6735
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Ravi Phulari
>
> Compiling fault-injection code while running ant tests increases test time by considerable amount of time.  It would be great if by default fi code is not compiled every time ant tests are run.
> We should add flag to run fault injection code on demand.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7841-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 20:06:18 2010
Return-Path: <common-issues-return-7841-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8217 invoked from network); 29 Apr 2010 20:06:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 20:06:17 -0000
Received: (qmail 79462 invoked by uid 500); 29 Apr 2010 20:06:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79396 invoked by uid 500); 29 Apr 2010 20:06:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79388 invoked by uid 99); 29 Apr 2010 20:06:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 20:06:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 20:06:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TK5rtN011389
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 20:05:54 GMT
Message-ID: <22382184.15501272571553733.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 16:05:53 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6735) Remove fault injection compilation
 from default ant compilation and ant test-core.
In-Reply-To: <6005215.15251272571076373.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862381#action_12862381 ] 

Konstantin Boudnik commented on HADOOP-6735:
--------------------------------------------

Fault injection tests are valuable part of the hadoop testing. E.g. it allowed to find many problems in HDFS' append feature. Yes, the compilation time is slightly increases but compare to the runtime of all the tests it barely adds a percent or less.

Having an optional flag to switch some testing on/off is almost a guarantee that this testing will never be run by anyone.

Besides, FI tests are only part of 'test-core' target. You can avoid running it by starting 'ant run-test-core' instead of 'ant test'

> Remove fault injection compilation from default ant compilation and ant test-core.
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6735
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6735
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Ravi Phulari
>
> Compiling fault-injection code while running ant tests increases test time by considerable amount of time.  It would be great if by default fi code is not compiled every time ant tests are run.
> We should add flag to run fault injection code on demand.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7842-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 20:08:18 2010
Return-Path: <common-issues-return-7842-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9405 invoked from network); 29 Apr 2010 20:08:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 20:08:18 -0000
Received: (qmail 85133 invoked by uid 500); 29 Apr 2010 20:08:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85104 invoked by uid 500); 29 Apr 2010 20:08:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85096 invoked by uid 99); 29 Apr 2010 20:08:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 20:08:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 20:08:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TK7rKd011435
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 20:07:54 GMT
Message-ID: <28688398.15531272571673849.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 16:07:53 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6735) Remove fault injection compilation
 from default ant compilation and ant test-core.
In-Reply-To: <6005215.15251272571076373.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari resolved HADOOP-6735.
----------------------------------

    Resolution: Not A Problem

Apparently there is ant run-test-core/hdfs/mapred  and ant run-test-unit options to achieve this,

{noformat}
 <target name="test-core" description="Run hdfs unit tests">
    <delete file="${test.build.dir}/testsfailed"/>
    <property name="continueOnFailure" value="true"/>
    <antcall target="run-test-hdfs"/>
    <antcall target="run-test-unit"/>
    <antcall target="run-test-hdfs-fault-inject"/>
    <available file="${test.build.dir}/testsfailed" property="testsfailed"/>
    <fail if="testsfailed">Tests failed!</fail>
  </target>
{noformat}

> Remove fault injection compilation from default ant compilation and ant test-core.
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6735
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6735
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Ravi Phulari
>
> Compiling fault-injection code while running ant tests increases test time by considerable amount of time.  It would be great if by default fi code is not compiled every time ant tests are run.
> We should add flag to run fault injection code on demand.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7843-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 20:10:20 2010
Return-Path: <common-issues-return-7843-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10539 invoked from network); 29 Apr 2010 20:10:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 20:10:20 -0000
Received: (qmail 88315 invoked by uid 500); 29 Apr 2010 20:10:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88258 invoked by uid 500); 29 Apr 2010 20:10:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88250 invoked by uid 99); 29 Apr 2010 20:10:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 20:10:19 +0000
X-ASF-Spam-Status: No, hits=-1368.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 20:10:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TK9wt2011462
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 20:09:58 GMT
Message-ID: <21530262.15591272571798466.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 16:09:58 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6730) Bug in FileContext#copy
In-Reply-To: <8913331.3881272480709335.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6730:
---------------------------------

    Attachment: HADOOP-6730.patch

Attaching patch for review. 

> Bug in FileContext#copy
> -----------------------
>
>                 Key: HADOOP-6730
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6730
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Ravi Phulari
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6730.patch
>
>
> Thanks to Eli, He noticed that there is no test for FileContext#Copy operation. 
> On further investigation with the help of Sanjay we found that there is bug in FileContext#checkDest.
> {noformat}
>   FileStatus dstFs = getFileStatus(dst);
>     try {
>       if (dstFs.isDir()) {
>         if (null == srcNa
> {noformat}
>  *FileStatus dstFs = getFileStatus(dst);* should be in try...catch block.
> {noformat}
>     try {
>        FileStatus dstFs = getFileStatus(dst);
>        if (dstFs.isDir()) {
>           if (null == srcNa
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7844-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 20:31:16 2010
Return-Path: <common-issues-return-7844-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20444 invoked from network); 29 Apr 2010 20:31:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 20:31:16 -0000
Received: (qmail 8071 invoked by uid 500); 29 Apr 2010 20:31:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7994 invoked by uid 500); 29 Apr 2010 20:31:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7904 invoked by uid 99); 29 Apr 2010 20:31:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 20:31:15 +0000
X-ASF-Spam-Status: No, hits=-1368.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 20:31:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TKUsAV011822
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 20:30:54 GMT
Message-ID: <7537676.16341272573054914.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 16:30:54 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6736) Add tests for checking exceptions
 thrown from FileContext for different file system implementation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Add tests for checking exceptions thrown from FileContext for different file system implementation
--------------------------------------------------------------------------------------------------

                 Key: HADOOP-6736
                 URL: https://issues.apache.org/jira/browse/HADOOP-6736
             Project: Hadoop Common
          Issue Type: Test
            Reporter: Suresh Srinivas
            Assignee: Suresh Srinivas


Need a test to check if the right exceptions are thrown by the FileContext for different file system implementations.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7845-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 20:53:19 2010
Return-Path: <common-issues-return-7845-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35426 invoked from network); 29 Apr 2010 20:53:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 20:53:19 -0000
Received: (qmail 38201 invoked by uid 500); 29 Apr 2010 20:53:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38173 invoked by uid 500); 29 Apr 2010 20:53:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38165 invoked by uid 99); 29 Apr 2010 20:53:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 20:53:18 +0000
X-ASF-Spam-Status: No, hits=-1368.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 20:53:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TKqufr012349
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 20:52:56 GMT
Message-ID: <24108585.17671272574376957.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 16:52:56 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6730) Bug in FileContext#copy
In-Reply-To: <8913331.3881272480709335.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862404#action_12862404 ] 

Eli Collins commented on HADOOP-6730:
-------------------------------------

Hey Ravi,  per the previous comment I left the getFileStatus outside the try block so the FileNotFoundException is now swallowed silently and is thrown up to copy (whose API wants FileNotFoundThrown). This deserves a comment.

The patch just has util classes - perhaps you forgot to add the test class in your diff?

> Bug in FileContext#copy
> -----------------------
>
>                 Key: HADOOP-6730
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6730
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Ravi Phulari
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6730.patch
>
>
> Thanks to Eli, He noticed that there is no test for FileContext#Copy operation. 
> On further investigation with the help of Sanjay we found that there is bug in FileContext#checkDest.
> {noformat}
>   FileStatus dstFs = getFileStatus(dst);
>     try {
>       if (dstFs.isDir()) {
>         if (null == srcNa
> {noformat}
>  *FileStatus dstFs = getFileStatus(dst);* should be in try...catch block.
> {noformat}
>     try {
>        FileStatus dstFs = getFileStatus(dst);
>        if (dstFs.isDir()) {
>           if (null == srcNa
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7846-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 21:27:18 2010
Return-Path: <common-issues-return-7846-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53287 invoked from network); 29 Apr 2010 21:27:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 21:27:18 -0000
Received: (qmail 78218 invoked by uid 500); 29 Apr 2010 21:27:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78190 invoked by uid 500); 29 Apr 2010 21:27:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78182 invoked by uid 99); 29 Apr 2010 21:27:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 21:27:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 21:27:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TLQtxQ012978
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 21:26:55 GMT
Message-ID: <5271754.19101272576415024.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 17:26:55 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6730) Bug in FileContext#copy
In-Reply-To: <8913331.3881272480709335.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862423#action_12862423 ] 

Ravi Phulari commented on HADOOP-6730:
--------------------------------------


 FileContext.Util.Copy() operation was failing with file not found exception if destination file does not exits , this was happening due to the fact that  getFileStatus was out of try block.

bq. The patch just has util classes - perhaps you forgot to add the test class in your diff?

I have included test runner with  util base.  Please check that there are two classes added with this patch  - {{TestFcLocalFsUtil.java}} and {{FileContextUtilBase.java}}

> Bug in FileContext#copy
> -----------------------
>
>                 Key: HADOOP-6730
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6730
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Ravi Phulari
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6730.patch
>
>
> Thanks to Eli, He noticed that there is no test for FileContext#Copy operation. 
> On further investigation with the help of Sanjay we found that there is bug in FileContext#checkDest.
> {noformat}
>   FileStatus dstFs = getFileStatus(dst);
>     try {
>       if (dstFs.isDir()) {
>         if (null == srcNa
> {noformat}
>  *FileStatus dstFs = getFileStatus(dst);* should be in try...catch block.
> {noformat}
>     try {
>        FileStatus dstFs = getFileStatus(dst);
>        if (dstFs.isDir()) {
>           if (null == srcNa
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7847-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 21:35:19 2010
Return-Path: <common-issues-return-7847-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57099 invoked from network); 29 Apr 2010 21:35:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 21:35:19 -0000
Received: (qmail 87025 invoked by uid 500); 29 Apr 2010 21:35:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86990 invoked by uid 500); 29 Apr 2010 21:35:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86982 invoked by uid 99); 29 Apr 2010 21:35:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 21:35:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 21:35:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TLYtfL013032
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 21:34:55 GMT
Message-ID: <20667884.19221272576895000.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 17:34:54 -0400 (EDT)
From: "Zheng Shao (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6515) Make maximum number of http threads
 configurable
In-Reply-To: <1919319763.85831264619136665.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zheng Shao updated HADOOP-6515:
-------------------------------

           Status: Resolved  (was: Patch Available)
     Hadoop Flags: [Reviewed]
     Release Note: HADOOP-6515. Make maximum number of http threads configurable (Scott Chen via zshao)
    Fix Version/s: 0.22.0
       Resolution: Fixed

Committed. Thanks Scott!


> Make maximum number of http threads configurable
> ------------------------------------------------
>
>                 Key: HADOOP-6515
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6515
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6515-v2.patch, HADOOP-6515-v3.1.patch, HADOOP-6515-v3.patch, HADOOP-6515.patch
>
>
> We found that http server threads may use considerable amount of resource in NameNode and JobTracker.
> It would be good if the number of threads is allowed to be configured to a smaller number.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7848-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 21:47:38 2010
Return-Path: <common-issues-return-7848-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63662 invoked from network); 29 Apr 2010 21:47:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 21:47:38 -0000
Received: (qmail 97827 invoked by uid 500); 29 Apr 2010 21:47:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97801 invoked by uid 500); 29 Apr 2010 21:47:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97793 invoked by uid 99); 29 Apr 2010 21:47:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 21:47:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 21:47:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TLlEu1013176
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 21:47:14 GMT
Message-ID: <15204331.19581272577634257.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 17:47:14 -0400 (EDT)
From: "Carl Steinbach (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6737) 0.20.1 tarball on archive.apache.org
 has bogus md5 checksum file
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

0.20.1 tarball on archive.apache.org has bogus md5 checksum file
----------------------------------------------------------------

                 Key: HADOOP-6737
                 URL: https://issues.apache.org/jira/browse/HADOOP-6737
             Project: Hadoop Common
          Issue Type: Bug
          Components: build
    Affects Versions: 0.20.1
            Reporter: Carl Steinbach


The MD5 checksum file for the 0.20.1 tarball contains an MD5 checksum along with
various SHA checksums and an RMD160 checksum:

{code}
% curl http://archive.apache.org/dist/hadoop/core/hadoop-0.20.1/hadoop-0.20.1.tar.gz.md5
hadoop-0.20.1.tar.gz:    MD5 = 71 9E 16 9B 77 60 C1 68  44 1B 49 F4 05 85 5B 72
hadoop-0.20.1.tar.gz:   SHA1 = 712A EE9C 279F 1031 1F83  657B 2B82 7ACA 0374
                               6613
hadoop-0.20.1.tar.gz: RMD160 = 4331 4350 27E9 E16D 055C  F23F FFEF 1564 E206
                               B144
hadoop-0.20.1.tar.gz: SHA224 = A5E4CBE9 EBBE5FE1 2020F3F1 BFBC6D3C C77A8E9B
                               E6C6062C A6484BDB
hadoop-0.20.1.tar.gz: SHA256 = 35024227 C51AFA34 6A2F3722 A652C4FB 4FEBFF35
                               E73BB364 3172F986 7233E2A4
hadoop-0.20.1.tar.gz: SHA384 = 54B5CB24 4708765A D3B930D8 73DB8A01 839B78DA
                               78A4A8B5 20ED0A88 5759E60F 92D3A90E 9AFB690A
                               C1EDB71F 6FB357AA
hadoop-0.20.1.tar.gz: SHA512 = 4D9DDEC5 27190749 87304D88 C2F9546E 6D9E62B0
                               EF8330C2 2D70BC2D 52CAFDD1 CD639E0E 8853177B
                               4E4F21EE EBEF316C 4DF3561A 10995B1D 95ED9047
                               16AC3013
{code}

The convention that most tools expect (including Ivy) is that a file named
xyz.tar.gz.md5 was generated by running the following command:

% md5sum xyz.tar.gz > xyz.tar.gz.md5

All of the other Hadoop releases on archive.apache.org follow this convention, with
the exception of hadoop-0.20.1, e.g:

{code}
% curl http://archive.apache.org/dist/hadoop/core/hadoop-0.20.0/hadoop-0.20.0.tar.gz.md5
6c751617e6429f23ec9b7ea7a02a73c2  hadoop-0.20.0.tar.gz
{code}

Can someone with access to the Apache archive repository please fix the 0.20.1
md5 checksum file?

Thanks.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7849-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 21:47:39 2010
Return-Path: <common-issues-return-7849-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63683 invoked from network); 29 Apr 2010 21:47:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 21:47:38 -0000
Received: (qmail 97965 invoked by uid 500); 29 Apr 2010 21:47:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97937 invoked by uid 500); 29 Apr 2010 21:47:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97929 invoked by uid 99); 29 Apr 2010 21:47:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 21:47:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 21:47:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TLlFOu013188
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 21:47:15 GMT
Message-ID: <12576793.19621272577635037.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 17:47:15 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6730) Bug in FileContext#copy
In-Reply-To: <8913331.3881272480709335.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6730:
---------------------------------

        Fix Version/s: 0.21.0
    Affects Version/s: 0.21.0

> Bug in FileContext#copy
> -----------------------
>
>                 Key: HADOOP-6730
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6730
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Eli Collins
>            Assignee: Ravi Phulari
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6730.patch
>
>
> Thanks to Eli, He noticed that there is no test for FileContext#Copy operation. 
> On further investigation with the help of Sanjay we found that there is bug in FileContext#checkDest.
> {noformat}
>   FileStatus dstFs = getFileStatus(dst);
>     try {
>       if (dstFs.isDir()) {
>         if (null == srcNa
> {noformat}
>  *FileStatus dstFs = getFileStatus(dst);* should be in try...catch block.
> {noformat}
>     try {
>        FileStatus dstFs = getFileStatus(dst);
>        if (dstFs.isDir()) {
>           if (null == srcNa
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7850-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 21:56:16 2010
Return-Path: <common-issues-return-7850-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68163 invoked from network); 29 Apr 2010 21:56:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 21:56:16 -0000
Received: (qmail 14753 invoked by uid 500); 29 Apr 2010 21:56:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14724 invoked by uid 500); 29 Apr 2010 21:56:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14716 invoked by uid 99); 29 Apr 2010 21:56:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 21:56:16 +0000
X-ASF-Spam-Status: No, hits=-1369.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 21:56:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TLttRs013282
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 21:55:55 GMT
Message-ID: <22624241.19781272578155159.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 17:55:55 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6730) Bug in FileContext#copy
In-Reply-To: <8913331.3881272480709335.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862435#action_12862435 ] 

Eli Collins commented on HADOOP-6730:
-------------------------------------

+1 Looks good to me.

> Bug in FileContext#copy
> -----------------------
>
>                 Key: HADOOP-6730
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6730
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Eli Collins
>            Assignee: Ravi Phulari
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6730.patch
>
>
> Thanks to Eli, He noticed that there is no test for FileContext#Copy operation. 
> On further investigation with the help of Sanjay we found that there is bug in FileContext#checkDest.
> {noformat}
>   FileStatus dstFs = getFileStatus(dst);
>     try {
>       if (dstFs.isDir()) {
>         if (null == srcNa
> {noformat}
>  *FileStatus dstFs = getFileStatus(dst);* should be in try...catch block.
> {noformat}
>     try {
>        FileStatus dstFs = getFileStatus(dst);
>        if (dstFs.isDir()) {
>           if (null == srcNa
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7851-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 22:18:22 2010
Return-Path: <common-issues-return-7851-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83175 invoked from network); 29 Apr 2010 22:18:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 22:18:21 -0000
Received: (qmail 48052 invoked by uid 500); 29 Apr 2010 22:18:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48023 invoked by uid 500); 29 Apr 2010 22:18:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48015 invoked by uid 99); 29 Apr 2010 22:18:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 22:18:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 22:18:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TMHwJt013848
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 22:17:58 GMT
Message-ID: <18164101.21271272579478303.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 18:17:58 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6289) Add interface classification stable
 & scope to common
In-Reply-To: <683197199.1254170476073.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White resolved HADOOP-6289.
-------------------------------

    Resolution: Duplicate

Duplicate of HADOOP-6668.

> Add interface classification stable & scope to common
> -----------------------------------------------------
>
>                 Key: HADOOP-6289
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6289
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.21.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.21.0
>
>         Attachments: common.interface.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7852-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 22:22:18 2010
Return-Path: <common-issues-return-7852-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86296 invoked from network); 29 Apr 2010 22:22:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 22:22:18 -0000
Received: (qmail 53310 invoked by uid 500); 29 Apr 2010 22:22:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53277 invoked by uid 500); 29 Apr 2010 22:22:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53269 invoked by uid 99); 29 Apr 2010 22:22:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 22:22:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 22:22:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TMLsLf013907
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 22:21:54 GMT
Message-ID: <8323850.21381272579714603.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 18:21:54 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6630) hadoop-config.sh fails to get
 executed if hadoop wrapper scripts are in path
In-Reply-To: <1961052931.216791268355387410.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862448#action_12862448 ] 

Allen Wittenauer commented on HADOOP-6630:
------------------------------------------

Can we get this committed prior to the 0.21 freeze please? :)

Thank you. :)

> hadoop-config.sh fails to get executed if hadoop wrapper scripts are in path
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-6630
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6630
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: Allen Wittenauer
>         Attachments: HADOOP-6630.txt
>
>
> If the hadoop/bin commands are in the path, hadoop-config.sh doesn't executed until much later in the stack because $0 lacks a path.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7853-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 22:40:16 2010
Return-Path: <common-issues-return-7853-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95370 invoked from network); 29 Apr 2010 22:40:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 22:40:16 -0000
Received: (qmail 71440 invoked by uid 500); 29 Apr 2010 22:40:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71412 invoked by uid 500); 29 Apr 2010 22:40:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71404 invoked by uid 99); 29 Apr 2010 22:40:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 22:40:16 +0000
X-ASF-Spam-Status: No, hits=-1369.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 22:40:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TMdsid014055
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 22:39:54 GMT
Message-ID: <10959668.21701272580794613.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 18:39:54 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6585) Add FileStatus#isDirectory and isFile
In-Reply-To: <649415366.425221266809667869.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6585:
--------------------------------

        Fix Version/s: 0.21.0
    Affects Version/s: 0.20.3
                       0.21.0
                           (was: 0.22.0)
             Priority: Blocker  (was: Major)

Need this in 21 since it preserves the old behavior of FileStatus#isDir.

> Add FileStatus#isDirectory and isFile
> -------------------------------------
>
>                 Key: HADOOP-6585
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6585
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.20.3, 0.21.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6585-1.patch
>
>
> Per Sanjay's suggestion in HADOOP-6421 let's deprecate FileStatus#isDir() and add isDirectory() and isFile() to compliment isSymlink. Currently clients assume !isDir() implies a file, which is no longer true with symlinks. I'll file a separate jira to change the various uses of !isDir() to be isFile() or isFile() or isSymlink() as appropriate.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7854-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 22:44:20 2010
Return-Path: <common-issues-return-7854-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97916 invoked from network); 29 Apr 2010 22:44:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 22:44:20 -0000
Received: (qmail 77427 invoked by uid 500); 29 Apr 2010 22:44:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77401 invoked by uid 500); 29 Apr 2010 22:44:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77393 invoked by uid 99); 29 Apr 2010 22:44:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 22:44:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 22:44:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TMhu4X014126
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 22:43:56 GMT
Message-ID: <8130217.21931272581036594.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 18:43:56 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6727) Remove UnresolvedLinkException from
 public FileContext APIs
In-Reply-To: <27900682.46541272411814280.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6727:
--------------------------------

    Priority: Blocker  (was: Minor)

API change so should be in 21, also an easy change.

> Remove UnresolvedLinkException from public FileContext APIs
> -----------------------------------------------------------
>
>                 Key: HADOOP-6727
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6727
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6727-1.patch
>
>
> HADOOP-6537 added UnresolvedLinkException to the throws clause and java docs of FileContext public APIs. FileContext fully resolves symbolic links, UnresolvedLinkException exception is only used internally by FSLinkResolver. I'll attach a patch which updates the APIs and javadoc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7855-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 22:56:18 2010
Return-Path: <common-issues-return-7855-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3584 invoked from network); 29 Apr 2010 22:56:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 22:56:17 -0000
Received: (qmail 84141 invoked by uid 500); 29 Apr 2010 22:56:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84109 invoked by uid 500); 29 Apr 2010 22:56:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84100 invoked by uid 99); 29 Apr 2010 22:56:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 22:56:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 22:56:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TMtrsF014222
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 22:55:54 GMT
Message-ID: <26228443.22131272581753755.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 18:55:53 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6738) Move cluster_setup.xml from MapReduce
 to Common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Move cluster_setup.xml from MapReduce to Common
-----------------------------------------------

                 Key: HADOOP-6738
                 URL: https://issues.apache.org/jira/browse/HADOOP-6738
             Project: Hadoop Common
          Issue Type: Test
            Reporter: Tom White
            Assignee: Tom White
            Priority: Blocker
             Fix For: 0.21.0


Common half of MAPREDUCE-1404.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7856-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 23:26:16 2010
Return-Path: <common-issues-return-7856-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17465 invoked from network); 29 Apr 2010 23:26:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 23:26:16 -0000
Received: (qmail 99424 invoked by uid 500); 29 Apr 2010 23:26:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99352 invoked by uid 500); 29 Apr 2010 23:26:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99344 invoked by uid 99); 29 Apr 2010 23:26:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 23:26:16 +0000
X-ASF-Spam-Status: No, hits=-1369.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 23:26:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TNPs56014495
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 23:25:54 GMT
Message-ID: <8558843.22681272583554625.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 19:25:54 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6630) hadoop-config.sh fails to get
 executed if hadoop wrapper scripts are in path
In-Reply-To: <1961052931.216791268355387410.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6630:
------------------------------

           Status: Resolved  (was: Patch Available)
     Hadoop Flags: [Reviewed]
         Assignee: Allen Wittenauer
    Fix Version/s: 0.21.0
       Resolution: Fixed

I've just committed this. Thanks Allen!

> hadoop-config.sh fails to get executed if hadoop wrapper scripts are in path
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-6630
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6630
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: Allen Wittenauer
>            Assignee: Allen Wittenauer
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6630.txt
>
>
> If the hadoop/bin commands are in the path, hadoop-config.sh doesn't executed until much later in the stack because $0 lacks a path.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7857-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 23:37:17 2010
Return-Path: <common-issues-return-7857-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21699 invoked from network); 29 Apr 2010 23:37:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 23:37:17 -0000
Received: (qmail 2165 invoked by uid 500); 29 Apr 2010 23:37:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2122 invoked by uid 500); 29 Apr 2010 23:37:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2114 invoked by uid 99); 29 Apr 2010 23:37:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 23:37:17 +0000
X-ASF-Spam-Status: No, hits=-1369.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 23:37:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TNat4b014562
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 23:36:55 GMT
Message-ID: <5551665.22801272584215334.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 19:36:55 -0400 (EDT)
From: "Scott Chen (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6590) Add a username check in
 hadoop-daemon.sh
In-Reply-To: <271804197.441811266874468125.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862476#action_12862476 ] 

Scott Chen commented on HADOOP-6590:
------------------------------------

We have been using this script internally for a while.
It stops me from running things as root a few times.
I think it is a good feature to have.

Could someone help take a look at it?

> Add a username check in hadoop-daemon.sh
> ----------------------------------------
>
>                 Key: HADOOP-6590
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6590
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.22.0
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6590.patch
>
>
> We experienced a case that sometimes we accidentally started HDFS or MAPREDUCE with root user. Then the directory permission will be modified and we have to chown them. It will be nice if there can be a username checking in the hadoop-daemon.sh script so that we always start with the desired username.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7858-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Apr 29 23:43:20 2010
Return-Path: <common-issues-return-7858-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24661 invoked from network); 29 Apr 2010 23:43:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Apr 2010 23:43:20 -0000
Received: (qmail 4853 invoked by uid 500); 29 Apr 2010 23:43:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4828 invoked by uid 500); 29 Apr 2010 23:43:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4820 invoked by uid 99); 29 Apr 2010 23:43:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 23:43:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Apr 2010 23:43:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3TNguiP014646
	for <common-issues@hadoop.apache.org>; Thu, 29 Apr 2010 23:42:56 GMT
Message-ID: <4063553.22981272584576200.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 19:42:56 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-3659) Patch to allow hadoop native to
 compile on Mac OS X
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862479#action_12862479 ] 

Allen Wittenauer commented on HADOOP-3659:
------------------------------------------

A couple of people have had problems getting a 32-bit or 64-bit build.  I'd recommend setting ANT_OPTS=-d32 or ANT_OPTS=-d64.  This seems to help the ant native compilation targets to build the proper version.

> Patch to allow hadoop native to compile on Mac OS X
> ---------------------------------------------------
>
>                 Key: HADOOP-3659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3659
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: native
>    Affects Versions: 0.20.0
>         Environment: Mac OS X 10.5.3
>            Reporter: Colin Evans
>            Assignee: Colin Evans
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-3659.patch, hadoop-native-mac.patch, hadoop-native-mac.patch
>
>
> This patch makes the autoconf script work on Mac OS X.  LZO needs to be installed (including the optional shared libraries) for the compile to succeed.  You'll want to regenerate the configure script using autoconf after applying this patch.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7859-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 00:11:21 2010
Return-Path: <common-issues-return-7859-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39337 invoked from network); 30 Apr 2010 00:11:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 00:11:20 -0000
Received: (qmail 21277 invoked by uid 500); 30 Apr 2010 00:11:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21228 invoked by uid 500); 30 Apr 2010 00:11:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21220 invoked by uid 99); 30 Apr 2010 00:11:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 00:11:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 00:11:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3U0AvmL014977
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 00:10:57 GMT
Message-ID: <28509274.23551272586257096.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 20:10:57 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-3659) Patch to allow hadoop native to
 compile on Mac OS X
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862486#action_12862486 ] 

Hong Tang commented on HADOOP-3659:
-----------------------------------

Hadoop requires Java 1.6, and on Mac OS X, Java 1.6 only comes with 64-bit version. So there is really no way to get 32-bit hadoop with native compilation, right?

> Patch to allow hadoop native to compile on Mac OS X
> ---------------------------------------------------
>
>                 Key: HADOOP-3659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3659
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: native
>    Affects Versions: 0.20.0
>         Environment: Mac OS X 10.5.3
>            Reporter: Colin Evans
>            Assignee: Colin Evans
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-3659.patch, hadoop-native-mac.patch, hadoop-native-mac.patch
>
>
> This patch makes the autoconf script work on Mac OS X.  LZO needs to be installed (including the optional shared libraries) for the compile to succeed.  You'll want to regenerate the configure script using autoconf after applying this patch.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7860-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 00:15:18 2010
Return-Path: <common-issues-return-7860-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41541 invoked from network); 30 Apr 2010 00:15:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 00:15:17 -0000
Received: (qmail 26092 invoked by uid 500); 30 Apr 2010 00:15:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26067 invoked by uid 500); 30 Apr 2010 00:15:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26059 invoked by uid 99); 30 Apr 2010 00:15:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 00:15:17 +0000
X-ASF-Spam-Status: No, hits=-1369.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 00:15:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3U0Eud6015014
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 00:14:56 GMT
Message-ID: <9929948.23651272586496194.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 20:14:56 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-3659) Patch to allow hadoop native to
 compile on Mac OS X
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-3659:
------------------------------

    Attachment: hadoop-0.20-compile-native-on-mac.patch
                hadoop-0.20-compile-native-on-mac-no-autogen-files.patch

BTW, in case it might be helpful, attached are two patches I have that allows Hadoop 0.20 to be compilable with -Dcompile.native=true. There are two variations, one with the bare minimal changes, and you need to run aclocal, autoconf and automake, and the other includes the artifacts of these commands.

> Patch to allow hadoop native to compile on Mac OS X
> ---------------------------------------------------
>
>                 Key: HADOOP-3659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3659
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: native
>    Affects Versions: 0.20.0
>         Environment: Mac OS X 10.5.3
>            Reporter: Colin Evans
>            Assignee: Colin Evans
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-0.20-compile-native-on-mac-no-autogen-files.patch, hadoop-0.20-compile-native-on-mac.patch, HADOOP-3659.patch, hadoop-native-mac.patch, hadoop-native-mac.patch
>
>
> This patch makes the autoconf script work on Mac OS X.  LZO needs to be installed (including the optional shared libraries) for the compile to succeed.  You'll want to regenerate the configure script using autoconf after applying this patch.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7861-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 00:19:18 2010
Return-Path: <common-issues-return-7861-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43280 invoked from network); 30 Apr 2010 00:19:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 00:19:17 -0000
Received: (qmail 27948 invoked by uid 500); 30 Apr 2010 00:19:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27922 invoked by uid 500); 30 Apr 2010 00:19:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27910 invoked by uid 99); 30 Apr 2010 00:19:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 00:19:17 +0000
X-ASF-Spam-Status: No, hits=-1370.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 00:19:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3U0ItiO015053
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 00:18:56 GMT
Message-ID: <602590.23771272586735920.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 20:18:55 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-3659) Patch to allow hadoop native to
 compile on Mac OS X
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862489#action_12862489 ] 

Allen Wittenauer commented on HADOOP-3659:
------------------------------------------


Java 1.6 on Snow Leopard supports both 32-bit and 64-bit.  (i.e., like Solaris, it honors -d from the same executable)  If it doesn't, Apple sure went through a lot of work:

$ file libjvm.dylib 
libjvm.dylib: Mach-O universal binary with 2 architectures
libjvm.dylib (for architecture i386):	Mach-O dynamically linked shared library i386
libjvm.dylib (for architecture x86_64):	Mach-O 64-bit dynamically linked shared library x86_64


> Patch to allow hadoop native to compile on Mac OS X
> ---------------------------------------------------
>
>                 Key: HADOOP-3659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3659
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: native
>    Affects Versions: 0.20.0
>         Environment: Mac OS X 10.5.3
>            Reporter: Colin Evans
>            Assignee: Colin Evans
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-0.20-compile-native-on-mac-no-autogen-files.patch, hadoop-0.20-compile-native-on-mac.patch, HADOOP-3659.patch, hadoop-native-mac.patch, hadoop-native-mac.patch
>
>
> This patch makes the autoconf script work on Mac OS X.  LZO needs to be installed (including the optional shared libraries) for the compile to succeed.  You'll want to regenerate the configure script using autoconf after applying this patch.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7862-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 00:39:18 2010
Return-Path: <common-issues-return-7862-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61839 invoked from network); 30 Apr 2010 00:39:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 00:39:18 -0000
Received: (qmail 43335 invoked by uid 500); 30 Apr 2010 00:39:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43296 invoked by uid 500); 30 Apr 2010 00:39:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43288 invoked by uid 99); 30 Apr 2010 00:39:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 00:39:18 +0000
X-ASF-Spam-Status: No, hits=-1370.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 00:39:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3U0cuRY015224
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 00:38:56 GMT
Message-ID: <28481401.24181272587936759.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 20:38:56 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-3659) Patch to allow hadoop native to
 compile on Mac OS X
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862494#action_12862494 ] 

Hong Tang commented on HADOOP-3659:
-----------------------------------

I see. I am using Leopard and it looks like that it does not support 32-bit 1.6.

{noformat}
% file Versions/1.6.0/Libraries/libjvm.dylib 
Versions/1.6.0/Libraries/libjvm.dylib: symbolic link to libclient.dylib
% file Versions/1.6.0/Libraries/libclient.dylib 
Versions/1.6.0/Libraries/libclient.dylib: Mach-O 64-bit dynamically linked shared library x86_64
{noformat}

> Patch to allow hadoop native to compile on Mac OS X
> ---------------------------------------------------
>
>                 Key: HADOOP-3659
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3659
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: native
>    Affects Versions: 0.20.0
>         Environment: Mac OS X 10.5.3
>            Reporter: Colin Evans
>            Assignee: Colin Evans
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-0.20-compile-native-on-mac-no-autogen-files.patch, hadoop-0.20-compile-native-on-mac.patch, HADOOP-3659.patch, hadoop-native-mac.patch, hadoop-native-mac.patch
>
>
> This patch makes the autoconf script work on Mac OS X.  LZO needs to be installed (including the optional shared libraries) for the compile to succeed.  You'll want to regenerate the configure script using autoconf after applying this patch.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7863-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 01:29:15 2010
Return-Path: <common-issues-return-7863-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88712 invoked from network); 30 Apr 2010 01:29:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 01:29:15 -0000
Received: (qmail 81352 invoked by uid 500); 30 Apr 2010 01:29:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81330 invoked by uid 500); 30 Apr 2010 01:29:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81322 invoked by uid 99); 30 Apr 2010 01:29:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 01:29:15 +0000
X-ASF-Spam-Status: No, hits=-1370.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 01:29:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3U1SrSx015594
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 01:28:53 GMT
Message-ID: <8061227.24681272590933422.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 21:28:53 -0400 (EDT)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6739) DistributedCache
 add(File|Archive)ToClassPath should only use FileSystem if path is not
 fully qualified
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

DistributedCache add(File|Archive)ToClassPath should only use FileSystem if path is not fully qualified
-------------------------------------------------------------------------------------------------------

                 Key: HADOOP-6739
                 URL: https://issues.apache.org/jira/browse/HADOOP-6739
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Alejandro Abdelnur


Currently setting a file/archive in the DistributedCache classpath creates a FileSystem instance to fully qualify the path, even if the path given is already fully qualified.

This forces a connection to the cluster.

The methods should check if the path is already fully qualified and if so it should not create a FileSystem instance and try to qualify the path.

This would allow creating a jobconf in disconnected mode until submission time.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7864-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 02:23:21 2010
Return-Path: <common-issues-return-7864-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12492 invoked from network); 30 Apr 2010 02:23:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 02:23:20 -0000
Received: (qmail 13364 invoked by uid 500); 30 Apr 2010 02:23:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13333 invoked by uid 500); 30 Apr 2010 02:23:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13325 invoked by uid 99); 30 Apr 2010 02:23:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 02:23:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 02:23:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3U2MuqN015888
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 02:22:56 GMT
Message-ID: <30721266.25141272594176111.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 22:22:56 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862515#action_12862515 ] 

Sanjay Radia commented on HADOOP-6668:
--------------------------------------

Hmm I and also suresh had missed this Jira. There was duplicate working going on in hadoop-6289 which was filed
in sept 09.
There is a proposal attached in the other one - we should sync the two.
Suresh and I spent a fair amount of time in the analysis of the common but were  waiting for feedback before generating the patch.

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7865-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 02:53:19 2010
Return-Path: <common-issues-return-7865-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24797 invoked from network); 30 Apr 2010 02:53:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 02:53:19 -0000
Received: (qmail 24392 invoked by uid 500); 30 Apr 2010 02:53:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24326 invoked by uid 500); 30 Apr 2010 02:53:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24318 invoked by uid 99); 30 Apr 2010 02:53:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 02:53:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 02:53:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3U2qski015998
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 02:52:54 GMT
Message-ID: <12057300.25271272595974409.JavaMail.jira@thor>
Date: Thu, 29 Apr 2010 22:52:54 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6730) Bug in FileContext#copy
In-Reply-To: <8913331.3881272480709335.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862517#action_12862517 ] 

Jakob Homan commented on HADOOP-6730:
-------------------------------------

Review of the patch:
* Lines 47-52: These notes aren't necessary since the test writers should know about @Before and @After. Also, the provided methods (setUp() and tearDown()) conflict with the provided names.
* Lines 55 & 56: These values should be all capitalized and final. Alternatively, it may be good to provide these values via functions so that implementing classes may override as needed.
* Line 64: catch should be on the same line as closing brace, per our coding style
* Line 70: fc should not be declared static. It is the responsibility of each implementing class to provide an instance of it. Also, fc should probably be protected
* Line 91: Please provide assert message for assertion failure
* Since this is a test that the copy method worked, we should prove it by comparing the contents of the copied file with the original.
* Lines 135-138: This code: {code}@After
public void tearDown() throws Exception {
  super.tearDown();
}{code} is a no-op and should be removed.

Nits:
* Rather than Assert.assertTrue() you can do a static import of org.junit.Assert.{assertTrue|*}.
* Line 57: Remove extra line
* Line 47: "Since this a junit 4" ->"Since this is a junit 4 test"

> Bug in FileContext#copy
> -----------------------
>
>                 Key: HADOOP-6730
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6730
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Eli Collins
>            Assignee: Ravi Phulari
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6730.patch
>
>
> Thanks to Eli, He noticed that there is no test for FileContext#Copy operation. 
> On further investigation with the help of Sanjay we found that there is bug in FileContext#checkDest.
> {noformat}
>   FileStatus dstFs = getFileStatus(dst);
>     try {
>       if (dstFs.isDir()) {
>         if (null == srcNa
> {noformat}
>  *FileStatus dstFs = getFileStatus(dst);* should be in try...catch block.
> {noformat}
>     try {
>        FileStatus dstFs = getFileStatus(dst);
>        if (dstFs.isDir()) {
>           if (null == srcNa
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7868-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 07:27:46 2010
Return-Path: <common-issues-return-7868-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2050 invoked from network); 30 Apr 2010 07:27:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 07:27:16 -0000
Received: (qmail 43419 invoked by uid 500); 30 Apr 2010 06:27:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43349 invoked by uid 500); 30 Apr 2010 06:27:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43341 invoked by uid 99); 30 Apr 2010 06:27:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 06:27:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 06:27:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3U6Quuj017791
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 06:26:56 GMT
Message-ID: <12079561.27171272608816185.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 02:26:56 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6740) Move commands_manual.xml from
 mapreduce into common
In-Reply-To: <17589870.27071272608574801.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V updated HADOOP-6740:
------------------------------

    Priority: Blocker  (was: Major)

> Move commands_manual.xml from mapreduce into common
> ---------------------------------------------------
>
>                 Key: HADOOP-6740
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6740
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.21.0
>            Reporter: Vinod K V
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>
> Long term, we want to split the commands manual separately into the sub-projects (MAPEDUCE-1079). But for 0.21 atleast we can move it into common. Having it in mapreduce is definitely inconvenient.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7869-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 07:47:22 2010
Return-Path: <common-issues-return-7869-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13882 invoked from network); 30 Apr 2010 07:47:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 07:47:16 -0000
Received: (qmail 59697 invoked by uid 500); 30 Apr 2010 06:47:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59657 invoked by uid 500); 30 Apr 2010 06:47:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59649 invoked by uid 99); 30 Apr 2010 06:47:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 06:47:15 +0000
X-ASF-Spam-Status: No, hits=-1370.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 06:47:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3U6kr9i017914
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 06:46:54 GMT
Message-ID: <13295333.27401272610013948.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 02:46:53 -0400 (EDT)
From: "Joni Niemi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6734) Jets3tNativeFileSystemStore wrongly
 calls S3Service.createBucket during initialisation
In-Reply-To: <30509879.3061272538794368.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862562#action_12862562 ] 

Joni Niemi commented on HADOOP-6734:
------------------------------------

Sure. I had to drop calling S3 straight from Hadoop code anyhow (due to us using S3 and  Amazon Elastic MapReduce in EU). I don't know what's Hadoop policy on making patches to older releases, so I won't mark this "won't fix".


> Jets3tNativeFileSystemStore wrongly calls S3Service.createBucket during initialisation
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6734
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6734
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 0.18.3
>         Environment: S3 Europe
>            Reporter: Joni Niemi
>
> Reason: If a bucket is created with CreateBucketConfiguration specified, s3service.createBucket will fail. 
> Symptoms: If a bucket has CreateBucketConfiguration, Jets3tNativeFileSystemStore will fail with BucketAlreadyOwnedByYou Error.
> A detailed descrioption from a blog (http://john.keyes.ie/boto-create_bucket-bucketalreadyownedbyyou-error/)
> {quote}This evening I encountered a problem with it though. When the bucket did not exist, the method behaved as expected. When the bucket did exist though I received the following error response:
> {code}<?xml version="1.0" encoding="UTF-8"?>
> <Error>
>     <Code>BucketAlreadyOwnedByYou</Code>
>     <Message>
>         Your previous request to create the named 
>         bucket succeeded and you already own it.
>     </Message>
>     ...
> </Error>{code}
> This problem only manifests itself with buckets that are hosted in the EU. If the bucket is created in the US then the create_bucket method behaves as described.
>     For buckets created with a <CreateBucketConfiguration>, you will receive an error if you attempt to recreate the same bucket.
> To create a bucket in the EU, the bucket is created with a CreateBucketConfiguration specified. I now use the following code to avoid the problem and it works for both US and EU buckets.
> {code}
> def get_bucket():
>     try:
>         bucket = conn.get_bucket('xxx', validate=True)
>     except S3ResponseError, e:
>         if e.code == "NoSuchBucket":
>             bucket = conn.create_bucket('xxx', location='EU')
>         else:
>             raise e
>     return bucket
> {code}
> {quote}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7867-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 08:09:59 2010
Return-Path: <common-issues-return-7867-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23159 invoked from network); 30 Apr 2010 08:09:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 08:09:57 -0000
Received: (qmail 40449 invoked by uid 500); 30 Apr 2010 06:23:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40326 invoked by uid 500); 30 Apr 2010 06:23:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40312 invoked by uid 99); 30 Apr 2010 06:23:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 06:23:16 +0000
X-ASF-Spam-Status: No, hits=-1370.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 06:23:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3U6MsBQ017750
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 06:22:55 GMT
Message-ID: <17589870.27071272608574801.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 02:22:54 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6740) Move commands_manual.xml from
 mapreduce into common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Move commands_manual.xml from mapreduce into common
---------------------------------------------------

                 Key: HADOOP-6740
                 URL: https://issues.apache.org/jira/browse/HADOOP-6740
             Project: Hadoop Common
          Issue Type: Bug
          Components: documentation
    Affects Versions: 0.21.0
            Reporter: Vinod K V
            Assignee: Tom White
             Fix For: 0.21.0


Long term, we want to split the commands manual separately into the sub-projects (MAPEDUCE-1079). But for 0.21 atleast we can move it into common. Having it in mapreduce is definitely inconvenient.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7866-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 08:21:07 2010
Return-Path: <common-issues-return-7866-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27309 invoked from network); 30 Apr 2010 08:21:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 08:21:01 -0000
Received: (qmail 6985 invoked by uid 500); 30 Apr 2010 05:34:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6866 invoked by uid 500); 30 Apr 2010 05:34:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6858 invoked by uid 99); 30 Apr 2010 05:34:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 05:34:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 05:34:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3U5XuOH017302
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 05:33:56 GMT
Message-ID: <24878834.26291272605636517.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 01:33:56 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862541#action_12862541 ] 

Tom White commented on HADOOP-6668:
-----------------------------------

Thanks, Sanjay. Yes, it's unfortunate that we missed each others JIRAs! I'll have a look at the differences tomorrow, so we can get this into 0.21 in the coming days. 

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7870-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 08:35:24 2010
Return-Path: <common-issues-return-7870-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32951 invoked from network); 30 Apr 2010 08:35:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 08:35:23 -0000
Received: (qmail 49853 invoked by uid 500); 30 Apr 2010 08:35:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49107 invoked by uid 500); 30 Apr 2010 08:35:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49091 invoked by uid 99); 30 Apr 2010 08:35:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 08:35:20 +0000
X-ASF-Spam-Status: No, hits=-1371.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 08:35:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3U8YwLV019095
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 08:34:58 GMT
Message-ID: <24291544.28721272616498875.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 04:34:58 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6730) Bug in FileContext#copy
In-Reply-To: <8913331.3881272480709335.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6730:
---------------------------------

    Attachment: HADOOP-6730.2.patch

Thanks for reviewing patch Jakob.
Uploading updated patch. 

> Bug in FileContext#copy
> -----------------------
>
>                 Key: HADOOP-6730
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6730
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Eli Collins
>            Assignee: Ravi Phulari
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6730.2.patch, HADOOP-6730.patch
>
>
> Thanks to Eli, He noticed that there is no test for FileContext#Copy operation. 
> On further investigation with the help of Sanjay we found that there is bug in FileContext#checkDest.
> {noformat}
>   FileStatus dstFs = getFileStatus(dst);
>     try {
>       if (dstFs.isDir()) {
>         if (null == srcNa
> {noformat}
>  *FileStatus dstFs = getFileStatus(dst);* should be in try...catch block.
> {noformat}
>     try {
>        FileStatus dstFs = getFileStatus(dst);
>        if (dstFs.isDir()) {
>           if (null == srcNa
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7871-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 12:47:20 2010
Return-Path: <common-issues-return-7871-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1034 invoked from network); 30 Apr 2010 12:47:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 12:47:20 -0000
Received: (qmail 15370 invoked by uid 500); 30 Apr 2010 12:47:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15326 invoked by uid 500); 30 Apr 2010 12:47:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15318 invoked by uid 99); 30 Apr 2010 12:47:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 12:47:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 12:47:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3UCksHI022311
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 12:46:55 GMT
Message-ID: <6557149.31751272631614957.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 08:46:54 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862653#action_12862653 ] 

Hadoop QA commented on HADOOP-6668:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12443216/HADOOP-6668.patch
  against trunk revision 939510.

    +1 @author.  The patch does not contain any @author tags.

    +0 tests included.  The patch appears to be a documentation patch that doesn't require tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/489/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/489/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/489/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/489/console

This message is automatically generated.

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7872-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 16:14:22 2010
Return-Path: <common-issues-return-7872-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52182 invoked from network); 30 Apr 2010 16:14:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 16:14:22 -0000
Received: (qmail 2487 invoked by uid 500); 30 Apr 2010 16:14:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2436 invoked by uid 500); 30 Apr 2010 16:14:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2428 invoked by uid 99); 30 Apr 2010 16:14:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 16:14:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 16:14:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3UGDvAl003940
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 16:13:57 GMT
Message-ID: <14380566.34781272644037099.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 12:13:57 -0400 (EDT)
From: "Neil Bliss (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6704) add support for Parascale filesystem
In-Reply-To: <6991209.115951271258268758.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Neil Bliss updated HADOOP-6704:
-------------------------------

           Status: Patch Available  (was: Open)
    Fix Version/s: 0.20.3

Adds support for Parascale storage as the backing store for Hadoop instances.

> add support for Parascale filesystem
> ------------------------------------
>
>                 Key: HADOOP-6704
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6704
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.20.2
>            Reporter: Neil Bliss
>             Fix For: 0.20.3
>
>         Attachments: HADOOP-6704.0.20.2.patch, HADOOP-6704.patch
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Parascale has developed an org.apache.hadoop.fs implementation that allows users to use Hadoop on Parascale storage clusters.  We'd like to contribute this work to the community. Should this be placed under contrib, or integrated into the org.apache.hadoop.fs space?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7873-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 16:40:21 2010
Return-Path: <common-issues-return-7873-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56463 invoked from network); 30 Apr 2010 16:40:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 16:40:21 -0000
Received: (qmail 27525 invoked by uid 500); 30 Apr 2010 16:40:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27499 invoked by uid 500); 30 Apr 2010 16:40:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27491 invoked by uid 99); 30 Apr 2010 16:40:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 16:40:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 16:40:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3UGduWI007408
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 16:39:57 GMT
Message-ID: <21387576.35151272645596687.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 12:39:56 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6704) add support for Parascale
 filesystem
In-Reply-To: <6991209.115951271258268758.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862740#action_12862740 ] 

Hadoop QA commented on HADOOP-6704:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12443204/HADOOP-6704.patch
  against trunk revision 939510.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 61 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The applied patch generated 1025 javac compiler warnings (more than the trunk's current 1017 warnings).

    -1 findbugs.  The patch appears to introduce 2 new Findbugs warnings.

    -1 release audit.  The applied patch generated 5 release audit warnings (more than the trunk's current 1 warnings).

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/490/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/490/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/490/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/490/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/490/console

This message is automatically generated.

> add support for Parascale filesystem
> ------------------------------------
>
>                 Key: HADOOP-6704
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6704
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.20.2
>            Reporter: Neil Bliss
>             Fix For: 0.20.3
>
>         Attachments: HADOOP-6704.0.20.2.patch, HADOOP-6704.patch
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Parascale has developed an org.apache.hadoop.fs implementation that allows users to use Hadoop on Parascale storage clusters.  We'd like to contribute this work to the community. Should this be placed under contrib, or integrated into the org.apache.hadoop.fs space?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7874-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 17:22:19 2010
Return-Path: <common-issues-return-7874-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69869 invoked from network); 30 Apr 2010 17:22:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 17:22:18 -0000
Received: (qmail 86224 invoked by uid 500); 30 Apr 2010 17:22:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86190 invoked by uid 500); 30 Apr 2010 17:22:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86179 invoked by uid 99); 30 Apr 2010 17:22:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 17:22:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 17:22:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3UHLsNL015428
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 17:21:54 GMT
Message-ID: <32379461.35691272648114288.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 13:21:54 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6741) Refactor Hadoop Command line
 interface.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Refactor Hadoop Command line interface.
---------------------------------------

                 Key: HADOOP-6741
                 URL: https://issues.apache.org/jira/browse/HADOOP-6741
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs
            Reporter: Ravi Phulari
             Fix For: 0.22.0


Currently Hadoop fs commands are scattered between FsShell , FsPermissions  and DfsAdmin .  It will be great if refactor hadoop CLI for better debugging, performance and code maintenance.

Any thoughts? 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7875-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 17:45:16 2010
Return-Path: <common-issues-return-7875-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75423 invoked from network); 30 Apr 2010 17:45:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 17:45:16 -0000
Received: (qmail 22930 invoked by uid 500); 30 Apr 2010 17:45:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22818 invoked by uid 500); 30 Apr 2010 17:45:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22810 invoked by uid 99); 30 Apr 2010 17:45:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 17:45:16 +0000
X-ASF-Spam-Status: No, hits=-1373.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 17:45:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3UHituG016484
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 17:44:55 GMT
Message-ID: <1101694.36311272649494993.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 13:44:54 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862763#action_12862763 ] 

Sanjay Radia commented on HADOOP-6668:
--------------------------------------

Tom, based on my first glance at the list of criteria that you state above and the patch you have attached it appears that you have also ignored the original jira Hadoop-5073 and the very concrete proposal in that Jira:
https://issues.apache.org/jira/browse/HADOOP-5073?focusedCommentId=12759723&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12759723
HADOOP-6289,  which was linked to Hadoop-5073,  used that concrete proposal as the starting point. 
(BTW HADOOP-6668 should have been linked to 5073 - if it had been,  the duplication might have been detected.)

I suggest that we first come to an agreement on the criteria. 
I will shortly post a my feedback on the criteria you have state above and contrast this against those used on the proposal in Hadoop-6289.


> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7876-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 18:07:17 2010
Return-Path: <common-issues-return-7876-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81759 invoked from network); 30 Apr 2010 18:07:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 18:07:16 -0000
Received: (qmail 44464 invoked by uid 500); 30 Apr 2010 18:07:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44438 invoked by uid 500); 30 Apr 2010 18:07:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44430 invoked by uid 99); 30 Apr 2010 18:07:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 18:07:16 +0000
X-ASF-Spam-Status: No, hits=-1373.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 18:07:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3UI6tFX019827
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 18:06:55 GMT
Message-ID: <11431844.36841272650815482.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 14:06:55 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862775#action_12862775 ] 

Tom White commented on HADOOP-6668:
-----------------------------------

Sanjay, I did see your proposal in HADOOP-5073, which informed my choice of classifications in this issue, but I missed the linked issue (HADOOP-6289). I think we are broadly in agreement on the actual classifications - I think going through these concrete differences to reconcile them would be the best way to get this resolved. I'll post a list of differences soon.

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7877-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 18:17:19 2010
Return-Path: <common-issues-return-7877-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85728 invoked from network); 30 Apr 2010 18:17:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 18:17:19 -0000
Received: (qmail 55937 invoked by uid 500); 30 Apr 2010 18:17:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55912 invoked by uid 500); 30 Apr 2010 18:17:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55904 invoked by uid 99); 30 Apr 2010 18:17:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 18:17:19 +0000
X-ASF-Spam-Status: No, hits=-1373.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 18:17:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3UIGwoF020429
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 18:16:58 GMT
Message-ID: <21550833.37211272651418404.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 14:16:58 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6741) Refactor Hadoop Command line
 interface.
In-Reply-To: <32379461.35691272648114288.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862782#action_12862782 ] 

Allen Wittenauer commented on HADOOP-6741:
------------------------------------------

caN wE diTch tHe camelCase toO?



> Refactor Hadoop Command line interface.
> ---------------------------------------
>
>                 Key: HADOOP-6741
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6741
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Ravi Phulari
>             Fix For: 0.22.0
>
>
> Currently Hadoop fs commands are scattered between FsShell , FsPermissions  and DfsAdmin .  It will be great if refactor hadoop CLI for better debugging, performance and code maintenance.
> Any thoughts? 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7878-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 18:27:22 2010
Return-Path: <common-issues-return-7878-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4487 invoked from network); 30 Apr 2010 18:27:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 18:27:22 -0000
Received: (qmail 65424 invoked by uid 500); 30 Apr 2010 18:27:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65379 invoked by uid 500); 30 Apr 2010 18:27:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65371 invoked by uid 99); 30 Apr 2010 18:27:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 18:27:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 18:27:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3UIQvJi020585
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 18:26:58 GMT
Message-ID: <6348996.37621272652017823.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 14:26:57 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862790#action_12862790 ] 

Suresh Srinivas commented on HADOOP-6563:
-----------------------------------------

+1 for the patch.

> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6563-1.patch, hadoop-6563-2.patch, hadoop-6563-3.patch, hadoop-6563-4.patch, hadoop-6563-5.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7879-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 18:57:18 2010
Return-Path: <common-issues-return-7879-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10830 invoked from network); 30 Apr 2010 18:57:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 18:57:18 -0000
Received: (qmail 94977 invoked by uid 500); 30 Apr 2010 18:57:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94912 invoked by uid 500); 30 Apr 2010 18:57:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94901 invoked by uid 99); 30 Apr 2010 18:57:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 18:57:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 18:57:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3UIusGc029786
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 18:56:54 GMT
Message-ID: <10694101.38311272653814407.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 14:56:54 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6730) Bug in FileContext#copy
In-Reply-To: <8913331.3881272480709335.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6730:
---------------------------------

    Attachment: HADOOP-6730.3.patch

Updated patch with changes.
1. Removed unnecessary  tearDown method.
2. Changed file read write to bytes from String.

> Bug in FileContext#copy
> -----------------------
>
>                 Key: HADOOP-6730
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6730
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Eli Collins
>            Assignee: Ravi Phulari
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6730.2.patch, HADOOP-6730.3.patch, HADOOP-6730.patch
>
>
> Thanks to Eli, He noticed that there is no test for FileContext#Copy operation. 
> On further investigation with the help of Sanjay we found that there is bug in FileContext#checkDest.
> {noformat}
>   FileStatus dstFs = getFileStatus(dst);
>     try {
>       if (dstFs.isDir()) {
>         if (null == srcNa
> {noformat}
>  *FileStatus dstFs = getFileStatus(dst);* should be in try...catch block.
> {noformat}
>     try {
>        FileStatus dstFs = getFileStatus(dst);
>        if (dstFs.isDir()) {
>           if (null == srcNa
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7880-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 19:29:20 2010
Return-Path: <common-issues-return-7880-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19117 invoked from network); 30 Apr 2010 19:29:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 19:29:20 -0000
Received: (qmail 37485 invoked by uid 500); 30 Apr 2010 19:29:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37424 invoked by uid 500); 30 Apr 2010 19:29:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37416 invoked by uid 99); 30 Apr 2010 19:29:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 19:29:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 19:29:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3UJSuNE004420
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 19:28:56 GMT
Message-ID: <30288370.39051272655736127.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 15:28:56 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6741) Refactor Hadoop Command line
 interface.
In-Reply-To: <32379461.35691272648114288.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862822#action_12862822 ] 

Eli Collins commented on HADOOP-6741:
-------------------------------------

See HADOOP-6425 (Refactor FsShell to make commands pluggable), agree that this should be expanded to FsPermissions and DfsAdmin.

> Refactor Hadoop Command line interface.
> ---------------------------------------
>
>                 Key: HADOOP-6741
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6741
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Ravi Phulari
>             Fix For: 0.22.0
>
>
> Currently Hadoop fs commands are scattered between FsShell , FsPermissions  and DfsAdmin .  It will be great if refactor hadoop CLI for better debugging, performance and code maintenance.
> Any thoughts? 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7881-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 19:29:22 2010
Return-Path: <common-issues-return-7881-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19216 invoked from network); 30 Apr 2010 19:29:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 19:29:22 -0000
Received: (qmail 38213 invoked by uid 500); 30 Apr 2010 19:29:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38180 invoked by uid 500); 30 Apr 2010 19:29:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38130 invoked by uid 99); 30 Apr 2010 19:29:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 19:29:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 19:29:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3UJSwTv004447
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 19:28:58 GMT
Message-ID: <1868700.39141272655738831.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 15:28:58 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6730) Bug in FileContext#copy
In-Reply-To: <8913331.3881272480709335.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6730:
---------------------------------

    Attachment: HADOOP-6730.4.patch

Updated patch.
Removed static from fc initialization.
Removed invalid comments.  

> Bug in FileContext#copy
> -----------------------
>
>                 Key: HADOOP-6730
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6730
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Eli Collins
>            Assignee: Ravi Phulari
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6730.2.patch, HADOOP-6730.3.patch, HADOOP-6730.4.patch, HADOOP-6730.patch
>
>
> Thanks to Eli, He noticed that there is no test for FileContext#Copy operation. 
> On further investigation with the help of Sanjay we found that there is bug in FileContext#checkDest.
> {noformat}
>   FileStatus dstFs = getFileStatus(dst);
>     try {
>       if (dstFs.isDir()) {
>         if (null == srcNa
> {noformat}
>  *FileStatus dstFs = getFileStatus(dst);* should be in try...catch block.
> {noformat}
>     try {
>        FileStatus dstFs = getFileStatus(dst);
>        if (dstFs.isDir()) {
>           if (null == srcNa
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7882-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 20:05:17 2010
Return-Path: <common-issues-return-7882-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25830 invoked from network); 30 Apr 2010 20:05:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 20:05:17 -0000
Received: (qmail 77242 invoked by uid 500); 30 Apr 2010 20:05:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77202 invoked by uid 500); 30 Apr 2010 20:05:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77194 invoked by uid 99); 30 Apr 2010 20:05:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 20:05:17 +0000
X-ASF-Spam-Status: No, hits=-1374.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 20:05:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3UK4tt4005623
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 20:04:55 GMT
Message-ID: <2308932.39751272657895868.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 16:04:55 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6563:
------------------------------------

    Status: Patch Available  (was: Open)

resubmitting the patch to trigger hudson tests.

> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6563-1.patch, hadoop-6563-2.patch, hadoop-6563-3.patch, hadoop-6563-4.patch, hadoop-6563-5.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7883-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 20:05:19 2010
Return-Path: <common-issues-return-7883-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25967 invoked from network); 30 Apr 2010 20:05:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 20:05:18 -0000
Received: (qmail 77381 invoked by uid 500); 30 Apr 2010 20:05:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77345 invoked by uid 500); 30 Apr 2010 20:05:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77337 invoked by uid 99); 30 Apr 2010 20:05:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 20:05:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 20:05:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3UK4sT1005608
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 20:04:54 GMT
Message-ID: <2278636.39701272657894651.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 16:04:54 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6563:
------------------------------------

    Status: Open  (was: Patch Available)

> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6563-1.patch, hadoop-6563-2.patch, hadoop-6563-3.patch, hadoop-6563-4.patch, hadoop-6563-5.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7884-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 20:29:18 2010
Return-Path: <common-issues-return-7884-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29546 invoked from network); 30 Apr 2010 20:29:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 20:29:17 -0000
Received: (qmail 99542 invoked by uid 500); 30 Apr 2010 20:29:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99509 invoked by uid 500); 30 Apr 2010 20:29:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99501 invoked by uid 99); 30 Apr 2010 20:29:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 20:29:17 +0000
X-ASF-Spam-Status: No, hits=-1374.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 20:29:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3UKSuOe006948
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 20:28:56 GMT
Message-ID: <16173696.40091272659336610.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 16:28:56 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862846#action_12862846 ] 

Hadoop QA commented on HADOOP-6563:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12443160/hadoop-6563-5.patch
  against trunk revision 939510.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/491/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/491/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/491/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/491/console

This message is automatically generated.

> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6563-1.patch, hadoop-6563-2.patch, hadoop-6563-3.patch, hadoop-6563-4.patch, hadoop-6563-5.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7885-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 21:15:24 2010
Return-Path: <common-issues-return-7885-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42467 invoked from network); 30 Apr 2010 21:15:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 21:15:24 -0000
Received: (qmail 52392 invoked by uid 500); 30 Apr 2010 21:15:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52360 invoked by uid 500); 30 Apr 2010 21:15:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52352 invoked by uid 99); 30 Apr 2010 21:15:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 21:15:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 21:15:22 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3ULF0KZ017994
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 21:15:00 GMT
Message-ID: <5210321.41771272662100198.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 17:15:00 -0400 (EDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6730) Bug in FileContext#copy
In-Reply-To: <8913331.3881272480709335.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6730:
---------------------------------

    Status: Patch Available  (was: Open)

Submitting patch for hudson queue.

> Bug in FileContext#copy
> -----------------------
>
>                 Key: HADOOP-6730
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6730
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Eli Collins
>            Assignee: Ravi Phulari
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6730.2.patch, HADOOP-6730.3.patch, HADOOP-6730.4.patch, HADOOP-6730.patch
>
>
> Thanks to Eli, He noticed that there is no test for FileContext#Copy operation. 
> On further investigation with the help of Sanjay we found that there is bug in FileContext#checkDest.
> {noformat}
>   FileStatus dstFs = getFileStatus(dst);
>     try {
>       if (dstFs.isDir()) {
>         if (null == srcNa
> {noformat}
>  *FileStatus dstFs = getFileStatus(dst);* should be in try...catch block.
> {noformat}
>     try {
>        FileStatus dstFs = getFileStatus(dst);
>        if (dstFs.isDir()) {
>           if (null == srcNa
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7886-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 21:17:18 2010
Return-Path: <common-issues-return-7886-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43163 invoked from network); 30 Apr 2010 21:17:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 21:17:18 -0000
Received: (qmail 55637 invoked by uid 500); 30 Apr 2010 21:17:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55615 invoked by uid 500); 30 Apr 2010 21:17:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55607 invoked by uid 99); 30 Apr 2010 21:17:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 21:17:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 21:17:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3ULGs2Y018353
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 21:16:54 GMT
Message-ID: <14770696.41801272662214311.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 17:16:54 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6693) Add metrics to track kerberos login
 activity
In-Reply-To: <1138587062.22291270763196714.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6693:
------------------------------------

    Attachment: HADOOP-6693.rel20.patch

20 version of the patch. I added bumping up metrics in loginUserFromKeytabAndReturnUGI also. 

The login code in UserGroupInformation  is repeated in many places. There is an opportunity to improve the code by making a method for it.

> Add metrics to track kerberos login activity
> --------------------------------------------
>
>                 Key: HADOOP-6693
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6693
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6693.1.patch, HADOOP-6693.patch, HADOOP-6693.rel20.patch
>
>
> Need metrics to track kerberos login activity such as login rate and the time taken for login.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7887-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 21:21:15 2010
Return-Path: <common-issues-return-7887-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43971 invoked from network); 30 Apr 2010 21:21:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 21:21:15 -0000
Received: (qmail 58629 invoked by uid 500); 30 Apr 2010 21:21:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58605 invoked by uid 500); 30 Apr 2010 21:21:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58597 invoked by uid 99); 30 Apr 2010 21:21:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 21:21:15 +0000
X-ASF-Spam-Status: No, hits=-1374.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 21:21:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3ULKs2N018754
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 21:20:54 GMT
Message-ID: <12171736.41831272662454241.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 17:20:54 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862871#action_12862871 ] 

Eli Collins commented on HADOOP-6563:
-------------------------------------

The TestFilterFileSystem failure is unrelated to this change, fails on trunk after HADOOP-6709. I'll track that down.

> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6563-1.patch, hadoop-6563-2.patch, hadoop-6563-3.patch, hadoop-6563-4.patch, hadoop-6563-5.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7888-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 21:27:19 2010
Return-Path: <common-issues-return-7888-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45795 invoked from network); 30 Apr 2010 21:27:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 21:27:19 -0000
Received: (qmail 64221 invoked by uid 500); 30 Apr 2010 21:27:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64194 invoked by uid 500); 30 Apr 2010 21:27:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64186 invoked by uid 99); 30 Apr 2010 21:27:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 21:27:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 21:27:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3ULQtTB020393
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 21:26:55 GMT
Message-ID: <25198091.41991272662815691.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 17:26:55 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6693) Add metrics to track kerberos login
 activity
In-Reply-To: <1138587062.22291270763196714.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6693:
------------------------------------

    Attachment: HADOOP-6693.rel20.1.patch

New patch with unnecessary debug logs removed. Tested it on cluster to ensure metrics is being updated.

> Add metrics to track kerberos login activity
> --------------------------------------------
>
>                 Key: HADOOP-6693
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6693
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6693.1.patch, HADOOP-6693.patch, HADOOP-6693.rel20.1.patch, HADOOP-6693.rel20.patch
>
>
> Need metrics to track kerberos login activity such as login rate and the time taken for login.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7889-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 21:31:17 2010
Return-Path: <common-issues-return-7889-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47005 invoked from network); 30 Apr 2010 21:31:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 21:31:17 -0000
Received: (qmail 67043 invoked by uid 500); 30 Apr 2010 21:31:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67006 invoked by uid 500); 30 Apr 2010 21:31:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66998 invoked by uid 99); 30 Apr 2010 21:31:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 21:31:17 +0000
X-ASF-Spam-Status: No, hits=-1374.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 21:31:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3ULUupG022265
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 21:30:56 GMT
Message-ID: <20555503.42111272663056017.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 17:30:56 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6693) Add metrics to track kerberos login
 activity
In-Reply-To: <1138587062.22291270763196714.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862878#action_12862878 ] 

Owen O'Malley commented on HADOOP-6693:
---------------------------------------

+1

> Add metrics to track kerberos login activity
> --------------------------------------------
>
>                 Key: HADOOP-6693
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6693
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6693.1.patch, HADOOP-6693.patch, HADOOP-6693.rel20.1.patch, HADOOP-6693.rel20.patch
>
>
> Need metrics to track kerberos login activity such as login rate and the time taken for login.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7890-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 21:37:17 2010
Return-Path: <common-issues-return-7890-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48178 invoked from network); 30 Apr 2010 21:37:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 21:37:17 -0000
Received: (qmail 70566 invoked by uid 500); 30 Apr 2010 21:37:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70524 invoked by uid 500); 30 Apr 2010 21:37:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70515 invoked by uid 99); 30 Apr 2010 21:37:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 21:37:16 +0000
X-ASF-Spam-Status: No, hits=-1374.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 21:37:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3ULatMa024434
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 21:36:55 GMT
Message-ID: <6478272.42281272663415062.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 17:36:55 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6709) Re-instate deprecated FileSystem
 methods that were removed after 0.20
In-Reply-To: <11903376.154751271374849451.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862884#action_12862884 ] 

Sanjay Radia commented on HADOOP-6709:
--------------------------------------

> Nit: please put @Deprecated
More than a Nit. We have to leave the deprecation in. The goal of this Jira is to merely reinstate deleted methods. 
Old deprecation should remain.

> Re-instate deprecated FileSystem methods that were removed after 0.20
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6709
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6709
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6709.patch
>
>
> To make the FileSystem API in the 0.21 release (which should be treated as a minor release) compatible with 0.20 the deprecated methods removed in HADOOP-4779 need to be re-instated. They should still be marked as deprecated, however.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7891-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 21:43:16 2010
Return-Path: <common-issues-return-7891-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49941 invoked from network); 30 Apr 2010 21:43:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 21:43:15 -0000
Received: (qmail 76257 invoked by uid 500); 30 Apr 2010 21:43:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76215 invoked by uid 500); 30 Apr 2010 21:43:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76207 invoked by uid 99); 30 Apr 2010 21:43:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 21:43:15 +0000
X-ASF-Spam-Status: No, hits=-1374.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 21:43:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3ULgsqV026032
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 21:42:54 GMT
Message-ID: <17092928.42411272663774457.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 17:42:54 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6563) Add more tests to
 FileContextSymlinkBaseTest that cover intermediate symlinks in paths
In-Reply-To: <293016623.222581265940867991.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6563:
------------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

I committed the patch. Thank you Eli.

> Add more tests to FileContextSymlinkBaseTest that cover intermediate symlinks in paths
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6563
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6563
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: fs, test
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6563-1.patch, hadoop-6563-2.patch, hadoop-6563-3.patch, hadoop-6563-4.patch, hadoop-6563-5.patch
>
>
> Intermediate symlinks in paths are covered by the tests that use /linkToDir/file, /linkToDir/subDir, etc  eg testCreateVia* in FileContextSymlinkBaseTest. I'll add additional tests to cover other basic operations on paths like /dir/linkToSomeDir/file beyond create() and open().

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7892-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 21:53:20 2010
Return-Path: <common-issues-return-7892-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51465 invoked from network); 30 Apr 2010 21:53:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 21:53:20 -0000
Received: (qmail 82595 invoked by uid 500); 30 Apr 2010 21:53:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82561 invoked by uid 500); 30 Apr 2010 21:53:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82553 invoked by uid 99); 30 Apr 2010 21:53:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 21:53:20 +0000
X-ASF-Spam-Status: No, hits=-1374.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 21:53:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3ULqx44027602
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 21:52:59 GMT
Message-ID: <22703058.42661272664379417.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 17:52:59 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6709) Re-instate deprecated FileSystem
 methods that were removed after 0.20
In-Reply-To: <11903376.154751271374849451.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862892#action_12862892 ] 

Tom White commented on HADOOP-6709:
-----------------------------------

To clarify, the nit was placement of the @Deprecated annotation (so that it was on its own line) rather than whether it was included or not.

> Re-instate deprecated FileSystem methods that were removed after 0.20
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6709
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6709
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6709.patch
>
>
> To make the FileSystem API in the 0.21 release (which should be treated as a minor release) compatible with 0.20 the deprecated methods removed in HADOOP-4779 need to be re-instated. They should still be marked as deprecated, however.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7893-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 22:03:17 2010
Return-Path: <common-issues-return-7893-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54991 invoked from network); 30 Apr 2010 22:03:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 22:03:17 -0000
Received: (qmail 98056 invoked by uid 500); 30 Apr 2010 22:03:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98030 invoked by uid 500); 30 Apr 2010 22:03:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98022 invoked by uid 99); 30 Apr 2010 22:03:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 22:03:17 +0000
X-ASF-Spam-Status: No, hits=-1374.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 22:03:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3UM2u0c029845
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 22:02:56 GMT
Message-ID: <4464681.42891272664976266.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 18:02:56 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6742) Add methods HADOOP-6709 from to
 TestFilterFileSystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Add methods HADOOP-6709 from to TestFilterFileSystem 
-----------------------------------------------------

                 Key: HADOOP-6742
                 URL: https://issues.apache.org/jira/browse/HADOOP-6742
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.21.0, 0.22.0
            Reporter: Eli Collins
            Assignee: Eli Collins
             Fix For: 0.21.0, 0.22.0


HADOOP-6709 added back deprecated methods, these need to be added to the don't check list in TestFilterFileSystem so that test passes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7894-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 22:07:20 2010
Return-Path: <common-issues-return-7894-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56134 invoked from network); 30 Apr 2010 22:07:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 22:07:19 -0000
Received: (qmail 3250 invoked by uid 500); 30 Apr 2010 22:07:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3208 invoked by uid 500); 30 Apr 2010 22:07:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3200 invoked by uid 99); 30 Apr 2010 22:07:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 22:07:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 22:07:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3UM6uOp000508
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 22:06:56 GMT
Message-ID: <12081955.43101272665216159.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 18:06:56 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6742) Add methods HADOOP-6709 from to
 TestFilterFileSystem
In-Reply-To: <4464681.42891272664976266.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6742:
--------------------------------

    Attachment: hadoop-6742-1.patch

Patch attached. TestFilterFileSystem passes for me locally now.

> Add methods HADOOP-6709 from to TestFilterFileSystem 
> -----------------------------------------------------
>
>                 Key: HADOOP-6742
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6742
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6742-1.patch
>
>
> HADOOP-6709 added back deprecated methods, these need to be added to the don't check list in TestFilterFileSystem so that test passes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7895-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 22:15:15 2010
Return-Path: <common-issues-return-7895-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57953 invoked from network); 30 Apr 2010 22:15:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 22:15:15 -0000
Received: (qmail 6624 invoked by uid 500); 30 Apr 2010 22:15:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6589 invoked by uid 500); 30 Apr 2010 22:15:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6581 invoked by uid 99); 30 Apr 2010 22:15:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 22:15:15 +0000
X-ASF-Spam-Status: No, hits=-1374.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 22:15:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3UMErEl002536
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 22:14:54 GMT
Message-ID: <5180755.43251272665693814.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 18:14:53 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6742) Add methods HADOOP-6709 from to
 TestFilterFileSystem
In-Reply-To: <4464681.42891272664976266.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6742:
--------------------------------

    Status: Patch Available  (was: Open)

> Add methods HADOOP-6709 from to TestFilterFileSystem 
> -----------------------------------------------------
>
>                 Key: HADOOP-6742
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6742
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6742-1.patch
>
>
> HADOOP-6709 added back deprecated methods, these need to be added to the don't check list in TestFilterFileSystem so that test passes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7896-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 22:30:18 2010
Return-Path: <common-issues-return-7896-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60406 invoked from network); 30 Apr 2010 22:30:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 22:30:18 -0000
Received: (qmail 17313 invoked by uid 500); 30 Apr 2010 22:30:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17280 invoked by uid 500); 30 Apr 2010 22:30:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17272 invoked by uid 99); 30 Apr 2010 22:30:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 22:30:18 +0000
X-ASF-Spam-Status: No, hits=-1375.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 22:30:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3UMTvn5006126
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 22:29:57 GMT
Message-ID: <32414544.43661272666597356.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 18:29:57 -0400 (EDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6194) Add service base class and tests to
 hadoop-common/util
In-Reply-To: <165334509.1250255474864.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862910#action_12862910 ] 

Steve Loughran commented on HADOOP-6194:
----------------------------------------

thanks Aaron, I will look at that script.

I have just stuck up onto MAPREDUCE-233 the patch that brings the TT and JT under the lifecycle. The JT is the tricky one, as it likes to wait for the filesystem to be live first since the JT added persistence; I think nobody has noticed that other than me as their filesystems come up more reliably than mine, but the problem is there in trunk too. 

> Add service base class and tests to hadoop-common/util
> ------------------------------------------------------
>
>                 Key: HADOOP-6194
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6194
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>             Fix For: 0.22.0
>
>         Attachments: find-new-warnings, HADOOP-6194-1.patch, HADOOP-6194.patch, HADOOP-6194.patch
>
>
> For a service base class, we need to add to common
> # The Service class
> # A mock Service class (it's not in test, its in src/, because it helps in functional tests downstream)
> # Unit tests
> This is the issue to cover this work

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7897-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 22:36:18 2010
Return-Path: <common-issues-return-7897-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61364 invoked from network); 30 Apr 2010 22:36:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 22:36:18 -0000
Received: (qmail 20597 invoked by uid 500); 30 Apr 2010 22:36:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20566 invoked by uid 500); 30 Apr 2010 22:36:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20558 invoked by uid 99); 30 Apr 2010 22:36:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 22:36:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 22:36:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3UMZr8O006778
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 22:35:54 GMT
Message-ID: <22784560.43741272666953804.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 18:35:53 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6742) Add methods HADOOP-6709 from to
 TestFilterFileSystem
In-Reply-To: <4464681.42891272664976266.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862913#action_12862913 ] 

Hadoop QA commented on HADOOP-6742:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12443335/hadoop-6742-1.patch
  against trunk revision 939827.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/62/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/62/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/62/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/62/console

This message is automatically generated.

> Add methods HADOOP-6709 from to TestFilterFileSystem 
> -----------------------------------------------------
>
>                 Key: HADOOP-6742
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6742
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6742-1.patch
>
>
> HADOOP-6709 added back deprecated methods, these need to be added to the don't check list in TestFilterFileSystem so that test passes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7898-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 22:51:15 2010
Return-Path: <common-issues-return-7898-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64729 invoked from network); 30 Apr 2010 22:51:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 22:51:15 -0000
Received: (qmail 39850 invoked by uid 500); 30 Apr 2010 22:51:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39764 invoked by uid 500); 30 Apr 2010 22:51:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39756 invoked by uid 99); 30 Apr 2010 22:51:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 22:51:15 +0000
X-ASF-Spam-Status: No, hits=-1375.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 22:51:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3UMor70010240
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 22:50:53 GMT
Message-ID: <944176.44031272667853676.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 18:50:53 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6742) Add methods HADOOP-6709 from to
 TestFilterFileSystem
In-Reply-To: <4464681.42891272664976266.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6742:
------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

I've just committed this. Thanks Eli!

> Add methods HADOOP-6709 from to TestFilterFileSystem 
> -----------------------------------------------------
>
>                 Key: HADOOP-6742
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6742
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6742-1.patch
>
>
> HADOOP-6709 added back deprecated methods, these need to be added to the don't check list in TestFilterFileSystem so that test passes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7899-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 23:21:20 2010
Return-Path: <common-issues-return-7899-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68535 invoked from network); 30 Apr 2010 23:21:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 23:21:20 -0000
Received: (qmail 65389 invoked by uid 500); 30 Apr 2010 23:21:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65366 invoked by uid 500); 30 Apr 2010 23:21:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65358 invoked by uid 99); 30 Apr 2010 23:21:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 23:21:20 +0000
X-ASF-Spam-Status: No, hits=-1375.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 23:21:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3UNKxPJ018228
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 23:20:59 GMT
Message-ID: <280471.44301272669659345.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 19:20:59 -0400 (EDT)
From: "Neil Bliss (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6704) add support for Parascale filesystem
In-Reply-To: <6991209.115951271258268758.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Neil Bliss updated HADOOP-6704:
-------------------------------

    Attachment: HADOOP-6704.patch.2

Second attempt at patch submission.

> add support for Parascale filesystem
> ------------------------------------
>
>                 Key: HADOOP-6704
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6704
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.20.2
>            Reporter: Neil Bliss
>             Fix For: 0.20.3
>
>         Attachments: HADOOP-6704.0.20.2.patch, HADOOP-6704.patch, HADOOP-6704.patch.2
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Parascale has developed an org.apache.hadoop.fs implementation that allows users to use Hadoop on Parascale storage clusters.  We'd like to contribute this work to the community. Should this be placed under contrib, or integrated into the org.apache.hadoop.fs space?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7900-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 23:21:21 2010
Return-Path: <common-issues-return-7900-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68581 invoked from network); 30 Apr 2010 23:21:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 23:21:21 -0000
Received: (qmail 65542 invoked by uid 500); 30 Apr 2010 23:21:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65503 invoked by uid 500); 30 Apr 2010 23:21:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65495 invoked by uid 99); 30 Apr 2010 23:21:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 23:21:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 23:21:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3UNKv1Y018198
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 23:20:57 GMT
Message-ID: <17875484.44201272669657383.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 19:20:57 -0400 (EDT)
From: "Neil Bliss (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6704) add support for Parascale filesystem
In-Reply-To: <6991209.115951271258268758.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Neil Bliss updated HADOOP-6704:
-------------------------------

    Status: Open  (was: Patch Available)

> add support for Parascale filesystem
> ------------------------------------
>
>                 Key: HADOOP-6704
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6704
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.20.2
>            Reporter: Neil Bliss
>             Fix For: 0.20.3
>
>         Attachments: HADOOP-6704.0.20.2.patch, HADOOP-6704.patch
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Parascale has developed an org.apache.hadoop.fs implementation that allows users to use Hadoop on Parascale storage clusters.  We'd like to contribute this work to the community. Should this be placed under contrib, or integrated into the org.apache.hadoop.fs space?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7901-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 23:22:43 2010
Return-Path: <common-issues-return-7901-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68746 invoked from network); 30 Apr 2010 23:22:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 23:22:43 -0000
Received: (qmail 66206 invoked by uid 500); 30 Apr 2010 23:22:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66141 invoked by uid 500); 30 Apr 2010 23:22:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66133 invoked by uid 99); 30 Apr 2010 23:22:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 23:22:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 23:22:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3UNMJX9018373
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 23:22:19 GMT
Message-ID: <18555123.44421272669739588.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 19:22:19 -0400 (EDT)
From: "Neil Bliss (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6704) add support for Parascale filesystem
In-Reply-To: <6991209.115951271258268758.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Neil Bliss updated HADOOP-6704:
-------------------------------

    Attachment:     (was: HADOOP-6704.patch.2)

> add support for Parascale filesystem
> ------------------------------------
>
>                 Key: HADOOP-6704
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6704
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.20.2
>            Reporter: Neil Bliss
>             Fix For: 0.20.3
>
>         Attachments: HADOOP-6704.0.20.2.patch, HADOOP-6704.patch
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Parascale has developed an org.apache.hadoop.fs implementation that allows users to use Hadoop on Parascale storage clusters.  We'd like to contribute this work to the community. Should this be placed under contrib, or integrated into the org.apache.hadoop.fs space?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7902-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 23:22:50 2010
Return-Path: <common-issues-return-7902-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68850 invoked from network); 30 Apr 2010 23:22:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 23:22:49 -0000
Received: (qmail 66444 invoked by uid 500); 30 Apr 2010 23:22:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66403 invoked by uid 500); 30 Apr 2010 23:22:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66395 invoked by uid 99); 30 Apr 2010 23:22:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 23:22:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 23:22:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3UNMQHI018430
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 23:22:26 GMT
Message-ID: <12096695.44591272669746035.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 19:22:26 -0400 (EDT)
From: "Neil Bliss (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6704) add support for Parascale filesystem
In-Reply-To: <6991209.115951271258268758.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Neil Bliss updated HADOOP-6704:
-------------------------------

    Attachment: HADOOP-6704-2.patch

Second attempt for patch submission.

> add support for Parascale filesystem
> ------------------------------------
>
>                 Key: HADOOP-6704
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6704
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.20.2
>            Reporter: Neil Bliss
>             Fix For: 0.20.3
>
>         Attachments: HADOOP-6704-2.patch, HADOOP-6704.0.20.2.patch, HADOOP-6704.patch
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Parascale has developed an org.apache.hadoop.fs implementation that allows users to use Hadoop on Parascale storage clusters.  We'd like to contribute this work to the community. Should this be placed under contrib, or integrated into the org.apache.hadoop.fs space?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7903-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 23:52:19 2010
Return-Path: <common-issues-return-7903-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72037 invoked from network); 30 Apr 2010 23:52:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 23:52:19 -0000
Received: (qmail 84437 invoked by uid 500); 30 Apr 2010 23:52:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84403 invoked by uid 500); 30 Apr 2010 23:52:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84395 invoked by uid 99); 30 Apr 2010 23:52:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 23:52:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 23:52:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3UNptr4023354
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 23:51:55 GMT
Message-ID: <16621789.44951272671515051.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 19:51:55 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6727) Remove UnresolvedLinkException from
 public FileContext APIs
In-Reply-To: <27900682.46541272411814280.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6727:
--------------------------------

    Attachment: hadoop-6727-2.patch

Attached patch merges with trunk.

> Remove UnresolvedLinkException from public FileContext APIs
> -----------------------------------------------------------
>
>                 Key: HADOOP-6727
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6727
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6727-1.patch, hadoop-6727-2.patch
>
>
> HADOOP-6537 added UnresolvedLinkException to the throws clause and java docs of FileContext public APIs. FileContext fully resolves symbolic links, UnresolvedLinkException exception is only used internally by FSLinkResolver. I'll attach a patch which updates the APIs and javadoc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7904-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 23:54:15 2010
Return-Path: <common-issues-return-7904-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72289 invoked from network); 30 Apr 2010 23:54:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 23:54:15 -0000
Received: (qmail 84799 invoked by uid 500); 30 Apr 2010 23:54:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84760 invoked by uid 500); 30 Apr 2010 23:54:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84752 invoked by uid 99); 30 Apr 2010 23:54:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 23:54:15 +0000
X-ASF-Spam-Status: No, hits=-1375.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 23:54:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3UNrrR6023365
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 23:53:54 GMT
Message-ID: <26080676.44981272671633803.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 19:53:53 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6727) Remove UnresolvedLinkException from
 public FileContext APIs
In-Reply-To: <27900682.46541272411814280.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6727:
--------------------------------

    Status: Open  (was: Patch Available)

> Remove UnresolvedLinkException from public FileContext APIs
> -----------------------------------------------------------
>
>                 Key: HADOOP-6727
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6727
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6727-1.patch, hadoop-6727-2.patch
>
>
> HADOOP-6537 added UnresolvedLinkException to the throws clause and java docs of FileContext public APIs. FileContext fully resolves symbolic links, UnresolvedLinkException exception is only used internally by FSLinkResolver. I'll attach a patch which updates the APIs and javadoc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-7905-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Apr 30 23:54:18 2010
Return-Path: <common-issues-return-7905-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72341 invoked from network); 30 Apr 2010 23:54:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Apr 2010 23:54:18 -0000
Received: (qmail 84941 invoked by uid 500); 30 Apr 2010 23:54:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84904 invoked by uid 500); 30 Apr 2010 23:54:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84896 invoked by uid 99); 30 Apr 2010 23:54:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 23:54:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Apr 2010 23:54:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o3UNrs0A023371
	for <common-issues@hadoop.apache.org>; Fri, 30 Apr 2010 23:53:54 GMT
Message-ID: <21781391.45001272671634467.JavaMail.jira@thor>
Date: Fri, 30 Apr 2010 19:53:54 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6727) Remove UnresolvedLinkException from
 public FileContext APIs
In-Reply-To: <27900682.46541272411814280.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6727:
--------------------------------

    Status: Patch Available  (was: Open)

> Remove UnresolvedLinkException from public FileContext APIs
> -----------------------------------------------------------
>
>                 Key: HADOOP-6727
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6727
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6727-1.patch, hadoop-6727-2.patch
>
>
> HADOOP-6537 added UnresolvedLinkException to the throws clause and java docs of FileContext public APIs. FileContext fully resolves symbolic links, UnresolvedLinkException exception is only used internally by FSLinkResolver. I'll attach a patch which updates the APIs and javadoc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-606-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 01 11:10:40 2009
Return-Path: <common-issues-return-606-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47513 invoked from network); 1 Aug 2009 11:10:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 1 Aug 2009 11:10:40 -0000
Received: (qmail 38643 invoked by uid 500); 1 Aug 2009 11:10:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38608 invoked by uid 500); 1 Aug 2009 11:10:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38598 invoked by uid 99); 1 Aug 2009 11:10:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 01 Aug 2009 11:10:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 01 Aug 2009 11:10:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CD8FC234C004
	for <common-issues@hadoop.apache.org>; Sat,  1 Aug 2009 04:10:14 -0700 (PDT)
Message-ID: <1951964825.1249125014834.JavaMail.jira@brutus>
Date: Sat, 1 Aug 2009 04:10:14 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6172) bin/hadoop version not working
In-Reply-To: <816666106.1249025655012.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737864#action_12737864 ] 

Hudson commented on HADOOP-6172:
--------------------------------

Integrated in Hadoop-Common-trunk #41 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/41/])
    . Fix jar file names in hadoop-config.sh and include ${build.src} as a part of the source list in build.xml.  Contributed by Hong Tang


> bin/hadoop version not working
> ------------------------------
>
>                 Key: HADOOP-6172
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6172
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6172-20090731.patch
>
>
> Two problems found:
> - ${build.src} not included in ant target "compile-core-classes", thus o.a.h.package-info.java is not compiled, which contains the version annotation.
> - bin/hadoop-config.sh attempts to include jar files matching pattern hadoop-*-core.jar rather than hadoop-core-*.jar.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-607-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 03 05:09:32 2009
Return-Path: <common-issues-return-607-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74765 invoked from network); 3 Aug 2009 05:09:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Aug 2009 05:09:32 -0000
Received: (qmail 4434 invoked by uid 500); 3 Aug 2009 05:09:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4394 invoked by uid 500); 3 Aug 2009 05:09:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4384 invoked by uid 99); 3 Aug 2009 05:09:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Aug 2009 05:09:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Aug 2009 05:09:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CA0B9234C004
	for <common-issues@hadoop.apache.org>; Sun,  2 Aug 2009 22:09:14 -0700 (PDT)
Message-ID: <1351622178.1249276154812.JavaMail.jira@brutus>
Date: Sun, 2 Aug 2009 22:09:14 -0700 (PDT)
From: "Mahadev konar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6169) Removing deprecated method calls in
 TFile
In-Reply-To: <927826654.1248487334841.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mahadev konar updated HADOOP-6169:
----------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I just committed this. thanks hong.

> Removing deprecated method calls in TFile
> -----------------------------------------
>
>                 Key: HADOOP-6169
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6169
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6169-20090725.patch
>
>
> We should remove the use of deprecated APIs in TFile.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-608-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 03 05:15:32 2009
Return-Path: <common-issues-return-608-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75633 invoked from network); 3 Aug 2009 05:15:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Aug 2009 05:15:32 -0000
Received: (qmail 9983 invoked by uid 500); 3 Aug 2009 05:15:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9939 invoked by uid 500); 3 Aug 2009 05:15:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9928 invoked by uid 99); 3 Aug 2009 05:15:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Aug 2009 05:15:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Aug 2009 05:15:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CF709234C004
	for <common-issues@hadoop.apache.org>; Sun,  2 Aug 2009 22:15:14 -0700 (PDT)
Message-ID: <1868024677.1249276514841.JavaMail.jira@brutus>
Date: Sun, 2 Aug 2009 22:15:14 -0700 (PDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4491) Per-job local data on the TaskTracker
 node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V updated HADOOP-4491:
------------------------------

    Attachment: HADOOP-4491-20090803.txt

Uploading a new patch.  The last patch for the common project holds.

 - This patch implements the above proposal in spirit with some minor modifications.
 - Re factors localization code in TaskTracker and TaskRunner a bit so as to support writing test cases.

The patch is functionally complete, but not the cleanest. Some of the minor review comments on the linux task-controller binary code are still in progress.




> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt, HADOOP-4491-20090803.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-610-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 03 05:17:32 2009
Return-Path: <common-issues-return-610-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76651 invoked from network); 3 Aug 2009 05:17:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Aug 2009 05:17:32 -0000
Received: (qmail 12783 invoked by uid 500); 3 Aug 2009 05:17:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12733 invoked by uid 500); 3 Aug 2009 05:17:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12716 invoked by uid 99); 3 Aug 2009 05:17:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Aug 2009 05:17:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Aug 2009 05:17:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0CA52234C48D
	for <common-issues@hadoop.apache.org>; Sun,  2 Aug 2009 22:17:15 -0700 (PDT)
Message-ID: <1984004912.1249276635051.JavaMail.jira@brutus>
Date: Sun, 2 Aug 2009 22:17:15 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6105) Provide a way to automatically
 handle backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12738191#action_12738191 ] 

V.V.Chaitanya Krishna commented on HADOOP-6105:
-----------------------------------------------

Uploaded the patch with the above suggested modifications done.

> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>         Attachments: HADOOP-6105-1.patch, HADOOP-6105.patch
>
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-609-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 03 05:17:32 2009
Return-Path: <common-issues-return-609-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76644 invoked from network); 3 Aug 2009 05:17:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Aug 2009 05:17:31 -0000
Received: (qmail 12704 invoked by uid 500); 3 Aug 2009 05:17:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12657 invoked by uid 500); 3 Aug 2009 05:17:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12647 invoked by uid 99); 3 Aug 2009 05:17:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Aug 2009 05:17:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Aug 2009 05:17:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D7053234C004
	for <common-issues@hadoop.apache.org>; Sun,  2 Aug 2009 22:17:14 -0700 (PDT)
Message-ID: <1601096607.1249276634866.JavaMail.jira@brutus>
Date: Sun, 2 Aug 2009 22:17:14 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6105) Provide a way to automatically handle
 backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna updated HADOOP-6105:
------------------------------------------

    Attachment: HADOOP-6105-1.patch

> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>         Attachments: HADOOP-6105-1.patch, HADOOP-6105.patch
>
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-611-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 03 09:23:38 2009
Return-Path: <common-issues-return-611-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47840 invoked from network); 3 Aug 2009 09:23:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Aug 2009 09:23:38 -0000
Received: (qmail 10644 invoked by uid 500); 3 Aug 2009 09:23:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10593 invoked by uid 500); 3 Aug 2009 09:23:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10576 invoked by uid 99); 3 Aug 2009 09:23:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Aug 2009 09:23:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Aug 2009 09:23:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CB550234C04B
	for <common-issues@hadoop.apache.org>; Mon,  3 Aug 2009 02:23:14 -0700 (PDT)
Message-ID: <2107476875.1249291394831.JavaMail.jira@brutus>
Date: Mon, 3 Aug 2009 02:23:14 -0700 (PDT)
From: "Urko Benito (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6175) Incorret version compilation with
 es_ES.ISO8859-15 locale on Solaris 10
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Incorret version compilation with es_ES.ISO8859-15 locale on Solaris 10
-----------------------------------------------------------------------

                 Key: HADOOP-6175
                 URL: https://issues.apache.org/jira/browse/HADOOP-6175
             Project: Hadoop Common
          Issue Type: Bug
          Components: build
    Affects Versions: 0.20.0
         Environment: SunOS 5.10
Locale:
LC_CTYPE=es_ES.ISO8859-15
LC_NUMERIC=es_ES.ISO8859-15
LC_TIME=es_ES.ISO8859-15
LC_COLLATE=es_ES.ISO8859-15
LC_MONETARY=es_ES.ISO8859-15
LC_MESSAGES=es

            Reporter: Urko Benito


When you compile _hadoop_ on Solaris 10 with locale _es_ES.ISO8859-15_ the _src/saveVersion.sh_ script generates incorrect date on _build/src/org/apache/hadoop/package-info.java_

The ploblem is that the _saveVersion.sh_ script unsets the *LC_CTYPE* to avoid the problem, but on Solaris the _date_ command uses the *LC_TIME* enviroment variable as you can see at the _man page_

{noformat}
Specifications of native language translations of month  and
weekday  names  are  supported.  The month and weekday names
used for a language are based on the locale specified by the
environment variable LC_TIME. See environ(5).
{noformat}

Here's an example about *date* on Solaris

{quote}
$ echo $LC_CTYPE
es_ES.ISO8859-15
$ echo $LC_TIME
es_ES.ISO8859-15
$ date
lunes  3 de agosto de 2009 11H10'25" CEST
$ unset LC_TYPE
$ date
lunes  3 de agosto de 2009 11H10'31" CEST
$ unset LC_TIME
$ date
Mon Aug  3 11:10:35 CEST 2009
{quote}

So the _saveVersion.sh_ script creates the _package-info.java_ file as 

{code}
/*
 * Generated by src/saveVersion.sh
 */
@HadoopVersionAnnotation(version="0.20.1-dev", revision="",
                         user="itily", date="lunes  3 de agosto de 2009 11H16'01" CEST", url="http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.0")
package org.apache.hadoop;
{code}

And if you run hadoop with _version_ argument it's says "Unknow", here's an example

{quote}
$ hadoop version
Hadoop Unknown
Subversion Unknown -r Unknown
Compiled by Unknown on Unknown
{quote}


To solve this issue is as simple as adding 

*unset LC_TIME*

to _saveVersion.sh_ script, and the output is as _C_ locale as
{code}
/*
 * Generated by src/saveVersion.sh
 */
@HadoopVersionAnnotation(version="0.20.1-dev", revision="",
                         user="itily", date="Mon Aug  3 11:19:41 CEST 2009", url="http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.0")
package org.apache.hadoop;
{code}




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-612-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 03 09:29:42 2009
Return-Path: <common-issues-return-612-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50432 invoked from network); 3 Aug 2009 09:29:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Aug 2009 09:29:42 -0000
Received: (qmail 17054 invoked by uid 500); 3 Aug 2009 09:29:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17004 invoked by uid 500); 3 Aug 2009 09:29:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16994 invoked by uid 99); 3 Aug 2009 09:29:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Aug 2009 09:29:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Aug 2009 09:29:37 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 2BEDA234C046
	for <common-issues@hadoop.apache.org>; Mon,  3 Aug 2009 02:29:16 -0700 (PDT)
Message-ID: <318718178.1249291756165.JavaMail.jira@brutus>
Date: Mon, 3 Aug 2009 02:29:16 -0700 (PDT)
From: "Urko Benito (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6175) Incorret version compilation with
 es_ES.ISO8859-15 locale on Solaris 10
In-Reply-To: <2107476875.1249291394831.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Urko Benito updated HADOOP-6175:
--------------------------------

    Attachment: HADOOP-6175.patch

Here's a patch to solve the issue, adding 
unset LC_TIME 

> Incorret version compilation with es_ES.ISO8859-15 locale on Solaris 10
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-6175
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6175
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.20.0
>         Environment: SunOS 5.10
> Locale:
> LC_CTYPE=es_ES.ISO8859-15
> LC_NUMERIC=es_ES.ISO8859-15
> LC_TIME=es_ES.ISO8859-15
> LC_COLLATE=es_ES.ISO8859-15
> LC_MONETARY=es_ES.ISO8859-15
> LC_MESSAGES=es
>            Reporter: Urko Benito
>         Attachments: HADOOP-6175.patch
>
>   Original Estimate: 0.08h
>  Remaining Estimate: 0.08h
>
> When you compile _hadoop_ on Solaris 10 with locale _es_ES.ISO8859-15_ the _src/saveVersion.sh_ script generates incorrect date on _build/src/org/apache/hadoop/package-info.java_
> The ploblem is that the _saveVersion.sh_ script unsets the *LC_CTYPE* to avoid the problem, but on Solaris the _date_ command uses the *LC_TIME* enviroment variable as you can see at the _man page_
> {noformat}
> Specifications of native language translations of month  and
> weekday  names  are  supported.  The month and weekday names
> used for a language are based on the locale specified by the
> environment variable LC_TIME. See environ(5).
> {noformat}
> Here's an example about *date* on Solaris
> {quote}
> $ echo $LC_CTYPE
> es_ES.ISO8859-15
> $ echo $LC_TIME
> es_ES.ISO8859-15
> $ date
> lunes  3 de agosto de 2009 11H10'25" CEST
> $ unset LC_TYPE
> $ date
> lunes  3 de agosto de 2009 11H10'31" CEST
> $ unset LC_TIME
> $ date
> Mon Aug  3 11:10:35 CEST 2009
> {quote}
> So the _saveVersion.sh_ script creates the _package-info.java_ file as 
> {code}
> /*
>  * Generated by src/saveVersion.sh
>  */
> @HadoopVersionAnnotation(version="0.20.1-dev", revision="",
>                          user="itily", date="lunes  3 de agosto de 2009 11H16'01" CEST", url="http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.0")
> package org.apache.hadoop;
> {code}
> And if you run hadoop with _version_ argument it's says "Unknow", here's an example
> {quote}
> $ hadoop version
> Hadoop Unknown
> Subversion Unknown -r Unknown
> Compiled by Unknown on Unknown
> {quote}
> To solve this issue is as simple as adding 
> *unset LC_TIME*
> to _saveVersion.sh_ script, and the output is as _C_ locale as
> {code}
> /*
>  * Generated by src/saveVersion.sh
>  */
> @HadoopVersionAnnotation(version="0.20.1-dev", revision="",
>                          user="itily", date="Mon Aug  3 11:19:41 CEST 2009", url="http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.0")
> package org.apache.hadoop;
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-613-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 03 10:15:33 2009
Return-Path: <common-issues-return-613-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72405 invoked from network); 3 Aug 2009 10:15:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Aug 2009 10:15:32 -0000
Received: (qmail 69505 invoked by uid 500); 3 Aug 2009 10:15:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69457 invoked by uid 500); 3 Aug 2009 10:15:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69427 invoked by uid 99); 3 Aug 2009 10:15:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Aug 2009 10:15:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Aug 2009 10:15:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D6F2A234C046
	for <common-issues@hadoop.apache.org>; Mon,  3 Aug 2009 03:15:14 -0700 (PDT)
Message-ID: <1525851587.1249294514875.JavaMail.jira@brutus>
Date: Mon, 3 Aug 2009 03:15:14 -0700 (PDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4491) Per-job local data on the TaskTracker
 node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V updated HADOOP-4491:
------------------------------

    Attachment: HADOOP-4491-20090803.1.txt

Updated patch including the review comments on the task-controller binary code.

> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt, HADOOP-4491-20090803.1.txt, HADOOP-4491-20090803.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-614-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 03 12:15:32 2009
Return-Path: <common-issues-return-614-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8070 invoked from network); 3 Aug 2009 12:15:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Aug 2009 12:15:32 -0000
Received: (qmail 7432 invoked by uid 500); 3 Aug 2009 12:15:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7410 invoked by uid 500); 3 Aug 2009 12:15:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7371 invoked by uid 99); 3 Aug 2009 12:15:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Aug 2009 12:15:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Aug 2009 12:15:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D2027234C045
	for <common-issues@hadoop.apache.org>; Mon,  3 Aug 2009 05:15:14 -0700 (PDT)
Message-ID: <2137978965.1249301714845.JavaMail.jira@brutus>
Date: Mon, 3 Aug 2009 05:15:14 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6169) Removing deprecated method calls in
 TFile
In-Reply-To: <927826654.1248487334841.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12738308#action_12738308 ] 

Hudson commented on HADOOP-6169:
--------------------------------

Integrated in Hadoop-Common-trunk #43 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/43/])
    . Removing deprecated method calls in TFile. (hong tang via mahadev)


> Removing deprecated method calls in TFile
> -----------------------------------------
>
>                 Key: HADOOP-6169
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6169
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6169-20090725.patch
>
>
> We should remove the use of deprecated APIs in TFile.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-615-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 03 12:54:35 2009
Return-Path: <common-issues-return-615-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27253 invoked from network); 3 Aug 2009 12:54:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 3 Aug 2009 12:54:35 -0000
Received: (qmail 55148 invoked by uid 500); 3 Aug 2009 12:54:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55053 invoked by uid 500); 3 Aug 2009 12:54:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55029 invoked by uid 99); 3 Aug 2009 12:54:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Aug 2009 12:54:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Aug 2009 12:54:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1F2DE234C055
	for <common-issues@hadoop.apache.org>; Mon,  3 Aug 2009 05:54:15 -0700 (PDT)
Message-ID: <2032932511.1249304055126.JavaMail.jira@brutus>
Date: Mon, 3 Aug 2009 05:54:15 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5454) SortedMapWritable: readFields() will
 not clear values before deserialization
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-5454:
------------------------------

    Affects Version/s: 0.19.1
        Fix Version/s:     (was: 0.19.2)

> SortedMapWritable: readFields() will not clear values before deserialization
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5454
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5454
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.19.1
>            Reporter: Stefan Podkowinski
>
> In case SortedMapWritable is used as value in a reducer, the user must explicitly call clear() on the map between iterating values. This is because SortedMapWritable will be reused once instantiated, but consecutive calls to readFields() will not reset the maps internal state, as e.g. done by MapWritable. Please add this.instance.clear(); on top of readFields().
> You may also want to consider HADOOP-5028 for fixing another issue with this class.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-616-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 04 03:18:33 2009
Return-Path: <common-issues-return-616-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56732 invoked from network); 4 Aug 2009 03:18:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Aug 2009 03:18:32 -0000
Received: (qmail 7873 invoked by uid 500); 4 Aug 2009 03:18:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7851 invoked by uid 500); 4 Aug 2009 03:18:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7836 invoked by uid 99); 4 Aug 2009 03:18:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 03:18:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 03:18:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 328F0234C04C
	for <common-issues@hadoop.apache.org>; Mon,  3 Aug 2009 20:18:15 -0700 (PDT)
Message-ID: <359725728.1249355895206.JavaMail.jira@brutus>
Date: Mon, 3 Aug 2009 20:18:15 -0700 (PDT)
From: "Derek Morr (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-3437) mapred.job.tracker default
 value/docs appear out of sync with code
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12738784#action_12738784 ] 

Derek Morr commented on HADOOP-3437:
------------------------------------

What would it take to enable IPv6 support in Hadoop?

> mapred.job.tracker default value/docs appear out of sync with code
> ------------------------------------------------------------------
>
>                 Key: HADOOP-3437
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3437
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.18.0
>            Reporter: Steve Loughran
>            Priority: Minor
>
> This is the settings for mapred.job.tracker in SVN_HEAD. If you dont override these, you get a RuntimeException
> <property>
>   <name>mapred.job.tracker</name>
>   <value>local</value>
>   <description>The host and port that the MapReduce job tracker runs
>   at.  If "local", then jobs are run in-process as a single map
>   and reduce task.
>   </description>
> </property>

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-617-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 04 03:41:40 2009
Return-Path: <common-issues-return-617-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59850 invoked from network); 4 Aug 2009 03:41:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Aug 2009 03:41:40 -0000
Received: (qmail 19525 invoked by uid 500); 4 Aug 2009 03:41:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19470 invoked by uid 500); 4 Aug 2009 03:41:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19460 invoked by uid 99); 4 Aug 2009 03:41:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 03:41:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 03:41:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D2BC2234C045
	for <common-issues@hadoop.apache.org>; Mon,  3 Aug 2009 20:41:14 -0700 (PDT)
Message-ID: <1057391574.1249357274861.JavaMail.jira@brutus>
Date: Mon, 3 Aug 2009 20:41:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6124) patchJavacWarnings and
 trunkJavacWarnings are not consistent.
In-Reply-To: <1339799848.1246409207795.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12738791#action_12738791 ] 

Giridharan Kesavan commented on HADOOP-6124:
--------------------------------------------

     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     -1 javac.  The applied patch generated 100 javac compiler warnings (more than the trunk's current 99 warnings).
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.

applied the hadoop-6124.patch and tested a hadoop patch that introduced a javac warning. Here is the result. 
test-patch was able to parse and show the increase in the no of javac warnings.



> patchJavacWarnings and trunkJavacWarnings are not consistent.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6124
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6124
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Giridharan Kesavan
>            Priority: Critical
>         Attachments: c6124_20090701.patch, HADOOP-6124.patch
>
>
> The values of patchJavacWarnings and trunkJavacWarnings are not consistent when running test-patch.sh with an empty patch over Common.  HDFS and MapReduce seem not having this problem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-618-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 04 09:29:40 2009
Return-Path: <common-issues-return-618-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49313 invoked from network); 4 Aug 2009 09:29:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Aug 2009 09:29:40 -0000
Received: (qmail 41351 invoked by uid 500); 4 Aug 2009 09:29:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41293 invoked by uid 500); 4 Aug 2009 09:29:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41270 invoked by uid 99); 4 Aug 2009 09:29:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 09:29:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 09:29:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CAD37234C045
	for <common-issues@hadoop.apache.org>; Tue,  4 Aug 2009 02:29:14 -0700 (PDT)
Message-ID: <64682192.1249378154816.JavaMail.jira@brutus>
Date: Tue, 4 Aug 2009 02:29:14 -0700 (PDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4491) Per-job local data on the
 TaskTracker node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12738914#action_12738914 ] 

Hemanth Yamijala commented on HADOOP-4491:
------------------------------------------

This is looking much simpler and good. Some comments, mostly minor:

- When mapred.local.dir is changed in TaskRunner for sandboxing the child, it affects the JobConf instance in the TaskInProgress. This seems error prone. Can we check if this can be moved to the Child directly.
- Add a comment saying why permissions are being set for the job and task directories at localization time itself, as opposed to inside the task-controller executable. This is mainly to narrow down the possibility of misuse, until we do a change of the group ownership.
- In the same line, set permissions to tasklog attempt directory to restrictive permissions.
- Add a comment to explain that split.dta is only used in the case of IsolationRunner where the changes are made in IsolationRunner.
- We are still passing the log directory to the task-controller - doesn't seem needed.
- I would suggest InitializeJobContext to be named as JobInitializationContext; doJobLocalization to localizeJobFiles and likewise for tasks.

I have still to look at the task-controller changes, will take the latest patch for the same.

> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt, HADOOP-4491-20090803.1.txt, HADOOP-4491-20090803.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-619-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 04 09:33:32 2009
Return-Path: <common-issues-return-619-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52267 invoked from network); 4 Aug 2009 09:33:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Aug 2009 09:33:31 -0000
Received: (qmail 47977 invoked by uid 500); 4 Aug 2009 09:33:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47951 invoked by uid 500); 4 Aug 2009 09:33:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47941 invoked by uid 99); 4 Aug 2009 09:33:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 09:33:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 09:33:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D1B02234C045
	for <common-issues@hadoop.apache.org>; Tue,  4 Aug 2009 02:33:14 -0700 (PDT)
Message-ID: <602602646.1249378394844.JavaMail.jira@brutus>
Date: Tue, 4 Aug 2009 02:33:14 -0700 (PDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4491) Per-job local data on the
 TaskTracker node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12738915#action_12738915 ] 

Hemanth Yamijala commented on HADOOP-4491:
------------------------------------------

One more point. I noticed that ReduceTask.ReduceCopier.configureClasspath() is using Task.getJobFile(). Will this be a problem given we are setting the value of the job file just before it is written to disk ?

> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt, HADOOP-4491-20090803.1.txt, HADOOP-4491-20090803.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-620-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 04 10:19:32 2009
Return-Path: <common-issues-return-620-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77257 invoked from network); 4 Aug 2009 10:19:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Aug 2009 10:19:32 -0000
Received: (qmail 3558 invoked by uid 500); 4 Aug 2009 10:19:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3515 invoked by uid 500); 4 Aug 2009 10:19:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3502 invoked by uid 99); 4 Aug 2009 10:19:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 10:19:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 10:19:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DB3FC234C045
	for <common-issues@hadoop.apache.org>; Tue,  4 Aug 2009 03:19:14 -0700 (PDT)
Message-ID: <1360712672.1249381154882.JavaMail.jira@brutus>
Date: Tue, 4 Aug 2009 03:19:14 -0700 (PDT)
From: "Abdul Qadeer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Work started: (HADOOP-4012) Providing splitting support for
 bzip2 compressed files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Work on HADOOP-4012 started by Abdul Qadeer.

> Providing splitting support for bzip2 compressed files
> ------------------------------------------------------
>
>                 Key: HADOOP-4012
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4012
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Abdul Qadeer
>            Assignee: Abdul Qadeer
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-4012-version1.patch, Hadoop-4012-version2.patch, Hadoop-4012-version3.patch, Hadoop-4012-version4.patch, Hadoop-4012-version5.patch, Hadoop-4012-version6.patch, Hadoop-4012-version7.patch, Hadoop-4012-version8.patch, Hadoop-4012-version9.patch
>
>
> Hadoop assumes that if the input data is compressed, it can not be split (mainly due to the limitation of many codecs that they need the whole input stream to decompress successfully).  So in such a case, Hadoop prepares only one split per compressed file, where the lower split limit is at 0 while the upper limit is the end of the file.  The consequence of this decision is that, one compress file goes to a single mapper. Although it circumvents the limitation of codecs (as mentioned above) but reduces the parallelism substantially, as it was possible otherwise in case of splitting.
> BZip2 is a compression / De-Compression algorithm which does compression on blocks of data and later these compressed blocks can be decompressed independent of each other.  This is indeed an opportunity that instead of one BZip2 compressed file going to one mapper, we can process chunks of file in parallel.  The correctness criteria of such a processing is that for a bzip2 compressed file, each compressed block should be processed by only one mapper and ultimately all the blocks of the file should be processed.  (By processing we mean the actual utilization of that un-compressed data (coming out of the codecs) in a mapper).
> We are writing the code to implement this suggested functionality.  Although we have used bzip2 as an example, but we have tried to extend Hadoop's compression interfaces so that any other codecs with the same capability as that of bzip2, could easily use the splitting support.  The details of these changes will be posted when we submit the code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-621-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 04 10:25:34 2009
Return-Path: <common-issues-return-621-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80880 invoked from network); 4 Aug 2009 10:25:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Aug 2009 10:25:34 -0000
Received: (qmail 9023 invoked by uid 500); 4 Aug 2009 10:25:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8976 invoked by uid 500); 4 Aug 2009 10:25:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8966 invoked by uid 99); 4 Aug 2009 10:25:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 10:25:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 10:25:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E140E234C045
	for <common-issues@hadoop.apache.org>; Tue,  4 Aug 2009 03:25:14 -0700 (PDT)
Message-ID: <1205348729.1249381514908.JavaMail.jira@brutus>
Date: Tue, 4 Aug 2009 03:25:14 -0700 (PDT)
From: "Abdul Qadeer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4012) Providing splitting support for bzip2
 compressed files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Abdul Qadeer updated HADOOP-4012:
---------------------------------

    Attachment: Hadoop-4012-version10.patch

The older patch has been split into two (due to the new Hadoop sub-projects: common, mapreduce and HDFS).  This patch has the work related to common while the portion going into MapReduce will be added soon on a separate MapReduce JIRA.

> Providing splitting support for bzip2 compressed files
> ------------------------------------------------------
>
>                 Key: HADOOP-4012
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4012
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Abdul Qadeer
>            Assignee: Abdul Qadeer
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-4012-version1.patch, Hadoop-4012-version10.patch, Hadoop-4012-version2.patch, Hadoop-4012-version3.patch, Hadoop-4012-version4.patch, Hadoop-4012-version5.patch, Hadoop-4012-version6.patch, Hadoop-4012-version7.patch, Hadoop-4012-version8.patch, Hadoop-4012-version9.patch
>
>
> Hadoop assumes that if the input data is compressed, it can not be split (mainly due to the limitation of many codecs that they need the whole input stream to decompress successfully).  So in such a case, Hadoop prepares only one split per compressed file, where the lower split limit is at 0 while the upper limit is the end of the file.  The consequence of this decision is that, one compress file goes to a single mapper. Although it circumvents the limitation of codecs (as mentioned above) but reduces the parallelism substantially, as it was possible otherwise in case of splitting.
> BZip2 is a compression / De-Compression algorithm which does compression on blocks of data and later these compressed blocks can be decompressed independent of each other.  This is indeed an opportunity that instead of one BZip2 compressed file going to one mapper, we can process chunks of file in parallel.  The correctness criteria of such a processing is that for a bzip2 compressed file, each compressed block should be processed by only one mapper and ultimately all the blocks of the file should be processed.  (By processing we mean the actual utilization of that un-compressed data (coming out of the codecs) in a mapper).
> We are writing the code to implement this suggested functionality.  Although we have used bzip2 as an example, but we have tried to extend Hadoop's compression interfaces so that any other codecs with the same capability as that of bzip2, could easily use the splitting support.  The details of these changes will be posted when we submit the code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-622-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 04 10:25:35 2009
Return-Path: <common-issues-return-622-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80941 invoked from network); 4 Aug 2009 10:25:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Aug 2009 10:25:34 -0000
Received: (qmail 9111 invoked by uid 500); 4 Aug 2009 10:25:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9058 invoked by uid 500); 4 Aug 2009 10:25:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9043 invoked by uid 99); 4 Aug 2009 10:25:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 10:25:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 10:25:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 8B6B8234C4B2
	for <common-issues@hadoop.apache.org>; Tue,  4 Aug 2009 03:25:15 -0700 (PDT)
Message-ID: <1349496213.1249381515570.JavaMail.jira@brutus>
Date: Tue, 4 Aug 2009 03:25:15 -0700 (PDT)
From: "Abdul Qadeer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4012) Providing splitting support for bzip2
 compressed files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Abdul Qadeer updated HADOOP-4012:
---------------------------------

    Status: Patch Available  (was: In Progress)

> Providing splitting support for bzip2 compressed files
> ------------------------------------------------------
>
>                 Key: HADOOP-4012
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4012
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Abdul Qadeer
>            Assignee: Abdul Qadeer
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-4012-version1.patch, Hadoop-4012-version10.patch, Hadoop-4012-version2.patch, Hadoop-4012-version3.patch, Hadoop-4012-version4.patch, Hadoop-4012-version5.patch, Hadoop-4012-version6.patch, Hadoop-4012-version7.patch, Hadoop-4012-version8.patch, Hadoop-4012-version9.patch
>
>
> Hadoop assumes that if the input data is compressed, it can not be split (mainly due to the limitation of many codecs that they need the whole input stream to decompress successfully).  So in such a case, Hadoop prepares only one split per compressed file, where the lower split limit is at 0 while the upper limit is the end of the file.  The consequence of this decision is that, one compress file goes to a single mapper. Although it circumvents the limitation of codecs (as mentioned above) but reduces the parallelism substantially, as it was possible otherwise in case of splitting.
> BZip2 is a compression / De-Compression algorithm which does compression on blocks of data and later these compressed blocks can be decompressed independent of each other.  This is indeed an opportunity that instead of one BZip2 compressed file going to one mapper, we can process chunks of file in parallel.  The correctness criteria of such a processing is that for a bzip2 compressed file, each compressed block should be processed by only one mapper and ultimately all the blocks of the file should be processed.  (By processing we mean the actual utilization of that un-compressed data (coming out of the codecs) in a mapper).
> We are writing the code to implement this suggested functionality.  Although we have used bzip2 as an example, but we have tried to extend Hadoop's compression interfaces so that any other codecs with the same capability as that of bzip2, could easily use the splitting support.  The details of these changes will be posted when we submit the code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-623-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 04 18:46:39 2009
Return-Path: <common-issues-return-623-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 523 invoked from network); 4 Aug 2009 18:46:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Aug 2009 18:46:39 -0000
Received: (qmail 55160 invoked by uid 500); 4 Aug 2009 18:46:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55109 invoked by uid 500); 4 Aug 2009 18:46:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55092 invoked by uid 99); 4 Aug 2009 18:46:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 18:46:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 18:46:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CAD6C234C045
	for <common-issues@hadoop.apache.org>; Tue,  4 Aug 2009 11:46:14 -0700 (PDT)
Message-ID: <1346564994.1249411574816.JavaMail.jira@brutus>
Date: Tue, 4 Aug 2009 11:46:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6124) patchJavacWarnings and
 trunkJavacWarnings are not consistent.
In-Reply-To: <1339799848.1246409207795.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6124:
-------------------------------------------

    Hadoop Flags: [Reviewed]

If the patch works for you, it may work for Hudson.  Let's give it a try.

+1 Patch looks good.

> patchJavacWarnings and trunkJavacWarnings are not consistent.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6124
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6124
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Giridharan Kesavan
>            Priority: Critical
>         Attachments: c6124_20090701.patch, HADOOP-6124.patch
>
>
> The values of patchJavacWarnings and trunkJavacWarnings are not consistent when running test-patch.sh with an empty patch over Common.  HDFS and MapReduce seem not having this problem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-624-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 04 18:50:40 2009
Return-Path: <common-issues-return-624-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3519 invoked from network); 4 Aug 2009 18:50:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Aug 2009 18:50:40 -0000
Received: (qmail 71466 invoked by uid 500); 4 Aug 2009 18:50:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71438 invoked by uid 500); 4 Aug 2009 18:50:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71423 invoked by uid 99); 4 Aug 2009 18:50:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 18:50:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 18:50:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C7FB9234C044
	for <common-issues@hadoop.apache.org>; Tue,  4 Aug 2009 11:50:14 -0700 (PDT)
Message-ID: <496903002.1249411814805.JavaMail.jira@brutus>
Date: Tue, 4 Aug 2009 11:50:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6124) patchJavacWarnings and
 trunkJavacWarnings are not consistent.
In-Reply-To: <1339799848.1246409207795.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12739090#action_12739090 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6124:
------------------------------------------------

Script testing is not yet supported (see HADOOP-6174).  So I am going to commit this without submitting to Hudson.

> patchJavacWarnings and trunkJavacWarnings are not consistent.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6124
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6124
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Giridharan Kesavan
>            Priority: Critical
>         Attachments: c6124_20090701.patch, HADOOP-6124.patch
>
>
> The values of patchJavacWarnings and trunkJavacWarnings are not consistent when running test-patch.sh with an empty patch over Common.  HDFS and MapReduce seem not having this problem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-625-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 04 19:56:41 2009
Return-Path: <common-issues-return-625-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32364 invoked from network); 4 Aug 2009 19:56:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Aug 2009 19:56:41 -0000
Received: (qmail 90959 invoked by uid 500); 4 Aug 2009 19:56:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90917 invoked by uid 500); 4 Aug 2009 19:56:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90907 invoked by uid 99); 4 Aug 2009 19:56:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 19:56:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 19:56:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id BCE0729A001C
	for <common-issues@hadoop.apache.org>; Tue,  4 Aug 2009 12:56:15 -0700 (PDT)
Message-ID: <908399066.1249415775772.JavaMail.jira@brutus>
Date: Tue, 4 Aug 2009 12:56:15 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6124) patchJavacWarnings and
 trunkJavacWarnings are not consistent.
In-Reply-To: <1339799848.1246409207795.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE resolved HADOOP-6124.
--------------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.21.0

I have committed this.  Thanks, Giridharan!

> patchJavacWarnings and trunkJavacWarnings are not consistent.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6124
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6124
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Giridharan Kesavan
>            Priority: Critical
>             Fix For: 0.21.0
>
>         Attachments: c6124_20090701.patch, HADOOP-6124.patch
>
>
> The values of patchJavacWarnings and trunkJavacWarnings are not consistent when running test-patch.sh with an empty patch over Common.  HDFS and MapReduce seem not having this problem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-626-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 04 21:20:35 2009
Return-Path: <common-issues-return-626-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70373 invoked from network); 4 Aug 2009 21:20:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Aug 2009 21:20:35 -0000
Received: (qmail 34722 invoked by uid 500); 4 Aug 2009 21:20:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34612 invoked by uid 500); 4 Aug 2009 21:20:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34561 invoked by uid 99); 4 Aug 2009 21:20:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 21:20:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 21:20:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0E2E629A0025
	for <common-issues@hadoop.apache.org>; Tue,  4 Aug 2009 14:20:15 -0700 (PDT)
Message-ID: <1874039112.1249420815057.JavaMail.jira@brutus>
Date: Tue, 4 Aug 2009 14:20:15 -0700 (PDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6176) Adding a couple private methods to
 AccessTokenHandler for testing puposes
In-Reply-To: <1121825252.1249420814969.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6176:
------------------------------

    Component/s: security

> Adding a couple private methods to AccessTokenHandler for testing puposes
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6176
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6176
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>
> To support some test cases being added as part of HDFS-409, a couple private methods need to be added to AccessTokenHandler. One is for setting token lifetime and another is checking if a token has expired.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-627-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 04 21:20:38 2009
Return-Path: <common-issues-return-627-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70786 invoked from network); 4 Aug 2009 21:20:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Aug 2009 21:20:38 -0000
Received: (qmail 35396 invoked by uid 500); 4 Aug 2009 21:20:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35272 invoked by uid 500); 4 Aug 2009 21:20:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35046 invoked by uid 99); 4 Aug 2009 21:20:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 21:20:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 21:20:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id ECD3529A001D
	for <common-issues@hadoop.apache.org>; Tue,  4 Aug 2009 14:20:14 -0700 (PDT)
Message-ID: <1121825252.1249420814969.JavaMail.jira@brutus>
Date: Tue, 4 Aug 2009 14:20:14 -0700 (PDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6176) Adding a couple private methods to
 AccessTokenHandler for testing puposes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Adding a couple private methods to AccessTokenHandler for testing puposes
-------------------------------------------------------------------------

                 Key: HADOOP-6176
                 URL: https://issues.apache.org/jira/browse/HADOOP-6176
             Project: Hadoop Common
          Issue Type: Test
            Reporter: Kan Zhang
            Assignee: Kan Zhang


To support some test cases being added as part of HDFS-409, a couple private methods need to be added to AccessTokenHandler. One is for setting token lifetime and another is checking if a token has expired.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-628-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 04 21:29:30 2009
Return-Path: <common-issues-return-628-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74854 invoked from network); 4 Aug 2009 21:29:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Aug 2009 21:29:29 -0000
Received: (qmail 44482 invoked by uid 500); 4 Aug 2009 21:29:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44433 invoked by uid 500); 4 Aug 2009 21:29:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44423 invoked by uid 99); 4 Aug 2009 21:29:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 21:29:34 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 21:29:31 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D05B329A001C
	for <common-issues@hadoop.apache.org>; Tue,  4 Aug 2009 14:29:10 -0700 (PDT)
Message-ID: <1512533732.1249421350852.JavaMail.jira@brutus>
Date: Tue, 4 Aug 2009 14:29:10 -0700 (PDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6176) Adding a couple private methods to
 AccessTokenHandler for testing puposes
In-Reply-To: <1121825252.1249420814969.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6176:
------------------------------

    Attachment: core-03.patch

> Adding a couple private methods to AccessTokenHandler for testing puposes
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6176
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6176
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: core-03.patch
>
>
> To support some test cases being added as part of HDFS-409, a couple private methods need to be added to AccessTokenHandler. One is for setting token lifetime and another is checking if a token has expired.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-629-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 04 21:50:34 2009
Return-Path: <common-issues-return-629-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89082 invoked from network); 4 Aug 2009 21:50:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 4 Aug 2009 21:50:34 -0000
Received: (qmail 70857 invoked by uid 500); 4 Aug 2009 21:50:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70812 invoked by uid 500); 4 Aug 2009 21:50:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70802 invoked by uid 99); 4 Aug 2009 21:50:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 21:50:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 21:50:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C693A29A0013
	for <common-issues@hadoop.apache.org>; Tue,  4 Aug 2009 14:50:14 -0700 (PDT)
Message-ID: <366394486.1249422614799.JavaMail.jira@brutus>
Date: Tue, 4 Aug 2009 14:50:14 -0700 (PDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6176) Adding a couple private methods to
 AccessTokenHandler for testing puposes
In-Reply-To: <1121825252.1249420814969.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6176:
------------------------------

    Status: Patch Available  (was: Open)

> Adding a couple private methods to AccessTokenHandler for testing puposes
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6176
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6176
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: core-03.patch
>
>
> To support some test cases being added as part of HDFS-409, a couple private methods need to be added to AccessTokenHandler. One is for setting token lifetime and another is checking if a token has expired.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-630-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 05 09:37:30 2009
Return-Path: <common-issues-return-630-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3685 invoked from network); 5 Aug 2009 09:37:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Aug 2009 09:37:29 -0000
Received: (qmail 19247 invoked by uid 500); 5 Aug 2009 09:37:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19199 invoked by uid 500); 5 Aug 2009 09:37:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19178 invoked by uid 99); 5 Aug 2009 09:37:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Aug 2009 09:37:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Aug 2009 09:37:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D6C15234C045
	for <common-issues@hadoop.apache.org>; Wed,  5 Aug 2009 02:37:14 -0700 (PDT)
Message-ID: <761170096.1249465034865.JavaMail.jira@brutus>
Date: Wed, 5 Aug 2009 02:37:14 -0700 (PDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4491) Per-job local data on the
 TaskTracker node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12739393#action_12739393 ] 

Hemanth Yamijala commented on HADOOP-4491:
------------------------------------------

Some comments on the task-controller code:

- Code in run_task_as_user can reuse code from initialize_task
- I think test-task-controller can be moved to a test folder following convention from other C++ modules.
- The log file for the taskcontroller in open_log_file should be closed if we can't change permissions to it.
- When there's an error in initializing job for any of the m/r local dirs, we should free the job_dir parameter, right ? Similarly in initializing task.
- Task log dir for cleanup tasks will not have the name task-attempt-id.cleanup. This seems fine right now. Because we check if the directory exists and return if it doesn't. Can we comment this ?
- Variable jars_dir not being used in run_task_as_user.
- job_dir needs to be freed before execlp


> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt, HADOOP-4491-20090803.1.txt, HADOOP-4491-20090803.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-631-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 05 10:51:30 2009
Return-Path: <common-issues-return-631-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39218 invoked from network); 5 Aug 2009 10:51:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Aug 2009 10:51:30 -0000
Received: (qmail 25917 invoked by uid 500); 5 Aug 2009 10:51:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25888 invoked by uid 500); 5 Aug 2009 10:51:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25661 invoked by uid 99); 5 Aug 2009 10:51:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Aug 2009 10:51:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Aug 2009 10:51:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 424D8234C045
	for <common-issues@hadoop.apache.org>; Wed,  5 Aug 2009 03:51:15 -0700 (PDT)
Message-ID: <798924081.1249469475270.JavaMail.jira@brutus>
Date: Wed, 5 Aug 2009 03:51:15 -0700 (PDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4491) Per-job local data on the
 TaskTracker node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12739425#action_12739425 ] 

Hemanth Yamijala commented on HADOOP-4491:
------------------------------------------

I missed out one more point. In initialize_job we should be setting the permissions as 2570 recursively. Only the work directory needs to open up write for the user.

> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt, HADOOP-4491-20090803.1.txt, HADOOP-4491-20090803.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-632-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 05 11:10:39 2009
Return-Path: <common-issues-return-632-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45743 invoked from network); 5 Aug 2009 11:10:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Aug 2009 11:10:38 -0000
Received: (qmail 46590 invoked by uid 500); 5 Aug 2009 11:10:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46542 invoked by uid 500); 5 Aug 2009 11:10:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46532 invoked by uid 99); 5 Aug 2009 11:10:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Aug 2009 11:10:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Aug 2009 11:10:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CF60E234C044
	for <common-issues@hadoop.apache.org>; Wed,  5 Aug 2009 04:10:14 -0700 (PDT)
Message-ID: <806486421.1249470614834.JavaMail.jira@brutus>
Date: Wed, 5 Aug 2009 04:10:14 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6124) patchJavacWarnings and
 trunkJavacWarnings are not consistent.
In-Reply-To: <1339799848.1246409207795.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12739431#action_12739431 ] 

Hudson commented on HADOOP-6124:
--------------------------------

Integrated in Hadoop-Common-trunk #45 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/45/])
    . Fix javac warning detection in test-patch.sh.  Contributed by Giridharan Kesavan


> patchJavacWarnings and trunkJavacWarnings are not consistent.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6124
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6124
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Giridharan Kesavan
>            Priority: Critical
>             Fix For: 0.21.0
>
>         Attachments: c6124_20090701.patch, HADOOP-6124.patch
>
>
> The values of patchJavacWarnings and trunkJavacWarnings are not consistent when running test-patch.sh with an empty patch over Common.  HDFS and MapReduce seem not having this problem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-633-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 05 12:38:39 2009
Return-Path: <common-issues-return-633-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84419 invoked from network); 5 Aug 2009 12:38:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Aug 2009 12:38:38 -0000
Received: (qmail 52227 invoked by uid 500); 5 Aug 2009 12:38:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52177 invoked by uid 500); 5 Aug 2009 12:38:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52167 invoked by uid 99); 5 Aug 2009 12:38:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Aug 2009 12:38:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Aug 2009 12:38:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 45769234C1E6
	for <common-issues@hadoop.apache.org>; Wed,  5 Aug 2009 05:38:15 -0700 (PDT)
Message-ID: <1906954071.1249475895283.JavaMail.jira@brutus>
Date: Wed, 5 Aug 2009 05:38:15 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6124) patchJavacWarnings and
 trunkJavacWarnings are not consistent.
In-Reply-To: <1339799848.1246409207795.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12739461#action_12739461 ] 

Hudson commented on HADOOP-6124:
--------------------------------

Integrated in Hadoop-Hdfs-trunk #40 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/40/])
    . Fix javac warning detection in test-patch.sh.  Contributed by Giridharan Kesavan


> patchJavacWarnings and trunkJavacWarnings are not consistent.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6124
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6124
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Giridharan Kesavan
>            Priority: Critical
>             Fix For: 0.21.0
>
>         Attachments: c6124_20090701.patch, HADOOP-6124.patch
>
>
> The values of patchJavacWarnings and trunkJavacWarnings are not consistent when running test-patch.sh with an empty patch over Common.  HDFS and MapReduce seem not having this problem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-634-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 05 15:58:36 2009
Return-Path: <common-issues-return-634-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87529 invoked from network); 5 Aug 2009 15:58:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Aug 2009 15:58:36 -0000
Received: (qmail 18713 invoked by uid 500); 5 Aug 2009 15:58:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18659 invoked by uid 500); 5 Aug 2009 15:58:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18649 invoked by uid 99); 5 Aug 2009 15:58:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Aug 2009 15:58:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Aug 2009 15:58:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C30E5234C044
	for <common-issues@hadoop.apache.org>; Wed,  5 Aug 2009 08:58:14 -0700 (PDT)
Message-ID: <1137883578.1249487894794.JavaMail.jira@brutus>
Date: Wed, 5 Aug 2009 08:58:14 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6124) patchJavacWarnings and
 trunkJavacWarnings are not consistent.
In-Reply-To: <1339799848.1246409207795.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12739574#action_12739574 ] 

Hudson commented on HADOOP-6124:
--------------------------------

Integrated in Hadoop-Mapreduce-trunk #39 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/39/])
    . Fix javac warning detection in test-patch.sh.  Contributed by Giridharan Kesavan


> patchJavacWarnings and trunkJavacWarnings are not consistent.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6124
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6124
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Giridharan Kesavan
>            Priority: Critical
>             Fix For: 0.21.0
>
>         Attachments: c6124_20090701.patch, HADOOP-6124.patch
>
>
> The values of patchJavacWarnings and trunkJavacWarnings are not consistent when running test-patch.sh with an empty patch over Common.  HDFS and MapReduce seem not having this problem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-635-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 05 21:15:30 2009
Return-Path: <common-issues-return-635-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29510 invoked from network); 5 Aug 2009 21:15:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Aug 2009 21:15:30 -0000
Received: (qmail 6192 invoked by uid 500); 5 Aug 2009 21:15:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6124 invoked by uid 500); 5 Aug 2009 21:15:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5803 invoked by uid 99); 5 Aug 2009 21:15:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Aug 2009 21:15:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Aug 2009 21:15:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DFAA729A001D
	for <common-issues@hadoop.apache.org>; Wed,  5 Aug 2009 14:15:14 -0700 (PDT)
Message-ID: <117574976.1249506914915.JavaMail.jira@brutus>
Date: Wed, 5 Aug 2009 14:15:14 -0700 (PDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6176) Adding a couple private methods to
 AccessTokenHandler for testing purposes
In-Reply-To: <1121825252.1249420814969.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6176:
------------------------------

    Summary: Adding a couple private methods to AccessTokenHandler for testing purposes  (was: Adding a couple private methods to AccessTokenHandler for testing puposes)

> Adding a couple private methods to AccessTokenHandler for testing purposes
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6176
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6176
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: core-03.patch
>
>
> To support some test cases being added as part of HDFS-409, a couple private methods need to be added to AccessTokenHandler. One is for setting token lifetime and another is checking if a token has expired.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-636-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 05 21:42:30 2009
Return-Path: <common-issues-return-636-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44462 invoked from network); 5 Aug 2009 21:42:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 5 Aug 2009 21:42:29 -0000
Received: (qmail 42667 invoked by uid 500); 5 Aug 2009 21:42:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42614 invoked by uid 500); 5 Aug 2009 21:42:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42510 invoked by uid 99); 5 Aug 2009 21:42:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Aug 2009 21:42:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Aug 2009 21:42:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E6FA729A0017
	for <common-issues@hadoop.apache.org>; Wed,  5 Aug 2009 14:42:14 -0700 (PDT)
Message-ID: <688375146.1249508534945.JavaMail.jira@brutus>
Date: Wed, 5 Aug 2009 14:42:14 -0700 (PDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5073) Hadoop 1.0 Interface Classification
 - scope (visibility - public/private) and stability
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12739768#action_12739768 ] 

Sanjay Radia commented on HADOOP-5073:
--------------------------------------

Most of org.apache.hadoop.security should be 

Limited-Private Evolving
    * Security (HDFS-MapReduce Private)

We are considering moving to using Jaas more extensively; if we do that some of these interfaces (esp UserGroupInformation) will change.


> Hadoop 1.0 Interface Classification - scope (visibility - public/private) and stability
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5073
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5073
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>
> This jira proposes an interface classification for hadoop interfaces.
> The discussion was started in email alias core-dev@hadoop.apache.org in Nov 2008.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-637-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 06 05:57:32 2009
Return-Path: <common-issues-return-637-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96609 invoked from network); 6 Aug 2009 05:57:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Aug 2009 05:57:32 -0000
Received: (qmail 31859 invoked by uid 500); 6 Aug 2009 05:57:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31805 invoked by uid 500); 6 Aug 2009 05:57:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31795 invoked by uid 99); 6 Aug 2009 05:57:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Aug 2009 05:57:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Aug 2009 05:57:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E521329A0011
	for <common-issues@hadoop.apache.org>; Wed,  5 Aug 2009 22:57:14 -0700 (PDT)
Message-ID: <688231599.1249538234924.JavaMail.jira@brutus>
Date: Wed, 5 Aug 2009 22:57:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4012) Providing splitting support for
 bzip2 compressed files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12739906#action_12739906 ] 

Hadoop QA commented on HADOOP-4012:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12415479/Hadoop-4012-version10.patch
  against trunk revision 800919.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    -1 javac.  The applied patch generated 118 javac compiler warnings (more than the trunk's current 116 warnings).

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/591/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/591/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/591/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/591/console

This message is automatically generated.

> Providing splitting support for bzip2 compressed files
> ------------------------------------------------------
>
>                 Key: HADOOP-4012
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4012
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Abdul Qadeer
>            Assignee: Abdul Qadeer
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-4012-version1.patch, Hadoop-4012-version10.patch, Hadoop-4012-version2.patch, Hadoop-4012-version3.patch, Hadoop-4012-version4.patch, Hadoop-4012-version5.patch, Hadoop-4012-version6.patch, Hadoop-4012-version7.patch, Hadoop-4012-version8.patch, Hadoop-4012-version9.patch
>
>
> Hadoop assumes that if the input data is compressed, it can not be split (mainly due to the limitation of many codecs that they need the whole input stream to decompress successfully).  So in such a case, Hadoop prepares only one split per compressed file, where the lower split limit is at 0 while the upper limit is the end of the file.  The consequence of this decision is that, one compress file goes to a single mapper. Although it circumvents the limitation of codecs (as mentioned above) but reduces the parallelism substantially, as it was possible otherwise in case of splitting.
> BZip2 is a compression / De-Compression algorithm which does compression on blocks of data and later these compressed blocks can be decompressed independent of each other.  This is indeed an opportunity that instead of one BZip2 compressed file going to one mapper, we can process chunks of file in parallel.  The correctness criteria of such a processing is that for a bzip2 compressed file, each compressed block should be processed by only one mapper and ultimately all the blocks of the file should be processed.  (By processing we mean the actual utilization of that un-compressed data (coming out of the codecs) in a mapper).
> We are writing the code to implement this suggested functionality.  Although we have used bzip2 as an example, but we have tried to extend Hadoop's compression interfaces so that any other codecs with the same capability as that of bzip2, could easily use the splitting support.  The details of these changes will be posted when we submit the code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-638-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 06 06:49:30 2009
Return-Path: <common-issues-return-638-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22883 invoked from network); 6 Aug 2009 06:49:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Aug 2009 06:49:30 -0000
Received: (qmail 63973 invoked by uid 500); 6 Aug 2009 06:49:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63928 invoked by uid 500); 6 Aug 2009 06:49:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63914 invoked by uid 99); 6 Aug 2009 06:49:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Aug 2009 06:49:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Aug 2009 06:49:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CDE6829A0011
	for <common-issues@hadoop.apache.org>; Wed,  5 Aug 2009 23:49:14 -0700 (PDT)
Message-ID: <1917932216.1249541354828.JavaMail.jira@brutus>
Date: Wed, 5 Aug 2009 23:49:14 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6105) Provide a way to automatically handle
 backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna updated HADOOP-6105:
------------------------------------------

    Attachment: HADOOP-6105.patch

Some changes made in the new patch are:

- method to check if the key is deprecated.
- method to check if the key is previously loaded with the value of the corresponding deprecated key.
- method to set value of deprecated key to the new keys ( when deprecated key occurs in xm files).
- method to generate appropriate warning message that needs to be logged when deprecated key is being considered.

> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>         Attachments: HADOOP-6105-1.patch, HADOOP-6105.patch, HADOOP-6105.patch
>
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-639-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 06 08:05:29 2009
Return-Path: <common-issues-return-639-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39436 invoked from network); 6 Aug 2009 08:05:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Aug 2009 08:05:29 -0000
Received: (qmail 22589 invoked by uid 500); 6 Aug 2009 08:05:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22554 invoked by uid 500); 6 Aug 2009 08:05:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22544 invoked by uid 99); 6 Aug 2009 08:05:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Aug 2009 08:05:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Aug 2009 08:05:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E5519234C045
	for <common-issues@hadoop.apache.org>; Thu,  6 Aug 2009 01:05:14 -0700 (PDT)
Message-ID: <681828712.1249545914938.JavaMail.jira@brutus>
Date: Thu, 6 Aug 2009 01:05:14 -0700 (PDT)
From: "Sreekanth Ramakrishnan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6105) Provide a way to automatically
 handle backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12739933#action_12739933 ] 

Sreekanth Ramakrishnan commented on HADOOP-6105:
------------------------------------------------

Took a look at the patch file:
The code changes looks fine. Following are the comments with respect to test case:

In testcase do assert that get of the old and new key matches correctly. We just need to do the following for these cases.
Old final new final --- Old. 
old non-final new final --- New
old final new non-final -- Old 
old non-final new non-final -- Old 

Add test case to check user code case also, i.e the configuration resource has only new keys. Then you can add deprecated keys using addDeprecation and then do get set and get of old keys.

> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>         Attachments: HADOOP-6105-1.patch, HADOOP-6105.patch, HADOOP-6105.patch
>
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-640-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 06 08:43:33 2009
Return-Path: <common-issues-return-640-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48337 invoked from network); 6 Aug 2009 08:43:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Aug 2009 08:43:33 -0000
Received: (qmail 54276 invoked by uid 500); 6 Aug 2009 08:43:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54225 invoked by uid 500); 6 Aug 2009 08:43:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54215 invoked by uid 99); 6 Aug 2009 08:43:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Aug 2009 08:43:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Aug 2009 08:43:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DBAFA234C04B
	for <common-issues@hadoop.apache.org>; Thu,  6 Aug 2009 01:43:14 -0700 (PDT)
Message-ID: <1634377601.1249548194898.JavaMail.jira@brutus>
Date: Thu, 6 Aug 2009 01:43:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6176) Adding a couple private methods to
 AccessTokenHandler for testing purposes
In-Reply-To: <1121825252.1249420814969.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12739943#action_12739943 ] 

Hadoop QA commented on HADOOP-6176:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12415521/core-03.patch
  against trunk revision 800919.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/592/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/592/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/592/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/592/console

This message is automatically generated.

> Adding a couple private methods to AccessTokenHandler for testing purposes
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6176
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6176
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: core-03.patch
>
>
> To support some test cases being added as part of HDFS-409, a couple private methods need to be added to AccessTokenHandler. One is for setting token lifetime and another is checking if a token has expired.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-641-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 06 09:29:59 2009
Return-Path: <common-issues-return-641-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64110 invoked from network); 6 Aug 2009 09:29:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Aug 2009 09:29:59 -0000
Received: (qmail 17451 invoked by uid 500); 6 Aug 2009 09:30:06 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17408 invoked by uid 500); 6 Aug 2009 09:30:06 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17177 invoked by uid 99); 6 Aug 2009 09:30:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Aug 2009 09:30:05 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Aug 2009 09:30:03 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D2031234C1ED
	for <common-issues@hadoop.apache.org>; Thu,  6 Aug 2009 02:29:43 -0700 (PDT)
Message-ID: <1428743334.1249550983859.JavaMail.jira@brutus>
Date: Thu, 6 Aug 2009 02:29:43 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6105) Provide a way to automatically handle
 backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna updated HADOOP-6105:
------------------------------------------

    Attachment: HADOOP-6105-2.patch

Uploading the new patch with the above suggestions being implemented.

> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>         Attachments: HADOOP-6105-1.patch, HADOOP-6105-2.patch, HADOOP-6105.patch, HADOOP-6105.patch
>
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-642-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 06 11:38:33 2009
Return-Path: <common-issues-return-642-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17553 invoked from network); 6 Aug 2009 11:38:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Aug 2009 11:38:33 -0000
Received: (qmail 77392 invoked by uid 500); 6 Aug 2009 11:38:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77340 invoked by uid 500); 6 Aug 2009 11:38:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77255 invoked by uid 99); 6 Aug 2009 11:38:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Aug 2009 11:38:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Aug 2009 11:38:37 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1D9CD234C1EA
	for <common-issues@hadoop.apache.org>; Thu,  6 Aug 2009 04:38:16 -0700 (PDT)
Message-ID: <1138010695.1249558696120.JavaMail.jira@brutus>
Date: Thu, 6 Aug 2009 04:38:16 -0700 (PDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4491) Per-job local data on the TaskTracker
 node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V updated HADOOP-4491:
------------------------------

    Attachment: HADOOP-4491-20090806.txt

Attaching patch that includes the changes suggest in the review. Also, has an added test that verifies permissions set on files at various steps. Running tests with the linux task-controller now.

> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt, HADOOP-4491-20090803.1.txt, HADOOP-4491-20090803.txt, HADOOP-4491-20090806.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-643-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 06 17:50:32 2009
Return-Path: <common-issues-return-643-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11004 invoked from network); 6 Aug 2009 17:50:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Aug 2009 17:50:32 -0000
Received: (qmail 41849 invoked by uid 500); 6 Aug 2009 17:50:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41815 invoked by uid 500); 6 Aug 2009 17:50:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41802 invoked by uid 99); 6 Aug 2009 17:50:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Aug 2009 17:50:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Aug 2009 17:50:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 6536D234C04C
	for <common-issues@hadoop.apache.org>; Thu,  6 Aug 2009 10:50:15 -0700 (PDT)
Message-ID: <2142026939.1249581015413.JavaMail.jira@brutus>
Date: Thu, 6 Aug 2009 10:50:15 -0700 (PDT)
From: "Abdul Qadeer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4012) Providing splitting support for bzip2
 compressed files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Abdul Qadeer updated HADOOP-4012:
---------------------------------

    Attachment: Hadoop-4012-version11.patch

Fixes javadoc and javac warnings.

> Providing splitting support for bzip2 compressed files
> ------------------------------------------------------
>
>                 Key: HADOOP-4012
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4012
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Abdul Qadeer
>            Assignee: Abdul Qadeer
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-4012-version1.patch, Hadoop-4012-version10.patch, Hadoop-4012-version11.patch, Hadoop-4012-version2.patch, Hadoop-4012-version3.patch, Hadoop-4012-version4.patch, Hadoop-4012-version5.patch, Hadoop-4012-version6.patch, Hadoop-4012-version7.patch, Hadoop-4012-version8.patch, Hadoop-4012-version9.patch
>
>
> Hadoop assumes that if the input data is compressed, it can not be split (mainly due to the limitation of many codecs that they need the whole input stream to decompress successfully).  So in such a case, Hadoop prepares only one split per compressed file, where the lower split limit is at 0 while the upper limit is the end of the file.  The consequence of this decision is that, one compress file goes to a single mapper. Although it circumvents the limitation of codecs (as mentioned above) but reduces the parallelism substantially, as it was possible otherwise in case of splitting.
> BZip2 is a compression / De-Compression algorithm which does compression on blocks of data and later these compressed blocks can be decompressed independent of each other.  This is indeed an opportunity that instead of one BZip2 compressed file going to one mapper, we can process chunks of file in parallel.  The correctness criteria of such a processing is that for a bzip2 compressed file, each compressed block should be processed by only one mapper and ultimately all the blocks of the file should be processed.  (By processing we mean the actual utilization of that un-compressed data (coming out of the codecs) in a mapper).
> We are writing the code to implement this suggested functionality.  Although we have used bzip2 as an example, but we have tried to extend Hadoop's compression interfaces so that any other codecs with the same capability as that of bzip2, could easily use the splitting support.  The details of these changes will be posted when we submit the code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-644-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 06 18:48:56 2009
Return-Path: <common-issues-return-644-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49509 invoked from network); 6 Aug 2009 18:48:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Aug 2009 18:48:56 -0000
Received: (qmail 24801 invoked by uid 500); 6 Aug 2009 18:48:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24626 invoked by uid 500); 6 Aug 2009 18:48:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24255 invoked by uid 99); 6 Aug 2009 18:47:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Aug 2009 18:47:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Aug 2009 18:47:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C5B58234C044
	for <common-issues@hadoop.apache.org>; Thu,  6 Aug 2009 11:47:15 -0700 (PDT)
Message-ID: <1994383238.1249584435602.JavaMail.jira@brutus>
Date: Thu, 6 Aug 2009 11:47:15 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6176) Adding a couple private methods to
 AccessTokenHandler for testing purposes
In-Reply-To: <1121825252.1249420814969.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12740191#action_12740191 ] 

Jakob Homan commented on HADOOP-6176:
-------------------------------------

+1 with a readability nit: it may be nice to explicitly state in a comment that the tests are needed to expose otherwise private methods for testing.  May be a reasonable assumption to make on the reader's part, but clarity is always a good thing.

> Adding a couple private methods to AccessTokenHandler for testing purposes
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6176
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6176
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: core-03.patch
>
>
> To support some test cases being added as part of HDFS-409, a couple private methods need to be added to AccessTokenHandler. One is for setting token lifetime and another is checking if a token has expired.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-645-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 06 19:53:32 2009
Return-Path: <common-issues-return-645-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71241 invoked from network); 6 Aug 2009 19:53:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 6 Aug 2009 19:53:32 -0000
Received: (qmail 8086 invoked by uid 500); 6 Aug 2009 19:53:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8032 invoked by uid 500); 6 Aug 2009 19:53:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8018 invoked by uid 99); 6 Aug 2009 19:53:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Aug 2009 19:53:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Aug 2009 19:53:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D3D80234C044
	for <common-issues@hadoop.apache.org>; Thu,  6 Aug 2009 12:53:14 -0700 (PDT)
Message-ID: <2059524330.1249588394853.JavaMail.jira@brutus>
Date: Thu, 6 Aug 2009 12:53:14 -0700 (PDT)
From: "Abdul Qadeer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4012) Providing splitting support for
 bzip2 compressed files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12740225#action_12740225 ] 

Abdul Qadeer commented on HADOOP-4012:
--------------------------------------

The MapReduce related stuff from the original HADOOP-4012 has been shifted to MapReduce-830 (https://issues.apache.org/jira/browse/MAPREDUCE-830).  It was necessary because of the recent forking of Hadoop into three sub-projects.  MapReduce-830 will only compile once this HADOOP-4012 commits.

> Providing splitting support for bzip2 compressed files
> ------------------------------------------------------
>
>                 Key: HADOOP-4012
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4012
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Abdul Qadeer
>            Assignee: Abdul Qadeer
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-4012-version1.patch, Hadoop-4012-version10.patch, Hadoop-4012-version11.patch, Hadoop-4012-version2.patch, Hadoop-4012-version3.patch, Hadoop-4012-version4.patch, Hadoop-4012-version5.patch, Hadoop-4012-version6.patch, Hadoop-4012-version7.patch, Hadoop-4012-version8.patch, Hadoop-4012-version9.patch
>
>
> Hadoop assumes that if the input data is compressed, it can not be split (mainly due to the limitation of many codecs that they need the whole input stream to decompress successfully).  So in such a case, Hadoop prepares only one split per compressed file, where the lower split limit is at 0 while the upper limit is the end of the file.  The consequence of this decision is that, one compress file goes to a single mapper. Although it circumvents the limitation of codecs (as mentioned above) but reduces the parallelism substantially, as it was possible otherwise in case of splitting.
> BZip2 is a compression / De-Compression algorithm which does compression on blocks of data and later these compressed blocks can be decompressed independent of each other.  This is indeed an opportunity that instead of one BZip2 compressed file going to one mapper, we can process chunks of file in parallel.  The correctness criteria of such a processing is that for a bzip2 compressed file, each compressed block should be processed by only one mapper and ultimately all the blocks of the file should be processed.  (By processing we mean the actual utilization of that un-compressed data (coming out of the codecs) in a mapper).
> We are writing the code to implement this suggested functionality.  Although we have used bzip2 as an example, but we have tried to extend Hadoop's compression interfaces so that any other codecs with the same capability as that of bzip2, could easily use the splitting support.  The details of these changes will be posted when we submit the code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-646-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 07 04:11:30 2009
Return-Path: <common-issues-return-646-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16932 invoked from network); 7 Aug 2009 04:11:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Aug 2009 04:11:29 -0000
Received: (qmail 86862 invoked by uid 500); 7 Aug 2009 04:11:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86812 invoked by uid 500); 7 Aug 2009 04:11:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86795 invoked by uid 99); 7 Aug 2009 04:11:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 04:11:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 04:11:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id F33BC234C055
	for <common-issues@hadoop.apache.org>; Thu,  6 Aug 2009 21:11:14 -0700 (PDT)
Message-ID: <1554871263.1249618274995.JavaMail.jira@brutus>
Date: Thu, 6 Aug 2009 21:11:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6177) FSInputChecker.getPos() would return
 position greater than the file size
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

FSInputChecker.getPos() would return position greater than the file size
------------------------------------------------------------------------

                 Key: HADOOP-6177
                 URL: https://issues.apache.org/jira/browse/HADOOP-6177
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Hong Tang


When using a small buffer (< 512 bytes) to read through the whole file, the final file position is 1+ the file size.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-648-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 07 04:17:33 2009
Return-Path: <common-issues-return-648-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17866 invoked from network); 7 Aug 2009 04:17:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Aug 2009 04:17:33 -0000
Received: (qmail 89788 invoked by uid 500); 7 Aug 2009 04:17:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89735 invoked by uid 500); 7 Aug 2009 04:17:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89715 invoked by uid 99); 7 Aug 2009 04:17:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 04:17:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 04:17:37 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id B9040234C044
	for <common-issues@hadoop.apache.org>; Thu,  6 Aug 2009 21:17:15 -0700 (PDT)
Message-ID: <67666835.1249618635753.JavaMail.jira@brutus>
Date: Thu, 6 Aug 2009 21:17:15 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6177) FSInputChecker.getPos() would return
 position greater than the file size
In-Reply-To: <1554871263.1249618274995.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-6177:
------------------------------

    Attachment: hadoop-6177-testcase.patch

The patch file modifies existing test cases to check that after reading through the file, the final offset would be the same as the file offset. On my local machine, it yields:

{noformat}
% ant test-core -Dtestcase=TestChecksumFileSystem
run-test-core:
   [delete] Deleting directory /Users/htang/Documents/Work/workspace/hadoop-common/build/test/data
    [mkdir] Created dir: /Users/htang/Documents/Work/workspace/hadoop-common/build/test/data
   [delete] Deleting directory /Users/htang/Documents/Work/workspace/hadoop-common/build/test/logs
    [mkdir] Created dir: /Users/htang/Documents/Work/workspace/hadoop-common/build/test/logs
    [junit] Running org.apache.hadoop.fs.TestChecksumFileSystem
    [junit] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 0.59 sec
    [junit] Test org.apache.hadoop.fs.TestChecksumFileSystem FAILED
% tail build/test/TEST-org.apache.hadoop.fs.TestChecksumFileSystem.txt
------------- ---------------- ---------------

Testcase: testgetChecksumLength took 0.196 sec
Testcase: testVerifyChecksum took 0.373 sec
        FAILED
expected:<7> but was:<8>
junit.framework.AssertionFailedError: expected:<7> but was:<8>
        at org.apache.hadoop.fs.TestLocalFileSystem.readFile(TestLocalFileSystem.java:45)
        at org.apache.hadoop.fs.TestChecksumFileSystem.testVerifyChecksum(TestChecksumFileSystem.java:57)
{noformat}

> FSInputChecker.getPos() would return position greater than the file size
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6177
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6177
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hong Tang
>         Attachments: hadoop-6177-testcase.patch
>
>
> When using a small buffer (< 512 bytes) to read through the whole file, the final file position is 1+ the file size.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-647-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 07 04:17:33 2009
Return-Path: <common-issues-return-647-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17883 invoked from network); 7 Aug 2009 04:17:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Aug 2009 04:17:33 -0000
Received: (qmail 89710 invoked by uid 500); 7 Aug 2009 04:17:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89660 invoked by uid 500); 7 Aug 2009 04:17:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89649 invoked by uid 99); 7 Aug 2009 04:17:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 04:17:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 04:17:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D5CA1234C056
	for <common-issues@hadoop.apache.org>; Thu,  6 Aug 2009 21:17:15 -0700 (PDT)
Message-ID: <924978193.1249618635874.JavaMail.jira@brutus>
Date: Thu, 6 Aug 2009 21:17:15 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6177) FSInputChecker.getPos() would return
 position greater than the file size
In-Reply-To: <1554871263.1249618274995.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang reassigned HADOOP-6177:
---------------------------------

    Assignee: Hong Tang

> FSInputChecker.getPos() would return position greater than the file size
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6177
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6177
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>         Attachments: hadoop-6177-testcase.patch
>
>
> When using a small buffer (< 512 bytes) to read through the whole file, the final file position is 1+ the file size.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-649-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 07 04:23:30 2009
Return-Path: <common-issues-return-649-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19021 invoked from network); 7 Aug 2009 04:23:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Aug 2009 04:23:30 -0000
Received: (qmail 95384 invoked by uid 500); 7 Aug 2009 04:23:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95356 invoked by uid 500); 7 Aug 2009 04:23:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95317 invoked by uid 99); 7 Aug 2009 04:23:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 04:23:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 04:23:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C40C1234C044
	for <common-issues@hadoop.apache.org>; Thu,  6 Aug 2009 21:23:14 -0700 (PDT)
Message-ID: <542448327.1249618994788.JavaMail.jira@brutus>
Date: Thu, 6 Aug 2009 21:23:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6177) FSInputChecker.getPos() would
 return position greater than the file size
In-Reply-To: <1554871263.1249618274995.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12740395#action_12740395 ] 

Hong Tang commented on HADOOP-6177:
-----------------------------------

The root cause is because readChecksumChunk returns -1 when the fs input stream reaches the end of the file. And in fill(), this leads to count == -1 (FSInputChecker.java:176). The fix is rather simple, adding the following line at the end of fill():
{noformat}
     count = readChecksumChunk(buf, 0, buf.length);
+    if (count < 0) count = 0; 
{noformat}

> FSInputChecker.getPos() would return position greater than the file size
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6177
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6177
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>         Attachments: hadoop-6177-testcase.patch
>
>
> When using a small buffer (< 512 bytes) to read through the whole file, the final file position is 1+ the file size.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-650-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 07 04:27:32 2009
Return-Path: <common-issues-return-650-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19260 invoked from network); 7 Aug 2009 04:27:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Aug 2009 04:27:31 -0000
Received: (qmail 96136 invoked by uid 500); 7 Aug 2009 04:27:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96085 invoked by uid 500); 7 Aug 2009 04:27:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96075 invoked by uid 99); 7 Aug 2009 04:27:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 04:27:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 04:27:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C76BC234C044
	for <common-issues@hadoop.apache.org>; Thu,  6 Aug 2009 21:27:14 -0700 (PDT)
Message-ID: <2145180782.1249619234801.JavaMail.jira@brutus>
Date: Thu, 6 Aug 2009 21:27:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6177) FSInputChecker.getPos() would return
 position greater than the file size
In-Reply-To: <1554871263.1249618274995.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-6177:
------------------------------

    Attachment: hadoop-6177-2090806.patch

Patch contains both modified test cases and the one line fix.

> FSInputChecker.getPos() would return position greater than the file size
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6177
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6177
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>         Attachments: hadoop-6177-2090806.patch, hadoop-6177-testcase.patch
>
>
> When using a small buffer (< 512 bytes) to read through the whole file, the final file position is 1+ the file size.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-651-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 07 05:03:29 2009
Return-Path: <common-issues-return-651-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24486 invoked from network); 7 Aug 2009 05:03:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Aug 2009 05:03:29 -0000
Received: (qmail 17299 invoked by uid 500); 7 Aug 2009 05:03:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17248 invoked by uid 500); 7 Aug 2009 05:03:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17238 invoked by uid 99); 7 Aug 2009 05:03:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 05:03:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 05:03:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C7F12234C044
	for <common-issues@hadoop.apache.org>; Thu,  6 Aug 2009 22:03:14 -0700 (PDT)
Message-ID: <1725085404.1249621394814.JavaMail.jira@brutus>
Date: Thu, 6 Aug 2009 22:03:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6177) FSInputChecker.getPos() would return
 position greater than the file size
In-Reply-To: <1554871263.1249618274995.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-6177:
------------------------------

        Fix Version/s: 0.21.0
    Affects Version/s: 0.21.0
               Status: Patch Available  (was: Open)

ant test and ant test-patch passed on my local machine:

     [exec] +1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 6 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.

> FSInputChecker.getPos() would return position greater than the file size
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6177
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6177
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6177-2090806.patch, hadoop-6177-testcase.patch
>
>
> When using a small buffer (< 512 bytes) to read through the whole file, the final file position is 1+ the file size.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-652-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 07 05:33:33 2009
Return-Path: <common-issues-return-652-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32921 invoked from network); 7 Aug 2009 05:33:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Aug 2009 05:33:33 -0000
Received: (qmail 45130 invoked by uid 500); 7 Aug 2009 05:33:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45092 invoked by uid 500); 7 Aug 2009 05:33:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44869 invoked by uid 99); 7 Aug 2009 05:33:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 05:33:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 05:33:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C9675234C044
	for <common-issues@hadoop.apache.org>; Thu,  6 Aug 2009 22:33:14 -0700 (PDT)
Message-ID: <1473773043.1249623194810.JavaMail.jira@brutus>
Date: Thu, 6 Aug 2009 22:33:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6177) FSInputChecker.getPos() would
 return position greater than the file size
In-Reply-To: <1554871263.1249618274995.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12740412#action_12740412 ] 

Hadoop QA commented on HADOOP-6177:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12415815/hadoop-6177-2090806.patch
  against trunk revision 800919.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/593/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/593/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/593/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/593/console

This message is automatically generated.

> FSInputChecker.getPos() would return position greater than the file size
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6177
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6177
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6177-2090806.patch, hadoop-6177-testcase.patch
>
>
> When using a small buffer (< 512 bytes) to read through the whole file, the final file position is 1+ the file size.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-653-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 07 06:47:30 2009
Return-Path: <common-issues-return-653-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63845 invoked from network); 7 Aug 2009 06:47:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Aug 2009 06:47:30 -0000
Received: (qmail 91585 invoked by uid 500); 7 Aug 2009 06:47:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91533 invoked by uid 500); 7 Aug 2009 06:47:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91523 invoked by uid 99); 7 Aug 2009 06:47:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 06:47:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 06:47:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id A19DE234C044
	for <common-issues@hadoop.apache.org>; Thu,  6 Aug 2009 23:47:15 -0700 (PDT)
Message-ID: <133461431.1249627635647.JavaMail.jira@brutus>
Date: Thu, 6 Aug 2009 23:47:15 -0700 (PDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4491) Per-job local data on the TaskTracker
 node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V updated HADOOP-4491:
------------------------------

    Attachment: HADOOP-4491-20090807.2.txt

Updated patch changing the forrest documentation to reflect the changes.

> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt, HADOOP-4491-20090803.1.txt, HADOOP-4491-20090803.txt, HADOOP-4491-20090806.txt, HADOOP-4491-20090807.2.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-654-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 07 09:49:30 2009
Return-Path: <common-issues-return-654-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10654 invoked from network); 7 Aug 2009 09:49:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Aug 2009 09:49:30 -0000
Received: (qmail 69239 invoked by uid 500); 7 Aug 2009 09:49:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69186 invoked by uid 500); 7 Aug 2009 09:49:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69176 invoked by uid 99); 7 Aug 2009 09:49:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 09:49:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED,FB_WORD1_END_DOLLAR
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 09:49:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CDFCF234C04B
	for <common-issues@hadoop.apache.org>; Fri,  7 Aug 2009 02:49:14 -0700 (PDT)
Message-ID: <198983034.1249638554842.JavaMail.jira@brutus>
Date: Fri, 7 Aug 2009 02:49:14 -0700 (PDT)
From: "Amar Kamat (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6178) GenericOptionsParser should ignore
 symlinks from archive filenames
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

GenericOptionsParser should ignore symlinks from archive filenames
------------------------------------------------------------------

                 Key: HADOOP-6178
                 URL: https://issues.apache.org/jira/browse/HADOOP-6178
             Project: Hadoop Common
          Issue Type: Bug
          Components: util
            Reporter: Amar Kamat


http://hadoop.apache.org/common/docs/r0.20.0/streaming.html#Large+files+and+archives+in+Hadoop+Streaming example doesnt work as expected. 
Stack trace :
{code}
trunk$  ./bin/hadoop jar build/contrib/streaming/hadoop-0.21.0-dev-streaming.jar  -D mapred.map.tasks=1 \
         -D mapred.reduce.tasks=1 -D mapred.job.name="Experiment" \
        -archives 'hdfs://namenode:port/user/me/cachedir.jar#testlink' -input input -output output -mapper "xargs cat" -reducer "cat"
java.io.FileNotFoundException: File hdfs://namenode:port/user/me/cachedir.jar#testlink does not exist.
	at org.apache.hadoop.util.GenericOptionsParser.validateFiles(GenericOptionsParser.java:349)
	at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:275)
	at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:375)
	at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:153)
	at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:138)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
	at org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreaming.java:50)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
{code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-655-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 07 13:08:30 2009
Return-Path: <common-issues-return-655-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79934 invoked from network); 7 Aug 2009 13:08:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Aug 2009 13:08:30 -0000
Received: (qmail 20196 invoked by uid 500); 7 Aug 2009 13:08:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20145 invoked by uid 500); 7 Aug 2009 13:08:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19945 invoked by uid 99); 7 Aug 2009 13:08:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 13:08:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 13:08:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D18FA234C04C
	for <common-issues@hadoop.apache.org>; Fri,  7 Aug 2009 06:08:14 -0700 (PDT)
Message-ID: <517820113.1249650494857.JavaMail.jira@brutus>
Date: Fri, 7 Aug 2009 06:08:14 -0700 (PDT)
From: "Karam Singh (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6179) Modify hadoop script  to pick correct
 jar from hadoop-mapred jar from HADOOP_HOME
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Modify hadoop script  to pick correct jar from hadoop-mapred jar from HADOOP_HOME
---------------------------------------------------------------------------------

                 Key: HADOOP-6179
                 URL: https://issues.apache.org/jira/browse/HADOOP-6179
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.21.0
            Reporter: Karam Singh


Currently when we checkout mapred trunk and use ant tar . It copies hadoop-mapred-0.21.0-dev.jar under HADOOP_HOME. So when deploy the tar and try start jobtracker/tracker as hadoop/mapred not able rleant classes in CLASSPATH so we have copy hadoop-mapred-0.21.0-dev.jar to HADOOP_HOME/lib directory

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-656-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 07 13:22:30 2009
Return-Path: <common-issues-return-656-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83186 invoked from network); 7 Aug 2009 13:22:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Aug 2009 13:22:30 -0000
Received: (qmail 45187 invoked by uid 500); 7 Aug 2009 13:22:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45147 invoked by uid 500); 7 Aug 2009 13:22:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45136 invoked by uid 99); 7 Aug 2009 13:22:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 13:22:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 13:22:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0F351234C044
	for <common-issues@hadoop.apache.org>; Fri,  7 Aug 2009 06:22:15 -0700 (PDT)
Message-ID: <1883983218.1249651335047.JavaMail.jira@brutus>
Date: Fri, 7 Aug 2009 06:22:15 -0700 (PDT)
From: "Karam Singh (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6179) Modify hadoop scripts  to pick
 correct jar from hadoop-mapred jar from HADOOP_HOME
In-Reply-To: <517820113.1249650494857.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Karam Singh updated HADOOP-6179:
--------------------------------

    Summary: Modify hadoop scripts  to pick correct jar from hadoop-mapred jar from HADOOP_HOME  (was: Modify hadoop script  to pick correct jar from hadoop-mapred jar from HADOOP_HOME)

> Modify hadoop scripts  to pick correct jar from hadoop-mapred jar from HADOOP_HOME
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6179
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6179
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Karam Singh
>
> Currently when we checkout mapred trunk and use ant tar . It copies hadoop-mapred-0.21.0-dev.jar under HADOOP_HOME. So when deploy the tar and try start jobtracker/tracker as hadoop/mapred not able rleant classes in CLASSPATH so we have copy hadoop-mapred-0.21.0-dev.jar to HADOOP_HOME/lib directory

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-657-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 07 17:55:30 2009
Return-Path: <common-issues-return-657-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95630 invoked from network); 7 Aug 2009 17:55:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Aug 2009 17:55:30 -0000
Received: (qmail 21392 invoked by uid 500); 7 Aug 2009 17:55:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21340 invoked by uid 500); 7 Aug 2009 17:55:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21330 invoked by uid 99); 7 Aug 2009 17:55:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 17:55:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 17:55:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D545E234C004
	for <common-issues@hadoop.apache.org>; Fri,  7 Aug 2009 10:55:14 -0700 (PDT)
Message-ID: <103536847.1249667714859.JavaMail.jira@brutus>
Date: Fri, 7 Aug 2009 10:55:14 -0700 (PDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Moved: (HADOOP-6180) Namenode slowed down when many files
 with same filename were moved to Trash
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik moved HDFS-124 to HADOOP-6180:
---------------------------------------------

        Key: HADOOP-6180  (was: HDFS-124)
    Project: Hadoop Common  (was: Hadoop HDFS)

> Namenode slowed down when many files with same filename were moved to Trash
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-6180
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6180
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Koji Noguchi
>            Assignee: Boris Shkolnik
>            Priority: Minor
>         Attachments: COMMON-124.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-658-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 07 18:01:35 2009
Return-Path: <common-issues-return-658-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98732 invoked from network); 7 Aug 2009 18:01:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Aug 2009 18:01:35 -0000
Received: (qmail 33367 invoked by uid 500); 7 Aug 2009 18:01:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33313 invoked by uid 500); 7 Aug 2009 18:01:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33275 invoked by uid 99); 7 Aug 2009 18:01:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 18:01:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 18:01:37 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 8BCAF234C004
	for <common-issues@hadoop.apache.org>; Fri,  7 Aug 2009 11:01:16 -0700 (PDT)
Message-ID: <1291552845.1249668076562.JavaMail.jira@brutus>
Date: Fri, 7 Aug 2009 11:01:16 -0700 (PDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6180) Namenode slowed down when many files
 with same filename were moved to Trash
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6180:
----------------------------------

       Resolution: Fixed
    Fix Version/s: 0.21.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I've just committed this. Thanks Boris!

> Namenode slowed down when many files with same filename were moved to Trash
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-6180
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6180
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Koji Noguchi
>            Assignee: Boris Shkolnik
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: COMMON-124.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-659-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 07 18:57:30 2009
Return-Path: <common-issues-return-659-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36055 invoked from network); 7 Aug 2009 18:57:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Aug 2009 18:57:30 -0000
Received: (qmail 21788 invoked by uid 500); 7 Aug 2009 18:57:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21740 invoked by uid 500); 7 Aug 2009 18:57:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21730 invoked by uid 99); 7 Aug 2009 18:57:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 18:57:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 18:57:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1113D234C04C
	for <common-issues@hadoop.apache.org>; Fri,  7 Aug 2009 11:57:15 -0700 (PDT)
Message-ID: <1588163258.1249671435068.JavaMail.jira@brutus>
Date: Fri, 7 Aug 2009 11:57:15 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6176) Adding a couple private methods to
 AccessTokenHandler for testing purposes
In-Reply-To: <1121825252.1249420814969.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12740679#action_12740679 ] 

Konstantin Boudnik commented on HADOOP-6176:
--------------------------------------------

There was an idea to express the essence of this modification via aspects. However, in order to deliver this functionality in a most timely manner this effort will be carried on separately. I might be filing a separate JIRA for it if a preliminary investigation will show that such approach is viable.
 

> Adding a couple private methods to AccessTokenHandler for testing purposes
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6176
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6176
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: core-03.patch
>
>
> To support some test cases being added as part of HDFS-409, a couple private methods need to be added to AccessTokenHandler. One is for setting token lifetime and another is checking if a token has expired.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-660-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 07 18:59:32 2009
Return-Path: <common-issues-return-660-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36385 invoked from network); 7 Aug 2009 18:59:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Aug 2009 18:59:31 -0000
Received: (qmail 24899 invoked by uid 500); 7 Aug 2009 18:59:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24864 invoked by uid 500); 7 Aug 2009 18:59:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24854 invoked by uid 99); 7 Aug 2009 18:59:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 18:59:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 18:59:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D0834234C004
	for <common-issues@hadoop.apache.org>; Fri,  7 Aug 2009 11:59:14 -0700 (PDT)
Message-ID: <498445413.1249671554839.JavaMail.jira@brutus>
Date: Fri, 7 Aug 2009 11:59:14 -0700 (PDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6180) Namenode slowed down when many files
 with same filename were moved to Trash
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6180:
-----------------------------------

    Release Note: 
This fix changes the naming of files moved to Trash.
Before files with the same name were appended with a running number 1,2,3.. and so on. Now it will have the current time in milliseconds attached to the name.

> Namenode slowed down when many files with same filename were moved to Trash
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-6180
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6180
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Koji Noguchi
>            Assignee: Boris Shkolnik
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: COMMON-124.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-661-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 07 21:43:30 2009
Return-Path: <common-issues-return-661-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86315 invoked from network); 7 Aug 2009 21:43:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Aug 2009 21:43:29 -0000
Received: (qmail 41881 invoked by uid 500); 7 Aug 2009 21:43:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41838 invoked by uid 500); 7 Aug 2009 21:43:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41827 invoked by uid 99); 7 Aug 2009 21:43:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 21:43:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 21:43:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C882F234C004
	for <common-issues@hadoop.apache.org>; Fri,  7 Aug 2009 14:43:14 -0700 (PDT)
Message-ID: <1274347518.1249681394806.JavaMail.jira@brutus>
Date: Fri, 7 Aug 2009 14:43:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6176) Adding a couple private methods to
 AccessTokenHandler for testing purposes
In-Reply-To: <1121825252.1249420814969.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6176:
-------------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.21.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I have committed this.  Thanks, Kan!

> Adding a couple private methods to AccessTokenHandler for testing purposes
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6176
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6176
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>             Fix For: 0.21.0
>
>         Attachments: core-03.patch
>
>
> To support some test cases being added as part of HDFS-409, a couple private methods need to be added to AccessTokenHandler. One is for setting token lifetime and another is checking if a token has expired.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-662-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 07 23:02:32 2009
Return-Path: <common-issues-return-662-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16791 invoked from network); 7 Aug 2009 23:02:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 7 Aug 2009 23:02:32 -0000
Received: (qmail 98847 invoked by uid 500); 7 Aug 2009 23:02:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98806 invoked by uid 500); 7 Aug 2009 23:02:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98796 invoked by uid 99); 7 Aug 2009 23:02:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 23:02:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 23:02:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CBDC4234C004
	for <common-issues@hadoop.apache.org>; Fri,  7 Aug 2009 16:02:14 -0700 (PDT)
Message-ID: <1000787390.1249686134827.JavaMail.jira@brutus>
Date: Fri, 7 Aug 2009 16:02:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6180) Namenode slowed down when many
 files with same filename were moved to Trash
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12740775#action_12740775 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6180:
------------------------------------------------

After updated hadoop-core-0.21.0-dev.jar and hadoop-core-test-0.21.0-dev.jar on hdfs, TestHDFSTrash failed.  Could someone take a look?

{noformat}
------------- Standard Error -----------------
rmr: Cannot move "hdfs://localhost:57940/user/tsz" to the trash, as it contains  
the trash
------------- ---------------- ---------------

Testcase: testTrash took 1.451 sec
        Caused an ERROR
null
java.lang.NullPointerException
        at org.apache.hadoop.fs.TestTrash.countSameDeletedFiles(TestTrash.java:77
5)
        at org.apache.hadoop.fs.TestTrash.trashShell(TestTrash.java:363)
        at org.apache.hadoop.hdfs.TestHDFSTrash.testTrash(TestHDFSTrash.java:54)
        at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
        at junit.extensions.TestSetup$1.protect(TestSetup.java:23)
        at junit.extensions.TestSetup.run(TestSetup.java:27)

Testcase: testNonDefaultFS took 0.262 sec
{noformat}

> Namenode slowed down when many files with same filename were moved to Trash
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-6180
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6180
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Koji Noguchi
>            Assignee: Boris Shkolnik
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: COMMON-124.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-664-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 08 11:02:30 2009
Return-Path: <common-issues-return-664-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95936 invoked from network); 8 Aug 2009 11:02:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Aug 2009 11:02:30 -0000
Received: (qmail 65844 invoked by uid 500); 8 Aug 2009 11:02:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65793 invoked by uid 500); 8 Aug 2009 11:02:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65779 invoked by uid 99); 8 Aug 2009 11:02:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 08 Aug 2009 11:02:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 08 Aug 2009 11:02:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 307B0234C4A7
	for <common-issues@hadoop.apache.org>; Sat,  8 Aug 2009 04:02:15 -0700 (PDT)
Message-ID: <421822988.1249729335197.JavaMail.jira@brutus>
Date: Sat, 8 Aug 2009 04:02:15 -0700 (PDT)
From: "Abdul Qadeer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4012) Providing splitting support for bzip2
 compressed files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Abdul Qadeer updated HADOOP-4012:
---------------------------------

    Status: Patch Available  (was: In Progress)

Re-submitting because the patch is not appearing in the Hudson queue.

> Providing splitting support for bzip2 compressed files
> ------------------------------------------------------
>
>                 Key: HADOOP-4012
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4012
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Abdul Qadeer
>            Assignee: Abdul Qadeer
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-4012-version1.patch, Hadoop-4012-version10.patch, Hadoop-4012-version11.patch, Hadoop-4012-version2.patch, Hadoop-4012-version3.patch, Hadoop-4012-version4.patch, Hadoop-4012-version5.patch, Hadoop-4012-version6.patch, Hadoop-4012-version7.patch, Hadoop-4012-version8.patch, Hadoop-4012-version9.patch
>
>
> Hadoop assumes that if the input data is compressed, it can not be split (mainly due to the limitation of many codecs that they need the whole input stream to decompress successfully).  So in such a case, Hadoop prepares only one split per compressed file, where the lower split limit is at 0 while the upper limit is the end of the file.  The consequence of this decision is that, one compress file goes to a single mapper. Although it circumvents the limitation of codecs (as mentioned above) but reduces the parallelism substantially, as it was possible otherwise in case of splitting.
> BZip2 is a compression / De-Compression algorithm which does compression on blocks of data and later these compressed blocks can be decompressed independent of each other.  This is indeed an opportunity that instead of one BZip2 compressed file going to one mapper, we can process chunks of file in parallel.  The correctness criteria of such a processing is that for a bzip2 compressed file, each compressed block should be processed by only one mapper and ultimately all the blocks of the file should be processed.  (By processing we mean the actual utilization of that un-compressed data (coming out of the codecs) in a mapper).
> We are writing the code to implement this suggested functionality.  Although we have used bzip2 as an example, but we have tried to extend Hadoop's compression interfaces so that any other codecs with the same capability as that of bzip2, could easily use the splitting support.  The details of these changes will be posted when we submit the code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-663-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 08 11:02:31 2009
Return-Path: <common-issues-return-663-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95986 invoked from network); 8 Aug 2009 11:02:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Aug 2009 11:02:31 -0000
Received: (qmail 65753 invoked by uid 500); 8 Aug 2009 11:02:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65700 invoked by uid 500); 8 Aug 2009 11:02:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65683 invoked by uid 99); 8 Aug 2009 11:02:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 08 Aug 2009 11:02:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 08 Aug 2009 11:02:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EFE25234C004
	for <common-issues@hadoop.apache.org>; Sat,  8 Aug 2009 04:02:14 -0700 (PDT)
Message-ID: <2048869361.1249729334967.JavaMail.jira@brutus>
Date: Sat, 8 Aug 2009 04:02:14 -0700 (PDT)
From: "Abdul Qadeer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4012) Providing splitting support for bzip2
 compressed files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Abdul Qadeer updated HADOOP-4012:
---------------------------------

    Status: In Progress  (was: Patch Available)

> Providing splitting support for bzip2 compressed files
> ------------------------------------------------------
>
>                 Key: HADOOP-4012
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4012
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Abdul Qadeer
>            Assignee: Abdul Qadeer
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-4012-version1.patch, Hadoop-4012-version10.patch, Hadoop-4012-version11.patch, Hadoop-4012-version2.patch, Hadoop-4012-version3.patch, Hadoop-4012-version4.patch, Hadoop-4012-version5.patch, Hadoop-4012-version6.patch, Hadoop-4012-version7.patch, Hadoop-4012-version8.patch, Hadoop-4012-version9.patch
>
>
> Hadoop assumes that if the input data is compressed, it can not be split (mainly due to the limitation of many codecs that they need the whole input stream to decompress successfully).  So in such a case, Hadoop prepares only one split per compressed file, where the lower split limit is at 0 while the upper limit is the end of the file.  The consequence of this decision is that, one compress file goes to a single mapper. Although it circumvents the limitation of codecs (as mentioned above) but reduces the parallelism substantially, as it was possible otherwise in case of splitting.
> BZip2 is a compression / De-Compression algorithm which does compression on blocks of data and later these compressed blocks can be decompressed independent of each other.  This is indeed an opportunity that instead of one BZip2 compressed file going to one mapper, we can process chunks of file in parallel.  The correctness criteria of such a processing is that for a bzip2 compressed file, each compressed block should be processed by only one mapper and ultimately all the blocks of the file should be processed.  (By processing we mean the actual utilization of that un-compressed data (coming out of the codecs) in a mapper).
> We are writing the code to implement this suggested functionality.  Although we have used bzip2 as an example, but we have tried to extend Hadoop's compression interfaces so that any other codecs with the same capability as that of bzip2, could easily use the splitting support.  The details of these changes will be posted when we submit the code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-665-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 08 11:06:31 2009
Return-Path: <common-issues-return-665-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96602 invoked from network); 8 Aug 2009 11:06:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Aug 2009 11:06:31 -0000
Received: (qmail 66647 invoked by uid 500); 8 Aug 2009 11:06:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66594 invoked by uid 500); 8 Aug 2009 11:06:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66583 invoked by uid 99); 8 Aug 2009 11:06:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 08 Aug 2009 11:06:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 08 Aug 2009 11:06:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CBE9A234C004
	for <common-issues@hadoop.apache.org>; Sat,  8 Aug 2009 04:06:14 -0700 (PDT)
Message-ID: <1360847263.1249729574820.JavaMail.jira@brutus>
Date: Sat, 8 Aug 2009 04:06:14 -0700 (PDT)
From: "Amr Awadallah (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4012) Providing splitting support for
 bzip2 compressed files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12740874#action_12740874 ] 

Amr Awadallah commented on HADOOP-4012:
---------------------------------------

I am out of office until Aug 14th. I will be checking my email
intermittently. If this is urgent then please call my cell phone,
otherwise I will reply to your email when I get back.

Thanks for your patience,

-- amr


> Providing splitting support for bzip2 compressed files
> ------------------------------------------------------
>
>                 Key: HADOOP-4012
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4012
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Abdul Qadeer
>            Assignee: Abdul Qadeer
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-4012-version1.patch, Hadoop-4012-version10.patch, Hadoop-4012-version11.patch, Hadoop-4012-version2.patch, Hadoop-4012-version3.patch, Hadoop-4012-version4.patch, Hadoop-4012-version5.patch, Hadoop-4012-version6.patch, Hadoop-4012-version7.patch, Hadoop-4012-version8.patch, Hadoop-4012-version9.patch
>
>
> Hadoop assumes that if the input data is compressed, it can not be split (mainly due to the limitation of many codecs that they need the whole input stream to decompress successfully).  So in such a case, Hadoop prepares only one split per compressed file, where the lower split limit is at 0 while the upper limit is the end of the file.  The consequence of this decision is that, one compress file goes to a single mapper. Although it circumvents the limitation of codecs (as mentioned above) but reduces the parallelism substantially, as it was possible otherwise in case of splitting.
> BZip2 is a compression / De-Compression algorithm which does compression on blocks of data and later these compressed blocks can be decompressed independent of each other.  This is indeed an opportunity that instead of one BZip2 compressed file going to one mapper, we can process chunks of file in parallel.  The correctness criteria of such a processing is that for a bzip2 compressed file, each compressed block should be processed by only one mapper and ultimately all the blocks of the file should be processed.  (By processing we mean the actual utilization of that un-compressed data (coming out of the codecs) in a mapper).
> We are writing the code to implement this suggested functionality.  Although we have used bzip2 as an example, but we have tried to extend Hadoop's compression interfaces so that any other codecs with the same capability as that of bzip2, could easily use the splitting support.  The details of these changes will be posted when we submit the code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-666-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 08 13:04:29 2009
Return-Path: <common-issues-return-666-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16576 invoked from network); 8 Aug 2009 13:04:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Aug 2009 13:04:29 -0000
Received: (qmail 27770 invoked by uid 500); 8 Aug 2009 13:04:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27721 invoked by uid 500); 8 Aug 2009 13:04:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27711 invoked by uid 99); 8 Aug 2009 13:04:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 08 Aug 2009 13:04:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 08 Aug 2009 13:04:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D56B7234C004
	for <common-issues@hadoop.apache.org>; Sat,  8 Aug 2009 06:04:14 -0700 (PDT)
Message-ID: <1300824219.1249736654862.JavaMail.jira@brutus>
Date: Sat, 8 Aug 2009 06:04:14 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6176) Adding a couple private methods to
 AccessTokenHandler for testing purposes
In-Reply-To: <1121825252.1249420814969.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12740892#action_12740892 ] 

Hudson commented on HADOOP-6176:
--------------------------------

Integrated in Hadoop-Common-trunk #48 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/48/])
    . Add a couple package private methods to AccessTokenHandler for testing.  Contributed by Kan Zhang


> Adding a couple private methods to AccessTokenHandler for testing purposes
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6176
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6176
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>             Fix For: 0.21.0
>
>         Attachments: core-03.patch
>
>
> To support some test cases being added as part of HDFS-409, a couple private methods need to be added to AccessTokenHandler. One is for setting token lifetime and another is checking if a token has expired.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-667-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 08 13:04:31 2009
Return-Path: <common-issues-return-667-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16615 invoked from network); 8 Aug 2009 13:04:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Aug 2009 13:04:31 -0000
Received: (qmail 27856 invoked by uid 500); 8 Aug 2009 13:04:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27806 invoked by uid 500); 8 Aug 2009 13:04:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27792 invoked by uid 99); 8 Aug 2009 13:04:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 08 Aug 2009 13:04:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 08 Aug 2009 13:04:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EE35B234C052
	for <common-issues@hadoop.apache.org>; Sat,  8 Aug 2009 06:04:14 -0700 (PDT)
Message-ID: <1177065004.1249736654974.JavaMail.jira@brutus>
Date: Sat, 8 Aug 2009 06:04:14 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6180) Namenode slowed down when many
 files with same filename were moved to Trash
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12740893#action_12740893 ] 

Hudson commented on HADOOP-6180:
--------------------------------

Integrated in Hadoop-Common-trunk #48 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/48/])
    . NameNode slowed down when many files with same filename were moved to Trash. Contributed by Boris Shkolnik.


> Namenode slowed down when many files with same filename were moved to Trash
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-6180
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6180
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Koji Noguchi
>            Assignee: Boris Shkolnik
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: COMMON-124.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-668-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 08 13:16:30 2009
Return-Path: <common-issues-return-668-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20362 invoked from network); 8 Aug 2009 13:16:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 8 Aug 2009 13:16:30 -0000
Received: (qmail 34295 invoked by uid 500); 8 Aug 2009 13:16:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34240 invoked by uid 500); 8 Aug 2009 13:16:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34230 invoked by uid 99); 8 Aug 2009 13:16:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 08 Aug 2009 13:16:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 08 Aug 2009 13:16:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4A0D6234C004
	for <common-issues@hadoop.apache.org>; Sat,  8 Aug 2009 06:16:15 -0700 (PDT)
Message-ID: <480411987.1249737375289.JavaMail.jira@brutus>
Date: Sat, 8 Aug 2009 06:16:15 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4012) Providing splitting support for
 bzip2 compressed files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12740894#action_12740894 ] 

Hadoop QA commented on HADOOP-4012:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12415766/Hadoop-4012-version11.patch
  against trunk revision 802224.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/594/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/594/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/594/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/594/console

This message is automatically generated.

> Providing splitting support for bzip2 compressed files
> ------------------------------------------------------
>
>                 Key: HADOOP-4012
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4012
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Abdul Qadeer
>            Assignee: Abdul Qadeer
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-4012-version1.patch, Hadoop-4012-version10.patch, Hadoop-4012-version11.patch, Hadoop-4012-version2.patch, Hadoop-4012-version3.patch, Hadoop-4012-version4.patch, Hadoop-4012-version5.patch, Hadoop-4012-version6.patch, Hadoop-4012-version7.patch, Hadoop-4012-version8.patch, Hadoop-4012-version9.patch
>
>
> Hadoop assumes that if the input data is compressed, it can not be split (mainly due to the limitation of many codecs that they need the whole input stream to decompress successfully).  So in such a case, Hadoop prepares only one split per compressed file, where the lower split limit is at 0 while the upper limit is the end of the file.  The consequence of this decision is that, one compress file goes to a single mapper. Although it circumvents the limitation of codecs (as mentioned above) but reduces the parallelism substantially, as it was possible otherwise in case of splitting.
> BZip2 is a compression / De-Compression algorithm which does compression on blocks of data and later these compressed blocks can be decompressed independent of each other.  This is indeed an opportunity that instead of one BZip2 compressed file going to one mapper, we can process chunks of file in parallel.  The correctness criteria of such a processing is that for a bzip2 compressed file, each compressed block should be processed by only one mapper and ultimately all the blocks of the file should be processed.  (By processing we mean the actual utilization of that un-compressed data (coming out of the codecs) in a mapper).
> We are writing the code to implement this suggested functionality.  Although we have used bzip2 as an example, but we have tried to extend Hadoop's compression interfaces so that any other codecs with the same capability as that of bzip2, could easily use the splitting support.  The details of these changes will be posted when we submit the code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-669-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 09 09:10:30 2009
Return-Path: <common-issues-return-669-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54687 invoked from network); 9 Aug 2009 09:10:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Aug 2009 09:10:30 -0000
Received: (qmail 95967 invoked by uid 500); 9 Aug 2009 09:10:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95918 invoked by uid 500); 9 Aug 2009 09:10:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95903 invoked by uid 99); 9 Aug 2009 09:10:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 09 Aug 2009 09:10:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 09 Aug 2009 09:10:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D6B54234C004
	for <common-issues@hadoop.apache.org>; Sun,  9 Aug 2009 02:10:14 -0700 (PDT)
Message-ID: <612035880.1249809014865.JavaMail.jira@brutus>
Date: Sun, 9 Aug 2009 02:10:14 -0700 (PDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4491) Per-job local data on the
 TaskTracker node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741052#action_12741052 ] 

Hemanth Yamijala commented on HADOOP-4491:
------------------------------------------

Getting close. A few minor points:

- Unrelated to this patch, in Child, we are adding the task's configuration file as a resource to the defaultConf of the Child.  This is used only to get the proxy, one time when the Child starts (before the config file is added). So, it appears the adding of config file to the defaultConf variable can be removed.
- The latest patch is not initializing the task attempt directories on all disks.  This would mean the task may need to create them at runtime under the job directory. We wanted to the job directory to not have writable permissions for the user. Would this not be a problem ?
- In the C code, the constant used for R and X for users is S_IREAD and S_IEXEC. These seem to be obsoleted in favor of S_IRUSR and S_IXUSR. Can we use those instead ?
- hadoop.log.dir in task-controller.cfg is not needed ?
- The test case for testing the permissions of the localized files and directories is very good. Thanks !
- secure_path, change_owner and change_mode can be marked static in task-controller.c as these are somewhat core to the security work in the patch, and therefore need not be exposed.

I also looked at the Forrest documentation and the javadocs:

- NIT: Typo in TaskTracker.initializeJobDirs() javadoc - 'alter' should be 'later'
- NIT: In javadocs, wherever we say 'download from DFS' should probably be 'download from FS', as it could be local also.
- C documentation - needs update for initialize_job, prepare_task_logs

> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt, HADOOP-4491-20090803.1.txt, HADOOP-4491-20090803.txt, HADOOP-4491-20090806.txt, HADOOP-4491-20090807.2.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-670-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 09 16:12:30 2009
Return-Path: <common-issues-return-670-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56227 invoked from network); 9 Aug 2009 16:12:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Aug 2009 16:12:30 -0000
Received: (qmail 89126 invoked by uid 500); 9 Aug 2009 16:12:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89076 invoked by uid 500); 9 Aug 2009 16:12:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89066 invoked by uid 99); 9 Aug 2009 16:12:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 09 Aug 2009 16:12:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 09 Aug 2009 16:12:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DD1DA29A0013
	for <common-issues@hadoop.apache.org>; Sun,  9 Aug 2009 09:12:14 -0700 (PDT)
Message-ID: <941471812.1249834334904.JavaMail.jira@brutus>
Date: Sun, 9 Aug 2009 09:12:14 -0700 (PDT)
From: "Carlos Valiente (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6181) Fixes for Eclipse template
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Fixes for Eclipse template
--------------------------

                 Key: HADOOP-6181
                 URL: https://issues.apache.org/jira/browse/HADOOP-6181
             Project: Hadoop Common
          Issue Type: Bug
          Components: build
    Affects Versions: 0.21.0
            Reporter: Carlos Valiente
            Priority: Minor
             Fix For: 0.21.0


The Avro jar file is missing, and the entry for jets3t does not match the version downloaded by Ivy

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-671-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 09 16:14:30 2009
Return-Path: <common-issues-return-671-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56545 invoked from network); 9 Aug 2009 16:14:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 9 Aug 2009 16:14:30 -0000
Received: (qmail 91118 invoked by uid 500); 9 Aug 2009 16:14:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91066 invoked by uid 500); 9 Aug 2009 16:14:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91056 invoked by uid 99); 9 Aug 2009 16:14:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 09 Aug 2009 16:14:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 09 Aug 2009 16:14:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C51E4234C004
	for <common-issues@hadoop.apache.org>; Sun,  9 Aug 2009 09:14:14 -0700 (PDT)
Message-ID: <1462179356.1249834454793.JavaMail.jira@brutus>
Date: Sun, 9 Aug 2009 09:14:14 -0700 (PDT)
From: "Carlos Valiente (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6181) Fixes for Eclipse template
In-Reply-To: <941471812.1249834334904.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Carlos Valiente updated HADOOP-6181:
------------------------------------

    Attachment: HADOOP-6181.patch

> Fixes for Eclipse template
> --------------------------
>
>                 Key: HADOOP-6181
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6181
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Carlos Valiente
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6181.patch
>
>
> The Avro jar file is missing, and the entry for jets3t does not match the version downloaded by Ivy

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-672-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 10 00:38:30 2009
Return-Path: <common-issues-return-672-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57980 invoked from network); 10 Aug 2009 00:38:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Aug 2009 00:38:30 -0000
Received: (qmail 2307 invoked by uid 500); 10 Aug 2009 00:38:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2249 invoked by uid 500); 10 Aug 2009 00:38:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2239 invoked by uid 99); 10 Aug 2009 00:38:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 00:38:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 00:38:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E5B4229A0018
	for <common-issues@hadoop.apache.org>; Sun,  9 Aug 2009 17:38:14 -0700 (PDT)
Message-ID: <1948161277.1249864694940.JavaMail.jira@brutus>
Date: Sun, 9 Aug 2009 17:38:14 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6133) ReflectionUtils performance
 regression
In-Reply-To: <1343704732.1247090834933.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6133:
--------------------------------

    Status: Patch Available  (was: Open)

> ReflectionUtils performance regression
> --------------------------------------
>
>                 Key: HADOOP-6133
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6133
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>    Affects Versions: 0.20.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-6133-0.20.patch, hadoop-6133-trunk.txt, Test.java
>
>
> HADOOP-4187 introduced extra calls to Class.forName in ReflectionUtils.setConf. This caused a fairly large performance regression. Attached is a microbenchmark that shows the following timings (ms) for 100M constructions of new instances:
> Explicit construction (new Test): around ~1.6sec
> Using Test.class.newInstance: around ~2.6sec
> ReflectionUtils on 0.18.3: ~8.0sec
> ReflectionUtils on 0.20.0: ~200sec
> This illustrates the ~80x slowdown caused by HADOOP-4187.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-673-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 10 00:38:32 2009
Return-Path: <common-issues-return-673-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58034 invoked from network); 10 Aug 2009 00:38:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Aug 2009 00:38:32 -0000
Received: (qmail 2385 invoked by uid 500); 10 Aug 2009 00:38:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2335 invoked by uid 500); 10 Aug 2009 00:38:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2325 invoked by uid 99); 10 Aug 2009 00:38:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 00:38:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 00:38:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CDC5C234C004
	for <common-issues@hadoop.apache.org>; Sun,  9 Aug 2009 17:38:14 -0700 (PDT)
Message-ID: <1074129214.1249864694829.JavaMail.jira@brutus>
Date: Sun, 9 Aug 2009 17:38:14 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6133) ReflectionUtils performance
 regression
In-Reply-To: <1343704732.1247090834933.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6133:
--------------------------------

    Attachment: hadoop-6133-trunk.txt

Here's the same patch against trunk

> ReflectionUtils performance regression
> --------------------------------------
>
>                 Key: HADOOP-6133
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6133
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>    Affects Versions: 0.20.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-6133-0.20.patch, hadoop-6133-trunk.txt, Test.java
>
>
> HADOOP-4187 introduced extra calls to Class.forName in ReflectionUtils.setConf. This caused a fairly large performance regression. Attached is a microbenchmark that shows the following timings (ms) for 100M constructions of new instances:
> Explicit construction (new Test): around ~1.6sec
> Using Test.class.newInstance: around ~2.6sec
> ReflectionUtils on 0.18.3: ~8.0sec
> ReflectionUtils on 0.20.0: ~200sec
> This illustrates the ~80x slowdown caused by HADOOP-4187.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-674-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 10 03:02:30 2009
Return-Path: <common-issues-return-674-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1763 invoked from network); 10 Aug 2009 03:02:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Aug 2009 03:02:30 -0000
Received: (qmail 52404 invoked by uid 500); 10 Aug 2009 03:02:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52354 invoked by uid 500); 10 Aug 2009 03:02:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52339 invoked by uid 99); 10 Aug 2009 03:02:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 03:02:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 03:02:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CE792234C004
	for <common-issues@hadoop.apache.org>; Sun,  9 Aug 2009 20:02:14 -0700 (PDT)
Message-ID: <40782602.1249873334830.JavaMail.jira@brutus>
Date: Sun, 9 Aug 2009 20:02:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6133) ReflectionUtils performance
 regression
In-Reply-To: <1343704732.1247090834933.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741168#action_12741168 ] 

Hadoop QA commented on HADOOP-6133:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12416013/hadoop-6133-trunk.txt
  against trunk revision 802224.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/595/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/595/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/595/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/595/console

This message is automatically generated.

> ReflectionUtils performance regression
> --------------------------------------
>
>                 Key: HADOOP-6133
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6133
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>    Affects Versions: 0.20.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-6133-0.20.patch, hadoop-6133-trunk.txt, Test.java
>
>
> HADOOP-4187 introduced extra calls to Class.forName in ReflectionUtils.setConf. This caused a fairly large performance regression. Attached is a microbenchmark that shows the following timings (ms) for 100M constructions of new instances:
> Explicit construction (new Test): around ~1.6sec
> Using Test.class.newInstance: around ~2.6sec
> ReflectionUtils on 0.18.3: ~8.0sec
> ReflectionUtils on 0.20.0: ~200sec
> This illustrates the ~80x slowdown caused by HADOOP-4187.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-675-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 10 03:36:31 2009
Return-Path: <common-issues-return-675-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11275 invoked from network); 10 Aug 2009 03:36:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Aug 2009 03:36:31 -0000
Received: (qmail 72506 invoked by uid 500); 10 Aug 2009 03:36:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72453 invoked by uid 500); 10 Aug 2009 03:36:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72424 invoked by uid 99); 10 Aug 2009 03:36:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 03:36:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 03:36:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0DBB129A0012
	for <common-issues@hadoop.apache.org>; Sun,  9 Aug 2009 20:36:15 -0700 (PDT)
Message-ID: <719956261.1249875375055.JavaMail.jira@brutus>
Date: Sun, 9 Aug 2009 20:36:15 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6133) ReflectionUtils performance
 regression
In-Reply-To: <1343704732.1247090834933.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741174#action_12741174 ] 

Todd Lipcon commented on HADOOP-6133:
-------------------------------------

No additional tests required - this code path is exercised widely by all parts of Hadoop

> ReflectionUtils performance regression
> --------------------------------------
>
>                 Key: HADOOP-6133
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6133
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>    Affects Versions: 0.20.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-6133-0.20.patch, hadoop-6133-trunk.txt, Test.java
>
>
> HADOOP-4187 introduced extra calls to Class.forName in ReflectionUtils.setConf. This caused a fairly large performance regression. Attached is a microbenchmark that shows the following timings (ms) for 100M constructions of new instances:
> Explicit construction (new Test): around ~1.6sec
> Using Test.class.newInstance: around ~2.6sec
> ReflectionUtils on 0.18.3: ~8.0sec
> ReflectionUtils on 0.20.0: ~200sec
> This illustrates the ~80x slowdown caused by HADOOP-4187.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-676-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 10 05:36:32 2009
Return-Path: <common-issues-return-676-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40133 invoked from network); 10 Aug 2009 05:36:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Aug 2009 05:36:31 -0000
Received: (qmail 23512 invoked by uid 500); 10 Aug 2009 05:36:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23454 invoked by uid 500); 10 Aug 2009 05:36:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23418 invoked by uid 99); 10 Aug 2009 05:36:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 05:36:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 05:36:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 601D929A001B
	for <common-issues@hadoop.apache.org>; Sun,  9 Aug 2009 22:36:15 -0700 (PDT)
Message-ID: <885876995.1249882575392.JavaMail.jira@brutus>
Date: Sun, 9 Aug 2009 22:36:15 -0700 (PDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6111) Support in LocalDirAllocator to set
 permissions of the paths created.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala resolved HADOOP-6111.
--------------------------------------

    Resolution: Later

In discussions that came up on HADOOP-4491, we realized that this change in LocalDirAllocator was no longer needed. Closing this bug for now.

> Support in LocalDirAllocator to set permissions of the paths created.
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6111
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6111
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Vinod K V
>            Assignee: Vinod K V
>
> Support is needed in LocalDirAllocator to set permissions of the paths created. This is needed for HADOOP-4491.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-677-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 10 05:48:30 2009
Return-Path: <common-issues-return-677-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44493 invoked from network); 10 Aug 2009 05:48:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Aug 2009 05:48:30 -0000
Received: (qmail 28998 invoked by uid 500); 10 Aug 2009 05:48:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28941 invoked by uid 500); 10 Aug 2009 05:48:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28930 invoked by uid 99); 10 Aug 2009 05:48:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 05:48:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 05:48:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D0E9C234C004
	for <common-issues@hadoop.apache.org>; Sun,  9 Aug 2009 22:48:14 -0700 (PDT)
Message-ID: <679730774.1249883294841.JavaMail.jira@brutus>
Date: Sun, 9 Aug 2009 22:48:14 -0700 (PDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4491) Per-job local data on the
 TaskTracker node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741194#action_12741194 ] 

Hemanth Yamijala commented on HADOOP-4491:
------------------------------------------

One more point which I discussed with Vinod. The changes in common are at this point just a simple convenience wrapper API around the java.io.File APIs for setting permissions. I propose we move this into the mapred patch itself and open a JIRA to discuss the changes to common more carefully.

> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt, HADOOP-4491-20090803.1.txt, HADOOP-4491-20090803.txt, HADOOP-4491-20090806.txt, HADOOP-4491-20090807.2.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-678-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 10 05:56:32 2009
Return-Path: <common-issues-return-678-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48177 invoked from network); 10 Aug 2009 05:56:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Aug 2009 05:56:32 -0000
Received: (qmail 32122 invoked by uid 500); 10 Aug 2009 05:56:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32060 invoked by uid 500); 10 Aug 2009 05:56:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32048 invoked by uid 99); 10 Aug 2009 05:56:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 05:56:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 05:56:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E7A1229A0017
	for <common-issues@hadoop.apache.org>; Sun,  9 Aug 2009 22:56:14 -0700 (PDT)
Message-ID: <739323326.1249883774947.JavaMail.jira@brutus>
Date: Sun, 9 Aug 2009 22:56:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6182) Adding Apache License Headers and
 reduce releaseaudit warnings to zero
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Adding Apache License Headers and reduce releaseaudit warnings to zero
----------------------------------------------------------------------

                 Key: HADOOP-6182
                 URL: https://issues.apache.org/jira/browse/HADOOP-6182
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Giridharan Kesavan


As of now rats tool shows 111 RA warnings 

[rat:report] Summary
[rat:report] -------
[rat:report] Notes: 18
[rat:report] Binaries: 118
[rat:report] Archives: 33
[rat:report] Standards: 942
[rat:report] 
[rat:report] Apache Licensed: 820
[rat:report] Generated Documents: 11
[rat:report] 
[rat:report] JavaDocs are generated and so license header is optional
[rat:report] Generated files do not required license headers
[rat:report] 
[rat:report] 111 Unknown Licenses
[rat:report] 
[rat:report] *******************************

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-679-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 10 07:04:31 2009
Return-Path: <common-issues-return-679-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82293 invoked from network); 10 Aug 2009 07:04:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Aug 2009 07:04:31 -0000
Received: (qmail 95161 invoked by uid 500); 10 Aug 2009 07:04:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95086 invoked by uid 500); 10 Aug 2009 07:04:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94906 invoked by uid 99); 10 Aug 2009 07:04:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 07:04:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 07:04:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DB3AB234C004
	for <common-issues@hadoop.apache.org>; Mon, 10 Aug 2009 00:04:14 -0700 (PDT)
Message-ID: <63524206.1249887854882.JavaMail.jira@brutus>
Date: Mon, 10 Aug 2009 00:04:14 -0700 (PDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4491) Per-job local data on the TaskTracker
 node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V updated HADOOP-4491:
------------------------------

    Attachment: HADOOP-4491-20090810.1.txt

Attaching patch including most of the above comments.

bq. hadoop.log.dir in task-controller.cfg is not needed?
hadoop.log.dir is still needed to set permissions on userlogs/attempt-id directories to be writable by the user's tasks and readable by TT for serving via web UI.
bq. One more point which I discussed with Vinod. The changes in common are at this point just a simple convenience wrapper API around the java.io.File APIs for setting permissions. I propose we move this into the mapred patch itself and open a JIRA to discuss the changes to common more carefully.
Moved these changes into TaskTracker.PermissionsHandler. In future (via MAPREDUCE-759),  this along with most of the remaining localization related code should be refactored into a separate class.


> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt, HADOOP-4491-20090803.1.txt, HADOOP-4491-20090803.txt, HADOOP-4491-20090806.txt, HADOOP-4491-20090807.2.txt, HADOOP-4491-20090810.1.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-680-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 10 07:04:32 2009
Return-Path: <common-issues-return-680-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82365 invoked from network); 10 Aug 2009 07:04:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Aug 2009 07:04:32 -0000
Received: (qmail 95482 invoked by uid 500); 10 Aug 2009 07:04:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95441 invoked by uid 500); 10 Aug 2009 07:04:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95427 invoked by uid 99); 10 Aug 2009 07:04:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 07:04:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 07:04:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0F1B529A001D
	for <common-issues@hadoop.apache.org>; Mon, 10 Aug 2009 00:04:15 -0700 (PDT)
Message-ID: <547966737.1249887855061.JavaMail.jira@brutus>
Date: Mon, 10 Aug 2009 00:04:15 -0700 (PDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4491) Per-job local data on the TaskTracker
 node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V updated HADOOP-4491:
------------------------------

    Status: Patch Available  (was: Open)

> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt, HADOOP-4491-20090803.1.txt, HADOOP-4491-20090803.txt, HADOOP-4491-20090806.txt, HADOOP-4491-20090807.2.txt, HADOOP-4491-20090810.1.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-681-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 10 09:00:32 2009
Return-Path: <common-issues-return-681-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4079 invoked from network); 10 Aug 2009 09:00:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Aug 2009 09:00:32 -0000
Received: (qmail 94024 invoked by uid 500); 10 Aug 2009 09:00:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94001 invoked by uid 500); 10 Aug 2009 09:00:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93991 invoked by uid 99); 10 Aug 2009 09:00:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 09:00:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 09:00:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 07B74234C04C
	for <common-issues@hadoop.apache.org>; Mon, 10 Aug 2009 02:00:15 -0700 (PDT)
Message-ID: <1354386720.1249894815030.JavaMail.jira@brutus>
Date: Mon, 10 Aug 2009 02:00:15 -0700 (PDT)
From: "Anjali M (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6183) improvement in fairscheduler
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

improvement in fairscheduler
----------------------------

                 Key: HADOOP-6183
                 URL: https://issues.apache.org/jira/browse/HADOOP-6183
             Project: Hadoop Common
          Issue Type: Improvement
          Components: contrib/hod
    Affects Versions: 0.19.1
         Environment: Fedora9
            Reporter: Anjali M
            Priority: Minor


I understand the hadoop fair scheduler does the scheduling at a task level. I think in the pool of jobs if we can queue up the jobs based on a backfilling strategy, the delay which long jobs might face in the current strategy, can be reduced fruther.  Has anybody tried such a strategy?


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-682-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 10 09:21:32 2009
Return-Path: <common-issues-return-682-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8614 invoked from network); 10 Aug 2009 09:21:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Aug 2009 09:21:32 -0000
Received: (qmail 6392 invoked by uid 500); 10 Aug 2009 09:21:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6342 invoked by uid 500); 10 Aug 2009 09:21:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6332 invoked by uid 99); 10 Aug 2009 09:21:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 09:21:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 09:21:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 285DC234C004
	for <common-issues@hadoop.apache.org>; Mon, 10 Aug 2009 02:21:15 -0700 (PDT)
Message-ID: <1089666315.1249896075160.JavaMail.jira@brutus>
Date: Mon, 10 Aug 2009 02:21:15 -0700 (PDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4491) Per-job local data on the
 TaskTracker node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741239#action_12741239 ] 

Hemanth Yamijala commented on HADOOP-4491:
------------------------------------------

Can we also weed out trace type log statements both in the the C and Java code, move them to a debug mode if needed ?

> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt, HADOOP-4491-20090803.1.txt, HADOOP-4491-20090803.txt, HADOOP-4491-20090806.txt, HADOOP-4491-20090807.2.txt, HADOOP-4491-20090810.1.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-683-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 10 10:15:30 2009
Return-Path: <common-issues-return-683-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39895 invoked from network); 10 Aug 2009 10:15:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Aug 2009 10:15:30 -0000
Received: (qmail 56494 invoked by uid 500); 10 Aug 2009 10:15:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56448 invoked by uid 500); 10 Aug 2009 10:15:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56438 invoked by uid 99); 10 Aug 2009 10:15:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 10:15:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 10:15:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CD1E6234C004
	for <common-issues@hadoop.apache.org>; Mon, 10 Aug 2009 03:15:14 -0700 (PDT)
Message-ID: <653522500.1249899314825.JavaMail.jira@brutus>
Date: Mon, 10 Aug 2009 03:15:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4491) Per-job local data on the
 TaskTracker node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741250#action_12741250 ] 

Hadoop QA commented on HADOOP-4491:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12416027/HADOOP-4491-20090810.1.txt
  against trunk revision 802224.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 47 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/596/console

This message is automatically generated.

> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt, HADOOP-4491-20090803.1.txt, HADOOP-4491-20090803.txt, HADOOP-4491-20090806.txt, HADOOP-4491-20090807.2.txt, HADOOP-4491-20090810.1.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-684-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 10 10:19:30 2009
Return-Path: <common-issues-return-684-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43573 invoked from network); 10 Aug 2009 10:19:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Aug 2009 10:19:30 -0000
Received: (qmail 60052 invoked by uid 500); 10 Aug 2009 10:19:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60001 invoked by uid 500); 10 Aug 2009 10:19:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59991 invoked by uid 99); 10 Aug 2009 10:19:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 10:19:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 10:19:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CA420234C004
	for <common-issues@hadoop.apache.org>; Mon, 10 Aug 2009 03:19:14 -0700 (PDT)
Message-ID: <1949150256.1249899554816.JavaMail.jira@brutus>
Date: Mon, 10 Aug 2009 03:19:14 -0700 (PDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4491) Per-job local data on the TaskTracker
 node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V updated HADOOP-4491:
------------------------------

    Status: Open  (was: Patch Available)

> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt, HADOOP-4491-20090803.1.txt, HADOOP-4491-20090803.txt, HADOOP-4491-20090806.txt, HADOOP-4491-20090807.2.txt, HADOOP-4491-20090810.1.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-685-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 10 10:23:34 2009
Return-Path: <common-issues-return-685-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46777 invoked from network); 10 Aug 2009 10:23:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Aug 2009 10:23:33 -0000
Received: (qmail 61906 invoked by uid 500); 10 Aug 2009 10:23:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61696 invoked by uid 500); 10 Aug 2009 10:23:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61664 invoked by uid 99); 10 Aug 2009 10:23:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 10:23:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 10:23:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D796C234C004
	for <common-issues@hadoop.apache.org>; Mon, 10 Aug 2009 03:23:14 -0700 (PDT)
Message-ID: <873828177.1249899794869.JavaMail.jira@brutus>
Date: Mon, 10 Aug 2009 03:23:14 -0700 (PDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4491) Per-job local data on the TaskTracker
 node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V updated HADOOP-4491:
------------------------------

    Attachment: HADOOP-4491-20090810.3.txt

Updated patch that
 - Downgrades some of the log statements from info to DEBUG level, both in JAVA as well as the c code
 - Removes unused exit codes, macros and method declarations
 - Fixes some of the glitches in Forrest documentation

> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt, HADOOP-4491-20090803.1.txt, HADOOP-4491-20090803.txt, HADOOP-4491-20090806.txt, HADOOP-4491-20090807.2.txt, HADOOP-4491-20090810.1.txt, HADOOP-4491-20090810.3.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-686-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 10 11:09:30 2009
Return-Path: <common-issues-return-686-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65588 invoked from network); 10 Aug 2009 11:09:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Aug 2009 11:09:30 -0000
Received: (qmail 95403 invoked by uid 500); 10 Aug 2009 11:09:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95352 invoked by uid 500); 10 Aug 2009 11:09:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95342 invoked by uid 99); 10 Aug 2009 11:09:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 11:09:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 11:09:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D833F234C04C
	for <common-issues@hadoop.apache.org>; Mon, 10 Aug 2009 04:09:14 -0700 (PDT)
Message-ID: <1276417070.1249902554884.JavaMail.jira@brutus>
Date: Mon, 10 Aug 2009 04:09:14 -0700 (PDT)
From: "rahul k singh (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6184) Provide a configuration dump in
 json format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741262#action_12741262 ] 

rahul k singh commented on HADOOP-6184:
---------------------------------------

In order to generate the  dump in standard format (mapreduce - 768 )
we need to provide a way in Configuration to generate dump in json format.

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-687-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 10 11:09:33 2009
Return-Path: <common-issues-return-687-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65652 invoked from network); 10 Aug 2009 11:09:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Aug 2009 11:09:33 -0000
Received: (qmail 95714 invoked by uid 500); 10 Aug 2009 11:09:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95619 invoked by uid 500); 10 Aug 2009 11:09:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95428 invoked by uid 99); 10 Aug 2009 11:09:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 11:09:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 11:09:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D4C4E234C044
	for <common-issues@hadoop.apache.org>; Mon, 10 Aug 2009 04:09:14 -0700 (PDT)
Message-ID: <1775349108.1249902554870.JavaMail.jira@brutus>
Date: Mon, 10 Aug 2009 04:09:14 -0700 (PDT)
From: "rahul k singh (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6184) Provide a configuration dump in json
 format.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Provide a configuration dump in json format.
--------------------------------------------

                 Key: HADOOP-6184
                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: rahul k singh


Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-688-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 10 11:30:32 2009
Return-Path: <common-issues-return-688-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69868 invoked from network); 10 Aug 2009 11:30:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Aug 2009 11:30:32 -0000
Received: (qmail 15405 invoked by uid 500); 10 Aug 2009 11:30:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15355 invoked by uid 500); 10 Aug 2009 11:30:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15345 invoked by uid 99); 10 Aug 2009 11:30:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 11:30:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 11:30:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DE604234C004
	for <common-issues@hadoop.apache.org>; Mon, 10 Aug 2009 04:30:14 -0700 (PDT)
Message-ID: <555071519.1249903814895.JavaMail.jira@brutus>
Date: Mon, 10 Aug 2009 04:30:14 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6184) Provide a configuration dump in json
 format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna updated HADOOP-6184:
------------------------------------------

    Attachment: HADOOP-6184.patch

Uploading patch which provides method to return all the properties of configuration object as a string of Json format.

The Json format is as follows:

{ "Properties" : [
       { "Key" : <key>,
          "Value" : <value>,
          "isFinal" : <true if the property is set to final>
          "resource" : <resource name which loads this property most recently> }
   ] }


> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>         Attachments: HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-689-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 10 11:56:32 2009
Return-Path: <common-issues-return-689-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73620 invoked from network); 10 Aug 2009 11:56:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Aug 2009 11:56:32 -0000
Received: (qmail 42318 invoked by uid 500); 10 Aug 2009 11:56:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42261 invoked by uid 500); 10 Aug 2009 11:56:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42197 invoked by uid 99); 10 Aug 2009 11:56:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 11:56:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 11:56:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 193E1234C004
	for <common-issues@hadoop.apache.org>; Mon, 10 Aug 2009 04:56:15 -0700 (PDT)
Message-ID: <1145553108.1249905375089.JavaMail.jira@brutus>
Date: Mon, 10 Aug 2009 04:56:15 -0700 (PDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4491) Per-job local data on the TaskTracker
 node should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V updated HADOOP-4491:
------------------------------

    Issue Type: Bug  (was: Sub-task)
        Parent:     (was: HADOOP-4487)

> Per-job local data on the TaskTracker node should have right access-control
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-4491
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4491
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: tasktracker
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>         Attachments: HADOOP-4491-20090623-common.1.txt, HADOOP-4491-20090623-mapred.1.txt, HADOOP-4491-20090703-common.1.txt, HADOOP-4491-20090703-common.txt, HADOOP-4491-20090703.1.txt, HADOOP-4491-20090703.txt, HADOOP-4491-20090707-common.txt, HADOOP-4491-20090707.txt, HADOOP-4491-20090716-mapred.txt, HADOOP-4491-20090803.1.txt, HADOOP-4491-20090803.txt, HADOOP-4491-20090806.txt, HADOOP-4491-20090807.2.txt, HADOOP-4491-20090810.1.txt, HADOOP-4491-20090810.3.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-690-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 10 16:44:32 2009
Return-Path: <common-issues-return-690-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96709 invoked from network); 10 Aug 2009 16:44:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Aug 2009 16:44:32 -0000
Received: (qmail 41014 invoked by uid 500); 10 Aug 2009 16:44:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40982 invoked by uid 500); 10 Aug 2009 16:44:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40972 invoked by uid 99); 10 Aug 2009 16:44:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 16:44:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 16:44:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CACDF234C004
	for <common-issues@hadoop.apache.org>; Mon, 10 Aug 2009 09:44:14 -0700 (PDT)
Message-ID: <480433999.1249922654816.JavaMail.jira@brutus>
Date: Mon, 10 Aug 2009 09:44:14 -0700 (PDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6177) FSInputChecker.getPos() would
 return position greater than the file size
In-Reply-To: <1554871263.1249618274995.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741419#action_12741419 ] 

Hairong Kuang commented on HADOOP-6177:
---------------------------------------

Good catch, Hong!
+1 The patch looks good.

> FSInputChecker.getPos() would return position greater than the file size
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6177
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6177
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6177-2090806.patch, hadoop-6177-testcase.patch
>
>
> When using a small buffer (< 512 bytes) to read through the whole file, the final file position is 1+ the file size.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-691-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 10 17:45:32 2009
Return-Path: <common-issues-return-691-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19651 invoked from network); 10 Aug 2009 17:45:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Aug 2009 17:45:32 -0000
Received: (qmail 19130 invoked by uid 500); 10 Aug 2009 17:45:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19049 invoked by uid 500); 10 Aug 2009 17:45:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19024 invoked by uid 99); 10 Aug 2009 17:45:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 17:45:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 17:45:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3F0E3234C004
	for <common-issues@hadoop.apache.org>; Mon, 10 Aug 2009 10:45:15 -0700 (PDT)
Message-ID: <342347492.1249926315244.JavaMail.jira@brutus>
Date: Mon, 10 Aug 2009 10:45:15 -0700 (PDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4493) Localized files from
 DistributedCache should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741439#action_12741439 ] 

Hemanth Yamijala commented on HADOOP-4493:
------------------------------------------

Had a chat with Owen to get more details. Here is the summary:

 - To download the files from distributed cache, we will use the user's JobConf for getting the file system handle. Owen feels any changes to Hadoop security will be abstracted here and hence we will not be affected.
- In this JIRA, we'll manage a per user cache where the downloaded cache files will be managed securely as done in MAPREDUCE-842 (ownership set to user, group ownership to the tasktracker user, 570 permissions).
- The per user cache will be under $ttroot/users/$user/archive
- Owen suggested we can change the jobcache also to be under $ttroot/users/$user/jobcache as part of this jira. (However, we will not change the direction of MAPREDUCE-842 which is quite close to the finish line now)
- There'll be symlinks from the task directories to the distributed cache files. 
- The current paging policy which sets a soft limit on the size of the distributed cache on disk will remain unchanged. We discussed there could be potentially work in the future to make this a hard limit, which could even have impacts on scheduling in the sense that the scheduler should schedule tasks to be within the limits). But this is certainly the scope of another jira.

> Localized files from DistributedCache should have right access-control
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-4493
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4493
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-692-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 10 19:58:30 2009
Return-Path: <common-issues-return-692-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73908 invoked from network); 10 Aug 2009 19:58:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Aug 2009 19:58:30 -0000
Received: (qmail 10888 invoked by uid 500); 10 Aug 2009 19:58:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10834 invoked by uid 500); 10 Aug 2009 19:58:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10824 invoked by uid 99); 10 Aug 2009 19:58:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 19:58:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 19:58:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CBF00234C004
	for <common-issues@hadoop.apache.org>; Mon, 10 Aug 2009 12:58:14 -0700 (PDT)
Message-ID: <2027490637.1249934294821.JavaMail.jira@brutus>
Date: Mon, 10 Aug 2009 12:58:14 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6184) Provide a configuration dump in
 json format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741519#action_12741519 ] 

Doug Cutting commented on HADOOP-6184:
--------------------------------------

What's the purpose of the "resource" field?

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>         Attachments: HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-693-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 10 22:46:33 2009
Return-Path: <common-issues-return-693-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33986 invoked from network); 10 Aug 2009 22:46:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Aug 2009 22:46:33 -0000
Received: (qmail 11678 invoked by uid 500); 10 Aug 2009 22:46:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11623 invoked by uid 500); 10 Aug 2009 22:46:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11613 invoked by uid 99); 10 Aug 2009 22:46:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 22:46:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 22:46:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E8E91234C1EC
	for <common-issues@hadoop.apache.org>; Mon, 10 Aug 2009 15:46:14 -0700 (PDT)
Message-ID: <1776076031.1249944374953.JavaMail.jira@brutus>
Date: Mon, 10 Aug 2009 15:46:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6166:
-------------------------------------------

    Attachment: c6166_20090810.patch

>> I think there may be some improvement possible in the 8_8 case in how the 9 XORs at the end are done. ...
>Yeah, we should try.

c6166_20090810.patch: tried various xor schemes for 8_8 and 16_16.

Seems Crc32_8_8d is the best choice.  I will generate a patch to replace current PureJavaCrc32 unless someone would like to run some benchmarks.

- Linux
java.version = 1.6.0_15
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_15-b03
java.vm.version = 14.1-b02
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = amd64
os.name = Linux
os.version = 2.6.9-55.ELsmp

Performance Table (The unit is MB/sec)
|| Num Bytes ||    CRC32 || PureJavaCrc32 || Crc32_8_8 || Crc32_8_8b || Crc32_8_8c || Crc32_8_8d || Crc32_16_16 || Crc32_16_16b || Crc32_16_16c || Crc32_16_16d ||
|          8 |    49.617 |        186.277 |    245.369 |     258.905 |     229.094 |     246.095 |      217.684 |       209.102 |       225.104 |       216.537 |
|         16 |    83.794 |        260.552 |    254.068 |     253.335 |     240.414 |     269.863 |      268.293 |       280.991 |       248.725 |       272.588 |
|         32 |   129.851 |        311.725 |    319.791 |     317.253 |     300.914 |     325.145 |      294.224 |       306.928 |       294.640 |       294.838 |
|         64 |   175.566 |        348.701 |    357.292 |     359.353 |     345.997 |     370.976 |      325.410 |       349.629 |       343.592 |       346.502 |
|        128 |   217.394 |        369.562 |    386.217 |     384.014 |     371.929 |     392.099 |      339.749 |       382.753 |       378.737 |       381.048 |
|        256 |   246.060 |        381.159 |    403.724 |     401.239 |     389.683 |     403.455 |      346.794 |       394.901 |       397.287 |       398.093 |
|        512 |   261.941 |        385.598 |    412.253 |     405.512 |     395.790 |     412.212 |      350.763 |       409.029 |       408.213 |       407.630 |
|       1024 |   271.043 |        390.318 |    408.391 |     408.592 |     398.981 |     413.389 |      352.213 |       414.423 |       414.213 |       413.736 |
|       2048 |   275.942 |        391.870 |    412.933 |     411.371 |     401.784 |     417.020 |      351.803 |       418.063 |       416.982 |       415.269 |
|       4096 |   280.018 |        393.816 |    412.645 |     411.781 |     402.632 |     418.353 |      353.348 |       420.783 |       416.263 |       416.360 |
|       8192 |   281.432 |        394.699 |    414.998 |     410.618 |     401.145 |     417.090 |      351.428 |       420.008 |       415.680 |       415.281 |
|      16384 |   279.658 |        391.788 |    413.657 |     411.491 |     402.747 |     418.249 |      349.888 |       419.431 |       414.399 |       414.312 |
|      32768 |   279.788 |        391.665 |    408.767 |     410.097 |     403.396 |     417.542 |      349.473 |       418.467 |       409.388 |       412.525 |
|      65536 |   280.168 |        391.557 |    411.674 |     410.447 |     404.971 |     419.271 |      350.824 |       419.866 |       412.099 |       413.847 |
|     131072 |   281.196 |        393.222 |    411.687 |     411.539 |     404.931 |     418.101 |      350.692 |       418.924 |       412.873 |       413.763 |
|     262144 |   281.874 |        392.158 |    411.668 |     411.660 |     405.089 |     419.301 |      350.236 |       418.776 |       412.855 |       413.918 |
|     524288 |   281.905 |        392.967 |    411.713 |     410.524 |     404.851 |     418.481 |      350.890 |       419.835 |       412.881 |       412.677 |
|    1048576 |   281.864 |        393.155 |    411.589 |     410.697 |     405.352 |     417.905 |      350.737 |       419.861 |       412.734 |       413.808 |
|    2097152 |   281.269 |        392.916 |    410.495 |     411.322 |     405.298 |     419.084 |      349.765 |       419.540 |       412.517 |       413.538 |
|    4194304 |   280.904 |        388.513 |    408.313 |     410.383 |     404.432 |     418.168 |      349.982 |       418.628 |       411.404 |       412.493 |
|    8388608 |   279.946 |        389.306 |    407.485 |     407.420 |     401.088 |     415.597 |      347.824 |       415.479 |       408.459 |       409.290 |
|   16777216 |   279.517 |        388.903 |    407.068 |     406.760 |     400.404 |     415.171 |      347.468 |       415.047 |       408.089 |       408.576 |

- Windows
java.version = 1.6.0_14
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_14-b08
java.vm.version = 14.0-b16
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = x86
os.name = Windows XP
os.version = 5.1

Performance Table (The unit is MB/sec)
|| Num Bytes ||     CRC32 || PureJavaCrc32 || Crc32_8_8 || Crc32_8_8b || Crc32_8_8c || Crc32_8_8d || Crc32_16_16 || Crc32_16_16b || Crc32_16_16c || Crc32_16_16d ||
|          8 |    30.582 |        165.684 |    222.712 |     239.031 |     192.301 |     192.855 |      121.520 |       118.156 |       121.212 |       119.522 |
|         16 |    54.720 |        215.567 |    237.607 |     232.193 |     230.172 |     215.288 |      269.855 |       279.729 |       254.891 |       251.049 |
|         32 |    93.370 |        250.110 |    290.161 |     300.234 |     272.414 |     266.170 |      288.042 |       293.170 |       273.936 |       278.459 |
|         64 |   142.510 |        271.193 |    317.292 |     331.031 |     293.492 |     298.169 |      314.142 |       310.036 |       309.833 |       315.505 |
|        128 |   193.336 |        285.876 |    340.127 |     355.663 |     302.081 |     319.978 |      330.662 |       318.949 |       326.466 |       333.227 |
|        256 |   237.137 |        292.450 |    349.322 |     366.198 |     308.711 |     328.690 |      335.694 |       323.855 |       328.766 |       342.473 |
|        512 |   266.200 |        293.008 |    352.708 |     370.587 |     307.753 |     330.876 |      335.443 |       320.305 |       338.866 |       342.883 |
|       1024 |   275.926 |        292.009 |    347.310 |     367.795 |     305.459 |     332.800 |      328.240 |       291.745 |       270.544 |       338.249 |
|       2048 |   285.404 |        282.665 |    338.805 |     356.377 |     300.794 |     333.204 |      334.669 |       317.972 |       327.810 |       347.768 |
|       4096 |   295.626 |        296.351 |    353.792 |     374.937 |     310.589 |     337.225 |      338.851 |       323.519 |       335.347 |       349.135 |
|       8192 |   299.106 |        297.335 |    352.159 |     371.486 |     310.073 |     334.584 |      336.462 |       320.744 |       340.072 |       343.119 |
|      16384 |   298.202 |        295.121 |    353.141 |     371.399 |     311.208 |     347.209 |      340.819 |       321.550 |       341.870 |       349.779 |
|      32768 |   301.017 |        294.772 |    355.933 |     374.572 |     311.367 |     354.163 |      335.798 |       320.324 |       344.780 |       351.859 |
|      65536 |   305.349 |        297.090 |    355.641 |     377.599 |     310.232 |     354.429 |      337.837 |       324.306 |       345.244 |       350.304 |
|     131072 |   305.814 |        297.094 |    356.518 |     380.588 |     310.183 |     354.515 |      339.324 |       319.657 |       344.429 |       347.836 |
|     262144 |   303.674 |        294.812 |    352.616 |     377.256 |     306.132 |     349.959 |      330.275 |       319.694 |       339.153 |       347.561 |
|     524288 |   306.440 |        296.660 |    352.876 |     371.992 |     308.378 |     353.454 |      339.735 |       322.835 |       344.267 |       350.561 |
|    1048576 |   303.807 |        297.499 |    351.199 |     375.630 |     307.785 |     350.496 |      338.899 |       321.673 |       342.487 |       350.763 |
|    2097152 |   302.701 |        295.785 |    352.618 |     376.966 |     306.529 |     351.045 |      335.405 |       318.857 |       340.192 |       349.454 |
|    4194304 |   300.318 |        293.945 |    345.822 |     373.705 |     306.829 |     348.913 |      334.297 |       318.580 |       339.454 |       346.533 |
|    8388608 |   299.847 |        293.259 |    348.667 |     373.015 |     305.414 |     338.622 |      333.332 |       317.337 |       339.085 |       346.144 |
|   16777216 |   298.157 |        293.100 |    349.116 |     372.502 |     306.273 |     348.192 |      334.259 |       316.192 |       336.934 |       347.545 |

> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt, c6166_20090727.patch, c6166_20090728.patch, c6166_20090810.patch
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-694-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 10 23:26:37 2009
Return-Path: <common-issues-return-694-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43163 invoked from network); 10 Aug 2009 23:26:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Aug 2009 23:26:37 -0000
Received: (qmail 47028 invoked by uid 500); 10 Aug 2009 23:26:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46973 invoked by uid 500); 10 Aug 2009 23:26:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46953 invoked by uid 99); 10 Aug 2009 23:26:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 23:26:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2009 23:26:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CF312234C004
	for <common-issues@hadoop.apache.org>; Mon, 10 Aug 2009 16:26:14 -0700 (PDT)
Message-ID: <994710173.1249946774833.JavaMail.jira@brutus>
Date: Mon, 10 Aug 2009 16:26:14 -0700 (PDT)
From: "Jay Booth (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6152) Hadoop scripts do not correctly put
 jars on the classpath
In-Reply-To: <283943906.1247689875148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741616#action_12741616 ] 

Jay Booth commented on HADOOP-6152:
-----------------------------------

Seems like hadoop-config.sh doesn't include the contrib jars from hdfs or mapred..  is this by design, are they listed elsewhere?

> Hadoop scripts do not correctly put jars on the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-6152
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6152
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>            Priority: Blocker
>         Attachments: HADOOP-6152.patch
>
>
> The various Hadoop scripts (bin/hadoop, bin/hdfs, bin/mapred) do not properly identify the jars needed to run Hadoop. They try to include hadoop-*-hdfs.jar, etc, rather than the hadoop-hdfs-*.jar that is actually created by the 'ant jar' and 'ant package' targets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-695-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 00:42:33 2009
Return-Path: <common-issues-return-695-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66609 invoked from network); 11 Aug 2009 00:42:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 00:42:33 -0000
Received: (qmail 13895 invoked by uid 500); 11 Aug 2009 00:42:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13818 invoked by uid 500); 11 Aug 2009 00:42:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13525 invoked by uid 99); 11 Aug 2009 00:42:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 00:42:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 00:42:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 2DD3129A0011
	for <common-issues@hadoop.apache.org>; Mon, 10 Aug 2009 17:42:15 -0700 (PDT)
Message-ID: <685653094.1249951335186.JavaMail.jira@brutus>
Date: Mon, 10 Aug 2009 17:42:15 -0700 (PDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6185) Replace FSDataOutputStream#sync() by
 hflush()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Replace FSDataOutputStream#sync() by hflush()
---------------------------------------------

                 Key: HADOOP-6185
                 URL: https://issues.apache.org/jira/browse/HADOOP-6185
             Project: Hadoop Common
          Issue Type: New Feature
          Components: fs
    Affects Versions: 0.21.0
            Reporter: Hairong Kuang
            Assignee: Hairong Kuang
            Priority: Blocker
             Fix For: 0.21.0


This jira aims add hflush() API to FSDataOutputStream and deprecates sync() API.

Note that the change should not commit to the trunk before hdfs append branch is merged to hdfs trunk.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-696-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 00:46:31 2009
Return-Path: <common-issues-return-696-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67235 invoked from network); 11 Aug 2009 00:46:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 00:46:31 -0000
Received: (qmail 16441 invoked by uid 500); 11 Aug 2009 00:46:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16390 invoked by uid 500); 11 Aug 2009 00:46:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16379 invoked by uid 99); 11 Aug 2009 00:46:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 00:46:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 00:46:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id AC51929A0014
	for <common-issues@hadoop.apache.org>; Mon, 10 Aug 2009 17:46:15 -0700 (PDT)
Message-ID: <1679160921.1249951575704.JavaMail.jira@brutus>
Date: Mon, 10 Aug 2009 17:46:15 -0700 (PDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5107) split the core, hdfs, and mapred
 jars from each other and publish them independently to the Maven repository
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741646#action_12741646 ] 

Arun C Murthy commented on HADOOP-5107:
---------------------------------------

Giri, any updates on this patch? It would be really nice to get people to resolver=local to do development across the common, hdfs and mapreduce sub-projects.

> split the core, hdfs, and mapred jars from each other and publish them independently to the Maven repository
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5107
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5107
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.0
>            Reporter: Owen O'Malley
>            Assignee: Giridharan Kesavan
>         Attachments: common-trunk.patch, hdfs-trunk.patch, mapreduce-trunk.patch
>
>
> I think to support splitting the projects, we should publish the jars for 0.20.0 as independent jars to the Maven repository 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-697-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 00:54:30 2009
Return-Path: <common-issues-return-697-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69392 invoked from network); 11 Aug 2009 00:54:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 00:54:30 -0000
Received: (qmail 20954 invoked by uid 500); 11 Aug 2009 00:54:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20903 invoked by uid 500); 11 Aug 2009 00:54:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20880 invoked by uid 99); 11 Aug 2009 00:54:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 00:54:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 00:54:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 35588234C004
	for <common-issues@hadoop.apache.org>; Mon, 10 Aug 2009 17:54:15 -0700 (PDT)
Message-ID: <1012715155.1249952055203.JavaMail.jira@brutus>
Date: Mon, 10 Aug 2009 17:54:15 -0700 (PDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6185) Replace FSDataOutputStream#sync() by
 hflush()
In-Reply-To: <685653094.1249951335186.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6185:
----------------------------------

    Attachment: commonHflush.patch

> Replace FSDataOutputStream#sync() by hflush()
> ---------------------------------------------
>
>                 Key: HADOOP-6185
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6185
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: commonHflush.patch
>
>
> This jira aims add hflush() API to FSDataOutputStream and deprecates sync() API.
> Note that the change should not commit to the trunk before hdfs append branch is merged to hdfs trunk.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-698-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 01:32:30 2009
Return-Path: <common-issues-return-698-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83310 invoked from network); 11 Aug 2009 01:32:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 01:32:30 -0000
Received: (qmail 52328 invoked by uid 500); 11 Aug 2009 01:32:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52275 invoked by uid 500); 11 Aug 2009 01:32:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52264 invoked by uid 99); 11 Aug 2009 01:32:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 01:32:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 01:32:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 2E1D1234C004
	for <common-issues@hadoop.apache.org>; Mon, 10 Aug 2009 18:32:15 -0700 (PDT)
Message-ID: <2076978097.1249954335173.JavaMail.jira@brutus>
Date: Mon, 10 Aug 2009 18:32:15 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6152) Hadoop scripts do not correctly put
 jars on the classpath
In-Reply-To: <283943906.1247689875148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6152:
-------------------------------------------

    Component/s: scripts

The patch cannot be applied.  Could you update it?

> Hadoop scripts do not correctly put jars on the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-6152
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6152
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>            Priority: Blocker
>         Attachments: HADOOP-6152.patch
>
>
> The various Hadoop scripts (bin/hadoop, bin/hdfs, bin/mapred) do not properly identify the jars needed to run Hadoop. They try to include hadoop-*-hdfs.jar, etc, rather than the hadoop-hdfs-*.jar that is actually created by the 'ant jar' and 'ant package' targets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-699-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 03:54:30 2009
Return-Path: <common-issues-return-699-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31624 invoked from network); 11 Aug 2009 03:54:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 03:54:30 -0000
Received: (qmail 28464 invoked by uid 500); 11 Aug 2009 03:54:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28411 invoked by uid 500); 11 Aug 2009 03:54:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28398 invoked by uid 99); 11 Aug 2009 03:54:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 03:54:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 03:54:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 37D80234C004
	for <common-issues@hadoop.apache.org>; Mon, 10 Aug 2009 20:54:15 -0700 (PDT)
Message-ID: <1765575332.1249962855214.JavaMail.jira@brutus>
Date: Mon, 10 Aug 2009 20:54:15 -0700 (PDT)
From: "Sreekanth Ramakrishnan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6184) Provide a configuration dump in
 json format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741679#action_12741679 ] 

Sreekanth Ramakrishnan commented on HADOOP-6184:
------------------------------------------------

in writeJson

* reloadConfiguration() call is not required.
* Can we make the method a static method with signature as following public static void writeJson(Configuration conf, Writer writer)
for(Map.Entry<Object,Object> item: getProps().entrySet()) { should be changed to  for(Map.Entry<Object,Object> item: conf.getProps().entrySet()) { where conf is your local variable.
* uncomment the line dumpGenerator.flush(); while iterating over each key
* Please dont import like import org.codehaus.jackson.*; import specific classes and interfaces.
* Comments in testcase.

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>         Attachments: HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-700-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 04:44:30 2009
Return-Path: <common-issues-return-700-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36874 invoked from network); 11 Aug 2009 04:44:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 04:44:30 -0000
Received: (qmail 50729 invoked by uid 500); 11 Aug 2009 04:44:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50676 invoked by uid 500); 11 Aug 2009 04:44:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50657 invoked by uid 99); 11 Aug 2009 04:44:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 04:44:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 04:44:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D7E30234C044
	for <common-issues@hadoop.apache.org>; Mon, 10 Aug 2009 21:44:14 -0700 (PDT)
Message-ID: <373147619.1249965854883.JavaMail.jira@brutus>
Date: Mon, 10 Aug 2009 21:44:14 -0700 (PDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6186) Enhance checkstyle results by Hudson
 Hadoop QA to provide a diff
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Enhance checkstyle results by Hudson Hadoop QA to provide a diff
----------------------------------------------------------------

                 Key: HADOOP-6186
                 URL: https://issues.apache.org/jira/browse/HADOOP-6186
             Project: Hadoop Common
          Issue Type: Bug
          Components: build
            Reporter: Vinod K V


Currently, when a patch is submitted to Hudson, it provides the checkstyle results indicating problems that happened throughout the project history. For e.g., http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/463/artifact/trunk/build/test/checkstyle-errors.html.

It is not useful at all in this format. It would be of better use to provide the style problems resulting ONLY OUT OF THE CURRENT PATCH.

Another point: Should we also consider a +1, -1 by Hudson w.r.t code style ala findBugs warnings, test results etc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-701-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 05:25:32 2009
Return-Path: <common-issues-return-701-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43563 invoked from network); 11 Aug 2009 05:25:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 05:25:32 -0000
Received: (qmail 78598 invoked by uid 500); 11 Aug 2009 05:25:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78555 invoked by uid 500); 11 Aug 2009 05:25:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78540 invoked by uid 99); 11 Aug 2009 05:25:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 05:25:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 05:25:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DBB75234C044
	for <common-issues@hadoop.apache.org>; Mon, 10 Aug 2009 22:25:14 -0700 (PDT)
Message-ID: <275506731.1249968314898.JavaMail.jira@brutus>
Date: Mon, 10 Aug 2009 22:25:14 -0700 (PDT)
From: "Pallavi Palleti (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6187) Allow Super user access only from
 certian trusted IP Range- This is to avoid spoofing by others as super user
 and gain access to the cluster
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Allow Super user access only from certian trusted IP Range- This is to avoid spoofing by others as super user and gain access to the cluster
--------------------------------------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-6187
                 URL: https://issues.apache.org/jira/browse/HADOOP-6187
             Project: Hadoop Common
          Issue Type: New Feature
          Components: ipc
    Affects Versions: 0.20.0
            Reporter: Pallavi Palleti


The current hadoop environment assumes  that everything is trusted environment and so doesn't have any security checks. This might lead to any one to spoof as super user and gain access to the cluster if the environment is open for every one. So, there is a need to restrict super user permission only from certain IP addresses. 




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-702-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 05:25:32 2009
Return-Path: <common-issues-return-702-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43585 invoked from network); 11 Aug 2009 05:25:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 05:25:32 -0000
Received: (qmail 78654 invoked by uid 500); 11 Aug 2009 05:25:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78621 invoked by uid 500); 11 Aug 2009 05:25:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78541 invoked by uid 99); 11 Aug 2009 05:25:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 05:25:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 05:25:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E6237234C1E6
	for <common-issues@hadoop.apache.org>; Mon, 10 Aug 2009 22:25:14 -0700 (PDT)
Message-ID: <535970231.1249968314941.JavaMail.jira@brutus>
Date: Mon, 10 Aug 2009 22:25:14 -0700 (PDT)
From: "Pallavi Palleti (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6187) Allow Super user access only from
 certian trusted IP Range- This is to avoid spoofing by others as super user
 and gain access to the cluster
In-Reply-To: <275506731.1249968314898.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pallavi Palleti updated HADOOP-6187:
------------------------------------

    Priority: Minor  (was: Major)

> Allow Super user access only from certian trusted IP Range- This is to avoid spoofing by others as super user and gain access to the cluster
> --------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6187
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6187
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.20.0
>            Reporter: Pallavi Palleti
>            Priority: Minor
>
> The current hadoop environment assumes  that everything is trusted environment and so doesn't have any security checks. This might lead to any one to spoof as super user and gain access to the cluster if the environment is open for every one. So, there is a need to restrict super user permission only from certain IP addresses. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-703-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 06:11:32 2009
Return-Path: <common-issues-return-703-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61659 invoked from network); 11 Aug 2009 06:11:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 06:11:32 -0000
Received: (qmail 17772 invoked by uid 500); 11 Aug 2009 06:11:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17727 invoked by uid 500); 11 Aug 2009 06:11:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17717 invoked by uid 99); 11 Aug 2009 06:11:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 06:11:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 06:11:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C5569234C004
	for <common-issues@hadoop.apache.org>; Mon, 10 Aug 2009 23:11:14 -0700 (PDT)
Message-ID: <789826872.1249971074793.JavaMail.jira@brutus>
Date: Mon, 10 Aug 2009 23:11:14 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6184) Provide a configuration dump in json
 format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna updated HADOOP-6184:
------------------------------------------

    Attachment: HADOOP-6184-1.patch

Uploading patch with the above comments implemented.

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-704-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 06:37:30 2009
Return-Path: <common-issues-return-704-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80222 invoked from network); 11 Aug 2009 06:37:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 06:37:30 -0000
Received: (qmail 46730 invoked by uid 500); 11 Aug 2009 06:37:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46680 invoked by uid 500); 11 Aug 2009 06:37:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46670 invoked by uid 99); 11 Aug 2009 06:37:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 06:37:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 06:37:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DC561234C046
	for <common-issues@hadoop.apache.org>; Mon, 10 Aug 2009 23:37:14 -0700 (PDT)
Message-ID: <124371820.1249972634901.JavaMail.jira@brutus>
Date: Mon, 10 Aug 2009 23:37:14 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6184) Provide a configuration dump in json
 format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna updated HADOOP-6184:
------------------------------------------

    Attachment: HADOOP-6184-2.patch

The previous patch does not handle the case when the key is first loaded by a resource file and then programatically set by the user. A new patch is uploaded which handles this issue.

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184-2.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-705-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 06:42:32 2009
Return-Path: <common-issues-return-705-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81649 invoked from network); 11 Aug 2009 06:42:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 06:42:32 -0000
Received: (qmail 55042 invoked by uid 500); 11 Aug 2009 06:42:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54991 invoked by uid 500); 11 Aug 2009 06:42:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54980 invoked by uid 99); 11 Aug 2009 06:42:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 06:42:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 06:42:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C4B5B234C004
	for <common-issues@hadoop.apache.org>; Mon, 10 Aug 2009 23:42:14 -0700 (PDT)
Message-ID: <741747192.1249972934798.JavaMail.jira@brutus>
Date: Mon, 10 Aug 2009 23:42:14 -0700 (PDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6103) Configuration clone constructor does
 not clone all the members.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amareshwari Sriramadasu updated HADOOP-6103:
--------------------------------------------

    Attachment: patch-6103.txt

Patch with the fix

> Configuration clone constructor does not clone all the members.
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6103
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6103
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>            Reporter: Amareshwari Sriramadasu
>             Fix For: 0.21.0
>
>         Attachments: patch-6103.txt
>
>
> Currently, Configuration(Configuration other) constructor does not clone all the members.
> It clones only resources, properties, overlay and finalParameters. It needs to clone loadDefaults, classLoader, defaultResources, quietmode.
> This resulted in bugs like HADOOP-4975

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-706-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 06:44:32 2009
Return-Path: <common-issues-return-706-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81932 invoked from network); 11 Aug 2009 06:44:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 06:44:32 -0000
Received: (qmail 57656 invoked by uid 500); 11 Aug 2009 06:44:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57625 invoked by uid 500); 11 Aug 2009 06:44:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57615 invoked by uid 99); 11 Aug 2009 06:44:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 06:44:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 06:44:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C1FFB234C004
	for <common-issues@hadoop.apache.org>; Mon, 10 Aug 2009 23:44:14 -0700 (PDT)
Message-ID: <1858352895.1249973054780.JavaMail.jira@brutus>
Date: Mon, 10 Aug 2009 23:44:14 -0700 (PDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6103) Configuration clone constructor does
 not clone all the members.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amareshwari Sriramadasu updated HADOOP-6103:
--------------------------------------------

    Assignee: Amareshwari Sriramadasu
      Status: Patch Available  (was: Open)

> Configuration clone constructor does not clone all the members.
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6103
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6103
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.21.0
>
>         Attachments: patch-6103.txt
>
>
> Currently, Configuration(Configuration other) constructor does not clone all the members.
> It clones only resources, properties, overlay and finalParameters. It needs to clone loadDefaults, classLoader, defaultResources, quietmode.
> This resulted in bugs like HADOOP-4975

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-707-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 07:01:31 2009
Return-Path: <common-issues-return-707-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84931 invoked from network); 11 Aug 2009 07:01:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 07:01:30 -0000
Received: (qmail 70517 invoked by uid 500); 11 Aug 2009 07:01:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70466 invoked by uid 500); 11 Aug 2009 07:01:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70454 invoked by uid 99); 11 Aug 2009 07:01:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 07:01:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 07:01:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CB351234C1E6
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 00:01:15 -0700 (PDT)
Message-ID: <932372129.1249974075831.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 00:01:15 -0700 (PDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4952) Improved files system interface for
 the application writer.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-4952:
---------------------------------

    Attachment: Files.java


The attached Files.java is a class for the proposed new interface for the application writer.
The FileSystem class will remain for the fs implementor (after some cleanup); it will also be used to implement the
Files class.
Most of the methods in the Files class do not have an implementation yet - the skeleton is to discuss the proposal. 

> Improved files system interface for the application writer.
> -----------------------------------------------------------
>
>                 Key: HADOOP-4952
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4952
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.21.0
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: Files.java, Files.java
>
>
> Currently the FIleSystem interface serves two purposes:
> - an application writer's interface for using the Hadoop file system
> - a file system implementer's interface (e.g. hdfs, local file system, kfs, etc)
> This Jira proposes that we provide a simpler interfaces for the application writer and leave the FilsSystem  interface for the implementer of a filesystem.
> - Filesystem interface  has a  confusing set of methods for the application writer
> - We could make it easier to take advantage of the URI file naming
> ** Current approach is to get FileSystem instance by supplying the URI and then access that name space. It is consistent for the FileSystem instance to not accept URIs for other schemes, but we can do better.
> ** The special copyFromLocalFIle can be generalized as a  copyFile where the src or target can be generalized to any URI, including the local one.
> ** The proposed scheme (below) simplifies this.
> -	The client side config can be simplified. 
> ** New config() by default uses the default config. Since this is the common usage pattern, one should not need to always pass the config as a parameter when accessing the file system.  
> -	
> ** It does not handle multiple file systems too well. Today a site.xml is derived from a single Hadoop cluster. This does not make sense for multiple Hadoop clusters which may have different defaults.
> ** Further one should need very little to configure the client side:
> *** Default files system.
> *** Block size 
> *** Replication factor
> *** Scheme to class mapping
> ** It should be possible to take Blocksize and replication factors defaults from the target file system, rather then the client size config.  I am not suggesting we don't allow setting client side defaults, but most clients do not care and would find it simpler to take the defaults for their systems  from the target file system. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-708-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 07:23:30 2009
Return-Path: <common-issues-return-708-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95336 invoked from network); 11 Aug 2009 07:23:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 07:23:30 -0000
Received: (qmail 95935 invoked by uid 500); 11 Aug 2009 07:23:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95880 invoked by uid 500); 11 Aug 2009 07:23:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95867 invoked by uid 99); 11 Aug 2009 07:23:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 07:23:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 07:23:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C78CB234C004
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 00:23:14 -0700 (PDT)
Message-ID: <290825956.1249975394803.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 00:23:14 -0700 (PDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6103) Configuration clone constructor
 does not clone all the members.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741732#action_12741732 ] 

Amareshwari Sriramadasu commented on HADOOP-6103:
-------------------------------------------------

test-patch result:
{noformat}
     [exec]
     [exec] +1 overall.
     [exec]
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec]
     [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
     [exec]
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec]
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec]
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec]
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec]
 {noformat}

ant test passed on my machine

> Configuration clone constructor does not clone all the members.
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6103
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6103
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.21.0
>
>         Attachments: patch-6103.txt
>
>
> Currently, Configuration(Configuration other) constructor does not clone all the members.
> It clones only resources, properties, overlay and finalParameters. It needs to clone loadDefaults, classLoader, defaultResources, quietmode.
> This resulted in bugs like HADOOP-4975

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-709-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 08:39:30 2009
Return-Path: <common-issues-return-709-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20062 invoked from network); 11 Aug 2009 08:39:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 08:39:30 -0000
Received: (qmail 98741 invoked by uid 500); 11 Aug 2009 08:39:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98688 invoked by uid 500); 11 Aug 2009 08:39:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98678 invoked by uid 99); 11 Aug 2009 08:39:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 08:39:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 08:39:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E1C4E234C1E7
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 01:39:14 -0700 (PDT)
Message-ID: <980256449.1249979954923.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 01:39:14 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6184) Provide a configuration dump in json
 format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna updated HADOOP-6184:
------------------------------------------

    Attachment: HADOOP-6184-3.patch

Had an offline discussion with Sreekanth and Rahul. It was proposed to have the following things in the Json format:

- Keys which are present in some resource and then are overriden using se method by the user should have resource displayed as "Unknown".
- Since currently, user would be looking for resource only when json format dump is needed, storage of key-resource mapping is done only when user wants to get dump in json format. This prevents redundant storage of key-resource mapping even when it is not required.

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184-2.patch, HADOOP-6184-3.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-710-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 08:43:30 2009
Return-Path: <common-issues-return-710-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21264 invoked from network); 11 Aug 2009 08:43:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 08:43:29 -0000
Received: (qmail 6324 invoked by uid 500); 11 Aug 2009 08:43:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6273 invoked by uid 500); 11 Aug 2009 08:43:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6260 invoked by uid 99); 11 Aug 2009 08:43:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 08:43:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 08:43:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C704C234C004
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 01:43:14 -0700 (PDT)
Message-ID: <532031645.1249980194801.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 01:43:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6103) Configuration clone constructor
 does not clone all the members.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741761#action_12741761 ] 

Hadoop QA commented on HADOOP-6103:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12416171/patch-6103.txt
  against trunk revision 802224.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/597/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/597/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/597/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/597/console

This message is automatically generated.

> Configuration clone constructor does not clone all the members.
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6103
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6103
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.21.0
>
>         Attachments: patch-6103.txt
>
>
> Currently, Configuration(Configuration other) constructor does not clone all the members.
> It clones only resources, properties, overlay and finalParameters. It needs to clone loadDefaults, classLoader, defaultResources, quietmode.
> This resulted in bugs like HADOOP-4975

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-711-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 08:59:31 2009
Return-Path: <common-issues-return-711-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26724 invoked from network); 11 Aug 2009 08:59:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 08:59:31 -0000
Received: (qmail 27841 invoked by uid 500); 11 Aug 2009 08:59:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27792 invoked by uid 500); 11 Aug 2009 08:59:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27653 invoked by uid 99); 11 Aug 2009 08:59:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 08:59:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 08:59:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D83A9234C044
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 01:59:14 -0700 (PDT)
Message-ID: <317975596.1249981154884.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 01:59:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6182) Adding Apache License Headers and
 reduce releaseaudit warnings to zero
In-Reply-To: <739323326.1249883774947.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan reassigned HADOOP-6182:
------------------------------------------

    Assignee: Giridharan Kesavan

> Adding Apache License Headers and reduce releaseaudit warnings to zero
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6182
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6182
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>
> As of now rats tool shows 111 RA warnings 
> [rat:report] Summary
> [rat:report] -------
> [rat:report] Notes: 18
> [rat:report] Binaries: 118
> [rat:report] Archives: 33
> [rat:report] Standards: 942
> [rat:report] 
> [rat:report] Apache Licensed: 820
> [rat:report] Generated Documents: 11
> [rat:report] 
> [rat:report] JavaDocs are generated and so license header is optional
> [rat:report] Generated files do not required license headers
> [rat:report] 
> [rat:report] 111 Unknown Licenses
> [rat:report] 
> [rat:report] *******************************

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-712-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 09:25:31 2009
Return-Path: <common-issues-return-712-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38947 invoked from network); 11 Aug 2009 09:25:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 09:25:30 -0000
Received: (qmail 54053 invoked by uid 500); 11 Aug 2009 09:25:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53999 invoked by uid 500); 11 Aug 2009 09:25:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53969 invoked by uid 99); 11 Aug 2009 09:25:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 09:25:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 09:25:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CDAF3234C004
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 02:25:14 -0700 (PDT)
Message-ID: <2052885195.1249982714824.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 02:25:14 -0700 (PDT)
From: "Sreekanth Ramakrishnan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6184) Provide a configuration dump in
 json format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741773#action_12741773 ] 

Sreekanth Ramakrishnan commented on HADOOP-6184:
------------------------------------------------

The code changes look fine to me.
+1

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184-2.patch, HADOOP-6184-3.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-713-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 09:49:32 2009
Return-Path: <common-issues-return-713-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53345 invoked from network); 11 Aug 2009 09:49:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 09:49:32 -0000
Received: (qmail 74785 invoked by uid 500); 11 Aug 2009 09:49:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74734 invoked by uid 500); 11 Aug 2009 09:49:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74718 invoked by uid 99); 11 Aug 2009 09:49:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 09:49:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 09:49:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1D82D234C004
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 02:49:15 -0700 (PDT)
Message-ID: <1463292409.1249984155106.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 02:49:15 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6182) Adding Apache License Headers and
 reduce releaseaudit warnings to zero
In-Reply-To: <739323326.1249883774947.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6182:
---------------------------------------

    Attachment: Hadoop-6182.patch

with this patch the releaseaudit warnings is down to 1.

[rat:report] 1 Unknown Licenses
[rat:report] *******************************
[rat:report] Unapproved licenses:
[rat:report]  src/java/org/apache/hadoop/fs/LengthFileChecksum.java

LengthFileChecksum.java this is an empty file. I'm not sure why we have this empty java files.
If we can remove this empty java file, we can get rid of this releaaseaudit warning.

Thanks,





> Adding Apache License Headers and reduce releaseaudit warnings to zero
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6182
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6182
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>         Attachments: Hadoop-6182.patch
>
>
> As of now rats tool shows 111 RA warnings 
> [rat:report] Summary
> [rat:report] -------
> [rat:report] Notes: 18
> [rat:report] Binaries: 118
> [rat:report] Archives: 33
> [rat:report] Standards: 942
> [rat:report] 
> [rat:report] Apache Licensed: 820
> [rat:report] Generated Documents: 11
> [rat:report] 
> [rat:report] JavaDocs are generated and so license header is optional
> [rat:report] Generated files do not required license headers
> [rat:report] 
> [rat:report] 111 Unknown Licenses
> [rat:report] 
> [rat:report] *******************************

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-714-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 10:29:37 2009
Return-Path: <common-issues-return-714-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74215 invoked from network); 11 Aug 2009 10:29:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 10:29:36 -0000
Received: (qmail 21242 invoked by uid 500); 11 Aug 2009 10:29:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21189 invoked by uid 500); 11 Aug 2009 10:29:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21178 invoked by uid 99); 11 Aug 2009 10:29:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 10:29:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 10:29:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 05422234C004
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 03:29:15 -0700 (PDT)
Message-ID: <1402843170.1249986555007.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 03:29:15 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6184) Provide a configuration dump in
 json format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741795#action_12741795 ] 

V.V.Chaitanya Krishna commented on HADOOP-6184:
-----------------------------------------------

Example for above comment: 
in mapred-config.sh:

export HADOOP_JOBTRACKER_OPTS="-Dmapred.jobtracker.dumpconfiguration=true

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184-2.patch, HADOOP-6184-3.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-715-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 10:47:32 2009
Return-Path: <common-issues-return-715-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80149 invoked from network); 11 Aug 2009 10:47:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 10:47:32 -0000
Received: (qmail 37154 invoked by uid 500); 11 Aug 2009 10:47:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37103 invoked by uid 500); 11 Aug 2009 10:47:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37093 invoked by uid 99); 11 Aug 2009 10:47:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 10:47:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 10:47:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C7AE4234C004
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 03:47:14 -0700 (PDT)
Message-ID: <2124593457.1249987634810.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 03:47:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6182) Adding Apache License Headers and
 reduce releaseaudit warnings to zero
In-Reply-To: <739323326.1249883774947.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6182:
---------------------------------------

    Status: Patch Available  (was: Open)

> Adding Apache License Headers and reduce releaseaudit warnings to zero
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6182
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6182
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>         Attachments: Hadoop-6182.patch
>
>
> As of now rats tool shows 111 RA warnings 
> [rat:report] Summary
> [rat:report] -------
> [rat:report] Notes: 18
> [rat:report] Binaries: 118
> [rat:report] Archives: 33
> [rat:report] Standards: 942
> [rat:report] 
> [rat:report] Apache Licensed: 820
> [rat:report] Generated Documents: 11
> [rat:report] 
> [rat:report] JavaDocs are generated and so license header is optional
> [rat:report] Generated files do not required license headers
> [rat:report] 
> [rat:report] 111 Unknown Licenses
> [rat:report] 
> [rat:report] *******************************

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-716-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 11:42:32 2009
Return-Path: <common-issues-return-716-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94232 invoked from network); 11 Aug 2009 11:42:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 11:42:32 -0000
Received: (qmail 1813 invoked by uid 500); 11 Aug 2009 11:42:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1773 invoked by uid 500); 11 Aug 2009 11:42:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1763 invoked by uid 99); 11 Aug 2009 11:42:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 11:42:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 11:42:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 5E2B2234C004
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 04:42:15 -0700 (PDT)
Message-ID: <174008838.1249990935370.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 04:42:15 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6182) Adding Apache License Headers and
 reduce releaseaudit warnings to zero
In-Reply-To: <739323326.1249883774947.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741814#action_12741814 ] 

Tom White commented on HADOOP-6182:
-----------------------------------

Looks like LengthFileChecksum was removed in HADOOP-3981, although the source file wasn't deleted. It should be possible to delete it since it is no longer used. It should be deleted from the 0.20 branch too.

> Adding Apache License Headers and reduce releaseaudit warnings to zero
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6182
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6182
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>         Attachments: Hadoop-6182.patch
>
>
> As of now rats tool shows 111 RA warnings 
> [rat:report] Summary
> [rat:report] -------
> [rat:report] Notes: 18
> [rat:report] Binaries: 118
> [rat:report] Archives: 33
> [rat:report] Standards: 942
> [rat:report] 
> [rat:report] Apache Licensed: 820
> [rat:report] Generated Documents: 11
> [rat:report] 
> [rat:report] JavaDocs are generated and so license header is optional
> [rat:report] Generated files do not required license headers
> [rat:report] 
> [rat:report] 111 Unknown Licenses
> [rat:report] 
> [rat:report] *******************************

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-717-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 13:54:34 2009
Return-Path: <common-issues-return-717-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69414 invoked from network); 11 Aug 2009 13:54:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 13:54:34 -0000
Received: (qmail 25418 invoked by uid 500); 11 Aug 2009 13:54:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25389 invoked by uid 500); 11 Aug 2009 13:54:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25332 invoked by uid 99); 11 Aug 2009 13:54:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 13:54:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 13:54:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D9ABD234C052
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 06:54:14 -0700 (PDT)
Message-ID: <409813019.1249998854890.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 06:54:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6182) Adding Apache License Headers and
 reduce releaseaudit warnings to zero
In-Reply-To: <739323326.1249883774947.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741876#action_12741876 ] 

Hadoop QA commented on HADOOP-6182:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12416182/Hadoop-6182.patch
  against trunk revision 802224.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 60 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/598/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/598/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/598/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/598/console

This message is automatically generated.

> Adding Apache License Headers and reduce releaseaudit warnings to zero
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6182
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6182
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>         Attachments: Hadoop-6182.patch
>
>
> As of now rats tool shows 111 RA warnings 
> [rat:report] Summary
> [rat:report] -------
> [rat:report] Notes: 18
> [rat:report] Binaries: 118
> [rat:report] Archives: 33
> [rat:report] Standards: 942
> [rat:report] 
> [rat:report] Apache Licensed: 820
> [rat:report] Generated Documents: 11
> [rat:report] 
> [rat:report] JavaDocs are generated and so license header is optional
> [rat:report] Generated files do not required license headers
> [rat:report] 
> [rat:report] 111 Unknown Licenses
> [rat:report] 
> [rat:report] *******************************

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-718-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 16:43:54 2009
Return-Path: <common-issues-return-718-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55087 invoked from network); 11 Aug 2009 16:43:53 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 16:43:53 -0000
Received: (qmail 8245 invoked by uid 500); 11 Aug 2009 16:23:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8227 invoked by uid 500); 11 Aug 2009 16:23:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8205 invoked by uid 99); 11 Aug 2009 16:23:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 16:23:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 16:23:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1AB7F234C045
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 09:23:15 -0700 (PDT)
Message-ID: <1545461582.1250007795108.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 09:23:15 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5107) split the core, hdfs, and mapred
 jars from each other and publish them independently to the Maven repository
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741936#action_12741936 ] 

Tom White commented on HADOOP-5107:
-----------------------------------

A few comments after playing with this:

* With these changes we should be able to use Ivy's support of transitive dependencies, so (amongst other things) we could remove the jets3t dependency from HDFS. Cleaning up the direct dependencies probably belongs in a follow up issue.
* I was having problems getting the build the use the version of Ivy it downloads. This was due to an older version in ~/.ant which was being picked up. It worked after I deleted it. (I just mention this in case others have this problem - no changes are required to the patches.)
* I had to make a few changes to ivysettings.xml:
** Define a maven2.pattern (it was having problems retrieving the RAT jar without it):
{code}
<property name="maven2.pattern" value="[organisation]/[module]/[revision]/[module]-[revision]"/>
{code}
** Define a default username (empty), since without it you get the error "The uri is in the wrong format" (for e.g. /home/${username}/ivyrepo/commons-cli/commons-cli/1.1/ivys/ivy.xml):
{code}
<property name="username" value=""/>
{code}

> split the core, hdfs, and mapred jars from each other and publish them independently to the Maven repository
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5107
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5107
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.0
>            Reporter: Owen O'Malley
>            Assignee: Giridharan Kesavan
>         Attachments: common-trunk.patch, hdfs-trunk.patch, mapreduce-trunk.patch
>
>
> I think to support splitting the projects, we should publish the jars for 0.20.0 as independent jars to the Maven repository 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-719-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 16:44:11 2009
Return-Path: <common-issues-return-719-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56126 invoked from network); 11 Aug 2009 16:44:11 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 16:44:11 -0000
Received: (qmail 10891 invoked by uid 500); 11 Aug 2009 16:29:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10870 invoked by uid 500); 11 Aug 2009 16:29:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10842 invoked by uid 99); 11 Aug 2009 16:29:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 16:29:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 16:29:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 121E6234C004
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 09:29:15 -0700 (PDT)
Message-ID: <59671136.1250008155060.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 09:29:15 -0700 (PDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6177) FSInputChecker.getPos() would return
 position greater than the file size
In-Reply-To: <1554871263.1249618274995.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6177:
----------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

I just committed this. Thanks Hong!

> FSInputChecker.getPos() would return position greater than the file size
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6177
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6177
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6177-2090806.patch, hadoop-6177-testcase.patch
>
>
> When using a small buffer (< 512 bytes) to read through the whole file, the final file position is 1+ the file size.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-720-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 16:44:39 2009
Return-Path: <common-issues-return-720-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58833 invoked from network); 11 Aug 2009 16:44:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 16:44:36 -0000
Received: (qmail 16488 invoked by uid 500); 11 Aug 2009 16:39:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16472 invoked by uid 500); 11 Aug 2009 16:39:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16462 invoked by uid 99); 11 Aug 2009 16:39:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 16:39:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 16:39:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E5DAD234C04C
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 09:39:14 -0700 (PDT)
Message-ID: <461429505.1250008754940.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 09:39:14 -0700 (PDT)
From: "Pallavi Palleti (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6187) Allow Super user access only from
 certian trusted IP Range- This is to avoid spoofing by others as super user
 and gain access to the cluster
In-Reply-To: <275506731.1249968314898.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pallavi Palleti updated HADOOP-6187:
------------------------------------

    Attachment: Hadoop-6187.patch

The patch restricts super user access only from some specified IP range if it is configured. if not, it works by default allowing super user access from any where. 

> Allow Super user access only from certian trusted IP Range- This is to avoid spoofing by others as super user and gain access to the cluster
> --------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6187
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6187
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.20.0
>            Reporter: Pallavi Palleti
>            Priority: Minor
>         Attachments: Hadoop-6187.patch
>
>
> The current hadoop environment assumes  that everything is trusted environment and so doesn't have any security checks. This might lead to any one to spoof as super user and gain access to the cluster if the environment is open for every one. So, there is a need to restrict super user permission only from certain IP addresses. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-721-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 17:03:30 2009
Return-Path: <common-issues-return-721-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69244 invoked from network); 11 Aug 2009 17:03:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 17:03:30 -0000
Received: (qmail 1354 invoked by uid 500); 11 Aug 2009 17:03:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1295 invoked by uid 500); 11 Aug 2009 17:03:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1283 invoked by uid 99); 11 Aug 2009 17:03:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 17:03:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 17:03:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C9BA4234C004
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 10:03:14 -0700 (PDT)
Message-ID: <2124770677.1250010194811.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 10:03:14 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5107) split the core, hdfs, and mapred
 jars from each other and publish them independently to the Maven repository
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741964#action_12741964 ] 

Doug Cutting commented on HADOOP-5107:
--------------------------------------

> I was having problems getting the build the use the version of Ivy it downloads.

Avro handles this better:

http://svn.apache.org/viewvc/hadoop/avro/trunk/build.xml?view=annotate#l128

Ivy's jar is stored in the lib/ directory.  If the specified version of Ivy isn't there, it removes all versions before downloading.  Also, one need never specify offline=true, since, so long as the specified version is there, it doesn't contact the network.

> split the core, hdfs, and mapred jars from each other and publish them independently to the Maven repository
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5107
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5107
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.0
>            Reporter: Owen O'Malley
>            Assignee: Giridharan Kesavan
>         Attachments: common-trunk.patch, hdfs-trunk.patch, mapreduce-trunk.patch
>
>
> I think to support splitting the projects, we should publish the jars for 0.20.0 as independent jars to the Maven repository 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-722-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 17:20:30 2009
Return-Path: <common-issues-return-722-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76605 invoked from network); 11 Aug 2009 17:20:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 17:20:30 -0000
Received: (qmail 62680 invoked by uid 500); 11 Aug 2009 17:20:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62623 invoked by uid 500); 11 Aug 2009 17:20:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62613 invoked by uid 99); 11 Aug 2009 17:20:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 17:20:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 17:20:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 801B8234C1F0
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 10:20:15 -0700 (PDT)
Message-ID: <867445791.1250011215523.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 10:20:15 -0700 (PDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6152) Hadoop scripts do not correctly put
 jars on the classpath
In-Reply-To: <283943906.1247689875148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron Kimball updated HADOOP-6152:
----------------------------------

    Status: Open  (was: Patch Available)

> Hadoop scripts do not correctly put jars on the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-6152
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6152
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>            Priority: Blocker
>         Attachments: HADOOP-6152.2.patch, HADOOP-6152.patch
>
>
> The various Hadoop scripts (bin/hadoop, bin/hdfs, bin/mapred) do not properly identify the jars needed to run Hadoop. They try to include hadoop-*-hdfs.jar, etc, rather than the hadoop-hdfs-*.jar that is actually created by the 'ant jar' and 'ant package' targets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-723-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 17:20:32 2009
Return-Path: <common-issues-return-723-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76699 invoked from network); 11 Aug 2009 17:20:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 17:20:32 -0000
Received: (qmail 62847 invoked by uid 500); 11 Aug 2009 17:20:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62783 invoked by uid 500); 11 Aug 2009 17:20:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62773 invoked by uid 99); 11 Aug 2009 17:20:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 17:20:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 17:20:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 6A104234C04C
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 10:20:15 -0700 (PDT)
Message-ID: <264056382.1250011215433.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 10:20:15 -0700 (PDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6152) Hadoop scripts do not correctly put
 jars on the classpath
In-Reply-To: <283943906.1247689875148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron Kimball updated HADOOP-6152:
----------------------------------

    Attachment: HADOOP-6152.2.patch

Attaching new resync'd patch.

> Hadoop scripts do not correctly put jars on the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-6152
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6152
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>            Priority: Blocker
>         Attachments: HADOOP-6152.2.patch, HADOOP-6152.patch
>
>
> The various Hadoop scripts (bin/hadoop, bin/hdfs, bin/mapred) do not properly identify the jars needed to run Hadoop. They try to include hadoop-*-hdfs.jar, etc, rather than the hadoop-hdfs-*.jar that is actually created by the 'ant jar' and 'ant package' targets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-724-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 17:20:33 2009
Return-Path: <common-issues-return-724-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76741 invoked from network); 11 Aug 2009 17:20:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 17:20:33 -0000
Received: (qmail 62963 invoked by uid 500); 11 Aug 2009 17:20:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62884 invoked by uid 500); 11 Aug 2009 17:20:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62845 invoked by uid 99); 11 Aug 2009 17:20:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 17:20:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 17:20:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 91258234C4AA
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 10:20:15 -0700 (PDT)
Message-ID: <1280970093.1250011215593.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 10:20:15 -0700 (PDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6152) Hadoop scripts do not correctly put
 jars on the classpath
In-Reply-To: <283943906.1247689875148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron Kimball updated HADOOP-6152:
----------------------------------

    Status: Patch Available  (was: Open)

> Hadoop scripts do not correctly put jars on the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-6152
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6152
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>            Priority: Blocker
>         Attachments: HADOOP-6152.2.patch, HADOOP-6152.patch
>
>
> The various Hadoop scripts (bin/hadoop, bin/hdfs, bin/mapred) do not properly identify the jars needed to run Hadoop. They try to include hadoop-*-hdfs.jar, etc, rather than the hadoop-hdfs-*.jar that is actually created by the 'ant jar' and 'ant package' targets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-725-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 17:28:32 2009
Return-Path: <common-issues-return-725-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82922 invoked from network); 11 Aug 2009 17:28:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 17:28:32 -0000
Received: (qmail 85235 invoked by uid 500); 11 Aug 2009 17:28:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85120 invoked by uid 500); 11 Aug 2009 17:28:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84829 invoked by uid 99); 11 Aug 2009 17:28:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 17:28:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 17:28:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D52BE234C045
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 10:28:14 -0700 (PDT)
Message-ID: <364050017.1250011694871.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 10:28:14 -0700 (PDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6188) TestHDFSTrash fails because of
 TestTrash in common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

TestHDFSTrash fails because of TestTrash in common
--------------------------------------------------

                 Key: HADOOP-6188
                 URL: https://issues.apache.org/jira/browse/HADOOP-6188
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
            Reporter: Boris Shkolnik




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-726-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 17:32:33 2009
Return-Path: <common-issues-return-726-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85760 invoked from network); 11 Aug 2009 17:32:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 17:32:33 -0000
Received: (qmail 94351 invoked by uid 500); 11 Aug 2009 17:32:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94288 invoked by uid 500); 11 Aug 2009 17:32:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94266 invoked by uid 99); 11 Aug 2009 17:32:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 17:32:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 17:32:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 9DF69234C055
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 10:32:15 -0700 (PDT)
Message-ID: <384339611.1250011935645.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 10:32:15 -0700 (PDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6188) TestHDFSTrash fails because of
 TestTrash in common
In-Reply-To: <364050017.1250011694871.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741979#action_12741979 ] 

Boris Shkolnik commented on HADOOP-6188:
----------------------------------------

Test is using Java File.list() method which works fine for LocalFileSystem. 
But the same test is being run in HDFS project with DFS and it failes. 
We need to change it to FileSystem.listStatus() , which works for both. 

> TestHDFSTrash fails because of TestTrash in common
> --------------------------------------------------
>
>                 Key: HADOOP-6188
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6188
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Boris Shkolnik
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-727-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 18:02:28 2009
Return-Path: <common-issues-return-727-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99218 invoked from network); 11 Aug 2009 18:02:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 18:02:28 -0000
Received: (qmail 45836 invoked by uid 500); 11 Aug 2009 18:01:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45542 invoked by uid 500); 11 Aug 2009 18:01:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45430 invoked by uid 99); 11 Aug 2009 18:00:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 18:00:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 18:00:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DED23234C004
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 11:00:14 -0700 (PDT)
Message-ID: <1655887469.1250013614900.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 11:00:14 -0700 (PDT)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741993#action_12741993 ] 

Scott Carey commented on HADOOP-6166:
-------------------------------------

We probably want to have Todd's concurrency test from HADOOP-5318 run to make sure the larger lookup table doesn't slow things down under concurrency.

We might also want to try the old four at a time code for 4 <= len < 8.

We should also confirm the results on one of the other systems we tested in the past. I won't be able to do that for a couple days, but it should be easy then.
How many other variants did you try? Intel's C code does some strange things to group CRC's by 3. Maybe something like the below is worth testing if it hasn't already been done haven't already:


{code}+public class Crc32_8_8e extends Crc32Base {
  public void update(byte[] b, int off, int len) {
    while(len > 7) {
      int c0 = b[off++] ^ crc;
      int c1 = b[off++] ^ (crc >>>= 8);
      int c2 = b[off++] ^ (crc >>>= 8);
      int c3 = b[off++] ^ (crc >>>= 8);
     
      crc = T8_7[c0 & 0xff] ^ T8_6[c1 & 0xff] ^ T8_5[c2 & 0xff] ^ T8_4[c3 & 0xff]; // three xors

      crc ^= T8_3[b[off++] & 0xff] ^ T8_2[b[off++] & 0xff]; // two xors

      crc ^= T8_1[b[off++] & 0xff] ^ T8_0[b[off++] & 0xff]; // two xors

      len -= 8;
    }
    while(len > 0) {
      crc = (crc >>> 8) ^ T8_0[(crc ^ b[off++]) & 0xff];
      len--;
    }
  }
}{code}


> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt, c6166_20090727.patch, c6166_20090728.patch, c6166_20090810.patch
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-728-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 18:08:32 2009
Return-Path: <common-issues-return-728-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7537 invoked from network); 11 Aug 2009 18:08:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 18:08:32 -0000
Received: (qmail 1811 invoked by uid 500); 11 Aug 2009 18:08:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1754 invoked by uid 500); 11 Aug 2009 18:08:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1744 invoked by uid 99); 11 Aug 2009 18:08:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 18:08:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 18:08:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E0A32234C045
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 11:08:14 -0700 (PDT)
Message-ID: <1238191954.1250014094919.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 11:08:14 -0700 (PDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6188) TestHDFSTrash fails because of
 TestTrash in common
In-Reply-To: <364050017.1250011694871.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6188:
-----------------------------------

    Attachment: HADOOP-6188.patch

> TestHDFSTrash fails because of TestTrash in common
> --------------------------------------------------
>
>                 Key: HADOOP-6188
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6188
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Boris Shkolnik
>         Attachments: HADOOP-6188.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-729-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 18:14:34 2009
Return-Path: <common-issues-return-729-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10371 invoked from network); 11 Aug 2009 18:14:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 18:14:33 -0000
Received: (qmail 17858 invoked by uid 500); 11 Aug 2009 18:14:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17822 invoked by uid 500); 11 Aug 2009 18:14:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17782 invoked by uid 99); 11 Aug 2009 18:14:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 18:14:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 18:14:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DC77E234C004
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 11:14:14 -0700 (PDT)
Message-ID: <972300077.1250014454889.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 11:14:14 -0700 (PDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6188) TestHDFSTrash fails because of
 TestTrash in common
In-Reply-To: <364050017.1250011694871.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741995#action_12741995 ] 

Boris Shkolnik commented on HADOOP-6188:
----------------------------------------

Ran testHDFSTrash on HDFS project..

run-test-hdfs:
   [delete] Deleting directory /Users/borya/work/hadoop/hadoop-hdfs/build/test/data
    [mkdir] Created dir: /Users/borya/work/hadoop/hadoop-hdfs/build/test/data
   [delete] Deleting directory /Users/borya/work/hadoop/hadoop-hdfs/build/test/logs
    [mkdir] Created dir: /Users/borya/work/hadoop/hadoop-hdfs/build/test/logs
    [junit] Running org.apache.hadoop.hdfs.TestHDFSTrash
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 5.526 sec


> TestHDFSTrash fails because of TestTrash in common
> --------------------------------------------------
>
>                 Key: HADOOP-6188
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6188
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Boris Shkolnik
>         Attachments: HADOOP-6188.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-730-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 18:18:32 2009
Return-Path: <common-issues-return-730-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14502 invoked from network); 11 Aug 2009 18:18:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 18:18:32 -0000
Received: (qmail 33490 invoked by uid 500); 11 Aug 2009 18:18:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33460 invoked by uid 500); 11 Aug 2009 18:18:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33450 invoked by uid 99); 11 Aug 2009 18:18:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 18:18:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 18:18:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1E1F4234C1E9
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 11:18:15 -0700 (PDT)
Message-ID: <563085064.1250014695122.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 11:18:15 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6188) TestHDFSTrash fails because of
 TestTrash in common
In-Reply-To: <364050017.1250011694871.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742002#action_12742002 ] 

Konstantin Boudnik commented on HADOOP-6188:
--------------------------------------------

I'd suggest to declare method's parameters final: it will improve readability and make testing easier.

> TestHDFSTrash fails because of TestTrash in common
> --------------------------------------------------
>
>                 Key: HADOOP-6188
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6188
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Boris Shkolnik
>         Attachments: HADOOP-6188.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-731-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 18:42:32 2009
Return-Path: <common-issues-return-731-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34731 invoked from network); 11 Aug 2009 18:42:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 18:42:32 -0000
Received: (qmail 9594 invoked by uid 500); 11 Aug 2009 18:42:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9534 invoked by uid 500); 11 Aug 2009 18:42:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9524 invoked by uid 99); 11 Aug 2009 18:42:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 18:42:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 18:42:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DD902234C045
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 11:42:14 -0700 (PDT)
Message-ID: <1093054545.1250016134906.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 11:42:14 -0700 (PDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6188) TestHDFSTrash fails because of
 TestTrash in common
In-Reply-To: <364050017.1250011694871.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742010#action_12742010 ] 

Boris Shkolnik commented on HADOOP-6188:
----------------------------------------

test-patch:

     [exec] +1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec] 
     [exec] 


> TestHDFSTrash fails because of TestTrash in common
> --------------------------------------------------
>
>                 Key: HADOOP-6188
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6188
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Boris Shkolnik
>         Attachments: HADOOP-6188.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-732-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 19:06:34 2009
Return-Path: <common-issues-return-732-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43599 invoked from network); 11 Aug 2009 19:06:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 19:06:33 -0000
Received: (qmail 2109 invoked by uid 500); 11 Aug 2009 19:06:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2066 invoked by uid 500); 11 Aug 2009 19:06:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2056 invoked by uid 99); 11 Aug 2009 19:06:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 19:06:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 19:06:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 7465029A0022
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 12:06:15 -0700 (PDT)
Message-ID: <1278490186.1250017575475.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 12:06:15 -0700 (PDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6188) TestHDFSTrash fails because of
 TestTrash in common
In-Reply-To: <364050017.1250011694871.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742019#action_12742019 ] 

Boris Shkolnik commented on HADOOP-6188:
----------------------------------------

test-core:
run-test-core:
    [mkdir] Created dir: /home/borya/work/hadoop/hadoop-common/build/test/data
    [mkdir] Created dir: /home/borya/work/hadoop/hadoop-common/build/test/logs
     [copy] Copying 1 file to /home/borya/work/hadoop/hadoop-common/build/test/extraconf
    [junit] Running org.apache.hadoop.cli.TestCLI
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.308 sec
    [junit] Running org.apache.hadoop.conf.TestConfiguration
    [junit] Tests run: 13, Failures: 0, Errors: 0, Time elapsed: 1.244 sec
    [junit] Running org.apache.hadoop.conf.TestConfigurationSubclass
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.495 sec
    [junit] Running org.apache.hadoop.conf.TestGetInstances
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.435 sec
    [junit] Running org.apache.hadoop.filecache.TestDistributedCache
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 1.202 sec
    [junit] Running org.apache.hadoop.fs.TestBlockLocation
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.213 sec
    [junit] Running org.apache.hadoop.fs.TestChecksumFileSystem
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.674 sec
    [junit] Running org.apache.hadoop.fs.TestDFVariations
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.405 sec
    [junit] Running org.apache.hadoop.fs.TestDU
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 5.323 sec
    [junit] Running org.apache.hadoop.fs.TestGetFileBlockLocations
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 1.224 sec
    [junit] Running org.apache.hadoop.fs.TestGlobExpander
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.056 sec
    [junit] Running org.apache.hadoop.fs.TestLocalDirAllocator
    [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 0.859 sec
    [junit] Running org.apache.hadoop.fs.TestLocalFileSystem
    [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 1.096 sec
    [junit] Running org.apache.hadoop.fs.TestLocalFileSystemPermission
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.567 sec
    [junit] Running org.apache.hadoop.fs.TestPath
    [junit] Tests run: 8, Failures: 0, Errors: 0, Time elapsed: 0.088 sec
    [junit] Running org.apache.hadoop.fs.TestTrash
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 2.221 sec
    [junit] Running org.apache.hadoop.fs.TestTruncatedInputBug
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.571 sec
    [junit] Running org.apache.hadoop.fs.kfs.TestKosmosFileSystem
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.861 sec
    [junit] Running org.apache.hadoop.fs.permission.TestFsPermission
    [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.093 sec
    [junit] Running org.apache.hadoop.fs.s3.TestINode
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.081 sec
    [junit] Running org.apache.hadoop.fs.s3.TestInMemoryS3FileSystemContract
    [junit] Tests run: 29, Failures: 0, Errors: 0, Time elapsed: 2.435 sec
    [junit] Running org.apache.hadoop.fs.s3.TestS3Credentials
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.23 sec
    [junit] Running org.apache.hadoop.fs.s3.TestS3FileSystem
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.264 sec
    [junit] Running org.apache.hadoop.fs.s3native.TestInMemoryNativeS3FileSystemContract
    [junit] Tests run: 35, Failures: 0, Errors: 0, Time elapsed: 2.494 sec
    [junit] Running org.apache.hadoop.http.TestGlobalFilter
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 2.639 sec
    [junit] Running org.apache.hadoop.http.TestServletFilter
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 2.008 sec
    [junit] Running org.apache.hadoop.io.TestArrayFile
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 6.843 sec
    [junit] Running org.apache.hadoop.io.TestArrayWritable
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.273 sec
    [junit] Running org.apache.hadoop.io.TestBloomMapFile
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 1.085 sec
    [junit] Running org.apache.hadoop.io.TestBytesWritable
    [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.208 sec
    [junit] Running org.apache.hadoop.io.TestDefaultStringifier
    [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.364 sec
    [junit] Running org.apache.hadoop.io.TestEnumSetWritable
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.236 sec
    [junit] Running org.apache.hadoop.io.TestGenericWritable
    [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 0.584 sec
    [junit] Running org.apache.hadoop.io.TestMD5Hash
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.073 sec
    [junit] Running org.apache.hadoop.io.TestMapFile
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.79 sec
    [junit] Running org.apache.hadoop.io.TestMapWritable
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.271 sec
    [junit] Running org.apache.hadoop.io.TestSequenceFileSerialization
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.641 sec
    [junit] Running org.apache.hadoop.io.TestSetFile
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 6.456 sec
    [junit] Running org.apache.hadoop.io.TestSortedMapWritable
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.267 sec
    [junit] Running org.apache.hadoop.io.TestText
    [junit] Tests run: 9, Failures: 0, Errors: 0, Time elapsed: 1.209 sec
    [junit] Running org.apache.hadoop.io.TestTextNonUTF8
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.046 sec
    [junit] Running org.apache.hadoop.io.TestUTF8
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.234 sec
    [junit] Running org.apache.hadoop.io.TestVersionedWritable
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.079 sec
    [junit] Running org.apache.hadoop.io.TestWritable
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.081 sec
    [junit] Running org.apache.hadoop.io.TestWritableName
    [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.222 sec
    [junit] Running org.apache.hadoop.io.TestWritableUtils
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.04 sec
    [junit] Running org.apache.hadoop.io.compress.TestCodec
    [junit] Tests run: 6, Failures: 0, Errors: 0, Time elapsed: 22.523 sec
    [junit] Running org.apache.hadoop.io.compress.TestCodecFactory
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.524 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestTFile
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 2.854 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestTFileByteArrays
    [junit] Tests run: 25, Failures: 0, Errors: 0, Time elapsed: 6.889 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestTFileComparators
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.562 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestTFileJClassComparatorByteArrays
    [junit] Tests run: 25, Failures: 0, Errors: 0, Time elapsed: 6.668 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestTFileLzoCodecsByteArrays
    [junit] Tests run: 25, Failures: 0, Errors: 0, Time elapsed: 0.273 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestTFileLzoCodecsStreams
    [junit] Tests run: 19, Failures: 0, Errors: 0, Time elapsed: 0.217 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestTFileNoneCodecsByteArrays
    [junit] Tests run: 25, Failures: 0, Errors: 0, Time elapsed: 3.59 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestTFileNoneCodecsJClassComparatorByteArrays
    [junit] Tests run: 25, Failures: 0, Errors: 0, Time elapsed: 3.614 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestTFileNoneCodecsStreams
    [junit] Tests run: 19, Failures: 0, Errors: 0, Time elapsed: 3.937 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestTFileSeek
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 6.799 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestTFileSeqFileComparison
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 55.548 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestTFileSplit
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 12.042 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestTFileStreams
    [junit] Tests run: 19, Failures: 0, Errors: 0, Time elapsed: 3.815 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestTFileUnsortedByteArrays
    [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.971 sec
    [junit] Running org.apache.hadoop.io.file.tfile.TestVLong
    [junit] Tests run: 9, Failures: 0, Errors: 0, Time elapsed: 3.778 sec
    [junit] Running org.apache.hadoop.io.retry.TestRetryProxy
    [junit] Tests run: 9, Failures: 0, Errors: 0, Time elapsed: 0.394 sec
    [junit] Running org.apache.hadoop.io.serializer.TestWritableSerialization
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.253 sec
    [junit] Running org.apache.hadoop.io.serializer.avro.TestAvroSerialization
    [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.658 sec
    [junit] Running org.apache.hadoop.ipc.TestIPC
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 15.36 sec
    [junit] Running org.apache.hadoop.ipc.TestIPCServerResponder
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 12.436 sec
    [junit] Running org.apache.hadoop.ipc.TestRPC
    [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 34.981 sec
    [junit] Running org.apache.hadoop.log.TestLogLevel
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 1.38 sec
    [junit] Running org.apache.hadoop.metrics.TestMetricsServlet
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.145 sec
    [junit] Running org.apache.hadoop.metrics.spi.TestOutputRecord
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.045 sec
    [junit] Running org.apache.hadoop.net.TestDNS
    [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 0.128 sec
    [junit] Running org.apache.hadoop.net.TestScriptBasedMapping
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.535 sec
    [junit] Running org.apache.hadoop.net.TestSocketIOWithTimeout
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 3.086 sec
    [junit] Running org.apache.hadoop.record.TestBuffer
    [junit] Tests run: 6, Failures: 0, Errors: 0, Time elapsed: 0.068 sec
    [junit] Running org.apache.hadoop.record.TestRecordIO
    [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 0.202 sec
    [junit] Running org.apache.hadoop.record.TestRecordVersioning
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.14 sec
    [junit] Running org.apache.hadoop.security.TestAccessControlList
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.064 sec
    [junit] Running org.apache.hadoop.security.TestAccessToken
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.989 sec
    [junit] Running org.apache.hadoop.security.TestUnixUserGroupInformation
    [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.464 sec
    [junit] Running org.apache.hadoop.security.authorize.TestConfiguredPolicy
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.416 sec
    [junit] Running org.apache.hadoop.util.TestCyclicIteration
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.059 sec
    [junit] Running org.apache.hadoop.util.TestGenericsUtil
    [junit] Tests run: 6, Failures: 0, Errors: 0, Time elapsed: 0.27 sec
    [junit] Running org.apache.hadoop.util.TestIndexedSort
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 2.144 sec
    [junit] Running org.apache.hadoop.util.TestProcfsBasedProcessTree
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 7.744 sec
    [junit] Running org.apache.hadoop.util.TestPureJavaCrc32
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 1.305 sec
    [junit] Running org.apache.hadoop.util.TestShell
    [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 4.283 sec
    [junit] Running org.apache.hadoop.util.TestStringUtils
    [junit] Tests run: 6, Failures: 0, Errors: 0, Time elapsed: 0.081 sec



> TestHDFSTrash fails because of TestTrash in common
> --------------------------------------------------
>
>                 Key: HADOOP-6188
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6188
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Boris Shkolnik
>         Attachments: HADOOP-6188.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-733-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 19:38:32 2009
Return-Path: <common-issues-return-733-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54102 invoked from network); 11 Aug 2009 19:38:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 19:38:31 -0000
Received: (qmail 27365 invoked by uid 500); 11 Aug 2009 19:38:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27322 invoked by uid 500); 11 Aug 2009 19:38:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27312 invoked by uid 99); 11 Aug 2009 19:38:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 19:38:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 19:38:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E55BE29A0017
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 12:38:14 -0700 (PDT)
Message-ID: <208483304.1250019494938.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 12:38:14 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6188) TestHDFSTrash fails because of
 TestTrash in common
In-Reply-To: <364050017.1250011694871.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6188:
--------------------------------

    Hadoop Flags: [Reviewed]

> TestHDFSTrash fails because of TestTrash in common
> --------------------------------------------------
>
>                 Key: HADOOP-6188
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6188
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Boris Shkolnik
>         Attachments: HADOOP-6188.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-734-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 19:38:32 2009
Return-Path: <common-issues-return-734-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54118 invoked from network); 11 Aug 2009 19:38:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 19:38:31 -0000
Received: (qmail 27582 invoked by uid 500); 11 Aug 2009 19:38:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27537 invoked by uid 500); 11 Aug 2009 19:38:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27404 invoked by uid 99); 11 Aug 2009 19:38:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 19:38:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 19:38:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CD5DE234C004
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 12:38:14 -0700 (PDT)
Message-ID: <1110066238.1250019494830.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 12:38:14 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6188) TestHDFSTrash fails because of
 TestTrash in common
In-Reply-To: <364050017.1250011694871.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742030#action_12742030 ] 

Jakob Homan commented on HADOOP-6188:
-------------------------------------

+1 on the new patch. I also manually tested that with the new jar TestHDFSTrash no longer fails.

 I'm not overly concerned with the non-final parameters as the function is just a few lines and the variables aren't being abused.  

Of note is the necessity of checking for null from listStatus.  This is to avoid the fact that the listStatus implementation differs between Local/ChecksumFileSystem and DistributedFileSystem.  I've opened HDFS-538 to address this.

> TestHDFSTrash fails because of TestTrash in common
> --------------------------------------------------
>
>                 Key: HADOOP-6188
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6188
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Boris Shkolnik
>         Attachments: HADOOP-6188.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-735-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 20:28:30 2009
Return-Path: <common-issues-return-735-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69315 invoked from network); 11 Aug 2009 20:28:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 20:28:30 -0000
Received: (qmail 28473 invoked by uid 500); 11 Aug 2009 20:28:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28385 invoked by uid 500); 11 Aug 2009 20:28:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28290 invoked by uid 99); 11 Aug 2009 20:28:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 20:28:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 20:28:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E374B29A0014
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 13:28:14 -0700 (PDT)
Message-ID: <1960562141.1250022494930.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 13:28:14 -0700 (PDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6188) TestHDFSTrash fails because of
 TestTrash in common
In-Reply-To: <364050017.1250011694871.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6188:
-----------------------------------

    Status: Patch Available  (was: Open)

> TestHDFSTrash fails because of TestTrash in common
> --------------------------------------------------
>
>                 Key: HADOOP-6188
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6188
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Boris Shkolnik
>         Attachments: HADOOP-6188.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-736-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 21:03:17 2009
Return-Path: <common-issues-return-736-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77676 invoked from network); 11 Aug 2009 21:03:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 21:03:17 -0000
Received: (qmail 98016 invoked by uid 500); 11 Aug 2009 20:58:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97997 invoked by uid 500); 11 Aug 2009 20:58:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97987 invoked by uid 99); 11 Aug 2009 20:58:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 20:58:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 20:58:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C93CB29A0011
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 13:58:14 -0700 (PDT)
Message-ID: <824054048.1250024294810.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 13:58:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6188) TestHDFSTrash fails because of
 TestTrash in common
In-Reply-To: <364050017.1250011694871.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6188:
-------------------------------------------

          Component/s:     (was: fs)
                       test
    Affects Version/s: 0.21.0
        Fix Version/s: 0.21.0
             Assignee: Boris Shkolnik

> TestHDFSTrash fails because of TestTrash in common
> --------------------------------------------------
>
>                 Key: HADOOP-6188
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6188
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6188.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-737-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 21:16:32 2009
Return-Path: <common-issues-return-737-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83336 invoked from network); 11 Aug 2009 21:16:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 21:16:32 -0000
Received: (qmail 93935 invoked by uid 500); 11 Aug 2009 21:16:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93884 invoked by uid 500); 11 Aug 2009 21:16:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93874 invoked by uid 99); 11 Aug 2009 21:16:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 21:16:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 21:16:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C47EF234C004
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 14:16:14 -0700 (PDT)
Message-ID: <236425854.1250025374788.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 14:16:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6188) TestHDFSTrash fails because of
 TestTrash in common
In-Reply-To: <364050017.1250011694871.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6188:
-------------------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I have committed this.  Thanks, Boris!

> TestHDFSTrash fails because of TestTrash in common
> --------------------------------------------------
>
>                 Key: HADOOP-6188
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6188
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6188.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-738-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 21:38:35 2009
Return-Path: <common-issues-return-738-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95630 invoked from network); 11 Aug 2009 21:38:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 21:38:35 -0000
Received: (qmail 72869 invoked by uid 500); 11 Aug 2009 21:38:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72840 invoked by uid 500); 11 Aug 2009 21:38:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72830 invoked by uid 99); 11 Aug 2009 21:38:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 21:38:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 21:38:38 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CF9D629A0016
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 14:38:17 -0700 (PDT)
Message-ID: <57783920.1250026697849.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 14:38:17 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6152) Hadoop scripts do not correctly put
 jars on the classpath
In-Reply-To: <283943906.1247689875148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742075#action_12742075 ] 

Hadoop QA commented on HADOOP-6152:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12416220/HADOOP-6152.2.patch
  against trunk revision 803296.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/599/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/599/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/599/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/599/console

This message is automatically generated.

> Hadoop scripts do not correctly put jars on the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-6152
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6152
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>            Priority: Blocker
>         Attachments: HADOOP-6152.2.patch, HADOOP-6152.patch
>
>
> The various Hadoop scripts (bin/hadoop, bin/hdfs, bin/mapred) do not properly identify the jars needed to run Hadoop. They try to include hadoop-*-hdfs.jar, etc, rather than the hadoop-hdfs-*.jar that is actually created by the 'ant jar' and 'ant package' targets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-739-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 11 22:22:15 2009
Return-Path: <common-issues-return-739-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16482 invoked from network); 11 Aug 2009 22:22:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 11 Aug 2009 22:22:15 -0000
Received: (qmail 51510 invoked by uid 500); 11 Aug 2009 22:02:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51476 invoked by uid 500); 11 Aug 2009 22:02:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51454 invoked by uid 99); 11 Aug 2009 22:02:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 22:02:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2009 22:02:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 21B4129A0023
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 15:02:15 -0700 (PDT)
Message-ID: <622378868.1250028135136.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 15:02:15 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6152) Hadoop scripts do not correctly put
 jars on the classpath
In-Reply-To: <283943906.1247689875148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742087#action_12742087 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6152:
------------------------------------------------

Look like that the jar file for tools is in mapreduce but not common.  Could you check it?
{code}
-for f in $HADOOP_CORE_HOME/hadoop-*-tools.jar; do
+for f in $HADOOP_CORE_HOME/hadoop-tools-*.jar; do
   TOOL_PATH=${TOOL_PATH}:$f;
 done
-for f in $HADOOP_CORE_HOME/build/hadoop-*-tools.jar; do
+for f in $HADOOP_CORE_HOME/build/hadoop-tools-*.jar; do
   TOOL_PATH=${TOOL_PATH}:$f;
 done
{code}

> Hadoop scripts do not correctly put jars on the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-6152
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6152
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>            Priority: Blocker
>         Attachments: HADOOP-6152.2.patch, HADOOP-6152.patch
>
>
> The various Hadoop scripts (bin/hadoop, bin/hdfs, bin/mapred) do not properly identify the jars needed to run Hadoop. They try to include hadoop-*-hdfs.jar, etc, rather than the hadoop-hdfs-*.jar that is actually created by the 'ant jar' and 'ant package' targets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-740-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 12 01:08:32 2009
Return-Path: <common-issues-return-740-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72783 invoked from network); 12 Aug 2009 01:08:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Aug 2009 01:08:31 -0000
Received: (qmail 77239 invoked by uid 500); 12 Aug 2009 01:08:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77146 invoked by uid 500); 12 Aug 2009 01:08:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76780 invoked by uid 99); 12 Aug 2009 01:08:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 01:08:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 01:08:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D367929A0011
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 18:08:14 -0700 (PDT)
Message-ID: <1028581514.1250039294864.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 18:08:14 -0700 (PDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6189) TestNameNodeMetrics fails
 intermittently
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

TestNameNodeMetrics fails intermittently
----------------------------------------

                 Key: HADOOP-6189
                 URL: https://issues.apache.org/jira/browse/HADOOP-6189
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.21.0
            Reporter: Suresh Srinivas
            Assignee: Suresh Srinivas
             Fix For: 0.21.0


TestNameNodeMetrics has strict timing constraint that relies on block management functionality and can fail intermittently.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-741-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 12 01:42:31 2009
Return-Path: <common-issues-return-741-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89856 invoked from network); 12 Aug 2009 01:42:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Aug 2009 01:42:30 -0000
Received: (qmail 56932 invoked by uid 500); 12 Aug 2009 01:42:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56862 invoked by uid 500); 12 Aug 2009 01:42:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56833 invoked by uid 99); 12 Aug 2009 01:42:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 01:42:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 01:42:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DA9FF29A0014
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 18:42:14 -0700 (PDT)
Message-ID: <1075643677.1250041334894.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 18:42:14 -0700 (PDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6152) Hadoop scripts do not correctly put
 jars on the classpath
In-Reply-To: <283943906.1247689875148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron Kimball updated HADOOP-6152:
----------------------------------

    Attachment: HADOOP-6152.3.patch

Good catch. Adding a new patch that handles this case.

> Hadoop scripts do not correctly put jars on the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-6152
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6152
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>            Priority: Blocker
>         Attachments: HADOOP-6152.2.patch, HADOOP-6152.3.patch, HADOOP-6152.patch
>
>
> The various Hadoop scripts (bin/hadoop, bin/hdfs, bin/mapred) do not properly identify the jars needed to run Hadoop. They try to include hadoop-*-hdfs.jar, etc, rather than the hadoop-hdfs-*.jar that is actually created by the 'ant jar' and 'ant package' targets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-742-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 12 01:42:31 2009
Return-Path: <common-issues-return-742-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89858 invoked from network); 12 Aug 2009 01:42:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Aug 2009 01:42:30 -0000
Received: (qmail 56966 invoked by uid 500); 12 Aug 2009 01:42:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56866 invoked by uid 500); 12 Aug 2009 01:42:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56834 invoked by uid 99); 12 Aug 2009 01:42:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 01:42:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 01:42:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EC27529A0020
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 18:42:14 -0700 (PDT)
Message-ID: <867638885.1250041334966.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 18:42:14 -0700 (PDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6152) Hadoop scripts do not correctly put
 jars on the classpath
In-Reply-To: <283943906.1247689875148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron Kimball updated HADOOP-6152:
----------------------------------

    Status: Patch Available  (was: Open)

> Hadoop scripts do not correctly put jars on the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-6152
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6152
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>            Priority: Blocker
>         Attachments: HADOOP-6152.2.patch, HADOOP-6152.3.patch, HADOOP-6152.patch
>
>
> The various Hadoop scripts (bin/hadoop, bin/hdfs, bin/mapred) do not properly identify the jars needed to run Hadoop. They try to include hadoop-*-hdfs.jar, etc, rather than the hadoop-hdfs-*.jar that is actually created by the 'ant jar' and 'ant package' targets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-743-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 12 01:42:32 2009
Return-Path: <common-issues-return-743-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89940 invoked from network); 12 Aug 2009 01:42:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Aug 2009 01:42:31 -0000
Received: (qmail 57177 invoked by uid 500); 12 Aug 2009 01:42:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57122 invoked by uid 500); 12 Aug 2009 01:42:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57112 invoked by uid 99); 12 Aug 2009 01:42:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 01:42:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 01:42:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E39FA29A001A
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 18:42:14 -0700 (PDT)
Message-ID: <741257113.1250041334931.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 18:42:14 -0700 (PDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6152) Hadoop scripts do not correctly put
 jars on the classpath
In-Reply-To: <283943906.1247689875148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron Kimball updated HADOOP-6152:
----------------------------------

    Status: Open  (was: Patch Available)

> Hadoop scripts do not correctly put jars on the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-6152
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6152
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>            Priority: Blocker
>         Attachments: HADOOP-6152.2.patch, HADOOP-6152.3.patch, HADOOP-6152.patch
>
>
> The various Hadoop scripts (bin/hadoop, bin/hdfs, bin/mapred) do not properly identify the jars needed to run Hadoop. They try to include hadoop-*-hdfs.jar, etc, rather than the hadoop-hdfs-*.jar that is actually created by the 'ant jar' and 'ant package' targets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-744-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 12 01:58:30 2009
Return-Path: <common-issues-return-744-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98133 invoked from network); 12 Aug 2009 01:58:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Aug 2009 01:58:30 -0000
Received: (qmail 6076 invoked by uid 500); 12 Aug 2009 01:58:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6048 invoked by uid 500); 12 Aug 2009 01:58:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6038 invoked by uid 99); 12 Aug 2009 01:58:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 01:58:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 01:58:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D2F8C29A0014
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 18:58:14 -0700 (PDT)
Message-ID: <632504967.1250042294863.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 18:58:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6152) Hadoop scripts do not correctly put
 jars on the classpath
In-Reply-To: <283943906.1247689875148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742179#action_12742179 ] 

Hadoop QA commented on HADOOP-6152:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12416278/HADOOP-6152.3.patch
  against trunk revision 803296.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/601/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/601/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/601/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/601/console

This message is automatically generated.

> Hadoop scripts do not correctly put jars on the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-6152
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6152
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>            Priority: Blocker
>         Attachments: HADOOP-6152.2.patch, HADOOP-6152.3.patch, HADOOP-6152.patch
>
>
> The various Hadoop scripts (bin/hadoop, bin/hdfs, bin/mapred) do not properly identify the jars needed to run Hadoop. They try to include hadoop-*-hdfs.jar, etc, rather than the hadoop-hdfs-*.jar that is actually created by the 'ant jar' and 'ant package' targets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-745-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 12 04:20:32 2009
Return-Path: <common-issues-return-745-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40713 invoked from network); 12 Aug 2009 04:20:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Aug 2009 04:20:32 -0000
Received: (qmail 4737 invoked by uid 500); 12 Aug 2009 04:20:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4683 invoked by uid 500); 12 Aug 2009 04:20:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4673 invoked by uid 99); 12 Aug 2009 04:20:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 04:20:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 04:20:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D47B4234C004
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 21:20:14 -0700 (PDT)
Message-ID: <878552299.1250050814854.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 21:20:14 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6105) Provide a way to automatically handle
 backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna updated HADOOP-6105:
------------------------------------------

    Attachment: HADOOP-6105-3.patch

uploaded patch with the following test cases being added:

1. User sets an old key and gets new key.

2. User sets a new key and gets old key.

The results should be the value which is set most recently, i.e., value corresponding to old key in the first case and value corresponding to new key in the second case.

> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>         Attachments: HADOOP-6105-1.patch, HADOOP-6105-2.patch, HADOOP-6105-3.patch, HADOOP-6105.patch, HADOOP-6105.patch
>
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-746-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 12 05:07:30 2009
Return-Path: <common-issues-return-746-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51139 invoked from network); 12 Aug 2009 05:07:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Aug 2009 05:07:30 -0000
Received: (qmail 47408 invoked by uid 500); 12 Aug 2009 05:07:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47349 invoked by uid 500); 12 Aug 2009 05:07:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47339 invoked by uid 99); 12 Aug 2009 05:07:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 05:07:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 05:07:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DC8EF234C004
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 22:07:14 -0700 (PDT)
Message-ID: <1065249550.1250053634896.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 22:07:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6166:
-------------------------------------------

    Attachment: c6166_20090811.patch

c6166_20090811.patch: added Crc32_8_8e and deleted some old classes.

> We probably want to have Todd's concurrency test from HADOOP-5318 run to make sure the larger lookup table doesn't slow things down under concurrency.
Todd, could you help running the test?

> We might also want to try the old four at a time code for 4 <= len < 8.
What do you mean exactly?

> We should also confirm the results on one of the other systems we tested in the past. I won't be able to do that for a couple days, but it should be easy then.
Hope you could find some time to do it soon.

> How many other variants did you try? Intel's C code does some strange things to group CRC's by 3. ...
Many others.  You know, there are many combinations.  I also tried different table sizes as shown before.

Included Crc32_8_8e below.  Crc32_8_8d still seems the best choice.


- java.version = 1.6.0_15
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_15-b03
java.vm.version = 14.1-b02
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = amd64
os.name = Linux
os.version = 2.6.9-55.ELsmp

Performance Table (The unit is MB/sec)
|| Num Bytes ||    CRC32 || PureJavaCrc32 || Crc32_8_8 || Crc32_8_8b || Crc32_8_8c || Crc32_8_8d || Crc32_8_8e || Crc32_16_16 || Crc32_16_16b || Crc32_16_16c || Crc32_16_16d ||
|          1 |     7.554 |         71.591 |     79.205 |     103.936 |      80.965 |      79.185 |      80.812 |       84.215 |        80.733 |        80.732 |        84.860 |
|          2 |    14.768 |        104.753 |    110.837 |     110.771 |     110.798 |     115.222 |     110.820 |      119.278 |       110.844 |       110.836 |       119.207 |
|          4 |    27.150 |        177.142 |    119.587 |     114.780 |     120.051 |     128.626 |     117.005 |      125.412 |       115.439 |       120.409 |       125.486 |
|          8 |    49.921 |        193.631 |    239.096 |     238.506 |     228.868 |     248.667 |     244.446 |      217.270 |       209.170 |       209.149 |       215.505 |
|         16 |    83.886 |        259.453 |    261.683 |     254.016 |     240.835 |     258.250 |     247.261 |      267.441 |       279.439 |       276.864 |       271.520 |
|         32 |   128.960 |        312.202 |    323.233 |     319.924 |     301.175 |     322.962 |     320.275 |      295.428 |       302.402 |       294.960 |       295.872 |
|         64 |   177.529 |        349.882 |    362.037 |     364.918 |     347.768 |     365.013 |     358.567 |      326.303 |       352.837 |       345.517 |       347.880 |
|        128 |   217.577 |        370.920 |    387.254 |     386.534 |     372.472 |     391.180 |     386.159 |      339.850 |       383.044 |       373.716 |       381.058 |
|        256 |   245.685 |        382.041 |    403.693 |     401.793 |     390.270 |     406.037 |     402.777 |      347.134 |       400.956 |       388.941 |       399.331 |
|        512 |   263.143 |        385.666 |    411.380 |     407.443 |     396.506 |     413.556 |     411.803 |      350.844 |       411.293 |       397.335 |       409.011 |
|       1024 |   271.941 |        390.055 |    406.949 |     406.830 |     399.163 |     415.963 |     424.095 |      352.875 |       417.516 |       401.168 |       414.229 |
|       2048 |   276.881 |        392.684 |    412.759 |     411.123 |     402.143 |     418.730 |     424.565 |      353.111 |       418.855 |       402.870 |       415.696 |
|       4096 |   279.541 |        393.738 |    418.844 |     413.271 |     403.645 |     419.106 |     424.975 |      353.582 |       421.574 |       403.834 |       417.058 |
|       8192 |   280.308 |        392.859 |    417.096 |     412.115 |     403.015 |     417.989 |     422.254 |      352.011 |       419.664 |       402.378 |       415.385 |
|      16384 |   280.420 |        393.006 |    415.629 |     409.566 |     403.498 |     418.107 |     420.676 |      350.423 |       418.062 |       401.090 |       412.468 |
|      32768 |   280.488 |        392.172 |    410.737 |     411.129 |     403.873 |     417.434 |     413.839 |      349.526 |       418.633 |       401.736 |       412.341 |
|      65536 |   281.809 |        393.565 |    411.490 |     412.331 |     405.347 |     419.190 |     414.866 |      350.781 |       418.473 |       401.637 |       412.297 |
|     131072 |   281.994 |        393.634 |    411.533 |     412.281 |     405.387 |     419.620 |     414.802 |      350.869 |       418.350 |       401.634 |       412.346 |
|     262144 |   282.106 |        393.732 |    411.339 |     411.562 |     404.172 |     416.299 |     413.072 |      349.416 |       416.190 |       400.395 |       410.680 |
|     524288 |   281.194 |        392.195 |    409.839 |     410.759 |     403.954 |     417.950 |     413.178 |      350.104 |       418.596 |       401.175 |       412.137 |
|    1048576 |   282.160 |        393.700 |    411.316 |     412.339 |     405.165 |     419.694 |     414.608 |      351.029 |       416.836 |       401.927 |       412.122 |
|    2097152 |   281.584 |        393.264 |    410.322 |     405.048 |     402.028 |     414.893 |     406.574 |      328.270 |       410.558 |       393.239 |       403.796 |
|    4194304 |   274.062 |        385.751 |    409.093 |     408.604 |     402.332 |     419.176 |     412.265 |      350.039 |       415.385 |       400.332 |       410.738 |
|    8388608 |   279.870 |        385.793 |    406.788 |     407.685 |     400.830 |     415.537 |     409.612 |      344.396 |       412.684 |       397.534 |       406.201 |
|   16777216 |   279.585 |        389.025 |    405.902 |     407.222 |     400.414 |     414.680 |     409.184 |      346.930 |       413.039 |       396.057 |       406.953 |

- java.version = 1.6.0_14
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_14-b08
java.vm.version = 14.0-b16
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = x86
os.name = Windows XP
os.version = 5.1

Performance Table (The unit is MB/sec)
|| Num Bytes ||    CRC32 || PureJavaCrc32 || Crc32_8_8 || Crc32_8_8b || Crc32_8_8c || Crc32_8_8d || Crc32_8_8e || Crc32_16_16 || Crc32_16_16b || Crc32_16_16c || Crc32_16_16d ||
|          1 |     4.669 |         62.483 |     67.995 |      64.360 |      64.700 |      64.430 |      51.386 |       61.391 |        62.175 |        68.944 |        62.271 |
|          2 |     9.959 |         79.342 |     88.566 |      79.881 |      91.326 |      88.695 |      81.976 |       89.846 |        83.984 |        91.166 |        84.918 |
|          4 |    19.448 |        136.996 |    119.941 |     129.939 |     120.992 |     119.898 |     132.349 |      118.682 |       116.076 |       122.505 |       114.090 |
|          8 |    36.518 |        168.433 |    240.284 |     219.896 |     213.865 |     212.471 |     209.605 |      115.325 |       135.040 |       145.009 |       139.316 |
|         16 |    66.157 |        233.146 |    272.227 |     258.220 |     250.948 |     245.771 |     253.216 |      313.174 |       321.156 |       289.904 |       285.301 |
|         32 |   111.982 |        283.081 |    327.881 |     326.083 |     311.988 |     300.416 |     299.799 |      331.685 |       338.723 |       316.987 |       319.500 |
|         64 |   169.448 |        315.802 |    364.829 |     368.059 |     348.219 |     340.445 |     323.045 |      360.273 |       357.086 |       352.497 |       359.832 |
|        128 |   227.468 |        335.743 |    384.770 |     394.813 |     373.775 |     368.234 |     338.247 |      379.686 |       363.607 |       372.900 |       382.183 |
|        256 |   274.901 |        348.319 |    395.674 |     410.081 |     386.566 |     388.908 |     345.215 |      385.142 |       368.920 |       383.446 |       392.874 |
|        512 |   307.317 |        353.626 |    401.379 |     422.212 |     392.543 |     399.756 |     348.747 |      391.548 |       372.486 |       389.674 |       400.035 |
|       1024 |   325.610 |        358.707 |    408.000 |     424.190 |     390.858 |     386.952 |     349.863 |      392.606 |       373.485 |       393.192 |       403.376 |
|       2048 |   337.261 |        361.642 |    410.563 |     429.318 |     395.794 |     388.887 |     351.070 |      392.865 |       373.458 |       394.564 |       404.084 |
|       4096 |   346.026 |        363.077 |    408.908 |     432.169 |     402.087 |     394.633 |     351.254 |      393.764 |       372.779 |       395.751 |       405.531 |
|       8192 |   348.807 |        363.550 |    407.264 |     431.891 |     409.774 |     400.138 |     350.242 |      393.582 |       373.939 |       394.498 |       405.981 |
|      16384 |   350.030 |        361.773 |    407.574 |     433.393 |     413.786 |     399.135 |     352.191 |      391.875 |       371.887 |       396.563 |       404.991 |
|      32768 |   350.552 |        360.131 |    408.495 |     428.463 |     413.372 |     406.904 |     352.049 |      390.723 |       371.160 |       393.841 |       403.526 |
|      65536 |   350.624 |        359.343 |    407.846 |     427.477 |     411.844 |     407.847 |     351.866 |      388.578 |       371.861 |       394.314 |       403.752 |
|     131072 |   351.383 |        358.687 |    407.854 |     426.513 |     411.453 |     407.655 |     352.480 |      388.961 |       372.343 |       394.181 |       401.394 |
|     262144 |   351.508 |        359.392 |    406.755 |     427.817 |     413.534 |     407.467 |     352.457 |      389.811 |       370.843 |       396.034 |       404.051 |
|     524288 |   350.264 |        358.659 |    407.698 |     427.750 |     411.986 |     408.091 |     351.678 |      390.126 |       371.458 |       395.884 |       404.374 |
|    1048576 |   351.092 |        358.452 |    406.796 |     426.634 |     413.797 |     408.162 |     351.430 |      389.716 |       372.157 |       394.826 |       403.300 |
|    2097152 |   349.731 |        357.048 |    407.095 |     426.922 |     411.427 |     405.970 |     351.830 |      388.308 |       371.514 |       394.656 |       401.170 |
|    4194304 |   345.948 |        354.513 |    401.117 |     423.269 |     408.442 |     401.112 |     348.512 |      384.834 |       365.403 |       390.539 |       399.911 |
|    8388608 |   343.162 |        350.602 |    400.009 |     415.757 |     403.662 |     399.569 |     344.909 |      382.301 |       365.704 |       386.523 |       397.156 |
|   16777216 |   343.989 |        351.502 |    400.036 |     417.893 |     403.717 |     400.511 |     345.505 |      381.932 |       363.491 |       385.044 |       393.422 |

> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt, c6166_20090727.patch, c6166_20090728.patch, c6166_20090810.patch, c6166_20090811.patch
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-747-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 12 05:17:30 2009
Return-Path: <common-issues-return-747-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56886 invoked from network); 12 Aug 2009 05:17:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Aug 2009 05:17:30 -0000
Received: (qmail 59108 invoked by uid 500); 12 Aug 2009 05:17:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59057 invoked by uid 500); 12 Aug 2009 05:17:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59047 invoked by uid 99); 12 Aug 2009 05:17:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 05:17:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 05:17:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C4C79234C004
	for <common-issues@hadoop.apache.org>; Tue, 11 Aug 2009 22:17:14 -0700 (PDT)
Message-ID: <956799676.1250054234801.JavaMail.jira@brutus>
Date: Tue, 11 Aug 2009 22:17:14 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742221#action_12742221 ] 

Todd Lipcon commented on HADOOP-6166:
-------------------------------------

bq. Todd, could you help running the test?

Yep, we're doing sprint planning tomorrow so I'll make sure to block off a couple hours to run benchmarks for this.

> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt, c6166_20090727.patch, c6166_20090728.patch, c6166_20090810.patch, c6166_20090811.patch
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-748-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 12 09:32:34 2009
Return-Path: <common-issues-return-748-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59495 invoked from network); 12 Aug 2009 09:32:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Aug 2009 09:32:34 -0000
Received: (qmail 96234 invoked by uid 500); 12 Aug 2009 09:32:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96138 invoked by uid 500); 12 Aug 2009 09:32:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95833 invoked by uid 99); 12 Aug 2009 09:32:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 09:32:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 09:32:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D1D12234C04C
	for <common-issues@hadoop.apache.org>; Wed, 12 Aug 2009 02:32:14 -0700 (PDT)
Message-ID: <198509755.1250069534858.JavaMail.jira@brutus>
Date: Wed, 12 Aug 2009 02:32:14 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6190) Links to Hadoop mailing list mbox
 files are broken
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Links to Hadoop mailing list mbox files are broken
--------------------------------------------------

                 Key: HADOOP-6190
                 URL: https://issues.apache.org/jira/browse/HADOOP-6190
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Tom White
             Fix For: site


http://hadoop.apache.org/mail/ returns a 404 (compare, for example, http://lucene.apache.org/mail/). There should be a symbolic link in /www/hadoop.apache.org/ from mail to /home/apmail/public-arch/hadoop.apache.org on people.apache.org.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-749-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 12 13:09:37 2009
Return-Path: <common-issues-return-749-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66631 invoked from network); 12 Aug 2009 13:09:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Aug 2009 13:09:37 -0000
Received: (qmail 21289 invoked by uid 500); 12 Aug 2009 13:09:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21254 invoked by uid 500); 12 Aug 2009 13:09:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21244 invoked by uid 99); 12 Aug 2009 13:09:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 13:09:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 13:09:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3934E234C04C
	for <common-issues@hadoop.apache.org>; Wed, 12 Aug 2009 06:09:15 -0700 (PDT)
Message-ID: <2034230348.1250082555233.JavaMail.jira@brutus>
Date: Wed, 12 Aug 2009 06:09:15 -0700 (PDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6032) reference to java1.5 in
 conf/hadoop-env.sh
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steve Loughran resolved HADOOP-6032.
------------------------------------

    Resolution: Duplicate

> reference to java1.5 in conf/hadoop-env.sh
> ------------------------------------------
>
>                 Key: HADOOP-6032
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6032
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Priority: Trivial
>
> conf/hadoop-env.sh has a commented out line
> # export JAVA_HOME=/usr/lib/j2sdk1.5-sun
> that path should be moved up to java 1.6

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-750-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 12 13:49:34 2009
Return-Path: <common-issues-return-750-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89481 invoked from network); 12 Aug 2009 13:49:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Aug 2009 13:49:34 -0000
Received: (qmail 93895 invoked by uid 500); 12 Aug 2009 13:49:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93861 invoked by uid 500); 12 Aug 2009 13:49:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93831 invoked by uid 99); 12 Aug 2009 13:49:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 13:49:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 13:49:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id F39EB234C045
	for <common-issues@hadoop.apache.org>; Wed, 12 Aug 2009 06:49:14 -0700 (PDT)
Message-ID: <971255749.1250084954997.JavaMail.jira@brutus>
Date: Wed, 12 Aug 2009 06:49:14 -0700 (PDT)
From: "Pallavi Palleti (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6191) Allow Super user access only from
 certian trusted IP Range- This is to avoid spoofing by others as super user
 and gain access to the cluster
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Allow Super user access only from certian trusted IP Range- This is to avoid spoofing by others as super user and gain access to the cluster
--------------------------------------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-6191
                 URL: https://issues.apache.org/jira/browse/HADOOP-6191
             Project: Hadoop Common
          Issue Type: New Feature
    Affects Versions: 0.18.2
            Reporter: Pallavi Palleti
            Priority: Minor


This is similar to https://issues.apache.org/jira/browse/HADOOP-6187 but for 18.2 version

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-751-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 12 13:59:32 2009
Return-Path: <common-issues-return-751-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95178 invoked from network); 12 Aug 2009 13:59:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Aug 2009 13:59:32 -0000
Received: (qmail 17740 invoked by uid 500); 12 Aug 2009 13:59:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17681 invoked by uid 500); 12 Aug 2009 13:59:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17621 invoked by uid 99); 12 Aug 2009 13:59:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 13:59:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 13:59:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D51D8234C04C
	for <common-issues@hadoop.apache.org>; Wed, 12 Aug 2009 06:59:14 -0700 (PDT)
Message-ID: <1447127228.1250085554872.JavaMail.jira@brutus>
Date: Wed, 12 Aug 2009 06:59:14 -0700 (PDT)
From: "Pallavi Palleti (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6191) Allow Super user access only from
 certian trusted IP Range- This is to avoid spoofing by others as super user
 and gain access to the cluster
In-Reply-To: <971255749.1250084954997.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pallavi Palleti updated HADOOP-6191:
------------------------------------

    Attachment: commons-net-ftp-2.0.jar
                Hadoop-6191.patch

The patch restricts super user access only from some specified IP range if it is configured. if not, it works by default allowing super user access from any where. Also, I have attached the jar file that is needed for it to work.

> Allow Super user access only from certian trusted IP Range- This is to avoid spoofing by others as super user and gain access to the cluster
> --------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6191
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6191
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.18.2
>            Reporter: Pallavi Palleti
>            Priority: Minor
>         Attachments: commons-net-ftp-2.0.jar, Hadoop-6191.patch
>
>
> This is similar to https://issues.apache.org/jira/browse/HADOOP-6187 but for 18.2 version

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-752-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 12 14:15:30 2009
Return-Path: <common-issues-return-752-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5567 invoked from network); 12 Aug 2009 14:15:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Aug 2009 14:15:30 -0000
Received: (qmail 57524 invoked by uid 500); 12 Aug 2009 14:15:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57471 invoked by uid 500); 12 Aug 2009 14:15:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57460 invoked by uid 99); 12 Aug 2009 14:15:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 14:15:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 14:15:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E8223234C04B
	for <common-issues@hadoop.apache.org>; Wed, 12 Aug 2009 07:15:14 -0700 (PDT)
Message-ID: <1662931740.1250086514950.JavaMail.jira@brutus>
Date: Wed, 12 Aug 2009 07:15:14 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6177) FSInputChecker.getPos() would
 return position greater than the file size
In-Reply-To: <1554871263.1249618274995.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742376#action_12742376 ] 

Hudson commented on HADOOP-6177:
--------------------------------

Integrated in Hadoop-Common-trunk #52 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/52/])
    . FSInputChecker.getPos() would return position greater than the file size. Contributed by Hong Tang.


> FSInputChecker.getPos() would return position greater than the file size
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6177
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6177
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6177-2090806.patch, hadoop-6177-testcase.patch
>
>
> When using a small buffer (< 512 bytes) to read through the whole file, the final file position is 1+ the file size.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-753-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 12 14:15:30 2009
Return-Path: <common-issues-return-753-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5619 invoked from network); 12 Aug 2009 14:15:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Aug 2009 14:15:30 -0000
Received: (qmail 57708 invoked by uid 500); 12 Aug 2009 14:15:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57661 invoked by uid 500); 12 Aug 2009 14:15:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57645 invoked by uid 99); 12 Aug 2009 14:15:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 14:15:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 14:15:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id ECA9B234C052
	for <common-issues@hadoop.apache.org>; Wed, 12 Aug 2009 07:15:14 -0700 (PDT)
Message-ID: <1523758820.1250086514968.JavaMail.jira@brutus>
Date: Wed, 12 Aug 2009 07:15:14 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6188) TestHDFSTrash fails because of
 TestTrash in common
In-Reply-To: <364050017.1250011694871.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742377#action_12742377 ] 

Hudson commented on HADOOP-6188:
--------------------------------

Integrated in Hadoop-Common-trunk #52 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/52/])
    . TestTrash uses java.io.File api but not hadoop FileSystem api.  Contributed by Boris Shkolnik


> TestHDFSTrash fails because of TestTrash in common
> --------------------------------------------------
>
>                 Key: HADOOP-6188
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6188
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6188.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-754-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 12 17:55:31 2009
Return-Path: <common-issues-return-754-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2130 invoked from network); 12 Aug 2009 17:55:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Aug 2009 17:55:30 -0000
Received: (qmail 83887 invoked by uid 500); 12 Aug 2009 17:55:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83840 invoked by uid 500); 12 Aug 2009 17:55:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83718 invoked by uid 99); 12 Aug 2009 17:55:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 17:55:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 17:55:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E42C129A0015
	for <common-issues@hadoop.apache.org>; Wed, 12 Aug 2009 10:55:14 -0700 (PDT)
Message-ID: <298786703.1250099714933.JavaMail.jira@brutus>
Date: Wed, 12 Aug 2009 10:55:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6152) Hadoop scripts do not correctly put
 jars on the classpath
In-Reply-To: <283943906.1247689875148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742480#action_12742480 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6152:
------------------------------------------------

The tools jar file name should be something like hadoop-mapred-tools-0.21.0-dev.jar.
{code}
+for f in $HADOOP_MAPRED_HOME/hadoop-tools-*.jar; do
+  TOOL_PATH=${TOOL_PATH}:$f;
+done
+for f in $HADOOP_MAPRED_HOME/build/hadoop-tools-*.jar; do
+  TOOL_PATH=${TOOL_PATH}:$f;
+done
{code}

> Hadoop scripts do not correctly put jars on the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-6152
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6152
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>            Priority: Blocker
>         Attachments: HADOOP-6152.2.patch, HADOOP-6152.3.patch, HADOOP-6152.patch
>
>
> The various Hadoop scripts (bin/hadoop, bin/hdfs, bin/mapred) do not properly identify the jars needed to run Hadoop. They try to include hadoop-*-hdfs.jar, etc, rather than the hadoop-hdfs-*.jar that is actually created by the 'ant jar' and 'ant package' targets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-755-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 12 18:09:34 2009
Return-Path: <common-issues-return-755-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11442 invoked from network); 12 Aug 2009 18:09:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Aug 2009 18:09:34 -0000
Received: (qmail 7755 invoked by uid 500); 12 Aug 2009 18:09:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7662 invoked by uid 500); 12 Aug 2009 18:09:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7637 invoked by uid 99); 12 Aug 2009 18:09:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 18:09:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 18:09:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4B1EB234C045
	for <common-issues@hadoop.apache.org>; Wed, 12 Aug 2009 11:09:15 -0700 (PDT)
Message-ID: <1320465570.1250100555306.JavaMail.jira@brutus>
Date: Wed, 12 Aug 2009 11:09:15 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6190) Links to Hadoop mailing list mbox
 files are broken
In-Reply-To: <198509755.1250069534858.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting resolved HADOOP-6190.
----------------------------------

    Resolution: Fixed
      Assignee: Doug Cutting

Good idea.  I created that link.  Thanks for noticing this, Tom.

> Links to Hadoop mailing list mbox files are broken
> --------------------------------------------------
>
>                 Key: HADOOP-6190
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6190
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Doug Cutting
>             Fix For: site
>
>
> http://hadoop.apache.org/mail/ returns a 404 (compare, for example, http://lucene.apache.org/mail/). There should be a symbolic link in /www/hadoop.apache.org/ from mail to /home/apmail/public-arch/hadoop.apache.org on people.apache.org.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-756-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 12 18:11:34 2009
Return-Path: <common-issues-return-756-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12444 invoked from network); 12 Aug 2009 18:11:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Aug 2009 18:11:34 -0000
Received: (qmail 9880 invoked by uid 500); 12 Aug 2009 18:11:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9822 invoked by uid 500); 12 Aug 2009 18:11:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9729 invoked by uid 99); 12 Aug 2009 18:11:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 18:11:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 18:11:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 2BEB729A001C
	for <common-issues@hadoop.apache.org>; Wed, 12 Aug 2009 11:11:15 -0700 (PDT)
Message-ID: <493015276.1250100675179.JavaMail.jira@brutus>
Date: Wed, 12 Aug 2009 11:11:15 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Reopened: (HADOOP-6190) Links to Hadoop mailing list mbox
 files are broken
In-Reply-To: <198509755.1250069534858.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting reopened HADOOP-6190:
----------------------------------


Oops.  I accidentally linked to lucene's mail archives!  I fixed that and will wait until it shows up in the mirrors before resolving this.

> Links to Hadoop mailing list mbox files are broken
> --------------------------------------------------
>
>                 Key: HADOOP-6190
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6190
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Doug Cutting
>             Fix For: site
>
>
> http://hadoop.apache.org/mail/ returns a 404 (compare, for example, http://lucene.apache.org/mail/). There should be a symbolic link in /www/hadoop.apache.org/ from mail to /home/apmail/public-arch/hadoop.apache.org on people.apache.org.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-757-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 12 20:15:30 2009
Return-Path: <common-issues-return-757-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74267 invoked from network); 12 Aug 2009 20:15:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Aug 2009 20:15:30 -0000
Received: (qmail 28445 invoked by uid 500); 12 Aug 2009 20:15:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28392 invoked by uid 500); 12 Aug 2009 20:15:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28381 invoked by uid 99); 12 Aug 2009 20:15:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 20:15:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 20:15:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D9BFB29A0012
	for <common-issues@hadoop.apache.org>; Wed, 12 Aug 2009 13:15:14 -0700 (PDT)
Message-ID: <1437381609.1250108114890.JavaMail.jira@brutus>
Date: Wed, 12 Aug 2009 13:15:14 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6190) Links to Hadoop mailing list mbox
 files are broken
In-Reply-To: <198509755.1250069534858.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White resolved HADOOP-6190.
-------------------------------

    Resolution: Fixed

Thanks Doug. This is fixed now. I've updated http://wiki.apache.org/hadoop/MailingListArchives too.

> Links to Hadoop mailing list mbox files are broken
> --------------------------------------------------
>
>                 Key: HADOOP-6190
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6190
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Doug Cutting
>             Fix For: site
>
>
> http://hadoop.apache.org/mail/ returns a 404 (compare, for example, http://lucene.apache.org/mail/). There should be a symbolic link in /www/hadoop.apache.org/ from mail to /home/apmail/public-arch/hadoop.apache.org on people.apache.org.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-758-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 12 22:34:31 2009
Return-Path: <common-issues-return-758-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46092 invoked from network); 12 Aug 2009 22:34:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Aug 2009 22:34:31 -0000
Received: (qmail 27095 invoked by uid 500); 12 Aug 2009 22:34:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26959 invoked by uid 500); 12 Aug 2009 22:34:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26937 invoked by uid 99); 12 Aug 2009 22:34:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 22:34:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 22:34:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E2D81234C045
	for <common-issues@hadoop.apache.org>; Wed, 12 Aug 2009 15:34:14 -0700 (PDT)
Message-ID: <2116170715.1250116454928.JavaMail.jira@brutus>
Date: Wed, 12 Aug 2009 15:34:14 -0700 (PDT)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5107) split the core, hdfs, and mapred
 jars from each other and publish them independently to the Maven repository
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742621#action_12742621 ] 

Nigel Daley commented on HADOOP-5107:
-------------------------------------

Giri's had too many problems trying to publish Hadoop jars to the Apache Maven repo using Ivy.  He's now going to change direction and try the Maven Ant tasks and individual POM files (one for every jar file) to replace the Ivy functionality.  If this works, he'll propagate the work to the other Hadoop subprojects.  As part of this, the maven-ant-tasks.jar would be checked into the lib directory.

> split the core, hdfs, and mapred jars from each other and publish them independently to the Maven repository
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5107
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5107
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.0
>            Reporter: Owen O'Malley
>            Assignee: Giridharan Kesavan
>         Attachments: common-trunk.patch, hdfs-trunk.patch, mapreduce-trunk.patch
>
>
> I think to support splitting the projects, we should publish the jars for 0.20.0 as independent jars to the Maven repository 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-759-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 12 22:40:32 2009
Return-Path: <common-issues-return-759-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49561 invoked from network); 12 Aug 2009 22:40:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 12 Aug 2009 22:40:32 -0000
Received: (qmail 35098 invoked by uid 500); 12 Aug 2009 22:40:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35017 invoked by uid 500); 12 Aug 2009 22:40:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34991 invoked by uid 99); 12 Aug 2009 22:40:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 22:40:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 22:40:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DC1C029A0012
	for <common-issues@hadoop.apache.org>; Wed, 12 Aug 2009 15:40:14 -0700 (PDT)
Message-ID: <1255789210.1250116814900.JavaMail.jira@brutus>
Date: Wed, 12 Aug 2009 15:40:14 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5107) split the core, hdfs, and mapred
 jars from each other and publish them independently to the Maven repository
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742628#action_12742628 ] 

Doug Cutting commented on HADOOP-5107:
--------------------------------------

I'd be curious to hear what the problems are, and what it means to publish using Ivy.  I publish Avro's jar with scp.

> split the core, hdfs, and mapred jars from each other and publish them independently to the Maven repository
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5107
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5107
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.0
>            Reporter: Owen O'Malley
>            Assignee: Giridharan Kesavan
>         Attachments: common-trunk.patch, hdfs-trunk.patch, mapreduce-trunk.patch
>
>
> I think to support splitting the projects, we should publish the jars for 0.20.0 as independent jars to the Maven repository 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-760-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 13 02:47:30 2009
Return-Path: <common-issues-return-760-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17776 invoked from network); 13 Aug 2009 02:47:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Aug 2009 02:47:30 -0000
Received: (qmail 29784 invoked by uid 500); 13 Aug 2009 02:47:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29744 invoked by uid 500); 13 Aug 2009 02:47:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29730 invoked by uid 99); 13 Aug 2009 02:47:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 02:47:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 02:47:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D219C234C004
	for <common-issues@hadoop.apache.org>; Wed, 12 Aug 2009 19:47:14 -0700 (PDT)
Message-ID: <1440306602.1250131634846.JavaMail.jira@brutus>
Date: Wed, 12 Aug 2009 19:47:14 -0700 (PDT)
From: "Sreekanth Ramakrishnan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6105) Provide a way to automatically
 handle backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742682#action_12742682 ] 

Sreekanth Ramakrishnan commented on HADOOP-6105:
------------------------------------------------

Took a look at the lastest patch, the last test condition i.e. simulation of the user setting deprecated value and framework using new value is not being asserted, can you please add the assert condition there? 

Apart from that all other changes look fine.


> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>         Attachments: HADOOP-6105-1.patch, HADOOP-6105-2.patch, HADOOP-6105-3.patch, HADOOP-6105.patch, HADOOP-6105.patch
>
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-761-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 13 03:27:33 2009
Return-Path: <common-issues-return-761-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25531 invoked from network); 13 Aug 2009 03:27:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Aug 2009 03:27:32 -0000
Received: (qmail 58741 invoked by uid 500); 13 Aug 2009 03:27:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58621 invoked by uid 500); 13 Aug 2009 03:27:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58592 invoked by uid 99); 13 Aug 2009 03:27:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 03:27:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 03:27:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EFA1929A0015
	for <common-issues@hadoop.apache.org>; Wed, 12 Aug 2009 20:27:14 -0700 (PDT)
Message-ID: <1884980595.1250134034980.JavaMail.jira@brutus>
Date: Wed, 12 Aug 2009 20:27:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5107) split the core, hdfs, and mapred
 jars from each other and publish them independently to the Maven repository
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742694#action_12742694 ] 

Giridharan Kesavan commented on HADOOP-5107:
--------------------------------------------

I was able to use ivy to publish ivy.xml and hadoop jars to a local filesystem using filesystem resolver ,scp resolver to people.apache.org:/home/<myhome>

And everyone had concerns about publishing on to the home folder including me using ivy scp resolver.

I tried using the respository.apache.org maven repository (nexus) for publishing ivy artifacts (ivy.xml and hadoop.jar) I coudnt do the publishing even to the snapshot repository. I get the forbidden error.(I verified the authentication as my userid has access to the maven repo.)

When I tried doing the same publishing to a local nexus instance I was able to publish it. 
After discussions with Brian (nexus repo admin), it looks like apache maven repo follows maven standards and he is not sure of publishing ivy.xml files to this repo.

Now I'm trying out with maven ant task.


 

> split the core, hdfs, and mapred jars from each other and publish them independently to the Maven repository
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5107
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5107
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.20.0
>            Reporter: Owen O'Malley
>            Assignee: Giridharan Kesavan
>         Attachments: common-trunk.patch, hdfs-trunk.patch, mapreduce-trunk.patch
>
>
> I think to support splitting the projects, we should publish the jars for 0.20.0 as independent jars to the Maven repository 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-762-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 13 03:34:30 2009
Return-Path: <common-issues-return-762-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27105 invoked from network); 13 Aug 2009 03:34:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Aug 2009 03:34:30 -0000
Received: (qmail 60897 invoked by uid 500); 13 Aug 2009 03:34:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60840 invoked by uid 500); 13 Aug 2009 03:34:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60829 invoked by uid 99); 13 Aug 2009 03:34:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 03:34:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 03:34:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D1627234C004
	for <common-issues@hadoop.apache.org>; Wed, 12 Aug 2009 20:34:14 -0700 (PDT)
Message-ID: <874821605.1250134454843.JavaMail.jira@brutus>
Date: Wed, 12 Aug 2009 20:34:14 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6105) Provide a way to automatically handle
 backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna updated HADOOP-6105:
------------------------------------------

    Attachment: HADOOP-6105-4.patch

Uploading the patch with the above mentioned correction done.

> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>         Attachments: HADOOP-6105-1.patch, HADOOP-6105-2.patch, HADOOP-6105-3.patch, HADOOP-6105-4.patch, HADOOP-6105.patch, HADOOP-6105.patch
>
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-763-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 13 06:28:30 2009
Return-Path: <common-issues-return-763-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76423 invoked from network); 13 Aug 2009 06:28:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Aug 2009 06:28:30 -0000
Received: (qmail 78685 invoked by uid 500); 13 Aug 2009 06:28:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78628 invoked by uid 500); 13 Aug 2009 06:28:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78618 invoked by uid 99); 13 Aug 2009 06:28:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 06:28:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 06:28:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 45E5829A001D
	for <common-issues@hadoop.apache.org>; Wed, 12 Aug 2009 23:28:15 -0700 (PDT)
Message-ID: <260752256.1250144895285.JavaMail.jira@brutus>
Date: Wed, 12 Aug 2009 23:28:15 -0700 (PDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4493) Localized files from DistributedCache
 should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V updated HADOOP-4493:
------------------------------

    Issue Type: Bug  (was: Sub-task)
        Parent:     (was: HADOOP-4487)

> Localized files from DistributedCache should have right access-control
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-4493
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4493
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: tasktracker
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-764-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 13 06:28:30 2009
Return-Path: <common-issues-return-764-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76459 invoked from network); 13 Aug 2009 06:28:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Aug 2009 06:28:30 -0000
Received: (qmail 78771 invoked by uid 500); 13 Aug 2009 06:28:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78717 invoked by uid 500); 13 Aug 2009 06:28:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78656 invoked by uid 99); 13 Aug 2009 06:28:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 06:28:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 06:28:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 363E129A0012
	for <common-issues@hadoop.apache.org>; Wed, 12 Aug 2009 23:28:15 -0700 (PDT)
Message-ID: <1059017137.1250144895221.JavaMail.jira@brutus>
Date: Wed, 12 Aug 2009 23:28:15 -0700 (PDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-4493) Localized files from
 DistributedCache should have right access-control
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V reassigned HADOOP-4493:
---------------------------------

    Assignee: Vinod K V

> Localized files from DistributedCache should have right access-control
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-4493
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4493
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: tasktracker
>            Reporter: Arun C Murthy
>            Assignee: Vinod K V
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-765-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 13 15:42:32 2009
Return-Path: <common-issues-return-765-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94046 invoked from network); 13 Aug 2009 15:42:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Aug 2009 15:42:32 -0000
Received: (qmail 69576 invoked by uid 500); 13 Aug 2009 15:42:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69534 invoked by uid 500); 13 Aug 2009 15:42:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69524 invoked by uid 99); 13 Aug 2009 15:42:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 15:42:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 15:42:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D0919234C004
	for <common-issues@hadoop.apache.org>; Thu, 13 Aug 2009 08:42:14 -0700 (PDT)
Message-ID: <2057045324.1250178134840.JavaMail.jira@brutus>
Date: Thu, 13 Aug 2009 08:42:14 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6165) Add metadata to Serializations
In-Reply-To: <1308804598.1248261914878.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6165:
------------------------------

    Attachment: HADOOP-6165-v2.patch

This patch implements 1, and 3.i (for sequence files) above but not 3.ii, 3.iii, or 5. I've changed the accept() method to just take the metadata map, and not the Class, following Doug's suggestion.

For the 0.21 release it is only necessary to make the API changes to the serialization framework. This means deprecating the Serialization/Serializer/Deserializer interfaces and introducing Base{Serialization,Serializer,Deserializer} (1).

Changes to take full advantage of the new framework in MapReduce (3.ii, 3.iii) can be introduced progressively in later JIRAs, since they will be additions and shouldn't affect backward compatibility.

> Add metadata to Serializations
> ------------------------------
>
>                 Key: HADOOP-6165
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6165
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: contrib/serialization
>            Reporter: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6165-v2.patch, HADOOP-6165.patch
>
>
> The Serialization framework only allows a class to be passed as metadata. This assumes there is a one-to-one mapping between types and Serializations, which is overly restrictive. By permitting applications to pass arbitrary metadata to Serializations, they can get more control over which Serialization is used, and would also allow, for example, one to pass an Avro schema to an Avro Serialization.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-766-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 13 15:42:32 2009
Return-Path: <common-issues-return-766-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94085 invoked from network); 13 Aug 2009 15:42:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Aug 2009 15:42:32 -0000
Received: (qmail 69677 invoked by uid 500); 13 Aug 2009 15:42:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69623 invoked by uid 500); 13 Aug 2009 15:42:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69597 invoked by uid 99); 13 Aug 2009 15:42:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 15:42:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 15:42:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1494B234C1EE
	for <common-issues@hadoop.apache.org>; Thu, 13 Aug 2009 08:42:15 -0700 (PDT)
Message-ID: <1494516801.1250178135083.JavaMail.jira@brutus>
Date: Thu, 13 Aug 2009 08:42:15 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6165) Add metadata to Serializations
In-Reply-To: <1308804598.1248261914878.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6165:
------------------------------

    Assignee: Tom White
      Status: Patch Available  (was: Open)

> Add metadata to Serializations
> ------------------------------
>
>                 Key: HADOOP-6165
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6165
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: contrib/serialization
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6165-v2.patch, HADOOP-6165.patch
>
>
> The Serialization framework only allows a class to be passed as metadata. This assumes there is a one-to-one mapping between types and Serializations, which is overly restrictive. By permitting applications to pass arbitrary metadata to Serializations, they can get more control over which Serialization is used, and would also allow, for example, one to pass an Avro schema to an Avro Serialization.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-767-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 13 16:04:32 2009
Return-Path: <common-issues-return-767-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2356 invoked from network); 13 Aug 2009 16:04:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Aug 2009 16:04:32 -0000
Received: (qmail 4528 invoked by uid 500); 13 Aug 2009 16:04:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4396 invoked by uid 500); 13 Aug 2009 16:04:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4357 invoked by uid 99); 13 Aug 2009 16:04:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 16:04:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 16:04:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id F0CE0234C1EC
	for <common-issues@hadoop.apache.org>; Thu, 13 Aug 2009 09:04:15 -0700 (PDT)
Message-ID: <1639565276.1250179455985.JavaMail.jira@brutus>
Date: Thu, 13 Aug 2009 09:04:15 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6165) Add metadata to Serializations
In-Reply-To: <1308804598.1248261914878.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742878#action_12742878 ] 

Hadoop QA commented on HADOOP-6165:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12416454/HADOOP-6165-v2.patch
  against trunk revision 803296.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The applied patch generated 169 javac compiler warnings (more than the trunk's current 116 warnings).

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/602/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/602/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/602/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/602/console

This message is automatically generated.

> Add metadata to Serializations
> ------------------------------
>
>                 Key: HADOOP-6165
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6165
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: contrib/serialization
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6165-v2.patch, HADOOP-6165.patch
>
>
> The Serialization framework only allows a class to be passed as metadata. This assumes there is a one-to-one mapping between types and Serializations, which is overly restrictive. By permitting applications to pass arbitrary metadata to Serializations, they can get more control over which Serialization is used, and would also allow, for example, one to pass an Avro schema to an Avro Serialization.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-768-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 13 19:57:33 2009
Return-Path: <common-issues-return-768-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96899 invoked from network); 13 Aug 2009 19:57:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Aug 2009 19:57:33 -0000
Received: (qmail 5339 invoked by uid 500); 13 Aug 2009 19:57:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5298 invoked by uid 500); 13 Aug 2009 19:57:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5283 invoked by uid 99); 13 Aug 2009 19:57:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 19:57:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 19:57:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4B1BA29A0014
	for <common-issues@hadoop.apache.org>; Thu, 13 Aug 2009 12:57:15 -0700 (PDT)
Message-ID: <1987057997.1250193435041.JavaMail.jira@brutus>
Date: Thu, 13 Aug 2009 12:57:15 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6165) Add metadata to Serializations
In-Reply-To: <1308804598.1248261914878.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742947#action_12742947 ] 

Doug Cutting commented on HADOOP-6165:
--------------------------------------

This looks very nice!  A few nits:
 - BaseSerialization and BaseDeserialization might be instead called SerializationBase and DeserializationBase.
 - BaseSerializationWrapper might instead be called LegacySerialization.  Similarly for BaseDeserializationWrapper.
 - Should this patch update AvroSerialization too?  In this case we could use something like SpecificRecord.class.isAssignableFrom(Class.forName(meta.get("class")), and, if that fails, use GenericDatumReader.  That way Avro data can be read whether or not a specific or reflect class is loaded.

> Add metadata to Serializations
> ------------------------------
>
>                 Key: HADOOP-6165
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6165
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: contrib/serialization
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6165-v2.patch, HADOOP-6165.patch
>
>
> The Serialization framework only allows a class to be passed as metadata. This assumes there is a one-to-one mapping between types and Serializations, which is overly restrictive. By permitting applications to pass arbitrary metadata to Serializations, they can get more control over which Serialization is used, and would also allow, for example, one to pass an Avro schema to an Avro Serialization.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-769-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 13 21:09:32 2009
Return-Path: <common-issues-return-769-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16378 invoked from network); 13 Aug 2009 21:09:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Aug 2009 21:09:32 -0000
Received: (qmail 13091 invoked by uid 500); 13 Aug 2009 21:09:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13034 invoked by uid 500); 13 Aug 2009 21:09:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13024 invoked by uid 99); 13 Aug 2009 21:09:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 21:09:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 21:09:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D639729A0014
	for <common-issues@hadoop.apache.org>; Thu, 13 Aug 2009 14:09:14 -0700 (PDT)
Message-ID: <474190989.1250197754876.JavaMail.jira@brutus>
Date: Thu, 13 Aug 2009 14:09:14 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6165) Add metadata to Serializations
In-Reply-To: <1308804598.1248261914878.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742961#action_12742961 ] 

Tom White commented on HADOOP-6165:
-----------------------------------

bq. That way Avro data can be read whether or not a specific or reflect class is loaded.

So there'd just be a single AvroSerialization?

> Add metadata to Serializations
> ------------------------------
>
>                 Key: HADOOP-6165
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6165
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: contrib/serialization
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6165-v2.patch, HADOOP-6165.patch
>
>
> The Serialization framework only allows a class to be passed as metadata. This assumes there is a one-to-one mapping between types and Serializations, which is overly restrictive. By permitting applications to pass arbitrary metadata to Serializations, they can get more control over which Serialization is used, and would also allow, for example, one to pass an Avro schema to an Avro Serialization.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-770-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 13 21:27:33 2009
Return-Path: <common-issues-return-770-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23230 invoked from network); 13 Aug 2009 21:27:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Aug 2009 21:27:30 -0000
Received: (qmail 44592 invoked by uid 500); 13 Aug 2009 21:27:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44553 invoked by uid 500); 13 Aug 2009 21:27:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44543 invoked by uid 99); 13 Aug 2009 21:27:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 21:27:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 21:27:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CAF5B234C004
	for <common-issues@hadoop.apache.org>; Thu, 13 Aug 2009 14:27:14 -0700 (PDT)
Message-ID: <1759011542.1250198834815.JavaMail.jira@brutus>
Date: Thu, 13 Aug 2009 14:27:14 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6165) Add metadata to Serializations
In-Reply-To: <1308804598.1248261914878.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742981#action_12742981 ] 

Doug Cutting commented on HADOOP-6165:
--------------------------------------

> So there'd just be a single AvroSerialization?

Perhaps.  An alternative might be to have three: reflect, specific and generic.  Each could accept records if they have the right base class.  But if you read a file that was written with, e.g., specific and don't have that class, or data written by python, that names no class, then you'd be unable to read that data.  Also, with Avro, you're not tied to records as the schema: values could be a union, a map, or an array.

If the data was written with reflect or specific, and you have the class used to write it loaded, then its probably best to use that.  But in all other cases generic is probably your best bet.  I guess this could be implemented by placing generic last on the list, so that it accepts anything that has an avro schema, with specific and reflect picking off things that have classes loaded.  Is that better?  I don't have a strong feeling.

> Add metadata to Serializations
> ------------------------------
>
>                 Key: HADOOP-6165
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6165
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: contrib/serialization
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6165-v2.patch, HADOOP-6165.patch
>
>
> The Serialization framework only allows a class to be passed as metadata. This assumes there is a one-to-one mapping between types and Serializations, which is overly restrictive. By permitting applications to pass arbitrary metadata to Serializations, they can get more control over which Serialization is used, and would also allow, for example, one to pass an Avro schema to an Avro Serialization.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-771-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 13 22:09:32 2009
Return-Path: <common-issues-return-771-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46787 invoked from network); 13 Aug 2009 22:09:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Aug 2009 22:09:32 -0000
Received: (qmail 90712 invoked by uid 500); 13 Aug 2009 22:09:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90652 invoked by uid 500); 13 Aug 2009 22:09:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90642 invoked by uid 99); 13 Aug 2009 22:09:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 22:09:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 22:09:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C9E8A234C004
	for <common-issues@hadoop.apache.org>; Thu, 13 Aug 2009 15:09:14 -0700 (PDT)
Message-ID: <1614955422.1250201354812.JavaMail.jira@brutus>
Date: Thu, 13 Aug 2009 15:09:14 -0700 (PDT)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6186) Enhance checkstyle results by
 Hudson Hadoop QA to provide a diff
In-Reply-To: <373147619.1249965854883.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742997#action_12742997 ] 

Nigel Daley commented on HADOOP-6186:
-------------------------------------

That would be great.  Unfortunately checkstyle doesn't come with any diffing tools (like findbugs does).

Another approach would be to get the warnings down to 0 through both tuning the rules and doing a mass cleanup of the source tree using an IDE.

> Enhance checkstyle results by Hudson Hadoop QA to provide a diff
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6186
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6186
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Vinod K V
>
> Currently, when a patch is submitted to Hudson, it provides the checkstyle results indicating problems that happened throughout the project history. For e.g., http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/463/artifact/trunk/build/test/checkstyle-errors.html.
> It is not useful at all in this format. It would be of better use to provide the style problems resulting ONLY OUT OF THE CURRENT PATCH.
> Another point: Should we also consider a +1, -1 by Hudson w.r.t code style ala findBugs warnings, test results etc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-772-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 13 23:32:30 2009
Return-Path: <common-issues-return-772-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72654 invoked from network); 13 Aug 2009 23:32:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Aug 2009 23:32:30 -0000
Received: (qmail 59716 invoked by uid 500); 13 Aug 2009 23:32:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59660 invoked by uid 500); 13 Aug 2009 23:32:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59650 invoked by uid 99); 13 Aug 2009 23:32:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 23:32:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 23:32:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id ED86129A0012
	for <common-issues@hadoop.apache.org>; Thu, 13 Aug 2009 16:32:14 -0700 (PDT)
Message-ID: <2147018514.1250206334972.JavaMail.jira@brutus>
Date: Thu, 13 Aug 2009 16:32:14 -0700 (PDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6192) Shell.getUlimitMemoryCommand is tied
 to Map-Reduce
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Shell.getUlimitMemoryCommand is tied to Map-Reduce
--------------------------------------------------

                 Key: HADOOP-6192
                 URL: https://issues.apache.org/jira/browse/HADOOP-6192
             Project: Hadoop Common
          Issue Type: Bug
          Components: util
            Reporter: Arun C Murthy
            Assignee: Arun C Murthy


Currently org.apache.hadoop.util.Shell.getUlimitMemoryCommand relies on a MAPREDUCE specific configuration property for the memory limit. We should break the link.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-773-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 13 23:36:30 2009
Return-Path: <common-issues-return-773-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73644 invoked from network); 13 Aug 2009 23:36:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Aug 2009 23:36:30 -0000
Received: (qmail 64972 invoked by uid 500); 13 Aug 2009 23:36:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64926 invoked by uid 500); 13 Aug 2009 23:36:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64916 invoked by uid 99); 13 Aug 2009 23:36:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 23:36:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 23:36:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C75A4234C004
	for <common-issues@hadoop.apache.org>; Thu, 13 Aug 2009 16:36:14 -0700 (PDT)
Message-ID: <1742690564.1250206574801.JavaMail.jira@brutus>
Date: Thu, 13 Aug 2009 16:36:14 -0700 (PDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6192) Shell.getUlimitMemoryCommand is tied
 to Map-Reduce
In-Reply-To: <2147018514.1250206334972.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arun C Murthy updated HADOOP-6192:
----------------------------------

    Attachment: HADOOP-6192_0_20090813.patch

Straight-forward patch. I'm adding test-cases via MAPREDUCE-478.

> Shell.getUlimitMemoryCommand is tied to Map-Reduce
> --------------------------------------------------
>
>                 Key: HADOOP-6192
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6192
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.20.0
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6192_0_20090813.patch
>
>
> Currently org.apache.hadoop.util.Shell.getUlimitMemoryCommand relies on a MAPREDUCE specific configuration property for the memory limit. We should break the link.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-774-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 13 23:36:32 2009
Return-Path: <common-issues-return-774-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73683 invoked from network); 13 Aug 2009 23:36:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Aug 2009 23:36:32 -0000
Received: (qmail 65074 invoked by uid 500); 13 Aug 2009 23:36:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65017 invoked by uid 500); 13 Aug 2009 23:36:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65007 invoked by uid 99); 13 Aug 2009 23:36:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 23:36:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 23:36:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D294529A0014
	for <common-issues@hadoop.apache.org>; Thu, 13 Aug 2009 16:36:14 -0700 (PDT)
Message-ID: <172550833.1250206574861.JavaMail.jira@brutus>
Date: Thu, 13 Aug 2009 16:36:14 -0700 (PDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6192) Shell.getUlimitMemoryCommand is tied
 to Map-Reduce
In-Reply-To: <2147018514.1250206334972.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arun C Murthy updated HADOOP-6192:
----------------------------------

        Fix Version/s: 0.21.0
    Affects Version/s: 0.20.0
               Status: Patch Available  (was: Open)

> Shell.getUlimitMemoryCommand is tied to Map-Reduce
> --------------------------------------------------
>
>                 Key: HADOOP-6192
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6192
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.20.0
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6192_0_20090813.patch
>
>
> Currently org.apache.hadoop.util.Shell.getUlimitMemoryCommand relies on a MAPREDUCE specific configuration property for the memory limit. We should break the link.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-775-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 13 23:54:32 2009
Return-Path: <common-issues-return-775-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76216 invoked from network); 13 Aug 2009 23:54:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 13 Aug 2009 23:54:32 -0000
Received: (qmail 78549 invoked by uid 500); 13 Aug 2009 23:54:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78493 invoked by uid 500); 13 Aug 2009 23:54:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78383 invoked by uid 99); 13 Aug 2009 23:54:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 23:54:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 23:54:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D18D8234C004
	for <common-issues@hadoop.apache.org>; Thu, 13 Aug 2009 16:54:14 -0700 (PDT)
Message-ID: <424512371.1250207654843.JavaMail.jira@brutus>
Date: Thu, 13 Aug 2009 16:54:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6192) Shell.getUlimitMemoryCommand is
 tied to Map-Reduce
In-Reply-To: <2147018514.1250206334972.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743025#action_12743025 ] 

Hadoop QA commented on HADOOP-6192:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12416504/HADOOP-6192_0_20090813.patch
  against trunk revision 803296.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/603/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/603/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/603/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/603/console

This message is automatically generated.

> Shell.getUlimitMemoryCommand is tied to Map-Reduce
> --------------------------------------------------
>
>                 Key: HADOOP-6192
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6192
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.20.0
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6192_0_20090813.patch
>
>
> Currently org.apache.hadoop.util.Shell.getUlimitMemoryCommand relies on a MAPREDUCE specific configuration property for the memory limit. We should break the link.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-776-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 00:02:32 2009
Return-Path: <common-issues-return-776-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78101 invoked from network); 14 Aug 2009 00:02:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 00:02:32 -0000
Received: (qmail 85055 invoked by uid 500); 14 Aug 2009 00:02:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85013 invoked by uid 500); 14 Aug 2009 00:02:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84906 invoked by uid 99); 14 Aug 2009 00:02:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 00:02:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 00:02:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D204929A0012
	for <common-issues@hadoop.apache.org>; Thu, 13 Aug 2009 17:02:14 -0700 (PDT)
Message-ID: <244811583.1250208134859.JavaMail.jira@brutus>
Date: Thu, 13 Aug 2009 17:02:14 -0700 (PDT)
From: "Adam Kramer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6193) JobTracker jobconf.jsp/job.xml should
 include <tt> or <pre> tags.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

JobTracker jobconf.jsp/job.xml should include <tt> or <pre> tags.
-----------------------------------------------------------------

                 Key: HADOOP-6193
                 URL: https://issues.apache.org/jira/browse/HADOOP-6193
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Adam Kramer


The job tracker allows me to click on a task, to view the "job file," and links me to a page that prints a name/value table of the job configuration options.

It would be lovely if this table, which currently is just a single <table>, would put <tt> tags inside the "value" field. Currently, there are several key/value pairs which are impossible to read without formatting. The formatting (newlines and whitespace) are passed through and appear in the source of this table, but are just munged together in the output (thanks, HTML). Inclusion of a <tt> tag in the name section of the table would solve this and make it pretty.

The jobconf.jsp / job.xml file makes no references to style sheets or anything, so it appears to just be a straight-up html representation of the right stuff...so I don't think it'd either be hard or counter-to-spec to do this. I also can't imagine that it would make the page look worse. :)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-777-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 01:12:30 2009
Return-Path: <common-issues-return-777-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99853 invoked from network); 14 Aug 2009 01:12:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 01:12:30 -0000
Received: (qmail 28747 invoked by uid 500); 14 Aug 2009 01:12:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28691 invoked by uid 500); 14 Aug 2009 01:12:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28680 invoked by uid 99); 14 Aug 2009 01:12:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 01:12:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 01:12:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CABC2234C004
	for <common-issues@hadoop.apache.org>; Thu, 13 Aug 2009 18:12:14 -0700 (PDT)
Message-ID: <763124955.1250212334815.JavaMail.jira@brutus>
Date: Thu, 13 Aug 2009 18:12:14 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6192) Shell.getUlimitMemoryCommand is
 tied to Map-Reduce
In-Reply-To: <2147018514.1250206334972.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743048#action_12743048 ] 

Chris Douglas commented on HADOOP-6192:
---------------------------------------

+1 (w/ testcases in MAPREDUCE-478 )

> Shell.getUlimitMemoryCommand is tied to Map-Reduce
> --------------------------------------------------
>
>                 Key: HADOOP-6192
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6192
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.20.0
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6192_0_20090813.patch
>
>
> Currently org.apache.hadoop.util.Shell.getUlimitMemoryCommand relies on a MAPREDUCE specific configuration property for the memory limit. We should break the link.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-778-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 02:20:32 2009
Return-Path: <common-issues-return-778-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23844 invoked from network); 14 Aug 2009 02:20:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 02:20:32 -0000
Received: (qmail 60398 invoked by uid 500); 14 Aug 2009 02:20:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60341 invoked by uid 500); 14 Aug 2009 02:20:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60331 invoked by uid 99); 14 Aug 2009 02:20:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 02:20:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 02:20:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CFCFE234C004
	for <common-issues@hadoop.apache.org>; Thu, 13 Aug 2009 19:20:14 -0700 (PDT)
Message-ID: <754975566.1250216414837.JavaMail.jira@brutus>
Date: Thu, 13 Aug 2009 19:20:14 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6166:
--------------------------------

    Attachment: graph.r
                Rplots.pdf

Attaching PDF of the concurrency benchmark from HADOOP-5318 run on all of these implementations. The first page also includes java.util.zip.CRC32 to verify that the benchmark script really is testing something and that we've fixed the problem originally reported in that ticket. The second page of the PDF is the same graph after dropping the super-slow one.

I've normalized all of the results as percentages of the very fastest. It looks like they're all within 5-10% of each other for this benchmark, and with the error bars it's pretty obvious that there is no clear winner for the 1-byte-write case at any concurrency level. If anything Crc32_6_6 seems to come out a little ahead, but really barely discernible.

I'd upload the script I used to run the benchmarks, but I had to add a silly little patch to Hadoop to be able to switch between implementations easily, so it's a hassle to run.

For reference, this system is a Nehalem box running 64-bit JDK 1.6.0u14, 8 cores w/ hyperthreading (16 logical).

> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt, c6166_20090727.patch, c6166_20090728.patch, c6166_20090810.patch, c6166_20090811.patch, graph.r, Rplots.pdf
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-779-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 02:30:32 2009
Return-Path: <common-issues-return-779-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26498 invoked from network); 14 Aug 2009 02:30:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 02:30:32 -0000
Received: (qmail 64053 invoked by uid 500); 14 Aug 2009 02:30:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63995 invoked by uid 500); 14 Aug 2009 02:30:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63985 invoked by uid 99); 14 Aug 2009 02:30:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 02:30:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 02:30:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D9010234C004
	for <common-issues@hadoop.apache.org>; Thu, 13 Aug 2009 19:30:14 -0700 (PDT)
Message-ID: <1755389735.1250217014874.JavaMail.jira@brutus>
Date: Thu, 13 Aug 2009 19:30:14 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6166:
--------------------------------

    Attachment: Rplots.pdf
                graph.r

Oops, excuse me - off-by-one error in my R script for matching up colors to datasets. Same results, except the one that appears to do the best is 8_8b

> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt, c6166_20090727.patch, c6166_20090728.patch, c6166_20090810.patch, c6166_20090811.patch, graph.r, graph.r, Rplots.pdf, Rplots.pdf
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-780-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 05:55:31 2009
Return-Path: <common-issues-return-780-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78348 invoked from network); 14 Aug 2009 05:55:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 05:55:31 -0000
Received: (qmail 90228 invoked by uid 500); 14 Aug 2009 05:55:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90175 invoked by uid 500); 14 Aug 2009 05:55:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90165 invoked by uid 99); 14 Aug 2009 05:55:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 05:55:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 05:55:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 55D2B234C004
	for <common-issues@hadoop.apache.org>; Thu, 13 Aug 2009 22:55:15 -0700 (PDT)
Message-ID: <1573387431.1250229315337.JavaMail.jira@brutus>
Date: Thu, 13 Aug 2009 22:55:15 -0700 (PDT)
From: "Nigel Daley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6182) Adding Apache License Headers and
 reduce releaseaudit warnings to zero
In-Reply-To: <739323326.1249883774947.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nigel Daley updated HADOOP-6182:
--------------------------------

    Hadoop Flags: [Reviewed]

+1 code review.

> Adding Apache License Headers and reduce releaseaudit warnings to zero
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6182
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6182
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>         Attachments: Hadoop-6182.patch
>
>
> As of now rats tool shows 111 RA warnings 
> [rat:report] Summary
> [rat:report] -------
> [rat:report] Notes: 18
> [rat:report] Binaries: 118
> [rat:report] Archives: 33
> [rat:report] Standards: 942
> [rat:report] 
> [rat:report] Apache Licensed: 820
> [rat:report] Generated Documents: 11
> [rat:report] 
> [rat:report] JavaDocs are generated and so license header is optional
> [rat:report] Generated files do not required license headers
> [rat:report] 
> [rat:report] 111 Unknown Licenses
> [rat:report] 
> [rat:report] *******************************

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-781-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 08:07:32 2009
Return-Path: <common-issues-return-781-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32344 invoked from network); 14 Aug 2009 08:07:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 08:07:32 -0000
Received: (qmail 14695 invoked by uid 500); 14 Aug 2009 08:07:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14637 invoked by uid 500); 14 Aug 2009 08:07:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14626 invoked by uid 99); 14 Aug 2009 08:07:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 08:07:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 08:07:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C9685234C004
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 01:07:14 -0700 (PDT)
Message-ID: <1049523023.1250237234810.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 01:07:14 -0700 (PDT)
From: "Adam Kramer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6193) JobTracker jobconf.jsp/job.xml
 should include <tt> or <pre> tags.
In-Reply-To: <244811583.1250208134859.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743126#action_12743126 ] 

Adam Kramer commented on HADOOP-6193:
-------------------------------------

Note that this can't just be done in the hive.query.string text as the jobtracker rewrites it; see https://issues.apache.org/jira/browse/HIVE-754

> JobTracker jobconf.jsp/job.xml should include <tt> or <pre> tags.
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6193
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6193
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Adam Kramer
>
> The job tracker allows me to click on a task, to view the "job file," and links me to a page that prints a name/value table of the job configuration options.
> It would be lovely if this table, which currently is just a single <table>, would put <tt> tags inside the "value" field. Currently, there are several key/value pairs which are impossible to read without formatting. The formatting (newlines and whitespace) are passed through and appear in the source of this table, but are just munged together in the output (thanks, HTML). Inclusion of a <tt> tag in the name section of the table would solve this and make it pretty.
> The jobconf.jsp / job.xml file makes no references to style sheets or anything, so it appears to just be a straight-up html representation of the right stuff...so I don't think it'd either be hard or counter-to-spec to do this. I also can't imagine that it would make the page look worse. :)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-782-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 08:17:32 2009
Return-Path: <common-issues-return-782-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34510 invoked from network); 14 Aug 2009 08:17:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 08:17:32 -0000
Received: (qmail 30214 invoked by uid 500); 14 Aug 2009 08:17:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30148 invoked by uid 500); 14 Aug 2009 08:17:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30070 invoked by uid 99); 14 Aug 2009 08:17:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 08:17:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 08:17:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DDA6D234C052
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 01:17:14 -0700 (PDT)
Message-ID: <763401939.1250237834907.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 01:17:14 -0700 (PDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6192) Shell.getUlimitMemoryCommand is tied
 to Map-Reduce
In-Reply-To: <2147018514.1250206334972.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arun C Murthy updated HADOOP-6192:
----------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I just committed this.

> Shell.getUlimitMemoryCommand is tied to Map-Reduce
> --------------------------------------------------
>
>                 Key: HADOOP-6192
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6192
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.20.0
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6192_0_20090813.patch
>
>
> Currently org.apache.hadoop.util.Shell.getUlimitMemoryCommand relies on a MAPREDUCE specific configuration property for the memory limit. We should break the link.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-783-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 10:25:30 2009
Return-Path: <common-issues-return-783-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90228 invoked from network); 14 Aug 2009 10:25:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 10:25:30 -0000
Received: (qmail 30263 invoked by uid 500); 14 Aug 2009 10:25:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30201 invoked by uid 500); 14 Aug 2009 10:25:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30190 invoked by uid 99); 14 Aug 2009 10:25:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 10:25:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 10:25:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D8FE3234C004
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 03:25:14 -0700 (PDT)
Message-ID: <1385130411.1250245514876.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 03:25:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6182) Adding Apache License Headers and
 reduce releaseaudit warnings to zero
In-Reply-To: <739323326.1249883774947.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6182:
---------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.21.0
           Status: Resolved  (was: Patch Available)

Thanks Nigel, I just committed this!


> Adding Apache License Headers and reduce releaseaudit warnings to zero
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6182
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6182
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-6182.patch
>
>
> As of now rats tool shows 111 RA warnings 
> [rat:report] Summary
> [rat:report] -------
> [rat:report] Notes: 18
> [rat:report] Binaries: 118
> [rat:report] Archives: 33
> [rat:report] Standards: 942
> [rat:report] 
> [rat:report] Apache Licensed: 820
> [rat:report] Generated Documents: 11
> [rat:report] 
> [rat:report] JavaDocs are generated and so license header is optional
> [rat:report] Generated files do not required license headers
> [rat:report] 
> [rat:report] 111 Unknown Licenses
> [rat:report] 
> [rat:report] *******************************

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-784-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 13:11:32 2009
Return-Path: <common-issues-return-784-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50362 invoked from network); 14 Aug 2009 13:11:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 13:11:31 -0000
Received: (qmail 34250 invoked by uid 500); 14 Aug 2009 13:11:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34173 invoked by uid 500); 14 Aug 2009 13:11:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34149 invoked by uid 99); 14 Aug 2009 13:11:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 13:11:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 13:11:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D35D8234C045
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 06:11:14 -0700 (PDT)
Message-ID: <165334509.1250255474864.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 06:11:14 -0700 (PDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6194) Add service base class and tests to
 hadoop-common/util
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Add service base class and tests to hadoop-common/util
------------------------------------------------------

                 Key: HADOOP-6194
                 URL: https://issues.apache.org/jira/browse/HADOOP-6194
             Project: Hadoop Common
          Issue Type: New Feature
          Components: util
    Affects Versions: 0.20.1
            Reporter: Steve Loughran
            Assignee: Steve Loughran


For a service base class, we need to add to common
# The Service class
# A mock Service class (it's not in test, its in src/, because it helps in functional tests downstream)
# Unit tests

This is the issue to cover this work

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-785-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 13:11:32 2009
Return-Path: <common-issues-return-785-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50413 invoked from network); 14 Aug 2009 13:11:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 13:11:32 -0000
Received: (qmail 35066 invoked by uid 500); 14 Aug 2009 13:11:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35031 invoked by uid 500); 14 Aug 2009 13:11:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35021 invoked by uid 99); 14 Aug 2009 13:11:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 13:11:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 13:11:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DBF8B234C052
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 06:11:14 -0700 (PDT)
Message-ID: <782432904.1250255474900.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 06:11:14 -0700 (PDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Work started: (HADOOP-6194) Add service base class and tests
 to hadoop-common/util
In-Reply-To: <165334509.1250255474864.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Work on HADOOP-6194 started by Steve Loughran.

> Add service base class and tests to hadoop-common/util
> ------------------------------------------------------
>
>                 Key: HADOOP-6194
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6194
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: util
>    Affects Versions: 0.20.1
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>
> For a service base class, we need to add to common
> # The Service class
> # A mock Service class (it's not in test, its in src/, because it helps in functional tests downstream)
> # Unit tests
> This is the issue to cover this work

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-786-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 13:37:31 2009
Return-Path: <common-issues-return-786-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65100 invoked from network); 14 Aug 2009 13:37:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 13:37:30 -0000
Received: (qmail 91525 invoked by uid 500); 14 Aug 2009 13:37:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91449 invoked by uid 500); 14 Aug 2009 13:37:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91384 invoked by uid 99); 14 Aug 2009 13:37:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 13:37:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 13:37:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EC352234C1E7
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 06:37:14 -0700 (PDT)
Message-ID: <1587452348.1250257034966.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 06:37:14 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6182) Adding Apache License Headers and
 reduce releaseaudit warnings to zero
In-Reply-To: <739323326.1249883774947.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743198#action_12743198 ] 

Hudson commented on HADOOP-6182:
--------------------------------

Integrated in Hadoop-Hdfs-trunk #49 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/49/])
    . Fix releaseaudit warnings by adding AL Headers.


> Adding Apache License Headers and reduce releaseaudit warnings to zero
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6182
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6182
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-6182.patch
>
>
> As of now rats tool shows 111 RA warnings 
> [rat:report] Summary
> [rat:report] -------
> [rat:report] Notes: 18
> [rat:report] Binaries: 118
> [rat:report] Archives: 33
> [rat:report] Standards: 942
> [rat:report] 
> [rat:report] Apache Licensed: 820
> [rat:report] Generated Documents: 11
> [rat:report] 
> [rat:report] JavaDocs are generated and so license header is optional
> [rat:report] Generated files do not required license headers
> [rat:report] 
> [rat:report] 111 Unknown Licenses
> [rat:report] 
> [rat:report] *******************************

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-787-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 13:37:32 2009
Return-Path: <common-issues-return-787-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65208 invoked from network); 14 Aug 2009 13:37:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 13:37:32 -0000
Received: (qmail 91622 invoked by uid 500); 14 Aug 2009 13:37:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91568 invoked by uid 500); 14 Aug 2009 13:37:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91554 invoked by uid 99); 14 Aug 2009 13:37:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 13:37:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 13:37:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E477E234C055
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 06:37:14 -0700 (PDT)
Message-ID: <1218377521.1250257034934.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 06:37:14 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6192) Shell.getUlimitMemoryCommand is
 tied to Map-Reduce
In-Reply-To: <2147018514.1250206334972.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743197#action_12743197 ] 

Hudson commented on HADOOP-6192:
--------------------------------

Integrated in Hadoop-Hdfs-trunk #49 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/49/])
    . Fix Shell.getUlimitMemoryCommand to not rely on Map-Reduce specific configs.


> Shell.getUlimitMemoryCommand is tied to Map-Reduce
> --------------------------------------------------
>
>                 Key: HADOOP-6192
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6192
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.20.0
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6192_0_20090813.patch
>
>
> Currently org.apache.hadoop.util.Shell.getUlimitMemoryCommand relies on a MAPREDUCE specific configuration property for the memory limit. We should break the link.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-788-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 15:20:30 2009
Return-Path: <common-issues-return-788-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8747 invoked from network); 14 Aug 2009 15:20:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 15:20:30 -0000
Received: (qmail 62155 invoked by uid 500); 14 Aug 2009 15:20:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62105 invoked by uid 500); 14 Aug 2009 15:20:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62090 invoked by uid 99); 14 Aug 2009 15:20:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 15:20:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 15:20:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CD54E234C004
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 08:20:14 -0700 (PDT)
Message-ID: <135328610.1250263214826.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 08:20:14 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6165) Add metadata to Serializations
In-Reply-To: <1308804598.1248261914878.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6165:
------------------------------

    Attachment: HADOOP-6165-v3.patch

Here's a new patch which adopts the names suggested by Doug. I've also added AvroGenericSerialization which looks for a schema in the metadata, and a test for it. I haven't added the fallback capability discussed, but it shouldn't be too hard to add.

I've also fixing failing tests, and reduced the number of deprecation warnings - I can't get rid of all of them until the deprecated interfaces are removed.

> Add metadata to Serializations
> ------------------------------
>
>                 Key: HADOOP-6165
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6165
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: contrib/serialization
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6165-v2.patch, HADOOP-6165-v3.patch, HADOOP-6165.patch
>
>
> The Serialization framework only allows a class to be passed as metadata. This assumes there is a one-to-one mapping between types and Serializations, which is overly restrictive. By permitting applications to pass arbitrary metadata to Serializations, they can get more control over which Serialization is used, and would also allow, for example, one to pass an Avro schema to an Avro Serialization.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-789-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 15:22:30 2009
Return-Path: <common-issues-return-789-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9111 invoked from network); 14 Aug 2009 15:22:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 15:22:30 -0000
Received: (qmail 63273 invoked by uid 500); 14 Aug 2009 15:22:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63218 invoked by uid 500); 14 Aug 2009 15:22:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63204 invoked by uid 99); 14 Aug 2009 15:22:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 15:22:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 15:22:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3DABA234C051
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 08:22:15 -0700 (PDT)
Message-ID: <192641375.1250263335251.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 08:22:15 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6165) Add metadata to Serializations
In-Reply-To: <1308804598.1248261914878.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6165:
------------------------------

    Status: Open  (was: Patch Available)

> Add metadata to Serializations
> ------------------------------
>
>                 Key: HADOOP-6165
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6165
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: contrib/serialization
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6165-v2.patch, HADOOP-6165-v3.patch, HADOOP-6165.patch
>
>
> The Serialization framework only allows a class to be passed as metadata. This assumes there is a one-to-one mapping between types and Serializations, which is overly restrictive. By permitting applications to pass arbitrary metadata to Serializations, they can get more control over which Serialization is used, and would also allow, for example, one to pass an Avro schema to an Avro Serialization.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-790-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 15:22:34 2009
Return-Path: <common-issues-return-790-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9184 invoked from network); 14 Aug 2009 15:22:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 15:22:34 -0000
Received: (qmail 63410 invoked by uid 500); 14 Aug 2009 15:22:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63334 invoked by uid 500); 14 Aug 2009 15:22:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63309 invoked by uid 99); 14 Aug 2009 15:22:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 15:22:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 15:22:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 5176D234C1F0
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 08:22:15 -0700 (PDT)
Message-ID: <1167484240.1250263335332.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 08:22:15 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6165) Add metadata to Serializations
In-Reply-To: <1308804598.1248261914878.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6165:
------------------------------

    Status: Patch Available  (was: Open)

> Add metadata to Serializations
> ------------------------------
>
>                 Key: HADOOP-6165
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6165
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: contrib/serialization
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6165-v2.patch, HADOOP-6165-v3.patch, HADOOP-6165.patch
>
>
> The Serialization framework only allows a class to be passed as metadata. This assumes there is a one-to-one mapping between types and Serializations, which is overly restrictive. By permitting applications to pass arbitrary metadata to Serializations, they can get more control over which Serialization is used, and would also allow, for example, one to pass an Avro schema to an Avro Serialization.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-791-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 15:31:32 2009
Return-Path: <common-issues-return-791-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13882 invoked from network); 14 Aug 2009 15:31:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 15:31:32 -0000
Received: (qmail 87725 invoked by uid 500); 14 Aug 2009 15:31:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87689 invoked by uid 500); 14 Aug 2009 15:31:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87679 invoked by uid 99); 14 Aug 2009 15:31:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 15:31:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 15:31:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EC854234C052
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 08:31:14 -0700 (PDT)
Message-ID: <1561183249.1250263874967.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 08:31:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6195) checkstyle target fails common trunk
 build.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

checkstyle target fails common trunk build.
-------------------------------------------

                 Key: HADOOP-6195
                 URL: https://issues.apache.org/jira/browse/HADOOP-6195
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Giridharan Kesavan




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-792-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 15:37:34 2009
Return-Path: <common-issues-return-792-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16191 invoked from network); 14 Aug 2009 15:37:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 15:37:34 -0000
Received: (qmail 1770 invoked by uid 500); 14 Aug 2009 15:37:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1727 invoked by uid 500); 14 Aug 2009 15:37:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1693 invoked by uid 99); 14 Aug 2009 15:37:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 15:37:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 15:37:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DDC77234C004
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 08:37:14 -0700 (PDT)
Message-ID: <601478744.1250264234893.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 08:37:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6195) checkstyle target fails common trunk
 build.
In-Reply-To: <1561183249.1250263874967.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6195:
---------------------------------------

    Attachment: Hadoop-6195.patch

this patch fixes the trunk build failure.

> checkstyle target fails common trunk build.
> -------------------------------------------
>
>                 Key: HADOOP-6195
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6195
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Giridharan Kesavan
>         Attachments: Hadoop-6195.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-793-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 15:45:30 2009
Return-Path: <common-issues-return-793-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19225 invoked from network); 14 Aug 2009 15:45:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 15:45:30 -0000
Received: (qmail 20866 invoked by uid 500); 14 Aug 2009 15:45:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20825 invoked by uid 500); 14 Aug 2009 15:45:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20815 invoked by uid 99); 14 Aug 2009 15:45:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 15:45:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 15:45:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C10D0234C004
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 08:45:14 -0700 (PDT)
Message-ID: <1082783457.1250264714786.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 08:45:14 -0700 (PDT)
From: "Lee Tucker (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6195) checkstyle target fails common trunk
 build.
In-Reply-To: <1561183249.1250263874967.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Lee Tucker updated HADOOP-6195:
-------------------------------

    Attachment: HADOOP-6195-v2.patch

instead of deleting <?xml version="1.0"?>, move the comment

> checkstyle target fails common trunk build.
> -------------------------------------------
>
>                 Key: HADOOP-6195
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6195
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Giridharan Kesavan
>         Attachments: HADOOP-6195-v2.patch, Hadoop-6195.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-794-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 16:21:30 2009
Return-Path: <common-issues-return-794-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31203 invoked from network); 14 Aug 2009 16:21:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 16:21:30 -0000
Received: (qmail 84448 invoked by uid 500); 14 Aug 2009 16:21:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84407 invoked by uid 500); 14 Aug 2009 16:21:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84397 invoked by uid 99); 14 Aug 2009 16:21:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 16:21:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 16:21:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0F084234C1EC
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 09:21:15 -0700 (PDT)
Message-ID: <1515298180.1250266875060.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 09:21:15 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6195) checkstyle target fails common trunk
 build.
In-Reply-To: <1561183249.1250263874967.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6195:
---------------------------------------

    Assignee: Giridharan Kesavan
      Status: Patch Available  (was: Open)

> checkstyle target fails common trunk build.
> -------------------------------------------
>
>                 Key: HADOOP-6195
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6195
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>         Attachments: HADOOP-6195-v2.patch, Hadoop-6195.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-795-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 16:25:32 2009
Return-Path: <common-issues-return-795-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33703 invoked from network); 14 Aug 2009 16:25:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 16:25:32 -0000
Received: (qmail 89397 invoked by uid 500); 14 Aug 2009 16:25:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89306 invoked by uid 500); 14 Aug 2009 16:25:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89284 invoked by uid 99); 14 Aug 2009 16:25:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 16:25:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 16:25:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id F3F11234C1F1
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 09:25:14 -0700 (PDT)
Message-ID: <530155726.1250267114998.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 09:25:14 -0700 (PDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6195) checkstyle target fails common
 trunk build.
In-Reply-To: <1561183249.1250263874967.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743274#action_12743274 ] 

Arun C Murthy commented on HADOOP-6195:
---------------------------------------

+1

> checkstyle target fails common trunk build.
> -------------------------------------------
>
>                 Key: HADOOP-6195
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6195
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>         Attachments: HADOOP-6195-v2.patch, Hadoop-6195.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-796-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 16:27:30 2009
Return-Path: <common-issues-return-796-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33959 invoked from network); 14 Aug 2009 16:27:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 16:27:30 -0000
Received: (qmail 90362 invoked by uid 500); 14 Aug 2009 16:27:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90304 invoked by uid 500); 14 Aug 2009 16:27:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90281 invoked by uid 99); 14 Aug 2009 16:27:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 16:27:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 16:27:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 04BCF234C1ED
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 09:27:15 -0700 (PDT)
Message-ID: <910160229.1250267235018.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 09:27:15 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6195) checkstyle target fails common
 trunk build.
In-Reply-To: <1561183249.1250263874967.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743276#action_12743276 ] 

Giridharan Kesavan commented on HADOOP-6195:
--------------------------------------------

tested checkstyle and releaseaudit target locally and the patch passes.

> checkstyle target fails common trunk build.
> -------------------------------------------
>
>                 Key: HADOOP-6195
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6195
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>         Attachments: HADOOP-6195-v2.patch, Hadoop-6195.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-797-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 16:47:32 2009
Return-Path: <common-issues-return-797-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40002 invoked from network); 14 Aug 2009 16:47:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 16:47:32 -0000
Received: (qmail 13911 invoked by uid 500); 14 Aug 2009 16:47:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13865 invoked by uid 500); 14 Aug 2009 16:47:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13800 invoked by uid 99); 14 Aug 2009 16:47:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 16:47:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 16:47:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DDC48234C051
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 09:47:14 -0700 (PDT)
Message-ID: <1288652300.1250268434907.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 09:47:14 -0700 (PDT)
From: "Lee Tucker (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6195) checkstyle target fails common
 trunk build.
In-Reply-To: <1561183249.1250263874967.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743289#action_12743289 ] 

Lee Tucker commented on HADOOP-6195:
------------------------------------

     [exec] +1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.


I'm not sure how it found modified unit tests...   But hey, the intelligence of scripts is inscrutable.

> checkstyle target fails common trunk build.
> -------------------------------------------
>
>                 Key: HADOOP-6195
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6195
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>         Attachments: HADOOP-6195-v2.patch, Hadoop-6195.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-798-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 17:01:32 2009
Return-Path: <common-issues-return-798-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46735 invoked from network); 14 Aug 2009 17:01:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 17:01:32 -0000
Received: (qmail 49077 invoked by uid 500); 14 Aug 2009 17:01:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49024 invoked by uid 500); 14 Aug 2009 17:01:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49009 invoked by uid 99); 14 Aug 2009 17:01:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 17:01:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 17:01:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 6866D234C1E6
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 10:01:15 -0700 (PDT)
Message-ID: <1491711419.1250269275426.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 10:01:15 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6195) checkstyle target fails common trunk
 build.
In-Reply-To: <1561183249.1250263874967.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6195:
---------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.21.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I just committed this, thanks Lee!

> checkstyle target fails common trunk build.
> -------------------------------------------
>
>                 Key: HADOOP-6195
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6195
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6195-v2.patch, Hadoop-6195.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-799-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 17:09:32 2009
Return-Path: <common-issues-return-799-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49216 invoked from network); 14 Aug 2009 17:09:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 17:09:32 -0000
Received: (qmail 59378 invoked by uid 500); 14 Aug 2009 17:09:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59335 invoked by uid 500); 14 Aug 2009 17:09:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59325 invoked by uid 99); 14 Aug 2009 17:09:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 17:09:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 17:09:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CAC5E234C004
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 10:09:14 -0700 (PDT)
Message-ID: <2022429671.1250269754816.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 10:09:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6195) checkstyle target fails common trunk
 build.
In-Reply-To: <1561183249.1250263874967.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6195:
-------------------------------------------

    Assignee: Lee Tucker  (was: Giridharan Kesavan)

>I'm not sure how it found modified unit tests...
It is because the patch changed src/test/checkstyle.xml, a file under src/test.  That why it said "The patch *appears* to include 3 new or modified tests."  I could not explain how it came up the number of modified tests.

> checkstyle target fails common trunk build.
> -------------------------------------------
>
>                 Key: HADOOP-6195
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6195
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Giridharan Kesavan
>            Assignee: Lee Tucker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6195-v2.patch, Hadoop-6195.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-800-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 18:25:31 2009
Return-Path: <common-issues-return-800-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92108 invoked from network); 14 Aug 2009 18:25:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 18:25:30 -0000
Received: (qmail 52440 invoked by uid 500); 14 Aug 2009 18:25:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52404 invoked by uid 500); 14 Aug 2009 18:25:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52394 invoked by uid 99); 14 Aug 2009 18:25:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 18:25:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 18:25:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E7A84234C1EB
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 11:25:14 -0700 (PDT)
Message-ID: <442660505.1250274314948.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 11:25:14 -0700 (PDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Reopened: (HADOOP-6195) checkstyle target fails common trunk
 build.
In-Reply-To: <1561183249.1250263874967.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arun C Murthy reopened HADOOP-6195:
-----------------------------------

      Assignee:     (was: Lee Tucker)

Re-opening to mark this as INVALID since HADOOP-6182 has been reverted.

> checkstyle target fails common trunk build.
> -------------------------------------------
>
>                 Key: HADOOP-6195
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6195
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Giridharan Kesavan
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6195-v2.patch, Hadoop-6195.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-801-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 18:25:31 2009
Return-Path: <common-issues-return-801-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92167 invoked from network); 14 Aug 2009 18:25:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 18:25:31 -0000
Received: (qmail 52818 invoked by uid 500); 14 Aug 2009 18:25:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52737 invoked by uid 500); 14 Aug 2009 18:25:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52559 invoked by uid 99); 14 Aug 2009 18:25:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 18:25:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 18:25:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E1EB9234C052
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 11:25:14 -0700 (PDT)
Message-ID: <498864308.1250274314924.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 11:25:14 -0700 (PDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Reopened: (HADOOP-6182) Adding Apache License Headers and
 reduce releaseaudit warnings to zero
In-Reply-To: <739323326.1249883774947.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arun C Murthy reopened HADOOP-6182:
-----------------------------------


I've had to revert this patch, it causes all unit-tests to fail since some of the xml config files are invalidated by the presence of the license headers (e.g. HADOOP-6195).

This patch shouldn't have been committed since hudson did catch the problem...

> Adding Apache License Headers and reduce releaseaudit warnings to zero
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6182
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6182
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-6182.patch
>
>
> As of now rats tool shows 111 RA warnings 
> [rat:report] Summary
> [rat:report] -------
> [rat:report] Notes: 18
> [rat:report] Binaries: 118
> [rat:report] Archives: 33
> [rat:report] Standards: 942
> [rat:report] 
> [rat:report] Apache Licensed: 820
> [rat:report] Generated Documents: 11
> [rat:report] 
> [rat:report] JavaDocs are generated and so license header is optional
> [rat:report] Generated files do not required license headers
> [rat:report] 
> [rat:report] 111 Unknown Licenses
> [rat:report] 
> [rat:report] *******************************

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-802-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 18:27:34 2009
Return-Path: <common-issues-return-802-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93185 invoked from network); 14 Aug 2009 18:27:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 18:27:34 -0000
Received: (qmail 56224 invoked by uid 500); 14 Aug 2009 18:27:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56160 invoked by uid 500); 14 Aug 2009 18:27:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55871 invoked by uid 99); 14 Aug 2009 18:27:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 18:27:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 18:27:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D281F234C045
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 11:27:14 -0700 (PDT)
Message-ID: <1785718647.1250274434861.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 11:27:14 -0700 (PDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6195) checkstyle target fails common trunk
 build.
In-Reply-To: <1561183249.1250263874967.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arun C Murthy resolved HADOOP-6195.
-----------------------------------

       Resolution: Invalid
    Fix Version/s:     (was: 0.21.0)

> checkstyle target fails common trunk build.
> -------------------------------------------
>
>                 Key: HADOOP-6195
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6195
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Giridharan Kesavan
>         Attachments: HADOOP-6195-v2.patch, Hadoop-6195.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-803-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 18:31:31 2009
Return-Path: <common-issues-return-803-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98901 invoked from network); 14 Aug 2009 18:31:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 18:31:31 -0000
Received: (qmail 63071 invoked by uid 500); 14 Aug 2009 18:31:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63012 invoked by uid 500); 14 Aug 2009 18:31:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62793 invoked by uid 99); 14 Aug 2009 18:31:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 18:31:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 18:31:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DD90E234C1E6
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 11:31:14 -0700 (PDT)
Message-ID: <851564939.1250274674906.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 11:31:14 -0700 (PDT)
From: "Lee Tucker (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6182) Adding Apache License Headers and
 reduce releaseaudit warnings to zero
In-Reply-To: <739323326.1249883774947.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743346#action_12743346 ] 

Lee Tucker commented on HADOOP-6182:
------------------------------------

Looks like the original patch inserted text blocks into the XML files ahead of the block:

 <?xml version="1.0"?>
 <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>


I corrected this for checkstyle in HADOOP-6185, but because the error is pervasive, it fails a wide variety of unit tests.

> Adding Apache License Headers and reduce releaseaudit warnings to zero
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6182
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6182
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-6182.patch
>
>
> As of now rats tool shows 111 RA warnings 
> [rat:report] Summary
> [rat:report] -------
> [rat:report] Notes: 18
> [rat:report] Binaries: 118
> [rat:report] Archives: 33
> [rat:report] Standards: 942
> [rat:report] 
> [rat:report] Apache Licensed: 820
> [rat:report] Generated Documents: 11
> [rat:report] 
> [rat:report] JavaDocs are generated and so license header is optional
> [rat:report] Generated files do not required license headers
> [rat:report] 
> [rat:report] 111 Unknown Licenses
> [rat:report] 
> [rat:report] *******************************

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-804-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 18:35:31 2009
Return-Path: <common-issues-return-804-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1603 invoked from network); 14 Aug 2009 18:35:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 18:35:30 -0000
Received: (qmail 70609 invoked by uid 500); 14 Aug 2009 18:35:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70572 invoked by uid 500); 14 Aug 2009 18:35:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70502 invoked by uid 99); 14 Aug 2009 18:35:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 18:35:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 18:35:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 64909234C1E7
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 11:35:15 -0700 (PDT)
Message-ID: <1094056240.1250274915410.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 11:35:15 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6182) Adding Apache License Headers and
 reduce releaseaudit warnings to zero
In-Reply-To: <739323326.1249883774947.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743351#action_12743351 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6182:
------------------------------------------------

> Looks like LengthFileChecksum was removed in HADOOP-3981, although the source file wasn't deleted. ...
Let's remove the file separately once the build is stable.

> Adding Apache License Headers and reduce releaseaudit warnings to zero
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6182
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6182
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-6182.patch
>
>
> As of now rats tool shows 111 RA warnings 
> [rat:report] Summary
> [rat:report] -------
> [rat:report] Notes: 18
> [rat:report] Binaries: 118
> [rat:report] Archives: 33
> [rat:report] Standards: 942
> [rat:report] 
> [rat:report] Apache Licensed: 820
> [rat:report] Generated Documents: 11
> [rat:report] 
> [rat:report] JavaDocs are generated and so license header is optional
> [rat:report] Generated files do not required license headers
> [rat:report] 
> [rat:report] 111 Unknown Licenses
> [rat:report] 
> [rat:report] *******************************

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-805-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 18:41:31 2009
Return-Path: <common-issues-return-805-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3520 invoked from network); 14 Aug 2009 18:41:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 18:41:31 -0000
Received: (qmail 75014 invoked by uid 500); 14 Aug 2009 18:41:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74911 invoked by uid 500); 14 Aug 2009 18:41:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74887 invoked by uid 99); 14 Aug 2009 18:41:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 18:41:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 18:41:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EEBE0234C1E6
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 11:41:14 -0700 (PDT)
Message-ID: <395666500.1250275274977.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 11:41:14 -0700 (PDT)
From: "Lee Tucker (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6182) Adding Apache License Headers and
 reduce releaseaudit warnings to zero
In-Reply-To: <739323326.1249883774947.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Lee Tucker updated HADOOP-6182:
-------------------------------

    Attachment: hadoop-6182.patch

Updated the patch, and moved the XML comment blocks so that they function correctly.

> Adding Apache License Headers and reduce releaseaudit warnings to zero
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6182
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6182
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6182.patch, Hadoop-6182.patch
>
>
> As of now rats tool shows 111 RA warnings 
> [rat:report] Summary
> [rat:report] -------
> [rat:report] Notes: 18
> [rat:report] Binaries: 118
> [rat:report] Archives: 33
> [rat:report] Standards: 942
> [rat:report] 
> [rat:report] Apache Licensed: 820
> [rat:report] Generated Documents: 11
> [rat:report] 
> [rat:report] JavaDocs are generated and so license header is optional
> [rat:report] Generated files do not required license headers
> [rat:report] 
> [rat:report] 111 Unknown Licenses
> [rat:report] 
> [rat:report] *******************************

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-806-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 19:06:30 2009
Return-Path: <common-issues-return-806-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9661 invoked from network); 14 Aug 2009 19:06:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 19:06:30 -0000
Received: (qmail 12873 invoked by uid 500); 14 Aug 2009 19:06:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12822 invoked by uid 500); 14 Aug 2009 19:06:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12812 invoked by uid 99); 14 Aug 2009 19:06:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 19:06:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 19:06:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C860A234C004
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 12:06:14 -0700 (PDT)
Message-ID: <1198255582.1250276774813.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 12:06:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6165) Add metadata to Serializations
In-Reply-To: <1308804598.1248261914878.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743366#action_12743366 ] 

Hadoop QA commented on HADOOP-6165:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12416560/HADOOP-6165-v3.patch
  against trunk revision 804317.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    -1 javac.  The applied patch generated 145 javac compiler warnings (more than the trunk's current 116 warnings).

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    -1 release audit.  The applied patch generated 112 release audit warnings (more than the trunk's current 111 warnings).

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/604/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/604/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/604/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/604/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/604/console

This message is automatically generated.

> Add metadata to Serializations
> ------------------------------
>
>                 Key: HADOOP-6165
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6165
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: contrib/serialization
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6165-v2.patch, HADOOP-6165-v3.patch, HADOOP-6165.patch
>
>
> The Serialization framework only allows a class to be passed as metadata. This assumes there is a one-to-one mapping between types and Serializations, which is overly restrictive. By permitting applications to pass arbitrary metadata to Serializations, they can get more control over which Serialization is used, and would also allow, for example, one to pass an Avro schema to an Avro Serialization.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-807-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 19:28:30 2009
Return-Path: <common-issues-return-807-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16554 invoked from network); 14 Aug 2009 19:28:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 19:28:30 -0000
Received: (qmail 39011 invoked by uid 500); 14 Aug 2009 19:28:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38982 invoked by uid 500); 14 Aug 2009 19:28:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38972 invoked by uid 99); 14 Aug 2009 19:28:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 19:28:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 19:28:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DAE06234C1E6
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 12:28:14 -0700 (PDT)
Message-ID: <1680105393.1250278094895.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 12:28:14 -0700 (PDT)
From: "Lee Tucker (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6182) Adding Apache License Headers and
 reduce releaseaudit warnings to zero
In-Reply-To: <739323326.1249883774947.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Lee Tucker reassigned HADOOP-6182:
----------------------------------

    Assignee: Lee Tucker  (was: Giridharan Kesavan)

> Adding Apache License Headers and reduce releaseaudit warnings to zero
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6182
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6182
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Giridharan Kesavan
>            Assignee: Lee Tucker
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6182.patch, Hadoop-6182.patch
>
>
> As of now rats tool shows 111 RA warnings 
> [rat:report] Summary
> [rat:report] -------
> [rat:report] Notes: 18
> [rat:report] Binaries: 118
> [rat:report] Archives: 33
> [rat:report] Standards: 942
> [rat:report] 
> [rat:report] Apache Licensed: 820
> [rat:report] Generated Documents: 11
> [rat:report] 
> [rat:report] JavaDocs are generated and so license header is optional
> [rat:report] Generated files do not required license headers
> [rat:report] 
> [rat:report] 111 Unknown Licenses
> [rat:report] 
> [rat:report] *******************************

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-808-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 19:28:32 2009
Return-Path: <common-issues-return-808-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16693 invoked from network); 14 Aug 2009 19:28:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 19:28:32 -0000
Received: (qmail 39717 invoked by uid 500); 14 Aug 2009 19:28:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39647 invoked by uid 500); 14 Aug 2009 19:28:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39588 invoked by uid 99); 14 Aug 2009 19:28:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 19:28:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 19:28:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D3CA3234C045
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 12:28:14 -0700 (PDT)
Message-ID: <307094581.1250278094866.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 12:28:14 -0700 (PDT)
From: "Lee Tucker (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6182) Adding Apache License Headers and
 reduce releaseaudit warnings to zero
In-Reply-To: <739323326.1249883774947.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743380#action_12743380 ] 

Lee Tucker commented on HADOOP-6182:
------------------------------------

Ran Test-patch locally:

     [exec] 
     [exec] +1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 60 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec] 
     [exec] 
     [exec] 
     [exec] 
     [exec] ======================================================================
     [exec] ======================================================================
     [exec]     Finished build.
     [exec] ======================================================================
     [exec] ======================================================================
     [exec] 


Also ran unit tests and all passed.

> Adding Apache License Headers and reduce releaseaudit warnings to zero
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6182
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6182
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6182.patch, Hadoop-6182.patch
>
>
> As of now rats tool shows 111 RA warnings 
> [rat:report] Summary
> [rat:report] -------
> [rat:report] Notes: 18
> [rat:report] Binaries: 118
> [rat:report] Archives: 33
> [rat:report] Standards: 942
> [rat:report] 
> [rat:report] Apache Licensed: 820
> [rat:report] Generated Documents: 11
> [rat:report] 
> [rat:report] JavaDocs are generated and so license header is optional
> [rat:report] Generated files do not required license headers
> [rat:report] 
> [rat:report] 111 Unknown Licenses
> [rat:report] 
> [rat:report] *******************************

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-809-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 19:30:32 2009
Return-Path: <common-issues-return-809-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17235 invoked from network); 14 Aug 2009 19:30:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 19:30:32 -0000
Received: (qmail 44975 invoked by uid 500); 14 Aug 2009 19:30:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44921 invoked by uid 500); 14 Aug 2009 19:30:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44911 invoked by uid 99); 14 Aug 2009 19:30:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 19:30:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 19:30:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CF360234C004
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 12:30:14 -0700 (PDT)
Message-ID: <1693457340.1250278214834.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 12:30:14 -0700 (PDT)
From: "Lee Tucker (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6182) Adding Apache License Headers and
 reduce releaseaudit warnings to zero
In-Reply-To: <739323326.1249883774947.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Lee Tucker updated HADOOP-6182:
-------------------------------

    Status: Patch Available  (was: Reopened)

This adjusts the previous version of the patch to put the commented license in the appropriate location in all XML files.

> Adding Apache License Headers and reduce releaseaudit warnings to zero
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6182
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6182
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Giridharan Kesavan
>            Assignee: Lee Tucker
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6182.patch, Hadoop-6182.patch
>
>
> As of now rats tool shows 111 RA warnings 
> [rat:report] Summary
> [rat:report] -------
> [rat:report] Notes: 18
> [rat:report] Binaries: 118
> [rat:report] Archives: 33
> [rat:report] Standards: 942
> [rat:report] 
> [rat:report] Apache Licensed: 820
> [rat:report] Generated Documents: 11
> [rat:report] 
> [rat:report] JavaDocs are generated and so license header is optional
> [rat:report] Generated files do not required license headers
> [rat:report] 
> [rat:report] 111 Unknown Licenses
> [rat:report] 
> [rat:report] *******************************

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-810-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 20:10:30 2009
Return-Path: <common-issues-return-810-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29591 invoked from network); 14 Aug 2009 20:10:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 20:10:30 -0000
Received: (qmail 83399 invoked by uid 500); 14 Aug 2009 20:10:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83345 invoked by uid 500); 14 Aug 2009 20:10:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83335 invoked by uid 99); 14 Aug 2009 20:10:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 20:10:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 20:10:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DC5B9234C051
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 13:10:14 -0700 (PDT)
Message-ID: <791371542.1250280614901.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 13:10:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6182) Adding Apache License Headers and
 reduce releaseaudit warnings to zero
In-Reply-To: <739323326.1249883774947.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743406#action_12743406 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6182:
------------------------------------------------

> > Looks like LengthFileChecksum was removed in HADOOP-3981, although the source file wasn't deleted. ...
> Let's remove the file separately once the build is stable.

Trunk is passing the unit tests (see [build #604|http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/604/testReport/]).
I have removed src/core/org/apache/hadoop/fs/LengthFileChecksum.java from 0.19 and above.

> Adding Apache License Headers and reduce releaseaudit warnings to zero
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6182
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6182
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Giridharan Kesavan
>            Assignee: Lee Tucker
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6182.patch, Hadoop-6182.patch
>
>
> As of now rats tool shows 111 RA warnings 
> [rat:report] Summary
> [rat:report] -------
> [rat:report] Notes: 18
> [rat:report] Binaries: 118
> [rat:report] Archives: 33
> [rat:report] Standards: 942
> [rat:report] 
> [rat:report] Apache Licensed: 820
> [rat:report] Generated Documents: 11
> [rat:report] 
> [rat:report] JavaDocs are generated and so license header is optional
> [rat:report] Generated files do not required license headers
> [rat:report] 
> [rat:report] 111 Unknown Licenses
> [rat:report] 
> [rat:report] *******************************

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-811-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 20:21:30 2009
Return-Path: <common-issues-return-811-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34660 invoked from network); 14 Aug 2009 20:21:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 20:21:30 -0000
Received: (qmail 97094 invoked by uid 500); 14 Aug 2009 20:21:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97048 invoked by uid 500); 14 Aug 2009 20:21:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96965 invoked by uid 99); 14 Aug 2009 20:21:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 20:21:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 20:21:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E807F234C051
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 13:21:14 -0700 (PDT)
Message-ID: <232040897.1250281274949.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 13:21:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743416#action_12743416 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6166:
------------------------------------------------

Thanks, Todd.  

8_8b is also best in my benchmark ran on Windows.

Question: Crc32Base and Crc32Table are not crc implementations.  What do they mean in the graph?

> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt, c6166_20090727.patch, c6166_20090728.patch, c6166_20090810.patch, c6166_20090811.patch, graph.r, graph.r, Rplots.pdf, Rplots.pdf
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-812-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 20:29:44 2009
Return-Path: <common-issues-return-812-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37745 invoked from network); 14 Aug 2009 20:29:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 20:29:42 -0000
Received: (qmail 10015 invoked by uid 500); 14 Aug 2009 20:29:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9888 invoked by uid 500); 14 Aug 2009 20:29:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9837 invoked by uid 99); 14 Aug 2009 20:29:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 20:29:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 20:29:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EF5EC234C055
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 13:29:14 -0700 (PDT)
Message-ID: <1756326370.1250281754979.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 13:29:14 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6165) Add metadata to Serializations
In-Reply-To: <1308804598.1248261914878.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743424#action_12743424 ] 

Doug Cutting commented on HADOOP-6165:
--------------------------------------

> I haven't added the fallback capability discussed, but it shouldn't be too hard to add. 

Since AvroGenericSerialization is added last, the fallback could simply be to change its accept method to accept anything that has AVRO_SCHEMA_KEY defined, no?

One other thing: we should probably adopt a naming convention for metadata keys.  Should they be Java-package-like strings, e.g., org.apache.hadoop.io.serialization.class, or HTTP/SMTP header-like things, e.g., Serialization-Class?

> Add metadata to Serializations
> ------------------------------
>
>                 Key: HADOOP-6165
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6165
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: contrib/serialization
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6165-v2.patch, HADOOP-6165-v3.patch, HADOOP-6165.patch
>
>
> The Serialization framework only allows a class to be passed as metadata. This assumes there is a one-to-one mapping between types and Serializations, which is overly restrictive. By permitting applications to pass arbitrary metadata to Serializations, they can get more control over which Serialization is used, and would also allow, for example, one to pass an Avro schema to an Avro Serialization.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-813-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 20:51:32 2009
Return-Path: <common-issues-return-813-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45725 invoked from network); 14 Aug 2009 20:51:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 20:51:32 -0000
Received: (qmail 36080 invoked by uid 500); 14 Aug 2009 20:51:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36022 invoked by uid 500); 14 Aug 2009 20:51:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36012 invoked by uid 99); 14 Aug 2009 20:51:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 20:51:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 20:51:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C6737234C004
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 13:51:14 -0700 (PDT)
Message-ID: <1612591599.1250283074808.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 13:51:14 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6166:
--------------------------------

    Attachment: Rplots.pdf

Oops - I had some old data files lying around in my "out" directory from a previous run where I had a mistake in my code. I've removed them and regenerated the graphs here.

I'm currently running CRC32 benchmarks without concurrency and will upload those when they finish.

> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt, c6166_20090727.patch, c6166_20090728.patch, c6166_20090810.patch, c6166_20090811.patch, graph.r, graph.r, Rplots.pdf, Rplots.pdf, Rplots.pdf
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-814-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 21:05:30 2009
Return-Path: <common-issues-return-814-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50527 invoked from network); 14 Aug 2009 21:05:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 21:05:30 -0000
Received: (qmail 58856 invoked by uid 500); 14 Aug 2009 21:05:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58800 invoked by uid 500); 14 Aug 2009 21:05:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58790 invoked by uid 99); 14 Aug 2009 21:05:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 21:05:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 21:05:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D923C234C045
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 14:05:14 -0700 (PDT)
Message-ID: <1157834086.1250283914888.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 14:05:14 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6166:
--------------------------------

    Attachment: Rplots-nehalem64.pdf
                Rplots-laptop.pdf
                Rplots-nehalem32.pdf

Looks like the benchmark has run long enough to get good data. Here are the benchmarks from TestPureJavaCrc32 run on three different test systems. nehalem32 is the same nehalem box (3MB L2 cache) running a 32-bit JVM. nehalem64 is that box with a 64-bit JVM. "laptop" is my MacBook Pro (Core 2 duo) running a 64-bit JVM.

Each PDF has several pages:
  - The first graph shows performance over the whole byte range tested. You'll definitely have to zoom in to be able to see anything here, and even then it's not that useful.
  - The remaining graphs show the different algorithms' performance on different sizes (same as the tables people have been pasting into JIRA)

I ran the whole benchmark suite 50+ times to generate the error bars. Hopefully they'll serve as a good visual indicator for where the differences are actually statistically significant.

In summary, here's how I interpret the data:

- For the 4-byte case, PureJavaCrc32 wins out on my laptop and the 32-bit JVM by a strong margin. On the 64-bit JVM it's within 5-10% of the rest (very little difference)
- The 8-byte case is interesting - all of the 16_16* CRCs perform worse then the _8_8 CRCs. On the 32-bit JDK it's especially obvious (nearly a factor of two)
- The 512-byte case (probably most common for DFS) - everyone is pretty much neck and neck. The 8_8d implementation wins significantly on nehalem64, and the 8_8b wins significantly on nehalem32. On my laptop they're all within the error bars except for 16_16 which is significantly worse
- The 16MB case is the same as the 512-byte case, just more pronounced. 8_8d wins on 64-bit, 8_8b on 32-bit, both by about 10%.

So, I think the next step here is to profile a couple of MR applications to see what sizes are most common.

My personal opinion is that we should target the 64-bit Nehalem architecture and the 128-byte size range. This would point to the 8_8d implementation as the winner.

> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt, c6166_20090727.patch, c6166_20090728.patch, c6166_20090810.patch, c6166_20090811.patch, graph.r, graph.r, Rplots-laptop.pdf, Rplots-nehalem32.pdf, Rplots-nehalem64.pdf, Rplots.pdf, Rplots.pdf, Rplots.pdf
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-815-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 21:35:33 2009
Return-Path: <common-issues-return-815-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62117 invoked from network); 14 Aug 2009 21:35:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 21:35:33 -0000
Received: (qmail 92760 invoked by uid 500); 14 Aug 2009 21:35:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92705 invoked by uid 500); 14 Aug 2009 21:35:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92695 invoked by uid 99); 14 Aug 2009 21:35:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 21:35:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 21:35:37 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D25EA234C004
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 14:35:14 -0700 (PDT)
Message-ID: <2085921440.1250285714846.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 14:35:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743453#action_12743453 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6166:
------------------------------------------------

> ... This would point to the 8_8d implementation as the winner.
I agree since Crc32_8_8d is the best on 64-bit platforms.

> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt, c6166_20090727.patch, c6166_20090728.patch, c6166_20090810.patch, c6166_20090811.patch, graph.r, graph.r, Rplots-laptop.pdf, Rplots-nehalem32.pdf, Rplots-nehalem64.pdf, Rplots.pdf, Rplots.pdf, Rplots.pdf
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-816-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 21:57:30 2009
Return-Path: <common-issues-return-816-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72396 invoked from network); 14 Aug 2009 21:57:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 21:57:30 -0000
Received: (qmail 10748 invoked by uid 500); 14 Aug 2009 21:57:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10692 invoked by uid 500); 14 Aug 2009 21:57:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10682 invoked by uid 99); 14 Aug 2009 21:57:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 21:57:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 21:57:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C437D234C004
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 14:57:14 -0700 (PDT)
Message-ID: <1533925802.1250287034799.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 14:57:14 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743463#action_12743463 ] 

Todd Lipcon commented on HADOOP-6166:
-------------------------------------

bq. I agree since Crc32_8_8d is the best on 64-bit platforms.

+1, though I just realized I didn't run my 32-bit JVM tests with -server. I should probably run that this afternoon before we make a final decision.

> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt, c6166_20090727.patch, c6166_20090728.patch, c6166_20090810.patch, c6166_20090811.patch, graph.r, graph.r, Rplots-laptop.pdf, Rplots-nehalem32.pdf, Rplots-nehalem64.pdf, Rplots.pdf, Rplots.pdf, Rplots.pdf
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-817-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 22:43:31 2009
Return-Path: <common-issues-return-817-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92787 invoked from network); 14 Aug 2009 22:43:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 22:43:30 -0000
Received: (qmail 64456 invoked by uid 500); 14 Aug 2009 22:43:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64400 invoked by uid 500); 14 Aug 2009 22:43:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64383 invoked by uid 99); 14 Aug 2009 22:43:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 22:43:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 22:43:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 64160234C4AA
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 15:43:15 -0700 (PDT)
Message-ID: <1607760945.1250289795409.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 15:43:15 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5073) Hadoop 1.0 Interface Classification -
 scope (visibility - public/private) and stability
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-5073:
--------------------------------

    Attachment: 5073_demo.png
                HADOOP-5073.patch

Did some work creating a test implementation of the above. Attached as a patch as a prototype of what such a system could look like, along with a screenshot of how the annotations end up looking in the Javadoc.  Just a proof of concept, don't ding me over file locations and such.

It turned out to be better to have separate annotations for each of the possible labels: IDE integration was much better and the values show up more clearly in the Javadoc and IDE  integreated Javadoc viewer.  

The biggest problem is that with the project split, these annotations are not carried over and we still have the fact that @Inherited and @Documented don't actually work correctly.  So, for instance, even though I annotated FileSystem, none of that carries over to DistributedFileSystem, since it's in a different jar.  Therefore, there would be a significant labeling initial and maintenance effort to make sure everything was marked as it should be.  Anyway, this is what the above scheme could look like if implemented.

> Hadoop 1.0 Interface Classification - scope (visibility - public/private) and stability
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5073
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5073
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: 5073_demo.png, HADOOP-5073.patch
>
>
> This jira proposes an interface classification for hadoop interfaces.
> The discussion was started in email alias core-dev@hadoop.apache.org in Nov 2008.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-818-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 14 23:46:32 2009
Return-Path: <common-issues-return-818-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5470 invoked from network); 14 Aug 2009 23:46:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 14 Aug 2009 23:46:32 -0000
Received: (qmail 40874 invoked by uid 500); 14 Aug 2009 23:46:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40820 invoked by uid 500); 14 Aug 2009 23:46:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40809 invoked by uid 99); 14 Aug 2009 23:46:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 23:46:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 23:46:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CEBD7234C004
	for <common-issues@hadoop.apache.org>; Fri, 14 Aug 2009 16:46:14 -0700 (PDT)
Message-ID: <1796055791.1250293574830.JavaMail.jira@brutus>
Date: Fri, 14 Aug 2009 16:46:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6182) Adding Apache License Headers and
 reduce releaseaudit warnings to zero
In-Reply-To: <739323326.1249883774947.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743491#action_12743491 ] 

Hadoop QA commented on HADOOP-6182:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12416590/hadoop-6182.patch
  against trunk revision 804352.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 60 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/606/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/606/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/606/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/606/console

This message is automatically generated.

> Adding Apache License Headers and reduce releaseaudit warnings to zero
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6182
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6182
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Giridharan Kesavan
>            Assignee: Lee Tucker
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6182.patch, Hadoop-6182.patch
>
>
> As of now rats tool shows 111 RA warnings 
> [rat:report] Summary
> [rat:report] -------
> [rat:report] Notes: 18
> [rat:report] Binaries: 118
> [rat:report] Archives: 33
> [rat:report] Standards: 942
> [rat:report] 
> [rat:report] Apache Licensed: 820
> [rat:report] Generated Documents: 11
> [rat:report] 
> [rat:report] JavaDocs are generated and so license header is optional
> [rat:report] Generated files do not required license headers
> [rat:report] 
> [rat:report] 111 Unknown Licenses
> [rat:report] 
> [rat:report] *******************************

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-819-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 15 12:38:30 2009
Return-Path: <common-issues-return-819-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67194 invoked from network); 15 Aug 2009 12:38:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Aug 2009 12:38:30 -0000
Received: (qmail 84065 invoked by uid 500); 15 Aug 2009 12:38:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84008 invoked by uid 500); 15 Aug 2009 12:38:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83998 invoked by uid 99); 15 Aug 2009 12:38:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Aug 2009 12:38:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Aug 2009 12:38:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E90CD234C055
	for <common-issues@hadoop.apache.org>; Sat, 15 Aug 2009 05:38:14 -0700 (PDT)
Message-ID: <1053998995.1250339894953.JavaMail.jira@brutus>
Date: Sat, 15 Aug 2009 05:38:14 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6192) Shell.getUlimitMemoryCommand is
 tied to Map-Reduce
In-Reply-To: <2147018514.1250206334972.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743699#action_12743699 ] 

Hudson commented on HADOOP-6192:
--------------------------------

Integrated in Hadoop-Common-trunk #55 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/55/])
    

> Shell.getUlimitMemoryCommand is tied to Map-Reduce
> --------------------------------------------------
>
>                 Key: HADOOP-6192
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6192
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.20.0
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6192_0_20090813.patch
>
>
> Currently org.apache.hadoop.util.Shell.getUlimitMemoryCommand relies on a MAPREDUCE specific configuration property for the memory limit. We should break the link.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-820-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 15 12:38:30 2009
Return-Path: <common-issues-return-820-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67227 invoked from network); 15 Aug 2009 12:38:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Aug 2009 12:38:30 -0000
Received: (qmail 84150 invoked by uid 500); 15 Aug 2009 12:38:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84093 invoked by uid 500); 15 Aug 2009 12:38:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84070 invoked by uid 99); 15 Aug 2009 12:38:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Aug 2009 12:38:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Aug 2009 12:38:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 05CD4234C1EE
	for <common-issues@hadoop.apache.org>; Sat, 15 Aug 2009 05:38:15 -0700 (PDT)
Message-ID: <2051430771.1250339895023.JavaMail.jira@brutus>
Date: Sat, 15 Aug 2009 05:38:15 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6182) Adding Apache License Headers and
 reduce releaseaudit warnings to zero
In-Reply-To: <739323326.1249883774947.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743701#action_12743701 ] 

Hudson commented on HADOOP-6182:
--------------------------------

Integrated in Hadoop-Common-trunk #55 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/55/])
    Reverting  and HADOOP-6195


> Adding Apache License Headers and reduce releaseaudit warnings to zero
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6182
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6182
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Giridharan Kesavan
>            Assignee: Lee Tucker
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6182.patch, Hadoop-6182.patch
>
>
> As of now rats tool shows 111 RA warnings 
> [rat:report] Summary
> [rat:report] -------
> [rat:report] Notes: 18
> [rat:report] Binaries: 118
> [rat:report] Archives: 33
> [rat:report] Standards: 942
> [rat:report] 
> [rat:report] Apache Licensed: 820
> [rat:report] Generated Documents: 11
> [rat:report] 
> [rat:report] JavaDocs are generated and so license header is optional
> [rat:report] Generated files do not required license headers
> [rat:report] 
> [rat:report] 111 Unknown Licenses
> [rat:report] 
> [rat:report] *******************************

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-821-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 15 12:38:32 2009
Return-Path: <common-issues-return-821-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67316 invoked from network); 15 Aug 2009 12:38:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Aug 2009 12:38:32 -0000
Received: (qmail 84247 invoked by uid 500); 15 Aug 2009 12:38:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84186 invoked by uid 500); 15 Aug 2009 12:38:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84173 invoked by uid 99); 15 Aug 2009 12:38:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Aug 2009 12:38:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Aug 2009 12:38:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D6EC7234C004
	for <common-issues@hadoop.apache.org>; Sat, 15 Aug 2009 05:38:14 -0700 (PDT)
Message-ID: <1835554190.1250339894876.JavaMail.jira@brutus>
Date: Sat, 15 Aug 2009 05:38:14 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-3981) Need a distributed file checksum
 algorithm for HDFS
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-3981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743698#action_12743698 ] 

Hudson commented on HADOOP-3981:
--------------------------------

Integrated in Hadoop-Common-trunk #55 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/55/])
    Delete the empty file src/core/org/apache/hadoop/fs/LengthFileChecksum.java which was removed by .


> Need a distributed file checksum algorithm for HDFS
> ---------------------------------------------------
>
>                 Key: HADOOP-3981
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3981
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.19.0
>
>         Attachments: 3981_20080909.patch, 3981_20080910.patch, 3981_20080910b.patch, 3981_20080912.patch
>
>
> Traditional message digest algorithms, like MD5, SHA1, etc., require reading the entire input message sequentially in a central location.  HDFS supports large files with multiple tera bytes.  The overhead of reading the entire file is huge. A distributed file checksum algorithm is needed for HDFS.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-822-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 15 12:38:32 2009
Return-Path: <common-issues-return-822-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67335 invoked from network); 15 Aug 2009 12:38:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Aug 2009 12:38:32 -0000
Received: (qmail 84314 invoked by uid 500); 15 Aug 2009 12:38:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84265 invoked by uid 500); 15 Aug 2009 12:38:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84218 invoked by uid 99); 15 Aug 2009 12:38:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Aug 2009 12:38:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Aug 2009 12:38:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id F122A234C1E7
	for <common-issues@hadoop.apache.org>; Sat, 15 Aug 2009 05:38:14 -0700 (PDT)
Message-ID: <223194344.1250339894987.JavaMail.jira@brutus>
Date: Sat, 15 Aug 2009 05:38:14 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6195) checkstyle target fails common
 trunk build.
In-Reply-To: <1561183249.1250263874967.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743700#action_12743700 ] 

Hudson commented on HADOOP-6195:
--------------------------------

Integrated in Hadoop-Common-trunk #55 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/55/])
    Reverting HADOOP-6182 and 
. Fix checkstyle xml version tag. Contributed by Lee Tucker


> checkstyle target fails common trunk build.
> -------------------------------------------
>
>                 Key: HADOOP-6195
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6195
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Giridharan Kesavan
>         Attachments: HADOOP-6195-v2.patch, Hadoop-6195.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-823-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 15 13:58:32 2009
Return-Path: <common-issues-return-823-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85425 invoked from network); 15 Aug 2009 13:58:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Aug 2009 13:58:32 -0000
Received: (qmail 22661 invoked by uid 500); 15 Aug 2009 13:58:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22598 invoked by uid 500); 15 Aug 2009 13:58:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22582 invoked by uid 99); 15 Aug 2009 13:58:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Aug 2009 13:58:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Aug 2009 13:58:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CD7AC234C004
	for <common-issues@hadoop.apache.org>; Sat, 15 Aug 2009 06:58:14 -0700 (PDT)
Message-ID: <1414590743.1250344694828.JavaMail.jira@brutus>
Date: Sat, 15 Aug 2009 06:58:14 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6195) checkstyle target fails common
 trunk build.
In-Reply-To: <1561183249.1250263874967.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743713#action_12743713 ] 

Hudson commented on HADOOP-6195:
--------------------------------

Integrated in Hadoop-Hdfs-trunk #50 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/50/])
    Reverting HADOOP-6182 and 


> checkstyle target fails common trunk build.
> -------------------------------------------
>
>                 Key: HADOOP-6195
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6195
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Giridharan Kesavan
>         Attachments: HADOOP-6195-v2.patch, Hadoop-6195.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-824-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 15 13:58:32 2009
Return-Path: <common-issues-return-824-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85457 invoked from network); 15 Aug 2009 13:58:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Aug 2009 13:58:32 -0000
Received: (qmail 22736 invoked by uid 500); 15 Aug 2009 13:58:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22676 invoked by uid 500); 15 Aug 2009 13:58:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22597 invoked by uid 99); 15 Aug 2009 13:58:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Aug 2009 13:58:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Aug 2009 13:58:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DD255234C052
	for <common-issues@hadoop.apache.org>; Sat, 15 Aug 2009 06:58:14 -0700 (PDT)
Message-ID: <160049310.1250344694905.JavaMail.jira@brutus>
Date: Sat, 15 Aug 2009 06:58:14 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6182) Adding Apache License Headers and
 reduce releaseaudit warnings to zero
In-Reply-To: <739323326.1249883774947.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743714#action_12743714 ] 

Hudson commented on HADOOP-6182:
--------------------------------

Integrated in Hadoop-Hdfs-trunk #50 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/50/])
    Reverting  and HADOOP-6195


> Adding Apache License Headers and reduce releaseaudit warnings to zero
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6182
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6182
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Giridharan Kesavan
>            Assignee: Lee Tucker
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6182.patch, Hadoop-6182.patch
>
>
> As of now rats tool shows 111 RA warnings 
> [rat:report] Summary
> [rat:report] -------
> [rat:report] Notes: 18
> [rat:report] Binaries: 118
> [rat:report] Archives: 33
> [rat:report] Standards: 942
> [rat:report] 
> [rat:report] Apache Licensed: 820
> [rat:report] Generated Documents: 11
> [rat:report] 
> [rat:report] JavaDocs are generated and so license header is optional
> [rat:report] Generated files do not required license headers
> [rat:report] 
> [rat:report] 111 Unknown Licenses
> [rat:report] 
> [rat:report] *******************************

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-825-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 15 14:06:30 2009
Return-Path: <common-issues-return-825-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88562 invoked from network); 15 Aug 2009 14:06:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Aug 2009 14:06:30 -0000
Received: (qmail 25528 invoked by uid 500); 15 Aug 2009 14:06:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25475 invoked by uid 500); 15 Aug 2009 14:06:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25465 invoked by uid 99); 15 Aug 2009 14:06:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Aug 2009 14:06:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Aug 2009 14:06:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 01FEB234C004
	for <common-issues@hadoop.apache.org>; Sat, 15 Aug 2009 07:06:15 -0700 (PDT)
Message-ID: <39428231.1250345174993.JavaMail.jira@brutus>
Date: Sat, 15 Aug 2009 07:06:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6182) Adding Apache License Headers and
 reduce releaseaudit warnings to zero
In-Reply-To: <739323326.1249883774947.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6182:
---------------------------------------

    Attachment: hadoop-6182.patch

> Adding Apache License Headers and reduce releaseaudit warnings to zero
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6182
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6182
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Giridharan Kesavan
>            Assignee: Lee Tucker
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6182.patch, hadoop-6182.patch, Hadoop-6182.patch
>
>
> As of now rats tool shows 111 RA warnings 
> [rat:report] Summary
> [rat:report] -------
> [rat:report] Notes: 18
> [rat:report] Binaries: 118
> [rat:report] Archives: 33
> [rat:report] Standards: 942
> [rat:report] 
> [rat:report] Apache Licensed: 820
> [rat:report] Generated Documents: 11
> [rat:report] 
> [rat:report] JavaDocs are generated and so license header is optional
> [rat:report] Generated files do not required license headers
> [rat:report] 
> [rat:report] 111 Unknown Licenses
> [rat:report] 
> [rat:report] *******************************

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-826-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 15 14:08:30 2009
Return-Path: <common-issues-return-826-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88941 invoked from network); 15 Aug 2009 14:08:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Aug 2009 14:08:30 -0000
Received: (qmail 25988 invoked by uid 500); 15 Aug 2009 14:08:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25934 invoked by uid 500); 15 Aug 2009 14:08:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25924 invoked by uid 99); 15 Aug 2009 14:08:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Aug 2009 14:08:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Aug 2009 14:08:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D548E234C004
	for <common-issues@hadoop.apache.org>; Sat, 15 Aug 2009 07:08:14 -0700 (PDT)
Message-ID: <1704696949.1250345294861.JavaMail.jira@brutus>
Date: Sat, 15 Aug 2009 07:08:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6182) Adding Apache License Headers and
 reduce releaseaudit warnings to zero
In-Reply-To: <739323326.1249883774947.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6182:
---------------------------------------

    Status: Open  (was: Patch Available)

> Adding Apache License Headers and reduce releaseaudit warnings to zero
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6182
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6182
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Giridharan Kesavan
>            Assignee: Lee Tucker
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6182.patch, hadoop-6182.patch, Hadoop-6182.patch
>
>
> As of now rats tool shows 111 RA warnings 
> [rat:report] Summary
> [rat:report] -------
> [rat:report] Notes: 18
> [rat:report] Binaries: 118
> [rat:report] Archives: 33
> [rat:report] Standards: 942
> [rat:report] 
> [rat:report] Apache Licensed: 820
> [rat:report] Generated Documents: 11
> [rat:report] 
> [rat:report] JavaDocs are generated and so license header is optional
> [rat:report] Generated files do not required license headers
> [rat:report] 
> [rat:report] 111 Unknown Licenses
> [rat:report] 
> [rat:report] *******************************

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-827-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 15 14:08:32 2009
Return-Path: <common-issues-return-827-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88999 invoked from network); 15 Aug 2009 14:08:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Aug 2009 14:08:32 -0000
Received: (qmail 26079 invoked by uid 500); 15 Aug 2009 14:08:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26021 invoked by uid 500); 15 Aug 2009 14:08:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26011 invoked by uid 99); 15 Aug 2009 14:08:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Aug 2009 14:08:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Aug 2009 14:08:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E0EE7234C1E6
	for <common-issues@hadoop.apache.org>; Sat, 15 Aug 2009 07:08:14 -0700 (PDT)
Message-ID: <2066530672.1250345294920.JavaMail.jira@brutus>
Date: Sat, 15 Aug 2009 07:08:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6182) Adding Apache License Headers and
 reduce releaseaudit warnings to zero
In-Reply-To: <739323326.1249883774947.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6182:
---------------------------------------

    Status: Patch Available  (was: Open)

> Adding Apache License Headers and reduce releaseaudit warnings to zero
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6182
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6182
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Giridharan Kesavan
>            Assignee: Lee Tucker
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6182.patch, hadoop-6182.patch, Hadoop-6182.patch
>
>
> As of now rats tool shows 111 RA warnings 
> [rat:report] Summary
> [rat:report] -------
> [rat:report] Notes: 18
> [rat:report] Binaries: 118
> [rat:report] Archives: 33
> [rat:report] Standards: 942
> [rat:report] 
> [rat:report] Apache Licensed: 820
> [rat:report] Generated Documents: 11
> [rat:report] 
> [rat:report] JavaDocs are generated and so license header is optional
> [rat:report] Generated files do not required license headers
> [rat:report] 
> [rat:report] 111 Unknown Licenses
> [rat:report] 
> [rat:report] *******************************

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-828-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 15 19:35:30 2009
Return-Path: <common-issues-return-828-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1295 invoked from network); 15 Aug 2009 19:35:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 15 Aug 2009 19:35:30 -0000
Received: (qmail 34720 invoked by uid 500); 15 Aug 2009 19:35:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34664 invoked by uid 500); 15 Aug 2009 19:35:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34654 invoked by uid 99); 15 Aug 2009 19:35:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Aug 2009 19:35:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 15 Aug 2009 19:35:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 39B70234C051
	for <common-issues@hadoop.apache.org>; Sat, 15 Aug 2009 12:35:15 -0700 (PDT)
Message-ID: <1250522822.1250364915222.JavaMail.jira@brutus>
Date: Sat, 15 Aug 2009 12:35:15 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6182) Adding Apache License Headers and
 reduce releaseaudit warnings to zero
In-Reply-To: <739323326.1249883774947.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743752#action_12743752 ] 

Hadoop QA commented on HADOOP-6182:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12416660/hadoop-6182.patch
  against trunk revision 804352.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 60 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/607/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/607/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/607/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/607/console

This message is automatically generated.

> Adding Apache License Headers and reduce releaseaudit warnings to zero
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6182
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6182
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Giridharan Kesavan
>            Assignee: Lee Tucker
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6182.patch, hadoop-6182.patch, Hadoop-6182.patch
>
>
> As of now rats tool shows 111 RA warnings 
> [rat:report] Summary
> [rat:report] -------
> [rat:report] Notes: 18
> [rat:report] Binaries: 118
> [rat:report] Archives: 33
> [rat:report] Standards: 942
> [rat:report] 
> [rat:report] Apache Licensed: 820
> [rat:report] Generated Documents: 11
> [rat:report] 
> [rat:report] JavaDocs are generated and so license header is optional
> [rat:report] Generated files do not required license headers
> [rat:report] 
> [rat:report] 111 Unknown Licenses
> [rat:report] 
> [rat:report] *******************************

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-829-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 16 09:10:32 2009
Return-Path: <common-issues-return-829-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23296 invoked from network); 16 Aug 2009 09:10:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Aug 2009 09:10:32 -0000
Received: (qmail 23450 invoked by uid 500); 16 Aug 2009 09:10:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23405 invoked by uid 500); 16 Aug 2009 09:10:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23395 invoked by uid 99); 16 Aug 2009 09:10:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 16 Aug 2009 09:10:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 16 Aug 2009 09:10:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CEAE8234C056
	for <common-issues@hadoop.apache.org>; Sun, 16 Aug 2009 02:10:14 -0700 (PDT)
Message-ID: <1782929485.1250413814845.JavaMail.jira@brutus>
Date: Sun, 16 Aug 2009 02:10:14 -0700 (PDT)
From: "Jay Booth (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6196) sync(0); next() breaks SequenceFile
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

sync(0); next() breaks SequenceFile
-----------------------------------

                 Key: HADOOP-6196
                 URL: https://issues.apache.org/jira/browse/HADOOP-6196
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Jay Booth


Currently, the end of the SequenceFile header is a sync block that isn't prefaced with SYNC_ESCAPE.  This means that sync(0) followed by next() fails.  Patch w/ test attached, bumps VERSION from 6 to 7.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-830-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 16 09:16:30 2009
Return-Path: <common-issues-return-830-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24301 invoked from network); 16 Aug 2009 09:16:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Aug 2009 09:16:30 -0000
Received: (qmail 24672 invoked by uid 500); 16 Aug 2009 09:16:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24618 invoked by uid 500); 16 Aug 2009 09:16:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24608 invoked by uid 99); 16 Aug 2009 09:16:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 16 Aug 2009 09:16:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 16 Aug 2009 09:16:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C6084234C055
	for <common-issues@hadoop.apache.org>; Sun, 16 Aug 2009 02:16:14 -0700 (PDT)
Message-ID: <2105324511.1250414174796.JavaMail.jira@brutus>
Date: Sun, 16 Aug 2009 02:16:14 -0700 (PDT)
From: "Jay Booth (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6196) sync(0); next() breaks SequenceFile
In-Reply-To: <1782929485.1250413814845.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jay Booth updated HADOOP-6196:
------------------------------

    Attachment: sync-bug.patch

Here's the patch.  To reproduce the problem, patch, then revert SequenceFile (to undo the fix) and run the attached test TestSequenceFileSync.  

Strangely, it works sometimes and breaks other times.  When it works, it'll tend to work for a few times in a row until a few seconds pass and then it starts breaking again (EOFException at readInt).  I don't get it, could be something weird with my environment, but the patch to SequenceFile means that it works all the time, so who cares :)  

The fix required bumping the version of SequenceFile from 6 to 7, writing the SYNC_ESCAPE int properly before writing the sync block at the end of the header file, and then scanning past that prior to loading sync when opening.  Now sync(0); next();  works properly.

> sync(0); next() breaks SequenceFile
> -----------------------------------
>
>                 Key: HADOOP-6196
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6196
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jay Booth
>         Attachments: sync-bug.patch
>
>
> Currently, the end of the SequenceFile header is a sync block that isn't prefaced with SYNC_ESCAPE.  This means that sync(0) followed by next() fails.  Patch w/ test attached, bumps VERSION from 6 to 7.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-831-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 16 09:18:30 2009
Return-Path: <common-issues-return-831-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24848 invoked from network); 16 Aug 2009 09:18:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Aug 2009 09:18:30 -0000
Received: (qmail 25030 invoked by uid 500); 16 Aug 2009 09:18:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24976 invoked by uid 500); 16 Aug 2009 09:18:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24966 invoked by uid 99); 16 Aug 2009 09:18:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 16 Aug 2009 09:18:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 16 Aug 2009 09:18:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CABA4234C055
	for <common-issues@hadoop.apache.org>; Sun, 16 Aug 2009 02:18:14 -0700 (PDT)
Message-ID: <441254149.1250414294816.JavaMail.jira@brutus>
Date: Sun, 16 Aug 2009 02:18:14 -0700 (PDT)
From: "Jay Booth (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6196) sync(0); next() breaks SequenceFile
In-Reply-To: <1782929485.1250413814845.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jay Booth updated HADOOP-6196:
------------------------------

    Release Note: The consequences of committing this would be that hadoop-0.20 can't read hadoop-0.21 sequence files unless we backport.  The change in itself is in the neighborhood of 5 lines of code, but does introduce an incompatibility.
          Status: Patch Available  (was: Open)

> sync(0); next() breaks SequenceFile
> -----------------------------------
>
>                 Key: HADOOP-6196
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6196
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jay Booth
>         Attachments: sync-bug.patch
>
>
> Currently, the end of the SequenceFile header is a sync block that isn't prefaced with SYNC_ESCAPE.  This means that sync(0) followed by next() fails.  Patch w/ test attached, bumps VERSION from 6 to 7.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-832-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 16 09:35:30 2009
Return-Path: <common-issues-return-832-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33309 invoked from network); 16 Aug 2009 09:35:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 16 Aug 2009 09:35:30 -0000
Received: (qmail 31292 invoked by uid 500); 16 Aug 2009 09:35:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31237 invoked by uid 500); 16 Aug 2009 09:35:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31226 invoked by uid 99); 16 Aug 2009 09:35:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 16 Aug 2009 09:35:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 16 Aug 2009 09:35:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DF0F4234C055
	for <common-issues@hadoop.apache.org>; Sun, 16 Aug 2009 02:35:14 -0700 (PDT)
Message-ID: <33221814.1250415314898.JavaMail.jira@brutus>
Date: Sun, 16 Aug 2009 02:35:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6196) sync(0); next() breaks SequenceFile
In-Reply-To: <1782929485.1250413814845.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743825#action_12743825 ] 

Hadoop QA commented on HADOOP-6196:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12416705/sync-bug.patch
  against trunk revision 804352.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/608/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/608/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/608/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/608/console

This message is automatically generated.

> sync(0); next() breaks SequenceFile
> -----------------------------------
>
>                 Key: HADOOP-6196
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6196
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jay Booth
>         Attachments: sync-bug.patch
>
>
> Currently, the end of the SequenceFile header is a sync block that isn't prefaced with SYNC_ESCAPE.  This means that sync(0) followed by next() fails.  Patch w/ test attached, bumps VERSION from 6 to 7.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-833-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 17 02:32:30 2009
Return-Path: <common-issues-return-833-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88563 invoked from network); 17 Aug 2009 02:32:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Aug 2009 02:32:30 -0000
Received: (qmail 49418 invoked by uid 500); 17 Aug 2009 02:32:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49362 invoked by uid 500); 17 Aug 2009 02:32:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49352 invoked by uid 99); 17 Aug 2009 02:32:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 02:32:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 02:32:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CB689234C046
	for <common-issues@hadoop.apache.org>; Sun, 16 Aug 2009 19:32:14 -0700 (PDT)
Message-ID: <476532120.1250476334819.JavaMail.jira@brutus>
Date: Sun, 16 Aug 2009 19:32:14 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743950#action_12743950 ] 

Todd Lipcon commented on HADOOP-6166:
-------------------------------------

Ran the same benchmarks with java -d32 -server and the results are indistinguishable (I guess since the benchmarks run a long time the client VM does just as much JIT as the server VM).

So, pending anyone coming up with evidence that we should fine tune for the really-small-checksum case on processors like my laptop, 8_8d wins by enough of a margin it seems worth doing.

> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt, c6166_20090727.patch, c6166_20090728.patch, c6166_20090810.patch, c6166_20090811.patch, graph.r, graph.r, Rplots-laptop.pdf, Rplots-nehalem32.pdf, Rplots-nehalem64.pdf, Rplots.pdf, Rplots.pdf, Rplots.pdf
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-834-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 17 04:06:30 2009
Return-Path: <common-issues-return-834-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14670 invoked from network); 17 Aug 2009 04:06:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Aug 2009 04:06:30 -0000
Received: (qmail 86111 invoked by uid 500); 17 Aug 2009 04:06:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86055 invoked by uid 500); 17 Aug 2009 04:06:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86045 invoked by uid 99); 17 Aug 2009 04:06:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 04:06:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 04:06:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CC67C234C046
	for <common-issues@hadoop.apache.org>; Sun, 16 Aug 2009 21:06:14 -0700 (PDT)
Message-ID: <75892994.1250481974823.JavaMail.jira@brutus>
Date: Sun, 16 Aug 2009 21:06:14 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6182) Adding Apache License Headers and
 reduce releaseaudit warnings to zero
In-Reply-To: <739323326.1249883774947.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743962#action_12743962 ] 

Hudson commented on HADOOP-6182:
--------------------------------

Integrated in Hadoop-Common-trunk #57 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/57/])
    . Fix ReleaseAudit warnings. Contributed by Giridharan Kesavan and Lee Tucker


> Adding Apache License Headers and reduce releaseaudit warnings to zero
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6182
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6182
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Giridharan Kesavan
>            Assignee: Lee Tucker
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6182.patch, hadoop-6182.patch, Hadoop-6182.patch
>
>
> As of now rats tool shows 111 RA warnings 
> [rat:report] Summary
> [rat:report] -------
> [rat:report] Notes: 18
> [rat:report] Binaries: 118
> [rat:report] Archives: 33
> [rat:report] Standards: 942
> [rat:report] 
> [rat:report] Apache Licensed: 820
> [rat:report] Generated Documents: 11
> [rat:report] 
> [rat:report] JavaDocs are generated and so license header is optional
> [rat:report] Generated files do not required license headers
> [rat:report] 
> [rat:report] 111 Unknown Licenses
> [rat:report] 
> [rat:report] *******************************

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-835-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 17 04:28:32 2009
Return-Path: <common-issues-return-835-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17307 invoked from network); 17 Aug 2009 04:28:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Aug 2009 04:28:32 -0000
Received: (qmail 99759 invoked by uid 500); 17 Aug 2009 04:28:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99705 invoked by uid 500); 17 Aug 2009 04:28:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99691 invoked by uid 99); 17 Aug 2009 04:28:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 04:28:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 04:28:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C73CC234C046
	for <common-issues@hadoop.apache.org>; Sun, 16 Aug 2009 21:28:14 -0700 (PDT)
Message-ID: <1368735177.1250483294801.JavaMail.jira@brutus>
Date: Sun, 16 Aug 2009 21:28:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6182) Adding Apache License Headers and
 reduce releaseaudit warnings to zero
In-Reply-To: <739323326.1249883774947.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6182:
---------------------------------------

    Resolution: Fixed
      Assignee: Giridharan Kesavan  (was: Lee Tucker)
        Status: Resolved  (was: Patch Available)

> Adding Apache License Headers and reduce releaseaudit warnings to zero
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6182
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6182
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6182.patch, hadoop-6182.patch, Hadoop-6182.patch
>
>
> As of now rats tool shows 111 RA warnings 
> [rat:report] Summary
> [rat:report] -------
> [rat:report] Notes: 18
> [rat:report] Binaries: 118
> [rat:report] Archives: 33
> [rat:report] Standards: 942
> [rat:report] 
> [rat:report] Apache Licensed: 820
> [rat:report] Generated Documents: 11
> [rat:report] 
> [rat:report] JavaDocs are generated and so license header is optional
> [rat:report] Generated files do not required license headers
> [rat:report] 
> [rat:report] 111 Unknown Licenses
> [rat:report] 
> [rat:report] *******************************

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-836-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 17 13:19:31 2009
Return-Path: <common-issues-return-836-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 212 invoked from network); 17 Aug 2009 13:19:31 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Aug 2009 13:19:31 -0000
Received: (qmail 8887 invoked by uid 500); 17 Aug 2009 13:19:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8844 invoked by uid 500); 17 Aug 2009 13:19:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8655 invoked by uid 99); 17 Aug 2009 13:19:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 13:19:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 13:19:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E5014234C052
	for <common-issues@hadoop.apache.org>; Mon, 17 Aug 2009 06:19:14 -0700 (PDT)
Message-ID: <1436235339.1250515154937.JavaMail.jira@brutus>
Date: Mon, 17 Aug 2009 06:19:14 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6182) Adding Apache License Headers and
 reduce releaseaudit warnings to zero
In-Reply-To: <739323326.1249883774947.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744054#action_12744054 ] 

Hudson commented on HADOOP-6182:
--------------------------------

Integrated in Hadoop-Hdfs-trunk #52 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/52/])
    . Fix ReleaseAudit warnings. Contributed by Giridharan Kesavan and Lee Tucker


> Adding Apache License Headers and reduce releaseaudit warnings to zero
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6182
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6182
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6182.patch, hadoop-6182.patch, Hadoop-6182.patch
>
>
> As of now rats tool shows 111 RA warnings 
> [rat:report] Summary
> [rat:report] -------
> [rat:report] Notes: 18
> [rat:report] Binaries: 118
> [rat:report] Archives: 33
> [rat:report] Standards: 942
> [rat:report] 
> [rat:report] Apache Licensed: 820
> [rat:report] Generated Documents: 11
> [rat:report] 
> [rat:report] JavaDocs are generated and so license header is optional
> [rat:report] Generated files do not required license headers
> [rat:report] 
> [rat:report] 111 Unknown Licenses
> [rat:report] 
> [rat:report] *******************************

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-837-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 17 17:10:30 2009
Return-Path: <common-issues-return-837-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99834 invoked from network); 17 Aug 2009 17:10:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Aug 2009 17:10:30 -0000
Received: (qmail 58312 invoked by uid 500); 17 Aug 2009 17:10:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58252 invoked by uid 500); 17 Aug 2009 17:10:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58237 invoked by uid 99); 17 Aug 2009 17:10:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 17:10:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 17:10:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E4ACE234C044
	for <common-issues@hadoop.apache.org>; Mon, 17 Aug 2009 10:10:14 -0700 (PDT)
Message-ID: <1148312246.1250529014919.JavaMail.jira@brutus>
Date: Mon, 17 Aug 2009 10:10:14 -0700 (PDT)
From: "Abdul Qadeer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4012) Providing splitting support for
 bzip2 compressed files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744136#action_12744136 ] 

Abdul Qadeer commented on HADOOP-4012:
--------------------------------------

This is a reminder for the Hadoop committers that HADOOP-4012 is ready to be reviewed.

> Providing splitting support for bzip2 compressed files
> ------------------------------------------------------
>
>                 Key: HADOOP-4012
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4012
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Abdul Qadeer
>            Assignee: Abdul Qadeer
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-4012-version1.patch, Hadoop-4012-version10.patch, Hadoop-4012-version11.patch, Hadoop-4012-version2.patch, Hadoop-4012-version3.patch, Hadoop-4012-version4.patch, Hadoop-4012-version5.patch, Hadoop-4012-version6.patch, Hadoop-4012-version7.patch, Hadoop-4012-version8.patch, Hadoop-4012-version9.patch
>
>
> Hadoop assumes that if the input data is compressed, it can not be split (mainly due to the limitation of many codecs that they need the whole input stream to decompress successfully).  So in such a case, Hadoop prepares only one split per compressed file, where the lower split limit is at 0 while the upper limit is the end of the file.  The consequence of this decision is that, one compress file goes to a single mapper. Although it circumvents the limitation of codecs (as mentioned above) but reduces the parallelism substantially, as it was possible otherwise in case of splitting.
> BZip2 is a compression / De-Compression algorithm which does compression on blocks of data and later these compressed blocks can be decompressed independent of each other.  This is indeed an opportunity that instead of one BZip2 compressed file going to one mapper, we can process chunks of file in parallel.  The correctness criteria of such a processing is that for a bzip2 compressed file, each compressed block should be processed by only one mapper and ultimately all the blocks of the file should be processed.  (By processing we mean the actual utilization of that un-compressed data (coming out of the codecs) in a mapper).
> We are writing the code to implement this suggested functionality.  Although we have used bzip2 as an example, but we have tried to extend Hadoop's compression interfaces so that any other codecs with the same capability as that of bzip2, could easily use the splitting support.  The details of these changes will be posted when we submit the code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-838-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 17 18:16:40 2009
Return-Path: <common-issues-return-838-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39685 invoked from network); 17 Aug 2009 18:16:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Aug 2009 18:16:34 -0000
Received: (qmail 72581 invoked by uid 500); 17 Aug 2009 18:16:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72326 invoked by uid 500); 17 Aug 2009 18:16:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72058 invoked by uid 99); 17 Aug 2009 18:16:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 18:16:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 18:16:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3E6E5234C498
	for <common-issues@hadoop.apache.org>; Mon, 17 Aug 2009 11:16:15 -0700 (PDT)
Message-ID: <1686425974.1250532975254.JavaMail.jira@brutus>
Date: Mon, 17 Aug 2009 11:16:15 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6196) sync(0); next() breaks SequenceFile
In-Reply-To: <1782929485.1250413814845.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744153#action_12744153 ] 

Konstantin Boudnik commented on HADOOP-6196:
--------------------------------------------

A few comments:
- new unit test is developed under JUnit3 conventions (e,g, it extends TestCase, imports junit.framework, etc.) I'd suggest to make it JUnit4 compliant: remove TestCase extention, add {{@Test}} annotation to the test method, use {{org.junit.*}} packages instead
- it seems that the unit test doesn't cover all the cases described in a comment above.
bq. Strangely, it works sometimes and breaks other times. When it works, it'll tend to work for a few times in a row until a few seconds pass and then it starts breaking again (EOFException at readInt)
It seems to be more sensible to do a number of {{seek(0); next()}} followed by a random pauses and {{seek()}} calls to locations differ from 0 
- {{SequenceFile.java}} has some white-space only modifications, e.g.
{noformat}
-      if (!blockCompressed) {
+      if (!blockCompressed) {        
{noformat}


> sync(0); next() breaks SequenceFile
> -----------------------------------
>
>                 Key: HADOOP-6196
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6196
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jay Booth
>         Attachments: sync-bug.patch
>
>
> Currently, the end of the SequenceFile header is a sync block that isn't prefaced with SYNC_ESCAPE.  This means that sync(0) followed by next() fails.  Patch w/ test attached, bumps VERSION from 6 to 7.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-839-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 17 20:51:32 2009
Return-Path: <common-issues-return-839-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96139 invoked from network); 17 Aug 2009 20:51:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Aug 2009 20:51:32 -0000
Received: (qmail 4439 invoked by uid 500); 17 Aug 2009 20:51:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4380 invoked by uid 500); 17 Aug 2009 20:51:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4368 invoked by uid 99); 17 Aug 2009 20:51:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 20:51:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 20:51:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CB21229A0011
	for <common-issues@hadoop.apache.org>; Mon, 17 Aug 2009 13:51:14 -0700 (PDT)
Message-ID: <1324647998.1250542274830.JavaMail.jira@brutus>
Date: Mon, 17 Aug 2009 13:51:14 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6197) Use aspects to introduce new methods
 required for testing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Use aspects to introduce new methods required for testing
---------------------------------------------------------

                 Key: HADOOP-6197
                 URL: https://issues.apache.org/jira/browse/HADOOP-6197
             Project: Hadoop Common
          Issue Type: Test
          Components: security, test
            Reporter: Konstantin Boudnik


HADOOP-6176 introduces two new methods which are essentially required for testing purposes. However, this bring a permanent modifications to the production code.

The same result can be reached by using aspect development. Thus the modifications will be brought in by byte code instrumentation and will exist only during the testing cycles.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-840-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 17 20:55:37 2009
Return-Path: <common-issues-return-840-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97420 invoked from network); 17 Aug 2009 20:55:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Aug 2009 20:55:37 -0000
Received: (qmail 10247 invoked by uid 500); 17 Aug 2009 20:55:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10194 invoked by uid 500); 17 Aug 2009 20:55:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10184 invoked by uid 99); 17 Aug 2009 20:55:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 20:55:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 20:55:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EC62729A0013
	for <common-issues@hadoop.apache.org>; Mon, 17 Aug 2009 13:55:14 -0700 (PDT)
Message-ID: <736620771.1250542514967.JavaMail.jira@brutus>
Date: Mon, 17 Aug 2009 13:55:14 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6197) Use aspects to introduce new methods
 required for testing
In-Reply-To: <1324647998.1250542274830.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6197:
---------------------------------------

    Attachment: HADOOP-6197.patch
                HADOOP-6197.sh

These two files show the essence of needed modifications. New aspect will instrument {{AccessTokenHandler}} class with two required methods. Thus, these aren't needed any longer in the source code of {{AccessTokenHandler}}.

At this time, the patch can't be applied because Common project doesn't use the fault injection framework introduced for HDFS (see HDFS-435). However, as soon as the framework will be ported over to the Common, we'll be able to use this very method of code instrumentation.

Also, the patch might require some extra fine tuning like moving some tests from a 'normal' test list into 'fault-injected' test list.

> Use aspects to introduce new methods required for testing
> ---------------------------------------------------------
>
>                 Key: HADOOP-6197
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6197
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: security, test
>            Reporter: Konstantin Boudnik
>         Attachments: HADOOP-6197.patch, HADOOP-6197.sh
>
>
> HADOOP-6176 introduces two new methods which are essentially required for testing purposes. However, this bring a permanent modifications to the production code.
> The same result can be reached by using aspect development. Thus the modifications will be brought in by byte code instrumentation and will exist only during the testing cycles.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-841-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 17 20:59:30 2009
Return-Path: <common-issues-return-841-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98747 invoked from network); 17 Aug 2009 20:59:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Aug 2009 20:59:30 -0000
Received: (qmail 22284 invoked by uid 500); 17 Aug 2009 20:59:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22244 invoked by uid 500); 17 Aug 2009 20:59:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22234 invoked by uid 99); 17 Aug 2009 20:59:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 20:59:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 20:59:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EEAE229A0013
	for <common-issues@hadoop.apache.org>; Mon, 17 Aug 2009 13:59:14 -0700 (PDT)
Message-ID: <1484104427.1250542754976.JavaMail.jira@brutus>
Date: Mon, 17 Aug 2009 13:59:14 -0700 (PDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6083) TestFsck takes nearly 10 minutes to
 run - a quarter of the entire hdfs-test time
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang reassigned HADOOP-6083:
-------------------------------------

    Assignee: Hairong Kuang

> TestFsck takes nearly 10 minutes to run - a quarter of the entire hdfs-test time
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6083
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6083
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>            Reporter: Jakob Homan
>            Assignee: Hairong Kuang
>
> run-test-hdfs was run repeatedly (n = 16) and the execution time of each test was recorded.  Testfsck was by far the longest test, with a median run time of 564 seconds, nearly 10 minutes.  This is a quarter of the entire time spent over the average 40 minute run for all the tests, and much too long for any individual test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-842-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 17 21:17:10 2009
Return-Path: <common-issues-return-842-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1053 invoked from network); 17 Aug 2009 21:05:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Aug 2009 21:05:32 -0000
Received: (qmail 31338 invoked by uid 500); 17 Aug 2009 21:05:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31286 invoked by uid 500); 17 Aug 2009 21:05:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31276 invoked by uid 99); 17 Aug 2009 21:05:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 21:05:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 21:05:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C3B77234C044
	for <common-issues@hadoop.apache.org>; Mon, 17 Aug 2009 14:05:14 -0700 (PDT)
Message-ID: <2071690098.1250543114797.JavaMail.jira@brutus>
Date: Mon, 17 Aug 2009 14:05:14 -0700 (PDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6083) TestFsck takes nearly 10 minutes to
 run - a quarter of the entire hdfs-test time
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744201#action_12744201 ] 

Hairong Kuang commented on HADOOP-6083:
---------------------------------------

TestFsck#testFsckMove corrupts a block file on disk and then depends on block reports to notify NN of the corrupt blocks. However, with the change in HADOOP-4584, block reports are always sent from in-memory volumeMap. The volumeMap depends on block scanner to scan the disk and update the in-memory volumeMap if there is a difference. The test needs to make the scanner period shorter.

> TestFsck takes nearly 10 minutes to run - a quarter of the entire hdfs-test time
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6083
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6083
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>            Reporter: Jakob Homan
>            Assignee: Hairong Kuang
>
> run-test-hdfs was run repeatedly (n = 16) and the execution time of each test was recorded.  Testfsck was by far the longest test, with a median run time of 564 seconds, nearly 10 minutes.  This is a quarter of the entire time spent over the average 40 minute run for all the tests, and much too long for any individual test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-843-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 17 22:15:18 2009
Return-Path: <common-issues-return-843-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28139 invoked from network); 17 Aug 2009 22:15:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 17 Aug 2009 22:15:18 -0000
Received: (qmail 41479 invoked by uid 500); 17 Aug 2009 21:15:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41422 invoked by uid 500); 17 Aug 2009 21:15:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41411 invoked by uid 99); 17 Aug 2009 21:15:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 21:15:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 21:15:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C9625234C044
	for <common-issues@hadoop.apache.org>; Mon, 17 Aug 2009 14:15:14 -0700 (PDT)
Message-ID: <1176954128.1250543714820.JavaMail.jira@brutus>
Date: Mon, 17 Aug 2009 14:15:14 -0700 (PDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6083) TestFsck takes nearly 10 minutes to
 run - a quarter of the entire hdfs-test time
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6083:
----------------------------------

    Attachment: slowFsck.patch

With this change, TestFsck runs around 1 minute.

> TestFsck takes nearly 10 minutes to run - a quarter of the entire hdfs-test time
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6083
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6083
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>            Reporter: Jakob Homan
>            Assignee: Hairong Kuang
>         Attachments: slowFsck.patch
>
>
> run-test-hdfs was run repeatedly (n = 16) and the execution time of each test was recorded.  Testfsck was by far the longest test, with a median run time of 564 seconds, nearly 10 minutes.  This is a quarter of the entire time spent over the average 40 minute run for all the tests, and much too long for any individual test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-844-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 18 04:19:27 2009
Return-Path: <common-issues-return-844-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77855 invoked from network); 18 Aug 2009 04:19:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Aug 2009 04:19:27 -0000
Received: (qmail 43919 invoked by uid 500); 18 Aug 2009 04:19:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43862 invoked by uid 500); 18 Aug 2009 04:19:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43850 invoked by uid 99); 18 Aug 2009 04:19:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 04:19:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 04:19:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1B0F629A0015
	for <common-issues@hadoop.apache.org>; Mon, 17 Aug 2009 21:19:15 -0700 (PDT)
Message-ID: <225352305.1250569155109.JavaMail.jira@brutus>
Date: Mon, 17 Aug 2009 21:19:15 -0700 (PDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6197) Use aspects to introduce new
 methods required for testing
In-Reply-To: <1324647998.1250542274830.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744353#action_12744353 ] 

Vinod K V commented on HADOOP-6197:
-----------------------------------

bq. However, as soon as the framework will be ported over to the Common, we'll be able to use this very method of code instrumentation.
Is there any JIRA issue already opened for this?

> Use aspects to introduce new methods required for testing
> ---------------------------------------------------------
>
>                 Key: HADOOP-6197
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6197
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: security, test
>            Reporter: Konstantin Boudnik
>         Attachments: HADOOP-6197.patch, HADOOP-6197.sh
>
>
> HADOOP-6176 introduces two new methods which are essentially required for testing purposes. However, this bring a permanent modifications to the production code.
> The same result can be reached by using aspect development. Thus the modifications will be brought in by byte code instrumentation and will exist only during the testing cycles.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-845-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 18 04:47:21 2009
Return-Path: <common-issues-return-845-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89874 invoked from network); 18 Aug 2009 04:47:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Aug 2009 04:47:20 -0000
Received: (qmail 74592 invoked by uid 500); 18 Aug 2009 04:47:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74533 invoked by uid 500); 18 Aug 2009 04:47:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74518 invoked by uid 99); 18 Aug 2009 04:47:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 04:47:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 04:47:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D1DC1234C044
	for <common-issues@hadoop.apache.org>; Mon, 17 Aug 2009 21:47:14 -0700 (PDT)
Message-ID: <72506238.1250570834846.JavaMail.jira@brutus>
Date: Mon, 17 Aug 2009 21:47:14 -0700 (PDT)
From: "Sreekanth Ramakrishnan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6105) Provide a way to automatically
 handle backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744359#action_12744359 ] 

Sreekanth Ramakrishnan commented on HADOOP-6105:
------------------------------------------------

The changes looks fine to me.
+1 to the patch.

> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>         Attachments: HADOOP-6105-1.patch, HADOOP-6105-2.patch, HADOOP-6105-3.patch, HADOOP-6105-4.patch, HADOOP-6105.patch, HADOOP-6105.patch
>
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-846-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 18 05:29:20 2009
Return-Path: <common-issues-return-846-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6910 invoked from network); 18 Aug 2009 05:29:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Aug 2009 05:29:20 -0000
Received: (qmail 11658 invoked by uid 500); 18 Aug 2009 05:29:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11618 invoked by uid 500); 18 Aug 2009 05:29:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11608 invoked by uid 99); 18 Aug 2009 05:29:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 05:29:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 05:29:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E5F4829A0014
	for <common-issues@hadoop.apache.org>; Mon, 17 Aug 2009 22:29:14 -0700 (PDT)
Message-ID: <439325976.1250573354940.JavaMail.jira@brutus>
Date: Mon, 17 Aug 2009 22:29:14 -0700 (PDT)
From: "Sreekanth Ramakrishnan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6105) Provide a way to automatically handle
 backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sreekanth Ramakrishnan updated HADOOP-6105:
-------------------------------------------

    Status: Patch Available  (was: Open)

> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>         Attachments: HADOOP-6105-1.patch, HADOOP-6105-2.patch, HADOOP-6105-3.patch, HADOOP-6105-4.patch, HADOOP-6105.patch, HADOOP-6105.patch
>
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-847-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 18 05:47:18 2009
Return-Path: <common-issues-return-847-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16734 invoked from network); 18 Aug 2009 05:47:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Aug 2009 05:47:17 -0000
Received: (qmail 27379 invoked by uid 500); 18 Aug 2009 05:47:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27324 invoked by uid 500); 18 Aug 2009 05:47:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27309 invoked by uid 99); 18 Aug 2009 05:47:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 05:47:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 05:47:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CDADA29A0011
	for <common-issues@hadoop.apache.org>; Mon, 17 Aug 2009 22:47:14 -0700 (PDT)
Message-ID: <702994081.1250574434841.JavaMail.jira@brutus>
Date: Mon, 17 Aug 2009 22:47:14 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6198) FileSystem filtering should work on
 FileStatus rather than Path objects
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

FileSystem filtering should work on FileStatus rather than Path objects
-----------------------------------------------------------------------

                 Key: HADOOP-6198
                 URL: https://issues.apache.org/jira/browse/HADOOP-6198
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Chris Douglas


There's an avoidable overhead in listing/globbing items with some property (e.g. owned by user foo, only files, files larger than _n_ bytes, etc.). Internally, the Path is extracted from a FileStatus object and passed to the PathFilter; simply passing the FileStatus object would allow one to filter on the information in the status object.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-848-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 18 06:15:20 2009
Return-Path: <common-issues-return-848-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29929 invoked from network); 18 Aug 2009 06:15:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Aug 2009 06:15:20 -0000
Received: (qmail 56313 invoked by uid 500); 18 Aug 2009 06:15:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56258 invoked by uid 500); 18 Aug 2009 06:15:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56248 invoked by uid 99); 18 Aug 2009 06:15:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 06:15:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 06:15:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E82C729A001A
	for <common-issues@hadoop.apache.org>; Mon, 17 Aug 2009 23:15:14 -0700 (PDT)
Message-ID: <882416560.1250576114949.JavaMail.jira@brutus>
Date: Mon, 17 Aug 2009 23:15:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6198) FileSystem filtering should work on
 FileStatus rather than Path objects
In-Reply-To: <702994081.1250574434841.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744384#action_12744384 ] 

Hong Tang commented on HADOOP-6198:
-----------------------------------

I'd argue that we may want to support both interfaces. The fact that the current path-based filtering retrieves a FileStatus object first is an implementation detail and it is conceivable that a different FS implementation may be more efficient to support path-based filtering than file-status based filtering.

> FileSystem filtering should work on FileStatus rather than Path objects
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-6198
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6198
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chris Douglas
>
> There's an avoidable overhead in listing/globbing items with some property (e.g. owned by user foo, only files, files larger than _n_ bytes, etc.). Internally, the Path is extracted from a FileStatus object and passed to the PathFilter; simply passing the FileStatus object would allow one to filter on the information in the status object.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-849-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 18 07:19:18 2009
Return-Path: <common-issues-return-849-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57905 invoked from network); 18 Aug 2009 07:19:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Aug 2009 07:19:18 -0000
Received: (qmail 46563 invoked by uid 500); 18 Aug 2009 07:19:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46538 invoked by uid 500); 18 Aug 2009 07:19:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46274 invoked by uid 99); 18 Aug 2009 07:19:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 07:19:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 07:19:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D948929A0013
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 00:19:14 -0700 (PDT)
Message-ID: <40586295.1250579954889.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 00:19:14 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6198) FileSystem filtering should work on
 FileStatus rather than Path objects
In-Reply-To: <702994081.1250574434841.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744392#action_12744392 ] 

Chris Douglas commented on HADOOP-6198:
---------------------------------------

Granted, but supporting two filtering interfaces will likely cause maintenance headaches and be more of a burden to a FS implementor; I'd rather pick an API and not support all possible variants. If a FileSystem is more efficient with path-based filtering, it can still work with FileStatus objects, either populating them lazily, filling them with defaults (what many shims do anyway), or even failing if a user queries unsupported data. Since globStatus returns FileStatus objects, any implementation will need to construct them for the set of accepted Paths, anyway. Given that the API seems biased toward FileStatus objects, I'd rather endure a penalty for the hypothetical FS that doesn't return this information, rather than maintain two separate filtering APIs.

> FileSystem filtering should work on FileStatus rather than Path objects
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-6198
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6198
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chris Douglas
>
> There's an avoidable overhead in listing/globbing items with some property (e.g. owned by user foo, only files, files larger than _n_ bytes, etc.). Internally, the Path is extracted from a FileStatus object and passed to the PathFilter; simply passing the FileStatus object would allow one to filter on the information in the status object.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-850-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 18 07:29:17 2009
Return-Path: <common-issues-return-850-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60786 invoked from network); 18 Aug 2009 07:29:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Aug 2009 07:29:17 -0000
Received: (qmail 58367 invoked by uid 500); 18 Aug 2009 07:29:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58314 invoked by uid 500); 18 Aug 2009 07:29:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58304 invoked by uid 99); 18 Aug 2009 07:29:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 07:29:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 07:29:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C8493234C044
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 00:29:14 -0700 (PDT)
Message-ID: <1114891432.1250580554808.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 00:29:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6198) FileSystem filtering should work on
 FileStatus rather than Path objects
In-Reply-To: <702994081.1250574434841.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744396#action_12744396 ] 

Hong Tang commented on HADOOP-6198:
-----------------------------------

I am not sure why it would be a burden for FS implementors. listStatus would be implemented by FileSystem and FS implementors should only override them when the default implementation is less efficient (and the implementors have the incentive of making them fast).

> FileSystem filtering should work on FileStatus rather than Path objects
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-6198
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6198
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chris Douglas
>
> There's an avoidable overhead in listing/globbing items with some property (e.g. owned by user foo, only files, files larger than _n_ bytes, etc.). Internally, the Path is extracted from a FileStatus object and passed to the PathFilter; simply passing the FileStatus object would allow one to filter on the information in the status object.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-851-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 18 10:16:33 2009
Return-Path: <common-issues-return-851-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19667 invoked from network); 18 Aug 2009 10:16:30 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Aug 2009 10:16:30 -0000
Received: (qmail 10025 invoked by uid 500); 18 Aug 2009 10:16:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9966 invoked by uid 500); 18 Aug 2009 10:16:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9828 invoked by uid 99); 18 Aug 2009 10:16:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 10:16:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 10:16:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0D21F234C1EB
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 03:16:15 -0700 (PDT)
Message-ID: <1169153491.1250590575052.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 03:16:15 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6105) Provide a way to automatically
 handle backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744444#action_12744444 ] 

Hadoop QA commented on HADOOP-6105:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12416391/HADOOP-6105-4.patch
  against trunk revision 804918.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs.  The patch appears to introduce 1 new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/609/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/609/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/609/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/609/console

This message is automatically generated.

> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>         Attachments: HADOOP-6105-1.patch, HADOOP-6105-2.patch, HADOOP-6105-3.patch, HADOOP-6105-4.patch, HADOOP-6105.patch, HADOOP-6105.patch
>
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-852-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 18 10:56:18 2009
Return-Path: <common-issues-return-852-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43192 invoked from network); 18 Aug 2009 10:56:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Aug 2009 10:56:18 -0000
Received: (qmail 71297 invoked by uid 500); 18 Aug 2009 10:56:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71258 invoked by uid 500); 18 Aug 2009 10:56:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71248 invoked by uid 99); 18 Aug 2009 10:56:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 10:56:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 10:56:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 07537234C044
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 03:56:15 -0700 (PDT)
Message-ID: <795421522.1250592975017.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 03:56:15 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6105) Provide a way to automatically
 handle backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna reassigned HADOOP-6105:
---------------------------------------------

    Assignee: V.V.Chaitanya Krishna

> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>            Assignee: V.V.Chaitanya Krishna
>         Attachments: HADOOP-6105-1.patch, HADOOP-6105-2.patch, HADOOP-6105-3.patch, HADOOP-6105-4.patch, HADOOP-6105.patch, HADOOP-6105.patch
>
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-853-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 18 11:02:20 2009
Return-Path: <common-issues-return-853-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44827 invoked from network); 18 Aug 2009 11:02:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Aug 2009 11:02:20 -0000
Received: (qmail 74490 invoked by uid 500); 18 Aug 2009 11:02:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74432 invoked by uid 500); 18 Aug 2009 11:02:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74422 invoked by uid 99); 18 Aug 2009 11:02:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 11:02:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 11:02:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C7DF9234C044
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 04:02:14 -0700 (PDT)
Message-ID: <328849891.1250593334813.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 04:02:14 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6198) FileSystem filtering should work on
 FileStatus rather than Path objects
In-Reply-To: <702994081.1250574434841.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744454#action_12744454 ] 

Chris Douglas commented on HADOOP-6198:
---------------------------------------

Is the proposal to retain two versions of globStatus, one taking a PathFilter and another taking e.g. StatusFilter? This seems like a premature optimization, given that we have no example of a FileSystem for which there is a significant performance difference, or any performance improvement. If one wants to implement a FileSystem that only supports filtering by Path, or that optimizes that particular call, then there are plenty of other ways to effect this without adding a separate method to the core API. Your point on possible performance gain is taken, but it doesn't seem like the right tradeoff, particularly without a concrete use case.

We'll have to keep PathFilter around for at least another release for backwards-compatibility, but I'd like to deprecate it.

> FileSystem filtering should work on FileStatus rather than Path objects
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-6198
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6198
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chris Douglas
>
> There's an avoidable overhead in listing/globbing items with some property (e.g. owned by user foo, only files, files larger than _n_ bytes, etc.). Internally, the Path is extracted from a FileStatus object and passed to the PathFilter; simply passing the FileStatus object would allow one to filter on the information in the status object.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-854-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 18 11:04:20 2009
Return-Path: <common-issues-return-854-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45206 invoked from network); 18 Aug 2009 11:04:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Aug 2009 11:04:20 -0000
Received: (qmail 75532 invoked by uid 500); 18 Aug 2009 11:04:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75488 invoked by uid 500); 18 Aug 2009 11:04:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75478 invoked by uid 99); 18 Aug 2009 11:04:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 11:04:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 11:04:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E57C7234C044
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 04:04:14 -0700 (PDT)
Message-ID: <1208158771.1250593454882.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 04:04:14 -0700 (PDT)
From: "Sreekanth Ramakrishnan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6105) Provide a way to automatically handle
 backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sreekanth Ramakrishnan updated HADOOP-6105:
-------------------------------------------

    Status: Open  (was: Patch Available)

> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>            Assignee: V.V.Chaitanya Krishna
>         Attachments: HADOOP-6105-1.patch, HADOOP-6105-2.patch, HADOOP-6105-3.patch, HADOOP-6105-4.patch, HADOOP-6105.patch, HADOOP-6105.patch
>
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-855-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 18 11:06:19 2009
Return-Path: <common-issues-return-855-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46694 invoked from network); 18 Aug 2009 11:06:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Aug 2009 11:06:18 -0000
Received: (qmail 79932 invoked by uid 500); 18 Aug 2009 11:06:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79883 invoked by uid 500); 18 Aug 2009 11:06:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79838 invoked by uid 99); 18 Aug 2009 11:06:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 11:06:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 11:06:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D54B9234C044
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 04:06:14 -0700 (PDT)
Message-ID: <638296700.1250593574859.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 04:06:14 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6105) Provide a way to automatically handle
 backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna updated HADOOP-6105:
------------------------------------------

    Attachment: HADOOP-6105-5.patch

uploading the patch with the findbugs warning fixed.

> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>            Assignee: V.V.Chaitanya Krishna
>         Attachments: HADOOP-6105-1.patch, HADOOP-6105-2.patch, HADOOP-6105-3.patch, HADOOP-6105-4.patch, HADOOP-6105-5.patch, HADOOP-6105.patch, HADOOP-6105.patch
>
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-856-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 18 11:20:17 2009
Return-Path: <common-issues-return-856-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54855 invoked from network); 18 Aug 2009 11:20:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Aug 2009 11:20:17 -0000
Received: (qmail 97395 invoked by uid 500); 18 Aug 2009 11:20:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97343 invoked by uid 500); 18 Aug 2009 11:20:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97333 invoked by uid 99); 18 Aug 2009 11:20:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 11:20:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 11:20:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E256E234C04B
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 04:20:14 -0700 (PDT)
Message-ID: <1428512061.1250594414926.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 04:20:14 -0700 (PDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6199) Add the documentation for
 io.map.index.skip in core-default
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Add the documentation for io.map.index.skip in core-default
-----------------------------------------------------------

                 Key: HADOOP-6199
                 URL: https://issues.apache.org/jira/browse/HADOOP-6199
             Project: Hadoop Common
          Issue Type: Bug
          Components: io
            Reporter: Amareshwari Sriramadasu
            Assignee: Amareshwari Sriramadasu
             Fix For: 0.21.0


io.map.index.skip is used in common. But is documented in MapReduce project. It should be documented in core-default.xml

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-857-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 18 11:22:17 2009
Return-Path: <common-issues-return-857-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55260 invoked from network); 18 Aug 2009 11:22:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Aug 2009 11:22:17 -0000
Received: (qmail 98731 invoked by uid 500); 18 Aug 2009 11:22:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98676 invoked by uid 500); 18 Aug 2009 11:22:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98666 invoked by uid 99); 18 Aug 2009 11:22:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 11:22:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 11:22:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DF3D4234C1EB
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 04:22:14 -0700 (PDT)
Message-ID: <1381327102.1250594534913.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 04:22:14 -0700 (PDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6199) Add the documentation for
 io.map.index.skip in core-default
In-Reply-To: <1428512061.1250594414926.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amareshwari Sriramadasu updated HADOOP-6199:
--------------------------------------------

    Status: Patch Available  (was: Open)

> Add the documentation for io.map.index.skip in core-default
> -----------------------------------------------------------
>
>                 Key: HADOOP-6199
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6199
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.21.0
>
>         Attachments: patch-6199.txt
>
>
> io.map.index.skip is used in common. But is documented in MapReduce project. It should be documented in core-default.xml

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-858-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 18 11:22:20 2009
Return-Path: <common-issues-return-858-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55333 invoked from network); 18 Aug 2009 11:22:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Aug 2009 11:22:20 -0000
Received: (qmail 98830 invoked by uid 500); 18 Aug 2009 11:22:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98774 invoked by uid 500); 18 Aug 2009 11:22:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98760 invoked by uid 99); 18 Aug 2009 11:22:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 11:22:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 11:22:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DA633234C04C
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 04:22:14 -0700 (PDT)
Message-ID: <1906108843.1250594534893.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 04:22:14 -0700 (PDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6199) Add the documentation for
 io.map.index.skip in core-default
In-Reply-To: <1428512061.1250594414926.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amareshwari Sriramadasu updated HADOOP-6199:
--------------------------------------------

    Attachment: patch-6199.txt

Patch adding the property.

> Add the documentation for io.map.index.skip in core-default
> -----------------------------------------------------------
>
>                 Key: HADOOP-6199
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6199
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.21.0
>
>         Attachments: patch-6199.txt
>
>
> io.map.index.skip is used in common. But is documented in MapReduce project. It should be documented in core-default.xml

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-859-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 18 14:13:01 2009
Return-Path: <common-issues-return-859-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25006 invoked from network); 18 Aug 2009 14:12:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Aug 2009 14:12:49 -0000
Received: (qmail 28338 invoked by uid 500); 18 Aug 2009 14:12:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28313 invoked by uid 500); 18 Aug 2009 14:12:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28287 invoked by uid 99); 18 Aug 2009 14:12:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 14:12:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 14:12:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E6393234C044
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 07:12:14 -0700 (PDT)
Message-ID: <1116261706.1250604734928.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 07:12:14 -0700 (PDT)
From: "Sreekanth Ramakrishnan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4490) Map and Reduce tasks should run as
 the user who submitted the job
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sreekanth Ramakrishnan updated HADOOP-4490:
-------------------------------------------

    Attachment: 4490-ydist.patch

Attaching new Yahoo! Distribution patch.

> Map and Reduce tasks should run as the user who submitted the job
> -----------------------------------------------------------------
>
>                 Key: HADOOP-4490
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4490
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>            Reporter: Arun C Murthy
>            Assignee: Hemanth Yamijala
>             Fix For: 0.21.0
>
>         Attachments: 4490-ydist.patch, cluster_setup.pdf, cluster_setup.pdf, HADOOP-4490-1.patch, HADOOP-4490-1.patch, hadoop-4490-10.patch, hadoop-4490-11.patch, hadoop-4490-12.patch, hadoop-4490-13.patch, hadoop-4490-14.patch, HADOOP-4490-2.patch, HADOOP-4490-3.patch, hadoop-4490-4.patch, hadoop-4490-5.patch, hadoop-4490-6.patch, hadoop-4490-7.patch, hadoop-4490-8.patch, hadoop-4490-9.patch, hadoop-4490-br20-3.2.patch, hadoop-4490-br20-3.patch, hadoop-4490-design.pdf, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490.patch, HADOOP-4490_streaming.patch
>
>
> Currently the TaskTracker spawns the map/reduce tasks, resulting in them running as the user who started the TaskTracker.
> For security and accounting purposes the tasks should be run as the job-owner.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-860-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 18 15:01:21 2009
Return-Path: <common-issues-return-860-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59550 invoked from network); 18 Aug 2009 15:01:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Aug 2009 15:01:21 -0000
Received: (qmail 68596 invoked by uid 500); 18 Aug 2009 15:01:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68559 invoked by uid 500); 18 Aug 2009 15:01:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68486 invoked by uid 99); 18 Aug 2009 15:01:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 15:01:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 15:01:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 51041234C4A7
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 08:01:15 -0700 (PDT)
Message-ID: <1254610124.1250607675330.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 08:01:15 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6199) Add the documentation for
 io.map.index.skip in core-default
In-Reply-To: <1428512061.1250594414926.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744536#action_12744536 ] 

Hadoop QA commented on HADOOP-6199:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12416863/patch-6199.txt
  against trunk revision 804918.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/610/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/610/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/610/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/610/console

This message is automatically generated.

> Add the documentation for io.map.index.skip in core-default
> -----------------------------------------------------------
>
>                 Key: HADOOP-6199
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6199
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.21.0
>
>         Attachments: patch-6199.txt
>
>
> io.map.index.skip is used in common. But is documented in MapReduce project. It should be documented in core-default.xml

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-861-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 18 15:29:34 2009
Return-Path: <common-issues-return-861-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18135 invoked from network); 18 Aug 2009 15:29:34 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Aug 2009 15:29:34 -0000
Received: (qmail 24721 invoked by uid 500); 18 Aug 2009 15:29:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24580 invoked by uid 500); 18 Aug 2009 15:29:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24556 invoked by uid 99); 18 Aug 2009 15:29:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 15:29:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 15:29:49 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 7FBA6234C1ED
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 08:29:29 -0700 (PDT)
Message-ID: <68853708.1250609369522.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 08:29:29 -0700 (PDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6200) glue together the different projects
 to make local builds easier
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

glue together the different projects to make local builds easier
----------------------------------------------------------------

                 Key: HADOOP-6200
                 URL: https://issues.apache.org/jira/browse/HADOOP-6200
             Project: Hadoop Common
          Issue Type: Bug
          Components: build
    Affects Versions: 0.21.0
            Reporter: Steve Loughran
            Priority: Minor


It's currently fairly tricky to get everything building/up to date in a single machine. We can and should improve this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-862-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 18 15:33:20 2009
Return-Path: <common-issues-return-862-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24729 invoked from network); 18 Aug 2009 15:33:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Aug 2009 15:33:20 -0000
Received: (qmail 31985 invoked by uid 500); 18 Aug 2009 15:33:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31929 invoked by uid 500); 18 Aug 2009 15:33:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31919 invoked by uid 99); 18 Aug 2009 15:33:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 15:33:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 15:33:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C6557234C044
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 08:33:14 -0700 (PDT)
Message-ID: <1622899296.1250609594807.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 08:33:14 -0700 (PDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6200) glue together the different
 projects to make local builds easier
In-Reply-To: <68853708.1250609369522.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744556#action_12744556 ] 

Steve Loughran commented on HADOOP-6200:
----------------------------------------

# create a wiki page documenting current best practises, symlink tricks, etc.
# add the option of using ivy to retrieve artifacts, by patching up the ivy.xml files
# deal with any cycles in the dependencies, so allowing ivy to order multiple builds (this is tricky because of testing, we can do some tricks here)
# provide an optional main build file (which repository?) to build everything in the right order


> glue together the different projects to make local builds easier
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6200
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6200
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Priority: Minor
>
> It's currently fairly tricky to get everything building/up to date in a single machine. We can and should improve this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-863-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 18 16:27:25 2009
Return-Path: <common-issues-return-863-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44711 invoked from network); 18 Aug 2009 16:27:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Aug 2009 16:27:25 -0000
Received: (qmail 50430 invoked by uid 500); 18 Aug 2009 16:27:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50390 invoked by uid 500); 18 Aug 2009 16:27:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50380 invoked by uid 99); 18 Aug 2009 16:27:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 16:27:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 16:27:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 09D2429A0015
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 09:27:15 -0700 (PDT)
Message-ID: <1011347012.1250612835039.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 09:27:15 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6197) Use aspects to introduce new
 methods required for testing
In-Reply-To: <1324647998.1250542274830.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744587#action_12744587 ] 

Konstantin Boudnik commented on HADOOP-6197:
--------------------------------------------

Not yet. I'll link this JIRA as soon as I'll have a new one for fault injection 
framework's porting.


> Use aspects to introduce new methods required for testing
> ---------------------------------------------------------
>
>                 Key: HADOOP-6197
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6197
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: security, test
>            Reporter: Konstantin Boudnik
>         Attachments: HADOOP-6197.patch, HADOOP-6197.sh
>
>
> HADOOP-6176 introduces two new methods which are essentially required for testing purposes. However, this bring a permanent modifications to the production code.
> The same result can be reached by using aspect development. Thus the modifications will be brought in by byte code instrumentation and will exist only during the testing cycles.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-864-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 18 17:26:20 2009
Return-Path: <common-issues-return-864-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75996 invoked from network); 18 Aug 2009 17:26:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Aug 2009 17:26:20 -0000
Received: (qmail 50247 invoked by uid 500); 18 Aug 2009 17:26:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50210 invoked by uid 500); 18 Aug 2009 17:26:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50200 invoked by uid 99); 18 Aug 2009 17:26:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 17:26:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 17:26:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C6257234C044
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 10:26:14 -0700 (PDT)
Message-ID: <123071649.1250616374797.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 10:26:14 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6200) glue together the different
 projects to make local builds easier
In-Reply-To: <68853708.1250609369522.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744610#action_12744610 ] 

Todd Lipcon commented on HADOOP-6200:
-------------------------------------

bq. deal with any cycles in the dependencies, so allowing ivy to order multiple builds (this is tricky because of testing, we can do some tricks here)

Where might we have cyclic dependencies? It seems to me, if we have such, they should be hunted down and destroyed mercilessly rather than worked around in the build process, right?

> glue together the different projects to make local builds easier
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6200
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6200
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Priority: Minor
>
> It's currently fairly tricky to get everything building/up to date in a single machine. We can and should improve this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-865-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 18 17:28:20 2009
Return-Path: <common-issues-return-865-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77135 invoked from network); 18 Aug 2009 17:28:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Aug 2009 17:28:20 -0000
Received: (qmail 50991 invoked by uid 500); 18 Aug 2009 17:28:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50944 invoked by uid 500); 18 Aug 2009 17:28:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50930 invoked by uid 99); 18 Aug 2009 17:28:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 17:28:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 17:28:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C8CBB234C044
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 10:28:14 -0700 (PDT)
Message-ID: <727809880.1250616494807.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 10:28:14 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6198) FileSystem filtering should work on
 FileStatus rather than Path objects
In-Reply-To: <702994081.1250574434841.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744611#action_12744611 ] 

Doug Cutting commented on HADOOP-6198:
--------------------------------------

> We'll have to keep PathFilter around for at least another release for backwards-compatibility, but I'd like to deprecate it.

+1 We've generally tried to switch FileSystem APIs from using Path to FileStatus whereever possible, deprecating and eventually removing Path-based methods, and this just seems like a case we've missed.

> FileSystem filtering should work on FileStatus rather than Path objects
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-6198
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6198
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Chris Douglas
>
> There's an avoidable overhead in listing/globbing items with some property (e.g. owned by user foo, only files, files larger than _n_ bytes, etc.). Internally, the Path is extracted from a FileStatus object and passed to the PathFilter; simply passing the FileStatus object would allow one to filter on the information in the status object.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-866-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 18 18:20:17 2009
Return-Path: <common-issues-return-866-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11578 invoked from network); 18 Aug 2009 18:20:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Aug 2009 18:20:17 -0000
Received: (qmail 37895 invoked by uid 500); 18 Aug 2009 18:20:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37836 invoked by uid 500); 18 Aug 2009 18:20:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37826 invoked by uid 99); 18 Aug 2009 18:20:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 18:20:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 18:20:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E504329A0013
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 11:20:14 -0700 (PDT)
Message-ID: <1093368448.1250619614937.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 11:20:14 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6201) FileSystem::ListStatus should throw
 FileNotFoundException
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

FileSystem::ListStatus should throw FileNotFoundException
---------------------------------------------------------

                 Key: HADOOP-6201
                 URL: https://issues.apache.org/jira/browse/HADOOP-6201
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs
            Reporter: Jakob Homan


As discussed in HDFS-538, it would be better for FileSystem and its implementations to throw FileNotFoundException when the file is not found, rather than returning null.  This will bring it in line with getFileStatus and, I expect, default user expectations.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-867-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 18 18:58:17 2009
Return-Path: <common-issues-return-867-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28023 invoked from network); 18 Aug 2009 18:58:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Aug 2009 18:58:17 -0000
Received: (qmail 8977 invoked by uid 500); 18 Aug 2009 18:58:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8930 invoked by uid 500); 18 Aug 2009 18:58:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8920 invoked by uid 99); 18 Aug 2009 18:58:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 18:58:36 +0000
X-ASF-Spam-Status: No, hits=-1998.5 required=10.0
	tests=ALL_TRUSTED,WEIRD_PORT
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 18:58:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E51E929A0012
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 11:58:14 -0700 (PDT)
Message-ID: <780429711.1250621894937.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 11:58:14 -0700 (PDT)
From: "Koji Noguchi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6202) harchive:   Har doesn't work on files
 having '%' character
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

harchive:   Har doesn't work on files having '%' character
----------------------------------------------------------

                 Key: HADOOP-6202
                 URL: https://issues.apache.org/jira/browse/HADOOP-6202
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
            Reporter: Koji Noguchi


If I have a harchive file test.har that contain 
/a/b <dir> 
/a/b/abc%cde <file>
/a/b/fgh <file>

{noformat}
$ hadoop dfs -cat test.har/_masterindex
1 
0 2046275926 0 244 
$ hadoop dfs -cat test.har/_index
/ dir none 0 0 user 
/user dir none 0 0 knoguchi 
/user/knoguchi/a/b dir none 0 0 abc%cde fgh 
/user/knoguchi dir none 0 0 a 
/user/knoguchi/a dir none 0 0 b 
/user/knoguchi/a/b/fgh file part-0 8 10 
/user/knoguchi/a/b/abc%cde file part-0 0 8 
$ hadoop dfs -lsr har:///user/knoguchi/test.har/
drw-r--r--   - knoguchi users          0 2009-08-18 18:52 /user/knoguchi/test.har/user
drw-r--r--   - knoguchi users          0 2009-08-18 18:52 /user/knoguchi/test.har/user/knoguchi
drw-r--r--   - knoguchi users          0 2009-08-18 18:52 /user/knoguchi/test.har/user/knoguchi/a
drw-r--r--   - knoguchi users          0 2009-08-18 18:52 /user/knoguchi/test.har/user/knoguchi/a/b
lsr: could not get get listing for 'har:/user/knoguchi/test.har/user/knoguchi/a/b' : File: har://hdfs-mithrilgold-nn1.gold.ygrid.yahoo.com:8020/user/knoguchi/test.har/user/knoguchi/a/b/abc%cde does not exist in har:///user/knoguchi/test.har
$
{noformat}


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-868-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 18 19:26:26 2009
Return-Path: <common-issues-return-868-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36532 invoked from network); 18 Aug 2009 19:26:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Aug 2009 19:26:26 -0000
Received: (qmail 53315 invoked by uid 500); 18 Aug 2009 19:26:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53272 invoked by uid 500); 18 Aug 2009 19:26:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53262 invoked by uid 99); 18 Aug 2009 19:26:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 19:26:45 +0000
X-ASF-Spam-Status: No, hits=-1998.5 required=10.0
	tests=ALL_TRUSTED,WEIRD_PORT
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 19:26:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C9D5B234C044
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 12:26:14 -0700 (PDT)
Message-ID: <1590493199.1250623574812.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 12:26:14 -0700 (PDT)
From: "Koji Noguchi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6202) harchive:   Har doesn't work on files
 having '%' character
In-Reply-To: <780429711.1250621894937.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Koji Noguchi updated HADOOP-6202:
---------------------------------

    Description: 
If I have a harchive file test.har that contain 
/a/b <dir> 
/a/b/abc%cde <file>
/a/b/fgh <file>

{noformat}
$ hadoop dfs -cat test.har/_masterindex
1 
0 2046275926 0 244 
$ hadoop dfs -cat test.har/_index
/ dir none 0 0 user 
/user dir none 0 0 knoguchi 
/user/knoguchi/a/b dir none 0 0 abc%cde fgh 
/user/knoguchi dir none 0 0 a 
/user/knoguchi/a dir none 0 0 b 
/user/knoguchi/a/b/fgh file part-0 8 10 
/user/knoguchi/a/b/abc%cde file part-0 0 8 
$ hadoop dfs -lsr har:///user/knoguchi/test.har/
drw-r--r--   - knoguchi users          0 2009-08-18 18:52 /user/knoguchi/test.har/user
drw-r--r--   - knoguchi users          0 2009-08-18 18:52 /user/knoguchi/test.har/user/knoguchi
drw-r--r--   - knoguchi users          0 2009-08-18 18:52 /user/knoguchi/test.har/user/knoguchi/a
drw-r--r--   - knoguchi users          0 2009-08-18 18:52 /user/knoguchi/test.har/user/knoguchi/a/b
lsr: could not get get listing for 'har:/user/knoguchi/test.har/user/knoguchi/a/b' : File: har://hdfs-aaa.bbb.ccc.com:8020/user/knoguchi/test.har/user/knoguchi/a/b/abc%cde does not exist in har:///user/knoguchi/test.har
$
{noformat}


  was:
If I have a harchive file test.har that contain 
/a/b <dir> 
/a/b/abc%cde <file>
/a/b/fgh <file>

{noformat}
$ hadoop dfs -cat test.har/_masterindex
1 
0 2046275926 0 244 
$ hadoop dfs -cat test.har/_index
/ dir none 0 0 user 
/user dir none 0 0 knoguchi 
/user/knoguchi/a/b dir none 0 0 abc%cde fgh 
/user/knoguchi dir none 0 0 a 
/user/knoguchi/a dir none 0 0 b 
/user/knoguchi/a/b/fgh file part-0 8 10 
/user/knoguchi/a/b/abc%cde file part-0 0 8 
$ hadoop dfs -lsr har:///user/knoguchi/test.har/
drw-r--r--   - knoguchi users          0 2009-08-18 18:52 /user/knoguchi/test.har/user
drw-r--r--   - knoguchi users          0 2009-08-18 18:52 /user/knoguchi/test.har/user/knoguchi
drw-r--r--   - knoguchi users          0 2009-08-18 18:52 /user/knoguchi/test.har/user/knoguchi/a
drw-r--r--   - knoguchi users          0 2009-08-18 18:52 /user/knoguchi/test.har/user/knoguchi/a/b
lsr: could not get get listing for 'har:/user/knoguchi/test.har/user/knoguchi/a/b' : File: har://hdfs-mithrilgold-nn1.gold.ygrid.yahoo.com:8020/user/knoguchi/test.har/user/knoguchi/a/b/abc%cde does not exist in har:///user/knoguchi/test.har
$
{noformat}



> harchive:   Har doesn't work on files having '%' character
> ----------------------------------------------------------
>
>                 Key: HADOOP-6202
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6202
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Koji Noguchi
>
> If I have a harchive file test.har that contain 
> /a/b <dir> 
> /a/b/abc%cde <file>
> /a/b/fgh <file>
> {noformat}
> $ hadoop dfs -cat test.har/_masterindex
> 1 
> 0 2046275926 0 244 
> $ hadoop dfs -cat test.har/_index
> / dir none 0 0 user 
> /user dir none 0 0 knoguchi 
> /user/knoguchi/a/b dir none 0 0 abc%cde fgh 
> /user/knoguchi dir none 0 0 a 
> /user/knoguchi/a dir none 0 0 b 
> /user/knoguchi/a/b/fgh file part-0 8 10 
> /user/knoguchi/a/b/abc%cde file part-0 0 8 
> $ hadoop dfs -lsr har:///user/knoguchi/test.har/
> drw-r--r--   - knoguchi users          0 2009-08-18 18:52 /user/knoguchi/test.har/user
> drw-r--r--   - knoguchi users          0 2009-08-18 18:52 /user/knoguchi/test.har/user/knoguchi
> drw-r--r--   - knoguchi users          0 2009-08-18 18:52 /user/knoguchi/test.har/user/knoguchi/a
> drw-r--r--   - knoguchi users          0 2009-08-18 18:52 /user/knoguchi/test.har/user/knoguchi/a/b
> lsr: could not get get listing for 'har:/user/knoguchi/test.har/user/knoguchi/a/b' : File: har://hdfs-aaa.bbb.ccc.com:8020/user/knoguchi/test.har/user/knoguchi/a/b/abc%cde does not exist in har:///user/knoguchi/test.har
> $
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-869-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 18 19:30:20 2009
Return-Path: <common-issues-return-869-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37690 invoked from network); 18 Aug 2009 19:30:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Aug 2009 19:30:20 -0000
Received: (qmail 65219 invoked by uid 500); 18 Aug 2009 19:30:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65193 invoked by uid 500); 18 Aug 2009 19:30:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65183 invoked by uid 99); 18 Aug 2009 19:30:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 19:30:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 19:30:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D3B4129A0013
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 12:30:14 -0700 (PDT)
Message-ID: <1489131604.1250623814866.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 12:30:14 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6203) Improve error message when moving to
 trash fails due to quota issue
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Improve error message when moving to trash fails due to quota issue
-------------------------------------------------------------------

                 Key: HADOOP-6203
                 URL: https://issues.apache.org/jira/browse/HADOOP-6203
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs
            Reporter: Jakob Homan


HADOOP-6080 provided an option for deleting files even when overquota, but the error message that's returned in this situation is unhelpful and doesn't suggest skipTrash as a remediation:
{noformat}$ hdfs -rmr /foo/bar/bat/boo

rmr: Failed to move to trash:
hdfs://cluster/foo/bar/bat/boo{noformat}
In this situation, the error message should say there was a quote problem and suggest -skipTrash.


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-870-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 18 20:20:26 2009
Return-Path: <common-issues-return-870-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50974 invoked from network); 18 Aug 2009 20:20:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Aug 2009 20:20:26 -0000
Received: (qmail 52457 invoked by uid 500); 18 Aug 2009 20:20:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52424 invoked by uid 500); 18 Aug 2009 20:20:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52403 invoked by uid 99); 18 Aug 2009 20:20:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 20:20:45 +0000
X-ASF-Spam-Status: No, hits=-1998.5 required=10.0
	tests=ALL_TRUSTED,WEIRD_PORT
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 20:20:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C9120234C044
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 13:20:14 -0700 (PDT)
Message-ID: <73730413.1250626814809.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 13:20:14 -0700 (PDT)
From: "Koji Noguchi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6202) harchive:   Har doesn't work on
 files having '%' character
In-Reply-To: <780429711.1250621894937.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744687#action_12744687 ] 

Koji Noguchi commented on HADOOP-6202:
--------------------------------------

Probably this is happening because
HarFileSystem.makeQualified uses URI.toString() which escapes '%' to '%25'.
So string matching fails (as well as incorrect hash value).

> harchive:   Har doesn't work on files having '%' character
> ----------------------------------------------------------
>
>                 Key: HADOOP-6202
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6202
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Koji Noguchi
>
> If I have a harchive file test.har that contain 
> /a/b <dir> 
> /a/b/abc%cde <file>
> /a/b/fgh <file>
> {noformat}
> $ hadoop dfs -cat test.har/_masterindex
> 1 
> 0 2046275926 0 244 
> $ hadoop dfs -cat test.har/_index
> / dir none 0 0 user 
> /user dir none 0 0 knoguchi 
> /user/knoguchi/a/b dir none 0 0 abc%cde fgh 
> /user/knoguchi dir none 0 0 a 
> /user/knoguchi/a dir none 0 0 b 
> /user/knoguchi/a/b/fgh file part-0 8 10 
> /user/knoguchi/a/b/abc%cde file part-0 0 8 
> $ hadoop dfs -lsr har:///user/knoguchi/test.har/
> drw-r--r--   - knoguchi users          0 2009-08-18 18:52 /user/knoguchi/test.har/user
> drw-r--r--   - knoguchi users          0 2009-08-18 18:52 /user/knoguchi/test.har/user/knoguchi
> drw-r--r--   - knoguchi users          0 2009-08-18 18:52 /user/knoguchi/test.har/user/knoguchi/a
> drw-r--r--   - knoguchi users          0 2009-08-18 18:52 /user/knoguchi/test.har/user/knoguchi/a/b
> lsr: could not get get listing for 'har:/user/knoguchi/test.har/user/knoguchi/a/b' : File: har://hdfs-aaa.bbb.ccc.com:8020/user/knoguchi/test.har/user/knoguchi/a/b/abc%cde does not exist in har:///user/knoguchi/test.har
> $
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-871-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 18 21:03:21 2009
Return-Path: <common-issues-return-871-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64577 invoked from network); 18 Aug 2009 21:03:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Aug 2009 21:03:20 -0000
Received: (qmail 19005 invoked by uid 500); 18 Aug 2009 21:03:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18920 invoked by uid 500); 18 Aug 2009 21:03:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18901 invoked by uid 99); 18 Aug 2009 21:03:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 21:03:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 21:03:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E680429A001D
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 14:03:14 -0700 (PDT)
Message-ID: <1837050058.1250629394943.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 14:03:14 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6204) Implementing aspects development and
 fault injeciton framework for Hadoop
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Implementing aspects development and fault injeciton framework for Hadoop
-------------------------------------------------------------------------

                 Key: HADOOP-6204
                 URL: https://issues.apache.org/jira/browse/HADOOP-6204
             Project: Hadoop Common
          Issue Type: Improvement
          Components: build, test
            Reporter: Konstantin Boudnik
            Assignee: Konstantin Boudnik


Fault injection framework implementation in HDFS (HDFS-435) turns out to be a very useful feature both for error handling testing and for various simulations.

There's certain demand for this framework, thus it need to be pulled up from HDFS and brought into Common, so other sub-projects will be able to share it if needed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-872-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 18 21:21:43 2009
Return-Path: <common-issues-return-872-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80947 invoked from network); 18 Aug 2009 21:21:41 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 18 Aug 2009 21:21:41 -0000
Received: (qmail 61116 invoked by uid 500); 18 Aug 2009 21:21:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60645 invoked by uid 500); 18 Aug 2009 21:21:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60622 invoked by uid 99); 18 Aug 2009 21:21:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 21:21:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2009 21:13:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CE7AC29A0013
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 14:13:14 -0700 (PDT)
Message-ID: <728993166.1250629994844.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 14:13:14 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6204) Implementing aspects development
 and fault injeciton framework for Hadoop
In-Reply-To: <1837050058.1250629394943.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744720#action_12744720 ] 

Konstantin Boudnik commented on HADOOP-6204:
--------------------------------------------

To implement it the fault injection framework will have to be pulled from HDFS and include it into Common project. This can be eventually represented in a form of an auxiliary build file and ivy configuration, which could be used by other subprojects as SNV externals. A couple of framework classes would have to be migrated from HDFS to Common permanently.

As the result, Common will need to provide one more jar file with instrumented classes, i.e. {{hadoop-core-0.21.0-dev-fi.jar}}

Besides, it might be a good time to think of different name for this framework, because it is covering not only fault injection, but rather generic code instrumentation needs.


> Implementing aspects development and fault injeciton framework for Hadoop
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6204
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6204
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>
> Fault injection framework implementation in HDFS (HDFS-435) turns out to be a very useful feature both for error handling testing and for various simulations.
> There's certain demand for this framework, thus it need to be pulled up from HDFS and brought into Common, so other sub-projects will be able to share it if needed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-873-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 19 00:10:18 2009
Return-Path: <common-issues-return-873-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46157 invoked from network); 19 Aug 2009 00:10:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Aug 2009 00:10:18 -0000
Received: (qmail 32645 invoked by uid 500); 19 Aug 2009 00:10:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32589 invoked by uid 500); 19 Aug 2009 00:10:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32579 invoked by uid 99); 19 Aug 2009 00:10:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 00:10:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 00:10:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D49EA234C044
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 17:10:14 -0700 (PDT)
Message-ID: <612977434.1250640614856.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 17:10:14 -0700 (PDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5640) Allow ServicePlugins to hook
 callbacks into key service events
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744791#action_12744791 ] 

Sanjay Radia commented on HADOOP-5640:
--------------------------------------

This Jira was motivated by HADOOP-5257, RPC plugins. I understand the motivation for RPC plugins.
However I fail to see why a RPC plugin needs to be notifed when a NN or DN starts. 
Further Todd says:
>>Do datanodes get notified on that kind of events? That would be the way to solve our problem in HADOOP-4707, when the namenode is restarted.
>Yep, there's a perfect place to insert that, which was the other motivation I forgot to mention for this ticket

Does that mean that such notification events are across the network (DN gets notified when NN restarts)?

If the motivation is general service lifecycle, then is it related to HDFS-326?


> Allow ServicePlugins to hook callbacks into key service events
> --------------------------------------------------------------
>
>                 Key: HADOOP-5640
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5640
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-5640.txt, hadoop-5640.txt, HADOOP-5640.v2.txt, hadoop-5640.v3.txt
>
>
> HADOOP-5257 added the ability for NameNode and DataNode to start and stop ServicePlugin implementations at NN/DN start/stop. However, this is insufficient integration for some common use cases.
> We should add some functionality for Plugins to subscribe to events generated by the service they're plugging into. Some potential hook points are:
> NameNode:
>   - new datanode registered
>   - datanode has died
>   - exception caught
>   - etc?
> DataNode:
>   - startup
>   - initial registration with NN complete (this is important for HADOOP-4707 to sync up datanode.dnRegistration.name with the NN-side registration)
>   - namenode reconnect
>   - some block transfer hooks?
>   - exception caught
> I see two potential routes for implementation:
> 1) We make an enum for the types of hookpoints and have a general function in the ServicePlugin interface. Something like:
> {code:java}
> enum HookPoint {
>   DN_STARTUP,
>   DN_RECEIVED_NEW_BLOCK,
>   DN_CAUGHT_EXCEPTION,
>  ...
> }
> void runHook(HookPoint hp, Object value);
> {code}
> 2) We make classes specific to each "pluggable" as was originally suggested in HADDOP-5257. Something like:
> {code:java}
> class DataNodePlugin {
>   void datanodeStarted() {}
>   void receivedNewBlock(block info, etc) {}
>   void caughtException(Exception e) {}
>   ...
> }
> {code}
> I personally prefer option (2) since we can ensure plugin API compatibility at compile-time, and we avoid an ugly switch statement in a runHook() function.
> Interested to hear what people's thoughts are here.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-874-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 19 05:25:46 2009
Return-Path: <common-issues-return-874-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57941 invoked from network); 19 Aug 2009 05:25:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Aug 2009 05:25:46 -0000
Received: (qmail 79961 invoked by uid 500); 19 Aug 2009 05:26:05 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79901 invoked by uid 500); 19 Aug 2009 05:26:05 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79888 invoked by uid 99); 19 Aug 2009 05:26:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 05:26:05 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 05:25:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E021D29A0011
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 22:25:14 -0700 (PDT)
Message-ID: <80050858.1250659514917.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 22:25:14 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6105) Provide a way to automatically handle
 backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna updated HADOOP-6105:
------------------------------------------

    Status: Patch Available  (was: Open)

> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>            Assignee: V.V.Chaitanya Krishna
>         Attachments: HADOOP-6105-1.patch, HADOOP-6105-2.patch, HADOOP-6105-3.patch, HADOOP-6105-4.patch, HADOOP-6105-5.patch, HADOOP-6105.patch, HADOOP-6105.patch
>
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-875-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 19 05:27:24 2009
Return-Path: <common-issues-return-875-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58821 invoked from network); 19 Aug 2009 05:27:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Aug 2009 05:27:21 -0000
Received: (qmail 81577 invoked by uid 500); 19 Aug 2009 05:27:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81538 invoked by uid 500); 19 Aug 2009 05:27:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81431 invoked by uid 99); 19 Aug 2009 05:27:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 05:27:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 05:27:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0389829A0021
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 22:27:15 -0700 (PDT)
Message-ID: <1425582317.1250659635013.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 22:27:15 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6184) Provide a configuration dump in json
 format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna reassigned HADOOP-6184:
---------------------------------------------

    Assignee: V.V.Chaitanya Krishna

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>            Assignee: V.V.Chaitanya Krishna
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184-2.patch, HADOOP-6184-3.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-876-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 19 05:56:20 2009
Return-Path: <common-issues-return-876-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74220 invoked from network); 19 Aug 2009 05:56:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Aug 2009 05:56:20 -0000
Received: (qmail 27921 invoked by uid 500); 19 Aug 2009 05:56:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27868 invoked by uid 500); 19 Aug 2009 05:56:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27848 invoked by uid 99); 19 Aug 2009 05:56:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 05:56:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 05:56:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E2C6C29A0012
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 22:56:14 -0700 (PDT)
Message-ID: <10653266.1250661374928.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 22:56:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6105) Provide a way to automatically
 handle backward compatibility of deprecated keys
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744906#action_12744906 ] 

Hadoop QA commented on HADOOP-6105:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12416861/HADOOP-6105-5.patch
  against trunk revision 804918.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/611/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/611/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/611/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/611/console

This message is automatically generated.

> Provide a way to automatically handle backward compatibility of deprecated keys
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6105
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6105
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>            Assignee: V.V.Chaitanya Krishna
>         Attachments: HADOOP-6105-1.patch, HADOOP-6105-2.patch, HADOOP-6105-3.patch, HADOOP-6105-4.patch, HADOOP-6105-5.patch, HADOOP-6105.patch, HADOOP-6105.patch
>
>
> There are cases when we have had to deprecate configuration keys. Use cases include, changing the names of variables to better match intent, splitting a single parameter into two - for maps, reduces etc.
> In such cases, we typically provide a backwards compatible option for the old keys. The handling of such cases might typically be common enough to actually add support for it in a generic fashion in the Configuration class. Some initial discussion around this started in HADOOP-5919, but since the project split happened in between we decided to open this issue to fix it in common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-877-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 19 06:28:20 2009
Return-Path: <common-issues-return-877-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98816 invoked from network); 19 Aug 2009 06:28:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Aug 2009 06:28:19 -0000
Received: (qmail 73709 invoked by uid 500); 19 Aug 2009 06:28:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73647 invoked by uid 500); 19 Aug 2009 06:28:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73628 invoked by uid 99); 19 Aug 2009 06:28:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 06:28:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 06:28:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D4963234C044
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 23:28:14 -0700 (PDT)
Message-ID: <1795086204.1250663294856.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 23:28:14 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6184) Provide a configuration dump in json
 format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna updated HADOOP-6184:
------------------------------------------

    Attachment: HADOOP-6184-4.patch

found a findbugs warning in the patch. Uploading a new patch with this warning fixed.

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>            Assignee: V.V.Chaitanya Krishna
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184-2.patch, HADOOP-6184-3.patch, HADOOP-6184-4.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-878-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 19 06:28:20 2009
Return-Path: <common-issues-return-878-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98850 invoked from network); 19 Aug 2009 06:28:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Aug 2009 06:28:20 -0000
Received: (qmail 73789 invoked by uid 500); 19 Aug 2009 06:28:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73724 invoked by uid 500); 19 Aug 2009 06:28:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73639 invoked by uid 99); 19 Aug 2009 06:28:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 06:28:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 06:28:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EC81A29A0016
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 23:28:14 -0700 (PDT)
Message-ID: <1354882199.1250663294967.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 23:28:14 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6184) Provide a configuration dump in json
 format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna updated HADOOP-6184:
------------------------------------------

    Status: Patch Available  (was: Open)

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>            Assignee: V.V.Chaitanya Krishna
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184-2.patch, HADOOP-6184-3.patch, HADOOP-6184-4.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-879-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 19 06:46:18 2009
Return-Path: <common-issues-return-879-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6260 invoked from network); 19 Aug 2009 06:46:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Aug 2009 06:46:17 -0000
Received: (qmail 92933 invoked by uid 500); 19 Aug 2009 06:46:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92875 invoked by uid 500); 19 Aug 2009 06:46:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92865 invoked by uid 99); 19 Aug 2009 06:46:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 06:46:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 06:46:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CF145234C044
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 23:46:14 -0700 (PDT)
Message-ID: <2709699.1250664374827.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 23:46:14 -0700 (PDT)
From: "Scott Beardsley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4675) Current Ganglia metrics
 implementation is incompatible with Ganglia 3.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Scott Beardsley updated HADOOP-4675:
------------------------------------

    Attachment: HADOOP-4675-v8.patch

add support for Ganglia 3.1 metrics

> Current Ganglia metrics implementation is incompatible with Ganglia 3.1
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-4675
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4675
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: metrics
>    Affects Versions: 0.18.2
>            Reporter: Brian Bockelman
>            Assignee: Brian Bockelman
>             Fix For: 0.21.0
>
>         Attachments: hadoop-4675-2.patch, hadoop-4675-3.patch, HADOOP-4675-4.patch, HADOOP-4675-v5.patch, HADOOP-4675-v6.patch, HADOOP-4675-v7.patch, HADOOP-4675-v8.patch, hadoop-4675.patch
>
>
> Ganglia changed its wire protocol in the 3.1.x series; the current implementation only works for 3.0.x.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-880-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 19 06:50:17 2009
Return-Path: <common-issues-return-880-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7306 invoked from network); 19 Aug 2009 06:50:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Aug 2009 06:50:17 -0000
Received: (qmail 99564 invoked by uid 500); 19 Aug 2009 06:50:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99503 invoked by uid 500); 19 Aug 2009 06:50:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99493 invoked by uid 99); 19 Aug 2009 06:50:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 06:50:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 06:50:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D520E234C044
	for <common-issues@hadoop.apache.org>; Tue, 18 Aug 2009 23:50:14 -0700 (PDT)
Message-ID: <487846452.1250664614858.JavaMail.jira@brutus>
Date: Tue, 18 Aug 2009 23:50:14 -0700 (PDT)
From: "Scott Beardsley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4675) Current Ganglia metrics
 implementation is incompatible with Ganglia 3.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Scott Beardsley updated HADOOP-4675:
------------------------------------

                 Tags: ganglia metrics
    Affects Version/s:     (was: 0.18.2)
                       0.21.0
         Release Note: Support for reporting metrics to Ganglia 3.1 servers
               Status: Patch Available  (was: Open)

These minor changes in this[1] patch should fix the findbugs warnings in previous versions. Tested against rev 805676 (current trunk)

[1] https://issues.apache.org/jira/secure/attachment/12416987/HADOOP-4675-v8.patch

--------
     [exec] +1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.


> Current Ganglia metrics implementation is incompatible with Ganglia 3.1
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-4675
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4675
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: metrics
>    Affects Versions: 0.21.0
>            Reporter: Brian Bockelman
>            Assignee: Brian Bockelman
>             Fix For: 0.21.0
>
>         Attachments: hadoop-4675-2.patch, hadoop-4675-3.patch, HADOOP-4675-4.patch, HADOOP-4675-v5.patch, HADOOP-4675-v6.patch, HADOOP-4675-v7.patch, HADOOP-4675-v8.patch, hadoop-4675.patch
>
>
> Ganglia changed its wire protocol in the 3.1.x series; the current implementation only works for 3.0.x.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-881-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 19 09:00:18 2009
Return-Path: <common-issues-return-881-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38827 invoked from network); 19 Aug 2009 09:00:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Aug 2009 09:00:17 -0000
Received: (qmail 1959 invoked by uid 500); 19 Aug 2009 09:00:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1903 invoked by uid 500); 19 Aug 2009 09:00:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1890 invoked by uid 99); 19 Aug 2009 09:00:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 09:00:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 09:00:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D6F81234C04B
	for <common-issues@hadoop.apache.org>; Wed, 19 Aug 2009 02:00:14 -0700 (PDT)
Message-ID: <506359037.1250672414871.JavaMail.jira@brutus>
Date: Wed, 19 Aug 2009 02:00:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6184) Provide a configuration dump in
 json format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744961#action_12744961 ] 

Hadoop QA commented on HADOOP-6184:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12416980/HADOOP-6184-4.patch
  against trunk revision 804918.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/612/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/612/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/612/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/612/console

This message is automatically generated.

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>            Assignee: V.V.Chaitanya Krishna
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184-2.patch, HADOOP-6184-3.patch, HADOOP-6184-4.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-882-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 19 10:22:18 2009
Return-Path: <common-issues-return-882-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78040 invoked from network); 19 Aug 2009 10:22:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Aug 2009 10:22:17 -0000
Received: (qmail 42737 invoked by uid 500); 19 Aug 2009 10:22:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42689 invoked by uid 500); 19 Aug 2009 10:22:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42674 invoked by uid 99); 19 Aug 2009 10:22:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 10:22:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 10:22:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CB77A234C044
	for <common-issues@hadoop.apache.org>; Wed, 19 Aug 2009 03:22:14 -0700 (PDT)
Message-ID: <1064925214.1250677334818.JavaMail.jira@brutus>
Date: Wed, 19 Aug 2009 03:22:14 -0700 (PDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6200) glue together the different
 projects to make local builds easier
In-Reply-To: <68853708.1250609369522.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744981#action_12744981 ] 

Steve Loughran commented on HADOOP-6200:
----------------------------------------

Hadoop-hdfs depends on hadoop-mapreduce for testing, hence, a cycle. 

Right now I am playing tricks with symlinks to hook up the lib directories, so what I build in one dir is automatically picked up by the adjacent project -and documenting what I am doing for the hadoop wiki- but its a bit complex.

options
# flatten : pull out hadoop-hdfs.run-test-hdfs-with-mr bit and move to a subproject that depends on hadoop-hdfs and hadoop-mapreduce
# boostrap via the central repository. Rather than have copies of artifacts in the different bits of SVN, stick some alpha releases of everything up onto the central repository. Then you can use ivy to pull things in, so when I build hdfs the latest version of common gets picked up, and the latest version of mapreduce. If I publish locally, I get the version I ask for, but the default would be to get the last release on the central repo. 

I'm coming round in favour of #2, because it helps us debug the publishing process with, say, a fortnightly alpha release of the artifacts (PMC approval still needed, incidentally), so that when the time comes to do real beta releases, the POMs and such like are stable. 

> glue together the different projects to make local builds easier
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6200
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6200
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Priority: Minor
>
> It's currently fairly tricky to get everything building/up to date in a single machine. We can and should improve this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-883-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 19 11:04:25 2009
Return-Path: <common-issues-return-883-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96094 invoked from network); 19 Aug 2009 11:04:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Aug 2009 11:04:25 -0000
Received: (qmail 1798 invoked by uid 500); 19 Aug 2009 11:04:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1751 invoked by uid 500); 19 Aug 2009 11:04:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1741 invoked by uid 99); 19 Aug 2009 11:04:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 11:04:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 11:04:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 59457234C1EF
	for <common-issues@hadoop.apache.org>; Wed, 19 Aug 2009 04:04:15 -0700 (PDT)
Message-ID: <140938831.1250679855364.JavaMail.jira@brutus>
Date: Wed, 19 Aug 2009 04:04:15 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6173) src/native/packageNativeHadoop.sh
 only packages files with "hadoop" in the name
In-Reply-To: <210252632.1249027454928.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-6173:
------------------------------

    Status: Patch Available  (was: Open)

resubmit patch for hudson

> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6173
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6173
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6174-20090731.patch
>
>
> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name. This becomes too restrictive when a user wants to inject third-party native libraries into his/her own tar build.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-884-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 19 11:04:25 2009
Return-Path: <common-issues-return-884-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96122 invoked from network); 19 Aug 2009 11:04:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Aug 2009 11:04:25 -0000
Received: (qmail 1879 invoked by uid 500); 19 Aug 2009 11:04:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1832 invoked by uid 500); 19 Aug 2009 11:04:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1752 invoked by uid 99); 19 Aug 2009 11:04:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 11:04:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 11:04:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CDC21234C044
	for <common-issues@hadoop.apache.org>; Wed, 19 Aug 2009 04:04:14 -0700 (PDT)
Message-ID: <1525897237.1250679854838.JavaMail.jira@brutus>
Date: Wed, 19 Aug 2009 04:04:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6173) src/native/packageNativeHadoop.sh
 only packages files with "hadoop" in the name
In-Reply-To: <210252632.1249027454928.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-6173:
------------------------------

    Fix Version/s: 0.21.0
           Status: Open  (was: Patch Available)

> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6173
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6173
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6174-20090731.patch
>
>
> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name. This becomes too restrictive when a user wants to inject third-party native libraries into his/her own tar build.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-885-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 19 11:06:26 2009
Return-Path: <common-issues-return-885-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97485 invoked from network); 19 Aug 2009 11:06:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Aug 2009 11:06:26 -0000
Received: (qmail 7877 invoked by uid 500); 19 Aug 2009 11:06:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7830 invoked by uid 500); 19 Aug 2009 11:06:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7819 invoked by uid 99); 19 Aug 2009 11:06:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 11:06:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 11:06:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DC7EF234C1E9
	for <common-issues@hadoop.apache.org>; Wed, 19 Aug 2009 04:06:14 -0700 (PDT)
Message-ID: <237368008.1250679974902.JavaMail.jira@brutus>
Date: Wed, 19 Aug 2009 04:06:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6173) src/native/packageNativeHadoop.sh
 only packages files with "hadoop" in the name
In-Reply-To: <210252632.1249027454928.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744999#action_12744999 ] 

Hong Tang commented on HADOOP-6173:
-----------------------------------

Could we move forward with this jira?

> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6173
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6173
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6174-20090731.patch
>
>
> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name. This becomes too restrictive when a user wants to inject third-party native libraries into his/her own tar build.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-886-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 19 11:48:20 2009
Return-Path: <common-issues-return-886-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14317 invoked from network); 19 Aug 2009 11:48:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Aug 2009 11:48:20 -0000
Received: (qmail 4300 invoked by uid 500); 19 Aug 2009 11:48:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4246 invoked by uid 500); 19 Aug 2009 11:48:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4232 invoked by uid 99); 19 Aug 2009 11:48:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 11:48:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 11:48:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1512F234C044
	for <common-issues@hadoop.apache.org>; Wed, 19 Aug 2009 04:48:15 -0700 (PDT)
Message-ID: <2104083855.1250682495072.JavaMail.jira@brutus>
Date: Wed, 19 Aug 2009 04:48:15 -0700 (PDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-1722) Make streaming to handle non-utf8
 byte array
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745011#action_12745011 ] 

Amareshwari Sriramadasu commented on HADOOP-1722:
-------------------------------------------------

Streaming documentation in forrest should be updated with this feature.

> Make streaming to handle non-utf8 byte array
> --------------------------------------------
>
>                 Key: HADOOP-1722
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1722
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Runping Qi
>            Assignee: Klaas Bosteels
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-1722-branch-0.18.patch, HADOOP-1722-branch-0.19.patch, HADOOP-1722-v2.patch, HADOOP-1722-v3.patch, HADOOP-1722-v4.patch, HADOOP-1722-v4.patch, HADOOP-1722-v5.patch, HADOOP-1722-v6.patch, HADOOP-1722.patch
>
>
> Right now, the streaming framework expects the output sof the steam process (mapper or reducer) are line 
> oriented UTF-8 text. This limit makes it impossible to use those programs whose outputs may be non-UTF-8
>  (international encoding, or maybe even binary data). Streaming can overcome this limit by introducing a simple
> encoding protocol. For example, it can allow the mapper/reducer to hexencode its keys/values, 
> the framework decodes them in the Java side.
> This way, as long as the mapper/reducer executables follow this encoding protocol, 
> they can output arabitary bytearray and the streaming framework can handle them.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-887-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 19 12:08:20 2009
Return-Path: <common-issues-return-887-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19320 invoked from network); 19 Aug 2009 12:08:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Aug 2009 12:08:20 -0000
Received: (qmail 42325 invoked by uid 500); 19 Aug 2009 12:08:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42276 invoked by uid 500); 19 Aug 2009 12:08:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42261 invoked by uid 99); 19 Aug 2009 12:08:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 12:08:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 12:08:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EEB91234C1E9
	for <common-issues@hadoop.apache.org>; Wed, 19 Aug 2009 05:08:14 -0700 (PDT)
Message-ID: <2093288753.1250683694976.JavaMail.jira@brutus>
Date: Wed, 19 Aug 2009 05:08:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4675) Current Ganglia metrics
 implementation is incompatible with Ganglia 3.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745023#action_12745023 ] 

Hadoop QA commented on HADOOP-4675:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12416987/HADOOP-4675-v8.patch
  against trunk revision 804918.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/613/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/613/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/613/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/613/console

This message is automatically generated.

> Current Ganglia metrics implementation is incompatible with Ganglia 3.1
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-4675
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4675
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: metrics
>    Affects Versions: 0.21.0
>            Reporter: Brian Bockelman
>            Assignee: Brian Bockelman
>             Fix For: 0.21.0
>
>         Attachments: hadoop-4675-2.patch, hadoop-4675-3.patch, HADOOP-4675-4.patch, HADOOP-4675-v5.patch, HADOOP-4675-v6.patch, HADOOP-4675-v7.patch, HADOOP-4675-v8.patch, hadoop-4675.patch
>
>
> Ganglia changed its wire protocol in the 3.1.x series; the current implementation only works for 3.0.x.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-888-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 19 12:14:17 2009
Return-Path: <common-issues-return-888-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21029 invoked from network); 19 Aug 2009 12:14:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Aug 2009 12:14:17 -0000
Received: (qmail 49778 invoked by uid 500); 19 Aug 2009 12:14:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49725 invoked by uid 500); 19 Aug 2009 12:14:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49715 invoked by uid 99); 19 Aug 2009 12:14:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 12:14:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 12:14:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C9E5E234C044
	for <common-issues@hadoop.apache.org>; Wed, 19 Aug 2009 05:14:14 -0700 (PDT)
Message-ID: <375012494.1250684054822.JavaMail.jira@brutus>
Date: Wed, 19 Aug 2009 05:14:14 -0700 (PDT)
From: "Vladimir Klimontovich (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6097) Multiple bugs w/ Hadoop archives
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745027#action_12745027 ] 

Vladimir Klimontovich commented on HADOOP-6097:
-----------------------------------------------

The patch contains:

+    /** Hadoop Archive connections cannot be cached by (scheme, authority,
+     * username) -- the path is significant as well. Come to think of it, they
+     * probably don't need to be cached at all, since they wrap another
+     * connection that does the actual networking.
+     */
+    if (scheme.equals("har")) {
+      return createFileSystem(uri, conf);
+    }
+
    return CACHE.get(uri, conf);
  }

I don't think it's a good way. 

I'd like to suggest to introduce property fs.[fs-name].impl.disable.cache. And don't use cache is this property exists and set to true.

> Multiple bugs w/ Hadoop archives
> --------------------------------
>
>                 Key: HADOOP-6097
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6097
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.18.0, 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1, 0.20.0
>            Reporter: Ben Slusky
>             Fix For: 0.20.1
>
>         Attachments: HADOOP-6097.patch
>
>
> Found and fixed several bugs involving Hadoop archives:
> - In makeQualified(), the sloppy conversion from Path to URI and back mangles the path if it contains an escape-worthy character.
> - It's possible that fileStatusInIndex() may have to read more than one segment of the index. The LineReader and count of bytes read need to be reset for each block.
> - har:// connections cannot be indexed by (scheme, authority, username) -- the path is significant as well. Caching them in this way limits a hadoop client to opening one archive per filesystem. It seems to be safe not to cache them, since they wrap another connection that does the actual networking.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-889-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 19 15:59:27 2009
Return-Path: <common-issues-return-889-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41504 invoked from network); 19 Aug 2009 15:59:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Aug 2009 15:59:27 -0000
Received: (qmail 22074 invoked by uid 500); 19 Aug 2009 15:59:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22004 invoked by uid 500); 19 Aug 2009 15:59:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21991 invoked by uid 99); 19 Aug 2009 15:59:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 15:59:44 +0000
X-ASF-Spam-Status: No, hits=-1998.5 required=10.0
	tests=ALL_TRUSTED,WEIRD_PORT
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 15:59:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D44F1234C04B
	for <common-issues@hadoop.apache.org>; Wed, 19 Aug 2009 08:59:14 -0700 (PDT)
Message-ID: <1908560978.1250697554868.JavaMail.jira@brutus>
Date: Wed, 19 Aug 2009 08:59:14 -0700 (PDT)
From: "Koji Noguchi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6202) harchive:   Har doesn't work on
 files having '%' character
In-Reply-To: <780429711.1250621894937.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Koji Noguchi resolved HADOOP-6202.
----------------------------------

    Resolution: Duplicate

This is reported in HADOOP-6097.

> harchive:   Har doesn't work on files having '%' character
> ----------------------------------------------------------
>
>                 Key: HADOOP-6202
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6202
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Koji Noguchi
>
> If I have a harchive file test.har that contain 
> /a/b <dir> 
> /a/b/abc%cde <file>
> /a/b/fgh <file>
> {noformat}
> $ hadoop dfs -cat test.har/_masterindex
> 1 
> 0 2046275926 0 244 
> $ hadoop dfs -cat test.har/_index
> / dir none 0 0 user 
> /user dir none 0 0 knoguchi 
> /user/knoguchi/a/b dir none 0 0 abc%cde fgh 
> /user/knoguchi dir none 0 0 a 
> /user/knoguchi/a dir none 0 0 b 
> /user/knoguchi/a/b/fgh file part-0 8 10 
> /user/knoguchi/a/b/abc%cde file part-0 0 8 
> $ hadoop dfs -lsr har:///user/knoguchi/test.har/
> drw-r--r--   - knoguchi users          0 2009-08-18 18:52 /user/knoguchi/test.har/user
> drw-r--r--   - knoguchi users          0 2009-08-18 18:52 /user/knoguchi/test.har/user/knoguchi
> drw-r--r--   - knoguchi users          0 2009-08-18 18:52 /user/knoguchi/test.har/user/knoguchi/a
> drw-r--r--   - knoguchi users          0 2009-08-18 18:52 /user/knoguchi/test.har/user/knoguchi/a/b
> lsr: could not get get listing for 'har:/user/knoguchi/test.har/user/knoguchi/a/b' : File: har://hdfs-aaa.bbb.ccc.com:8020/user/knoguchi/test.har/user/knoguchi/a/b/abc%cde does not exist in har:///user/knoguchi/test.har
> $
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-890-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 19 16:03:18 2009
Return-Path: <common-issues-return-890-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43316 invoked from network); 19 Aug 2009 16:03:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Aug 2009 16:03:18 -0000
Received: (qmail 29600 invoked by uid 500); 19 Aug 2009 16:03:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29557 invoked by uid 500); 19 Aug 2009 16:03:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29547 invoked by uid 99); 19 Aug 2009 16:03:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 16:03:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 16:03:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C8D71234C044
	for <common-issues@hadoop.apache.org>; Wed, 19 Aug 2009 09:03:14 -0700 (PDT)
Message-ID: <1989431709.1250697794818.JavaMail.jira@brutus>
Date: Wed, 19 Aug 2009 09:03:14 -0700 (PDT)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745095#action_12745095 ] 

Scott Carey commented on HADOOP-6166:
-------------------------------------

Great test results!

The conclusions I draw from these:

* All of these perform well under concurrency, regardless of whether its 4K, 8K, or 16K of lookup tables. 
* General performance trends in a more sophisticated test like yours line up with the simpler Perf test we have, so we can be confident in those results if they are consistent enough run to run.

The 32 bit JVM likes 8_8b, 64 bit 8_8d.  I agree that d is the winner here.

Because the '8' variants shift to one byte at a time if the input is less than 8 bytes, they perform worse than the old PureJavaCrc32 at the 4 byte to 7 byte level.   Is this important?   It would be useful to know how often the crc code is called on small byte chunks.  We can get this to near PureJavaCrc32 speeds for 4 byte sizes if we add a four byte at a time block to 8_8d.   

That is, we can go to something of the form: 

{code}while(len > 7) {
       < 8 at a time code here >
    }
    while(len > 3) {
       < 4 at a time code here>
    }
    while(len > 0) {
       < 1 at a time code here>
    }
{code}

Whether that is important depends on the use cases and frequency of requests  in the 4-7 and 12-15 byte range.  We can even have a block for two-at a time optimization.

> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt, c6166_20090727.patch, c6166_20090728.patch, c6166_20090810.patch, c6166_20090811.patch, graph.r, graph.r, Rplots-laptop.pdf, Rplots-nehalem32.pdf, Rplots-nehalem64.pdf, Rplots.pdf, Rplots.pdf, Rplots.pdf
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-891-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 19 16:07:17 2009
Return-Path: <common-issues-return-891-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45488 invoked from network); 19 Aug 2009 16:07:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Aug 2009 16:07:17 -0000
Received: (qmail 36046 invoked by uid 500); 19 Aug 2009 16:07:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36003 invoked by uid 500); 19 Aug 2009 16:07:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35993 invoked by uid 99); 19 Aug 2009 16:07:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 16:07:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 16:07:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CF160234C044
	for <common-issues@hadoop.apache.org>; Wed, 19 Aug 2009 09:07:14 -0700 (PDT)
Message-ID: <1659457716.1250698034833.JavaMail.jira@brutus>
Date: Wed, 19 Aug 2009 09:07:14 -0700 (PDT)
From: "Koji Noguchi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6097) Multiple bugs w/ Hadoop archives
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745096#action_12745096 ] 

Koji Noguchi commented on HADOOP-6097:
--------------------------------------

Ben, for the caching part, 
HarFileSystem.java has a comment 
{quote}
   * the uri of Har is 
   * har://underlyingfsscheme-host:port/archivepath.
   * or 
   * har:///archivepath.
{quote}

So it already creates a cache for each harpath?
 

> Multiple bugs w/ Hadoop archives
> --------------------------------
>
>                 Key: HADOOP-6097
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6097
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.18.0, 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1, 0.20.0
>            Reporter: Ben Slusky
>             Fix For: 0.20.1
>
>         Attachments: HADOOP-6097.patch
>
>
> Found and fixed several bugs involving Hadoop archives:
> - In makeQualified(), the sloppy conversion from Path to URI and back mangles the path if it contains an escape-worthy character.
> - It's possible that fileStatusInIndex() may have to read more than one segment of the index. The LineReader and count of bytes read need to be reset for each block.
> - har:// connections cannot be indexed by (scheme, authority, username) -- the path is significant as well. Caching them in this way limits a hadoop client to opening one archive per filesystem. It seems to be safe not to cache them, since they wrap another connection that does the actual networking.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-892-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 19 16:15:18 2009
Return-Path: <common-issues-return-892-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49771 invoked from network); 19 Aug 2009 16:15:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Aug 2009 16:15:18 -0000
Received: (qmail 55712 invoked by uid 500); 19 Aug 2009 16:15:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55656 invoked by uid 500); 19 Aug 2009 16:15:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55550 invoked by uid 99); 19 Aug 2009 16:15:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 16:15:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 16:15:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0F098234C1ED
	for <common-issues@hadoop.apache.org>; Wed, 19 Aug 2009 09:15:15 -0700 (PDT)
Message-ID: <1868726835.1250698515060.JavaMail.jira@brutus>
Date: Wed, 19 Aug 2009 09:15:15 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745100#action_12745100 ] 

Todd Lipcon commented on HADOOP-6166:
-------------------------------------

Personally, I'm satisfied at this point :) But, if others feel like squeezing that last ounce out, I'm happy to rerun the tests when you have some new candidates. Just let me know.

Regarding the importance of the different sizes, I think the next step is to instrument the CRC32 to log every invocation to a file and run some single-node wordcounts or something. With a histogram of the results it should be easy to see whether the small-buffer case matters at all in real scenarios.

> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt, c6166_20090727.patch, c6166_20090728.patch, c6166_20090810.patch, c6166_20090811.patch, graph.r, graph.r, Rplots-laptop.pdf, Rplots-nehalem32.pdf, Rplots-nehalem64.pdf, Rplots.pdf, Rplots.pdf, Rplots.pdf
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-893-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 19 16:27:20 2009
Return-Path: <common-issues-return-893-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58515 invoked from network); 19 Aug 2009 16:27:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Aug 2009 16:27:20 -0000
Received: (qmail 89077 invoked by uid 500); 19 Aug 2009 16:27:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89029 invoked by uid 500); 19 Aug 2009 16:27:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88838 invoked by uid 99); 19 Aug 2009 16:27:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 16:27:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 16:27:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D6913234C04B
	for <common-issues@hadoop.apache.org>; Wed, 19 Aug 2009 09:27:14 -0700 (PDT)
Message-ID: <189160968.1250699234877.JavaMail.jira@brutus>
Date: Wed, 19 Aug 2009 09:27:14 -0700 (PDT)
From: "Ben Slusky (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6097) Multiple bugs w/ Hadoop archives
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745106#action_12745106 ] 

Ben Slusky commented on HADOOP-6097:
------------------------------------

Koji, I'm not entirely sure what you're saying. When using the har filesystem, you create a connection for each harpath. This is not a network connection itself, but it wraps a network connection to the underlying filesystem, which is cached. I think this caching is sufficient, and it is not necessary to cache the wrapping har filesystem "connection".

> Multiple bugs w/ Hadoop archives
> --------------------------------
>
>                 Key: HADOOP-6097
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6097
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.18.0, 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1, 0.20.0
>            Reporter: Ben Slusky
>             Fix For: 0.20.1
>
>         Attachments: HADOOP-6097.patch
>
>
> Found and fixed several bugs involving Hadoop archives:
> - In makeQualified(), the sloppy conversion from Path to URI and back mangles the path if it contains an escape-worthy character.
> - It's possible that fileStatusInIndex() may have to read more than one segment of the index. The LineReader and count of bytes read need to be reset for each block.
> - har:// connections cannot be indexed by (scheme, authority, username) -- the path is significant as well. Caching them in this way limits a hadoop client to opening one archive per filesystem. It seems to be safe not to cache them, since they wrap another connection that does the actual networking.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-894-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 19 16:29:20 2009
Return-Path: <common-issues-return-894-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59035 invoked from network); 19 Aug 2009 16:29:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Aug 2009 16:29:19 -0000
Received: (qmail 90932 invoked by uid 500); 19 Aug 2009 16:29:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90880 invoked by uid 500); 19 Aug 2009 16:29:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90851 invoked by uid 99); 19 Aug 2009 16:29:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 16:29:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 16:29:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CBE9E234C044
	for <common-issues@hadoop.apache.org>; Wed, 19 Aug 2009 09:29:14 -0700 (PDT)
Message-ID: <1983473824.1250699354834.JavaMail.jira@brutus>
Date: Wed, 19 Aug 2009 09:29:14 -0700 (PDT)
From: "Ben Slusky (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6097) Multiple bugs w/ Hadoop archives
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745107#action_12745107 ] 

Ben Slusky commented on HADOOP-6097:
------------------------------------

Vladimir, I will implement that and submit a new patch.

> Multiple bugs w/ Hadoop archives
> --------------------------------
>
>                 Key: HADOOP-6097
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6097
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.18.0, 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1, 0.20.0
>            Reporter: Ben Slusky
>             Fix For: 0.20.1
>
>         Attachments: HADOOP-6097.patch
>
>
> Found and fixed several bugs involving Hadoop archives:
> - In makeQualified(), the sloppy conversion from Path to URI and back mangles the path if it contains an escape-worthy character.
> - It's possible that fileStatusInIndex() may have to read more than one segment of the index. The LineReader and count of bytes read need to be reset for each block.
> - har:// connections cannot be indexed by (scheme, authority, username) -- the path is significant as well. Caching them in this way limits a hadoop client to opening one archive per filesystem. It seems to be safe not to cache them, since they wrap another connection that does the actual networking.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-895-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 19 16:53:18 2009
Return-Path: <common-issues-return-895-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72090 invoked from network); 19 Aug 2009 16:53:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Aug 2009 16:53:18 -0000
Received: (qmail 31564 invoked by uid 500); 19 Aug 2009 16:53:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31443 invoked by uid 500); 19 Aug 2009 16:53:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31390 invoked by uid 99); 19 Aug 2009 16:53:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 16:53:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 16:53:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E1B10234C1EF
	for <common-issues@hadoop.apache.org>; Wed, 19 Aug 2009 09:53:14 -0700 (PDT)
Message-ID: <676929675.1250700794923.JavaMail.jira@brutus>
Date: Wed, 19 Aug 2009 09:53:14 -0700 (PDT)
From: "Mahadev konar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6097) Multiple bugs w/ Hadoop archives
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745122#action_12745122 ] 

Mahadev konar commented on HADOOP-6097:
---------------------------------------

ben,
  koji is right. The caching is just a filesystem caching. The filesystem cache has a cache for each scheme cached. So for har filesystem its caching the scheme and harpath to create a cache for a filesystem meaning that a har filesystem is uniquely identified by a har:///archivepath. the connection caching has nothing to do with this filesystem cache.  The connection caching is done via the RPC layer and you cwould not be able to cache connections at the har filesystem layer. 



> Multiple bugs w/ Hadoop archives
> --------------------------------
>
>                 Key: HADOOP-6097
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6097
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.18.0, 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1, 0.20.0
>            Reporter: Ben Slusky
>             Fix For: 0.20.1
>
>         Attachments: HADOOP-6097.patch
>
>
> Found and fixed several bugs involving Hadoop archives:
> - In makeQualified(), the sloppy conversion from Path to URI and back mangles the path if it contains an escape-worthy character.
> - It's possible that fileStatusInIndex() may have to read more than one segment of the index. The LineReader and count of bytes read need to be reset for each block.
> - har:// connections cannot be indexed by (scheme, authority, username) -- the path is significant as well. Caching them in this way limits a hadoop client to opening one archive per filesystem. It seems to be safe not to cache them, since they wrap another connection that does the actual networking.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-896-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 19 17:11:27 2009
Return-Path: <common-issues-return-896-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83175 invoked from network); 19 Aug 2009 17:11:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Aug 2009 17:11:27 -0000
Received: (qmail 83541 invoked by uid 500); 19 Aug 2009 17:11:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83500 invoked by uid 500); 19 Aug 2009 17:11:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83490 invoked by uid 99); 19 Aug 2009 17:11:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 17:11:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 17:11:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DB62E234C1F2
	for <common-issues@hadoop.apache.org>; Wed, 19 Aug 2009 10:11:14 -0700 (PDT)
Message-ID: <1268442602.1250701874897.JavaMail.jira@brutus>
Date: Wed, 19 Aug 2009 10:11:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6173) src/native/packageNativeHadoop.sh
 only packages files with "hadoop" in the name
In-Reply-To: <210252632.1249027454928.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745133#action_12745133 ] 

Hadoop QA commented on HADOOP-6173:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12415106/hadoop-6174-20090731.patch
  against trunk revision 804918.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/614/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/614/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/614/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/614/console

This message is automatically generated.

> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6173
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6173
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6174-20090731.patch
>
>
> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name. This becomes too restrictive when a user wants to inject third-party native libraries into his/her own tar build.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-897-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 19 17:29:18 2009
Return-Path: <common-issues-return-897-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93981 invoked from network); 19 Aug 2009 17:29:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Aug 2009 17:29:17 -0000
Received: (qmail 16497 invoked by uid 500); 19 Aug 2009 17:29:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16438 invoked by uid 500); 19 Aug 2009 17:29:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16345 invoked by uid 99); 19 Aug 2009 17:29:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 17:29:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 17:29:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CD790234C04B
	for <common-issues@hadoop.apache.org>; Wed, 19 Aug 2009 10:29:14 -0700 (PDT)
Message-ID: <492569326.1250702954840.JavaMail.jira@brutus>
Date: Wed, 19 Aug 2009 10:29:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745143#action_12745143 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6166:
------------------------------------------------

> Because the '8' variants shift to one byte at a time if the input is less than 8 bytes, they perform worse than the old PureJavaCrc32 at the 4 byte to 7 byte level. Is this important? It would be useful to know how often the crc code is called on small byte chunks. We can get this to near PureJavaCrc32 speeds for 4 byte sizes if we add a four byte at a time block to 8_8d.

I tried this before but it did not help.  BTW, the while-loop in the middle should be an if-statement since we have len <= 7.

> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt, c6166_20090727.patch, c6166_20090728.patch, c6166_20090810.patch, c6166_20090811.patch, graph.r, graph.r, Rplots-laptop.pdf, Rplots-nehalem32.pdf, Rplots-nehalem64.pdf, Rplots.pdf, Rplots.pdf, Rplots.pdf
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-898-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 19 18:28:20 2009
Return-Path: <common-issues-return-898-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38340 invoked from network); 19 Aug 2009 18:28:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Aug 2009 18:28:20 -0000
Received: (qmail 2419 invoked by uid 500); 19 Aug 2009 18:28:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2394 invoked by uid 500); 19 Aug 2009 18:28:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2384 invoked by uid 99); 19 Aug 2009 18:28:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 18:28:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 18:28:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 135E5234C4A7
	for <common-issues@hadoop.apache.org>; Wed, 19 Aug 2009 11:28:15 -0700 (PDT)
Message-ID: <1345758741.1250706495078.JavaMail.jira@brutus>
Date: Wed, 19 Aug 2009 11:28:15 -0700 (PDT)
From: "Ben Slusky (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6097) Multiple bugs w/ Hadoop archives
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745159#action_12745159 ] 

Ben Slusky commented on HADOOP-6097:
------------------------------------

Mahadev,

Yes, we are talking about the filesystem cache, not the connection cache. Sorry for the confusion. As you say, har filesystems are identified by har://authority/archivepath, but the filesystem cache does not index filesystems by path -- only by scheme, authority, and username. This leads directly to the misbehavior I described above. I worked around it by not caching har fielsystems. Another option is to cache by scheme, authority, username, AND path. I could submit a patch to do that if anyone has a strong preference for it.

> Multiple bugs w/ Hadoop archives
> --------------------------------
>
>                 Key: HADOOP-6097
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6097
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.18.0, 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1, 0.20.0
>            Reporter: Ben Slusky
>             Fix For: 0.20.1
>
>         Attachments: HADOOP-6097.patch
>
>
> Found and fixed several bugs involving Hadoop archives:
> - In makeQualified(), the sloppy conversion from Path to URI and back mangles the path if it contains an escape-worthy character.
> - It's possible that fileStatusInIndex() may have to read more than one segment of the index. The LineReader and count of bytes read need to be reset for each block.
> - har:// connections cannot be indexed by (scheme, authority, username) -- the path is significant as well. Caching them in this way limits a hadoop client to opening one archive per filesystem. It seems to be safe not to cache them, since they wrap another connection that does the actual networking.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-899-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 19 20:32:20 2009
Return-Path: <common-issues-return-899-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86530 invoked from network); 19 Aug 2009 20:32:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Aug 2009 20:32:20 -0000
Received: (qmail 76905 invoked by uid 500); 19 Aug 2009 20:32:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76849 invoked by uid 500); 19 Aug 2009 20:32:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76835 invoked by uid 99); 19 Aug 2009 20:32:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 20:32:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 20:32:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D19CA234C044
	for <common-issues@hadoop.apache.org>; Wed, 19 Aug 2009 13:32:14 -0700 (PDT)
Message-ID: <1588890167.1250713934843.JavaMail.jira@brutus>
Date: Wed, 19 Aug 2009 13:32:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6166:
-------------------------------------------

    Attachment: c6166_20090819.patch

> I tried this before but it did not help. ...
I should be more clear: it did help for the 4-byte case but it hurt the performance in some other cases.

c6166_20090819.patch: added Crc32_8_8b2 and Crc32_8_8d2.  See whether anyone wants to play with them.

Here is my results.
- java.version = 1.6.0_15
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_15-b03
java.vm.version = 14.1-b02
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = amd64
os.name = Linux
os.version = 2.6.9-55.ELsmp

Performance Table (The unit is MB/sec)
|| Num Bytes ||    CRC32 || PureJavaCrc32 || Crc32_8_8b || Crc32_8_8b2 || Crc32_8_8d || Crc32_8_8d2 ||
|          1 |     7.236 |        106.903 |      58.534 |       61.106 |     109.742 |       67.673 |
|          2 |    14.605 |        104.665 |     104.573 |      104.952 |     115.005 |      102.944 |
|          4 |    27.566 |        177.502 |     223.599 |      228.648 |     131.890 |      218.530 |
|          8 |    49.708 |        193.198 |     196.794 |      175.006 |     251.469 |      204.796 |
|         16 |    83.804 |        259.008 |     234.906 |      221.473 |     258.878 |      236.937 |
|         32 |   128.918 |        313.157 |     299.622 |      273.306 |     322.913 |      300.376 |
|         64 |   176.797 |        349.738 |     339.404 |      320.543 |     365.522 |      348.328 |
|        128 |   217.087 |        370.589 |     362.759 |      353.613 |     390.209 |      384.627 |
|        256 |   244.869 |        381.630 |     376.096 |      369.157 |     405.767 |      401.649 |
|        512 |   261.282 |        385.554 |     382.974 |      378.841 |     413.286 |      411.211 |
|       1024 |   271.205 |        390.170 |     384.064 |      384.586 |     415.817 |      413.505 |
|       2048 |   276.269 |        392.448 |     386.215 |      387.585 |     418.559 |      417.466 |
|       4096 |   279.280 |        393.668 |     387.571 |      388.814 |     418.777 |      419.686 |
|       8192 |   280.938 |        394.176 |     388.399 |      389.576 |     420.204 |      420.571 |
|      16384 |   281.385 |        393.164 |     389.518 |      388.856 |     420.236 |      420.392 |
|      32768 |   281.442 |        391.935 |     389.115 |      387.306 |     419.440 |      417.362 |
|      65536 |   281.605 |        391.986 |     389.124 |      387.297 |     419.405 |      417.123 |
|     131072 |   281.334 |        393.221 |     389.093 |      384.810 |     417.836 |      415.334 |
|     262144 |   280.363 |        391.701 |     386.141 |      385.937 |     417.907 |      415.452 |
|     524288 |   280.373 |        391.727 |     387.646 |      385.911 |     417.851 |      415.510 |
|    1048576 |   280.295 |        391.585 |     387.535 |      385.745 |     417.587 |      414.222 |
|    2097152 |   280.866 |        391.217 |     387.413 |      385.603 |     417.605 |      414.120 |
|    4194304 |   280.273 |        390.247 |     387.650 |      385.971 |     418.284 |      414.302 |
|    8388608 |   279.799 |        388.960 |     385.093 |      383.170 |     414.179 |      412.095 |
|   16777216 |   279.473 |        388.359 |     384.629 |      382.737 |     413.717 |      411.523 |

- java.version = 1.6.0_14
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_14-b08
java.vm.version = 14.0-b16
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = x86
os.name = Windows XP
os.version = 5.1

Performance Table (The unit is MB/sec)
|| Num Bytes ||    CRC32 || PureJavaCrc32 || Crc32_8_8b || Crc32_8_8b2 || Crc32_8_8d || Crc32_8_8d2 ||
|          1 |     3.744 |         49.901 |      47.314 |       49.920 |      51.063 |       45.306 |
|          2 |     7.397 |         78.714 |      64.757 |       63.904 |      69.411 |       69.742 |
|          4 |    14.129 |        118.892 |     116.559 |      122.589 |     100.686 |      120.545 |
|          8 |    27.169 |        141.788 |     178.545 |      179.537 |     177.252 |      176.913 |
|         16 |    48.864 |        197.479 |     199.851 |      206.187 |     205.112 |      197.922 |
|         32 |    87.972 |        242.980 |     252.241 |      243.120 |     248.911 |      230.985 |
|         64 |   128.459 |        262.461 |     272.747 |      284.252 |     276.611 |      264.473 |
|        128 |   178.257 |        265.661 |     305.895 |      303.167 |     293.562 |      286.069 |
|        256 |   213.740 |        280.286 |     307.318 |      324.065 |     296.782 |      297.760 |
|        512 |   247.829 |        287.490 |     334.125 |      332.208 |     317.107 |      310.138 |
|       1024 |   266.095 |        289.453 |     343.593 |      340.496 |     322.506 |      330.703 |
|       2048 |   284.693 |        286.926 |     328.546 |      338.577 |     323.831 |      324.033 |
|       4096 |   292.326 |        304.553 |     354.299 |      351.903 |     332.555 |      334.845 |
|       8192 |   276.416 |        286.101 |     343.856 |      336.716 |     327.936 |      317.006 |
|      16384 |   284.944 |        292.414 |     338.566 |      333.038 |     323.622 |      314.212 |
|      32768 |   275.270 |        292.939 |     339.562 |      332.072 |     332.398 |      294.426 |
|      65536 |   271.285 |        294.269 |     336.092 |      339.905 |     327.122 |      306.583 |
|     131072 |   288.692 |        290.441 |     337.711 |      312.378 |     240.516 |      299.113 |
|     262144 |   288.789 |        290.830 |     342.317 |      331.144 |     322.446 |      312.372 |
|     524288 |   277.527 |        288.056 |     337.654 |      329.802 |     326.141 |      301.740 |
|    1048576 |   278.305 |        288.149 |     332.875 |      328.402 |     320.754 |      300.196 |
|    2097152 |   278.992 |        291.338 |     324.239 |      325.872 |     324.888 |      294.238 |
|    4194304 |   282.476 |        281.645 |     331.034 |      332.346 |     318.677 |      296.442 |
|    8388608 |   283.086 |        287.335 |     328.317 |      331.580 |     325.669 |      306.567 |
|   16777216 |   290.031 |        296.875 |     339.540 |      332.101 |     323.918 |      307.348 |


> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt, c6166_20090727.patch, c6166_20090728.patch, c6166_20090810.patch, c6166_20090811.patch, c6166_20090819.patch, graph.r, graph.r, Rplots-laptop.pdf, Rplots-nehalem32.pdf, Rplots-nehalem64.pdf, Rplots.pdf, Rplots.pdf, Rplots.pdf
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-900-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 19 21:01:18 2009
Return-Path: <common-issues-return-900-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96606 invoked from network); 19 Aug 2009 21:01:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Aug 2009 21:01:18 -0000
Received: (qmail 28404 invoked by uid 500); 19 Aug 2009 21:01:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28368 invoked by uid 500); 19 Aug 2009 21:01:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28358 invoked by uid 99); 19 Aug 2009 21:01:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 21:01:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 21:01:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 22B1F29A0013
	for <common-issues@hadoop.apache.org>; Wed, 19 Aug 2009 14:01:15 -0700 (PDT)
Message-ID: <1611284488.1250715675141.JavaMail.jira@brutus>
Date: Wed, 19 Aug 2009 14:01:15 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6201) FileSystem::ListStatus should throw
 FileNotFoundException
In-Reply-To: <1093368448.1250619614937.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan reassigned HADOOP-6201:
-----------------------------------

    Assignee: Jakob Homan

> FileSystem::ListStatus should throw FileNotFoundException
> ---------------------------------------------------------
>
>                 Key: HADOOP-6201
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6201
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>
> As discussed in HDFS-538, it would be better for FileSystem and its implementations to throw FileNotFoundException when the file is not found, rather than returning null.  This will bring it in line with getFileStatus and, I expect, default user expectations.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-901-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 19 21:15:22 2009
Return-Path: <common-issues-return-901-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 277 invoked from network); 19 Aug 2009 21:15:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 19 Aug 2009 21:15:21 -0000
Received: (qmail 42982 invoked by uid 500); 19 Aug 2009 21:15:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42936 invoked by uid 500); 19 Aug 2009 21:15:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42858 invoked by uid 99); 19 Aug 2009 21:15:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 21:15:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Aug 2009 21:15:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 124FC29A001D
	for <common-issues@hadoop.apache.org>; Wed, 19 Aug 2009 14:15:15 -0700 (PDT)
Message-ID: <2142940272.1250716515074.JavaMail.jira@brutus>
Date: Wed, 19 Aug 2009 14:15:15 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6166:
-------------------------------------------

    Attachment: c6166_20090819review.patch

c6166_20090819review.patch: patch for reviewing.

- replaced PureJavaCrc32 implementation with 8_8d
- updated TestPureJavaCrc32 to junit 4
- added TestPureJavaCrc32.Table
- changed TestPureJavaCrc32.PerformanceTest to print system information

> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt, c6166_20090727.patch, c6166_20090728.patch, c6166_20090810.patch, c6166_20090811.patch, c6166_20090819.patch, c6166_20090819review.patch, graph.r, graph.r, Rplots-laptop.pdf, Rplots-nehalem32.pdf, Rplots-nehalem64.pdf, Rplots.pdf, Rplots.pdf, Rplots.pdf
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-902-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 20 00:12:18 2009
Return-Path: <common-issues-return-902-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72069 invoked from network); 20 Aug 2009 00:12:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Aug 2009 00:12:17 -0000
Received: (qmail 60344 invoked by uid 500); 20 Aug 2009 00:12:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60285 invoked by uid 500); 20 Aug 2009 00:12:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60275 invoked by uid 99); 20 Aug 2009 00:12:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 00:12:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 00:12:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CACBA234C044
	for <common-issues@hadoop.apache.org>; Wed, 19 Aug 2009 17:12:14 -0700 (PDT)
Message-ID: <1978324173.1250727134816.JavaMail.jira@brutus>
Date: Wed, 19 Aug 2009 17:12:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6166:
-------------------------------------------

        Fix Version/s: 0.21.0
    Affects Version/s: 0.21.0
               Status: Patch Available  (was: Open)

> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.21.0
>
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt, c6166_20090727.patch, c6166_20090728.patch, c6166_20090810.patch, c6166_20090811.patch, c6166_20090819.patch, c6166_20090819review.patch, graph.r, graph.r, Rplots-laptop.pdf, Rplots-nehalem32.pdf, Rplots-nehalem64.pdf, Rplots.pdf, Rplots.pdf, Rplots.pdf
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-903-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 20 00:34:24 2009
Return-Path: <common-issues-return-903-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82917 invoked from network); 20 Aug 2009 00:34:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Aug 2009 00:34:24 -0000
Received: (qmail 71441 invoked by uid 500); 20 Aug 2009 00:34:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71385 invoked by uid 500); 20 Aug 2009 00:34:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71375 invoked by uid 99); 20 Aug 2009 00:34:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 00:34:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 00:34:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CBB0F234C044
	for <common-issues@hadoop.apache.org>; Wed, 19 Aug 2009 17:34:14 -0700 (PDT)
Message-ID: <957456708.1250728454819.JavaMail.jira@brutus>
Date: Wed, 19 Aug 2009 17:34:14 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6173) src/native/packageNativeHadoop.sh
 only packages files with "hadoop" in the name
In-Reply-To: <210252632.1249027454928.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6173:
-------------------------------------------

     Component/s:     (was: build)
                  scripts
    Hadoop Flags: [Reviewed]

+1 patch looks good

> ... And Cos created a new issue HADOOP-6174 to track the unit tests for scripts. ...
We are currently doing manual tests for scripts.  Could you post the manual test results and "Also please list what manual steps were performed to verify this patch"?

> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6173
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6173
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: scripts
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6174-20090731.patch
>
>
> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name. This becomes too restrictive when a user wants to inject third-party native libraries into his/her own tar build.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-904-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 20 00:59:24 2009
Return-Path: <common-issues-return-904-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88713 invoked from network); 20 Aug 2009 00:59:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Aug 2009 00:59:24 -0000
Received: (qmail 82789 invoked by uid 500); 20 Aug 2009 00:59:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82733 invoked by uid 500); 20 Aug 2009 00:59:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82718 invoked by uid 99); 20 Aug 2009 00:59:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 00:59:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 00:59:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C5D97234C044
	for <common-issues@hadoop.apache.org>; Wed, 19 Aug 2009 17:59:14 -0700 (PDT)
Message-ID: <2003544340.1250729954796.JavaMail.jira@brutus>
Date: Wed, 19 Aug 2009 17:59:14 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6173) src/native/packageNativeHadoop.sh
 only packages files with "hadoop" in the name
In-Reply-To: <210252632.1249027454928.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745260#action_12745260 ] 

Hong Tang commented on HADOOP-6173:
-----------------------------------

Thanks Nicolas. I verified the patch through manual testing. Steps are listed as follows:

Step 0: download hadoop-gpl-compression source repository and build jar files and native library.
Step 1: copied hadoop-gpl-compression/build/hadoop-gpl-compression-0.1.0-dev/hadoop-gpl-compression-0.1.0-dev.jar  => hadoop-common/lib; hadoop-gpl-compression/build/hadoop-gpl-compression-0.1.0-dev/lib/native/* => hadoop-common/lib/native
Step 2: under hadoop-common, do "ant tar"
Step 3: tar -tzf hadoop-common/build/hadoop-core-0.21.0-dev.tar.gz | grep libgplcompression => returning nothing
Step 4: apply the patch, and redo steps 2-3, and grep shows libgplcompression.* are included in the tar


> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6173
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6173
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: scripts
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6174-20090731.patch
>
>
> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name. This becomes too restrictive when a user wants to inject third-party native libraries into his/her own tar build.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-905-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 20 01:16:46 2009
Return-Path: <common-issues-return-905-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91807 invoked from network); 20 Aug 2009 01:16:44 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Aug 2009 01:16:44 -0000
Received: (qmail 93821 invoked by uid 500); 20 Aug 2009 01:17:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93763 invoked by uid 500); 20 Aug 2009 01:17:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93746 invoked by uid 99); 20 Aug 2009 01:17:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 01:17:03 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 01:15:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C8D81234C044
	for <common-issues@hadoop.apache.org>; Wed, 19 Aug 2009 18:15:14 -0700 (PDT)
Message-ID: <659986870.1250730914808.JavaMail.jira@brutus>
Date: Wed, 19 Aug 2009 18:15:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745269#action_12745269 ] 

Hadoop QA commented on HADOOP-6166:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12417054/c6166_20090819review.patch
  against trunk revision 804918.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/615/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/615/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/615/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/615/console

This message is automatically generated.

> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.21.0
>
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt, c6166_20090727.patch, c6166_20090728.patch, c6166_20090810.patch, c6166_20090811.patch, c6166_20090819.patch, c6166_20090819review.patch, graph.r, graph.r, Rplots-laptop.pdf, Rplots-nehalem32.pdf, Rplots-nehalem64.pdf, Rplots.pdf, Rplots.pdf, Rplots.pdf
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-906-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 20 07:17:18 2009
Return-Path: <common-issues-return-906-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15819 invoked from network); 20 Aug 2009 07:17:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Aug 2009 07:17:17 -0000
Received: (qmail 49739 invoked by uid 500); 20 Aug 2009 07:17:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49681 invoked by uid 500); 20 Aug 2009 07:17:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49666 invoked by uid 99); 20 Aug 2009 07:17:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 07:17:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 07:17:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CF5E0234C044
	for <common-issues@hadoop.apache.org>; Thu, 20 Aug 2009 00:17:14 -0700 (PDT)
Message-ID: <1162759844.1250752634844.JavaMail.jira@brutus>
Date: Thu, 20 Aug 2009 00:17:14 -0700 (PDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5556) A few improvements to DataNodeCluster
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-5556:
---------------------------------

    Attachment: HADOOP-5556.patch

Submitting updated patch . 
Changed path to work with pro split code .

> A few improvements to DataNodeCluster
> -------------------------------------
>
>                 Key: HADOOP-5556
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5556
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.21.0
>
>         Attachments: DataNodeCluster.patch, HADOOP-5556.patch
>
>
> DataNodeCluster is a great tool to simulate a large scale DFS cluster using a small set of machines. A few suggestions to improve this tool:
> # DataNodeCluster uses MiniDFSCluster#startDataNode to start multiple instances of DataNode on one machine. MiniDFSCluster sets DataNode's address to be 127.0.0.1. We should allow to set its address to 0.0.0.0 so DataNodes in different machines could communicate.
> # Currently the size of the blocks injected to DataNode and created in CreatedEditsLog is hardcoded as 10. It would be more convenient if this could be configurable. Also we need to make sure that both use the same block size.
> # If the replication factor of blocks is larger than 1, currently a DataNode in DataNodeCluster will be injected blocks multiple times and therefore it sends block reports to NameNode multiple times. Initial block reports contain only a portion of its blocks and therefore may cause unnecessary block replications. It would be cleaner if only one block report with all its blocks is sent. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-907-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 20 07:19:20 2009
Return-Path: <common-issues-return-907-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16684 invoked from network); 20 Aug 2009 07:19:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Aug 2009 07:19:19 -0000
Received: (qmail 53868 invoked by uid 500); 20 Aug 2009 07:19:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53830 invoked by uid 500); 20 Aug 2009 07:19:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53820 invoked by uid 99); 20 Aug 2009 07:19:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 07:19:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 07:19:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CC1A229A0012
	for <common-issues@hadoop.apache.org>; Thu, 20 Aug 2009 00:19:14 -0700 (PDT)
Message-ID: <787315253.1250752754824.JavaMail.jira@brutus>
Date: Thu, 20 Aug 2009 00:19:14 -0700 (PDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5556) A few improvements to DataNodeCluster
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-5556:
---------------------------------

    Status: Patch Available  (was: Open)

HADOOP-5556.patch is for 0.21  , patch specific to pro split code. 
successfully ran ant test-core  , waiting for Hudson .

> A few improvements to DataNodeCluster
> -------------------------------------
>
>                 Key: HADOOP-5556
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5556
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.21.0
>
>         Attachments: DataNodeCluster.patch, HADOOP-5556.patch
>
>
> DataNodeCluster is a great tool to simulate a large scale DFS cluster using a small set of machines. A few suggestions to improve this tool:
> # DataNodeCluster uses MiniDFSCluster#startDataNode to start multiple instances of DataNode on one machine. MiniDFSCluster sets DataNode's address to be 127.0.0.1. We should allow to set its address to 0.0.0.0 so DataNodes in different machines could communicate.
> # Currently the size of the blocks injected to DataNode and created in CreatedEditsLog is hardcoded as 10. It would be more convenient if this could be configurable. Also we need to make sure that both use the same block size.
> # If the replication factor of blocks is larger than 1, currently a DataNode in DataNodeCluster will be injected blocks multiple times and therefore it sends block reports to NameNode multiple times. Initial block reports contain only a portion of its blocks and therefore may cause unnecessary block replications. It would be cleaner if only one block report with all its blocks is sent. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-908-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 20 07:27:20 2009
Return-Path: <common-issues-return-908-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19424 invoked from network); 20 Aug 2009 07:27:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Aug 2009 07:27:20 -0000
Received: (qmail 63674 invoked by uid 500); 20 Aug 2009 07:27:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63631 invoked by uid 500); 20 Aug 2009 07:27:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63620 invoked by uid 99); 20 Aug 2009 07:27:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 07:27:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 07:27:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C3956234C044
	for <common-issues@hadoop.apache.org>; Thu, 20 Aug 2009 00:27:14 -0700 (PDT)
Message-ID: <1815982495.1250753234786.JavaMail.jira@brutus>
Date: Thu, 20 Aug 2009 00:27:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5556) A few improvements to
 DataNodeCluster
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745339#action_12745339 ] 

Hadoop QA commented on HADOOP-5556:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12417111/HADOOP-5556.patch
  against trunk revision 804918.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 12 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/616/console

This message is automatically generated.

> A few improvements to DataNodeCluster
> -------------------------------------
>
>                 Key: HADOOP-5556
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5556
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.21.0
>
>         Attachments: DataNodeCluster.patch, HADOOP-5556.patch
>
>
> DataNodeCluster is a great tool to simulate a large scale DFS cluster using a small set of machines. A few suggestions to improve this tool:
> # DataNodeCluster uses MiniDFSCluster#startDataNode to start multiple instances of DataNode on one machine. MiniDFSCluster sets DataNode's address to be 127.0.0.1. We should allow to set its address to 0.0.0.0 so DataNodes in different machines could communicate.
> # Currently the size of the blocks injected to DataNode and created in CreatedEditsLog is hardcoded as 10. It would be more convenient if this could be configurable. Also we need to make sure that both use the same block size.
> # If the replication factor of blocks is larger than 1, currently a DataNode in DataNodeCluster will be injected blocks multiple times and therefore it sends block reports to NameNode multiple times. Initial block reports contain only a portion of its blocks and therefore may cause unnecessary block replications. It would be cleaner if only one block report with all its blocks is sent. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-909-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 20 10:27:23 2009
Return-Path: <common-issues-return-909-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86430 invoked from network); 20 Aug 2009 10:27:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Aug 2009 10:27:20 -0000
Received: (qmail 57240 invoked by uid 500); 20 Aug 2009 10:27:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57157 invoked by uid 500); 20 Aug 2009 10:27:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56991 invoked by uid 99); 20 Aug 2009 10:27:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 10:27:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 10:27:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1F92C234C498
	for <common-issues@hadoop.apache.org>; Thu, 20 Aug 2009 03:27:15 -0700 (PDT)
Message-ID: <183009020.1250764035128.JavaMail.jira@brutus>
Date: Thu, 20 Aug 2009 03:27:15 -0700 (PDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6184) Provide a configuration dump in
 json format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745379#action_12745379 ] 

Hemanth Yamijala commented on HADOOP-6184:
------------------------------------------

- updatingResouce need not be static. Also, I think it should not be created unless the flag to store resources is set to true.
- Leave a space between control statement (like if) and the condition
- Rename writeJson to dumpConfiguration. Document in writeJson that it will not work if configuration is loaded from the an input stream. Update documentation to say the format in which the dump is generated. It would also be good to document the format itself as an example.
- Why is flush called after each item written ? Can it be done only once ?
- Should handle the case where default resources are not loaded. When a configuration object is cloned in writeJson, it loses knowledge of whether default resources were loaded with the original config being dumped or not.
- Testcase: 
-- the writers created should be closed.
-- the way we are looking up for the key from the json properties can do with some improvement. Can't we build a hashtable once the deserialization happens and do a lookup ?


> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>            Assignee: V.V.Chaitanya Krishna
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184-2.patch, HADOOP-6184-3.patch, HADOOP-6184-4.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-910-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 20 15:09:18 2009
Return-Path: <common-issues-return-910-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18265 invoked from network); 20 Aug 2009 15:09:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Aug 2009 15:09:18 -0000
Received: (qmail 1201 invoked by uid 500); 20 Aug 2009 15:09:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1145 invoked by uid 500); 20 Aug 2009 15:09:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1122 invoked by uid 99); 20 Aug 2009 15:09:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 15:09:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 15:09:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EA33A234C044
	for <common-issues@hadoop.apache.org>; Thu, 20 Aug 2009 08:09:14 -0700 (PDT)
Message-ID: <1258819303.1250780954944.JavaMail.jira@brutus>
Date: Thu, 20 Aug 2009 08:09:14 -0700 (PDT)
From: "stephen mulcahy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-3744) Eclipse Plugin does not work with
 Eclipse Ganymede (3.4)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-3744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745454#action_12745454 ] 

stephen mulcahy commented on HADOOP-3744:
-----------------------------------------

I ran into the same problem with Eclipse 3.4 on Debian Linux 5.0 - I could browse DFS with the eclipse plugin from hadoop 0.19.2 but I couldn't run a Hadoop Job.

I checked out http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.19.2/ and rebuilt the plugin against my local eclipse as follows,

ant -Declipse.home=/usr/local/eclipse clean package

and copied the resulting plugin .jar file to /usr/local/eclipse/plugins.

For some reason the MapReduce perspective was no longer available and I couldn't see any error from Eclipse. I figure maybe moving from one version of the plugin to another might have corrupted something in Eclipse?

Anyways - since I was at it, I tried installing the latest Eclipse (3.5, Galileo) and rebuilt the plugin against this version. I then installed it as above and start Eclipse.

This time the plugin works perfectly - I can both browse DFS and run a job against Hadoop. So, in conclusion, the patch works against Eclipse 3.5 (on Linux at least) also.

+1 on including this in the release.


> Eclipse Plugin does not work with Eclipse Ganymede (3.4)
> --------------------------------------------------------
>
>                 Key: HADOOP-3744
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3744
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: contrib/eclipse-plugin
>    Affects Versions: 0.16.4, 0.17.0, 0.19.2
>         Environment: Windows XP with Eclipse "Ganymede" 3.4
>            Reporter: Aaron Kimball
>            Assignee: Christophe Taton
>         Attachments: HADOOP-3744.patch, HadoopApplicationLaunchShortcut.diff
>
>
> The newest version of Eclipse seems incompatible with the plugin. The plugin as released in 0.16.4 will allow you to add/remove MapReduce servers, and will allow you to browse/manipulate the DFS in the DFS Browser, but will not allow you to run programs. Clicking "Run As * Run On Hadoop" will simply not cause the run-on-hadoop server selection window to appear. No error message is given.
> Dropping the 0.17.1 copy of the plugin JAR into the eclipse/plugins/ directory does not fix the issue; it is in fact worse: Eclipse does not seem to regard the 0.17 plugin as real. No "MapReduce Perspective" is made available in the perspectives selection window.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-911-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 20 15:09:20 2009
Return-Path: <common-issues-return-911-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18277 invoked from network); 20 Aug 2009 15:09:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Aug 2009 15:09:18 -0000
Received: (qmail 1234 invoked by uid 500); 20 Aug 2009 15:09:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1147 invoked by uid 500); 20 Aug 2009 15:09:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1121 invoked by uid 99); 20 Aug 2009 15:09:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 15:09:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 15:09:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3DDD3234C4AB
	for <common-issues@hadoop.apache.org>; Thu, 20 Aug 2009 08:09:15 -0700 (PDT)
Message-ID: <1518827507.1250780955252.JavaMail.jira@brutus>
Date: Thu, 20 Aug 2009 08:09:15 -0700 (PDT)
From: "stephen mulcahy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-3744) Eclipse Plugin does not work with
 Eclipse Ganymede (3.4)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-3744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stephen mulcahy updated HADOOP-3744:
------------------------------------

    Affects Version/s: 0.19.2

Added 0.19.2 to list of affected versions

> Eclipse Plugin does not work with Eclipse Ganymede (3.4)
> --------------------------------------------------------
>
>                 Key: HADOOP-3744
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3744
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: contrib/eclipse-plugin
>    Affects Versions: 0.16.4, 0.17.0, 0.19.2
>         Environment: Windows XP with Eclipse "Ganymede" 3.4
>            Reporter: Aaron Kimball
>            Assignee: Christophe Taton
>         Attachments: HADOOP-3744.patch, HadoopApplicationLaunchShortcut.diff
>
>
> The newest version of Eclipse seems incompatible with the plugin. The plugin as released in 0.16.4 will allow you to add/remove MapReduce servers, and will allow you to browse/manipulate the DFS in the DFS Browser, but will not allow you to run programs. Clicking "Run As * Run On Hadoop" will simply not cause the run-on-hadoop server selection window to appear. No error message is given.
> Dropping the 0.17.1 copy of the plugin JAR into the eclipse/plugins/ directory does not fix the issue; it is in fact worse: Eclipse does not seem to regard the 0.17 plugin as real. No "MapReduce Perspective" is made available in the perspectives selection window.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-912-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 20 16:26:18 2009
Return-Path: <common-issues-return-912-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52538 invoked from network); 20 Aug 2009 16:26:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Aug 2009 16:26:18 -0000
Received: (qmail 82202 invoked by uid 500); 20 Aug 2009 16:26:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82165 invoked by uid 500); 20 Aug 2009 16:26:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82155 invoked by uid 99); 20 Aug 2009 16:26:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 16:26:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 16:26:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CA37C234C04B
	for <common-issues@hadoop.apache.org>; Thu, 20 Aug 2009 09:26:14 -0700 (PDT)
Message-ID: <780169680.1250785574823.JavaMail.jira@brutus>
Date: Thu, 20 Aug 2009 09:26:14 -0700 (PDT)
From: "Koji Noguchi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6097) Multiple bugs w/ Hadoop archives
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745477#action_12745477 ] 

Koji Noguchi commented on HADOOP-6097:
--------------------------------------

Ben, sorry about that. I was wrong.

I got confused since HarFileSystem.getUri() returns har://archivepath, 
I mistakenly thought FileSystem.CACHE would use the path as part of the hash key.

{noformat}
$ hadoop dfs -ls har:///user/knoguchi/test.har har:///user/knoguchi/test2.har                                
Found 1 items
drw-r--r--   - knoguchi users          0 2009-08-18 18:52 /user/knoguchi/test.har/user
ls: Invalid file name: /user/knoguchi/test2.har in har:///user/knoguchi/test.har


$ hadoop dfs -ls har:///user/knoguchi/test2.har har:///user/knoguchi/test.har
Found 1 items
drw-------   - knoguchi users          0 2009-08-17 19:15 /user/knoguchi/test2.har/user
ls: Invalid file name: /user/knoguchi/test.har in har:///user/knoguchi/test2.har
$ 
{noformat}

> Multiple bugs w/ Hadoop archives
> --------------------------------
>
>                 Key: HADOOP-6097
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6097
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.18.0, 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1, 0.20.0
>            Reporter: Ben Slusky
>             Fix For: 0.20.1
>
>         Attachments: HADOOP-6097.patch
>
>
> Found and fixed several bugs involving Hadoop archives:
> - In makeQualified(), the sloppy conversion from Path to URI and back mangles the path if it contains an escape-worthy character.
> - It's possible that fileStatusInIndex() may have to read more than one segment of the index. The LineReader and count of bytes read need to be reset for each block.
> - har:// connections cannot be indexed by (scheme, authority, username) -- the path is significant as well. Caching them in this way limits a hadoop client to opening one archive per filesystem. It seems to be safe not to cache them, since they wrap another connection that does the actual networking.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-913-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 20 18:19:18 2009
Return-Path: <common-issues-return-913-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67100 invoked from network); 20 Aug 2009 18:19:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Aug 2009 18:19:18 -0000
Received: (qmail 64812 invoked by uid 500); 20 Aug 2009 18:19:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64761 invoked by uid 500); 20 Aug 2009 18:19:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64623 invoked by uid 99); 20 Aug 2009 18:19:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 18:19:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 18:19:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D4ABA234C044
	for <common-issues@hadoop.apache.org>; Thu, 20 Aug 2009 11:19:14 -0700 (PDT)
Message-ID: <33013252.1250792354856.JavaMail.jira@brutus>
Date: Thu, 20 Aug 2009 11:19:14 -0700 (PDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5556) A few improvements to DataNodeCluster
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-5556:
---------------------------------

    Attachment:     (was: HADOOP-5556.patch)

> A few improvements to DataNodeCluster
> -------------------------------------
>
>                 Key: HADOOP-5556
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5556
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.21.0
>
>         Attachments: DataNodeCluster.patch
>
>
> DataNodeCluster is a great tool to simulate a large scale DFS cluster using a small set of machines. A few suggestions to improve this tool:
> # DataNodeCluster uses MiniDFSCluster#startDataNode to start multiple instances of DataNode on one machine. MiniDFSCluster sets DataNode's address to be 127.0.0.1. We should allow to set its address to 0.0.0.0 so DataNodes in different machines could communicate.
> # Currently the size of the blocks injected to DataNode and created in CreatedEditsLog is hardcoded as 10. It would be more convenient if this could be configurable. Also we need to make sure that both use the same block size.
> # If the replication factor of blocks is larger than 1, currently a DataNode in DataNodeCluster will be injected blocks multiple times and therefore it sends block reports to NameNode multiple times. Initial block reports contain only a portion of its blocks and therefore may cause unnecessary block replications. It would be cleaner if only one block report with all its blocks is sent. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-914-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 20 18:25:18 2009
Return-Path: <common-issues-return-914-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69101 invoked from network); 20 Aug 2009 18:25:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Aug 2009 18:25:18 -0000
Received: (qmail 72320 invoked by uid 500); 20 Aug 2009 18:25:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72225 invoked by uid 500); 20 Aug 2009 18:25:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72175 invoked by uid 99); 20 Aug 2009 18:25:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 18:25:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 18:25:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 3EDA5234C4B0
	for <common-issues@hadoop.apache.org>; Thu, 20 Aug 2009 11:25:15 -0700 (PDT)
Message-ID: <958805239.1250792715256.JavaMail.jira@brutus>
Date: Thu, 20 Aug 2009 11:25:15 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6200) glue together the different
 projects to make local builds easier
In-Reply-To: <68853708.1250609369522.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745545#action_12745545 ] 

Doug Cutting commented on HADOOP-6200:
--------------------------------------

It's not actually a cycle, since tests are layered after.  Each can be built independently.  They can't be fully tested independently, but that's different.  Perhaps we should separate tests that require mapreduce from those that do not.

A simple lo-tek solution might be to reference ../mapreduce/build/ in the mapreduce test classpath and require that developers who wish to run those tests check things out in sibling directories.  We could even then have single build target that builds and tests everything.


> glue together the different projects to make local builds easier
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6200
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6200
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Priority: Minor
>
> It's currently fairly tricky to get everything building/up to date in a single machine. We can and should improve this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-915-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 20 18:25:20 2009
Return-Path: <common-issues-return-915-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69263 invoked from network); 20 Aug 2009 18:25:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Aug 2009 18:25:20 -0000
Received: (qmail 72565 invoked by uid 500); 20 Aug 2009 18:25:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72507 invoked by uid 500); 20 Aug 2009 18:25:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72497 invoked by uid 99); 20 Aug 2009 18:25:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 18:25:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 18:25:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E0499234C044
	for <common-issues@hadoop.apache.org>; Thu, 20 Aug 2009 11:25:14 -0700 (PDT)
Message-ID: <1330664896.1250792714904.JavaMail.jira@brutus>
Date: Thu, 20 Aug 2009 11:25:14 -0700 (PDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5640) Allow ServicePlugins to hook
 callbacks into key service events
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745543#action_12745543 ] 

Todd Lipcon commented on HADOOP-5640:
-------------------------------------

Hey Sanjay,

It's been a while, but I think the issue was this:

- In the Thrift/HDFS RPC stuff, both the DN and the NN have plugins. The DN exposes things to let the client read chunks out of blocks, and the NN of course exposes the metadata ops, getBlockLocations, etc.
- The trick is that we don't want to force the DN thrift plugins to bind to a hardcoded port. So, by default the DN plugin binds to an ephemeral port, and once the server is up and listening, it passes along the actual port to the NN Plugin via Thrift. Thus, the NN Plugin maintains a list of DatanodeID -> DN Plugin Thrift Port and can give out the thrift port to clients in response to getBlockLocations.
- This is all well and good, since that registration happens via thrift call from DN to NN when the DN plugin starts. The issue comes up when the NN restarts while the DN stays up. In that case, the DN service loop automatically re-registers itself with the new NN, but the DN plugin has no idea that it has to re-register. Hence the DatanodeID->ThriftPort map in the NN Plugin is empty and has no way of repopulating itself.

The only solutions here were (a) to add a hook on registration so the DN Plugin can reregister, or (b) add a second heartbeat from the DN Plugin to the NN Plugin. We decided that the heartbeats added unnecessary complexity as well as extra load on the NN, so went with the hook solution.

> Allow ServicePlugins to hook callbacks into key service events
> --------------------------------------------------------------
>
>                 Key: HADOOP-5640
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5640
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hadoop-5640.txt, hadoop-5640.txt, HADOOP-5640.v2.txt, hadoop-5640.v3.txt
>
>
> HADOOP-5257 added the ability for NameNode and DataNode to start and stop ServicePlugin implementations at NN/DN start/stop. However, this is insufficient integration for some common use cases.
> We should add some functionality for Plugins to subscribe to events generated by the service they're plugging into. Some potential hook points are:
> NameNode:
>   - new datanode registered
>   - datanode has died
>   - exception caught
>   - etc?
> DataNode:
>   - startup
>   - initial registration with NN complete (this is important for HADOOP-4707 to sync up datanode.dnRegistration.name with the NN-side registration)
>   - namenode reconnect
>   - some block transfer hooks?
>   - exception caught
> I see two potential routes for implementation:
> 1) We make an enum for the types of hookpoints and have a general function in the ServicePlugin interface. Something like:
> {code:java}
> enum HookPoint {
>   DN_STARTUP,
>   DN_RECEIVED_NEW_BLOCK,
>   DN_CAUGHT_EXCEPTION,
>  ...
> }
> void runHook(HookPoint hp, Object value);
> {code}
> 2) We make classes specific to each "pluggable" as was originally suggested in HADDOP-5257. Something like:
> {code:java}
> class DataNodePlugin {
>   void datanodeStarted() {}
>   void receivedNewBlock(block info, etc) {}
>   void caughtException(Exception e) {}
>   ...
> }
> {code}
> I personally prefer option (2) since we can ensure plugin API compatibility at compile-time, and we avoid an ugly switch statement in a runHook() function.
> Interested to hear what people's thoughts are here.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-916-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 20 18:27:18 2009
Return-Path: <common-issues-return-916-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69843 invoked from network); 20 Aug 2009 18:27:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Aug 2009 18:27:18 -0000
Received: (qmail 73709 invoked by uid 500); 20 Aug 2009 18:27:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73645 invoked by uid 500); 20 Aug 2009 18:27:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73608 invoked by uid 99); 20 Aug 2009 18:27:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 18:27:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 18:27:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E0F38234C1E9
	for <common-issues@hadoop.apache.org>; Thu, 20 Aug 2009 11:27:14 -0700 (PDT)
Message-ID: <626549835.1250792834920.JavaMail.jira@brutus>
Date: Thu, 20 Aug 2009 11:27:14 -0700 (PDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5556) A few improvements to DataNodeCluster
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-5556:
---------------------------------

    Status: Patch Available  (was: Open)

> A few improvements to DataNodeCluster
> -------------------------------------
>
>                 Key: HADOOP-5556
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5556
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.21.0
>
>         Attachments: DataNodeCluster.patch
>
>
> DataNodeCluster is a great tool to simulate a large scale DFS cluster using a small set of machines. A few suggestions to improve this tool:
> # DataNodeCluster uses MiniDFSCluster#startDataNode to start multiple instances of DataNode on one machine. MiniDFSCluster sets DataNode's address to be 127.0.0.1. We should allow to set its address to 0.0.0.0 so DataNodes in different machines could communicate.
> # Currently the size of the blocks injected to DataNode and created in CreatedEditsLog is hardcoded as 10. It would be more convenient if this could be configurable. Also we need to make sure that both use the same block size.
> # If the replication factor of blocks is larger than 1, currently a DataNode in DataNodeCluster will be injected blocks multiple times and therefore it sends block reports to NameNode multiple times. Initial block reports contain only a portion of its blocks and therefore may cause unnecessary block replications. It would be cleaner if only one block report with all its blocks is sent. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-917-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 20 18:27:18 2009
Return-Path: <common-issues-return-917-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69852 invoked from network); 20 Aug 2009 18:27:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Aug 2009 18:27:18 -0000
Received: (qmail 73775 invoked by uid 500); 20 Aug 2009 18:27:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73704 invoked by uid 500); 20 Aug 2009 18:27:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73630 invoked by uid 99); 20 Aug 2009 18:27:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 18:27:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 18:27:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C864D234C044
	for <common-issues@hadoop.apache.org>; Thu, 20 Aug 2009 11:27:14 -0700 (PDT)
Message-ID: <1763724048.1250792834806.JavaMail.jira@brutus>
Date: Thu, 20 Aug 2009 11:27:14 -0700 (PDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5556) A few improvements to DataNodeCluster
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-5556:
---------------------------------

    Status: Open  (was: Patch Available)

Canceling patch . 

> A few improvements to DataNodeCluster
> -------------------------------------
>
>                 Key: HADOOP-5556
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5556
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.21.0
>
>         Attachments: DataNodeCluster.patch
>
>
> DataNodeCluster is a great tool to simulate a large scale DFS cluster using a small set of machines. A few suggestions to improve this tool:
> # DataNodeCluster uses MiniDFSCluster#startDataNode to start multiple instances of DataNode on one machine. MiniDFSCluster sets DataNode's address to be 127.0.0.1. We should allow to set its address to 0.0.0.0 so DataNodes in different machines could communicate.
> # Currently the size of the blocks injected to DataNode and created in CreatedEditsLog is hardcoded as 10. It would be more convenient if this could be configurable. Also we need to make sure that both use the same block size.
> # If the replication factor of blocks is larger than 1, currently a DataNode in DataNodeCluster will be injected blocks multiple times and therefore it sends block reports to NameNode multiple times. Initial block reports contain only a portion of its blocks and therefore may cause unnecessary block replications. It would be cleaner if only one block report with all its blocks is sent. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-918-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 20 19:46:20 2009
Return-Path: <common-issues-return-918-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95430 invoked from network); 20 Aug 2009 19:46:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Aug 2009 19:46:20 -0000
Received: (qmail 97708 invoked by uid 500); 20 Aug 2009 19:46:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97679 invoked by uid 500); 20 Aug 2009 19:46:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97669 invoked by uid 99); 20 Aug 2009 19:46:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 19:46:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 19:46:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CEECD234C044
	for <common-issues@hadoop.apache.org>; Thu, 20 Aug 2009 12:46:14 -0700 (PDT)
Message-ID: <2108675829.1250797574846.JavaMail.jira@brutus>
Date: Thu, 20 Aug 2009 12:46:14 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6205) test-patch provides misleading
 message about JavaDoc warnings
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

test-patch provides misleading message about JavaDoc warnings
-------------------------------------------------------------

                 Key: HADOOP-6205
                 URL: https://issues.apache.org/jira/browse/HADOOP-6205
             Project: Hadoop Common
          Issue Type: Bug
          Components: scripts
            Reporter: Konstantin Boudnik


test-patch generated message usually includes a line
{noformat}
+1 javadoc. The javadoc tool did not generate any warning messages.
{noformat}

However the message is highly misleading, because the project code has tons of JavaDoc's warnings and even errors. What the message should say is 
"The javadoc tool did not generate new warning messages"


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-919-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 20 19:50:18 2009
Return-Path: <common-issues-return-919-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95863 invoked from network); 20 Aug 2009 19:50:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Aug 2009 19:50:18 -0000
Received: (qmail 99250 invoked by uid 500); 20 Aug 2009 19:50:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99190 invoked by uid 500); 20 Aug 2009 19:50:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99180 invoked by uid 99); 20 Aug 2009 19:50:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 19:50:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 19:50:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C4914234C004
	for <common-issues@hadoop.apache.org>; Thu, 20 Aug 2009 12:50:14 -0700 (PDT)
Message-ID: <821311128.1250797814791.JavaMail.jira@brutus>
Date: Thu, 20 Aug 2009 12:50:14 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6205) test-patch provides misleading
 message about JavaDoc warnings
In-Reply-To: <2108675829.1250797574846.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745596#action_12745596 ] 

Konstantin Boudnik commented on HADOOP-6205:
--------------------------------------------

Also, the other related message by test-patch says:
{noformat}
    -1 javadoc.  The javadoc tool appears to have generated $javadocWarnings warning messages."
{noformat}

It seems to be more appropriate to generate 
{noformat}
    -1 javadoc.  The javadoc tool appears to have generated $javadocWarnings new warning messages."
{noformat}
in this case

> test-patch provides misleading message about JavaDoc warnings
> -------------------------------------------------------------
>
>                 Key: HADOOP-6205
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6205
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Konstantin Boudnik
>
> test-patch generated message usually includes a line
> {noformat}
> +1 javadoc. The javadoc tool did not generate any warning messages.
> {noformat}
> However the message is highly misleading, because the project code has tons of JavaDoc's warnings and even errors. What the message should say is 
> "The javadoc tool did not generate new warning messages"

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-920-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 20 21:16:20 2009
Return-Path: <common-issues-return-920-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31231 invoked from network); 20 Aug 2009 21:16:20 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Aug 2009 21:16:20 -0000
Received: (qmail 24479 invoked by uid 500); 20 Aug 2009 21:16:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24425 invoked by uid 500); 20 Aug 2009 21:16:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24415 invoked by uid 99); 20 Aug 2009 21:16:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 21:16:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 21:16:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C439C234C004
	for <common-issues@hadoop.apache.org>; Thu, 20 Aug 2009 14:16:14 -0700 (PDT)
Message-ID: <1705900852.1250802974790.JavaMail.jira@brutus>
Date: Thu, 20 Aug 2009 14:16:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5556) A few improvements to
 DataNodeCluster
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745662#action_12745662 ] 

Hadoop QA commented on HADOOP-5556:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12403474/DataNodeCluster.patch
  against trunk revision 804918.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 12 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/617/console

This message is automatically generated.

> A few improvements to DataNodeCluster
> -------------------------------------
>
>                 Key: HADOOP-5556
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5556
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.21.0
>
>         Attachments: DataNodeCluster.patch
>
>
> DataNodeCluster is a great tool to simulate a large scale DFS cluster using a small set of machines. A few suggestions to improve this tool:
> # DataNodeCluster uses MiniDFSCluster#startDataNode to start multiple instances of DataNode on one machine. MiniDFSCluster sets DataNode's address to be 127.0.0.1. We should allow to set its address to 0.0.0.0 so DataNodes in different machines could communicate.
> # Currently the size of the blocks injected to DataNode and created in CreatedEditsLog is hardcoded as 10. It would be more convenient if this could be configurable. Also we need to make sure that both use the same block size.
> # If the replication factor of blocks is larger than 1, currently a DataNode in DataNodeCluster will be injected blocks multiple times and therefore it sends block reports to NameNode multiple times. Initial block reports contain only a portion of its blocks and therefore may cause unnecessary block replications. It would be cleaner if only one block report with all its blocks is sent. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-921-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 20 21:29:32 2009
Return-Path: <common-issues-return-921-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35556 invoked from network); 20 Aug 2009 21:29:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Aug 2009 21:29:32 -0000
Received: (qmail 39010 invoked by uid 500); 20 Aug 2009 21:29:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38951 invoked by uid 500); 20 Aug 2009 21:29:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38941 invoked by uid 99); 20 Aug 2009 21:29:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 21:29:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 21:29:42 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CB4BB234C004
	for <common-issues@hadoop.apache.org>; Thu, 20 Aug 2009 14:29:22 -0700 (PDT)
Message-ID: <1988164592.1250803762818.JavaMail.jira@brutus>
Date: Thu, 20 Aug 2009 14:29:22 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6173) src/native/packageNativeHadoop.sh
 only packages files with "hadoop" in the name
In-Reply-To: <210252632.1249027454928.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6173:
-------------------------------------------

    Component/s: build

> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6173
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6173
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build, scripts
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6174-20090731.patch
>
>
> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name. This becomes too restrictive when a user wants to inject third-party native libraries into his/her own tar build.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-922-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 20 21:29:32 2009
Return-Path: <common-issues-return-922-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35546 invoked from network); 20 Aug 2009 21:29:32 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Aug 2009 21:29:32 -0000
Received: (qmail 39071 invoked by uid 500); 20 Aug 2009 21:29:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39033 invoked by uid 500); 20 Aug 2009 21:29:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38976 invoked by uid 99); 20 Aug 2009 21:29:51 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 21:29:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 21:29:42 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EAF4029A0017
	for <common-issues@hadoop.apache.org>; Thu, 20 Aug 2009 14:29:22 -0700 (PDT)
Message-ID: <45976102.1250803762961.JavaMail.jira@brutus>
Date: Thu, 20 Aug 2009 14:29:22 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6173) src/native/packageNativeHadoop.sh
 only packages files with "hadoop" in the name
In-Reply-To: <210252632.1249027454928.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6173:
-------------------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I have committed this.  Thanks, Hong!

> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6173
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6173
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build, scripts
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6174-20090731.patch
>
>
> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name. This becomes too restrictive when a user wants to inject third-party native libraries into his/her own tar build.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-923-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 20 23:18:19 2009
Return-Path: <common-issues-return-923-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67880 invoked from network); 20 Aug 2009 23:18:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Aug 2009 23:18:18 -0000
Received: (qmail 78536 invoked by uid 500); 20 Aug 2009 23:18:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78468 invoked by uid 500); 20 Aug 2009 23:18:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78451 invoked by uid 99); 20 Aug 2009 23:18:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 23:18:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 23:18:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D3087234C004
	for <common-issues@hadoop.apache.org>; Thu, 20 Aug 2009 16:18:14 -0700 (PDT)
Message-ID: <59159433.1250810294850.JavaMail.jira@brutus>
Date: Thu, 20 Aug 2009 16:18:14 -0700 (PDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6152) Hadoop scripts do not correctly put
 jars on the classpath
In-Reply-To: <283943906.1247689875148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron Kimball updated HADOOP-6152:
----------------------------------

    Attachment: HADOOP-6152.4.patch

Fixed mapred-tools jar path; attached new patch.

> Hadoop scripts do not correctly put jars on the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-6152
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6152
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>            Priority: Blocker
>         Attachments: HADOOP-6152.2.patch, HADOOP-6152.3.patch, HADOOP-6152.4.patch, HADOOP-6152.patch
>
>
> The various Hadoop scripts (bin/hadoop, bin/hdfs, bin/mapred) do not properly identify the jars needed to run Hadoop. They try to include hadoop-*-hdfs.jar, etc, rather than the hadoop-hdfs-*.jar that is actually created by the 'ant jar' and 'ant package' targets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-924-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 20 23:18:20 2009
Return-Path: <common-issues-return-924-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67886 invoked from network); 20 Aug 2009 23:18:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Aug 2009 23:18:18 -0000
Received: (qmail 78567 invoked by uid 500); 20 Aug 2009 23:18:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78470 invoked by uid 500); 20 Aug 2009 23:18:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78448 invoked by uid 99); 20 Aug 2009 23:18:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 23:18:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 23:18:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id F29BA29A001D
	for <common-issues@hadoop.apache.org>; Thu, 20 Aug 2009 16:18:14 -0700 (PDT)
Message-ID: <1010617516.1250810294993.JavaMail.jira@brutus>
Date: Thu, 20 Aug 2009 16:18:14 -0700 (PDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6152) Hadoop scripts do not correctly put
 jars on the classpath
In-Reply-To: <283943906.1247689875148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron Kimball updated HADOOP-6152:
----------------------------------

    Status: Patch Available  (was: Open)

> Hadoop scripts do not correctly put jars on the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-6152
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6152
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>            Priority: Blocker
>         Attachments: HADOOP-6152.2.patch, HADOOP-6152.3.patch, HADOOP-6152.4.patch, HADOOP-6152.patch
>
>
> The various Hadoop scripts (bin/hadoop, bin/hdfs, bin/mapred) do not properly identify the jars needed to run Hadoop. They try to include hadoop-*-hdfs.jar, etc, rather than the hadoop-hdfs-*.jar that is actually created by the 'ant jar' and 'ant package' targets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-925-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 20 23:18:20 2009
Return-Path: <common-issues-return-925-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67949 invoked from network); 20 Aug 2009 23:18:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 20 Aug 2009 23:18:19 -0000
Received: (qmail 78681 invoked by uid 500); 20 Aug 2009 23:18:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78623 invoked by uid 500); 20 Aug 2009 23:18:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78613 invoked by uid 99); 20 Aug 2009 23:18:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 23:18:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Aug 2009 23:18:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E9C3C29A0017
	for <common-issues@hadoop.apache.org>; Thu, 20 Aug 2009 16:18:14 -0700 (PDT)
Message-ID: <134125879.1250810294956.JavaMail.jira@brutus>
Date: Thu, 20 Aug 2009 16:18:14 -0700 (PDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6152) Hadoop scripts do not correctly put
 jars on the classpath
In-Reply-To: <283943906.1247689875148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron Kimball updated HADOOP-6152:
----------------------------------

    Status: Open  (was: Patch Available)

> Hadoop scripts do not correctly put jars on the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-6152
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6152
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>            Priority: Blocker
>         Attachments: HADOOP-6152.2.patch, HADOOP-6152.3.patch, HADOOP-6152.4.patch, HADOOP-6152.patch
>
>
> The various Hadoop scripts (bin/hadoop, bin/hdfs, bin/mapred) do not properly identify the jars needed to run Hadoop. They try to include hadoop-*-hdfs.jar, etc, rather than the hadoop-hdfs-*.jar that is actually created by the 'ant jar' and 'ant package' targets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-926-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 21 01:12:18 2009
Return-Path: <common-issues-return-926-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98237 invoked from network); 21 Aug 2009 01:12:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Aug 2009 01:12:17 -0000
Received: (qmail 92283 invoked by uid 500); 21 Aug 2009 01:12:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92247 invoked by uid 500); 21 Aug 2009 01:12:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92237 invoked by uid 99); 21 Aug 2009 01:12:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 01:12:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 01:12:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C630F234C004
	for <common-issues@hadoop.apache.org>; Thu, 20 Aug 2009 18:12:14 -0700 (PDT)
Message-ID: <1986331563.1250817134807.JavaMail.jira@brutus>
Date: Thu, 20 Aug 2009 18:12:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6152) Hadoop scripts do not correctly put
 jars on the classpath
In-Reply-To: <283943906.1247689875148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745760#action_12745760 ] 

Hadoop QA commented on HADOOP-6152:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12417192/HADOOP-6152.4.patch
  against trunk revision 806368.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/618/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/618/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/618/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/618/console

This message is automatically generated.

> Hadoop scripts do not correctly put jars on the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-6152
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6152
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>            Priority: Blocker
>         Attachments: HADOOP-6152.2.patch, HADOOP-6152.3.patch, HADOOP-6152.4.patch, HADOOP-6152.patch
>
>
> The various Hadoop scripts (bin/hadoop, bin/hdfs, bin/mapred) do not properly identify the jars needed to run Hadoop. They try to include hadoop-*-hdfs.jar, etc, rather than the hadoop-hdfs-*.jar that is actually created by the 'ant jar' and 'ant package' targets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-927-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 21 03:36:19 2009
Return-Path: <common-issues-return-927-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22315 invoked from network); 21 Aug 2009 03:36:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Aug 2009 03:36:18 -0000
Received: (qmail 9216 invoked by uid 500); 21 Aug 2009 03:36:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9140 invoked by uid 500); 21 Aug 2009 03:36:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9130 invoked by uid 99); 21 Aug 2009 03:36:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 03:36:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 03:36:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C5948234C004
	for <common-issues@hadoop.apache.org>; Thu, 20 Aug 2009 20:36:14 -0700 (PDT)
Message-ID: <526006076.1250825774795.JavaMail.jira@brutus>
Date: Thu, 20 Aug 2009 20:36:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6205) test-patch provides misleading
 message about JavaDoc warnings
In-Reply-To: <2108675829.1250797574846.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745777#action_12745777 ] 

Giridharan Kesavan commented on HADOOP-6205:
--------------------------------------------

  [javadoc] Loading source files for package org.apache.hadoop.security.authorize...
  [javadoc] Loading source files for package org.apache.hadoop.util...
  [javadoc] Loading source files for package org.apache.hadoop.util.bloom...
  [javadoc] Loading source files for package org.apache.hadoop.util.hash...
  [javadoc] Loading source files for package org.apache.hadoop.contrib.failmon...
  [javadoc] Constructing Javadoc information...
  [javadoc] Standard Doclet version 1.6.0_07
  [javadoc] Building tree for all the packages and classes...
  [javadoc] Building index for all the packages and classes...
  [javadoc] Building index for all classes...

BUILD SUCCESSFUL
Total time: 17 seconds

bq. However the message is highly misleading, because the project code has tons of JavaDoc's warnings and even errors. What the message should say is "The javadoc tool did not generate new warning messages"

I dont see any javadoc warnings when I compile common trunk; Project codes are expected to have 0 javadoc warnings and test-patch runs with the assumption that any warning that is thrown during javadoc executing is being introduced by the patch being test for.

Please help me If I'm missing something here.

> test-patch provides misleading message about JavaDoc warnings
> -------------------------------------------------------------
>
>                 Key: HADOOP-6205
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6205
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Konstantin Boudnik
>
> test-patch generated message usually includes a line
> {noformat}
> +1 javadoc. The javadoc tool did not generate any warning messages.
> {noformat}
> However the message is highly misleading, because the project code has tons of JavaDoc's warnings and even errors. What the message should say is 
> "The javadoc tool did not generate new warning messages"

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-928-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 21 05:48:20 2009
Return-Path: <common-issues-return-928-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48752 invoked from network); 21 Aug 2009 05:48:19 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Aug 2009 05:48:19 -0000
Received: (qmail 15696 invoked by uid 500); 21 Aug 2009 05:48:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15631 invoked by uid 500); 21 Aug 2009 05:48:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15616 invoked by uid 99); 21 Aug 2009 05:48:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 05:48:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 05:48:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D83EA29A0014
	for <common-issues@hadoop.apache.org>; Thu, 20 Aug 2009 22:48:14 -0700 (PDT)
Message-ID: <799830860.1250833694884.JavaMail.jira@brutus>
Date: Thu, 20 Aug 2009 22:48:14 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6184) Provide a configuration dump in json
 format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna updated HADOOP-6184:
------------------------------------------

    Attachment: HADOOP-6184-5.patch

uploading patch with the above mentioned points implemented.

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>            Assignee: V.V.Chaitanya Krishna
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184-2.patch, HADOOP-6184-3.patch, HADOOP-6184-4.patch, HADOOP-6184-5.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-930-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 21 08:20:57 2009
Return-Path: <common-issues-return-930-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15008 invoked from network); 21 Aug 2009 08:20:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Aug 2009 08:20:57 -0000
Received: (qmail 44867 invoked by uid 500); 21 Aug 2009 07:54:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44811 invoked by uid 500); 21 Aug 2009 07:54:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44797 invoked by uid 99); 21 Aug 2009 07:54:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 07:54:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 07:54:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D26EC234C044
	for <common-issues@hadoop.apache.org>; Fri, 21 Aug 2009 00:54:14 -0700 (PDT)
Message-ID: <1178680078.1250841254860.JavaMail.jira@brutus>
Date: Fri, 21 Aug 2009 00:54:14 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6103) Configuration clone constructor
 does not clone all the members.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745844#action_12745844 ] 

Hudson commented on HADOOP-6103:
--------------------------------

Integrated in Hadoop-Common-trunk #63 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/63/])
    . Clones the classloader as part of Configuration clone. Contributed by Amareshwari Sriramadasu.


> Configuration clone constructor does not clone all the members.
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6103
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6103
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.21.0
>
>         Attachments: patch-6103.txt
>
>
> Currently, Configuration(Configuration other) constructor does not clone all the members.
> It clones only resources, properties, overlay and finalParameters. It needs to clone loadDefaults, classLoader, defaultResources, quietmode.
> This resulted in bugs like HADOOP-4975

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-931-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 21 08:21:00 2009
Return-Path: <common-issues-return-931-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15082 invoked from network); 21 Aug 2009 08:21:00 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Aug 2009 08:21:00 -0000
Received: (qmail 44974 invoked by uid 500); 21 Aug 2009 07:54:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44912 invoked by uid 500); 21 Aug 2009 07:54:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44897 invoked by uid 99); 21 Aug 2009 07:54:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 07:54:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 07:54:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EC09A234C053
	for <common-issues@hadoop.apache.org>; Fri, 21 Aug 2009 00:54:14 -0700 (PDT)
Message-ID: <1760958206.1250841254966.JavaMail.jira@brutus>
Date: Fri, 21 Aug 2009 00:54:14 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6173) src/native/packageNativeHadoop.sh
 only packages files with "hadoop" in the name
In-Reply-To: <210252632.1249027454928.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745845#action_12745845 ] 

Hudson commented on HADOOP-6173:
--------------------------------

Integrated in Hadoop-Common-trunk #63 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/63/])
    . Change src/native/packageNativeHadoop.sh to package all native library files.  Contributed by Hong Tang


> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6173
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6173
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build, scripts
>    Affects Versions: 0.21.0
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6174-20090731.patch
>
>
> src/native/packageNativeHadoop.sh only packages files with "hadoop" in the name. This becomes too restrictive when a user wants to inject third-party native libraries into his/her own tar build.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-929-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 21 08:47:02 2009
Return-Path: <common-issues-return-929-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31168 invoked from network); 21 Aug 2009 08:46:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Aug 2009 08:46:56 -0000
Received: (qmail 82863 invoked by uid 500); 21 Aug 2009 07:00:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82805 invoked by uid 500); 21 Aug 2009 07:00:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82795 invoked by uid 99); 21 Aug 2009 07:00:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 07:00:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 07:00:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 98DCA29A0018
	for <common-issues@hadoop.apache.org>; Fri, 21 Aug 2009 00:00:16 -0700 (PDT)
Message-ID: <228445429.1250838016625.JavaMail.jira@brutus>
Date: Fri, 21 Aug 2009 00:00:16 -0700 (PDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6103) Configuration clone constructor does
 not clone all the members.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6103:
--------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I just committed this. Thanks, Amareshwari!

> Configuration clone constructor does not clone all the members.
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6103
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6103
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.21.0
>
>         Attachments: patch-6103.txt
>
>
> Currently, Configuration(Configuration other) constructor does not clone all the members.
> It clones only resources, properties, overlay and finalParameters. It needs to clone loadDefaults, classLoader, defaultResources, quietmode.
> This resulted in bugs like HADOOP-4975

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-932-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 21 09:04:15 2009
Return-Path: <common-issues-return-932-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41131 invoked from network); 21 Aug 2009 09:04:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Aug 2009 09:04:15 -0000
Received: (qmail 52119 invoked by uid 500); 21 Aug 2009 09:04:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52053 invoked by uid 500); 21 Aug 2009 09:04:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51768 invoked by uid 99); 21 Aug 2009 09:04:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 09:04:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 09:04:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DA014234C1E9
	for <common-issues@hadoop.apache.org>; Fri, 21 Aug 2009 02:04:14 -0700 (PDT)
Message-ID: <975591878.1250845454892.JavaMail.jira@brutus>
Date: Fri, 21 Aug 2009 02:04:14 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6184) Provide a configuration dump in json
 format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna updated HADOOP-6184:
------------------------------------------

    Attachment: HADOOP-6184-6.patch

The previous patch is not valid with the recent commits done. Uploading a new patch with this issue handled. 

Also, the sample parser for json format can be checked in TestConfiguration.java

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>            Assignee: V.V.Chaitanya Krishna
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184-2.patch, HADOOP-6184-3.patch, HADOOP-6184-4.patch, HADOOP-6184-5.patch, HADOOP-6184-6.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-933-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 21 09:49:16 2009
Return-Path: <common-issues-return-933-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68047 invoked from network); 21 Aug 2009 09:49:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Aug 2009 09:49:16 -0000
Received: (qmail 15788 invoked by uid 500); 21 Aug 2009 09:49:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15739 invoked by uid 500); 21 Aug 2009 09:49:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15724 invoked by uid 99); 21 Aug 2009 09:49:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 09:49:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 09:49:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CDB29234C004
	for <common-issues@hadoop.apache.org>; Fri, 21 Aug 2009 02:49:14 -0700 (PDT)
Message-ID: <258090474.1250848154827.JavaMail.jira@brutus>
Date: Fri, 21 Aug 2009 02:49:14 -0700 (PDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6184) Provide a configuration dump in
 json format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745878#action_12745878 ] 

Hemanth Yamijala commented on HADOOP-6184:
------------------------------------------

- The commit of HADOOP-6103 ensured that the loadDefaults variable is being cloned in the copy constructor. So, it can be removed from the private constructor you are introducing.

bq. the writers created should be closed.
- I meant the BufferedWriter to be closed in the testcase, not the StringWriter.

- Also for the test case related to default resources, I think we should ideally have that in a separate test case. And it should make sure that the default size of the json properties is 0. Then we can add a new resource that adds a couple of strings and makes sure those alone are dumped.


> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>            Assignee: V.V.Chaitanya Krishna
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184-2.patch, HADOOP-6184-3.patch, HADOOP-6184-4.patch, HADOOP-6184-5.patch, HADOOP-6184-6.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-934-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 21 10:57:14 2009
Return-Path: <common-issues-return-934-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97859 invoked from network); 21 Aug 2009 10:57:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Aug 2009 10:57:14 -0000
Received: (qmail 1284 invoked by uid 500); 21 Aug 2009 10:57:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1225 invoked by uid 500); 21 Aug 2009 10:57:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1210 invoked by uid 99); 21 Aug 2009 10:57:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 10:57:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 10:57:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CA138234C004
	for <common-issues@hadoop.apache.org>; Fri, 21 Aug 2009 03:57:14 -0700 (PDT)
Message-ID: <924500548.1250852234813.JavaMail.jira@brutus>
Date: Fri, 21 Aug 2009 03:57:14 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6184) Provide a configuration dump in json
 format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna updated HADOOP-6184:
------------------------------------------

    Attachment: HADOOP-6184-7.patch

uploading the patch with the above points implemented.

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>            Assignee: V.V.Chaitanya Krishna
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184-2.patch, HADOOP-6184-3.patch, HADOOP-6184-4.patch, HADOOP-6184-5.patch, HADOOP-6184-6.patch, HADOOP-6184-7.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-936-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 21 11:23:15 2009
Return-Path: <common-issues-return-936-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6280 invoked from network); 21 Aug 2009 11:23:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Aug 2009 11:23:15 -0000
Received: (qmail 37753 invoked by uid 500); 21 Aug 2009 11:23:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37633 invoked by uid 500); 21 Aug 2009 11:23:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37546 invoked by uid 99); 21 Aug 2009 11:23:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 11:23:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 11:23:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 203DE234C4B1
	for <common-issues@hadoop.apache.org>; Fri, 21 Aug 2009 04:23:15 -0700 (PDT)
Message-ID: <750846820.1250853795131.JavaMail.jira@brutus>
Date: Fri, 21 Aug 2009 04:23:15 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6165) Add metadata to Serializations
In-Reply-To: <1308804598.1248261914878.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6165:
------------------------------

    Status: Patch Available  (was: Open)

> Add metadata to Serializations
> ------------------------------
>
>                 Key: HADOOP-6165
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6165
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: contrib/serialization
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6165-v2.patch, HADOOP-6165-v3.patch, HADOOP-6165-v4.patch, HADOOP-6165.patch
>
>
> The Serialization framework only allows a class to be passed as metadata. This assumes there is a one-to-one mapping between types and Serializations, which is overly restrictive. By permitting applications to pass arbitrary metadata to Serializations, they can get more control over which Serialization is used, and would also allow, for example, one to pass an Avro schema to an Avro Serialization.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-935-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 21 11:23:15 2009
Return-Path: <common-issues-return-935-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6324 invoked from network); 21 Aug 2009 11:23:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Aug 2009 11:23:15 -0000
Received: (qmail 37694 invoked by uid 500); 21 Aug 2009 11:23:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37572 invoked by uid 500); 21 Aug 2009 11:23:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37542 invoked by uid 99); 21 Aug 2009 11:23:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 11:23:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 11:23:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0E130234C4A7
	for <common-issues@hadoop.apache.org>; Fri, 21 Aug 2009 04:23:15 -0700 (PDT)
Message-ID: <1721753619.1250853795056.JavaMail.jira@brutus>
Date: Fri, 21 Aug 2009 04:23:15 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6165) Add metadata to Serializations
In-Reply-To: <1308804598.1248261914878.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6165:
------------------------------

    Status: Open  (was: Patch Available)

> Add metadata to Serializations
> ------------------------------
>
>                 Key: HADOOP-6165
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6165
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: contrib/serialization
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6165-v2.patch, HADOOP-6165-v3.patch, HADOOP-6165-v4.patch, HADOOP-6165.patch
>
>
> The Serialization framework only allows a class to be passed as metadata. This assumes there is a one-to-one mapping between types and Serializations, which is overly restrictive. By permitting applications to pass arbitrary metadata to Serializations, they can get more control over which Serialization is used, and would also allow, for example, one to pass an Avro schema to an Avro Serialization.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-937-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 21 11:23:16 2009
Return-Path: <common-issues-return-937-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6508 invoked from network); 21 Aug 2009 11:23:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Aug 2009 11:23:16 -0000
Received: (qmail 38193 invoked by uid 500); 21 Aug 2009 11:23:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38168 invoked by uid 500); 21 Aug 2009 11:23:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38158 invoked by uid 99); 21 Aug 2009 11:23:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 11:23:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 11:23:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E0011234C004
	for <common-issues@hadoop.apache.org>; Fri, 21 Aug 2009 04:23:14 -0700 (PDT)
Message-ID: <184176547.1250853794903.JavaMail.jira@brutus>
Date: Fri, 21 Aug 2009 04:23:14 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6165) Add metadata to Serializations
In-Reply-To: <1308804598.1248261914878.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6165:
------------------------------

    Attachment: HADOOP-6165-v4.patch

bq. Since AvroGenericSerialization is added last, the fallback could simply be to change its accept method to accept anything that has AVRO_SCHEMA_KEY defined, no?

Done.

bq. One other thing: we should probably adopt a naming convention for metadata keys. Should they be Java-package-like strings, e.g., org.apache.hadoop.io.serialization.class, or HTTP/SMTP header-like things, e.g., Serialization-Class?

In this case I don't have a strong feeling either way, so I've changed the keys to be named as "header-like things".

> Add metadata to Serializations
> ------------------------------
>
>                 Key: HADOOP-6165
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6165
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: contrib/serialization
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6165-v2.patch, HADOOP-6165-v3.patch, HADOOP-6165-v4.patch, HADOOP-6165.patch
>
>
> The Serialization framework only allows a class to be passed as metadata. This assumes there is a one-to-one mapping between types and Serializations, which is overly restrictive. By permitting applications to pass arbitrary metadata to Serializations, they can get more control over which Serialization is used, and would also allow, for example, one to pass an Avro schema to an Avro Serialization.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-938-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 21 11:25:15 2009
Return-Path: <common-issues-return-938-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7023 invoked from network); 21 Aug 2009 11:25:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Aug 2009 11:25:15 -0000
Received: (qmail 40785 invoked by uid 500); 21 Aug 2009 11:25:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40720 invoked by uid 500); 21 Aug 2009 11:25:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40413 invoked by uid 99); 21 Aug 2009 11:25:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 11:25:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 11:25:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E578B234C1EA
	for <common-issues@hadoop.apache.org>; Fri, 21 Aug 2009 04:25:14 -0700 (PDT)
Message-ID: <990268832.1250853914939.JavaMail.jira@brutus>
Date: Fri, 21 Aug 2009 04:25:14 -0700 (PDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6103) Configuration clone constructor
 does not clone all the members.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745907#action_12745907 ] 

Amareshwari Sriramadasu commented on HADOOP-6103:
-------------------------------------------------

I ran all mapreduce and hdfs tests on my machine with the build http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/63/. All tests ran except TestRecoveryManager (due to MAPREDUCE-880) and TestReduceFetch (due to MAPREDUCE-884).

> Configuration clone constructor does not clone all the members.
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6103
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6103
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.21.0
>
>         Attachments: patch-6103.txt
>
>
> Currently, Configuration(Configuration other) constructor does not clone all the members.
> It clones only resources, properties, overlay and finalParameters. It needs to clone loadDefaults, classLoader, defaultResources, quietmode.
> This resulted in bugs like HADOOP-4975

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-939-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 21 11:29:14 2009
Return-Path: <common-issues-return-939-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8014 invoked from network); 21 Aug 2009 11:29:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Aug 2009 11:29:14 -0000
Received: (qmail 45793 invoked by uid 500); 21 Aug 2009 11:29:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45732 invoked by uid 500); 21 Aug 2009 11:29:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45718 invoked by uid 99); 21 Aug 2009 11:29:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 11:29:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 11:29:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D1D6E234C004
	for <common-issues@hadoop.apache.org>; Fri, 21 Aug 2009 04:29:14 -0700 (PDT)
Message-ID: <965814804.1250854154845.JavaMail.jira@brutus>
Date: Fri, 21 Aug 2009 04:29:14 -0700 (PDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6200) glue together the different
 projects to make local builds easier
In-Reply-To: <68853708.1250609369522.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745908#action_12745908 ] 

Steve Loughran commented on HADOOP-6200:
----------------------------------------

When I tried to bump up the artifact version number by way of a shared {{build.properties}} file, hdfs was not happy, as in "refuses to build the base JAR not happy". Therefore, a cycle exists in the jar build process, even if the dependencies only come together at test time.
{code}
init:
    [mkdir] Created dir: /Users/slo/Java/Hadoop/lifecycle/hadoop-hdfs/build/classes
    [mkdir] Created dir: /Users/slo/Java/Hadoop/lifecycle/hadoop-hdfs/build/src
    [mkdir] Created dir: /Users/slo/Java/Hadoop/lifecycle/hadoop-hdfs/build/webapps/hdfs/WEB-INF
    [mkdir] Created dir: /Users/slo/Java/Hadoop/lifecycle/hadoop-hdfs/build/webapps/datanode/WEB-INF
    [mkdir] Created dir: /Users/slo/Java/Hadoop/lifecycle/hadoop-hdfs/build/webapps/secondary/WEB-INF
    [mkdir] Created dir: /Users/slo/Java/Hadoop/lifecycle/hadoop-hdfs/build/ant
    [mkdir] Created dir: /Users/slo/Java/Hadoop/lifecycle/hadoop-hdfs/build/test
    [mkdir] Created dir: /Users/slo/Java/Hadoop/lifecycle/hadoop-hdfs/build/test/classes
    [mkdir] Created dir: /Users/slo/Java/Hadoop/lifecycle/hadoop-hdfs/build/test/extraconf
    [touch] Creating /var/folders/6j/6jD1CUYiGs43jrHUr7BepU+++TI/-Tmp-/null2099585006
   [delete] Deleting: /var/folders/6j/6jD1CUYiGs43jrHUr7BepU+++TI/-Tmp-/null2099585006
     [copy] Copying 2 files to /Users/slo/Java/Hadoop/lifecycle/hadoop-hdfs/build/webapps

BUILD FAILED
/Users/slo/Java/Hadoop/lifecycle/hadoop-hdfs/build.xml:259: src '/Users/slo/Java/Hadoop/lifecycle/hadoop-hdfs/lib/hadoop-mapred-0.21.0-alpha-15.jar' doesn't exist.
{code}
I've documented what I've done to get the build working:
http://wiki.apache.org/hadoop/BuildingHadoopFromSVN

This has
* symbolic links to hook up the files. This is why you should be using a unix to build on.
* a boot process where you don't flip the version marker on mapreduce until you've build hdfs.

That wiki entry documents what we have today, its the starting point to what we have to do to simplify things. 
# We could have a shared base build.xml used by all projects, externally pulled in by -hdfs and -mapreduce
# I would like to use ivy to glue together stuff locally and remotely. For this to work we need at least one alpha release of 0.21 up in the big ibiblio repository
# the hdfs build could be tweaked to only bail out at test-time if the mapred JAR is missing, because that is the only time it is needed. 

> glue together the different projects to make local builds easier
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6200
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6200
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Priority: Minor
>
> It's currently fairly tricky to get everything building/up to date in a single machine. We can and should improve this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-940-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 21 11:31:14 2009
Return-Path: <common-issues-return-940-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8451 invoked from network); 21 Aug 2009 11:31:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Aug 2009 11:31:14 -0000
Received: (qmail 47693 invoked by uid 500); 21 Aug 2009 11:31:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47660 invoked by uid 500); 21 Aug 2009 11:31:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47650 invoked by uid 99); 21 Aug 2009 11:31:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 11:31:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 11:31:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C5FCA234C004
	for <common-issues@hadoop.apache.org>; Fri, 21 Aug 2009 04:31:14 -0700 (PDT)
Message-ID: <381053793.1250854274796.JavaMail.jira@brutus>
Date: Fri, 21 Aug 2009 04:31:14 -0700 (PDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6103) Configuration clone constructor
 does not clone all the members.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745909#action_12745909 ] 

Devaraj Das commented on HADOOP-6103:
-------------------------------------

Committed the common jar to mapreduce and hdfs.

> Configuration clone constructor does not clone all the members.
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6103
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6103
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.21.0
>
>         Attachments: patch-6103.txt
>
>
> Currently, Configuration(Configuration other) constructor does not clone all the members.
> It clones only resources, properties, overlay and finalParameters. It needs to clone loadDefaults, classLoader, defaultResources, quietmode.
> This resulted in bugs like HADOOP-4975

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-941-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 21 11:45:14 2009
Return-Path: <common-issues-return-941-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12442 invoked from network); 21 Aug 2009 11:45:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Aug 2009 11:45:14 -0000
Received: (qmail 70541 invoked by uid 500); 21 Aug 2009 11:45:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70478 invoked by uid 500); 21 Aug 2009 11:45:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70468 invoked by uid 99); 21 Aug 2009 11:45:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 11:45:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 11:45:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1D391234C4A7
	for <common-issues@hadoop.apache.org>; Fri, 21 Aug 2009 04:45:15 -0700 (PDT)
Message-ID: <766371184.1250855115118.JavaMail.jira@brutus>
Date: Fri, 21 Aug 2009 04:45:15 -0700 (PDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6184) Provide a configuration dump in json
 format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala updated HADOOP-6184:
-------------------------------------

    Status: Patch Available  (was: Open)

Looks good. +1. Running through Hudson

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>            Assignee: V.V.Chaitanya Krishna
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184-2.patch, HADOOP-6184-3.patch, HADOOP-6184-4.patch, HADOOP-6184-5.patch, HADOOP-6184-6.patch, HADOOP-6184-7.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-942-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 21 11:45:16 2009
Return-Path: <common-issues-return-942-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12549 invoked from network); 21 Aug 2009 11:45:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Aug 2009 11:45:16 -0000
Received: (qmail 70716 invoked by uid 500); 21 Aug 2009 11:45:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70660 invoked by uid 500); 21 Aug 2009 11:45:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70643 invoked by uid 99); 21 Aug 2009 11:45:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 11:45:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 11:45:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EE3DB234C1F1
	for <common-issues@hadoop.apache.org>; Fri, 21 Aug 2009 04:45:14 -0700 (PDT)
Message-ID: <1776825930.1250855114974.JavaMail.jira@brutus>
Date: Fri, 21 Aug 2009 04:45:14 -0700 (PDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6184) Provide a configuration dump in json
 format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala updated HADOOP-6184:
-------------------------------------

    Status: Open  (was: Patch Available)

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>            Assignee: V.V.Chaitanya Krishna
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184-2.patch, HADOOP-6184-3.patch, HADOOP-6184-4.patch, HADOOP-6184-5.patch, HADOOP-6184-6.patch, HADOOP-6184-7.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-943-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 21 13:01:17 2009
Return-Path: <common-issues-return-943-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54720 invoked from network); 21 Aug 2009 13:01:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Aug 2009 13:01:17 -0000
Received: (qmail 82037 invoked by uid 500); 21 Aug 2009 13:01:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81976 invoked by uid 500); 21 Aug 2009 13:01:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81966 invoked by uid 99); 21 Aug 2009 13:01:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 13:01:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 13:01:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 00FF3234C004
	for <common-issues@hadoop.apache.org>; Fri, 21 Aug 2009 06:01:15 -0700 (PDT)
Message-ID: <1977633573.1250859674993.JavaMail.jira@brutus>
Date: Fri, 21 Aug 2009 06:01:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6165) Add metadata to Serializations
In-Reply-To: <1308804598.1248261914878.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745953#action_12745953 ] 

Hadoop QA commented on HADOOP-6165:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12417254/HADOOP-6165-v4.patch
  against trunk revision 806430.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The applied patch generated 145 javac compiler warnings (more than the trunk's current 116 warnings).

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/619/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/619/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/619/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/619/console

This message is automatically generated.

> Add metadata to Serializations
> ------------------------------
>
>                 Key: HADOOP-6165
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6165
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: contrib/serialization
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6165-v2.patch, HADOOP-6165-v3.patch, HADOOP-6165-v4.patch, HADOOP-6165.patch
>
>
> The Serialization framework only allows a class to be passed as metadata. This assumes there is a one-to-one mapping between types and Serializations, which is overly restrictive. By permitting applications to pass arbitrary metadata to Serializations, they can get more control over which Serialization is used, and would also allow, for example, one to pass an Avro schema to an Avro Serialization.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-944-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 21 15:16:19 2009
Return-Path: <common-issues-return-944-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30679 invoked from network); 21 Aug 2009 15:16:18 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Aug 2009 15:16:18 -0000
Received: (qmail 66349 invoked by uid 500); 21 Aug 2009 15:16:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66318 invoked by uid 500); 21 Aug 2009 15:16:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66304 invoked by uid 99); 21 Aug 2009 15:16:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 15:16:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 15:16:37 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id A91E8234C1E9
	for <common-issues@hadoop.apache.org>; Fri, 21 Aug 2009 08:16:16 -0700 (PDT)
Message-ID: <2118015204.1250867776691.JavaMail.jira@brutus>
Date: Fri, 21 Aug 2009 08:16:16 -0700 (PDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6206) Proposed enhancements/tuning to
 hadoop-common/build.xml
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Proposed enhancements/tuning to hadoop-common/build.xml
-------------------------------------------------------

                 Key: HADOOP-6206
                 URL: https://issues.apache.org/jira/browse/HADOOP-6206
             Project: Hadoop Common
          Issue Type: Improvement
          Components: build
    Affects Versions: 0.21.0
            Reporter: Steve Loughran
            Assignee: Steve Loughran
            Priority: Minor


Minor enhancements to the build.xml of hadoop-common for better ivy integration/reporting

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-945-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 21 15:20:14 2009
Return-Path: <common-issues-return-945-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32312 invoked from network); 21 Aug 2009 15:20:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Aug 2009 15:20:14 -0000
Received: (qmail 75528 invoked by uid 500); 21 Aug 2009 15:20:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75484 invoked by uid 500); 21 Aug 2009 15:20:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75474 invoked by uid 99); 21 Aug 2009 15:20:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 15:20:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 15:20:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D9B07234C1EA
	for <common-issues@hadoop.apache.org>; Fri, 21 Aug 2009 08:20:14 -0700 (PDT)
Message-ID: <493183434.1250868014890.JavaMail.jira@brutus>
Date: Fri, 21 Aug 2009 08:20:14 -0700 (PDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6206) Proposed enhancements/tuning to
 hadoop-common/build.xml
In-Reply-To: <2118015204.1250867776691.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12746011#action_12746011 ] 

Steve Loughran commented on HADOOP-6206:
----------------------------------------

# make project name hadoop-common (in lower case), so that ivy-publish publishes the artifact in that case, to aid others to pick it up
# ivy-report to depend on ivy-resolve, so generating the reports for every configuration


> Proposed enhancements/tuning to hadoop-common/build.xml
> -------------------------------------------------------
>
>                 Key: HADOOP-6206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6206
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Minor
>
> Minor enhancements to the build.xml of hadoop-common for better ivy integration/reporting

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-946-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 21 15:36:18 2009
Return-Path: <common-issues-return-946-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43614 invoked from network); 21 Aug 2009 15:36:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Aug 2009 15:36:16 -0000
Received: (qmail 28797 invoked by uid 500); 21 Aug 2009 15:36:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28766 invoked by uid 500); 21 Aug 2009 15:36:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28756 invoked by uid 99); 21 Aug 2009 15:36:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 15:36:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 15:36:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1BA9C234C1EA
	for <common-issues@hadoop.apache.org>; Fri, 21 Aug 2009 08:36:15 -0700 (PDT)
Message-ID: <142401428.1250868975112.JavaMail.jira@brutus>
Date: Fri, 21 Aug 2009 08:36:15 -0700 (PDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Work started: (HADOOP-6206) Proposed enhancements/tuning to
 hadoop-common/build.xml
In-Reply-To: <2118015204.1250867776691.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Work on HADOOP-6206 started by Steve Loughran.

> Proposed enhancements/tuning to hadoop-common/build.xml
> -------------------------------------------------------
>
>                 Key: HADOOP-6206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6206
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Minor
>
> Minor enhancements to the build.xml of hadoop-common for better ivy integration/reporting

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-947-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 21 16:06:16 2009
Return-Path: <common-issues-return-947-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55607 invoked from network); 21 Aug 2009 16:06:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Aug 2009 16:06:16 -0000
Received: (qmail 85409 invoked by uid 500); 21 Aug 2009 16:06:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85376 invoked by uid 500); 21 Aug 2009 16:06:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85366 invoked by uid 99); 21 Aug 2009 16:06:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 16:06:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 16:06:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E02E9234C004
	for <common-issues@hadoop.apache.org>; Fri, 21 Aug 2009 09:06:14 -0700 (PDT)
Message-ID: <376141158.1250870774903.JavaMail.jira@brutus>
Date: Fri, 21 Aug 2009 09:06:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6184) Provide a configuration dump in
 json format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12746039#action_12746039 ] 

Hadoop QA commented on HADOOP-6184:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12417252/HADOOP-6184-7.patch
  against trunk revision 806430.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/620/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/620/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/620/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/620/console

This message is automatically generated.

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>            Assignee: V.V.Chaitanya Krishna
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184-2.patch, HADOOP-6184-3.patch, HADOOP-6184-4.patch, HADOOP-6184-5.patch, HADOOP-6184-6.patch, HADOOP-6184-7.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-948-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 21 16:32:17 2009
Return-Path: <common-issues-return-948-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66420 invoked from network); 21 Aug 2009 16:32:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Aug 2009 16:32:16 -0000
Received: (qmail 34535 invoked by uid 500); 21 Aug 2009 16:32:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34455 invoked by uid 500); 21 Aug 2009 16:32:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34438 invoked by uid 99); 21 Aug 2009 16:32:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 16:32:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 16:32:36 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 1D560234C4AB
	for <common-issues@hadoop.apache.org>; Fri, 21 Aug 2009 09:32:15 -0700 (PDT)
Message-ID: <879100088.1250872335119.JavaMail.jira@brutus>
Date: Fri, 21 Aug 2009 09:32:15 -0700 (PDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6206) Proposed enhancements/tuning to
 hadoop-common/build.xml
In-Reply-To: <2118015204.1250867776691.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12746068#action_12746068 ] 

Steve Loughran commented on HADOOP-6206:
----------------------------------------

Also, ivy/hadoop-core.pom should be moved/edited to become hadoop-common.pom; with appropriate changes in the build file

> Proposed enhancements/tuning to hadoop-common/build.xml
> -------------------------------------------------------
>
>                 Key: HADOOP-6206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6206
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Minor
>
> Minor enhancements to the build.xml of hadoop-common for better ivy integration/reporting

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-949-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 21 16:48:15 2009
Return-Path: <common-issues-return-949-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72164 invoked from network); 21 Aug 2009 16:48:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Aug 2009 16:48:15 -0000
Received: (qmail 54150 invoked by uid 500); 21 Aug 2009 16:48:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54056 invoked by uid 500); 21 Aug 2009 16:48:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53757 invoked by uid 99); 21 Aug 2009 16:48:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 16:48:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 16:48:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DF857234C1EB
	for <common-issues@hadoop.apache.org>; Fri, 21 Aug 2009 09:48:14 -0700 (PDT)
Message-ID: <1069048243.1250873294914.JavaMail.jira@brutus>
Date: Fri, 21 Aug 2009 09:48:14 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6206) Proposed enhancements/tuning to
 hadoop-common/build.xml
In-Reply-To: <2118015204.1250867776691.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12746075#action_12746075 ] 

Giridharan Kesavan commented on HADOOP-6206:
--------------------------------------------

On a similar not Im working on ant-maven-task that would publish common, hdfs and mapreduce jars to the apache maven repo.
https://issues.apache.org/jira/browse/HADOOP-5107
I 've already published common snapshot jars to the apache repo using the ant maven task.





> Proposed enhancements/tuning to hadoop-common/build.xml
> -------------------------------------------------------
>
>                 Key: HADOOP-6206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6206
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Minor
>
> Minor enhancements to the build.xml of hadoop-common for better ivy integration/reporting

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-950-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 21 16:50:17 2009
Return-Path: <common-issues-return-950-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73021 invoked from network); 21 Aug 2009 16:50:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Aug 2009 16:50:14 -0000
Received: (qmail 56790 invoked by uid 500); 21 Aug 2009 16:50:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56756 invoked by uid 500); 21 Aug 2009 16:50:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56664 invoked by uid 99); 21 Aug 2009 16:50:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 16:50:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 16:50:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 19E75234C4AC
	for <common-issues@hadoop.apache.org>; Fri, 21 Aug 2009 09:50:15 -0700 (PDT)
Message-ID: <1914736587.1250873415105.JavaMail.jira@brutus>
Date: Fri, 21 Aug 2009 09:50:15 -0700 (PDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6206) Proposed enhancements/tuning to
 hadoop-common/build.xml
In-Reply-To: <2118015204.1250867776691.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12746080#action_12746080 ] 

Steve Loughran commented on HADOOP-6206:
----------------------------------------

# move the hard coded version numbers for avro, jackson-mapper.asl and panamer into libraries.properties, add matching entries to the POM

> Proposed enhancements/tuning to hadoop-common/build.xml
> -------------------------------------------------------
>
>                 Key: HADOOP-6206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6206
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Minor
>
> Minor enhancements to the build.xml of hadoop-common for better ivy integration/reporting

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-951-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 21 17:00:15 2009
Return-Path: <common-issues-return-951-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77277 invoked from network); 21 Aug 2009 17:00:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 21 Aug 2009 17:00:14 -0000
Received: (qmail 75466 invoked by uid 500); 21 Aug 2009 17:00:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75436 invoked by uid 500); 21 Aug 2009 17:00:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75426 invoked by uid 99); 21 Aug 2009 17:00:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 17:00:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Aug 2009 17:00:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CA9E2234C1E9
	for <common-issues@hadoop.apache.org>; Fri, 21 Aug 2009 10:00:14 -0700 (PDT)
Message-ID: <508477156.1250874014818.JavaMail.jira@brutus>
Date: Fri, 21 Aug 2009 10:00:14 -0700 (PDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6206) Proposed enhancements/tuning to
 hadoop-common/build.xml
In-Reply-To: <2118015204.1250867776691.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12746087#action_12746087 ] 

Steve Loughran commented on HADOOP-6206:
----------------------------------------

gkesavan - I'm playing with this in my own branch, initially just use Ivy to glue together the outputs of the 3 projects into my own build, all via ivy. I can help with the POM files. 

> Proposed enhancements/tuning to hadoop-common/build.xml
> -------------------------------------------------------
>
>                 Key: HADOOP-6206
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6206
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Minor
>
> Minor enhancements to the build.xml of hadoop-common for better ivy integration/reporting

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-971-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 22 07:59:14 2009
Return-Path: <common-issues-return-971-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59190 invoked from network); 22 Aug 2009 07:59:14 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Aug 2009 07:59:14 -0000
Received: (qmail 71750 invoked by uid 500); 22 Aug 2009 07:59:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71693 invoked by uid 500); 22 Aug 2009 07:59:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71683 invoked by uid 99); 22 Aug 2009 07:59:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 Aug 2009 07:59:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 Aug 2009 07:59:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CD049234C004
	for <common-issues@hadoop.apache.org>; Sat, 22 Aug 2009 00:59:14 -0700 (PDT)
Message-ID: <1053899613.1250927954825.JavaMail.jira@brutus>
Date: Sat, 22 Aug 2009 00:59:14 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6152) Hadoop scripts do not correctly put
 jars on the classpath
In-Reply-To: <283943906.1247689875148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12746425#action_12746425 ] 

Hadoop QA commented on HADOOP-6152:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12417331/HADOOP-6152.5.patch
  against trunk revision 806745.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 2 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/623/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/623/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/623/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/623/console

This message is automatically generated.

> Hadoop scripts do not correctly put jars on the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-6152
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6152
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6152.2.patch, HADOOP-6152.3.patch, HADOOP-6152.4.patch, HADOOP-6152.5.patch, HADOOP-6152.patch
>
>
> The various Hadoop scripts (bin/hadoop, bin/hdfs, bin/mapred) do not properly identify the jars needed to run Hadoop. They try to include hadoop-*-hdfs.jar, etc, rather than the hadoop-hdfs-*.jar that is actually created by the 'ant jar' and 'ant package' targets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-972-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 22 12:40:15 2009
Return-Path: <common-issues-return-972-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96360 invoked from network); 22 Aug 2009 12:40:15 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Aug 2009 12:40:15 -0000
Received: (qmail 11217 invoked by uid 500); 22 Aug 2009 12:40:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11157 invoked by uid 500); 22 Aug 2009 12:40:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11147 invoked by uid 99); 22 Aug 2009 12:40:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 Aug 2009 12:40:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 Aug 2009 12:40:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 07903234C004
	for <common-issues@hadoop.apache.org>; Sat, 22 Aug 2009 05:40:15 -0700 (PDT)
Message-ID: <311603914.1250944815014.JavaMail.jira@brutus>
Date: Sat, 22 Aug 2009 05:40:15 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6184) Provide a configuration dump in
 json format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12746446#action_12746446 ] 

Hadoop QA commented on HADOOP-6184:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12417351/HADOOP-6184-8.patch
  against trunk revision 806745.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/624/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/624/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/624/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/624/console

This message is automatically generated.

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>            Assignee: V.V.Chaitanya Krishna
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184-2.patch, HADOOP-6184-3.patch, HADOOP-6184-4.patch, HADOOP-6184-5.patch, HADOOP-6184-6.patch, HADOOP-6184-7.patch, HADOOP-6184-8.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-973-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 22 16:16:18 2009
Return-Path: <common-issues-return-973-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28411 invoked from network); 22 Aug 2009 16:16:17 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 22 Aug 2009 16:16:17 -0000
Received: (qmail 36294 invoked by uid 500); 22 Aug 2009 16:16:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36249 invoked by uid 500); 22 Aug 2009 16:16:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36239 invoked by uid 99); 22 Aug 2009 16:16:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 Aug 2009 16:16:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 Aug 2009 16:16:35 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D222F234C004
	for <common-issues@hadoop.apache.org>; Sat, 22 Aug 2009 09:16:14 -0700 (PDT)
Message-ID: <1471667570.1250957774846.JavaMail.jira@brutus>
Date: Sat, 22 Aug 2009 09:16:14 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6201) FileSystem::ListStatus should throw
 FileNotFoundException
In-Reply-To: <1093368448.1250619614937.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12746476#action_12746476 ] 

Hudson commented on HADOOP-6201:
--------------------------------

Integrated in Hadoop-Common-trunk #65 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/65/])
    . Change FileSystem::listStatus contract to throw
FileNotFoundException if the directory does not exist, rather than letting
this be implementation-specific. Contributed by Jakob Homan.


> FileSystem::ListStatus should throw FileNotFoundException
> ---------------------------------------------------------
>
>                 Key: HADOOP-6201
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6201
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6201.patch
>
>
> As discussed in HDFS-538, it would be better for FileSystem and its implementations to throw FileNotFoundException when the file is not found, rather than returning null.  This will bring it in line with getFileStatus and, I expect, default user expectations.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-974-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 24 05:21:02 2009
Return-Path: <common-issues-return-974-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88703 invoked from network); 24 Aug 2009 05:21:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Aug 2009 05:21:01 -0000
Received: (qmail 95502 invoked by uid 500); 24 Aug 2009 05:21:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95442 invoked by uid 500); 24 Aug 2009 05:21:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95432 invoked by uid 99); 24 Aug 2009 05:21:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 05:21:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 05:21:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4A458234C004
	for <common-issues@hadoop.apache.org>; Sun, 23 Aug 2009 22:20:59 -0700 (PDT)
Message-ID: <1741244635.1251091259299.JavaMail.jira@brutus>
Date: Sun, 23 Aug 2009 22:20:59 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6184) Provide a configuration dump in json
 format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna updated HADOOP-6184:
------------------------------------------

    Attachment: HADOOP-6184-9.patch

As pointed by  Steve Loughran in MAPREDUCE-768, the expansion of configuration properties needs to be done. Uploading the patch with this being implemented.

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>            Assignee: V.V.Chaitanya Krishna
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184-2.patch, HADOOP-6184-3.patch, HADOOP-6184-4.patch, HADOOP-6184-5.patch, HADOOP-6184-6.patch, HADOOP-6184-7.patch, HADOOP-6184-8.patch, HADOOP-6184-9.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-975-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 24 06:19:02 2009
Return-Path: <common-issues-return-975-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94792 invoked from network); 24 Aug 2009 06:19:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Aug 2009 06:19:02 -0000
Received: (qmail 30461 invoked by uid 500); 24 Aug 2009 06:19:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30399 invoked by uid 500); 24 Aug 2009 06:19:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30381 invoked by uid 99); 24 Aug 2009 06:19:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 06:19:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 06:19:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4D552234C004
	for <common-issues@hadoop.apache.org>; Sun, 23 Aug 2009 23:18:59 -0700 (PDT)
Message-ID: <1352202564.1251094739301.JavaMail.jira@brutus>
Date: Sun, 23 Aug 2009 23:18:59 -0700 (PDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6184) Provide a configuration dump in json
 format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala updated HADOOP-6184:
-------------------------------------

    Status: Open  (was: Patch Available)

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>            Assignee: V.V.Chaitanya Krishna
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184-2.patch, HADOOP-6184-3.patch, HADOOP-6184-4.patch, HADOOP-6184-5.patch, HADOOP-6184-6.patch, HADOOP-6184-7.patch, HADOOP-6184-8.patch, HADOOP-6184-9.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-976-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 24 06:19:04 2009
Return-Path: <common-issues-return-976-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94856 invoked from network); 24 Aug 2009 06:19:04 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Aug 2009 06:19:04 -0000
Received: (qmail 30557 invoked by uid 500); 24 Aug 2009 06:19:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30495 invoked by uid 500); 24 Aug 2009 06:19:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30481 invoked by uid 99); 24 Aug 2009 06:19:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 06:19:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 06:19:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 74CE5234C1E6
	for <common-issues@hadoop.apache.org>; Sun, 23 Aug 2009 23:18:59 -0700 (PDT)
Message-ID: <1739748937.1251094739477.JavaMail.jira@brutus>
Date: Sun, 23 Aug 2009 23:18:59 -0700 (PDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6184) Provide a configuration dump in json
 format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala updated HADOOP-6184:
-------------------------------------

    Status: Patch Available  (was: Open)

Changes look fine to me. Trying Hudson again.

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>            Assignee: V.V.Chaitanya Krishna
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184-2.patch, HADOOP-6184-3.patch, HADOOP-6184-4.patch, HADOOP-6184-5.patch, HADOOP-6184-6.patch, HADOOP-6184-7.patch, HADOOP-6184-8.patch, HADOOP-6184-9.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-977-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 24 06:25:02 2009
Return-Path: <common-issues-return-977-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95321 invoked from network); 24 Aug 2009 06:25:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Aug 2009 06:25:02 -0000
Received: (qmail 32457 invoked by uid 500); 24 Aug 2009 06:25:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32400 invoked by uid 500); 24 Aug 2009 06:25:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32390 invoked by uid 99); 24 Aug 2009 06:25:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 06:25:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 06:25:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 54F20234C004
	for <common-issues@hadoop.apache.org>; Sun, 23 Aug 2009 23:24:59 -0700 (PDT)
Message-ID: <1300612678.1251095099333.JavaMail.jira@brutus>
Date: Sun, 23 Aug 2009 23:24:59 -0700 (PDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5073) Hadoop 1.0 Interface Classification
 - scope (visibility - public/private) and stability
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12746744#action_12746744 ] 

Sanjay Radia commented on HADOOP-5073:
--------------------------------------

Doug said:
>The implication on documentation should be something like:
>    * Public/stable should be documented.
>    * Public/evolving should documented, but must be well-marked as evolving.
>    * Private, limited or otherwise, should not be included in public documentation, but only in internal documents.

Agreed, However private/stable and private/evolving also need to be document even though they 
are not visible in the public documentation, only in the developer documentation.
Developers need to understand which private/internal interfaces are stable since their stability needs to be maintained across releases, For example the client protocols need to be stable even though they are
private for wire compatibility.


> Hadoop 1.0 Interface Classification - scope (visibility - public/private) and stability
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5073
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5073
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: 5073_demo.png, HADOOP-5073.patch
>
>
> This jira proposes an interface classification for hadoop interfaces.
> The discussion was started in email alias core-dev@hadoop.apache.org in Nov 2008.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-980-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 24 16:35:37 2009
Return-Path: <common-issues-return-980-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9411 invoked from network); 24 Aug 2009 16:35:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Aug 2009 16:35:36 -0000
Received: (qmail 53077 invoked by uid 500); 24 Aug 2009 11:09:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53018 invoked by uid 500); 24 Aug 2009 11:09:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52992 invoked by uid 99); 24 Aug 2009 11:09:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 11:09:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 11:09:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 625F5234C056
	for <common-issues@hadoop.apache.org>; Mon, 24 Aug 2009 04:08:59 -0700 (PDT)
Message-ID: <445952209.1251112139402.JavaMail.jira@brutus>
Date: Mon, 24 Aug 2009 04:08:59 -0700 (PDT)
From: "kate (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5945) Support running multiple
 DataNodes/TaskTrackers simultaneously in a single node
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12746815#action_12746815 ] 

kate commented on HADOOP-5945:
------------------------------

Hello! How can I use the DataNodeCluster tool to run multiple datanodes in one node?

Thx, Kate

> Support running multiple DataNodes/TaskTrackers simultaneously in a single node
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-5945
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5945
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: He Yongqiang
>
> We should support multiple datanodes/tasktrackers running at a same node, only if they do not share same port/local fs dir etc. I think Hadoop can be easily adapted to meet this.  
> I guess at first and the major step is that we should modify the script to let it support startting multiple datanode/tasktracker daemons in a same node.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-978-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 24 16:40:38 2009
Return-Path: <common-issues-return-978-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18266 invoked from network); 24 Aug 2009 16:40:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Aug 2009 16:40:38 -0000
Received: (qmail 13603 invoked by uid 500); 24 Aug 2009 09:34:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13544 invoked by uid 500); 24 Aug 2009 09:34:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13534 invoked by uid 99); 24 Aug 2009 09:34:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 09:34:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 09:34:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id A3537234C1E9
	for <common-issues@hadoop.apache.org>; Mon, 24 Aug 2009 02:33:59 -0700 (PDT)
Message-ID: <703610395.1251106439668.JavaMail.jira@brutus>
Date: Mon, 24 Aug 2009 02:33:59 -0700 (PDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6184) Provide a configuration dump in
 json format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12746787#action_12746787 ] 

Hemanth Yamijala commented on HADOOP-6184:
------------------------------------------

When we complete HADOOP-6105, I can think of a useful enhancement where we can dump if some deprecated keys are being defined in the configuration. 

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>            Assignee: V.V.Chaitanya Krishna
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184-2.patch, HADOOP-6184-3.patch, HADOOP-6184-4.patch, HADOOP-6184-5.patch, HADOOP-6184-6.patch, HADOOP-6184-7.patch, HADOOP-6184-8.patch, HADOOP-6184-9.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-981-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 24 16:51:38 2009
Return-Path: <common-issues-return-981-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33167 invoked from network); 24 Aug 2009 16:51:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Aug 2009 16:51:37 -0000
Received: (qmail 67174 invoked by uid 500); 24 Aug 2009 11:25:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67122 invoked by uid 500); 24 Aug 2009 11:25:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67102 invoked by uid 99); 24 Aug 2009 11:25:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 11:25:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 11:25:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 481D2234C004
	for <common-issues@hadoop.apache.org>; Mon, 24 Aug 2009 04:24:59 -0700 (PDT)
Message-ID: <333673332.1251113099281.JavaMail.jira@brutus>
Date: Mon, 24 Aug 2009 04:24:59 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6184) Provide a configuration dump in json
 format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna updated HADOOP-6184:
------------------------------------------

    Attachment: HADOOP-6184-ydist.patch

uploading patch for Yahoo! distribution.

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>            Assignee: V.V.Chaitanya Krishna
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184-2.patch, HADOOP-6184-3.patch, HADOOP-6184-4.patch, HADOOP-6184-5.patch, HADOOP-6184-6.patch, HADOOP-6184-7.patch, HADOOP-6184-8.patch, HADOOP-6184-9.patch, HADOOP-6184-ydist.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-984-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 24 16:58:37 2009
Return-Path: <common-issues-return-984-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44128 invoked from network); 24 Aug 2009 16:58:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Aug 2009 16:58:36 -0000
Received: (qmail 30168 invoked by uid 500); 24 Aug 2009 14:12:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30111 invoked by uid 500); 24 Aug 2009 14:12:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30098 invoked by uid 99); 24 Aug 2009 14:12:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 14:12:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 14:12:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 6074F234C04C
	for <common-issues@hadoop.apache.org>; Mon, 24 Aug 2009 07:11:59 -0700 (PDT)
Message-ID: <966702947.1251123119380.JavaMail.jira@brutus>
Date: Mon, 24 Aug 2009 07:11:59 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5889) Allow writing to output directories
 that exist, as long as they are empty
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12746884#action_12746884 ] 

Tom White commented on HADOOP-5889:
-----------------------------------

I don't see why we wouldn't make this change to both old and new APIs. 

There is a precedent for having the same test for the old and new APIs (e.g. the one for LazyOutput), so yes, I would create a new org.apache.hadoop.mapreduce.lib.output.TestFileOutputFormat.

> Allow writing to output directories that exist, as long as they are empty
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-5889
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5889
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.18.3
>            Reporter: Ian Nowland
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5889-0.patch
>
>
> The current behavior in FileOutputFormat.checkOutputSpecs is to fail if the path specified by mapred.output.dir exists at the start of the job. This is to protect from accidentally overwriting existing data. There seems no harm then in slightly relaxing this check to allow the case for the output to exist if it is an empty directory.
> At a minimum this would allow outputting to the root of S3N buckets, which is currently impossible (https://issues.apache.org/jira/browse/HADOOP-5805).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-985-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 24 16:58:38 2009
Return-Path: <common-issues-return-985-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44299 invoked from network); 24 Aug 2009 16:58:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Aug 2009 16:58:38 -0000
Received: (qmail 30999 invoked by uid 500); 24 Aug 2009 14:12:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30953 invoked by uid 500); 24 Aug 2009 14:12:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30943 invoked by uid 99); 24 Aug 2009 14:12:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 14:12:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 14:12:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 79E10234C1F0
	for <common-issues@hadoop.apache.org>; Mon, 24 Aug 2009 07:11:59 -0700 (PDT)
Message-ID: <1802874054.1251123119498.JavaMail.jira@brutus>
Date: Mon, 24 Aug 2009 07:11:59 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6009) S3N listStatus incorrectly returns
 null instead of empty array when called on empty root
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White reassigned HADOOP-6009:
---------------------------------

    Assignee: Ian Nowland

> S3N listStatus incorrectly returns null instead of empty array when called on empty root
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6009
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6009
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 0.18.3
>            Reporter: Ian Nowland
>            Assignee: Ian Nowland
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6009-0.patch
>
>
> Null means the directory does not exist, which is obviously not the case.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-979-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 24 16:58:38 2009
Return-Path: <common-issues-return-979-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44298 invoked from network); 24 Aug 2009 16:58:38 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Aug 2009 16:58:38 -0000
Received: (qmail 57402 invoked by uid 500); 24 Aug 2009 09:52:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57349 invoked by uid 500); 24 Aug 2009 09:52:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57339 invoked by uid 99); 24 Aug 2009 09:52:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 09:52:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 09:52:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 46EBD234C004
	for <common-issues@hadoop.apache.org>; Mon, 24 Aug 2009 02:51:59 -0700 (PDT)
Message-ID: <1068115433.1251107519276.JavaMail.jira@brutus>
Date: Mon, 24 Aug 2009 02:51:59 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6184) Provide a configuration dump in
 json format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12746794#action_12746794 ] 

Hadoop QA commented on HADOOP-6184:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12417439/HADOOP-6184-9.patch
  against trunk revision 806745.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/625/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/625/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/625/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/625/console

This message is automatically generated.

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>            Assignee: V.V.Chaitanya Krishna
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184-2.patch, HADOOP-6184-3.patch, HADOOP-6184-4.patch, HADOOP-6184-5.patch, HADOOP-6184-6.patch, HADOOP-6184-7.patch, HADOOP-6184-8.patch, HADOOP-6184-9.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-986-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 24 17:00:36 2009
Return-Path: <common-issues-return-986-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47736 invoked from network); 24 Aug 2009 17:00:35 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Aug 2009 17:00:35 -0000
Received: (qmail 34153 invoked by uid 500); 24 Aug 2009 14:14:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34094 invoked by uid 500); 24 Aug 2009 14:14:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34084 invoked by uid 99); 24 Aug 2009 14:14:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 14:14:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 14:14:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 504FD234C04C
	for <common-issues@hadoop.apache.org>; Mon, 24 Aug 2009 07:13:59 -0700 (PDT)
Message-ID: <114500089.1251123239313.JavaMail.jira@brutus>
Date: Mon, 24 Aug 2009 07:13:59 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6009) S3N listStatus incorrectly returns
 null instead of empty array when called on empty root
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6009:
------------------------------

    Hadoop Flags: [Reviewed]
          Status: Patch Available  (was: Open)

+1 looks good to me.

> S3N listStatus incorrectly returns null instead of empty array when called on empty root
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6009
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6009
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 0.18.3
>            Reporter: Ian Nowland
>            Assignee: Ian Nowland
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6009-0.patch
>
>
> Null means the directory does not exist, which is obviously not the case.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-982-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 24 17:13:37 2009
Return-Path: <common-issues-return-982-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65498 invoked from network); 24 Aug 2009 17:13:37 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Aug 2009 17:13:37 -0000
Received: (qmail 88901 invoked by uid 500); 24 Aug 2009 11:47:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88873 invoked by uid 500); 24 Aug 2009 11:47:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88863 invoked by uid 99); 24 Aug 2009 11:47:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 11:47:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 11:47:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 46733234C004
	for <common-issues@hadoop.apache.org>; Mon, 24 Aug 2009 04:46:59 -0700 (PDT)
Message-ID: <378583755.1251114419274.JavaMail.jira@brutus>
Date: Mon, 24 Aug 2009 04:46:59 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6184) Provide a configuration dump in json
 format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna updated HADOOP-6184:
------------------------------------------

    Attachment: HADOOP-6184-ydist-1.patch

Uploading new patch for Yahoo! distribution with the compiler error fixed.

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>            Assignee: V.V.Chaitanya Krishna
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184-2.patch, HADOOP-6184-3.patch, HADOOP-6184-4.patch, HADOOP-6184-5.patch, HADOOP-6184-6.patch, HADOOP-6184-7.patch, HADOOP-6184-8.patch, HADOOP-6184-9.patch, HADOOP-6184-ydist-1.patch, HADOOP-6184-ydist.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-987-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 24 17:14:37 2009
Return-Path: <common-issues-return-987-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66218 invoked from network); 24 Aug 2009 17:14:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Aug 2009 17:14:36 -0000
Received: (qmail 64935 invoked by uid 500); 24 Aug 2009 14:28:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64879 invoked by uid 500); 24 Aug 2009 14:28:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64864 invoked by uid 99); 24 Aug 2009 14:28:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 14:28:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 14:28:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 50EE4234C1E6
	for <common-issues@hadoop.apache.org>; Mon, 24 Aug 2009 07:27:59 -0700 (PDT)
Message-ID: <276097209.1251124079317.JavaMail.jira@brutus>
Date: Mon, 24 Aug 2009 07:27:59 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6184) Provide a configuration dump in
 json format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12746893#action_12746893 ] 

Hudson commented on HADOOP-6184:
--------------------------------

Integrated in Hadoop-Common-trunk #68 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/68/])
    . Provide an API to dump Configuration in a JSON format. Contributed by V.V.Chaitanya Krishna.


> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>            Assignee: V.V.Chaitanya Krishna
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184-2.patch, HADOOP-6184-3.patch, HADOOP-6184-4.patch, HADOOP-6184-5.patch, HADOOP-6184-6.patch, HADOOP-6184-7.patch, HADOOP-6184-8.patch, HADOOP-6184-9.patch, HADOOP-6184-ydist-1.patch, HADOOP-6184-ydist.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-983-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 24 17:17:42 2009
Return-Path: <common-issues-return-983-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69517 invoked from network); 24 Aug 2009 17:17:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Aug 2009 17:17:42 -0000
Received: (qmail 91467 invoked by uid 500); 24 Aug 2009 11:51:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91409 invoked by uid 500); 24 Aug 2009 11:51:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91399 invoked by uid 99); 24 Aug 2009 11:51:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 11:51:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 11:51:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 805C1234C1EC
	for <common-issues@hadoop.apache.org>; Mon, 24 Aug 2009 04:50:59 -0700 (PDT)
Message-ID: <749233397.1251114659524.JavaMail.jira@brutus>
Date: Mon, 24 Aug 2009 04:50:59 -0700 (PDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6184) Provide a configuration dump in json
 format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala updated HADOOP-6184:
-------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.21.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

I just committed this to trunk. Thanks, Chaitanya !

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>            Assignee: V.V.Chaitanya Krishna
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184-2.patch, HADOOP-6184-3.patch, HADOOP-6184-4.patch, HADOOP-6184-5.patch, HADOOP-6184-6.patch, HADOOP-6184-7.patch, HADOOP-6184-8.patch, HADOOP-6184-9.patch, HADOOP-6184-ydist-1.patch, HADOOP-6184-ydist.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-988-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 24 17:55:58 2009
Return-Path: <common-issues-return-988-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6390 invoked from network); 24 Aug 2009 17:55:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Aug 2009 17:55:58 -0000
Received: (qmail 59388 invoked by uid 500); 24 Aug 2009 17:56:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59329 invoked by uid 500); 24 Aug 2009 17:56:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59319 invoked by uid 99); 24 Aug 2009 17:56:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 17:56:23 +0000
X-ASF-Spam-Status: No, hits=-1998.5 required=10.0
	tests=ALL_TRUSTED,WEIRD_PORT
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 17:56:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 66EA1234C1E7
	for <common-issues@hadoop.apache.org>; Mon, 24 Aug 2009 10:55:59 -0700 (PDT)
Message-ID: <1603486505.1251136559420.JavaMail.jira@brutus>
Date: Mon, 24 Aug 2009 10:55:59 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6152) Hadoop scripts do not correctly put
 jars on the classpath
In-Reply-To: <283943906.1247689875148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12746979#action_12746979 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6152:
------------------------------------------------

> Prove that it works: ...
Hi Aaron, I tried an earlier version of the patch (with some additional fixes).  Everything works fine except for the jsp web pages (e.g. http://namenode:50070/dfshealth.jsp).  Have you checked it with your latest patch?

+1 on the latest patch which fixes many problems.   We may commit it first and fix the jsp problem, if there is any, in a new jira.

> Hadoop scripts do not correctly put jars on the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-6152
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6152
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6152.2.patch, HADOOP-6152.3.patch, HADOOP-6152.4.patch, HADOOP-6152.5.patch, HADOOP-6152.patch
>
>
> The various Hadoop scripts (bin/hadoop, bin/hdfs, bin/mapred) do not properly identify the jars needed to run Hadoop. They try to include hadoop-*-hdfs.jar, etc, rather than the hadoop-hdfs-*.jar that is actually created by the 'ant jar' and 'ant package' targets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-989-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 24 18:21:56 2009
Return-Path: <common-issues-return-989-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17853 invoked from network); 24 Aug 2009 18:21:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Aug 2009 18:21:56 -0000
Received: (qmail 94143 invoked by uid 500); 24 Aug 2009 18:22:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94083 invoked by uid 500); 24 Aug 2009 18:22:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94073 invoked by uid 99); 24 Aug 2009 18:22:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 18:22:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 18:22:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 57BEE234C1EB
	for <common-issues@hadoop.apache.org>; Mon, 24 Aug 2009 11:21:59 -0700 (PDT)
Message-ID: <394456699.1251138119358.JavaMail.jira@brutus>
Date: Mon, 24 Aug 2009 11:21:59 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6009) S3N listStatus incorrectly returns
 null instead of empty array when called on empty root
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12746989#action_12746989 ] 

Hadoop QA commented on HADOOP-6009:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12410346/HADOOP-6009-0.patch
  against trunk revision 807149.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 5 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/626/console

This message is automatically generated.

> S3N listStatus incorrectly returns null instead of empty array when called on empty root
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6009
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6009
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 0.18.3
>            Reporter: Ian Nowland
>            Assignee: Ian Nowland
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6009-0.patch
>
>
> Null means the directory does not exist, which is obviously not the case.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-991-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 24 18:31:33 2009
Return-Path: <common-issues-return-991-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19535 invoked from network); 24 Aug 2009 18:31:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Aug 2009 18:31:33 -0000
Received: (qmail 2448 invoked by uid 500); 24 Aug 2009 18:31:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2417 invoked by uid 500); 24 Aug 2009 18:31:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2407 invoked by uid 99); 24 Aug 2009 18:31:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 18:31:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 18:31:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 7C74A234C4AA
	for <common-issues@hadoop.apache.org>; Mon, 24 Aug 2009 11:30:59 -0700 (PDT)
Message-ID: <1206124269.1251138659509.JavaMail.jira@brutus>
Date: Mon, 24 Aug 2009 11:30:59 -0700 (PDT)
From: "Aaron Kimball (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6152) Hadoop scripts do not correctly put
 jars on the classpath
In-Reply-To: <283943906.1247689875148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12746994#action_12746994 ] 

Aaron Kimball commented on HADOOP-6152:
---------------------------------------

This doesn't fix the JSP issue. I think that'll require a separate issue. I've gotten closer to fixing some JSP build-process stuff in my MAPREDUCE-679 and HDFS-453 patches which are as-yet uncommitted. But they still don't totally fix the issue. (To test them, I am manually copying the build/webapps/ and build/classes/ hierarchies on top of one another.)

> Hadoop scripts do not correctly put jars on the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-6152
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6152
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6152.2.patch, HADOOP-6152.3.patch, HADOOP-6152.4.patch, HADOOP-6152.5.patch, HADOOP-6152.patch
>
>
> The various Hadoop scripts (bin/hadoop, bin/hdfs, bin/mapred) do not properly identify the jars needed to run Hadoop. They try to include hadoop-*-hdfs.jar, etc, rather than the hadoop-hdfs-*.jar that is actually created by the 'ant jar' and 'ant package' targets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-990-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 24 18:31:55 2009
Return-Path: <common-issues-return-990-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19726 invoked from network); 24 Aug 2009 18:31:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Aug 2009 18:31:54 -0000
Received: (qmail 2044 invoked by uid 500); 24 Aug 2009 18:31:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1983 invoked by uid 500); 24 Aug 2009 18:31:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1899 invoked by uid 99); 24 Aug 2009 18:31:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 18:31:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 18:31:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 70CAA234C1F2
	for <common-issues@hadoop.apache.org>; Mon, 24 Aug 2009 11:30:59 -0700 (PDT)
Message-ID: <2133703492.1251138659461.JavaMail.jira@brutus>
Date: Mon, 24 Aug 2009 11:30:59 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6201) FileSystem::ListStatus should throw
 FileNotFoundException
In-Reply-To: <1093368448.1250619614937.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6201:
--------------------------------

    Release Note: 
Solidify FileSystem contract for listStatus method when the provided path does not exist. The method is now guaranteed to throw a FileNotFoundException, whereas some FileSystem implementations had previously returned null in this case. Existing code that relies on checking for null should be updated to catch or propagate a FileNotFoundException.


Adding release note.

> FileSystem::ListStatus should throw FileNotFoundException
> ---------------------------------------------------------
>
>                 Key: HADOOP-6201
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6201
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6201.patch
>
>
> As discussed in HDFS-538, it would be better for FileSystem and its implementations to throw FileNotFoundException when the file is not found, rather than returning null.  This will bring it in line with getFileStatus and, I expect, default user expectations.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-992-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 24 22:07:58 2009
Return-Path: <common-issues-return-992-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33887 invoked from network); 24 Aug 2009 22:07:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 24 Aug 2009 22:07:58 -0000
Received: (qmail 73893 invoked by uid 500); 24 Aug 2009 22:08:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73784 invoked by uid 500); 24 Aug 2009 22:08:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73737 invoked by uid 99); 24 Aug 2009 22:08:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 22:08:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Aug 2009 22:08:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 6B9DA234C04C
	for <common-issues@hadoop.apache.org>; Mon, 24 Aug 2009 15:07:59 -0700 (PDT)
Message-ID: <769809492.1251151679426.JavaMail.jira@brutus>
Date: Mon, 24 Aug 2009 15:07:59 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6152) Hadoop scripts do not correctly put
 jars on the classpath
In-Reply-To: <283943906.1247689875148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6152:
-------------------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I have committed this.  Thanks, Aaron!

> Hadoop scripts do not correctly put jars on the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-6152
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6152
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6152.2.patch, HADOOP-6152.3.patch, HADOOP-6152.4.patch, HADOOP-6152.5.patch, HADOOP-6152.patch
>
>
> The various Hadoop scripts (bin/hadoop, bin/hdfs, bin/mapred) do not properly identify the jars needed to run Hadoop. They try to include hadoop-*-hdfs.jar, etc, rather than the hadoop-hdfs-*.jar that is actually created by the 'ant jar' and 'ant package' targets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-993-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 25 01:49:56 2009
Return-Path: <common-issues-return-993-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3420 invoked from network); 25 Aug 2009 01:49:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Aug 2009 01:49:56 -0000
Received: (qmail 46524 invoked by uid 500); 25 Aug 2009 01:50:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46473 invoked by uid 500); 25 Aug 2009 01:50:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46457 invoked by uid 99); 25 Aug 2009 01:50:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 01:50:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 01:50:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 57ED3234C04C
	for <common-issues@hadoop.apache.org>; Mon, 24 Aug 2009 18:49:59 -0700 (PDT)
Message-ID: <1928898175.1251164999345.JavaMail.jira@brutus>
Date: Mon, 24 Aug 2009 18:49:59 -0700 (PDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4952) Improved files system interface for
 the application writer.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-4952:
---------------------------------

    Attachment: FilesContext1.patch

The attached FilesContext.java is a class for the proposed new interface for the application writer. (I have use the name FIlesContext instead of the original class name Files since the class/object is really a context in which files
are named and created; further it avoids confusion with the Java class File.
The FileSystem class will remain for the fs implementing FSs (after some cleanup); We will probably rename it
to AbstractFileSystem or VirtualFileSystem (since it is very similar to the Unix VFS).
The FilesContext class is thing layer that mostly calls FileSystem.

There are still a few methods that remain to be implemented (such as copy, move, glob).

Special things to note when reviewing the new interface
* See the examples on how to use FilesContext in the javadoc.
* Have added notion of using ServerSide defaults using SERVER_DEFAULT
   (essentially its value is -1 and is passed across to the NN to use its defaults
   for ReplicationFactor, blockSize etc.
* Unlike FileSystem, FilesContext provides only 2  methods for create:
## FSDataOutputStream create(Path f, FsPermission permission  
                                                   EnumSet<CreateFlag> createFlag)
## FSDataOutputStream create(Path f,
                                    FsPermission permission,
                                    EnumSet<CreateFlag> createFlag,
                                    int bufferSize,
                                    short replication,
                                    long blockSize,
                                    Progressable progress)
** Note that the first one uses the defaults as expected.
In the 2nd one  can still use defaults for individual parameters; e.g.
  create("foo", FSPermissions.default(), EnumSet.of(CREATE), 4096, SERVER_DEFAULT, CONFIG_DEFAULT, null)
In this case the buf size is 4096, the SS default is used for replication factor and the
config default for the blockSize.

* Note the utility functions are in a inner class called Util.
The reason I have not put them in independent class is that one needs the
FilesContext for these function (ie to knopw what / and wd are and also for 
replication factor etc for new files created by the utility methods such as copy().
   

> Improved files system interface for the application writer.
> -----------------------------------------------------------
>
>                 Key: HADOOP-4952
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4952
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.21.0
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: Files.java, Files.java, FilesContext1.patch
>
>
> Currently the FIleSystem interface serves two purposes:
> - an application writer's interface for using the Hadoop file system
> - a file system implementer's interface (e.g. hdfs, local file system, kfs, etc)
> This Jira proposes that we provide a simpler interfaces for the application writer and leave the FilsSystem  interface for the implementer of a filesystem.
> - Filesystem interface  has a  confusing set of methods for the application writer
> - We could make it easier to take advantage of the URI file naming
> ** Current approach is to get FileSystem instance by supplying the URI and then access that name space. It is consistent for the FileSystem instance to not accept URIs for other schemes, but we can do better.
> ** The special copyFromLocalFIle can be generalized as a  copyFile where the src or target can be generalized to any URI, including the local one.
> ** The proposed scheme (below) simplifies this.
> -	The client side config can be simplified. 
> ** New config() by default uses the default config. Since this is the common usage pattern, one should not need to always pass the config as a parameter when accessing the file system.  
> -	
> ** It does not handle multiple file systems too well. Today a site.xml is derived from a single Hadoop cluster. This does not make sense for multiple Hadoop clusters which may have different defaults.
> ** Further one should need very little to configure the client side:
> *** Default files system.
> *** Block size 
> *** Replication factor
> *** Scheme to class mapping
> ** It should be possible to take Blocksize and replication factors defaults from the target file system, rather then the client size config.  I am not suggesting we don't allow setting client side defaults, but most clients do not care and would find it simpler to take the defaults for their systems  from the target file system. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-994-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 25 09:20:58 2009
Return-Path: <common-issues-return-994-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 625 invoked from network); 25 Aug 2009 09:20:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Aug 2009 09:20:58 -0000
Received: (qmail 79301 invoked by uid 500); 25 Aug 2009 09:21:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79257 invoked by uid 500); 25 Aug 2009 09:21:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79246 invoked by uid 99); 25 Aug 2009 09:21:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 09:21:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 09:21:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 5FA38234C1E7
	for <common-issues@hadoop.apache.org>; Tue, 25 Aug 2009 02:20:59 -0700 (PDT)
Message-ID: <1887060823.1251192059390.JavaMail.jira@brutus>
Date: Tue, 25 Aug 2009 02:20:59 -0700 (PDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6184) Provide a configuration dump in
 json format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747321#action_12747321 ] 

Hemanth Yamijala commented on HADOOP-6184:
------------------------------------------

I committed the common jar and test jar to the MAPREDUCE sub-project after verifying tests pass with the new jar.

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>            Assignee: V.V.Chaitanya Krishna
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184-2.patch, HADOOP-6184-3.patch, HADOOP-6184-4.patch, HADOOP-6184-5.patch, HADOOP-6184-6.patch, HADOOP-6184-7.patch, HADOOP-6184-8.patch, HADOOP-6184-9.patch, HADOOP-6184-ydist-1.patch, HADOOP-6184-ydist.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-995-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 25 15:29:58 2009
Return-Path: <common-issues-return-995-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24317 invoked from network); 25 Aug 2009 15:29:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Aug 2009 15:29:58 -0000
Received: (qmail 94666 invoked by uid 500); 25 Aug 2009 15:30:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94607 invoked by uid 500); 25 Aug 2009 15:30:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94596 invoked by uid 99); 25 Aug 2009 15:30:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 15:30:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 15:30:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4D4B7234C04C
	for <common-issues@hadoop.apache.org>; Tue, 25 Aug 2009 08:29:59 -0700 (PDT)
Message-ID: <47259695.1251214199302.JavaMail.jira@brutus>
Date: Tue, 25 Aug 2009 08:29:59 -0700 (PDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6200) glue together the different
 projects to make local builds easier
In-Reply-To: <68853708.1250609369522.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747461#action_12747461 ] 

Steve Loughran commented on HADOOP-6200:
----------------------------------------

Actually, it turns out to really be a cycle. the hadoop-hdfs.init target requires hadoop-mapred to exist, because it unzips the webapps. 
    <unzip src="${lib.dir}/hadoop-mapred-${hadoop-mr.version}.jar"
        dest="${build.dir}">
      <patternset>
        <include name="webapps/**"/>
      </patternset>
    </unzip>

_you cannot build hdfs without mapred, you cannot build mapred without hdfs_
 Therefore, the graph of build dependencies is not technically acyclic.


> glue together the different projects to make local builds easier
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6200
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6200
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Priority: Minor
>
> It's currently fairly tricky to get everything building/up to date in a single machine. We can and should improve this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-996-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 25 16:41:58 2009
Return-Path: <common-issues-return-996-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89820 invoked from network); 25 Aug 2009 16:41:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Aug 2009 16:41:58 -0000
Received: (qmail 13835 invoked by uid 500); 25 Aug 2009 16:42:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13811 invoked by uid 500); 25 Aug 2009 16:42:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13801 invoked by uid 99); 25 Aug 2009 16:42:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 16:42:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 16:42:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 50BAF234C04C
	for <common-issues@hadoop.apache.org>; Tue, 25 Aug 2009 09:41:59 -0700 (PDT)
Message-ID: <1845055420.1251218519316.JavaMail.jira@brutus>
Date: Tue, 25 Aug 2009 09:41:59 -0700 (PDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4952) Improved files system interface for
 the application writer.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-4952:
---------------------------------

    Attachment: FilesContext2.patch

> Improved files system interface for the application writer.
> -----------------------------------------------------------
>
>                 Key: HADOOP-4952
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4952
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.21.0
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: Files.java, Files.java, FilesContext1.patch, FilesContext2.patch
>
>
> Currently the FIleSystem interface serves two purposes:
> - an application writer's interface for using the Hadoop file system
> - a file system implementer's interface (e.g. hdfs, local file system, kfs, etc)
> This Jira proposes that we provide a simpler interfaces for the application writer and leave the FilsSystem  interface for the implementer of a filesystem.
> - Filesystem interface  has a  confusing set of methods for the application writer
> - We could make it easier to take advantage of the URI file naming
> ** Current approach is to get FileSystem instance by supplying the URI and then access that name space. It is consistent for the FileSystem instance to not accept URIs for other schemes, but we can do better.
> ** The special copyFromLocalFIle can be generalized as a  copyFile where the src or target can be generalized to any URI, including the local one.
> ** The proposed scheme (below) simplifies this.
> -	The client side config can be simplified. 
> ** New config() by default uses the default config. Since this is the common usage pattern, one should not need to always pass the config as a parameter when accessing the file system.  
> -	
> ** It does not handle multiple file systems too well. Today a site.xml is derived from a single Hadoop cluster. This does not make sense for multiple Hadoop clusters which may have different defaults.
> ** Further one should need very little to configure the client side:
> *** Default files system.
> *** Block size 
> *** Replication factor
> *** Scheme to class mapping
> ** It should be possible to take Blocksize and replication factors defaults from the target file system, rather then the client size config.  I am not suggesting we don't allow setting client side defaults, but most clients do not care and would find it simpler to take the defaults for their systems  from the target file system. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-997-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 25 17:20:59 2009
Return-Path: <common-issues-return-997-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5748 invoked from network); 25 Aug 2009 17:20:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Aug 2009 17:20:59 -0000
Received: (qmail 75016 invoked by uid 500); 25 Aug 2009 17:21:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74909 invoked by uid 500); 25 Aug 2009 17:21:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74898 invoked by uid 99); 25 Aug 2009 17:21:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 17:21:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 17:21:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 5EB13234C48C
	for <common-issues@hadoop.apache.org>; Tue, 25 Aug 2009 10:20:59 -0700 (PDT)
Message-ID: <1937599251.1251220859387.JavaMail.jira@brutus>
Date: Tue, 25 Aug 2009 10:20:59 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4952) Improved files system interface for
 the application writer.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747493#action_12747493 ] 

Doug Cutting commented on HADOOP-4952:
--------------------------------------

Some naming nits:
 - I'd prefer to name this just Files or perhaps FileContext rather than FilesContext.
 - FSConfigUtil could simply be called FSConfig.  Moreover, its methods could be FileContext methods, no?  Why do we need this as a separate class?
 - Why do we need the inner class FSConfig?  Its fields seem like they should be FileContext fields directly, no?


> Improved files system interface for the application writer.
> -----------------------------------------------------------
>
>                 Key: HADOOP-4952
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4952
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.21.0
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: Files.java, Files.java, FilesContext1.patch, FilesContext2.patch
>
>
> Currently the FIleSystem interface serves two purposes:
> - an application writer's interface for using the Hadoop file system
> - a file system implementer's interface (e.g. hdfs, local file system, kfs, etc)
> This Jira proposes that we provide a simpler interfaces for the application writer and leave the FilsSystem  interface for the implementer of a filesystem.
> - Filesystem interface  has a  confusing set of methods for the application writer
> - We could make it easier to take advantage of the URI file naming
> ** Current approach is to get FileSystem instance by supplying the URI and then access that name space. It is consistent for the FileSystem instance to not accept URIs for other schemes, but we can do better.
> ** The special copyFromLocalFIle can be generalized as a  copyFile where the src or target can be generalized to any URI, including the local one.
> ** The proposed scheme (below) simplifies this.
> -	The client side config can be simplified. 
> ** New config() by default uses the default config. Since this is the common usage pattern, one should not need to always pass the config as a parameter when accessing the file system.  
> -	
> ** It does not handle multiple file systems too well. Today a site.xml is derived from a single Hadoop cluster. This does not make sense for multiple Hadoop clusters which may have different defaults.
> ** Further one should need very little to configure the client side:
> *** Default files system.
> *** Block size 
> *** Replication factor
> *** Scheme to class mapping
> ** It should be possible to take Blocksize and replication factors defaults from the target file system, rather then the client size config.  I am not suggesting we don't allow setting client side defaults, but most clients do not care and would find it simpler to take the defaults for their systems  from the target file system. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-998-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 25 17:22:56 2009
Return-Path: <common-issues-return-998-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7050 invoked from network); 25 Aug 2009 17:22:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Aug 2009 17:22:56 -0000
Received: (qmail 78676 invoked by uid 500); 25 Aug 2009 17:23:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78616 invoked by uid 500); 25 Aug 2009 17:23:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78595 invoked by uid 99); 25 Aug 2009 17:23:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 17:23:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 17:23:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 62F5A234C1EE
	for <common-issues@hadoop.apache.org>; Tue, 25 Aug 2009 10:22:59 -0700 (PDT)
Message-ID: <2060288550.1251220979404.JavaMail.jira@brutus>
Date: Tue, 25 Aug 2009 10:22:59 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6200) glue together the different
 projects to make local builds easier
In-Reply-To: <68853708.1250609369522.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747494#action_12747494 ] 

Doug Cutting commented on HADOOP-6200:
--------------------------------------

> the hadoop-hdfs.init target requires hadoop-mapred to exist, because it unzips the webapps

That seems like a bug we should fix, no?

> glue together the different projects to make local builds easier
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6200
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6200
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Priority: Minor
>
> It's currently fairly tricky to get everything building/up to date in a single machine. We can and should improve this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-999-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 25 17:22:58 2009
Return-Path: <common-issues-return-999-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7231 invoked from network); 25 Aug 2009 17:22:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Aug 2009 17:22:58 -0000
Received: (qmail 79636 invoked by uid 500); 25 Aug 2009 17:23:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79587 invoked by uid 500); 25 Aug 2009 17:23:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79577 invoked by uid 99); 25 Aug 2009 17:23:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 17:23:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 17:23:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 5FB76234C1E7
	for <common-issues@hadoop.apache.org>; Tue, 25 Aug 2009 10:22:59 -0700 (PDT)
Message-ID: <548899358.1251220979391.JavaMail.jira@brutus>
Date: Tue, 25 Aug 2009 10:22:59 -0700 (PDT)
From: "Bradley Buda (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6208) Block loss in S3FS due to S3
 inconsistency on file rename
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Block loss in S3FS due to S3 inconsistency on file rename
---------------------------------------------------------

                 Key: HADOOP-6208
                 URL: https://issues.apache.org/jira/browse/HADOOP-6208
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs/s3
    Affects Versions: 0.20.0
         Environment: Ubuntu Linux 8.04 on EC2, Mac OS X 10.5, likely to affect any Hadoop environment
            Reporter: Bradley Buda


Under certain S3 consistency scenarios, Hadoop's S3FileSystem can 'truncate' files, especially when writing reduce outputs.  We've noticed this at tracksimple where we use the S3FS as the direct input and output of our MapReduce jobs.  The symptom of this problem is a file in the filesystem that is an exact multiple of the FS block size - exactly 32MB, 64MB, 96MB, etc. in length.

The issue appears to be caused by renaming a file that has recently been written, and getting a stale INode read from S3.  When a reducer is writing job output to the S3FS, the normal series of S3 key writes for a 3-block file looks something like this:

Task Output:
1) Write the first block (block_99)
2) Write an INode (/myjob/_temporary/_attempt_200907142159_0306_r_000133_0/part-00133.gz) containing [block_99]
3) Write the second block (block_81)
4) Rewrite the INode with new contents [block_99, block_81]
5) Write the last block (block_-101)
6) Rewrite the INode with the final contents [block_99, block_81, block_-101]

Copy Output to Final Location (ReduceTask#copyOutput):
1) Read the INode contents from /myjob/_temporary/_attempt_200907142159_0306_r_000133_0/part-00133.gz, which gives [block_99, block_81, block_-101]
2) Write the data from #1 to the final location, /myjob/part-00133.gz
3) Delete the old INode 

The output file is truncated if S3 serves a stale copy of the temporary INode.  In copyOutput, step 1 above, it is possible for S3 to return a version of the temporary INode that contains just [block_99, block_81].  In this case, we write this new data to the final output location, and 'lose' block_-101 in the process.  Since we then delete the temporary INode, we've lost all references to the final block of this file and it's orphaned in the S3 bucket.

This type of consistency error is infrequent but not impossible. We've observed these failures about once a week for one of our large jobs which runs daily and has 200 reduce outputs; so we're seeing an error rate of something like 0.07% per reduce.

These kind of errors are generally difficult to handle in a system like S3.  We have a few ideas about how to fix this:
1) HACK! Sleep during S3OutputStream#close or #flush to wait for S3 to catch up and make these less likely.
2) Poll for updated MD5 or INode data in Jets3tFileSystemStore#storeINode until S3 says the INode contents are the same as our local copy.  This could be a config option - "fs.s3.verifyInodeWrites" or something like that.
3) Cache INode contents in-process, so we don't have to go back to S3 to ask for the current version of an INode.
4) Only write INodes once, when the output stream is closed.  This would basically make S3OutputStream#flush() a no-op.
5) Modify the S3FS to somehow version INodes (unclear how we would do this, need some design work).
6) Avoid using the S3FS for temporary task attempt files.
7) Avoid using the S3FS completely.

We wanted to get some guidance from the community before we went down any of these paths.  Has anyone seen this issue?  Any other suggested workarounds?  We at tracksimple are willing to invest some time in fixing this and (of course) contributing our fix back, but we wanted to get an 'ack' from others before we try anything crazy :-).

I've attached a test app if anyone wants to try and reproduce this themselves.  It takes a while to run (depending on the 'weather' in S3 right now), but should eventually detect a consistency 'error' that manifests itself as a truncated file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1000-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 25 17:41:57 2009
Return-Path: <common-issues-return-1000-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18362 invoked from network); 25 Aug 2009 17:41:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Aug 2009 17:41:57 -0000
Received: (qmail 1559 invoked by uid 500); 25 Aug 2009 17:42:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1499 invoked by uid 500); 25 Aug 2009 17:42:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1489 invoked by uid 99); 25 Aug 2009 17:42:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 17:42:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 17:42:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 8EC02234C1EB
	for <common-issues@hadoop.apache.org>; Tue, 25 Aug 2009 10:42:00 -0700 (PDT)
Message-ID: <1940888334.1251222120583.JavaMail.jira@brutus>
Date: Tue, 25 Aug 2009 10:42:00 -0700 (PDT)
From: "Bradley Buda (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6208) Block loss in S3FS due to S3
 inconsistency on file rename
In-Reply-To: <548899358.1251220979391.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bradley Buda updated HADOOP-6208:
---------------------------------

    Attachment: S3FSConsistencyTest.java

Test app that can reproduce truncated files in S3FS

> Block loss in S3FS due to S3 inconsistency on file rename
> ---------------------------------------------------------
>
>                 Key: HADOOP-6208
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6208
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 0.20.0
>         Environment: Ubuntu Linux 8.04 on EC2, Mac OS X 10.5, likely to affect any Hadoop environment
>            Reporter: Bradley Buda
>         Attachments: S3FSConsistencyTest.java
>
>
> Under certain S3 consistency scenarios, Hadoop's S3FileSystem can 'truncate' files, especially when writing reduce outputs.  We've noticed this at tracksimple where we use the S3FS as the direct input and output of our MapReduce jobs.  The symptom of this problem is a file in the filesystem that is an exact multiple of the FS block size - exactly 32MB, 64MB, 96MB, etc. in length.
> The issue appears to be caused by renaming a file that has recently been written, and getting a stale INode read from S3.  When a reducer is writing job output to the S3FS, the normal series of S3 key writes for a 3-block file looks something like this:
> Task Output:
> 1) Write the first block (block_99)
> 2) Write an INode (/myjob/_temporary/_attempt_200907142159_0306_r_000133_0/part-00133.gz) containing [block_99]
> 3) Write the second block (block_81)
> 4) Rewrite the INode with new contents [block_99, block_81]
> 5) Write the last block (block_-101)
> 6) Rewrite the INode with the final contents [block_99, block_81, block_-101]
> Copy Output to Final Location (ReduceTask#copyOutput):
> 1) Read the INode contents from /myjob/_temporary/_attempt_200907142159_0306_r_000133_0/part-00133.gz, which gives [block_99, block_81, block_-101]
> 2) Write the data from #1 to the final location, /myjob/part-00133.gz
> 3) Delete the old INode 
> The output file is truncated if S3 serves a stale copy of the temporary INode.  In copyOutput, step 1 above, it is possible for S3 to return a version of the temporary INode that contains just [block_99, block_81].  In this case, we write this new data to the final output location, and 'lose' block_-101 in the process.  Since we then delete the temporary INode, we've lost all references to the final block of this file and it's orphaned in the S3 bucket.
> This type of consistency error is infrequent but not impossible. We've observed these failures about once a week for one of our large jobs which runs daily and has 200 reduce outputs; so we're seeing an error rate of something like 0.07% per reduce.
> These kind of errors are generally difficult to handle in a system like S3.  We have a few ideas about how to fix this:
> 1) HACK! Sleep during S3OutputStream#close or #flush to wait for S3 to catch up and make these less likely.
> 2) Poll for updated MD5 or INode data in Jets3tFileSystemStore#storeINode until S3 says the INode contents are the same as our local copy.  This could be a config option - "fs.s3.verifyInodeWrites" or something like that.
> 3) Cache INode contents in-process, so we don't have to go back to S3 to ask for the current version of an INode.
> 4) Only write INodes once, when the output stream is closed.  This would basically make S3OutputStream#flush() a no-op.
> 5) Modify the S3FS to somehow version INodes (unclear how we would do this, need some design work).
> 6) Avoid using the S3FS for temporary task attempt files.
> 7) Avoid using the S3FS completely.
> We wanted to get some guidance from the community before we went down any of these paths.  Has anyone seen this issue?  Any other suggested workarounds?  We at tracksimple are willing to invest some time in fixing this and (of course) contributing our fix back, but we wanted to get an 'ack' from others before we try anything crazy :-).
> I've attached a test app if anyone wants to try and reproduce this themselves.  It takes a while to run (depending on the 'weather' in S3 right now), but should eventually detect a consistency 'error' that manifests itself as a truncated file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1001-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 25 18:38:02 2009
Return-Path: <common-issues-return-1001-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57794 invoked from network); 25 Aug 2009 18:38:01 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Aug 2009 18:38:01 -0000
Received: (qmail 96800 invoked by uid 500); 25 Aug 2009 18:38:26 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96714 invoked by uid 500); 25 Aug 2009 18:38:26 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96532 invoked by uid 99); 25 Aug 2009 18:38:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 18:38:26 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 18:38:24 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 8ABE3234C04C
	for <common-issues@hadoop.apache.org>; Tue, 25 Aug 2009 11:38:04 -0700 (PDT)
Message-ID: <547946396.1251225484553.JavaMail.jira@brutus>
Date: Tue, 25 Aug 2009 11:38:04 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-6166:
----------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

+1

Very impressive work. Thanks Nicholas, Todd, and Scott!

> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.21.0
>
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt, c6166_20090727.patch, c6166_20090728.patch, c6166_20090810.patch, c6166_20090811.patch, c6166_20090819.patch, c6166_20090819review.patch, graph.r, graph.r, Rplots-laptop.pdf, Rplots-nehalem32.pdf, Rplots-nehalem64.pdf, Rplots.pdf, Rplots.pdf, Rplots.pdf
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1002-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 25 19:29:57 2009
Return-Path: <common-issues-return-1002-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75809 invoked from network); 25 Aug 2009 19:29:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Aug 2009 19:29:56 -0000
Received: (qmail 87073 invoked by uid 500); 25 Aug 2009 19:30:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86983 invoked by uid 500); 25 Aug 2009 19:30:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86777 invoked by uid 99); 25 Aug 2009 19:30:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 19:30:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 19:30:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 5C64A234C04C
	for <common-issues@hadoop.apache.org>; Tue, 25 Aug 2009 12:29:59 -0700 (PDT)
Message-ID: <109566683.1251228599296.JavaMail.jira@brutus>
Date: Tue, 25 Aug 2009 12:29:59 -0700 (PDT)
From: "Raghu Angadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6208) Block loss in S3FS due to S3
 inconsistency on file rename
In-Reply-To: <548899358.1251220979391.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747591#action_12747591 ] 

Raghu Angadi commented on HADOOP-6208:
--------------------------------------

This is the first time I am looking at S3FS. It is essentially a thin client managed FS layer over S3.. it would pretty hard to avoid such inconsistencies especially S3 itself can return stale info. isn't there an option for S3 client to get the latest copy (thinking Dynamo-like key-value reads)?

Between options (1) to (5) : (2) is probably the best. It is a bit analogous to what HDFS does : when close() completes successfully, it implies all the blocks have at least one replica reported.

Regd (6)-(7) :  a loosely managed FS will always such issues. I am not sure how much is S3FS is used in production, I would be curious to know.

For those familiar with HDFS: S3FS is a broadly similar where all the metadata is manipulated by S3FS client rather than a central server (e.g. rename moves metadata for each file under the tree to a new location). While writing, each block is written locally and stored as an S3Object when full.

> Block loss in S3FS due to S3 inconsistency on file rename
> ---------------------------------------------------------
>
>                 Key: HADOOP-6208
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6208
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 0.20.0
>         Environment: Ubuntu Linux 8.04 on EC2, Mac OS X 10.5, likely to affect any Hadoop environment
>            Reporter: Bradley Buda
>         Attachments: S3FSConsistencyTest.java
>
>
> Under certain S3 consistency scenarios, Hadoop's S3FileSystem can 'truncate' files, especially when writing reduce outputs.  We've noticed this at tracksimple where we use the S3FS as the direct input and output of our MapReduce jobs.  The symptom of this problem is a file in the filesystem that is an exact multiple of the FS block size - exactly 32MB, 64MB, 96MB, etc. in length.
> The issue appears to be caused by renaming a file that has recently been written, and getting a stale INode read from S3.  When a reducer is writing job output to the S3FS, the normal series of S3 key writes for a 3-block file looks something like this:
> Task Output:
> 1) Write the first block (block_99)
> 2) Write an INode (/myjob/_temporary/_attempt_200907142159_0306_r_000133_0/part-00133.gz) containing [block_99]
> 3) Write the second block (block_81)
> 4) Rewrite the INode with new contents [block_99, block_81]
> 5) Write the last block (block_-101)
> 6) Rewrite the INode with the final contents [block_99, block_81, block_-101]
> Copy Output to Final Location (ReduceTask#copyOutput):
> 1) Read the INode contents from /myjob/_temporary/_attempt_200907142159_0306_r_000133_0/part-00133.gz, which gives [block_99, block_81, block_-101]
> 2) Write the data from #1 to the final location, /myjob/part-00133.gz
> 3) Delete the old INode 
> The output file is truncated if S3 serves a stale copy of the temporary INode.  In copyOutput, step 1 above, it is possible for S3 to return a version of the temporary INode that contains just [block_99, block_81].  In this case, we write this new data to the final output location, and 'lose' block_-101 in the process.  Since we then delete the temporary INode, we've lost all references to the final block of this file and it's orphaned in the S3 bucket.
> This type of consistency error is infrequent but not impossible. We've observed these failures about once a week for one of our large jobs which runs daily and has 200 reduce outputs; so we're seeing an error rate of something like 0.07% per reduce.
> These kind of errors are generally difficult to handle in a system like S3.  We have a few ideas about how to fix this:
> 1) HACK! Sleep during S3OutputStream#close or #flush to wait for S3 to catch up and make these less likely.
> 2) Poll for updated MD5 or INode data in Jets3tFileSystemStore#storeINode until S3 says the INode contents are the same as our local copy.  This could be a config option - "fs.s3.verifyInodeWrites" or something like that.
> 3) Cache INode contents in-process, so we don't have to go back to S3 to ask for the current version of an INode.
> 4) Only write INodes once, when the output stream is closed.  This would basically make S3OutputStream#flush() a no-op.
> 5) Modify the S3FS to somehow version INodes (unclear how we would do this, need some design work).
> 6) Avoid using the S3FS for temporary task attempt files.
> 7) Avoid using the S3FS completely.
> We wanted to get some guidance from the community before we went down any of these paths.  Has anyone seen this issue?  Any other suggested workarounds?  We at tracksimple are willing to invest some time in fixing this and (of course) contributing our fix back, but we wanted to get an 'ack' from others before we try anything crazy :-).
> I've attached a test app if anyone wants to try and reproduce this themselves.  It takes a while to run (depending on the 'weather' in S3 right now), but should eventually detect a consistency 'error' that manifests itself as a truncated file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1003-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 25 20:11:56 2009
Return-Path: <common-issues-return-1003-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87188 invoked from network); 25 Aug 2009 20:11:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Aug 2009 20:11:56 -0000
Received: (qmail 37011 invoked by uid 500); 25 Aug 2009 20:12:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36959 invoked by uid 500); 25 Aug 2009 20:12:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36949 invoked by uid 99); 25 Aug 2009 20:12:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 20:12:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 20:12:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id B0337234C1E7
	for <common-issues@hadoop.apache.org>; Tue, 25 Aug 2009 13:11:59 -0700 (PDT)
Message-ID: <1585712928.1251231119720.JavaMail.jira@brutus>
Date: Tue, 25 Aug 2009 13:11:59 -0700 (PDT)
From: "Bradley Buda (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6208) Block loss in S3FS due to S3
 inconsistency on file rename
In-Reply-To: <548899358.1251220979391.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747616#action_12747616 ] 

Bradley Buda commented on HADOOP-6208:
--------------------------------------

Thanks for the quick reply.  You're right; any use of S3FS is going to run into inconsistencies, no matter how clever the client is.  What I'm looking for is some way to make any failures due to inconsistency more visible, and less likely to lose data.  Of the ideas I threw out, I think I also like #2 the best - that way users like us who do a ton of S3 I/O can pay the extra performance hit to verify writes without impacting everyone.  I'll bounce this idea around with teammates and see if I can put together a patch.

> Block loss in S3FS due to S3 inconsistency on file rename
> ---------------------------------------------------------
>
>                 Key: HADOOP-6208
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6208
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 0.20.0
>         Environment: Ubuntu Linux 8.04 on EC2, Mac OS X 10.5, likely to affect any Hadoop environment
>            Reporter: Bradley Buda
>         Attachments: S3FSConsistencyTest.java
>
>
> Under certain S3 consistency scenarios, Hadoop's S3FileSystem can 'truncate' files, especially when writing reduce outputs.  We've noticed this at tracksimple where we use the S3FS as the direct input and output of our MapReduce jobs.  The symptom of this problem is a file in the filesystem that is an exact multiple of the FS block size - exactly 32MB, 64MB, 96MB, etc. in length.
> The issue appears to be caused by renaming a file that has recently been written, and getting a stale INode read from S3.  When a reducer is writing job output to the S3FS, the normal series of S3 key writes for a 3-block file looks something like this:
> Task Output:
> 1) Write the first block (block_99)
> 2) Write an INode (/myjob/_temporary/_attempt_200907142159_0306_r_000133_0/part-00133.gz) containing [block_99]
> 3) Write the second block (block_81)
> 4) Rewrite the INode with new contents [block_99, block_81]
> 5) Write the last block (block_-101)
> 6) Rewrite the INode with the final contents [block_99, block_81, block_-101]
> Copy Output to Final Location (ReduceTask#copyOutput):
> 1) Read the INode contents from /myjob/_temporary/_attempt_200907142159_0306_r_000133_0/part-00133.gz, which gives [block_99, block_81, block_-101]
> 2) Write the data from #1 to the final location, /myjob/part-00133.gz
> 3) Delete the old INode 
> The output file is truncated if S3 serves a stale copy of the temporary INode.  In copyOutput, step 1 above, it is possible for S3 to return a version of the temporary INode that contains just [block_99, block_81].  In this case, we write this new data to the final output location, and 'lose' block_-101 in the process.  Since we then delete the temporary INode, we've lost all references to the final block of this file and it's orphaned in the S3 bucket.
> This type of consistency error is infrequent but not impossible. We've observed these failures about once a week for one of our large jobs which runs daily and has 200 reduce outputs; so we're seeing an error rate of something like 0.07% per reduce.
> These kind of errors are generally difficult to handle in a system like S3.  We have a few ideas about how to fix this:
> 1) HACK! Sleep during S3OutputStream#close or #flush to wait for S3 to catch up and make these less likely.
> 2) Poll for updated MD5 or INode data in Jets3tFileSystemStore#storeINode until S3 says the INode contents are the same as our local copy.  This could be a config option - "fs.s3.verifyInodeWrites" or something like that.
> 3) Cache INode contents in-process, so we don't have to go back to S3 to ask for the current version of an INode.
> 4) Only write INodes once, when the output stream is closed.  This would basically make S3OutputStream#flush() a no-op.
> 5) Modify the S3FS to somehow version INodes (unclear how we would do this, need some design work).
> 6) Avoid using the S3FS for temporary task attempt files.
> 7) Avoid using the S3FS completely.
> We wanted to get some guidance from the community before we went down any of these paths.  Has anyone seen this issue?  Any other suggested workarounds?  We at tracksimple are willing to invest some time in fixing this and (of course) contributing our fix back, but we wanted to get an 'ack' from others before we try anything crazy :-).
> I've attached a test app if anyone wants to try and reproduce this themselves.  It takes a while to run (depending on the 'weather' in S3 right now), but should eventually detect a consistency 'error' that manifests itself as a truncated file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1004-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 25 21:11:57 2009
Return-Path: <common-issues-return-1004-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8497 invoked from network); 25 Aug 2009 21:11:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Aug 2009 21:11:56 -0000
Received: (qmail 6734 invoked by uid 500); 25 Aug 2009 21:12:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6674 invoked by uid 500); 25 Aug 2009 21:12:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6664 invoked by uid 99); 25 Aug 2009 21:12:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 21:12:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 21:12:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 54BE6234C1EE
	for <common-issues@hadoop.apache.org>; Tue, 25 Aug 2009 14:11:59 -0700 (PDT)
Message-ID: <597495258.1251234719346.JavaMail.jira@brutus>
Date: Tue, 25 Aug 2009 14:11:59 -0700 (PDT)
From: "Raghu Angadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6208) Block loss in S3FS due to S3
 inconsistency on file rename
In-Reply-To: <548899358.1251220979391.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747646#action_12747646 ] 

Raghu Angadi commented on HADOOP-6208:
--------------------------------------

Regd option #2 : If one client gets the latest value, does it imply all subsequent reads (from any client) will? Otherwise one read might provide latest value, but very next read might return older value since it might read from a different replica (I don't know much about S3 internals). The client needs to somehow make sure all the replicas got the update.

> Block loss in S3FS due to S3 inconsistency on file rename
> ---------------------------------------------------------
>
>                 Key: HADOOP-6208
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6208
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 0.20.0
>         Environment: Ubuntu Linux 8.04 on EC2, Mac OS X 10.5, likely to affect any Hadoop environment
>            Reporter: Bradley Buda
>         Attachments: S3FSConsistencyTest.java
>
>
> Under certain S3 consistency scenarios, Hadoop's S3FileSystem can 'truncate' files, especially when writing reduce outputs.  We've noticed this at tracksimple where we use the S3FS as the direct input and output of our MapReduce jobs.  The symptom of this problem is a file in the filesystem that is an exact multiple of the FS block size - exactly 32MB, 64MB, 96MB, etc. in length.
> The issue appears to be caused by renaming a file that has recently been written, and getting a stale INode read from S3.  When a reducer is writing job output to the S3FS, the normal series of S3 key writes for a 3-block file looks something like this:
> Task Output:
> 1) Write the first block (block_99)
> 2) Write an INode (/myjob/_temporary/_attempt_200907142159_0306_r_000133_0/part-00133.gz) containing [block_99]
> 3) Write the second block (block_81)
> 4) Rewrite the INode with new contents [block_99, block_81]
> 5) Write the last block (block_-101)
> 6) Rewrite the INode with the final contents [block_99, block_81, block_-101]
> Copy Output to Final Location (ReduceTask#copyOutput):
> 1) Read the INode contents from /myjob/_temporary/_attempt_200907142159_0306_r_000133_0/part-00133.gz, which gives [block_99, block_81, block_-101]
> 2) Write the data from #1 to the final location, /myjob/part-00133.gz
> 3) Delete the old INode 
> The output file is truncated if S3 serves a stale copy of the temporary INode.  In copyOutput, step 1 above, it is possible for S3 to return a version of the temporary INode that contains just [block_99, block_81].  In this case, we write this new data to the final output location, and 'lose' block_-101 in the process.  Since we then delete the temporary INode, we've lost all references to the final block of this file and it's orphaned in the S3 bucket.
> This type of consistency error is infrequent but not impossible. We've observed these failures about once a week for one of our large jobs which runs daily and has 200 reduce outputs; so we're seeing an error rate of something like 0.07% per reduce.
> These kind of errors are generally difficult to handle in a system like S3.  We have a few ideas about how to fix this:
> 1) HACK! Sleep during S3OutputStream#close or #flush to wait for S3 to catch up and make these less likely.
> 2) Poll for updated MD5 or INode data in Jets3tFileSystemStore#storeINode until S3 says the INode contents are the same as our local copy.  This could be a config option - "fs.s3.verifyInodeWrites" or something like that.
> 3) Cache INode contents in-process, so we don't have to go back to S3 to ask for the current version of an INode.
> 4) Only write INodes once, when the output stream is closed.  This would basically make S3OutputStream#flush() a no-op.
> 5) Modify the S3FS to somehow version INodes (unclear how we would do this, need some design work).
> 6) Avoid using the S3FS for temporary task attempt files.
> 7) Avoid using the S3FS completely.
> We wanted to get some guidance from the community before we went down any of these paths.  Has anyone seen this issue?  Any other suggested workarounds?  We at tracksimple are willing to invest some time in fixing this and (of course) contributing our fix back, but we wanted to get an 'ack' from others before we try anything crazy :-).
> I've attached a test app if anyone wants to try and reproduce this themselves.  It takes a while to run (depending on the 'weather' in S3 right now), but should eventually detect a consistency 'error' that manifests itself as a truncated file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1005-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 25 21:53:57 2009
Return-Path: <common-issues-return-1005-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57555 invoked from network); 25 Aug 2009 21:53:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Aug 2009 21:53:57 -0000
Received: (qmail 77877 invoked by uid 500); 25 Aug 2009 21:54:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77820 invoked by uid 500); 25 Aug 2009 21:54:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77810 invoked by uid 99); 25 Aug 2009 21:54:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 21:54:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 21:54:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 605D2234C1F0
	for <common-issues@hadoop.apache.org>; Tue, 25 Aug 2009 14:53:59 -0700 (PDT)
Message-ID: <1890725100.1251237239393.JavaMail.jira@brutus>
Date: Tue, 25 Aug 2009 14:53:59 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6205) Project java code has a significant
 number of declaration javadoc warnings and errors
In-Reply-To: <2108675829.1250797574846.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik reassigned HADOOP-6205:
------------------------------------------

    Assignee: Konstantin Boudnik

> Project java code has a significant number of declaration javadoc warnings and errors
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6205
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6205
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>
> The issue here is that code inspections (say by IntelliJ IDEA) show that a number of classes have javadoc problems in declarations. Some of those are minor, e.g. missing @return, @param, @throws tags.
> Some of them are more severe, e.g. using non-existing parameter names in a method javadoc (see HDFS' BlockManager.removeFromInvalidates(..) for an example)
> I'd like to add more sophisticated code analysis to be able to fix at least javadoc declaration errors in order to produce better code quality.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1006-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 25 21:54:04 2009
Return-Path: <common-issues-return-1006-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57670 invoked from network); 25 Aug 2009 21:54:04 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Aug 2009 21:54:04 -0000
Received: (qmail 78033 invoked by uid 500); 25 Aug 2009 21:54:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77976 invoked by uid 500); 25 Aug 2009 21:54:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77966 invoked by uid 99); 25 Aug 2009 21:54:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 21:54:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 21:54:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4C8B5234C1E7
	for <common-issues@hadoop.apache.org>; Tue, 25 Aug 2009 14:53:59 -0700 (PDT)
Message-ID: <1902704708.1251237239298.JavaMail.jira@brutus>
Date: Tue, 25 Aug 2009 14:53:59 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6197) Use aspects to introduce new methods
 required for testing
In-Reply-To: <1324647998.1250542274830.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik reassigned HADOOP-6197:
------------------------------------------

    Assignee: Konstantin Boudnik

> Use aspects to introduce new methods required for testing
> ---------------------------------------------------------
>
>                 Key: HADOOP-6197
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6197
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: security, test
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-6197.patch, HADOOP-6197.sh
>
>
> HADOOP-6176 introduces two new methods which are essentially required for testing purposes. However, this bring a permanent modifications to the production code.
> The same result can be reached by using aspect development. Thus the modifications will be brought in by byte code instrumentation and will exist only during the testing cycles.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1007-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 25 22:46:58 2009
Return-Path: <common-issues-return-1007-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83459 invoked from network); 25 Aug 2009 22:46:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Aug 2009 22:46:58 -0000
Received: (qmail 45780 invoked by uid 500); 25 Aug 2009 22:47:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45720 invoked by uid 500); 25 Aug 2009 22:47:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45685 invoked by uid 99); 25 Aug 2009 22:47:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 22:47:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 22:47:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 5D8EE234C1EF
	for <common-issues@hadoop.apache.org>; Tue, 25 Aug 2009 15:46:59 -0700 (PDT)
Message-ID: <1264434986.1251240419382.JavaMail.jira@brutus>
Date: Tue, 25 Aug 2009 15:46:59 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Moved: (HADOOP-6209) chmod fails on Windows+cygwin
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE moved HDFS-569 to HADOOP-6209:
-----------------------------------------------------

    Component/s:     (was: hdfs client)
                 fs
            Key: HADOOP-6209  (was: HDFS-569)
        Project: Hadoop Common  (was: Hadoop HDFS)

> chmod fails on Windows+cygwin
> -----------------------------
>
>                 Key: HADOOP-6209
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6209
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Kohsuke Kawaguchi
>            Assignee: Kohsuke Kawaguchi
>         Attachments: HDFS-569.diff
>
>
> When run on Windows + cygwin, chmod fails because it fails to call {{makeShellPath}} on the parameter.
> The strack trace with 0.19.1 below:
> {noformat}
> org.apache.hadoop.util.Shell$ExitCodeException: chmod: changing permissions of `C:\\files\\hudson\\hadoop\\task-tracker\\taskTracker\\jobcache\\job_200908191713_0008\\job.xml': No such file or directory
> 	at org.apache.hadoop.util.Shell.runCommand(Shell.java:195)
> 	at org.apache.hadoop.util.Shell.run(Shell.java:134)
> 	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:286)
> 	at org.apache.hadoop.util.Shell.execCommand(Shell.java:338)
> 	at org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:540)
> 	at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:532)
> 	at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:274)
> 	at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:364)
> 	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:487)
> 	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:468)
> 	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:375)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:208)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)
> 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1214)
> 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1195)
> 	at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:750)
> 	at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1656)
> 	at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:102)
> 	at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1621)
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1008-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 25 23:33:42 2009
Return-Path: <common-issues-return-1008-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95738 invoked from network); 25 Aug 2009 23:33:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 25 Aug 2009 23:33:42 -0000
Received: (qmail 89286 invoked by uid 500); 25 Aug 2009 23:34:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89184 invoked by uid 500); 25 Aug 2009 23:34:02 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88828 invoked by uid 99); 25 Aug 2009 23:34:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 23:34:01 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2009 23:32:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 58096234C1E7
	for <common-issues@hadoop.apache.org>; Tue, 25 Aug 2009 16:31:59 -0700 (PDT)
Message-ID: <397903046.1251243119359.JavaMail.jira@brutus>
Date: Tue, 25 Aug 2009 16:31:59 -0700 (PDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6210) Bind Hadoop RPC outgoing to a
 specific nic
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Bind Hadoop RPC outgoing to a specific nic
------------------------------------------

                 Key: HADOOP-6210
                 URL: https://issues.apache.org/jira/browse/HADOOP-6210
             Project: Hadoop Common
          Issue Type: New Feature
    Affects Versions: 0.20.0
            Reporter: Allen Wittenauer


In environments with multiple interfaces on a machine, it would be useful to be able to bind outgoing traffic to specific interfaces rather than letting the outbound be picked at random.  This is especially important for things like Solaris IPMP.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1009-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 00:23:56 2009
Return-Path: <common-issues-return-1009-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14943 invoked from network); 26 Aug 2009 00:23:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 00:23:56 -0000
Received: (qmail 23094 invoked by uid 500); 26 Aug 2009 00:24:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23033 invoked by uid 500); 26 Aug 2009 00:24:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23022 invoked by uid 99); 26 Aug 2009 00:24:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 00:24:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 00:24:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 5E95B234C04C
	for <common-issues@hadoop.apache.org>; Tue, 25 Aug 2009 17:23:59 -0700 (PDT)
Message-ID: <1035672908.1251246239372.JavaMail.jira@brutus>
Date: Tue, 25 Aug 2009 17:23:59 -0700 (PDT)
From: "Bill Zeller (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5073) Hadoop 1.0 Interface Classification
 - scope (visibility - public/private) and stability
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747718#action_12747718 ] 

Bill Zeller commented on HADOOP-5073:
-------------------------------------

If all methods of a subclass are not explicitly documented, I worry that it will become difficult to program against the code directly (without looking at generated javadoc). For example, I imagine needing to trace a method's ancestors or class hierarchy to determine its classification. Would it be too difficult or bloated to add public/stable (etc) to every applicable method?

> Hadoop 1.0 Interface Classification - scope (visibility - public/private) and stability
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5073
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5073
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: 5073_demo.png, HADOOP-5073.patch
>
>
> This jira proposes an interface classification for hadoop interfaces.
> The discussion was started in email alias core-dev@hadoop.apache.org in Nov 2008.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1010-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 01:25:56 2009
Return-Path: <common-issues-return-1010-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28910 invoked from network); 26 Aug 2009 01:25:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 01:25:56 -0000
Received: (qmail 92612 invoked by uid 500); 26 Aug 2009 01:26:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92575 invoked by uid 500); 26 Aug 2009 01:26:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92565 invoked by uid 99); 26 Aug 2009 01:26:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 01:26:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 01:26:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4D85F234C1EB
	for <common-issues@hadoop.apache.org>; Tue, 25 Aug 2009 18:25:59 -0700 (PDT)
Message-ID: <217995990.1251249959312.JavaMail.jira@brutus>
Date: Tue, 25 Aug 2009 18:25:59 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5073) Hadoop 1.0 Interface Classification -
 scope (visibility - public/private) and stability
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-5073:
-------------------------------------------

    Attachment: c5073_20090825.png
                c5073_20090825.patch

> It turned out to be better to have separate annotations for each of the possible labels: IDE integration was much better and the values show up more clearly in the Javadoc and IDE integreated Javadoc viewer.
Hi Jakob, how are the separated annotations better than a single annotation with a parameter?

c5073_20090825.patch: I tried a little example.  It seems that a single annotation works well.
c5073_20090825.png: the javadoc screen shot.

> Hadoop 1.0 Interface Classification - scope (visibility - public/private) and stability
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5073
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5073
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: 5073_demo.png, c5073_20090825.patch, c5073_20090825.png, HADOOP-5073.patch
>
>
> This jira proposes an interface classification for hadoop interfaces.
> The discussion was started in email alias core-dev@hadoop.apache.org in Nov 2008.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1011-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 02:26:57 2009
Return-Path: <common-issues-return-1011-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50993 invoked from network); 26 Aug 2009 02:26:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 02:26:56 -0000
Received: (qmail 41083 invoked by uid 500); 26 Aug 2009 02:27:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41034 invoked by uid 500); 26 Aug 2009 02:27:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41023 invoked by uid 99); 26 Aug 2009 02:27:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 02:27:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 02:27:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 44D76234C04C
	for <common-issues@hadoop.apache.org>; Tue, 25 Aug 2009 19:26:59 -0700 (PDT)
Message-ID: <1557437862.1251253619268.JavaMail.jira@brutus>
Date: Tue, 25 Aug 2009 19:26:59 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6205) Project java code has a significant
 number of declaration javadoc warnings and errors
In-Reply-To: <2108675829.1250797574846.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6205:
---------------------------------------

    Attachment: Eclipse-JavaDoc-errors.png

To reproduce the same situation in the Eclipse it needs to be configured as show in the attachment. These settings reveal 1000+ issues with missing tags or fields/parameters which can't be resolved. And this is for {{public}} methods only

If missing tags might be considered as a warnings then unresolvable parameters have to be looked upon as errors and addressed accordingly.

I can't find how to reproduce this analysis with JavaDoc+javac tools yet..

I'm going to attach the list of 'unresolved parameters' for future consideration.

> Project java code has a significant number of declaration javadoc warnings and errors
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6205
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6205
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: Eclipse-JavaDoc-errors.png
>
>
> The issue here is that code inspections (say by IntelliJ IDEA) show that a number of classes have javadoc problems in declarations. Some of those are minor, e.g. missing @return, @param, @throws tags.
> Some of them are more severe, e.g. using non-existing parameter names in a method javadoc (see HDFS' BlockManager.removeFromInvalidates(..) for an example)
> I'd like to add more sophisticated code analysis to be able to fix at least javadoc declaration errors in order to produce better code quality.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1012-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 04:11:59 2009
Return-Path: <common-issues-return-1012-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89877 invoked from network); 26 Aug 2009 04:11:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 04:11:59 -0000
Received: (qmail 27756 invoked by uid 500); 26 Aug 2009 04:12:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27692 invoked by uid 500); 26 Aug 2009 04:12:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27682 invoked by uid 99); 26 Aug 2009 04:12:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 04:12:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 04:12:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 605A7234C1F1
	for <common-issues@hadoop.apache.org>; Tue, 25 Aug 2009 21:11:59 -0700 (PDT)
Message-ID: <1106088236.1251259919393.JavaMail.jira@brutus>
Date: Tue, 25 Aug 2009 21:11:59 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6211) Hdfs source code has significant
 number of JavaDocs errors and warnings
In-Reply-To: <1500634751.1251259919382.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747759#action_12747759 ] 

Konstantin Boudnik commented on HADOOP-6211:
--------------------------------------------


{noformat}
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	DFSClient.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs	line 230
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	FSDataset.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/datanode	line 1581
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	FSDataset.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/datanode	line 1582
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	EditLogBackupInputStream.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/namenode	line 25
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	EditLogBackupOutputStream.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/namenode	line 35
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	EditLogFileInputStream.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/namenode	line 26
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	EditLogFileOutputStream.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/namenode	line 33
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	EditLogInputStream.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/namenode	line 30
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	JournalStream.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/namenode	line 31
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	JournalStream.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/namenode	line 31
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	FSNamesystemMetrics.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/namenode/metrics	line 41
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	IOMapperBase.java	Hdfs.inj/src/test/hdfs-with-mr/org/apache/hadoop/fs	line 30
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	IOMapperBase.java	Hdfs.inj/src/test/hdfs-with-mr/org/apache/hadoop/fs	line 31
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	IOMapperBase.java	Hdfs.inj/src/test/hdfs-with-mr/org/apache/hadoop/fs	line 103
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	IOMapperBase.java	Hdfs.inj/src/test/hdfs-with-mr/org/apache/hadoop/fs	line 105
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TestHeartbeatHandling.java	Hdfs.inj/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode	line 21
Javadoc: 'public' visibility for malformed doc comments hides this 'protected' reference	UpgradeManager.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/common	line 29
Javadoc: any cannot be resolved to a type	Balancer.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/balancer	line 1459
Javadoc: doMain cannot be resolved or is not a field	DfsTask.java	Hdfs.inj/src/ant/org/apache/hadoop/ant	line 167
Javadoc: Exception IOException is not declared	MiniDFSCluster.java	Hdfs.inj/src/test/hdfs/org/apache/hadoop/hdfs	line 937
Javadoc: FSEditLog.EditLogFileInputStream cannot be resolved to a type	JournalStream.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/namenode	line 29
Javadoc: FSEditLog.EditLogFileOutputStream cannot be resolved to a type	JournalStream.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/namenode	line 28
Javadoc: Invalid param tag name	FSImage.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/namenode	line 1889
Javadoc: Invalid param tag name	FSImage.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/namenode	line 1890
Javadoc: Invalid param tag name	ReplicationTargetChooser.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/namenode	line 62
Javadoc: Invalid param tag name	ReplicationTargetChooser.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/namenode	line 63
Javadoc: Invalid param tag name	ReplicationTargetChooser.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/namenode	line 64
Javadoc: Invalid param tag name	ReplicationTargetChooser.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/namenode	line 65
Javadoc: Invalid param tag name	ReplicationTargetChooser.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/namenode	line 86
Javadoc: Invalid param tag name	ReplicationTargetChooser.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/namenode	line 87
Javadoc: Invalid param tag name	ReplicationTargetChooser.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/namenode	line 88
Javadoc: Invalid param tag name	ReplicationTargetChooser.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/namenode	line 89
Javadoc: Invalid param tag name	ReplicationTargetChooser.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/namenode	line 90
Javadoc: Invalid param tag name	DataTransferTestUtil.java	Hdfs.inj/src/test/aop/org/apache/hadoop/fi	line 114
Javadoc: Invalid param tag name	MiniDFSCluster.java	Hdfs.inj/src/test/hdfs/org/apache/hadoop/hdfs	line 696
Javadoc: Invalid param tag name	MiniDFSCluster.java	Hdfs.inj/src/test/hdfs/org/apache/hadoop/hdfs	line 697
Javadoc: Missing class name	DFSClient.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs	line 795
Javadoc: Missing class name	DFSClient.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs	line 813
Javadoc: Missing class name	ClientProtocol.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/protocol	line 254
Javadoc: Missing class name	Storage.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/common	line 343
Javadoc: Missing return type description	FindFreePort.java	Hdfs.inj/src/contrib/hdfsproxy/src/test/org/apache/hadoop/hdfsproxy	line 65
Javadoc: org.apache.hadoop.hdfs.server.datanode.metrics.DataNodeStatisticsMBean cannot be resolved to a type	FSDatasetMBean.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/datanode/metrics	line 34
Javadoc: Parameter FileToMd5 is not declared	TestStorageRestore.java	Hdfs.inj/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode	line 165
Javadoc: Parameter num is not declared	FindFreePort.java	Hdfs.inj/src/contrib/hdfsproxy/src/test/org/apache/hadoop/hdfsproxy	line 32
Javadoc: RemoteException cannot be resolved to a type	NamenodeProtocol.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/protocol	line 65
Javadoc: The method computeInvalidateWork(int) is undefined for the type FSNamesystem	TestComputeInvalidateWork.java	Hdfs.inj/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode	line 15
Javadoc: The method newNamespaceID() from the type FSImage is not visible	FSNamesystem.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/namenode	line 1894
Javadoc: The method register() from the type DataNode is not visible	FSNamesystem.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/namenode	line 1748
Javadoc: Unexpected tag	ProxyForwardServlet.java	Hdfs.inj/src/contrib/hdfsproxy/src/java/org/apache/hadoop/hdfsproxy	line 79
Javadoc: Unexpected tag	ProxyForwardServlet.java	Hdfs.inj/src/contrib/hdfsproxy/src/java/org/apache/hadoop/hdfsproxy	line 93
Javadoc: Unexpected tag	BlockMetadataHeader.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/datanode	line 102
Javadoc: Unexpected tag	FSDirectory.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/namenode	line 1093
Javadoc: Unexpected tag	FSPermissionChecker.java	Hdfs.inj/src/java/org/apache/hadoop/hdfs/server/namenode	line 65
Javadoc: Unexpected tag	DataTransferTestUtil.java	Hdfs.inj/src/test/aop/org/apache/hadoop/fi	line 113
{noformat}

> Hdfs source code has significant number of JavaDocs errors and warnings
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-6211
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6211
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Konstantin Boudnik
>
> By turning some extra source code inspections in Eclipse one can see many JavaDoc warnings and errors in the source code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1013-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 04:14:41 2009
Return-Path: <common-issues-return-1013-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90107 invoked from network); 26 Aug 2009 04:14:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 04:14:40 -0000
Received: (qmail 28543 invoked by uid 500); 26 Aug 2009 04:15:05 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28444 invoked by uid 500); 26 Aug 2009 04:15:05 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28271 invoked by uid 99); 26 Aug 2009 04:15:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 04:15:03 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 04:12:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 5D924234C1EF
	for <common-issues@hadoop.apache.org>; Tue, 25 Aug 2009 21:11:59 -0700 (PDT)
Message-ID: <1500634751.1251259919382.JavaMail.jira@brutus>
Date: Tue, 25 Aug 2009 21:11:59 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6211) Hdfs source code has significant
 number of JavaDocs errors and warnings
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Hdfs source code has significant number of JavaDocs errors and warnings
-----------------------------------------------------------------------

                 Key: HADOOP-6211
                 URL: https://issues.apache.org/jira/browse/HADOOP-6211
             Project: Hadoop Common
          Issue Type: Sub-task
            Reporter: Konstantin Boudnik


By turning some extra source code inspections in Eclipse one can see many JavaDoc warnings and errors in the source code.


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1014-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 04:21:58 2009
Return-Path: <common-issues-return-1014-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91208 invoked from network); 26 Aug 2009 04:21:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 04:21:58 -0000
Received: (qmail 34398 invoked by uid 500); 26 Aug 2009 04:22:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34330 invoked by uid 500); 26 Aug 2009 04:22:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34308 invoked by uid 99); 26 Aug 2009 04:22:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 04:22:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 04:22:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4DF40234C04C
	for <common-issues@hadoop.apache.org>; Tue, 25 Aug 2009 21:21:59 -0700 (PDT)
Message-ID: <1727594795.1251260519304.JavaMail.jira@brutus>
Date: Tue, 25 Aug 2009 21:21:59 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6205) Project java code has a significant
 number of declaration javadoc warnings and errors
In-Reply-To: <2108675829.1250797574846.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747761#action_12747761 ] 

Konstantin Boudnik commented on HADOOP-6205:
--------------------------------------------

List of JavaDoc errors for Common project
{noformat}
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	CapacityTaskScheduler.java	MR/src/contrib/capacity-scheduler/src/java/org/apache/hadoop/mapred	line 40
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	JobInitializationPoller.java	MR/src/contrib/capacity-scheduler/src/java/org/apache/hadoop/mapred	line 36
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	JobQueuesManager.java	MR/src/contrib/capacity-scheduler/src/java/org/apache/hadoop/mapred	line 35
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TestCapacitySchedulerWithJobTracker.java	MR/src/contrib/capacity-scheduler/src/test/org/apache/hadoop/mapred	line 32
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TestCapacitySchedulerWithJobTracker.java	MR/src/contrib/capacity-scheduler/src/test/org/apache/hadoop/mapred	line 64
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	DynamicPriorityScheduler.java	MR/src/contrib/dynamic-scheduler/src/java/org/apache/hadoop/mapred	line 43
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	DynamicPriorityServlet.java	MR/src/contrib/dynamic-scheduler/src/java/org/apache/hadoop/mapred	line 42
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	PriorityScheduler.java	MR/src/contrib/dynamic-scheduler/src/java/org/apache/hadoop/mapred	line 47
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	FairScheduler.java	MR/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred	line 41
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	FairSchedulerEventLog.java	MR/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred	line 49
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	LoadManager.java	MR/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred	line 28
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskSelector.java	MR/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred	line 29
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TestTestDriver.java	MR/src/contrib/mrunit/src/test/org/apache/hadoop/mrunit	line 35
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	CleanupQueue.java	MR/src/java/org/apache/hadoop/mapred	line 42
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	EagerTaskInitializationListener.java	MR/src/java/org/apache/hadoop/mapred	line 34
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	EagerTaskInitializationListener.java	MR/src/java/org/apache/hadoop/mapred	line 35
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	IFileInputStream.java	MR/src/java/org/apache/hadoop/mapred	line 30
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	InterTrackerProtocol.java	MR/src/java/org/apache/hadoop/mapred	line 33
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	InterTrackerProtocol.java	MR/src/java/org/apache/hadoop/mapred	line 76
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	InterTrackerProtocol.java	MR/src/java/org/apache/hadoop/mapred	line 93
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	JobChangeEvent.java	MR/src/java/org/apache/hadoop/mapred	line 21
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	JobInProgressListener.java	MR/src/java/org/apache/hadoop/mapred	line 43
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	JobQueueJobInProgressListener.java	MR/src/java/org/apache/hadoop/mapred	line 29
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	JobQueueTaskScheduler.java	MR/src/java/org/apache/hadoop/mapred	line 31
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	JobStatusChangeEvent.java	MR/src/java/org/apache/hadoop/mapred	line 21
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	JobTracker.java	MR/src/java/org/apache/hadoop/mapred	line 2832
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	JobTracker.java	MR/src/java/org/apache/hadoop/mapred	line 3940
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	LimitTasksPerJobTaskScheduler.java	MR/src/java/org/apache/hadoop/mapred	line 32
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	LinuxTaskController.java	MR/src/java/org/apache/hadoop/mapred	line 37
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	LinuxTaskController.java	MR/src/java/org/apache/hadoop/mapred	line 41
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	LinuxTaskController.java	MR/src/java/org/apache/hadoop/mapred	line 48
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	LocalJobRunner.java	MR/src/java/org/apache/hadoop/mapred	line 466
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	QueueManager.java	MR/src/java/org/apache/hadoop/mapred	line 117
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	QueueManager.java	MR/src/java/org/apache/hadoop/mapred	line 137
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	QueueManager.java	MR/src/java/org/apache/hadoop/mapred	line 145
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TIPStatus.java	MR/src/java/org/apache/hadoop/mapred	line 20
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskScheduler.java	MR/src/java/org/apache/hadoop/mapred	line 32
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskScheduler.java	MR/src/java/org/apache/hadoop/mapred	line 33
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskScheduler.java	MR/src/java/org/apache/hadoop/mapred	line 35
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskScheduler.java	MR/src/java/org/apache/hadoop/mapred	line 38
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskScheduler.java	MR/src/java/org/apache/hadoop/mapred	line 40
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskScheduler.java	MR/src/java/org/apache/hadoop/mapred	line 41
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskTrackerManager.java	MR/src/java/org/apache/hadoop/mapred	line 47
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskTrackerManager.java	MR/src/java/org/apache/hadoop/mapred	line 48
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskTrackerManager.java	MR/src/java/org/apache/hadoop/mapred	line 49
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskTrackerManager.java	MR/src/java/org/apache/hadoop/mapred	line 54
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskTrackerManager.java	MR/src/java/org/apache/hadoop/mapred	line 55
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskTrackerManager.java	MR/src/java/org/apache/hadoop/mapred	line 56
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskTrackerManager.java	MR/src/java/org/apache/hadoop/mapred	line 61
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskTrackerManager.java	MR/src/java/org/apache/hadoop/mapred	line 62
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskTrackerManager.java	MR/src/java/org/apache/hadoop/mapred	line 64
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskTrackerStatus.java	MR/src/java/org/apache/hadoop/mapred	line 256
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskTrackerStatus.java	MR/src/java/org/apache/hadoop/mapred	line 258
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskUmbilicalProtocol.java	MR/src/java/org/apache/hadoop/mapred	line 146
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	PipesNonJavaInputFormat.java	MR/src/java/org/apache/hadoop/mapred/pipes	line 38
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	MarkableIterator.java	MR/src/java/org/apache/hadoop/mapreduce	line 26
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TestLocalizationWithLinuxTaskController.java	MR/src/test/mapred/org/apache/hadoop/mapred	line 39
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TestLocalizationWithLinuxTaskController.java	MR/src/test/mapred/org/apache/hadoop/mapred	line 92
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TestLocalizationWithLinuxTaskController.java	MR/src/test/mapred/org/apache/hadoop/mapred	line 167
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TestRecoveryManager.java	MR/src/test/mapred/org/apache/hadoop/mapred	line 38
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TestRecoveryManager.java	MR/src/test/mapred/org/apache/hadoop/mapred	line 39
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TestRecoveryManager.java	MR/src/test/mapred/org/apache/hadoop/mapred	line 51
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TestRecoveryManager.java	MR/src/test/mapred/org/apache/hadoop/mapred	line 146
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TestTTMemoryReporting.java	MR/src/test/mapred/org/apache/hadoop/mapred	line 38
Javadoc: 'public' visibility for malformed doc comments hides this 'protected' reference	Schedulable.java	MR/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred	line 102
Javadoc: 'public' visibility for malformed doc comments hides this 'protected' reference	TestTestDriver.java	MR/src/contrib/mrunit/src/test/org/apache/hadoop/mrunit	line 96
Javadoc: 'public' visibility for malformed doc comments hides this 'protected' reference	FileInputFormat.java	MR/src/java/org/apache/hadoop/mapred	line 52
Javadoc: 'public' visibility for malformed doc comments hides this 'protected' reference	FileInputFormat.java	MR/src/java/org/apache/hadoop/mapred	line 196
Javadoc: 'public' visibility for malformed doc comments hides this 'protected' reference	Job.java	MR/src/java/org/apache/hadoop/mapreduce	line 235
Javadoc: 'public' visibility for malformed doc comments hides this 'protected' reference	Mapper.java	MR/src/java/org/apache/hadoop/mapreduce	line 41
Javadoc: 'public' visibility for malformed doc comments hides this 'protected' reference	Mapper.java	MR/src/java/org/apache/hadoop/mapreduce	line 44
Javadoc: 'public' visibility for malformed doc comments hides this 'protected' reference	FileInputFormat.java	MR/src/java/org/apache/hadoop/mapreduce/lib/input	line 48
Javadoc: Exception Exception is not declared	ImportOptions.java	MR/src/contrib/sqoop/src/java/org/apache/hadoop/sqoop	line 389
Javadoc: Exception Exception is not declared	ImportOptions.java	MR/src/contrib/sqoop/src/java/org/apache/hadoop/sqoop	line 526
Javadoc: Exception IOException is not declared	DistributedCache.java	MR/src/java/org/apache/hadoop/mapreduce/filecache	line 596
Javadoc: Exception IOException is not declared	DistributedCache.java	MR/src/java/org/apache/hadoop/mapreduce/filecache	line 607
Javadoc: HalfWaitingMapper cannot be resolved to a type	TestJobTrackerRestart.java	MR/src/test/mapred/org/apache/hadoop/mapred	line 145
Javadoc: Invalid member type qualification	SequenceFileInputFilter.java	MR/src/java/org/apache/hadoop/mapreduce/lib/input	line 135
Javadoc: Invalid member type qualification	SequenceFileInputFilter.java	MR/src/java/org/apache/hadoop/mapreduce/lib/input	line 179
Javadoc: Invalid member type qualification	SequenceFileInputFilter.java	MR/src/java/org/apache/hadoop/mapreduce/lib/input	line 240
Javadoc: Invalid param tag name	PriorityAuthorization.java	MR/src/contrib/dynamic-scheduler/src/java/org/apache/hadoop/mapred	line 172
Javadoc: Job cannot be resolved to a type	DelegatingMapper.java	MR/src/java/org/apache/hadoop/mapreduce/lib/input	line 31
Javadoc: JobContext cannot be resolved to a type	FileSplit.java	MR/src/java/org/apache/hadoop/mapreduce/lib/input	line 33
Javadoc: Malformed link reference	JobConf.java	MR/src/java/org/apache/hadoop/mapred	line 1545
Javadoc: Missing #: "Configuration.getJar()}),"	TaskRunner.java	MR/src/java/org/apache/hadoop/mapred	line 702
Javadoc: Missing closing brace for inline tag	MultithreadedMapper.java	MR/src/java/org/apache/hadoop/mapreduce/lib/map	line 51
Javadoc: Missing reference	UlimitApp.java	MR/src/contrib/streaming/src/test/org/apache/hadoop/streaming	line 27
Javadoc: Missing return type description	CapacitySchedulerConf.java	MR/src/contrib/capacity-scheduler/src/java/org/apache/hadoop/mapred	line 376
Javadoc: Missing return type description	CapacitySchedulerConf.java	MR/src/contrib/capacity-scheduler/src/java/org/apache/hadoop/mapred	line 394
Javadoc: Missing return type description	JSPUtil.java	MR/src/java/org/apache/hadoop/mapred	line 95
Javadoc: Missing return type description	TaskInProgress.java	MR/src/java/org/apache/hadoop/mapred	line 795
Javadoc: Missing return type description	TaskScheduler.java	MR/src/java/org/apache/hadoop/mapred	line 91
Javadoc: OutputFormat cannot be resolved to a type	RawKeyTextOutputFormat.java	MR/src/contrib/sqoop/src/java/org/apache/hadoop/sqoop/mapred	line 37
Javadoc: Parameter aStartingDir is not declared	FileListing.java	MR/src/contrib/sqoop/src/java/org/apache/hadoop/sqoop/util	line 108
Javadoc: Parameter database is not declared	ImportOptions.java	MR/src/contrib/sqoop/src/java/org/apache/hadoop/sqoop	line 126
Javadoc: Parameter datatype is not declared	JobStatisticsInterface.java	MR/src/contrib/vaidya/src/java/org/apache/hadoop/vaidya/statistics/job	line 64
Javadoc: Parameter debugCommand is not declared	TestMiniMRMapRedDebugScript.java	MR/src/test/mapred/org/apache/hadoop/mapred	line 107
Javadoc: Parameter fileName is not declared	XMLUtils.java	MR/src/contrib/vaidya/src/java/org/apache/hadoop/vaidya/util	line 151
Javadoc: Parameter jobConf is not declared	JobStatistics.java	MR/src/contrib/vaidya/src/java/org/apache/hadoop/vaidya/statistics/job	line 76
Javadoc: Parameter jobHistoryLogFile is not declared	PostExPerformanceDiagnoser.java	MR/src/contrib/vaidya/src/java/org/apache/hadoop/vaidya/postexdiagnosis	line 94
Javadoc: Parameter jobInProgressListener is not declared	TaskTrackerManager.java	MR/src/java/org/apache/hadoop/mapred	line 49
Javadoc: Parameter jobInProgressListener is not declared	TaskTrackerManager.java	MR/src/java/org/apache/hadoop/mapred	line 56
Javadoc: Parameter recordOffset is not declared	BufferSorter.java	MR/src/java/org/apache/hadoop/mapred	line 47
Javadoc: Parameter specificMgr is not declared	SqlManager.java	MR/src/contrib/sqoop/src/java/org/apache/hadoop/sqoop/manager	line 57
Javadoc: Parameter tableClassName is not declared	ClassLoaderStack.java	MR/src/contrib/sqoop/src/java/org/apache/hadoop/sqoop/util	line 57
Javadoc: Parameter testsConfFile is not declared	PostExPerformanceDiagnoser.java	MR/src/contrib/vaidya/src/java/org/apache/hadoop/vaidya/postexdiagnosis	line 95
Javadoc: RM_CONF_FILE cannot be resolved to a type	CapacitySchedulerConf.java	MR/src/contrib/capacity-scheduler/src/java/org/apache/hadoop/mapred	line 109
Javadoc: The method assignTasks(TaskTracker) in the type TaskScheduler is not applicable for the arguments (TaskTrackerStatus)	TestTTMemoryReporting.java	MR/src/test/mapred/org/apache/hadoop/mapred	line 41
Javadoc: The method equals(Object) in the type Pair is not applicable for the arguments (Pair)	TestPair.java	MR/src/contrib/mrunit/src/test/org/apache/hadoop/mrunit/types	line 34
Javadoc: The method getNumberOfThreads(JobContext) in the type MultithreadedMapper is not applicable for the arguments (Configuration)	MultithreadedMapper.java	MR/src/java/org/apache/hadoop/mapreduce/lib/map	line 51
Javadoc: The method map(Object, Object, Mapper.Context) in the type Mapper is not applicable for the arguments (Object, Object, Mapper<KEYIN,VALUEIN,KEYOUT,VALUEOUT>.Context)	Mapper.java	MR/src/java/org/apache/hadoop/mapreduce	line 42
Javadoc: The method obtainNewMapTask(TaskTrackerStatus, int, int) in the type JobInProgress is not applicable for the arguments (TaskTrackerStatus, int)	DefaultTaskSelector.java	MR/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred	line 25
Javadoc: The method obtainNewReduceTask(TaskTrackerStatus, int, int) in the type JobInProgress is not applicable for the arguments (TaskTrackerStatus, int)	DefaultTaskSelector.java	MR/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred	line 26
Javadoc: The method reduce(Object, Iterable, Reducer.Context) in the type Reducer is not applicable for the arguments (Object, Iterable, Reducer<KEYIN,VALUEIN,KEYOUT,VALUEOUT>.Context)	Reducer.java	MR/src/java/org/apache/hadoop/mapreduce	line 86
Javadoc: The method setMapperClass(Job, Class<? extends Mapper<K1,V1,K2,V2>>) in the type MultithreadedMapper is not applicable for the arguments (Configuration, Class)	MultithreadedMapper.java	MR/src/java/org/apache/hadoop/mapreduce/lib/map	line 49
Javadoc: The method setSchedulingInfo(String, Object) is undefined for the type QueueManager	QueueManager.java	MR/src/java/org/apache/hadoop/mapred	line 245
Javadoc: The method write(KEYOUT, VALUEOUT) in the type TaskInputOutputContext<KEYIN,VALUEIN,KEYOUT,VALUEOUT> is not applicable for the arguments (Object, Object)	Reducer.java	MR/src/java/org/apache/hadoop/mapreduce	line 91
Javadoc: Unexpected tag	TestDriver.java	MR/src/contrib/mrunit/src/java/org/apache/hadoop/mrunit	line 63
Javadoc: Unexpected tag	TestDriver.java	MR/src/contrib/mrunit/src/java/org/apache/hadoop/mrunit	line 126
Javadoc: Unexpected tag	MultithreadedMapRunner.java	MR/src/java/org/apache/hadoop/mapred/lib	line 40
Javadoc: Unexpected tag	ValueAggregatorReducer.java	MR/src/java/org/apache/hadoop/mapred/lib/aggregate	line 49
Javadoc: Unexpected tag	MultithreadedMapper.java	MR/src/java/org/apache/hadoop/mapreduce/lib/map	line 43
Javadoc: Unexpected tag	TestQueueAclsForCurrentUser.java	MR/src/test/mapred/org/apache/hadoop/mapred	line 58
Javadoc: WordCount.MapClass cannot be resolved to a type	MultiFileWordCount.java	MR/src/examples/org/apache/hadoop/examples	line 202
{noformat}

> Project java code has a significant number of declaration javadoc warnings and errors
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6205
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6205
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: Eclipse-JavaDoc-errors.png
>
>
> The issue here is that code inspections (say by IntelliJ IDEA) show that a number of classes have javadoc problems in declarations. Some of those are minor, e.g. missing @return, @param, @throws tags.
> Some of them are more severe, e.g. using non-existing parameter names in a method javadoc (see HDFS' BlockManager.removeFromInvalidates(..) for an example)
> I'd like to add more sophisticated code analysis to be able to fix at least javadoc declaration errors in order to produce better code quality.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1015-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 04:23:56 2009
Return-Path: <common-issues-return-1015-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91734 invoked from network); 26 Aug 2009 04:23:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 04:23:56 -0000
Received: (qmail 37544 invoked by uid 500); 26 Aug 2009 04:24:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37471 invoked by uid 500); 26 Aug 2009 04:24:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37455 invoked by uid 99); 26 Aug 2009 04:24:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 04:24:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 04:24:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 61794234C1E7
	for <common-issues@hadoop.apache.org>; Tue, 25 Aug 2009 21:23:59 -0700 (PDT)
Message-ID: <1113513611.1251260639398.JavaMail.jira@brutus>
Date: Tue, 25 Aug 2009 21:23:59 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6212) MapReduce source code has significant
 number of JavaDocs errors and warnings
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

MapReduce source code has significant number of JavaDocs errors and warnings
----------------------------------------------------------------------------

                 Key: HADOOP-6212
                 URL: https://issues.apache.org/jira/browse/HADOOP-6212
             Project: Hadoop Common
          Issue Type: Sub-task
            Reporter: Konstantin Boudnik




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1016-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 04:23:56 2009
Return-Path: <common-issues-return-1016-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91776 invoked from network); 26 Aug 2009 04:23:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 04:23:56 -0000
Received: (qmail 37624 invoked by uid 500); 26 Aug 2009 04:24:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37574 invoked by uid 500); 26 Aug 2009 04:24:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37553 invoked by uid 99); 26 Aug 2009 04:24:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 04:24:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 04:24:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 7EA58234C48D
	for <common-issues@hadoop.apache.org>; Tue, 25 Aug 2009 21:23:59 -0700 (PDT)
Message-ID: <1722315570.1251260639518.JavaMail.jira@brutus>
Date: Tue, 25 Aug 2009 21:23:59 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6212) MapReduce source code has
 significant number of JavaDocs errors and warnings
In-Reply-To: <1113513611.1251260639398.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747763#action_12747763 ] 

Konstantin Boudnik commented on HADOOP-6212:
--------------------------------------------

{noformat}
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	CapacityTaskScheduler.java	MR/src/contrib/capacity-scheduler/src/java/org/apache/hadoop/mapred	line 40
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	JobInitializationPoller.java	MR/src/contrib/capacity-scheduler/src/java/org/apache/hadoop/mapred	line 36
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	JobQueuesManager.java	MR/src/contrib/capacity-scheduler/src/java/org/apache/hadoop/mapred	line 35
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TestCapacitySchedulerWithJobTracker.java	MR/src/contrib/capacity-scheduler/src/test/org/apache/hadoop/mapred	line 32
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TestCapacitySchedulerWithJobTracker.java	MR/src/contrib/capacity-scheduler/src/test/org/apache/hadoop/mapred	line 64
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	DynamicPriorityScheduler.java	MR/src/contrib/dynamic-scheduler/src/java/org/apache/hadoop/mapred	line 43
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	DynamicPriorityServlet.java	MR/src/contrib/dynamic-scheduler/src/java/org/apache/hadoop/mapred	line 42
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	PriorityScheduler.java	MR/src/contrib/dynamic-scheduler/src/java/org/apache/hadoop/mapred	line 47
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	FairScheduler.java	MR/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred	line 41
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	FairSchedulerEventLog.java	MR/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred	line 49
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	LoadManager.java	MR/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred	line 28
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskSelector.java	MR/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred	line 29
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TestTestDriver.java	MR/src/contrib/mrunit/src/test/org/apache/hadoop/mrunit	line 35
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	CleanupQueue.java	MR/src/java/org/apache/hadoop/mapred	line 42
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	EagerTaskInitializationListener.java	MR/src/java/org/apache/hadoop/mapred	line 34
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	EagerTaskInitializationListener.java	MR/src/java/org/apache/hadoop/mapred	line 35
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	IFileInputStream.java	MR/src/java/org/apache/hadoop/mapred	line 30
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	InterTrackerProtocol.java	MR/src/java/org/apache/hadoop/mapred	line 33
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	InterTrackerProtocol.java	MR/src/java/org/apache/hadoop/mapred	line 76
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	InterTrackerProtocol.java	MR/src/java/org/apache/hadoop/mapred	line 93
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	JobChangeEvent.java	MR/src/java/org/apache/hadoop/mapred	line 21
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	JobInProgressListener.java	MR/src/java/org/apache/hadoop/mapred	line 43
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	JobQueueJobInProgressListener.java	MR/src/java/org/apache/hadoop/mapred	line 29
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	JobQueueTaskScheduler.java	MR/src/java/org/apache/hadoop/mapred	line 31
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	JobStatusChangeEvent.java	MR/src/java/org/apache/hadoop/mapred	line 21
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	JobTracker.java	MR/src/java/org/apache/hadoop/mapred	line 2832
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	JobTracker.java	MR/src/java/org/apache/hadoop/mapred	line 3940
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	LimitTasksPerJobTaskScheduler.java	MR/src/java/org/apache/hadoop/mapred	line 32
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	LinuxTaskController.java	MR/src/java/org/apache/hadoop/mapred	line 37
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	LinuxTaskController.java	MR/src/java/org/apache/hadoop/mapred	line 41
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	LinuxTaskController.java	MR/src/java/org/apache/hadoop/mapred	line 48
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	LocalJobRunner.java	MR/src/java/org/apache/hadoop/mapred	line 466
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	QueueManager.java	MR/src/java/org/apache/hadoop/mapred	line 117
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	QueueManager.java	MR/src/java/org/apache/hadoop/mapred	line 137
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	QueueManager.java	MR/src/java/org/apache/hadoop/mapred	line 145
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TIPStatus.java	MR/src/java/org/apache/hadoop/mapred	line 20
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskScheduler.java	MR/src/java/org/apache/hadoop/mapred	line 32
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskScheduler.java	MR/src/java/org/apache/hadoop/mapred	line 33
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskScheduler.java	MR/src/java/org/apache/hadoop/mapred	line 35
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskScheduler.java	MR/src/java/org/apache/hadoop/mapred	line 38
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskScheduler.java	MR/src/java/org/apache/hadoop/mapred	line 40
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskScheduler.java	MR/src/java/org/apache/hadoop/mapred	line 41
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskTrackerManager.java	MR/src/java/org/apache/hadoop/mapred	line 47
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskTrackerManager.java	MR/src/java/org/apache/hadoop/mapred	line 48
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskTrackerManager.java	MR/src/java/org/apache/hadoop/mapred	line 49
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskTrackerManager.java	MR/src/java/org/apache/hadoop/mapred	line 54
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskTrackerManager.java	MR/src/java/org/apache/hadoop/mapred	line 55
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskTrackerManager.java	MR/src/java/org/apache/hadoop/mapred	line 56
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskTrackerManager.java	MR/src/java/org/apache/hadoop/mapred	line 61
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskTrackerManager.java	MR/src/java/org/apache/hadoop/mapred	line 62
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskTrackerManager.java	MR/src/java/org/apache/hadoop/mapred	line 64
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskTrackerStatus.java	MR/src/java/org/apache/hadoop/mapred	line 256
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskTrackerStatus.java	MR/src/java/org/apache/hadoop/mapred	line 258
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TaskUmbilicalProtocol.java	MR/src/java/org/apache/hadoop/mapred	line 146
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	PipesNonJavaInputFormat.java	MR/src/java/org/apache/hadoop/mapred/pipes	line 38
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	MarkableIterator.java	MR/src/java/org/apache/hadoop/mapreduce	line 26
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TestLocalizationWithLinuxTaskController.java	MR/src/test/mapred/org/apache/hadoop/mapred	line 39
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TestLocalizationWithLinuxTaskController.java	MR/src/test/mapred/org/apache/hadoop/mapred	line 92
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TestLocalizationWithLinuxTaskController.java	MR/src/test/mapred/org/apache/hadoop/mapred	line 167
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TestRecoveryManager.java	MR/src/test/mapred/org/apache/hadoop/mapred	line 38
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TestRecoveryManager.java	MR/src/test/mapred/org/apache/hadoop/mapred	line 39
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TestRecoveryManager.java	MR/src/test/mapred/org/apache/hadoop/mapred	line 51
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TestRecoveryManager.java	MR/src/test/mapred/org/apache/hadoop/mapred	line 146
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	TestTTMemoryReporting.java	MR/src/test/mapred/org/apache/hadoop/mapred	line 38
Javadoc: 'public' visibility for malformed doc comments hides this 'protected' reference	Schedulable.java	MR/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred	line 102
Javadoc: 'public' visibility for malformed doc comments hides this 'protected' reference	TestTestDriver.java	MR/src/contrib/mrunit/src/test/org/apache/hadoop/mrunit	line 96
Javadoc: 'public' visibility for malformed doc comments hides this 'protected' reference	FileInputFormat.java	MR/src/java/org/apache/hadoop/mapred	line 52
Javadoc: 'public' visibility for malformed doc comments hides this 'protected' reference	FileInputFormat.java	MR/src/java/org/apache/hadoop/mapred	line 196
Javadoc: 'public' visibility for malformed doc comments hides this 'protected' reference	Job.java	MR/src/java/org/apache/hadoop/mapreduce	line 235
Javadoc: 'public' visibility for malformed doc comments hides this 'protected' reference	Mapper.java	MR/src/java/org/apache/hadoop/mapreduce	line 41
Javadoc: 'public' visibility for malformed doc comments hides this 'protected' reference	Mapper.java	MR/src/java/org/apache/hadoop/mapreduce	line 44
Javadoc: 'public' visibility for malformed doc comments hides this 'protected' reference	FileInputFormat.java	MR/src/java/org/apache/hadoop/mapreduce/lib/input	line 48
Javadoc: Exception Exception is not declared	ImportOptions.java	MR/src/contrib/sqoop/src/java/org/apache/hadoop/sqoop	line 389
Javadoc: Exception Exception is not declared	ImportOptions.java	MR/src/contrib/sqoop/src/java/org/apache/hadoop/sqoop	line 526
Javadoc: Exception IOException is not declared	DistributedCache.java	MR/src/java/org/apache/hadoop/mapreduce/filecache	line 596
Javadoc: Exception IOException is not declared	DistributedCache.java	MR/src/java/org/apache/hadoop/mapreduce/filecache	line 607
Javadoc: HalfWaitingMapper cannot be resolved to a type	TestJobTrackerRestart.java	MR/src/test/mapred/org/apache/hadoop/mapred	line 145
Javadoc: Invalid member type qualification	SequenceFileInputFilter.java	MR/src/java/org/apache/hadoop/mapreduce/lib/input	line 135
Javadoc: Invalid member type qualification	SequenceFileInputFilter.java	MR/src/java/org/apache/hadoop/mapreduce/lib/input	line 179
Javadoc: Invalid member type qualification	SequenceFileInputFilter.java	MR/src/java/org/apache/hadoop/mapreduce/lib/input	line 240
Javadoc: Invalid param tag name	PriorityAuthorization.java	MR/src/contrib/dynamic-scheduler/src/java/org/apache/hadoop/mapred	line 172
Javadoc: Job cannot be resolved to a type	DelegatingMapper.java	MR/src/java/org/apache/hadoop/mapreduce/lib/input	line 31
Javadoc: JobContext cannot be resolved to a type	FileSplit.java	MR/src/java/org/apache/hadoop/mapreduce/lib/input	line 33
Javadoc: Malformed link reference	JobConf.java	MR/src/java/org/apache/hadoop/mapred	line 1545
Javadoc: Missing #: "Configuration.getJar()}),"	TaskRunner.java	MR/src/java/org/apache/hadoop/mapred	line 702
Javadoc: Missing closing brace for inline tag	MultithreadedMapper.java	MR/src/java/org/apache/hadoop/mapreduce/lib/map	line 51
Javadoc: Missing reference	UlimitApp.java	MR/src/contrib/streaming/src/test/org/apache/hadoop/streaming	line 27
Javadoc: Missing return type description	CapacitySchedulerConf.java	MR/src/contrib/capacity-scheduler/src/java/org/apache/hadoop/mapred	line 376
Javadoc: Missing return type description	CapacitySchedulerConf.java	MR/src/contrib/capacity-scheduler/src/java/org/apache/hadoop/mapred	line 394
Javadoc: Missing return type description	JSPUtil.java	MR/src/java/org/apache/hadoop/mapred	line 95
Javadoc: Missing return type description	TaskInProgress.java	MR/src/java/org/apache/hadoop/mapred	line 795
Javadoc: Missing return type description	TaskScheduler.java	MR/src/java/org/apache/hadoop/mapred	line 91
Javadoc: OutputFormat cannot be resolved to a type	RawKeyTextOutputFormat.java	MR/src/contrib/sqoop/src/java/org/apache/hadoop/sqoop/mapred	line 37
Javadoc: Parameter aStartingDir is not declared	FileListing.java	MR/src/contrib/sqoop/src/java/org/apache/hadoop/sqoop/util	line 108
Javadoc: Parameter database is not declared	ImportOptions.java	MR/src/contrib/sqoop/src/java/org/apache/hadoop/sqoop	line 126
Javadoc: Parameter datatype is not declared	JobStatisticsInterface.java	MR/src/contrib/vaidya/src/java/org/apache/hadoop/vaidya/statistics/job	line 64
Javadoc: Parameter debugCommand is not declared	TestMiniMRMapRedDebugScript.java	MR/src/test/mapred/org/apache/hadoop/mapred	line 107
Javadoc: Parameter fileName is not declared	XMLUtils.java	MR/src/contrib/vaidya/src/java/org/apache/hadoop/vaidya/util	line 151
Javadoc: Parameter jobConf is not declared	JobStatistics.java	MR/src/contrib/vaidya/src/java/org/apache/hadoop/vaidya/statistics/job	line 76
Javadoc: Parameter jobHistoryLogFile is not declared	PostExPerformanceDiagnoser.java	MR/src/contrib/vaidya/src/java/org/apache/hadoop/vaidya/postexdiagnosis	line 94
Javadoc: Parameter jobInProgressListener is not declared	TaskTrackerManager.java	MR/src/java/org/apache/hadoop/mapred	line 49
Javadoc: Parameter jobInProgressListener is not declared	TaskTrackerManager.java	MR/src/java/org/apache/hadoop/mapred	line 56
Javadoc: Parameter recordOffset is not declared	BufferSorter.java	MR/src/java/org/apache/hadoop/mapred	line 47
Javadoc: Parameter specificMgr is not declared	SqlManager.java	MR/src/contrib/sqoop/src/java/org/apache/hadoop/sqoop/manager	line 57
Javadoc: Parameter tableClassName is not declared	ClassLoaderStack.java	MR/src/contrib/sqoop/src/java/org/apache/hadoop/sqoop/util	line 57
Javadoc: Parameter testsConfFile is not declared	PostExPerformanceDiagnoser.java	MR/src/contrib/vaidya/src/java/org/apache/hadoop/vaidya/postexdiagnosis	line 95
Javadoc: RM_CONF_FILE cannot be resolved to a type	CapacitySchedulerConf.java	MR/src/contrib/capacity-scheduler/src/java/org/apache/hadoop/mapred	line 109
Javadoc: The method assignTasks(TaskTracker) in the type TaskScheduler is not applicable for the arguments (TaskTrackerStatus)	TestTTMemoryReporting.java	MR/src/test/mapred/org/apache/hadoop/mapred	line 41
Javadoc: The method equals(Object) in the type Pair is not applicable for the arguments (Pair)	TestPair.java	MR/src/contrib/mrunit/src/test/org/apache/hadoop/mrunit/types	line 34
Javadoc: The method getNumberOfThreads(JobContext) in the type MultithreadedMapper is not applicable for the arguments (Configuration)	MultithreadedMapper.java	MR/src/java/org/apache/hadoop/mapreduce/lib/map	line 51
Javadoc: The method map(Object, Object, Mapper.Context) in the type Mapper is not applicable for the arguments (Object, Object, Mapper<KEYIN,VALUEIN,KEYOUT,VALUEOUT>.Context)	Mapper.java	MR/src/java/org/apache/hadoop/mapreduce	line 42
Javadoc: The method obtainNewMapTask(TaskTrackerStatus, int, int) in the type JobInProgress is not applicable for the arguments (TaskTrackerStatus, int)	DefaultTaskSelector.java	MR/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred	line 25
Javadoc: The method obtainNewReduceTask(TaskTrackerStatus, int, int) in the type JobInProgress is not applicable for the arguments (TaskTrackerStatus, int)	DefaultTaskSelector.java	MR/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred	line 26
Javadoc: The method reduce(Object, Iterable, Reducer.Context) in the type Reducer is not applicable for the arguments (Object, Iterable, Reducer<KEYIN,VALUEIN,KEYOUT,VALUEOUT>.Context)	Reducer.java	MR/src/java/org/apache/hadoop/mapreduce	line 86
Javadoc: The method setMapperClass(Job, Class<? extends Mapper<K1,V1,K2,V2>>) in the type MultithreadedMapper is not applicable for the arguments (Configuration, Class)	MultithreadedMapper.java	MR/src/java/org/apache/hadoop/mapreduce/lib/map	line 49
Javadoc: The method setSchedulingInfo(String, Object) is undefined for the type QueueManager	QueueManager.java	MR/src/java/org/apache/hadoop/mapred	line 245
Javadoc: The method write(KEYOUT, VALUEOUT) in the type TaskInputOutputContext<KEYIN,VALUEIN,KEYOUT,VALUEOUT> is not applicable for the arguments (Object, Object)	Reducer.java	MR/src/java/org/apache/hadoop/mapreduce	line 91
Javadoc: Unexpected tag	TestDriver.java	MR/src/contrib/mrunit/src/java/org/apache/hadoop/mrunit	line 63
Javadoc: Unexpected tag	TestDriver.java	MR/src/contrib/mrunit/src/java/org/apache/hadoop/mrunit	line 126
Javadoc: Unexpected tag	MultithreadedMapRunner.java	MR/src/java/org/apache/hadoop/mapred/lib	line 40
Javadoc: Unexpected tag	ValueAggregatorReducer.java	MR/src/java/org/apache/hadoop/mapred/lib/aggregate	line 49
Javadoc: Unexpected tag	MultithreadedMapper.java	MR/src/java/org/apache/hadoop/mapreduce/lib/map	line 43
Javadoc: Unexpected tag	TestQueueAclsForCurrentUser.java	MR/src/test/mapred/org/apache/hadoop/mapred	line 58
Javadoc: WordCount.MapClass cannot be resolved to a type	MultiFileWordCount.java	MR/src/examples/org/apache/hadoop/examples	line 202

{noformat}

> MapReduce source code has significant number of JavaDocs errors and warnings
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-6212
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6212
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Konstantin Boudnik
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1017-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 04:25:57 2009
Return-Path: <common-issues-return-1017-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92481 invoked from network); 26 Aug 2009 04:25:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 04:25:57 -0000
Received: (qmail 40660 invoked by uid 500); 26 Aug 2009 04:26:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40603 invoked by uid 500); 26 Aug 2009 04:26:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40592 invoked by uid 99); 26 Aug 2009 04:26:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 04:26:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 04:26:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4507B234C04C
	for <common-issues@hadoop.apache.org>; Tue, 25 Aug 2009 21:25:59 -0700 (PDT)
Message-ID: <449102482.1251260759267.JavaMail.jira@brutus>
Date: Tue, 25 Aug 2009 21:25:59 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6205) Project java code has a significant
 number of declaration javadoc warnings and errors
In-Reply-To: <2108675829.1250797574846.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747765#action_12747765 ] 

Konstantin Boudnik commented on HADOOP-6205:
--------------------------------------------

Last comment contains wrong list of errors - they belong to MapReduce project and have to be included to MAPREDUCE-6212.

> Project java code has a significant number of declaration javadoc warnings and errors
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6205
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6205
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: Eclipse-JavaDoc-errors.png
>
>
> The issue here is that code inspections (say by IntelliJ IDEA) show that a number of classes have javadoc problems in declarations. Some of those are minor, e.g. missing @return, @param, @throws tags.
> Some of them are more severe, e.g. using non-existing parameter names in a method javadoc (see HDFS' BlockManager.removeFromInvalidates(..) for an example)
> I'd like to add more sophisticated code analysis to be able to fix at least javadoc declaration errors in order to produce better code quality.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1018-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 04:40:00 2009
Return-Path: <common-issues-return-1018-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95873 invoked from network); 26 Aug 2009 04:39:59 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 04:39:59 -0000
Received: (qmail 56700 invoked by uid 500); 26 Aug 2009 04:40:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56639 invoked by uid 500); 26 Aug 2009 04:40:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56628 invoked by uid 99); 26 Aug 2009 04:40:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 04:40:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 04:40:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 40AC1234C04C
	for <common-issues@hadoop.apache.org>; Tue, 25 Aug 2009 21:39:59 -0700 (PDT)
Message-ID: <349573131.1251261599250.JavaMail.jira@brutus>
Date: Tue, 25 Aug 2009 21:39:59 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6205) Project java code has a significant
 number of declaration javadoc warnings and errors
In-Reply-To: <2108675829.1250797574846.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747771#action_12747771 ] 

Konstantin Boudnik commented on HADOOP-6205:
--------------------------------------------

Corrent list of JavaDoc errors for the project Common
{noformat}
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	FileMetadata.java	Common/src/java/org/apache/hadoop/fs/s3native	line 23
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	PartialListing.java	Common/src/java/org/apache/hadoop/fs/s3native	line 24
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	PartialListing.java	Common/src/java/org/apache/hadoop/fs/s3native	line 25
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	PartialListing.java	Common/src/java/org/apache/hadoop/fs/s3native	line 32
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	ConnectionHeader.java	Common/src/java/org/apache/hadoop/ipc	line 44
Javadoc: 'public' visibility for malformed doc comments hides this 'default' reference	InMemoryNativeFileSystemStore.java	Common/src/test/core/org/apache/hadoop/fs/s3native	line 45
Javadoc: 'public' visibility for malformed doc comments hides this 'private' reference	MetricsTimeVaryingInt.java	Common/src/java/org/apache/hadoop/metrics/util	line 98
Javadoc: 'public' visibility for malformed doc comments hides this 'private' reference	MetricsTimeVaryingLong.java	Common/src/java/org/apache/hadoop/metrics/util	line 94
Javadoc: 'public' visibility for malformed doc comments hides this 'protected' reference	FileSystemContractBaseTest.java	Common/src/test/core/org/apache/hadoop/fs	line 41
Javadoc: DataInput cannot be resolved to a type	VersionMismatchException.java	Common/src/java/org/apache/hadoop/io	line 23
Javadoc: Exception IOException is not declared	LocalDirAllocator.java	Common/src/java/org/apache/hadoop/fs	line 177
Javadoc: Exception IOException is not declared	LineReader.java	Common/src/java/org/apache/hadoop/util	line 47
Javadoc: Exception IOException is not declared	LineReader.java	Common/src/java/org/apache/hadoop/util	line 58
Javadoc: Invalid param tag name	FsShell.java	Common/src/java/org/apache/hadoop/fs	line 153
Javadoc: Invalid param tag name	FsShell.java	Common/src/java/org/apache/hadoop/fs	line 154
Javadoc: Invalid param tag name	FsShell.java	Common/src/java/org/apache/hadoop/fs	line 286
Javadoc: Invalid param tag name	FsShell.java	Common/src/java/org/apache/hadoop/fs	line 287
Javadoc: Invalid param tag name	FsShell.java	Common/src/java/org/apache/hadoop/fs	line 303
Javadoc: Invalid param tag name	FsShell.java	Common/src/java/org/apache/hadoop/fs	line 304
Javadoc: Invalid param tag name	FsShell.java	Common/src/java/org/apache/hadoop/fs	line 305
Javadoc: Invalid param tag name	FsShell.java	Common/src/java/org/apache/hadoop/fs	line 336
Javadoc: Invalid param tag name	FsShell.java	Common/src/java/org/apache/hadoop/fs	line 553
Javadoc: Invalid param tag name	FsShell.java	Common/src/java/org/apache/hadoop/fs	line 554
Javadoc: Invalid param tag name	MapFile.java	Common/src/java/org/apache/hadoop/io	line 444
Javadoc: Invalid param tag name	MapFile.java	Common/src/java/org/apache/hadoop/io	line 569
Javadoc: Invalid param tag name	SerializationTestUtil.java	Common/src/test/core/org/apache/hadoop/io/serializer	line 32
Javadoc: Invalid parameters declaration	CreateFlag.java	Common/src/java/org/apache/hadoop/fs	line 25
Javadoc: Invalid tag	FsShell.java	Common/src/java/org/apache/hadoop/fs	line 155
Javadoc: Invalid tag	FsShell.java	Common/src/java/org/apache/hadoop/fs	line 288
Javadoc: Invalid tag	FsShell.java	Common/src/java/org/apache/hadoop/fs	line 306
Javadoc: Invalid tag	FsShell.java	Common/src/java/org/apache/hadoop/fs	line 337
Javadoc: Invalid tag	FsShell.java	Common/src/java/org/apache/hadoop/fs	line 946
Javadoc: Invalid tag	FsShell.java	Common/src/java/org/apache/hadoop/fs	line 1031
Javadoc: LocalDirAllocator.SIZE_UNKNOWN cannot be resolved to a type	LocalDirAllocator.java	Common/src/java/org/apache/hadoop/fs	line 352
Javadoc: MetricsException cannot be resolved to a type	MetricsRecordImpl.java	Common/src/java/org/apache/hadoop/metrics/spi	line 63
Javadoc: MetricsException cannot be resolved to a type	MetricsRecordImpl.java	Common/src/java/org/apache/hadoop/metrics/spi	line 77
Javadoc: MetricsException cannot be resolved to a type	MetricsRecordImpl.java	Common/src/java/org/apache/hadoop/metrics/spi	line 88
Javadoc: MetricsException cannot be resolved to a type	MetricsRecordImpl.java	Common/src/java/org/apache/hadoop/metrics/spi	line 99
Javadoc: MetricsException cannot be resolved to a type	MetricsRecordImpl.java	Common/src/java/org/apache/hadoop/metrics/spi	line 110
Javadoc: MetricsException cannot be resolved to a type	MetricsRecordImpl.java	Common/src/java/org/apache/hadoop/metrics/spi	line 128
Javadoc: MetricsException cannot be resolved to a type	MetricsRecordImpl.java	Common/src/java/org/apache/hadoop/metrics/spi	line 140
Javadoc: MetricsException cannot be resolved to a type	MetricsRecordImpl.java	Common/src/java/org/apache/hadoop/metrics/spi	line 152
Javadoc: MetricsException cannot be resolved to a type	MetricsRecordImpl.java	Common/src/java/org/apache/hadoop/metrics/spi	line 164
Javadoc: MetricsException cannot be resolved to a type	MetricsRecordImpl.java	Common/src/java/org/apache/hadoop/metrics/spi	line 176
Javadoc: MetricsException cannot be resolved to a type	MetricsRecordImpl.java	Common/src/java/org/apache/hadoop/metrics/spi	line 188
Javadoc: MetricsException cannot be resolved to a type	MetricsRecordImpl.java	Common/src/java/org/apache/hadoop/metrics/spi	line 200
Javadoc: MetricsException cannot be resolved to a type	MetricsRecordImpl.java	Common/src/java/org/apache/hadoop/metrics/spi	line 212
Javadoc: MetricsException cannot be resolved to a type	MetricsRecordImpl.java	Common/src/java/org/apache/hadoop/metrics/spi	line 224
Javadoc: MetricsException cannot be resolved to a type	MetricsRecordImpl.java	Common/src/java/org/apache/hadoop/metrics/spi	line 236
Javadoc: NetUtils cannot be resolved or is not a field	NetUtils.java	Common/src/java/org/apache/hadoop/net	line 248
Javadoc: NetUtils cannot be resolved or is not a field	NetUtils.java	Common/src/java/org/apache/hadoop/net	line 270
Javadoc: NetUtils cannot be resolved or is not a field	NetUtils.java	Common/src/java/org/apache/hadoop/net	line 299
Javadoc: NetUtils cannot be resolved or is not a field	NetUtils.java	Common/src/java/org/apache/hadoop/net	line 321
Javadoc: Unexpected tag	NetUtils.java	Common/src/java/org/apache/hadoop/net	line 415
Javadoc: Unexpected tag	TestLocalDirAllocator.java	Common/src/test/core/org/apache/hadoop/fs	line 182
{noformat}

> Project java code has a significant number of declaration javadoc warnings and errors
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6205
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6205
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: Eclipse-JavaDoc-errors.png
>
>
> The issue here is that code inspections (say by IntelliJ IDEA) show that a number of classes have javadoc problems in declarations. Some of those are minor, e.g. missing @return, @param, @throws tags.
> Some of them are more severe, e.g. using non-existing parameter names in a method javadoc (see HDFS' BlockManager.removeFromInvalidates(..) for an example)
> I'd like to add more sophisticated code analysis to be able to fix at least javadoc declaration errors in order to produce better code quality.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1019-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 05:25:23 2009
Return-Path: <common-issues-return-1019-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8476 invoked from network); 26 Aug 2009 05:25:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 05:25:23 -0000
Received: (qmail 89581 invoked by uid 500); 26 Aug 2009 05:25:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89523 invoked by uid 500); 26 Aug 2009 05:25:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89467 invoked by uid 99); 26 Aug 2009 05:25:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 05:25:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 05:25:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 6E984234C1F0
	for <common-issues@hadoop.apache.org>; Tue, 25 Aug 2009 22:24:59 -0700 (PDT)
Message-ID: <1343172920.1251264299452.JavaMail.jira@brutus>
Date: Tue, 25 Aug 2009 22:24:59 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747784#action_12747784 ] 

Hudson commented on HADOOP-6166:
--------------------------------

Integrated in TestBuilds #6 (See [http://hudson.zones.apache.org/hudson/job/TestBuilds/6/])
    . Further improve the performance of the pure-Java CRC32
implementation. Contributed by Tsz Wo (Nicholas), SZE


> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.21.0
>
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt, c6166_20090727.patch, c6166_20090728.patch, c6166_20090810.patch, c6166_20090811.patch, c6166_20090819.patch, c6166_20090819review.patch, graph.r, graph.r, Rplots-laptop.pdf, Rplots-nehalem32.pdf, Rplots-nehalem64.pdf, Rplots.pdf, Rplots.pdf, Rplots.pdf
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1020-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 06:35:24 2009
Return-Path: <common-issues-return-1020-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50268 invoked from network); 26 Aug 2009 06:35:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 06:35:24 -0000
Received: (qmail 69755 invoked by uid 500); 26 Aug 2009 06:35:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69704 invoked by uid 500); 26 Aug 2009 06:35:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69685 invoked by uid 99); 26 Aug 2009 06:35:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 06:35:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 06:35:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 53F98234C1E7
	for <common-issues@hadoop.apache.org>; Tue, 25 Aug 2009 23:34:59 -0700 (PDT)
Message-ID: <2079571856.1251268499342.JavaMail.jira@brutus>
Date: Tue, 25 Aug 2009 23:34:59 -0700 (PDT)
From: "Amar Kamat (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6213) Remove commons dependency on
 commons-cli2
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Remove commons dependency on commons-cli2
-----------------------------------------

                 Key: HADOOP-6213
                 URL: https://issues.apache.org/jira/browse/HADOOP-6213
             Project: Hadoop Common
          Issue Type: Sub-task
          Components: util
            Reporter: Amar Kamat


GenericOptionsParser depends on commons-cli2

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1021-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 06:41:21 2009
Return-Path: <common-issues-return-1021-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52306 invoked from network); 26 Aug 2009 06:41:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 06:41:21 -0000
Received: (qmail 77314 invoked by uid 500); 26 Aug 2009 06:41:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77248 invoked by uid 500); 26 Aug 2009 06:41:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77237 invoked by uid 99); 26 Aug 2009 06:41:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 06:41:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 06:41:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 5E37A234C48C
	for <common-issues@hadoop.apache.org>; Tue, 25 Aug 2009 23:40:59 -0700 (PDT)
Message-ID: <1053346140.1251268859385.JavaMail.jira@brutus>
Date: Tue, 25 Aug 2009 23:40:59 -0700 (PDT)
From: "Amar Kamat (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6213) Remove commons dependency on
 commons-cli2
In-Reply-To: <2079571856.1251268499342.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amar Kamat updated HADOOP-6213:
-------------------------------

    Affects Version/s: 0.20.0
        Fix Version/s: 0.20.0

> Remove commons dependency on commons-cli2
> -----------------------------------------
>
>                 Key: HADOOP-6213
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6213
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: util
>    Affects Versions: 0.20.0
>            Reporter: Amar Kamat
>             Fix For: 0.20.0
>
>
> GenericOptionsParser depends on commons-cli2

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1022-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 07:40:23 2009
Return-Path: <common-issues-return-1022-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68988 invoked from network); 26 Aug 2009 07:40:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 07:40:23 -0000
Received: (qmail 35733 invoked by uid 500); 26 Aug 2009 07:40:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35671 invoked by uid 500); 26 Aug 2009 07:40:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35659 invoked by uid 99); 26 Aug 2009 07:40:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 07:40:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 07:40:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4C55F234C04C
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 00:39:59 -0700 (PDT)
Message-ID: <952355659.1251272399298.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 00:39:59 -0700 (PDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Moved: (HADOOP-6214) HDFS Web UI displays comments from
 dfs.exclude file and counts them as dead nodes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari moved HDFS-1 to HADOOP-6214:
-----------------------------------------

        Key: HADOOP-6214  (was: HDFS-1)
    Project: Hadoop Common  (was: Hadoop HDFS)

> HDFS Web UI displays comments from dfs.exclude file and counts them as dead nodes
> ---------------------------------------------------------------------------------
>
>                 Key: HADOOP-6214
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6214
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jim Huang
>            Assignee: Ravi Phulari
>         Attachments: HDFS-1.patch
>
>
> I am putting comments in dfs.exclude file such as:
> {noformat}
>  # 32 GB memory upgrades
> {noformat}
> HDFS Web UI counts each word in the commented line as a dead node.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1023-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 08:16:23 2009
Return-Path: <common-issues-return-1023-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82064 invoked from network); 26 Aug 2009 08:16:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 08:16:23 -0000
Received: (qmail 86015 invoked by uid 500); 26 Aug 2009 08:16:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85937 invoked by uid 500); 26 Aug 2009 08:16:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85889 invoked by uid 99); 26 Aug 2009 08:16:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 08:16:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 08:16:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 5A8D6234C1F0
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 01:15:59 -0700 (PDT)
Message-ID: <1380300348.1251274559370.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 01:15:59 -0700 (PDT)
From: "Amar Kamat (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6213) Remove commons dependency on
 commons-cli2
In-Reply-To: <2079571856.1251268499342.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amar Kamat reassigned HADOOP-6213:
----------------------------------

    Assignee: Amar Kamat

> Remove commons dependency on commons-cli2
> -----------------------------------------
>
>                 Key: HADOOP-6213
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6213
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: util
>    Affects Versions: 0.20.0
>            Reporter: Amar Kamat
>            Assignee: Amar Kamat
>             Fix For: 0.20.0
>
>
> GenericOptionsParser depends on commons-cli2

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1024-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 08:30:21 2009
Return-Path: <common-issues-return-1024-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85294 invoked from network); 26 Aug 2009 08:30:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 08:30:21 -0000
Received: (qmail 4228 invoked by uid 500); 26 Aug 2009 08:30:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4168 invoked by uid 500); 26 Aug 2009 08:30:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4157 invoked by uid 99); 26 Aug 2009 08:30:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 08:30:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 08:30:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 44FD3234C04C
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 01:29:59 -0700 (PDT)
Message-ID: <723212905.1251275399266.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 01:29:59 -0700 (PDT)
From: "Amar Kamat (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6213) Remove commons dependency on
 commons-cli2
In-Reply-To: <2079571856.1251268499342.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amar Kamat updated HADOOP-6213:
-------------------------------

    Priority: Blocker  (was: Major)

> Remove commons dependency on commons-cli2
> -----------------------------------------
>
>                 Key: HADOOP-6213
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6213
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: util
>    Affects Versions: 0.20.0
>            Reporter: Amar Kamat
>            Assignee: Amar Kamat
>            Priority: Blocker
>             Fix For: 0.20.0
>
>
> GenericOptionsParser depends on commons-cli2

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1025-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 08:56:21 2009
Return-Path: <common-issues-return-1025-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90696 invoked from network); 26 Aug 2009 08:56:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 08:56:21 -0000
Received: (qmail 27284 invoked by uid 500); 26 Aug 2009 08:56:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27226 invoked by uid 500); 26 Aug 2009 08:56:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27216 invoked by uid 99); 26 Aug 2009 08:56:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 08:56:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 08:56:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 5C88B234C04C
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 01:55:59 -0700 (PDT)
Message-ID: <1787273122.1251276959357.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 01:55:59 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6208) Block loss in S3FS due to S3
 inconsistency on file rename
In-Reply-To: <548899358.1251220979391.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747842#action_12747842 ] 

Tom White commented on HADOOP-6208:
-----------------------------------

Option 2 seems like a reasonable approach. Since there is a single writer, do we only need to poll for consistency on file close?

Also, do we need to make this configurable? Always doing the polling seems acceptable since it guarantees correctness (at least for a given client).


> Block loss in S3FS due to S3 inconsistency on file rename
> ---------------------------------------------------------
>
>                 Key: HADOOP-6208
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6208
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 0.20.0
>         Environment: Ubuntu Linux 8.04 on EC2, Mac OS X 10.5, likely to affect any Hadoop environment
>            Reporter: Bradley Buda
>         Attachments: S3FSConsistencyTest.java
>
>
> Under certain S3 consistency scenarios, Hadoop's S3FileSystem can 'truncate' files, especially when writing reduce outputs.  We've noticed this at tracksimple where we use the S3FS as the direct input and output of our MapReduce jobs.  The symptom of this problem is a file in the filesystem that is an exact multiple of the FS block size - exactly 32MB, 64MB, 96MB, etc. in length.
> The issue appears to be caused by renaming a file that has recently been written, and getting a stale INode read from S3.  When a reducer is writing job output to the S3FS, the normal series of S3 key writes for a 3-block file looks something like this:
> Task Output:
> 1) Write the first block (block_99)
> 2) Write an INode (/myjob/_temporary/_attempt_200907142159_0306_r_000133_0/part-00133.gz) containing [block_99]
> 3) Write the second block (block_81)
> 4) Rewrite the INode with new contents [block_99, block_81]
> 5) Write the last block (block_-101)
> 6) Rewrite the INode with the final contents [block_99, block_81, block_-101]
> Copy Output to Final Location (ReduceTask#copyOutput):
> 1) Read the INode contents from /myjob/_temporary/_attempt_200907142159_0306_r_000133_0/part-00133.gz, which gives [block_99, block_81, block_-101]
> 2) Write the data from #1 to the final location, /myjob/part-00133.gz
> 3) Delete the old INode 
> The output file is truncated if S3 serves a stale copy of the temporary INode.  In copyOutput, step 1 above, it is possible for S3 to return a version of the temporary INode that contains just [block_99, block_81].  In this case, we write this new data to the final output location, and 'lose' block_-101 in the process.  Since we then delete the temporary INode, we've lost all references to the final block of this file and it's orphaned in the S3 bucket.
> This type of consistency error is infrequent but not impossible. We've observed these failures about once a week for one of our large jobs which runs daily and has 200 reduce outputs; so we're seeing an error rate of something like 0.07% per reduce.
> These kind of errors are generally difficult to handle in a system like S3.  We have a few ideas about how to fix this:
> 1) HACK! Sleep during S3OutputStream#close or #flush to wait for S3 to catch up and make these less likely.
> 2) Poll for updated MD5 or INode data in Jets3tFileSystemStore#storeINode until S3 says the INode contents are the same as our local copy.  This could be a config option - "fs.s3.verifyInodeWrites" or something like that.
> 3) Cache INode contents in-process, so we don't have to go back to S3 to ask for the current version of an INode.
> 4) Only write INodes once, when the output stream is closed.  This would basically make S3OutputStream#flush() a no-op.
> 5) Modify the S3FS to somehow version INodes (unclear how we would do this, need some design work).
> 6) Avoid using the S3FS for temporary task attempt files.
> 7) Avoid using the S3FS completely.
> We wanted to get some guidance from the community before we went down any of these paths.  Has anyone seen this issue?  Any other suggested workarounds?  We at tracksimple are willing to invest some time in fixing this and (of course) contributing our fix back, but we wanted to get an 'ack' from others before we try anything crazy :-).
> I've attached a test app if anyone wants to try and reproduce this themselves.  It takes a while to run (depending on the 'weather' in S3 right now), but should eventually detect a consistency 'error' that manifests itself as a truncated file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1026-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 09:14:23 2009
Return-Path: <common-issues-return-1026-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95493 invoked from network); 26 Aug 2009 09:14:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 09:14:23 -0000
Received: (qmail 45085 invoked by uid 500); 26 Aug 2009 09:14:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45041 invoked by uid 500); 26 Aug 2009 09:14:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45031 invoked by uid 99); 26 Aug 2009 09:14:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 09:14:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 09:14:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 59E89234C1EE
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 02:13:59 -0700 (PDT)
Message-ID: <1886586965.1251278039367.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 02:13:59 -0700 (PDT)
From: "Amar Kamat (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6213) Remove commons dependency on
 commons-cli2
In-Reply-To: <2079571856.1251268499342.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amar Kamat updated HADOOP-6213:
-------------------------------

    Attachment: HADOOP-6213-v1.0.patch

Attaching a patch that removes the dependency on GenericOptionsParser on commons-cli2. 
[exec] -1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
     [exec]                         Please justify why no tests are needed for this patch.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     -1 Eclipse classpath. The patch causes the Eclipse classpath to differ from the contents of the lib directories.

Eclipse classpath check fails because commons-cli2 has to be manually deleted.

Running ant tests.

> Remove commons dependency on commons-cli2
> -----------------------------------------
>
>                 Key: HADOOP-6213
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6213
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: util
>    Affects Versions: 0.20.0
>            Reporter: Amar Kamat
>            Assignee: Amar Kamat
>            Priority: Blocker
>             Fix For: 0.20.0
>
>         Attachments: HADOOP-6213-v1.0.patch
>
>
> GenericOptionsParser depends on commons-cli2

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1027-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 11:32:26 2009
Return-Path: <common-issues-return-1027-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52099 invoked from network); 26 Aug 2009 11:32:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 11:32:26 -0000
Received: (qmail 21946 invoked by uid 500); 26 Aug 2009 11:32:26 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21887 invoked by uid 500); 26 Aug 2009 11:32:26 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21877 invoked by uid 99); 26 Aug 2009 11:32:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 11:32:26 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 11:32:23 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id B3F47234C04B
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 04:32:02 -0700 (PDT)
Message-ID: <1850357457.1251286322735.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 04:32:02 -0700 (PDT)
From: "Amar Kamat (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6213) Remove commons dependency on
 commons-cli2
In-Reply-To: <2079571856.1251268499342.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747892#action_12747892 ] 

Amar Kamat commented on HADOOP-6213:
------------------------------------

All test except  TestDistributedFileSystem and TestReduceFetch passed. Testing contrib now.

> Remove commons dependency on commons-cli2
> -----------------------------------------
>
>                 Key: HADOOP-6213
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6213
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: util
>    Affects Versions: 0.20.0
>            Reporter: Amar Kamat
>            Assignee: Amar Kamat
>            Priority: Blocker
>             Fix For: 0.20.0
>
>         Attachments: HADOOP-6213-v1.0.patch
>
>
> GenericOptionsParser depends on commons-cli2

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1028-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 12:04:23 2009
Return-Path: <common-issues-return-1028-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60908 invoked from network); 26 Aug 2009 12:04:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 12:04:23 -0000
Received: (qmail 62924 invoked by uid 500); 26 Aug 2009 12:04:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62868 invoked by uid 500); 26 Aug 2009 12:04:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62858 invoked by uid 99); 26 Aug 2009 12:04:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 12:04:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 12:04:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 6C035234C1F0
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 05:03:59 -0700 (PDT)
Message-ID: <235316153.1251288239441.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 05:03:59 -0700 (PDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-2681) NullPointerException in
 TaskRunner.java when system property "hadoop.log.dir" is not set
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747906#action_12747906 ] 

Steve Loughran commented on HADOOP-2681:
----------------------------------------

This problem still exists in 0.21 code,  not at line 401

    // Setup the log4j prop
    vargs.add("-Dhadoop.log.dir=" + 
        new File(System.getProperty("hadoop.log.dir")
        ).getAbsolutePath());

The value of hadoop.log.dir should be driven by the configuration, not a system property. That system property could be an default, but it should be sufficient for a value in the configuration to be defined.

> NullPointerException in TaskRunner.java when system property "hadoop.log.dir" is not set
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-2681
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2681
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.15.2
>            Reporter: Xu Zhang
>            Priority: Minor
>
> Currently, NullPointerException exception is thrown on line 321 in TaskRunner.java when system property "hadoop.log.dir" is not set.  Instead of a NullPointerException exception, I expected a default value for "hadoop.log.dir" to be used, or to see a more meaningful error message that could have helped me figure out what was wrong (like, telling me that I needed to set "hadoop.log.dir" and how to do so).
> Here is one instance of such exceptions:
> WARN mapred.TaskRunner: task_200801181719_0001_m_000000_0 Child Error
> java.lang.NullPointerException
> 	at java.io.File.<init>(File.java:222)
> 	at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:321)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1029-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 12:22:23 2009
Return-Path: <common-issues-return-1029-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72616 invoked from network); 26 Aug 2009 12:22:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 12:22:23 -0000
Received: (qmail 79199 invoked by uid 500); 26 Aug 2009 12:22:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79158 invoked by uid 500); 26 Aug 2009 12:22:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79148 invoked by uid 99); 26 Aug 2009 12:22:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 12:22:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 12:22:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 412B8234C04B
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 05:21:59 -0700 (PDT)
Message-ID: <2133206750.1251289319262.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 05:21:59 -0700 (PDT)
From: "Amar Kamat (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6213) Remove commons dependency on
 commons-cli2
In-Reply-To: <2079571856.1251268499342.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747910#action_12747910 ] 

Amar Kamat commented on HADOOP-6213:
------------------------------------

TestStreamingWithExitStatus, TestJobInitialization (timeout) and TestQueueCapacities (timeout) failed in contrib.

> Remove commons dependency on commons-cli2
> -----------------------------------------
>
>                 Key: HADOOP-6213
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6213
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: util
>    Affects Versions: 0.20.0
>            Reporter: Amar Kamat
>            Assignee: Amar Kamat
>            Priority: Blocker
>             Fix For: 0.20.0
>
>         Attachments: HADOOP-6213-v1.0.patch
>
>
> GenericOptionsParser depends on commons-cli2

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1030-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 12:40:21 2009
Return-Path: <common-issues-return-1030-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78208 invoked from network); 26 Aug 2009 12:40:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 12:40:21 -0000
Received: (qmail 6475 invoked by uid 500); 26 Aug 2009 12:40:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6413 invoked by uid 500); 26 Aug 2009 12:40:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6403 invoked by uid 99); 26 Aug 2009 12:40:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 12:40:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 12:40:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 6C1E7234C044
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 05:39:59 -0700 (PDT)
Message-ID: <2115814964.1251290399427.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 05:39:59 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6166) Improve PureJavaCrc32
In-Reply-To: <1460486577.1248313274837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747916#action_12747916 ] 

Hudson commented on HADOOP-6166:
--------------------------------

Integrated in Hadoop-Common-trunk #70 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/70/])
    . Further improve the performance of the pure-Java CRC32
implementation. Contributed by Tsz Wo (Nicholas), SZE


> Improve PureJavaCrc32
> ---------------------
>
>                 Key: HADOOP-6166
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6166
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>             Fix For: 0.21.0
>
>         Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt, c6166_20090722_benchmark_64VM.txt, c6166_20090727.patch, c6166_20090728.patch, c6166_20090810.patch, c6166_20090811.patch, c6166_20090819.patch, c6166_20090819review.patch, graph.r, graph.r, Rplots-laptop.pdf, Rplots-nehalem32.pdf, Rplots-nehalem64.pdf, Rplots.pdf, Rplots.pdf, Rplots.pdf
>
>
> Got some ideas to improve CRC32 calculation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1031-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 12:40:23 2009
Return-Path: <common-issues-return-1031-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78251 invoked from network); 26 Aug 2009 12:40:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 12:40:23 -0000
Received: (qmail 6562 invoked by uid 500); 26 Aug 2009 12:40:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6501 invoked by uid 500); 26 Aug 2009 12:40:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6491 invoked by uid 99); 26 Aug 2009 12:40:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 12:40:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 12:40:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C24DB234C1ED
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 05:39:59 -0700 (PDT)
Message-ID: <975656902.1251290399794.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 05:39:59 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6152) Hadoop scripts do not correctly put
 jars on the classpath
In-Reply-To: <283943906.1247689875148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747917#action_12747917 ] 

Hudson commented on HADOOP-6152:
--------------------------------

Integrated in Hadoop-Common-trunk #70 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/70/])
    

> Hadoop scripts do not correctly put jars on the classpath
> ---------------------------------------------------------
>
>                 Key: HADOOP-6152
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6152
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6152.2.patch, HADOOP-6152.3.patch, HADOOP-6152.4.patch, HADOOP-6152.5.patch, HADOOP-6152.patch
>
>
> The various Hadoop scripts (bin/hadoop, bin/hdfs, bin/mapred) do not properly identify the jars needed to run Hadoop. They try to include hadoop-*-hdfs.jar, etc, rather than the hadoop-hdfs-*.jar that is actually created by the 'ant jar' and 'ant package' targets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1032-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 12:46:23 2009
Return-Path: <common-issues-return-1032-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79683 invoked from network); 26 Aug 2009 12:46:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 12:46:23 -0000
Received: (qmail 17437 invoked by uid 500); 26 Aug 2009 12:46:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17378 invoked by uid 500); 26 Aug 2009 12:46:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17367 invoked by uid 99); 26 Aug 2009 12:46:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 12:46:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 12:46:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 479C3234C044
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 05:45:59 -0700 (PDT)
Message-ID: <1392953463.1251290759278.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 05:45:59 -0700 (PDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6209) chmod fails on Windows+cygwin
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747920#action_12747920 ] 

Vinod K V commented on HADOOP-6209:
-----------------------------------

A quick look at DF.java and DU.java shows that they may also result in this same exception. Can you check it by running the DF and DU classes directly?

> chmod fails on Windows+cygwin
> -----------------------------
>
>                 Key: HADOOP-6209
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6209
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Kohsuke Kawaguchi
>            Assignee: Kohsuke Kawaguchi
>         Attachments: HDFS-569.diff
>
>
> When run on Windows + cygwin, chmod fails because it fails to call {{makeShellPath}} on the parameter.
> The strack trace with 0.19.1 below:
> {noformat}
> org.apache.hadoop.util.Shell$ExitCodeException: chmod: changing permissions of `C:\\files\\hudson\\hadoop\\task-tracker\\taskTracker\\jobcache\\job_200908191713_0008\\job.xml': No such file or directory
> 	at org.apache.hadoop.util.Shell.runCommand(Shell.java:195)
> 	at org.apache.hadoop.util.Shell.run(Shell.java:134)
> 	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:286)
> 	at org.apache.hadoop.util.Shell.execCommand(Shell.java:338)
> 	at org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:540)
> 	at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:532)
> 	at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:274)
> 	at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:364)
> 	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:487)
> 	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:468)
> 	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:375)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:208)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)
> 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1214)
> 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1195)
> 	at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:750)
> 	at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1656)
> 	at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:102)
> 	at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1621)
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1033-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 12:54:24 2009
Return-Path: <common-issues-return-1033-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81705 invoked from network); 26 Aug 2009 12:54:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 12:54:24 -0000
Received: (qmail 29535 invoked by uid 500); 26 Aug 2009 12:54:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29474 invoked by uid 500); 26 Aug 2009 12:54:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29259 invoked by uid 99); 26 Aug 2009 12:54:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 12:54:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 12:54:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 53210234C045
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 05:53:59 -0700 (PDT)
Message-ID: <1419603551.1251291239339.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 05:53:59 -0700 (PDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6213) Remove commons dependency on
 commons-cli2
In-Reply-To: <2079571856.1251268499342.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sharad Agarwal resolved HADOOP-6213.
------------------------------------

       Resolution: Fixed
    Fix Version/s:     (was: 0.20.0)
                   0.20.1
     Hadoop Flags: [Reviewed]

I just committed this. Thanks Amar!

> Remove commons dependency on commons-cli2
> -----------------------------------------
>
>                 Key: HADOOP-6213
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6213
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: util
>    Affects Versions: 0.20.0
>            Reporter: Amar Kamat
>            Assignee: Amar Kamat
>            Priority: Blocker
>             Fix For: 0.20.1
>
>         Attachments: HADOOP-6213-v1.0.patch
>
>
> GenericOptionsParser depends on commons-cli2

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1034-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 15:10:25 2009
Return-Path: <common-issues-return-1034-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55970 invoked from network); 26 Aug 2009 15:10:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 15:10:25 -0000
Received: (qmail 37751 invoked by uid 500); 26 Aug 2009 15:10:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37646 invoked by uid 500); 26 Aug 2009 15:10:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37369 invoked by uid 99); 26 Aug 2009 15:10:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 15:10:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 15:10:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 7F850234C1ED
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 08:09:59 -0700 (PDT)
Message-ID: <944007286.1251299399521.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 08:09:59 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Reopened: (HADOOP-6213) Remove commons dependency on
 commons-cli2
In-Reply-To: <2079571856.1251268499342.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley reopened HADOOP-6213:
-----------------------------------


We missed the split("=", 2) here...

> Remove commons dependency on commons-cli2
> -----------------------------------------
>
>                 Key: HADOOP-6213
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6213
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: util
>    Affects Versions: 0.20.0
>            Reporter: Amar Kamat
>            Assignee: Amar Kamat
>            Priority: Blocker
>             Fix For: 0.20.1
>
>         Attachments: HADOOP-6213-v1.0.patch
>
>
> GenericOptionsParser depends on commons-cli2

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1035-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 15:18:23 2009
Return-Path: <common-issues-return-1035-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69950 invoked from network); 26 Aug 2009 15:18:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 15:18:23 -0000
Received: (qmail 60403 invoked by uid 500); 26 Aug 2009 15:18:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60289 invoked by uid 500); 26 Aug 2009 15:18:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60244 invoked by uid 99); 26 Aug 2009 15:18:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 15:18:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 15:18:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 72ECF234C48C
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 08:17:59 -0700 (PDT)
Message-ID: <200246398.1251299879470.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 08:17:59 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6213) Remove commons dependency on
 commons-cli2
In-Reply-To: <2079571856.1251268499342.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley resolved HADOOP-6213.
-----------------------------------

    Resolution: Fixed

Opened a new jira HADOOP-6215 to resolve the issue in trunk and 0.20.

> Remove commons dependency on commons-cli2
> -----------------------------------------
>
>                 Key: HADOOP-6213
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6213
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: util
>    Affects Versions: 0.20.0
>            Reporter: Amar Kamat
>            Assignee: Amar Kamat
>            Priority: Blocker
>             Fix For: 0.20.1
>
>         Attachments: HADOOP-6213-v1.0.patch
>
>
> GenericOptionsParser depends on commons-cli2

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1036-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 15:18:23 2009
Return-Path: <common-issues-return-1036-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69976 invoked from network); 26 Aug 2009 15:18:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 15:18:23 -0000
Received: (qmail 60447 invoked by uid 500); 26 Aug 2009 15:18:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60295 invoked by uid 500); 26 Aug 2009 15:18:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60257 invoked by uid 99); 26 Aug 2009 15:18:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 15:18:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 15:18:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 5A2ED234C045
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 08:17:59 -0700 (PDT)
Message-ID: <2129574511.1251299879368.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 08:17:59 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6215) fix GenericOptionParser to deal with
 -D with '=' in the value
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

fix GenericOptionParser to deal with -D with '=' in the value
-------------------------------------------------------------

                 Key: HADOOP-6215
                 URL: https://issues.apache.org/jira/browse/HADOOP-6215
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Owen O'Malley
            Assignee: Amar Kamat
             Fix For: 0.20.1


Currently the code ignores any -D option that has a "=" in the value. This is clearly wrong.
In particular, -Dx=y=z should work and assign "y=z" to "x". This is a regression from HADOOP-6213.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1037-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 16:18:21 2009
Return-Path: <common-issues-return-1037-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99794 invoked from network); 26 Aug 2009 16:18:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 16:18:21 -0000
Received: (qmail 70765 invoked by uid 500); 26 Aug 2009 16:18:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70720 invoked by uid 500); 26 Aug 2009 16:18:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70706 invoked by uid 99); 26 Aug 2009 16:18:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 16:18:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 16:18:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4EF2E234C04B
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 09:17:59 -0700 (PDT)
Message-ID: <1329566032.1251303479322.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 09:17:59 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6211) Hdfs source code has significant
 number of JavaDocs errors and warnings
In-Reply-To: <1500634751.1251259919382.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6211:
---------------------------------------

    Issue Type: Bug  (was: Sub-task)
        Parent:     (was: HADOOP-6205)

> Hdfs source code has significant number of JavaDocs errors and warnings
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-6211
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6211
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Konstantin Boudnik
>
> By turning some extra source code inspections in Eclipse one can see many JavaDoc warnings and errors in the source code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1038-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 16:20:23 2009
Return-Path: <common-issues-return-1038-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 725 invoked from network); 26 Aug 2009 16:20:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 16:20:23 -0000
Received: (qmail 72452 invoked by uid 500); 26 Aug 2009 16:20:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72409 invoked by uid 500); 26 Aug 2009 16:20:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72399 invoked by uid 99); 26 Aug 2009 16:20:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 16:20:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 16:20:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 5F790234C04B
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 09:19:59 -0700 (PDT)
Message-ID: <343906821.1251303599390.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 09:19:59 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6212) MapReduce source code has significant
 number of JavaDocs errors and warnings
In-Reply-To: <1113513611.1251260639398.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6212:
---------------------------------------

    Issue Type: Bug  (was: Sub-task)
        Parent:     (was: HADOOP-6205)

> MapReduce source code has significant number of JavaDocs errors and warnings
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-6212
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6212
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Konstantin Boudnik
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1039-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 16:42:23 2009
Return-Path: <common-issues-return-1039-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8377 invoked from network); 26 Aug 2009 16:42:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 16:42:23 -0000
Received: (qmail 5418 invoked by uid 500); 26 Aug 2009 16:42:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5367 invoked by uid 500); 26 Aug 2009 16:42:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5357 invoked by uid 99); 26 Aug 2009 16:42:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 16:42:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 16:42:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 7E948234C1EB
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 09:41:59 -0700 (PDT)
Message-ID: <2101140498.1251304919517.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 09:41:59 -0700 (PDT)
From: "Kohsuke Kawaguchi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6209) chmod fails on Windows+cygwin
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748019#action_12748019 ] 

Kohsuke Kawaguchi commented on HADOOP-6209:
-------------------------------------------

HDFS-197 addresses the DU/DF problem.

> chmod fails on Windows+cygwin
> -----------------------------
>
>                 Key: HADOOP-6209
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6209
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Kohsuke Kawaguchi
>            Assignee: Kohsuke Kawaguchi
>         Attachments: HDFS-569.diff
>
>
> When run on Windows + cygwin, chmod fails because it fails to call {{makeShellPath}} on the parameter.
> The strack trace with 0.19.1 below:
> {noformat}
> org.apache.hadoop.util.Shell$ExitCodeException: chmod: changing permissions of `C:\\files\\hudson\\hadoop\\task-tracker\\taskTracker\\jobcache\\job_200908191713_0008\\job.xml': No such file or directory
> 	at org.apache.hadoop.util.Shell.runCommand(Shell.java:195)
> 	at org.apache.hadoop.util.Shell.run(Shell.java:134)
> 	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:286)
> 	at org.apache.hadoop.util.Shell.execCommand(Shell.java:338)
> 	at org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:540)
> 	at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:532)
> 	at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:274)
> 	at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:364)
> 	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:487)
> 	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:468)
> 	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:375)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:208)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)
> 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1214)
> 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1195)
> 	at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:750)
> 	at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1656)
> 	at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:102)
> 	at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1621)
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1040-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 16:54:21 2009
Return-Path: <common-issues-return-1040-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11322 invoked from network); 26 Aug 2009 16:54:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 16:54:21 -0000
Received: (qmail 19395 invoked by uid 500); 26 Aug 2009 16:54:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19363 invoked by uid 500); 26 Aug 2009 16:54:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19353 invoked by uid 99); 26 Aug 2009 16:54:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 16:54:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 16:54:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 5794F234C04B
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 09:53:59 -0700 (PDT)
Message-ID: <586626753.1251305639357.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 09:53:59 -0700 (PDT)
From: "Amar Kamat (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6213) Remove commons dependency on
 commons-cli2
In-Reply-To: <2079571856.1251268499342.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amar Kamat updated HADOOP-6213:
-------------------------------

    Release Note: GenericOptionsParser depends on commons-cli2. This jira removed the dependency of branch 0.20 on commons-cli2.

> Remove commons dependency on commons-cli2
> -----------------------------------------
>
>                 Key: HADOOP-6213
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6213
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: util
>    Affects Versions: 0.20.0
>            Reporter: Amar Kamat
>            Assignee: Amar Kamat
>            Priority: Blocker
>             Fix For: 0.20.1
>
>         Attachments: HADOOP-6213-v1.0.patch
>
>
> GenericOptionsParser depends on commons-cli2

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1041-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 17:18:21 2009
Return-Path: <common-issues-return-1041-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22009 invoked from network); 26 Aug 2009 17:18:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 17:18:21 -0000
Received: (qmail 67786 invoked by uid 500); 26 Aug 2009 17:18:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67720 invoked by uid 500); 26 Aug 2009 17:18:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67710 invoked by uid 99); 26 Aug 2009 17:18:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 17:18:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 17:18:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 5CD15234C04B
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 10:17:59 -0700 (PDT)
Message-ID: <250386492.1251307079379.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 10:17:59 -0700 (PDT)
From: "Amar Kamat (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6213) Remove commons dependency on
 commons-cli2
In-Reply-To: <2079571856.1251268499342.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amar Kamat updated HADOOP-6213:
-------------------------------

    Release Note: GenericOptionsParser in branch 0.20 depends on commons-cli2. This jira removes the dependency of branch 0.20 on commons-cli2 completely. The problem is seen after 'ant binary' where all the library files are copied to '$hadoop-home/lib' which already has commons-cli2.   (was: GenericOptionsParser depends on commons-cli2. This jira removed the dependency of branch 0.20 on commons-cli2.)

> Remove commons dependency on commons-cli2
> -----------------------------------------
>
>                 Key: HADOOP-6213
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6213
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: util
>    Affects Versions: 0.20.0
>            Reporter: Amar Kamat
>            Assignee: Amar Kamat
>            Priority: Blocker
>             Fix For: 0.20.1
>
>         Attachments: HADOOP-6213-v1.0.patch
>
>
> GenericOptionsParser depends on commons-cli2

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1042-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 17:28:21 2009
Return-Path: <common-issues-return-1042-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28125 invoked from network); 26 Aug 2009 17:28:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 17:28:21 -0000
Received: (qmail 84243 invoked by uid 500); 26 Aug 2009 17:28:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84185 invoked by uid 500); 26 Aug 2009 17:28:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84169 invoked by uid 99); 26 Aug 2009 17:28:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 17:28:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 17:28:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 49B29234C044
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 10:27:59 -0700 (PDT)
Message-ID: <497784667.1251307679286.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 10:27:59 -0700 (PDT)
From: "Amar Kamat (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6215) fix GenericOptionParser to deal
 with -D with '=' in the value
In-Reply-To: <2129574511.1251299879368.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748041#action_12748041 ] 

Amar Kamat commented on HADOOP-6215:
------------------------------------

The patch in HADOOP-6213 makes GenericOptionsParser same as in commons-trunk. We can fix both here.

> fix GenericOptionParser to deal with -D with '=' in the value
> -------------------------------------------------------------
>
>                 Key: HADOOP-6215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6215
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Amar Kamat
>             Fix For: 0.20.1
>
>
> Currently the code ignores any -D option that has a "=" in the value. This is clearly wrong.
> In particular, -Dx=y=z should work and assign "y=z" to "x". This is a regression from HADOOP-6213.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1043-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 18:20:21 2009
Return-Path: <common-issues-return-1043-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62574 invoked from network); 26 Aug 2009 18:20:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 18:20:21 -0000
Received: (qmail 49315 invoked by uid 500); 26 Aug 2009 18:20:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49261 invoked by uid 500); 26 Aug 2009 18:20:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49251 invoked by uid 99); 26 Aug 2009 18:20:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 18:20:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 18:20:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 441E9234C044
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 11:19:59 -0700 (PDT)
Message-ID: <1962382589.1251310799265.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 11:19:59 -0700 (PDT)
From: "Amar Kamat (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6215) fix GenericOptionParser to deal with
 -D with '=' in the value
In-Reply-To: <2129574511.1251299879368.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amar Kamat updated HADOOP-6215:
-------------------------------

    Attachment: HADOOP-6215-v1.0-branch-0.20.patch
                HADOOP-6215-v1.0.patch

Attaching a patch for commons and branch 0.20. Testing in progress.

> fix GenericOptionParser to deal with -D with '=' in the value
> -------------------------------------------------------------
>
>                 Key: HADOOP-6215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6215
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Amar Kamat
>             Fix For: 0.20.1
>
>         Attachments: HADOOP-6215-v1.0-branch-0.20.patch, HADOOP-6215-v1.0.patch
>
>
> Currently the code ignores any -D option that has a "=" in the value. This is clearly wrong.
> In particular, -Dx=y=z should work and assign "y=z" to "x". This is a regression from HADOOP-6213.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1044-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 19:26:23 2009
Return-Path: <common-issues-return-1044-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89000 invoked from network); 26 Aug 2009 19:26:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 19:26:23 -0000
Received: (qmail 62597 invoked by uid 500); 26 Aug 2009 19:26:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62532 invoked by uid 500); 26 Aug 2009 19:26:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62514 invoked by uid 99); 26 Aug 2009 19:26:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 19:26:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 19:26:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 6BB5E234C04B
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 12:25:59 -0700 (PDT)
Message-ID: <1296705081.1251314759440.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 12:25:59 -0700 (PDT)
From: "Amar Kamat (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6215) fix GenericOptionParser to deal
 with -D with '=' in the value
In-Reply-To: <2129574511.1251299879368.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748098#action_12748098 ] 

Amar Kamat commented on HADOOP-6215:
------------------------------------

Result of test-patch 
[exec] +1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.

Ant tests passed on my box.

> fix GenericOptionParser to deal with -D with '=' in the value
> -------------------------------------------------------------
>
>                 Key: HADOOP-6215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6215
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Amar Kamat
>             Fix For: 0.20.1
>
>         Attachments: HADOOP-6215-v1.0-branch-0.20.patch, HADOOP-6215-v1.0.patch
>
>
> Currently the code ignores any -D option that has a "=" in the value. This is clearly wrong.
> In particular, -Dx=y=z should work and assign "y=z" to "x". This is a regression from HADOOP-6213.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1045-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 19:28:23 2009
Return-Path: <common-issues-return-1045-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89556 invoked from network); 26 Aug 2009 19:28:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 19:28:23 -0000
Received: (qmail 65951 invoked by uid 500); 26 Aug 2009 19:28:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65886 invoked by uid 500); 26 Aug 2009 19:28:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65876 invoked by uid 99); 26 Aug 2009 19:28:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 19:28:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 19:28:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 5F03A234C04C
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 12:27:59 -0700 (PDT)
Message-ID: <494424578.1251314879388.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 12:27:59 -0700 (PDT)
From: "Amar Kamat (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6215) fix GenericOptionParser to deal
 with -D with '=' in the value
In-Reply-To: <2129574511.1251299879368.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748100#action_12748100 ] 

Amar Kamat commented on HADOOP-6215:
------------------------------------

Testing on branch 20

> fix GenericOptionParser to deal with -D with '=' in the value
> -------------------------------------------------------------
>
>                 Key: HADOOP-6215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6215
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Amar Kamat
>             Fix For: 0.20.1
>
>         Attachments: HADOOP-6215-v1.0-branch-0.20.patch, HADOOP-6215-v1.0.patch
>
>
> Currently the code ignores any -D option that has a "=" in the value. This is clearly wrong.
> In particular, -Dx=y=z should work and assign "y=z" to "x". This is a regression from HADOOP-6213.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1046-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 20:20:21 2009
Return-Path: <common-issues-return-1046-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3310 invoked from network); 26 Aug 2009 20:20:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 20:20:21 -0000
Received: (qmail 48087 invoked by uid 500); 26 Aug 2009 20:20:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48047 invoked by uid 500); 26 Aug 2009 20:20:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48028 invoked by uid 99); 26 Aug 2009 20:20:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 20:20:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 20:20:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 784EE234C498
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 13:19:59 -0700 (PDT)
Message-ID: <1847432007.1251317999491.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 13:19:59 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Moved: (HADOOP-6216) HDFS Web UI displays comments from
 dfs.exclude file and counts them as dead nodes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE moved HDFS-572 to HADOOP-6216:
-----------------------------------------------------

    Component/s:     (was: name-node)
                 util
            Key: HADOOP-6216  (was: HDFS-572)
        Project: Hadoop Common  (was: Hadoop HDFS)

> HDFS Web UI displays comments from dfs.exclude file and counts them as dead nodes
> ---------------------------------------------------------------------------------
>
>                 Key: HADOOP-6216
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6216
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>            Reporter: Jim Huang
>            Assignee: Ravi Phulari
>         Attachments: HDFS-1.patch
>
>
> I am putting comments in dfs.exclude file such as:
> {noformat}
>  # 32 GB memory upgrades
> {noformat}
> HDFS Web UI counts each word in the commented line as a dead node.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1047-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 20:35:04 2009
Return-Path: <common-issues-return-1047-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7379 invoked from network); 26 Aug 2009 20:35:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 20:35:03 -0000
Received: (qmail 63783 invoked by uid 500); 26 Aug 2009 20:35:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63655 invoked by uid 500); 26 Aug 2009 20:35:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63578 invoked by uid 99); 26 Aug 2009 20:35:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 20:35:03 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 20:32:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4DFD4234C045
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 13:31:59 -0700 (PDT)
Message-ID: <13494760.1251318719318.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 13:31:59 -0700 (PDT)
From: "Corinne Chandel (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6217) Hadoop Doc Split: Common Docs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Hadoop Doc Split: Common Docs
-----------------------------

                 Key: HADOOP-6217
                 URL: https://issues.apache.org/jira/browse/HADOOP-6217
             Project: Hadoop Common
          Issue Type: Task
          Components: documentation
    Affects Versions: 0.21.0
            Reporter: Corinne Chandel
            Assignee: Owen O'Malley
            Priority: Minor
             Fix For: 0.21.0


Hadoop Doc Split: Common Docs

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1048-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 20:38:23 2009
Return-Path: <common-issues-return-1048-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8417 invoked from network); 26 Aug 2009 20:38:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 20:38:23 -0000
Received: (qmail 71240 invoked by uid 500); 26 Aug 2009 20:38:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71213 invoked by uid 500); 26 Aug 2009 20:38:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71203 invoked by uid 99); 26 Aug 2009 20:38:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 20:38:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 20:38:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 58AA7234C04B
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 13:37:59 -0700 (PDT)
Message-ID: <626435659.1251319079362.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 13:37:59 -0700 (PDT)
From: "Corinne Chandel (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6217) Hadoop Doc Split: Common Docs
In-Reply-To: <13494760.1251318719318.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Corinne Chandel updated HADOOP-6217:
------------------------------------

    Attachment: Hadoop-Doc-Split.doc
                hadoop-6217-common.patch

(1) hadoop-6217-common.patch - path file

(2) Hadoop-Doc-Split.doc - explains file changes

> Hadoop Doc Split: Common Docs
> -----------------------------
>
>                 Key: HADOOP-6217
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6217
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>    Affects Versions: 0.21.0
>            Reporter: Corinne Chandel
>            Assignee: Owen O'Malley
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6217-common.patch, Hadoop-Doc-Split.doc
>
>
> Hadoop Doc Split: Common Docs

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1049-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 20:42:21 2009
Return-Path: <common-issues-return-1049-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9245 invoked from network); 26 Aug 2009 20:42:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 20:42:21 -0000
Received: (qmail 75384 invoked by uid 500); 26 Aug 2009 20:42:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75315 invoked by uid 500); 26 Aug 2009 20:42:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75305 invoked by uid 99); 26 Aug 2009 20:42:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 20:42:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 20:42:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4F63B234C044
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 13:41:59 -0700 (PDT)
Message-ID: <1369123033.1251319319311.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 13:41:59 -0700 (PDT)
From: "Corinne Chandel (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6217) Hadoop Doc Split: Common Docs
In-Reply-To: <13494760.1251318719318.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Corinne Chandel updated HADOOP-6217:
------------------------------------

    Status: Patch Available  (was: Open)

Apply this patch to: https://svn.apache.org/repos/asf/hadoop/common/trunk

Note: No new test code required; changes to documentation only.

> Hadoop Doc Split: Common Docs
> -----------------------------
>
>                 Key: HADOOP-6217
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6217
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>    Affects Versions: 0.21.0
>            Reporter: Corinne Chandel
>            Assignee: Owen O'Malley
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6217-common.patch, Hadoop-Doc-Split.doc
>
>
> Hadoop Doc Split: Common Docs

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1050-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 20:42:23 2009
Return-Path: <common-issues-return-1050-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9302 invoked from network); 26 Aug 2009 20:42:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 20:42:23 -0000
Received: (qmail 75465 invoked by uid 500); 26 Aug 2009 20:42:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75414 invoked by uid 500); 26 Aug 2009 20:42:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75404 invoked by uid 99); 26 Aug 2009 20:42:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 20:42:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 20:42:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 59E7D234C04C
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 13:41:59 -0700 (PDT)
Message-ID: <620036921.1251319319367.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 13:41:59 -0700 (PDT)
From: "Corinne Chandel (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6217) Hadoop Doc Split: Common Docs
In-Reply-To: <13494760.1251318719318.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Corinne Chandel updated HADOOP-6217:
------------------------------------

    Description: 
Hadoop Doc Split: Common Docs

Please note that I am unable to directly check all of the new links. Some links may break and will need to be updated.

  was:Hadoop Doc Split: Common Docs


> Hadoop Doc Split: Common Docs
> -----------------------------
>
>                 Key: HADOOP-6217
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6217
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>    Affects Versions: 0.21.0
>            Reporter: Corinne Chandel
>            Assignee: Owen O'Malley
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6217-common.patch, Hadoop-Doc-Split.doc
>
>
> Hadoop Doc Split: Common Docs
> Please note that I am unable to directly check all of the new links. Some links may break and will need to be updated.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1051-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 21:17:21 2009
Return-Path: <common-issues-return-1051-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19376 invoked from network); 26 Aug 2009 21:17:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 21:17:21 -0000
Received: (qmail 17251 invoked by uid 500); 26 Aug 2009 21:17:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17230 invoked by uid 500); 26 Aug 2009 21:17:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17220 invoked by uid 99); 26 Aug 2009 21:17:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 21:17:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 21:17:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4C878234C044
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 14:16:59 -0700 (PDT)
Message-ID: <618244718.1251321419298.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 14:16:59 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6216) HDFS Web UI displays comments from
 dfs.exclude file and counts them as dead nodes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748152#action_12748152 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6216:
------------------------------------------------

How about changing line.startsWith("#") to line.trim().startsWith("#")?  Also, could you use our [Coding Style|http://wiki.apache.org/hadoop/CodeReviewChecklist]?

> HDFS Web UI displays comments from dfs.exclude file and counts them as dead nodes
> ---------------------------------------------------------------------------------
>
>                 Key: HADOOP-6216
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6216
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>            Reporter: Jim Huang
>            Assignee: Ravi Phulari
>         Attachments: HDFS-1.patch
>
>
> I am putting comments in dfs.exclude file such as:
> {noformat}
>  # 32 GB memory upgrades
> {noformat}
> HDFS Web UI counts each word in the commented line as a dead node.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1052-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 22:10:08 2009
Return-Path: <common-issues-return-1052-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42156 invoked from network); 26 Aug 2009 22:10:08 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 22:10:08 -0000
Received: (qmail 66549 invoked by uid 500); 26 Aug 2009 22:04:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66527 invoked by uid 500); 26 Aug 2009 22:04:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66512 invoked by uid 99); 26 Aug 2009 22:04:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 22:04:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 22:04:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4F470234C04C
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 15:03:59 -0700 (PDT)
Message-ID: <1085778007.1251324239323.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 15:03:59 -0700 (PDT)
From: "Corinne Chandel (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6217) Hadoop Doc Split: Common Docs
In-Reply-To: <13494760.1251318719318.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748179#action_12748179 ] 

Corinne Chandel commented on HADOOP-6217:
-----------------------------------------

These wiki pages, linked to from the docs, need to be created:

http://wiki.apache.org/hadoop/Common

http://wiki.apache.org/hadoop/Common/FAQ


> Hadoop Doc Split: Common Docs
> -----------------------------
>
>                 Key: HADOOP-6217
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6217
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>    Affects Versions: 0.21.0
>            Reporter: Corinne Chandel
>            Assignee: Owen O'Malley
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6217-common.patch, Hadoop-Doc-Split.doc
>
>
> Hadoop Doc Split: Common Docs
> Please note that I am unable to directly check all of the new links. Some links may break and will need to be updated.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1053-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 22:25:24 2009
Return-Path: <common-issues-return-1053-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50394 invoked from network); 26 Aug 2009 22:25:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 22:25:24 -0000
Received: (qmail 92199 invoked by uid 500); 26 Aug 2009 22:25:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92130 invoked by uid 500); 26 Aug 2009 22:25:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91825 invoked by uid 99); 26 Aug 2009 22:25:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 22:25:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 22:25:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 81B56234C04C
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 15:24:59 -0700 (PDT)
Message-ID: <668717721.1251325499530.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 15:24:59 -0700 (PDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6143) FS shell commands returns incorrect
 exit code  when error occurs
In-Reply-To: <636161787.1247269154845.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6143:
---------------------------------

    Attachment: HADOOP-6143-1.patch

> FS shell commands returns incorrect exit code  when error occurs
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6143
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6143
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Ravi Phulari
>            Assignee: Ravi Phulari
>         Attachments: HADOOP-6143-1.patch, HADOOP-6143.patch
>
>
> HDFS documentation ( http://hadoop.apache.org/core/docs/current/hdfs_shell.html#du )  mentions that 
> {noformat}
> Exit Code:
> Returns 0 on success and -1 on error. 
> {noformat} 
> Current Fs shell behavior is buggy with this agreement.
> {code}
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -ls foo
> ls: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -lsr foo
> lsr: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -du foo
> du: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -dus foo
> dus: Cannot access foo: No such file or directory.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -cp foo f2
> cp: File does not exist: foo
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -copyToLocal foo f2
> copyToLocal: null
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> statepick-lm:Hadoop rphulari$ bin/hadoop fs -copyFromLocal foo f2
> copyFromLocal: File foo does not exist.
> statepick-lm:Hadoop rphulari$ echo $?
> 255
> {code}  
> In all above cases exit code on error should be -1 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1054-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 23:29:21 2009
Return-Path: <common-issues-return-1054-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68688 invoked from network); 26 Aug 2009 23:29:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 23:29:21 -0000
Received: (qmail 48046 invoked by uid 500); 26 Aug 2009 23:29:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47982 invoked by uid 500); 26 Aug 2009 23:29:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47972 invoked by uid 99); 26 Aug 2009 23:29:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 23:29:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 23:29:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 7C721234C498
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 16:28:59 -0700 (PDT)
Message-ID: <790806686.1251329339508.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 16:28:59 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6170) add Avro-based RPC serialization
In-Reply-To: <1566370037.1248818534917.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-6170:
---------------------------------

    Status: Patch Available  (was: Open)

> add Avro-based RPC serialization
> --------------------------------
>
>                 Key: HADOOP-6170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6170
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6170.patch
>
>
> Permit RPC protocols to use Avro to serialize requests and responses, so that protocols may better evolve without breaking compatibility.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1055-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 23:29:23 2009
Return-Path: <common-issues-return-1055-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68761 invoked from network); 26 Aug 2009 23:29:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 23:29:23 -0000
Received: (qmail 48196 invoked by uid 500); 26 Aug 2009 23:29:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48132 invoked by uid 500); 26 Aug 2009 23:29:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48121 invoked by uid 99); 26 Aug 2009 23:29:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 23:29:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 23:29:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 57804234C044
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 16:28:59 -0700 (PDT)
Message-ID: <432475870.1251329339344.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 16:28:59 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6170) add Avro-based RPC serialization
In-Reply-To: <1566370037.1248818534917.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-6170:
---------------------------------

    Attachment: HADOOP-6170.patch

> add Avro-based RPC serialization
> --------------------------------
>
>                 Key: HADOOP-6170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6170
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6170.patch
>
>
> Permit RPC protocols to use Avro to serialize requests and responses, so that protocols may better evolve without breaking compatibility.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1056-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 26 23:57:23 2009
Return-Path: <common-issues-return-1056-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74167 invoked from network); 26 Aug 2009 23:57:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 26 Aug 2009 23:57:23 -0000
Received: (qmail 65375 invoked by uid 500); 26 Aug 2009 23:57:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65325 invoked by uid 500); 26 Aug 2009 23:57:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65306 invoked by uid 99); 26 Aug 2009 23:57:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 23:57:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Aug 2009 23:57:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 8F9F9234C053
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 16:56:59 -0700 (PDT)
Message-ID: <1693021070.1251331019587.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 16:56:59 -0700 (PDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5363) Proxying for multiple HDFS clusters
 of different versions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-5363:
------------------------------

    Attachment: h5363-0.20.patch

attaching a partially backported patch for 0.20 release, which removes pickOneAddress functionality from HFTP client.

> Proxying for multiple HDFS clusters of different versions
> ---------------------------------------------------------
>
>                 Key: HADOOP-5363
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5363
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Kan Zhang
>            Assignee: zhiyong zhang
>             Fix For: 0.21.0
>
>         Attachments: h5363-0.20.patch, HADOOP-5363.patch, HADOOP-5363.patch, HADOOP-5363.patch, HADOOP-5363.patch, ProxyTestPlan.html
>
>
> A single hdfsproxy server should be able to proxy for multiple HDFS clusters, whose Hadoop versions may be different from each other.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1057-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 00:21:21 2009
Return-Path: <common-issues-return-1057-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84770 invoked from network); 27 Aug 2009 00:21:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 00:21:21 -0000
Received: (qmail 81110 invoked by uid 500); 27 Aug 2009 00:21:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81051 invoked by uid 500); 27 Aug 2009 00:21:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81041 invoked by uid 99); 27 Aug 2009 00:21:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 00:21:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 00:21:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 56E94234C04B
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 17:20:59 -0700 (PDT)
Message-ID: <831051817.1251332459355.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 17:20:59 -0700 (PDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5780) Fix slightly confusing log from
 "-metaSave" on NameNode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-5780:
-----------------------------------------

    Attachment: HADOOP-5780.hadoop-0.20.patch

Patch for hadoop-0.20

> Fix slightly confusing log from "-metaSave" on NameNode
> -------------------------------------------------------
>
>                 Key: HADOOP-5780
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5780
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.0
>            Reporter: Raghu Angadi
>            Assignee: Raghu Angadi
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5780.hadoop-0.20.patch, HADOOP-5780.patch, HADOOP-5780.patch, HADOOP-5780.patch
>
>
> After HADOOP-4103, "-metaSave" on NameNode prints "MISSING" in the line for a block that does not have any good replicas left. 
> e.g. : {noformat}
> Metasave: Blocks waiting for replication: 1
> blk_6735500019364591152_0 (replicas: l: 0 d: 0 c: 2 e: 0 MISSING) 74.6.132.200:50010 :  192.168.0.107:50010 :
> {noformat}
> "0 MISSING" in the log might imply to the user that there is no missing block, though it implies "e(xcess replicas) : 0".
> We should fix that. Also as part of this I would like to indicate which datanodes are decommissioned or contain the corrupt replicas.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1058-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 00:31:23 2009
Return-Path: <common-issues-return-1058-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87618 invoked from network); 27 Aug 2009 00:31:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 00:31:23 -0000
Received: (qmail 89783 invoked by uid 500); 27 Aug 2009 00:31:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89725 invoked by uid 500); 27 Aug 2009 00:31:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89715 invoked by uid 99); 27 Aug 2009 00:31:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 00:31:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 00:31:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4BCFA234C044
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 17:30:59 -0700 (PDT)
Message-ID: <1221497007.1251333059295.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 17:30:59 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6217) Hadoop Doc Split: Common Docs
In-Reply-To: <13494760.1251318719318.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748219#action_12748219 ] 

Hadoop QA commented on HADOOP-6217:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12417788/Hadoop-Doc-Split.doc
  against trunk revision 807753.

    +1 @author.  The patch does not contain any @author tags.

    +0 tests included.  The patch appears to be a documentation patch that doesn't require tests.

    -1 patch.  The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/627/console

This message is automatically generated.

> Hadoop Doc Split: Common Docs
> -----------------------------
>
>                 Key: HADOOP-6217
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6217
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>    Affects Versions: 0.21.0
>            Reporter: Corinne Chandel
>            Assignee: Owen O'Malley
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6217-common.patch, Hadoop-Doc-Split.doc
>
>
> Hadoop Doc Split: Common Docs
> Please note that I am unable to directly check all of the new links. Some links may break and will need to be updated.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1059-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 00:39:21 2009
Return-Path: <common-issues-return-1059-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88810 invoked from network); 27 Aug 2009 00:39:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 00:39:21 -0000
Received: (qmail 93853 invoked by uid 500); 27 Aug 2009 00:39:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93794 invoked by uid 500); 27 Aug 2009 00:39:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93784 invoked by uid 99); 27 Aug 2009 00:39:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 00:39:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 00:39:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 92789234C4AB
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 17:38:59 -0700 (PDT)
Message-ID: <134867415.1251333539599.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 17:38:59 -0700 (PDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5363) Proxying for multiple HDFS clusters
 of different versions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-5363:
------------------------------

    Attachment:     (was: h5363-0.20.patch)

> Proxying for multiple HDFS clusters of different versions
> ---------------------------------------------------------
>
>                 Key: HADOOP-5363
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5363
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Kan Zhang
>            Assignee: zhiyong zhang
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5363.patch, HADOOP-5363.patch, HADOOP-5363.patch, HADOOP-5363.patch, ProxyTestPlan.html
>
>
> A single hdfsproxy server should be able to proxy for multiple HDFS clusters, whose Hadoop versions may be different from each other.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1060-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 00:43:21 2009
Return-Path: <common-issues-return-1060-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89464 invoked from network); 27 Aug 2009 00:43:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 00:43:21 -0000
Received: (qmail 95483 invoked by uid 500); 27 Aug 2009 00:43:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95423 invoked by uid 500); 27 Aug 2009 00:43:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95413 invoked by uid 99); 27 Aug 2009 00:43:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 00:43:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 00:43:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4612A234C044
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 17:42:59 -0700 (PDT)
Message-ID: <76374638.1251333779272.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 17:42:59 -0700 (PDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5363) Proxying for multiple HDFS clusters
 of different versions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-5363:
------------------------------

    Attachment: 5363-backport.1.patch

attaching a new patch for back-porting to 0.20 (not for checking in)

> Proxying for multiple HDFS clusters of different versions
> ---------------------------------------------------------
>
>                 Key: HADOOP-5363
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5363
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Kan Zhang
>            Assignee: zhiyong zhang
>             Fix For: 0.21.0
>
>         Attachments: 5363-backport.1.patch, HADOOP-5363.patch, HADOOP-5363.patch, HADOOP-5363.patch, HADOOP-5363.patch, ProxyTestPlan.html
>
>
> A single hdfsproxy server should be able to proxy for multiple HDFS clusters, whose Hadoop versions may be different from each other.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1061-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 01:07:23 2009
Return-Path: <common-issues-return-1061-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93975 invoked from network); 27 Aug 2009 01:07:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 01:07:23 -0000
Received: (qmail 16774 invoked by uid 500); 27 Aug 2009 01:07:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16737 invoked by uid 500); 27 Aug 2009 01:07:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16727 invoked by uid 99); 27 Aug 2009 01:07:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 01:07:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 01:07:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4B804234C044
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 18:06:59 -0700 (PDT)
Message-ID: <1816954648.1251335219304.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 18:06:59 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5879) GzipCodec should read compression
 level etc from configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-5879:
----------------------------------

    Status: Patch Available  (was: Open)

> GzipCodec should read compression level etc from configuration
> --------------------------------------------------------------
>
>                 Key: HADOOP-5879
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5879
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Zheng Shao
>            Assignee: He Yongqiang
>         Attachments: hadoop-5879-5-21.patch, hadoop-5879-7-13-2.patch, hadoop-5879-7-13-3.patch, hadoop-5879-7-18-3.patch, hadoop-5879-7-26.patch
>
>
> GzipCodec currently uses the default compression level. We should allow overriding the default value from Configuration.
> {code}
>   static final class GzipZlibCompressor extends ZlibCompressor {
>     public GzipZlibCompressor() {
>       super(ZlibCompressor.CompressionLevel.DEFAULT_COMPRESSION,
>           ZlibCompressor.CompressionStrategy.DEFAULT_STRATEGY,
>           ZlibCompressor.CompressionHeader.GZIP_FORMAT, 64*1024);
>     }
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1062-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 02:39:23 2009
Return-Path: <common-issues-return-1062-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27740 invoked from network); 27 Aug 2009 02:39:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 02:39:23 -0000
Received: (qmail 88983 invoked by uid 500); 27 Aug 2009 02:39:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88923 invoked by uid 500); 27 Aug 2009 02:39:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88908 invoked by uid 99); 27 Aug 2009 02:39:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 02:39:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 02:39:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 7196F234C1EE
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 19:38:59 -0700 (PDT)
Message-ID: <950933792.1251340739464.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 19:38:59 -0700 (PDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4952) Improved files system interface for
 the application writer.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-4952:
---------------------------------

    Attachment: FileContext3.patch

> Improved files system interface for the application writer.
> -----------------------------------------------------------
>
>                 Key: HADOOP-4952
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4952
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.21.0
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: FileContext3.patch, Files.java, Files.java, FilesContext1.patch, FilesContext2.patch
>
>
> Currently the FIleSystem interface serves two purposes:
> - an application writer's interface for using the Hadoop file system
> - a file system implementer's interface (e.g. hdfs, local file system, kfs, etc)
> This Jira proposes that we provide a simpler interfaces for the application writer and leave the FilsSystem  interface for the implementer of a filesystem.
> - Filesystem interface  has a  confusing set of methods for the application writer
> - We could make it easier to take advantage of the URI file naming
> ** Current approach is to get FileSystem instance by supplying the URI and then access that name space. It is consistent for the FileSystem instance to not accept URIs for other schemes, but we can do better.
> ** The special copyFromLocalFIle can be generalized as a  copyFile where the src or target can be generalized to any URI, including the local one.
> ** The proposed scheme (below) simplifies this.
> -	The client side config can be simplified. 
> ** New config() by default uses the default config. Since this is the common usage pattern, one should not need to always pass the config as a parameter when accessing the file system.  
> -	
> ** It does not handle multiple file systems too well. Today a site.xml is derived from a single Hadoop cluster. This does not make sense for multiple Hadoop clusters which may have different defaults.
> ** Further one should need very little to configure the client side:
> *** Default files system.
> *** Block size 
> *** Replication factor
> *** Scheme to class mapping
> ** It should be possible to take Blocksize and replication factors defaults from the target file system, rather then the client size config.  I am not suggesting we don't allow setting client side defaults, but most clients do not care and would find it simpler to take the defaults for their systems  from the target file system. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1063-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 04:15:23 2009
Return-Path: <common-issues-return-1063-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45991 invoked from network); 27 Aug 2009 04:15:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 04:15:23 -0000
Received: (qmail 42417 invoked by uid 500); 27 Aug 2009 04:15:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42355 invoked by uid 500); 27 Aug 2009 04:15:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42340 invoked by uid 99); 27 Aug 2009 04:15:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 04:15:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 04:15:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4E3AD234C044
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 21:14:59 -0700 (PDT)
Message-ID: <998549500.1251346499305.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 21:14:59 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6170) add Avro-based RPC serialization
In-Reply-To: <1566370037.1248818534917.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748254#action_12748254 ] 

Hadoop QA commented on HADOOP-6170:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12417826/HADOOP-6170.patch
  against trunk revision 807753.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 7 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The applied patch generated 119 javac compiler warnings (more than the trunk's current 116 warnings).

    -1 findbugs.  The patch appears to introduce 1 new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/628/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/628/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/628/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/628/console

This message is automatically generated.

> add Avro-based RPC serialization
> --------------------------------
>
>                 Key: HADOOP-6170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6170
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6170.patch
>
>
> Permit RPC protocols to use Avro to serialize requests and responses, so that protocols may better evolve without breaking compatibility.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1064-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 05:55:23 2009
Return-Path: <common-issues-return-1064-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71172 invoked from network); 27 Aug 2009 05:55:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 05:55:23 -0000
Received: (qmail 6959 invoked by uid 500); 27 Aug 2009 05:55:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6898 invoked by uid 500); 27 Aug 2009 05:55:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6882 invoked by uid 99); 27 Aug 2009 05:55:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 05:55:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 05:55:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 68CE7234C1F0
	for <common-issues@hadoop.apache.org>; Wed, 26 Aug 2009 22:54:59 -0700 (PDT)
Message-ID: <340009043.1251352499428.JavaMail.jira@brutus>
Date: Wed, 26 Aug 2009 22:54:59 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5879) GzipCodec should read compression
 level etc from configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748267#action_12748267 ] 

Hadoop QA commented on HADOOP-5879:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12414549/hadoop-5879-7-26.patch
  against trunk revision 807753.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs.  The patch appears to introduce 4 new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/629/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/629/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/629/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/629/console

This message is automatically generated.

> GzipCodec should read compression level etc from configuration
> --------------------------------------------------------------
>
>                 Key: HADOOP-5879
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5879
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Zheng Shao
>            Assignee: He Yongqiang
>         Attachments: hadoop-5879-5-21.patch, hadoop-5879-7-13-2.patch, hadoop-5879-7-13-3.patch, hadoop-5879-7-18-3.patch, hadoop-5879-7-26.patch
>
>
> GzipCodec currently uses the default compression level. We should allow overriding the default value from Configuration.
> {code}
>   static final class GzipZlibCompressor extends ZlibCompressor {
>     public GzipZlibCompressor() {
>       super(ZlibCompressor.CompressionLevel.DEFAULT_COMPRESSION,
>           ZlibCompressor.CompressionStrategy.DEFAULT_STRATEGY,
>           ZlibCompressor.CompressionHeader.GZIP_FORMAT, 64*1024);
>     }
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1065-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 07:51:24 2009
Return-Path: <common-issues-return-1065-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10559 invoked from network); 27 Aug 2009 07:51:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 07:51:24 -0000
Received: (qmail 29559 invoked by uid 500); 27 Aug 2009 07:51:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29496 invoked by uid 500); 27 Aug 2009 07:51:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29486 invoked by uid 99); 27 Aug 2009 07:51:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 07:51:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 07:51:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 8830C234C48D
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 00:50:59 -0700 (PDT)
Message-ID: <1592343681.1251359459557.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 00:50:59 -0700 (PDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4952) Improved files system interface for
 the application writer.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-4952:
---------------------------------

    Attachment: FileContext5.patch

Updated patch.
(Patch is not complete yet - has missing parts and some bugs I know of).

Addressed Doug's naming nits + some bug fixes.
>    *  I'd prefer to name this just Files or perhaps FileContext rather than FilesContext.
Called it FileContext.
The term context is appropriate since FileContext is the context in which file names are resolved.
Many folks confused Files with FileSystem (thinking I really meant FileSystem but since the
name was taken I called it Files).
>    * FSConfigUtil could simply be called FSConfig. Moreover, its methods could be FileContext methods, >no? Why do we need this as a separate class?
Renamed it.
Kept it as a separate class because it has nothing to do with FilesContext; it merely sets/gets
keys from the Config. Besides other parts of fs may need to access the keys or their default
values.

>   * Why do we need the inner class FSConfig? Its fields seem like they should be FileContext fields directly, no?
Fixed it.


> Improved files system interface for the application writer.
> -----------------------------------------------------------
>
>                 Key: HADOOP-4952
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4952
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.21.0
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: FileContext3.patch, FileContext5.patch, Files.java, Files.java, FilesContext1.patch, FilesContext2.patch
>
>
> Currently the FIleSystem interface serves two purposes:
> - an application writer's interface for using the Hadoop file system
> - a file system implementer's interface (e.g. hdfs, local file system, kfs, etc)
> This Jira proposes that we provide a simpler interfaces for the application writer and leave the FilsSystem  interface for the implementer of a filesystem.
> - Filesystem interface  has a  confusing set of methods for the application writer
> - We could make it easier to take advantage of the URI file naming
> ** Current approach is to get FileSystem instance by supplying the URI and then access that name space. It is consistent for the FileSystem instance to not accept URIs for other schemes, but we can do better.
> ** The special copyFromLocalFIle can be generalized as a  copyFile where the src or target can be generalized to any URI, including the local one.
> ** The proposed scheme (below) simplifies this.
> -	The client side config can be simplified. 
> ** New config() by default uses the default config. Since this is the common usage pattern, one should not need to always pass the config as a parameter when accessing the file system.  
> -	
> ** It does not handle multiple file systems too well. Today a site.xml is derived from a single Hadoop cluster. This does not make sense for multiple Hadoop clusters which may have different defaults.
> ** Further one should need very little to configure the client side:
> *** Default files system.
> *** Block size 
> *** Replication factor
> *** Scheme to class mapping
> ** It should be possible to take Blocksize and replication factors defaults from the target file system, rather then the client size config.  I am not suggesting we don't allow setting client side defaults, but most clients do not care and would find it simpler to take the defaults for their systems  from the target file system. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1066-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 09:13:21 2009
Return-Path: <common-issues-return-1066-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31198 invoked from network); 27 Aug 2009 09:13:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 09:13:21 -0000
Received: (qmail 21935 invoked by uid 500); 27 Aug 2009 09:13:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21901 invoked by uid 500); 27 Aug 2009 09:13:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21889 invoked by uid 99); 27 Aug 2009 09:13:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 09:13:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 09:13:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4C779234C045
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 02:12:59 -0700 (PDT)
Message-ID: <1807071602.1251364379312.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 02:12:59 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6218) Split TFile by Record Sequence Number
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Split TFile by Record Sequence Number
-------------------------------------

                 Key: HADOOP-6218
                 URL: https://issues.apache.org/jira/browse/HADOOP-6218
             Project: Hadoop Common
          Issue Type: New Feature
            Reporter: Hong Tang


It would be nice if TFile can be split by Record Sequence Number. This way, columnar storage like PIG-833 can align fields that belong to the same row but in different columns.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1067-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 09:27:21 2009
Return-Path: <common-issues-return-1067-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35602 invoked from network); 27 Aug 2009 09:27:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 09:27:21 -0000
Received: (qmail 38821 invoked by uid 500); 27 Aug 2009 09:27:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38777 invoked by uid 500); 27 Aug 2009 09:27:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38767 invoked by uid 99); 27 Aug 2009 09:27:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 09:27:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 09:27:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 71371234C044
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 02:26:59 -0700 (PDT)
Message-ID: <1829416477.1251365219459.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 02:26:59 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6218) Split TFile by Record Sequence Number
In-Reply-To: <1807071602.1251364379312.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-6218:
------------------------------

    Attachment: hadoop-6218-20090827.patch

Preliminary patch for review. Need to add more test cases.

> Split TFile by Record Sequence Number
> -------------------------------------
>
>                 Key: HADOOP-6218
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6218
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Hong Tang
>         Attachments: hadoop-6218-20090827.patch
>
>
> It would be nice if TFile can be split by Record Sequence Number. This way, columnar storage like PIG-833 can align fields that belong to the same row but in different columns.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1068-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 11:15:21 2009
Return-Path: <common-issues-return-1068-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82258 invoked from network); 27 Aug 2009 11:15:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 11:15:21 -0000
Received: (qmail 68130 invoked by uid 500); 27 Aug 2009 11:15:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68093 invoked by uid 500); 27 Aug 2009 11:15:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68082 invoked by uid 99); 27 Aug 2009 11:15:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 11:15:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 11:15:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 49627234C044
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 04:14:59 -0700 (PDT)
Message-ID: <1487620282.1251371699286.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 04:14:59 -0700 (PDT)
From: "Sreekanth Ramakrishnan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Moved: (HADOOP-6219) Add dumpConfiguration option in hadoop
 help message
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sreekanth Ramakrishnan moved MAPREDUCE-919 to HADOOP-6219:
----------------------------------------------------------

          Component/s:     (was: jobtracker)
    Affects Version/s:     (was: 0.20.1)
                       0.20.1
           Issue Type: Bug  (was: Improvement)
                  Key: HADOOP-6219  (was: MAPREDUCE-919)
              Project: Hadoop Common  (was: Hadoop Map/Reduce)

> Add dumpConfiguration option in hadoop help message
> ---------------------------------------------------
>
>                 Key: HADOOP-6219
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6219
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.1
>            Reporter: Ramya R
>            Priority: Trivial
>
> Execution of bin/hadoop should show the -dumpConfiguration option introduced in MAPREDUCE-768

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1069-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 11:17:23 2009
Return-Path: <common-issues-return-1069-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82738 invoked from network); 27 Aug 2009 11:17:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 11:17:23 -0000
Received: (qmail 69547 invoked by uid 500); 27 Aug 2009 11:17:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69487 invoked by uid 500); 27 Aug 2009 11:17:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69477 invoked by uid 99); 27 Aug 2009 11:17:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 11:17:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 11:17:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4A43F234C044
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 04:16:59 -0700 (PDT)
Message-ID: <314859896.1251371819280.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 04:16:59 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6219) Add dumpConfiguration option in
 hadoop help message
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna updated HADOOP-6219:
------------------------------------------

    Attachment: MAPREDUCE-919.patch

uploading the patch with changes made to mapred script.

> Add dumpConfiguration option in hadoop help message
> ---------------------------------------------------
>
>                 Key: HADOOP-6219
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6219
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.1
>            Reporter: Ramya R
>            Priority: Trivial
>         Attachments: MAPREDUCE-919.patch
>
>
> Execution of bin/hadoop should show the -dumpConfiguration option introduced in MAPREDUCE-768

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1070-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 11:33:23 2009
Return-Path: <common-issues-return-1070-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85922 invoked from network); 27 Aug 2009 11:33:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 11:33:23 -0000
Received: (qmail 85181 invoked by uid 500); 27 Aug 2009 11:33:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85124 invoked by uid 500); 27 Aug 2009 11:33:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85113 invoked by uid 99); 27 Aug 2009 11:33:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 11:33:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 11:33:21 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 05124234C044
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 04:33:00 -0700 (PDT)
Message-ID: <927115566.1251372780748.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 04:33:00 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6219) Add dumpConfiguration option in
 hadoop help message
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna updated HADOOP-6219:
------------------------------------------

    Attachment: HADOOP-6184-ydist.patch

uploading the patch for Yahoo! internal distribution.

> Add dumpConfiguration option in hadoop help message
> ---------------------------------------------------
>
>                 Key: HADOOP-6219
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6219
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.1
>            Reporter: Ramya R
>            Priority: Trivial
>         Attachments: HADOOP-6184-ydist.patch, MAPREDUCE-919.patch
>
>
> Execution of bin/hadoop should show the -dumpConfiguration option introduced in MAPREDUCE-768

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1071-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 12:03:24 2009
Return-Path: <common-issues-return-1071-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91903 invoked from network); 27 Aug 2009 12:03:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 12:03:24 -0000
Received: (qmail 12455 invoked by uid 500); 27 Aug 2009 12:03:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12329 invoked by uid 500); 27 Aug 2009 12:03:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12317 invoked by uid 99); 27 Aug 2009 12:03:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 12:03:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 12:03:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 6498C234C04B
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 05:02:59 -0700 (PDT)
Message-ID: <740685219.1251374579411.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 05:02:59 -0700 (PDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6220) Jetty wraps InterruptedExceptions by
 IOExceptions if interrupted in startup
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Jetty wraps InterruptedExceptions by IOExceptions if interrupted in startup
---------------------------------------------------------------------------

                 Key: HADOOP-6220
                 URL: https://issues.apache.org/jira/browse/HADOOP-6220
             Project: Hadoop Common
          Issue Type: Bug
          Components: util
    Affects Versions: 0.21.0
            Reporter: Steve Loughran
            Priority: Minor


Following on some discusson on mapred-dev, we should keep an eye on the fact that Jetty uses sleeps when starting up; jetty can be a big part of the delays of bringing up a node. When interrupted, the exception is wrapped by an IOException, the root cause is still there, just hidden.

If we want callers to distinguish InterruptedExceptions from IOEs, then this exception should be extracted. Some helper method to start an http daemon could do this -catch the IOE, and if there is a nested interrupted exception, rethrow it, otherwise rethrowing the original IOE


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1072-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 12:07:24 2009
Return-Path: <common-issues-return-1072-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93441 invoked from network); 27 Aug 2009 12:07:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 12:07:24 -0000
Received: (qmail 16774 invoked by uid 500); 27 Aug 2009 12:07:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16665 invoked by uid 500); 27 Aug 2009 12:07:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16642 invoked by uid 99); 27 Aug 2009 12:07:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 12:07:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 12:07:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 5355C234C04B
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 05:06:59 -0700 (PDT)
Message-ID: <1205196426.1251374819340.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 05:06:59 -0700 (PDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6221) RPC.WaitForProxy swallows
 InterruptedException
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

RPC.WaitForProxy swallows InterruptedException
----------------------------------------------

                 Key: HADOOP-6221
                 URL: https://issues.apache.org/jira/browse/HADOOP-6221
             Project: Hadoop Common
          Issue Type: Bug
          Components: ipc
            Reporter: Steve Loughran


RPC.waitForProxy swallows any attempts to interrupt it while waiting for a proxy; this makes it hard to shutdown a service that you are starting; you have to wait for the timeouts. 

There are only 4-5 places in the code that use either of the two overloaded methods, removing the catch and changing the signature should not be too painful, unless anyone is using the method outside the hadoop codebase. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1073-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 12:51:21 2009
Return-Path: <common-issues-return-1073-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14589 invoked from network); 27 Aug 2009 12:51:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 12:51:21 -0000
Received: (qmail 79101 invoked by uid 500); 27 Aug 2009 12:51:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79068 invoked by uid 500); 27 Aug 2009 12:51:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79058 invoked by uid 99); 27 Aug 2009 12:51:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 12:51:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 12:51:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 48F80234C044
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 05:50:59 -0700 (PDT)
Message-ID: <1987267686.1251377459284.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 05:50:59 -0700 (PDT)
From: "Amar Kamat (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6215) fix GenericOptionParser to deal
 with -D with '=' in the value
In-Reply-To: <2129574511.1251299879368.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748358#action_12748358 ] 

Amar Kamat commented on HADOOP-6215:
------------------------------------

All core tests except TestDistributedFileSystem and TestReduceFetch failed on core.
All contrib tests except TestStreamingExitStatus, TestJobInitialization and TestQueueCapacities passed.

> fix GenericOptionParser to deal with -D with '=' in the value
> -------------------------------------------------------------
>
>                 Key: HADOOP-6215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6215
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Amar Kamat
>             Fix For: 0.20.1
>
>         Attachments: HADOOP-6215-v1.0-branch-0.20.patch, HADOOP-6215-v1.0.patch
>
>
> Currently the code ignores any -D option that has a "=" in the value. This is clearly wrong.
> In particular, -Dx=y=z should work and assign "y=z" to "x". This is a regression from HADOOP-6213.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1074-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 13:02:25 2009
Return-Path: <common-issues-return-1074-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17166 invoked from network); 27 Aug 2009 13:02:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 13:02:25 -0000
Received: (qmail 90368 invoked by uid 500); 27 Aug 2009 13:02:25 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90330 invoked by uid 500); 27 Aug 2009 13:02:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90315 invoked by uid 99); 27 Aug 2009 13:02:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 13:02:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 13:02:22 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 0287A234C1EB
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 06:02:01 -0700 (PDT)
Message-ID: <1359174077.1251378121009.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 06:02:01 -0700 (PDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6220) HttpServer wraps
 InterruptedExceptions by IOExceptions if interrupted in startup
In-Reply-To: <740685219.1251374579411.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steve Loughran updated HADOOP-6220:
-----------------------------------

    Summary: HttpServer wraps InterruptedExceptions by IOExceptions if interrupted in startup  (was: Jetty wraps InterruptedExceptions by IOExceptions if interrupted in startup)

> HttpServer wraps InterruptedExceptions by IOExceptions if interrupted in startup
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6220
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6220
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Priority: Minor
>
> Following on some discusson on mapred-dev, we should keep an eye on the fact that Jetty uses sleeps when starting up; jetty can be a big part of the delays of bringing up a node. When interrupted, the exception is wrapped by an IOException, the root cause is still there, just hidden.
> If we want callers to distinguish InterruptedExceptions from IOEs, then this exception should be extracted. Some helper method to start an http daemon could do this -catch the IOE, and if there is a nested interrupted exception, rethrow it, otherwise rethrowing the original IOE

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1075-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 13:57:25 2009
Return-Path: <common-issues-return-1075-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43163 invoked from network); 27 Aug 2009 13:57:25 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 13:57:25 -0000
Received: (qmail 56495 invoked by uid 500); 27 Aug 2009 13:57:25 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56467 invoked by uid 500); 27 Aug 2009 13:57:25 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56457 invoked by uid 99); 27 Aug 2009 13:57:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 13:57:25 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 13:57:22 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 83377234C1ED
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 06:57:01 -0700 (PDT)
Message-ID: <2145653574.1251381421536.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 06:57:01 -0700 (PDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6215) fix GenericOptionParser to deal with
 -D with '=' in the value
In-Reply-To: <2129574511.1251299879368.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sharad Agarwal resolved HADOOP-6215.
------------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]

+1 I committed to trunk and branch 20. Thanks Amar!

> fix GenericOptionParser to deal with -D with '=' in the value
> -------------------------------------------------------------
>
>                 Key: HADOOP-6215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6215
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Amar Kamat
>             Fix For: 0.20.1
>
>         Attachments: HADOOP-6215-v1.0-branch-0.20.patch, HADOOP-6215-v1.0.patch
>
>
> Currently the code ignores any -D option that has a "=" in the value. This is clearly wrong.
> In particular, -Dx=y=z should work and assign "y=z" to "x". This is a regression from HADOOP-6213.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1076-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 14:11:24 2009
Return-Path: <common-issues-return-1076-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49113 invoked from network); 27 Aug 2009 14:11:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 14:11:23 -0000
Received: (qmail 76976 invoked by uid 500); 27 Aug 2009 14:11:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76921 invoked by uid 500); 27 Aug 2009 14:11:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76909 invoked by uid 99); 27 Aug 2009 14:11:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 14:11:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 14:11:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 74D23234C1ED
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 07:10:59 -0700 (PDT)
Message-ID: <129350089.1251382259477.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 07:10:59 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6215) fix GenericOptionParser to deal
 with -D with '=' in the value
In-Reply-To: <2129574511.1251299879368.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748382#action_12748382 ] 

Hudson commented on HADOOP-6215:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #2 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/2/])
    . fix GenericOptionParser to deal with -D with '=' in the value. Contributed by Amar Kamat.


> fix GenericOptionParser to deal with -D with '=' in the value
> -------------------------------------------------------------
>
>                 Key: HADOOP-6215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6215
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Amar Kamat
>             Fix For: 0.20.1
>
>         Attachments: HADOOP-6215-v1.0-branch-0.20.patch, HADOOP-6215-v1.0.patch
>
>
> Currently the code ignores any -D option that has a "=" in the value. This is clearly wrong.
> In particular, -Dx=y=z should work and assign "y=z" to "x". This is a regression from HADOOP-6213.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1077-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 16:09:22 2009
Return-Path: <common-issues-return-1077-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5895 invoked from network); 27 Aug 2009 16:09:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 16:09:21 -0000
Received: (qmail 72617 invoked by uid 500); 27 Aug 2009 16:09:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72560 invoked by uid 500); 27 Aug 2009 16:09:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72548 invoked by uid 99); 27 Aug 2009 16:09:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 16:09:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 16:09:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 96A77234C044
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 09:08:59 -0700 (PDT)
Message-ID: <500039846.1251389339606.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 09:08:59 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6170) add Avro-based RPC serialization
In-Reply-To: <1566370037.1248818534917.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-6170:
---------------------------------

    Status: Open  (was: Patch Available)

> add Avro-based RPC serialization
> --------------------------------
>
>                 Key: HADOOP-6170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6170
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6170.patch, HADOOP-6170.patch
>
>
> Permit RPC protocols to use Avro to serialize requests and responses, so that protocols may better evolve without breaking compatibility.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1078-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 16:09:23 2009
Return-Path: <common-issues-return-1078-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5937 invoked from network); 27 Aug 2009 16:09:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 16:09:23 -0000
Received: (qmail 72722 invoked by uid 500); 27 Aug 2009 16:09:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72659 invoked by uid 500); 27 Aug 2009 16:09:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72649 invoked by uid 99); 27 Aug 2009 16:09:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 16:09:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 16:09:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id B4E36234C4AC
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 09:08:59 -0700 (PDT)
Message-ID: <1852696718.1251389339739.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 09:08:59 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6170) add Avro-based RPC serialization
In-Reply-To: <1566370037.1248818534917.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-6170:
---------------------------------

    Attachment: HADOOP-6170.patch

New patch that fixes lint warnings.

> add Avro-based RPC serialization
> --------------------------------
>
>                 Key: HADOOP-6170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6170
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6170.patch, HADOOP-6170.patch
>
>
> Permit RPC protocols to use Avro to serialize requests and responses, so that protocols may better evolve without breaking compatibility.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1079-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 16:09:23 2009
Return-Path: <common-issues-return-1079-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6006 invoked from network); 27 Aug 2009 16:09:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 16:09:23 -0000
Received: (qmail 72812 invoked by uid 500); 27 Aug 2009 16:09:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72761 invoked by uid 500); 27 Aug 2009 16:09:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72719 invoked by uid 99); 27 Aug 2009 16:09:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 16:09:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 16:09:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CA87D234C4BB
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 09:08:59 -0700 (PDT)
Message-ID: <42928619.1251389339828.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 09:08:59 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6170) add Avro-based RPC serialization
In-Reply-To: <1566370037.1248818534917.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-6170:
---------------------------------

    Status: Patch Available  (was: Open)

> add Avro-based RPC serialization
> --------------------------------
>
>                 Key: HADOOP-6170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6170
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6170.patch, HADOOP-6170.patch
>
>
> Permit RPC protocols to use Avro to serialize requests and responses, so that protocols may better evolve without breaking compatibility.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1080-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 17:09:24 2009
Return-Path: <common-issues-return-1080-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47573 invoked from network); 27 Aug 2009 17:09:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 17:09:23 -0000
Received: (qmail 69609 invoked by uid 500); 27 Aug 2009 17:09:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69547 invoked by uid 500); 27 Aug 2009 17:09:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69537 invoked by uid 99); 27 Aug 2009 17:09:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 17:09:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 17:09:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 7D951234C48D
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 10:08:59 -0700 (PDT)
Message-ID: <22176831.1251392939513.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 10:08:59 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4952) Improved files system interface for
 the application writer.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748449#action_12748449 ] 

Doug Cutting commented on HADOOP-4952:
--------------------------------------

> Kept it as a separate class because it has nothing to do with FilesContext; it merely sets/gets
> keys from the Config.

It sets/gets the keys that FileContext uses, no?  I suppose others may read them, but, if most file access is through FileContext, then FileContext has a primary association with these keys.

> Besides other parts of fs may need to access the keys or their default
> values.

They're static -- other parts of the FS can access them.

I still don't see why these static methods belong on a separate class when they're so closely tied to FileContext.

> Improved files system interface for the application writer.
> -----------------------------------------------------------
>
>                 Key: HADOOP-4952
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4952
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.21.0
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: FileContext3.patch, FileContext5.patch, Files.java, Files.java, FilesContext1.patch, FilesContext2.patch
>
>
> Currently the FIleSystem interface serves two purposes:
> - an application writer's interface for using the Hadoop file system
> - a file system implementer's interface (e.g. hdfs, local file system, kfs, etc)
> This Jira proposes that we provide a simpler interfaces for the application writer and leave the FilsSystem  interface for the implementer of a filesystem.
> - Filesystem interface  has a  confusing set of methods for the application writer
> - We could make it easier to take advantage of the URI file naming
> ** Current approach is to get FileSystem instance by supplying the URI and then access that name space. It is consistent for the FileSystem instance to not accept URIs for other schemes, but we can do better.
> ** The special copyFromLocalFIle can be generalized as a  copyFile where the src or target can be generalized to any URI, including the local one.
> ** The proposed scheme (below) simplifies this.
> -	The client side config can be simplified. 
> ** New config() by default uses the default config. Since this is the common usage pattern, one should not need to always pass the config as a parameter when accessing the file system.  
> -	
> ** It does not handle multiple file systems too well. Today a site.xml is derived from a single Hadoop cluster. This does not make sense for multiple Hadoop clusters which may have different defaults.
> ** Further one should need very little to configure the client side:
> *** Default files system.
> *** Block size 
> *** Replication factor
> *** Scheme to class mapping
> ** It should be possible to take Blocksize and replication factors defaults from the target file system, rather then the client size config.  I am not suggesting we don't allow setting client side defaults, but most clients do not care and would find it simpler to take the defaults for their systems  from the target file system. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1081-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 17:17:23 2009
Return-Path: <common-issues-return-1081-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51599 invoked from network); 27 Aug 2009 17:17:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 17:17:23 -0000
Received: (qmail 82744 invoked by uid 500); 27 Aug 2009 17:17:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82704 invoked by uid 500); 27 Aug 2009 17:17:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82694 invoked by uid 99); 27 Aug 2009 17:17:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 17:17:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 17:17:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 695BA234C1EF
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 10:16:59 -0700 (PDT)
Message-ID: <1665508276.1251393419430.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 10:16:59 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6170) add Avro-based RPC serialization
In-Reply-To: <1566370037.1248818534917.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-6170:
---------------------------------

    Attachment: HADOOP-6170.patch

Ignore the warning rather than add bogus code to make it go away.

> add Avro-based RPC serialization
> --------------------------------
>
>                 Key: HADOOP-6170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6170
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6170.patch, HADOOP-6170.patch, HADOOP-6170.patch
>
>
> Permit RPC protocols to use Avro to serialize requests and responses, so that protocols may better evolve without breaking compatibility.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1082-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 17:19:23 2009
Return-Path: <common-issues-return-1082-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52841 invoked from network); 27 Aug 2009 17:19:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 17:19:23 -0000
Received: (qmail 86360 invoked by uid 500); 27 Aug 2009 17:19:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86305 invoked by uid 500); 27 Aug 2009 17:19:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86265 invoked by uid 99); 27 Aug 2009 17:19:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 17:19:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 17:19:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 67D6E234C1EE
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 10:18:59 -0700 (PDT)
Message-ID: <549721336.1251393539424.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 10:18:59 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6170) add Avro-based RPC serialization
In-Reply-To: <1566370037.1248818534917.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748454#action_12748454 ] 

Hadoop QA commented on HADOOP-6170:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12417892/HADOOP-6170.patch
  against trunk revision 808415.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 7 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/630/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/630/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/630/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/630/console

This message is automatically generated.

> add Avro-based RPC serialization
> --------------------------------
>
>                 Key: HADOOP-6170
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6170
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6170.patch, HADOOP-6170.patch, HADOOP-6170.patch
>
>
> Permit RPC protocols to use Avro to serialize requests and responses, so that protocols may better evolve without breaking compatibility.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1083-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 18:38:21 2009
Return-Path: <common-issues-return-1083-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1573 invoked from network); 27 Aug 2009 18:38:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 18:38:21 -0000
Received: (qmail 94294 invoked by uid 500); 27 Aug 2009 18:38:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94195 invoked by uid 500); 27 Aug 2009 18:38:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94140 invoked by uid 99); 27 Aug 2009 18:38:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 18:38:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 18:38:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 6334F234C1EF
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 11:37:59 -0700 (PDT)
Message-ID: <1300675905.1251398279405.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 11:37:59 -0700 (PDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6203) Improve error message when moving
 to trash fails due to quota issue
In-Reply-To: <1489131604.1250623814866.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748495#action_12748495 ] 

Boris Shkolnik commented on HADOOP-6203:
----------------------------------------

Well, there are few ways of solving this.
1. print out the "cause" exception together with the error message. That will tell the person that the problem was with Quota. We cannot make suggestions about "-skipTrash" in this case.
2. catch and rethrow exception in delete function when trying to move to Trash. This way we know that the problem is with trash and we can issue a message suggesting using "-skipTrash". The disadvantage is that this way we issue two separate messages.

> Improve error message when moving to trash fails due to quota issue
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6203
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6203
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Jakob Homan
>
> HADOOP-6080 provided an option for deleting files even when overquota, but the error message that's returned in this situation is unhelpful and doesn't suggest skipTrash as a remediation:
> {noformat}$ hdfs -rmr /foo/bar/bat/boo
> rmr: Failed to move to trash:
> hdfs://cluster/foo/bar/bat/boo{noformat}
> In this situation, the error message should say there was a quote problem and suggest -skipTrash.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1084-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 20:26:21 2009
Return-Path: <common-issues-return-1084-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32643 invoked from network); 27 Aug 2009 20:26:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 20:26:21 -0000
Received: (qmail 59786 invoked by uid 500); 27 Aug 2009 20:26:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59729 invoked by uid 500); 27 Aug 2009 20:26:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59714 invoked by uid 99); 27 Aug 2009 20:26:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 20:26:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 20:26:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 46B6D234C044
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 13:25:59 -0700 (PDT)
Message-ID: <1508561314.1251404759274.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 13:25:59 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6204) Implementing aspects development
 and fault injeciton framework for Hadoop
In-Reply-To: <1837050058.1250629394943.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748553#action_12748553 ] 

Konstantin Boudnik commented on HADOOP-6204:
--------------------------------------------

In order to avoid some massive ANT code duplication, I have sketched a solution for this as follows:
  - all FI logic is represented by a separate aop.xml file placed in build.aux folder
  - that aop.xml file is imported into build.xml
  - to make FI framework project independent I've renamed FI related targets to {{run-test-primary-fault-inject}}. {{run-test-secondary-fault-inject}}, {{jar-fault-inject}}, and {{jar-test-fault-inject}}
  - project specific build.properties should contains actual target to be executed by aforementioned generic names.

Now, build.aux can be pulled from Common through svn:externals and then everything will be working in a very similar way. The above solution has 2 issues in my opinion:
  - an indirection level is being added. One way to avoid the abstraction is to define a number of macros in the aop.xml file and parametrize them from build.xml. It might be better approach.
  - svn:externals aren't universally acceptable. E.g. if one wants to pull SVN repository into a git one, then externals are have to be processed as a special case.

I'd appreciate if anyone has any comment or preferences on this.

> Implementing aspects development and fault injeciton framework for Hadoop
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6204
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6204
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>
> Fault injection framework implementation in HDFS (HDFS-435) turns out to be a very useful feature both for error handling testing and for various simulations.
> There's certain demand for this framework, thus it need to be pulled up from HDFS and brought into Common, so other sub-projects will be able to share it if needed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1085-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 20:37:23 2009
Return-Path: <common-issues-return-1085-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36354 invoked from network); 27 Aug 2009 20:37:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 20:37:23 -0000
Received: (qmail 71843 invoked by uid 500); 27 Aug 2009 20:37:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71786 invoked by uid 500); 27 Aug 2009 20:37:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71776 invoked by uid 99); 27 Aug 2009 20:37:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 20:37:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 20:37:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4EFC0234C044
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 13:36:59 -0700 (PDT)
Message-ID: <1330296161.1251405419309.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 13:36:59 -0700 (PDT)
From: "stack (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4675) Current Ganglia metrics
 implementation is incompatible with Ganglia 3.1
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748556#action_12748556 ] 

stack commented on HADOOP-4675:
-------------------------------

I will commit this patch in next day or so unless objection.

> Current Ganglia metrics implementation is incompatible with Ganglia 3.1
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-4675
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4675
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: metrics
>    Affects Versions: 0.21.0
>            Reporter: Brian Bockelman
>            Assignee: Brian Bockelman
>             Fix For: 0.21.0
>
>         Attachments: hadoop-4675-2.patch, hadoop-4675-3.patch, HADOOP-4675-4.patch, HADOOP-4675-v5.patch, HADOOP-4675-v6.patch, HADOOP-4675-v7.patch, HADOOP-4675-v8.patch, hadoop-4675.patch
>
>
> Ganglia changed its wire protocol in the 3.1.x series; the current implementation only works for 3.0.x.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1086-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 21:17:27 2009
Return-Path: <common-issues-return-1086-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49368 invoked from network); 27 Aug 2009 21:17:27 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 21:17:27 -0000
Received: (qmail 19049 invoked by uid 500); 27 Aug 2009 21:17:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19024 invoked by uid 500); 27 Aug 2009 21:17:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19014 invoked by uid 99); 27 Aug 2009 21:17:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 21:17:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 21:17:24 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 8641B234C044
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 14:17:03 -0700 (PDT)
Message-ID: <1308723959.1251407823534.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 14:17:03 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6204) Implementing aspects development and
 fault injeciton framework for Hadoop
In-Reply-To: <1837050058.1250629394943.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6204:
---------------------------------------

    Attachment: HADOOP-6204.patch.withmacros

This is the solution with macros in place and without additional indirection level for target names

> Implementing aspects development and fault injeciton framework for Hadoop
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6204
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6204
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-6204.patch.withmacros
>
>
> Fault injection framework implementation in HDFS (HDFS-435) turns out to be a very useful feature both for error handling testing and for various simulations.
> There's certain demand for this framework, thus it need to be pulled up from HDFS and brought into Common, so other sub-projects will be able to share it if needed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1087-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 21:19:21 2009
Return-Path: <common-issues-return-1087-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49696 invoked from network); 27 Aug 2009 21:19:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 21:19:21 -0000
Received: (qmail 21076 invoked by uid 500); 27 Aug 2009 21:19:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21026 invoked by uid 500); 27 Aug 2009 21:19:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21016 invoked by uid 99); 27 Aug 2009 21:19:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 21:19:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 21:19:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 478E4234C044
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 14:18:59 -0700 (PDT)
Message-ID: <1927072709.1251407939279.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 14:18:59 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6204) Implementing aspects development and
 fault injeciton framework for Hadoop
In-Reply-To: <1837050058.1250629394943.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6204:
---------------------------------------

    Attachment: HADOOP-6204.patch.indirect

This is the solution with name resolutions (i.e. indirection layer)

Both are created on top of existing HDFS' FI framework and will be adopted for Common in the final version of the patch.

> Implementing aspects development and fault injeciton framework for Hadoop
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6204
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6204
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-6204.patch.indirect, HADOOP-6204.patch.withmacros
>
>
> Fault injection framework implementation in HDFS (HDFS-435) turns out to be a very useful feature both for error handling testing and for various simulations.
> There's certain demand for this framework, thus it need to be pulled up from HDFS and brought into Common, so other sub-projects will be able to share it if needed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1088-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 21:21:21 2009
Return-Path: <common-issues-return-1088-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50762 invoked from network); 27 Aug 2009 21:21:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 21:21:21 -0000
Received: (qmail 25164 invoked by uid 500); 27 Aug 2009 21:21:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25134 invoked by uid 500); 27 Aug 2009 21:21:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25124 invoked by uid 99); 27 Aug 2009 21:21:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 21:21:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 21:21:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 474DD234C044
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 14:20:59 -0700 (PDT)
Message-ID: <939364923.1251408059277.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 14:20:59 -0700 (PDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6203) Improve error message when moving
 to trash fails due to quota issue
In-Reply-To: <1489131604.1250623814866.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748576#action_12748576 ] 

Jakob Homan commented on HADOOP-6203:
-------------------------------------

2 is probably preferable.  The real implementation problem is that the Quota exception is in HDFS and so can't be caught and identified in Common.  

> Improve error message when moving to trash fails due to quota issue
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6203
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6203
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Jakob Homan
>
> HADOOP-6080 provided an option for deleting files even when overquota, but the error message that's returned in this situation is unhelpful and doesn't suggest skipTrash as a remediation:
> {noformat}$ hdfs -rmr /foo/bar/bat/boo
> rmr: Failed to move to trash:
> hdfs://cluster/foo/bar/bat/boo{noformat}
> In this situation, the error message should say there was a quote problem and suggest -skipTrash.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1089-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 27 22:05:23 2009
Return-Path: <common-issues-return-1089-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74161 invoked from network); 27 Aug 2009 22:05:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 27 Aug 2009 22:05:23 -0000
Received: (qmail 70048 invoked by uid 500); 27 Aug 2009 22:05:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69997 invoked by uid 500); 27 Aug 2009 22:05:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69987 invoked by uid 99); 27 Aug 2009 22:05:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 22:05:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Aug 2009 22:05:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id A0B4F234C044
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 15:04:59 -0700 (PDT)
Message-ID: <1606899960.1251410699646.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 15:04:59 -0700 (PDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6216) HDFS Web UI displays comments from
 dfs.exclude file and counts them as dead nodes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6216:
---------------------------------

    Attachment: HADOOP-6216.patch

Nicolas,Thanks for reviewing.
Attaching updated patch with unit test and changes suggested by Nicolas.

> HDFS Web UI displays comments from dfs.exclude file and counts them as dead nodes
> ---------------------------------------------------------------------------------
>
>                 Key: HADOOP-6216
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6216
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>            Reporter: Jim Huang
>            Assignee: Ravi Phulari
>         Attachments: HADOOP-6216.patch, HDFS-1.patch
>
>
> I am putting comments in dfs.exclude file such as:
> {noformat}
>  # 32 GB memory upgrades
> {noformat}
> HDFS Web UI counts each word in the commented line as a dead node.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1090-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 28 02:17:42 2009
Return-Path: <common-issues-return-1090-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64597 invoked from network); 28 Aug 2009 02:17:42 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Aug 2009 02:17:42 -0000
Received: (qmail 60109 invoked by uid 500); 28 Aug 2009 02:17:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60054 invoked by uid 500); 28 Aug 2009 02:17:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60039 invoked by uid 99); 28 Aug 2009 02:17:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 02:17:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 02:17:40 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 48B02234C044
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 19:14:59 -0700 (PDT)
Message-ID: <1172289327.1251425699282.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 19:14:59 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6196) sync(0); next() breaks SequenceFile
In-Reply-To: <1782929485.1250413814845.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-6196:
----------------------------------

    Status: Open  (was: Patch Available)

Cancelling patch until Cos's comments are addressed

> sync(0); next() breaks SequenceFile
> -----------------------------------
>
>                 Key: HADOOP-6196
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6196
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jay Booth
>         Attachments: sync-bug.patch
>
>
> Currently, the end of the SequenceFile header is a sync block that isn't prefaced with SYNC_ESCAPE.  This means that sync(0) followed by next() fails.  Patch w/ test attached, bumps VERSION from 6 to 7.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1091-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 28 03:35:22 2009
Return-Path: <common-issues-return-1091-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89282 invoked from network); 28 Aug 2009 03:35:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Aug 2009 03:35:22 -0000
Received: (qmail 9225 invoked by uid 500); 28 Aug 2009 03:35:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9164 invoked by uid 500); 28 Aug 2009 03:35:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9144 invoked by uid 99); 28 Aug 2009 03:35:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 03:35:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 03:35:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 49644234C044
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 20:34:59 -0700 (PDT)
Message-ID: <433230606.1251430499286.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 20:34:59 -0700 (PDT)
From: "Jay Booth (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6196) sync(0); next() breaks SequenceFile
In-Reply-To: <1782929485.1250413814845.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748660#action_12748660 ] 

Jay Booth commented on HADOOP-6196:
-----------------------------------

Hey, will address this weekend, sorry for the slow turnaround

> sync(0); next() breaks SequenceFile
> -----------------------------------
>
>                 Key: HADOOP-6196
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6196
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jay Booth
>         Attachments: sync-bug.patch
>
>
> Currently, the end of the SequenceFile header is a sync block that isn't prefaced with SYNC_ESCAPE.  This means that sync(0) followed by next() fails.  Patch w/ test attached, bumps VERSION from 6 to 7.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1092-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 28 03:47:22 2009
Return-Path: <common-issues-return-1092-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90564 invoked from network); 28 Aug 2009 03:47:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Aug 2009 03:47:22 -0000
Received: (qmail 20315 invoked by uid 500); 28 Aug 2009 03:47:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20246 invoked by uid 500); 28 Aug 2009 03:47:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20223 invoked by uid 99); 28 Aug 2009 03:47:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 03:47:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 03:47:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 45F7C234C044
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 20:46:59 -0700 (PDT)
Message-ID: <827709663.1251431219272.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 20:46:59 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6219) Add dumpConfiguration option in
 hadoop help message
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna updated HADOOP-6219:
------------------------------------------

    Attachment: HADOOP-6219-ydist.patch

The previously uploaded patch is wrong. Uploading the correct patch.

> Add dumpConfiguration option in hadoop help message
> ---------------------------------------------------
>
>                 Key: HADOOP-6219
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6219
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.1
>            Reporter: Ramya R
>            Priority: Trivial
>         Attachments: HADOOP-6184-ydist.patch, HADOOP-6219-ydist.patch, MAPREDUCE-919.patch
>
>
> Execution of bin/hadoop should show the -dumpConfiguration option introduced in MAPREDUCE-768

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1093-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 28 04:09:21 2009
Return-Path: <common-issues-return-1093-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93723 invoked from network); 28 Aug 2009 04:09:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Aug 2009 04:09:21 -0000
Received: (qmail 32850 invoked by uid 500); 28 Aug 2009 04:09:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32791 invoked by uid 500); 28 Aug 2009 04:09:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32777 invoked by uid 99); 28 Aug 2009 04:09:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 04:09:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 04:09:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 53AEE234C044
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 21:08:59 -0700 (PDT)
Message-ID: <1157721001.1251432539328.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 21:08:59 -0700 (PDT)
From: "Sreekanth Ramakrishnan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5488) HADOOP-2721 doesn't clean up
 descendant processes of a jvm that exits cleanly after running a task
 successfully
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sreekanth Ramakrishnan updated HADOOP-5488:
-------------------------------------------

    Release Note: Issue cleans up the descendants of the successful task JVM.

> HADOOP-2721 doesn't clean up descendant processes of a jvm that exits cleanly after running a task successfully
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5488
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5488
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Vinod K V
>            Assignee: Ravi Gummadi
>             Fix For: 0.21.0
>
>         Attachments: hadoop-5488-ydist.patch, HADOOP-5488.patch, HADOOP_5488_5614.patch, HADOOP_5488_5614.v1.2.patch, HADOOP_5488_5614.v1.3.patch, HADOOP_5488_5614.v1.patch, testcase5488.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1094-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 28 04:09:23 2009
Return-Path: <common-issues-return-1094-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93749 invoked from network); 28 Aug 2009 04:09:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Aug 2009 04:09:23 -0000
Received: (qmail 32914 invoked by uid 500); 28 Aug 2009 04:09:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32871 invoked by uid 500); 28 Aug 2009 04:09:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32861 invoked by uid 99); 28 Aug 2009 04:09:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 04:09:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 04:09:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 6EAD5234C48C
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 21:08:59 -0700 (PDT)
Message-ID: <249545744.1251432539452.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 21:08:59 -0700 (PDT)
From: "Sreekanth Ramakrishnan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5420) Support killing of process groups in
 LinuxTaskController binary
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sreekanth Ramakrishnan updated HADOOP-5420:
-------------------------------------------

    Release Note: Provide facility within in Linux task controller to clean up descendants of the successful task JVMs.

> Support killing of process groups in LinuxTaskController binary
> ---------------------------------------------------------------
>
>                 Key: HADOOP-5420
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5420
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Sreekanth Ramakrishnan
>            Assignee: Sreekanth Ramakrishnan
>             Fix For: 0.21.0
>
>         Attachments: 5420-fix-ydist.patch, 5420-ydist.patch.txt, hadoop-5420-1.patch, hadoop-5420-10.patch, hadoop-5420-11.patch, hadoop-5420-12.patch, hadoop-5420-2.patch, hadoop-5420-3.patch, hadoop-5420-4.patch, hadoop-5420-5.patch, hadoop-5420-6.patch, hadoop-5420-7.patch, hadoop-5420-8.patch, hadoop-5420-9.patch, HADOOP-5420-v20.patch, hadoop-5420.patch
>
>
> Support setsid based kill in LinuxTaskController.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1095-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 28 04:21:24 2009
Return-Path: <common-issues-return-1095-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95434 invoked from network); 28 Aug 2009 04:21:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Aug 2009 04:21:24 -0000
Received: (qmail 40203 invoked by uid 500); 28 Aug 2009 04:21:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40142 invoked by uid 500); 28 Aug 2009 04:21:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40132 invoked by uid 99); 28 Aug 2009 04:21:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 04:21:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 04:21:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 597FC234C044
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 21:20:59 -0700 (PDT)
Message-ID: <1630169193.1251433259352.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 21:20:59 -0700 (PDT)
From: "V.V.Chaitanya Krishna (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6184) Provide a configuration dump in json
 format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

V.V.Chaitanya Krishna updated HADOOP-6184:
------------------------------------------

    Release Note: The API dumpConfiguration(Configuration, Writer) in Configuration.java writes out all the parameters and their properties (final and resource) to the given Writer.  The format of the output would be { "properties" : [ {key1,value1,key1.isFinal,key1.resource}, {key2,value2,key2.isFinal,key2.resource}... ] }. It does not output the parameters of the configuration object which is loaded from an input stream.

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>            Assignee: V.V.Chaitanya Krishna
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184-2.patch, HADOOP-6184-3.patch, HADOOP-6184-4.patch, HADOOP-6184-5.patch, HADOOP-6184-6.patch, HADOOP-6184-7.patch, HADOOP-6184-8.patch, HADOOP-6184-9.patch, HADOOP-6184-ydist-1.patch, HADOOP-6184-ydist.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1096-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 28 05:59:23 2009
Return-Path: <common-issues-return-1096-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23992 invoked from network); 28 Aug 2009 05:59:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Aug 2009 05:59:23 -0000
Received: (qmail 5170 invoked by uid 500); 28 Aug 2009 05:59:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5122 invoked by uid 500); 28 Aug 2009 05:59:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5112 invoked by uid 99); 28 Aug 2009 05:59:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 05:59:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 05:59:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 445E9234C044
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 22:58:59 -0700 (PDT)
Message-ID: <225579764.1251439139266.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 22:58:59 -0700 (PDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5420) Support killing of process groups in
 LinuxTaskController binary
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala updated HADOOP-5420:
-------------------------------------

    Release Note: The LinuxTaskController was fixed to support proper cleanup of tasks and their sub-processes in order to ensure there were no 'hung' processes on tasktracker nodes.  (was: Provide facility within in Linux task controller to clean up descendants of the successful task JVMs.)

> Support killing of process groups in LinuxTaskController binary
> ---------------------------------------------------------------
>
>                 Key: HADOOP-5420
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5420
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Sreekanth Ramakrishnan
>            Assignee: Sreekanth Ramakrishnan
>             Fix For: 0.21.0
>
>         Attachments: 5420-fix-ydist.patch, 5420-ydist.patch.txt, hadoop-5420-1.patch, hadoop-5420-10.patch, hadoop-5420-11.patch, hadoop-5420-12.patch, hadoop-5420-2.patch, hadoop-5420-3.patch, hadoop-5420-4.patch, hadoop-5420-5.patch, hadoop-5420-6.patch, hadoop-5420-7.patch, hadoop-5420-8.patch, hadoop-5420-9.patch, HADOOP-5420-v20.patch, hadoop-5420.patch
>
>
> Support setsid based kill in LinuxTaskController.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1097-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 28 06:01:22 2009
Return-Path: <common-issues-return-1097-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25248 invoked from network); 28 Aug 2009 06:01:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Aug 2009 06:01:22 -0000
Received: (qmail 6181 invoked by uid 500); 28 Aug 2009 06:01:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6122 invoked by uid 500); 28 Aug 2009 06:01:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6112 invoked by uid 99); 28 Aug 2009 06:01:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 06:01:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 06:01:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 9051A234C044
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 23:01:00 -0700 (PDT)
Message-ID: <1979057345.1251439260582.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 23:01:00 -0700 (PDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5488) HADOOP-2721 doesn't clean up
 descendant processes of a jvm that exits cleanly after running a task
 successfully
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala updated HADOOP-5488:
-------------------------------------

    Release Note: Tasktracker was modified to ensure that successful and failed tasks and their sub-processes are cleaned up upon completion to prevent 'hung' processes on tasktracker nodes.  (was: Issue cleans up the descendants of the successful task JVM.)

> HADOOP-2721 doesn't clean up descendant processes of a jvm that exits cleanly after running a task successfully
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5488
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5488
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Vinod K V
>            Assignee: Ravi Gummadi
>             Fix For: 0.21.0
>
>         Attachments: hadoop-5488-ydist.patch, HADOOP-5488.patch, HADOOP_5488_5614.patch, HADOOP_5488_5614.v1.2.patch, HADOOP_5488_5614.v1.3.patch, HADOOP_5488_5614.v1.patch, testcase5488.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1098-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 28 06:03:23 2009
Return-Path: <common-issues-return-1098-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26189 invoked from network); 28 Aug 2009 06:03:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Aug 2009 06:03:23 -0000
Received: (qmail 7169 invoked by uid 500); 28 Aug 2009 06:03:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7114 invoked by uid 500); 28 Aug 2009 06:03:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7103 invoked by uid 99); 28 Aug 2009 06:03:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 06:03:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 06:03:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 50642234C1F0
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 23:02:59 -0700 (PDT)
Message-ID: <545340453.1251439379328.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 23:02:59 -0700 (PDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6184) Provide a configuration dump in json
 format.
In-Reply-To: <1775349108.1249902554870.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala updated HADOOP-6184:
-------------------------------------

    Release Note: The API dumpConfiguration(Configuration, Writer) was added to Configuration.java. Calling this API would write out all the configuration parameters and their properties (final and resource) to a specified Writer in the JSON format.  The format of the output would be { "properties" : [ {key1,value1,key1.isFinal,key1.resource}, {key2,value2,key2.isFinal,key2.resource}... ] }.  (was: The API dumpConfiguration(Configuration, Writer) in Configuration.java writes out all the parameters and their properties (final and resource) to the given Writer.  The format of the output would be { "properties" : [ {key1,value1,key1.isFinal,key1.resource}, {key2,value2,key2.isFinal,key2.resource}... ] }. It does not output the parameters of the configuration object which is loaded from an input stream.)

> Provide a configuration dump in json format.
> --------------------------------------------
>
>                 Key: HADOOP-6184
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6184
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: rahul k singh
>            Assignee: V.V.Chaitanya Krishna
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6184-1.patch, HADOOP-6184-2.patch, HADOOP-6184-3.patch, HADOOP-6184-4.patch, HADOOP-6184-5.patch, HADOOP-6184-6.patch, HADOOP-6184-7.patch, HADOOP-6184-8.patch, HADOOP-6184-9.patch, HADOOP-6184-ydist-1.patch, HADOOP-6184-ydist.patch, HADOOP-6184.patch
>
>
> Configuration dump in json format.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1099-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 28 06:39:21 2009
Return-Path: <common-issues-return-1099-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50685 invoked from network); 28 Aug 2009 06:39:21 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Aug 2009 06:39:21 -0000
Received: (qmail 27442 invoked by uid 500); 28 Aug 2009 06:39:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27385 invoked by uid 500); 28 Aug 2009 06:39:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27375 invoked by uid 99); 28 Aug 2009 06:39:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 06:39:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 06:39:19 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 481CB234C044
	for <common-issues@hadoop.apache.org>; Thu, 27 Aug 2009 23:38:59 -0700 (PDT)
Message-ID: <1307592037.1251441539279.JavaMail.jira@brutus>
Date: Thu, 27 Aug 2009 23:38:59 -0700 (PDT)
From: "Amar Kamat (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6215) fix GenericOptionParser to deal
 with -D with '=' in the value
In-Reply-To: <2129574511.1251299879368.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748690#action_12748690 ] 

Amar Kamat commented on HADOOP-6215:
------------------------------------

Note that this is not a regression caused because of HADOOP-6213. I tested the code pre-H6213 and found that "-D x=y=z" throws parse-exception. 

> fix GenericOptionParser to deal with -D with '=' in the value
> -------------------------------------------------------------
>
>                 Key: HADOOP-6215
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6215
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Amar Kamat
>             Fix For: 0.20.1
>
>         Attachments: HADOOP-6215-v1.0-branch-0.20.patch, HADOOP-6215-v1.0.patch
>
>
> Currently the code ignores any -D option that has a "=" in the value. This is clearly wrong.
> In particular, -Dx=y=z should work and assign "y=z" to "x". This is a regression from HADOOP-6213.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1104-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 28 17:14:23 2009
Return-Path: <common-issues-return-1104-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24008 invoked from network); 28 Aug 2009 17:14:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Aug 2009 17:14:23 -0000
Received: (qmail 5441 invoked by uid 500); 28 Aug 2009 16:14:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5380 invoked by uid 500); 28 Aug 2009 16:14:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5370 invoked by uid 99); 28 Aug 2009 16:14:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 16:14:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 16:14:20 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E6BBE234C044
	for <common-issues@hadoop.apache.org>; Fri, 28 Aug 2009 09:13:57 -0700 (PDT)
Message-ID: <124283550.1251476037940.JavaMail.jira@brutus>
Date: Fri, 28 Aug 2009 09:13:57 -0700 (PDT)
From: "Edward Capriolo (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5564) hadoop command uses large JVM heap
 size
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748872#action_12748872 ] 

Edward Capriolo commented on HADOOP-5564:
-----------------------------------------

Steve,

You definitely need this patch with java update 14&15. I have 18.3 and u 15. There are no surface errors! but my map tasks produced no output! Good thing I caught it relatively quickly. 




> hadoop command uses large JVM heap size
> ---------------------------------------
>
>                 Key: HADOOP-5564
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5564
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>            Priority: Blocker
>             Fix For: 0.20.0
>
>         Attachments: 5564.patch, 5564.trunk.patch, 5564.trunk.patch
>
>
> Command used to determine JAVA_PLATFORM in bin/hadoop command does not set the heap size. The command uses default 1GB heap size. The tasks invoking hadoop command end up  using large heap size in streaming jobs. If the maximum memory that can be used by a task is restricted, this could result in map/reduce job failures.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1105-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 28 18:46:55 2009
Return-Path: <common-issues-return-1105-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14010 invoked from network); 28 Aug 2009 18:46:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Aug 2009 18:46:55 -0000
Received: (qmail 67575 invoked by uid 500); 28 Aug 2009 18:46:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67516 invoked by uid 500); 28 Aug 2009 18:46:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67436 invoked by uid 99); 28 Aug 2009 18:46:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 18:46:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 18:46:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C4753234C1EF
	for <common-issues@hadoop.apache.org>; Fri, 28 Aug 2009 11:46:32 -0700 (PDT)
Message-ID: <1768107896.1251485192803.JavaMail.jira@brutus>
Date: Fri, 28 Aug 2009 11:46:32 -0700 (PDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6203) Improve error message when moving to
 trash fails due to quota issue
In-Reply-To: <1489131604.1250623814866.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6203:
-----------------------------------

    Attachment: HADOOP-6203.patch

> Improve error message when moving to trash fails due to quota issue
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6203
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6203
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Jakob Homan
>         Attachments: HADOOP-6203.patch
>
>
> HADOOP-6080 provided an option for deleting files even when overquota, but the error message that's returned in this situation is unhelpful and doesn't suggest skipTrash as a remediation:
> {noformat}$ hdfs -rmr /foo/bar/bat/boo
> rmr: Failed to move to trash:
> hdfs://cluster/foo/bar/bat/boo{noformat}
> In this situation, the error message should say there was a quote problem and suggest -skipTrash.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1106-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 28 18:48:54 2009
Return-Path: <common-issues-return-1106-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14301 invoked from network); 28 Aug 2009 18:48:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Aug 2009 18:48:54 -0000
Received: (qmail 69948 invoked by uid 500); 28 Aug 2009 18:48:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69893 invoked by uid 500); 28 Aug 2009 18:48:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69883 invoked by uid 99); 28 Aug 2009 18:48:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 18:48:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 18:48:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C3371234C1EF
	for <common-issues@hadoop.apache.org>; Fri, 28 Aug 2009 11:48:32 -0700 (PDT)
Message-ID: <1581085373.1251485312798.JavaMail.jira@brutus>
Date: Fri, 28 Aug 2009 11:48:32 -0700 (PDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6203) Improve error message when moving to
 trash fails due to quota issue
In-Reply-To: <1489131604.1250623814866.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6203:
-----------------------------------

    Attachment:     (was: HADOOP-6203.patch)

> Improve error message when moving to trash fails due to quota issue
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6203
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6203
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Jakob Homan
>         Attachments: HADOOP-6203.patch
>
>
> HADOOP-6080 provided an option for deleting files even when overquota, but the error message that's returned in this situation is unhelpful and doesn't suggest skipTrash as a remediation:
> {noformat}$ hdfs -rmr /foo/bar/bat/boo
> rmr: Failed to move to trash:
> hdfs://cluster/foo/bar/bat/boo{noformat}
> In this situation, the error message should say there was a quote problem and suggest -skipTrash.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1107-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 28 18:48:56 2009
Return-Path: <common-issues-return-1107-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14340 invoked from network); 28 Aug 2009 18:48:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Aug 2009 18:48:56 -0000
Received: (qmail 70028 invoked by uid 500); 28 Aug 2009 18:48:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69976 invoked by uid 500); 28 Aug 2009 18:48:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69966 invoked by uid 99); 28 Aug 2009 18:48:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 18:48:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 18:48:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C6565234C1F1
	for <common-issues@hadoop.apache.org>; Fri, 28 Aug 2009 11:48:32 -0700 (PDT)
Message-ID: <1690768054.1251485312811.JavaMail.jira@brutus>
Date: Fri, 28 Aug 2009 11:48:32 -0700 (PDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6203) Improve error message when moving to
 trash fails due to quota issue
In-Reply-To: <1489131604.1250623814866.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6203:
-----------------------------------

    Attachment: HADOOP-6203.patch

> Improve error message when moving to trash fails due to quota issue
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6203
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6203
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Jakob Homan
>         Attachments: HADOOP-6203.patch
>
>
> HADOOP-6080 provided an option for deleting files even when overquota, but the error message that's returned in this situation is unhelpful and doesn't suggest skipTrash as a remediation:
> {noformat}$ hdfs -rmr /foo/bar/bat/boo
> rmr: Failed to move to trash:
> hdfs://cluster/foo/bar/bat/boo{noformat}
> In this situation, the error message should say there was a quote problem and suggest -skipTrash.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1108-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 28 21:11:58 2009
Return-Path: <common-issues-return-1108-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62699 invoked from network); 28 Aug 2009 21:11:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Aug 2009 21:11:57 -0000
Received: (qmail 46803 invoked by uid 500); 28 Aug 2009 21:11:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46744 invoked by uid 500); 28 Aug 2009 21:11:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46734 invoked by uid 99); 28 Aug 2009 21:11:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 21:11:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 21:11:54 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CD145234C1EB
	for <common-issues@hadoop.apache.org>; Fri, 28 Aug 2009 14:11:32 -0700 (PDT)
Message-ID: <1797398304.1251493892839.JavaMail.jira@brutus>
Date: Fri, 28 Aug 2009 14:11:32 -0700 (PDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6203) Improve error message when moving to
 trash fails due to quota issue
In-Reply-To: <1489131604.1250623814866.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6203:
-----------------------------------

    Status: Patch Available  (was: Open)

> Improve error message when moving to trash fails due to quota issue
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6203
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6203
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Jakob Homan
>         Attachments: HADOOP-6203.patch
>
>
> HADOOP-6080 provided an option for deleting files even when overquota, but the error message that's returned in this situation is unhelpful and doesn't suggest skipTrash as a remediation:
> {noformat}$ hdfs -rmr /foo/bar/bat/boo
> rmr: Failed to move to trash:
> hdfs://cluster/foo/bar/bat/boo{noformat}
> In this situation, the error message should say there was a quote problem and suggest -skipTrash.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1109-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 28 21:35:57 2009
Return-Path: <common-issues-return-1109-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67901 invoked from network); 28 Aug 2009 21:35:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Aug 2009 21:35:57 -0000
Received: (qmail 67992 invoked by uid 500); 28 Aug 2009 21:35:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67968 invoked by uid 500); 28 Aug 2009 21:35:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67948 invoked by uid 99); 28 Aug 2009 21:35:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 21:35:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 21:35:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C8DD6234C1F0
	for <common-issues@hadoop.apache.org>; Fri, 28 Aug 2009 14:35:32 -0700 (PDT)
Message-ID: <1941261301.1251495332821.JavaMail.jira@brutus>
Date: Fri, 28 Aug 2009 14:35:32 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6203) Improve error message when moving
 to trash fails due to quota issue
In-Reply-To: <1489131604.1250623814866.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748992#action_12748992 ] 

Hadoop QA commented on HADOOP-6203:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12418014/HADOOP-6203.patch
  against trunk revision 808415.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/631/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/631/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/631/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/631/console

This message is automatically generated.

> Improve error message when moving to trash fails due to quota issue
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6203
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6203
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Jakob Homan
>         Attachments: HADOOP-6203.patch
>
>
> HADOOP-6080 provided an option for deleting files even when overquota, but the error message that's returned in this situation is unhelpful and doesn't suggest skipTrash as a remediation:
> {noformat}$ hdfs -rmr /foo/bar/bat/boo
> rmr: Failed to move to trash:
> hdfs://cluster/foo/bar/bat/boo{noformat}
> In this situation, the error message should say there was a quote problem and suggest -skipTrash.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1110-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 28 21:37:54 2009
Return-Path: <common-issues-return-1110-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68585 invoked from network); 28 Aug 2009 21:37:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Aug 2009 21:37:54 -0000
Received: (qmail 69455 invoked by uid 500); 28 Aug 2009 21:37:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69416 invoked by uid 500); 28 Aug 2009 21:37:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69406 invoked by uid 99); 28 Aug 2009 21:37:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 21:37:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 21:37:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id B9376234C044
	for <common-issues@hadoop.apache.org>; Fri, 28 Aug 2009 14:37:32 -0700 (PDT)
Message-ID: <499862303.1251495452740.JavaMail.jira@brutus>
Date: Fri, 28 Aug 2009 14:37:32 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6140)
 DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
In-Reply-To: <1165046874.1247245875107.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-6140:
----------------------------------

    Status: Open  (was: Patch Available)

All the work for this issue should probably be in MAPREDUCE-752.

If this fix is to be included on the 0.18 branch, we'll also need fixes for the 0.19 and 0.20 branches.

> DistributedCache.addArchiveToClassPath doesn't work in 0.18.x branch
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6140
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6140
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.18.3
>            Reporter: Vladimir Klimontovich
>         Attachments: HADOOP-6140-ver4.patch
>
>
> addArchiveToClassPath is a method of DistributedCache class. It should be called before running a task. It accepts path to a jar file on a DFS. After it
> this method should put this jar file on sitribuuted cache and than add this file to classpath to each map/reduce process on job tracker.
> This method didn't work.
> Bug 1:
> addArchiveToClassPath adds DFS-path to archive to mapred.job.classpath.archives property. It uses System.getProperty("path.separator") as delimiter of multiple path.
> getFileClassPaths that is called from TaskRunner uses splits mapred.job.classpath.archives using System.getProperty("path.separator").
> In unix systems System.getProperty("path.separator") equals to ":". DFS-path urls is hdfs://host:port/path. It means that a result of split will be
> [ hdfs,//host,port/path].
> Suggested solution: use "," instead of  
> Bug 2:
> in TaskRunner there is an algorithm that looks for correspondence between DFS paths and local paths in distributed cache. 
> It compares
>    if (archives[i].getPath().equals(
>                                                archiveClasspaths[j].toString())){
> instead of
>     if (archives[i].toString().equals(
>                                                archiveClasspaths[j].toString())) 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1111-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 28 22:15:56 2009
Return-Path: <common-issues-return-1111-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80172 invoked from network); 28 Aug 2009 22:15:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Aug 2009 22:15:56 -0000
Received: (qmail 568 invoked by uid 500); 28 Aug 2009 22:15:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 514 invoked by uid 500); 28 Aug 2009 22:15:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 500 invoked by uid 99); 28 Aug 2009 22:15:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 22:15:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 22:15:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id ACF69234C044
	for <common-issues@hadoop.apache.org>; Fri, 28 Aug 2009 15:15:32 -0700 (PDT)
Message-ID: <1734604859.1251497732693.JavaMail.jira@brutus>
Date: Fri, 28 Aug 2009 15:15:32 -0700 (PDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6203) Improve error message when moving to
 trash fails due to quota issue
In-Reply-To: <1489131604.1250623814866.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6203:
-----------------------------------

    Attachment: HADOOP-6203.patch

> Improve error message when moving to trash fails due to quota issue
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6203
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6203
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Jakob Homan
>         Attachments: HADOOP-6203.patch, HADOOP-6203.patch
>
>
> HADOOP-6080 provided an option for deleting files even when overquota, but the error message that's returned in this situation is unhelpful and doesn't suggest skipTrash as a remediation:
> {noformat}$ hdfs -rmr /foo/bar/bat/boo
> rmr: Failed to move to trash:
> hdfs://cluster/foo/bar/bat/boo{noformat}
> In this situation, the error message should say there was a quote problem and suggest -skipTrash.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1112-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 28 22:50:54 2009
Return-Path: <common-issues-return-1112-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91903 invoked from network); 28 Aug 2009 22:50:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Aug 2009 22:50:54 -0000
Received: (qmail 24667 invoked by uid 500); 28 Aug 2009 22:50:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24609 invoked by uid 500); 28 Aug 2009 22:50:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24599 invoked by uid 99); 28 Aug 2009 22:50:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 22:50:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 22:50:52 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CCD70234C046
	for <common-issues@hadoop.apache.org>; Fri, 28 Aug 2009 15:50:32 -0700 (PDT)
Message-ID: <565101244.1251499832837.JavaMail.jira@brutus>
Date: Fri, 28 Aug 2009 15:50:32 -0700 (PDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6222) Core doesn't have TestCommonCLI
 facility
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Core doesn't have TestCommonCLI facility
----------------------------------------

                 Key: HADOOP-6222
                 URL: https://issues.apache.org/jira/browse/HADOOP-6222
             Project: Hadoop Common
          Issue Type: Test
            Reporter: Boris Shkolnik


TestCLI is a base class, which cannot run FS type of commands.
We need a "copy" of TestHDFSCLI as TestCommonCLI to be able to test CLI stuff in common.

I suggest we create TestCommonCLI.java in hadoop-common

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1113-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 28 22:50:57 2009
Return-Path: <common-issues-return-1113-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91943 invoked from network); 28 Aug 2009 22:50:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Aug 2009 22:50:57 -0000
Received: (qmail 24788 invoked by uid 500); 28 Aug 2009 22:50:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24701 invoked by uid 500); 28 Aug 2009 22:50:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24691 invoked by uid 99); 28 Aug 2009 22:50:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 22:50:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 22:50:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D1186234C1EF
	for <common-issues@hadoop.apache.org>; Fri, 28 Aug 2009 15:50:32 -0700 (PDT)
Message-ID: <1243890492.1251499832855.JavaMail.jira@brutus>
Date: Fri, 28 Aug 2009 15:50:32 -0700 (PDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6222) Core doesn't have TestCommonCLI
 facility
In-Reply-To: <565101244.1251499832837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6222:
-----------------------------------

    Component/s: test

> Core doesn't have TestCommonCLI facility
> ----------------------------------------
>
>                 Key: HADOOP-6222
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6222
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>            Reporter: Boris Shkolnik
>
> TestCLI is a base class, which cannot run FS type of commands.
> We need a "copy" of TestHDFSCLI as TestCommonCLI to be able to test CLI stuff in common.
> I suggest we create TestCommonCLI.java in hadoop-common

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1114-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 28 23:40:57 2009
Return-Path: <common-issues-return-1114-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7928 invoked from network); 28 Aug 2009 23:40:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Aug 2009 23:40:57 -0000
Received: (qmail 47342 invoked by uid 500); 28 Aug 2009 23:40:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47292 invoked by uid 500); 28 Aug 2009 23:40:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47279 invoked by uid 99); 28 Aug 2009 23:40:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 23:40:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 23:40:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id B722B234C044
	for <common-issues@hadoop.apache.org>; Fri, 28 Aug 2009 16:40:32 -0700 (PDT)
Message-ID: <248480523.1251502832733.JavaMail.jira@brutus>
Date: Fri, 28 Aug 2009 16:40:32 -0700 (PDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6203) Improve error message when moving to
 trash fails due to quota issue
In-Reply-To: <1489131604.1250623814866.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik reassigned HADOOP-6203:
--------------------------------------

    Assignee: Boris Shkolnik

> Improve error message when moving to trash fails due to quota issue
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6203
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6203
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Jakob Homan
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6203.patch, HADOOP-6203.patch
>
>
> HADOOP-6080 provided an option for deleting files even when overquota, but the error message that's returned in this situation is unhelpful and doesn't suggest skipTrash as a remediation:
> {noformat}$ hdfs -rmr /foo/bar/bat/boo
> rmr: Failed to move to trash:
> hdfs://cluster/foo/bar/bat/boo{noformat}
> In this situation, the error message should say there was a quote problem and suggest -skipTrash.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1115-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 28 23:42:54 2009
Return-Path: <common-issues-return-1115-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9150 invoked from network); 28 Aug 2009 23:42:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Aug 2009 23:42:54 -0000
Received: (qmail 47990 invoked by uid 500); 28 Aug 2009 23:42:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47932 invoked by uid 500); 28 Aug 2009 23:42:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47922 invoked by uid 99); 28 Aug 2009 23:42:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 23:42:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 23:42:52 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CB5A1234C044
	for <common-issues@hadoop.apache.org>; Fri, 28 Aug 2009 16:42:32 -0700 (PDT)
Message-ID: <1857640246.1251502952819.JavaMail.jira@brutus>
Date: Fri, 28 Aug 2009 16:42:32 -0700 (PDT)
From: "Yuri Pradkin (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4012) Providing splitting support for
 bzip2 compressed files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749029#action_12749029 ] 

Yuri Pradkin commented on HADOOP-4012:
--------------------------------------

Gentle Hadoop folk,

may I ask what we are waiting for while not committing this patch?  If somebody is unhappy with this patch, it would be good to know their concern(s); likewise, if there are some internal discussions going on, it would be wonderful if somebody occasionally filled us in on that.  This patch has been in the queue for a long time and now that Hudson is finally happy with it (and I know for various reasons it is hard to satisfy ;-)  can somebody take charge and check it in?  If you are unwilling to check it in, please explain!

I think by delaying it, we're risking running into conflicts yet again and creating unnecessary work for ourselves.

Thank you!

> Providing splitting support for bzip2 compressed files
> ------------------------------------------------------
>
>                 Key: HADOOP-4012
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4012
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Abdul Qadeer
>            Assignee: Abdul Qadeer
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-4012-version1.patch, Hadoop-4012-version10.patch, Hadoop-4012-version11.patch, Hadoop-4012-version2.patch, Hadoop-4012-version3.patch, Hadoop-4012-version4.patch, Hadoop-4012-version5.patch, Hadoop-4012-version6.patch, Hadoop-4012-version7.patch, Hadoop-4012-version8.patch, Hadoop-4012-version9.patch
>
>
> Hadoop assumes that if the input data is compressed, it can not be split (mainly due to the limitation of many codecs that they need the whole input stream to decompress successfully).  So in such a case, Hadoop prepares only one split per compressed file, where the lower split limit is at 0 while the upper limit is the end of the file.  The consequence of this decision is that, one compress file goes to a single mapper. Although it circumvents the limitation of codecs (as mentioned above) but reduces the parallelism substantially, as it was possible otherwise in case of splitting.
> BZip2 is a compression / De-Compression algorithm which does compression on blocks of data and later these compressed blocks can be decompressed independent of each other.  This is indeed an opportunity that instead of one BZip2 compressed file going to one mapper, we can process chunks of file in parallel.  The correctness criteria of such a processing is that for a bzip2 compressed file, each compressed block should be processed by only one mapper and ultimately all the blocks of the file should be processed.  (By processing we mean the actual utilization of that un-compressed data (coming out of the codecs) in a mapper).
> We are writing the code to implement this suggested functionality.  Although we have used bzip2 as an example, but we have tried to extend Hadoop's compression interfaces so that any other codecs with the same capability as that of bzip2, could easily use the splitting support.  The details of these changes will be posted when we submit the code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1116-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 28 23:42:55 2009
Return-Path: <common-issues-return-1116-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9203 invoked from network); 28 Aug 2009 23:42:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 28 Aug 2009 23:42:54 -0000
Received: (qmail 48080 invoked by uid 500); 28 Aug 2009 23:42:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48023 invoked by uid 500); 28 Aug 2009 23:42:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48013 invoked by uid 99); 28 Aug 2009 23:42:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 23:42:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Aug 2009 23:42:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 2282A234C4B6
	for <common-issues@hadoop.apache.org>; Fri, 28 Aug 2009 16:42:33 -0700 (PDT)
Message-ID: <1070082730.1251502953140.JavaMail.jira@brutus>
Date: Fri, 28 Aug 2009 16:42:33 -0700 (PDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6203) Improve error message when moving
 to trash fails due to quota issue
In-Reply-To: <1489131604.1250623814866.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749030#action_12749030 ] 

Boris Shkolnik commented on HADOOP-6203:
----------------------------------------

Currently there is no CLI facility in hadoop-common, so I resorted to manual testing:

>bin/hdfs dfs -lsr /
-rw-r--r--   1            0 2009-08-28 16:36 /file2
-rw-r--r--   1            0 2009-08-28 16:37 /file3
drwxr-xr-x   -            0 2009-08-28 16:37 /user
drwxr-xr-x   -            0 2009-08-28 16:37 /user/borya
drwxr-xr-x   -            0 2009-08-28 16:37 /user/borya/.Trash
drwx------   -          0 2009-08-28 16:37 /user/borya/.Trash/Current
-rw-r--r--   1            0 2009-08-28 16:36 /user/borya/.Trash/Current/file1

>bin/hdfs dfs -count -q /
           3              -3            none             inf            4            2                  0 hdfs://localhost:9000/

>bin/hdfs dfs -rm /file3
Problem with Trash. Consider using -skipTrash option   <<<<<<<<< THIS IS the new line
rm: Failed to move to trash: hdfs://localhost:9000/file3


> Improve error message when moving to trash fails due to quota issue
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6203
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6203
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Jakob Homan
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6203.patch, HADOOP-6203.patch
>
>
> HADOOP-6080 provided an option for deleting files even when overquota, but the error message that's returned in this situation is unhelpful and doesn't suggest skipTrash as a remediation:
> {noformat}$ hdfs -rmr /foo/bar/bat/boo
> rmr: Failed to move to trash:
> hdfs://cluster/foo/bar/bat/boo{noformat}
> In this situation, the error message should say there was a quote problem and suggest -skipTrash.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1117-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 29 01:40:55 2009
Return-Path: <common-issues-return-1117-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37101 invoked from network); 29 Aug 2009 01:40:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Aug 2009 01:40:54 -0000
Received: (qmail 91590 invoked by uid 500); 29 Aug 2009 01:40:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91558 invoked by uid 500); 29 Aug 2009 01:40:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91533 invoked by uid 99); 29 Aug 2009 01:40:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 29 Aug 2009 01:40:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 29 Aug 2009 01:40:52 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D478B234C044
	for <common-issues@hadoop.apache.org>; Fri, 28 Aug 2009 18:40:32 -0700 (PDT)
Message-ID: <1121508504.1251510032855.JavaMail.jira@brutus>
Date: Fri, 28 Aug 2009 18:40:32 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4012) Providing splitting support for
 bzip2 compressed files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749055#action_12749055 ] 

Chris Douglas commented on HADOOP-4012:
---------------------------------------

Fair point.

I will try to complete a review this weekend.

> Providing splitting support for bzip2 compressed files
> ------------------------------------------------------
>
>                 Key: HADOOP-4012
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4012
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Abdul Qadeer
>            Assignee: Abdul Qadeer
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-4012-version1.patch, Hadoop-4012-version10.patch, Hadoop-4012-version11.patch, Hadoop-4012-version2.patch, Hadoop-4012-version3.patch, Hadoop-4012-version4.patch, Hadoop-4012-version5.patch, Hadoop-4012-version6.patch, Hadoop-4012-version7.patch, Hadoop-4012-version8.patch, Hadoop-4012-version9.patch
>
>
> Hadoop assumes that if the input data is compressed, it can not be split (mainly due to the limitation of many codecs that they need the whole input stream to decompress successfully).  So in such a case, Hadoop prepares only one split per compressed file, where the lower split limit is at 0 while the upper limit is the end of the file.  The consequence of this decision is that, one compress file goes to a single mapper. Although it circumvents the limitation of codecs (as mentioned above) but reduces the parallelism substantially, as it was possible otherwise in case of splitting.
> BZip2 is a compression / De-Compression algorithm which does compression on blocks of data and later these compressed blocks can be decompressed independent of each other.  This is indeed an opportunity that instead of one BZip2 compressed file going to one mapper, we can process chunks of file in parallel.  The correctness criteria of such a processing is that for a bzip2 compressed file, each compressed block should be processed by only one mapper and ultimately all the blocks of the file should be processed.  (By processing we mean the actual utilization of that un-compressed data (coming out of the codecs) in a mapper).
> We are writing the code to implement this suggested functionality.  Although we have used bzip2 as an example, but we have tried to extend Hadoop's compression interfaces so that any other codecs with the same capability as that of bzip2, could easily use the splitting support.  The details of these changes will be posted when we submit the code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1118-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 29 03:34:55 2009
Return-Path: <common-issues-return-1118-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54596 invoked from network); 29 Aug 2009 03:34:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Aug 2009 03:34:55 -0000
Received: (qmail 32419 invoked by uid 500); 29 Aug 2009 03:34:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32363 invoked by uid 500); 29 Aug 2009 03:34:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32353 invoked by uid 99); 29 Aug 2009 03:34:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 29 Aug 2009 03:34:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 29 Aug 2009 03:34:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EF4CE234C044
	for <common-issues@hadoop.apache.org>; Fri, 28 Aug 2009 20:34:32 -0700 (PDT)
Message-ID: <538933666.1251516872948.JavaMail.jira@brutus>
Date: Fri, 28 Aug 2009 20:34:32 -0700 (PDT)
From: "John Heidemann (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4012) Providing splitting support for
 bzip2 compressed files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749063#action_12749063 ] 

John Heidemann commented on HADOOP-4012:
----------------------------------------

Thanks!

> Providing splitting support for bzip2 compressed files
> ------------------------------------------------------
>
>                 Key: HADOOP-4012
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4012
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Abdul Qadeer
>            Assignee: Abdul Qadeer
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-4012-version1.patch, Hadoop-4012-version10.patch, Hadoop-4012-version11.patch, Hadoop-4012-version2.patch, Hadoop-4012-version3.patch, Hadoop-4012-version4.patch, Hadoop-4012-version5.patch, Hadoop-4012-version6.patch, Hadoop-4012-version7.patch, Hadoop-4012-version8.patch, Hadoop-4012-version9.patch
>
>
> Hadoop assumes that if the input data is compressed, it can not be split (mainly due to the limitation of many codecs that they need the whole input stream to decompress successfully).  So in such a case, Hadoop prepares only one split per compressed file, where the lower split limit is at 0 while the upper limit is the end of the file.  The consequence of this decision is that, one compress file goes to a single mapper. Although it circumvents the limitation of codecs (as mentioned above) but reduces the parallelism substantially, as it was possible otherwise in case of splitting.
> BZip2 is a compression / De-Compression algorithm which does compression on blocks of data and later these compressed blocks can be decompressed independent of each other.  This is indeed an opportunity that instead of one BZip2 compressed file going to one mapper, we can process chunks of file in parallel.  The correctness criteria of such a processing is that for a bzip2 compressed file, each compressed block should be processed by only one mapper and ultimately all the blocks of the file should be processed.  (By processing we mean the actual utilization of that un-compressed data (coming out of the codecs) in a mapper).
> We are writing the code to implement this suggested functionality.  Although we have used bzip2 as an example, but we have tried to extend Hadoop's compression interfaces so that any other codecs with the same capability as that of bzip2, could easily use the splitting support.  The details of these changes will be posted when we submit the code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1119-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 29 13:53:55 2009
Return-Path: <common-issues-return-1119-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63475 invoked from network); 29 Aug 2009 13:53:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Aug 2009 13:53:55 -0000
Received: (qmail 64018 invoked by uid 500); 29 Aug 2009 13:53:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63960 invoked by uid 500); 29 Aug 2009 13:53:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63882 invoked by uid 99); 29 Aug 2009 13:53:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 29 Aug 2009 13:53:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 29 Aug 2009 13:53:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id DEEC3234C004
	for <common-issues@hadoop.apache.org>; Sat, 29 Aug 2009 06:53:32 -0700 (PDT)
Message-ID: <208111220.1251554012899.JavaMail.jira@brutus>
Date: Sat, 29 Aug 2009 06:53:32 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6103) Configuration clone constructor
 does not clone all the members.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749137#action_12749137 ] 

Hudson commented on HADOOP-6103:
--------------------------------

Integrated in Hadoop-Hdfs-trunk #65 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/65/])
    

> Configuration clone constructor does not clone all the members.
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6103
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6103
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.21.0
>
>         Attachments: patch-6103.txt
>
>
> Currently, Configuration(Configuration other) constructor does not clone all the members.
> It clones only resources, properties, overlay and finalParameters. It needs to clone loadDefaults, classLoader, defaultResources, quietmode.
> This resulted in bugs like HADOOP-4975

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1120-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 29 13:54:16 2009
Return-Path: <common-issues-return-1120-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63720 invoked from network); 29 Aug 2009 13:54:16 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 29 Aug 2009 13:54:16 -0000
Received: (qmail 64470 invoked by uid 500); 29 Aug 2009 13:54:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64409 invoked by uid 500); 29 Aug 2009 13:54:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64392 invoked by uid 99); 29 Aug 2009 13:54:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 29 Aug 2009 13:54:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 29 Aug 2009 13:54:14 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 4D626234C4BB
	for <common-issues@hadoop.apache.org>; Sat, 29 Aug 2009 06:53:33 -0700 (PDT)
Message-ID: <323761186.1251554013316.JavaMail.jira@brutus>
Date: Sat, 29 Aug 2009 06:53:33 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6201) FileSystem::ListStatus should throw
 FileNotFoundException
In-Reply-To: <1093368448.1250619614937.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749144#action_12749144 ] 

Hudson commented on HADOOP-6201:
--------------------------------

Integrated in Hadoop-Hdfs-trunk #65 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/65/])
    

> FileSystem::ListStatus should throw FileNotFoundException
> ---------------------------------------------------------
>
>                 Key: HADOOP-6201
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6201
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6201.patch
>
>
> As discussed in HDFS-538, it would be better for FileSystem and its implementations to throw FileNotFoundException when the file is not found, rather than returning null.  This will bring it in line with getFileStatus and, I expect, default user expectations.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1121-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 30 21:49:57 2009
Return-Path: <common-issues-return-1121-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60085 invoked from network); 30 Aug 2009 21:49:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Aug 2009 21:49:57 -0000
Received: (qmail 38243 invoked by uid 500); 30 Aug 2009 21:49:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38199 invoked by uid 500); 30 Aug 2009 21:49:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38189 invoked by uid 99); 30 Aug 2009 21:49:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 30 Aug 2009 21:49:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 30 Aug 2009 21:49:54 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EA80A234C004
	for <common-issues@hadoop.apache.org>; Sun, 30 Aug 2009 14:49:32 -0700 (PDT)
Message-ID: <598070000.1251668972953.JavaMail.jira@brutus>
Date: Sun, 30 Aug 2009 14:49:32 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4012) Providing splitting support for bzip2
 compressed files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-4012:
----------------------------------

    Status: Open  (was: Patch Available)

This is what we were looking for. Thank you.

* {{CBZip2InputStream::skipToNextMarker}} should add IOException (for bsR) to its throws list, not Exception, and throw IllegalArgumentException if markerBitLength is larger than 63. This permits the removal of the empty catch clauses in calls to this method. The javadoc should include a description of this behavior, and note that marker is the EOB delimiter (w/ {{@param}} javadoc directives). The javadoc may be self-evident, but it's a public API.
* {{updateProcessedByteCount}} and {{updateReportedByteCount}} are public methods, but they seem very, very specialized. The comment explains that the client may manipulate the compressed stream, but any client sophisticated enough to do that is likely bound to this class. Would making these methods protected make sense?
* This method:
{noformat}
+  private static void reportCRCError() throws IOException {
+    throw new IOException("crc error");
+  }
{noformat}
Seems unnecessary. If the intent is to add a hook, then the method should be protected and non-static. Otherwise, there's no reason not to simply throw the IOE at its callers.
* The API issue with InputStreamCreationResultant is clearer to me, now (I hadn't seen why a reader might need a modified start). Other than synchronizing on the codec before creating new streams (to avoid the race condition), I don't see a better way to do this without pushing other API changes. Unless someone has a better idea, I think documenting this requirement on {{SplittableCompressionCodec}} is sufficient for now (and making these methods synchronized in {{BZip2Codec}}).

> Providing splitting support for bzip2 compressed files
> ------------------------------------------------------
>
>                 Key: HADOOP-4012
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4012
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.21.0
>            Reporter: Abdul Qadeer
>            Assignee: Abdul Qadeer
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-4012-version1.patch, Hadoop-4012-version10.patch, Hadoop-4012-version11.patch, Hadoop-4012-version2.patch, Hadoop-4012-version3.patch, Hadoop-4012-version4.patch, Hadoop-4012-version5.patch, Hadoop-4012-version6.patch, Hadoop-4012-version7.patch, Hadoop-4012-version8.patch, Hadoop-4012-version9.patch
>
>
> Hadoop assumes that if the input data is compressed, it can not be split (mainly due to the limitation of many codecs that they need the whole input stream to decompress successfully).  So in such a case, Hadoop prepares only one split per compressed file, where the lower split limit is at 0 while the upper limit is the end of the file.  The consequence of this decision is that, one compress file goes to a single mapper. Although it circumvents the limitation of codecs (as mentioned above) but reduces the parallelism substantially, as it was possible otherwise in case of splitting.
> BZip2 is a compression / De-Compression algorithm which does compression on blocks of data and later these compressed blocks can be decompressed independent of each other.  This is indeed an opportunity that instead of one BZip2 compressed file going to one mapper, we can process chunks of file in parallel.  The correctness criteria of such a processing is that for a bzip2 compressed file, each compressed block should be processed by only one mapper and ultimately all the blocks of the file should be processed.  (By processing we mean the actual utilization of that un-compressed data (coming out of the codecs) in a mapper).
> We are writing the code to implement this suggested functionality.  Although we have used bzip2 as an example, but we have tried to extend Hadoop's compression interfaces so that any other codecs with the same capability as that of bzip2, could easily use the splitting support.  The details of these changes will be posted when we submit the code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1122-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 30 23:55:55 2009
Return-Path: <common-issues-return-1122-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78631 invoked from network); 30 Aug 2009 23:55:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 30 Aug 2009 23:55:55 -0000
Received: (qmail 90124 invoked by uid 500); 30 Aug 2009 23:55:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90060 invoked by uid 500); 30 Aug 2009 23:55:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90050 invoked by uid 99); 30 Aug 2009 23:55:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 30 Aug 2009 23:55:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 30 Aug 2009 23:55:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id BEC75234C004
	for <common-issues@hadoop.apache.org>; Sun, 30 Aug 2009 16:55:32 -0700 (PDT)
Message-ID: <1419273718.1251676532761.JavaMail.jira@brutus>
Date: Sun, 30 Aug 2009 16:55:32 -0700 (PDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4952) Improved files system interface for
 the application writer.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749343#action_12749343 ] 

Sanjay Radia commented on HADOOP-4952:
--------------------------------------

Doug, I would like to keep them in a separate class:
 * Lower layers that need these params should not rely on a higher layer like FileContext. This is layer violation.
 * The config variable layer cleanup jira (HDFS-531) needs such a class. That Jira may get in before this one - this further argues for the
separation.
 * We tend to create huge classes with inner classes that are accessed outside. I dislike that. Smaller classes are easier to 
maintain. Further i rather reply on creating sub-packages if we need lexical protection.

> Improved files system interface for the application writer.
> -----------------------------------------------------------
>
>                 Key: HADOOP-4952
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4952
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.21.0
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: FileContext3.patch, FileContext5.patch, Files.java, Files.java, FilesContext1.patch, FilesContext2.patch
>
>
> Currently the FIleSystem interface serves two purposes:
> - an application writer's interface for using the Hadoop file system
> - a file system implementer's interface (e.g. hdfs, local file system, kfs, etc)
> This Jira proposes that we provide a simpler interfaces for the application writer and leave the FilsSystem  interface for the implementer of a filesystem.
> - Filesystem interface  has a  confusing set of methods for the application writer
> - We could make it easier to take advantage of the URI file naming
> ** Current approach is to get FileSystem instance by supplying the URI and then access that name space. It is consistent for the FileSystem instance to not accept URIs for other schemes, but we can do better.
> ** The special copyFromLocalFIle can be generalized as a  copyFile where the src or target can be generalized to any URI, including the local one.
> ** The proposed scheme (below) simplifies this.
> -	The client side config can be simplified. 
> ** New config() by default uses the default config. Since this is the common usage pattern, one should not need to always pass the config as a parameter when accessing the file system.  
> -	
> ** It does not handle multiple file systems too well. Today a site.xml is derived from a single Hadoop cluster. This does not make sense for multiple Hadoop clusters which may have different defaults.
> ** Further one should need very little to configure the client side:
> *** Default files system.
> *** Block size 
> *** Replication factor
> *** Scheme to class mapping
> ** It should be possible to take Blocksize and replication factors defaults from the target file system, rather then the client size config.  I am not suggesting we don't allow setting client side defaults, but most clients do not care and would find it simpler to take the defaults for their systems  from the target file system. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1123-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 00:03:55 2009
Return-Path: <common-issues-return-1123-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79664 invoked from network); 31 Aug 2009 00:03:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 00:03:55 -0000
Received: (qmail 91655 invoked by uid 500); 31 Aug 2009 00:03:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91597 invoked by uid 500); 31 Aug 2009 00:03:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91587 invoked by uid 99); 31 Aug 2009 00:03:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 00:03:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 00:03:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id B5923234C004
	for <common-issues@hadoop.apache.org>; Sun, 30 Aug 2009 17:03:32 -0700 (PDT)
Message-ID: <1865041072.1251677012736.JavaMail.jira@brutus>
Date: Sun, 30 Aug 2009 17:03:32 -0700 (PDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4952) Improved files system interface for
 the application writer.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749344#action_12749344 ] 

Sanjay Radia commented on HADOOP-4952:
--------------------------------------

The proposed FileContext API, in addition to being a more convenient interface for application writer,  will also simplify the FileSystem layer, freeing that layer from dealing with notions of default filesystem (i.e. /) and wd and also freeing it from having to deal with default values of FS variables like default block size (BS), replication factor (RF), umask, etc

The original design was to allow options 1 and 2 below for config values. A few folks felt that we needed to retain the notion of client-side defaults (option 3 below).:
 # Specify the desired values as parameters in the methods such as create. [as before]
 # Use SS defaults - different FS and their deployments can have their own defaults and also frees the admin from distributing the default config to all client nodes. [New]
 # Use client side defaults derived from the config. [as before]
The uploaded patch attempts to give all 3 options.

After further thinking, I am proposing that we support only options 1 and 2; not 3.
Reasons:
 * Simpler config 
 * Most folks are going to be happy with SS defaults. The admin has set the defaults based on various considerations including cluster size etc. Further in a federated environment, each cluster may have different defaults that a single client-side config cannot accommodate.
 * Turns out that supporting client-side defaults (option 3) complicates the implementation and forces the FileSystem layer to deal with default values for BS, RF etc.   The primary reason for that is that the original design of FileSystem allows *each distinct FileSystem class* to have its own client-side defaults for BS, RF!!
 ** If you look at my patch, you will realize that it is incorrect. When resolving a fully qualified URI it needs to look up the defaults for that specific FileSystem. The patch incorrectly uses the defaults for the default filesystem. Turns out the patch for Symbolic links (HDFS-245) also makes the same mistake when it follows symbolic links to foreign file systems.
 ** Yes this is fixable, but will require that the FileSystem layer retain notion of defaults for BS, RF. It is much simpler to let the FileContext be the only layer that needs to deal with the context for /, wd and defaults for BS, RF, etc.
 * I believe the notion of client-side defaults is bad idea.  SS defaults are sufficient and furthermore they deal with multiple deployments of the same FS while client-side defaults do not deal with that situation. 


> Improved files system interface for the application writer.
> -----------------------------------------------------------
>
>                 Key: HADOOP-4952
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4952
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.21.0
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: FileContext3.patch, FileContext5.patch, Files.java, Files.java, FilesContext1.patch, FilesContext2.patch
>
>
> Currently the FIleSystem interface serves two purposes:
> - an application writer's interface for using the Hadoop file system
> - a file system implementer's interface (e.g. hdfs, local file system, kfs, etc)
> This Jira proposes that we provide a simpler interfaces for the application writer and leave the FilsSystem  interface for the implementer of a filesystem.
> - Filesystem interface  has a  confusing set of methods for the application writer
> - We could make it easier to take advantage of the URI file naming
> ** Current approach is to get FileSystem instance by supplying the URI and then access that name space. It is consistent for the FileSystem instance to not accept URIs for other schemes, but we can do better.
> ** The special copyFromLocalFIle can be generalized as a  copyFile where the src or target can be generalized to any URI, including the local one.
> ** The proposed scheme (below) simplifies this.
> -	The client side config can be simplified. 
> ** New config() by default uses the default config. Since this is the common usage pattern, one should not need to always pass the config as a parameter when accessing the file system.  
> -	
> ** It does not handle multiple file systems too well. Today a site.xml is derived from a single Hadoop cluster. This does not make sense for multiple Hadoop clusters which may have different defaults.
> ** Further one should need very little to configure the client side:
> *** Default files system.
> *** Block size 
> *** Replication factor
> *** Scheme to class mapping
> ** It should be possible to take Blocksize and replication factors defaults from the target file system, rather then the client size config.  I am not suggesting we don't allow setting client side defaults, but most clients do not care and would find it simpler to take the defaults for their systems  from the target file system. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1124-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 00:33:57 2009
Return-Path: <common-issues-return-1124-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85629 invoked from network); 31 Aug 2009 00:33:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 00:33:56 -0000
Received: (qmail 8210 invoked by uid 500); 31 Aug 2009 00:33:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8180 invoked by uid 500); 31 Aug 2009 00:33:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8170 invoked by uid 99); 31 Aug 2009 00:33:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 00:33:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 00:33:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CC31F234C051
	for <common-issues@hadoop.apache.org>; Sun, 30 Aug 2009 17:33:32 -0700 (PDT)
Message-ID: <9646781.1251678812835.JavaMail.jira@brutus>
Date: Sun, 30 Aug 2009 17:33:32 -0700 (PDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6223) New improved FileSystem interface for
 those implementing new files systems.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

New improved FileSystem interface for those implementing new files systems.
---------------------------------------------------------------------------

                 Key: HADOOP-6223
                 URL: https://issues.apache.org/jira/browse/HADOOP-6223
             Project: Hadoop Common
          Issue Type: Sub-task
            Reporter: Sanjay Radia


The FileContext API (HADOOP-4952) provides an improved interface for the application writer.
This lets us simplify the FileSystem API since it will no longer need to deal with notions of default filesystem [ / ],  wd, and config
defaults for blocksize, replication factor etc. Further it will not need the many overloaded methods for create() and open() since
the FileContext API provides that convenience.
The FileSystem API can be simplified and can now be restricted to those implementing new file systems.


This jira proposes that we create new file system API,  and deprecate FileSystem API after a few releases.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1125-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 00:37:57 2009
Return-Path: <common-issues-return-1125-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86001 invoked from network); 31 Aug 2009 00:37:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 00:37:57 -0000
Received: (qmail 11474 invoked by uid 500); 31 Aug 2009 00:37:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11411 invoked by uid 500); 31 Aug 2009 00:37:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11401 invoked by uid 99); 31 Aug 2009 00:37:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 00:37:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 00:37:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id B0142234C004
	for <common-issues@hadoop.apache.org>; Sun, 30 Aug 2009 17:37:32 -0700 (PDT)
Message-ID: <951439942.1251679052705.JavaMail.jira@brutus>
Date: Sun, 30 Aug 2009 17:37:32 -0700 (PDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6223) New improved FileSystem interface
 for those implementing new files systems.
In-Reply-To: <9646781.1251678812835.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749348#action_12749348 ] 

Sanjay Radia commented on HADOOP-6223:
--------------------------------------

I propose that we call the new class AbstractFileSystem and deprecate the FileSystem class in a few releases after folks have
migrated to FileContext.

Note: one could deprecate parts of FileSystem and them delete or make-package private its methods. It is simpler
to simply create a new class called AbstractFileSystem (or VirtualFileSystem a la Unix).

> New improved FileSystem interface for those implementing new files systems.
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-6223
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6223
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Sanjay Radia
>
> The FileContext API (HADOOP-4952) provides an improved interface for the application writer.
> This lets us simplify the FileSystem API since it will no longer need to deal with notions of default filesystem [ / ],  wd, and config
> defaults for blocksize, replication factor etc. Further it will not need the many overloaded methods for create() and open() since
> the FileContext API provides that convenience.
> The FileSystem API can be simplified and can now be restricted to those implementing new file systems.
> This jira proposes that we create new file system API,  and deprecate FileSystem API after a few releases.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1126-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 03:47:57 2009
Return-Path: <common-issues-return-1126-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17070 invoked from network); 31 Aug 2009 03:47:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 03:47:57 -0000
Received: (qmail 81717 invoked by uid 500); 31 Aug 2009 03:47:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81659 invoked by uid 500); 31 Aug 2009 03:47:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81645 invoked by uid 99); 31 Aug 2009 03:47:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 03:47:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 03:47:54 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CAB55234C004
	for <common-issues@hadoop.apache.org>; Sun, 30 Aug 2009 20:47:32 -0700 (PDT)
Message-ID: <1286563855.1251690452814.JavaMail.jira@brutus>
Date: Sun, 30 Aug 2009 20:47:32 -0700 (PDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-4952) Improved files system interface for
 the application writer.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-4952:
---------------------------------

    Attachment: FileContext6.patch

> Improved files system interface for the application writer.
> -----------------------------------------------------------
>
>                 Key: HADOOP-4952
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4952
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.21.0
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: FileContext3.patch, FileContext5.patch, FileContext6.patch, Files.java, Files.java, FilesContext1.patch, FilesContext2.patch
>
>
> Currently the FIleSystem interface serves two purposes:
> - an application writer's interface for using the Hadoop file system
> - a file system implementer's interface (e.g. hdfs, local file system, kfs, etc)
> This Jira proposes that we provide a simpler interfaces for the application writer and leave the FilsSystem  interface for the implementer of a filesystem.
> - Filesystem interface  has a  confusing set of methods for the application writer
> - We could make it easier to take advantage of the URI file naming
> ** Current approach is to get FileSystem instance by supplying the URI and then access that name space. It is consistent for the FileSystem instance to not accept URIs for other schemes, but we can do better.
> ** The special copyFromLocalFIle can be generalized as a  copyFile where the src or target can be generalized to any URI, including the local one.
> ** The proposed scheme (below) simplifies this.
> -	The client side config can be simplified. 
> ** New config() by default uses the default config. Since this is the common usage pattern, one should not need to always pass the config as a parameter when accessing the file system.  
> -	
> ** It does not handle multiple file systems too well. Today a site.xml is derived from a single Hadoop cluster. This does not make sense for multiple Hadoop clusters which may have different defaults.
> ** Further one should need very little to configure the client side:
> *** Default files system.
> *** Block size 
> *** Replication factor
> *** Scheme to class mapping
> ** It should be possible to take Blocksize and replication factors defaults from the target file system, rather then the client size config.  I am not suggesting we don't allow setting client side defaults, but most clients do not care and would find it simpler to take the defaults for their systems  from the target file system. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1127-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 04:51:55 2009
Return-Path: <common-issues-return-1127-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30464 invoked from network); 31 Aug 2009 04:51:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 04:51:54 -0000
Received: (qmail 2599 invoked by uid 500); 31 Aug 2009 04:51:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2541 invoked by uid 500); 31 Aug 2009 04:51:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2524 invoked by uid 99); 31 Aug 2009 04:51:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 04:51:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 04:51:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CCC31234C044
	for <common-issues@hadoop.apache.org>; Sun, 30 Aug 2009 21:51:32 -0700 (PDT)
Message-ID: <107649796.1251694292837.JavaMail.jira@brutus>
Date: Sun, 30 Aug 2009 21:51:32 -0700 (PDT)
From: "Jothi Padmanabhan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6224) Add some utility methods to Text and
 DataOutputBuffer to be used by MAPREDUCE-318
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Add some utility methods to Text and DataOutputBuffer to be used by MAPREDUCE-318
---------------------------------------------------------------------------------

                 Key: HADOOP-6224
                 URL: https://issues.apache.org/jira/browse/HADOOP-6224
             Project: Hadoop Common
          Issue Type: Improvement
          Components: io
            Reporter: Jothi Padmanabhan
            Assignee: Jothi Padmanabhan
             Fix For: 0.21.0


MAPREDUCE-318 needs some minor extensions to the Text and the DataOutputBuffer classes. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1128-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 04:53:57 2009
Return-Path: <common-issues-return-1128-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31427 invoked from network); 31 Aug 2009 04:53:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 04:53:56 -0000
Received: (qmail 4898 invoked by uid 500); 31 Aug 2009 04:53:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4819 invoked by uid 500); 31 Aug 2009 04:53:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4799 invoked by uid 99); 31 Aug 2009 04:53:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 04:53:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 04:53:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id B4DB5234C004
	for <common-issues@hadoop.apache.org>; Sun, 30 Aug 2009 21:53:32 -0700 (PDT)
Message-ID: <1249427112.1251694412725.JavaMail.jira@brutus>
Date: Sun, 30 Aug 2009 21:53:32 -0700 (PDT)
From: "Jothi Padmanabhan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6224) Add some utility methods to Text and
 DataOutputBuffer to be used by MAPREDUCE-318
In-Reply-To: <107649796.1251694292837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jothi Padmanabhan updated HADOOP-6224:
--------------------------------------

    Attachment: common.patch

Straight Forward Patch

> Add some utility methods to Text and DataOutputBuffer to be used by MAPREDUCE-318
> ---------------------------------------------------------------------------------
>
>                 Key: HADOOP-6224
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6224
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Jothi Padmanabhan
>            Assignee: Jothi Padmanabhan
>             Fix For: 0.21.0
>
>         Attachments: common.patch
>
>
> MAPREDUCE-318 needs some minor extensions to the Text and the DataOutputBuffer classes. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1129-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 04:53:57 2009
Return-Path: <common-issues-return-1129-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31430 invoked from network); 31 Aug 2009 04:53:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 04:53:56 -0000
Received: (qmail 4924 invoked by uid 500); 31 Aug 2009 04:53:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4821 invoked by uid 500); 31 Aug 2009 04:53:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4800 invoked by uid 99); 31 Aug 2009 04:53:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 04:53:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 04:53:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id C1348234C046
	for <common-issues@hadoop.apache.org>; Sun, 30 Aug 2009 21:53:32 -0700 (PDT)
Message-ID: <1760455143.1251694412790.JavaMail.jira@brutus>
Date: Sun, 30 Aug 2009 21:53:32 -0700 (PDT)
From: "Jothi Padmanabhan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6224) Add some utility methods to Text and
 DataOutputBuffer to be used by MAPREDUCE-318
In-Reply-To: <107649796.1251694292837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jothi Padmanabhan updated HADOOP-6224:
--------------------------------------

    Status: Patch Available  (was: Open)

> Add some utility methods to Text and DataOutputBuffer to be used by MAPREDUCE-318
> ---------------------------------------------------------------------------------
>
>                 Key: HADOOP-6224
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6224
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Jothi Padmanabhan
>            Assignee: Jothi Padmanabhan
>             Fix For: 0.21.0
>
>         Attachments: common.patch
>
>
> MAPREDUCE-318 needs some minor extensions to the Text and the DataOutputBuffer classes. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1130-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 05:15:55 2009
Return-Path: <common-issues-return-1130-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34763 invoked from network); 31 Aug 2009 05:15:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 05:15:54 -0000
Received: (qmail 17385 invoked by uid 500); 31 Aug 2009 05:15:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17327 invoked by uid 500); 31 Aug 2009 05:15:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17317 invoked by uid 99); 31 Aug 2009 05:15:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 05:15:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 05:15:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id F24C6234C004
	for <common-issues@hadoop.apache.org>; Sun, 30 Aug 2009 22:15:32 -0700 (PDT)
Message-ID: <1187669439.1251695732913.JavaMail.jira@brutus>
Date: Sun, 30 Aug 2009 22:15:32 -0700 (PDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6224) Add some utility methods to Text
 and DataOutputBuffer to be used by MAPREDUCE-318
In-Reply-To: <107649796.1251694292837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749382#action_12749382 ] 

Hadoop QA commented on HADOOP-6224:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12418118/common.patch
  against trunk revision 808415.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/632/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/632/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/632/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/632/console

This message is automatically generated.

> Add some utility methods to Text and DataOutputBuffer to be used by MAPREDUCE-318
> ---------------------------------------------------------------------------------
>
>                 Key: HADOOP-6224
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6224
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Jothi Padmanabhan
>            Assignee: Jothi Padmanabhan
>             Fix For: 0.21.0
>
>         Attachments: common.patch
>
>
> MAPREDUCE-318 needs some minor extensions to the Text and the DataOutputBuffer classes. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1131-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 05:19:56 2009
Return-Path: <common-issues-return-1131-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36308 invoked from network); 31 Aug 2009 05:19:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 05:19:56 -0000
Received: (qmail 18613 invoked by uid 500); 31 Aug 2009 05:19:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18581 invoked by uid 500); 31 Aug 2009 05:19:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18571 invoked by uid 99); 31 Aug 2009 05:19:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 05:19:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 05:19:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id BA91E234C004
	for <common-issues@hadoop.apache.org>; Sun, 30 Aug 2009 22:19:32 -0700 (PDT)
Message-ID: <1340674827.1251695972749.JavaMail.jira@brutus>
Date: Sun, 30 Aug 2009 22:19:32 -0700 (PDT)
From: "Jothi Padmanabhan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6224) Add some utility methods to Text
 and DataOutputBuffer to be used by MAPREDUCE-318
In-Reply-To: <107649796.1251694292837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749383#action_12749383 ] 

Jothi Padmanabhan commented on HADOOP-6224:
-------------------------------------------

bq, Please justify why no new tests are needed for this patch.

This functionality will be comprehensively tested by MAPREDUCE-318

> Add some utility methods to Text and DataOutputBuffer to be used by MAPREDUCE-318
> ---------------------------------------------------------------------------------
>
>                 Key: HADOOP-6224
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6224
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Jothi Padmanabhan
>            Assignee: Jothi Padmanabhan
>             Fix For: 0.21.0
>
>         Attachments: common.patch
>
>
> MAPREDUCE-318 needs some minor extensions to the Text and the DataOutputBuffer classes. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1132-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 05:35:54 2009
Return-Path: <common-issues-return-1132-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38186 invoked from network); 31 Aug 2009 05:35:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 05:35:54 -0000
Received: (qmail 23811 invoked by uid 500); 31 Aug 2009 05:35:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23756 invoked by uid 500); 31 Aug 2009 05:35:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23746 invoked by uid 99); 31 Aug 2009 05:35:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 05:35:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 05:35:52 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id CD42B234C055
	for <common-issues@hadoop.apache.org>; Sun, 30 Aug 2009 22:35:32 -0700 (PDT)
Message-ID: <948310371.1251696932839.JavaMail.jira@brutus>
Date: Sun, 30 Aug 2009 22:35:32 -0700 (PDT)
From: "Jay Booth (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6196) sync(0); next() breaks SequenceFile
In-Reply-To: <1782929485.1250413814845.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jay Booth updated HADOOP-6196:
------------------------------

    Attachment:     (was: sync-bug.patch)

> sync(0); next() breaks SequenceFile
> -----------------------------------
>
>                 Key: HADOOP-6196
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6196
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jay Booth
>         Attachments: sync-bug.patch
>
>
> Currently, the end of the SequenceFile header is a sync block that isn't prefaced with SYNC_ESCAPE.  This means that sync(0) followed by next() fails.  Patch w/ test attached, bumps VERSION from 6 to 7.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1133-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 05:35:56 2009
Return-Path: <common-issues-return-1133-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38243 invoked from network); 31 Aug 2009 05:35:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 05:35:56 -0000
Received: (qmail 23911 invoked by uid 500); 31 Aug 2009 05:35:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23855 invoked by uid 500); 31 Aug 2009 05:35:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23830 invoked by uid 99); 31 Aug 2009 05:35:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 05:35:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 05:35:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id BBCE0234C004
	for <common-issues@hadoop.apache.org>; Sun, 30 Aug 2009 22:35:32 -0700 (PDT)
Message-ID: <585272477.1251696932754.JavaMail.jira@brutus>
Date: Sun, 30 Aug 2009 22:35:32 -0700 (PDT)
From: "Jay Booth (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6196) sync(0); next() breaks SequenceFile
In-Reply-To: <1782929485.1250413814845.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jay Booth updated HADOOP-6196:
------------------------------

    Attachment: sync-bug.patch

So, it turns out that SequenceFile.Writer generates its sync block and headers statically at classloader time, so if it generates the 'right' binary data, it'll evade this bug.  I can't reproduce the bug reliably without running the test multiple times in a row by hand, unless someone knows a way to re-execute static code in a running jvm.  It occurs about 2/3 of the time or so.  Modifying SequenceFile to expose those values for writing by a test seems like overkill... should we just leave as is?  If the build runs frequently enough, it'll catch regressions eventually, right?  All the other ideas I have are hacky.

Issues with JUnit 4 and whitespace in the patch have been addressed.

> sync(0); next() breaks SequenceFile
> -----------------------------------
>
>                 Key: HADOOP-6196
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6196
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jay Booth
>         Attachments: sync-bug.patch
>
>
> Currently, the end of the SequenceFile header is a sync block that isn't prefaced with SYNC_ESCAPE.  This means that sync(0) followed by next() fails.  Patch w/ test attached, bumps VERSION from 6 to 7.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1134-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 06:39:55 2009
Return-Path: <common-issues-return-1134-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49615 invoked from network); 31 Aug 2009 06:39:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 06:39:55 -0000
Received: (qmail 62758 invoked by uid 500); 31 Aug 2009 06:39:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62708 invoked by uid 500); 31 Aug 2009 06:39:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62697 invoked by uid 99); 31 Aug 2009 06:39:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 06:39:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 06:39:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 37B93234C004
	for <common-issues@hadoop.apache.org>; Sun, 30 Aug 2009 23:39:33 -0700 (PDT)
Message-ID: <2070136732.1251700773213.JavaMail.jira@brutus>
Date: Sun, 30 Aug 2009 23:39:33 -0700 (PDT)
From: "Ravi Phulari (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6216) HDFS Web UI displays comments from
 dfs.exclude file and counts them as dead nodes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Phulari updated HADOOP-6216:
---------------------------------

    Attachment: HADOOP-6216.patch

> HDFS Web UI displays comments from dfs.exclude file and counts them as dead nodes
> ---------------------------------------------------------------------------------
>
>                 Key: HADOOP-6216
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6216
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>            Reporter: Jim Huang
>            Assignee: Ravi Phulari
>         Attachments: HADOOP-6216.patch, HADOOP-6216.patch, HDFS-1.patch
>
>
> I am putting comments in dfs.exclude file such as:
> {noformat}
>  # 32 GB memory upgrades
> {noformat}
> HDFS Web UI counts each word in the commented line as a dead node.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1135-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 07:57:56 2009
Return-Path: <common-issues-return-1135-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72878 invoked from network); 31 Aug 2009 07:57:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 07:57:56 -0000
Received: (qmail 18175 invoked by uid 500); 31 Aug 2009 07:57:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18121 invoked by uid 500); 31 Aug 2009 07:57:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18111 invoked by uid 99); 31 Aug 2009 07:57:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 07:57:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 07:57:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id AE153234C004
	for <common-issues@hadoop.apache.org>; Mon, 31 Aug 2009 00:57:32 -0700 (PDT)
Message-ID: <1152801114.1251705452698.JavaMail.jira@brutus>
Date: Mon, 31 Aug 2009 00:57:32 -0700 (PDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6224) Add some utility methods to Text and
 DataOutputBuffer to be used by MAPREDUCE-318
In-Reply-To: <107649796.1251694292837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6224:
--------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I just committed this. Thanks, Jothi and Arun!

> Add some utility methods to Text and DataOutputBuffer to be used by MAPREDUCE-318
> ---------------------------------------------------------------------------------
>
>                 Key: HADOOP-6224
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6224
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Jothi Padmanabhan
>            Assignee: Jothi Padmanabhan
>             Fix For: 0.21.0
>
>         Attachments: common.patch
>
>
> MAPREDUCE-318 needs some minor extensions to the Text and the DataOutputBuffer classes. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1136-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 08:11:57 2009
Return-Path: <common-issues-return-1136-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82423 invoked from network); 31 Aug 2009 08:11:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 08:11:56 -0000
Received: (qmail 33342 invoked by uid 500); 31 Aug 2009 08:11:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33285 invoked by uid 500); 31 Aug 2009 08:11:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33275 invoked by uid 99); 31 Aug 2009 08:11:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 08:11:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 08:11:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D9A08234C004
	for <common-issues@hadoop.apache.org>; Mon, 31 Aug 2009 01:11:32 -0700 (PDT)
Message-ID: <286435704.1251706292874.JavaMail.jira@brutus>
Date: Mon, 31 Aug 2009 01:11:32 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6224) Add some utility methods to Text
 and DataOutputBuffer to be used by MAPREDUCE-318
In-Reply-To: <107649796.1251694292837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749414#action_12749414 ] 

Hudson commented on HADOOP-6224:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #3 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/3/])
    . Adds methods to read strings safely, makes the Buffer class in DataOutputBuffer public, and introduces public constructors there. These changes are required for MAPREDUCE-318. Contributed by Jothi Padmanabhan and Arun Murthy.


> Add some utility methods to Text and DataOutputBuffer to be used by MAPREDUCE-318
> ---------------------------------------------------------------------------------
>
>                 Key: HADOOP-6224
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6224
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Jothi Padmanabhan
>            Assignee: Jothi Padmanabhan
>             Fix For: 0.21.0
>
>         Attachments: common.patch
>
>
> MAPREDUCE-318 needs some minor extensions to the Text and the DataOutputBuffer classes. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1137-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 10:24:57 2009
Return-Path: <common-issues-return-1137-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24452 invoked from network); 31 Aug 2009 10:24:57 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 10:24:57 -0000
Received: (qmail 79103 invoked by uid 500); 31 Aug 2009 10:24:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79081 invoked by uid 500); 31 Aug 2009 10:24:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78913 invoked by uid 99); 31 Aug 2009 10:24:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 10:24:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 10:24:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id D71FC234C052
	for <common-issues@hadoop.apache.org>; Mon, 31 Aug 2009 03:24:32 -0700 (PDT)
Message-ID: <2058306579.1251714272880.JavaMail.jira@brutus>
Date: Mon, 31 Aug 2009 03:24:32 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6077) test-patch target does not validate
 that forrest docs are built correctly
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749457#action_12749457 ] 

Giridharan Kesavan commented on HADOOP-6077:
--------------------------------------------

this issue is fixed with https://issues.apache.org/jira/browse/HADOOP-6124


> test-patch target does not validate that forrest docs are built correctly
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6077
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6077
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Hemanth Yamijala
>            Assignee: Giridharan Kesavan
>
> The test-patch target does not explicitly check if the new patch breaks forrest documentation. It actually runs the 'tar' target while checking for javac warnings, but does not seem to validate if the target ran successfully or not.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1138-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 10:26:56 2009
Return-Path: <common-issues-return-1138-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24966 invoked from network); 31 Aug 2009 10:26:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 10:26:56 -0000
Received: (qmail 82583 invoked by uid 500); 31 Aug 2009 10:26:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82550 invoked by uid 500); 31 Aug 2009 10:26:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82538 invoked by uid 99); 31 Aug 2009 10:26:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 10:26:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 10:26:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EF26F234C044
	for <common-issues@hadoop.apache.org>; Mon, 31 Aug 2009 03:26:32 -0700 (PDT)
Message-ID: <2065267188.1251714392978.JavaMail.jira@brutus>
Date: Mon, 31 Aug 2009 03:26:32 -0700 (PDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6077) test-patch target does not validate
 that forrest docs are built correctly
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan resolved HADOOP-6077.
----------------------------------------

    Resolution: Duplicate

marking it as duplicate.

> test-patch target does not validate that forrest docs are built correctly
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6077
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6077
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Hemanth Yamijala
>            Assignee: Giridharan Kesavan
>
> The test-patch target does not explicitly check if the new patch breaks forrest documentation. It actually runs the 'tar' target while checking for javac warnings, but does not seem to validate if the target ran successfully or not.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1139-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 11:10:56 2009
Return-Path: <common-issues-return-1139-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33728 invoked from network); 31 Aug 2009 11:10:56 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 11:10:56 -0000
Received: (qmail 50262 invoked by uid 500); 31 Aug 2009 11:10:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50150 invoked by uid 500); 31 Aug 2009 11:10:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50108 invoked by uid 99); 31 Aug 2009 11:10:56 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 11:10:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 11:10:54 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 7186C234C4C9
	for <common-issues@hadoop.apache.org>; Mon, 31 Aug 2009 04:10:34 -0700 (PDT)
Message-ID: <1200992474.1251717034463.JavaMail.jira@brutus>
Date: Mon, 31 Aug 2009 04:10:34 -0700 (PDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6224) Add some utility methods to Text
 and DataOutputBuffer to be used by MAPREDUCE-318
In-Reply-To: <107649796.1251694292837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749465#action_12749465 ] 

Hudson commented on HADOOP-6224:
--------------------------------

Integrated in Hadoop-Common-trunk #80 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/80/])
    . Adds methods to read strings safely, makes the Buffer class in DataOutputBuffer public, and introduces public constructors there. These changes are required for MAPREDUCE-318. Contributed by Jothi Padmanabhan and Arun Murthy.


> Add some utility methods to Text and DataOutputBuffer to be used by MAPREDUCE-318
> ---------------------------------------------------------------------------------
>
>                 Key: HADOOP-6224
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6224
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Jothi Padmanabhan
>            Assignee: Jothi Padmanabhan
>             Fix For: 0.21.0
>
>         Attachments: common.patch
>
>
> MAPREDUCE-318 needs some minor extensions to the Text and the DataOutputBuffer classes. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1140-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 11:56:54 2009
Return-Path: <common-issues-return-1140-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43102 invoked from network); 31 Aug 2009 11:56:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 11:56:54 -0000
Received: (qmail 10102 invoked by uid 500); 31 Aug 2009 11:56:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10053 invoked by uid 500); 31 Aug 2009 11:56:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10043 invoked by uid 99); 31 Aug 2009 11:56:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 11:56:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 11:56:52 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id A9DE9234C004
	for <common-issues@hadoop.apache.org>; Mon, 31 Aug 2009 04:56:32 -0700 (PDT)
Message-ID: <610788083.1251719792691.JavaMail.jira@brutus>
Date: Mon, 31 Aug 2009 04:56:32 -0700 (PDT)
From: "Jothi Padmanabhan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6224) Add some utility methods to Text
 and DataOutputBuffer to be used by MAPREDUCE-318
In-Reply-To: <107649796.1251694292837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749473#action_12749473 ] 

Jothi Padmanabhan commented on HADOOP-6224:
-------------------------------------------

I ran HDFS and MapRed tests using this jar, they all ran fine except for the known test failures, TestRecoveryManager and TestKillSubProcessesWithLinuxTaskController

> Add some utility methods to Text and DataOutputBuffer to be used by MAPREDUCE-318
> ---------------------------------------------------------------------------------
>
>                 Key: HADOOP-6224
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6224
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Jothi Padmanabhan
>            Assignee: Jothi Padmanabhan
>             Fix For: 0.21.0
>
>         Attachments: common.patch
>
>
> MAPREDUCE-318 needs some minor extensions to the Text and the DataOutputBuffer classes. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1141-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 18:15:54 2009
Return-Path: <common-issues-return-1141-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49826 invoked from network); 31 Aug 2009 18:15:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 18:15:54 -0000
Received: (qmail 81967 invoked by uid 500); 31 Aug 2009 18:15:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81931 invoked by uid 500); 31 Aug 2009 18:15:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81921 invoked by uid 99); 31 Aug 2009 18:15:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 18:15:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 18:15:52 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id AFBEB234C004
	for <common-issues@hadoop.apache.org>; Mon, 31 Aug 2009 11:15:32 -0700 (PDT)
Message-ID: <643411594.1251742532705.JavaMail.jira@brutus>
Date: Mon, 31 Aug 2009 11:15:32 -0700 (PDT)
From: "Corinne Chandel (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6217) Hadoop Doc Split: Common Docs
In-Reply-To: <13494760.1251318719318.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Corinne Chandel updated HADOOP-6217:
------------------------------------

    Priority: Blocker  (was: Minor)

> Hadoop Doc Split: Common Docs
> -----------------------------
>
>                 Key: HADOOP-6217
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6217
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>    Affects Versions: 0.21.0
>            Reporter: Corinne Chandel
>            Assignee: Owen O'Malley
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6217-common.patch, Hadoop-Doc-Split.doc
>
>
> Hadoop Doc Split: Common Docs
> Please note that I am unable to directly check all of the new links. Some links may break and will need to be updated.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1142-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 18:25:54 2009
Return-Path: <common-issues-return-1142-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58226 invoked from network); 31 Aug 2009 18:25:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 18:25:54 -0000
Received: (qmail 93408 invoked by uid 500); 31 Aug 2009 18:25:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93349 invoked by uid 500); 31 Aug 2009 18:25:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93336 invoked by uid 99); 31 Aug 2009 18:25:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 18:25:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 18:25:52 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id BC882234C044
	for <common-issues@hadoop.apache.org>; Mon, 31 Aug 2009 11:25:32 -0700 (PDT)
Message-ID: <1694332292.1251743132771.JavaMail.jira@brutus>
Date: Mon, 31 Aug 2009 11:25:32 -0700 (PDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6168) HADOOP_HEAPSIZE cannot be done
 per-server easily
In-Reply-To: <1003483123.1248379634876.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Allen Wittenauer resolved HADOOP-6168.
--------------------------------------

    Resolution: Duplicate

agreed that this a dupe.

> HADOOP_HEAPSIZE cannot be done per-server easily
> ------------------------------------------------
>
>                 Key: HADOOP-6168
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6168
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.18.3
>            Reporter: Allen Wittenauer
>
> The hadoop script forces a heap that cannot be easily overridden if one wants to push the same config everywhere.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1143-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 18:29:58 2009
Return-Path: <common-issues-return-1143-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59198 invoked from network); 31 Aug 2009 18:29:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 18:29:58 -0000
Received: (qmail 96187 invoked by uid 500); 31 Aug 2009 18:29:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96127 invoked by uid 500); 31 Aug 2009 18:29:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96117 invoked by uid 99); 31 Aug 2009 18:29:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 18:29:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 18:29:55 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 09741234C046
	for <common-issues@hadoop.apache.org>; Mon, 31 Aug 2009 11:29:34 -0700 (PDT)
Message-ID: <462588370.1251743374036.JavaMail.jira@brutus>
Date: Mon, 31 Aug 2009 11:29:34 -0700 (PDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6167) bin/hadoop script doesn't allow for
 different memory settings for each daemon type
In-Reply-To: <128618595.1248313995284.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749565#action_12749565 ] 

Allen Wittenauer commented on HADOOP-6167:
------------------------------------------

I agree with Scott.  This should be fixed.

> bin/hadoop script doesn't allow for different memory settings for each daemon type
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6167
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6167
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.0
>            Reporter: Fernando
>         Attachments: hadoop, hadoop-script.diff
>
>
> bin/hadoop assumes that all daemon types ( namenode, datanode, jobtracker, tasktracker ), all use the same memory settings.. (HADOOP_HEAPSIZE).
> I propose changes to that script to allow overriding the default memory ( HADOOP_HEAPSIZE ), with daemon specific OPTS (HADOOP_NAMENODE_OPTS, etc ).
> Basically at the bottom of the bin/hadoop script, it will check to see if the user has already set "-Xmx" in the HADOOP_OPTS variable.. if so, then it will ignore the JAVA_HEAP_SIZE variable..
> as such:
> # run it
> if [[ $HADOOP_OPTS == *-Xmx* ]]; then
>   exec "$JAVA" $HADOOP_OPTS -classpath "$CLASSPATH" $CLASS "$@"
> else
>   exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS -classpath "$CLASSPATH" $CLASS "$@"
> fi
> I will attach the file as I have modified it..

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1144-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 18:38:04 2009
Return-Path: <common-issues-return-1144-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61908 invoked from network); 31 Aug 2009 18:38:04 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 18:38:04 -0000
Received: (qmail 2091 invoked by uid 500); 31 Aug 2009 18:38:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2069 invoked by uid 500); 31 Aug 2009 18:38:04 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2059 invoked by uid 99); 31 Aug 2009 18:38:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 18:38:04 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 18:37:54 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id EEC67234C044
	for <common-issues@hadoop.apache.org>; Mon, 31 Aug 2009 11:37:32 -0700 (PDT)
Message-ID: <1907497064.1251743852977.JavaMail.jira@brutus>
Date: Mon, 31 Aug 2009 11:37:32 -0700 (PDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6225) Source checksum computation in
 saveVersion.sh doesn't work on MacOS X
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Source checksum computation in saveVersion.sh doesn't work on MacOS X
---------------------------------------------------------------------

                 Key: HADOOP-6225
                 URL: https://issues.apache.org/jira/browse/HADOOP-6225
             Project: Hadoop Common
          Issue Type: Bug
          Components: build
    Affects Versions: 0.21.0
            Reporter: Chris Douglas
            Priority: Minor


HADOOP-5203, added a "source checksum" field computed with {{md5sum}}. MacOS X uses an equivalent tool called {{md5}}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1145-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 18:49:54 2009
Return-Path: <common-issues-return-1145-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67845 invoked from network); 31 Aug 2009 18:49:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 18:49:54 -0000
Received: (qmail 15837 invoked by uid 500); 31 Aug 2009 18:49:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15788 invoked by uid 500); 31 Aug 2009 18:49:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15778 invoked by uid 99); 31 Aug 2009 18:49:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 18:49:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 18:49:52 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id B40A0234C004
	for <common-issues@hadoop.apache.org>; Mon, 31 Aug 2009 11:49:32 -0700 (PDT)
Message-ID: <589789978.1251744572723.JavaMail.jira@brutus>
Date: Mon, 31 Aug 2009 11:49:32 -0700 (PDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6222) Core doesn't have TestCommonCLI
 facility
In-Reply-To: <565101244.1251499832837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749572#action_12749572 ] 

Konstantin Boudnik commented on HADOOP-6222:
--------------------------------------------

It sounds great, but it shouldn't be a literal 'copy' cause it's a sure way for code duplication and creating a very different version of the little framework in the future.

In the past, I was able to pull-off TestCLI from HDFS and make it independent for the sake of Owl project. I'm pretty sure something similar has to be repeated in this case.

> Core doesn't have TestCommonCLI facility
> ----------------------------------------
>
>                 Key: HADOOP-6222
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6222
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>            Reporter: Boris Shkolnik
>
> TestCLI is a base class, which cannot run FS type of commands.
> We need a "copy" of TestHDFSCLI as TestCommonCLI to be able to test CLI stuff in common.
> I suggest we create TestCommonCLI.java in hadoop-common

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1146-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 20:57:55 2009
Return-Path: <common-issues-return-1146-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3018 invoked from network); 31 Aug 2009 20:57:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 20:57:54 -0000
Received: (qmail 70975 invoked by uid 500); 31 Aug 2009 20:57:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70917 invoked by uid 500); 31 Aug 2009 20:57:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70907 invoked by uid 99); 31 Aug 2009 20:57:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 20:57:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 20:57:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id B3B7F234C004
	for <common-issues@hadoop.apache.org>; Mon, 31 Aug 2009 13:57:32 -0700 (PDT)
Message-ID: <84983940.1251752252725.JavaMail.jira@brutus>
Date: Mon, 31 Aug 2009 13:57:32 -0700 (PDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6222) Core doesn't have TestCommonCLI
 facility
In-Reply-To: <565101244.1251499832837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749621#action_12749621 ] 

Boris Shkolnik commented on HADOOP-6222:
----------------------------------------

I agree with Konstantin that we shouldn't copy the code. 
But since we need to support both FS and DFSADMIN commands in common, we may need to either move most of the functionality from TestHDFSCLI to TestCLI or create another class (something like TestShellCLI), move FS/DFSADMIN command processing into it and extend it for HDFS.

> Core doesn't have TestCommonCLI facility
> ----------------------------------------
>
>                 Key: HADOOP-6222
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6222
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>            Reporter: Boris Shkolnik
>
> TestCLI is a base class, which cannot run FS type of commands.
> We need a "copy" of TestHDFSCLI as TestCommonCLI to be able to test CLI stuff in common.
> I suggest we create TestCommonCLI.java in hadoop-common

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1147-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 21:12:03 2009
Return-Path: <common-issues-return-1147-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5822 invoked from network); 31 Aug 2009 21:12:03 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 21:12:03 -0000
Received: (qmail 87444 invoked by uid 500); 31 Aug 2009 21:12:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87425 invoked by uid 500); 31 Aug 2009 21:12:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87415 invoked by uid 99); 31 Aug 2009 21:12:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 21:12:03 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 21:11:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id BB6C6234C004
	for <common-issues@hadoop.apache.org>; Mon, 31 Aug 2009 14:11:32 -0700 (PDT)
Message-ID: <1724881406.1251753092746.JavaMail.jira@brutus>
Date: Mon, 31 Aug 2009 14:11:32 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6225) Source checksum computation in
 saveVersion.sh doesn't work on MacOS X
In-Reply-To: <1907497064.1251743852977.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749625#action_12749625 ] 

Owen O'Malley commented on HADOOP-6225:
---------------------------------------

MacPort includes md5sum, so that is the easy way out. But code to detect and use the appropriate md5 tool would be fine.

> Source checksum computation in saveVersion.sh doesn't work on MacOS X
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6225
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6225
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Chris Douglas
>            Priority: Minor
>
> HADOOP-5203, added a "source checksum" field computed with {{md5sum}}. MacOS X uses an equivalent tool called {{md5}}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1148-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 21:22:02 2009
Return-Path: <common-issues-return-1148-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7815 invoked from network); 31 Aug 2009 21:22:02 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 21:22:02 -0000
Received: (qmail 10758 invoked by uid 500); 31 Aug 2009 21:22:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10733 invoked by uid 500); 31 Aug 2009 21:22:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10669 invoked by uid 99); 31 Aug 2009 21:22:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 21:22:01 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 21:21:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 32283234C004
	for <common-issues@hadoop.apache.org>; Mon, 31 Aug 2009 14:21:33 -0700 (PDT)
Message-ID: <356788716.1251753693187.JavaMail.jira@brutus>
Date: Mon, 31 Aug 2009 14:21:33 -0700 (PDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6225) Source checksum computation in
 saveVersion.sh doesn't work on MacOS X
In-Reply-To: <1907497064.1251743852977.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749629#action_12749629 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6225:
------------------------------------------------

cygwin also has a md5sum command.

Actually, it is easy to write a little java program to compute MD5. One way is to add a main method in org.apache.hadoop.io.MD5Hash.

> Source checksum computation in saveVersion.sh doesn't work on MacOS X
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6225
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6225
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Chris Douglas
>            Priority: Minor
>
> HADOOP-5203, added a "source checksum" field computed with {{md5sum}}. MacOS X uses an equivalent tool called {{md5}}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1149-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 21:27:55 2009
Return-Path: <common-issues-return-1149-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8661 invoked from network); 31 Aug 2009 21:27:55 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 21:27:55 -0000
Received: (qmail 14971 invoked by uid 500); 31 Aug 2009 21:27:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14952 invoked by uid 500); 31 Aug 2009 21:27:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14942 invoked by uid 99); 31 Aug 2009 21:27:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 21:27:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 21:27:53 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 2332D234C004
	for <common-issues@hadoop.apache.org>; Mon, 31 Aug 2009 14:27:33 -0700 (PDT)
Message-ID: <1304020973.1251754053129.JavaMail.jira@brutus>
Date: Mon, 31 Aug 2009 14:27:33 -0700 (PDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6203) Improve error message when moving
 to trash fails due to quota issue
In-Reply-To: <1489131604.1250623814866.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749630#action_12749630 ] 

Suresh Srinivas commented on HADOOP-6203:
-----------------------------------------

# The bug description talks about indicating that there was quota related problem while performing rm. That still is not captured with the new change. Also since the existing code just captures IOException from {{trashTmp.moveToTrash()}}, it not setup to capture quota issue.
# FSShell.delete() - This probably exists in the previous version of the code also. What happens if {{trashTmp.moveToTrash()}} returns -1 when skipTrash flag is true?

> Improve error message when moving to trash fails due to quota issue
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6203
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6203
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Jakob Homan
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6203.patch, HADOOP-6203.patch
>
>
> HADOOP-6080 provided an option for deleting files even when overquota, but the error message that's returned in this situation is unhelpful and doesn't suggest skipTrash as a remediation:
> {noformat}$ hdfs -rmr /foo/bar/bat/boo
> rmr: Failed to move to trash:
> hdfs://cluster/foo/bar/bat/boo{noformat}
> In this situation, the error message should say there was a quote problem and suggest -skipTrash.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1150-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 21:47:54 2009
Return-Path: <common-issues-return-1150-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13931 invoked from network); 31 Aug 2009 21:47:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 21:47:54 -0000
Received: (qmail 41295 invoked by uid 500); 31 Aug 2009 21:47:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41255 invoked by uid 500); 31 Aug 2009 21:47:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41236 invoked by uid 99); 31 Aug 2009 21:47:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 21:47:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 21:47:52 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id E455F234C1E9
	for <common-issues@hadoop.apache.org>; Mon, 31 Aug 2009 14:47:32 -0700 (PDT)
Message-ID: <1723917273.1251755252934.JavaMail.jira@brutus>
Date: Mon, 31 Aug 2009 14:47:32 -0700 (PDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6224) Add some utility methods to Text
 and DataOutputBuffer to be used by MAPREDUCE-318
In-Reply-To: <107649796.1251694292837.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749639#action_12749639 ] 

Hong Tang commented on HADOOP-6224:
-----------------------------------

I am actually a bit surprised by the patch. 

- One minor point: Would it be better readStringSafely move to WritableUtils?
- One serious question: I am not sure why we need to open up the internal Buffer class of DataOutputBuffer, or need to support the construction based on an external byte[]. For the least, the ctor should clearly identify the ownership of the byte[]. But based on the implementation of DataOutputBuffer, it seems that the ownership has to be transferred to DataOutputBuffer and the caller of this ctor should not retain the reference to byte[].

> Add some utility methods to Text and DataOutputBuffer to be used by MAPREDUCE-318
> ---------------------------------------------------------------------------------
>
>                 Key: HADOOP-6224
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6224
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Jothi Padmanabhan
>            Assignee: Jothi Padmanabhan
>             Fix For: 0.21.0
>
>         Attachments: common.patch
>
>
> MAPREDUCE-318 needs some minor extensions to the Text and the DataOutputBuffer classes. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1151-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 22:11:58 2009
Return-Path: <common-issues-return-1151-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19172 invoked from network); 31 Aug 2009 22:11:58 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 22:11:58 -0000
Received: (qmail 67237 invoked by uid 500); 31 Aug 2009 22:11:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67180 invoked by uid 500); 31 Aug 2009 22:11:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67170 invoked by uid 99); 31 Aug 2009 22:11:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 22:11:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 22:11:55 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id B95E1234C004
	for <common-issues@hadoop.apache.org>; Mon, 31 Aug 2009 15:11:32 -0700 (PDT)
Message-ID: <793815693.1251756692744.JavaMail.jira@brutus>
Date: Mon, 31 Aug 2009 15:11:32 -0700 (PDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4952) Improved files system interface for
 the application writer.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749649#action_12749649 ] 

Sanjay Radia commented on HADOOP-4952:
--------------------------------------

I forgot to mention that the class FsConfig is suppose to be package private (I had accidental made it public).
This should clear up the confusion between Doug and me.

> Improved files system interface for the application writer.
> -----------------------------------------------------------
>
>                 Key: HADOOP-4952
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4952
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.21.0
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>         Attachments: FileContext3.patch, FileContext5.patch, FileContext6.patch, Files.java, Files.java, FilesContext1.patch, FilesContext2.patch
>
>
> Currently the FIleSystem interface serves two purposes:
> - an application writer's interface for using the Hadoop file system
> - a file system implementer's interface (e.g. hdfs, local file system, kfs, etc)
> This Jira proposes that we provide a simpler interfaces for the application writer and leave the FilsSystem  interface for the implementer of a filesystem.
> - Filesystem interface  has a  confusing set of methods for the application writer
> - We could make it easier to take advantage of the URI file naming
> ** Current approach is to get FileSystem instance by supplying the URI and then access that name space. It is consistent for the FileSystem instance to not accept URIs for other schemes, but we can do better.
> ** The special copyFromLocalFIle can be generalized as a  copyFile where the src or target can be generalized to any URI, including the local one.
> ** The proposed scheme (below) simplifies this.
> -	The client side config can be simplified. 
> ** New config() by default uses the default config. Since this is the common usage pattern, one should not need to always pass the config as a parameter when accessing the file system.  
> -	
> ** It does not handle multiple file systems too well. Today a site.xml is derived from a single Hadoop cluster. This does not make sense for multiple Hadoop clusters which may have different defaults.
> ** Further one should need very little to configure the client side:
> *** Default files system.
> *** Block size 
> *** Replication factor
> *** Scheme to class mapping
> ** It should be possible to take Blocksize and replication factors defaults from the target file system, rather then the client size config.  I am not suggesting we don't allow setting client side defaults, but most clients do not care and would find it simpler to take the defaults for their systems  from the target file system. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-1152-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 31 23:34:54 2009
Return-Path: <common-issues-return-1152-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38712 invoked from network); 31 Aug 2009 23:34:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 31 Aug 2009 23:34:54 -0000
Received: (qmail 61319 invoked by uid 500); 31 Aug 2009 23:34:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61260 invoked by uid 500); 31 Aug 2009 23:34:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61249 invoked by uid 99); 31 Aug 2009 23:34:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 23:34:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Aug 2009 23:34:52 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id B2C42234C004
	for <common-issues@hadoop.apache.org>; Mon, 31 Aug 2009 16:34:32 -0700 (PDT)
Message-ID: <4271091.1251761672717.JavaMail.jira@brutus>
Date: Mon, 31 Aug 2009 16:34:32 -0700 (PDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6203) Improve error message when moving
 to trash fails due to quota issue
In-Reply-To: <1489131604.1250623814866.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749675#action_12749675 ] 

Boris Shkolnik commented on HADOOP-6203:
----------------------------------------

1. The idea is to make sure they know they can use skipTrash option. They can find real reason by looking at the logs. We can print exception.cause.message also, but that will make the message rather long. It will look like this:
>bin/hdfs dfs -rm /file3
Problem with Trash.org.apache.hadoop.hdfs.protocol.NSQuotaExceededException: The NameSpace quota (directories and files) of directory  is exceeded: quota=3 file count=7. Consider using -skipTrash option
rm: Failed to move to trash: hdfs://localhost:9000/file3

We cannot catch Quota exception directly because it is part of hadoop-hdfs. Parsing string to find out that it is quota related doesn't sound like a good idea.

Please specify if you like the longer version of the message.


2. if skipTrash is true - moveToTrash() won't be called.

> Improve error message when moving to trash fails due to quota issue
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6203
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6203
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Jakob Homan
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6203.patch, HADOOP-6203.patch
>
>
> HADOOP-6080 provided an option for deleting files even when overquota, but the error message that's returned in this situation is unhelpful and doesn't suggest skipTrash as a remediation:
> {noformat}$ hdfs -rmr /foo/bar/bat/boo
> rmr: Failed to move to trash:
> hdfs://cluster/foo/bar/bat/boo{noformat}
> In this situation, the error message should say there was a quote problem and suggest -skipTrash.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9309-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 01 01:28:22 2010
Return-Path: <common-issues-return-9309-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52402 invoked from network); 1 Jul 2010 01:28:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jul 2010 01:28:22 -0000
Received: (qmail 18220 invoked by uid 500); 1 Jul 2010 01:28:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18175 invoked by uid 500); 1 Jul 2010 01:28:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17979 invoked by uid 99); 1 Jul 2010 01:28:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 01:28:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 01:28:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o611RusO021632
	for <common-issues@hadoop.apache.org>; Thu, 1 Jul 2010 01:27:56 GMT
Message-ID: <6458645.145611277947676511.JavaMail.jira@thor>
Date: Wed, 30 Jun 2010 21:27:56 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884120#action_12884120 ] 

Greg Roelofs commented on HADOOP-6835:
--------------------------------------

OK, so auto-patch doesn't know about the split projects, and hadoop-mapreduce patches don't work in hadoop-common.

I tried moving the test case to hadoop-common/src/test/core/org/apache/hadoop/io/compress, but it depends on too many MR classes to be workable (JobConf, Reporter, RecordReader, FileSplit, InputSplit, FileInputFormat, TextInputFormat).  So I guess the procedure is (1) reupload the hadoop-common patch and mark as patch-available [actually, I missed a fix in TestCodec, so I need to upload a new one anyway]; (2) get that past QA and checked in; (3) open a separate MR JIRA, attach the test case, and mark it patch-available; and (4) get _that_ past QA and checked in, too.

If there's an easier way, feel free to enlighten me.  (On a related note, I was told by Reliable Sources to blame Owen, but he just left town.  Coincidence?)


> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>             Fix For: 0.22.0
>
>         Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v4.trunk-hadoop-common.patch, HADOOP-6835.v4.trunk-hadoop-mapreduce.patch, HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9310-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 01 01:38:22 2010
Return-Path: <common-issues-return-9310-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56745 invoked from network); 1 Jul 2010 01:38:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jul 2010 01:38:21 -0000
Received: (qmail 29747 invoked by uid 500); 1 Jul 2010 01:38:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29698 invoked by uid 500); 1 Jul 2010 01:38:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29685 invoked by uid 99); 1 Jul 2010 01:38:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 01:38:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 01:38:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o611bvJE021778
	for <common-issues@hadoop.apache.org>; Thu, 1 Jul 2010 01:37:57 GMT
Message-ID: <31602450.145971277948277758.JavaMail.jira@thor>
Date: Wed, 30 Jun 2010 21:37:57 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Greg Roelofs updated HADOOP-6835:
---------------------------------

    Attachment: HADOOP-6835.v5.trunk-hadoop-common.patch

Main (hadoop-common) patch, version 5:  includes TestCodec fix.

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>             Fix For: 0.22.0
>
>         Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v4.trunk-hadoop-common.patch, HADOOP-6835.v4.trunk-hadoop-mapreduce.patch, HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v5.trunk-hadoop-common.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9311-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 01 02:00:19 2010
Return-Path: <common-issues-return-9311-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64234 invoked from network); 1 Jul 2010 02:00:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jul 2010 02:00:19 -0000
Received: (qmail 43578 invoked by uid 500); 1 Jul 2010 02:00:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43510 invoked by uid 500); 1 Jul 2010 02:00:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43502 invoked by uid 99); 1 Jul 2010 02:00:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 02:00:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 02:00:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o611xs0J021927
	for <common-issues@hadoop.apache.org>; Thu, 1 Jul 2010 01:59:54 GMT
Message-ID: <9776609.146271277949594318.JavaMail.jira@thor>
Date: Wed, 30 Jun 2010 21:59:54 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Greg Roelofs updated HADOOP-6835:
---------------------------------

    Status: Open  (was: Patch Available)

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>             Fix For: 0.22.0
>
>         Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v4.trunk-hadoop-common.patch, HADOOP-6835.v4.trunk-hadoop-mapreduce.patch, HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v5.trunk-hadoop-common.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9312-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 01 02:00:21 2010
Return-Path: <common-issues-return-9312-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64283 invoked from network); 1 Jul 2010 02:00:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jul 2010 02:00:21 -0000
Received: (qmail 43692 invoked by uid 500); 1 Jul 2010 02:00:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43666 invoked by uid 500); 1 Jul 2010 02:00:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43658 invoked by uid 99); 1 Jul 2010 02:00:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 02:00:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 02:00:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o611xu9o021972
	for <common-issues@hadoop.apache.org>; Thu, 1 Jul 2010 01:59:56 GMT
Message-ID: <23648750.146421277949596754.JavaMail.jira@thor>
Date: Wed, 30 Jun 2010 21:59:56 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Greg Roelofs updated HADOOP-6835:
---------------------------------

    Status: Patch Available  (was: Open)

hadoop-common v5 this time.

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>             Fix For: 0.22.0
>
>         Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v4.trunk-hadoop-common.patch, HADOOP-6835.v4.trunk-hadoop-mapreduce.patch, HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v5.trunk-hadoop-common.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9313-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 01 02:29:49 2010
Return-Path: <common-issues-return-9313-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75823 invoked from network); 1 Jul 2010 02:29:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jul 2010 02:29:49 -0000
Received: (qmail 62594 invoked by uid 500); 1 Jul 2010 02:29:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62536 invoked by uid 500); 1 Jul 2010 02:29:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62528 invoked by uid 99); 1 Jul 2010 02:29:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 02:29:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 02:29:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o612LsiN022158
	for <common-issues@hadoop.apache.org>; Thu, 1 Jul 2010 02:21:54 GMT
Message-ID: <20648759.146761277950914193.JavaMail.jira@thor>
Date: Wed, 30 Jun 2010 22:21:54 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884135#action_12884135 ] 

Hadoop QA commented on HADOOP-6835:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12448469/HADOOP-6835.v5.trunk-hadoop-common.patch
  against trunk revision 957074.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The patch appears to cause tar ant target to fail.

    -1 findbugs.  The patch appears to cause Findbugs to fail.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    -1 contrib tests.  The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/598/testReport/
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/598/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/598/console

This message is automatically generated.

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>             Fix For: 0.22.0
>
>         Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v4.trunk-hadoop-common.patch, HADOOP-6835.v4.trunk-hadoop-mapreduce.patch, HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v5.trunk-hadoop-common.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9314-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 01 03:26:50 2010
Return-Path: <common-issues-return-9314-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89223 invoked from network); 1 Jul 2010 03:26:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jul 2010 03:26:50 -0000
Received: (qmail 94427 invoked by uid 500); 1 Jul 2010 03:26:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94302 invoked by uid 500); 1 Jul 2010 03:26:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94294 invoked by uid 99); 1 Jul 2010 03:26:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 03:26:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 03:26:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o613Iqvp022445
	for <common-issues@hadoop.apache.org>; Thu, 1 Jul 2010 03:18:52 GMT
Message-ID: <19139050.147161277954332422.JavaMail.jira@thor>
Date: Wed, 30 Jun 2010 23:18:52 -0400 (EDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5436) job history directory grows without
 bound, locks up job tracker on new job submission
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884145#action_12884145 ] 

Amareshwari Sriramadasu commented on HADOOP-5436:
-------------------------------------------------

Jobhistory directory structure will be fixed by MAPREDUCE-323. That should solve the issue.

> job history directory grows without bound, locks up job tracker on new job submission
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5436
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5436
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.19.0, 0.20.0, 0.20.1, 0.20.2
>            Reporter: Tim Williamson
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-5436.patch
>
>
> An unpleasant surprise upgrading to 0.19: requests to jobtracker.jsp would take a long time or even time out whenever new jobs where submitted.  Investigation showed the call to JobInProgress.initTasks() was calling JobHistory.JobInfo.logSubmitted() which in turn was calling JobHistory.getJobHistoryFileName() which was pegging the CPU for a couple minutes.  Further investigation showed the were 200,000+ files in the job history folder -- and every submission was creating a FileStatus for them all, then applying a regular expression to just the name.  All this just on the off chance the job tracker had been restarted (see HADOOP-3245).  To make matters worse, these files cannot be safely deleted while the job tracker is running, as the disappearance of a history file at the wrong time causes a FileNotFoundException.
> So to summarize the issues:
> - having Hadoop default to storing all the history files in a single directory is a Bad Idea
> - doing expensive processing of every history file on every job submission is a Worse Idea
> - doing expensive processing of every history file on every job submission while holding a lock on the JobInProgress object and thereby blocking the jobtracker.jsp from rendering is a Terrible Idea (note: haven't confirmed this, but a cursory glance suggests that's what's going on)
> - not being able to clean up the mess without taking down the job tracker is just Unfortunate

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9315-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 01 04:26:19 2010
Return-Path: <common-issues-return-9315-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1937 invoked from network); 1 Jul 2010 04:26:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jul 2010 04:26:19 -0000
Received: (qmail 16622 invoked by uid 500); 1 Jul 2010 04:26:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16491 invoked by uid 500); 1 Jul 2010 04:26:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16483 invoked by uid 99); 1 Jul 2010 04:26:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 04:26:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 04:26:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o614Pph4023201
	for <common-issues@hadoop.apache.org>; Thu, 1 Jul 2010 04:25:51 GMT
Message-ID: <17262777.147431277958351555.JavaMail.jira@thor>
Date: Thu, 1 Jul 2010 00:25:51 -0400 (EDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6800) Harmonize JAR library versions
In-Reply-To: <8215141.145191275515876547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884151#action_12884151 ] 

Hemanth Yamijala commented on HADOOP-6800:
------------------------------------------

Tom, Hudson's results on MAPREDUCE-1870 seem to be OK. I checked HDFS-1212 and the test failures had age > 0; so I assume this is fine too. All patches seem ready except for reaching out to contrib devs as we discussed above. Do you need any further assistance ?

> Harmonize JAR library versions
> ------------------------------
>
>                 Key: HADOOP-6800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: build.sh, HADOOP-6800.patch, HADOOP-6800.patch, HADOOP-6800.patch, ivy-declarations.dump.merged, jars-in-lib.dump.merged, tar-munge
>
>
> Currently, multiple versions of the same library JAR are being pulled in for various reasons.
> * Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
> * Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9316-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 01 04:28:18 2010
Return-Path: <common-issues-return-9316-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2226 invoked from network); 1 Jul 2010 04:28:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jul 2010 04:28:18 -0000
Received: (qmail 18910 invoked by uid 500); 1 Jul 2010 04:28:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18786 invoked by uid 500); 1 Jul 2010 04:28:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18778 invoked by uid 99); 1 Jul 2010 04:28:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 04:28:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 04:28:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o614RoYT023226
	for <common-issues@hadoop.apache.org>; Thu, 1 Jul 2010 04:27:51 GMT
Message-ID: <13732257.147511277958470874.JavaMail.jira@thor>
Date: Thu, 1 Jul 2010 00:27:50 -0400 (EDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6800) Harmonize JAR library versions
In-Reply-To: <8215141.145191275515876547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala updated HADOOP-6800:
-------------------------------------

    Attachment: HADOOP-6800.patch

Re-attaching the same patch as Tom's last one, so can run it past Hudson.

> Harmonize JAR library versions
> ------------------------------
>
>                 Key: HADOOP-6800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: build.sh, HADOOP-6800.patch, HADOOP-6800.patch, HADOOP-6800.patch, HADOOP-6800.patch, ivy-declarations.dump.merged, jars-in-lib.dump.merged, tar-munge
>
>
> Currently, multiple versions of the same library JAR are being pulled in for various reasons.
> * Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
> * Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9317-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 01 04:33:49 2010
Return-Path: <common-issues-return-9317-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2669 invoked from network); 1 Jul 2010 04:33:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jul 2010 04:33:49 -0000
Received: (qmail 22691 invoked by uid 500); 1 Jul 2010 04:33:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22557 invoked by uid 500); 1 Jul 2010 04:33:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22540 invoked by uid 99); 1 Jul 2010 04:33:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 04:33:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 04:33:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o614PqtP023213
	for <common-issues@hadoop.apache.org>; Thu, 1 Jul 2010 04:25:52 GMT
Message-ID: <9535798.147471277958352602.JavaMail.jira@thor>
Date: Thu, 1 Jul 2010 00:25:52 -0400 (EDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6800) Harmonize JAR library versions
In-Reply-To: <8215141.145191275515876547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala updated HADOOP-6800:
-------------------------------------

    Status: Open  (was: Patch Available)

> Harmonize JAR library versions
> ------------------------------
>
>                 Key: HADOOP-6800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: build.sh, HADOOP-6800.patch, HADOOP-6800.patch, HADOOP-6800.patch, ivy-declarations.dump.merged, jars-in-lib.dump.merged, tar-munge
>
>
> Currently, multiple versions of the same library JAR are being pulled in for various reasons.
> * Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
> * Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9318-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 01 04:35:49 2010
Return-Path: <common-issues-return-9318-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2875 invoked from network); 1 Jul 2010 04:35:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jul 2010 04:35:49 -0000
Received: (qmail 23729 invoked by uid 500); 1 Jul 2010 04:35:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23605 invoked by uid 500); 1 Jul 2010 04:35:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23596 invoked by uid 99); 1 Jul 2010 04:35:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 04:35:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 04:35:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o614RqTb023238
	for <common-issues@hadoop.apache.org>; Thu, 1 Jul 2010 04:27:52 GMT
Message-ID: <28800186.147551277958472351.JavaMail.jira@thor>
Date: Thu, 1 Jul 2010 00:27:52 -0400 (EDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6800) Harmonize JAR library versions
In-Reply-To: <8215141.145191275515876547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala updated HADOOP-6800:
-------------------------------------

    Status: Patch Available  (was: Open)

> Harmonize JAR library versions
> ------------------------------
>
>                 Key: HADOOP-6800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: build.sh, HADOOP-6800.patch, HADOOP-6800.patch, HADOOP-6800.patch, HADOOP-6800.patch, ivy-declarations.dump.merged, jars-in-lib.dump.merged, tar-munge
>
>
> Currently, multiple versions of the same library JAR are being pulled in for various reasons.
> * Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
> * Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9319-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 01 04:47:48 2010
Return-Path: <common-issues-return-9319-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4291 invoked from network); 1 Jul 2010 04:47:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jul 2010 04:47:48 -0000
Received: (qmail 28932 invoked by uid 500); 1 Jul 2010 04:47:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28797 invoked by uid 500); 1 Jul 2010 04:47:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28785 invoked by uid 99); 1 Jul 2010 04:47:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 04:47:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 04:47:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o614doKO023339
	for <common-issues@hadoop.apache.org>; Thu, 1 Jul 2010 04:39:51 GMT
Message-ID: <19128278.147741277959190940.JavaMail.jira@thor>
Date: Thu, 1 Jul 2010 00:39:50 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6800) Harmonize JAR library versions
In-Reply-To: <8215141.145191275515876547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6800:
------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

I've just committed this. (Hudson has already run against the patch.) Thanks for your help Hemanth.

> Harmonize JAR library versions
> ------------------------------
>
>                 Key: HADOOP-6800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: build.sh, HADOOP-6800.patch, HADOOP-6800.patch, HADOOP-6800.patch, HADOOP-6800.patch, ivy-declarations.dump.merged, jars-in-lib.dump.merged, tar-munge
>
>
> Currently, multiple versions of the same library JAR are being pulled in for various reasons.
> * Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
> * Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9320-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 01 04:54:20 2010
Return-Path: <common-issues-return-9320-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6282 invoked from network); 1 Jul 2010 04:54:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jul 2010 04:54:20 -0000
Received: (qmail 32324 invoked by uid 500); 1 Jul 2010 04:54:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32133 invoked by uid 500); 1 Jul 2010 04:54:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32121 invoked by uid 99); 1 Jul 2010 04:54:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 04:54:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 04:54:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o614rqmX023405
	for <common-issues@hadoop.apache.org>; Thu, 1 Jul 2010 04:53:52 GMT
Message-ID: <31326081.147861277960032302.JavaMail.jira@thor>
Date: Thu, 1 Jul 2010 00:53:52 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6800) Harmonize JAR library versions
In-Reply-To: <8215141.145191275515876547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884161#action_12884161 ] 

Hudson commented on HADOOP-6800:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #317 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/317/])
    HADOOP-6800.  Harmonize JAR library versions.


> Harmonize JAR library versions
> ------------------------------
>
>                 Key: HADOOP-6800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: build.sh, HADOOP-6800.patch, HADOOP-6800.patch, HADOOP-6800.patch, HADOOP-6800.patch, ivy-declarations.dump.merged, jars-in-lib.dump.merged, tar-munge
>
>
> Currently, multiple versions of the same library JAR are being pulled in for various reasons.
> * Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
> * Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9321-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 01 11:13:21 2010
Return-Path: <common-issues-return-9321-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63777 invoked from network); 1 Jul 2010 11:13:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jul 2010 11:13:21 -0000
Received: (qmail 60513 invoked by uid 500); 1 Jul 2010 11:13:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60301 invoked by uid 500); 1 Jul 2010 11:13:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60291 invoked by uid 99); 1 Jul 2010 11:13:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 11:13:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 11:13:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o61BCpQt026588
	for <common-issues@hadoop.apache.org>; Thu, 1 Jul 2010 11:12:52 GMT
Message-ID: <23254584.152221277982771665.JavaMail.jira@thor>
Date: Thu, 1 Jul 2010 07:12:51 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6800) Harmonize JAR library versions
In-Reply-To: <8215141.145191275515876547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884247#action_12884247 ] 

Hudson commented on HADOOP-6800:
--------------------------------

Integrated in Hadoop-Common-trunk #383 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/383/])
    HADOOP-6800.  Harmonize JAR library versions.


> Harmonize JAR library versions
> ------------------------------
>
>                 Key: HADOOP-6800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: build.sh, HADOOP-6800.patch, HADOOP-6800.patch, HADOOP-6800.patch, HADOOP-6800.patch, ivy-declarations.dump.merged, jars-in-lib.dump.merged, tar-munge
>
>
> Currently, multiple versions of the same library JAR are being pulled in for various reasons.
> * Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
> * Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9322-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 01 22:08:54 2010
Return-Path: <common-issues-return-9322-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14135 invoked from network); 1 Jul 2010 22:08:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jul 2010 22:08:54 -0000
Received: (qmail 2602 invoked by uid 500); 1 Jul 2010 22:08:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2475 invoked by uid 500); 1 Jul 2010 22:08:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2459 invoked by uid 99); 1 Jul 2010 22:08:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 22:08:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 22:08:50 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o61M0wUg002761
	for <common-issues@hadoop.apache.org>; Thu, 1 Jul 2010 22:00:58 GMT
Message-ID: <15189316.162981278021658765.JavaMail.jira@thor>
Date: Thu, 1 Jul 2010 18:00:58 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6756) Clean up and add documentation for
 configuration keys in CommonConfigurationKeys.java
In-Reply-To: <23380183.20521273191468124.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Erik Steffl updated HADOOP-6756:
--------------------------------

    Attachment: HADOOP-6756-0.22.0-2.patch

Patch HADOOP-6756-0.22.0-2.patch has a new improved  documentation for io.map.index.interval.

> Clean up and add documentation for configuration keys in CommonConfigurationKeys.java
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6756
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6756
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.3, 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.20.3, 0.22.0
>
>         Attachments: HADOOP-6756-0.22.0-1.patch, HADOOP-6756-0.22.0-2.patch, HADOOP-6756-0.22.0.patch, jira.HADOOP-6756-0.20-1-FS_DEFAULT_NAME_KEY.patch, jira.HADOOP-6756-0.20-1.patch, jira.HADOOP-6756-0.20.patch
>
>
> Configuration keys in CommonConfigurationKeys.java should be cleaned up and documented (javadoc comments, appropriate *-default.xml descriptions).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9323-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 01 23:58:51 2010
Return-Path: <common-issues-return-9323-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37265 invoked from network); 1 Jul 2010 23:58:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jul 2010 23:58:50 -0000
Received: (qmail 96129 invoked by uid 500); 1 Jul 2010 23:58:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95511 invoked by uid 500); 1 Jul 2010 23:58:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95267 invoked by uid 99); 1 Jul 2010 23:58:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 23:58:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 23:58:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o61Nos0t003425
	for <common-issues@hadoop.apache.org>; Thu, 1 Jul 2010 23:50:54 GMT
Message-ID: <4968696.164341278028254035.JavaMail.jira@thor>
Date: Thu, 1 Jul 2010 19:50:54 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6844) Update docs to reflect new jar names
In-Reply-To: <21611623.80411277663810310.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6844:
------------------------------

    Fix Version/s:     (was: 0.21.0)

> Update docs to reflect new jar names
> ------------------------------------
>
>                 Key: HADOOP-6844
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6844
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>    Affects Versions: 0.20.3
>            Reporter: Eli Collins
>             Fix For: 0.20.3, 0.22.0
>
>
> HADOOP-6382 changed the way jars are named, the docs need to be updated to reflect this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9324-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 01 23:58:51 2010
Return-Path: <common-issues-return-9324-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37280 invoked from network); 1 Jul 2010 23:58:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jul 2010 23:58:50 -0000
Received: (qmail 96469 invoked by uid 500); 1 Jul 2010 23:58:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96095 invoked by uid 500); 1 Jul 2010 23:58:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95737 invoked by uid 99); 1 Jul 2010 23:58:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 23:58:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 23:58:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o61Notcr003437
	for <common-issues@hadoop.apache.org>; Thu, 1 Jul 2010 23:50:55 GMT
Message-ID: <23314948.164381278028255284.JavaMail.jira@thor>
Date: Thu, 1 Jul 2010 19:50:55 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6688) FileSystem.delete(...)
 implementations should not throw FileNotFoundException
In-Reply-To: <1780197215.37411270627173494.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6688:
------------------------------

    Fix Version/s:     (was: 0.21.0)

> FileSystem.delete(...) implementations should not throw FileNotFoundException
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-6688
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6688
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, fs/s3
>    Affects Versions: 0.20.2
>         Environment: Amazon EC2/S3
>            Reporter: Danny Leshem
>            Priority: Minor
>             Fix For: 0.20.3, 0.22.0
>
>
> S3FileSystem.delete(Path path, boolean recursive) may fail and throw a FileNotFoundException if a directory is being deleted while at the same time some of its files are deleted in the background.
> This is definitely not the expected behavior of a delete method. If one of the to-be-deleted files is found missing, the method should not fail and simply continue. This is true for the general contract of FileSystem.delete, and also for its various implementations: RawLocalFileSystem (and specifically FileUtil.fullyDelete) exhibits the same problem.
> The fix is to silently catch and ignore FileNotFoundExceptions in delete loops. This can very easily be unit-tested, at least for RawLocalFileSystem.
> The reason this issue bothers me is that the cleanup part of a long (Mahout) MR job inconsistently fails for me, and I think this is the root problem. The log shows:
> {code}
> java.io.FileNotFoundException: s3://S3-BUCKET/tmp/0008E25BF7554CA9/2521362836721872/DistributedMatrix.times.outputVector/_temporary/_attempt_201004061215_0092_r_000002_0/part-00002: No such file or directory.
> 	at org.apache.hadoop.fs.s3.S3FileSystem.getFileStatus(S3FileSystem.java:334)
> 	at org.apache.hadoop.fs.s3.S3FileSystem.listStatus(S3FileSystem.java:193)
> 	at org.apache.hadoop.fs.s3.S3FileSystem.delete(S3FileSystem.java:303)
> 	at org.apache.hadoop.fs.s3.S3FileSystem.delete(S3FileSystem.java:312)
> 	at org.apache.hadoop.mapred.FileOutputCommitter.cleanupJob(FileOutputCommitter.java:64)
> 	at org.apache.hadoop.mapred.OutputCommitter.cleanupJob(OutputCommitter.java:135)
> 	at org.apache.hadoop.mapred.Task.runJobCleanupTask(Task.java:826)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:292)
> 	at org.apache.hadoop.mapred.Child.main(Child.java:170)
> {code}
> (similar errors are displayed for ReduceTask.run)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9325-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 05:06:19 2010
Return-Path: <common-issues-return-9325-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2837 invoked from network); 2 Jul 2010 05:06:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 05:06:19 -0000
Received: (qmail 48800 invoked by uid 500); 2 Jul 2010 05:06:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48678 invoked by uid 500); 2 Jul 2010 05:06:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48670 invoked by uid 99); 2 Jul 2010 05:06:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 05:06:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 05:06:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6255pXo005765
	for <common-issues@hadoop.apache.org>; Fri, 2 Jul 2010 05:05:52 GMT
Message-ID: <13621831.167311278047151837.JavaMail.jira@thor>
Date: Fri, 2 Jul 2010 01:05:51 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6584) Provide Kerberized SSL encryption
 for webservices
In-Reply-To: <713706873.424711266804869916.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884544#action_12884544 ] 

Devaraj Das commented on HADOOP-6584:
-------------------------------------

Looks good. +1

> Provide Kerberized SSL encryption for webservices
> -------------------------------------------------
>
>                 Key: HADOOP-6584
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6584
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>         Attachments: c6584-02.patch, HADOOP-6584-FixJavadoc-Y20.patch, HADOOP-6584-Y20-1.patch, HADOOP-6584-Y20-2.patch, HADOOP-6584-Y20-3.patch, HADOOP-6584-Y20-4.patch, HADOOP-6584.patch
>
>
> Some web services should be authenticated via SSL backed by Kerberos, both to provide cross-cluster secure communication and to provide intra-cluster server authentication for services such as the {Name,SecondaryName,Backup,Checkpoint}node's image transfer and balancer.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9326-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 05:46:19 2010
Return-Path: <common-issues-return-9326-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13548 invoked from network); 2 Jul 2010 05:46:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 05:46:19 -0000
Received: (qmail 73034 invoked by uid 500); 2 Jul 2010 05:46:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72902 invoked by uid 500); 2 Jul 2010 05:46:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72886 invoked by uid 99); 2 Jul 2010 05:46:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 05:46:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 05:46:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o625jor9005929
	for <common-issues@hadoop.apache.org>; Fri, 2 Jul 2010 05:45:50 GMT
Message-ID: <12606214.167451278049550119.JavaMail.jira@thor>
Date: Fri, 2 Jul 2010 01:45:50 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6846) Scripts for building Hadoop 0.21.0
 release
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Scripts for building Hadoop 0.21.0 release
------------------------------------------

                 Key: HADOOP-6846
                 URL: https://issues.apache.org/jira/browse/HADOOP-6846
             Project: Hadoop Common
          Issue Type: Task
          Components: build
            Reporter: Tom White
            Assignee: Tom White




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9327-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 07:04:17 2010
Return-Path: <common-issues-return-9327-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40192 invoked from network); 2 Jul 2010 07:04:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 07:04:17 -0000
Received: (qmail 53107 invoked by uid 500); 2 Jul 2010 07:04:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53082 invoked by uid 500); 2 Jul 2010 07:04:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53074 invoked by uid 99); 2 Jul 2010 07:04:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 07:04:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 07:04:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6273nVU006481
	for <common-issues@hadoop.apache.org>; Fri, 2 Jul 2010 07:03:49 GMT
Message-ID: <28118738.168371278054229909.JavaMail.jira@thor>
Date: Fri, 2 Jul 2010 03:03:49 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6847) Problem staging 0.21.0 artifacts to
 Apache Nexus Maven Repository
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Problem staging 0.21.0 artifacts to Apache Nexus Maven Repository
-----------------------------------------------------------------

                 Key: HADOOP-6847
                 URL: https://issues.apache.org/jira/browse/HADOOP-6847
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.21.0
            Reporter: Tom White
            Priority: Blocker


I get this error:
{noformat}
-mvn-system-deploy:
[artifact:install-provider] Installing provider: org.apache.maven.wagon:wagon-http:jar:1.0-beta-2:runtime
[artifact:deploy] Deploying to https://repository.apache.org/content/repositories/snapshots
[artifact:deploy] Uploading: org/apache/hadoop/hadoop-common-instrumented/0.21.0/hadoop-common-instrumented-0.21.0.jar to apache.snapshots.https
[artifact:deploy] Uploaded 1331K
[artifact:deploy] An error has occurred while processing the Maven artifact tasks.
[artifact:deploy]  Diagnosis:
[artifact:deploy] 
[artifact:deploy] Error deploying artifact 'org.apache.hadoop:hadoop-common-instrumented:jar': Error deploying artifact: Failed to transfer file: https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common-instrumented/0.21.0/hadoop-common-instrumented-0.21.0.jar. Return code is: 400
{noformat}

Note that 400 is "Bad Request", not an authentication error (401).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9328-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 07:30:17 2010
Return-Path: <common-issues-return-9328-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52661 invoked from network); 2 Jul 2010 07:30:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 07:30:17 -0000
Received: (qmail 84692 invoked by uid 500); 2 Jul 2010 07:30:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84598 invoked by uid 500); 2 Jul 2010 07:30:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84071 invoked by uid 99); 2 Jul 2010 07:30:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 07:30:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 07:30:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o627ToFl007120
	for <common-issues@hadoop.apache.org>; Fri, 2 Jul 2010 07:29:50 GMT
Message-ID: <11186715.169911278055790011.JavaMail.jira@thor>
Date: Fri, 2 Jul 2010 03:29:50 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6846) Scripts for building Hadoop 0.21.0
 release
In-Reply-To: <12606214.167451278049550119.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6846:
------------------------------

    Attachment: release-scripts.tar.gz

Here are the scripts I used to build the release candidate in conjunction with the instructions at http://wiki.apache.org/hadoop/HowToRelease. I used an EC2 instance (AMI ami-c38d6caa) for the build.

> Scripts for building Hadoop 0.21.0 release
> ------------------------------------------
>
>                 Key: HADOOP-6846
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6846
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: build
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: release-scripts.tar.gz
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9329-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 16:21:47 2010
Return-Path: <common-issues-return-9329-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92699 invoked from network); 2 Jul 2010 16:21:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 16:21:47 -0000
Received: (qmail 70239 invoked by uid 500); 2 Jul 2010 16:21:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70080 invoked by uid 500); 2 Jul 2010 16:21:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69760 invoked by uid 99); 2 Jul 2010 16:21:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 16:21:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 16:21:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o62GDpCe011278
	for <common-issues@hadoop.apache.org>; Fri, 2 Jul 2010 16:13:52 GMT
Message-ID: <7971623.176621278087231805.JavaMail.jira@thor>
Date: Fri, 2 Jul 2010 12:13:51 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6756) Clean up and add documentation for
 configuration keys in CommonConfigurationKeys.java
In-Reply-To: <23380183.20521273191468124.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Erik Steffl updated HADOOP-6756:
--------------------------------

    Attachment: HADOOP-6756-0.22.0-3.patch

HADOOP-6756-0.22.0-3.patch applies to current trunk on 2010/6/2

> Clean up and add documentation for configuration keys in CommonConfigurationKeys.java
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6756
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6756
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.3, 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.20.3, 0.22.0
>
>         Attachments: HADOOP-6756-0.22.0-1.patch, HADOOP-6756-0.22.0-2.patch, HADOOP-6756-0.22.0-3.patch, HADOOP-6756-0.22.0.patch, jira.HADOOP-6756-0.20-1-FS_DEFAULT_NAME_KEY.patch, jira.HADOOP-6756-0.20-1.patch, jira.HADOOP-6756-0.20.patch
>
>
> Configuration keys in CommonConfigurationKeys.java should be cleaned up and documented (javadoc comments, appropriate *-default.xml descriptions).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9330-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 17:22:15 2010
Return-Path: <common-issues-return-9330-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9537 invoked from network); 2 Jul 2010 17:22:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 17:22:15 -0000
Received: (qmail 37603 invoked by uid 500); 2 Jul 2010 17:22:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37536 invoked by uid 500); 2 Jul 2010 17:22:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37528 invoked by uid 99); 2 Jul 2010 17:22:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 17:22:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 17:22:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o62HLoOi011766
	for <common-issues@hadoop.apache.org>; Fri, 2 Jul 2010 17:21:50 GMT
Message-ID: <22214536.177091278091310612.JavaMail.jira@thor>
Date: Fri, 2 Jul 2010 13:21:50 -0400 (EDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6056) Use java.net.preferIPv4Stack to force
 IPv4
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steve Loughran updated HADOOP-6056:
-----------------------------------

    Status: Patch Available  (was: Open)

resubmitting

> Use java.net.preferIPv4Stack to force IPv4
> ------------------------------------------
>
>                 Key: HADOOP-6056
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6056
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Steve Loughran
>            Assignee: Michele Catasta
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6056.patch, hadoop-6056.txt
>
>
> This was mentioned on HADOOP-3427, there is a property,  java.net.preferIPv4Stack, which you set to true for the java net process to switch to IPv4 everywhere. 
> As Hadoop doesn't work on IPv6, this should be set to true in the startup scripts. Hopefully this will ensure that Jetty will also pick it up.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9331-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 17:53:15 2010
Return-Path: <common-issues-return-9331-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18561 invoked from network); 2 Jul 2010 17:53:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 17:53:15 -0000
Received: (qmail 81887 invoked by uid 500); 2 Jul 2010 17:53:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81828 invoked by uid 500); 2 Jul 2010 17:53:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81820 invoked by uid 99); 2 Jul 2010 17:53:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 17:53:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 17:53:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o62HqoZd011935
	for <common-issues@hadoop.apache.org>; Fri, 2 Jul 2010 17:52:50 GMT
Message-ID: <31529542.177471278093170339.JavaMail.jira@thor>
Date: Fri, 2 Jul 2010 13:52:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6056) Use java.net.preferIPv4Stack to
 force IPv4
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884754#action_12884754 ] 

Hadoop QA commented on HADOOP-6056:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12439189/HADOOP-6056.patch
  against trunk revision 959501.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/600/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/600/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/600/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/600/console

This message is automatically generated.

> Use java.net.preferIPv4Stack to force IPv4
> ------------------------------------------
>
>                 Key: HADOOP-6056
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6056
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Steve Loughran
>            Assignee: Michele Catasta
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6056.patch, hadoop-6056.txt
>
>
> This was mentioned on HADOOP-3427, there is a property,  java.net.preferIPv4Stack, which you set to true for the java net process to switch to IPv4 everywhere. 
> As Hadoop doesn't work on IPv6, this should be set to true in the startup scripts. Hopefully this will ensure that Jetty will also pick it up.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9332-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 17:59:15 2010
Return-Path: <common-issues-return-9332-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19439 invoked from network); 2 Jul 2010 17:59:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 17:59:14 -0000
Received: (qmail 85584 invoked by uid 500); 2 Jul 2010 17:59:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85476 invoked by uid 500); 2 Jul 2010 17:59:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85468 invoked by uid 99); 2 Jul 2010 17:59:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 17:59:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 17:59:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o62Hwn7k011995
	for <common-issues@hadoop.apache.org>; Fri, 2 Jul 2010 17:58:50 GMT
Message-ID: <10947781.177581278093529874.JavaMail.jira@thor>
Date: Fri, 2 Jul 2010 13:58:49 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Moved: (HADOOP-6848) FsShell have resource leak
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins moved HDFS-1279 to HADOOP-6848:
-------------------------------------------

              Project: Hadoop Common  (was: Hadoop HDFS)
                  Key: HADOOP-6848  (was: HDFS-1279)
    Affects Version/s: 0.20.2
                       0.20.1
                       0.20.3
                       0.20-append
                           (was: 0.20.1)
                           (was: 0.20.2)
                           (was: 0.20.3)
                           (was: 0.20-append)
        Fix Version/s: 0.20.3
                       0.20-append
                           (was: 0.20.3)
                           (was: 0.20-append)
          Component/s: fs
                           (was: hdfs client)

> FsShell have resource leak
> --------------------------
>
>                 Key: HADOOP-6848
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6848
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.2, 0.20.1, 0.20.3, 0.20-append
>            Reporter: Vladimir Klimontovich
>             Fix For: 0.20.3, 0.20-append
>
>
> When FsShell exectutes -text command it's using TextRecordInputStream. This class doesn't close inbuf and outbuf (with underlying socket in case of HDFS fs) opened in constructor. It's ok in "classic" "command per JVM" case when -text command is being used from command-line. But when FsShell is being used in same JVM several times (for example via http://martiansoftware.com/nailgun/index.html project) it cause socket leak.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9333-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 18:37:45 2010
Return-Path: <common-issues-return-9333-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38270 invoked from network); 2 Jul 2010 18:37:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 18:37:44 -0000
Received: (qmail 34088 invoked by uid 500); 2 Jul 2010 18:37:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34029 invoked by uid 500); 2 Jul 2010 18:37:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34021 invoked by uid 99); 2 Jul 2010 18:37:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 18:37:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 18:37:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o62IToe7012347
	for <common-issues@hadoop.apache.org>; Fri, 2 Jul 2010 18:29:50 GMT
Message-ID: <12672882.178271278095390170.JavaMail.jira@thor>
Date: Fri, 2 Jul 2010 14:29:50 -0400 (EDT)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6756) Clean up and add documentation for
 configuration keys in CommonConfigurationKeys.java
In-Reply-To: <23380183.20521273191468124.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Shvachko resolved HADOOP-6756.
-----------------------------------------

     Hadoop Flags: [Reviewed]
    Fix Version/s:     (was: 0.20.3)
       Resolution: Fixed

I just committed this. Thank you Erik.

> Clean up and add documentation for configuration keys in CommonConfigurationKeys.java
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6756
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6756
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.3, 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6756-0.22.0-1.patch, HADOOP-6756-0.22.0-2.patch, HADOOP-6756-0.22.0-3.patch, HADOOP-6756-0.22.0.patch, jira.HADOOP-6756-0.20-1-FS_DEFAULT_NAME_KEY.patch, jira.HADOOP-6756-0.20-1.patch, jira.HADOOP-6756-0.20.patch
>
>
> Configuration keys in CommonConfigurationKeys.java should be cleaned up and documented (javadoc comments, appropriate *-default.xml descriptions).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9334-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 18:43:49 2010
Return-Path: <common-issues-return-9334-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39995 invoked from network); 2 Jul 2010 18:43:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 18:43:49 -0000
Received: (qmail 53236 invoked by uid 500); 2 Jul 2010 18:43:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53194 invoked by uid 500); 2 Jul 2010 18:43:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53186 invoked by uid 99); 2 Jul 2010 18:43:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 18:43:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 18:43:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o62IZsYO012448
	for <common-issues@hadoop.apache.org>; Fri, 2 Jul 2010 18:35:54 GMT
Message-ID: <12142679.178521278095754448.JavaMail.jira@thor>
Date: Fri, 2 Jul 2010 14:35:54 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6756) Clean up and add documentation for
 configuration keys in CommonConfigurationKeys.java
In-Reply-To: <23380183.20521273191468124.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884771#action_12884771 ] 

Hudson commented on HADOOP-6756:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #318 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/318/])
    HADOOP-6756. Documentation for common configuration keys. Contributed by Erik Steffl.


> Clean up and add documentation for configuration keys in CommonConfigurationKeys.java
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6756
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6756
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.3, 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6756-0.22.0-1.patch, HADOOP-6756-0.22.0-2.patch, HADOOP-6756-0.22.0-3.patch, HADOOP-6756-0.22.0.patch, jira.HADOOP-6756-0.20-1-FS_DEFAULT_NAME_KEY.patch, jira.HADOOP-6756-0.20-1.patch, jira.HADOOP-6756-0.20.patch
>
>
> Configuration keys in CommonConfigurationKeys.java should be cleaned up and documented (javadoc comments, appropriate *-default.xml descriptions).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9335-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 18:46:16 2010
Return-Path: <common-issues-return-9335-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41063 invoked from network); 2 Jul 2010 18:46:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 18:46:16 -0000
Received: (qmail 58344 invoked by uid 500); 2 Jul 2010 18:46:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58287 invoked by uid 500); 2 Jul 2010 18:46:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58279 invoked by uid 99); 2 Jul 2010 18:46:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 18:46:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 18:46:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o62IjpMV012523
	for <common-issues@hadoop.apache.org>; Fri, 2 Jul 2010 18:45:51 GMT
Message-ID: <24828873.178691278096351841.JavaMail.jira@thor>
Date: Fri, 2 Jul 2010 14:45:51 -0400 (EDT)
From: "Michele Catasta (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6056) Use java.net.preferIPv4Stack to
 force IPv4
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884774#action_12884774 ] 

Michele Catasta commented on HADOOP-6056:
-----------------------------------------

bash-only patch: lack of javadoc is self-explanatory.
@Steve, @Todd: could you please state if you tested this patch in your cluster as well?

> Use java.net.preferIPv4Stack to force IPv4
> ------------------------------------------
>
>                 Key: HADOOP-6056
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6056
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Steve Loughran
>            Assignee: Michele Catasta
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6056.patch, hadoop-6056.txt
>
>
> This was mentioned on HADOOP-3427, there is a property,  java.net.preferIPv4Stack, which you set to true for the java net process to switch to IPv4 everywhere. 
> As Hadoop doesn't work on IPv6, this should be set to true in the startup scripts. Hopefully this will ensure that Jetty will also pick it up.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9336-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 21:01:48 2010
Return-Path: <common-issues-return-9336-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71958 invoked from network); 2 Jul 2010 21:01:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 21:01:48 -0000
Received: (qmail 81305 invoked by uid 500); 2 Jul 2010 21:01:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81252 invoked by uid 500); 2 Jul 2010 21:01:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81244 invoked by uid 99); 2 Jul 2010 21:01:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 21:01:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 21:01:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o62Krsfr013754
	for <common-issues@hadoop.apache.org>; Fri, 2 Jul 2010 20:53:54 GMT
Message-ID: <7319459.181571278104034283.JavaMail.jira@thor>
Date: Fri, 2 Jul 2010 16:53:54 -0400 (EDT)
From: "Bharath Mundlapudi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6713) The RPC server Listener thread is a
 scalability bottleneck
In-Reply-To: <20387500.4981271656489920.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bharath Mundlapudi updated HADOOP-6713:
---------------------------------------

    Attachment: HADOOP-6713-rel20.patch

Attached the patch for Y20S branch.

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6713-rel20.patch, HADOOP-6713.2.patch, HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9337-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 21:59:50 2010
Return-Path: <common-issues-return-9337-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83160 invoked from network); 2 Jul 2010 21:59:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 21:59:49 -0000
Received: (qmail 19520 invoked by uid 500); 2 Jul 2010 21:59:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19499 invoked by uid 500); 2 Jul 2010 21:59:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19488 invoked by uid 99); 2 Jul 2010 21:59:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 21:59:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 21:59:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o62Lpt5l014096
	for <common-issues@hadoop.apache.org>; Fri, 2 Jul 2010 21:51:55 GMT
Message-ID: <30115294.182251278107515404.JavaMail.jira@thor>
Date: Fri, 2 Jul 2010 17:51:55 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6713) The RPC server Listener thread is a
 scalability bottleneck
In-Reply-To: <20387500.4981271656489920.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884817#action_12884817 ] 

Suresh Srinivas commented on HADOOP-6713:
-----------------------------------------

Comments for Y20 version of the patch.
# Please remove YAHOO-CHANGES.txt from the patch
# Remove blank changes that is diff between two patches: lines 51, 73, 138, 171, 203, 220. There could be others.
# line 283 exceeds 80 chars


> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6713-rel20.patch, HADOOP-6713.2.patch, HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9338-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 23:36:20 2010
Return-Path: <common-issues-return-9338-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5347 invoked from network); 2 Jul 2010 23:36:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 23:36:20 -0000
Received: (qmail 92115 invoked by uid 500); 2 Jul 2010 23:36:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92048 invoked by uid 500); 2 Jul 2010 23:36:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92039 invoked by uid 99); 2 Jul 2010 23:36:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 23:36:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 23:36:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o62NZtVs015052
	for <common-issues@hadoop.apache.org>; Fri, 2 Jul 2010 23:35:55 GMT
Message-ID: <28355889.184541278113754997.JavaMail.jira@thor>
Date: Fri, 2 Jul 2010 19:35:54 -0400 (EDT)
From: "Bharath Mundlapudi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6713) The RPC server Listener thread is a
 scalability bottleneck
In-Reply-To: <20387500.4981271656489920.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bharath Mundlapudi updated HADOOP-6713:
---------------------------------------

    Attachment: HADOOP-6713-rel20.patch.1

Attached the patch addressing the comments.

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6713-rel20.patch, HADOOP-6713-rel20.patch.1, HADOOP-6713.2.patch, HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9339-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 23:38:21 2010
Return-Path: <common-issues-return-9339-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5891 invoked from network); 2 Jul 2010 23:38:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 23:38:20 -0000
Received: (qmail 93586 invoked by uid 500); 2 Jul 2010 23:38:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93260 invoked by uid 500); 2 Jul 2010 23:38:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93252 invoked by uid 99); 2 Jul 2010 23:38:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 23:38:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 23:38:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o62Nbthg015124
	for <common-issues@hadoop.apache.org>; Fri, 2 Jul 2010 23:37:55 GMT
Message-ID: <13775566.184781278113875906.JavaMail.jira@thor>
Date: Fri, 2 Jul 2010 19:37:55 -0400 (EDT)
From: "Bharath Mundlapudi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6713) The RPC server Listener thread is a
 scalability bottleneck
In-Reply-To: <20387500.4981271656489920.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bharath Mundlapudi updated HADOOP-6713:
---------------------------------------

    Attachment:     (was: HADOOP-6713-rel20.patch.1)

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6713-rel20.2.patch, HADOOP-6713-rel20.patch, HADOOP-6713.2.patch, HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9340-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 02 23:38:22 2010
Return-Path: <common-issues-return-9340-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6006 invoked from network); 2 Jul 2010 23:38:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jul 2010 23:38:21 -0000
Received: (qmail 93735 invoked by uid 500); 2 Jul 2010 23:38:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93705 invoked by uid 500); 2 Jul 2010 23:38:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93697 invoked by uid 99); 2 Jul 2010 23:38:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 23:38:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jul 2010 23:38:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o62NbwMB015178
	for <common-issues@hadoop.apache.org>; Fri, 2 Jul 2010 23:37:58 GMT
Message-ID: <20481285.184961278113878261.JavaMail.jira@thor>
Date: Fri, 2 Jul 2010 19:37:58 -0400 (EDT)
From: "Bharath Mundlapudi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6713) The RPC server Listener thread is a
 scalability bottleneck
In-Reply-To: <20387500.4981271656489920.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bharath Mundlapudi updated HADOOP-6713:
---------------------------------------

    Attachment: HADOOP-6713-rel20.2.patch

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6713-rel20.2.patch, HADOOP-6713-rel20.patch, HADOOP-6713.2.patch, HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9341-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 03 00:05:16 2010
Return-Path: <common-issues-return-9341-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12995 invoked from network); 3 Jul 2010 00:05:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jul 2010 00:05:16 -0000
Received: (qmail 5414 invoked by uid 500); 3 Jul 2010 00:05:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5340 invoked by uid 500); 3 Jul 2010 00:05:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5332 invoked by uid 99); 3 Jul 2010 00:05:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Jul 2010 00:05:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Jul 2010 00:05:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6304pRh015433
	for <common-issues@hadoop.apache.org>; Sat, 3 Jul 2010 00:04:51 GMT
Message-ID: <14974890.185481278115491881.JavaMail.jira@thor>
Date: Fri, 2 Jul 2010 20:04:51 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6584) Provide Kerberized SSL encryption for
 webservices
In-Reply-To: <713706873.424711266804869916.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6584:
--------------------------------

           Status: Resolved  (was: Patch Available)
     Hadoop Flags: [Reviewed]
    Fix Version/s: 0.22.0
       Resolution: Fixed

I've committed this.  Resolving as fixed.

> Provide Kerberized SSL encryption for webservices
> -------------------------------------------------
>
>                 Key: HADOOP-6584
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6584
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: c6584-02.patch, HADOOP-6584-FixJavadoc-Y20.patch, HADOOP-6584-Y20-1.patch, HADOOP-6584-Y20-2.patch, HADOOP-6584-Y20-3.patch, HADOOP-6584-Y20-4.patch, HADOOP-6584.patch
>
>
> Some web services should be authenticated via SSL backed by Kerberos, both to provide cross-cluster secure communication and to provide intra-cluster server authentication for services such as the {Name,SecondaryName,Backup,Checkpoint}node's image transfer and balancer.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9342-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 03 00:17:48 2010
Return-Path: <common-issues-return-9342-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14319 invoked from network); 3 Jul 2010 00:17:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jul 2010 00:17:48 -0000
Received: (qmail 13252 invoked by uid 500); 3 Jul 2010 00:17:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13179 invoked by uid 500); 3 Jul 2010 00:17:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13171 invoked by uid 99); 3 Jul 2010 00:17:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Jul 2010 00:17:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Jul 2010 00:17:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6309rCe015505
	for <common-issues@hadoop.apache.org>; Sat, 3 Jul 2010 00:09:54 GMT
Message-ID: <2913806.185701278115793661.JavaMail.jira@thor>
Date: Fri, 2 Jul 2010 20:09:53 -0400 (EDT)
From: "Bharath Mundlapudi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6713) The RPC server Listener thread is a
 scalability bottleneck
In-Reply-To: <20387500.4981271656489920.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bharath Mundlapudi updated HADOOP-6713:
---------------------------------------

    Attachment: HADOOP-6713-rel20.3.patch

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6713-rel20.2.patch, HADOOP-6713-rel20.3.patch, HADOOP-6713-rel20.patch, HADOOP-6713.2.patch, HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9343-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 03 00:33:51 2010
Return-Path: <common-issues-return-9343-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23026 invoked from network); 3 Jul 2010 00:33:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jul 2010 00:33:50 -0000
Received: (qmail 23321 invoked by uid 500); 3 Jul 2010 00:33:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23131 invoked by uid 500); 3 Jul 2010 00:33:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23041 invoked by uid 99); 3 Jul 2010 00:33:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Jul 2010 00:33:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Jul 2010 00:33:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o630PuNm015653
	for <common-issues@hadoop.apache.org>; Sat, 3 Jul 2010 00:25:56 GMT
Message-ID: <221556.186111278116756579.JavaMail.jira@thor>
Date: Fri, 2 Jul 2010 20:25:56 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6584) Provide Kerberized SSL encryption
 for webservices
In-Reply-To: <713706873.424711266804869916.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884870#action_12884870 ] 

Hudson commented on HADOOP-6584:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #319 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/319/])
    HADOOP-6584. Provide Kerberized SSL encryption for webservices.


> Provide Kerberized SSL encryption for webservices
> -------------------------------------------------
>
>                 Key: HADOOP-6584
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6584
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: c6584-02.patch, HADOOP-6584-FixJavadoc-Y20.patch, HADOOP-6584-Y20-1.patch, HADOOP-6584-Y20-2.patch, HADOOP-6584-Y20-3.patch, HADOOP-6584-Y20-4.patch, HADOOP-6584.patch
>
>
> Some web services should be authenticated via SSL backed by Kerberos, both to provide cross-cluster secure communication and to provide intra-cluster server authentication for services such as the {Name,SecondaryName,Backup,Checkpoint}node's image transfer and balancer.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9344-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 03 02:41:48 2010
Return-Path: <common-issues-return-9344-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44508 invoked from network); 3 Jul 2010 02:41:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jul 2010 02:41:47 -0000
Received: (qmail 79745 invoked by uid 500); 3 Jul 2010 02:41:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79578 invoked by uid 500); 3 Jul 2010 02:41:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79562 invoked by uid 99); 3 Jul 2010 02:41:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Jul 2010 02:41:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Jul 2010 02:41:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o632XqrP016309
	for <common-issues@hadoop.apache.org>; Sat, 3 Jul 2010 02:33:52 GMT
Message-ID: <24719292.187301278124432110.JavaMail.jira@thor>
Date: Fri, 2 Jul 2010 22:33:52 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6584) Provide Kerberized SSL encryption
 for webservices
In-Reply-To: <713706873.424711266804869916.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884890#action_12884890 ] 

Hudson commented on HADOOP-6584:
--------------------------------

Integrated in Hadoop-Hdfs-trunk-Commit #328 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/328/])
    HDFS-1004. Update NN to support Kerberized SSL from HADOOP-6584.


> Provide Kerberized SSL encryption for webservices
> -------------------------------------------------
>
>                 Key: HADOOP-6584
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6584
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: c6584-02.patch, HADOOP-6584-FixJavadoc-Y20.patch, HADOOP-6584-Y20-1.patch, HADOOP-6584-Y20-2.patch, HADOOP-6584-Y20-3.patch, HADOOP-6584-Y20-4.patch, HADOOP-6584.patch
>
>
> Some web services should be authenticated via SSL backed by Kerberos, both to provide cross-cluster secure communication and to provide intra-cluster server authentication for services such as the {Name,SecondaryName,Backup,Checkpoint}node's image transfer and balancer.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9345-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 03 11:23:52 2010
Return-Path: <common-issues-return-9345-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39734 invoked from network); 3 Jul 2010 11:23:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jul 2010 11:23:52 -0000
Received: (qmail 38064 invoked by uid 500); 3 Jul 2010 11:23:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37924 invoked by uid 500); 3 Jul 2010 11:23:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37905 invoked by uid 99); 3 Jul 2010 11:23:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Jul 2010 11:23:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Jul 2010 11:23:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o63BFqMD018525
	for <common-issues@hadoop.apache.org>; Sat, 3 Jul 2010 11:15:53 GMT
Message-ID: <3017784.188361278155752939.JavaMail.jira@thor>
Date: Sat, 3 Jul 2010 07:15:52 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6584) Provide Kerberized SSL encryption
 for webservices
In-Reply-To: <713706873.424711266804869916.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884907#action_12884907 ] 

Hudson commented on HADOOP-6584:
--------------------------------

Integrated in Hadoop-Common-trunk #385 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/385/])
    HADOOP-6584. Provide Kerberized SSL encryption for webservices.


> Provide Kerberized SSL encryption for webservices
> -------------------------------------------------
>
>                 Key: HADOOP-6584
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6584
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: c6584-02.patch, HADOOP-6584-FixJavadoc-Y20.patch, HADOOP-6584-Y20-1.patch, HADOOP-6584-Y20-2.patch, HADOOP-6584-Y20-3.patch, HADOOP-6584-Y20-4.patch, HADOOP-6584.patch
>
>
> Some web services should be authenticated via SSL backed by Kerberos, both to provide cross-cluster secure communication and to provide intra-cluster server authentication for services such as the {Name,SecondaryName,Backup,Checkpoint}node's image transfer and balancer.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9346-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 03 11:23:53 2010
Return-Path: <common-issues-return-9346-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39763 invoked from network); 3 Jul 2010 11:23:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jul 2010 11:23:53 -0000
Received: (qmail 38121 invoked by uid 500); 3 Jul 2010 11:23:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37934 invoked by uid 500); 3 Jul 2010 11:23:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37901 invoked by uid 99); 3 Jul 2010 11:23:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Jul 2010 11:23:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Jul 2010 11:23:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o63BFrxT018537
	for <common-issues@hadoop.apache.org>; Sat, 3 Jul 2010 11:15:53 GMT
Message-ID: <5869194.188401278155753764.JavaMail.jira@thor>
Date: Sat, 3 Jul 2010 07:15:53 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6756) Clean up and add documentation for
 configuration keys in CommonConfigurationKeys.java
In-Reply-To: <23380183.20521273191468124.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884908#action_12884908 ] 

Hudson commented on HADOOP-6756:
--------------------------------

Integrated in Hadoop-Common-trunk #385 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/385/])
    HADOOP-6756. Documentation for common configuration keys. Contributed by Erik Steffl.


> Clean up and add documentation for configuration keys in CommonConfigurationKeys.java
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6756
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6756
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.3, 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6756-0.22.0-1.patch, HADOOP-6756-0.22.0-2.patch, HADOOP-6756-0.22.0-3.patch, HADOOP-6756-0.22.0.patch, jira.HADOOP-6756-0.20-1-FS_DEFAULT_NAME_KEY.patch, jira.HADOOP-6756-0.20-1.patch, jira.HADOOP-6756-0.20.patch
>
>
> Configuration keys in CommonConfigurationKeys.java should be cleaned up and documented (javadoc comments, appropriate *-default.xml descriptions).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9347-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 03 17:56:45 2010
Return-Path: <common-issues-return-9347-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14732 invoked from network); 3 Jul 2010 17:56:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jul 2010 17:56:45 -0000
Received: (qmail 14914 invoked by uid 500); 3 Jul 2010 17:56:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14872 invoked by uid 500); 3 Jul 2010 17:56:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14864 invoked by uid 99); 3 Jul 2010 17:56:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Jul 2010 17:56:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Jul 2010 17:56:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o63HmnAb020012
	for <common-issues@hadoop.apache.org>; Sat, 3 Jul 2010 17:48:49 GMT
Message-ID: <28072519.190301278179329541.JavaMail.jira@thor>
Date: Sat, 3 Jul 2010 13:48:49 -0400 (EDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6849) Have getLocalPathForWrite
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Have getLocalPathForWrite
-------------------------

                 Key: HADOOP-6849
                 URL: https://issues.apache.org/jira/browse/HADOOP-6849
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs
    Affects Versions: 0.20.2
            Reporter: Steve Loughran
            Priority: Minor


A stack trace makes it way to me, of a reduce failing
{code}
Caused by: org\.apache\.hadoop\.util\.DiskChecker$DiskErrorException: Could not find any valid local directory for file:/mnt/data/dfs/data/mapred/local/taskTracker/jobcache/job_201007011427_0001/attempt_201007011427_0001_r_000000_1/output/map_96\.out
      at org\.apache\.hadoop\.fs\.LocalDirAllocator$AllocatorPerContext\.getLocalPathForWrite(LocalDirAllocator\.java:343)

      at org\.apache\.hadoop\.fs\.LocalDirAllocator\.getLocalPathForWrite(LocalDirAllocator\.java:124)

      at org\.apache\.hadoop\.mapred\.ReduceTask$ReduceCopier$LocalFSMerger\.run(ReduceTask\.java:2434)
{code}

We're probably running out of HDD space, if not its configuration problems. Either way, some more hints in the exception would be handy.
# Include the size of the output file looked for if known
# Include the list of dirs examined and their reason for rejection (not found or if not enough room, available space).
This would make it easier to diagnose problems after the event, with nothing but emailed logs for diagnostics.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9348-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 03 18:36:46 2010
Return-Path: <common-issues-return-9348-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33728 invoked from network); 3 Jul 2010 18:36:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jul 2010 18:36:46 -0000
Received: (qmail 33507 invoked by uid 500); 3 Jul 2010 18:36:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33431 invoked by uid 500); 3 Jul 2010 18:36:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33423 invoked by uid 99); 3 Jul 2010 18:36:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Jul 2010 18:36:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 03 Jul 2010 18:36:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o63ISpdQ020209
	for <common-issues@hadoop.apache.org>; Sat, 3 Jul 2010 18:28:52 GMT
Message-ID: <10097968.190561278181731785.JavaMail.jira@thor>
Date: Sat, 3 Jul 2010 14:28:51 -0400 (EDT)
From: "Brahma Reddy Battula (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-445) Parallel data/socket writing for
 DFSOutputStream
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884959#action_12884959 ] 

Brahma Reddy Battula commented on HADOOP-445:
---------------------------------------------

I am using the 20.1 version ,by using two threads i am doing the all the operations(write,read,copyfromlocal..) continously after some time im getting the following exception

ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(192.168.3.26:50010, storageID=DS-2095362429-192.168.3.26-50010-1244004238916, infoPort=50075, ipcPort=50020):DataXceiver
java.net.SocketTimeoutException: 480000 millis timeout while waiting for channel to be ready for write. ch : java.nio.channels.SocketChannel[connected local=/192.168.3.26:50010 remote=/192.168.3.26:34243]

> Parallel data/socket writing for DFSOutputStream
> ------------------------------------------------
>
>                 Key: HADOOP-445
>                 URL: https://issues.apache.org/jira/browse/HADOOP-445
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.5.0
>            Reporter: Benjamin Reed
>            Assignee: Sameer Paranjpye
>         Attachments: fastClientWrite.patch
>
>
> Currently, as DFS clients output blocks they write the entire block to disk before starting to transmit to the datanode. By writing to disk the client is able to retry a block write if the datanode files in the middle of a block transfer. Writing to disk and then to the datanode adds latency. Hopefully, the common case is that block transfers to datanodes are successful. This patch writes to the datanode and the disk in parallel. If the write to the datanode fails, it falls back to current behavior.
> In my tests of transmits of 237M and 946M datasets using -copyFromLocal I'm seeing a 20-25% improvement in throughput.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9349-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 05 10:39:19 2010
Return-Path: <common-issues-return-9349-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88625 invoked from network); 5 Jul 2010 10:39:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jul 2010 10:39:19 -0000
Received: (qmail 27675 invoked by uid 500); 5 Jul 2010 10:39:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27540 invoked by uid 500); 5 Jul 2010 10:39:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27532 invoked by uid 99); 5 Jul 2010 10:39:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jul 2010 10:39:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jul 2010 10:39:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o65AcoP0006079
	for <common-issues@hadoop.apache.org>; Mon, 5 Jul 2010 10:38:50 GMT
Message-ID: <27205205.199821278326330526.JavaMail.jira@thor>
Date: Mon, 5 Jul 2010 06:38:50 -0400 (EDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6056) Use java.net.preferIPv4Stack to
 force IPv4
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885174#action_12885174 ] 

Steve Loughran commented on HADOOP-6056:
----------------------------------------

I don't start my machines via the shell scripts, so can't say whether or not it works; i've tweaked how my own JVMs start and added some in-JVM tests for IPV4

looking at the code, my main concern is what happens if there isn't an /sbin/sysctl on the path, like on cygwin, OS/X or other systems. We probably need a check round there to look for that function before calling it...

> Use java.net.preferIPv4Stack to force IPv4
> ------------------------------------------
>
>                 Key: HADOOP-6056
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6056
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Steve Loughran
>            Assignee: Michele Catasta
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6056.patch, hadoop-6056.txt
>
>
> This was mentioned on HADOOP-3427, there is a property,  java.net.preferIPv4Stack, which you set to true for the java net process to switch to IPv4 everywhere. 
> As Hadoop doesn't work on IPv6, this should be set to true in the startup scripts. Hopefully this will ensure that Jetty will also pick it up.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9350-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 05 11:45:18 2010
Return-Path: <common-issues-return-9350-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15498 invoked from network); 5 Jul 2010 11:45:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jul 2010 11:45:18 -0000
Received: (qmail 69604 invoked by uid 500); 5 Jul 2010 11:45:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69476 invoked by uid 500); 5 Jul 2010 11:45:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69467 invoked by uid 99); 5 Jul 2010 11:45:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jul 2010 11:45:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jul 2010 11:45:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o65Bin8q006538
	for <common-issues@hadoop.apache.org>; Mon, 5 Jul 2010 11:44:50 GMT
Message-ID: <3569302.200491278330289727.JavaMail.jira@thor>
Date: Mon, 5 Jul 2010 07:44:49 -0400 (EDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6849) Have
 LocalDirAllocator.AllocatorPerContext.getLocalPathForWrite fail more
 meaningfully
In-Reply-To: <28072519.190301278179329541.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885187#action_12885187 ] 

Steve Loughran commented on HADOOP-6849:
----------------------------------------

Obviously, building up an error string during the directory search is a waste of effort for every successful operation, so the diagnostics should only be created on failure. A list of directories and the target file size may be enough, as it would catch directory config issues.

> Have LocalDirAllocator.AllocatorPerContext.getLocalPathForWrite fail more meaningfully
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6849
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6849
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.20.2
>            Reporter: Steve Loughran
>            Priority: Minor
>
> A stack trace makes it way to me, of a reduce failing
> {code}
> Caused by: org\.apache\.hadoop\.util\.DiskChecker$DiskErrorException: Could not find any valid local directory for file:/mnt/data/dfs/data/mapred/local/taskTracker/jobcache/job_201007011427_0001/attempt_201007011427_0001_r_000000_1/output/map_96\.out
>       at org\.apache\.hadoop\.fs\.LocalDirAllocator$AllocatorPerContext\.getLocalPathForWrite(LocalDirAllocator\.java:343)
>       at org\.apache\.hadoop\.fs\.LocalDirAllocator\.getLocalPathForWrite(LocalDirAllocator\.java:124)
>       at org\.apache\.hadoop\.mapred\.ReduceTask$ReduceCopier$LocalFSMerger\.run(ReduceTask\.java:2434)
> {code}
> We're probably running out of HDD space, if not its configuration problems. Either way, some more hints in the exception would be handy.
> # Include the size of the output file looked for if known
> # Include the list of dirs examined and their reason for rejection (not found or if not enough room, available space).
> This would make it easier to diagnose problems after the event, with nothing but emailed logs for diagnostics.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9351-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 05 11:50:47 2010
Return-Path: <common-issues-return-9351-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16143 invoked from network); 5 Jul 2010 11:50:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jul 2010 11:50:47 -0000
Received: (qmail 73540 invoked by uid 500); 5 Jul 2010 11:50:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73313 invoked by uid 500); 5 Jul 2010 11:50:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73288 invoked by uid 99); 5 Jul 2010 11:50:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jul 2010 11:50:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jul 2010 11:50:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o65Bgohc006519
	for <common-issues@hadoop.apache.org>; Mon, 5 Jul 2010 11:42:50 GMT
Message-ID: <25265206.200451278330170352.JavaMail.jira@thor>
Date: Mon, 5 Jul 2010 07:42:50 -0400 (EDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6849) Have
 LocalDirAllocator.AllocatorPerContext.getLocalPathForWrite fail more
 meaningfully
In-Reply-To: <28072519.190301278179329541.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steve Loughran updated HADOOP-6849:
-----------------------------------

    Summary: Have LocalDirAllocator.AllocatorPerContext.getLocalPathForWrite fail more meaningfully  (was: Have getLocalPathForWrite)

> Have LocalDirAllocator.AllocatorPerContext.getLocalPathForWrite fail more meaningfully
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6849
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6849
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.20.2
>            Reporter: Steve Loughran
>            Priority: Minor
>
> A stack trace makes it way to me, of a reduce failing
> {code}
> Caused by: org\.apache\.hadoop\.util\.DiskChecker$DiskErrorException: Could not find any valid local directory for file:/mnt/data/dfs/data/mapred/local/taskTracker/jobcache/job_201007011427_0001/attempt_201007011427_0001_r_000000_1/output/map_96\.out
>       at org\.apache\.hadoop\.fs\.LocalDirAllocator$AllocatorPerContext\.getLocalPathForWrite(LocalDirAllocator\.java:343)
>       at org\.apache\.hadoop\.fs\.LocalDirAllocator\.getLocalPathForWrite(LocalDirAllocator\.java:124)
>       at org\.apache\.hadoop\.mapred\.ReduceTask$ReduceCopier$LocalFSMerger\.run(ReduceTask\.java:2434)
> {code}
> We're probably running out of HDD space, if not its configuration problems. Either way, some more hints in the exception would be handy.
> # Include the size of the output file looked for if known
> # Include the list of dirs examined and their reason for rejection (not found or if not enough room, available space).
> This would make it easier to diagnose problems after the event, with nothing but emailed logs for diagnostics.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9352-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 05 18:03:18 2010
Return-Path: <common-issues-return-9352-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8761 invoked from network); 5 Jul 2010 18:03:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jul 2010 18:03:18 -0000
Received: (qmail 99678 invoked by uid 500); 5 Jul 2010 18:03:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99650 invoked by uid 500); 5 Jul 2010 18:03:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99642 invoked by uid 99); 5 Jul 2010 18:03:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jul 2010 18:03:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jul 2010 18:03:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o65I2r3g009183
	for <common-issues@hadoop.apache.org>; Mon, 5 Jul 2010 18:02:53 GMT
Message-ID: <15539820.204421278352973438.JavaMail.jira@thor>
Date: Mon, 5 Jul 2010 14:02:53 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6056) Use java.net.preferIPv4Stack to
 force IPv4
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885287#action_12885287 ] 

Todd Lipcon commented on HADOOP-6056:
-------------------------------------

bq. @Steve, @Todd: could you please state if you tested this patch in your cluster as well?

I've tested on my laptop, but not out in the wide world of varied deployments.

> Use java.net.preferIPv4Stack to force IPv4
> ------------------------------------------
>
>                 Key: HADOOP-6056
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6056
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Steve Loughran
>            Assignee: Michele Catasta
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6056.patch, hadoop-6056.txt
>
>
> This was mentioned on HADOOP-3427, there is a property,  java.net.preferIPv4Stack, which you set to true for the java net process to switch to IPv4 everywhere. 
> As Hadoop doesn't work on IPv6, this should be set to true in the startup scripts. Hopefully this will ensure that Jetty will also pick it up.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9353-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 05 18:12:45 2010
Return-Path: <common-issues-return-9353-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13957 invoked from network); 5 Jul 2010 18:12:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jul 2010 18:12:45 -0000
Received: (qmail 6056 invoked by uid 500); 5 Jul 2010 18:12:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6006 invoked by uid 500); 5 Jul 2010 18:12:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5998 invoked by uid 99); 5 Jul 2010 18:12:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jul 2010 18:12:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jul 2010 18:12:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o65I4ou0009216
	for <common-issues@hadoop.apache.org>; Mon, 5 Jul 2010 18:04:50 GMT
Message-ID: <13664275.204511278353090225.JavaMail.jira@thor>
Date: Mon, 5 Jul 2010 14:04:50 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6849) Have
 LocalDirAllocator.AllocatorPerContext.getLocalPathForWrite fail more
 meaningfully
In-Reply-To: <28072519.190301278179329541.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885289#action_12885289 ] 

Todd Lipcon commented on HADOOP-6849:
-------------------------------------

I agree, debugging this issue is always a pain with customer clusters.

> Have LocalDirAllocator.AllocatorPerContext.getLocalPathForWrite fail more meaningfully
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6849
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6849
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.20.2
>            Reporter: Steve Loughran
>            Priority: Minor
>
> A stack trace makes it way to me, of a reduce failing
> {code}
> Caused by: org\.apache\.hadoop\.util\.DiskChecker$DiskErrorException: Could not find any valid local directory for file:/mnt/data/dfs/data/mapred/local/taskTracker/jobcache/job_201007011427_0001/attempt_201007011427_0001_r_000000_1/output/map_96\.out
>       at org\.apache\.hadoop\.fs\.LocalDirAllocator$AllocatorPerContext\.getLocalPathForWrite(LocalDirAllocator\.java:343)
>       at org\.apache\.hadoop\.fs\.LocalDirAllocator\.getLocalPathForWrite(LocalDirAllocator\.java:124)
>       at org\.apache\.hadoop\.mapred\.ReduceTask$ReduceCopier$LocalFSMerger\.run(ReduceTask\.java:2434)
> {code}
> We're probably running out of HDD space, if not its configuration problems. Either way, some more hints in the exception would be handy.
> # Include the size of the output file looked for if known
> # Include the list of dirs examined and their reason for rejection (not found or if not enough room, available space).
> This would make it easier to diagnose problems after the event, with nothing but emailed logs for diagnostics.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9354-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 14:32:16 2010
Return-Path: <common-issues-return-9354-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12223 invoked from network); 6 Jul 2010 14:32:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 14:32:16 -0000
Received: (qmail 40525 invoked by uid 500); 6 Jul 2010 14:32:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40487 invoked by uid 500); 6 Jul 2010 14:32:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40257 invoked by uid 99); 6 Jul 2010 14:32:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 14:32:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 14:32:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o66EVoxY017782
	for <common-issues@hadoop.apache.org>; Tue, 6 Jul 2010 14:31:50 GMT
Message-ID: <1642525.216061278426710500.JavaMail.jira@thor>
Date: Tue, 6 Jul 2010 10:31:50 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6847) Problem staging 0.21.0 artifacts to
 Apache Nexus Maven Repository
In-Reply-To: <28118738.168371278054229909.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6847:
---------------------------------------

    Attachment: hadoop-6847.patch

this patch should fix the issue with uploading artifacts to the nexus maven repo.

command to upload artifact to the staging rep is ;

ant mvn-deploy -Dversion="standard version string" -Drepo=staging

command to upload artifact to the snapshot repo is:

ant mvn-deploy

I 've tried publishing artifacts to the staging repo and it worked.
https://repository.apache.org/index.html#staging;6a441994c87797,  this stating url is not closed yet for to get a public view. 


> Problem staging 0.21.0 artifacts to Apache Nexus Maven Repository
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6847
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6847
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Tom White
>            Priority: Blocker
>         Attachments: hadoop-6847.patch
>
>
> I get this error:
> {noformat}
> -mvn-system-deploy:
> [artifact:install-provider] Installing provider: org.apache.maven.wagon:wagon-http:jar:1.0-beta-2:runtime
> [artifact:deploy] Deploying to https://repository.apache.org/content/repositories/snapshots
> [artifact:deploy] Uploading: org/apache/hadoop/hadoop-common-instrumented/0.21.0/hadoop-common-instrumented-0.21.0.jar to apache.snapshots.https
> [artifact:deploy] Uploaded 1331K
> [artifact:deploy] An error has occurred while processing the Maven artifact tasks.
> [artifact:deploy]  Diagnosis:
> [artifact:deploy] 
> [artifact:deploy] Error deploying artifact 'org.apache.hadoop:hadoop-common-instrumented:jar': Error deploying artifact: Failed to transfer file: https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common-instrumented/0.21.0/hadoop-common-instrumented-0.21.0.jar. Return code is: 400
> {noformat}
> Note that 400 is "Bad Request", not an authentication error (401).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9355-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 17:12:18 2010
Return-Path: <common-issues-return-9355-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62745 invoked from network); 6 Jul 2010 17:12:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 17:12:18 -0000
Received: (qmail 60032 invoked by uid 500); 6 Jul 2010 17:12:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59963 invoked by uid 500); 6 Jul 2010 17:12:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59955 invoked by uid 99); 6 Jul 2010 17:12:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 17:12:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 17:12:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o66HBrwq019546
	for <common-issues@hadoop.apache.org>; Tue, 6 Jul 2010 17:11:53 GMT
Message-ID: <20753722.218911278436313105.JavaMail.jira@thor>
Date: Tue, 6 Jul 2010 13:11:53 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6713) The RPC server Listener thread is a
 scalability bottleneck
In-Reply-To: <20387500.4981271656489920.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885605#action_12885605 ] 

Suresh Srinivas commented on HADOOP-6713:
-----------------------------------------

+1 for the patch.

> The RPC server Listener thread is a scalability bottleneck
> ----------------------------------------------------------
>
>                 Key: HADOOP-6713
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6713
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6713-rel20.2.patch, HADOOP-6713-rel20.3.patch, HADOOP-6713-rel20.patch, HADOOP-6713.2.patch, HADOOP-6713.patch
>
>
> The Hadoop RPC Server implementation has a single Listener thread that reads data from the socket and puts them into a call queue. This means that this single thread can pull RPC requests off the network only as fast as a single CPU can execute. This is a scalability bottlneck in our cluster.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9356-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 19:24:16 2010
Return-Path: <common-issues-return-9356-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6407 invoked from network); 6 Jul 2010 19:24:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 19:24:16 -0000
Received: (qmail 3097 invoked by uid 500); 6 Jul 2010 19:24:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3047 invoked by uid 500); 6 Jul 2010 19:24:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3034 invoked by uid 99); 6 Jul 2010 19:24:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 19:24:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 19:24:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o66JNosB020702
	for <common-issues@hadoop.apache.org>; Tue, 6 Jul 2010 19:23:51 GMT
Message-ID: <2071724.221181278444230928.JavaMail.jira@thor>
Date: Tue, 6 Jul 2010 15:23:50 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6847) Problem staging 0.21.0 artifacts to
 Apache Nexus Maven Repository
In-Reply-To: <28118738.168371278054229909.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6847:
---------------------------------------

          Status: Patch Available  (was: Open)
    Hadoop Flags: [Reviewed]

+1 patch looks good. Let's verify it.

> Problem staging 0.21.0 artifacts to Apache Nexus Maven Repository
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6847
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6847
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Tom White
>            Priority: Blocker
>         Attachments: hadoop-6847.patch
>
>
> I get this error:
> {noformat}
> -mvn-system-deploy:
> [artifact:install-provider] Installing provider: org.apache.maven.wagon:wagon-http:jar:1.0-beta-2:runtime
> [artifact:deploy] Deploying to https://repository.apache.org/content/repositories/snapshots
> [artifact:deploy] Uploading: org/apache/hadoop/hadoop-common-instrumented/0.21.0/hadoop-common-instrumented-0.21.0.jar to apache.snapshots.https
> [artifact:deploy] Uploaded 1331K
> [artifact:deploy] An error has occurred while processing the Maven artifact tasks.
> [artifact:deploy]  Diagnosis:
> [artifact:deploy] 
> [artifact:deploy] Error deploying artifact 'org.apache.hadoop:hadoop-common-instrumented:jar': Error deploying artifact: Failed to transfer file: https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common-instrumented/0.21.0/hadoop-common-instrumented-0.21.0.jar. Return code is: 400
> {noformat}
> Note that 400 is "Bad Request", not an authentication error (401).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9357-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 19:30:18 2010
Return-Path: <common-issues-return-9357-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7274 invoked from network); 6 Jul 2010 19:30:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 19:30:18 -0000
Received: (qmail 10722 invoked by uid 500); 6 Jul 2010 19:30:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9936 invoked by uid 500); 6 Jul 2010 19:30:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9923 invoked by uid 99); 6 Jul 2010 19:30:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 19:30:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 19:30:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o66JTo5m020749
	for <common-issues@hadoop.apache.org>; Tue, 6 Jul 2010 19:29:50 GMT
Message-ID: <17876497.221281278444590187.JavaMail.jira@thor>
Date: Tue, 6 Jul 2010 15:29:50 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6850) Allow sequence file to be created
 with different bytes per checksum
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Allow sequence file to be created with different bytes per checksum
-------------------------------------------------------------------

                 Key: HADOOP-6850
                 URL: https://issues.apache.org/jira/browse/HADOOP-6850
             Project: Hadoop Common
          Issue Type: New Feature
          Components: io
    Affects Versions: 0.20-append
            Reporter: Hairong Kuang
            Assignee: Hairong Kuang
             Fix For: 0.22.0, 0.20-append


This jira allows a sequence file to be created with a configured bytes per checksum. So we could experiment how this parameter could effect the performance of random reads of a sequence file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9358-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 21:52:16 2010
Return-Path: <common-issues-return-9358-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50900 invoked from network); 6 Jul 2010 21:52:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 21:52:15 -0000
Received: (qmail 75964 invoked by uid 500); 6 Jul 2010 21:52:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75894 invoked by uid 500); 6 Jul 2010 21:52:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75667 invoked by uid 99); 6 Jul 2010 21:52:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 21:52:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 21:52:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o66Lpo2e022477
	for <common-issues@hadoop.apache.org>; Tue, 6 Jul 2010 21:51:50 GMT
Message-ID: <28960691.224821278453110527.JavaMail.jira@thor>
Date: Tue, 6 Jul 2010 17:51:50 -0400 (EDT)
From: "Nicolas Spiegelberg (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6851) Fix '$bin' path duplication in setup
 scripts
In-Reply-To: <13265146.224771278452993844.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicolas Spiegelberg updated HADOOP-6851:
----------------------------------------

    Attachment: HADOOP-6851-0.22.patch

> Fix '$bin' path duplication in setup scripts
> --------------------------------------------
>
>                 Key: HADOOP-6851
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6851
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Nicolas Spiegelberg
>            Priority: Trivial
>             Fix For: 0.21.0, 0.22.0, 0.20-append
>
>         Attachments: HADOOP-6851-0.22.patch
>
>
> I have my bash environment setup to echo absolute pathnames when a relative one is specified in 'cd'. This caused problems with all the Hadoop bash scripts because the script accidentally sets the $bin variable twice in this setup. (e.g. would set $bin="/path/bin/hadoop\n/path/bin/hadoop").
> This jira is for common scripts.  I filed a separate jira for HDFS scripts, which share the same pattern.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9359-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 21:57:48 2010
Return-Path: <common-issues-return-9359-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52606 invoked from network); 6 Jul 2010 21:57:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 21:57:48 -0000
Received: (qmail 86135 invoked by uid 500); 6 Jul 2010 21:57:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85804 invoked by uid 500); 6 Jul 2010 21:57:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85580 invoked by uid 99); 6 Jul 2010 21:57:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 21:57:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 21:57:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o66Lnrv9022460
	for <common-issues@hadoop.apache.org>; Tue, 6 Jul 2010 21:49:53 GMT
Message-ID: <13265146.224771278452993844.JavaMail.jira@thor>
Date: Tue, 6 Jul 2010 17:49:53 -0400 (EDT)
From: "Nicolas Spiegelberg (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6851) Fix '$bin' path duplication in setup
 scripts
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Fix '$bin' path duplication in setup scripts
--------------------------------------------

                 Key: HADOOP-6851
                 URL: https://issues.apache.org/jira/browse/HADOOP-6851
             Project: Hadoop Common
          Issue Type: Bug
          Components: scripts
            Reporter: Nicolas Spiegelberg
            Priority: Trivial
             Fix For: 0.21.0, 0.22.0, 0.20-append


I have my bash environment setup to echo absolute pathnames when a relative one is specified in 'cd'. This caused problems with all the Hadoop bash scripts because the script accidentally sets the $bin variable twice in this setup. (e.g. would set $bin="/path/bin/hadoop\n/path/bin/hadoop").

This jira is for common scripts.  I filed a separate jira for HDFS scripts, which share the same pattern.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9360-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 22:22:19 2010
Return-Path: <common-issues-return-9360-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62158 invoked from network); 6 Jul 2010 22:22:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 22:22:18 -0000
Received: (qmail 9402 invoked by uid 500); 6 Jul 2010 22:22:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9355 invoked by uid 500); 6 Jul 2010 22:22:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9347 invoked by uid 99); 6 Jul 2010 22:22:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 22:22:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 22:22:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o66MLsMN022804
	for <common-issues@hadoop.apache.org>; Tue, 6 Jul 2010 22:21:54 GMT
Message-ID: <25896394.225531278454914209.JavaMail.jira@thor>
Date: Tue, 6 Jul 2010 18:21:54 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Greg Roelofs updated HADOOP-6835:
---------------------------------

    Status: Open  (was: Patch Available)

Cancelling and resubmitting per ArunM:  "sometimes Hudson just gets weird" (or words to that effect).

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>             Fix For: 0.22.0
>
>         Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v4.trunk-hadoop-common.patch, HADOOP-6835.v4.trunk-hadoop-mapreduce.patch, HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v5.trunk-hadoop-common.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9361-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 22:26:18 2010
Return-Path: <common-issues-return-9361-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63292 invoked from network); 6 Jul 2010 22:26:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 22:26:18 -0000
Received: (qmail 16340 invoked by uid 500); 6 Jul 2010 22:26:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16267 invoked by uid 500); 6 Jul 2010 22:26:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16259 invoked by uid 99); 6 Jul 2010 22:26:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 22:26:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 22:26:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o66MPqbc022856
	for <common-issues@hadoop.apache.org>; Tue, 6 Jul 2010 22:25:53 GMT
Message-ID: <12739789.225671278455152858.JavaMail.jira@thor>
Date: Tue, 6 Jul 2010 18:25:52 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Greg Roelofs updated HADOOP-6835:
---------------------------------

    Status: Patch Available  (was: Open)

No change to patch, just retrying Hudson.  Error lines were:

    [javac] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/io/compress/GzipCodec.java:195: cannot find symbol
    [javac] symbol  : class BuiltInGzipDecompressor
    [javac] location: class org.apache.hadoop.io.compress.GzipCodec
    [javac]       : new BuiltInGzipDecompressor();
    [javac]             ^
    [javac] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/io/compress/GzipCodec.java:201: cannot find symbol
    [javac] symbol  : class BuiltInGzipDecompressor
    [javac] location: class org.apache.hadoop.io.compress.GzipCodec
    [javac]       : BuiltInGzipDecompressor.class;
    [javac]         ^

However, BuiltInGzipDecompressor lives in org.apache.hadoop.io.compress.zlib, and GzipCodec.java includes "import org.apache.hadoop.io.compress.zlib.*;", so this failure makes no sense to either of us.

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>             Fix For: 0.22.0
>
>         Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v4.trunk-hadoop-common.patch, HADOOP-6835.v4.trunk-hadoop-mapreduce.patch, HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v5.trunk-hadoop-common.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9362-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 06 23:30:45 2010
Return-Path: <common-issues-return-9362-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77350 invoked from network); 6 Jul 2010 23:30:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jul 2010 23:30:44 -0000
Received: (qmail 75624 invoked by uid 500); 6 Jul 2010 23:30:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75569 invoked by uid 500); 6 Jul 2010 23:30:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75561 invoked by uid 99); 6 Jul 2010 23:30:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 23:30:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jul 2010 23:30:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o66NMntN023217
	for <common-issues@hadoop.apache.org>; Tue, 6 Jul 2010 23:22:50 GMT
Message-ID: <15483385.226461278458569944.JavaMail.jira@thor>
Date: Tue, 6 Jul 2010 19:22:49 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6847) Problem staging 0.21.0 artifacts to
 Apache Nexus Maven Repository
In-Reply-To: <28118738.168371278054229909.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885731#action_12885731 ] 

Hadoop QA commented on HADOOP-6847:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12448784/hadoop-6847.patch
  against trunk revision 960137.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/601/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/601/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/601/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/601/console

This message is automatically generated.

> Problem staging 0.21.0 artifacts to Apache Nexus Maven Repository
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6847
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6847
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Tom White
>            Priority: Blocker
>         Attachments: hadoop-6847.patch
>
>
> I get this error:
> {noformat}
> -mvn-system-deploy:
> [artifact:install-provider] Installing provider: org.apache.maven.wagon:wagon-http:jar:1.0-beta-2:runtime
> [artifact:deploy] Deploying to https://repository.apache.org/content/repositories/snapshots
> [artifact:deploy] Uploading: org/apache/hadoop/hadoop-common-instrumented/0.21.0/hadoop-common-instrumented-0.21.0.jar to apache.snapshots.https
> [artifact:deploy] Uploaded 1331K
> [artifact:deploy] An error has occurred while processing the Maven artifact tasks.
> [artifact:deploy]  Diagnosis:
> [artifact:deploy] 
> [artifact:deploy] Error deploying artifact 'org.apache.hadoop:hadoop-common-instrumented:jar': Error deploying artifact: Failed to transfer file: https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common-instrumented/0.21.0/hadoop-common-instrumented-0.21.0.jar. Return code is: 400
> {noformat}
> Note that 400 is "Bad Request", not an authentication error (401).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9363-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 07 01:00:18 2010
Return-Path: <common-issues-return-9363-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98031 invoked from network); 7 Jul 2010 01:00:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jul 2010 01:00:18 -0000
Received: (qmail 28267 invoked by uid 500); 7 Jul 2010 01:00:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28229 invoked by uid 500); 7 Jul 2010 01:00:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28221 invoked by uid 99); 7 Jul 2010 01:00:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 01:00:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 01:00:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o670xrGb023926
	for <common-issues@hadoop.apache.org>; Wed, 7 Jul 2010 00:59:54 GMT
Message-ID: <3283913.227821278464393740.JavaMail.jira@thor>
Date: Tue, 6 Jul 2010 20:59:53 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Greg Roelofs updated HADOOP-6835:
---------------------------------

    Status: Open  (was: Patch Available)

Pilot error, sigh.  Patches work better when new files are git added first...

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>             Fix For: 0.22.0
>
>         Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v4.trunk-hadoop-common.patch, HADOOP-6835.v4.trunk-hadoop-mapreduce.patch, HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v5.trunk-hadoop-common.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9364-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 07 01:02:19 2010
Return-Path: <common-issues-return-9364-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98338 invoked from network); 7 Jul 2010 01:02:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jul 2010 01:02:19 -0000
Received: (qmail 28631 invoked by uid 500); 7 Jul 2010 01:02:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28591 invoked by uid 500); 7 Jul 2010 01:02:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28583 invoked by uid 99); 7 Jul 2010 01:02:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 01:02:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 01:02:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6711sD6023993
	for <common-issues@hadoop.apache.org>; Wed, 7 Jul 2010 01:01:54 GMT
Message-ID: <3335078.227961278464514466.JavaMail.jira@thor>
Date: Tue, 6 Jul 2010 21:01:54 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Greg Roelofs updated HADOOP-6835:
---------------------------------

    Attachment: HADOOP-6835.v6.trunk-hadoop-common.patch

Same as v5 except with BuiltInGzipDecompressor.java included.

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>             Fix For: 0.22.0
>
>         Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v4.trunk-hadoop-common.patch, HADOOP-6835.v4.trunk-hadoop-mapreduce.patch, HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v5.trunk-hadoop-common.patch, HADOOP-6835.v6.trunk-hadoop-common.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9365-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 07 01:09:51 2010
Return-Path: <common-issues-return-9365-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3424 invoked from network); 7 Jul 2010 01:09:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jul 2010 01:09:50 -0000
Received: (qmail 33192 invoked by uid 500); 7 Jul 2010 01:09:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33080 invoked by uid 500); 7 Jul 2010 01:09:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33072 invoked by uid 99); 7 Jul 2010 01:09:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 01:09:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 01:09:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6711ux3024035
	for <common-issues@hadoop.apache.org>; Wed, 7 Jul 2010 01:01:56 GMT
Message-ID: <3378118.228101278464516603.JavaMail.jira@thor>
Date: Tue, 6 Jul 2010 21:01:56 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Greg Roelofs updated HADOOP-6835:
---------------------------------

    Status: Patch Available  (was: Open)

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>             Fix For: 0.22.0
>
>         Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v4.trunk-hadoop-common.patch, HADOOP-6835.v4.trunk-hadoop-mapreduce.patch, HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v5.trunk-hadoop-common.patch, HADOOP-6835.v6.trunk-hadoop-common.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9366-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 07 01:34:19 2010
Return-Path: <common-issues-return-9366-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8912 invoked from network); 7 Jul 2010 01:34:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jul 2010 01:34:18 -0000
Received: (qmail 46929 invoked by uid 500); 7 Jul 2010 01:34:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46833 invoked by uid 500); 7 Jul 2010 01:34:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46825 invoked by uid 99); 7 Jul 2010 01:34:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 01:34:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 01:34:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o671XrxX024333
	for <common-issues@hadoop.apache.org>; Wed, 7 Jul 2010 01:33:53 GMT
Message-ID: <16592710.228851278466433535.JavaMail.jira@thor>
Date: Tue, 6 Jul 2010 21:33:53 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885767#action_12885767 ] 

Hadoop QA commented on HADOOP-6835:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12448841/HADOOP-6835.v6.trunk-hadoop-common.patch
  against trunk revision 960137.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    -1 javac.  The applied patch generated 1019 javac compiler warnings (more than the trunk's current 1017 warnings).

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/602/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/602/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/602/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/602/console

This message is automatically generated.

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>             Fix For: 0.22.0
>
>         Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v4.trunk-hadoop-common.patch, HADOOP-6835.v4.trunk-hadoop-mapreduce.patch, HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v5.trunk-hadoop-common.patch, HADOOP-6835.v6.trunk-hadoop-common.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9367-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 07 02:00:21 2010
Return-Path: <common-issues-return-9367-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11553 invoked from network); 7 Jul 2010 02:00:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jul 2010 02:00:21 -0000
Received: (qmail 56949 invoked by uid 500); 7 Jul 2010 02:00:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56848 invoked by uid 500); 7 Jul 2010 02:00:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56840 invoked by uid 99); 7 Jul 2010 02:00:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 02:00:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 02:00:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o671xuAM024583
	for <common-issues@hadoop.apache.org>; Wed, 7 Jul 2010 01:59:56 GMT
Message-ID: <15154329.229371278467996480.JavaMail.jira@thor>
Date: Tue, 6 Jul 2010 21:59:56 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Greg Roelofs updated HADOOP-6835:
---------------------------------

    Attachment: HADOOP-6835.v7.trunk-hadoop-common.patch

Fixed extra pair of javac warnings (unnecessary int typecasts in {{readUShortLE()}}).

The javadoc output file (http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/602/artifact/trunk/patchprocess/patchJavadocWarnings.txt) shows 15 + 6 warnings, all of which are existing warnings in the security code.  I have no idea which one it thinks I'm responsible for (particularly since there's no corresponding "trunkJavadocWarnings.txt" link), but I plead not guilty...

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>             Fix For: 0.22.0
>
>         Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v4.trunk-hadoop-common.patch, HADOOP-6835.v4.trunk-hadoop-mapreduce.patch, HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v5.trunk-hadoop-common.patch, HADOOP-6835.v6.trunk-hadoop-common.patch, HADOOP-6835.v7.trunk-hadoop-common.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9368-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 07 02:08:22 2010
Return-Path: <common-issues-return-9368-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16895 invoked from network); 7 Jul 2010 02:08:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jul 2010 02:08:22 -0000
Received: (qmail 64250 invoked by uid 500); 7 Jul 2010 02:08:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64226 invoked by uid 500); 7 Jul 2010 02:08:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64218 invoked by uid 99); 7 Jul 2010 02:08:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 02:08:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 02:08:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6727vDN024775
	for <common-issues@hadoop.apache.org>; Wed, 7 Jul 2010 02:07:57 GMT
Message-ID: <17456857.229901278468477488.JavaMail.jira@thor>
Date: Tue, 6 Jul 2010 22:07:57 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Greg Roelofs updated HADOOP-6835:
---------------------------------

    Status: Patch Available  (was: Open)

(Not sure this will work...  Sorting by "date attached" currently shows 8:59PM coming after 9:59pm.  Oooookay, then.)

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>             Fix For: 0.22.0
>
>         Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v4.trunk-hadoop-common.patch, HADOOP-6835.v4.trunk-hadoop-mapreduce.patch, HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v5.trunk-hadoop-common.patch, HADOOP-6835.v6.trunk-hadoop-common.patch, HADOOP-6835.v7.trunk-hadoop-common.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9369-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 07 02:13:49 2010
Return-Path: <common-issues-return-9369-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17484 invoked from network); 7 Jul 2010 02:13:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jul 2010 02:13:48 -0000
Received: (qmail 66423 invoked by uid 500); 7 Jul 2010 02:13:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66382 invoked by uid 500); 7 Jul 2010 02:13:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66374 invoked by uid 99); 7 Jul 2010 02:13:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 02:13:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 02:13:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6725snK024699
	for <common-issues@hadoop.apache.org>; Wed, 7 Jul 2010 02:05:54 GMT
Message-ID: <516971.229651278468354025.JavaMail.jira@thor>
Date: Tue, 6 Jul 2010 22:05:54 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Greg Roelofs updated HADOOP-6835:
---------------------------------

    Status: Open  (was: Patch Available)

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>             Fix For: 0.22.0
>
>         Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v4.trunk-hadoop-common.patch, HADOOP-6835.v4.trunk-hadoop-mapreduce.patch, HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v5.trunk-hadoop-common.patch, HADOOP-6835.v6.trunk-hadoop-common.patch, HADOOP-6835.v7.trunk-hadoop-common.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9370-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 07 02:32:24 2010
Return-Path: <common-issues-return-9370-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20404 invoked from network); 7 Jul 2010 02:32:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jul 2010 02:32:24 -0000
Received: (qmail 81379 invoked by uid 500); 7 Jul 2010 02:32:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81266 invoked by uid 500); 7 Jul 2010 02:32:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81253 invoked by uid 99); 7 Jul 2010 02:32:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 02:32:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 02:32:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o672VwdK025065
	for <common-issues@hadoop.apache.org>; Wed, 7 Jul 2010 02:31:58 GMT
Message-ID: <25712370.230461278469918210.JavaMail.jira@thor>
Date: Tue, 6 Jul 2010 22:31:58 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885791#action_12885791 ] 

Hadoop QA commented on HADOOP-6835:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12448847/HADOOP-6835.v7.trunk-hadoop-common.patch
  against trunk revision 960137.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/603/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/603/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/603/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/603/console

This message is automatically generated.

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>             Fix For: 0.22.0
>
>         Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v4.trunk-hadoop-common.patch, HADOOP-6835.v4.trunk-hadoop-mapreduce.patch, HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v5.trunk-hadoop-common.patch, HADOOP-6835.v6.trunk-hadoop-common.patch, HADOOP-6835.v7.trunk-hadoop-common.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9371-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 07 03:52:21 2010
Return-Path: <common-issues-return-9371-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36522 invoked from network); 7 Jul 2010 03:52:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jul 2010 03:52:21 -0000
Received: (qmail 35443 invoked by uid 500); 7 Jul 2010 03:52:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35230 invoked by uid 500); 7 Jul 2010 03:52:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35222 invoked by uid 99); 7 Jul 2010 03:52:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 03:52:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 03:52:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o673psSS025581
	for <common-issues@hadoop.apache.org>; Wed, 7 Jul 2010 03:51:54 GMT
Message-ID: <4126971.231361278474714095.JavaMail.jira@thor>
Date: Tue, 6 Jul 2010 23:51:54 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Greg Roelofs updated HADOOP-6835:
---------------------------------

    Status: Open  (was: Patch Available)

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>             Fix For: 0.22.0
>
>         Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v4.trunk-hadoop-common.patch, HADOOP-6835.v4.trunk-hadoop-mapreduce.patch, HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v5.trunk-hadoop-common.patch, HADOOP-6835.v6.trunk-hadoop-common.patch, HADOOP-6835.v7.trunk-hadoop-common.patch, HADOOP-6835.v8.trunk-hadoop-common.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9372-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 07 03:52:22 2010
Return-Path: <common-issues-return-9372-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36583 invoked from network); 7 Jul 2010 03:52:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jul 2010 03:52:21 -0000
Received: (qmail 35501 invoked by uid 500); 7 Jul 2010 03:52:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35440 invoked by uid 500); 7 Jul 2010 03:52:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35309 invoked by uid 99); 7 Jul 2010 03:52:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 03:52:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 03:52:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o673puet025623
	for <common-issues@hadoop.apache.org>; Wed, 7 Jul 2010 03:51:56 GMT
Message-ID: <10827214.231501278474716715.JavaMail.jira@thor>
Date: Tue, 6 Jul 2010 23:51:56 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Greg Roelofs updated HADOOP-6835:
---------------------------------

    Status: Patch Available  (was: Open)

Ready for checkin.  The javadoc warnings are all preexisting Kerberos/security-related ones (verified with Chris).

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>             Fix For: 0.22.0
>
>         Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v4.trunk-hadoop-common.patch, HADOOP-6835.v4.trunk-hadoop-mapreduce.patch, HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v5.trunk-hadoop-common.patch, HADOOP-6835.v6.trunk-hadoop-common.patch, HADOOP-6835.v7.trunk-hadoop-common.patch, HADOOP-6835.v8.trunk-hadoop-common.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9373-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 07 03:57:51 2010
Return-Path: <common-issues-return-9373-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37301 invoked from network); 7 Jul 2010 03:57:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jul 2010 03:57:51 -0000
Received: (qmail 38040 invoked by uid 500); 7 Jul 2010 03:57:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37927 invoked by uid 500); 7 Jul 2010 03:57:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37919 invoked by uid 99); 7 Jul 2010 03:57:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 03:57:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 03:57:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o673nqbC025538
	for <common-issues@hadoop.apache.org>; Wed, 7 Jul 2010 03:49:53 GMT
Message-ID: <25769679.231221278474592837.JavaMail.jira@thor>
Date: Tue, 6 Jul 2010 23:49:52 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Greg Roelofs updated HADOOP-6835:
---------------------------------

    Attachment: HADOOP-6835.v8.trunk-hadoop-common.patch

Same as v7 but with a pair of new TestCodec unit (sub)tests provided by Chris Douglas.  (These represent a low-level, hadoop-common complement to the existing ones in the hadoop-mapreduce patch.)

All uses of the deprecated "hadoop.native.lib" option in TestCodec are replaced by "io.native.lib.available", too.

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>             Fix For: 0.22.0
>
>         Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v4.trunk-hadoop-common.patch, HADOOP-6835.v4.trunk-hadoop-mapreduce.patch, HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v5.trunk-hadoop-common.patch, HADOOP-6835.v6.trunk-hadoop-common.patch, HADOOP-6835.v7.trunk-hadoop-common.patch, HADOOP-6835.v8.trunk-hadoop-common.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9374-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 07 06:49:52 2010
Return-Path: <common-issues-return-9374-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80924 invoked from network); 7 Jul 2010 06:49:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jul 2010 06:49:51 -0000
Received: (qmail 52298 invoked by uid 500); 7 Jul 2010 06:49:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52165 invoked by uid 500); 7 Jul 2010 06:49:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52157 invoked by uid 99); 7 Jul 2010 06:49:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 06:49:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 06:49:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o676fqLY027724
	for <common-issues@hadoop.apache.org>; Wed, 7 Jul 2010 06:41:52 GMT
Message-ID: <13580729.234821278484912655.JavaMail.jira@thor>
Date: Wed, 7 Jul 2010 02:41:52 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885850#action_12885850 ] 

Hadoop QA commented on HADOOP-6835:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12448852/HADOOP-6835.v8.trunk-hadoop-common.patch
  against trunk revision 960137.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/83/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/83/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/83/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/83/console

This message is automatically generated.

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>             Fix For: 0.22.0
>
>         Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v4.trunk-hadoop-common.patch, HADOOP-6835.v4.trunk-hadoop-mapreduce.patch, HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v5.trunk-hadoop-common.patch, HADOOP-6835.v6.trunk-hadoop-common.patch, HADOOP-6835.v7.trunk-hadoop-common.patch, HADOOP-6835.v8.trunk-hadoop-common.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9375-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 07 07:29:51 2010
Return-Path: <common-issues-return-9375-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92334 invoked from network); 7 Jul 2010 07:29:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jul 2010 07:29:51 -0000
Received: (qmail 84226 invoked by uid 500); 7 Jul 2010 07:29:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84046 invoked by uid 500); 7 Jul 2010 07:29:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84028 invoked by uid 99); 7 Jul 2010 07:29:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 07:29:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 07:29:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o677LrLC028153
	for <common-issues@hadoop.apache.org>; Wed, 7 Jul 2010 07:21:53 GMT
Message-ID: <25828286.235431278487313378.JavaMail.jira@thor>
Date: Wed, 7 Jul 2010 03:21:53 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885860#action_12885860 ] 

Hadoop QA commented on HADOOP-6835:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12448852/HADOOP-6835.v8.trunk-hadoop-common.patch
  against trunk revision 960137.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/604/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/604/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/604/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/604/console

This message is automatically generated.

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>             Fix For: 0.22.0
>
>         Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v4.trunk-hadoop-common.patch, HADOOP-6835.v4.trunk-hadoop-mapreduce.patch, HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v5.trunk-hadoop-common.patch, HADOOP-6835.v6.trunk-hadoop-common.patch, HADOOP-6835.v7.trunk-hadoop-common.patch, HADOOP-6835.v8.trunk-hadoop-common.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9376-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 07 11:56:18 2010
Return-Path: <common-issues-return-9376-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55601 invoked from network); 7 Jul 2010 11:56:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jul 2010 11:56:18 -0000
Received: (qmail 75852 invoked by uid 500); 7 Jul 2010 11:56:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75730 invoked by uid 500); 7 Jul 2010 11:56:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75718 invoked by uid 99); 7 Jul 2010 11:56:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 11:56:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 11:56:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o67Btolc000332
	for <common-issues@hadoop.apache.org>; Wed, 7 Jul 2010 11:55:50 GMT
Message-ID: <30568307.238481278503750009.JavaMail.jira@thor>
Date: Wed, 7 Jul 2010 07:55:50 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6847) Problem staging 0.21.0 artifacts to
 Apache Nexus Maven Repository
In-Reply-To: <28118738.168371278054229909.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White reassigned HADOOP-6847:
---------------------------------

    Assignee: Giridharan Kesavan

> Problem staging 0.21.0 artifacts to Apache Nexus Maven Repository
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6847
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6847
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Tom White
>            Assignee: Giridharan Kesavan
>            Priority: Blocker
>         Attachments: hadoop-6847.patch
>
>
> I get this error:
> {noformat}
> -mvn-system-deploy:
> [artifact:install-provider] Installing provider: org.apache.maven.wagon:wagon-http:jar:1.0-beta-2:runtime
> [artifact:deploy] Deploying to https://repository.apache.org/content/repositories/snapshots
> [artifact:deploy] Uploading: org/apache/hadoop/hadoop-common-instrumented/0.21.0/hadoop-common-instrumented-0.21.0.jar to apache.snapshots.https
> [artifact:deploy] Uploaded 1331K
> [artifact:deploy] An error has occurred while processing the Maven artifact tasks.
> [artifact:deploy]  Diagnosis:
> [artifact:deploy] 
> [artifact:deploy] Error deploying artifact 'org.apache.hadoop:hadoop-common-instrumented:jar': Error deploying artifact: Failed to transfer file: https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common-instrumented/0.21.0/hadoop-common-instrumented-0.21.0.jar. Return code is: 400
> {noformat}
> Note that 400 is "Bad Request", not an authentication error (401).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9377-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 07 12:00:47 2010
Return-Path: <common-issues-return-9377-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56428 invoked from network); 7 Jul 2010 12:00:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jul 2010 12:00:47 -0000
Received: (qmail 78840 invoked by uid 500); 7 Jul 2010 12:00:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78707 invoked by uid 500); 7 Jul 2010 12:00:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78698 invoked by uid 99); 7 Jul 2010 12:00:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 12:00:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 12:00:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o67BqoN0000293
	for <common-issues@hadoop.apache.org>; Wed, 7 Jul 2010 11:52:50 GMT
Message-ID: <33147030.238441278503570212.JavaMail.jira@thor>
Date: Wed, 7 Jul 2010 07:52:50 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6847) Problem staging 0.21.0 artifacts to
 Apache Nexus Maven Repository
In-Reply-To: <28118738.168371278054229909.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885922#action_12885922 ] 

Tom White commented on HADOOP-6847:
-----------------------------------

+1

I managed to publish artifacts to the staging repo too (I had to add a server section to ~/.m2/settings.xml with an ID of apache.staging.https for it to work). Thanks for fixing this, Giri. I think we need equivalent fixes for HDFS and MapReduce.

> Problem staging 0.21.0 artifacts to Apache Nexus Maven Repository
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6847
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6847
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Tom White
>            Priority: Blocker
>         Attachments: hadoop-6847.patch
>
>
> I get this error:
> {noformat}
> -mvn-system-deploy:
> [artifact:install-provider] Installing provider: org.apache.maven.wagon:wagon-http:jar:1.0-beta-2:runtime
> [artifact:deploy] Deploying to https://repository.apache.org/content/repositories/snapshots
> [artifact:deploy] Uploading: org/apache/hadoop/hadoop-common-instrumented/0.21.0/hadoop-common-instrumented-0.21.0.jar to apache.snapshots.https
> [artifact:deploy] Uploaded 1331K
> [artifact:deploy] An error has occurred while processing the Maven artifact tasks.
> [artifact:deploy]  Diagnosis:
> [artifact:deploy] 
> [artifact:deploy] Error deploying artifact 'org.apache.hadoop:hadoop-common-instrumented:jar': Error deploying artifact: Failed to transfer file: https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common-instrumented/0.21.0/hadoop-common-instrumented-0.21.0.jar. Return code is: 400
> {noformat}
> Note that 400 is "Bad Request", not an authentication error (401).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9378-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 07 22:33:47 2010
Return-Path: <common-issues-return-9378-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78304 invoked from network); 7 Jul 2010 22:33:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jul 2010 22:33:47 -0000
Received: (qmail 17936 invoked by uid 500); 7 Jul 2010 22:33:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17747 invoked by uid 500); 7 Jul 2010 22:33:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17729 invoked by uid 99); 7 Jul 2010 22:33:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 22:33:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 22:33:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o67MPpxh009049
	for <common-issues@hadoop.apache.org>; Wed, 7 Jul 2010 22:25:51 GMT
Message-ID: <503604.248281278541551065.JavaMail.jira@thor>
Date: Wed, 7 Jul 2010 18:25:51 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6852) apparent bug in concatenated-bzip2
 support (decoding)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

apparent bug in concatenated-bzip2 support (decoding)
-----------------------------------------------------

                 Key: HADOOP-6852
                 URL: https://issues.apache.org/jira/browse/HADOOP-6852
             Project: Hadoop Common
          Issue Type: Bug
          Components: io
    Affects Versions: 0.22.0
         Environment: Linux x86_64 running 32-bit Hadoop, JDK 1.6.0_15
            Reporter: Greg Roelofs


The following simplified code (manually picked out of testMoreBzip2() in https://issues.apache.org/jira/secure/attachment/12448272/HADOOP-6835.v4.trunk-hadoop-mapreduce.patch) triggers a "java.io.IOException: bad block header" in org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.initBlock( CBZip2InputStream.java:527):

    JobConf jobConf = new JobConf(defaultConf);

    CompressionCodec bzip2 = new BZip2Codec();
    ReflectionUtils.setConf(bzip2, jobConf);
    localFs.delete(workDir, true);

    // copy multiple-member test file to HDFS
    String fn2 = "testCompressThenConcat.txt" + bzip2.getDefaultExtension();
    Path fnLocal2 = new Path(System.getProperty("test.concat.data","/tmp"),fn2);
    Path fnHDFS2  = new Path(workDir, fn2);
    localFs.copyFromLocalFile(fnLocal2, fnHDFS2);

    FileInputFormat.setInputPaths(jobConf, workDir);

    final FileInputStream in2 = new FileInputStream(fnLocal2.toString());
    CompressionInputStream cin2 = bzip2.createInputStream(in2);
    LineReader in = new LineReader(cin2);
    Text out = new Text();

    int numBytes, totalBytes=0, lineNum=0;
    while ((numBytes = in.readLine(out)) > 0) {
      ++lineNum;
      totalBytes += numBytes;
    }
    in.close();

The specified file is also included in the H-6835 patch linked above, and some additional debug output is included in the commented-out test loop above.  (Only in the linked, "v4" version of the patch, however--I'm about to remove the debug stuff for checkin.)

It's possible I've done something completely boneheaded here, but the file, at least, checks out in a subsequent set of subtests and with stock bzip2 itself.  Only the code above is problematic; it reads through the first concatenated chunk (17 lines of text) just fine but chokes on the header of the second one.  Altogether, the test file contains 84 lines of text and 4 concatenated bzip2 files.

(It's possible this is a mapreduce issue rather than common, but note that the identical gzip test works fine.  Possibly it's related to the stream-vs-decompressor dichotomy, though; intentionally not supported?)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9379-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 07 22:35:16 2010
Return-Path: <common-issues-return-9379-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78874 invoked from network); 7 Jul 2010 22:35:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jul 2010 22:35:15 -0000
Received: (qmail 22005 invoked by uid 500); 7 Jul 2010 22:35:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21913 invoked by uid 500); 7 Jul 2010 22:35:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21880 invoked by uid 99); 7 Jul 2010 22:35:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 22:35:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 22:35:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o67MYouu009606
	for <common-issues@hadoop.apache.org>; Wed, 7 Jul 2010 22:34:50 GMT
Message-ID: <20848420.248771278542090605.JavaMail.jira@thor>
Date: Wed, 7 Jul 2010 18:34:50 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6852) apparent bug in concatenated-bzip2
 support (decoding)
In-Reply-To: <503604.248281278541551065.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Greg Roelofs updated HADOOP-6852:
---------------------------------

    Description: 
The following simplified code (manually picked out of testMoreBzip2() in https://issues.apache.org/jira/secure/attachment/12448272/HADOOP-6835.v4.trunk-hadoop-mapreduce.patch) triggers a "java.io.IOException: bad block header" in org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.initBlock( CBZip2InputStream.java:527):

{noformat}
    JobConf jobConf = new JobConf(defaultConf);

    CompressionCodec bzip2 = new BZip2Codec();
    ReflectionUtils.setConf(bzip2, jobConf);
    localFs.delete(workDir, true);

    // copy multiple-member test file to HDFS
    String fn2 = "testCompressThenConcat.txt" + bzip2.getDefaultExtension();
    Path fnLocal2 = new Path(System.getProperty("test.concat.data","/tmp"),fn2);
    Path fnHDFS2  = new Path(workDir, fn2);
    localFs.copyFromLocalFile(fnLocal2, fnHDFS2);

    FileInputFormat.setInputPaths(jobConf, workDir);

    final FileInputStream in2 = new FileInputStream(fnLocal2.toString());
    CompressionInputStream cin2 = bzip2.createInputStream(in2);
    LineReader in = new LineReader(cin2);
    Text out = new Text();

    int numBytes, totalBytes=0, lineNum=0;
    while ((numBytes = in.readLine(out)) > 0) {
      ++lineNum;
      totalBytes += numBytes;
    }
    in.close();
{noformat}

The specified file is also included in the H-6835 patch linked above, and some additional debug output is included in the commented-out test loop above.  (Only in the linked, "v4" version of the patch, however--I'm about to remove the debug stuff for checkin.)

It's possible I've done something completely boneheaded here, but the file, at least, checks out in a subsequent set of subtests and with stock bzip2 itself.  Only the code above is problematic; it reads through the first concatenated chunk (17 lines of text) just fine but chokes on the header of the second one.  Altogether, the test file contains 84 lines of text and 4 concatenated bzip2 files.

(It's possible this is a mapreduce issue rather than common, but note that the identical gzip test works fine.  Possibly it's related to the stream-vs-decompressor dichotomy, though; intentionally not supported?)

  was:
The following simplified code (manually picked out of testMoreBzip2() in https://issues.apache.org/jira/secure/attachment/12448272/HADOOP-6835.v4.trunk-hadoop-mapreduce.patch) triggers a "java.io.IOException: bad block header" in org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.initBlock( CBZip2InputStream.java:527):

    JobConf jobConf = new JobConf(defaultConf);

    CompressionCodec bzip2 = new BZip2Codec();
    ReflectionUtils.setConf(bzip2, jobConf);
    localFs.delete(workDir, true);

    // copy multiple-member test file to HDFS
    String fn2 = "testCompressThenConcat.txt" + bzip2.getDefaultExtension();
    Path fnLocal2 = new Path(System.getProperty("test.concat.data","/tmp"),fn2);
    Path fnHDFS2  = new Path(workDir, fn2);
    localFs.copyFromLocalFile(fnLocal2, fnHDFS2);

    FileInputFormat.setInputPaths(jobConf, workDir);

    final FileInputStream in2 = new FileInputStream(fnLocal2.toString());
    CompressionInputStream cin2 = bzip2.createInputStream(in2);
    LineReader in = new LineReader(cin2);
    Text out = new Text();

    int numBytes, totalBytes=0, lineNum=0;
    while ((numBytes = in.readLine(out)) > 0) {
      ++lineNum;
      totalBytes += numBytes;
    }
    in.close();

The specified file is also included in the H-6835 patch linked above, and some additional debug output is included in the commented-out test loop above.  (Only in the linked, "v4" version of the patch, however--I'm about to remove the debug stuff for checkin.)

It's possible I've done something completely boneheaded here, but the file, at least, checks out in a subsequent set of subtests and with stock bzip2 itself.  Only the code above is problematic; it reads through the first concatenated chunk (17 lines of text) just fine but chokes on the header of the second one.  Altogether, the test file contains 84 lines of text and 4 concatenated bzip2 files.

(It's possible this is a mapreduce issue rather than common, but note that the identical gzip test works fine.  Possibly it's related to the stream-vs-decompressor dichotomy, though; intentionally not supported?)


> apparent bug in concatenated-bzip2 support (decoding)
> -----------------------------------------------------
>
>                 Key: HADOOP-6852
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6852
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.22.0
>         Environment: Linux x86_64 running 32-bit Hadoop, JDK 1.6.0_15
>            Reporter: Greg Roelofs
>
> The following simplified code (manually picked out of testMoreBzip2() in https://issues.apache.org/jira/secure/attachment/12448272/HADOOP-6835.v4.trunk-hadoop-mapreduce.patch) triggers a "java.io.IOException: bad block header" in org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.initBlock( CBZip2InputStream.java:527):
> {noformat}
>     JobConf jobConf = new JobConf(defaultConf);
>     CompressionCodec bzip2 = new BZip2Codec();
>     ReflectionUtils.setConf(bzip2, jobConf);
>     localFs.delete(workDir, true);
>     // copy multiple-member test file to HDFS
>     String fn2 = "testCompressThenConcat.txt" + bzip2.getDefaultExtension();
>     Path fnLocal2 = new Path(System.getProperty("test.concat.data","/tmp"),fn2);
>     Path fnHDFS2  = new Path(workDir, fn2);
>     localFs.copyFromLocalFile(fnLocal2, fnHDFS2);
>     FileInputFormat.setInputPaths(jobConf, workDir);
>     final FileInputStream in2 = new FileInputStream(fnLocal2.toString());
>     CompressionInputStream cin2 = bzip2.createInputStream(in2);
>     LineReader in = new LineReader(cin2);
>     Text out = new Text();
>     int numBytes, totalBytes=0, lineNum=0;
>     while ((numBytes = in.readLine(out)) > 0) {
>       ++lineNum;
>       totalBytes += numBytes;
>     }
>     in.close();
> {noformat}
> The specified file is also included in the H-6835 patch linked above, and some additional debug output is included in the commented-out test loop above.  (Only in the linked, "v4" version of the patch, however--I'm about to remove the debug stuff for checkin.)
> It's possible I've done something completely boneheaded here, but the file, at least, checks out in a subsequent set of subtests and with stock bzip2 itself.  Only the code above is problematic; it reads through the first concatenated chunk (17 lines of text) just fine but chokes on the header of the second one.  Altogether, the test file contains 84 lines of text and 4 concatenated bzip2 files.
> (It's possible this is a mapreduce issue rather than common, but note that the identical gzip test works fine.  Possibly it's related to the stream-vs-decompressor dichotomy, though; intentionally not supported?)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9380-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 07 23:27:20 2010
Return-Path: <common-issues-return-9380-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91829 invoked from network); 7 Jul 2010 23:27:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jul 2010 23:27:19 -0000
Received: (qmail 65470 invoked by uid 500); 7 Jul 2010 23:27:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65406 invoked by uid 500); 7 Jul 2010 23:27:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65395 invoked by uid 99); 7 Jul 2010 23:27:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 23:27:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 23:27:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o67NQsdY011889
	for <common-issues@hadoop.apache.org>; Wed, 7 Jul 2010 23:26:54 GMT
Message-ID: <830957.250051278545214289.JavaMail.jira@thor>
Date: Wed, 7 Jul 2010 19:26:54 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-6835:
----------------------------------

    Attachment: C6835-9.patch

Very minor modifications to v8 prior to commit
* Use {{PureJavaCRC32}} per comments
* Removed unused GzipCodec::GzipInputStream
* Replace config literals with CommonConfigurationKeys constants

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>             Fix For: 0.22.0
>
>         Attachments: C6835-9.patch, grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v4.trunk-hadoop-common.patch, HADOOP-6835.v4.trunk-hadoop-mapreduce.patch, HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v5.trunk-hadoop-common.patch, HADOOP-6835.v6.trunk-hadoop-common.patch, HADOOP-6835.v7.trunk-hadoop-common.patch, HADOOP-6835.v8.trunk-hadoop-common.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9381-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 07 23:32:48 2010
Return-Path: <common-issues-return-9381-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92196 invoked from network); 7 Jul 2010 23:32:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jul 2010 23:32:48 -0000
Received: (qmail 67745 invoked by uid 500); 7 Jul 2010 23:32:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67677 invoked by uid 500); 7 Jul 2010 23:32:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67669 invoked by uid 99); 7 Jul 2010 23:32:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 23:32:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 23:32:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o67NOrpO011845
	for <common-issues@hadoop.apache.org>; Wed, 7 Jul 2010 23:24:53 GMT
Message-ID: <32038927.249911278545093619.JavaMail.jira@thor>
Date: Wed, 7 Jul 2010 19:24:53 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-6835:
----------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Incompatible change, Reviewed]  (was: [Incompatible change])
      Resolution: Fixed

+1

I committed this. Thanks, Greg!

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>             Fix For: 0.22.0
>
>         Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v4.trunk-hadoop-common.patch, HADOOP-6835.v4.trunk-hadoop-mapreduce.patch, HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v5.trunk-hadoop-common.patch, HADOOP-6835.v6.trunk-hadoop-common.patch, HADOOP-6835.v7.trunk-hadoop-common.patch, HADOOP-6835.v8.trunk-hadoop-common.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9382-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 07 23:52:50 2010
Return-Path: <common-issues-return-9382-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95531 invoked from network); 7 Jul 2010 23:52:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jul 2010 23:52:50 -0000
Received: (qmail 81677 invoked by uid 500); 7 Jul 2010 23:52:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81561 invoked by uid 500); 7 Jul 2010 23:52:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81553 invoked by uid 99); 7 Jul 2010 23:52:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 23:52:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jul 2010 23:52:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o67NisdA011977
	for <common-issues@hadoop.apache.org>; Wed, 7 Jul 2010 23:44:54 GMT
Message-ID: <17697055.250231278546294595.JavaMail.jira@thor>
Date: Wed, 7 Jul 2010 19:44:54 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886154#action_12886154 ] 

Hudson commented on HADOOP-6835:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #320 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/320/])
    HADOOP-6835. Add support for concatenated gzip input. Contributed by Greg Roelofs


> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>             Fix For: 0.22.0
>
>         Attachments: C6835-9.patch, grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v4.trunk-hadoop-common.patch, HADOOP-6835.v4.trunk-hadoop-mapreduce.patch, HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v5.trunk-hadoop-common.patch, HADOOP-6835.v6.trunk-hadoop-common.patch, HADOOP-6835.v7.trunk-hadoop-common.patch, HADOOP-6835.v8.trunk-hadoop-common.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9383-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 00:29:18 2010
Return-Path: <common-issues-return-9383-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14508 invoked from network); 8 Jul 2010 00:29:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 00:29:17 -0000
Received: (qmail 12232 invoked by uid 500); 8 Jul 2010 00:29:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11774 invoked by uid 500); 8 Jul 2010 00:29:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11766 invoked by uid 99); 8 Jul 2010 00:29:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 00:29:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 00:29:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o680SpJY012357
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 00:28:52 GMT
Message-ID: <13947313.250911278548931492.JavaMail.jira@thor>
Date: Wed, 7 Jul 2010 20:28:51 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6853) Common component of HDFS-1045
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Common component of HDFS-1045
-----------------------------

                 Key: HADOOP-6853
                 URL: https://issues.apache.org/jira/browse/HADOOP-6853
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Jakob Homan
            Assignee: Jakob Homan
             Fix For: 0.22.0


HDFS-1045 modified UGI, which is in Common on trunk.  This JIRA is for that change.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9384-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 01:06:16 2010
Return-Path: <common-issues-return-9384-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25673 invoked from network); 8 Jul 2010 01:06:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 01:06:15 -0000
Received: (qmail 40696 invoked by uid 500); 8 Jul 2010 01:06:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40638 invoked by uid 500); 8 Jul 2010 01:06:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40630 invoked by uid 99); 8 Jul 2010 01:06:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 01:06:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 01:06:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6815poe012608
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 01:05:51 GMT
Message-ID: <8781134.251321278551151081.JavaMail.jira@thor>
Date: Wed, 7 Jul 2010 21:05:51 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6853) Common component of HDFS-1045
In-Reply-To: <13947313.250911278548931492.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6853:
--------------------------------

               Status: Patch Available  (was: Open)
    Affects Version/s: 0.22.0

Submitting patch.

> Common component of HDFS-1045
> -----------------------------
>
>                 Key: HADOOP-6853
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6853
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6853.patch
>
>
> HDFS-1045 modified UGI, which is in Common on trunk.  This JIRA is for that change.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9385-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 01:13:45 2010
Return-Path: <common-issues-return-9385-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26351 invoked from network); 8 Jul 2010 01:13:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 01:13:45 -0000
Received: (qmail 44881 invoked by uid 500); 8 Jul 2010 01:13:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44768 invoked by uid 500); 8 Jul 2010 01:13:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44760 invoked by uid 99); 8 Jul 2010 01:13:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 01:13:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 01:13:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6815oBK012602
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 01:05:50 GMT
Message-ID: <11873629.251301278551150307.JavaMail.jira@thor>
Date: Wed, 7 Jul 2010 21:05:50 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6853) Common component of HDFS-1045
In-Reply-To: <13947313.250911278548931492.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6853:
--------------------------------

    Attachment: HADOOP-6853.patch

Straight patch of loginUserFromKeytabAndReturnUGI in UGI, updated with changes from HADOOP-6649, which has already been committed.  At the moment, not unit-testable... sigh.

> Common component of HDFS-1045
> -----------------------------
>
>                 Key: HADOOP-6853
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6853
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6853.patch
>
>
> HDFS-1045 modified UGI, which is in Common on trunk.  This JIRA is for that change.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9386-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 02:00:17 2010
Return-Path: <common-issues-return-9386-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32424 invoked from network); 8 Jul 2010 02:00:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 02:00:16 -0000
Received: (qmail 83365 invoked by uid 500); 8 Jul 2010 02:00:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83054 invoked by uid 500); 8 Jul 2010 02:00:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83046 invoked by uid 99); 8 Jul 2010 02:00:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 02:00:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 02:00:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o681xoFd012945
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 01:59:50 GMT
Message-ID: <2613111.252031278554390518.JavaMail.jira@thor>
Date: Wed, 7 Jul 2010 21:59:50 -0400 (EDT)
From: "Nicolas Spiegelberg (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6854) Cannot Configure 'progress' with
 CreateOpts API
In-Reply-To: <15361457.252001278554389949.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicolas Spiegelberg updated HADOOP-6854:
----------------------------------------

    Attachment: progress.patch

3-line fix should apply cleanly on 0.21 + 0.22.

> Cannot Configure 'progress' with CreateOpts API
> -----------------------------------------------
>
>                 Key: HADOOP-6854
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6854
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Nicolas Spiegelberg
>            Priority: Minor
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: progress.patch
>
>
> Noticed that CreateOpts class specifies a Progress class, but does not allow you to specify a Progressable object.  Need to add static function to set this API element.  Trying to get FileContext on parity with old FileSystem API.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9387-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 02:07:44 2010
Return-Path: <common-issues-return-9387-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37606 invoked from network); 8 Jul 2010 02:07:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 02:07:44 -0000
Received: (qmail 88907 invoked by uid 500); 8 Jul 2010 02:07:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88844 invoked by uid 500); 8 Jul 2010 02:07:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88836 invoked by uid 99); 8 Jul 2010 02:07:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 02:07:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 02:07:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o681xnZv012936
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 01:59:50 GMT
Message-ID: <15361457.252001278554389949.JavaMail.jira@thor>
Date: Wed, 7 Jul 2010 21:59:49 -0400 (EDT)
From: "Nicolas Spiegelberg (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6854) Cannot Configure 'progress' with
 CreateOpts API
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Cannot Configure 'progress' with CreateOpts API
-----------------------------------------------

                 Key: HADOOP-6854
                 URL: https://issues.apache.org/jira/browse/HADOOP-6854
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
    Affects Versions: 0.21.0, 0.22.0
            Reporter: Nicolas Spiegelberg
            Priority: Minor
             Fix For: 0.21.0, 0.22.0
         Attachments: progress.patch

Noticed that CreateOpts class specifies a Progress class, but does not allow you to specify a Progressable object.  Need to add static function to set this API element.  Trying to get FileContext on parity with old FileSystem API.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9388-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 02:17:15 2010
Return-Path: <common-issues-return-9388-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39230 invoked from network); 8 Jul 2010 02:17:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 02:17:15 -0000
Received: (qmail 140 invoked by uid 500); 8 Jul 2010 02:17:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99973 invoked by uid 500); 8 Jul 2010 02:17:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99965 invoked by uid 99); 8 Jul 2010 02:17:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 02:17:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 02:17:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o682GoVv013104
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 02:16:50 GMT
Message-ID: <22599358.252371278555410181.JavaMail.jira@thor>
Date: Wed, 7 Jul 2010 22:16:50 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6847) Problem staging 0.21.0 artifacts to
 Apache Nexus Maven Repository
In-Reply-To: <28118738.168371278054229909.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886197#action_12886197 ] 

Konstantin Boudnik commented on HADOOP-6847:
--------------------------------------------

JavaDoc warning is caused by some FindBugs issue and isn't related to the patch. I'll be committing this shortly.

> Problem staging 0.21.0 artifacts to Apache Nexus Maven Repository
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6847
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6847
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Tom White
>            Assignee: Giridharan Kesavan
>            Priority: Blocker
>         Attachments: hadoop-6847.patch
>
>
> I get this error:
> {noformat}
> -mvn-system-deploy:
> [artifact:install-provider] Installing provider: org.apache.maven.wagon:wagon-http:jar:1.0-beta-2:runtime
> [artifact:deploy] Deploying to https://repository.apache.org/content/repositories/snapshots
> [artifact:deploy] Uploading: org/apache/hadoop/hadoop-common-instrumented/0.21.0/hadoop-common-instrumented-0.21.0.jar to apache.snapshots.https
> [artifact:deploy] Uploaded 1331K
> [artifact:deploy] An error has occurred while processing the Maven artifact tasks.
> [artifact:deploy]  Diagnosis:
> [artifact:deploy] 
> [artifact:deploy] Error deploying artifact 'org.apache.hadoop:hadoop-common-instrumented:jar': Error deploying artifact: Failed to transfer file: https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common-instrumented/0.21.0/hadoop-common-instrumented-0.21.0.jar. Return code is: 400
> {noformat}
> Note that 400 is "Bad Request", not an authentication error (401).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9389-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 02:22:49 2010
Return-Path: <common-issues-return-9389-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39677 invoked from network); 8 Jul 2010 02:22:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 02:22:49 -0000
Received: (qmail 4952 invoked by uid 500); 8 Jul 2010 02:22:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4845 invoked by uid 500); 8 Jul 2010 02:22:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4837 invoked by uid 99); 8 Jul 2010 02:22:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 02:22:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 02:22:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o682Es2I013090
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 02:14:54 GMT
Message-ID: <17802648.252331278555294488.JavaMail.jira@thor>
Date: Wed, 7 Jul 2010 22:14:54 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Greg Roelofs updated HADOOP-6835:
---------------------------------

    Attachment: HADOOP-6835.v9.yahoo-0.20.2xx-branch.patch

Final Yahoo 0.20.2xx version, incorporating all relevant trunk updates through "v9".

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>             Fix For: 0.22.0
>
>         Attachments: C6835-9.patch, grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v4.trunk-hadoop-common.patch, HADOOP-6835.v4.trunk-hadoop-mapreduce.patch, HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v5.trunk-hadoop-common.patch, HADOOP-6835.v6.trunk-hadoop-common.patch, HADOOP-6835.v7.trunk-hadoop-common.patch, HADOOP-6835.v8.trunk-hadoop-common.patch, HADOOP-6835.v9.yahoo-0.20.2xx-branch.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9390-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 02:35:17 2010
Return-Path: <common-issues-return-9390-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41084 invoked from network); 8 Jul 2010 02:35:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 02:35:16 -0000
Received: (qmail 13159 invoked by uid 500); 8 Jul 2010 02:35:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13036 invoked by uid 500); 8 Jul 2010 02:35:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13020 invoked by uid 99); 8 Jul 2010 02:35:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 02:35:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 02:35:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o682YoMD013239
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 02:34:50 GMT
Message-ID: <1608963.252671278556490456.JavaMail.jira@thor>
Date: Wed, 7 Jul 2010 22:34:50 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6847) Problem staging 0.21.0 artifacts to
 Apache Nexus Maven Repository
In-Reply-To: <28118738.168371278054229909.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6847:
---------------------------------------

           Status: Resolved  (was: Patch Available)
    Fix Version/s: 0.21.0
       Resolution: Fixed

I have committed this both to branch0.21 and the trunk. Thanks Giri!

> Problem staging 0.21.0 artifacts to Apache Nexus Maven Repository
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6847
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6847
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Tom White
>            Assignee: Giridharan Kesavan
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6847.patch
>
>
> I get this error:
> {noformat}
> -mvn-system-deploy:
> [artifact:install-provider] Installing provider: org.apache.maven.wagon:wagon-http:jar:1.0-beta-2:runtime
> [artifact:deploy] Deploying to https://repository.apache.org/content/repositories/snapshots
> [artifact:deploy] Uploading: org/apache/hadoop/hadoop-common-instrumented/0.21.0/hadoop-common-instrumented-0.21.0.jar to apache.snapshots.https
> [artifact:deploy] Uploaded 1331K
> [artifact:deploy] An error has occurred while processing the Maven artifact tasks.
> [artifact:deploy]  Diagnosis:
> [artifact:deploy] 
> [artifact:deploy] Error deploying artifact 'org.apache.hadoop:hadoop-common-instrumented:jar': Error deploying artifact: Failed to transfer file: https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common-instrumented/0.21.0/hadoop-common-instrumented-0.21.0.jar. Return code is: 400
> {noformat}
> Note that 400 is "Bad Request", not an authentication error (401).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9391-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 02:37:45 2010
Return-Path: <common-issues-return-9391-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41275 invoked from network); 8 Jul 2010 02:37:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 02:37:45 -0000
Received: (qmail 13908 invoked by uid 500); 8 Jul 2010 02:37:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13804 invoked by uid 500); 8 Jul 2010 02:37:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13796 invoked by uid 99); 8 Jul 2010 02:37:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 02:37:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 02:37:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o682TnAh013208
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 02:29:50 GMT
Message-ID: <16480621.252631278556189930.JavaMail.jira@thor>
Date: Wed, 7 Jul 2010 22:29:49 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6853) Common component of HDFS-1045
In-Reply-To: <13947313.250911278548931492.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886203#action_12886203 ] 

Hadoop QA commented on HADOOP-6853:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12448936/HADOOP-6853.patch
  against trunk revision 961532.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/605/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/605/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/605/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/605/console

This message is automatically generated.

> Common component of HDFS-1045
> -----------------------------
>
>                 Key: HADOOP-6853
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6853
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6853.patch
>
>
> HDFS-1045 modified UGI, which is in Common on trunk.  This JIRA is for that change.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9392-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 02:42:46 2010
Return-Path: <common-issues-return-9392-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41750 invoked from network); 8 Jul 2010 02:42:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 02:42:46 -0000
Received: (qmail 16032 invoked by uid 500); 8 Jul 2010 02:42:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15908 invoked by uid 500); 8 Jul 2010 02:42:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15900 invoked by uid 99); 8 Jul 2010 02:42:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 02:42:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 02:42:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o682YpG5013251
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 02:34:51 GMT
Message-ID: <27738531.252711278556491138.JavaMail.jira@thor>
Date: Wed, 7 Jul 2010 22:34:51 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6847) Problem staging 0.21.0 artifacts to
 Apache Nexus Maven Repository
In-Reply-To: <28118738.168371278054229909.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6847:
---------------------------------------

    Component/s: build

> Problem staging 0.21.0 artifacts to Apache Nexus Maven Repository
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6847
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6847
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Tom White
>            Assignee: Giridharan Kesavan
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6847.patch
>
>
> I get this error:
> {noformat}
> -mvn-system-deploy:
> [artifact:install-provider] Installing provider: org.apache.maven.wagon:wagon-http:jar:1.0-beta-2:runtime
> [artifact:deploy] Deploying to https://repository.apache.org/content/repositories/snapshots
> [artifact:deploy] Uploading: org/apache/hadoop/hadoop-common-instrumented/0.21.0/hadoop-common-instrumented-0.21.0.jar to apache.snapshots.https
> [artifact:deploy] Uploaded 1331K
> [artifact:deploy] An error has occurred while processing the Maven artifact tasks.
> [artifact:deploy]  Diagnosis:
> [artifact:deploy] 
> [artifact:deploy] Error deploying artifact 'org.apache.hadoop:hadoop-common-instrumented:jar': Error deploying artifact: Failed to transfer file: https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common-instrumented/0.21.0/hadoop-common-instrumented-0.21.0.jar. Return code is: 400
> {noformat}
> Note that 400 is "Bad Request", not an authentication error (401).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9393-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 02:54:47 2010
Return-Path: <common-issues-return-9393-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42953 invoked from network); 8 Jul 2010 02:54:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 02:54:47 -0000
Received: (qmail 29951 invoked by uid 500); 8 Jul 2010 02:54:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29823 invoked by uid 500); 8 Jul 2010 02:54:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29809 invoked by uid 99); 8 Jul 2010 02:54:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 02:54:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 02:54:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o682koOh013289
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 02:46:50 GMT
Message-ID: <20452092.252771278557210390.JavaMail.jira@thor>
Date: Wed, 7 Jul 2010 22:46:50 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6847) Problem staging 0.21.0 artifacts to
 Apache Nexus Maven Repository
In-Reply-To: <28118738.168371278054229909.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886206#action_12886206 ] 

Hudson commented on HADOOP-6847:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #321 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/321/])
    HADOOP-6847. Problem staging 0.21.0 artifacts to Apache Nexus Maven Repository. Contributed by Giridharan Kesavan


> Problem staging 0.21.0 artifacts to Apache Nexus Maven Repository
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6847
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6847
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Tom White
>            Assignee: Giridharan Kesavan
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6847.patch
>
>
> I get this error:
> {noformat}
> -mvn-system-deploy:
> [artifact:install-provider] Installing provider: org.apache.maven.wagon:wagon-http:jar:1.0-beta-2:runtime
> [artifact:deploy] Deploying to https://repository.apache.org/content/repositories/snapshots
> [artifact:deploy] Uploading: org/apache/hadoop/hadoop-common-instrumented/0.21.0/hadoop-common-instrumented-0.21.0.jar to apache.snapshots.https
> [artifact:deploy] Uploaded 1331K
> [artifact:deploy] An error has occurred while processing the Maven artifact tasks.
> [artifact:deploy]  Diagnosis:
> [artifact:deploy] 
> [artifact:deploy] Error deploying artifact 'org.apache.hadoop:hadoop-common-instrumented:jar': Error deploying artifact: Failed to transfer file: https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common-instrumented/0.21.0/hadoop-common-instrumented-0.21.0.jar. Return code is: 400
> {noformat}
> Note that 400 is "Bad Request", not an authentication error (401).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9394-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 03:05:17 2010
Return-Path: <common-issues-return-9394-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55908 invoked from network); 8 Jul 2010 03:05:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 03:05:17 -0000
Received: (qmail 46402 invoked by uid 500); 8 Jul 2010 03:05:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46253 invoked by uid 500); 8 Jul 2010 03:05:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46236 invoked by uid 99); 8 Jul 2010 03:05:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 03:05:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 03:05:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6834obj013502
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 03:04:50 GMT
Message-ID: <1635893.253071278558290051.JavaMail.jira@thor>
Date: Wed, 7 Jul 2010 23:04:50 -0400 (EDT)
From: "Nicolas Spiegelberg (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6840) Support non-recursive create() in
 FileSystem & SequenceFile.Writer
In-Reply-To: <30089846.67411277503610667.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicolas Spiegelberg updated HADOOP-6840:
----------------------------------------

    Attachment: HADOOP-6840_0.21.patch

0.21 patch, also applies on 0.22 cleanly.  

Kept this patch simple.  Adds 2 static create() API calls to SequenceFile.  The first one adds a 'boolean createParent' and is marked as deprecated.  I would like to add this functionality to 0.20-append, which will use this API call instead of having to import the FileContext code.  The 2nd API call uses the new FileContext class and should be used for clients that don't need 0.20-append backward compatibility.  Please sanity check my usage of the new API.  

No existing APIs were touched to make this change.  Did fix a minor bug where BlockCompressWriter() was not properly looking at io.seqfile.compress.blocksize.

> Support non-recursive create() in FileSystem & SequenceFile.Writer
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6840
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6840
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs, io
>    Affects Versions: 0.21.0, 0.20-append
>            Reporter: Nicolas Spiegelberg
>            Priority: Minor
>         Attachments: HADOOP-6840_0.21.patch
>
>
> The proposed solution for HBASE-2312 requires the sequence file to handle a non-recursive create.  This is already supported by HDFS, but needs to have an equivalent FileSystem & SequenceFile.Writer API.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9395-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 04:03:46 2010
Return-Path: <common-issues-return-9395-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62548 invoked from network); 8 Jul 2010 04:03:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 04:03:46 -0000
Received: (qmail 78552 invoked by uid 500); 8 Jul 2010 04:03:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78427 invoked by uid 500); 8 Jul 2010 04:03:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78419 invoked by uid 99); 8 Jul 2010 04:03:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 04:03:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 04:03:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o683tor9013752
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 03:55:50 GMT
Message-ID: <21492498.253341278561350414.JavaMail.jira@thor>
Date: Wed, 7 Jul 2010 23:55:50 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6850) Allow sequence file to be created
 with different bytes per checksum
In-Reply-To: <17876497.221281278444590187.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886218#action_12886218 ] 

Hairong Kuang commented on HADOOP-6850:
---------------------------------------

It looks that I have to add a new parameter "bytesPerChecksum" to FileSystem#create interface especially that we need this in 0.20. Is there any way that I could get around this FileSystem interface change?

> Allow sequence file to be created with different bytes per checksum
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6850
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6850
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.20-append
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>
> This jira allows a sequence file to be created with a configured bytes per checksum. So we could experiment how this parameter could effect the performance of random reads of a sequence file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9396-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 04:11:48 2010
Return-Path: <common-issues-return-9396-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67179 invoked from network); 8 Jul 2010 04:11:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 04:11:48 -0000
Received: (qmail 82840 invoked by uid 500); 8 Jul 2010 04:11:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82660 invoked by uid 500); 8 Jul 2010 04:11:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82646 invoked by uid 99); 8 Jul 2010 04:11:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 04:11:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 04:11:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6843oN2014078
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 04:03:51 GMT
Message-ID: <16634203.253471278561830831.JavaMail.jira@thor>
Date: Thu, 8 Jul 2010 00:03:50 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6853) Common component of HDFS-1045
In-Reply-To: <13947313.250911278548931492.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886221#action_12886221 ] 

Devaraj Das commented on HADOOP-6853:
-------------------------------------

+1

> Common component of HDFS-1045
> -----------------------------
>
>                 Key: HADOOP-6853
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6853
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6853.patch
>
>
> HDFS-1045 modified UGI, which is in Common on trunk.  This JIRA is for that change.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9397-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 05:16:47 2010
Return-Path: <common-issues-return-9397-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77926 invoked from network); 8 Jul 2010 05:16:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 05:16:47 -0000
Received: (qmail 12847 invoked by uid 500); 8 Jul 2010 05:16:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12708 invoked by uid 500); 8 Jul 2010 05:16:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12700 invoked by uid 99); 8 Jul 2010 05:16:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 05:16:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 05:16:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6858nOi014806
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 05:08:50 GMT
Message-ID: <17513743.254251278565729958.JavaMail.jira@thor>
Date: Thu, 8 Jul 2010 01:08:49 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6854) Cannot Configure 'progress' with
 CreateOpts API
In-Reply-To: <15361457.252001278554389949.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886236#action_12886236 ] 

Chris Douglas commented on HADOOP-6854:
---------------------------------------

A similar case for rename came up in MAPREDUCE-972. Would it be worth considering adding {{Progresable}} support for other {{FileContext}} operations?

> Cannot Configure 'progress' with CreateOpts API
> -----------------------------------------------
>
>                 Key: HADOOP-6854
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6854
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Nicolas Spiegelberg
>            Priority: Minor
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: progress.patch
>
>
> Noticed that CreateOpts class specifies a Progress class, but does not allow you to specify a Progressable object.  Need to add static function to set this API element.  Trying to get FileContext on parity with old FileSystem API.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9398-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 09:17:35 2010
Return-Path: <common-issues-return-9398-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39293 invoked from network); 8 Jul 2010 09:17:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 09:17:35 -0000
Received: (qmail 38779 invoked by uid 500); 8 Jul 2010 09:17:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36815 invoked by uid 500); 8 Jul 2010 09:17:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36801 invoked by uid 99); 8 Jul 2010 09:17:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 09:17:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 09:17:28 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o689H6td017089
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 09:17:06 GMT
Message-ID: <4700367.257631278580626161.JavaMail.jira@thor>
Date: Thu, 8 Jul 2010 05:17:06 -0400 (EDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6849) Have
 LocalDirAllocator.AllocatorPerContext.getLocalPathForWrite fail more
 meaningfully
In-Reply-To: <28072519.190301278179329541.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886290#action_12886290 ] 

Steve Loughran commented on HADOOP-6849:
----------------------------------------

-it's been mentioned to me that as well as no disk space, permissions can trigger these problems. It is similar to the DN and NN startup routines where the code runs through a list of dirs and tries to find any it can work with.

 we could have some helper class that "diagnoses" a directory: does it exist, what is its permissions, can you write to it, etc, which we'd then use in this method and any other that has similar problems. We could unit test that code, then on an error, pass in the list of dirs and let it generate the diagnostics to log and attach to the exception.

> Have LocalDirAllocator.AllocatorPerContext.getLocalPathForWrite fail more meaningfully
> --------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6849
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6849
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.20.2
>            Reporter: Steve Loughran
>            Priority: Minor
>
> A stack trace makes it way to me, of a reduce failing
> {code}
> Caused by: org\.apache\.hadoop\.util\.DiskChecker$DiskErrorException: Could not find any valid local directory for file:/mnt/data/dfs/data/mapred/local/taskTracker/jobcache/job_201007011427_0001/attempt_201007011427_0001_r_000000_1/output/map_96\.out
>       at org\.apache\.hadoop\.fs\.LocalDirAllocator$AllocatorPerContext\.getLocalPathForWrite(LocalDirAllocator\.java:343)
>       at org\.apache\.hadoop\.fs\.LocalDirAllocator\.getLocalPathForWrite(LocalDirAllocator\.java:124)
>       at org\.apache\.hadoop\.mapred\.ReduceTask$ReduceCopier$LocalFSMerger\.run(ReduceTask\.java:2434)
> {code}
> We're probably running out of HDD space, if not its configuration problems. Either way, some more hints in the exception would be handy.
> # Include the size of the output file looked for if known
> # Include the list of dirs examined and their reason for rejection (not found or if not enough room, available space).
> This would make it easier to diagnose problems after the event, with nothing but emailed logs for diagnostics.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9399-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 09:33:19 2010
Return-Path: <common-issues-return-9399-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42737 invoked from network); 8 Jul 2010 09:33:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 09:33:18 -0000
Received: (qmail 58953 invoked by uid 500); 8 Jul 2010 09:33:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58829 invoked by uid 500); 8 Jul 2010 09:33:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58816 invoked by uid 99); 8 Jul 2010 09:33:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 09:33:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 09:33:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o689Wp0D017209
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 09:32:51 GMT
Message-ID: <31187941.257821278581571422.JavaMail.jira@thor>
Date: Thu, 8 Jul 2010 05:32:51 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5913) Allow administrators to be able to
 start and stop queues
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-5913:
----------------------------------

    Attachment: C5913-15y20s.patch

Avoid incompatible change to queue configuration.

> Allow administrators to be able to start and stop queues 
> ---------------------------------------------------------
>
>                 Key: HADOOP-5913
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5913
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: rahul k singh
>            Assignee: rahul k singh
>             Fix For: 0.21.0
>
>         Attachments: C5913-14y20s.patch, C5913-15y20s.patch, hadoop-5913-1.patch, hadoop-5913-10.patch, HADOOP-5913-11.patch, HADOOP-5913-12.patch, HADOOP-5913-13.patch, HADOOP-5913-14.patch, hadoop-5913-2.patch, hadoop-5913-3.patch, hadoop-5913-4.patch, hadoop-5913-5.patch, hadoop-5913-6.patch, hadoop-5913-7.patch, hadoop-5913-8.patch, hadoop-5913-9.patch
>
>
>   This feature would provide functionality to stop and start queues in Hadoop at runtime. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9400-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 14:10:50 2010
Return-Path: <common-issues-return-9400-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30026 invoked from network); 8 Jul 2010 14:10:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 14:10:50 -0000
Received: (qmail 89641 invoked by uid 500); 8 Jul 2010 14:10:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89328 invoked by uid 500); 8 Jul 2010 14:10:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89311 invoked by uid 99); 8 Jul 2010 14:10:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 14:10:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 14:10:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o68E2r0f019109
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 14:02:53 GMT
Message-ID: <29018164.259791278597773514.JavaMail.jira@thor>
Date: Thu, 8 Jul 2010 10:02:53 -0400 (EDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6850) Allow sequence file to be created
 with different bytes per checksum
In-Reply-To: <17876497.221281278444590187.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886326#action_12886326 ] 

dhruba borthakur commented on HADOOP-6850:
------------------------------------------

typically, io.bytes.per.checksum is a server side configuration because an administrator usually wants all the files in his/her cluster to have to the same value for this config. Will this be sufficient for you?

> Allow sequence file to be created with different bytes per checksum
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6850
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6850
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.20-append
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>
> This jira allows a sequence file to be created with a configured bytes per checksum. So we could experiment how this parameter could effect the performance of random reads of a sequence file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9401-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 16:27:46 2010
Return-Path: <common-issues-return-9401-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74699 invoked from network); 8 Jul 2010 16:27:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 16:27:46 -0000
Received: (qmail 4659 invoked by uid 500); 8 Jul 2010 16:27:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4590 invoked by uid 500); 8 Jul 2010 16:27:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4582 invoked by uid 99); 8 Jul 2010 16:27:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 16:27:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 16:27:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o68GJoMo020200
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 16:19:50 GMT
Message-ID: <20718048.261321278605990515.JavaMail.jira@thor>
Date: Thu, 8 Jul 2010 12:19:50 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6850) Allow sequence file to be created
 with different bytes per checksum
In-Reply-To: <17876497.221281278444590187.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886359#action_12886359 ] 

Hairong Kuang commented on HADOOP-6850:
---------------------------------------

In the use case of HBASE-2478, we want to configure HLog & HFile with different values of bytes per checksum.

> Allow sequence file to be created with different bytes per checksum
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6850
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6850
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>    Affects Versions: 0.20-append
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>
> This jira allows a sequence file to be created with a configured bytes per checksum. So we could experiment how this parameter could effect the performance of random reads of a sequence file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9402-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 19:07:45 2010
Return-Path: <common-issues-return-9402-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32937 invoked from network); 8 Jul 2010 19:07:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 19:07:45 -0000
Received: (qmail 91959 invoked by uid 500); 8 Jul 2010 19:07:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91919 invoked by uid 500); 8 Jul 2010 19:07:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91911 invoked by uid 99); 8 Jul 2010 19:07:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 19:07:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 19:07:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o68IxoOn021865
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 18:59:50 GMT
Message-ID: <22553053.264841278615590269.JavaMail.jira@thor>
Date: Thu, 8 Jul 2010 14:59:50 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6845) TokenStorage renamed to
 Credentials.
In-Reply-To: <11027372.98871277758850634.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886411#action_12886411 ] 

Devaraj Das commented on HADOOP-6845:
-------------------------------------

Looks good. Minor nits - Credentials.java should be annotated with the same annotations that was there in TokenStorage,java. Also, the class TestJobCredentials should be renamed to TestCredentials. Although not related to this refactoring patch, do you want to have a teardown method in the testcase that removes the directory created in the setup method?

> TokenStorage renamed to Credentials.
> ------------------------------------
>
>                 Key: HADOOP-6845
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6845
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6845.1.patch
>
>
> This jira tracks common changes for MAPREDUCE-1528.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9403-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 19:59:16 2010
Return-Path: <common-issues-return-9403-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42388 invoked from network); 8 Jul 2010 19:59:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 19:59:15 -0000
Received: (qmail 29536 invoked by uid 500); 8 Jul 2010 19:59:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29472 invoked by uid 500); 8 Jul 2010 19:59:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29464 invoked by uid 99); 8 Jul 2010 19:59:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 19:59:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 19:59:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o68JwomA022676
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 19:58:50 GMT
Message-ID: <27641021.266641278619130491.JavaMail.jira@thor>
Date: Thu, 8 Jul 2010 15:58:50 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6853) Common component of HDFS-1045
In-Reply-To: <13947313.250911278548931492.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886437#action_12886437 ] 

Jakob Homan commented on HADOOP-6853:
-------------------------------------

Re: reported new javadoc warning...
{noformat}     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:31: warning: sun.security.krb5.Config is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.Config;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:32: warning: sun.security.krb5.KrbException is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.KrbException;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:81: warning: sun.security.krb5.Config is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc]   private static Config kerbConf;
     [exec]   [javadoc]                  ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/SecurityUtil.java:33: warning: sun.security.jgss.krb5.Krb5Util is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.jgss.krb5.Krb5Util;
     [exec]   [javadoc]                              ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/SecurityUtil.java:34: warning: sun.security.krb5.Credentials is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.Credentials;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/SecurityUtil.java:35: warning: sun.security.krb5.PrincipalName is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.PrincipalName;
{noformat}
This patch doesn't touch those files, so I'm not sure where the warning is coming from.  It looks like HADOOP-6854 got hit with it too.  I'm calling bogus and will commit.

> Common component of HDFS-1045
> -----------------------------
>
>                 Key: HADOOP-6853
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6853
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6853.patch
>
>
> HDFS-1045 modified UGI, which is in Common on trunk.  This JIRA is for that change.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9404-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 20:12:45 2010
Return-Path: <common-issues-return-9404-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51097 invoked from network); 8 Jul 2010 20:12:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 20:12:44 -0000
Received: (qmail 39540 invoked by uid 500); 8 Jul 2010 20:12:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39497 invoked by uid 500); 8 Jul 2010 20:12:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39489 invoked by uid 99); 8 Jul 2010 20:12:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 20:12:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 20:12:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o68K4nhV022755
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 20:04:50 GMT
Message-ID: <6331424.266681278619489912.JavaMail.jira@thor>
Date: Thu, 8 Jul 2010 16:04:49 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6853) Common component of HDFS-1045
In-Reply-To: <13947313.250911278548931492.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6853:
--------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

I've committed this.  Resolving as fixed.

> Common component of HDFS-1045
> -----------------------------
>
>                 Key: HADOOP-6853
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6853
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6853.patch
>
>
> HDFS-1045 modified UGI, which is in Common on trunk.  This JIRA is for that change.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9405-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 20:23:19 2010
Return-Path: <common-issues-return-9405-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53382 invoked from network); 8 Jul 2010 20:23:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 20:23:18 -0000
Received: (qmail 44484 invoked by uid 500); 8 Jul 2010 20:23:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44434 invoked by uid 500); 8 Jul 2010 20:23:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44425 invoked by uid 99); 8 Jul 2010 20:23:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 20:23:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 20:23:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o68KMs4Q022912
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 20:22:54 GMT
Message-ID: <30122127.267011278620574135.JavaMail.jira@thor>
Date: Thu, 8 Jul 2010 16:22:54 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6853) Common component of HDFS-1045
In-Reply-To: <13947313.250911278548931492.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886450#action_12886450 ] 

Hudson commented on HADOOP-6853:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #322 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/322/])
    HADOOP-6853. Common component of HDFS-1045.


> Common component of HDFS-1045
> -----------------------------
>
>                 Key: HADOOP-6853
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6853
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6853.patch
>
>
> HDFS-1045 modified UGI, which is in Common on trunk.  This JIRA is for that change.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9406-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 20:32:54 2010
Return-Path: <common-issues-return-9406-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57006 invoked from network); 8 Jul 2010 20:32:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 20:32:54 -0000
Received: (qmail 53841 invoked by uid 500); 8 Jul 2010 20:32:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53704 invoked by uid 500); 8 Jul 2010 20:32:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53696 invoked by uid 99); 8 Jul 2010 20:32:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 20:32:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 20:32:50 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o68KOxln023029
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 20:24:59 GMT
Message-ID: <820970.267291278620699456.JavaMail.jira@thor>
Date: Thu, 8 Jul 2010 16:24:59 -0400 (EDT)
From: "Nicholas Carlini (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6349) Implement FastLZCodec for
 fastlz/lzo algorithm
In-Reply-To: <289817036.1256941439448.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886452#action_12886452 ] 

Nicholas Carlini commented on HADOOP-6349:
------------------------------------------

Is anyone currently working on this? I'm currently working on adding LZMA compression ([HADOOP-6837|https://issues.apache.org/jira/browse/HADOOP-6837]), but after I finish that if no one else is then I'll work on this.

> Implement FastLZCodec for fastlz/lzo algorithm
> ----------------------------------------------
>
>                 Key: HADOOP-6349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6349
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: William Kinney
>         Attachments: HADOOP-6349-TestFastLZCodec.patch, HADOOP-6349.patch, TestCodecPerformance.java, TestCodecPerformance.java, testCodecPerfResults.tsv
>
>
> Per  [HADOOP-4874|http://issues.apache.org/jira/browse/HADOOP-4874], FastLZ is a good (speed, license) alternative to LZO. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9407-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 21:22:16 2010
Return-Path: <common-issues-return-9407-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75187 invoked from network); 8 Jul 2010 21:22:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 21:22:16 -0000
Received: (qmail 11764 invoked by uid 500); 8 Jul 2010 21:22:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11725 invoked by uid 500); 8 Jul 2010 21:22:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11714 invoked by uid 99); 8 Jul 2010 21:22:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 21:22:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 21:22:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o68LLpOp023726
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 21:21:51 GMT
Message-ID: <31625574.268541278624111423.JavaMail.jira@thor>
Date: Thu, 8 Jul 2010 17:21:51 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6845) TokenStorage renamed to Credentials.
In-Reply-To: <11027372.98871277758850634.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6845:
-----------------------------------------

    Status: Open  (was: Patch Available)

> TokenStorage renamed to Credentials.
> ------------------------------------
>
>                 Key: HADOOP-6845
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6845
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6845.1.patch, HADOOP-6845.2.patch
>
>
> This jira tracks common changes for MAPREDUCE-1528.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9408-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 21:22:18 2010
Return-Path: <common-issues-return-9408-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75238 invoked from network); 8 Jul 2010 21:22:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 21:22:18 -0000
Received: (qmail 11917 invoked by uid 500); 8 Jul 2010 21:22:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11881 invoked by uid 500); 8 Jul 2010 21:22:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11873 invoked by uid 99); 8 Jul 2010 21:22:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 21:22:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 21:22:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o68LLrDg023744
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 21:21:53 GMT
Message-ID: <13562973.268601278624113818.JavaMail.jira@thor>
Date: Thu, 8 Jul 2010 17:21:53 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6845) TokenStorage renamed to Credentials.
In-Reply-To: <11027372.98871277758850634.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6845:
-----------------------------------------

    Status: Patch Available  (was: Open)

> TokenStorage renamed to Credentials.
> ------------------------------------
>
>                 Key: HADOOP-6845
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6845
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6845.1.patch, HADOOP-6845.2.patch
>
>
> This jira tracks common changes for MAPREDUCE-1528.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9409-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 21:29:45 2010
Return-Path: <common-issues-return-9409-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77875 invoked from network); 8 Jul 2010 21:29:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 21:29:45 -0000
Received: (qmail 25747 invoked by uid 500); 8 Jul 2010 21:29:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25701 invoked by uid 500); 8 Jul 2010 21:29:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25692 invoked by uid 99); 8 Jul 2010 21:29:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 21:29:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 21:29:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o68LLot8023720
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 21:21:51 GMT
Message-ID: <22588922.268521278624110914.JavaMail.jira@thor>
Date: Thu, 8 Jul 2010 17:21:50 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6845) TokenStorage renamed to Credentials.
In-Reply-To: <11027372.98871277758850634.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6845:
-----------------------------------------

    Attachment: HADOOP-6845.2.patch

New patch addressing the comments.

> TokenStorage renamed to Credentials.
> ------------------------------------
>
>                 Key: HADOOP-6845
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6845
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6845.1.patch, HADOOP-6845.2.patch
>
>
> This jira tracks common changes for MAPREDUCE-1528.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9410-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 21:33:25 2010
Return-Path: <common-issues-return-9410-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79769 invoked from network); 8 Jul 2010 21:33:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 21:33:24 -0000
Received: (qmail 35532 invoked by uid 500); 8 Jul 2010 21:33:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35420 invoked by uid 500); 8 Jul 2010 21:33:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35412 invoked by uid 99); 8 Jul 2010 21:33:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 21:33:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 21:33:21 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o68LWxi7024064
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 21:32:59 GMT
Message-ID: <11011299.269351278624779341.JavaMail.jira@thor>
Date: Thu, 8 Jul 2010 17:32:59 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6349) Implement FastLZCodec for
 fastlz/lzo algorithm
In-Reply-To: <289817036.1256941439448.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886495#action_12886495 ] 

Eli Collins commented on HADOOP-6349:
-------------------------------------

I don't think so. I pinged William a while back picked up the patch and fixed a bug or two but haven't had the cycles recently, will try to rebase on trunk and post soon.  Would be awesome to have you pick this up.

> Implement FastLZCodec for fastlz/lzo algorithm
> ----------------------------------------------
>
>                 Key: HADOOP-6349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6349
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: William Kinney
>         Attachments: HADOOP-6349-TestFastLZCodec.patch, HADOOP-6349.patch, TestCodecPerformance.java, TestCodecPerformance.java, testCodecPerfResults.tsv
>
>
> Per  [HADOOP-4874|http://issues.apache.org/jira/browse/HADOOP-4874], FastLZ is a good (speed, license) alternative to LZO. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9411-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 21:53:15 2010
Return-Path: <common-issues-return-9411-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89702 invoked from network); 8 Jul 2010 21:53:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 21:53:15 -0000
Received: (qmail 73687 invoked by uid 500); 8 Jul 2010 21:53:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73622 invoked by uid 500); 8 Jul 2010 21:53:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73606 invoked by uid 99); 8 Jul 2010 21:53:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 21:53:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 21:53:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o68LqobE024452
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 21:52:50 GMT
Message-ID: <28555124.270201278625970824.JavaMail.jira@thor>
Date: Thu, 8 Jul 2010 17:52:50 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6845) TokenStorage renamed to Credentials.
In-Reply-To: <11027372.98871277758850634.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6845:
-----------------------------------------

    Status: Open  (was: Patch Available)

> TokenStorage renamed to Credentials.
> ------------------------------------
>
>                 Key: HADOOP-6845
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6845
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6845.1.patch, HADOOP-6845.2.patch, HADOOP-6845.3.patch
>
>
> This jira tracks common changes for MAPREDUCE-1528.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9412-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 21:53:16 2010
Return-Path: <common-issues-return-9412-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89749 invoked from network); 8 Jul 2010 21:53:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 21:53:16 -0000
Received: (qmail 73847 invoked by uid 500); 8 Jul 2010 21:53:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73632 invoked by uid 500); 8 Jul 2010 21:53:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73607 invoked by uid 99); 8 Jul 2010 21:53:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 21:53:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 21:53:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o68LqoGq024446
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 21:52:50 GMT
Message-ID: <17783087.270181278625970306.JavaMail.jira@thor>
Date: Thu, 8 Jul 2010 17:52:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6845) TokenStorage renamed to
 Credentials.
In-Reply-To: <11027372.98871277758850634.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886512#action_12886512 ] 

Hadoop QA commented on HADOOP-6845:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12449007/HADOOP-6845.2.patch
  against trunk revision 961911.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/606/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/606/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/606/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/606/console

This message is automatically generated.

> TokenStorage renamed to Credentials.
> ------------------------------------
>
>                 Key: HADOOP-6845
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6845
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6845.1.patch, HADOOP-6845.2.patch, HADOOP-6845.3.patch
>
>
> This jira tracks common changes for MAPREDUCE-1528.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9413-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 21:53:16 2010
Return-Path: <common-issues-return-9413-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89759 invoked from network); 8 Jul 2010 21:53:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 21:53:16 -0000
Received: (qmail 73860 invoked by uid 500); 8 Jul 2010 21:53:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73778 invoked by uid 500); 8 Jul 2010 21:53:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73623 invoked by uid 99); 8 Jul 2010 21:53:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 21:53:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 21:53:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o68LqpVS024458
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 21:52:51 GMT
Message-ID: <30561283.270221278625971167.JavaMail.jira@thor>
Date: Thu, 8 Jul 2010 17:52:51 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6845) TokenStorage renamed to Credentials.
In-Reply-To: <11027372.98871277758850634.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6845:
-----------------------------------------

    Attachment: HADOOP-6845.3.patch

Updated InterfaceAudience annotation for Credentials to include HDFS as well.

> TokenStorage renamed to Credentials.
> ------------------------------------
>
>                 Key: HADOOP-6845
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6845
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6845.1.patch, HADOOP-6845.2.patch, HADOOP-6845.3.patch
>
>
> This jira tracks common changes for MAPREDUCE-1528.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9414-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 22:02:45 2010
Return-Path: <common-issues-return-9414-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93700 invoked from network); 8 Jul 2010 22:02:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 22:02:45 -0000
Received: (qmail 85784 invoked by uid 500); 8 Jul 2010 22:02:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85715 invoked by uid 500); 8 Jul 2010 22:02:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85707 invoked by uid 99); 8 Jul 2010 22:02:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 22:02:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 22:02:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o68LsoLK024470
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 21:54:50 GMT
Message-ID: <21651296.270241278626090258.JavaMail.jira@thor>
Date: Thu, 8 Jul 2010 17:54:50 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6845) TokenStorage renamed to Credentials.
In-Reply-To: <11027372.98871277758850634.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6845:
-----------------------------------------

    Status: Patch Available  (was: Open)

> TokenStorage renamed to Credentials.
> ------------------------------------
>
>                 Key: HADOOP-6845
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6845
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6845.1.patch, HADOOP-6845.2.patch, HADOOP-6845.3.patch
>
>
> This jira tracks common changes for MAPREDUCE-1528.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9415-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 08 22:32:18 2010
Return-Path: <common-issues-return-9415-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4783 invoked from network); 8 Jul 2010 22:32:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jul 2010 22:32:17 -0000
Received: (qmail 12544 invoked by uid 500); 8 Jul 2010 22:32:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12422 invoked by uid 500); 8 Jul 2010 22:32:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12217 invoked by uid 99); 8 Jul 2010 22:32:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 22:32:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Jul 2010 22:32:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o68MVq2J025122
	for <common-issues@hadoop.apache.org>; Thu, 8 Jul 2010 22:31:52 GMT
Message-ID: <8107855.271821278628312339.JavaMail.jira@thor>
Date: Thu, 8 Jul 2010 18:31:52 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6845) TokenStorage renamed to
 Credentials.
In-Reply-To: <11027372.98871277758850634.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886527#action_12886527 ] 

Hadoop QA commented on HADOOP-6845:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12449017/HADOOP-6845.3.patch
  against trunk revision 961911.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/607/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/607/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/607/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/607/console

This message is automatically generated.

> TokenStorage renamed to Credentials.
> ------------------------------------
>
>                 Key: HADOOP-6845
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6845
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6845.1.patch, HADOOP-6845.2.patch, HADOOP-6845.3.patch
>
>
> This jira tracks common changes for MAPREDUCE-1528.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9416-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 00:33:17 2010
Return-Path: <common-issues-return-9416-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56246 invoked from network); 9 Jul 2010 00:33:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 00:33:17 -0000
Received: (qmail 23626 invoked by uid 500); 9 Jul 2010 00:33:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23441 invoked by uid 500); 9 Jul 2010 00:33:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23431 invoked by uid 99); 9 Jul 2010 00:33:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 00:33:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 00:33:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o690WoFI026466
	for <common-issues@hadoop.apache.org>; Fri, 9 Jul 2010 00:32:51 GMT
Message-ID: <23521712.275021278635570824.JavaMail.jira@thor>
Date: Thu, 8 Jul 2010 20:32:50 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6855) Add ability to get groups for ACLs
 from 'getent netgroup'
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Add ability to get groups for ACLs from 'getent netgroup'
---------------------------------------------------------

                 Key: HADOOP-6855
                 URL: https://issues.apache.org/jira/browse/HADOOP-6855
             Project: Hadoop Common
          Issue Type: New Feature
          Components: security
    Affects Versions: 0.22.0
            Reporter: Erik Steffl
             Fix For: 0.22.0


Add ability to specify netgroups in ACLs (see class AccessControlList.java). Membership of users in netgroups will be determined by running 'getent negroups $groupName'. Netgroups will be differentiated from regular unix groups by having '@' as a first character.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9417-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 01:06:16 2010
Return-Path: <common-issues-return-9417-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66070 invoked from network); 9 Jul 2010 01:06:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 01:06:16 -0000
Received: (qmail 57424 invoked by uid 500); 9 Jul 2010 01:06:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57380 invoked by uid 500); 9 Jul 2010 01:06:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57370 invoked by uid 99); 9 Jul 2010 01:06:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 01:06:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 01:06:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6915pwE026635
	for <common-issues@hadoop.apache.org>; Fri, 9 Jul 2010 01:05:51 GMT
Message-ID: <12758641.275211278637551021.JavaMail.jira@thor>
Date: Thu, 8 Jul 2010 21:05:51 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6845) TokenStorage renamed to
 Credentials.
In-Reply-To: <11027372.98871277758850634.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886572#action_12886572 ] 

Devaraj Das commented on HADOOP-6845:
-------------------------------------

+1

> TokenStorage renamed to Credentials.
> ------------------------------------
>
>                 Key: HADOOP-6845
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6845
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6845.1.patch, HADOOP-6845.2.patch, HADOOP-6845.3.patch
>
>
> This jira tracks common changes for MAPREDUCE-1528.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9418-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 01:08:15 2010
Return-Path: <common-issues-return-9418-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66462 invoked from network); 9 Jul 2010 01:08:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 01:08:15 -0000
Received: (qmail 58109 invoked by uid 500); 9 Jul 2010 01:08:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58070 invoked by uid 500); 9 Jul 2010 01:08:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58062 invoked by uid 99); 9 Jul 2010 01:08:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 01:08:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 01:08:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6917oMo026648
	for <common-issues@hadoop.apache.org>; Fri, 9 Jul 2010 01:07:50 GMT
Message-ID: <23548238.275251278637670062.JavaMail.jira@thor>
Date: Thu, 8 Jul 2010 21:07:50 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6845) TokenStorage renamed to
 Credentials.
In-Reply-To: <11027372.98871277758850634.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886574#action_12886574 ] 

Jitendra Nath Pandey commented on HADOOP-6845:
----------------------------------------------

Javadoc warnings are in KerberosName.java and SecurityUtil.java which are unrelated.

> TokenStorage renamed to Credentials.
> ------------------------------------
>
>                 Key: HADOOP-6845
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6845
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6845.1.patch, HADOOP-6845.2.patch, HADOOP-6845.3.patch
>
>
> This jira tracks common changes for MAPREDUCE-1528.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9419-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 11:14:19 2010
Return-Path: <common-issues-return-9419-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25129 invoked from network); 9 Jul 2010 11:14:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 11:14:19 -0000
Received: (qmail 75607 invoked by uid 500); 9 Jul 2010 11:14:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75354 invoked by uid 500); 9 Jul 2010 11:14:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75336 invoked by uid 99); 9 Jul 2010 11:14:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 11:14:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 11:14:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o69BDoiK001565
	for <common-issues@hadoop.apache.org>; Fri, 9 Jul 2010 11:13:51 GMT
Message-ID: <27997701.282171278674030766.JavaMail.jira@thor>
Date: Fri, 9 Jul 2010 07:13:50 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6853) Common component of HDFS-1045
In-Reply-To: <13947313.250911278548931492.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886680#action_12886680 ] 

Hudson commented on HADOOP-6853:
--------------------------------

Integrated in Hadoop-Common-trunk #390 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/390/])
    HADOOP-6853. Common component of HDFS-1045.


> Common component of HDFS-1045
> -----------------------------
>
>                 Key: HADOOP-6853
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6853
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6853.patch
>
>
> HDFS-1045 modified UGI, which is in Common on trunk.  This JIRA is for that change.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9420-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 11:14:21 2010
Return-Path: <common-issues-return-9420-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25181 invoked from network); 9 Jul 2010 11:14:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 11:14:21 -0000
Received: (qmail 75677 invoked by uid 500); 9 Jul 2010 11:14:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75627 invoked by uid 500); 9 Jul 2010 11:14:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75616 invoked by uid 99); 9 Jul 2010 11:14:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 11:14:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 11:14:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o69BDtqL001614
	for <common-issues@hadoop.apache.org>; Fri, 9 Jul 2010 11:13:55 GMT
Message-ID: <12188569.282331278674035615.JavaMail.jira@thor>
Date: Fri, 9 Jul 2010 07:13:55 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886682#action_12886682 ] 

Hudson commented on HADOOP-6835:
--------------------------------

Integrated in Hadoop-Common-trunk #390 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/390/])
    

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.20.2
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>             Fix For: 0.22.0
>
>         Attachments: C6835-9.patch, grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v4.trunk-hadoop-common.patch, HADOOP-6835.v4.trunk-hadoop-mapreduce.patch, HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v5.trunk-hadoop-common.patch, HADOOP-6835.v6.trunk-hadoop-common.patch, HADOOP-6835.v7.trunk-hadoop-common.patch, HADOOP-6835.v8.trunk-hadoop-common.patch, HADOOP-6835.v9.yahoo-0.20.2xx-branch.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9421-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 11:21:53 2010
Return-Path: <common-issues-return-9421-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28426 invoked from network); 9 Jul 2010 11:21:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 11:21:53 -0000
Received: (qmail 83751 invoked by uid 500); 9 Jul 2010 11:21:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83622 invoked by uid 500); 9 Jul 2010 11:21:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83608 invoked by uid 99); 9 Jul 2010 11:21:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 11:21:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 11:21:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o69BDu6m001626
	for <common-issues@hadoop.apache.org>; Fri, 9 Jul 2010 11:13:56 GMT
Message-ID: <16902239.282371278674036168.JavaMail.jira@thor>
Date: Fri, 9 Jul 2010 07:13:56 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6847) Problem staging 0.21.0 artifacts to
 Apache Nexus Maven Repository
In-Reply-To: <28118738.168371278054229909.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886683#action_12886683 ] 

Hudson commented on HADOOP-6847:
--------------------------------

Integrated in Hadoop-Common-trunk #390 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/390/])
    

> Problem staging 0.21.0 artifacts to Apache Nexus Maven Repository
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6847
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6847
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Tom White
>            Assignee: Giridharan Kesavan
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6847.patch
>
>
> I get this error:
> {noformat}
> -mvn-system-deploy:
> [artifact:install-provider] Installing provider: org.apache.maven.wagon:wagon-http:jar:1.0-beta-2:runtime
> [artifact:deploy] Deploying to https://repository.apache.org/content/repositories/snapshots
> [artifact:deploy] Uploading: org/apache/hadoop/hadoop-common-instrumented/0.21.0/hadoop-common-instrumented-0.21.0.jar to apache.snapshots.https
> [artifact:deploy] Uploaded 1331K
> [artifact:deploy] An error has occurred while processing the Maven artifact tasks.
> [artifact:deploy]  Diagnosis:
> [artifact:deploy] 
> [artifact:deploy] Error deploying artifact 'org.apache.hadoop:hadoop-common-instrumented:jar': Error deploying artifact: Failed to transfer file: https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-common-instrumented/0.21.0/hadoop-common-instrumented-0.21.0.jar. Return code is: 400
> {noformat}
> Note that 400 is "Bad Request", not an authentication error (401).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9422-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 16:06:15 2010
Return-Path: <common-issues-return-9422-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30123 invoked from network); 9 Jul 2010 16:06:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 16:06:15 -0000
Received: (qmail 75612 invoked by uid 500); 9 Jul 2010 16:06:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75510 invoked by uid 500); 9 Jul 2010 16:06:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75495 invoked by uid 99); 9 Jul 2010 16:06:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 16:06:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 16:06:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o69G5nqE005098
	for <common-issues@hadoop.apache.org>; Fri, 9 Jul 2010 16:05:50 GMT
Message-ID: <19355698.285171278691549769.JavaMail.jira@thor>
Date: Fri, 9 Jul 2010 12:05:49 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6856) SequenceFile and MapFile need cleanup
 to remove redundant constructors
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

SequenceFile and MapFile need cleanup to remove redundant constructors
----------------------------------------------------------------------

                 Key: HADOOP-6856
                 URL: https://issues.apache.org/jira/browse/HADOOP-6856
             Project: Hadoop Common
          Issue Type: Improvement
          Components: io
            Reporter: Owen O'Malley
            Assignee: Owen O'Malley
             Fix For: 0.22.0


Currently there are 2 public SequenceFile.Reader constructors, 3 public SequenceFile.Writer constructors, 9 public SequenceFile.createWriter, 2 public MapFile.Reader constructors, and 8 public MapFile.Writer constructors. All of with various historical combinations of parameters that don't cover the entire space.

All of this makes it *very* difficult to add new optional parameters to SequenceFile and MapFile. 

I'd like change to the style of FileContext.create with option parameters. I'll implement one public SequenceFile.Reader constructor and one public SequenceFile.createWriter and implement all of the current variants based on those two. I'll do the same for MapFile.Reader and MapFile.Writer including passing parameters down to the underlying SequenceFile.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9423-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 18:26:17 2010
Return-Path: <common-issues-return-9423-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73463 invoked from network); 9 Jul 2010 18:26:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 18:26:16 -0000
Received: (qmail 40258 invoked by uid 500); 9 Jul 2010 18:26:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40186 invoked by uid 500); 9 Jul 2010 18:26:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40178 invoked by uid 99); 9 Jul 2010 18:26:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 18:26:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 18:26:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o69IPqfR006802
	for <common-issues@hadoop.apache.org>; Fri, 9 Jul 2010 18:25:52 GMT
Message-ID: <20535388.289261278699952332.JavaMail.jira@thor>
Date: Fri, 9 Jul 2010 14:25:52 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6656:
--------------------------------

    Attachment: 6656-trunk-1.patch

Attaching the trunk patch. Unfortunately, I realized towards the end that the patch requires the "common" part of MAPREDUCE-1566 to be committed first. So this won't compile now but should merge cleanly once the common part of MAPREDUCE-1566 patch is committed.

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 6656-trunk-1.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9424-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 19:31:17 2010
Return-Path: <common-issues-return-9424-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89673 invoked from network); 9 Jul 2010 19:31:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 19:31:16 -0000
Received: (qmail 15612 invoked by uid 500); 9 Jul 2010 19:31:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15544 invoked by uid 500); 9 Jul 2010 19:31:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15536 invoked by uid 99); 9 Jul 2010 19:31:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 19:31:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 19:31:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o69JUqgW007698
	for <common-issues@hadoop.apache.org>; Fri, 9 Jul 2010 19:30:52 GMT
Message-ID: <28959100.291501278703852046.JavaMail.jira@thor>
Date: Fri, 9 Jul 2010 15:30:52 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6836) [Herriot] Test Autoamtion : Generic
 method for adding/modifying the attributes for new configuration.
In-Reply-To: <23644117.9641277269730469.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886849#action_12886849 ] 

Vinay Kumar Thota commented on HADOOP-6836:
-------------------------------------------

Cos, patch is working fine and I don't see any issues.Please commit into trunk.

> [Herriot] Test Autoamtion : Generic method for adding/modifying the attributes for new configuration.
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6836
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6836
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>            Reporter: Iyappan Srinivasan
>            Assignee: Vinay Kumar Thota
>         Attachments: 6836-ydist-security.patch, HADOOP-6836.patch
>
>
> HADOP-6772 deals with 
> Common utilities for system tests.
> 1. A method for restarting the daemon with new configuration.
> c throws Exception; 
> 2. A method for restarting the daemon with default configuration.
> public void restart() throws Exception;
> 3. A method for waiting until daemon is stop.
> public void waitForClusterToStop() throws Exception;
> In this  some variables are made of String, instead of Long. Those needs ot be changed too. So, can this method 
> "public static void restartClusterWithNewConfig(Hashtable<String,Long> props, String confFile) throws Exception;"
> be generalized to accepts string too. All otehr methods should work as usual.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9425-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 19:36:45 2010
Return-Path: <common-issues-return-9425-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90334 invoked from network); 9 Jul 2010 19:36:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 19:36:45 -0000
Received: (qmail 21282 invoked by uid 500); 9 Jul 2010 19:36:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21055 invoked by uid 500); 9 Jul 2010 19:36:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21047 invoked by uid 99); 9 Jul 2010 19:36:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 19:36:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 19:36:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o69JSoZI007643
	for <common-issues@hadoop.apache.org>; Fri, 9 Jul 2010 19:28:50 GMT
Message-ID: <11895030.291341278703730442.JavaMail.jira@thor>
Date: Fri, 9 Jul 2010 15:28:50 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6819) [Herriot] Shell command for getting
 the new exceptions in the logs returning exitcode 1 after executing
 successfully.
In-Reply-To: <25211660.39321276233134711.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886846#action_12886846 ] 

Vinay Kumar Thota commented on HADOOP-6819:
-------------------------------------------

Cos, It was there in patch available state quite a long time.Please commit it.

> [Herriot] Shell command for getting the new exceptions in the logs returning exitcode 1 after executing successfully.
> ---------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6819
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6819
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6819-ydist-security.patch, HADOOP-6819.patch
>
>
> I have found an corner case issue while executing the shell command for getting the new exceptions in the logs. The problem could see whenever the log files are empty and trying to get the new exceptions count, it returns the output is 0 ,however it returns the exitcode is 1 instead of 0. grep -vc in the command is not  giving the expected exitcode because its not handling the multiple in options.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9426-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 20:09:17 2010
Return-Path: <common-issues-return-9426-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 533 invoked from network); 9 Jul 2010 20:09:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 20:09:17 -0000
Received: (qmail 53480 invoked by uid 500); 9 Jul 2010 20:09:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53439 invoked by uid 500); 9 Jul 2010 20:09:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53431 invoked by uid 99); 9 Jul 2010 20:09:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 20:09:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 20:09:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o69K8qnj008053
	for <common-issues@hadoop.apache.org>; Fri, 9 Jul 2010 20:08:52 GMT
Message-ID: <21683644.292271278706132580.JavaMail.jira@thor>
Date: Fri, 9 Jul 2010 16:08:52 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6845) TokenStorage renamed to Credentials.
In-Reply-To: <11027372.98871277758850634.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6845:
--------------------------------

           Status: Resolved  (was: Patch Available)
    Fix Version/s: 0.22.0
       Resolution: Fixed

I just committed this. Thanks, Jitendra!

> TokenStorage renamed to Credentials.
> ------------------------------------
>
>                 Key: HADOOP-6845
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6845
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6845.1.patch, HADOOP-6845.2.patch, HADOOP-6845.3.patch
>
>
> This jira tracks common changes for MAPREDUCE-1528.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9427-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 20:24:15 2010
Return-Path: <common-issues-return-9427-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2600 invoked from network); 9 Jul 2010 20:24:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 20:24:15 -0000
Received: (qmail 68971 invoked by uid 500); 9 Jul 2010 20:24:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68932 invoked by uid 500); 9 Jul 2010 20:24:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68924 invoked by uid 99); 9 Jul 2010 20:24:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 20:24:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 20:24:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o69KNoB6008184
	for <common-issues@hadoop.apache.org>; Fri, 9 Jul 2010 20:23:50 GMT
Message-ID: <13688969.292591278707030044.JavaMail.jira@thor>
Date: Fri, 9 Jul 2010 16:23:50 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6845) TokenStorage renamed to
 Credentials.
In-Reply-To: <11027372.98871277758850634.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886863#action_12886863 ] 

Hudson commented on HADOOP-6845:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #323 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/323/])
    HADOOP-6845. Renames the TokenStorage class to Credentials. Contributed by Jitendra Pandey.


> TokenStorage renamed to Credentials.
> ------------------------------------
>
>                 Key: HADOOP-6845
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6845
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6845.1.patch, HADOOP-6845.2.patch, HADOOP-6845.3.patch
>
>
> This jira tracks common changes for MAPREDUCE-1528.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9428-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 21:13:16 2010
Return-Path: <common-issues-return-9428-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13389 invoked from network); 9 Jul 2010 21:13:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 21:13:15 -0000
Received: (qmail 4445 invoked by uid 500); 9 Jul 2010 21:13:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3679 invoked by uid 500); 9 Jul 2010 21:13:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3635 invoked by uid 99); 9 Jul 2010 21:13:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 21:13:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 21:13:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o69LCo8m008698
	for <common-issues@hadoop.apache.org>; Fri, 9 Jul 2010 21:12:50 GMT
Message-ID: <283378.293791278709970688.JavaMail.jira@thor>
Date: Fri, 9 Jul 2010 17:12:50 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6845) TokenStorage renamed to
 Credentials.
In-Reply-To: <11027372.98871277758850634.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886884#action_12886884 ] 

Hudson commented on HADOOP-6845:
--------------------------------

Integrated in Hadoop-Hdfs-trunk-Commit #337 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/337/])
    HDFS-1272. Fixes to take care of the changes in HADOOP-6845. Contributed by Jitendra Pandey.


> TokenStorage renamed to Credentials.
> ------------------------------------
>
>                 Key: HADOOP-6845
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6845
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6845.1.patch, HADOOP-6845.2.patch, HADOOP-6845.3.patch
>
>
> This jira tracks common changes for MAPREDUCE-1528.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9429-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 21:42:15 2010
Return-Path: <common-issues-return-9429-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18844 invoked from network); 9 Jul 2010 21:42:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 21:42:15 -0000
Received: (qmail 39351 invoked by uid 500); 9 Jul 2010 21:42:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39300 invoked by uid 500); 9 Jul 2010 21:42:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39292 invoked by uid 99); 9 Jul 2010 21:42:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 21:42:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 21:42:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o69LforW009200
	for <common-issues@hadoop.apache.org>; Fri, 9 Jul 2010 21:41:50 GMT
Message-ID: <1257176.295251278711710798.JavaMail.jira@thor>
Date: Fri, 9 Jul 2010 17:41:50 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6836) [Herriot] Test Autoamtion : Generic
 method for adding/modifying the attributes for new configuration.
In-Reply-To: <23644117.9641277269730469.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6836:
---------------------------------------

               Status: Patch Available  (was: Open)
         Hadoop Flags: [Reviewed]
    Affects Version/s: 0.21.0

Vinay, usual verification means running a patch through {{test-patch}} process. One can do it by clicking on 'Submit patch' link in a JIRA.

> [Herriot] Test Autoamtion : Generic method for adding/modifying the attributes for new configuration.
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6836
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6836
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Iyappan Srinivasan
>            Assignee: Vinay Kumar Thota
>         Attachments: 6836-ydist-security.patch, HADOOP-6836.patch
>
>
> HADOP-6772 deals with 
> Common utilities for system tests.
> 1. A method for restarting the daemon with new configuration.
> c throws Exception; 
> 2. A method for restarting the daemon with default configuration.
> public void restart() throws Exception;
> 3. A method for waiting until daemon is stop.
> public void waitForClusterToStop() throws Exception;
> In this  some variables are made of String, instead of Long. Those needs ot be changed too. So, can this method 
> "public static void restartClusterWithNewConfig(Hashtable<String,Long> props, String confFile) throws Exception;"
> be generalized to accepts string too. All otehr methods should work as usual.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9430-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 22:08:53 2010
Return-Path: <common-issues-return-9430-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26580 invoked from network); 9 Jul 2010 22:08:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 22:08:53 -0000
Received: (qmail 70272 invoked by uid 500); 9 Jul 2010 22:08:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70220 invoked by uid 500); 9 Jul 2010 22:08:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70207 invoked by uid 99); 9 Jul 2010 22:08:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 22:08:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 22:08:50 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o69M0wCb009422
	for <common-issues@hadoop.apache.org>; Fri, 9 Jul 2010 22:00:59 GMT
Message-ID: <25030547.295741278712858879.JavaMail.jira@thor>
Date: Fri, 9 Jul 2010 18:00:58 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6648) Credentials should ignore null tokens
In-Reply-To: <626471334.379531269043887761.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6648:
--------------------------------

    Attachment: h-6648-trunk.patch

Patch for trunk.

> Credentials should ignore null tokens
> -------------------------------------
>
>                 Key: HADOOP-6648
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6648
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.22.0
>
>         Attachments: c-6648-y20.patch, h-6648-trunk.patch
>
>
> When hftp goes to a non-secure cluster, getDelegationToken returns null. Credentials.addToken needs to ignore null tokens to avoid a NPE in serialization.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9431-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 22:10:45 2010
Return-Path: <common-issues-return-9431-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26833 invoked from network); 9 Jul 2010 22:10:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 22:10:45 -0000
Received: (qmail 70926 invoked by uid 500); 9 Jul 2010 22:10:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70898 invoked by uid 500); 9 Jul 2010 22:10:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70890 invoked by uid 99); 9 Jul 2010 22:10:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 22:10:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 22:10:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o69M2oaD009433
	for <common-issues@hadoop.apache.org>; Fri, 9 Jul 2010 22:02:50 GMT
Message-ID: <30798191.295771278712970583.JavaMail.jira@thor>
Date: Fri, 9 Jul 2010 18:02:50 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6648) Credentials should ignore null tokens
In-Reply-To: <626471334.379531269043887761.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6648:
--------------------------------

      Status: Patch Available  (was: Open)
    Assignee: Devaraj Das  (was: Owen O'Malley)

> Credentials should ignore null tokens
> -------------------------------------
>
>                 Key: HADOOP-6648
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6648
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: c-6648-y20.patch, h-6648-trunk.patch
>
>
> When hftp goes to a non-secure cluster, getDelegationToken returns null. Credentials.addToken needs to ignore null tokens to avoid a NPE in serialization.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9432-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 22:15:44 2010
Return-Path: <common-issues-return-9432-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27457 invoked from network); 9 Jul 2010 22:15:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 22:15:44 -0000
Received: (qmail 72585 invoked by uid 500); 9 Jul 2010 22:15:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72558 invoked by uid 500); 9 Jul 2010 22:15:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72550 invoked by uid 99); 9 Jul 2010 22:15:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 22:15:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 22:15:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o69M7oup009468
	for <common-issues@hadoop.apache.org>; Fri, 9 Jul 2010 22:07:50 GMT
Message-ID: <24173444.295861278713270286.JavaMail.jira@thor>
Date: Fri, 9 Jul 2010 18:07:50 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6648) Credentials should ignore null tokens
In-Reply-To: <626471334.379531269043887761.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6648:
-----------------------------------

    Hadoop Flags: [Reviewed]

+1

> Credentials should ignore null tokens
> -------------------------------------
>
>                 Key: HADOOP-6648
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6648
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: c-6648-y20.patch, h-6648-trunk.patch
>
>
> When hftp goes to a non-secure cluster, getDelegationToken returns null. Credentials.addToken needs to ignore null tokens to avoid a NPE in serialization.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9433-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 22:34:48 2010
Return-Path: <common-issues-return-9433-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30216 invoked from network); 9 Jul 2010 22:34:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 22:34:48 -0000
Received: (qmail 84912 invoked by uid 500); 9 Jul 2010 22:34:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84853 invoked by uid 500); 9 Jul 2010 22:34:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84839 invoked by uid 99); 9 Jul 2010 22:34:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 22:34:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 22:34:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o69MQrFD009629
	for <common-issues@hadoop.apache.org>; Fri, 9 Jul 2010 22:26:53 GMT
Message-ID: <7370443.296261278714413470.JavaMail.jira@thor>
Date: Fri, 9 Jul 2010 18:26:53 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6648) Credentials should ignore null
 tokens
In-Reply-To: <626471334.379531269043887761.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886921#action_12886921 ] 

Hadoop QA commented on HADOOP-6648:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12449123/h-6648-trunk.patch
  against trunk revision 962677.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/84/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/84/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/84/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/84/console

This message is automatically generated.

> Credentials should ignore null tokens
> -------------------------------------
>
>                 Key: HADOOP-6648
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6648
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: c-6648-y20.patch, h-6648-trunk.patch
>
>
> When hftp goes to a non-secure cluster, getDelegationToken returns null. Credentials.addToken needs to ignore null tokens to avoid a NPE in serialization.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9434-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 23:12:45 2010
Return-Path: <common-issues-return-9434-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40341 invoked from network); 9 Jul 2010 23:12:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 23:12:45 -0000
Received: (qmail 13049 invoked by uid 500); 9 Jul 2010 23:12:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12977 invoked by uid 500); 9 Jul 2010 23:12:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12969 invoked by uid 99); 9 Jul 2010 23:12:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 23:12:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 23:12:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o69N4oa9010063
	for <common-issues@hadoop.apache.org>; Fri, 9 Jul 2010 23:04:50 GMT
Message-ID: <32342353.297081278716690450.JavaMail.jira@thor>
Date: Fri, 9 Jul 2010 19:04:50 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6855) Add ability to get groups for ACLs
 from 'getent netgroup'
In-Reply-To: <23521712.275021278635570824.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Erik Steffl updated HADOOP-6855:
--------------------------------

    Attachment: HADOOP-6855-0.20.patch

> Add ability to get groups for ACLs from 'getent netgroup'
> ---------------------------------------------------------
>
>                 Key: HADOOP-6855
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6855
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6855-0.20.patch
>
>
> Add ability to specify netgroups in ACLs (see class AccessControlList.java). Membership of users in netgroups will be determined by running 'getent negroups $groupName'. Netgroups will be differentiated from regular unix groups by having '@' as a first character.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9435-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 09 23:57:19 2010
Return-Path: <common-issues-return-9435-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46572 invoked from network); 9 Jul 2010 23:57:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jul 2010 23:57:19 -0000
Received: (qmail 35319 invoked by uid 500); 9 Jul 2010 23:57:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35288 invoked by uid 500); 9 Jul 2010 23:57:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35280 invoked by uid 99); 9 Jul 2010 23:57:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 23:57:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Jul 2010 23:57:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o69NusDj010601
	for <common-issues@hadoop.apache.org>; Fri, 9 Jul 2010 23:56:54 GMT
Message-ID: <26180294.298531278719814075.JavaMail.jira@thor>
Date: Fri, 9 Jul 2010 19:56:54 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6856) SequenceFile and MapFile need cleanup
 to remove redundant constructors
In-Reply-To: <19355698.285171278691549769.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6856:
----------------------------------

    Attachment: h-6856.patch

This patch adds the new constructors and makes all of the old (now deprecated) constructors call the new one.

> SequenceFile and MapFile need cleanup to remove redundant constructors
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6856
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6856
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.22.0
>
>         Attachments: h-6856.patch
>
>
> Currently there are 2 public SequenceFile.Reader constructors, 3 public SequenceFile.Writer constructors, 9 public SequenceFile.createWriter, 2 public MapFile.Reader constructors, and 8 public MapFile.Writer constructors. All of with various historical combinations of parameters that don't cover the entire space.
> All of this makes it *very* difficult to add new optional parameters to SequenceFile and MapFile. 
> I'd like change to the style of FileContext.create with option parameters. I'll implement one public SequenceFile.Reader constructor and one public SequenceFile.createWriter and implement all of the current variants based on those two. I'll do the same for MapFile.Reader and MapFile.Writer including passing parameters down to the underlying SequenceFile.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9436-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 10 00:11:45 2010
Return-Path: <common-issues-return-9436-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52319 invoked from network); 10 Jul 2010 00:11:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jul 2010 00:11:45 -0000
Received: (qmail 39403 invoked by uid 500); 10 Jul 2010 00:11:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39339 invoked by uid 500); 10 Jul 2010 00:11:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39331 invoked by uid 99); 10 Jul 2010 00:11:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Jul 2010 00:11:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Jul 2010 00:11:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6A03o2F010709
	for <common-issues@hadoop.apache.org>; Sat, 10 Jul 2010 00:03:50 GMT
Message-ID: <535355.298751278720230053.JavaMail.jira@thor>
Date: Fri, 9 Jul 2010 20:03:50 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6647) balancer fails with "is not
 authorized for protocol interface NamenodeProtocol" in secure environment
In-Reply-To: <387797205.373021269030508879.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886952#action_12886952 ] 

Owen O'Malley commented on HADOOP-6647:
---------------------------------------

Allen,

The Namenode's configuration defines the mapping from long names to short names. It defaults to:

*@YOUR.DOMAIN -> *

With that mapping, someone coming in from another domain will fail, even with the cross-realm stuff set up.

hdfs@BAD.DOMAIN fails....

At Yahoo, we have two domains and we have rules for exactly how they map, but they amount to:

*@YGRID.YAHOO.COM -> *
*@CORP.YAHOO.COM -> *

So those two realms work, but anything else will fail. Depending on the translation that operations defines, they *can* make a cluster insecure. 

joe@CORP.YAHOO.COM -> root

would be really convenient for joe, but not secure. *grin*


> balancer fails with "is not authorized for protocol interface NamenodeProtocol" in secure environment
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6647
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6647
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6647-BP20.patch, HADOOP-6647.patch
>
>
> user logs in as hdfs/something@something and tries to run balancer.
> balancer is using NameNode Protocol which authorizes based on server principal key.
> but NameNode key is hdfs/_HOST@.. now. so it fails. 
> To fix we need to compare the short names only.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9437-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 10 00:13:45 2010
Return-Path: <common-issues-return-9437-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52432 invoked from network); 10 Jul 2010 00:13:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jul 2010 00:13:45 -0000
Received: (qmail 42052 invoked by uid 500); 10 Jul 2010 00:13:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41921 invoked by uid 500); 10 Jul 2010 00:13:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41913 invoked by uid 99); 10 Jul 2010 00:13:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Jul 2010 00:13:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Jul 2010 00:13:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6A05ooM010724
	for <common-issues@hadoop.apache.org>; Sat, 10 Jul 2010 00:05:50 GMT
Message-ID: <33164813.298781278720350425.JavaMail.jira@thor>
Date: Fri, 9 Jul 2010 20:05:50 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6856) SequenceFile and MapFile need cleanup
 to remove redundant constructors
In-Reply-To: <19355698.285171278691549769.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6856:
----------------------------------

    Status: Patch Available  (was: Open)

> SequenceFile and MapFile need cleanup to remove redundant constructors
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6856
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6856
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.22.0
>
>         Attachments: h-6856.patch
>
>
> Currently there are 2 public SequenceFile.Reader constructors, 3 public SequenceFile.Writer constructors, 9 public SequenceFile.createWriter, 2 public MapFile.Reader constructors, and 8 public MapFile.Writer constructors. All of with various historical combinations of parameters that don't cover the entire space.
> All of this makes it *very* difficult to add new optional parameters to SequenceFile and MapFile. 
> I'd like change to the style of FileContext.create with option parameters. I'll implement one public SequenceFile.Reader constructor and one public SequenceFile.createWriter and implement all of the current variants based on those two. I'll do the same for MapFile.Reader and MapFile.Writer including passing parameters down to the underlying SequenceFile.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9438-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 10 00:44:45 2010
Return-Path: <common-issues-return-9438-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61903 invoked from network); 10 Jul 2010 00:44:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jul 2010 00:44:44 -0000
Received: (qmail 55043 invoked by uid 500); 10 Jul 2010 00:44:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54992 invoked by uid 500); 10 Jul 2010 00:44:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54984 invoked by uid 99); 10 Jul 2010 00:44:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Jul 2010 00:44:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Jul 2010 00:44:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6A0aoNA010928
	for <common-issues@hadoop.apache.org>; Sat, 10 Jul 2010 00:36:50 GMT
Message-ID: <13977054.299271278722210015.JavaMail.jira@thor>
Date: Fri, 9 Jul 2010 20:36:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6856) SequenceFile and MapFile need
 cleanup to remove redundant constructors
In-Reply-To: <19355698.285171278691549769.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886959#action_12886959 ] 

Hadoop QA commented on HADOOP-6856:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12449138/h-6856.patch
  against trunk revision 962677.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    -1 javac.  The applied patch generated 1065 javac compiler warnings (more than the trunk's current 1017 warnings).

    -1 findbugs.  The patch appears to introduce 1 new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/85/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/85/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/85/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/85/console

This message is automatically generated.

> SequenceFile and MapFile need cleanup to remove redundant constructors
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6856
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6856
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.22.0
>
>         Attachments: h-6856.patch
>
>
> Currently there are 2 public SequenceFile.Reader constructors, 3 public SequenceFile.Writer constructors, 9 public SequenceFile.createWriter, 2 public MapFile.Reader constructors, and 8 public MapFile.Writer constructors. All of with various historical combinations of parameters that don't cover the entire space.
> All of this makes it *very* difficult to add new optional parameters to SequenceFile and MapFile. 
> I'd like change to the style of FileContext.create with option parameters. I'll implement one public SequenceFile.Reader constructor and one public SequenceFile.createWriter and implement all of the current variants based on those two. I'll do the same for MapFile.Reader and MapFile.Writer including passing parameters down to the underlying SequenceFile.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9439-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 10 01:02:46 2010
Return-Path: <common-issues-return-9439-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63430 invoked from network); 10 Jul 2010 01:02:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jul 2010 01:02:46 -0000
Received: (qmail 57754 invoked by uid 500); 10 Jul 2010 01:02:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57708 invoked by uid 500); 10 Jul 2010 01:02:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57700 invoked by uid 99); 10 Jul 2010 01:02:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Jul 2010 01:02:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Jul 2010 01:02:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6A0spNj011038
	for <common-issues@hadoop.apache.org>; Sat, 10 Jul 2010 00:54:51 GMT
Message-ID: <15638960.299531278723291304.JavaMail.jira@thor>
Date: Fri, 9 Jul 2010 20:54:51 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6825) FileStatus needs unit tests
In-Reply-To: <28758544.71131276463653554.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886961#action_12886961 ] 

Rodrigo Schmidt commented on HADOOP-6825:
-----------------------------------------

Any further comments? Can this be committed?

> FileStatus needs unit tests
> ---------------------------
>
>                 Key: HADOOP-6825
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6825
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6825.0.patch
>
>
> We need some unit tests for FileStatus to prevent problems like those we recently had on HADOOP-6796 and MAPREDUCE-1858.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9440-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 10 06:58:19 2010
Return-Path: <common-issues-return-9440-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53897 invoked from network); 10 Jul 2010 06:58:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jul 2010 06:58:19 -0000
Received: (qmail 85849 invoked by uid 500); 10 Jul 2010 06:58:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85719 invoked by uid 500); 10 Jul 2010 06:58:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85691 invoked by uid 99); 10 Jul 2010 06:58:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Jul 2010 06:58:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Jul 2010 06:58:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6A6vpbZ013310
	for <common-issues@hadoop.apache.org>; Sat, 10 Jul 2010 06:57:51 GMT
Message-ID: <2645027.302651278745071116.JavaMail.jira@thor>
Date: Sat, 10 Jul 2010 02:57:51 -0400 (EDT)
From: "Lance Norskog (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6209) chmod fails on Windows+cygwin
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12887004#action_12887004 ] 

Lance Norskog commented on HADOOP-6209:
---------------------------------------

Please revive this effort.

Thanks!

> chmod fails on Windows+cygwin
> -----------------------------
>
>                 Key: HADOOP-6209
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6209
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Kohsuke Kawaguchi
>            Assignee: Kohsuke Kawaguchi
>         Attachments: HDFS-569.diff
>
>
> When run on Windows + cygwin, chmod fails because it fails to call {{makeShellPath}} on the parameter.
> The strack trace with 0.19.1 below:
> {noformat}
> org.apache.hadoop.util.Shell$ExitCodeException: chmod: changing permissions of `C:\\files\\hudson\\hadoop\\task-tracker\\taskTracker\\jobcache\\job_200908191713_0008\\job.xml': No such file or directory
> 	at org.apache.hadoop.util.Shell.runCommand(Shell.java:195)
> 	at org.apache.hadoop.util.Shell.run(Shell.java:134)
> 	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:286)
> 	at org.apache.hadoop.util.Shell.execCommand(Shell.java:338)
> 	at org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:540)
> 	at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:532)
> 	at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:274)
> 	at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:364)
> 	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:487)
> 	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:468)
> 	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:375)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:208)
> 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)
> 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1214)
> 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1195)
> 	at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:750)
> 	at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1656)
> 	at org.apache.hadoop.mapred.TaskTracker.access$1200(TaskTracker.java:102)
> 	at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:1621)
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9441-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 10 07:21:39 2010
Return-Path: <common-issues-return-9441-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60199 invoked from network); 10 Jul 2010 07:21:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jul 2010 07:21:39 -0000
Received: (qmail 96426 invoked by uid 500); 10 Jul 2010 07:21:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96294 invoked by uid 500); 10 Jul 2010 07:21:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96274 invoked by uid 99); 10 Jul 2010 07:21:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Jul 2010 07:21:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Jul 2010 07:21:33 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6A7LAgh013515
	for <common-issues@hadoop.apache.org>; Sat, 10 Jul 2010 07:21:11 GMT
Message-ID: <30813312.302821278746470241.JavaMail.jira@thor>
Date: Sat, 10 Jul 2010 03:21:10 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6856) SequenceFile and MapFile need cleanup
 to remove redundant constructors
In-Reply-To: <19355698.285171278691549769.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6856:
----------------------------------

    Attachment: h-6856.patch

Fixed the javac and javadoc warnings. 

> SequenceFile and MapFile need cleanup to remove redundant constructors
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6856
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6856
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.22.0
>
>         Attachments: h-6856.patch, h-6856.patch
>
>
> Currently there are 2 public SequenceFile.Reader constructors, 3 public SequenceFile.Writer constructors, 9 public SequenceFile.createWriter, 2 public MapFile.Reader constructors, and 8 public MapFile.Writer constructors. All of with various historical combinations of parameters that don't cover the entire space.
> All of this makes it *very* difficult to add new optional parameters to SequenceFile and MapFile. 
> I'd like change to the style of FileContext.create with option parameters. I'll implement one public SequenceFile.Reader constructor and one public SequenceFile.createWriter and implement all of the current variants based on those two. I'll do the same for MapFile.Reader and MapFile.Writer including passing parameters down to the underlying SequenceFile.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9442-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 10 07:21:41 2010
Return-Path: <common-issues-return-9442-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60263 invoked from network); 10 Jul 2010 07:21:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jul 2010 07:21:40 -0000
Received: (qmail 96476 invoked by uid 500); 10 Jul 2010 07:21:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96355 invoked by uid 500); 10 Jul 2010 07:21:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96282 invoked by uid 99); 10 Jul 2010 07:21:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Jul 2010 07:21:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Jul 2010 07:21:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6A7LCZQ013524
	for <common-issues@hadoop.apache.org>; Sat, 10 Jul 2010 07:21:12 GMT
Message-ID: <2561100.302851278746472647.JavaMail.jira@thor>
Date: Sat, 10 Jul 2010 03:21:12 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6856) SequenceFile and MapFile need cleanup
 to remove redundant constructors
In-Reply-To: <19355698.285171278691549769.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6856:
----------------------------------

    Status: Open  (was: Patch Available)

> SequenceFile and MapFile need cleanup to remove redundant constructors
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6856
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6856
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.22.0
>
>         Attachments: h-6856.patch, h-6856.patch
>
>
> Currently there are 2 public SequenceFile.Reader constructors, 3 public SequenceFile.Writer constructors, 9 public SequenceFile.createWriter, 2 public MapFile.Reader constructors, and 8 public MapFile.Writer constructors. All of with various historical combinations of parameters that don't cover the entire space.
> All of this makes it *very* difficult to add new optional parameters to SequenceFile and MapFile. 
> I'd like change to the style of FileContext.create with option parameters. I'll implement one public SequenceFile.Reader constructor and one public SequenceFile.createWriter and implement all of the current variants based on those two. I'll do the same for MapFile.Reader and MapFile.Writer including passing parameters down to the underlying SequenceFile.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9443-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 10 07:21:41 2010
Return-Path: <common-issues-return-9443-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60264 invoked from network); 10 Jul 2010 07:21:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jul 2010 07:21:40 -0000
Received: (qmail 96624 invoked by uid 500); 10 Jul 2010 07:21:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96473 invoked by uid 500); 10 Jul 2010 07:21:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96295 invoked by uid 99); 10 Jul 2010 07:21:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Jul 2010 07:21:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Jul 2010 07:21:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6A7LD50013533
	for <common-issues@hadoop.apache.org>; Sat, 10 Jul 2010 07:21:13 GMT
Message-ID: <21751135.302881278746473154.JavaMail.jira@thor>
Date: Sat, 10 Jul 2010 03:21:13 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6856) SequenceFile and MapFile need cleanup
 to remove redundant constructors
In-Reply-To: <19355698.285171278691549769.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6856:
----------------------------------

    Status: Patch Available  (was: Open)

> SequenceFile and MapFile need cleanup to remove redundant constructors
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6856
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6856
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.22.0
>
>         Attachments: h-6856.patch, h-6856.patch
>
>
> Currently there are 2 public SequenceFile.Reader constructors, 3 public SequenceFile.Writer constructors, 9 public SequenceFile.createWriter, 2 public MapFile.Reader constructors, and 8 public MapFile.Writer constructors. All of with various historical combinations of parameters that don't cover the entire space.
> All of this makes it *very* difficult to add new optional parameters to SequenceFile and MapFile. 
> I'd like change to the style of FileContext.create with option parameters. I'll implement one public SequenceFile.Reader constructor and one public SequenceFile.createWriter and implement all of the current variants based on those two. I'll do the same for MapFile.Reader and MapFile.Writer including passing parameters down to the underlying SequenceFile.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9444-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 10 07:48:18 2010
Return-Path: <common-issues-return-9444-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62919 invoked from network); 10 Jul 2010 07:48:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jul 2010 07:48:18 -0000
Received: (qmail 1121 invoked by uid 500); 10 Jul 2010 07:48:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 984 invoked by uid 500); 10 Jul 2010 07:48:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 976 invoked by uid 99); 10 Jul 2010 07:48:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Jul 2010 07:48:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Jul 2010 07:48:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6A7lo6M013649
	for <common-issues@hadoop.apache.org>; Sat, 10 Jul 2010 07:47:50 GMT
Message-ID: <5120470.302961278748070405.JavaMail.jira@thor>
Date: Sat, 10 Jul 2010 03:47:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6856) SequenceFile and MapFile need
 cleanup to remove redundant constructors
In-Reply-To: <19355698.285171278691549769.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12887010#action_12887010 ] 

Hadoop QA commented on HADOOP-6856:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12449147/h-6856.patch
  against trunk revision 962677.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 2 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    -1 javac.  The applied patch generated 1035 javac compiler warnings (more than the trunk's current 1017 warnings).

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/86/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/86/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/86/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/86/console

This message is automatically generated.

> SequenceFile and MapFile need cleanup to remove redundant constructors
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6856
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6856
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.22.0
>
>         Attachments: h-6856.patch, h-6856.patch
>
>
> Currently there are 2 public SequenceFile.Reader constructors, 3 public SequenceFile.Writer constructors, 9 public SequenceFile.createWriter, 2 public MapFile.Reader constructors, and 8 public MapFile.Writer constructors. All of with various historical combinations of parameters that don't cover the entire space.
> All of this makes it *very* difficult to add new optional parameters to SequenceFile and MapFile. 
> I'd like change to the style of FileContext.create with option parameters. I'll implement one public SequenceFile.Reader constructor and one public SequenceFile.createWriter and implement all of the current variants based on those two. I'll do the same for MapFile.Reader and MapFile.Writer including passing parameters down to the underlying SequenceFile.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9445-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 10 11:22:49 2010
Return-Path: <common-issues-return-9445-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5176 invoked from network); 10 Jul 2010 11:22:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jul 2010 11:22:49 -0000
Received: (qmail 82741 invoked by uid 500); 10 Jul 2010 11:22:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82610 invoked by uid 500); 10 Jul 2010 11:22:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82602 invoked by uid 99); 10 Jul 2010 11:22:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Jul 2010 11:22:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Jul 2010 11:22:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6ABEopF014591
	for <common-issues@hadoop.apache.org>; Sat, 10 Jul 2010 11:14:50 GMT
Message-ID: <25088328.303631278760490092.JavaMail.jira@thor>
Date: Sat, 10 Jul 2010 07:14:50 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6845) TokenStorage renamed to
 Credentials.
In-Reply-To: <11027372.98871277758850634.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12887019#action_12887019 ] 

Hudson commented on HADOOP-6845:
--------------------------------

Integrated in Hadoop-Common-trunk #391 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/391/])
    HADOOP-6845. Renames the TokenStorage class to Credentials. Contributed by Jitendra Pandey.


> TokenStorage renamed to Credentials.
> ------------------------------------
>
>                 Key: HADOOP-6845
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6845
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6845.1.patch, HADOOP-6845.2.patch, HADOOP-6845.3.patch
>
>
> This jira tracks common changes for MAPREDUCE-1528.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9446-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 10 20:59:15 2010
Return-Path: <common-issues-return-9446-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28453 invoked from network); 10 Jul 2010 20:59:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jul 2010 20:59:15 -0000
Received: (qmail 36966 invoked by uid 500); 10 Jul 2010 20:59:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36935 invoked by uid 500); 10 Jul 2010 20:59:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36927 invoked by uid 99); 10 Jul 2010 20:59:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Jul 2010 20:59:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Jul 2010 20:59:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6AKwobV017321
	for <common-issues@hadoop.apache.org>; Sat, 10 Jul 2010 20:58:50 GMT
Message-ID: <20533737.307601278795530108.JavaMail.jira@thor>
Date: Sat, 10 Jul 2010 16:58:50 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6819) [Herriot] Shell command for getting
 the new exceptions in the logs returning exitcode 1 after executing
 successfully.
In-Reply-To: <25211660.39321276233134711.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6819:
---------------------------------------

           Status: Resolved  (was: Patch Available)
    Fix Version/s: 0.21.0
       Resolution: Fixed

I have committed this to trunk and 0.21 branch. Thanks Vinay!

> [Herriot] Shell command for getting the new exceptions in the logs returning exitcode 1 after executing successfully.
> ---------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6819
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6819
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>             Fix For: 0.21.0
>
>         Attachments: 6819-ydist-security.patch, HADOOP-6819.patch
>
>
> I have found an corner case issue while executing the shell command for getting the new exceptions in the logs. The problem could see whenever the log files are empty and trying to get the new exceptions count, it returns the output is 0 ,however it returns the exitcode is 1 instead of 0. grep -vc in the command is not  giving the expected exitcode because its not handling the multiple in options.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9447-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 10 22:09:15 2010
Return-Path: <common-issues-return-9447-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52028 invoked from network); 10 Jul 2010 22:09:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jul 2010 22:09:15 -0000
Received: (qmail 67809 invoked by uid 500); 10 Jul 2010 22:09:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67756 invoked by uid 500); 10 Jul 2010 22:09:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67748 invoked by uid 99); 10 Jul 2010 22:09:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Jul 2010 22:09:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Jul 2010 22:09:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6AM8oo2017667
	for <common-issues@hadoop.apache.org>; Sat, 10 Jul 2010 22:08:50 GMT
Message-ID: <9001879.307931278799730449.JavaMail.jira@thor>
Date: Sat, 10 Jul 2010 18:08:50 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6670) UserGroupInformation doesn't support
 use in hash tables
In-Reply-To: <242202762.620161270078590059.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6670:
------------------------------

    Attachment: c6670-02.patch

Attaching a straight-forward port for trunk. Changes made to TestFileSystem in the original patch is now applied to TestFileSystemCaching. Changes made to FileSystem.java are ignored since only debug messages were added. Changes made to TestTrackerDistributedCacheManager are also ignored for the same reason. Devaraj's bug fix for Gridmix is not ported at this time since related Gridmix files are not forward ported yet. It's better to incorporate the bug fix when porting Gridmix.

> UserGroupInformation doesn't support use in hash tables
> -------------------------------------------------------
>
>                 Key: HADOOP-6670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6670
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 6670-gridmix-test.bugfix.patch, c6670-02.patch, fs-close.patch, fs-close.patch, fs-close.patch
>
>
> The UserGroupInformation objects are mutable, but they are used as keys in hash tables. This leads to serious problems in the FileSystem cache and RPC connection cache. We need to change the hashCode to be the identity hash code of the Subject and change equals to use == between the Subjects.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9448-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 10 22:19:44 2010
Return-Path: <common-issues-return-9448-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53118 invoked from network); 10 Jul 2010 22:19:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jul 2010 22:19:44 -0000
Received: (qmail 75007 invoked by uid 500); 10 Jul 2010 22:19:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74958 invoked by uid 500); 10 Jul 2010 22:19:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74950 invoked by uid 99); 10 Jul 2010 22:19:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Jul 2010 22:19:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Jul 2010 22:19:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6AMBoeG017691
	for <common-issues@hadoop.apache.org>; Sat, 10 Jul 2010 22:11:50 GMT
Message-ID: <17818427.307981278799910199.JavaMail.jira@thor>
Date: Sat, 10 Jul 2010 18:11:50 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6670) UserGroupInformation doesn't support
 use in hash tables
In-Reply-To: <242202762.620161270078590059.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6670:
------------------------------

    Status: Patch Available  (was: Open)

> UserGroupInformation doesn't support use in hash tables
> -------------------------------------------------------
>
>                 Key: HADOOP-6670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6670
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 6670-gridmix-test.bugfix.patch, c6670-02.patch, fs-close.patch, fs-close.patch, fs-close.patch
>
>
> The UserGroupInformation objects are mutable, but they are used as keys in hash tables. This leads to serious problems in the FileSystem cache and RPC connection cache. We need to change the hashCode to be the identity hash code of the Subject and change equals to use == between the Subjects.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9449-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jul 11 06:39:18 2010
Return-Path: <common-issues-return-9449-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80658 invoked from network); 11 Jul 2010 06:39:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jul 2010 06:39:18 -0000
Received: (qmail 79073 invoked by uid 500); 11 Jul 2010 06:39:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78949 invoked by uid 500); 11 Jul 2010 06:39:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78941 invoked by uid 99); 11 Jul 2010 06:39:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 11 Jul 2010 06:39:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 11 Jul 2010 06:39:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6B6coGI020915
	for <common-issues@hadoop.apache.org>; Sun, 11 Jul 2010 06:38:50 GMT
Message-ID: <28044025.312031278830330641.JavaMail.jira@thor>
Date: Sun, 11 Jul 2010 02:38:50 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6647) balancer fails with "is not
 authorized for protocol interface NamenodeProtocol" in secure environment
In-Reply-To: <387797205.373021269030508879.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12887143#action_12887143 ] 

Devaraj Das commented on HADOOP-6647:
-------------------------------------

+1

> balancer fails with "is not authorized for protocol interface NamenodeProtocol" in secure environment
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6647
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6647
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6647-BP20.patch, HADOOP-6647.patch
>
>
> user logs in as hdfs/something@something and tries to run balancer.
> balancer is using NameNode Protocol which authorizes based on server principal key.
> but NameNode key is hdfs/_HOST@.. now. so it fails. 
> To fix we need to compare the short names only.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9450-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jul 11 06:45:17 2010
Return-Path: <common-issues-return-9450-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81137 invoked from network); 11 Jul 2010 06:45:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jul 2010 06:45:17 -0000
Received: (qmail 79747 invoked by uid 500); 11 Jul 2010 06:45:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79623 invoked by uid 500); 11 Jul 2010 06:45:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79615 invoked by uid 99); 11 Jul 2010 06:45:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 11 Jul 2010 06:45:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 11 Jul 2010 06:45:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6B6ioWG020939
	for <common-issues@hadoop.apache.org>; Sun, 11 Jul 2010 06:44:50 GMT
Message-ID: <22269869.312061278830690371.JavaMail.jira@thor>
Date: Sun, 11 Jul 2010 02:44:50 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6648) Credentials should ignore null tokens
In-Reply-To: <626471334.379531269043887761.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6648:
--------------------------------

        Status: Resolved  (was: Patch Available)
    Resolution: Fixed

I just committed this (the javadoc warning is not due to this patch).

> Credentials should ignore null tokens
> -------------------------------------
>
>                 Key: HADOOP-6648
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6648
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: c-6648-y20.patch, h-6648-trunk.patch
>
>
> When hftp goes to a non-secure cluster, getDelegationToken returns null. Credentials.addToken needs to ignore null tokens to avoid a NPE in serialization.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9451-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 12 06:03:20 2010
Return-Path: <common-issues-return-9451-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6936 invoked from network); 12 Jul 2010 06:03:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jul 2010 06:03:19 -0000
Received: (qmail 34698 invoked by uid 500); 12 Jul 2010 06:03:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34483 invoked by uid 500); 12 Jul 2010 06:03:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34468 invoked by uid 99); 12 Jul 2010 06:03:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 06:03:14 +0000
X-ASF-Spam-Status: No, hits=-1996.4 required=10.0
	tests=ALL_TRUSTED,FS_REPLICA
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 06:03:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6C62nQZ002929
	for <common-issues@hadoop.apache.org>; Mon, 12 Jul 2010 06:02:50 GMT
Message-ID: <1268643.319901278914569769.JavaMail.jira@thor>
Date: Mon, 12 Jul 2010 02:02:49 -0400 (EDT)
From: "Alex Kozlov (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6857) FsShell should report raw disk usage
 including replication factor
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

FsShell should report raw disk usage including replication factor
-----------------------------------------------------------------

                 Key: HADOOP-6857
                 URL: https://issues.apache.org/jira/browse/HADOOP-6857
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs
    Affects Versions: 0.20.2
            Reporter: Alex Kozlov


Currently FsShell report HDFS usage with "hadoop fs -dus <path>" command.  Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe "hadoop fs -dus -raw <path>"?).  This will allow to assess resource usage more accurately.  -- Alex K


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9452-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 12 17:25:48 2010
Return-Path: <common-issues-return-9452-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51632 invoked from network); 12 Jul 2010 17:25:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jul 2010 17:25:48 -0000
Received: (qmail 42035 invoked by uid 500); 12 Jul 2010 17:25:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41993 invoked by uid 500); 12 Jul 2010 17:25:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41977 invoked by uid 99); 12 Jul 2010 17:25:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 17:25:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 17:25:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6CHHsgf008704
	for <common-issues@hadoop.apache.org>; Mon, 12 Jul 2010 17:17:54 GMT
Message-ID: <3277355.328521278955074330.JavaMail.jira@thor>
Date: Mon, 12 Jul 2010 13:17:54 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6670) UserGroupInformation doesn't support
 use in hash tables
In-Reply-To: <242202762.620161270078590059.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6670:
------------------------------

    Status: Patch Available  (was: Open)

> UserGroupInformation doesn't support use in hash tables
> -------------------------------------------------------
>
>                 Key: HADOOP-6670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6670
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 6670-gridmix-test.bugfix.patch, c6670-02.patch, fs-close.patch, fs-close.patch, fs-close.patch
>
>
> The UserGroupInformation objects are mutable, but they are used as keys in hash tables. This leads to serious problems in the FileSystem cache and RPC connection cache. We need to change the hashCode to be the identity hash code of the Subject and change equals to use == between the Subjects.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9453-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 12 17:25:49 2010
Return-Path: <common-issues-return-9453-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51676 invoked from network); 12 Jul 2010 17:25:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jul 2010 17:25:48 -0000
Received: (qmail 42180 invoked by uid 500); 12 Jul 2010 17:25:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41995 invoked by uid 500); 12 Jul 2010 17:25:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41980 invoked by uid 99); 12 Jul 2010 17:25:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 17:25:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 17:25:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6CHHra9008689
	for <common-issues@hadoop.apache.org>; Mon, 12 Jul 2010 17:17:53 GMT
Message-ID: <384114.328471278955073688.JavaMail.jira@thor>
Date: Mon, 12 Jul 2010 13:17:53 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6670) UserGroupInformation doesn't support
 use in hash tables
In-Reply-To: <242202762.620161270078590059.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6670:
------------------------------

    Status: Open  (was: Patch Available)

> UserGroupInformation doesn't support use in hash tables
> -------------------------------------------------------
>
>                 Key: HADOOP-6670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6670
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 6670-gridmix-test.bugfix.patch, c6670-02.patch, fs-close.patch, fs-close.patch, fs-close.patch
>
>
> The UserGroupInformation objects are mutable, but they are used as keys in hash tables. This leads to serious problems in the FileSystem cache and RPC connection cache. We need to change the hashCode to be the identity hash code of the Subject and change equals to use == between the Subjects.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9454-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 12 18:09:16 2010
Return-Path: <common-issues-return-9454-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68845 invoked from network); 12 Jul 2010 18:09:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jul 2010 18:09:15 -0000
Received: (qmail 95895 invoked by uid 500); 12 Jul 2010 18:09:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95816 invoked by uid 500); 12 Jul 2010 18:09:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95808 invoked by uid 99); 12 Jul 2010 18:09:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 18:09:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 18:09:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6CI8oWs009227
	for <common-issues@hadoop.apache.org>; Mon, 12 Jul 2010 18:08:50 GMT
Message-ID: <19247850.329571278958130662.JavaMail.jira@thor>
Date: Mon, 12 Jul 2010 14:08:50 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6839) [Herriot] Implement a functionality
 for getting the user list for creating proxy users.
In-Reply-To: <26284266.54901277454290297.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12887448#action_12887448 ] 

Konstantin Boudnik commented on HADOOP-6839:
--------------------------------------------

Looks good overall. Please fix the formatting (white space are inconsistent) and make exception and error messages more meaningful. Submit the patch for verification whenever is ready.

> [Herriot] Implement a functionality for getting the user list for creating proxy users.
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6839
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6839
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6839-ydist-security.patch, HADOOP-6839.patch
>
>
> Develop a new method for getting the user list. 
> Method signature is 
> public ArrayList<String> getHadoopMultiUsersList() throws IOException;
> Add new attribute in system-test.xml  file for getting userlist path.
> For submitting the jobs as different user a proxy user id is needed. So,get the available users from a userlist and then pass the user as proxy instead of hardcoding user id in a test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9455-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 12 18:41:17 2010
Return-Path: <common-issues-return-9455-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85218 invoked from network); 12 Jul 2010 18:41:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jul 2010 18:41:16 -0000
Received: (qmail 36836 invoked by uid 500); 12 Jul 2010 18:41:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36771 invoked by uid 500); 12 Jul 2010 18:41:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36763 invoked by uid 99); 12 Jul 2010 18:41:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 18:41:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 18:41:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6CIepnO009581
	for <common-issues@hadoop.apache.org>; Mon, 12 Jul 2010 18:40:51 GMT
Message-ID: <17133573.330451278960051728.JavaMail.jira@thor>
Date: Mon, 12 Jul 2010 14:40:51 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6839) [Herriot] Implement a functionality
 for getting the user list for creating proxy users.
In-Reply-To: <26284266.54901277454290297.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinay Kumar Thota updated HADOOP-6839:
--------------------------------------

    Attachment: 6839-ydist-security.patch
                HADOOP-6839.patch

Addressed the cos comments and attached new patches.

> [Herriot] Implement a functionality for getting the user list for creating proxy users.
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6839
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6839
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6839-ydist-security.patch, 6839-ydist-security.patch, HADOOP-6839.patch, HADOOP-6839.patch
>
>
> Develop a new method for getting the user list. 
> Method signature is 
> public ArrayList<String> getHadoopMultiUsersList() throws IOException;
> Add new attribute in system-test.xml  file for getting userlist path.
> For submitting the jobs as different user a proxy user id is needed. So,get the available users from a userlist and then pass the user as proxy instead of hardcoding user id in a test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9456-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 12 18:50:47 2010
Return-Path: <common-issues-return-9456-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88878 invoked from network); 12 Jul 2010 18:50:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jul 2010 18:50:47 -0000
Received: (qmail 46423 invoked by uid 500); 12 Jul 2010 18:50:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46381 invoked by uid 500); 12 Jul 2010 18:50:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46373 invoked by uid 99); 12 Jul 2010 18:50:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 18:50:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 18:50:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6CIgq9P009633
	for <common-issues@hadoop.apache.org>; Mon, 12 Jul 2010 18:42:52 GMT
Message-ID: <14291401.330591278960172535.JavaMail.jira@thor>
Date: Mon, 12 Jul 2010 14:42:52 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6839) [Herriot] Implement a functionality
 for getting the user list for creating proxy users.
In-Reply-To: <26284266.54901277454290297.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinay Kumar Thota updated HADOOP-6839:
--------------------------------------

               Status: Patch Available  (was: Open)
         Hadoop Flags: [Reviewed]
    Affects Version/s: 0.21.0

> [Herriot] Implement a functionality for getting the user list for creating proxy users.
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6839
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6839
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6839-ydist-security.patch, 6839-ydist-security.patch, HADOOP-6839.patch, HADOOP-6839.patch
>
>
> Develop a new method for getting the user list. 
> Method signature is 
> public ArrayList<String> getHadoopMultiUsersList() throws IOException;
> Add new attribute in system-test.xml  file for getting userlist path.
> For submitting the jobs as different user a proxy user id is needed. So,get the available users from a userlist and then pass the user as proxy instead of hardcoding user id in a test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9457-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 12 18:56:16 2010
Return-Path: <common-issues-return-9457-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90384 invoked from network); 12 Jul 2010 18:56:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jul 2010 18:56:16 -0000
Received: (qmail 51478 invoked by uid 500); 12 Jul 2010 18:56:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51433 invoked by uid 500); 12 Jul 2010 18:56:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51425 invoked by uid 99); 12 Jul 2010 18:56:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 18:56:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 18:56:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6CItpJI009752
	for <common-issues@hadoop.apache.org>; Mon, 12 Jul 2010 18:55:51 GMT
Message-ID: <5417362.330841278960951601.JavaMail.jira@thor>
Date: Mon, 12 Jul 2010 14:55:51 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6839) [Herriot] Implement a functionality
 for getting the user list for creating proxy users.
In-Reply-To: <26284266.54901277454290297.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12887467#action_12887467 ] 

Konstantin Boudnik commented on HADOOP-6839:
--------------------------------------------

- CONF_HADOOP_MULTI_USER_LIST has to be unquoted if you want to print the value of the constant.
+      LOG.error("Multi users list path not passed "
+          + "for CONF_HADOOP_MULTI_USER_LIST ");


> [Herriot] Implement a functionality for getting the user list for creating proxy users.
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6839
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6839
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6839-ydist-security.patch, 6839-ydist-security.patch, HADOOP-6839.patch, HADOOP-6839.patch
>
>
> Develop a new method for getting the user list. 
> Method signature is 
> public ArrayList<String> getHadoopMultiUsersList() throws IOException;
> Add new attribute in system-test.xml  file for getting userlist path.
> For submitting the jobs as different user a proxy user id is needed. So,get the available users from a userlist and then pass the user as proxy instead of hardcoding user id in a test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9458-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 12 20:55:48 2010
Return-Path: <common-issues-return-9458-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25715 invoked from network); 12 Jul 2010 20:55:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jul 2010 20:55:47 -0000
Received: (qmail 76393 invoked by uid 500); 12 Jul 2010 20:55:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76195 invoked by uid 500); 12 Jul 2010 20:55:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76183 invoked by uid 99); 12 Jul 2010 20:55:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 20:55:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 20:55:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6CKlqSb011079
	for <common-issues@hadoop.apache.org>; Mon, 12 Jul 2010 20:47:52 GMT
Message-ID: <6167885.333171278967672413.JavaMail.jira@thor>
Date: Mon, 12 Jul 2010 16:47:52 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6825) FileStatus needs unit tests
In-Reply-To: <28758544.71131276463653554.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12887514#action_12887514 ] 

Eli Collins commented on HADOOP-6825:
-------------------------------------

Thanks for the explanation.

+1   I'll commit this soon.


> FileStatus needs unit tests
> ---------------------------
>
>                 Key: HADOOP-6825
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6825
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6825.0.patch
>
>
> We need some unit tests for FileStatus to prevent problems like those we recently had on HADOOP-6796 and MAPREDUCE-1858.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9459-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 12 21:29:46 2010
Return-Path: <common-issues-return-9459-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41440 invoked from network); 12 Jul 2010 21:29:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jul 2010 21:29:46 -0000
Received: (qmail 31407 invoked by uid 500); 12 Jul 2010 21:29:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31365 invoked by uid 500); 12 Jul 2010 21:29:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31357 invoked by uid 99); 12 Jul 2010 21:29:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 21:29:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 21:29:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6CLLpaP011691
	for <common-issues@hadoop.apache.org>; Mon, 12 Jul 2010 21:21:51 GMT
Message-ID: <32267276.334651278969711650.JavaMail.jira@thor>
Date: Mon, 12 Jul 2010 17:21:51 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6647) balancer fails with "is not
 authorized for protocol interface NamenodeProtocol" in secure environment
In-Reply-To: <387797205.373021269030508879.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6647:
-----------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

committed to trunk.

javadoc warning is related to use of "Sun proprietary API and may be removed in a future release" packages introduces elsewhere.

> balancer fails with "is not authorized for protocol interface NamenodeProtocol" in secure environment
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6647
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6647
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6647-BP20.patch, HADOOP-6647.patch
>
>
> user logs in as hdfs/something@something and tries to run balancer.
> balancer is using NameNode Protocol which authorizes based on server principal key.
> but NameNode key is hdfs/_HOST@.. now. so it fails. 
> To fix we need to compare the short names only.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9460-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 12 21:35:16 2010
Return-Path: <common-issues-return-9460-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42935 invoked from network); 12 Jul 2010 21:35:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jul 2010 21:35:15 -0000
Received: (qmail 34876 invoked by uid 500); 12 Jul 2010 21:35:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34812 invoked by uid 500); 12 Jul 2010 21:35:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34800 invoked by uid 99); 12 Jul 2010 21:35:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 21:35:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 21:35:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6CLYodw011784
	for <common-issues@hadoop.apache.org>; Mon, 12 Jul 2010 21:34:50 GMT
Message-ID: <5542226.334851278970490280.JavaMail.jira@thor>
Date: Mon, 12 Jul 2010 17:34:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6836) [Herriot] Test Autoamtion : Generic
 method for adding/modifying the attributes for new configuration.
In-Reply-To: <23644117.9641277269730469.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12887547#action_12887547 ] 

Hadoop QA commented on HADOOP-6836:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12448033/HADOOP-6836.patch
  against trunk revision 962998.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/608/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/608/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/608/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/608/console

This message is automatically generated.

> [Herriot] Test Autoamtion : Generic method for adding/modifying the attributes for new configuration.
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6836
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6836
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Iyappan Srinivasan
>            Assignee: Vinay Kumar Thota
>         Attachments: 6836-ydist-security.patch, HADOOP-6836.patch
>
>
> HADOP-6772 deals with 
> Common utilities for system tests.
> 1. A method for restarting the daemon with new configuration.
> c throws Exception; 
> 2. A method for restarting the daemon with default configuration.
> public void restart() throws Exception;
> 3. A method for waiting until daemon is stop.
> public void waitForClusterToStop() throws Exception;
> In this  some variables are made of String, instead of Long. Those needs ot be changed too. So, can this method 
> "public static void restartClusterWithNewConfig(Hashtable<String,Long> props, String confFile) throws Exception;"
> be generalized to accepts string too. All otehr methods should work as usual.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9461-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 12 21:54:16 2010
Return-Path: <common-issues-return-9461-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49416 invoked from network); 12 Jul 2010 21:54:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jul 2010 21:54:16 -0000
Received: (qmail 62822 invoked by uid 500); 12 Jul 2010 21:54:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62776 invoked by uid 500); 12 Jul 2010 21:54:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62764 invoked by uid 99); 12 Jul 2010 21:54:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 21:54:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 21:54:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6CLroxj012019
	for <common-issues@hadoop.apache.org>; Mon, 12 Jul 2010 21:53:50 GMT
Message-ID: <11808671.335481278971630534.JavaMail.jira@thor>
Date: Mon, 12 Jul 2010 17:53:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6670) UserGroupInformation doesn't
 support use in hash tables
In-Reply-To: <242202762.620161270078590059.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12887555#action_12887555 ] 

Hadoop QA commented on HADOOP-6670:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12449172/c6670-02.patch
  against trunk revision 963490.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/609/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/609/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/609/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/609/console

This message is automatically generated.

> UserGroupInformation doesn't support use in hash tables
> -------------------------------------------------------
>
>                 Key: HADOOP-6670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6670
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 6670-gridmix-test.bugfix.patch, c6670-02.patch, fs-close.patch, fs-close.patch, fs-close.patch
>
>
> The UserGroupInformation objects are mutable, but they are used as keys in hash tables. This leads to serious problems in the FileSystem cache and RPC connection cache. We need to change the hashCode to be the identity hash code of the Subject and change equals to use == between the Subjects.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9462-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 12 21:59:49 2010
Return-Path: <common-issues-return-9462-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50320 invoked from network); 12 Jul 2010 21:59:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jul 2010 21:59:49 -0000
Received: (qmail 75717 invoked by uid 500); 12 Jul 2010 21:59:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75540 invoked by uid 500); 12 Jul 2010 21:59:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75531 invoked by uid 99); 12 Jul 2010 21:59:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 21:59:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 21:59:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6CLps0p011940
	for <common-issues@hadoop.apache.org>; Mon, 12 Jul 2010 21:51:54 GMT
Message-ID: <24734683.335221278971514644.JavaMail.jira@thor>
Date: Mon, 12 Jul 2010 17:51:54 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6839) [Herriot] Implement a functionality
 for getting the user list for creating proxy users.
In-Reply-To: <26284266.54901277454290297.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12887551#action_12887551 ] 

Hadoop QA commented on HADOOP-6839:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12449274/HADOOP-6839.patch
  against trunk revision 963490.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/87/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/87/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/87/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/87/console

This message is automatically generated.

> [Herriot] Implement a functionality for getting the user list for creating proxy users.
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6839
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6839
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6839-ydist-security.patch, 6839-ydist-security.patch, HADOOP-6839.patch, HADOOP-6839.patch
>
>
> Develop a new method for getting the user list. 
> Method signature is 
> public ArrayList<String> getHadoopMultiUsersList() throws IOException;
> Add new attribute in system-test.xml  file for getting userlist path.
> For submitting the jobs as different user a proxy user id is needed. So,get the available users from a userlist and then pass the user as proxy instead of hardcoding user id in a test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9463-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 12 23:50:46 2010
Return-Path: <common-issues-return-9463-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78447 invoked from network); 12 Jul 2010 23:50:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jul 2010 23:50:46 -0000
Received: (qmail 70997 invoked by uid 500); 12 Jul 2010 23:50:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70934 invoked by uid 500); 12 Jul 2010 23:50:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70926 invoked by uid 99); 12 Jul 2010 23:50:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 23:50:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jul 2010 23:50:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6CNgpDl012958
	for <common-issues@hadoop.apache.org>; Mon, 12 Jul 2010 23:42:51 GMT
Message-ID: <17611388.337461278978171190.JavaMail.jira@thor>
Date: Mon, 12 Jul 2010 19:42:51 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6670) UserGroupInformation doesn't
 support use in hash tables
In-Reply-To: <242202762.620161270078590059.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12887588#action_12887588 ] 

Kan Zhang commented on HADOOP-6670:
-----------------------------------

The javadoc warnings are due to KerberosName.java and SecurityUtil.java, and not related to this patch.

> UserGroupInformation doesn't support use in hash tables
> -------------------------------------------------------
>
>                 Key: HADOOP-6670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6670
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 6670-gridmix-test.bugfix.patch, c6670-02.patch, fs-close.patch, fs-close.patch, fs-close.patch
>
>
> The UserGroupInformation objects are mutable, but they are used as keys in hash tables. This leads to serious problems in the FileSystem cache and RPC connection cache. We need to change the hashCode to be the identity hash code of the Subject and change equals to use == between the Subjects.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9464-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 13 00:16:16 2010
Return-Path: <common-issues-return-9464-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87403 invoked from network); 13 Jul 2010 00:16:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jul 2010 00:16:16 -0000
Received: (qmail 88757 invoked by uid 500); 13 Jul 2010 00:16:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88689 invoked by uid 500); 13 Jul 2010 00:16:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88680 invoked by uid 99); 13 Jul 2010 00:16:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 00:16:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 00:16:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6D0Fp19013315
	for <common-issues@hadoop.apache.org>; Tue, 13 Jul 2010 00:15:51 GMT
Message-ID: <29969415.338211278980151549.JavaMail.jira@thor>
Date: Mon, 12 Jul 2010 20:15:51 -0400 (EDT)
From: "Scott Chen (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6590) Add a username check in
 hadoop-daemon.sh
In-Reply-To: <271804197.441811266874468125.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Scott Chen updated HADOOP-6590:
-------------------------------

    Status: Open  (was: Patch Available)

> Add a username check in hadoop-daemon.sh
> ----------------------------------------
>
>                 Key: HADOOP-6590
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6590
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.22.0
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6590-2010-07-12.txt, HADOOP-6590.patch
>
>
> We experienced a case that sometimes we accidentally started HDFS or MAPREDUCE with root user. Then the directory permission will be modified and we have to chown them. It will be nice if there can be a username checking in the hadoop-daemon.sh script so that we always start with the desired username.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9465-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 13 00:16:17 2010
Return-Path: <common-issues-return-9465-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87449 invoked from network); 13 Jul 2010 00:16:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jul 2010 00:16:17 -0000
Received: (qmail 88881 invoked by uid 500); 13 Jul 2010 00:16:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88838 invoked by uid 500); 13 Jul 2010 00:16:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88690 invoked by uid 99); 13 Jul 2010 00:16:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 00:16:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 00:16:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6D0FqcW013330
	for <common-issues@hadoop.apache.org>; Tue, 13 Jul 2010 00:15:52 GMT
Message-ID: <18394177.338261278980152410.JavaMail.jira@thor>
Date: Mon, 12 Jul 2010 20:15:52 -0400 (EDT)
From: "Scott Chen (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6590) Add a username check in
 hadoop-daemon.sh
In-Reply-To: <271804197.441811266874468125.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Scott Chen updated HADOOP-6590:
-------------------------------

    Status: Patch Available  (was: Open)

> Add a username check in hadoop-daemon.sh
> ----------------------------------------
>
>                 Key: HADOOP-6590
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6590
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.22.0
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6590-2010-07-12.txt, HADOOP-6590.patch
>
>
> We experienced a case that sometimes we accidentally started HDFS or MAPREDUCE with root user. Then the directory permission will be modified and we have to chown them. It will be nice if there can be a username checking in the hadoop-daemon.sh script so that we always start with the desired username.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9466-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 13 00:19:52 2010
Return-Path: <common-issues-return-9466-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88187 invoked from network); 13 Jul 2010 00:19:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jul 2010 00:19:51 -0000
Received: (qmail 91891 invoked by uid 500); 13 Jul 2010 00:19:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91853 invoked by uid 500); 13 Jul 2010 00:19:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91843 invoked by uid 99); 13 Jul 2010 00:19:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 00:19:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 00:19:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6D0BoAb013282
	for <common-issues@hadoop.apache.org>; Tue, 13 Jul 2010 00:11:50 GMT
Message-ID: <3993299.338121278979910543.JavaMail.jira@thor>
Date: Mon, 12 Jul 2010 20:11:50 -0400 (EDT)
From: "Scott Chen (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6590) Add a username check in
 hadoop-daemon.sh
In-Reply-To: <271804197.441811266874468125.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Scott Chen updated HADOOP-6590:
-------------------------------

    Attachment: HADOOP-6590-2010-07-12.txt

Update the patch to fit the current code.

> Add a username check in hadoop-daemon.sh
> ----------------------------------------
>
>                 Key: HADOOP-6590
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6590
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.22.0
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6590-2010-07-12.txt, HADOOP-6590.patch
>
>
> We experienced a case that sometimes we accidentally started HDFS or MAPREDUCE with root user. Then the directory permission will be modified and we have to chown them. It will be nice if there can be a username checking in the hadoop-daemon.sh script so that we always start with the desired username.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9467-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 13 00:35:44 2010
Return-Path: <common-issues-return-9467-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98792 invoked from network); 13 Jul 2010 00:35:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jul 2010 00:35:44 -0000
Received: (qmail 2153 invoked by uid 500); 13 Jul 2010 00:35:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2094 invoked by uid 500); 13 Jul 2010 00:35:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2086 invoked by uid 99); 13 Jul 2010 00:35:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 00:35:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 00:35:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6D0Rowl013378
	for <common-issues@hadoop.apache.org>; Tue, 13 Jul 2010 00:27:50 GMT
Message-ID: <20536612.338341278980870283.JavaMail.jira@thor>
Date: Mon, 12 Jul 2010 20:27:50 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6724) IPC doesn't properly handle IOEs
 thrown by socket factory
In-Reply-To: <4486259.186941272248270321.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Allen Wittenauer updated HADOOP-6724:
-------------------------------------

    Fix Version/s: 0.21.0

> IPC doesn't properly handle IOEs thrown by socket factory
> ---------------------------------------------------------
>
>                 Key: HADOOP-6724
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6724
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.20.3, 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6724-20-1.patch, hadoop-6724.txt
>
>
> If the socket factory throws an IOE inside setupIOStreams, then handleConnectionFailure will be called with socket still null, and thus generate an NPE on socket.close(). This ends up orphaning clients, etc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9468-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 13 02:57:17 2010
Return-Path: <common-issues-return-9468-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42745 invoked from network); 13 Jul 2010 02:57:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jul 2010 02:57:17 -0000
Received: (qmail 77452 invoked by uid 500); 13 Jul 2010 02:57:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77330 invoked by uid 500); 13 Jul 2010 02:57:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77320 invoked by uid 99); 13 Jul 2010 02:57:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 02:57:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 02:57:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6D2uoog014428
	for <common-issues@hadoop.apache.org>; Tue, 13 Jul 2010 02:56:51 GMT
Message-ID: <26722143.340311278989810887.JavaMail.jira@thor>
Date: Mon, 12 Jul 2010 22:56:50 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6825) FileStatus needs unit tests
In-Reply-To: <28758544.71131276463653554.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6825:
--------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

I've committed this. Thanks Rodrigo!

> FileStatus needs unit tests
> ---------------------------
>
>                 Key: HADOOP-6825
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6825
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6825.0.patch
>
>
> We need some unit tests for FileStatus to prevent problems like those we recently had on HADOOP-6796 and MAPREDUCE-1858.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9469-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 13 17:16:18 2010
Return-Path: <common-issues-return-9469-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62044 invoked from network); 13 Jul 2010 17:16:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jul 2010 17:16:18 -0000
Received: (qmail 56482 invoked by uid 500); 13 Jul 2010 17:16:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56369 invoked by uid 500); 13 Jul 2010 17:16:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56361 invoked by uid 99); 13 Jul 2010 17:16:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 17:16:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 17:16:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6DHFqnj022400
	for <common-issues@hadoop.apache.org>; Tue, 13 Jul 2010 17:15:53 GMT
Message-ID: <29243473.351231279041352904.JavaMail.jira@thor>
Date: Tue, 13 Jul 2010 13:15:52 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6647) balancer fails with "is not
 authorized for protocol interface NamenodeProtocol" in secure environment
In-Reply-To: <387797205.373021269030508879.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12887856#action_12887856 ] 

Hudson commented on HADOOP-6647:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #324 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/324/])
    HADOOP-6647. balancer fails with "is not authorized for protocol interface NamenodeProtocol" in secure environment


> balancer fails with "is not authorized for protocol interface NamenodeProtocol" in secure environment
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6647
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6647
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6647-BP20.patch, HADOOP-6647.patch
>
>
> user logs in as hdfs/something@something and tries to run balancer.
> balancer is using NameNode Protocol which authorizes based on server principal key.
> but NameNode key is hdfs/_HOST@.. now. so it fails. 
> To fix we need to compare the short names only.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9471-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 13 17:16:19 2010
Return-Path: <common-issues-return-9471-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62096 invoked from network); 13 Jul 2010 17:16:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jul 2010 17:16:19 -0000
Received: (qmail 56595 invoked by uid 500); 13 Jul 2010 17:16:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56531 invoked by uid 500); 13 Jul 2010 17:16:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56378 invoked by uid 99); 13 Jul 2010 17:16:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 17:16:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 17:16:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6DHFsFm022424
	for <common-issues@hadoop.apache.org>; Tue, 13 Jul 2010 17:15:54 GMT
Message-ID: <24188160.351311279041354248.JavaMail.jira@thor>
Date: Tue, 13 Jul 2010 13:15:54 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6648) Credentials should ignore null
 tokens
In-Reply-To: <626471334.379531269043887761.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12887858#action_12887858 ] 

Hudson commented on HADOOP-6648:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #324 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/324/])
    HADOOP-6648. Adds a check for null tokens in Credentials.addToken api. Contributed by Owen O'Malley and Devaraj Das.


> Credentials should ignore null tokens
> -------------------------------------
>
>                 Key: HADOOP-6648
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6648
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: c-6648-y20.patch, h-6648-trunk.patch
>
>
> When hftp goes to a non-secure cluster, getDelegationToken returns null. Credentials.addToken needs to ignore null tokens to avoid a NPE in serialization.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9470-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 13 17:16:20 2010
Return-Path: <common-issues-return-9470-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62090 invoked from network); 13 Jul 2010 17:16:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jul 2010 17:16:19 -0000
Received: (qmail 56582 invoked by uid 500); 13 Jul 2010 17:16:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56388 invoked by uid 500); 13 Jul 2010 17:16:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56370 invoked by uid 99); 13 Jul 2010 17:16:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 17:16:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 17:16:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6DHFrh2022415
	for <common-issues@hadoop.apache.org>; Tue, 13 Jul 2010 17:15:53 GMT
Message-ID: <16864975.351281279041353807.JavaMail.jira@thor>
Date: Tue, 13 Jul 2010 13:15:53 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6819) [Herriot] Shell command for getting
 the new exceptions in the logs returning exitcode 1 after executing
 successfully.
In-Reply-To: <25211660.39321276233134711.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12887857#action_12887857 ] 

Hudson commented on HADOOP-6819:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #324 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/324/])
    HADOOP-6819. [Herriot] Shell command for getting the new exceptions in
the logs returning exitcode 1 after executing successfully. Contributed
by Vinay Thota.


> [Herriot] Shell command for getting the new exceptions in the logs returning exitcode 1 after executing successfully.
> ---------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6819
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6819
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>             Fix For: 0.21.0
>
>         Attachments: 6819-ydist-security.patch, HADOOP-6819.patch
>
>
> I have found an corner case issue while executing the shell command for getting the new exceptions in the logs. The problem could see whenever the log files are empty and trying to get the new exceptions count, it returns the output is 0 ,however it returns the exitcode is 1 instead of 0. grep -vc in the command is not  giving the expected exitcode because its not handling the multiple in options.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9472-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 13 17:16:20 2010
Return-Path: <common-issues-return-9472-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62126 invoked from network); 13 Jul 2010 17:16:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jul 2010 17:16:19 -0000
Received: (qmail 56868 invoked by uid 500); 13 Jul 2010 17:16:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56823 invoked by uid 500); 13 Jul 2010 17:16:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56815 invoked by uid 99); 13 Jul 2010 17:16:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 17:16:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 17:16:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6DHFtm5022439
	for <common-issues@hadoop.apache.org>; Tue, 13 Jul 2010 17:15:55 GMT
Message-ID: <27615480.351361279041355479.JavaMail.jira@thor>
Date: Tue, 13 Jul 2010 13:15:55 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6825) FileStatus needs unit tests
In-Reply-To: <28758544.71131276463653554.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12887859#action_12887859 ] 

Hudson commented on HADOOP-6825:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #324 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/324/])
    HADOOP-6825. FileStatus needs unit tests. Contributed by Rodrigo Schmidt.


> FileStatus needs unit tests
> ---------------------------
>
>                 Key: HADOOP-6825
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6825
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6825.0.patch
>
>
> We need some unit tests for FileStatus to prevent problems like those we recently had on HADOOP-6796 and MAPREDUCE-1858.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9473-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 13 17:30:47 2010
Return-Path: <common-issues-return-9473-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49142 invoked from network); 13 Jul 2010 17:30:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jul 2010 17:30:45 -0000
Received: (qmail 92469 invoked by uid 500); 13 Jul 2010 17:30:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92404 invoked by uid 500); 13 Jul 2010 17:30:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92391 invoked by uid 99); 13 Jul 2010 17:30:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 17:30:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 17:30:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6DHMoeU022486
	for <common-issues@hadoop.apache.org>; Tue, 13 Jul 2010 17:22:50 GMT
Message-ID: <26762544.351441279041770706.JavaMail.jira@thor>
Date: Tue, 13 Jul 2010 13:22:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6590) Add a username check in
 hadoop-daemon.sh
In-Reply-To: <271804197.441811266874468125.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12887861#action_12887861 ] 

Hadoop QA commented on HADOOP-6590:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12449308/HADOOP-6590-2010-07-12.txt
  against trunk revision 963593.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/610/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/610/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/610/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/610/console

This message is automatically generated.

> Add a username check in hadoop-daemon.sh
> ----------------------------------------
>
>                 Key: HADOOP-6590
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6590
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 0.22.0
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6590-2010-07-12.txt, HADOOP-6590.patch
>
>
> We experienced a case that sometimes we accidentally started HDFS or MAPREDUCE with root user. Then the directory permission will be modified and we have to chown them. It will be nice if there can be a username checking in the hadoop-daemon.sh script so that we always start with the desired username.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9474-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 13 17:34:15 2010
Return-Path: <common-issues-return-9474-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77901 invoked from network); 13 Jul 2010 17:34:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jul 2010 17:34:15 -0000
Received: (qmail 659 invoked by uid 500); 13 Jul 2010 17:34:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 549 invoked by uid 500); 13 Jul 2010 17:34:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 536 invoked by uid 99); 13 Jul 2010 17:34:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 17:34:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 17:34:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6DHXoM3022681
	for <common-issues@hadoop.apache.org>; Tue, 13 Jul 2010 17:33:50 GMT
Message-ID: <23012236.351981279042430459.JavaMail.jira@thor>
Date: Tue, 13 Jul 2010 13:33:50 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6855) Add ability to get groups for ACLs
 from 'getent netgroup'
In-Reply-To: <23521712.275021278635570824.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12887869#action_12887869 ] 

Boris Shkolnik commented on HADOOP-6855:
----------------------------------------

Here are some comments so far..
1. commented log lines Shellbasedunixgroup:122 and after could be removed

2. getNetgroups - returns list that is passed to it as an argument. Do you need this return value?
 
3. List groups = new LinkedList(netgroupToUsersMap.keySet()); - shouldn't it be a generic LinkedList<E>.


4. ShellBasedUnixGroupsNetgroupMappingTestWrapper: execShellGetUnixGroups - seems to return wrong data  - call for groups for user foo doesn't return user name as the first entry (not sure if it is used in tests though).


5. Patch seems to introduce 2 new warnings:
ShellBasedUnixGroupsNetgroupMapping.java:60: warning: [unchecked] unchecked call to Lin
kedList(java.util.Collection<? extends E>) as a member of the raw type java.util.LinkedList
    [javac]     List groups = new LinkedList(netgroupToUsersMap.keySet());
    [javac]                   ^
    [javac] ShellBasedUnixGroupsNetgroupMapping.java:62: warning: [unchecked] unchecked conversion
    [javac] found   : java.util.List
    [javac] required: java.util.List<java.lang.String>
    [javac]     cacheGroupsAdd(groups);
    [javac]                    ^


> Add ability to get groups for ACLs from 'getent netgroup'
> ---------------------------------------------------------
>
>                 Key: HADOOP-6855
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6855
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6855-0.20.patch
>
>
> Add ability to specify netgroups in ACLs (see class AccessControlList.java). Membership of users in netgroups will be determined by running 'getent negroups $groupName'. Netgroups will be differentiated from regular unix groups by having '@' as a first character.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9475-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 13 21:08:15 2010
Return-Path: <common-issues-return-9475-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95460 invoked from network); 13 Jul 2010 21:08:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jul 2010 21:08:15 -0000
Received: (qmail 12174 invoked by uid 500); 13 Jul 2010 21:08:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12103 invoked by uid 500); 13 Jul 2010 21:08:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12094 invoked by uid 99); 13 Jul 2010 21:08:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 21:08:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 21:08:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6DL7n1G025802
	for <common-issues@hadoop.apache.org>; Tue, 13 Jul 2010 21:07:50 GMT
Message-ID: <10952334.359241279055269951.JavaMail.jira@thor>
Date: Tue, 13 Jul 2010 17:07:49 -0400 (EDT)
From: "Andrew Ryan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6858) Enable rotateable JVM garbage
 collection logs for Hadoop daemons
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Enable rotateable JVM garbage collection logs for Hadoop daemons
----------------------------------------------------------------

                 Key: HADOOP-6858
                 URL: https://issues.apache.org/jira/browse/HADOOP-6858
             Project: Hadoop Common
          Issue Type: New Feature
          Components: scripts
    Affects Versions: 0.22.0
            Reporter: Andrew Ryan


The purpose of this enhancement is to make it easier to collect garbage collection logs and insure that they persist across restarts in the same way that the standard output files of Hadoop daemon JVM's currently does.

Garbage collection logs are a vital debugging tool for administrators and developers. In our production environments, at some point or another, every single type of Hadoop daemon has OOM'ed or experienced other significant issues related to GC and/or lack of heap memory. For the longest time, we have put in garbage collection logs in our HADOOP_NAMENODE_OPTS, HADOOP_JOBTRACKER_OPTS, etc. by using options like "-XX:+PrintGCDateStamps -XX:+PrintGCDetails -Xloggc:$HADOOP_LOG_DIR/jobtracker.gc.log".

Unfortunately, these logs don't survive a restart of the node, so if a node OOM's and then is restarted automatically, or manually by someone who is unaware, we lose the GC logs forever. We also have to manually add GC log options to each daemon. This patch:
1) Creates a single, optional, off by default, parameter for specifying GC logging.
2) If that parameter is set, automatically enables GC logging for all daemons in the cluster. The parameter is flexible enough to allow for the different ways various vendor's JVM's require garbage collection logging to be specified. 
3) If GC logging is on, insures that the GC log files for each daemon are rotated with up to 5 copies kept, same as the .out files currently.

We are currently running a variation of this patch in our 0.20 install. This patch actually includes changes to common, mapred, and hdfs, so it obviously cannot be applied as-is, but is included here for review and comments.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9476-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 13 21:17:18 2010
Return-Path: <common-issues-return-9476-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99599 invoked from network); 13 Jul 2010 21:17:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jul 2010 21:17:18 -0000
Received: (qmail 28345 invoked by uid 500); 13 Jul 2010 21:17:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28299 invoked by uid 500); 13 Jul 2010 21:17:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28291 invoked by uid 99); 13 Jul 2010 21:17:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 21:17:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 21:17:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6DLGrjb025983
	for <common-issues@hadoop.apache.org>; Tue, 13 Jul 2010 21:16:53 GMT
Message-ID: <32693001.359711279055813052.JavaMail.jira@thor>
Date: Tue, 13 Jul 2010 17:16:53 -0400 (EDT)
From: "Andrew Ryan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6858) Enable rotateable JVM garbage
 collection logs for Hadoop daemons
In-Reply-To: <10952334.359241279055269951.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Ryan updated HADOOP-6858:
--------------------------------

    Attachment: HADOOP-6858.patch

This patch requires changes in common, mapred, and hdfs. So this patch will not actually apply to Hudson yet.

> Enable rotateable JVM garbage collection logs for Hadoop daemons
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6858
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6858
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: scripts
>    Affects Versions: 0.22.0
>            Reporter: Andrew Ryan
>         Attachments: HADOOP-6858.patch
>
>
> The purpose of this enhancement is to make it easier to collect garbage collection logs and insure that they persist across restarts in the same way that the standard output files of Hadoop daemon JVM's currently does.
> Garbage collection logs are a vital debugging tool for administrators and developers. In our production environments, at some point or another, every single type of Hadoop daemon has OOM'ed or experienced other significant issues related to GC and/or lack of heap memory. For the longest time, we have put in garbage collection logs in our HADOOP_NAMENODE_OPTS, HADOOP_JOBTRACKER_OPTS, etc. by using options like "-XX:+PrintGCDateStamps -XX:+PrintGCDetails -Xloggc:$HADOOP_LOG_DIR/jobtracker.gc.log".
> Unfortunately, these logs don't survive a restart of the node, so if a node OOM's and then is restarted automatically, or manually by someone who is unaware, we lose the GC logs forever. We also have to manually add GC log options to each daemon. This patch:
> 1) Creates a single, optional, off by default, parameter for specifying GC logging.
> 2) If that parameter is set, automatically enables GC logging for all daemons in the cluster. The parameter is flexible enough to allow for the different ways various vendor's JVM's require garbage collection logging to be specified. 
> 3) If GC logging is on, insures that the GC log files for each daemon are rotated with up to 5 copies kept, same as the .out files currently.
> We are currently running a variation of this patch in our 0.20 install. This patch actually includes changes to common, mapred, and hdfs, so it obviously cannot be applied as-is, but is included here for review and comments.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9477-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 13 21:26:47 2010
Return-Path: <common-issues-return-9477-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13535 invoked from network); 13 Jul 2010 21:26:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jul 2010 21:26:46 -0000
Received: (qmail 43625 invoked by uid 500); 13 Jul 2010 21:26:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43519 invoked by uid 500); 13 Jul 2010 21:26:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43511 invoked by uid 99); 13 Jul 2010 21:26:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 21:26:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 21:26:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6DLIpFM025991
	for <common-issues@hadoop.apache.org>; Tue, 13 Jul 2010 21:18:51 GMT
Message-ID: <15870867.359731279055931488.JavaMail.jira@thor>
Date: Tue, 13 Jul 2010 17:18:51 -0400 (EDT)
From: "Andrew Ryan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6858) Enable rotateable JVM garbage
 collection logs for Hadoop daemons
In-Reply-To: <10952334.359241279055269951.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Ryan updated HADOOP-6858:
--------------------------------

    Status: Patch Available  (was: Open)

Patch submitted for review. This won't apply, because it needs patching common, mapred, and hdfs. If there is agreement on how to move forward, I can regenerate different patches for each.

> Enable rotateable JVM garbage collection logs for Hadoop daemons
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6858
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6858
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: scripts
>    Affects Versions: 0.22.0
>            Reporter: Andrew Ryan
>         Attachments: HADOOP-6858.patch
>
>
> The purpose of this enhancement is to make it easier to collect garbage collection logs and insure that they persist across restarts in the same way that the standard output files of Hadoop daemon JVM's currently does.
> Garbage collection logs are a vital debugging tool for administrators and developers. In our production environments, at some point or another, every single type of Hadoop daemon has OOM'ed or experienced other significant issues related to GC and/or lack of heap memory. For the longest time, we have put in garbage collection logs in our HADOOP_NAMENODE_OPTS, HADOOP_JOBTRACKER_OPTS, etc. by using options like "-XX:+PrintGCDateStamps -XX:+PrintGCDetails -Xloggc:$HADOOP_LOG_DIR/jobtracker.gc.log".
> Unfortunately, these logs don't survive a restart of the node, so if a node OOM's and then is restarted automatically, or manually by someone who is unaware, we lose the GC logs forever. We also have to manually add GC log options to each daemon. This patch:
> 1) Creates a single, optional, off by default, parameter for specifying GC logging.
> 2) If that parameter is set, automatically enables GC logging for all daemons in the cluster. The parameter is flexible enough to allow for the different ways various vendor's JVM's require garbage collection logging to be specified. 
> 3) If GC logging is on, insures that the GC log files for each daemon are rotated with up to 5 copies kept, same as the .out files currently.
> We are currently running a variation of this patch in our 0.20 install. This patch actually includes changes to common, mapred, and hdfs, so it obviously cannot be applied as-is, but is included here for review and comments.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9478-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 13 21:44:45 2010
Return-Path: <common-issues-return-9478-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30767 invoked from network); 13 Jul 2010 21:44:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jul 2010 21:44:45 -0000
Received: (qmail 75653 invoked by uid 500); 13 Jul 2010 21:44:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75594 invoked by uid 500); 13 Jul 2010 21:44:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75586 invoked by uid 99); 13 Jul 2010 21:44:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 21:44:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 21:44:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6DLao6N026120
	for <common-issues@hadoop.apache.org>; Tue, 13 Jul 2010 21:36:50 GMT
Message-ID: <1227924.360041279057010259.JavaMail.jira@thor>
Date: Tue, 13 Jul 2010 17:36:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6858) Enable rotateable JVM garbage
 collection logs for Hadoop daemons
In-Reply-To: <10952334.359241279055269951.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888025#action_12888025 ] 

Hadoop QA commented on HADOOP-6858:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12449386/HADOOP-6858.patch
  against trunk revision 963593.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 patch.  The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/611/console

This message is automatically generated.

> Enable rotateable JVM garbage collection logs for Hadoop daemons
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6858
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6858
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: scripts
>    Affects Versions: 0.22.0
>            Reporter: Andrew Ryan
>         Attachments: HADOOP-6858.patch
>
>
> The purpose of this enhancement is to make it easier to collect garbage collection logs and insure that they persist across restarts in the same way that the standard output files of Hadoop daemon JVM's currently does.
> Garbage collection logs are a vital debugging tool for administrators and developers. In our production environments, at some point or another, every single type of Hadoop daemon has OOM'ed or experienced other significant issues related to GC and/or lack of heap memory. For the longest time, we have put in garbage collection logs in our HADOOP_NAMENODE_OPTS, HADOOP_JOBTRACKER_OPTS, etc. by using options like "-XX:+PrintGCDateStamps -XX:+PrintGCDetails -Xloggc:$HADOOP_LOG_DIR/jobtracker.gc.log".
> Unfortunately, these logs don't survive a restart of the node, so if a node OOM's and then is restarted automatically, or manually by someone who is unaware, we lose the GC logs forever. We also have to manually add GC log options to each daemon. This patch:
> 1) Creates a single, optional, off by default, parameter for specifying GC logging.
> 2) If that parameter is set, automatically enables GC logging for all daemons in the cluster. The parameter is flexible enough to allow for the different ways various vendor's JVM's require garbage collection logging to be specified. 
> 3) If GC logging is on, insures that the GC log files for each daemon are rotated with up to 5 copies kept, same as the .out files currently.
> We are currently running a variation of this patch in our 0.20 install. This patch actually includes changes to common, mapred, and hdfs, so it obviously cannot be applied as-is, but is included here for review and comments.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9479-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 13 21:46:17 2010
Return-Path: <common-issues-return-9479-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31098 invoked from network); 13 Jul 2010 21:46:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jul 2010 21:46:17 -0000
Received: (qmail 78975 invoked by uid 500); 13 Jul 2010 21:46:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78837 invoked by uid 500); 13 Jul 2010 21:46:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78797 invoked by uid 99); 13 Jul 2010 21:46:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 21:46:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 21:46:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6DLjp2H026262
	for <common-issues@hadoop.apache.org>; Tue, 13 Jul 2010 21:45:52 GMT
Message-ID: <3466971.360371279057551878.JavaMail.jira@thor>
Date: Tue, 13 Jul 2010 17:45:51 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6859) Introduce additional statistics to
 FileSystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Introduce additional statistics to FileSystem
---------------------------------------------

                 Key: HADOOP-6859
                 URL: https://issues.apache.org/jira/browse/HADOOP-6859
             Project: Hadoop Common
          Issue Type: Improvement
    Affects Versions: 0.22.0
            Reporter: Suresh Srinivas
            Assignee: Suresh Srinivas
             Fix For: 0.22.0


Currently FileSystem#statistics tracks bytesRead and bytesWritten. Additional statistics that gives summary of operations performed will be useful for tracking file system use.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9480-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 13 22:01:47 2010
Return-Path: <common-issues-return-9480-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46839 invoked from network); 13 Jul 2010 22:01:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jul 2010 22:01:46 -0000
Received: (qmail 98884 invoked by uid 500); 13 Jul 2010 22:01:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98214 invoked by uid 500); 13 Jul 2010 22:01:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98205 invoked by uid 99); 13 Jul 2010 22:01:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 22:01:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 22:01:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6DLrn25026289
	for <common-issues@hadoop.apache.org>; Tue, 13 Jul 2010 21:53:50 GMT
Message-ID: <25381891.360431279058029788.JavaMail.jira@thor>
Date: Tue, 13 Jul 2010 17:53:49 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6859) Introduce additional statistics to
 FileSystem
In-Reply-To: <3466971.360371279057551878.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888032#action_12888032 ] 

Suresh Srinivas commented on HADOOP-6859:
-----------------------------------------

I am planning to introduce the following additional statistics, that is accumulated at the client per file system as it is done now.
# read operations - number of read operations such as listStatus, getFileBlockLocations, open etc.
# write operations - number of write operations such as create, append, setPermission etc.
# large read operations - on file system, most of the operations are small except listFiles for a large directory. Iterative listFiles was introduced in HDFS to break down a single large operation into smaller steps. This counter is incremented is incremented for every iteration of listFiles, when listing files under a large directory.

These statistics are collected in job history for analysis of how HDFS is loaded by map reduce tasks. This is useful in the interim to identify jobs that heavily load HDFS. In future this could also be used to throttle the load at the map reduce framework.

> Introduce additional statistics to FileSystem
> ---------------------------------------------
>
>                 Key: HADOOP-6859
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6859
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>
> Currently FileSystem#statistics tracks bytesRead and bytesWritten. Additional statistics that gives summary of operations performed will be useful for tracking file system use.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9481-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 13 22:13:46 2010
Return-Path: <common-issues-return-9481-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95882 invoked from network); 13 Jul 2010 22:13:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jul 2010 22:13:45 -0000
Received: (qmail 27195 invoked by uid 500); 13 Jul 2010 22:13:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27086 invoked by uid 500); 13 Jul 2010 22:13:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27078 invoked by uid 99); 13 Jul 2010 22:13:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 22:13:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 22:13:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6DM5pAq026477
	for <common-issues@hadoop.apache.org>; Tue, 13 Jul 2010 22:05:51 GMT
Message-ID: <29057041.360751279058751296.JavaMail.jira@thor>
Date: Tue, 13 Jul 2010 18:05:51 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6859) Introduce additional statistics to
 FileSystem
In-Reply-To: <3466971.360371279057551878.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6859:
------------------------------------

    Attachment: HADOOP-6859.patch

Attached path adds new statistics.

> Introduce additional statistics to FileSystem
> ---------------------------------------------
>
>                 Key: HADOOP-6859
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6859
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6859.patch
>
>
> Currently FileSystem#statistics tracks bytesRead and bytesWritten. Additional statistics that gives summary of operations performed will be useful for tracking file system use.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9482-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 13 22:24:46 2010
Return-Path: <common-issues-return-9482-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31450 invoked from network); 13 Jul 2010 22:24:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jul 2010 22:24:46 -0000
Received: (qmail 40996 invoked by uid 500); 13 Jul 2010 22:24:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40941 invoked by uid 500); 13 Jul 2010 22:24:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40723 invoked by uid 99); 13 Jul 2010 22:24:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 22:24:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 22:24:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6DMGpnw026656
	for <common-issues@hadoop.apache.org>; Tue, 13 Jul 2010 22:16:51 GMT
Message-ID: <26007616.361221279059411452.JavaMail.jira@thor>
Date: Tue, 13 Jul 2010 18:16:51 -0400 (EDT)
From: "Philip Zeyliger (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6859) Introduce additional statistics to
 FileSystem
In-Reply-To: <3466971.360371279057551878.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888049#action_12888049 ] 

Philip Zeyliger commented on HADOOP-6859:
-----------------------------------------

Patch looks fine.  You might want to mention in the javadoc what you mentioned here in the JIRA about what a "large read" operation may be.

> Introduce additional statistics to FileSystem
> ---------------------------------------------
>
>                 Key: HADOOP-6859
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6859
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6859.patch
>
>
> Currently FileSystem#statistics tracks bytesRead and bytesWritten. Additional statistics that gives summary of operations performed will be useful for tracking file system use.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9483-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 13 22:44:46 2010
Return-Path: <common-issues-return-9483-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63767 invoked from network); 13 Jul 2010 22:44:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jul 2010 22:44:45 -0000
Received: (qmail 81519 invoked by uid 500); 13 Jul 2010 22:44:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81476 invoked by uid 500); 13 Jul 2010 22:44:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81468 invoked by uid 99); 13 Jul 2010 22:44:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 22:44:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 22:44:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6DMaoxu026829
	for <common-issues@hadoop.apache.org>; Tue, 13 Jul 2010 22:36:50 GMT
Message-ID: <9668808.361641279060610494.JavaMail.jira@thor>
Date: Tue, 13 Jul 2010 18:36:50 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6858) Enable rotateable JVM garbage
 collection logs for Hadoop daemons
In-Reply-To: <10952334.359241279055269951.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888061#action_12888061 ] 

Allen Wittenauer commented on HADOOP-6858:
------------------------------------------

We just put a date on our logs.

i.e., gc.log.`date +blahblah`.



> Enable rotateable JVM garbage collection logs for Hadoop daemons
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6858
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6858
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: scripts
>    Affects Versions: 0.22.0
>            Reporter: Andrew Ryan
>         Attachments: HADOOP-6858.patch
>
>
> The purpose of this enhancement is to make it easier to collect garbage collection logs and insure that they persist across restarts in the same way that the standard output files of Hadoop daemon JVM's currently does.
> Garbage collection logs are a vital debugging tool for administrators and developers. In our production environments, at some point or another, every single type of Hadoop daemon has OOM'ed or experienced other significant issues related to GC and/or lack of heap memory. For the longest time, we have put in garbage collection logs in our HADOOP_NAMENODE_OPTS, HADOOP_JOBTRACKER_OPTS, etc. by using options like "-XX:+PrintGCDateStamps -XX:+PrintGCDetails -Xloggc:$HADOOP_LOG_DIR/jobtracker.gc.log".
> Unfortunately, these logs don't survive a restart of the node, so if a node OOM's and then is restarted automatically, or manually by someone who is unaware, we lose the GC logs forever. We also have to manually add GC log options to each daemon. This patch:
> 1) Creates a single, optional, off by default, parameter for specifying GC logging.
> 2) If that parameter is set, automatically enables GC logging for all daemons in the cluster. The parameter is flexible enough to allow for the different ways various vendor's JVM's require garbage collection logging to be specified. 
> 3) If GC logging is on, insures that the GC log files for each daemon are rotated with up to 5 copies kept, same as the .out files currently.
> We are currently running a variation of this patch in our 0.20 install. This patch actually includes changes to common, mapred, and hdfs, so it obviously cannot be applied as-is, but is included here for review and comments.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9484-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 13 23:31:16 2010
Return-Path: <common-issues-return-9484-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29586 invoked from network); 13 Jul 2010 23:31:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jul 2010 23:31:15 -0000
Received: (qmail 36753 invoked by uid 500); 13 Jul 2010 23:31:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36692 invoked by uid 500); 13 Jul 2010 23:31:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36684 invoked by uid 99); 13 Jul 2010 23:31:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 23:31:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Jul 2010 23:31:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6DNUoZr027617
	for <common-issues@hadoop.apache.org>; Tue, 13 Jul 2010 23:30:50 GMT
Message-ID: <31781284.363771279063850583.JavaMail.jira@thor>
Date: Tue, 13 Jul 2010 19:30:50 -0400 (EDT)
From: "Andrew Ryan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6858) Enable rotateable JVM garbage
 collection logs for Hadoop daemons
In-Reply-To: <10952334.359241279055269951.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888105#action_12888105 ] 

Andrew Ryan commented on HADOOP-6858:
-------------------------------------

We chose not to go that route, because date stamping in that way would address the "logs get blown away on startup" problem, but then it creates a "gc logs are never cleaned up automatically by hadoop" problem. But there are certainly many ways to solve this problem. We're hoping to reach some kind of consensus.

> Enable rotateable JVM garbage collection logs for Hadoop daemons
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6858
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6858
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: scripts
>    Affects Versions: 0.22.0
>            Reporter: Andrew Ryan
>         Attachments: HADOOP-6858.patch
>
>
> The purpose of this enhancement is to make it easier to collect garbage collection logs and insure that they persist across restarts in the same way that the standard output files of Hadoop daemon JVM's currently does.
> Garbage collection logs are a vital debugging tool for administrators and developers. In our production environments, at some point or another, every single type of Hadoop daemon has OOM'ed or experienced other significant issues related to GC and/or lack of heap memory. For the longest time, we have put in garbage collection logs in our HADOOP_NAMENODE_OPTS, HADOOP_JOBTRACKER_OPTS, etc. by using options like "-XX:+PrintGCDateStamps -XX:+PrintGCDetails -Xloggc:$HADOOP_LOG_DIR/jobtracker.gc.log".
> Unfortunately, these logs don't survive a restart of the node, so if a node OOM's and then is restarted automatically, or manually by someone who is unaware, we lose the GC logs forever. We also have to manually add GC log options to each daemon. This patch:
> 1) Creates a single, optional, off by default, parameter for specifying GC logging.
> 2) If that parameter is set, automatically enables GC logging for all daemons in the cluster. The parameter is flexible enough to allow for the different ways various vendor's JVM's require garbage collection logging to be specified. 
> 3) If GC logging is on, insures that the GC log files for each daemon are rotated with up to 5 copies kept, same as the .out files currently.
> We are currently running a variation of this patch in our 0.20 install. This patch actually includes changes to common, mapred, and hdfs, so it obviously cannot be applied as-is, but is included here for review and comments.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9485-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 00:12:18 2010
Return-Path: <common-issues-return-9485-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19157 invoked from network); 14 Jul 2010 00:12:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 00:12:18 -0000
Received: (qmail 68990 invoked by uid 500); 14 Jul 2010 00:12:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68939 invoked by uid 500); 14 Jul 2010 00:12:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68931 invoked by uid 99); 14 Jul 2010 00:12:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 00:12:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 00:12:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6E0Brbb027989
	for <common-issues@hadoop.apache.org>; Wed, 14 Jul 2010 00:11:53 GMT
Message-ID: <8725188.364581279066313431.JavaMail.jira@thor>
Date: Tue, 13 Jul 2010 20:11:53 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6855) Add ability to get groups for ACLs
 from 'getent netgroup'
In-Reply-To: <23521712.275021278635570824.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Erik Steffl updated HADOOP-6855:
--------------------------------

    Attachment: HADOOP-6855-0.20-1.patch

> Add ability to get groups for ACLs from 'getent netgroup'
> ---------------------------------------------------------
>
>                 Key: HADOOP-6855
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6855
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6855-0.20-1.patch, HADOOP-6855-0.20.patch
>
>
> Add ability to specify netgroups in ACLs (see class AccessControlList.java). Membership of users in netgroups will be determined by running 'getent negroups $groupName'. Netgroups will be differentiated from regular unix groups by having '@' as a first character.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9486-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 00:26:46 2010
Return-Path: <common-issues-return-9486-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38351 invoked from network); 14 Jul 2010 00:26:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 00:26:46 -0000
Received: (qmail 79996 invoked by uid 500); 14 Jul 2010 00:26:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79951 invoked by uid 500); 14 Jul 2010 00:26:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79943 invoked by uid 99); 14 Jul 2010 00:26:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 00:26:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 00:26:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6E0Iqkw028037
	for <common-issues@hadoop.apache.org>; Wed, 14 Jul 2010 00:18:52 GMT
Message-ID: <3751457.364711279066732120.JavaMail.jira@thor>
Date: Tue, 13 Jul 2010 20:18:52 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6855) Add ability to get groups for ACLs
 from 'getent netgroup'
In-Reply-To: <23521712.275021278635570824.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888124#action_12888124 ] 

Erik Steffl commented on HADOOP-6855:
-------------------------------------

HADOOP-6855-0.20-1.patch fixes problems pointed out by review:

1. ShellBasedUnixGroupsNetgroupMapping.java: commented out log lines removed

2.  ShellBasedUnixGroupsNetgroupMapping.java: getNetgroups, getUnixGroups return void now

3. ShellBasedUnixGroupsNetgroupMapping.java: LinkedList<String> is used

4. ShellBasedUnixGroupsNetgroupMappingTestWrapper.java: yes, the first element was user, removed. And yes, it doesn't really matter for tests

5. No warnings, was caused by usage of LinkedList instead of LinkedList<String>

Also, per our discussion, I improved parsing of netgroup output, previously supported format:

group ( , user, ) ...

Currently supported format (both domain and host are optional, not used if present):

group ( , user, ) ( domain, anotherUser, host.com)


> Add ability to get groups for ACLs from 'getent netgroup'
> ---------------------------------------------------------
>
>                 Key: HADOOP-6855
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6855
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6855-0.20-1.patch, HADOOP-6855-0.20.patch
>
>
> Add ability to specify netgroups in ACLs (see class AccessControlList.java). Membership of users in netgroups will be determined by running 'getent negroups $groupName'. Netgroups will be differentiated from regular unix groups by having '@' as a first character.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9487-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 11:16:21 2010
Return-Path: <common-issues-return-9487-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52994 invoked from network); 14 Jul 2010 11:16:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 11:16:21 -0000
Received: (qmail 70513 invoked by uid 500); 14 Jul 2010 11:16:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70375 invoked by uid 500); 14 Jul 2010 11:16:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70348 invoked by uid 99); 14 Jul 2010 11:16:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 11:16:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 11:16:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6EBFp6T004526
	for <common-issues@hadoop.apache.org>; Wed, 14 Jul 2010 11:15:52 GMT
Message-ID: <23568832.372291279106151878.JavaMail.jira@thor>
Date: Wed, 14 Jul 2010 07:15:51 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6647) balancer fails with "is not
 authorized for protocol interface NamenodeProtocol" in secure environment
In-Reply-To: <387797205.373021269030508879.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888289#action_12888289 ] 

Hudson commented on HADOOP-6647:
--------------------------------

Integrated in Hadoop-Common-trunk #392 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/392/])
    HADOOP-6647. balancer fails with "is not authorized for protocol interface NamenodeProtocol" in secure environment


> balancer fails with "is not authorized for protocol interface NamenodeProtocol" in secure environment
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6647
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6647
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6647-BP20.patch, HADOOP-6647.patch
>
>
> user logs in as hdfs/something@something and tries to run balancer.
> balancer is using NameNode Protocol which authorizes based on server principal key.
> but NameNode key is hdfs/_HOST@.. now. so it fails. 
> To fix we need to compare the short names only.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9488-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 11:16:22 2010
Return-Path: <common-issues-return-9488-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53039 invoked from network); 14 Jul 2010 11:16:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 11:16:22 -0000
Received: (qmail 70571 invoked by uid 500); 14 Jul 2010 11:16:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70386 invoked by uid 500); 14 Jul 2010 11:16:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70367 invoked by uid 99); 14 Jul 2010 11:16:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 11:16:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 11:16:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6EBFrXa004550
	for <common-issues@hadoop.apache.org>; Wed, 14 Jul 2010 11:15:53 GMT
Message-ID: <1267816.372371279106153473.JavaMail.jira@thor>
Date: Wed, 14 Jul 2010 07:15:53 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6648) Credentials should ignore null
 tokens
In-Reply-To: <626471334.379531269043887761.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888291#action_12888291 ] 

Hudson commented on HADOOP-6648:
--------------------------------

Integrated in Hadoop-Common-trunk #392 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/392/])
    HADOOP-6648. Adds a check for null tokens in Credentials.addToken api. Contributed by Owen O'Malley and Devaraj Das.


> Credentials should ignore null tokens
> -------------------------------------
>
>                 Key: HADOOP-6648
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6648
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: c-6648-y20.patch, h-6648-trunk.patch
>
>
> When hftp goes to a non-secure cluster, getDelegationToken returns null. Credentials.addToken needs to ignore null tokens to avoid a NPE in serialization.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9489-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 11:16:23 2010
Return-Path: <common-issues-return-9489-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53057 invoked from network); 14 Jul 2010 11:16:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 11:16:22 -0000
Received: (qmail 70744 invoked by uid 500); 14 Jul 2010 11:16:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70573 invoked by uid 500); 14 Jul 2010 11:16:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70376 invoked by uid 99); 14 Jul 2010 11:16:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 11:16:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 11:16:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6EBFsTg004565
	for <common-issues@hadoop.apache.org>; Wed, 14 Jul 2010 11:15:54 GMT
Message-ID: <15032694.372421279106154774.JavaMail.jira@thor>
Date: Wed, 14 Jul 2010 07:15:54 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6825) FileStatus needs unit tests
In-Reply-To: <28758544.71131276463653554.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888292#action_12888292 ] 

Hudson commented on HADOOP-6825:
--------------------------------

Integrated in Hadoop-Common-trunk #392 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/392/])
    HADOOP-6825. FileStatus needs unit tests. Contributed by Rodrigo Schmidt.


> FileStatus needs unit tests
> ---------------------------
>
>                 Key: HADOOP-6825
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6825
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6825.0.patch
>
>
> We need some unit tests for FileStatus to prevent problems like those we recently had on HADOOP-6796 and MAPREDUCE-1858.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9490-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 11:23:50 2010
Return-Path: <common-issues-return-9490-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58124 invoked from network); 14 Jul 2010 11:23:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 11:23:50 -0000
Received: (qmail 81614 invoked by uid 500); 14 Jul 2010 11:23:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81487 invoked by uid 500); 14 Jul 2010 11:23:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81459 invoked by uid 99); 14 Jul 2010 11:23:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 11:23:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 11:23:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6EBFrcK004541
	for <common-issues@hadoop.apache.org>; Wed, 14 Jul 2010 11:15:53 GMT
Message-ID: <3673971.372341279106153024.JavaMail.jira@thor>
Date: Wed, 14 Jul 2010 07:15:53 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6819) [Herriot] Shell command for getting
 the new exceptions in the logs returning exitcode 1 after executing
 successfully.
In-Reply-To: <25211660.39321276233134711.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888290#action_12888290 ] 

Hudson commented on HADOOP-6819:
--------------------------------

Integrated in Hadoop-Common-trunk #392 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/392/])
    HADOOP-6819. [Herriot] Shell command for getting the new exceptions in
the logs returning exitcode 1 after executing successfully. Contributed
by Vinay Thota.


> [Herriot] Shell command for getting the new exceptions in the logs returning exitcode 1 after executing successfully.
> ---------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6819
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6819
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>             Fix For: 0.21.0
>
>         Attachments: 6819-ydist-security.patch, HADOOP-6819.patch
>
>
> I have found an corner case issue while executing the shell command for getting the new exceptions in the logs. The problem could see whenever the log files are empty and trying to get the new exceptions count, it returns the output is 0 ,however it returns the exitcode is 1 instead of 0. grep -vc in the command is not  giving the expected exitcode because its not handling the multiple in options.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9491-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 11:51:19 2010
Return-Path: <common-issues-return-9491-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71561 invoked from network); 14 Jul 2010 11:51:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 11:51:18 -0000
Received: (qmail 13024 invoked by uid 500); 14 Jul 2010 11:51:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12887 invoked by uid 500); 14 Jul 2010 11:51:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12872 invoked by uid 99); 14 Jul 2010 11:51:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 11:51:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 11:51:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6EBooYl004801
	for <common-issues@hadoop.apache.org>; Wed, 14 Jul 2010 11:50:51 GMT
Message-ID: <4332458.372791279108250910.JavaMail.jira@thor>
Date: Wed, 14 Jul 2010 07:50:50 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6836) [Herriot] Test Autoamtion : Generic
 method for adding/modifying the attributes for new configuration.
In-Reply-To: <23644117.9641277269730469.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888302#action_12888302 ] 

Vinay Kumar Thota commented on HADOOP-6836:
-------------------------------------------

I can see there are 6 javadoc warnings and all of them are related to warning: sun.security.krb5.Config.So, I don't think the patch could raise the number of java doc warning considering the scope of the patch.


> [Herriot] Test Autoamtion : Generic method for adding/modifying the attributes for new configuration.
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6836
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6836
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Iyappan Srinivasan
>            Assignee: Vinay Kumar Thota
>         Attachments: 6836-ydist-security.patch, HADOOP-6836.patch
>
>
> HADOP-6772 deals with 
> Common utilities for system tests.
> 1. A method for restarting the daemon with new configuration.
> c throws Exception; 
> 2. A method for restarting the daemon with default configuration.
> public void restart() throws Exception;
> 3. A method for waiting until daemon is stop.
> public void waitForClusterToStop() throws Exception;
> In this  some variables are made of String, instead of Long. Those needs ot be changed too. So, can this method 
> "public static void restartClusterWithNewConfig(Hashtable<String,Long> props, String confFile) throws Exception;"
> be generalized to accepts string too. All otehr methods should work as usual.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9492-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 12:05:18 2010
Return-Path: <common-issues-return-9492-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98891 invoked from network); 14 Jul 2010 12:05:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 12:05:18 -0000
Received: (qmail 38635 invoked by uid 500); 14 Jul 2010 12:05:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38505 invoked by uid 500); 14 Jul 2010 12:05:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38494 invoked by uid 99); 14 Jul 2010 12:05:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 12:05:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 12:05:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6EC4oo8005040
	for <common-issues@hadoop.apache.org>; Wed, 14 Jul 2010 12:04:51 GMT
Message-ID: <11426841.373251279109090883.JavaMail.jira@thor>
Date: Wed, 14 Jul 2010 08:04:50 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6839) [Herriot] Implement a functionality
 for getting the user list for creating proxy users.
In-Reply-To: <26284266.54901277454290297.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinay Kumar Thota updated HADOOP-6839:
--------------------------------------

    Attachment: HADOOP-6839.patch

Addressed the cos comments and attached latest patch for trunk.

> [Herriot] Implement a functionality for getting the user list for creating proxy users.
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6839
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6839
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6839-ydist-security.patch, 6839-ydist-security.patch, HADOOP-6839.patch, HADOOP-6839.patch, HADOOP-6839.patch
>
>
> Develop a new method for getting the user list. 
> Method signature is 
> public ArrayList<String> getHadoopMultiUsersList() throws IOException;
> Add new attribute in system-test.xml  file for getting userlist path.
> For submitting the jobs as different user a proxy user id is needed. So,get the available users from a userlist and then pass the user as proxy instead of hardcoding user id in a test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9493-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 12:12:49 2010
Return-Path: <common-issues-return-9493-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8494 invoked from network); 14 Jul 2010 12:12:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 12:12:49 -0000
Received: (qmail 47574 invoked by uid 500); 14 Jul 2010 12:12:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47256 invoked by uid 500); 14 Jul 2010 12:12:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47244 invoked by uid 99); 14 Jul 2010 12:12:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 12:12:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 12:12:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6EC4qEL005073
	for <common-issues@hadoop.apache.org>; Wed, 14 Jul 2010 12:04:52 GMT
Message-ID: <25729924.373341279109092840.JavaMail.jira@thor>
Date: Wed, 14 Jul 2010 08:04:52 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6839) [Herriot] Implement a functionality
 for getting the user list for creating proxy users.
In-Reply-To: <26284266.54901277454290297.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888310#action_12888310 ] 

Vinay Kumar Thota commented on HADOOP-6839:
-------------------------------------------

I can see there are 6 javadoc warnings and all of them are related to warning: sun.security.krb5.Config. So, I don't think the patch could raise the number of java doc warning considering the scope of the patch.


> [Herriot] Implement a functionality for getting the user list for creating proxy users.
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6839
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6839
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6839-ydist-security.patch, 6839-ydist-security.patch, HADOOP-6839.patch, HADOOP-6839.patch, HADOOP-6839.patch
>
>
> Develop a new method for getting the user list. 
> Method signature is 
> public ArrayList<String> getHadoopMultiUsersList() throws IOException;
> Add new attribute in system-test.xml  file for getting userlist path.
> For submitting the jobs as different user a proxy user id is needed. So,get the available users from a userlist and then pass the user as proxy instead of hardcoding user id in a test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9494-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 17:11:15 2010
Return-Path: <common-issues-return-9494-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27878 invoked from network); 14 Jul 2010 17:11:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 17:11:15 -0000
Received: (qmail 16261 invoked by uid 500); 14 Jul 2010 17:11:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16143 invoked by uid 500); 14 Jul 2010 17:11:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16135 invoked by uid 99); 14 Jul 2010 17:11:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 17:11:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 17:11:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6EHAohi008200
	for <common-issues@hadoop.apache.org>; Wed, 14 Jul 2010 17:10:50 GMT
Message-ID: <13502327.378481279127450433.JavaMail.jira@thor>
Date: Wed, 14 Jul 2010 13:10:50 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6855) Add ability to get groups for ACLs
 from 'getent netgroup'
In-Reply-To: <23521712.275021278635570824.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6855:
-----------------------------------

    Hadoop Flags: [Reviewed]

+1

small nit, there was one more commented line which is not needed (// netgroupToUsersMap.put(group, users); )

> Add ability to get groups for ACLs from 'getent netgroup'
> ---------------------------------------------------------
>
>                 Key: HADOOP-6855
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6855
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6855-0.20-1.patch, HADOOP-6855-0.20.patch
>
>
> Add ability to specify netgroups in ACLs (see class AccessControlList.java). Membership of users in netgroups will be determined by running 'getent negroups $groupName'. Netgroups will be differentiated from regular unix groups by having '@' as a first character.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9495-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 18:07:17 2010
Return-Path: <common-issues-return-9495-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 618 invoked from network); 14 Jul 2010 18:07:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 18:07:17 -0000
Received: (qmail 4802 invoked by uid 500); 14 Jul 2010 18:07:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4741 invoked by uid 500); 14 Jul 2010 18:07:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4733 invoked by uid 99); 14 Jul 2010 18:07:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 18:07:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 18:07:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6EI6qhM008872
	for <common-issues@hadoop.apache.org>; Wed, 14 Jul 2010 18:06:52 GMT
Message-ID: <7915557.379601279130812573.JavaMail.jira@thor>
Date: Wed, 14 Jul 2010 14:06:52 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6839) [Herriot] Implement a functionality
 for getting the user list for creating proxy users.
In-Reply-To: <26284266.54901277454290297.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888463#action_12888463 ] 

Konstantin Boudnik commented on HADOOP-6839:
--------------------------------------------

+1 patch looks good. I'll commit it by COB today if hear no objections.

> [Herriot] Implement a functionality for getting the user list for creating proxy users.
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6839
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6839
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6839-ydist-security.patch, 6839-ydist-security.patch, HADOOP-6839.patch, HADOOP-6839.patch, HADOOP-6839.patch
>
>
> Develop a new method for getting the user list. 
> Method signature is 
> public ArrayList<String> getHadoopMultiUsersList() throws IOException;
> Add new attribute in system-test.xml  file for getting userlist path.
> For submitting the jobs as different user a proxy user id is needed. So,get the available users from a userlist and then pass the user as proxy instead of hardcoding user id in a test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9496-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 18:25:22 2010
Return-Path: <common-issues-return-9496-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53449 invoked from network); 14 Jul 2010 18:25:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 18:25:22 -0000
Received: (qmail 35043 invoked by uid 500); 14 Jul 2010 18:25:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34990 invoked by uid 500); 14 Jul 2010 18:25:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34776 invoked by uid 99); 14 Jul 2010 18:25:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 18:25:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 18:25:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6EIOvvL009108
	for <common-issues@hadoop.apache.org>; Wed, 14 Jul 2010 18:24:57 GMT
Message-ID: <22401780.380201279131897262.JavaMail.jira@thor>
Date: Wed, 14 Jul 2010 14:24:57 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6836) [Herriot] Test Autoamtion : Generic
 method for adding/modifying the attributes for new configuration.
In-Reply-To: <23644117.9641277269730469.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888475#action_12888475 ] 

Konstantin Boudnik commented on HADOOP-6836:
--------------------------------------------

+1 patch looks good. The warnings are apparently unrelated. Will commit it shortly.

> [Herriot] Test Autoamtion : Generic method for adding/modifying the attributes for new configuration.
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6836
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6836
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Iyappan Srinivasan
>            Assignee: Vinay Kumar Thota
>         Attachments: 6836-ydist-security.patch, HADOOP-6836.patch
>
>
> HADOP-6772 deals with 
> Common utilities for system tests.
> 1. A method for restarting the daemon with new configuration.
> c throws Exception; 
> 2. A method for restarting the daemon with default configuration.
> public void restart() throws Exception;
> 3. A method for waiting until daemon is stop.
> public void waitForClusterToStop() throws Exception;
> In this  some variables are made of String, instead of Long. Those needs ot be changed too. So, can this method 
> "public static void restartClusterWithNewConfig(Hashtable<String,Long> props, String confFile) throws Exception;"
> be generalized to accepts string too. All otehr methods should work as usual.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9497-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 18:29:23 2010
Return-Path: <common-issues-return-9497-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64421 invoked from network); 14 Jul 2010 18:29:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 18:29:23 -0000
Received: (qmail 44093 invoked by uid 500); 14 Jul 2010 18:29:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44038 invoked by uid 500); 14 Jul 2010 18:29:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43972 invoked by uid 99); 14 Jul 2010 18:29:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 18:29:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 18:29:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6EISwuV009167
	for <common-issues@hadoop.apache.org>; Wed, 14 Jul 2010 18:28:58 GMT
Message-ID: <6165445.380291279132138267.JavaMail.jira@thor>
Date: Wed, 14 Jul 2010 14:28:58 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6855) Add ability to get groups for ACLs
 from 'getent netgroup'
In-Reply-To: <23521712.275021278635570824.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Erik Steffl updated HADOOP-6855:
--------------------------------

    Attachment: HADOOP-6855-0.20-2.patch

> Add ability to get groups for ACLs from 'getent netgroup'
> ---------------------------------------------------------
>
>                 Key: HADOOP-6855
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6855
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6855-0.20-1.patch, HADOOP-6855-0.20-2.patch, HADOOP-6855-0.20.patch
>
>
> Add ability to specify netgroups in ACLs (see class AccessControlList.java). Membership of users in netgroups will be determined by running 'getent negroups $groupName'. Netgroups will be differentiated from regular unix groups by having '@' as a first character.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9498-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 18:29:24 2010
Return-Path: <common-issues-return-9498-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64468 invoked from network); 14 Jul 2010 18:29:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 18:29:23 -0000
Received: (qmail 44222 invoked by uid 500); 14 Jul 2010 18:29:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44048 invoked by uid 500); 14 Jul 2010 18:29:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44039 invoked by uid 99); 14 Jul 2010 18:29:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 18:29:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 18:29:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6EISx6Z009182
	for <common-issues@hadoop.apache.org>; Wed, 14 Jul 2010 18:28:59 GMT
Message-ID: <7094324.380341279132139172.JavaMail.jira@thor>
Date: Wed, 14 Jul 2010 14:28:59 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6855) Add ability to get groups for ACLs
 from 'getent netgroup'
In-Reply-To: <23521712.275021278635570824.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888476#action_12888476 ] 

Erik Steffl commented on HADOOP-6855:
-------------------------------------

Patch HADOOP-6855-0.20-2.patch removes the bogus comment (// netgroupToUsersMap.put(group, users); )

> Add ability to get groups for ACLs from 'getent netgroup'
> ---------------------------------------------------------
>
>                 Key: HADOOP-6855
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6855
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6855-0.20-1.patch, HADOOP-6855-0.20-2.patch, HADOOP-6855-0.20.patch
>
>
> Add ability to specify netgroups in ACLs (see class AccessControlList.java). Membership of users in netgroups will be determined by running 'getent negroups $groupName'. Netgroups will be differentiated from regular unix groups by having '@' as a first character.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9499-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 18:34:17 2010
Return-Path: <common-issues-return-9499-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76041 invoked from network); 14 Jul 2010 18:34:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 18:34:17 -0000
Received: (qmail 52624 invoked by uid 500); 14 Jul 2010 18:34:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52537 invoked by uid 500); 14 Jul 2010 18:34:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52527 invoked by uid 99); 14 Jul 2010 18:34:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 18:34:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 18:34:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6EIXq3t009231
	for <common-issues@hadoop.apache.org>; Wed, 14 Jul 2010 18:33:52 GMT
Message-ID: <14794126.380461279132432379.JavaMail.jira@thor>
Date: Wed, 14 Jul 2010 14:33:52 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6860)  'compile-fault-inject' should never
 be called directly.
In-Reply-To: <25103470.380431279132431950.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6860:
---------------------------------------

    Attachment: HADOOP-6860.patch

The fix.

>  'compile-fault-inject' should never be called directly.
> --------------------------------------------------------
>
>                 Key: HADOOP-6860
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6860
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>            Priority: Minor
>         Attachments: HADOOP-6860.patch
>
>
> Similar to HDFS-1299 a  helper target  'compile-fault-inject' should never be called directly.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9500-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 18:41:46 2010
Return-Path: <common-issues-return-9500-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94427 invoked from network); 14 Jul 2010 18:41:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 18:41:46 -0000
Received: (qmail 62752 invoked by uid 500); 14 Jul 2010 18:41:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62707 invoked by uid 500); 14 Jul 2010 18:41:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62697 invoked by uid 99); 14 Jul 2010 18:41:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 18:41:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 18:41:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6EIXpMD009222
	for <common-issues@hadoop.apache.org>; Wed, 14 Jul 2010 18:33:51 GMT
Message-ID: <25103470.380431279132431950.JavaMail.jira@thor>
Date: Wed, 14 Jul 2010 14:33:51 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6860)  'compile-fault-inject' should never
 be called directly.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

 'compile-fault-inject' should never be called directly.
--------------------------------------------------------

                 Key: HADOOP-6860
                 URL: https://issues.apache.org/jira/browse/HADOOP-6860
             Project: Hadoop Common
          Issue Type: Bug
          Components: build
    Affects Versions: 0.21.0
            Reporter: Konstantin Boudnik
            Assignee: Konstantin Boudnik
            Priority: Minor
         Attachments: HADOOP-6860.patch

Similar to HDFS-1299 a  helper target  'compile-fault-inject' should never be called directly.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9501-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 18:48:18 2010
Return-Path: <common-issues-return-9501-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6651 invoked from network); 14 Jul 2010 18:48:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 18:48:18 -0000
Received: (qmail 75629 invoked by uid 500); 14 Jul 2010 18:48:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75584 invoked by uid 500); 14 Jul 2010 18:48:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75575 invoked by uid 99); 14 Jul 2010 18:48:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 18:48:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 18:48:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6EIlrUG009341
	for <common-issues@hadoop.apache.org>; Wed, 14 Jul 2010 18:47:53 GMT
Message-ID: <16004573.380701279133273333.JavaMail.jira@thor>
Date: Wed, 14 Jul 2010 14:47:53 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6839) [Herriot] Implement a functionality
 for getting the user list for creating proxy users.
In-Reply-To: <26284266.54901277454290297.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6839:
---------------------------------------

           Status: Resolved  (was: Patch Available)
    Fix Version/s: 0.21.0
       Resolution: Fixed

I have just committed this. Thanks Vinay.

> [Herriot] Implement a functionality for getting the user list for creating proxy users.
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6839
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6839
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>             Fix For: 0.21.0
>
>         Attachments: 6839-ydist-security.patch, 6839-ydist-security.patch, HADOOP-6839.patch, HADOOP-6839.patch, HADOOP-6839.patch
>
>
> Develop a new method for getting the user list. 
> Method signature is 
> public ArrayList<String> getHadoopMultiUsersList() throws IOException;
> Add new attribute in system-test.xml  file for getting userlist path.
> For submitting the jobs as different user a proxy user id is needed. So,get the available users from a userlist and then pass the user as proxy instead of hardcoding user id in a test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9502-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 18:51:16 2010
Return-Path: <common-issues-return-9502-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10946 invoked from network); 14 Jul 2010 18:51:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 18:51:16 -0000
Received: (qmail 83247 invoked by uid 500); 14 Jul 2010 18:51:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83147 invoked by uid 500); 14 Jul 2010 18:51:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83139 invoked by uid 99); 14 Jul 2010 18:51:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 18:51:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 18:51:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6EIoouL009370
	for <common-issues@hadoop.apache.org>; Wed, 14 Jul 2010 18:50:51 GMT
Message-ID: <17043629.380761279133450795.JavaMail.jira@thor>
Date: Wed, 14 Jul 2010 14:50:50 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6836) [Herriot]: Generic method for
 adding/modifying the attributes for new configuration.
In-Reply-To: <23644117.9641277269730469.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6836:
---------------------------------------

    Summary: [Herriot]: Generic method for adding/modifying the attributes for new configuration.  (was: [Herriot] Test Autoamtion : Generic method for adding/modifying the attributes for new configuration.)

> [Herriot]: Generic method for adding/modifying the attributes for new configuration.
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6836
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6836
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Iyappan Srinivasan
>            Assignee: Vinay Kumar Thota
>             Fix For: 0.21.0
>
>         Attachments: 6836-ydist-security.patch, HADOOP-6836.patch
>
>
> HADOP-6772 deals with 
> Common utilities for system tests.
> 1. A method for restarting the daemon with new configuration.
> c throws Exception; 
> 2. A method for restarting the daemon with default configuration.
> public void restart() throws Exception;
> 3. A method for waiting until daemon is stop.
> public void waitForClusterToStop() throws Exception;
> In this  some variables are made of String, instead of Long. Those needs ot be changed too. So, can this method 
> "public static void restartClusterWithNewConfig(Hashtable<String,Long> props, String confFile) throws Exception;"
> be generalized to accepts string too. All otehr methods should work as usual.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9503-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 18:51:18 2010
Return-Path: <common-issues-return-9503-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11054 invoked from network); 14 Jul 2010 18:51:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 18:51:18 -0000
Received: (qmail 83950 invoked by uid 500); 14 Jul 2010 18:51:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83905 invoked by uid 500); 14 Jul 2010 18:51:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83897 invoked by uid 99); 14 Jul 2010 18:51:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 18:51:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 18:51:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6EIormI009397
	for <common-issues@hadoop.apache.org>; Wed, 14 Jul 2010 18:50:53 GMT
Message-ID: <6898984.380851279133452987.JavaMail.jira@thor>
Date: Wed, 14 Jul 2010 14:50:52 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6836) [Herriot]: Generic method for
 adding/modifying the attributes for new configuration.
In-Reply-To: <23644117.9641277269730469.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6836:
---------------------------------------

           Status: Resolved  (was: Patch Available)
    Fix Version/s: 0.21.0
       Resolution: Fixed

I have committed this. Thanks Vinay.

> [Herriot]: Generic method for adding/modifying the attributes for new configuration.
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6836
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6836
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Iyappan Srinivasan
>            Assignee: Vinay Kumar Thota
>             Fix For: 0.21.0
>
>         Attachments: 6836-ydist-security.patch, HADOOP-6836.patch
>
>
> HADOP-6772 deals with 
> Common utilities for system tests.
> 1. A method for restarting the daemon with new configuration.
> c throws Exception; 
> 2. A method for restarting the daemon with default configuration.
> public void restart() throws Exception;
> 3. A method for waiting until daemon is stop.
> public void waitForClusterToStop() throws Exception;
> In this  some variables are made of String, instead of Long. Those needs ot be changed too. So, can this method 
> "public static void restartClusterWithNewConfig(Hashtable<String,Long> props, String confFile) throws Exception;"
> be generalized to accepts string too. All otehr methods should work as usual.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9504-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 19:06:18 2010
Return-Path: <common-issues-return-9504-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20872 invoked from network); 14 Jul 2010 19:06:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 19:06:17 -0000
Received: (qmail 255 invoked by uid 500); 14 Jul 2010 19:06:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 206 invoked by uid 500); 14 Jul 2010 19:06:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 198 invoked by uid 99); 14 Jul 2010 19:06:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 19:06:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 19:06:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6EJ5rWG009560
	for <common-issues@hadoop.apache.org>; Wed, 14 Jul 2010 19:05:53 GMT
Message-ID: <2376821.381061279134352986.JavaMail.jira@thor>
Date: Wed, 14 Jul 2010 15:05:52 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6836) [Herriot]: Generic method for
 adding/modifying the attributes for new configuration.
In-Reply-To: <23644117.9641277269730469.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888489#action_12888489 ] 

Hudson commented on HADOOP-6836:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #325 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/325/])
    HADOOP-6836. [Herriot]: Generic method for adding/modifying the attributes for new configuration. Contributed by Vinay Thota.


> [Herriot]: Generic method for adding/modifying the attributes for new configuration.
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6836
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6836
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Iyappan Srinivasan
>            Assignee: Vinay Kumar Thota
>             Fix For: 0.21.0
>
>         Attachments: 6836-ydist-security.patch, HADOOP-6836.patch
>
>
> HADOP-6772 deals with 
> Common utilities for system tests.
> 1. A method for restarting the daemon with new configuration.
> c throws Exception; 
> 2. A method for restarting the daemon with default configuration.
> public void restart() throws Exception;
> 3. A method for waiting until daemon is stop.
> public void waitForClusterToStop() throws Exception;
> In this  some variables are made of String, instead of Long. Those needs ot be changed too. So, can this method 
> "public static void restartClusterWithNewConfig(Hashtable<String,Long> props, String confFile) throws Exception;"
> be generalized to accepts string too. All otehr methods should work as usual.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9505-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 19:13:46 2010
Return-Path: <common-issues-return-9505-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23911 invoked from network); 14 Jul 2010 19:13:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 19:13:46 -0000
Received: (qmail 11284 invoked by uid 500); 14 Jul 2010 19:13:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11136 invoked by uid 500); 14 Jul 2010 19:13:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11055 invoked by uid 99); 14 Jul 2010 19:13:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 19:13:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 19:13:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6EJ5qk9009542
	for <common-issues@hadoop.apache.org>; Wed, 14 Jul 2010 19:05:52 GMT
Message-ID: <6917357.381001279134352090.JavaMail.jira@thor>
Date: Wed, 14 Jul 2010 15:05:52 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6839) [Herriot] Implement a functionality
 for getting the user list for creating proxy users.
In-Reply-To: <26284266.54901277454290297.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888488#action_12888488 ] 

Hudson commented on HADOOP-6839:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #325 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/325/])
    HADOOP-6839. [Herriot] Implement a functionality for getting the user list for creating proxy users. Contributed by Vinay Thota.


> [Herriot] Implement a functionality for getting the user list for creating proxy users.
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6839
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6839
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>             Fix For: 0.21.0
>
>         Attachments: 6839-ydist-security.patch, 6839-ydist-security.patch, HADOOP-6839.patch, HADOOP-6839.patch, HADOOP-6839.patch
>
>
> Develop a new method for getting the user list. 
> Method signature is 
> public ArrayList<String> getHadoopMultiUsersList() throws IOException;
> Add new attribute in system-test.xml  file for getting userlist path.
> For submitting the jobs as different user a proxy user id is needed. So,get the available users from a userlist and then pass the user as proxy instead of hardcoding user id in a test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9506-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 20:12:21 2010
Return-Path: <common-issues-return-9506-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52018 invoked from network); 14 Jul 2010 20:12:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 20:12:21 -0000
Received: (qmail 64165 invoked by uid 500); 14 Jul 2010 20:12:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63753 invoked by uid 500); 14 Jul 2010 20:12:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63744 invoked by uid 99); 14 Jul 2010 20:12:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 20:12:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 20:12:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6EKBtNZ010188
	for <common-issues@hadoop.apache.org>; Wed, 14 Jul 2010 20:11:55 GMT
Message-ID: <2821515.382131279138315526.JavaMail.jira@thor>
Date: Wed, 14 Jul 2010 16:11:55 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6670) UserGroupInformation doesn't
 support use in hash tables
In-Reply-To: <242202762.620161270078590059.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888513#action_12888513 ] 

Jitendra Nath Pandey commented on HADOOP-6670:
----------------------------------------------

In TestUserGroupInformation.java there is a repeated check for equality of two UGIs with same subject which can be removed.

> UserGroupInformation doesn't support use in hash tables
> -------------------------------------------------------
>
>                 Key: HADOOP-6670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6670
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 6670-gridmix-test.bugfix.patch, c6670-02.patch, fs-close.patch, fs-close.patch, fs-close.patch
>
>
> The UserGroupInformation objects are mutable, but they are used as keys in hash tables. This leads to serious problems in the FileSystem cache and RPC connection cache. We need to change the hashCode to be the identity hash code of the Subject and change equals to use == between the Subjects.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9507-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 21:27:16 2010
Return-Path: <common-issues-return-9507-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39478 invoked from network); 14 Jul 2010 21:27:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 21:27:16 -0000
Received: (qmail 77796 invoked by uid 500); 14 Jul 2010 21:27:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77739 invoked by uid 500); 14 Jul 2010 21:27:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77728 invoked by uid 99); 14 Jul 2010 21:27:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 21:27:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 21:27:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6ELQp1N010954
	for <common-issues@hadoop.apache.org>; Wed, 14 Jul 2010 21:26:51 GMT
Message-ID: <25998346.383631279142811429.JavaMail.jira@thor>
Date: Wed, 14 Jul 2010 17:26:51 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6860)  'compile-fault-inject' should
 never be called directly.
In-Reply-To: <25103470.380431279132431950.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888547#action_12888547 ] 

Eli Collins commented on HADOOP-6860:
-------------------------------------

+1

>  'compile-fault-inject' should never be called directly.
> --------------------------------------------------------
>
>                 Key: HADOOP-6860
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6860
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>            Priority: Minor
>         Attachments: HADOOP-6860.patch
>
>
> Similar to HDFS-1299 a  helper target  'compile-fault-inject' should never be called directly.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9508-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 22:15:46 2010
Return-Path: <common-issues-return-9508-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9060 invoked from network); 14 Jul 2010 22:15:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 22:15:46 -0000
Received: (qmail 48917 invoked by uid 500); 14 Jul 2010 22:15:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48843 invoked by uid 500); 14 Jul 2010 22:15:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48832 invoked by uid 99); 14 Jul 2010 22:15:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 22:15:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 22:15:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6EM7oSB011484
	for <common-issues@hadoop.apache.org>; Wed, 14 Jul 2010 22:07:50 GMT
Message-ID: <21462038.384681279145270696.JavaMail.jira@thor>
Date: Wed, 14 Jul 2010 18:07:50 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6861) Method in Credentials to read and
 write a token storage file.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Method in Credentials to read and write a token storage file.
-------------------------------------------------------------

                 Key: HADOOP-6861
                 URL: https://issues.apache.org/jira/browse/HADOOP-6861
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Jitendra Nath Pandey
            Assignee: Jitendra Nath Pandey
         Attachments: HADOOP-6861.1.patch

The jira covers the changes in common corresponding to MAPREDUCE-1566. 

This jira adds  new non-static methods in Credentials to read and write token storage file. A method to copy tokens from another credential object is also added. Static method readTokensAndLoadInUGI is removed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9510-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 22:15:49 2010
Return-Path: <common-issues-return-9510-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9202 invoked from network); 14 Jul 2010 22:15:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 22:15:47 -0000
Received: (qmail 49774 invoked by uid 500); 14 Jul 2010 22:15:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49685 invoked by uid 500); 14 Jul 2010 22:15:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49197 invoked by uid 99); 14 Jul 2010 22:15:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 22:15:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 22:15:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6EM7rdH011508
	for <common-issues@hadoop.apache.org>; Wed, 14 Jul 2010 22:07:53 GMT
Message-ID: <20910932.384761279145273615.JavaMail.jira@thor>
Date: Wed, 14 Jul 2010 18:07:53 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6861) Method in Credentials to read and
 write a token storage file.
In-Reply-To: <21462038.384681279145270696.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6861:
-----------------------------------------

    Status: Patch Available  (was: Open)

> Method in Credentials to read and write a token storage file.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6861
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6861
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6861.1.patch
>
>
> The jira covers the changes in common corresponding to MAPREDUCE-1566. 
> This jira adds  new non-static methods in Credentials to read and write token storage file. A method to copy tokens from another credential object is also added. Static method readTokensAndLoadInUGI is removed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9509-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 22:15:49 2010
Return-Path: <common-issues-return-9509-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9182 invoked from network); 14 Jul 2010 22:15:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 22:15:47 -0000
Received: (qmail 49709 invoked by uid 500); 14 Jul 2010 22:15:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49027 invoked by uid 500); 14 Jul 2010 22:15:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49019 invoked by uid 99); 14 Jul 2010 22:15:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 22:15:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 22:15:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6EM7r35011502
	for <common-issues@hadoop.apache.org>; Wed, 14 Jul 2010 22:07:53 GMT
Message-ID: <7994266.384741279145273207.JavaMail.jira@thor>
Date: Wed, 14 Jul 2010 18:07:53 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6861) Method in Credentials to read and
 write a token storage file.
In-Reply-To: <21462038.384681279145270696.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6861:
-----------------------------------------

    Attachment: HADOOP-6861.1.patch

Patch uploaded.

> Method in Credentials to read and write a token storage file.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6861
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6861
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6861.1.patch
>
>
> The jira covers the changes in common corresponding to MAPREDUCE-1566. 
> This jira adds  new non-static methods in Credentials to read and write token storage file. A method to copy tokens from another credential object is also added. Static method readTokensAndLoadInUGI is removed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9511-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 14 22:17:14 2010
Return-Path: <common-issues-return-9511-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9769 invoked from network); 14 Jul 2010 22:17:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jul 2010 22:17:13 -0000
Received: (qmail 53637 invoked by uid 500); 14 Jul 2010 22:17:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53615 invoked by uid 500); 14 Jul 2010 22:17:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53607 invoked by uid 99); 14 Jul 2010 22:17:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 22:17:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 22:17:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6EMGnOI011573
	for <common-issues@hadoop.apache.org>; Wed, 14 Jul 2010 22:16:49 GMT
Message-ID: <20657979.384871279145809913.JavaMail.jira@thor>
Date: Wed, 14 Jul 2010 18:16:49 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6861) Method in Credentials to read and
 write a token storage file.
In-Reply-To: <21462038.384681279145270696.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6861:
-----------------------------------------

    Status: Open  (was: Patch Available)

> Method in Credentials to read and write a token storage file.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6861
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6861
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6861.1.patch
>
>
> The jira covers the changes in common corresponding to MAPREDUCE-1566. 
> This jira adds  new non-static methods in Credentials to read and write token storage file. A method to copy tokens from another credential object is also added. Static method readTokensAndLoadInUGI is removed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9512-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 01:00:47 2010
Return-Path: <common-issues-return-9512-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58732 invoked from network); 15 Jul 2010 01:00:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 01:00:46 -0000
Received: (qmail 87337 invoked by uid 500); 15 Jul 2010 01:00:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87280 invoked by uid 500); 15 Jul 2010 01:00:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87272 invoked by uid 99); 15 Jul 2010 01:00:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 01:00:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 01:00:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6F0qpTF013547
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 00:52:51 GMT
Message-ID: <19700809.388391279155171584.JavaMail.jira@thor>
Date: Wed, 14 Jul 2010 20:52:51 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6818) Provide a JNI-based implementation
 of GroupMappingServiceProvider
In-Reply-To: <17262845.29701276200793990.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888647#action_12888647 ] 

Erik Steffl commented on HADOOP-6818:
-------------------------------------

+1

> Provide a JNI-based implementation of GroupMappingServiceProvider
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6818
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6818
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: JNIGroupMapping.patch
>
>
> The default implementation of GroupMappingServiceProvider does a fork of a unix command to get the groups of a user. Since the group resolution happens in the servers, this might be costly. This jira aims at providing a JNI-based implementation for GroupMappingServiceProvider.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9513-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 01:03:48 2010
Return-Path: <common-issues-return-9513-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59452 invoked from network); 15 Jul 2010 01:03:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 01:03:48 -0000
Received: (qmail 90717 invoked by uid 500); 15 Jul 2010 01:03:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90542 invoked by uid 500); 15 Jul 2010 01:03:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90534 invoked by uid 99); 15 Jul 2010 01:03:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 01:03:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 01:03:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6F0tr5f013611
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 00:55:54 GMT
Message-ID: <1961490.388521279155353466.JavaMail.jira@thor>
Date: Wed, 14 Jul 2010 20:55:53 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6861) Method in Credentials to read and
 write a token storage file.
In-Reply-To: <21462038.384681279145270696.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6861:
-----------------------------------------

    Attachment: HADOOP-6861.2.patch

> Method in Credentials to read and write a token storage file.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6861
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6861
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6861.1.patch, HADOOP-6861.2.patch
>
>
> The jira covers the changes in common corresponding to MAPREDUCE-1566. 
> This jira adds  new non-static methods in Credentials to read and write token storage file. A method to copy tokens from another credential object is also added. Static method readTokensAndLoadInUGI is removed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9514-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 01:22:16 2010
Return-Path: <common-issues-return-9514-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66774 invoked from network); 15 Jul 2010 01:22:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 01:22:15 -0000
Received: (qmail 14345 invoked by uid 500); 15 Jul 2010 01:22:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14279 invoked by uid 500); 15 Jul 2010 01:22:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14271 invoked by uid 99); 15 Jul 2010 01:22:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 01:22:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 01:22:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6F1LoEs013794
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 01:21:50 GMT
Message-ID: <17786483.388771279156910516.JavaMail.jira@thor>
Date: Wed, 14 Jul 2010 21:21:50 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6855) Add ability to get groups for ACLs
 from 'getent netgroup'
In-Reply-To: <23521712.275021278635570824.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888656#action_12888656 ] 

Erik Steffl commented on HADOOP-6855:
-------------------------------------

Testing: ant test fails with the same errors with or without patch.

> Add ability to get groups for ACLs from 'getent netgroup'
> ---------------------------------------------------------
>
>                 Key: HADOOP-6855
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6855
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6855-0.20-1.patch, HADOOP-6855-0.20-2.patch, HADOOP-6855-0.20.patch
>
>
> Add ability to specify netgroups in ACLs (see class AccessControlList.java). Membership of users in netgroups will be determined by running 'getent negroups $groupName'. Netgroups will be differentiated from regular unix groups by having '@' as a first character.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9515-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 03:12:50 2010
Return-Path: <common-issues-return-9515-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33680 invoked from network); 15 Jul 2010 03:12:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 03:12:50 -0000
Received: (qmail 84736 invoked by uid 500); 15 Jul 2010 03:12:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84550 invoked by uid 500); 15 Jul 2010 03:12:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84530 invoked by uid 99); 15 Jul 2010 03:12:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 03:12:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 03:12:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6F34orB014550
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 03:04:51 GMT
Message-ID: <27683426.390031279163090762.JavaMail.jira@thor>
Date: Wed, 14 Jul 2010 23:04:50 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6669) zlib.compress.level  ignored for
 DefaultCodec initialization
In-Reply-To: <1208299192.618051270074867352.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-6669:
----------------------------------

    Attachment: 6669-0y20.patch

Backport to y20

> zlib.compress.level  ignored for DefaultCodec initialization 
> -------------------------------------------------------------
>
>                 Key: HADOOP-6669
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6669
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Koji Noguchi
>            Assignee: Koji Noguchi
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: 6669-0y20.patch, HADOOP-6669-0.patch
>
>
> HADOOP-5879 added a compression level for codecs, but DefaultCodec seems to ignore this conf value when initialized.
> This is only when codec is first created.  reinit() probably sets it right. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9516-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 04:21:18 2010
Return-Path: <common-issues-return-9516-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73360 invoked from network); 15 Jul 2010 04:21:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 04:21:17 -0000
Received: (qmail 25402 invoked by uid 500); 15 Jul 2010 04:21:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25275 invoked by uid 500); 15 Jul 2010 04:21:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25267 invoked by uid 99); 15 Jul 2010 04:21:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 04:21:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 04:21:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6F4KoAP015557
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 04:20:51 GMT
Message-ID: <8888330.390641279167650802.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 00:20:50 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6839) [Herriot] Implement a functionality
 for getting the user list for creating proxy users.
In-Reply-To: <26284266.54901277454290297.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinay Kumar Thota updated HADOOP-6839:
--------------------------------------

    Attachment: 6839-ydist-security.patch

> [Herriot] Implement a functionality for getting the user list for creating proxy users.
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6839
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6839
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>             Fix For: 0.21.0
>
>         Attachments: 6839-ydist-security.patch, 6839-ydist-security.patch, 6839-ydist-security.patch, HADOOP-6839.patch, HADOOP-6839.patch, HADOOP-6839.patch
>
>
> Develop a new method for getting the user list. 
> Method signature is 
> public ArrayList<String> getHadoopMultiUsersList() throws IOException;
> Add new attribute in system-test.xml  file for getting userlist path.
> For submitting the jobs as different user a proxy user id is needed. So,get the available users from a userlist and then pass the user as proxy instead of hardcoding user id in a test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9517-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 06:11:49 2010
Return-Path: <common-issues-return-9517-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48118 invoked from network); 15 Jul 2010 06:11:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 06:11:48 -0000
Received: (qmail 80196 invoked by uid 500); 15 Jul 2010 06:11:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80088 invoked by uid 500); 15 Jul 2010 06:11:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80080 invoked by uid 99); 15 Jul 2010 06:11:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 06:11:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 06:11:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6F63oEd016293
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 06:03:50 GMT
Message-ID: <6885195.391831279173830267.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 02:03:50 -0400 (EDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6862) Add api to add user/group to
 AccessControlList
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Add api to add user/group to AccessControlList
----------------------------------------------

                 Key: HADOOP-6862
                 URL: https://issues.apache.org/jira/browse/HADOOP-6862
             Project: Hadoop Common
          Issue Type: Improvement
          Components: security
            Reporter: Amareshwari Sriramadasu
            Assignee: Amareshwari Sriramadasu
             Fix For: 0.22.0


Add api addUser(String user) and addGroup(String group) to add user/group to AccessControlList

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9518-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 06:29:48 2010
Return-Path: <common-issues-return-9518-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56018 invoked from network); 15 Jul 2010 06:29:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 06:29:48 -0000
Received: (qmail 87185 invoked by uid 500); 15 Jul 2010 06:29:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87059 invoked by uid 500); 15 Jul 2010 06:29:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87039 invoked by uid 99); 15 Jul 2010 06:29:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 06:29:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 06:29:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6F6LopI016352
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 06:21:50 GMT
Message-ID: <4972502.391901279174910694.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 02:21:50 -0400 (EDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6862) Add api to add user/group to
 AccessControlList
In-Reply-To: <6885195.391831279173830267.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amareshwari Sriramadasu updated HADOOP-6862:
--------------------------------------------

    Attachment: patch-6862.txt

Patch adds addUser(String user) and addGroup(String group) to AccessControlList

> Add api to add user/group to AccessControlList
> ----------------------------------------------
>
>                 Key: HADOOP-6862
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6862
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.22.0
>
>         Attachments: patch-6862.txt
>
>
> Add api addUser(String user) and addGroup(String group) to add user/group to AccessControlList

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9519-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 06:29:49 2010
Return-Path: <common-issues-return-9519-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56033 invoked from network); 15 Jul 2010 06:29:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 06:29:49 -0000
Received: (qmail 87240 invoked by uid 500); 15 Jul 2010 06:29:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87062 invoked by uid 500); 15 Jul 2010 06:29:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87038 invoked by uid 99); 15 Jul 2010 06:29:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 06:29:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 06:29:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6F6LprK016358
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 06:21:51 GMT
Message-ID: <31109016.391921279174911199.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 02:21:51 -0400 (EDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6862) Add api to add user/group to
 AccessControlList
In-Reply-To: <6885195.391831279173830267.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amareshwari Sriramadasu updated HADOOP-6862:
--------------------------------------------

    Status: Patch Available  (was: Open)

> Add api to add user/group to AccessControlList
> ----------------------------------------------
>
>                 Key: HADOOP-6862
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6862
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.22.0
>
>         Attachments: patch-6862.txt
>
>
> Add api addUser(String user) and addGroup(String group) to add user/group to AccessControlList

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9520-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 06:58:47 2010
Return-Path: <common-issues-return-9520-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61747 invoked from network); 15 Jul 2010 06:58:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 06:58:47 -0000
Received: (qmail 10135 invoked by uid 500); 15 Jul 2010 06:58:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10008 invoked by uid 500); 15 Jul 2010 06:58:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9998 invoked by uid 99); 15 Jul 2010 06:58:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 06:58:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 06:58:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6F6on81016496
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 06:50:50 GMT
Message-ID: <28104248.392081279176649977.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 02:50:49 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6862) Add api to add user/group to
 AccessControlList
In-Reply-To: <6885195.391831279173830267.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888721#action_12888721 ] 

Hadoop QA commented on HADOOP-6862:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12449539/patch-6862.txt
  against trunk revision 964134.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/613/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/613/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/613/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/613/console

This message is automatically generated.

> Add api to add user/group to AccessControlList
> ----------------------------------------------
>
>                 Key: HADOOP-6862
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6862
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.22.0
>
>         Attachments: patch-6862.txt
>
>
> Add api addUser(String user) and addGroup(String group) to add user/group to AccessControlList

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9521-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 08:48:22 2010
Return-Path: <common-issues-return-9521-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51599 invoked from network); 15 Jul 2010 08:48:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 08:48:21 -0000
Received: (qmail 31076 invoked by uid 500); 15 Jul 2010 08:48:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30247 invoked by uid 500); 15 Jul 2010 08:48:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30223 invoked by uid 99); 15 Jul 2010 08:48:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 08:48:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 08:48:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6F8lrxK017586
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 08:47:53 GMT
Message-ID: <26117428.393121279183673000.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 04:47:52 -0400 (EDT)
From: "Ravi Gummadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6862) Add api to add user/group to
 AccessControlList
In-Reply-To: <6885195.391831279173830267.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888748#action_12888748 ] 

Ravi Gummadi commented on HADOOP-6862:
--------------------------------------

How about api for removing user/group ?

> Add api to add user/group to AccessControlList
> ----------------------------------------------
>
>                 Key: HADOOP-6862
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6862
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.22.0
>
>         Attachments: patch-6862.txt
>
>
> Add api addUser(String user) and addGroup(String group) to add user/group to AccessControlList

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9522-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 10:53:18 2010
Return-Path: <common-issues-return-9522-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36214 invoked from network); 15 Jul 2010 10:53:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 10:53:18 -0000
Received: (qmail 59890 invoked by uid 500); 15 Jul 2010 10:53:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59746 invoked by uid 500); 15 Jul 2010 10:53:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59738 invoked by uid 99); 15 Jul 2010 10:53:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 10:53:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 10:53:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6FAqoh5018440
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 10:52:50 GMT
Message-ID: <5599131.394251279191170172.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 06:52:50 -0400 (EDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6862) Add api to add user/group to
 AccessControlList
In-Reply-To: <6885195.391831279173830267.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amareshwari Sriramadasu updated HADOOP-6862:
--------------------------------------------

    Status: Open  (was: Patch Available)

> Add api to add user/group to AccessControlList
> ----------------------------------------------
>
>                 Key: HADOOP-6862
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6862
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.22.0
>
>         Attachments: patch-6862.txt
>
>
> Add api addUser(String user) and addGroup(String group) to add user/group to AccessControlList

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9524-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 11:02:48 2010
Return-Path: <common-issues-return-9524-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47014 invoked from network); 15 Jul 2010 11:02:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 11:02:47 -0000
Received: (qmail 65406 invoked by uid 500); 15 Jul 2010 11:02:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65260 invoked by uid 500); 15 Jul 2010 11:02:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65223 invoked by uid 99); 15 Jul 2010 11:02:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 11:02:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 11:02:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6FAsoh1018456
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 10:54:50 GMT
Message-ID: <10177517.394281279191290347.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 06:54:50 -0400 (EDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6862) Add api to add user/group to
 AccessControlList
In-Reply-To: <6885195.391831279173830267.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amareshwari Sriramadasu updated HADOOP-6862:
--------------------------------------------

    Attachment: patch-6862-1.txt

Adds removeUser and removeGroup api also.

> Add api to add user/group to AccessControlList
> ----------------------------------------------
>
>                 Key: HADOOP-6862
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6862
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.22.0
>
>         Attachments: patch-6862-1.txt, patch-6862.txt
>
>
> Add api addUser(String user) and addGroup(String group) to add user/group to AccessControlList

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9523-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 11:02:48 2010
Return-Path: <common-issues-return-9523-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47015 invoked from network); 15 Jul 2010 11:02:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 11:02:47 -0000
Received: (qmail 65383 invoked by uid 500); 15 Jul 2010 11:02:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65249 invoked by uid 500); 15 Jul 2010 11:02:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65224 invoked by uid 99); 15 Jul 2010 11:02:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 11:02:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 11:02:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6FAsp99018471
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 10:54:51 GMT
Message-ID: <24277939.394331279191291291.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 06:54:51 -0400 (EDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6862) Add api to add user/group to
 AccessControlList
In-Reply-To: <6885195.391831279173830267.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amareshwari Sriramadasu updated HADOOP-6862:
--------------------------------------------

    Status: Patch Available  (was: Open)

> Add api to add user/group to AccessControlList
> ----------------------------------------------
>
>                 Key: HADOOP-6862
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6862
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.22.0
>
>         Attachments: patch-6862-1.txt, patch-6862.txt
>
>
> Add api addUser(String user) and addGroup(String group) to add user/group to AccessControlList

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9525-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 11:21:49 2010
Return-Path: <common-issues-return-9525-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97369 invoked from network); 15 Jul 2010 11:21:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 11:21:49 -0000
Received: (qmail 78452 invoked by uid 500); 15 Jul 2010 11:21:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78320 invoked by uid 500); 15 Jul 2010 11:21:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78298 invoked by uid 99); 15 Jul 2010 11:21:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 11:21:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 11:21:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6FBDoG3018627
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 11:13:51 GMT
Message-ID: <23196046.394541279192430750.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 07:13:50 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6839) [Herriot] Implement a functionality
 for getting the user list for creating proxy users.
In-Reply-To: <26284266.54901277454290297.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888778#action_12888778 ] 

Hudson commented on HADOOP-6839:
--------------------------------

Integrated in Hadoop-Common-trunk #393 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/393/])
    HADOOP-6839. [Herriot] Implement a functionality for getting the user list for creating proxy users. Contributed by Vinay Thota.


> [Herriot] Implement a functionality for getting the user list for creating proxy users.
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6839
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6839
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>             Fix For: 0.21.0
>
>         Attachments: 6839-ydist-security.patch, 6839-ydist-security.patch, 6839-ydist-security.patch, HADOOP-6839.patch, HADOOP-6839.patch, HADOOP-6839.patch
>
>
> Develop a new method for getting the user list. 
> Method signature is 
> public ArrayList<String> getHadoopMultiUsersList() throws IOException;
> Add new attribute in system-test.xml  file for getting userlist path.
> For submitting the jobs as different user a proxy user id is needed. So,get the available users from a userlist and then pass the user as proxy instead of hardcoding user id in a test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9526-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 11:21:50 2010
Return-Path: <common-issues-return-9526-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97415 invoked from network); 15 Jul 2010 11:21:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 11:21:50 -0000
Received: (qmail 78495 invoked by uid 500); 15 Jul 2010 11:21:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78329 invoked by uid 500); 15 Jul 2010 11:21:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78311 invoked by uid 99); 15 Jul 2010 11:21:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 11:21:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 11:21:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6FBDqFr018645
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 11:13:52 GMT
Message-ID: <8096768.394601279192432034.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 07:13:52 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6836) [Herriot]: Generic method for
 adding/modifying the attributes for new configuration.
In-Reply-To: <23644117.9641277269730469.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888779#action_12888779 ] 

Hudson commented on HADOOP-6836:
--------------------------------

Integrated in Hadoop-Common-trunk #393 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/393/])
    HADOOP-6836. [Herriot]: Generic method for adding/modifying the attributes for new configuration. Contributed by Vinay Thota.


> [Herriot]: Generic method for adding/modifying the attributes for new configuration.
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6836
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6836
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Iyappan Srinivasan
>            Assignee: Vinay Kumar Thota
>             Fix For: 0.21.0
>
>         Attachments: 6836-ydist-security.patch, HADOOP-6836.patch
>
>
> HADOP-6772 deals with 
> Common utilities for system tests.
> 1. A method for restarting the daemon with new configuration.
> c throws Exception; 
> 2. A method for restarting the daemon with default configuration.
> public void restart() throws Exception;
> 3. A method for waiting until daemon is stop.
> public void waitForClusterToStop() throws Exception;
> In this  some variables are made of String, instead of Long. Those needs ot be changed too. So, can this method 
> "public static void restartClusterWithNewConfig(Hashtable<String,Long> props, String confFile) throws Exception;"
> be generalized to accepts string too. All otehr methods should work as usual.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9527-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 11:37:47 2010
Return-Path: <common-issues-return-9527-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15647 invoked from network); 15 Jul 2010 11:37:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 11:37:47 -0000
Received: (qmail 94997 invoked by uid 500); 15 Jul 2010 11:37:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94865 invoked by uid 500); 15 Jul 2010 11:37:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94857 invoked by uid 99); 15 Jul 2010 11:37:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 11:37:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 11:37:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6FBTnBZ018744
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 11:29:50 GMT
Message-ID: <16219926.394721279193389973.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 07:29:49 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6862) Add api to add user/group to
 AccessControlList
In-Reply-To: <6885195.391831279173830267.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888780#action_12888780 ] 

Hadoop QA commented on HADOOP-6862:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12449552/patch-6862-1.txt
  against trunk revision 964134.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/614/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/614/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/614/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/614/console

This message is automatically generated.

> Add api to add user/group to AccessControlList
> ----------------------------------------------
>
>                 Key: HADOOP-6862
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6862
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.22.0
>
>         Attachments: patch-6862-1.txt, patch-6862.txt
>
>
> Add api addUser(String user) and addGroup(String group) to add user/group to AccessControlList

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9528-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 11:49:47 2010
Return-Path: <common-issues-return-9528-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32380 invoked from network); 15 Jul 2010 11:49:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 11:49:47 -0000
Received: (qmail 5428 invoked by uid 500); 15 Jul 2010 11:49:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5290 invoked by uid 500); 15 Jul 2010 11:49:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5270 invoked by uid 99); 15 Jul 2010 11:49:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 11:49:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 11:49:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6FBfoi1018811
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 11:41:50 GMT
Message-ID: <6015722.394811279194109995.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 07:41:49 -0400 (EDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6862) Add api to add user/group to
 AccessControlList
In-Reply-To: <6885195.391831279173830267.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888783#action_12888783 ] 

Amareshwari Sriramadasu commented on HADOOP-6862:
-------------------------------------------------

bq. -1 javadoc. 
Is because of HADOOP-6830. No new javadoc warnings are introduced in the patch.

> Add api to add user/group to AccessControlList
> ----------------------------------------------
>
>                 Key: HADOOP-6862
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6862
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.22.0
>
>         Attachments: patch-6862-1.txt, patch-6862.txt
>
>
> Add api addUser(String user) and addGroup(String group) to add user/group to AccessControlList

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9529-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 17:49:47 2010
Return-Path: <common-issues-return-9529-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66014 invoked from network); 15 Jul 2010 17:49:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 17:49:47 -0000
Received: (qmail 94614 invoked by uid 500); 15 Jul 2010 17:49:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94568 invoked by uid 500); 15 Jul 2010 17:49:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94560 invoked by uid 99); 15 Jul 2010 17:49:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 17:49:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 17:49:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6FHfpB4022293
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 17:41:52 GMT
Message-ID: <15527493.399121279215711904.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 13:41:51 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6632) Support for using different Kerberos
 keys for different instances of Hadoop services
In-Reply-To: <1542465557.251491268540967230.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6632:
------------------------------

    Attachment: c6632-05.patch

A port for trunk.

> Support for using different Kerberos keys for different instances of Hadoop services
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6632
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6632
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: 6632.mr.patch, c6632-05.patch, HADOOP-6632-Y20S-18.patch, HADOOP-6632-Y20S-22.patch
>
>
> We tested using the same Kerberos key for all datanodes in a HDFS cluster or the same Kerberos key for all TaskTarckers in a MapRed cluster. But it doesn't work. The reason is that when datanodes try to authenticate to the namenode all at once, the Kerberos authenticators they send to the namenode may have the same timestamp and will be rejected as replay requests. This JIRA makes it possible to use a unique key for each service instance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9530-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 17:49:47 2010
Return-Path: <common-issues-return-9530-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66060 invoked from network); 15 Jul 2010 17:49:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 17:49:47 -0000
Received: (qmail 94750 invoked by uid 500); 15 Jul 2010 17:49:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94579 invoked by uid 500); 15 Jul 2010 17:49:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94569 invoked by uid 99); 15 Jul 2010 17:49:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 17:49:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 17:49:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6FHfqn8022305
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 17:41:52 GMT
Message-ID: <8117389.399161279215712785.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 13:41:52 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6632) Support for using different Kerberos
 keys for different instances of Hadoop services
In-Reply-To: <1542465557.251491268540967230.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6632:
------------------------------

    Status: Patch Available  (was: Open)

> Support for using different Kerberos keys for different instances of Hadoop services
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6632
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6632
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: 6632.mr.patch, c6632-05.patch, HADOOP-6632-Y20S-18.patch, HADOOP-6632-Y20S-22.patch
>
>
> We tested using the same Kerberos key for all datanodes in a HDFS cluster or the same Kerberos key for all TaskTarckers in a MapRed cluster. But it doesn't work. The reason is that when datanodes try to authenticate to the namenode all at once, the Kerberos authenticators they send to the namenode may have the same timestamp and will be rejected as replay requests. This JIRA makes it possible to use a unique key for each service instance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9531-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 18:06:16 2010
Return-Path: <common-issues-return-9531-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73379 invoked from network); 15 Jul 2010 18:06:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 18:06:15 -0000
Received: (qmail 15595 invoked by uid 500); 15 Jul 2010 18:06:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15514 invoked by uid 500); 15 Jul 2010 18:06:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15506 invoked by uid 99); 15 Jul 2010 18:06:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 18:06:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 18:06:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6FI5oa6022493
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 18:05:50 GMT
Message-ID: <10794212.399401279217150449.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 14:05:50 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6861) Method in Credentials to read and
 write a token storage file.
In-Reply-To: <21462038.384681279145270696.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6861:
-----------------------------------------

    Status: Patch Available  (was: Open)

> Method in Credentials to read and write a token storage file.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6861
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6861
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6861.1.patch, HADOOP-6861.2.patch
>
>
> The jira covers the changes in common corresponding to MAPREDUCE-1566. 
> This jira adds  new non-static methods in Credentials to read and write token storage file. A method to copy tokens from another credential object is also added. Static method readTokensAndLoadInUGI is removed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9532-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 18:10:15 2010
Return-Path: <common-issues-return-9532-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74829 invoked from network); 15 Jul 2010 18:10:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 18:10:15 -0000
Received: (qmail 19667 invoked by uid 500); 15 Jul 2010 18:10:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19550 invoked by uid 500); 15 Jul 2010 18:10:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19541 invoked by uid 99); 15 Jul 2010 18:10:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 18:10:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 18:10:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6FI9oPS022540
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 18:09:50 GMT
Message-ID: <147967.399481279217390432.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 14:09:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6632) Support for using different
 Kerberos keys for different instances of Hadoop services
In-Reply-To: <1542465557.251491268540967230.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888873#action_12888873 ] 

Hadoop QA commented on HADOOP-6632:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12449571/c6632-05.patch
  against trunk revision 964134.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/615/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/615/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/615/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/615/console

This message is automatically generated.

> Support for using different Kerberos keys for different instances of Hadoop services
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6632
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6632
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: 6632.mr.patch, c6632-05.patch, HADOOP-6632-Y20S-18.patch, HADOOP-6632-Y20S-22.patch
>
>
> We tested using the same Kerberos key for all datanodes in a HDFS cluster or the same Kerberos key for all TaskTarckers in a MapRed cluster. But it doesn't work. The reason is that when datanodes try to authenticate to the namenode all at once, the Kerberos authenticators they send to the namenode may have the same timestamp and will be rejected as replay requests. This JIRA makes it possible to use a unique key for each service instance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9533-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 18:36:45 2010
Return-Path: <common-issues-return-9533-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90275 invoked from network); 15 Jul 2010 18:36:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 18:36:45 -0000
Received: (qmail 55580 invoked by uid 500); 15 Jul 2010 18:36:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55510 invoked by uid 500); 15 Jul 2010 18:36:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55502 invoked by uid 99); 15 Jul 2010 18:36:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 18:36:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 18:36:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6FISouI022722
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 18:28:50 GMT
Message-ID: <19964725.399931279218530547.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 14:28:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6861) Method in Credentials to read and
 write a token storage file.
In-Reply-To: <21462038.384681279145270696.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888881#action_12888881 ] 

Hadoop QA commented on HADOOP-6861:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12449514/HADOOP-6861.2.patch
  against trunk revision 964134.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/616/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/616/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/616/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/616/console

This message is automatically generated.

> Method in Credentials to read and write a token storage file.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6861
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6861
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6861.1.patch, HADOOP-6861.2.patch
>
>
> The jira covers the changes in common corresponding to MAPREDUCE-1566. 
> This jira adds  new non-static methods in Credentials to read and write token storage file. A method to copy tokens from another credential object is also added. Static method readTokensAndLoadInUGI is removed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9534-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 20:26:22 2010
Return-Path: <common-issues-return-9534-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35341 invoked from network); 15 Jul 2010 20:26:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 20:26:22 -0000
Received: (qmail 80747 invoked by uid 500); 15 Jul 2010 20:26:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80676 invoked by uid 500); 15 Jul 2010 20:26:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80668 invoked by uid 99); 15 Jul 2010 20:26:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 20:26:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 20:26:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6FKPuZQ024109
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 20:25:56 GMT
Message-ID: <12540237.402981279225556222.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 16:25:56 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6859) Introduce additional statistics to
 FileSystem
In-Reply-To: <3466971.360371279057551878.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888909#action_12888909 ] 

Suresh Srinivas commented on HADOOP-6859:
-----------------------------------------

Large read operation is left to the file system implementation. I am explaining that in the context of HDFS in this jira. I have it also documented in javadoc for getLargeReadOps() method.

> Introduce additional statistics to FileSystem
> ---------------------------------------------
>
>                 Key: HADOOP-6859
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6859
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6859.patch
>
>
> Currently FileSystem#statistics tracks bytesRead and bytesWritten. Additional statistics that gives summary of operations performed will be useful for tracking file system use.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9535-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 21:38:16 2010
Return-Path: <common-issues-return-9535-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60027 invoked from network); 15 Jul 2010 21:38:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 21:38:16 -0000
Received: (qmail 76171 invoked by uid 500); 15 Jul 2010 21:38:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76125 invoked by uid 500); 15 Jul 2010 21:38:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76117 invoked by uid 99); 15 Jul 2010 21:38:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 21:38:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 21:38:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6FLbpCl024791
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 21:37:51 GMT
Message-ID: <29638809.404411279229870988.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 17:37:50 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6834) TFile.append compares initial key
 against null lastKey
In-Reply-To: <10707501.8491277191496663.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-6834:
------------------------------

    Status: Patch Available  (was: Open)

> TFile.append compares initial key against null lastKey  
> --------------------------------------------------------
>
>                 Key: HADOOP-6834
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6834
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.2, 0.20.1
>            Reporter: Ahad Rana
>            Assignee: Hong Tang
>         Attachments: hadoop-6834-20100715.patch
>
>
> The following code in TFile.KeyReigster.close: 
>             byte[] lastKey = lastKeyBufferOS.getBuffer();
>             int lastLen = lastKeyBufferOS.size();
>             if (tfileMeta.getComparator().compare(key, 0, len, lastKey, 0,
>                 lastLen) < 0) {
>               throw new IOException("Keys are not added in sorted order");
>             }
> compares the initial  key (passed in via  TFile.Writer.append) against a technically NULL lastKey. lastKey is not initialized until after the first call to TFile.Writer.append. The underlying RawComparator interface used for comparisons does not stipulate the proper behavior when either length 1  or length 2 is zero. In the case of LongWritable, its WritableComparator implementation does an unsafe read on the passed in byte arrays b1 and b2. Since TFile pre-allocates the buffer used for storing lastKey, this passes a valid buffer with zero count to LongWritable's comparator, which ignores length and thus produces incorrect results. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9536-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 21:41:45 2010
Return-Path: <common-issues-return-9536-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62232 invoked from network); 15 Jul 2010 21:41:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 21:41:45 -0000
Received: (qmail 80528 invoked by uid 500); 15 Jul 2010 21:41:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80474 invoked by uid 500); 15 Jul 2010 21:41:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80466 invoked by uid 99); 15 Jul 2010 21:41:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 21:41:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 21:41:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6FLXoxT024761
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 21:33:50 GMT
Message-ID: <7653456.404351279229630510.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 17:33:50 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6834) TFile.append compares initial key
 against null lastKey
In-Reply-To: <10707501.8491277191496663.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang reassigned HADOOP-6834:
---------------------------------

    Assignee: Hong Tang

> TFile.append compares initial key against null lastKey  
> --------------------------------------------------------
>
>                 Key: HADOOP-6834
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6834
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Ahad Rana
>            Assignee: Hong Tang
>
> The following code in TFile.KeyReigster.close: 
>             byte[] lastKey = lastKeyBufferOS.getBuffer();
>             int lastLen = lastKeyBufferOS.size();
>             if (tfileMeta.getComparator().compare(key, 0, len, lastKey, 0,
>                 lastLen) < 0) {
>               throw new IOException("Keys are not added in sorted order");
>             }
> compares the initial  key (passed in via  TFile.Writer.append) against a technically NULL lastKey. lastKey is not initialized until after the first call to TFile.Writer.append. The underlying RawComparator interface used for comparisons does not stipulate the proper behavior when either length 1  or length 2 is zero. In the case of LongWritable, its WritableComparator implementation does an unsafe read on the passed in byte arrays b1 and b2. Since TFile pre-allocates the buffer used for storing lastKey, this passes a valid buffer with zero count to LongWritable's comparator, which ignores length and thus produces incorrect results. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9537-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 21:45:44 2010
Return-Path: <common-issues-return-9537-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64805 invoked from network); 15 Jul 2010 21:45:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 21:45:44 -0000
Received: (qmail 86090 invoked by uid 500); 15 Jul 2010 21:45:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86061 invoked by uid 500); 15 Jul 2010 21:45:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86053 invoked by uid 99); 15 Jul 2010 21:45:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 21:45:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 21:45:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6FLboFT024782
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 21:37:50 GMT
Message-ID: <32305390.404381279229870241.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 17:37:50 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6834) TFile.append compares initial key
 against null lastKey
In-Reply-To: <10707501.8491277191496663.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-6834:
------------------------------

    Attachment: hadoop-6834-20100715.patch

Straightforward patch with a new unit test.

> TFile.append compares initial key against null lastKey  
> --------------------------------------------------------
>
>                 Key: HADOOP-6834
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6834
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Ahad Rana
>            Assignee: Hong Tang
>         Attachments: hadoop-6834-20100715.patch
>
>
> The following code in TFile.KeyReigster.close: 
>             byte[] lastKey = lastKeyBufferOS.getBuffer();
>             int lastLen = lastKeyBufferOS.size();
>             if (tfileMeta.getComparator().compare(key, 0, len, lastKey, 0,
>                 lastLen) < 0) {
>               throw new IOException("Keys are not added in sorted order");
>             }
> compares the initial  key (passed in via  TFile.Writer.append) against a technically NULL lastKey. lastKey is not initialized until after the first call to TFile.Writer.append. The underlying RawComparator interface used for comparisons does not stipulate the proper behavior when either length 1  or length 2 is zero. In the case of LongWritable, its WritableComparator implementation does an unsafe read on the passed in byte arrays b1 and b2. Since TFile pre-allocates the buffer used for storing lastKey, this passes a valid buffer with zero count to LongWritable's comparator, which ignores length and thus produces incorrect results. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9538-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 21:57:45 2010
Return-Path: <common-issues-return-9538-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72481 invoked from network); 15 Jul 2010 21:57:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 21:57:45 -0000
Received: (qmail 12217 invoked by uid 500); 15 Jul 2010 21:57:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12160 invoked by uid 500); 15 Jul 2010 21:57:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12145 invoked by uid 99); 15 Jul 2010 21:57:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 21:57:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 21:57:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6FLnorO024891
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 21:49:50 GMT
Message-ID: <18053660.404611279230589983.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 17:49:49 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6863) Compile-native still fails on Mac OS
 X.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Compile-native still fails on Mac OS X.
---------------------------------------

                 Key: HADOOP-6863
                 URL: https://issues.apache.org/jira/browse/HADOOP-6863
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Hong Tang


I am still getting failures when I try to compile native library on Mac OS X (10.5.8 Leopard). The problems are two fold:
- Although "aclocal.m4" is changed after HADOOP-3659 is committed, the corresponding "configure" script is not re-generated.
- It cannot find libjvm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9539-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 22:08:51 2010
Return-Path: <common-issues-return-9539-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80984 invoked from network); 15 Jul 2010 22:08:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 22:08:51 -0000
Received: (qmail 23253 invoked by uid 500); 15 Jul 2010 22:08:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23204 invoked by uid 500); 15 Jul 2010 22:08:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23192 invoked by uid 99); 15 Jul 2010 22:08:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 22:08:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 22:08:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6FM0tI7024978
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 22:00:56 GMT
Message-ID: <16575846.404681279231255798.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 18:00:55 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6834) TFile.append compares initial key
 against null lastKey
In-Reply-To: <10707501.8491277191496663.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888939#action_12888939 ] 

Hadoop QA commented on HADOOP-6834:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12449597/hadoop-6834-20100715.patch
  against trunk revision 964134.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 2 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/617/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/617/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/617/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/617/console

This message is automatically generated.

> TFile.append compares initial key against null lastKey  
> --------------------------------------------------------
>
>                 Key: HADOOP-6834
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6834
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Ahad Rana
>            Assignee: Hong Tang
>         Attachments: hadoop-6834-20100715.patch
>
>
> The following code in TFile.KeyReigster.close: 
>             byte[] lastKey = lastKeyBufferOS.getBuffer();
>             int lastLen = lastKeyBufferOS.size();
>             if (tfileMeta.getComparator().compare(key, 0, len, lastKey, 0,
>                 lastLen) < 0) {
>               throw new IOException("Keys are not added in sorted order");
>             }
> compares the initial  key (passed in via  TFile.Writer.append) against a technically NULL lastKey. lastKey is not initialized until after the first call to TFile.Writer.append. The underlying RawComparator interface used for comparisons does not stipulate the proper behavior when either length 1  or length 2 is zero. In the case of LongWritable, its WritableComparator implementation does an unsafe read on the passed in byte arrays b1 and b2. Since TFile pre-allocates the buffer used for storing lastKey, this passes a valid buffer with zero count to LongWritable's comparator, which ignores length and thus produces incorrect results. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9540-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 22:28:47 2010
Return-Path: <common-issues-return-9540-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86231 invoked from network); 15 Jul 2010 22:28:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 22:28:47 -0000
Received: (qmail 40953 invoked by uid 500); 15 Jul 2010 22:28:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40909 invoked by uid 500); 15 Jul 2010 22:28:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40901 invoked by uid 99); 15 Jul 2010 22:28:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 22:28:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 22:28:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6FMKqWP025181
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 22:20:52 GMT
Message-ID: <3583444.404951279232452952.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 18:20:52 -0400 (EDT)
From: "Luke Forehand (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5859) FindBugs : fix "wait() or sleep()
 with locks held" warnings in hdfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888943#action_12888943 ] 

Luke Forehand commented on HADOOP-5859:
---------------------------------------

Is this patch applied to 0.20.1 by chance?  We are experiencing this issue with regularity on a 3 node cluster attempting to load 15GB of data.

> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
>                 Key: HADOOP-5859
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5859
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>             Fix For: 0.21.0
>
>         Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-33.patch, 5859-35.patch, 5859-36.patch, 5859-38.patch, 5859-4.patch, 5859-40.patch, 5859-41.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL	org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW	wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW	wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW	wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9541-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 23:01:46 2010
Return-Path: <common-issues-return-9541-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94047 invoked from network); 15 Jul 2010 23:01:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 23:01:45 -0000
Received: (qmail 63539 invoked by uid 500); 15 Jul 2010 23:01:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63489 invoked by uid 500); 15 Jul 2010 23:01:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63475 invoked by uid 99); 15 Jul 2010 23:01:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 23:01:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 23:01:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6FMrpRZ025471
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 22:53:51 GMT
Message-ID: <13820223.405691279234431078.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 18:53:51 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6863) Compile-native still fails on Mac
 OS X.
In-Reply-To: <18053660.404611279230589983.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888947#action_12888947 ] 

Allen Wittenauer commented on HADOOP-6863:
------------------------------------------

These are pretty much two separate problems.

I have a fix for b, but it isn't very clean. :)

> Compile-native still fails on Mac OS X.
> ---------------------------------------
>
>                 Key: HADOOP-6863
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6863
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hong Tang
>
> I am still getting failures when I try to compile native library on Mac OS X (10.5.8 Leopard). The problems are two fold:
> - Although "aclocal.m4" is changed after HADOOP-3659 is committed, the corresponding "configure" script is not re-generated.
> - It cannot find libjvm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9542-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 15 23:29:45 2010
Return-Path: <common-issues-return-9542-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5543 invoked from network); 15 Jul 2010 23:29:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jul 2010 23:29:45 -0000
Received: (qmail 89028 invoked by uid 500); 15 Jul 2010 23:29:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88936 invoked by uid 500); 15 Jul 2010 23:29:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88928 invoked by uid 99); 15 Jul 2010 23:29:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 23:29:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jul 2010 23:29:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6FNLnXd025792
	for <common-issues@hadoop.apache.org>; Thu, 15 Jul 2010 23:21:50 GMT
Message-ID: <11586290.406181279236109974.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 19:21:49 -0400 (EDT)
From: "Ahad Rana (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6834) TFile.append compares initial key
 against null lastKey
In-Reply-To: <10707501.8491277191496663.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888949#action_12888949 ] 

Ahad Rana commented on HADOOP-6834:
-----------------------------------

I apologize Hong,  this was on my todo list, but obviously I did not get to it in time. Your patch and test case look good. One question: Is a NULL key allowed in the TFile API ? It seems that KeyRegister's close does accepts a zero length key.

Ahad

> TFile.append compares initial key against null lastKey  
> --------------------------------------------------------
>
>                 Key: HADOOP-6834
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6834
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Ahad Rana
>            Assignee: Hong Tang
>         Attachments: hadoop-6834-20100715.patch
>
>
> The following code in TFile.KeyReigster.close: 
>             byte[] lastKey = lastKeyBufferOS.getBuffer();
>             int lastLen = lastKeyBufferOS.size();
>             if (tfileMeta.getComparator().compare(key, 0, len, lastKey, 0,
>                 lastLen) < 0) {
>               throw new IOException("Keys are not added in sorted order");
>             }
> compares the initial  key (passed in via  TFile.Writer.append) against a technically NULL lastKey. lastKey is not initialized until after the first call to TFile.Writer.append. The underlying RawComparator interface used for comparisons does not stipulate the proper behavior when either length 1  or length 2 is zero. In the case of LongWritable, its WritableComparator implementation does an unsafe read on the passed in byte arrays b1 and b2. Since TFile pre-allocates the buffer used for storing lastKey, this passes a valid buffer with zero count to LongWritable's comparator, which ignores length and thus produces incorrect results. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9543-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 00:08:15 2010
Return-Path: <common-issues-return-9543-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14646 invoked from network); 16 Jul 2010 00:08:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 00:08:15 -0000
Received: (qmail 16354 invoked by uid 500); 16 Jul 2010 00:08:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16336 invoked by uid 500); 16 Jul 2010 00:08:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16328 invoked by uid 99); 16 Jul 2010 00:08:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 00:08:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 00:08:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6G07oaN026183
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 00:07:50 GMT
Message-ID: <24602984.406891279238870213.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 20:07:50 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6834) TFile.append compares initial key
 against null lastKey
In-Reply-To: <10707501.8491277191496663.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888963#action_12888963 ] 

Hong Tang commented on HADOOP-6834:
-----------------------------------

The javadoc warning reported by test-patch seems to be false positive. (Log: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/617/artifact/trunk/patchprocess/patchJavadocWarnings.txt).

> TFile.append compares initial key against null lastKey  
> --------------------------------------------------------
>
>                 Key: HADOOP-6834
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6834
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Ahad Rana
>            Assignee: Hong Tang
>         Attachments: hadoop-6834-20100715.patch
>
>
> The following code in TFile.KeyReigster.close: 
>             byte[] lastKey = lastKeyBufferOS.getBuffer();
>             int lastLen = lastKeyBufferOS.size();
>             if (tfileMeta.getComparator().compare(key, 0, len, lastKey, 0,
>                 lastLen) < 0) {
>               throw new IOException("Keys are not added in sorted order");
>             }
> compares the initial  key (passed in via  TFile.Writer.append) against a technically NULL lastKey. lastKey is not initialized until after the first call to TFile.Writer.append. The underlying RawComparator interface used for comparisons does not stipulate the proper behavior when either length 1  or length 2 is zero. In the case of LongWritable, its WritableComparator implementation does an unsafe read on the passed in byte arrays b1 and b2. Since TFile pre-allocates the buffer used for storing lastKey, this passes a valid buffer with zero count to LongWritable's comparator, which ignores length and thus produces incorrect results. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9544-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 00:15:17 2010
Return-Path: <common-issues-return-9544-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15527 invoked from network); 16 Jul 2010 00:15:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 00:15:17 -0000
Received: (qmail 19261 invoked by uid 500); 16 Jul 2010 00:15:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19191 invoked by uid 500); 16 Jul 2010 00:15:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19182 invoked by uid 99); 16 Jul 2010 00:15:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 00:15:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 00:15:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6G0Eps9026270
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 00:14:52 GMT
Message-ID: <23023571.407061279239291960.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 20:14:51 -0400 (EDT)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6859) Introduce additional statistics to
 FileSystem
In-Reply-To: <3466971.360371279057551878.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888966#action_12888966 ] 

Konstantin Shvachko commented on HADOOP-6859:
---------------------------------------------

+1 Both patches look good to me.

> Introduce additional statistics to FileSystem
> ---------------------------------------------
>
>                 Key: HADOOP-6859
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6859
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6859.patch
>
>
> Currently FileSystem#statistics tracks bytesRead and bytesWritten. Additional statistics that gives summary of operations performed will be useful for tracking file system use.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9545-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 00:18:45 2010
Return-Path: <common-issues-return-9545-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15797 invoked from network); 16 Jul 2010 00:18:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 00:18:45 -0000
Received: (qmail 21517 invoked by uid 500); 16 Jul 2010 00:18:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21451 invoked by uid 500); 16 Jul 2010 00:18:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21443 invoked by uid 99); 16 Jul 2010 00:18:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 00:18:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 00:18:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6G0AoCq026207
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 00:10:50 GMT
Message-ID: <13243745.406921279239050349.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 20:10:50 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6834) TFile.append compares initial key
 against null lastKey
In-Reply-To: <10707501.8491277191496663.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888964#action_12888964 ] 

Hong Tang commented on HADOOP-6834:
-----------------------------------

@ahad, by "NULL" key, are you referring to zero-length key? It is certainly allowed by TFile (and you can even have multiple of them). 

> TFile.append compares initial key against null lastKey  
> --------------------------------------------------------
>
>                 Key: HADOOP-6834
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6834
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Ahad Rana
>            Assignee: Hong Tang
>         Attachments: hadoop-6834-20100715.patch
>
>
> The following code in TFile.KeyReigster.close: 
>             byte[] lastKey = lastKeyBufferOS.getBuffer();
>             int lastLen = lastKeyBufferOS.size();
>             if (tfileMeta.getComparator().compare(key, 0, len, lastKey, 0,
>                 lastLen) < 0) {
>               throw new IOException("Keys are not added in sorted order");
>             }
> compares the initial  key (passed in via  TFile.Writer.append) against a technically NULL lastKey. lastKey is not initialized until after the first call to TFile.Writer.append. The underlying RawComparator interface used for comparisons does not stipulate the proper behavior when either length 1  or length 2 is zero. In the case of LongWritable, its WritableComparator implementation does an unsafe read on the passed in byte arrays b1 and b2. Since TFile pre-allocates the buffer used for storing lastKey, this passes a valid buffer with zero count to LongWritable's comparator, which ignores length and thus produces incorrect results. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9546-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 00:22:50 2010
Return-Path: <common-issues-return-9546-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16434 invoked from network); 16 Jul 2010 00:22:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 00:22:49 -0000
Received: (qmail 25101 invoked by uid 500); 16 Jul 2010 00:22:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25069 invoked by uid 500); 16 Jul 2010 00:22:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25061 invoked by uid 99); 16 Jul 2010 00:22:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 00:22:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 00:22:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6G0EtiX026303
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 00:14:55 GMT
Message-ID: <30946696.407171279239295016.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 20:14:55 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5859) FindBugs : fix "wait() or sleep()
 with locks held" warnings in hdfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888967#action_12888967 ] 

Suresh Srinivas commented on HADOOP-5859:
-----------------------------------------

This patch is not applied to 0.20.1

> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
>                 Key: HADOOP-5859
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5859
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>             Fix For: 0.21.0
>
>         Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-33.patch, 5859-35.patch, 5859-36.patch, 5859-38.patch, 5859-4.patch, 5859-40.patch, 5859-41.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL	org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW	wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW	wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW	wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9547-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 00:29:47 2010
Return-Path: <common-issues-return-9547-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24835 invoked from network); 16 Jul 2010 00:29:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 00:29:47 -0000
Received: (qmail 30257 invoked by uid 500); 16 Jul 2010 00:29:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30190 invoked by uid 500); 16 Jul 2010 00:29:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30181 invoked by uid 99); 16 Jul 2010 00:29:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 00:29:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 00:29:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6G0Lpcg026387
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 00:21:51 GMT
Message-ID: <18607926.407411279239711060.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 20:21:51 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6859) Introduce additional statistics to
 FileSystem
In-Reply-To: <3466971.360371279057551878.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6859:
------------------------------------

    Status: Patch Available  (was: Open)

> Introduce additional statistics to FileSystem
> ---------------------------------------------
>
>                 Key: HADOOP-6859
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6859
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6859.patch
>
>
> Currently FileSystem#statistics tracks bytesRead and bytesWritten. Additional statistics that gives summary of operations performed will be useful for tracking file system use.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9548-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 00:37:17 2010
Return-Path: <common-issues-return-9548-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26604 invoked from network); 16 Jul 2010 00:37:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 00:37:17 -0000
Received: (qmail 37190 invoked by uid 500); 16 Jul 2010 00:37:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37118 invoked by uid 500); 16 Jul 2010 00:37:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37110 invoked by uid 99); 16 Jul 2010 00:37:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 00:37:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 00:37:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6G0apuN026444
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 00:36:52 GMT
Message-ID: <25705700.407521279240611742.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 20:36:51 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6863) Compile-native still fails on Mac OS
 X.
In-Reply-To: <18053660.404611279230589983.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-6863:
------------------------------

    Attachment: hadoop-6863-20100715.patch

Got a simple fix:
- Remove the hardcoded "-ljvm" in two Makefile.am files.
- Run "autoreconf".

With the attached patch, ant compile-native passes on both Mac OS-X and Linux RHEL 5.1.



> Compile-native still fails on Mac OS X.
> ---------------------------------------
>
>                 Key: HADOOP-6863
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6863
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hong Tang
>         Attachments: hadoop-6863-20100715.patch
>
>
> I am still getting failures when I try to compile native library on Mac OS X (10.5.8 Leopard). The problems are two fold:
> - Although "aclocal.m4" is changed after HADOOP-3659 is committed, the corresponding "configure" script is not re-generated.
> - It cannot find libjvm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9549-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 00:37:18 2010
Return-Path: <common-issues-return-9549-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26649 invoked from network); 16 Jul 2010 00:37:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 00:37:17 -0000
Received: (qmail 37313 invoked by uid 500); 16 Jul 2010 00:37:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37270 invoked by uid 500); 16 Jul 2010 00:37:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37120 invoked by uid 99); 16 Jul 2010 00:37:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 00:37:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 00:37:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6G0arPb026456
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 00:36:53 GMT
Message-ID: <13498187.407561279240613078.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 20:36:53 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6863) Compile-native still fails on Mac
 OS X.
In-Reply-To: <18053660.404611279230589983.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888974#action_12888974 ] 

Hong Tang commented on HADOOP-6863:
-----------------------------------

Note, the real changes to the existing code base are really in two files. The remaining changes are caused by "autoreconf".
- ./src/native/Makefile.am
- ./src/native/src/org/apache/hadoop/io/compress/zlib/Makefile.am

> Compile-native still fails on Mac OS X.
> ---------------------------------------
>
>                 Key: HADOOP-6863
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6863
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hong Tang
>         Attachments: hadoop-6863-20100715.patch
>
>
> I am still getting failures when I try to compile native library on Mac OS X (10.5.8 Leopard). The problems are two fold:
> - Although "aclocal.m4" is changed after HADOOP-3659 is committed, the corresponding "configure" script is not re-generated.
> - It cannot find libjvm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9550-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 00:37:18 2010
Return-Path: <common-issues-return-9550-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26665 invoked from network); 16 Jul 2010 00:37:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 00:37:17 -0000
Received: (qmail 37394 invoked by uid 500); 16 Jul 2010 00:37:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37280 invoked by uid 500); 16 Jul 2010 00:37:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37271 invoked by uid 99); 16 Jul 2010 00:37:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 00:37:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 00:37:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6G0ar4W026468
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 00:36:53 GMT
Message-ID: <31421821.407601279240613703.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 20:36:53 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6863) Compile-native still fails on Mac OS
 X.
In-Reply-To: <18053660.404611279230589983.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-6863:
------------------------------

      Status: Patch Available  (was: Open)
    Assignee: Hong Tang

> Compile-native still fails on Mac OS X.
> ---------------------------------------
>
>                 Key: HADOOP-6863
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6863
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>         Attachments: hadoop-6863-20100715.patch
>
>
> I am still getting failures when I try to compile native library on Mac OS X (10.5.8 Leopard). The problems are two fold:
> - Although "aclocal.m4" is changed after HADOOP-3659 is committed, the corresponding "configure" script is not re-generated.
> - It cannot find libjvm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9551-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 00:51:16 2010
Return-Path: <common-issues-return-9551-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29317 invoked from network); 16 Jul 2010 00:51:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 00:51:16 -0000
Received: (qmail 52176 invoked by uid 500); 16 Jul 2010 00:51:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52129 invoked by uid 500); 16 Jul 2010 00:51:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52121 invoked by uid 99); 16 Jul 2010 00:51:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 00:51:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 00:51:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6G0opON026525
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 00:50:51 GMT
Message-ID: <16342645.407711279241451803.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 20:50:51 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6859) Introduce additional statistics to
 FileSystem
In-Reply-To: <3466971.360371279057551878.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888975#action_12888975 ] 

Hadoop QA commented on HADOOP-6859:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12449390/HADOOP-6859.patch
  against trunk revision 964134.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/618/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/618/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/618/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/618/console

This message is automatically generated.

> Introduce additional statistics to FileSystem
> ---------------------------------------------
>
>                 Key: HADOOP-6859
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6859
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6859.patch
>
>
> Currently FileSystem#statistics tracks bytesRead and bytesWritten. Additional statistics that gives summary of operations performed will be useful for tracking file system use.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9552-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 01:05:46 2010
Return-Path: <common-issues-return-9552-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36571 invoked from network); 16 Jul 2010 01:05:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 01:05:46 -0000
Received: (qmail 64807 invoked by uid 500); 16 Jul 2010 01:05:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64719 invoked by uid 500); 16 Jul 2010 01:05:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64711 invoked by uid 99); 16 Jul 2010 01:05:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 01:05:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 01:05:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6G0vpUt026567
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 00:57:51 GMT
Message-ID: <29481646.407781279241871420.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 20:57:51 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6863) Compile-native still fails on Mac
 OS X.
In-Reply-To: <18053660.404611279230589983.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888976#action_12888976 ] 

Hadoop QA commented on HADOOP-6863:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12449607/hadoop-6863-20100715.patch
  against trunk revision 964134.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 1 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/88/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/88/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/88/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/88/console

This message is automatically generated.

> Compile-native still fails on Mac OS X.
> ---------------------------------------
>
>                 Key: HADOOP-6863
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6863
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>         Attachments: hadoop-6863-20100715.patch
>
>
> I am still getting failures when I try to compile native library on Mac OS X (10.5.8 Leopard). The problems are two fold:
> - Although "aclocal.m4" is changed after HADOOP-3659 is committed, the corresponding "configure" script is not re-generated.
> - It cannot find libjvm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9553-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 01:09:16 2010
Return-Path: <common-issues-return-9553-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36930 invoked from network); 16 Jul 2010 01:09:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 01:09:16 -0000
Received: (qmail 73869 invoked by uid 500); 16 Jul 2010 01:09:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73817 invoked by uid 500); 16 Jul 2010 01:09:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73809 invoked by uid 99); 16 Jul 2010 01:09:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 01:09:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 01:09:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6G18pCH026648
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 01:08:51 GMT
Message-ID: <9394191.407871279242531433.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 21:08:51 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6863) Compile-native still fails on Mac
 OS X.
In-Reply-To: <18053660.404611279230589983.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888978#action_12888978 ] 

Hong Tang commented on HADOOP-6863:
-----------------------------------

Hudson is wrong again. The patch does not include new tests, nor does it change any javadocs.

> Compile-native still fails on Mac OS X.
> ---------------------------------------
>
>                 Key: HADOOP-6863
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6863
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>         Attachments: hadoop-6863-20100715.patch
>
>
> I am still getting failures when I try to compile native library on Mac OS X (10.5.8 Leopard). The problems are two fold:
> - Although "aclocal.m4" is changed after HADOOP-3659 is committed, the corresponding "configure" script is not re-generated.
> - It cannot find libjvm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9554-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 01:40:45 2010
Return-Path: <common-issues-return-9554-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40444 invoked from network); 16 Jul 2010 01:40:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 01:40:45 -0000
Received: (qmail 95761 invoked by uid 500); 16 Jul 2010 01:40:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95707 invoked by uid 500); 16 Jul 2010 01:40:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95699 invoked by uid 99); 16 Jul 2010 01:40:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 01:40:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 01:40:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6G1WoCW026851
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 01:32:50 GMT
Message-ID: <28720028.408171279243970585.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 21:32:50 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6632) Support for using different
 Kerberos keys for different instances of Hadoop services
In-Reply-To: <1542465557.251491268540967230.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888982#action_12888982 ] 

Kan Zhang commented on HADOOP-6632:
-----------------------------------

The javadoc warnings are unrelated to this patch. Manually verified the feature on a single node cluster.

> Support for using different Kerberos keys for different instances of Hadoop services
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6632
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6632
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: 6632.mr.patch, c6632-05.patch, HADOOP-6632-Y20S-18.patch, HADOOP-6632-Y20S-22.patch
>
>
> We tested using the same Kerberos key for all datanodes in a HDFS cluster or the same Kerberos key for all TaskTarckers in a MapRed cluster. But it doesn't work. The reason is that when datanodes try to authenticate to the namenode all at once, the Kerberos authenticators they send to the namenode may have the same timestamp and will be rejected as replay requests. This JIRA makes it possible to use a unique key for each service instance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9555-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 02:34:05 2010
Return-Path: <common-issues-return-9555-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51785 invoked from network); 16 Jul 2010 02:34:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 02:34:05 -0000
Received: (qmail 29331 invoked by uid 500); 16 Jul 2010 02:34:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29128 invoked by uid 500); 16 Jul 2010 02:34:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29099 invoked by uid 99); 16 Jul 2010 02:34:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 02:34:02 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 02:33:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6G2PpXU027183
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 02:25:51 GMT
Message-ID: <7056991.408651279247151763.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 22:25:51 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6818) Provide a JNI-based implementation of
 GroupMappingServiceProvider
In-Reply-To: <17262845.29701276200793990.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6818:
--------------------------------

    Attachment: hadoop-6818-1.patch

Attaching a patch that simplifies the JNI part.

> Provide a JNI-based implementation of GroupMappingServiceProvider
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6818
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6818
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6818-1.patch, JNIGroupMapping.patch
>
>
> The default implementation of GroupMappingServiceProvider does a fork of a unix command to get the groups of a user. Since the group resolution happens in the servers, this might be costly. This jira aims at providing a JNI-based implementation for GroupMappingServiceProvider.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9556-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 03:02:18 2010
Return-Path: <common-issues-return-9556-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57719 invoked from network); 16 Jul 2010 03:02:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 03:02:18 -0000
Received: (qmail 46486 invoked by uid 500); 16 Jul 2010 03:02:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46294 invoked by uid 500); 16 Jul 2010 03:02:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46270 invoked by uid 99); 16 Jul 2010 03:02:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 03:02:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 03:01:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6G30qAx027367
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 03:00:52 GMT
Message-ID: <27540225.408871279249252552.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 23:00:52 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6818) Provide a JNI-based implementation of
 GroupMappingServiceProvider
In-Reply-To: <17262845.29701276200793990.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6818:
--------------------------------

    Attachment: hadoop-6818-2.patch

A bit of an update..

> Provide a JNI-based implementation of GroupMappingServiceProvider
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6818
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6818
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6818-1.patch, hadoop-6818-2.patch, JNIGroupMapping.patch
>
>
> The default implementation of GroupMappingServiceProvider does a fork of a unix command to get the groups of a user. Since the group resolution happens in the servers, this might be costly. This jira aims at providing a JNI-based implementation for GroupMappingServiceProvider.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9557-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 03:18:20 2010
Return-Path: <common-issues-return-9557-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64272 invoked from network); 16 Jul 2010 03:18:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 03:18:20 -0000
Received: (qmail 53721 invoked by uid 500); 16 Jul 2010 03:18:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53590 invoked by uid 500); 16 Jul 2010 03:18:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53582 invoked by uid 99); 16 Jul 2010 03:18:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 03:18:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 03:18:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6G3Hq6U027477
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 03:17:52 GMT
Message-ID: <29535746.409031279250271991.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 23:17:51 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6818) Provide a JNI-based implementation
 of GroupMappingServiceProvider
In-Reply-To: <17262845.29701276200793990.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888997#action_12888997 ] 

Erik Steffl commented on HADOOP-6818:
-------------------------------------

+1 for hadoop-6818-2.patch

> Provide a JNI-based implementation of GroupMappingServiceProvider
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6818
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6818
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6818-1.patch, hadoop-6818-2.patch, JNIGroupMapping.patch
>
>
> The default implementation of GroupMappingServiceProvider does a fork of a unix command to get the groups of a user. Since the group resolution happens in the servers, this might be costly. This jira aims at providing a JNI-based implementation for GroupMappingServiceProvider.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9558-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 03:54:20 2010
Return-Path: <common-issues-return-9558-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68889 invoked from network); 16 Jul 2010 03:54:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 03:54:19 -0000
Received: (qmail 73437 invoked by uid 500); 16 Jul 2010 03:54:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73299 invoked by uid 500); 16 Jul 2010 03:54:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73288 invoked by uid 99); 16 Jul 2010 03:54:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 03:54:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 03:54:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6G3rptm027602
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 03:53:51 GMT
Message-ID: <3720381.409261279252431682.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 23:53:51 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6859) Introduce additional statistics to
 FileSystem
In-Reply-To: <3466971.360371279057551878.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6859:
------------------------------------

    Status: Open  (was: Patch Available)

> Introduce additional statistics to FileSystem
> ---------------------------------------------
>
>                 Key: HADOOP-6859
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6859
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6859.patch
>
>
> Currently FileSystem#statistics tracks bytesRead and bytesWritten. Additional statistics that gives summary of operations performed will be useful for tracking file system use.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9559-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 04:01:49 2010
Return-Path: <common-issues-return-9559-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69898 invoked from network); 16 Jul 2010 04:01:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 04:01:49 -0000
Received: (qmail 79233 invoked by uid 500); 16 Jul 2010 04:01:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79107 invoked by uid 500); 16 Jul 2010 04:01:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79093 invoked by uid 99); 16 Jul 2010 04:01:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 04:01:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 04:01:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6G3rqWL027626
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 03:53:52 GMT
Message-ID: <18470425.409341279252432727.JavaMail.jira@thor>
Date: Thu, 15 Jul 2010 23:53:52 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6859) Introduce additional statistics to
 FileSystem
In-Reply-To: <3466971.360371279057551878.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6859:
------------------------------------

    Status: Patch Available  (was: Open)

I do not get javadoc warning when I run testpatch. Submitting it again to see if it is flagged again.

> Introduce additional statistics to FileSystem
> ---------------------------------------------
>
>                 Key: HADOOP-6859
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6859
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6859.patch
>
>
> Currently FileSystem#statistics tracks bytesRead and bytesWritten. Additional statistics that gives summary of operations performed will be useful for tracking file system use.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9560-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 04:31:20 2010
Return-Path: <common-issues-return-9560-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77577 invoked from network); 16 Jul 2010 04:31:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 04:31:20 -0000
Received: (qmail 95216 invoked by uid 500); 16 Jul 2010 04:31:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95085 invoked by uid 500); 16 Jul 2010 04:31:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95077 invoked by uid 99); 16 Jul 2010 04:31:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 04:31:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 04:31:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6G4UpH1028278
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 04:30:51 GMT
Message-ID: <17148276.409511279254651535.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 00:30:51 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6859) Introduce additional statistics to
 FileSystem
In-Reply-To: <3466971.360371279057551878.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889006#action_12889006 ] 

Hadoop QA commented on HADOOP-6859:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12449390/HADOOP-6859.patch
  against trunk revision 964134.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/619/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/619/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/619/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/619/console

This message is automatically generated.

> Introduce additional statistics to FileSystem
> ---------------------------------------------
>
>                 Key: HADOOP-6859
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6859
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6859.patch
>
>
> Currently FileSystem#statistics tracks bytesRead and bytesWritten. Additional statistics that gives summary of operations performed will be useful for tracking file system use.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9561-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 06:10:20 2010
Return-Path: <common-issues-return-9561-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 487 invoked from network); 16 Jul 2010 06:10:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 06:10:20 -0000
Received: (qmail 43227 invoked by uid 500); 16 Jul 2010 06:10:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43102 invoked by uid 500); 16 Jul 2010 06:10:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43086 invoked by uid 99); 16 Jul 2010 06:10:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 06:10:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 06:10:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6G69pTU029080
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 06:09:51 GMT
Message-ID: <8094849.410511279260591842.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 02:09:51 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6859) Introduce additional statistics to
 FileSystem
In-Reply-To: <3466971.360371279057551878.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889019#action_12889019 ] 

Suresh Srinivas commented on HADOOP-6859:
-----------------------------------------

This patch does not include tests. Tests are done as a part of specific file system implementation. Tests for HDFS will be added in HDFS-1298.

> Introduce additional statistics to FileSystem
> ---------------------------------------------
>
>                 Key: HADOOP-6859
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6859
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6859.patch
>
>
> Currently FileSystem#statistics tracks bytesRead and bytesWritten. Additional statistics that gives summary of operations performed will be useful for tracking file system use.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9562-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 06:10:22 2010
Return-Path: <common-issues-return-9562-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 543 invoked from network); 16 Jul 2010 06:10:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 06:10:22 -0000
Received: (qmail 43285 invoked by uid 500); 16 Jul 2010 06:10:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43104 invoked by uid 500); 16 Jul 2010 06:10:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43092 invoked by uid 99); 16 Jul 2010 06:10:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 06:10:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 06:10:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6G69qaA029101
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 06:09:52 GMT
Message-ID: <8796921.410581279260592624.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 02:09:52 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6859) Introduce additional statistics to
 FileSystem
In-Reply-To: <3466971.360371279057551878.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6859:
------------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

I committed the patch.

> Introduce additional statistics to FileSystem
> ---------------------------------------------
>
>                 Key: HADOOP-6859
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6859
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6859.patch
>
>
> Currently FileSystem#statistics tracks bytesRead and bytesWritten. Additional statistics that gives summary of operations performed will be useful for tracking file system use.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9563-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 06:14:50 2010
Return-Path: <common-issues-return-9563-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 915 invoked from network); 16 Jul 2010 06:14:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 06:14:49 -0000
Received: (qmail 46904 invoked by uid 500); 16 Jul 2010 06:14:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46779 invoked by uid 500); 16 Jul 2010 06:14:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46709 invoked by uid 99); 16 Jul 2010 06:14:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 06:14:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 06:14:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6G66qDq029050
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 06:06:52 GMT
Message-ID: <25877483.410441279260412529.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 02:06:52 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6859) Introduce additional statistics to
 FileSystem
In-Reply-To: <3466971.360371279057551878.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889018#action_12889018 ] 

Suresh Srinivas commented on HADOOP-6859:
-----------------------------------------

Looking at Hudson console output, the javadoc warning is unrelated to this patch. Here are the javadoc warnings:
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:31: warning: sun.security.krb5.Config is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.Config;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:32: warning: sun.security.krb5.KrbException is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.KrbException;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:81: warning: sun.security.krb5.Config is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc]   private static Config kerbConf;
     [exec]   [javadoc]                  ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/SecurityUtil.java:33: warning: sun.security.jgss.krb5.Krb5Util is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.jgss.krb5.Krb5Util;
     [exec]   [javadoc]                              ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/SecurityUtil.java:34: warning: sun.security.krb5.Credentials is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.Credentials;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/SecurityUtil.java:35: warning: sun.security.krb5.PrincipalName is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.PrincipalName;


> Introduce additional statistics to FileSystem
> ---------------------------------------------
>
>                 Key: HADOOP-6859
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6859
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6859.patch
>
>
> Currently FileSystem#statistics tracks bytesRead and bytesWritten. Additional statistics that gives summary of operations performed will be useful for tracking file system use.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9564-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 06:31:52 2010
Return-Path: <common-issues-return-9564-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9380 invoked from network); 16 Jul 2010 06:31:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 06:31:52 -0000
Received: (qmail 54866 invoked by uid 500); 16 Jul 2010 06:31:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54663 invoked by uid 500); 16 Jul 2010 06:31:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54645 invoked by uid 99); 16 Jul 2010 06:31:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 06:31:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 06:31:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6G6Nsaw029174
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 06:23:54 GMT
Message-ID: <21328145.410721279261434576.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 02:23:54 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6859) Introduce additional statistics to
 FileSystem
In-Reply-To: <3466971.360371279057551878.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889022#action_12889022 ] 

Hudson commented on HADOOP-6859:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #326 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/326/])
    HADOOP-6859 - Introduce additional statistics to FileSystem to track file system operations. Contributed by Suresh Srinivas.


> Introduce additional statistics to FileSystem
> ---------------------------------------------
>
>                 Key: HADOOP-6859
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6859
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6859.patch
>
>
> Currently FileSystem#statistics tracks bytesRead and bytesWritten. Additional statistics that gives summary of operations performed will be useful for tracking file system use.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9565-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 09:08:18 2010
Return-Path: <common-issues-return-9565-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48435 invoked from network); 16 Jul 2010 09:08:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 09:08:18 -0000
Received: (qmail 13984 invoked by uid 500); 16 Jul 2010 09:08:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13349 invoked by uid 500); 16 Jul 2010 09:08:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13336 invoked by uid 99); 16 Jul 2010 09:08:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 09:08:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 09:08:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6G90JKS000937
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 09:00:20 GMT
Message-ID: <21486340.413571279270819596.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 05:00:19 -0400 (EDT)
From: "Carl Steinbach (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5303) Oozie, Hadoop Workflow System
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889070#action_12889070 ] 

Carl Steinbach commented on HADOOP-5303:
----------------------------------------

Yahoo!'s Official Oozie Site: http://yahoo.github.com/oozie/

> Oozie, Hadoop Workflow System
> -----------------------------
>
>                 Key: HADOOP-5303
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5303
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>         Attachments: Hadoop_Summit_Oozie.pdf, hws-preso-v1_0_2009FEB22.pdf, hws-spec2009MAR09.pdf, hws-v1_0_2009FEB22.pdf, oozie-0.18.3.o0.1-SNAPSHOT-distro.tar.gz, oozie-patch.txt, oozie-spec-20090521.pdf, oozie-src-20090605.tar.gz, packages.tar.gz
>
>
> This is a proposal for a system specialized in running Hadoop/Pig jobs in a control dependency DAG (Direct Acyclic Graph), a Hadoop workflow application.
> Attached there is a complete specification and a high level overview presentation.
> ----
> *Highlights* 
> A Workflow application is DAG that coordinates the following types of actions: Hadoop, Pig, Ssh, Http, Email and sub-workflows. 
> Flow control operations within the workflow applications can be done using decision, fork and join nodes. Cycles in workflows are not supported.
> Actions and decisions can be parameterized with job properties, actions output (i.e. Hadoop counters, Ssh key/value pairs output) and file information (file exists, file size, etc). Formal parameters are expressed in the workflow definition as {{${VAR}}} variables.
> A Workflow application is a ZIP file that contains the workflow definition (an XML file), all the necessary files to run all the actions: JAR files for Map/Reduce jobs, shells for streaming Map/Reduce jobs, native libraries, Pig scripts, and other resource files.
> Before running a workflow job, the corresponding workflow application must be deployed in HWS.
> Deploying workflow application and running workflow jobs can be done via command line tools, a WS API and a Java API.
> Monitoring the system and workflow jobs can be done via a web console, command line tools, a WS API and a Java API.
> When submitting a workflow job, a set of properties resolving all the formal parameters in the workflow definitions must be provided. This set of properties is a Hadoop configuration.
> Possible states for a workflow jobs are: {{CREATED}}, {{RUNNING}}, {{SUSPENDED}}, {{SUCCEEDED}}, {{KILLED}} and {{FAILED}}.
> In the case of a action failure in a workflow job, depending on the type of failure, HWS will attempt automatic retries, it will request a manual retry or it will fail the workflow job.
> HWS can make HTTP callback notifications on action start/end/failure events and workflow end/failure events.
> In the case of workflow job failure, the workflow job can be resubmitted skipping previously completed actions. Before doing a resubmission the workflow application could be updated with a patch to fix a problem in the workflow application code.
> ----

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9566-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 11:16:20 2010
Return-Path: <common-issues-return-9566-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8584 invoked from network); 16 Jul 2010 11:16:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 11:16:20 -0000
Received: (qmail 94044 invoked by uid 500); 16 Jul 2010 11:16:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93914 invoked by uid 500); 16 Jul 2010 11:16:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93906 invoked by uid 99); 16 Jul 2010 11:16:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 11:16:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 11:16:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6GBFpEb002538
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 11:15:51 GMT
Message-ID: <5680307.416291279278951806.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 07:15:51 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6859) Introduce additional statistics to
 FileSystem
In-Reply-To: <3466971.360371279057551878.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889131#action_12889131 ] 

Hudson commented on HADOOP-6859:
--------------------------------

Integrated in Hadoop-Common-trunk #394 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/394/])
    HADOOP-6859 - Introduce additional statistics to FileSystem to track file system operations. Contributed by Suresh Srinivas.


> Introduce additional statistics to FileSystem
> ---------------------------------------------
>
>                 Key: HADOOP-6859
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6859
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6859.patch
>
>
> Currently FileSystem#statistics tracks bytesRead and bytesWritten. Additional statistics that gives summary of operations performed will be useful for tracking file system use.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9567-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 17:27:17 2010
Return-Path: <common-issues-return-9567-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42518 invoked from network); 16 Jul 2010 17:27:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 17:27:17 -0000
Received: (qmail 33296 invoked by uid 500); 16 Jul 2010 17:27:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33235 invoked by uid 500); 16 Jul 2010 17:27:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33227 invoked by uid 99); 16 Jul 2010 17:27:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 17:27:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 17:27:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6GHQqFt006182
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 17:26:52 GMT
Message-ID: <28552117.422471279301212324.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 13:26:52 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6859) Introduce additional statistics to
 FileSystem
In-Reply-To: <3466971.360371279057551878.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6859:
------------------------------------

    Attachment: HADOOP-6859.y20.patch

y20 version of the patch

> Introduce additional statistics to FileSystem
> ---------------------------------------------
>
>                 Key: HADOOP-6859
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6859
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6859.patch, HADOOP-6859.y20.patch
>
>
> Currently FileSystem#statistics tracks bytesRead and bytesWritten. Additional statistics that gives summary of operations performed will be useful for tracking file system use.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9568-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 18:14:07 2010
Return-Path: <common-issues-return-9568-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62613 invoked from network); 16 Jul 2010 18:14:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 18:14:07 -0000
Received: (qmail 82919 invoked by uid 500); 16 Jul 2010 18:14:07 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82720 invoked by uid 500); 16 Jul 2010 18:14:06 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82695 invoked by uid 99); 16 Jul 2010 18:14:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 18:14:06 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 18:13:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6GI5pEM006612
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 18:05:52 GMT
Message-ID: <204126.423281279303551730.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 14:05:51 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6863) Compile-native still fails on Mac
 OS X.
In-Reply-To: <18053660.404611279230589983.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889248#action_12889248 ] 

Allen Wittenauer commented on HADOOP-6863:
------------------------------------------

I'll try to test this on a multi-arch JVM (Solaris, OS X) today.  

> Compile-native still fails on Mac OS X.
> ---------------------------------------
>
>                 Key: HADOOP-6863
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6863
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>         Attachments: hadoop-6863-20100715.patch
>
>
> I am still getting failures when I try to compile native library on Mac OS X (10.5.8 Leopard). The problems are two fold:
> - Although "aclocal.m4" is changed after HADOOP-3659 is committed, the corresponding "configure" script is not re-generated.
> - It cannot find libjvm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9569-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 18:32:15 2010
Return-Path: <common-issues-return-9569-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78859 invoked from network); 16 Jul 2010 18:32:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 18:32:15 -0000
Received: (qmail 11519 invoked by uid 500); 16 Jul 2010 18:32:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11459 invoked by uid 500); 16 Jul 2010 18:32:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11451 invoked by uid 99); 16 Jul 2010 18:32:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 18:32:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 18:32:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6GIVoYk006953
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 18:31:50 GMT
Message-ID: <32850767.423991279305110123.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 14:31:50 -0400 (EDT)
From: "Mahadev konar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6834) TFile.append compares initial key
 against null lastKey
In-Reply-To: <10707501.8491277191496663.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889261#action_12889261 ] 

Mahadev konar commented on HADOOP-6834:
---------------------------------------

+1 the patch looks good.

> TFile.append compares initial key against null lastKey  
> --------------------------------------------------------
>
>                 Key: HADOOP-6834
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6834
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Ahad Rana
>            Assignee: Hong Tang
>         Attachments: hadoop-6834-20100715.patch
>
>
> The following code in TFile.KeyReigster.close: 
>             byte[] lastKey = lastKeyBufferOS.getBuffer();
>             int lastLen = lastKeyBufferOS.size();
>             if (tfileMeta.getComparator().compare(key, 0, len, lastKey, 0,
>                 lastLen) < 0) {
>               throw new IOException("Keys are not added in sorted order");
>             }
> compares the initial  key (passed in via  TFile.Writer.append) against a technically NULL lastKey. lastKey is not initialized until after the first call to TFile.Writer.append. The underlying RawComparator interface used for comparisons does not stipulate the proper behavior when either length 1  or length 2 is zero. In the case of LongWritable, its WritableComparator implementation does an unsafe read on the passed in byte arrays b1 and b2. Since TFile pre-allocates the buffer used for storing lastKey, this passes a valid buffer with zero count to LongWritable's comparator, which ignores length and thus produces incorrect results. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9570-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 18:40:20 2010
Return-Path: <common-issues-return-9570-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83217 invoked from network); 16 Jul 2010 18:40:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 18:40:16 -0000
Received: (qmail 28596 invoked by uid 500); 16 Jul 2010 18:40:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28556 invoked by uid 500); 16 Jul 2010 18:40:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28548 invoked by uid 99); 16 Jul 2010 18:40:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 18:40:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 18:40:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6GIdpdB007099
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 18:39:51 GMT
Message-ID: <31180623.424261279305591868.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 14:39:51 -0400 (EDT)
From: "Mahadev konar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6834) TFile.append compares initial key
 against null lastKey
In-Reply-To: <10707501.8491277191496663.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mahadev konar updated HADOOP-6834:
----------------------------------

           Status: Resolved  (was: Patch Available)
     Hadoop Flags: [Reviewed]
    Fix Version/s: 0.22.0
       Resolution: Fixed

> TFile.append compares initial key against null lastKey  
> --------------------------------------------------------
>
>                 Key: HADOOP-6834
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6834
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Ahad Rana
>            Assignee: Hong Tang
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6834-20100715.patch
>
>
> The following code in TFile.KeyReigster.close: 
>             byte[] lastKey = lastKeyBufferOS.getBuffer();
>             int lastLen = lastKeyBufferOS.size();
>             if (tfileMeta.getComparator().compare(key, 0, len, lastKey, 0,
>                 lastLen) < 0) {
>               throw new IOException("Keys are not added in sorted order");
>             }
> compares the initial  key (passed in via  TFile.Writer.append) against a technically NULL lastKey. lastKey is not initialized until after the first call to TFile.Writer.append. The underlying RawComparator interface used for comparisons does not stipulate the proper behavior when either length 1  or length 2 is zero. In the case of LongWritable, its WritableComparator implementation does an unsafe read on the passed in byte arrays b1 and b2. Since TFile pre-allocates the buffer used for storing lastKey, this passes a valid buffer with zero count to LongWritable's comparator, which ignores length and thus produces incorrect results. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9571-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 19:02:48 2010
Return-Path: <common-issues-return-9571-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91551 invoked from network); 16 Jul 2010 19:02:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 19:02:48 -0000
Received: (qmail 57007 invoked by uid 500); 16 Jul 2010 19:02:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56934 invoked by uid 500); 16 Jul 2010 19:02:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56926 invoked by uid 99); 16 Jul 2010 19:02:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 19:02:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 19:02:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6GIsrqP007200
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 18:54:53 GMT
Message-ID: <25401572.424461279306493652.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 14:54:53 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6834) TFile.append compares initial key
 against null lastKey
In-Reply-To: <10707501.8491277191496663.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889268#action_12889268 ] 

Hudson commented on HADOOP-6834:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #327 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/327/])
    HADOOP-6834. TFile.append compares initial key against null lastKey (hong tang via mahadev)


> TFile.append compares initial key against null lastKey  
> --------------------------------------------------------
>
>                 Key: HADOOP-6834
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6834
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Ahad Rana
>            Assignee: Hong Tang
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6834-20100715.patch
>
>
> The following code in TFile.KeyReigster.close: 
>             byte[] lastKey = lastKeyBufferOS.getBuffer();
>             int lastLen = lastKeyBufferOS.size();
>             if (tfileMeta.getComparator().compare(key, 0, len, lastKey, 0,
>                 lastLen) < 0) {
>               throw new IOException("Keys are not added in sorted order");
>             }
> compares the initial  key (passed in via  TFile.Writer.append) against a technically NULL lastKey. lastKey is not initialized until after the first call to TFile.Writer.append. The underlying RawComparator interface used for comparisons does not stipulate the proper behavior when either length 1  or length 2 is zero. In the case of LongWritable, its WritableComparator implementation does an unsafe read on the passed in byte arrays b1 and b2. Since TFile pre-allocates the buffer used for storing lastKey, this passes a valid buffer with zero count to LongWritable's comparator, which ignores length and thus produces incorrect results. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9572-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 19:51:45 2010
Return-Path: <common-issues-return-9572-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12245 invoked from network); 16 Jul 2010 19:51:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 19:51:45 -0000
Received: (qmail 15382 invoked by uid 500); 16 Jul 2010 19:51:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15308 invoked by uid 500); 16 Jul 2010 19:51:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15287 invoked by uid 99); 16 Jul 2010 19:51:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 19:51:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 19:51:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6GJhoXs007621
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 19:43:50 GMT
Message-ID: <17930831.425261279309430174.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 15:43:50 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6860)  'compile-fault-inject' should
 never be called directly.
In-Reply-To: <25103470.380431279132431950.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889285#action_12889285 ] 

Konstantin Boudnik commented on HADOOP-6860:
--------------------------------------------

Apparently, {{test-patch.sh}} won't catch anything in this kind of situation (as proven by HDFS-1299). So, I have ran build with a couple of different targets to make sure it is still Ok. 
I'll commit without validation by Hudson it if no objections.

>  'compile-fault-inject' should never be called directly.
> --------------------------------------------------------
>
>                 Key: HADOOP-6860
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6860
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>            Priority: Minor
>         Attachments: HADOOP-6860.patch
>
>
> Similar to HDFS-1299 a  helper target  'compile-fault-inject' should never be called directly.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9573-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 20:37:46 2010
Return-Path: <common-issues-return-9573-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26460 invoked from network); 16 Jul 2010 20:37:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 20:37:46 -0000
Received: (qmail 81378 invoked by uid 500); 16 Jul 2010 20:37:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81298 invoked by uid 500); 16 Jul 2010 20:37:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81272 invoked by uid 99); 16 Jul 2010 20:37:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 20:37:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 20:37:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6GKTpO0008146
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 20:29:51 GMT
Message-ID: <32239872.426301279312191078.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 16:29:51 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6860)  'compile-fault-inject' should never
 be called directly.
In-Reply-To: <25103470.380431279132431950.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik resolved HADOOP-6860.
----------------------------------------

     Hadoop Flags: [Reviewed]
    Fix Version/s: 0.21.0
       Resolution: Fixed

I have committed this.

>  'compile-fault-inject' should never be called directly.
> --------------------------------------------------------
>
>                 Key: HADOOP-6860
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6860
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6860.patch
>
>
> Similar to HDFS-1299 a  helper target  'compile-fault-inject' should never be called directly.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9574-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 20:54:53 2010
Return-Path: <common-issues-return-9574-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31536 invoked from network); 16 Jul 2010 20:54:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 20:54:46 -0000
Received: (qmail 95889 invoked by uid 500); 16 Jul 2010 20:54:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95833 invoked by uid 500); 16 Jul 2010 20:54:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95425 invoked by uid 99); 16 Jul 2010 20:54:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 20:54:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 20:54:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6GKkp5v008302
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 20:46:51 GMT
Message-ID: <20496167.426591279313211014.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 16:46:51 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6860)  'compile-fault-inject' should
 never be called directly.
In-Reply-To: <25103470.380431279132431950.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889315#action_12889315 ] 

Hudson commented on HADOOP-6860:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #328 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/328/])
    HADOOP-6860. 'compile-fault-inject' should never be called directly. Contributed by Konstantin Boudnik.


>  'compile-fault-inject' should never be called directly.
> --------------------------------------------------------
>
>                 Key: HADOOP-6860
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6860
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6860.patch
>
>
> Similar to HDFS-1299 a  helper target  'compile-fault-inject' should never be called directly.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9575-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 21:12:47 2010
Return-Path: <common-issues-return-9575-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46935 invoked from network); 16 Jul 2010 21:12:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 21:12:46 -0000
Received: (qmail 27685 invoked by uid 500); 16 Jul 2010 21:12:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27612 invoked by uid 500); 16 Jul 2010 21:12:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27604 invoked by uid 99); 16 Jul 2010 21:12:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 21:12:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 21:12:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6GL4qJp008605
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 21:04:52 GMT
Message-ID: <21778001.427181279314292151.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 17:04:52 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6863) Compile-native still fails on Mac
 OS X.
In-Reply-To: <18053660.404611279230589983.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889327#action_12889327 ] 

Hong Tang commented on HADOOP-6863:
-----------------------------------

@allen, thanks for helping out.

> Compile-native still fails on Mac OS X.
> ---------------------------------------
>
>                 Key: HADOOP-6863
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6863
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>         Attachments: hadoop-6863-20100715.patch
>
>
> I am still getting failures when I try to compile native library on Mac OS X (10.5.8 Leopard). The problems are two fold:
> - Although "aclocal.m4" is changed after HADOOP-3659 is committed, the corresponding "configure" script is not re-generated.
> - It cannot find libjvm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9576-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 22:23:15 2010
Return-Path: <common-issues-return-9576-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67210 invoked from network); 16 Jul 2010 22:23:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 22:23:15 -0000
Received: (qmail 81784 invoked by uid 500); 16 Jul 2010 22:23:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81728 invoked by uid 500); 16 Jul 2010 22:23:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81720 invoked by uid 99); 16 Jul 2010 22:23:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 22:23:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 22:23:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6GMMoV5009536
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 22:22:50 GMT
Message-ID: <4089270.429351279318970358.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 18:22:50 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6861) Method in Credentials to read and
 write a token storage file.
In-Reply-To: <21462038.384681279145270696.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889361#action_12889361 ] 

Devaraj Das commented on HADOOP-6861:
-------------------------------------

+1

> Method in Credentials to read and write a token storage file.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6861
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6861
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6861.1.patch, HADOOP-6861.2.patch
>
>
> The jira covers the changes in common corresponding to MAPREDUCE-1566. 
> This jira adds  new non-static methods in Credentials to read and write token storage file. A method to copy tokens from another credential object is also added. Static method readTokensAndLoadInUGI is removed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9577-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 22:25:19 2010
Return-Path: <common-issues-return-9577-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67997 invoked from network); 16 Jul 2010 22:25:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 22:25:18 -0000
Received: (qmail 85467 invoked by uid 500); 16 Jul 2010 22:25:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85132 invoked by uid 500); 16 Jul 2010 22:25:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85071 invoked by uid 99); 16 Jul 2010 22:25:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 22:25:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 22:25:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6GMOr0W009626
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 22:24:53 GMT
Message-ID: <9859836.429631279319093943.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 18:24:53 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6670) UserGroupInformation doesn't support
 use in hash tables
In-Reply-To: <242202762.620161270078590059.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6670:
------------------------------

    Status: Open  (was: Patch Available)

> UserGroupInformation doesn't support use in hash tables
> -------------------------------------------------------
>
>                 Key: HADOOP-6670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6670
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 6670-gridmix-test.bugfix.patch, c6670-02.patch, c6670-03.patch, fs-close.patch, fs-close.patch, fs-close.patch
>
>
> The UserGroupInformation objects are mutable, but they are used as keys in hash tables. This leads to serious problems in the FileSystem cache and RPC connection cache. We need to change the hashCode to be the identity hash code of the Subject and change equals to use == between the Subjects.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9578-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 22:32:15 2010
Return-Path: <common-issues-return-9578-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69649 invoked from network); 16 Jul 2010 22:32:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 22:32:15 -0000
Received: (qmail 92520 invoked by uid 500); 16 Jul 2010 22:32:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92465 invoked by uid 500); 16 Jul 2010 22:32:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92457 invoked by uid 99); 16 Jul 2010 22:32:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 22:32:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 22:32:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6GMVox3009733
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 22:31:50 GMT
Message-ID: <25411009.429961279319510332.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 18:31:50 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6855) Add ability to get groups for ACLs
 from 'getent netgroup'
In-Reply-To: <23521712.275021278635570824.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889373#action_12889373 ] 

Allen Wittenauer commented on HADOOP-6855:
------------------------------------------

-1 

getent is not portable.

> Add ability to get groups for ACLs from 'getent netgroup'
> ---------------------------------------------------------
>
>                 Key: HADOOP-6855
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6855
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6855-0.20-1.patch, HADOOP-6855-0.20-2.patch, HADOOP-6855-0.20.patch
>
>
> Add ability to specify netgroups in ACLs (see class AccessControlList.java). Membership of users in netgroups will be determined by running 'getent negroups $groupName'. Netgroups will be differentiated from regular unix groups by having '@' as a first character.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9579-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 22:32:15 2010
Return-Path: <common-issues-return-9579-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69693 invoked from network); 16 Jul 2010 22:32:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 22:32:15 -0000
Received: (qmail 92679 invoked by uid 500); 16 Jul 2010 22:32:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92475 invoked by uid 500); 16 Jul 2010 22:32:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92466 invoked by uid 99); 16 Jul 2010 22:32:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 22:32:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 22:32:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6GMVoIA009742
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 22:31:50 GMT
Message-ID: <8337602.429991279319510916.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 18:31:50 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6864) Provide a JNI-based implementation
 of ShellBasedUnixGroupsNetgroupMapping (implementation of
 GroupMappingServiceProvider)
In-Reply-To: <17354207.429501279319091600.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889374#action_12889374 ] 

Boris Shkolnik commented on HADOOP-6864:
----------------------------------------

please review the possibility of using JNA instead of JNI (https://jna.dev.java.net/).

> Provide a JNI-based implementation of ShellBasedUnixGroupsNetgroupMapping (implementation of GroupMappingServiceProvider)
> -------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6864
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6864
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Erik Steffl
>             Fix For: 0.22.0
>
>
> The netgroups implementation of GroupMappingServiceProvider (see ShellBasedUnixGroupsNetgroupMapping.java) does a fork of a unix command to get the netgroups of a user. Since the group resolution happens in the servers, this might be costly. This jira aims at providing a JNI-based implementation for GroupMappingServiceProvider.
> Note that this is similar to what https://issues.apache.org/jira/browse/HADOOP-6818 does for implementation of GroupMappingServiceProvider that  supports only unix groups.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9580-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 22:32:45 2010
Return-Path: <common-issues-return-9580-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69739 invoked from network); 16 Jul 2010 22:32:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 22:32:45 -0000
Received: (qmail 92835 invoked by uid 500); 16 Jul 2010 22:32:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92794 invoked by uid 500); 16 Jul 2010 22:32:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92784 invoked by uid 99); 16 Jul 2010 22:32:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 22:32:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 22:32:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6GMOpsM009587
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 22:24:51 GMT
Message-ID: <17354207.429501279319091600.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 18:24:51 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6864) Provide a JNI-based implementation of
 ShellBasedUnixGroupsNetgroupMapping (implementation of
 GroupMappingServiceProvider)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Provide a JNI-based implementation of ShellBasedUnixGroupsNetgroupMapping (implementation of GroupMappingServiceProvider)
-------------------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-6864
                 URL: https://issues.apache.org/jira/browse/HADOOP-6864
             Project: Hadoop Common
          Issue Type: Improvement
          Components: security
            Reporter: Erik Steffl
             Fix For: 0.22.0


The netgroups implementation of GroupMappingServiceProvider (see ShellBasedUnixGroupsNetgroupMapping.java) does a fork of a unix command to get the netgroups of a user. Since the group resolution happens in the servers, this might be costly. This jira aims at providing a JNI-based implementation for GroupMappingServiceProvider.

Note that this is similar to what https://issues.apache.org/jira/browse/HADOOP-6818 does for implementation of GroupMappingServiceProvider that  supports only unix groups.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9581-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 22:32:47 2010
Return-Path: <common-issues-return-9581-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69843 invoked from network); 16 Jul 2010 22:32:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 22:32:46 -0000
Received: (qmail 93118 invoked by uid 500); 16 Jul 2010 22:32:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93065 invoked by uid 500); 16 Jul 2010 22:32:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93057 invoked by uid 99); 16 Jul 2010 22:32:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 22:32:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 22:32:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6GMOrtm009611
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 22:24:53 GMT
Message-ID: <23781079.429581279319093293.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 18:24:53 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6670) UserGroupInformation doesn't support
 use in hash tables
In-Reply-To: <242202762.620161270078590059.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6670:
------------------------------

    Attachment: c6670-03.patch

Attaching a new patch that removes the redundant checking in test per Jitendra's comment. Otherwise, it stays the same.

> UserGroupInformation doesn't support use in hash tables
> -------------------------------------------------------
>
>                 Key: HADOOP-6670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6670
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 6670-gridmix-test.bugfix.patch, c6670-02.patch, c6670-03.patch, fs-close.patch, fs-close.patch, fs-close.patch
>
>
> The UserGroupInformation objects are mutable, but they are used as keys in hash tables. This leads to serious problems in the FileSystem cache and RPC connection cache. We need to change the hashCode to be the identity hash code of the Subject and change equals to use == between the Subjects.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9582-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 22:32:50 2010
Return-Path: <common-issues-return-9582-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69906 invoked from network); 16 Jul 2010 22:32:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 22:32:50 -0000
Received: (qmail 93276 invoked by uid 500); 16 Jul 2010 22:32:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93232 invoked by uid 500); 16 Jul 2010 22:32:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93224 invoked by uid 99); 16 Jul 2010 22:32:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 22:32:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 22:32:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6GMOuj6009665
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 22:24:56 GMT
Message-ID: <19357743.429761279319096834.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 18:24:56 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6670) UserGroupInformation doesn't support
 use in hash tables
In-Reply-To: <242202762.620161270078590059.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6670:
------------------------------

    Status: Patch Available  (was: Open)

> UserGroupInformation doesn't support use in hash tables
> -------------------------------------------------------
>
>                 Key: HADOOP-6670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6670
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 6670-gridmix-test.bugfix.patch, c6670-02.patch, c6670-03.patch, fs-close.patch, fs-close.patch, fs-close.patch
>
>
> The UserGroupInformation objects are mutable, but they are used as keys in hash tables. This leads to serious problems in the FileSystem cache and RPC connection cache. We need to change the hashCode to be the identity hash code of the Subject and change equals to use == between the Subjects.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9583-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 23:09:47 2010
Return-Path: <common-issues-return-9583-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85620 invoked from network); 16 Jul 2010 23:09:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 23:09:47 -0000
Received: (qmail 29177 invoked by uid 500); 16 Jul 2010 23:09:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29096 invoked by uid 500); 16 Jul 2010 23:09:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29087 invoked by uid 99); 16 Jul 2010 23:09:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 23:09:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 23:09:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6GN1qCG010104
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 23:01:52 GMT
Message-ID: <13771888.430731279321312602.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 19:01:52 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6670) UserGroupInformation doesn't
 support use in hash tables
In-Reply-To: <242202762.620161270078590059.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889385#action_12889385 ] 

Hadoop QA commented on HADOOP-6670:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12449720/c6670-03.patch
  against trunk revision 964939.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/620/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/620/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/620/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/620/console

This message is automatically generated.

> UserGroupInformation doesn't support use in hash tables
> -------------------------------------------------------
>
>                 Key: HADOOP-6670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6670
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 6670-gridmix-test.bugfix.patch, c6670-02.patch, c6670-03.patch, fs-close.patch, fs-close.patch, fs-close.patch
>
>
> The UserGroupInformation objects are mutable, but they are used as keys in hash tables. This leads to serious problems in the FileSystem cache and RPC connection cache. We need to change the hashCode to be the identity hash code of the Subject and change equals to use == between the Subjects.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9584-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 23:16:14 2010
Return-Path: <common-issues-return-9584-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86900 invoked from network); 16 Jul 2010 23:16:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 23:16:14 -0000
Received: (qmail 33293 invoked by uid 500); 16 Jul 2010 23:16:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33232 invoked by uid 500); 16 Jul 2010 23:16:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33224 invoked by uid 99); 16 Jul 2010 23:16:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 23:16:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 23:16:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6GNFnSt010246
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 23:15:49 GMT
Message-ID: <33172567.431091279322149866.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 19:15:49 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6805) add buildDTServiceName method to
 SecurityUtil (as part of MAPREDUCE-1718)
In-Reply-To: <5393650.148181275524636022.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6805:
-----------------------------------

    Attachment: HADOOP-6805.patch

> add buildDTServiceName method to SecurityUtil (as part of MAPREDUCE-1718)
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6805
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6805
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6805.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9585-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 23:16:15 2010
Return-Path: <common-issues-return-9585-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86954 invoked from network); 16 Jul 2010 23:16:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 23:16:15 -0000
Received: (qmail 33447 invoked by uid 500); 16 Jul 2010 23:16:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33398 invoked by uid 500); 16 Jul 2010 23:16:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33390 invoked by uid 99); 16 Jul 2010 23:16:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 23:16:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 23:16:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6GNFo04010261
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 23:15:51 GMT
Message-ID: <21250890.431141279322150974.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 19:15:50 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6805) add buildDTServiceName method to
 SecurityUtil (as part of MAPREDUCE-1718)
In-Reply-To: <5393650.148181275524636022.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6805:
-----------------------------------

    Status: Patch Available  (was: Open)

> add buildDTServiceName method to SecurityUtil (as part of MAPREDUCE-1718)
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6805
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6805
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6805.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9586-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 23:18:15 2010
Return-Path: <common-issues-return-9586-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87292 invoked from network); 16 Jul 2010 23:18:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 23:18:15 -0000
Received: (qmail 34877 invoked by uid 500); 16 Jul 2010 23:18:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34815 invoked by uid 500); 16 Jul 2010 23:18:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34807 invoked by uid 99); 16 Jul 2010 23:18:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 23:18:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 23:18:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6GNHoQB010288
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 23:17:50 GMT
Message-ID: <2336786.431211279322270207.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 19:17:50 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6805) add buildDTServiceName method to
 SecurityUtil (as part of MAPREDUCE-1718)
In-Reply-To: <5393650.148181275524636022.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6805:
--------------------------------

    Hadoop Flags: [Reviewed]

+1

> add buildDTServiceName method to SecurityUtil (as part of MAPREDUCE-1718)
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6805
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6805
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6805.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9587-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 23:40:16 2010
Return-Path: <common-issues-return-9587-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90938 invoked from network); 16 Jul 2010 23:40:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 23:40:15 -0000
Received: (qmail 47367 invoked by uid 500); 16 Jul 2010 23:40:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47298 invoked by uid 500); 16 Jul 2010 23:40:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47289 invoked by uid 99); 16 Jul 2010 23:40:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 23:40:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 23:40:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6GNdoYg010535
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 23:39:50 GMT
Message-ID: <10268703.431871279323590475.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 19:39:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6805) add buildDTServiceName method to
 SecurityUtil (as part of MAPREDUCE-1718)
In-Reply-To: <5393650.148181275524636022.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889410#action_12889410 ] 

Hadoop QA commented on HADOOP-6805:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12449729/HADOOP-6805.patch
  against trunk revision 964939.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/621/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/621/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/621/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/621/console

This message is automatically generated.

> add buildDTServiceName method to SecurityUtil (as part of MAPREDUCE-1718)
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6805
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6805
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6805.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9588-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 16 23:41:46 2010
Return-Path: <common-issues-return-9588-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91678 invoked from network); 16 Jul 2010 23:41:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jul 2010 23:41:46 -0000
Received: (qmail 48142 invoked by uid 500); 16 Jul 2010 23:41:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48117 invoked by uid 500); 16 Jul 2010 23:41:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48109 invoked by uid 99); 16 Jul 2010 23:41:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 23:41:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Jul 2010 23:41:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6GNXpHC010436
	for <common-issues@hadoop.apache.org>; Fri, 16 Jul 2010 23:33:51 GMT
Message-ID: <16816112.431601279323231785.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 19:33:51 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6855) Add ability to get groups for ACLs
 from 'getent netgroup'
In-Reply-To: <23521712.275021278635570824.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889405#action_12889405 ] 

Erik Steffl commented on HADOOP-6855:
-------------------------------------

Re https://issues.apache.org/jira/browse/HADOOP-6855?focusedCommentId=12889373&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12889373

This patch is adding an optional group to users mapping mechanism for users who want to use getent (often used to get group to users mapping from LDAP). Users who want unix group mapping (itself not entirely portable) can continue to use existing mechanism of group to users mapping.

> Add ability to get groups for ACLs from 'getent netgroup'
> ---------------------------------------------------------
>
>                 Key: HADOOP-6855
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6855
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6855-0.20-1.patch, HADOOP-6855-0.20-2.patch, HADOOP-6855-0.20.patch
>
>
> Add ability to specify netgroups in ACLs (see class AccessControlList.java). Membership of users in netgroups will be determined by running 'getent negroups $groupName'. Netgroups will be differentiated from regular unix groups by having '@' as a first character.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9589-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 17 00:10:17 2010
Return-Path: <common-issues-return-9589-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3851 invoked from network); 17 Jul 2010 00:10:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jul 2010 00:10:17 -0000
Received: (qmail 71366 invoked by uid 500); 17 Jul 2010 00:10:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71250 invoked by uid 500); 17 Jul 2010 00:10:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71242 invoked by uid 99); 17 Jul 2010 00:10:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Jul 2010 00:10:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Jul 2010 00:10:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6H09qhL010820
	for <common-issues@hadoop.apache.org>; Sat, 17 Jul 2010 00:09:52 GMT
Message-ID: <24679136.432491279325392078.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 20:09:52 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6855) Add ability to get groups for ACLs
 from 'getent netgroup'
In-Reply-To: <23521712.275021278635570824.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889425#action_12889425 ] 

Allen Wittenauer commented on HADOOP-6855:
------------------------------------------

LDAP and netgroups are portable. For netgroups, most Unices have getnetgrent() and friends.  For LDAP, you can query it natively from Java with JNDI. 

The problem is the getent command itself.  It is a SysV command, so many BSD's do not have it, including OS X.

> Add ability to get groups for ACLs from 'getent netgroup'
> ---------------------------------------------------------
>
>                 Key: HADOOP-6855
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6855
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6855-0.20-1.patch, HADOOP-6855-0.20-2.patch, HADOOP-6855-0.20.patch
>
>
> Add ability to specify netgroups in ACLs (see class AccessControlList.java). Membership of users in netgroups will be determined by running 'getent negroups $groupName'. Netgroups will be differentiated from regular unix groups by having '@' as a first character.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9590-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 17 01:44:15 2010
Return-Path: <common-issues-return-9590-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24243 invoked from network); 17 Jul 2010 01:44:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jul 2010 01:44:15 -0000
Received: (qmail 2809 invoked by uid 500); 17 Jul 2010 01:44:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2735 invoked by uid 500); 17 Jul 2010 01:44:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2727 invoked by uid 99); 17 Jul 2010 01:44:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Jul 2010 01:44:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Jul 2010 01:44:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6H1ho5s013452
	for <common-issues@hadoop.apache.org>; Sat, 17 Jul 2010 01:43:50 GMT
Message-ID: <24782594.433541279331030115.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 21:43:50 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6670) UserGroupInformation doesn't support
 use in hash tables
In-Reply-To: <242202762.620161270078590059.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6670:
--------------------------------

           Status: Resolved  (was: Patch Available)
    Fix Version/s: 0.22.0
       Resolution: Fixed

I just committed this. Thanks, Owen and Kan!

> UserGroupInformation doesn't support use in hash tables
> -------------------------------------------------------
>
>                 Key: HADOOP-6670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6670
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.22.0
>
>         Attachments: 6670-gridmix-test.bugfix.patch, c6670-02.patch, c6670-03.patch, fs-close.patch, fs-close.patch, fs-close.patch
>
>
> The UserGroupInformation objects are mutable, but they are used as keys in hash tables. This leads to serious problems in the FileSystem cache and RPC connection cache. We need to change the hashCode to be the identity hash code of the Subject and change equals to use == between the Subjects.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9591-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 17 02:11:46 2010
Return-Path: <common-issues-return-9591-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31128 invoked from network); 17 Jul 2010 02:11:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jul 2010 02:11:46 -0000
Received: (qmail 13380 invoked by uid 500); 17 Jul 2010 02:11:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13329 invoked by uid 500); 17 Jul 2010 02:11:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13316 invoked by uid 99); 17 Jul 2010 02:11:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Jul 2010 02:11:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Jul 2010 02:11:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6H23pNI013527
	for <common-issues@hadoop.apache.org>; Sat, 17 Jul 2010 02:03:51 GMT
Message-ID: <19752579.433611279332231667.JavaMail.jira@thor>
Date: Fri, 16 Jul 2010 22:03:51 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6670) UserGroupInformation doesn't
 support use in hash tables
In-Reply-To: <242202762.620161270078590059.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889451#action_12889451 ] 

Hudson commented on HADOOP-6670:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #329 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/329/])
    HADOOP-6670. Use the UserGroupInformation's Subject as the criteria for equals and hashCode. Contributed by Owen O'Malley and Kan Zhang.


> UserGroupInformation doesn't support use in hash tables
> -------------------------------------------------------
>
>                 Key: HADOOP-6670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6670
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.22.0
>
>         Attachments: 6670-gridmix-test.bugfix.patch, c6670-02.patch, c6670-03.patch, fs-close.patch, fs-close.patch, fs-close.patch
>
>
> The UserGroupInformation objects are mutable, but they are used as keys in hash tables. This leads to serious problems in the FileSystem cache and RPC connection cache. We need to change the hashCode to be the identity hash code of the Subject and change equals to use == between the Subjects.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9592-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 17 09:09:19 2010
Return-Path: <common-issues-return-9592-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11541 invoked from network); 17 Jul 2010 09:09:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jul 2010 09:09:18 -0000
Received: (qmail 18848 invoked by uid 500); 17 Jul 2010 09:09:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18483 invoked by uid 500); 17 Jul 2010 09:09:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18475 invoked by uid 99); 17 Jul 2010 09:09:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Jul 2010 09:09:14 +0000
X-ASF-Spam-Status: No, hits=-1996.4 required=10.0
	tests=ALL_TRUSTED,FS_REPLICA
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Jul 2010 09:09:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6H98nbk015675
	for <common-issues@hadoop.apache.org>; Sat, 17 Jul 2010 09:08:50 GMT
Message-ID: <14555596.434731279357729812.JavaMail.jira@thor>
Date: Sat, 17 Jul 2010 05:08:49 -0400 (EDT)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6857) FsShell should report raw disk usage
 including replication factor
In-Reply-To: <1268643.319901278914569769.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-6857:
-----------------------------------

    Status: Patch Available  (was: Open)

> FsShell should report raw disk usage including replication factor
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6857
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6857
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.20.2
>            Reporter: Alex Kozlov
>         Attachments: show-space-consumed.txt
>
>
> Currently FsShell report HDFS usage with "hadoop fs -dus <path>" command.  Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe "hadoop fs -dus -raw <path>"?).  This will allow to assess resource usage more accurately.  -- Alex K

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9593-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 17 09:14:47 2010
Return-Path: <common-issues-return-9593-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11931 invoked from network); 17 Jul 2010 09:14:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jul 2010 09:14:47 -0000
Received: (qmail 20064 invoked by uid 500); 17 Jul 2010 09:14:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19904 invoked by uid 500); 17 Jul 2010 09:14:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19886 invoked by uid 99); 17 Jul 2010 09:14:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Jul 2010 09:14:44 +0000
X-ASF-Spam-Status: No, hits=-1996.4 required=10.0
	tests=ALL_TRUSTED,FS_REPLICA
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Jul 2010 09:14:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6H96o3d015668
	for <common-issues@hadoop.apache.org>; Sat, 17 Jul 2010 09:06:50 GMT
Message-ID: <14058176.434711279357610582.JavaMail.jira@thor>
Date: Sat, 17 Jul 2010 05:06:50 -0400 (EDT)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6857) FsShell should report raw disk usage
 including replication factor
In-Reply-To: <1268643.319901278914569769.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HADOOP-6857:
-----------------------------------

    Attachment: show-space-consumed.txt

This patch adds a new column to the output of "hadoop fs -du" and "hadoop fs -dus" which shows the disk space consumed (file size * per-file replication factor) of the paths matched by these commands.

> FsShell should report raw disk usage including replication factor
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6857
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6857
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.20.2
>            Reporter: Alex Kozlov
>         Attachments: show-space-consumed.txt
>
>
> Currently FsShell report HDFS usage with "hadoop fs -dus <path>" command.  Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe "hadoop fs -dus -raw <path>"?).  This will allow to assess resource usage more accurately.  -- Alex K

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9594-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 17 09:29:17 2010
Return-Path: <common-issues-return-9594-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13576 invoked from network); 17 Jul 2010 09:29:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jul 2010 09:29:17 -0000
Received: (qmail 29259 invoked by uid 500); 17 Jul 2010 09:29:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29125 invoked by uid 500); 17 Jul 2010 09:29:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29114 invoked by uid 99); 17 Jul 2010 09:29:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Jul 2010 09:29:14 +0000
X-ASF-Spam-Status: No, hits=-1996.4 required=10.0
	tests=ALL_TRUSTED,FS_REPLICA
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Jul 2010 09:29:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6H9Sopt015740
	for <common-issues@hadoop.apache.org>; Sat, 17 Jul 2010 09:28:50 GMT
Message-ID: <9951569.434821279358930292.JavaMail.jira@thor>
Date: Sat, 17 Jul 2010 05:28:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6857) FsShell should report raw disk
 usage including replication factor
In-Reply-To: <1268643.319901278914569769.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889481#action_12889481 ] 

Hadoop QA commented on HADOOP-6857:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12449749/show-space-consumed.txt
  against trunk revision 964993.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/622/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/622/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/622/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/622/console

This message is automatically generated.

> FsShell should report raw disk usage including replication factor
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6857
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6857
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.20.2
>            Reporter: Alex Kozlov
>         Attachments: show-space-consumed.txt
>
>
> Currently FsShell report HDFS usage with "hadoop fs -dus <path>" command.  Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe "hadoop fs -dus -raw <path>"?).  This will allow to assess resource usage more accurately.  -- Alex K

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9595-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 17 11:15:11 2010
Return-Path: <common-issues-return-9595-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47689 invoked from network); 17 Jul 2010 11:15:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jul 2010 11:15:11 -0000
Received: (qmail 66632 invoked by uid 500); 17 Jul 2010 11:15:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66499 invoked by uid 500); 17 Jul 2010 11:15:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66462 invoked by uid 99); 17 Jul 2010 11:15:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Jul 2010 11:15:07 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Jul 2010 11:14:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6HBDo8p016335
	for <common-issues@hadoop.apache.org>; Sat, 17 Jul 2010 11:13:51 GMT
Message-ID: <6021307.435491279365230914.JavaMail.jira@thor>
Date: Sat, 17 Jul 2010 07:13:50 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6670) UserGroupInformation doesn't
 support use in hash tables
In-Reply-To: <242202762.620161270078590059.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889493#action_12889493 ] 

Hudson commented on HADOOP-6670:
--------------------------------

Integrated in Hadoop-Common-trunk #395 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/395/])
    HADOOP-6670. Use the UserGroupInformation's Subject as the criteria for equals and hashCode. Contributed by Owen O'Malley and Kan Zhang.


> UserGroupInformation doesn't support use in hash tables
> -------------------------------------------------------
>
>                 Key: HADOOP-6670
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6670
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.22.0
>
>         Attachments: 6670-gridmix-test.bugfix.patch, c6670-02.patch, c6670-03.patch, fs-close.patch, fs-close.patch, fs-close.patch
>
>
> The UserGroupInformation objects are mutable, but they are used as keys in hash tables. This leads to serious problems in the FileSystem cache and RPC connection cache. We need to change the hashCode to be the identity hash code of the Subject and change equals to use == between the Subjects.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9596-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 17 11:15:13 2010
Return-Path: <common-issues-return-9596-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47742 invoked from network); 17 Jul 2010 11:15:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jul 2010 11:15:13 -0000
Received: (qmail 66774 invoked by uid 500); 17 Jul 2010 11:15:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66504 invoked by uid 500); 17 Jul 2010 11:15:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66461 invoked by uid 99); 17 Jul 2010 11:15:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Jul 2010 11:15:07 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Jul 2010 11:14:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6HBDpqA016344
	for <common-issues@hadoop.apache.org>; Sat, 17 Jul 2010 11:13:51 GMT
Message-ID: <898122.435521279365231620.JavaMail.jira@thor>
Date: Sat, 17 Jul 2010 07:13:51 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6834) TFile.append compares initial key
 against null lastKey
In-Reply-To: <10707501.8491277191496663.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889494#action_12889494 ] 

Hudson commented on HADOOP-6834:
--------------------------------

Integrated in Hadoop-Common-trunk #395 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/395/])
    HADOOP-6834. TFile.append compares initial key against null lastKey (hong tang via mahadev)


> TFile.append compares initial key against null lastKey  
> --------------------------------------------------------
>
>                 Key: HADOOP-6834
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6834
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Ahad Rana
>            Assignee: Hong Tang
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6834-20100715.patch
>
>
> The following code in TFile.KeyReigster.close: 
>             byte[] lastKey = lastKeyBufferOS.getBuffer();
>             int lastLen = lastKeyBufferOS.size();
>             if (tfileMeta.getComparator().compare(key, 0, len, lastKey, 0,
>                 lastLen) < 0) {
>               throw new IOException("Keys are not added in sorted order");
>             }
> compares the initial  key (passed in via  TFile.Writer.append) against a technically NULL lastKey. lastKey is not initialized until after the first call to TFile.Writer.append. The underlying RawComparator interface used for comparisons does not stipulate the proper behavior when either length 1  or length 2 is zero. In the case of LongWritable, its WritableComparator implementation does an unsafe read on the passed in byte arrays b1 and b2. Since TFile pre-allocates the buffer used for storing lastKey, this passes a valid buffer with zero count to LongWritable's comparator, which ignores length and thus produces incorrect results. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9597-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 17 11:21:48 2010
Return-Path: <common-issues-return-9597-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48103 invoked from network); 17 Jul 2010 11:21:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jul 2010 11:21:48 -0000
Received: (qmail 68568 invoked by uid 500); 17 Jul 2010 11:21:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68440 invoked by uid 500); 17 Jul 2010 11:21:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68432 invoked by uid 99); 17 Jul 2010 11:21:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Jul 2010 11:21:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Jul 2010 11:21:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6HBDq2t016353
	for <common-issues@hadoop.apache.org>; Sat, 17 Jul 2010 11:13:52 GMT
Message-ID: <22914781.435551279365232031.JavaMail.jira@thor>
Date: Sat, 17 Jul 2010 07:13:52 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6860)  'compile-fault-inject' should
 never be called directly.
In-Reply-To: <25103470.380431279132431950.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889495#action_12889495 ] 

Hudson commented on HADOOP-6860:
--------------------------------

Integrated in Hadoop-Common-trunk #395 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/395/])
    HADOOP-6860. 'compile-fault-inject' should never be called directly. Contributed by Konstantin Boudnik.


>  'compile-fault-inject' should never be called directly.
> --------------------------------------------------------
>
>                 Key: HADOOP-6860
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6860
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6860.patch
>
>
> Similar to HDFS-1299 a  helper target  'compile-fault-inject' should never be called directly.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9598-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 17 13:40:49 2010
Return-Path: <common-issues-return-9598-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77119 invoked from network); 17 Jul 2010 13:40:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jul 2010 13:40:49 -0000
Received: (qmail 23691 invoked by uid 500); 17 Jul 2010 13:40:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23560 invoked by uid 500); 17 Jul 2010 13:40:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23539 invoked by uid 99); 17 Jul 2010 13:40:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Jul 2010 13:40:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Jul 2010 13:40:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6HDWpNo016838
	for <common-issues@hadoop.apache.org>; Sat, 17 Jul 2010 13:32:51 GMT
Message-ID: <9668853.435891279373571387.JavaMail.jira@thor>
Date: Sat, 17 Jul 2010 09:32:51 -0400 (EDT)
From: "Evan Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6865) there will be ant error if ant ran
 without network connected
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

there will be ant error if ant ran without network connected
------------------------------------------------------------

                 Key: HADOOP-6865
                 URL: https://issues.apache.org/jira/browse/HADOOP-6865
             Project: Hadoop Common
          Issue Type: Bug
          Components: build
    Affects Versions: 0.20.2
         Environment: centos 5.4
            Reporter: Evan Wang


If you run `ant` without network connected, there will be an error below. And even if you connect your network, the error will exist.

ivy-init-antlib:
  [typedef] java.util.zip.ZipException: error in opening zip file
  [typedef]     at java.util.zip.ZipFile.open(Native Method)
  [typedef]     at java.util.zip.ZipFile.<init>(ZipFile.java:114)
  [typedef]     at java.util.zip.ZipFile.<init>(ZipFile.java:131)
  [typedef]     at org.apache.tools.ant.AntClassLoader.getResourceURL(AntClassLoader.java:1028)
  [typedef]     at org.apache.tools.ant.AntClassLoader$ResourceEnumeration.findNextResource(AntClassLoader.java:147)
  [typedef]     at org.apache.tools.ant.AntClassLoader$ResourceEnumeration.<init>(AntClassLoader.java:109)
  [typedef]     at org.apache.tools.ant.AntClassLoader.findResources(AntClassLoader.java:975)
  [typedef]     at java.lang.ClassLoader.getResources(ClassLoader.java:1016)
  [typedef]     at org.apache.tools.ant.taskdefs.Definer.resourceToURLs(Definer.java:364)
  [typedef]     at org.apache.tools.ant.taskdefs.Definer.execute(Definer.java:256)
  [typedef]     at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288)
  [typedef]     at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
  [typedef]     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  [typedef]     at java.lang.reflect.Method.invoke(Method.java:597)
  [typedef]     at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
  [typedef]     at org.apache.tools.ant.Task.perform(Task.java:348)
  [typedef]     at org.apache.tools.ant.Target.execute(Target.java:357)
  [typedef]     at org.apache.tools.ant.Target.performTasks(Target.java:385)
  [typedef]     at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1337)
  [typedef]     at org.apache.tools.ant.Project.executeTarget(Project.java:1306)
  [typedef]     at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
  [typedef]     at org.apache.tools.ant.Project.executeTargets(Project.java:1189)
  [typedef]     at org.apache.tools.ant.Main.runBuild(Main.java:758)
  [typedef]     at org.apache.tools.ant.Main.startAnt(Main.java:217)
  [typedef]     at org.apache.tools.ant.launch.Launcher.run(Launcher.java:257)
  [typedef]     at org.apache.tools.ant.launch.Launcher.main(Launcher.java:104)
  [typedef] Could not load definitions from resource org/apache/ivy/ant/antlib.xml. It could not be found.

BUILD FAILED
/opt/hadoop-0.20.2/build.xml:1644: You need Apache Ivy 2.0 or later from http://ant.apache.org/
      It could not be loaded from http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.0.0-rc2/ivy-2.0.0-rc2.jar


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9599-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 17 22:01:16 2010
Return-Path: <common-issues-return-9599-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74393 invoked from network); 17 Jul 2010 22:01:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jul 2010 22:01:16 -0000
Received: (qmail 34135 invoked by uid 500); 17 Jul 2010 22:01:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34106 invoked by uid 500); 17 Jul 2010 22:01:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34098 invoked by uid 99); 17 Jul 2010 22:01:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Jul 2010 22:01:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 17 Jul 2010 22:01:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6HM0pNN018803
	for <common-issues@hadoop.apache.org>; Sat, 17 Jul 2010 22:00:51 GMT
Message-ID: <25968078.437261279404051537.JavaMail.jira@thor>
Date: Sat, 17 Jul 2010 18:00:51 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6863) Compile-native still fails on Mac
 OS X.
In-Reply-To: <18053660.404611279230589983.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889540#action_12889540 ] 

Allen Wittenauer commented on HADOOP-6863:
------------------------------------------

I haven't had a chance to try the patch yet, but it should be noted that it looks it was built with newer automake and I'm guessing newer libtool than the base version we have documented...

> Compile-native still fails on Mac OS X.
> ---------------------------------------
>
>                 Key: HADOOP-6863
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6863
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>         Attachments: hadoop-6863-20100715.patch
>
>
> I am still getting failures when I try to compile native library on Mac OS X (10.5.8 Leopard). The problems are two fold:
> - Although "aclocal.m4" is changed after HADOOP-3659 is committed, the corresponding "configure" script is not re-generated.
> - It cannot find libjvm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9600-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 19 01:29:19 2010
Return-Path: <common-issues-return-9600-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67244 invoked from network); 19 Jul 2010 01:29:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Jul 2010 01:29:19 -0000
Received: (qmail 31028 invoked by uid 500); 19 Jul 2010 01:29:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30895 invoked by uid 500); 19 Jul 2010 01:29:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30887 invoked by uid 99); 19 Jul 2010 01:29:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 01:29:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 01:29:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6J1Sr8R001875
	for <common-issues@hadoop.apache.org>; Mon, 19 Jul 2010 01:28:54 GMT
Message-ID: <29817395.446181279502933688.JavaMail.jira@thor>
Date: Sun, 18 Jul 2010 21:28:53 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6863) Compile-native still fails on Mac
 OS X.
In-Reply-To: <18053660.404611279230589983.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889727#action_12889727 ] 

Hong Tang commented on HADOOP-6863:
-----------------------------------

I used the version installed on my Mac.  What are the base versions we have documented?

Sent from my iPhone





> Compile-native still fails on Mac OS X.
> ---------------------------------------
>
>                 Key: HADOOP-6863
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6863
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>         Attachments: hadoop-6863-20100715.patch
>
>
> I am still getting failures when I try to compile native library on Mac OS X (10.5.8 Leopard). The problems are two fold:
> - Although "aclocal.m4" is changed after HADOOP-3659 is committed, the corresponding "configure" script is not re-generated.
> - It cannot find libjvm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9601-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 19 06:22:50 2010
Return-Path: <common-issues-return-9601-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41227 invoked from network); 19 Jul 2010 06:22:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Jul 2010 06:22:50 -0000
Received: (qmail 24762 invoked by uid 500); 19 Jul 2010 06:22:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24621 invoked by uid 500); 19 Jul 2010 06:22:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24613 invoked by uid 99); 19 Jul 2010 06:22:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 06:22:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 06:22:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6J6EpNl004103
	for <common-issues@hadoop.apache.org>; Mon, 19 Jul 2010 06:14:51 GMT
Message-ID: <22334492.448971279520091173.JavaMail.jira@thor>
Date: Mon, 19 Jul 2010 02:14:51 -0400 (EDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6536) FileUtil.fullyDelete(dir) behavior is
 not defined when we pass a symlink as the argument
In-Reply-To: <508958946.48471265176519280.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amareshwari Sriramadasu updated HADOOP-6536:
--------------------------------------------

    Status: Open  (was: Patch Available)

Changes look good.

bq. I would also add that fullyDelete should delete dangling links (it currently does but we should add a test). 
Can you add a test for deleting dangling links also?

> FileUtil.fullyDelete(dir) behavior is not defined when we pass a symlink as the argument
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6536
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6536
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Ravi Gummadi
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6536.patch, HADOOP-6536.v1.patch
>
>
> FileUtil.fullyDelete(dir) deletes contents of sym-linked directory when we pass a symlink. If this is the behavior, it should be documented as so. 
> Or it should be changed not to delete the contents of the sym-linked directory.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9602-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 19 08:09:17 2010
Return-Path: <common-issues-return-9602-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68652 invoked from network); 19 Jul 2010 08:09:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Jul 2010 08:09:17 -0000
Received: (qmail 96716 invoked by uid 500); 19 Jul 2010 08:09:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96589 invoked by uid 500); 19 Jul 2010 08:09:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96570 invoked by uid 99); 19 Jul 2010 08:09:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 08:09:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 08:09:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6J88nY9004911
	for <common-issues@hadoop.apache.org>; Mon, 19 Jul 2010 08:08:50 GMT
Message-ID: <20694622.449841279526929381.JavaMail.jira@thor>
Date: Mon, 19 Jul 2010 04:08:49 -0400 (EDT)
From: "Amar Kamat (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6866) Tool interface should also support
 getUsage()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Tool interface should also support getUsage()
---------------------------------------------

                 Key: HADOOP-6866
                 URL: https://issues.apache.org/jira/browse/HADOOP-6866
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Amar Kamat


Currently each and every _tool_ implementing the {{Tool}} interface is forced to manage their usage string. Since its a common piece of code, its better we factor it out. This can be useful in the following ways
# A proper lib like support for usage strings
# Forcing _tools_ (implementers of {{Tool}}) to expose their usage string
# Test cases can now use these well defined and exposed usage strings to test

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9603-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 19 09:05:50 2010
Return-Path: <common-issues-return-9603-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81629 invoked from network); 19 Jul 2010 09:05:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Jul 2010 09:05:50 -0000
Received: (qmail 36774 invoked by uid 500); 19 Jul 2010 09:05:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36642 invoked by uid 500); 19 Jul 2010 09:05:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36634 invoked by uid 99); 19 Jul 2010 09:05:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 09:05:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 09:05:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6J8vqfw005243
	for <common-issues@hadoop.apache.org>; Mon, 19 Jul 2010 08:57:52 GMT
Message-ID: <1572829.450331279529872035.JavaMail.jira@thor>
Date: Mon, 19 Jul 2010 04:57:52 -0400 (EDT)
From: "Evan Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6865) there will be ant error if ant ran
 without network connected
In-Reply-To: <9668853.435891279373571387.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889790#action_12889790 ] 

Evan Wang commented on HADOOP-6865:
-----------------------------------

There is a method to slove this problem. When ant built the project, ivy-2.0.0-rc2.jar would be rewritten. If network was not connected, this jar would have some error that cannot be repaired automatically. But you can use a original ivy-2.0.0-rc2.jar to replace it, then ant building will be ok.

> there will be ant error if ant ran without network connected
> ------------------------------------------------------------
>
>                 Key: HADOOP-6865
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6865
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.20.2
>         Environment: centos 5.4
>            Reporter: Evan Wang
>
> If you run `ant` without network connected, there will be an error below. And even if you connect your network, the error will exist.
> ivy-init-antlib:
>   [typedef] java.util.zip.ZipException: error in opening zip file
>   [typedef]     at java.util.zip.ZipFile.open(Native Method)
>   [typedef]     at java.util.zip.ZipFile.<init>(ZipFile.java:114)
>   [typedef]     at java.util.zip.ZipFile.<init>(ZipFile.java:131)
>   [typedef]     at org.apache.tools.ant.AntClassLoader.getResourceURL(AntClassLoader.java:1028)
>   [typedef]     at org.apache.tools.ant.AntClassLoader$ResourceEnumeration.findNextResource(AntClassLoader.java:147)
>   [typedef]     at org.apache.tools.ant.AntClassLoader$ResourceEnumeration.<init>(AntClassLoader.java:109)
>   [typedef]     at org.apache.tools.ant.AntClassLoader.findResources(AntClassLoader.java:975)
>   [typedef]     at java.lang.ClassLoader.getResources(ClassLoader.java:1016)
>   [typedef]     at org.apache.tools.ant.taskdefs.Definer.resourceToURLs(Definer.java:364)
>   [typedef]     at org.apache.tools.ant.taskdefs.Definer.execute(Definer.java:256)
>   [typedef]     at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288)
>   [typedef]     at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
>   [typedef]     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   [typedef]     at java.lang.reflect.Method.invoke(Method.java:597)
>   [typedef]     at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
>   [typedef]     at org.apache.tools.ant.Task.perform(Task.java:348)
>   [typedef]     at org.apache.tools.ant.Target.execute(Target.java:357)
>   [typedef]     at org.apache.tools.ant.Target.performTasks(Target.java:385)
>   [typedef]     at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1337)
>   [typedef]     at org.apache.tools.ant.Project.executeTarget(Project.java:1306)
>   [typedef]     at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
>   [typedef]     at org.apache.tools.ant.Project.executeTargets(Project.java:1189)
>   [typedef]     at org.apache.tools.ant.Main.runBuild(Main.java:758)
>   [typedef]     at org.apache.tools.ant.Main.startAnt(Main.java:217)
>   [typedef]     at org.apache.tools.ant.launch.Launcher.run(Launcher.java:257)
>   [typedef]     at org.apache.tools.ant.launch.Launcher.main(Launcher.java:104)
>   [typedef] Could not load definitions from resource org/apache/ivy/ant/antlib.xml. It could not be found.
> BUILD FAILED
> /opt/hadoop-0.20.2/build.xml:1644: You need Apache Ivy 2.0 or later from http://ant.apache.org/
>       It could not be loaded from http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.0.0-rc2/ivy-2.0.0-rc2.jar

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9604-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 19 09:27:49 2010
Return-Path: <common-issues-return-9604-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84639 invoked from network); 19 Jul 2010 09:27:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Jul 2010 09:27:48 -0000
Received: (qmail 48809 invoked by uid 500); 19 Jul 2010 09:27:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48668 invoked by uid 500); 19 Jul 2010 09:27:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48653 invoked by uid 99); 19 Jul 2010 09:27:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 09:27:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 09:27:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6J9JoY2005404
	for <common-issues@hadoop.apache.org>; Mon, 19 Jul 2010 09:19:50 GMT
Message-ID: <25280595.450471279531190434.JavaMail.jira@thor>
Date: Mon, 19 Jul 2010 05:19:50 -0400 (EDT)
From: "Jeff Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6866) Tool interface should also support
 getUsage()
In-Reply-To: <20694622.449841279526929381.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889792#action_12889792 ] 

Jeff Zhang commented on HADOOP-6866:
------------------------------------

This is a good point, but I concern that it will bring in incompatibility problem.  It will force the users who use the old Tool interface to add the getUsage() method.




> Tool interface should also support getUsage()
> ---------------------------------------------
>
>                 Key: HADOOP-6866
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6866
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Amar Kamat
>
> Currently each and every _tool_ implementing the {{Tool}} interface is forced to manage their usage string. Since its a common piece of code, its better we factor it out. This can be useful in the following ways
> # A proper lib like support for usage strings
> # Forcing _tools_ (implementers of {{Tool}}) to expose their usage string
> # Test cases can now use these well defined and exposed usage strings to test

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9605-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 19 10:04:19 2010
Return-Path: <common-issues-return-9605-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91011 invoked from network); 19 Jul 2010 10:04:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Jul 2010 10:04:19 -0000
Received: (qmail 76303 invoked by uid 500); 19 Jul 2010 10:04:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76176 invoked by uid 500); 19 Jul 2010 10:04:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76168 invoked by uid 99); 19 Jul 2010 10:04:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 10:04:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 10:04:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6JA3pGT005751
	for <common-issues@hadoop.apache.org>; Mon, 19 Jul 2010 10:03:51 GMT
Message-ID: <11121968.450951279533831682.JavaMail.jira@thor>
Date: Mon, 19 Jul 2010 06:03:51 -0400 (EDT)
From: "Ravi Gummadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6536) FileUtil.fullyDelete(dir) behavior is
 not defined when we pass a symlink as the argument
In-Reply-To: <508958946.48471265176519280.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated HADOOP-6536:
---------------------------------

    Attachment: HADOOP-6536.v1.1.patch

Attaching patch adding more testcases.

> FileUtil.fullyDelete(dir) behavior is not defined when we pass a symlink as the argument
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6536
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6536
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Ravi Gummadi
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6536.patch, HADOOP-6536.v1.1.patch, HADOOP-6536.v1.patch
>
>
> FileUtil.fullyDelete(dir) deletes contents of sym-linked directory when we pass a symlink. If this is the behavior, it should be documented as so. 
> Or it should be changed not to delete the contents of the sym-linked directory.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9606-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 19 11:04:49 2010
Return-Path: <common-issues-return-9606-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3058 invoked from network); 19 Jul 2010 11:04:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Jul 2010 11:04:49 -0000
Received: (qmail 10808 invoked by uid 500); 19 Jul 2010 11:04:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10677 invoked by uid 500); 19 Jul 2010 11:04:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10660 invoked by uid 99); 19 Jul 2010 11:04:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 11:04:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 11:04:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6JAuoEc006115
	for <common-issues@hadoop.apache.org>; Mon, 19 Jul 2010 10:56:51 GMT
Message-ID: <30814017.451461279537010789.JavaMail.jira@thor>
Date: Mon, 19 Jul 2010 06:56:50 -0400 (EDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6536) FileUtil.fullyDelete(dir) behavior is
 not defined when we pass a symlink as the argument
In-Reply-To: <508958946.48471265176519280.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amareshwari Sriramadasu updated HADOOP-6536:
--------------------------------------------

    Status: Patch Available  (was: Open)

Patch looks good.

> FileUtil.fullyDelete(dir) behavior is not defined when we pass a symlink as the argument
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6536
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6536
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Ravi Gummadi
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6536.patch, HADOOP-6536.v1.1.patch, HADOOP-6536.v1.patch
>
>
> FileUtil.fullyDelete(dir) deletes contents of sym-linked directory when we pass a symlink. If this is the behavior, it should be documented as so. 
> Or it should be changed not to delete the contents of the sym-linked directory.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9607-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 19 11:31:19 2010
Return-Path: <common-issues-return-9607-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10801 invoked from network); 19 Jul 2010 11:31:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Jul 2010 11:31:19 -0000
Received: (qmail 33646 invoked by uid 500); 19 Jul 2010 11:31:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33528 invoked by uid 500); 19 Jul 2010 11:31:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33510 invoked by uid 99); 19 Jul 2010 11:31:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 11:31:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 11:31:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6JBUogn006342
	for <common-issues@hadoop.apache.org>; Mon, 19 Jul 2010 11:30:51 GMT
Message-ID: <27114782.451831279539050844.JavaMail.jira@thor>
Date: Mon, 19 Jul 2010 07:30:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6536) FileUtil.fullyDelete(dir) behavior
 is not defined when we pass a symlink as the argument
In-Reply-To: <508958946.48471265176519280.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889822#action_12889822 ] 

Hadoop QA commented on HADOOP-6536:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12449821/HADOOP-6536.v1.1.patch
  against trunk revision 964993.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/623/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/623/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/623/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/623/console

This message is automatically generated.

> FileUtil.fullyDelete(dir) behavior is not defined when we pass a symlink as the argument
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6536
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6536
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Ravi Gummadi
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6536.patch, HADOOP-6536.v1.1.patch, HADOOP-6536.v1.patch
>
>
> FileUtil.fullyDelete(dir) deletes contents of sym-linked directory when we pass a symlink. If this is the behavior, it should be documented as so. 
> Or it should be changed not to delete the contents of the sym-linked directory.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9608-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 19 15:55:53 2010
Return-Path: <common-issues-return-9608-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3780 invoked from network); 19 Jul 2010 15:55:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Jul 2010 15:55:53 -0000
Received: (qmail 15197 invoked by uid 500); 19 Jul 2010 15:55:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15071 invoked by uid 500); 19 Jul 2010 15:55:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15063 invoked by uid 99); 19 Jul 2010 15:55:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 15:55:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 15:55:49 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6JFlwFa008599
	for <common-issues@hadoop.apache.org>; Mon, 19 Jul 2010 15:47:59 GMT
Message-ID: <9147831.455691279554478586.JavaMail.jira@thor>
Date: Mon, 19 Jul 2010 11:47:58 -0400 (EDT)
From: "Nicholas Carlini (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6349) Implement FastLZCodec for
 fastlz/lzo algorithm
In-Reply-To: <289817036.1256941439448.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889893#action_12889893 ] 

Nicholas Carlini commented on HADOOP-6349:
------------------------------------------

Eli - did you make an updated patch? If you haven't that's okay -- I can rebase it on trunk if you haven't yet.

> Implement FastLZCodec for fastlz/lzo algorithm
> ----------------------------------------------
>
>                 Key: HADOOP-6349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6349
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: William Kinney
>         Attachments: HADOOP-6349-TestFastLZCodec.patch, HADOOP-6349.patch, TestCodecPerformance.java, TestCodecPerformance.java, testCodecPerfResults.tsv
>
>
> Per  [HADOOP-4874|http://issues.apache.org/jira/browse/HADOOP-4874], FastLZ is a good (speed, license) alternative to LZO. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9609-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 19 16:50:17 2010
Return-Path: <common-issues-return-9609-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8481 invoked from network); 19 Jul 2010 16:50:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Jul 2010 16:50:16 -0000
Received: (qmail 80566 invoked by uid 500); 19 Jul 2010 16:50:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80406 invoked by uid 500); 19 Jul 2010 16:50:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80394 invoked by uid 99); 19 Jul 2010 16:50:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 16:50:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 16:50:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6JGnppr009286
	for <common-issues@hadoop.apache.org>; Mon, 19 Jul 2010 16:49:51 GMT
Message-ID: <21773772.457021279558191424.JavaMail.jira@thor>
Date: Mon, 19 Jul 2010 12:49:51 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6805) add buildDTServiceName method to
 SecurityUtil (as part of MAPREDUCE-1718)
In-Reply-To: <5393650.148181275524636022.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889914#action_12889914 ] 

Boris Shkolnik commented on HADOOP-6805:
----------------------------------------

javadoc - are the old javadoc warnings (about sun private pacakges) introduced elswhere..
no test - this is not a new code, this patch moves a method from one class to another.

> add buildDTServiceName method to SecurityUtil (as part of MAPREDUCE-1718)
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6805
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6805
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6805.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9610-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 19 16:54:15 2010
Return-Path: <common-issues-return-9610-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9081 invoked from network); 19 Jul 2010 16:54:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Jul 2010 16:54:15 -0000
Received: (qmail 84413 invoked by uid 500); 19 Jul 2010 16:54:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84348 invoked by uid 500); 19 Jul 2010 16:54:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84340 invoked by uid 99); 19 Jul 2010 16:54:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 16:54:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 16:54:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6JGrohW009331
	for <common-issues@hadoop.apache.org>; Mon, 19 Jul 2010 16:53:50 GMT
Message-ID: <3852287.457161279558430207.JavaMail.jira@thor>
Date: Mon, 19 Jul 2010 12:53:50 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6805) add buildDTServiceName method to
 SecurityUtil (as part of MAPREDUCE-1718)
In-Reply-To: <5393650.148181275524636022.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6805:
-----------------------------------


committed to trunk

> add buildDTServiceName method to SecurityUtil (as part of MAPREDUCE-1718)
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6805
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6805
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6805.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9611-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 19 17:48:45 2010
Return-Path: <common-issues-return-9611-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26705 invoked from network); 19 Jul 2010 17:48:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Jul 2010 17:48:45 -0000
Received: (qmail 83556 invoked by uid 500); 19 Jul 2010 17:48:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83526 invoked by uid 500); 19 Jul 2010 17:48:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83518 invoked by uid 99); 19 Jul 2010 17:48:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 17:48:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 17:48:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6JHepsR009744
	for <common-issues@hadoop.apache.org>; Mon, 19 Jul 2010 17:40:51 GMT
Message-ID: <16405142.457761279561251019.JavaMail.jira@thor>
Date: Mon, 19 Jul 2010 13:40:51 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6805) add buildDTServiceName method to
 SecurityUtil (as part of MAPREDUCE-1718)
In-Reply-To: <5393650.148181275524636022.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889936#action_12889936 ] 

Hudson commented on HADOOP-6805:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #330 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/330/])
    HADOOP-6805. add buildDTServiceName method to SecurityUtil (as part of MAPREDUCE-1718)


> add buildDTServiceName method to SecurityUtil (as part of MAPREDUCE-1718)
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6805
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6805
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6805.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9612-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 19 18:35:47 2010
Return-Path: <common-issues-return-9612-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48278 invoked from network); 19 Jul 2010 18:35:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Jul 2010 18:35:47 -0000
Received: (qmail 53684 invoked by uid 500); 19 Jul 2010 18:35:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53259 invoked by uid 500); 19 Jul 2010 18:35:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53244 invoked by uid 99); 19 Jul 2010 18:35:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 18:35:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 18:35:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6JIRph6010262
	for <common-issues@hadoop.apache.org>; Mon, 19 Jul 2010 18:27:51 GMT
Message-ID: <23399636.458721279564071100.JavaMail.jira@thor>
Date: Mon, 19 Jul 2010 14:27:51 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6865) there will be ant error if ant ran
 without network connected
In-Reply-To: <9668853.435891279373571387.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889956#action_12889956 ] 

Konstantin Boudnik commented on HADOOP-6865:
--------------------------------------------

Looks like you have a broken Icy zip file. Try to remove it.

> there will be ant error if ant ran without network connected
> ------------------------------------------------------------
>
>                 Key: HADOOP-6865
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6865
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.20.2
>         Environment: centos 5.4
>            Reporter: Evan Wang
>
> If you run `ant` without network connected, there will be an error below. And even if you connect your network, the error will exist.
> ivy-init-antlib:
>   [typedef] java.util.zip.ZipException: error in opening zip file
>   [typedef]     at java.util.zip.ZipFile.open(Native Method)
>   [typedef]     at java.util.zip.ZipFile.<init>(ZipFile.java:114)
>   [typedef]     at java.util.zip.ZipFile.<init>(ZipFile.java:131)
>   [typedef]     at org.apache.tools.ant.AntClassLoader.getResourceURL(AntClassLoader.java:1028)
>   [typedef]     at org.apache.tools.ant.AntClassLoader$ResourceEnumeration.findNextResource(AntClassLoader.java:147)
>   [typedef]     at org.apache.tools.ant.AntClassLoader$ResourceEnumeration.<init>(AntClassLoader.java:109)
>   [typedef]     at org.apache.tools.ant.AntClassLoader.findResources(AntClassLoader.java:975)
>   [typedef]     at java.lang.ClassLoader.getResources(ClassLoader.java:1016)
>   [typedef]     at org.apache.tools.ant.taskdefs.Definer.resourceToURLs(Definer.java:364)
>   [typedef]     at org.apache.tools.ant.taskdefs.Definer.execute(Definer.java:256)
>   [typedef]     at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288)
>   [typedef]     at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
>   [typedef]     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   [typedef]     at java.lang.reflect.Method.invoke(Method.java:597)
>   [typedef]     at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
>   [typedef]     at org.apache.tools.ant.Task.perform(Task.java:348)
>   [typedef]     at org.apache.tools.ant.Target.execute(Target.java:357)
>   [typedef]     at org.apache.tools.ant.Target.performTasks(Target.java:385)
>   [typedef]     at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1337)
>   [typedef]     at org.apache.tools.ant.Project.executeTarget(Project.java:1306)
>   [typedef]     at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
>   [typedef]     at org.apache.tools.ant.Project.executeTargets(Project.java:1189)
>   [typedef]     at org.apache.tools.ant.Main.runBuild(Main.java:758)
>   [typedef]     at org.apache.tools.ant.Main.startAnt(Main.java:217)
>   [typedef]     at org.apache.tools.ant.launch.Launcher.run(Launcher.java:257)
>   [typedef]     at org.apache.tools.ant.launch.Launcher.main(Launcher.java:104)
>   [typedef] Could not load definitions from resource org/apache/ivy/ant/antlib.xml. It could not be found.
> BUILD FAILED
> /opt/hadoop-0.20.2/build.xml:1644: You need Apache Ivy 2.0 or later from http://ant.apache.org/
>       It could not be loaded from http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.0.0-rc2/ivy-2.0.0-rc2.jar

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9613-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 19 21:10:20 2010
Return-Path: <common-issues-return-9613-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8592 invoked from network); 19 Jul 2010 21:10:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Jul 2010 21:10:19 -0000
Received: (qmail 88347 invoked by uid 500); 19 Jul 2010 21:10:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88289 invoked by uid 500); 19 Jul 2010 21:10:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88281 invoked by uid 99); 19 Jul 2010 21:10:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 21:10:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 21:10:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6JL9thJ012100
	for <common-issues@hadoop.apache.org>; Mon, 19 Jul 2010 21:09:55 GMT
Message-ID: <20410586.462451279573794996.JavaMail.jira@thor>
Date: Mon, 19 Jul 2010 17:09:54 -0400 (EDT)
From: "Nicholas Carlini (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicholas Carlini updated HADOOP-6837:
-------------------------------------

    Attachment: HADOOP-6837-lzma-c-20100719.patch

Uploaded C code with LzmaNativeInputStream and LzmaNativeOutputStream. Testing is the same as that for the Java code. The documentation is limited on the C side, and there are still (commented out) debug statements scattered all over.

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-c-20100719.patch, HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9614-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 19 21:12:50 2010
Return-Path: <common-issues-return-9614-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9586 invoked from network); 19 Jul 2010 21:12:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Jul 2010 21:12:50 -0000
Received: (qmail 94206 invoked by uid 500); 19 Jul 2010 21:12:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94161 invoked by uid 500); 19 Jul 2010 21:12:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94153 invoked by uid 99); 19 Jul 2010 21:12:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 21:12:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 21:12:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6JLCPkB012153
	for <common-issues@hadoop.apache.org>; Mon, 19 Jul 2010 21:12:25 GMT
Message-ID: <26284056.462611279573945618.JavaMail.jira@thor>
Date: Mon, 19 Jul 2010 17:12:25 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6855) Add ability to get groups for ACLs
 from 'getent netgroup'
In-Reply-To: <23521712.275021278635570824.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890041#action_12890041 ] 

Erik Steffl commented on HADOOP-6855:
-------------------------------------

This patch is for people who want to use 'getent netgroup $group' command to provide groups to user mapping. It does not affect people who do not explicitly configure usage of the group mapping service added in this patch (ShellBasedUnixGroupsNetgroupMapping).

There is another patch coming that will provide JNI (or JNA) based implementation see https://issues.apache.org/jira/browse/HADOOP-6864

> Add ability to get groups for ACLs from 'getent netgroup'
> ---------------------------------------------------------
>
>                 Key: HADOOP-6855
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6855
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6855-0.20-1.patch, HADOOP-6855-0.20-2.patch, HADOOP-6855-0.20.patch
>
>
> Add ability to specify netgroups in ACLs (see class AccessControlList.java). Membership of users in netgroups will be determined by running 'getent negroups $groupName'. Netgroups will be differentiated from regular unix groups by having '@' as a first character.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9615-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 19 21:29:49 2010
Return-Path: <common-issues-return-9615-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19470 invoked from network); 19 Jul 2010 21:29:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Jul 2010 21:29:48 -0000
Received: (qmail 27746 invoked by uid 500); 19 Jul 2010 21:29:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27683 invoked by uid 500); 19 Jul 2010 21:29:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27675 invoked by uid 99); 19 Jul 2010 21:29:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 21:29:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 21:29:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6JLLs7A012241
	for <common-issues@hadoop.apache.org>; Mon, 19 Jul 2010 21:21:54 GMT
Message-ID: <4067517.462751279574514036.JavaMail.jira@thor>
Date: Mon, 19 Jul 2010 17:21:54 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890043#action_12890043 ] 

Hong Tang commented on HADOOP-6837:
-----------------------------------

@nicolas, per our offline conversation last week, have you looked into whether the licensing of liblzma is suitable for inclusion in Hadoop? Liblzma seems better in the sense that its API resembles closely the APIs of other compression libraries like bzip or zlib and should shrink the amount of coding work needed to support C (and Java over JNI).

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-c-20100719.patch, HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9616-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 19 21:47:50 2010
Return-Path: <common-issues-return-9616-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28511 invoked from network); 19 Jul 2010 21:47:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Jul 2010 21:47:50 -0000
Received: (qmail 52041 invoked by uid 500); 19 Jul 2010 21:47:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51979 invoked by uid 500); 19 Jul 2010 21:47:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51971 invoked by uid 99); 19 Jul 2010 21:47:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 21:47:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 21:47:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6JLduQl012434
	for <common-issues@hadoop.apache.org>; Mon, 19 Jul 2010 21:39:56 GMT
Message-ID: <1938524.463161279575596381.JavaMail.jira@thor>
Date: Mon, 19 Jul 2010 17:39:56 -0400 (EDT)
From: "Nicholas Carlini (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890051#action_12890051 ] 

Nicholas Carlini commented on HADOOP-6837:
------------------------------------------

I spoke with Greg about it just now and he said it would probably be better for me to work on FastLZ first, and come back to doing that latter.

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-c-20100719.patch, HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9617-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 19 22:05:48 2010
Return-Path: <common-issues-return-9617-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37944 invoked from network); 19 Jul 2010 22:05:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Jul 2010 22:05:47 -0000
Received: (qmail 83009 invoked by uid 500); 19 Jul 2010 22:05:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82809 invoked by uid 500); 19 Jul 2010 22:05:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82801 invoked by uid 99); 19 Jul 2010 22:05:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 22:05:46 +0000
X-ASF-Spam-Status: No, hits=-1996.4 required=10.0
	tests=ALL_TRUSTED,FS_REPLICA
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 22:05:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6JLvrle012567
	for <common-issues@hadoop.apache.org>; Mon, 19 Jul 2010 21:57:53 GMT
Message-ID: <18620743.463391279576673532.JavaMail.jira@thor>
Date: Mon, 19 Jul 2010 17:57:53 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6857) FsShell should report raw disk usage
 including replication factor
In-Reply-To: <1268643.319901278914569769.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6857:
--------------------------------

         Hadoop Flags: [Incompatible change]
        Fix Version/s: 0.22.0
    Affects Version/s:     (was: 0.20.2)

Hey Aaron,

Patch looks good.  Mind creating o.a.h.fs.TestFsShell.java and adding a test that shows files with two different replication levels works? (might need to mock up the replication level).  Also, please test that TestShell and TestHDFSCLI in HDFS still pass for sanity.

Wrt to rationale I think this change is kosher since FileStatus#getReplication is not hdfs-specific. 

Marking the jira as an incompatible change since IIRC the FsShell is considered a public API.

Thanks,
Eli

> FsShell should report raw disk usage including replication factor
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6857
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6857
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Alex Kozlov
>             Fix For: 0.22.0
>
>         Attachments: show-space-consumed.txt
>
>
> Currently FsShell report HDFS usage with "hadoop fs -dus <path>" command.  Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe "hadoop fs -dus -raw <path>"?).  This will allow to assess resource usage more accurately.  -- Alex K

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9618-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 19 22:28:46 2010
Return-Path: <common-issues-return-9618-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44701 invoked from network); 19 Jul 2010 22:28:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Jul 2010 22:28:46 -0000
Received: (qmail 10429 invoked by uid 500); 19 Jul 2010 22:28:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10365 invoked by uid 500); 19 Jul 2010 22:28:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10356 invoked by uid 99); 19 Jul 2010 22:28:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 22:28:45 +0000
X-ASF-Spam-Status: No, hits=-1996.4 required=10.0
	tests=ALL_TRUSTED,FS_REPLICA
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 22:28:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6JMKplJ012982
	for <common-issues@hadoop.apache.org>; Mon, 19 Jul 2010 22:20:51 GMT
Message-ID: <11781049.464411279578051250.JavaMail.jira@thor>
Date: Mon, 19 Jul 2010 18:20:51 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6857) FsShell should report raw disk
 usage including replication factor
In-Reply-To: <1268643.319901278914569769.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890070#action_12890070 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6857:
------------------------------------------------

We already have "fs -count <path>" which counts bytes including replications. Is it good enough?

> FsShell should report raw disk usage including replication factor
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6857
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6857
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Alex Kozlov
>             Fix For: 0.22.0
>
>         Attachments: show-space-consumed.txt
>
>
> Currently FsShell report HDFS usage with "hadoop fs -dus <path>" command.  Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe "hadoop fs -dus -raw <path>"?).  This will allow to assess resource usage more accurately.  -- Alex K

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9619-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 19 23:23:46 2010
Return-Path: <common-issues-return-9619-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62252 invoked from network); 19 Jul 2010 23:23:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Jul 2010 23:23:46 -0000
Received: (qmail 47327 invoked by uid 500); 19 Jul 2010 23:23:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47250 invoked by uid 500); 19 Jul 2010 23:23:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47242 invoked by uid 99); 19 Jul 2010 23:23:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 23:23:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 19 Jul 2010 23:23:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6JNFpR9013438
	for <common-issues@hadoop.apache.org>; Mon, 19 Jul 2010 23:15:51 GMT
Message-ID: <7746361.465401279581351297.JavaMail.jira@thor>
Date: Mon, 19 Jul 2010 19:15:51 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6632) Support for using different Kerberos
 keys for different instances of Hadoop services
In-Reply-To: <1542465557.251491268540967230.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6632:
------------------------------

    Attachment: c6632-07.patch

Uploading a new patch that simply merges with latest trunk changes. No semantic change from previous patch.

> Support for using different Kerberos keys for different instances of Hadoop services
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6632
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6632
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: 6632.mr.patch, c6632-05.patch, c6632-07.patch, HADOOP-6632-Y20S-18.patch, HADOOP-6632-Y20S-22.patch
>
>
> We tested using the same Kerberos key for all datanodes in a HDFS cluster or the same Kerberos key for all TaskTarckers in a MapRed cluster. But it doesn't work. The reason is that when datanodes try to authenticate to the namenode all at once, the Kerberos authenticators they send to the namenode may have the same timestamp and will be rejected as replay requests. This JIRA makes it possible to use a unique key for each service instance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9620-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 00:07:45 2010
Return-Path: <common-issues-return-9620-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75886 invoked from network); 20 Jul 2010 00:07:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 00:07:45 -0000
Received: (qmail 77927 invoked by uid 500); 20 Jul 2010 00:07:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77849 invoked by uid 500); 20 Jul 2010 00:07:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77838 invoked by uid 99); 20 Jul 2010 00:07:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 00:07:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 00:07:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6JNxo23013634
	for <common-issues@hadoop.apache.org>; Mon, 19 Jul 2010 23:59:50 GMT
Message-ID: <12366916.465701279583990677.JavaMail.jira@thor>
Date: Mon, 19 Jul 2010 19:59:50 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6632) Support for using different Kerberos
 keys for different instances of Hadoop services
In-Reply-To: <1542465557.251491268540967230.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6632:
------------------------------

    Status: Open  (was: Patch Available)

> Support for using different Kerberos keys for different instances of Hadoop services
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6632
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6632
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: 6632.mr.patch, c6632-05.patch, c6632-07.patch, HADOOP-6632-Y20S-18.patch, HADOOP-6632-Y20S-22.patch
>
>
> We tested using the same Kerberos key for all datanodes in a HDFS cluster or the same Kerberos key for all TaskTarckers in a MapRed cluster. But it doesn't work. The reason is that when datanodes try to authenticate to the namenode all at once, the Kerberos authenticators they send to the namenode may have the same timestamp and will be rejected as replay requests. This JIRA makes it possible to use a unique key for each service instance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9621-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 00:07:46 2010
Return-Path: <common-issues-return-9621-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75937 invoked from network); 20 Jul 2010 00:07:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 00:07:46 -0000
Received: (qmail 78042 invoked by uid 500); 20 Jul 2010 00:07:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77864 invoked by uid 500); 20 Jul 2010 00:07:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77846 invoked by uid 99); 20 Jul 2010 00:07:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 00:07:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 00:07:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6JNxprL013646
	for <common-issues@hadoop.apache.org>; Mon, 19 Jul 2010 23:59:51 GMT
Message-ID: <8113664.465741279583991429.JavaMail.jira@thor>
Date: Mon, 19 Jul 2010 19:59:51 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6632) Support for using different Kerberos
 keys for different instances of Hadoop services
In-Reply-To: <1542465557.251491268540967230.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6632:
------------------------------

    Status: Patch Available  (was: Open)

> Support for using different Kerberos keys for different instances of Hadoop services
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6632
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6632
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: 6632.mr.patch, c6632-05.patch, c6632-07.patch, HADOOP-6632-Y20S-18.patch, HADOOP-6632-Y20S-22.patch
>
>
> We tested using the same Kerberos key for all datanodes in a HDFS cluster or the same Kerberos key for all TaskTarckers in a MapRed cluster. But it doesn't work. The reason is that when datanodes try to authenticate to the namenode all at once, the Kerberos authenticators they send to the namenode may have the same timestamp and will be rejected as replay requests. This JIRA makes it possible to use a unique key for each service instance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9622-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 00:34:16 2010
Return-Path: <common-issues-return-9622-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88317 invoked from network); 20 Jul 2010 00:34:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 00:34:16 -0000
Received: (qmail 96521 invoked by uid 500); 20 Jul 2010 00:34:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96466 invoked by uid 500); 20 Jul 2010 00:34:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96448 invoked by uid 99); 20 Jul 2010 00:34:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 00:34:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 00:34:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6K0XoD4013866
	for <common-issues@hadoop.apache.org>; Tue, 20 Jul 2010 00:33:50 GMT
Message-ID: <14347346.465951279586030428.JavaMail.jira@thor>
Date: Mon, 19 Jul 2010 20:33:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6632) Support for using different
 Kerberos keys for different instances of Hadoop services
In-Reply-To: <1542465557.251491268540967230.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890104#action_12890104 ] 

Hadoop QA commented on HADOOP-6632:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12449900/c6632-07.patch
  against trunk revision 965556.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/624/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/624/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/624/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/624/console

This message is automatically generated.

> Support for using different Kerberos keys for different instances of Hadoop services
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6632
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6632
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: 6632.mr.patch, c6632-05.patch, c6632-07.patch, HADOOP-6632-Y20S-18.patch, HADOOP-6632-Y20S-22.patch
>
>
> We tested using the same Kerberos key for all datanodes in a HDFS cluster or the same Kerberos key for all TaskTarckers in a MapRed cluster. But it doesn't work. The reason is that when datanodes try to authenticate to the namenode all at once, the Kerberos authenticators they send to the namenode may have the same timestamp and will be rejected as replay requests. This JIRA makes it possible to use a unique key for each service instance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9623-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 00:59:47 2010
Return-Path: <common-issues-return-9623-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2804 invoked from network); 20 Jul 2010 00:59:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 00:59:47 -0000
Received: (qmail 21934 invoked by uid 500); 20 Jul 2010 00:59:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21875 invoked by uid 500); 20 Jul 2010 00:59:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21867 invoked by uid 99); 20 Jul 2010 00:59:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 00:59:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 00:59:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6K0pora013960
	for <common-issues@hadoop.apache.org>; Tue, 20 Jul 2010 00:51:50 GMT
Message-ID: <2579802.466091279587110371.JavaMail.jira@thor>
Date: Mon, 19 Jul 2010 20:51:50 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6632) Support for using different Kerberos
 keys for different instances of Hadoop services
In-Reply-To: <1542465557.251491268540967230.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6632:
--------------------------------

           Status: Resolved  (was: Patch Available)
    Fix Version/s: 0.22.0
       Resolution: Fixed

I just committed this. Thanks, Kan & Jitendra!

> Support for using different Kerberos keys for different instances of Hadoop services
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6632
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6632
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>             Fix For: 0.22.0
>
>         Attachments: 6632.mr.patch, c6632-05.patch, c6632-07.patch, HADOOP-6632-Y20S-18.patch, HADOOP-6632-Y20S-22.patch
>
>
> We tested using the same Kerberos key for all datanodes in a HDFS cluster or the same Kerberos key for all TaskTarckers in a MapRed cluster. But it doesn't work. The reason is that when datanodes try to authenticate to the namenode all at once, the Kerberos authenticators they send to the namenode may have the same timestamp and will be rejected as replay requests. This JIRA makes it possible to use a unique key for each service instance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9624-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 01:04:23 2010
Return-Path: <common-issues-return-9624-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3934 invoked from network); 20 Jul 2010 01:04:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 01:04:23 -0000
Received: (qmail 26224 invoked by uid 500); 20 Jul 2010 01:04:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26112 invoked by uid 500); 20 Jul 2010 01:04:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26104 invoked by uid 99); 20 Jul 2010 01:04:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 01:04:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 01:04:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6K13oPr014113
	for <common-issues@hadoop.apache.org>; Tue, 20 Jul 2010 01:03:51 GMT
Message-ID: <5778765.466321279587830969.JavaMail.jira@thor>
Date: Mon, 19 Jul 2010 21:03:50 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6632) Support for using different
 Kerberos keys for different instances of Hadoop services
In-Reply-To: <1542465557.251491268540967230.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890112#action_12890112 ] 

Hudson commented on HADOOP-6632:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #331 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/331/])
    HADOOP-6632. Adds support for using different keytabs for different servers in a Hadoop cluster. In the earier implementation, all servers of a certain type \(like TaskTracker\), would have the same keytab and the same principal. Now the principal name is a pattern that has _HOST in it. Contributed by Kan Zhang & Jitendra Pandey.


> Support for using different Kerberos keys for different instances of Hadoop services
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6632
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6632
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>             Fix For: 0.22.0
>
>         Attachments: 6632.mr.patch, c6632-05.patch, c6632-07.patch, HADOOP-6632-Y20S-18.patch, HADOOP-6632-Y20S-22.patch
>
>
> We tested using the same Kerberos key for all datanodes in a HDFS cluster or the same Kerberos key for all TaskTarckers in a MapRed cluster. But it doesn't work. The reason is that when datanodes try to authenticate to the namenode all at once, the Kerberos authenticators they send to the namenode may have the same timestamp and will be rejected as replay requests. This JIRA makes it possible to use a unique key for each service instance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9625-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 01:21:25 2010
Return-Path: <common-issues-return-9625-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12795 invoked from network); 20 Jul 2010 01:21:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 01:21:24 -0000
Received: (qmail 37782 invoked by uid 500); 20 Jul 2010 01:21:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37722 invoked by uid 500); 20 Jul 2010 01:21:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37714 invoked by uid 99); 20 Jul 2010 01:21:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 01:21:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 01:21:21 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6K1KxjL014340
	for <common-issues@hadoop.apache.org>; Tue, 20 Jul 2010 01:20:59 GMT
Message-ID: <29867039.466831279588859875.JavaMail.jira@thor>
Date: Mon, 19 Jul 2010 21:20:59 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6349) Implement FastLZCodec for fastlz/lzo
 algorithm
In-Reply-To: <289817036.1256941439448.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6349:
--------------------------------

    Attachment: hadoop-6349-1.patch

Patch attached, this was as far as I got before getting distracted.
* Re-based on trunk. Cleans up carriage returns and indenting, removes most gratuitous uses of "this"
* Fixes some bugs in TestCodecPerformance.java
* Got JFastLZCompressor working, at least for the seeds that were breaking (the finished method was incorrect)
* Made some progress on JFastLZDecompressor but found a seed (see the one used in TestCodecPerformance) that causes a checksum missmatch and getRemaining needs to be implemented.

> Implement FastLZCodec for fastlz/lzo algorithm
> ----------------------------------------------
>
>                 Key: HADOOP-6349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6349
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: William Kinney
>         Attachments: hadoop-6349-1.patch, HADOOP-6349-TestFastLZCodec.patch, HADOOP-6349.patch, TestCodecPerformance.java, TestCodecPerformance.java, testCodecPerfResults.tsv
>
>
> Per  [HADOOP-4874|http://issues.apache.org/jira/browse/HADOOP-4874], FastLZ is a good (speed, license) alternative to LZO. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9626-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 01:49:00 2010
Return-Path: <common-issues-return-9626-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20780 invoked from network); 20 Jul 2010 01:48:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 01:48:59 -0000
Received: (qmail 56324 invoked by uid 500); 20 Jul 2010 01:48:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56264 invoked by uid 500); 20 Jul 2010 01:48:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56256 invoked by uid 99); 20 Jul 2010 01:48:58 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 01:48:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 01:48:56 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6K1f4wO014576
	for <common-issues@hadoop.apache.org>; Tue, 20 Jul 2010 01:41:05 GMT
Message-ID: <23514348.467461279590064931.JavaMail.jira@thor>
Date: Mon, 19 Jul 2010 21:41:04 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6349) Implement FastLZCodec for
 fastlz/lzo algorithm
In-Reply-To: <289817036.1256941439448.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890128#action_12890128 ] 

Eli Collins commented on HADOOP-6349:
-------------------------------------

Here's some performance data I collected a while back. I compressed and uncompressed a 135mb file (~ hdfs block size) of json data using the default command line utility for the codec, compiled with -Wall -O3 -fomit-frame-pointer on a nehalem-based system. I used an in-memory file system so the times were stable and took the best of 5 runs. These should be taken with a grain of salt since the same command line utility was not used with each codec. According to [1] zippy is 22% faster than lzo, fastlz is also 22% faster than lzo in the data below so ballpark performance is reasonable. We could optimize the native version further, curious to see what the overhead of calling out to jni is.

||codec	    ||fastlz(1)	||fastlz(2)||quicklz(1)	||quicklz(2) ||quicklz(3)||lzo (default)	  ||lzo (-9)||lzf||
|size	    |70m	|63M      |58M	        |49M	     |48M	 |61M	  |47M|69M|
|compress   |1.092s	|1.113s   |0.913s	|1.409s      |5.343s     |1.414s  |20.495s|1.110s|
|decompress |0.649s	|0.639s   |0.697s	|0.699s      |0.486s     |0.630s  |0.665s|0.729s|

1. http://feedblog.org/2008/10/12/google-bigtable-compression-zippy-and-bmdiff

> Implement FastLZCodec for fastlz/lzo algorithm
> ----------------------------------------------
>
>                 Key: HADOOP-6349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6349
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: William Kinney
>         Attachments: hadoop-6349-1.patch, HADOOP-6349-TestFastLZCodec.patch, HADOOP-6349.patch, TestCodecPerformance.java, TestCodecPerformance.java, testCodecPerfResults.tsv
>
>
> Per  [HADOOP-4874|http://issues.apache.org/jira/browse/HADOOP-4874], FastLZ is a good (speed, license) alternative to LZO. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9627-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 02:33:22 2010
Return-Path: <common-issues-return-9627-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40213 invoked from network); 20 Jul 2010 02:33:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 02:33:22 -0000
Received: (qmail 95844 invoked by uid 500); 20 Jul 2010 02:33:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95707 invoked by uid 500); 20 Jul 2010 02:33:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95699 invoked by uid 99); 20 Jul 2010 02:33:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 02:33:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 02:33:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6K2WuN9015021
	for <common-issues@hadoop.apache.org>; Tue, 20 Jul 2010 02:32:56 GMT
Message-ID: <21284241.468261279593176010.JavaMail.jira@thor>
Date: Mon, 19 Jul 2010 22:32:56 -0400 (EDT)
From: "Evan Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6865) there will be ant error if ant ran
 without network connected
In-Reply-To: <9668853.435891279373571387.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890134#action_12890134 ] 

Evan Wang commented on HADOOP-6865:
-----------------------------------

Yep, it is possible that ivy-2.0.0-rc2.jar is the broken lcy zip file. I remove it and put a original ivy-2.0.0-rc2.jar and ant works well. But how to revise ant build to do it automatically and how about building without network supporting. By the way, I am just a new user of ant, looking for a solution.

> there will be ant error if ant ran without network connected
> ------------------------------------------------------------
>
>                 Key: HADOOP-6865
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6865
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.20.2
>         Environment: centos 5.4
>            Reporter: Evan Wang
>
> If you run `ant` without network connected, there will be an error below. And even if you connect your network, the error will exist.
> ivy-init-antlib:
>   [typedef] java.util.zip.ZipException: error in opening zip file
>   [typedef]     at java.util.zip.ZipFile.open(Native Method)
>   [typedef]     at java.util.zip.ZipFile.<init>(ZipFile.java:114)
>   [typedef]     at java.util.zip.ZipFile.<init>(ZipFile.java:131)
>   [typedef]     at org.apache.tools.ant.AntClassLoader.getResourceURL(AntClassLoader.java:1028)
>   [typedef]     at org.apache.tools.ant.AntClassLoader$ResourceEnumeration.findNextResource(AntClassLoader.java:147)
>   [typedef]     at org.apache.tools.ant.AntClassLoader$ResourceEnumeration.<init>(AntClassLoader.java:109)
>   [typedef]     at org.apache.tools.ant.AntClassLoader.findResources(AntClassLoader.java:975)
>   [typedef]     at java.lang.ClassLoader.getResources(ClassLoader.java:1016)
>   [typedef]     at org.apache.tools.ant.taskdefs.Definer.resourceToURLs(Definer.java:364)
>   [typedef]     at org.apache.tools.ant.taskdefs.Definer.execute(Definer.java:256)
>   [typedef]     at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288)
>   [typedef]     at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
>   [typedef]     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>   [typedef]     at java.lang.reflect.Method.invoke(Method.java:597)
>   [typedef]     at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
>   [typedef]     at org.apache.tools.ant.Task.perform(Task.java:348)
>   [typedef]     at org.apache.tools.ant.Target.execute(Target.java:357)
>   [typedef]     at org.apache.tools.ant.Target.performTasks(Target.java:385)
>   [typedef]     at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1337)
>   [typedef]     at org.apache.tools.ant.Project.executeTarget(Project.java:1306)
>   [typedef]     at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
>   [typedef]     at org.apache.tools.ant.Project.executeTargets(Project.java:1189)
>   [typedef]     at org.apache.tools.ant.Main.runBuild(Main.java:758)
>   [typedef]     at org.apache.tools.ant.Main.startAnt(Main.java:217)
>   [typedef]     at org.apache.tools.ant.launch.Launcher.run(Launcher.java:257)
>   [typedef]     at org.apache.tools.ant.launch.Launcher.main(Launcher.java:104)
>   [typedef] Could not load definitions from resource org/apache/ivy/ant/antlib.xml. It could not be found.
> BUILD FAILED
> /opt/hadoop-0.20.2/build.xml:1644: You need Apache Ivy 2.0 or later from http://ant.apache.org/
>       It could not be loaded from http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.0.0-rc2/ivy-2.0.0-rc2.jar

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9628-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 02:39:02 2010
Return-Path: <common-issues-return-9628-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40633 invoked from network); 20 Jul 2010 02:39:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 02:39:02 -0000
Received: (qmail 98148 invoked by uid 500); 20 Jul 2010 02:39:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98014 invoked by uid 500); 20 Jul 2010 02:39:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98006 invoked by uid 99); 20 Jul 2010 02:39:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 02:39:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 02:38:57 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6K2V6Wn014993
	for <common-issues@hadoop.apache.org>; Tue, 20 Jul 2010 02:31:06 GMT
Message-ID: <2159590.468181279593066751.JavaMail.jira@thor>
Date: Mon, 19 Jul 2010 22:31:06 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6632) Support for using different
 Kerberos keys for different instances of Hadoop services
In-Reply-To: <1542465557.251491268540967230.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890132#action_12890132 ] 

Hudson commented on HADOOP-6632:
--------------------------------

Integrated in Hadoop-Hdfs-trunk-Commit #346 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/346/])
    HDFS-1201. The HDFS component for HADOOP-6632. Contributed by Kan Zhang & Jitendra Pandey.


> Support for using different Kerberos keys for different instances of Hadoop services
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6632
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6632
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>             Fix For: 0.22.0
>
>         Attachments: 6632.mr.patch, c6632-05.patch, c6632-07.patch, HADOOP-6632-Y20S-18.patch, HADOOP-6632-Y20S-22.patch
>
>
> We tested using the same Kerberos key for all datanodes in a HDFS cluster or the same Kerberos key for all TaskTarckers in a MapRed cluster. But it doesn't work. The reason is that when datanodes try to authenticate to the namenode all at once, the Kerberos authenticators they send to the namenode may have the same timestamp and will be rejected as replay requests. This JIRA makes it possible to use a unique key for each service instance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9629-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 03:43:18 2010
Return-Path: <common-issues-return-9629-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62963 invoked from network); 20 Jul 2010 03:43:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 03:43:18 -0000
Received: (qmail 55284 invoked by uid 500); 20 Jul 2010 03:43:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54968 invoked by uid 500); 20 Jul 2010 03:43:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54957 invoked by uid 99); 20 Jul 2010 03:43:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 03:43:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 03:43:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6K3gomH017231
	for <common-issues@hadoop.apache.org>; Tue, 20 Jul 2010 03:42:50 GMT
Message-ID: <12121562.469801279597370860.JavaMail.jira@thor>
Date: Mon, 19 Jul 2010 23:42:50 -0400 (EDT)
From: "Jordan Sissel (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6867) Using socket address for datanode
 registry breaks multihoming
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Using socket address for datanode registry breaks multihoming
-------------------------------------------------------------

                 Key: HADOOP-6867
                 URL: https://issues.apache.org/jira/browse/HADOOP-6867
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.20.2
         Environment: hadoop-0.20-0.20.2+228-1, centos 5, distcp
            Reporter: Jordan Sissel


Related: 
* https://issues.apache.org/jira/browse/HADOOP-985
* https://issues.apache.org/jira/secure/attachment/12350813/HADOOP-985-1.patch
* http://old.nabble.com/public-IP-for-datanode-on-EC2-td19336240.html
* http://www.cloudera.com/blog/2008/12/securing-a-hadoop-cluster-through-a-gateway/ 

Datanodes register using their dns name (even configurable with dfs.datanode.dns.interface). However, the Namenode only really uses the source address that the registration came from when sharing it to clients wanting to write to HDFS.

Specific environment that causes this problem:
* Datanode and Namenode multihomed on two networks.
* Datanode registers to namenode using dns name on network #1
* Client (distcp) connects to namenode on network #2 (*) and is told to write to datanodes on network #1, which doesn't work for us.

(*) Allowing contact to the namenode on multiple networks was achieved with a socat proxy hack that tunnels network#2 to network#1 port 8020. This is unrelated to the issue at hand.


The cloudera link above recommends proxying for other reasons than multihoming, but it would work, but it doesn't sound like it would well (bandwidth, multiplicity, multitenant, etc).

Our specific scenario is wanting to distcp over a different network interface than the datanodes register themselves on, but it would be nice if both (all) interfaces worked. We are internally going to patch hadoop to roll back parts of the patch mentioned above so that we rely the datanode name rather than the socket address it uses to talk to the namenode. The alternate option is to push config changes to all nodes that force them to listen/register on one specific interface only. This helps us work around our specific problem, but doesn't really help with multihoming. 

I would propose that datanodes register all interface addresses during the registration/heartbeat/whatever process does this and hdfs clients would be given all addresses for a specific node to perform operations against and they could select accordingly (or 'whichever worked first') just like round-robin dns does.


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9630-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 04:04:19 2010
Return-Path: <common-issues-return-9630-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65827 invoked from network); 20 Jul 2010 04:04:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 04:04:19 -0000
Received: (qmail 72758 invoked by uid 500); 20 Jul 2010 04:04:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72624 invoked by uid 500); 20 Jul 2010 04:04:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72615 invoked by uid 99); 20 Jul 2010 04:04:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 04:04:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 04:04:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6K43ppM017619
	for <common-issues@hadoop.apache.org>; Tue, 20 Jul 2010 04:03:51 GMT
Message-ID: <22896652.469931279598631079.JavaMail.jira@thor>
Date: Tue, 20 Jul 2010 00:03:51 -0400 (EDT)
From: "Ravi Gummadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6536) FileUtil.fullyDelete(dir) behavior is
 not defined when we pass a symlink as the argument
In-Reply-To: <508958946.48471265176519280.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated HADOOP-6536:
---------------------------------

    Status: Open  (was: Patch Available)

> FileUtil.fullyDelete(dir) behavior is not defined when we pass a symlink as the argument
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6536
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6536
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Ravi Gummadi
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6536.patch, HADOOP-6536.v1.1.patch, HADOOP-6536.v1.patch
>
>
> FileUtil.fullyDelete(dir) deletes contents of sym-linked directory when we pass a symlink. If this is the behavior, it should be documented as so. 
> Or it should be changed not to delete the contents of the sym-linked directory.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9631-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 04:13:50 2010
Return-Path: <common-issues-return-9631-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72053 invoked from network); 20 Jul 2010 04:13:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 04:13:50 -0000
Received: (qmail 80177 invoked by uid 500); 20 Jul 2010 04:13:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79937 invoked by uid 500); 20 Jul 2010 04:13:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79920 invoked by uid 99); 20 Jul 2010 04:13:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 04:13:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 04:13:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6K45p7a017858
	for <common-issues@hadoop.apache.org>; Tue, 20 Jul 2010 04:05:52 GMT
Message-ID: <31952715.470021279598751586.JavaMail.jira@thor>
Date: Tue, 20 Jul 2010 00:05:51 -0400 (EDT)
From: "Ravi Gummadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6536) FileUtil.fullyDelete(dir) behavior is
 not defined when we pass a symlink as the argument
In-Reply-To: <508958946.48471265176519280.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated HADOOP-6536:
---------------------------------

    Status: Patch Available  (was: Open)

The failed test TestTrash passes on my local machine. Allowing to go through Hudson again...

> FileUtil.fullyDelete(dir) behavior is not defined when we pass a symlink as the argument
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6536
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6536
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Ravi Gummadi
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6536.patch, HADOOP-6536.v1.1.patch, HADOOP-6536.v1.patch
>
>
> FileUtil.fullyDelete(dir) deletes contents of sym-linked directory when we pass a symlink. If this is the behavior, it should be documented as so. 
> Or it should be changed not to delete the contents of the sym-linked directory.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9632-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 04:40:21 2010
Return-Path: <common-issues-return-9632-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75834 invoked from network); 20 Jul 2010 04:40:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 04:40:21 -0000
Received: (qmail 97939 invoked by uid 500); 20 Jul 2010 04:40:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97807 invoked by uid 500); 20 Jul 2010 04:40:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97623 invoked by uid 99); 20 Jul 2010 04:40:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 04:40:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 04:40:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6K4drR4018026
	for <common-issues@hadoop.apache.org>; Tue, 20 Jul 2010 04:39:53 GMT
Message-ID: <23139464.470171279600793097.JavaMail.jira@thor>
Date: Tue, 20 Jul 2010 00:39:53 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6536) FileUtil.fullyDelete(dir) behavior
 is not defined when we pass a symlink as the argument
In-Reply-To: <508958946.48471265176519280.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890180#action_12890180 ] 

Hadoop QA commented on HADOOP-6536:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12449821/HADOOP-6536.v1.1.patch
  against trunk revision 965696.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/625/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/625/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/625/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/625/console

This message is automatically generated.

> FileUtil.fullyDelete(dir) behavior is not defined when we pass a symlink as the argument
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6536
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6536
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Ravi Gummadi
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6536.patch, HADOOP-6536.v1.1.patch, HADOOP-6536.v1.patch
>
>
> FileUtil.fullyDelete(dir) deletes contents of sym-linked directory when we pass a symlink. If this is the behavior, it should be documented as so. 
> Or it should be changed not to delete the contents of the sym-linked directory.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9633-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 05:14:18 2010
Return-Path: <common-issues-return-9633-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84077 invoked from network); 20 Jul 2010 05:14:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 05:14:18 -0000
Received: (qmail 26703 invoked by uid 500); 20 Jul 2010 05:14:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26566 invoked by uid 500); 20 Jul 2010 05:14:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26558 invoked by uid 99); 20 Jul 2010 05:14:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 05:14:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 05:14:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6K5DpbC018411
	for <common-issues@hadoop.apache.org>; Tue, 20 Jul 2010 05:13:51 GMT
Message-ID: <6979432.470541279602831037.JavaMail.jira@thor>
Date: Tue, 20 Jul 2010 01:13:51 -0400 (EDT)
From: "Ravi Gummadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6536) FileUtil.fullyDelete(dir) behavior
 is not defined when we pass a symlink as the argument
In-Reply-To: <508958946.48471265176519280.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890183#action_12890183 ] 

Ravi Gummadi commented on HADOOP-6536:
--------------------------------------

>> -1 javadoc. The javadoc tool appears to have generated 1 warning messages.
No javadoc warning is introduced in this patch.

> FileUtil.fullyDelete(dir) behavior is not defined when we pass a symlink as the argument
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6536
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6536
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Ravi Gummadi
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6536.patch, HADOOP-6536.v1.1.patch, HADOOP-6536.v1.patch
>
>
> FileUtil.fullyDelete(dir) deletes contents of sym-linked directory when we pass a symlink. If this is the behavior, it should be documented as so. 
> Or it should be changed not to delete the contents of the sym-linked directory.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9634-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 05:25:19 2010
Return-Path: <common-issues-return-9634-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85766 invoked from network); 20 Jul 2010 05:25:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 05:25:19 -0000
Received: (qmail 30424 invoked by uid 500); 20 Jul 2010 05:25:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30299 invoked by uid 500); 20 Jul 2010 05:25:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30291 invoked by uid 99); 20 Jul 2010 05:25:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 05:25:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 05:25:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6K5OpQ7018974
	for <common-issues@hadoop.apache.org>; Tue, 20 Jul 2010 05:24:51 GMT
Message-ID: <28029932.470611279603491347.JavaMail.jira@thor>
Date: Tue, 20 Jul 2010 01:24:51 -0400 (EDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6536) FileUtil.fullyDelete(dir) behavior is
 not defined when we pass a symlink as the argument
In-Reply-To: <508958946.48471265176519280.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amareshwari Sriramadasu updated HADOOP-6536:
--------------------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

I just committed this. Thanks Ravi!

> FileUtil.fullyDelete(dir) behavior is not defined when we pass a symlink as the argument
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6536
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6536
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Ravi Gummadi
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6536.patch, HADOOP-6536.v1.1.patch, HADOOP-6536.v1.patch
>
>
> FileUtil.fullyDelete(dir) deletes contents of sym-linked directory when we pass a symlink. If this is the behavior, it should be documented as so. 
> Or it should be changed not to delete the contents of the sym-linked directory.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9635-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 05:45:20 2010
Return-Path: <common-issues-return-9635-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89256 invoked from network); 20 Jul 2010 05:45:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 05:45:20 -0000
Received: (qmail 37787 invoked by uid 500); 20 Jul 2010 05:45:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37661 invoked by uid 500); 20 Jul 2010 05:45:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37653 invoked by uid 99); 20 Jul 2010 05:45:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 05:45:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 05:45:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6K5ipKO019684
	for <common-issues@hadoop.apache.org>; Tue, 20 Jul 2010 05:44:52 GMT
Message-ID: <10986176.470761279604691786.JavaMail.jira@thor>
Date: Tue, 20 Jul 2010 01:44:51 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6536) FileUtil.fullyDelete(dir) behavior
 is not defined when we pass a symlink as the argument
In-Reply-To: <508958946.48471265176519280.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890188#action_12890188 ] 

Hudson commented on HADOOP-6536:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #332 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/332/])
    HADOOP-6536. Fixes FileUtil.fullyDelete() not to delete the contents of the sym-linked directory. Contributed by Ravi Gummadi


> FileUtil.fullyDelete(dir) behavior is not defined when we pass a symlink as the argument
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6536
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6536
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Ravi Gummadi
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6536.patch, HADOOP-6536.v1.1.patch, HADOOP-6536.v1.patch
>
>
> FileUtil.fullyDelete(dir) deletes contents of sym-linked directory when we pass a symlink. If this is the behavior, it should be documented as so. 
> Or it should be changed not to delete the contents of the sym-linked directory.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9636-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 11:22:19 2010
Return-Path: <common-issues-return-9636-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83452 invoked from network); 20 Jul 2010 11:22:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 11:22:19 -0000
Received: (qmail 58732 invoked by uid 500); 20 Jul 2010 11:22:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58597 invoked by uid 500); 20 Jul 2010 11:22:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58583 invoked by uid 99); 20 Jul 2010 11:22:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 11:22:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 11:22:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6KBLqab023947
	for <common-issues@hadoop.apache.org>; Tue, 20 Jul 2010 11:21:52 GMT
Message-ID: <8723889.473451279624912193.JavaMail.jira@thor>
Date: Tue, 20 Jul 2010 07:21:52 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6805) add buildDTServiceName method to
 SecurityUtil (as part of MAPREDUCE-1718)
In-Reply-To: <5393650.148181275524636022.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890238#action_12890238 ] 

Hudson commented on HADOOP-6805:
--------------------------------

Integrated in Hadoop-Common-trunk #398 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/398/])
    HADOOP-6805. add buildDTServiceName method to SecurityUtil (as part of MAPREDUCE-1718)


> add buildDTServiceName method to SecurityUtil (as part of MAPREDUCE-1718)
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6805
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6805
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6805.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9637-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 11:22:20 2010
Return-Path: <common-issues-return-9637-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83522 invoked from network); 20 Jul 2010 11:22:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 11:22:20 -0000
Received: (qmail 58871 invoked by uid 500); 20 Jul 2010 11:22:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58607 invoked by uid 500); 20 Jul 2010 11:22:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58537 invoked by uid 99); 20 Jul 2010 11:22:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 11:22:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 11:22:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6KBLp1R023941
	for <common-issues@hadoop.apache.org>; Tue, 20 Jul 2010 11:21:51 GMT
Message-ID: <5845586.473431279624911458.JavaMail.jira@thor>
Date: Tue, 20 Jul 2010 07:21:51 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6536) FileUtil.fullyDelete(dir) behavior
 is not defined when we pass a symlink as the argument
In-Reply-To: <508958946.48471265176519280.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890237#action_12890237 ] 

Hudson commented on HADOOP-6536:
--------------------------------

Integrated in Hadoop-Common-trunk #398 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/398/])
    HADOOP-6536. Fixes FileUtil.fullyDelete() not to delete the contents of the sym-linked directory. Contributed by Ravi Gummadi


> FileUtil.fullyDelete(dir) behavior is not defined when we pass a symlink as the argument
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6536
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6536
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Ravi Gummadi
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6536.patch, HADOOP-6536.v1.1.patch, HADOOP-6536.v1.patch
>
>
> FileUtil.fullyDelete(dir) deletes contents of sym-linked directory when we pass a symlink. If this is the behavior, it should be documented as so. 
> Or it should be changed not to delete the contents of the sym-linked directory.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9638-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 11:29:51 2010
Return-Path: <common-issues-return-9638-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84917 invoked from network); 20 Jul 2010 11:29:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 11:29:50 -0000
Received: (qmail 72235 invoked by uid 500); 20 Jul 2010 11:29:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72020 invoked by uid 500); 20 Jul 2010 11:29:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72005 invoked by uid 99); 20 Jul 2010 11:29:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 11:29:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 11:29:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6KBLrej023965
	for <common-issues@hadoop.apache.org>; Tue, 20 Jul 2010 11:21:53 GMT
Message-ID: <14960005.473511279624913086.JavaMail.jira@thor>
Date: Tue, 20 Jul 2010 07:21:53 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6632) Support for using different
 Kerberos keys for different instances of Hadoop services
In-Reply-To: <1542465557.251491268540967230.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890240#action_12890240 ] 

Hudson commented on HADOOP-6632:
--------------------------------

Integrated in Hadoop-Common-trunk #398 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/398/])
    HADOOP-6632. Adds support for using different keytabs for different servers in a Hadoop cluster. In the earier implementation, all servers of a certain type \(like TaskTracker\), would have the same keytab and the same principal. Now the principal name is a pattern that has _HOST in it. Contributed by Kan Zhang & Jitendra Pandey.


> Support for using different Kerberos keys for different instances of Hadoop services
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6632
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6632
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>             Fix For: 0.22.0
>
>         Attachments: 6632.mr.patch, c6632-05.patch, c6632-07.patch, HADOOP-6632-Y20S-18.patch, HADOOP-6632-Y20S-22.patch
>
>
> We tested using the same Kerberos key for all datanodes in a HDFS cluster or the same Kerberos key for all TaskTarckers in a MapRed cluster. But it doesn't work. The reason is that when datanodes try to authenticate to the namenode all at once, the Kerberos authenticators they send to the namenode may have the same timestamp and will be rejected as replay requests. This JIRA makes it possible to use a unique key for each service instance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9639-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 14:29:46 2010
Return-Path: <common-issues-return-9639-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43328 invoked from network); 20 Jul 2010 14:29:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 14:29:46 -0000
Received: (qmail 20524 invoked by uid 500); 20 Jul 2010 14:29:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20397 invoked by uid 500); 20 Jul 2010 14:29:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20389 invoked by uid 99); 20 Jul 2010 14:29:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 14:29:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 14:29:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6KELoHv027272
	for <common-issues@hadoop.apache.org>; Tue, 20 Jul 2010 14:21:50 GMT
Message-ID: <18500159.474631279635710075.JavaMail.jira@thor>
Date: Tue, 20 Jul 2010 10:21:50 -0400 (EDT)
From: "Ted Yu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6868) Text class should provide method to
 return byte array of getLength()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Text class should provide method to return byte array of getLength()
--------------------------------------------------------------------

                 Key: HADOOP-6868
                 URL: https://issues.apache.org/jira/browse/HADOOP-6868
             Project: Hadoop Common
          Issue Type: Bug
          Components: util
    Affects Versions: 0.20.2
            Reporter: Ted Yu


People would use the following code to convert Text to String:
String valueString = new String(valueText.getBytes(), "UTF-8");

However, if Text is reused, the above call would return String of monotonically increasing length.
>From 'Hadoop and XML' discussion thread:
The problem I am seeing is between the Map phase and the
Reduce phase, the XML is getting munged.  For Example:
 </PrivateRate>
 </PrivateRateSet>te>

Text should provide method to return byte array of getLength() length.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9640-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 16:48:45 2010
Return-Path: <common-issues-return-9640-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96975 invoked from network); 20 Jul 2010 16:48:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 16:48:45 -0000
Received: (qmail 46491 invoked by uid 500); 20 Jul 2010 16:48:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46442 invoked by uid 500); 20 Jul 2010 16:48:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46434 invoked by uid 99); 20 Jul 2010 16:48:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 16:48:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 16:48:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6KGeohj029911
	for <common-issues@hadoop.apache.org>; Tue, 20 Jul 2010 16:40:50 GMT
Message-ID: <8400181.477331279644050140.JavaMail.jira@thor>
Date: Tue, 20 Jul 2010 12:40:50 -0400 (EDT)
From: "Ted Yu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6868) Text class should provide method to
 return byte array of getLength()
In-Reply-To: <18500159.474631279635710075.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890305#action_12890305 ] 

Ted Yu commented on HADOOP-6868:
--------------------------------

So the correct call in the above use case should be:
String valueString = new String(valueText.getBytes(), 0, valueText.getLength(), "UTF-8");

Text class Java doc should document explicitly about this use case.

> Text class should provide method to return byte array of getLength()
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6868
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6868
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.20.2
>            Reporter: Ted Yu
>
> People would use the following code to convert Text to String:
> String valueString = new String(valueText.getBytes(), "UTF-8");
> However, if Text is reused, the above call would return String of monotonically increasing length.
> From 'Hadoop and XML' discussion thread:
> The problem I am seeing is between the Map phase and the
> Reduce phase, the XML is getting munged.  For Example:
>  </PrivateRate>
>  </PrivateRateSet>te>
> Text should provide method to return byte array of getLength() length.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9641-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 16:57:46 2010
Return-Path: <common-issues-return-9641-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 428 invoked from network); 20 Jul 2010 16:57:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 16:57:45 -0000
Received: (qmail 54990 invoked by uid 500); 20 Jul 2010 16:57:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54501 invoked by uid 500); 20 Jul 2010 16:57:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54438 invoked by uid 99); 20 Jul 2010 16:57:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 16:57:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 16:57:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6KGnpBp000105
	for <common-issues@hadoop.apache.org>; Tue, 20 Jul 2010 16:49:51 GMT
Message-ID: <18316223.477571279644591588.JavaMail.jira@thor>
Date: Tue, 20 Jul 2010 12:49:51 -0400 (EDT)
From: "Ted Yu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6868) Text class should provide method to
 return byte array of getLength()
In-Reply-To: <18500159.474631279635710075.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890312#action_12890312 ] 

Ted Yu commented on HADOOP-6868:
--------------------------------

>From Peter Minearo:
Let's say you create a Text object and drop in a String that sets the byte array length to 200.  Then drop in a a second String that sets the byte array length to 500.  Since, the new length is greater than the previous length; the byte array length is reset to the longer length.  Now, if you drop in a third String that would set the byte array length to 350; the Text object does not replace the byte array with a new length of 350; it utilizes the greater length of 500 and sets an extra variable to track the "real" length.

So: Text.getBytes().length != Text.getLength()

This does 2 things:

1. Passes around more data than what is needed
2. Makes the Text object confusing to work with

Text.getBytes().length == Text.getLength() - should be the correct behavior.

> Text class should provide method to return byte array of getLength()
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6868
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6868
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.20.2
>            Reporter: Ted Yu
>
> People would use the following code to convert Text to String:
> String valueString = new String(valueText.getBytes(), "UTF-8");
> However, if Text is reused, the above call would return String of monotonically increasing length.
> From 'Hadoop and XML' discussion thread:
> The problem I am seeing is between the Map phase and the
> Reduce phase, the XML is getting munged.  For Example:
>  </PrivateRate>
>  </PrivateRateSet>te>
> Text should provide method to return byte array of getLength() length.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9642-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 18:35:45 2010
Return-Path: <common-issues-return-9642-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45514 invoked from network); 20 Jul 2010 18:35:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 18:35:45 -0000
Received: (qmail 14072 invoked by uid 500); 20 Jul 2010 18:35:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14039 invoked by uid 500); 20 Jul 2010 18:35:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14031 invoked by uid 99); 20 Jul 2010 18:35:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 18:35:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 18:35:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6KIRnMd001896
	for <common-issues@hadoop.apache.org>; Tue, 20 Jul 2010 18:27:50 GMT
Message-ID: <20016878.480251279650469922.JavaMail.jira@thor>
Date: Tue, 20 Jul 2010 14:27:49 -0400 (EDT)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6868) Text class should provide method to
 return byte array of getLength()
In-Reply-To: <18500159.474631279635710075.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890374#action_12890374 ] 

Scott Carey commented on HADOOP-6868:
-------------------------------------


I don't think that is a bug.  Passing around byte arrays larger than the valid data is common practice in Java for performance reasons.  Hence, the common method signature containing  (byte[] bytes, int len, int offset) and similar.   Creating a new byte array for each resize defeats the purpose of re-using the byte array and the Text object -- lower memory allocation and improved CPU cache locality. The byte array here is a buffer, it does not represent the entire string.

Instead, use the 'helper' method, toString(). 



> Text class should provide method to return byte array of getLength()
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6868
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6868
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.20.2
>            Reporter: Ted Yu
>
> People would use the following code to convert Text to String:
> String valueString = new String(valueText.getBytes(), "UTF-8");
> However, if Text is reused, the above call would return String of monotonically increasing length.
> From 'Hadoop and XML' discussion thread:
> The problem I am seeing is between the Map phase and the
> Reduce phase, the XML is getting munged.  For Example:
>  </PrivateRate>
>  </PrivateRateSet>te>
> Text should provide method to return byte array of getLength() length.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9643-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 20 20:01:16 2010
Return-Path: <common-issues-return-9643-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73141 invoked from network); 20 Jul 2010 20:01:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jul 2010 20:01:16 -0000
Received: (qmail 13253 invoked by uid 500); 20 Jul 2010 20:01:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13160 invoked by uid 500); 20 Jul 2010 20:01:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13035 invoked by uid 99); 20 Jul 2010 20:01:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 20:01:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Jul 2010 20:01:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6KK0pcX003259
	for <common-issues@hadoop.apache.org>; Tue, 20 Jul 2010 20:00:51 GMT
Message-ID: <3148671.482421279656051365.JavaMail.jira@thor>
Date: Tue, 20 Jul 2010 16:00:51 -0400 (EDT)
From: "Ted Yu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6868) Text class should provide method to
 return byte array of getLength()
In-Reply-To: <18500159.474631279635710075.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ted Yu resolved HADOOP-6868.
----------------------------

    Resolution: Not A Problem

> Text class should provide method to return byte array of getLength()
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6868
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6868
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.20.2
>            Reporter: Ted Yu
>
> People would use the following code to convert Text to String:
> String valueString = new String(valueText.getBytes(), "UTF-8");
> However, if Text is reused, the above call would return String of monotonically increasing length.
> From 'Hadoop and XML' discussion thread:
> The problem I am seeing is between the Map phase and the
> Reduce phase, the XML is getting munged.  For Example:
>  </PrivateRate>
>  </PrivateRateSet>te>
> Text should provide method to return byte array of getLength() length.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9644-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 17:07:17 2010
Return-Path: <common-issues-return-9644-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81989 invoked from network); 21 Jul 2010 17:07:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 17:07:16 -0000
Received: (qmail 34059 invoked by uid 500); 21 Jul 2010 17:07:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33568 invoked by uid 500); 21 Jul 2010 17:07:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33553 invoked by uid 99); 21 Jul 2010 17:07:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 17:07:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 17:07:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6LH6oXQ015758
	for <common-issues@hadoop.apache.org>; Wed, 21 Jul 2010 17:06:50 GMT
Message-ID: <1398369.499011279732010273.JavaMail.jira@thor>
Date: Wed, 21 Jul 2010 13:06:50 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6869) [Herriot] Implement a functionality
 for creating either file or folder in task attempt  folder while job is
 running.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

[Herriot] Implement a functionality for creating either file or folder in task attempt  folder while job is running.
--------------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-6869
                 URL: https://issues.apache.org/jira/browse/HADOOP-6869
             Project: Hadoop Common
          Issue Type: Task
          Components: test
            Reporter: Vinay Kumar Thota


Functionality for creating either files or folders in task attempt folder while job is running. The functionality covers the following methods.

1. public void DaemonProtocol.createFile(String path, String fileName, boolean local) throws IOException; 
It uses to create a file with full permissions.

2.   public void DaemonProtocol.createFile(String path, String fileName, FsPermission permission, boolean local) throws IOException; 
It uses to create a file with given permissions.

3.   public void DaemonProtocol.createFolder(String path, String folderName, boolean local) throws IOException;
It uses to create a file with full permissions.

4.   public void DaemonProtocol.createFolder(String path, String folderName, FsPermission permission, boolean local) throws IOException;
It uses to create a folder with given permissions.


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9645-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 17:18:46 2010
Return-Path: <common-issues-return-9645-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86030 invoked from network); 21 Jul 2010 17:18:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 17:18:46 -0000
Received: (qmail 49766 invoked by uid 500); 21 Jul 2010 17:18:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49724 invoked by uid 500); 21 Jul 2010 17:18:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49708 invoked by uid 99); 21 Jul 2010 17:18:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 17:18:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 17:18:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6LHAoBq015793
	for <common-issues@hadoop.apache.org>; Wed, 21 Jul 2010 17:10:50 GMT
Message-ID: <18208934.499081279732250961.JavaMail.jira@thor>
Date: Wed, 21 Jul 2010 13:10:50 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6869) [Herriot] Implement a functionality
 for creating either file or folder in task attempt  folder while job is
 running.
In-Reply-To: <1398369.499011279732010273.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinay Kumar Thota reassigned HADOOP-6869:
-----------------------------------------

    Assignee: Vinay Kumar Thota

> [Herriot] Implement a functionality for creating either file or folder in task attempt  folder while job is running.
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6869
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6869
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6869-ydist-security.patch
>
>
> Functionality for creating either files or folders in task attempt folder while job is running. The functionality covers the following methods.
> 1. public void DaemonProtocol.createFile(String path, String fileName, boolean local) throws IOException; 
> It uses to create a file with full permissions.
> 2.   public void DaemonProtocol.createFile(String path, String fileName, FsPermission permission, boolean local) throws IOException; 
> It uses to create a file with given permissions.
> 3.   public void DaemonProtocol.createFolder(String path, String folderName, boolean local) throws IOException;
> It uses to create a file with full permissions.
> 4.   public void DaemonProtocol.createFolder(String path, String folderName, FsPermission permission, boolean local) throws IOException;
> It uses to create a folder with given permissions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9646-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 17:18:47 2010
Return-Path: <common-issues-return-9646-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86081 invoked from network); 21 Jul 2010 17:18:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 17:18:47 -0000
Received: (qmail 49901 invoked by uid 500); 21 Jul 2010 17:18:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49756 invoked by uid 500); 21 Jul 2010 17:18:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49709 invoked by uid 99); 21 Jul 2010 17:18:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 17:18:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 17:18:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6LHAo7I015787
	for <common-issues@hadoop.apache.org>; Wed, 21 Jul 2010 17:10:50 GMT
Message-ID: <947740.499061279732250502.JavaMail.jira@thor>
Date: Wed, 21 Jul 2010 13:10:50 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6869) [Herriot] Implement a functionality
 for creating either file or folder in task attempt  folder while job is
 running.
In-Reply-To: <1398369.499011279732010273.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinay Kumar Thota updated HADOOP-6869:
--------------------------------------

    Attachment: 6869-ydist-security.patch

Initial patch for yahoo dist security branch.

> [Herriot] Implement a functionality for creating either file or folder in task attempt  folder while job is running.
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6869
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6869
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>            Reporter: Vinay Kumar Thota
>         Attachments: 6869-ydist-security.patch
>
>
> Functionality for creating either files or folders in task attempt folder while job is running. The functionality covers the following methods.
> 1. public void DaemonProtocol.createFile(String path, String fileName, boolean local) throws IOException; 
> It uses to create a file with full permissions.
> 2.   public void DaemonProtocol.createFile(String path, String fileName, FsPermission permission, boolean local) throws IOException; 
> It uses to create a file with given permissions.
> 3.   public void DaemonProtocol.createFolder(String path, String folderName, boolean local) throws IOException;
> It uses to create a file with full permissions.
> 4.   public void DaemonProtocol.createFolder(String path, String folderName, FsPermission permission, boolean local) throws IOException;
> It uses to create a folder with given permissions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9647-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 17:19:16 2010
Return-Path: <common-issues-return-9647-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86154 invoked from network); 21 Jul 2010 17:19:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 17:19:16 -0000
Received: (qmail 50441 invoked by uid 500); 21 Jul 2010 17:19:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50384 invoked by uid 500); 21 Jul 2010 17:19:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50275 invoked by uid 99); 21 Jul 2010 17:19:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 17:19:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 17:19:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6LHInq1015914
	for <common-issues@hadoop.apache.org>; Wed, 21 Jul 2010 17:18:50 GMT
Message-ID: <4694242.499101279732729916.JavaMail.jira@thor>
Date: Wed, 21 Jul 2010 13:18:49 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6869) [Herriot] Implement a functionality
 for creating either file or folder in task attempt  folder while job is
 running.
In-Reply-To: <1398369.499011279732010273.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinay Kumar Thota updated HADOOP-6869:
--------------------------------------

    Attachment:     (was: 6869-ydist-security.patch)

> [Herriot] Implement a functionality for creating either file or folder in task attempt  folder while job is running.
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6869
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6869
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6869-ydist-security.patch
>
>
> Functionality for creating either files or folders in task attempt folder while job is running. The functionality covers the following methods.
> 1. public void DaemonProtocol.createFile(String path, String fileName, boolean local) throws IOException; 
> It uses to create a file with full permissions.
> 2.   public void DaemonProtocol.createFile(String path, String fileName, FsPermission permission, boolean local) throws IOException; 
> It uses to create a file with given permissions.
> 3.   public void DaemonProtocol.createFolder(String path, String folderName, boolean local) throws IOException;
> It uses to create a file with full permissions.
> 4.   public void DaemonProtocol.createFolder(String path, String folderName, FsPermission permission, boolean local) throws IOException;
> It uses to create a folder with given permissions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9648-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 17:26:50 2010
Return-Path: <common-issues-return-9648-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88596 invoked from network); 21 Jul 2010 17:26:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 17:26:46 -0000
Received: (qmail 60128 invoked by uid 500); 21 Jul 2010 17:26:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60071 invoked by uid 500); 21 Jul 2010 17:26:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60050 invoked by uid 99); 21 Jul 2010 17:26:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 17:26:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 17:26:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6LHIoX1015920
	for <common-issues@hadoop.apache.org>; Wed, 21 Jul 2010 17:18:50 GMT
Message-ID: <19300947.499121279732730606.JavaMail.jira@thor>
Date: Wed, 21 Jul 2010 13:18:50 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6869) [Herriot] Implement a functionality
 for creating either file or folder in task attempt  folder while job is
 running.
In-Reply-To: <1398369.499011279732010273.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinay Kumar Thota updated HADOOP-6869:
--------------------------------------

    Attachment: 6869-ydist-security.patch

> [Herriot] Implement a functionality for creating either file or folder in task attempt  folder while job is running.
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6869
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6869
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6869-ydist-security.patch
>
>
> Functionality for creating either files or folders in task attempt folder while job is running. The functionality covers the following methods.
> 1. public void DaemonProtocol.createFile(String path, String fileName, boolean local) throws IOException; 
> It uses to create a file with full permissions.
> 2.   public void DaemonProtocol.createFile(String path, String fileName, FsPermission permission, boolean local) throws IOException; 
> It uses to create a file with given permissions.
> 3.   public void DaemonProtocol.createFolder(String path, String folderName, boolean local) throws IOException;
> It uses to create a file with full permissions.
> 4.   public void DaemonProtocol.createFolder(String path, String folderName, FsPermission permission, boolean local) throws IOException;
> It uses to create a folder with given permissions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9649-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 17:35:44 2010
Return-Path: <common-issues-return-9649-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93452 invoked from network); 21 Jul 2010 17:35:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 17:35:44 -0000
Received: (qmail 73088 invoked by uid 500); 21 Jul 2010 17:35:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73043 invoked by uid 500); 21 Jul 2010 17:35:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73035 invoked by uid 99); 21 Jul 2010 17:35:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 17:35:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 17:35:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6LHRncM016152
	for <common-issues@hadoop.apache.org>; Wed, 21 Jul 2010 17:27:49 GMT
Message-ID: <14300994.499601279733269648.JavaMail.jira@thor>
Date: Wed, 21 Jul 2010 13:27:49 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6869) [Herriot] Implement a functionality
 for creating either file or folder in task attempt  folder while job is
 running.
In-Reply-To: <1398369.499011279732010273.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinay Kumar Thota updated HADOOP-6869:
--------------------------------------

    Attachment: HADOOP-6869.patch

Patch for trunk.

> [Herriot] Implement a functionality for creating either file or folder in task attempt  folder while job is running.
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6869
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6869
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6869-ydist-security.patch, HADOOP-6869.patch
>
>
> Functionality for creating either files or folders in task attempt folder while job is running. The functionality covers the following methods.
> 1. public void DaemonProtocol.createFile(String path, String fileName, boolean local) throws IOException; 
> It uses to create a file with full permissions.
> 2.   public void DaemonProtocol.createFile(String path, String fileName, FsPermission permission, boolean local) throws IOException; 
> It uses to create a file with given permissions.
> 3.   public void DaemonProtocol.createFolder(String path, String folderName, boolean local) throws IOException;
> It uses to create a file with full permissions.
> 4.   public void DaemonProtocol.createFolder(String path, String folderName, FsPermission permission, boolean local) throws IOException;
> It uses to create a folder with given permissions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9650-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 18:15:17 2010
Return-Path: <common-issues-return-9650-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12554 invoked from network); 21 Jul 2010 18:15:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 18:15:17 -0000
Received: (qmail 27374 invoked by uid 500); 21 Jul 2010 18:15:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27312 invoked by uid 500); 21 Jul 2010 18:15:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27302 invoked by uid 99); 21 Jul 2010 18:15:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 18:15:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 18:15:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6LIEpiw016776
	for <common-issues@hadoop.apache.org>; Wed, 21 Jul 2010 18:14:52 GMT
Message-ID: <5917221.500381279736091889.JavaMail.jira@thor>
Date: Wed, 21 Jul 2010 14:14:51 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6810) Extract a subset of tests for smoke
 (DOA) validation
In-Reply-To: <10151977.193211275695163699.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik resolved HADOOP-6810.
----------------------------------------

    Resolution: Later

I'd like to close it for now as 'Later' until Common has enough tests to separate a smoke suite. Right not it doesn't make much sense.

> Extract a subset of tests for smoke (DOA) validation
> ----------------------------------------------------
>
>                 Key: HADOOP-6810
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6810
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: smoke-tests.1.patch, smoke-tests.patch, smoke-tests.patch, smoke-tests.patch, smoke-tests.patch, smoke-tests.patch
>
>
> Currently, there's no way to quickly find out if a new build of Hadoop is reasonably viable and can be used for future development/testing cycle.
> Similarly to 'commit' testing it'd be great to have a wider subset of existing tests to be executed inside of 30 minutes. The main goal of such a subset is to have a reasonable guarantees that different parts of Hadoop's functionality aren't broken and the system is not DOA.
> Full test run takes about 3+ hours and it is a too long wait time in many cases. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9651-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 18:18:20 2010
Return-Path: <common-issues-return-9651-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13254 invoked from network); 21 Jul 2010 18:18:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 18:18:20 -0000
Received: (qmail 32512 invoked by uid 500); 21 Jul 2010 18:18:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32354 invoked by uid 500); 21 Jul 2010 18:18:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32346 invoked by uid 99); 21 Jul 2010 18:18:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 18:18:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 18:18:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6LIHtjV016850
	for <common-issues@hadoop.apache.org>; Wed, 21 Jul 2010 18:17:55 GMT
Message-ID: <21327290.500611279736275037.JavaMail.jira@thor>
Date: Wed, 21 Jul 2010 14:17:55 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6786) test-patch needs to verify Herriot
 integrity
In-Reply-To: <19336182.46451274829212745.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890807#action_12890807 ] 

Konstantin Boudnik commented on HADOOP-6786:
--------------------------------------------

A review, please? All blockers were resolved.

> test-patch needs to verify Herriot integrity
> --------------------------------------------
>
>                 Key: HADOOP-6786
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6786
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-6786.patch
>
>
> Whenever a new patch is submitted for verification {{test-patch}} process has to make sure that none of Herriot bindings were broken.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9652-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 21:23:48 2010
Return-Path: <common-issues-return-9652-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98830 invoked from network); 21 Jul 2010 21:23:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 21:23:47 -0000
Received: (qmail 83199 invoked by uid 500); 21 Jul 2010 21:23:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82860 invoked by uid 500); 21 Jul 2010 21:23:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82843 invoked by uid 99); 21 Jul 2010 21:23:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 21:23:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 21:23:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6LLFp3P019549
	for <common-issues@hadoop.apache.org>; Wed, 21 Jul 2010 21:15:51 GMT
Message-ID: <19114477.505371279746951096.JavaMail.jira@thor>
Date: Wed, 21 Jul 2010 17:15:51 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6864) Provide a JNI-based implementation
 of ShellBasedUnixGroupsNetgroupMapping (implementation of
 GroupMappingServiceProvider)
In-Reply-To: <17354207.429501279319091600.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890896#action_12890896 ] 

Erik Steffl commented on HADOOP-6864:
-------------------------------------

JNA is licenced under Lesser General Public License (LGPL v. 2.1) which is not allowed to be used in Apache projects, see http://www.apache.org/legal/resolved.html

> Provide a JNI-based implementation of ShellBasedUnixGroupsNetgroupMapping (implementation of GroupMappingServiceProvider)
> -------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6864
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6864
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Erik Steffl
>             Fix For: 0.22.0
>
>
> The netgroups implementation of GroupMappingServiceProvider (see ShellBasedUnixGroupsNetgroupMapping.java) does a fork of a unix command to get the netgroups of a user. Since the group resolution happens in the servers, this might be costly. This jira aims at providing a JNI-based implementation for GroupMappingServiceProvider.
> Note that this is similar to what https://issues.apache.org/jira/browse/HADOOP-6818 does for implementation of GroupMappingServiceProvider that  supports only unix groups.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9653-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 22:02:45 2010
Return-Path: <common-issues-return-9653-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11590 invoked from network); 21 Jul 2010 22:02:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 22:02:45 -0000
Received: (qmail 24756 invoked by uid 500); 21 Jul 2010 22:02:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24689 invoked by uid 500); 21 Jul 2010 22:02:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24681 invoked by uid 99); 21 Jul 2010 22:02:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 22:02:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 22:02:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6LLsnRg019981
	for <common-issues@hadoop.apache.org>; Wed, 21 Jul 2010 21:54:49 GMT
Message-ID: <13042840.506371279749289722.JavaMail.jira@thor>
Date: Wed, 21 Jul 2010 17:54:49 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6870) Add FileSystem#listLocatedStatus to
 list a directory's content together with each file's block locations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Add FileSystem#listLocatedStatus to list a directory's content together with each file's block locations
--------------------------------------------------------------------------------------------------------

                 Key: HADOOP-6870
                 URL: https://issues.apache.org/jira/browse/HADOOP-6870
             Project: Hadoop Common
          Issue Type: New Feature
          Components: fs
    Affects Versions: 0.22.0
            Reporter: Hairong Kuang
            Assignee: Hairong Kuang
             Fix For: 0.22.0


This jira implements the new FileSystem API as proposed in HDFS-202. The new API aims to eliminate individual "getFileBlockLocations" calls to NN for each file in the input directory of a job. Instead, a file's block locations are returned together with FileStatus when listing a directory, thus improving getSplits performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9654-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 22:05:15 2010
Return-Path: <common-issues-return-9654-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16006 invoked from network); 21 Jul 2010 22:05:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 22:05:15 -0000
Received: (qmail 28409 invoked by uid 500); 21 Jul 2010 22:05:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28318 invoked by uid 500); 21 Jul 2010 22:05:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28304 invoked by uid 99); 21 Jul 2010 22:05:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 22:05:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 22:05:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6LM4oiC020141
	for <common-issues@hadoop.apache.org>; Wed, 21 Jul 2010 22:04:50 GMT
Message-ID: <10098205.506541279749890622.JavaMail.jira@thor>
Date: Wed, 21 Jul 2010 18:04:50 -0400 (EDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6790) Instrumented (Herriot) build uses too
 wide mask to include aspect files.
In-Reply-To: <22445865.9971274913097186.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sharad Agarwal updated HADOOP-6790:
-----------------------------------

    Status: Open  (was: Patch Available)

Rerunning hudson.

> Instrumented (Herriot) build uses too wide mask to include aspect files.
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6790
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6790
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>            Priority: Minor
>         Attachments: HADOOP-6790.patch
>
>
> AspectJ compiler needs to use narrower mask to look for sources and aspect files. 
> Otherwise, {{injectfaults}} might fail if called after {{inject-system-faults}}.
> This is very minor issue and won't appear until we'll be having actually FI tests in the Common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9655-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 22:05:15 2010
Return-Path: <common-issues-return-9655-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16004 invoked from network); 21 Jul 2010 22:05:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 22:05:15 -0000
Received: (qmail 28432 invoked by uid 500); 21 Jul 2010 22:05:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28324 invoked by uid 500); 21 Jul 2010 22:05:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28301 invoked by uid 99); 21 Jul 2010 22:05:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 22:05:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 22:05:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6LM4oBZ020135
	for <common-issues@hadoop.apache.org>; Wed, 21 Jul 2010 22:04:50 GMT
Message-ID: <29906494.506521279749890112.JavaMail.jira@thor>
Date: Wed, 21 Jul 2010 18:04:50 -0400 (EDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6790) Instrumented (Herriot) build uses
 too wide mask to include aspect files.
In-Reply-To: <22445865.9971274913097186.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890919#action_12890919 ] 

Sharad Agarwal commented on HADOOP-6790:
----------------------------------------

Patch looks fine.

> Instrumented (Herriot) build uses too wide mask to include aspect files.
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6790
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6790
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>            Priority: Minor
>         Attachments: HADOOP-6790.patch
>
>
> AspectJ compiler needs to use narrower mask to look for sources and aspect files. 
> Otherwise, {{injectfaults}} might fail if called after {{inject-system-faults}}.
> This is very minor issue and won't appear until we'll be having actually FI tests in the Common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9656-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 22:09:16 2010
Return-Path: <common-issues-return-9656-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17058 invoked from network); 21 Jul 2010 22:09:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 22:09:16 -0000
Received: (qmail 33719 invoked by uid 500); 21 Jul 2010 22:09:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33450 invoked by uid 500); 21 Jul 2010 22:09:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33434 invoked by uid 99); 21 Jul 2010 22:09:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 22:09:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 22:09:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6LM8nXt020166
	for <common-issues@hadoop.apache.org>; Wed, 21 Jul 2010 22:08:50 GMT
Message-ID: <4673721.506611279750129934.JavaMail.jira@thor>
Date: Wed, 21 Jul 2010 18:08:49 -0400 (EDT)
From: "Arvind Prabhakar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6871) When the value of a configuration key
 is set to its unresolved form, it causes the IllegalStateException in
 Configuration.get() stating that substitution depth is too large.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

When the value of a configuration key is set to its unresolved form, it causes the IllegalStateException in Configuration.get() stating that substitution depth is too large.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-6871
                 URL: https://issues.apache.org/jira/browse/HADOOP-6871
             Project: Hadoop Common
          Issue Type: Bug
          Components: conf
            Reporter: Arvind Prabhakar


When a configuration value is set to its unresolved expression string, it leads to recursive substitution attempts in {{Configuration.substituteVars(String)}} method until the max substitution check kicks in and raises an IllegalStateException indicating that the substitution depth is too large. For example, the configuration key "{{foobar}}" with a value set to "{{$\{foobar\}}}" will cause this behavior. 

While this is not a usual use case, it can happen in build environments where a property value is not specified and yet being passed into the test mechanism leading to failures due to this limitation.


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9657-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 22:12:45 2010
Return-Path: <common-issues-return-9657-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17681 invoked from network); 21 Jul 2010 22:12:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 22:12:45 -0000
Received: (qmail 39995 invoked by uid 500); 21 Jul 2010 22:12:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39950 invoked by uid 500); 21 Jul 2010 22:12:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39942 invoked by uid 99); 21 Jul 2010 22:12:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 22:12:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 22:12:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6LM4oJH020147
	for <common-issues@hadoop.apache.org>; Wed, 21 Jul 2010 22:04:50 GMT
Message-ID: <32523106.506561279749890918.JavaMail.jira@thor>
Date: Wed, 21 Jul 2010 18:04:50 -0400 (EDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6790) Instrumented (Herriot) build uses too
 wide mask to include aspect files.
In-Reply-To: <22445865.9971274913097186.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sharad Agarwal updated HADOOP-6790:
-----------------------------------

    Status: Patch Available  (was: Open)

> Instrumented (Herriot) build uses too wide mask to include aspect files.
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6790
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6790
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>            Priority: Minor
>         Attachments: HADOOP-6790.patch
>
>
> AspectJ compiler needs to use narrower mask to look for sources and aspect files. 
> Otherwise, {{injectfaults}} might fail if called after {{inject-system-faults}}.
> This is very minor issue and won't appear until we'll be having actually FI tests in the Common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9658-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 22:13:15 2010
Return-Path: <common-issues-return-9658-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17739 invoked from network); 21 Jul 2010 22:13:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 22:13:15 -0000
Received: (qmail 40209 invoked by uid 500); 21 Jul 2010 22:13:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40140 invoked by uid 500); 21 Jul 2010 22:13:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40132 invoked by uid 99); 21 Jul 2010 22:13:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 22:13:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 22:13:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6LMComf020259
	for <common-issues@hadoop.apache.org>; Wed, 21 Jul 2010 22:12:50 GMT
Message-ID: <3631048.506851279750370442.JavaMail.jira@thor>
Date: Wed, 21 Jul 2010 18:12:50 -0400 (EDT)
From: "Arvind Prabhakar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6871) When the value of a configuration key
 is set to its unresolved form, it causes the IllegalStateException in
 Configuration.get() stating that substitution depth is too large.
In-Reply-To: <4673721.506611279750129934.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arvind Prabhakar updated HADOOP-6871:
-------------------------------------

    Attachment: HADOOP-6871.patch

> When the value of a configuration key is set to its unresolved form, it causes the IllegalStateException in Configuration.get() stating that substitution depth is too large.
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6871
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6871
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>            Reporter: Arvind Prabhakar
>         Attachments: HADOOP-6871.patch
>
>
> When a configuration value is set to its unresolved expression string, it leads to recursive substitution attempts in {{Configuration.substituteVars(String)}} method until the max substitution check kicks in and raises an IllegalStateException indicating that the substitution depth is too large. For example, the configuration key "{{foobar}}" with a value set to "{{$\{foobar\}}}" will cause this behavior. 
> While this is not a usual use case, it can happen in build environments where a property value is not specified and yet being passed into the test mechanism leading to failures due to this limitation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9659-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 22:17:43 2010
Return-Path: <common-issues-return-9659-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18474 invoked from network); 21 Jul 2010 22:17:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 22:17:43 -0000
Received: (qmail 43999 invoked by uid 500); 21 Jul 2010 22:17:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43940 invoked by uid 500); 21 Jul 2010 22:17:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43931 invoked by uid 99); 21 Jul 2010 22:17:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 22:17:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 22:17:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6LMHIiO020315
	for <common-issues@hadoop.apache.org>; Wed, 21 Jul 2010 22:17:18 GMT
Message-ID: <10714066.506961279750638878.JavaMail.jira@thor>
Date: Wed, 21 Jul 2010 18:17:18 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6870) Add FileSystem#listLocatedStatus to
 list a directory's content together with each file's block locations
In-Reply-To: <13042840.506371279749289722.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890928#action_12890928 ] 

Hairong Kuang commented on HADOOP-6870:
---------------------------------------

Note that listLocatedStatus lists only files. So the API name might be better to be listFiles().
{code}
Iterator<LocatedFileStatus> listFiles(Path path, boolean isRecursive);
{code}

> Add FileSystem#listLocatedStatus to list a directory's content together with each file's block locations
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6870
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6870
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>
> This jira implements the new FileSystem API as proposed in HDFS-202. The new API aims to eliminate individual "getFileBlockLocations" calls to NN for each file in the input directory of a job. Instead, a file's block locations are returned together with FileStatus when listing a directory, thus improving getSplits performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9660-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 22:24:45 2010
Return-Path: <common-issues-return-9660-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19876 invoked from network); 21 Jul 2010 22:24:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 22:24:45 -0000
Received: (qmail 52483 invoked by uid 500); 21 Jul 2010 22:24:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52449 invoked by uid 500); 21 Jul 2010 22:24:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52441 invoked by uid 99); 21 Jul 2010 22:24:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 22:24:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 22:24:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6LMGo4Y020286
	for <common-issues@hadoop.apache.org>; Wed, 21 Jul 2010 22:16:50 GMT
Message-ID: <12629359.506891279750610093.JavaMail.jira@thor>
Date: Wed, 21 Jul 2010 18:16:50 -0400 (EDT)
From: "Arvind Prabhakar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6871) When the value of a configuration key
 is set to its unresolved form, it causes the IllegalStateException in
 Configuration.get() stating that substitution depth is too large.
In-Reply-To: <4673721.506611279750129934.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arvind Prabhakar updated HADOOP-6871:
-------------------------------------

    Status: Patch Available  (was: Open)

This patch fixes this problem by maintaining a set of intermediate evaluated expressions to test if there is a cycle detected during the resolution loop. If a cycle is detected, the then intermediate form is returned instead of raising any exception.

Note that since my formatter setting is such that it substitutes all tabs with spaces, there appear to be a lot of changes - most of which are only white-space related. If there is a need to preserve the current tab/space format, I can regenerate the patch with this disabled.

PS - I am not able to assign the jira to myself for some reason. Admin(s) can you please add me to the list of assignees?

> When the value of a configuration key is set to its unresolved form, it causes the IllegalStateException in Configuration.get() stating that substitution depth is too large.
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6871
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6871
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>            Reporter: Arvind Prabhakar
>         Attachments: HADOOP-6871.patch
>
>
> When a configuration value is set to its unresolved expression string, it leads to recursive substitution attempts in {{Configuration.substituteVars(String)}} method until the max substitution check kicks in and raises an IllegalStateException indicating that the substitution depth is too large. For example, the configuration key "{{foobar}}" with a value set to "{{$\{foobar\}}}" will cause this behavior. 
> While this is not a usual use case, it can happen in build environments where a property value is not specified and yet being passed into the test mechanism leading to failures due to this limitation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9661-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 22:39:46 2010
Return-Path: <common-issues-return-9661-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25834 invoked from network); 21 Jul 2010 22:39:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 22:39:45 -0000
Received: (qmail 72965 invoked by uid 500); 21 Jul 2010 22:39:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72918 invoked by uid 500); 21 Jul 2010 22:39:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72910 invoked by uid 99); 21 Jul 2010 22:39:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 22:39:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 22:39:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6LMVn74020428
	for <common-issues@hadoop.apache.org>; Wed, 21 Jul 2010 22:31:50 GMT
Message-ID: <7021219.507201279751509778.JavaMail.jira@thor>
Date: Wed, 21 Jul 2010 18:31:49 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6870) Add FileSystem#listLocatedStatus to
 list a directory's content together with each file's block locations
In-Reply-To: <13042840.506371279749289722.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6870:
----------------------------------

    Attachment: listFiles.patch

An initial patch for review.

> Add FileSystem#listLocatedStatus to list a directory's content together with each file's block locations
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6870
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6870
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listFiles.patch
>
>
> This jira implements the new FileSystem API as proposed in HDFS-202. The new API aims to eliminate individual "getFileBlockLocations" calls to NN for each file in the input directory of a job. Instead, a file's block locations are returned together with FileStatus when listing a directory, thus improving getSplits performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9662-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 22:41:44 2010
Return-Path: <common-issues-return-9662-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26138 invoked from network); 21 Jul 2010 22:41:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 22:41:43 -0000
Received: (qmail 73788 invoked by uid 500); 21 Jul 2010 22:41:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73752 invoked by uid 500); 21 Jul 2010 22:41:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73744 invoked by uid 99); 21 Jul 2010 22:41:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 22:41:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 22:41:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6LMXnGf020445
	for <common-issues@hadoop.apache.org>; Wed, 21 Jul 2010 22:33:49 GMT
Message-ID: <2931482.507231279751629413.JavaMail.jira@thor>
Date: Wed, 21 Jul 2010 18:33:49 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6872) ChecksumFs#listStatus should filter
 out .crc files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

ChecksumFs#listStatus should filter out .crc files
--------------------------------------------------

                 Key: HADOOP-6872
                 URL: https://issues.apache.org/jira/browse/HADOOP-6872
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
    Affects Versions: 0.21.0
            Reporter: Hairong Kuang
             Fix For: 0.22.0


Currently ChecksumFs#listStatus listing not only regular files but also .crc files. Crc files should be filtered out.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9663-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 22:42:13 2010
Return-Path: <common-issues-return-9663-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26252 invoked from network); 21 Jul 2010 22:42:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 22:42:13 -0000
Received: (qmail 74241 invoked by uid 500); 21 Jul 2010 22:42:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74112 invoked by uid 500); 21 Jul 2010 22:42:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74104 invoked by uid 99); 21 Jul 2010 22:42:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 22:42:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 22:42:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6LMfnqD020500
	for <common-issues@hadoop.apache.org>; Wed, 21 Jul 2010 22:41:49 GMT
Message-ID: <22584969.507371279752109437.JavaMail.jira@thor>
Date: Wed, 21 Jul 2010 18:41:49 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6790) Instrumented (Herriot) build uses
 too wide mask to include aspect files.
In-Reply-To: <22445865.9971274913097186.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890940#action_12890940 ] 

Hadoop QA commented on HADOOP-6790:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12445592/HADOOP-6790.patch
  against trunk revision 965725.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/626/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/626/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/626/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/626/console

This message is automatically generated.

> Instrumented (Herriot) build uses too wide mask to include aspect files.
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6790
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6790
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>            Priority: Minor
>         Attachments: HADOOP-6790.patch
>
>
> AspectJ compiler needs to use narrower mask to look for sources and aspect files. 
> Otherwise, {{injectfaults}} might fail if called after {{inject-system-faults}}.
> This is very minor issue and won't appear until we'll be having actually FI tests in the Common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9664-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 21 22:57:45 2010
Return-Path: <common-issues-return-9664-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29368 invoked from network); 21 Jul 2010 22:57:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jul 2010 22:57:45 -0000
Received: (qmail 88164 invoked by uid 500); 21 Jul 2010 22:57:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88108 invoked by uid 500); 21 Jul 2010 22:57:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88100 invoked by uid 99); 21 Jul 2010 22:57:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 22:57:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2010 22:57:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6LMnonL020529
	for <common-issues@hadoop.apache.org>; Wed, 21 Jul 2010 22:49:50 GMT
Message-ID: <28518193.507411279752590034.JavaMail.jira@thor>
Date: Wed, 21 Jul 2010 18:49:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6871) When the value of a configuration
 key is set to its unresolved form, it causes the IllegalStateException in
 Configuration.get() stating that substitution depth is too large.
In-Reply-To: <4673721.506611279750129934.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890941#action_12890941 ] 

Hadoop QA commented on HADOOP-6871:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12450102/HADOOP-6871.patch
  against trunk revision 965725.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/89/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/89/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/89/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/89/console

This message is automatically generated.

> When the value of a configuration key is set to its unresolved form, it causes the IllegalStateException in Configuration.get() stating that substitution depth is too large.
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6871
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6871
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>            Reporter: Arvind Prabhakar
>         Attachments: HADOOP-6871.patch
>
>
> When a configuration value is set to its unresolved expression string, it leads to recursive substitution attempts in {{Configuration.substituteVars(String)}} method until the max substitution check kicks in and raises an IllegalStateException indicating that the substitution depth is too large. For example, the configuration key "{{foobar}}" with a value set to "{{$\{foobar\}}}" will cause this behavior. 
> While this is not a usual use case, it can happen in build environments where a property value is not specified and yet being passed into the test mechanism leading to failures due to this limitation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9665-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 22 00:15:15 2010
Return-Path: <common-issues-return-9665-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54649 invoked from network); 22 Jul 2010 00:15:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jul 2010 00:15:15 -0000
Received: (qmail 48658 invoked by uid 500); 22 Jul 2010 00:15:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48558 invoked by uid 500); 22 Jul 2010 00:15:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48550 invoked by uid 99); 22 Jul 2010 00:15:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 00:15:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 00:15:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6M0EoTm021088
	for <common-issues@hadoop.apache.org>; Thu, 22 Jul 2010 00:14:50 GMT
Message-ID: <2384126.508441279757690907.JavaMail.jira@thor>
Date: Wed, 21 Jul 2010 20:14:50 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6873) using delegation token over hftp for
 long running clients (part of hdfs 1296)
In-Reply-To: <3672204.508361279757392441.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6873:
-----------------------------------

    Status: Patch Available  (was: Open)

> using delegation token over hftp for long running clients (part of hdfs 1296)
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-6873
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6873
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6873.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9667-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 22 00:17:47 2010
Return-Path: <common-issues-return-9667-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55142 invoked from network); 22 Jul 2010 00:17:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jul 2010 00:17:47 -0000
Received: (qmail 51096 invoked by uid 500); 22 Jul 2010 00:17:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51037 invoked by uid 500); 22 Jul 2010 00:17:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51022 invoked by uid 99); 22 Jul 2010 00:17:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 00:17:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 00:17:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6M09ppP021046
	for <common-issues@hadoop.apache.org>; Thu, 22 Jul 2010 00:09:51 GMT
Message-ID: <12036773.508341279757391889.JavaMail.jira@thor>
Date: Wed, 21 Jul 2010 20:09:51 -0400 (EDT)
From: "Arvind Prabhakar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6871) When the value of a configuration key
 is set to its unresolved form, it causes the IllegalStateException in
 Configuration.get() stating that substitution depth is too large.
In-Reply-To: <4673721.506611279750129934.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arvind Prabhakar updated HADOOP-6871:
-------------------------------------

    Attachment: HADOOP-6871-1.patch

Removed the whitespace formatting as that may have caused the javadoc warning from the previous automated test build.

> When the value of a configuration key is set to its unresolved form, it causes the IllegalStateException in Configuration.get() stating that substitution depth is too large.
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6871
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6871
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>            Reporter: Arvind Prabhakar
>         Attachments: HADOOP-6871-1.patch, HADOOP-6871.patch
>
>
> When a configuration value is set to its unresolved expression string, it leads to recursive substitution attempts in {{Configuration.substituteVars(String)}} method until the max substitution check kicks in and raises an IllegalStateException indicating that the substitution depth is too large. For example, the configuration key "{{foobar}}" with a value set to "{{$\{foobar\}}}" will cause this behavior. 
> While this is not a usual use case, it can happen in build environments where a property value is not specified and yet being passed into the test mechanism leading to failures due to this limitation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9666-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 22 00:17:47 2010
Return-Path: <common-issues-return-9666-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55141 invoked from network); 22 Jul 2010 00:17:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jul 2010 00:17:47 -0000
Received: (qmail 51090 invoked by uid 500); 22 Jul 2010 00:17:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51018 invoked by uid 500); 22 Jul 2010 00:17:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51009 invoked by uid 99); 22 Jul 2010 00:17:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 00:17:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 00:17:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6M09qir021052
	for <common-issues@hadoop.apache.org>; Thu, 22 Jul 2010 00:09:52 GMT
Message-ID: <3672204.508361279757392441.JavaMail.jira@thor>
Date: Wed, 21 Jul 2010 20:09:52 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6873) using delegation token over hftp for
 long running clients (part of hdfs 1296)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

using delegation token over hftp for long running clients (part of hdfs 1296)
-----------------------------------------------------------------------------

                 Key: HADOOP-6873
                 URL: https://issues.apache.org/jira/browse/HADOOP-6873
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Boris Shkolnik
            Assignee: Boris Shkolnik




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9668-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 22 00:22:45 2010
Return-Path: <common-issues-return-9668-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61136 invoked from network); 22 Jul 2010 00:22:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jul 2010 00:22:45 -0000
Received: (qmail 57250 invoked by uid 500); 22 Jul 2010 00:22:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57199 invoked by uid 500); 22 Jul 2010 00:22:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57191 invoked by uid 99); 22 Jul 2010 00:22:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 00:22:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 00:22:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6M0EoGD021082
	for <common-issues@hadoop.apache.org>; Thu, 22 Jul 2010 00:14:50 GMT
Message-ID: <487041.508421279757690407.JavaMail.jira@thor>
Date: Wed, 21 Jul 2010 20:14:50 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6873) using delegation token over hftp for
 long running clients (part of hdfs 1296)
In-Reply-To: <3672204.508361279757392441.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6873:
-----------------------------------

    Attachment: HADOOP-6873.patch

> using delegation token over hftp for long running clients (part of hdfs 1296)
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-6873
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6873
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6873.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9669-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 22 00:33:46 2010
Return-Path: <common-issues-return-9669-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66729 invoked from network); 22 Jul 2010 00:33:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jul 2010 00:33:45 -0000
Received: (qmail 64615 invoked by uid 500); 22 Jul 2010 00:33:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64547 invoked by uid 500); 22 Jul 2010 00:33:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64539 invoked by uid 99); 22 Jul 2010 00:33:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 00:33:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 00:33:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6M0PngV021285
	for <common-issues@hadoop.apache.org>; Thu, 22 Jul 2010 00:25:49 GMT
Message-ID: <2671444.508881279758349792.JavaMail.jira@thor>
Date: Wed, 21 Jul 2010 20:25:49 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6873) using delegation token over hftp
 for long running clients (part of hdfs 1296)
In-Reply-To: <3672204.508361279757392441.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890969#action_12890969 ] 

Hadoop QA commented on HADOOP-6873:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12450116/HADOOP-6873.patch
  against trunk revision 965725.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 patch.  The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/627/console

This message is automatically generated.

> using delegation token over hftp for long running clients (part of hdfs 1296)
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-6873
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6873
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6873.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9670-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 22 01:34:45 2010
Return-Path: <common-issues-return-9670-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80808 invoked from network); 22 Jul 2010 01:34:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jul 2010 01:34:44 -0000
Received: (qmail 602 invoked by uid 500); 22 Jul 2010 01:34:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 549 invoked by uid 500); 22 Jul 2010 01:34:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 541 invoked by uid 99); 22 Jul 2010 01:34:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 01:34:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 01:34:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6M1QnH4021745
	for <common-issues@hadoop.apache.org>; Thu, 22 Jul 2010 01:26:49 GMT
Message-ID: <10881263.509571279762009663.JavaMail.jira@thor>
Date: Wed, 21 Jul 2010 21:26:49 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6790) Instrumented (Herriot) build uses
 too wide mask to include aspect files.
In-Reply-To: <22445865.9971274913097186.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890983#action_12890983 ] 

Konstantin Boudnik commented on HADOOP-6790:
--------------------------------------------

Same old deprecation warnings from security stuff: unrelated. I am going to commit this.

> Instrumented (Herriot) build uses too wide mask to include aspect files.
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6790
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6790
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>            Priority: Minor
>         Attachments: HADOOP-6790.patch
>
>
> AspectJ compiler needs to use narrower mask to look for sources and aspect files. 
> Otherwise, {{injectfaults}} might fail if called after {{inject-system-faults}}.
> This is very minor issue and won't appear until we'll be having actually FI tests in the Common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9671-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 22 04:44:48 2010
Return-Path: <common-issues-return-9671-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27746 invoked from network); 22 Jul 2010 04:44:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jul 2010 04:44:47 -0000
Received: (qmail 12323 invoked by uid 500); 22 Jul 2010 04:44:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12192 invoked by uid 500); 22 Jul 2010 04:44:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12172 invoked by uid 99); 22 Jul 2010 04:44:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 04:44:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 04:44:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6M4anhb023455
	for <common-issues@hadoop.apache.org>; Thu, 22 Jul 2010 04:36:49 GMT
Message-ID: <768991.511221279773409593.JavaMail.jira@thor>
Date: Thu, 22 Jul 2010 00:36:49 -0400 (EDT)
From: "Franklin Ta (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6874) JvmMetrics can't distinguish between
 jvms with same processName
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

JvmMetrics can't distinguish between jvms with same processName
---------------------------------------------------------------

                 Key: HADOOP-6874
                 URL: https://issues.apache.org/jira/browse/HADOOP-6874
             Project: Hadoop Common
          Issue Type: Bug
          Components: metrics
    Affects Versions: 0.20.2
            Reporter: Franklin Ta
            Priority: Minor


JvmMetrics has three tags: hostName, processName, and sessionId. For processes such tasktracker/jobtracker/namenode/datanode which there is only one of on each host, these tags are fine. But for process names such as "MAP" and "REDUCE", since there might be multiple jvms running map/reduce tasks, we might end up with multiple set of metrics which all have the same tags, and no way to find out which jvm they actually correspond to. (In addition, since there is jvm reuse, those process names might not correspond to the actual task being ran)

A quick fix is to change this line in Child.java
JvmMetrics.init(task.getPhase().toString(), job.getSessionId());
to this
JvmMetrics.init(jvmId.toString(), job.getSessionId());
so that we are using the jvm id for the process name instead.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9672-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 22 05:46:18 2010
Return-Path: <common-issues-return-9672-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41156 invoked from network); 22 Jul 2010 05:46:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jul 2010 05:46:18 -0000
Received: (qmail 34408 invoked by uid 500); 22 Jul 2010 05:46:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34279 invoked by uid 500); 22 Jul 2010 05:46:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34271 invoked by uid 99); 22 Jul 2010 05:46:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 05:46:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 05:46:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6M5jnNf023908
	for <common-issues@hadoop.apache.org>; Thu, 22 Jul 2010 05:45:50 GMT
Message-ID: <14926355.511821279777549921.JavaMail.jira@thor>
Date: Thu, 22 Jul 2010 01:45:49 -0400 (EDT)
From: "Franklin Ta (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6874) JvmMetrics can't distinguish between
 jvms with same processName
In-Reply-To: <768991.511221279773409593.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Franklin Ta updated HADOOP-6874:
--------------------------------

    Description: 
JvmMetrics has three tags: hostName, processName, and sessionId. For processes such as tasktracker/jobtracker/namenode/datanode which there is only one of on each host, these tags are fine. But for process names such as "MAP" and "REDUCE", since there might be multiple jvms running map/reduce tasks, we might end up with multiple set of metrics which all have the same tags, and no way to find out which jvm they actually correspond to. (In addition, since there is jvm reuse, those process names might not correspond to the actual task being ran)

A quick fix is to change this line in Child.java
JvmMetrics.init(task.getPhase().toString(), job.getSessionId());
to this
JvmMetrics.init(jvmId.toString(), job.getSessionId());
so that we are using the jvm id for the process name instead.

  was:
JvmMetrics has three tags: hostName, processName, and sessionId. For processes such tasktracker/jobtracker/namenode/datanode which there is only one of on each host, these tags are fine. But for process names such as "MAP" and "REDUCE", since there might be multiple jvms running map/reduce tasks, we might end up with multiple set of metrics which all have the same tags, and no way to find out which jvm they actually correspond to. (In addition, since there is jvm reuse, those process names might not correspond to the actual task being ran)

A quick fix is to change this line in Child.java
JvmMetrics.init(task.getPhase().toString(), job.getSessionId());
to this
JvmMetrics.init(jvmId.toString(), job.getSessionId());
so that we are using the jvm id for the process name instead.


> JvmMetrics can't distinguish between jvms with same processName
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6874
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6874
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 0.20.2
>            Reporter: Franklin Ta
>            Priority: Minor
>
> JvmMetrics has three tags: hostName, processName, and sessionId. For processes such as tasktracker/jobtracker/namenode/datanode which there is only one of on each host, these tags are fine. But for process names such as "MAP" and "REDUCE", since there might be multiple jvms running map/reduce tasks, we might end up with multiple set of metrics which all have the same tags, and no way to find out which jvm they actually correspond to. (In addition, since there is jvm reuse, those process names might not correspond to the actual task being ran)
> A quick fix is to change this line in Child.java
> JvmMetrics.init(task.getPhase().toString(), job.getSessionId());
> to this
> JvmMetrics.init(jvmId.toString(), job.getSessionId());
> so that we are using the jvm id for the process name instead.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9673-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 22 05:58:49 2010
Return-Path: <common-issues-return-9673-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42460 invoked from network); 22 Jul 2010 05:58:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jul 2010 05:58:49 -0000
Received: (qmail 40068 invoked by uid 500); 22 Jul 2010 05:58:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39391 invoked by uid 500); 22 Jul 2010 05:58:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39383 invoked by uid 99); 22 Jul 2010 05:58:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 05:58:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 05:58:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6M5opgu023942
	for <common-issues@hadoop.apache.org>; Thu, 22 Jul 2010 05:50:51 GMT
Message-ID: <23544132.511891279777851012.JavaMail.jira@thor>
Date: Thu, 22 Jul 2010 01:50:51 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6790) Instrumented (Herriot) build uses too
 wide mask to include aspect files.
In-Reply-To: <22445865.9971274913097186.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6790:
---------------------------------------

           Status: Resolved  (was: Patch Available)
     Hadoop Flags: [Reviewed]
    Fix Version/s: 0.21.0
       Resolution: Fixed

I have committed this.

> Instrumented (Herriot) build uses too wide mask to include aspect files.
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6790
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6790
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6790.patch
>
>
> AspectJ compiler needs to use narrower mask to look for sources and aspect files. 
> Otherwise, {{injectfaults}} might fail if called after {{inject-system-faults}}.
> This is very minor issue and won't appear until we'll be having actually FI tests in the Common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9674-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 22 06:13:48 2010
Return-Path: <common-issues-return-9674-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50753 invoked from network); 22 Jul 2010 06:13:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jul 2010 06:13:48 -0000
Received: (qmail 49415 invoked by uid 500); 22 Jul 2010 06:13:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49279 invoked by uid 500); 22 Jul 2010 06:13:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49265 invoked by uid 99); 22 Jul 2010 06:13:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 06:13:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 06:13:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6M65oFj024092
	for <common-issues@hadoop.apache.org>; Thu, 22 Jul 2010 06:05:51 GMT
Message-ID: <26595772.512001279778750944.JavaMail.jira@thor>
Date: Thu, 22 Jul 2010 02:05:50 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6790) Instrumented (Herriot) build uses
 too wide mask to include aspect files.
In-Reply-To: <22445865.9971274913097186.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891016#action_12891016 ] 

Hudson commented on HADOOP-6790:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #333 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/333/])
    HADOOP-6790. Instrumented (Herriot) build uses too wide mask to include aspect files. Contributed by Konstantin Boudnik.


> Instrumented (Herriot) build uses too wide mask to include aspect files.
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6790
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6790
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6790.patch
>
>
> AspectJ compiler needs to use narrower mask to look for sources and aspect files. 
> Otherwise, {{injectfaults}} might fail if called after {{inject-system-faults}}.
> This is very minor issue and won't appear until we'll be having actually FI tests in the Common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9675-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 22 08:06:51 2010
Return-Path: <common-issues-return-9675-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82619 invoked from network); 22 Jul 2010 08:06:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jul 2010 08:06:50 -0000
Received: (qmail 25744 invoked by uid 500); 22 Jul 2010 08:06:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25311 invoked by uid 500); 22 Jul 2010 08:06:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25282 invoked by uid 99); 22 Jul 2010 08:06:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 08:06:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 08:06:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6M7wobM025029
	for <common-issues@hadoop.apache.org>; Thu, 22 Jul 2010 07:58:51 GMT
Message-ID: <28139830.513221279785530734.JavaMail.jira@thor>
Date: Thu, 22 Jul 2010 03:58:50 -0400 (EDT)
From: "Balaji Rajagopalan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6869) [Herriot] Implement a functionality
 for creating either file or folder in task attempt  folder while job is
 running.
In-Reply-To: <1398369.499011279732010273.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891050#action_12891050 ] 

Balaji Rajagopalan commented on HADOOP-6869:
--------------------------------------------

+1

> [Herriot] Implement a functionality for creating either file or folder in task attempt  folder while job is running.
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6869
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6869
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6869-ydist-security.patch, HADOOP-6869.patch
>
>
> Functionality for creating either files or folders in task attempt folder while job is running. The functionality covers the following methods.
> 1. public void DaemonProtocol.createFile(String path, String fileName, boolean local) throws IOException; 
> It uses to create a file with full permissions.
> 2.   public void DaemonProtocol.createFile(String path, String fileName, FsPermission permission, boolean local) throws IOException; 
> It uses to create a file with given permissions.
> 3.   public void DaemonProtocol.createFolder(String path, String folderName, boolean local) throws IOException;
> It uses to create a file with full permissions.
> 4.   public void DaemonProtocol.createFolder(String path, String folderName, FsPermission permission, boolean local) throws IOException;
> It uses to create a folder with given permissions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9676-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 22 11:16:23 2010
Return-Path: <common-issues-return-9676-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35041 invoked from network); 22 Jul 2010 11:16:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jul 2010 11:16:23 -0000
Received: (qmail 37490 invoked by uid 500); 22 Jul 2010 11:16:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36320 invoked by uid 500); 22 Jul 2010 11:16:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36310 invoked by uid 99); 22 Jul 2010 11:16:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 11:16:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 11:16:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6MBFrwP026844
	for <common-issues@hadoop.apache.org>; Thu, 22 Jul 2010 11:15:53 GMT
Message-ID: <3135723.515421279797353098.JavaMail.jira@thor>
Date: Thu, 22 Jul 2010 07:15:53 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6790) Instrumented (Herriot) build uses
 too wide mask to include aspect files.
In-Reply-To: <22445865.9971274913097186.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891095#action_12891095 ] 

Hudson commented on HADOOP-6790:
--------------------------------

Integrated in Hadoop-Common-trunk #400 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/400/])
    HADOOP-6790. Instrumented (Herriot) build uses too wide mask to include aspect files. Contributed by Konstantin Boudnik.


> Instrumented (Herriot) build uses too wide mask to include aspect files.
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6790
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6790
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6790.patch
>
>
> AspectJ compiler needs to use narrower mask to look for sources and aspect files. 
> Otherwise, {{injectfaults}} might fail if called after {{inject-system-faults}}.
> This is very minor issue and won't appear until we'll be having actually FI tests in the Common.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9677-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 22 16:31:17 2010
Return-Path: <common-issues-return-9677-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51413 invoked from network); 22 Jul 2010 16:31:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jul 2010 16:31:17 -0000
Received: (qmail 75782 invoked by uid 500); 22 Jul 2010 16:31:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75723 invoked by uid 500); 22 Jul 2010 16:31:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75714 invoked by uid 99); 22 Jul 2010 16:31:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 16:31:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 16:31:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6MGUp9C029992
	for <common-issues@hadoop.apache.org>; Thu, 22 Jul 2010 16:30:51 GMT
Message-ID: <28631338.521021279816251209.JavaMail.jira@thor>
Date: Thu, 22 Jul 2010 12:30:51 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Moved: (HADOOP-6875) [Herriot] Couple of issues occurred
 while running the tests in a cluster with security enabled.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinay Kumar Thota moved MAPREDUCE-1913 to HADOOP-6875:
------------------------------------------------------

        Project: Hadoop Common  (was: Hadoop Map/Reduce)
            Key: HADOOP-6875  (was: MAPREDUCE-1913)
    Component/s: test
                     (was: test)

> [Herriot] Couple of issues occurred while running the tests in a cluster with security enabled.
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6875
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6875
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 1913-ydist-security.patch, MAPREDUCE-1913.patch
>
>
> 1. New configuration directory is not cleaning up after resetting to default configuration directory in a pushconfig functionality. Because of this reason, it's giving  permission denied problem for a folder, if  other user tried running the tests in the same cluster with pushconfig functionality. I could see this issue while running the tests on a cluster with security enabled and different user.
> I have added the functionality for above issue and attaching the patch
> 2.  Throwing IOException and it says token is expired while running  the tests. I could see this issue in a secure cluster.
> This issue has been resolved by setting the following attribute in the configuration. 
> mapreduce.job.complete.cancel.delegation.tokens=false
>  
> adding/updating this attribute in the push configuration functionality while creating the new configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9678-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 22 16:42:18 2010
Return-Path: <common-issues-return-9678-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55170 invoked from network); 22 Jul 2010 16:42:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jul 2010 16:42:17 -0000
Received: (qmail 92005 invoked by uid 500); 22 Jul 2010 16:42:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91919 invoked by uid 500); 22 Jul 2010 16:42:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91911 invoked by uid 99); 22 Jul 2010 16:42:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 16:42:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 16:42:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6MGfqTw000172
	for <common-issues@hadoop.apache.org>; Thu, 22 Jul 2010 16:41:52 GMT
Message-ID: <29993815.521491279816912157.JavaMail.jira@thor>
Date: Thu, 22 Jul 2010 12:41:52 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6875) [Herriot] Couple of issues occurred
 while running the tests in a cluster with security enabled.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891235#action_12891235 ] 

Vinay Kumar Thota commented on HADOOP-6875:
-------------------------------------------

bq.the config key is already defined as in mapreduce project(src/java/org/apache/hadoop/mapreduce/MRJobConfig.java)

The patch belongs to common project and it doesn't have defined config key.

> [Herriot] Couple of issues occurred while running the tests in a cluster with security enabled.
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6875
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6875
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 1913-ydist-security.patch, MAPREDUCE-1913.patch
>
>
> 1. New configuration directory is not cleaning up after resetting to default configuration directory in a pushconfig functionality. Because of this reason, it's giving  permission denied problem for a folder, if  other user tried running the tests in the same cluster with pushconfig functionality. I could see this issue while running the tests on a cluster with security enabled and different user.
> I have added the functionality for above issue and attaching the patch
> 2.  Throwing IOException and it says token is expired while running  the tests. I could see this issue in a secure cluster.
> This issue has been resolved by setting the following attribute in the configuration. 
> mapreduce.job.complete.cancel.delegation.tokens=false
>  
> adding/updating this attribute in the push configuration functionality while creating the new configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9679-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 22 16:45:46 2010
Return-Path: <common-issues-return-9679-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56705 invoked from network); 22 Jul 2010 16:45:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jul 2010 16:45:46 -0000
Received: (qmail 99701 invoked by uid 500); 22 Jul 2010 16:45:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99644 invoked by uid 500); 22 Jul 2010 16:45:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99633 invoked by uid 99); 22 Jul 2010 16:45:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 16:45:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 16:45:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6MGbpVa000088
	for <common-issues@hadoop.apache.org>; Thu, 22 Jul 2010 16:37:51 GMT
Message-ID: <23356983.521251279816671269.JavaMail.jira@thor>
Date: Thu, 22 Jul 2010 12:37:51 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6875) [Herriot] Couple of issues occurred
 while running the tests in a cluster with security enabled.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891233#action_12891233 ] 

Vinay Kumar Thota commented on HADOOP-6875:
-------------------------------------------

bq. the config key is already defined as in mapreduce project(src/java/org/apache/hadoop/mapreduce/MRJobConfig.java). But the patch belongs to common project and it doesn't have defined config key.

> [Herriot] Couple of issues occurred while running the tests in a cluster with security enabled.
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6875
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6875
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 1913-ydist-security.patch, MAPREDUCE-1913.patch
>
>
> 1. New configuration directory is not cleaning up after resetting to default configuration directory in a pushconfig functionality. Because of this reason, it's giving  permission denied problem for a folder, if  other user tried running the tests in the same cluster with pushconfig functionality. I could see this issue while running the tests on a cluster with security enabled and different user.
> I have added the functionality for above issue and attaching the patch
> 2.  Throwing IOException and it says token is expired while running  the tests. I could see this issue in a secure cluster.
> This issue has been resolved by setting the following attribute in the configuration. 
> mapreduce.job.complete.cancel.delegation.tokens=false
>  
> adding/updating this attribute in the push configuration functionality while creating the new configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9680-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 22 17:05:20 2010
Return-Path: <common-issues-return-9680-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65975 invoked from network); 22 Jul 2010 17:05:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jul 2010 17:05:18 -0000
Received: (qmail 33329 invoked by uid 500); 22 Jul 2010 17:05:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32395 invoked by uid 500); 22 Jul 2010 17:05:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31552 invoked by uid 99); 22 Jul 2010 17:05:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 17:05:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 17:04:50 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6MGux5i000403
	for <common-issues@hadoop.apache.org>; Thu, 22 Jul 2010 16:56:59 GMT
Message-ID: <20991216.521991279817819437.JavaMail.jira@thor>
Date: Thu, 22 Jul 2010 12:56:59 -0400 (EDT)
From: "Nicholas Carlini (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicholas Carlini updated HADOOP-6837:
-------------------------------------

    Attachment: HADOOP-6837-lzma-1-20100722.patch

Attached a patch merging the java and c code together. Fixed two bugs in the C code, and removed some unneeded methods/arguments. Ran TestCodec on both of them for several hours each, and didn't find any bugs. Tested (again with TestCodec) all possible dictionary sizes from 1<<12 to 1<<23. Also added more (much needed) documentation.

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-1-20100722.patch, HADOOP-6837-lzma-c-20100719.patch, HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9681-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 22 17:21:45 2010
Return-Path: <common-issues-return-9681-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72845 invoked from network); 22 Jul 2010 17:21:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jul 2010 17:21:45 -0000
Received: (qmail 63198 invoked by uid 500); 22 Jul 2010 17:21:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63163 invoked by uid 500); 22 Jul 2010 17:21:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63155 invoked by uid 99); 22 Jul 2010 17:21:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 17:21:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 17:21:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6MHDpEp000580
	for <common-issues@hadoop.apache.org>; Thu, 22 Jul 2010 17:13:51 GMT
Message-ID: <28872292.522201279818831553.JavaMail.jira@thor>
Date: Thu, 22 Jul 2010 13:13:51 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6875) [Herriot] Couple of issues occurred
 while running the tests in a cluster with security enabled.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinay Kumar Thota updated HADOOP-6875:
--------------------------------------

    Attachment: HADOOP-6875.patch

> [Herriot] Couple of issues occurred while running the tests in a cluster with security enabled.
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6875
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6875
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 1913-ydist-security.patch, HADOOP-6875.patch, MAPREDUCE-1913.patch
>
>
> 1. New configuration directory is not cleaning up after resetting to default configuration directory in a pushconfig functionality. Because of this reason, it's giving  permission denied problem for a folder, if  other user tried running the tests in the same cluster with pushconfig functionality. I could see this issue while running the tests on a cluster with security enabled and different user.
> I have added the functionality for above issue and attaching the patch
> 2.  Throwing IOException and it says token is expired while running  the tests. I could see this issue in a secure cluster.
> This issue has been resolved by setting the following attribute in the configuration. 
> mapreduce.job.complete.cancel.delegation.tokens=false
>  
> adding/updating this attribute in the push configuration functionality while creating the new configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9682-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 22 21:12:16 2010
Return-Path: <common-issues-return-9682-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68194 invoked from network); 22 Jul 2010 21:12:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jul 2010 21:12:16 -0000
Received: (qmail 26205 invoked by uid 500); 22 Jul 2010 21:12:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26068 invoked by uid 500); 22 Jul 2010 21:12:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26059 invoked by uid 99); 22 Jul 2010 21:12:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 21:12:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 21:12:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6MLBprk003192
	for <common-issues@hadoop.apache.org>; Thu, 22 Jul 2010 21:11:51 GMT
Message-ID: <18145915.526581279833111713.JavaMail.jira@thor>
Date: Thu, 22 Jul 2010 17:11:51 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6875) [Herriot] Couple of issues occurred
 while running the tests in a cluster with security enabled.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891338#action_12891338 ] 

Konstantin Boudnik commented on HADOOP-6875:
--------------------------------------------

bq. The patch belongs to common project and it doesn't have defined config key.
This is an indication why this patch shoudn't go to the Common project then. If a base project has a knowledge about an upstream module - this is plain wrong!
I missed that during the first review - I think the name of the patch file confused me.

So, it is big no-no. -1

> [Herriot] Couple of issues occurred while running the tests in a cluster with security enabled.
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6875
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6875
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 1913-ydist-security.patch, HADOOP-6875.patch, MAPREDUCE-1913.patch
>
>
> 1. New configuration directory is not cleaning up after resetting to default configuration directory in a pushconfig functionality. Because of this reason, it's giving  permission denied problem for a folder, if  other user tried running the tests in the same cluster with pushconfig functionality. I could see this issue while running the tests on a cluster with security enabled and different user.
> I have added the functionality for above issue and attaching the patch
> 2.  Throwing IOException and it says token is expired while running  the tests. I could see this issue in a secure cluster.
> This issue has been resolved by setting the following attribute in the configuration. 
> mapreduce.job.complete.cancel.delegation.tokens=false
>  
> adding/updating this attribute in the push configuration functionality while creating the new configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9683-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 22 21:14:13 2010
Return-Path: <common-issues-return-9683-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70257 invoked from network); 22 Jul 2010 21:14:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jul 2010 21:14:12 -0000
Received: (qmail 29595 invoked by uid 500); 22 Jul 2010 21:14:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29479 invoked by uid 500); 22 Jul 2010 21:14:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29471 invoked by uid 99); 22 Jul 2010 21:14:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 21:14:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 21:14:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6MLDo3F003211
	for <common-issues@hadoop.apache.org>; Thu, 22 Jul 2010 21:13:51 GMT
Message-ID: <16870534.526631279833230701.JavaMail.jira@thor>
Date: Thu, 22 Jul 2010 17:13:50 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6875) [Herriot] Couple of issues occurred
 while running the tests in a cluster with security enabled.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891339#action_12891339 ] 

Konstantin Boudnik commented on HADOOP-6875:
--------------------------------------------

Basically, if you need to have this kind of properties for MR - it should go into concrete implementation of a cluster. I believe this is MRCLuster in this case.

> [Herriot] Couple of issues occurred while running the tests in a cluster with security enabled.
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6875
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6875
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 1913-ydist-security.patch, HADOOP-6875.patch, MAPREDUCE-1913.patch
>
>
> 1. New configuration directory is not cleaning up after resetting to default configuration directory in a pushconfig functionality. Because of this reason, it's giving  permission denied problem for a folder, if  other user tried running the tests in the same cluster with pushconfig functionality. I could see this issue while running the tests on a cluster with security enabled and different user.
> I have added the functionality for above issue and attaching the patch
> 2.  Throwing IOException and it says token is expired while running  the tests. I could see this issue in a secure cluster.
> This issue has been resolved by setting the following attribute in the configuration. 
> mapreduce.job.complete.cancel.delegation.tokens=false
>  
> adding/updating this attribute in the push configuration functionality while creating the new configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9684-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 22 21:52:15 2010
Return-Path: <common-issues-return-9684-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15981 invoked from network); 22 Jul 2010 21:52:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jul 2010 21:52:14 -0000
Received: (qmail 97951 invoked by uid 500); 22 Jul 2010 21:52:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97747 invoked by uid 500); 22 Jul 2010 21:52:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97732 invoked by uid 99); 22 Jul 2010 21:52:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 21:52:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 21:52:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6MLprG5003937
	for <common-issues@hadoop.apache.org>; Thu, 22 Jul 2010 21:51:53 GMT
Message-ID: <24368989.527261279835513262.JavaMail.jira@thor>
Date: Thu, 22 Jul 2010 17:51:53 -0400 (EDT)
From: "Jordan Sissel (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6876) Path.suffix(...) on a path url with
 no "path" results in NullPointerException
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Path.suffix(...) on a path url with no "path" results in NullPointerException
-----------------------------------------------------------------------------

                 Key: HADOOP-6876
                 URL: https://issues.apache.org/jira/browse/HADOOP-6876
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.20.2
            Reporter: Jordan Sissel


I can most briefly demo this using the hbase jruby shell (as it has hadoop in the classpath:

{noformat}

hbase(main):001:0> path = org.apache.hadoop.fs.Path.new("hdfs://testing/")
=> #<Java::OrgApacheHadoopFs::Path:0x6128453c @java_object=#<Java::JavaObject:0x1ad997f9>>
hbase(main):002:0> path.suffix("Test")
NativeException: java.lang.NullPointerException: null
{noformat}

I expected instead that path.suffix("Test") would simply generate a new path with 'Test' appended. For above, this would be the path "hdfs://testing/Test"

Workaround: This should work - instead of using path.suffix(), use this (java):
{noformat}
   Path newpath = new Path(oldpath.toString() + "mysuffix");
{noformat}



-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9685-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 22 21:52:17 2010
Return-Path: <common-issues-return-9685-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16055 invoked from network); 22 Jul 2010 21:52:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jul 2010 21:52:16 -0000
Received: (qmail 98382 invoked by uid 500); 22 Jul 2010 21:52:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98044 invoked by uid 500); 22 Jul 2010 21:52:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97971 invoked by uid 99); 22 Jul 2010 21:52:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 21:52:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 21:52:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6MLpoir003916
	for <common-issues@hadoop.apache.org>; Thu, 22 Jul 2010 21:51:50 GMT
Message-ID: <22471744.527191279835510536.JavaMail.jira@thor>
Date: Thu, 22 Jul 2010 17:51:50 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6869) [Herriot] Implement a functionality
 for creating either file or folder in task attempt  folder while job is
 running.
In-Reply-To: <1398369.499011279732010273.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891356#action_12891356 ] 

Konstantin Boudnik commented on HADOOP-6869:
--------------------------------------------

- Looks good with one nit. I'd suggest to have methods with fewer arguments simply call ones with more args. E.g. in this case 
{noformat}
public void createFile(String path, String fileName, boolean local) throws IOException {
  createFile(path, fileName, null, local);
}
{noformat}
instead of duplicating their exact implementation. You have done similar in {{DaemonProtocolAspect}} already.

- Also, in {{DaemonProtocolAspect}} you already have a ref to the filesystem so you can just call its {{createFile}} methods instead of implementing your own logic 
{noformat}
+  public void DaemonProtocol.createFile(String path, String fileName, 
+     FsPermission permission, boolean local) throws IOException {
+    Path p = new Path(path); 
+    FileSystem fs = getFS(p, local);
{noformat}
just call {{fs.createNewFile()}} to have new file created for you. 

- also you are writing some content into this file which seems to be an inadvertent side-effect. Doesn't look like a good idea.

> [Herriot] Implement a functionality for creating either file or folder in task attempt  folder while job is running.
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6869
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6869
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6869-ydist-security.patch, HADOOP-6869.patch
>
>
> Functionality for creating either files or folders in task attempt folder while job is running. The functionality covers the following methods.
> 1. public void DaemonProtocol.createFile(String path, String fileName, boolean local) throws IOException; 
> It uses to create a file with full permissions.
> 2.   public void DaemonProtocol.createFile(String path, String fileName, FsPermission permission, boolean local) throws IOException; 
> It uses to create a file with given permissions.
> 3.   public void DaemonProtocol.createFolder(String path, String folderName, boolean local) throws IOException;
> It uses to create a file with full permissions.
> 4.   public void DaemonProtocol.createFolder(String path, String folderName, FsPermission permission, boolean local) throws IOException;
> It uses to create a folder with given permissions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9686-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 22 22:47:21 2010
Return-Path: <common-issues-return-9686-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35897 invoked from network); 22 Jul 2010 22:47:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jul 2010 22:47:21 -0000
Received: (qmail 73506 invoked by uid 500); 22 Jul 2010 22:47:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73444 invoked by uid 500); 22 Jul 2010 22:47:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73436 invoked by uid 99); 22 Jul 2010 22:47:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 22:47:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 22:47:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6MMkqvj004751
	for <common-issues@hadoop.apache.org>; Thu, 22 Jul 2010 22:46:52 GMT
Message-ID: <30384159.528351279838812010.JavaMail.jira@thor>
Date: Thu, 22 Jul 2010 18:46:52 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6864) Provide a JNI-based implementation of
 ShellBasedUnixGroupsNetgroupMapping (implementation of
 GroupMappingServiceProvider)
In-Reply-To: <17354207.429501279319091600.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Erik Steffl updated HADOOP-6864:
--------------------------------

    Attachment: HADOOP-6864-0.20.1xx.patch

> Provide a JNI-based implementation of ShellBasedUnixGroupsNetgroupMapping (implementation of GroupMappingServiceProvider)
> -------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6864
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6864
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6864-0.20.1xx.patch
>
>
> The netgroups implementation of GroupMappingServiceProvider (see ShellBasedUnixGroupsNetgroupMapping.java) does a fork of a unix command to get the netgroups of a user. Since the group resolution happens in the servers, this might be costly. This jira aims at providing a JNI-based implementation for GroupMappingServiceProvider.
> Note that this is similar to what https://issues.apache.org/jira/browse/HADOOP-6818 does for implementation of GroupMappingServiceProvider that  supports only unix groups.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9687-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 22 23:17:12 2010
Return-Path: <common-issues-return-9687-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47485 invoked from network); 22 Jul 2010 23:17:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jul 2010 23:17:12 -0000
Received: (qmail 10767 invoked by uid 500); 22 Jul 2010 23:17:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10643 invoked by uid 500); 22 Jul 2010 23:17:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10634 invoked by uid 99); 22 Jul 2010 23:17:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 23:17:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 23:17:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6MNGoMM005728
	for <common-issues@hadoop.apache.org>; Thu, 22 Jul 2010 23:16:50 GMT
Message-ID: <23819841.528821279840610179.JavaMail.jira@thor>
Date: Thu, 22 Jul 2010 19:16:50 -0400 (EDT)
From: "Jordan Sissel (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6876) Path.suffix(...) on a path url with
 no "path" results in NullPointerException
In-Reply-To: <24368989.527261279835513262.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891390#action_12891390 ] 

Jordan Sissel commented on HADOOP-6876:
---------------------------------------

Updated my own code and confirmed that my suggested workaround does work.

> Path.suffix(...) on a path url with no "path" results in NullPointerException
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-6876
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6876
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: Jordan Sissel
>
> I can most briefly demo this using the hbase jruby shell (as it has hadoop in the classpath:
> {noformat}
> hbase(main):001:0> path = org.apache.hadoop.fs.Path.new("hdfs://testing/")
> => #<Java::OrgApacheHadoopFs::Path:0x6128453c @java_object=#<Java::JavaObject:0x1ad997f9>>
> hbase(main):002:0> path.suffix("Test")
> NativeException: java.lang.NullPointerException: null
> {noformat}
> I expected instead that path.suffix("Test") would simply generate a new path with 'Test' appended. For above, this would be the path "hdfs://testing/Test"
> Workaround: This should work - instead of using path.suffix(), use this (java):
> {noformat}
>    Path newpath = new Path(oldpath.toString() + "mysuffix");
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9688-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 00:07:13 2010
Return-Path: <common-issues-return-9688-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65099 invoked from network); 23 Jul 2010 00:07:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 00:07:12 -0000
Received: (qmail 82217 invoked by uid 500); 23 Jul 2010 00:07:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82055 invoked by uid 500); 23 Jul 2010 00:07:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82047 invoked by uid 99); 23 Jul 2010 00:07:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 00:07:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 00:07:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6N06o80006330
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 00:06:50 GMT
Message-ID: <24551011.529791279843610838.JavaMail.jira@thor>
Date: Thu, 22 Jul 2010 20:06:50 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6861) Method in Credentials to read and
 write a token storage file.
In-Reply-To: <21462038.384681279145270696.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6861:
--------------------------------

           Status: Resolved  (was: Patch Available)
    Fix Version/s: 0.22.0
       Resolution: Fixed

I just committed this. Thanks, Jitendra & Owen!

> Method in Credentials to read and write a token storage file.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6861
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6861
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6861.1.patch, HADOOP-6861.2.patch
>
>
> The jira covers the changes in common corresponding to MAPREDUCE-1566. 
> This jira adds  new non-static methods in Credentials to read and write token storage file. A method to copy tokens from another credential object is also added. Static method readTokensAndLoadInUGI is removed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9689-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 00:18:16 2010
Return-Path: <common-issues-return-9689-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66383 invoked from network); 23 Jul 2010 00:18:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 00:18:15 -0000
Received: (qmail 87520 invoked by uid 500); 23 Jul 2010 00:18:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87395 invoked by uid 500); 23 Jul 2010 00:18:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87387 invoked by uid 99); 23 Jul 2010 00:18:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 00:18:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 00:18:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6N0Hsgw006442
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 00:17:54 GMT
Message-ID: <28566454.530061279844274377.JavaMail.jira@thor>
Date: Thu, 22 Jul 2010 20:17:54 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6863) Compile-native still fails on Mac OS
 X.
In-Reply-To: <18053660.404611279230589983.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-6863:
------------------------------

    Status: Patch Available  (was: Open)

> Compile-native still fails on Mac OS X.
> ---------------------------------------
>
>                 Key: HADOOP-6863
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6863
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>         Attachments: hadoop-6863-20100715.patch, hadoop-6863-20100721.patch
>
>
> I am still getting failures when I try to compile native library on Mac OS X (10.5.8 Leopard). The problems are two fold:
> - Although "aclocal.m4" is changed after HADOOP-3659 is committed, the corresponding "configure" script is not re-generated.
> - It cannot find libjvm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9690-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 00:18:18 2010
Return-Path: <common-issues-return-9690-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66434 invoked from network); 23 Jul 2010 00:18:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 00:18:17 -0000
Received: (qmail 87577 invoked by uid 500); 23 Jul 2010 00:18:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87551 invoked by uid 500); 23 Jul 2010 00:18:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87543 invoked by uid 99); 23 Jul 2010 00:18:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 00:18:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 00:18:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6N0Hqv7006412
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 00:17:53 GMT
Message-ID: <19326023.529961279844272620.JavaMail.jira@thor>
Date: Thu, 22 Jul 2010 20:17:52 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6863) Compile-native still fails on Mac OS
 X.
In-Reply-To: <18053660.404611279230589983.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-6863:
------------------------------

    Status: Open  (was: Patch Available)

> Compile-native still fails on Mac OS X.
> ---------------------------------------
>
>                 Key: HADOOP-6863
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6863
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>         Attachments: hadoop-6863-20100715.patch, hadoop-6863-20100721.patch
>
>
> I am still getting failures when I try to compile native library on Mac OS X (10.5.8 Leopard). The problems are two fold:
> - Although "aclocal.m4" is changed after HADOOP-3659 is committed, the corresponding "configure" script is not re-generated.
> - It cannot find libjvm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9691-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 00:18:18 2010
Return-Path: <common-issues-return-9691-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66469 invoked from network); 23 Jul 2010 00:18:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 00:18:17 -0000
Received: (qmail 87742 invoked by uid 500); 23 Jul 2010 00:18:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87708 invoked by uid 500); 23 Jul 2010 00:18:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87652 invoked by uid 99); 23 Jul 2010 00:18:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 00:18:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 00:18:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6N0Hr1X006427
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 00:17:53 GMT
Message-ID: <11521406.530011279844273706.JavaMail.jira@thor>
Date: Thu, 22 Jul 2010 20:17:53 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6863) Compile-native still fails on Mac OS
 X.
In-Reply-To: <18053660.404611279230589983.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-6863:
------------------------------

    Attachment: hadoop-6863-20100721.patch

New patch that is generated using automake 1.9 aclocal 1.9 and autoconf 2.59.

> Compile-native still fails on Mac OS X.
> ---------------------------------------
>
>                 Key: HADOOP-6863
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6863
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>         Attachments: hadoop-6863-20100715.patch, hadoop-6863-20100721.patch
>
>
> I am still getting failures when I try to compile native library on Mac OS X (10.5.8 Leopard). The problems are two fold:
> - Although "aclocal.m4" is changed after HADOOP-3659 is committed, the corresponding "configure" script is not re-generated.
> - It cannot find libjvm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9692-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 00:34:15 2010
Return-Path: <common-issues-return-9692-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77604 invoked from network); 23 Jul 2010 00:34:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 00:34:15 -0000
Received: (qmail 98514 invoked by uid 500); 23 Jul 2010 00:34:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98454 invoked by uid 500); 23 Jul 2010 00:34:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98446 invoked by uid 99); 23 Jul 2010 00:34:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 00:34:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 00:34:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6N0Xrhe006578
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 00:33:54 GMT
Message-ID: <27912704.530381279845233881.JavaMail.jira@thor>
Date: Thu, 22 Jul 2010 20:33:53 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6861) Method in Credentials to read and
 write a token storage file.
In-Reply-To: <21462038.384681279145270696.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891423#action_12891423 ] 

Hudson commented on HADOOP-6861:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #334 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/334/])
    HADOOP-6861. Adds new non-static methods in Credentials to read and write token storage file. Contributed by Jitendra Pandey & Owen O'Malley.


> Method in Credentials to read and write a token storage file.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6861
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6861
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6861.1.patch, HADOOP-6861.2.patch
>
>
> The jira covers the changes in common corresponding to MAPREDUCE-1566. 
> This jira adds  new non-static methods in Credentials to read and write token storage file. A method to copy tokens from another credential object is also added. Static method readTokensAndLoadInUGI is removed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9693-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 00:43:15 2010
Return-Path: <common-issues-return-9693-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79484 invoked from network); 23 Jul 2010 00:43:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 00:43:14 -0000
Received: (qmail 9310 invoked by uid 500); 23 Jul 2010 00:43:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9237 invoked by uid 500); 23 Jul 2010 00:43:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9229 invoked by uid 99); 23 Jul 2010 00:43:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 00:43:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 00:43:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6N0gq6V006652
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 00:42:53 GMT
Message-ID: <31979004.530551279845772859.JavaMail.jira@thor>
Date: Thu, 22 Jul 2010 20:42:52 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6863) Compile-native still fails on Mac
 OS X.
In-Reply-To: <18053660.404611279230589983.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891426#action_12891426 ] 

Hadoop QA commented on HADOOP-6863:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12450246/hadoop-6863-20100721.patch
  against trunk revision 966911.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/628/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/628/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/628/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/628/console

This message is automatically generated.

> Compile-native still fails on Mac OS X.
> ---------------------------------------
>
>                 Key: HADOOP-6863
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6863
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>         Attachments: hadoop-6863-20100715.patch, hadoop-6863-20100721.patch
>
>
> I am still getting failures when I try to compile native library on Mac OS X (10.5.8 Leopard). The problems are two fold:
> - Although "aclocal.m4" is changed after HADOOP-3659 is committed, the corresponding "configure" script is not re-generated.
> - It cannot find libjvm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9694-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 00:51:15 2010
Return-Path: <common-issues-return-9694-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81171 invoked from network); 23 Jul 2010 00:51:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 00:51:15 -0000
Received: (qmail 16326 invoked by uid 500); 23 Jul 2010 00:51:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16155 invoked by uid 500); 23 Jul 2010 00:51:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16147 invoked by uid 99); 23 Jul 2010 00:51:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 00:51:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 00:51:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6N0oobo006728
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 00:50:50 GMT
Message-ID: <10000515.530721279846250311.JavaMail.jira@thor>
Date: Thu, 22 Jul 2010 20:50:50 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6869) [Herriot] Implement a functionality
 for creating either file or folder in task attempt  folder while job is
 running.
In-Reply-To: <1398369.499011279732010273.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891429#action_12891429 ] 

Konstantin Boudnik commented on HADOOP-6869:
--------------------------------------------

In continuation of the comment above: since there's {{getFS()}} method which exposes filesystem API from a particular daemon then creating additional 4 wrappers around the standard API doesn't make much sense to me. These wrappers are too obvious and then will simply crowd Herriot API without adding much value.

> [Herriot] Implement a functionality for creating either file or folder in task attempt  folder while job is running.
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6869
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6869
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6869-ydist-security.patch, HADOOP-6869.patch
>
>
> Functionality for creating either files or folders in task attempt folder while job is running. The functionality covers the following methods.
> 1. public void DaemonProtocol.createFile(String path, String fileName, boolean local) throws IOException; 
> It uses to create a file with full permissions.
> 2.   public void DaemonProtocol.createFile(String path, String fileName, FsPermission permission, boolean local) throws IOException; 
> It uses to create a file with given permissions.
> 3.   public void DaemonProtocol.createFolder(String path, String folderName, boolean local) throws IOException;
> It uses to create a file with full permissions.
> 4.   public void DaemonProtocol.createFolder(String path, String folderName, FsPermission permission, boolean local) throws IOException;
> It uses to create a folder with given permissions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9695-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 01:51:14 2010
Return-Path: <common-issues-return-9695-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11574 invoked from network); 23 Jul 2010 01:51:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 01:51:14 -0000
Received: (qmail 62295 invoked by uid 500); 23 Jul 2010 01:51:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62258 invoked by uid 500); 23 Jul 2010 01:51:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62250 invoked by uid 99); 23 Jul 2010 01:51:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 01:51:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 01:51:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6N1orFQ007347
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 01:50:53 GMT
Message-ID: <28028839.531871279849853033.JavaMail.jira@thor>
Date: Thu, 22 Jul 2010 21:50:53 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6861) Method in Credentials to read and
 write a token storage file.
In-Reply-To: <21462038.384681279145270696.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891445#action_12891445 ] 

Hudson commented on HADOOP-6861:
--------------------------------

Integrated in Hadoop-Hdfs-trunk-Commit #350 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/350/])
    HDFS-1302. The HDFS side of the changes corresponding to HADOOP-6861. Contributed by Jitendra Pandey & Owen O'Malley.


> Method in Credentials to read and write a token storage file.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6861
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6861
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6861.1.patch, HADOOP-6861.2.patch
>
>
> The jira covers the changes in common corresponding to MAPREDUCE-1566. 
> This jira adds  new non-static methods in Credentials to read and write token storage file. A method to copy tokens from another credential object is also added. Static method readTokensAndLoadInUGI is removed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9696-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 06:14:15 2010
Return-Path: <common-issues-return-9696-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74953 invoked from network); 23 Jul 2010 06:14:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 06:14:15 -0000
Received: (qmail 5451 invoked by uid 500); 23 Jul 2010 06:14:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5317 invoked by uid 500); 23 Jul 2010 06:14:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5309 invoked by uid 99); 23 Jul 2010 06:14:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 06:14:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 06:14:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6N6Do1P009472
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 06:13:50 GMT
Message-ID: <10638478.534021279865630084.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 02:13:50 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6869) [Herriot] Implement a functionality
 for creating either file or folder in task attempt  folder while job is
 running.
In-Reply-To: <1398369.499011279732010273.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891486#action_12891486 ] 

Vinay Kumar Thota commented on HADOOP-6869:
-------------------------------------------

Hi Cos, I do agree with you. However my requirement is creating the user define files and folders in task attempt id folder with different kind of permissions while job is running. Later while cleaning up, i just wanted make sure whether all the user defined files and folders are cleaned up or not for that job. So that I have defined the above methods for fulfilling my requirement. As my understand, i don't think so, the FileSystem is providing the no such straight forward methods(createNewFile and createNewFolder) like you mentioned.

In the four methods, the first method creates the files with full permission.Suppose if user wants to give his own permissions while creating file in that case he can use the second method.Same way for folder creation also.

bq. also you are writing some content into this file which seems to be an inadvertent side-effect. Doesn't look like a good idea

I will remove this part in the code.



> [Herriot] Implement a functionality for creating either file or folder in task attempt  folder while job is running.
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6869
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6869
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6869-ydist-security.patch, HADOOP-6869.patch
>
>
> Functionality for creating either files or folders in task attempt folder while job is running. The functionality covers the following methods.
> 1. public void DaemonProtocol.createFile(String path, String fileName, boolean local) throws IOException; 
> It uses to create a file with full permissions.
> 2.   public void DaemonProtocol.createFile(String path, String fileName, FsPermission permission, boolean local) throws IOException; 
> It uses to create a file with given permissions.
> 3.   public void DaemonProtocol.createFolder(String path, String folderName, boolean local) throws IOException;
> It uses to create a file with full permissions.
> 4.   public void DaemonProtocol.createFolder(String path, String folderName, FsPermission permission, boolean local) throws IOException;
> It uses to create a folder with given permissions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9697-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 07:15:14 2010
Return-Path: <common-issues-return-9697-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98086 invoked from network); 23 Jul 2010 07:15:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 07:15:14 -0000
Received: (qmail 51031 invoked by uid 500); 23 Jul 2010 07:15:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50893 invoked by uid 500); 23 Jul 2010 07:15:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50885 invoked by uid 99); 23 Jul 2010 07:15:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 07:15:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 07:15:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6N7Eoq5010059
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 07:14:50 GMT
Message-ID: <27464753.534891279869290465.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 03:14:50 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6875) [Herriot] Couple of issues occurred
 while running the tests in a cluster with security enabled.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891499#action_12891499 ] 

Vinay Kumar Thota commented on HADOOP-6875:
-------------------------------------------

I totally agreed with you and It's valid. So I have created separate ticket(MAPREDUCE-1962) for that path and remaining thing keeping as it is.

> [Herriot] Couple of issues occurred while running the tests in a cluster with security enabled.
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6875
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6875
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 1913-ydist-security.patch, HADOOP-6875.patch, MAPREDUCE-1913.patch
>
>
> 1. New configuration directory is not cleaning up after resetting to default configuration directory in a pushconfig functionality. Because of this reason, it's giving  permission denied problem for a folder, if  other user tried running the tests in the same cluster with pushconfig functionality. I could see this issue while running the tests on a cluster with security enabled and different user.
> I have added the functionality for above issue and attaching the patch
> 2.  Throwing IOException and it says token is expired while running  the tests. I could see this issue in a secure cluster.
> This issue has been resolved by setting the following attribute in the configuration. 
> mapreduce.job.complete.cancel.delegation.tokens=false
>  
> adding/updating this attribute in the push configuration functionality while creating the new configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9698-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 09:37:19 2010
Return-Path: <common-issues-return-9698-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36238 invoked from network); 23 Jul 2010 09:37:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 09:37:19 -0000
Received: (qmail 17642 invoked by uid 500); 23 Jul 2010 09:37:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16813 invoked by uid 500); 23 Jul 2010 09:37:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16793 invoked by uid 99); 23 Jul 2010 09:37:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 09:37:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 09:37:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6N9artZ011792
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 09:36:53 GMT
Message-ID: <5682361.538171279877813267.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 05:36:53 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6875) [Herriot] Couple of issues occurred
 while running the tests in a cluster with security enabled.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinay Kumar Thota updated HADOOP-6875:
--------------------------------------

    Attachment: 6875-ydist-security.patch

Excluded the mapreduce related content in the patch.

> [Herriot] Couple of issues occurred while running the tests in a cluster with security enabled.
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6875
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6875
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 1913-ydist-security.patch, 6875-ydist-security.patch, HADOOP-6875.patch, MAPREDUCE-1913.patch
>
>
> 1. New configuration directory is not cleaning up after resetting to default configuration directory in a pushconfig functionality. Because of this reason, it's giving  permission denied problem for a folder, if  other user tried running the tests in the same cluster with pushconfig functionality. I could see this issue while running the tests on a cluster with security enabled and different user.
> I have added the functionality for above issue and attaching the patch
> 2.  Throwing IOException and it says token is expired while running  the tests. I could see this issue in a secure cluster.
> This issue has been resolved by setting the following attribute in the configuration. 
> mapreduce.job.complete.cancel.delegation.tokens=false
>  
> adding/updating this attribute in the push configuration functionality while creating the new configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9699-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 10:13:15 2010
Return-Path: <common-issues-return-9699-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47451 invoked from network); 23 Jul 2010 10:13:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 10:13:14 -0000
Received: (qmail 53807 invoked by uid 500); 23 Jul 2010 10:13:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53417 invoked by uid 500); 23 Jul 2010 10:13:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53404 invoked by uid 99); 23 Jul 2010 10:13:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 10:13:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 10:13:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6NACoY1012507
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 10:12:50 GMT
Message-ID: <10246965.539761279879970273.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 06:12:50 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6875) [Herriot] Couple of issues occurred
 while running the tests in a cluster with security enabled.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinay Kumar Thota updated HADOOP-6875:
--------------------------------------

    Attachment: 6875-ydist-security.patch
                HADOOP-6875.patch

> [Herriot] Couple of issues occurred while running the tests in a cluster with security enabled.
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6875
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6875
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 1913-ydist-security.patch, 6875-ydist-security.patch, 6875-ydist-security.patch, HADOOP-6875.patch, HADOOP-6875.patch, MAPREDUCE-1913.patch
>
>
> 1. New configuration directory is not cleaning up after resetting to default configuration directory in a pushconfig functionality. Because of this reason, it's giving  permission denied problem for a folder, if  other user tried running the tests in the same cluster with pushconfig functionality. I could see this issue while running the tests on a cluster with security enabled and different user.
> I have added the functionality for above issue and attaching the patch
> 2.  Throwing IOException and it says token is expired while running  the tests. I could see this issue in a secure cluster.
> This issue has been resolved by setting the following attribute in the configuration. 
> mapreduce.job.complete.cancel.delegation.tokens=false
>  
> adding/updating this attribute in the push configuration functionality while creating the new configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9700-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 10:20:16 2010
Return-Path: <common-issues-return-9700-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49039 invoked from network); 23 Jul 2010 10:20:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 10:20:16 -0000
Received: (qmail 58981 invoked by uid 500); 23 Jul 2010 10:20:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58949 invoked by uid 500); 23 Jul 2010 10:20:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58941 invoked by uid 99); 23 Jul 2010 10:20:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 10:20:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 10:20:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6NAJpx4012686
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 10:19:51 GMT
Message-ID: <31584420.540261279880391399.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 06:19:51 -0400 (EDT)
From: "Balaji Rajagopalan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6875) [Herriot] Couple of issues occurred
 while running the tests in a cluster with security enabled.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891548#action_12891548 ] 

Balaji Rajagopalan commented on HADOOP-6875:
--------------------------------------------

+1

> [Herriot] Couple of issues occurred while running the tests in a cluster with security enabled.
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6875
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6875
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 1913-ydist-security.patch, 6875-ydist-security.patch, 6875-ydist-security.patch, HADOOP-6875.patch, HADOOP-6875.patch, MAPREDUCE-1913.patch
>
>
> 1. New configuration directory is not cleaning up after resetting to default configuration directory in a pushconfig functionality. Because of this reason, it's giving  permission denied problem for a folder, if  other user tried running the tests in the same cluster with pushconfig functionality. I could see this issue while running the tests on a cluster with security enabled and different user.
> I have added the functionality for above issue and attaching the patch
> 2.  Throwing IOException and it says token is expired while running  the tests. I could see this issue in a secure cluster.
> This issue has been resolved by setting the following attribute in the configuration. 
> mapreduce.job.complete.cancel.delegation.tokens=false
>  
> adding/updating this attribute in the push configuration functionality while creating the new configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9701-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 18:07:18 2010
Return-Path: <common-issues-return-9701-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44199 invoked from network); 23 Jul 2010 18:07:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 18:07:18 -0000
Received: (qmail 98656 invoked by uid 500); 23 Jul 2010 18:07:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98598 invoked by uid 500); 23 Jul 2010 18:07:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98590 invoked by uid 99); 23 Jul 2010 18:07:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 18:07:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 18:07:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6NI6rhg018568
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 18:06:53 GMT
Message-ID: <4006945.550711279908413685.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 14:06:53 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6875) [Herriot] Couple of issues occurred
 while running the tests in a cluster with security enabled.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6875:
---------------------------------------

          Status: Patch Available  (was: Open)
    Hadoop Flags: [Reviewed]

+1 patch looks good.

> [Herriot] Couple of issues occurred while running the tests in a cluster with security enabled.
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6875
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6875
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 1913-ydist-security.patch, 6875-ydist-security.patch, 6875-ydist-security.patch, HADOOP-6875.patch, HADOOP-6875.patch, MAPREDUCE-1913.patch
>
>
> 1. New configuration directory is not cleaning up after resetting to default configuration directory in a pushconfig functionality. Because of this reason, it's giving  permission denied problem for a folder, if  other user tried running the tests in the same cluster with pushconfig functionality. I could see this issue while running the tests on a cluster with security enabled and different user.
> I have added the functionality for above issue and attaching the patch
> 2.  Throwing IOException and it says token is expired while running  the tests. I could see this issue in a secure cluster.
> This issue has been resolved by setting the following attribute in the configuration. 
> mapreduce.job.complete.cancel.delegation.tokens=false
>  
> adding/updating this attribute in the push configuration functionality while creating the new configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9702-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 18:19:13 2010
Return-Path: <common-issues-return-9702-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50436 invoked from network); 23 Jul 2010 18:19:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 18:19:13 -0000
Received: (qmail 28716 invoked by uid 500); 23 Jul 2010 18:19:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28685 invoked by uid 500); 23 Jul 2010 18:19:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28677 invoked by uid 99); 23 Jul 2010 18:19:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 18:19:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 18:19:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6NIIpFE018901
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 18:18:51 GMT
Message-ID: <19334908.551731279909131537.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 14:18:51 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6864) Provide a JNI-based implementation
 of ShellBasedUnixGroupsNetgroupMapping (implementation of
 GroupMappingServiceProvider)
In-Reply-To: <17354207.429501279319091600.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891703#action_12891703 ] 

Boris Shkolnik commented on HADOOP-6864:
----------------------------------------

1. put a comment before (and other places, including C code)
 users = getUsersForNetgroupJNI(netgroup.substring(1)); 
2. instead  Arrays.asList(new String[0]); - just construct an empty list.
3. Please put a disclaimer that JNI implementation returns only groups which appear in the ACL
4. synchronize excess to the JNI functionality
5. Please test valid and invalid cases (empty or non-existing groups)..
6. Please test again :)

otherwise +1



> Provide a JNI-based implementation of ShellBasedUnixGroupsNetgroupMapping (implementation of GroupMappingServiceProvider)
> -------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6864
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6864
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6864-0.20.1xx.patch
>
>
> The netgroups implementation of GroupMappingServiceProvider (see ShellBasedUnixGroupsNetgroupMapping.java) does a fork of a unix command to get the netgroups of a user. Since the group resolution happens in the servers, this might be costly. This jira aims at providing a JNI-based implementation for GroupMappingServiceProvider.
> Note that this is similar to what https://issues.apache.org/jira/browse/HADOOP-6818 does for implementation of GroupMappingServiceProvider that  supports only unix groups.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9703-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 18:32:16 2010
Return-Path: <common-issues-return-9703-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63543 invoked from network); 23 Jul 2010 18:32:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 18:32:16 -0000
Received: (qmail 60630 invoked by uid 500); 23 Jul 2010 18:32:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60594 invoked by uid 500); 23 Jul 2010 18:32:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60583 invoked by uid 99); 23 Jul 2010 18:32:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 18:32:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 18:32:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6NIVpZT019062
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 18:31:51 GMT
Message-ID: <16965821.552141279909911206.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 14:31:51 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6875) [Herriot] Couple of issues occurred
 while running the tests in a cluster with security enabled.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891713#action_12891713 ] 

Hadoop QA commented on HADOOP-6875:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12450293/HADOOP-6875.patch
  against trunk revision 966919.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/629/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/629/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/629/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/629/console

This message is automatically generated.

> [Herriot] Couple of issues occurred while running the tests in a cluster with security enabled.
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6875
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6875
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 1913-ydist-security.patch, 6875-ydist-security.patch, 6875-ydist-security.patch, HADOOP-6875.patch, HADOOP-6875.patch, MAPREDUCE-1913.patch
>
>
> 1. New configuration directory is not cleaning up after resetting to default configuration directory in a pushconfig functionality. Because of this reason, it's giving  permission denied problem for a folder, if  other user tried running the tests in the same cluster with pushconfig functionality. I could see this issue while running the tests on a cluster with security enabled and different user.
> I have added the functionality for above issue and attaching the patch
> 2.  Throwing IOException and it says token is expired while running  the tests. I could see this issue in a secure cluster.
> This issue has been resolved by setting the following attribute in the configuration. 
> mapreduce.job.complete.cancel.delegation.tokens=false
>  
> adding/updating this attribute in the push configuration functionality while creating the new configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9704-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 18:37:19 2010
Return-Path: <common-issues-return-9704-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65477 invoked from network); 23 Jul 2010 18:37:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 18:37:19 -0000
Received: (qmail 72780 invoked by uid 500); 23 Jul 2010 18:37:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72718 invoked by uid 500); 23 Jul 2010 18:37:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72710 invoked by uid 99); 23 Jul 2010 18:37:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 18:37:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 18:37:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6NIaraf019150
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 18:36:54 GMT
Message-ID: <955050.552411279910213733.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 14:36:53 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6869) [Herriot] Implement a functionality
 for creating either file or folder in task attempt  folder while job is
 running.
In-Reply-To: <1398369.499011279732010273.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891719#action_12891719 ] 

Konstantin Boudnik commented on HADOOP-6869:
--------------------------------------------

bq. my requirement is creating the user define files and folders in task attempt id folder with different kind of permissions while job is running

This reasoning sounds like an argument to move this functionality up the stack to the MR cluster concrete implementation. But let's suppose they are useful enough and let them be in the Common.

bq. In the four methods, the first method creates the files with full permission.Suppose if user wants to give his own permissions while creating file in that case he can use the second method.Same way for folder creation also.

I do understand that. What I have said is that you need a generic enough method which does most of needed functionality (i.e. create a file with specified permissions) and a wrapper around it which creates a file with all permissions. The implementation of the latter is essentially a call to the former with *all* permissions being passed. This is already done in this very patch for the implementation of {{createFolder(..)}} methods. Look at your own code.

> [Herriot] Implement a functionality for creating either file or folder in task attempt  folder while job is running.
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6869
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6869
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6869-ydist-security.patch, HADOOP-6869.patch
>
>
> Functionality for creating either files or folders in task attempt folder while job is running. The functionality covers the following methods.
> 1. public void DaemonProtocol.createFile(String path, String fileName, boolean local) throws IOException; 
> It uses to create a file with full permissions.
> 2.   public void DaemonProtocol.createFile(String path, String fileName, FsPermission permission, boolean local) throws IOException; 
> It uses to create a file with given permissions.
> 3.   public void DaemonProtocol.createFolder(String path, String folderName, boolean local) throws IOException;
> It uses to create a file with full permissions.
> 4.   public void DaemonProtocol.createFolder(String path, String folderName, FsPermission permission, boolean local) throws IOException;
> It uses to create a folder with given permissions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9705-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 18:41:30 2010
Return-Path: <common-issues-return-9705-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66543 invoked from network); 23 Jul 2010 18:41:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 18:41:30 -0000
Received: (qmail 85525 invoked by uid 500); 23 Jul 2010 18:41:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85467 invoked by uid 500); 23 Jul 2010 18:41:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85459 invoked by uid 99); 23 Jul 2010 18:41:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 18:41:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 18:41:28 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6NIf8YK019360
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 18:41:08 GMT
Message-ID: <19808856.553071279910468802.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 14:41:08 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6863) Compile-native still fails on Mac
 OS X.
In-Reply-To: <18053660.404611279230589983.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891723#action_12891723 ] 

Hong Tang commented on HADOOP-6863:
-----------------------------------

bq. -1 tests included. The patch doesn't appear to include any new or modified tests.
This is expected as the patch only touches the build system.

bq. -1 javadoc. The javadoc tool appears to have generated 1 warning messages.
This is false alarm.

> Compile-native still fails on Mac OS X.
> ---------------------------------------
>
>                 Key: HADOOP-6863
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6863
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>         Attachments: hadoop-6863-20100715.patch, hadoop-6863-20100721.patch
>
>
> I am still getting failures when I try to compile native library on Mac OS X (10.5.8 Leopard). The problems are two fold:
> - Although "aclocal.m4" is changed after HADOOP-3659 is committed, the corresponding "configure" script is not re-generated.
> - It cannot find libjvm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9706-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 18:57:16 2010
Return-Path: <common-issues-return-9706-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72175 invoked from network); 23 Jul 2010 18:57:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 18:57:15 -0000
Received: (qmail 14768 invoked by uid 500); 23 Jul 2010 18:57:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14689 invoked by uid 500); 23 Jul 2010 18:57:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14592 invoked by uid 99); 23 Jul 2010 18:57:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 18:57:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 18:57:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6NIusjx019637
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 18:56:54 GMT
Message-ID: <12118830.553761279911414290.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 14:56:54 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6864) Provide a JNI-based implementation of
 ShellBasedUnixGroupsNetgroupMapping (implementation of
 GroupMappingServiceProvider)
In-Reply-To: <17354207.429501279319091600.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Erik Steffl updated HADOOP-6864:
--------------------------------

    Attachment: HADOOP-6864-0.20.1xx-1.patch

HADOOP-6864-0.20.1xx-1.patch addresses the comments in review.

Testing: since it depends on system setup it's not completely automatic, the way I tested it I created a copy of TestAccessControlList and set it up to use JniBasedUnixGroupsNetgroupMapping for group mapping and set up /etc/netgroup accordingly (essentially same as what ShellBasedUnixGroupsNetgroupMappingTestWrapper.java returns).

Note that /etc/nsswitch.conf must have line 'netgroup:       files' in it for the settings in /etc/netgroup to be used.

> Provide a JNI-based implementation of ShellBasedUnixGroupsNetgroupMapping (implementation of GroupMappingServiceProvider)
> -------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6864
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6864
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6864-0.20.1xx-1.patch, HADOOP-6864-0.20.1xx.patch
>
>
> The netgroups implementation of GroupMappingServiceProvider (see ShellBasedUnixGroupsNetgroupMapping.java) does a fork of a unix command to get the netgroups of a user. Since the group resolution happens in the servers, this might be costly. This jira aims at providing a JNI-based implementation for GroupMappingServiceProvider.
> Note that this is similar to what https://issues.apache.org/jira/browse/HADOOP-6818 does for implementation of GroupMappingServiceProvider that  supports only unix groups.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9707-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 19:15:16 2010
Return-Path: <common-issues-return-9707-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86245 invoked from network); 23 Jul 2010 19:15:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 19:15:15 -0000
Received: (qmail 43729 invoked by uid 500); 23 Jul 2010 19:15:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43682 invoked by uid 500); 23 Jul 2010 19:15:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43674 invoked by uid 99); 23 Jul 2010 19:15:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 19:15:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 19:15:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6NJEsHc019978
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 19:14:54 GMT
Message-ID: <6474928.554491279912494315.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 15:14:54 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6875) [Herriot] Couple of issues occurred
 while running the tests in a cluster with security enabled.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891746#action_12891746 ] 

Konstantin Boudnik commented on HADOOP-6875:
--------------------------------------------

Going to commit it because all javadoc warnings seem to be coming from security related classes. Current patch is unlikely to introduce any of those.

> [Herriot] Couple of issues occurred while running the tests in a cluster with security enabled.
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6875
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6875
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 1913-ydist-security.patch, 6875-ydist-security.patch, 6875-ydist-security.patch, HADOOP-6875.patch, HADOOP-6875.patch, MAPREDUCE-1913.patch
>
>
> 1. New configuration directory is not cleaning up after resetting to default configuration directory in a pushconfig functionality. Because of this reason, it's giving  permission denied problem for a folder, if  other user tried running the tests in the same cluster with pushconfig functionality. I could see this issue while running the tests on a cluster with security enabled and different user.
> I have added the functionality for above issue and attaching the patch
> 2.  Throwing IOException and it says token is expired while running  the tests. I could see this issue in a secure cluster.
> This issue has been resolved by setting the following attribute in the configuration. 
> mapreduce.job.complete.cancel.delegation.tokens=false
>  
> adding/updating this attribute in the push configuration functionality while creating the new configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9708-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 19:38:16 2010
Return-Path: <common-issues-return-9708-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92966 invoked from network); 23 Jul 2010 19:38:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 19:38:16 -0000
Received: (qmail 77489 invoked by uid 500); 23 Jul 2010 19:38:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77427 invoked by uid 500); 23 Jul 2010 19:38:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77414 invoked by uid 99); 23 Jul 2010 19:38:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 19:38:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 19:38:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6NJbpwk020183
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 19:37:51 GMT
Message-ID: <12290198.554961279913871228.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 15:37:51 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6875) [Herriot] Cleanup of temp.
 configurations is needed upon restart of a cluster
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6875:
---------------------------------------

              Summary: [Herriot] Cleanup of temp. configurations is needed upon restart of a cluster  (was: [Herriot] Couple of issues occurred while running the tests in a cluster with security enabled.)
    Affects Version/s: 0.21.0

> [Herriot] Cleanup of temp. configurations is needed upon restart of a cluster
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-6875
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6875
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 1913-ydist-security.patch, 6875-ydist-security.patch, 6875-ydist-security.patch, HADOOP-6875.patch, HADOOP-6875.patch, MAPREDUCE-1913.patch
>
>
> 1. New configuration directory is not cleaning up after resetting to default configuration directory in a pushconfig functionality. Because of this reason, it's giving  permission denied problem for a folder, if  other user tried running the tests in the same cluster with pushconfig functionality. I could see this issue while running the tests on a cluster with security enabled and different user.
> I have added the functionality for above issue and attaching the patch
> 2.  Throwing IOException and it says token is expired while running  the tests. I could see this issue in a secure cluster.
> This issue has been resolved by setting the following attribute in the configuration. 
> mapreduce.job.complete.cancel.delegation.tokens=false
>  
> adding/updating this attribute in the push configuration functionality while creating the new configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9709-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 19:40:14 2010
Return-Path: <common-issues-return-9709-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94915 invoked from network); 23 Jul 2010 19:40:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 19:40:13 -0000
Received: (qmail 85482 invoked by uid 500); 23 Jul 2010 19:40:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85434 invoked by uid 500); 23 Jul 2010 19:40:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85424 invoked by uid 99); 23 Jul 2010 19:40:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 19:40:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 19:40:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6NJdqmi020227
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 19:39:52 GMT
Message-ID: <25480380.555081279913992465.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 15:39:52 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6875) [Herriot] Cleanup of temp.
 configurations is needed upon restart of a cluster
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6875:
---------------------------------------

           Status: Resolved  (was: Patch Available)
    Fix Version/s: 0.21.0
       Resolution: Fixed

I have committed this to the trunk and 0.21 branch. Thanks Vinay.

> [Herriot] Cleanup of temp. configurations is needed upon restart of a cluster
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-6875
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6875
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>             Fix For: 0.21.0
>
>         Attachments: 1913-ydist-security.patch, 6875-ydist-security.patch, 6875-ydist-security.patch, HADOOP-6875.patch, HADOOP-6875.patch, MAPREDUCE-1913.patch
>
>
> 1. New configuration directory is not cleaning up after resetting to default configuration directory in a pushconfig functionality. Because of this reason, it's giving  permission denied problem for a folder, if  other user tried running the tests in the same cluster with pushconfig functionality. I could see this issue while running the tests on a cluster with security enabled and different user.
> I have added the functionality for above issue and attaching the patch
> 2.  Throwing IOException and it says token is expired while running  the tests. I could see this issue in a secure cluster.
> This issue has been resolved by setting the following attribute in the configuration. 
> mapreduce.job.complete.cancel.delegation.tokens=false
>  
> adding/updating this attribute in the push configuration functionality while creating the new configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9710-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 19:45:13 2010
Return-Path: <common-issues-return-9710-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97977 invoked from network); 23 Jul 2010 19:45:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 19:45:12 -0000
Received: (qmail 92075 invoked by uid 500); 23 Jul 2010 19:45:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92027 invoked by uid 500); 23 Jul 2010 19:45:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92019 invoked by uid 99); 23 Jul 2010 19:45:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 19:45:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 19:45:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6NJiotQ020248
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 19:44:51 GMT
Message-ID: <32479425.555111279914290561.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 15:44:50 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6870) Add FileSystem#listLocatedStatus to
 list a directory's content together with each file's block locations
In-Reply-To: <13042840.506371279749289722.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891759#action_12891759 ] 

Doug Cutting commented on HADOOP-6870:
--------------------------------------

Errors in iteration should turn into a subclass of RuntimeException, not silently fail, no?

Also, it looks to me that recursive file listings across symlinks don't work.  Instinctively, it seems to me they should.

> Add FileSystem#listLocatedStatus to list a directory's content together with each file's block locations
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6870
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6870
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listFiles.patch
>
>
> This jira implements the new FileSystem API as proposed in HDFS-202. The new API aims to eliminate individual "getFileBlockLocations" calls to NN for each file in the input directory of a job. Instead, a file's block locations are returned together with FileStatus when listing a directory, thus improving getSplits performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9711-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 20:09:13 2010
Return-Path: <common-issues-return-9711-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13179 invoked from network); 23 Jul 2010 20:09:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 20:09:12 -0000
Received: (qmail 23751 invoked by uid 500); 23 Jul 2010 20:09:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23687 invoked by uid 500); 23 Jul 2010 20:09:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23679 invoked by uid 99); 23 Jul 2010 20:09:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 20:09:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 20:09:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6NK8pCZ020518
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 20:08:51 GMT
Message-ID: <24836604.555631279915731444.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 16:08:51 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6656:
--------------------------------

           Status: Patch Available  (was: Open)
         Assignee: Devaraj Das  (was: Owen O'Malley)
    Fix Version/s: 0.22.0

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6656-trunk-1.patch, 6656-trunk-2.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9712-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 20:09:15 2010
Return-Path: <common-issues-return-9712-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13227 invoked from network); 23 Jul 2010 20:09:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 20:09:15 -0000
Received: (qmail 23894 invoked by uid 500); 23 Jul 2010 20:09:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23863 invoked by uid 500); 23 Jul 2010 20:09:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23855 invoked by uid 99); 23 Jul 2010 20:09:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 20:09:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 20:09:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6NK8oEn020506
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 20:08:51 GMT
Message-ID: <7886407.555591279915730758.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 16:08:50 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6656:
--------------------------------

    Attachment: 6656-trunk-2.patch

Attaching a patch w.r.t the current trunk.

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.22.0
>
>         Attachments: 6656-trunk-1.patch, 6656-trunk-2.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9713-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 20:48:13 2010
Return-Path: <common-issues-return-9713-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25320 invoked from network); 23 Jul 2010 20:48:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 20:48:11 -0000
Received: (qmail 83823 invoked by uid 500); 23 Jul 2010 20:48:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83231 invoked by uid 500); 23 Jul 2010 20:48:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83049 invoked by uid 99); 23 Jul 2010 20:48:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 20:48:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 20:48:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6NKlo0l020917
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 20:47:50 GMT
Message-ID: <10223047.556651279918070153.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 16:47:50 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6877) Common part of HDFS-1178
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Common part of HDFS-1178
------------------------

                 Key: HADOOP-6877
                 URL: https://issues.apache.org/jira/browse/HADOOP-6877
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Kan Zhang
            Assignee: Kan Zhang


This is the Common part of HDFS-1178.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9714-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 20:50:15 2010
Return-Path: <common-issues-return-9714-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25786 invoked from network); 23 Jul 2010 20:50:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 20:50:15 -0000
Received: (qmail 87841 invoked by uid 500); 23 Jul 2010 20:50:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87805 invoked by uid 500); 23 Jul 2010 20:50:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87795 invoked by uid 99); 23 Jul 2010 20:50:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 20:50:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 20:50:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6NKnoj1020945
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 20:49:50 GMT
Message-ID: <33037089.556721279918190380.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 16:49:50 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6877) Common part of HDFS-1178
In-Reply-To: <10223047.556651279918070153.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6877:
------------------------------

    Attachment: c6877-01.patch

> Common part of HDFS-1178
> ------------------------
>
>                 Key: HADOOP-6877
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6877
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6877-01.patch
>
>
> This is the Common part of HDFS-1178.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9715-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 20:50:18 2010
Return-Path: <common-issues-return-9715-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25905 invoked from network); 23 Jul 2010 20:50:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 20:50:18 -0000
Received: (qmail 88096 invoked by uid 500); 23 Jul 2010 20:50:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88046 invoked by uid 500); 23 Jul 2010 20:50:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88038 invoked by uid 99); 23 Jul 2010 20:50:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 20:50:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 20:50:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6NKnshY020981
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 20:49:54 GMT
Message-ID: <13308792.556841279918194656.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 16:49:54 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6877) Common part of HDFS-1178
In-Reply-To: <10223047.556651279918070153.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6877:
------------------------------

    Status: Patch Available  (was: Open)

> Common part of HDFS-1178
> ------------------------
>
>                 Key: HADOOP-6877
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6877
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6877-01.patch
>
>
> This is the Common part of HDFS-1178.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9716-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 21:50:12 2010
Return-Path: <common-issues-return-9716-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53267 invoked from network); 23 Jul 2010 21:50:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 21:50:12 -0000
Received: (qmail 56762 invoked by uid 500); 23 Jul 2010 21:50:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56735 invoked by uid 500); 23 Jul 2010 21:50:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56725 invoked by uid 99); 23 Jul 2010 21:50:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 21:50:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 21:50:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6NLnohw021713
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 21:49:50 GMT
Message-ID: <15776563.558551279921790728.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 17:49:50 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6873) using delegation token over hftp for
 long running clients (part of hdfs 1296)
In-Reply-To: <3672204.508361279757392441.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6873:
-----------------------------------

    Attachment: HADOOP-6873-2.patch

merged with trunk

> using delegation token over hftp for long running clients (part of hdfs 1296)
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-6873
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6873
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6873-2.patch, HADOOP-6873.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9717-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 21:50:15 2010
Return-Path: <common-issues-return-9717-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53307 invoked from network); 23 Jul 2010 21:50:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 21:50:15 -0000
Received: (qmail 56944 invoked by uid 500); 23 Jul 2010 21:50:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56899 invoked by uid 500); 23 Jul 2010 21:50:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56891 invoked by uid 99); 23 Jul 2010 21:50:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 21:50:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 21:50:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6NLnou9021707
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 21:49:50 GMT
Message-ID: <10646601.558531279921790143.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 17:49:50 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6873) using delegation token over hftp for
 long running clients (part of hdfs 1296)
In-Reply-To: <3672204.508361279757392441.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6873:
-----------------------------------

    Status: Open  (was: Patch Available)

> using delegation token over hftp for long running clients (part of hdfs 1296)
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-6873
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6873
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6873-2.patch, HADOOP-6873.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9718-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 21:50:15 2010
Return-Path: <common-issues-return-9718-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53326 invoked from network); 23 Jul 2010 21:50:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 21:50:15 -0000
Received: (qmail 57011 invoked by uid 500); 23 Jul 2010 21:50:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56909 invoked by uid 500); 23 Jul 2010 21:50:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56900 invoked by uid 99); 23 Jul 2010 21:50:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 21:50:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 21:50:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6NLnp1q021719
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 21:49:51 GMT
Message-ID: <15905319.558571279921790995.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 17:49:50 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6873) using delegation token over hftp for
 long running clients (part of hdfs 1296)
In-Reply-To: <3672204.508361279757392441.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6873:
-----------------------------------

    Status: Patch Available  (was: Open)

> using delegation token over hftp for long running clients (part of hdfs 1296)
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-6873
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6873
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6873-2.patch, HADOOP-6873.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9719-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 22:28:15 2010
Return-Path: <common-issues-return-9719-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64644 invoked from network); 23 Jul 2010 22:28:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 22:28:14 -0000
Received: (qmail 95148 invoked by uid 500); 23 Jul 2010 22:28:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94994 invoked by uid 500); 23 Jul 2010 22:28:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94922 invoked by uid 99); 23 Jul 2010 22:28:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 22:28:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 22:28:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6NMRnUq022230
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 22:27:50 GMT
Message-ID: <29673510.559771279924069583.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 18:27:49 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6878) Use NetgroupCache.java in
 ShellBasedUnixGroupsNetgroupMapping.java
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Use NetgroupCache.java in ShellBasedUnixGroupsNetgroupMapping.java
------------------------------------------------------------------

                 Key: HADOOP-6878
                 URL: https://issues.apache.org/jira/browse/HADOOP-6878
             Project: Hadoop Common
          Issue Type: Improvement
          Components: security
    Affects Versions: 0.22.0
            Reporter: Erik Steffl
             Fix For: 0.22.0


Code to handle netgroups cache is factored out of JniBasedUnixGroupsNetgroupMapping class and implemented in a separate class NetgroupCache.

ShellBasedUnixGroupsNetgroupMapping should also use the same NetgroupCache class instead of implementing the same cache.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9720-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 22:38:11 2010
Return-Path: <common-issues-return-9720-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66959 invoked from network); 23 Jul 2010 22:38:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 22:38:11 -0000
Received: (qmail 6978 invoked by uid 500); 23 Jul 2010 22:38:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6842 invoked by uid 500); 23 Jul 2010 22:38:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6834 invoked by uid 99); 23 Jul 2010 22:38:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 22:38:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 22:38:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6NMbnok022308
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 22:37:50 GMT
Message-ID: <11006512.559861279924669873.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 18:37:49 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6878) Use NetgroupCache.java in
 ShellBasedUnixGroupsNetgroupMapping.java
In-Reply-To: <29673510.559771279924069583.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Erik Steffl reassigned HADOOP-6878:
-----------------------------------

    Assignee: Erik Steffl

> Use NetgroupCache.java in ShellBasedUnixGroupsNetgroupMapping.java
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6878
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6878
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>
> Code to handle netgroups cache is factored out of JniBasedUnixGroupsNetgroupMapping class and implemented in a separate class NetgroupCache.
> ShellBasedUnixGroupsNetgroupMapping should also use the same NetgroupCache class instead of implementing the same cache.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9721-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 23 22:47:14 2010
Return-Path: <common-issues-return-9721-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69227 invoked from network); 23 Jul 2010 22:47:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jul 2010 22:47:14 -0000
Received: (qmail 15999 invoked by uid 500); 23 Jul 2010 22:47:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15902 invoked by uid 500); 23 Jul 2010 22:47:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15894 invoked by uid 99); 23 Jul 2010 22:47:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 22:47:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Jul 2010 22:47:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6NMkocP022393
	for <common-issues@hadoop.apache.org>; Fri, 23 Jul 2010 22:46:50 GMT
Message-ID: <17743155.560071279925210092.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 18:46:50 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6805) add buildDTServiceName method to
 SecurityUtil (as part of MAPREDUCE-1718)
In-Reply-To: <5393650.148181275524636022.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6805:
-----------------------------------

        Status: Resolved  (was: Patch Available)
    Resolution: Fixed

> add buildDTServiceName method to SecurityUtil (as part of MAPREDUCE-1718)
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6805
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6805
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6805.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9722-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 24 00:19:12 2010
Return-Path: <common-issues-return-9722-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91661 invoked from network); 24 Jul 2010 00:19:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Jul 2010 00:19:12 -0000
Received: (qmail 80463 invoked by uid 500); 24 Jul 2010 00:19:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80393 invoked by uid 500); 24 Jul 2010 00:19:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80382 invoked by uid 99); 24 Jul 2010 00:19:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Jul 2010 00:19:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Jul 2010 00:19:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6O0IoeP023518
	for <common-issues@hadoop.apache.org>; Sat, 24 Jul 2010 00:18:50 GMT
Message-ID: <27681641.562471279930730360.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 20:18:50 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6870) Add FileSystem#listLocatedStatus to
 list a directory's content together with each file's block locations
In-Reply-To: <13042840.506371279749289722.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6870:
----------------------------------

    Attachment: listFiles1.patch

This patch addresses Doug's comments.

Unit tests for recursive listing of symbolic links will be added after HADOOP-6872 is fixed.

> Add FileSystem#listLocatedStatus to list a directory's content together with each file's block locations
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6870
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6870
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listFiles.patch, listFiles1.patch
>
>
> This jira implements the new FileSystem API as proposed in HDFS-202. The new API aims to eliminate individual "getFileBlockLocations" calls to NN for each file in the input directory of a job. Instead, a file's block locations are returned together with FileStatus when listing a directory, thus improving getSplits performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9723-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 24 00:22:12 2010
Return-Path: <common-issues-return-9723-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92577 invoked from network); 24 Jul 2010 00:22:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Jul 2010 00:22:11 -0000
Received: (qmail 80960 invoked by uid 500); 24 Jul 2010 00:22:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80827 invoked by uid 500); 24 Jul 2010 00:22:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80819 invoked by uid 99); 24 Jul 2010 00:22:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Jul 2010 00:22:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Jul 2010 00:22:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6O0Lov5023536
	for <common-issues@hadoop.apache.org>; Sat, 24 Jul 2010 00:21:50 GMT
Message-ID: <10591474.562511279930910150.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 20:21:50 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6870) Add FileSystem#listLocatedStatus to
 list a directory's content together with each file's block locations
In-Reply-To: <13042840.506371279749289722.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6870:
----------------------------------

    Attachment:     (was: listFiles1.patch)

> Add FileSystem#listLocatedStatus to list a directory's content together with each file's block locations
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6870
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6870
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listFiles.patch
>
>
> This jira implements the new FileSystem API as proposed in HDFS-202. The new API aims to eliminate individual "getFileBlockLocations" calls to NN for each file in the input directory of a job. Instead, a file's block locations are returned together with FileStatus when listing a directory, thus improving getSplits performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9724-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 24 00:46:20 2010
Return-Path: <common-issues-return-9724-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1861 invoked from network); 24 Jul 2010 00:46:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Jul 2010 00:46:20 -0000
Received: (qmail 90381 invoked by uid 500); 24 Jul 2010 00:46:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90337 invoked by uid 500); 24 Jul 2010 00:46:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90328 invoked by uid 99); 24 Jul 2010 00:46:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Jul 2010 00:46:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Jul 2010 00:46:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6O0jw3J023767
	for <common-issues@hadoop.apache.org>; Sat, 24 Jul 2010 00:45:58 GMT
Message-ID: <22169867.563131279932358299.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 20:45:58 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6870) Add FileSystem#listLocatedStatus to
 list a directory's content together with each file's block locations
In-Reply-To: <13042840.506371279749289722.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6870:
----------------------------------

    Attachment: listFiles1.patch

> Add FileSystem#listLocatedStatus to list a directory's content together with each file's block locations
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6870
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6870
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listFiles.patch, listFiles1.patch
>
>
> This jira implements the new FileSystem API as proposed in HDFS-202. The new API aims to eliminate individual "getFileBlockLocations" calls to NN for each file in the input directory of a job. Instead, a file's block locations are returned together with FileStatus when listing a directory, thus improving getSplits performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9725-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 24 00:46:21 2010
Return-Path: <common-issues-return-9725-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1904 invoked from network); 24 Jul 2010 00:46:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Jul 2010 00:46:20 -0000
Received: (qmail 90534 invoked by uid 500); 24 Jul 2010 00:46:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90380 invoked by uid 500); 24 Jul 2010 00:46:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90338 invoked by uid 99); 24 Jul 2010 00:46:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Jul 2010 00:46:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Jul 2010 00:46:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6O0jwUC023776
	for <common-issues@hadoop.apache.org>; Sat, 24 Jul 2010 00:45:58 GMT
Message-ID: <21450958.563161279932358966.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 20:45:58 -0400 (EDT)
From: "Arvind Prabhakar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6871) When the value of a configuration key
 is set to its unresolved form, it causes the IllegalStateException in
 Configuration.get() stating that substitution depth is too large.
In-Reply-To: <4673721.506611279750129934.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arvind Prabhakar updated HADOOP-6871:
-------------------------------------

    Status: Open  (was: Patch Available)

Canceling and resubmitting patch to trigger the QA build.

> When the value of a configuration key is set to its unresolved form, it causes the IllegalStateException in Configuration.get() stating that substitution depth is too large.
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6871
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6871
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>            Reporter: Arvind Prabhakar
>         Attachments: HADOOP-6871-1.patch, HADOOP-6871.patch
>
>
> When a configuration value is set to its unresolved expression string, it leads to recursive substitution attempts in {{Configuration.substituteVars(String)}} method until the max substitution check kicks in and raises an IllegalStateException indicating that the substitution depth is too large. For example, the configuration key "{{foobar}}" with a value set to "{{$\{foobar\}}}" will cause this behavior. 
> While this is not a usual use case, it can happen in build environments where a property value is not specified and yet being passed into the test mechanism leading to failures due to this limitation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9726-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 24 00:46:23 2010
Return-Path: <common-issues-return-9726-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1959 invoked from network); 24 Jul 2010 00:46:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Jul 2010 00:46:23 -0000
Received: (qmail 90837 invoked by uid 500); 24 Jul 2010 00:46:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90794 invoked by uid 500); 24 Jul 2010 00:46:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90786 invoked by uid 99); 24 Jul 2010 00:46:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Jul 2010 00:46:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Jul 2010 00:46:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6O0jxlP023785
	for <common-issues@hadoop.apache.org>; Sat, 24 Jul 2010 00:45:59 GMT
Message-ID: <22211025.563191279932359492.JavaMail.jira@thor>
Date: Fri, 23 Jul 2010 20:45:59 -0400 (EDT)
From: "Arvind Prabhakar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6871) When the value of a configuration key
 is set to its unresolved form, it causes the IllegalStateException in
 Configuration.get() stating that substitution depth is too large.
In-Reply-To: <4673721.506611279750129934.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arvind Prabhakar updated HADOOP-6871:
-------------------------------------

    Status: Patch Available  (was: Open)

Canceling and resubmitting patch to trigger the QA build.

> When the value of a configuration key is set to its unresolved form, it causes the IllegalStateException in Configuration.get() stating that substitution depth is too large.
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6871
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6871
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>            Reporter: Arvind Prabhakar
>         Attachments: HADOOP-6871-1.patch, HADOOP-6871.patch
>
>
> When a configuration value is set to its unresolved expression string, it leads to recursive substitution attempts in {{Configuration.substituteVars(String)}} method until the max substitution check kicks in and raises an IllegalStateException indicating that the substitution depth is too large. For example, the configuration key "{{foobar}}" with a value set to "{{$\{foobar\}}}" will cause this behavior. 
> While this is not a usual use case, it can happen in build environments where a property value is not specified and yet being passed into the test mechanism leading to failures due to this limitation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9727-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 24 07:28:18 2010
Return-Path: <common-issues-return-9727-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86312 invoked from network); 24 Jul 2010 07:28:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Jul 2010 07:28:18 -0000
Received: (qmail 21516 invoked by uid 500); 24 Jul 2010 07:28:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20242 invoked by uid 500); 24 Jul 2010 07:28:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20229 invoked by uid 99); 24 Jul 2010 07:28:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Jul 2010 07:28:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Jul 2010 07:28:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6O7RnHn026422
	for <common-issues@hadoop.apache.org>; Sat, 24 Jul 2010 07:27:50 GMT
Message-ID: <11760779.565921279956469755.JavaMail.jira@thor>
Date: Sat, 24 Jul 2010 03:27:49 -0400 (EDT)
From: "Iyappan Srinivasan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6879) Jsch jar si required for ivy
 resolution to be included int eh build
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Jsch jar si required for ivy resolution to be included int eh build
-------------------------------------------------------------------

                 Key: HADOOP-6879
                 URL: https://issues.apache.org/jira/browse/HADOOP-6879
             Project: Hadoop Common
          Issue Type: Improvement
    Affects Versions: 0.20.1
            Reporter: Iyappan Srinivasan
            Assignee: Iyappan Srinivasan


http://mvnrepository.com/
com.jcraft =C2=BB jsch=20
0.1.42 version needs to be included in the build. This is  required for her=
riot testcases using jsch.

Please include this in ivy.

jsch is originally located in http://www.jcraft.com/jsch/

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9728-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 24 07:30:14 2010
Return-Path: <common-issues-return-9728-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86691 invoked from network); 24 Jul 2010 07:30:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Jul 2010 07:30:14 -0000
Received: (qmail 22046 invoked by uid 500); 24 Jul 2010 07:30:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21934 invoked by uid 500); 24 Jul 2010 07:30:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21925 invoked by uid 99); 24 Jul 2010 07:30:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Jul 2010 07:30:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Jul 2010 07:30:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6O7ToTV026448
	for <common-issues@hadoop.apache.org>; Sat, 24 Jul 2010 07:29:50 GMT
Message-ID: <2334901.565981279956590112.JavaMail.jira@thor>
Date: Sat, 24 Jul 2010 03:29:50 -0400 (EDT)
From: "Iyappan Srinivasan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6879) Jsch jar is required for ivy
 resolution to be included in the build
In-Reply-To: <11760779.565921279956469755.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6879?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Iyappan Srinivasan updated HADOOP-6879:
---------------------------------------

    Summary: Jsch jar is required for ivy resolution to be included in the =
build  (was: Jsch jar si required for ivy resolution to be included int eh =
build)

> Jsch jar is required for ivy resolution to be included in the build
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6879
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6879
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.1
>            Reporter: Iyappan Srinivasan
>            Assignee: Iyappan Srinivasan
>
> http://mvnrepository.com/
> com.jcraft =C2=BB jsch=20
> 0.1.42 version needs to be included in the build. This is  required for h=
erriot testcases using jsch.
> Please include this in ivy.
> jsch is originally located in http://www.jcraft.com/jsch/

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9729-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 24 20:13:15 2010
Return-Path: <common-issues-return-9729-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62934 invoked from network); 24 Jul 2010 20:13:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Jul 2010 20:13:15 -0000
Received: (qmail 77114 invoked by uid 500); 24 Jul 2010 20:13:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77073 invoked by uid 500); 24 Jul 2010 20:13:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77065 invoked by uid 99); 24 Jul 2010 20:13:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Jul 2010 20:13:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Jul 2010 20:13:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6OKCn2V029669
	for <common-issues@hadoop.apache.org>; Sat, 24 Jul 2010 20:12:50 GMT
Message-ID: <19396611.569471280002369761.JavaMail.jira@thor>
Date: Sat, 24 Jul 2010 16:12:49 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6877) Common part of HDFS-1178
In-Reply-To: <10223047.556651279918070153.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6877:
--------------------------------

    Hadoop Flags: [Reviewed]

+1

> Common part of HDFS-1178
> ------------------------
>
>                 Key: HADOOP-6877
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6877
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6877-01.patch
>
>
> This is the Common part of HDFS-1178.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9730-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 24 20:15:14 2010
Return-Path: <common-issues-return-9730-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63287 invoked from network); 24 Jul 2010 20:15:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Jul 2010 20:15:13 -0000
Received: (qmail 78595 invoked by uid 500); 24 Jul 2010 20:15:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78552 invoked by uid 500); 24 Jul 2010 20:15:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78543 invoked by uid 99); 24 Jul 2010 20:15:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Jul 2010 20:15:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Jul 2010 20:15:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6OKEqH4029702
	for <common-issues@hadoop.apache.org>; Sat, 24 Jul 2010 20:14:52 GMT
Message-ID: <11359595.569561280002492537.JavaMail.jira@thor>
Date: Sat, 24 Jul 2010 16:14:52 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6877) Common part of HDFS-1178
In-Reply-To: <10223047.556651279918070153.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6877:
--------------------------------

    Status: Open  (was: Patch Available)

> Common part of HDFS-1178
> ------------------------
>
>                 Key: HADOOP-6877
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6877
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6877-01.patch
>
>
> This is the Common part of HDFS-1178.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9731-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 24 20:15:17 2010
Return-Path: <common-issues-return-9731-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63338 invoked from network); 24 Jul 2010 20:15:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Jul 2010 20:15:16 -0000
Received: (qmail 78777 invoked by uid 500); 24 Jul 2010 20:15:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78735 invoked by uid 500); 24 Jul 2010 20:15:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78727 invoked by uid 99); 24 Jul 2010 20:15:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Jul 2010 20:15:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Jul 2010 20:15:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6OKEqcN029708
	for <common-issues@hadoop.apache.org>; Sat, 24 Jul 2010 20:14:52 GMT
Message-ID: <11457531.569581280002492817.JavaMail.jira@thor>
Date: Sat, 24 Jul 2010 16:14:52 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6877) Common part of HDFS-1178
In-Reply-To: <10223047.556651279918070153.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6877:
--------------------------------

    Status: Patch Available  (was: Open)

Trying to get Hudson's attention.

> Common part of HDFS-1178
> ------------------------
>
>                 Key: HADOOP-6877
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6877
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6877-01.patch
>
>
> This is the Common part of HDFS-1178.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9732-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 24 22:11:15 2010
Return-Path: <common-issues-return-9732-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83820 invoked from network); 24 Jul 2010 22:11:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Jul 2010 22:11:15 -0000
Received: (qmail 25951 invoked by uid 500); 24 Jul 2010 22:11:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25927 invoked by uid 500); 24 Jul 2010 22:11:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25918 invoked by uid 99); 24 Jul 2010 22:11:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Jul 2010 22:11:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Jul 2010 22:11:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6OMAoJJ000162
	for <common-issues@hadoop.apache.org>; Sat, 24 Jul 2010 22:10:50 GMT
Message-ID: <2701759.570251280009450410.JavaMail.jira@thor>
Date: Sat, 24 Jul 2010 18:10:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892038#action_12892038 ] 

Hadoop QA commented on HADOOP-6656:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12450347/6656-trunk-2.patch
  against trunk revision 967220.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/630/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/630/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/630/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/630/console

This message is automatically generated.

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6656-trunk-1.patch, 6656-trunk-2.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9733-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 24 22:28:16 2010
Return-Path: <common-issues-return-9733-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85744 invoked from network); 24 Jul 2010 22:28:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Jul 2010 22:28:15 -0000
Received: (qmail 27824 invoked by uid 500); 24 Jul 2010 22:28:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27748 invoked by uid 500); 24 Jul 2010 22:28:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27740 invoked by uid 99); 24 Jul 2010 22:28:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Jul 2010 22:28:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Jul 2010 22:28:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6OMRo9j000200
	for <common-issues@hadoop.apache.org>; Sat, 24 Jul 2010 22:27:50 GMT
Message-ID: <7581856.570291280010470181.JavaMail.jira@thor>
Date: Sat, 24 Jul 2010 18:27:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6873) using delegation token over hftp
 for long running clients (part of hdfs 1296)
In-Reply-To: <3672204.508361279757392441.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892040#action_12892040 ] 

Hadoop QA commented on HADOOP-6873:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12450364/HADOOP-6873-2.patch
  against trunk revision 967220.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/631/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/631/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/631/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/631/console

This message is automatically generated.

> using delegation token over hftp for long running clients (part of hdfs 1296)
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-6873
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6873
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6873-2.patch, HADOOP-6873.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9734-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 24 22:48:16 2010
Return-Path: <common-issues-return-9734-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87781 invoked from network); 24 Jul 2010 22:48:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Jul 2010 22:48:16 -0000
Received: (qmail 33377 invoked by uid 500); 24 Jul 2010 22:48:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33340 invoked by uid 500); 24 Jul 2010 22:48:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33332 invoked by uid 99); 24 Jul 2010 22:48:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Jul 2010 22:48:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 24 Jul 2010 22:48:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6OMlooV000296
	for <common-issues@hadoop.apache.org>; Sat, 24 Jul 2010 22:47:50 GMT
Message-ID: <9330920.570401280011670499.JavaMail.jira@thor>
Date: Sat, 24 Jul 2010 18:47:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6871) When the value of a configuration
 key is set to its unresolved form, it causes the IllegalStateException in
 Configuration.get() stating that substitution depth is too large.
In-Reply-To: <4673721.506611279750129934.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892045#action_12892045 ] 

Hadoop QA commented on HADOOP-6871:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12450115/HADOOP-6871-1.patch
  against trunk revision 967220.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/632/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/632/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/632/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/632/console

This message is automatically generated.

> When the value of a configuration key is set to its unresolved form, it causes the IllegalStateException in Configuration.get() stating that substitution depth is too large.
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6871
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6871
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>            Reporter: Arvind Prabhakar
>         Attachments: HADOOP-6871-1.patch, HADOOP-6871.patch
>
>
> When a configuration value is set to its unresolved expression string, it leads to recursive substitution attempts in {{Configuration.substituteVars(String)}} method until the max substitution check kicks in and raises an IllegalStateException indicating that the substitution depth is too large. For example, the configuration key "{{foobar}}" with a value set to "{{$\{foobar\}}}" will cause this behavior. 
> While this is not a usual use case, it can happen in build environments where a property value is not specified and yet being passed into the test mechanism leading to failures due to this limitation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9735-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jul 25 03:32:17 2010
Return-Path: <common-issues-return-9735-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42266 invoked from network); 25 Jul 2010 03:32:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Jul 2010 03:32:17 -0000
Received: (qmail 93632 invoked by uid 500); 25 Jul 2010 03:32:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93501 invoked by uid 500); 25 Jul 2010 03:32:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93489 invoked by uid 99); 25 Jul 2010 03:32:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Jul 2010 03:32:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Jul 2010 03:32:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6P3VnGH001207
	for <common-issues@hadoop.apache.org>; Sun, 25 Jul 2010 03:31:50 GMT
Message-ID: <21166371.571201280028709788.JavaMail.jira@thor>
Date: Sat, 24 Jul 2010 23:31:49 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6869) [Herriot] Implement a functionality
 for creating either file or folder in task attempt  folder while job is
 running.
In-Reply-To: <1398369.499011279732010273.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892065#action_12892065 ] 

Vinay Kumar Thota commented on HADOOP-6869:
-------------------------------------------

Cos,
Please look at my patch once again because I have not duplicated the exact implementation for create file. I have created a wrapper around the actual method with full permission like which i coded for create folder.Might be you have missed to look at that portion in my patch.

> [Herriot] Implement a functionality for creating either file or folder in task attempt  folder while job is running.
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6869
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6869
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6869-ydist-security.patch, HADOOP-6869.patch
>
>
> Functionality for creating either files or folders in task attempt folder while job is running. The functionality covers the following methods.
> 1. public void DaemonProtocol.createFile(String path, String fileName, boolean local) throws IOException; 
> It uses to create a file with full permissions.
> 2.   public void DaemonProtocol.createFile(String path, String fileName, FsPermission permission, boolean local) throws IOException; 
> It uses to create a file with given permissions.
> 3.   public void DaemonProtocol.createFolder(String path, String folderName, boolean local) throws IOException;
> It uses to create a file with full permissions.
> 4.   public void DaemonProtocol.createFolder(String path, String folderName, FsPermission permission, boolean local) throws IOException;
> It uses to create a folder with given permissions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9736-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 06:49:19 2010
Return-Path: <common-issues-return-9736-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67566 invoked from network); 26 Jul 2010 06:49:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 06:49:19 -0000
Received: (qmail 59829 invoked by uid 500); 26 Jul 2010 06:49:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58616 invoked by uid 500); 26 Jul 2010 06:49:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58600 invoked by uid 99); 26 Jul 2010 06:49:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 06:49:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 06:49:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6Q6mo39015208
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 06:48:50 GMT
Message-ID: <6168171.580591280126930054.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 02:48:50 -0400 (EDT)
From: "huang jian (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6880) Documentation: Chinese (cn) doc is
 old
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Documentation: Chinese (cn) doc is old
--------------------------------------

                 Key: HADOOP-6880
                 URL: https://issues.apache.org/jira/browse/HADOOP-6880
             Project: Hadoop Common
          Issue Type: Bug
          Components: documentation
    Affects Versions: 0.20.2
            Reporter: huang jian


Documentation: Chinese (cn) doc is old.
http://hadoop.apache.org/common/docs/r0.20.2/quickstart.html
JavaTM 1.6.x, preferably from Sun, must be installed.=20
Configuration

Use the following:
conf/core-site.xml:
<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>


conf/hdfs-site.xml:
<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>


conf/mapred-site.xml:
<configuration>
  <property>
    <name>mapred.job.tracker</name>
    <value>localhost:9001</value>
  </property>
</configuration>

http://hadoop.apache.org/common/docs/r0.20.2/cn/quickstart.html
JavaTM1.5.x=EF=BC=8C=E5=BF=85=E9=A1=BB=E5=AE=89=E8=A3=85=EF=BC=8C=E5=BB=BA=
=E8=AE=AE=E9=80=89=E6=8B=A9Sun=E5=85=AC=E5=8F=B8=E5=8F=91=E8=A1=8C=E7=9A=84=
Java=E7=89=88=E6=9C=AC=E3=80=82=20
=E9=85=8D=E7=BD=AE

=E4=BD=BF=E7=94=A8=E5=A6=82=E4=B8=8B=E7=9A=84 conf/hadoop-site.xml:
<configuration>
  <property>
    <name>fs.default.name</name>
    <value>localhost:9000</value>
  </property>
  <property>
    <name>mapred.job.tracker</name>
    <value>localhost:9001</value>
  </property>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9737-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 13:45:15 2010
Return-Path: <common-issues-return-9737-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73680 invoked from network); 26 Jul 2010 13:45:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 13:45:15 -0000
Received: (qmail 74836 invoked by uid 500); 26 Jul 2010 13:45:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74700 invoked by uid 500); 26 Jul 2010 13:45:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74692 invoked by uid 99); 26 Jul 2010 13:45:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 13:45:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 13:45:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QDiouO018970
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 13:44:50 GMT
Message-ID: <3566887.586381280151890367.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 09:44:50 -0400 (EDT)
From: "Iyappan Srinivasan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6879) Jsch jar is required for ivy
 resolution to be included in the build
In-Reply-To: <11760779.565921279956469755.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6879?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Iyappan Srinivasan updated HADOOP-6879:
---------------------------------------

    Attachment: 6879-ydist-security-patch.txt

patch for 0.20.1.xxx

> Jsch jar is required for ivy resolution to be included in the build
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6879
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6879
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.1
>            Reporter: Iyappan Srinivasan
>            Assignee: Iyappan Srinivasan
>         Attachments: 6879-ydist-security-patch.txt
>
>
> http://mvnrepository.com/
> com.jcraft =C2=BB jsch=20
> 0.1.42 version needs to be included in the build. This is  required for h=
erriot testcases using jsch.
> Please include this in ivy.
> jsch is originally located in http://www.jcraft.com/jsch/

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9738-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 16:55:42 2010
Return-Path: <common-issues-return-9738-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99907 invoked from network); 26 Jul 2010 16:55:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 16:55:41 -0000
Received: (qmail 5086 invoked by uid 500); 26 Jul 2010 16:55:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5046 invoked by uid 500); 26 Jul 2010 16:55:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5030 invoked by uid 99); 26 Jul 2010 16:55:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 16:55:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 16:55:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QGtGtr020750
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 16:55:17 GMT
Message-ID: <33436071.191280163316580.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 12:55:16 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6877) Common part of HDFS-1178
In-Reply-To: <10223047.556651279918070153.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6877:
--------------------------------

    Status: Open  (was: Patch Available)

> Common part of HDFS-1178
> ------------------------
>
>                 Key: HADOOP-6877
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6877
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6877-01.patch
>
>
> This is the Common part of HDFS-1178.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9739-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 16:55:42 2010
Return-Path: <common-issues-return-9739-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99967 invoked from network); 26 Jul 2010 16:55:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 16:55:42 -0000
Received: (qmail 5281 invoked by uid 500); 26 Jul 2010 16:55:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5057 invoked by uid 500); 26 Jul 2010 16:55:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5037 invoked by uid 99); 26 Jul 2010 16:55:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 16:55:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 16:55:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QGtHdW020756
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 16:55:17 GMT
Message-ID: <10962779.211280163317227.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 12:55:17 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6877) Common part of HDFS-1178
In-Reply-To: <10223047.556651279918070153.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6877:
--------------------------------

    Status: Patch Available  (was: Open)

> Common part of HDFS-1178
> ------------------------
>
>                 Key: HADOOP-6877
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6877
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6877-01.patch
>
>
> This is the Common part of HDFS-1178.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9740-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 17:20:48 2010
Return-Path: <common-issues-return-9740-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9721 invoked from network); 26 Jul 2010 17:20:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 17:20:48 -0000
Received: (qmail 40927 invoked by uid 500); 26 Jul 2010 17:20:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40873 invoked by uid 500); 26 Jul 2010 17:20:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40865 invoked by uid 99); 26 Jul 2010 17:20:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 17:20:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 17:20:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QHKMrM021129
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 17:20:22 GMT
Message-ID: <19905883.1011280164822873.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 13:20:22 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6877) Common part of HDFS-1178
In-Reply-To: <10223047.556651279918070153.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892377#action_12892377 ] 

Hadoop QA commented on HADOOP-6877:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12450355/c6877-01.patch
  against trunk revision 967220.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/633/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/633/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/633/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/633/console

This message is automatically generated.

> Common part of HDFS-1178
> ------------------------
>
>                 Key: HADOOP-6877
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6877
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6877-01.patch
>
>
> This is the Common part of HDFS-1178.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9741-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 17:38:41 2010
Return-Path: <common-issues-return-9741-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15701 invoked from network); 26 Jul 2010 17:38:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 17:38:41 -0000
Received: (qmail 77005 invoked by uid 500); 26 Jul 2010 17:38:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76947 invoked by uid 500); 26 Jul 2010 17:38:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76939 invoked by uid 99); 26 Jul 2010 17:38:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 17:38:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 17:38:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QHcGLl021340
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 17:38:16 GMT
Message-ID: <32256795.1551280165896706.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 13:38:16 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6877) Common part of HDFS-1178
In-Reply-To: <10223047.556651279918070153.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892381#action_12892381 ] 

Kan Zhang commented on HADOOP-6877:
-----------------------------------

The javadoc warnings are unrelated. No new test needed since this patch simply removes a utility method.

> Common part of HDFS-1178
> ------------------------
>
>                 Key: HADOOP-6877
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6877
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6877-01.patch
>
>
> This is the Common part of HDFS-1178.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9742-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 17:45:41 2010
Return-Path: <common-issues-return-9742-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17724 invoked from network); 26 Jul 2010 17:45:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 17:45:41 -0000
Received: (qmail 87334 invoked by uid 500); 26 Jul 2010 17:45:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87293 invoked by uid 500); 26 Jul 2010 17:45:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87285 invoked by uid 99); 26 Jul 2010 17:45:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 17:45:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 17:45:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QHjJk0021424
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 17:45:19 GMT
Message-ID: <26420239.1761280166319351.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 13:45:19 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6870) Add FileSystem#listLocatedStatus to
 list a directory's content together with each file's block locations
In-Reply-To: <13042840.506371279749289722.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892387#action_12892387 ] 

Doug Cutting commented on HADOOP-6870:
--------------------------------------

The next() method in the iterator for FileSystem#listStatus() still returns a LocatedFileStatus when getFileBlockLocations() throws an exception.  I think throwing an exception here would be better.  The list() method in that class also still swallows exceptions that should probably be thrown.

> Unit tests for recursive listing of symbolic links will be added after HADOOP-6872 is fixed.

Can't we test this independently?  There are other symlink-based unit tests independent of HDFS.


> Add FileSystem#listLocatedStatus to list a directory's content together with each file's block locations
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6870
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6870
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listFiles.patch, listFiles1.patch
>
>
> This jira implements the new FileSystem API as proposed in HDFS-202. The new API aims to eliminate individual "getFileBlockLocations" calls to NN for each file in the input directory of a job. Instead, a file's block locations are returned together with FileStatus when listing a directory, thus improving getSplits performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9743-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 17:55:38 2010
Return-Path: <common-issues-return-9743-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19633 invoked from network); 26 Jul 2010 17:55:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 17:55:38 -0000
Received: (qmail 97615 invoked by uid 500); 26 Jul 2010 17:55:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97481 invoked by uid 500); 26 Jul 2010 17:55:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97469 invoked by uid 99); 26 Jul 2010 17:55:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 17:55:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 17:55:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QHtG01021574
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 17:55:16 GMT
Message-ID: <3189526.2111280166916543.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 13:55:16 -0400 (EDT)
From: "Arvind Prabhakar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6871) When the value of a configuration
 key is set to its unresolved form, it causes the IllegalStateException in
 Configuration.get() stating that substitution depth is too large.
In-Reply-To: <4673721.506611279750129934.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892390#action_12892390 ] 

Arvind Prabhakar commented on HADOOP-6871:
------------------------------------------

I am not very sure what exactly is causing the Javadoc warning in the automated build. Can anyone comment/clarify what exactly is causing this issue? 

> When the value of a configuration key is set to its unresolved form, it causes the IllegalStateException in Configuration.get() stating that substitution depth is too large.
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6871
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6871
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>            Reporter: Arvind Prabhakar
>         Attachments: HADOOP-6871-1.patch, HADOOP-6871.patch
>
>
> When a configuration value is set to its unresolved expression string, it leads to recursive substitution attempts in {{Configuration.substituteVars(String)}} method until the max substitution check kicks in and raises an IllegalStateException indicating that the substitution depth is too large. For example, the configuration key "{{foobar}}" with a value set to "{{$\{foobar\}}}" will cause this behavior. 
> While this is not a usual use case, it can happen in build environments where a property value is not specified and yet being passed into the test mechanism leading to failures due to this limitation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9744-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 17:55:42 2010
Return-Path: <common-issues-return-9744-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19695 invoked from network); 26 Jul 2010 17:55:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 17:55:41 -0000
Received: (qmail 97687 invoked by uid 500); 26 Jul 2010 17:55:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97643 invoked by uid 500); 26 Jul 2010 17:55:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97635 invoked by uid 99); 26 Jul 2010 17:55:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 17:55:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 17:55:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QHtHvm021580
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 17:55:17 GMT
Message-ID: <2139731.2131280166917408.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 13:55:17 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6877) Common part of HDFS-1178
In-Reply-To: <10223047.556651279918070153.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6877:
--------------------------------

           Status: Resolved  (was: Patch Available)
    Fix Version/s: 0.22.0
       Resolution: Fixed

I've committed this.  Resolving as fixed.  Thanks, Kan.

> Common part of HDFS-1178
> ------------------------
>
>                 Key: HADOOP-6877
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6877
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>             Fix For: 0.22.0
>
>         Attachments: c6877-01.patch
>
>
> This is the Common part of HDFS-1178.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9745-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 17:59:38 2010
Return-Path: <common-issues-return-9745-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20249 invoked from network); 26 Jul 2010 17:59:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 17:59:38 -0000
Received: (qmail 2125 invoked by uid 500); 26 Jul 2010 17:59:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2067 invoked by uid 500); 26 Jul 2010 17:59:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2059 invoked by uid 99); 26 Jul 2010 17:59:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 17:59:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 17:59:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QHxGHF021673
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 17:59:16 GMT
Message-ID: <4581419.2211280167156575.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 13:59:16 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6877) Common part of HDFS-1178
In-Reply-To: <10223047.556651279918070153.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6877:
--------------------------------

           Issue Type: Improvement  (was: Bug)
    Affects Version/s: 0.22.0
          Component/s: ipc

> Common part of HDFS-1178
> ------------------------
>
>                 Key: HADOOP-6877
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6877
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>             Fix For: 0.22.0
>
>         Attachments: c6877-01.patch
>
>
> This is the Common part of HDFS-1178.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9746-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 18:53:38 2010
Return-Path: <common-issues-return-9746-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49607 invoked from network); 26 Jul 2010 18:53:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 18:53:38 -0000
Received: (qmail 91038 invoked by uid 500); 26 Jul 2010 18:53:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91004 invoked by uid 500); 26 Jul 2010 18:53:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90996 invoked by uid 99); 26 Jul 2010 18:53:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 18:53:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 18:53:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QIrG2Y022151
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 18:53:16 GMT
Message-ID: <28363686.3111280170396575.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 14:53:16 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6881) The efficient comparators aren't
 always used except for BytesWritable and Text
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

The efficient comparators aren't always used except for BytesWritable and Text
------------------------------------------------------------------------------

                 Key: HADOOP-6881
                 URL: https://issues.apache.org/jira/browse/HADOOP-6881
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.20.0
            Reporter: Owen O'Malley
            Assignee: Owen O'Malley


When we moved from Java 4 to Java 5 (and then 6), there was a change in the JVM semantics such that references to a class such as IntWritable.class no longer forces initialization. Since all of the Writables depend on their class static blocks to register their fast comparators, that can happen *after* we look up the comparator. In that case, the framework will fall back to the generic comparator that deserializes both keys and does the object compare, which may cause a huge slow down in the sort.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9747-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 19:26:47 2010
Return-Path: <common-issues-return-9747-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60213 invoked from network); 26 Jul 2010 19:26:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 19:26:47 -0000
Received: (qmail 34484 invoked by uid 500); 26 Jul 2010 19:26:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34386 invoked by uid 500); 26 Jul 2010 19:26:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34378 invoked by uid 99); 26 Jul 2010 19:26:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 19:26:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 19:26:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QJQM6O022562
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 19:26:22 GMT
Message-ID: <4075835.3761280172382113.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 15:26:22 -0400 (EDT)
From: "Nicholas Carlini (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicholas Carlini updated HADOOP-6837:
-------------------------------------

    Attachment: hadoop-6349-2.patch

Attached an update patch.

Fixed the checksum mismatch. It was possible for the decompressor to run out of input after reading the header bytes but not notice if the block ID was 1. So if there were fewer than 26 bytes in the input (but more than 16) and the byte ID was 1 then it wouldn't notice and just use whatever happened to be in the buffer at the time.

Fixed a bug in the decompressor where it would incorrectly indicate it was finished if at the end of decompressing a block there was no more input left to decompress and decompress() was then called again (TestCodec seed 1333275328, 2011623221, -1402938700 or -1990280158; generate 50,000 records). Actually, the decompressor never returns finished now. This is because the only time the decompressor should return true is if it somehow knows the end of the stream has been reached and it doesn't, it just guesses that if it has read all the bytes it currently has then it's done, which is not the case.

Implemented getRemaining().

Removed iOff from both the compressor and decompressor. It was initialized to zero from the start and was only ever modified after that by setting it to 0.

Modified TestCodec to accept a seed as an argument.

Removed the rest of the carriage returns.


I will be adding a native version over the next few days and will upload that patch when it's done.

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: hadoop-6349-2.patch, HADOOP-6837-lzma-1-20100722.patch, HADOOP-6837-lzma-c-20100719.patch, HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9748-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 19:30:44 2010
Return-Path: <common-issues-return-9748-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60905 invoked from network); 26 Jul 2010 19:30:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 19:30:44 -0000
Received: (qmail 35852 invoked by uid 500); 26 Jul 2010 19:30:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35828 invoked by uid 500); 26 Jul 2010 19:30:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35820 invoked by uid 99); 26 Jul 2010 19:30:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 19:30:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 19:30:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QJUJMv022609
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 19:30:19 GMT
Message-ID: <28660369.3881280172619558.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 15:30:19 -0400 (EDT)
From: "Nicholas Carlini (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892425#action_12892425 ] 

Nicholas Carlini commented on HADOOP-6837:
------------------------------------------

... that was supposed to go on HADOOP-6349, not here. Ignore that.

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: hadoop-6349-2.patch, HADOOP-6837-lzma-1-20100722.patch, HADOOP-6837-lzma-c-20100719.patch, HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9749-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 19:32:49 2010
Return-Path: <common-issues-return-9749-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61430 invoked from network); 26 Jul 2010 19:32:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 19:32:49 -0000
Received: (qmail 37212 invoked by uid 500); 26 Jul 2010 19:32:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37170 invoked by uid 500); 26 Jul 2010 19:32:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37162 invoked by uid 99); 26 Jul 2010 19:32:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 19:32:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 19:32:48 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QJWSXa022742
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 19:32:28 GMT
Message-ID: <1796594.4321280172748466.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 15:32:28 -0400 (EDT)
From: "Nicholas Carlini (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicholas Carlini updated HADOOP-6837:
-------------------------------------

    Attachment:     (was: hadoop-6349-2.patch)

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-1-20100722.patch, HADOOP-6837-lzma-c-20100719.patch, HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9750-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 19:32:50 2010
Return-Path: <common-issues-return-9750-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61471 invoked from network); 26 Jul 2010 19:32:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 19:32:49 -0000
Received: (qmail 37367 invoked by uid 500); 26 Jul 2010 19:32:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37326 invoked by uid 500); 26 Jul 2010 19:32:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37318 invoked by uid 99); 26 Jul 2010 19:32:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 19:32:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 19:32:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QJWPSV022706
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 19:32:25 GMT
Message-ID: <14966125.4201280172745123.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 15:32:25 -0400 (EDT)
From: "Nicholas Carlini (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6349) Implement FastLZCodec for fastlz/lzo
 algorithm
In-Reply-To: <289817036.1256941439448.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicholas Carlini updated HADOOP-6349:
-------------------------------------

    Attachment: hadoop-6349-2.patch

Attached an update patch.

Fixed the checksum mismatch. It was possible for the decompressor to run out of input after reading the header bytes but not notice if the block ID was 1. So if there were fewer than 26 bytes in the input (but more than 16) and the byte ID was 1 then it wouldn't notice and just use whatever happened to be in the buffer at the time.

Fixed a bug in the decompressor where it would incorrectly indicate it was finished if at the end of decompressing a block there was no more input left to decompress and decompress() was then called again (TestCodec seed 1333275328, 2011623221, -1402938700 or -1990280158; generate 50,000 records). Actually, the decompressor never returns finished now. This is because the only time the decompressor should return true is if it somehow knows the end of the stream has been reached and it doesn't, it just guesses that if it has read all the bytes it currently has then it's done, which is not the case.

Implemented getRemaining().

Removed iOff from both the compressor and decompressor. It was initialized to zero from the start and was only ever modified after that by setting it to 0.

Modified TestCodec to accept a seed as an argument.

Removed the rest of the carriage returns.

I will be adding a native version over the next few days and will upload that patch when it's done.

> Implement FastLZCodec for fastlz/lzo algorithm
> ----------------------------------------------
>
>                 Key: HADOOP-6349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6349
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: William Kinney
>         Attachments: hadoop-6349-1.patch, hadoop-6349-2.patch, HADOOP-6349-TestFastLZCodec.patch, HADOOP-6349.patch, TestCodecPerformance.java, TestCodecPerformance.java, testCodecPerfResults.tsv
>
>
> Per  [HADOOP-4874|http://issues.apache.org/jira/browse/HADOOP-4874], FastLZ is a good (speed, license) alternative to LZO. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9751-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 20:20:54 2010
Return-Path: <common-issues-return-9751-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79679 invoked from network); 26 Jul 2010 20:20:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 20:20:53 -0000
Received: (qmail 7551 invoked by uid 500); 26 Jul 2010 20:20:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7478 invoked by uid 500); 26 Jul 2010 20:20:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7468 invoked by uid 99); 26 Jul 2010 20:20:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 20:20:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 20:20:50 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QKKTtg023377
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 20:20:29 GMT
Message-ID: <17503629.5861280175629204.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 16:20:29 -0400 (EDT)
From: "Nicholas Carlini (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6349) Implement FastLZCodec for fastlz/lzo
 algorithm
In-Reply-To: <289817036.1256941439448.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicholas Carlini updated HADOOP-6349:
-------------------------------------

    Attachment:     (was: hadoop-6349-2.patch)

> Implement FastLZCodec for fastlz/lzo algorithm
> ----------------------------------------------
>
>                 Key: HADOOP-6349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6349
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: William Kinney
>         Attachments: hadoop-6349-1.patch, hadoop-6349-2.patch, HADOOP-6349-TestFastLZCodec.patch, HADOOP-6349.patch, TestCodecPerformance.java, TestCodecPerformance.java, testCodecPerfResults.tsv
>
>
> Per  [HADOOP-4874|http://issues.apache.org/jira/browse/HADOOP-4874], FastLZ is a good (speed, license) alternative to LZO. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9752-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 20:20:55 2010
Return-Path: <common-issues-return-9752-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79732 invoked from network); 26 Jul 2010 20:20:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 20:20:54 -0000
Received: (qmail 7683 invoked by uid 500); 26 Jul 2010 20:20:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7639 invoked by uid 500); 26 Jul 2010 20:20:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7526 invoked by uid 99); 26 Jul 2010 20:20:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 20:20:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 20:20:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QKKXSt023464
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 20:20:33 GMT
Message-ID: <22590918.6151280175633331.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 16:20:33 -0400 (EDT)
From: "Nicholas Carlini (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6349) Implement FastLZCodec for fastlz/lzo
 algorithm
In-Reply-To: <289817036.1256941439448.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicholas Carlini updated HADOOP-6349:
-------------------------------------

    Attachment: hadoop-6349-2.patch

> Implement FastLZCodec for fastlz/lzo algorithm
> ----------------------------------------------
>
>                 Key: HADOOP-6349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6349
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: William Kinney
>         Attachments: hadoop-6349-1.patch, hadoop-6349-2.patch, HADOOP-6349-TestFastLZCodec.patch, HADOOP-6349.patch, TestCodecPerformance.java, TestCodecPerformance.java, testCodecPerfResults.tsv
>
>
> Per  [HADOOP-4874|http://issues.apache.org/jira/browse/HADOOP-4874], FastLZ is a good (speed, license) alternative to LZO. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9753-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 21:09:42 2010
Return-Path: <common-issues-return-9753-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1407 invoked from network); 26 Jul 2010 21:09:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 21:09:41 -0000
Received: (qmail 78457 invoked by uid 500); 26 Jul 2010 21:09:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78408 invoked by uid 500); 26 Jul 2010 21:09:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78400 invoked by uid 99); 26 Jul 2010 21:09:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 21:09:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 21:09:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QL9G98023955
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 21:09:17 GMT
Message-ID: <20731804.7141280178556900.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 17:09:16 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6881) The efficient comparators aren't
 always used except for BytesWritable and Text
In-Reply-To: <28363686.3111280170396575.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6881:
----------------------------------

    Attachment: h-6881.patch

This patch does two things:
  1. If a comparator is not defined, forces the class to be initialized by the class loader.
  2. cache the comparator so that we reuse the comparator even if it is the generic one.

> The efficient comparators aren't always used except for BytesWritable and Text
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6881
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6881
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.0
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: h-6881.patch
>
>
> When we moved from Java 4 to Java 5 (and then 6), there was a change in the JVM semantics such that references to a class such as IntWritable.class no longer forces initialization. Since all of the Writables depend on their class static blocks to register their fast comparators, that can happen *after* we look up the comparator. In that case, the framework will fall back to the generic comparator that deserializes both keys and does the object compare, which may cause a huge slow down in the sort.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9754-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 21:21:39 2010
Return-Path: <common-issues-return-9754-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6649 invoked from network); 26 Jul 2010 21:21:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 21:21:39 -0000
Received: (qmail 94770 invoked by uid 500); 26 Jul 2010 21:21:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94703 invoked by uid 500); 26 Jul 2010 21:21:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94695 invoked by uid 99); 26 Jul 2010 21:21:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 21:21:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 21:21:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QLLHbb024069
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 21:21:17 GMT
Message-ID: <22321305.7421280179277547.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 17:21:17 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6881) The efficient comparators aren't
 always used except for BytesWritable and Text
In-Reply-To: <28363686.3111280170396575.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6881:
----------------------------------

           Status: Patch Available  (was: Open)
    Fix Version/s: 0.20.3

> The efficient comparators aren't always used except for BytesWritable and Text
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6881
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6881
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.0
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.3
>
>         Attachments: h-6881.patch
>
>
> When we moved from Java 4 to Java 5 (and then 6), there was a change in the JVM semantics such that references to a class such as IntWritable.class no longer forces initialization. Since all of the Writables depend on their class static blocks to register their fast comparators, that can happen *after* we look up the comparator. In that case, the framework will fall back to the generic comparator that deserializes both keys and does the object compare, which may cause a huge slow down in the sort.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9755-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 21:36:42 2010
Return-Path: <common-issues-return-9755-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14324 invoked from network); 26 Jul 2010 21:36:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 21:36:42 -0000
Received: (qmail 20690 invoked by uid 500); 26 Jul 2010 21:36:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20562 invoked by uid 500); 26 Jul 2010 21:36:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20554 invoked by uid 99); 26 Jul 2010 21:36:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 21:36:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 21:36:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QLaGFn024288
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 21:36:17 GMT
Message-ID: <8190268.7931280180176915.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 17:36:16 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6881) The efficient comparators aren't
 always used except for BytesWritable and Text
In-Reply-To: <28363686.3111280170396575.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-6881:
---------------------------------

    Attachment: HADOOP-6881.patch

Here's a version of the patch that adds a unit test.  This test fails for me without the patch.

> The efficient comparators aren't always used except for BytesWritable and Text
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6881
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6881
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.0
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.3
>
>         Attachments: h-6881.patch, HADOOP-6881.patch
>
>
> When we moved from Java 4 to Java 5 (and then 6), there was a change in the JVM semantics such that references to a class such as IntWritable.class no longer forces initialization. Since all of the Writables depend on their class static blocks to register their fast comparators, that can happen *after* we look up the comparator. In that case, the framework will fall back to the generic comparator that deserializes both keys and does the object compare, which may cause a huge slow down in the sort.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9756-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 21:42:39 2010
Return-Path: <common-issues-return-9756-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16527 invoked from network); 26 Jul 2010 21:42:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 21:42:38 -0000
Received: (qmail 35373 invoked by uid 500); 26 Jul 2010 21:42:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35201 invoked by uid 500); 26 Jul 2010 21:42:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35188 invoked by uid 99); 26 Jul 2010 21:42:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 21:42:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 21:42:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QLgGYK024403
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 21:42:16 GMT
Message-ID: <7355803.8251280180536685.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 17:42:16 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6881) The efficient comparators aren't
 always used except for BytesWritable and Text
In-Reply-To: <28363686.3111280170396575.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-6881:
---------------------------------

    Attachment: HADOOP-6881.patch

Slightly cleaned up the test code.

> The efficient comparators aren't always used except for BytesWritable and Text
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6881
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6881
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.0
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.3
>
>         Attachments: h-6881.patch, HADOOP-6881.patch, HADOOP-6881.patch
>
>
> When we moved from Java 4 to Java 5 (and then 6), there was a change in the JVM semantics such that references to a class such as IntWritable.class no longer forces initialization. Since all of the Writables depend on their class static blocks to register their fast comparators, that can happen *after* we look up the comparator. In that case, the framework will fall back to the generic comparator that deserializes both keys and does the object compare, which may cause a huge slow down in the sort.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9757-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 21:52:39 2010
Return-Path: <common-issues-return-9757-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19439 invoked from network); 26 Jul 2010 21:52:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 21:52:39 -0000
Received: (qmail 47557 invoked by uid 500); 26 Jul 2010 21:52:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47433 invoked by uid 500); 26 Jul 2010 21:52:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47421 invoked by uid 99); 26 Jul 2010 21:52:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 21:52:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 21:52:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QLqHAE024532
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 21:52:17 GMT
Message-ID: <25801973.8551280181137361.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 17:52:17 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6881) The efficient comparators aren't
 always used except for BytesWritable and Text
In-Reply-To: <28363686.3111280170396575.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892488#action_12892488 ] 

Hadoop QA commented on HADOOP-6881:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12450519/h-6881.patch
  against trunk revision 979387.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/634/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/634/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/634/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/634/console

This message is automatically generated.

> The efficient comparators aren't always used except for BytesWritable and Text
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6881
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6881
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.0
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.3
>
>         Attachments: h-6881.patch, HADOOP-6881.patch, HADOOP-6881.patch
>
>
> When we moved from Java 4 to Java 5 (and then 6), there was a change in the JVM semantics such that references to a class such as IntWritable.class no longer forces initialization. Since all of the Writables depend on their class static blocks to register their fast comparators, that can happen *after* we look up the comparator. In that case, the framework will fall back to the generic comparator that deserializes both keys and does the object compare, which may cause a huge slow down in the sort.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9758-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 22:01:49 2010
Return-Path: <common-issues-return-9758-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21789 invoked from network); 26 Jul 2010 22:01:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 22:01:48 -0000
Received: (qmail 56971 invoked by uid 500); 26 Jul 2010 22:01:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56909 invoked by uid 500); 26 Jul 2010 22:01:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56900 invoked by uid 99); 26 Jul 2010 22:01:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 22:01:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 22:01:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QM1Q38024826
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 22:01:27 GMT
Message-ID: <14932268.9241280181686974.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 18:01:26 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6870) Add FileSystem#listLocatedStatus to
 list a directory's content together with each file's block locations
In-Reply-To: <13042840.506371279749289722.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6870:
----------------------------------

    Attachment: listFiles2.patch

"listFiles2.patch" makes FileSystem to throw a RuntimeException when getBlockLocations throws an exception.

> Add FileSystem#listLocatedStatus to list a directory's content together with each file's block locations
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6870
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6870
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listFiles.patch, listFiles1.patch, listFiles2.patch
>
>
> This jira implements the new FileSystem API as proposed in HDFS-202. The new API aims to eliminate individual "getFileBlockLocations" calls to NN for each file in the input directory of a job. Instead, a file's block locations are returned together with FileStatus when listing a directory, thus improving getSplits performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9759-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 22:01:51 2010
Return-Path: <common-issues-return-9759-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21875 invoked from network); 26 Jul 2010 22:01:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 22:01:51 -0000
Received: (qmail 57321 invoked by uid 500); 26 Jul 2010 22:01:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57274 invoked by uid 500); 26 Jul 2010 22:01:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57108 invoked by uid 99); 26 Jul 2010 22:01:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 22:01:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 22:01:50 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QM1Uu8024859
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 22:01:30 GMT
Message-ID: <7892286.9351280181690371.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 18:01:30 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6870) Add FileSystem#listLocatedStatus to
 list a directory's content together with each file's block locations
In-Reply-To: <13042840.506371279749289722.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6870:
----------------------------------

    Status: Patch Available  (was: Open)

> Add FileSystem#listLocatedStatus to list a directory's content together with each file's block locations
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6870
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6870
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listFiles.patch, listFiles1.patch, listFiles2.patch
>
>
> This jira implements the new FileSystem API as proposed in HDFS-202. The new API aims to eliminate individual "getFileBlockLocations" calls to NN for each file in the input directory of a job. Instead, a file's block locations are returned together with FileStatus when listing a directory, thus improving getSplits performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9760-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 22:05:41 2010
Return-Path: <common-issues-return-9760-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27490 invoked from network); 26 Jul 2010 22:05:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 22:05:41 -0000
Received: (qmail 65674 invoked by uid 500); 26 Jul 2010 22:05:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65618 invoked by uid 500); 26 Jul 2010 22:05:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65610 invoked by uid 99); 26 Jul 2010 22:05:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 22:05:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 22:05:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QM5Gux024911
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 22:05:16 GMT
Message-ID: <23540000.9391280181916248.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 18:05:16 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6882) Update the patch level of Jetty
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Update the patch level of Jetty
-------------------------------

                 Key: HADOOP-6882
                 URL: https://issues.apache.org/jira/browse/HADOOP-6882
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Owen O'Malley
            Assignee: Owen O'Malley


I'd like to move to a newer patch level of Jetty. 6.1.23 (instead of our current 6.1.14) has been suggested. As seen in http://svn.codehaus.org/jetty/jetty/branches/jetty-6.1/VERSION.txt, that represents 18 months of bug fixes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9761-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 22:14:42 2010
Return-Path: <common-issues-return-9761-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32350 invoked from network); 26 Jul 2010 22:14:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 22:14:42 -0000
Received: (qmail 85893 invoked by uid 500); 26 Jul 2010 22:14:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85783 invoked by uid 500); 26 Jul 2010 22:14:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85775 invoked by uid 99); 26 Jul 2010 22:14:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 22:14:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 22:14:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QMEH1S025011
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 22:14:17 GMT
Message-ID: <1404050.9641280182457017.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 18:14:17 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6881) The efficient comparators aren't
 always used except for BytesWritable and Text
In-Reply-To: <28363686.3111280170396575.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-6881:
---------------------------------

    Status: Open  (was: Patch Available)

> The efficient comparators aren't always used except for BytesWritable and Text
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6881
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6881
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.0
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.3
>
>         Attachments: h-6881.patch, HADOOP-6881.patch, HADOOP-6881.patch
>
>
> When we moved from Java 4 to Java 5 (and then 6), there was a change in the JVM semantics such that references to a class such as IntWritable.class no longer forces initialization. Since all of the Writables depend on their class static blocks to register their fast comparators, that can happen *after* we look up the comparator. In that case, the framework will fall back to the generic comparator that deserializes both keys and does the object compare, which may cause a huge slow down in the sort.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9762-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 22:14:43 2010
Return-Path: <common-issues-return-9762-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32401 invoked from network); 26 Jul 2010 22:14:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 22:14:42 -0000
Received: (qmail 85979 invoked by uid 500); 26 Jul 2010 22:14:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85833 invoked by uid 500); 26 Jul 2010 22:14:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85784 invoked by uid 99); 26 Jul 2010 22:14:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 22:14:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 22:14:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QMEILY025026
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 22:14:18 GMT
Message-ID: <27568516.9691280182458128.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 18:14:18 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6881) The efficient comparators aren't
 always used except for BytesWritable and Text
In-Reply-To: <28363686.3111280170396575.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-6881:
---------------------------------

    Status: Patch Available  (was: Open)

Get Hudson to try the test I added.

The javadoc warnings all seem unrelated, but rather are about use of proprietary Sun APIs and cannot be supressed.

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6476630


> The efficient comparators aren't always used except for BytesWritable and Text
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6881
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6881
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.0
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.3
>
>         Attachments: h-6881.patch, HADOOP-6881.patch, HADOOP-6881.patch
>
>
> When we moved from Java 4 to Java 5 (and then 6), there was a change in the JVM semantics such that references to a class such as IntWritable.class no longer forces initialization. Since all of the Writables depend on their class static blocks to register their fast comparators, that can happen *after* we look up the comparator. In that case, the framework will fall back to the generic comparator that deserializes both keys and does the object compare, which may cause a huge slow down in the sort.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9763-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 22:30:39 2010
Return-Path: <common-issues-return-9763-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42544 invoked from network); 26 Jul 2010 22:30:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 22:30:38 -0000
Received: (qmail 18594 invoked by uid 500); 26 Jul 2010 22:30:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18545 invoked by uid 500); 26 Jul 2010 22:30:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18520 invoked by uid 99); 26 Jul 2010 22:30:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 22:30:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 22:30:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QMUHHw025232
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 22:30:17 GMT
Message-ID: <17016987.10241280183417286.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 18:30:17 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6870) Add FileSystem#listLocatedStatus to
 list a directory's content together with each file's block locations
In-Reply-To: <13042840.506371279749289722.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892512#action_12892512 ] 

Hadoop QA commented on HADOOP-6870:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12450529/listFiles2.patch
  against trunk revision 979387.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 4 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs.  The patch appears to introduce 1 new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/635/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/635/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/635/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/635/console

This message is automatically generated.

> Add FileSystem#listLocatedStatus to list a directory's content together with each file's block locations
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6870
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6870
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listFiles.patch, listFiles1.patch, listFiles2.patch
>
>
> This jira implements the new FileSystem API as proposed in HDFS-202. The new API aims to eliminate individual "getFileBlockLocations" calls to NN for each file in the input directory of a job. Instead, a file's block locations are returned together with FileStatus when listing a directory, thus improving getSplits performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9764-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 22:48:39 2010
Return-Path: <common-issues-return-9764-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46052 invoked from network); 26 Jul 2010 22:48:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 22:48:39 -0000
Received: (qmail 32695 invoked by uid 500); 26 Jul 2010 22:48:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32643 invoked by uid 500); 26 Jul 2010 22:48:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32635 invoked by uid 99); 26 Jul 2010 22:48:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 22:48:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 22:48:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QMmIhX025471
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 22:48:18 GMT
Message-ID: <23263943.10871280184498108.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 18:48:18 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6881) The efficient comparators aren't
 always used except for BytesWritable and Text
In-Reply-To: <28363686.3111280170396575.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892521#action_12892521 ] 

Hadoop QA commented on HADOOP-6881:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12450525/HADOOP-6881.patch
  against trunk revision 979387.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/636/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/636/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/636/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/636/console

This message is automatically generated.

> The efficient comparators aren't always used except for BytesWritable and Text
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6881
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6881
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.0
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.3
>
>         Attachments: h-6881.patch, HADOOP-6881.patch, HADOOP-6881.patch
>
>
> When we moved from Java 4 to Java 5 (and then 6), there was a change in the JVM semantics such that references to a class such as IntWritable.class no longer forces initialization. Since all of the Writables depend on their class static blocks to register their fast comparators, that can happen *after* we look up the comparator. In that case, the framework will fall back to the generic comparator that deserializes both keys and does the object compare, which may cause a huge slow down in the sort.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9765-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 23:09:39 2010
Return-Path: <common-issues-return-9765-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54759 invoked from network); 26 Jul 2010 23:09:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 23:09:39 -0000
Received: (qmail 45602 invoked by uid 500); 26 Jul 2010 23:09:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45556 invoked by uid 500); 26 Jul 2010 23:09:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45548 invoked by uid 99); 26 Jul 2010 23:09:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 23:09:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 23:09:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QN9Hp5025696
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 23:09:17 GMT
Message-ID: <18974435.11291280185757416.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 19:09:17 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6706:
-----------------------------------------

    Attachment: HADOOP-6706.2.patch

Patch for trunk.
Incorporates HADOOP-6718 and HADOOP-6776

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706-bp20-2.patch, 6706.bp20.1.patch, 6706.bp20.patch, HADOOP-6706-BP20-fix1.patch, HADOOP-6706-BP20-fix2.patch, HADOOP-6706-BP20-fix3.patch, HADOOP-6706.2.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9766-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 23:09:43 2010
Return-Path: <common-issues-return-9766-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54877 invoked from network); 26 Jul 2010 23:09:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 23:09:43 -0000
Received: (qmail 45859 invoked by uid 500); 26 Jul 2010 23:09:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45818 invoked by uid 500); 26 Jul 2010 23:09:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45810 invoked by uid 99); 26 Jul 2010 23:09:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 23:09:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 23:09:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QN9JNm025714
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 23:09:19 GMT
Message-ID: <15253079.11351280185759180.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 19:09:19 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6706:
-----------------------------------------

    Status: Patch Available  (was: Open)

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706-bp20-2.patch, 6706.bp20.1.patch, 6706.bp20.patch, HADOOP-6706-BP20-fix1.patch, HADOOP-6706-BP20-fix2.patch, HADOOP-6706-BP20-fix3.patch, HADOOP-6706.2.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9767-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 23:11:42 2010
Return-Path: <common-issues-return-9767-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55766 invoked from network); 26 Jul 2010 23:11:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 23:11:39 -0000
Received: (qmail 48451 invoked by uid 500); 26 Jul 2010 23:11:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48413 invoked by uid 500); 26 Jul 2010 23:11:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48405 invoked by uid 99); 26 Jul 2010 23:11:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 23:11:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 23:11:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QNBIhH025739
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 23:11:18 GMT
Message-ID: <8103529.11431280185878447.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 19:11:18 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6870) Add FileSystem#listLocatedStatus to
 list a directory's content together with each file's block locations
In-Reply-To: <13042840.506371279749289722.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892533#action_12892533 ] 

Doug Cutting commented on HADOOP-6870:
--------------------------------------

LocatedFileStatus() still ignores an exception.  Even if it should never be thrown, throwing a RuntimeException is probably better than failing silently.

> Add FileSystem#listLocatedStatus to list a directory's content together with each file's block locations
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6870
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6870
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listFiles.patch, listFiles1.patch, listFiles2.patch
>
>
> This jira implements the new FileSystem API as proposed in HDFS-202. The new API aims to eliminate individual "getFileBlockLocations" calls to NN for each file in the input directory of a job. Instead, a file's block locations are returned together with FileStatus when listing a directory, thus improving getSplits performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9768-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 23:19:38 2010
Return-Path: <common-issues-return-9768-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59437 invoked from network); 26 Jul 2010 23:19:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 23:19:38 -0000
Received: (qmail 57778 invoked by uid 500); 26 Jul 2010 23:19:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57745 invoked by uid 500); 26 Jul 2010 23:19:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57737 invoked by uid 99); 26 Jul 2010 23:19:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 23:19:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 23:19:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QNJGcF025871
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 23:19:17 GMT
Message-ID: <17759837.11791280186356906.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 19:19:16 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6881) The efficient comparators aren't
 always used except for BytesWritable and Text
In-Reply-To: <28363686.3111280170396575.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892537#action_12892537 ] 

Doug Cutting commented on HADOOP-6881:
--------------------------------------

I just committed this.  Thanks, Owen!

> The efficient comparators aren't always used except for BytesWritable and Text
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6881
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6881
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.0
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.3
>
>         Attachments: h-6881.patch, HADOOP-6881.patch, HADOOP-6881.patch
>
>
> When we moved from Java 4 to Java 5 (and then 6), there was a change in the JVM semantics such that references to a class such as IntWritable.class no longer forces initialization. Since all of the Writables depend on their class static blocks to register their fast comparators, that can happen *after* we look up the comparator. In that case, the framework will fall back to the generic comparator that deserializes both keys and does the object compare, which may cause a huge slow down in the sort.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9769-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 23:31:39 2010
Return-Path: <common-issues-return-9769-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63949 invoked from network); 26 Jul 2010 23:31:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 23:31:38 -0000
Received: (qmail 63341 invoked by uid 500); 26 Jul 2010 23:31:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63285 invoked by uid 500); 26 Jul 2010 23:31:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63277 invoked by uid 99); 26 Jul 2010 23:31:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 23:31:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 23:31:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QNVGeb025926
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 23:31:17 GMT
Message-ID: <1765319.11921280187076595.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 19:31:16 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892540#action_12892540 ] 

Hadoop QA commented on HADOOP-6706:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12450536/HADOOP-6706.2.patch
  against trunk revision 979387.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/637/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/637/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/637/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/637/console

This message is automatically generated.

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706-bp20-2.patch, 6706.bp20.1.patch, 6706.bp20.patch, HADOOP-6706-BP20-fix1.patch, HADOOP-6706-BP20-fix2.patch, HADOOP-6706-BP20-fix3.patch, HADOOP-6706.2.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9770-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 26 23:55:42 2010
Return-Path: <common-issues-return-9770-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67206 invoked from network); 26 Jul 2010 23:55:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jul 2010 23:55:42 -0000
Received: (qmail 85024 invoked by uid 500); 26 Jul 2010 23:55:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84962 invoked by uid 500); 26 Jul 2010 23:55:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84953 invoked by uid 99); 26 Jul 2010 23:55:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 23:55:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 23:55:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6QNtHXB026119
	for <common-issues@hadoop.apache.org>; Mon, 26 Jul 2010 23:55:17 GMT
Message-ID: <2468887.12431280188517108.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 19:55:17 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892548#action_12892548 ] 

Jitendra Nath Pandey commented on HADOOP-6706:
----------------------------------------------

HADOOP-6706.2.patch is not for commit. It also includes the patch for HADOOP-6656.

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706-bp20-2.patch, 6706.bp20.1.patch, 6706.bp20.patch, HADOOP-6706-BP20-fix1.patch, HADOOP-6706-BP20-fix2.patch, HADOOP-6706-BP20-fix3.patch, HADOOP-6706.2.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9771-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 00:36:41 2010
Return-Path: <common-issues-return-9771-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91365 invoked from network); 27 Jul 2010 00:36:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 00:36:41 -0000
Received: (qmail 12150 invoked by uid 500); 27 Jul 2010 00:36:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11879 invoked by uid 500); 27 Jul 2010 00:36:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11871 invoked by uid 99); 27 Jul 2010 00:36:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 00:36:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 00:36:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6R0aGoq026403
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 00:36:16 GMT
Message-ID: <27281242.12981280190976761.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 20:36:16 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6870) Add FileSystem#listLocatedStatus to
 list a directory's content together with each file's block locations
In-Reply-To: <13042840.506371279749289722.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6870:
----------------------------------

    Attachment: listFiles3.patch

Doug, can not believe that you did not miss any exception! listFiles3.patch addressed your concern on getSymLink.

> Add FileSystem#listLocatedStatus to list a directory's content together with each file's block locations
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6870
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6870
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listFiles.patch, listFiles1.patch, listFiles2.patch, listFiles3.patch
>
>
> This jira implements the new FileSystem API as proposed in HDFS-202. The new API aims to eliminate individual "getFileBlockLocations" calls to NN for each file in the input directory of a job. Instead, a file's block locations are returned together with FileStatus when listing a directory, thus improving getSplits performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9772-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 01:16:47 2010
Return-Path: <common-issues-return-9772-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3770 invoked from network); 27 Jul 2010 01:16:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 01:16:47 -0000
Received: (qmail 43065 invoked by uid 500); 27 Jul 2010 01:16:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43005 invoked by uid 500); 27 Jul 2010 01:16:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42997 invoked by uid 99); 27 Jul 2010 01:16:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 01:16:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 01:16:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6R1GPwE026982
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 01:16:25 GMT
Message-ID: <24109754.14471280193385747.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 21:16:25 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5859) FindBugs : fix "wait() or sleep()
 with locks held" warnings in hdfs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-5859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892592#action_12892592 ] 

Todd Lipcon commented on HADOOP-5859:
-------------------------------------

Note that this patch doesn't address the root issue in most cases (I've seen confusion from several users on this front).

Usually the root issue here is that a thread gets stuck in processDatanodeError, particularly the RPC to the recovering DN. I've seen this in a number of situations, usually due to other IPC bugs that cause hung clients in various error scenarios, and sometimes due to bugs in recovery on the DN side.

So, if you see a stack trace that looks like this issue, it's probably a symptom of some other bug, and this patch would just rejigger the stack traces, but not actually solve the underlying problem.

Luke: do you have the stack trace from the DataStreamer thread? Is it stuck in an IPC call? If so, do you have the jstack from the recovery datanode?

> FindBugs : fix "wait() or sleep() with locks held" warnings in hdfs
> -------------------------------------------------------------------
>
>                 Key: HADOOP-5859
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5859
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>             Fix For: 0.21.0
>
>         Attachments: 5859-21.patch, 5859-22.patch, 5859-26.patch, 5859-33.patch, 5859-35.patch, 5859-36.patch, 5859-38.patch, 5859-4.patch, 5859-40.patch, 5859-41.patch, 5859-5.patch, 5859-8.patch
>
>
> This JIRA fixes the following warnings:
> SWL	org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal() calls Thread.sleep() with a lock held
> TLW	wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW	wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal()
> TLW	wait() with two locks held in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(byte[], int, int, byte[])

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9773-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 01:50:42 2010
Return-Path: <common-issues-return-9773-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10631 invoked from network); 27 Jul 2010 01:50:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 01:50:42 -0000
Received: (qmail 59835 invoked by uid 500); 27 Jul 2010 01:50:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59780 invoked by uid 500); 27 Jul 2010 01:50:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59772 invoked by uid 99); 27 Jul 2010 01:50:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 01:50:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 01:50:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6R1oHKr027259
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 01:50:17 GMT
Message-ID: <9621305.15081280195417111.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 21:50:17 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892605#action_12892605 ] 

Kan Zhang commented on HADOOP-6656:
-----------------------------------

getTGT() method should be replaced by or merged with SecurityUtil.getTgtFromSubject(). I don't think getTGT() handles cross-realm case.

Shouldn't User.setLastLogin() and User.getLastLogin() be synchronized methods? In current code, only synchronized methods in UGI use them, which is fine. But it's safer to synchronize at User class, and not relying on users of User class to synchronize. Same for other getters and setters in User.

hasSufficientTimeElapsed() has the side-effect of setting the last login time to now if it returns true, which is not intuitive to me.

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6656-trunk-1.patch, 6656-trunk-2.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9774-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 02:01:42 2010
Return-Path: <common-issues-return-9774-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11853 invoked from network); 27 Jul 2010 02:01:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 02:01:42 -0000
Received: (qmail 61909 invoked by uid 500); 27 Jul 2010 02:01:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61831 invoked by uid 500); 27 Jul 2010 02:01:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61821 invoked by uid 99); 27 Jul 2010 02:01:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 02:01:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 02:01:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6R21H2k027318
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 02:01:17 GMT
Message-ID: <26292337.15121280196077120.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 22:01:17 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892606#action_12892606 ] 

Kan Zhang commented on HADOOP-6656:
-----------------------------------

> I don't think getTGT() handles cross-realm case.
Sorry, it does. But I still recommend it to be merged with SecurityUtil.getTgtFromSubject().

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6656-trunk-1.patch, 6656-trunk-2.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9775-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 02:28:43 2010
Return-Path: <common-issues-return-9775-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21375 invoked from network); 27 Jul 2010 02:28:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 02:28:42 -0000
Received: (qmail 80829 invoked by uid 500); 27 Jul 2010 02:28:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80756 invoked by uid 500); 27 Jul 2010 02:28:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80748 invoked by uid 99); 27 Jul 2010 02:28:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 02:28:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 02:28:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6R2SHwS027613
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 02:28:18 GMT
Message-ID: <5648607.15751280197697765.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 22:28:17 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6870) Add FileSystem#listLocatedStatus to
 list a directory's content together with each file's block locations
In-Reply-To: <13042840.506371279749289722.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6870:
----------------------------------

    Status: Open  (was: Patch Available)

> Add FileSystem#listLocatedStatus to list a directory's content together with each file's block locations
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6870
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6870
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listFiles.patch, listFiles1.patch, listFiles2.patch, listFiles3.patch
>
>
> This jira implements the new FileSystem API as proposed in HDFS-202. The new API aims to eliminate individual "getFileBlockLocations" calls to NN for each file in the input directory of a job. Instead, a file's block locations are returned together with FileStatus when listing a directory, thus improving getSplits performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9776-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 02:28:43 2010
Return-Path: <common-issues-return-9776-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21391 invoked from network); 27 Jul 2010 02:28:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 02:28:42 -0000
Received: (qmail 81051 invoked by uid 500); 27 Jul 2010 02:28:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80916 invoked by uid 500); 27 Jul 2010 02:28:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80841 invoked by uid 99); 27 Jul 2010 02:28:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 02:28:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 02:28:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6R2SIak027628
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 02:28:18 GMT
Message-ID: <15151094.15801280197698709.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 22:28:18 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6870) Add FileSystem#listLocatedStatus to
 list a directory's content together with each file's block locations
In-Reply-To: <13042840.506371279749289722.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6870:
----------------------------------

    Status: Patch Available  (was: Open)

> Add FileSystem#listLocatedStatus to list a directory's content together with each file's block locations
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6870
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6870
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listFiles.patch, listFiles1.patch, listFiles2.patch, listFiles3.patch
>
>
> This jira implements the new FileSystem API as proposed in HDFS-202. The new API aims to eliminate individual "getFileBlockLocations" calls to NN for each file in the input directory of a job. Instead, a file's block locations are returned together with FileStatus when listing a directory, thus improving getSplits performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9777-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 02:51:40 2010
Return-Path: <common-issues-return-9777-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26659 invoked from network); 27 Jul 2010 02:51:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 02:51:40 -0000
Received: (qmail 93157 invoked by uid 500); 27 Jul 2010 02:51:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93023 invoked by uid 500); 27 Jul 2010 02:51:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93014 invoked by uid 99); 27 Jul 2010 02:51:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 02:51:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 02:51:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6R2pHr2027729
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 02:51:17 GMT
Message-ID: <7490828.15991280199077671.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 22:51:17 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6870) Add FileSystem#listLocatedStatus to
 list a directory's content together with each file's block locations
In-Reply-To: <13042840.506371279749289722.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892618#action_12892618 ] 

Hadoop QA commented on HADOOP-6870:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12450541/listFiles3.patch
  against trunk revision 979485.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 7 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/638/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/638/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/638/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/638/console

This message is automatically generated.

> Add FileSystem#listLocatedStatus to list a directory's content together with each file's block locations
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6870
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6870
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listFiles.patch, listFiles1.patch, listFiles2.patch, listFiles3.patch
>
>
> This jira implements the new FileSystem API as proposed in HDFS-202. The new API aims to eliminate individual "getFileBlockLocations" calls to NN for each file in the input directory of a job. Instead, a file's block locations are returned together with FileStatus when listing a directory, thus improving getSplits performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9778-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 04:54:41 2010
Return-Path: <common-issues-return-9778-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61420 invoked from network); 27 Jul 2010 04:54:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 04:54:41 -0000
Received: (qmail 47964 invoked by uid 500); 27 Jul 2010 04:54:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47824 invoked by uid 500); 27 Jul 2010 04:54:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47810 invoked by uid 99); 27 Jul 2010 04:54:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 04:54:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 04:54:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6R4sG6d028937
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 04:54:16 GMT
Message-ID: <7897109.17241280206456578.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 00:54:16 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6869) [Herriot] Implement a functionality
 for creating either file or folder in task attempt  folder while job is
 running.
In-Reply-To: <1398369.499011279732010273.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinay Kumar Thota updated HADOOP-6869:
--------------------------------------

    Attachment: 6869-ydist-security.patch
                HADOOP-6869.patch

Addressed cos comments.

> [Herriot] Implement a functionality for creating either file or folder in task attempt  folder while job is running.
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6869
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6869
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6869-ydist-security.patch, 6869-ydist-security.patch, HADOOP-6869.patch, HADOOP-6869.patch
>
>
> Functionality for creating either files or folders in task attempt folder while job is running. The functionality covers the following methods.
> 1. public void DaemonProtocol.createFile(String path, String fileName, boolean local) throws IOException; 
> It uses to create a file with full permissions.
> 2.   public void DaemonProtocol.createFile(String path, String fileName, FsPermission permission, boolean local) throws IOException; 
> It uses to create a file with given permissions.
> 3.   public void DaemonProtocol.createFolder(String path, String folderName, boolean local) throws IOException;
> It uses to create a file with full permissions.
> 4.   public void DaemonProtocol.createFolder(String path, String folderName, FsPermission permission, boolean local) throws IOException;
> It uses to create a folder with given permissions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9779-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 06:15:42 2010
Return-Path: <common-issues-return-9779-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86173 invoked from network); 27 Jul 2010 06:15:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 06:15:42 -0000
Received: (qmail 93438 invoked by uid 500); 27 Jul 2010 06:15:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93297 invoked by uid 500); 27 Jul 2010 06:15:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93277 invoked by uid 99); 27 Jul 2010 06:15:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 06:15:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 06:15:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6R6FJ4a029448
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 06:15:19 GMT
Message-ID: <26099452.17941280211319064.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 02:15:19 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6870) Add FileSystem#listLocatedStatus to
 list a directory's content together with each file's block locations
In-Reply-To: <13042840.506371279749289722.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6870:
----------------------------------

    Status: Open  (was: Patch Available)

> Add FileSystem#listLocatedStatus to list a directory's content together with each file's block locations
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6870
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6870
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listFiles.patch, listFiles1.patch, listFiles2.patch, listFiles3.patch, listFiles4.patch
>
>
> This jira implements the new FileSystem API as proposed in HDFS-202. The new API aims to eliminate individual "getFileBlockLocations" calls to NN for each file in the input directory of a job. Instead, a file's block locations are returned together with FileStatus when listing a directory, thus improving getSplits performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9780-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 06:15:44 2010
Return-Path: <common-issues-return-9780-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86226 invoked from network); 27 Jul 2010 06:15:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 06:15:43 -0000
Received: (qmail 93502 invoked by uid 500); 27 Jul 2010 06:15:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93336 invoked by uid 500); 27 Jul 2010 06:15:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93298 invoked by uid 99); 27 Jul 2010 06:15:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 06:15:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 06:15:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6R6FHMi029433
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 06:15:17 GMT
Message-ID: <8196138.17891280211317531.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 02:15:17 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6870) Add FileSystem#listLocatedStatus to
 list a directory's content together with each file's block locations
In-Reply-To: <13042840.506371279749289722.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6870:
----------------------------------

    Attachment: listFiles4.patch

"listFiles4.patch" fixed javadoc warnings & the failed unit test.

> Add FileSystem#listLocatedStatus to list a directory's content together with each file's block locations
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6870
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6870
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listFiles.patch, listFiles1.patch, listFiles2.patch, listFiles3.patch, listFiles4.patch
>
>
> This jira implements the new FileSystem API as proposed in HDFS-202. The new API aims to eliminate individual "getFileBlockLocations" calls to NN for each file in the input directory of a job. Instead, a file's block locations are returned together with FileStatus when listing a directory, thus improving getSplits performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9781-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 06:15:44 2010
Return-Path: <common-issues-return-9781-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86272 invoked from network); 27 Jul 2010 06:15:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 06:15:44 -0000
Received: (qmail 93635 invoked by uid 500); 27 Jul 2010 06:15:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93471 invoked by uid 500); 27 Jul 2010 06:15:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93454 invoked by uid 99); 27 Jul 2010 06:15:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 06:15:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 06:15:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6R6FKYX029463
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 06:15:20 GMT
Message-ID: <28110071.17991280211320093.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 02:15:20 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6870) Add FileSystem#listLocatedStatus to
 list a directory's content together with each file's block locations
In-Reply-To: <13042840.506371279749289722.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6870:
----------------------------------

    Status: Patch Available  (was: Open)

> Add FileSystem#listLocatedStatus to list a directory's content together with each file's block locations
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6870
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6870
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listFiles.patch, listFiles1.patch, listFiles2.patch, listFiles3.patch, listFiles4.patch
>
>
> This jira implements the new FileSystem API as proposed in HDFS-202. The new API aims to eliminate individual "getFileBlockLocations" calls to NN for each file in the input directory of a job. Instead, a file's block locations are returned together with FileStatus when listing a directory, thus improving getSplits performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9782-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 06:29:43 2010
Return-Path: <common-issues-return-9782-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96042 invoked from network); 27 Jul 2010 06:29:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 06:29:43 -0000
Received: (qmail 4014 invoked by uid 500); 27 Jul 2010 06:29:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3879 invoked by uid 500); 27 Jul 2010 06:29:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3857 invoked by uid 99); 27 Jul 2010 06:29:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 06:29:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 06:29:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6R6TH32029544
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 06:29:17 GMT
Message-ID: <29019932.18071280212157040.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 02:29:17 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6869) [Herriot] Implement a functionality
 for creating either file or folder in task attempt  folder while job is
 running.
In-Reply-To: <1398369.499011279732010273.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6869:
---------------------------------------

               Status: Patch Available  (was: Open)
    Affects Version/s: 0.21.0

> [Herriot] Implement a functionality for creating either file or folder in task attempt  folder while job is running.
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6869
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6869
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6869-ydist-security.patch, 6869-ydist-security.patch, HADOOP-6869.patch, HADOOP-6869.patch
>
>
> Functionality for creating either files or folders in task attempt folder while job is running. The functionality covers the following methods.
> 1. public void DaemonProtocol.createFile(String path, String fileName, boolean local) throws IOException; 
> It uses to create a file with full permissions.
> 2.   public void DaemonProtocol.createFile(String path, String fileName, FsPermission permission, boolean local) throws IOException; 
> It uses to create a file with given permissions.
> 3.   public void DaemonProtocol.createFolder(String path, String folderName, boolean local) throws IOException;
> It uses to create a file with full permissions.
> 4.   public void DaemonProtocol.createFolder(String path, String folderName, FsPermission permission, boolean local) throws IOException;
> It uses to create a folder with given permissions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9783-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 06:29:45 2010
Return-Path: <common-issues-return-9783-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96091 invoked from network); 27 Jul 2010 06:29:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 06:29:44 -0000
Received: (qmail 4077 invoked by uid 500); 27 Jul 2010 06:29:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3902 invoked by uid 500); 27 Jul 2010 06:29:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3860 invoked by uid 99); 27 Jul 2010 06:29:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 06:29:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 06:29:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6R6TGrm029535
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 06:29:16 GMT
Message-ID: <7560342.18041280212156413.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 02:29:16 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6869) [Herriot] Implement a functionality
 for creating either file or folder in task attempt  folder while job is
 running.
In-Reply-To: <1398369.499011279732010273.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892657#action_12892657 ] 

Konstantin Boudnik commented on HADOOP-6869:
--------------------------------------------

I am glad we have understood each other :)

+1 patch looks good. Let's verify. Also, please make sure that it works in real cluster setup because we can't verify it in Hudson yet.

> [Herriot] Implement a functionality for creating either file or folder in task attempt  folder while job is running.
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6869
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6869
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6869-ydist-security.patch, 6869-ydist-security.patch, HADOOP-6869.patch, HADOOP-6869.patch
>
>
> Functionality for creating either files or folders in task attempt folder while job is running. The functionality covers the following methods.
> 1. public void DaemonProtocol.createFile(String path, String fileName, boolean local) throws IOException; 
> It uses to create a file with full permissions.
> 2.   public void DaemonProtocol.createFile(String path, String fileName, FsPermission permission, boolean local) throws IOException; 
> It uses to create a file with given permissions.
> 3.   public void DaemonProtocol.createFolder(String path, String folderName, boolean local) throws IOException;
> It uses to create a file with full permissions.
> 4.   public void DaemonProtocol.createFolder(String path, String folderName, FsPermission permission, boolean local) throws IOException;
> It uses to create a folder with given permissions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9784-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 06:33:44 2010
Return-Path: <common-issues-return-9784-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98668 invoked from network); 27 Jul 2010 06:33:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 06:33:43 -0000
Received: (qmail 11561 invoked by uid 500); 27 Jul 2010 06:33:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11195 invoked by uid 500); 27 Jul 2010 06:33:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10634 invoked by uid 99); 27 Jul 2010 06:33:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 06:33:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 06:33:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6R6XIcb029591
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 06:33:18 GMT
Message-ID: <28420454.18221280212398034.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 02:33:18 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6879) Jsch jar is required for ivy
 resolution to be included in the build
In-Reply-To: <11760779.565921279956469755.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6879?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D128=
92662#action_12892662 ]=20

Konstantin Boudnik commented on HADOOP-6879:
--------------------------------------------

Please have the trunk patch as well. For trunk, I think, this jar file need=
s to go to test->system profile, not master.

> Jsch jar is required for ivy resolution to be included in the build
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6879
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6879
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.1
>            Reporter: Iyappan Srinivasan
>            Assignee: Iyappan Srinivasan
>         Attachments: 6879-ydist-security-patch.txt
>
>
> http://mvnrepository.com/
> com.jcraft =C2=BB jsch=20
> 0.1.42 version needs to be included in the build. This is  required for h=
erriot testcases using jsch.
> Please include this in ivy.
> jsch is originally located in http://www.jcraft.com/jsch/

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9785-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 06:39:44 2010
Return-Path: <common-issues-return-9785-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1066 invoked from network); 27 Jul 2010 06:39:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 06:39:44 -0000
Received: (qmail 18982 invoked by uid 500); 27 Jul 2010 06:39:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18516 invoked by uid 500); 27 Jul 2010 06:39:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18508 invoked by uid 99); 27 Jul 2010 06:39:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 06:39:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 06:39:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6R6dHq8029663
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 06:39:17 GMT
Message-ID: <30645063.18381280212757077.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 02:39:17 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6870) Add FileSystem#listLocatedStatus to
 list a directory's content together with each file's block locations
In-Reply-To: <13042840.506371279749289722.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892665#action_12892665 ] 

Hadoop QA commented on HADOOP-6870:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12450567/listFiles4.patch
  against trunk revision 979485.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 10 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/639/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/639/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/639/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/639/console

This message is automatically generated.

> Add FileSystem#listLocatedStatus to list a directory's content together with each file's block locations
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6870
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6870
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listFiles.patch, listFiles1.patch, listFiles2.patch, listFiles3.patch, listFiles4.patch
>
>
> This jira implements the new FileSystem API as proposed in HDFS-202. The new API aims to eliminate individual "getFileBlockLocations" calls to NN for each file in the input directory of a job. Instead, a file's block locations are returned together with FileStatus when listing a directory, thus improving getSplits performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9786-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 06:47:41 2010
Return-Path: <common-issues-return-9786-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2447 invoked from network); 27 Jul 2010 06:47:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 06:47:41 -0000
Received: (qmail 25725 invoked by uid 500); 27 Jul 2010 06:47:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25589 invoked by uid 500); 27 Jul 2010 06:47:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25581 invoked by uid 99); 27 Jul 2010 06:47:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 06:47:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 06:47:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6R6lHjo029708
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 06:47:17 GMT
Message-ID: <12855287.18491280213237030.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 02:47:17 -0400 (EDT)
From: "Balaji Rajagopalan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6879) Jsch jar is required for ivy
 resolution to be included in the build
In-Reply-To: <11760779.565921279956469755.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6879?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D128=
92668#action_12892668 ]=20

Balaji Rajagopalan commented on HADOOP-6879:
--------------------------------------------

+1

> Jsch jar is required for ivy resolution to be included in the build
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6879
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6879
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.1
>            Reporter: Iyappan Srinivasan
>            Assignee: Iyappan Srinivasan
>         Attachments: 6879-ydist-security-patch.txt
>
>
> http://mvnrepository.com/
> com.jcraft =C2=BB jsch=20
> 0.1.42 version needs to be included in the build. This is  required for h=
erriot testcases using jsch.
> Please include this in ivy.
> jsch is originally located in http://www.jcraft.com/jsch/

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9787-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 06:57:42 2010
Return-Path: <common-issues-return-9787-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3711 invoked from network); 27 Jul 2010 06:57:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 06:57:42 -0000
Received: (qmail 33613 invoked by uid 500); 27 Jul 2010 06:57:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33410 invoked by uid 500); 27 Jul 2010 06:57:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33400 invoked by uid 99); 27 Jul 2010 06:57:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 06:57:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 06:57:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6R6vHIA029779
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 06:57:17 GMT
Message-ID: <33174879.18651280213837387.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 02:57:17 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6869) [Herriot] Implement a functionality
 for creating either file or folder in task attempt  folder while job is
 running.
In-Reply-To: <1398369.499011279732010273.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892672#action_12892672 ] 

Hadoop QA commented on HADOOP-6869:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12450562/HADOOP-6869.patch
  against trunk revision 979485.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 9 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/640/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/640/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/640/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/640/console

This message is automatically generated.

> [Herriot] Implement a functionality for creating either file or folder in task attempt  folder while job is running.
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6869
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6869
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6869-ydist-security.patch, 6869-ydist-security.patch, HADOOP-6869.patch, HADOOP-6869.patch
>
>
> Functionality for creating either files or folders in task attempt folder while job is running. The functionality covers the following methods.
> 1. public void DaemonProtocol.createFile(String path, String fileName, boolean local) throws IOException; 
> It uses to create a file with full permissions.
> 2.   public void DaemonProtocol.createFile(String path, String fileName, FsPermission permission, boolean local) throws IOException; 
> It uses to create a file with given permissions.
> 3.   public void DaemonProtocol.createFolder(String path, String folderName, boolean local) throws IOException;
> It uses to create a file with full permissions.
> 4.   public void DaemonProtocol.createFolder(String path, String folderName, FsPermission permission, boolean local) throws IOException;
> It uses to create a folder with given permissions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9788-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 07:13:45 2010
Return-Path: <common-issues-return-9788-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12292 invoked from network); 27 Jul 2010 07:13:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 07:13:45 -0000
Received: (qmail 57049 invoked by uid 500); 27 Jul 2010 07:13:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56820 invoked by uid 500); 27 Jul 2010 07:13:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56811 invoked by uid 99); 27 Jul 2010 07:13:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 07:13:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 07:13:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6R7DIUh029994
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 07:13:18 GMT
Message-ID: <29262733.18831280214798103.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 03:13:18 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6869) [Herriot] Implement a functionality
 for creating either file or folder in task attempt  folder while job is
 running.
In-Reply-To: <1398369.499011279732010273.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892677#action_12892677 ] 

Vinay Kumar Thota commented on HADOOP-6869:
-------------------------------------------

java doc warnings are not related to this patch.

> [Herriot] Implement a functionality for creating either file or folder in task attempt  folder while job is running.
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6869
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6869
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6869-ydist-security.patch, 6869-ydist-security.patch, HADOOP-6869.patch, HADOOP-6869.patch
>
>
> Functionality for creating either files or folders in task attempt folder while job is running. The functionality covers the following methods.
> 1. public void DaemonProtocol.createFile(String path, String fileName, boolean local) throws IOException; 
> It uses to create a file with full permissions.
> 2.   public void DaemonProtocol.createFile(String path, String fileName, FsPermission permission, boolean local) throws IOException; 
> It uses to create a file with given permissions.
> 3.   public void DaemonProtocol.createFolder(String path, String folderName, boolean local) throws IOException;
> It uses to create a file with full permissions.
> 4.   public void DaemonProtocol.createFolder(String path, String folderName, FsPermission permission, boolean local) throws IOException;
> It uses to create a folder with given permissions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9789-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 14:05:43 2010
Return-Path: <common-issues-return-9789-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92722 invoked from network); 27 Jul 2010 14:05:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 14:05:43 -0000
Received: (qmail 86372 invoked by uid 500); 27 Jul 2010 14:05:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86143 invoked by uid 500); 27 Jul 2010 14:05:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85247 invoked by uid 99); 27 Jul 2010 14:05:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 14:05:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 14:05:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6RE5GWG003599
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 14:05:16 GMT
Message-ID: <17962447.24221280239516227.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 10:05:16 -0400 (EDT)
From: "Gordon Sommers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6883) Text.toString violates its
 abstraction
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Text.toString violates its abstraction
--------------------------------------

                 Key: HADOOP-6883
                 URL: https://issues.apache.org/jira/browse/HADOOP-6883
             Project: Hadoop Common
          Issue Type: Bug
          Components: io
    Affects Versions: 0.20.1
         Environment: Linux
            Reporter: Gordon Sommers


I stumbled upon this when encoding a google protocol buffer in base64, and storing it in a Text object for serialization. Compare the following two lines:

byte [] decoded = b64.decode(val.getBytes())
//this does not return the same bytes as below and the result, after decoding the base64 successfully, is a very mangled protocol buffer

byte [] decoded = b64.decode(val.toString().getBytes());
//YES, toString() FIXES IT

Elsewhere in my code I also have: 
Text curline = new Text(values.next().toString());
byte [] raw = base64.decode(curline.getBytes());
//This does work.

It looks like the Text object must be toString'd (just once, somewhere, even if its later repacked in a Text) before it will have the proper byte representation. I would classify this as a leaky abstraction and ask that the reason please be isolated and the api fixed somehow so that other developers dont have to spend 3 days figuring out when Text.getBytes isn't returning the right bytes even though Text.toString prints exactly the right string representation and Text.toString.getBytes does return the right bytes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9790-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 15:17:45 2010
Return-Path: <common-issues-return-9790-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33285 invoked from network); 27 Jul 2010 15:17:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 15:17:45 -0000
Received: (qmail 90145 invoked by uid 500); 27 Jul 2010 15:17:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89829 invoked by uid 500); 27 Jul 2010 15:17:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89745 invoked by uid 99); 27 Jul 2010 15:17:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 15:17:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 15:17:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6RFHJcw004382
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 15:17:19 GMT
Message-ID: <26512072.25621280243839478.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 11:17:19 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6883) Text.toString violates its
 abstraction
In-Reply-To: <17962447.24221280239516227.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley resolved HADOOP-6883.
-----------------------------------

    Resolution: Invalid

The proper call is:

{code}
b64.decode(val.getBytes(), 0, val.getLength());
{code}

Yes, it is confusing, but doing anything else would not perform acceptably. If you look at the javadoc for getBytes(), you'll see why your call fails.

> Text.toString violates its abstraction
> --------------------------------------
>
>                 Key: HADOOP-6883
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6883
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.1
>         Environment: Linux
>            Reporter: Gordon Sommers
>
> I stumbled upon this when encoding a google protocol buffer in base64, and storing it in a Text object for serialization. Compare the following two lines:
> byte [] decoded = b64.decode(val.getBytes())
> //this does not return the same bytes as below and the result, after decoding the base64 successfully, is a very mangled protocol buffer
> byte [] decoded = b64.decode(val.toString().getBytes());
> //YES, toString() FIXES IT
> Elsewhere in my code I also have: 
> Text curline = new Text(values.next().toString());
> byte [] raw = base64.decode(curline.getBytes());
> //This does work.
> It looks like the Text object must be toString'd (just once, somewhere, even if its later repacked in a Text) before it will have the proper byte representation. I would classify this as a leaky abstraction and ask that the reason please be isolated and the api fixed somehow so that other developers dont have to spend 3 days figuring out when Text.getBytes isn't returning the right bytes even though Text.toString prints exactly the right string representation and Text.toString.getBytes does return the right bytes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9791-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 15:21:38 2010
Return-Path: <common-issues-return-9791-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35158 invoked from network); 27 Jul 2010 15:21:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 15:21:38 -0000
Received: (qmail 97648 invoked by uid 500); 27 Jul 2010 15:21:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97589 invoked by uid 500); 27 Jul 2010 15:21:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97581 invoked by uid 99); 27 Jul 2010 15:21:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 15:21:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 15:21:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6RFLG45004424
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 15:21:17 GMT
Message-ID: <27670491.25721280244076884.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 11:21:16 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6883) Text.toString violates its
 abstraction
In-Reply-To: <17962447.24221280239516227.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892820#action_12892820 ] 

Owen O'Malley commented on HADOOP-6883:
---------------------------------------

I should also comment that I'm almost done with a patch that re-designs the generic serialization framework that includes native support for ProtocolBuffers, Thrift, Avro, and Writables. I already have all of the serializations working with both SequenceFiles and TFiles (with serialization layered on top named OFiles).

> Text.toString violates its abstraction
> --------------------------------------
>
>                 Key: HADOOP-6883
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6883
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.1
>         Environment: Linux
>            Reporter: Gordon Sommers
>
> I stumbled upon this when encoding a google protocol buffer in base64, and storing it in a Text object for serialization. Compare the following two lines:
> byte [] decoded = b64.decode(val.getBytes())
> //this does not return the same bytes as below and the result, after decoding the base64 successfully, is a very mangled protocol buffer
> byte [] decoded = b64.decode(val.toString().getBytes());
> //YES, toString() FIXES IT
> Elsewhere in my code I also have: 
> Text curline = new Text(values.next().toString());
> byte [] raw = base64.decode(curline.getBytes());
> //This does work.
> It looks like the Text object must be toString'd (just once, somewhere, even if its later repacked in a Text) before it will have the proper byte representation. I would classify this as a leaky abstraction and ask that the reason please be isolated and the api fixed somehow so that other developers dont have to spend 3 days figuring out when Text.getBytes isn't returning the right bytes even though Text.toString prints exactly the right string representation and Text.toString.getBytes does return the right bytes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9792-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 16:52:39 2010
Return-Path: <common-issues-return-9792-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66272 invoked from network); 27 Jul 2010 16:52:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 16:52:39 -0000
Received: (qmail 23594 invoked by uid 500); 27 Jul 2010 16:52:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23478 invoked by uid 500); 27 Jul 2010 16:52:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23470 invoked by uid 99); 27 Jul 2010 16:52:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 16:52:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 16:52:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6RGqHP0005496
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 16:52:17 GMT
Message-ID: <12134151.27041280249537433.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 12:52:17 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6870) Add FileSystem#listLocatedStatus to
 list a directory's content together with each file's block locations
In-Reply-To: <13042840.506371279749289722.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892850#action_12892850 ] 

Hairong Kuang commented on HADOOP-6870:
---------------------------------------

All six javadoc warning are related to security and not introduced by this patch.

> Add FileSystem#listLocatedStatus to list a directory's content together with each file's block locations
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6870
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6870
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listFiles.patch, listFiles1.patch, listFiles2.patch, listFiles3.patch, listFiles4.patch
>
>
> This jira implements the new FileSystem API as proposed in HDFS-202. The new API aims to eliminate individual "getFileBlockLocations" calls to NN for each file in the input directory of a job. Instead, a file's block locations are returned together with FileStatus when listing a directory, thus improving getSplits performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9793-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 17:27:44 2010
Return-Path: <common-issues-return-9793-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82474 invoked from network); 27 Jul 2010 17:27:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 17:27:43 -0000
Received: (qmail 78411 invoked by uid 500); 27 Jul 2010 17:27:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78359 invoked by uid 500); 27 Jul 2010 17:27:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78239 invoked by uid 99); 27 Jul 2010 17:27:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 17:27:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 17:27:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6RHRITq005866
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 17:27:18 GMT
Message-ID: <24028340.27731280251638118.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 13:27:18 -0400 (EDT)
From: "Gordon Sommers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6883) Text.toString violates its
 abstraction
In-Reply-To: <17962447.24221280239516227.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892861#action_12892861 ] 

Gordon Sommers commented on HADOOP-6883:
----------------------------------------

Thanks for the quick feedback. I'm not sure I understand why calling it like that would make a difference -- if the bytes we want to decode amount to the entire string that should have the same effect as b64.decode(val.getBytes()), no? I looked at the api here: http://www.docjar.com/docs/api/org/apache/hadoop/io/Text.html#getBytes and it didn't specify anything unusual. Is Text encoding the bytes differently from String?

In either case, I'm glad to hear that you're working on improving the serialization framework -- I think a lot of people, myself included, will really appreciate that when it gets released. Thanks!

> Text.toString violates its abstraction
> --------------------------------------
>
>                 Key: HADOOP-6883
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6883
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.1
>         Environment: Linux
>            Reporter: Gordon Sommers
>
> I stumbled upon this when encoding a google protocol buffer in base64, and storing it in a Text object for serialization. Compare the following two lines:
> byte [] decoded = b64.decode(val.getBytes())
> //this does not return the same bytes as below and the result, after decoding the base64 successfully, is a very mangled protocol buffer
> byte [] decoded = b64.decode(val.toString().getBytes());
> //YES, toString() FIXES IT
> Elsewhere in my code I also have: 
> Text curline = new Text(values.next().toString());
> byte [] raw = base64.decode(curline.getBytes());
> //This does work.
> It looks like the Text object must be toString'd (just once, somewhere, even if its later repacked in a Text) before it will have the proper byte representation. I would classify this as a leaky abstraction and ask that the reason please be isolated and the api fixed somehow so that other developers dont have to spend 3 days figuring out when Text.getBytes isn't returning the right bytes even though Text.toString prints exactly the right string representation and Text.toString.getBytes does return the right bytes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9794-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 17:30:42 2010
Return-Path: <common-issues-return-9794-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83936 invoked from network); 27 Jul 2010 17:30:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 17:30:41 -0000
Received: (qmail 83696 invoked by uid 500); 27 Jul 2010 17:30:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83337 invoked by uid 500); 27 Jul 2010 17:30:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83166 invoked by uid 99); 27 Jul 2010 17:30:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 17:30:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 17:30:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6RHUGnd005905
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 17:30:16 GMT
Message-ID: <22237834.27831280251816522.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 13:30:16 -0400 (EDT)
From: "Gordon Sommers (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6883) Text.toString violates its
 abstraction
In-Reply-To: <17962447.24221280239516227.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892862#action_12892862 ] 

Gordon Sommers commented on HADOOP-6883:
----------------------------------------

Ah, just found the javadocs hosted by apache -- now I see what the issue was. Thanks. Is there a case where it's useful to dump the entire byte array and not just the valid ones?

> Text.toString violates its abstraction
> --------------------------------------
>
>                 Key: HADOOP-6883
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6883
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.1
>         Environment: Linux
>            Reporter: Gordon Sommers
>
> I stumbled upon this when encoding a google protocol buffer in base64, and storing it in a Text object for serialization. Compare the following two lines:
> byte [] decoded = b64.decode(val.getBytes())
> //this does not return the same bytes as below and the result, after decoding the base64 successfully, is a very mangled protocol buffer
> byte [] decoded = b64.decode(val.toString().getBytes());
> //YES, toString() FIXES IT
> Elsewhere in my code I also have: 
> Text curline = new Text(values.next().toString());
> byte [] raw = base64.decode(curline.getBytes());
> //This does work.
> It looks like the Text object must be toString'd (just once, somewhere, even if its later repacked in a Text) before it will have the proper byte representation. I would classify this as a leaky abstraction and ask that the reason please be isolated and the api fixed somehow so that other developers dont have to spend 3 days figuring out when Text.getBytes isn't returning the right bytes even though Text.toString prints exactly the right string representation and Text.toString.getBytes does return the right bytes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9795-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 18:14:42 2010
Return-Path: <common-issues-return-9795-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 874 invoked from network); 27 Jul 2010 18:14:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 18:14:42 -0000
Received: (qmail 42377 invoked by uid 500); 27 Jul 2010 18:14:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42272 invoked by uid 500); 27 Jul 2010 18:14:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42264 invoked by uid 99); 27 Jul 2010 18:14:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 18:14:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 18:14:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6RIEHNg006310
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 18:14:17 GMT
Message-ID: <10240586.28581280254457135.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 14:14:17 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6881) The efficient comparators aren't
 always used except for BytesWritable and Text
In-Reply-To: <28363686.3111280170396575.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-6881:
---------------------------------

           Status: Resolved  (was: Patch Available)
     Hadoop Flags: [Reviewed]
    Fix Version/s: 0.21.0
       Resolution: Fixed

Patch merged to both 0.20 and 0.21 branches.

> The efficient comparators aren't always used except for BytesWritable and Text
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6881
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6881
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.0
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.20.3, 0.21.0
>
>         Attachments: h-6881.patch, HADOOP-6881.patch, HADOOP-6881.patch
>
>
> When we moved from Java 4 to Java 5 (and then 6), there was a change in the JVM semantics such that references to a class such as IntWritable.class no longer forces initialization. Since all of the Writables depend on their class static blocks to register their fast comparators, that can happen *after* we look up the comparator. In that case, the framework will fall back to the generic comparator that deserializes both keys and does the object compare, which may cause a huge slow down in the sort.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9796-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 18:27:40 2010
Return-Path: <common-issues-return-9796-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10442 invoked from network); 27 Jul 2010 18:27:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 18:27:40 -0000
Received: (qmail 55907 invoked by uid 500); 27 Jul 2010 18:27:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55819 invoked by uid 500); 27 Jul 2010 18:27:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55810 invoked by uid 99); 27 Jul 2010 18:27:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 18:27:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 18:27:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6RIRIIx006396
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 18:27:18 GMT
Message-ID: <1014404.28781280255238513.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 14:27:18 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6883) Text.toString violates its
 abstraction
In-Reply-To: <17962447.24221280239516227.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892880#action_12892880 ] 

Owen O'Malley commented on HADOOP-6883:
---------------------------------------

No, it is never useful to look at the trailing bytes. They are just random bytes left over from a previous value.

> Text.toString violates its abstraction
> --------------------------------------
>
>                 Key: HADOOP-6883
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6883
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.1
>         Environment: Linux
>            Reporter: Gordon Sommers
>
> I stumbled upon this when encoding a google protocol buffer in base64, and storing it in a Text object for serialization. Compare the following two lines:
> byte [] decoded = b64.decode(val.getBytes())
> //this does not return the same bytes as below and the result, after decoding the base64 successfully, is a very mangled protocol buffer
> byte [] decoded = b64.decode(val.toString().getBytes());
> //YES, toString() FIXES IT
> Elsewhere in my code I also have: 
> Text curline = new Text(values.next().toString());
> byte [] raw = base64.decode(curline.getBytes());
> //This does work.
> It looks like the Text object must be toString'd (just once, somewhere, even if its later repacked in a Text) before it will have the proper byte representation. I would classify this as a leaky abstraction and ask that the reason please be isolated and the api fixed somehow so that other developers dont have to spend 3 days figuring out when Text.getBytes isn't returning the right bytes even though Text.toString prints exactly the right string representation and Text.toString.getBytes does return the right bytes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9797-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 18:49:40 2010
Return-Path: <common-issues-return-9797-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17035 invoked from network); 27 Jul 2010 18:49:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 18:49:40 -0000
Received: (qmail 81301 invoked by uid 500); 27 Jul 2010 18:49:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81236 invoked by uid 500); 27 Jul 2010 18:49:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81228 invoked by uid 99); 27 Jul 2010 18:49:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 18:49:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 18:49:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6RInJck006656
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 18:49:19 GMT
Message-ID: <17862551.29221280256559027.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 14:49:19 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6656:
--------------------------------

    Attachment: 6656-trunk-3.patch

Addressing Kan's comments..

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6656-trunk-1.patch, 6656-trunk-2.patch, 6656-trunk-3.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9798-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 18:53:39 2010
Return-Path: <common-issues-return-9798-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17600 invoked from network); 27 Jul 2010 18:53:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 18:53:39 -0000
Received: (qmail 85395 invoked by uid 500); 27 Jul 2010 18:53:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85319 invoked by uid 500); 27 Jul 2010 18:53:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85311 invoked by uid 99); 27 Jul 2010 18:53:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 18:53:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 18:53:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6RIrHuu006713
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 18:53:17 GMT
Message-ID: <19503364.29271280256797623.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 14:53:17 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6656:
--------------------------------

    Status: Open  (was: Patch Available)

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6656-trunk-1.patch, 6656-trunk-2.patch, 6656-trunk-3.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9799-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 18:53:39 2010
Return-Path: <common-issues-return-9799-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17644 invoked from network); 27 Jul 2010 18:53:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 18:53:39 -0000
Received: (qmail 85512 invoked by uid 500); 27 Jul 2010 18:53:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85358 invoked by uid 500); 27 Jul 2010 18:53:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85321 invoked by uid 99); 27 Jul 2010 18:53:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 18:53:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 18:53:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6RIrIQC006725
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 18:53:18 GMT
Message-ID: <12743720.29311280256798223.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 14:53:18 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6656:
--------------------------------

    Status: Patch Available  (was: Open)

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6656-trunk-1.patch, 6656-trunk-2.patch, 6656-trunk-3.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9800-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 19:07:36 2010
Return-Path: <common-issues-return-9800-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25203 invoked from network); 27 Jul 2010 19:07:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 19:07:36 -0000
Received: (qmail 6569 invoked by uid 500); 27 Jul 2010 19:07:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6535 invoked by uid 500); 27 Jul 2010 19:07:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6527 invoked by uid 99); 27 Jul 2010 19:07:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 19:07:35 +0000
X-ASF-Spam-Status: No, hits=-2.3 required=10.0
	tests=RCVD_IN_DNSWL_MED,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of gary.lutz@thomsonreuters.com designates 163.231.12.71 as permitted sender)
Received: from [163.231.12.71] (HELO mailout4-tlr.thomson.com) (163.231.12.71)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 19:07:26 +0000
Received: from mailout4-tlr.thomson.com (localhost.localdomain [127.0.0.1])
	by mailout4-tlr.thomson.com (Switch-3.3.2/Switch-3.3.2) with ESMTP id o6RJ73LD030090
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 19:07:03 GMT
Received: from tlrusmneagfe01.ERF.THOMSON.COM (tlrusmneagfe01.erf.thomson.com [163.231.22.161])
	by mailout4-tlr.thomson.com (Switch-3.3.2/Switch-3.3.2) with ESMTP id o6RJ73EE030082
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 19:07:03 GMT
Received: from TLRUSMNEAGMBX12.ERF.THOMSON.COM ([163.231.30.198]) by tlrusmneagfe01.ERF.THOMSON.COM with Microsoft SMTPSVC(6.0.3790.4675);
	 Tue, 27 Jul 2010 14:07:03 -0500
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="UTF-8"
Content-Transfer-Encoding: base64
Subject: RE: [jira] Commented: (HADOOP-6883) Text.toString violates its abstraction
Date: Tue, 27 Jul 2010 14:07:03 -0500
Message-ID: <426D301441E459438A88C8BE21031F8F1067D9@TLRUSMNEAGMBX12.ERF.THOMSON.COM>
In-Reply-To: <1014404.28781280255238513.JavaMail.jira@thor>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: [jira] Commented: (HADOOP-6883) Text.toString violates its abstraction
Thread-Index: AcstuWrrBU+CEargSb2hZOzmVf2pdwABVOZg
References: <17962447.24221280239516227.JavaMail.jira@thor> <1014404.28781280255238513.JavaMail.jira@thor>
From: <gary.lutz@thomsonreuters.com>
To: <common-issues@hadoop.apache.org>
X-OriginalArrivalTime: 27 Jul 2010 19:07:03.0431 (UTC) FILETIME=[E5E66970:01CB2DBE]
X-Virus-Checked: Checked by ClamAV on apache.org

UGxlYXNlIHJlbW92ZSBtZSBmcm9tIHRoZSBsaXN0IC0gdGhhbmtzIA0KDQpHYXJ5Lkx1dHpAdGhv
bXNvbi5jb20NCg0KLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCkZyb206IE93ZW4gTydNYWxs
ZXkgKEpJUkEpIFttYWlsdG86amlyYUBhcGFjaGUub3JnXSANClNlbnQ6IFR1ZXNkYXksIEp1bHkg
MjcsIDIwMTAgMToyNyBQTQ0KVG86IGNvbW1vbi1pc3N1ZXNAaGFkb29wLmFwYWNoZS5vcmcNClN1
YmplY3Q6IFtqaXJhXSBDb21tZW50ZWQ6IChIQURPT1AtNjg4MykgVGV4dC50b1N0cmluZyB2aW9s
YXRlcyBpdHMgYWJzdHJhY3Rpb24NCg0KDQogICAgWyBodHRwczovL2lzc3Vlcy5hcGFjaGUub3Jn
L2ppcmEvYnJvd3NlL0hBRE9PUC02ODgzP3BhZ2U9Y29tLmF0bGFzc2lhbi5qaXJhLnBsdWdpbi5z
eXN0ZW0uaXNzdWV0YWJwYW5lbHM6Y29tbWVudC10YWJwYW5lbCZmb2N1c2VkQ29tbWVudElkPTEy
ODkyODgwI2FjdGlvbl8xMjg5Mjg4MCBdIA0KDQpPd2VuIE8nTWFsbGV5IGNvbW1lbnRlZCBvbiBI
QURPT1AtNjg4MzoNCi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KDQpO
bywgaXQgaXMgbmV2ZXIgdXNlZnVsIHRvIGxvb2sgYXQgdGhlIHRyYWlsaW5nIGJ5dGVzLiBUaGV5
IGFyZSBqdXN0IHJhbmRvbSBieXRlcyBsZWZ0IG92ZXIgZnJvbSBhIHByZXZpb3VzIHZhbHVlLg0K
DQo+IFRleHQudG9TdHJpbmcgdmlvbGF0ZXMgaXRzIGFic3RyYWN0aW9uDQo+IC0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+DQo+ICAgICAgICAgICAgICAgICBLZXk6IEhB
RE9PUC02ODgzDQo+ICAgICAgICAgICAgICAgICBVUkw6IGh0dHBzOi8vaXNzdWVzLmFwYWNoZS5v
cmcvamlyYS9icm93c2UvSEFET09QLTY4ODMNCj4gICAgICAgICAgICAgUHJvamVjdDogSGFkb29w
IENvbW1vbg0KPiAgICAgICAgICBJc3N1ZSBUeXBlOiBCdWcNCj4gICAgICAgICAgQ29tcG9uZW50
czogaW8NCj4gICAgQWZmZWN0cyBWZXJzaW9uczogMC4yMC4xDQo+ICAgICAgICAgRW52aXJvbm1l
bnQ6IExpbnV4DQo+ICAgICAgICAgICAgUmVwb3J0ZXI6IEdvcmRvbiBTb21tZXJzDQo+DQo+IEkg
c3R1bWJsZWQgdXBvbiB0aGlzIHdoZW4gZW5jb2RpbmcgYSBnb29nbGUgcHJvdG9jb2wgYnVmZmVy
IGluIGJhc2U2NCwgYW5kIHN0b3JpbmcgaXQgaW4gYSBUZXh0IG9iamVjdCBmb3Igc2VyaWFsaXph
dGlvbi4gQ29tcGFyZSB0aGUgZm9sbG93aW5nIHR3byBsaW5lczoNCj4gYnl0ZSBbXSBkZWNvZGVk
ID0gYjY0LmRlY29kZSh2YWwuZ2V0Qnl0ZXMoKSkNCj4gLy90aGlzIGRvZXMgbm90IHJldHVybiB0
aGUgc2FtZSBieXRlcyBhcyBiZWxvdyBhbmQgdGhlIHJlc3VsdCwgYWZ0ZXIgZGVjb2RpbmcgdGhl
IGJhc2U2NCBzdWNjZXNzZnVsbHksIGlzIGEgdmVyeSBtYW5nbGVkIHByb3RvY29sIGJ1ZmZlcg0K
PiBieXRlIFtdIGRlY29kZWQgPSBiNjQuZGVjb2RlKHZhbC50b1N0cmluZygpLmdldEJ5dGVzKCkp
Ow0KPiAvL1lFUywgdG9TdHJpbmcoKSBGSVhFUyBJVA0KPiBFbHNld2hlcmUgaW4gbXkgY29kZSBJ
IGFsc28gaGF2ZTogDQo+IFRleHQgY3VybGluZSA9IG5ldyBUZXh0KHZhbHVlcy5uZXh0KCkudG9T
dHJpbmcoKSk7DQo+IGJ5dGUgW10gcmF3ID0gYmFzZTY0LmRlY29kZShjdXJsaW5lLmdldEJ5dGVz
KCkpOw0KPiAvL1RoaXMgZG9lcyB3b3JrLg0KPiBJdCBsb29rcyBsaWtlIHRoZSBUZXh0IG9iamVj
dCBtdXN0IGJlIHRvU3RyaW5nJ2QgKGp1c3Qgb25jZSwgc29tZXdoZXJlLCBldmVuIGlmIGl0cyBs
YXRlciByZXBhY2tlZCBpbiBhIFRleHQpIGJlZm9yZSBpdCB3aWxsIGhhdmUgdGhlIHByb3BlciBi
eXRlIHJlcHJlc2VudGF0aW9uLiBJIHdvdWxkIGNsYXNzaWZ5IHRoaXMgYXMgYSBsZWFreSBhYnN0
cmFjdGlvbiBhbmQgYXNrIHRoYXQgdGhlIHJlYXNvbiBwbGVhc2UgYmUgaXNvbGF0ZWQgYW5kIHRo
ZSBhcGkgZml4ZWQgc29tZWhvdyBzbyB0aGF0IG90aGVyIGRldmVsb3BlcnMgZG9udCBoYXZlIHRv
IHNwZW5kIDMgZGF5cyBmaWd1cmluZyBvdXQgd2hlbiBUZXh0LmdldEJ5dGVzIGlzbid0IHJldHVy
bmluZyB0aGUgcmlnaHQgYnl0ZXMgZXZlbiB0aG91Z2ggVGV4dC50b1N0cmluZyBwcmludHMgZXhh
Y3RseSB0aGUgcmlnaHQgc3RyaW5nIHJlcHJlc2VudGF0aW9uIGFuZCBUZXh0LnRvU3RyaW5nLmdl
dEJ5dGVzIGRvZXMgcmV0dXJuIHRoZSByaWdodCBieXRlcy4NCg0KLS0gDQpUaGlzIG1lc3NhZ2Ug
aXMgYXV0b21hdGljYWxseSBnZW5lcmF0ZWQgYnkgSklSQS4NCi0NCllvdSBjYW4gcmVwbHkgdG8g
dGhpcyBlbWFpbCB0byBhZGQgYSBjb21tZW50IHRvIHRoZSBpc3N1ZSBvbmxpbmUuDQoNCg0K

From common-issues-return-9801-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 19:22:42 2010
Return-Path: <common-issues-return-9801-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31977 invoked from network); 27 Jul 2010 19:22:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 19:22:42 -0000
Received: (qmail 23765 invoked by uid 500); 27 Jul 2010 19:22:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23704 invoked by uid 500); 27 Jul 2010 19:22:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23696 invoked by uid 99); 27 Jul 2010 19:22:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 19:22:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 19:22:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6RJMH3q007047
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 19:22:17 GMT
Message-ID: <23482245.29751280258537283.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 15:22:17 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892899#action_12892899 ] 

Hadoop QA commented on HADOOP-6656:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12450614/6656-trunk-3.patch
  against trunk revision 979785.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/641/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/641/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/641/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/641/console

This message is automatically generated.

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6656-trunk-1.patch, 6656-trunk-2.patch, 6656-trunk-3.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9802-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 19:26:42 2010
Return-Path: <common-issues-return-9802-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33798 invoked from network); 27 Jul 2010 19:26:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 19:26:42 -0000
Received: (qmail 28146 invoked by uid 500); 27 Jul 2010 19:26:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28092 invoked by uid 500); 27 Jul 2010 19:26:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28084 invoked by uid 99); 27 Jul 2010 19:26:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 19:26:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 19:26:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6RJQHaT007087
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 19:26:17 GMT
Message-ID: <22536682.29821280258777506.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 15:26:17 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6869) [Herriot] Implement a functionality
 for creating either file or folder in task attempt  folder while job is
 running.
In-Reply-To: <1398369.499011279732010273.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892902#action_12892902 ] 

Konstantin Boudnik commented on HADOOP-6869:
--------------------------------------------

I'll commit it later today if won't hear otherwise.

> [Herriot] Implement a functionality for creating either file or folder in task attempt  folder while job is running.
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6869
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6869
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6869-ydist-security.patch, 6869-ydist-security.patch, HADOOP-6869.patch, HADOOP-6869.patch
>
>
> Functionality for creating either files or folders in task attempt folder while job is running. The functionality covers the following methods.
> 1. public void DaemonProtocol.createFile(String path, String fileName, boolean local) throws IOException; 
> It uses to create a file with full permissions.
> 2.   public void DaemonProtocol.createFile(String path, String fileName, FsPermission permission, boolean local) throws IOException; 
> It uses to create a file with given permissions.
> 3.   public void DaemonProtocol.createFolder(String path, String folderName, boolean local) throws IOException;
> It uses to create a file with full permissions.
> 4.   public void DaemonProtocol.createFolder(String path, String folderName, FsPermission permission, boolean local) throws IOException;
> It uses to create a folder with given permissions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9803-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 19:34:41 2010
Return-Path: <common-issues-return-9803-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35695 invoked from network); 27 Jul 2010 19:34:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 19:34:41 -0000
Received: (qmail 41470 invoked by uid 500); 27 Jul 2010 19:34:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41412 invoked by uid 500); 27 Jul 2010 19:34:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41404 invoked by uid 99); 27 Jul 2010 19:34:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 19:34:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 19:34:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6RJYGpQ007147
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 19:34:17 GMT
Message-ID: <10975126.29911280259256921.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 15:34:16 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892904#action_12892904 ] 

Kan Zhang commented on HADOOP-6656:
-----------------------------------

Sorry, I now see why you didn't use SecurityUtil.getTgtFromSubject() in the first place. You already have a "subject" to work with. For this reason I think it's better to use your original code, since I'm not sure what your AccessControlContext is when you call getTgtFromSubject(). But I like your way of figuring out whether a ticket is an original TGT. Is it possible to share that logic with SecurityUtil.isOriginalTgt()?

Otherwise, +1 for the patch.

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6656-trunk-1.patch, 6656-trunk-2.patch, 6656-trunk-3.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9804-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 21:25:39 2010
Return-Path: <common-issues-return-9804-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77489 invoked from network); 27 Jul 2010 21:25:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 21:25:38 -0000
Received: (qmail 88973 invoked by uid 500); 27 Jul 2010 21:25:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88879 invoked by uid 500); 27 Jul 2010 21:25:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88866 invoked by uid 99); 27 Jul 2010 21:25:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 21:25:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 21:25:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6RLPGq1008282
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 21:25:17 GMT
Message-ID: <19488633.31491280265916962.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 17:25:16 -0400 (EDT)
From: "Nicolas Spiegelberg (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6840) Support non-recursive create() in
 FileSystem & SequenceFile.Writer
In-Reply-To: <30089846.67411277503610667.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicolas Spiegelberg updated HADOOP-6840:
----------------------------------------

    Attachment: HADOOP-6840_0.21-2.patch

fixed bug in original patch. constructors with the FSDataOutputStream param will not confer ownership of the stream by default, so SequenceFile.close() would not call os.close().  Added ownsStream() package-private API to set ownership after construction for the new API changes.  again, this does not touch existing APIs

> Support non-recursive create() in FileSystem & SequenceFile.Writer
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6840
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6840
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs, io
>    Affects Versions: 0.21.0, 0.20-append
>            Reporter: Nicolas Spiegelberg
>            Priority: Minor
>         Attachments: HADOOP-6840_0.21-2.patch, HADOOP-6840_0.21.patch
>
>
> The proposed solution for HBASE-2312 requires the sequence file to handle a non-recursive create.  This is already supported by HDFS, but needs to have an equivalent FileSystem & SequenceFile.Writer API.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9805-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 21:29:39 2010
Return-Path: <common-issues-return-9805-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79093 invoked from network); 27 Jul 2010 21:29:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 21:29:39 -0000
Received: (qmail 90860 invoked by uid 500); 27 Jul 2010 21:29:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90701 invoked by uid 500); 27 Jul 2010 21:29:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90693 invoked by uid 99); 27 Jul 2010 21:29:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 21:29:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 21:29:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6RLTGZu008310
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 21:29:17 GMT
Message-ID: <22563239.31551280266156737.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 17:29:16 -0400 (EDT)
From: "Nicolas Spiegelberg (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6840) Support non-recursive create() in
 FileSystem & SequenceFile.Writer
In-Reply-To: <30089846.67411277503610667.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicolas Spiegelberg updated HADOOP-6840:
----------------------------------------

    Attachment: HADOOP-6840_0.20-append.patch

version for 0.20-append.  requires HDFS-617 as a prerequisite

> Support non-recursive create() in FileSystem & SequenceFile.Writer
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6840
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6840
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs, io
>    Affects Versions: 0.21.0, 0.20-append
>            Reporter: Nicolas Spiegelberg
>            Priority: Minor
>         Attachments: HADOOP-6840_0.20-append.patch, HADOOP-6840_0.21-2.patch, HADOOP-6840_0.21.patch
>
>
> The proposed solution for HBASE-2312 requires the sequence file to handle a non-recursive create.  This is already supported by HDFS, but needs to have an equivalent FileSystem & SequenceFile.Writer API.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9806-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 22:46:50 2010
Return-Path: <common-issues-return-9806-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5541 invoked from network); 27 Jul 2010 22:46:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 22:46:49 -0000
Received: (qmail 80836 invoked by uid 500); 27 Jul 2010 22:46:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80315 invoked by uid 500); 27 Jul 2010 22:46:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80137 invoked by uid 99); 27 Jul 2010 22:46:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 22:46:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 22:46:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6RMkOEm009480
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 22:46:24 GMT
Message-ID: <8688001.34681280270784520.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 18:46:24 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Add LOG.isDebugEnabled() guard for each LOG.debug("...")
--------------------------------------------------------

                 Key: HADOOP-6884
                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
             Project: Hadoop Common
          Issue Type: Improvement
    Affects Versions: 0.22.0
            Reporter: Erik Steffl
             Fix For: 0.22.0


Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9807-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 22:57:38 2010
Return-Path: <common-issues-return-9807-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6373 invoked from network); 27 Jul 2010 22:57:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 22:57:38 -0000
Received: (qmail 84568 invoked by uid 500); 27 Jul 2010 22:57:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84487 invoked by uid 500); 27 Jul 2010 22:57:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84479 invoked by uid 99); 27 Jul 2010 22:57:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 22:57:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 22:57:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6RMvGLh009543
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 22:57:16 GMT
Message-ID: <1361606.34801280271436532.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 18:57:16 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Erik Steffl updated HADOOP-6884:
--------------------------------

    Attachment: HADOOP-6884-0.22.patch

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9808-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 22:57:41 2010
Return-Path: <common-issues-return-9808-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6451 invoked from network); 27 Jul 2010 22:57:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 22:57:41 -0000
Received: (qmail 84791 invoked by uid 500); 27 Jul 2010 22:57:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84761 invoked by uid 500); 27 Jul 2010 22:57:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84753 invoked by uid 99); 27 Jul 2010 22:57:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 22:57:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 22:57:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6RMvHQ6009549
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 22:57:17 GMT
Message-ID: <7126195.34821280271436998.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 18:57:16 -0400 (EDT)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892972#action_12892972 ] 

Alejandro Abdelnur commented on HADOOP-6884:
--------------------------------------------

While common practice, this result in lot of code uglyfication/noise.

I'd rather suggest the following (we are doing this in Oozie):

Create a Log wrapper, XLog which extends Log and provides the following additional methods:

  info(String msgTemplate, Object ... args)
  warn(String msgTemplate, Object ... args)
  debug(String msgTemplate, Object ... args)
  trace(String msgTemplate, Object ... args)

In each one of these methods, if the log is enabled use the JDK MessageFormat class to create the log message and call the corresponding Log method, else do nothing.

Caveat: the last args has to be tested for being a Throwable and if so the corresponding Log method with (String, Throwable) signature.


> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9809-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 23:11:39 2010
Return-Path: <common-issues-return-9809-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15058 invoked from network); 27 Jul 2010 23:11:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 23:11:39 -0000
Received: (qmail 95829 invoked by uid 500); 27 Jul 2010 23:11:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95794 invoked by uid 500); 27 Jul 2010 23:11:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95786 invoked by uid 99); 27 Jul 2010 23:11:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 23:11:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 23:11:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6RNBHNu009704
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 23:11:17 GMT
Message-ID: <30399929.35081280272277685.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 19:11:17 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6745) adding some java doc to
 Server.RpcMetrics, UGI
In-Reply-To: <5413783.17731272909726168.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6745:
-----------------------------------------

    Status: Patch Available  (was: Open)

> adding some java doc to Server.RpcMetrics, UGI
> ----------------------------------------------
>
>                 Key: HADOOP-6745
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6745
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>            Priority: Minor
>         Attachments: HADOOP-6745-BP20-1.patch, HADOOP-6745-BP20-2.patch, HADOOP-6745.1.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9810-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 23:11:41 2010
Return-Path: <common-issues-return-9810-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15137 invoked from network); 27 Jul 2010 23:11:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 23:11:41 -0000
Received: (qmail 96132 invoked by uid 500); 27 Jul 2010 23:11:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96081 invoked by uid 500); 27 Jul 2010 23:11:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96073 invoked by uid 99); 27 Jul 2010 23:11:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 23:11:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 23:11:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6RNBHBw009695
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 23:11:17 GMT
Message-ID: <23736818.35051280272277014.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 19:11:17 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6745) adding some java doc to
 Server.RpcMetrics, UGI
In-Reply-To: <5413783.17731272909726168.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6745:
-----------------------------------------

    Attachment: HADOOP-6745.1.patch

Patch for trunk.

> adding some java doc to Server.RpcMetrics, UGI
> ----------------------------------------------
>
>                 Key: HADOOP-6745
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6745
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>            Priority: Minor
>         Attachments: HADOOP-6745-BP20-1.patch, HADOOP-6745-BP20-2.patch, HADOOP-6745.1.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9811-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jul 27 23:41:44 2010
Return-Path: <common-issues-return-9811-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22376 invoked from network); 27 Jul 2010 23:41:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jul 2010 23:41:43 -0000
Received: (qmail 16011 invoked by uid 500); 27 Jul 2010 23:41:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15957 invoked by uid 500); 27 Jul 2010 23:41:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15820 invoked by uid 99); 27 Jul 2010 23:41:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 23:41:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jul 2010 23:41:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6RNfJdX009925
	for <common-issues@hadoop.apache.org>; Tue, 27 Jul 2010 23:41:19 GMT
Message-ID: <4213030.35661280274078960.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 19:41:18 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6745) adding some java doc to
 Server.RpcMetrics, UGI
In-Reply-To: <5413783.17731272909726168.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892988#action_12892988 ] 

Hadoop QA commented on HADOOP-6745:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12450644/HADOOP-6745.1.patch
  against trunk revision 979785.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/642/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/642/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/642/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/642/console

This message is automatically generated.

> adding some java doc to Server.RpcMetrics, UGI
> ----------------------------------------------
>
>                 Key: HADOOP-6745
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6745
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>            Priority: Minor
>         Attachments: HADOOP-6745-BP20-1.patch, HADOOP-6745-BP20-2.patch, HADOOP-6745.1.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9812-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 00:51:42 2010
Return-Path: <common-issues-return-9812-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45641 invoked from network); 28 Jul 2010 00:51:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 00:51:42 -0000
Received: (qmail 57264 invoked by uid 500); 28 Jul 2010 00:51:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57220 invoked by uid 500); 28 Jul 2010 00:51:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57212 invoked by uid 99); 28 Jul 2010 00:51:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 00:51:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 00:51:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6S0pHLj010382
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 00:51:17 GMT
Message-ID: <3177293.36431280278277486.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 20:51:17 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6745) adding some java doc to
 Server.RpcMetrics, UGI
In-Reply-To: <5413783.17731272909726168.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893007#action_12893007 ] 

Jitendra Nath Pandey commented on HADOOP-6745:
----------------------------------------------

The javadoc warnings are in KerberosName.java and SecurityUtil.java which are related to kerberos API and are unrelated to this patch.

No tests included because this patch changes only javadocs.

> adding some java doc to Server.RpcMetrics, UGI
> ----------------------------------------------
>
>                 Key: HADOOP-6745
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6745
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>            Priority: Minor
>         Attachments: HADOOP-6745-BP20-1.patch, HADOOP-6745-BP20-2.patch, HADOOP-6745.1.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9813-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 00:53:39 2010
Return-Path: <common-issues-return-9813-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45824 invoked from network); 28 Jul 2010 00:53:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 00:53:38 -0000
Received: (qmail 57979 invoked by uid 500); 28 Jul 2010 00:53:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57930 invoked by uid 500); 28 Jul 2010 00:53:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57920 invoked by uid 99); 28 Jul 2010 00:53:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 00:53:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 00:53:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6S0rGZs010422
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 00:53:17 GMT
Message-ID: <9199873.36531280278396836.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 20:53:16 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6482) GenericOptionsParser constructor that
 takes Options and String[] ignores options
In-Reply-To: <947197260.111021262909095193.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6482:
--------------------------------

    Attachment: hadoop-6482-1.patch

Patch attached. Fixes this and adds a test. 

Not sure why users need to pass in generic options to GenericOptionsParser given that we hard-code the hadoop options, but at least this fix will allow us to pull the hadoop-specific options out of this class. 

> GenericOptionsParser constructor that takes Options and String[] ignores options
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6482
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6482
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.22.0
>            Reporter: Chris Wilkes
>            Priority: Minor
>         Attachments: hadoop-6482-1.patch
>
>   Original Estimate: 0.08h
>  Remaining Estimate: 0.08h
>
> This constructor:
>   public GenericOptionsParser(Options opts, String[] args) {
>     this(new Configuration(), new Options(), args);
>   }
> should do a this(new Configuration(), opts, args)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9814-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 00:53:39 2010
Return-Path: <common-issues-return-9814-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45842 invoked from network); 28 Jul 2010 00:53:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 00:53:38 -0000
Received: (qmail 58120 invoked by uid 500); 28 Jul 2010 00:53:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57945 invoked by uid 500); 28 Jul 2010 00:53:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57931 invoked by uid 99); 28 Jul 2010 00:53:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 00:53:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 00:53:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6S0rHXm010434
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 00:53:17 GMT
Message-ID: <8344115.36571280278397668.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 20:53:17 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6482) GenericOptionsParser constructor
 that takes Options and String[] ignores options
In-Reply-To: <947197260.111021262909095193.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins reassigned HADOOP-6482:
-----------------------------------

    Assignee: Eli Collins

> GenericOptionsParser constructor that takes Options and String[] ignores options
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6482
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6482
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.22.0
>            Reporter: Chris Wilkes
>            Assignee: Eli Collins
>            Priority: Minor
>         Attachments: hadoop-6482-1.patch
>
>   Original Estimate: 0.08h
>  Remaining Estimate: 0.08h
>
> This constructor:
>   public GenericOptionsParser(Options opts, String[] args) {
>     this(new Configuration(), new Options(), args);
>   }
> should do a this(new Configuration(), opts, args)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9815-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 00:53:40 2010
Return-Path: <common-issues-return-9815-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45918 invoked from network); 28 Jul 2010 00:53:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 00:53:39 -0000
Received: (qmail 58279 invoked by uid 500); 28 Jul 2010 00:53:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58249 invoked by uid 500); 28 Jul 2010 00:53:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58241 invoked by uid 99); 28 Jul 2010 00:53:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 00:53:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 00:53:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6S0rJp7010458
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 00:53:19 GMT
Message-ID: <32042265.36651280278399284.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 20:53:19 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6482) GenericOptionsParser constructor that
 takes Options and String[] ignores options
In-Reply-To: <947197260.111021262909095193.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6482:
--------------------------------

    Status: Patch Available  (was: Open)

> GenericOptionsParser constructor that takes Options and String[] ignores options
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6482
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6482
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.22.0
>            Reporter: Chris Wilkes
>            Assignee: Eli Collins
>            Priority: Minor
>         Attachments: hadoop-6482-1.patch
>
>   Original Estimate: 0.08h
>  Remaining Estimate: 0.08h
>
> This constructor:
>   public GenericOptionsParser(Options opts, String[] args) {
>     this(new Configuration(), new Options(), args);
>   }
> should do a this(new Configuration(), opts, args)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9816-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 00:55:38 2010
Return-Path: <common-issues-return-9816-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46104 invoked from network); 28 Jul 2010 00:55:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 00:55:38 -0000
Received: (qmail 58910 invoked by uid 500); 28 Jul 2010 00:55:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58882 invoked by uid 500); 28 Jul 2010 00:55:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58874 invoked by uid 99); 28 Jul 2010 00:55:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 00:55:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 00:55:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6S0tGCF010479
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 00:55:17 GMT
Message-ID: <9485375.36681280278516950.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 20:55:16 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6745) adding some java doc to
 Server.RpcMetrics, UGI
In-Reply-To: <5413783.17731272909726168.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6745:
--------------------------------

    Hadoop Flags: [Reviewed]

+1

> adding some java doc to Server.RpcMetrics, UGI
> ----------------------------------------------
>
>                 Key: HADOOP-6745
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6745
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>            Priority: Minor
>         Attachments: HADOOP-6745-BP20-1.patch, HADOOP-6745-BP20-2.patch, HADOOP-6745.1.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9817-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 00:58:41 2010
Return-Path: <common-issues-return-9817-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46297 invoked from network); 28 Jul 2010 00:58:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 00:58:40 -0000
Received: (qmail 59853 invoked by uid 500); 28 Jul 2010 00:58:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59816 invoked by uid 500); 28 Jul 2010 00:58:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59808 invoked by uid 99); 28 Jul 2010 00:58:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 00:58:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 00:58:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6S0wGPu010516
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 00:58:17 GMT
Message-ID: <6895469.36771280278696929.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 20:58:16 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6745) adding some java doc to
 Server.RpcMetrics, UGI
In-Reply-To: <5413783.17731272909726168.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6745:
--------------------------------

           Status: Resolved  (was: Patch Available)
    Fix Version/s: 0.22.0
       Resolution: Fixed

I've committed this.  Thanks Jitendra and Boris.  Resolving as fixed.

> adding some java doc to Server.RpcMetrics, UGI
> ----------------------------------------------
>
>                 Key: HADOOP-6745
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6745
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6745-BP20-1.patch, HADOOP-6745-BP20-2.patch, HADOOP-6745.1.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9818-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 01:00:41 2010
Return-Path: <common-issues-return-9818-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47129 invoked from network); 28 Jul 2010 01:00:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 01:00:41 -0000
Received: (qmail 60743 invoked by uid 500); 28 Jul 2010 01:00:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60684 invoked by uid 500); 28 Jul 2010 01:00:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60676 invoked by uid 99); 28 Jul 2010 01:00:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 01:00:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 01:00:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6S10GPw010557
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 01:00:17 GMT
Message-ID: <9570114.36801280278816895.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 21:00:16 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6745) adding some java doc to
 Server.RpcMetrics, UGI
In-Reply-To: <5413783.17731272909726168.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6745:
--------------------------------

    Affects Version/s: 0.22.0
          Component/s: ipc

> adding some java doc to Server.RpcMetrics, UGI
> ----------------------------------------------
>
>                 Key: HADOOP-6745
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6745
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6745-BP20-1.patch, HADOOP-6745-BP20-2.patch, HADOOP-6745.1.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9819-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 01:22:38 2010
Return-Path: <common-issues-return-9819-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4700 invoked from network); 28 Jul 2010 01:22:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 01:22:38 -0000
Received: (qmail 67497 invoked by uid 500); 28 Jul 2010 01:22:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67449 invoked by uid 500); 28 Jul 2010 01:22:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67441 invoked by uid 99); 28 Jul 2010 01:22:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 01:22:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 01:22:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6S1MG8u010805
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 01:22:17 GMT
Message-ID: <8986297.37431280280136752.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 21:22:16 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6482) GenericOptionsParser constructor
 that takes Options and String[] ignores options
In-Reply-To: <947197260.111021262909095193.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893024#action_12893024 ] 

Hadoop QA commented on HADOOP-6482:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12450655/hadoop-6482-1.patch
  against trunk revision 979919.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/643/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/643/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/643/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/643/console

This message is automatically generated.

> GenericOptionsParser constructor that takes Options and String[] ignores options
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6482
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6482
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.22.0
>            Reporter: Chris Wilkes
>            Assignee: Eli Collins
>            Priority: Minor
>         Attachments: hadoop-6482-1.patch
>
>   Original Estimate: 0.08h
>  Remaining Estimate: 0.08h
>
> This constructor:
>   public GenericOptionsParser(Options opts, String[] args) {
>     this(new Configuration(), new Options(), args);
>   }
> should do a this(new Configuration(), opts, args)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9820-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 01:47:41 2010
Return-Path: <common-issues-return-9820-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49977 invoked from network); 28 Jul 2010 01:47:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 01:47:40 -0000
Received: (qmail 80496 invoked by uid 500); 28 Jul 2010 01:47:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79748 invoked by uid 500); 28 Jul 2010 01:47:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79601 invoked by uid 99); 28 Jul 2010 01:47:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 01:47:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 01:47:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6S1lI1D011085
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 01:47:18 GMT
Message-ID: <10231472.38071280281638807.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 21:47:18 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6885) Fix java doc warnings in Groups and
 RefreshUserMappingsProtocol
In-Reply-To: <14122960.38041280281637802.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6885:
--------------------------------

    Attachment: hadoop-6885-1.patch

Patch attached.

> Fix java doc warnings in Groups and RefreshUserMappingsProtocol
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6885
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6885
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.20.2
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6885-1.patch
>
>
> There are a couple java docs warnings in Groups and RefreshUserMappingsProtocol.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9821-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 01:47:42 2010
Return-Path: <common-issues-return-9821-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50148 invoked from network); 28 Jul 2010 01:47:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 01:47:42 -0000
Received: (qmail 81036 invoked by uid 500); 28 Jul 2010 01:47:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80991 invoked by uid 500); 28 Jul 2010 01:47:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80983 invoked by uid 99); 28 Jul 2010 01:47:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 01:47:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 01:47:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6S1lHln011076
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 01:47:18 GMT
Message-ID: <14122960.38041280281637802.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 21:47:17 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6885) Fix java doc warnings in Groups and
 RefreshUserMappingsProtocol
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Fix java doc warnings in Groups and RefreshUserMappingsProtocol
---------------------------------------------------------------

                 Key: HADOOP-6885
                 URL: https://issues.apache.org/jira/browse/HADOOP-6885
             Project: Hadoop Common
          Issue Type: Bug
          Components: security
    Affects Versions: 0.20.2
            Reporter: Eli Collins
            Assignee: Eli Collins
             Fix For: 0.22.0
         Attachments: hadoop-6885-1.patch

There are a couple java docs warnings in Groups and RefreshUserMappingsProtocol.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9822-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 01:47:43 2010
Return-Path: <common-issues-return-9822-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50210 invoked from network); 28 Jul 2010 01:47:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 01:47:43 -0000
Received: (qmail 81187 invoked by uid 500); 28 Jul 2010 01:47:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81148 invoked by uid 500); 28 Jul 2010 01:47:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81140 invoked by uid 99); 28 Jul 2010 01:47:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 01:47:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 01:47:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6S1lJS6011091
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 01:47:19 GMT
Message-ID: <30117899.38091280281639349.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 21:47:19 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6885) Fix java doc warnings in Groups and
 RefreshUserMappingsProtocol
In-Reply-To: <14122960.38041280281637802.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6885:
--------------------------------

    Status: Patch Available  (was: Open)

> Fix java doc warnings in Groups and RefreshUserMappingsProtocol
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6885
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6885
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.20.2
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6885-1.patch
>
>
> There are a couple java docs warnings in Groups and RefreshUserMappingsProtocol.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9823-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 01:49:38 2010
Return-Path: <common-issues-return-9823-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50456 invoked from network); 28 Jul 2010 01:49:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 01:49:37 -0000
Received: (qmail 81653 invoked by uid 500); 28 Jul 2010 01:49:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81619 invoked by uid 500); 28 Jul 2010 01:49:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81611 invoked by uid 99); 28 Jul 2010 01:49:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 01:49:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 01:49:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6S1nG1G011111
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 01:49:17 GMT
Message-ID: <17103418.38131280281756653.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 21:49:16 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6482) GenericOptionsParser constructor
 that takes Options and String[] ignores options
In-Reply-To: <947197260.111021262909095193.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893033#action_12893033 ] 

Eli Collins commented on HADOOP-6482:
-------------------------------------

javadoc warnings are due to HADOOP-6885.

> GenericOptionsParser constructor that takes Options and String[] ignores options
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6482
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6482
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.22.0
>            Reporter: Chris Wilkes
>            Assignee: Eli Collins
>            Priority: Minor
>         Attachments: hadoop-6482-1.patch
>
>   Original Estimate: 0.08h
>  Remaining Estimate: 0.08h
>
> This constructor:
>   public GenericOptionsParser(Options opts, String[] args) {
>     this(new Configuration(), new Options(), args);
>   }
> should do a this(new Configuration(), opts, args)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9824-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 01:59:38 2010
Return-Path: <common-issues-return-9824-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51423 invoked from network); 28 Jul 2010 01:59:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 01:59:38 -0000
Received: (qmail 83110 invoked by uid 500); 28 Jul 2010 01:59:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83081 invoked by uid 500); 28 Jul 2010 01:59:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83073 invoked by uid 99); 28 Jul 2010 01:59:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 01:59:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 01:59:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6S1xHBc011171
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 01:59:17 GMT
Message-ID: <32750619.38211280282357283.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 21:59:17 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6482) GenericOptionsParser constructor that
 takes Options and String[] ignores options
In-Reply-To: <947197260.111021262909095193.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6482:
--------------------------------

    Attachment: hadoop-6482-2.patch

Updated the patch to include a test that shows that GenericOptionsParser will use multiple -conf arguments. This is required when using GenericOptionsParser to create a Configuration using multiple configuration files (ie post project split). 

> GenericOptionsParser constructor that takes Options and String[] ignores options
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6482
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6482
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.22.0
>            Reporter: Chris Wilkes
>            Assignee: Eli Collins
>            Priority: Minor
>         Attachments: hadoop-6482-1.patch, hadoop-6482-2.patch
>
>   Original Estimate: 0.08h
>  Remaining Estimate: 0.08h
>
> This constructor:
>   public GenericOptionsParser(Options opts, String[] args) {
>     this(new Configuration(), new Options(), args);
>   }
> should do a this(new Configuration(), opts, args)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9825-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 02:19:38 2010
Return-Path: <common-issues-return-9825-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59544 invoked from network); 28 Jul 2010 02:19:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 02:19:38 -0000
Received: (qmail 99513 invoked by uid 500); 28 Jul 2010 02:19:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99482 invoked by uid 500); 28 Jul 2010 02:19:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99474 invoked by uid 99); 28 Jul 2010 02:19:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 02:19:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 02:19:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6S2JGAU011275
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 02:19:17 GMT
Message-ID: <28890514.38321280283556675.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 22:19:16 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6885) Fix java doc warnings in Groups and
 RefreshUserMappingsProtocol
In-Reply-To: <14122960.38041280281637802.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893039#action_12893039 ] 

Hadoop QA commented on HADOOP-6885:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12450658/hadoop-6885-1.patch
  against trunk revision 979919.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/644/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/644/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/644/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/644/console

This message is automatically generated.

> Fix java doc warnings in Groups and RefreshUserMappingsProtocol
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6885
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6885
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.20.2
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6885-1.patch
>
>
> There are a couple java docs warnings in Groups and RefreshUserMappingsProtocol.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9826-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 03:19:46 2010
Return-Path: <common-issues-return-9826-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82176 invoked from network); 28 Jul 2010 03:19:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 03:19:45 -0000
Received: (qmail 13757 invoked by uid 500); 28 Jul 2010 03:19:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13617 invoked by uid 500); 28 Jul 2010 03:19:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13070 invoked by uid 99); 28 Jul 2010 03:19:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 03:19:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 03:19:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6S3JIZH011625
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 03:19:18 GMT
Message-ID: <20555059.38561280287158560.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 23:19:18 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6869) Functionality to creating file or
 folder on a remote daemon side
In-Reply-To: <1398369.499011279732010273.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6869:
---------------------------------------

         Summary: Functionality to creating file or folder on a remote daemon side  (was: [Herriot] Implement a functionality for creating either file or folder in task attempt  folder while job is running.)
    Hadoop Flags: [Reviewed]
      Issue Type: New Feature  (was: Task)

> Functionality to creating file or folder on a remote daemon side
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6869
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6869
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6869-ydist-security.patch, 6869-ydist-security.patch, HADOOP-6869.patch, HADOOP-6869.patch
>
>
> Functionality for creating either files or folders in task attempt folder while job is running. The functionality covers the following methods.
> 1. public void DaemonProtocol.createFile(String path, String fileName, boolean local) throws IOException; 
> It uses to create a file with full permissions.
> 2.   public void DaemonProtocol.createFile(String path, String fileName, FsPermission permission, boolean local) throws IOException; 
> It uses to create a file with given permissions.
> 3.   public void DaemonProtocol.createFolder(String path, String folderName, boolean local) throws IOException;
> It uses to create a file with full permissions.
> 4.   public void DaemonProtocol.createFolder(String path, String folderName, FsPermission permission, boolean local) throws IOException;
> It uses to create a folder with given permissions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9827-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 03:23:43 2010
Return-Path: <common-issues-return-9827-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83437 invoked from network); 28 Jul 2010 03:23:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 03:23:43 -0000
Received: (qmail 15086 invoked by uid 500); 28 Jul 2010 03:23:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14989 invoked by uid 500); 28 Jul 2010 03:23:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14977 invoked by uid 99); 28 Jul 2010 03:23:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 03:23:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 03:23:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6S3NHLN011671
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 03:23:17 GMT
Message-ID: <20618267.38701280287397002.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 23:23:17 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6869) Functionality to create file or
 folder on a remote daemon side
In-Reply-To: <1398369.499011279732010273.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6869:
---------------------------------------

    Summary: Functionality to create file or folder on a remote daemon side  (was: Functionality to creating file or folder on a remote daemon side)

> Functionality to create file or folder on a remote daemon side
> --------------------------------------------------------------
>
>                 Key: HADOOP-6869
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6869
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>             Fix For: 0.21.0
>
>         Attachments: 6869-ydist-security.patch, 6869-ydist-security.patch, HADOOP-6869.patch, HADOOP-6869.patch
>
>
> Functionality for creating either files or folders in task attempt folder while job is running. The functionality covers the following methods.
> 1. public void DaemonProtocol.createFile(String path, String fileName, boolean local) throws IOException; 
> It uses to create a file with full permissions.
> 2.   public void DaemonProtocol.createFile(String path, String fileName, FsPermission permission, boolean local) throws IOException; 
> It uses to create a file with given permissions.
> 3.   public void DaemonProtocol.createFolder(String path, String folderName, boolean local) throws IOException;
> It uses to create a file with full permissions.
> 4.   public void DaemonProtocol.createFolder(String path, String folderName, FsPermission permission, boolean local) throws IOException;
> It uses to create a folder with given permissions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9828-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 03:23:44 2010
Return-Path: <common-issues-return-9828-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83489 invoked from network); 28 Jul 2010 03:23:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 03:23:44 -0000
Received: (qmail 15212 invoked by uid 500); 28 Jul 2010 03:23:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15001 invoked by uid 500); 28 Jul 2010 03:23:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14990 invoked by uid 99); 28 Jul 2010 03:23:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 03:23:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 03:23:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6S3NK6J011707
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 03:23:20 GMT
Message-ID: <2226905.38821280287400880.JavaMail.jira@thor>
Date: Tue, 27 Jul 2010 23:23:20 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6869) Functionality to create file or
 folder on a remote daemon side
In-Reply-To: <1398369.499011279732010273.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6869:
---------------------------------------

           Status: Resolved  (was: Patch Available)
    Fix Version/s: 0.21.0
       Resolution: Fixed

I have just committed this. Thanks Vinay.

> Functionality to create file or folder on a remote daemon side
> --------------------------------------------------------------
>
>                 Key: HADOOP-6869
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6869
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>             Fix For: 0.21.0
>
>         Attachments: 6869-ydist-security.patch, 6869-ydist-security.patch, HADOOP-6869.patch, HADOOP-6869.patch
>
>
> Functionality for creating either files or folders in task attempt folder while job is running. The functionality covers the following methods.
> 1. public void DaemonProtocol.createFile(String path, String fileName, boolean local) throws IOException; 
> It uses to create a file with full permissions.
> 2.   public void DaemonProtocol.createFile(String path, String fileName, FsPermission permission, boolean local) throws IOException; 
> It uses to create a file with given permissions.
> 3.   public void DaemonProtocol.createFolder(String path, String folderName, boolean local) throws IOException;
> It uses to create a file with full permissions.
> 4.   public void DaemonProtocol.createFolder(String path, String folderName, FsPermission permission, boolean local) throws IOException;
> It uses to create a folder with given permissions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9829-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 05:46:41 2010
Return-Path: <common-issues-return-9829-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31972 invoked from network); 28 Jul 2010 05:46:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 05:46:41 -0000
Received: (qmail 75728 invoked by uid 500); 28 Jul 2010 05:46:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75581 invoked by uid 500); 28 Jul 2010 05:46:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75573 invoked by uid 99); 28 Jul 2010 05:46:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 05:46:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 05:46:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6S5kGqU013021
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 05:46:16 GMT
Message-ID: <29262092.39841280295976234.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 01:46:16 -0400 (EDT)
From: "Nicolas Spiegelberg (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6886) LocalFileSystem Needs
 createNonRecursive API
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

LocalFileSystem Needs createNonRecursive API
--------------------------------------------

                 Key: HADOOP-6886
                 URL: https://issues.apache.org/jira/browse/HADOOP-6886
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs
    Affects Versions: 0.20-append
            Reporter: Nicolas Spiegelberg
            Priority: Minor
             Fix For: 0.20-append


While running sanity check tests for HBASE-2312, I noticed that HDFS-617 did not include createNonRecursive() support for the LocalFileSystem.  This is a problem for HBase, which allows the user to run over the LocalFS instead of HDFS for local cluster testing.  I think this only affects 0.20-append, but may affect the trunk based upon how exactly FileContext handles non-recursive creates.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9830-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 05:48:43 2010
Return-Path: <common-issues-return-9830-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32475 invoked from network); 28 Jul 2010 05:48:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 05:48:42 -0000
Received: (qmail 78049 invoked by uid 500); 28 Jul 2010 05:48:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77960 invoked by uid 500); 28 Jul 2010 05:48:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77952 invoked by uid 99); 28 Jul 2010 05:48:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 05:48:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 05:48:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6S5mGBX013033
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 05:48:16 GMT
Message-ID: <3096463.39871280296096322.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 01:48:16 -0400 (EDT)
From: "Nicolas Spiegelberg (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6886) LocalFileSystem Needs
 createNonRecursive API
In-Reply-To: <29262092.39841280295976234.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicolas Spiegelberg updated HADOOP-6886:
----------------------------------------

    Attachment: HADOOP-6886_20-append.patch

> LocalFileSystem Needs createNonRecursive API
> --------------------------------------------
>
>                 Key: HADOOP-6886
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6886
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.20-append
>            Reporter: Nicolas Spiegelberg
>            Priority: Minor
>             Fix For: 0.20-append
>
>         Attachments: HADOOP-6886_20-append.patch
>
>
> While running sanity check tests for HBASE-2312, I noticed that HDFS-617 did not include createNonRecursive() support for the LocalFileSystem.  This is a problem for HBase, which allows the user to run over the LocalFS instead of HDFS for local cluster testing.  I think this only affects 0.20-append, but may affect the trunk based upon how exactly FileContext handles non-recursive creates.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9831-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 06:27:44 2010
Return-Path: <common-issues-return-9831-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53188 invoked from network); 28 Jul 2010 06:27:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 06:27:44 -0000
Received: (qmail 97763 invoked by uid 500); 28 Jul 2010 06:27:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97623 invoked by uid 500); 28 Jul 2010 06:27:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97613 invoked by uid 99); 28 Jul 2010 06:27:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 06:27:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 06:27:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6S6RKBi013388
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 06:27:21 GMT
Message-ID: <19176928.40391280298440823.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 02:27:20 -0400 (EDT)
From: "Tatu Saloranta (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6389) Add support for LZF compression
In-Reply-To: <21384827.1259089719643.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893076#action_12893076 ] 

Tatu Saloranta commented on HADOOP-6389:
----------------------------------------

Although I have not worked on integration, I have been able to get a simple reusable LZF block codec published, available from github (http://github.com/ning/compress), and main Maven repo (group com.ning, artifact compress-lzf). So at least simple part (codec itself) is ready for anyone with enough familiarity to handle full integration, ideally supporting access at least at block level (can read starting from block boundaries; blocks are byte-aligned, contain compress and uncompressed block lengths to support somewhat efficient skipping of blocks).


> Add support for LZF compression
> -------------------------------
>
>                 Key: HADOOP-6389
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6389
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: Tatu Saloranta
>
> (note: related to [HADOOP-4874])
> As per Doug's earlier comments, LZF does indeed look like a good compressor candidate for fast compression/decompression, good enough compression rate.
> From my testing it seems at least twice as fast at compression, and somewhat faster for decompressing than gzip.
> Code from [http://h2database.googlecode.com/svn/trunk/h2/src/main/org/h2/compress/] is applicable, and I have tested it with json data.
> I hope to have more to spend on this in near future, but if someone else gets to this first that'd be good too.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9832-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 11:02:48 2010
Return-Path: <common-issues-return-9832-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45922 invoked from network); 28 Jul 2010 11:02:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 11:02:48 -0000
Received: (qmail 44654 invoked by uid 500); 28 Jul 2010 11:02:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44498 invoked by uid 500); 28 Jul 2010 11:02:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44487 invoked by uid 99); 28 Jul 2010 11:02:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 11:02:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 11:02:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6SB2HIT015508
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 11:02:17 GMT
Message-ID: <3859622.42971280314937336.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 07:02:17 -0400 (EDT)
From: "Iyappan Srinivasan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6879) Jsch jar is required for ivy
 resolution to be included in the build
In-Reply-To: <11760779.565921279956469755.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6879?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Iyappan Srinivasan updated HADOOP-6879:
---------------------------------------

    Attachment: HADOOP-6879.patch

patch for trunk

> Jsch jar is required for ivy resolution to be included in the build
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6879
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6879
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.1
>            Reporter: Iyappan Srinivasan
>            Assignee: Iyappan Srinivasan
>         Attachments: 6879-ydist-security-patch.txt, HADOOP-6879.patch
>
>
> http://mvnrepository.com/
> com.jcraft =C2=BB jsch=20
> 0.1.42 version needs to be included in the build. This is  required for h=
erriot testcases using jsch.
> Please include this in ivy.
> jsch is originally located in http://www.jcraft.com/jsch/

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9833-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 17:14:44 2010
Return-Path: <common-issues-return-9833-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91295 invoked from network); 28 Jul 2010 17:14:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 17:14:43 -0000
Received: (qmail 94526 invoked by uid 500); 28 Jul 2010 17:14:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94437 invoked by uid 500); 28 Jul 2010 17:14:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94429 invoked by uid 99); 28 Jul 2010 17:14:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 17:14:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 17:14:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6SHEIH3018863
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 17:14:19 GMT
Message-ID: <12852131.48251280337258789.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 13:14:18 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893258#action_12893258 ] 

Luke Lu commented on HADOOP-6884:
---------------------------------

The debug log check is critical for performance sensitive paths. Due to lack of macros in Java, log wrappers doesn't really help if additional objects need to be created for the messages (the template stuff helps but not in many cases.)

-1 for yet another home brew log wrappers.
+1 for slf4j.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9834-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 17:52:59 2010
Return-Path: <common-issues-return-9834-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2113 invoked from network); 28 Jul 2010 17:52:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 17:52:58 -0000
Received: (qmail 33344 invoked by uid 500); 28 Jul 2010 17:52:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33235 invoked by uid 500); 28 Jul 2010 17:52:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33227 invoked by uid 99); 28 Jul 2010 17:52:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 17:52:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 17:52:57 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6SHqbb7019330
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 17:52:37 GMT
Message-ID: <3098376.49291280339557406.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 13:52:37 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893277#action_12893277 ] 

Suresh Srinivas commented on HADOOP-6884:
-----------------------------------------

Alejandro, with the methods you are proposing, an array or Objects created to pass the varargs. It is still expensive.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9835-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 19:22:44 2010
Return-Path: <common-issues-return-9835-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42806 invoked from network); 28 Jul 2010 19:22:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 19:22:44 -0000
Received: (qmail 28502 invoked by uid 500); 28 Jul 2010 19:22:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28441 invoked by uid 500); 28 Jul 2010 19:22:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28433 invoked by uid 99); 28 Jul 2010 19:22:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 19:22:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 19:22:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6SJMMvD020520
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 19:22:22 GMT
Message-ID: <8577479.51911280344942961.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 15:22:22 -0400 (EDT)
From: "Dmytro Molkov (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6870) Add FileSystem#listLocatedStatus to
 list a directory's content together with each file's block locations
In-Reply-To: <13042840.506371279749289722.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893320#action_12893320 ] 

Dmytro Molkov commented on HADOOP-6870:
---------------------------------------

I looked over the patch and overall looks good to me.
The only question I had is the hash/equals of the LocatedFileStatus. Do you ever see a need to have two instances of LocatedFileStatus that differ only in block locations be not equal? Because right now the equality is the same as the FileStatus equality and doesn't take block locations into account, right?

> Add FileSystem#listLocatedStatus to list a directory's content together with each file's block locations
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6870
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6870
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listFiles.patch, listFiles1.patch, listFiles2.patch, listFiles3.patch, listFiles4.patch
>
>
> This jira implements the new FileSystem API as proposed in HDFS-202. The new API aims to eliminate individual "getFileBlockLocations" calls to NN for each file in the input directory of a job. Instead, a file's block locations are returned together with FileStatus when listing a directory, thus improving getSplits performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9836-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 20:36:43 2010
Return-Path: <common-issues-return-9836-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85374 invoked from network); 28 Jul 2010 20:36:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 20:36:43 -0000
Received: (qmail 11688 invoked by uid 500); 28 Jul 2010 20:36:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11650 invoked by uid 500); 28 Jul 2010 20:36:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11642 invoked by uid 99); 28 Jul 2010 20:36:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 20:36:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 20:36:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6SKaIvq021367
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 20:36:18 GMT
Message-ID: <19271149.53661280349378947.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 16:36:18 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6870) Add FileSystem#listLocatedStatus to
 list a directory's content together with each file's block locations
In-Reply-To: <13042840.506371279749289722.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893356#action_12893356 ] 

Hairong Kuang commented on HADOOP-6870:
---------------------------------------

FileStatus uses a very loose equality semantics. It compares only path names. So I decided that LocatedFileStatus uses the same semantics.

> Add FileSystem#listLocatedStatus to list a directory's content together with each file's block locations
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6870
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6870
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listFiles.patch, listFiles1.patch, listFiles2.patch, listFiles3.patch, listFiles4.patch
>
>
> This jira implements the new FileSystem API as proposed in HDFS-202. The new API aims to eliminate individual "getFileBlockLocations" calls to NN for each file in the input directory of a job. Instead, a file's block locations are returned together with FileStatus when listing a directory, thus improving getSplits performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9837-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 21:13:45 2010
Return-Path: <common-issues-return-9837-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5202 invoked from network); 28 Jul 2010 21:13:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 21:13:37 -0000
Received: (qmail 50895 invoked by uid 500); 28 Jul 2010 21:13:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50832 invoked by uid 500); 28 Jul 2010 21:13:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50823 invoked by uid 99); 28 Jul 2010 21:13:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 21:13:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 21:13:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6SLDGPc021607
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 21:13:16 GMT
Message-ID: <25805095.53911280351596010.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 17:13:16 -0400 (EDT)
From: "Bharath Mundlapudi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6887) Need a separate metrics per garbage
 collector
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Need a separate metrics per garbage collector
---------------------------------------------

                 Key: HADOOP-6887
                 URL: https://issues.apache.org/jira/browse/HADOOP-6887
             Project: Hadoop Common
          Issue Type: Improvement
          Components: metrics
            Reporter: Bharath Mundlapudi
             Fix For: 0.22.0


In addition to current GC metrics which are the sum of all the collectors, Need separate metrics for monitoring young generation and old generation collections per collector w.r.t collection count and collection time. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9838-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 22:09:42 2010
Return-Path: <common-issues-return-9838-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71443 invoked from network); 28 Jul 2010 22:09:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 22:09:41 -0000
Received: (qmail 32786 invoked by uid 500); 28 Jul 2010 22:09:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32464 invoked by uid 500); 28 Jul 2010 22:09:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32362 invoked by uid 99); 28 Jul 2010 22:09:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 22:09:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 22:09:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6SM9Hpt022182
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 22:09:17 GMT
Message-ID: <11905003.55021280354956989.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 18:09:16 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6888) Being able to close all cached
 FileSystem objects for a given UGI
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Being able to close all cached FileSystem objects for a given UGI
-----------------------------------------------------------------

                 Key: HADOOP-6888
                 URL: https://issues.apache.org/jira/browse/HADOOP-6888
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Kan Zhang
            Assignee: Kan Zhang


This is the Common part of MAPREDUCE-1900. It adds a utility method to FileSystem that closes all cached filesystems for a given UGI.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9839-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 22:24:43 2010
Return-Path: <common-issues-return-9839-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77842 invoked from network); 28 Jul 2010 22:24:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 22:24:42 -0000
Received: (qmail 45456 invoked by uid 500); 28 Jul 2010 22:24:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45315 invoked by uid 500); 28 Jul 2010 22:24:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45305 invoked by uid 99); 28 Jul 2010 22:24:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 22:24:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 22:24:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6SMOLnu022375
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 22:24:21 GMT
Message-ID: <2062239.55341280355861372.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 18:24:21 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6888) Being able to close all cached
 FileSystem objects for a given UGI
In-Reply-To: <11905003.55021280354956989.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6888:
------------------------------

    Attachment: c6888-02.patch

> Being able to close all cached FileSystem objects for a given UGI
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6888
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6888
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6888-02.patch
>
>
> This is the Common part of MAPREDUCE-1900. It adds a utility method to FileSystem that closes all cached filesystems for a given UGI.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9840-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 22:24:44 2010
Return-Path: <common-issues-return-9840-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77905 invoked from network); 28 Jul 2010 22:24:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 22:24:43 -0000
Received: (qmail 46162 invoked by uid 500); 28 Jul 2010 22:24:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45398 invoked by uid 500); 28 Jul 2010 22:24:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45317 invoked by uid 99); 28 Jul 2010 22:24:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 22:24:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 22:24:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6SMOIRd022348
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 22:24:18 GMT
Message-ID: <26726400.55251280355858032.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 18:24:18 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6888) Being able to close all cached
 FileSystem objects for a given UGI
In-Reply-To: <11905003.55021280354956989.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6888:
------------------------------

    Component/s: fs
                 security

> Being able to close all cached FileSystem objects for a given UGI
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6888
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6888
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6888-02.patch
>
>
> This is the Common part of MAPREDUCE-1900. It adds a utility method to FileSystem that closes all cached filesystems for a given UGI.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9841-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 22:27:41 2010
Return-Path: <common-issues-return-9841-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79796 invoked from network); 28 Jul 2010 22:27:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 22:27:41 -0000
Received: (qmail 52999 invoked by uid 500); 28 Jul 2010 22:27:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52767 invoked by uid 500); 28 Jul 2010 22:27:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52759 invoked by uid 99); 28 Jul 2010 22:27:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 22:27:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 22:27:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6SMRHc4022392
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 22:27:17 GMT
Message-ID: <28327311.55381280356037167.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 18:27:17 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6888) Being able to close all cached
 FileSystem objects for a given UGI
In-Reply-To: <11905003.55021280354956989.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6888:
------------------------------

    Status: Patch Available  (was: Open)

> Being able to close all cached FileSystem objects for a given UGI
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6888
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6888
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6888-02.patch
>
>
> This is the Common part of MAPREDUCE-1900. It adds a utility method to FileSystem that closes all cached filesystems for a given UGI.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9842-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 22:51:42 2010
Return-Path: <common-issues-return-9842-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91969 invoked from network); 28 Jul 2010 22:51:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 22:51:42 -0000
Received: (qmail 80694 invoked by uid 500); 28 Jul 2010 22:51:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80594 invoked by uid 500); 28 Jul 2010 22:51:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80583 invoked by uid 99); 28 Jul 2010 22:51:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 22:51:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 22:51:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6SMpHq5022776
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 22:51:17 GMT
Message-ID: <8153581.56301280357477560.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 18:51:17 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6888) Being able to close all cached
 FileSystem objects for a given UGI
In-Reply-To: <11905003.55021280354956989.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893408#action_12893408 ] 

Hadoop QA commented on HADOOP-6888:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12450763/c6888-02.patch
  against trunk revision 979944.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/645/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/645/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/645/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/645/console

This message is automatically generated.

> Being able to close all cached FileSystem objects for a given UGI
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6888
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6888
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6888-02.patch
>
>
> This is the Common part of MAPREDUCE-1900. It adds a utility method to FileSystem that closes all cached filesystems for a given UGI.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9843-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 22:54:41 2010
Return-Path: <common-issues-return-9843-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95055 invoked from network); 28 Jul 2010 22:54:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 22:54:41 -0000
Received: (qmail 83862 invoked by uid 500); 28 Jul 2010 22:54:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83805 invoked by uid 500); 28 Jul 2010 22:54:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83796 invoked by uid 99); 28 Jul 2010 22:54:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 22:54:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 22:54:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6SMsHMZ022806
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 22:54:17 GMT
Message-ID: <9226066.56351280357657079.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 18:54:17 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6870) Add FileSystem#listLocatedStatus to
 list a directory's content together with each file's block locations
In-Reply-To: <13042840.506371279749289722.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893410#action_12893410 ] 

Doug Cutting commented on HADOOP-6870:
--------------------------------------

Looks fine to me now.  Thanks!

> Add FileSystem#listLocatedStatus to list a directory's content together with each file's block locations
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6870
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6870
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listFiles.patch, listFiles1.patch, listFiles2.patch, listFiles3.patch, listFiles4.patch
>
>
> This jira implements the new FileSystem API as proposed in HDFS-202. The new API aims to eliminate individual "getFileBlockLocations" calls to NN for each file in the input directory of a job. Instead, a file's block locations are returned together with FileStatus when listing a directory, thus improving getSplits performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9844-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 22:59:41 2010
Return-Path: <common-issues-return-9844-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98886 invoked from network); 28 Jul 2010 22:59:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 22:59:41 -0000
Received: (qmail 90633 invoked by uid 500); 28 Jul 2010 22:59:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90595 invoked by uid 500); 28 Jul 2010 22:59:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90586 invoked by uid 99); 28 Jul 2010 22:59:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 22:59:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 22:59:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6SMxGcp022841
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 22:59:16 GMT
Message-ID: <20229717.56421280357956844.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 18:59:16 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6888) Being able to close all cached
 FileSystem objects for a given UGI
In-Reply-To: <11905003.55021280354956989.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893412#action_12893412 ] 

Kan Zhang commented on HADOOP-6888:
-----------------------------------

The javadoc warnings are due to KerberosName and SecurityUtil, and unrelated to this patch.

> Being able to close all cached FileSystem objects for a given UGI
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6888
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6888
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6888-02.patch
>
>
> This is the Common part of MAPREDUCE-1900. It adds a utility method to FileSystem that closes all cached filesystems for a given UGI.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9845-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jul 28 23:08:41 2010
Return-Path: <common-issues-return-9845-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7649 invoked from network); 28 Jul 2010 23:08:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Jul 2010 23:08:41 -0000
Received: (qmail 3158 invoked by uid 500); 28 Jul 2010 23:08:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3113 invoked by uid 500); 28 Jul 2010 23:08:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3105 invoked by uid 99); 28 Jul 2010 23:08:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 23:08:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Jul 2010 23:08:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6SN8G0m022944
	for <common-issues@hadoop.apache.org>; Wed, 28 Jul 2010 23:08:16 GMT
Message-ID: <18464067.56511280358496537.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 19:08:16 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6873) using delegation token over hftp
 for long running clients (part of hdfs 1296)
In-Reply-To: <3672204.508361279757392441.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893416#action_12893416 ] 

Jitendra Nath Pandey commented on HADOOP-6873:
----------------------------------------------

+1 for the patch.

> using delegation token over hftp for long running clients (part of hdfs 1296)
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-6873
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6873
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6873-2.patch, HADOOP-6873.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9846-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 00:17:42 2010
Return-Path: <common-issues-return-9846-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45250 invoked from network); 29 Jul 2010 00:17:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 00:17:42 -0000
Received: (qmail 71481 invoked by uid 500); 29 Jul 2010 00:17:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71408 invoked by uid 500); 29 Jul 2010 00:17:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71400 invoked by uid 99); 29 Jul 2010 00:17:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 00:17:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 00:17:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6T0HHuf023467
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 00:17:18 GMT
Message-ID: <22033978.57571280362637970.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 20:17:17 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6870) Add FileSystem#listLocatedStatus to
 list a directory's content together with each file's block locations
In-Reply-To: <13042840.506371279749289722.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6870:
----------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

I've committed this!

> Add FileSystem#listLocatedStatus to list a directory's content together with each file's block locations
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6870
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6870
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listFiles.patch, listFiles1.patch, listFiles2.patch, listFiles3.patch, listFiles4.patch
>
>
> This jira implements the new FileSystem API as proposed in HDFS-202. The new API aims to eliminate individual "getFileBlockLocations" calls to NN for each file in the input directory of a job. Instead, a file's block locations are returned together with FileStatus when listing a directory, thus improving getSplits performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9847-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 00:52:38 2010
Return-Path: <common-issues-return-9847-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66215 invoked from network); 29 Jul 2010 00:52:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 00:52:38 -0000
Received: (qmail 3242 invoked by uid 500); 29 Jul 2010 00:52:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3207 invoked by uid 500); 29 Jul 2010 00:52:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3199 invoked by uid 99); 29 Jul 2010 00:52:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 00:52:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 00:52:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6T0qHU1023791
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 00:52:17 GMT
Message-ID: <25869623.58341280364737040.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 20:52:17 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6656:
--------------------------------

    Status: Open  (was: Patch Available)

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6656-trunk-1.patch, 6656-trunk-2.patch, 6656-trunk-3.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9848-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 00:54:42 2010
Return-Path: <common-issues-return-9848-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66403 invoked from network); 29 Jul 2010 00:54:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 00:54:42 -0000
Received: (qmail 3985 invoked by uid 500); 29 Jul 2010 00:54:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3947 invoked by uid 500); 29 Jul 2010 00:54:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3939 invoked by uid 99); 29 Jul 2010 00:54:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 00:54:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 00:54:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6T0sIQ5023813
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 00:54:18 GMT
Message-ID: <7321566.58391280364858137.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 20:54:18 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6656:
--------------------------------

    Attachment: 6656-trunk-4.patch

Addressed Kan's comments..

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6656-trunk-1.patch, 6656-trunk-2.patch, 6656-trunk-3.patch, 6656-trunk-4.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9849-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 00:54:43 2010
Return-Path: <common-issues-return-9849-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66430 invoked from network); 29 Jul 2010 00:54:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 00:54:42 -0000
Received: (qmail 4193 invoked by uid 500); 29 Jul 2010 00:54:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4103 invoked by uid 500); 29 Jul 2010 00:54:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4052 invoked by uid 99); 29 Jul 2010 00:54:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 00:54:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 00:54:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6T0sJ4n023828
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 00:54:19 GMT
Message-ID: <16164879.58441280364859001.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 20:54:18 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6656:
--------------------------------

    Status: Patch Available  (was: Open)

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6656-trunk-1.patch, 6656-trunk-2.patch, 6656-trunk-3.patch, 6656-trunk-4.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9850-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 01:11:40 2010
Return-Path: <common-issues-return-9850-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75423 invoked from network); 29 Jul 2010 01:11:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 01:11:40 -0000
Received: (qmail 13166 invoked by uid 500); 29 Jul 2010 01:11:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13123 invoked by uid 500); 29 Jul 2010 01:11:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13115 invoked by uid 99); 29 Jul 2010 01:11:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 01:11:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 01:11:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6T1BJL8024154
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 01:11:19 GMT
Message-ID: <2806883.59221280365879584.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 21:11:19 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6656:
--------------------------------

    Status: Open  (was: Patch Available)

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6656-trunk-1.patch, 6656-trunk-2.patch, 6656-trunk-3.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9851-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 01:11:44 2010
Return-Path: <common-issues-return-9851-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75505 invoked from network); 29 Jul 2010 01:11:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 01:11:44 -0000
Received: (qmail 13335 invoked by uid 500); 29 Jul 2010 01:11:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13292 invoked by uid 500); 29 Jul 2010 01:11:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13284 invoked by uid 99); 29 Jul 2010 01:11:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 01:11:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 01:11:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6T1BKek024169
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 01:11:20 GMT
Message-ID: <17149370.59271280365880357.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 21:11:20 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6656:
--------------------------------

    Attachment: 6656-trunk-4.patch

This patch has some improved javadocs.

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6656-trunk-1.patch, 6656-trunk-2.patch, 6656-trunk-3.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9852-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 01:13:38 2010
Return-Path: <common-issues-return-9852-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77239 invoked from network); 29 Jul 2010 01:13:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 01:13:38 -0000
Received: (qmail 15116 invoked by uid 500); 29 Jul 2010 01:13:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15075 invoked by uid 500); 29 Jul 2010 01:13:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15067 invoked by uid 99); 29 Jul 2010 01:13:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 01:13:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 01:13:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6T1DHoA024201
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 01:13:17 GMT
Message-ID: <31867268.59371280365997544.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 21:13:17 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893464#action_12893464 ] 

Kan Zhang commented on HADOOP-6656:
-----------------------------------

+1.

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6656-trunk-1.patch, 6656-trunk-2.patch, 6656-trunk-3.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9853-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 01:13:40 2010
Return-Path: <common-issues-return-9853-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77303 invoked from network); 29 Jul 2010 01:13:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 01:13:40 -0000
Received: (qmail 15284 invoked by uid 500); 29 Jul 2010 01:13:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15241 invoked by uid 500); 29 Jul 2010 01:13:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15233 invoked by uid 99); 29 Jul 2010 01:13:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 01:13:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 01:13:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6T1DGfm024186
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 01:13:16 GMT
Message-ID: <18301121.59321280365996836.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 21:13:16 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6656:
--------------------------------

    Status: Patch Available  (was: Open)

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6656-trunk-1.patch, 6656-trunk-2.patch, 6656-trunk-3.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9854-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 01:22:41 2010
Return-Path: <common-issues-return-9854-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81872 invoked from network); 29 Jul 2010 01:22:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 01:22:40 -0000
Received: (qmail 22122 invoked by uid 500); 29 Jul 2010 01:22:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22094 invoked by uid 500); 29 Jul 2010 01:22:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22086 invoked by uid 99); 29 Jul 2010 01:22:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 01:22:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 01:22:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6T1MHDj024264
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 01:22:17 GMT
Message-ID: <10937874.59501280366537005.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 21:22:17 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893465#action_12893465 ] 

Hadoop QA commented on HADOOP-6656:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12450781/6656-trunk-4.patch
  against trunk revision 980271.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs.  The patch appears to introduce 1 new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/646/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/646/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/646/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/646/console

This message is automatically generated.

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6656-trunk-1.patch, 6656-trunk-2.patch, 6656-trunk-3.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9855-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 01:31:41 2010
Return-Path: <common-issues-return-9855-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86082 invoked from network); 29 Jul 2010 01:31:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 01:31:41 -0000
Received: (qmail 25928 invoked by uid 500); 29 Jul 2010 01:31:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25906 invoked by uid 500); 29 Jul 2010 01:31:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25898 invoked by uid 99); 29 Jul 2010 01:31:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 01:31:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 01:31:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6T1VKXm024337
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 01:31:20 GMT
Message-ID: <7714310.59671280367080449.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 21:31:20 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6656:
--------------------------------

    Status: Patch Available  (was: Open)

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6656-trunk-1.patch, 6656-trunk-2.patch, 6656-trunk-3.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9856-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 01:31:41 2010
Return-Path: <common-issues-return-9856-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86122 invoked from network); 29 Jul 2010 01:31:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 01:31:41 -0000
Received: (qmail 26114 invoked by uid 500); 29 Jul 2010 01:31:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26062 invoked by uid 500); 29 Jul 2010 01:31:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26054 invoked by uid 99); 29 Jul 2010 01:31:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 01:31:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 01:31:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6T1VHcg024301
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 01:31:17 GMT
Message-ID: <17837437.59551280367077348.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 21:31:17 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6656:
--------------------------------

    Attachment: 6656-trunk-4.patch

Attaching a patch fixing the findbugs warning. The javadoc warning is unrelated.

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6656-trunk-1.patch, 6656-trunk-2.patch, 6656-trunk-3.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9857-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 01:31:43 2010
Return-Path: <common-issues-return-9857-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86174 invoked from network); 29 Jul 2010 01:31:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 01:31:43 -0000
Received: (qmail 26272 invoked by uid 500); 29 Jul 2010 01:31:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26240 invoked by uid 500); 29 Jul 2010 01:31:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26232 invoked by uid 99); 29 Jul 2010 01:31:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 01:31:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 01:31:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6T1VJq5024322
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 01:31:19 GMT
Message-ID: <32918396.59621280367079549.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 21:31:19 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6656:
--------------------------------

    Status: Open  (was: Patch Available)

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6656-trunk-1.patch, 6656-trunk-2.patch, 6656-trunk-3.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9858-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 01:40:38 2010
Return-Path: <common-issues-return-9858-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88480 invoked from network); 29 Jul 2010 01:40:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 01:40:38 -0000
Received: (qmail 29729 invoked by uid 500); 29 Jul 2010 01:40:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29592 invoked by uid 500); 29 Jul 2010 01:40:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29583 invoked by uid 99); 29 Jul 2010 01:40:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 01:40:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 01:40:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6T1eGqh024398
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 01:40:16 GMT
Message-ID: <32377991.59741280367616674.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 21:40:16 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893469#action_12893469 ] 

Hadoop QA commented on HADOOP-6656:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12450783/6656-trunk-4.patch
  against trunk revision 980271.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs.  The patch appears to introduce 1 new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/647/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/647/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/647/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/647/console

This message is automatically generated.

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6656-trunk-1.patch, 6656-trunk-2.patch, 6656-trunk-3.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9859-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 01:58:39 2010
Return-Path: <common-issues-return-9859-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90175 invoked from network); 29 Jul 2010 01:58:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 01:58:39 -0000
Received: (qmail 37186 invoked by uid 500); 29 Jul 2010 01:58:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37136 invoked by uid 500); 29 Jul 2010 01:58:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37128 invoked by uid 99); 29 Jul 2010 01:58:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 01:58:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 01:58:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6T1wHDg024507
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 01:58:17 GMT
Message-ID: <15750809.59891280368697318.JavaMail.jira@thor>
Date: Wed, 28 Jul 2010 21:58:17 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893473#action_12893473 ] 

Hadoop QA commented on HADOOP-6656:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12450785/6656-trunk-4.patch
  against trunk revision 980271.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs.  The patch appears to introduce 1 new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/648/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/648/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/648/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/648/console

This message is automatically generated.

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6656-trunk-1.patch, 6656-trunk-2.patch, 6656-trunk-3.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9860-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 06:05:44 2010
Return-Path: <common-issues-return-9860-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77597 invoked from network); 29 Jul 2010 06:05:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 06:05:44 -0000
Received: (qmail 76090 invoked by uid 500); 29 Jul 2010 06:05:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75960 invoked by uid 500); 29 Jul 2010 06:05:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75952 invoked by uid 99); 29 Jul 2010 06:05:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 06:05:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 06:05:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6T65GZF026586
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 06:05:17 GMT
Message-ID: <29859752.62201280383516905.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 02:05:16 -0400 (EDT)
From: "Meng Mao (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6092) No space left on device
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893521#action_12893521 ] 

Meng Mao commented on HADOOP-6092:
----------------------------------

We've been seeing what we think is the same thing in Hadoop 0.20.1.

Here's what the dfs health page says:
Configured Capacity	 :	 77.22 TB
 DFS Used	 :	 58.18 TB
 Non DFS Used	 :	 5.69 TB
 DFS Remaining	 :	 13.34 TB
 DFS Used%	 :	 75.35 %
 DFS Remaining%	 :	 17.27 %
 Live Nodes 	 :	 45
 Dead Nodes 	 :	 0

Even the most occupied node has just under 80% used space.

For both Java and Streaming jobs, we've seen exceptions like the below, which is from a streaming run:
Failed to merge on the local FSorg.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
	at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:192)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
	at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
	at java.io.DataOutputStream.write(DataOutputStream.java:90)
	at org.apache.hadoop.mapred.IFileOutputStream.write(IFileOutputStream.java:84)
	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
	at java.io.DataOutputStream.write(DataOutputStream.java:90)
	at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:218)
	at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:157)
	at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(ReduceTask.java:2533)
Caused by: java.io.IOException: No space left on device
	at java.io.FileOutputStream.writeBytes(Native Method)
	at java.io.FileOutputStream.write(FileOutputStream.java:260)
	at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:190)
	... 10 more

This job had about 100GB of input, and I'm guessing around that level of data being passed into the reducers. During the job execution, watching dfsnodelist.jsp?whatNodes=LIVE doesn't indicate much growth, certainly not to the point where a node hits 100%. Not that I know that kind of tracking of disk use is actually supported.

AFAIK, we have the physical disk space to match what the dfs pages indicate. 

What can we do to investigate how we're running out of space?

> No space left on device
> -----------------------
>
>                 Key: HADOOP-6092
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6092
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.19.0
>         Environment: ubuntu0.8.4
>            Reporter: mawanqiang
>
> Exception in thread "main" org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
>         at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
>         at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
>         at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
>         at java.io.FilterOutputStream.close(FilterOutputStream.java:140)
>         at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
>         at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
>         at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.close(ChecksumFileSystem.java:339)
>         at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
>         at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
>         at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:825)
>         at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1142)
>         at org.apache.nutch.indexer.Indexer.index(Indexer.java:72)
>         at org.apache.nutch.indexer.Indexer.run(Indexer.java:92)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>         at org.apache.nutch.indexer.Indexer.main(Indexer.java:101)
> Caused by: java.io.IOException: No space left on device
>         at java.io.FileOutputStream.writeBytes(Native Method)
>         at java.io.FileOutputStream.write(FileOutputStream.java:260)
>         at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:197)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9861-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 06:10:44 2010
Return-Path: <common-issues-return-9861-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78927 invoked from network); 29 Jul 2010 06:10:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 06:10:44 -0000
Received: (qmail 77818 invoked by uid 500); 29 Jul 2010 06:10:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77656 invoked by uid 500); 29 Jul 2010 06:10:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77635 invoked by uid 99); 29 Jul 2010 06:10:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 06:10:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 06:10:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6T6AJ78026677
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 06:10:19 GMT
Message-ID: <64627.62481280383819557.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 02:10:19 -0400 (EDT)
From: "Meng Mao (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6092) No space left on device
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893523#action_12893523 ] 

Meng Mao commented on HADOOP-6092:
----------------------------------

one more comment --
we have a nightly process composed of Java jobs, that runs over roughly the same data set size each time. One of the big upfront filtering jobs seems to currently the most at-risk for triggering a No space left error.
As noted above, we have about 13TB free. As this number shrinks past 10TB, the incidence of No space left errors rises. This is only empirical, but we try to keep more than 10TB free just to avoid it.

Without any knowledge of what's going on internally, it seems that there's certainly correlation between HDFS space left and likelihood of a job on roughly the same input size failing with No space left. What's puzzling us is why it's when there's still as much as 10TB free, for a job that's on the order of 100GB of input.

> No space left on device
> -----------------------
>
>                 Key: HADOOP-6092
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6092
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.19.0
>         Environment: ubuntu0.8.4
>            Reporter: mawanqiang
>
> Exception in thread "main" org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
>         at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
>         at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
>         at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
>         at java.io.FilterOutputStream.close(FilterOutputStream.java:140)
>         at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
>         at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
>         at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.close(ChecksumFileSystem.java:339)
>         at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
>         at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
>         at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:825)
>         at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1142)
>         at org.apache.nutch.indexer.Indexer.index(Indexer.java:72)
>         at org.apache.nutch.indexer.Indexer.run(Indexer.java:92)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>         at org.apache.nutch.indexer.Indexer.main(Indexer.java:101)
> Caused by: java.io.IOException: No space left on device
>         at java.io.FileOutputStream.writeBytes(Native Method)
>         at java.io.FileOutputStream.write(FileOutputStream.java:260)
>         at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:197)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9862-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 12:54:45 2010
Return-Path: <common-issues-return-9862-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32245 invoked from network); 29 Jul 2010 12:54:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 12:54:45 -0000
Received: (qmail 80207 invoked by uid 500); 29 Jul 2010 12:54:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80057 invoked by uid 500); 29 Jul 2010 12:54:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80042 invoked by uid 99); 29 Jul 2010 12:54:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 12:54:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 12:54:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6TCsK94001523
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 12:54:20 GMT
Message-ID: <26740551.66531280408060563.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 08:54:20 -0400 (EDT)
From: "Iyappan Srinivasan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6879) Jsch jar is required for ivy
 resolution to be included in the build
In-Reply-To: <11760779.565921279956469755.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6879?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Iyappan Srinivasan updated HADOOP-6879:
---------------------------------------

    Attachment: HADOOP-6879.patch

One minor change

> Jsch jar is required for ivy resolution to be included in the build
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6879
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6879
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.1
>            Reporter: Iyappan Srinivasan
>            Assignee: Iyappan Srinivasan
>         Attachments: 6879-ydist-security-patch.txt, HADOOP-6879.patch, HA=
DOOP-6879.patch
>
>
> http://mvnrepository.com/
> com.jcraft =C2=BB jsch=20
> 0.1.42 version needs to be included in the build. This is  required for h=
erriot testcases using jsch.
> Please include this in ivy.
> jsch is originally located in http://www.jcraft.com/jsch/

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9863-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 16:52:48 2010
Return-Path: <common-issues-return-9863-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29307 invoked from network); 29 Jul 2010 16:52:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 16:52:42 -0000
Received: (qmail 94412 invoked by uid 500); 29 Jul 2010 16:52:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94358 invoked by uid 500); 29 Jul 2010 16:52:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94275 invoked by uid 99); 29 Jul 2010 16:52:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 16:52:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 16:52:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6TGqLOt016574
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 16:52:21 GMT
Message-ID: <20229400.70961280422341006.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 12:52:21 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6656:
--------------------------------

    Attachment: 6656-trunk-4.patch

This should take care of findbugs.

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6656-trunk-1.patch, 6656-trunk-2.patch, 6656-trunk-3.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9864-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 16:52:51 2010
Return-Path: <common-issues-return-9864-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29332 invoked from network); 29 Jul 2010 16:52:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 16:52:43 -0000
Received: (qmail 94584 invoked by uid 500); 29 Jul 2010 16:52:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94381 invoked by uid 500); 29 Jul 2010 16:52:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94135 invoked by uid 99); 29 Jul 2010 16:52:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 16:52:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 16:52:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6TGqHvU016531
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 16:52:17 GMT
Message-ID: <2904349.70821280422337523.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 12:52:17 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6656:
--------------------------------

    Status: Open  (was: Patch Available)

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6656-trunk-1.patch, 6656-trunk-2.patch, 6656-trunk-3.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9865-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 16:52:52 2010
Return-Path: <common-issues-return-9865-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29350 invoked from network); 29 Jul 2010 16:52:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 16:52:43 -0000
Received: (qmail 94620 invoked by uid 500); 29 Jul 2010 16:52:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94530 invoked by uid 500); 29 Jul 2010 16:52:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94366 invoked by uid 99); 29 Jul 2010 16:52:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 16:52:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 16:52:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6TGqLlU016589
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 16:52:21 GMT
Message-ID: <30566633.71011280422341741.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 12:52:21 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6656:
--------------------------------

    Status: Patch Available  (was: Open)

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6656-trunk-1.patch, 6656-trunk-2.patch, 6656-trunk-3.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9866-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 17:22:44 2010
Return-Path: <common-issues-return-9866-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48643 invoked from network); 29 Jul 2010 17:22:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 17:22:44 -0000
Received: (qmail 40301 invoked by uid 500); 29 Jul 2010 17:22:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40232 invoked by uid 500); 29 Jul 2010 17:22:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40223 invoked by uid 99); 29 Jul 2010 17:22:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 17:22:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 17:22:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6THMMs2016982
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 17:22:22 GMT
Message-ID: <11341520.71901280424142218.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 13:22:22 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893718#action_12893718 ] 

Hadoop QA commented on HADOOP-6656:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12450843/6656-trunk-4.patch
  against trunk revision 980271.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/649/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/649/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/649/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/649/console

This message is automatically generated.

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6656-trunk-1.patch, 6656-trunk-2.patch, 6656-trunk-3.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9867-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 17:26:42 2010
Return-Path: <common-issues-return-9867-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49874 invoked from network); 29 Jul 2010 17:26:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 17:26:42 -0000
Received: (qmail 42300 invoked by uid 500); 29 Jul 2010 17:26:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42135 invoked by uid 500); 29 Jul 2010 17:26:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42127 invoked by uid 99); 29 Jul 2010 17:26:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 17:26:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 17:26:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6THQHS0017023
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 17:26:17 GMT
Message-ID: <8182122.72011280424377345.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 13:26:17 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893720#action_12893720 ] 

Kan Zhang commented on HADOOP-6656:
-----------------------------------

+1 on the latest patch.

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6656-trunk-1.patch, 6656-trunk-2.patch, 6656-trunk-3.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9868-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 17:28:41 2010
Return-Path: <common-issues-return-9868-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50768 invoked from network); 29 Jul 2010 17:28:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 17:28:41 -0000
Received: (qmail 43198 invoked by uid 500); 29 Jul 2010 17:28:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43158 invoked by uid 500); 29 Jul 2010 17:28:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43150 invoked by uid 99); 29 Jul 2010 17:28:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 17:28:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 17:28:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6THSHWf017062
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 17:28:17 GMT
Message-ID: <26362013.72131280424497839.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 13:28:17 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6888) Being able to close all cached
 FileSystem objects for a given UGI
In-Reply-To: <11905003.55021280354956989.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6888:
-------------------------------------------

    Hadoop Flags: [Reviewed]

+1 patch looks good.

> Being able to close all cached FileSystem objects for a given UGI
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6888
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6888
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6888-02.patch
>
>
> This is the Common part of MAPREDUCE-1900. It adds a utility method to FileSystem that closes all cached filesystems for a given UGI.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9869-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 17:32:38 2010
Return-Path: <common-issues-return-9869-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52836 invoked from network); 29 Jul 2010 17:32:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 17:32:38 -0000
Received: (qmail 49910 invoked by uid 500); 29 Jul 2010 17:32:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49827 invoked by uid 500); 29 Jul 2010 17:32:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49819 invoked by uid 99); 29 Jul 2010 17:32:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 17:32:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 17:32:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6THWGYL017116
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 17:32:17 GMT
Message-ID: <28390090.72281280424736683.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 13:32:16 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6656) Security framework needs to renew
 Kerberos tickets while the process is running
In-Reply-To: <140327789.427581269326907684.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6656:
--------------------------------

        Status: Resolved  (was: Patch Available)
    Resolution: Fixed

I just committed this. Thanks, Owen for the early patches on this.

> Security framework needs to renew Kerberos tickets while the process is running
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-6656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6656
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6656-trunk-1.patch, 6656-trunk-2.patch, 6656-trunk-3.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, 6656-trunk-4.patch, c-6656-y20-internal.patch, refresh.patch
>
>
> While a client process is running, there should be a thread that periodically renews the Kerberos credentials to ensure they don't expire.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9870-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 17:50:41 2010
Return-Path: <common-issues-return-9870-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60351 invoked from network); 29 Jul 2010 17:50:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 17:50:41 -0000
Received: (qmail 66650 invoked by uid 500); 29 Jul 2010 17:50:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66600 invoked by uid 500); 29 Jul 2010 17:50:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66592 invoked by uid 99); 29 Jul 2010 17:50:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 17:50:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 17:50:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6THoGEL017244
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 17:50:16 GMT
Message-ID: <5663850.72531280425816167.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 13:50:16 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6653) NullPointerException in
 setupSaslConnection when browsing directories
In-Reply-To: <2110980760.395301269206007274.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das resolved HADOOP-6653.
---------------------------------

    Resolution: Invalid

This is not applicable in trunk anymore.

> NullPointerException in setupSaslConnection when browsing directories
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6653
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6653
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: c-6653-y20.patch
>
>
> e currently get a NullPointerException when setting up SASL RPC connection as part of browsing the filesystem after
> being redirected to a datanode.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9871-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 17:57:42 2010
Return-Path: <common-issues-return-9871-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63365 invoked from network); 29 Jul 2010 17:57:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 17:57:41 -0000
Received: (qmail 82265 invoked by uid 500); 29 Jul 2010 17:57:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82216 invoked by uid 500); 29 Jul 2010 17:57:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82208 invoked by uid 99); 29 Jul 2010 17:57:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 17:57:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 17:57:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6THvHYe017326
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 17:57:17 GMT
Message-ID: <31929985.72751280426237121.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 13:57:17 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6633) normalize property names for JT/NN
 kerberos principal names in configuration
In-Reply-To: <1678703232.278051268697507309.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6633:
--------------------------------

        Status: Resolved  (was: Patch Available)
    Resolution: Fixed

Resolving.

> normalize property names for JT/NN kerberos principal names in configuration
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-6633
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6633
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6633-BP20-1.patch, HADOOP-6633-BP20-2.patch, HADOOP-6633-BP20.patch, HADOOP-6633.patch
>
>
> change:
> DFS_NAMENODE_USER_NAME_KEY = "dfs.namenode.user.name" to "dfs.namenode.kerberos.user.name";
> DFS_NAMENODE_KRB_HTTPS_USER_NAME_KEY = "dfs.namenode.https.user.name" to  "dfs.namenode.kerberos.https.user.name"
> DFS_SECONDARY_NAMENODE_USER_NAME_KEY = "dfs.secondary.namenode.user.name" to "dfs.secondary.namenode.kerberos.user.name";
> DFS_SECONDARY_NAMENODE_KRB_HTTPS_USER_NAME_KEY = "dfs.secondary.namenode.https.user.name" to "dfs.secondary.namenode.kerberos.https.user.name";
> DFS_DATANODE_USER_NAME_KEY = "dfs.datanode.user.name" to "dfs.datanode.kerberos.user.name" 
> JT_USER_NAME = "mapreduce.jobtracker.user.name" to "mapreduce.jobtracker.kerberos.user.name";
> TT_USER_NAME = "mapreduce.tasktracker.user.name" to "mapreduce.tasktracker.kerberos.user.name"

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9872-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 18:06:42 2010
Return-Path: <common-issues-return-9872-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70669 invoked from network); 29 Jul 2010 18:06:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 18:06:41 -0000
Received: (qmail 88819 invoked by uid 500); 29 Jul 2010 18:06:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88781 invoked by uid 500); 29 Jul 2010 18:06:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88773 invoked by uid 99); 29 Jul 2010 18:06:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 18:06:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 18:06:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6TI6KV4017534
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 18:06:20 GMT
Message-ID: <1928742.73241280426780508.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 14:06:20 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6888) Being able to close all cached
 FileSystem objects for a given UGI
In-Reply-To: <11905003.55021280354956989.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6888:
-------------------------------------------

           Status: Resolved  (was: Patch Available)
    Fix Version/s: 0.22.0
       Resolution: Fixed

I have committed this.  Thanks, Kan!

> Being able to close all cached FileSystem objects for a given UGI
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6888
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6888
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>             Fix For: 0.22.0
>
>         Attachments: c6888-02.patch
>
>
> This is the Common part of MAPREDUCE-1900. It adds a utility method to FileSystem that closes all cached filesystems for a given UGI.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9873-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 18:33:46 2010
Return-Path: <common-issues-return-9873-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86992 invoked from network); 29 Jul 2010 18:33:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 18:33:45 -0000
Received: (qmail 22696 invoked by uid 500); 29 Jul 2010 18:33:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22641 invoked by uid 500); 29 Jul 2010 18:33:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22633 invoked by uid 99); 29 Jul 2010 18:33:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 18:33:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 18:33:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6TIXGUU017753
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 18:33:17 GMT
Message-ID: <20638354.73661280428396721.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 14:33:16 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6888) Being able to close all cached
 FileSystem objects for a given UGI
In-Reply-To: <11905003.55021280354956989.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893759#action_12893759 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6888:
------------------------------------------------

Also thanks Devaraj for working on this.

> Being able to close all cached FileSystem objects for a given UGI
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6888
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6888
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>             Fix For: 0.22.0
>
>         Attachments: c6888-02.patch
>
>
> This is the Common part of MAPREDUCE-1900. It adds a utility method to FileSystem that closes all cached filesystems for a given UGI.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9874-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 18:39:44 2010
Return-Path: <common-issues-return-9874-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88689 invoked from network); 29 Jul 2010 18:39:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 18:39:44 -0000
Received: (qmail 28553 invoked by uid 500); 29 Jul 2010 18:39:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28517 invoked by uid 500); 29 Jul 2010 18:39:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28509 invoked by uid 99); 29 Jul 2010 18:39:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 18:39:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 18:39:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6TIdHi5017790
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 18:39:17 GMT
Message-ID: <2264120.73731280428757563.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 14:39:17 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6870) Add FileSystem#listLocatedStatus to
 list a directory's content together with each file's block locations
In-Reply-To: <13042840.506371279749289722.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893760#action_12893760 ] 

Suresh Srinivas commented on HADOOP-6870:
-----------------------------------------

Sorry for posting the comments late. I was busy.

# General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS. 
# Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in {{stack}} and was not found later when iterating through it?
# FileSystem.java
#* listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
#* listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
# TestListFiles.java
#* testDirectory() - comments {{test empty directory}} and  {{test directory with 1 file}} should be moved up to relevant sections of the test. 



> Add FileSystem#listLocatedStatus to list a directory's content together with each file's block locations
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6870
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6870
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listFiles.patch, listFiles1.patch, listFiles2.patch, listFiles3.patch, listFiles4.patch
>
>
> This jira implements the new FileSystem API as proposed in HDFS-202. The new API aims to eliminate individual "getFileBlockLocations" calls to NN for each file in the input directory of a job. Instead, a file's block locations are returned together with FileStatus when listing a directory, thus improving getSplits performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9875-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 20:15:41 2010
Return-Path: <common-issues-return-9875-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23065 invoked from network); 29 Jul 2010 20:15:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 20:15:41 -0000
Received: (qmail 40354 invoked by uid 500); 29 Jul 2010 20:15:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40280 invoked by uid 500); 29 Jul 2010 20:15:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40272 invoked by uid 99); 29 Jul 2010 20:15:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 20:15:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 20:15:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6TKFGQG018681
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 20:15:16 GMT
Message-ID: <30758566.75561280434516176.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 16:15:16 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6889) Make RPC to have an option to timeout
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Make RPC to have an option to timeout
-------------------------------------

                 Key: HADOOP-6889
                 URL: https://issues.apache.org/jira/browse/HADOOP-6889
             Project: Hadoop Common
          Issue Type: New Feature
          Components: ipc
    Affects Versions: 0.22.0
            Reporter: Hairong Kuang
            Assignee: Hairong Kuang
             Fix For: 0.22.0, 0.20-append


Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.

But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica's length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.

I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9876-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 21:37:38 2010
Return-Path: <common-issues-return-9876-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54046 invoked from network); 29 Jul 2010 21:37:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 21:37:38 -0000
Received: (qmail 37025 invoked by uid 500); 29 Jul 2010 21:37:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36976 invoked by uid 500); 29 Jul 2010 21:37:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36968 invoked by uid 99); 29 Jul 2010 21:37:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 21:37:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 21:37:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6TLbFXO019475
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 21:37:16 GMT
Message-ID: <29884485.77211280439435901.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 17:37:15 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6889) Make RPC to have an option to
 timeout
In-Reply-To: <30758566.75561280434516176.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893833#action_12893833 ] 

Hairong Kuang commented on HADOOP-6889:
---------------------------------------

On a second thought, it would be nice if RPC proxies could configure differently. For example, one may want to have timeout on RPCs to DataNodes but no timeout on RPCs to NameNode. So I propose to add an additional parameter, maxRpcWaitingTime, to RPC#getProxy, meaning for every RPC call to this proxy, SocketTimeout is thrown if a client has not received a response in maxRpcWaitingTime. If maxRpcWaitingTime is zero, a RPC call will not timeout.

In this way, a client could configure maxRpcWaitingTime differently for its communicatations to NameNode and DataNodes. 



> Make RPC to have an option to timeout
> -------------------------------------
>
>                 Key: HADOOP-6889
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6889
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>
> Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.
> But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica's length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.
> I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9877-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 22:02:42 2010
Return-Path: <common-issues-return-9877-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58009 invoked from network); 29 Jul 2010 22:02:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 22:02:41 -0000
Received: (qmail 64346 invoked by uid 500); 29 Jul 2010 22:02:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64279 invoked by uid 500); 29 Jul 2010 22:02:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64271 invoked by uid 99); 29 Jul 2010 22:02:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 22:02:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 22:02:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6TM2Hs1019704
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 22:02:17 GMT
Message-ID: <3225048.77521280440937625.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 18:02:17 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6706:
-----------------------------------------

    Status: Open  (was: Patch Available)

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706-bp20-2.patch, 6706.bp20.1.patch, 6706.bp20.patch, HADOOP-6706-BP20-fix1.patch, HADOOP-6706-BP20-fix2.patch, HADOOP-6706-BP20-fix3.patch, HADOOP-6706.2.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9878-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 22:04:43 2010
Return-Path: <common-issues-return-9878-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58622 invoked from network); 29 Jul 2010 22:04:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 22:04:42 -0000
Received: (qmail 68457 invoked by uid 500); 29 Jul 2010 22:04:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68411 invoked by uid 500); 29 Jul 2010 22:04:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68403 invoked by uid 99); 29 Jul 2010 22:04:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 22:04:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 22:04:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6TM4LXY019756
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 22:04:21 GMT
Message-ID: <29574039.77661280441061559.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 18:04:21 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6706:
-----------------------------------------

    Status: Patch Available  (was: Open)

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706-bp20-2.patch, 6706.bp20.1.patch, 6706.bp20.patch, HADOOP-6706-BP20-fix1.patch, HADOOP-6706-BP20-fix2.patch, HADOOP-6706-BP20-fix3.patch, HADOOP-6706.2.patch, HADOOP-6706.4.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9879-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 22:04:45 2010
Return-Path: <common-issues-return-9879-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58676 invoked from network); 29 Jul 2010 22:04:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 22:04:45 -0000
Received: (qmail 69859 invoked by uid 500); 29 Jul 2010 22:04:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69827 invoked by uid 500); 29 Jul 2010 22:04:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69797 invoked by uid 99); 29 Jul 2010 22:04:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 22:04:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 22:04:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6TM4KFE019744
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 22:04:21 GMT
Message-ID: <2975506.77621280441060739.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 18:04:20 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6706:
-----------------------------------------

    Attachment: HADOOP-6706.4.patch

Patch for latest trunk.

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706-bp20-2.patch, 6706.bp20.1.patch, 6706.bp20.patch, HADOOP-6706-BP20-fix1.patch, HADOOP-6706-BP20-fix2.patch, HADOOP-6706-BP20-fix3.patch, HADOOP-6706.2.patch, HADOOP-6706.4.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9880-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 22:26:38 2010
Return-Path: <common-issues-return-9880-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68688 invoked from network); 29 Jul 2010 22:26:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 22:26:38 -0000
Received: (qmail 84443 invoked by uid 500); 29 Jul 2010 22:26:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84363 invoked by uid 500); 29 Jul 2010 22:26:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84355 invoked by uid 99); 29 Jul 2010 22:26:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 22:26:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 22:26:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6TMQGnZ019852
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 22:26:16 GMT
Message-ID: <6227281.77781280442376526.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 18:26:16 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Improve listFiles API introduced by HADOOP-6870
-----------------------------------------------

                 Key: HADOOP-6890
                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs
    Affects Versions: 0.22.0
            Reporter: Hairong Kuang
            Assignee: Hairong Kuang
             Fix For: 0.22.0


This jira is mainly for addressing Suresh's review comments for HADOOP-6870:

   1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
   2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
   3. FileSystem.java
          * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
          * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
   4. TestListFiles.java
          * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9881-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 22:32:39 2010
Return-Path: <common-issues-return-9881-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70379 invoked from network); 29 Jul 2010 22:32:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 22:32:39 -0000
Received: (qmail 91216 invoked by uid 500); 29 Jul 2010 22:32:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91162 invoked by uid 500); 29 Jul 2010 22:32:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91154 invoked by uid 99); 29 Jul 2010 22:32:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 22:32:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 22:32:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6TMWHfX019888
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 22:32:17 GMT
Message-ID: <25085916.77851280442737350.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 18:32:17 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893844#action_12893844 ] 

Hadoop QA commented on HADOOP-6706:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12450869/HADOOP-6706.4.patch
  against trunk revision 980533.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/650/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/650/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/650/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/650/console

This message is automatically generated.

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706-bp20-2.patch, 6706.bp20.1.patch, 6706.bp20.patch, HADOOP-6706-BP20-fix1.patch, HADOOP-6706-BP20-fix2.patch, HADOOP-6706-BP20-fix3.patch, HADOOP-6706.2.patch, HADOOP-6706.4.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9882-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 22:32:40 2010
Return-Path: <common-issues-return-9882-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70385 invoked from network); 29 Jul 2010 22:32:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 22:32:39 -0000
Received: (qmail 91230 invoked by uid 500); 29 Jul 2010 22:32:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91172 invoked by uid 500); 29 Jul 2010 22:32:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91163 invoked by uid 99); 29 Jul 2010 22:32:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 22:32:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 22:32:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6TMWHao019894
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 22:32:17 GMT
Message-ID: <13170683.77871280442737942.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 18:32:17 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893845#action_12893845 ] 

Hairong Kuang commented on HADOOP-6890:
---------------------------------------

> I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
We could put this new API in FileContext.utils. FileSystem will be deprecated soon so no need to worry about it.

> Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. 
> Should we ignore if a directory was in stack and was not found later when iterating through it?
GetFiles is used by MapReduce job client. In the scenario that you described, it is good to throw an exception so fail the job earlier. Otherwise, the job won't fail until map tasks are launched.

I will incorporate the rest of the comments. 

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9883-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 22:43:38 2010
Return-Path: <common-issues-return-9883-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71841 invoked from network); 29 Jul 2010 22:43:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 22:43:38 -0000
Received: (qmail 98059 invoked by uid 500); 29 Jul 2010 22:43:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98008 invoked by uid 500); 29 Jul 2010 22:43:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97997 invoked by uid 99); 29 Jul 2010 22:43:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 22:43:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 22:43:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6TMhGdg019934
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 22:43:16 GMT
Message-ID: <30789876.77921280443396083.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 18:43:16 -0400 (EDT)
From: "Saikat Kanjilal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6891) write Junit test cases that run the
 Hadoop examples
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

write Junit test cases that run the Hadoop examples
---------------------------------------------------

                 Key: HADOOP-6891
                 URL: https://issues.apache.org/jira/browse/HADOOP-6891
             Project: Hadoop Common
          Issue Type: New Feature
          Components: test
    Affects Versions: 0.20.2
         Environment: N/A
            Reporter: Saikat Kanjilal
            Priority: Minor
             Fix For: 0.20.3


I will be adding junit test cases that run the hadoop examples, specifically I'll be adding a test suite that tests everything under the src/examples/org/apache/hadoop/examples package.  Since this is my first contribution to the project I'll need some help to figure out how to submit this

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9884-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 22:49:41 2010
Return-Path: <common-issues-return-9884-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72612 invoked from network); 29 Jul 2010 22:49:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 22:49:41 -0000
Received: (qmail 4775 invoked by uid 500); 29 Jul 2010 22:49:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4732 invoked by uid 500); 29 Jul 2010 22:49:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4724 invoked by uid 99); 29 Jul 2010 22:49:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 22:49:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 22:49:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6TMnGAx019976
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 22:49:16 GMT
Message-ID: <33155323.77991280443756466.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 18:49:16 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6891) write Junit test cases that run the
 Hadoop examples
In-Reply-To: <30789876.77921280443396083.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893847#action_12893847 ] 

Jakob Homan commented on HADOOP-6891:
-------------------------------------

Hi Saikat.  What were you hoping to accomplish with these new unit tests? The example programs are meant to be a quick demo of Hadoop's abilitiles; how you are hoping to fit them into the unit tests?

> write Junit test cases that run the Hadoop examples
> ---------------------------------------------------
>
>                 Key: HADOOP-6891
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6891
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: test
>    Affects Versions: 0.20.2
>         Environment: N/A
>            Reporter: Saikat Kanjilal
>            Priority: Minor
>             Fix For: 0.20.3
>
>   Original Estimate: 20h
>  Remaining Estimate: 20h
>
> I will be adding junit test cases that run the hadoop examples, specifically I'll be adding a test suite that tests everything under the src/examples/org/apache/hadoop/examples package.  Since this is my first contribution to the project I'll need some help to figure out how to submit this

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9885-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 22:56:44 2010
Return-Path: <common-issues-return-9885-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73175 invoked from network); 29 Jul 2010 22:56:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 22:56:44 -0000
Received: (qmail 9921 invoked by uid 500); 29 Jul 2010 22:56:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9880 invoked by uid 500); 29 Jul 2010 22:56:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9872 invoked by uid 99); 29 Jul 2010 22:56:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 22:56:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 22:56:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6TMuJVh020056
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 22:56:19 GMT
Message-ID: <31137292.78131280444179831.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 18:56:19 -0400 (EDT)
From: "Saikat Kanjilal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6891) write Junit test cases that run the
 Hadoop examples
In-Reply-To: <30789876.77921280443396083.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893849#action_12893849 ] 

Saikat Kanjilal commented on HADOOP-6891:
-----------------------------------------

Hello Jakob,
I am interested in contributing to the hadoop project overall and am new to the community, to start this effort I took a look at the Project Suggestions page and picked the first task, please take a look at http://wiki.apache.org/hadoop/ProjectSuggestions and follow through to the first task under the Test Projects that has 16 hours allocated to it.  This is what I was hoping to tackle,  please let me know if I am misunderstanding what this task is asking but I thought this would be a good way to begin my contributions.

Regards

> write Junit test cases that run the Hadoop examples
> ---------------------------------------------------
>
>                 Key: HADOOP-6891
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6891
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: test
>    Affects Versions: 0.20.2
>         Environment: N/A
>            Reporter: Saikat Kanjilal
>            Priority: Minor
>             Fix For: 0.20.3
>
>   Original Estimate: 20h
>  Remaining Estimate: 20h
>
> I will be adding junit test cases that run the hadoop examples, specifically I'll be adding a test suite that tests everything under the src/examples/org/apache/hadoop/examples package.  Since this is my first contribution to the project I'll need some help to figure out how to submit this

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9886-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 23:12:42 2010
Return-Path: <common-issues-return-9886-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81005 invoked from network); 29 Jul 2010 23:12:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 23:12:42 -0000
Received: (qmail 14407 invoked by uid 500); 29 Jul 2010 23:12:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14348 invoked by uid 500); 29 Jul 2010 23:12:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14340 invoked by uid 99); 29 Jul 2010 23:12:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 23:12:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 23:12:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6TNCH9w020193
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 23:12:17 GMT
Message-ID: <22333113.78381280445137087.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 19:12:17 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6891) write Junit test cases that run the
 Hadoop examples
In-Reply-To: <30789876.77921280443396083.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893853#action_12893853 ] 

Jakob Homan commented on HADOOP-6891:
-------------------------------------

Ah, I see.  I think I know what the intention of this project idea was.  The examples could used as good integration tests and junit would be a reasonable way to execute them.  The tests themselves would consist of starting up minidfsclusters, prepping them for each example as needed, and executing them.  Post-project split the examples are located in the MapReduce project, so this JIRA should probably end up there.  This would be a good step toward clarifying the difference between unit and integration tests, which is a bit muddled at the moment in the source tree.

> write Junit test cases that run the Hadoop examples
> ---------------------------------------------------
>
>                 Key: HADOOP-6891
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6891
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: test
>    Affects Versions: 0.20.2
>         Environment: N/A
>            Reporter: Saikat Kanjilal
>            Priority: Minor
>             Fix For: 0.20.3
>
>   Original Estimate: 20h
>  Remaining Estimate: 20h
>
> I will be adding junit test cases that run the hadoop examples, specifically I'll be adding a test suite that tests everything under the src/examples/org/apache/hadoop/examples package.  Since this is my first contribution to the project I'll need some help to figure out how to submit this

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9887-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 23:21:42 2010
Return-Path: <common-issues-return-9887-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83752 invoked from network); 29 Jul 2010 23:21:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 23:21:41 -0000
Received: (qmail 20306 invoked by uid 500); 29 Jul 2010 23:21:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20239 invoked by uid 500); 29 Jul 2010 23:21:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20095 invoked by uid 99); 29 Jul 2010 23:21:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 23:21:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 23:21:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6TNLHHp020310
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 23:21:17 GMT
Message-ID: <25979782.78711280445677152.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 19:21:17 -0400 (EDT)
From: "Saikat Kanjilal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6891) write Junit test cases that run the
 Hadoop examples
In-Reply-To: <30789876.77921280443396083.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893857#action_12893857 ] 

Saikat Kanjilal commented on HADOOP-6891:
-----------------------------------------

Jakob,
Thanks for the big picture explanation, I'll first finish the test suite containing test classes to drive all the examples, move this jira over to the MapReduce project and then as I mentioned earlier may need some help to submit this (docs etc on the submittal process for this would be most helpful, I see some docs on submitting patches so not sure if this would be considered a patch or would follow some other process) from y'all since this will be my first submittal.
Regards

> write Junit test cases that run the Hadoop examples
> ---------------------------------------------------
>
>                 Key: HADOOP-6891
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6891
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: test
>    Affects Versions: 0.20.2
>         Environment: N/A
>            Reporter: Saikat Kanjilal
>            Priority: Minor
>             Fix For: 0.20.3
>
>   Original Estimate: 20h
>  Remaining Estimate: 20h
>
> I will be adding junit test cases that run the hadoop examples, specifically I'll be adding a test suite that tests everything under the src/examples/org/apache/hadoop/examples package.  Since this is my first contribution to the project I'll need some help to figure out how to submit this

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9888-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 29 23:55:38 2010
Return-Path: <common-issues-return-9888-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88977 invoked from network); 29 Jul 2010 23:55:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Jul 2010 23:55:38 -0000
Received: (qmail 37416 invoked by uid 500); 29 Jul 2010 23:55:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37380 invoked by uid 500); 29 Jul 2010 23:55:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37372 invoked by uid 99); 29 Jul 2010 23:55:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 23:55:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Jul 2010 23:55:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6TNtGTK020492
	for <common-issues@hadoop.apache.org>; Thu, 29 Jul 2010 23:55:17 GMT
Message-ID: <11117700.78981280447716743.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 19:55:16 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6889) Make RPC to have an option to
 timeout
In-Reply-To: <30758566.75561280434516176.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893863#action_12893863 ] 

Todd Lipcon commented on HADOOP-6889:
-------------------------------------

Hi Hairong. I agree that it should be per-proxy, and also that this is useful. We have a customer who has occasionally run into this with problematic "stuck" datanodes blocking pipeline recovery indefinitely (at least until ops notices the hung machine and powers it down, causing TCP connection to drop).

> Make RPC to have an option to timeout
> -------------------------------------
>
>                 Key: HADOOP-6889
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6889
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>
> Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.
> But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica's length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.
> I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9889-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 30 00:22:39 2010
Return-Path: <common-issues-return-9889-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98535 invoked from network); 30 Jul 2010 00:22:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jul 2010 00:22:38 -0000
Received: (qmail 55118 invoked by uid 500); 30 Jul 2010 00:22:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55076 invoked by uid 500); 30 Jul 2010 00:22:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55068 invoked by uid 99); 30 Jul 2010 00:22:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 00:22:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 00:22:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6U0MGBr020762
	for <common-issues@hadoop.apache.org>; Fri, 30 Jul 2010 00:22:17 GMT
Message-ID: <31462712.79571280449336755.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 20:22:16 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6840) Support non-recursive create() in
 FileSystem & SequenceFile.Writer
In-Reply-To: <30089846.67411277503610667.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893874#action_12893874 ] 

Todd Lipcon commented on HADOOP-6840:
-------------------------------------

20-append patch looks good.

> Support non-recursive create() in FileSystem & SequenceFile.Writer
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6840
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6840
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs, io
>    Affects Versions: 0.21.0, 0.20-append
>            Reporter: Nicolas Spiegelberg
>            Priority: Minor
>         Attachments: HADOOP-6840_0.20-append.patch, HADOOP-6840_0.21-2.patch, HADOOP-6840_0.21.patch
>
>
> The proposed solution for HBASE-2312 requires the sequence file to handle a non-recursive create.  This is already supported by HDFS, but needs to have an equivalent FileSystem & SequenceFile.Writer API.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9890-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 30 00:26:39 2010
Return-Path: <common-issues-return-9890-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99507 invoked from network); 30 Jul 2010 00:26:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jul 2010 00:26:38 -0000
Received: (qmail 55611 invoked by uid 500); 30 Jul 2010 00:26:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55571 invoked by uid 500); 30 Jul 2010 00:26:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55563 invoked by uid 99); 30 Jul 2010 00:26:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 00:26:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 00:26:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6U0QGmN020783
	for <common-issues@hadoop.apache.org>; Fri, 30 Jul 2010 00:26:17 GMT
Message-ID: <31633372.79601280449576700.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 20:26:16 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6886) LocalFileSystem Needs
 createNonRecursive API
In-Reply-To: <29262092.39841280295976234.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893875#action_12893875 ] 

Todd Lipcon commented on HADOOP-6886:
-------------------------------------

Looks good except one note:
- Change to the log message in FileSystem is missing a " " before class name


> LocalFileSystem Needs createNonRecursive API
> --------------------------------------------
>
>                 Key: HADOOP-6886
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6886
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.20-append
>            Reporter: Nicolas Spiegelberg
>            Priority: Minor
>             Fix For: 0.20-append
>
>         Attachments: HADOOP-6886_20-append.patch
>
>
> While running sanity check tests for HBASE-2312, I noticed that HDFS-617 did not include createNonRecursive() support for the LocalFileSystem.  This is a problem for HBase, which allows the user to run over the LocalFS instead of HDFS for local cluster testing.  I think this only affects 0.20-append, but may affect the trunk based upon how exactly FileContext handles non-recursive creates.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9891-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 30 00:54:39 2010
Return-Path: <common-issues-return-9891-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11927 invoked from network); 30 Jul 2010 00:54:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jul 2010 00:54:39 -0000
Received: (qmail 81867 invoked by uid 500); 30 Jul 2010 00:54:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81797 invoked by uid 500); 30 Jul 2010 00:54:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81789 invoked by uid 99); 30 Jul 2010 00:54:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 00:54:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 00:54:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6U0sHB4021153
	for <common-issues@hadoop.apache.org>; Fri, 30 Jul 2010 00:54:17 GMT
Message-ID: <2119646.80631280451257392.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 20:54:17 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6889) Make RPC to have an option to
 timeout
In-Reply-To: <30758566.75561280434516176.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893887#action_12893887 ] 

Doug Cutting commented on HADOOP-6889:
--------------------------------------

As I recall, RPC calls used to have a timeout that, combined with the retry proxy, would spiral load on servers.  But I think timeouts when retries go to a different server should not have the same problem.

> Make RPC to have an option to timeout
> -------------------------------------
>
>                 Key: HADOOP-6889
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6889
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>
> Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.
> But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica's length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.
> I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9892-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 30 01:35:41 2010
Return-Path: <common-issues-return-9892-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26911 invoked from network); 30 Jul 2010 01:35:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jul 2010 01:35:41 -0000
Received: (qmail 2702 invoked by uid 500); 30 Jul 2010 01:35:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2571 invoked by uid 500); 30 Jul 2010 01:35:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2563 invoked by uid 99); 30 Jul 2010 01:35:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 01:35:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 01:35:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6U1ZJ4L021458
	for <common-issues@hadoop.apache.org>; Fri, 30 Jul 2010 01:35:19 GMT
Message-ID: <3980749.81161280453719001.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 21:35:19 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6879) Jsch jar is required for ivy
 resolution to be included in the build
In-Reply-To: <11760779.565921279956469755.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6879?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D128=
93897#action_12893897 ]=20

Konstantin Boudnik commented on HADOOP-6879:
--------------------------------------------

bq. I think, this jar file needs to go to test->system profile, not master.=
=20

Apparently there's no Ivy system profile in Common (unlike Hdfs and MR). Ho=
wever, this dependency needs to go to test instead of master.=20

> Jsch jar is required for ivy resolution to be included in the build
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6879
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6879
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.1
>            Reporter: Iyappan Srinivasan
>            Assignee: Iyappan Srinivasan
>         Attachments: 6879-ydist-security-patch.txt, HADOOP-6879.patch, HA=
DOOP-6879.patch
>
>
> http://mvnrepository.com/
> com.jcraft =C2=BB jsch=20
> 0.1.42 version needs to be included in the build. This is  required for h=
erriot testcases using jsch.
> Please include this in ivy.
> jsch is originally located in http://www.jcraft.com/jsch/

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9893-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 30 02:20:43 2010
Return-Path: <common-issues-return-9893-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45371 invoked from network); 30 Jul 2010 02:20:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jul 2010 02:20:39 -0000
Received: (qmail 38924 invoked by uid 500); 30 Jul 2010 02:20:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38866 invoked by uid 500); 30 Jul 2010 02:20:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38851 invoked by uid 99); 30 Jul 2010 02:20:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 02:20:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 02:20:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6U2KHQG021946
	for <common-issues@hadoop.apache.org>; Fri, 30 Jul 2010 02:20:17 GMT
Message-ID: <23025161.82251280456416987.JavaMail.jira@thor>
Date: Thu, 29 Jul 2010 22:20:16 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6891) write Junit test cases that run the
 Hadoop examples
In-Reply-To: <30789876.77921280443396083.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893916#action_12893916 ] 

Konstantin Boudnik commented on HADOOP-6891:
--------------------------------------------

bq. This would be a good step toward clarifying the difference between unit and integration tests, which is a bit muddled at the moment in the source tree.

Right on! Because examples aren't unit tests in any sense of this word. They might be functional tests at best. As for integration tests: I'd agree that having something around mini-clusters might pass as integration. However, I tend to suggest that actual integration validation needs to be done in a real cluster environment.


> write Junit test cases that run the Hadoop examples
> ---------------------------------------------------
>
>                 Key: HADOOP-6891
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6891
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: test
>    Affects Versions: 0.20.2
>         Environment: N/A
>            Reporter: Saikat Kanjilal
>            Priority: Minor
>             Fix For: 0.20.3
>
>   Original Estimate: 20h
>  Remaining Estimate: 20h
>
> I will be adding junit test cases that run the hadoop examples, specifically I'll be adding a test suite that tests everything under the src/examples/org/apache/hadoop/examples package.  Since this is my first contribution to the project I'll need some help to figure out how to submit this

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9894-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 30 04:59:45 2010
Return-Path: <common-issues-return-9894-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86015 invoked from network); 30 Jul 2010 04:59:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jul 2010 04:59:45 -0000
Received: (qmail 10846 invoked by uid 500); 30 Jul 2010 04:59:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10627 invoked by uid 500); 30 Jul 2010 04:59:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10616 invoked by uid 99); 30 Jul 2010 04:59:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 04:59:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 04:59:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6U4xGqe023051
	for <common-issues@hadoop.apache.org>; Fri, 30 Jul 2010 04:59:17 GMT
Message-ID: <19763098.82761280465956635.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 00:59:16 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6873) using delegation token over hftp for
 long running clients (part of hdfs 1296)
In-Reply-To: <3672204.508361279757392441.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6873:
-----------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

committed to trunk

> using delegation token over hftp for long running clients (part of hdfs 1296)
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-6873
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6873
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6873-2.patch, HADOOP-6873.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9895-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 30 06:59:45 2010
Return-Path: <common-issues-return-9895-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43689 invoked from network); 30 Jul 2010 06:59:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jul 2010 06:59:45 -0000
Received: (qmail 79468 invoked by uid 500); 30 Jul 2010 06:59:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79330 invoked by uid 500); 30 Jul 2010 06:59:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79322 invoked by uid 99); 30 Jul 2010 06:59:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 06:59:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 06:59:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6U6xHmi023943
	for <common-issues@hadoop.apache.org>; Fri, 30 Jul 2010 06:59:17 GMT
Message-ID: <15807433.83841280473157312.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 02:59:17 -0400 (EDT)
From: "Meng Mao (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6092) No space left on device
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893951#action_12893951 ] 

Meng Mao commented on HADOOP-6092:
----------------------------------

Well tonight we had a failure with more than 12TB free. So the problem gets worse for us. Here is a sample of the errors we got:

java.io.IOException: Task: attempt_201005271420_14408_r_000000_0 - The reduce copier failed
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
	at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for file:/hadoop/hadoop-metadata/cache/mapred/local/taskTracker/jobcache/job_201005271420_14408/attempt_201005271420_14408_r_000000_0/output/map_1043.out
	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:343)
	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
	at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(ReduceTask.java:2513)


Error: java.io.IOException: No space left on device
	at java.io.FileOutputStream.writeBytes(Native Method)
	at java.io.FileOutputStream.write(FileOutputStream.java:260)
	at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:190)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
	at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
	at java.io.DataOutputStream.write(DataOutputStream.java:90)
	at org.apache.hadoop.mapred.IFileOutputStream.write(IFileOutputStream.java:84)
	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
	at java.io.DataOutputStream.write(DataOutputStream.java:90)
	at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:217)
	at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:157)
	at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(ReduceTask.java:2533)


Error: java.io.IOException: No space left on device
	at java.io.FileOutputStream.writeBytes(Native Method)
	at java.io.FileOutputStream.write(FileOutputStream.java:260)
	at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:190)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
	at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
	at java.io.DataOutputStream.write(DataOutputStream.java:90)
	at org.apache.hadoop.mapred.IFileOutputStream.write(IFileOutputStream.java:84)
	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
	at java.io.DataOutputStream.write(DataOutputStream.java:90)
	at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:189)
	at org.apache.hadoop.mapred.Task$CombineOutputCollector.collect(Task.java:880)
	at org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorCombiner.reduce(ValueAggregatorCombiner.java:68)
	at com.visiblemeasures.overdrive.hadoop.metrics.OverdriveAggregator$OverdriveAggregatorCombiner.reduce(OverdriveAggregator.java:315)
	at com.visiblemeasures.overdrive.hadoop.metrics.OverdriveAggregator$OverdriveAggregatorCombiner.reduce(OverdriveAggregator.java:286)
	at org.apache.hadoop.mapred.Task$OldCombinerRunner.combine(Task.java:1148)
	at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2642)
	at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:2580)

> No space left on device
> -----------------------
>
>                 Key: HADOOP-6092
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6092
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.19.0
>         Environment: ubuntu0.8.4
>            Reporter: mawanqiang
>
> Exception in thread "main" org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
>         at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
>         at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
>         at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
>         at java.io.FilterOutputStream.close(FilterOutputStream.java:140)
>         at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
>         at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
>         at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.close(ChecksumFileSystem.java:339)
>         at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
>         at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
>         at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:825)
>         at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1142)
>         at org.apache.nutch.indexer.Indexer.index(Indexer.java:72)
>         at org.apache.nutch.indexer.Indexer.run(Indexer.java:92)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>         at org.apache.nutch.indexer.Indexer.main(Indexer.java:101)
> Caused by: java.io.IOException: No space left on device
>         at java.io.FileOutputStream.writeBytes(Native Method)
>         at java.io.FileOutputStream.write(FileOutputStream.java:260)
>         at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:197)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9896-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 30 18:25:40 2010
Return-Path: <common-issues-return-9896-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9168 invoked from network); 30 Jul 2010 18:25:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jul 2010 18:25:40 -0000
Received: (qmail 36774 invoked by uid 500); 30 Jul 2010 18:25:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36604 invoked by uid 500); 30 Jul 2010 18:25:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36591 invoked by uid 99); 30 Jul 2010 18:25:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 18:25:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 18:25:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6UIPI2i029594
	for <common-issues@hadoop.apache.org>; Fri, 30 Jul 2010 18:25:18 GMT
Message-ID: <30191362.90571280514318260.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 14:25:18 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6887) Need a separate metrics per garbage
 collector
In-Reply-To: <25805095.53911280351596010.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu reassigned HADOOP-6887:
-------------------------------

    Assignee: Luke Lu

> Need a separate metrics per garbage collector
> ---------------------------------------------
>
>                 Key: HADOOP-6887
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6887
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: metrics
>            Reporter: Bharath Mundlapudi
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>
> In addition to current GC metrics which are the sum of all the collectors, Need separate metrics for monitoring young generation and old generation collections per collector w.r.t collection count and collection time. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9897-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 30 18:49:45 2010
Return-Path: <common-issues-return-9897-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17828 invoked from network); 30 Jul 2010 18:49:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jul 2010 18:49:45 -0000
Received: (qmail 66306 invoked by uid 500); 30 Jul 2010 18:49:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66235 invoked by uid 500); 30 Jul 2010 18:49:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66227 invoked by uid 99); 30 Jul 2010 18:49:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 18:49:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 18:49:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6UInKO1029782
	for <common-issues@hadoop.apache.org>; Fri, 30 Jul 2010 18:49:21 GMT
Message-ID: <22221965.90891280515760886.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 14:49:20 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6706:
-----------------------------------------

    Status: Open  (was: Patch Available)

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706-bp20-2.patch, 6706.bp20.1.patch, 6706.bp20.patch, HADOOP-6706-BP20-fix1.patch, HADOOP-6706-BP20-fix2.patch, HADOOP-6706-BP20-fix3.patch, HADOOP-6706.2.patch, HADOOP-6706.4.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9898-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 30 18:51:38 2010
Return-Path: <common-issues-return-9898-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18130 invoked from network); 30 Jul 2010 18:51:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jul 2010 18:51:38 -0000
Received: (qmail 67444 invoked by uid 500); 30 Jul 2010 18:51:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67303 invoked by uid 500); 30 Jul 2010 18:51:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67295 invoked by uid 99); 30 Jul 2010 18:51:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 18:51:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 18:51:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6UIpGxq029801
	for <common-issues@hadoop.apache.org>; Fri, 30 Jul 2010 18:51:16 GMT
Message-ID: <12598281.90931280515876693.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 14:51:16 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6706:
-----------------------------------------

    Attachment: HADOOP-6706.5.patch

New patch uploaded. Previous patch had an error.

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706-bp20-2.patch, 6706.bp20.1.patch, 6706.bp20.patch, HADOOP-6706-BP20-fix1.patch, HADOOP-6706-BP20-fix2.patch, HADOOP-6706-BP20-fix3.patch, HADOOP-6706.2.patch, HADOOP-6706.4.patch, HADOOP-6706.5.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9899-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 30 18:51:42 2010
Return-Path: <common-issues-return-9899-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18230 invoked from network); 30 Jul 2010 18:51:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jul 2010 18:51:41 -0000
Received: (qmail 67661 invoked by uid 500); 30 Jul 2010 18:51:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67573 invoked by uid 500); 30 Jul 2010 18:51:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67565 invoked by uid 99); 30 Jul 2010 18:51:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 18:51:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 18:51:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6UIpHOC029813
	for <common-issues@hadoop.apache.org>; Fri, 30 Jul 2010 18:51:17 GMT
Message-ID: <19317528.90971280515877449.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 14:51:17 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6706:
-----------------------------------------

    Status: Patch Available  (was: Open)

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706-bp20-2.patch, 6706.bp20.1.patch, 6706.bp20.patch, HADOOP-6706-BP20-fix1.patch, HADOOP-6706-BP20-fix2.patch, HADOOP-6706-BP20-fix3.patch, HADOOP-6706.2.patch, HADOOP-6706.4.patch, HADOOP-6706.5.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9900-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 30 18:58:42 2010
Return-Path: <common-issues-return-9900-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19924 invoked from network); 30 Jul 2010 18:58:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jul 2010 18:58:41 -0000
Received: (qmail 77545 invoked by uid 500); 30 Jul 2010 18:58:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77380 invoked by uid 500); 30 Jul 2010 18:58:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77176 invoked by uid 99); 30 Jul 2010 18:58:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 18:58:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 18:58:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6UIwHs3029925
	for <common-issues@hadoop.apache.org>; Fri, 30 Jul 2010 18:58:17 GMT
Message-ID: <23872254.91151280516297050.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 14:58:17 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6890:
----------------------------------

    Attachment: improveListFiles.patch

This patch
1. Move listFiles to be a method of FileContext#Util;
2. Modify FileContext#Util#listFiles so that the subtree is traversed in the depth-first order.
3. Improve the comments.

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: improveListFiles.patch
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9901-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 30 19:00:47 2010
Return-Path: <common-issues-return-9901-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21192 invoked from network); 30 Jul 2010 19:00:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jul 2010 19:00:46 -0000
Received: (qmail 82693 invoked by uid 500); 30 Jul 2010 19:00:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82455 invoked by uid 500); 30 Jul 2010 19:00:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82349 invoked by uid 99); 30 Jul 2010 19:00:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 19:00:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 19:00:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6UJ0OI6029998
	for <common-issues@hadoop.apache.org>; Fri, 30 Jul 2010 19:00:25 GMT
Message-ID: <16435132.91301280516424954.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 15:00:24 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6890:
----------------------------------

    Attachment:     (was: improveListFiles.patch)

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: improveListFiles.patch
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9902-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 30 19:00:48 2010
Return-Path: <common-issues-return-9902-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21251 invoked from network); 30 Jul 2010 19:00:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jul 2010 19:00:47 -0000
Received: (qmail 82855 invoked by uid 500); 30 Jul 2010 19:00:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82663 invoked by uid 500); 30 Jul 2010 19:00:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82636 invoked by uid 99); 30 Jul 2010 19:00:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 19:00:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 19:00:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6UJ0QUr000017
	for <common-issues@hadoop.apache.org>; Fri, 30 Jul 2010 19:00:26 GMT
Message-ID: <15594481.91341280516426026.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 15:00:26 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6890:
----------------------------------

    Attachment: improveListFiles.patch

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: improveListFiles.patch
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9903-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 30 19:07:41 2010
Return-Path: <common-issues-return-9903-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28306 invoked from network); 30 Jul 2010 19:07:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jul 2010 19:07:41 -0000
Received: (qmail 91848 invoked by uid 500); 30 Jul 2010 19:07:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91780 invoked by uid 500); 30 Jul 2010 19:07:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91772 invoked by uid 99); 30 Jul 2010 19:07:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 19:07:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 19:07:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6UJ7GC9000077
	for <common-issues@hadoop.apache.org>; Fri, 30 Jul 2010 19:07:17 GMT
Message-ID: <30207319.91401280516836848.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 15:07:16 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6890:
----------------------------------

    Status: Patch Available  (was: Open)

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: improveListFiles.patch
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9904-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 30 19:22:42 2010
Return-Path: <common-issues-return-9904-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31786 invoked from network); 30 Jul 2010 19:22:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jul 2010 19:22:42 -0000
Received: (qmail 16254 invoked by uid 500); 30 Jul 2010 19:22:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16208 invoked by uid 500); 30 Jul 2010 19:22:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16200 invoked by uid 99); 30 Jul 2010 19:22:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 19:22:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 19:22:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6UJMHLN000179
	for <common-issues@hadoop.apache.org>; Fri, 30 Jul 2010 19:22:18 GMT
Message-ID: <1638852.91521280517737840.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 15:22:17 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894117#action_12894117 ] 

Hadoop QA commented on HADOOP-6706:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12450920/HADOOP-6706.5.patch
  against trunk revision 980648.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/651/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/651/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/651/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/651/console

This message is automatically generated.

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706-bp20-2.patch, 6706.bp20.1.patch, 6706.bp20.patch, HADOOP-6706-BP20-fix1.patch, HADOOP-6706-BP20-fix2.patch, HADOOP-6706-BP20-fix3.patch, HADOOP-6706.2.patch, HADOOP-6706.4.patch, HADOOP-6706.5.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9905-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 30 19:42:38 2010
Return-Path: <common-issues-return-9905-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39584 invoked from network); 30 Jul 2010 19:42:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jul 2010 19:42:38 -0000
Received: (qmail 39908 invoked by uid 500); 30 Jul 2010 19:42:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39815 invoked by uid 500); 30 Jul 2010 19:42:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39807 invoked by uid 99); 30 Jul 2010 19:42:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 19:42:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 19:42:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6UJgGLv000317
	for <common-issues@hadoop.apache.org>; Fri, 30 Jul 2010 19:42:16 GMT
Message-ID: <23713215.91761280518936552.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 15:42:16 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894123#action_12894123 ] 

Hadoop QA commented on HADOOP-6890:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12450922/improveListFiles.patch
  against trunk revision 980648.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/652/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/652/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/652/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/652/console

This message is automatically generated.

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: improveListFiles.patch
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9906-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 30 20:36:42 2010
Return-Path: <common-issues-return-9906-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58215 invoked from network); 30 Jul 2010 20:36:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jul 2010 20:36:41 -0000
Received: (qmail 90831 invoked by uid 500); 30 Jul 2010 20:36:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90764 invoked by uid 500); 30 Jul 2010 20:36:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90753 invoked by uid 99); 30 Jul 2010 20:36:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 20:36:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 20:36:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6UKaH5V000954
	for <common-issues@hadoop.apache.org>; Fri, 30 Jul 2010 20:36:17 GMT
Message-ID: <10649627.93151280522177122.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 16:36:17 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894140#action_12894140 ] 

Suresh Srinivas commented on HADOOP-6890:
-----------------------------------------

+1 for the patch.

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: improveListFiles.patch
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9907-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 30 20:56:39 2010
Return-Path: <common-issues-return-9907-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62496 invoked from network); 30 Jul 2010 20:56:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jul 2010 20:56:38 -0000
Received: (qmail 19875 invoked by uid 500); 30 Jul 2010 20:56:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19797 invoked by uid 500); 30 Jul 2010 20:56:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19789 invoked by uid 99); 30 Jul 2010 20:56:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 20:56:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 20:56:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6UKuHXD001088
	for <common-issues@hadoop.apache.org>; Fri, 30 Jul 2010 20:56:17 GMT
Message-ID: <24901483.93311280523377489.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 16:56:17 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6890:
----------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

I've committed this!

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: improveListFiles.patch
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9908-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 30 20:56:41 2010
Return-Path: <common-issues-return-9908-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62556 invoked from network); 30 Jul 2010 20:56:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jul 2010 20:56:41 -0000
Received: (qmail 19997 invoked by uid 500); 30 Jul 2010 20:56:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19963 invoked by uid 500); 30 Jul 2010 20:56:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19955 invoked by uid 99); 30 Jul 2010 20:56:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 20:56:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 20:56:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6UKuGuf001076
	for <common-issues@hadoop.apache.org>; Fri, 30 Jul 2010 20:56:16 GMT
Message-ID: <17204266.93271280523376713.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 16:56:16 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894143#action_12894143 ] 

Hairong Kuang commented on HADOOP-6890:
---------------------------------------

Thanks Suresh for your quick review! 

The 6 javadoc warnings are all security related, not introduced by this patch.

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: improveListFiles.patch
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9909-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 30 21:41:42 2010
Return-Path: <common-issues-return-9909-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86311 invoked from network); 30 Jul 2010 21:41:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jul 2010 21:41:42 -0000
Received: (qmail 62530 invoked by uid 500); 30 Jul 2010 21:41:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62460 invoked by uid 500); 30 Jul 2010 21:41:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62452 invoked by uid 99); 30 Jul 2010 21:41:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 21:41:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 21:41:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6ULfHiQ001502
	for <common-issues@hadoop.apache.org>; Fri, 30 Jul 2010 21:41:17 GMT
Message-ID: <8004891.94141280526077534.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 17:41:17 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6632) Support for using different
 Kerberos keys for different instances of Hadoop services
In-Reply-To: <1542465557.251491268540967230.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894158#action_12894158 ] 

Todd Lipcon commented on HADOOP-6632:
-------------------------------------

It looks like the 6632.mr.patch portion was committed to ydist but not trunk - was this intentional?

> Support for using different Kerberos keys for different instances of Hadoop services
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6632
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6632
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>             Fix For: 0.22.0
>
>         Attachments: 6632.mr.patch, c6632-05.patch, c6632-07.patch, HADOOP-6632-Y20S-18.patch, HADOOP-6632-Y20S-22.patch
>
>
> We tested using the same Kerberos key for all datanodes in a HDFS cluster or the same Kerberos key for all TaskTarckers in a MapRed cluster. But it doesn't work. The reason is that when datanodes try to authenticate to the namenode all at once, the Kerberos authenticators they send to the namenode may have the same timestamp and will be rejected as replay requests. This JIRA makes it possible to use a unique key for each service instance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9910-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 30 21:55:39 2010
Return-Path: <common-issues-return-9910-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89442 invoked from network); 30 Jul 2010 21:55:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jul 2010 21:55:39 -0000
Received: (qmail 75142 invoked by uid 500); 30 Jul 2010 21:55:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75044 invoked by uid 500); 30 Jul 2010 21:55:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75030 invoked by uid 99); 30 Jul 2010 21:55:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 21:55:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 21:55:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6ULtHE1001673
	for <common-issues@hadoop.apache.org>; Fri, 30 Jul 2010 21:55:17 GMT
Message-ID: <743364.94511280526917087.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 17:55:17 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6632) Support for using different
 Kerberos keys for different instances of Hadoop services
In-Reply-To: <1542465557.251491268540967230.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894170#action_12894170 ] 

Devaraj Das commented on HADOOP-6632:
-------------------------------------

Yes this was intentional. The mr patch seemed like a hack and that's why we didn't commit it to trunk, and instead raised MAPREDUCE-1824 to discuss that... BTW, the problem which the mr patch attempted to address would be significantly less once we have HADOOP-6706 committed that does retries in case of failures due to the false replay attack detection by the rpc servers. MAPREDUCE-1824 takes a low priority..

> Support for using different Kerberos keys for different instances of Hadoop services
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6632
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6632
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>             Fix For: 0.22.0
>
>         Attachments: 6632.mr.patch, c6632-05.patch, c6632-07.patch, HADOOP-6632-Y20S-18.patch, HADOOP-6632-Y20S-22.patch
>
>
> We tested using the same Kerberos key for all datanodes in a HDFS cluster or the same Kerberos key for all TaskTarckers in a MapRed cluster. But it doesn't work. The reason is that when datanodes try to authenticate to the namenode all at once, the Kerberos authenticators they send to the namenode may have the same timestamp and will be rejected as replay requests. This JIRA makes it possible to use a unique key for each service instance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9911-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 30 22:53:42 2010
Return-Path: <common-issues-return-9911-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9005 invoked from network); 30 Jul 2010 22:53:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jul 2010 22:53:42 -0000
Received: (qmail 23833 invoked by uid 500); 30 Jul 2010 22:53:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23752 invoked by uid 500); 30 Jul 2010 22:53:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23735 invoked by uid 99); 30 Jul 2010 22:53:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 22:53:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 22:53:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6UMrGwa002200
	for <common-issues@hadoop.apache.org>; Fri, 30 Jul 2010 22:53:17 GMT
Message-ID: <11193881.95511280530396700.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 18:53:16 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6892) Common component of HDFS-1150 (Verify
 datanodes' identities to clients in secure clusters)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Common component of HDFS-1150 (Verify datanodes' identities to clients in secure clusters)
------------------------------------------------------------------------------------------

                 Key: HADOOP-6892
                 URL: https://issues.apache.org/jira/browse/HADOOP-6892
             Project: Hadoop Common
          Issue Type: New Feature
          Components: security
    Affects Versions: 0.22.0
            Reporter: Jakob Homan
            Assignee: Jakob Homan
             Fix For: 0.22.0


HDFS-1150 will have changes to the start-up scripts and HttpServer.  These are handled here.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9912-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 30 23:02:41 2010
Return-Path: <common-issues-return-9912-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10406 invoked from network); 30 Jul 2010 23:02:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jul 2010 23:02:41 -0000
Received: (qmail 28863 invoked by uid 500); 30 Jul 2010 23:02:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28798 invoked by uid 500); 30 Jul 2010 23:02:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28782 invoked by uid 99); 30 Jul 2010 23:02:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 23:02:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 23:02:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6UN2GSn002294
	for <common-issues@hadoop.apache.org>; Fri, 30 Jul 2010 23:02:17 GMT
Message-ID: <26752926.95541280530936617.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 19:02:16 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6892) Common component of HDFS-1150 (Verify
 datanodes' identities to clients in secure clusters)
In-Reply-To: <11193881.95511280530396700.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6892:
--------------------------------

    Attachment: HADOOP-6892.patch

Patch for trunk.  It's actually smaller than it appears.  HttpServer had some strangely indented code that patch moved in and out of.  Straight forward port of 20 work.

> Common component of HDFS-1150 (Verify datanodes' identities to clients in secure clusters)
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6892
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6892
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6892.patch
>
>
> HDFS-1150 will have changes to the start-up scripts and HttpServer.  These are handled here.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9913-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 30 23:02:42 2010
Return-Path: <common-issues-return-9913-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10453 invoked from network); 30 Jul 2010 23:02:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jul 2010 23:02:42 -0000
Received: (qmail 28988 invoked by uid 500); 30 Jul 2010 23:02:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28865 invoked by uid 500); 30 Jul 2010 23:02:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28789 invoked by uid 99); 30 Jul 2010 23:02:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 23:02:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 23:02:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6UN2H59002300
	for <common-issues@hadoop.apache.org>; Fri, 30 Jul 2010 23:02:17 GMT
Message-ID: <32799557.95561280530937182.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 19:02:17 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6892) Common component of HDFS-1150 (Verify
 datanodes' identities to clients in secure clusters)
In-Reply-To: <11193881.95511280530396700.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6892:
--------------------------------

    Status: Patch Available  (was: Open)

submitting patch.

> Common component of HDFS-1150 (Verify datanodes' identities to clients in secure clusters)
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6892
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6892
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6892.patch
>
>
> HDFS-1150 will have changes to the start-up scripts and HttpServer.  These are handled here.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9914-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 30 23:14:43 2010
Return-Path: <common-issues-return-9914-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19684 invoked from network); 30 Jul 2010 23:14:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jul 2010 23:14:42 -0000
Received: (qmail 36790 invoked by uid 500); 30 Jul 2010 23:14:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36731 invoked by uid 500); 30 Jul 2010 23:14:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36723 invoked by uid 99); 30 Jul 2010 23:14:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 23:14:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 23:14:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6UNEH0a002458
	for <common-issues@hadoop.apache.org>; Fri, 30 Jul 2010 23:14:17 GMT
Message-ID: <13057516.96001280531657474.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 19:14:17 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6632) Support for using different
 Kerberos keys for different instances of Hadoop services
In-Reply-To: <1542465557.251491268540967230.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894192#action_12894192 ] 

Todd Lipcon commented on HADOOP-6632:
-------------------------------------

Thanks, Deveraj. That makes sense.

> Support for using different Kerberos keys for different instances of Hadoop services
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6632
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6632
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>             Fix For: 0.22.0
>
>         Attachments: 6632.mr.patch, c6632-05.patch, c6632-07.patch, HADOOP-6632-Y20S-18.patch, HADOOP-6632-Y20S-22.patch
>
>
> We tested using the same Kerberos key for all datanodes in a HDFS cluster or the same Kerberos key for all TaskTarckers in a MapRed cluster. But it doesn't work. The reason is that when datanodes try to authenticate to the namenode all at once, the Kerberos authenticators they send to the namenode may have the same timestamp and will be rejected as replay requests. This JIRA makes it possible to use a unique key for each service instance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9915-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 30 23:31:42 2010
Return-Path: <common-issues-return-9915-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25286 invoked from network); 30 Jul 2010 23:31:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jul 2010 23:31:41 -0000
Received: (qmail 47384 invoked by uid 500); 30 Jul 2010 23:31:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47324 invoked by uid 500); 30 Jul 2010 23:31:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47316 invoked by uid 99); 30 Jul 2010 23:31:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 23:31:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 23:31:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6UNVGUv002665
	for <common-issues@hadoop.apache.org>; Fri, 30 Jul 2010 23:31:17 GMT
Message-ID: <1902711.96561280532676929.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 19:31:16 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6892) Common component of HDFS-1150
 (Verify datanodes' identities to clients in secure clusters)
In-Reply-To: <11193881.95511280530396700.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894196#action_12894196 ] 

Hadoop QA commented on HADOOP-6892:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12450926/HADOOP-6892.patch
  against trunk revision 980953.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/653/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/653/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/653/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/653/console

This message is automatically generated.

> Common component of HDFS-1150 (Verify datanodes' identities to clients in secure clusters)
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6892
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6892
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6892.patch
>
>
> HDFS-1150 will have changes to the start-up scripts and HttpServer.  These are handled here.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9916-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jul 30 23:52:41 2010
Return-Path: <common-issues-return-9916-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30205 invoked from network); 30 Jul 2010 23:52:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Jul 2010 23:52:41 -0000
Received: (qmail 54732 invoked by uid 500); 30 Jul 2010 23:52:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54661 invoked by uid 500); 30 Jul 2010 23:52:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54653 invoked by uid 99); 30 Jul 2010 23:52:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 23:52:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Jul 2010 23:52:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6UNqJHL002861
	for <common-issues@hadoop.apache.org>; Fri, 30 Jul 2010 23:52:19 GMT
Message-ID: <14284895.96971280533939483.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 19:52:19 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894206#action_12894206 ] 

Kan Zhang commented on HADOOP-6706:
-----------------------------------

+1, patch looks good. Couple nits.

bq. It's better to call disposeSasl() immediately after closeConnection() in handleSaslConnectionFailure().

bq. Please don't format lines you're not changing otherwise. This will make patch easier to read.

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706-bp20-2.patch, 6706.bp20.1.patch, 6706.bp20.patch, HADOOP-6706-BP20-fix1.patch, HADOOP-6706-BP20-fix2.patch, HADOOP-6706-BP20-fix3.patch, HADOOP-6706.2.patch, HADOOP-6706.4.patch, HADOOP-6706.5.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9917-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 31 00:03:42 2010
Return-Path: <common-issues-return-9917-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32412 invoked from network); 31 Jul 2010 00:03:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Jul 2010 00:03:42 -0000
Received: (qmail 61803 invoked by uid 500); 31 Jul 2010 00:03:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61766 invoked by uid 500); 31 Jul 2010 00:03:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61757 invoked by uid 99); 31 Jul 2010 00:03:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 31 Jul 2010 00:03:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 31 Jul 2010 00:03:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6V03HXX002957
	for <common-issues@hadoop.apache.org>; Sat, 31 Jul 2010 00:03:17 GMT
Message-ID: <5601971.97071280534597347.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 20:03:17 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6706:
-----------------------------------------

    Status: Open  (was: Patch Available)

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706-bp20-2.patch, 6706.bp20.1.patch, 6706.bp20.patch, HADOOP-6706-BP20-fix1.patch, HADOOP-6706-BP20-fix2.patch, HADOOP-6706-BP20-fix3.patch, HADOOP-6706.2.patch, HADOOP-6706.4.patch, HADOOP-6706.5.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9918-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 31 00:05:39 2010
Return-Path: <common-issues-return-9918-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36666 invoked from network); 31 Jul 2010 00:05:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Jul 2010 00:05:39 -0000
Received: (qmail 62270 invoked by uid 500); 31 Jul 2010 00:05:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62224 invoked by uid 500); 31 Jul 2010 00:05:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62210 invoked by uid 99); 31 Jul 2010 00:05:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 31 Jul 2010 00:05:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 31 Jul 2010 00:05:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6V05HMb002980
	for <common-issues@hadoop.apache.org>; Sat, 31 Jul 2010 00:05:18 GMT
Message-ID: <27119849.97111280534717550.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 20:05:17 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6706:
-----------------------------------------

    Attachment: HADOOP-6706.6.patch

New patch addressing the comments.

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706-bp20-2.patch, 6706.bp20.1.patch, 6706.bp20.patch, HADOOP-6706-BP20-fix1.patch, HADOOP-6706-BP20-fix2.patch, HADOOP-6706-BP20-fix3.patch, HADOOP-6706.2.patch, HADOOP-6706.4.patch, HADOOP-6706.5.patch, HADOOP-6706.6.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9919-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 31 00:05:40 2010
Return-Path: <common-issues-return-9919-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36707 invoked from network); 31 Jul 2010 00:05:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Jul 2010 00:05:40 -0000
Received: (qmail 62424 invoked by uid 500); 31 Jul 2010 00:05:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62386 invoked by uid 500); 31 Jul 2010 00:05:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62378 invoked by uid 99); 31 Jul 2010 00:05:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 31 Jul 2010 00:05:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 31 Jul 2010 00:05:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6V05JQE002992
	for <common-issues@hadoop.apache.org>; Sat, 31 Jul 2010 00:05:19 GMT
Message-ID: <32249475.97151280534719207.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 20:05:19 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6706:
-----------------------------------------

    Status: Patch Available  (was: Open)

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706-bp20-2.patch, 6706.bp20.1.patch, 6706.bp20.patch, HADOOP-6706-BP20-fix1.patch, HADOOP-6706-BP20-fix2.patch, HADOOP-6706-BP20-fix3.patch, HADOOP-6706.2.patch, HADOOP-6706.4.patch, HADOOP-6706.5.patch, HADOOP-6706.6.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9920-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 31 00:29:39 2010
Return-Path: <common-issues-return-9920-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49159 invoked from network); 31 Jul 2010 00:29:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Jul 2010 00:29:39 -0000
Received: (qmail 68075 invoked by uid 500); 31 Jul 2010 00:29:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68005 invoked by uid 500); 31 Jul 2010 00:29:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67997 invoked by uid 99); 31 Jul 2010 00:29:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 31 Jul 2010 00:29:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 31 Jul 2010 00:29:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6V0THMB003287
	for <common-issues@hadoop.apache.org>; Sat, 31 Jul 2010 00:29:17 GMT
Message-ID: <10249767.97821280536157892.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 20:29:17 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894219#action_12894219 ] 

Hadoop QA commented on HADOOP-6706:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12450934/HADOOP-6706.6.patch
  against trunk revision 980953.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/654/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/654/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/654/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/654/console

This message is automatically generated.

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706-bp20-2.patch, 6706.bp20.1.patch, 6706.bp20.patch, HADOOP-6706-BP20-fix1.patch, HADOOP-6706-BP20-fix2.patch, HADOOP-6706-BP20-fix3.patch, HADOOP-6706.2.patch, HADOOP-6706.4.patch, HADOOP-6706.5.patch, HADOOP-6706.6.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9921-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 31 00:40:42 2010
Return-Path: <common-issues-return-9921-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51634 invoked from network); 31 Jul 2010 00:40:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Jul 2010 00:40:41 -0000
Received: (qmail 74541 invoked by uid 500); 31 Jul 2010 00:40:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74461 invoked by uid 500); 31 Jul 2010 00:40:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74453 invoked by uid 99); 31 Jul 2010 00:40:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 31 Jul 2010 00:40:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 31 Jul 2010 00:40:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6V0eHKO003459
	for <common-issues@hadoop.apache.org>; Sat, 31 Jul 2010 00:40:17 GMT
Message-ID: <9186586.98301280536817347.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 20:40:17 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6890:
----------------------------------

    Attachment: listFilesInFS.patch

This patch makes the implementation of listFiles in FileSystem is the same as the one in FileContext except that it does not handle symbolic links. This will make the code easier to maintain later one.

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: improveListFiles.patch, listFilesInFS.patch
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9922-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 31 03:08:47 2010
Return-Path: <common-issues-return-9922-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83156 invoked from network); 31 Jul 2010 03:08:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Jul 2010 03:08:47 -0000
Received: (qmail 5355 invoked by uid 500); 31 Jul 2010 03:08:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5216 invoked by uid 500); 31 Jul 2010 03:08:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5208 invoked by uid 99); 31 Jul 2010 03:08:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 31 Jul 2010 03:08:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 31 Jul 2010 03:08:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6V38Ktx004253
	for <common-issues@hadoop.apache.org>; Sat, 31 Jul 2010 03:08:20 GMT
Message-ID: <8289341.99671280545700343.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 23:08:20 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Greg Roelofs updated HADOOP-6837:
---------------------------------

    Attachment: HADOOP-6837-lzma-1-20100722.non-trivial.pseudo-patch

Hand-edited pseudo-patch for easier reviewing.

"New" files from the LZMA SDK were diffed against that (version 9.12), and patch-hunks containing only trivial changes (Java package, formatting, Apache boilerplate, white space, etc.) were replaced with a corresponding meta-comment.  Note that the 9.12 diffs were "diff -ruw" (i.e., ignore whitespace-only changes on single lines) and only one directory level above the relevant code ("LZMA-SDK" vs. "LZMA").  The Java and C diffs were then combined, and the truly new files (e.g., LzmaCodec.java) were then added back from the original patch.

3700 lines vs. original 10400+

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-1-20100722.non-trivial.pseudo-patch, HADOOP-6837-lzma-1-20100722.patch, HADOOP-6837-lzma-c-20100719.patch, HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9923-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 31 03:42:47 2010
Return-Path: <common-issues-return-9923-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90635 invoked from network); 31 Jul 2010 03:42:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Jul 2010 03:42:47 -0000
Received: (qmail 22455 invoked by uid 500); 31 Jul 2010 03:42:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22324 invoked by uid 500); 31 Jul 2010 03:42:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22316 invoked by uid 99); 31 Jul 2010 03:42:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 31 Jul 2010 03:42:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 31 Jul 2010 03:42:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6V3gNYJ004404
	for <common-issues@hadoop.apache.org>; Sat, 31 Jul 2010 03:42:23 GMT
Message-ID: <4329798.99991280547743556.JavaMail.jira@thor>
Date: Fri, 30 Jul 2010 23:42:23 -0400 (EDT)
From: "Nicholas Carlini (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6349) Implement FastLZCodec for fastlz/lzo
 algorithm
In-Reply-To: <289817036.1256941439448.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicholas Carlini updated HADOOP-6349:
-------------------------------------

    Attachment: hadoop-6349-3.patch

Another patch. There is still debug code scattered about (commented out), as I might need to put it to use at some point. This code isn't tested as well as the last patch.

Adds support for native compression/decompression. Native compression is 230% faster than java. Native decompression is 70% faster than java.

Somewhat-large redesign of the compressor. Compression is now fifty times faster when compressing around 64MB. The compressor used to keep in memory all input it had previously processed, and arraycopy it to a new array every time it needed more space, so through the process of compressing 64MB of data calling write every 64k, it would end up copying ~32GB through memory (this is how it was for my test case). Instead compress 128MB of data and write every 1k, and you copy 8.8TB through memory.

Also modified compressor to include an end-of-stream marker. This way the decompressor can set to "finished" so the stream can return -1. The end of stream mark is indicated by setting the four unused bytes after the input size to high in the last chunk of length 0. By this way, any decompressor which does not support the end of stream marker will never read those bytes and will just decompress an empty block and not notice anything is wrong.

Adds another method to TestCodecPerformance which haves it load a (relatively small) input file to memory, and from it generate 64MB of data to compress. (It does this by taking random substrings from 16 to 128 bytes at random offsets until there are 64MB.) It then directly compresses the 64MB from memory to memory and times that. These times seem to be more reflective than timing the compression of "key %d value %d" and of timing the compression of random data. Right now this mode is enabled by calling it with the -input flag.

Ported code for Adler32 to C, uses it when using native libraries.

Added a constant in the compressor to allow for uncompressible data to instead be copied over byte for byte. This decreases the speed of the compressor by ~10% as it results in another memcpy, but it can more than double the speed of decompression.  



Here's what the new part of the test codec performance gives when given a log file. For comparison: DefaultCodec gets the size down to 11% and the BZip2Codec down to 8%.

Previous patch:
10/07/29 11:51:39 INFO compress.TestCodecPerformance: Total decompressed size: 640 MB.
10/07/29 11:51:39 INFO compress.TestCodecPerformance: Total compressed size: 177 MB (27% of original).
10/07/29 11:51:39 INFO compress.TestCodecPerformance: Total compression time: 381868 ms (1716 KBps).
10/07/29 11:51:39 INFO compress.TestCodecPerformance: Total decompression time: 5051 ms (126 MBps).

Current patch:
Native C:
10/07/29 11:56:57 INFO compress.TestCodecPerformance: Total decompressed size: 640 MB.
10/07/29 11:56:57 INFO compress.TestCodecPerformance: Total compressed size: 177 MB (27% of original).
10/07/29 11:56:57 INFO compress.TestCodecPerformance: Total compression time: 3314 ms (193 MBps).
10/07/29 11:56:57 INFO compress.TestCodecPerformance: Total decompression time: 2861 ms (223 MBps).

Current patch:
Pure Java:
10/07/29 12:15:50 INFO compress.TestCodecPerformance: Total decompressed size: 640 MB.
10/07/29 12:15:50 INFO compress.TestCodecPerformance: Total compressed size: 177 MB (27% of original).
10/07/29 12:15:50 INFO compress.TestCodecPerformance: Total compression time: 7891 ms (81 MBps).
10/07/29 12:15:50 INFO compress.TestCodecPerformance: Total decompression time: 5077 ms (126 MBps).

> Implement FastLZCodec for fastlz/lzo algorithm
> ----------------------------------------------
>
>                 Key: HADOOP-6349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6349
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: William Kinney
>         Attachments: hadoop-6349-1.patch, hadoop-6349-2.patch, hadoop-6349-3.patch, HADOOP-6349-TestFastLZCodec.patch, HADOOP-6349.patch, TestCodecPerformance.java, TestCodecPerformance.java, testCodecPerfResults.tsv
>
>
> Per  [HADOOP-4874|http://issues.apache.org/jira/browse/HADOOP-4874], FastLZ is a good (speed, license) alternative to LZO. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9924-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jul 31 04:56:48 2010
Return-Path: <common-issues-return-9924-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5498 invoked from network); 31 Jul 2010 04:56:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Jul 2010 04:56:48 -0000
Received: (qmail 43685 invoked by uid 500); 31 Jul 2010 04:56:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43552 invoked by uid 500); 31 Jul 2010 04:56:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43540 invoked by uid 99); 31 Jul 2010 04:56:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 31 Jul 2010 04:56:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 31 Jul 2010 04:56:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o6V4uKX9005099
	for <common-issues@hadoop.apache.org>; Sat, 31 Jul 2010 04:56:20 GMT
Message-ID: <28796306.100201280552180313.JavaMail.jira@thor>
Date: Sat, 31 Jul 2010 00:56:20 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894246#action_12894246 ] 

Greg Roelofs commented on HADOOP-6837:
--------------------------------------

First, apologies for having steered Nicholas wrong on the liblzma issue.  As
Hong noted, it provides a much saner (that is, zlib-like) API for this sort
of thing, but I mistakenly thought it shared the GPL license of (parts of)
xz, so we ignored it and he worked on the LZMA SDK code instead.  (The latter
did include a Java port, however; liblzma does not.)

Overall, the 20100722 patch looks pretty decent (given the starting material),
but it does include some less-than-beautiful workarounds to cope with the
impedance mismatch between push- and pull-style I/O models.  In light of the
fact that liblzma is, in fact, public-domain software (every file under
xz-4.999.9beta-143-g3e49/src/liblzma is either explicitly in the public
domain or has been automatically generated by such a file), I'm going to ask
that Nicholas redo the native-code version to use liblzma rather than the
SDK.  (Unfortunately, it looks like the transformation from C SDK to liblzma
was a significant amount of work, so it doesn't appear that a trivial "liblzma-
ification" of the Java SDK code is likely.  If Nicholas concurs with that
assessment, we can instead file a separate JIRA to port the liblzma C code
to Java.)  Note that liblzma includes an LZMA2 codec, so Scott Carey's
splittable-codec suggestion is within reach, too.

OK, enough preamble.  There were a number of relatively minor style issues,
which I'll simply list below, but the five main concerns were:

 - FakeInputStream.java, FakeOutputStream.java:  the linked lists of byte
   arrays are tough to swallow, even given the push/pull problem, even given
   our previous discussions on the matter.  It would be good to know what the
   stats are on these things in "typical" cases--how frequently does overflow
   occur in LzmaInputStream, for example, and how many buffers are used?

 - Is the Code(..., len) call in LzmaInputStream guaranteed to produce len
   bytes if it returns true?  The calling read() function assumes this, but
   it's not at all obvious to me; the parameter is outSize in Code(), and I
   don't see that it's decremented or even really used at all (other than
   being stored in oldOutSize), unless it's buried inside macros defined
   elsewhere.

The next two (or perhaps three) are no longer directly relevant, but they're
general things to watch out for:

 - The return value from inStream.read() in LzmaNativeInputStream.java is
   ignored even though there's no guarantee the call will return the requested
   number of bytes.  A comment ("never have to call ... again") reiterates
   this error.

 - There's no need for recursion in LzmaNativeOutputStream.java's write()
   method; iterative solutions tend to be far cleaner, I think.  (Probably
   ditto for LzmaNativeInputStream.java's read() method.)

 - LzmaCompressor.c has a pair of memleaks (state->outStream, state->inStream).

Here are the minor readability/maintainability/cosmetic/style issues:

 * add LZMA SDK version (apparently 9.12) and perhaps its release date to the
   boilerplate
 * tabs/formatting of LZMA SDK code (CRC.java, BinTree, etc.):  I _think_
   tabs are frowned upon in Hadoop, though I might be wrong; at any rate,
   they seem to be rarely used
   ** for easy Hadoop-style formatting, "indent -i2 -br -l80" is a start (though
     it's sometimes confused by Java/C++ constructs)
 * reuse existing CRC implementation(s)?  (JDK has one, Hadoop has another)
 * prefer lowercase "lzma" for subdirs
 * use uppercase "LZMA" when referring to codec/algorithm (e.g., comments)
 * add README mentioning anything special/weird/etc. (e.g., weird hashtable
   issue); summary of changes made for Hadoop; potential Java/C diffs; binary
   compatibility between various output formats (other than trivial headers/
   footers); LZMA2 (splittable, not yet implemented); liblzma (much cleaner,
   more zlib-like implementation, still PD); etc.
 * "ant javadoc" run yet?  (apparently not recently)
 * line lengths, particularly comments (LzmaInputStream.java, etc.):  should
   be no more than 80 columns in general (Hadoop guidelines)
 * avoid generic variable names for globals and class members; use existing
   conventions where possible (e.g., look at gzip/zlib and bzip2 code)
 * LzmaCodec.java:
   ** uppercase "LZMA" when referring to codec/algorithm in general
   ** "funcionality" x 4
   ** "throws ... {" continuation line:  don't further indent
 * LZ/InWindow.java
   ** leftover debug code at end
 * RangeCoder/Encoder.java
   ** spurious blank line (else just boilerplate)
 * FakeOutputStream.java:
   ** "stuffeed"
   ** "ammount"
   ** isOverflow() -> didOverflow()
 * LzmaInputStream.java:
   ** [uses FakeOutputStream]
   ** "bufferd"
   ** "we 've"
   ** index -> overflowIndex (or similar):  too generic
   ** hasRead -> initialized
   ** doInit():  use LzmaCodec getDefaultExtension() (or simply change text to
     "LZMA") instead of hardcoded ".lzma"
     *** if higher-level code extended to support .xz, would this code even know?
   ** read():  do doInit() after args-checking/throws
   ** 3rd read() has bogus @return comment
   ** overflow approach:  data on how frequent, how many buffers?
     *** no reuse:  creating new FakeOutputStream on every read()
 * LzmaOutputStream.java
   ** [uses FakeInputStream]
   ** hasWritten -> initialized
   ** 6 -> static final int defaultCompressionLevel
   ** ctor:  level -> this.level; l -> level (broken javadoc currently)
   ** "may cause a lower compression ratio":  measured?  by how much?
 * LzmaNativeInputStream.java
   ** line lengths
   ** hasRead -> initialized
   ** tmp -> tmpBuf
   ** max -> maxBlockSize?  ...?
   ** index -> ...?
   ** buffered, sendData -> compressedDirectBuf, uncompressedDirectBuf (~zlib)
   ** what -> something better
   ** 1<<16, 1<<12 -> static final whatever ints, and use N*1024 instead
   ** allocateDirect() should be using config'd buffer size, I think (check
     zlib/similar impls for use of io.file.buffer.size)
   ** giveMore():  return val not used, but if it were, true/false seems backward
     (expect:  if !giveMore() => nothing available)
   ** read():  point of extras just to support recursion?  (extras, what: what??)
   ** "decompresed"
 * LzmaNativeOutputStream.java
   ** buffered, sendData -> compressedDirectBuf, uncompressedDirectBuf (~zlib)
   ** 1<<16, 1<<17 -> static final whatever int (and 64*1024, etc.)
   ** hasWritten -> initialized
   ** tmp -> tmpBuf
 * LZMA/Encoder.java
   ** "their" -> "original SDK" or similar
   ** "Setup the encoder" -> "Set up the encoder"
   ** "Do one encoding of a block" -> "Encode one block" ?
 * LzmaCompressor.c
   ** buffered, sendData -> compressedDirectBuf, uncompressedDirectBuf (~zlib)
 * LzmaDecompressor.c
   ** inPos vs. posInInBuf:  ???
   ** buffered, sendData -> compressedDirectBuf, uncompressedDirectBuf (~zlib)
   ** 1<<16, 1<<12, difference -> macros or static const ints
 * Makefile.am
   ** zlib -> LZMA
   ** "All these are setup" -> "All these are set up"
   ** "are also done" -> "is also done"
 * LzFind.c
   ** remove commented-out stuff
   ** (1<<15)+(1<<14) -> 48*1024 (etc.)
 * LZMA/LzmaEnc.c
   ** tabs


> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-1-20100722.non-trivial.pseudo-patch, HADOOP-6837-lzma-1-20100722.patch, HADOOP-6837-lzma-c-20100719.patch, HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9925-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 05:20:46 2010
Return-Path: <common-issues-return-9925-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69853 invoked from network); 2 Aug 2010 05:20:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 05:20:46 -0000
Received: (qmail 10838 invoked by uid 500); 2 Aug 2010 05:20:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9742 invoked by uid 500); 2 Aug 2010 05:20:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9620 invoked by uid 99); 2 Aug 2010 05:20:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 05:20:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 05:20:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o725KHK3027142
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 05:20:17 GMT
Message-ID: <14318368.112951280726417561.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 01:20:17 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-4851) Change filesystem permissions to use
 JAAS rather than UGI
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-4851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley resolved HADOOP-4851.
-----------------------------------

    Resolution: Won't Fix

I don't think this is a good idea. Having implemented on top of the JAAS stuff, the impedance mismatch is pretty annoying. Can I close this as won't fix?

> Change filesystem permissions to use JAAS rather than UGI
> ---------------------------------------------------------
>
>                 Key: HADOOP-4851
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4851
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.0
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>
> Hadoop embraced JAAS via HADOOP-4348.
> We need to fix HDFS to use JAAS concepts such as Subject, Principal, Permission etc. rather than UserGroupInformation for file-system permissions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9926-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 05:28:42 2010
Return-Path: <common-issues-return-9926-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71843 invoked from network); 2 Aug 2010 05:28:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 05:28:42 -0000
Received: (qmail 11737 invoked by uid 500); 2 Aug 2010 05:28:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11672 invoked by uid 500); 2 Aug 2010 05:28:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11655 invoked by uid 99); 2 Aug 2010 05:28:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 05:28:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 05:28:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o725SJxK027211
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 05:28:19 GMT
Message-ID: <12314272.113111280726899551.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 01:28:19 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6893) FileContext needs a method to test
 for existence of a file or directory.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

FileContext needs a method to test for existence of a file or directory.
------------------------------------------------------------------------

                 Key: HADOOP-6893
                 URL: https://issues.apache.org/jira/browse/HADOOP-6893
             Project: Hadoop Common
          Issue Type: New Feature
          Components: fs
            Reporter: Owen O'Malley


FileContext is missing a method to ask if a file or directory exists. Currently the only mechanism is to use getFileStatus, which throws an exception if the file doesn't exist, which I don't think is an appropriate use of that method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9927-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 05:38:44 2010
Return-Path: <common-issues-return-9927-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73186 invoked from network); 2 Aug 2010 05:38:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 05:38:43 -0000
Received: (qmail 18577 invoked by uid 500); 2 Aug 2010 05:38:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18441 invoked by uid 500); 2 Aug 2010 05:38:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18433 invoked by uid 99); 2 Aug 2010 05:38:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 05:38:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 05:38:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o725cGeJ027247
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 05:38:16 GMT
Message-ID: <15708894.113141280727496179.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 01:38:16 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6893) FileContext needs a method to test
 for existence of a file or directory.
In-Reply-To: <12314272.113111280726899551.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894465#action_12894465 ] 

Owen O'Malley commented on HADOOP-6893:
---------------------------------------

I see now that there is a Util method that checks exist, but it still uses getFileStatus and catches the exception. There should be some way of asking an AbstractFileSystem whether a name exists that doesn't throw an exception.

> FileContext needs a method to test for existence of a file or directory.
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6893
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6893
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>            Reporter: Owen O'Malley
>
> FileContext is missing a method to ask if a file or directory exists. Currently the only mechanism is to use getFileStatus, which throws an exception if the file doesn't exist, which I don't think is an appropriate use of that method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9928-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 05:41:40 2010
Return-Path: <common-issues-return-9928-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73295 invoked from network); 2 Aug 2010 05:41:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 05:41:40 -0000
Received: (qmail 18828 invoked by uid 500); 2 Aug 2010 05:41:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18701 invoked by uid 500); 2 Aug 2010 05:41:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18693 invoked by uid 99); 2 Aug 2010 05:41:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 05:41:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 05:41:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o725fH2j027273
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 05:41:17 GMT
Message-ID: <10064811.113191280727677005.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 01:41:17 -0400 (EDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5759) IllegalArgumentException when
 CombineFileInputFormat is used as job InputFormat
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-5759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amareshwari Sriramadasu updated HADOOP-5759:
--------------------------------------------

    Attachment: patch-5759-ydist.txt

Patch for Yahoo! distribution

> IllegalArgumentException when CombineFileInputFormat is used as job InputFormat
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-5759
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5759
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.1
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.20.2
>
>         Attachments: patch-5759-1.txt, patch-5759-2.txt, patch-5759-ydist.txt, patch-5759.txt
>
>
> As per my understanding, CombineFileInputFormat is creating splits with rackname as split location. 
> When I use CombineFileInputFormat as the InputFormat for job, job initialization fails with following exception :
> 2009-04-28 14:10:40,162 ERROR mapred.EagerTaskInitializationListener (EagerTaskInitializationListener.java:run(83)) - Job initialization failed:
> java.lang.IllegalArgumentException: Network location name contains /: /default-rack
>   at org.apache.hadoop.net.NodeBase.set(NodeBase.java:76)
>   at org.apache.hadoop.net.NodeBase.<init>(NodeBase.java:57)
>   at org.apache.hadoop.mapred.JobTracker.addHostToNodeMapping(JobTracker.java:2342)
>   at org.apache.hadoop.mapred.JobTracker.resolveAndAddToTopology(JobTracker.java:2336)
>   at org.apache.hadoop.mapred.JobInProgress.createCache(JobInProgress.java:344)
>   at org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:441)
>   at org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:81)
>   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
>   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>   at java.lang.Thread.run(Thread.java:619)
> When I changed CombineFileInputFormat to pass just rackname (without '/'), JT wrongly resolves  the node as /default-rack/<rack-name>.
> Solution is to pass hostnames holding the block(on the rack),  instead of rackname.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9929-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 07:24:41 2010
Return-Path: <common-issues-return-9929-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6337 invoked from network); 2 Aug 2010 07:24:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 07:24:41 -0000
Received: (qmail 65705 invoked by uid 500); 2 Aug 2010 07:24:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65573 invoked by uid 500); 2 Aug 2010 07:24:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65565 invoked by uid 99); 2 Aug 2010 07:24:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 07:24:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 07:24:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o727OGgS027953
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 07:24:16 GMT
Message-ID: <10134473.113661280733856346.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 03:24:16 -0400 (EDT)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6894) Common foundation for Hadoop client
 tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Common foundation for Hadoop client tools
-----------------------------------------

                 Key: HADOOP-6894
                 URL: https://issues.apache.org/jira/browse/HADOOP-6894
             Project: Hadoop Common
          Issue Type: New Feature
            Reporter: Alejandro Abdelnur


As Hadoop widespreads and matures the number of tools and utilities for users keeps growing.

Some of them are bundled with Hadoop core, some with Hadoop contrib, some on their own, some are full fledged servers on their own. For example, just to name a few: distcp, streaming, pipes, har, pig, hive, oozie.

Today there is no standard mechanism for making these tools available to users. Neither there is a standard mechanism for these tools to integrate and distributed them with each other.

The lack of a common foundation creates issues for developers and users.


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9930-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 07:27:40 2010
Return-Path: <common-issues-return-9930-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7353 invoked from network); 2 Aug 2010 07:27:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 07:27:40 -0000
Received: (qmail 67567 invoked by uid 500); 2 Aug 2010 07:27:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67248 invoked by uid 500); 2 Aug 2010 07:27:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67240 invoked by uid 99); 2 Aug 2010 07:27:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 07:27:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 07:27:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o727RGsK027968
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 07:27:16 GMT
Message-ID: <23338976.113691280734036299.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 03:27:16 -0400 (EDT)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6894) Common foundation for Hadoop client
 tools
In-Reply-To: <10134473.113661280733856346.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-6894:
---------------------------------------

    Attachment: HadoopClientTools.pdf

This proposal defines simple and well understood guidelines for:

* Development (optional)
* Directory layout (mandatory)
* Packaging (mandatory)
* Platform packaging (optional)
* Stack release (optional)
* Execution (mandatory)

The Hadoop hadoop command line tool is out of scope.

Server tools (like start/stop Hadoop or Oozie) are out of scope.

> Common foundation for Hadoop client tools
> -----------------------------------------
>
>                 Key: HADOOP-6894
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6894
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Alejandro Abdelnur
>         Attachments: HadoopClientTools.pdf
>
>
> As Hadoop widespreads and matures the number of tools and utilities for users keeps growing.
> Some of them are bundled with Hadoop core, some with Hadoop contrib, some on their own, some are full fledged servers on their own. For example, just to name a few: distcp, streaming, pipes, har, pig, hive, oozie.
> Today there is no standard mechanism for making these tools available to users. Neither there is a standard mechanism for these tools to integrate and distributed them with each other.
> The lack of a common foundation creates issues for developers and users.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9931-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 17:26:41 2010
Return-Path: <common-issues-return-9931-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45964 invoked from network); 2 Aug 2010 17:26:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 17:26:41 -0000
Received: (qmail 96848 invoked by uid 500); 2 Aug 2010 17:26:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96757 invoked by uid 500); 2 Aug 2010 17:26:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96743 invoked by uid 99); 2 Aug 2010 17:26:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 17:26:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 17:26:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72HQIdi002728
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 17:26:19 GMT
Message-ID: <26672875.119141280769978779.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 13:26:18 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6890:
----------------------------------

    Attachment: improveListFiles.patch

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: improveListFiles.patch, listFilesInFS.patch
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9932-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 17:26:42 2010
Return-Path: <common-issues-return-9932-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46016 invoked from network); 2 Aug 2010 17:26:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 17:26:41 -0000
Received: (qmail 96959 invoked by uid 500); 2 Aug 2010 17:26:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96775 invoked by uid 500); 2 Aug 2010 17:26:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96758 invoked by uid 99); 2 Aug 2010 17:26:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 17:26:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 17:26:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72HQJP0002743
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 17:26:19 GMT
Message-ID: <26370499.119191280769979782.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 13:26:19 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6890:
----------------------------------

    Attachment:     (was: improveListFiles.patch)

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: improveListFiles.patch, listFilesInFS.patch
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9933-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 17:26:42 2010
Return-Path: <common-issues-return-9933-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46031 invoked from network); 2 Aug 2010 17:26:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 17:26:41 -0000
Received: (qmail 97126 invoked by uid 500); 2 Aug 2010 17:26:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97093 invoked by uid 500); 2 Aug 2010 17:26:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96988 invoked by uid 99); 2 Aug 2010 17:26:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 17:26:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 17:26:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72HQLZ5002773
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 17:26:21 GMT
Message-ID: <6656466.119291280769981054.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 13:26:21 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6890:
----------------------------------

    Attachment: improveListFiles.patch

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: improveListFiles.patch, listFilesInFS.patch
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9934-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 17:26:43 2010
Return-Path: <common-issues-return-9934-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46089 invoked from network); 2 Aug 2010 17:26:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 17:26:42 -0000
Received: (qmail 97293 invoked by uid 500); 2 Aug 2010 17:26:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97261 invoked by uid 500); 2 Aug 2010 17:26:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97253 invoked by uid 99); 2 Aug 2010 17:26:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 17:26:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 17:26:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72HQLeK002788
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 17:26:21 GMT
Message-ID: <30841605.119341280769981723.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 13:26:21 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Reopened: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang reopened HADOOP-6890:
-----------------------------------


> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: improveListFiles.patch, listFilesInFS.patch
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9935-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 17:26:45 2010
Return-Path: <common-issues-return-9935-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46209 invoked from network); 2 Aug 2010 17:26:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 17:26:44 -0000
Received: (qmail 98091 invoked by uid 500); 2 Aug 2010 17:26:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98044 invoked by uid 500); 2 Aug 2010 17:26:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97997 invoked by uid 99); 2 Aug 2010 17:26:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 17:26:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 17:26:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72HQKed002758
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 17:26:20 GMT
Message-ID: <4922825.119241280769980426.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 13:26:20 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6890:
----------------------------------

    Attachment:     (was: improveListFiles.patch)

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: improveListFiles.patch, listFilesInFS.patch
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9936-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 17:26:46 2010
Return-Path: <common-issues-return-9936-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46321 invoked from network); 2 Aug 2010 17:26:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 17:26:45 -0000
Received: (qmail 98734 invoked by uid 500); 2 Aug 2010 17:26:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98709 invoked by uid 500); 2 Aug 2010 17:26:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98701 invoked by uid 99); 2 Aug 2010 17:26:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 17:26:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 17:26:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72HQMm2002806
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 17:26:22 GMT
Message-ID: <3833802.119401280769982504.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 13:26:22 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6890:
----------------------------------

    Status: Patch Available  (was: Reopened)

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: improveListFiles.patch, listFilesInFS.patch
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9937-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 17:31:39 2010
Return-Path: <common-issues-return-9937-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47322 invoked from network); 2 Aug 2010 17:31:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 17:31:39 -0000
Received: (qmail 3361 invoked by uid 500); 2 Aug 2010 17:31:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3311 invoked by uid 500); 2 Aug 2010 17:31:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3303 invoked by uid 99); 2 Aug 2010 17:31:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 17:31:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 17:31:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72HVHxA002855
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 17:31:17 GMT
Message-ID: <31203020.119511280770277353.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 13:31:17 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6890:
----------------------------------

    Attachment:     (was: improveListFiles.patch)

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: improveListFiles.patch, listFilesInFS.patch
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9938-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 17:31:43 2010
Return-Path: <common-issues-return-9938-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47383 invoked from network); 2 Aug 2010 17:31:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 17:31:43 -0000
Received: (qmail 3530 invoked by uid 500); 2 Aug 2010 17:31:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3485 invoked by uid 500); 2 Aug 2010 17:31:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3477 invoked by uid 99); 2 Aug 2010 17:31:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 17:31:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 17:31:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72HVI2C002874
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 17:31:18 GMT
Message-ID: <7308996.119571280770278607.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 13:31:18 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6890:
----------------------------------

    Attachment: improveListFiles.patch

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: improveListFiles.patch, listFilesInFS.patch
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9939-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 17:54:42 2010
Return-Path: <common-issues-return-9939-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57905 invoked from network); 2 Aug 2010 17:54:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 17:54:42 -0000
Received: (qmail 45840 invoked by uid 500); 2 Aug 2010 17:54:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45792 invoked by uid 500); 2 Aug 2010 17:54:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45784 invoked by uid 99); 2 Aug 2010 17:54:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 17:54:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 17:54:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72HsGUP003142
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 17:54:17 GMT
Message-ID: <26345646.120051280771656885.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 13:54:16 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894621#action_12894621 ] 

Hadoop QA commented on HADOOP-6890:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12451056/improveListFiles.patch
  against trunk revision 980953.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/655/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/655/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/655/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/655/console

This message is automatically generated.

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: improveListFiles.patch, listFilesInFS.patch
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9940-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 17:54:43 2010
Return-Path: <common-issues-return-9940-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57951 invoked from network); 2 Aug 2010 17:54:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 17:54:43 -0000
Received: (qmail 46001 invoked by uid 500); 2 Aug 2010 17:54:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45958 invoked by uid 500); 2 Aug 2010 17:54:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45950 invoked by uid 99); 2 Aug 2010 17:54:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 17:54:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 17:54:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72HsJPS003167
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 17:54:19 GMT
Message-ID: <27297706.120131280771659082.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 13:54:19 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6894) Common foundation for Hadoop client
 tools
In-Reply-To: <10134473.113661280733856346.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894622#action_12894622 ] 

Owen O'Malley commented on HADOOP-6894:
---------------------------------------

Comments:
  1. We desperately need a packaging and deployment strategy in Hadoop. The current tarballs based deployment is  insufficient.
  2. A lot of discussion of potential layouts for deployment already happened on HADOOP-6255, which you've completely ignored. Installing each tool is a separate directory with the tool and version name is how it was done before we had package managers. Package managers have taken over because they are far more convenient than tarballs with install instructions. I don't think this proposal is moving the project in the correct direction. Granted, we probably need to produce tarballs for operating systems without package managers, but certainly I'd expect most installations to use packages.
  3. Any packaging and/or deployment strategy that ignores cluster deployment won't work.
  4. Naming versions based on the Hadoop version makes sense, but should only be to the branch. (ie. pig-0.7-0.20 not pig-0.7-0.20.2). Naturally this Jira isn't the place to discuss changes in other sub-projects (ie Pig).



> Common foundation for Hadoop client tools
> -----------------------------------------
>
>                 Key: HADOOP-6894
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6894
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Alejandro Abdelnur
>         Attachments: HadoopClientTools.pdf
>
>
> As Hadoop widespreads and matures the number of tools and utilities for users keeps growing.
> Some of them are bundled with Hadoop core, some with Hadoop contrib, some on their own, some are full fledged servers on their own. For example, just to name a few: distcp, streaming, pipes, har, pig, hive, oozie.
> Today there is no standard mechanism for making these tools available to users. Neither there is a standard mechanism for these tools to integrate and distributed them with each other.
> The lack of a common foundation creates issues for developers and users.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9941-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 18:17:42 2010
Return-Path: <common-issues-return-9941-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69390 invoked from network); 2 Aug 2010 18:17:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 18:17:40 -0000
Received: (qmail 86781 invoked by uid 500); 2 Aug 2010 18:17:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86700 invoked by uid 500); 2 Aug 2010 18:17:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86623 invoked by uid 99); 2 Aug 2010 18:17:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 18:17:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 18:17:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72IHJBQ003547
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 18:17:19 GMT
Message-ID: <16462933.120781280773039051.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 14:17:19 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6890:
----------------------------------

    Status: Open  (was: Patch Available)

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: improveListFiles.patch, improveListFiles1.patch, listFilesInFS.patch
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9942-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 18:17:42 2010
Return-Path: <common-issues-return-9942-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69422 invoked from network); 2 Aug 2010 18:17:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 18:17:41 -0000
Received: (qmail 86919 invoked by uid 500); 2 Aug 2010 18:17:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86874 invoked by uid 500); 2 Aug 2010 18:17:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86774 invoked by uid 99); 2 Aug 2010 18:17:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 18:17:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 18:17:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72IHKDg003566
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 18:17:20 GMT
Message-ID: <6600695.120841280773039995.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 14:17:19 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6890:
----------------------------------

    Status: Patch Available  (was: Open)

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: improveListFiles.patch, improveListFiles1.patch, listFilesInFS.patch
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9943-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 18:17:42 2010
Return-Path: <common-issues-return-9943-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69441 invoked from network); 2 Aug 2010 18:17:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 18:17:41 -0000
Received: (qmail 87079 invoked by uid 500); 2 Aug 2010 18:17:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87024 invoked by uid 500); 2 Aug 2010 18:17:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86880 invoked by uid 99); 2 Aug 2010 18:17:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 18:17:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 18:17:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72IHHWx003521
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 18:17:17 GMT
Message-ID: <1067010.120701280773037491.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 14:17:17 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6890:
----------------------------------

    Attachment: improveListFiles1.patch

Fixed the failed test.

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: improveListFiles.patch, improveListFiles1.patch, listFilesInFS.patch
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9944-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 18:35:39 2010
Return-Path: <common-issues-return-9944-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82635 invoked from network); 2 Aug 2010 18:35:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 18:35:39 -0000
Received: (qmail 14501 invoked by uid 500); 2 Aug 2010 18:35:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14476 invoked by uid 500); 2 Aug 2010 18:35:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14468 invoked by uid 99); 2 Aug 2010 18:35:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 18:35:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 18:35:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72IZH87003721
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 18:35:17 GMT
Message-ID: <14481747.121081280774117141.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 14:35:17 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6890:
----------------------------------

    Attachment: improveListFiles1.patch

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: improveListFiles.patch, improveListFiles1.patch, listFilesInFS.patch
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9945-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 18:35:43 2010
Return-Path: <common-issues-return-9945-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82691 invoked from network); 2 Aug 2010 18:35:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 18:35:42 -0000
Received: (qmail 14681 invoked by uid 500); 2 Aug 2010 18:35:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14650 invoked by uid 500); 2 Aug 2010 18:35:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14642 invoked by uid 99); 2 Aug 2010 18:35:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 18:35:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 18:35:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72IZIxO003740
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 18:35:18 GMT
Message-ID: <19341652.121141280774118300.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 14:35:18 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6890:
----------------------------------

    Attachment:     (was: improveListFiles1.patch)

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: improveListFiles.patch, improveListFiles1.patch, listFilesInFS.patch
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9946-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 18:44:44 2010
Return-Path: <common-issues-return-9946-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84730 invoked from network); 2 Aug 2010 18:44:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 18:44:44 -0000
Received: (qmail 23088 invoked by uid 500); 2 Aug 2010 18:44:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23052 invoked by uid 500); 2 Aug 2010 18:44:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23044 invoked by uid 99); 2 Aug 2010 18:44:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 18:44:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 18:44:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72IiK73003836
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 18:44:20 GMT
Message-ID: <32318850.121341280774660237.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 14:44:20 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6786) test-patch needs to verify Herriot
 integrity
In-Reply-To: <19336182.46451274829212745.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894642#action_12894642 ] 

Giridharan Kesavan commented on HADOOP-6786:
--------------------------------------------

{quote}
+  $PS auxwww | $GREP HadoopPatchProcess | /usr/bin/nawk '{print $2}' | /usr/bin/xargs -t -I {} /usr/bin/kill -9 {} > /dev/null
{quote}

all the patch test builds run on ubuntu machines and kill cmd seem to reside in /bin/kill 


> test-patch needs to verify Herriot integrity
> --------------------------------------------
>
>                 Key: HADOOP-6786
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6786
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-6786.patch
>
>
> Whenever a new patch is submitted for verification {{test-patch}} process has to make sure that none of Herriot bindings were broken.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9947-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 18:47:40 2010
Return-Path: <common-issues-return-9947-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85359 invoked from network); 2 Aug 2010 18:47:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 18:47:39 -0000
Received: (qmail 25917 invoked by uid 500); 2 Aug 2010 18:47:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25854 invoked by uid 500); 2 Aug 2010 18:47:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25846 invoked by uid 99); 2 Aug 2010 18:47:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 18:47:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 18:47:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72IlHIs003860
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 18:47:18 GMT
Message-ID: <15944848.121401280774837853.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 14:47:17 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894643#action_12894643 ] 

Hadoop QA commented on HADOOP-6890:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12451058/improveListFiles1.patch
  against trunk revision 980953.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/656/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/656/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/656/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/656/console

This message is automatically generated.

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: improveListFiles.patch, improveListFiles1.patch, listFilesInFS.patch
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9948-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 18:56:42 2010
Return-Path: <common-issues-return-9948-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87825 invoked from network); 2 Aug 2010 18:56:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 18:56:41 -0000
Received: (qmail 42606 invoked by uid 500); 2 Aug 2010 18:56:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42547 invoked by uid 500); 2 Aug 2010 18:56:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42539 invoked by uid 99); 2 Aug 2010 18:56:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 18:56:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 18:56:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72IuGe6003992
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 18:56:17 GMT
Message-ID: <22292055.121701280775376874.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 14:56:16 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6893) FileContext needs a method to test
 for existence of a file or directory.
In-Reply-To: <12314272.113111280726899551.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894650#action_12894650 ] 

Eli Collins commented on HADOOP-6893:
-------------------------------------

Hey Owen,

FileContext#exists was removed in HADOOP-6678. The motivation from Hairong was "in our production clusters, we often see that the number of getFileStatus calls is multiple times of the open calls. If we remove isFile, isDirectory, and exists from FileContext, users have to explicitly call getFileStatus first, it is more likely that they will write more efficient code"

Adding AFS#exists that doesn't throw an exception seems reasonable, out of curiosity what's the particular change you need this for?

Thanks,
Eli  

> FileContext needs a method to test for existence of a file or directory.
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6893
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6893
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>            Reporter: Owen O'Malley
>
> FileContext is missing a method to ask if a file or directory exists. Currently the only mechanism is to use getFileStatus, which throws an exception if the file doesn't exist, which I don't think is an appropriate use of that method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9949-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 19:23:39 2010
Return-Path: <common-issues-return-9949-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98553 invoked from network); 2 Aug 2010 19:23:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 19:23:39 -0000
Received: (qmail 78641 invoked by uid 500); 2 Aug 2010 19:23:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78200 invoked by uid 500); 2 Aug 2010 19:23:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78082 invoked by uid 99); 2 Aug 2010 19:23:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 19:23:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 19:23:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72JNHLY004295
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 19:23:17 GMT
Message-ID: <29835313.122281280776997114.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 15:23:17 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6890:
----------------------------------

    Attachment: improveListFiles2.patch

I can not believe that such a trivial patch made hudson failed twice. Sorry for not running all unit tests locally before submitting the patch previously.

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: improveListFiles.patch, improveListFiles1.patch, improveListFiles2.patch, listFilesInFS.patch
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9950-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 19:25:42 2010
Return-Path: <common-issues-return-9950-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98889 invoked from network); 2 Aug 2010 19:25:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 19:25:42 -0000
Received: (qmail 80301 invoked by uid 500); 2 Aug 2010 19:25:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80271 invoked by uid 500); 2 Aug 2010 19:25:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80263 invoked by uid 99); 2 Aug 2010 19:25:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 19:25:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 19:25:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72JPHrx004324
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 19:25:17 GMT
Message-ID: <31920802.122341280777117169.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 15:25:17 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6890:
----------------------------------

    Status: Open  (was: Patch Available)

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: improveListFiles.patch, improveListFiles1.patch, improveListFiles2.patch, listFilesInFS.patch
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9951-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 19:25:43 2010
Return-Path: <common-issues-return-9951-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98935 invoked from network); 2 Aug 2010 19:25:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 19:25:42 -0000
Received: (qmail 80471 invoked by uid 500); 2 Aug 2010 19:25:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80439 invoked by uid 500); 2 Aug 2010 19:25:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80431 invoked by uid 99); 2 Aug 2010 19:25:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 19:25:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 19:25:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72JPIHS004342
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 19:25:18 GMT
Message-ID: <11952749.122401280777118247.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 15:25:18 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6890:
----------------------------------

    Status: Patch Available  (was: Open)

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: improveListFiles.patch, improveListFiles1.patch, improveListFiles2.patch, listFilesInFS.patch
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9952-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 19:50:44 2010
Return-Path: <common-issues-return-9952-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7860 invoked from network); 2 Aug 2010 19:50:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 19:50:44 -0000
Received: (qmail 27934 invoked by uid 500); 2 Aug 2010 19:50:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27875 invoked by uid 500); 2 Aug 2010 19:50:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27867 invoked by uid 99); 2 Aug 2010 19:50:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 19:50:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 19:50:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72JoJ7h004628
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 19:50:19 GMT
Message-ID: <10936724.123081280778619705.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 15:50:19 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894674#action_12894674 ] 

Hadoop QA commented on HADOOP-6890:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12451066/improveListFiles2.patch
  against trunk revision 980953.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/657/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/657/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/657/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/657/console

This message is automatically generated.

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: improveListFiles.patch, improveListFiles1.patch, improveListFiles2.patch, listFilesInFS.patch
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9953-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 20:01:41 2010
Return-Path: <common-issues-return-9953-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11705 invoked from network); 2 Aug 2010 20:01:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 20:01:41 -0000
Received: (qmail 45890 invoked by uid 500); 2 Aug 2010 20:01:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45819 invoked by uid 500); 2 Aug 2010 20:01:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45811 invoked by uid 99); 2 Aug 2010 20:01:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 20:01:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 20:01:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72K1K14004892
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 20:01:20 GMT
Message-ID: <8210696.123741280779279985.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 16:01:19 -0400 (EDT)
From: "Stephen Watt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6895) Native Libraries do not load if a
 different platform signature is returned from
 org.apache.hadoop.util.PlatformName
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Native Libraries do not load if a different platform signature is returned from org.apache.hadoop.util.PlatformName
-------------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-6895
                 URL: https://issues.apache.org/jira/browse/HADOOP-6895
             Project: Hadoop Common
          Issue Type: Bug
          Components: native
         Environment: SLES 10, IBM Java 6, Hadoop 0.21.0-rc0
            Reporter: Stephen Watt
            Priority: Minor
             Fix For: 0.21.0, 0.22.0


bin/hadoop-config.sh has an environment variable called JAVA_PLATFORM which is set to to the results returned by org.apache.hadoop.util.PlatformName . These results are sometimes unique to the JRE being used. Although the value returned for 64 Bit Sun/Oracle Java and 64 Bit IBM Java is the same, it is different for the corresponding 32 Bit JREs. 

The issue is that the value returned is used in creating the path to the native libraries on disk, i.e ${HADOOP_COMMON_HOME}/lib/native/${JAVA_PLATFORM}

Since the path on disk is fixed with the Sun JRE value /lib/native/Linux-i386-32 it therefore fails when it attempts to load the native libraries with the value returned with 32 Bit IBM Java, /lib/native/Linux-x86-32

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9954-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 20:05:39 2010
Return-Path: <common-issues-return-9954-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17122 invoked from network); 2 Aug 2010 20:05:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 20:05:39 -0000
Received: (qmail 51823 invoked by uid 500); 2 Aug 2010 20:05:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51778 invoked by uid 500); 2 Aug 2010 20:05:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51770 invoked by uid 99); 2 Aug 2010 20:05:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 20:05:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 20:05:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72K5GcU004911
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 20:05:17 GMT
Message-ID: <21814161.123771280779516768.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 16:05:16 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6892) Common component of HDFS-1150
 (Verify datanodes' identities to clients in secure clusters)
In-Reply-To: <11193881.95511280530396700.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894680#action_12894680 ] 

Jakob Homan commented on HADOOP-6892:
-------------------------------------

I believe the javadoc warning is bogus.  None of the reported warnings have to do with this patch:
{noformat}
     [exec]   [javadoc] Constructing Javadoc information...
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:31: warning: sun.security.krb5.Config is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.Config;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:32: warning: sun.security.krb5.KrbException is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.KrbException;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:81: warning: sun.security.krb5.Config is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc]   private static Config kerbConf;
     [exec]   [javadoc]                  ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/SecurityUtil.java:39: warning: sun.security.jgss.krb5.Krb5Util is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.jgss.krb5.Krb5Util;
     [exec]   [javadoc]                              ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/SecurityUtil.java:40: warning: sun.security.krb5.Credentials is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.Credentials;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/SecurityUtil.java:41: warning: sun.security.krb5.PrincipalName is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.PrincipalName;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] ExcludePrivateAnnotationsStandardDoclet{noformat}

> Common component of HDFS-1150 (Verify datanodes' identities to clients in secure clusters)
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6892
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6892
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6892.patch
>
>
> HDFS-1150 will have changes to the start-up scripts and HttpServer.  These are handled here.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9955-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 20:05:42 2010
Return-Path: <common-issues-return-9955-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17183 invoked from network); 2 Aug 2010 20:05:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 20:05:42 -0000
Received: (qmail 51977 invoked by uid 500); 2 Aug 2010 20:05:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51949 invoked by uid 500); 2 Aug 2010 20:05:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51941 invoked by uid 99); 2 Aug 2010 20:05:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 20:05:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 20:05:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72K5IgD004923
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 20:05:18 GMT
Message-ID: <25597509.123811280779518308.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 16:05:18 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894681#action_12894681 ] 

Jitendra Nath Pandey commented on HADOOP-6706:
----------------------------------------------

javadoc warnings are in KerberosName.java and SecurityUtil.java and are unrelated to this patch.

The patch was manually tested on Y20 branch.

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706-bp20-2.patch, 6706.bp20.1.patch, 6706.bp20.patch, HADOOP-6706-BP20-fix1.patch, HADOOP-6706-BP20-fix2.patch, HADOOP-6706-BP20-fix3.patch, HADOOP-6706.2.patch, HADOOP-6706.4.patch, HADOOP-6706.5.patch, HADOOP-6706.6.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9956-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 20:08:47 2010
Return-Path: <common-issues-return-9956-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18495 invoked from network); 2 Aug 2010 20:08:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 20:08:46 -0000
Received: (qmail 53189 invoked by uid 500); 2 Aug 2010 20:08:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53119 invoked by uid 500); 2 Aug 2010 20:08:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53111 invoked by uid 99); 2 Aug 2010 20:08:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 20:08:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 20:08:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72K8PXH005032
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 20:08:25 GMT
Message-ID: <3258677.124151280779705368.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 16:08:25 -0400 (EDT)
From: "Stephen Watt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6895) Native Libraries do not load if a
 different platform signature is returned from
 org.apache.hadoop.util.PlatformName
In-Reply-To: <8210696.123741280779279985.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stephen Watt updated HADOOP-6895:
---------------------------------

    Attachment: HADOOP-6895.patch

> Native Libraries do not load if a different platform signature is returned from org.apache.hadoop.util.PlatformName
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6895
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6895
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>         Environment: SLES 10, IBM Java 6, Hadoop 0.21.0-rc0
>            Reporter: Stephen Watt
>            Priority: Minor
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6895.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> bin/hadoop-config.sh has an environment variable called JAVA_PLATFORM which is set to to the results returned by org.apache.hadoop.util.PlatformName . These results are sometimes unique to the JRE being used. Although the value returned for 64 Bit Sun/Oracle Java and 64 Bit IBM Java is the same, it is different for the corresponding 32 Bit JREs. 
> The issue is that the value returned is used in creating the path to the native libraries on disk, i.e ${HADOOP_COMMON_HOME}/lib/native/${JAVA_PLATFORM}
> Since the path on disk is fixed with the Sun JRE value /lib/native/Linux-i386-32 it therefore fails when it attempts to load the native libraries with the value returned with 32 Bit IBM Java, /lib/native/Linux-x86-32

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9957-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 20:08:49 2010
Return-Path: <common-issues-return-9957-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18569 invoked from network); 2 Aug 2010 20:08:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 20:08:49 -0000
Received: (qmail 53498 invoked by uid 500); 2 Aug 2010 20:08:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53298 invoked by uid 500); 2 Aug 2010 20:08:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53290 invoked by uid 99); 2 Aug 2010 20:08:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 20:08:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 20:08:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72K8PLx005026
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 20:08:25 GMT
Message-ID: <21942432.124131280779705143.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 16:08:25 -0400 (EDT)
From: "Stephen Watt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6895) Native Libraries do not load if a
 different platform signature is returned from
 org.apache.hadoop.util.PlatformName
In-Reply-To: <8210696.123741280779279985.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stephen Watt updated HADOOP-6895:
---------------------------------

    Status: Patch Available  (was: Open)

Since the native libraries work regardless of JRE, I am applying a simple patch that checks for the IBM 32 JRE Signature and if found sets the JAVA_PLATFORM variable to the Oracle/Sun Signature.

> Native Libraries do not load if a different platform signature is returned from org.apache.hadoop.util.PlatformName
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6895
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6895
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>         Environment: SLES 10, IBM Java 6, Hadoop 0.21.0-rc0
>            Reporter: Stephen Watt
>            Priority: Minor
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6895.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> bin/hadoop-config.sh has an environment variable called JAVA_PLATFORM which is set to to the results returned by org.apache.hadoop.util.PlatformName . These results are sometimes unique to the JRE being used. Although the value returned for 64 Bit Sun/Oracle Java and 64 Bit IBM Java is the same, it is different for the corresponding 32 Bit JREs. 
> The issue is that the value returned is used in creating the path to the native libraries on disk, i.e ${HADOOP_COMMON_HOME}/lib/native/${JAVA_PLATFORM}
> Since the path on disk is fixed with the Sun JRE value /lib/native/Linux-i386-32 it therefore fails when it attempts to load the native libraries with the value returned with 32 Bit IBM Java, /lib/native/Linux-x86-32

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9958-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 20:26:41 2010
Return-Path: <common-issues-return-9958-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24813 invoked from network); 2 Aug 2010 20:26:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 20:26:41 -0000
Received: (qmail 75633 invoked by uid 500); 2 Aug 2010 20:26:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75574 invoked by uid 500); 2 Aug 2010 20:26:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75566 invoked by uid 99); 2 Aug 2010 20:26:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 20:26:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 20:26:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72KQGvM005217
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 20:26:16 GMT
Message-ID: <22659403.124581280780776442.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 16:26:16 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894690#action_12894690 ] 

Devaraj Das commented on HADOOP-6706:
-------------------------------------

Looks like due to the order in which the patches from HADOOP-6706 and HADOOP-6718 got applied, there is a small problem in the part where the exception thrown by setupSaslConnection is handled. handleSaslConnection is not called anymore. Please remove that call from your patch.

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706-bp20-2.patch, 6706.bp20.1.patch, 6706.bp20.patch, HADOOP-6706-BP20-fix1.patch, HADOOP-6706-BP20-fix2.patch, HADOOP-6706-BP20-fix3.patch, HADOOP-6706.2.patch, HADOOP-6706.4.patch, HADOOP-6706.5.patch, HADOOP-6706.6.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9959-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 20:30:41 2010
Return-Path: <common-issues-return-9959-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26360 invoked from network); 2 Aug 2010 20:30:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 20:30:40 -0000
Received: (qmail 83525 invoked by uid 500); 2 Aug 2010 20:30:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83458 invoked by uid 500); 2 Aug 2010 20:30:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83450 invoked by uid 99); 2 Aug 2010 20:30:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 20:30:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 20:30:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72KUIGP005274
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 20:30:18 GMT
Message-ID: <19280514.124721280781018510.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 16:30:18 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6895) Native Libraries do not load if a
 different platform signature is returned from
 org.apache.hadoop.util.PlatformName
In-Reply-To: <8210696.123741280779279985.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894694#action_12894694 ] 

Hadoop QA commented on HADOOP-6895:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12451070/HADOOP-6895.patch
  against trunk revision 980953.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/658/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/658/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/658/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/658/console

This message is automatically generated.

> Native Libraries do not load if a different platform signature is returned from org.apache.hadoop.util.PlatformName
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6895
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6895
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>         Environment: SLES 10, IBM Java 6, Hadoop 0.21.0-rc0
>            Reporter: Stephen Watt
>            Priority: Minor
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6895.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> bin/hadoop-config.sh has an environment variable called JAVA_PLATFORM which is set to to the results returned by org.apache.hadoop.util.PlatformName . These results are sometimes unique to the JRE being used. Although the value returned for 64 Bit Sun/Oracle Java and 64 Bit IBM Java is the same, it is different for the corresponding 32 Bit JREs. 
> The issue is that the value returned is used in creating the path to the native libraries on disk, i.e ${HADOOP_COMMON_HOME}/lib/native/${JAVA_PLATFORM}
> Since the path on disk is fixed with the Sun JRE value /lib/native/Linux-i386-32 it therefore fails when it attempts to load the native libraries with the value returned with 32 Bit IBM Java, /lib/native/Linux-x86-32

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9960-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 20:30:41 2010
Return-Path: <common-issues-return-9960-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26377 invoked from network); 2 Aug 2010 20:30:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 20:30:40 -0000
Received: (qmail 83547 invoked by uid 500); 2 Aug 2010 20:30:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83469 invoked by uid 500); 2 Aug 2010 20:30:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83460 invoked by uid 99); 2 Aug 2010 20:30:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 20:30:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 20:30:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72KUJEt005292
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 20:30:19 GMT
Message-ID: <9810095.124781280781019327.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 16:30:19 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6890) Improve listFiles API introduced by
 HADOOP-6870
In-Reply-To: <6227281.77781280442376526.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6890:
----------------------------------

        Status: Resolved  (was: Patch Available)
    Resolution: Fixed

Most of "improveListFiles2.patch" copies the change made in FileContext into FileSystem. Since the change is pretty trivial and FileSystem is an interface to be deprecated soon, so I just committed it.

> Improve listFiles API introduced by HADOOP-6870
> -----------------------------------------------
>
>                 Key: HADOOP-6890
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6890
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: improveListFiles.patch, improveListFiles1.patch, improveListFiles2.patch, listFilesInFS.patch
>
>
> This jira is mainly for addressing Suresh's review comments for HADOOP-6870:
>    1. General comment: I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
>    2. Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result. Should we ignore if a directory was in stack and was not found later when iterating through it?
>    3. FileSystem.java
>           * listFile() - method javadoc could be better organized - first write about if path is directory and two cases recursive=true and false. Then if path is file and two cases recursive=true or false.
>           * listFile() - document throwing RuntimeException, UnsupportedOperationException and the possible cause. IOException is no longer thrown.
>    4. TestListFiles.java
>           * testDirectory() - comments test empty directory and test directory with 1 file should be moved up to relevant sections of the test.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9961-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 20:57:38 2010
Return-Path: <common-issues-return-9961-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31597 invoked from network); 2 Aug 2010 20:57:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 20:57:38 -0000
Received: (qmail 17513 invoked by uid 500); 2 Aug 2010 20:57:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17441 invoked by uid 500); 2 Aug 2010 20:57:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17432 invoked by uid 99); 2 Aug 2010 20:57:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 20:57:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 20:57:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72KvHUp005599
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 20:57:17 GMT
Message-ID: <2848601.125431280782637090.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 16:57:17 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6892) Common component of HDFS-1150
 (Verify datanodes' identities to clients in secure clusters)
In-Reply-To: <11193881.95511280530396700.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894708#action_12894708 ] 

Devaraj Das commented on HADOOP-6892:
-------------------------------------

+1


> Common component of HDFS-1150 (Verify datanodes' identities to clients in secure clusters)
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6892
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6892
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6892.patch
>
>
> HDFS-1150 will have changes to the start-up scripts and HttpServer.  These are handled here.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9962-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 21:11:39 2010
Return-Path: <common-issues-return-9962-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40188 invoked from network); 2 Aug 2010 21:11:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 21:11:38 -0000
Received: (qmail 28823 invoked by uid 500); 2 Aug 2010 21:11:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28736 invoked by uid 500); 2 Aug 2010 21:11:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28728 invoked by uid 99); 2 Aug 2010 21:11:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 21:11:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 21:11:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72LBHQU005692
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 21:11:17 GMT
Message-ID: <1784737.125501280783477019.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 17:11:17 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6896) Remove the Util separation in
 FileContext
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Remove the Util separation in FileContext
-----------------------------------------

                 Key: HADOOP-6896
                 URL: https://issues.apache.org/jira/browse/HADOOP-6896
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs
            Reporter: Owen O'Malley
            Assignee: Sanjay Radia


I think from a usability point of view, it would be nicer to write:

{code}
fc.exist(path);
{code}

instead of:

{code}
fc.utils().exist(path);
{code}

We should use the javadoc to mark the fact that the util functions are not atomic.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9963-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 21:14:39 2010
Return-Path: <common-issues-return-9963-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41243 invoked from network); 2 Aug 2010 21:14:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 21:14:39 -0000
Received: (qmail 31016 invoked by uid 500); 2 Aug 2010 21:14:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30931 invoked by uid 500); 2 Aug 2010 21:14:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30923 invoked by uid 99); 2 Aug 2010 21:14:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 21:14:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 21:14:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72LEH3i005720
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 21:14:17 GMT
Message-ID: <7210988.125561280783657895.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 17:14:17 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6892) Common component of HDFS-1150 (Verify
 datanodes' identities to clients in secure clusters)
In-Reply-To: <11193881.95511280530396700.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6892:
--------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

Thanks for the review, Devaraj.  I've committed this.  Resolving as fixed. 

> Common component of HDFS-1150 (Verify datanodes' identities to clients in secure clusters)
> ------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6892
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6892
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6892.patch
>
>
> HDFS-1150 will have changes to the start-up scripts and HttpServer.  These are handled here.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9964-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 21:33:42 2010
Return-Path: <common-issues-return-9964-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49186 invoked from network); 2 Aug 2010 21:33:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 21:33:42 -0000
Received: (qmail 58189 invoked by uid 500); 2 Aug 2010 21:33:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58114 invoked by uid 500); 2 Aug 2010 21:33:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58106 invoked by uid 99); 2 Aug 2010 21:33:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 21:33:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 21:33:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72LXHTj005894
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 21:33:18 GMT
Message-ID: <22833335.126021280784797520.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 17:33:17 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6786) test-patch needs to verify Herriot
 integrity
In-Reply-To: <19336182.46451274829212745.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894717#action_12894717 ] 

Konstantin Boudnik commented on HADOOP-6786:
--------------------------------------------

bq. all the patch test builds run on ubuntu machines and kill cmd seem to reside in /bin/kill 
I wonder how it works in two other places (for core and contrib tests) then?

> test-patch needs to verify Herriot integrity
> --------------------------------------------
>
>                 Key: HADOOP-6786
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6786
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-6786.patch
>
>
> Whenever a new patch is submitted for verification {{test-patch}} process has to make sure that none of Herriot bindings were broken.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9965-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 21:49:41 2010
Return-Path: <common-issues-return-9965-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53354 invoked from network); 2 Aug 2010 21:49:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 21:49:41 -0000
Received: (qmail 84254 invoked by uid 500); 2 Aug 2010 21:49:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84227 invoked by uid 500); 2 Aug 2010 21:49:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84219 invoked by uid 99); 2 Aug 2010 21:49:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 21:49:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 21:49:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72LnHVv006337
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 21:49:17 GMT
Message-ID: <14795254.127311280785757112.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 17:49:17 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6706:
-----------------------------------------

    Attachment: HADOOP-6706.7.patch

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706-bp20-2.patch, 6706.bp20.1.patch, 6706.bp20.patch, HADOOP-6706-BP20-fix1.patch, HADOOP-6706-BP20-fix2.patch, HADOOP-6706-BP20-fix3.patch, HADOOP-6706.2.patch, HADOOP-6706.4.patch, HADOOP-6706.5.patch, HADOOP-6706.6.patch, HADOOP-6706.7.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9966-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 21:49:41 2010
Return-Path: <common-issues-return-9966-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53399 invoked from network); 2 Aug 2010 21:49:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 21:49:41 -0000
Received: (qmail 84426 invoked by uid 500); 2 Aug 2010 21:49:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84387 invoked by uid 500); 2 Aug 2010 21:49:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84379 invoked by uid 99); 2 Aug 2010 21:49:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 21:49:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 21:49:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72LnHEC006349
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 21:49:17 GMT
Message-ID: <3117594.127351280785757902.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 17:49:17 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6706:
-----------------------------------------

    Status: Open  (was: Patch Available)

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706-bp20-2.patch, 6706.bp20.1.patch, 6706.bp20.patch, HADOOP-6706-BP20-fix1.patch, HADOOP-6706-BP20-fix2.patch, HADOOP-6706-BP20-fix3.patch, HADOOP-6706.2.patch, HADOOP-6706.4.patch, HADOOP-6706.5.patch, HADOOP-6706.6.patch, HADOOP-6706.7.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9967-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 21:51:42 2010
Return-Path: <common-issues-return-9967-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53601 invoked from network); 2 Aug 2010 21:51:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 21:51:42 -0000
Received: (qmail 85585 invoked by uid 500); 2 Aug 2010 21:51:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85295 invoked by uid 500); 2 Aug 2010 21:51:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85287 invoked by uid 99); 2 Aug 2010 21:51:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 21:51:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 21:51:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72LpHLY006368
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 21:51:17 GMT
Message-ID: <9873288.127411280785877966.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 17:51:17 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6706:
-----------------------------------------

    Status: Patch Available  (was: Open)

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706-bp20-2.patch, 6706.bp20.1.patch, 6706.bp20.patch, HADOOP-6706-BP20-fix1.patch, HADOOP-6706-BP20-fix2.patch, HADOOP-6706-BP20-fix3.patch, HADOOP-6706.2.patch, HADOOP-6706.4.patch, HADOOP-6706.5.patch, HADOOP-6706.6.patch, HADOOP-6706.7.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9968-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 21:51:43 2010
Return-Path: <common-issues-return-9968-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53644 invoked from network); 2 Aug 2010 21:51:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 21:51:42 -0000
Received: (qmail 85711 invoked by uid 500); 2 Aug 2010 21:51:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85530 invoked by uid 500); 2 Aug 2010 21:51:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85444 invoked by uid 99); 2 Aug 2010 21:51:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 21:51:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 21:51:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72LpIn2006381
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 21:51:18 GMT
Message-ID: <8323978.127451280785878373.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 17:51:18 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894724#action_12894724 ] 

Jitendra Nath Pandey commented on HADOOP-6706:
----------------------------------------------

New patch addressing the comment.

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706-bp20-2.patch, 6706.bp20.1.patch, 6706.bp20.patch, HADOOP-6706-BP20-fix1.patch, HADOOP-6706-BP20-fix2.patch, HADOOP-6706-BP20-fix3.patch, HADOOP-6706.2.patch, HADOOP-6706.4.patch, HADOOP-6706.5.patch, HADOOP-6706.6.patch, HADOOP-6706.7.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9969-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 21:58:46 2010
Return-Path: <common-issues-return-9969-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55794 invoked from network); 2 Aug 2010 21:58:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 21:58:46 -0000
Received: (qmail 93320 invoked by uid 500); 2 Aug 2010 21:58:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93263 invoked by uid 500); 2 Aug 2010 21:58:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93255 invoked by uid 99); 2 Aug 2010 21:58:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 21:58:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 21:58:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72LwHJr006437
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 21:58:17 GMT
Message-ID: <24464104.127521280786297287.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 17:58:17 -0400 (EDT)
From: "Stephen Watt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6895) Native Libraries do not load if a
 different platform signature is returned from
 org.apache.hadoop.util.PlatformName
In-Reply-To: <8210696.123741280779279985.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894725#action_12894725 ] 

Stephen Watt commented on HADOOP-6895:
--------------------------------------

There is a JRE security library exception being thrown when attempting to format the namenode or starting Hadoop and using IBM Java as the JRE. This makes the patch hard to validate when testing with IBM Java. I'll write up a JIRA ticket for the security issue and link it to this one as a dependency.

> Native Libraries do not load if a different platform signature is returned from org.apache.hadoop.util.PlatformName
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6895
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6895
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>         Environment: SLES 10, IBM Java 6, Hadoop 0.21.0-rc0
>            Reporter: Stephen Watt
>            Priority: Minor
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6895.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> bin/hadoop-config.sh has an environment variable called JAVA_PLATFORM which is set to to the results returned by org.apache.hadoop.util.PlatformName . These results are sometimes unique to the JRE being used. Although the value returned for 64 Bit Sun/Oracle Java and 64 Bit IBM Java is the same, it is different for the corresponding 32 Bit JREs. 
> The issue is that the value returned is used in creating the path to the native libraries on disk, i.e ${HADOOP_COMMON_HOME}/lib/native/${JAVA_PLATFORM}
> Since the path on disk is fixed with the Sun JRE value /lib/native/Linux-i386-32 it therefore fails when it attempts to load the native libraries with the value returned with 32 Bit IBM Java, /lib/native/Linux-x86-32

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9970-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 22:22:41 2010
Return-Path: <common-issues-return-9970-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68888 invoked from network); 2 Aug 2010 22:22:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 22:22:41 -0000
Received: (qmail 19565 invoked by uid 500); 2 Aug 2010 22:22:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19498 invoked by uid 500); 2 Aug 2010 22:22:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19490 invoked by uid 99); 2 Aug 2010 22:22:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 22:22:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 22:22:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72MMGL5006646
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 22:22:16 GMT
Message-ID: <6216628.127811280787736527.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 18:22:16 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894731#action_12894731 ] 

Hadoop QA commented on HADOOP-6706:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12451082/HADOOP-6706.7.patch
  against trunk revision 981688.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs.  The patch appears to introduce 1 new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/659/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/659/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/659/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/659/console

This message is automatically generated.

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706-bp20-2.patch, 6706.bp20.1.patch, 6706.bp20.patch, HADOOP-6706-BP20-fix1.patch, HADOOP-6706-BP20-fix2.patch, HADOOP-6706-BP20-fix3.patch, HADOOP-6706.2.patch, HADOOP-6706.4.patch, HADOOP-6706.5.patch, HADOOP-6706.6.patch, HADOOP-6706.7.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9971-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 23:43:42 2010
Return-Path: <common-issues-return-9971-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96842 invoked from network); 2 Aug 2010 23:43:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 23:43:42 -0000
Received: (qmail 84226 invoked by uid 500); 2 Aug 2010 23:43:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84163 invoked by uid 500); 2 Aug 2010 23:43:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84155 invoked by uid 99); 2 Aug 2010 23:43:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 23:43:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 23:43:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72NhIWk007351
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 23:43:18 GMT
Message-ID: <33473576.129041280792598314.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 19:43:18 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6706:
-----------------------------------------

    Attachment: HADOOP-6706.8.patch

Updated patch fixing the findbugs warning.

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706-bp20-2.patch, 6706.bp20.1.patch, 6706.bp20.patch, HADOOP-6706-BP20-fix1.patch, HADOOP-6706-BP20-fix2.patch, HADOOP-6706-BP20-fix3.patch, HADOOP-6706.2.patch, HADOOP-6706.4.patch, HADOOP-6706.5.patch, HADOOP-6706.6.patch, HADOOP-6706.7.patch, HADOOP-6706.8.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9972-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 23:45:39 2010
Return-Path: <common-issues-return-9972-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98364 invoked from network); 2 Aug 2010 23:45:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 23:45:39 -0000
Received: (qmail 87098 invoked by uid 500); 2 Aug 2010 23:45:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87034 invoked by uid 500); 2 Aug 2010 23:45:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87026 invoked by uid 99); 2 Aug 2010 23:45:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 23:45:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 23:45:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72NjHZ9007393
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 23:45:17 GMT
Message-ID: <4724725.129151280792717720.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 19:45:17 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6706:
-----------------------------------------

    Status: Patch Available  (was: Open)

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706-bp20-2.patch, 6706.bp20.1.patch, 6706.bp20.patch, HADOOP-6706-BP20-fix1.patch, HADOOP-6706-BP20-fix2.patch, HADOOP-6706-BP20-fix3.patch, HADOOP-6706.2.patch, HADOOP-6706.4.patch, HADOOP-6706.5.patch, HADOOP-6706.6.patch, HADOOP-6706.7.patch, HADOOP-6706.8.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9973-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 02 23:45:41 2010
Return-Path: <common-issues-return-9973-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98417 invoked from network); 2 Aug 2010 23:45:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Aug 2010 23:45:40 -0000
Received: (qmail 87234 invoked by uid 500); 2 Aug 2010 23:45:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87203 invoked by uid 500); 2 Aug 2010 23:45:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87195 invoked by uid 99); 2 Aug 2010 23:45:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 23:45:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 23:45:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o72NjH8Q007380
	for <common-issues@hadoop.apache.org>; Mon, 2 Aug 2010 23:45:17 GMT
Message-ID: <22224639.129111280792717043.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 19:45:17 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6706:
-----------------------------------------

    Status: Open  (was: Patch Available)

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706-bp20-2.patch, 6706.bp20.1.patch, 6706.bp20.patch, HADOOP-6706-BP20-fix1.patch, HADOOP-6706-BP20-fix2.patch, HADOOP-6706-BP20-fix3.patch, HADOOP-6706.2.patch, HADOOP-6706.4.patch, HADOOP-6706.5.patch, HADOOP-6706.6.patch, HADOOP-6706.7.patch, HADOOP-6706.8.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9974-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 03 00:11:42 2010
Return-Path: <common-issues-return-9974-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6180 invoked from network); 3 Aug 2010 00:11:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Aug 2010 00:11:42 -0000
Received: (qmail 6044 invoked by uid 500); 3 Aug 2010 00:11:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5924 invoked by uid 500); 3 Aug 2010 00:11:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5857 invoked by uid 99); 3 Aug 2010 00:11:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Aug 2010 00:11:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Aug 2010 00:11:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o730BHKu007566
	for <common-issues@hadoop.apache.org>; Tue, 3 Aug 2010 00:11:17 GMT
Message-ID: <19705877.129401280794277390.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 20:11:17 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894754#action_12894754 ] 

Hadoop QA commented on HADOOP-6706:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12451090/HADOOP-6706.8.patch
  against trunk revision 981688.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/660/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/660/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/660/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/660/console

This message is automatically generated.

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706-bp20-2.patch, 6706.bp20.1.patch, 6706.bp20.patch, HADOOP-6706-BP20-fix1.patch, HADOOP-6706-BP20-fix2.patch, HADOOP-6706-BP20-fix3.patch, HADOOP-6706.2.patch, HADOOP-6706.4.patch, HADOOP-6706.5.patch, HADOOP-6706.6.patch, HADOOP-6706.7.patch, HADOOP-6706.8.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9975-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 03 00:55:42 2010
Return-Path: <common-issues-return-9975-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26077 invoked from network); 3 Aug 2010 00:55:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Aug 2010 00:55:41 -0000
Received: (qmail 40367 invoked by uid 500); 3 Aug 2010 00:55:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40271 invoked by uid 500); 3 Aug 2010 00:55:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40261 invoked by uid 99); 3 Aug 2010 00:55:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Aug 2010 00:55:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Aug 2010 00:55:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o730tGe0008022
	for <common-issues@hadoop.apache.org>; Tue, 3 Aug 2010 00:55:17 GMT
Message-ID: <9303427.130511280796916945.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 20:55:16 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6706) Relogin behavior for RPC clients
 could be improved
In-Reply-To: <19329091.128961271291643642.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6706:
--------------------------------

        Status: Resolved  (was: Patch Available)
    Resolution: Fixed

I just committed this. Thanks, Jitendra for the trunk patches.

> Relogin behavior for RPC clients could be improved
> --------------------------------------------------
>
>                 Key: HADOOP-6706
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6706
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6706-bp20-2.patch, 6706.bp20.1.patch, 6706.bp20.patch, HADOOP-6706-BP20-fix1.patch, HADOOP-6706-BP20-fix2.patch, HADOOP-6706-BP20-fix3.patch, HADOOP-6706.2.patch, HADOOP-6706.4.patch, HADOOP-6706.5.patch, HADOOP-6706.6.patch, HADOOP-6706.7.patch, HADOOP-6706.8.patch
>
>
> Currently, the relogin in the RPC client happens on only a SaslException. But we have seen cases where other exceptions are thrown (like IllegalStateException when the client's ticket is invalid). This jira is to fix that behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9976-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 03 01:18:46 2010
Return-Path: <common-issues-return-9976-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35272 invoked from network); 3 Aug 2010 01:18:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Aug 2010 01:18:46 -0000
Received: (qmail 50241 invoked by uid 500); 3 Aug 2010 01:18:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50159 invoked by uid 500); 3 Aug 2010 01:18:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50140 invoked by uid 99); 3 Aug 2010 01:18:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Aug 2010 01:18:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Aug 2010 01:18:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o731IKT2008256
	for <common-issues@hadoop.apache.org>; Tue, 3 Aug 2010 01:18:21 GMT
Message-ID: <151844.130821280798300978.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 21:18:20 -0400 (EDT)
From: "Nicholas Carlini (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894773#action_12894773 ] 

Nicholas Carlini commented on HADOOP-6837:
------------------------------------------

Responding to the major comments -- will upload a patch that fixes these and the smaller comments soon.

FakeInputStream LinkedList:
This LinkedList can get fairly long, depending on how write is called. Worst case it can have upwards of 12 million elements, which is far beyond acceptable. This is the case if the write(single_byte) is called over and over. Each call will add a new link. Looking back at this, linked list probably wasn't the best way to go.

There are two (obvious) ways that write() could have worked. One is using linked lists as I did. The other way would be to create a byte array that can hold forceWriteLen bytes and just copy into it; however this can be as large as 12MB. There are then two other ways to make this work. The first is just allocating the 12MB right up front. The other way is to start it with maybe just 64k, and make it grow (by powers of two) until it reaches 12MB, however this would end up arraycopying a little under 12MB in total more than the other solution. I will implement one of these for the patch.


FakeOutputStream LinkedList:
This linked list has a more reasonable use. Its purpose is to hold extra bytes just in case the input stream gives too many. I am fairly confident that at most 272 bytes (maximum number of fast bytes - 1) can be written to it. The reason I used a linked list, however, is that I couldn't formally prove this after going through code. I wanted to be safe and just in case their code doesn't behave as it should, everything will work on the OutputStream end.


Code(..., len)
I think I remember figuring out that Code(...) will return at least (but possibly more than) len bytes with the one exception that when the end of the stream is reached it will only read up to the end of the stream. I will modify the decompressor to no longer assume this and use the actual number of bytes read instead.


Fixed the inStream.read() bug (and will be in patch I upload). Added a while loop to read until EOF is reached so the assumptions are true.


Tail call recursive methods -> while loop. Java should add tail-call optimizations when methods only call themselves recursively (which would require no changes to the bytecode).


Fixed memory leaks.

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-1-20100722.non-trivial.pseudo-patch, HADOOP-6837-lzma-1-20100722.patch, HADOOP-6837-lzma-c-20100719.patch, HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9977-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 03 01:48:41 2010
Return-Path: <common-issues-return-9977-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39035 invoked from network); 3 Aug 2010 01:48:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Aug 2010 01:48:41 -0000
Received: (qmail 65472 invoked by uid 500); 3 Aug 2010 01:48:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65409 invoked by uid 500); 3 Aug 2010 01:48:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65401 invoked by uid 99); 3 Aug 2010 01:48:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Aug 2010 01:48:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Aug 2010 01:48:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o731mJ1B008479
	for <common-issues@hadoop.apache.org>; Tue, 3 Aug 2010 01:48:19 GMT
Message-ID: <25165849.131181280800099659.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 21:48:19 -0400 (EDT)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6894) Common foundation for Hadoop client
 tools
In-Reply-To: <10134473.113661280733856346.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894779#action_12894779 ] 

Alejandro Abdelnur commented on HADOOP-6894:
--------------------------------------------

Thanks Owen,

I have not ignored HADOOP-6255, the proposal integrates some of the ideas discussed there (there is still no consensus on some of them in HADOOP-6255):

* A common directory layout
* Support for native components
* TAR files as initial distro from where native packages are created

Regarding your other comments:

1. Yes, we need clear story for packaging.
2. Tarballs on its own are not sufficient. IMO tarballs with metadata that allows the creation of native packages are. If you are using tarballs directly, you have to resolve dependencies by hand, if you are using native packages things are done for you. A packager project can use the tarballs+metadata to create native packages.
3. Cluster deployment is a complete different beast and that is why it is ignored by this proposal, this proposal focuses on a solution for users (the client side of things).
4. This Jira only recommends (it does not impose) a consistent versioning for client tools that would make things simpler for the users. 


> Common foundation for Hadoop client tools
> -----------------------------------------
>
>                 Key: HADOOP-6894
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6894
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Alejandro Abdelnur
>         Attachments: HadoopClientTools.pdf
>
>
> As Hadoop widespreads and matures the number of tools and utilities for users keeps growing.
> Some of them are bundled with Hadoop core, some with Hadoop contrib, some on their own, some are full fledged servers on their own. For example, just to name a few: distcp, streaming, pipes, har, pig, hive, oozie.
> Today there is no standard mechanism for making these tools available to users. Neither there is a standard mechanism for these tools to integrate and distributed them with each other.
> The lack of a common foundation creates issues for developers and users.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9978-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 03 22:32:44 2010
Return-Path: <common-issues-return-9978-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37553 invoked from network); 3 Aug 2010 22:32:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Aug 2010 22:32:43 -0000
Received: (qmail 54572 invoked by uid 500); 3 Aug 2010 22:32:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54481 invoked by uid 500); 3 Aug 2010 22:32:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54473 invoked by uid 99); 3 Aug 2010 22:32:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Aug 2010 22:32:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Aug 2010 22:32:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o73MWI0O020458
	for <common-issues@hadoop.apache.org>; Tue, 3 Aug 2010 22:32:18 GMT
Message-ID: <10866986.146901280874738314.JavaMail.jira@thor>
Date: Tue, 3 Aug 2010 18:32:18 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6889) Make RPC to have an option to timeout
In-Reply-To: <30758566.75561280434516176.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6889:
----------------------------------

    Attachment: ipcTimeout.patch

This patch passes a rpcTimout parameter to RPC#getProxy method. A a non-positive rpcTimeout means that RPC does not timeout as the default behavior. If rpcTimeout is positive, a RPC client throws SocketTimeoutException if the client has not received a response in rpcTimeout period.

> Make RPC to have an option to timeout
> -------------------------------------
>
>                 Key: HADOOP-6889
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6889
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>         Attachments: ipcTimeout.patch
>
>
> Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.
> But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica's length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.
> I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9979-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 03 22:38:38 2010
Return-Path: <common-issues-return-9979-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39222 invoked from network); 3 Aug 2010 22:38:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Aug 2010 22:38:38 -0000
Received: (qmail 57193 invoked by uid 500); 3 Aug 2010 22:38:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57158 invoked by uid 500); 3 Aug 2010 22:38:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57144 invoked by uid 99); 3 Aug 2010 22:38:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Aug 2010 22:38:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Aug 2010 22:38:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o73McGKw020490
	for <common-issues@hadoop.apache.org>; Tue, 3 Aug 2010 22:38:16 GMT
Message-ID: <33260487.146971280875096374.JavaMail.jira@thor>
Date: Tue, 3 Aug 2010 18:38:16 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6897) FileSystem#mkdirs(FileSystem, Path,
 FsPermission) should not call setPermission if mkdirs failled
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

FileSystem#mkdirs(FileSystem, Path, FsPermission) should not call setPermission if mkdirs failled
-------------------------------------------------------------------------------------------------

                 Key: HADOOP-6897
                 URL: https://issues.apache.org/jira/browse/HADOOP-6897
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.22.0
            Reporter: Hairong Kuang
            Assignee: Hairong Kuang
             Fix For: 0.22.0


Here is the piece of code that has the bug. fs.setPermission should not be called if result is false.
{code}
  public static boolean mkdirs(FileSystem fs, Path dir, FsPermission permission)
  throws IOException {
    // create the directory using the default permission
    boolean result = fs.mkdirs(dir);
    // set its permission to be the supplied one
    fs.setPermission(dir, permission);
    return result;
  }
{code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9980-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 03 22:41:43 2010
Return-Path: <common-issues-return-9980-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40519 invoked from network); 3 Aug 2010 22:41:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Aug 2010 22:41:42 -0000
Received: (qmail 62690 invoked by uid 500); 3 Aug 2010 22:41:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62249 invoked by uid 500); 3 Aug 2010 22:41:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62164 invoked by uid 99); 3 Aug 2010 22:41:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Aug 2010 22:41:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Aug 2010 22:41:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o73MfGE0020514
	for <common-issues@hadoop.apache.org>; Tue, 3 Aug 2010 22:41:17 GMT
Message-ID: <18078536.147021280875276768.JavaMail.jira@thor>
Date: Tue, 3 Aug 2010 18:41:16 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6897) FileSystem#mkdirs(FileSystem, Path,
 FsPermission) should not call setPermission if mkdirs failled
In-Reply-To: <33260487.146971280875096374.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6897:
----------------------------------

    Attachment: mkdirs.patch

The fix is trivial. So no unit test is included.

> FileSystem#mkdirs(FileSystem, Path, FsPermission) should not call setPermission if mkdirs failled
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6897
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6897
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: mkdirs.patch
>
>
> Here is the piece of code that has the bug. fs.setPermission should not be called if result is false.
> {code}
>   public static boolean mkdirs(FileSystem fs, Path dir, FsPermission permission)
>   throws IOException {
>     // create the directory using the default permission
>     boolean result = fs.mkdirs(dir);
>     // set its permission to be the supplied one
>     fs.setPermission(dir, permission);
>     return result;
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9981-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 03 22:45:41 2010
Return-Path: <common-issues-return-9981-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42674 invoked from network); 3 Aug 2010 22:45:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Aug 2010 22:45:41 -0000
Received: (qmail 70014 invoked by uid 500); 3 Aug 2010 22:45:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69969 invoked by uid 500); 3 Aug 2010 22:45:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69961 invoked by uid 99); 3 Aug 2010 22:45:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Aug 2010 22:45:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Aug 2010 22:45:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o73MjGmj020574
	for <common-issues@hadoop.apache.org>; Tue, 3 Aug 2010 22:45:16 GMT
Message-ID: <8015301.147131280875516192.JavaMail.jira@thor>
Date: Tue, 3 Aug 2010 18:45:16 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6897) FileSystem#mkdirs(FileSystem, Path,
 FsPermission) should not call setPermission if mkdirs failled
In-Reply-To: <33260487.146971280875096374.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6897:
----------------------------------

    Status: Patch Available  (was: Open)

> FileSystem#mkdirs(FileSystem, Path, FsPermission) should not call setPermission if mkdirs failled
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6897
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6897
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: mkdirs.patch
>
>
> Here is the piece of code that has the bug. fs.setPermission should not be called if result is false.
> {code}
>   public static boolean mkdirs(FileSystem fs, Path dir, FsPermission permission)
>   throws IOException {
>     // create the directory using the default permission
>     boolean result = fs.mkdirs(dir);
>     // set its permission to be the supplied one
>     fs.setPermission(dir, permission);
>     return result;
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9982-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 03 23:12:39 2010
Return-Path: <common-issues-return-9982-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56395 invoked from network); 3 Aug 2010 23:12:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Aug 2010 23:12:39 -0000
Received: (qmail 177 invoked by uid 500); 3 Aug 2010 23:12:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99985 invoked by uid 500); 3 Aug 2010 23:12:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99977 invoked by uid 99); 3 Aug 2010 23:12:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Aug 2010 23:12:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Aug 2010 23:12:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o73NCHx7020862
	for <common-issues@hadoop.apache.org>; Tue, 3 Aug 2010 23:12:17 GMT
Message-ID: <9245835.147741280877137229.JavaMail.jira@thor>
Date: Tue, 3 Aug 2010 19:12:17 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6897) FileSystem#mkdirs(FileSystem, Path,
 FsPermission) should not call setPermission if mkdirs failled
In-Reply-To: <33260487.146971280875096374.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895078#action_12895078 ] 

Hadoop QA commented on HADOOP-6897:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12451172/mkdirs.patch
  against trunk revision 981714.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/661/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/661/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/661/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/661/console

This message is automatically generated.

> FileSystem#mkdirs(FileSystem, Path, FsPermission) should not call setPermission if mkdirs failled
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6897
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6897
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: mkdirs.patch
>
>
> Here is the piece of code that has the bug. fs.setPermission should not be called if result is false.
> {code}
>   public static boolean mkdirs(FileSystem fs, Path dir, FsPermission permission)
>   throws IOException {
>     // create the directory using the default permission
>     boolean result = fs.mkdirs(dir);
>     // set its permission to be the supplied one
>     fs.setPermission(dir, permission);
>     return result;
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9983-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 03 23:26:42 2010
Return-Path: <common-issues-return-9983-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66017 invoked from network); 3 Aug 2010 23:26:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Aug 2010 23:26:41 -0000
Received: (qmail 22944 invoked by uid 500); 3 Aug 2010 23:26:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22850 invoked by uid 500); 3 Aug 2010 23:26:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22834 invoked by uid 99); 3 Aug 2010 23:26:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Aug 2010 23:26:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Aug 2010 23:26:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o73NQGWp020996
	for <common-issues@hadoop.apache.org>; Tue, 3 Aug 2010 23:26:16 GMT
Message-ID: <25894414.148031280877976739.JavaMail.jira@thor>
Date: Tue, 3 Aug 2010 19:26:16 -0400 (EDT)
From: "Dick King (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6897) FileSystem#mkdirs(FileSystem, Path,
 FsPermission) should not call setPermission if mkdirs failled
In-Reply-To: <33260487.146971280875096374.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895084#action_12895084 ] 

Dick King commented on HADOOP-6897:
-----------------------------------

There's a wierd incompatibility within the various overloads of {{FileSystem.mkdirs}} .

{{static boolean FileSystem.mkdirs(FileSystem fs, Path path, FsPermission permission)}} assigns the specified permission to all directories constructed on the way to path , including the last one, {{path}} itself .

{{fs.mkdirs(path, permission)}} behaves very differently.  The intermediate directories, if any, that get created, are born with the permissions of {{permission}} , but the leaf directory, {{path}} , seems to get its permissions set to a MASKED permission mask, at least when {{FileSystem}} is a DFS.

At the very least, this behavior should be documented.  A better remedy, in my opinion, would be for {{FileSystem.mkdirs(Path, FsPermission)}} to change behavior so all directories created have permission bits named by the {{FsPermission}} parameter in all of the calls that have one.

> FileSystem#mkdirs(FileSystem, Path, FsPermission) should not call setPermission if mkdirs failled
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6897
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6897
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: mkdirs.patch
>
>
> Here is the piece of code that has the bug. fs.setPermission should not be called if result is false.
> {code}
>   public static boolean mkdirs(FileSystem fs, Path dir, FsPermission permission)
>   throws IOException {
>     // create the directory using the default permission
>     boolean result = fs.mkdirs(dir);
>     // set its permission to be the supplied one
>     fs.setPermission(dir, permission);
>     return result;
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9984-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 04 00:32:41 2010
Return-Path: <common-issues-return-9984-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90812 invoked from network); 4 Aug 2010 00:32:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Aug 2010 00:32:40 -0000
Received: (qmail 367 invoked by uid 500); 4 Aug 2010 00:32:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 248 invoked by uid 500); 4 Aug 2010 00:32:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 237 invoked by uid 99); 4 Aug 2010 00:32:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 00:32:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 00:32:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o740WIO5021661
	for <common-issues@hadoop.apache.org>; Wed, 4 Aug 2010 00:32:18 GMT
Message-ID: <3130073.149351280881938702.JavaMail.jira@thor>
Date: Tue, 3 Aug 2010 20:32:18 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6889) Make RPC to have an option to timeout
In-Reply-To: <30758566.75561280434516176.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6889:
----------------------------------

    Attachment: ipcTimeout1.patch

ipcTimeout1.patch additionally makes waitForProxy has rpcTimeout as a parameter.

> Make RPC to have an option to timeout
> -------------------------------------
>
>                 Key: HADOOP-6889
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6889
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>         Attachments: ipcTimeout.patch, ipcTimeout1.patch
>
>
> Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.
> But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica's length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.
> I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9985-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 04 00:52:38 2010
Return-Path: <common-issues-return-9985-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95978 invoked from network); 4 Aug 2010 00:52:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Aug 2010 00:52:38 -0000
Received: (qmail 21773 invoked by uid 500); 4 Aug 2010 00:52:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21492 invoked by uid 500); 4 Aug 2010 00:52:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21484 invoked by uid 99); 4 Aug 2010 00:52:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 00:52:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 00:52:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o740qHH5021832
	for <common-issues@hadoop.apache.org>; Wed, 4 Aug 2010 00:52:17 GMT
Message-ID: <12444083.149691280883137011.JavaMail.jira@thor>
Date: Tue, 3 Aug 2010 20:52:17 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6898) FileSystem.copyToLocal creates files
 with 777 permissions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

FileSystem.copyToLocal creates files with 777 permissions
---------------------------------------------------------

                 Key: HADOOP-6898
                 URL: https://issues.apache.org/jira/browse/HADOOP-6898
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs, security
            Reporter: Todd Lipcon
            Priority: Critical
             Fix For: 0.22.0


FileSystem.copyToLocal ends up calling through to FileUtil.copy, which calls create() on the target file system without passing any permission object. Therefore, the file ends up getting created locally with 777 permissions, which is dangerous -- even if the caller then fixes up permissions afterwards, it exposes a window in which an attacker can open the file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9986-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 04 03:49:44 2010
Return-Path: <common-issues-return-9986-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69748 invoked from network); 4 Aug 2010 03:49:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Aug 2010 03:49:43 -0000
Received: (qmail 34383 invoked by uid 500); 4 Aug 2010 03:49:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34294 invoked by uid 500); 4 Aug 2010 03:49:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34281 invoked by uid 99); 4 Aug 2010 03:49:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 03:49:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 03:49:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o743nGk8023014
	for <common-issues@hadoop.apache.org>; Wed, 4 Aug 2010 03:49:16 GMT
Message-ID: <14900551.151031280893756511.JavaMail.jira@thor>
Date: Tue, 3 Aug 2010 23:49:16 -0400 (EDT)
From: "Ranjit Mathew (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6898) FileSystem.copyToLocal creates
 files with 777 permissions
In-Reply-To: <12444083.149691280883137011.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895130#action_12895130 ] 

Ranjit Mathew commented on HADOOP-6898:
---------------------------------------

Is there no umask set for the corresponding user?

(Even if the default permissions are 777, the umask setting should have stripped some of them out upon file-creation.)

> FileSystem.copyToLocal creates files with 777 permissions
> ---------------------------------------------------------
>
>                 Key: HADOOP-6898
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6898
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, security
>            Reporter: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>
> FileSystem.copyToLocal ends up calling through to FileUtil.copy, which calls create() on the target file system without passing any permission object. Therefore, the file ends up getting created locally with 777 permissions, which is dangerous -- even if the caller then fixes up permissions afterwards, it exposes a window in which an attacker can open the file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9987-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 04 04:03:41 2010
Return-Path: <common-issues-return-9987-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75139 invoked from network); 4 Aug 2010 04:03:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Aug 2010 04:03:41 -0000
Received: (qmail 40674 invoked by uid 500); 4 Aug 2010 04:03:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40534 invoked by uid 500); 4 Aug 2010 04:03:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40526 invoked by uid 99); 4 Aug 2010 04:03:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 04:03:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 04:03:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7443Gw5023326
	for <common-issues@hadoop.apache.org>; Wed, 4 Aug 2010 04:03:16 GMT
Message-ID: <25977285.151071280894596424.JavaMail.jira@thor>
Date: Wed, 4 Aug 2010 00:03:16 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6898) FileSystem.copyToLocal creates
 files with 777 permissions
In-Reply-To: <12444083.149691280883137011.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895131#action_12895131 ] 

Todd Lipcon commented on HADOOP-6898:
-------------------------------------

The umask is set, but create() actually specifically chmods the file to 777 after it's created.

> FileSystem.copyToLocal creates files with 777 permissions
> ---------------------------------------------------------
>
>                 Key: HADOOP-6898
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6898
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, security
>            Reporter: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.22.0
>
>
> FileSystem.copyToLocal ends up calling through to FileUtil.copy, which calls create() on the target file system without passing any permission object. Therefore, the file ends up getting created locally with 777 permissions, which is dangerous -- even if the caller then fixes up permissions afterwards, it exposes a window in which an attacker can open the file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9988-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 04 10:32:45 2010
Return-Path: <common-issues-return-9988-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37064 invoked from network); 4 Aug 2010 10:32:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Aug 2010 10:32:45 -0000
Received: (qmail 61088 invoked by uid 500); 4 Aug 2010 10:32:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60946 invoked by uid 500); 4 Aug 2010 10:32:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60938 invoked by uid 99); 4 Aug 2010 10:32:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 10:32:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 10:32:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o74AWIEp027084
	for <common-issues@hadoop.apache.org>; Wed, 4 Aug 2010 10:32:18 GMT
Message-ID: <31713408.155791280917938877.JavaMail.jira@thor>
Date: Wed, 4 Aug 2010 06:32:18 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6862) Add api to add user/group to
 AccessControlList
In-Reply-To: <6885195.391831279173830267.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895211#action_12895211 ] 

Vinod K V commented on HADOOP-6862:
-----------------------------------

The nice little patch looks fine except for the point that adding or removing the wild card character either as part of users list of the groups doesn't change the ACL at all.

You can also separate the newly added cases into a new test :)

> Add api to add user/group to AccessControlList
> ----------------------------------------------
>
>                 Key: HADOOP-6862
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6862
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.22.0
>
>         Attachments: patch-6862-1.txt, patch-6862.txt
>
>
> Add api addUser(String user) and addGroup(String group) to add user/group to AccessControlList

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9989-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 04 10:50:43 2010
Return-Path: <common-issues-return-9989-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43406 invoked from network); 4 Aug 2010 10:50:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Aug 2010 10:50:43 -0000
Received: (qmail 80984 invoked by uid 500); 4 Aug 2010 10:50:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80524 invoked by uid 500); 4 Aug 2010 10:50:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80512 invoked by uid 99); 4 Aug 2010 10:50:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 10:50:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 10:50:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o74AoGj5027186
	for <common-issues@hadoop.apache.org>; Wed, 4 Aug 2010 10:50:16 GMT
Message-ID: <33071816.155921280919016495.JavaMail.jira@thor>
Date: Wed, 4 Aug 2010 06:50:16 -0400 (EDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6862) Add api to add user/group to
 AccessControlList
In-Reply-To: <6885195.391831279173830267.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amareshwari Sriramadasu updated HADOOP-6862:
--------------------------------------------

    Status: Open  (was: Patch Available)

> Add api to add user/group to AccessControlList
> ----------------------------------------------
>
>                 Key: HADOOP-6862
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6862
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.22.0
>
>         Attachments: patch-6862-1.txt, patch-6862.txt
>
>
> Add api addUser(String user) and addGroup(String group) to add user/group to AccessControlList

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9990-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 04 11:01:44 2010
Return-Path: <common-issues-return-9990-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45344 invoked from network); 4 Aug 2010 11:01:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Aug 2010 11:01:44 -0000
Received: (qmail 96499 invoked by uid 500); 4 Aug 2010 11:01:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96356 invoked by uid 500); 4 Aug 2010 11:01:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96075 invoked by uid 99); 4 Aug 2010 11:01:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 11:01:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 11:01:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o74B1HIu027311
	for <common-issues@hadoop.apache.org>; Wed, 4 Aug 2010 11:01:17 GMT
Message-ID: <1591517.155981280919677174.JavaMail.jira@thor>
Date: Wed, 4 Aug 2010 07:01:17 -0400 (EDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6862) Add api to add user/group to
 AccessControlList
In-Reply-To: <6885195.391831279173830267.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895217#action_12895217 ] 

Amareshwari Sriramadasu commented on HADOOP-6862:
-------------------------------------------------

I think adding/removing a wild card character through this api should not be allowed, because adding a '*' (the only allowed wild card character in ACL) to an already created ACL does not make sense. Also, It makes sense to accept it only in the constructor as it is now, which defines the behavior to be simple. For example, adding it twice by addUser('*") and removing it by removeUser("*") once would result in allowing all the users.

I will submit a patch which throws an exception saying "wild card not allowed" if a wild card character is passed.

Thoughts?

> Add api to add user/group to AccessControlList
> ----------------------------------------------
>
>                 Key: HADOOP-6862
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6862
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.22.0
>
>         Attachments: patch-6862-1.txt, patch-6862.txt
>
>
> Add api addUser(String user) and addGroup(String group) to add user/group to AccessControlList

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9991-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 04 15:50:39 2010
Return-Path: <common-issues-return-9991-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67779 invoked from network); 4 Aug 2010 15:50:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Aug 2010 15:50:39 -0000
Received: (qmail 94456 invoked by uid 500); 4 Aug 2010 15:50:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94385 invoked by uid 500); 4 Aug 2010 15:50:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94377 invoked by uid 99); 4 Aug 2010 15:50:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 15:50:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 15:50:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o74FoGg9029985
	for <common-issues@hadoop.apache.org>; Wed, 4 Aug 2010 15:50:17 GMT
Message-ID: <23638324.159791280937016648.JavaMail.jira@thor>
Date: Wed, 4 Aug 2010 11:50:16 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6862) Add api to add user/group to
 AccessControlList
In-Reply-To: <6885195.391831279173830267.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895299#action_12895299 ] 

Vinod K V commented on HADOOP-6862:
-----------------------------------

bq. I will submit a patch which throws an exception saying "wild card not allowed" if a wild card character is passed.
+1. As an aside, all the new methods should return immediately if allAllowed is already true, no point in maintaining any more user/group names.

> Add api to add user/group to AccessControlList
> ----------------------------------------------
>
>                 Key: HADOOP-6862
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6862
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.22.0
>
>         Attachments: patch-6862-1.txt, patch-6862.txt
>
>
> Add api addUser(String user) and addGroup(String group) to add user/group to AccessControlList

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9992-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 04 16:42:41 2010
Return-Path: <common-issues-return-9992-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95131 invoked from network); 4 Aug 2010 16:42:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Aug 2010 16:42:40 -0000
Received: (qmail 86904 invoked by uid 500); 4 Aug 2010 16:42:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86762 invoked by uid 500); 4 Aug 2010 16:42:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86747 invoked by uid 99); 4 Aug 2010 16:42:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 16:42:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 16:42:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o74GgJFX000770
	for <common-issues@hadoop.apache.org>; Wed, 4 Aug 2010 16:42:19 GMT
Message-ID: <32772950.160281280940139284.JavaMail.jira@thor>
Date: Wed, 4 Aug 2010 12:42:19 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6897) FileSystem#mkdirs(FileSystem, Path,
 FsPermission) should not call setPermission if mkdirs failled
In-Reply-To: <33260487.146971280875096374.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895311#action_12895311 ] 

Hairong Kuang commented on HADOOP-6897:
---------------------------------------

Dick, you made a good observation. But my understanding of their semantics is a little different from yours.

I believe static boolean FileSystem.mkdirs(FileSystem fs, Path path, FsPermission permission) assigns masked permission to intermediate directories, if any, that get created, and assigns the given permission to the leaf directory.

On the other hand fs.mkdirs(path, permission) assign the masked permission to all newly created directories.

Another subtle difference is that if the directory already exists, static boolean FileSystem mkdirs would go ahead and change the directory's permission. But boolean fs.mkdirs would simply return true without changing its permission.

> FileSystem#mkdirs(FileSystem, Path, FsPermission) should not call setPermission if mkdirs failled
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6897
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6897
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: mkdirs.patch
>
>
> Here is the piece of code that has the bug. fs.setPermission should not be called if result is false.
> {code}
>   public static boolean mkdirs(FileSystem fs, Path dir, FsPermission permission)
>   throws IOException {
>     // create the directory using the default permission
>     boolean result = fs.mkdirs(dir);
>     // set its permission to be the supplied one
>     fs.setPermission(dir, permission);
>     return result;
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9993-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 04 16:48:38 2010
Return-Path: <common-issues-return-9993-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97905 invoked from network); 4 Aug 2010 16:48:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Aug 2010 16:48:38 -0000
Received: (qmail 7657 invoked by uid 500); 4 Aug 2010 16:48:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7600 invoked by uid 500); 4 Aug 2010 16:48:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7592 invoked by uid 99); 4 Aug 2010 16:48:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 16:48:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 16:48:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o74GmHQp001040
	for <common-issues@hadoop.apache.org>; Wed, 4 Aug 2010 16:48:17 GMT
Message-ID: <23975859.160741280940497060.JavaMail.jira@thor>
Date: Wed, 4 Aug 2010 12:48:17 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6897) FileSystem#mkdirs(FileSystem, Path,
 FsPermission) should not call setPermission if mkdirs failled
In-Reply-To: <33260487.146971280875096374.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895316#action_12895316 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6897:
------------------------------------------------

Hi Hairong, could you also clarify javadoc for the expected behavior of public static boolean mkdirs(FileSystem fs, Path dir, FsPermission permission)?

> FileSystem#mkdirs(FileSystem, Path, FsPermission) should not call setPermission if mkdirs failled
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6897
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6897
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: mkdirs.patch
>
>
> Here is the piece of code that has the bug. fs.setPermission should not be called if result is false.
> {code}
>   public static boolean mkdirs(FileSystem fs, Path dir, FsPermission permission)
>   throws IOException {
>     // create the directory using the default permission
>     boolean result = fs.mkdirs(dir);
>     // set its permission to be the supplied one
>     fs.setPermission(dir, permission);
>     return result;
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9994-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 04 19:03:45 2010
Return-Path: <common-issues-return-9994-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59054 invoked from network); 4 Aug 2010 19:03:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Aug 2010 19:03:45 -0000
Received: (qmail 2993 invoked by uid 500); 4 Aug 2010 19:03:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2802 invoked by uid 500); 4 Aug 2010 19:03:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2793 invoked by uid 99); 4 Aug 2010 19:03:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 19:03:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 19:03:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o74J3Kgq008612
	for <common-issues@hadoop.apache.org>; Wed, 4 Aug 2010 19:03:21 GMT
Message-ID: <1278284.163441280948600820.JavaMail.jira@thor>
Date: Wed, 4 Aug 2010 15:03:20 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6889) Make RPC to have an option to
 timeout
In-Reply-To: <30758566.75561280434516176.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895368#action_12895368 ] 

Doug Cutting commented on HADOOP-6889:
--------------------------------------

+1 this patch looks reasonable to me, pending Hudson's scrutiny.

> Make RPC to have an option to timeout
> -------------------------------------
>
>                 Key: HADOOP-6889
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6889
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>         Attachments: ipcTimeout.patch, ipcTimeout1.patch
>
>
> Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.
> But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica's length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.
> I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9995-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 04 19:55:53 2010
Return-Path: <common-issues-return-9995-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82638 invoked from network); 4 Aug 2010 19:55:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Aug 2010 19:55:52 -0000
Received: (qmail 71068 invoked by uid 500); 4 Aug 2010 19:55:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71013 invoked by uid 500); 4 Aug 2010 19:55:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71005 invoked by uid 99); 4 Aug 2010 19:55:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 19:55:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 19:55:49 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o74JtSRX018795
	for <common-issues@hadoop.apache.org>; Wed, 4 Aug 2010 19:55:28 GMT
Message-ID: <5529002.164701280951728110.JavaMail.jira@thor>
Date: Wed, 4 Aug 2010 15:55:28 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6889) Make RPC to have an option to timeout
In-Reply-To: <30758566.75561280434516176.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6889:
----------------------------------

          Status: Patch Available  (was: Open)
    Hadoop Flags: [Reviewed]

> Make RPC to have an option to timeout
> -------------------------------------
>
>                 Key: HADOOP-6889
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6889
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>         Attachments: ipcTimeout.patch, ipcTimeout1.patch
>
>
> Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.
> But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica's length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.
> I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9996-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 04 20:24:43 2010
Return-Path: <common-issues-return-9996-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99724 invoked from network); 4 Aug 2010 20:24:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Aug 2010 20:24:43 -0000
Received: (qmail 14587 invoked by uid 500); 4 Aug 2010 20:24:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14532 invoked by uid 500); 4 Aug 2010 20:24:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14524 invoked by uid 99); 4 Aug 2010 20:24:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 20:24:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 20:24:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o74KOIhS020240
	for <common-issues@hadoop.apache.org>; Wed, 4 Aug 2010 20:24:18 GMT
Message-ID: <8873006.165981280953457990.JavaMail.jira@thor>
Date: Wed, 4 Aug 2010 16:24:17 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6889) Make RPC to have an option to
 timeout
In-Reply-To: <30758566.75561280434516176.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895403#action_12895403 ] 

Hadoop QA commented on HADOOP-6889:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12451180/ipcTimeout1.patch
  against trunk revision 981714.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/662/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/662/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/662/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/662/console

This message is automatically generated.

> Make RPC to have an option to timeout
> -------------------------------------
>
>                 Key: HADOOP-6889
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6889
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>         Attachments: ipcTimeout.patch, ipcTimeout1.patch
>
>
> Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.
> But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica's length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.
> I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9997-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 04 22:51:45 2010
Return-Path: <common-issues-return-9997-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32462 invoked from network); 4 Aug 2010 22:51:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Aug 2010 22:51:45 -0000
Received: (qmail 7448 invoked by uid 500); 4 Aug 2010 22:51:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7384 invoked by uid 500); 4 Aug 2010 22:51:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7368 invoked by uid 99); 4 Aug 2010 22:51:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 22:51:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 22:51:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o74MpKYT020424
	for <common-issues@hadoop.apache.org>; Wed, 4 Aug 2010 22:51:20 GMT
Message-ID: <2326417.169371280962280722.JavaMail.jira@thor>
Date: Wed, 4 Aug 2010 18:51:20 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6889) Make RPC to have an option to timeout
In-Reply-To: <30758566.75561280434516176.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6889:
----------------------------------

    Attachment: ipcTimeout2.patch

This fixed the failed tests.

> Make RPC to have an option to timeout
> -------------------------------------
>
>                 Key: HADOOP-6889
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6889
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>         Attachments: ipcTimeout.patch, ipcTimeout1.patch, ipcTimeout2.patch
>
>
> Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.
> But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica's length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.
> I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9998-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 04 22:51:46 2010
Return-Path: <common-issues-return-9998-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32510 invoked from network); 4 Aug 2010 22:51:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Aug 2010 22:51:45 -0000
Received: (qmail 7578 invoked by uid 500); 4 Aug 2010 22:51:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7392 invoked by uid 500); 4 Aug 2010 22:51:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7371 invoked by uid 99); 4 Aug 2010 22:51:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 22:51:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 22:51:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o74MpNN8020480
	for <common-issues@hadoop.apache.org>; Wed, 4 Aug 2010 22:51:23 GMT
Message-ID: <10363869.169551280962283805.JavaMail.jira@thor>
Date: Wed, 4 Aug 2010 18:51:23 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6889) Make RPC to have an option to timeout
In-Reply-To: <30758566.75561280434516176.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6889:
----------------------------------

    Status: Patch Available  (was: Open)

> Make RPC to have an option to timeout
> -------------------------------------
>
>                 Key: HADOOP-6889
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6889
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>         Attachments: ipcTimeout.patch, ipcTimeout1.patch, ipcTimeout2.patch
>
>
> Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.
> But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica's length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.
> I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9999-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 04 22:51:46 2010
Return-Path: <common-issues-return-9999-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32523 invoked from network); 4 Aug 2010 22:51:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Aug 2010 22:51:45 -0000
Received: (qmail 7743 invoked by uid 500); 4 Aug 2010 22:51:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7706 invoked by uid 500); 4 Aug 2010 22:51:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7668 invoked by uid 99); 4 Aug 2010 22:51:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 22:51:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 22:51:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o74MpM1v020452
	for <common-issues@hadoop.apache.org>; Wed, 4 Aug 2010 22:51:22 GMT
Message-ID: <6479171.169461280962282078.JavaMail.jira@thor>
Date: Wed, 4 Aug 2010 18:51:22 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6889) Make RPC to have an option to timeout
In-Reply-To: <30758566.75561280434516176.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6889:
----------------------------------

    Status: Open  (was: Patch Available)

> Make RPC to have an option to timeout
> -------------------------------------
>
>                 Key: HADOOP-6889
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6889
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>         Attachments: ipcTimeout.patch, ipcTimeout1.patch, ipcTimeout2.patch
>
>
> Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.
> But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica's length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.
> I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10000-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 04 23:29:38 2010
Return-Path: <common-issues-return-10000-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46745 invoked from network); 4 Aug 2010 23:29:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Aug 2010 23:29:38 -0000
Received: (qmail 29996 invoked by uid 500); 4 Aug 2010 23:29:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29905 invoked by uid 500); 4 Aug 2010 23:29:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29896 invoked by uid 99); 4 Aug 2010 23:29:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 23:29:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 23:29:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o74NTGEs020715
	for <common-issues@hadoop.apache.org>; Wed, 4 Aug 2010 23:29:16 GMT
Message-ID: <23450587.169791280964556270.JavaMail.jira@thor>
Date: Wed, 4 Aug 2010 19:29:16 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6899) FileSystem#setWorkingDir() does not
 work for relative names
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

FileSystem#setWorkingDir() does not work for relative names
-----------------------------------------------------------

                 Key: HADOOP-6899
                 URL: https://issues.apache.org/jira/browse/HADOOP-6899
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.20.2
            Reporter: Sanjay Radia
            Assignee: Sanjay Radia
             Fix For: 0.22.0


FileSystem#setWorkingDir() does not work for relative names

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10001-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 04 23:31:38 2010
Return-Path: <common-issues-return-10001-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47196 invoked from network); 4 Aug 2010 23:31:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Aug 2010 23:31:38 -0000
Received: (qmail 31311 invoked by uid 500); 4 Aug 2010 23:31:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31281 invoked by uid 500); 4 Aug 2010 23:31:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31273 invoked by uid 99); 4 Aug 2010 23:31:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 23:31:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 23:31:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o74NVGxT020724
	for <common-issues@hadoop.apache.org>; Wed, 4 Aug 2010 23:31:17 GMT
Message-ID: <29253908.169821280964676744.JavaMail.jira@thor>
Date: Wed, 4 Aug 2010 19:31:16 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6900) FileSystem#listLocatedStatus API
 needs changes to indicate complete set of error conditions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

FileSystem#listLocatedStatus API needs changes to indicate complete set of error conditions
-------------------------------------------------------------------------------------------

                 Key: HADOOP-6900
                 URL: https://issues.apache.org/jira/browse/HADOOP-6900
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Suresh Srinivas


HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Through the iterator application does not get all the error conditions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10002-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 04 23:36:38 2010
Return-Path: <common-issues-return-10002-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48830 invoked from network); 4 Aug 2010 23:36:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Aug 2010 23:36:38 -0000
Received: (qmail 33570 invoked by uid 500); 4 Aug 2010 23:36:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33539 invoked by uid 500); 4 Aug 2010 23:36:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33530 invoked by uid 99); 4 Aug 2010 23:36:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 23:36:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 23:36:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o74NaHDY020750
	for <common-issues@hadoop.apache.org>; Wed, 4 Aug 2010 23:36:17 GMT
Message-ID: <17317741.169871280964977768.JavaMail.jira@thor>
Date: Wed, 4 Aug 2010 19:36:17 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6899) FileSystem#setWorkingDir() does not
 work for relative names
In-Reply-To: <23450587.169791280964556270.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895467#action_12895467 ] 

Sanjay Radia commented on HADOOP-6899:
--------------------------------------

RawLocalFileSystem#setWorkingDiectory(path) if path is itself a relative path. 
It does not bother to convert it to an absolute path internally. Also, sadly, the existing tests for 
working directory never correctly checked if the setWorkingDirectory worked. 




> FileSystem#setWorkingDir() does not work for relative names
> -----------------------------------------------------------
>
>                 Key: HADOOP-6899
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6899
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.22.0
>
>
> FileSystem#setWorkingDir() does not work for relative names

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10003-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 04 23:36:39 2010
Return-Path: <common-issues-return-10003-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48858 invoked from network); 4 Aug 2010 23:36:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Aug 2010 23:36:38 -0000
Received: (qmail 33751 invoked by uid 500); 4 Aug 2010 23:36:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33700 invoked by uid 500); 4 Aug 2010 23:36:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33692 invoked by uid 99); 4 Aug 2010 23:36:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 23:36:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 23:36:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o74NaII3020756
	for <common-issues@hadoop.apache.org>; Wed, 4 Aug 2010 23:36:18 GMT
Message-ID: <18820120.169891280964978030.JavaMail.jira@thor>
Date: Wed, 4 Aug 2010 19:36:18 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6899) RawLocalFileSystem#setWorkingDir()
 does not work for relative names
In-Reply-To: <23450587.169791280964556270.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-6899:
---------------------------------

        Summary: RawLocalFileSystem#setWorkingDir() does not work for relative names  (was: FileSystem#setWorkingDir() does not work for relative names)
    Description: RawLocalFileSystem#setWorkingDir() does not work for relative names  (was: FileSystem#setWorkingDir() does not work for relative names)

> RawLocalFileSystem#setWorkingDir() does not work for relative names
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6899
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6899
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.22.0
>
>
> RawLocalFileSystem#setWorkingDir() does not work for relative names

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10004-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 04 23:40:42 2010
Return-Path: <common-issues-return-10004-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49397 invoked from network); 4 Aug 2010 23:40:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Aug 2010 23:40:42 -0000
Received: (qmail 34855 invoked by uid 500); 4 Aug 2010 23:40:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34823 invoked by uid 500); 4 Aug 2010 23:40:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34815 invoked by uid 99); 4 Aug 2010 23:40:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 23:40:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 23:40:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o74NeI2M020789
	for <common-issues@hadoop.apache.org>; Wed, 4 Aug 2010 23:40:18 GMT
Message-ID: <15255406.169971280965218470.JavaMail.jira@thor>
Date: Wed, 4 Aug 2010 19:40:18 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6899) RawLocalFileSystem#setWorkingDir()
 does not work for relative names
In-Reply-To: <23450587.169791280964556270.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-6899:
---------------------------------

    Attachment: 6899.patch

The attached path fixes the bug. I also backported one of more extensive tests for FileContext to FileSystem.

> RawLocalFileSystem#setWorkingDir() does not work for relative names
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6899
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6899
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.22.0
>
>         Attachments: 6899.patch
>
>
> RawLocalFileSystem#setWorkingDir() does not work for relative names

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10005-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 04 23:40:42 2010
Return-Path: <common-issues-return-10005-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49431 invoked from network); 4 Aug 2010 23:40:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Aug 2010 23:40:42 -0000
Received: (qmail 35034 invoked by uid 500); 4 Aug 2010 23:40:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34992 invoked by uid 500); 4 Aug 2010 23:40:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34983 invoked by uid 99); 4 Aug 2010 23:40:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 23:40:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 23:40:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o74NeIpG020795
	for <common-issues@hadoop.apache.org>; Wed, 4 Aug 2010 23:40:18 GMT
Message-ID: <8836954.169991280965218777.JavaMail.jira@thor>
Date: Wed, 4 Aug 2010 19:40:18 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6900) FileSystem#listLocatedStatus API
 needs changes to indicate complete set of error conditions
In-Reply-To: <29253908.169821280964676744.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895470#action_12895470 ] 

Suresh Srinivas commented on HADOOP-6900:
-----------------------------------------

Iterator#hasNext() has to indicate the following error conditions:
# next element exists - returns true
# next element does not exist - returns false
# could not get status for the next element, because a directory that was available before iteration started does not exist any more.
# could not talk to the remote server to get status.
# remote server throws IOException due to an error while getting the status.

For the last 3 conditions, currently RuntimeException is thrown. Throwing RTE forces application to catch it, to handle the error condition. This results in application catching other RTE as well. Worse still, an application that does not catch RTE might exit.

The problem is, Java Iterator interface is not suitable for implementation that use RPC underneath. We should have a different interface, which could indicate error conditions more clearly.

Additionally I feel when a directory gets deleted in the middle of iteration, we should ignore it, or add provision to the iterator to ignore it.

> FileSystem#listLocatedStatus API needs changes to indicate complete set of error conditions
> -------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6900
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6900
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Suresh Srinivas
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Through the iterator application does not get all the error conditions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10006-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 04 23:42:37 2010
Return-Path: <common-issues-return-10006-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49958 invoked from network); 4 Aug 2010 23:42:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Aug 2010 23:42:37 -0000
Received: (qmail 38761 invoked by uid 500); 4 Aug 2010 23:42:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38728 invoked by uid 500); 4 Aug 2010 23:42:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38720 invoked by uid 99); 4 Aug 2010 23:42:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 23:42:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 23:42:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o74NgGhr020810
	for <common-issues@hadoop.apache.org>; Wed, 4 Aug 2010 23:42:16 GMT
Message-ID: <18610148.170031280965336760.JavaMail.jira@thor>
Date: Wed, 4 Aug 2010 19:42:16 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6899) RawLocalFileSystem#setWorkingDir()
 does not work for relative names
In-Reply-To: <23450587.169791280964556270.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-6899:
---------------------------------

    Status: Patch Available  (was: Open)

> RawLocalFileSystem#setWorkingDir() does not work for relative names
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6899
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6899
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.22.0
>
>         Attachments: 6899.patch
>
>
> RawLocalFileSystem#setWorkingDir() does not work for relative names

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10007-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 05 01:42:38 2010
Return-Path: <common-issues-return-10007-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96521 invoked from network); 5 Aug 2010 01:42:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Aug 2010 01:42:38 -0000
Received: (qmail 12732 invoked by uid 500); 5 Aug 2010 01:42:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12664 invoked by uid 500); 5 Aug 2010 01:42:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12656 invoked by uid 99); 5 Aug 2010 01:42:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 01:42:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 01:42:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o751gGal021812
	for <common-issues@hadoop.apache.org>; Thu, 5 Aug 2010 01:42:16 GMT
Message-ID: <12268119.171801280972536282.JavaMail.jira@thor>
Date: Wed, 4 Aug 2010 21:42:16 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6899) RawLocalFileSystem#setWorkingDir()
 does not work for relative names
In-Reply-To: <23450587.169791280964556270.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-6899:
---------------------------------

    Attachment:     (was: 6899.patch)

> RawLocalFileSystem#setWorkingDir() does not work for relative names
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6899
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6899
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.22.0
>
>         Attachments: 6899.patch
>
>
> RawLocalFileSystem#setWorkingDir() does not work for relative names

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10008-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 05 01:42:42 2010
Return-Path: <common-issues-return-10008-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96571 invoked from network); 5 Aug 2010 01:42:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Aug 2010 01:42:41 -0000
Received: (qmail 12873 invoked by uid 500); 5 Aug 2010 01:42:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12844 invoked by uid 500); 5 Aug 2010 01:42:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12836 invoked by uid 99); 5 Aug 2010 01:42:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 01:42:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 01:42:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o751gGhn021818
	for <common-issues@hadoop.apache.org>; Thu, 5 Aug 2010 01:42:16 GMT
Message-ID: <26291045.171821280972536750.JavaMail.jira@thor>
Date: Wed, 4 Aug 2010 21:42:16 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6899) RawLocalFileSystem#setWorkingDir()
 does not work for relative names
In-Reply-To: <23450587.169791280964556270.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-6899:
---------------------------------

    Attachment: 6899.patch

> RawLocalFileSystem#setWorkingDir() does not work for relative names
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6899
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6899
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.22.0
>
>         Attachments: 6899.patch
>
>
> RawLocalFileSystem#setWorkingDir() does not work for relative names

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10009-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 05 04:24:41 2010
Return-Path: <common-issues-return-10009-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40819 invoked from network); 5 Aug 2010 04:24:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Aug 2010 04:24:40 -0000
Received: (qmail 92387 invoked by uid 500); 5 Aug 2010 04:24:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92252 invoked by uid 500); 5 Aug 2010 04:24:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92244 invoked by uid 99); 5 Aug 2010 04:24:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 04:24:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 04:24:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o754OHCB023383
	for <common-issues@hadoop.apache.org>; Thu, 5 Aug 2010 04:24:17 GMT
Message-ID: <11769232.173441280982257059.JavaMail.jira@thor>
Date: Thu, 5 Aug 2010 00:24:17 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6900) FileSystem#listLocatedStatus should
 not throw generic RuntimeException to indicate error conditions
In-Reply-To: <29253908.169821280964676744.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6900:
------------------------------------

        Summary: FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions  (was: FileSystem#listLocatedStatus API needs changes to indicate complete set of error conditions)
    Description: HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.  (was: HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Through the iterator application does not get all the error conditions.)

> FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6900
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6900
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Suresh Srinivas
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10010-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 05 04:30:45 2010
Return-Path: <common-issues-return-10010-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42939 invoked from network); 5 Aug 2010 04:30:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Aug 2010 04:30:45 -0000
Received: (qmail 93805 invoked by uid 500); 5 Aug 2010 04:30:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93420 invoked by uid 500); 5 Aug 2010 04:30:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93412 invoked by uid 99); 5 Aug 2010 04:30:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 04:30:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 04:30:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o754UITK023445
	for <common-issues@hadoop.apache.org>; Thu, 5 Aug 2010 04:30:18 GMT
Message-ID: <25666717.173521280982618454.JavaMail.jira@thor>
Date: Thu, 5 Aug 2010 00:30:18 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6900) FileSystem#listLocatedStatus should
 not throw generic RuntimeException to indicate error conditions
In-Reply-To: <29253908.169821280964676744.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang reassigned HADOOP-6900:
-------------------------------------

    Assignee: Hairong Kuang

> FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6900
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6900
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Suresh Srinivas
>            Assignee: Hairong Kuang
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10011-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 05 04:33:42 2010
Return-Path: <common-issues-return-10011-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43512 invoked from network); 5 Aug 2010 04:33:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Aug 2010 04:33:42 -0000
Received: (qmail 94460 invoked by uid 500); 5 Aug 2010 04:33:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94314 invoked by uid 500); 5 Aug 2010 04:33:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94300 invoked by uid 99); 5 Aug 2010 04:33:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 04:33:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 04:33:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o754XHJ9023466
	for <common-issues@hadoop.apache.org>; Thu, 5 Aug 2010 04:33:18 GMT
Message-ID: <20511067.173571280982797908.JavaMail.jira@thor>
Date: Thu, 5 Aug 2010 00:33:17 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6899) RawLocalFileSystem#setWorkingDir()
 does not work for relative names
In-Reply-To: <23450587.169791280964556270.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-6899:
---------------------------------

    Attachment: 6899-2.patch

> RawLocalFileSystem#setWorkingDir() does not work for relative names
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6899
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6899
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.22.0
>
>         Attachments: 6899-2.patch, 6899.patch
>
>
> RawLocalFileSystem#setWorkingDir() does not work for relative names

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10012-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 05 04:55:44 2010
Return-Path: <common-issues-return-10012-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46503 invoked from network); 5 Aug 2010 04:55:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Aug 2010 04:55:44 -0000
Received: (qmail 4340 invoked by uid 500); 5 Aug 2010 04:55:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4201 invoked by uid 500); 5 Aug 2010 04:55:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4193 invoked by uid 99); 5 Aug 2010 04:55:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 04:55:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 04:55:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o754tHD9023593
	for <common-issues@hadoop.apache.org>; Thu, 5 Aug 2010 04:55:17 GMT
Message-ID: <10102203.173701280984117255.JavaMail.jira@thor>
Date: Thu, 5 Aug 2010 00:55:17 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6897) FileSystem#mkdirs(FileSystem, Path,
 FsPermission) should not call setPermission if mkdirs failled
In-Reply-To: <33260487.146971280875096374.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895541#action_12895541 ] 

Hairong Kuang commented on HADOOP-6897:
---------------------------------------

Nicholas, shall we discuss the expected behavior of the static mkdirs method? At least I think it should be consistent with the regular mkdirs except that it sets the absolute permission.

> FileSystem#mkdirs(FileSystem, Path, FsPermission) should not call setPermission if mkdirs failled
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6897
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6897
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: mkdirs.patch
>
>
> Here is the piece of code that has the bug. fs.setPermission should not be called if result is false.
> {code}
>   public static boolean mkdirs(FileSystem fs, Path dir, FsPermission permission)
>   throws IOException {
>     // create the directory using the default permission
>     boolean result = fs.mkdirs(dir);
>     // set its permission to be the supplied one
>     fs.setPermission(dir, permission);
>     return result;
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10013-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 05 04:57:47 2010
Return-Path: <common-issues-return-10013-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46716 invoked from network); 5 Aug 2010 04:57:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Aug 2010 04:57:46 -0000
Received: (qmail 4773 invoked by uid 500); 5 Aug 2010 04:57:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4548 invoked by uid 500); 5 Aug 2010 04:57:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4536 invoked by uid 99); 5 Aug 2010 04:57:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 04:57:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 04:57:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o754vJv1023642
	for <common-issues@hadoop.apache.org>; Thu, 5 Aug 2010 04:57:20 GMT
Message-ID: <15254989.173811280984239632.JavaMail.jira@thor>
Date: Thu, 5 Aug 2010 00:57:19 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6889) Make RPC to have an option to timeout
In-Reply-To: <30758566.75561280434516176.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6889:
----------------------------------

    Status: Open  (was: Patch Available)

> Make RPC to have an option to timeout
> -------------------------------------
>
>                 Key: HADOOP-6889
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6889
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>         Attachments: ipcTimeout.patch, ipcTimeout1.patch, ipcTimeout2.patch
>
>
> Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.
> But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica's length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.
> I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10014-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 05 04:57:48 2010
Return-Path: <common-issues-return-10014-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46746 invoked from network); 5 Aug 2010 04:57:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Aug 2010 04:57:48 -0000
Received: (qmail 4818 invoked by uid 500); 5 Aug 2010 04:57:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4669 invoked by uid 500); 5 Aug 2010 04:57:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4544 invoked by uid 99); 5 Aug 2010 04:57:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 04:57:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 04:57:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o754vLr9023669
	for <common-issues@hadoop.apache.org>; Thu, 5 Aug 2010 04:57:21 GMT
Message-ID: <7398963.173901280984241215.JavaMail.jira@thor>
Date: Thu, 5 Aug 2010 00:57:21 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6889) Make RPC to have an option to timeout
In-Reply-To: <30758566.75561280434516176.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6889:
----------------------------------

    Status: Patch Available  (was: Open)

Resubmit since the previous one seemed to stuck.

> Make RPC to have an option to timeout
> -------------------------------------
>
>                 Key: HADOOP-6889
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6889
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>         Attachments: ipcTimeout.patch, ipcTimeout1.patch, ipcTimeout2.patch
>
>
> Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.
> But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica's length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.
> I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10015-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 05 16:23:43 2010
Return-Path: <common-issues-return-10015-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48940 invoked from network); 5 Aug 2010 16:23:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Aug 2010 16:23:43 -0000
Received: (qmail 30796 invoked by uid 500); 5 Aug 2010 16:23:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30708 invoked by uid 500); 5 Aug 2010 16:23:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30699 invoked by uid 99); 5 Aug 2010 16:23:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 16:23:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 16:23:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o75GNKbL006791
	for <common-issues@hadoop.apache.org>; Thu, 5 Aug 2010 16:23:20 GMT
Message-ID: <22343032.181611281025400870.JavaMail.jira@thor>
Date: Thu, 5 Aug 2010 12:23:20 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6897) FileSystem#mkdirs(FileSystem, Path,
 FsPermission) should not call setPermission if mkdirs failled
In-Reply-To: <33260487.146971280875096374.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895722#action_12895722 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6897:
------------------------------------------------

I think that the static mkdirs(..) is confusing to users.  There are only a few lines of codes inside.  Why don't we just deprecate it now and remove it in the future?

In the meantime, we could update the javadoc as follows:
{code}
/** 
 * This method is equivalent to the following codes:
 *
 *    if (fs.mkdirs(dir)) {
 *      fs.setPermission(dir, permission);
 *      return true;
 *    }
 *    return false;
 */
{code}

> FileSystem#mkdirs(FileSystem, Path, FsPermission) should not call setPermission if mkdirs failled
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6897
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6897
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: mkdirs.patch
>
>
> Here is the piece of code that has the bug. fs.setPermission should not be called if result is false.
> {code}
>   public static boolean mkdirs(FileSystem fs, Path dir, FsPermission permission)
>   throws IOException {
>     // create the directory using the default permission
>     boolean result = fs.mkdirs(dir);
>     // set its permission to be the supplied one
>     fs.setPermission(dir, permission);
>     return result;
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10016-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 05 16:23:47 2010
Return-Path: <common-issues-return-10016-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49142 invoked from network); 5 Aug 2010 16:23:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Aug 2010 16:23:46 -0000
Received: (qmail 32191 invoked by uid 500); 5 Aug 2010 16:23:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31680 invoked by uid 500); 5 Aug 2010 16:23:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31601 invoked by uid 99); 5 Aug 2010 16:23:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 16:23:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 16:23:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o75GNLNB006797
	for <common-issues@hadoop.apache.org>; Thu, 5 Aug 2010 16:23:21 GMT
Message-ID: <19395150.181631281025401452.JavaMail.jira@thor>
Date: Thu, 5 Aug 2010 12:23:21 -0400 (EDT)
From: "Rick Weber (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6901) Parsing large compressed files with
 HADOOP-1722 spawns multiple mappers per file
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Parsing large compressed files with HADOOP-1722 spawns multiple mappers per file
--------------------------------------------------------------------------------

                 Key: HADOOP-6901
                 URL: https://issues.apache.org/jira/browse/HADOOP-6901
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.21.0
         Environment: Hadoop v0.20.2 + HADOOP-1722
            Reporter: Rick Weber


This was originally discovered while using Dumbo to parse a very large (2G) compressed file.  By default, Dumbo will attempt to use the AutoInputFormat as the input format.  

Here is my use case:

I have a large (2Gb) compressed file.  It's compressed using the default method, which is Gzip based and is unsplittable.  Due to the size, the default implementation of AutoInputFormat says that this file is splittable. As a result, this file is split into about 35 parts, and each one is assigned to a Map job.

However, since the file itself is unsplittable, each Map job winds up parsing the file again from the beginning.  This basically means my job has 35x the data, and takes 35x long to run.

If I set "-inputformat text", this problem does not appear in dumbo. If I manually call the streaming jar and use AutoInputFormat, this
problem appears.

Looking at the code in AutoInputFormat, it appears that it uses the default isSplittable() method from InputFormat, which indicates everything is splittable.  I think that this class should define it's own isSplittable method similar to what is defined in the TextInputFormat class, which basically says it's splittable if it's not compressed.



-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10017-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 05 16:35:40 2010
Return-Path: <common-issues-return-10017-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53143 invoked from network); 5 Aug 2010 16:35:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Aug 2010 16:35:40 -0000
Received: (qmail 41147 invoked by uid 500); 5 Aug 2010 16:35:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41053 invoked by uid 500); 5 Aug 2010 16:35:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41044 invoked by uid 99); 5 Aug 2010 16:35:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 16:35:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 16:35:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o75GZI52006901
	for <common-issues@hadoop.apache.org>; Thu, 5 Aug 2010 16:35:18 GMT
Message-ID: <8865988.181801281026118450.JavaMail.jira@thor>
Date: Thu, 5 Aug 2010 12:35:18 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6889) Make RPC to have an option to
 timeout
In-Reply-To: <30758566.75561280434516176.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895724#action_12895724 ] 

Hairong Kuang commented on HADOOP-6889:
---------------------------------------

Hudson is down. I ran ant clean test on my linux box twice. They all passed:

BUILD SUCCESSFUL
Total time: 11 minutes 8 seconds

BUILD SUCCESSFUL
Total time: 9 minutes 55 seconds

> Make RPC to have an option to timeout
> -------------------------------------
>
>                 Key: HADOOP-6889
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6889
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>         Attachments: ipcTimeout.patch, ipcTimeout1.patch, ipcTimeout2.patch
>
>
> Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.
> But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica's length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.
> I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10018-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 05 16:43:43 2010
Return-Path: <common-issues-return-10018-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55325 invoked from network); 5 Aug 2010 16:43:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Aug 2010 16:43:43 -0000
Received: (qmail 50018 invoked by uid 500); 5 Aug 2010 16:43:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49970 invoked by uid 500); 5 Aug 2010 16:43:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49962 invoked by uid 99); 5 Aug 2010 16:43:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 16:43:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 16:43:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o75GhJ56007017
	for <common-issues@hadoop.apache.org>; Thu, 5 Aug 2010 16:43:19 GMT
Message-ID: <306126.182121281026599259.JavaMail.jira@thor>
Date: Thu, 5 Aug 2010 12:43:19 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6889) Make RPC to have an option to timeout
In-Reply-To: <30758566.75561280434516176.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6889:
----------------------------------

    Attachment:     (was: ipcTimeout2.patch)

> Make RPC to have an option to timeout
> -------------------------------------
>
>                 Key: HADOOP-6889
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6889
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>         Attachments: ipcTimeout.patch, ipcTimeout1.patch
>
>
> Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.
> But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica's length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.
> I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10019-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 05 16:45:41 2010
Return-Path: <common-issues-return-10019-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55593 invoked from network); 5 Aug 2010 16:45:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Aug 2010 16:45:40 -0000
Received: (qmail 50608 invoked by uid 500); 5 Aug 2010 16:45:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50519 invoked by uid 500); 5 Aug 2010 16:45:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50511 invoked by uid 99); 5 Aug 2010 16:45:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 16:45:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 16:45:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o75GjJSj007055
	for <common-issues@hadoop.apache.org>; Thu, 5 Aug 2010 16:45:19 GMT
Message-ID: <15665798.182221281026719000.JavaMail.jira@thor>
Date: Thu, 5 Aug 2010 12:45:18 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6889) Make RPC to have an option to timeout
In-Reply-To: <30758566.75561280434516176.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6889:
----------------------------------

    Attachment: ipcTimeout2.patch

> Make RPC to have an option to timeout
> -------------------------------------
>
>                 Key: HADOOP-6889
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6889
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>         Attachments: ipcTimeout.patch, ipcTimeout1.patch, ipcTimeout2.patch
>
>
> Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.
> But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica's length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.
> I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10020-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 05 16:45:41 2010
Return-Path: <common-issues-return-10020-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55638 invoked from network); 5 Aug 2010 16:45:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Aug 2010 16:45:41 -0000
Received: (qmail 50718 invoked by uid 500); 5 Aug 2010 16:45:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50693 invoked by uid 500); 5 Aug 2010 16:45:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50685 invoked by uid 99); 5 Aug 2010 16:45:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 16:45:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 16:45:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o75GjKS1007085
	for <common-issues@hadoop.apache.org>; Thu, 5 Aug 2010 16:45:20 GMT
Message-ID: <20255257.182321281026720808.JavaMail.jira@thor>
Date: Thu, 5 Aug 2010 12:45:20 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6889) Make RPC to have an option to timeout
In-Reply-To: <30758566.75561280434516176.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6889:
----------------------------------

        Status: Resolved  (was: Patch Available)
    Resolution: Fixed

I've just committed this!

> Make RPC to have an option to timeout
> -------------------------------------
>
>                 Key: HADOOP-6889
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6889
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>         Attachments: ipcTimeout.patch, ipcTimeout1.patch, ipcTimeout2.patch
>
>
> Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.
> But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica's length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.
> I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10021-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 05 17:15:39 2010
Return-Path: <common-issues-return-10021-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70008 invoked from network); 5 Aug 2010 17:15:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Aug 2010 17:15:39 -0000
Received: (qmail 81692 invoked by uid 500); 5 Aug 2010 17:15:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81628 invoked by uid 500); 5 Aug 2010 17:15:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81620 invoked by uid 99); 5 Aug 2010 17:15:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 17:15:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 17:15:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o75HFHAR007496
	for <common-issues@hadoop.apache.org>; Thu, 5 Aug 2010 17:15:17 GMT
Message-ID: <23448471.182941281028517865.JavaMail.jira@thor>
Date: Thu, 5 Aug 2010 13:15:17 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6899) RawLocalFileSystem#setWorkingDir()
 does not work for relative names
In-Reply-To: <23450587.169791280964556270.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-6899:
---------------------------------

    Status: Patch Available  (was: Open)

> RawLocalFileSystem#setWorkingDir() does not work for relative names
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6899
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6899
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.22.0
>
>         Attachments: 6899-2.patch, 6899.patch
>
>
> RawLocalFileSystem#setWorkingDir() does not work for relative names

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10022-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 05 17:15:41 2010
Return-Path: <common-issues-return-10022-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70049 invoked from network); 5 Aug 2010 17:15:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Aug 2010 17:15:41 -0000
Received: (qmail 81820 invoked by uid 500); 5 Aug 2010 17:15:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81792 invoked by uid 500); 5 Aug 2010 17:15:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81784 invoked by uid 99); 5 Aug 2010 17:15:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 17:15:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 17:15:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o75HFHuQ007487
	for <common-issues@hadoop.apache.org>; Thu, 5 Aug 2010 17:15:17 GMT
Message-ID: <9068842.182911281028517253.JavaMail.jira@thor>
Date: Thu, 5 Aug 2010 13:15:17 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6899) RawLocalFileSystem#setWorkingDir()
 does not work for relative names
In-Reply-To: <23450587.169791280964556270.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-6899:
---------------------------------

    Status: Open  (was: Patch Available)

> RawLocalFileSystem#setWorkingDir() does not work for relative names
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6899
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6899
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.22.0
>
>         Attachments: 6899-2.patch, 6899.patch
>
>
> RawLocalFileSystem#setWorkingDir() does not work for relative names

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10023-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 05 21:25:43 2010
Return-Path: <common-issues-return-10023-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90916 invoked from network); 5 Aug 2010 21:25:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Aug 2010 21:25:43 -0000
Received: (qmail 88498 invoked by uid 500); 5 Aug 2010 21:25:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88431 invoked by uid 500); 5 Aug 2010 21:25:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88423 invoked by uid 99); 5 Aug 2010 21:25:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 21:25:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 21:25:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o75LPIZO010174
	for <common-issues@hadoop.apache.org>; Thu, 5 Aug 2010 21:25:18 GMT
Message-ID: <9217988.186961281043518534.JavaMail.jira@thor>
Date: Thu, 5 Aug 2010 17:25:18 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6889) Make RPC to have an option to
 timeout
In-Reply-To: <30758566.75561280434516176.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895814#action_12895814 ] 

Kan Zhang commented on HADOOP-6889:
-----------------------------------

I know I'm a little late to the party. But I'm just wondering if it's better to pass rcpTimeout using the existing conf param rather than adding a new parameter to getProxy()?

> Make RPC to have an option to timeout
> -------------------------------------
>
>                 Key: HADOOP-6889
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6889
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>         Attachments: ipcTimeout.patch, ipcTimeout1.patch, ipcTimeout2.patch
>
>
> Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.
> But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica's length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.
> I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10024-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 05 21:45:43 2010
Return-Path: <common-issues-return-10024-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98722 invoked from network); 5 Aug 2010 21:45:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Aug 2010 21:45:42 -0000
Received: (qmail 5068 invoked by uid 500); 5 Aug 2010 21:45:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5021 invoked by uid 500); 5 Aug 2010 21:45:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5012 invoked by uid 99); 5 Aug 2010 21:45:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 21:45:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 21:45:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o75LjLnq010453
	for <common-issues@hadoop.apache.org>; Thu, 5 Aug 2010 21:45:21 GMT
Message-ID: <18636236.187651281044721001.JavaMail.jira@thor>
Date: Thu, 5 Aug 2010 17:45:20 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6889) Make RPC to have an option to
 timeout
In-Reply-To: <30758566.75561280434516176.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895823#action_12895823 ] 

Hairong Kuang commented on HADOOP-6889:
---------------------------------------

Kan, thanks for taking a look at the patch!

I thought about introducing a configuration parameter. But clients or DataNodes want to have timeout for RPCs to DataNodes but no timeout for RPCs to NameNodes. Adding a rpcTimeout parameter makes this easy.

> Make RPC to have an option to timeout
> -------------------------------------
>
>                 Key: HADOOP-6889
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6889
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>         Attachments: ipcTimeout.patch, ipcTimeout1.patch, ipcTimeout2.patch
>
>
> Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.
> But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica's length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.
> I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10025-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 05 21:45:48 2010
Return-Path: <common-issues-return-10025-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99004 invoked from network); 5 Aug 2010 21:45:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Aug 2010 21:45:48 -0000
Received: (qmail 6415 invoked by uid 500); 5 Aug 2010 21:45:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6387 invoked by uid 500); 5 Aug 2010 21:45:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6379 invoked by uid 99); 5 Aug 2010 21:45:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 21:45:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 21:45:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o75LjO85010487
	for <common-issues@hadoop.apache.org>; Thu, 5 Aug 2010 21:45:24 GMT
Message-ID: <17911648.187761281044724328.JavaMail.jira@thor>
Date: Thu, 5 Aug 2010 17:45:24 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6897) FileSystem#mkdirs(FileSystem, Path,
 FsPermission) should not call setPermission if mkdirs failled
In-Reply-To: <33260487.146971280875096374.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895824#action_12895824 ] 

Hairong Kuang commented on HADOOP-6897:
---------------------------------------

This utility is used by Map/Reduce. So removing it may not be an option. :-(

> FileSystem#mkdirs(FileSystem, Path, FsPermission) should not call setPermission if mkdirs failled
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6897
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6897
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: mkdirs.patch
>
>
> Here is the piece of code that has the bug. fs.setPermission should not be called if result is false.
> {code}
>   public static boolean mkdirs(FileSystem fs, Path dir, FsPermission permission)
>   throws IOException {
>     // create the directory using the default permission
>     boolean result = fs.mkdirs(dir);
>     // set its permission to be the supplied one
>     fs.setPermission(dir, permission);
>     return result;
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10026-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 05 21:49:44 2010
Return-Path: <common-issues-return-10026-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99801 invoked from network); 5 Aug 2010 21:49:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Aug 2010 21:49:43 -0000
Received: (qmail 12078 invoked by uid 500); 5 Aug 2010 21:49:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12020 invoked by uid 500); 5 Aug 2010 21:49:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12012 invoked by uid 99); 5 Aug 2010 21:49:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 21:49:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 21:49:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o75LnJbO010593
	for <common-issues@hadoop.apache.org>; Thu, 5 Aug 2010 21:49:19 GMT
Message-ID: <8084469.188061281044959349.JavaMail.jira@thor>
Date: Thu, 5 Aug 2010 17:49:19 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6889) Make RPC to have an option to
 timeout
In-Reply-To: <30758566.75561280434516176.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895827#action_12895827 ] 

Kan Zhang commented on HADOOP-6889:
-----------------------------------

I see.

> Make RPC to have an option to timeout
> -------------------------------------
>
>                 Key: HADOOP-6889
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6889
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>         Attachments: ipcTimeout.patch, ipcTimeout1.patch, ipcTimeout2.patch
>
>
> Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.
> But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica's length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.
> I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10027-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 05 21:53:40 2010
Return-Path: <common-issues-return-10027-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 503 invoked from network); 5 Aug 2010 21:53:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Aug 2010 21:53:40 -0000
Received: (qmail 14722 invoked by uid 500); 5 Aug 2010 21:53:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14646 invoked by uid 500); 5 Aug 2010 21:53:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14638 invoked by uid 99); 5 Aug 2010 21:53:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 21:53:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 21:53:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o75LrJqr010662
	for <common-issues@hadoop.apache.org>; Thu, 5 Aug 2010 21:53:19 GMT
Message-ID: <22174880.188241281045199272.JavaMail.jira@thor>
Date: Thu, 5 Aug 2010 17:53:19 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6897) FileSystem#mkdirs(FileSystem, Path,
 FsPermission) should not call setPermission if mkdirs failled
In-Reply-To: <33260487.146971280875096374.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895830#action_12895830 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6897:
------------------------------------------------

Map/Reduce could just replace it with two or three lines of codes.

I forgot to mention that there is another drawback about the static mkdir(..).  Some users calling it do not realize that it is actually more expensive compared with the non-static mkdir(..) -- it requires one more rpc calls.

> FileSystem#mkdirs(FileSystem, Path, FsPermission) should not call setPermission if mkdirs failled
> -------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6897
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6897
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: mkdirs.patch
>
>
> Here is the piece of code that has the bug. fs.setPermission should not be called if result is false.
> {code}
>   public static boolean mkdirs(FileSystem fs, Path dir, FsPermission permission)
>   throws IOException {
>     // create the directory using the default permission
>     boolean result = fs.mkdirs(dir);
>     // set its permission to be the supplied one
>     fs.setPermission(dir, permission);
>     return result;
>   }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10028-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 05 22:43:44 2010
Return-Path: <common-issues-return-10028-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16782 invoked from network); 5 Aug 2010 22:43:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Aug 2010 22:43:43 -0000
Received: (qmail 72334 invoked by uid 500); 5 Aug 2010 22:43:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72234 invoked by uid 500); 5 Aug 2010 22:43:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72226 invoked by uid 99); 5 Aug 2010 22:43:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 22:43:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 22:43:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o75MhIh5011399
	for <common-issues@hadoop.apache.org>; Thu, 5 Aug 2010 22:43:18 GMT
Message-ID: <2821125.189721281048198694.JavaMail.jira@thor>
Date: Thu, 5 Aug 2010 18:43:18 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6900) FileSystem#listLocatedStatus should
 not throw generic RuntimeException to indicate error conditions
In-Reply-To: <29253908.169821280964676744.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895852#action_12895852 ] 

Hairong Kuang commented on HADOOP-6900:
---------------------------------------

Suresh, good idea! I appreciate your thoughts on this.

I will create a new Iterator interface that throws IOException on next() and hasNext().

I will make listFiles, listLocatedStatus, AND listStatus to return this new iterator iterface. How does this sound to you?

> FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6900
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6900
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Suresh Srinivas
>            Assignee: Hairong Kuang
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10029-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 05 23:35:42 2010
Return-Path: <common-issues-return-10029-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34458 invoked from network); 5 Aug 2010 23:35:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Aug 2010 23:35:42 -0000
Received: (qmail 14626 invoked by uid 500); 5 Aug 2010 23:35:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14429 invoked by uid 500); 5 Aug 2010 23:35:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14421 invoked by uid 99); 5 Aug 2010 23:35:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 23:35:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Aug 2010 23:35:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o75NZIOf011900
	for <common-issues@hadoop.apache.org>; Thu, 5 Aug 2010 23:35:18 GMT
Message-ID: <31093459.190711281051317983.JavaMail.jira@thor>
Date: Thu, 5 Aug 2010 19:35:17 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6902) RPC server's SASL_PROPS shouldn't be
 re-initialized every time an RPC client is created
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

RPC server's SASL_PROPS shouldn't be re-initialized every time an RPC client is created
---------------------------------------------------------------------------------------

                 Key: HADOOP-6902
                 URL: https://issues.apache.org/jira/browse/HADOOP-6902
             Project: Hadoop Common
          Issue Type: Bug
          Components: ipc, security
            Reporter: Kan Zhang


SaslRpcServer.SASL_PROPS is a SASL server property and should stay constant after initialization. In the initial implementation, we assumed all SASL clients will use the same constant value. If different clients might use different values depending on the conf in the getProxy() call (as current code implies), each client should have its own copy. In any case, a client shouldn't re-initialize server's copy.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10030-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 00:04:41 2010
Return-Path: <common-issues-return-10030-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41093 invoked from network); 6 Aug 2010 00:04:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 00:04:41 -0000
Received: (qmail 35042 invoked by uid 500); 6 Aug 2010 00:04:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34960 invoked by uid 500); 6 Aug 2010 00:04:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34952 invoked by uid 99); 6 Aug 2010 00:04:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 00:04:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 00:04:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7604KB5012113
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 00:04:20 GMT
Message-ID: <21449593.191011281053059997.JavaMail.jira@thor>
Date: Thu, 5 Aug 2010 20:04:19 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6900) FileSystem#listLocatedStatus should
 not throw generic RuntimeException to indicate error conditions
In-Reply-To: <29253908.169821280964676744.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6900:
----------------------------------

    Attachment: listFilesIO.patch

A patch for review.

> FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6900
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6900
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Suresh Srinivas
>            Assignee: Hairong Kuang
>         Attachments: listFilesIO.patch
>
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10031-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 01:17:40 2010
Return-Path: <common-issues-return-10031-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83065 invoked from network); 6 Aug 2010 01:17:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 01:17:39 -0000
Received: (qmail 24698 invoked by uid 500); 6 Aug 2010 01:17:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24612 invoked by uid 500); 6 Aug 2010 01:17:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24604 invoked by uid 99); 6 Aug 2010 01:17:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 01:17:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 01:17:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o761HH4N019781
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 01:17:17 GMT
Message-ID: <23453598.192091281057437429.JavaMail.jira@thor>
Date: Thu, 5 Aug 2010 21:17:17 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6786) test-patch needs to verify Herriot
 integrity
In-Reply-To: <19336182.46451274829212745.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895907#action_12895907 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6786:
------------------------------------------------

I may have missed something:
- In the code below, why setting PreTestTarget to a empty string?
{code}
+  PreTestTarget=""
+
+  echo "$ANT_HOME/bin/ant -Dversion="${VERSION}" -DHadoopPatchProcess= -Dtest.junit.output.format=xml -Dtest.output=yes -Dcompile.c++=yes -Dforrest.home=$FORREST_HOME -Djava5.home=$JAVA5_HOME $PreTestTarget inject-system-faults"
+  $ANT_HOME/bin/ant -Dversion="${VERSION}" -DHadoopPatchProcess= -Dtest.junit.output.format=xml -Dtest.output=yes -Dcompile.c++=yes -Dforrest.home=$FORREST_HOME -Djava5.home=$JAVA5_HOME $PreTestTarget inject-system-faults
{code}
- The inject-system-faults target is not found in build.xml.  Where is it defined?

> test-patch needs to verify Herriot integrity
> --------------------------------------------
>
>                 Key: HADOOP-6786
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6786
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-6786.patch
>
>
> Whenever a new patch is submitted for verification {{test-patch}} process has to make sure that none of Herriot bindings were broken.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10032-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 01:19:42 2010
Return-Path: <common-issues-return-10032-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83224 invoked from network); 6 Aug 2010 01:19:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 01:19:41 -0000
Received: (qmail 24902 invoked by uid 500); 6 Aug 2010 01:19:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24854 invoked by uid 500); 6 Aug 2010 01:19:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24846 invoked by uid 99); 6 Aug 2010 01:19:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 01:19:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 01:19:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o761JHqA020056
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 01:19:17 GMT
Message-ID: <8878558.192121281057557094.JavaMail.jira@thor>
Date: Thu, 5 Aug 2010 21:19:17 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6899) RawLocalFileSystem#setWorkingDir()
 does not work for relative names
In-Reply-To: <23450587.169791280964556270.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-6899:
---------------------------------

    Attachment: 6899-3.patch

> RawLocalFileSystem#setWorkingDir() does not work for relative names
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6899
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6899
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.22.0
>
>         Attachments: 6899-2.patch, 6899-3.patch, 6899.patch
>
>
> RawLocalFileSystem#setWorkingDir() does not work for relative names

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10033-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 01:55:41 2010
Return-Path: <common-issues-return-10033-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93129 invoked from network); 6 Aug 2010 01:55:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 01:55:41 -0000
Received: (qmail 59012 invoked by uid 500); 6 Aug 2010 01:55:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58951 invoked by uid 500); 6 Aug 2010 01:55:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58943 invoked by uid 99); 6 Aug 2010 01:55:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 01:55:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 01:55:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o761tHIY025180
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 01:55:17 GMT
Message-ID: <12908669.192531281059716997.JavaMail.jira@thor>
Date: Thu, 5 Aug 2010 21:55:16 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6786) test-patch needs to verify Herriot
 integrity
In-Reply-To: <19336182.46451274829212745.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6786:
---------------------------------------

    Attachment: HADOOP-6786.patch

{{PreTestTarget}} is a garbage I have missed out. Removed.
The target {{inject-system-faults}} is defined in src/test/aop/build/aop.xml

> test-patch needs to verify Herriot integrity
> --------------------------------------------
>
>                 Key: HADOOP-6786
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6786
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-6786.patch, HADOOP-6786.patch
>
>
> Whenever a new patch is submitted for verification {{test-patch}} process has to make sure that none of Herriot bindings were broken.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10034-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 02:05:40 2010
Return-Path: <common-issues-return-10034-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98370 invoked from network); 6 Aug 2010 02:05:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 02:05:40 -0000
Received: (qmail 60867 invoked by uid 500); 6 Aug 2010 02:05:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60799 invoked by uid 500); 6 Aug 2010 02:05:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60791 invoked by uid 99); 6 Aug 2010 02:05:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 02:05:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 02:05:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7625Idf026751
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 02:05:18 GMT
Message-ID: <15444068.192741281060318829.JavaMail.jira@thor>
Date: Thu, 5 Aug 2010 22:05:18 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6899) RawLocalFileSystem#setWorkingDir()
 does not work for relative names
In-Reply-To: <23450587.169791280964556270.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-6899:
---------------------------------

    Status: Patch Available  (was: Open)

> RawLocalFileSystem#setWorkingDir() does not work for relative names
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6899
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6899
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.22.0
>
>         Attachments: 6899-2.patch, 6899-3.patch, 6899-4.patch, 6899.patch
>
>
> RawLocalFileSystem#setWorkingDir() does not work for relative names

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10035-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 02:05:43 2010
Return-Path: <common-issues-return-10035-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98425 invoked from network); 6 Aug 2010 02:05:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 02:05:43 -0000
Received: (qmail 61182 invoked by uid 500); 6 Aug 2010 02:05:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61107 invoked by uid 500); 6 Aug 2010 02:05:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61099 invoked by uid 99); 6 Aug 2010 02:05:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 02:05:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 02:05:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7625HkI026729
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 02:05:17 GMT
Message-ID: <28149470.192681281060317670.JavaMail.jira@thor>
Date: Thu, 5 Aug 2010 22:05:17 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6899) RawLocalFileSystem#setWorkingDir()
 does not work for relative names
In-Reply-To: <23450587.169791280964556270.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-6899:
---------------------------------

    Attachment: 6899-4.patch

Addressed Nicolas's feedback.

> RawLocalFileSystem#setWorkingDir() does not work for relative names
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6899
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6899
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.22.0
>
>         Attachments: 6899-2.patch, 6899-3.patch, 6899-4.patch, 6899.patch
>
>
> RawLocalFileSystem#setWorkingDir() does not work for relative names

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10036-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 02:05:43 2010
Return-Path: <common-issues-return-10036-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98439 invoked from network); 6 Aug 2010 02:05:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 02:05:43 -0000
Received: (qmail 61224 invoked by uid 500); 6 Aug 2010 02:05:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61117 invoked by uid 500); 6 Aug 2010 02:05:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61108 invoked by uid 99); 6 Aug 2010 02:05:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 02:05:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 02:05:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7625ISe026738
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 02:05:18 GMT
Message-ID: <30885605.192711281060318358.JavaMail.jira@thor>
Date: Thu, 5 Aug 2010 22:05:18 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6899) RawLocalFileSystem#setWorkingDir()
 does not work for relative names
In-Reply-To: <23450587.169791280964556270.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-6899:
---------------------------------

    Status: Open  (was: Patch Available)

> RawLocalFileSystem#setWorkingDir() does not work for relative names
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6899
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6899
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.22.0
>
>         Attachments: 6899-2.patch, 6899-3.patch, 6899-4.patch, 6899.patch
>
>
> RawLocalFileSystem#setWorkingDir() does not work for relative names

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10037-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 05:55:42 2010
Return-Path: <common-issues-return-10037-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81534 invoked from network); 6 Aug 2010 05:55:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 05:55:41 -0000
Received: (qmail 59059 invoked by uid 500); 6 Aug 2010 05:55:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58903 invoked by uid 500); 6 Aug 2010 05:55:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58885 invoked by uid 99); 6 Aug 2010 05:55:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 05:55:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 05:55:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o765tGkb001516
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 05:55:17 GMT
Message-ID: <14012018.194341281074116525.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 01:55:16 -0400 (EDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6862) Add api to add user/group to
 AccessControlList
In-Reply-To: <6885195.391831279173830267.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amareshwari Sriramadasu updated HADOOP-6862:
--------------------------------------------

    Attachment: patch-6862-2.txt

Patch with suggested changes

> Add api to add user/group to AccessControlList
> ----------------------------------------------
>
>                 Key: HADOOP-6862
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6862
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.22.0
>
>         Attachments: patch-6862-1.txt, patch-6862-2.txt, patch-6862.txt
>
>
> Add api addUser(String user) and addGroup(String group) to add user/group to AccessControlList

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10038-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 05:55:43 2010
Return-Path: <common-issues-return-10038-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81560 invoked from network); 6 Aug 2010 05:55:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 05:55:43 -0000
Received: (qmail 59131 invoked by uid 500); 6 Aug 2010 05:55:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58923 invoked by uid 500); 6 Aug 2010 05:55:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58888 invoked by uid 99); 6 Aug 2010 05:55:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 05:55:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 05:55:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o765tHAM001525
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 05:55:17 GMT
Message-ID: <32440504.194371281074117521.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 01:55:17 -0400 (EDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6862) Add api to add user/group to
 AccessControlList
In-Reply-To: <6885195.391831279173830267.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amareshwari Sriramadasu updated HADOOP-6862:
--------------------------------------------

    Status: Patch Available  (was: Open)

> Add api to add user/group to AccessControlList
> ----------------------------------------------
>
>                 Key: HADOOP-6862
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6862
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.22.0
>
>         Attachments: patch-6862-1.txt, patch-6862-2.txt, patch-6862.txt
>
>
> Add api addUser(String user) and addGroup(String group) to add user/group to AccessControlList

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10039-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 07:42:42 2010
Return-Path: <common-issues-return-10039-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33749 invoked from network); 6 Aug 2010 07:42:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 07:42:42 -0000
Received: (qmail 36705 invoked by uid 500); 6 Aug 2010 07:42:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36048 invoked by uid 500); 6 Aug 2010 07:42:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36034 invoked by uid 99); 6 Aug 2010 07:42:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 07:42:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 07:42:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o767gH5E002678
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 07:42:17 GMT
Message-ID: <6350216.196101281080537278.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 03:42:17 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6899) RawLocalFileSystem#setWorkingDir()
 does not work for relative names
In-Reply-To: <23450587.169791280964556270.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6899:
-------------------------------------------

    Hadoop Flags: [Reviewed]
     Component/s: fs

+1 patch looks good.

> RawLocalFileSystem#setWorkingDir() does not work for relative names
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6899
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6899
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.2
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.22.0
>
>         Attachments: 6899-2.patch, 6899-3.patch, 6899-4.patch, 6899.patch
>
>
> RawLocalFileSystem#setWorkingDir() does not work for relative names

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10040-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 07:58:41 2010
Return-Path: <common-issues-return-10040-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35608 invoked from network); 6 Aug 2010 07:58:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 07:58:41 -0000
Received: (qmail 44655 invoked by uid 500); 6 Aug 2010 07:58:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44518 invoked by uid 500); 6 Aug 2010 07:58:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44510 invoked by uid 99); 6 Aug 2010 07:58:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 07:58:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 07:58:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o767wG5q002885
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 07:58:17 GMT
Message-ID: <22166007.196441281081496717.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 03:58:16 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6862) Add api to add user/group to
 AccessControlList
In-Reply-To: <6885195.391831279173830267.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895986#action_12895986 ] 

Vinod K V commented on HADOOP-6862:
-----------------------------------

Patch looks good. +1.

> Add api to add user/group to AccessControlList
> ----------------------------------------------
>
>                 Key: HADOOP-6862
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6862
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.22.0
>
>         Attachments: patch-6862-1.txt, patch-6862-2.txt, patch-6862.txt
>
>
> Add api addUser(String user) and addGroup(String group) to add user/group to AccessControlList

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10041-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 10:09:41 2010
Return-Path: <common-issues-return-10041-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92487 invoked from network); 6 Aug 2010 10:09:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 10:09:41 -0000
Received: (qmail 64450 invoked by uid 500); 6 Aug 2010 10:09:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64309 invoked by uid 500); 6 Aug 2010 10:09:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64301 invoked by uid 99); 6 Aug 2010 10:09:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 10:09:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 10:09:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o76A9G1x006317
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 10:09:17 GMT
Message-ID: <19995888.197781281089356975.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 06:09:16 -0400 (EDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6862) Add api to add user/group to
 AccessControlList
In-Reply-To: <6885195.391831279173830267.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896012#action_12896012 ] 

Amareshwari Sriramadasu commented on HADOOP-6862:
-------------------------------------------------

test-patch result:
{noformat}
     [exec]
     [exec] -1 overall.
     [exec]
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec]
     [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
     [exec]
     [exec]     -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.
     [exec]
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec]
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec]
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec]
{noformat}
bq -1 javadoc.  
Is due to HADOOP-6830. Verified there no new javadoc warnings introduced in the patch.

All unit tests passed on my local machine.



> Add api to add user/group to AccessControlList
> ----------------------------------------------
>
>                 Key: HADOOP-6862
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6862
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.22.0
>
>         Attachments: patch-6862-1.txt, patch-6862-2.txt, patch-6862.txt
>
>
> Add api addUser(String user) and addGroup(String group) to add user/group to AccessControlList

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10042-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 14:26:49 2010
Return-Path: <common-issues-return-10042-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11272 invoked from network); 6 Aug 2010 14:26:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 14:26:44 -0000
Received: (qmail 22440 invoked by uid 500); 6 Aug 2010 14:26:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22267 invoked by uid 500); 6 Aug 2010 14:26:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22248 invoked by uid 99); 6 Aug 2010 14:26:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 14:26:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 14:26:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o76EQIEl008060
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 14:26:19 GMT
Message-ID: <30079419.199041281104778284.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 10:26:18 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6786) test-patch needs to verify Herriot
 integrity
In-Reply-To: <19336182.46451274829212745.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896046#action_12896046 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6786:
------------------------------------------------

I think we need
{code}
  (( RESULT = RESULT + $? ))
{code}
after calling checkInjectSystemFaults.

> test-patch needs to verify Herriot integrity
> --------------------------------------------
>
>                 Key: HADOOP-6786
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6786
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-6786.patch, HADOOP-6786.patch
>
>
> Whenever a new patch is submitted for verification {{test-patch}} process has to make sure that none of Herriot bindings were broken.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10043-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 14:43:41 2010
Return-Path: <common-issues-return-10043-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22443 invoked from network); 6 Aug 2010 14:43:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 14:43:40 -0000
Received: (qmail 50565 invoked by uid 500); 6 Aug 2010 14:43:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50416 invoked by uid 500); 6 Aug 2010 14:43:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50401 invoked by uid 99); 6 Aug 2010 14:43:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 14:43:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 14:43:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o76EhIWj008199
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 14:43:18 GMT
Message-ID: <4483068.199231281105798038.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 10:43:18 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6899) RawLocalFileSystem#setWorkingDir()
 does not work for relative names
In-Reply-To: <23450587.169791280964556270.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896052#action_12896052 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6899:
------------------------------------------------

{noformat}
     [exec] -1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 20 new or modified tests.
     [exec] 
     [exec]     -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.
     [exec] 
     [exec]     -1 javac.  The applied patch generated 1018 javac compiler warnings (more than the trunk's current 1017 warnings).
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
{noformat}

> RawLocalFileSystem#setWorkingDir() does not work for relative names
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6899
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6899
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.2
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.22.0
>
>         Attachments: 6899-2.patch, 6899-3.patch, 6899-4.patch, 6899.patch
>
>
> RawLocalFileSystem#setWorkingDir() does not work for relative names

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10044-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 14:51:43 2010
Return-Path: <common-issues-return-10044-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26959 invoked from network); 6 Aug 2010 14:51:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 14:51:43 -0000
Received: (qmail 71910 invoked by uid 500); 6 Aug 2010 14:51:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71748 invoked by uid 500); 6 Aug 2010 14:51:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71740 invoked by uid 99); 6 Aug 2010 14:51:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 14:51:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 14:51:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o76EpL5P008261
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 14:51:21 GMT
Message-ID: <416484.199371281106281595.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 10:51:21 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6899) RawLocalFileSystem#setWorkingDir()
 does not work for relative names
In-Reply-To: <23450587.169791280964556270.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896057#action_12896057 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6899:
------------------------------------------------

Here is the new javac warning
{noformat}
    [javac] /home/tsz/hadoop/common/testpatch-trunk/src/test/core/org/apache/hadoop/fs/FSMainOperationsBaseTest.java:1056:
 warning: [deprecation] rename(org.apache.hadoop.fs.Path,org.apache.hadoop.fs.Path,org.apache.hadoop.fs.Options.Rename...)
 in org.apache.hadoop.fs.FileSystem has been deprecated
    [javac]     fSys.rename(src, dst, options);
    [javac]         ^
{noformat}
The javadoc warnings already exists in trunk and not related to this.

The patch also passed all unit tests.

> RawLocalFileSystem#setWorkingDir() does not work for relative names
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6899
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6899
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.2
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.22.0
>
>         Attachments: 6899-2.patch, 6899-3.patch, 6899-4.patch, 6899.patch
>
>
> RawLocalFileSystem#setWorkingDir() does not work for relative names

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10045-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 17:58:41 2010
Return-Path: <common-issues-return-10045-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12511 invoked from network); 6 Aug 2010 17:58:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 17:58:41 -0000
Received: (qmail 53837 invoked by uid 500); 6 Aug 2010 17:58:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53776 invoked by uid 500); 6 Aug 2010 17:58:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53768 invoked by uid 99); 6 Aug 2010 17:58:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 17:58:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 17:58:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o76HwHmU010143
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 17:58:17 GMT
Message-ID: <20172438.202041281117497300.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 13:58:17 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6899) RawLocalFileSystem#setWorkingDir()
 does not work for relative names
In-Reply-To: <23450587.169791280964556270.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sanjay Radia updated HADOOP-6899:
---------------------------------

    Attachment: 6899-5.patch

The calling of the obsolete rename was intended (the tests we ported use that).
Add a suppress warning tag.

> RawLocalFileSystem#setWorkingDir() does not work for relative names
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6899
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6899
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.2
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.22.0
>
>         Attachments: 6899-2.patch, 6899-3.patch, 6899-4.patch, 6899-5.patch, 6899.patch
>
>
> RawLocalFileSystem#setWorkingDir() does not work for relative names

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10046-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 19:10:39 2010
Return-Path: <common-issues-return-10046-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58073 invoked from network); 6 Aug 2010 19:10:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 19:10:38 -0000
Received: (qmail 63325 invoked by uid 500); 6 Aug 2010 19:10:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63243 invoked by uid 500); 6 Aug 2010 19:10:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63234 invoked by uid 99); 6 Aug 2010 19:10:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 19:10:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 19:10:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o76JAH52010976
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 19:10:17 GMT
Message-ID: <8317270.203351281121817023.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 15:10:17 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6786) test-patch needs to verify Herriot
 integrity
In-Reply-To: <19336182.46451274829212745.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6786:
---------------------------------------

    Status: Open  (was: Patch Available)

> test-patch needs to verify Herriot integrity
> --------------------------------------------
>
>                 Key: HADOOP-6786
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6786
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-6786.patch, HADOOP-6786.patch, HADOOP-6786.patch
>
>
> Whenever a new patch is submitted for verification {{test-patch}} process has to make sure that none of Herriot bindings were broken.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10047-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 19:10:39 2010
Return-Path: <common-issues-return-10047-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58121 invoked from network); 6 Aug 2010 19:10:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 19:10:39 -0000
Received: (qmail 63441 invoked by uid 500); 6 Aug 2010 19:10:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63301 invoked by uid 500); 6 Aug 2010 19:10:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63250 invoked by uid 99); 6 Aug 2010 19:10:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 19:10:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 19:10:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o76JAHqL010991
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 19:10:17 GMT
Message-ID: <22940035.203401281121817886.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 15:10:17 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6786) test-patch needs to verify Herriot
 integrity
In-Reply-To: <19336182.46451274829212745.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6786:
---------------------------------------

    Attachment: HADOOP-6786.patch

Thanks for catching it up, Nicholas. (note to self: there's no 'simple' patches, damn it). Attaching new version and re-submitting the patch.

> test-patch needs to verify Herriot integrity
> --------------------------------------------
>
>                 Key: HADOOP-6786
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6786
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-6786.patch, HADOOP-6786.patch, HADOOP-6786.patch
>
>
> Whenever a new patch is submitted for verification {{test-patch}} process has to make sure that none of Herriot bindings were broken.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10048-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 19:12:41 2010
Return-Path: <common-issues-return-10048-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58688 invoked from network); 6 Aug 2010 19:12:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 19:12:40 -0000
Received: (qmail 66917 invoked by uid 500); 6 Aug 2010 19:12:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66881 invoked by uid 500); 6 Aug 2010 19:12:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66873 invoked by uid 99); 6 Aug 2010 19:12:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 19:12:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 19:12:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o76JCHwK011004
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 19:12:17 GMT
Message-ID: <3225437.203441281121937175.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 15:12:17 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6786) test-patch needs to verify Herriot
 integrity
In-Reply-To: <19336182.46451274829212745.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6786:
---------------------------------------

    Status: Patch Available  (was: Open)

> test-patch needs to verify Herriot integrity
> --------------------------------------------
>
>                 Key: HADOOP-6786
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6786
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-6786.patch, HADOOP-6786.patch, HADOOP-6786.patch
>
>
> Whenever a new patch is submitted for verification {{test-patch}} process has to make sure that none of Herriot bindings were broken.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10049-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 19:16:40 2010
Return-Path: <common-issues-return-10049-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59718 invoked from network); 6 Aug 2010 19:16:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 19:16:40 -0000
Received: (qmail 70241 invoked by uid 500); 6 Aug 2010 19:16:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70184 invoked by uid 500); 6 Aug 2010 19:16:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70176 invoked by uid 99); 6 Aug 2010 19:16:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 19:16:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 19:16:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o76JGG6M011039
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 19:16:16 GMT
Message-ID: <8679405.203491281122176280.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 15:16:16 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6903) Make AbstractFileSystem's methods
 public to allow filter-Fs like implementions in a differnt package than fs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Make AbstractFileSystem's methods public to allow filter-Fs like implementions in a differnt package than fs
------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-6903
                 URL: https://issues.apache.org/jira/browse/HADOOP-6903
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Sanjay Radia


Make AbstractFileSystem's methods public to allow filter-Fs like implementions in a differnt package than fs

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10050-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 19:20:39 2010
Return-Path: <common-issues-return-10050-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61234 invoked from network); 6 Aug 2010 19:20:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 19:20:39 -0000
Received: (qmail 77048 invoked by uid 500); 6 Aug 2010 19:20:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77000 invoked by uid 500); 6 Aug 2010 19:20:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76850 invoked by uid 99); 6 Aug 2010 19:20:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 19:20:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 19:20:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o76JKHEP011108
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 19:20:17 GMT
Message-ID: <15363045.203611281122417504.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 15:20:17 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6903) Make AbstractFileSystem's methods
 public to allow filter-Fs like implementions in a differnt package than fs
In-Reply-To: <8679405.203491281122176280.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896130#action_12896130 ] 

Sanjay Radia commented on HADOOP-6903:
--------------------------------------

I was implementing a mount table file system in a package called viewFs. As part of this I  had to call an instance of AbstractFilesSystem. Unfortunately the methods are protected.
I am proposing that we make the methods public to allow the convenience of using other packages besides o.a.h.fs.

> Make AbstractFileSystem's methods public to allow filter-Fs like implementions in a differnt package than fs
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6903
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6903
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>
> Make AbstractFileSystem's methods public to allow filter-Fs like implementions in a differnt package than fs

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10051-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 21:43:40 2010
Return-Path: <common-issues-return-10051-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13624 invoked from network); 6 Aug 2010 21:43:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 21:43:40 -0000
Received: (qmail 27609 invoked by uid 500); 6 Aug 2010 21:43:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27557 invoked by uid 500); 6 Aug 2010 21:43:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27545 invoked by uid 99); 6 Aug 2010 21:43:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 21:43:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 21:43:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o76LhIgU012357
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 21:43:18 GMT
Message-ID: <33108294.205771281130998692.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 17:43:18 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Moved: (HADOOP-6904) A baby step towards inter-version
 communications between dfs client and NameNode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang moved HDFS-1329 to HADOOP-6904:
---------------------------------------------

              Project: Hadoop Common  (was: Hadoop HDFS)
                  Key: HADOOP-6904  (was: HDFS-1329)
    Affects Version/s: 0.22.0
                           (was: 0.22.0)
        Fix Version/s: 0.22.0
                           (was: 0.22.0)
          Component/s: ipc
                           (was: hdfs client)

> A baby step towards inter-version communications between dfs client and NameNode
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6904
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6904
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>
> Currently RPC communications in Hadoop is very strict. If a client has a different version from that of the server, a VersionMismatched exception is thrown and the client can not connect to the server. This force us to update both client and server all at once if a RPC protocol is changed. But sometime different versions do not mean the client & server are not compatible. It would be nice if we could relax this restriction and allows us to support inter-version communications.
> My idea is that DfsClient catches VersionMismatched exception when it connects to NameNode. It then checks if the client & the server is compatible. If yes, it sets the NameNode version in the dfs client and allows the client to continue talking to NameNode. Otherwise, rethrow the VersionMismatch exception.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10052-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 21:45:40 2010
Return-Path: <common-issues-return-10052-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13936 invoked from network); 6 Aug 2010 21:45:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 21:45:40 -0000
Received: (qmail 30101 invoked by uid 500); 6 Aug 2010 21:45:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30061 invoked by uid 500); 6 Aug 2010 21:45:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30053 invoked by uid 99); 6 Aug 2010 21:45:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 21:45:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 21:45:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o76LjIb9012408
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 21:45:18 GMT
Message-ID: <3010059.205921281131118560.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 17:45:18 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6904) A baby step towards inter-version
 communications between dfs client and NameNode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896167#action_12896167 ] 

Hairong Kuang commented on HADOOP-6904:
---------------------------------------

Looks that all the changes I need to make is in ipc. So I move this jira from hdfs to common.

> A baby step towards inter-version communications between dfs client and NameNode
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6904
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6904
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>
> Currently RPC communications in Hadoop is very strict. If a client has a different version from that of the server, a VersionMismatched exception is thrown and the client can not connect to the server. This force us to update both client and server all at once if a RPC protocol is changed. But sometime different versions do not mean the client & server are not compatible. It would be nice if we could relax this restriction and allows us to support inter-version communications.
> My idea is that DfsClient catches VersionMismatched exception when it connects to NameNode. It then checks if the client & the server is compatible. If yes, it sets the NameNode version in the dfs client and allows the client to continue talking to NameNode. Otherwise, rethrow the VersionMismatch exception.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10053-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 21:49:40 2010
Return-Path: <common-issues-return-10053-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14561 invoked from network); 6 Aug 2010 21:49:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 21:49:40 -0000
Received: (qmail 32078 invoked by uid 500); 6 Aug 2010 21:49:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31935 invoked by uid 500); 6 Aug 2010 21:49:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31927 invoked by uid 99); 6 Aug 2010 21:49:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 21:49:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 21:49:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o76LnJHR012455
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 21:49:19 GMT
Message-ID: <14838056.206041281131359380.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 17:49:19 -0400 (EDT)
From: "Nicholas Carlini (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicholas Carlini updated HADOOP-6837:
-------------------------------------

    Attachment: HADOOP-6837-lzma-2-20100806.patch

I found a bug where Java compression would set a very, very wrong dictionary size. Instead of setting the dictionary size to, say, 2^24, it would set it to 24 (which would then be forced up to the min size, but still, very very wrong).

I also added a fairly long package.html (~3000 words) with documentation about how what I did works, so anyone else who wants to modify it hopefully won't need to spend forever exploring the code again figuring out how it works.

And also, I was both right and wrong about giveMore(). I was right when I first wrote it, and wrong when I said I fixed it by using the return value. The return values were actually left over from when I was checking for the end of stream in the Java end, but I realized that it was possible (because of the semi-circular buffer) for Java to indicate an EOF but for it not to be really true. So I had moved that check to the C code and just never removed that code from that Java end.

Fixed the linked list stuff.

Also a fairly significant directory restructure. The modified SDK code now is in src/contrib/SevenZip. Java code under src/java and C code is under src/native. I removed all of my re-formatting of their code so should a future version of the SDK be released, it shouldn't be as hard to do a diff and apply the patch to make this code better. The makefile from there builds to the same build tree as otherwise for java.

In order to get it building correctly, I had to modify the base build.xml and the contrib/build.xml. compile-core-classes now also depends on compile-contrib-before. compile-contrib-before now calls compile-before on contrib/build.xml. From there contrib/build.xml calls compile-before on contrib/*/build.xml, with failonerror set to false so this change will not break any other build scripts. (This change is required because Lzma{Input,Output}Stream stream requires the classes to be built first.)

I also cleaned up the code and fixed all the review comments.

There will be at least one more version of this patch for things I didn't catch.

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-1-20100722.non-trivial.pseudo-patch, HADOOP-6837-lzma-1-20100722.patch, HADOOP-6837-lzma-2-20100806.patch, HADOOP-6837-lzma-c-20100719.patch, HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10054-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 21:55:40 2010
Return-Path: <common-issues-return-10054-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15380 invoked from network); 6 Aug 2010 21:55:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 21:55:40 -0000
Received: (qmail 40418 invoked by uid 500); 6 Aug 2010 21:55:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40315 invoked by uid 500); 6 Aug 2010 21:55:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40307 invoked by uid 99); 6 Aug 2010 21:55:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 21:55:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 21:55:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o76LtIYX012512
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 21:55:18 GMT
Message-ID: <4266434.206171281131718110.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 17:55:18 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6904) A baby step towards inter-version
 communications between dfs client and NameNode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6904:
----------------------------------

    Attachment: rpcVersion.patch

Here is a patch for review.

1. It introduces a new exception VersionIncompatible means that the server is not able to serve the client because they are not compatible, while VersionMismatch only means that the client and server have different versions and the server could not determine if the client is compatible with the server or not.

2. It adds a set of unit tests to illustrate how client & server can talk across-versions.

Other changes are trivial.

> A baby step towards inter-version communications between dfs client and NameNode
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6904
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6904
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: rpcVersion.patch
>
>
> Currently RPC communications in Hadoop is very strict. If a client has a different version from that of the server, a VersionMismatched exception is thrown and the client can not connect to the server. This force us to update both client and server all at once if a RPC protocol is changed. But sometime different versions do not mean the client & server are not compatible. It would be nice if we could relax this restriction and allows us to support inter-version communications.
> My idea is that DfsClient catches VersionMismatched exception when it connects to NameNode. It then checks if the client & the server is compatible. If yes, it sets the NameNode version in the dfs client and allows the client to continue talking to NameNode. Otherwise, rethrow the VersionMismatch exception.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10055-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 22:36:40 2010
Return-Path: <common-issues-return-10055-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32436 invoked from network); 6 Aug 2010 22:36:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 22:36:40 -0000
Received: (qmail 76397 invoked by uid 500); 6 Aug 2010 22:36:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76313 invoked by uid 500); 6 Aug 2010 22:36:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76305 invoked by uid 99); 6 Aug 2010 22:36:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 22:36:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 22:36:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o76MaJ7k012852
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 22:36:19 GMT
Message-ID: <20708287.206621281134179010.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 18:36:19 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6904) A baby step towards inter-version
 communications between dfs client and NameNode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896178#action_12896178 ] 

Doug Cutting commented on HADOOP-6904:
--------------------------------------

I think the long-term plan is to try to avoid version incompatibilities.  With Avro-based RPC (HADOOP-6659) we could:
 - handle the addition of new parameters and fields with reasonable default values
 - handle changes to method semantics by adding new methods with the new semantics (eventually perhaps removing the old methods).

We'd still need to be careful to design and test for back-compatibility, and only, e.g., make sure new parameters and fields have good defaults, add new methods when semantics change, and only remove an old method in a major release where we're willing to lose some back-compatibility.  This approach might not need an explicit check of which versions are compatible, since Avro could detect the missing method.

Unfortunately work on HADOOP-6659 has been stopped by a veto.

The approach you propose here could be useful until work on HADOOP-6659 is resumed, and may be even useful afterwards, as it might permit more concise incompatibility errors.  It's not a very large or complex change, so I don't see it creating a lot of risk or adding significant maintenance costs.

> A baby step towards inter-version communications between dfs client and NameNode
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6904
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6904
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: rpcVersion.patch
>
>
> Currently RPC communications in Hadoop is very strict. If a client has a different version from that of the server, a VersionMismatched exception is thrown and the client can not connect to the server. This force us to update both client and server all at once if a RPC protocol is changed. But sometime different versions do not mean the client & server are not compatible. It would be nice if we could relax this restriction and allows us to support inter-version communications.
> My idea is that DfsClient catches VersionMismatched exception when it connects to NameNode. It then checks if the client & the server is compatible. If yes, it sets the NameNode version in the dfs client and allows the client to continue talking to NameNode. Otherwise, rethrow the VersionMismatch exception.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10056-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 23:14:38 2010
Return-Path: <common-issues-return-10056-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48166 invoked from network); 6 Aug 2010 23:14:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 23:14:38 -0000
Received: (qmail 6155 invoked by uid 500); 6 Aug 2010 23:14:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6086 invoked by uid 500); 6 Aug 2010 23:14:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6078 invoked by uid 99); 6 Aug 2010 23:14:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 23:14:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 23:14:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o76NEHqe013225
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 23:14:17 GMT
Message-ID: <8035666.207121281136457056.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 19:14:17 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6899) RawLocalFileSystem#setWorkingDir()
 does not work for relative names
In-Reply-To: <23450587.169791280964556270.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6899:
------------------------------------

    Attachment: 6899.y20.patch

y20 version of the patch

> RawLocalFileSystem#setWorkingDir() does not work for relative names
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6899
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6899
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.2
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>             Fix For: 0.22.0
>
>         Attachments: 6899-2.patch, 6899-3.patch, 6899-4.patch, 6899-5.patch, 6899.patch, 6899.y20.patch
>
>
> RawLocalFileSystem#setWorkingDir() does not work for relative names

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10057-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 23:19:41 2010
Return-Path: <common-issues-return-10057-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49347 invoked from network); 6 Aug 2010 23:19:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 23:19:41 -0000
Received: (qmail 11696 invoked by uid 500); 6 Aug 2010 23:19:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11663 invoked by uid 500); 6 Aug 2010 23:19:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11655 invoked by uid 99); 6 Aug 2010 23:19:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 23:19:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 23:19:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o76NJGP7013246
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 23:19:16 GMT
Message-ID: <12774985.207161281136756838.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 19:19:16 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6905) Better logging messages when a
 delegation token is invalid
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Better logging messages when a delegation token is invalid
----------------------------------------------------------

                 Key: HADOOP-6905
                 URL: https://issues.apache.org/jira/browse/HADOOP-6905
             Project: Hadoop Common
          Issue Type: Improvement
          Components: security
            Reporter: Kan Zhang
            Assignee: Kan Zhang


>From our production logs, we see some logging messages of "token is expired or doesn't exist". It would be helpful to know whose token it was.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10058-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 23:26:39 2010
Return-Path: <common-issues-return-10058-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52360 invoked from network); 6 Aug 2010 23:26:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 23:26:39 -0000
Received: (qmail 14725 invoked by uid 500); 6 Aug 2010 23:26:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14654 invoked by uid 500); 6 Aug 2010 23:26:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14646 invoked by uid 99); 6 Aug 2010 23:26:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 23:26:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 23:26:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o76NQHcd013300
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 23:26:17 GMT
Message-ID: <1302813.207311281137177705.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 19:26:17 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6905) Better logging messages when a
 delegation token is invalid
In-Reply-To: <12774985.207161281136756838.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896186#action_12896186 ] 

Allen Wittenauer commented on HADOOP-6905:
------------------------------------------

Maybe you can fix some of the other issues from HADOOP-6586 while you're at it.

> Better logging messages when a delegation token is invalid
> ----------------------------------------------------------
>
>                 Key: HADOOP-6905
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6905
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>
> From our production logs, we see some logging messages of "token is expired or doesn't exist". It would be helpful to know whose token it was.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10059-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 23:37:42 2010
Return-Path: <common-issues-return-10059-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55870 invoked from network); 6 Aug 2010 23:37:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 23:37:42 -0000
Received: (qmail 24011 invoked by uid 500); 6 Aug 2010 23:37:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23962 invoked by uid 500); 6 Aug 2010 23:37:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23954 invoked by uid 99); 6 Aug 2010 23:37:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 23:37:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 23:37:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o76NbHPn013343
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 23:37:17 GMT
Message-ID: <25870104.207381281137837062.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 19:37:17 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6905) Better logging messages when a
 delegation token is invalid
In-Reply-To: <12774985.207161281136756838.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6905:
------------------------------

    Attachment: c6905-01.patch

attaching a trivial patch.

> Better logging messages when a delegation token is invalid
> ----------------------------------------------------------
>
>                 Key: HADOOP-6905
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6905
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6905-01.patch
>
>
> From our production logs, we see some logging messages of "token is expired or doesn't exist". It would be helpful to know whose token it was.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10060-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 23:39:39 2010
Return-Path: <common-issues-return-10060-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57166 invoked from network); 6 Aug 2010 23:39:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 23:39:39 -0000
Received: (qmail 29436 invoked by uid 500); 6 Aug 2010 23:39:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29281 invoked by uid 500); 6 Aug 2010 23:39:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29273 invoked by uid 99); 6 Aug 2010 23:39:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 23:39:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 23:39:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o76NdHbo013389
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 23:39:17 GMT
Message-ID: <12989788.207511281137957729.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 19:39:17 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6905) Better logging messages when a
 delegation token is invalid
In-Reply-To: <12774985.207161281136756838.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6905:
------------------------------

    Status: Patch Available  (was: Open)

> Better logging messages when a delegation token is invalid
> ----------------------------------------------------------
>
>                 Key: HADOOP-6905
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6905
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6905-01.patch
>
>
> From our production logs, we see some logging messages of "token is expired or doesn't exist". It would be helpful to know whose token it was.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10061-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 23:46:41 2010
Return-Path: <common-issues-return-10061-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58145 invoked from network); 6 Aug 2010 23:46:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 23:46:41 -0000
Received: (qmail 32056 invoked by uid 500); 6 Aug 2010 23:46:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31995 invoked by uid 500); 6 Aug 2010 23:46:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31987 invoked by uid 99); 6 Aug 2010 23:46:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 23:46:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 23:46:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o76NkGxD013417
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 23:46:17 GMT
Message-ID: <15704451.207551281138376934.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 19:46:16 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6905) Better logging messages when a
 delegation token is invalid
In-Reply-To: <12774985.207161281136756838.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896191#action_12896191 ] 

Kan Zhang commented on HADOOP-6905:
-----------------------------------

Allen, I think that's a different issue (format rather than content), and I suggest you open a new JIRA so that you can make the format consistent in all the places you care.

> Better logging messages when a delegation token is invalid
> ----------------------------------------------------------
>
>                 Key: HADOOP-6905
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6905
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6905-01.patch
>
>
> From our production logs, we see some logging messages of "token is expired or doesn't exist". It would be helpful to know whose token it was.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10062-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 23:48:41 2010
Return-Path: <common-issues-return-10062-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58359 invoked from network); 6 Aug 2010 23:48:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 23:48:41 -0000
Received: (qmail 34377 invoked by uid 500); 6 Aug 2010 23:48:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34308 invoked by uid 500); 6 Aug 2010 23:48:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34296 invoked by uid 99); 6 Aug 2010 23:48:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 23:48:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 23:48:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o76NmJum013435
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 23:48:19 GMT
Message-ID: <9226736.207611281138499083.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 19:48:19 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6863) Compile-native still fails on Mac
 OS X.
In-Reply-To: <18053660.404611279230589983.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896192#action_12896192 ] 

Hong Tang commented on HADOOP-6863:
-----------------------------------

@allen, do you have objections of having this patch committed?

> Compile-native still fails on Mac OS X.
> ---------------------------------------
>
>                 Key: HADOOP-6863
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6863
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>         Attachments: hadoop-6863-20100715.patch, hadoop-6863-20100721.patch
>
>
> I am still getting failures when I try to compile native library on Mac OS X (10.5.8 Leopard). The problems are two fold:
> - Although "aclocal.m4" is changed after HADOOP-3659 is committed, the corresponding "configure" script is not re-generated.
> - It cannot find libjvm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10063-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 23:52:39 2010
Return-Path: <common-issues-return-10063-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58600 invoked from network); 6 Aug 2010 23:52:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 23:52:39 -0000
Received: (qmail 37507 invoked by uid 500); 6 Aug 2010 23:52:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37364 invoked by uid 500); 6 Aug 2010 23:52:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37356 invoked by uid 99); 6 Aug 2010 23:52:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 23:52:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 23:52:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o76NqIpr013470
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 23:52:18 GMT
Message-ID: <14331953.207671281138738271.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 19:52:18 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6863) Compile-native still fails on Mac
 OS X.
In-Reply-To: <18053660.404611279230589983.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896193#action_12896193 ] 

Allen Wittenauer commented on HADOOP-6863:
------------------------------------------

Right now, yes, because it needs to be tested on a bi-arch jvm.

> Compile-native still fails on Mac OS X.
> ---------------------------------------
>
>                 Key: HADOOP-6863
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6863
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>         Attachments: hadoop-6863-20100715.patch, hadoop-6863-20100721.patch
>
>
> I am still getting failures when I try to compile native library on Mac OS X (10.5.8 Leopard). The problems are two fold:
> - Although "aclocal.m4" is changed after HADOOP-3659 is committed, the corresponding "configure" script is not re-generated.
> - It cannot find libjvm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10064-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 06 23:59:42 2010
Return-Path: <common-issues-return-10064-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61857 invoked from network); 6 Aug 2010 23:59:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Aug 2010 23:59:42 -0000
Received: (qmail 45023 invoked by uid 500); 6 Aug 2010 23:59:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44976 invoked by uid 500); 6 Aug 2010 23:59:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44968 invoked by uid 99); 6 Aug 2010 23:59:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 23:59:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Aug 2010 23:59:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o76NxHKB013518
	for <common-issues@hadoop.apache.org>; Fri, 6 Aug 2010 23:59:17 GMT
Message-ID: <20430404.207751281139157725.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 19:59:17 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6863) Compile-native still fails on Mac
 OS X.
In-Reply-To: <18053660.404611279230589983.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896194#action_12896194 ] 

Hong Tang commented on HADOOP-6863:
-----------------------------------

bq. Right now, yes, because it needs to be tested on a bi-arch jvm.
I have tested the patch for both Linux and Mac OS X. Would you help test on other archs?

> Compile-native still fails on Mac OS X.
> ---------------------------------------
>
>                 Key: HADOOP-6863
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6863
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>         Attachments: hadoop-6863-20100715.patch, hadoop-6863-20100721.patch
>
>
> I am still getting failures when I try to compile native library on Mac OS X (10.5.8 Leopard). The problems are two fold:
> - Although "aclocal.m4" is changed after HADOOP-3659 is committed, the corresponding "configure" script is not re-generated.
> - It cannot find libjvm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10065-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 07 03:25:45 2010
Return-Path: <common-issues-return-10065-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28169 invoked from network); 7 Aug 2010 03:25:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Aug 2010 03:25:45 -0000
Received: (qmail 76480 invoked by uid 500); 7 Aug 2010 03:25:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76309 invoked by uid 500); 7 Aug 2010 03:25:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76301 invoked by uid 99); 7 Aug 2010 03:25:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 Aug 2010 03:25:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 Aug 2010 03:25:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o773PJnN014480
	for <common-issues@hadoop.apache.org>; Sat, 7 Aug 2010 03:25:20 GMT
Message-ID: <22252034.208931281151519948.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 23:25:19 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896213#action_12896213 ] 

Greg Roelofs commented on HADOOP-6837:
--------------------------------------

20100806 version looks pretty good.  The first couple of issues are the main ones:

 * FakeOutputStream LinkedList of 1 KB buffers (BLOCKSIZE = 1024):  supposed to be 128 KB buffers (package.html)
 * directory structure = unholy mix of 7zip, Hadoop  (expected directory structure similar to LZMA SDK tarball contents, e.g.,  lzma912/Java/SevenZip/Compression/LZMA/Decoder.java and lzma912/C/LzFind.c)
   ** OK to trim some of tarball's levels out (and to be different for Java, C), and top level need not be "lzma912" (though it makes connection to download more obvious)--I know I suggested "SevenZip", but I thought that appeared in both the Java and the C paths:  suggest lzma912 or lzma-9.12 instead
 * kDefaultDictionaryLogSize no longer changed to 16:  OK?
 * apparently bogus files:
   ** src/contrib/SevenZip/ivy/libraries.properties
   ** src/contrib/ec2/bin/hadoop-ec2-env.sh
 * "LZMA SDK" -> "LZMA SDK 9.12" in all boilerplate
 * CRC:  prefer to reuse existing (e.g., PureJavaCrc32); should be compatible
 * LzmaNativeInputStream.java:
   ** "circularwould"
   ** read():  fast busy-loop not thread-friendly...and not necessary:  read(b[]) (InputStream) blocks until at least 1 byte available--zero returned only if b[] has length zero, which is not true of oneByte[]
   ** read():  t unnecessary; just do:  return (int)oneByte[0] & 0xFF;
     or even:  return (ret == -1)? -1 : (int)oneByte[0] & 0xFF;
   ** 1<<13
 * LzmaNativeOutputStream.java:
   ** buffered, sendData -> compressedDirectBuf, uncompressedDirectBuf as above
   ** 1<<16
 * LzmaOutputStream.java:
   ** 1<<17
 * Makefile.am
   ** "All these are setup" -> "All these are set up"
   ** "are also done" -> "is also done"

(Per previous feedback, please change all "1<<N" to the Java equivalent of static const int kSomeBufferSize = M * 1024:  easier to read, easier to change later.)

Btw, it's always best to follow the existing style consistently than to use your own for your changes (with the possible exception of the boilerplate comment).  Perhaps Emacs hides the issue, but with 8-space tabs, your changes to the contrib LZMA files are a complete mismatch to their style.

Be sure to run "ant javadoc" (and fix any new issues) before the next patch, and give "ant test" a shot, too (over the weekend if you happen to see this--it takes several hours to run).  I'll work with you to get "ant test-patch" going.


> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-1-20100722.non-trivial.pseudo-patch, HADOOP-6837-lzma-1-20100722.patch, HADOOP-6837-lzma-2-20100806.patch, HADOOP-6837-lzma-c-20100719.patch, HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10066-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 07 03:27:44 2010
Return-Path: <common-issues-return-10066-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30291 invoked from network); 7 Aug 2010 03:27:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Aug 2010 03:27:43 -0000
Received: (qmail 77819 invoked by uid 500); 7 Aug 2010 03:27:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77667 invoked by uid 500); 7 Aug 2010 03:27:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77659 invoked by uid 99); 7 Aug 2010 03:27:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 Aug 2010 03:27:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 Aug 2010 03:27:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o773RJcn014520
	for <common-issues@hadoop.apache.org>; Sat, 7 Aug 2010 03:27:19 GMT
Message-ID: <2775934.209051281151639681.JavaMail.jira@thor>
Date: Fri, 6 Aug 2010 23:27:19 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896214#action_12896214 ] 

Greg Roelofs commented on HADOOP-6837:
--------------------------------------

On second thought, _if_ you put the version into the src/contrib path as I suggested, there's no need to add it to the boilerplate text, too.  That will make future forward-ports simpler (i.e., they can use the same boilerplate text).

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-1-20100722.non-trivial.pseudo-patch, HADOOP-6837-lzma-1-20100722.patch, HADOOP-6837-lzma-2-20100806.patch, HADOOP-6837-lzma-c-20100719.patch, HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10067-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 07 05:47:47 2010
Return-Path: <common-issues-return-10067-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76063 invoked from network); 7 Aug 2010 05:47:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Aug 2010 05:47:46 -0000
Received: (qmail 15802 invoked by uid 500); 7 Aug 2010 05:47:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15650 invoked by uid 500); 7 Aug 2010 05:47:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15640 invoked by uid 99); 7 Aug 2010 05:47:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 Aug 2010 05:47:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 Aug 2010 05:47:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o775lJPD016074
	for <common-issues@hadoop.apache.org>; Sat, 7 Aug 2010 05:47:19 GMT
Message-ID: <11837050.211281281160039000.JavaMail.jira@thor>
Date: Sat, 7 Aug 2010 01:47:18 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6904) A baby step towards inter-version
 communications between dfs client and NameNode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6904:
----------------------------------

    Attachment: rpcVersion1.patch

This patch changed a comment.

> A baby step towards inter-version communications between dfs client and NameNode
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6904
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6904
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: rpcVersion.patch, rpcVersion1.patch
>
>
> Currently RPC communications in Hadoop is very strict. If a client has a different version from that of the server, a VersionMismatched exception is thrown and the client can not connect to the server. This force us to update both client and server all at once if a RPC protocol is changed. But sometime different versions do not mean the client & server are not compatible. It would be nice if we could relax this restriction and allows us to support inter-version communications.
> My idea is that DfsClient catches VersionMismatched exception when it connects to NameNode. It then checks if the client & the server is compatible. If yes, it sets the NameNode version in the dfs client and allows the client to continue talking to NameNode. Otherwise, rethrow the VersionMismatch exception.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10068-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 07 05:47:49 2010
Return-Path: <common-issues-return-10068-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76095 invoked from network); 7 Aug 2010 05:47:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Aug 2010 05:47:49 -0000
Received: (qmail 15858 invoked by uid 500); 7 Aug 2010 05:47:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15827 invoked by uid 500); 7 Aug 2010 05:47:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15819 invoked by uid 99); 7 Aug 2010 05:47:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 Aug 2010 05:47:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 Aug 2010 05:47:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o775lR4f016170
	for <common-issues@hadoop.apache.org>; Sat, 7 Aug 2010 05:47:27 GMT
Message-ID: <26097595.211601281160047255.JavaMail.jira@thor>
Date: Sat, 7 Aug 2010 01:47:27 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6349) Implement FastLZCodec for
 fastlz/lzo algorithm
In-Reply-To: <289817036.1256941439448.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896222#action_12896222 ] 

Greg Roelofs commented on HADOOP-6349:
--------------------------------------

OK, with respect to the 20100730 patch, the biggest issue by far is the buffering, both compression and decompression, as we've discussed in person:

 * since FastLZ is block-based with an uncompressed block size of 64 KB, it should never require a buffer larger than 64 KB for either input or output, and certainly never the entirety of the file
   ** it sounds like a (relatively) quick fix is possible, but otherwise it might be worth looking at SplittableCompressionCodec and/or BlockCompressorStream, per comments below

The other items are mostly style-related:

 * move imported code to src/contrib, per offline discussion with Owen O'Malley
 * boilerplate should probably mention original MIT license
 * no src/java/core-default.xml change? (add to io.compression.codecs)
 * [cosmetic: nuke commented-out System.currentTimeMillis(), debug printfs, etc.]
 * [cosmetic: strip all trailing blanks on new, non-3rdparty/imported (i.e., non-src/contrib) files]
 * prefer to drop "J" prefix on fastlz/*.java (consistent with C, other codecs)
 * add package.html
   ** 64 KB block size
   ** mainly level 2 with fallback to level 1 for partial blocks
   ** weird triple-self-inclusion of fastlz.c
   ** possible future speed/etc. improvements:
     *** use existing Adler-32 in java.util.zip or zlib or wherever?
     *** parallel Adler-32 (and CRC-32) capability in zlib, pigz
 * FastLZCodec.java
   ** originally was going to suggest that it should extend DefaultCodec (like GzipCodec) rather than copy it, but as a block-based codec, Splittable*Codec is better: we'll file separate JIRA for splittable support later (also/alternatively, per Arun Murthy, BlockCompressorStream?)
 * JFastLZ.java
   ** DEFAULT_ARCHIVE_EXTENSION not used
   ** readU32() BigInteger version should be nuked in favor of (fixed) readU32Fast() (not used much in any case, but...ewww)
   ** why isn't all of "Moved." block (line 686) commented out/deleted?
   ** invert "Never runs b/c" block so true path is first
     *** does !FASTLZ_STRICT_ALIGN option even work? if not, nuke altogether
   ** no need for double parens on while condition (line 768)
 * JFastLZCompressor.java
   ** lose empty else in static block
   ** "v. low" -> "very low", "b/c" -> "because", etc. (in general, avoid abbreviations in comments; English isn't everyone's primary language)
   ** had hoped b/iBuff arraycopy ("so CompressorStream doesn't muck w/ our data") was unnecessary, but doesn't look like it: b[] comes straight from user via write() call
   ** "XXX" ?
   ** growOutputBuffer() used for apparently tiny deltas: should simply double size (unless strong statistical or theoretical arg why small increments are rare) [this is subsumed by whole fix-the-buffering issue above]
   ** use "L" for long-constant appendage: lowercase looks too much like "1"
   ** [cosmetic: need spaces on outside of for-loop and if-conditional parens]
   ** why isn't "if (totalCompressed == 26)" block (line 356) deleted?
   ** "enought"
   ** looks like native checksum call is wasted if incompressible data: should be moved down to other half of "if (!nativeLoaded)" conditional (line 416) (more symmetric that way, too)
   ** [cosmetic: prefer variables at top of class (SAFE_PACK)]
 * JFastLZDecompressor.java
   ** [cosmetic: wrap long lines (native methods)]
   ** lose empty else in static block
   ** "v." -> "very", etc. (as above)
   ** DEFAULT_BLOCK_SIZE hardcoded at 64K ... configured value completely ignored (OK, probably related to "native" FastLZ block size...never mind)

Btw, this one is clearly much, much simpler than the LZMA patch, so if it comes down to one or the other, please concentrate on LZMA. If necessary, I (or someone) can pick this up and finish it without much trouble, I think.

> Implement FastLZCodec for fastlz/lzo algorithm
> ----------------------------------------------
>
>                 Key: HADOOP-6349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6349
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: William Kinney
>         Attachments: hadoop-6349-1.patch, hadoop-6349-2.patch, hadoop-6349-3.patch, HADOOP-6349-TestFastLZCodec.patch, HADOOP-6349.patch, TestCodecPerformance.java, TestCodecPerformance.java, testCodecPerfResults.tsv
>
>
> Per  [HADOOP-4874|http://issues.apache.org/jira/browse/HADOOP-4874], FastLZ is a good (speed, license) alternative to LZO. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10069-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 08 17:30:44 2010
Return-Path: <common-issues-return-10069-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5223 invoked from network); 8 Aug 2010 17:30:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Aug 2010 17:30:44 -0000
Received: (qmail 54856 invoked by uid 500); 8 Aug 2010 17:30:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54768 invoked by uid 500); 8 Aug 2010 17:30:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54760 invoked by uid 99); 8 Aug 2010 17:30:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 Aug 2010 17:30:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 Aug 2010 17:30:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o78HUJQg002845
	for <common-issues@hadoop.apache.org>; Sun, 8 Aug 2010 17:30:19 GMT
Message-ID: <22362779.224341281288619134.JavaMail.jira@thor>
Date: Sun, 8 Aug 2010 13:30:19 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6904) A baby step towards inter-version
 communications between dfs client and NameNode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6904:
----------------------------------

    Attachment:     (was: rpcVersion1.patch)

> A baby step towards inter-version communications between dfs client and NameNode
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6904
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6904
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: rpcVersion.patch
>
>
> Currently RPC communications in Hadoop is very strict. If a client has a different version from that of the server, a VersionMismatched exception is thrown and the client can not connect to the server. This force us to update both client and server all at once if a RPC protocol is changed. But sometime different versions do not mean the client & server are not compatible. It would be nice if we could relax this restriction and allows us to support inter-version communications.
> My idea is that DfsClient catches VersionMismatched exception when it connects to NameNode. It then checks if the client & the server is compatible. If yes, it sets the NameNode version in the dfs client and allows the client to continue talking to NameNode. Otherwise, rethrow the VersionMismatch exception.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10070-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 08 17:32:42 2010
Return-Path: <common-issues-return-10070-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6105 invoked from network); 8 Aug 2010 17:32:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Aug 2010 17:32:42 -0000
Received: (qmail 56944 invoked by uid 500); 8 Aug 2010 17:32:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56833 invoked by uid 500); 8 Aug 2010 17:32:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56825 invoked by uid 99); 8 Aug 2010 17:32:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 Aug 2010 17:32:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 Aug 2010 17:32:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o78HWIYn002879
	for <common-issues@hadoop.apache.org>; Sun, 8 Aug 2010 17:32:18 GMT
Message-ID: <5646390.224451281288738095.JavaMail.jira@thor>
Date: Sun, 8 Aug 2010 13:32:18 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6904) A baby step towards inter-version
 communications between dfs client and NameNode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6904:
----------------------------------

    Attachment: rpcVersion1.patch

> A baby step towards inter-version communications between dfs client and NameNode
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6904
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6904
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: rpcVersion.patch, rpcVersion1.patch
>
>
> Currently RPC communications in Hadoop is very strict. If a client has a different version from that of the server, a VersionMismatched exception is thrown and the client can not connect to the server. This force us to update both client and server all at once if a RPC protocol is changed. But sometime different versions do not mean the client & server are not compatible. It would be nice if we could relax this restriction and allows us to support inter-version communications.
> My idea is that DfsClient catches VersionMismatched exception when it connects to NameNode. It then checks if the client & the server is compatible. If yes, it sets the NameNode version in the dfs client and allows the client to continue talking to NameNode. Otherwise, rethrow the VersionMismatch exception.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10071-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 08 18:04:43 2010
Return-Path: <common-issues-return-10071-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12480 invoked from network); 8 Aug 2010 18:04:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Aug 2010 18:04:43 -0000
Received: (qmail 72343 invoked by uid 500); 8 Aug 2010 18:04:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72205 invoked by uid 500); 8 Aug 2010 18:04:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72197 invoked by uid 99); 8 Aug 2010 18:04:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 Aug 2010 18:04:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 Aug 2010 18:04:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o78I4Kdp003094
	for <common-issues@hadoop.apache.org>; Sun, 8 Aug 2010 18:04:21 GMT
Message-ID: <22200988.224751281290660959.JavaMail.jira@thor>
Date: Sun, 8 Aug 2010 14:04:20 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6904) A baby step towards inter-version
 communications between dfs client and NameNode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896389#action_12896389 ] 

Hairong Kuang commented on HADOOP-6904:
---------------------------------------

Hudson is stuck. I ran tests on my local box. Here are the results:

ant test-patch
     [exec] -1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to i
     [exec] nclude 3 new or modified tests.
     [exec] 
     [exec]     -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.

Javadoc warnings are not related to this patch.

ant test:
BUILD SUCCESSFUL
Total time: 10 minutes 20 seconds

> A baby step towards inter-version communications between dfs client and NameNode
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6904
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6904
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: rpcVersion.patch, rpcVersion1.patch
>
>
> Currently RPC communications in Hadoop is very strict. If a client has a different version from that of the server, a VersionMismatched exception is thrown and the client can not connect to the server. This force us to update both client and server all at once if a RPC protocol is changed. But sometime different versions do not mean the client & server are not compatible. It would be nice if we could relax this restriction and allows us to support inter-version communications.
> My idea is that DfsClient catches VersionMismatched exception when it connects to NameNode. It then checks if the client & the server is compatible. If yes, it sets the NameNode version in the dfs client and allows the client to continue talking to NameNode. Otherwise, rethrow the VersionMismatch exception.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10072-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 09 03:54:44 2010
Return-Path: <common-issues-return-10072-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54527 invoked from network); 9 Aug 2010 03:54:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Aug 2010 03:54:44 -0000
Received: (qmail 87551 invoked by uid 500); 9 Aug 2010 03:54:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87399 invoked by uid 500); 9 Aug 2010 03:54:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87391 invoked by uid 99); 9 Aug 2010 03:54:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 03:54:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 03:54:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o793sJ5t005837
	for <common-issues@hadoop.apache.org>; Mon, 9 Aug 2010 03:54:19 GMT
Message-ID: <6214801.227851281326059477.JavaMail.jira@thor>
Date: Sun, 8 Aug 2010 23:54:19 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6863) Compile-native still fails on Mac
 OS X.
In-Reply-To: <18053660.404611279230589983.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896446#action_12896446 ] 

Allen Wittenauer commented on HADOOP-6863:
------------------------------------------

Java on Snow Leopard is very different than what is on Leopard due it properly supporting the -d flag to flip from 32-bit and 64-bit JVM support.  Last I heard, Yahoo! was still using Leopard.

> Compile-native still fails on Mac OS X.
> ---------------------------------------
>
>                 Key: HADOOP-6863
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6863
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>         Attachments: hadoop-6863-20100715.patch, hadoop-6863-20100721.patch
>
>
> I am still getting failures when I try to compile native library on Mac OS X (10.5.8 Leopard). The problems are two fold:
> - Although "aclocal.m4" is changed after HADOOP-3659 is committed, the corresponding "configure" script is not re-generated.
> - It cannot find libjvm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10073-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 09 04:40:51 2010
Return-Path: <common-issues-return-10073-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69746 invoked from network); 9 Aug 2010 04:40:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Aug 2010 04:40:51 -0000
Received: (qmail 8436 invoked by uid 500); 9 Aug 2010 04:40:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8289 invoked by uid 500); 9 Aug 2010 04:40:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8281 invoked by uid 99); 9 Aug 2010 04:40:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 04:40:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 04:40:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o794eMnO006509
	for <common-issues@hadoop.apache.org>; Mon, 9 Aug 2010 04:40:22 GMT
Message-ID: <24870182.228141281328822652.JavaMail.jira@thor>
Date: Mon, 9 Aug 2010 00:40:22 -0400 (EDT)
From: "Grady Laksmono (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5342) DataNodes do not start up because
 InconsistentFSStateException on just part of the disks in use
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896450#action_12896450 ] 

Grady Laksmono commented on HADOOP-5342:
----------------------------------------

I'm also experiencing this issue, is there a quick solution for now? This happened to me once before, then I reformat the HDFS and it works fine, but this happened again, but this time I have files on my HDFS..


10/08/08 21:33:01 INFO common.Storage: Storage directory /tmp/hadoop-grady/dfs/name does not exist.
10/08/08 21:33:01 ERROR namenode.FSNamesystem: FSNamesystem initialization failed.
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /tmp/hadoop-grady/dfs/name is in an inconsistent state: storage directory does not exist or is not accessible.
	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:290)
	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:292)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:201)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
10/08/08 21:33:01 INFO ipc.Server: Stopping server on 9000
10/08/08 21:33:01 ERROR namenode.NameNode: org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /tmp/hadoop-grady/dfs/name is in an inconsistent state: storage directory does not exist or is not accessible.
	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:290)
	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:292)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:201)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)


> DataNodes do not start up because InconsistentFSStateException on just part of the disks in use
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-5342
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5342
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.18.2
>            Reporter: Christian Kunz
>            Assignee: Hairong Kuang
>            Priority: Critical
>
> After restarting a cluster (including rebooting) the dfs got corrupted because many DataNodes did not start up, running into the following exception:
> 2009-02-26 22:33:53,774 ERROR org.apache.hadoop.dfs.DataNode: org.apache.hadoop.dfs.InconsistentFSStateException: Directory xxx  is in an inconsistent state: version file in current directory is missing.
> 	at org.apache.hadoop.dfs.Storage$StorageDirectory.analyzeStorage(Storage.java:326)
> 	at org.apache.hadoop.dfs.DataStorage.recoverTransitionRead(DataStorage.java:105)
> 	at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:306)
> 	at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:223)
> 	at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:3030)
> 	at org.apache.hadoop.dfs.DataNode.instantiateDataNode(DataNode.java:2985)
> 	at org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:2993)
> 	at org.apache.hadoop.dfs.DataNode.main(DataNode.java:3115)
> This happens when using multiple disks with at least one previously marked as read-only, such that the storage version became out-dated, but after reboot it was mounted read-write, resulting in the DataNode not starting because of out-dated version.
> This is a big headache. If a DataNode has multiple disks of which at least one has the correct storage version then out-dated versions should not bring down the DataNode.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10074-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 09 10:21:45 2010
Return-Path: <common-issues-return-10074-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4424 invoked from network); 9 Aug 2010 10:21:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Aug 2010 10:21:45 -0000
Received: (qmail 22264 invoked by uid 500); 9 Aug 2010 10:21:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22123 invoked by uid 500); 9 Aug 2010 10:21:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22108 invoked by uid 99); 9 Aug 2010 10:21:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 10:21:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 10:21:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o79ALGQx009187
	for <common-issues@hadoop.apache.org>; Mon, 9 Aug 2010 10:21:17 GMT
Message-ID: <31039707.230631281349276933.JavaMail.jira@thor>
Date: Mon, 9 Aug 2010 06:21:16 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6906) FileContext copy() utility doesn't
 work with recursive copying of directories.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

FileContext copy() utility doesn't work with recursive copying of directories.
------------------------------------------------------------------------------

                 Key: HADOOP-6906
                 URL: https://issues.apache.org/jira/browse/HADOOP-6906
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
    Affects Versions: 0.22.0
            Reporter: Vinod K V




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10075-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 09 10:29:40 2010
Return-Path: <common-issues-return-10075-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7354 invoked from network); 9 Aug 2010 10:29:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Aug 2010 10:29:40 -0000
Received: (qmail 29347 invoked by uid 500); 9 Aug 2010 10:29:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29252 invoked by uid 500); 9 Aug 2010 10:29:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29243 invoked by uid 99); 9 Aug 2010 10:29:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 10:29:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 10:29:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o79ATGfg009271
	for <common-issues@hadoop.apache.org>; Mon, 9 Aug 2010 10:29:16 GMT
Message-ID: <31165085.230681281349756274.JavaMail.jira@thor>
Date: Mon, 9 Aug 2010 06:29:16 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6906) FileContext copy() utility doesn't
 work with recursive copying of directories.
In-Reply-To: <31039707.230631281349276933.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V updated HADOOP-6906:
------------------------------

    Attachment: HADOOP-6906-regression.1.txt

Attaching a regression test. Running TestFcLocalFsUtil after applying this patch verifies that a simple recursive copy of a directory from local file system to the same FS fails. I could verify the same failure with HDFS to HDFS copy also.

> FileContext copy() utility doesn't work with recursive copying of directories.
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6906
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6906
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Vinod K V
>         Attachments: HADOOP-6906-regression.1.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10076-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 09 10:29:42 2010
Return-Path: <common-issues-return-10076-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7425 invoked from network); 9 Aug 2010 10:29:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Aug 2010 10:29:42 -0000
Received: (qmail 29444 invoked by uid 500); 9 Aug 2010 10:29:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29430 invoked by uid 500); 9 Aug 2010 10:29:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29422 invoked by uid 99); 9 Aug 2010 10:29:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 10:29:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 10:29:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o79ATHPh009277
	for <common-issues@hadoop.apache.org>; Mon, 9 Aug 2010 10:29:17 GMT
Message-ID: <21007999.230701281349757394.JavaMail.jira@thor>
Date: Mon, 9 Aug 2010 06:29:17 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6906) FileContext copy() utility doesn't
 work with recursive copying of directories.
In-Reply-To: <31039707.230631281349276933.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V reassigned HADOOP-6906:
---------------------------------

    Assignee: Vinod K V

> FileContext copy() utility doesn't work with recursive copying of directories.
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6906
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6906
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Vinod K V
>            Assignee: Vinod K V
>         Attachments: HADOOP-6906-regression.1.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10077-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 09 11:17:41 2010
Return-Path: <common-issues-return-10077-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26897 invoked from network); 9 Aug 2010 11:17:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Aug 2010 11:17:41 -0000
Received: (qmail 65694 invoked by uid 500); 9 Aug 2010 11:17:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65526 invoked by uid 500); 9 Aug 2010 11:17:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65518 invoked by uid 99); 9 Aug 2010 11:17:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 11:17:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 11:17:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o79BHHn4009787
	for <common-issues@hadoop.apache.org>; Mon, 9 Aug 2010 11:17:17 GMT
Message-ID: <13043300.231311281352637222.JavaMail.jira@thor>
Date: Mon, 9 Aug 2010 07:17:17 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6906) FileContext copy() utility doesn't
 work with recursive copying of directories.
In-Reply-To: <31039707.230631281349276933.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V updated HADOOP-6906:
------------------------------

    Attachment: HADOOP-6906-20100809.1.txt

Attaching a patch to address this issue. Please review this patch. It does the following:
 - Fixes the main bug which incorrectly constructs path names of contents of a directory while copying (FileContext.java +2012 with the patch)
 - Modifies {{FileContext.isSameFS()}} to avoid null errors when using file:/// URIs which don't have an authority part.
 - LocalFs extends ChecksumFs and *has* a RawLocalFs object. RawLocalFs is backed by RawLocalFileSystem which doesn't know about .crc files. Because of this, files written via LocalFs automatically write .crc files transparently, but listing of files in a LocalFs doesn't hide .crc files. The modified test fails because of this. I changed ChecksumFs to override listStatus() and swallow .crc files. Please correct me if I am wrong.

> FileContext copy() utility doesn't work with recursive copying of directories.
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6906
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6906
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Vinod K V
>            Assignee: Vinod K V
>         Attachments: HADOOP-6906-20100809.1.txt, HADOOP-6906-regression.1.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10078-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 09 11:19:40 2010
Return-Path: <common-issues-return-10078-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27564 invoked from network); 9 Aug 2010 11:19:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Aug 2010 11:19:40 -0000
Received: (qmail 74258 invoked by uid 500); 9 Aug 2010 11:19:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74104 invoked by uid 500); 9 Aug 2010 11:19:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74095 invoked by uid 99); 9 Aug 2010 11:19:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 11:19:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 11:19:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o79BJGo0009807
	for <common-issues@hadoop.apache.org>; Mon, 9 Aug 2010 11:19:16 GMT
Message-ID: <11885973.231331281352756633.JavaMail.jira@thor>
Date: Mon, 9 Aug 2010 07:19:16 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6906) FileContext copy() utility doesn't
 work with recursive copying of directories.
In-Reply-To: <31039707.230631281349276933.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V updated HADOOP-6906:
------------------------------

    Status: Patch Available  (was: Open)

> FileContext copy() utility doesn't work with recursive copying of directories.
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6906
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6906
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Vinod K V
>            Assignee: Vinod K V
>         Attachments: HADOOP-6906-20100809.1.txt, HADOOP-6906-regression.1.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10079-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 09 18:30:40 2010
Return-Path: <common-issues-return-10079-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 681 invoked from network); 9 Aug 2010 18:30:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Aug 2010 18:30:40 -0000
Received: (qmail 95887 invoked by uid 500); 9 Aug 2010 18:30:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95751 invoked by uid 500); 9 Aug 2010 18:30:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95743 invoked by uid 99); 9 Aug 2010 18:30:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 18:30:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 18:30:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o79IUIUW014699
	for <common-issues@hadoop.apache.org>; Mon, 9 Aug 2010 18:30:18 GMT
Message-ID: <26112314.238351281378618120.JavaMail.jira@thor>
Date: Mon, 9 Aug 2010 14:30:18 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6904) A baby step towards inter-version
 communications between dfs client and NameNode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896656#action_12896656 ] 

Doug Cutting commented on HADOOP-6904:
--------------------------------------

I'd like to have a better understanding of how this will be used (HDFS-1335) before it is committed.

> A baby step towards inter-version communications between dfs client and NameNode
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6904
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6904
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: rpcVersion.patch, rpcVersion1.patch
>
>
> Currently RPC communications in Hadoop is very strict. If a client has a different version from that of the server, a VersionMismatched exception is thrown and the client can not connect to the server. This force us to update both client and server all at once if a RPC protocol is changed. But sometime different versions do not mean the client & server are not compatible. It would be nice if we could relax this restriction and allows us to support inter-version communications.
> My idea is that DfsClient catches VersionMismatched exception when it connects to NameNode. It then checks if the client & the server is compatible. If yes, it sets the NameNode version in the dfs client and allows the client to continue talking to NameNode. Otherwise, rethrow the VersionMismatch exception.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10080-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 09 21:35:42 2010
Return-Path: <common-issues-return-10080-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73019 invoked from network); 9 Aug 2010 21:35:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Aug 2010 21:35:42 -0000
Received: (qmail 78222 invoked by uid 500); 9 Aug 2010 21:35:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78166 invoked by uid 500); 9 Aug 2010 21:35:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78158 invoked by uid 99); 9 Aug 2010 21:35:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 21:35:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 21:35:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o79LZHXI017407
	for <common-issues@hadoop.apache.org>; Mon, 9 Aug 2010 21:35:17 GMT
Message-ID: <31919203.243071281389717457.JavaMail.jira@thor>
Date: Mon, 9 Aug 2010 17:35:17 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6900) FileSystem#listLocatedStatus should
 not throw generic RuntimeException to indicate error conditions
In-Reply-To: <29253908.169821280964676744.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896701#action_12896701 ] 

Hairong Kuang commented on HADOOP-6900:
---------------------------------------

> Additionally I feel when a directory gets deleted in the middle of iteration, we should ignore it, or add provision to the iterator to ignore it.

Unfortunately not allowing concurrent modification on the input directories is the semantics that map/reduce has it now. I do not want to introduce an incompatible change for now.

> FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6900
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6900
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Suresh Srinivas
>            Assignee: Hairong Kuang
>         Attachments: listFilesIO.patch
>
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10081-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 09 21:37:40 2010
Return-Path: <common-issues-return-10081-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73480 invoked from network); 9 Aug 2010 21:37:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Aug 2010 21:37:40 -0000
Received: (qmail 80660 invoked by uid 500); 9 Aug 2010 21:37:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80612 invoked by uid 500); 9 Aug 2010 21:37:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80604 invoked by uid 99); 9 Aug 2010 21:37:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 21:37:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 21:37:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o79LbIxN017426
	for <common-issues@hadoop.apache.org>; Mon, 9 Aug 2010 21:37:19 GMT
Message-ID: <9029429.243131281389838773.JavaMail.jira@thor>
Date: Mon, 9 Aug 2010 17:37:18 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6907) Rpc client doesn't use the
 per-connection conf to figure out server's Kerberos principal
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Rpc client doesn't use the per-connection conf to figure out server's Kerberos principal
----------------------------------------------------------------------------------------

                 Key: HADOOP-6907
                 URL: https://issues.apache.org/jira/browse/HADOOP-6907
             Project: Hadoop Common
          Issue Type: Bug
          Components: ipc, security
            Reporter: Kan Zhang
            Assignee: Kan Zhang


Currently, RPC client caches the conf that was passed in to its constructor and uses that same conf (or values obtained from it) for every connection it sets up. This is not sufficient for security since each connection needs to figure out server's Kerberos principal on a per-connection basis. It's not reasonable to expect the first conf used by a user to contain all the Kerberos principals that her future connections will ever need. Or worse, if her first conf contains an incorrect principal name, it will prevent the user from connecting to the server even if she later on passes in a correct conf on retry (RPC.getProxy()).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10082-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 09 21:39:41 2010
Return-Path: <common-issues-return-10082-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73747 invoked from network); 9 Aug 2010 21:39:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Aug 2010 21:39:41 -0000
Received: (qmail 82426 invoked by uid 500); 9 Aug 2010 21:39:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82388 invoked by uid 500); 9 Aug 2010 21:39:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82380 invoked by uid 99); 9 Aug 2010 21:39:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 21:39:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 21:39:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o79LdH9V017457
	for <common-issues@hadoop.apache.org>; Mon, 9 Aug 2010 21:39:17 GMT
Message-ID: <8260284.243211281389957859.JavaMail.jira@thor>
Date: Mon, 9 Aug 2010 17:39:17 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Erik Steffl updated HADOOP-6884:
--------------------------------

    Attachment: HADOOP-6884-0.22-1.patch

Patch HADOOP-6884-0.22-1.patch adds more changes that were previosuly missed (or new .debug() calls).

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10083-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 09 21:45:45 2010
Return-Path: <common-issues-return-10083-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74770 invoked from network); 9 Aug 2010 21:45:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Aug 2010 21:45:44 -0000
Received: (qmail 88926 invoked by uid 500); 9 Aug 2010 21:45:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88870 invoked by uid 500); 9 Aug 2010 21:45:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88860 invoked by uid 99); 9 Aug 2010 21:45:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 21:45:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 21:45:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o79LjJ8a017653
	for <common-issues@hadoop.apache.org>; Mon, 9 Aug 2010 21:45:20 GMT
Message-ID: <8032486.243761281390319934.JavaMail.jira@thor>
Date: Mon, 9 Aug 2010 17:45:19 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6907) Rpc client doesn't use the
 per-connection conf to figure out server's Kerberos principal
In-Reply-To: <9029429.243131281389838773.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6907?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6907:
------------------------------

    Description: Currently, RPC client caches the conf that was passed in t=
o its constructor and uses that same conf (or values obtained from it) for =
every connection it sets up. This is not sufficient for security since each=
 connection needs to figure out server's Kerberos principal on a per-connec=
tion basis. It's not reasonable to expect the first conf used by a user to =
contain all the Kerberos principals that her future connections will ever n=
eed. Or worse, if her first conf contains an incorrect principal name, it w=
ill prevent the user from connecting to the server even if she later on pas=
ses in a correct conf on retry (by calling RPC.getProxy()).  (was: Currentl=
y, RPC client caches the conf that was passed in to its constructor and use=
s that same conf (or values obtained from it) for every connection it sets =
up. This is not sufficient for security since each connection needs to figu=
re out server's Kerberos principal on a per-connection basis. It's not reas=
onable to expect the first conf used by a user to contain all the Kerberos =
principals that her future connections will ever need. Or worse, if her fir=
st conf contains an incorrect principal name, it will prevent the user from=
 connecting to the server even if she later on passes in a correct conf on =
retry (RPC.getProxy()).)

> Rpc client doesn't use the per-connection conf to figure out server's Ker=
beros principal
> -------------------------------------------------------------------------=
---------------
>
>                 Key: HADOOP-6907
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6907
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>
> Currently, RPC client caches the conf that was passed in to its construct=
or and uses that same conf (or values obtained from it) for every connectio=
n it sets up. This is not sufficient for security since each connection nee=
ds to figure out server's Kerberos principal on a per-connection basis. It'=
s not reasonable to expect the first conf used by a user to contain all the=
 Kerberos principals that her future connections will ever need. Or worse, =
if her first conf contains an incorrect principal name, it will prevent the=
 user from connecting to the server even if she later on passes in a correc=
t conf on retry (by calling RPC.getProxy()).

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10084-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 09 22:16:38 2010
Return-Path: <common-issues-return-10084-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85866 invoked from network); 9 Aug 2010 22:16:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Aug 2010 22:16:38 -0000
Received: (qmail 16756 invoked by uid 500); 9 Aug 2010 22:16:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16717 invoked by uid 500); 9 Aug 2010 22:16:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16705 invoked by uid 99); 9 Aug 2010 22:16:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 22:16:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 22:16:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o79MGGs2017899
	for <common-issues@hadoop.apache.org>; Mon, 9 Aug 2010 22:16:16 GMT
Message-ID: <3099972.244001281392176377.JavaMail.jira@thor>
Date: Mon, 9 Aug 2010 18:16:16 -0400 (EDT)
From: "Mathias Gug (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6908) Provide a "-y" option that skips the
 confirmation question of the namenode for use in scripts.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Provide a "-y" option that skips the confirmation question of the namenode for use in scripts.
----------------------------------------------------------------------------------------------

                 Key: HADOOP-6908
                 URL: https://issues.apache.org/jira/browse/HADOOP-6908
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Mathias Gug


As suggested in [HDFS 718|https://issues.apache.org/jira/browse/HDFS-718?focusedCommentId=12776315&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12776315] having an option to skip the confirmation question when formating a NameNode would prove to be very useful when packaging hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10085-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 09 22:21:45 2010
Return-Path: <common-issues-return-10085-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86888 invoked from network); 9 Aug 2010 22:21:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Aug 2010 22:21:45 -0000
Received: (qmail 22450 invoked by uid 500); 9 Aug 2010 22:21:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22422 invoked by uid 500); 9 Aug 2010 22:21:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22414 invoked by uid 99); 9 Aug 2010 22:21:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 22:21:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 22:21:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o79MLKra017951
	for <common-issues@hadoop.apache.org>; Mon, 9 Aug 2010 22:21:20 GMT
Message-ID: <14154807.244131281392480309.JavaMail.jira@thor>
Date: Mon, 9 Aug 2010 18:21:20 -0400 (EDT)
From: "Nicholas Carlini (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicholas Carlini updated HADOOP-6837:
-------------------------------------

    Attachment: HADOOP-6837-lzma-3-20100809.patch

I'm going to respond to each one this time to make sure I don't miss anything again...

FakeOutputStream isn't the one I'm talking about in package.html. That's for the OutputStream/FakeInputStream. FakeOutputStream is just the one where I couldn't justify the maximum acting correctly (wrtiting a max of 273 bytes extra) so I added the linked list in case anything goes wrong.

Changed directory structure as suggested -- src/contrib/lzma912/src/SevenZip/* has the java files and src/contrib/lzma912/src/native has the native code.

The default dictionary size doesn't matter, because it gets set to something else on initialization. I left it there to keep the minimum set of changes.

I needed library.properties there for it to build correctly: I have no idea what it does, but if I delete it then it doesn't build. And when it's empty it doesn't complain, so that's why it's there.

Sorry about the ec2 file; I didn't realize that went into the patch! It's not supposed to be there.

Deleted CRC.java ... apparently it was never used.

Removed the read() while loop and wrote as suggested.

Fixed names (buffered/sendData) again, I had fixed them but then reverted to an older version when I introduced a bug.

Removed 1<<x's to y*1024.

Makefile.am, fixed grammar.

Yeah, sorry. I didn't realize that happened with the off-indentation. Fixed it.

I'll talk to you about ant test-patch. I'm running ant test right now and if anything goes wrong I'll push another patch. ant javadoc has no warnings.

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-1-20100722.non-trivial.pseudo-patch, HADOOP-6837-lzma-1-20100722.patch, HADOOP-6837-lzma-2-20100806.patch, HADOOP-6837-lzma-3-20100809.patch, HADOOP-6837-lzma-c-20100719.patch, HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10086-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 09 23:29:42 2010
Return-Path: <common-issues-return-10086-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14661 invoked from network); 9 Aug 2010 23:29:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Aug 2010 23:29:42 -0000
Received: (qmail 63400 invoked by uid 500); 9 Aug 2010 23:29:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63338 invoked by uid 500); 9 Aug 2010 23:29:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63325 invoked by uid 99); 9 Aug 2010 23:29:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 23:29:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 23:29:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o79NTHW1018478
	for <common-issues@hadoop.apache.org>; Mon, 9 Aug 2010 23:29:17 GMT
Message-ID: <19317095.245081281396557626.JavaMail.jira@thor>
Date: Mon, 9 Aug 2010 19:29:17 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6863) Compile-native still fails on Mac
 OS X.
In-Reply-To: <18053660.404611279230589983.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896727#action_12896727 ] 

Allen Wittenauer commented on HADOOP-6863:
------------------------------------------

Does anyone else blow configure warnings with this patch?

For example:
configure.ac:46: warning: AC_CACHE_VAL(lt_prog_compiler_static_works, ...): suspicious cache-id, must contain _cv_ to be cached


> Compile-native still fails on Mac OS X.
> ---------------------------------------
>
>                 Key: HADOOP-6863
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6863
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>         Attachments: hadoop-6863-20100715.patch, hadoop-6863-20100721.patch
>
>
> I am still getting failures when I try to compile native library on Mac OS X (10.5.8 Leopard). The problems are two fold:
> - Although "aclocal.m4" is changed after HADOOP-3659 is committed, the corresponding "configure" script is not re-generated.
> - It cannot find libjvm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10087-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 09 23:52:52 2010
Return-Path: <common-issues-return-10087-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17905 invoked from network); 9 Aug 2010 23:52:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Aug 2010 23:52:52 -0000
Received: (qmail 72486 invoked by uid 500); 9 Aug 2010 23:52:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72431 invoked by uid 500); 9 Aug 2010 23:52:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72423 invoked by uid 99); 9 Aug 2010 23:52:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 23:52:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Aug 2010 23:52:48 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o79NqRXt018727
	for <common-issues@hadoop.apache.org>; Mon, 9 Aug 2010 23:52:27 GMT
Message-ID: <26465758.245621281397947209.JavaMail.jira@thor>
Date: Mon, 9 Aug 2010 19:52:27 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6889) Make RPC to have an option to
 timeout
In-Reply-To: <30758566.75561280434516176.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896735#action_12896735 ] 

Kan Zhang commented on HADOOP-6889:
-----------------------------------

{code}
-    @Override
+    @Override  // simply use the default Object#hashcode() ?
     public int hashCode() {
-      return (address.hashCode() + PRIME * System.identityHashCode(protocol)) ^ 
-             (ticket == null ? 0 : ticket.hashCode());
+      return (address.hashCode() + PRIME * (
+                PRIME * (
+                  PRIME * System.identityHashCode(protocol) ^
+                  System.identityHashCode(ticket)
+                ) ^ System.identityHashCode(rpcTimeout)
+              ));
{code}

I wonder if System.identityHashCode(rpcTimeout) is correct in the above code. I think Java does an autoboxing to get an Integer object and what identityHashCode() returns is the address of this object (it doesn't call the object's hashcode() method). So even if two ConnectionId objects have the same rpcTimeout you will get two different hashcodes. I think you can simply use rpcTimeout itself as the hashcode.

> Make RPC to have an option to timeout
> -------------------------------------
>
>                 Key: HADOOP-6889
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6889
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>         Attachments: ipcTimeout.patch, ipcTimeout1.patch, ipcTimeout2.patch
>
>
> Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.
> But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica's length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.
> I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10088-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 00:02:40 2010
Return-Path: <common-issues-return-10088-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20033 invoked from network); 10 Aug 2010 00:02:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 00:02:40 -0000
Received: (qmail 75062 invoked by uid 500); 10 Aug 2010 00:02:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75017 invoked by uid 500); 10 Aug 2010 00:02:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75009 invoked by uid 99); 10 Aug 2010 00:02:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 00:02:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 00:02:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7A02IKe018909
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 00:02:18 GMT
Message-ID: <18069626.245911281398538387.JavaMail.jira@thor>
Date: Mon, 9 Aug 2010 20:02:18 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6889) Make RPC to have an option to
 timeout
In-Reply-To: <30758566.75561280434516176.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896738#action_12896738 ] 

Kan Zhang commented on HADOOP-6889:
-----------------------------------

> I think you can simply use rpcTimeout itself as the hashcode.
If that's what you intended. I haven't considered its randomness.

> Make RPC to have an option to timeout
> -------------------------------------
>
>                 Key: HADOOP-6889
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6889
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>         Attachments: ipcTimeout.patch, ipcTimeout1.patch, ipcTimeout2.patch
>
>
> Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.
> But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica's length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.
> I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10089-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 00:04:41 2010
Return-Path: <common-issues-return-10089-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21005 invoked from network); 10 Aug 2010 00:04:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 00:04:41 -0000
Received: (qmail 76483 invoked by uid 500); 10 Aug 2010 00:04:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76342 invoked by uid 500); 10 Aug 2010 00:04:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76334 invoked by uid 99); 10 Aug 2010 00:04:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 00:04:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 00:04:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7A04JuB018957
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 00:04:19 GMT
Message-ID: <26743550.246031281398659632.JavaMail.jira@thor>
Date: Mon, 9 Aug 2010 20:04:19 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896739#action_12896739 ] 

Greg Roelofs commented on HADOOP-6837:
--------------------------------------

bq. FakeOutputStream isn't the one I'm talking about in package.html. That's for the OutputStream/ FakeInputStream. FakeOutputStream is just the one where I couldn't justify the maximum acting correctly (wrtiting a max of 273 bytes extra) so I added the linked list in case anything goes wrong.

Argh, yes, of course...we discussed that at least twice already.  Sorry for spacing.

Did you ever instrument it to emit a warning if it did go above 273 extra bytes?

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-1-20100722.non-trivial.pseudo-patch, HADOOP-6837-lzma-1-20100722.patch, HADOOP-6837-lzma-2-20100806.patch, HADOOP-6837-lzma-3-20100809.patch, HADOOP-6837-lzma-c-20100719.patch, HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10090-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 00:09:40 2010
Return-Path: <common-issues-return-10090-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26078 invoked from network); 10 Aug 2010 00:09:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 00:09:40 -0000
Received: (qmail 79386 invoked by uid 500); 10 Aug 2010 00:09:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79338 invoked by uid 500); 10 Aug 2010 00:09:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79330 invoked by uid 99); 10 Aug 2010 00:09:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 00:09:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 00:09:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7A09IdR019028
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 00:09:19 GMT
Message-ID: <9585113.246161281398958913.JavaMail.jira@thor>
Date: Mon, 9 Aug 2010 20:09:18 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6889) Make RPC to have an option to
 timeout
In-Reply-To: <30758566.75561280434516176.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896740#action_12896740 ] 

Kan Zhang commented on HADOOP-6889:
-----------------------------------

PS. Maybe Java optimizes on reusing the same object for the same integer. I don't know if this is something we can depend on.

> Make RPC to have an option to timeout
> -------------------------------------
>
>                 Key: HADOOP-6889
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6889
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>         Attachments: ipcTimeout.patch, ipcTimeout1.patch, ipcTimeout2.patch
>
>
> Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.
> But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica's length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.
> I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10091-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 00:17:48 2010
Return-Path: <common-issues-return-10091-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28671 invoked from network); 10 Aug 2010 00:17:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 00:17:47 -0000
Received: (qmail 86616 invoked by uid 500); 10 Aug 2010 00:17:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86555 invoked by uid 500); 10 Aug 2010 00:17:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86547 invoked by uid 99); 10 Aug 2010 00:17:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 00:17:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 00:17:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7A0HMrM019181
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 00:17:22 GMT
Message-ID: <19327946.246581281399442291.JavaMail.jira@thor>
Date: Mon, 9 Aug 2010 20:17:22 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6253) Add a Ceph FileSystem interface.
In-Reply-To: <937733780.1252699621879.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896745#action_12896745 ] 

Allen Wittenauer commented on HADOOP-6253:
------------------------------------------

Should this have a version associated with it?  Should it get committed to trunk?  Should this be in contrib since it is experimental?

> Add a Ceph FileSystem interface.
> --------------------------------
>
>                 Key: HADOOP-6253
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6253
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>            Reporter: Gregory Farnum
>            Assignee: Gregory Farnum
>            Priority: Minor
>         Attachments: HADOOP-6253.patch, HADOOP-6253.patch, HADOOP-6253.patch, HADOOP-6253.patch, HADOOP-6253.patch
>
>
> The experimental distributed filesystem Ceph does not have a single point of failure, uses a distributed metadata cluster instead of a single in-memory server, and might be of use to some Hadoop users.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10092-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 00:21:41 2010
Return-Path: <common-issues-return-10092-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29722 invoked from network); 10 Aug 2010 00:21:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 00:21:41 -0000
Received: (qmail 92539 invoked by uid 500); 10 Aug 2010 00:21:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92518 invoked by uid 500); 10 Aug 2010 00:21:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92510 invoked by uid 99); 10 Aug 2010 00:21:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 00:21:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 00:21:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7A0LG3g019218
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 00:21:17 GMT
Message-ID: <25612323.246681281399676976.JavaMail.jira@thor>
Date: Mon, 9 Aug 2010 20:21:16 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6900) FileSystem#listLocatedStatus should
 not throw generic RuntimeException to indicate error conditions
In-Reply-To: <29253908.169821280964676744.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang updated HADOOP-6900:
----------------------------------

    Attachment: listFilesIO1.patch

This patch adds more comments and changes some FileContext tests to use the new listStatus interface.

> FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6900
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6900
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Suresh Srinivas
>            Assignee: Hairong Kuang
>         Attachments: listFilesIO.patch, listFilesIO1.patch
>
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10093-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 00:45:41 2010
Return-Path: <common-issues-return-10093-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44728 invoked from network); 10 Aug 2010 00:45:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 00:45:40 -0000
Received: (qmail 7340 invoked by uid 500); 10 Aug 2010 00:45:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7270 invoked by uid 500); 10 Aug 2010 00:45:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7259 invoked by uid 99); 10 Aug 2010 00:45:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 00:45:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 00:45:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7A0jJLZ019329
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 00:45:19 GMT
Message-ID: <30915388.246901281401119460.JavaMail.jira@thor>
Date: Mon, 9 Aug 2010 20:45:19 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6889) Make RPC to have an option to
 timeout
In-Reply-To: <30758566.75561280434516176.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896750#action_12896750 ] 

Hairong Kuang commented on HADOOP-6889:
---------------------------------------

Kan, thanks for your insight. It seems that you are right, identifyHashcode is based on references.

Do you mind if we simply use rpcTime as the hashcode?

> Make RPC to have an option to timeout
> -------------------------------------
>
>                 Key: HADOOP-6889
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6889
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>         Attachments: ipcTimeout.patch, ipcTimeout1.patch, ipcTimeout2.patch
>
>
> Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.
> But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica's length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.
> I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10094-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 00:45:47 2010
Return-Path: <common-issues-return-10094-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44833 invoked from network); 10 Aug 2010 00:45:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 00:45:47 -0000
Received: (qmail 7616 invoked by uid 500); 10 Aug 2010 00:45:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7563 invoked by uid 500); 10 Aug 2010 00:45:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7555 invoked by uid 99); 10 Aug 2010 00:45:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 00:45:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 00:45:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7A0jNvM019365
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 00:45:23 GMT
Message-ID: <5076434.247021281401123305.JavaMail.jira@thor>
Date: Mon, 9 Aug 2010 20:45:23 -0400 (EDT)
From: "Nicholas Carlini (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896751#action_12896751 ] 

Nicholas Carlini commented on HADOOP-6837:
------------------------------------------

I did. The average number of overflow bytes is 24. I never saw it go above 120. A quick sed/dc script tells me the standard deviation is 18. So I'm fairly sure that I am correct in that it will never go above 273. Trying with setting the number of fast bytes to 273 gives average of 37 and standard deviation of 26.

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-1-20100722.non-trivial.pseudo-patch, HADOOP-6837-lzma-1-20100722.patch, HADOOP-6837-lzma-2-20100806.patch, HADOOP-6837-lzma-3-20100809.patch, HADOOP-6837-lzma-c-20100719.patch, HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10095-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 00:58:50 2010
Return-Path: <common-issues-return-10095-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46064 invoked from network); 10 Aug 2010 00:58:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 00:58:50 -0000
Received: (qmail 14552 invoked by uid 500); 10 Aug 2010 00:58:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14519 invoked by uid 500); 10 Aug 2010 00:58:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14510 invoked by uid 99); 10 Aug 2010 00:58:50 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 00:58:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 00:58:48 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7A0wPNX019513
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 00:58:26 GMT
Message-ID: <26981512.247361281401905978.JavaMail.jira@thor>
Date: Mon, 9 Aug 2010 20:58:25 -0400 (EDT)
From: "Nicholas Carlini (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6349) Implement FastLZCodec for
 fastlz/lzo algorithm
In-Reply-To: <289817036.1256941439448.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896753#action_12896753 ] 

Nicholas Carlini commented on HADOOP-6349:
------------------------------------------

I believe I've fixed the buffering problems with very few changes to the compressor and decompressor. I'll need to verify that these changes didn't break anything else, but it seems like a simple issue to fix.

> Implement FastLZCodec for fastlz/lzo algorithm
> ----------------------------------------------
>
>                 Key: HADOOP-6349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6349
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: William Kinney
>         Attachments: hadoop-6349-1.patch, hadoop-6349-2.patch, hadoop-6349-3.patch, HADOOP-6349-TestFastLZCodec.patch, HADOOP-6349.patch, TestCodecPerformance.java, TestCodecPerformance.java, testCodecPerfResults.tsv
>
>
> Per  [HADOOP-4874|http://issues.apache.org/jira/browse/HADOOP-4874], FastLZ is a good (speed, license) alternative to LZO. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10096-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 01:11:43 2010
Return-Path: <common-issues-return-10096-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52980 invoked from network); 10 Aug 2010 01:11:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 01:11:43 -0000
Received: (qmail 21719 invoked by uid 500); 10 Aug 2010 01:11:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21652 invoked by uid 500); 10 Aug 2010 01:11:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21641 invoked by uid 99); 10 Aug 2010 01:11:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 01:11:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 01:11:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7A1BIAi019611
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 01:11:18 GMT
Message-ID: <19537508.247491281402678602.JavaMail.jira@thor>
Date: Mon, 9 Aug 2010 21:11:18 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6889) Make RPC to have an option to
 timeout
In-Reply-To: <30758566.75561280434516176.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896754#action_12896754 ] 

Kan Zhang commented on HADOOP-6889:
-----------------------------------

If we can confirm Java always uses the same object for the same integer, your code is correct. Otherwise, I'm fine with using rpcTimeout as hashcode. I've seen recommendations on computing hashcode iteratively as follows.
{code}
hash = PRIME * hash + component_value
{code}
Seems we are doing ^ instead of +. Does it matter?

> Make RPC to have an option to timeout
> -------------------------------------
>
>                 Key: HADOOP-6889
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6889
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>         Attachments: ipcTimeout.patch, ipcTimeout1.patch, ipcTimeout2.patch
>
>
> Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.
> But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica's length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.
> I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10097-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 06:28:44 2010
Return-Path: <common-issues-return-10097-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40845 invoked from network); 10 Aug 2010 06:28:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 06:28:43 -0000
Received: (qmail 63052 invoked by uid 500); 10 Aug 2010 06:28:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62894 invoked by uid 500); 10 Aug 2010 06:28:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62877 invoked by uid 99); 10 Aug 2010 06:28:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 06:28:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 06:28:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7A6SIqA021742
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 06:28:18 GMT
Message-ID: <2932342.248941281421698164.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 02:28:18 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6863) Compile-native still fails on Mac
 OS X.
In-Reply-To: <18053660.404611279230589983.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896776#action_12896776 ] 

Hong Tang commented on HADOOP-6863:
-----------------------------------

bq. Java on Snow Leopard is very different than what is on Leopard due it properly supporting the -d flag to flip from 32-bit and 64-bit JVM support. Last I heard, Yahoo! was still using Leopard.
Ok, I will test it with snow leopard and see if it works.

bq. configure.ac:46: warning: AC_CACHE_VAL(lt_prog_compiler_static_works, ...): suspicious cache-id, must contain cv to be cached
No. I have not seen this. What is the command line that triggers this message? What versions of autotools you are using?

> Compile-native still fails on Mac OS X.
> ---------------------------------------
>
>                 Key: HADOOP-6863
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6863
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>         Attachments: hadoop-6863-20100715.patch, hadoop-6863-20100721.patch
>
>
> I am still getting failures when I try to compile native library on Mac OS X (10.5.8 Leopard). The problems are two fold:
> - Although "aclocal.m4" is changed after HADOOP-3659 is committed, the corresponding "configure" script is not re-generated.
> - It cannot find libjvm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10098-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 08:03:43 2010
Return-Path: <common-issues-return-10098-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63651 invoked from network); 10 Aug 2010 08:03:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 08:03:43 -0000
Received: (qmail 36542 invoked by uid 500); 10 Aug 2010 08:03:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35906 invoked by uid 500); 10 Aug 2010 08:03:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35883 invoked by uid 99); 10 Aug 2010 08:03:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 08:03:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 08:03:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7A83G4Z022649
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 08:03:16 GMT
Message-ID: <5122865.249891281427396576.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 04:03:16 -0400 (EDT)
From: "Ravi Gummadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6715) AccessControlList.toString() returns
 empty string when we set acl to "*"
In-Reply-To: <7375715.82441271756441695.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated HADOOP-6715:
---------------------------------

    Attachment: 6715.patch

Attaching patch for trunk.

AccessControlList.toString() returns a descriptive String of users and groups that are part of this ACL. This is same as that is there in 6715.20S.6.patch.

 Also added a new public method AccessControlList.getAclString() that returns the exact String that can be used for creating a new instance of AccessControlList by passing it to the constructor.

> AccessControlList.toString() returns empty string when we set acl to "*"
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6715
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6715
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security, util
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>         Attachments: 6715.20S.6.patch, 6715.patch
>
>
> AccessControlList.toString() returns empty string when we set the acl to "\*" and also when we set it to empty(i.e. " "). This is causing wrong values for ACLs shown on jobdetails.jsp and jobdetailshistory.jsp web pages when acls are set to "\*".
> I think AccessControlList.toString() needs to be changed to return "\*" when we set the acl to "\*".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10099-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 08:41:40 2010
Return-Path: <common-issues-return-10099-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77799 invoked from network); 10 Aug 2010 08:41:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 08:41:40 -0000
Received: (qmail 82595 invoked by uid 500); 10 Aug 2010 08:41:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82446 invoked by uid 500); 10 Aug 2010 08:41:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82438 invoked by uid 99); 10 Aug 2010 08:41:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 08:41:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 08:41:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7A8fG2O022953
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 08:41:16 GMT
Message-ID: <3551240.250251281429676439.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 04:41:16 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6862) Add api to add user/group to
 AccessControlList
In-Reply-To: <6885195.391831279173830267.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896803#action_12896803 ] 

Hudson commented on HADOOP-6862:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #354 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/354/])
    HADOOP-6862. Adds api to add/remove user and group to AccessControlList. Contributed by Amareshwari Sriramadasu


> Add api to add user/group to AccessControlList
> ----------------------------------------------
>
>                 Key: HADOOP-6862
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6862
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.22.0
>
>         Attachments: patch-6862-1.txt, patch-6862-2.txt, patch-6862.txt
>
>
> Add api addUser(String user) and addGroup(String group) to add user/group to AccessControlList

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10100-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 08:54:42 2010
Return-Path: <common-issues-return-10100-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81185 invoked from network); 10 Aug 2010 08:54:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 08:54:42 -0000
Received: (qmail 6885 invoked by uid 500); 10 Aug 2010 08:54:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6766 invoked by uid 500); 10 Aug 2010 08:54:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6758 invoked by uid 99); 10 Aug 2010 08:54:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 08:54:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 08:54:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7A8sHbS023093
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 08:54:18 GMT
Message-ID: <17946982.250491281430457034.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 04:54:17 -0400 (EDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6862) Add api to add user/group to
 AccessControlList
In-Reply-To: <6885195.391831279173830267.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amareshwari Sriramadasu updated HADOOP-6862:
--------------------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

I just committed this.

> Add api to add user/group to AccessControlList
> ----------------------------------------------
>
>                 Key: HADOOP-6862
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6862
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.22.0
>
>         Attachments: patch-6862-1.txt, patch-6862-2.txt, patch-6862.txt
>
>
> Add api addUser(String user) and addGroup(String group) to add user/group to AccessControlList

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10101-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 09:01:41 2010
Return-Path: <common-issues-return-10101-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82964 invoked from network); 10 Aug 2010 09:01:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 09:01:41 -0000
Received: (qmail 16265 invoked by uid 500); 10 Aug 2010 09:01:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16100 invoked by uid 500); 10 Aug 2010 09:01:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16087 invoked by uid 99); 10 Aug 2010 09:01:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 09:01:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 09:01:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7A91HcK023255
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 09:01:17 GMT
Message-ID: <24835402.250661281430877702.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 05:01:17 -0400 (EDT)
From: "Ravi Gummadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6715) AccessControlList.toString() returns
 empty string when we set acl to "*"
In-Reply-To: <7375715.82441271756441695.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated HADOOP-6715:
---------------------------------

    Attachment: 6715.v1.patch

Attaching new patch for current trunk because HADOOP-6862 just got merged to trunk.

> AccessControlList.toString() returns empty string when we set acl to "*"
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6715
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6715
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security, util
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>         Attachments: 6715.20S.6.patch, 6715.patch, 6715.v1.patch
>
>
> AccessControlList.toString() returns empty string when we set the acl to "\*" and also when we set it to empty(i.e. " "). This is causing wrong values for ACLs shown on jobdetails.jsp and jobdetailshistory.jsp web pages when acls are set to "\*".
> I think AccessControlList.toString() needs to be changed to return "\*" when we set the acl to "\*".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10102-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 10:08:41 2010
Return-Path: <common-issues-return-10102-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9481 invoked from network); 10 Aug 2010 10:08:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 10:08:41 -0000
Received: (qmail 77092 invoked by uid 500); 10 Aug 2010 10:08:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76945 invoked by uid 500); 10 Aug 2010 10:08:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76932 invoked by uid 99); 10 Aug 2010 10:08:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 10:08:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 10:08:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7AA8G9P023967
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 10:08:16 GMT
Message-ID: <19390443.251401281434896709.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 06:08:16 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6715) AccessControlList.toString() returns
 empty string when we set acl to "*"
In-Reply-To: <7375715.82441271756441695.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V updated HADOOP-6715:
------------------------------

          Status: Patch Available  (was: Open)
    Hadoop Flags: [Reviewed]

Patch looks good. +1.

We should commit this together with MAPREDUCE-1780 so that mapreduce doesn't get broken.

> AccessControlList.toString() returns empty string when we set acl to "*"
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6715
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6715
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security, util
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>         Attachments: 6715.20S.6.patch, 6715.patch, 6715.v1.patch
>
>
> AccessControlList.toString() returns empty string when we set the acl to "\*" and also when we set it to empty(i.e. " "). This is causing wrong values for ACLs shown on jobdetails.jsp and jobdetailshistory.jsp web pages when acls are set to "\*".
> I think AccessControlList.toString() needs to be changed to return "\*" when we set the acl to "\*".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10103-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 16:41:43 2010
Return-Path: <common-issues-return-10103-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69842 invoked from network); 10 Aug 2010 16:41:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 16:41:42 -0000
Received: (qmail 60161 invoked by uid 500); 10 Aug 2010 16:41:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59797 invoked by uid 500); 10 Aug 2010 16:41:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59782 invoked by uid 99); 10 Aug 2010 16:41:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 16:41:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 16:41:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7AGfHRL027693
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 16:41:17 GMT
Message-ID: <5214465.255261281458477662.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 12:41:17 -0400 (EDT)
From: "Jeff Hammerbacher (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6909) Umbrella ticket for reducing the
 number of libraries in Common
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Umbrella ticket for reducing the number of libraries in Common
--------------------------------------------------------------

                 Key: HADOOP-6909
                 URL: https://issues.apache.org/jira/browse/HADOOP-6909
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Jeff Hammerbacher


There seems to be some consensus that moving libraries out of Common, with the eventual goal of getting rid of the Common package all together, is a good idea: http://hadoop.markmail.org/thread/jhiactdekhywznac. Hopefully we can collect the incremental tasks needed to achieve that goal here.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10104-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 16:43:42 2010
Return-Path: <common-issues-return-10104-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70546 invoked from network); 10 Aug 2010 16:43:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 16:43:41 -0000
Received: (qmail 62554 invoked by uid 500); 10 Aug 2010 16:43:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62341 invoked by uid 500); 10 Aug 2010 16:43:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62333 invoked by uid 99); 10 Aug 2010 16:43:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 16:43:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 16:43:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7AGhKGw027755
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 16:43:20 GMT
Message-ID: <21025141.255421281458600377.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 12:43:20 -0400 (EDT)
From: "Jeff Hammerbacher (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6910) Replace or merge Configuration with
 Jakarta Commons Configuration
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Replace or merge Configuration with Jakarta Commons Configuration
-----------------------------------------------------------------

                 Key: HADOOP-6910
                 URL: https://issues.apache.org/jira/browse/HADOOP-6910
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Jeff Hammerbacher


Jakarta Commons Configuration:  http://commons.apache.org/configuration/

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10105-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 16:45:39 2010
Return-Path: <common-issues-return-10105-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71381 invoked from network); 10 Aug 2010 16:45:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 16:45:38 -0000
Received: (qmail 67319 invoked by uid 500); 10 Aug 2010 16:45:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67263 invoked by uid 500); 10 Aug 2010 16:45:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67255 invoked by uid 99); 10 Aug 2010 16:45:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 16:45:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 16:45:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7AGjGMU027845
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 16:45:17 GMT
Message-ID: <6789645.255661281458716960.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 12:45:16 -0400 (EDT)
From: "Jeff Hammerbacher (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6909) Umbrella ticket for reducing the
 number of libraries in Common
In-Reply-To: <5214465.255261281458477662.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896938#action_12896938 ] 

Jeff Hammerbacher commented on HADOOP-6909:
-------------------------------------------

Doug also mentions on that thread that the metrics and fs packages might also be moved to Jakarta Commons.

> Umbrella ticket for reducing the number of libraries in Common
> --------------------------------------------------------------
>
>                 Key: HADOOP-6909
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6909
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jeff Hammerbacher
>
> There seems to be some consensus that moving libraries out of Common, with the eventual goal of getting rid of the Common package all together, is a good idea: http://hadoop.markmail.org/thread/jhiactdekhywznac. Hopefully we can collect the incremental tasks needed to achieve that goal here.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10106-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 16:52:41 2010
Return-Path: <common-issues-return-10106-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73110 invoked from network); 10 Aug 2010 16:52:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 16:52:41 -0000
Received: (qmail 76212 invoked by uid 500); 10 Aug 2010 16:52:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76144 invoked by uid 500); 10 Aug 2010 16:52:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76135 invoked by uid 99); 10 Aug 2010 16:52:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 16:52:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 16:52:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7AGqGWE027956
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 16:52:16 GMT
Message-ID: <29838122.255971281459136595.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 12:52:16 -0400 (EDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6909) Umbrella ticket for reducing the
 number of libraries in Common
In-Reply-To: <5214465.255261281458477662.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896940#action_12896940 ] 

Arun C Murthy commented on HADOOP-6909:
---------------------------------------

Moving fs to HDFS might be reasonable, but moving metrics to Commons means we will have another dependency, plus it isn't clear metrics is usable in a wider context.

> Umbrella ticket for reducing the number of libraries in Common
> --------------------------------------------------------------
>
>                 Key: HADOOP-6909
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6909
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jeff Hammerbacher
>
> There seems to be some consensus that moving libraries out of Common, with the eventual goal of getting rid of the Common package all together, is a good idea: http://hadoop.markmail.org/thread/jhiactdekhywznac. Hopefully we can collect the incremental tasks needed to achieve that goal here.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10107-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 16:52:42 2010
Return-Path: <common-issues-return-10107-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73151 invoked from network); 10 Aug 2010 16:52:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 16:52:41 -0000
Received: (qmail 76335 invoked by uid 500); 10 Aug 2010 16:52:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76307 invoked by uid 500); 10 Aug 2010 16:52:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76234 invoked by uid 99); 10 Aug 2010 16:52:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 16:52:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 16:52:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7AGqHl3027971
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 16:52:17 GMT
Message-ID: <21322532.256021281459137803.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 12:52:17 -0400 (EDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6909) Umbrella ticket for reducing the
 number of libraries in Common
In-Reply-To: <5214465.255261281458477662.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896941#action_12896941 ] 

Arun C Murthy commented on HADOOP-6909:
---------------------------------------

bq. There seems to be some consensus that moving libraries out of Common

I'm sure I'd call it 'consensus' yet, we need a discussion before we reach consensus.

> Umbrella ticket for reducing the number of libraries in Common
> --------------------------------------------------------------
>
>                 Key: HADOOP-6909
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6909
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jeff Hammerbacher
>
> There seems to be some consensus that moving libraries out of Common, with the eventual goal of getting rid of the Common package all together, is a good idea: http://hadoop.markmail.org/thread/jhiactdekhywznac. Hopefully we can collect the incremental tasks needed to achieve that goal here.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10108-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 17:00:44 2010
Return-Path: <common-issues-return-10108-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75398 invoked from network); 10 Aug 2010 17:00:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 17:00:44 -0000
Received: (qmail 85643 invoked by uid 500); 10 Aug 2010 17:00:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85577 invoked by uid 500); 10 Aug 2010 17:00:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85569 invoked by uid 99); 10 Aug 2010 17:00:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:00:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:00:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7AH0Jls028222
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 17:00:19 GMT
Message-ID: <21986722.256671281459619392.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 13:00:19 -0400 (EDT)
From: "Jeff Hammerbacher (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6909) Umbrella ticket for reducing the
 number of libraries in Common
In-Reply-To: <5214465.255261281458477662.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896949#action_12896949 ] 

Jeff Hammerbacher commented on HADOOP-6909:
-------------------------------------------

bq. I'm sure I'd call it 'consensus' yet, we need a discussion before we reach consensus.

Sure, and hopefully that discussion can take place here, to avoid derailing the thread on combining the committer lists.

> Umbrella ticket for reducing the number of libraries in Common
> --------------------------------------------------------------
>
>                 Key: HADOOP-6909
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6909
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jeff Hammerbacher
>
> There seems to be some consensus that moving libraries out of Common, with the eventual goal of getting rid of the Common package all together, is a good idea: http://hadoop.markmail.org/thread/jhiactdekhywznac. Hopefully we can collect the incremental tasks needed to achieve that goal here.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10109-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 17:04:40 2010
Return-Path: <common-issues-return-10109-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76764 invoked from network); 10 Aug 2010 17:04:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 17:04:40 -0000
Received: (qmail 96007 invoked by uid 500); 10 Aug 2010 17:04:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95960 invoked by uid 500); 10 Aug 2010 17:04:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95952 invoked by uid 99); 10 Aug 2010 17:04:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:04:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:04:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7AH4ILB028330
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 17:04:18 GMT
Message-ID: <22529989.256841281459858384.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 13:04:18 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6889) Make RPC to have an option to
 timeout
In-Reply-To: <30758566.75561280434516176.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896951#action_12896951 ] 

Hairong Kuang commented on HADOOP-6889:
---------------------------------------

It looks that Java returns the same value for every call of System.indentifyHashcode(x) as long as int x remains the same. But I feel more comfortable using x as hashcode directly. 

Either ^ or + should be fine. But I think we should consistently use either ^ or +.

> Make RPC to have an option to timeout
> -------------------------------------
>
>                 Key: HADOOP-6889
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6889
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>         Attachments: ipcTimeout.patch, ipcTimeout1.patch, ipcTimeout2.patch
>
>
> Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.
> But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica's length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.
> I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10110-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 17:04:40 2010
Return-Path: <common-issues-return-10110-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76806 invoked from network); 10 Aug 2010 17:04:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 17:04:40 -0000
Received: (qmail 96156 invoked by uid 500); 10 Aug 2010 17:04:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95993 invoked by uid 500); 10 Aug 2010 17:04:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95962 invoked by uid 99); 10 Aug 2010 17:04:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:04:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:04:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7AH4Jhc028345
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 17:04:19 GMT
Message-ID: <4702441.256891281459859133.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 13:04:19 -0400 (EDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6909) Umbrella ticket for reducing the
 number of libraries in Common
In-Reply-To: <5214465.255261281458477662.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896952#action_12896952 ] 

Arun C Murthy commented on HADOOP-6909:
---------------------------------------

Yep.

> Umbrella ticket for reducing the number of libraries in Common
> --------------------------------------------------------------
>
>                 Key: HADOOP-6909
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6909
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jeff Hammerbacher
>
> There seems to be some consensus that moving libraries out of Common, with the eventual goal of getting rid of the Common package all together, is a good idea: http://hadoop.markmail.org/thread/jhiactdekhywznac. Hopefully we can collect the incremental tasks needed to achieve that goal here.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10111-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 17:10:41 2010
Return-Path: <common-issues-return-10111-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83042 invoked from network); 10 Aug 2010 17:10:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 17:10:41 -0000
Received: (qmail 8737 invoked by uid 500); 10 Aug 2010 17:10:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8681 invoked by uid 500); 10 Aug 2010 17:10:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8673 invoked by uid 99); 10 Aug 2010 17:10:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:10:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:10:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7AHAGvY028458
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 17:10:17 GMT
Message-ID: <25485112.257161281460216896.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 13:10:16 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6910) Replace or merge Configuration with
 Jakarta Commons Configuration
In-Reply-To: <21025141.255421281458600377.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896957#action_12896957 ] 

Owen O'Malley commented on HADOOP-6910:
---------------------------------------

I'm certainly not a fan of the current Configuration class, but this change seems difficult to pull off (with backwards compatibility) for pretty marginal gains.

What are the advantages of Jakarta configurations? Can they support the current xml format or will you need to implement a new reader for them? Can it support final parameters? Are all of their configurations guaranteed to be serializable? They have a put that takes Objects, so how are those written in to XML or binary?

> Replace or merge Configuration with Jakarta Commons Configuration
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6910
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6910
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jeff Hammerbacher
>
> Jakarta Commons Configuration:  http://commons.apache.org/configuration/

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10112-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 17:12:41 2010
Return-Path: <common-issues-return-10112-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83770 invoked from network); 10 Aug 2010 17:12:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 17:12:40 -0000
Received: (qmail 12884 invoked by uid 500); 10 Aug 2010 17:12:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12841 invoked by uid 500); 10 Aug 2010 17:12:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12833 invoked by uid 99); 10 Aug 2010 17:12:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:12:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:12:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7AHCJiZ028523
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 17:12:19 GMT
Message-ID: <22251033.257341281460339405.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 13:12:19 -0400 (EDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6910) Replace or merge Configuration with
 Jakarta Commons Configuration
In-Reply-To: <21025141.255421281458600377.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896961#action_12896961 ] 

Arun C Murthy commented on HADOOP-6910:
---------------------------------------

Not clear if Commons Config is a good fit:
# It doesn't seem to have have support for 'final' parameters which is critical for the server-side, we need to at least open a feature request there.
# Commons config doesn't seem to have support for context sensitive configs e.g. getMemory("foo", 1KB) - another feature reqeust. I believe Philip had a good proposal, I can't seem to find the jira.
# Commons config doesnt' seem to have support for min/max values per config knob- another feature request. Again, Philip's jira covered if I remember right.

None of these are blockers but for support for 'final' parameters, but we need to be careful we make the switch.

> Replace or merge Configuration with Jakarta Commons Configuration
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6910
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6910
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jeff Hammerbacher
>
> Jakarta Commons Configuration:  http://commons.apache.org/configuration/

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10113-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 17:12:43 2010
Return-Path: <common-issues-return-10113-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83817 invoked from network); 10 Aug 2010 17:12:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 17:12:43 -0000
Received: (qmail 13056 invoked by uid 500); 10 Aug 2010 17:12:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13019 invoked by uid 500); 10 Aug 2010 17:12:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13011 invoked by uid 99); 10 Aug 2010 17:12:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:12:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:12:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7AHCI7g028508
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 17:12:18 GMT
Message-ID: <2810777.257291281460338321.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 13:12:18 -0400 (EDT)
From: "Jeff Hammerbacher (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6909) Umbrella ticket for reducing the
 number of libraries in Common
In-Reply-To: <5214465.255261281458477662.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896960#action_12896960 ] 

Jeff Hammerbacher commented on HADOOP-6909:
-------------------------------------------

Also, I apologize if I jumped the gun a little early, I became irrationally exuberant when I thought about moving to a generic configuration system.

> Umbrella ticket for reducing the number of libraries in Common
> --------------------------------------------------------------
>
>                 Key: HADOOP-6909
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6909
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jeff Hammerbacher
>
> There seems to be some consensus that moving libraries out of Common, with the eventual goal of getting rid of the Common package all together, is a good idea: http://hadoop.markmail.org/thread/jhiactdekhywznac. Hopefully we can collect the incremental tasks needed to achieve that goal here.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10114-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 17:14:43 2010
Return-Path: <common-issues-return-10114-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84687 invoked from network); 10 Aug 2010 17:14:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 17:14:43 -0000
Received: (qmail 17571 invoked by uid 500); 10 Aug 2010 17:14:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17509 invoked by uid 500); 10 Aug 2010 17:14:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17501 invoked by uid 99); 10 Aug 2010 17:14:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:14:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:14:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7AHEI7V028556
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 17:14:18 GMT
Message-ID: <6652594.257431281460458478.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 13:14:18 -0400 (EDT)
From: "Jay Booth (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6909) Umbrella ticket for reducing the
 number of libraries in Common
In-Reply-To: <5214465.255261281458477662.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896962#action_12896962 ] 

Jay Booth commented on HADOOP-6909:
-----------------------------------

Well, if we're encourage to derail this thread with config-related stuff, what do people think about a variable-substituted properties file instead of XML?  

#comment
key=value
#other comment
key2=${key}

> Umbrella ticket for reducing the number of libraries in Common
> --------------------------------------------------------------
>
>                 Key: HADOOP-6909
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6909
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jeff Hammerbacher
>
> There seems to be some consensus that moving libraries out of Common, with the eventual goal of getting rid of the Common package all together, is a good idea: http://hadoop.markmail.org/thread/jhiactdekhywznac. Hopefully we can collect the incremental tasks needed to achieve that goal here.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10115-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 17:16:39 2010
Return-Path: <common-issues-return-10115-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85315 invoked from network); 10 Aug 2010 17:16:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 17:16:38 -0000
Received: (qmail 19142 invoked by uid 500); 10 Aug 2010 17:16:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19075 invoked by uid 500); 10 Aug 2010 17:16:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19065 invoked by uid 99); 10 Aug 2010 17:16:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:16:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:16:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7AHGHwf028619
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 17:16:17 GMT
Message-ID: <9346592.257601281460577390.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 13:16:17 -0400 (EDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6909) Umbrella ticket for reducing the
 number of libraries in Common
In-Reply-To: <5214465.255261281458477662.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896964#action_12896964 ] 

Arun C Murthy commented on HADOOP-6909:
---------------------------------------

Jay, can we please use HADOOP-6910 for discussions on configuration? Thanks.

> Umbrella ticket for reducing the number of libraries in Common
> --------------------------------------------------------------
>
>                 Key: HADOOP-6909
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6909
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jeff Hammerbacher
>
> There seems to be some consensus that moving libraries out of Common, with the eventual goal of getting rid of the Common package all together, is a good idea: http://hadoop.markmail.org/thread/jhiactdekhywznac. Hopefully we can collect the incremental tasks needed to achieve that goal here.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10116-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 17:22:44 2010
Return-Path: <common-issues-return-10116-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87482 invoked from network); 10 Aug 2010 17:22:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 17:22:44 -0000
Received: (qmail 30872 invoked by uid 500); 10 Aug 2010 17:22:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30832 invoked by uid 500); 10 Aug 2010 17:22:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30824 invoked by uid 99); 10 Aug 2010 17:22:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:22:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:22:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7AHMJhd028703
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 17:22:19 GMT
Message-ID: <22658417.257831281460939207.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 13:22:19 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6909) Umbrella ticket for reducing the
 number of libraries in Common
In-Reply-To: <5214465.255261281458477662.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896968#action_12896968 ] 

Owen O'Malley commented on HADOOP-6909:
---------------------------------------

Moving FileSystem and FileContext to Jakarta is a non-starter. Having it in Common where we are all committers is hard enough.

We've discussed moving FileSystem to HDFS, but that is both problematic (it moves a lot of non-HDFS stuff into HDFS) and breaks the abstraction that HDFS implements the contract required by FileSystem.

I don't think this is a good direction.

> Umbrella ticket for reducing the number of libraries in Common
> --------------------------------------------------------------
>
>                 Key: HADOOP-6909
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6909
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jeff Hammerbacher
>
> There seems to be some consensus that moving libraries out of Common, with the eventual goal of getting rid of the Common package all together, is a good idea: http://hadoop.markmail.org/thread/jhiactdekhywznac. Hopefully we can collect the incremental tasks needed to achieve that goal here.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10117-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 17:41:41 2010
Return-Path: <common-issues-return-10117-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92571 invoked from network); 10 Aug 2010 17:41:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 17:41:41 -0000
Received: (qmail 64416 invoked by uid 500); 10 Aug 2010 17:41:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64384 invoked by uid 500); 10 Aug 2010 17:41:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64376 invoked by uid 99); 10 Aug 2010 17:41:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:41:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:41:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7AHfGa9028891
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 17:41:17 GMT
Message-ID: <14471169.258151281462076762.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 13:41:16 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6911) doc update for DelegationTokenFetcher
 (part of HDFS-1036)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

doc update for DelegationTokenFetcher (part of HDFS-1036)
---------------------------------------------------------

                 Key: HADOOP-6911
                 URL: https://issues.apache.org/jira/browse/HADOOP-6911
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Boris Shkolnik
            Priority: Minor




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10118-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 17:43:39 2010
Return-Path: <common-issues-return-10118-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93022 invoked from network); 10 Aug 2010 17:43:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 17:43:39 -0000
Received: (qmail 65514 invoked by uid 500); 10 Aug 2010 17:43:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65473 invoked by uid 500); 10 Aug 2010 17:43:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65465 invoked by uid 99); 10 Aug 2010 17:43:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:43:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:43:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7AHhHrf028935
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 17:43:18 GMT
Message-ID: <23046915.258241281462197969.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 13:43:17 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6786) test-patch needs to verify Herriot
 integrity
In-Reply-To: <19336182.46451274829212745.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6786:
---------------------------------------

    Status: Patch Available  (was: Open)

Seems to be stuck. resubmitting

> test-patch needs to verify Herriot integrity
> --------------------------------------------
>
>                 Key: HADOOP-6786
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6786
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-6786.patch, HADOOP-6786.patch, HADOOP-6786.patch
>
>
> Whenever a new patch is submitted for verification {{test-patch}} process has to make sure that none of Herriot bindings were broken.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10119-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 17:43:41 2010
Return-Path: <common-issues-return-10119-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93086 invoked from network); 10 Aug 2010 17:43:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 17:43:41 -0000
Received: (qmail 65892 invoked by uid 500); 10 Aug 2010 17:43:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65866 invoked by uid 500); 10 Aug 2010 17:43:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65858 invoked by uid 99); 10 Aug 2010 17:43:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:43:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:43:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7AHhH2o028923
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 17:43:17 GMT
Message-ID: <8537151.258201281462197273.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 13:43:17 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6786) test-patch needs to verify Herriot
 integrity
In-Reply-To: <19336182.46451274829212745.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6786:
---------------------------------------

    Status: Open  (was: Patch Available)

> test-patch needs to verify Herriot integrity
> --------------------------------------------
>
>                 Key: HADOOP-6786
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6786
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-6786.patch, HADOOP-6786.patch, HADOOP-6786.patch
>
>
> Whenever a new patch is submitted for verification {{test-patch}} process has to make sure that none of Herriot bindings were broken.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10120-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 17:47:42 2010
Return-Path: <common-issues-return-10120-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94264 invoked from network); 10 Aug 2010 17:47:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 17:47:42 -0000
Received: (qmail 74377 invoked by uid 500); 10 Aug 2010 17:47:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74307 invoked by uid 500); 10 Aug 2010 17:47:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74298 invoked by uid 99); 10 Aug 2010 17:47:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:47:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:47:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7AHlI6G029012
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 17:47:18 GMT
Message-ID: <17360244.258411281462438066.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 13:47:18 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6786) test-patch needs to verify Herriot
 integrity
In-Reply-To: <19336182.46451274829212745.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896981#action_12896981 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6786:
------------------------------------------------

+1 the new patch looks good.

> test-patch needs to verify Herriot integrity
> --------------------------------------------
>
>                 Key: HADOOP-6786
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6786
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-6786.patch, HADOOP-6786.patch, HADOOP-6786.patch
>
>
> Whenever a new patch is submitted for verification {{test-patch}} process has to make sure that none of Herriot bindings were broken.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10121-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 17:55:42 2010
Return-Path: <common-issues-return-10121-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96529 invoked from network); 10 Aug 2010 17:55:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 17:55:42 -0000
Received: (qmail 87398 invoked by uid 500); 10 Aug 2010 17:55:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87349 invoked by uid 500); 10 Aug 2010 17:55:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87340 invoked by uid 99); 10 Aug 2010 17:55:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:55:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 17:55:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7AHtHCT029120
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 17:55:17 GMT
Message-ID: <8868151.258621281462917406.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 13:55:17 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6910) Replace or merge Configuration with
 Jakarta Commons Configuration
In-Reply-To: <21025141.255421281458600377.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896986#action_12896986 ] 

Allen Wittenauer commented on HADOOP-6910:
------------------------------------------

Is it that time of year again?  HADOOP-5670

> Replace or merge Configuration with Jakarta Commons Configuration
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6910
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6910
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jeff Hammerbacher
>
> Jakarta Commons Configuration:  http://commons.apache.org/configuration/

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10122-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 18:51:45 2010
Return-Path: <common-issues-return-10122-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23660 invoked from network); 10 Aug 2010 18:51:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 18:51:45 -0000
Received: (qmail 80004 invoked by uid 500); 10 Aug 2010 18:51:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79931 invoked by uid 500); 10 Aug 2010 18:51:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79923 invoked by uid 99); 10 Aug 2010 18:51:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 18:51:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 18:51:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7AIpKiV029642
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 18:51:21 GMT
Message-ID: <14838146.259481281466280782.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 14:51:20 -0400 (EDT)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6909) Umbrella ticket for reducing the
 number of libraries in Common
In-Reply-To: <5214465.255261281458477662.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12896998#action_12896998 ] 

Konstantin Shvachko commented on HADOOP-6909:
---------------------------------------------

- Getting rid of common, as Dough proposed, is a good idea, imo.
- I think it makes sense to move fs APIs like FileSystem and FileContext into HDFS. The same way as mapreduce APIs like Mapper and Reducer are in mapreduce project.
- I looked at [commons.vfs|http://commons.apache.org/vfs/apidocs/index.html] packages. There are clearly some similarities with fs.FileSystem, but the differences are substantial. I believe one can implement {{commons.vfs.HDFS}} using {{hadoop.fs.FileSystem}}. But using commons.vfs as a new API for HDFS would be a big, incompatible, hard to justify change, which also lacks intrinsic for HDFS apis like getBlockLocations().
So at this point I don't understand what the proposal of moving fs packages into Jakarta Commons means.

> Umbrella ticket for reducing the number of libraries in Common
> --------------------------------------------------------------
>
>                 Key: HADOOP-6909
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6909
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jeff Hammerbacher
>
> There seems to be some consensus that moving libraries out of Common, with the eventual goal of getting rid of the Common package all together, is a good idea: http://hadoop.markmail.org/thread/jhiactdekhywznac. Hopefully we can collect the incremental tasks needed to achieve that goal here.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10123-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 19:01:49 2010
Return-Path: <common-issues-return-10123-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26760 invoked from network); 10 Aug 2010 19:01:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 19:01:49 -0000
Received: (qmail 92480 invoked by uid 500); 10 Aug 2010 19:01:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92444 invoked by uid 500); 10 Aug 2010 19:01:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92436 invoked by uid 99); 10 Aug 2010 19:01:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 19:01:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 19:01:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7AJ1OkD029809
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 19:01:25 GMT
Message-ID: <2646218.259591281466884784.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 15:01:24 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6911) doc update for DelegationTokenFetcher
 (part of HDFS-1036)
In-Reply-To: <14471169.258151281462076762.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6911:
-----------------------------------

    Attachment: HADOOP-6911-doc.patch

> doc update for DelegationTokenFetcher (part of HDFS-1036)
> ---------------------------------------------------------
>
>                 Key: HADOOP-6911
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6911
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Boris Shkolnik
>            Priority: Minor
>         Attachments: HADOOP-6911-doc.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10124-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 21:19:42 2010
Return-Path: <common-issues-return-10124-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89559 invoked from network); 10 Aug 2010 21:19:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 21:19:41 -0000
Received: (qmail 70889 invoked by uid 500); 10 Aug 2010 21:19:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70818 invoked by uid 500); 10 Aug 2010 21:19:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70805 invoked by uid 99); 10 Aug 2010 21:19:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 21:19:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 21:19:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7ALJI5F001387
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 21:19:20 GMT
Message-ID: <3115078.262501281475158966.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 17:19:18 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6910) Replace or merge Configuration with
 Jakarta Commons Configuration
In-Reply-To: <21025141.255421281458600377.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897045#action_12897045 ] 

Luke Lu commented on HADOOP-6910:
---------------------------------

Last time I checked, the commons configuration is not serializable yet (CONFIGURATION-203) and appears to be non-trivial to implement due to all its existing features and dependencies. OTOH, it's designed to have multiple backends, so a ZookeeperConfiguration (a la DatabaseConfiguration) is a natural fit :)

> Replace or merge Configuration with Jakarta Commons Configuration
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6910
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6910
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jeff Hammerbacher
>
> Jakarta Commons Configuration:  http://commons.apache.org/configuration/

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10125-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 21:50:41 2010
Return-Path: <common-issues-return-10125-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98815 invoked from network); 10 Aug 2010 21:50:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 21:50:41 -0000
Received: (qmail 2187 invoked by uid 500); 10 Aug 2010 21:50:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2125 invoked by uid 500); 10 Aug 2010 21:50:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2117 invoked by uid 99); 10 Aug 2010 21:50:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 21:50:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 21:50:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7ALoGsC001680
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 21:50:16 GMT
Message-ID: <29846672.263061281477016561.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 17:50:16 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6912) Guard against NPE when calling
 UGI.isLoginKeytabBased()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Guard against NPE when calling UGI.isLoginKeytabBased()
-------------------------------------------------------

                 Key: HADOOP-6912
                 URL: https://issues.apache.org/jira/browse/HADOOP-6912
             Project: Hadoop Common
          Issue Type: Bug
          Components: security
            Reporter: Kan Zhang
            Assignee: Kan Zhang


NPE can happen when isLoginKeytabBased() is called before a login is performed. See MAPREDUCE-1992 for an example.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10126-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 22:08:46 2010
Return-Path: <common-issues-return-10126-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8989 invoked from network); 10 Aug 2010 22:08:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 22:08:45 -0000
Received: (qmail 23903 invoked by uid 500); 10 Aug 2010 22:08:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23694 invoked by uid 500); 10 Aug 2010 22:08:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23685 invoked by uid 99); 10 Aug 2010 22:08:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 22:08:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 22:08:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7AM8OYV001933
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 22:08:24 GMT
Message-ID: <9248751.263561281478104065.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 18:08:24 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6912) Guard against NPE when calling
 UGI.isLoginKeytabBased()
In-Reply-To: <29846672.263061281477016561.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6912:
------------------------------

    Attachment: c6912-01.patch

Attaching a trivial patch. This change is already present in Y20S branch. Somehow it got left out during forward porting. 

> Guard against NPE when calling UGI.isLoginKeytabBased()
> -------------------------------------------------------
>
>                 Key: HADOOP-6912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6912
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6912-01.patch
>
>
> NPE can happen when isLoginKeytabBased() is called before a login is performed. See MAPREDUCE-1992 for an example.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10127-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 22:08:46 2010
Return-Path: <common-issues-return-10127-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9041 invoked from network); 10 Aug 2010 22:08:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 22:08:46 -0000
Received: (qmail 24185 invoked by uid 500); 10 Aug 2010 22:08:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23828 invoked by uid 500); 10 Aug 2010 22:08:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23695 invoked by uid 99); 10 Aug 2010 22:08:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 22:08:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 22:08:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7AM8O4I001939
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 22:08:24 GMT
Message-ID: <9527453.263581281478104394.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 18:08:24 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6912) Guard against NPE when calling
 UGI.isLoginKeytabBased()
In-Reply-To: <29846672.263061281477016561.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6912:
------------------------------

    Status: Patch Available  (was: Open)

> Guard against NPE when calling UGI.isLoginKeytabBased()
> -------------------------------------------------------
>
>                 Key: HADOOP-6912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6912
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6912-01.patch
>
>
> NPE can happen when isLoginKeytabBased() is called before a login is performed. See MAPREDUCE-1992 for an example.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10128-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 22:29:40 2010
Return-Path: <common-issues-return-10128-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15341 invoked from network); 10 Aug 2010 22:29:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 22:29:39 -0000
Received: (qmail 49021 invoked by uid 500); 10 Aug 2010 22:29:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48719 invoked by uid 500); 10 Aug 2010 22:29:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48682 invoked by uid 99); 10 Aug 2010 22:29:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 22:29:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 22:29:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7AMTGh1002063
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 22:29:16 GMT
Message-ID: <21628148.263841281479356652.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 18:29:16 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6911) doc update for DelegationTokenFetcher
 (part of HDFS-1036)
In-Reply-To: <14471169.258151281462076762.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6911:
-----------------------------------

    Attachment: HADOOP-6911-doc-1.patch

according to Jittendra's comment
put file_name instead "path" in fetchdt command syntax to match the description.

> doc update for DelegationTokenFetcher (part of HDFS-1036)
> ---------------------------------------------------------
>
>                 Key: HADOOP-6911
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6911
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Boris Shkolnik
>            Priority: Minor
>         Attachments: HADOOP-6911-doc-1.patch, HADOOP-6911-doc.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10129-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 22:33:39 2010
Return-Path: <common-issues-return-10129-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16543 invoked from network); 10 Aug 2010 22:33:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 22:33:39 -0000
Received: (qmail 55030 invoked by uid 500); 10 Aug 2010 22:33:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54968 invoked by uid 500); 10 Aug 2010 22:33:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54960 invoked by uid 99); 10 Aug 2010 22:33:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 22:33:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 22:33:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7AMXHj2002129
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 22:33:17 GMT
Message-ID: <30443024.264041281479597750.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 18:33:17 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6905) Better logging messages when a
 delegation token is invalid
In-Reply-To: <12774985.207161281136756838.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6905:
------------------------------

    Status: Open  (was: Patch Available)

> Better logging messages when a delegation token is invalid
> ----------------------------------------------------------
>
>                 Key: HADOOP-6905
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6905
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6905-01.patch
>
>
> From our production logs, we see some logging messages of "token is expired or doesn't exist". It would be helpful to know whose token it was.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10130-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 22:33:40 2010
Return-Path: <common-issues-return-10130-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16584 invoked from network); 10 Aug 2010 22:33:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 22:33:39 -0000
Received: (qmail 55169 invoked by uid 500); 10 Aug 2010 22:33:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54987 invoked by uid 500); 10 Aug 2010 22:33:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54969 invoked by uid 99); 10 Aug 2010 22:33:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 22:33:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 22:33:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7AMXIgf002138
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 22:33:18 GMT
Message-ID: <20859129.264071281479598264.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 18:33:18 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6905) Better logging messages when a
 delegation token is invalid
In-Reply-To: <12774985.207161281136756838.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6905:
------------------------------

    Status: Patch Available  (was: Open)

> Better logging messages when a delegation token is invalid
> ----------------------------------------------------------
>
>                 Key: HADOOP-6905
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6905
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6905-01.patch
>
>
> From our production logs, we see some logging messages of "token is expired or doesn't exist". It would be helpful to know whose token it was.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10131-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 22:43:09 2010
Return-Path: <common-issues-return-10131-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19743 invoked from network); 10 Aug 2010 22:43:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 22:43:09 -0000
Received: (qmail 67899 invoked by uid 500); 10 Aug 2010 22:43:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67852 invoked by uid 500); 10 Aug 2010 22:43:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67836 invoked by uid 99); 10 Aug 2010 22:43:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 22:43:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 22:43:06 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7AMgi34002220
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 22:42:44 GMT
Message-ID: <4044874.264241281480164784.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 18:42:44 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6911) doc update for
 DelegationTokenFetcher (part of HDFS-1036)
In-Reply-To: <14471169.258151281462076762.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik resolved HADOOP-6911.
------------------------------------

    Hadoop Flags: [Reviewed]
      Resolution: Fixed

committed doc to trunk

> doc update for DelegationTokenFetcher (part of HDFS-1036)
> ---------------------------------------------------------
>
>                 Key: HADOOP-6911
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6911
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Boris Shkolnik
>            Priority: Minor
>         Attachments: HADOOP-6911-doc-1.patch, HADOOP-6911-doc.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10132-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 23:01:41 2010
Return-Path: <common-issues-return-10132-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27626 invoked from network); 10 Aug 2010 23:01:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 23:01:41 -0000
Received: (qmail 85822 invoked by uid 500); 10 Aug 2010 23:01:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85786 invoked by uid 500); 10 Aug 2010 23:01:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85778 invoked by uid 99); 10 Aug 2010 23:01:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 23:01:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 23:01:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7AN1GVq002320
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 23:01:16 GMT
Message-ID: <5310816.264321281481276566.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 19:01:16 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6911) doc update for
 DelegationTokenFetcher (part of HDFS-1036)
In-Reply-To: <14471169.258151281462076762.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897078#action_12897078 ] 

Hudson commented on HADOOP-6911:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #355 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/355/])
    HADOOP-6911. doc update for DelegationTokenFetcher


> doc update for DelegationTokenFetcher (part of HDFS-1036)
> ---------------------------------------------------------
>
>                 Key: HADOOP-6911
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6911
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Boris Shkolnik
>            Priority: Minor
>         Attachments: HADOOP-6911-doc-1.patch, HADOOP-6911-doc.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10133-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 23:17:41 2010
Return-Path: <common-issues-return-10133-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34991 invoked from network); 10 Aug 2010 23:17:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 23:17:41 -0000
Received: (qmail 97187 invoked by uid 500); 10 Aug 2010 23:17:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97160 invoked by uid 500); 10 Aug 2010 23:17:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97152 invoked by uid 99); 10 Aug 2010 23:17:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 23:17:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 23:17:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7ANHJ3E002490
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 23:17:19 GMT
Message-ID: <20036790.264661281482239518.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 19:17:19 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6904) A baby step towards inter-version
 communications between dfs client and NameNode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897081#action_12897081 ] 

Hairong Kuang commented on HADOOP-6904:
---------------------------------------

Doug, now that the patch to HDFS-1335 is out there, do you see any issue with this approach? Today I rewrote HDFS-946 to make it backward compatible and tested on my local machine. It worked out very well. 

> A baby step towards inter-version communications between dfs client and NameNode
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6904
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6904
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: rpcVersion.patch, rpcVersion1.patch
>
>
> Currently RPC communications in Hadoop is very strict. If a client has a different version from that of the server, a VersionMismatched exception is thrown and the client can not connect to the server. This force us to update both client and server all at once if a RPC protocol is changed. But sometime different versions do not mean the client & server are not compatible. It would be nice if we could relax this restriction and allows us to support inter-version communications.
> My idea is that DfsClient catches VersionMismatched exception when it connects to NameNode. It then checks if the client & the server is compatible. If yes, it sets the NameNode version in the dfs client and allows the client to continue talking to NameNode. Otherwise, rethrow the VersionMismatch exception.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10134-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 23:38:46 2010
Return-Path: <common-issues-return-10134-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43096 invoked from network); 10 Aug 2010 23:38:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 23:38:45 -0000
Received: (qmail 12791 invoked by uid 500); 10 Aug 2010 23:38:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12727 invoked by uid 500); 10 Aug 2010 23:38:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12719 invoked by uid 99); 10 Aug 2010 23:38:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 23:38:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 23:38:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7ANcKKp002593
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 23:38:20 GMT
Message-ID: <32627828.264821281483500033.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 19:38:20 -0400 (EDT)
From: "Alex Rovner (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4162)
 CodecPool.getDecompressor(LzopCodec) always creates a brand-new
 decompressor.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897084#action_12897084 ] 

Alex Rovner commented on HADOOP-4162:
-------------------------------------

We are using 0.20 and LZO can be used for map compression. In which case the same issue appears:

[2010-08-10 18:19:18,459] INFO  (JvmMetrics.java:71) - Initializing JVM Metrics with processName=SHUFFLE, sessionId=
[2010-08-10 18:19:18,729] INFO  (GPLNativeCodeLoader.java:34) - Loaded native gpl library
[2010-08-10 18:19:18,739] INFO  (LzoCodec.java:65) - Successfully loaded & initialized native-lzo library [hadoop-lzo rev fatal: Not a git repository]
[2010-08-10 18:19:18,793] INFO  (ReduceTask.java:1014) - ShuffleRamManager: MemoryLimit=358881696, MaxSingleShuffleLimit=89720424
[2010-08-10 18:19:18,836] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,839] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,841] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,842] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,844] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,845] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,847] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,849] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,851] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,852] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,854] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,855] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,857] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,859] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,863] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,865] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,867] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,881] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,885] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,886] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,888] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,889] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,891] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,892] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,894] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,895] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,897] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,899] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,901] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,902] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,904] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,905] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,907] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,908] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,910] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,912] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,914] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,915] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,917] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,918] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,920] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,921] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,923] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,924] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,926] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,928] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,930] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,931] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,933] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,934] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,936] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,938] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,939] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,941] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,942] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,943] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,945] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,946] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,947] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,948] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,949] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,950] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,951] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,953] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,954] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,955] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,956] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,957] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,958] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,960] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,961] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,962] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,963] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,964] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,964] INFO  (CodecPool.java:121) - Got brand-new decompressor
[2010-08-10 18:19:18,970] INFO  (ReduceTask.java:2468) - attempt_201008101538_0280_r_000000_0 Thread started: Thread for merging on-disk files
[2010-08-10 18:19:18,971] INFO  (ReduceTask.java:2473) - attempt_201008101538_0280_r_000000_0 Thread waiting: Thread for merging on-disk files


> CodecPool.getDecompressor(LzopCodec) always creates a brand-new decompressor.
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-4162
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4162
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.18.0
>            Reporter: Hong Tang
>            Assignee: Arun C Murthy
>         Attachments: HADOOP-4162_0_20080911.patch
>
>
> CodecPool.getDecompressor(LzopCodec) always creates a brand-new decompressor. I investigated the code, the reason seems to be the following:
> LzopCodec inherits from LzoCodec. The getDecompressorType() method is supposed to return the concrete Decompressor class type the specific Codec class creates. In this case, LzopCodec creates LzopDecompressors and should return LzopDecompressor.class. But instead, it uses the getDecompressorType() method defined in the parent and returns LzoDecompressor.class.
> This leads to CodecPool unable to properly recycle the decompressors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10135-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 10 23:52:46 2010
Return-Path: <common-issues-return-10135-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44904 invoked from network); 10 Aug 2010 23:52:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Aug 2010 23:52:45 -0000
Received: (qmail 18645 invoked by uid 500); 10 Aug 2010 23:52:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18535 invoked by uid 500); 10 Aug 2010 23:52:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18527 invoked by uid 99); 10 Aug 2010 23:52:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 23:52:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Aug 2010 23:52:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7ANqJ4V002652
	for <common-issues@hadoop.apache.org>; Tue, 10 Aug 2010 23:52:20 GMT
Message-ID: <24571943.264931281484339784.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 19:52:19 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4162)
 CodecPool.getDecompressor(LzopCodec) always creates a brand-new
 decompressor.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-4162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897086#action_12897086 ] 

Hong Tang commented on HADOOP-4162:
-----------------------------------

@alex, your description looks like a very different problem from this jira (LzopCodec instead of LzoCodec). Additionally, LZO code is moved to github. You may want to create a new issue there. http://github.com/omalley/hadoop-gpl-compression/issues.

> CodecPool.getDecompressor(LzopCodec) always creates a brand-new decompressor.
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-4162
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4162
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.18.0
>            Reporter: Hong Tang
>            Assignee: Arun C Murthy
>         Attachments: HADOOP-4162_0_20080911.patch
>
>
> CodecPool.getDecompressor(LzopCodec) always creates a brand-new decompressor. I investigated the code, the reason seems to be the following:
> LzopCodec inherits from LzoCodec. The getDecompressorType() method is supposed to return the concrete Decompressor class type the specific Codec class creates. In this case, LzopCodec creates LzopDecompressors and should return LzopDecompressor.class. But instead, it uses the getDecompressorType() method defined in the parent and returns LzoDecompressor.class.
> This leads to CodecPool unable to properly recycle the decompressors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10136-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 00:00:39 2010
Return-Path: <common-issues-return-10136-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46354 invoked from network); 11 Aug 2010 00:00:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 00:00:39 -0000
Received: (qmail 25363 invoked by uid 500); 11 Aug 2010 00:00:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25216 invoked by uid 500); 11 Aug 2010 00:00:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25208 invoked by uid 99); 11 Aug 2010 00:00:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 00:00:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 00:00:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7B00HOd002750
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 00:00:17 GMT
Message-ID: <3201096.265081281484817482.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 20:00:17 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6900) FileSystem#listLocatedStatus should
 not throw generic RuntimeException to indicate error conditions
In-Reply-To: <29253908.169821280964676744.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897088#action_12897088 ] 

Suresh Srinivas commented on HADOOP-6900:
-----------------------------------------

+ 1 for the patch.

bq. Unfortunately not allowing concurrent modification on the input directories is the semantics that map/reduce has it now. I do not want to introduce an incompatible change for now.

Given that this is a new API, I am not sure handling concurrent modification and ignoring deleted directories is a real backward compatibility issue for mapreduce. Mapreduce folks can comment about what is more suitable. We can address this issue in a separate jira though.

> FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6900
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6900
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Suresh Srinivas
>            Assignee: Hairong Kuang
>         Attachments: listFilesIO.patch, listFilesIO1.patch
>
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10137-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 00:19:42 2010
Return-Path: <common-issues-return-10137-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55023 invoked from network); 11 Aug 2010 00:19:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 00:19:41 -0000
Received: (qmail 34755 invoked by uid 500); 11 Aug 2010 00:19:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34693 invoked by uid 500); 11 Aug 2010 00:19:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34685 invoked by uid 99); 11 Aug 2010 00:19:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 00:19:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 00:19:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7B0JGah002893
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 00:19:17 GMT
Message-ID: <24824352.265261281485956742.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 20:19:16 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6900) FileSystem#listLocatedStatus should
 not throw generic RuntimeException to indicate error conditions
In-Reply-To: <29253908.169821280964676744.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897092#action_12897092 ] 

Sanjay Radia commented on HADOOP-6900:
--------------------------------------

>I do not want to introduce an incompatible change for now.
I don't understand the compatibility argument since this is new API.

We could add a parameter to indicate whether or not continue if a change in the sub-tree has occurred. 
I can live with that as long as folks not interpret it to mean that the list returned is atomic - if a subdir's content has been 
returned then the iterator will not detect it.
Hairong's patch can go though as Suresh has suggested and we can file another one to finalize this API. 




> FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6900
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6900
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Suresh Srinivas
>            Assignee: Hairong Kuang
>         Attachments: listFilesIO.patch, listFilesIO1.patch
>
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10138-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 00:27:40 2010
Return-Path: <common-issues-return-10138-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65264 invoked from network); 11 Aug 2010 00:27:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 00:27:40 -0000
Received: (qmail 42841 invoked by uid 500); 11 Aug 2010 00:27:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42792 invoked by uid 500); 11 Aug 2010 00:27:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42784 invoked by uid 99); 11 Aug 2010 00:27:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 00:27:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 00:27:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7B0RI5w003012
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 00:27:18 GMT
Message-ID: <673483.265611281486438864.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 20:27:18 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6900) FileSystem#listLocatedStatus should
 not throw generic RuntimeException to indicate error conditions
In-Reply-To: <29253908.169821280964676744.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897099#action_12897099 ] 

Hairong Kuang commented on HADOOP-6900:
---------------------------------------

> I don't understand the compatibility argument since this is new API.

Even though this is a new API, but mapreduce already has some logic in their code traversing the input directories and getting file status/block locations. Existing code does not allow any concurrent modification to input directories.

I agree that allowing concurrent modification could be an option, but lets do it in a different jira.

> FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6900
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6900
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Suresh Srinivas
>            Assignee: Hairong Kuang
>         Attachments: listFilesIO.patch, listFilesIO1.patch
>
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10139-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 00:27:43 2010
Return-Path: <common-issues-return-10139-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65377 invoked from network); 11 Aug 2010 00:27:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 00:27:43 -0000
Received: (qmail 43138 invoked by uid 500); 11 Aug 2010 00:27:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43087 invoked by uid 500); 11 Aug 2010 00:27:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42992 invoked by uid 99); 11 Aug 2010 00:27:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 00:27:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 00:27:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7B0RIuf002997
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 00:27:18 GMT
Message-ID: <24933333.265561281486438071.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 20:27:18 -0400 (EDT)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Shvachko reassigned HADOOP-6884:
-------------------------------------------

    Assignee: Erik Steffl

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10140-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 01:30:42 2010
Return-Path: <common-issues-return-10140-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91100 invoked from network); 11 Aug 2010 01:30:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 01:30:42 -0000
Received: (qmail 83071 invoked by uid 500); 11 Aug 2010 01:30:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83019 invoked by uid 500); 11 Aug 2010 01:30:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83011 invoked by uid 99); 11 Aug 2010 01:30:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 01:30:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 01:30:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7B1UHTM003383
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 01:30:17 GMT
Message-ID: <21539093.265951281490217306.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 21:30:17 -0400 (EDT)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897106#action_12897106 ] 

Konstantin Shvachko commented on HADOOP-6884:
---------------------------------------------

+1 the patch looks good.
I'll commit both debug-logging related patches if there are no other opinions.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10141-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 03:39:44 2010
Return-Path: <common-issues-return-10141-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29038 invoked from network); 11 Aug 2010 03:39:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 03:39:44 -0000
Received: (qmail 54202 invoked by uid 500); 11 Aug 2010 03:39:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54054 invoked by uid 500); 11 Aug 2010 03:39:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54043 invoked by uid 99); 11 Aug 2010 03:39:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 03:39:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 03:39:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7B3dJJ2004462
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 03:39:19 GMT
Message-ID: <8529262.267941281497959091.JavaMail.jira@thor>
Date: Tue, 10 Aug 2010 23:39:19 -0400 (EDT)
From: "Ranjit Mathew (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6909) Umbrella ticket for reducing the
 number of libraries in Common
In-Reply-To: <5214465.255261281458477662.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897129#action_12897129 ] 

Ranjit Mathew commented on HADOOP-6909:
---------------------------------------

??I think it makes sense to move fs APIs like FileSystem and FileContext into HDFS. The same way as mapreduce APIs like Mapper and Reducer are in mapreduce project.??

I believe HDFS is _a_ file-system supported by Hadoop and it could potentially end up supporting others (e.g Ceph via HADOOP-6253).
So it makes sense to have a common FS API that then has a concrete implementation in HDFS.

> Umbrella ticket for reducing the number of libraries in Common
> --------------------------------------------------------------
>
>                 Key: HADOOP-6909
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6909
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jeff Hammerbacher
>
> There seems to be some consensus that moving libraries out of Common, with the eventual goal of getting rid of the Common package all together, is a good idea: http://hadoop.markmail.org/thread/jhiactdekhywznac. Hopefully we can collect the incremental tasks needed to achieve that goal here.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10142-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 04:28:44 2010
Return-Path: <common-issues-return-10142-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39966 invoked from network); 11 Aug 2010 04:28:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 04:28:43 -0000
Received: (qmail 73095 invoked by uid 500); 11 Aug 2010 04:28:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72946 invoked by uid 500); 11 Aug 2010 04:28:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72928 invoked by uid 99); 11 Aug 2010 04:28:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 04:28:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 04:28:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7B4SHvZ005175
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 04:28:18 GMT
Message-ID: <11616535.268121281500897843.JavaMail.jira@thor>
Date: Wed, 11 Aug 2010 00:28:17 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6900) FileSystem#listLocatedStatus should
 not throw generic RuntimeException to indicate error conditions
In-Reply-To: <29253908.169821280964676744.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897132#action_12897132 ] 

Hairong Kuang commented on HADOOP-6900:
---------------------------------------

Ant test-patch result:

     [exec] -1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 12 new or modified tests.
     [exec] 
     [exec]     -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.
     [exec] 
     [exec]     +1 javac.
     [exec]   The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.

Javadoc warnings are not related to this patch.

Ant test:
BUILD SUCCESSFUL
Total time: 11 minutes 2 seconds


> FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6900
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6900
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Suresh Srinivas
>            Assignee: Hairong Kuang
>         Attachments: listFilesIO.patch, listFilesIO1.patch
>
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10143-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 04:46:43 2010
Return-Path: <common-issues-return-10143-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43194 invoked from network); 11 Aug 2010 04:46:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 04:46:43 -0000
Received: (qmail 86590 invoked by uid 500); 11 Aug 2010 04:46:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86555 invoked by uid 500); 11 Aug 2010 04:46:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86546 invoked by uid 99); 11 Aug 2010 04:46:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 04:46:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 04:46:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7B4kHVH005311
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 04:46:17 GMT
Message-ID: <12505996.268341281501977560.JavaMail.jira@thor>
Date: Wed, 11 Aug 2010 00:46:17 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6900) FileSystem#listLocatedStatus should
 not throw generic RuntimeException to indicate error conditions
In-Reply-To: <29253908.169821280964676744.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hairong Kuang resolved HADOOP-6900.
-----------------------------------

     Hadoop Flags: [Reviewed]
    Fix Version/s: 0.22.0
       Resolution: Fixed

I've just committed this!

> FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6900
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6900
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Suresh Srinivas
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listFilesIO.patch, listFilesIO1.patch
>
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10144-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 05:11:46 2010
Return-Path: <common-issues-return-10144-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51310 invoked from network); 11 Aug 2010 05:11:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 05:11:45 -0000
Received: (qmail 5739 invoked by uid 500); 11 Aug 2010 05:11:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5596 invoked by uid 500); 11 Aug 2010 05:11:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5588 invoked by uid 99); 11 Aug 2010 05:11:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 05:11:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 05:11:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7B5BHn6005656
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 05:11:18 GMT
Message-ID: <19239161.268961281503477785.JavaMail.jira@thor>
Date: Wed, 11 Aug 2010 01:11:17 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6900) FileSystem#listLocatedStatus should
 not throw generic RuntimeException to indicate error conditions
In-Reply-To: <29253908.169821280964676744.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897142#action_12897142 ] 

Hudson commented on HADOOP-6900:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #356 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/356/])
    HADOOP-6900. Make the iterator returned by FileSystem#listLocatedStatus to throw IOException rather than RuntimeException when there is an IO error fetching the next file. Contributed by Hairong Kuang.


> FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6900
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6900
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Suresh Srinivas
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listFilesIO.patch, listFilesIO1.patch
>
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10145-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 05:30:45 2010
Return-Path: <common-issues-return-10145-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55187 invoked from network); 11 Aug 2010 05:30:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 05:30:44 -0000
Received: (qmail 18635 invoked by uid 500); 11 Aug 2010 05:30:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18495 invoked by uid 500); 11 Aug 2010 05:30:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18487 invoked by uid 99); 11 Aug 2010 05:30:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 05:30:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 05:30:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7B5UHL9005848
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 05:30:17 GMT
Message-ID: <9771494.269361281504617296.JavaMail.jira@thor>
Date: Wed, 11 Aug 2010 01:30:17 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6861) Method in Credentials to read and
 write a token storage file.
In-Reply-To: <21462038.384681279145270696.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897148#action_12897148 ] 

Hudson commented on HADOOP-6861:
--------------------------------

Integrated in Hadoop-Common-trunk #421 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/421/])
    

> Method in Credentials to read and write a token storage file.
> -------------------------------------------------------------
>
>                 Key: HADOOP-6861
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6861
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6861.1.patch, HADOOP-6861.2.patch
>
>
> The jira covers the changes in common corresponding to MAPREDUCE-1566. 
> This jira adds  new non-static methods in Credentials to read and write token storage file. A method to copy tokens from another credential object is also added. Static method readTokensAndLoadInUGI is removed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10146-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 10:41:46 2010
Return-Path: <common-issues-return-10146-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52360 invoked from network); 11 Aug 2010 10:41:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 10:41:46 -0000
Received: (qmail 72503 invoked by uid 500); 11 Aug 2010 10:41:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71138 invoked by uid 500); 11 Aug 2010 10:41:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70273 invoked by uid 99); 11 Aug 2010 10:41:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 10:41:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 10:41:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7BAfHC3008708
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 10:41:17 GMT
Message-ID: <11605648.273261281523277747.JavaMail.jira@thor>
Date: Wed, 11 Aug 2010 06:41:17 -0400 (EDT)
From: "Ravi Gummadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6715) AccessControlList.toString()
 returns empty string when we set acl to "*"
In-Reply-To: <7375715.82441271756441695.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897211#action_12897211 ] 

Ravi Gummadi commented on HADOOP-6715:
--------------------------------------

Hudson didn't come back for 1 day.
I manually ran unit tests and test-patch.
All unit tests passed.
test-patch gave:

     [exec] -1 overall.
     [exec]
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec]
     [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
     [exec]
     [exec]     -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.
     [exec]
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec]
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec]
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.


I verified that the existing javadoc warnings are not related to this patch.

> AccessControlList.toString() returns empty string when we set acl to "*"
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6715
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6715
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security, util
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>         Attachments: 6715.20S.6.patch, 6715.patch, 6715.v1.patch
>
>
> AccessControlList.toString() returns empty string when we set the acl to "\*" and also when we set it to empty(i.e. " "). This is causing wrong values for ACLs shown on jobdetails.jsp and jobdetailshistory.jsp web pages when acls are set to "\*".
> I think AccessControlList.toString() needs to be changed to return "\*" when we set the acl to "\*".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10147-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 14:14:45 2010
Return-Path: <common-issues-return-10147-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48797 invoked from network); 11 Aug 2010 14:14:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 14:14:45 -0000
Received: (qmail 58534 invoked by uid 500); 11 Aug 2010 14:14:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58156 invoked by uid 500); 11 Aug 2010 14:14:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57834 invoked by uid 99); 11 Aug 2010 14:14:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 14:14:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 14:14:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7BEEJ5e011168
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 14:14:20 GMT
Message-ID: <2145623.276741281536059802.JavaMail.jira@thor>
Date: Wed, 11 Aug 2010 10:14:19 -0400 (EDT)
From: "Pelle Nilsson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897277#action_12897277 ] 

Pelle Nilsson commented on HADOOP-6837:
---------------------------------------

Do I read these comments correctly that LZMA2/xz is not included in the current patch, and might not be included as part of this issue since the LZMA Java lib does not support it?

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-1-20100722.non-trivial.pseudo-patch, HADOOP-6837-lzma-1-20100722.patch, HADOOP-6837-lzma-2-20100806.patch, HADOOP-6837-lzma-3-20100809.patch, HADOOP-6837-lzma-c-20100719.patch, HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10148-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 15:32:58 2010
Return-Path: <common-issues-return-10148-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96979 invoked from network); 11 Aug 2010 15:32:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 15:32:50 -0000
Received: (qmail 2447 invoked by uid 500); 11 Aug 2010 15:32:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2373 invoked by uid 500); 11 Aug 2010 15:32:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2360 invoked by uid 99); 11 Aug 2010 15:32:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 15:32:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 15:32:48 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7BFWSMA012203
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 15:32:28 GMT
Message-ID: <21349092.278501281540748259.JavaMail.jira@thor>
Date: Wed, 11 Aug 2010 11:32:28 -0400 (EDT)
From: "Nicholas Carlini (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6349) Implement FastLZCodec for fastlz/lzo
 algorithm
In-Reply-To: <289817036.1256941439448.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicholas Carlini updated HADOOP-6349:
-------------------------------------

    Attachment: hadoop-6349-4.patch

Fixed the buffering issues. There is still work to do here, though. In the case where compress() is called with an array that is big enough, then instead of compressing to a temporary buffer and then to the byte array given, it should compress directly to that buffer if possible. It is possible to not do this if the compressor was changed to keep its state and resume compression from the middle, but that seems like it would be more work than it's worth, at very little cost. This would, however, mean that the buffer size used by the CompressorStream would need to be around 64k so that the majority of the time bytes could be written directly to it. (That applies to decompression, too.)

Fixed a bug where it was possible for the end of stream mark to show up in the wrong place. 

Fixed another bug from the compressor where calling write(int) on the output stream n times would add 47*n bytes to the output stream. This is because each time write() is called, so is compress(), which means 26 bytes for a header block and 16 bytes of a header, and then 1 byte of uncompressed data. Fixed this by adding another case where if the buffer has fewer than 2^16 (default block size) bytes, then it'll return true from needsInput, and so it won't compress.

Changed the way that the test codec performance will call write() several times instead of writing all at once. It now calls write() with a random length until it's out of input.

Got rid of the uses of BigInteger ...

Removed the moved code ... no idea why it was ever there.

Made abbreviations in comments real words.

FASTLZ_STRICT_ALIGN mode when set to false doesn't even work. Deleted.

FASTLZ_SAFE mode set to false has no performance increases. All it does is make a few if statements happen (which don't have any loops or anything). Deleted.

At some point, the header blocks (with block ID 1) should get removed. They have no purpose and just remain from the port from 6pack.c. Maybe even make the header smaller by removing the ID now that all blocks will have ID of 17. And the 'options' are only one bit for compressed or just uncompressible data.

> Implement FastLZCodec for fastlz/lzo algorithm
> ----------------------------------------------
>
>                 Key: HADOOP-6349
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6349
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: William Kinney
>         Attachments: hadoop-6349-1.patch, hadoop-6349-2.patch, hadoop-6349-3.patch, hadoop-6349-4.patch, HADOOP-6349-TestFastLZCodec.patch, HADOOP-6349.patch, TestCodecPerformance.java, TestCodecPerformance.java, testCodecPerfResults.tsv
>
>
> Per  [HADOOP-4874|http://issues.apache.org/jira/browse/HADOOP-4874], FastLZ is a good (speed, license) alternative to LZO. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10149-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 17:01:03 2010
Return-Path: <common-issues-return-10149-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17625 invoked from network); 11 Aug 2010 17:01:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 17:01:02 -0000
Received: (qmail 83557 invoked by uid 500); 11 Aug 2010 17:01:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83498 invoked by uid 500); 11 Aug 2010 17:01:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83490 invoked by uid 99); 11 Aug 2010 17:01:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 17:01:01 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 17:01:01 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7BH0ft7013024
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 17:00:41 GMT
Message-ID: <8312018.280091281546041216.JavaMail.jira@thor>
Date: Wed, 11 Aug 2010 13:00:41 -0400 (EDT)
From: "Nicholas Carlini (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897337#action_12897337 ] 

Nicholas Carlini commented on HADOOP-6837:
------------------------------------------

The current patch does not support LZMA2, and correct, it probably won't be included in this patch. There will probably be another JIRA filed to swap out the current C code with the LibLzma library to make the C side of things cleaner. LibLzma also has LZMA2 support, so that would be free. It should be trivial to add LibLzma which has a very Zlib-like interface. And then there will probably be another JIRA to port LibLzma to Java, and make everything nice and clean.

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-1-20100722.non-trivial.pseudo-patch, HADOOP-6837-lzma-1-20100722.patch, HADOOP-6837-lzma-2-20100806.patch, HADOOP-6837-lzma-3-20100809.patch, HADOOP-6837-lzma-c-20100719.patch, HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10150-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 18:03:42 2010
Return-Path: <common-issues-return-10150-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56526 invoked from network); 11 Aug 2010 18:03:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 18:03:41 -0000
Received: (qmail 95956 invoked by uid 500); 11 Aug 2010 18:03:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95812 invoked by uid 500); 11 Aug 2010 18:03:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95790 invoked by uid 99); 11 Aug 2010 18:03:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 18:03:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 18:03:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7BI3JaE013745
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 18:03:19 GMT
Message-ID: <26292317.281221281549799534.JavaMail.jira@thor>
Date: Wed, 11 Aug 2010 14:03:19 -0400 (EDT)
From: "Nicholas Carlini (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicholas Carlini updated HADOOP-6837:
-------------------------------------

    Attachment: HADOOP-6837-lzma-4-20100811.patch

Fixed two FindBugs warnings. Made one variable static and removed another.

Also I fixed build.xml (I think, it works correctly when doing 'ant package' on src/contrib, and it didn't work on the previous patch) to not crash when ant package is run. Had to override all of the build-contrib targets with empty ones. Otherwise it'd go looking for directories which didn't exist (because I don't need them to exist). 

I also changed build-contrib.xml (which is included by the other build.xml's) to have a compile-before target in it so there won't be any errors (even if I did set failonerror to false). Safer that way.

FindBugs fails because of four errors in the original SevenZip code. I could change those to static to make FindBugs happy but that would make more work to apply a patch to the java code, so unless there's a particularly good reason to do so I won't.
SIC Should SevenZip.Compression.LZMA.Decoder$LenDecoder be a _static_ inner class? 
SIC Should SevenZip.Compression.LZMA.Decoder$LiteralDecoder be a _static_ inner class? 
SIC Should SevenZip.Compression.LZMA.Encoder$LiteralEncoder be a _static_ inner class? 
SIC Should SevenZip.Compression.LZMA.Encoder$Optimal be a _static_ inner class?

And JavaDoc find an error:
[javadoc] javadoc: error - Illegal package name: ""
However, I can not reproduce this on my local machine. Let's see if it happens with Hudson.

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-1-20100722.non-trivial.pseudo-patch, HADOOP-6837-lzma-1-20100722.patch, HADOOP-6837-lzma-2-20100806.patch, HADOOP-6837-lzma-3-20100809.patch, HADOOP-6837-lzma-4-20100811.patch, HADOOP-6837-lzma-c-20100719.patch, HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10151-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 18:10:41 2010
Return-Path: <common-issues-return-10151-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62928 invoked from network); 11 Aug 2010 18:10:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 18:10:41 -0000
Received: (qmail 1721 invoked by uid 500); 11 Aug 2010 18:10:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1578 invoked by uid 500); 11 Aug 2010 18:10:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1568 invoked by uid 99); 11 Aug 2010 18:10:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 18:10:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 18:10:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7BIAKsU013807
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 18:10:20 GMT
Message-ID: <14965425.281391281550220387.JavaMail.jira@thor>
Date: Wed, 11 Aug 2010 14:10:20 -0400 (EDT)
From: "Nicholas Carlini (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicholas Carlini updated HADOOP-6837:
-------------------------------------

          Status: Patch Available  (was: Open)
    Hadoop Flags: [Reviewed]

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-1-20100722.non-trivial.pseudo-patch, HADOOP-6837-lzma-1-20100722.patch, HADOOP-6837-lzma-2-20100806.patch, HADOOP-6837-lzma-3-20100809.patch, HADOOP-6837-lzma-4-20100811.patch, HADOOP-6837-lzma-c-20100719.patch, HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10153-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 18:10:43 2010
Return-Path: <common-issues-return-10153-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62996 invoked from network); 11 Aug 2010 18:10:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 18:10:42 -0000
Received: (qmail 1864 invoked by uid 500); 11 Aug 2010 18:10:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1754 invoked by uid 500); 11 Aug 2010 18:10:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1745 invoked by uid 99); 11 Aug 2010 18:10:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 18:10:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 18:10:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7BIALfh013831
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 18:10:21 GMT
Message-ID: <4021179.281471281550221472.JavaMail.jira@thor>
Date: Wed, 11 Aug 2010 14:10:21 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6786) test-patch needs to verify Herriot
 integrity
In-Reply-To: <19336182.46451274829212745.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6786:
---------------------------------------

          Status: Patch Available  (was: Open)
    Hadoop Flags: [Reviewed]

Hudson doesn't pick it up :( even after the queue has been cleaned... One more

> test-patch needs to verify Herriot integrity
> --------------------------------------------
>
>                 Key: HADOOP-6786
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6786
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-6786.patch, HADOOP-6786.patch, HADOOP-6786.patch
>
>
> Whenever a new patch is submitted for verification {{test-patch}} process has to make sure that none of Herriot bindings were broken.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10152-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 18:10:43 2010
Return-Path: <common-issues-return-10152-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62995 invoked from network); 11 Aug 2010 18:10:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 18:10:42 -0000
Received: (qmail 1787 invoked by uid 500); 11 Aug 2010 18:10:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1591 invoked by uid 500); 11 Aug 2010 18:10:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1580 invoked by uid 99); 11 Aug 2010 18:10:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 18:10:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 18:10:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7BIAKbd013819
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 18:10:21 GMT
Message-ID: <1569054.281431281550220978.JavaMail.jira@thor>
Date: Wed, 11 Aug 2010 14:10:20 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6786) test-patch needs to verify Herriot
 integrity
In-Reply-To: <19336182.46451274829212745.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6786:
---------------------------------------

    Status: Open  (was: Patch Available)

> test-patch needs to verify Herriot integrity
> --------------------------------------------
>
>                 Key: HADOOP-6786
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6786
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-6786.patch, HADOOP-6786.patch, HADOOP-6786.patch
>
>
> Whenever a new patch is submitted for verification {{test-patch}} process has to make sure that none of Herriot bindings were broken.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10154-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 18:16:39 2010
Return-Path: <common-issues-return-10154-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66013 invoked from network); 11 Aug 2010 18:16:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 18:16:39 -0000
Received: (qmail 10535 invoked by uid 500); 11 Aug 2010 18:16:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10465 invoked by uid 500); 11 Aug 2010 18:16:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10454 invoked by uid 99); 11 Aug 2010 18:16:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 18:16:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 18:16:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7BIGHG7013922
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 18:16:18 GMT
Message-ID: <27327567.281671281550577971.JavaMail.jira@thor>
Date: Wed, 11 Aug 2010 14:16:17 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6913) Circular initialization between
 UserGroupInformation and KerberosName
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Circular initialization between UserGroupInformation and KerberosName
---------------------------------------------------------------------

                 Key: HADOOP-6913
                 URL: https://issues.apache.org/jira/browse/HADOOP-6913
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Kan Zhang
            Assignee: Kan Zhang


If the first call to UGI is UGI.setConfiguration(conf), it will try to initialize UGI class. During this initialization, the code calls KerberosName.setConfiguration(). KerberosName's static initializer will in turn call UGI.isSecurityEnabled(). Since UGI hasn't been completely initialized yet, isSecurityEnabled() will re-initialize UGI with a DEFAULT conf. As a result, the original conf used in UGI.setConfiguration(conf) will be overwritten by the DEFAULT conf.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10155-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 18:19:39 2010
Return-Path: <common-issues-return-10155-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67871 invoked from network); 11 Aug 2010 18:19:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 18:19:38 -0000
Received: (qmail 17181 invoked by uid 500); 11 Aug 2010 18:19:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17153 invoked by uid 500); 11 Aug 2010 18:19:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17145 invoked by uid 99); 11 Aug 2010 18:19:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 18:19:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 18:19:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7BIJHj9013956
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 18:19:17 GMT
Message-ID: <21925910.281751281550757307.JavaMail.jira@thor>
Date: Wed, 11 Aug 2010 14:19:17 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6913) Circular initialization between
 UserGroupInformation and KerberosName
In-Reply-To: <27327567.281671281550577971.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6913:
------------------------------

    Component/s: security

> Circular initialization between UserGroupInformation and KerberosName
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6913
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6913
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>
> If the first call to UGI is UGI.setConfiguration(conf), it will try to initialize UGI class. During this initialization, the code calls KerberosName.setConfiguration(). KerberosName's static initializer will in turn call UGI.isSecurityEnabled(). Since UGI hasn't been completely initialized yet, isSecurityEnabled() will re-initialize UGI with a DEFAULT conf. As a result, the original conf used in UGI.setConfiguration(conf) will be overwritten by the DEFAULT conf.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10156-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 18:30:43 2010
Return-Path: <common-issues-return-10156-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79330 invoked from network); 11 Aug 2010 18:30:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 18:30:43 -0000
Received: (qmail 36622 invoked by uid 500); 11 Aug 2010 18:30:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36566 invoked by uid 500); 11 Aug 2010 18:30:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36558 invoked by uid 99); 11 Aug 2010 18:30:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 18:30:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 18:30:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7BIUIR1014181
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 18:30:18 GMT
Message-ID: <1554837.282421281551418215.JavaMail.jira@thor>
Date: Wed, 11 Aug 2010 14:30:18 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6913) Circular initialization between
 UserGroupInformation and KerberosName
In-Reply-To: <27327567.281671281550577971.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897370#action_12897370 ] 

Kan Zhang commented on HADOOP-6913:
-----------------------------------

TestBlockToken fails  (HDFS-1284) on some platforms because of this. Before the test starts, it tries to initialize UGI using a customized conf with property "hadoop.security.authentication" set to "kerberos". However, because of this bug, the value eventually been used is "simple" from the default conf. And the test fails as a result.

> Circular initialization between UserGroupInformation and KerberosName
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6913
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6913
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>
> If the first call to UGI is UGI.setConfiguration(conf), it will try to initialize UGI class. During this initialization, the code calls KerberosName.setConfiguration(). KerberosName's static initializer will in turn call UGI.isSecurityEnabled(). Since UGI hasn't been completely initialized yet, isSecurityEnabled() will re-initialize UGI with a DEFAULT conf. As a result, the original conf used in UGI.setConfiguration(conf) will be overwritten by the DEFAULT conf.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10157-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 18:59:43 2010
Return-Path: <common-issues-return-10157-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89639 invoked from network); 11 Aug 2010 18:59:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 18:59:42 -0000
Received: (qmail 92932 invoked by uid 500); 11 Aug 2010 18:59:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92657 invoked by uid 500); 11 Aug 2010 18:59:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92616 invoked by uid 99); 11 Aug 2010 18:59:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 18:59:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 18:59:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7BIxHbX014679
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 18:59:18 GMT
Message-ID: <20102333.283601281553157954.JavaMail.jira@thor>
Date: Wed, 11 Aug 2010 14:59:17 -0400 (EDT)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6900) FileSystem#listLocatedStatus should
 not throw generic RuntimeException to indicate error conditions
In-Reply-To: <29253908.169821280964676744.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897389#action_12897389 ] 

Konstantin Shvachko commented on HADOOP-6900:
---------------------------------------------

Hairong. You changed the prototype of {{AbstractFileSystem.listStatusIterator(Path)}} but did not change its overloaded version {{Hdfs.listStatusIterator(Path)}}. Could you please fix it, as current build is broken as it is.


> FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6900
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6900
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Suresh Srinivas
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listFilesIO.patch, listFilesIO1.patch
>
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10158-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 19:02:47 2010
Return-Path: <common-issues-return-10158-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91958 invoked from network); 11 Aug 2010 19:02:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 19:02:46 -0000
Received: (qmail 10209 invoked by uid 500); 11 Aug 2010 19:02:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10155 invoked by uid 500); 11 Aug 2010 19:02:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10147 invoked by uid 99); 11 Aug 2010 19:02:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 19:02:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 19:02:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7BJ2PqM014789
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 19:02:25 GMT
Message-ID: <32870825.283811281553345358.JavaMail.jira@thor>
Date: Wed, 11 Aug 2010 15:02:25 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897391#action_12897391 ] 

Hong Tang commented on HADOOP-6837:
-----------------------------------

Nicholas's has done some interesting works. But unfortunately I will -1 for marking it "patch available". The currently patch carries a modified version of LZMA SDK. This is a huge maintenance overhead going forward where a much simpler solution clearly exists. We should explore the liblzma route first as I mentioned in http://bit.ly/cDz2Pk.

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-1-20100722.non-trivial.pseudo-patch, HADOOP-6837-lzma-1-20100722.patch, HADOOP-6837-lzma-2-20100806.patch, HADOOP-6837-lzma-3-20100809.patch, HADOOP-6837-lzma-4-20100811.patch, HADOOP-6837-lzma-c-20100719.patch, HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10159-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 19:15:47 2010
Return-Path: <common-issues-return-10159-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2374 invoked from network); 11 Aug 2010 19:15:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 19:15:46 -0000
Received: (qmail 26870 invoked by uid 500); 11 Aug 2010 19:15:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26805 invoked by uid 500); 11 Aug 2010 19:15:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26797 invoked by uid 99); 11 Aug 2010 19:15:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 19:15:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 19:15:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7BJFKv8014999
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 19:15:21 GMT
Message-ID: <4712047.284351281554120849.JavaMail.jira@thor>
Date: Wed, 11 Aug 2010 15:15:20 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897400#action_12897400 ] 

Jakob Homan commented on HADOOP-6837:
-------------------------------------

It may be good to address the code style issue now, since this patch diverges significantly from our standard: http://wiki.apache.org/hadoop/CodeReviewChecklist  Eclipse can re-format everything into Hadoop's style pretty well, if that will save time.

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-1-20100722.non-trivial.pseudo-patch, HADOOP-6837-lzma-1-20100722.patch, HADOOP-6837-lzma-2-20100806.patch, HADOOP-6837-lzma-3-20100809.patch, HADOOP-6837-lzma-4-20100811.patch, HADOOP-6837-lzma-c-20100719.patch, HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10160-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 19:24:54 2010
Return-Path: <common-issues-return-10160-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5796 invoked from network); 11 Aug 2010 19:24:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 19:24:51 -0000
Received: (qmail 45347 invoked by uid 500); 11 Aug 2010 19:24:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44853 invoked by uid 500); 11 Aug 2010 19:24:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44725 invoked by uid 99); 11 Aug 2010 19:24:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 19:24:50 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 19:24:50 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7BJOTgV015157
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 19:24:30 GMT
Message-ID: <520958.284801281554669706.JavaMail.jira@thor>
Date: Wed, 11 Aug 2010 15:24:29 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6693) Add metrics to track kerberos login
 activity
In-Reply-To: <1138587062.22291270763196714.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6693:
------------------------------------

    Attachment: HADOOP-6693.patch

Trunk version of the patch.

> Add metrics to track kerberos login activity
> --------------------------------------------
>
>                 Key: HADOOP-6693
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6693
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6693.1.patch, HADOOP-6693.patch, HADOOP-6693.patch, HADOOP-6693.rel20.1.patch, HADOOP-6693.rel20.patch
>
>
> Need metrics to track kerberos login activity such as login rate and the time taken for login.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10162-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 19:26:38 2010
Return-Path: <common-issues-return-10162-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6732 invoked from network); 11 Aug 2010 19:26:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 19:26:38 -0000
Received: (qmail 46627 invoked by uid 500); 11 Aug 2010 19:26:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46566 invoked by uid 500); 11 Aug 2010 19:26:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46557 invoked by uid 99); 11 Aug 2010 19:26:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 19:26:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 19:26:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7BJQHBw015178
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 19:26:17 GMT
Message-ID: <24307447.284861281554777402.JavaMail.jira@thor>
Date: Wed, 11 Aug 2010 15:26:17 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6693) Add metrics to track kerberos login
 activity
In-Reply-To: <1138587062.22291270763196714.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6693:
------------------------------------

    Attachment:     (was: HADOOP-6693.1.patch)

> Add metrics to track kerberos login activity
> --------------------------------------------
>
>                 Key: HADOOP-6693
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6693
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6693.patch, HADOOP-6693.rel20.1.patch, HADOOP-6693.rel20.patch
>
>
> Need metrics to track kerberos login activity such as login rate and the time taken for login.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10161-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 19:26:39 2010
Return-Path: <common-issues-return-10161-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6728 invoked from network); 11 Aug 2010 19:26:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 19:26:38 -0000
Received: (qmail 46626 invoked by uid 500); 11 Aug 2010 19:26:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46555 invoked by uid 500); 11 Aug 2010 19:26:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46547 invoked by uid 99); 11 Aug 2010 19:26:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 19:26:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 19:26:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7BJQGHO015169
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 19:26:16 GMT
Message-ID: <13930179.284831281554776859.JavaMail.jira@thor>
Date: Wed, 11 Aug 2010 15:26:16 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6693) Add metrics to track kerberos login
 activity
In-Reply-To: <1138587062.22291270763196714.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6693:
------------------------------------

    Attachment:     (was: HADOOP-6693.patch)

> Add metrics to track kerberos login activity
> --------------------------------------------
>
>                 Key: HADOOP-6693
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6693
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6693.patch, HADOOP-6693.rel20.1.patch, HADOOP-6693.rel20.patch
>
>
> Need metrics to track kerberos login activity such as login rate and the time taken for login.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10163-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 20:19:45 2010
Return-Path: <common-issues-return-10163-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27687 invoked from network); 11 Aug 2010 20:19:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 20:19:45 -0000
Received: (qmail 4685 invoked by uid 500); 11 Aug 2010 20:19:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4606 invoked by uid 500); 11 Aug 2010 20:19:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4598 invoked by uid 99); 11 Aug 2010 20:19:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 20:19:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 20:19:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7BKJKar015868
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 20:19:20 GMT
Message-ID: <19643037.286321281557960121.JavaMail.jira@thor>
Date: Wed, 11 Aug 2010 16:19:20 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hong Tang updated HADOOP-6837:
------------------------------

    Status: Open  (was: Patch Available)

Canceling the patch.

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-1-20100722.non-trivial.pseudo-patch, HADOOP-6837-lzma-1-20100722.patch, HADOOP-6837-lzma-2-20100806.patch, HADOOP-6837-lzma-3-20100809.patch, HADOOP-6837-lzma-4-20100811.patch, HADOOP-6837-lzma-c-20100719.patch, HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10164-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 20:32:39 2010
Return-Path: <common-issues-return-10164-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34743 invoked from network); 11 Aug 2010 20:32:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 20:32:39 -0000
Received: (qmail 20121 invoked by uid 500); 11 Aug 2010 20:32:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20087 invoked by uid 500); 11 Aug 2010 20:32:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20079 invoked by uid 99); 11 Aug 2010 20:32:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 20:32:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 20:32:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7BKWHH8016146
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 20:32:17 GMT
Message-ID: <23886466.287101281558737563.JavaMail.jira@thor>
Date: Wed, 11 Aug 2010 16:32:17 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6900) FileSystem#listLocatedStatus should
 not throw generic RuntimeException to indicate error conditions
In-Reply-To: <29253908.169821280964676744.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897437#action_12897437 ] 

Hairong Kuang commented on HADOOP-6900:
---------------------------------------

Yes, I can not change HDFS in common but have to do it in HDFS-202. This is side effect of having fs in common. :(

> FileSystem#listLocatedStatus should not throw generic RuntimeException to indicate error conditions
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6900
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6900
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Suresh Srinivas
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listFilesIO.patch, listFilesIO1.patch
>
>
> HDFS-6870 introduced FileSystem#listLocatedStatus(), that returns an Iterator to iterate through LocatedFileStatus for files under a directory or recursively under a sub-directory. Iterator currently throws generic RuntimeException to indicate error conditions. API needs to be changed to throw appropriate exceptions to indicate error conditions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10165-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 22:03:39 2010
Return-Path: <common-issues-return-10165-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69334 invoked from network); 11 Aug 2010 22:03:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 22:03:39 -0000
Received: (qmail 30154 invoked by uid 500); 11 Aug 2010 22:03:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30070 invoked by uid 500); 11 Aug 2010 22:03:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30059 invoked by uid 99); 11 Aug 2010 22:03:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 22:03:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 22:03:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7BM3H7E017431
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 22:03:17 GMT
Message-ID: <16648832.290101281564197605.JavaMail.jira@thor>
Date: Wed, 11 Aug 2010 18:03:17 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6913) Circular initialization between
 UserGroupInformation and KerberosName
In-Reply-To: <27327567.281671281550577971.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6913:
------------------------------

    Status: Patch Available  (was: Open)

> Circular initialization between UserGroupInformation and KerberosName
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6913
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6913
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6913-01.patch
>
>
> If the first call to UGI is UGI.setConfiguration(conf), it will try to initialize UGI class. During this initialization, the code calls KerberosName.setConfiguration(). KerberosName's static initializer will in turn call UGI.isSecurityEnabled(). Since UGI hasn't been completely initialized yet, isSecurityEnabled() will re-initialize UGI with a DEFAULT conf. As a result, the original conf used in UGI.setConfiguration(conf) will be overwritten by the DEFAULT conf.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10166-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 22:03:41 2010
Return-Path: <common-issues-return-10166-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69425 invoked from network); 11 Aug 2010 22:03:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 22:03:41 -0000
Received: (qmail 30482 invoked by uid 500); 11 Aug 2010 22:03:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30417 invoked by uid 500); 11 Aug 2010 22:03:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30409 invoked by uid 99); 11 Aug 2010 22:03:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 22:03:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 22:03:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7BM3GUk017419
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 22:03:16 GMT
Message-ID: <19805553.290061281564196583.JavaMail.jira@thor>
Date: Wed, 11 Aug 2010 18:03:16 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6913) Circular initialization between
 UserGroupInformation and KerberosName
In-Reply-To: <27327567.281671281550577971.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6913:
------------------------------

    Attachment: c6913-01.patch

A patch that initializes UGI completely before initializing others.

> Circular initialization between UserGroupInformation and KerberosName
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6913
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6913
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6913-01.patch
>
>
> If the first call to UGI is UGI.setConfiguration(conf), it will try to initialize UGI class. During this initialization, the code calls KerberosName.setConfiguration(). KerberosName's static initializer will in turn call UGI.isSecurityEnabled(). Since UGI hasn't been completely initialized yet, isSecurityEnabled() will re-initialize UGI with a DEFAULT conf. As a result, the original conf used in UGI.setConfiguration(conf) will be overwritten by the DEFAULT conf.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10167-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 23:05:42 2010
Return-Path: <common-issues-return-10167-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97655 invoked from network); 11 Aug 2010 23:05:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 23:05:42 -0000
Received: (qmail 6218 invoked by uid 500); 11 Aug 2010 23:05:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6126 invoked by uid 500); 11 Aug 2010 23:05:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6118 invoked by uid 99); 11 Aug 2010 23:05:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 23:05:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 23:05:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7BN5Hbx018457
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 23:05:17 GMT
Message-ID: <14584710.292531281567917300.JavaMail.jira@thor>
Date: Wed, 11 Aug 2010 19:05:17 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6913) Circular initialization between
 UserGroupInformation and KerberosName
In-Reply-To: <27327567.281671281550577971.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897506#action_12897506 ] 

Devaraj Das commented on HADOOP-6913:
-------------------------------------

+1

> Circular initialization between UserGroupInformation and KerberosName
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6913
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6913
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6913-01.patch
>
>
> If the first call to UGI is UGI.setConfiguration(conf), it will try to initialize UGI class. During this initialization, the code calls KerberosName.setConfiguration(). KerberosName's static initializer will in turn call UGI.isSecurityEnabled(). Since UGI hasn't been completely initialized yet, isSecurityEnabled() will re-initialize UGI with a DEFAULT conf. As a result, the original conf used in UGI.setConfiguration(conf) will be overwritten by the DEFAULT conf.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10168-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 11 23:51:49 2010
Return-Path: <common-issues-return-10168-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8088 invoked from network); 11 Aug 2010 23:51:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Aug 2010 23:51:48 -0000
Received: (qmail 41541 invoked by uid 500); 11 Aug 2010 23:51:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41456 invoked by uid 500); 11 Aug 2010 23:51:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41448 invoked by uid 99); 11 Aug 2010 23:51:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 23:51:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Aug 2010 23:51:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7BNpOLS018841
	for <common-issues@hadoop.apache.org>; Wed, 11 Aug 2010 23:51:24 GMT
Message-ID: <8576764.293421281570684083.JavaMail.jira@thor>
Date: Wed, 11 Aug 2010 19:51:24 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897512#action_12897512 ] 

Greg Roelofs commented on HADOOP-6837:
--------------------------------------

bq. The currently patch carries a modified version of LZMA SDK. This is a huge maintenance overhead going forward where a much simpler solution clearly exists.

If the modifications are rolled into the SDK, this issue goes away.  Nicholas, can you create a current diff of the src/contrib part relative to lzma912's original path structure (i.e., so it applies cleanly to a stock lzma912 codebase)?  Then we can send it off to the 7Zip folks and see if they're willing to incorporate it.

(And only "partly exists."  There's no Java in liblzma.  On the other hand, consensus around here seems to be that built-in Java support isn't necessary.)

bq. It may be good to address the code style issue now, since this patch diverges significantly from our standard

Only the src/contrib portion does, and that was intentional.  bzip2 is no longer actively developed, so an in-tree, heavily modified port is no big deal.  LZMA, however, is still a very active project, and if we ever wanted to upgrade to a newer release (e.g., for performance or correctness reasons), we do _not_ want a lot of whitespace noise hiding the real diffs.

But this issue also largely disappears if the substantive modifications are accepted upstream; then the formatting is fairly irrelevant, though still a pain for diffs and patches.  Either way, I don't think style rules are or should necessarily be applicable to contrib code (in the outside-the-core-codebase sense of "contrib").

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-1-20100722.non-trivial.pseudo-patch, HADOOP-6837-lzma-1-20100722.patch, HADOOP-6837-lzma-2-20100806.patch, HADOOP-6837-lzma-3-20100809.patch, HADOOP-6837-lzma-4-20100811.patch, HADOOP-6837-lzma-c-20100719.patch, HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10169-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 12 01:10:40 2010
Return-Path: <common-issues-return-10169-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43728 invoked from network); 12 Aug 2010 01:10:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Aug 2010 01:10:40 -0000
Received: (qmail 4883 invoked by uid 500); 12 Aug 2010 01:10:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4824 invoked by uid 500); 12 Aug 2010 01:10:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4815 invoked by uid 99); 12 Aug 2010 01:10:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 01:10:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 01:10:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7C1AIMo019623
	for <common-issues@hadoop.apache.org>; Thu, 12 Aug 2010 01:10:18 GMT
Message-ID: <11503489.294961281575418546.JavaMail.jira@thor>
Date: Wed, 11 Aug 2010 21:10:18 -0400 (EDT)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Shvachko updated HADOOP-6884:
----------------------------------------

          Status: Patch Available  (was: Open)
    Hadoop Flags: [Reviewed]

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10170-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 12 01:19:40 2010
Return-Path: <common-issues-return-10170-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46589 invoked from network); 12 Aug 2010 01:19:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Aug 2010 01:19:39 -0000
Received: (qmail 12635 invoked by uid 500); 12 Aug 2010 01:19:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12597 invoked by uid 500); 12 Aug 2010 01:19:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12589 invoked by uid 99); 12 Aug 2010 01:19:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 01:19:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 01:19:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7C1JIBJ019728
	for <common-issues@hadoop.apache.org>; Thu, 12 Aug 2010 01:19:18 GMT
Message-ID: <9743198.295201281575958433.JavaMail.jira@thor>
Date: Wed, 11 Aug 2010 21:19:18 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897536#action_12897536 ] 

Doug Cutting commented on HADOOP-6884:
--------------------------------------

This bloats the code when the logged string is a constant.  Also, in the vast majority of cases, this is not a significant optimization.  I'd rather keep most code more readable and only perform the check in performance-critical code.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10171-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 12 06:37:43 2010
Return-Path: <common-issues-return-10171-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53681 invoked from network); 12 Aug 2010 06:37:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Aug 2010 06:37:43 -0000
Received: (qmail 87131 invoked by uid 500); 12 Aug 2010 06:37:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86993 invoked by uid 500); 12 Aug 2010 06:37:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86985 invoked by uid 99); 12 Aug 2010 06:37:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 06:37:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 06:37:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7C6bGVQ022325
	for <common-issues@hadoop.apache.org>; Thu, 12 Aug 2010 06:37:16 GMT
Message-ID: <28705544.298291281595036012.JavaMail.jira@thor>
Date: Thu, 12 Aug 2010 02:37:16 -0400 (EDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6715) AccessControlList.toString() returns
 empty string when we set acl to "*"
In-Reply-To: <7375715.82441271756441695.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amareshwari Sriramadasu updated HADOOP-6715:
--------------------------------------------

           Status: Resolved  (was: Patch Available)
    Fix Version/s: 0.22.0
       Resolution: Fixed

I just committed this. Thanks Ravi !


> AccessControlList.toString() returns empty string when we set acl to "*"
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6715
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6715
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security, util
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.22.0
>
>         Attachments: 6715.20S.6.patch, 6715.patch, 6715.v1.patch
>
>
> AccessControlList.toString() returns empty string when we set the acl to "\*" and also when we set it to empty(i.e. " "). This is causing wrong values for ACLs shown on jobdetails.jsp and jobdetailshistory.jsp web pages when acls are set to "\*".
> I think AccessControlList.toString() needs to be changed to return "\*" when we set the acl to "\*".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10172-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 12 07:02:46 2010
Return-Path: <common-issues-return-10172-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57741 invoked from network); 12 Aug 2010 07:02:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Aug 2010 07:02:46 -0000
Received: (qmail 5288 invoked by uid 500); 12 Aug 2010 07:02:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5128 invoked by uid 500); 12 Aug 2010 07:02:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5105 invoked by uid 99); 12 Aug 2010 07:02:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 07:02:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 07:02:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7C72L30022551
	for <common-issues@hadoop.apache.org>; Thu, 12 Aug 2010 07:02:21 GMT
Message-ID: <2601326.298581281596541125.JavaMail.jira@thor>
Date: Thu, 12 Aug 2010 03:02:21 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6715) AccessControlList.toString()
 returns empty string when we set acl to "*"
In-Reply-To: <7375715.82441271756441695.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897596#action_12897596 ] 

Hudson commented on HADOOP-6715:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #357 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/357/])
    HADOOP-6715. Fixes AccessControlList.toString() to return a descriptive String representation of the ACL. Contributed by Ravi Gummadi


> AccessControlList.toString() returns empty string when we set acl to "*"
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6715
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6715
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security, util
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.22.0
>
>         Attachments: 6715.20S.6.patch, 6715.patch, 6715.v1.patch
>
>
> AccessControlList.toString() returns empty string when we set the acl to "\*" and also when we set it to empty(i.e. " "). This is causing wrong values for ACLs shown on jobdetails.jsp and jobdetailshistory.jsp web pages when acls are set to "\*".
> I think AccessControlList.toString() needs to be changed to return "\*" when we set the acl to "\*".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10173-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 12 11:24:45 2010
Return-Path: <common-issues-return-10173-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53241 invoked from network); 12 Aug 2010 11:24:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Aug 2010 11:24:45 -0000
Received: (qmail 83495 invoked by uid 500); 12 Aug 2010 11:24:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83345 invoked by uid 500); 12 Aug 2010 11:24:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83332 invoked by uid 99); 12 Aug 2010 11:24:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 11:24:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 11:24:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7CBOGV6025337
	for <common-issues@hadoop.apache.org>; Thu, 12 Aug 2010 11:24:16 GMT
Message-ID: <30497466.302591281612256329.JavaMail.jira@thor>
Date: Thu, 12 Aug 2010 07:24:16 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6715) AccessControlList.toString()
 returns empty string when we set acl to "*"
In-Reply-To: <7375715.82441271756441695.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897698#action_12897698 ] 

Hudson commented on HADOOP-6715:
--------------------------------

Integrated in Hadoop-Common-trunk #422 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/422/])
    HADOOP-6715. Fixes AccessControlList.toString() to return a descriptive String representation of the ACL. Contributed by Ravi Gummadi


> AccessControlList.toString() returns empty string when we set acl to "*"
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-6715
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6715
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security, util
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>             Fix For: 0.22.0
>
>         Attachments: 6715.20S.6.patch, 6715.patch, 6715.v1.patch
>
>
> AccessControlList.toString() returns empty string when we set the acl to "\*" and also when we set it to empty(i.e. " "). This is causing wrong values for ACLs shown on jobdetails.jsp and jobdetailshistory.jsp web pages when acls are set to "\*".
> I think AccessControlList.toString() needs to be changed to return "\*" when we set the acl to "\*".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10174-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 12 15:41:38 2010
Return-Path: <common-issues-return-10174-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64631 invoked from network); 12 Aug 2010 15:41:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Aug 2010 15:41:38 -0000
Received: (qmail 51379 invoked by uid 500); 12 Aug 2010 15:41:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51240 invoked by uid 500); 12 Aug 2010 15:41:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51232 invoked by uid 99); 12 Aug 2010 15:41:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 15:41:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 15:41:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7CFfGBT028447
	for <common-issues@hadoop.apache.org>; Thu, 12 Aug 2010 15:41:16 GMT
Message-ID: <21494390.306931281627676321.JavaMail.jira@thor>
Date: Thu, 12 Aug 2010 11:41:16 -0400 (EDT)
From: "Bern Hassmo (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6872) ChecksumFs#listStatus should filter
 out .crc files
In-Reply-To: <2931482.507231279751629413.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897801#action_12897801 ] 

Bern Hassmo commented on HADOOP-6872:
-------------------------------------

This is fixed, at least in trunk.

> ChecksumFs#listStatus should filter out .crc files
> --------------------------------------------------
>
>                 Key: HADOOP-6872
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6872
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Hairong Kuang
>             Fix For: 0.22.0
>
>
> Currently ChecksumFs#listStatus listing not only regular files but also .crc files. Crc files should be filtered out.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10175-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 12 16:21:42 2010
Return-Path: <common-issues-return-10175-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79395 invoked from network); 12 Aug 2010 16:21:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Aug 2010 16:21:41 -0000
Received: (qmail 10776 invoked by uid 500); 12 Aug 2010 16:21:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10623 invoked by uid 500); 12 Aug 2010 16:21:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10615 invoked by uid 99); 12 Aug 2010 16:21:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 16:21:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 16:21:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7CGLHiK028808
	for <common-issues@hadoop.apache.org>; Thu, 12 Aug 2010 16:21:17 GMT
Message-ID: <14439772.307481281630077348.JavaMail.jira@thor>
Date: Thu, 12 Aug 2010 12:21:17 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6872) ChecksumFs#listStatus should filter
 out .crc files
In-Reply-To: <2931482.507231279751629413.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897816#action_12897816 ] 

Hairong Kuang commented on HADOOP-6872:
---------------------------------------

Are you sure? I do not see any code in ChecksumFs that filter out .crc files. Did you look at ChecksumFileSystem?

> ChecksumFs#listStatus should filter out .crc files
> --------------------------------------------------
>
>                 Key: HADOOP-6872
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6872
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Hairong Kuang
>             Fix For: 0.22.0
>
>
> Currently ChecksumFs#listStatus listing not only regular files but also .crc files. Crc files should be filtered out.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10176-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 12 16:45:38 2010
Return-Path: <common-issues-return-10176-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86841 invoked from network); 12 Aug 2010 16:45:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Aug 2010 16:45:38 -0000
Received: (qmail 50124 invoked by uid 500); 12 Aug 2010 16:45:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50091 invoked by uid 500); 12 Aug 2010 16:45:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50083 invoked by uid 99); 12 Aug 2010 16:45:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 16:45:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 16:45:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7CGjGUL029004
	for <common-issues@hadoop.apache.org>; Thu, 12 Aug 2010 16:45:16 GMT
Message-ID: <5585632.307801281631516601.JavaMail.jira@thor>
Date: Thu, 12 Aug 2010 12:45:16 -0400 (EDT)
From: "Bern Hassmo (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6872) ChecksumFs#listStatus should filter
 out .crc files
In-Reply-To: <2931482.507231279751629413.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897827#action_12897827 ] 

Bern Hassmo commented on HADOOP-6872:
-------------------------------------

Yes, I looked at ChecksumFileSystem. Sorry, didn't notice the subtle difference. I'll have a look at ChecksumFs instead.

> ChecksumFs#listStatus should filter out .crc files
> --------------------------------------------------
>
>                 Key: HADOOP-6872
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6872
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Hairong Kuang
>             Fix For: 0.22.0
>
>
> Currently ChecksumFs#listStatus listing not only regular files but also .crc files. Crc files should be filtered out.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10177-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 12 19:15:39 2010
Return-Path: <common-issues-return-10177-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53626 invoked from network); 12 Aug 2010 19:15:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Aug 2010 19:15:39 -0000
Received: (qmail 92422 invoked by uid 500); 12 Aug 2010 19:15:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92340 invoked by uid 500); 12 Aug 2010 19:15:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92332 invoked by uid 99); 12 Aug 2010 19:15:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 19:15:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 19:15:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7CJFHsm000909
	for <common-issues@hadoop.apache.org>; Thu, 12 Aug 2010 19:15:17 GMT
Message-ID: <18140006.311621281640517260.JavaMail.jira@thor>
Date: Thu, 12 Aug 2010 15:15:17 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6693) Add metrics to track kerberos login
 activity
In-Reply-To: <1138587062.22291270763196714.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6693:
------------------------------------

    Attachment: HADOOP-6693.1.patch

New trunk patch that incorporates additional change from HADOOP-6808.

> Add metrics to track kerberos login activity
> --------------------------------------------
>
>                 Key: HADOOP-6693
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6693
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6693.1.patch, HADOOP-6693.patch, HADOOP-6693.rel20.1.patch, HADOOP-6693.rel20.patch
>
>
> Need metrics to track kerberos login activity such as login rate and the time taken for login.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10178-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 12 20:52:40 2010
Return-Path: <common-issues-return-10178-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88525 invoked from network); 12 Aug 2010 20:52:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Aug 2010 20:52:40 -0000
Received: (qmail 23819 invoked by uid 500); 12 Aug 2010 20:52:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23776 invoked by uid 500); 12 Aug 2010 20:52:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23768 invoked by uid 99); 12 Aug 2010 20:52:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 20:52:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 20:52:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7CKqGNU001890
	for <common-issues@hadoop.apache.org>; Thu, 12 Aug 2010 20:52:16 GMT
Message-ID: <9870966.313421281646336187.JavaMail.jira@thor>
Date: Thu, 12 Aug 2010 16:52:16 -0400 (EDT)
From: "Richard Lackey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6914) IPC Client Connection close does not
 protect against partially Connection objects
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

IPC Client Connection close does not protect against partially Connection objects
---------------------------------------------------------------------------------

                 Key: HADOOP-6914
                 URL: https://issues.apache.org/jira/browse/HADOOP-6914
             Project: Hadoop Common
          Issue Type: Bug
          Components: ipc
    Affects Versions: 0.20.0
            Reporter: Richard Lackey


The purpose of this ticket is to cross-reference hbase-2042, opened 12/12/2009. close() is used to release Connection resources. It is invoked even though the Connection object may not have been completely initialized. For example, the file descriptor ulimit may have been exceeded in which case no socket would have been acquired. The result would be an NPE in close. The implied contract for close() seems to be that it will never fail. 

There is a proposed patch in the hbase-2042 ticket.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10179-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 12 20:54:42 2010
Return-Path: <common-issues-return-10179-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89117 invoked from network); 12 Aug 2010 20:54:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Aug 2010 20:54:41 -0000
Received: (qmail 26091 invoked by uid 500); 12 Aug 2010 20:54:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26057 invoked by uid 500); 12 Aug 2010 20:54:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26049 invoked by uid 99); 12 Aug 2010 20:54:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 20:54:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 20:54:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7CKsHBw001915
	for <common-issues@hadoop.apache.org>; Thu, 12 Aug 2010 20:54:17 GMT
Message-ID: <21020407.313491281646457571.JavaMail.jira@thor>
Date: Thu, 12 Aug 2010 16:54:17 -0400 (EDT)
From: "Richard Lackey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6914) IPC Client Connection close does not
 protect against partially Connection objects
In-Reply-To: <9870966.313421281646336187.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Richard Lackey updated HADOOP-6914:
-----------------------------------

    Attachment: hbase-2042.patch

This is the proposed patch to Client based upon trunk. There are no compatibility issues.

> IPC Client Connection close does not protect against partially Connection objects
> ---------------------------------------------------------------------------------
>
>                 Key: HADOOP-6914
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6914
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.20.0
>            Reporter: Richard Lackey
>         Attachments: hbase-2042.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> The purpose of this ticket is to cross-reference hbase-2042, opened 12/12/2009. close() is used to release Connection resources. It is invoked even though the Connection object may not have been completely initialized. For example, the file descriptor ulimit may have been exceeded in which case no socket would have been acquired. The result would be an NPE in close. The implied contract for close() seems to be that it will never fail. 
> There is a proposed patch in the hbase-2042 ticket.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10180-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 12 20:57:42 2010
Return-Path: <common-issues-return-10180-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89613 invoked from network); 12 Aug 2010 20:57:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Aug 2010 20:57:41 -0000
Received: (qmail 32659 invoked by uid 500); 12 Aug 2010 20:57:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32334 invoked by uid 500); 12 Aug 2010 20:57:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32326 invoked by uid 99); 12 Aug 2010 20:57:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 20:57:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 20:57:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7CKvKml002004
	for <common-issues@hadoop.apache.org>; Thu, 12 Aug 2010 20:57:20 GMT
Message-ID: <18087280.313711281646640158.JavaMail.jira@thor>
Date: Thu, 12 Aug 2010 16:57:20 -0400 (EDT)
From: "Richard Lackey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6914) IPC Client Connection close does not
 protect against partially Connection objects
In-Reply-To: <9870966.313421281646336187.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Richard Lackey updated HADOOP-6914:
-----------------------------------

    Status: Patch Available  (was: Open)

The fix is isolated to the close() method. It does not change the normal flow. It only affects the RuntimeException flow.

> IPC Client Connection close does not protect against partially Connection objects
> ---------------------------------------------------------------------------------
>
>                 Key: HADOOP-6914
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6914
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.20.0
>            Reporter: Richard Lackey
>         Attachments: hbase-2042.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> The purpose of this ticket is to cross-reference hbase-2042, opened 12/12/2009. close() is used to release Connection resources. It is invoked even though the Connection object may not have been completely initialized. For example, the file descriptor ulimit may have been exceeded in which case no socket would have been acquired. The result would be an NPE in close. The implied contract for close() seems to be that it will never fail. 
> There is a proposed patch in the hbase-2042 ticket.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10181-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 12 21:58:39 2010
Return-Path: <common-issues-return-10181-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12182 invoked from network); 12 Aug 2010 21:58:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Aug 2010 21:58:39 -0000
Received: (qmail 16689 invoked by uid 500); 12 Aug 2010 21:58:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16648 invoked by uid 500); 12 Aug 2010 21:58:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16640 invoked by uid 99); 12 Aug 2010 21:58:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 21:58:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 21:58:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7CLwH49002563
	for <common-issues@hadoop.apache.org>; Thu, 12 Aug 2010 21:58:17 GMT
Message-ID: <11790974.314791281650297630.JavaMail.jira@thor>
Date: Thu, 12 Aug 2010 17:58:17 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6693) Add metrics to track kerberos login
 activity
In-Reply-To: <1138587062.22291270763196714.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6693:
------------------------------------

    Attachment: HADOOP-6693.patch

New patch to fix a patch merge problem

> Add metrics to track kerberos login activity
> --------------------------------------------
>
>                 Key: HADOOP-6693
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6693
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6693.patch, HADOOP-6693.rel20.1.patch, HADOOP-6693.rel20.patch
>
>
> Need metrics to track kerberos login activity such as login rate and the time taken for login.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10182-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 12 21:58:41 2010
Return-Path: <common-issues-return-10182-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12231 invoked from network); 12 Aug 2010 21:58:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Aug 2010 21:58:40 -0000
Received: (qmail 16859 invoked by uid 500); 12 Aug 2010 21:58:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16814 invoked by uid 500); 12 Aug 2010 21:58:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16806 invoked by uid 99); 12 Aug 2010 21:58:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 21:58:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 21:58:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7CLwJ0o002596
	for <common-issues@hadoop.apache.org>; Thu, 12 Aug 2010 21:58:19 GMT
Message-ID: <1554598.314901281650299644.JavaMail.jira@thor>
Date: Thu, 12 Aug 2010 17:58:19 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6693) Add metrics to track kerberos login
 activity
In-Reply-To: <1138587062.22291270763196714.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6693:
------------------------------------

    Attachment:     (was: HADOOP-6693.patch)

> Add metrics to track kerberos login activity
> --------------------------------------------
>
>                 Key: HADOOP-6693
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6693
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6693.patch, HADOOP-6693.rel20.1.patch, HADOOP-6693.rel20.patch
>
>
> Need metrics to track kerberos login activity such as login rate and the time taken for login.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10183-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 12 21:58:42 2010
Return-Path: <common-issues-return-10183-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12272 invoked from network); 12 Aug 2010 21:58:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Aug 2010 21:58:41 -0000
Received: (qmail 17014 invoked by uid 500); 12 Aug 2010 21:58:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16981 invoked by uid 500); 12 Aug 2010 21:58:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16973 invoked by uid 99); 12 Aug 2010 21:58:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 21:58:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 21:58:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7CLwIb5002575
	for <common-issues@hadoop.apache.org>; Thu, 12 Aug 2010 21:58:18 GMT
Message-ID: <11517723.314831281650298250.JavaMail.jira@thor>
Date: Thu, 12 Aug 2010 17:58:18 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6693) Add metrics to track kerberos login
 activity
In-Reply-To: <1138587062.22291270763196714.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6693:
------------------------------------

    Attachment:     (was: HADOOP-6693.1.patch)

> Add metrics to track kerberos login activity
> --------------------------------------------
>
>                 Key: HADOOP-6693
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6693
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6693.patch, HADOOP-6693.rel20.1.patch, HADOOP-6693.rel20.patch
>
>
> Need metrics to track kerberos login activity such as login rate and the time taken for login.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10184-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 12 22:00:46 2010
Return-Path: <common-issues-return-10184-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13020 invoked from network); 12 Aug 2010 22:00:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Aug 2010 22:00:45 -0000
Received: (qmail 19754 invoked by uid 500); 12 Aug 2010 22:00:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19716 invoked by uid 500); 12 Aug 2010 22:00:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19705 invoked by uid 99); 12 Aug 2010 22:00:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 22:00:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 22:00:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7CM0OHX002618
	for <common-issues@hadoop.apache.org>; Thu, 12 Aug 2010 22:00:25 GMT
Message-ID: <4052793.314921281650424735.JavaMail.jira@thor>
Date: Thu, 12 Aug 2010 18:00:24 -0400 (EDT)
From: "Nicolas Spiegelberg (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6915) Add daemon restart option
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Add daemon restart option
-------------------------

                 Key: HADOOP-6915
                 URL: https://issues.apache.org/jira/browse/HADOOP-6915
             Project: Hadoop Common
          Issue Type: Improvement
          Components: scripts
            Reporter: Nicolas Spiegelberg
            Priority: Minor
             Fix For: 0.22.0, 0.20-append


With the addition of AvatarNode, we are working on a rolling restart option for upgrading a cluster.  It seems that a generic restart option would be useful for this purpose and perhaps other administrative tasks.  Add a 'restart' option to hadoop-daemon.sh and also modify 'slaves.sh' to run operations in parallel or serially.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10185-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 12 22:05:45 2010
Return-Path: <common-issues-return-10185-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19100 invoked from network); 12 Aug 2010 22:05:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Aug 2010 22:05:45 -0000
Received: (qmail 29623 invoked by uid 500); 12 Aug 2010 22:05:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29564 invoked by uid 500); 12 Aug 2010 22:05:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29556 invoked by uid 99); 12 Aug 2010 22:05:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 22:05:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 22:05:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7CM5Gbt002715
	for <common-issues@hadoop.apache.org>; Thu, 12 Aug 2010 22:05:16 GMT
Message-ID: <3020268.315081281650716441.JavaMail.jira@thor>
Date: Thu, 12 Aug 2010 18:05:16 -0400 (EDT)
From: "Nicolas Spiegelberg (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6915) Add daemon restart option
In-Reply-To: <4052793.314921281650424735.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicolas Spiegelberg updated HADOOP-6915:
----------------------------------------

    Attachment: HADOOP-6915_0.22.patch
                HADOOP-6915_0.20-append.patch

> Add daemon restart option
> -------------------------
>
>                 Key: HADOOP-6915
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6915
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Nicolas Spiegelberg
>            Priority: Minor
>             Fix For: 0.22.0, 0.20-append
>
>         Attachments: HADOOP-6915_0.20-append.patch, HADOOP-6915_0.22.patch
>
>
> With the addition of AvatarNode, we are working on a rolling restart option for upgrading a cluster.  It seems that a generic restart option would be useful for this purpose and perhaps other administrative tasks.  Add a 'restart' option to hadoop-daemon.sh and also modify 'slaves.sh' to run operations in parallel or serially.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10186-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 12 22:07:41 2010
Return-Path: <common-issues-return-10186-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19644 invoked from network); 12 Aug 2010 22:07:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Aug 2010 22:07:41 -0000
Received: (qmail 30847 invoked by uid 500); 12 Aug 2010 22:07:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30803 invoked by uid 500); 12 Aug 2010 22:07:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30795 invoked by uid 99); 12 Aug 2010 22:07:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 22:07:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 22:07:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7CM7K84002799
	for <common-issues@hadoop.apache.org>; Thu, 12 Aug 2010 22:07:20 GMT
Message-ID: <7882701.315291281650840010.JavaMail.jira@thor>
Date: Thu, 12 Aug 2010 18:07:20 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6904) A baby step towards inter-version
 communications between dfs client and NameNode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897971#action_12897971 ] 

Erik Steffl commented on HADOOP-6904:
-------------------------------------

Was it ever considered to take the capabilities approach instead of version checking?

See e.g. IMAP protocol for example (chose it as an example because it's a widely used protocol that has number of client and server implementations working successfully with each other), this looks like a fairly comprehensive link to protocol specs http://www.networksorcery.com/enp/protocol/imap.htm

Seems like as Hadoop matures it could make sense to provide a method for both client and server to negotiate how they talk to each other (and possibly adjust to each other). At this point the many implementation scenario is not valid yet but capabilities are useful anyway (cross version) and (very likely in my opinion) sooner or later we will get different implementations (happens with pretty much any successfull non-proprietary client/server system (even some proprietary ones)).

> A baby step towards inter-version communications between dfs client and NameNode
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6904
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6904
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: rpcVersion.patch, rpcVersion1.patch
>
>
> Currently RPC communications in Hadoop is very strict. If a client has a different version from that of the server, a VersionMismatched exception is thrown and the client can not connect to the server. This force us to update both client and server all at once if a RPC protocol is changed. But sometime different versions do not mean the client & server are not compatible. It would be nice if we could relax this restriction and allows us to support inter-version communications.
> My idea is that DfsClient catches VersionMismatched exception when it connects to NameNode. It then checks if the client & the server is compatible. If yes, it sets the NameNode version in the dfs client and allows the client to continue talking to NameNode. Otherwise, rethrow the VersionMismatch exception.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10187-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 12 22:07:42 2010
Return-Path: <common-issues-return-10187-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19660 invoked from network); 12 Aug 2010 22:07:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Aug 2010 22:07:41 -0000
Received: (qmail 31000 invoked by uid 500); 12 Aug 2010 22:07:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30968 invoked by uid 500); 12 Aug 2010 22:07:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30924 invoked by uid 99); 12 Aug 2010 22:07:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 22:07:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 22:07:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7CM7H0b002760
	for <common-issues@hadoop.apache.org>; Thu, 12 Aug 2010 22:07:17 GMT
Message-ID: <30384557.315161281650837910.JavaMail.jira@thor>
Date: Thu, 12 Aug 2010 18:07:17 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6693) Add metrics to track kerberos login
 activity
In-Reply-To: <1138587062.22291270763196714.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6693:
------------------------------------

    Status: Patch Available  (was: Open)

> Add metrics to track kerberos login activity
> --------------------------------------------
>
>                 Key: HADOOP-6693
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6693
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6693.patch, HADOOP-6693.rel20.1.patch, HADOOP-6693.rel20.patch
>
>
> Need metrics to track kerberos login activity such as login rate and the time taken for login.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10188-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 12 23:09:42 2010
Return-Path: <common-issues-return-10188-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37479 invoked from network); 12 Aug 2010 23:09:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Aug 2010 23:09:42 -0000
Received: (qmail 95767 invoked by uid 500); 12 Aug 2010 23:09:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95672 invoked by uid 500); 12 Aug 2010 23:09:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95664 invoked by uid 99); 12 Aug 2010 23:09:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 23:09:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 23:09:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7CN9HcN003525
	for <common-issues@hadoop.apache.org>; Thu, 12 Aug 2010 23:09:17 GMT
Message-ID: <30192217.317061281654557723.JavaMail.jira@thor>
Date: Thu, 12 Aug 2010 19:09:17 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6693) Add metrics to track kerberos login
 activity
In-Reply-To: <1138587062.22291270763196714.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897996#action_12897996 ] 

Jitendra Nath Pandey commented on HADOOP-6693:
----------------------------------------------

+1.
The kinit login in spawnAutoRenewalThreadForUserCreds is not tracked by these metrics, which is fine because this thread is spawned only at clients and  purpose of these metrics is to track logins on the servers.

> Add metrics to track kerberos login activity
> --------------------------------------------
>
>                 Key: HADOOP-6693
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6693
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6693.patch, HADOOP-6693.rel20.1.patch, HADOOP-6693.rel20.patch
>
>
> Need metrics to track kerberos login activity such as login rate and the time taken for login.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10189-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 12 23:48:39 2010
Return-Path: <common-issues-return-10189-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48859 invoked from network); 12 Aug 2010 23:48:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Aug 2010 23:48:39 -0000
Received: (qmail 36709 invoked by uid 500); 12 Aug 2010 23:48:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36648 invoked by uid 500); 12 Aug 2010 23:48:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36640 invoked by uid 99); 12 Aug 2010 23:48:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 23:48:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Aug 2010 23:48:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7CNmHCG003755
	for <common-issues@hadoop.apache.org>; Thu, 12 Aug 2010 23:48:17 GMT
Message-ID: <14834938.317501281656897478.JavaMail.jira@thor>
Date: Thu, 12 Aug 2010 19:48:17 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898007#action_12898007 ] 

Erik Steffl commented on HADOOP-6884:
-------------------------------------

Re https://issues.apache.org/jira/browse/HADOOP-6884?focusedCommentId=12897536&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12897536

Me too however then there's no way to automatically or at least semi-automatically check for the cases where the optimisation would be helpful.

If all debug() logging calls have isDebugEnabled() guard it's easy to check the whole codebase in few minutes (I have a simple script that only produces few false positives that need to be eyeballed). If the guard is used in minority of cases where it's actually useful then verification whether it's used everywhere where it's needed is very expensive.

Possibly better solutions but I don't think they are realistic in short/medium term:

  - use scala and lazy values (arguments for debug() are not constructed unless used)

  - use extensive profiling and see where debug() calls matter (this could be part of automatic builds/tests)

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10190-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 13 00:24:41 2010
Return-Path: <common-issues-return-10190-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68877 invoked from network); 13 Aug 2010 00:24:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Aug 2010 00:24:41 -0000
Received: (qmail 69070 invoked by uid 500); 13 Aug 2010 00:24:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69033 invoked by uid 500); 13 Aug 2010 00:24:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69024 invoked by uid 99); 13 Aug 2010 00:24:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 00:24:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 00:24:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7D0OGqO004214
	for <common-issues@hadoop.apache.org>; Fri, 13 Aug 2010 00:24:17 GMT
Message-ID: <23448170.318591281659056963.JavaMail.jira@thor>
Date: Thu, 12 Aug 2010 20:24:16 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898018#action_12898018 ] 

Luke Lu commented on HADOOP-6884:
---------------------------------

I'd prefer fixing the tools instead of bloating the code.  Some simple heuristics that could work:

* Skip checking all the test code, which is trivial, as they live in different trees.
* Put some marker, say a trailing /// for verified debug calls that are not expensive, so a grep -v '///$' can filter them out etc.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10191-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 13 00:46:38 2010
Return-Path: <common-issues-return-10191-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76323 invoked from network); 13 Aug 2010 00:46:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Aug 2010 00:46:38 -0000
Received: (qmail 6176 invoked by uid 500); 13 Aug 2010 00:46:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6118 invoked by uid 500); 13 Aug 2010 00:46:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6110 invoked by uid 99); 13 Aug 2010 00:46:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 00:46:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 00:46:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7D0kGFQ004361
	for <common-issues@hadoop.apache.org>; Fri, 13 Aug 2010 00:46:17 GMT
Message-ID: <19544546.318851281660376872.JavaMail.jira@thor>
Date: Thu, 12 Aug 2010 20:46:16 -0400 (EDT)
From: "Jeff Hammerbacher (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6915) Add daemon restart option
In-Reply-To: <4052793.314921281650424735.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898025#action_12898025 ] 

Jeff Hammerbacher commented on HADOOP-6915:
-------------------------------------------

In a magical world where ponies run free under rainbows, adding restart() as a method to HDFS-326 would be another way to tackle this sort of problem.

> Add daemon restart option
> -------------------------
>
>                 Key: HADOOP-6915
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6915
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>            Reporter: Nicolas Spiegelberg
>            Priority: Minor
>             Fix For: 0.22.0, 0.20-append
>
>         Attachments: HADOOP-6915_0.20-append.patch, HADOOP-6915_0.22.patch
>
>
> With the addition of AvatarNode, we are working on a rolling restart option for upgrading a cluster.  It seems that a generic restart option would be useful for this purpose and perhaps other administrative tasks.  Add a 'restart' option to hadoop-daemon.sh and also modify 'slaves.sh' to run operations in parallel or serially.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10192-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 13 03:49:46 2010
Return-Path: <common-issues-return-10192-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34258 invoked from network); 13 Aug 2010 03:49:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Aug 2010 03:49:45 -0000
Received: (qmail 14669 invoked by uid 500); 13 Aug 2010 03:49:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13403 invoked by uid 500); 13 Aug 2010 03:49:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12388 invoked by uid 99); 13 Aug 2010 03:49:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 03:49:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 03:49:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7D3nH0v005947
	for <common-issues@hadoop.apache.org>; Fri, 13 Aug 2010 03:49:17 GMT
Message-ID: <4175985.321711281671357501.JavaMail.jira@thor>
Date: Thu, 12 Aug 2010 23:49:17 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6872) ChecksumFs#listStatus should filter
 out .crc files
In-Reply-To: <2931482.507231279751629413.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898064#action_12898064 ] 

Vinod K V commented on HADOOP-6872:
-----------------------------------

I incorporated this as part of HADOOP-6906, the patch is up for review there. Can I close this as a duplicate?

> ChecksumFs#listStatus should filter out .crc files
> --------------------------------------------------
>
>                 Key: HADOOP-6872
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6872
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Hairong Kuang
>             Fix For: 0.22.0
>
>
> Currently ChecksumFs#listStatus listing not only regular files but also .crc files. Crc files should be filtered out.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10193-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 13 07:38:02 2010
Return-Path: <common-issues-return-10193-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22434 invoked from network); 13 Aug 2010 07:38:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Aug 2010 07:38:02 -0000
Received: (qmail 59890 invoked by uid 500); 13 Aug 2010 07:38:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59139 invoked by uid 500); 13 Aug 2010 07:37:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59124 invoked by uid 99); 13 Aug 2010 07:37:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 07:37:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 07:37:56 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7D7bZ4k008489
	for <common-issues@hadoop.apache.org>; Fri, 13 Aug 2010 07:37:36 GMT
Message-ID: <1861157.325111281685055796.JavaMail.jira@thor>
Date: Fri, 13 Aug 2010 03:37:35 -0400 (EDT)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-5303) Oozie, Hadoop Workflow System
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-5303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur resolved HADOOP-5303.
----------------------------------------

    Resolution: Fixed

> Oozie, Hadoop Workflow System
> -----------------------------
>
>                 Key: HADOOP-5303
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5303
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>         Attachments: Hadoop_Summit_Oozie.pdf, hws-preso-v1_0_2009FEB22.pdf, hws-spec2009MAR09.pdf, hws-v1_0_2009FEB22.pdf, oozie-0.18.3.o0.1-SNAPSHOT-distro.tar.gz, oozie-patch.txt, oozie-spec-20090521.pdf, oozie-src-20090605.tar.gz, packages.tar.gz
>
>
> This is a proposal for a system specialized in running Hadoop/Pig jobs in a control dependency DAG (Direct Acyclic Graph), a Hadoop workflow application.
> Attached there is a complete specification and a high level overview presentation.
> ----
> *Highlights* 
> A Workflow application is DAG that coordinates the following types of actions: Hadoop, Pig, Ssh, Http, Email and sub-workflows. 
> Flow control operations within the workflow applications can be done using decision, fork and join nodes. Cycles in workflows are not supported.
> Actions and decisions can be parameterized with job properties, actions output (i.e. Hadoop counters, Ssh key/value pairs output) and file information (file exists, file size, etc). Formal parameters are expressed in the workflow definition as {{${VAR}}} variables.
> A Workflow application is a ZIP file that contains the workflow definition (an XML file), all the necessary files to run all the actions: JAR files for Map/Reduce jobs, shells for streaming Map/Reduce jobs, native libraries, Pig scripts, and other resource files.
> Before running a workflow job, the corresponding workflow application must be deployed in HWS.
> Deploying workflow application and running workflow jobs can be done via command line tools, a WS API and a Java API.
> Monitoring the system and workflow jobs can be done via a web console, command line tools, a WS API and a Java API.
> When submitting a workflow job, a set of properties resolving all the formal parameters in the workflow definitions must be provided. This set of properties is a Hadoop configuration.
> Possible states for a workflow jobs are: {{CREATED}}, {{RUNNING}}, {{SUSPENDED}}, {{SUCCEEDED}}, {{KILLED}} and {{FAILED}}.
> In the case of a action failure in a workflow job, depending on the type of failure, HWS will attempt automatic retries, it will request a manual retry or it will fail the workflow job.
> HWS can make HTTP callback notifications on action start/end/failure events and workflow end/failure events.
> In the case of workflow job failure, the workflow job can be resubmitted skipping previously completed actions. Before doing a resubmission the workflow application could be updated with a patch to fix a problem in the workflow application code.
> ----

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10194-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 13 12:38:45 2010
Return-Path: <common-issues-return-10194-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42597 invoked from network); 13 Aug 2010 12:38:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Aug 2010 12:38:45 -0000
Received: (qmail 84544 invoked by uid 500); 13 Aug 2010 12:38:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84367 invoked by uid 500); 13 Aug 2010 12:38:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84359 invoked by uid 99); 13 Aug 2010 12:38:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 12:38:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 12:38:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7DCcHKD011106
	for <common-issues@hadoop.apache.org>; Fri, 13 Aug 2010 12:38:17 GMT
Message-ID: <17202148.329191281703097037.JavaMail.jira@thor>
Date: Fri, 13 Aug 2010 08:38:17 -0400 (EDT)
From: "Bern Hassmo (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6906) FileContext copy() utility doesn't
 work with recursive copying of directories.
In-Reply-To: <31039707.230631281349276933.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898206#action_12898206 ] 

Bern Hassmo commented on HADOOP-6906:
-------------------------------------

Regarding HADOOP-6872 - just a quick question: Why not provide a listStatus(Path,Filter) method in FilterFs and do the filtering there? Sounds more natural to me, since it is already a filtering fs. Other subclasses of FilterFs might benefit from this as well. Minor detail, just curious why it is in ChecksumFS now. 

> FileContext copy() utility doesn't work with recursive copying of directories.
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6906
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6906
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Vinod K V
>            Assignee: Vinod K V
>         Attachments: HADOOP-6906-20100809.1.txt, HADOOP-6906-regression.1.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10195-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 13 15:56:39 2010
Return-Path: <common-issues-return-10195-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38640 invoked from network); 13 Aug 2010 15:56:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Aug 2010 15:56:39 -0000
Received: (qmail 33071 invoked by uid 500); 13 Aug 2010 15:56:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32980 invoked by uid 500); 13 Aug 2010 15:56:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32971 invoked by uid 99); 13 Aug 2010 15:56:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 15:56:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 15:56:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7DFuHaB013352
	for <common-issues@hadoop.apache.org>; Fri, 13 Aug 2010 15:56:17 GMT
Message-ID: <18389856.333301281714977060.JavaMail.jira@thor>
Date: Fri, 13 Aug 2010 11:56:17 -0400 (EDT)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898287#action_12898287 ] 

Scott Carey commented on HADOOP-6884:
-------------------------------------

One could use something like ASM to re-write log.debug() statements with the guard around it at class load time, or some build system trickery can do something similar at build time.  An AOP tool could do this sort of thing.

I wonder if any logging tool project has already done such a thing?

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10196-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 13 17:23:41 2010
Return-Path: <common-issues-return-10196-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79813 invoked from network); 13 Aug 2010 17:23:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Aug 2010 17:23:41 -0000
Received: (qmail 24053 invoked by uid 500); 13 Aug 2010 17:23:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23969 invoked by uid 500); 13 Aug 2010 17:23:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23961 invoked by uid 99); 13 Aug 2010 17:23:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 17:23:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 17:23:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7DHNJWF014850
	for <common-issues@hadoop.apache.org>; Fri, 13 Aug 2010 17:23:19 GMT
Message-ID: <15471847.334931281720199385.JavaMail.jira@thor>
Date: Fri, 13 Aug 2010 13:23:19 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6904) A baby step towards inter-version
 communications between dfs client and NameNode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898324#action_12898324 ] 

Hairong Kuang commented on HADOOP-6904:
---------------------------------------

Erik, I believe that there are many solutions to version compatible problem. I would be very happy to see a proposal from you taking the capabilities approach.

As the title indicates, my proposal is a baby step towards the solution. The idea is simple and needs least amount of change to the existing code.

> A baby step towards inter-version communications between dfs client and NameNode
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6904
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6904
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: rpcVersion.patch, rpcVersion1.patch
>
>
> Currently RPC communications in Hadoop is very strict. If a client has a different version from that of the server, a VersionMismatched exception is thrown and the client can not connect to the server. This force us to update both client and server all at once if a RPC protocol is changed. But sometime different versions do not mean the client & server are not compatible. It would be nice if we could relax this restriction and allows us to support inter-version communications.
> My idea is that DfsClient catches VersionMismatched exception when it connects to NameNode. It then checks if the client & the server is compatible. If yes, it sets the NameNode version in the dfs client and allows the client to continue talking to NameNode. Otherwise, rethrow the VersionMismatch exception.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10197-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 13 17:59:48 2010
Return-Path: <common-issues-return-10197-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99475 invoked from network); 13 Aug 2010 17:59:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Aug 2010 17:59:48 -0000
Received: (qmail 78951 invoked by uid 500); 13 Aug 2010 17:59:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78886 invoked by uid 500); 13 Aug 2010 17:59:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78878 invoked by uid 99); 13 Aug 2010 17:59:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 17:59:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 17:59:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7DHxN2R015401
	for <common-issues@hadoop.apache.org>; Fri, 13 Aug 2010 17:59:23 GMT
Message-ID: <33504853.336291281722363129.JavaMail.jira@thor>
Date: Fri, 13 Aug 2010 13:59:23 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6904) A baby step towards inter-version
 communications between dfs client and NameNode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898347#action_12898347 ] 

Doug Cutting commented on HADOOP-6904:
--------------------------------------

I'll repeat here my comments from HDFS-1335: This patch seems fine: it's simple and does not alone introduce significant risk.

However subsequent changes to ProtocolCompatible will need to be carefully reviewed and tested.  We don't yet have a testing framework to do that.  So we're delaying that work (interoperability testing of different versions) until we actually have two versions that are meant to interoperate.  This is a technical debt will be due in full the first time ProtocolCompatible is altered.

> A baby step towards inter-version communications between dfs client and NameNode
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6904
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6904
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: rpcVersion.patch, rpcVersion1.patch
>
>
> Currently RPC communications in Hadoop is very strict. If a client has a different version from that of the server, a VersionMismatched exception is thrown and the client can not connect to the server. This force us to update both client and server all at once if a RPC protocol is changed. But sometime different versions do not mean the client & server are not compatible. It would be nice if we could relax this restriction and allows us to support inter-version communications.
> My idea is that DfsClient catches VersionMismatched exception when it connects to NameNode. It then checks if the client & the server is compatible. If yes, it sets the NameNode version in the dfs client and allows the client to continue talking to NameNode. Otherwise, rethrow the VersionMismatch exception.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10198-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 13 19:06:40 2010
Return-Path: <common-issues-return-10198-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40345 invoked from network); 13 Aug 2010 19:06:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Aug 2010 19:06:40 -0000
Received: (qmail 60822 invoked by uid 500); 13 Aug 2010 19:06:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60668 invoked by uid 500); 13 Aug 2010 19:06:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60660 invoked by uid 99); 13 Aug 2010 19:06:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 19:06:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 19:06:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7DJ6IvQ016023
	for <common-issues@hadoop.apache.org>; Fri, 13 Aug 2010 19:06:18 GMT
Message-ID: <24040257.337241281726378509.JavaMail.jira@thor>
Date: Fri, 13 Aug 2010 15:06:18 -0400 (EDT)
From: "Soundararajan Velu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6092) No space left on device
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898371#action_12898371 ] 

Soundararajan Velu commented on HADOOP-6092:
--------------------------------------------

Meng, from what I can guess looking at your cluster configuration and job that you are executing, Is that One of the nodes in your cluster is running out of space, and when map/reduce task tries to write out intermediate result data to its local disks (non HDFS disk space) it is running out of space causing job failures, Though your HDFS cluster had more than 12TB of free space, the temp space on the failed node should have fallen well below limits. just check the free space on the disks holding the temp paths configured for map/reduce.

> No space left on device
> -----------------------
>
>                 Key: HADOOP-6092
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6092
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.19.0
>         Environment: ubuntu0.8.4
>            Reporter: mawanqiang
>
> Exception in thread "main" org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
>         at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
>         at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
>         at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
>         at java.io.FilterOutputStream.close(FilterOutputStream.java:140)
>         at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
>         at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
>         at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.close(ChecksumFileSystem.java:339)
>         at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
>         at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
>         at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:825)
>         at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1142)
>         at org.apache.nutch.indexer.Indexer.index(Indexer.java:72)
>         at org.apache.nutch.indexer.Indexer.run(Indexer.java:92)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>         at org.apache.nutch.indexer.Indexer.main(Indexer.java:101)
> Caused by: java.io.IOException: No space left on device
>         at java.io.FileOutputStream.writeBytes(Native Method)
>         at java.io.FileOutputStream.write(FileOutputStream.java:260)
>         at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:197)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10199-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 13 19:39:40 2010
Return-Path: <common-issues-return-10199-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56060 invoked from network); 13 Aug 2010 19:39:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Aug 2010 19:39:40 -0000
Received: (qmail 256 invoked by uid 500); 13 Aug 2010 19:39:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 154 invoked by uid 500); 13 Aug 2010 19:39:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 146 invoked by uid 99); 13 Aug 2010 19:39:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 19:39:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 19:39:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7DJdIdP016420
	for <common-issues@hadoop.apache.org>; Fri, 13 Aug 2010 19:39:18 GMT
Message-ID: <27172506.338201281728358199.JavaMail.jira@thor>
Date: Fri, 13 Aug 2010 15:39:18 -0400 (EDT)
From: "Meng Mao (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6092) No space left on device
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898387#action_12898387 ] 

Meng Mao commented on HADOOP-6092:
----------------------------------

I see. Yes, our physical disks are definitely full to bursting. Looking on one of the nodes that recently reported several no disk space errors, our configured temp dir (/disk1/hadoop/hadoop-metadata/cache/mapred/local) has 62 GB of stuff in it. 
And only 16GB used by the current job.

There's about 1500 uncleaned attempt_ directories lying around in local/. The earliest have been around since the last restart of our grid. Is it safe to delete these? How do they accumulate?

> No space left on device
> -----------------------
>
>                 Key: HADOOP-6092
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6092
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.19.0
>         Environment: ubuntu0.8.4
>            Reporter: mawanqiang
>
> Exception in thread "main" org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
>         at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
>         at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
>         at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
>         at java.io.FilterOutputStream.close(FilterOutputStream.java:140)
>         at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
>         at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
>         at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.close(ChecksumFileSystem.java:339)
>         at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
>         at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
>         at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:825)
>         at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1142)
>         at org.apache.nutch.indexer.Indexer.index(Indexer.java:72)
>         at org.apache.nutch.indexer.Indexer.run(Indexer.java:92)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>         at org.apache.nutch.indexer.Indexer.main(Indexer.java:101)
> Caused by: java.io.IOException: No space left on device
>         at java.io.FileOutputStream.writeBytes(Native Method)
>         at java.io.FileOutputStream.write(FileOutputStream.java:260)
>         at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:197)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10200-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 13 20:17:44 2010
Return-Path: <common-issues-return-10200-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71549 invoked from network); 13 Aug 2010 20:17:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Aug 2010 20:17:44 -0000
Received: (qmail 55719 invoked by uid 500); 13 Aug 2010 20:17:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55587 invoked by uid 500); 13 Aug 2010 20:17:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55576 invoked by uid 99); 13 Aug 2010 20:17:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 20:17:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 20:17:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7DKHJ49016809
	for <common-issues@hadoop.apache.org>; Fri, 13 Aug 2010 20:17:19 GMT
Message-ID: <32144822.338891281730639255.JavaMail.jira@thor>
Date: Fri, 13 Aug 2010 16:17:19 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6904) A baby step towards inter-version
 communications between dfs client and NameNode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898402#action_12898402 ] 

Sanjay Radia commented on HADOOP-6904:
--------------------------------------

>Could you provide a few examples of compatible changes to ClientProtocol? 
Hairong I have the same question as Nicolas. Can you give a couple of concrete examples of changes to the dfs client
protocol but it continues to remain compatible.

> A baby step towards inter-version communications between dfs client and NameNode
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6904
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6904
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: rpcVersion.patch, rpcVersion1.patch
>
>
> Currently RPC communications in Hadoop is very strict. If a client has a different version from that of the server, a VersionMismatched exception is thrown and the client can not connect to the server. This force us to update both client and server all at once if a RPC protocol is changed. But sometime different versions do not mean the client & server are not compatible. It would be nice if we could relax this restriction and allows us to support inter-version communications.
> My idea is that DfsClient catches VersionMismatched exception when it connects to NameNode. It then checks if the client & the server is compatible. If yes, it sets the NameNode version in the dfs client and allows the client to continue talking to NameNode. Otherwise, rethrow the VersionMismatch exception.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10201-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 13 20:30:44 2010
Return-Path: <common-issues-return-10201-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77029 invoked from network); 13 Aug 2010 20:30:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Aug 2010 20:30:44 -0000
Received: (qmail 79191 invoked by uid 500); 13 Aug 2010 20:30:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79145 invoked by uid 500); 13 Aug 2010 20:30:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79137 invoked by uid 99); 13 Aug 2010 20:30:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 20:30:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 20:30:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7DKUJ0T016917
	for <common-issues@hadoop.apache.org>; Fri, 13 Aug 2010 20:30:19 GMT
Message-ID: <21386660.339171281731419849.JavaMail.jira@thor>
Date: Fri, 13 Aug 2010 16:30:19 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6904) A baby step towards inter-version
 communications between dfs client and NameNode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898407#action_12898407 ] 

Sanjay Radia commented on HADOOP-6904:
--------------------------------------

Sorry I hit <add> by mistake.
For example will be be able to handle addition of new methods. (I don't believe adding new parameters work).
What would be interesting is to loo at the 20 protocol and compare it to the trunck protocol and determine which 
of those changes are compatible.

This also bags the the questions. If the changes were compatible then why did we bump the version number in the first place.
Is it because we have a simple rule: "any changes bump the version number"?

Longer term the I agree with Doug that  using a stronger serialization mechanism like Avro is *part* of the answer.



> A baby step towards inter-version communications between dfs client and NameNode
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6904
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6904
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: rpcVersion.patch, rpcVersion1.patch
>
>
> Currently RPC communications in Hadoop is very strict. If a client has a different version from that of the server, a VersionMismatched exception is thrown and the client can not connect to the server. This force us to update both client and server all at once if a RPC protocol is changed. But sometime different versions do not mean the client & server are not compatible. It would be nice if we could relax this restriction and allows us to support inter-version communications.
> My idea is that DfsClient catches VersionMismatched exception when it connects to NameNode. It then checks if the client & the server is compatible. If yes, it sets the NameNode version in the dfs client and allows the client to continue talking to NameNode. Otherwise, rethrow the VersionMismatch exception.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10202-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 13 20:35:47 2010
Return-Path: <common-issues-return-10202-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78223 invoked from network); 13 Aug 2010 20:35:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Aug 2010 20:35:47 -0000
Received: (qmail 89680 invoked by uid 500); 13 Aug 2010 20:35:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89621 invoked by uid 500); 13 Aug 2010 20:35:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89612 invoked by uid 99); 13 Aug 2010 20:35:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 20:35:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 20:35:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7DKZMI7016982
	for <common-issues@hadoop.apache.org>; Fri, 13 Aug 2010 20:35:22 GMT
Message-ID: <25161130.339361281731721989.JavaMail.jira@thor>
Date: Fri, 13 Aug 2010 16:35:21 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6904) A baby step towards inter-version
 communications between dfs client and NameNode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898410#action_12898410 ] 

Hairong Kuang commented on HADOOP-6904:
---------------------------------------

As a practice, I rewrote HDFS-946, improve listStatus performance, in a backward compatibility way. So the idea is that if the NN is an old server, the client sends the old RPC getListing or getFileInfo, which send back FileStatus in full path name. If NN is a new server, the client send the new RPCs, getHdfsLising or getHdfsFileInfo, which send back HdfsFileStatus that carries only a local name in byte format.

Potentially HDFS-202 could also written in an backward compatible way. If new server, client sends a request getting FileStatus with locations back together; Otherwise, client first get FileStatus then sends separate RPC for block location.

> A baby step towards inter-version communications between dfs client and NameNode
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6904
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6904
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: rpcVersion.patch, rpcVersion1.patch
>
>
> Currently RPC communications in Hadoop is very strict. If a client has a different version from that of the server, a VersionMismatched exception is thrown and the client can not connect to the server. This force us to update both client and server all at once if a RPC protocol is changed. But sometime different versions do not mean the client & server are not compatible. It would be nice if we could relax this restriction and allows us to support inter-version communications.
> My idea is that DfsClient catches VersionMismatched exception when it connects to NameNode. It then checks if the client & the server is compatible. If yes, it sets the NameNode version in the dfs client and allows the client to continue talking to NameNode. Otherwise, rethrow the VersionMismatch exception.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10203-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 13 20:40:40 2010
Return-Path: <common-issues-return-10203-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79169 invoked from network); 13 Aug 2010 20:40:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Aug 2010 20:40:40 -0000
Received: (qmail 92809 invoked by uid 500); 13 Aug 2010 20:40:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92734 invoked by uid 500); 13 Aug 2010 20:40:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92726 invoked by uid 99); 13 Aug 2010 20:40:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 20:40:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 20:40:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7DKeIPv017037
	for <common-issues@hadoop.apache.org>; Fri, 13 Aug 2010 20:40:19 GMT
Message-ID: <3571545.339511281732018645.JavaMail.jira@thor>
Date: Fri, 13 Aug 2010 16:40:18 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6904) A baby step towards inter-version
 communications between dfs client and NameNode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898411#action_12898411 ] 

Hairong Kuang commented on HADOOP-6904:
---------------------------------------

Yes, Sanjay, you got the idea! I am happy to see your thoughtful ideas here.

> If the changes were compatible then why did we bump the version number in the first place.

If we do not bump the version number, old client can continue to talk to a new server, but we also want a new client to talk to an old server.

> A baby step towards inter-version communications between dfs client and NameNode
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6904
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6904
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: rpcVersion.patch, rpcVersion1.patch
>
>
> Currently RPC communications in Hadoop is very strict. If a client has a different version from that of the server, a VersionMismatched exception is thrown and the client can not connect to the server. This force us to update both client and server all at once if a RPC protocol is changed. But sometime different versions do not mean the client & server are not compatible. It would be nice if we could relax this restriction and allows us to support inter-version communications.
> My idea is that DfsClient catches VersionMismatched exception when it connects to NameNode. It then checks if the client & the server is compatible. If yes, it sets the NameNode version in the dfs client and allows the client to continue talking to NameNode. Otherwise, rethrow the VersionMismatch exception.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10204-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 13 20:44:40 2010
Return-Path: <common-issues-return-10204-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81251 invoked from network); 13 Aug 2010 20:44:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Aug 2010 20:44:39 -0000
Received: (qmail 5602 invoked by uid 500); 13 Aug 2010 20:44:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5502 invoked by uid 500); 13 Aug 2010 20:44:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5494 invoked by uid 99); 13 Aug 2010 20:44:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 20:44:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 20:44:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7DKiIMf017096
	for <common-issues@hadoop.apache.org>; Fri, 13 Aug 2010 20:44:18 GMT
Message-ID: <23204045.339681281732258000.JavaMail.jira@thor>
Date: Fri, 13 Aug 2010 16:44:17 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6904) A baby step towards inter-version
 communications between dfs client and NameNode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898413#action_12898413 ] 

Hairong Kuang commented on HADOOP-6904:
---------------------------------------

This framework supports both backward & forward compatibility. :-)

I agree with both you and Doug, this does not exclude Avro. For changes like adding a parameter to a RPC, Avro makes it much simpler.

> A baby step towards inter-version communications between dfs client and NameNode
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6904
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6904
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: rpcVersion.patch, rpcVersion1.patch
>
>
> Currently RPC communications in Hadoop is very strict. If a client has a different version from that of the server, a VersionMismatched exception is thrown and the client can not connect to the server. This force us to update both client and server all at once if a RPC protocol is changed. But sometime different versions do not mean the client & server are not compatible. It would be nice if we could relax this restriction and allows us to support inter-version communications.
> My idea is that DfsClient catches VersionMismatched exception when it connects to NameNode. It then checks if the client & the server is compatible. If yes, it sets the NameNode version in the dfs client and allows the client to continue talking to NameNode. Otherwise, rethrow the VersionMismatch exception.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10205-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 13 20:50:45 2010
Return-Path: <common-issues-return-10205-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82267 invoked from network); 13 Aug 2010 20:50:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Aug 2010 20:50:40 -0000
Received: (qmail 7924 invoked by uid 500); 13 Aug 2010 20:50:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7860 invoked by uid 500); 13 Aug 2010 20:50:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7851 invoked by uid 99); 13 Aug 2010 20:50:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 20:50:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 20:50:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7DKoI7x017146
	for <common-issues@hadoop.apache.org>; Fri, 13 Aug 2010 20:50:19 GMT
Message-ID: <2880803.339811281732618948.JavaMail.jira@thor>
Date: Fri, 13 Aug 2010 16:50:18 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6904) A baby step towards inter-version
 communications between dfs client and NameNode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898414#action_12898414 ] 

Sanjay Radia commented on HADOOP-6904:
--------------------------------------

Slight improvement to your design:
If a new server knows that it is compatible with an older client then it should accept the client's version.
If the client is new then the decision can only be made on the client side. 

> A baby step towards inter-version communications between dfs client and NameNode
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6904
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6904
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: rpcVersion.patch, rpcVersion1.patch
>
>
> Currently RPC communications in Hadoop is very strict. If a client has a different version from that of the server, a VersionMismatched exception is thrown and the client can not connect to the server. This force us to update both client and server all at once if a RPC protocol is changed. But sometime different versions do not mean the client & server are not compatible. It would be nice if we could relax this restriction and allows us to support inter-version communications.
> My idea is that DfsClient catches VersionMismatched exception when it connects to NameNode. It then checks if the client & the server is compatible. If yes, it sets the NameNode version in the dfs client and allows the client to continue talking to NameNode. Otherwise, rethrow the VersionMismatch exception.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10206-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 13 20:56:41 2010
Return-Path: <common-issues-return-10206-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83616 invoked from network); 13 Aug 2010 20:56:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Aug 2010 20:56:41 -0000
Received: (qmail 12040 invoked by uid 500); 13 Aug 2010 20:56:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11966 invoked by uid 500); 13 Aug 2010 20:56:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11958 invoked by uid 99); 13 Aug 2010 20:56:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 20:56:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 20:56:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7DKuJvI017225
	for <common-issues@hadoop.apache.org>; Fri, 13 Aug 2010 20:56:19 GMT
Message-ID: <10303970.340011281732979165.JavaMail.jira@thor>
Date: Fri, 13 Aug 2010 16:56:19 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6904) A baby step towards inter-version
 communications between dfs client and NameNode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898416#action_12898416 ] 

Sanjay Radia commented on HADOOP-6904:
--------------------------------------

>If we do not bump the version number, old client can continue to talk to a new server, but we also want a new client to talk to an old server.
But the new client will get the "now such method" exception and will know that he is talking to an older client.

(BTW I am not at this point suggesting we modify our practice of changing version #s (yet). 
For example with Avro or PBs, if you call a old server with a new method you simple get back a "not such method" exception. 
Correct Doug?


> A baby step towards inter-version communications between dfs client and NameNode
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6904
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6904
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: rpcVersion.patch, rpcVersion1.patch
>
>
> Currently RPC communications in Hadoop is very strict. If a client has a different version from that of the server, a VersionMismatched exception is thrown and the client can not connect to the server. This force us to update both client and server all at once if a RPC protocol is changed. But sometime different versions do not mean the client & server are not compatible. It would be nice if we could relax this restriction and allows us to support inter-version communications.
> My idea is that DfsClient catches VersionMismatched exception when it connects to NameNode. It then checks if the client & the server is compatible. If yes, it sets the NameNode version in the dfs client and allows the client to continue talking to NameNode. Otherwise, rethrow the VersionMismatch exception.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10207-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 13 21:06:40 2010
Return-Path: <common-issues-return-10207-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91023 invoked from network); 13 Aug 2010 21:06:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Aug 2010 21:06:40 -0000
Received: (qmail 21690 invoked by uid 500); 13 Aug 2010 21:06:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21626 invoked by uid 500); 13 Aug 2010 21:06:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21618 invoked by uid 99); 13 Aug 2010 21:06:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 21:06:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 21:06:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7DL6ISZ017362
	for <common-issues@hadoop.apache.org>; Fri, 13 Aug 2010 21:06:18 GMT
Message-ID: <16828224.340231281733578288.JavaMail.jira@thor>
Date: Fri, 13 Aug 2010 17:06:18 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6904) A baby step towards inter-version
 communications between dfs client and NameNode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898418#action_12898418 ] 

Hairong Kuang commented on HADOOP-6904:
---------------------------------------

> But the new client will get the "now such method" exception and will know that he is talking to an older client. 
Sanjay I think you meant the old client will get the "no such method" exception and will know he is talking to an older server.

Yes, this adds one more RPC and does not seem to be elegant. Furthermore,

If a protocol has a method x, if you add an improved version of x, called method y, to the protocol. Later on add method an improved version of y, called method z,  a protocol. Both changes do not bump version #. When a client failed on calling Z, it does not know it should fail back to y or z.

I think bumping up version number is good, clearly indicating a change are made to a protocol. It should be the application itself to decide how to deal with the change in a compatible way.

> A baby step towards inter-version communications between dfs client and NameNode
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6904
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6904
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: rpcVersion.patch, rpcVersion1.patch
>
>
> Currently RPC communications in Hadoop is very strict. If a client has a different version from that of the server, a VersionMismatched exception is thrown and the client can not connect to the server. This force us to update both client and server all at once if a RPC protocol is changed. But sometime different versions do not mean the client & server are not compatible. It would be nice if we could relax this restriction and allows us to support inter-version communications.
> My idea is that DfsClient catches VersionMismatched exception when it connects to NameNode. It then checks if the client & the server is compatible. If yes, it sets the NameNode version in the dfs client and allows the client to continue talking to NameNode. Otherwise, rethrow the VersionMismatch exception.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10208-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 13 21:34:42 2010
Return-Path: <common-issues-return-10208-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 814 invoked from network); 13 Aug 2010 21:34:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Aug 2010 21:34:42 -0000
Received: (qmail 61105 invoked by uid 500); 13 Aug 2010 21:34:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61048 invoked by uid 500); 13 Aug 2010 21:34:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61040 invoked by uid 99); 13 Aug 2010 21:34:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 21:34:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 21:34:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7DLYHpF017589
	for <common-issues@hadoop.apache.org>; Fri, 13 Aug 2010 21:34:17 GMT
Message-ID: <14270283.340731281735257938.JavaMail.jira@thor>
Date: Fri, 13 Aug 2010 17:34:17 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898426#action_12898426 ] 

Erik Steffl commented on HADOOP-6884:
-------------------------------------

Why not committing a simple patch now and work on a good solution later?

Luke: we all prefer fixing the tools. Consider the timeframe and effort required. In the meantime we can have proposed patch as an intermediate solution. Leaving out test code doesn't change the situation. Adding comments is also code bloat (code bloat is mostly relevant for people reading the code) plus it's not guaranteed to work (isDebugEnabled() is guaranteed to work but people will add a marker more or less randomly without considering price of debug call).

Scott: build time - means we wouldn't be able to enable debug at runtime, that's a significant restriction compared to current state and limits troubleshooting capabilities.

I appreciate comments but how about separating a short term quick fix and long term proper solution? At this point it's not clear what a proper solution is. If you want I'll file a jira for long term solution so that we don't forget about it? I mean we probably spent more time discussing this that it take to both create a patch and reverse it (once we have the proper solution).

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10209-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 13 21:58:44 2010
Return-Path: <common-issues-return-10209-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10590 invoked from network); 13 Aug 2010 21:58:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Aug 2010 21:58:43 -0000
Received: (qmail 86105 invoked by uid 500); 13 Aug 2010 21:58:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86096 invoked by uid 500); 13 Aug 2010 21:58:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86088 invoked by uid 99); 13 Aug 2010 21:58:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 21:58:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 21:58:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7DLwIou017775
	for <common-issues@hadoop.apache.org>; Fri, 13 Aug 2010 21:58:19 GMT
Message-ID: <10034916.341131281736698737.JavaMail.jira@thor>
Date: Fri, 13 Aug 2010 17:58:18 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6904) A baby step towards inter-version
 communications between dfs client and NameNode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898434#action_12898434 ] 

Hairong Kuang commented on HADOOP-6904:
---------------------------------------

> Slight improvement to your design:
> If a new server knows that it is compatible with an older client then it should accept the client's version.
> If the client is new then the decision can only be made on the client side. 

Sanjay, this is exactly what I did in my patch to HDFS-1335. Here is the algorithm for getProtocolVersion:

If client is old and server is new
  server checks compatibility. if not compatible, throws VersionIncompatibleException. Connection fails.
  Otherwise, return server version; RPC client will throw a VersionMismatch. DFS client catches VersionMismatch but ignores it because server is newer. Connection succeeds.

if client is newer and server is old
  server simply returns the server version;
  RPC client throws VersionMismatch. DFS client catches VersionMismatch and then performs a compatibility check; If not compatible, throws VersionIncompatible and connection fails. Otherwise, connection succeeds.

> A baby step towards inter-version communications between dfs client and NameNode
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6904
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6904
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: rpcVersion.patch, rpcVersion1.patch
>
>
> Currently RPC communications in Hadoop is very strict. If a client has a different version from that of the server, a VersionMismatched exception is thrown and the client can not connect to the server. This force us to update both client and server all at once if a RPC protocol is changed. But sometime different versions do not mean the client & server are not compatible. It would be nice if we could relax this restriction and allows us to support inter-version communications.
> My idea is that DfsClient catches VersionMismatched exception when it connects to NameNode. It then checks if the client & the server is compatible. If yes, it sets the NameNode version in the dfs client and allows the client to continue talking to NameNode. Otherwise, rethrow the VersionMismatch exception.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10210-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 13 22:08:42 2010
Return-Path: <common-issues-return-10210-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18349 invoked from network); 13 Aug 2010 22:08:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Aug 2010 22:08:42 -0000
Received: (qmail 95139 invoked by uid 500); 13 Aug 2010 22:08:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94766 invoked by uid 500); 13 Aug 2010 22:08:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94722 invoked by uid 99); 13 Aug 2010 22:08:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 22:08:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 22:08:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7DM8HxZ017954
	for <common-issues@hadoop.apache.org>; Fri, 13 Aug 2010 22:08:17 GMT
Message-ID: <4282927.341451281737297620.JavaMail.jira@thor>
Date: Fri, 13 Aug 2010 18:08:17 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898437#action_12898437 ] 

Doug Cutting commented on HADOOP-6884:
--------------------------------------

> code bloat is mostly relevant for people reading the code

Bloat hinders maintainability, a perennial goal.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10211-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 13 22:57:39 2010
Return-Path: <common-issues-return-10211-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30787 invoked from network); 13 Aug 2010 22:57:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Aug 2010 22:57:39 -0000
Received: (qmail 35825 invoked by uid 500); 13 Aug 2010 22:57:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35765 invoked by uid 500); 13 Aug 2010 22:57:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35757 invoked by uid 99); 13 Aug 2010 22:57:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 22:57:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 22:57:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7DMvH2a018412
	for <common-issues@hadoop.apache.org>; Fri, 13 Aug 2010 22:57:17 GMT
Message-ID: <33016253.342561281740237242.JavaMail.jira@thor>
Date: Fri, 13 Aug 2010 18:57:17 -0400 (EDT)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898461#action_12898461 ] 

Scott Carey commented on HADOOP-6884:
-------------------------------------

bq. Scott: build time - means we wouldn't be able to enable debug at runtime, that's a significant restriction compared to current state and limits troubleshooting capabilities.

Thats not what I meant.   I mean that at build time, the source code:
{code}
log.debug("hello " + "world" + "!");
{code}

can be transformed into:
{code}
if (log.isDebugEnabled()) {
  log.debug("hello " + "world" + "!");
}
{code}

The transformation can be at the source code level prior to compilation OR at byte code level post compilation.
Obviously, that is a longer term option.

Do we have any information about how much improved the performance is with the patch here?

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10212-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 13 23:02:42 2010
Return-Path: <common-issues-return-10212-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32654 invoked from network); 13 Aug 2010 23:02:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Aug 2010 23:02:42 -0000
Received: (qmail 38485 invoked by uid 500); 13 Aug 2010 23:02:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38435 invoked by uid 500); 13 Aug 2010 23:02:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38426 invoked by uid 99); 13 Aug 2010 23:02:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 23:02:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 23:02:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7DN2HJu018488
	for <common-issues@hadoop.apache.org>; Fri, 13 Aug 2010 23:02:17 GMT
Message-ID: <15263392.342621281740537493.JavaMail.jira@thor>
Date: Fri, 13 Aug 2010 19:02:17 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898462#action_12898462 ] 

Luke Lu commented on HADOOP-6884:
---------------------------------

bq. Leaving out test code doesn't change the situation.

Why? It would make the patches smaller and quicker to review.

bq. build time - means we wouldn't be able to enable debug at runtime, that's a significant restriction compared to current state and limits troubleshooting capabilities.

AOP can insert if LOG.isDebugEnabled()) guards at build time, so you can definitely enable debug at runtime.

The problem with the current approach is that you'd have to keep patching people's new code manually.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10213-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 13 23:28:41 2010
Return-Path: <common-issues-return-10213-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43616 invoked from network); 13 Aug 2010 23:28:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Aug 2010 23:28:40 -0000
Received: (qmail 54083 invoked by uid 500); 13 Aug 2010 23:28:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54047 invoked by uid 500); 13 Aug 2010 23:28:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54039 invoked by uid 99); 13 Aug 2010 23:28:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 23:28:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 23:28:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7DNSJL8018714
	for <common-issues@hadoop.apache.org>; Fri, 13 Aug 2010 23:28:19 GMT
Message-ID: <17750867.343161281742099098.JavaMail.jira@thor>
Date: Fri, 13 Aug 2010 19:28:19 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6904) A baby step towards inter-version
 communications between dfs client and NameNode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898472#action_12898472 ] 

Sanjay Radia commented on HADOOP-6904:
--------------------------------------

> But the new client will get the "now such method" exception and will know that he is talking to an older client. 
I meant - But the new client will get the "now such method" exception and will know that he is talking to an *older* server. 

> A baby step towards inter-version communications between dfs client and NameNode
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6904
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6904
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: rpcVersion.patch, rpcVersion1.patch
>
>
> Currently RPC communications in Hadoop is very strict. If a client has a different version from that of the server, a VersionMismatched exception is thrown and the client can not connect to the server. This force us to update both client and server all at once if a RPC protocol is changed. But sometime different versions do not mean the client & server are not compatible. It would be nice if we could relax this restriction and allows us to support inter-version communications.
> My idea is that DfsClient catches VersionMismatched exception when it connects to NameNode. It then checks if the client & the server is compatible. If yes, it sets the NameNode version in the dfs client and allows the client to continue talking to NameNode. Otherwise, rethrow the VersionMismatch exception.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10214-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 13 23:36:44 2010
Return-Path: <common-issues-return-10214-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47314 invoked from network); 13 Aug 2010 23:36:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Aug 2010 23:36:43 -0000
Received: (qmail 62379 invoked by uid 500); 13 Aug 2010 23:36:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62324 invoked by uid 500); 13 Aug 2010 23:36:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62316 invoked by uid 99); 13 Aug 2010 23:36:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 23:36:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 23:36:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7DNaI80018775
	for <common-issues@hadoop.apache.org>; Fri, 13 Aug 2010 23:36:18 GMT
Message-ID: <24621568.343321281742578704.JavaMail.jira@thor>
Date: Fri, 13 Aug 2010 19:36:18 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6904) A baby step towards inter-version
 communications between dfs client and NameNode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898474#action_12898474 ] 

Sanjay Radia commented on HADOOP-6904:
--------------------------------------

>Yes, it is going to be a hard coded compatibility list. 
Is it something in the dfs-client or as a config info  that the rpc layer  uses?

Would we be able to use the major and minor version number to automate this.
ie if major numbers mismath then incompatible otherwise compatible.

Clearly the special calls to call older less optimized methods when talking to an old server have to be hard coded in the client side.

> A baby step towards inter-version communications between dfs client and NameNode
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6904
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6904
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: rpcVersion.patch, rpcVersion1.patch
>
>
> Currently RPC communications in Hadoop is very strict. If a client has a different version from that of the server, a VersionMismatched exception is thrown and the client can not connect to the server. This force us to update both client and server all at once if a RPC protocol is changed. But sometime different versions do not mean the client & server are not compatible. It would be nice if we could relax this restriction and allows us to support inter-version communications.
> My idea is that DfsClient catches VersionMismatched exception when it connects to NameNode. It then checks if the client & the server is compatible. If yes, it sets the NameNode version in the dfs client and allows the client to continue talking to NameNode. Otherwise, rethrow the VersionMismatch exception.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10215-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 13 23:58:43 2010
Return-Path: <common-issues-return-10215-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50792 invoked from network); 13 Aug 2010 23:58:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Aug 2010 23:58:43 -0000
Received: (qmail 71057 invoked by uid 500); 13 Aug 2010 23:58:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71000 invoked by uid 500); 13 Aug 2010 23:58:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70992 invoked by uid 99); 13 Aug 2010 23:58:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 23:58:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Aug 2010 23:58:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7DNwMhw018911
	for <common-issues@hadoop.apache.org>; Fri, 13 Aug 2010 23:58:22 GMT
Message-ID: <1564404.343601281743902009.JavaMail.jira@thor>
Date: Fri, 13 Aug 2010 19:58:22 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6904) A baby step towards inter-version
 communications between dfs client and NameNode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898479#action_12898479 ] 

Hairong Kuang commented on HADOOP-6904:
---------------------------------------

> Would we be able to use the major and minor version number to automate this.

I like this idea!

> A baby step towards inter-version communications between dfs client and NameNode
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6904
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6904
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: rpcVersion.patch, rpcVersion1.patch
>
>
> Currently RPC communications in Hadoop is very strict. If a client has a different version from that of the server, a VersionMismatched exception is thrown and the client can not connect to the server. This force us to update both client and server all at once if a RPC protocol is changed. But sometime different versions do not mean the client & server are not compatible. It would be nice if we could relax this restriction and allows us to support inter-version communications.
> My idea is that DfsClient catches VersionMismatched exception when it connects to NameNode. It then checks if the client & the server is compatible. If yes, it sets the NameNode version in the dfs client and allows the client to continue talking to NameNode. Otherwise, rethrow the VersionMismatch exception.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10216-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 14 00:10:38 2010
Return-Path: <common-issues-return-10216-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58251 invoked from network); 14 Aug 2010 00:10:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Aug 2010 00:10:38 -0000
Received: (qmail 79642 invoked by uid 500); 14 Aug 2010 00:10:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79566 invoked by uid 500); 14 Aug 2010 00:10:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79558 invoked by uid 99); 14 Aug 2010 00:10:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Aug 2010 00:10:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Aug 2010 00:10:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7E0AGd7019009
	for <common-issues@hadoop.apache.org>; Sat, 14 Aug 2010 00:10:16 GMT
Message-ID: <24894879.343651281744616527.JavaMail.jira@thor>
Date: Fri, 13 Aug 2010 20:10:16 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6905) Better logging messages when a
 delegation token is invalid
In-Reply-To: <12774985.207161281136756838.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898480#action_12898480 ] 

Devaraj Das commented on HADOOP-6905:
-------------------------------------

+1

> Better logging messages when a delegation token is invalid
> ----------------------------------------------------------
>
>                 Key: HADOOP-6905
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6905
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6905-01.patch
>
>
> From our production logs, we see some logging messages of "token is expired or doesn't exist". It would be helpful to know whose token it was.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10217-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 14 00:14:40 2010
Return-Path: <common-issues-return-10217-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60225 invoked from network); 14 Aug 2010 00:14:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Aug 2010 00:14:40 -0000
Received: (qmail 81528 invoked by uid 500); 14 Aug 2010 00:14:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81502 invoked by uid 500); 14 Aug 2010 00:14:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81494 invoked by uid 99); 14 Aug 2010 00:14:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Aug 2010 00:14:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Aug 2010 00:14:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7E0EGM8019042
	for <common-issues@hadoop.apache.org>; Sat, 14 Aug 2010 00:14:16 GMT
Message-ID: <21815396.343721281744856055.JavaMail.jira@thor>
Date: Fri, 13 Aug 2010 20:14:16 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6912) Guard against NPE when calling
 UGI.isLoginKeytabBased()
In-Reply-To: <29846672.263061281477016561.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898482#action_12898482 ] 

Devaraj Das commented on HADOOP-6912:
-------------------------------------

Could you please check whether the HDFS or MR builds break since you changed the signature of a public method that's probably used by these other projects... (I am guessing it shouldn't but do check).

> Guard against NPE when calling UGI.isLoginKeytabBased()
> -------------------------------------------------------
>
>                 Key: HADOOP-6912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6912
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6912-01.patch
>
>
> NPE can happen when isLoginKeytabBased() is called before a login is performed. See MAPREDUCE-1992 for an example.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10218-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 14 02:22:38 2010
Return-Path: <common-issues-return-10218-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4268 invoked from network); 14 Aug 2010 02:22:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Aug 2010 02:22:37 -0000
Received: (qmail 47873 invoked by uid 500); 14 Aug 2010 02:22:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47731 invoked by uid 500); 14 Aug 2010 02:22:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47723 invoked by uid 99); 14 Aug 2010 02:22:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Aug 2010 02:22:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Aug 2010 02:22:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7E2MGLb019912
	for <common-issues@hadoop.apache.org>; Sat, 14 Aug 2010 02:22:16 GMT
Message-ID: <842899.345251281752536389.JavaMail.jira@thor>
Date: Fri, 13 Aug 2010 22:22:16 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6906) FileContext copy() utility doesn't
 work with recursive copying of directories.
In-Reply-To: <31039707.230631281349276933.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898506#action_12898506 ] 

Hairong Kuang commented on HADOOP-6906:
---------------------------------------

I think that AbstractFileSystem should have an API listStatus(Path, PathFilter).

> FileContext copy() utility doesn't work with recursive copying of directories.
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6906
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6906
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Vinod K V
>            Assignee: Vinod K V
>         Attachments: HADOOP-6906-20100809.1.txt, HADOOP-6906-regression.1.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10219-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 14 06:01:47 2010
Return-Path: <common-issues-return-10219-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67989 invoked from network); 14 Aug 2010 06:01:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Aug 2010 06:01:47 -0000
Received: (qmail 3105 invoked by uid 500); 14 Aug 2010 06:01:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2949 invoked by uid 500); 14 Aug 2010 06:01:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2941 invoked by uid 99); 14 Aug 2010 06:01:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Aug 2010 06:01:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Aug 2010 06:01:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7E61Jop021263
	for <common-issues@hadoop.apache.org>; Sat, 14 Aug 2010 06:01:19 GMT
Message-ID: <5340903.346081281765679204.JavaMail.jira@thor>
Date: Sat, 14 Aug 2010 02:01:19 -0400 (EDT)
From: "Soundararajan Velu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6092) No space left on device
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898515#action_12898515 ] 

Soundararajan Velu commented on HADOOP-6092:
--------------------------------------------

I guess this is a known issue with Map/Reduce, even our clusters ran out of space after a few months due to these attempt files and cache files created by map/reduce. Ideally clean up of these files happen after successful job completion but in this case due to failures these are not getting cleaned up. I think its OK to purge these files when there are no Job's running, 

There are a few open bugs on Mapreduce relating to this issue, you might want to refer to a few of these fixes. 

https://issues.apache.org/jira/browse/MAPREDUCE-1693

If this is the case then you might want to close this against MAPREDUCE-1693.

> No space left on device
> -----------------------
>
>                 Key: HADOOP-6092
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6092
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.19.0
>         Environment: ubuntu0.8.4
>            Reporter: mawanqiang
>
> Exception in thread "main" org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
>         at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
>         at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
>         at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
>         at java.io.FilterOutputStream.close(FilterOutputStream.java:140)
>         at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
>         at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
>         at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.close(ChecksumFileSystem.java:339)
>         at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
>         at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
>         at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:825)
>         at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1142)
>         at org.apache.nutch.indexer.Indexer.index(Indexer.java:72)
>         at org.apache.nutch.indexer.Indexer.run(Indexer.java:92)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>         at org.apache.nutch.indexer.Indexer.main(Indexer.java:101)
> Caused by: java.io.IOException: No space left on device
>         at java.io.FileOutputStream.writeBytes(Native Method)
>         at java.io.FileOutputStream.write(FileOutputStream.java:260)
>         at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:197)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10220-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 14:34:45 2010
Return-Path: <common-issues-return-10220-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17612 invoked from network); 16 Aug 2010 14:34:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 14:34:44 -0000
Received: (qmail 74352 invoked by uid 500); 16 Aug 2010 14:34:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73423 invoked by uid 500); 16 Aug 2010 14:34:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73408 invoked by uid 99); 16 Aug 2010 14:34:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 14:34:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 14:34:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7GEYHJS015946
	for <common-issues@hadoop.apache.org>; Mon, 16 Aug 2010 14:34:17 GMT
Message-ID: <6991431.365871281969257661.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 10:34:17 -0400 (EDT)
From: "Oleg Aleshko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6916) Implement append operation for
 S3FileSystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Implement append operation for S3FileSystem
-------------------------------------------

                 Key: HADOOP-6916
                 URL: https://issues.apache.org/jira/browse/HADOOP-6916
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs/s3
    Affects Versions: 0.20.2
            Reporter: Oleg Aleshko
            Priority: Minor


Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException("Not supported");
S3FileSystem should be able to support appending, possibly via common block storage logic.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10221-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 14:40:48 2010
Return-Path: <common-issues-return-10221-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23607 invoked from network); 16 Aug 2010 14:40:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 14:40:47 -0000
Received: (qmail 85673 invoked by uid 500); 16 Aug 2010 14:40:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85507 invoked by uid 500); 16 Aug 2010 14:40:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85375 invoked by uid 99); 16 Aug 2010 14:40:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 14:40:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 14:40:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7GEeIaR015983
	for <common-issues@hadoop.apache.org>; Mon, 16 Aug 2010 14:40:18 GMT
Message-ID: <17649791.365921281969618123.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 10:40:18 -0400 (EDT)
From: "Oleg Aleshko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6916) Implement append operation for
 S3FileSystem
In-Reply-To: <6991431.365871281969257661.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Oleg Aleshko updated HADOOP-6916:
---------------------------------

    Status: Patch Available  (was: Open)

> Implement append operation for S3FileSystem
> -------------------------------------------
>
>                 Key: HADOOP-6916
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6916
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 0.20.2
>            Reporter: Oleg Aleshko
>            Priority: Minor
>
> Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException("Not supported");
> S3FileSystem should be able to support appending, possibly via common block storage logic.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10222-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 14:40:49 2010
Return-Path: <common-issues-return-10222-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23667 invoked from network); 16 Aug 2010 14:40:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 14:40:47 -0000
Received: (qmail 86916 invoked by uid 500); 16 Aug 2010 14:40:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85530 invoked by uid 500); 16 Aug 2010 14:40:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85493 invoked by uid 99); 16 Aug 2010 14:40:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 14:40:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 14:40:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7GEeI1T015989
	for <common-issues@hadoop.apache.org>; Mon, 16 Aug 2010 14:40:18 GMT
Message-ID: <1391406.365941281969618858.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 10:40:18 -0400 (EDT)
From: "Oleg Aleshko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6916) Implement append operation for
 S3FileSystem
In-Reply-To: <6991431.365871281969257661.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Oleg Aleshko updated HADOOP-6916:
---------------------------------

    Status: Open  (was: Patch Available)

> Implement append operation for S3FileSystem
> -------------------------------------------
>
>                 Key: HADOOP-6916
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6916
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 0.20.2
>            Reporter: Oleg Aleshko
>            Priority: Minor
>
> Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException("Not supported");
> S3FileSystem should be able to support appending, possibly via common block storage logic.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10223-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 14:42:40 2010
Return-Path: <common-issues-return-10223-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24405 invoked from network); 16 Aug 2010 14:42:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 14:42:40 -0000
Received: (qmail 90330 invoked by uid 500); 16 Aug 2010 14:42:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90273 invoked by uid 500); 16 Aug 2010 14:42:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90265 invoked by uid 99); 16 Aug 2010 14:42:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 14:42:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 14:42:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7GEgIei016041
	for <common-issues@hadoop.apache.org>; Mon, 16 Aug 2010 14:42:18 GMT
Message-ID: <200588.366081281969737993.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 10:42:17 -0400 (EDT)
From: "Oleg Aleshko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6916) Implement append operation for
 S3FileSystem
In-Reply-To: <6991431.365871281969257661.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Oleg Aleshko updated HADOOP-6916:
---------------------------------

    Attachment: s3_append1.patch

attached patch, code review needed

> Implement append operation for S3FileSystem
> -------------------------------------------
>
>                 Key: HADOOP-6916
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6916
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 0.20.2
>            Reporter: Oleg Aleshko
>            Priority: Minor
>         Attachments: s3_append1.patch
>
>
> Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException("Not supported");
> S3FileSystem should be able to support appending, possibly via common block storage logic.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10224-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 14:42:44 2010
Return-Path: <common-issues-return-10224-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24514 invoked from network); 16 Aug 2010 14:42:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 14:42:43 -0000
Received: (qmail 90681 invoked by uid 500); 16 Aug 2010 14:42:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90453 invoked by uid 500); 16 Aug 2010 14:42:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90445 invoked by uid 99); 16 Aug 2010 14:42:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 14:42:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 14:42:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7GEgIG3016047
	for <common-issues@hadoop.apache.org>; Mon, 16 Aug 2010 14:42:18 GMT
Message-ID: <7814023.366101281969738675.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 10:42:18 -0400 (EDT)
From: "Oleg Aleshko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6916) Implement append operation for
 S3FileSystem
In-Reply-To: <6991431.365871281969257661.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Oleg Aleshko updated HADOOP-6916:
---------------------------------

    Status: Patch Available  (was: Open)

> Implement append operation for S3FileSystem
> -------------------------------------------
>
>                 Key: HADOOP-6916
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6916
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 0.20.2
>            Reporter: Oleg Aleshko
>            Priority: Minor
>         Attachments: s3_append1.patch
>
>
> Currently org.apache.hadoop.fs.s3.S3FileSystem#append throws an IOException("Not supported");
> S3FileSystem should be able to support appending, possibly via common block storage logic.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10225-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 15:30:53 2010
Return-Path: <common-issues-return-10225-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48021 invoked from network); 16 Aug 2010 15:30:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 15:30:53 -0000
Received: (qmail 57886 invoked by uid 500); 16 Aug 2010 15:30:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57776 invoked by uid 500); 16 Aug 2010 15:30:52 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57768 invoked by uid 99); 16 Aug 2010 15:30:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 15:30:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 15:30:50 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7GFUNNd016658
	for <common-issues@hadoop.apache.org>; Mon, 16 Aug 2010 15:30:24 GMT
Message-ID: <4756941.367081281972623951.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 11:30:23 -0400 (EDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898967#action_12898967 ] 

Steve Loughran commented on HADOOP-6884:
----------------------------------------

-1 to ASM-style work as it just complicates build/test cycles
-0 to SLF4J. I don't see the point. I use commons-logging as the front end with either Log4J or other tools as the back end.

Now, we could improve commons-logging to add varargs...

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10226-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 19:02:52 2010
Return-Path: <common-issues-return-10226-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86825 invoked from network); 16 Aug 2010 19:02:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 19:02:52 -0000
Received: (qmail 84562 invoked by uid 500); 16 Aug 2010 19:02:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84517 invoked by uid 500); 16 Aug 2010 19:02:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84509 invoked by uid 99); 16 Aug 2010 19:02:51 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 19:02:51 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 19:02:48 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7GJ2RET019319
	for <common-issues@hadoop.apache.org>; Mon, 16 Aug 2010 19:02:27 GMT
Message-ID: <30749955.372021281985347587.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 15:02:27 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899034#action_12899034 ] 

Suresh Srinivas commented on HADOOP-6884:
-----------------------------------------

As I indicated in my previous patch, java varargs is not a good idea. It creates object array to pass as varargs.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10227-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 20:57:39 2010
Return-Path: <common-issues-return-10227-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42921 invoked from network); 16 Aug 2010 20:57:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 20:57:39 -0000
Received: (qmail 67610 invoked by uid 500); 16 Aug 2010 20:57:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67550 invoked by uid 500); 16 Aug 2010 20:57:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67542 invoked by uid 99); 16 Aug 2010 20:57:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 20:57:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 20:57:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7GKvH8X020927
	for <common-issues@hadoop.apache.org>; Mon, 16 Aug 2010 20:57:17 GMT
Message-ID: <9416603.375811281992237470.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 16:57:17 -0400 (EDT)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899084#action_12899084 ] 

Scott Carey commented on HADOOP-6884:
-------------------------------------

At most two of these can be true:

1. Code must remain as readable/simple as possible and maintainable (no wrapping isDebugEnabled() in all of source code).
2. Code must perform best (no varargs, autoboxing, string concatenation, etc for unused debug lines)
3. The build system can't re-write code (AOP, source modification, or bytecode modification)


There appear to be "-1"'s that require the above to all be true.  Therefore, this ticket will go nowhere until there is consensus to let one of the three be false.   Implicitly, #2 is false if no action is taken.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10228-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 21:09:44 2010
Return-Path: <common-issues-return-10228-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51740 invoked from network); 16 Aug 2010 21:09:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 21:09:44 -0000
Received: (qmail 78884 invoked by uid 500); 16 Aug 2010 21:09:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78822 invoked by uid 500); 16 Aug 2010 21:09:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78814 invoked by uid 99); 16 Aug 2010 21:09:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 21:09:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 21:09:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7GL9J8C021132
	for <common-issues@hadoop.apache.org>; Mon, 16 Aug 2010 21:09:19 GMT
Message-ID: <1368036.376091281992959661.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 17:09:19 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899089#action_12899089 ] 

Suresh Srinivas commented on HADOOP-6884:
-----------------------------------------

I have additional concerns about AOP. Herriot plans to introduce AOP based interfaces for exposing system internals. This should not be deployed on production clusters. Using AOP to check isDebugEnabled(), results in two flavors of AOP code. One that must be deployed on the system and the other that should not be. Confusion here could result in serious issues.

Second, with AOP, if some one deploys the version without AOP based debug check, there is no way to know if the check is happening or not, by the system behavior alone.

My vote is to keep it simple for now and check in the code changes that Erik has. This is a widely used java coding practice and not a huge code bloat. If others still feel that this causes code bloat, we could further choose one of the choices proposed in the jira. At that time, we need to any way remove all debug enabled checks that are currently being made.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10229-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 21:11:40 2010
Return-Path: <common-issues-return-10229-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52411 invoked from network); 16 Aug 2010 21:11:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 21:11:40 -0000
Received: (qmail 82350 invoked by uid 500); 16 Aug 2010 21:11:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82295 invoked by uid 500); 16 Aug 2010 21:11:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82287 invoked by uid 99); 16 Aug 2010 21:11:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 21:11:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 21:11:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7GLBIah021165
	for <common-issues@hadoop.apache.org>; Mon, 16 Aug 2010 21:11:19 GMT
Message-ID: <17335024.376201281993078831.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 17:11:18 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899091#action_12899091 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6884:
------------------------------------------------

Thanks Scott for a nice [summary|https://issues.apache.org/jira/browse/HADOOP-6884?focusedCommentId=12899084&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12899084].

Actually, #1 is subjective.  I don't think adding an if-statement per LOG.debug(..) make the codes hard to read or maintain.  On the other hand, #2 is very important in production systems.  Source-code/byte-code rewriting mentioned in #3 needs a lot of works.  Why don't we just commit Erik's patch?

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10230-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 21:30:41 2010
Return-Path: <common-issues-return-10230-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62463 invoked from network); 16 Aug 2010 21:30:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 21:30:41 -0000
Received: (qmail 15409 invoked by uid 500); 16 Aug 2010 21:30:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15343 invoked by uid 500); 16 Aug 2010 21:30:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15333 invoked by uid 99); 16 Aug 2010 21:30:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 21:30:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 21:30:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7GLUKmJ021441
	for <common-issues@hadoop.apache.org>; Mon, 16 Aug 2010 21:30:20 GMT
Message-ID: <17873668.376951281994219981.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 17:30:19 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899102#action_12899102 ] 

Jakob Homan commented on HADOOP-6884:
-------------------------------------

The current patch seems to be a bit of a sledge hammer where a ball-peen might be more appropriate.  However, since Java doesn't give us any way of lazily evaluating the arguments to a method, this check does seem to be a standard way of working around this limitation.  I'm a bit surprised at the debug logs in the test cases.  Doesn't it seem more reasonable to do all logging in tests at an always-on level?

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10231-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 21:35:43 2010
Return-Path: <common-issues-return-10231-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66134 invoked from network); 16 Aug 2010 21:35:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 21:35:43 -0000
Received: (qmail 25356 invoked by uid 500); 16 Aug 2010 21:35:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25121 invoked by uid 500); 16 Aug 2010 21:35:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25113 invoked by uid 99); 16 Aug 2010 21:35:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 21:35:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 21:35:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7GLZI2b021506
	for <common-issues@hadoop.apache.org>; Mon, 16 Aug 2010 21:35:18 GMT
Message-ID: <15439193.377111281994518895.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 17:35:18 -0400 (EDT)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899104#action_12899104 ] 

Konstantin Shvachko commented on HADOOP-6884:
---------------------------------------------

Luke> The problem with the current approach is that you'd have to keep patching people's new code manually.

I understand that Erik has a script, which automates the process of finding calls to {{debug()}} unguarded with {{isDebugEnabled()}}. If we run it as a target of ant build, we can enforce people to maintain the guards automatically.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10232-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 21:46:40 2010
Return-Path: <common-issues-return-10232-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69307 invoked from network); 16 Aug 2010 21:46:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 21:46:39 -0000
Received: (qmail 38291 invoked by uid 500); 16 Aug 2010 21:46:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38225 invoked by uid 500); 16 Aug 2010 21:46:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38217 invoked by uid 99); 16 Aug 2010 21:46:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 21:46:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 21:46:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7GLkIYm021645
	for <common-issues@hadoop.apache.org>; Mon, 16 Aug 2010 21:46:18 GMT
Message-ID: <11525031.377491281995178287.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 17:46:18 -0400 (EDT)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899110#action_12899110 ] 

Scott Carey commented on HADOOP-6884:
-------------------------------------

It must also be ensured that isDebugEnabled() is NOT wrapped around an info, warn, error, or fatal.  Someone might check in a change that turns a debug into an info without removing the guard.


> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10233-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 21:56:58 2010
Return-Path: <common-issues-return-10233-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71101 invoked from network); 16 Aug 2010 21:56:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 21:56:57 -0000
Received: (qmail 51050 invoked by uid 500); 16 Aug 2010 21:56:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50995 invoked by uid 500); 16 Aug 2010 21:56:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50987 invoked by uid 99); 16 Aug 2010 21:56:56 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 21:56:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 21:56:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7GLuXRU021936
	for <common-issues@hadoop.apache.org>; Mon, 16 Aug 2010 21:56:33 GMT
Message-ID: <26806169.378291281995793358.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 17:56:33 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899116#action_12899116 ] 

Jakob Homan commented on HADOOP-6884:
-------------------------------------

bq. If we run it as a target of ant build, we can enforce people to maintain the guards automatically.
I'd like to avoid any potential false positives the script may identify.  Checking for appropriate if blocks around costly debug logging should be a responsibility of the patch reviewer.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10234-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 22:08:42 2010
Return-Path: <common-issues-return-10234-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78904 invoked from network); 16 Aug 2010 22:08:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 22:08:42 -0000
Received: (qmail 62249 invoked by uid 500); 16 Aug 2010 22:08:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62160 invoked by uid 500); 16 Aug 2010 22:08:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62152 invoked by uid 99); 16 Aug 2010 22:08:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 22:08:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 22:08:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7GM8HAa022137
	for <common-issues@hadoop.apache.org>; Mon, 16 Aug 2010 22:08:18 GMT
Message-ID: <14551655.378651281996497954.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 18:08:17 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899127#action_12899127 ] 

Luke Lu commented on HADOOP-6884:
---------------------------------

We can save ~296 lines from the patch if we exclude the test code:

{code}
[...hadoop]$ find */src/java -name \*.java | xargs grep '\.debug(' | wc -l
     410
[...hadoop]$ find */src/test -name \*.java | xargs grep '\.debug(' | wc -l
     148
{code}

Also #3 seems to be fairly simple with a single "around" advice in aspectj and a couple of lines for ajc target in build.xml. Why use an ad-hoc script that's known (according Erik himself) to have false positive that require manual inspection (which is also not part of the version controlled source tree), when you we have a built-in (aspectj*.jars are already included as build/runtime dependencies) tool specifically for such tasks?

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10235-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 22:35:44 2010
Return-Path: <common-issues-return-10235-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88200 invoked from network); 16 Aug 2010 22:35:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 22:35:43 -0000
Received: (qmail 83723 invoked by uid 500); 16 Aug 2010 22:35:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83647 invoked by uid 500); 16 Aug 2010 22:35:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83639 invoked by uid 99); 16 Aug 2010 22:35:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 22:35:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 22:35:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7GMZLJV022606
	for <common-issues@hadoop.apache.org>; Mon, 16 Aug 2010 22:35:22 GMT
Message-ID: <1100279.379881281998121884.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 18:35:21 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899145#action_12899145 ] 

Doug Cutting commented on HADOOP-6884:
--------------------------------------

> 1. Code must remain as readable/simple as possible and maintainable (no wrapping isDebugEnabled() in all of source code).

Does anyone believe that?  I think the guards should ideally be limited to performance sensitive parts of the code.  But if folks don't trust that can be maintained, then a warning when non-constant strings are logged might be acceptable.  Constant strings should be as fast as the guard.

> 2. Code must perform best (no varargs, autoboxing, string concatenation, etc for unused debug lines)

We could easily implement a guard that both performs well and does not bloat code, e.g.:
{code}
public static void debugLog(String message) { LOG.debug(message); }
public static void debugLog(String message, Object o1) {
  if (LOG.isDebugEnabled())
    LOG.debug(MessageFormat.format(message, new Object[] {o1}));
}
public static void debugLog(String message, Object o1, Object o2) {
  if (LOG.isDebugEnabled())
    LOG.debug(MessageFormat.format(message, new Object[] {o1, o2}));
}
public static void debugLog(String message, Object o1, Object o2, Object o3) {
  if (LOG.isDebugEnabled())
    LOG.debug(MessageFormat.format(message, new Object[] {o1, o2, o3}));
}
...
{code}

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10236-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 22:39:42 2010
Return-Path: <common-issues-return-10236-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88671 invoked from network); 16 Aug 2010 22:39:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 22:39:42 -0000
Received: (qmail 86293 invoked by uid 500); 16 Aug 2010 22:39:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86261 invoked by uid 500); 16 Aug 2010 22:39:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86253 invoked by uid 99); 16 Aug 2010 22:39:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 22:39:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 22:39:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7GMdHNf022655
	for <common-issues@hadoop.apache.org>; Mon, 16 Aug 2010 22:39:18 GMT
Message-ID: <28098284.379981281998357757.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 18:39:17 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899147#action_12899147 ] 

Luke Lu commented on HADOOP-6884:
---------------------------------

@Steve & Suresh: slf4j support "{}" holders in the format string (which has a much faster implementation than jdk MessageFormat), so we don't have to create new strings objects with all these + operators and/or implicit toString() calls. One can create a few overloads (for number of args under n, and delegate to vargs a la slf4j (n=3) for larger n) to avoid vargs (hence the new Object array) overhead in common cases. Feature/design/performance aside, the main problem of commons logging is that there are no active maintainers working on it (last release in 2007), unlike slf4j.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10237-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 22:48:41 2010
Return-Path: <common-issues-return-10237-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91919 invoked from network); 16 Aug 2010 22:48:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 22:48:41 -0000
Received: (qmail 3204 invoked by uid 500); 16 Aug 2010 22:48:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3157 invoked by uid 500); 16 Aug 2010 22:48:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3149 invoked by uid 99); 16 Aug 2010 22:48:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 22:48:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 22:48:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7GMmJoA022786
	for <common-issues@hadoop.apache.org>; Mon, 16 Aug 2010 22:48:19 GMT
Message-ID: <13683530.380331281998899926.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 18:48:19 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6907) Rpc client doesn't use the
 per-connection conf to figure out server's Kerberos principal
In-Reply-To: <9029429.243131281389838773.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6907:
------------------------------

    Attachment: c6907-12.patch

This patch does the following.

1. It refactored a bunch of connection-related params from being global (stay the same for new connections) to become per-connection params and their values are obtained from the conf used when calling RPC.getProxy(). This allows new connections to be set up based on the conf used for that connection, which is more intuitive.

2. In particular, the server's Kerberos principal is now obtained from the per connection conf.

3. The conf cached globally in Client is not removed since it provides a cache for Classes that ObjectWritable.NullInstance relies on to figure out the declaredClass. It seems to provide a useful function and the cache can be shared among connections. No per-connection params are obtained from this global conf.

> Rpc client doesn't use the per-connection conf to figure out server's Kerberos principal
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6907
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6907
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6907-12.patch
>
>
> Currently, RPC client caches the conf that was passed in to its constructor and uses that same conf (or values obtained from it) for every connection it sets up. This is not sufficient for security since each connection needs to figure out server's Kerberos principal on a per-connection basis. It's not reasonable to expect the first conf used by a user to contain all the Kerberos principals that her future connections will ever need. Or worse, if her first conf contains an incorrect principal name, it will prevent the user from connecting to the server even if she later on passes in a correct conf on retry (by calling RPC.getProxy()).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10238-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 22:48:42 2010
Return-Path: <common-issues-return-10238-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91963 invoked from network); 16 Aug 2010 22:48:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 22:48:42 -0000
Received: (qmail 3354 invoked by uid 500); 16 Aug 2010 22:48:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3325 invoked by uid 500); 16 Aug 2010 22:48:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3284 invoked by uid 99); 16 Aug 2010 22:48:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 22:48:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 22:48:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7GMmKMO022807
	for <common-issues@hadoop.apache.org>; Mon, 16 Aug 2010 22:48:20 GMT
Message-ID: <19124149.380401281998900885.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 18:48:20 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6907) Rpc client doesn't use the
 per-connection conf to figure out server's Kerberos principal
In-Reply-To: <9029429.243131281389838773.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6907:
------------------------------

    Status: Patch Available  (was: Open)

> Rpc client doesn't use the per-connection conf to figure out server's Kerberos principal
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6907
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6907
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6907-12.patch
>
>
> Currently, RPC client caches the conf that was passed in to its constructor and uses that same conf (or values obtained from it) for every connection it sets up. This is not sufficient for security since each connection needs to figure out server's Kerberos principal on a per-connection basis. It's not reasonable to expect the first conf used by a user to contain all the Kerberos principals that her future connections will ever need. Or worse, if her first conf contains an incorrect principal name, it will prevent the user from connecting to the server even if she later on passes in a correct conf on retry (by calling RPC.getProxy()).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10239-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 23:08:44 2010
Return-Path: <common-issues-return-10239-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2914 invoked from network); 16 Aug 2010 23:08:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 23:08:43 -0000
Received: (qmail 21366 invoked by uid 500); 16 Aug 2010 23:08:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21271 invoked by uid 500); 16 Aug 2010 23:08:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21263 invoked by uid 99); 16 Aug 2010 23:08:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 23:08:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 23:08:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7GN8J6x023100
	for <common-issues@hadoop.apache.org>; Mon, 16 Aug 2010 23:08:19 GMT
Message-ID: <5511292.381021282000099260.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 19:08:19 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6889) Make RPC to have an option to
 timeout
In-Reply-To: <30758566.75561280434516176.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899171#action_12899171 ] 

Kan Zhang commented on HADOOP-6889:
-----------------------------------

This patch's hashcode() implementation also changed the UGI semantics. It used to be that two UGI's having the same subject object are equal. Now they are not equal. I hope to fix both issues in HADOOP-6907. 

> Make RPC to have an option to timeout
> -------------------------------------
>
>                 Key: HADOOP-6889
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6889
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0, 0.20-append
>
>         Attachments: ipcTimeout.patch, ipcTimeout1.patch, ipcTimeout2.patch
>
>
> Currently Hadoop RPC does not timeout when the RPC server is alive. What it currently does is that a RPC client sends a ping to the server whenever a socket timeout happens. If the server is still alive, it continues to wait instead of throwing a SocketTimeoutException. This is to avoid a client to retry when a server is busy and thus making the server even busier. This works great if the RPC server is NameNode.
> But Hadoop RPC is also used for some of client to DataNode communications, for example, for getting a replica's length. When a client comes across a problematic DataNode, it gets stuck and can not switch to a different DataNode. In this case, it would be better that the client receives a timeout exception.
> I plan to add a new configuration ipc.client.max.pings that specifies the max number of pings that a client could try. If a response can not be received after the specified max number of pings, a SocketTimeoutException is thrown. If this configuration property is not set, a client maintains the current semantics, waiting forever.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10240-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 23:39:38 2010
Return-Path: <common-issues-return-10240-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21189 invoked from network); 16 Aug 2010 23:39:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 23:39:38 -0000
Received: (qmail 45302 invoked by uid 500); 16 Aug 2010 23:39:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45203 invoked by uid 500); 16 Aug 2010 23:39:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45194 invoked by uid 99); 16 Aug 2010 23:39:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 23:39:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 23:39:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7GNdGv6023664
	for <common-issues@hadoop.apache.org>; Mon, 16 Aug 2010 23:39:16 GMT
Message-ID: <27417627.382681282001956560.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 19:39:16 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6912) Guard against NPE when calling
 UGI.isLoginKeytabBased()
In-Reply-To: <29846672.263061281477016561.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899191#action_12899191 ] 

Kan Zhang commented on HADOOP-6912:
-----------------------------------

I checked. This method is currently not used in either HDFS or MR trunk.

> Guard against NPE when calling UGI.isLoginKeytabBased()
> -------------------------------------------------------
>
>                 Key: HADOOP-6912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6912
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6912-01.patch
>
>
> NPE can happen when isLoginKeytabBased() is called before a login is performed. See MAPREDUCE-1992 for an example.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10241-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 23:39:39 2010
Return-Path: <common-issues-return-10241-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21232 invoked from network); 16 Aug 2010 23:39:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 23:39:38 -0000
Received: (qmail 45427 invoked by uid 500); 16 Aug 2010 23:39:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45221 invoked by uid 500); 16 Aug 2010 23:39:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45204 invoked by uid 99); 16 Aug 2010 23:39:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 23:39:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 23:39:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7GNdGdp023670
	for <common-issues@hadoop.apache.org>; Mon, 16 Aug 2010 23:39:16 GMT
Message-ID: <13049776.382701282001956808.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 19:39:16 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6912) Guard against NPE when calling
 UGI.isLoginKeytabBased()
In-Reply-To: <29846672.263061281477016561.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6912:
------------------------------

    Status: Open  (was: Patch Available)

> Guard against NPE when calling UGI.isLoginKeytabBased()
> -------------------------------------------------------
>
>                 Key: HADOOP-6912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6912
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6912-01.patch
>
>
> NPE can happen when isLoginKeytabBased() is called before a login is performed. See MAPREDUCE-1992 for an example.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10242-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 23:39:39 2010
Return-Path: <common-issues-return-10242-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21248 invoked from network); 16 Aug 2010 23:39:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 23:39:38 -0000
Received: (qmail 45465 invoked by uid 500); 16 Aug 2010 23:39:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45379 invoked by uid 500); 16 Aug 2010 23:39:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45212 invoked by uid 99); 16 Aug 2010 23:39:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 23:39:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 23:39:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7GNdGfW023676
	for <common-issues@hadoop.apache.org>; Mon, 16 Aug 2010 23:39:17 GMT
Message-ID: <32942961.382721282001956982.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 19:39:16 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6912) Guard against NPE when calling
 UGI.isLoginKeytabBased()
In-Reply-To: <29846672.263061281477016561.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6912:
------------------------------

    Status: Patch Available  (was: Open)

> Guard against NPE when calling UGI.isLoginKeytabBased()
> -------------------------------------------------------
>
>                 Key: HADOOP-6912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6912
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6912-01.patch
>
>
> NPE can happen when isLoginKeytabBased() is called before a login is performed. See MAPREDUCE-1992 for an example.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10243-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 23:44:41 2010
Return-Path: <common-issues-return-10243-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22437 invoked from network); 16 Aug 2010 23:44:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 23:44:40 -0000
Received: (qmail 48134 invoked by uid 500); 16 Aug 2010 23:44:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47621 invoked by uid 500); 16 Aug 2010 23:44:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47408 invoked by uid 99); 16 Aug 2010 23:44:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 23:44:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 23:44:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7GNiIm8023734
	for <common-issues@hadoop.apache.org>; Mon, 16 Aug 2010 23:44:18 GMT
Message-ID: <28272967.382801282002258342.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 19:44:18 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6913) Circular initialization between
 UserGroupInformation and KerberosName
In-Reply-To: <27327567.281671281550577971.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6913:
------------------------------

    Status: Open  (was: Patch Available)

> Circular initialization between UserGroupInformation and KerberosName
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6913
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6913
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6913-01.patch
>
>
> If the first call to UGI is UGI.setConfiguration(conf), it will try to initialize UGI class. During this initialization, the code calls KerberosName.setConfiguration(). KerberosName's static initializer will in turn call UGI.isSecurityEnabled(). Since UGI hasn't been completely initialized yet, isSecurityEnabled() will re-initialize UGI with a DEFAULT conf. As a result, the original conf used in UGI.setConfiguration(conf) will be overwritten by the DEFAULT conf.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10244-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 16 23:44:42 2010
Return-Path: <common-issues-return-10244-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22472 invoked from network); 16 Aug 2010 23:44:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Aug 2010 23:44:40 -0000
Received: (qmail 48518 invoked by uid 500); 16 Aug 2010 23:44:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48141 invoked by uid 500); 16 Aug 2010 23:44:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47470 invoked by uid 99); 16 Aug 2010 23:44:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 23:44:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Aug 2010 23:44:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7GNiI5k023740
	for <common-issues@hadoop.apache.org>; Mon, 16 Aug 2010 23:44:18 GMT
Message-ID: <25756799.382821282002258852.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 19:44:18 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6913) Circular initialization between
 UserGroupInformation and KerberosName
In-Reply-To: <27327567.281671281550577971.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6913:
------------------------------

    Status: Patch Available  (was: Open)

> Circular initialization between UserGroupInformation and KerberosName
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6913
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6913
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6913-01.patch
>
>
> If the first call to UGI is UGI.setConfiguration(conf), it will try to initialize UGI class. During this initialization, the code calls KerberosName.setConfiguration(). KerberosName's static initializer will in turn call UGI.isSecurityEnabled(). Since UGI hasn't been completely initialized yet, isSecurityEnabled() will re-initialize UGI with a DEFAULT conf. As a result, the original conf used in UGI.setConfiguration(conf) will be overwritten by the DEFAULT conf.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10245-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 01:23:40 2010
Return-Path: <common-issues-return-10245-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76932 invoked from network); 17 Aug 2010 01:23:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 01:23:40 -0000
Received: (qmail 37581 invoked by uid 500); 17 Aug 2010 01:23:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37533 invoked by uid 500); 17 Aug 2010 01:23:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37525 invoked by uid 99); 17 Aug 2010 01:23:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 01:23:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 01:23:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7H1NIhI024581
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 01:23:18 GMT
Message-ID: <8932584.384371282008198270.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 21:23:18 -0400 (EDT)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899228#action_12899228 ] 

Scott Carey commented on HADOOP-6884:
-------------------------------------

bq. Also #3 seems to be fairly simple with a single "around" advice in aspectj and a couple of lines for ajc target in build.xml. Why use an ad-hoc script that's known (according Erik himself) to have false positive that require manual inspection (which is also not part of the version controlled source tree), when you we have a built-in (aspectj*.jars are already included as build/runtime dependencies) tool specifically for such tasks?

You can do this for info messages too for more performance when that is off.  There are no extra lines of code or maintenance; it is the simplest solution.  As long as the change is applied at build time, I don't see any real drawback.



> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10246-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 01:31:38 2010
Return-Path: <common-issues-return-10246-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80051 invoked from network); 17 Aug 2010 01:31:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 01:31:37 -0000
Received: (qmail 42785 invoked by uid 500); 17 Aug 2010 01:31:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42735 invoked by uid 500); 17 Aug 2010 01:31:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42727 invoked by uid 99); 17 Aug 2010 01:31:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 01:31:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 01:31:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7H1VGMK024598
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 01:31:16 GMT
Message-ID: <18677733.384391282008676257.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 21:31:16 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6885) Fix java doc warnings in Groups and
 RefreshUserMappingsProtocol
In-Reply-To: <14122960.38041280281637802.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899229#action_12899229 ] 

Jakob Homan commented on HADOOP-6885:
-------------------------------------

Eli-
   Funny, I don't get these as warnings when I run {{ant javadoc}}.  I do however, get:
{noformat}
  [javadoc] /Users/jhoman/work/git/hadoop-common/src/java/org/apache/hadoop/ipc/Client.java:945: warning - Tag @link: 
can't find call(Writable, InetSocketAddress, Class, UserGroupInformation) in org.apache.hadoop.ipc.Client
  [javadoc] /Users/jhoman/work/git/hadoop-common/src/java/org/apache/hadoop/ipc/Client.java:958: warning - Tag @link: 
can't find call(Writable, InetSocketAddress, Class, UserGroupInformation) in org.apache.hadoop.ipc.Client
{noformat}

Could you update the patch to fix these warnings as well and I'll +1?

> Fix java doc warnings in Groups and RefreshUserMappingsProtocol
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6885
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6885
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.20.2
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6885-1.patch
>
>
> There are a couple java docs warnings in Groups and RefreshUserMappingsProtocol.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10247-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 01:59:37 2010
Return-Path: <common-issues-return-10247-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87822 invoked from network); 17 Aug 2010 01:59:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 01:59:37 -0000
Received: (qmail 60395 invoked by uid 500); 17 Aug 2010 01:59:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60364 invoked by uid 500); 17 Aug 2010 01:59:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60355 invoked by uid 99); 17 Aug 2010 01:59:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 01:59:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 01:59:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7H1xGIN024847
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 01:59:16 GMT
Message-ID: <10509062.384931282010356110.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 21:59:16 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6885) Fix java doc warnings in Groups and
 RefreshUserMappingsProtocol
In-Reply-To: <14122960.38041280281637802.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6885:
--------------------------------

    Status: Open  (was: Patch Available)

Canceling patch post-review.

> Fix java doc warnings in Groups and RefreshUserMappingsProtocol
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6885
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6885
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.20.2
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6885-1.patch
>
>
> There are a couple java docs warnings in Groups and RefreshUserMappingsProtocol.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10248-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 02:41:39 2010
Return-Path: <common-issues-return-10248-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6036 invoked from network); 17 Aug 2010 02:41:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 02:41:39 -0000
Received: (qmail 91497 invoked by uid 500); 17 Aug 2010 02:41:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91341 invoked by uid 500); 17 Aug 2010 02:41:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91331 invoked by uid 99); 17 Aug 2010 02:41:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 02:41:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 02:41:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7H2fGEC025170
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 02:41:16 GMT
Message-ID: <30123252.385491282012876681.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 22:41:16 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6905) Better logging messages when a
 delegation token is invalid
In-Reply-To: <12774985.207161281136756838.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6905:
--------------------------------

           Status: Resolved  (was: Patch Available)
     Hadoop Flags: [Incompatible change, Reviewed]
    Fix Version/s: 0.22.0
       Resolution: Fixed

I've committed this, although I just noticed that Hudson hadn't run.  Wandered off again, apparently.  It's a trivial patch, so this should be ok.  Resolving as fixed.  Thanks, Kan.

> Better logging messages when a delegation token is invalid
> ----------------------------------------------------------
>
>                 Key: HADOOP-6905
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6905
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>             Fix For: 0.22.0
>
>         Attachments: c6905-01.patch
>
>
> From our production logs, we see some logging messages of "token is expired or doesn't exist". It would be helpful to know whose token it was.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10249-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 03:01:41 2010
Return-Path: <common-issues-return-10249-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13398 invoked from network); 17 Aug 2010 03:01:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 03:01:41 -0000
Received: (qmail 23898 invoked by uid 500); 17 Aug 2010 03:01:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23820 invoked by uid 500); 17 Aug 2010 03:01:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23812 invoked by uid 99); 17 Aug 2010 03:01:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 03:01:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 03:01:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7H31HV1025414
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 03:01:17 GMT
Message-ID: <29113558.385981282014077666.JavaMail.jira@thor>
Date: Mon, 16 Aug 2010 23:01:17 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6905) Better logging messages when a
 delegation token is invalid
In-Reply-To: <12774985.207161281136756838.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899257#action_12899257 ] 

Hudson commented on HADOOP-6905:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #358 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/358/])
    HADOOP-6905. Better logging messages when a delegation token is invalid. Contributed by Kan Zhang.


> Better logging messages when a delegation token is invalid
> ----------------------------------------------------------
>
>                 Key: HADOOP-6905
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6905
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>             Fix For: 0.22.0
>
>         Attachments: c6905-01.patch
>
>
> From our production logs, we see some logging messages of "token is expired or doesn't exist". It would be helpful to know whose token it was.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10250-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 04:07:44 2010
Return-Path: <common-issues-return-10250-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39305 invoked from network); 17 Aug 2010 04:07:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 04:07:44 -0000
Received: (qmail 11737 invoked by uid 500); 17 Aug 2010 04:07:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11585 invoked by uid 500); 17 Aug 2010 04:07:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11559 invoked by uid 99); 17 Aug 2010 04:07:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 04:07:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 04:07:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7H47HH2026239
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 04:07:17 GMT
Message-ID: <6031122.386641282018037183.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 00:07:17 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6101) CDPATH environment variable causes
 bin scripts to fail.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899263#action_12899263 ] 

Eli Collins commented on HADOOP-6101:
-------------------------------------

What version of bash are you using and what's your CDPATH?  Looking at the way the bin scripts from 19 to trunk, cd is given an absolute path, ".", or one that starts with "./"  from dirname, and bash won't use CDPATH (and print to stdout) if that's the case doesn't seem to use CDPATH used to stdout if that's the case (even though the manual claims it . 

{code}
bin=`dirname "$0"`   # or bin=`dirname "${BASH_SOURCE-$0}"`
bin=`cd "$bin"; pwd`
{code}

1. http://www.gnu.org/software/bash/manual/html_node/Bourne-Shell-Builtins.html
2. http://www.opengroup.org/onlinepubs/009695399/utilities/cd.html

We can unset CDPATH in the bin scripts but it doesn't look like it should fix anything.


> CDPATH environment variable causes bin scripts to fail.
> -------------------------------------------------------
>
>                 Key: HADOOP-6101
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6101
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.19.1
>            Reporter: Phil Hagelberg
>            Priority: Minor
>
> Most of the scripts in bin/* assume that cd produces no output. But when using bash (and some other shells) cd will output the destination directory if the CDPATH environment variable is set. CDPATH is very useful, and it's unfortunate to have to unset it to use Hadoop.
> The offending line (in start-all.sh, though most of the scripts exhibit the problem) is:
>   bin=`cd "$bin"; pwd`
> Adding this to the top of each affected script will fix the problem:
>   unset CDPATH

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10251-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 06:16:50 2010
Return-Path: <common-issues-return-10251-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85014 invoked from network); 17 Aug 2010 06:16:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 06:16:50 -0000
Received: (qmail 99227 invoked by uid 500); 17 Aug 2010 06:16:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98960 invoked by uid 500); 17 Aug 2010 06:16:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98946 invoked by uid 99); 17 Aug 2010 06:16:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 06:16:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 06:16:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7H6GOnt027228
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 06:16:24 GMT
Message-ID: <16076739.388541282025784817.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 02:16:24 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6693) Add metrics to track kerberos login
 activity
In-Reply-To: <1138587062.22291270763196714.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6693:
------------------------------------

    Status: Patch Available  (was: Open)

> Add metrics to track kerberos login activity
> --------------------------------------------
>
>                 Key: HADOOP-6693
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6693
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6693.patch, HADOOP-6693.rel20.1.patch, HADOOP-6693.rel20.patch
>
>
> Need metrics to track kerberos login activity such as login rate and the time taken for login.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10252-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 06:17:05 2010
Return-Path: <common-issues-return-10252-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85130 invoked from network); 17 Aug 2010 06:17:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 06:17:04 -0000
Received: (qmail 99290 invoked by uid 500); 17 Aug 2010 06:17:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99262 invoked by uid 500); 17 Aug 2010 06:17:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99254 invoked by uid 99); 17 Aug 2010 06:17:02 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 06:17:02 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 06:16:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7H6GO5T027216
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 06:16:24 GMT
Message-ID: <16151336.388501282025784296.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 02:16:24 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6693) Add metrics to track kerberos login
 activity
In-Reply-To: <1138587062.22291270763196714.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6693:
------------------------------------

    Status: Open  (was: Patch Available)

> Add metrics to track kerberos login activity
> --------------------------------------------
>
>                 Key: HADOOP-6693
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6693
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6693.patch, HADOOP-6693.rel20.1.patch, HADOOP-6693.rel20.patch
>
>
> Need metrics to track kerberos login activity such as login rate and the time taken for login.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10253-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 09:07:43 2010
Return-Path: <common-issues-return-10253-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57014 invoked from network); 17 Aug 2010 09:07:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 09:07:43 -0000
Received: (qmail 67282 invoked by uid 500); 17 Aug 2010 09:07:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67123 invoked by uid 500); 17 Aug 2010 09:07:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67094 invoked by uid 99); 17 Aug 2010 09:07:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 09:07:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 09:07:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7H97IYu028954
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 09:07:18 GMT
Message-ID: <19212529.391331282036038517.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 05:07:18 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6863) Compile-native still fails on Mac
 OS X.
In-Reply-To: <18053660.404611279230589983.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899350#action_12899350 ] 

Hong Tang commented on HADOOP-6863:
-----------------------------------

bq. Ok, I will test it with snow leopard and see if it works.
ant compile-native and ant test-core pass on Snow Leopard.

> Compile-native still fails on Mac OS X.
> ---------------------------------------
>
>                 Key: HADOOP-6863
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6863
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Hong Tang
>            Assignee: Hong Tang
>         Attachments: hadoop-6863-20100715.patch, hadoop-6863-20100721.patch
>
>
> I am still getting failures when I try to compile native library on Mac OS X (10.5.8 Leopard). The problems are two fold:
> - Although "aclocal.m4" is changed after HADOOP-3659 is committed, the corresponding "configure" script is not re-generated.
> - It cannot find libjvm.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10254-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 09:07:46 2010
Return-Path: <common-issues-return-10254-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57066 invoked from network); 17 Aug 2010 09:07:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 09:07:46 -0000
Received: (qmail 67339 invoked by uid 500); 17 Aug 2010 09:07:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67311 invoked by uid 500); 17 Aug 2010 09:07:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67302 invoked by uid 99); 17 Aug 2010 09:07:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 09:07:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 09:07:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7H97Kwo028987
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 09:07:20 GMT
Message-ID: <23055230.391441282036040734.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 05:07:20 -0400 (EDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899352#action_12899352 ] 

Steve Loughran commented on HADOOP-6884:
----------------------------------------

Luke Lu _said Feature/design/performance aside, the main problem of commons logging is that there are no active maintainers working on it (last release in 2007), unlike slf4j. _

We can fix that, if the project is idle it could be taken over, but I fear the #of open bugreps. The thought appeals though. Strip out avalon support, pare back the POM files and assume log4j and java.util are the two backends. 

Maybe I've just had too much negative experience of SLF4J the redistributable, rather than as an API.
# it's another JAR
# you also need to choose at redist time what your back end will be and include that JAR
# If you don't want to log things like Jetty through it you have to write your own bridge specifically for that app [1] and make sure it's on the classpath.

so: I get to deal with the downstream problems, things not logging, things not logging in forked JVMs, etc, the SLF4J api is something I have no knowledge of.

Of course, the other tactic is to say "no layers, go to log4j" and work with the log4j team on having log4j push out data to chukwa and scribus, feed it back into Hadoop for analysis. Just a thought.

[1] http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/components/jetty/src/org/smartfrog/services/jetty/log/JettyToCommonsLogger.java?revision=8363&view=markup


> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10255-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 11:23:42 2010
Return-Path: <common-issues-return-10255-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6899 invoked from network); 17 Aug 2010 11:23:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 11:23:42 -0000
Received: (qmail 27976 invoked by uid 500); 17 Aug 2010 11:23:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27830 invoked by uid 500); 17 Aug 2010 11:23:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27819 invoked by uid 99); 17 Aug 2010 11:23:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 11:23:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 11:23:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HBNHZL029919
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 11:23:17 GMT
Message-ID: <14213939.392161282044197371.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 07:23:17 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6905) Better logging messages when a
 delegation token is invalid
In-Reply-To: <12774985.207161281136756838.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899375#action_12899375 ] 

Hudson commented on HADOOP-6905:
--------------------------------

Integrated in Hadoop-Common-trunk #426 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/426/])
    HADOOP-6905. Better logging messages when a delegation token is invalid. Contributed by Kan Zhang.


> Better logging messages when a delegation token is invalid
> ----------------------------------------------------------
>
>                 Key: HADOOP-6905
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6905
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>             Fix For: 0.22.0
>
>         Attachments: c6905-01.patch
>
>
> From our production logs, we see some logging messages of "token is expired or doesn't exist". It would be helpful to know whose token it was.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10256-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 15:27:39 2010
Return-Path: <common-issues-return-10256-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28547 invoked from network); 17 Aug 2010 15:27:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 15:27:39 -0000
Received: (qmail 87247 invoked by uid 500); 17 Aug 2010 15:27:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87166 invoked by uid 500); 17 Aug 2010 15:27:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87158 invoked by uid 99); 17 Aug 2010 15:27:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 15:27:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 15:27:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HFRHTc002404
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 15:27:17 GMT
Message-ID: <21888371.395041282058837109.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 11:27:17 -0400 (EDT)
From: "Richard Lackey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6914) IPC Client Connection close does not
 protect against partially Connection objects
In-Reply-To: <9870966.313421281646336187.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Richard Lackey updated HADOOP-6914:
-----------------------------------

        Status: Resolved  (was: Patch Available)
    Resolution: Fixed

This should resolve a host of unexplained issues by providing problem source information

> IPC Client Connection close does not protect against partially Connection objects
> ---------------------------------------------------------------------------------
>
>                 Key: HADOOP-6914
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6914
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.20.0
>            Reporter: Richard Lackey
>         Attachments: hbase-2042.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> The purpose of this ticket is to cross-reference hbase-2042, opened 12/12/2009. close() is used to release Connection resources. It is invoked even though the Connection object may not have been completely initialized. For example, the file descriptor ulimit may have been exceeded in which case no socket would have been acquired. The result would be an NPE in close. The implied contract for close() seems to be that it will never fail. 
> There is a proposed patch in the hbase-2042 ticket.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10257-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 15:43:39 2010
Return-Path: <common-issues-return-10257-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35180 invoked from network); 17 Aug 2010 15:43:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 15:43:38 -0000
Received: (qmail 18994 invoked by uid 500); 17 Aug 2010 15:43:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18832 invoked by uid 500); 17 Aug 2010 15:43:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18784 invoked by uid 99); 17 Aug 2010 15:43:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 15:43:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 15:43:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HFhH8h002524
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 15:43:17 GMT
Message-ID: <4261119.395181282059797056.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 11:43:17 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6693) Add metrics to track kerberos login
 activity
In-Reply-To: <1138587062.22291270763196714.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6693:
------------------------------------

    Status: Open  (was: Patch Available)

> Add metrics to track kerberos login activity
> --------------------------------------------
>
>                 Key: HADOOP-6693
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6693
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6693.patch, HADOOP-6693.rel20.1.patch, HADOOP-6693.rel20.patch
>
>
> Need metrics to track kerberos login activity such as login rate and the time taken for login.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10258-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 15:43:57 2010
Return-Path: <common-issues-return-10258-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35224 invoked from network); 17 Aug 2010 15:43:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 15:43:57 -0000
Received: (qmail 19126 invoked by uid 500); 17 Aug 2010 15:43:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19082 invoked by uid 500); 17 Aug 2010 15:43:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19074 invoked by uid 99); 17 Aug 2010 15:43:56 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 15:43:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 15:43:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HFhHrF002536
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 15:43:17 GMT
Message-ID: <24257355.395221282059797924.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 11:43:17 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6693) Add metrics to track kerberos login
 activity
In-Reply-To: <1138587062.22291270763196714.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6693:
------------------------------------

    Status: Patch Available  (was: Open)

> Add metrics to track kerberos login activity
> --------------------------------------------
>
>                 Key: HADOOP-6693
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6693
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6693.patch, HADOOP-6693.rel20.1.patch, HADOOP-6693.rel20.patch
>
>
> Need metrics to track kerberos login activity such as login rate and the time taken for login.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10259-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 15:56:57 2010
Return-Path: <common-issues-return-10259-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38400 invoked from network); 17 Aug 2010 15:56:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 15:56:57 -0000
Received: (qmail 46988 invoked by uid 500); 17 Aug 2010 15:56:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46929 invoked by uid 500); 17 Aug 2010 15:56:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46921 invoked by uid 99); 17 Aug 2010 15:56:56 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 15:56:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 15:56:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HFuHnX002653
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 15:56:18 GMT
Message-ID: <20715289.395371282060577867.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 11:56:17 -0400 (EDT)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899441#action_12899441 ] 

Scott Carey commented on HADOOP-6884:
-------------------------------------

bq. Of course, the other tactic is to say "no layers, go to log4j" and work with the log4j team on having log4j push out data to chukwa and scribus, feed it back into Hadoop for analysis. Just a thought.

slf4j is log4j's successor, the 'founder' of slf4j is the same person that started log4j.  Like commons logging,  log4j is in maintenance mode.  http://logging.apache.org/log4j/1.2/changes-report.html

One maintenance release in 2.5 years.  Thats good, its stable and simple.  But new features or adapters? Unlikely.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10260-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 15:56:59 2010
Return-Path: <common-issues-return-10260-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38442 invoked from network); 17 Aug 2010 15:56:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 15:56:59 -0000
Received: (qmail 47145 invoked by uid 500); 17 Aug 2010 15:56:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47098 invoked by uid 500); 17 Aug 2010 15:56:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47090 invoked by uid 99); 17 Aug 2010 15:56:58 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 15:56:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 15:56:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HFuJHT002677
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 15:56:19 GMT
Message-ID: <12503775.395451282060579278.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 11:56:19 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899442#action_12899442 ] 

Doug Cutting commented on HADOOP-6884:
--------------------------------------

As Luke implies, slf4j already implements the API style I proposed above:

http://www.slf4j.org/faq.html#logging_performance

As for commons logging, there are a large number of articles suggesting that it should not be revived:

http://articles.qos.ch/thinkAgain.html
http://tapestryjava.blogspot.com/2007/08/so-long-commons-logging-hello-slf4j.html

So, to my thinking, a good approach might be:
 - switch to slf4j
 - replace string concatenation with slf4j format strings
 - add a checkstyle or other test that warns about string concatenation in log expressions
 
> you also need to choose at redist time what your back end will be and include that JAR

Unless we want to break things incompatibly, Hadoop's standard logging backend will remain log4j for the forseeable future.  So slf4j is only being proposed here as a front end API.  To my thinking, the question is whether we should invent our own improved front-end API for log4j that makes it easier to be efficient, or whether we should just use an existing efficient API like slf4j.  I lean towards the latter.



> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10261-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 16:32:55 2010
Return-Path: <common-issues-return-10261-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58219 invoked from network); 17 Aug 2010 16:32:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 16:32:54 -0000
Received: (qmail 583 invoked by uid 500); 17 Aug 2010 16:32:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 396 invoked by uid 500); 17 Aug 2010 16:32:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 292 invoked by uid 99); 17 Aug 2010 16:32:54 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 16:32:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 16:32:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HGWXTs003260
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 16:32:33 GMT
Message-ID: <23591115.396561282062753079.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 12:32:33 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899455#action_12899455 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6884:
------------------------------------------------

{code}
int a = ...;
long b = ...;
double c = ...;
LOG.debug("a=" + a + ", b=" + b + ", c=" + c);
{code}
If I understand the slf4j API correctly, slf4j does not really help for the codes above.  It has to create Integer, Long and  Double objects for the primitive data type and an array for vararg.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10262-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 16:41:42 2010
Return-Path: <common-issues-return-10262-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61118 invoked from network); 17 Aug 2010 16:41:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 16:41:40 -0000
Received: (qmail 17848 invoked by uid 500); 17 Aug 2010 16:41:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17794 invoked by uid 500); 17 Aug 2010 16:41:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17786 invoked by uid 99); 17 Aug 2010 16:41:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 16:41:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 16:41:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HGfHEP003373
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 16:41:18 GMT
Message-ID: <1784275.396831282063277877.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 12:41:17 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899461#action_12899461 ] 

Luke Lu commented on HADOOP-6884:
---------------------------------

Did some digging on the aspectj wrapping approach. Looks like it's a no go performance wise, as it cannot solve the arguments building cost issue without fixing aspectj itself, as someone else tried to do exactly the same thing here:

http://dev.eclipse.org/mhonarc/lists/aspectj-users/msg09294.html

Looks the most reasonable current course of action is just committing the patch as it looks correct, low risk and significantly lower gc stress (especially in namenode code, according to Suresh in offline discussions.) 

We should file a separate jira to explore the switching to slf4j api (still using log4j as backend) approach. One open issue even with slf4j API is that it doesn't solve the autoboxing cost issue for primitive types, which we use a lot the logs:

http://qos.ch/pipermail/slf4j-dev/2005-August/000241.html (old but still reflecting the current API design.)

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10263-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 16:43:58 2010
Return-Path: <common-issues-return-10263-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63037 invoked from network); 17 Aug 2010 16:43:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 16:43:57 -0000
Received: (qmail 24763 invoked by uid 500); 17 Aug 2010 16:43:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24685 invoked by uid 500); 17 Aug 2010 16:43:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24677 invoked by uid 99); 17 Aug 2010 16:43:56 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 16:43:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 16:43:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HGhHt8003413
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 16:43:18 GMT
Message-ID: <20946511.396911282063397868.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 12:43:17 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899462#action_12899462 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6884:
------------------------------------------------

For aspectj, first of all, we currently don't use aspectj in Hadoop except for testing.  I beg there are people against using aspectj or other byte-code rewriting technique on the non-testing codes since it is hard to debug.

Let's be academic (as what I see in this issue) and ignore the statement above.  I cannot see a simple solution on using aspectj.  Clearly, if we define a point cut per LOG.debug(..), aspectj will works but this is too much of works and, in some cases, we need to change the codes in order to define a point cut.  For the people suggesting aspectj and saying that it is simple, could you please provide a simple example?

I heard one suggestion from Luke yesterday.  He claimed we could use the "around" advice.  However, it seems to me that around does not prevent parameter evaluation.  I have not tested it yet.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10264-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 16:47:39 2010
Return-Path: <common-issues-return-10264-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66079 invoked from network); 17 Aug 2010 16:47:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 16:47:38 -0000
Received: (qmail 35888 invoked by uid 500); 17 Aug 2010 16:47:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35824 invoked by uid 500); 17 Aug 2010 16:47:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35816 invoked by uid 99); 17 Aug 2010 16:47:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 16:47:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 16:47:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HGlHoS003487
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 16:47:17 GMT
Message-ID: <8131792.397031282063637281.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 12:47:17 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899464#action_12899464 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6884:
------------------------------------------------

Thanks Luke for clarfying aspectj.

Let's be academic again: one more fantasy idea is to write our own java compiler or JVM.  It definitely will work.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10265-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 17:08:56 2010
Return-Path: <common-issues-return-10265-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77746 invoked from network); 17 Aug 2010 17:08:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 17:08:56 -0000
Received: (qmail 57378 invoked by uid 500); 17 Aug 2010 17:08:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57309 invoked by uid 500); 17 Aug 2010 17:08:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57301 invoked by uid 99); 17 Aug 2010 17:08:55 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 17:08:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 17:08:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HH8GgS003906
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 17:08:16 GMT
Message-ID: <22473455.397701282064896482.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 13:08:16 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6914) IPC Client Connection close does
 not protect against partially Connection objects
In-Reply-To: <9870966.313421281646336187.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899474#action_12899474 ] 

Todd Lipcon commented on HADOOP-6914:
-------------------------------------

Is this a dup of HADOOP-6724?

> IPC Client Connection close does not protect against partially Connection objects
> ---------------------------------------------------------------------------------
>
>                 Key: HADOOP-6914
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6914
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.20.0
>            Reporter: Richard Lackey
>         Attachments: hbase-2042.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> The purpose of this ticket is to cross-reference hbase-2042, opened 12/12/2009. close() is used to release Connection resources. It is invoked even though the Connection object may not have been completely initialized. For example, the file descriptor ulimit may have been exceeded in which case no socket would have been acquired. The result would be an NPE in close. The implied contract for close() seems to be that it will never fail. 
> There is a proposed patch in the hbase-2042 ticket.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10266-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 17:38:39 2010
Return-Path: <common-issues-return-10266-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89591 invoked from network); 17 Aug 2010 17:38:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 17:38:39 -0000
Received: (qmail 285 invoked by uid 500); 17 Aug 2010 17:38:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 207 invoked by uid 500); 17 Aug 2010 17:38:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 199 invoked by uid 99); 17 Aug 2010 17:38:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 17:38:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 17:38:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HHcHvA004322
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 17:38:17 GMT
Message-ID: <29911135.398401282066697275.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 13:38:17 -0400 (EDT)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899484#action_12899484 ] 

Scott Carey commented on HADOOP-6884:
-------------------------------------

bq. Thanks Luke for clarfying aspectj.

If aspectj can't do this, then its likely a no-go.  Spring or Guice cannot do it either (they both change methods by overriding them, not wrapping them).
That leaves ASM or low level bytecode changes which definitely will work but would not be easy at all and would require significant testing.  At this point, a feature like that should be part of the logging framework.

bq. If I understand the slf4j API correctly, slf4j does not really help for the codes above. It has to create Integer, Long and Double objects for the primitive data type and an array for vararg.

It helps a lot, even with the autoboxing and varargs.  A String has a hashCode and byte array member variable, a minimum memory overhead much larger than a boxed object.
The autoboxing and varargs is significantly less expensive than string concatenation.

{code}
int a = 0; long b = 1L; double c = 0.0d;
LOG.debug("a={}, b={}, c={}", a, b, c);
{code}

In slf4j this requires allocating a varargs array with 3 elements (~32 bytes), and one Integer, one Long, and one Double (16, 24, and 24 bytes).
With log4j, it does not require the vararg array, but creates a string out of each number -- which uses CPU and allocates more memory than just the boxed object, and then concatenates that into another String.  The end result is at minimum 15 characters (30 bytes) + String overhead (~48 bytes), and the intermediate result overhead is larger.  If any of the numbers are not so small, the overhead grows quickly.

The JVM is getting smarter with its Escape Analysis optimizations too -- which will either eliminate many autobox and vararg object allocations or push them on the stack.  I think Sun's JVM will already avoid the varargs Object[] construction, and the Double, but not the Long or Integer, if +UseEscapeAnalysis is on.  That flag becomes the default soon.  



> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10267-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 17:40:57 2010
Return-Path: <common-issues-return-10267-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89947 invoked from network); 17 Aug 2010 17:40:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 17:40:57 -0000
Received: (qmail 1777 invoked by uid 500); 17 Aug 2010 17:40:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1647 invoked by uid 500); 17 Aug 2010 17:40:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1638 invoked by uid 99); 17 Aug 2010 17:40:56 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 17:40:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 17:40:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HHeH5r004348
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 17:40:17 GMT
Message-ID: <10583685.398481282066817763.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 13:40:17 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899485#action_12899485 ] 

Doug Cutting commented on HADOOP-6884:
--------------------------------------

> LOG.debug("a=" + a + ", b=" + b + ", c=" + c);

The way you'd write this is:

LOG.debug("a={} b={} c={}", a, b, c);

This need not be a varargs call, but can a normal, 4-arg method call.

That said, numeric values would get boxed.  In the attached patch there are some calls that pass a single numeric value that would get boxed.  If we're concerned about the performance cost of the boxing in these cases, then we could define methods like:

public static void debug(String format, int i) {  if (log.isDebugEnabled())  log.debug(format, i); }
public static void debug(String format, long l) {  if (log.isDebugEnabled())  log.debug(format, l); }
public static void debug(String format, float f) {  if (log.isDebugEnabled())  log.debug(format, f); }
public static void debug(String format, double d) {  if (log.isDebugEnabled())  log.debug(format, d); }

If benchmarks show this to be a significant optimization, this might argue for creating a Hadoop-specific logging wrapper, as slf4j does not make this optimization.

Here's a random idea: use assertions for logging!  So folks could write:
  assert LOG.debug(....);
where LOG.debug() would need to return 'true'.


    

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10268-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 18:10:00 2010
Return-Path: <common-issues-return-10268-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7054 invoked from network); 17 Aug 2010 18:10:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 18:10:00 -0000
Received: (qmail 53749 invoked by uid 500); 17 Aug 2010 18:10:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53707 invoked by uid 500); 17 Aug 2010 18:09:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53699 invoked by uid 99); 17 Aug 2010 18:09:59 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 18:09:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 18:09:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HI9JMi004837
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 18:09:20 GMT
Message-ID: <2827943.399261282068559909.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 14:09:19 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899500#action_12899500 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6884:
------------------------------------------------

>The way you'd write this is:
>
>LOG.debug("a={} b={} c={}", a, b, c);

Dong, I have checked the [slf4j Logger API|http://www.slf4j.org/apidocs/org/slf4j/Logger.html] but does find a method like that.  There is one requiring creating an array, debug(String format, Object[] argArray).  From [the FAQ link|http://www.slf4j.org/faq.html#logging_performance] you provided earlier, it also suggests creating an array in this case.  Below is quoted from the FAQ.
{quote}
If three or more arguments need to be passed, you can make use of the Object[] variant. For example, you can write:

logger.debug("Value {} was inserted between {} and {}.", new Object[] {newVal, below, above});
{quote}


> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10269-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 18:10:01 2010
Return-Path: <common-issues-return-10269-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7112 invoked from network); 17 Aug 2010 18:10:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 18:10:01 -0000
Received: (qmail 53913 invoked by uid 500); 17 Aug 2010 18:10:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53879 invoked by uid 500); 17 Aug 2010 18:10:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53871 invoked by uid 99); 17 Aug 2010 18:10:00 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 18:10:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 18:09:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HI9LtO004867
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 18:09:21 GMT
Message-ID: <12315255.399361282068561616.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 14:09:21 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899501#action_12899501 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6884:
------------------------------------------------

Doug, sorry that I misspelled you name.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10270-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 18:16:02 2010
Return-Path: <common-issues-return-10270-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10100 invoked from network); 17 Aug 2010 18:16:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 18:16:01 -0000
Received: (qmail 60525 invoked by uid 500); 17 Aug 2010 18:16:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60483 invoked by uid 500); 17 Aug 2010 18:16:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60475 invoked by uid 99); 17 Aug 2010 18:16:00 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 18:16:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 18:15:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HIFGUD004984
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 18:15:17 GMT
Message-ID: <32374149.399571282068916454.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 14:15:16 -0400 (EDT)
From: "Richard Lackey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6914) IPC Client Connection close does
 not protect against partially Connection objects
In-Reply-To: <9870966.313421281646336187.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899505#action_12899505 ] 

Richard Lackey commented on HADOOP-6914:
----------------------------------------

It appears to be a duplicate. 
The proposed fix is different. The patch insures that close does not percolate a RuntimeException, and checks the socket and associated streams streams. This appears to be more inline with the implied contract for Client.close().

> IPC Client Connection close does not protect against partially Connection objects
> ---------------------------------------------------------------------------------
>
>                 Key: HADOOP-6914
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6914
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.20.0
>            Reporter: Richard Lackey
>         Attachments: hbase-2042.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> The purpose of this ticket is to cross-reference hbase-2042, opened 12/12/2009. close() is used to release Connection resources. It is invoked even though the Connection object may not have been completely initialized. For example, the file descriptor ulimit may have been exceeded in which case no socket would have been acquired. The result would be an NPE in close. The implied contract for close() seems to be that it will never fail. 
> There is a proposed patch in the hbase-2042 ticket.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10271-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 18:52:58 2010
Return-Path: <common-issues-return-10271-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31014 invoked from network); 17 Aug 2010 18:52:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 18:52:58 -0000
Received: (qmail 8470 invoked by uid 500); 17 Aug 2010 18:52:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8406 invoked by uid 500); 17 Aug 2010 18:52:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8394 invoked by uid 99); 17 Aug 2010 18:52:57 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 18:52:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 18:52:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HIqHlP005454
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 18:52:18 GMT
Message-ID: <20250664.400421282071137840.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 14:52:17 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899522#action_12899522 ] 

Doug Cutting commented on HADOOP-6884:
--------------------------------------

Nicholas>  I have checked the slf4j Logger API but does find a method like that.

That's right.  I said, such an API "need not be a varargs call, but can a normal, 4-arg method call".  With slf4j it would be varrargs.  Also note, however, that the current patch doesn't contain any log examples with three parameters.

Scott> The autoboxing and varargs is significantly less expensive than string concatenation.
Scott> Sun's JVM will already avoid the varargs Object[] construction, and the Double, but not the Long or Integer, if +UseEscapeAnalysis is on. That flag becomes the default soon.

Good points, Scott.

This issue lacks benchmarks.  Proposed optimizations should include benchmarks.  The following article has comments that indicate that, as Scott suggests, autoboxing and varargs are pretty fast.  With escape analysis they might be even faster.

http://www.nearinfinity.com/blogs/jeff_kunkle/performance_of_guarding_debug_log.html

If someone adds debug log lines in performance sensitive code, e.g., when calculating CRC32 or somesuch, then explicitly calling isDebugEnabled() would probably be a significant optimization, but such cases are rare, and performance sensitive code should be modified cautiously anyway, since other minor changes can have big effects.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10272-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 19:13:57 2010
Return-Path: <common-issues-return-10272-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50815 invoked from network); 17 Aug 2010 19:13:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 19:13:57 -0000
Received: (qmail 54767 invoked by uid 500); 17 Aug 2010 19:13:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54525 invoked by uid 500); 17 Aug 2010 19:13:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54517 invoked by uid 99); 17 Aug 2010 19:13:56 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 19:13:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 19:13:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HJDH4M005907
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 19:13:17 GMT
Message-ID: <1198076.401321282072397272.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 15:13:17 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6786) test-patch needs to verify Herriot
 integrity
In-Reply-To: <19336182.46451274829212745.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6786:
---------------------------------------

    Status: Open  (was: Patch Available)

No luck with Hudson. :(

> test-patch needs to verify Herriot integrity
> --------------------------------------------
>
>                 Key: HADOOP-6786
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6786
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-6786.patch, HADOOP-6786.patch, HADOOP-6786.patch
>
>
> Whenever a new patch is submitted for verification {{test-patch}} process has to make sure that none of Herriot bindings were broken.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10273-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 19:20:46 2010
Return-Path: <common-issues-return-10273-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54411 invoked from network); 17 Aug 2010 19:20:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 19:20:38 -0000
Received: (qmail 62786 invoked by uid 500); 17 Aug 2010 19:20:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62710 invoked by uid 500); 17 Aug 2010 19:20:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62702 invoked by uid 99); 17 Aug 2010 19:20:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 19:20:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 19:20:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HJKGed006014
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 19:20:16 GMT
Message-ID: <10771638.401541282072816727.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 15:20:16 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6786) test-patch needs to verify Herriot
 integrity
In-Reply-To: <19336182.46451274829212745.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6786:
---------------------------------------

    Status: Patch Available  (was: Open)

> test-patch needs to verify Herriot integrity
> --------------------------------------------
>
>                 Key: HADOOP-6786
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6786
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-6786.patch, HADOOP-6786.patch, HADOOP-6786.patch
>
>
> Whenever a new patch is submitted for verification {{test-patch}} process has to make sure that none of Herriot bindings were broken.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10274-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 19:59:58 2010
Return-Path: <common-issues-return-10274-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67314 invoked from network); 17 Aug 2010 19:59:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 19:59:58 -0000
Received: (qmail 19546 invoked by uid 500); 17 Aug 2010 19:59:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19488 invoked by uid 500); 17 Aug 2010 19:59:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19480 invoked by uid 99); 17 Aug 2010 19:59:57 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 19:59:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 19:59:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HJxIQi006631
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 19:59:19 GMT
Message-ID: <31110331.402891282075158932.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 15:59:18 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6907) Rpc client doesn't use the
 per-connection conf to figure out server's Kerberos principal
In-Reply-To: <9029429.243131281389838773.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899550#action_12899550 ] 

Jakob Homan commented on HADOOP-6907:
-------------------------------------

Patch review:
* Methods added for only unit testing should be marked as Private and Unstable
* Move Client::getConnectionId to getConnectionID itself.  With these changes ConnectionID may be large enough to warrant its own class rather than being nested in Client.
* The custom hash method in ConnectionID seems a bit odd.  Would the default provided by Eclipse be more workable? (as well as Eclipse-provided equals to guarantee hash/equals equivalency?)
* Provide messages for asserts in unit tests

> Rpc client doesn't use the per-connection conf to figure out server's Kerberos principal
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6907
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6907
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6907-12.patch
>
>
> Currently, RPC client caches the conf that was passed in to its constructor and uses that same conf (or values obtained from it) for every connection it sets up. This is not sufficient for security since each connection needs to figure out server's Kerberos principal on a per-connection basis. It's not reasonable to expect the first conf used by a user to contain all the Kerberos principals that her future connections will ever need. Or worse, if her first conf contains an incorrect principal name, it will prevent the user from connecting to the server even if she later on passes in a correct conf on retry (by calling RPC.getProxy()).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10275-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 20:01:39 2010
Return-Path: <common-issues-return-10275-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68543 invoked from network); 17 Aug 2010 20:01:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 20:01:38 -0000
Received: (qmail 22216 invoked by uid 500); 17 Aug 2010 20:01:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22174 invoked by uid 500); 17 Aug 2010 20:01:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22165 invoked by uid 99); 17 Aug 2010 20:01:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 20:01:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 20:01:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HK1HA0006698
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 20:01:17 GMT
Message-ID: <28924845.402941282075277381.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 16:01:17 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6907) Rpc client doesn't use the
 per-connection conf to figure out server's Kerberos principal
In-Reply-To: <9029429.243131281389838773.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6907:
--------------------------------

    Status: Open  (was: Patch Available)

Canceling patch post-review.

> Rpc client doesn't use the per-connection conf to figure out server's Kerberos principal
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6907
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6907
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6907-12.patch
>
>
> Currently, RPC client caches the conf that was passed in to its constructor and uses that same conf (or values obtained from it) for every connection it sets up. This is not sufficient for security since each connection needs to figure out server's Kerberos principal on a per-connection basis. It's not reasonable to expect the first conf used by a user to contain all the Kerberos principals that her future connections will ever need. Or worse, if her first conf contains an incorrect principal name, it will prevent the user from connecting to the server even if she later on passes in a correct conf on retry (by calling RPC.getProxy()).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10276-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 20:05:56 2010
Return-Path: <common-issues-return-10276-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74123 invoked from network); 17 Aug 2010 20:05:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 20:05:56 -0000
Received: (qmail 25192 invoked by uid 500); 17 Aug 2010 20:05:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25144 invoked by uid 500); 17 Aug 2010 20:05:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25136 invoked by uid 99); 17 Aug 2010 20:05:55 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 20:05:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 20:05:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HK5Gpa006722
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 20:05:16 GMT
Message-ID: <22496691.402981282075516566.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 16:05:16 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6693) Add metrics to track kerberos login
 activity
In-Reply-To: <1138587062.22291270763196714.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899552#action_12899552 ] 

Suresh Srinivas commented on HADOOP-6693:
-----------------------------------------

Since hudson has some issues, ran the unit tests and testpatch. All the tests passed.


> Add metrics to track kerberos login activity
> --------------------------------------------
>
>                 Key: HADOOP-6693
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6693
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6693.patch, HADOOP-6693.rel20.1.patch, HADOOP-6693.rel20.patch
>
>
> Need metrics to track kerberos login activity such as login rate and the time taken for login.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10277-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 20:08:39 2010
Return-Path: <common-issues-return-10277-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75726 invoked from network); 17 Aug 2010 20:08:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 20:08:39 -0000
Received: (qmail 29929 invoked by uid 500); 17 Aug 2010 20:08:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29691 invoked by uid 500); 17 Aug 2010 20:08:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29683 invoked by uid 99); 17 Aug 2010 20:08:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 20:08:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 20:08:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HK8HQI006755
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 20:08:17 GMT
Message-ID: <27558906.403061282075697710.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 16:08:17 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6693) Add metrics to track kerberos login
 activity
In-Reply-To: <1138587062.22291270763196714.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suresh Srinivas updated HADOOP-6693:
------------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
    Release Note: New metrics "login" of type MetricTimeVaryingRate is added under new metrics context name "ugi" and metrics record name "ugi".
      Resolution: Fixed

I committed the patch.

> Add metrics to track kerberos login activity
> --------------------------------------------
>
>                 Key: HADOOP-6693
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6693
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6693.patch, HADOOP-6693.rel20.1.patch, HADOOP-6693.rel20.patch
>
>
> Need metrics to track kerberos login activity such as login rate and the time taken for login.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10278-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 20:35:58 2010
Return-Path: <common-issues-return-10278-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87791 invoked from network); 17 Aug 2010 20:35:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 20:35:57 -0000
Received: (qmail 69743 invoked by uid 500); 17 Aug 2010 20:35:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69684 invoked by uid 500); 17 Aug 2010 20:35:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69676 invoked by uid 99); 17 Aug 2010 20:35:57 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 20:35:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 20:35:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HKZHlo007023
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 20:35:18 GMT
Message-ID: <16817999.403581282077317829.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 16:35:17 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6693) Add metrics to track kerberos login
 activity
In-Reply-To: <1138587062.22291270763196714.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899565#action_12899565 ] 

Hudson commented on HADOOP-6693:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #359 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/359/])
    HADOOP-6693. Add metrics to track kerberol login activity. Contributed by Suresh Srinivas.


> Add metrics to track kerberos login activity
> --------------------------------------------
>
>                 Key: HADOOP-6693
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6693
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6693.patch, HADOOP-6693.rel20.1.patch, HADOOP-6693.rel20.patch
>
>
> Need metrics to track kerberos login activity such as login rate and the time taken for login.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10279-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 21:44:42 2010
Return-Path: <common-issues-return-10279-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16292 invoked from network); 17 Aug 2010 21:44:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 21:44:42 -0000
Received: (qmail 57887 invoked by uid 500); 17 Aug 2010 21:44:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57554 invoked by uid 500); 17 Aug 2010 21:44:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57340 invoked by uid 99); 17 Aug 2010 21:44:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 21:44:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 21:44:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HLiK8k007872
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 21:44:20 GMT
Message-ID: <18508241.405391282081460831.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 17:44:20 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6885) Fix java doc warnings in Groups and
 RefreshUserMappingsProtocol
In-Reply-To: <14122960.38041280281637802.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6885:
--------------------------------

    Attachment: hadoop-6885-2.patch

Updated the patch to fix the IPC javadoc warnings. The earlier warnings are seen when using javadoc-dev.  Both targets build cleanly on trunk with this patch.  Thanks Jakob.


> Fix java doc warnings in Groups and RefreshUserMappingsProtocol
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6885
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6885
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.20.2
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6885-1.patch, hadoop-6885-2.patch
>
>
> There are a couple java docs warnings in Groups and RefreshUserMappingsProtocol.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10280-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 21:46:43 2010
Return-Path: <common-issues-return-10280-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16910 invoked from network); 17 Aug 2010 21:46:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 21:46:43 -0000
Received: (qmail 61358 invoked by uid 500); 17 Aug 2010 21:46:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61312 invoked by uid 500); 17 Aug 2010 21:46:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61304 invoked by uid 99); 17 Aug 2010 21:46:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 21:46:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 21:46:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HLkMJE007887
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 21:46:22 GMT
Message-ID: <19196156.405431282081582101.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 17:46:22 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6885) Fix java doc warnings in Groups and
 RefreshUserMappingsProtocol
In-Reply-To: <14122960.38041280281637802.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6885:
--------------------------------

    Status: Patch Available  (was: Open)

> Fix java doc warnings in Groups and RefreshUserMappingsProtocol
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6885
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6885
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6885-1.patch, hadoop-6885-2.patch
>
>
> There are a couple java docs warnings in Groups and RefreshUserMappingsProtocol.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10281-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 21:47:00 2010
Return-Path: <common-issues-return-10281-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17056 invoked from network); 17 Aug 2010 21:47:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 21:47:00 -0000
Received: (qmail 61725 invoked by uid 500); 17 Aug 2010 21:47:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61701 invoked by uid 500); 17 Aug 2010 21:47:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61693 invoked by uid 99); 17 Aug 2010 21:47:00 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 21:47:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 21:46:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HLkLfF007881
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 21:46:21 GMT
Message-ID: <19175658.405411282081581608.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 17:46:21 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6885) Fix java doc warnings in Groups and
 RefreshUserMappingsProtocol
In-Reply-To: <14122960.38041280281637802.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6885:
--------------------------------

    Affects Version/s: 0.22.0
                           (was: 0.20.2)

> Fix java doc warnings in Groups and RefreshUserMappingsProtocol
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6885
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6885
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6885-1.patch, hadoop-6885-2.patch
>
>
> There are a couple java docs warnings in Groups and RefreshUserMappingsProtocol.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10282-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 22:27:57 2010
Return-Path: <common-issues-return-10282-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38734 invoked from network); 17 Aug 2010 22:27:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 22:27:56 -0000
Received: (qmail 8218 invoked by uid 500); 17 Aug 2010 22:27:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8171 invoked by uid 500); 17 Aug 2010 22:27:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8163 invoked by uid 99); 17 Aug 2010 22:27:56 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 22:27:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 22:27:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HMRHtN008292
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 22:27:17 GMT
Message-ID: <33163986.406021282084037165.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 18:27:17 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6803) Add native gzip read/write coverage
 to TestCodec
In-Reply-To: <20681213.146461275520316684.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6803:
--------------------------------

    Attachment: hadoop-6803-3.patch

Updated patch resolves with trunk.

> Add native gzip read/write coverage to TestCodec 
> -------------------------------------------------
>
>                 Key: HADOOP-6803
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6803
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6803-1.patch, hadoop-6803-2.patch, hadoop-6803-3.patch
>
>
> Looking at ZlibCompressor I noticed that the finished member is never modified, and is therefore always false. This means ZlibCompressor#finished will always return false so CompressorStream#close loops indefinitely in finish:
> {code} 
>       while (!compressor.finished()) {
>         compress();
>       }
> {code}
> I modifed TestCodec#testGzipCodecWrite to also cover writing using the native lib and confirmed the hang with jstack. The fix is simple, ZlibCompressor should record when it's been finished.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10283-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 22:27:57 2010
Return-Path: <common-issues-return-10283-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38786 invoked from network); 17 Aug 2010 22:27:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 22:27:57 -0000
Received: (qmail 8385 invoked by uid 500); 17 Aug 2010 22:27:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8181 invoked by uid 500); 17 Aug 2010 22:27:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8172 invoked by uid 99); 17 Aug 2010 22:27:56 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 22:27:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 22:27:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HMRIM4008307
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 22:27:18 GMT
Message-ID: <32187056.406071282084038048.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 18:27:18 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6803) Add native gzip read/write coverage
 to TestCodec
In-Reply-To: <20681213.146461275520316684.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6803:
--------------------------------

    Status: Open  (was: Patch Available)

> Add native gzip read/write coverage to TestCodec 
> -------------------------------------------------
>
>                 Key: HADOOP-6803
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6803
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6803-1.patch, hadoop-6803-2.patch, hadoop-6803-3.patch
>
>
> Looking at ZlibCompressor I noticed that the finished member is never modified, and is therefore always false. This means ZlibCompressor#finished will always return false so CompressorStream#close loops indefinitely in finish:
> {code} 
>       while (!compressor.finished()) {
>         compress();
>       }
> {code}
> I modifed TestCodec#testGzipCodecWrite to also cover writing using the native lib and confirmed the hang with jstack. The fix is simple, ZlibCompressor should record when it's been finished.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10284-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 22:27:58 2010
Return-Path: <common-issues-return-10284-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38858 invoked from network); 17 Aug 2010 22:27:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 22:27:58 -0000
Received: (qmail 8625 invoked by uid 500); 17 Aug 2010 22:27:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8589 invoked by uid 500); 17 Aug 2010 22:27:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8581 invoked by uid 99); 17 Aug 2010 22:27:58 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 22:27:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 22:27:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HMRJtp008328
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 22:27:19 GMT
Message-ID: <12364875.406141282084039274.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 18:27:19 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6803) Add native gzip read/write coverage
 to TestCodec
In-Reply-To: <20681213.146461275520316684.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6803:
--------------------------------

    Status: Patch Available  (was: Open)

> Add native gzip read/write coverage to TestCodec 
> -------------------------------------------------
>
>                 Key: HADOOP-6803
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6803
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6803-1.patch, hadoop-6803-2.patch, hadoop-6803-3.patch
>
>
> Looking at ZlibCompressor I noticed that the finished member is never modified, and is therefore always false. This means ZlibCompressor#finished will always return false so CompressorStream#close loops indefinitely in finish:
> {code} 
>       while (!compressor.finished()) {
>         compress();
>       }
> {code}
> I modifed TestCodec#testGzipCodecWrite to also cover writing using the native lib and confirmed the hang with jstack. The fix is simple, ZlibCompressor should record when it's been finished.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10285-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 22:47:42 2010
Return-Path: <common-issues-return-10285-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45075 invoked from network); 17 Aug 2010 22:47:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 22:47:41 -0000
Received: (qmail 40076 invoked by uid 500); 17 Aug 2010 22:47:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40027 invoked by uid 500); 17 Aug 2010 22:47:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40018 invoked by uid 99); 17 Aug 2010 22:47:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 22:47:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 22:47:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HMlGsg008530
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 22:47:17 GMT
Message-ID: <11527428.406621282085236857.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 18:47:16 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6803) Add native gzip read/write coverage
 to TestCodec
In-Reply-To: <20681213.146461275520316684.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899630#action_12899630 ] 

Tom White commented on HADOOP-6803:
-----------------------------------

+1 Looks good.

> Add native gzip read/write coverage to TestCodec 
> -------------------------------------------------
>
>                 Key: HADOOP-6803
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6803
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6803-1.patch, hadoop-6803-2.patch, hadoop-6803-3.patch
>
>
> Looking at ZlibCompressor I noticed that the finished member is never modified, and is therefore always false. This means ZlibCompressor#finished will always return false so CompressorStream#close loops indefinitely in finish:
> {code} 
>       while (!compressor.finished()) {
>         compress();
>       }
> {code}
> I modifed TestCodec#testGzipCodecWrite to also cover writing using the native lib and confirmed the hang with jstack. The fix is simple, ZlibCompressor should record when it's been finished.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10286-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 22:51:43 2010
Return-Path: <common-issues-return-10286-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46195 invoked from network); 17 Aug 2010 22:51:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 22:51:42 -0000
Received: (qmail 50036 invoked by uid 500); 17 Aug 2010 22:51:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49989 invoked by uid 500); 17 Aug 2010 22:51:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49981 invoked by uid 99); 17 Aug 2010 22:51:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 22:51:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 22:51:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HMpIp1008560
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 22:51:18 GMT
Message-ID: <33520139.406701282085478498.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 18:51:18 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6784) Create Wiki document about Herriot
 Test Framework and test development guide
In-Reply-To: <15254185.17241274730143948.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik resolved HADOOP-6784.
----------------------------------------

    Resolution: Fixed

The document is published 
  http://wiki.apache.org/hadoop/HowToUseSystemTestFramework

> Create Wiki document about Herriot Test Framework and test development guide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-6784
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6784
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>            Reporter: Konstantin Boudnik
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10287-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 17 22:51:47 2010
Return-Path: <common-issues-return-10287-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46281 invoked from network); 17 Aug 2010 22:51:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Aug 2010 22:51:46 -0000
Received: (qmail 50308 invoked by uid 500); 17 Aug 2010 22:51:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50262 invoked by uid 500); 17 Aug 2010 22:51:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50254 invoked by uid 99); 17 Aug 2010 22:51:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 22:51:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Aug 2010 22:51:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7HMpP8v008587
	for <common-issues@hadoop.apache.org>; Tue, 17 Aug 2010 22:51:25 GMT
Message-ID: <16894455.406791282085485587.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 18:51:25 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6784) Create Wiki document about Herriot
 Test Framework and test development guide
In-Reply-To: <15254185.17241274730143948.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik reassigned HADOOP-6784:
------------------------------------------

    Assignee: Konstantin Boudnik

> Create Wiki document about Herriot Test Framework and test development guide
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-6784
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6784
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: documentation
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10288-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 01:17:59 2010
Return-Path: <common-issues-return-10288-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20857 invoked from network); 18 Aug 2010 01:17:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 01:17:59 -0000
Received: (qmail 88765 invoked by uid 500); 18 Aug 2010 01:17:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88701 invoked by uid 500); 18 Aug 2010 01:17:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88693 invoked by uid 99); 18 Aug 2010 01:17:58 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 01:17:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 01:17:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7I1HJpM010387
	for <common-issues@hadoop.apache.org>; Wed, 18 Aug 2010 01:17:19 GMT
Message-ID: <2112761.410791282094239754.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 21:17:19 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6907) Rpc client doesn't use the
 per-connection conf to figure out server's Kerberos principal
In-Reply-To: <9029429.243131281389838773.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6907:
------------------------------

    Attachment: c6907-15.patch

new patch addressing Jakob's comments. My equals() and hashcode() implementations are the same as Eclipse-provided defaults. I see Eclipse's hashcode() method is easier to read. So I coupled it with my own equals().

> Rpc client doesn't use the per-connection conf to figure out server's Kerberos principal
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6907
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6907
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6907-12.patch, c6907-15.patch
>
>
> Currently, RPC client caches the conf that was passed in to its constructor and uses that same conf (or values obtained from it) for every connection it sets up. This is not sufficient for security since each connection needs to figure out server's Kerberos principal on a per-connection basis. It's not reasonable to expect the first conf used by a user to contain all the Kerberos principals that her future connections will ever need. Or worse, if her first conf contains an incorrect principal name, it will prevent the user from connecting to the server even if she later on passes in a correct conf on retry (by calling RPC.getProxy()).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10289-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 01:20:58 2010
Return-Path: <common-issues-return-10289-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21848 invoked from network); 18 Aug 2010 01:20:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 01:20:57 -0000
Received: (qmail 90558 invoked by uid 500); 18 Aug 2010 01:20:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90422 invoked by uid 500); 18 Aug 2010 01:20:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90414 invoked by uid 99); 18 Aug 2010 01:20:57 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 01:20:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 01:20:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7I1KI7w010426
	for <common-issues@hadoop.apache.org>; Wed, 18 Aug 2010 01:20:18 GMT
Message-ID: <1686274.410911282094418641.JavaMail.jira@thor>
Date: Tue, 17 Aug 2010 21:20:18 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6907) Rpc client doesn't use the
 per-connection conf to figure out server's Kerberos principal
In-Reply-To: <9029429.243131281389838773.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6907:
------------------------------

    Status: Patch Available  (was: Open)

> Rpc client doesn't use the per-connection conf to figure out server's Kerberos principal
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6907
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6907
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6907-12.patch, c6907-15.patch
>
>
> Currently, RPC client caches the conf that was passed in to its constructor and uses that same conf (or values obtained from it) for every connection it sets up. This is not sufficient for security since each connection needs to figure out server's Kerberos principal on a per-connection basis. It's not reasonable to expect the first conf used by a user to contain all the Kerberos principals that her future connections will ever need. Or worse, if her first conf contains an incorrect principal name, it will prevent the user from connecting to the server even if she later on passes in a correct conf on retry (by calling RPC.getProxy()).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10290-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 06:27:50 2010
Return-Path: <common-issues-return-10290-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26607 invoked from network); 18 Aug 2010 06:27:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 06:27:50 -0000
Received: (qmail 59443 invoked by uid 500); 18 Aug 2010 06:27:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59299 invoked by uid 500); 18 Aug 2010 06:27:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59289 invoked by uid 99); 18 Aug 2010 06:27:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 06:27:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 06:27:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7I6ROR2013303
	for <common-issues@hadoop.apache.org>; Wed, 18 Aug 2010 06:27:24 GMT
Message-ID: <8793731.415221282112844674.JavaMail.jira@thor>
Date: Wed, 18 Aug 2010 02:27:24 -0400 (EDT)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6894) Common foundation for Hadoop client
 tools
In-Reply-To: <10134473.113661280733856346.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-6894:
---------------------------------------

    Attachment: HadoopClientToolsV2.pdf

Attached is a proposal (V2) Owen and I worked out together.

This proposal enables a complementary integration with HADOOP-6255

What has changed:

* hadoop client commands are just another tool
* 'tools/' dir is now 'share/'
* there can be a special tool 'commons' where 'plugin' tools like har, viewfs, hdfsprx are added as dependencies
* all tools registered in 'commons' are always avail to any tool that depends on 'commons'
* there is an 'etc/' directory where tools configurations live

> Common foundation for Hadoop client tools
> -----------------------------------------
>
>                 Key: HADOOP-6894
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6894
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Alejandro Abdelnur
>         Attachments: HadoopClientTools.pdf, HadoopClientToolsV2.pdf
>
>
> As Hadoop widespreads and matures the number of tools and utilities for users keeps growing.
> Some of them are bundled with Hadoop core, some with Hadoop contrib, some on their own, some are full fledged servers on their own. For example, just to name a few: distcp, streaming, pipes, har, pig, hive, oozie.
> Today there is no standard mechanism for making these tools available to users. Neither there is a standard mechanism for these tools to integrate and distributed them with each other.
> The lack of a common foundation creates issues for developers and users.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10291-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 12:10:43 2010
Return-Path: <common-issues-return-10291-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64707 invoked from network); 18 Aug 2010 12:10:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 12:10:42 -0000
Received: (qmail 2378 invoked by uid 500); 18 Aug 2010 12:10:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2230 invoked by uid 500); 18 Aug 2010 12:10:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2222 invoked by uid 99); 18 Aug 2010 12:10:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 12:10:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 12:10:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7ICAHZU016348
	for <common-issues@hadoop.apache.org>; Wed, 18 Aug 2010 12:10:17 GMT
Message-ID: <5829404.418901282133417170.JavaMail.jira@thor>
Date: Wed, 18 Aug 2010 08:10:17 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6693) Add metrics to track kerberos login
 activity
In-Reply-To: <1138587062.22291270763196714.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12899818#action_12899818 ] 

Hudson commented on HADOOP-6693:
--------------------------------

Integrated in Hadoop-Common-trunk #427 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/427/])
    HADOOP-6693. Add metrics to track kerberol login activity. Contributed by Suresh Srinivas.


> Add metrics to track kerberos login activity
> --------------------------------------------
>
>                 Key: HADOOP-6693
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6693
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6693.patch, HADOOP-6693.rel20.1.patch, HADOOP-6693.rel20.patch
>
>
> Need metrics to track kerberos login activity such as login rate and the time taken for login.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10292-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 19:45:58 2010
Return-Path: <common-issues-return-10292-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22223 invoked from network); 18 Aug 2010 19:45:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 19:45:58 -0000
Received: (qmail 25735 invoked by uid 500); 18 Aug 2010 19:45:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25662 invoked by uid 500); 18 Aug 2010 19:45:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25654 invoked by uid 99); 18 Aug 2010 19:45:57 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 19:45:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 19:45:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7IJjIUA021455
	for <common-issues@hadoop.apache.org>; Wed, 18 Aug 2010 19:45:18 GMT
Message-ID: <8429102.427321282160718294.JavaMail.jira@thor>
Date: Wed, 18 Aug 2010 15:45:18 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6860)  'compile-fault-inject' should never
 be called directly.
In-Reply-To: <25103470.380431279132431950.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6860:
---------------------------------------

    Attachment: hadoop-6860.y20.patch

Re-fit for y20. Not to commit here.

>  'compile-fault-inject' should never be called directly.
> --------------------------------------------------------
>
>                 Key: HADOOP-6860
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6860
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6860.patch, hadoop-6860.y20.patch
>
>
> Similar to HDFS-1299 a  helper target  'compile-fault-inject' should never be called directly.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10293-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 21:11:00 2010
Return-Path: <common-issues-return-10293-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65508 invoked from network); 18 Aug 2010 21:11:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 21:11:00 -0000
Received: (qmail 13730 invoked by uid 500); 18 Aug 2010 21:11:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13639 invoked by uid 500); 18 Aug 2010 21:10:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13626 invoked by uid 99); 18 Aug 2010 21:10:59 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 21:10:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 21:10:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7ILAH6g022803
	for <common-issues@hadoop.apache.org>; Wed, 18 Aug 2010 21:10:17 GMT
Message-ID: <32460563.430341282165817396.JavaMail.jira@thor>
Date: Wed, 18 Aug 2010 17:10:17 -0400 (EDT)
From: "Mahadev konar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6906) FileContext copy() utility doesn't
 work with recursive copying of directories.
In-Reply-To: <31039707.230631281349276933.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900034#action_12900034 ] 

Mahadev konar commented on HADOOP-6906:
---------------------------------------

+1 for the fix. as per the listStatus(Path, PathFilter) api, it would be good to have it, but again this fix should neways be there.

 

> FileContext copy() utility doesn't work with recursive copying of directories.
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6906
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6906
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Vinod K V
>            Assignee: Vinod K V
>         Attachments: HADOOP-6906-20100809.1.txt, HADOOP-6906-regression.1.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10295-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 21:11:01 2010
Return-Path: <common-issues-return-10295-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65587 invoked from network); 18 Aug 2010 21:11:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 21:11:00 -0000
Received: (qmail 13895 invoked by uid 500); 18 Aug 2010 21:11:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13817 invoked by uid 500); 18 Aug 2010 21:11:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13774 invoked by uid 99); 18 Aug 2010 21:11:00 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 21:10:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 21:10:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7ILAIO6022815
	for <common-issues@hadoop.apache.org>; Wed, 18 Aug 2010 21:10:18 GMT
Message-ID: <10658396.430381282165817997.JavaMail.jira@thor>
Date: Wed, 18 Aug 2010 17:10:17 -0400 (EDT)
From: "Mahadev konar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6906) FileContext copy() utility doesn't
 work with recursive copying of directories.
In-Reply-To: <31039707.230631281349276933.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mahadev konar updated HADOOP-6906:
----------------------------------

    Status: Open  (was: Patch Available)

> FileContext copy() utility doesn't work with recursive copying of directories.
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6906
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6906
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Vinod K V
>            Assignee: Vinod K V
>         Attachments: HADOOP-6906-20100809.1.txt, HADOOP-6906-regression.1.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10294-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 21:11:01 2010
Return-Path: <common-issues-return-10294-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65589 invoked from network); 18 Aug 2010 21:11:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 21:11:00 -0000
Received: (qmail 13875 invoked by uid 500); 18 Aug 2010 21:11:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13815 invoked by uid 500); 18 Aug 2010 21:11:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13777 invoked by uid 99); 18 Aug 2010 21:11:00 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 21:10:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 21:10:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7ILAI5C022827
	for <common-issues@hadoop.apache.org>; Wed, 18 Aug 2010 21:10:18 GMT
Message-ID: <1991277.430421282165818591.JavaMail.jira@thor>
Date: Wed, 18 Aug 2010 17:10:18 -0400 (EDT)
From: "Mahadev konar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6906) FileContext copy() utility doesn't
 work with recursive copying of directories.
In-Reply-To: <31039707.230631281349276933.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mahadev konar updated HADOOP-6906:
----------------------------------

    Status: Patch Available  (was: Open)

> FileContext copy() utility doesn't work with recursive copying of directories.
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6906
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6906
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Vinod K V
>            Assignee: Vinod K V
>         Attachments: HADOOP-6906-20100809.1.txt, HADOOP-6906-regression.1.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10296-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 21:11:03 2010
Return-Path: <common-issues-return-10296-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65682 invoked from network); 18 Aug 2010 21:11:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 21:11:03 -0000
Received: (qmail 14193 invoked by uid 500); 18 Aug 2010 21:11:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14155 invoked by uid 500); 18 Aug 2010 21:11:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14147 invoked by uid 99); 18 Aug 2010 21:11:03 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 21:11:03 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 21:10:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7ILAJqb022839
	for <common-issues@hadoop.apache.org>; Wed, 18 Aug 2010 21:10:19 GMT
Message-ID: <14151870.430461282165819447.JavaMail.jira@thor>
Date: Wed, 18 Aug 2010 17:10:19 -0400 (EDT)
From: "Mahadev konar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6906) FileContext copy() utility doesn't
 work with recursive copying of directories.
In-Reply-To: <31039707.230631281349276933.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mahadev konar updated HADOOP-6906:
----------------------------------

    Fix Version/s: 0.22.0

> FileContext copy() utility doesn't work with recursive copying of directories.
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6906
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6906
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Vinod K V
>            Assignee: Vinod K V
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6906-20100809.1.txt, HADOOP-6906-regression.1.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10297-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 21:53:39 2010
Return-Path: <common-issues-return-10297-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84295 invoked from network); 18 Aug 2010 21:53:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 21:53:38 -0000
Received: (qmail 75107 invoked by uid 500); 18 Aug 2010 21:53:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75044 invoked by uid 500); 18 Aug 2010 21:53:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75035 invoked by uid 99); 18 Aug 2010 21:53:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 21:53:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 21:53:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7ILrHOQ023308
	for <common-issues@hadoop.apache.org>; Wed, 18 Aug 2010 21:53:17 GMT
Message-ID: <7560402.431361282168397043.JavaMail.jira@thor>
Date: Wed, 18 Aug 2010 17:53:17 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6885) Fix java doc warnings in Groups and
 RefreshUserMappingsProtocol
In-Reply-To: <14122960.38041280281637802.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6885:
--------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

+1.  Verified javadoc is fixed.  Hudson is completely AWOL and this is a minor documentation patch, so I've committed it.  Resolving as fixed.  Thanks, Eli.

> Fix java doc warnings in Groups and RefreshUserMappingsProtocol
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6885
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6885
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6885-1.patch, hadoop-6885-2.patch
>
>
> There are a couple java docs warnings in Groups and RefreshUserMappingsProtocol.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10298-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 21:53:40 2010
Return-Path: <common-issues-return-10298-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84344 invoked from network); 18 Aug 2010 21:53:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 21:53:39 -0000
Received: (qmail 75255 invoked by uid 500); 18 Aug 2010 21:53:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75218 invoked by uid 500); 18 Aug 2010 21:53:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75210 invoked by uid 99); 18 Aug 2010 21:53:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 21:53:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 21:53:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7ILrIoh023326
	for <common-issues@hadoop.apache.org>; Wed, 18 Aug 2010 21:53:18 GMT
Message-ID: <25029002.431421282168398737.JavaMail.jira@thor>
Date: Wed, 18 Aug 2010 17:53:18 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6697) script to that would let us checkout
 code from different repos
In-Reply-To: <314161499.9971270831131213.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6697:
--------------------------------

    Status: Open  (was: Patch Available)

Canceling patch available status, since there's no patch.

> script to that would let us checkout code from different repos
> --------------------------------------------------------------
>
>                 Key: HADOOP-6697
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6697
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: build
>            Reporter: Giridharan Kesavan
>
> To write a shell script that would let us checkout code from two different repository , where we cant use svn:externals.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10299-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 21:59:39 2010
Return-Path: <common-issues-return-10299-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86256 invoked from network); 18 Aug 2010 21:59:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 21:59:39 -0000
Received: (qmail 82785 invoked by uid 500); 18 Aug 2010 21:59:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82620 invoked by uid 500); 18 Aug 2010 21:59:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82469 invoked by uid 99); 18 Aug 2010 21:59:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 21:59:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 21:59:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7ILxH3X023490
	for <common-issues@hadoop.apache.org>; Wed, 18 Aug 2010 21:59:17 GMT
Message-ID: <18435886.431881282168757880.JavaMail.jira@thor>
Date: Wed, 18 Aug 2010 17:59:17 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6803) Add native gzip read/write coverage
 to TestCodec
In-Reply-To: <20681213.146461275520316684.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6803:
------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

I've just committed this. Thanks Eli!

> Add native gzip read/write coverage to TestCodec 
> -------------------------------------------------
>
>                 Key: HADOOP-6803
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6803
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6803-1.patch, hadoop-6803-2.patch, hadoop-6803-3.patch
>
>
> Looking at ZlibCompressor I noticed that the finished member is never modified, and is therefore always false. This means ZlibCompressor#finished will always return false so CompressorStream#close loops indefinitely in finish:
> {code} 
>       while (!compressor.finished()) {
>         compress();
>       }
> {code}
> I modifed TestCodec#testGzipCodecWrite to also cover writing using the native lib and confirmed the hang with jstack. The fix is simple, ZlibCompressor should record when it's been finished.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10300-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 22:01:46 2010
Return-Path: <common-issues-return-10300-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87251 invoked from network); 18 Aug 2010 22:01:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 22:01:46 -0000
Received: (qmail 84016 invoked by uid 500); 18 Aug 2010 22:01:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83980 invoked by uid 500); 18 Aug 2010 22:01:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83968 invoked by uid 99); 18 Aug 2010 22:01:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 22:01:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 22:01:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7IM1Ohc023577
	for <common-issues@hadoop.apache.org>; Wed, 18 Aug 2010 22:01:24 GMT
Message-ID: <28877164.432001282168884530.JavaMail.jira@thor>
Date: Wed, 18 Aug 2010 18:01:24 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6523) add an expiration date field to Token
 class.
In-Reply-To: <2119481605.137151264795714568.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6523:
-----------------------------------

        Status: Resolved  (was: Patch Available)
    Resolution: Not A Problem

this is not needed anymore.

> add an expiration date field to Token class. 
> ---------------------------------------------
>
>                 Key: HADOOP-6523
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6523
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6523.patch
>
>
> Needed by MAPREDUCE-1430

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10301-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 22:14:57 2010
Return-Path: <common-issues-return-10301-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98207 invoked from network); 18 Aug 2010 22:14:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 22:14:57 -0000
Received: (qmail 2061 invoked by uid 500); 18 Aug 2010 22:14:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1988 invoked by uid 500); 18 Aug 2010 22:14:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1980 invoked by uid 99); 18 Aug 2010 22:14:56 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 22:14:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 22:14:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7IMEH63023653
	for <common-issues@hadoop.apache.org>; Wed, 18 Aug 2010 22:14:17 GMT
Message-ID: <17236725.432131282169657243.JavaMail.jira@thor>
Date: Wed, 18 Aug 2010 18:14:17 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6453) Hadoop wrapper script shouldn't
 ignore an existing JAVA_LIBRARY_PATH
In-Reply-To: <909267607.1261086858225.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6453:
--------------------------------

           Status: Open  (was: Patch Available)
    Fix Version/s: 0.22.0

The trunk patch has unfortunately gone stale.  However, this seems like a good change to make with essentially no risk.  This will treat JAVA_LIBRARY_PATH consistently with JAVA_HOME, and can be marked as incompatible change.  cygwin should be fine.  Chad, can you please spin up a new patch and, barring any objections, I'll review and commit it.

> Hadoop wrapper script shouldn't ignore an existing JAVA_LIBRARY_PATH
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6453
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6453
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2, 0.21.0, 0.22.0
>            Reporter: Chad Metcalf
>            Assignee: Chad Metcalf
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6453-0.20.patch, HADOOP-6453-0.20v2.patch, HADOOP-6453-trunkv2.patch, HADOOP-6453.trunk.patch
>
>
> Currently the hadoop wrapper script assumes its the only place that uses JAVA_LIBRARY_PATH and initializes it to a blank line.
> JAVA_LIBRARY_PATH=''
> This prevents anyone from setting this outside of the hadoop wrapper (say hadoop-config.sh) for their own native libraries.
> The fix is pretty simple. Don't initialize it to '' and append the native libs like normal. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10302-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 22:22:38 2010
Return-Path: <common-issues-return-10302-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2344 invoked from network); 18 Aug 2010 22:22:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 22:22:38 -0000
Received: (qmail 13785 invoked by uid 500); 18 Aug 2010 22:22:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13746 invoked by uid 500); 18 Aug 2010 22:22:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13738 invoked by uid 99); 18 Aug 2010 22:22:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 22:22:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 22:22:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7IMMG98023705
	for <common-issues@hadoop.apache.org>; Wed, 18 Aug 2010 22:22:16 GMT
Message-ID: <467280.432251282170136471.JavaMail.jira@thor>
Date: Wed, 18 Aug 2010 18:22:16 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6625) Hadoop 0.20 doesn't generate
 hadoop-test pom, existing pom has bad dependencies, doesnt build
 javadoc,sources jar
In-Reply-To: <670483963.168041268185827623.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6625:
--------------------------------

        Status: Resolved  (was: Patch Available)
    Resolution: Duplicate

> Hadoop 0.20 doesn't generate hadoop-test pom, existing pom has bad dependencies, doesnt build javadoc,sources jar
> -----------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6625
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6625
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: ryan rawson
>             Fix For: 0.20.3
>
>         Attachments: 0001-fix-build-pom-and-maven-exporting.patch
>
>
> The build for hadoop 0.20 attempts to build a POM and JAR for maven projects. However it includes a few broken dependencies, no hadoop-test.pom, and doesn't build javadoc or sources jars.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10303-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 22:24:40 2010
Return-Path: <common-issues-return-10303-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3572 invoked from network); 18 Aug 2010 22:24:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 22:24:40 -0000
Received: (qmail 15133 invoked by uid 500); 18 Aug 2010 22:24:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15097 invoked by uid 500); 18 Aug 2010 22:24:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15089 invoked by uid 99); 18 Aug 2010 22:24:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 22:24:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 22:24:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7IMOJbo023742
	for <common-issues@hadoop.apache.org>; Wed, 18 Aug 2010 22:24:19 GMT
Message-ID: <31826484.432371282170259362.JavaMail.jira@thor>
Date: Wed, 18 Aug 2010 18:24:19 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6885) Fix java doc warnings in Groups and
 RefreshUserMappingsProtocol
In-Reply-To: <14122960.38041280281637802.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900069#action_12900069 ] 

Hudson commented on HADOOP-6885:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #360 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/360/])
    HADOOP-6885. Fix java doc warnings in Groups and RefreshUserMappingsProtocol. Contributed by Eli Collins.


> Fix java doc warnings in Groups and RefreshUserMappingsProtocol
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6885
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6885
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6885-1.patch, hadoop-6885-2.patch
>
>
> There are a couple java docs warnings in Groups and RefreshUserMappingsProtocol.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10304-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 22:24:58 2010
Return-Path: <common-issues-return-10304-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3792 invoked from network); 18 Aug 2010 22:24:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 22:24:58 -0000
Received: (qmail 15810 invoked by uid 500); 18 Aug 2010 22:24:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15777 invoked by uid 500); 18 Aug 2010 22:24:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15769 invoked by uid 99); 18 Aug 2010 22:24:57 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 22:24:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 22:24:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7IMOJhi023736
	for <common-issues@hadoop.apache.org>; Wed, 18 Aug 2010 22:24:19 GMT
Message-ID: <5317477.432351282170259112.JavaMail.jira@thor>
Date: Wed, 18 Aug 2010 18:24:19 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6803) Add native gzip read/write coverage
 to TestCodec
In-Reply-To: <20681213.146461275520316684.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900068#action_12900068 ] 

Hudson commented on HADOOP-6803:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #360 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/360/])
    HADOOP-6803. Add native gzip read/write coverage to TestCodec. Contributed by Eli Collins.


> Add native gzip read/write coverage to TestCodec 
> -------------------------------------------------
>
>                 Key: HADOOP-6803
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6803
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6803-1.patch, hadoop-6803-2.patch, hadoop-6803-3.patch
>
>
> Looking at ZlibCompressor I noticed that the finished member is never modified, and is therefore always false. This means ZlibCompressor#finished will always return false so CompressorStream#close loops indefinitely in finish:
> {code} 
>       while (!compressor.finished()) {
>         compress();
>       }
> {code}
> I modifed TestCodec#testGzipCodecWrite to also cover writing using the native lib and confirmed the hang with jstack. The fix is simple, ZlibCompressor should record when it's been finished.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10305-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 22:49:39 2010
Return-Path: <common-issues-return-10305-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13350 invoked from network); 18 Aug 2010 22:49:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 22:49:39 -0000
Received: (qmail 42259 invoked by uid 500); 18 Aug 2010 22:49:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42222 invoked by uid 500); 18 Aug 2010 22:49:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42214 invoked by uid 99); 18 Aug 2010 22:49:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 22:49:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 22:49:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7IMnI80024029
	for <common-issues@hadoop.apache.org>; Wed, 18 Aug 2010 22:49:18 GMT
Message-ID: <10925638.432981282171758629.JavaMail.jira@thor>
Date: Wed, 18 Aug 2010 18:49:18 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6482) GenericOptionsParser constructor that
 takes Options and String[] ignores options
In-Reply-To: <947197260.111021262909095193.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6482:
--------------------------------

     Hadoop Flags: [Reviewed]
    Fix Version/s: 0.22.0

+1.  Modified method is private, so not an incompatible change.

> GenericOptionsParser constructor that takes Options and String[] ignores options
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6482
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6482
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.22.0
>            Reporter: Chris Wilkes
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6482-1.patch, hadoop-6482-2.patch
>
>   Original Estimate: 0.08h
>  Remaining Estimate: 0.08h
>
> This constructor:
>   public GenericOptionsParser(Options opts, String[] args) {
>     this(new Configuration(), new Options(), args);
>   }
> should do a this(new Configuration(), opts, args)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10306-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 22:53:39 2010
Return-Path: <common-issues-return-10306-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13878 invoked from network); 18 Aug 2010 22:53:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 22:53:39 -0000
Received: (qmail 46212 invoked by uid 500); 18 Aug 2010 22:53:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46169 invoked by uid 500); 18 Aug 2010 22:53:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46140 invoked by uid 99); 18 Aug 2010 22:53:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 22:53:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 22:53:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7IMrIIf024075
	for <common-issues@hadoop.apache.org>; Wed, 18 Aug 2010 22:53:18 GMT
Message-ID: <7742174.433091282171998363.JavaMail.jira@thor>
Date: Wed, 18 Aug 2010 18:53:18 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6884:
-------------------------------------------

    Attachment: FunAgain.java

FunAgain.java: Benchmarks comparing the following calls
- LOG.isDebugEnabled()
- static debug(..) as suggested by Doug
- LOG.debug(..), i.e. do nothing.

Result 1:
{noformat}
java.version = 1.6.0_10
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_10-b33
java.vm.version = 11.0-b15
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = amd64
os.name = Linux
os.version = 2.6.9-55.ELsmp

n=10000000
LOG.isDebugEnabled(): 82 ms
static debug(..)    : 502 ms
LOG.debug(..)       : 11644 ms
{noformat}

Result 2:
{noformat}
java.version = 1.6.0_16
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_16-b01
java.vm.version = 14.2-b01
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) Client VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = x86
os.name = Windows XP
os.version = 5.1

n=10000000
LOG.isDebugEnabled(): 172 ms
static debug(..)    : 547 ms
LOG.debug(..)       : 40421 ms
{noformat}

+1 on committing the existing patch.  We may work on the static debug(..) in a separated JIRA.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10307-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 22:53:40 2010
Return-Path: <common-issues-return-10307-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13910 invoked from network); 18 Aug 2010 22:53:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 22:53:39 -0000
Received: (qmail 46375 invoked by uid 500); 18 Aug 2010 22:53:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46338 invoked by uid 500); 18 Aug 2010 22:53:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46330 invoked by uid 99); 18 Aug 2010 22:53:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 22:53:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 22:53:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7IMrJlL024087
	for <common-issues@hadoop.apache.org>; Wed, 18 Aug 2010 22:53:19 GMT
Message-ID: <17448262.433131282171998991.JavaMail.jira@thor>
Date: Wed, 18 Aug 2010 18:53:18 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6482) GenericOptionsParser constructor that
 takes Options and String[] ignores options
In-Reply-To: <947197260.111021262909095193.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6482:
--------------------------------

        Status: Resolved  (was: Patch Available)
    Resolution: Fixed

I've committed this.  Resolving as fixed.  Thanks, Chris and Eli.

> GenericOptionsParser constructor that takes Options and String[] ignores options
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6482
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6482
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.22.0
>            Reporter: Chris Wilkes
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6482-1.patch, hadoop-6482-2.patch
>
>   Original Estimate: 0.08h
>  Remaining Estimate: 0.08h
>
> This constructor:
>   public GenericOptionsParser(Options opts, String[] args) {
>     this(new Configuration(), new Options(), args);
>   }
> should do a this(new Configuration(), opts, args)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10308-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 23:01:38 2010
Return-Path: <common-issues-return-10308-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15517 invoked from network); 18 Aug 2010 23:01:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 23:01:38 -0000
Received: (qmail 51294 invoked by uid 500); 18 Aug 2010 23:01:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51201 invoked by uid 500); 18 Aug 2010 23:01:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51193 invoked by uid 99); 18 Aug 2010 23:01:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 23:01:38 +0000
X-ASF-Spam-Status: No, hits=-1996.4 required=10.0
	tests=ALL_TRUSTED,FS_REPLICA
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 23:01:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7IN1HdK024251
	for <common-issues@hadoop.apache.org>; Wed, 18 Aug 2010 23:01:17 GMT
Message-ID: <31062676.433471282172477666.JavaMail.jira@thor>
Date: Wed, 18 Aug 2010 19:01:17 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6857) FsShell should report raw disk usage
 including replication factor
In-Reply-To: <1268643.319901278914569769.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6857:
--------------------------------

    Status: Open  (was: Patch Available)

Canceling patch until Nicholas' comments are addressed.

> FsShell should report raw disk usage including replication factor
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6857
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6857
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Alex Kozlov
>             Fix For: 0.22.0
>
>         Attachments: show-space-consumed.txt
>
>
> Currently FsShell report HDFS usage with "hadoop fs -dus <path>" command.  Since replication level is per file level, it would be nice to add raw disk usage including the replication factor (maybe "hadoop fs -dus -raw <path>"?).  This will allow to assess resource usage more accurately.  -- Alex K

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10309-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 23:10:40 2010
Return-Path: <common-issues-return-10309-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22073 invoked from network); 18 Aug 2010 23:10:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 23:10:40 -0000
Received: (qmail 59633 invoked by uid 500); 18 Aug 2010 23:10:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59597 invoked by uid 500); 18 Aug 2010 23:10:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59587 invoked by uid 99); 18 Aug 2010 23:10:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 23:10:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 23:10:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7INAIF1024451
	for <common-issues@hadoop.apache.org>; Wed, 18 Aug 2010 23:10:18 GMT
Message-ID: <15265969.433851282173018524.JavaMail.jira@thor>
Date: Wed, 18 Aug 2010 19:10:18 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6397) topology script called with host
 names instead of IP addresses
In-Reply-To: <1893554762.1259519900657.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6397:
--------------------------------

        Status: Resolved  (was: Patch Available)
    Resolution: Duplicate

Resolving as duplicate as this was independently discovered and fixed in HADOOP-6682 (although without the unit test, which would have been nice).  Further improvement to this function was delegated to HADOOP-6824.

> topology script called with host names instead of IP addresses
> --------------------------------------------------------------
>
>                 Key: HADOOP-6397
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6397
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.22.0
>            Reporter: Christian Kunz
>         Attachments: HADOOP-6397-v2.patch, HADOOP-6397.patch
>
>
> According to the 'Hadoop Rack Awareness' section on http://hadoop.apache.org/common/docs/r0.20.0/cluster_setup.html the script/program topology.script.file.name can assume that names are passed in as IP addresses.
> This seems to work for hdfs, but the rack-awareness of the JobTracker does not always work, depending on the hostnames of the cluster, because of a bug in the method normalizeHostName of org.apache.hadoop.net.NetUtils.java:
> It returns the host name as-is when it decides that the string passed-in is already an IP but actually it checks whether the string starts with a valid hex character!
> There is no need to check at all, because InetAddress.getByName is smart enough to not do a dns lookup when the name is an IP address.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10310-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 23:23:58 2010
Return-Path: <common-issues-return-10310-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26763 invoked from network); 18 Aug 2010 23:23:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 23:23:57 -0000
Received: (qmail 72909 invoked by uid 500); 18 Aug 2010 23:23:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72850 invoked by uid 500); 18 Aug 2010 23:23:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72842 invoked by uid 99); 18 Aug 2010 23:23:57 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 23:23:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 23:23:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7INNITe024650
	for <common-issues@hadoop.apache.org>; Wed, 18 Aug 2010 23:23:18 GMT
Message-ID: <33537824.434331282173798460.JavaMail.jira@thor>
Date: Wed, 18 Aug 2010 19:23:18 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6482) GenericOptionsParser constructor
 that takes Options and String[] ignores options
In-Reply-To: <947197260.111021262909095193.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900106#action_12900106 ] 

Hudson commented on HADOOP-6482:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #361 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/361/])
    HADOOP-6482. GenericOptionsParser constructor that takes Options and String[] ignores options. Contributed by Eli Collins.


> GenericOptionsParser constructor that takes Options and String[] ignores options
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6482
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6482
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.22.0
>            Reporter: Chris Wilkes
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6482-1.patch, hadoop-6482-2.patch
>
>   Original Estimate: 0.08h
>  Remaining Estimate: 0.08h
>
> This constructor:
>   public GenericOptionsParser(Options opts, String[] args) {
>     this(new Configuration(), new Options(), args);
>   }
> should do a this(new Configuration(), opts, args)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10311-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 18 23:33:42 2010
Return-Path: <common-issues-return-10311-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30940 invoked from network); 18 Aug 2010 23:33:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Aug 2010 23:33:39 -0000
Received: (qmail 78426 invoked by uid 500); 18 Aug 2010 23:33:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78392 invoked by uid 500); 18 Aug 2010 23:33:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78384 invoked by uid 99); 18 Aug 2010 23:33:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 23:33:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Aug 2010 23:33:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7INXHiT024709
	for <common-issues@hadoop.apache.org>; Wed, 18 Aug 2010 23:33:18 GMT
Message-ID: <6501525.434471282174397740.JavaMail.jira@thor>
Date: Wed, 18 Aug 2010 19:33:17 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6194) Add service base class and tests to
 hadoop-common/util
In-Reply-To: <165334509.1250255474864.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900109#action_12900109 ] 

Tom White commented on HADOOP-6194:
-----------------------------------

I just came across the [Service|http://guava-libraries.googlecode.com/svn/trunk/javadoc/com/google/common/base/Service.html] class in the [Guava|http://guava-libraries.googlecode.com/] library, which looks like it does something very similar. Would it be worth considering using this?

Interestingly, there's also a [@Beta|http://guava-libraries.googlecode.com/svn/trunk/javadoc/com/google/common/annotations/Beta.html] annotation, which is like InterfaceStability.Unstable.

> Add service base class and tests to hadoop-common/util
> ------------------------------------------------------
>
>                 Key: HADOOP-6194
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6194
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>             Fix For: 0.22.0
>
>         Attachments: find-new-warnings, HADOOP-6194-1.patch, HADOOP-6194.patch, HADOOP-6194.patch
>
>
> For a service base class, we need to add to common
> # The Service class
> # A mock Service class (it's not in test, its in src/, because it helps in functional tests downstream)
> # Unit tests
> This is the issue to cover this work

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10312-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 19 11:22:42 2010
Return-Path: <common-issues-return-10312-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97030 invoked from network); 19 Aug 2010 11:22:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Aug 2010 11:22:42 -0000
Received: (qmail 70116 invoked by uid 500); 19 Aug 2010 11:22:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69958 invoked by uid 500); 19 Aug 2010 11:22:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69946 invoked by uid 99); 19 Aug 2010 11:22:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 11:22:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 11:22:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7JBMHXJ001290
	for <common-issues@hadoop.apache.org>; Thu, 19 Aug 2010 11:22:17 GMT
Message-ID: <13589200.444331282216937005.JavaMail.jira@thor>
Date: Thu, 19 Aug 2010 07:22:17 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6803) Add native gzip read/write coverage
 to TestCodec
In-Reply-To: <20681213.146461275520316684.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900250#action_12900250 ] 

Hudson commented on HADOOP-6803:
--------------------------------

Integrated in Hadoop-Common-trunk #428 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/428/])
    HADOOP-6803. Add native gzip read/write coverage to TestCodec. Contributed by Eli Collins.


> Add native gzip read/write coverage to TestCodec 
> -------------------------------------------------
>
>                 Key: HADOOP-6803
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6803
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6803-1.patch, hadoop-6803-2.patch, hadoop-6803-3.patch
>
>
> Looking at ZlibCompressor I noticed that the finished member is never modified, and is therefore always false. This means ZlibCompressor#finished will always return false so CompressorStream#close loops indefinitely in finish:
> {code} 
>       while (!compressor.finished()) {
>         compress();
>       }
> {code}
> I modifed TestCodec#testGzipCodecWrite to also cover writing using the native lib and confirmed the hang with jstack. The fix is simple, ZlibCompressor should record when it's been finished.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10313-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 19 11:22:44 2010
Return-Path: <common-issues-return-10313-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97079 invoked from network); 19 Aug 2010 11:22:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Aug 2010 11:22:43 -0000
Received: (qmail 70159 invoked by uid 500); 19 Aug 2010 11:22:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70000 invoked by uid 500); 19 Aug 2010 11:22:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69955 invoked by uid 99); 19 Aug 2010 11:22:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 11:22:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 11:22:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7JBMI4x001308
	for <common-issues@hadoop.apache.org>; Thu, 19 Aug 2010 11:22:18 GMT
Message-ID: <24408310.444391282216938353.JavaMail.jira@thor>
Date: Thu, 19 Aug 2010 07:22:18 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6482) GenericOptionsParser constructor
 that takes Options and String[] ignores options
In-Reply-To: <947197260.111021262909095193.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900252#action_12900252 ] 

Hudson commented on HADOOP-6482:
--------------------------------

Integrated in Hadoop-Common-trunk #428 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/428/])
    HADOOP-6482. GenericOptionsParser constructor that takes Options and String[] ignores options. Contributed by Eli Collins.


> GenericOptionsParser constructor that takes Options and String[] ignores options
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-6482
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6482
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.22.0
>            Reporter: Chris Wilkes
>            Assignee: Eli Collins
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6482-1.patch, hadoop-6482-2.patch
>
>   Original Estimate: 0.08h
>  Remaining Estimate: 0.08h
>
> This constructor:
>   public GenericOptionsParser(Options opts, String[] args) {
>     this(new Configuration(), new Options(), args);
>   }
> should do a this(new Configuration(), opts, args)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10314-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 19 11:22:59 2010
Return-Path: <common-issues-return-10314-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97197 invoked from network); 19 Aug 2010 11:22:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Aug 2010 11:22:58 -0000
Received: (qmail 70483 invoked by uid 500); 19 Aug 2010 11:22:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70326 invoked by uid 500); 19 Aug 2010 11:22:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70313 invoked by uid 99); 19 Aug 2010 11:22:56 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 11:22:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 11:22:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7JBMHgt001296
	for <common-issues@hadoop.apache.org>; Thu, 19 Aug 2010 11:22:17 GMT
Message-ID: <8060820.444351282216937734.JavaMail.jira@thor>
Date: Thu, 19 Aug 2010 07:22:17 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6885) Fix java doc warnings in Groups and
 RefreshUserMappingsProtocol
In-Reply-To: <14122960.38041280281637802.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900251#action_12900251 ] 

Hudson commented on HADOOP-6885:
--------------------------------

Integrated in Hadoop-Common-trunk #428 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/428/])
    HADOOP-6885. Fix java doc warnings in Groups and RefreshUserMappingsProtocol. Contributed by Eli Collins.


> Fix java doc warnings in Groups and RefreshUserMappingsProtocol
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6885
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6885
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6885-1.patch, hadoop-6885-2.patch
>
>
> There are a couple java docs warnings in Groups and RefreshUserMappingsProtocol.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10315-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 19 16:36:00 2010
Return-Path: <common-issues-return-10315-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76435 invoked from network); 19 Aug 2010 16:36:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Aug 2010 16:36:00 -0000
Received: (qmail 36853 invoked by uid 500); 19 Aug 2010 16:35:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36796 invoked by uid 500); 19 Aug 2010 16:35:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36788 invoked by uid 99); 19 Aug 2010 16:35:58 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 16:35:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 16:35:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7JGZJgx004880
	for <common-issues@hadoop.apache.org>; Thu, 19 Aug 2010 16:35:19 GMT
Message-ID: <11721304.449711282235719464.JavaMail.jira@thor>
Date: Thu, 19 Aug 2010 12:35:19 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated HADOOP-6884:
---------------------------------

    Attachment: FunAgain.java

You've benchmarked a case that isn't in the existing patch (3 variables) and certainly isn't typical.  Your 'static' case still boxes.

So your benchmark mostly shows that boxing costs, and string concatenation costs even more yet.  I don't see how it shows that any of these costs are significant in the Hadoop codebase.

I've attached a version that benchmarks the more typical single-parameter using slf4j and a static version that avoids boxing.

The output this gives is:

java.version = 1.6.0_20
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_20-b02
java.vm.version = 16.3-b01
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = i386
os.name = Linux
os.version = 2.6.32-24-generic-pae

n=10000000
LOG.isDebugEnabled(): 13 ms
static debug(..)    : 78 ms
LOG.debug(..)       : 472 ms

What does this show?  It shows that simply switching to slf4j using format strings would remove string concatenation costs speeding log statements by around 10x without any bloat.  It does not show whether that improvement would be significant in any larger Hadoop benchmark, but at least the cost in code readability would be null.  Further it shows that, if we find that logging costs are significant somewhere due to boxing, we could optimize that with static methods, gaining another 5x without losing any readability.  Finally, if we find that logging costs are still somewhere significant, we can improve them yet another 6x with some impairment to readability.  This is a bit like manual loop unrolling.  We'll do it in certain performance-critical areas when it's shown to provide a significant advantage, but we shouldn't do it blindly for every loop.


> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10316-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 19 17:41:45 2010
Return-Path: <common-issues-return-10316-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13111 invoked from network); 19 Aug 2010 17:41:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Aug 2010 17:41:45 -0000
Received: (qmail 67114 invoked by uid 500); 19 Aug 2010 17:41:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67066 invoked by uid 500); 19 Aug 2010 17:41:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67058 invoked by uid 99); 19 Aug 2010 17:41:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 17:41:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 17:41:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7JHfK9n005657
	for <common-issues@hadoop.apache.org>; Thu, 19 Aug 2010 17:41:20 GMT
Message-ID: <32372569.450871282239680197.JavaMail.jira@thor>
Date: Thu, 19 Aug 2010 13:41:20 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900373#action_12900373 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6884:
------------------------------------------------

> You've benchmarked a case that isn't in the existing patch (3 variables) and certainly isn't typical. ...

Which patch are you talking about?  I found at least two three-variable cases in Erik's latest patch.

{code}
-        LOG.debug("Exception while invoking " + method.getName()
-                 + " of " + implementation.getClass() + ". Retrying."
-                 + StringUtils.stringifyException(e));
+        if(LOG.isDebugEnabled()) {
+          LOG.debug("Exception while invoking " + method.getName()
+              + " of " + implementation.getClass() + ". Retrying."
+              + StringUtils.stringifyException(e));
+        }
{code}


{code}
-    LOG.debug("for protocol authorization compare (" + clientPrincipal + "): " 
-        + shortName + " with " + user.getShortUserName());
+    if(LOG.isDebugEnabled()) {
+      LOG.debug("for protocol authorization compare (" + clientPrincipal +
+          "): " + shortName + " with " + user.getShortUserName());
{code}

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10317-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 19 17:57:58 2010
Return-Path: <common-issues-return-10317-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17866 invoked from network); 19 Aug 2010 17:57:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Aug 2010 17:57:58 -0000
Received: (qmail 89609 invoked by uid 500); 19 Aug 2010 17:57:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89471 invoked by uid 500); 19 Aug 2010 17:57:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89460 invoked by uid 99); 19 Aug 2010 17:57:57 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 17:57:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 17:57:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7JHvIWN005802
	for <common-issues@hadoop.apache.org>; Thu, 19 Aug 2010 17:57:18 GMT
Message-ID: <24575077.451091282240638189.JavaMail.jira@thor>
Date: Thu, 19 Aug 2010 13:57:18 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900377#action_12900377 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6884:
------------------------------------------------

> What does this show? ...

Your benchmark also shows that LOG.isDebugEnabled() provides the best results.  A patch using this approach is already there.  Do you really want to stop committing it?

There may be other comparable solutions out there but (1) we don't have a patch yet, (2) it may require further modifying the codes and (3) the performance is not as good as LOG.isDebugEnabled().

I forgot to mention that my benchmark only measures running time but it does not account the GC overhead induced by unnecessary object creation due to auto-boxing.  This is another reason that LOG.isDebugEnabled() is better than the other suggested solution.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10318-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 19 18:23:03 2010
Return-Path: <common-issues-return-10318-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35720 invoked from network); 19 Aug 2010 18:23:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Aug 2010 18:23:03 -0000
Received: (qmail 25805 invoked by uid 500); 19 Aug 2010 18:23:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25706 invoked by uid 500); 19 Aug 2010 18:23:02 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25698 invoked by uid 99); 19 Aug 2010 18:23:02 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 18:23:02 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 18:22:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7JIMNTC006151
	for <common-issues@hadoop.apache.org>; Thu, 19 Aug 2010 18:22:23 GMT
Message-ID: <19354378.451651282242143804.JavaMail.jira@thor>
Date: Thu, 19 Aug 2010 14:22:23 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900390#action_12900390 ] 

Doug Cutting commented on HADOOP-6884:
--------------------------------------

>  I found at least two three-variable cases in Erik's latest patch.

I missed those, but it's still different from your benchmark, which focussed on boxed variables.  Those examples might look something like:

debug("Exception while invoking {} of {}. Retrying.", method.getName(), implementation.getClass(), StringUtils.stringifyException(e));

That last parameter would make things slow, but, I'd be surprised if you can show that optimizing this substantially improves Hadoop performance, since this only runs in the case that a retryable method fails with an exception.  If we ever found that such cases were problematic, we might add methods like:

static void debug(String message, Throwable t);
static void debug(String message, Throwable t, Object p1);
static void debug(String message, Throwable t, Object p1, Object p2);
static void debug(String message, Throwable t, Object... parameters);

Perhaps we could submit a patch to slf4j adding such methods there.

The second case could become something like:

debug("for protocol authorization compare ({}):{} with ", clientPrincipal, shortName, user.getShortUserName));

In this case all parameters are strings, so no boxing would occur.

> Your benchmark also shows that LOG.isDebugEnabled() provides the best results.

It provides significantly better results in a micro-benchmark.  But optimizations that increase code size should show significant improvement in real application performance, not just micro-benchmarks.  Until someone's shown that this optimization measurably improves end performance, I see no urgency to commit it.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10319-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 19 21:46:45 2010
Return-Path: <common-issues-return-10319-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39777 invoked from network); 19 Aug 2010 21:46:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Aug 2010 21:46:44 -0000
Received: (qmail 76497 invoked by uid 500); 19 Aug 2010 21:46:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76422 invoked by uid 500); 19 Aug 2010 21:46:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76414 invoked by uid 99); 19 Aug 2010 21:46:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 21:46:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 21:46:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7JLkN61008655
	for <common-issues@hadoop.apache.org>; Thu, 19 Aug 2010 21:46:23 GMT
Message-ID: <1378780.455961282254383306.JavaMail.jira@thor>
Date: Thu, 19 Aug 2010 17:46:23 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900480#action_12900480 ] 

Erik Steffl commented on HADOOP-6884:
-------------------------------------

To put things in perspective:

  - the practice of using isDebugEnabled() is already common in Hadoop code, this patch just makes it more consistent

  - HDFS 187 in patch, 86 before patch

  - Common 63 in patch, 52 before patch

Note: patch adds isDebugEnabled to tests as well so the counts that matter are even lower.

None of the proposed solutions is solving all the problems and none of them is a short term solution. The only complaint against the patch is that it looks inelegant but from the discussion in this bug (and looking up others' effort to do the same) there's really no elegant solution for this problem in Java.

Claiming that patch adds code is a bit tricky. Yes it adds to total code written but in production runs it means less code being executed and less objects created (in alternative proposals even if the objects are not stringified they are still created). Given that we already have problems with GC that seems like a good idea.

We can still work on a long term better solution, once we have one we would have to convert all the code to the new solution anyway (at least remove already existing isDebugEnabled calls). Patch does not add significant (possibly none) additional work for that case.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10320-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 19 21:50:57 2010
Return-Path: <common-issues-return-10320-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40188 invoked from network); 19 Aug 2010 21:50:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Aug 2010 21:50:56 -0000
Received: (qmail 77240 invoked by uid 500); 19 Aug 2010 21:50:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77190 invoked by uid 500); 19 Aug 2010 21:50:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77182 invoked by uid 99); 19 Aug 2010 21:50:56 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 21:50:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 21:50:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7JLoHfe008689
	for <common-issues@hadoop.apache.org>; Thu, 19 Aug 2010 21:50:17 GMT
Message-ID: <19041039.456051282254617368.JavaMail.jira@thor>
Date: Thu, 19 Aug 2010 17:50:17 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900481#action_12900481 ] 

Suresh Srinivas commented on HADOOP-6884:
-----------------------------------------

I agree with Erik. I am actually surprised by how many comments this simple proposal has generated!

My +1 for committing the patch with a minor change - we do not need isDebugEnabled() check for tests.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10321-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 20 00:21:39 2010
Return-Path: <common-issues-return-10321-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97774 invoked from network); 20 Aug 2010 00:21:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Aug 2010 00:21:39 -0000
Received: (qmail 18171 invoked by uid 500); 20 Aug 2010 00:21:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18110 invoked by uid 500); 20 Aug 2010 00:21:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18101 invoked by uid 99); 20 Aug 2010 00:21:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 00:21:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 00:21:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7K0LHOg010406
	for <common-issues@hadoop.apache.org>; Fri, 20 Aug 2010 00:21:17 GMT
Message-ID: <4692429.459761282263677537.JavaMail.jira@thor>
Date: Thu, 19 Aug 2010 20:21:17 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900533#action_12900533 ] 

Doug Cutting commented on HADOOP-6884:
--------------------------------------

The unanswered question is whether this patch does anything useful for the project.  Does it actually make Hadoop measurably faster?  Shouldn't we demonstrate that for optimizations that increase the code size, a cost/benefit analysis?

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10322-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 20 03:48:42 2010
Return-Path: <common-issues-return-10322-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64813 invoked from network); 20 Aug 2010 03:48:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Aug 2010 03:48:42 -0000
Received: (qmail 6791 invoked by uid 500); 20 Aug 2010 03:48:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6640 invoked by uid 500); 20 Aug 2010 03:48:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6631 invoked by uid 99); 20 Aug 2010 03:48:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 03:48:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 03:48:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7K3mHtw011767
	for <common-issues@hadoop.apache.org>; Fri, 20 Aug 2010 03:48:17 GMT
Message-ID: <27447198.461561282276097452.JavaMail.jira@thor>
Date: Thu, 19 Aug 2010 23:48:17 -0400 (EDT)
From: "Mahadev konar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6906) FileContext copy() utility doesn't
 work with recursive copying of directories.
In-Reply-To: <31039707.230631281349276933.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mahadev konar updated HADOOP-6906:
----------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

I just committed this.

thanks vinod!


> FileContext copy() utility doesn't work with recursive copying of directories.
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6906
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6906
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Vinod K V
>            Assignee: Vinod K V
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6906-20100809.1.txt, HADOOP-6906-regression.1.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10323-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 20 04:12:41 2010
Return-Path: <common-issues-return-10323-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73209 invoked from network); 20 Aug 2010 04:12:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Aug 2010 04:12:41 -0000
Received: (qmail 13843 invoked by uid 500); 20 Aug 2010 04:12:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13704 invoked by uid 500); 20 Aug 2010 04:12:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13696 invoked by uid 99); 20 Aug 2010 04:12:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 04:12:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 04:12:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7K4CHdl012396
	for <common-issues@hadoop.apache.org>; Fri, 20 Aug 2010 04:12:17 GMT
Message-ID: <17069114.461751282277537057.JavaMail.jira@thor>
Date: Fri, 20 Aug 2010 00:12:17 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6906) FileContext copy() utility doesn't
 work with recursive copying of directories.
In-Reply-To: <31039707.230631281349276933.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900569#action_12900569 ] 

Hudson commented on HADOOP-6906:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #362 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/362/])
    HADOOP-6906.  FileContext copy() utility doesn't work with recursive copying of directories. (vinod k v via mahadev)


> FileContext copy() utility doesn't work with recursive copying of directories.
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6906
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6906
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Vinod K V
>            Assignee: Vinod K V
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6906-20100809.1.txt, HADOOP-6906-regression.1.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10324-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 20 08:01:22 2010
Return-Path: <common-issues-return-10324-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74364 invoked from network); 20 Aug 2010 08:01:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Aug 2010 08:01:21 -0000
Received: (qmail 86249 invoked by uid 500); 20 Aug 2010 08:01:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85856 invoked by uid 500); 20 Aug 2010 08:01:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85726 invoked by uid 99); 20 Aug 2010 08:01:17 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 08:01:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 08:01:05 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7K80iUQ014500
	for <common-issues@hadoop.apache.org>; Fri, 20 Aug 2010 08:00:44 GMT
Message-ID: <31572721.465111282291244338.JavaMail.jira@thor>
Date: Fri, 20 Aug 2010 04:00:44 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6870) Add FileSystem#listLocatedStatus to
 list a directory's content together with each file's block locations
In-Reply-To: <13042840.506371279749289722.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900626#action_12900626 ] 

Hudson commented on HADOOP-6870:
--------------------------------

Integrated in Hadoop-Hdfs-trunk-Commit #370 (See [https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/370/])
    

> Add FileSystem#listLocatedStatus to list a directory's content together with each file's block locations
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6870
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6870
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.22.0
>
>         Attachments: listFiles.patch, listFiles1.patch, listFiles2.patch, listFiles3.patch, listFiles4.patch
>
>
> This jira implements the new FileSystem API as proposed in HDFS-202. The new API aims to eliminate individual "getFileBlockLocations" calls to NN for each file in the input directory of a job. Instead, a file's block locations are returned together with FileStatus when listing a directory, thus improving getSplits performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10325-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 20 11:28:43 2010
Return-Path: <common-issues-return-10325-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50778 invoked from network); 20 Aug 2010 11:28:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Aug 2010 11:28:43 -0000
Received: (qmail 76696 invoked by uid 500); 20 Aug 2010 11:28:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76545 invoked by uid 500); 20 Aug 2010 11:28:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76535 invoked by uid 99); 20 Aug 2010 11:28:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 11:28:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 11:28:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7KBSI5t016454
	for <common-issues@hadoop.apache.org>; Fri, 20 Aug 2010 11:28:18 GMT
Message-ID: <24022679.467711282303698381.JavaMail.jira@thor>
Date: Fri, 20 Aug 2010 07:28:18 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6906) FileContext copy() utility doesn't
 work with recursive copying of directories.
In-Reply-To: <31039707.230631281349276933.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900668#action_12900668 ] 

Hudson commented on HADOOP-6906:
--------------------------------

Integrated in Hadoop-Common-trunk #429 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/429/])
    HADOOP-6906.  FileContext copy() utility doesn't work with recursive copying of directories. (vinod k v via mahadev)


> FileContext copy() utility doesn't work with recursive copying of directories.
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6906
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6906
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Vinod K V
>            Assignee: Vinod K V
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6906-20100809.1.txt, HADOOP-6906-regression.1.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10326-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 20 12:53:03 2010
Return-Path: <common-issues-return-10326-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83869 invoked from network); 20 Aug 2010 12:53:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Aug 2010 12:53:03 -0000
Received: (qmail 68714 invoked by uid 500); 20 Aug 2010 12:53:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68440 invoked by uid 500); 20 Aug 2010 12:53:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68428 invoked by uid 99); 20 Aug 2010 12:52:58 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 12:52:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 12:52:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7KCqJP6017227
	for <common-issues@hadoop.apache.org>; Fri, 20 Aug 2010 12:52:20 GMT
Message-ID: <23971861.468581282308739879.JavaMail.jira@thor>
Date: Fri, 20 Aug 2010 08:52:19 -0400 (EDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6194) Add service base class and tests to
 hadoop-common/util
In-Reply-To: <165334509.1250255474864.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900682#action_12900682 ] 

Steve Loughran commented on HADOOP-6194:
----------------------------------------

Looks similar, as their basic lifecycle set of states matches fairly well, they have a stopping state to describe slow shutdowns which could make sense. It's the same philosophy of how you design a class to represent a subclassable service with a lifecycle that can be aggregated.

Their base class itself is a bit minimal, I like to record why something failed. "Failed" isn't enough, exceptions are handy. I'd also like it to implement Closeable as that's generally handy.

If consensus was to adopt the guava lib as yet-another-dependency, there's no reason why their AbstractServiceBase couldn't be used, but I'd still want a HadoopServiceBase under this which would be our base class and have extra methods in there. 

However, we don't need to rush to Guava just because we would gain a lot from having our own service base class, be it our own or descended from the Guava one.  

> Add service base class and tests to hadoop-common/util
> ------------------------------------------------------
>
>                 Key: HADOOP-6194
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6194
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>             Fix For: 0.22.0
>
>         Attachments: find-new-warnings, HADOOP-6194-1.patch, HADOOP-6194.patch, HADOOP-6194.patch
>
>
> For a service base class, we need to add to common
> # The Service class
> # A mock Service class (it's not in test, its in src/, because it helps in functional tests downstream)
> # Unit tests
> This is the issue to cover this work

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10327-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 20 13:10:46 2010
Return-Path: <common-issues-return-10327-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93965 invoked from network); 20 Aug 2010 13:10:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Aug 2010 13:10:45 -0000
Received: (qmail 99675 invoked by uid 500); 20 Aug 2010 13:10:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99393 invoked by uid 500); 20 Aug 2010 13:10:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99384 invoked by uid 99); 20 Aug 2010 13:10:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 13:10:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 13:10:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7KDALSk017453
	for <common-issues@hadoop.apache.org>; Fri, 20 Aug 2010 13:10:21 GMT
Message-ID: <13688256.468811282309821661.JavaMail.jira@thor>
Date: Fri, 20 Aug 2010 09:10:21 -0400 (EDT)
From: "Mahadev konar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6906) FileContext copy() utility doesn't
 work with recursive copying of directories.
In-Reply-To: <31039707.230631281349276933.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900686#action_12900686 ] 

Mahadev konar commented on HADOOP-6906:
---------------------------------------

sorry I forgot to mention I ran ant test and 
and test-patch. They both pass!

> FileContext copy() utility doesn't work with recursive copying of directories.
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6906
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6906
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Vinod K V
>            Assignee: Vinod K V
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6906-20100809.1.txt, HADOOP-6906-regression.1.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10328-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 20 14:34:00 2010
Return-Path: <common-issues-return-10328-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28903 invoked from network); 20 Aug 2010 14:34:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Aug 2010 14:34:00 -0000
Received: (qmail 49745 invoked by uid 500); 20 Aug 2010 14:34:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49588 invoked by uid 500); 20 Aug 2010 14:33:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49577 invoked by uid 99); 20 Aug 2010 14:33:57 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 14:33:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 14:33:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7KEXIfr027846
	for <common-issues@hadoop.apache.org>; Fri, 20 Aug 2010 14:33:18 GMT
Message-ID: <17874503.469941282314798336.JavaMail.jira@thor>
Date: Fri, 20 Aug 2010 10:33:18 -0400 (EDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6194) Add service base class and tests to
 hadoop-common/util
In-Reply-To: <165334509.1250255474864.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900707#action_12900707 ] 

Steve Loughran commented on HADOOP-6194:
----------------------------------------

one other thing to consider is that the jetty instances that HDFS and the JT brings up are a service too, and they should be manageable as well, as are, effectively the in-JVM HDFS and MR clusters. I haven't gone near that code, yet, but it would make things consistent.

> Add service base class and tests to hadoop-common/util
> ------------------------------------------------------
>
>                 Key: HADOOP-6194
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6194
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: util
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>             Fix For: 0.22.0
>
>         Attachments: find-new-warnings, HADOOP-6194-1.patch, HADOOP-6194.patch, HADOOP-6194.patch
>
>
> For a service base class, we need to add to common
> # The Service class
> # A mock Service class (it's not in test, its in src/, because it helps in functional tests downstream)
> # Unit tests
> This is the issue to cover this work

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10329-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 20 18:29:38 2010
Return-Path: <common-issues-return-10329-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28766 invoked from network); 20 Aug 2010 18:29:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Aug 2010 18:29:38 -0000
Received: (qmail 90591 invoked by uid 500); 20 Aug 2010 18:29:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90539 invoked by uid 500); 20 Aug 2010 18:29:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90531 invoked by uid 99); 20 Aug 2010 18:29:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 18:29:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 18:29:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7KITGNa000536
	for <common-issues@hadoop.apache.org>; Fri, 20 Aug 2010 18:29:16 GMT
Message-ID: <5727198.474711282328956153.JavaMail.jira@thor>
Date: Fri, 20 Aug 2010 14:29:16 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6917) Remove empty
 FTPFileSystemConfigKeys.java file
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Remove empty FTPFileSystemConfigKeys.java file
----------------------------------------------

                 Key: HADOOP-6917
                 URL: https://issues.apache.org/jira/browse/HADOOP-6917
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.21.0
            Reporter: Tom White


FTPFileSystemConfigKeys.java is empty and not used so should be deleted.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10330-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 20 19:12:40 2010
Return-Path: <common-issues-return-10330-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46524 invoked from network); 20 Aug 2010 19:12:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Aug 2010 19:12:40 -0000
Received: (qmail 51647 invoked by uid 500); 20 Aug 2010 19:12:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51529 invoked by uid 500); 20 Aug 2010 19:12:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51476 invoked by uid 99); 20 Aug 2010 19:12:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 19:12:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 19:12:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7KJCHXv001108
	for <common-issues@hadoop.apache.org>; Fri, 20 Aug 2010 19:12:17 GMT
Message-ID: <32421393.475691282331537072.JavaMail.jira@thor>
Date: Fri, 20 Aug 2010 15:12:17 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6786) test-patch needs to verify Herriot
 integrity
In-Reply-To: <19336182.46451274829212745.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900832#action_12900832 ] 

Konstantin Boudnik commented on HADOOP-6786:
--------------------------------------------

Hudson seems to be not at his best for the last a couple of weeks. I have ran {{test-patch.sh}} on my dev. machine.
{noformat}
-1 overall.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 4 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.
{noformat}

The javadoc warning is apparently not connected to the patch itself. The only warnings I can see are in security related classes.


> test-patch needs to verify Herriot integrity
> --------------------------------------------
>
>                 Key: HADOOP-6786
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6786
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-6786.patch, HADOOP-6786.patch, HADOOP-6786.patch
>
>
> Whenever a new patch is submitted for verification {{test-patch}} process has to make sure that none of Herriot bindings were broken.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10331-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 20 19:12:41 2010
Return-Path: <common-issues-return-10331-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46575 invoked from network); 20 Aug 2010 19:12:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Aug 2010 19:12:41 -0000
Received: (qmail 51785 invoked by uid 500); 20 Aug 2010 19:12:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51577 invoked by uid 500); 20 Aug 2010 19:12:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51482 invoked by uid 99); 20 Aug 2010 19:12:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 19:12:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 19:12:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7KJCIrQ001129
	for <common-issues@hadoop.apache.org>; Fri, 20 Aug 2010 19:12:18 GMT
Message-ID: <15475783.475761282331538503.JavaMail.jira@thor>
Date: Fri, 20 Aug 2010 15:12:18 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6786) test-patch needs to verify Herriot
 integrity
In-Reply-To: <19336182.46451274829212745.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12900834#action_12900834 ] 

Konstantin Boudnik commented on HADOOP-6786:
--------------------------------------------

I'll be committing this by end of today or Monday if there's no objections.

> test-patch needs to verify Herriot integrity
> --------------------------------------------
>
>                 Key: HADOOP-6786
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6786
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-6786.patch, HADOOP-6786.patch, HADOOP-6786.patch
>
>
> Whenever a new patch is submitted for verification {{test-patch}} process has to make sure that none of Herriot bindings were broken.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10332-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 20 20:38:39 2010
Return-Path: <common-issues-return-10332-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83227 invoked from network); 20 Aug 2010 20:38:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Aug 2010 20:38:39 -0000
Received: (qmail 34638 invoked by uid 500); 20 Aug 2010 20:38:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34564 invoked by uid 500); 20 Aug 2010 20:38:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34551 invoked by uid 99); 20 Aug 2010 20:38:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 20:38:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 20:38:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7KKcH3N001912
	for <common-issues@hadoop.apache.org>; Fri, 20 Aug 2010 20:38:17 GMT
Message-ID: <23032818.477051282336697507.JavaMail.jira@thor>
Date: Fri, 20 Aug 2010 16:38:17 -0400 (EDT)
From: "Chad Metcalf (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6453) Hadoop wrapper script shouldn't
 ignore an existing JAVA_LIBRARY_PATH
In-Reply-To: <909267607.1261086858225.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chad Metcalf updated HADOOP-6453:
---------------------------------

    Attachment: HADOOP-6453-trunkv3.patch

Thanks.

> Hadoop wrapper script shouldn't ignore an existing JAVA_LIBRARY_PATH
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6453
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6453
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2, 0.21.0, 0.22.0
>            Reporter: Chad Metcalf
>            Assignee: Chad Metcalf
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6453-0.20.patch, HADOOP-6453-0.20v2.patch, HADOOP-6453-trunkv2.patch, HADOOP-6453-trunkv3.patch, HADOOP-6453.trunk.patch
>
>
> Currently the hadoop wrapper script assumes its the only place that uses JAVA_LIBRARY_PATH and initializes it to a blank line.
> JAVA_LIBRARY_PATH=''
> This prevents anyone from setting this outside of the hadoop wrapper (say hadoop-config.sh) for their own native libraries.
> The fix is pretty simple. Don't initialize it to '' and append the native libs like normal. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10333-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 20 20:38:40 2010
Return-Path: <common-issues-return-10333-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83268 invoked from network); 20 Aug 2010 20:38:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Aug 2010 20:38:40 -0000
Received: (qmail 34784 invoked by uid 500); 20 Aug 2010 20:38:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34734 invoked by uid 500); 20 Aug 2010 20:38:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34726 invoked by uid 99); 20 Aug 2010 20:38:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 20:38:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 20:38:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7KKcJ0C001948
	for <common-issues@hadoop.apache.org>; Fri, 20 Aug 2010 20:38:19 GMT
Message-ID: <22149066.477171282336699439.JavaMail.jira@thor>
Date: Fri, 20 Aug 2010 16:38:19 -0400 (EDT)
From: "Chad Metcalf (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6453) Hadoop wrapper script shouldn't
 ignore an existing JAVA_LIBRARY_PATH
In-Reply-To: <909267607.1261086858225.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chad Metcalf updated HADOOP-6453:
---------------------------------

    Attachment: HADOOP-6453-trunkv3.patch

Thanks

> Hadoop wrapper script shouldn't ignore an existing JAVA_LIBRARY_PATH
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6453
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6453
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2, 0.21.0, 0.22.0
>            Reporter: Chad Metcalf
>            Assignee: Chad Metcalf
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6453-0.20.patch, HADOOP-6453-0.20v2.patch, HADOOP-6453-trunkv2.patch, HADOOP-6453-trunkv3.patch, HADOOP-6453.trunk.patch
>
>
> Currently the hadoop wrapper script assumes its the only place that uses JAVA_LIBRARY_PATH and initializes it to a blank line.
> JAVA_LIBRARY_PATH=''
> This prevents anyone from setting this outside of the hadoop wrapper (say hadoop-config.sh) for their own native libraries.
> The fix is pretty simple. Don't initialize it to '' and append the native libs like normal. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10334-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 20 20:38:41 2010
Return-Path: <common-issues-return-10334-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83306 invoked from network); 20 Aug 2010 20:38:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Aug 2010 20:38:41 -0000
Received: (qmail 34948 invoked by uid 500); 20 Aug 2010 20:38:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34913 invoked by uid 500); 20 Aug 2010 20:38:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34905 invoked by uid 99); 20 Aug 2010 20:38:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 20:38:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 20:38:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7KKcKrs001966
	for <common-issues@hadoop.apache.org>; Fri, 20 Aug 2010 20:38:20 GMT
Message-ID: <21832633.477231282336700326.JavaMail.jira@thor>
Date: Fri, 20 Aug 2010 16:38:20 -0400 (EDT)
From: "Chad Metcalf (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6453) Hadoop wrapper script shouldn't
 ignore an existing JAVA_LIBRARY_PATH
In-Reply-To: <909267607.1261086858225.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chad Metcalf updated HADOOP-6453:
---------------------------------

          Status: Patch Available  (was: Open)
    Hadoop Flags: [Incompatible change]

> Hadoop wrapper script shouldn't ignore an existing JAVA_LIBRARY_PATH
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6453
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6453
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2, 0.21.0, 0.22.0
>            Reporter: Chad Metcalf
>            Assignee: Chad Metcalf
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6453-0.20.patch, HADOOP-6453-0.20v2.patch, HADOOP-6453-trunkv2.patch, HADOOP-6453-trunkv3.patch, HADOOP-6453.trunk.patch
>
>
> Currently the hadoop wrapper script assumes its the only place that uses JAVA_LIBRARY_PATH and initializes it to a blank line.
> JAVA_LIBRARY_PATH=''
> This prevents anyone from setting this outside of the hadoop wrapper (say hadoop-config.sh) for their own native libraries.
> The fix is pretty simple. Don't initialize it to '' and append the native libs like normal. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10335-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 20 20:38:57 2010
Return-Path: <common-issues-return-10335-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83465 invoked from network); 20 Aug 2010 20:38:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Aug 2010 20:38:57 -0000
Received: (qmail 37953 invoked by uid 500); 20 Aug 2010 20:38:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37919 invoked by uid 500); 20 Aug 2010 20:38:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37911 invoked by uid 99); 20 Aug 2010 20:38:56 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 20:38:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Aug 2010 20:38:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7KKcI5i001930
	for <common-issues@hadoop.apache.org>; Fri, 20 Aug 2010 20:38:18 GMT
Message-ID: <9299573.477111282336698361.JavaMail.jira@thor>
Date: Fri, 20 Aug 2010 16:38:18 -0400 (EDT)
From: "Chad Metcalf (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6453) Hadoop wrapper script shouldn't
 ignore an existing JAVA_LIBRARY_PATH
In-Reply-To: <909267607.1261086858225.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chad Metcalf updated HADOOP-6453:
---------------------------------

    Attachment:     (was: HADOOP-6453-trunkv3.patch)

> Hadoop wrapper script shouldn't ignore an existing JAVA_LIBRARY_PATH
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6453
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6453
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2, 0.21.0, 0.22.0
>            Reporter: Chad Metcalf
>            Assignee: Chad Metcalf
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6453-0.20.patch, HADOOP-6453-0.20v2.patch, HADOOP-6453-trunkv2.patch, HADOOP-6453-trunkv3.patch, HADOOP-6453.trunk.patch
>
>
> Currently the hadoop wrapper script assumes its the only place that uses JAVA_LIBRARY_PATH and initializes it to a blank line.
> JAVA_LIBRARY_PATH=''
> This prevents anyone from setting this outside of the hadoop wrapper (say hadoop-config.sh) for their own native libraries.
> The fix is pretty simple. Don't initialize it to '' and append the native libs like normal. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10336-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 21 00:12:39 2010
Return-Path: <common-issues-return-10336-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55064 invoked from network); 21 Aug 2010 00:12:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Aug 2010 00:12:39 -0000
Received: (qmail 64803 invoked by uid 500); 21 Aug 2010 00:12:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64649 invoked by uid 500); 21 Aug 2010 00:12:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64641 invoked by uid 99); 21 Aug 2010 00:12:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Aug 2010 00:12:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Aug 2010 00:12:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7L0CHtY004250
	for <common-issues@hadoop.apache.org>; Sat, 21 Aug 2010 00:12:17 GMT
Message-ID: <8947862.481731282349537597.JavaMail.jira@thor>
Date: Fri, 20 Aug 2010 20:12:17 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6902) RPC server's SASL_PROPS shouldn't be
 re-initialized every time an RPC client is created
In-Reply-To: <31093459.190711281051317983.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6902:
------------------------------

    Attachment: c6902-03.patch

Attaching a preliminary patch for review. 

1. To simplify the code, I chose the simple use model of a cluster-wide property that either you use encryption for all your RPC connections or not. There is no support for selective encryption on a per-connection basis. 

2. Removed splitKerberosName() method and use KerberosName class instead

3. re-factored createSaslServer methods from Server to SaslRpcServer to have better encapsulation of SASL_PROS.

4. SASL_PROS is set according to the conf used to call UGI.setConfiguration(conf). Otherwise, it defaults to no-encryption. Anyone sees any problem here?


> RPC server's SASL_PROPS shouldn't be re-initialized every time an RPC client is created
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6902
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6902
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, security
>            Reporter: Kan Zhang
>         Attachments: c6902-03.patch
>
>
> SaslRpcServer.SASL_PROPS is a SASL server property and should stay constant after initialization. In the initial implementation, we assumed all SASL clients will use the same constant value. If different clients might use different values depending on the conf in the getProxy() call (as current code implies), each client should have its own copy. In any case, a client shouldn't re-initialize server's copy.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10337-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 21 01:36:00 2010
Return-Path: <common-issues-return-10337-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82277 invoked from network); 21 Aug 2010 01:35:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Aug 2010 01:35:59 -0000
Received: (qmail 98663 invoked by uid 500); 21 Aug 2010 01:35:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98593 invoked by uid 500); 21 Aug 2010 01:35:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98585 invoked by uid 99); 21 Aug 2010 01:35:59 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Aug 2010 01:35:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Aug 2010 01:35:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7L1ZKsi004797
	for <common-issues@hadoop.apache.org>; Sat, 21 Aug 2010 01:35:20 GMT
Message-ID: <28227334.482861282354520301.JavaMail.jira@thor>
Date: Fri, 20 Aug 2010 21:35:20 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6728) Overhaul metrics framework
In-Reply-To: <17506266.48501272420692262.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-6728:
----------------------------

    Attachment: metrics2-uml-r2.png
                metrics2-record-r2.png
                metrics2-mutable-r2.png

> Overhaul metrics framework
> --------------------------
>
>                 Key: HADOOP-6728
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6728
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.2
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: metrics1-flow.png, metrics2-builder-r2.png, metrics2-builder.png, metrics2-flow.png, metrics2-mutable-r2.png, metrics2-mutable.png, metrics2-record-r2.png, metrics2-record.png, metrics2-uml-r2.png, metrics2-uml.png
>
>
> Per discussions with Arun, Chris, Hong and Rajiv, et al, we concluded that the current metrics framework needs an overhaul to:
> * Allow multiple plugins for different monitoring systems simultaneously (see also: HADOOP-6508).
> * Refresh metrics plugin config without server restart.
> ** Including filtering of metrics per plugin.
> * Support metrics schema for plugins.
> The jira will be resolved when core hadoop components (hdfs, mapreduce) are updated to use the new framework . Updates to external components that use the existing metrics framework will be tracked by different issues.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10338-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 21 01:36:01 2010
Return-Path: <common-issues-return-10338-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82325 invoked from network); 21 Aug 2010 01:36:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Aug 2010 01:36:01 -0000
Received: (qmail 98792 invoked by uid 500); 21 Aug 2010 01:36:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98766 invoked by uid 500); 21 Aug 2010 01:36:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98758 invoked by uid 99); 21 Aug 2010 01:36:01 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Aug 2010 01:36:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Aug 2010 01:35:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7L1ZMBa004854
	for <common-issues@hadoop.apache.org>; Sat, 21 Aug 2010 01:35:22 GMT
Message-ID: <24636052.483051282354522212.JavaMail.jira@thor>
Date: Fri, 20 Aug 2010 21:35:22 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6728) Overhaul metrics framework
In-Reply-To: <17506266.48501272420692262.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-6728:
----------------------------

    Attachment: metrics2-builder-r2.png

> Overhaul metrics framework
> --------------------------
>
>                 Key: HADOOP-6728
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6728
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.2
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: metrics1-flow.png, metrics2-builder-r2.png, metrics2-builder.png, metrics2-flow.png, metrics2-mutable-r2.png, metrics2-mutable.png, metrics2-record-r2.png, metrics2-record.png, metrics2-uml-r2.png, metrics2-uml.png
>
>
> Per discussions with Arun, Chris, Hong and Rajiv, et al, we concluded that the current metrics framework needs an overhaul to:
> * Allow multiple plugins for different monitoring systems simultaneously (see also: HADOOP-6508).
> * Refresh metrics plugin config without server restart.
> ** Including filtering of metrics per plugin.
> * Support metrics schema for plugins.
> The jira will be resolved when core hadoop components (hdfs, mapreduce) are updated to use the new framework . Updates to external components that use the existing metrics framework will be tracked by different issues.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10339-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 21 01:43:44 2010
Return-Path: <common-issues-return-10339-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83010 invoked from network); 21 Aug 2010 01:43:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Aug 2010 01:43:44 -0000
Received: (qmail 2246 invoked by uid 500); 21 Aug 2010 01:43:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2169 invoked by uid 500); 21 Aug 2010 01:43:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2161 invoked by uid 99); 21 Aug 2010 01:43:43 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Aug 2010 01:43:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Aug 2010 01:43:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7L1hMeD005018
	for <common-issues@hadoop.apache.org>; Sat, 21 Aug 2010 01:43:22 GMT
Message-ID: <21512094.483531282355002745.JavaMail.jira@thor>
Date: Fri, 20 Aug 2010 21:43:22 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6728) Overhaul metrics framework
In-Reply-To: <17506266.48501272420692262.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-6728:
----------------------------

    Attachment: metrics2-uml-r2.png

> Overhaul metrics framework
> --------------------------
>
>                 Key: HADOOP-6728
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6728
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.2
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: metrics1-flow.png, metrics2-builder-r2.png, metrics2-builder.png, metrics2-flow.png, metrics2-mutable-r2.png, metrics2-mutable.png, metrics2-record-r2.png, metrics2-record.png, metrics2-uml-r2.png, metrics2-uml.png
>
>
> Per discussions with Arun, Chris, Hong and Rajiv, et al, we concluded that the current metrics framework needs an overhaul to:
> * Allow multiple plugins for different monitoring systems simultaneously (see also: HADOOP-6508).
> * Refresh metrics plugin config without server restart.
> ** Including filtering of metrics per plugin.
> * Support metrics schema for plugins.
> The jira will be resolved when core hadoop components (hdfs, mapreduce) are updated to use the new framework . Updates to external components that use the existing metrics framework will be tracked by different issues.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10340-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 21 01:44:00 2010
Return-Path: <common-issues-return-10340-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83079 invoked from network); 21 Aug 2010 01:44:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Aug 2010 01:44:00 -0000
Received: (qmail 2409 invoked by uid 500); 21 Aug 2010 01:44:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2354 invoked by uid 500); 21 Aug 2010 01:43:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2342 invoked by uid 99); 21 Aug 2010 01:43:59 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Aug 2010 01:43:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Aug 2010 01:43:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7L1hKdi004961
	for <common-issues@hadoop.apache.org>; Sat, 21 Aug 2010 01:43:20 GMT
Message-ID: <6893464.483341282355000830.JavaMail.jira@thor>
Date: Fri, 20 Aug 2010 21:43:20 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6728) Overhaul metrics framework
In-Reply-To: <17506266.48501272420692262.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-6728:
----------------------------

    Attachment:     (was: metrics2-uml-r2.png)

> Overhaul metrics framework
> --------------------------
>
>                 Key: HADOOP-6728
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6728
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.2
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: metrics1-flow.png, metrics2-builder-r2.png, metrics2-builder.png, metrics2-flow.png, metrics2-mutable-r2.png, metrics2-mutable.png, metrics2-record-r2.png, metrics2-record.png, metrics2-uml-r2.png, metrics2-uml.png
>
>
> Per discussions with Arun, Chris, Hong and Rajiv, et al, we concluded that the current metrics framework needs an overhaul to:
> * Allow multiple plugins for different monitoring systems simultaneously (see also: HADOOP-6508).
> * Refresh metrics plugin config without server restart.
> ** Including filtering of metrics per plugin.
> * Support metrics schema for plugins.
> The jira will be resolved when core hadoop components (hdfs, mapreduce) are updated to use the new framework . Updates to external components that use the existing metrics framework will be tracked by different issues.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10341-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 21 02:07:43 2010
Return-Path: <common-issues-return-10341-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91433 invoked from network); 21 Aug 2010 02:07:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Aug 2010 02:07:43 -0000
Received: (qmail 17791 invoked by uid 500); 21 Aug 2010 02:07:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17751 invoked by uid 500); 21 Aug 2010 02:07:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17743 invoked by uid 99); 21 Aug 2010 02:07:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Aug 2010 02:07:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 21 Aug 2010 02:07:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7L27K54005183
	for <common-issues@hadoop.apache.org>; Sat, 21 Aug 2010 02:07:21 GMT
Message-ID: <6869328.483821282356440838.JavaMail.jira@thor>
Date: Fri, 20 Aug 2010 22:07:20 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6728) Overhaul metrics framework
In-Reply-To: <17506266.48501272420692262.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-6728:
----------------------------

    Attachment: hadoop-6728-y20.104.patch

The patch is a combined (commons/mapreduce/hdfs) patch against yahoo-hadoop-0.20.104 (published at http://github.com/yahoo/hadoop-common/). The code should be quite stable (based on ongoing scale testing.)

The trunk patch will include significant API improvement with annotations (not included in the above patch). The design notes have been updated to reflect the current code: [rev2|http://wiki.apache.org/hadoop/HADOOP-6728-MetricsV2?action=recall&rev=2]

Note, the patch completely removes the old metrics framework, which is appropriate for our use cases (and forced porting of everything). The main question for the community is: do we want to provide a mechanism to have the two framework coexists in 0.22, where the metrics framework can be switched at runtime or let's move everything to v2 in 0.22?

The answer to the question would alter the trunk patch significantly.

I prefer the latter as the former (dual framework switchable at runtime per process) would cause more confusion to users (and more throwaway work for me.). As an incentive for the latter, I offer to port ganglia 3.1+ plugin for the new framework, even though we don't use it :)


> Overhaul metrics framework
> --------------------------
>
>                 Key: HADOOP-6728
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6728
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.2
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6728-y20.104.patch, metrics1-flow.png, metrics2-builder-r2.png, metrics2-builder.png, metrics2-flow.png, metrics2-mutable-r2.png, metrics2-mutable.png, metrics2-record-r2.png, metrics2-record.png, metrics2-uml-r2.png, metrics2-uml.png
>
>
> Per discussions with Arun, Chris, Hong and Rajiv, et al, we concluded that the current metrics framework needs an overhaul to:
> * Allow multiple plugins for different monitoring systems simultaneously (see also: HADOOP-6508).
> * Refresh metrics plugin config without server restart.
> ** Including filtering of metrics per plugin.
> * Support metrics schema for plugins.
> The jira will be resolved when core hadoop components (hdfs, mapreduce) are updated to use the new framework . Updates to external components that use the existing metrics framework will be tracked by different issues.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10342-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 23 17:01:02 2010
Return-Path: <common-issues-return-10342-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18615 invoked from network); 23 Aug 2010 17:01:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Aug 2010 17:01:02 -0000
Received: (qmail 78524 invoked by uid 500); 23 Aug 2010 17:01:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78468 invoked by uid 500); 23 Aug 2010 17:01:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78455 invoked by uid 99); 23 Aug 2010 17:01:00 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 17:01:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 17:00:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7NH0GIr008929
	for <common-issues@hadoop.apache.org>; Mon, 23 Aug 2010 17:00:17 GMT
Message-ID: <15841216.511671282582816729.JavaMail.jira@thor>
Date: Mon, 23 Aug 2010 13:00:16 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6918) jjjjjjjjj
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

jjjjjjjjj
---------

                 Key: HADOOP-6918
                 URL: https://issues.apache.org/jira/browse/HADOOP-6918
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Luke Lu




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10343-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 23 17:24:56 2010
Return-Path: <common-issues-return-10343-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30929 invoked from network); 23 Aug 2010 17:24:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Aug 2010 17:24:56 -0000
Received: (qmail 10660 invoked by uid 500); 23 Aug 2010 17:24:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10596 invoked by uid 500); 23 Aug 2010 17:24:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10588 invoked by uid 99); 23 Aug 2010 17:24:55 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 17:24:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 17:24:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7NHOGWr009207
	for <common-issues@hadoop.apache.org>; Mon, 23 Aug 2010 17:24:17 GMT
Message-ID: <12754027.512141282584256841.JavaMail.jira@thor>
Date: Mon, 23 Aug 2010 13:24:16 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6453) Hadoop wrapper script shouldn't
 ignore an existing JAVA_LIBRARY_PATH
In-Reply-To: <909267607.1261086858225.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6453:
--------------------------------

    Hadoop Flags: [Incompatible change, Reviewed]  (was: [Incompatible change])
     Component/s: scripts

+1

> Hadoop wrapper script shouldn't ignore an existing JAVA_LIBRARY_PATH
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6453
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6453
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.20.2, 0.21.0, 0.22.0
>            Reporter: Chad Metcalf
>            Assignee: Chad Metcalf
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6453-0.20.patch, HADOOP-6453-0.20v2.patch, HADOOP-6453-trunkv2.patch, HADOOP-6453-trunkv3.patch, HADOOP-6453.trunk.patch
>
>
> Currently the hadoop wrapper script assumes its the only place that uses JAVA_LIBRARY_PATH and initializes it to a blank line.
> JAVA_LIBRARY_PATH=''
> This prevents anyone from setting this outside of the hadoop wrapper (say hadoop-config.sh) for their own native libraries.
> The fix is pretty simple. Don't initialize it to '' and append the native libs like normal. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10344-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 23 17:26:39 2010
Return-Path: <common-issues-return-10344-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31565 invoked from network); 23 Aug 2010 17:26:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Aug 2010 17:26:39 -0000
Received: (qmail 11067 invoked by uid 500); 23 Aug 2010 17:26:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11041 invoked by uid 500); 23 Aug 2010 17:26:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11033 invoked by uid 99); 23 Aug 2010 17:26:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 17:26:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 17:26:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7NHQIOB009237
	for <common-issues@hadoop.apache.org>; Mon, 23 Aug 2010 17:26:18 GMT
Message-ID: <22158887.512241282584378343.JavaMail.jira@thor>
Date: Mon, 23 Aug 2010 13:26:18 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6453) Hadoop wrapper script shouldn't
 ignore an existing JAVA_LIBRARY_PATH
In-Reply-To: <909267607.1261086858225.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12901493#action_12901493 ] 

Jakob Homan commented on HADOOP-6453:
-------------------------------------

Hudson has completely disappeared and test-patch was already run on a prior version of this patch, so I'm going to go ahead and commit this as-is.

> Hadoop wrapper script shouldn't ignore an existing JAVA_LIBRARY_PATH
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6453
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6453
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.20.2, 0.21.0, 0.22.0
>            Reporter: Chad Metcalf
>            Assignee: Chad Metcalf
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6453-0.20.patch, HADOOP-6453-0.20v2.patch, HADOOP-6453-trunkv2.patch, HADOOP-6453-trunkv3.patch, HADOOP-6453.trunk.patch
>
>
> Currently the hadoop wrapper script assumes its the only place that uses JAVA_LIBRARY_PATH and initializes it to a blank line.
> JAVA_LIBRARY_PATH=''
> This prevents anyone from setting this outside of the hadoop wrapper (say hadoop-config.sh) for their own native libraries.
> The fix is pretty simple. Don't initialize it to '' and append the native libs like normal. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10345-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 23 17:28:39 2010
Return-Path: <common-issues-return-10345-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32463 invoked from network); 23 Aug 2010 17:28:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Aug 2010 17:28:39 -0000
Received: (qmail 12958 invoked by uid 500); 23 Aug 2010 17:28:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12899 invoked by uid 500); 23 Aug 2010 17:28:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12664 invoked by uid 99); 23 Aug 2010 17:28:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 17:28:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 17:28:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7NHSG2B009253
	for <common-issues@hadoop.apache.org>; Mon, 23 Aug 2010 17:28:17 GMT
Message-ID: <8048692.512271282584496956.JavaMail.jira@thor>
Date: Mon, 23 Aug 2010 13:28:16 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6918) Make metrics naming consistent
In-Reply-To: <15841216.511671282582816729.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-6918:
----------------------------

              Summary: Make metrics naming consistent  (was: jjjjjjjjj)
             Assignee: Luke Lu
        Fix Version/s: 0.22.0
                 Tags: metrics
    Affects Version/s: 0.20.2
          Description: 
While working HADOOP-6728, I noticed that our metrics naming style is all over the place:

* Capitalized camel case: e.g., "FilesCreated" in namenode metrics and some rpc metrics
* uncapitalized camel case: e.g, "threadsBlocked" in jvm metrics and some rpc metrics
* lowercased underscored: e.g., "bytes_written" in datanode metrics and mapreduce metrics

Let's make them consistent. How about uncapitalized camel case? My main reason for the camel case: some backends have limits on the name length and underscore is wasteful.

Once we have a consistent naming style we can do:
@Metric("Number of INodes created") MutableCounterLong filesCreated;

instead of the more redundant:
@Metric({"FilesCreated", "Number of INodes created"}) MutableCounterLong filesCreated;
          Component/s: metrics

> Make metrics naming consistent
> ------------------------------
>
>                 Key: HADOOP-6918
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6918
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: metrics
>    Affects Versions: 0.20.2
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>
> While working HADOOP-6728, I noticed that our metrics naming style is all over the place:
> * Capitalized camel case: e.g., "FilesCreated" in namenode metrics and some rpc metrics
> * uncapitalized camel case: e.g, "threadsBlocked" in jvm metrics and some rpc metrics
> * lowercased underscored: e.g., "bytes_written" in datanode metrics and mapreduce metrics
> Let's make them consistent. How about uncapitalized camel case? My main reason for the camel case: some backends have limits on the name length and underscore is wasteful.
> Once we have a consistent naming style we can do:
> @Metric("Number of INodes created") MutableCounterLong filesCreated;
> instead of the more redundant:
> @Metric({"FilesCreated", "Number of INodes created"}) MutableCounterLong filesCreated;

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10346-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 23 17:32:57 2010
Return-Path: <common-issues-return-10346-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34755 invoked from network); 23 Aug 2010 17:32:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Aug 2010 17:32:57 -0000
Received: (qmail 21217 invoked by uid 500); 23 Aug 2010 17:32:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21168 invoked by uid 500); 23 Aug 2010 17:32:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21160 invoked by uid 99); 23 Aug 2010 17:32:56 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 17:32:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 17:32:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7NHWHed009287
	for <common-issues@hadoop.apache.org>; Mon, 23 Aug 2010 17:32:17 GMT
Message-ID: <15128183.512351282584737427.JavaMail.jira@thor>
Date: Mon, 23 Aug 2010 13:32:17 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6453) Hadoop wrapper script shouldn't
 ignore an existing JAVA_LIBRARY_PATH
In-Reply-To: <909267607.1261086858225.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6453:
--------------------------------

        Status: Resolved  (was: Patch Available)
    Resolution: Fixed

I've committed this.  Resolving as fixed.  Thanks, Chad!

> Hadoop wrapper script shouldn't ignore an existing JAVA_LIBRARY_PATH
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6453
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6453
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.20.2, 0.21.0, 0.22.0
>            Reporter: Chad Metcalf
>            Assignee: Chad Metcalf
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6453-0.20.patch, HADOOP-6453-0.20v2.patch, HADOOP-6453-trunkv2.patch, HADOOP-6453-trunkv3.patch, HADOOP-6453.trunk.patch
>
>
> Currently the hadoop wrapper script assumes its the only place that uses JAVA_LIBRARY_PATH and initializes it to a blank line.
> JAVA_LIBRARY_PATH=''
> This prevents anyone from setting this outside of the hadoop wrapper (say hadoop-config.sh) for their own native libraries.
> The fix is pretty simple. Don't initialize it to '' and append the native libs like normal. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10347-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 23 20:53:38 2010
Return-Path: <common-issues-return-10347-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35173 invoked from network); 23 Aug 2010 20:53:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Aug 2010 20:53:38 -0000
Received: (qmail 45238 invoked by uid 500); 23 Aug 2010 20:53:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45133 invoked by uid 500); 23 Aug 2010 20:53:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45122 invoked by uid 99); 23 Aug 2010 20:53:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 20:53:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 20:53:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7NKrG7C011771
	for <common-issues@hadoop.apache.org>; Mon, 23 Aug 2010 20:53:16 GMT
Message-ID: <3493931.517161282596796384.JavaMail.jira@thor>
Date: Mon, 23 Aug 2010 16:53:16 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6919) Metrics2: metrics framework
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Metrics2: metrics framework
---------------------------

                 Key: HADOOP-6919
                 URL: https://issues.apache.org/jira/browse/HADOOP-6919
             Project: Hadoop Common
          Issue Type: Sub-task
          Components: metrics
    Affects Versions: 0.20.2
            Reporter: Luke Lu
            Assignee: Luke Lu
             Fix For: 0.22.0


The jira tracks the new metrics framework only changes, i.e., it doesn't track the instrumentation changes (and compatibility issues) to existing code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10348-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 23 21:01:39 2010
Return-Path: <common-issues-return-10348-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38272 invoked from network); 23 Aug 2010 21:01:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Aug 2010 21:01:38 -0000
Received: (qmail 70683 invoked by uid 500); 23 Aug 2010 21:01:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70632 invoked by uid 500); 23 Aug 2010 21:01:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70624 invoked by uid 99); 23 Aug 2010 21:01:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 21:01:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 21:01:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7NL1GgN011847
	for <common-issues@hadoop.apache.org>; Mon, 23 Aug 2010 21:01:17 GMT
Message-ID: <14012363.517231282597276859.JavaMail.jira@thor>
Date: Mon, 23 Aug 2010 17:01:16 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6920) Metrics2: metrics instrumentation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Metrics2: metrics instrumentation
---------------------------------

                 Key: HADOOP-6920
                 URL: https://issues.apache.org/jira/browse/HADOOP-6920
             Project: Hadoop Common
          Issue Type: Sub-task
    Affects Versions: 0.20.2
            Reporter: Luke Lu
            Assignee: Luke Lu
             Fix For: 0.22.0


The jira tracks the metrics instrumentation for the new framework, i.e., porting jvm and rpc metrics to the new framework.

This issue (esp. the "incompatible change flag") depends on the outcome of the discussion (proposed in HADOOP-6728) on whether we should support backward compatibility (at some cost.)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10349-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 23 21:07:03 2010
Return-Path: <common-issues-return-10349-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44873 invoked from network); 23 Aug 2010 21:07:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Aug 2010 21:07:03 -0000
Received: (qmail 77730 invoked by uid 500); 23 Aug 2010 21:07:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77456 invoked by uid 500); 23 Aug 2010 21:07:02 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77448 invoked by uid 99); 23 Aug 2010 21:07:02 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 21:07:02 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 21:06:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7NL6NMM012010
	for <common-issues@hadoop.apache.org>; Mon, 23 Aug 2010 21:06:23 GMT
Message-ID: <22990252.517611282597583575.JavaMail.jira@thor>
Date: Mon, 23 Aug 2010 17:06:23 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6921) metrics2: metrics plugins
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

metrics2: metrics plugins
-------------------------

                 Key: HADOOP-6921
                 URL: https://issues.apache.org/jira/browse/HADOOP-6921
             Project: Hadoop Common
          Issue Type: Sub-task
    Affects Versions: 0.20.2
            Reporter: Luke Lu
            Assignee: Luke Lu
             Fix For: 0.22.0


This jira tracks the porting of builtin metrics sink plugins (file, ganglia) for the new metrics framework.

Whether or not ganglia 3.0/3.1 plugins will be ported depends on the outcome of the discussion (proposed in the parent issue: HADOOP-6728) on backward compatibility (at some cost/limitations of course.)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10350-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 23 22:24:07 2010
Return-Path: <common-issues-return-10350-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80494 invoked from network); 23 Aug 2010 22:24:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Aug 2010 22:24:07 -0000
Received: (qmail 73082 invoked by uid 500); 23 Aug 2010 22:24:07 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73032 invoked by uid 500); 23 Aug 2010 22:24:06 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73024 invoked by uid 99); 23 Aug 2010 22:24:06 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 22:24:06 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 22:23:49 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7NMNRCs013116
	for <common-issues@hadoop.apache.org>; Mon, 23 Aug 2010 22:23:27 GMT
Message-ID: <16006005.520151282602207321.JavaMail.jira@thor>
Date: Mon, 23 Aug 2010 18:23:27 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6728) Overhaul metrics framework
In-Reply-To: <17506266.48501272420692262.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-6728:
----------------------------

    Attachment: metrics2-uml-r2.png

> Overhaul metrics framework
> --------------------------
>
>                 Key: HADOOP-6728
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6728
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.2
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6728-y20.104.patch, metrics1-flow.png, metrics2-builder-r2.png, metrics2-builder.png, metrics2-flow.png, metrics2-mutable-r2.png, metrics2-mutable.png, metrics2-record-r2.png, metrics2-record.png, metrics2-uml-r2.png, metrics2-uml-r2.png, metrics2-uml.png
>
>
> Per discussions with Arun, Chris, Hong and Rajiv, et al, we concluded that the current metrics framework needs an overhaul to:
> * Allow multiple plugins for different monitoring systems simultaneously (see also: HADOOP-6508).
> * Refresh metrics plugin config without server restart.
> ** Including filtering of metrics per plugin.
> * Support metrics schema for plugins.
> The jira will be resolved when core hadoop components (hdfs, mapreduce) are updated to use the new framework . Updates to external components that use the existing metrics framework will be tracked by different issues.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10351-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 23 22:25:46 2010
Return-Path: <common-issues-return-10351-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81499 invoked from network); 23 Aug 2010 22:25:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Aug 2010 22:25:46 -0000
Received: (qmail 73625 invoked by uid 500); 23 Aug 2010 22:25:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73596 invoked by uid 500); 23 Aug 2010 22:25:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73588 invoked by uid 99); 23 Aug 2010 22:25:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 22:25:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 22:25:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7NMPOL1013183
	for <common-issues@hadoop.apache.org>; Mon, 23 Aug 2010 22:25:25 GMT
Message-ID: <15430844.520371282602324767.JavaMail.jira@thor>
Date: Mon, 23 Aug 2010 18:25:24 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6728) Overhaul metrics framework
In-Reply-To: <17506266.48501272420692262.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-6728:
----------------------------

    Attachment:     (was: metrics2-uml-r2.png)

> Overhaul metrics framework
> --------------------------
>
>                 Key: HADOOP-6728
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6728
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.2
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6728-y20.104.patch, metrics1-flow.png, metrics2-builder-r2.png, metrics2-builder.png, metrics2-flow.png, metrics2-mutable-r2.png, metrics2-mutable.png, metrics2-record-r2.png, metrics2-record.png, metrics2-uml-r2.png, metrics2-uml.png
>
>
> Per discussions with Arun, Chris, Hong and Rajiv, et al, we concluded that the current metrics framework needs an overhaul to:
> * Allow multiple plugins for different monitoring systems simultaneously (see also: HADOOP-6508).
> * Refresh metrics plugin config without server restart.
> ** Including filtering of metrics per plugin.
> * Support metrics schema for plugins.
> The jira will be resolved when core hadoop components (hdfs, mapreduce) are updated to use the new framework . Updates to external components that use the existing metrics framework will be tracked by different issues.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10352-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 23 22:44:02 2010
Return-Path: <common-issues-return-10352-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85749 invoked from network); 23 Aug 2010 22:44:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Aug 2010 22:44:02 -0000
Received: (qmail 96209 invoked by uid 500); 23 Aug 2010 22:44:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96157 invoked by uid 500); 23 Aug 2010 22:44:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96149 invoked by uid 99); 23 Aug 2010 22:44:01 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 22:44:01 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Aug 2010 22:43:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7NMhMDH013391
	for <common-issues@hadoop.apache.org>; Mon, 23 Aug 2010 22:43:22 GMT
Message-ID: <13739966.520941282603402569.JavaMail.jira@thor>
Date: Mon, 23 Aug 2010 18:43:22 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6728) Overhaul metrics framework
In-Reply-To: <17506266.48501272420692262.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12901639#action_12901639 ] 

Luke Lu commented on HADOOP-6728:
---------------------------------

[Design doc rev3|http://wiki.apache.org/hadoop/HADOOP-6728-MetricsV2?action=recall&rev=3] fixes some [inconsistencies in examples and wording|http://wiki.apache.org/hadoop/HADOOP-6728-MetricsV2?action=diff&rev2=3&rev1=2].

> Overhaul metrics framework
> --------------------------
>
>                 Key: HADOOP-6728
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6728
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.2
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6728-y20.104.patch, metrics1-flow.png, metrics2-builder-r2.png, metrics2-builder.png, metrics2-flow.png, metrics2-mutable-r2.png, metrics2-mutable.png, metrics2-record-r2.png, metrics2-record.png, metrics2-uml-r2.png, metrics2-uml.png
>
>
> Per discussions with Arun, Chris, Hong and Rajiv, et al, we concluded that the current metrics framework needs an overhaul to:
> * Allow multiple plugins for different monitoring systems simultaneously (see also: HADOOP-6508).
> * Refresh metrics plugin config without server restart.
> ** Including filtering of metrics per plugin.
> * Support metrics schema for plugins.
> The jira will be resolved when core hadoop components (hdfs, mapreduce) are updated to use the new framework . Updates to external components that use the existing metrics framework will be tracked by different issues.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10353-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 09:16:42 2010
Return-Path: <common-issues-return-10353-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10161 invoked from network); 24 Aug 2010 09:16:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 09:16:42 -0000
Received: (qmail 57277 invoked by uid 500); 24 Aug 2010 09:16:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57111 invoked by uid 500); 24 Aug 2010 09:16:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57094 invoked by uid 99); 24 Aug 2010 09:16:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 09:16:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 09:16:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7O9GHWx018128
	for <common-issues@hadoop.apache.org>; Tue, 24 Aug 2010 09:16:17 GMT
Message-ID: <23487377.527071282641377033.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 05:16:17 -0400 (EDT)
From: "Ravi Gummadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6922) COMMON part of MAPREDUCE-1664
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

COMMON part of MAPREDUCE-1664
-----------------------------

                 Key: HADOOP-6922
                 URL: https://issues.apache.org/jira/browse/HADOOP-6922
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Ravi Gummadi


MAPREDUCE-1664 changes the behavior of queue acls and job acls. This needs documentation changes to cluster_setup.xml and a small change in AccessControlList.java

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10354-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 09:16:44 2010
Return-Path: <common-issues-return-10354-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10267 invoked from network); 24 Aug 2010 09:16:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 09:16:44 -0000
Received: (qmail 57750 invoked by uid 500); 24 Aug 2010 09:16:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57200 invoked by uid 500); 24 Aug 2010 09:16:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57099 invoked by uid 99); 24 Aug 2010 09:16:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 09:16:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 09:16:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7O9GHdb018137
	for <common-issues@hadoop.apache.org>; Tue, 24 Aug 2010 09:16:17 GMT
Message-ID: <10529152.527101282641377679.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 05:16:17 -0400 (EDT)
From: "Ravi Gummadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6922) COMMON part of MAPREDUCE-1664
In-Reply-To: <23487377.527071282641377033.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi reassigned HADOOP-6922:
------------------------------------

    Assignee: Ravi Gummadi

> COMMON part of MAPREDUCE-1664
> -----------------------------
>
>                 Key: HADOOP-6922
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6922
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>
> MAPREDUCE-1664 changes the behavior of queue acls and job acls. This needs documentation changes to cluster_setup.xml and a small change in AccessControlList.java

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10355-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 09:20:42 2010
Return-Path: <common-issues-return-10355-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11557 invoked from network); 24 Aug 2010 09:20:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 09:20:41 -0000
Received: (qmail 67576 invoked by uid 500); 24 Aug 2010 09:20:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67413 invoked by uid 500); 24 Aug 2010 09:20:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67405 invoked by uid 99); 24 Aug 2010 09:20:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 09:20:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 09:20:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7O9KHlJ018236
	for <common-issues@hadoop.apache.org>; Tue, 24 Aug 2010 09:20:17 GMT
Message-ID: <30876326.527211282641617376.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 05:20:17 -0400 (EDT)
From: "Ravi Gummadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6922) COMMON part of MAPREDUCE-1664
In-Reply-To: <23487377.527071282641377033.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated HADOOP-6922:
---------------------------------

    Attachment: 6922.patch

Attaching patch for trunk.

This patch adds the documentation changes to cluster_setup.xml and modifies AccessControlList so that it is writable and creation of objects through Reflection utils is possible.

Please review and provide your comments.

> COMMON part of MAPREDUCE-1664
> -----------------------------
>
>                 Key: HADOOP-6922
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6922
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>         Attachments: 6922.patch
>
>
> MAPREDUCE-1664 changes the behavior of queue acls and job acls. This needs documentation changes to cluster_setup.xml and a small change in AccessControlList.java

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10356-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 09:58:00 2010
Return-Path: <common-issues-return-10356-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20806 invoked from network); 24 Aug 2010 09:58:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 09:58:00 -0000
Received: (qmail 23673 invoked by uid 500); 24 Aug 2010 09:58:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23521 invoked by uid 500); 24 Aug 2010 09:57:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23513 invoked by uid 99); 24 Aug 2010 09:57:56 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 09:57:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 09:57:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7O9vHTs018614
	for <common-issues@hadoop.apache.org>; Tue, 24 Aug 2010 09:57:17 GMT
Message-ID: <25490161.527871282643837689.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 05:57:17 -0400 (EDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6922) COMMON part of MAPREDUCE-1664
In-Reply-To: <23487377.527071282641377033.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12901805#action_12901805 ] 

Amareshwari Sriramadasu commented on HADOOP-6922:
-------------------------------------------------

The configuration name for the acls enabled should be mapreduce.cluster.acls.enabled following the nomenclature in branch 0.21 and trunk.

> COMMON part of MAPREDUCE-1664
> -----------------------------
>
>                 Key: HADOOP-6922
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6922
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>         Attachments: 6922.patch
>
>
> MAPREDUCE-1664 changes the behavior of queue acls and job acls. This needs documentation changes to cluster_setup.xml and a small change in AccessControlList.java

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10357-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 11:20:42 2010
Return-Path: <common-issues-return-10357-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58025 invoked from network); 24 Aug 2010 11:20:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 11:20:42 -0000
Received: (qmail 93366 invoked by uid 500); 24 Aug 2010 11:20:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93215 invoked by uid 500); 24 Aug 2010 11:20:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93207 invoked by uid 99); 24 Aug 2010 11:20:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 11:20:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 11:20:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7OBKHUs019275
	for <common-issues@hadoop.apache.org>; Tue, 24 Aug 2010 11:20:17 GMT
Message-ID: <9480639.528791282648817411.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 07:20:17 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6453) Hadoop wrapper script shouldn't
 ignore an existing JAVA_LIBRARY_PATH
In-Reply-To: <909267607.1261086858225.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12901822#action_12901822 ] 

Hudson commented on HADOOP-6453:
--------------------------------

Integrated in Hadoop-Common-trunk #432 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/432/])
    HADOOP-6453. Hadoop wrapper script shouldn't ignore an existing JAVA_LIBRARY_PATH. Contributed by Chad Metcalf.


> Hadoop wrapper script shouldn't ignore an existing JAVA_LIBRARY_PATH
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6453
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6453
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.20.2, 0.21.0, 0.22.0
>            Reporter: Chad Metcalf
>            Assignee: Chad Metcalf
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6453-0.20.patch, HADOOP-6453-0.20v2.patch, HADOOP-6453-trunkv2.patch, HADOOP-6453-trunkv3.patch, HADOOP-6453.trunk.patch
>
>
> Currently the hadoop wrapper script assumes its the only place that uses JAVA_LIBRARY_PATH and initializes it to a blank line.
> JAVA_LIBRARY_PATH=''
> This prevents anyone from setting this outside of the hadoop wrapper (say hadoop-config.sh) for their own native libraries.
> The fix is pretty simple. Don't initialize it to '' and append the native libs like normal. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10358-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 19:30:58 2010
Return-Path: <common-issues-return-10358-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85117 invoked from network); 24 Aug 2010 19:30:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 19:30:58 -0000
Received: (qmail 12717 invoked by uid 500); 24 Aug 2010 19:30:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12615 invoked by uid 500); 24 Aug 2010 19:30:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12607 invoked by uid 99); 24 Aug 2010 19:30:57 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 19:30:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 19:30:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7OJUIUu024528
	for <common-issues@hadoop.apache.org>; Tue, 24 Aug 2010 19:30:18 GMT
Message-ID: <920355.537841282678218330.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 15:30:18 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6453) Hadoop wrapper script shouldn't
 ignore an existing JAVA_LIBRARY_PATH
In-Reply-To: <909267607.1261086858225.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902032#action_12902032 ] 

Hudson commented on HADOOP-6453:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #364 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/364/])
    

> Hadoop wrapper script shouldn't ignore an existing JAVA_LIBRARY_PATH
> --------------------------------------------------------------------
>
>                 Key: HADOOP-6453
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6453
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.20.2, 0.21.0, 0.22.0
>            Reporter: Chad Metcalf
>            Assignee: Chad Metcalf
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6453-0.20.patch, HADOOP-6453-0.20v2.patch, HADOOP-6453-trunkv2.patch, HADOOP-6453-trunkv3.patch, HADOOP-6453.trunk.patch
>
>
> Currently the hadoop wrapper script assumes its the only place that uses JAVA_LIBRARY_PATH and initializes it to a blank line.
> JAVA_LIBRARY_PATH=''
> This prevents anyone from setting this outside of the hadoop wrapper (say hadoop-config.sh) for their own native libraries.
> The fix is pretty simple. Don't initialize it to '' and append the native libs like normal. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10359-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 19:50:57 2010
Return-Path: <common-issues-return-10359-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89737 invoked from network); 24 Aug 2010 19:50:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 19:50:57 -0000
Received: (qmail 33432 invoked by uid 500); 24 Aug 2010 19:50:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33243 invoked by uid 500); 24 Aug 2010 19:50:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33230 invoked by uid 99); 24 Aug 2010 19:50:56 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 19:50:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 19:50:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7OJoHPm024788
	for <common-issues@hadoop.apache.org>; Tue, 24 Aug 2010 19:50:17 GMT
Message-ID: <26891645.538181282679417132.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 15:50:17 -0400 (EDT)
From: "Stephen Watt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6923) Native Libraries do not load if a
 different platform signature is returned from
 org.apache.hadoop.util.PlatformName
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Native Libraries do not load if a different platform signature is returned from org.apache.hadoop.util.PlatformName
-------------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-6923
                 URL: https://issues.apache.org/jira/browse/HADOOP-6923
             Project: Hadoop Common
          Issue Type: Bug
          Components: native
    Affects Versions: 0.20.2, 0.20.1, 0.20.0
         Environment: SLES 10, IBM Java 6, Apache Hadoop 0.20.x
            Reporter: Stephen Watt
             Fix For: 0.20.3


The bin/hadoop script has an environment variable called JAVA_PLATFORM which is set to to the results returned by org.apache.hadoop.util.PlatformName . These results are sometimes unique to the JRE being used. Although the value returned for 64 Bit Sun/Oracle Java and 64 Bit IBM Java is the same, it is different for the corresponding 32 Bit JREs.
The issue is that the value returned is used in creating the path to the native libraries on disk, i.e ${HADOOP_COMMON_HOME}/lib/native/${JAVA_PLATFORM}

Since the path on disk is fixed with the Sun JRE value /lib/native/Linux-i386-32 it therefore fails when it attempts to load the native libraries with the value returned with 32 Bit IBM Java, /lib/native/Linux-x86-32

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10360-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 19:57:39 2010
Return-Path: <common-issues-return-10360-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90825 invoked from network); 24 Aug 2010 19:57:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 19:57:38 -0000
Received: (qmail 45692 invoked by uid 500); 24 Aug 2010 19:57:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45633 invoked by uid 500); 24 Aug 2010 19:57:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45625 invoked by uid 99); 24 Aug 2010 19:57:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 19:57:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 19:57:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7OJvHXQ024975
	for <common-issues@hadoop.apache.org>; Tue, 24 Aug 2010 19:57:17 GMT
Message-ID: <26608562.538611282679837029.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 15:57:17 -0400 (EDT)
From: "Stephen Watt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6923) Native Libraries do not load if a
 different platform signature is returned from
 org.apache.hadoop.util.PlatformName
In-Reply-To: <26891645.538181282679417132.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stephen Watt updated HADOOP-6923:
---------------------------------

    Attachment: HADOOP-6923.patch

Since the native libraries work regardless of JRE signature, the patches checks for the IBM 32 bit JRE signature and if found sets the JAVA_PLATFORM environment variable to the same signature of the Oracle/Sun JRE

> Native Libraries do not load if a different platform signature is returned from org.apache.hadoop.util.PlatformName
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6923
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6923
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.20.0, 0.20.1, 0.20.2
>         Environment: SLES 10, IBM Java 6, Apache Hadoop 0.20.x
>            Reporter: Stephen Watt
>             Fix For: 0.20.3
>
>         Attachments: HADOOP-6923.patch
>
>
> The bin/hadoop script has an environment variable called JAVA_PLATFORM which is set to to the results returned by org.apache.hadoop.util.PlatformName . These results are sometimes unique to the JRE being used. Although the value returned for 64 Bit Sun/Oracle Java and 64 Bit IBM Java is the same, it is different for the corresponding 32 Bit JREs.
> The issue is that the value returned is used in creating the path to the native libraries on disk, i.e ${HADOOP_COMMON_HOME}/lib/native/${JAVA_PLATFORM}
> Since the path on disk is fixed with the Sun JRE value /lib/native/Linux-i386-32 it therefore fails when it attempts to load the native libraries with the value returned with 32 Bit IBM Java, /lib/native/Linux-x86-32

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10361-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 19:58:05 2010
Return-Path: <common-issues-return-10361-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90959 invoked from network); 24 Aug 2010 19:58:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 19:58:05 -0000
Received: (qmail 46270 invoked by uid 500); 24 Aug 2010 19:58:05 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46245 invoked by uid 500); 24 Aug 2010 19:58:04 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46237 invoked by uid 99); 24 Aug 2010 19:58:04 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 19:58:04 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 19:57:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7OJvP3E024990
	for <common-issues@hadoop.apache.org>; Tue, 24 Aug 2010 19:57:25 GMT
Message-ID: <6659312.538661282679845915.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 15:57:25 -0400 (EDT)
From: "Stephen Watt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6923) Native Libraries do not load if a
 different platform signature is returned from
 org.apache.hadoop.util.PlatformName
In-Reply-To: <26891645.538181282679417132.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stephen Watt updated HADOOP-6923:
---------------------------------

    Status: Patch Available  (was: Open)

> Native Libraries do not load if a different platform signature is returned from org.apache.hadoop.util.PlatformName
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6923
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6923
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.20.2, 0.20.1, 0.20.0
>         Environment: SLES 10, IBM Java 6, Apache Hadoop 0.20.x
>            Reporter: Stephen Watt
>             Fix For: 0.20.3
>
>         Attachments: HADOOP-6923.patch
>
>
> The bin/hadoop script has an environment variable called JAVA_PLATFORM which is set to to the results returned by org.apache.hadoop.util.PlatformName . These results are sometimes unique to the JRE being used. Although the value returned for 64 Bit Sun/Oracle Java and 64 Bit IBM Java is the same, it is different for the corresponding 32 Bit JREs.
> The issue is that the value returned is used in creating the path to the native libraries on disk, i.e ${HADOOP_COMMON_HOME}/lib/native/${JAVA_PLATFORM}
> Since the path on disk is fixed with the Sun JRE value /lib/native/Linux-i386-32 it therefore fails when it attempts to load the native libraries with the value returned with 32 Bit IBM Java, /lib/native/Linux-x86-32

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10362-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 20:07:42 2010
Return-Path: <common-issues-return-10362-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 311 invoked from network); 24 Aug 2010 20:07:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 20:07:42 -0000
Received: (qmail 58684 invoked by uid 500); 24 Aug 2010 20:07:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58645 invoked by uid 500); 24 Aug 2010 20:07:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58637 invoked by uid 99); 24 Aug 2010 20:07:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 20:07:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 20:07:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7OK7KpF025158
	for <common-issues@hadoop.apache.org>; Tue, 24 Aug 2010 20:07:20 GMT
Message-ID: <11981519.538891282680440329.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 16:07:20 -0400 (EDT)
From: "David Rosenstrauch (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6298) BytesWritable#getBytes is a bad
 name that leads to programming mistakes
In-Reply-To: <2125297599.1254952892405.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902051#action_12902051 ] 

David Rosenstrauch commented on HADOOP-6298:
--------------------------------------------

We just got hit with this problem as well, and lost several hours on this due to getBytes giving me back an 18-byte array instead of the expected 12.

I realize this method is part of the API out in the field now, and so likely can't be changed.  But I think there should be *some* solution implemented for this.  i.e., some new method that's a variation on the name "getBytes" that provides the expected behavior.

It's just asking for trouble leaving this little land mine out there for developers to keep stumbling upon.

> BytesWritable#getBytes is a bad name that leads to programming mistakes
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-6298
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6298
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.1
>            Reporter: Nathan Marz
>
> Pretty much everyone at Rapleaf who has worked with Hadoop has misused BytesWritable#getBytes at some point, not expecting the byte array to be padded. I think we can completely alleviate these programming mistakes by deprecating and renaming this method (again) to be more descriptive. I propose "getPaddedBytes()" or "getPaddedValue()". It would also be helpful to have a helper method "getNonPaddedValue()" that makes a copy into a non-padded byte array. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10363-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 20:12:13 2010
Return-Path: <common-issues-return-10363-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2637 invoked from network); 24 Aug 2010 20:12:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 20:12:12 -0000
Received: (qmail 68136 invoked by uid 500); 24 Aug 2010 20:12:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68077 invoked by uid 500); 24 Aug 2010 20:12:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68069 invoked by uid 99); 24 Aug 2010 20:12:12 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 20:12:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 20:11:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7OKBW4H025181
	for <common-issues@hadoop.apache.org>; Tue, 24 Aug 2010 20:11:33 GMT
Message-ID: <18990854.538931282680692890.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 16:11:32 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6923) Native Libraries do not load if a
 different platform signature is returned from
 org.apache.hadoop.util.PlatformName
In-Reply-To: <26891645.538181282679417132.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902052#action_12902052 ] 

Greg Roelofs commented on HADOOP-6923:
--------------------------------------

Other than the reported version numbers, isn't this identical to HADOOP-6895?

> Native Libraries do not load if a different platform signature is returned from org.apache.hadoop.util.PlatformName
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6923
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6923
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.20.0, 0.20.1, 0.20.2
>         Environment: SLES 10, IBM Java 6, Apache Hadoop 0.20.x
>            Reporter: Stephen Watt
>             Fix For: 0.20.3
>
>         Attachments: HADOOP-6923.patch
>
>
> The bin/hadoop script has an environment variable called JAVA_PLATFORM which is set to to the results returned by org.apache.hadoop.util.PlatformName . These results are sometimes unique to the JRE being used. Although the value returned for 64 Bit Sun/Oracle Java and 64 Bit IBM Java is the same, it is different for the corresponding 32 Bit JREs.
> The issue is that the value returned is used in creating the path to the native libraries on disk, i.e ${HADOOP_COMMON_HOME}/lib/native/${JAVA_PLATFORM}
> Since the path on disk is fixed with the Sun JRE value /lib/native/Linux-i386-32 it therefore fails when it attempts to load the native libraries with the value returned with 32 Bit IBM Java, /lib/native/Linux-x86-32

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10364-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 20:20:58 2010
Return-Path: <common-issues-return-10364-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6396 invoked from network); 24 Aug 2010 20:20:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 20:20:57 -0000
Received: (qmail 78912 invoked by uid 500); 24 Aug 2010 20:20:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78354 invoked by uid 500); 24 Aug 2010 20:20:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78146 invoked by uid 99); 24 Aug 2010 20:20:56 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 20:20:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 20:20:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7OKKHag025223
	for <common-issues@hadoop.apache.org>; Tue, 24 Aug 2010 20:20:17 GMT
Message-ID: <28038853.538991282681217622.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 16:20:17 -0400 (EDT)
From: "Stephen Watt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6923) Native Libraries do not load if a
 different platform signature is returned from
 org.apache.hadoop.util.PlatformName
In-Reply-To: <26891645.538181282679417132.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902053#action_12902053 ] 

Stephen Watt commented on HADOOP-6923:
--------------------------------------

Identical Problem, *slightly* different resolutions. HADOOP-6895 is for Hadoop 0.21 where the code has been moved out of the bin/hadoop script into the bin/hadoop-config.sh file, so the 6895 patch modifies the hadoop-config.sh file, whereas this patch (for 0.20.3) modifies the bin/hadoop file.

> Native Libraries do not load if a different platform signature is returned from org.apache.hadoop.util.PlatformName
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6923
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6923
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.20.0, 0.20.1, 0.20.2
>         Environment: SLES 10, IBM Java 6, Apache Hadoop 0.20.x
>            Reporter: Stephen Watt
>             Fix For: 0.20.3
>
>         Attachments: HADOOP-6923.patch
>
>
> The bin/hadoop script has an environment variable called JAVA_PLATFORM which is set to to the results returned by org.apache.hadoop.util.PlatformName . These results are sometimes unique to the JRE being used. Although the value returned for 64 Bit Sun/Oracle Java and 64 Bit IBM Java is the same, it is different for the corresponding 32 Bit JREs.
> The issue is that the value returned is used in creating the path to the native libraries on disk, i.e ${HADOOP_COMMON_HOME}/lib/native/${JAVA_PLATFORM}
> Since the path on disk is fixed with the Sun JRE value /lib/native/Linux-i386-32 it therefore fails when it attempts to load the native libraries with the value returned with 32 Bit IBM Java, /lib/native/Linux-x86-32

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10365-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 20:48:38 2010
Return-Path: <common-issues-return-10365-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14221 invoked from network); 24 Aug 2010 20:48:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 20:48:38 -0000
Received: (qmail 5132 invoked by uid 500); 24 Aug 2010 20:48:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5107 invoked by uid 500); 24 Aug 2010 20:48:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5099 invoked by uid 99); 24 Aug 2010 20:48:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 20:48:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 20:48:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7OKmGBr025472
	for <common-issues@hadoop.apache.org>; Tue, 24 Aug 2010 20:48:16 GMT
Message-ID: <25715248.539481282682896678.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 16:48:16 -0400 (EDT)
From: "Stephen Watt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6924) Build fails with non-Sun JREs due to
 different pathing to the operating system architecture shared libraries
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Build fails with non-Sun JREs due to different pathing to the operating system architecture shared libraries
------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-6924
                 URL: https://issues.apache.org/jira/browse/HADOOP-6924
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.20.2, 0.20.1, 0.20.0
         Environment: SLES 10, IBM Java 6
            Reporter: Stephen Watt
             Fix For: 0.20.3


The src/native/configure script used to build the native libraries has an environment variable called JNI_LDFLAGS which is set as follows:

JNI_LDFLAGS="-L$JAVA_HOME/jre/lib/$OS_ARCH/server"

This pathing convention to the shared libraries for the operating system architecture is unique to Oracle/Sun Java and thus on other flavors of Java the path will not exist and will result in a build failure with the following exception:

     [exec] gcc -shared  ../src/org/apache/hadoop/io/compress/zlib/.libs/ZlibCompressor.o ../src/org/apache/hadoop/io/compress/zlib/.libs/ZlibDecompressor.o  -L/home/hadoop/Java-Versions/ibm-java-i386-60/jre/lib/x86/server -ljvm -ldl  -m32 -m32 -Wl,-soname -Wl,libhadoop.so.1 -o .libs/libhadoop.so.1.0.0
     [exec] /usr/lib/gcc/i586-suse-linux/4.1.2/../../../../i586-suse-linux/bin/ld: cannot find -ljvm
     [exec] collect2: ld returned 1 exit status

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10366-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 21:07:04 2010
Return-Path: <common-issues-return-10366-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22646 invoked from network); 24 Aug 2010 21:07:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 21:07:03 -0000
Received: (qmail 25719 invoked by uid 500); 24 Aug 2010 21:07:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25657 invoked by uid 500); 24 Aug 2010 21:07:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25649 invoked by uid 99); 24 Aug 2010 21:07:03 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 21:07:02 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 21:06:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7OL6OH1025861
	for <common-issues@hadoop.apache.org>; Tue, 24 Aug 2010 21:06:24 GMT
Message-ID: <13740654.540101282683984174.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 17:06:24 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6924) Build fails with non-Sun JREs due
 to different pathing to the operating system architecture shared libraries
In-Reply-To: <25715248.539481282682896678.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902070#action_12902070 ] 

Allen Wittenauer commented on HADOOP-6924:
------------------------------------------

I've got a local patch that basically asks the running JVM where the libjvm.so file is at.  I should really clean it up and post it.

> Build fails with non-Sun JREs due to different pathing to the operating system architecture shared libraries
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6924
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6924
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.0, 0.20.1, 0.20.2
>         Environment: SLES 10, IBM Java 6
>            Reporter: Stephen Watt
>             Fix For: 0.20.3
>
>
> The src/native/configure script used to build the native libraries has an environment variable called JNI_LDFLAGS which is set as follows:
> JNI_LDFLAGS="-L$JAVA_HOME/jre/lib/$OS_ARCH/server"
> This pathing convention to the shared libraries for the operating system architecture is unique to Oracle/Sun Java and thus on other flavors of Java the path will not exist and will result in a build failure with the following exception:
>      [exec] gcc -shared  ../src/org/apache/hadoop/io/compress/zlib/.libs/ZlibCompressor.o ../src/org/apache/hadoop/io/compress/zlib/.libs/ZlibDecompressor.o  -L/home/hadoop/Java-Versions/ibm-java-i386-60/jre/lib/x86/server -ljvm -ldl  -m32 -m32 -Wl,-soname -Wl,libhadoop.so.1 -o .libs/libhadoop.so.1.0.0
>      [exec] /usr/lib/gcc/i586-suse-linux/4.1.2/../../../../i586-suse-linux/bin/ld: cannot find -ljvm
>      [exec] collect2: ld returned 1 exit status

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10367-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 21:19:39 2010
Return-Path: <common-issues-return-10367-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38338 invoked from network); 24 Aug 2010 21:19:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 21:19:38 -0000
Received: (qmail 41688 invoked by uid 500); 24 Aug 2010 21:19:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41659 invoked by uid 500); 24 Aug 2010 21:19:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41651 invoked by uid 99); 24 Aug 2010 21:19:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 21:19:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 21:19:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7OLJH5V026052
	for <common-issues@hadoop.apache.org>; Tue, 24 Aug 2010 21:19:17 GMT
Message-ID: <13571684.540511282684757043.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 17:19:17 -0400 (EDT)
From: "Stephen Watt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6924) Build fails with non-Sun JREs due
 to different pathing to the operating system architecture shared libraries
In-Reply-To: <25715248.539481282682896678.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902078#action_12902078 ] 

Stephen Watt commented on HADOOP-6924:
--------------------------------------

That sounds like a good approach Allen. Something that greps forward from the $JAVA_HOME/jre/lib directory for the libjvm.so to determine the path would be quite elegant and much simpler than trying to divine the OS Architecture signature and build the right path.

> Build fails with non-Sun JREs due to different pathing to the operating system architecture shared libraries
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6924
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6924
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.0, 0.20.1, 0.20.2
>         Environment: SLES 10, IBM Java 6
>            Reporter: Stephen Watt
>             Fix For: 0.20.3
>
>
> The src/native/configure script used to build the native libraries has an environment variable called JNI_LDFLAGS which is set as follows:
> JNI_LDFLAGS="-L$JAVA_HOME/jre/lib/$OS_ARCH/server"
> This pathing convention to the shared libraries for the operating system architecture is unique to Oracle/Sun Java and thus on other flavors of Java the path will not exist and will result in a build failure with the following exception:
>      [exec] gcc -shared  ../src/org/apache/hadoop/io/compress/zlib/.libs/ZlibCompressor.o ../src/org/apache/hadoop/io/compress/zlib/.libs/ZlibDecompressor.o  -L/home/hadoop/Java-Versions/ibm-java-i386-60/jre/lib/x86/server -ljvm -ldl  -m32 -m32 -Wl,-soname -Wl,libhadoop.so.1 -o .libs/libhadoop.so.1.0.0
>      [exec] /usr/lib/gcc/i586-suse-linux/4.1.2/../../../../i586-suse-linux/bin/ld: cannot find -ljvm
>      [exec] collect2: ld returned 1 exit status

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10368-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 21:49:48 2010
Return-Path: <common-issues-return-10368-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48331 invoked from network); 24 Aug 2010 21:49:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 21:49:48 -0000
Received: (qmail 82137 invoked by uid 500); 24 Aug 2010 21:49:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82073 invoked by uid 500); 24 Aug 2010 21:49:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82064 invoked by uid 99); 24 Aug 2010 21:49:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 21:49:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 21:49:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7OLnReB026586
	for <common-issues@hadoop.apache.org>; Tue, 24 Aug 2010 21:49:27 GMT
Message-ID: <17438612.541721282686567090.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 17:49:27 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6923) Native Libraries do not load if a
 different platform signature is returned from
 org.apache.hadoop.util.PlatformName
In-Reply-To: <26891645.538181282679417132.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902089#action_12902089 ] 

Greg Roelofs commented on HADOOP-6923:
--------------------------------------

Gotcha, thanks.  Might be good to link the two issues, though.

> Native Libraries do not load if a different platform signature is returned from org.apache.hadoop.util.PlatformName
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6923
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6923
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 0.20.0, 0.20.1, 0.20.2
>         Environment: SLES 10, IBM Java 6, Apache Hadoop 0.20.x
>            Reporter: Stephen Watt
>             Fix For: 0.20.3
>
>         Attachments: HADOOP-6923.patch
>
>
> The bin/hadoop script has an environment variable called JAVA_PLATFORM which is set to to the results returned by org.apache.hadoop.util.PlatformName . These results are sometimes unique to the JRE being used. Although the value returned for 64 Bit Sun/Oracle Java and 64 Bit IBM Java is the same, it is different for the corresponding 32 Bit JREs.
> The issue is that the value returned is used in creating the path to the native libraries on disk, i.e ${HADOOP_COMMON_HOME}/lib/native/${JAVA_PLATFORM}
> Since the path on disk is fixed with the Sun JRE value /lib/native/Linux-i386-32 it therefore fails when it attempts to load the native libraries with the value returned with 32 Bit IBM Java, /lib/native/Linux-x86-32

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10369-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 22:11:38 2010
Return-Path: <common-issues-return-10369-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60371 invoked from network); 24 Aug 2010 22:11:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 22:11:38 -0000
Received: (qmail 6573 invoked by uid 500); 24 Aug 2010 22:11:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6513 invoked by uid 500); 24 Aug 2010 22:11:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6505 invoked by uid 99); 24 Aug 2010 22:11:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 22:11:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 22:11:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7OMBGHZ027018
	for <common-issues@hadoop.apache.org>; Tue, 24 Aug 2010 22:11:16 GMT
Message-ID: <8553385.542391282687876583.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 18:11:16 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6925) BZip2Codec incorrectly implements
 read()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

BZip2Codec incorrectly implements read()
----------------------------------------

                 Key: HADOOP-6925
                 URL: https://issues.apache.org/jira/browse/HADOOP-6925
             Project: Hadoop Common
          Issue Type: Bug
          Components: io
    Affects Versions: 0.21.0, 0.22.0
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
            Priority: Critical
         Attachments: hadoop-6925.txt

HADOOP-4012 added an implementation of read() in BZip2InputStream that doesn't work correctly when reading bytes > 0x80. This causes EOFExceptions when working with BZip2 compressed data inside of sequence files in some datasets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10370-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 22:11:56 2010
Return-Path: <common-issues-return-10370-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60438 invoked from network); 24 Aug 2010 22:11:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 22:11:56 -0000
Received: (qmail 6830 invoked by uid 500); 24 Aug 2010 22:11:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6730 invoked by uid 500); 24 Aug 2010 22:11:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6721 invoked by uid 99); 24 Aug 2010 22:11:55 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 22:11:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 22:11:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7OMBHYI027027
	for <common-issues@hadoop.apache.org>; Tue, 24 Aug 2010 22:11:17 GMT
Message-ID: <25830067.542421282687877239.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 18:11:17 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6925) BZip2Codec incorrectly implements
 read()
In-Reply-To: <8553385.542391282687876583.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6925:
--------------------------------

    Attachment: hadoop-6925.txt

Patch fixes the read() implementation to correctly mask with 0xff before upcasting to int. Also augments the unit test to check the single-byte read() function - the new test fails before this patch.

> BZip2Codec incorrectly implements read()
> ----------------------------------------
>
>                 Key: HADOOP-6925
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6925
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>         Attachments: hadoop-6925.txt
>
>
> HADOOP-4012 added an implementation of read() in BZip2InputStream that doesn't work correctly when reading bytes > 0x80. This causes EOFExceptions when working with BZip2 compressed data inside of sequence files in some datasets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10371-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 22:13:55 2010
Return-Path: <common-issues-return-10371-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61550 invoked from network); 24 Aug 2010 22:13:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 22:13:55 -0000
Received: (qmail 9226 invoked by uid 500); 24 Aug 2010 22:13:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9127 invoked by uid 500); 24 Aug 2010 22:13:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9118 invoked by uid 99); 24 Aug 2010 22:13:55 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 22:13:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 22:13:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7OMDGgp027039
	for <common-issues@hadoop.apache.org>; Tue, 24 Aug 2010 22:13:16 GMT
Message-ID: <10770132.542441282687996205.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 18:13:16 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6925) BZip2Codec incorrectly implements
 read()
In-Reply-To: <8553385.542391282687876583.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6925:
--------------------------------

    Status: Patch Available  (was: Open)

> BZip2Codec incorrectly implements read()
> ----------------------------------------
>
>                 Key: HADOOP-6925
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6925
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>         Attachments: hadoop-6925.txt
>
>
> HADOOP-4012 added an implementation of read() in BZip2InputStream that doesn't work correctly when reading bytes > 0x80. This causes EOFExceptions when working with BZip2 compressed data inside of sequence files in some datasets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10372-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 22:30:39 2010
Return-Path: <common-issues-return-10372-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68070 invoked from network); 24 Aug 2010 22:30:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 22:30:38 -0000
Received: (qmail 21485 invoked by uid 500); 24 Aug 2010 22:30:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21405 invoked by uid 500); 24 Aug 2010 22:30:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21397 invoked by uid 99); 24 Aug 2010 22:30:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 22:30:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 22:30:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7OMUHnv027269
	for <common-issues@hadoop.apache.org>; Tue, 24 Aug 2010 22:30:17 GMT
Message-ID: <3163232.542961282689017102.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 18:30:17 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6925) BZip2Codec incorrectly implements
 read()
In-Reply-To: <8553385.542391282687876583.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902112#action_12902112 ] 

Greg Roelofs commented on HADOOP-6925:
--------------------------------------

This may fix HADOOP-6852, as well.  (The unit test in question is available as a standalone patch in MAPREDUCE-1927, which is awaiting review.)  If so, I can update the unit test to uncomment the relevant bzip2 test.

I'll make a note to check this soon...

> BZip2Codec incorrectly implements read()
> ----------------------------------------
>
>                 Key: HADOOP-6925
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6925
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>         Attachments: hadoop-6925.txt
>
>
> HADOOP-4012 added an implementation of read() in BZip2InputStream that doesn't work correctly when reading bytes > 0x80. This causes EOFExceptions when working with BZip2 compressed data inside of sequence files in some datasets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10373-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 22:30:58 2010
Return-Path: <common-issues-return-10373-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68273 invoked from network); 24 Aug 2010 22:30:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 22:30:58 -0000
Received: (qmail 22608 invoked by uid 500); 24 Aug 2010 22:30:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22578 invoked by uid 500); 24 Aug 2010 22:30:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22570 invoked by uid 99); 24 Aug 2010 22:30:57 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 22:30:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 22:30:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7OMUIg2027281
	for <common-issues@hadoop.apache.org>; Tue, 24 Aug 2010 22:30:18 GMT
Message-ID: <31714619.543001282689018655.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 18:30:18 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6924) Build fails with non-Sun JREs due
 to different pathing to the operating system architecture shared libraries
In-Reply-To: <25715248.539481282682896678.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902113#action_12902113 ] 

Allen Wittenauer commented on HADOOP-6924:
------------------------------------------

No, no, there's an easier way.

The system property sun.boot.library.path (at least on Sun JRE's) will tell you the location of libjvm. So I have an autoconf that builds a small java app that prints out this property.

> Build fails with non-Sun JREs due to different pathing to the operating system architecture shared libraries
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6924
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6924
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.0, 0.20.1, 0.20.2
>         Environment: SLES 10, IBM Java 6
>            Reporter: Stephen Watt
>             Fix For: 0.20.3
>
>
> The src/native/configure script used to build the native libraries has an environment variable called JNI_LDFLAGS which is set as follows:
> JNI_LDFLAGS="-L$JAVA_HOME/jre/lib/$OS_ARCH/server"
> This pathing convention to the shared libraries for the operating system architecture is unique to Oracle/Sun Java and thus on other flavors of Java the path will not exist and will result in a build failure with the following exception:
>      [exec] gcc -shared  ../src/org/apache/hadoop/io/compress/zlib/.libs/ZlibCompressor.o ../src/org/apache/hadoop/io/compress/zlib/.libs/ZlibDecompressor.o  -L/home/hadoop/Java-Versions/ibm-java-i386-60/jre/lib/x86/server -ljvm -ldl  -m32 -m32 -Wl,-soname -Wl,libhadoop.so.1 -o .libs/libhadoop.so.1.0.0
>      [exec] /usr/lib/gcc/i586-suse-linux/4.1.2/../../../../i586-suse-linux/bin/ld: cannot find -ljvm
>      [exec] collect2: ld returned 1 exit status

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10374-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 22:32:40 2010
Return-Path: <common-issues-return-10374-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68910 invoked from network); 24 Aug 2010 22:32:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 22:32:39 -0000
Received: (qmail 25669 invoked by uid 500); 24 Aug 2010 22:32:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25626 invoked by uid 500); 24 Aug 2010 22:32:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25618 invoked by uid 99); 24 Aug 2010 22:32:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 22:32:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 22:32:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7OMWIYT027303
	for <common-issues@hadoop.apache.org>; Tue, 24 Aug 2010 22:32:18 GMT
Message-ID: <7328286.543051282689138035.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 18:32:18 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6925) BZip2Codec incorrectly implements
 read()
In-Reply-To: <8553385.542391282687876583.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6925:
--------------------------------

    Hadoop Flags: [Reviewed]

+1

Looks good. Nice find.

> BZip2Codec incorrectly implements read()
> ----------------------------------------
>
>                 Key: HADOOP-6925
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6925
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>         Attachments: hadoop-6925.txt
>
>
> HADOOP-4012 added an implementation of read() in BZip2InputStream that doesn't work correctly when reading bytes > 0x80. This causes EOFExceptions when working with BZip2 compressed data inside of sequence files in some datasets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10375-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 22:35:01 2010
Return-Path: <common-issues-return-10375-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69673 invoked from network); 24 Aug 2010 22:35:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 22:35:01 -0000
Received: (qmail 28569 invoked by uid 500); 24 Aug 2010 22:35:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28532 invoked by uid 500); 24 Aug 2010 22:35:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28524 invoked by uid 99); 24 Aug 2010 22:35:00 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 22:35:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 22:34:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7OMYLRM027352
	for <common-issues@hadoop.apache.org>; Tue, 24 Aug 2010 22:34:21 GMT
Message-ID: <16117339.543181282689261661.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 18:34:21 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6925) BZip2Codec incorrectly implements
 read()
In-Reply-To: <8553385.542391282687876583.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902118#action_12902118 ] 

Chris Douglas commented on HADOOP-6925:
---------------------------------------

+1

*sigh* This was introduced between v9 of the patch and v11. Good catch

> BZip2Codec incorrectly implements read()
> ----------------------------------------
>
>                 Key: HADOOP-6925
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6925
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>         Attachments: hadoop-6925.txt
>
>
> HADOOP-4012 added an implementation of read() in BZip2InputStream that doesn't work correctly when reading bytes > 0x80. This causes EOFExceptions when working with BZip2 compressed data inside of sequence files in some datasets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10376-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 22:40:44 2010
Return-Path: <common-issues-return-10376-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71906 invoked from network); 24 Aug 2010 22:40:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 22:40:44 -0000
Received: (qmail 41844 invoked by uid 500); 24 Aug 2010 22:40:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41797 invoked by uid 500); 24 Aug 2010 22:40:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41789 invoked by uid 99); 24 Aug 2010 22:40:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 22:40:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 22:40:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7OMeNSI027467
	for <common-issues@hadoop.apache.org>; Tue, 24 Aug 2010 22:40:23 GMT
Message-ID: <6335491.543431282689623460.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 18:40:23 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6926) SocketInputStream incorrectly
 implements read()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

SocketInputStream incorrectly implements read()
-----------------------------------------------

                 Key: HADOOP-6926
                 URL: https://issues.apache.org/jira/browse/HADOOP-6926
             Project: Hadoop Common
          Issue Type: Bug
          Components: io
    Affects Versions: 0.21.0, 0.20.2
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
            Priority: Minor
             Fix For: 0.22.0


SocketInputStream's read() implementation doesn't upcast to int correctly, so it can't read bytes > 0x80. This is the same bug as HADOOP-6925, but in a different spot.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10377-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 22:44:43 2010
Return-Path: <common-issues-return-10377-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72845 invoked from network); 24 Aug 2010 22:44:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 22:44:43 -0000
Received: (qmail 44603 invoked by uid 500); 24 Aug 2010 22:44:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44581 invoked by uid 500); 24 Aug 2010 22:44:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44565 invoked by uid 99); 24 Aug 2010 22:44:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 22:44:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 22:44:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7OMiMEF027498
	for <common-issues@hadoop.apache.org>; Tue, 24 Aug 2010 22:44:22 GMT
Message-ID: <24046870.543501282689862124.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 18:44:22 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6926) SocketInputStream incorrectly
 implements read()
In-Reply-To: <6335491.543431282689623460.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6926:
--------------------------------

               Status: Patch Available  (was: Open)
    Affects Version/s: 0.22.0
        Fix Version/s:     (was: 0.22.0)

> SocketInputStream incorrectly implements read()
> -----------------------------------------------
>
>                 Key: HADOOP-6926
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6926
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.21.0, 0.20.2, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-6926.txt
>
>
> SocketInputStream's read() implementation doesn't upcast to int correctly, so it can't read bytes > 0x80. This is the same bug as HADOOP-6925, but in a different spot.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10378-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 22:44:44 2010
Return-Path: <common-issues-return-10378-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72898 invoked from network); 24 Aug 2010 22:44:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 22:44:44 -0000
Received: (qmail 44799 invoked by uid 500); 24 Aug 2010 22:44:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44583 invoked by uid 500); 24 Aug 2010 22:44:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44566 invoked by uid 99); 24 Aug 2010 22:44:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 22:44:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 22:44:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7OMiLwJ027492
	for <common-issues@hadoop.apache.org>; Tue, 24 Aug 2010 22:44:21 GMT
Message-ID: <31879940.543481282689861687.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 18:44:21 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6926) SocketInputStream incorrectly
 implements read()
In-Reply-To: <6335491.543431282689623460.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6926:
--------------------------------

    Attachment: hadoop-6926.txt

> SocketInputStream incorrectly implements read()
> -----------------------------------------------
>
>                 Key: HADOOP-6926
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6926
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.2, 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-6926.txt
>
>
> SocketInputStream's read() implementation doesn't upcast to int correctly, so it can't read bytes > 0x80. This is the same bug as HADOOP-6925, but in a different spot.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10379-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 22:49:42 2010
Return-Path: <common-issues-return-10379-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74532 invoked from network); 24 Aug 2010 22:49:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 22:49:42 -0000
Received: (qmail 52948 invoked by uid 500); 24 Aug 2010 22:49:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52582 invoked by uid 500); 24 Aug 2010 22:49:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52574 invoked by uid 99); 24 Aug 2010 22:49:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 22:49:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 22:49:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7OMnKIV027557
	for <common-issues@hadoop.apache.org>; Tue, 24 Aug 2010 22:49:20 GMT
Message-ID: <10876119.543651282690160695.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 18:49:20 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6925) BZip2Codec incorrectly implements
 read()
In-Reply-To: <8553385.542391282687876583.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6925:
--------------------------------

           Status: Resolved  (was: Patch Available)
    Fix Version/s: 0.21.1
                   0.22.0
       Resolution: Fixed

Committed to trunk and branch-0.20.  Thanks Todd.

> BZip2Codec incorrectly implements read()
> ----------------------------------------
>
>                 Key: HADOOP-6925
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6925
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.21.1, 0.22.0
>
>         Attachments: hadoop-6925.txt
>
>
> HADOOP-4012 added an implementation of read() in BZip2InputStream that doesn't work correctly when reading bytes > 0x80. This causes EOFExceptions when working with BZip2 compressed data inside of sequence files in some datasets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10380-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 22:51:46 2010
Return-Path: <common-issues-return-10380-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74832 invoked from network); 24 Aug 2010 22:51:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 22:51:46 -0000
Received: (qmail 55145 invoked by uid 500); 24 Aug 2010 22:51:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55011 invoked by uid 500); 24 Aug 2010 22:51:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55000 invoked by uid 99); 24 Aug 2010 22:51:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 22:51:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 22:51:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7OMpOY2027597
	for <common-issues@hadoop.apache.org>; Tue, 24 Aug 2010 22:51:24 GMT
Message-ID: <31354702.543781282690284539.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 18:51:24 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6925) BZip2Codec incorrectly implements
 read()
In-Reply-To: <8553385.542391282687876583.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902127#action_12902127 ] 

Eli Collins commented on HADOOP-6925:
-------------------------------------

Forgot to mention that I verified TestCodec passed on trunk and the merge to branch-0.21. Previous comment has a typo, meant *branch-0.21* not branch-0.20.

> BZip2Codec incorrectly implements read()
> ----------------------------------------
>
>                 Key: HADOOP-6925
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6925
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.21.1, 0.22.0
>
>         Attachments: hadoop-6925.txt
>
>
> HADOOP-4012 added an implementation of read() in BZip2InputStream that doesn't work correctly when reading bytes > 0x80. This causes EOFExceptions when working with BZip2 compressed data inside of sequence files in some datasets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10381-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 24 23:33:01 2010
Return-Path: <common-issues-return-10381-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92652 invoked from network); 24 Aug 2010 23:33:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Aug 2010 23:33:01 -0000
Received: (qmail 95375 invoked by uid 500); 24 Aug 2010 23:33:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95323 invoked by uid 500); 24 Aug 2010 23:33:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95315 invoked by uid 99); 24 Aug 2010 23:33:00 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 23:33:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Aug 2010 23:32:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7ONWLCd028195
	for <common-issues@hadoop.apache.org>; Tue, 24 Aug 2010 23:32:21 GMT
Message-ID: <29485999.544981282692741125.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 19:32:21 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6822) Provide information as to whether
 or not security is enabled on web interface
In-Reply-To: <18772311.54571276297093402.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902156#action_12902156 ] 

Boris Shkolnik commented on HADOOP-6822:
----------------------------------------

I guess we should add it at least for NN.

something like this:
Started: 	 Mon Aug 23 16:06:03 PDT 2010
Version: 	0.20.200.0-SNAPSHOT, r60bf953db32b5e803c400ff11095bb07caf0ccb7
Compiled: 	Mon Aug 23 16:04:53 PDT 2010 by borya
Upgrades: 	There are no upgrades in progress.
SecureMode: 	Off 

> Provide information as to whether or not security is enabled on web interface
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-6822
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6822
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Jakob Homan
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10382-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 25 01:09:00 2010
Return-Path: <common-issues-return-10382-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32238 invoked from network); 25 Aug 2010 01:09:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Aug 2010 01:09:00 -0000
Received: (qmail 88316 invoked by uid 500); 25 Aug 2010 01:09:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88231 invoked by uid 500); 25 Aug 2010 01:08:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88223 invoked by uid 99); 25 Aug 2010 01:08:59 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 01:08:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 01:08:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7P18K00029695
	for <common-issues@hadoop.apache.org>; Wed, 25 Aug 2010 01:08:20 GMT
Message-ID: <1148293.548921282698500196.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 21:08:20 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6925) BZip2Codec incorrectly implements
 read()
In-Reply-To: <8553385.542391282687876583.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902247#action_12902247 ] 

Hudson commented on HADOOP-6925:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #365 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/365/])
    HADOOP-6925. BZip2Codec incorrectly implements read(). Contributed by Todd Lipcon.


> BZip2Codec incorrectly implements read()
> ----------------------------------------
>
>                 Key: HADOOP-6925
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6925
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.21.1, 0.22.0
>
>         Attachments: hadoop-6925.txt
>
>
> HADOOP-4012 added an implementation of read() in BZip2InputStream that doesn't work correctly when reading bytes > 0x80. This causes EOFExceptions when working with BZip2 compressed data inside of sequence files in some datasets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10383-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 25 01:42:57 2010
Return-Path: <common-issues-return-10383-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40850 invoked from network); 25 Aug 2010 01:42:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Aug 2010 01:42:57 -0000
Received: (qmail 11227 invoked by uid 500); 25 Aug 2010 01:42:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11154 invoked by uid 500); 25 Aug 2010 01:42:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11146 invoked by uid 99); 25 Aug 2010 01:42:56 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 01:42:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 01:42:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7P1gHuJ000050
	for <common-issues@hadoop.apache.org>; Wed, 25 Aug 2010 01:42:17 GMT
Message-ID: <17147644.549651282700537251.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 21:42:17 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6925) BZip2Codec incorrectly implements
 read()
In-Reply-To: <8553385.542391282687876583.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902267#action_12902267 ] 

Greg Roelofs commented on HADOOP-6925:
--------------------------------------

bq. This may fix HADOOP-6852, as well.

Nope, concatenation is still broken.  Ah, well.

> BZip2Codec incorrectly implements read()
> ----------------------------------------
>
>                 Key: HADOOP-6925
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6925
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.21.1, 0.22.0
>
>         Attachments: hadoop-6925.txt
>
>
> HADOOP-4012 added an implementation of read() in BZip2InputStream that doesn't work correctly when reading bytes > 0x80. This causes EOFExceptions when working with BZip2 compressed data inside of sequence files in some datasets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10384-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 25 01:48:39 2010
Return-Path: <common-issues-return-10384-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41618 invoked from network); 25 Aug 2010 01:48:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Aug 2010 01:48:38 -0000
Received: (qmail 14863 invoked by uid 500); 25 Aug 2010 01:48:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14768 invoked by uid 500); 25 Aug 2010 01:48:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14757 invoked by uid 99); 25 Aug 2010 01:48:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 01:48:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 01:48:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7P1mGUD000087
	for <common-issues@hadoop.apache.org>; Wed, 25 Aug 2010 01:48:17 GMT
Message-ID: <16354924.549711282700896873.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 21:48:16 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6852) apparent bug in concatenated-bzip2
 support (decoding)
In-Reply-To: <503604.248281278541551065.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902268#action_12902268 ] 

Greg Roelofs commented on HADOOP-6852:
--------------------------------------

Alas, it doesn't. :-(

> apparent bug in concatenated-bzip2 support (decoding)
> -----------------------------------------------------
>
>                 Key: HADOOP-6852
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6852
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.22.0
>         Environment: Linux x86_64 running 32-bit Hadoop, JDK 1.6.0_15
>            Reporter: Greg Roelofs
>
> The following simplified code (manually picked out of testMoreBzip2() in https://issues.apache.org/jira/secure/attachment/12448272/HADOOP-6835.v4.trunk-hadoop-mapreduce.patch) triggers a "java.io.IOException: bad block header" in org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.initBlock( CBZip2InputStream.java:527):
> {noformat}
>     JobConf jobConf = new JobConf(defaultConf);
>     CompressionCodec bzip2 = new BZip2Codec();
>     ReflectionUtils.setConf(bzip2, jobConf);
>     localFs.delete(workDir, true);
>     // copy multiple-member test file to HDFS
>     String fn2 = "testCompressThenConcat.txt" + bzip2.getDefaultExtension();
>     Path fnLocal2 = new Path(System.getProperty("test.concat.data","/tmp"),fn2);
>     Path fnHDFS2  = new Path(workDir, fn2);
>     localFs.copyFromLocalFile(fnLocal2, fnHDFS2);
>     FileInputFormat.setInputPaths(jobConf, workDir);
>     final FileInputStream in2 = new FileInputStream(fnLocal2.toString());
>     CompressionInputStream cin2 = bzip2.createInputStream(in2);
>     LineReader in = new LineReader(cin2);
>     Text out = new Text();
>     int numBytes, totalBytes=0, lineNum=0;
>     while ((numBytes = in.readLine(out)) > 0) {
>       ++lineNum;
>       totalBytes += numBytes;
>     }
>     in.close();
> {noformat}
> The specified file is also included in the H-6835 patch linked above, and some additional debug output is included in the commented-out test loop above.  (Only in the linked, "v4" version of the patch, however--I'm about to remove the debug stuff for checkin.)
> It's possible I've done something completely boneheaded here, but the file, at least, checks out in a subsequent set of subtests and with stock bzip2 itself.  Only the code above is problematic; it reads through the first concatenated chunk (17 lines of text) just fine but chokes on the header of the second one.  Altogether, the test file contains 84 lines of text and 4 concatenated bzip2 files.
> (It's possible this is a mapreduce issue rather than common, but note that the identical gzip test works fine.  Possibly it's related to the stream-vs-decompressor dichotomy, though; intentionally not supported?)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10385-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 25 11:22:02 2010
Return-Path: <common-issues-return-10385-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34964 invoked from network); 25 Aug 2010 11:22:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Aug 2010 11:22:02 -0000
Received: (qmail 44799 invoked by uid 500); 25 Aug 2010 11:22:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44634 invoked by uid 500); 25 Aug 2010 11:21:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44621 invoked by uid 99); 25 Aug 2010 11:21:58 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 11:21:58 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 11:21:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7PBLHEY004887
	for <common-issues@hadoop.apache.org>; Wed, 25 Aug 2010 11:21:17 GMT
Message-ID: <24816146.556001282735277298.JavaMail.jira@thor>
Date: Wed, 25 Aug 2010 07:21:17 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6925) BZip2Codec incorrectly implements
 read()
In-Reply-To: <8553385.542391282687876583.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902380#action_12902380 ] 

Hudson commented on HADOOP-6925:
--------------------------------

Integrated in Hadoop-Common-trunk #433 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/433/])
    HADOOP-6925. BZip2Codec incorrectly implements read(). Contributed by Todd Lipcon.


> BZip2Codec incorrectly implements read()
> ----------------------------------------
>
>                 Key: HADOOP-6925
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6925
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.21.1, 0.22.0
>
>         Attachments: hadoop-6925.txt
>
>
> HADOOP-4012 added an implementation of read() in BZip2InputStream that doesn't work correctly when reading bytes > 0x80. This causes EOFExceptions when working with BZip2 compressed data inside of sequence files in some datasets.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10386-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 25 18:20:43 2010
Return-Path: <common-issues-return-10386-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14831 invoked from network); 25 Aug 2010 18:20:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Aug 2010 18:20:43 -0000
Received: (qmail 38730 invoked by uid 500); 25 Aug 2010 18:20:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38701 invoked by uid 500); 25 Aug 2010 18:20:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38693 invoked by uid 99); 25 Aug 2010 18:20:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 18:20:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 18:20:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7PIKHRg009205
	for <common-issues@hadoop.apache.org>; Wed, 25 Aug 2010 18:20:18 GMT
Message-ID: <33169650.563181282760417886.JavaMail.jira@thor>
Date: Wed, 25 Aug 2010 14:20:17 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902556#action_12902556 ] 

Todd Lipcon commented on HADOOP-6762:
-------------------------------------

Hi Sam,

With this patch, I see occasinal failures of TestGridmixSubmission when the JobMonitor gets interrupted:

{noformat}
10/08/25 11:05:20 WARN ipc.Client: interrupted waiting to send params to server
java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1215)
        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
        at java.util.concurrent.FutureTask.get(FutureTask.java:83)
        at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:754)
        at org.apache.hadoop.ipc.Client.call(Client.java:1001)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224)
        at org.apache.hadoop.mapred.$Proxy10.getJobStatus(Unknown Source)
        at org.apache.hadoop.mapred.JobClient$NetworkedJob.updateStatus(JobClient.java:250)
        at org.apache.hadoop.mapred.JobClient$NetworkedJob.isSuccessful(JobClient.java:339)
        at org.apache.hadoop.mapreduce.Job.isSuccessful(Job.java:332)
        at org.apache.hadoop.mapred.gridmix.JobMonitor$MonitorThread.process(JobMonitor.java:134)
        at org.apache.hadoop.mapred.gridmix.JobMonitor$MonitorThread.run(JobMonitor.java:175)
10/08/25 11:05:20 WARN gridmix.JobMonitor: Lost job GRIDMIX00000
java.io.IOException: java.lang.InterruptedException
        at org.apache.hadoop.ipc.Client.call(Client.java:1007)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224)
        at org.apache.hadoop.mapred.$Proxy10.getJobStatus(Unknown Source)
        at org.apache.hadoop.mapred.JobClient$NetworkedJob.updateStatus(JobClient.java:250)
        at org.apache.hadoop.mapred.JobClient$NetworkedJob.isSuccessful(JobClient.java:339)
        at org.apache.hadoop.mapreduce.Job.isSuccessful(Job.java:332)
        at org.apache.hadoop.mapred.gridmix.JobMonitor$MonitorThread.process(JobMonitor.java:134)
        at org.apache.hadoop.mapred.gridmix.JobMonitor$MonitorThread.run(JobMonitor.java:175)
Caused by: java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1215)
        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
        at java.util.concurrent.FutureTask.get(FutureTask.java:83)
        at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:754)
        at org.apache.hadoop.ipc.Client.call(Client.java:1001)
        ... 7 more
{noformat}

I think the patch is doing the right thing here for the most part, but it should throw InterruptedIOException instead of just a normal IOException. Then the caller can at least catch it distinctly from a normal IOException.

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt, hadoop-6762-7.txt, hadoop-6762-8.txt, hadoop-6762-9.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10387-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 25 18:26:39 2010
Return-Path: <common-issues-return-10387-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23968 invoked from network); 25 Aug 2010 18:26:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Aug 2010 18:26:39 -0000
Received: (qmail 49993 invoked by uid 500); 25 Aug 2010 18:26:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49958 invoked by uid 500); 25 Aug 2010 18:26:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49932 invoked by uid 99); 25 Aug 2010 18:26:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 18:26:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 18:26:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7PIQHhl009255
	for <common-issues@hadoop.apache.org>; Wed, 25 Aug 2010 18:26:17 GMT
Message-ID: <32371906.563261282760777754.JavaMail.jira@thor>
Date: Wed, 25 Aug 2010 14:26:17 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902557#action_12902557 ] 

Todd Lipcon commented on HADOOP-6762:
-------------------------------------

actually, I might take it back - it looks like you can't pass cause information into an InterruptedIOException, so it's better to have callers check e.getCause() instanceof InterruptedException instead? What do you think? The gridmix case above already checks for ClosedByInterruptException, so it can easy check for InterruptedException too.

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt, hadoop-6762-7.txt, hadoop-6762-8.txt, hadoop-6762-9.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10388-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 25 22:41:58 2010
Return-Path: <common-issues-return-10388-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36319 invoked from network); 25 Aug 2010 22:41:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Aug 2010 22:41:57 -0000
Received: (qmail 99430 invoked by uid 500); 25 Aug 2010 22:41:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99121 invoked by uid 500); 25 Aug 2010 22:41:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99105 invoked by uid 99); 25 Aug 2010 22:41:56 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 22:41:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 22:41:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7PMfHcD012969
	for <common-issues@hadoop.apache.org>; Wed, 25 Aug 2010 22:41:17 GMT
Message-ID: <28323308.571051282776077877.JavaMail.jira@thor>
Date: Wed, 25 Aug 2010 18:41:17 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902685#action_12902685 ] 

sam rash commented on HADOOP-6762:
----------------------------------

I think IOException makes sense for 2 reasons

1. the reason you stated--we can pass the actual cause
2. my read is that only if you are the thread doing the IO should you throw InterruptedIOException.  In this case, the thread truly was interrupted--the IO happens in the other thread.  
http://download-llnw.oracle.com/javase/6/docs/api/java/io/InterruptedIOException.html
it also has a field for the partial bytes transfered.  In fact, this patch is supposed to keep the IO from actually being interrupted :)

how can we make it so the test will pass?  does the test have a faulty assumption that I should fix?

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt, hadoop-6762-7.txt, hadoop-6762-8.txt, hadoop-6762-9.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10389-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 25 22:57:57 2010
Return-Path: <common-issues-return-10389-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39135 invoked from network); 25 Aug 2010 22:57:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Aug 2010 22:57:56 -0000
Received: (qmail 17133 invoked by uid 500); 25 Aug 2010 22:57:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17021 invoked by uid 500); 25 Aug 2010 22:57:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17013 invoked by uid 99); 25 Aug 2010 22:57:56 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 22:57:56 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 22:57:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7PMvHw1013204
	for <common-issues@hadoop.apache.org>; Wed, 25 Aug 2010 22:57:17 GMT
Message-ID: <20857438.571651282777037128.JavaMail.jira@thor>
Date: Wed, 25 Aug 2010 18:57:17 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902692#action_12902692 ] 

Todd Lipcon commented on HADOOP-6762:
-------------------------------------

I think we just need this fix to the gridmix test:
{quote}
--- a/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/JobMonitor.java
+++ b/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/JobMonitor.java
@@ -177,7 +177,8 @@ class JobMonitor implements Gridmix.Component<Job> {
                 continue;
               }
             } catch (IOException e) {
-              if (e.getCause() instanceof ClosedByInterruptException) {
+              if (e.getCause() instanceof ClosedByInterruptException ||
+                  e.getCause() instanceof InterruptedException) {
                 // Job doesn't throw InterruptedException, but RPC socket layer
                 // is blocking and may throw a wrapped Exception if this thread
                 // is interrupted. Since the lower level cleared the flag,
{quote}

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt, hadoop-6762-7.txt, hadoop-6762-8.txt, hadoop-6762-9.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10390-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Aug 25 23:56:00 2010
Return-Path: <common-issues-return-10390-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57439 invoked from network); 25 Aug 2010 23:56:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Aug 2010 23:56:00 -0000
Received: (qmail 56316 invoked by uid 500); 25 Aug 2010 23:56:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56239 invoked by uid 500); 25 Aug 2010 23:55:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56231 invoked by uid 99); 25 Aug 2010 23:55:59 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 23:55:59 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Aug 2010 23:55:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7PNtKZp013781
	for <common-issues@hadoop.apache.org>; Wed, 25 Aug 2010 23:55:20 GMT
Message-ID: <5828386.572961282780520211.JavaMail.jira@thor>
Date: Wed, 25 Aug 2010 19:55:20 -0400 (EDT)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902709#action_12902709 ] 

Konstantin Shvachko commented on HADOOP-6884:
---------------------------------------------

h3. This issue

Based on what is said in this jira and discussions with developers I'd like to make the following proposal.
There are 3 main goals the change should be targeting for:
# *debug() statement optimization*: do not calculate the arguments if they are not going to be logged.
# *code readability*: avoid if-debug-enabled statement before logging.
# *new code verification*: provide an automatic procedure or simple rules to verify that the newly introduced code does not introduces that same inefficiencies.

My approach is a variant of Doug's proposal and SLF4J formatting.
# Let's create a new class {{org.apache.hadoop.log.Logger}}
We will have only static methods in the class for now:
{code}
class Logger {
  // Optimized logging
  static void logDebug(Log log, Object a1);
  static void logDebug(Log log, Object a1, Object a2);
  static void logDebug(Log log, Object a1, Object a2, Object a3);
  static void logDebug(Log log, Object a1, Object a2, Object a3, Object a4);

  // not efficient, do not use on performance critical paths
  static void logDebug(Log log, Object... args);
  static void logDebug(Log log, String formatter, Object... args);
  static void logDebug(Log log, Throwable e, Object... args);
}
{code}
# Each statement
{{Log.debug(a1 + a2 + ...)}} 
in current code is replaced by one of the methods above:
{{Logger.logDebug(Log, a1, a2, ...)}}
Looking at our code, about 90% of debug calls will be replaced by the first 4 methods,
which do not compute the arguments and do not instantiate an array object.
Most of the rest debug calls will be converted into a {{logDebug(Log log, Object... args)}} call.
Which is less efficient, but still faster than building a string.
The remaining calls can be transformed into "formatter" version. We should make sure these are not
performance-critical parts of the code, or otherwise rewrite the messages to be able to call one 
of the optimized versions of {{logDebug()}}.
Autoboxing of primitive types should be ok with these less efficient versions of {{logDebug()}}.
# The implementation of {{logDebug()}} should use optimized string building tools:
{code}
  static void logDebug(Log log, Object a1, Object a2) {
    if(!log.isDebugEnabled())
      return;
    StringBuilder str = new StringBuilder().append(a1).append(a2);
    log.debug(str);
  }
{code}
# The JavaDoc for {{logDebug()}} methods should state which methods are optimized, and 
what to avoid when calling them, like primitive type autoboxing, string concatenation, etc.

This will give us all three goals above, up to a point. Efficiency and readability should be obvious. 
The tool to verify should be checking that there are no "naked" calls of ".debug" anywhere but 
in the Logger. And that calls of {{logDebug()}} do not contain "+" operations.

This is not the ideal solution, but it is practical imo.

h3. Future directions

In the future we can turn {{org.apache.hadoop.log.Logger}} into a log wrapper by converting
the static methods into non static, and adding methods logInfo(), logError(), logWarn().
This should optimize logging in Hadoop in general. Non optimal string building in info-level
logging hearts the performance right now as I write it.
The idea is to replace {{org.apache.commons.logging.Log}} with {{org.apache.hadoop.log.Logger}}.
{{logging.Log}} will become a member of the {{Logger}}.
This is a bigger change and should be done in a different jira.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10391-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 00:31:38 2010
Return-Path: <common-issues-return-10391-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74498 invoked from network); 26 Aug 2010 00:31:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 00:31:38 -0000
Received: (qmail 73480 invoked by uid 500); 26 Aug 2010 00:31:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73268 invoked by uid 500); 26 Aug 2010 00:31:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73259 invoked by uid 99); 26 Aug 2010 00:31:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 00:31:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 00:31:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7Q0VGUY014262
	for <common-issues@hadoop.apache.org>; Thu, 26 Aug 2010 00:31:16 GMT
Message-ID: <16874682.574151282782676914.JavaMail.jira@thor>
Date: Wed, 25 Aug 2010 20:31:16 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6833) IPC leaks call parameters when
 exceptions thrown
In-Reply-To: <21655383.3791277174455275.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902729#action_12902729 ] 

Eli Collins commented on HADOOP-6833:
-------------------------------------

Hey Todd,

Looks like the test added in this patch is racy, it failed for me after I looped it for a couple minutes. When System.gc returns it just means the gc has made a best effort to run, not that it actually has, so in the cases where the gc hasn't run the test will fail because myArg was not reaped.

The change itself looks correct. Since this is a one-liner and memory leaks like this are hard to test I'd be OK with a new patch that just fixes the bug as long as it passes the existing RPC tests. Any other suggestions on how to add test coverage of course are welcome, doesn't seem worth eg adding a getter that returns the number of calls. 

Thanks,
Eli

> IPC leaks call parameters when exceptions thrown
> ------------------------------------------------
>
>                 Key: HADOOP-6833
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6833
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2, 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Blocker
>         Attachments: hadoop-6833.txt
>
>
> HADOOP-6498 moved the calls.remove() call lower into the SUCCESS clause of receiveResponse(), but didn't put a similar calls.remove into the ERROR clause. So, any RPC call that throws an exception ends up orphaning the Call object in the connection's "calls" hashtable. This prevents cleanup of the connection and is a memory leak for the call parameters.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10392-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 09:05:39 2010
Return-Path: <common-issues-return-10392-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37580 invoked from network); 26 Aug 2010 09:05:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 09:05:39 -0000
Received: (qmail 38416 invoked by uid 500); 26 Aug 2010 09:05:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38265 invoked by uid 500); 26 Aug 2010 09:05:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38254 invoked by uid 99); 26 Aug 2010 09:05:35 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 09:05:35 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 09:05:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7Q94tC6002688
	for <common-issues@hadoop.apache.org>; Thu, 26 Aug 2010 09:04:56 GMT
Message-ID: <27036842.1811282813495832.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 05:04:55 -0400 (EDT)
From: "Simone Leo (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6388) Hadoop uses invalid names for
 environment variables
In-Reply-To: <286500122.1259076579674.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Simone Leo resolved HADOOP-6388.
--------------------------------

    Resolution: Not A Problem

The old behavior has been restored in bash 4.1. Just Stay away from bash 4.0 and you'll be fine :)

> Hadoop uses invalid names for environment variables
> ---------------------------------------------------
>
>                 Key: HADOOP-6388
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6388
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.1
>            Reporter: Simone Leo
>
> Variable names cannot contain dots and this is enforced starting from bash 4.0, see [http://www.mail-archive.com/bug-bash@gnu.org/msg05726.html].
> An example of Hadoop functionality that breaks after upgrading to bash 4.0 is running Pipes applications: the "hadoop.pipes.command.port" var is not passed and applications crash with "Hadoop Pipes Exception: Illegal text protocol command".

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10393-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 10:38:18 2010
Return-Path: <common-issues-return-10393-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60104 invoked from network); 26 Aug 2010 10:38:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 10:38:18 -0000
Received: (qmail 20339 invoked by uid 500); 26 Aug 2010 10:38:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20186 invoked by uid 500); 26 Aug 2010 10:38:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20178 invoked by uid 99); 26 Aug 2010 10:38:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 10:38:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 10:38:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7QAbr9O003453
	for <common-issues@hadoop.apache.org>; Thu, 26 Aug 2010 10:37:53 GMT
Message-ID: <22066225.2621282819073773.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 06:37:53 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6922) COMMON part of MAPREDUCE-1664
In-Reply-To: <23487377.527071282641377033.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902813#action_12902813 ] 

Vinod K V commented on HADOOP-6922:
-----------------------------------

Fix cluster_setup.xml to document mapred.acls.enabled that it is needed in both JT's and TT's conf files if at all.

Other than that the patch looks good.


> COMMON part of MAPREDUCE-1664
> -----------------------------
>
>                 Key: HADOOP-6922
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6922
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>         Attachments: 6922.patch
>
>
> MAPREDUCE-1664 changes the behavior of queue acls and job acls. This needs documentation changes to cluster_setup.xml and a small change in AccessControlList.java

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10394-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 16:26:17 2010
Return-Path: <common-issues-return-10394-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79901 invoked from network); 26 Aug 2010 16:26:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 16:26:17 -0000
Received: (qmail 25197 invoked by uid 500); 26 Aug 2010 16:26:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25073 invoked by uid 500); 26 Aug 2010 16:26:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25064 invoked by uid 99); 26 Aug 2010 16:26:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 16:26:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 16:26:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7QGPt3K025813
	for <common-issues@hadoop.apache.org>; Thu, 26 Aug 2010 16:25:55 GMT
Message-ID: <28136490.7531282839955302.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 12:25:55 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902917#action_12902917 ] 

Doug Cutting commented on HADOOP-6884:
--------------------------------------

Konstantin, this sounds like it could be a workable approach.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10395-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 16:49:36 2010
Return-Path: <common-issues-return-10395-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88235 invoked from network); 26 Aug 2010 16:49:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 16:49:35 -0000
Received: (qmail 67874 invoked by uid 500); 26 Aug 2010 16:49:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67834 invoked by uid 500); 26 Aug 2010 16:49:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67826 invoked by uid 99); 26 Aug 2010 16:49:34 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 16:49:34 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 16:49:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7QGmugD028004
	for <common-issues@hadoop.apache.org>; Thu, 26 Aug 2010 16:48:56 GMT
Message-ID: <30824268.7881282841336257.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 12:48:56 -0400 (EDT)
From: "Al Thompson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6927) Herriot NN and DN clients should vend
 statistics
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Herriot NN and DN clients should vend statistics
------------------------------------------------

                 Key: HADOOP-6927
                 URL: https://issues.apache.org/jira/browse/HADOOP-6927
             Project: Hadoop Common
          Issue Type: Improvement
          Components: test
            Reporter: Al Thompson


The HDFS web user interface serves useful information through dfshealth.jsp and dfsnodelist.jsp.

The Herriot interface to the namenode and datanode (as implemented in NNClient and DNClient, respectively) would benefit from the addition of some way to channel this information. In the case of DNClient this can be an injected method that returns a DatanodeDescriptor relevant to the underlying datanode.

There seems to be no analagous NamenodeDescriptor. It may be useful to add this as a facade to a visitor that aggregates values across the filesystem datanodes. These values are (from dfshealth JSP):

Configured Capacity
DFS Used
Non DFS Used
DFS Remaining
DFS Used%
DFS Remaining%
Live Nodes
Dead Nodes
Decommissioning Nodes
Number of Under-Replicated Blocks

Attributes reflecting the web user interface header may also be useful such as When-Started, Version, When-Compiled, and Upgrade-Status.

A NamenodeDescriptor would essentially "push down" the code in dfshealth web UI behind a more general abstraction. If it is objectionable to make this class available in HDFS, perhaps this could be packaged in a Herriot specific way.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10396-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 16:53:18 2010
Return-Path: <common-issues-return-10396-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90181 invoked from network); 26 Aug 2010 16:53:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 16:53:18 -0000
Received: (qmail 74750 invoked by uid 500); 26 Aug 2010 16:53:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74694 invoked by uid 500); 26 Aug 2010 16:53:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74686 invoked by uid 99); 26 Aug 2010 16:53:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 16:53:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 16:53:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7QGqvix001997
	for <common-issues@hadoop.apache.org>; Thu, 26 Aug 2010 16:52:57 GMT
Message-ID: <10318639.8201282841577197.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 12:52:57 -0400 (EDT)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902930#action_12902930 ] 

Scott Carey commented on HADOOP-6884:
-------------------------------------

This is not much different than just using SLF4J, the only difference is that there would be some extra Object[] creation with SLF4J (which is really not a big performance problem, about the same as auto-boxing).  But SLF4J would have a much faster and easier to use formatting implementation, and have performance benefits for INFO and other debug levels.   

Additionally, we can still create a wrappper around that to add signatures to prevent object array creation and/or autoboxing for a limited subset of cases.

In short, why not just use SLF4J's more featured and flexible wrapper and either extend that or push for adding a few extra signatures on that side?  It even has a tool to go through the source code and change stuff from log4j for you.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10397-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 16:53:33 2010
Return-Path: <common-issues-return-10397-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90235 invoked from network); 26 Aug 2010 16:53:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 16:53:33 -0000
Received: (qmail 74995 invoked by uid 500); 26 Aug 2010 16:53:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74956 invoked by uid 500); 26 Aug 2010 16:53:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74948 invoked by uid 99); 26 Aug 2010 16:53:32 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 16:53:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 16:53:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7QGqsDH001967
	for <common-issues@hadoop.apache.org>; Thu, 26 Aug 2010 16:52:54 GMT
Message-ID: <32649298.8101282841573990.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 12:52:53 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6928) Fix BooleanWritable comparator in
 0.20
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Fix BooleanWritable comparator in 0.20
--------------------------------------

                 Key: HADOOP-6928
                 URL: https://issues.apache.org/jira/browse/HADOOP-6928
             Project: Hadoop Common
          Issue Type: Bug
          Components: io
    Affects Versions: 0.20.2, 0.20.1, 0.20.0
            Reporter: Owen O'Malley
             Fix For: 0.20.3


The RawComparator for BooleanWritable was fixed as part of HADOOP-5699 in 0.21 and trunk. The fix should be pushed back into 0.20.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10398-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 16:56:16 2010
Return-Path: <common-issues-return-10398-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91351 invoked from network); 26 Aug 2010 16:56:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 16:56:16 -0000
Received: (qmail 81651 invoked by uid 500); 26 Aug 2010 16:56:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81446 invoked by uid 500); 26 Aug 2010 16:56:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81437 invoked by uid 99); 26 Aug 2010 16:56:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 16:56:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 16:56:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7QGtsc7002491
	for <common-issues@hadoop.apache.org>; Thu, 26 Aug 2010 16:55:54 GMT
Message-ID: <11837833.8281282841754746.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 12:55:54 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6928) Fix BooleanWritable comparator in
 0.20
In-Reply-To: <32649298.8101282841573990.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6928:
----------------------------------

    Attachment: h-6928.patch

This is the relevant part of the patch that was applied to 0.21 and trunk.

> Fix BooleanWritable comparator in 0.20
> --------------------------------------
>
>                 Key: HADOOP-6928
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6928
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.0, 0.20.1, 0.20.2
>            Reporter: Owen O'Malley
>             Fix For: 0.20.3
>
>         Attachments: h-6928.patch
>
>
> The RawComparator for BooleanWritable was fixed as part of HADOOP-5699 in 0.21 and trunk. The fix should be pushed back into 0.20.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10399-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 16:56:17 2010
Return-Path: <common-issues-return-10399-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91433 invoked from network); 26 Aug 2010 16:56:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 16:56:16 -0000
Received: (qmail 82018 invoked by uid 500); 26 Aug 2010 16:56:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81563 invoked by uid 500); 26 Aug 2010 16:56:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81457 invoked by uid 99); 26 Aug 2010 16:56:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 16:56:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 16:56:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7QGttn3002567
	for <common-issues@hadoop.apache.org>; Thu, 26 Aug 2010 16:55:55 GMT
Message-ID: <9408793.8321282841755489.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 12:55:55 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6928) Fix BooleanWritable comparator in
 0.20
In-Reply-To: <32649298.8101282841573990.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6928:
----------------------------------

    Status: Patch Available  (was: Open)

> Fix BooleanWritable comparator in 0.20
> --------------------------------------
>
>                 Key: HADOOP-6928
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6928
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.2, 0.20.1, 0.20.0
>            Reporter: Owen O'Malley
>             Fix For: 0.20.3
>
>         Attachments: h-6928.patch
>
>
> The RawComparator for BooleanWritable was fixed as part of HADOOP-5699 in 0.21 and trunk. The fix should be pushed back into 0.20.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10400-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 17:10:16 2010
Return-Path: <common-issues-return-10400-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99543 invoked from network); 26 Aug 2010 17:10:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 17:10:15 -0000
Received: (qmail 99106 invoked by uid 500); 26 Aug 2010 17:10:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98997 invoked by uid 500); 26 Aug 2010 17:10:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98980 invoked by uid 99); 26 Aug 2010 17:10:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 17:10:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 17:10:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7QH9sFQ010505
	for <common-issues@hadoop.apache.org>; Thu, 26 Aug 2010 17:09:54 GMT
Message-ID: <16004183.8381282842594018.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 13:09:54 -0400 (EDT)
From: "Al Thompson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6927) Herriot NN and DN clients should
 vend statistics
In-Reply-To: <30824268.7881282841336257.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Al Thompson resolved HADOOP-6927.
---------------------------------

    Resolution: Invalid

Best to use HDFS JMX support for gathering this info.

> Herriot NN and DN clients should vend statistics
> ------------------------------------------------
>
>                 Key: HADOOP-6927
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6927
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>            Reporter: Al Thompson
>
> The HDFS web user interface serves useful information through dfshealth.jsp and dfsnodelist.jsp.
> The Herriot interface to the namenode and datanode (as implemented in NNClient and DNClient, respectively) would benefit from the addition of some way to channel this information. In the case of DNClient this can be an injected method that returns a DatanodeDescriptor relevant to the underlying datanode.
> There seems to be no analagous NamenodeDescriptor. It may be useful to add this as a facade to a visitor that aggregates values across the filesystem datanodes. These values are (from dfshealth JSP):
> Configured Capacity
> DFS Used
> Non DFS Used
> DFS Remaining
> DFS Used%
> DFS Remaining%
> Live Nodes
> Dead Nodes
> Decommissioning Nodes
> Number of Under-Replicated Blocks
> Attributes reflecting the web user interface header may also be useful such as When-Started, Version, When-Compiled, and Upgrade-Status.
> A NamenodeDescriptor would essentially "push down" the code in dfshealth web UI behind a more general abstraction. If it is objectionable to make this class available in HDFS, perhaps this could be packaged in a Herriot specific way.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10401-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 17:28:16 2010
Return-Path: <common-issues-return-10401-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5552 invoked from network); 26 Aug 2010 17:28:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 17:28:16 -0000
Received: (qmail 23025 invoked by uid 500); 26 Aug 2010 17:28:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22948 invoked by uid 500); 26 Aug 2010 17:28:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22940 invoked by uid 99); 26 Aug 2010 17:28:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 17:28:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 17:28:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7QHRtLV010741
	for <common-issues@hadoop.apache.org>; Thu, 26 Aug 2010 17:27:55 GMT
Message-ID: <9726580.8901282843675111.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 13:27:55 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902943#action_12902943 ] 

Doug Cutting commented on HADOOP-6884:
--------------------------------------

Scott> This is not much different than just using SLF4J [ ... ]

I too would prefer we simply switch to slf4j.  In performance-critical code that requires complex log messages we might still ocasionally use isDebugEnabled(), as we might make other local code optimizations in critical sections.  To be clear, I find Konstantin's approach acceptable, but feel a move to slf4j would be simpler with essentially the same benefits. 

String concatenation is the biggest potential performance problem.  Adding an automated test that detects string concatenation in log statements would be a good addition.  Before we can add that test though, we need a logging API that does not require string concatenation and does not bloat code.  Slf4j and Konstantin's approach both provide one.


> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10402-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 18:49:16 2010
Return-Path: <common-issues-return-10402-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35340 invoked from network); 26 Aug 2010 18:49:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 18:49:16 -0000
Received: (qmail 49598 invoked by uid 500); 26 Aug 2010 18:49:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49538 invoked by uid 500); 26 Aug 2010 18:49:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49530 invoked by uid 99); 26 Aug 2010 18:49:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 18:49:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 18:49:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7QImsRp015949
	for <common-issues@hadoop.apache.org>; Thu, 26 Aug 2010 18:48:55 GMT
Message-ID: <32445652.10761282848534970.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 14:48:54 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6833) IPC leaks call parameters when
 exceptions thrown
In-Reply-To: <21655383.3791277174455275.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6833:
--------------------------------

    Attachment: hadoop-6833.txt

Here's the same patch minus the unit test.

As for manual/cluster testing, we've been running with this fix in HBase's copy of RPC for a couple months. We used to eventually run out of RAM, heap analysis showed this was the problem. This fix got rid of the issue.

> IPC leaks call parameters when exceptions thrown
> ------------------------------------------------
>
>                 Key: HADOOP-6833
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6833
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2, 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Blocker
>         Attachments: hadoop-6833.txt, hadoop-6833.txt
>
>
> HADOOP-6498 moved the calls.remove() call lower into the SUCCESS clause of receiveResponse(), but didn't put a similar calls.remove into the ERROR clause. So, any RPC call that throws an exception ends up orphaning the Call object in the connection's "calls" hashtable. This prevents cleanup of the connection and is a memory leak for the call parameters.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10403-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 19:01:55 2010
Return-Path: <common-issues-return-10403-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37777 invoked from network); 26 Aug 2010 19:01:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 19:01:54 -0000
Received: (qmail 71808 invoked by uid 500); 26 Aug 2010 19:01:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71734 invoked by uid 500); 26 Aug 2010 19:01:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71726 invoked by uid 99); 26 Aug 2010 19:01:53 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 19:01:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 19:01:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7QJ1E1c016084
	for <common-issues@hadoop.apache.org>; Thu, 26 Aug 2010 19:01:14 GMT
Message-ID: <6014598.11001282849274488.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 15:01:14 -0400 (EDT)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902985#action_12902985 ] 

Konstantin Shvachko commented on HADOOP-6884:
---------------------------------------------

I am reluctant to migrate to SLF4J at the moment for 3 main reasons. 
# Yes, SLF4J does provide a way to log more efficiently, but it also has the regular method {{void debug(String msg)}} so people will just use it as they did before. In my approach I intentionally excluded it from the api.
# Migrating to SLF4J will mean changing logging for all levels in the entire project(s). This issue is targeting only debug messages.
Logging is a big deal. apache.commons worked for us for a long time, will SLF4J - I don't know. There are analytic tools out there relying on the format of the logs, will SLF4J break them? All this needs investigation. Just optimizing debug logging will not affect anything except improving the performance.
# In fact, people in this discussion mentioned that creating extra object[] is a performance problem. I tried to address this issue in my proposal. And I think this makes a big difference compared to SLF4J, because 90% of the calls will end up executing the optimized versions of logDebug().

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10404-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 19:17:17 2010
Return-Path: <common-issues-return-10404-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44773 invoked from network); 26 Aug 2010 19:17:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 19:17:16 -0000
Received: (qmail 93657 invoked by uid 500); 26 Aug 2010 19:17:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93593 invoked by uid 500); 26 Aug 2010 19:17:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93585 invoked by uid 99); 26 Aug 2010 19:17:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 19:17:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 19:17:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7QJGsEc020169
	for <common-issues@hadoop.apache.org>; Thu, 26 Aug 2010 19:16:55 GMT
Message-ID: <22946324.12011282850214877.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 15:16:54 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902994#action_12902994 ] 

Doug Cutting commented on HADOOP-6884:
--------------------------------------

> SLF4J [ ...]  has the regular method void debug(String msg) so people will just use it as they did before.

Not if we have an automated test that looks for string concatenation in log statements.  That's required for your proposal too, no?

> Migrating to SLF4J will mean changing logging for all levels in the entire project(s).

SLF4J wraps log4j, just like commons logging.  So existing log configuration & format should be unaffected.

> creating extra object[] is a performance problem

No performance problems have actually been identified.  Some have been suggested, but none shown to be significant in real operations.

That said, micro-benchmarks show that, were there a performance problem, getting rid of string concatenation would reduce the impact of logging by around 90%.  Removing Object[] allocation would improve it further, but it's not clear, given that even that the string concatenation has not shown to be significant, that this would be a significant improvement.  Plus, the majority of log statements have a single parameter and would not require an Object[] allocation.  So, unless someone can show that Object[] allocation in multi-parameter log statements is slowing things significantly, slf4j would suffice.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10405-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 19:43:19 2010
Return-Path: <common-issues-return-10405-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51449 invoked from network); 26 Aug 2010 19:43:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 19:43:19 -0000
Received: (qmail 28209 invoked by uid 500); 26 Aug 2010 19:43:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28142 invoked by uid 500); 26 Aug 2010 19:43:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28134 invoked by uid 99); 26 Aug 2010 19:43:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 19:43:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 19:43:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7QJgvgD020523
	for <common-issues@hadoop.apache.org>; Thu, 26 Aug 2010 19:42:57 GMT
Message-ID: <5975018.12761282851777599.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 15:42:57 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903003#action_12903003 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6884:
------------------------------------------------

> ...  , given that even that the string concatenation has not shown to be significant, ...

Doug, have you done any benchmark to support the above statement?

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10406-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 19:54:18 2010
Return-Path: <common-issues-return-10406-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53671 invoked from network); 26 Aug 2010 19:54:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 19:54:18 -0000
Received: (qmail 40614 invoked by uid 500); 26 Aug 2010 19:54:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40546 invoked by uid 500); 26 Aug 2010 19:54:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40534 invoked by uid 99); 26 Aug 2010 19:54:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 19:54:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 19:54:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7QJrunU020699
	for <common-issues@hadoop.apache.org>; Thu, 26 Aug 2010 19:53:57 GMT
Message-ID: <26292039.13251282852436677.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 15:53:56 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903009#action_12903009 ] 

Doug Cutting commented on HADOOP-6884:
--------------------------------------

No, I have not demonstrated that the cost string concatenation in debug existing log messages is insignificant, but neither has anyone else to my knowledge shown that this cost *is* significant.  I'm willing to accept optimizations that don't make the code any larger without proof of their significance, but optimizations that increase code size should first be shown to significantly improve performance, no?

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10407-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 20:10:15 2010
Return-Path: <common-issues-return-10407-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61812 invoked from network); 26 Aug 2010 20:10:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 20:10:15 -0000
Received: (qmail 66715 invoked by uid 500); 26 Aug 2010 20:10:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66669 invoked by uid 500); 26 Aug 2010 20:10:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66661 invoked by uid 99); 26 Aug 2010 20:10:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 20:10:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 20:10:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7QK9ro1020898
	for <common-issues@hadoop.apache.org>; Thu, 26 Aug 2010 20:09:54 GMT
Message-ID: <9637227.13511282853393848.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 16:09:53 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6902) RPC server's SASL_PROPS shouldn't be
 re-initialized every time an RPC client is created
In-Reply-To: <31093459.190711281051317983.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang reassigned HADOOP-6902:
---------------------------------

    Assignee: Kan Zhang

> RPC server's SASL_PROPS shouldn't be re-initialized every time an RPC client is created
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6902
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6902
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6902-03.patch
>
>
> SaslRpcServer.SASL_PROPS is a SASL server property and should stay constant after initialization. In the initial implementation, we assumed all SASL clients will use the same constant value. If different clients might use different values depending on the conf in the getProxy() call (as current code implies), each client should have its own copy. In any case, a client shouldn't re-initialize server's copy.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10408-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 20:49:18 2010
Return-Path: <common-issues-return-10408-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70369 invoked from network); 26 Aug 2010 20:49:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 20:49:17 -0000
Received: (qmail 8214 invoked by uid 500); 26 Aug 2010 20:49:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8001 invoked by uid 500); 26 Aug 2010 20:49:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7988 invoked by uid 99); 26 Aug 2010 20:49:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 20:49:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 20:49:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7QKms38021567
	for <common-issues@hadoop.apache.org>; Thu, 26 Aug 2010 20:48:55 GMT
Message-ID: <29776327.15071282855734948.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 16:48:54 -0400 (EDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6929) RPC should have a way to pass
 Security information other than protocol annotations
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

RPC should have a way to pass Security information other than protocol annotations
----------------------------------------------------------------------------------

                 Key: HADOOP-6929
                 URL: https://issues.apache.org/jira/browse/HADOOP-6929
             Project: Hadoop Common
          Issue Type: Improvement
          Components: ipc, security
            Reporter: Sharad Agarwal
             Fix For: 0.22.0


Currently Hadoop RPC allows protocol annotations as the only way to pass security information. This becomes a problem if protocols are generated and not hand written. For example protocols generated via Avro and passed over Avro tunnel (AvroRpcEngine.java) can't pass the security information.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10409-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 21:00:42 2010
Return-Path: <common-issues-return-10409-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72273 invoked from network); 26 Aug 2010 21:00:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 21:00:41 -0000
Received: (qmail 19232 invoked by uid 500); 26 Aug 2010 21:00:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19025 invoked by uid 500); 26 Aug 2010 21:00:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19015 invoked by uid 99); 26 Aug 2010 21:00:40 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 21:00:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 21:00:23 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7QL02qH021759
	for <common-issues@hadoop.apache.org>; Thu, 26 Aug 2010 21:00:02 GMT
Message-ID: <6489936.15501282856402081.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 17:00:02 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-584) Calling shell scripts from build.xml
 discriminates Windows user minority.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903046#action_12903046 ] 

Konstantin Boudnik commented on HADOOP-584:
-------------------------------------------

I'd say to hell with windows minority :)

> Calling shell scripts from build.xml discriminates Windows user minority.
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-584
>                 URL: https://issues.apache.org/jira/browse/HADOOP-584
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.7.0
>         Environment: Windows
>            Reporter: Konstantin Shvachko
>
> This is introduced by HADOOP-567.
> The problem is that now I cannot even build hadoop in Eclipse under Windows unless I run it under Cygwin. 
> This is in a way the same as calling make in build.xml, which was recently fixed HADOOP-537.
> I think we should not introducing more dependencies on Cygwin just in order to show something in Web UI.
> I also don't remember we claimed that Cygwin or anything else except for Ant is required for Hadoop builds. 
> Is there another way of solving this?
> build.xml defines "version" property, Ant has "user.name" property. URL is not changing very often.
> Or may be the web ui should obtain these properties in run-time.
> Or may be the Packaging is a better solution, as you guys discussed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10410-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 21:04:15 2010
Return-Path: <common-issues-return-10410-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73059 invoked from network); 26 Aug 2010 21:04:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 21:04:15 -0000
Received: (qmail 22325 invoked by uid 500); 26 Aug 2010 21:04:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22118 invoked by uid 500); 26 Aug 2010 21:04:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22105 invoked by uid 99); 26 Aug 2010 21:04:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 21:04:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 21:04:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7QL3rRo021848
	for <common-issues@hadoop.apache.org>; Thu, 26 Aug 2010 21:03:54 GMT
Message-ID: <32541354.15611282856633769.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 17:03:53 -0400 (EDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6930) AvroRpcEngine doesn't work with
 generated Avro code
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

AvroRpcEngine doesn't work with generated Avro code
---------------------------------------------------

                 Key: HADOOP-6930
                 URL: https://issues.apache.org/jira/browse/HADOOP-6930
             Project: Hadoop Common
          Issue Type: Bug
          Components: ipc
            Reporter: Sharad Agarwal


AvroRpcEngine uses 'reflect' based java implementation. There should be a way to have it work with 'specific' (generated code from avro idl).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10411-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 21:13:15 2010
Return-Path: <common-issues-return-10411-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78768 invoked from network); 26 Aug 2010 21:13:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 21:13:15 -0000
Received: (qmail 36163 invoked by uid 500); 26 Aug 2010 21:13:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36109 invoked by uid 500); 26 Aug 2010 21:13:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36101 invoked by uid 99); 26 Aug 2010 21:13:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 21:13:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 21:13:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7QLCr5Z022002
	for <common-issues@hadoop.apache.org>; Thu, 26 Aug 2010 21:12:53 GMT
Message-ID: <24985184.16041282857173665.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 17:12:53 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6931) Namenode start (init) fails because
 of invalid kerberos key, even when security set to "simple"
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Namenode start (init) fails because of invalid kerberos key, even when security set to "simple"
-----------------------------------------------------------------------------------------------

                 Key: HADOOP-6931
                 URL: https://issues.apache.org/jira/browse/HADOOP-6931
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Boris Shkolnik
             Fix For: 0.22.0


NameNode.initialize() calls login() method even when security set to simple

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10412-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 21:13:15 2010
Return-Path: <common-issues-return-10412-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78819 invoked from network); 26 Aug 2010 21:13:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 21:13:15 -0000
Received: (qmail 36327 invoked by uid 500); 26 Aug 2010 21:13:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36138 invoked by uid 500); 26 Aug 2010 21:13:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36110 invoked by uid 99); 26 Aug 2010 21:13:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 21:13:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 21:13:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7QLCsVk022011
	for <common-issues@hadoop.apache.org>; Thu, 26 Aug 2010 21:12:54 GMT
Message-ID: <4038300.16071282857174131.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 17:12:54 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6931) Namenode start (init) fails because
 of invalid kerberos key, even when security set to "simple"
In-Reply-To: <24985184.16041282857173665.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik reassigned HADOOP-6931:
--------------------------------------

    Assignee: Boris Shkolnik

> Namenode start (init) fails because of invalid kerberos key, even when security set to "simple"
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6931
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6931
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>             Fix For: 0.22.0
>
>
> NameNode.initialize() calls login() method even when security set to simple

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10413-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 21:15:34 2010
Return-Path: <common-issues-return-10413-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79714 invoked from network); 26 Aug 2010 21:15:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 21:15:34 -0000
Received: (qmail 39563 invoked by uid 500); 26 Aug 2010 21:15:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39525 invoked by uid 500); 26 Aug 2010 21:15:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39517 invoked by uid 99); 26 Aug 2010 21:15:33 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 21:15:33 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 21:15:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7QLEssm022058
	for <common-issues@hadoop.apache.org>; Thu, 26 Aug 2010 21:14:54 GMT
Message-ID: <28363713.16211282857294298.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 17:14:54 -0400 (EDT)
From: "Wouter de Bie (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6852) apparent bug in concatenated-bzip2
 support (decoding)
In-Reply-To: <503604.248281278541551065.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903063#action_12903063 ] 

Wouter de Bie commented on HADOOP-6852:
---------------------------------------

I'm having the same problem. We write files from our logging application using the BZip2Codec, but our application rotates files on an hourly basis. Next to that, when we flush log lines, we compress, which in turn causes different bzip2 block sizes.
Anyway, we're also getting the 'bad block header' exception.
Looking through some code and trying different things, we get things to work when we change the following in org.apache.hadoop.io.compress.bzip2.CBZip2InputStream:

{noformat}
  public CBZip2InputStream(final InputStream in) throws IOException {
    this(in, READ_MODE.CONTINUOUS);
  }
{noformat}

to

{noformat}
  public CBZip2InputStream(final InputStream in) throws IOException {
    this(in, READ_MODE.BYBLOCK);
  }
{noformat}

Which causes CBZip2InputStream to use the block size in initBlock(), instead of looking for the magic numbers.

I'll be digging into some code more tomorrow, but to quickly move forward on this issue, I would like to know why is CONTINUOUS the default mode and is there a place where the read mode is determined? BZip2 is block based, so why not default to that? 

I'll also try the above piece of code and see if that works with BYBLOCK.



> apparent bug in concatenated-bzip2 support (decoding)
> -----------------------------------------------------
>
>                 Key: HADOOP-6852
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6852
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.22.0
>         Environment: Linux x86_64 running 32-bit Hadoop, JDK 1.6.0_15
>            Reporter: Greg Roelofs
>
> The following simplified code (manually picked out of testMoreBzip2() in https://issues.apache.org/jira/secure/attachment/12448272/HADOOP-6835.v4.trunk-hadoop-mapreduce.patch) triggers a "java.io.IOException: bad block header" in org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.initBlock( CBZip2InputStream.java:527):
> {noformat}
>     JobConf jobConf = new JobConf(defaultConf);
>     CompressionCodec bzip2 = new BZip2Codec();
>     ReflectionUtils.setConf(bzip2, jobConf);
>     localFs.delete(workDir, true);
>     // copy multiple-member test file to HDFS
>     String fn2 = "testCompressThenConcat.txt" + bzip2.getDefaultExtension();
>     Path fnLocal2 = new Path(System.getProperty("test.concat.data","/tmp"),fn2);
>     Path fnHDFS2  = new Path(workDir, fn2);
>     localFs.copyFromLocalFile(fnLocal2, fnHDFS2);
>     FileInputFormat.setInputPaths(jobConf, workDir);
>     final FileInputStream in2 = new FileInputStream(fnLocal2.toString());
>     CompressionInputStream cin2 = bzip2.createInputStream(in2);
>     LineReader in = new LineReader(cin2);
>     Text out = new Text();
>     int numBytes, totalBytes=0, lineNum=0;
>     while ((numBytes = in.readLine(out)) > 0) {
>       ++lineNum;
>       totalBytes += numBytes;
>     }
>     in.close();
> {noformat}
> The specified file is also included in the H-6835 patch linked above, and some additional debug output is included in the commented-out test loop above.  (Only in the linked, "v4" version of the patch, however--I'm about to remove the debug stuff for checkin.)
> It's possible I've done something completely boneheaded here, but the file, at least, checks out in a subsequent set of subtests and with stock bzip2 itself.  Only the code above is problematic; it reads through the first concatenated chunk (17 lines of text) just fine but chokes on the header of the second one.  Altogether, the test file contains 84 lines of text and 4 concatenated bzip2 files.
> (It's possible this is a mapreduce issue rather than common, but note that the identical gzip test works fine.  Possibly it's related to the stream-vs-decompressor dichotomy, though; intentionally not supported?)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10414-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 21:17:54 2010
Return-Path: <common-issues-return-10414-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79894 invoked from network); 26 Aug 2010 21:17:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 21:17:53 -0000
Received: (qmail 40331 invoked by uid 500); 26 Aug 2010 21:17:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40283 invoked by uid 500); 26 Aug 2010 21:17:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40275 invoked by uid 99); 26 Aug 2010 21:17:53 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 21:17:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 21:17:53 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7QLHX4w022138
	for <common-issues@hadoop.apache.org>; Thu, 26 Aug 2010 21:17:33 GMT
Message-ID: <30462671.16391282857453085.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 17:17:33 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6918) Make metrics naming consistent
In-Reply-To: <15841216.511671282582816729.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903069#action_12903069 ] 

Hong Tang commented on HADOOP-6918:
-----------------------------------

Not a strong preference, but I like CapitalizedCamelCase better.

> Make metrics naming consistent
> ------------------------------
>
>                 Key: HADOOP-6918
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6918
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: metrics
>    Affects Versions: 0.20.2
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>
> While working HADOOP-6728, I noticed that our metrics naming style is all over the place:
> * Capitalized camel case: e.g., "FilesCreated" in namenode metrics and some rpc metrics
> * uncapitalized camel case: e.g, "threadsBlocked" in jvm metrics and some rpc metrics
> * lowercased underscored: e.g., "bytes_written" in datanode metrics and mapreduce metrics
> Let's make them consistent. How about uncapitalized camel case? My main reason for the camel case: some backends have limits on the name length and underscore is wasteful.
> Once we have a consistent naming style we can do:
> @Metric("Number of INodes created") MutableCounterLong filesCreated;
> instead of the more redundant:
> @Metric({"FilesCreated", "Number of INodes created"}) MutableCounterLong filesCreated;

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10415-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 23:00:35 2010
Return-Path: <common-issues-return-10415-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14054 invoked from network); 26 Aug 2010 23:00:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 23:00:35 -0000
Received: (qmail 60807 invoked by uid 500); 26 Aug 2010 23:00:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60747 invoked by uid 500); 26 Aug 2010 23:00:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60737 invoked by uid 99); 26 Aug 2010 23:00:34 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 23:00:34 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 23:00:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7QMxt5B023696
	for <common-issues@hadoop.apache.org>; Thu, 26 Aug 2010 22:59:55 GMT
Message-ID: <3422411.20081282863595693.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 18:59:55 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6930) AvroRpcEngine doesn't work with
 generated Avro code
In-Reply-To: <32541354.15611282856633769.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903131#action_12903131 ] 

Doug Cutting commented on HADOOP-6930:
--------------------------------------

Avro 1.4, which should be released next week, better integrates 'specific' and 'reflect'.  With a little more effort this could be make to 'just work', i.e., ReflectRequestor and ReflectResponder would be able to handle Specific data.  Glancing at the code, it looks like mostly what remains is fixing a few isX() methods in ReflectData so that unions would be dispatched correctly.  Making GenericArray to implement Collection would also simplify this.  Anyway, it's close.  If this sounds of interest, please file an Avro Jira requesting this.

Alternately, we could simply configure things, protocol-by-protocol to use either specific or reflect.  But intermixing might then cause problems, and it might be nice to be able to move more incrementally.

> AvroRpcEngine doesn't work with generated Avro code
> ---------------------------------------------------
>
>                 Key: HADOOP-6930
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6930
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>            Reporter: Sharad Agarwal
>
> AvroRpcEngine uses 'reflect' based java implementation. There should be a way to have it work with 'specific' (generated code from avro idl).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10416-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 23:27:34 2010
Return-Path: <common-issues-return-10416-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23091 invoked from network); 26 Aug 2010 23:27:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 23:27:34 -0000
Received: (qmail 89436 invoked by uid 500); 26 Aug 2010 23:27:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89378 invoked by uid 500); 26 Aug 2010 23:27:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89366 invoked by uid 99); 26 Aug 2010 23:27:33 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 23:27:33 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 23:27:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7QNQsHj023976
	for <common-issues@hadoop.apache.org>; Thu, 26 Aug 2010 23:26:54 GMT
Message-ID: <27600645.20481282865214558.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 19:26:54 -0400 (EDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6930) AvroRpcEngine doesn't work with
 generated Avro code
In-Reply-To: <32541354.15611282856633769.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903139#action_12903139 ] 

Sharad Agarwal commented on HADOOP-6930:
----------------------------------------

Having 'reflect' and 'specific' types as interoperable would be a great thing. But for now I am thinking of a simpler fix -> Have AvroSpecificRpcEngine (that extends AvroRpcEngine) and let AvroRpcEngine to have protected factory methods for creating Requestor and Responder.
Here clients HAVE to choose between 'reflect' and 'specific' Avro engine, but that should be ok for now.

> AvroRpcEngine doesn't work with generated Avro code
> ---------------------------------------------------
>
>                 Key: HADOOP-6930
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6930
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>            Reporter: Sharad Agarwal
>
> AvroRpcEngine uses 'reflect' based java implementation. There should be a way to have it work with 'specific' (generated code from avro idl).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10417-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 23:31:34 2010
Return-Path: <common-issues-return-10417-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25791 invoked from network); 26 Aug 2010 23:31:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 23:31:34 -0000
Received: (qmail 641 invoked by uid 500); 26 Aug 2010 23:31:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 389 invoked by uid 500); 26 Aug 2010 23:31:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 381 invoked by uid 99); 26 Aug 2010 23:31:33 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 23:31:33 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 23:31:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7QNUs1e023997
	for <common-issues@hadoop.apache.org>; Thu, 26 Aug 2010 23:30:54 GMT
Message-ID: <18024712.20531282865454199.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 19:30:54 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Moved: (HADOOP-6932) Namenode start (init) fails because of
 invalid kerberos key, even when security set to "simple"
In-Reply-To: <24985184.16041282857173665.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik moved HDFS-1358 to HADOOP-6932:
----------------------------------------------

          Project: Hadoop Common  (was: Hadoop HDFS)
              Key: HADOOP-6932  (was: HDFS-1358)
    Fix Version/s: 0.22.0
                       (was: 0.22.0)

> Namenode start (init) fails because of invalid kerberos key, even when security set to "simple"
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6932
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6932
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>             Fix For: 0.22.0
>
>         Attachments: HDFS-1358.patch
>
>
> NameNode.initialize() calls login() method even when security set to simple

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10418-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Aug 26 23:59:15 2010
Return-Path: <common-issues-return-10418-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31575 invoked from network); 26 Aug 2010 23:59:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Aug 2010 23:59:15 -0000
Received: (qmail 23763 invoked by uid 500); 26 Aug 2010 23:59:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23724 invoked by uid 500); 26 Aug 2010 23:59:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23716 invoked by uid 99); 26 Aug 2010 23:59:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 23:59:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Aug 2010 23:59:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7QNwr1H024133
	for <common-issues@hadoop.apache.org>; Thu, 26 Aug 2010 23:58:53 GMT
Message-ID: <1053376.20641282867133806.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 19:58:53 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6932) Namenode start (init) fails because
 of invalid kerberos key, even when security set to "simple"
In-Reply-To: <24985184.16041282857173665.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6932:
-----------------------------------

    Attachment: HADOOP-6932.patch

as per Kan's suggestion

> Namenode start (init) fails because of invalid kerberos key, even when security set to "simple"
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6932
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6932
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6932.patch, HDFS-1358.patch
>
>
> NameNode.initialize() calls login() method even when security set to simple

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10419-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 00:18:36 2010
Return-Path: <common-issues-return-10419-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38784 invoked from network); 27 Aug 2010 00:18:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 00:18:36 -0000
Received: (qmail 32790 invoked by uid 500); 27 Aug 2010 00:18:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32709 invoked by uid 500); 27 Aug 2010 00:18:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32701 invoked by uid 99); 27 Aug 2010 00:18:35 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 00:18:35 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 00:18:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7R0Hu7g024318
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 00:17:56 GMT
Message-ID: <23751331.20921282868276787.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 20:17:56 -0400 (EDT)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903150#action_12903150 ] 

Scott Carey commented on HADOOP-6884:
-------------------------------------

What has been shown is that over 80% of the performance problem, if it exists (I believe it does, but no one has actually shown that here or in the sibling tickets), is corrected by removing String concatenation in favor of varargs and Object[] creation.

So therefore Object[] creation is at least 5x faster than string concatenation.  Generally speaking, autoboxing and transient creation of Object[] would be similar in cost, so it is probably closer to 10x faster.

Pushing on the SLF4J project (which is actively developed, unlike Log4j and commons-logging, etc) to add a few method signatures that reduce Object[] creation and autoboxing feels like a better idea to me than re-inventing SLF4J -- which was created in the first place to solve this problem along with a few others.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10420-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 00:51:17 2010
Return-Path: <common-issues-return-10420-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50110 invoked from network); 27 Aug 2010 00:51:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 00:51:17 -0000
Received: (qmail 49216 invoked by uid 500); 27 Aug 2010 00:51:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49050 invoked by uid 500); 27 Aug 2010 00:51:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49042 invoked by uid 99); 27 Aug 2010 00:51:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 00:51:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 00:51:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7R0oti9024603
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 00:50:55 GMT
Message-ID: <1057729.21591282870255257.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 20:50:55 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903164#action_12903164 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6884:
------------------------------------------------

This issue draw up an unbelievably long discussion.  Erik's patch is great.  Of course, there is a room for improvement as always, and as many of other patches.  Quite a few people have proposed "better" ideas but only Erik has provided patches.  I have fine with ideas like Doug's or Konstantin's as long as some one is willing to work on it.  IMO, trying to make the logging perfect is a kind of wasting resources in this community.

Konstantin and I have +1'ed earlier but no one has  -1'ed yet.  I will commit Erik's patch 24 hours later if there is no -1.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10421-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 02:39:35 2010
Return-Path: <common-issues-return-10421-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73903 invoked from network); 27 Aug 2010 02:39:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 02:39:35 -0000
Received: (qmail 14200 invoked by uid 500); 27 Aug 2010 02:39:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14042 invoked by uid 500); 27 Aug 2010 02:39:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14033 invoked by uid 99); 27 Aug 2010 02:39:33 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 02:39:33 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 02:39:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7R2cr6g025359
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 02:38:53 GMT
Message-ID: <2573447.22781282876733425.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 22:38:53 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6879) Include Jsch jar to ivy dependencies
In-Reply-To: <11760779.565921279956469755.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6879?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6879:
---------------------------------------

              Summary: Include Jsch jar to ivy dependencies  (was: Jsch jar=
 is required for ivy resolution to be included in the build)
    Affects Version/s: 0.22.0
                           (was: 0.20.1)
          Description:=20
http://mvnrepository.com/
com.jcraft =C2=BB jsch=20
0.1.42 version needs to be included in the build. This is  needed to facili=
tate implementation of some system (Herriot) testcases .

Please include this in ivy.

jsch is originally located in http://www.jcraft.com/jsch/

  was:
http://mvnrepository.com/
com.jcraft =C2=BB jsch=20
0.1.42 version needs to be included in the build. This is  required for her=
riot testcases using jsch.

Please include this in ivy.

jsch is originally located in http://www.jcraft.com/jsch/

          Component/s: build
                       test

> Include Jsch jar to ivy dependencies
> ------------------------------------
>
>                 Key: HADOOP-6879
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6879
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>    Affects Versions: 0.22.0
>            Reporter: Iyappan Srinivasan
>            Assignee: Iyappan Srinivasan
>         Attachments: 6879-ydist-security-patch.txt, HADOOP-6879.patch, HA=
DOOP-6879.patch
>
>
> http://mvnrepository.com/
> com.jcraft =C2=BB jsch=20
> 0.1.42 version needs to be included in the build. This is  needed to faci=
litate implementation of some system (Herriot) testcases .
> Please include this in ivy.
> jsch is originally located in http://www.jcraft.com/jsch/

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10422-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 02:43:38 2010
Return-Path: <common-issues-return-10422-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74489 invoked from network); 27 Aug 2010 02:43:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 02:43:38 -0000
Received: (qmail 15724 invoked by uid 500); 27 Aug 2010 02:43:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15581 invoked by uid 500); 27 Aug 2010 02:43:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15573 invoked by uid 99); 27 Aug 2010 02:43:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 02:43:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 02:43:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7R2hFER025411
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 02:43:15 GMT
Message-ID: <19492703.22921282876995342.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 22:43:15 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6833) IPC leaks call parameters when
 exceptions thrown
In-Reply-To: <21655383.3791277174455275.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903195#action_12903195 ] 

Eli Collins commented on HADOOP-6833:
-------------------------------------

+1    

Here are the test-patch results. Not sure why it -1 javadoc, ant javadoc|javadoc-dev are clean.

{noformat}
     [exec] -1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
     [exec]                         Please justify why no new tests are needed for this patch.
     [exec]                         Also please list what manual steps were performed to verify this patch.
     [exec] 
     [exec]     -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
{noformat}


> IPC leaks call parameters when exceptions thrown
> ------------------------------------------------
>
>                 Key: HADOOP-6833
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6833
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2, 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Blocker
>         Attachments: hadoop-6833.txt, hadoop-6833.txt
>
>
> HADOOP-6498 moved the calls.remove() call lower into the SUCCESS clause of receiveResponse(), but didn't put a similar calls.remove into the ERROR clause. So, any RPC call that throws an exception ends up orphaning the Call object in the connection's "calls" hashtable. This prevents cleanup of the connection and is a memory leak for the call parameters.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10423-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 03:05:34 2010
Return-Path: <common-issues-return-10423-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81255 invoked from network); 27 Aug 2010 03:05:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 03:05:34 -0000
Received: (qmail 31398 invoked by uid 500); 27 Aug 2010 03:05:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31253 invoked by uid 500); 27 Aug 2010 03:05:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31245 invoked by uid 99); 27 Aug 2010 03:05:32 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 03:05:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 03:05:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7R34so4025549
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 03:04:54 GMT
Message-ID: <21629617.23001282878294125.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 23:04:54 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6833) IPC leaks call parameters when
 exceptions thrown
In-Reply-To: <21655383.3791277174455275.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6833:
--------------------------------

    Attachment: hadoop-6833-20.patch

Attaching patch for branch-0.20.

> IPC leaks call parameters when exceptions thrown
> ------------------------------------------------
>
>                 Key: HADOOP-6833
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6833
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2, 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Blocker
>         Attachments: hadoop-6833-20.patch, hadoop-6833.txt, hadoop-6833.txt
>
>
> HADOOP-6498 moved the calls.remove() call lower into the SUCCESS clause of receiveResponse(), but didn't put a similar calls.remove into the ERROR clause. So, any RPC call that throws an exception ends up orphaning the Call object in the connection's "calls" hashtable. This prevents cleanup of the connection and is a memory leak for the call parameters.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10424-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 03:08:35 2010
Return-Path: <common-issues-return-10424-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81574 invoked from network); 27 Aug 2010 03:08:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 03:08:34 -0000
Received: (qmail 34729 invoked by uid 500); 27 Aug 2010 03:08:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34587 invoked by uid 500); 27 Aug 2010 03:08:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34579 invoked by uid 99); 27 Aug 2010 03:08:32 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 03:08:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 03:08:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7R37rG6025573
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 03:07:54 GMT
Message-ID: <22586203.23051282878473916.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 23:07:53 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6833) IPC leaks call parameters when
 exceptions thrown
In-Reply-To: <21655383.3791277174455275.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6833:
--------------------------------

           Status: Resolved  (was: Patch Available)
     Hadoop Flags: [Reviewed]
    Fix Version/s: 0.20.3
                   0.21.1
                   0.22.0
       Resolution: Fixed

I committed this to 20, 21 and trunk.   Thanks Todd!

> IPC leaks call parameters when exceptions thrown
> ------------------------------------------------
>
>                 Key: HADOOP-6833
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6833
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2, 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.20.3, 0.21.1, 0.22.0
>
>         Attachments: hadoop-6833-20.patch, hadoop-6833.txt, hadoop-6833.txt
>
>
> HADOOP-6498 moved the calls.remove() call lower into the SUCCESS clause of receiveResponse(), but didn't put a similar calls.remove into the ERROR clause. So, any RPC call that throws an exception ends up orphaning the Call object in the connection's "calls" hashtable. This prevents cleanup of the connection and is a memory leak for the call parameters.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10425-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 03:20:36 2010
Return-Path: <common-issues-return-10425-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82637 invoked from network); 27 Aug 2010 03:20:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 03:20:36 -0000
Received: (qmail 41708 invoked by uid 500); 27 Aug 2010 03:20:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41566 invoked by uid 500); 27 Aug 2010 03:20:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41558 invoked by uid 99); 27 Aug 2010 03:20:33 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 03:20:33 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 03:20:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7R3JsQ2025646
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 03:19:54 GMT
Message-ID: <31729417.23151282879194018.JavaMail.jira@thor>
Date: Thu, 26 Aug 2010 23:19:54 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6833) IPC leaks call parameters when
 exceptions thrown
In-Reply-To: <21655383.3791277174455275.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903200#action_12903200 ] 

Hudson commented on HADOOP-6833:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #366 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/366/])
    HADOOP-6833. IPC leaks call parameters when exceptions thrown. Contributed by Todd Lipcon.


> IPC leaks call parameters when exceptions thrown
> ------------------------------------------------
>
>                 Key: HADOOP-6833
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6833
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2, 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.20.3, 0.21.1, 0.22.0
>
>         Attachments: hadoop-6833-20.patch, hadoop-6833.txt, hadoop-6833.txt
>
>
> HADOOP-6498 moved the calls.remove() call lower into the SUCCESS clause of receiveResponse(), but didn't put a similar calls.remove into the ERROR clause. So, any RPC call that throws an exception ends up orphaning the Call object in the connection's "calls" hashtable. This prevents cleanup of the connection and is a memory leak for the call parameters.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10426-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 04:03:37 2010
Return-Path: <common-issues-return-10426-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90985 invoked from network); 27 Aug 2010 04:03:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 04:03:36 -0000
Received: (qmail 79117 invoked by uid 500); 27 Aug 2010 04:03:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78946 invoked by uid 500); 27 Aug 2010 04:03:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78936 invoked by uid 99); 27 Aug 2010 04:03:32 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 04:03:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 04:03:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7R42snh026061
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 04:02:54 GMT
Message-ID: <20126466.23701282881774076.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 00:02:54 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6879) Include Jsch jar to ivy dependencies
In-Reply-To: <11760779.565921279956469755.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6879?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6879:
---------------------------------------

    Attachment: HADOOP-6879.patch

Moving over (and fixing issues in the original patch) proposed functionalit=
y from MAPREDUCE-1882 and refitting original patch for the trunk.

Remote exec API needs some extra work to be done on it especially in the pa=
rt of getting user's identities, but I just want to have it in the same pla=
ce for now.

> Include Jsch jar to ivy dependencies
> ------------------------------------
>
>                 Key: HADOOP-6879
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6879
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>    Affects Versions: 0.22.0
>            Reporter: Iyappan Srinivasan
>            Assignee: Iyappan Srinivasan
>         Attachments: 6879-ydist-security-patch.txt, HADOOP-6879.patch, HA=
DOOP-6879.patch, HADOOP-6879.patch
>
>
> http://mvnrepository.com/
> com.jcraft =C2=BB jsch=20
> 0.1.42 version needs to be included in the build. This is  needed to faci=
litate implementation of some system (Herriot) testcases .
> Please include this in ivy.
> jsch is originally located in http://www.jcraft.com/jsch/

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10427-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 04:11:19 2010
Return-Path: <common-issues-return-10427-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95909 invoked from network); 27 Aug 2010 04:11:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 04:11:19 -0000
Received: (qmail 82141 invoked by uid 500); 27 Aug 2010 04:11:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81938 invoked by uid 500); 27 Aug 2010 04:11:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81927 invoked by uid 99); 27 Aug 2010 04:11:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 04:11:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 04:11:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7R4Atx5026528
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 04:10:55 GMT
Message-ID: <32827257.23921282882255227.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 00:10:55 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6879) Provide SSH based (Jsch) remote
 execution API for system tests
In-Reply-To: <11760779.565921279956469755.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6879?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6879:
---------------------------------------

    Summary: Provide SSH based (Jsch) remote execution API for system tests=
  (was: Include Jsch jar to ivy dependencies)

> Provide SSH based (Jsch) remote execution API for system tests
> --------------------------------------------------------------
>
>                 Key: HADOOP-6879
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6879
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build, test
>    Affects Versions: 0.22.0
>            Reporter: Iyappan Srinivasan
>            Assignee: Iyappan Srinivasan
>         Attachments: 6879-ydist-security-patch.txt, HADOOP-6879.patch, HA=
DOOP-6879.patch, HADOOP-6879.patch
>
>
> http://mvnrepository.com/
> com.jcraft =C2=BB jsch=20
> 0.1.42 version needs to be included in the build. This is  needed to faci=
litate implementation of some system (Herriot) testcases .
> Please include this in ivy.
> jsch is originally located in http://www.jcraft.com/jsch/

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10428-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 09:31:23 2010
Return-Path: <common-issues-return-10428-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75066 invoked from network); 27 Aug 2010 09:31:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 09:31:22 -0000
Received: (qmail 39615 invoked by uid 500); 27 Aug 2010 09:31:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39454 invoked by uid 500); 27 Aug 2010 09:31:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39438 invoked by uid 99); 27 Aug 2010 09:31:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 09:31:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 09:31:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7R9Uw0L029889
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 09:30:58 GMT
Message-ID: <2387779.28891282901458214.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 05:30:58 -0400 (EDT)
From: "Alejandro Abdelnur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6894) Common foundation for Hadoop client
 tools
In-Reply-To: <10134473.113661280733856346.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated HADOOP-6894:
---------------------------------------

    Attachment: HadoopClientToolsV2_1.pdf

taken care of some typos.

clarifying that etc/ must be in the classpath

> Common foundation for Hadoop client tools
> -----------------------------------------
>
>                 Key: HADOOP-6894
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6894
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Alejandro Abdelnur
>         Attachments: HadoopClientTools.pdf, HadoopClientToolsV2.pdf, HadoopClientToolsV2_1.pdf
>
>
> As Hadoop widespreads and matures the number of tools and utilities for users keeps growing.
> Some of them are bundled with Hadoop core, some with Hadoop contrib, some on their own, some are full fledged servers on their own. For example, just to name a few: distcp, streaming, pipes, har, pig, hive, oozie.
> Today there is no standard mechanism for making these tools available to users. Neither there is a standard mechanism for these tools to integrate and distributed them with each other.
> The lack of a common foundation creates issues for developers and users.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10429-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 11:18:19 2010
Return-Path: <common-issues-return-10429-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5628 invoked from network); 27 Aug 2010 11:18:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 11:18:19 -0000
Received: (qmail 44893 invoked by uid 500); 27 Aug 2010 11:18:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44749 invoked by uid 500); 27 Aug 2010 11:18:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44741 invoked by uid 99); 27 Aug 2010 11:18:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 11:18:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 11:18:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7RBHsjL000795
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 11:17:54 GMT
Message-ID: <25149569.29721282907874020.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 07:17:54 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6833) IPC leaks call parameters when
 exceptions thrown
In-Reply-To: <21655383.3791277174455275.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903370#action_12903370 ] 

Hudson commented on HADOOP-6833:
--------------------------------

Integrated in Hadoop-Common-trunk #435 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/435/])
    HADOOP-6833. IPC leaks call parameters when exceptions thrown. Contributed by Todd Lipcon.


> IPC leaks call parameters when exceptions thrown
> ------------------------------------------------
>
>                 Key: HADOOP-6833
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6833
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2, 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Blocker
>             Fix For: 0.20.3, 0.21.1, 0.22.0
>
>         Attachments: hadoop-6833-20.patch, hadoop-6833.txt, hadoop-6833.txt
>
>
> HADOOP-6498 moved the calls.remove() call lower into the SUCCESS clause of receiveResponse(), but didn't put a similar calls.remove into the ERROR clause. So, any RPC call that throws an exception ends up orphaning the Call object in the connection's "calls" hashtable. This prevents cleanup of the connection and is a memory leak for the call parameters.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10430-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 16:46:23 2010
Return-Path: <common-issues-return-10430-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4609 invoked from network); 27 Aug 2010 16:46:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 16:46:22 -0000
Received: (qmail 14715 invoked by uid 500); 27 Aug 2010 16:46:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14074 invoked by uid 500); 27 Aug 2010 16:46:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13756 invoked by uid 99); 27 Aug 2010 16:46:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 16:46:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 16:46:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7RGjs4w004013
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 16:45:55 GMT
Message-ID: <85044.34231282927554950.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 12:45:54 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903473#action_12903473 ] 

Doug Cutting commented on HADOOP-6884:
--------------------------------------

-0.  This patch is not well motivated.  Reasonable alternatives have been proposed.  No urgency has been demonstrated.  No great effort was required in creating this patch: if it goes stale it would be easy to reconstruct.  Further effort to reach a solution acceptable to all before anything is committed seems well warranted.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10431-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 17:00:15 2010
Return-Path: <common-issues-return-10431-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6388 invoked from network); 27 Aug 2010 17:00:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 17:00:15 -0000
Received: (qmail 28160 invoked by uid 500); 27 Aug 2010 17:00:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28124 invoked by uid 500); 27 Aug 2010 17:00:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28116 invoked by uid 99); 27 Aug 2010 17:00:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 17:00:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 17:00:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7RGxs5Y004117
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 16:59:54 GMT
Message-ID: <25567829.34411282928393992.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 12:59:53 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6932) Namenode start (init) fails because
 of invalid kerberos key, even when security set to "simple"
In-Reply-To: <24985184.16041282857173665.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903479#action_12903479 ] 

Boris Shkolnik commented on HADOOP-6932:
----------------------------------------

ran ant test. All passed.

> Namenode start (init) fails because of invalid kerberos key, even when security set to "simple"
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6932
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6932
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6932.patch, HDFS-1358.patch
>
>
> NameNode.initialize() calls login() method even when security set to simple

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10432-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 17:03:34 2010
Return-Path: <common-issues-return-10432-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7410 invoked from network); 27 Aug 2010 17:03:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 17:03:33 -0000
Received: (qmail 34327 invoked by uid 500); 27 Aug 2010 17:03:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34266 invoked by uid 500); 27 Aug 2010 17:03:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34257 invoked by uid 99); 27 Aug 2010 17:03:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 17:03:33 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 17:03:32 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7RH3COT004241
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 17:03:12 GMT
Message-ID: <20414765.34601282928592479.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 13:03:12 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6930) AvroRpcEngine doesn't work with
 generated Avro code
In-Reply-To: <32541354.15611282856633769.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903483#action_12903483 ] 

Doug Cutting commented on HADOOP-6930:
--------------------------------------

> Here clients HAVE to choose between 'reflect' and 'specific' Avro engine, but that should be ok for now.

That would work if you only transition protocols to Specific and never wish to rely on Reflect.  Since some parameter types are used by multiple protocols, this requires you to transition more (all?) protocols at once, whereas, if some parameters could use Reflect, you could transition things from Writable to Avro protocol-by-protocol, which might be easier to debug.  But AvroSpecificRpcEngine should be easy to write, and, long-term, when all protocols use Specific, we'll want this, so it'd be a good addition.  +1

> AvroRpcEngine doesn't work with generated Avro code
> ---------------------------------------------------
>
>                 Key: HADOOP-6930
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6930
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>            Reporter: Sharad Agarwal
>
> AvroRpcEngine uses 'reflect' based java implementation. There should be a way to have it work with 'specific' (generated code from avro idl).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10433-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 18:47:35 2010
Return-Path: <common-issues-return-10433-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41665 invoked from network); 27 Aug 2010 18:47:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 18:47:35 -0000
Received: (qmail 90608 invoked by uid 500); 27 Aug 2010 18:47:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90525 invoked by uid 500); 27 Aug 2010 18:47:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90517 invoked by uid 99); 27 Aug 2010 18:47:34 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 18:47:34 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 18:47:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7RIkt3f005188
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 18:46:55 GMT
Message-ID: <18879677.36411282934815294.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 14:46:55 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903524#action_12903524 ] 

Erik Steffl commented on HADOOP-6884:
-------------------------------------

Doug, I don't see your point - the patch only makes the code consistent, as I mentioned previously lot of debug() calls are already guarded by isDebugEnabled().

If I started a new practice or tried to bring in new coding style/practice I would understand your point. But defending inconsistent code?

Patch might not be nice to look at but:

  - it's best from performance viewpoint

  - even though it adds code still LESS code is executed

  - code is less readable on small scale (individual statement is longer) but more readable on big scale - no surprises (all debug messages are same), easy to maintain (easy to find debug messages that are not guarded thus potential performance problems), it's obvious that debug message is optimal (don't need to investigate whether it's on critical path, whether it's good enough - in some cases the arguments themselves might be expensive, i.e. it's a function call or a new object is created)

  - it's relatively standard practice (i.e. java language doesn't offer much better solution, as seen by this long discussion)

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10434-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 19:05:17 2010
Return-Path: <common-issues-return-10434-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48634 invoked from network); 27 Aug 2010 19:05:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 19:05:17 -0000
Received: (qmail 13514 invoked by uid 500); 27 Aug 2010 19:05:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13489 invoked by uid 500); 27 Aug 2010 19:05:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13481 invoked by uid 99); 27 Aug 2010 19:05:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 19:05:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 19:05:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7RJ4tnX005469
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 19:04:55 GMT
Message-ID: <6888223.36921282935895862.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 15:04:55 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903534#action_12903534 ] 

Eli Collins commented on HADOOP-6884:
-------------------------------------

Seems like a version of the patch that only added the ifs statements when needed (when the log argument is not a constant string) would be a nice middle ground since it addresses both the performance and code readability concerns, and does not require a lot of effort. I think we should implement Konstantin/Doug's proposal in a separate jira and make forward progress on this one.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10435-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 19:07:35 2010
Return-Path: <common-issues-return-10435-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49026 invoked from network); 27 Aug 2010 19:07:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 19:07:34 -0000
Received: (qmail 16166 invoked by uid 500); 27 Aug 2010 19:07:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16120 invoked by uid 500); 27 Aug 2010 19:07:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16112 invoked by uid 99); 27 Aug 2010 19:07:34 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 19:07:34 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 19:07:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7RJ6tBl005517
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 19:06:55 GMT
Message-ID: <5423662.37051282936015196.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 15:06:55 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903536#action_12903536 ] 

Jakob Homan commented on HADOOP-6884:
-------------------------------------

bq. Konstantin and I have +1'ed earlier but no one has -1'ed yet. I will commit Erik's patch 24 hours later if there is no -1.
+1

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10436-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 20:21:19 2010
Return-Path: <common-issues-return-10436-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66600 invoked from network); 27 Aug 2010 20:21:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 20:21:19 -0000
Received: (qmail 2575 invoked by uid 500); 27 Aug 2010 20:21:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2513 invoked by uid 500); 27 Aug 2010 20:21:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2503 invoked by uid 99); 27 Aug 2010 20:21:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 20:21:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 20:21:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7RKKvsj006398
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 20:20:57 GMT
Message-ID: <14520609.38901282940457643.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 16:20:57 -0400 (EDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903586#action_12903586 ] 

Suresh Srinivas commented on HADOOP-6884:
-----------------------------------------

I had posted my +1 for the patch earlier. But echoing what Eli has said already, we should open another jira and explore some of the good ideas that this jira has generated.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10437-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 20:50:35 2010
Return-Path: <common-issues-return-10437-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75329 invoked from network); 27 Aug 2010 20:50:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 20:50:35 -0000
Received: (qmail 31030 invoked by uid 500); 27 Aug 2010 20:50:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30957 invoked by uid 500); 27 Aug 2010 20:50:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30947 invoked by uid 99); 27 Aug 2010 20:50:34 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 20:50:34 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 20:50:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7RKntnW006862
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 20:49:55 GMT
Message-ID: <7145927.40091282942195679.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 16:49:55 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6932) Namenode start (init) fails because
 of invalid kerberos key, even when security set to "simple"
In-Reply-To: <24985184.16041282857173665.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6932:
-----------------------------------

    Attachment: HADOOP-6932-1.patch

> Namenode start (init) fails because of invalid kerberos key, even when security set to "simple"
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6932
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6932
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6932-1.patch, HADOOP-6932.patch, HDFS-1358.patch
>
>
> NameNode.initialize() calls login() method even when security set to simple

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10438-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 20:56:16 2010
Return-Path: <common-issues-return-10438-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76150 invoked from network); 27 Aug 2010 20:56:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 20:56:16 -0000
Received: (qmail 37592 invoked by uid 500); 27 Aug 2010 20:56:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37479 invoked by uid 500); 27 Aug 2010 20:56:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37471 invoked by uid 99); 27 Aug 2010 20:56:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 20:56:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 20:56:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7RKtt9U006945
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 20:55:55 GMT
Message-ID: <5540863.40291282942554987.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 16:55:54 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6932) Namenode start (init) fails because
 of invalid kerberos key, even when security set to "simple"
In-Reply-To: <24985184.16041282857173665.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903623#action_12903623 ] 

Kan Zhang commented on HADOOP-6932:
-----------------------------------

Please make sure we don't break any existing code by throwing an exception when keytabFilename == null.
Otherwise, +1 on the patch.

> Namenode start (init) fails because of invalid kerberos key, even when security set to "simple"
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6932
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6932
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6932-1.patch, HADOOP-6932.patch, HDFS-1358.patch
>
>
> NameNode.initialize() calls login() method even when security set to simple

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10439-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 21:02:15 2010
Return-Path: <common-issues-return-10439-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78371 invoked from network); 27 Aug 2010 21:02:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 21:02:15 -0000
Received: (qmail 46966 invoked by uid 500); 27 Aug 2010 21:02:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46881 invoked by uid 500); 27 Aug 2010 21:02:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46873 invoked by uid 99); 27 Aug 2010 21:02:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 21:02:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 21:02:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7RL1sAU007104
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 21:01:54 GMT
Message-ID: <24618575.40521282942914108.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 17:01:54 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6932) Namenode start (init) fails because
 of invalid kerberos key, even when security set to "simple"
In-Reply-To: <24985184.16041282857173665.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903632#action_12903632 ] 

Kan Zhang commented on HADOOP-6932:
-----------------------------------

One minor nit, can we add a condition for keytabFilename.length() == 0 as well?

> Namenode start (init) fails because of invalid kerberos key, even when security set to "simple"
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6932
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6932
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6932-1.patch, HADOOP-6932.patch, HDFS-1358.patch
>
>
> NameNode.initialize() calls login() method even when security set to simple

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10440-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 21:33:39 2010
Return-Path: <common-issues-return-10440-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92464 invoked from network); 27 Aug 2010 21:33:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 21:33:35 -0000
Received: (qmail 82085 invoked by uid 500); 27 Aug 2010 21:33:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82051 invoked by uid 500); 27 Aug 2010 21:33:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82043 invoked by uid 99); 27 Aug 2010 21:33:34 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 21:33:34 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 21:33:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7RLWsru007546
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 21:32:54 GMT
Message-ID: <23470252.41481282944774366.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 17:32:54 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6907) Rpc client doesn't use the
 per-connection conf to figure out server's Kerberos principal
In-Reply-To: <9029429.243131281389838773.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903649#action_12903649 ] 

Hadoop QA commented on HADOOP-6907:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12452346/c6907-15.patch
  against trunk revision 989999.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/39/console

This message is automatically generated.

> Rpc client doesn't use the per-connection conf to figure out server's Kerberos principal
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6907
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6907
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6907-12.patch, c6907-15.patch
>
>
> Currently, RPC client caches the conf that was passed in to its constructor and uses that same conf (or values obtained from it) for every connection it sets up. This is not sufficient for security since each connection needs to figure out server's Kerberos principal on a per-connection basis. It's not reasonable to expect the first conf used by a user to contain all the Kerberos principals that her future connections will ever need. Or worse, if her first conf contains an incorrect principal name, it will prevent the user from connecting to the server even if she later on passes in a correct conf on retry (by calling RPC.getProxy()).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10441-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 21:48:34 2010
Return-Path: <common-issues-return-10441-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99392 invoked from network); 27 Aug 2010 21:48:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 21:48:34 -0000
Received: (qmail 98215 invoked by uid 500); 27 Aug 2010 21:48:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98151 invoked by uid 500); 27 Aug 2010 21:48:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98143 invoked by uid 99); 27 Aug 2010 21:48:33 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 21:48:33 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 21:48:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7RLlsHX007696
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 21:47:54 GMT
Message-ID: <18646833.41841282945674701.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 17:47:54 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6907) Rpc client doesn't use the
 per-connection conf to figure out server's Kerberos principal
In-Reply-To: <9029429.243131281389838773.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6907:
------------------------------

    Attachment: c6907-16.patch

an updated patch for current trunk.

> Rpc client doesn't use the per-connection conf to figure out server's Kerberos principal
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6907
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6907
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6907-12.patch, c6907-15.patch, c6907-16.patch
>
>
> Currently, RPC client caches the conf that was passed in to its constructor and uses that same conf (or values obtained from it) for every connection it sets up. This is not sufficient for security since each connection needs to figure out server's Kerberos principal on a per-connection basis. It's not reasonable to expect the first conf used by a user to contain all the Kerberos principals that her future connections will ever need. Or worse, if her first conf contains an incorrect principal name, it will prevent the user from connecting to the server even if she later on passes in a correct conf on retry (by calling RPC.getProxy()).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10442-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 21:52:16 2010
Return-Path: <common-issues-return-10442-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 735 invoked from network); 27 Aug 2010 21:52:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 21:52:16 -0000
Received: (qmail 6801 invoked by uid 500); 27 Aug 2010 21:52:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6746 invoked by uid 500); 27 Aug 2010 21:52:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6530 invoked by uid 99); 27 Aug 2010 21:52:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 21:52:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 21:52:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7RLpskv007820
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 21:51:54 GMT
Message-ID: <13098560.42201282945914619.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 17:51:54 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6913) Circular initialization between
 UserGroupInformation and KerberosName
In-Reply-To: <27327567.281671281550577971.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903663#action_12903663 ] 

Hadoop QA commented on HADOOP-6913:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12451830/c6913-01.patch
  against trunk revision 989999.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/40/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/40/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/40/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/40/console

This message is automatically generated.

> Circular initialization between UserGroupInformation and KerberosName
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6913
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6913
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6913-01.patch
>
>
> If the first call to UGI is UGI.setConfiguration(conf), it will try to initialize UGI class. During this initialization, the code calls KerberosName.setConfiguration(). KerberosName's static initializer will in turn call UGI.isSecurityEnabled(). Since UGI hasn't been completely initialized yet, isSecurityEnabled() will re-initialize UGI with a DEFAULT conf. As a result, the original conf used in UGI.setConfiguration(conf) will be overwritten by the DEFAULT conf.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10443-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 22:00:15 2010
Return-Path: <common-issues-return-10443-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1651 invoked from network); 27 Aug 2010 22:00:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 22:00:15 -0000
Received: (qmail 17163 invoked by uid 500); 27 Aug 2010 22:00:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17081 invoked by uid 500); 27 Aug 2010 22:00:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17073 invoked by uid 99); 27 Aug 2010 22:00:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 22:00:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 22:00:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7RLxrAr007863
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 21:59:54 GMT
Message-ID: <16067309.42251282946393661.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 17:59:53 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6913) Circular initialization between
 UserGroupInformation and KerberosName
In-Reply-To: <27327567.281671281550577971.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903665#action_12903665 ] 

Kan Zhang commented on HADOOP-6913:
-----------------------------------

The javadoc warnings are due to KerberosName.java and SecurityUtil.java and unrelated to this patch. TestBlockToken in HDFS has been failing because of this bug. It can serve as the test for this patch (once HDFS-1284 is committed).

> Circular initialization between UserGroupInformation and KerberosName
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6913
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6913
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6913-01.patch
>
>
> If the first call to UGI is UGI.setConfiguration(conf), it will try to initialize UGI class. During this initialization, the code calls KerberosName.setConfiguration(). KerberosName's static initializer will in turn call UGI.isSecurityEnabled(). Since UGI hasn't been completely initialized yet, isSecurityEnabled() will re-initialize UGI with a DEFAULT conf. As a result, the original conf used in UGI.setConfiguration(conf) will be overwritten by the DEFAULT conf.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10444-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 22:04:33 2010
Return-Path: <common-issues-return-10444-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2972 invoked from network); 27 Aug 2010 22:04:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 22:04:33 -0000
Received: (qmail 28098 invoked by uid 500); 27 Aug 2010 22:04:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28065 invoked by uid 500); 27 Aug 2010 22:04:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28057 invoked by uid 99); 27 Aug 2010 22:04:32 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 22:04:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 22:04:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7RM3rm0007947
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 22:03:53 GMT
Message-ID: <14615079.42351282946633507.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 18:03:53 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6933) TestListFiles is flaky
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

TestListFiles is flaky
----------------------

                 Key: HADOOP-6933
                 URL: https://issues.apache.org/jira/browse/HADOOP-6933
             Project: Hadoop Common
          Issue Type: Bug
          Components: test
    Affects Versions: 0.22.0
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
            Priority: Minor


TestListFiles assumes a particular order of the files returned by the directory iterator. There's no such guarantee made by the underlying API, so the test fails on some hosts.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10445-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 22:06:15 2010
Return-Path: <common-issues-return-10445-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7130 invoked from network); 27 Aug 2010 22:06:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 22:06:15 -0000
Received: (qmail 29229 invoked by uid 500); 27 Aug 2010 22:06:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29171 invoked by uid 500); 27 Aug 2010 22:06:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29155 invoked by uid 99); 27 Aug 2010 22:06:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 22:06:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 22:06:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7RM5rCW007958
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 22:05:53 GMT
Message-ID: <30293300.42381282946753839.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 18:05:53 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6933) TestListFiles is flaky
In-Reply-To: <14615079.42351282946633507.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6933:
--------------------------------

    Attachment: hadoop-6933.txt

> TestListFiles is flaky
> ----------------------
>
>                 Key: HADOOP-6933
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6933
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-6933.txt
>
>
> TestListFiles assumes a particular order of the files returned by the directory iterator. There's no such guarantee made by the underlying API, so the test fails on some hosts.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10446-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 22:06:16 2010
Return-Path: <common-issues-return-10446-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7136 invoked from network); 27 Aug 2010 22:06:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 22:06:15 -0000
Received: (qmail 29230 invoked by uid 500); 27 Aug 2010 22:06:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29174 invoked by uid 500); 27 Aug 2010 22:06:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29156 invoked by uid 99); 27 Aug 2010 22:06:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 22:06:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 22:06:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7RM5sNB007964
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 22:05:54 GMT
Message-ID: <10945063.42401282946754087.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 18:05:54 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6933) TestListFiles is flaky
In-Reply-To: <14615079.42351282946633507.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6933:
--------------------------------

    Status: Patch Available  (was: Open)

> TestListFiles is flaky
> ----------------------
>
>                 Key: HADOOP-6933
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6933
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-6933.txt
>
>
> TestListFiles assumes a particular order of the files returned by the directory iterator. There's no such guarantee made by the underlying API, so the test fails on some hosts.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10447-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 22:06:16 2010
Return-Path: <common-issues-return-10447-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7153 invoked from network); 27 Aug 2010 22:06:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 22:06:15 -0000
Received: (qmail 29478 invoked by uid 500); 27 Aug 2010 22:06:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29415 invoked by uid 500); 27 Aug 2010 22:06:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29218 invoked by uid 99); 27 Aug 2010 22:06:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 22:06:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 22:06:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7RM5saP007976
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 22:05:54 GMT
Message-ID: <6157153.42441282946754735.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 18:05:54 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6926) SocketInputStream incorrectly
 implements read()
In-Reply-To: <6335491.543431282689623460.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903667#action_12903667 ] 

Todd Lipcon commented on HADOOP-6926:
-------------------------------------

Ran unit tests locally. All passed except for TestListFiles (fixed by HADOOP-6933).

> SocketInputStream incorrectly implements read()
> -----------------------------------------------
>
>                 Key: HADOOP-6926
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6926
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.2, 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-6926.txt
>
>
> SocketInputStream's read() implementation doesn't upcast to int correctly, so it can't read bytes > 0x80. This is the same bug as HADOOP-6925, but in a different spot.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10448-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 22:08:33 2010
Return-Path: <common-issues-return-10448-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7432 invoked from network); 27 Aug 2010 22:08:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 22:08:33 -0000
Received: (qmail 32602 invoked by uid 500); 27 Aug 2010 22:08:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32571 invoked by uid 500); 27 Aug 2010 22:08:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32563 invoked by uid 99); 27 Aug 2010 22:08:32 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 22:08:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 22:08:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7RM7rp8007997
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 22:07:53 GMT
Message-ID: <32666694.42481282946873885.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 18:07:53 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6932) Namenode start (init) fails because
 of invalid kerberos key, even when security set to "simple"
In-Reply-To: <24985184.16041282857173665.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6932:
-----------------------------------

    Attachment: HADOOP-6932-2.patch

addressed Kan's comment and added test

> Namenode start (init) fails because of invalid kerberos key, even when security set to "simple"
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6932
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6932
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6932-1.patch, HADOOP-6932-2.patch, HADOOP-6932.patch, HDFS-1358.patch
>
>
> NameNode.initialize() calls login() method even when security set to simple

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10449-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 22:12:33 2010
Return-Path: <common-issues-return-10449-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7908 invoked from network); 27 Aug 2010 22:12:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 22:12:33 -0000
Received: (qmail 36921 invoked by uid 500); 27 Aug 2010 22:12:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36891 invoked by uid 500); 27 Aug 2010 22:12:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36883 invoked by uid 99); 27 Aug 2010 22:12:32 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 22:12:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 22:12:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7RMBsE5008027
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 22:11:54 GMT
Message-ID: <15843832.42541282947114032.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 18:11:54 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6932) Namenode start (init) fails because
 of invalid kerberos key, even when security set to "simple"
In-Reply-To: <24985184.16041282857173665.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903670#action_12903670 ] 

Kan Zhang commented on HADOOP-6932:
-----------------------------------

+1.

> Namenode start (init) fails because of invalid kerberos key, even when security set to "simple"
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6932
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6932
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6932-1.patch, HADOOP-6932-2.patch, HADOOP-6932.patch, HDFS-1358.patch
>
>
> NameNode.initialize() calls login() method even when security set to simple

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10450-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 22:28:18 2010
Return-Path: <common-issues-return-10450-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12237 invoked from network); 27 Aug 2010 22:28:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 22:28:18 -0000
Received: (qmail 58119 invoked by uid 500); 27 Aug 2010 22:28:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58045 invoked by uid 500); 27 Aug 2010 22:28:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57993 invoked by uid 99); 27 Aug 2010 22:28:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 22:28:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 22:28:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7RMRu9N008315
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 22:27:56 GMT
Message-ID: <2378960.43321282948076789.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 18:27:56 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6907) Rpc client doesn't use the
 per-connection conf to figure out server's Kerberos principal
In-Reply-To: <9029429.243131281389838773.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903684#action_12903684 ] 

Hadoop QA commented on HADOOP-6907:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12453285/c6907-16.patch
  against trunk revision 989999.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    -1 javac.  The applied patch generated 1025 javac compiler warnings (more than the trunk's current 1017 warnings).

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/41/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/41/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/41/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h9.grid.sp2.yahoo.net/41/console

This message is automatically generated.

> Rpc client doesn't use the per-connection conf to figure out server's Kerberos principal
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6907
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6907
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6907-12.patch, c6907-15.patch, c6907-16.patch
>
>
> Currently, RPC client caches the conf that was passed in to its constructor and uses that same conf (or values obtained from it) for every connection it sets up. This is not sufficient for security since each connection needs to figure out server's Kerberos principal on a per-connection basis. It's not reasonable to expect the first conf used by a user to contain all the Kerberos principals that her future connections will ever need. Or worse, if her first conf contains an incorrect principal name, it will prevent the user from connecting to the server even if she later on passes in a correct conf on retry (by calling RPC.getProxy()).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10451-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Aug 27 23:05:17 2010
Return-Path: <common-issues-return-10451-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22086 invoked from network); 27 Aug 2010 23:05:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Aug 2010 23:05:17 -0000
Received: (qmail 82323 invoked by uid 500); 27 Aug 2010 23:05:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82243 invoked by uid 500); 27 Aug 2010 23:05:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82233 invoked by uid 99); 27 Aug 2010 23:05:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 23:05:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Aug 2010 23:05:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7RN4tTq008617
	for <common-issues@hadoop.apache.org>; Fri, 27 Aug 2010 23:04:55 GMT
Message-ID: <16519120.43841282950295235.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 19:04:55 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6907) Rpc client doesn't use the
 per-connection conf to figure out server's Kerberos principal
In-Reply-To: <9029429.243131281389838773.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903698#action_12903698 ] 

Hadoop QA commented on HADOOP-6907:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12453285/c6907-16.patch
  against trunk revision 989999.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    -1 javac.  The applied patch generated 1025 javac compiler warnings (more than the trunk's current 1017 warnings).

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/664/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/664/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/664/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/664/console

This message is automatically generated.

> Rpc client doesn't use the per-connection conf to figure out server's Kerberos principal
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6907
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6907
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6907-12.patch, c6907-15.patch, c6907-16.patch
>
>
> Currently, RPC client caches the conf that was passed in to its constructor and uses that same conf (or values obtained from it) for every connection it sets up. This is not sufficient for security since each connection needs to figure out server's Kerberos principal on a per-connection basis. It's not reasonable to expect the first conf used by a user to contain all the Kerberos principals that her future connections will ever need. Or worse, if her first conf contains an incorrect principal name, it will prevent the user from connecting to the server even if she later on passes in a correct conf on retry (by calling RPC.getProxy()).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10452-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 28 00:33:17 2010
Return-Path: <common-issues-return-10452-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49874 invoked from network); 28 Aug 2010 00:33:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Aug 2010 00:33:17 -0000
Received: (qmail 29265 invoked by uid 500); 28 Aug 2010 00:33:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29140 invoked by uid 500); 28 Aug 2010 00:33:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29132 invoked by uid 99); 28 Aug 2010 00:33:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Aug 2010 00:33:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Aug 2010 00:33:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7S0Wt7U009530
	for <common-issues@hadoop.apache.org>; Sat, 28 Aug 2010 00:32:55 GMT
Message-ID: <2812059.45891282955575647.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 20:32:55 -0400 (EDT)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903734#action_12903734 ] 

Konstantin Shvachko commented on HADOOP-6884:
---------------------------------------------

I think Doug's request to provide benchmarks is reasonable. We should benchmark more.
So I modified {{NNThroughputBenchmark}}, to benchmark {{getFileStatus()}}. In current trunk {{getFileStatus()}} calls {{Groups.getGroups()}}, which has a LOG.debug() in it. In {{NNThroughputBenchmark}} you can specify a logLevel, so if I run it with {{logLevel = DEBUG}} the debug message from {{Groups.getGroups()}} is printed for every {{getFileStatus()}} call, and no other messages are printed.

I compared two versions of the code with {{logLevel = INFO}}:
# Unmodified {{Groups.getGroups()}}, where the argument for the debug call is calculated as a sum of strings.
# Modified {{Groups.getGroups()}} with {{if(LOG.isDebugEnabled())}} before the logging statement.

Results:
# 10,000 calls of {{getFileStatus()}} in (1) yield 22,905 ops/sec
 10,000 calls of {{getFileStatus()}} in (2) yield 24,610 ops/sec
 This about 7% difference.
# I increased the namespace to 100,000 files and performed that many calls of {{getFileStatus()}} for both setups. I still see 3-4% improvement in case (2).
  The gain is less because it is absorbed by the increased cost of navigating down the namespace tree.
# Did the same with 1,000 files. (2) is better about 12-13%.

{{Groups.getGroups()}} is a very popular method it is a part of user authentication, so this log message is a part of each and every name-node call. In many cases the cost of concatenating strings in it will be absorbed by disk io operations, but it is still good to have this improvement.

I'll post the patch for {{NNThroughputBenchmark}} in another jira shortly.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10453-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 28 00:46:35 2010
Return-Path: <common-issues-return-10453-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53383 invoked from network); 28 Aug 2010 00:46:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Aug 2010 00:46:34 -0000
Received: (qmail 37491 invoked by uid 500); 28 Aug 2010 00:46:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37448 invoked by uid 500); 28 Aug 2010 00:46:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37440 invoked by uid 99); 28 Aug 2010 00:46:33 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Aug 2010 00:46:33 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Aug 2010 00:46:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7S0jtWW009694
	for <common-issues@hadoop.apache.org>; Sat, 28 Aug 2010 00:45:55 GMT
Message-ID: <16421374.46311282956355414.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 20:45:55 -0400 (EDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903742#action_12903742 ] 

Doug Cutting commented on HADOOP-6884:
--------------------------------------

Konstantin, thanks very much for running this benchmark!  It sounds like that, or any other LOG.debug() call that might be called while servicing an otherwise inexpensive RPC request, should be guarded with an isDebugEnabled() or replaced with an slf4j or some other wrapper that avoids string concatenation.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10454-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 28 01:17:36 2010
Return-Path: <common-issues-return-10454-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59944 invoked from network); 28 Aug 2010 01:17:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Aug 2010 01:17:35 -0000
Received: (qmail 47869 invoked by uid 500); 28 Aug 2010 01:17:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47808 invoked by uid 500); 28 Aug 2010 01:17:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47800 invoked by uid 99); 28 Aug 2010 01:17:34 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Aug 2010 01:17:34 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Aug 2010 01:17:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7S1Gtii009892
	for <common-issues@hadoop.apache.org>; Sat, 28 Aug 2010 01:16:56 GMT
Message-ID: <14366721.46541282958215797.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 21:16:55 -0400 (EDT)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903747#action_12903747 ] 

Konstantin Shvachko commented on HADOOP-6884:
---------------------------------------------

I've got a feeling that Scott is very strong about moving Hadoop to SLF4J. This clearly does not fall within the scope of this issue as I explained before. But if you would like to invest in this subject you are welcome to create an issue and provide arguments for the transition, may be your experience with SLF4J in Avro, performance comparison, etc.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10455-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 28 01:19:18 2010
Return-Path: <common-issues-return-10455-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60247 invoked from network); 28 Aug 2010 01:19:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Aug 2010 01:19:18 -0000
Received: (qmail 49774 invoked by uid 500); 28 Aug 2010 01:19:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49628 invoked by uid 500); 28 Aug 2010 01:19:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49620 invoked by uid 99); 28 Aug 2010 01:19:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Aug 2010 01:19:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Aug 2010 01:19:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7S1Iu3j009957
	for <common-issues@hadoop.apache.org>; Sat, 28 Aug 2010 01:18:56 GMT
Message-ID: <6608521.46751282958336749.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 21:18:56 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903750#action_12903750 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6884:
------------------------------------------------

Thanks everyone for spending time on this issue.  Again, I do not opposite the other ideas but they are just outside the scope of this issue.  You are very welcome to work on them.

(Erik is running tests on his patch.)

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10456-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 28 01:23:15 2010
Return-Path: <common-issues-return-10456-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60594 invoked from network); 28 Aug 2010 01:23:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Aug 2010 01:23:15 -0000
Received: (qmail 50186 invoked by uid 500); 28 Aug 2010 01:23:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50109 invoked by uid 500); 28 Aug 2010 01:23:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50101 invoked by uid 99); 28 Aug 2010 01:23:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Aug 2010 01:23:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Aug 2010 01:23:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7S1Mrmo009976
	for <common-issues@hadoop.apache.org>; Sat, 28 Aug 2010 01:22:54 GMT
Message-ID: <8425330.46781282958573872.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 21:22:53 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6933) TestListFiles is flaky
In-Reply-To: <14615079.42351282946633507.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6933:
-------------------------------------------

    Hadoop Flags: [Reviewed]

+1 patch looks good.

> TestListFiles is flaky
> ----------------------
>
>                 Key: HADOOP-6933
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6933
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>         Attachments: hadoop-6933.txt
>
>
> TestListFiles assumes a particular order of the files returned by the directory iterator. There's no such guarantee made by the underlying API, so the test fails on some hosts.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10457-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 28 02:05:35 2010
Return-Path: <common-issues-return-10457-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69148 invoked from network); 28 Aug 2010 02:05:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Aug 2010 02:05:35 -0000
Received: (qmail 64621 invoked by uid 500); 28 Aug 2010 02:05:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64601 invoked by uid 500); 28 Aug 2010 02:05:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64587 invoked by uid 99); 28 Aug 2010 02:05:34 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Aug 2010 02:05:34 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Aug 2010 02:05:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7S24tvL010232
	for <common-issues@hadoop.apache.org>; Sat, 28 Aug 2010 02:04:55 GMT
Message-ID: <30933128.47151282961095267.JavaMail.jira@thor>
Date: Fri, 27 Aug 2010 22:04:55 -0400 (EDT)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903759#action_12903759 ] 

Scott Carey commented on HADOOP-6884:
-------------------------------------

bq.  I've got a feeling that Scott is very strong about moving Hadoop to SLF4J.

Moreso that if there was going to be a lot of work to build custom wrappers and change logging signatures across the code-base, that SLF4J should be a strong contender since that is what it does.   
If building such a thing is outside the scope here, then guard statements surely work.

In a blue-sky world, SLF4J (or some framework) would modify bytecode to push its guard statement from inside its log method to around it at call sites.  But now I'm just dreaming.

I have had good experience with a proprietary logging wrapper that lowered string concatenation churn caused by method invocations in the past -- many years before SLF4J did it -- and with an additional 200 or so method signatures to avoid autoboxing in 99% of use cases.  A bit ugly, but very effective and we didn't have to teach developers when or why to guard their logging statements.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10458-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 28 16:52:34 2010
Return-Path: <common-issues-return-10458-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51146 invoked from network); 28 Aug 2010 16:52:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Aug 2010 16:52:34 -0000
Received: (qmail 95799 invoked by uid 500); 28 Aug 2010 16:52:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95725 invoked by uid 500); 28 Aug 2010 16:52:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95717 invoked by uid 99); 28 Aug 2010 16:52:32 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Aug 2010 16:52:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Aug 2010 16:52:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7SGpr1r014979
	for <common-issues@hadoop.apache.org>; Sat, 28 Aug 2010 16:51:54 GMT
Message-ID: <28049262.51361283014313963.JavaMail.jira@thor>
Date: Sat, 28 Aug 2010 12:51:53 -0400 (EDT)
From: "Johannes Zillmann (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6928) Fix BooleanWritable comparator in
 0.20
In-Reply-To: <32649298.8101282841573990.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Johannes Zillmann updated HADOOP-6928:
--------------------------------------

    Attachment: h-6928-2.patch

added [^h-6928-2.patch] wich includes a test

> Fix BooleanWritable comparator in 0.20
> --------------------------------------
>
>                 Key: HADOOP-6928
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6928
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.0, 0.20.1, 0.20.2
>            Reporter: Owen O'Malley
>             Fix For: 0.20.3
>
>         Attachments: h-6928-2.patch, h-6928.patch
>
>
> The RawComparator for BooleanWritable was fixed as part of HADOOP-5699 in 0.21 and trunk. The fix should be pushed back into 0.20.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10459-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 28 17:50:33 2010
Return-Path: <common-issues-return-10459-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61894 invoked from network); 28 Aug 2010 17:50:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Aug 2010 17:50:33 -0000
Received: (qmail 13256 invoked by uid 500); 28 Aug 2010 17:50:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13165 invoked by uid 500); 28 Aug 2010 17:50:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13157 invoked by uid 99); 28 Aug 2010 17:50:32 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Aug 2010 17:50:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Aug 2010 17:50:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7SHnrbE015230
	for <common-issues@hadoop.apache.org>; Sat, 28 Aug 2010 17:49:53 GMT
Message-ID: <11925651.51571283017793361.JavaMail.jira@thor>
Date: Sat, 28 Aug 2010 13:49:53 -0400 (EDT)
From: "Johannes Zillmann (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6934) test for ByteWritable comparator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

test for ByteWritable comparator
--------------------------------

                 Key: HADOOP-6934
                 URL: https://issues.apache.org/jira/browse/HADOOP-6934
             Project: Hadoop Common
          Issue Type: Test
          Components: record
    Affects Versions: 0.21.0
            Reporter: Johannes Zillmann
             Fix For: 0.21.1


The test for the ByteWritable comparator bug (see HADOOP-6928), which is already fixed in 0.21.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10460-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 28 17:56:15 2010
Return-Path: <common-issues-return-10460-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62563 invoked from network); 28 Aug 2010 17:56:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Aug 2010 17:56:15 -0000
Received: (qmail 17319 invoked by uid 500); 28 Aug 2010 17:56:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17282 invoked by uid 500); 28 Aug 2010 17:56:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17274 invoked by uid 99); 28 Aug 2010 17:56:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Aug 2010 17:56:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Aug 2010 17:56:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7SHtrCQ015276
	for <common-issues@hadoop.apache.org>; Sat, 28 Aug 2010 17:55:53 GMT
Message-ID: <9394300.51661283018153204.JavaMail.jira@thor>
Date: Sat, 28 Aug 2010 13:55:53 -0400 (EDT)
From: "Johannes Zillmann (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6934) test for ByteWritable comparator
In-Reply-To: <11925651.51571283017793361.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Johannes Zillmann updated HADOOP-6934:
--------------------------------------

    Attachment: h-6934.patch

Patch [^h-6934.patch] with the test attached.

> test for ByteWritable comparator
> --------------------------------
>
>                 Key: HADOOP-6934
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6934
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: record
>    Affects Versions: 0.21.0
>            Reporter: Johannes Zillmann
>             Fix For: 0.21.1
>
>         Attachments: h-6934.patch
>
>
> The test for the ByteWritable comparator bug (see HADOOP-6928), which is already fixed in 0.21.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10461-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 28 17:58:33 2010
Return-Path: <common-issues-return-10461-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62762 invoked from network); 28 Aug 2010 17:58:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Aug 2010 17:58:33 -0000
Received: (qmail 18700 invoked by uid 500); 28 Aug 2010 17:58:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18645 invoked by uid 500); 28 Aug 2010 17:58:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18637 invoked by uid 99); 28 Aug 2010 17:58:32 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Aug 2010 17:58:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Aug 2010 17:58:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7SHvrtX015290
	for <common-issues@hadoop.apache.org>; Sat, 28 Aug 2010 17:57:53 GMT
Message-ID: <1147705.51681283018273516.JavaMail.jira@thor>
Date: Sat, 28 Aug 2010 13:57:53 -0400 (EDT)
From: "Johannes Zillmann (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6934) test for ByteWritable comparator
In-Reply-To: <11925651.51571283017793361.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Johannes Zillmann updated HADOOP-6934:
--------------------------------------

    Status: Patch Available  (was: Open)

> test for ByteWritable comparator
> --------------------------------
>
>                 Key: HADOOP-6934
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6934
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: record
>    Affects Versions: 0.21.0
>            Reporter: Johannes Zillmann
>             Fix For: 0.21.1
>
>         Attachments: h-6934.patch
>
>
> The test for the ByteWritable comparator bug (see HADOOP-6928), which is already fixed in 0.21.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10462-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 28 19:23:20 2010
Return-Path: <common-issues-return-10462-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92614 invoked from network); 28 Aug 2010 19:23:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Aug 2010 19:23:20 -0000
Received: (qmail 69143 invoked by uid 500); 28 Aug 2010 19:23:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69070 invoked by uid 500); 28 Aug 2010 19:23:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69062 invoked by uid 99); 28 Aug 2010 19:23:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Aug 2010 19:23:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Aug 2010 19:23:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7SJMtrg015952
	for <common-issues@hadoop.apache.org>; Sat, 28 Aug 2010 19:22:55 GMT
Message-ID: <6907411.52831283023375222.JavaMail.jira@thor>
Date: Sat, 28 Aug 2010 15:22:55 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903877#action_12903877 ] 

Erik Steffl commented on HADOOP-6884:
-------------------------------------

HADOOP-6884-0.22-1.patch 'ant test' and 'ant test-patch' results (ran manually because Hudson is down).

Results of 'ant test':

BUILD SUCCESSFUL
Total time: 6 minutes 44 seconds

Results of 'ant test-patch':

     [exec] There appear to be 2 release audit warnings before the patch and 2 release audit warnings after applying the patch.
     [exec]
     [exec]
     [exec]
     [exec]
     [exec] -1 overall.
     [exec]
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec]
     [exec]     +1 tests included.  The patch appears to include 18 new or modified tests.
     [exec]
     [exec]     -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.
     [exec]
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec]
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec]
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec]

Javadoc warnings are unrelated to the patch:

     [exec]   [javadoc] Constructing Javadoc information...
     [exec]   [javadoc] /home/steffl/work/svn.isDebugEnabled/common-trunk/src/java/org/apache/hadoop/security/SecurityUtil.java:39: warning: sun.security.jgss.krb5.Krb5Util is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.jgss.krb5.Krb5Util;
     [exec]   [javadoc]                              ^
     [exec]   [javadoc] /home/steffl/work/svn.isDebugEnabled/common-trunk/src/java/org/apache/hadoop/security/SecurityUtil.java:40: warning: sun.security.krb5.Credentials is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.Credentials;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /home/steffl/work/svn.isDebugEnabled/common-trunk/src/java/org/apache/hadoop/security/SecurityUtil.java:41: warning: sun.security.krb5.PrincipalName is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.PrincipalName;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /home/steffl/work/svn.isDebugEnabled/common-trunk/src/java/org/apache/hadoop/security/KerberosName.java:31: warning: sun.security.krb5.Config is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.Config;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /home/steffl/work/svn.isDebugEnabled/common-trunk/src/java/org/apache/hadoop/security/KerberosName.java:32: warning: sun.security.krb5.KrbException is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.KrbException;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /home/steffl/work/svn.isDebugEnabled/common-trunk/src/java/org/apache/hadoop/security/KerberosName.java:81: warning: sun.security.krb5.Config is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc]   private static Config kerbConf;
     [exec]   [javadoc]                  ^
     [exec]   [javadoc] ExcludePrivateAnnotationsStandardDoclet
     [exec]   [javadoc] Standard Doclet version 1.6.0_06
     [exec]   [javadoc] Building tree for all the packages and classes...
     [exec]   [javadoc] Building index for all the packages and classes...
     [exec]   [javadoc] Building index for all classes...
     [exec]   [javadoc] Generating /home/steffl/work/svn.isDebugEnabled/common-trunk/build/docs/api/stylesheet.css...
     [exec]   [javadoc] 6 warnings


> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10463-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 28 21:31:18 2010
Return-Path: <common-issues-return-10463-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26392 invoked from network); 28 Aug 2010 21:31:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Aug 2010 21:31:18 -0000
Received: (qmail 26173 invoked by uid 500); 28 Aug 2010 21:31:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26103 invoked by uid 500); 28 Aug 2010 21:31:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26083 invoked by uid 99); 28 Aug 2010 21:31:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Aug 2010 21:31:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Aug 2010 21:31:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7SLUret016635
	for <common-issues@hadoop.apache.org>; Sat, 28 Aug 2010 21:30:53 GMT
Message-ID: <24780218.53611283031053767.JavaMail.jira@thor>
Date: Sat, 28 Aug 2010 17:30:53 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6928) Fix BooleanWritable comparator in
 0.20
In-Reply-To: <32649298.8101282841573990.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903889#action_12903889 ] 

Eli Collins commented on HADOOP-6928:
-------------------------------------

+1

Here are test-patch results from branch-0.20.  I think the eclipse classpath error is bogus.

{noformat}
     [exec] -1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 2 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     -1 Eclipse classpath. The patch causes the Eclipse classpath to differ from the contents of the lib directories.
{noformat}

> Fix BooleanWritable comparator in 0.20
> --------------------------------------
>
>                 Key: HADOOP-6928
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6928
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.0, 0.20.1, 0.20.2
>            Reporter: Owen O'Malley
>             Fix For: 0.20.3
>
>         Attachments: h-6928-2.patch, h-6928.patch
>
>
> The RawComparator for BooleanWritable was fixed as part of HADOOP-5699 in 0.21 and trunk. The fix should be pushed back into 0.20.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10464-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 28 22:50:20 2010
Return-Path: <common-issues-return-10464-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40120 invoked from network); 28 Aug 2010 22:50:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Aug 2010 22:50:19 -0000
Received: (qmail 53571 invoked by uid 500); 28 Aug 2010 22:50:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53500 invoked by uid 500); 28 Aug 2010 22:50:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53492 invoked by uid 99); 28 Aug 2010 22:50:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Aug 2010 22:50:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Aug 2010 22:50:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7SMnvn0017097
	for <common-issues@hadoop.apache.org>; Sat, 28 Aug 2010 22:49:58 GMT
Message-ID: <7941574.54371283035797920.JavaMail.jira@thor>
Date: Sat, 28 Aug 2010 18:49:57 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6884:
-------------------------------------------

        Status: Resolved  (was: Patch Available)
    Resolution: Fixed

I have committed this.  Thanks, Erik!

Thanks everyone again.

> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10465-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Aug 28 23:10:35 2010
Return-Path: <common-issues-return-10465-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46111 invoked from network); 28 Aug 2010 23:10:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 28 Aug 2010 23:10:35 -0000
Received: (qmail 60733 invoked by uid 500); 28 Aug 2010 23:10:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60679 invoked by uid 500); 28 Aug 2010 23:10:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60671 invoked by uid 99); 28 Aug 2010 23:10:34 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Aug 2010 23:10:34 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Aug 2010 23:10:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7SN9u0r017250
	for <common-issues@hadoop.apache.org>; Sat, 28 Aug 2010 23:09:56 GMT
Message-ID: <3055809.54621283036996068.JavaMail.jira@thor>
Date: Sat, 28 Aug 2010 19:09:56 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903912#action_12903912 ] 

Hudson commented on HADOOP-6884:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #367 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/367/])
    HADOOP-6884. Add LOG.isDebugEnabled() guard for each LOG.debug(..).  Contributed by Erik Steffl


> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10466-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 29 11:18:24 2010
Return-Path: <common-issues-return-10466-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20657 invoked from network); 29 Aug 2010 11:18:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Aug 2010 11:18:24 -0000
Received: (qmail 82685 invoked by uid 500); 29 Aug 2010 11:18:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82528 invoked by uid 500); 29 Aug 2010 11:18:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82520 invoked by uid 99); 29 Aug 2010 11:18:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 Aug 2010 11:18:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 Aug 2010 11:18:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7TBHtIj009206
	for <common-issues@hadoop.apache.org>; Sun, 29 Aug 2010 11:17:55 GMT
Message-ID: <5308420.57331283080675636.JavaMail.jira@thor>
Date: Sun, 29 Aug 2010 07:17:55 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6884) Add LOG.isDebugEnabled() guard for
 each LOG.debug("...")
In-Reply-To: <8688001.34681280270784520.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903956#action_12903956 ] 

Hudson commented on HADOOP-6884:
--------------------------------

Integrated in Hadoop-Common-trunk #437 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/437/])
    HADOOP-6884. Add LOG.isDebugEnabled() guard for each LOG.debug(..).  Contributed by Erik Steffl


> Add LOG.isDebugEnabled() guard for each LOG.debug("...")
> --------------------------------------------------------
>
>                 Key: HADOOP-6884
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6884
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.22.0
>
>         Attachments: FunAgain.java, FunAgain.java, HADOOP-6884-0.22-1.patch, HADOOP-6884-0.22.patch
>
>
> Each LOG.debug("...") should be executed only if LOG.isDebugEnabled() is true, in some cases it's expensive to construct the string that is being printed to log. It's much easier to always use LOG.isDebugEnabled() because it's easier to check (rather than in each case reason whether it's necessary or not).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10467-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 29 18:24:16 2010
Return-Path: <common-issues-return-10467-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18541 invoked from network); 29 Aug 2010 18:24:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 29 Aug 2010 18:24:16 -0000
Received: (qmail 46677 invoked by uid 500); 29 Aug 2010 18:24:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46542 invoked by uid 500); 29 Aug 2010 18:24:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46534 invoked by uid 99); 29 Aug 2010 18:24:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 Aug 2010 18:24:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 29 Aug 2010 18:24:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7TINsVl029763
	for <common-issues@hadoop.apache.org>; Sun, 29 Aug 2010 18:23:54 GMT
Message-ID: <18014567.59991283106234400.JavaMail.jira@thor>
Date: Sun, 29 Aug 2010 14:23:54 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6851) Fix '$bin' path duplication in
 setup scripts
In-Reply-To: <13265146.224771278452993844.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904007#action_12904007 ] 

Eli Collins commented on HADOOP-6851:
-------------------------------------

Isn't this a dupe of HADOOP-6101? I don't see how the bin scripts hit this since they don't appear to use relative paths.

> Fix '$bin' path duplication in setup scripts
> --------------------------------------------
>
>                 Key: HADOOP-6851
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6851
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>            Reporter: Nicolas Spiegelberg
>            Priority: Trivial
>             Fix For: 0.21.1, 0.22.0, 0.20-append
>
>         Attachments: HADOOP-6851-0.22.patch
>
>
> I have my bash environment setup to echo absolute pathnames when a relative one is specified in 'cd'. This caused problems with all the Hadoop bash scripts because the script accidentally sets the $bin variable twice in this setup. (e.g. would set $bin="/path/bin/hadoop\n/path/bin/hadoop").
> This jira is for common scripts.  I filed a separate jira for HDFS scripts, which share the same pattern.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10468-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 01:39:16 2010
Return-Path: <common-issues-return-10468-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53702 invoked from network); 30 Aug 2010 01:39:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 01:39:16 -0000
Received: (qmail 62053 invoked by uid 500); 30 Aug 2010 01:39:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61979 invoked by uid 500); 30 Aug 2010 01:39:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61971 invoked by uid 99); 30 Aug 2010 01:39:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 01:39:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 01:39:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7U1ctcV002015
	for <common-issues@hadoop.apache.org>; Mon, 30 Aug 2010 01:38:55 GMT
Message-ID: <18648471.62311283132335131.JavaMail.jira@thor>
Date: Sun, 29 Aug 2010 21:38:55 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6786) test-patch needs to verify Herriot
 integrity
In-Reply-To: <19336182.46451274829212745.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6786:
---------------------------------------

    Fix Version/s: 0.21.1

> test-patch needs to verify Herriot integrity
> --------------------------------------------
>
>                 Key: HADOOP-6786
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6786
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>             Fix For: 0.21.1
>
>         Attachments: HADOOP-6786.patch, HADOOP-6786.patch, HADOOP-6786.patch
>
>
> Whenever a new patch is submitted for verification {{test-patch}} process has to make sure that none of Herriot bindings were broken.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10469-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 01:39:34 2010
Return-Path: <common-issues-return-10469-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53793 invoked from network); 30 Aug 2010 01:39:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 01:39:34 -0000
Received: (qmail 62309 invoked by uid 500); 30 Aug 2010 01:39:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62253 invoked by uid 500); 30 Aug 2010 01:39:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62241 invoked by uid 99); 30 Aug 2010 01:39:33 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 01:39:33 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 01:39:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7U1csFm002003
	for <common-issues@hadoop.apache.org>; Mon, 30 Aug 2010 01:38:54 GMT
Message-ID: <21479850.62271283132334216.JavaMail.jira@thor>
Date: Sun, 29 Aug 2010 21:38:54 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6786) test-patch needs to verify Herriot
 integrity
In-Reply-To: <19336182.46451274829212745.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6786:
---------------------------------------

        Status: Resolved  (was: Patch Available)
    Resolution: Fixed

I have committed this to 0.21 and trunk.

> test-patch needs to verify Herriot integrity
> --------------------------------------------
>
>                 Key: HADOOP-6786
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6786
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: HADOOP-6786.patch, HADOOP-6786.patch, HADOOP-6786.patch
>
>
> Whenever a new patch is submitted for verification {{test-patch}} process has to make sure that none of Herriot bindings were broken.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10470-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 02:04:16 2010
Return-Path: <common-issues-return-10470-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58112 invoked from network); 30 Aug 2010 02:04:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 02:04:15 -0000
Received: (qmail 75437 invoked by uid 500); 30 Aug 2010 02:04:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75385 invoked by uid 500); 30 Aug 2010 02:04:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75377 invoked by uid 99); 30 Aug 2010 02:04:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 02:04:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 02:04:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7U23sQ9002198
	for <common-issues@hadoop.apache.org>; Mon, 30 Aug 2010 02:03:54 GMT
Message-ID: <15370439.62481283133834074.JavaMail.jira@thor>
Date: Sun, 29 Aug 2010 22:03:54 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6786) test-patch needs to verify Herriot
 integrity
In-Reply-To: <19336182.46451274829212745.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904065#action_12904065 ] 

Hudson commented on HADOOP-6786:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #368 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/368/])
    HADOOP-6786. test-patch needs to verify Herriot integrity. Contributed by Konstantin Boudnik


> test-patch needs to verify Herriot integrity
> --------------------------------------------
>
>                 Key: HADOOP-6786
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6786
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>             Fix For: 0.21.1
>
>         Attachments: HADOOP-6786.patch, HADOOP-6786.patch, HADOOP-6786.patch
>
>
> Whenever a new patch is submitted for verification {{test-patch}} process has to make sure that none of Herriot bindings were broken.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10471-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 02:57:58 2010
Return-Path: <common-issues-return-10471-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68731 invoked from network); 30 Aug 2010 02:57:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 02:57:57 -0000
Received: (qmail 95960 invoked by uid 500); 30 Aug 2010 02:57:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95818 invoked by uid 500); 30 Aug 2010 02:57:56 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95810 invoked by uid 99); 30 Aug 2010 02:57:55 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 02:57:55 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 02:57:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7U2vFct002388
	for <common-issues@hadoop.apache.org>; Mon, 30 Aug 2010 02:57:15 GMT
Message-ID: <21789388.62631283137035296.JavaMail.jira@thor>
Date: Sun, 29 Aug 2010 22:57:15 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6786) test-patch needs to verify Herriot
 integrity
In-Reply-To: <19336182.46451274829212745.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904069#action_12904069 ] 

Hudson commented on HADOOP-6786:
--------------------------------

Integrated in Hadoop-Hdfs-trunk-Commit #377 (See [https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/377/])
    HADOOP-6786. test-patch needs to verify Herriot integrity. Contributed by Konstantin Boudnik


> test-patch needs to verify Herriot integrity
> --------------------------------------------
>
>                 Key: HADOOP-6786
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6786
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>             Fix For: 0.21.1
>
>         Attachments: HADOOP-6786.patch, HADOOP-6786.patch, HADOOP-6786.patch
>
>
> Whenever a new patch is submitted for verification {{test-patch}} process has to make sure that none of Herriot bindings were broken.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10472-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 09:32:22 2010
Return-Path: <common-issues-return-10472-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81075 invoked from network); 30 Aug 2010 09:32:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 09:32:22 -0000
Received: (qmail 6218 invoked by uid 500); 30 Aug 2010 09:32:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6060 invoked by uid 500); 30 Aug 2010 09:32:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6041 invoked by uid 99); 30 Aug 2010 09:32:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 09:32:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 09:32:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7U9VwkZ005387
	for <common-issues@hadoop.apache.org>; Mon, 30 Aug 2010 09:31:58 GMT
Message-ID: <5001813.65451283160717994.JavaMail.jira@thor>
Date: Mon, 30 Aug 2010 05:31:57 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6342) Create a script to squash a common,
 hdfs, and mapreduce tarball into a single hadoop tarball
In-Reply-To: <1136640537.1256834099525.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V updated HADOOP-6342:
------------------------------

    Fix Version/s: 0.21.1

Looks like this is missed in the 0.21.0 release.

> Create a script to squash a common, hdfs, and mapreduce tarball into a single hadoop tarball
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6342
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6342
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>            Priority: Blocker
>             Fix For: 0.21.1
>
>         Attachments: h-6342.patch, HADOOP-6342.2.patch, HADOOP-6342.patch, tar-munge, tar-munge
>
>
> It would be convenient for the transition if we had a script to take a set of common, hdfs, and mapreduce tarballs and merge them into a single tarball. This is intended just to help users who don't want to transition to split projects for deployment immediately.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10473-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 09:42:23 2010
Return-Path: <common-issues-return-10473-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83106 invoked from network); 30 Aug 2010 09:42:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 09:42:23 -0000
Received: (qmail 9595 invoked by uid 500); 30 Aug 2010 09:42:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9518 invoked by uid 500); 30 Aug 2010 09:42:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9510 invoked by uid 99); 30 Aug 2010 09:42:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 09:42:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 09:42:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7U9fuXH005494
	for <common-issues@hadoop.apache.org>; Mon, 30 Aug 2010 09:41:57 GMT
Message-ID: <5484726.65631283161316870.JavaMail.jira@thor>
Date: Mon, 30 Aug 2010 05:41:56 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6929) RPC should have a way to pass
 Security information other than protocol annotations
In-Reply-To: <29776327.15071282855734948.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904119#action_12904119 ] 

Vinod K V commented on HADOOP-6929:
-----------------------------------


Can we do something like this? (An updated version of Sharad's offline proposal)

{code}
interface SecurityContext {
  KerberosContext getKerberosContext(Class protocol);
  TokenContext getTokenContext(Class protocol);
}

//get the info from annotations
AnnotatedSecurityContext implements SecurityContext {
  KerberosContext getKerborosContext(Class protocol) {
     // construct KerberosContext from annotation KerberosInfo
  }

  TokenContext getTokenContext(Class protocol) {
     // construct TokenContext from annotation TokenInfo
  }
}

// get the information from hand-crafted context object.
HandCraftedSecurityContext implements SecurityContext {
}

{code}

We will need to pass this context information all through into the RPC layer. (Is this why annotations were originally used?)

> RPC should have a way to pass Security information other than protocol annotations
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6929
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6929
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc, security
>            Reporter: Sharad Agarwal
>             Fix For: 0.22.0
>
>
> Currently Hadoop RPC allows protocol annotations as the only way to pass security information. This becomes a problem if protocols are generated and not hand written. For example protocols generated via Avro and passed over Avro tunnel (AvroRpcEngine.java) can't pass the security information.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10474-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 13:15:33 2010
Return-Path: <common-issues-return-10474-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60762 invoked from network); 30 Aug 2010 13:15:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 13:15:33 -0000
Received: (qmail 32961 invoked by uid 500); 30 Aug 2010 13:15:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32442 invoked by uid 500); 30 Aug 2010 13:15:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32429 invoked by uid 99); 30 Aug 2010 13:15:28 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 13:15:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 13:15:27 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7UDF7OM007592
	for <common-issues@hadoop.apache.org>; Mon, 30 Aug 2010 13:15:07 GMT
Message-ID: <10669153.68721283174107314.JavaMail.jira@thor>
Date: Mon, 30 Aug 2010 09:15:07 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6786) test-patch needs to verify Herriot
 integrity
In-Reply-To: <19336182.46451274829212745.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904190#action_12904190 ] 

Hudson commented on HADOOP-6786:
--------------------------------

Integrated in Hadoop-Common-trunk #438 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/438/])
    HADOOP-6786. test-patch needs to verify Herriot integrity. Contributed by Konstantin Boudnik


> test-patch needs to verify Herriot integrity
> --------------------------------------------
>
>                 Key: HADOOP-6786
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6786
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>             Fix For: 0.21.1
>
>         Attachments: HADOOP-6786.patch, HADOOP-6786.patch, HADOOP-6786.patch
>
>
> Whenever a new patch is submitted for verification {{test-patch}} process has to make sure that none of Herriot bindings were broken.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10475-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 17:56:15 2010
Return-Path: <common-issues-return-10475-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67728 invoked from network); 30 Aug 2010 17:56:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 17:56:15 -0000
Received: (qmail 41725 invoked by uid 500); 30 Aug 2010 17:56:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41678 invoked by uid 500); 30 Aug 2010 17:56:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41655 invoked by uid 99); 30 Aug 2010 17:56:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 17:56:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 17:56:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7UHtrQS010169
	for <common-issues@hadoop.apache.org>; Mon, 30 Aug 2010 17:55:53 GMT
Message-ID: <10889828.72241283190953414.JavaMail.jira@thor>
Date: Mon, 30 Aug 2010 13:55:53 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6913) Circular initialization between
 UserGroupInformation and KerberosName
In-Reply-To: <27327567.281671281550577971.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6913:
---------------------------------------

    Status: Open  (was: Patch Available)

resubmitting to test the hudson patch testing

> Circular initialization between UserGroupInformation and KerberosName
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6913
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6913
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6913-01.patch
>
>
> If the first call to UGI is UGI.setConfiguration(conf), it will try to initialize UGI class. During this initialization, the code calls KerberosName.setConfiguration(). KerberosName's static initializer will in turn call UGI.isSecurityEnabled(). Since UGI hasn't been completely initialized yet, isSecurityEnabled() will re-initialize UGI with a DEFAULT conf. As a result, the original conf used in UGI.setConfiguration(conf) will be overwritten by the DEFAULT conf.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10476-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 17:58:15 2010
Return-Path: <common-issues-return-10476-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67955 invoked from network); 30 Aug 2010 17:58:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 17:58:15 -0000
Received: (qmail 42550 invoked by uid 500); 30 Aug 2010 17:58:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42526 invoked by uid 500); 30 Aug 2010 17:58:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42518 invoked by uid 99); 30 Aug 2010 17:58:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 17:58:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 17:58:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7UHvskG010184
	for <common-issues@hadoop.apache.org>; Mon, 30 Aug 2010 17:57:54 GMT
Message-ID: <23809269.72261283191073992.JavaMail.jira@thor>
Date: Mon, 30 Aug 2010 13:57:53 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6913) Circular initialization between
 UserGroupInformation and KerberosName
In-Reply-To: <27327567.281671281550577971.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6913:
---------------------------------------

    Status: Patch Available  (was: Open)

> Circular initialization between UserGroupInformation and KerberosName
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6913
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6913
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6913-01.patch
>
>
> If the first call to UGI is UGI.setConfiguration(conf), it will try to initialize UGI class. During this initialization, the code calls KerberosName.setConfiguration(). KerberosName's static initializer will in turn call UGI.isSecurityEnabled(). Since UGI hasn't been completely initialized yet, isSecurityEnabled() will re-initialize UGI with a DEFAULT conf. As a result, the original conf used in UGI.setConfiguration(conf) will be overwritten by the DEFAULT conf.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10477-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 18:26:16 2010
Return-Path: <common-issues-return-10477-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84106 invoked from network); 30 Aug 2010 18:26:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 18:26:16 -0000
Received: (qmail 65663 invoked by uid 500); 30 Aug 2010 18:26:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65600 invoked by uid 500); 30 Aug 2010 18:26:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65403 invoked by uid 99); 30 Aug 2010 18:26:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 18:26:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 18:26:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7UIPtiW010407
	for <common-issues@hadoop.apache.org>; Mon, 30 Aug 2010 18:25:55 GMT
Message-ID: <33057867.72551283192755246.JavaMail.jira@thor>
Date: Mon, 30 Aug 2010 14:25:55 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6913) Circular initialization between
 UserGroupInformation and KerberosName
In-Reply-To: <27327567.281671281550577971.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6913:
---------------------------------------

    Status: Patch Available  (was: Open)

> Circular initialization between UserGroupInformation and KerberosName
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6913
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6913
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6913-01.patch
>
>
> If the first call to UGI is UGI.setConfiguration(conf), it will try to initialize UGI class. During this initialization, the code calls KerberosName.setConfiguration(). KerberosName's static initializer will in turn call UGI.isSecurityEnabled(). Since UGI hasn't been completely initialized yet, isSecurityEnabled() will re-initialize UGI with a DEFAULT conf. As a result, the original conf used in UGI.setConfiguration(conf) will be overwritten by the DEFAULT conf.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10478-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 18:26:17 2010
Return-Path: <common-issues-return-10478-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84150 invoked from network); 30 Aug 2010 18:26:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 18:26:17 -0000
Received: (qmail 65791 invoked by uid 500); 30 Aug 2010 18:26:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65603 invoked by uid 500); 30 Aug 2010 18:26:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65358 invoked by uid 99); 30 Aug 2010 18:26:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 18:26:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 18:26:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7UIPsb0010401
	for <common-issues@hadoop.apache.org>; Mon, 30 Aug 2010 18:25:55 GMT
Message-ID: <24203398.72531283192754787.JavaMail.jira@thor>
Date: Mon, 30 Aug 2010 14:25:54 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6913) Circular initialization between
 UserGroupInformation and KerberosName
In-Reply-To: <27327567.281671281550577971.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6913:
---------------------------------------

    Status: Open  (was: Patch Available)

> Circular initialization between UserGroupInformation and KerberosName
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6913
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6913
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6913-01.patch
>
>
> If the first call to UGI is UGI.setConfiguration(conf), it will try to initialize UGI class. During this initialization, the code calls KerberosName.setConfiguration(). KerberosName's static initializer will in turn call UGI.isSecurityEnabled(). Since UGI hasn't been completely initialized yet, isSecurityEnabled() will re-initialize UGI with a DEFAULT conf. As a result, the original conf used in UGI.setConfiguration(conf) will be overwritten by the DEFAULT conf.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10479-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 18:51:35 2010
Return-Path: <common-issues-return-10479-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90807 invoked from network); 30 Aug 2010 18:51:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 18:51:35 -0000
Received: (qmail 98026 invoked by uid 500); 30 Aug 2010 18:51:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97927 invoked by uid 500); 30 Aug 2010 18:51:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97919 invoked by uid 99); 30 Aug 2010 18:51:34 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 18:51:34 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 18:51:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7UIotTr010647
	for <common-issues@hadoop.apache.org>; Mon, 30 Aug 2010 18:50:55 GMT
Message-ID: <27953661.73001283194255436.JavaMail.jira@thor>
Date: Mon, 30 Aug 2010 14:50:55 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6907) Rpc client doesn't use the
 per-connection conf to figure out server's Kerberos principal
In-Reply-To: <9029429.243131281389838773.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904283#action_12904283 ] 

Kan Zhang commented on HADOOP-6907:
-----------------------------------

The 6 javadoc warnings are from SecurityUtil.java and KerberosName.java and not related to this patch. The number of javac warnings from compile-core-classes stayed at 15, I don't know why test-patch reported there is an increase of javac warnings.

> Rpc client doesn't use the per-connection conf to figure out server's Kerberos principal
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6907
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6907
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6907-12.patch, c6907-15.patch, c6907-16.patch
>
>
> Currently, RPC client caches the conf that was passed in to its constructor and uses that same conf (or values obtained from it) for every connection it sets up. This is not sufficient for security since each connection needs to figure out server's Kerberos principal on a per-connection basis. It's not reasonable to expect the first conf used by a user to contain all the Kerberos principals that her future connections will ever need. Or worse, if her first conf contains an incorrect principal name, it will prevent the user from connecting to the server even if she later on passes in a correct conf on retry (by calling RPC.getProxy()).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10480-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 19:11:16 2010
Return-Path: <common-issues-return-10480-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98925 invoked from network); 30 Aug 2010 19:11:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 19:11:16 -0000
Received: (qmail 26058 invoked by uid 500); 30 Aug 2010 19:11:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25954 invoked by uid 500); 30 Aug 2010 19:11:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25946 invoked by uid 99); 30 Aug 2010 19:11:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 19:11:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 19:11:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7UJAstE010917
	for <common-issues@hadoop.apache.org>; Mon, 30 Aug 2010 19:10:55 GMT
Message-ID: <29403716.73471283195454470.JavaMail.jira@thor>
Date: Mon, 30 Aug 2010 15:10:54 -0400 (EDT)
From: "Stephen Watt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6924) Build fails with non-Sun JREs due to
 different pathing to the operating system architecture shared libraries
In-Reply-To: <25715248.539481282682896678.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stephen Watt updated HADOOP-6924:
---------------------------------

        Fix Version/s: 0.21.1
                       0.22.0
    Affects Version/s: 0.21.0

> Build fails with non-Sun JREs due to different pathing to the operating system architecture shared libraries
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6924
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6924
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.0, 0.20.1, 0.20.2, 0.21.0
>         Environment: SLES 10, IBM Java 6
>            Reporter: Stephen Watt
>             Fix For: 0.20.3, 0.21.1, 0.22.0
>
>
> The src/native/configure script used to build the native libraries has an environment variable called JNI_LDFLAGS which is set as follows:
> JNI_LDFLAGS="-L$JAVA_HOME/jre/lib/$OS_ARCH/server"
> This pathing convention to the shared libraries for the operating system architecture is unique to Oracle/Sun Java and thus on other flavors of Java the path will not exist and will result in a build failure with the following exception:
>      [exec] gcc -shared  ../src/org/apache/hadoop/io/compress/zlib/.libs/ZlibCompressor.o ../src/org/apache/hadoop/io/compress/zlib/.libs/ZlibDecompressor.o  -L/home/hadoop/Java-Versions/ibm-java-i386-60/jre/lib/x86/server -ljvm -ldl  -m32 -m32 -Wl,-soname -Wl,libhadoop.so.1 -o .libs/libhadoop.so.1.0.0
>      [exec] /usr/lib/gcc/i586-suse-linux/4.1.2/../../../../i586-suse-linux/bin/ld: cannot find -ljvm
>      [exec] collect2: ld returned 1 exit status

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10481-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 19:18:16 2010
Return-Path: <common-issues-return-10481-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 631 invoked from network); 30 Aug 2010 19:18:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 19:18:16 -0000
Received: (qmail 36842 invoked by uid 500); 30 Aug 2010 19:18:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36697 invoked by uid 500); 30 Aug 2010 19:18:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36674 invoked by uid 99); 30 Aug 2010 19:18:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 19:18:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 19:18:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7UJHtbf010975
	for <common-issues@hadoop.apache.org>; Mon, 30 Aug 2010 19:17:55 GMT
Message-ID: <22335863.73571283195875029.JavaMail.jira@thor>
Date: Mon, 30 Aug 2010 15:17:55 -0400 (EDT)
From: "Stephen Watt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6924) Build fails with non-Sun JREs due to
 different pathing to the operating system architecture shared libraries
In-Reply-To: <25715248.539481282682896678.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stephen Watt updated HADOOP-6924:
---------------------------------

    Attachment: HADOOP-6924.patch

This is a one line fix that identifies the location of the appropriate libjvm.so file. For Sun systems this is under the jre/lib/$OS_ARCH/server folder for IBM JREs this is under the jre/lib/$OS_ARCH/classic folder. Non-Sun JREs do not have a sun.boot.library.path environment variable that can be relied upon. 

This is also easily extensible as other JREs (such as Apache Harmony) can simply extend the regular expression by adding in an "OR" expression for a folder that is unique to their JRE.

> Build fails with non-Sun JREs due to different pathing to the operating system architecture shared libraries
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6924
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6924
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.0, 0.20.1, 0.20.2, 0.21.0
>         Environment: SLES 10, IBM Java 6
>            Reporter: Stephen Watt
>             Fix For: 0.20.3, 0.21.1, 0.22.0
>
>         Attachments: HADOOP-6924.patch
>
>
> The src/native/configure script used to build the native libraries has an environment variable called JNI_LDFLAGS which is set as follows:
> JNI_LDFLAGS="-L$JAVA_HOME/jre/lib/$OS_ARCH/server"
> This pathing convention to the shared libraries for the operating system architecture is unique to Oracle/Sun Java and thus on other flavors of Java the path will not exist and will result in a build failure with the following exception:
>      [exec] gcc -shared  ../src/org/apache/hadoop/io/compress/zlib/.libs/ZlibCompressor.o ../src/org/apache/hadoop/io/compress/zlib/.libs/ZlibDecompressor.o  -L/home/hadoop/Java-Versions/ibm-java-i386-60/jre/lib/x86/server -ljvm -ldl  -m32 -m32 -Wl,-soname -Wl,libhadoop.so.1 -o .libs/libhadoop.so.1.0.0
>      [exec] /usr/lib/gcc/i586-suse-linux/4.1.2/../../../../i586-suse-linux/bin/ld: cannot find -ljvm
>      [exec] collect2: ld returned 1 exit status

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10482-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 19:18:20 2010
Return-Path: <common-issues-return-10482-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 765 invoked from network); 30 Aug 2010 19:18:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 19:18:19 -0000
Received: (qmail 37057 invoked by uid 500); 30 Aug 2010 19:18:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37025 invoked by uid 500); 30 Aug 2010 19:18:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37017 invoked by uid 99); 30 Aug 2010 19:18:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 19:18:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 19:18:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7UJHw4M011005
	for <common-issues@hadoop.apache.org>; Mon, 30 Aug 2010 19:17:58 GMT
Message-ID: <26231722.73671283195878476.JavaMail.jira@thor>
Date: Mon, 30 Aug 2010 15:17:58 -0400 (EDT)
From: "Stephen Watt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6924) Build fails with non-Sun JREs due to
 different pathing to the operating system architecture shared libraries
In-Reply-To: <25715248.539481282682896678.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stephen Watt updated HADOOP-6924:
---------------------------------

    Status: Patch Available  (was: Open)

> Build fails with non-Sun JREs due to different pathing to the operating system architecture shared libraries
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6924
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6924
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.21.0, 0.20.2, 0.20.1, 0.20.0
>         Environment: SLES 10, IBM Java 6
>            Reporter: Stephen Watt
>             Fix For: 0.20.3, 0.21.1, 0.22.0
>
>         Attachments: HADOOP-6924.patch
>
>
> The src/native/configure script used to build the native libraries has an environment variable called JNI_LDFLAGS which is set as follows:
> JNI_LDFLAGS="-L$JAVA_HOME/jre/lib/$OS_ARCH/server"
> This pathing convention to the shared libraries for the operating system architecture is unique to Oracle/Sun Java and thus on other flavors of Java the path will not exist and will result in a build failure with the following exception:
>      [exec] gcc -shared  ../src/org/apache/hadoop/io/compress/zlib/.libs/ZlibCompressor.o ../src/org/apache/hadoop/io/compress/zlib/.libs/ZlibDecompressor.o  -L/home/hadoop/Java-Versions/ibm-java-i386-60/jre/lib/x86/server -ljvm -ldl  -m32 -m32 -Wl,-soname -Wl,libhadoop.so.1 -o .libs/libhadoop.so.1.0.0
>      [exec] /usr/lib/gcc/i586-suse-linux/4.1.2/../../../../i586-suse-linux/bin/ld: cannot find -ljvm
>      [exec] collect2: ld returned 1 exit status

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10483-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 21:21:16 2010
Return-Path: <common-issues-return-10483-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36728 invoked from network); 30 Aug 2010 21:21:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 21:21:16 -0000
Received: (qmail 67671 invoked by uid 500); 30 Aug 2010 21:21:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67560 invoked by uid 500); 30 Aug 2010 21:21:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67552 invoked by uid 99); 30 Aug 2010 21:21:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 21:21:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 21:21:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7ULKsBU012146
	for <common-issues@hadoop.apache.org>; Mon, 30 Aug 2010 21:20:55 GMT
Message-ID: <32143289.75601283203254980.JavaMail.jira@thor>
Date: Mon, 30 Aug 2010 17:20:54 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6907) Rpc client doesn't use the
 per-connection conf to figure out server's Kerberos principal
In-Reply-To: <9029429.243131281389838773.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6907:
------------------------------

    Attachment: c6907-18.patch

I guess it was due to newly deprecated methods. Uploading a new patch.

> Rpc client doesn't use the per-connection conf to figure out server's Kerberos principal
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6907
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6907
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6907-12.patch, c6907-15.patch, c6907-16.patch, c6907-18.patch
>
>
> Currently, RPC client caches the conf that was passed in to its constructor and uses that same conf (or values obtained from it) for every connection it sets up. This is not sufficient for security since each connection needs to figure out server's Kerberos principal on a per-connection basis. It's not reasonable to expect the first conf used by a user to contain all the Kerberos principals that her future connections will ever need. Or worse, if her first conf contains an incorrect principal name, it will prevent the user from connecting to the server even if she later on passes in a correct conf on retry (by calling RPC.getProxy()).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10484-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 21:25:15 2010
Return-Path: <common-issues-return-10484-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37830 invoked from network); 30 Aug 2010 21:25:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 21:25:15 -0000
Received: (qmail 72339 invoked by uid 500); 30 Aug 2010 21:25:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72271 invoked by uid 500); 30 Aug 2010 21:25:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72263 invoked by uid 99); 30 Aug 2010 21:25:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 21:25:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 21:25:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7ULOrV3012268
	for <common-issues@hadoop.apache.org>; Mon, 30 Aug 2010 21:24:53 GMT
Message-ID: <27754989.75941283203493814.JavaMail.jira@thor>
Date: Mon, 30 Aug 2010 17:24:53 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6913) Circular initialization between
 UserGroupInformation and KerberosName
In-Reply-To: <27327567.281671281550577971.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6913:
---------------------------------------

    Status: Open  (was: Patch Available)

> Circular initialization between UserGroupInformation and KerberosName
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6913
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6913
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6913-01.patch
>
>
> If the first call to UGI is UGI.setConfiguration(conf), it will try to initialize UGI class. During this initialization, the code calls KerberosName.setConfiguration(). KerberosName's static initializer will in turn call UGI.isSecurityEnabled(). Since UGI hasn't been completely initialized yet, isSecurityEnabled() will re-initialize UGI with a DEFAULT conf. As a result, the original conf used in UGI.setConfiguration(conf) will be overwritten by the DEFAULT conf.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10485-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 21:25:33 2010
Return-Path: <common-issues-return-10485-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37932 invoked from network); 30 Aug 2010 21:25:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 21:25:33 -0000
Received: (qmail 72812 invoked by uid 500); 30 Aug 2010 21:25:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72748 invoked by uid 500); 30 Aug 2010 21:25:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72740 invoked by uid 99); 30 Aug 2010 21:25:32 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 21:25:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 21:25:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7ULOsAo012274
	for <common-issues@hadoop.apache.org>; Mon, 30 Aug 2010 21:24:54 GMT
Message-ID: <25408617.75961283203494138.JavaMail.jira@thor>
Date: Mon, 30 Aug 2010 17:24:54 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6913) Circular initialization between
 UserGroupInformation and KerberosName
In-Reply-To: <27327567.281671281550577971.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6913:
---------------------------------------

    Status: Patch Available  (was: Open)

> Circular initialization between UserGroupInformation and KerberosName
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6913
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6913
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6913-01.patch
>
>
> If the first call to UGI is UGI.setConfiguration(conf), it will try to initialize UGI class. During this initialization, the code calls KerberosName.setConfiguration(). KerberosName's static initializer will in turn call UGI.isSecurityEnabled(). Since UGI hasn't been completely initialized yet, isSecurityEnabled() will re-initialize UGI with a DEFAULT conf. As a result, the original conf used in UGI.setConfiguration(conf) will be overwritten by the DEFAULT conf.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10486-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 21:54:34 2010
Return-Path: <common-issues-return-10486-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45308 invoked from network); 30 Aug 2010 21:54:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 21:54:34 -0000
Received: (qmail 90827 invoked by uid 500); 30 Aug 2010 21:54:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90745 invoked by uid 500); 30 Aug 2010 21:54:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90737 invoked by uid 99); 30 Aug 2010 21:54:33 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 21:54:33 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 21:54:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7ULrt5f012672
	for <common-issues@hadoop.apache.org>; Mon, 30 Aug 2010 21:53:55 GMT
Message-ID: <12792908.76991283205235257.JavaMail.jira@thor>
Date: Mon, 30 Aug 2010 17:53:55 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6432) Statistics support in FileContext
In-Reply-To: <1573206980.1260484938204.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6432:
-----------------------------------------

    Status: Open  (was: Patch Available)

> Statistics support in FileContext
> ---------------------------------
>
>                 Key: HADOOP-6432
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6432
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6432.1.patch, HADOOP-6432.2.patch, HADOOP-6432.3.patch, HADOOP-6432.4.patch
>
>
> FileContext should have API to get statistics from underlying file systems.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10487-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 22:27:15 2010
Return-Path: <common-issues-return-10487-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64660 invoked from network); 30 Aug 2010 22:27:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 22:27:15 -0000
Received: (qmail 32603 invoked by uid 500); 30 Aug 2010 22:27:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32527 invoked by uid 500); 30 Aug 2010 22:27:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32519 invoked by uid 99); 30 Aug 2010 22:27:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 22:27:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 22:27:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7UMQs9h013199
	for <common-issues@hadoop.apache.org>; Mon, 30 Aug 2010 22:26:54 GMT
Message-ID: <16571.78161283207214248.JavaMail.jira@thor>
Date: Mon, 30 Aug 2010 18:26:54 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6913) Circular initialization between
 UserGroupInformation and KerberosName
In-Reply-To: <27327567.281671281550577971.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6913:
---------------------------------------

    Status: Patch Available  (was: Open)

> Circular initialization between UserGroupInformation and KerberosName
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6913
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6913
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6913-01.patch
>
>
> If the first call to UGI is UGI.setConfiguration(conf), it will try to initialize UGI class. During this initialization, the code calls KerberosName.setConfiguration(). KerberosName's static initializer will in turn call UGI.isSecurityEnabled(). Since UGI hasn't been completely initialized yet, isSecurityEnabled() will re-initialize UGI with a DEFAULT conf. As a result, the original conf used in UGI.setConfiguration(conf) will be overwritten by the DEFAULT conf.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10488-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 22:27:33 2010
Return-Path: <common-issues-return-10488-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64826 invoked from network); 30 Aug 2010 22:27:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 22:27:33 -0000
Received: (qmail 32771 invoked by uid 500); 30 Aug 2010 22:27:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32731 invoked by uid 500); 30 Aug 2010 22:27:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32723 invoked by uid 99); 30 Aug 2010 22:27:32 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 22:27:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 22:27:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7UMQrf5013193
	for <common-issues@hadoop.apache.org>; Mon, 30 Aug 2010 22:26:54 GMT
Message-ID: <20561595.78141283207213769.JavaMail.jira@thor>
Date: Mon, 30 Aug 2010 18:26:53 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6913) Circular initialization between
 UserGroupInformation and KerberosName
In-Reply-To: <27327567.281671281550577971.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6913:
---------------------------------------

    Status: Open  (was: Patch Available)

> Circular initialization between UserGroupInformation and KerberosName
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6913
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6913
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6913-01.patch
>
>
> If the first call to UGI is UGI.setConfiguration(conf), it will try to initialize UGI class. During this initialization, the code calls KerberosName.setConfiguration(). KerberosName's static initializer will in turn call UGI.isSecurityEnabled(). Since UGI hasn't been completely initialized yet, isSecurityEnabled() will re-initialize UGI with a DEFAULT conf. As a result, the original conf used in UGI.setConfiguration(conf) will be overwritten by the DEFAULT conf.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10489-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 23:26:19 2010
Return-Path: <common-issues-return-10489-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85934 invoked from network); 30 Aug 2010 23:26:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 23:26:19 -0000
Received: (qmail 10446 invoked by uid 500); 30 Aug 2010 23:26:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10389 invoked by uid 500); 30 Aug 2010 23:26:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10381 invoked by uid 99); 30 Aug 2010 23:26:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 23:26:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 23:26:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7UNPsGN014077
	for <common-issues@hadoop.apache.org>; Mon, 30 Aug 2010 23:25:54 GMT
Message-ID: <7071704.80241283210754289.JavaMail.jira@thor>
Date: Mon, 30 Aug 2010 19:25:54 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6924) Build fails with non-Sun JREs due
 to different pathing to the operating system architecture shared libraries
In-Reply-To: <25715248.539481282682896678.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904408#action_12904408 ] 

Allen Wittenauer commented on HADOOP-6924:
------------------------------------------

This will need to tested on OS X Snow Leopard, which is both bi-arch, a Sun-based JVM and doesn't use jre/lib/server. :)



> Build fails with non-Sun JREs due to different pathing to the operating system architecture shared libraries
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6924
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6924
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.0, 0.20.1, 0.20.2, 0.21.0
>         Environment: SLES 10, IBM Java 6
>            Reporter: Stephen Watt
>             Fix For: 0.20.3, 0.21.1, 0.22.0
>
>         Attachments: HADOOP-6924.patch
>
>
> The src/native/configure script used to build the native libraries has an environment variable called JNI_LDFLAGS which is set as follows:
> JNI_LDFLAGS="-L$JAVA_HOME/jre/lib/$OS_ARCH/server"
> This pathing convention to the shared libraries for the operating system architecture is unique to Oracle/Sun Java and thus on other flavors of Java the path will not exist and will result in a build failure with the following exception:
>      [exec] gcc -shared  ../src/org/apache/hadoop/io/compress/zlib/.libs/ZlibCompressor.o ../src/org/apache/hadoop/io/compress/zlib/.libs/ZlibDecompressor.o  -L/home/hadoop/Java-Versions/ibm-java-i386-60/jre/lib/x86/server -ljvm -ldl  -m32 -m32 -Wl,-soname -Wl,libhadoop.so.1 -o .libs/libhadoop.so.1.0.0
>      [exec] /usr/lib/gcc/i586-suse-linux/4.1.2/../../../../i586-suse-linux/bin/ld: cannot find -ljvm
>      [exec] collect2: ld returned 1 exit status

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10490-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Aug 30 23:45:16 2010
Return-Path: <common-issues-return-10490-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90811 invoked from network); 30 Aug 2010 23:45:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Aug 2010 23:45:16 -0000
Received: (qmail 23808 invoked by uid 500); 30 Aug 2010 23:45:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23755 invoked by uid 500); 30 Aug 2010 23:45:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23747 invoked by uid 99); 30 Aug 2010 23:45:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 23:45:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Aug 2010 23:45:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7UNisKQ014326
	for <common-issues@hadoop.apache.org>; Mon, 30 Aug 2010 23:44:54 GMT
Message-ID: <5993206.80921283211894763.JavaMail.jira@thor>
Date: Mon, 30 Aug 2010 19:44:54 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6932) Namenode start (init) fails because
 of invalid kerberos key, even when security set to "simple"
In-Reply-To: <24985184.16041282857173665.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6932:
-----------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

ran tests. all passed (except TestHdfsTrash, which fails anyway).

committed to trunk.

> Namenode start (init) fails because of invalid kerberos key, even when security set to "simple"
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6932
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6932
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6932-1.patch, HADOOP-6932-2.patch, HADOOP-6932.patch, HDFS-1358.patch
>
>
> NameNode.initialize() calls login() method even when security set to simple

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10491-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 31 00:10:17 2010
Return-Path: <common-issues-return-10491-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99240 invoked from network); 31 Aug 2010 00:10:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Aug 2010 00:10:17 -0000
Received: (qmail 60775 invoked by uid 500); 31 Aug 2010 00:10:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60718 invoked by uid 500); 31 Aug 2010 00:10:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60710 invoked by uid 99); 31 Aug 2010 00:10:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 00:10:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 00:10:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7V09t8X014721
	for <common-issues@hadoop.apache.org>; Tue, 31 Aug 2010 00:09:56 GMT
Message-ID: <29530640.81871283213395802.JavaMail.jira@thor>
Date: Mon, 30 Aug 2010 20:09:55 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6932) Namenode start (init) fails because
 of invalid kerberos key, even when security set to "simple"
In-Reply-To: <24985184.16041282857173665.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904434#action_12904434 ] 

Hudson commented on HADOOP-6932:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #369 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/369/])
    HADOOP-6932.  Namenode start (init) fails because of invalid kerberos key, even when security set to simple


> Namenode start (init) fails because of invalid kerberos key, even when security set to "simple"
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6932
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6932
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6932-1.patch, HADOOP-6932-2.patch, HADOOP-6932.patch, HDFS-1358.patch
>
>
> NameNode.initialize() calls login() method even when security set to simple

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10492-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 31 00:12:18 2010
Return-Path: <common-issues-return-10492-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99406 invoked from network); 31 Aug 2010 00:12:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Aug 2010 00:12:18 -0000
Received: (qmail 61530 invoked by uid 500); 31 Aug 2010 00:12:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61502 invoked by uid 500); 31 Aug 2010 00:12:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61494 invoked by uid 99); 31 Aug 2010 00:12:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 00:12:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 00:12:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7V0BrYv014743
	for <common-issues@hadoop.apache.org>; Tue, 31 Aug 2010 00:11:53 GMT
Message-ID: <11973969.81911283213513579.JavaMail.jira@thor>
Date: Mon, 30 Aug 2010 20:11:53 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6913) Circular initialization between
 UserGroupInformation and KerberosName
In-Reply-To: <27327567.281671281550577971.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6913:
-----------------------------------

        Status: Resolved  (was: Patch Available)
    Resolution: Fixed

Committed to trunk. Thanks Kan.

> Circular initialization between UserGroupInformation and KerberosName
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6913
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6913
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6913-01.patch
>
>
> If the first call to UGI is UGI.setConfiguration(conf), it will try to initialize UGI class. During this initialization, the code calls KerberosName.setConfiguration(). KerberosName's static initializer will in turn call UGI.isSecurityEnabled(). Since UGI hasn't been completely initialized yet, isSecurityEnabled() will re-initialize UGI with a DEFAULT conf. As a result, the original conf used in UGI.setConfiguration(conf) will be overwritten by the DEFAULT conf.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10493-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 31 00:30:19 2010
Return-Path: <common-issues-return-10493-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8802 invoked from network); 31 Aug 2010 00:30:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Aug 2010 00:30:19 -0000
Received: (qmail 86268 invoked by uid 500); 31 Aug 2010 00:30:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86245 invoked by uid 500); 31 Aug 2010 00:30:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86237 invoked by uid 99); 31 Aug 2010 00:30:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 00:30:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 00:30:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7V0Twwe015016
	for <common-issues@hadoop.apache.org>; Tue, 31 Aug 2010 00:29:58 GMT
Message-ID: <8363213.82661283214598632.JavaMail.jira@thor>
Date: Mon, 30 Aug 2010 20:29:58 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6913) Circular initialization between
 UserGroupInformation and KerberosName
In-Reply-To: <27327567.281671281550577971.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904446#action_12904446 ] 

Hudson commented on HADOOP-6913:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #370 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/370/])
    HADOOP-6913. Circular initialization between UserGroupInformation and KerberosName


> Circular initialization between UserGroupInformation and KerberosName
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6913
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6913
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6913-01.patch
>
>
> If the first call to UGI is UGI.setConfiguration(conf), it will try to initialize UGI class. During this initialization, the code calls KerberosName.setConfiguration(). KerberosName's static initializer will in turn call UGI.isSecurityEnabled(). Since UGI hasn't been completely initialized yet, isSecurityEnabled() will re-initialize UGI with a DEFAULT conf. As a result, the original conf used in UGI.setConfiguration(conf) will be overwritten by the DEFAULT conf.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10494-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 31 11:21:20 2010
Return-Path: <common-issues-return-10494-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73422 invoked from network); 31 Aug 2010 11:21:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Aug 2010 11:21:20 -0000
Received: (qmail 81436 invoked by uid 500); 31 Aug 2010 11:21:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81275 invoked by uid 500); 31 Aug 2010 11:21:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81262 invoked by uid 99); 31 Aug 2010 11:21:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 11:21:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 11:21:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7VBKsfq021119
	for <common-issues@hadoop.apache.org>; Tue, 31 Aug 2010 11:20:54 GMT
Message-ID: <32901688.91661283253654460.JavaMail.jira@thor>
Date: Tue, 31 Aug 2010 07:20:54 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6932) Namenode start (init) fails because
 of invalid kerberos key, even when security set to "simple"
In-Reply-To: <24985184.16041282857173665.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904600#action_12904600 ] 

Hudson commented on HADOOP-6932:
--------------------------------

Integrated in Hadoop-Common-trunk #439 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/439/])
    HADOOP-6932.  Namenode start (init) fails because of invalid kerberos key, even when security set to simple


> Namenode start (init) fails because of invalid kerberos key, even when security set to "simple"
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6932
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6932
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6932-1.patch, HADOOP-6932-2.patch, HADOOP-6932.patch, HDFS-1358.patch
>
>
> NameNode.initialize() calls login() method even when security set to simple

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10495-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 31 11:21:21 2010
Return-Path: <common-issues-return-10495-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73471 invoked from network); 31 Aug 2010 11:21:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Aug 2010 11:21:21 -0000
Received: (qmail 81489 invoked by uid 500); 31 Aug 2010 11:21:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81352 invoked by uid 500); 31 Aug 2010 11:21:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81282 invoked by uid 99); 31 Aug 2010 11:21:17 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 11:21:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 11:21:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7VBKsJw021125
	for <common-issues@hadoop.apache.org>; Tue, 31 Aug 2010 11:20:54 GMT
Message-ID: <17410510.91681283253654901.JavaMail.jira@thor>
Date: Tue, 31 Aug 2010 07:20:54 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6913) Circular initialization between
 UserGroupInformation and KerberosName
In-Reply-To: <27327567.281671281550577971.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904601#action_12904601 ] 

Hudson commented on HADOOP-6913:
--------------------------------

Integrated in Hadoop-Common-trunk #439 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/439/])
    HADOOP-6913. Circular initialization between UserGroupInformation and KerberosName


> Circular initialization between UserGroupInformation and KerberosName
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6913
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6913
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6913-01.patch
>
>
> If the first call to UGI is UGI.setConfiguration(conf), it will try to initialize UGI class. During this initialization, the code calls KerberosName.setConfiguration(). KerberosName's static initializer will in turn call UGI.isSecurityEnabled(). Since UGI hasn't been completely initialized yet, isSecurityEnabled() will re-initialize UGI with a DEFAULT conf. As a result, the original conf used in UGI.setConfiguration(conf) will be overwritten by the DEFAULT conf.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10496-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 31 18:45:37 2010
Return-Path: <common-issues-return-10496-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49571 invoked from network); 31 Aug 2010 18:45:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Aug 2010 18:45:37 -0000
Received: (qmail 90398 invoked by uid 500); 31 Aug 2010 18:45:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90372 invoked by uid 500); 31 Aug 2010 18:45:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90364 invoked by uid 99); 31 Aug 2010 18:45:36 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 18:45:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 18:45:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7VIiv4E025352
	for <common-issues@hadoop.apache.org>; Tue, 31 Aug 2010 18:44:57 GMT
Message-ID: <33418367.98071283280297666.JavaMail.jira@thor>
Date: Tue, 31 Aug 2010 14:44:57 -0400 (EDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6930) AvroRpcEngine doesn't work with
 generated Avro code
In-Reply-To: <32541354.15611282856633769.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sharad Agarwal reassigned HADOOP-6930:
--------------------------------------

    Assignee: Sharad Agarwal

> AvroRpcEngine doesn't work with generated Avro code
> ---------------------------------------------------
>
>                 Key: HADOOP-6930
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6930
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>
> AvroRpcEngine uses 'reflect' based java implementation. There should be a way to have it work with 'specific' (generated code from avro idl).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10497-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 31 18:49:16 2010
Return-Path: <common-issues-return-10497-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50013 invoked from network); 31 Aug 2010 18:49:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Aug 2010 18:49:16 -0000
Received: (qmail 93401 invoked by uid 500); 31 Aug 2010 18:49:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93340 invoked by uid 500); 31 Aug 2010 18:49:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93332 invoked by uid 99); 31 Aug 2010 18:49:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 18:49:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 18:49:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7VImtb1025413
	for <common-issues@hadoop.apache.org>; Tue, 31 Aug 2010 18:48:55 GMT
Message-ID: <28126586.98201283280535112.JavaMail.jira@thor>
Date: Tue, 31 Aug 2010 14:48:55 -0400 (EDT)
From: "Sharad Agarwal (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6930) AvroRpcEngine doesn't work with
 generated Avro code
In-Reply-To: <32541354.15611282856633769.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sharad Agarwal updated HADOOP-6930:
-----------------------------------

    Attachment: Hadoop-6930.patch

Here is the patch adding AvroSpecificRpcEngine.

Should we make the RpcEngine a public API instead of package private ? In order to use these engine, right now I need to be in org.apache.hadoop.ipc package.



> AvroRpcEngine doesn't work with generated Avro code
> ---------------------------------------------------
>
>                 Key: HADOOP-6930
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6930
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>         Attachments: Hadoop-6930.patch
>
>
> AvroRpcEngine uses 'reflect' based java implementation. There should be a way to have it work with 'specific' (generated code from avro idl).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10498-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 31 20:31:17 2010
Return-Path: <common-issues-return-10498-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80272 invoked from network); 31 Aug 2010 20:31:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Aug 2010 20:31:16 -0000
Received: (qmail 86866 invoked by uid 500); 31 Aug 2010 20:31:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86796 invoked by uid 500); 31 Aug 2010 20:31:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86788 invoked by uid 99); 31 Aug 2010 20:31:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 20:31:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 20:31:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7VKUtQM026506
	for <common-issues@hadoop.apache.org>; Tue, 31 Aug 2010 20:30:55 GMT
Message-ID: <11043920.100151283286654987.JavaMail.jira@thor>
Date: Tue, 31 Aug 2010 16:30:54 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6907) Rpc client doesn't use the
 per-connection conf to figure out server's Kerberos principal
In-Reply-To: <9029429.243131281389838773.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6907:
---------------------------------------

    Status: Open  (was: Patch Available)

> Rpc client doesn't use the per-connection conf to figure out server's Kerberos principal
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6907
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6907
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6907-12.patch, c6907-15.patch, c6907-16.patch, c6907-18.patch
>
>
> Currently, RPC client caches the conf that was passed in to its constructor and uses that same conf (or values obtained from it) for every connection it sets up. This is not sufficient for security since each connection needs to figure out server's Kerberos principal on a per-connection basis. It's not reasonable to expect the first conf used by a user to contain all the Kerberos principals that her future connections will ever need. Or worse, if her first conf contains an incorrect principal name, it will prevent the user from connecting to the server even if she later on passes in a correct conf on retry (by calling RPC.getProxy()).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10499-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 31 20:31:17 2010
Return-Path: <common-issues-return-10499-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80315 invoked from network); 31 Aug 2010 20:31:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Aug 2010 20:31:17 -0000
Received: (qmail 86998 invoked by uid 500); 31 Aug 2010 20:31:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86900 invoked by uid 500); 31 Aug 2010 20:31:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86798 invoked by uid 99); 31 Aug 2010 20:31:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 20:31:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 20:31:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7VKUt14026524
	for <common-issues@hadoop.apache.org>; Tue, 31 Aug 2010 20:30:56 GMT
Message-ID: <22900635.100211283286655975.JavaMail.jira@thor>
Date: Tue, 31 Aug 2010 16:30:55 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6907) Rpc client doesn't use the
 per-connection conf to figure out server's Kerberos principal
In-Reply-To: <9029429.243131281389838773.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6907:
---------------------------------------

    Status: Patch Available  (was: Open)

> Rpc client doesn't use the per-connection conf to figure out server's Kerberos principal
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6907
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6907
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6907-12.patch, c6907-15.patch, c6907-16.patch, c6907-18.patch
>
>
> Currently, RPC client caches the conf that was passed in to its constructor and uses that same conf (or values obtained from it) for every connection it sets up. This is not sufficient for security since each connection needs to figure out server's Kerberos principal on a per-connection basis. It's not reasonable to expect the first conf used by a user to contain all the Kerberos principals that her future connections will ever need. Or worse, if her first conf contains an incorrect principal name, it will prevent the user from connecting to the server even if she later on passes in a correct conf on retry (by calling RPC.getProxy()).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10500-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 31 20:35:20 2010
Return-Path: <common-issues-return-10500-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81237 invoked from network); 31 Aug 2010 20:35:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Aug 2010 20:35:19 -0000
Received: (qmail 93239 invoked by uid 500); 31 Aug 2010 20:35:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93192 invoked by uid 500); 31 Aug 2010 20:35:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93184 invoked by uid 99); 31 Aug 2010 20:35:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 20:35:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 20:35:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7VKYtsv026596
	for <common-issues@hadoop.apache.org>; Tue, 31 Aug 2010 20:34:55 GMT
Message-ID: <22185727.100351283286895218.JavaMail.jira@thor>
Date: Tue, 31 Aug 2010 16:34:55 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6907) Rpc client doesn't use the
 per-connection conf to figure out server's Kerberos principal
In-Reply-To: <9029429.243131281389838773.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6907:
---------------------------------------

    Status: Open  (was: Patch Available)

> Rpc client doesn't use the per-connection conf to figure out server's Kerberos principal
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6907
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6907
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6907-12.patch, c6907-15.patch, c6907-16.patch, c6907-18.patch
>
>
> Currently, RPC client caches the conf that was passed in to its constructor and uses that same conf (or values obtained from it) for every connection it sets up. This is not sufficient for security since each connection needs to figure out server's Kerberos principal on a per-connection basis. It's not reasonable to expect the first conf used by a user to contain all the Kerberos principals that her future connections will ever need. Or worse, if her first conf contains an incorrect principal name, it will prevent the user from connecting to the server even if she later on passes in a correct conf on retry (by calling RPC.getProxy()).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10501-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 31 20:35:22 2010
Return-Path: <common-issues-return-10501-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81345 invoked from network); 31 Aug 2010 20:35:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Aug 2010 20:35:22 -0000
Received: (qmail 93664 invoked by uid 500); 31 Aug 2010 20:35:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93597 invoked by uid 500); 31 Aug 2010 20:35:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93376 invoked by uid 99); 31 Aug 2010 20:35:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 20:35:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 20:35:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7VKYvCG026620
	for <common-issues@hadoop.apache.org>; Tue, 31 Aug 2010 20:34:57 GMT
Message-ID: <31130519.100431283286897499.JavaMail.jira@thor>
Date: Tue, 31 Aug 2010 16:34:57 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6907) Rpc client doesn't use the
 per-connection conf to figure out server's Kerberos principal
In-Reply-To: <9029429.243131281389838773.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6907:
---------------------------------------

    Status: Patch Available  (was: Open)

> Rpc client doesn't use the per-connection conf to figure out server's Kerberos principal
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6907
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6907
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6907-12.patch, c6907-15.patch, c6907-16.patch, c6907-18.patch
>
>
> Currently, RPC client caches the conf that was passed in to its constructor and uses that same conf (or values obtained from it) for every connection it sets up. This is not sufficient for security since each connection needs to figure out server's Kerberos principal on a per-connection basis. It's not reasonable to expect the first conf used by a user to contain all the Kerberos principals that her future connections will ever need. Or worse, if her first conf contains an incorrect principal name, it will prevent the user from connecting to the server even if she later on passes in a correct conf on retry (by calling RPC.getProxy()).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10502-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 31 20:59:19 2010
Return-Path: <common-issues-return-10502-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84730 invoked from network); 31 Aug 2010 20:59:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Aug 2010 20:59:19 -0000
Received: (qmail 21882 invoked by uid 500); 31 Aug 2010 20:59:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21844 invoked by uid 500); 31 Aug 2010 20:59:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21836 invoked by uid 99); 31 Aug 2010 20:59:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 20:59:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 20:59:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7VKwshG026836
	for <common-issues@hadoop.apache.org>; Tue, 31 Aug 2010 20:58:54 GMT
Message-ID: <1508115.100731283288334830.JavaMail.jira@thor>
Date: Tue, 31 Aug 2010 16:58:54 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6934) test for ByteWritable comparator
In-Reply-To: <11925651.51571283017793361.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6934:
---------------------------------------

    Status: Open  (was: Patch Available)

re-submiting to test hudson patch testing

> test for ByteWritable comparator
> --------------------------------
>
>                 Key: HADOOP-6934
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6934
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: record
>    Affects Versions: 0.21.0
>            Reporter: Johannes Zillmann
>             Fix For: 0.21.1
>
>         Attachments: h-6934.patch
>
>
> The test for the ByteWritable comparator bug (see HADOOP-6928), which is already fixed in 0.21.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10503-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 31 23:17:17 2010
Return-Path: <common-issues-return-10503-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32675 invoked from network); 31 Aug 2010 23:17:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Aug 2010 23:17:17 -0000
Received: (qmail 84086 invoked by uid 500); 31 Aug 2010 23:17:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84017 invoked by uid 500); 31 Aug 2010 23:17:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84008 invoked by uid 99); 31 Aug 2010 23:17:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 23:17:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 23:17:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7VNGtjr028713
	for <common-issues@hadoop.apache.org>; Tue, 31 Aug 2010 23:16:55 GMT
Message-ID: <25624598.104981283296615107.JavaMail.jira@thor>
Date: Tue, 31 Aug 2010 19:16:55 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6907) Rpc client doesn't use the
 per-connection conf to figure out server's Kerberos principal
In-Reply-To: <9029429.243131281389838773.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6907:
---------------------------------------

    Status: Open  (was: Patch Available)

> Rpc client doesn't use the per-connection conf to figure out server's Kerberos principal
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6907
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6907
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6907-12.patch, c6907-15.patch, c6907-16.patch, c6907-18.patch
>
>
> Currently, RPC client caches the conf that was passed in to its constructor and uses that same conf (or values obtained from it) for every connection it sets up. This is not sufficient for security since each connection needs to figure out server's Kerberos principal on a per-connection basis. It's not reasonable to expect the first conf used by a user to contain all the Kerberos principals that her future connections will ever need. Or worse, if her first conf contains an incorrect principal name, it will prevent the user from connecting to the server even if she later on passes in a correct conf on retry (by calling RPC.getProxy()).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10504-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 31 23:17:36 2010
Return-Path: <common-issues-return-10504-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33087 invoked from network); 31 Aug 2010 23:17:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Aug 2010 23:17:36 -0000
Received: (qmail 84841 invoked by uid 500); 31 Aug 2010 23:17:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84800 invoked by uid 500); 31 Aug 2010 23:17:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84787 invoked by uid 99); 31 Aug 2010 23:17:34 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 23:17:34 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 23:17:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7VNGuTB028731
	for <common-issues@hadoop.apache.org>; Tue, 31 Aug 2010 23:16:56 GMT
Message-ID: <24415970.105041283296616116.JavaMail.jira@thor>
Date: Tue, 31 Aug 2010 19:16:56 -0400 (EDT)
From: "Giridharan Kesavan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6907) Rpc client doesn't use the
 per-connection conf to figure out server's Kerberos principal
In-Reply-To: <9029429.243131281389838773.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HADOOP-6907:
---------------------------------------

    Status: Patch Available  (was: Open)

> Rpc client doesn't use the per-connection conf to figure out server's Kerberos principal
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6907
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6907
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6907-12.patch, c6907-15.patch, c6907-16.patch, c6907-18.patch
>
>
> Currently, RPC client caches the conf that was passed in to its constructor and uses that same conf (or values obtained from it) for every connection it sets up. This is not sufficient for security since each connection needs to figure out server's Kerberos principal on a per-connection basis. It's not reasonable to expect the first conf used by a user to contain all the Kerberos principals that her future connections will ever need. Or worse, if her first conf contains an incorrect principal name, it will prevent the user from connecting to the server even if she later on passes in a correct conf on retry (by calling RPC.getProxy()).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10505-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 31 23:41:16 2010
Return-Path: <common-issues-return-10505-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38947 invoked from network); 31 Aug 2010 23:41:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Aug 2010 23:41:15 -0000
Received: (qmail 2651 invoked by uid 500); 31 Aug 2010 23:41:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2504 invoked by uid 500); 31 Aug 2010 23:41:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2495 invoked by uid 99); 31 Aug 2010 23:41:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 23:41:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 23:41:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7VNesM7029073
	for <common-issues@hadoop.apache.org>; Tue, 31 Aug 2010 23:40:54 GMT
Message-ID: <19422002.105761283298054242.JavaMail.jira@thor>
Date: Tue, 31 Aug 2010 19:40:54 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6912) Guard against NPE when calling
 UGI.isLoginKeytabBased()
In-Reply-To: <29846672.263061281477016561.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6912:
------------------------------

    Status: Open  (was: Patch Available)

> Guard against NPE when calling UGI.isLoginKeytabBased()
> -------------------------------------------------------
>
>                 Key: HADOOP-6912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6912
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6912-01.patch
>
>
> NPE can happen when isLoginKeytabBased() is called before a login is performed. See MAPREDUCE-1992 for an example.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-10506-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Aug 31 23:41:33 2010
Return-Path: <common-issues-return-10506-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39069 invoked from network); 31 Aug 2010 23:41:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 31 Aug 2010 23:41:33 -0000
Received: (qmail 3234 invoked by uid 500); 31 Aug 2010 23:41:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3200 invoked by uid 500); 31 Aug 2010 23:41:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3192 invoked by uid 99); 31 Aug 2010 23:41:32 -0000
Received: from Unknown (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 23:41:32 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Aug 2010 23:41:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o7VNesku029079
	for <common-issues@hadoop.apache.org>; Tue, 31 Aug 2010 23:40:54 GMT
Message-ID: <21067651.105781283298054494.JavaMail.jira@thor>
Date: Tue, 31 Aug 2010 19:40:54 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6912) Guard against NPE when calling
 UGI.isLoginKeytabBased()
In-Reply-To: <29846672.263061281477016561.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6912:
------------------------------

    Status: Patch Available  (was: Open)

> Guard against NPE when calling UGI.isLoginKeytabBased()
> -------------------------------------------------------
>
>                 Key: HADOOP-6912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6912
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>         Attachments: c6912-01.patch
>
>
> NPE can happen when isLoginKeytabBased() is called before a login is performed. See MAPREDUCE-1992 for an example.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8602-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 08:15:03 2010
Return-Path: <common-issues-return-8602-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43885 invoked from network); 1 Jun 2010 08:15:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 08:15:03 -0000
Received: (qmail 72992 invoked by uid 500); 1 Jun 2010 08:15:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72904 invoked by uid 500); 1 Jun 2010 08:15:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72888 invoked by uid 99); 1 Jun 2010 08:14:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 08:14:59 +0000
X-ASF-Spam-Status: No, hits=-1484.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 08:14:58 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o518Eb2B015560
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 08:14:38 GMT
Message-ID: <10721777.99811275380077714.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 04:14:37 -0400 (EDT)
From: "Shunsuke Mikami (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6795) FsShell 'hadoop fs -text' does not
 work with other  file systems
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

FsShell 'hadoop fs -text' does not work with other  file systems 
-----------------------------------------------------------------

                 Key: HADOOP-6795
                 URL: https://issues.apache.org/jira/browse/HADOOP-6795
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
    Affects Versions: 0.20.2
            Reporter: Shunsuke Mikami
            Priority: Minor


FsShell 'hadoop fs -text' can only work with file system which set by fs.default.name.
I use Gfarm file system from Hadoop.
https://gfarm.svn.sourceforge.net/svnroot/gfarm/gfarm_hadoop/trunk/
If i set fs.default.name to hdfs, the error "Wrong FS" occurred when i submit 'hadoop fs -text' to file on gfarm file system.

 $ hadoop fs -text gfarmfs:///home/mikami/random/part-00000
 text: Wrong FS: gfarmfs://null/home/mikami/random/part-00000, expected: hdfs://hostname:9000

if i set fs.default.name to gfarmfs:///, i can get correct result.

this command's result shouldn't depend on fs.default.name.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8603-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 10:45:03 2010
Return-Path: <common-issues-return-8603-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77240 invoked from network); 1 Jun 2010 10:45:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 10:45:03 -0000
Received: (qmail 25495 invoked by uid 500); 1 Jun 2010 10:45:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25138 invoked by uid 500); 1 Jun 2010 10:45:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24377 invoked by uid 99); 1 Jun 2010 10:45:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 10:45:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 10:44:58 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51Aiaea016663
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 10:44:36 GMT
Message-ID: <8732335.101241275389076505.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 06:44:36 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

FileStatus allows null srcPath but crashes if that's done
---------------------------------------------------------

                 Key: HADOOP-6796
                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
            Reporter: Rodrigo Schmidt
            Assignee: Rodrigo Schmidt
            Priority: Minor


FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8604-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 10:47:03 2010
Return-Path: <common-issues-return-8604-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77518 invoked from network); 1 Jun 2010 10:47:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 10:47:03 -0000
Received: (qmail 30047 invoked by uid 500); 1 Jun 2010 10:47:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30000 invoked by uid 500); 1 Jun 2010 10:47:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29987 invoked by uid 99); 1 Jun 2010 10:47:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 10:47:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 10:46:58 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51Akbn2016684
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 10:46:37 GMT
Message-ID: <11862014.101291275389197225.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 06:46:37 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6796:
------------------------------------

    Attachment: HADOOP-6796.patch

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8605-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 10:47:05 2010
Return-Path: <common-issues-return-8605-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77563 invoked from network); 1 Jun 2010 10:47:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 10:47:04 -0000
Received: (qmail 30213 invoked by uid 500); 1 Jun 2010 10:47:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30059 invoked by uid 500); 1 Jun 2010 10:47:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29984 invoked by uid 99); 1 Jun 2010 10:47:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 10:47:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 10:46:58 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51AkaIm016678
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 10:46:37 GMT
Message-ID: <16284854.101271275389196749.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 06:46:36 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6796:
------------------------------------

        Fix Version/s: 0.22.0
    Affects Version/s: 0.22.0

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8606-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 10:48:58 2010
Return-Path: <common-issues-return-8606-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78106 invoked from network); 1 Jun 2010 10:48:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 10:48:58 -0000
Received: (qmail 35079 invoked by uid 500); 1 Jun 2010 10:48:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34997 invoked by uid 500); 1 Jun 2010 10:48:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34985 invoked by uid 99); 1 Jun 2010 10:48:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 10:48:57 +0000
X-ASF-Spam-Status: No, hits=-1484.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 10:48:57 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51Amack016691
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 10:48:37 GMT
Message-ID: <2529684.101311275389316735.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 06:48:36 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6796:
------------------------------------

    Status: Patch Available  (was: Open)

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8607-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 11:48:02 2010
Return-Path: <common-issues-return-8607-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94109 invoked from network); 1 Jun 2010 11:48:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 11:48:02 -0000
Received: (qmail 1430 invoked by uid 500); 1 Jun 2010 11:48:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1390 invoked by uid 500); 1 Jun 2010 11:48:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1258 invoked by uid 99); 1 Jun 2010 11:48:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 11:48:01 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 11:47:59 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51BlbKn017123
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 11:47:37 GMT
Message-ID: <33178339.101931275392857496.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 07:47:37 -0400 (EDT)
From: "Johannes Zillmann (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6797) DNS#getHosts() fallback leads to mix
 of network-interface addresse in case reverse lookup fails
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

DNS#getHosts() fallback leads to mix of network-interface addresse in case reverse lookup fails
-----------------------------------------------------------------------------------------------

                 Key: HADOOP-6797
                 URL: https://issues.apache.org/jira/browse/HADOOP-6797
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.20.2
            Reporter: Johannes Zillmann


org.apache.hadoop.net.DNS#getHosts():
{noformat}
public static String[] getHosts(String strInterface, String nameserver)
    throws UnknownHostException {
    String[] ips = getIPs(strInterface);
    Vector<String> hosts = new Vector<String>();
    for (int ctr = 0; ctr < ips.length; ctr++)
      try {
        hosts.add(reverseDns(InetAddress.getByName(ips[ctr]),
                             nameserver));
      } catch (Exception e) {
      }

    if (hosts.size() == 0)
      return new String[] { InetAddress.getLocalHost().getCanonicalHostName() };
    else
      return hosts.toArray(new String[] {});
  }
{noformat}

I have the situation where i choosing eth1 as network interface (fs.datanode.dns.interface and mapred.tasktracker.dns.interface) which is the internal interface.
The reverse lookup fails for eth1 so the fallback:
{noformat}
    if (hosts.size() == 0)
      return new String[] { InetAddress.getLocalHost().getCanonicalHostName() };
{noformat}
comes to action. The dns of eth0 is returned which is the external interface.

This leads in my case to a combination of internal ip and external dns as the default ip/host.



-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8608-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 11:50:04 2010
Return-Path: <common-issues-return-8608-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94511 invoked from network); 1 Jun 2010 11:50:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 11:50:04 -0000
Received: (qmail 6778 invoked by uid 500); 1 Jun 2010 11:50:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6649 invoked by uid 500); 1 Jun 2010 11:50:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6634 invoked by uid 99); 1 Jun 2010 11:50:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 11:50:02 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 11:50:00 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51BndGT017140
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 11:49:39 GMT
Message-ID: <25512263.101981275392979017.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 07:49:39 -0400 (EDT)
From: "Johannes Zillmann (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6797) DNS#getHosts() fallback leads to
 mix of network-interface addresse in case reverse lookup fails
In-Reply-To: <33178339.101931275392857496.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12873963#action_12873963 ] 

Johannes Zillmann commented on HADOOP-6797:
-------------------------------------------

In case we would add or replace the fallback with :
{noformat}
InetAddress.getByName(ips[ctr]).getCanonicalHostName()
{noformat}

the default ip would match up again with the default dns.

> DNS#getHosts() fallback leads to mix of network-interface addresse in case reverse lookup fails
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6797
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6797
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: Johannes Zillmann
>
> org.apache.hadoop.net.DNS#getHosts():
> {noformat}
> public static String[] getHosts(String strInterface, String nameserver)
>     throws UnknownHostException {
>     String[] ips = getIPs(strInterface);
>     Vector<String> hosts = new Vector<String>();
>     for (int ctr = 0; ctr < ips.length; ctr++)
>       try {
>         hosts.add(reverseDns(InetAddress.getByName(ips[ctr]),
>                              nameserver));
>       } catch (Exception e) {
>       }
>     if (hosts.size() == 0)
>       return new String[] { InetAddress.getLocalHost().getCanonicalHostName() };
>     else
>       return hosts.toArray(new String[] {});
>   }
> {noformat}
> I have the situation where i choosing eth1 as network interface (fs.datanode.dns.interface and mapred.tasktracker.dns.interface) which is the internal interface.
> The reverse lookup fails for eth1 so the fallback:
> {noformat}
>     if (hosts.size() == 0)
>       return new String[] { InetAddress.getLocalHost().getCanonicalHostName() };
> {noformat}
> comes to action. The dns of eth0 is returned which is the external interface.
> This leads in my case to a combination of internal ip and external dns as the default ip/host.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8609-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 13:44:05 2010
Return-Path: <common-issues-return-8609-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35462 invoked from network); 1 Jun 2010 13:44:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 13:44:05 -0000
Received: (qmail 64034 invoked by uid 500); 1 Jun 2010 13:44:05 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64002 invoked by uid 500); 1 Jun 2010 13:44:04 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63994 invoked by uid 99); 1 Jun 2010 13:44:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 13:44:04 +0000
X-ASF-Spam-Status: No, hits=-1484.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 13:44:03 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51Dhgvd018129
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 13:43:43 GMT
Message-ID: <25066887.103451275399822749.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 09:43:42 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6794) Move configuration and script files
 post split
In-Reply-To: <11039662.93131275342217664.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12873999#action_12873999 ] 

Vinod K V commented on HADOOP-6794:
-----------------------------------

The following need to be added to the patch.

conf/hadoop-policy.xml.template:
 - {{security.admin.operations.protocol.acl}} configuration property is not present in common-project but is present and needed in mapreduce.

conf/log4j.properties
 - Nine settings for 'Job Summary Appender' are present and needed in mapreduce project but are not there in common project.
 - Two settings for mapreduce task logs are present and needed in mapreduce project but are not there in common project.
    -- hadoop.tasklog.iscleanup=false
    -- log4j.appender.TLA.isCleanup=${hadoop.tasklog.iscleanup}
 - One more thing: copying this file over from mapreduce will not work because six settings for 'Security appender' are in common but not present in the mapreduce project's log4j.properties file.

May be we should also split both the above files per project. But that may be long term, will open a separate ticket if you agree.

Other issues
 - Minor: bin/rcc still refers to hadoop-core at one place.
 - when i run `ant binary` in hdfs bin directory isn't getting copied into the packaged binary, neither are the hdfs script files copied.
 - when i run `ant binary` in mapreduce, files aren't getting copied into the packaged bin directory

Verified the rest of the changes in all the three patches, they look good.

Still couldn't get the cluster up with different installation directories, will be trying in the meanwhile..

> Move configuration and script files post split
> ----------------------------------------------
>
>                 Key: HADOOP-6794
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6794
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, scripts
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6794.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8610-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 13:46:04 2010
Return-Path: <common-issues-return-8610-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35944 invoked from network); 1 Jun 2010 13:46:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 13:46:04 -0000
Received: (qmail 66372 invoked by uid 500); 1 Jun 2010 13:46:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66345 invoked by uid 500); 1 Jun 2010 13:46:04 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66337 invoked by uid 99); 1 Jun 2010 13:46:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 13:46:04 +0000
X-ASF-Spam-Status: No, hits=-1484.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 13:46:03 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51Djg1D018148
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 13:45:43 GMT
Message-ID: <31097099.103491275399942806.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 09:45:42 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874001#action_12874001 ] 

Hadoop QA commented on HADOOP-6796:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446012/HADOOP-6796.patch
  against trunk revision 949658.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/556/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/556/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/556/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/556/console

This message is automatically generated.

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8611-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 14:19:07 2010
Return-Path: <common-issues-return-8611-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55091 invoked from network); 1 Jun 2010 14:19:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 14:19:06 -0000
Received: (qmail 53433 invoked by uid 500); 1 Jun 2010 14:19:06 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53404 invoked by uid 500); 1 Jun 2010 14:19:06 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53391 invoked by uid 99); 1 Jun 2010 14:19:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 14:19:06 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 14:19:04 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51EIgGb018618
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 14:18:43 GMT
Message-ID: <18656243.104141275401922940.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 10:18:42 -0400 (EDT)
From: "Sam Walker (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6472) add tokenCache option to
 GenericOptionsParser for passing file with secret keys to a map reduce job
In-Reply-To: <1407870644.1262042429493.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874022#action_12874022 ] 

Sam Walker commented on HADOOP-6472:
------------------------------------

Alpha Bravo Charlie

> add tokenCache option to GenericOptionsParser for passing file with secret keys to a map reduce job
> ---------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6472
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6472
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6472-1.patch, HADOOP-6472-2.patch, HADOOP-6472.patch
>
>
> for MAPRED-1338 - we need an option to pass a file with secreteKeys (tokens) to a mapreduce Job.
> Name of the file is set into the config and will be picked up by JobSubmiter.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8612-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 14:19:10 2010
Return-Path: <common-issues-return-8612-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55189 invoked from network); 1 Jun 2010 14:19:10 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 14:19:10 -0000
Received: (qmail 53591 invoked by uid 500); 1 Jun 2010 14:19:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53550 invoked by uid 500); 1 Jun 2010 14:19:10 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53542 invoked by uid 99); 1 Jun 2010 14:19:10 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 14:19:10 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 14:19:07 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51EIk9l018654
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 14:18:46 GMT
Message-ID: <13649709.104261275401926247.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 10:18:46 -0400 (EDT)
From: "Sam Walker (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6308) make number of IPC accepts
 configurable
In-Reply-To: <1628092885.1255130671325.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874023#action_12874023 ] 

Sam Walker commented on HADOOP-6308:
------------------------------------

Alpha Bravo Charlie

> make number of IPC accepts configurable
> ---------------------------------------
>
>                 Key: HADOOP-6308
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6308
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.20.0
>         Environment: Linux, running Yahoo-based 0.20
>            Reporter: Andrew Ryan
>         Attachments: HADOOP-6308.patch
>
>
> We were recently seeing issues in our environments where HDFS clients would experience RST's from the NN when trying to do RPC to get file info, which would cause the task to fatal out. After some debugging we identified this to be that the IPC server listen queue -- ipc.server.listen.queue.size -- was far too low, we had been using the default value of 128 and found we needed to bump it up to 10240 before resets went away (although this value is a bit suspect, as I will explain later in the issue).
> When a large map job starts, lots of clients very quickly start to issue RPC requests to the namenode, which creates this listen queue filling up problem, because clients are opening connections faster than Hadoop's RPC server can process them. We went back to our 0.17 cluster and instrumented that with tcpdump and found that we had been sending RST's for a long time there, but the retry handling was implemented differently back in 0.17 so a single TCP failure wasn't task-fatal.
> In our environment we have our TCP stack set to explicitly send resets when the listen queue gets overflowed (syctl net.ipv4.tcp_abort_on_overflow = 1), default linux behavior is to start dropping SYN packets and let the client retransmit. Other people may be experiencing this issue and not noticing it because they are using the default behavior, which is to let the NN drop packets on the floor and let clients retransmit.
> So we've identified (at least) 3 improvements that can be made here:
> 1) In src/core/org/apache/hadoop/ipc/Server.java, Listener.doAccept() is currently hardcoded to do 10 accept()'s at a time, then it will start to read. We feel that it would be better to allow the server to be configured to support more than 10 accept's at one time using a configurable parameter. We can still leave 10 as the default.
> 2) Increase the default value of ipc.server.listen.queue.size from 128, or at least document that people with larger clusters starting thousands of mappers at once should increase this value. I wonder if a lot of people running larger clusters are dropping packets and don't realize it because TCP is covering them up. One one hand, yay TCP, on the other hand, those are needless delays and retries because the server can handle more connections.
> 3) Document that ipc.server.listen.queue.size may be limited to the value of SOMAXCONN (linux sysctl net.core.somaxconn ; default 4096 on our systems). The Java docs are not completely clear about this, and it's difficult to test because you can't query the backlog of a listening socket. We were under some time pressure in our case and tried 1024 which was not enough, and 10240 which worked, so we stuck with that.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8613-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 14:49:06 2010
Return-Path: <common-issues-return-8613-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65135 invoked from network); 1 Jun 2010 14:49:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 14:49:06 -0000
Received: (qmail 7800 invoked by uid 500); 1 Jun 2010 14:49:06 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7760 invoked by uid 500); 1 Jun 2010 14:49:06 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7752 invoked by uid 99); 1 Jun 2010 14:49:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 14:49:06 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 14:49:04 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51EmgXn019054
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 14:48:42 GMT
Message-ID: <11426086.105111275403722615.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 10:48:42 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874036#action_12874036 ] 

Hairong Kuang commented on HADOOP-6668:
---------------------------------------

Hi Tom, yes, I see your use case, but this is not the intended use when we designed those interfaces. I think we made a mistake to let FsDataInputStream/FsDataOutputStream implement those interfaces. 

As we discussed in our meeting, shall we mark them as "Public evolving"?

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: common.interface-tw-annotated.txt, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8614-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 16:12:00 2010
Return-Path: <common-issues-return-8614-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2790 invoked from network); 1 Jun 2010 16:12:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 16:12:00 -0000
Received: (qmail 43538 invoked by uid 500); 1 Jun 2010 16:12:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43513 invoked by uid 500); 1 Jun 2010 16:12:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43505 invoked by uid 99); 1 Jun 2010 16:12:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 16:12:00 +0000
X-ASF-Spam-Status: No, hits=-1485.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 16:11:59 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51GBcBF020200
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 16:11:39 GMT
Message-ID: <20901164.107161275408698759.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 12:11:38 -0400 (EDT)
From: "Sanjay Radia (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874089#action_12874089 ] 

Sanjay Radia commented on HADOOP-6668:
--------------------------------------

> PositionedReadable, Seekable and Syncable:

The definitions of the methods are in the PositionedReadable, Seekable and Syncable Java interfaces. 
The definitions of the methods are needed in the user javadoc; only APis's marked as audience public
appear in the user javadoc.
The user can reasonably  expect to  lookup  an extended  interfaces of an audience-public API. 
The only way to avoid this is to actually copy the method signatures but not extend the interfaces.
Unless we are prepared to do that, I think  we are stuck with making them audience public. 
Hairong is probably right that the originally these interfaces were targeted for the implementor.

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: common.interface-tw-annotated.txt, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8615-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 18:26:02 2010
Return-Path: <common-issues-return-8615-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62665 invoked from network); 1 Jun 2010 18:26:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 18:26:01 -0000
Received: (qmail 59098 invoked by uid 500); 1 Jun 2010 18:26:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58977 invoked by uid 500); 1 Jun 2010 18:26:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58969 invoked by uid 99); 1 Jun 2010 18:26:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 18:26:01 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 18:25:59 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51IPbbR021650
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 18:25:37 GMT
Message-ID: <19712300.110201275416737755.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 14:25:37 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6798) Align Ivy version for all Hadoop
 subprojects.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Align Ivy version for all Hadoop subprojects.
---------------------------------------------

                 Key: HADOOP-6798
                 URL: https://issues.apache.org/jira/browse/HADOOP-6798
             Project: Hadoop Common
          Issue Type: Improvement
          Components: build
    Affects Versions: 0.21.0
            Reporter: Konstantin Boudnik
            Assignee: Konstantin Boudnik


HDFS-1177 and MAPREDUCE-1830 are upgrading Ivy to 2.1.0. Common still has Ivy version at 2.1.0-rc1

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8616-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 18:29:57 2010
Return-Path: <common-issues-return-8616-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63400 invoked from network); 1 Jun 2010 18:29:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 18:29:57 -0000
Received: (qmail 62943 invoked by uid 500); 1 Jun 2010 18:29:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62825 invoked by uid 500); 1 Jun 2010 18:29:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62816 invoked by uid 99); 1 Jun 2010 18:29:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 18:29:57 +0000
X-ASF-Spam-Status: No, hits=-1486.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 18:29:57 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51ITa1G021682
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 18:29:36 GMT
Message-ID: <8841837.110281275416976906.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 14:29:36 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6798) Align Ivy version for all Hadoop
 subprojects.
In-Reply-To: <19712300.110201275416737755.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6798:
---------------------------------------

           Status: Patch Available  (was: Open)
    Fix Version/s: 0.21.0

> Align Ivy version for all Hadoop subprojects.
> ---------------------------------------------
>
>                 Key: HADOOP-6798
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6798
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6798.patch
>
>
> HDFS-1177 and MAPREDUCE-1830 are upgrading Ivy to 2.1.0. Common still has Ivy version at 2.1.0-rc1

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8617-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 18:30:00 2010
Return-Path: <common-issues-return-8617-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63455 invoked from network); 1 Jun 2010 18:30:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 18:30:00 -0000
Received: (qmail 63019 invoked by uid 500); 1 Jun 2010 18:30:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62970 invoked by uid 500); 1 Jun 2010 18:30:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62962 invoked by uid 99); 1 Jun 2010 18:30:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 18:30:00 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 18:29:58 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51ITauE021676
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 18:29:36 GMT
Message-ID: <11940910.110261275416976448.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 14:29:36 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6798) Align Ivy version for all Hadoop
 subprojects.
In-Reply-To: <19712300.110201275416737755.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6798:
---------------------------------------

    Attachment: hadoop-6798.patch

Version update.

> Align Ivy version for all Hadoop subprojects.
> ---------------------------------------------
>
>                 Key: HADOOP-6798
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6798
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6798.patch
>
>
> HDFS-1177 and MAPREDUCE-1830 are upgrading Ivy to 2.1.0. Common still has Ivy version at 2.1.0-rc1

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8618-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 19:12:02 2010
Return-Path: <common-issues-return-8618-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79469 invoked from network); 1 Jun 2010 19:12:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 19:12:02 -0000
Received: (qmail 37652 invoked by uid 500); 1 Jun 2010 19:12:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37627 invoked by uid 500); 1 Jun 2010 19:12:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37619 invoked by uid 99); 1 Jun 2010 19:12:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 19:12:01 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 19:11:59 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51JBc66022299
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 19:11:38 GMT
Message-ID: <13004283.111641275419498053.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 15:11:38 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-6787:
----------------------------

    Attachment: hadoop-6787-y20s-v1.patch

Patches for y20s branch.

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Luke Lu
>         Attachments: hadoop-6787-y20s-v1.patch
>
>
> The GlobFilter class (especially the error-prone setRegex method) is duplicated in both FileContext and Filesystem. And I'd like to be able to reuse the most of the setRegex code for glob pattern matching in metrics filter configurations.
> We could at least factor out the hairy setRegex code and package them into a GlobPattern class with a minimum compile method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8619-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 19:25:01 2010
Return-Path: <common-issues-return-8619-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83598 invoked from network); 1 Jun 2010 19:25:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 19:25:01 -0000
Received: (qmail 56811 invoked by uid 500); 1 Jun 2010 19:25:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56773 invoked by uid 500); 1 Jun 2010 19:25:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56765 invoked by uid 99); 1 Jun 2010 19:25:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 19:25:01 +0000
X-ASF-Spam-Status: No, hits=-1486.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 19:25:01 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51JOf8I022573
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 19:24:41 GMT
Message-ID: <10861124.112401275420281061.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 15:24:41 -0400 (EDT)
From: "Dmytro Molkov (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874175#action_12874175 ] 

Dmytro Molkov commented on HADOOP-6796:
---------------------------------------

Did you think of using an empty string instead of root for path? like other strings (owner, group) do? 

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8620-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 19:26:57 2010
Return-Path: <common-issues-return-8620-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84236 invoked from network); 1 Jun 2010 19:26:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 19:26:57 -0000
Received: (qmail 60381 invoked by uid 500); 1 Jun 2010 19:26:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60332 invoked by uid 500); 1 Jun 2010 19:26:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60324 invoked by uid 99); 1 Jun 2010 19:26:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 19:26:57 +0000
X-ASF-Spam-Status: No, hits=-1486.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 19:26:56 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51JQad3022591
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 19:26:36 GMT
Message-ID: <8394214.112431275420396544.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 15:26:36 -0400 (EDT)
From: "Scott Chen (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874176#action_12874176 ] 

Scott Chen commented on HADOOP-6796:
------------------------------------

Rodrigo: Will returning "/" change some of the existing behavior? Is there any code path that does the following?
{code}
if (getPath() == null) {
 ...
}
{code}

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8621-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 19:30:59 2010
Return-Path: <common-issues-return-8621-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84986 invoked from network); 1 Jun 2010 19:30:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 19:30:58 -0000
Received: (qmail 65421 invoked by uid 500); 1 Jun 2010 19:30:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65378 invoked by uid 500); 1 Jun 2010 19:30:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65370 invoked by uid 99); 1 Jun 2010 19:30:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 19:30:58 +0000
X-ASF-Spam-Status: No, hits=-1486.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 19:30:58 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51JUbUN022623
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 19:30:38 GMT
Message-ID: <2075776.112521275420637654.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 15:30:37 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-6787:
----------------------------

    Attachment: hadoop-6787-trunk-v1.patch

Ported to trunk

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-y20s-v1.patch
>
>
> The GlobFilter class (especially the error-prone setRegex method) is duplicated in both FileContext and Filesystem. And I'd like to be able to reuse the most of the setRegex code for glob pattern matching in metrics filter configurations.
> We could at least factor out the hairy setRegex code and package them into a GlobPattern class with a minimum compile method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8622-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 19:31:02 2010
Return-Path: <common-issues-return-8622-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85032 invoked from network); 1 Jun 2010 19:31:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 19:31:02 -0000
Received: (qmail 65562 invoked by uid 500); 1 Jun 2010 19:31:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65527 invoked by uid 500); 1 Jun 2010 19:31:02 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65519 invoked by uid 99); 1 Jun 2010 19:31:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 19:31:02 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 19:31:00 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51JUcuc022638
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 19:30:38 GMT
Message-ID: <10518487.112571275420638640.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 15:30:38 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-6787:
----------------------------

           Status: Patch Available  (was: Open)
         Assignee: Luke Lu
    Fix Version/s: 0.22.0

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-y20s-v1.patch
>
>
> The GlobFilter class (especially the error-prone setRegex method) is duplicated in both FileContext and Filesystem. And I'd like to be able to reuse the most of the setRegex code for glob pattern matching in metrics filter configurations.
> We could at least factor out the hairy setRegex code and package them into a GlobPattern class with a minimum compile method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8623-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 19:53:03 2010
Return-Path: <common-issues-return-8623-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41648 invoked from network); 1 Jun 2010 19:53:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 19:53:03 -0000
Received: (qmail 4028 invoked by uid 500); 1 Jun 2010 19:53:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3999 invoked by uid 500); 1 Jun 2010 19:53:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3991 invoked by uid 99); 1 Jun 2010 19:53:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 19:53:03 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 19:53:01 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51Jqd1L023055
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 19:52:39 GMT
Message-ID: <9749650.113701275421959463.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 15:52:39 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5438) Merge FileSystem.create and
 FileSystem.append
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874200#action_12874200 ] 

Eli Collins commented on HADOOP-5438:
-------------------------------------

HDFS-609 (create with the append flag doesn't work) is marked as a blocker for 21. Since the combined create/append API does not work and there's still some debate as to what the right API is it seems like the right thing to do here is revert this change for 21 (and on trunk) and figure out the right API before we put it in an official release. Reasonable?

> Merge FileSystem.create and FileSystem.append
> ---------------------------------------------
>
>                 Key: HADOOP-5438
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5438
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: He Yongqiang
>            Assignee: He Yongqiang
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-5438(2009-04-06).patch, Hadoop-5438-2009-03-30.patch, Hadoop-5438-2009-03-31-2.patch, Hadoop-5438-2009-03-31.patch, Hadoop-5438-2009-05-10.patch, Hadoop-5438-2009-05-15.patch, Hadoop-5438-2009-05-19.patch, Hadoop-5438-2009-05-5.patch
>
>
> Currently, when a user wants to modify a file, the user first calls exists() to know if this file is already there. And then uses create() or append() according to whether the file exists or not.
> the code looks like:
> {code}
> FSDataOutputStream out_1 = null;
> if (fs.exists(path_1))
>    out_1 = fs.append(path_1);
> else
>    out_1 = fs.create(path_1);
> {code}
> . On the performace side,It involes two RPCs. On the easy-of-use side, it is not very convient in contrast to the traditional open interface.
> It will more complicate if there is a overwrite parameter specified. I donot know whether there is a bug about 'overwrite' in 0.19, some times it takes a long time for overwrite creates to reture. So i make the write file code with overwrite param works like:
> {code}
> boolean exists = fs.exists(name);
> if (overwrite) {
>     if (exists)
>        fs.delete(name, true);
>      this.out = fs.create(name, overwrite, bufferSize, replication,
> 				    blockSize, progress);
>      this.currentRowID = 0;
>  } else {
>    if (!exists)
> 	this.out = fs.create(name, overwrite, bufferSize,
> 					replication, blockSize, progress);
>    else
> 	this.out = fs.append(name, bufferSize, progress);
> {code}
> Some code statements there are really redundant and not needed, especialy with the delete(). But without deleting first, the overwrite takes a long time to reture.
> BTW, i will create another issue about the overwrite problem. If it is not a bug at all or a duplicate, someone please close it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8624-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 19:57:02 2010
Return-Path: <common-issues-return-8624-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42202 invoked from network); 1 Jun 2010 19:57:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 19:57:02 -0000
Received: (qmail 7496 invoked by uid 500); 1 Jun 2010 19:57:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7462 invoked by uid 500); 1 Jun 2010 19:57:02 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7454 invoked by uid 99); 1 Jun 2010 19:57:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 19:57:02 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 19:56:59 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51Juc6L023093
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 19:56:38 GMT
Message-ID: <31138921.113771275422198156.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 15:56:38 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874203#action_12874203 ] 

Hadoop QA commented on HADOOP-6787:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446049/hadoop-6787-trunk-v1.patch
  against trunk revision 949658.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 2 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/71/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/71/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/71/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/71/console

This message is automatically generated.

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-y20s-v1.patch
>
>
> The GlobFilter class (especially the error-prone setRegex method) is duplicated in both FileContext and Filesystem. And I'd like to be able to reuse the most of the setRegex code for glob pattern matching in metrics filter configurations.
> We could at least factor out the hairy setRegex code and package them into a GlobPattern class with a minimum compile method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8625-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 20:01:04 2010
Return-Path: <common-issues-return-8625-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43909 invoked from network); 1 Jun 2010 20:01:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 20:01:03 -0000
Received: (qmail 15947 invoked by uid 500); 1 Jun 2010 20:01:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15912 invoked by uid 500); 1 Jun 2010 20:01:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15904 invoked by uid 99); 1 Jun 2010 20:01:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 20:01:03 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 20:01:01 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51K0eqb023194
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 20:00:40 GMT
Message-ID: <28918255.113891275422440006.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 16:00:40 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874204#action_12874204 ] 

Eli Collins commented on HADOOP-6787:
-------------------------------------

Looks good.

Nit: not your change but public GlobFilter methods need javadocs, spaces in between the end of a method and start of the next.


> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-y20s-v1.patch
>
>
> The GlobFilter class (especially the error-prone setRegex method) is duplicated in both FileContext and Filesystem. And I'd like to be able to reuse the most of the setRegex code for glob pattern matching in metrics filter configurations.
> We could at least factor out the hairy setRegex code and package them into a GlobPattern class with a minimum compile method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8626-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 20:16:59 2010
Return-Path: <common-issues-return-8626-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50752 invoked from network); 1 Jun 2010 20:16:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 20:16:59 -0000
Received: (qmail 33220 invoked by uid 500); 1 Jun 2010 20:16:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33196 invoked by uid 500); 1 Jun 2010 20:16:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33188 invoked by uid 99); 1 Jun 2010 20:16:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 20:16:58 +0000
X-ASF-Spam-Status: No, hits=-1486.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 20:16:58 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51KGbM9023405
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 20:16:38 GMT
Message-ID: <24281340.114321275423397870.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 16:16:37 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874212#action_12874212 ] 

Eli Collins commented on HADOOP-6796:
-------------------------------------

Why does anyone create a FileStatus with a null path?  Seems like we should make those callers all pass in "/" if that's the intended behavior rather than automatically convert null to Path("/") in the constructor.  Adding a path != null assertion in the constructor should prevent future cases of people passing in null.  

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8627-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 20:18:18 2010
Return-Path: <common-issues-return-8627-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51221 invoked from network); 1 Jun 2010 20:18:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 20:18:17 -0000
Received: (qmail 34203 invoked by uid 500); 1 Jun 2010 20:18:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34175 invoked by uid 500); 1 Jun 2010 20:18:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34167 invoked by uid 99); 1 Jun 2010 20:18:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 20:18:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 20:18:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51KHruu023446
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 20:17:53 GMT
Message-ID: <25732532.114421275423473316.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 16:17:53 -0400 (EDT)
From: "He Yongqiang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5438) Merge FileSystem.create and
 FileSystem.append
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874213#action_12874213 ] 

He Yongqiang commented on HADOOP-5438:
--------------------------------------

Here is not about append and create. it is also create and overwrite.

> Merge FileSystem.create and FileSystem.append
> ---------------------------------------------
>
>                 Key: HADOOP-5438
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5438
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: He Yongqiang
>            Assignee: He Yongqiang
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-5438(2009-04-06).patch, Hadoop-5438-2009-03-30.patch, Hadoop-5438-2009-03-31-2.patch, Hadoop-5438-2009-03-31.patch, Hadoop-5438-2009-05-10.patch, Hadoop-5438-2009-05-15.patch, Hadoop-5438-2009-05-19.patch, Hadoop-5438-2009-05-5.patch
>
>
> Currently, when a user wants to modify a file, the user first calls exists() to know if this file is already there. And then uses create() or append() according to whether the file exists or not.
> the code looks like:
> {code}
> FSDataOutputStream out_1 = null;
> if (fs.exists(path_1))
>    out_1 = fs.append(path_1);
> else
>    out_1 = fs.create(path_1);
> {code}
> . On the performace side,It involes two RPCs. On the easy-of-use side, it is not very convient in contrast to the traditional open interface.
> It will more complicate if there is a overwrite parameter specified. I donot know whether there is a bug about 'overwrite' in 0.19, some times it takes a long time for overwrite creates to reture. So i make the write file code with overwrite param works like:
> {code}
> boolean exists = fs.exists(name);
> if (overwrite) {
>     if (exists)
>        fs.delete(name, true);
>      this.out = fs.create(name, overwrite, bufferSize, replication,
> 				    blockSize, progress);
>      this.currentRowID = 0;
>  } else {
>    if (!exists)
> 	this.out = fs.create(name, overwrite, bufferSize,
> 					replication, blockSize, progress);
>    else
> 	this.out = fs.append(name, bufferSize, progress);
> {code}
> Some code statements there are really redundant and not needed, especialy with the delete(). But without deleting first, the overwrite takes a long time to reture.
> BTW, i will create another issue about the overwrite problem. If it is not a bug at all or a duplicate, someone please close it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8628-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 20:26:03 2010
Return-Path: <common-issues-return-8628-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53794 invoked from network); 1 Jun 2010 20:26:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 20:26:03 -0000
Received: (qmail 47667 invoked by uid 500); 1 Jun 2010 20:26:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47564 invoked by uid 500); 1 Jun 2010 20:26:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47556 invoked by uid 99); 1 Jun 2010 20:26:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 20:26:03 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 20:26:01 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51KPduc023562
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 20:25:39 GMT
Message-ID: <9687148.114701275423939843.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 16:25:39 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5438) Merge FileSystem.create and
 FileSystem.append
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874217#action_12874217 ] 

Eli Collins commented on HADOOP-5438:
-------------------------------------

Does it make sense to do a release with the current API, ie is the merge of the APIs ready enough to release?  From the last thread of comments it sounds like it's not clear what API people want (eg vararg create flags).

> Merge FileSystem.create and FileSystem.append
> ---------------------------------------------
>
>                 Key: HADOOP-5438
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5438
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: He Yongqiang
>            Assignee: He Yongqiang
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-5438(2009-04-06).patch, Hadoop-5438-2009-03-30.patch, Hadoop-5438-2009-03-31-2.patch, Hadoop-5438-2009-03-31.patch, Hadoop-5438-2009-05-10.patch, Hadoop-5438-2009-05-15.patch, Hadoop-5438-2009-05-19.patch, Hadoop-5438-2009-05-5.patch
>
>
> Currently, when a user wants to modify a file, the user first calls exists() to know if this file is already there. And then uses create() or append() according to whether the file exists or not.
> the code looks like:
> {code}
> FSDataOutputStream out_1 = null;
> if (fs.exists(path_1))
>    out_1 = fs.append(path_1);
> else
>    out_1 = fs.create(path_1);
> {code}
> . On the performace side,It involes two RPCs. On the easy-of-use side, it is not very convient in contrast to the traditional open interface.
> It will more complicate if there is a overwrite parameter specified. I donot know whether there is a bug about 'overwrite' in 0.19, some times it takes a long time for overwrite creates to reture. So i make the write file code with overwrite param works like:
> {code}
> boolean exists = fs.exists(name);
> if (overwrite) {
>     if (exists)
>        fs.delete(name, true);
>      this.out = fs.create(name, overwrite, bufferSize, replication,
> 				    blockSize, progress);
>      this.currentRowID = 0;
>  } else {
>    if (!exists)
> 	this.out = fs.create(name, overwrite, bufferSize,
> 					replication, blockSize, progress);
>    else
> 	this.out = fs.append(name, bufferSize, progress);
> {code}
> Some code statements there are really redundant and not needed, especialy with the delete(). But without deleting first, the overwrite takes a long time to reture.
> BTW, i will create another issue about the overwrite problem. If it is not a bug at all or a duplicate, someone please close it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8629-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 20:29:03 2010
Return-Path: <common-issues-return-8629-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54755 invoked from network); 1 Jun 2010 20:29:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 20:29:03 -0000
Received: (qmail 50783 invoked by uid 500); 1 Jun 2010 20:29:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50740 invoked by uid 500); 1 Jun 2010 20:29:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50732 invoked by uid 99); 1 Jun 2010 20:29:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 20:29:03 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 20:29:00 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51KSdx6023593
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 20:28:39 GMT
Message-ID: <30061067.114771275424119353.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 16:28:39 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874219#action_12874219 ] 

Luke Lu commented on HADOOP-6787:
---------------------------------

Thanks for the review, Eli. Not sure I completely follow your comment. The GlobFilter class is not a public class. The visibility is package only except for the accept method from the PathFilter interface which has javadoc.

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-y20s-v1.patch
>
>
> The GlobFilter class (especially the error-prone setRegex method) is duplicated in both FileContext and Filesystem. And I'd like to be able to reuse the most of the setRegex code for glob pattern matching in metrics filter configurations.
> We could at least factor out the hairy setRegex code and package them into a GlobPattern class with a minimum compile method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8630-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 20:48:59 2010
Return-Path: <common-issues-return-8630-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59540 invoked from network); 1 Jun 2010 20:48:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 20:48:59 -0000
Received: (qmail 76002 invoked by uid 500); 1 Jun 2010 20:48:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75961 invoked by uid 500); 1 Jun 2010 20:48:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75953 invoked by uid 99); 1 Jun 2010 20:48:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 20:48:59 +0000
X-ASF-Spam-Status: No, hits=-1486.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 20:48:58 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51KmcKB023816
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 20:48:38 GMT
Message-ID: <14375792.115211275425318572.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 16:48:38 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874228#action_12874228 ] 

Rodrigo Schmidt commented on HADOOP-6796:
-----------------------------------------

@Dmytro: srcPath is currently defined as a Path object and I don't want to change that. Calling 'new Path("")', crashes because "" is not a valid path. That's why I defaulted it to "/"

@Scott: No checks for getPath() == null afaik, on the other hand, if srcPath is null, all the method calls I mentioned will crash.

@Eli: You create a FileStatus with a null srcPath if you do 'stat = new FileStatus();'. This patch actually changes srcPath to Path("/") whenever it's null, which is exactly the behavior you proposed, but without triggering the exception when someone does that. I don't think passing null should be prevented, since we don't do that for other parameters like "owner" or "group". In such cases we default the value to empty strings. The equivalent for srcPath should be Path("/") IMO.

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8631-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 21:51:59 2010
Return-Path: <common-issues-return-8631-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77361 invoked from network); 1 Jun 2010 21:51:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 21:51:59 -0000
Received: (qmail 56074 invoked by uid 500); 1 Jun 2010 21:51:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56031 invoked by uid 500); 1 Jun 2010 21:51:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56023 invoked by uid 99); 1 Jun 2010 21:51:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 21:51:58 +0000
X-ASF-Spam-Status: No, hits=-1486.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 21:51:58 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51Lpbd5024534
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 21:51:37 GMT
Message-ID: <8327090.116311275429097727.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 17:51:37 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6798) Align Ivy version for all Hadoop
 subprojects.
In-Reply-To: <19712300.110201275416737755.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874257#action_12874257 ] 

Hadoop QA commented on HADOOP-6798:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446038/hadoop-6798.patch
  against trunk revision 949658.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/557/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/557/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/557/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/557/console

This message is automatically generated.

> Align Ivy version for all Hadoop subprojects.
> ---------------------------------------------
>
>                 Key: HADOOP-6798
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6798
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6798.patch
>
>
> HDFS-1177 and MAPREDUCE-1830 are upgrading Ivy to 2.1.0. Common still has Ivy version at 2.1.0-rc1

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8632-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 22:02:08 2010
Return-Path: <common-issues-return-8632-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79920 invoked from network); 1 Jun 2010 22:02:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 22:02:08 -0000
Received: (qmail 75410 invoked by uid 500); 1 Jun 2010 22:02:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75369 invoked by uid 500); 1 Jun 2010 22:02:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75361 invoked by uid 99); 1 Jun 2010 22:02:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 22:02:08 +0000
X-ASF-Spam-Status: No, hits=-1487.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 22:02:07 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51M1lwd024727
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 22:01:47 GMT
Message-ID: <24504196.116651275429707170.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 18:01:47 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6798) Align Ivy version for all Hadoop
 subprojects.
In-Reply-To: <19712300.110201275416737755.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874265#action_12874265 ] 

Tom White commented on HADOOP-6798:
-----------------------------------

+1

> Align Ivy version for all Hadoop subprojects.
> ---------------------------------------------
>
>                 Key: HADOOP-6798
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6798
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6798.patch
>
>
> HDFS-1177 and MAPREDUCE-1830 are upgrading Ivy to 2.1.0. Common still has Ivy version at 2.1.0-rc1

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8633-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 22:15:59 2010
Return-Path: <common-issues-return-8633-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86663 invoked from network); 1 Jun 2010 22:15:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 22:15:58 -0000
Received: (qmail 99731 invoked by uid 500); 1 Jun 2010 22:15:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99705 invoked by uid 500); 1 Jun 2010 22:15:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99697 invoked by uid 99); 1 Jun 2010 22:15:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 22:15:58 +0000
X-ASF-Spam-Status: No, hits=-1487.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 22:15:58 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51MFcSj024879
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 22:15:38 GMT
Message-ID: <23792106.117041275430538032.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 18:15:38 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874275#action_12874275 ] 

Eli Collins commented on HADOOP-6796:
-------------------------------------

Shouldn't people who want a FileStatus that represents root just pass in "/"?  It seems weird that new FileStatus() gives you a file status for root.

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8634-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 22:16:02 2010
Return-Path: <common-issues-return-8634-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86707 invoked from network); 1 Jun 2010 22:16:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 22:16:02 -0000
Received: (qmail 99885 invoked by uid 500); 1 Jun 2010 22:16:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99855 invoked by uid 500); 1 Jun 2010 22:16:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99847 invoked by uid 99); 1 Jun 2010 22:16:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 22:16:01 +0000
X-ASF-Spam-Status: No, hits=-1487.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 22:16:01 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51MFfu0024897
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 22:15:41 GMT
Message-ID: <22264929.117101275430541263.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 18:15:41 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6794) Move configuration and script files
 post split
In-Reply-To: <11039662.93131275342217664.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6794:
------------------------------

    Attachment: HADOOP-6794.patch

Vinod, many thanks for the thorough review. I've made fixes for all the errors you pointed out. Common's conf/log4j.properties is now a merged superset of the log4j.properties - I think we can split it out further later as needed.

I tried running with different installation directories, and successfully ran a job in pseudo-distributed mode. However, I did notice that the web UIs were not working, which I think will be fixed as a part of HADOOP-6461, although I'm still investigating that. 

> Move configuration and script files post split
> ----------------------------------------------
>
>                 Key: HADOOP-6794
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6794
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, scripts
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6794.patch, HADOOP-6794.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8635-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 22:19:58 2010
Return-Path: <common-issues-return-8635-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87908 invoked from network); 1 Jun 2010 22:19:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 22:19:58 -0000
Received: (qmail 6710 invoked by uid 500); 1 Jun 2010 22:19:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6677 invoked by uid 500); 1 Jun 2010 22:19:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6669 invoked by uid 99); 1 Jun 2010 22:19:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 22:19:58 +0000
X-ASF-Spam-Status: No, hits=-1487.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 22:19:58 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51MJbec024993
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 22:19:37 GMT
Message-ID: <9252686.117391275430777470.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 18:19:37 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874282#action_12874282 ] 

Eli Collins commented on HADOOP-6787:
-------------------------------------

You're right, you don't need full javadocs for those methods, just suggesting a comment explaining what the public methods do would be useful, but feel free to defer.

+1 for the patch.

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-y20s-v1.patch
>
>
> The GlobFilter class (especially the error-prone setRegex method) is duplicated in both FileContext and Filesystem. And I'd like to be able to reuse the most of the setRegex code for glob pattern matching in metrics filter configurations.
> We could at least factor out the hairy setRegex code and package them into a GlobPattern class with a minimum compile method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8636-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 22:31:57 2010
Return-Path: <common-issues-return-8636-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89711 invoked from network); 1 Jun 2010 22:31:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 22:31:57 -0000
Received: (qmail 16422 invoked by uid 500); 1 Jun 2010 22:31:57 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16398 invoked by uid 500); 1 Jun 2010 22:31:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16390 invoked by uid 99); 1 Jun 2010 22:31:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 22:31:57 +0000
X-ASF-Spam-Status: No, hits=-1487.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 22:31:57 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51MVa3g025044
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 22:31:36 GMT
Message-ID: <32839972.117471275431496934.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 18:31:36 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874284#action_12874284 ] 

Rodrigo Schmidt commented on HADOOP-6796:
-----------------------------------------

What should be the srcPath for "new FileStatus()" then?

I don't think disabling this constructor is a good choice. As a Writable, people might want to create a new empty object and read from some DataOutput (that was exactly how I found the problem, by the way :-)).

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8637-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 23:28:04 2010
Return-Path: <common-issues-return-8637-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2337 invoked from network); 1 Jun 2010 23:28:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 23:28:01 -0000
Received: (qmail 68897 invoked by uid 500); 1 Jun 2010 23:28:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68862 invoked by uid 500); 1 Jun 2010 23:28:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68854 invoked by uid 99); 1 Jun 2010 23:28:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 23:28:01 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 23:27:58 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51NRbeS025313
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 23:27:37 GMT
Message-ID: <3577543.117891275434857006.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 19:27:37 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6799) GzipCodec/CompressionOutputStream
 resetState() fails to reset gzip header and CRC
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

GzipCodec/CompressionOutputStream resetState() fails to reset gzip header and CRC
---------------------------------------------------------------------------------

                 Key: HADOOP-6799
                 URL: https://issues.apache.org/jira/browse/HADOOP-6799
             Project: Hadoop Common
          Issue Type: Bug
          Components: io
    Affects Versions: 0.21.0
         Environment: Linux/x86-32, Java 1.6.0_15
            Reporter: Greg Roelofs
            Priority: Minor


    CompressionCodec gzip = new GzipCodec();
    Path fnHDFS = new Path(workDir, "concat" + gzip.getDefaultExtension());
    OutputStream out = localFs.create(fnHDFS);
    Compressor gzCmp = gzip.createCompressor();
    CompressionOutputStream gzOStm = gzip.createOutputStream(out, gzCmp);

    gzOStm.write("first gzip concat\n member\nwith three lines\n".getBytes());
    gzOStm.finish();
    gzOStm.resetState();
    gzOStm.write("2nd gzip concat member\n".getBytes());
    gzOStm.finish();
    gzOStm.resetState();
    gzOStm.write("gzip concat\nmember #3\n".getBytes());
    gzOStm.close();

This should create a 3-member concatenated gzip file (i.e., equivalent to concatenation of three individual gzip files).  However, resetState() simply resets the underlying deflate engine; it does not reset the running CRC-32, nor does it set a flag to write a new gzip header.  As a result, the output stream is corrupted--i.e., only the first member is readable.

I also have hex dumps, but they don't work well with proportional fonts.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8638-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 23:44:05 2010
Return-Path: <common-issues-return-8638-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12920 invoked from network); 1 Jun 2010 23:44:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 23:44:04 -0000
Received: (qmail 97384 invoked by uid 500); 1 Jun 2010 23:44:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97352 invoked by uid 500); 1 Jun 2010 23:44:04 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97344 invoked by uid 99); 1 Jun 2010 23:44:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 23:44:04 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 23:44:02 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51NheZm025478
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 23:43:40 GMT
Message-ID: <17550530.118241275435820614.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 19:43:40 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6798) Align Ivy version for all Hadoop
 subprojects.
In-Reply-To: <19712300.110201275416737755.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6798:
---------------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

I have just committed this to 0.21 and trunk.

> Align Ivy version for all Hadoop subprojects.
> ---------------------------------------------
>
>                 Key: HADOOP-6798
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6798
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6798.patch
>
>
> HDFS-1177 and MAPREDUCE-1830 are upgrading Ivy to 2.1.0. Common still has Ivy version at 2.1.0-rc1

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8639-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 01 23:52:04 2010
Return-Path: <common-issues-return-8639-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15276 invoked from network); 1 Jun 2010 23:52:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 1 Jun 2010 23:52:04 -0000
Received: (qmail 7944 invoked by uid 500); 1 Jun 2010 23:52:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7914 invoked by uid 500); 1 Jun 2010 23:52:04 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7905 invoked by uid 99); 1 Jun 2010 23:52:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 23:52:04 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Jun 2010 23:52:02 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o51Npe2k025583
	for <common-issues@hadoop.apache.org>; Tue, 1 Jun 2010 23:51:40 GMT
Message-ID: <21312797.118521275436300136.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 19:51:40 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874311#action_12874311 ] 

Luke Lu commented on HADOOP-6787:
---------------------------------

Thanks for the +1!

Most IDEs would indicate (in my case, with a green circle with I in it) an implementing method. Clicking or hovering on the indicator would show you the  javadoc of the interface method. I personally prefer keeping the javadocs in interfaces/abstract classes and not duplicating them in implementing classes.   I hope that it's not a bug in Hudson's javadoc tool to +1 my patch as well :)

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-y20s-v1.patch
>
>
> The GlobFilter class (especially the error-prone setRegex method) is duplicated in both FileContext and Filesystem. And I'd like to be able to reuse the most of the setRegex code for glob pattern matching in metrics filter configurations.
> We could at least factor out the hairy setRegex code and package them into a GlobPattern class with a minimum compile method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8640-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 00:03:59 2010
Return-Path: <common-issues-return-8640-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19115 invoked from network); 2 Jun 2010 00:03:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 00:03:59 -0000
Received: (qmail 23405 invoked by uid 500); 2 Jun 2010 00:03:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23284 invoked by uid 500); 2 Jun 2010 00:03:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23276 invoked by uid 99); 2 Jun 2010 00:03:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 00:03:58 +0000
X-ASF-Spam-Status: No, hits=-1487.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 00:03:58 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5203bSM025968
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 00:03:38 GMT
Message-ID: <24734203.118951275437017828.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 20:03:37 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6798) Align Ivy version for all Hadoop
 subprojects.
In-Reply-To: <19712300.110201275416737755.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874322#action_12874322 ] 

Hudson commented on HADOOP-6798:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #273 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/273/])
    HADOOP-6798. Align Ivy version for all Hadoop subprojects. Contributed by Konstantin Boudnik.


> Align Ivy version for all Hadoop subprojects.
> ---------------------------------------------
>
>                 Key: HADOOP-6798
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6798
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6798.patch
>
>
> HDFS-1177 and MAPREDUCE-1830 are upgrading Ivy to 2.1.0. Common still has Ivy version at 2.1.0-rc1

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8641-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 00:36:03 2010
Return-Path: <common-issues-return-8641-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35405 invoked from network); 2 Jun 2010 00:36:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 00:36:03 -0000
Received: (qmail 44128 invoked by uid 500); 2 Jun 2010 00:36:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44030 invoked by uid 500); 2 Jun 2010 00:36:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44022 invoked by uid 99); 2 Jun 2010 00:36:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 00:36:03 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 00:36:01 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o520ZdKB026226
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 00:35:39 GMT
Message-ID: <23616396.119531275438939306.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 20:35:39 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874338#action_12874338 ] 

Eli Collins commented on HADOOP-6796:
-------------------------------------

In cases where you want to explicitly tolerate objects that are only partially constructed after calling the constructor (ie the user is responsible for initializing the internals) it's often useful to provide an isValid method that returns true if all the requisite fields have been initialized and public methods assert isValid before they're called. This way there's no side effects (ie you forget to set the src path but it happens to work because a default value was specified, but the user doesn't know what that is without looking at the object's internals) and the assert clearly indicates what field needs to be set.  This feels like overkill to me. Objects that can be partially constructed are special cases where the user needs to know what fields are required, to me it's better to get an NPE indicating the user needs to set the source path than automatically constructing a path object for them using a value they may not be aware of.

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8642-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 00:38:05 2010
Return-Path: <common-issues-return-8642-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35742 invoked from network); 2 Jun 2010 00:38:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 00:38:05 -0000
Received: (qmail 45337 invoked by uid 500); 2 Jun 2010 00:38:05 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45219 invoked by uid 500); 2 Jun 2010 00:38:04 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45210 invoked by uid 99); 2 Jun 2010 00:38:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 00:38:04 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 00:38:02 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o520bfAk026266
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 00:37:41 GMT
Message-ID: <31127735.119641275439061204.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 20:37:41 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874341#action_12874341 ] 

Eli Collins commented on HADOOP-6787:
-------------------------------------

Thanks Luke.  I will commit this soon.

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-y20s-v1.patch
>
>
> The GlobFilter class (especially the error-prone setRegex method) is duplicated in both FileContext and Filesystem. And I'd like to be able to reuse the most of the setRegex code for glob pattern matching in metrics filter configurations.
> We could at least factor out the hairy setRegex code and package them into a GlobPattern class with a minimum compile method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8643-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 00:50:01 2010
Return-Path: <common-issues-return-8643-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37970 invoked from network); 2 Jun 2010 00:50:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 00:50:00 -0000
Received: (qmail 52139 invoked by uid 500); 2 Jun 2010 00:50:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52081 invoked by uid 500); 2 Jun 2010 00:50:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52073 invoked by uid 99); 2 Jun 2010 00:50:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 00:50:00 +0000
X-ASF-Spam-Status: No, hits=-1487.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 00:50:00 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o520ne1N026441
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 00:49:40 GMT
Message-ID: <1135501.120101275439780082.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 20:49:40 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874347#action_12874347 ] 

Rodrigo Schmidt commented on HADOOP-6796:
-----------------------------------------

So are you proposing to get rid of the FileStatus() constructor, and signal NPE when the user forgets to specify the srcPath?


> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8644-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 01:32:03 2010
Return-Path: <common-issues-return-8644-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48083 invoked from network); 2 Jun 2010 01:32:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 01:32:03 -0000
Received: (qmail 87507 invoked by uid 500); 2 Jun 2010 01:32:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87460 invoked by uid 500); 2 Jun 2010 01:32:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87451 invoked by uid 99); 2 Jun 2010 01:32:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 01:32:03 +0000
X-ASF-Spam-Status: No, hits=-1487.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 01:32:02 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o521Vgbq026901
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 01:31:42 GMT
Message-ID: <6697961.120871275442302595.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 21:31:42 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874362#action_12874362 ] 

Jakob Homan commented on HADOOP-6796:
-------------------------------------

bq. So are you proposing to get rid of the FileStatus() constructor, and signal NPE when the user forgets to specify the srcPath? 
FileStatus is a Writable and therefore requires a default constructor, unfortunately.  Were one to start looking at the classes that implement Writable, I imagine quite a few would be susceptible to this problem.  It's a limitation of Writable (and Java...)

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8645-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 01:32:05 2010
Return-Path: <common-issues-return-8645-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48140 invoked from network); 2 Jun 2010 01:32:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 01:32:04 -0000
Received: (qmail 87658 invoked by uid 500); 2 Jun 2010 01:32:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87621 invoked by uid 500); 2 Jun 2010 01:32:04 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87613 invoked by uid 99); 2 Jun 2010 01:32:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 01:32:04 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 01:32:02 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o521VehY026883
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 01:31:41 GMT
Message-ID: <22484741.120811275442300903.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 21:31:40 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874361#action_12874361 ] 

Eli Collins commented on HADOOP-6796:
-------------------------------------

If you can get rid of the FileStatus() constructor that's preferable since it's a useful invariant that objects are fully initialized after their constructors are called (users and methods don't need to worry about partially constructed objects) but that might not be possible or worth the trouble with FileStatus. I think it basically comes down to whether there is a good default value for a FileStatus path.  Getting an NPE or hitting an assert when you try to use a partially constructed object seems preferable to accidentally using a value that wasn't intended (ie make users specify the path for a file status). My 2c. 


> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8646-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 02:14:01 2010
Return-Path: <common-issues-return-8646-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58207 invoked from network); 2 Jun 2010 02:14:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 02:14:00 -0000
Received: (qmail 25202 invoked by uid 500); 2 Jun 2010 02:14:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25083 invoked by uid 500); 2 Jun 2010 02:14:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25069 invoked by uid 99); 2 Jun 2010 02:14:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 02:14:00 +0000
X-ASF-Spam-Status: No, hits=-1487.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 02:14:00 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o522DcQq027303
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 02:13:39 GMT
Message-ID: <2067745.121461275444818032.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 22:13:38 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874382#action_12874382 ] 

Rodrigo Schmidt commented on HADOOP-6796:
-----------------------------------------

Wow! I didn't know this would create so much discussion. I thought it was such a simple code cleanup and unit tests. :-)

@Eli: Unfortunately we can't get rid of the default constructor because FileStatus is a Writable(). So, I think your proposal boils down to just leaving it the way it is.

I don't like it because it leaves room for people to use the default constructor or a null srcPath for some corner case conditions and crash unexpectedly down the road with an NPE. 

So, my opinion is that any valid Path is better than null as a default value, and "/" seems reasonable.

The options are the following:

1) Leave it the way it is
2) Use Path("/") if the srcPath is null (current patch)
3) Start accepting a null srcPath and change the methods that currently crash because of it

My patch does (2) because I didn't think it was worth implementing (3). However, I definitely prefer (3) to (1), so I could definitely implement it.

What do the others think?


> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8647-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 03:18:04 2010
Return-Path: <common-issues-return-8647-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71501 invoked from network); 2 Jun 2010 03:18:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 03:18:04 -0000
Received: (qmail 64139 invoked by uid 500); 2 Jun 2010 03:18:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64097 invoked by uid 500); 2 Jun 2010 03:18:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64088 invoked by uid 99); 2 Jun 2010 03:18:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 03:18:03 +0000
X-ASF-Spam-Status: No, hits=-1487.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 03:18:02 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o523HfDv027743
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 03:17:42 GMT
Message-ID: <7055634.122021275448661261.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 23:17:41 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874396#action_12874396 ] 

Eli Collins commented on HADOOP-6796:
-------------------------------------

You could also modify Path to accept Path("") or Path(null) so that instead of throwing an exception they silently convert to Path("/"). That runs into the same rationale, ie it's useful that Path throws an exception when given invalid arguments instead of converting the path to something else (this way people know when they pass in a null by mistake).  You could also make sure FileStatus objects are created via a method that sets the required fields (ie use a factory).

What's the code that triggers this? If that code just wants a FileStatus that represents "/" then could you just modify that code to set the path explicitly to "/"?

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8648-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 03:39:59 2010
Return-Path: <common-issues-return-8648-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74827 invoked from network); 2 Jun 2010 03:39:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 03:39:59 -0000
Received: (qmail 77396 invoked by uid 500); 2 Jun 2010 03:39:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77368 invoked by uid 500); 2 Jun 2010 03:39:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77360 invoked by uid 99); 2 Jun 2010 03:39:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 03:39:58 +0000
X-ASF-Spam-Status: No, hits=-1487.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 03:39:57 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o523dbmb027846
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 03:39:37 GMT
Message-ID: <6077391.122231275449977385.JavaMail.jira@thor>
Date: Tue, 1 Jun 2010 23:39:37 -0400 (EDT)
From: "Balaji Rajagopalan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6788) Exception exclusion functionality
 is not working correctly.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874400#action_12874400 ] 

Balaji Rajagopalan commented on HADOOP-6788:
--------------------------------------------

Looks good to me. 

> Exception exclusion functionality is not working correctly.
> -----------------------------------------------------------
>
>                 Key: HADOOP-6788
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6788
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Balaji Rajagopalan
>         Attachments: 6788-ydist-security.patch, HADOOP-6788.patch
>
>
> Exception exclusion functionality is not working correctly because of that tests are failing by not matching the error count.
> I debugged the issue and found that the problem with shell command which is generating in the getNumberOfMatchesInLogFile function.
> Currently building the shell command in the following way. 
> if(list != null){
>   for(int i =0; i < list.length; ++i)
>   {
>     filePattern.append(" | grep -v " + list[i] );
>   }
> }
>     String[] cmd =
>         new String[] {
>             "bash",
>             "-c",
>             "grep -c "
>                 + pattern + " " + filePattern
>                 + " | awk -F: '{s+=$2} END {print s}'" };    
> However, The above commnad won't work correctly because you are counting the exceptions in the file before excluding the known exceptions.
> In this case it gives the mismatch error counts everytime.The shell command should be in the following way to work correctly.
> if (list != null) {
>   int index = 0;
>   for (String excludeExp : list) {
>     filePattern.append((++index < list.length)? "| grep -v " : 
>             "| grep -vc " + list[i] );  
>   }
> }
> String[] cmd =
>    new String[] {
>        "bash",
>        "-c",
>        "grep "
>            + pattern + " " + filePattern
>            + " | awk -F: '{s+=$2} END {print s}'" };  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8649-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 04:02:03 2010
Return-Path: <common-issues-return-8649-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78350 invoked from network); 2 Jun 2010 04:02:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 04:02:02 -0000
Received: (qmail 83095 invoked by uid 500); 2 Jun 2010 04:02:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83071 invoked by uid 500); 2 Jun 2010 04:02:02 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83061 invoked by uid 99); 2 Jun 2010 04:02:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 04:02:01 +0000
X-ASF-Spam-Status: No, hits=-1488.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 04:02:01 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5241fLM028102
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 04:01:41 GMT
Message-ID: <13477676.122491275451301004.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 00:01:41 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6788) Exception exclusion functionality
 is not working correctly.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874406#action_12874406 ] 

Konstantin Boudnik commented on HADOOP-6788:
--------------------------------------------

This looks like a whitespace modification to me:
{noformat}
-                + " | awk -F: '{s+=$2} END {print s}'" };    
+                + " | awk -F: '{s+=$2} END {print s}'" };
{noformat}
Please fix it. Other than that looks ok.


> Exception exclusion functionality is not working correctly.
> -----------------------------------------------------------
>
>                 Key: HADOOP-6788
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6788
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Balaji Rajagopalan
>         Attachments: 6788-ydist-security.patch, HADOOP-6788.patch
>
>
> Exception exclusion functionality is not working correctly because of that tests are failing by not matching the error count.
> I debugged the issue and found that the problem with shell command which is generating in the getNumberOfMatchesInLogFile function.
> Currently building the shell command in the following way. 
> if(list != null){
>   for(int i =0; i < list.length; ++i)
>   {
>     filePattern.append(" | grep -v " + list[i] );
>   }
> }
>     String[] cmd =
>         new String[] {
>             "bash",
>             "-c",
>             "grep -c "
>                 + pattern + " " + filePattern
>                 + " | awk -F: '{s+=$2} END {print s}'" };    
> However, The above commnad won't work correctly because you are counting the exceptions in the file before excluding the known exceptions.
> In this case it gives the mismatch error counts everytime.The shell command should be in the following way to work correctly.
> if (list != null) {
>   int index = 0;
>   for (String excludeExp : list) {
>     filePattern.append((++index < list.length)? "| grep -v " : 
>             "| grep -vc " + list[i] );  
>   }
> }
> String[] cmd =
>    new String[] {
>        "bash",
>        "-c",
>        "grep "
>            + pattern + " " + filePattern
>            + " | awk -F: '{s+=$2} END {print s}'" };  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8650-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 04:23:04 2010
Return-Path: <common-issues-return-8650-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85123 invoked from network); 2 Jun 2010 04:23:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 04:23:04 -0000
Received: (qmail 94976 invoked by uid 500); 2 Jun 2010 04:23:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94938 invoked by uid 500); 2 Jun 2010 04:23:04 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94930 invoked by uid 99); 2 Jun 2010 04:23:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 04:23:04 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 04:23:01 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o524Mdbp028619
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 04:22:39 GMT
Message-ID: <13448489.122651275452559611.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 00:22:39 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6788) Exception exclusion functionality
 is not working correctly.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874409#action_12874409 ] 

Vinay Kumar Thota commented on HADOOP-6788:
-------------------------------------------

Cos,
Yes, you are correct. Earlier there was four white spaces after that statement and I have removed them while modifying code,because I felt that its unnecessary to keep the white spaces after the statement.  I  just curious to know, is there a necessity to create a another patch for this? I feels that its not an issue right.



> Exception exclusion functionality is not working correctly.
> -----------------------------------------------------------
>
>                 Key: HADOOP-6788
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6788
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Balaji Rajagopalan
>         Attachments: 6788-ydist-security.patch, HADOOP-6788.patch
>
>
> Exception exclusion functionality is not working correctly because of that tests are failing by not matching the error count.
> I debugged the issue and found that the problem with shell command which is generating in the getNumberOfMatchesInLogFile function.
> Currently building the shell command in the following way. 
> if(list != null){
>   for(int i =0; i < list.length; ++i)
>   {
>     filePattern.append(" | grep -v " + list[i] );
>   }
> }
>     String[] cmd =
>         new String[] {
>             "bash",
>             "-c",
>             "grep -c "
>                 + pattern + " " + filePattern
>                 + " | awk -F: '{s+=$2} END {print s}'" };    
> However, The above commnad won't work correctly because you are counting the exceptions in the file before excluding the known exceptions.
> In this case it gives the mismatch error counts everytime.The shell command should be in the following way to work correctly.
> if (list != null) {
>   int index = 0;
>   for (String excludeExp : list) {
>     filePattern.append((++index < list.length)? "| grep -v " : 
>             "| grep -vc " + list[i] );  
>   }
> }
> String[] cmd =
>    new String[] {
>        "bash",
>        "-c",
>        "grep "
>            + pattern + " " + filePattern
>            + " | awk -F: '{s+=$2} END {print s}'" };  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8651-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 05:05:00 2010
Return-Path: <common-issues-return-8651-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90944 invoked from network); 2 Jun 2010 05:05:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 05:05:00 -0000
Received: (qmail 18257 invoked by uid 500); 2 Jun 2010 05:05:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18214 invoked by uid 500); 2 Jun 2010 05:04:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18096 invoked by uid 99); 2 Jun 2010 05:04:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 05:04:58 +0000
X-ASF-Spam-Status: No, hits=-1488.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 05:04:58 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5254ccm028935
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 05:04:38 GMT
Message-ID: <5285832.123081275455078370.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 01:04:38 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6789) set-version should precede Herriot
 artifacts installation target
In-Reply-To: <13948127.9831274912496806.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6789:
---------------------------------------

    Affects Version/s: 0.22.0
                           (was: 0.21.0)

> set-version should precede Herriot artifacts installation target
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6789
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6789
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>            Priority: Minor
>         Attachments: HADOOP-6789.patch
>
>
> {{set-version}} should be preceding the invocation of {{-mvn-system-install}}. Otherwise it will be failing when called on a clean workspace. This problem only affects local installation of artifacts.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8652-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 05:05:04 2010
Return-Path: <common-issues-return-8652-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91429 invoked from network); 2 Jun 2010 05:05:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 05:05:04 -0000
Received: (qmail 18817 invoked by uid 500); 2 Jun 2010 05:05:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18780 invoked by uid 500); 2 Jun 2010 05:05:04 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18772 invoked by uid 99); 2 Jun 2010 05:05:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 05:05:04 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 05:05:02 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5254e2C028968
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 05:04:40 GMT
Message-ID: <26108495.123191275455080580.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 01:04:40 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6461) webapps aren't located correctly
 post-split
In-Reply-To: <428473464.1261365498148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6461:
------------------------------

    Attachment: HADOOP-6461.patch

+1

With this fix (updated to apply to trunk) the daemons' web UIs work correctly. I tested in conjunction with HADOOP-6794 and related JIRAs.

> webapps aren't located correctly post-split
> -------------------------------------------
>
>                 Key: HADOOP-6461
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6461
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Steve Loughran
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6461.patch, HADOOP-6461.patch, HADOOP-6461.patch, HADOOP-6461.patch, hadoop-6461.patch, hadoop-6461.patch, hadoop-6461.patch, HADOOP-6461.txt, hadoop-6461.txt, hadoop-6461.txt
>
>
> Post-split, when one builds common it creates an empty build/webapps dir. If you then try to launch the NN for example using HADOOP_HDFS_HOME=/path/to/hdfs hdfs namenode, HttpServer.getWebAppsPath locates the empty common webapps dir, and the NN web UI fails to load.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8653-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 05:07:01 2010
Return-Path: <common-issues-return-8653-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95350 invoked from network); 2 Jun 2010 05:07:00 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 05:07:00 -0000
Received: (qmail 19886 invoked by uid 500); 2 Jun 2010 05:07:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19858 invoked by uid 500); 2 Jun 2010 05:07:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19843 invoked by uid 99); 2 Jun 2010 05:06:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 05:06:59 +0000
X-ASF-Spam-Status: No, hits=-1488.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 05:06:59 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5256cmp029007
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 05:06:38 GMT
Message-ID: <3389260.123291275455198777.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 01:06:38 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6788) Exception exclusion functionality is
 not working correctly.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinay Kumar Thota updated HADOOP-6788:
--------------------------------------

    Attachment: HADOOP-6788.patch

New patch for trunk.

> Exception exclusion functionality is not working correctly.
> -----------------------------------------------------------
>
>                 Key: HADOOP-6788
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6788
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Balaji Rajagopalan
>         Attachments: 6788-ydist-security.patch, 6788-ydist-security.patch, HADOOP-6788.patch, HADOOP-6788.patch
>
>
> Exception exclusion functionality is not working correctly because of that tests are failing by not matching the error count.
> I debugged the issue and found that the problem with shell command which is generating in the getNumberOfMatchesInLogFile function.
> Currently building the shell command in the following way. 
> if(list != null){
>   for(int i =0; i < list.length; ++i)
>   {
>     filePattern.append(" | grep -v " + list[i] );
>   }
> }
>     String[] cmd =
>         new String[] {
>             "bash",
>             "-c",
>             "grep -c "
>                 + pattern + " " + filePattern
>                 + " | awk -F: '{s+=$2} END {print s}'" };    
> However, The above commnad won't work correctly because you are counting the exceptions in the file before excluding the known exceptions.
> In this case it gives the mismatch error counts everytime.The shell command should be in the following way to work correctly.
> if (list != null) {
>   int index = 0;
>   for (String excludeExp : list) {
>     filePattern.append((++index < list.length)? "| grep -v " : 
>             "| grep -vc " + list[i] );  
>   }
> }
> String[] cmd =
>    new String[] {
>        "bash",
>        "-c",
>        "grep "
>            + pattern + " " + filePattern
>            + " | awk -F: '{s+=$2} END {print s}'" };  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8654-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 05:07:02 2010
Return-Path: <common-issues-return-8654-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95405 invoked from network); 2 Jun 2010 05:07:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 05:07:01 -0000
Received: (qmail 20068 invoked by uid 500); 2 Jun 2010 05:07:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20007 invoked by uid 500); 2 Jun 2010 05:07:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19999 invoked by uid 99); 2 Jun 2010 05:07:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 05:07:01 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 05:06:59 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5256baH028992
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 05:06:37 GMT
Message-ID: <16849117.123241275455197244.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 01:06:37 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6788) Exception exclusion functionality is
 not working correctly.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinay Kumar Thota updated HADOOP-6788:
--------------------------------------

    Attachment: 6788-ydist-security.patch

Attached new patch based on cos comments. Patch for yahoo security distribution branch.

> Exception exclusion functionality is not working correctly.
> -----------------------------------------------------------
>
>                 Key: HADOOP-6788
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6788
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Balaji Rajagopalan
>         Attachments: 6788-ydist-security.patch, 6788-ydist-security.patch, HADOOP-6788.patch, HADOOP-6788.patch
>
>
> Exception exclusion functionality is not working correctly because of that tests are failing by not matching the error count.
> I debugged the issue and found that the problem with shell command which is generating in the getNumberOfMatchesInLogFile function.
> Currently building the shell command in the following way. 
> if(list != null){
>   for(int i =0; i < list.length; ++i)
>   {
>     filePattern.append(" | grep -v " + list[i] );
>   }
> }
>     String[] cmd =
>         new String[] {
>             "bash",
>             "-c",
>             "grep -c "
>                 + pattern + " " + filePattern
>                 + " | awk -F: '{s+=$2} END {print s}'" };    
> However, The above commnad won't work correctly because you are counting the exceptions in the file before excluding the known exceptions.
> In this case it gives the mismatch error counts everytime.The shell command should be in the following way to work correctly.
> if (list != null) {
>   int index = 0;
>   for (String excludeExp : list) {
>     filePattern.append((++index < list.length)? "| grep -v " : 
>             "| grep -vc " + list[i] );  
>   }
> }
> String[] cmd =
>    new String[] {
>        "bash",
>        "-c",
>        "grep "
>            + pattern + " " + filePattern
>            + " | awk -F: '{s+=$2} END {print s}'" };  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8655-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 05:09:06 2010
Return-Path: <common-issues-return-8655-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95652 invoked from network); 2 Jun 2010 05:09:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 05:09:06 -0000
Received: (qmail 20524 invoked by uid 500); 2 Jun 2010 05:09:06 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20476 invoked by uid 500); 2 Jun 2010 05:09:05 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20379 invoked by uid 99); 2 Jun 2010 05:09:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 05:09:05 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 05:09:02 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5258fiG029067
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 05:08:41 GMT
Message-ID: <11996478.123481275455321555.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 01:08:41 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6461) webapps aren't located correctly
 post-split
In-Reply-To: <428473464.1261365498148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6461:
------------------------------

    Status: Open  (was: Patch Available)

> webapps aren't located correctly post-split
> -------------------------------------------
>
>                 Key: HADOOP-6461
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6461
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Steve Loughran
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6461.patch, HADOOP-6461.patch, HADOOP-6461.patch, HADOOP-6461.patch, hadoop-6461.patch, hadoop-6461.patch, hadoop-6461.patch, HADOOP-6461.txt, hadoop-6461.txt, hadoop-6461.txt
>
>
> Post-split, when one builds common it creates an empty build/webapps dir. If you then try to launch the NN for example using HADOOP_HDFS_HOME=/path/to/hdfs hdfs namenode, HttpServer.getWebAppsPath locates the empty common webapps dir, and the NN web UI fails to load.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8656-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 05:11:02 2010
Return-Path: <common-issues-return-8656-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95894 invoked from network); 2 Jun 2010 05:11:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 05:11:01 -0000
Received: (qmail 21051 invoked by uid 500); 2 Jun 2010 05:11:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21018 invoked by uid 500); 2 Jun 2010 05:11:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20997 invoked by uid 99); 2 Jun 2010 05:11:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 05:11:00 +0000
X-ASF-Spam-Status: No, hits=-1488.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 05:11:00 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o525AdpU029097
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 05:10:39 GMT
Message-ID: <9596494.123571275455439962.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 01:10:39 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6794) Move configuration and script files
 post split
In-Reply-To: <11039662.93131275342217664.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6794:
------------------------------

    Attachment: HADOOP-6794.patch

Updated to pick up webapps from the classpath. With this change (and HADOOP-6461) the web UIs work correctly.

> Move configuration and script files post split
> ----------------------------------------------
>
>                 Key: HADOOP-6794
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6794
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, scripts
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6794.patch, HADOOP-6794.patch, HADOOP-6794.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8657-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 05:11:03 2010
Return-Path: <common-issues-return-8657-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95942 invoked from network); 2 Jun 2010 05:11:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 05:11:02 -0000
Received: (qmail 21206 invoked by uid 500); 2 Jun 2010 05:11:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21167 invoked by uid 500); 2 Jun 2010 05:11:02 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21159 invoked by uid 99); 2 Jun 2010 05:11:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 05:11:02 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 05:11:00 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o525AcHr029085
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 05:10:39 GMT
Message-ID: <22843189.123531275455438918.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 01:10:38 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6461) webapps aren't located correctly
 post-split
In-Reply-To: <428473464.1261365498148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6461:
------------------------------

    Status: Patch Available  (was: Open)

> webapps aren't located correctly post-split
> -------------------------------------------
>
>                 Key: HADOOP-6461
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6461
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Steve Loughran
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6461.patch, HADOOP-6461.patch, HADOOP-6461.patch, HADOOP-6461.patch, hadoop-6461.patch, hadoop-6461.patch, hadoop-6461.patch, HADOOP-6461.txt, hadoop-6461.txt, hadoop-6461.txt
>
>
> Post-split, when one builds common it creates an empty build/webapps dir. If you then try to launch the NN for example using HADOOP_HDFS_HOME=/path/to/hdfs hdfs namenode, HttpServer.getWebAppsPath locates the empty common webapps dir, and the NN web UI fails to load.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8658-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 05:13:12 2010
Return-Path: <common-issues-return-8658-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96460 invoked from network); 2 Jun 2010 05:13:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 05:13:09 -0000
Received: (qmail 23493 invoked by uid 500); 2 Jun 2010 05:13:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23445 invoked by uid 500); 2 Jun 2010 05:13:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23374 invoked by uid 99); 2 Jun 2010 05:13:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 05:13:08 +0000
X-ASF-Spam-Status: No, hits=-1488.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 05:13:08 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o525ClhA029170
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 05:12:47 GMT
Message-ID: <2779588.123771275455567955.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 01:12:47 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6794) Move configuration and script files
 post split
In-Reply-To: <11039662.93131275342217664.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6794:
------------------------------

    Status: Open  (was: Patch Available)

> Move configuration and script files post split
> ----------------------------------------------
>
>                 Key: HADOOP-6794
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6794
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, scripts
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6794.patch, HADOOP-6794.patch, HADOOP-6794.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8659-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 05:13:18 2010
Return-Path: <common-issues-return-8659-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96527 invoked from network); 2 Jun 2010 05:13:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 05:13:12 -0000
Received: (qmail 23744 invoked by uid 500); 2 Jun 2010 05:13:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23723 invoked by uid 500); 2 Jun 2010 05:13:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23714 invoked by uid 99); 2 Jun 2010 05:13:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 05:13:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 05:13:09 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o525CmaO029182
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 05:12:48 GMT
Message-ID: <25831095.123811275455568450.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 01:12:48 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6794) Move configuration and script files
 post split
In-Reply-To: <11039662.93131275342217664.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6794:
------------------------------

    Status: Patch Available  (was: Open)

> Move configuration and script files post split
> ----------------------------------------------
>
>                 Key: HADOOP-6794
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6794
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, scripts
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6794.patch, HADOOP-6794.patch, HADOOP-6794.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8660-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 05:41:05 2010
Return-Path: <common-issues-return-8660-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 643 invoked from network); 2 Jun 2010 05:41:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 05:41:04 -0000
Received: (qmail 37646 invoked by uid 500); 2 Jun 2010 05:41:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37590 invoked by uid 500); 2 Jun 2010 05:41:04 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37582 invoked by uid 99); 2 Jun 2010 05:41:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 05:41:03 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 05:41:01 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o525edVg029411
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 05:40:39 GMT
Message-ID: <2406870.124391275457239593.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 01:40:39 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6461) webapps aren't located correctly
 post-split
In-Reply-To: <428473464.1261365498148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874429#action_12874429 ] 

Hadoop QA commented on HADOOP-6461:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446101/HADOOP-6461.patch
  against trunk revision 950309.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 24 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/558/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/558/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/558/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/558/console

This message is automatically generated.

> webapps aren't located correctly post-split
> -------------------------------------------
>
>                 Key: HADOOP-6461
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6461
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Steve Loughran
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6461.patch, HADOOP-6461.patch, HADOOP-6461.patch, HADOOP-6461.patch, hadoop-6461.patch, hadoop-6461.patch, hadoop-6461.patch, HADOOP-6461.txt, hadoop-6461.txt, hadoop-6461.txt
>
>
> Post-split, when one builds common it creates an empty build/webapps dir. If you then try to launch the NN for example using HADOOP_HDFS_HOME=/path/to/hdfs hdfs namenode, HttpServer.getWebAppsPath locates the empty common webapps dir, and the NN web UI fails to load.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8661-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 05:45:01 2010
Return-Path: <common-issues-return-8661-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1258 invoked from network); 2 Jun 2010 05:45:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 05:45:01 -0000
Received: (qmail 39707 invoked by uid 500); 2 Jun 2010 05:45:00 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39680 invoked by uid 500); 2 Jun 2010 05:45:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39672 invoked by uid 99); 2 Jun 2010 05:45:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 05:45:00 +0000
X-ASF-Spam-Status: No, hits=-1488.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 05:44:59 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o525idgr029464
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 05:44:39 GMT
Message-ID: <8242270.124521275457479271.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 01:44:39 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6788) Exception exclusion functionality
 is not working correctly.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874431#action_12874431 ] 

Konstantin Boudnik commented on HADOOP-6788:
--------------------------------------------

Vinay, code re-formats should be done separately from code modifications unless they are a part of some fixes. The reason is simple: formatting in the patches makes reviewing much harder, especially for bigger patches. 

I have applied the patch to the Common's code, built the artifacts and then re-built HDFS using fresh instrumented Common. All code seem to be in place.

+1 patch looks good.

> Exception exclusion functionality is not working correctly.
> -----------------------------------------------------------
>
>                 Key: HADOOP-6788
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6788
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Balaji Rajagopalan
>         Attachments: 6788-ydist-security.patch, 6788-ydist-security.patch, HADOOP-6788.patch, HADOOP-6788.patch
>
>
> Exception exclusion functionality is not working correctly because of that tests are failing by not matching the error count.
> I debugged the issue and found that the problem with shell command which is generating in the getNumberOfMatchesInLogFile function.
> Currently building the shell command in the following way. 
> if(list != null){
>   for(int i =0; i < list.length; ++i)
>   {
>     filePattern.append(" | grep -v " + list[i] );
>   }
> }
>     String[] cmd =
>         new String[] {
>             "bash",
>             "-c",
>             "grep -c "
>                 + pattern + " " + filePattern
>                 + " | awk -F: '{s+=$2} END {print s}'" };    
> However, The above commnad won't work correctly because you are counting the exceptions in the file before excluding the known exceptions.
> In this case it gives the mismatch error counts everytime.The shell command should be in the following way to work correctly.
> if (list != null) {
>   int index = 0;
>   for (String excludeExp : list) {
>     filePattern.append((++index < list.length)? "| grep -v " : 
>             "| grep -vc " + list[i] );  
>   }
> }
> String[] cmd =
>    new String[] {
>        "bash",
>        "-c",
>        "grep "
>            + pattern + " " + filePattern
>            + " | awk -F: '{s+=$2} END {print s}'" };  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8662-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 05:47:02 2010
Return-Path: <common-issues-return-8662-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1587 invoked from network); 2 Jun 2010 05:47:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 05:47:02 -0000
Received: (qmail 41754 invoked by uid 500); 2 Jun 2010 05:47:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41720 invoked by uid 500); 2 Jun 2010 05:47:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41712 invoked by uid 99); 2 Jun 2010 05:47:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 05:47:01 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 05:46:59 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o525kboo029486
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 05:46:37 GMT
Message-ID: <7613248.124571275457597272.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 01:46:37 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6788) Exception exclusion functionality is
 not working correctly.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6788:
---------------------------------------

               Status: Patch Available  (was: Open)
         Hadoop Flags: [Reviewed]
    Affects Version/s: 0.21.0

To make sure that everything is all right.

> Exception exclusion functionality is not working correctly.
> -----------------------------------------------------------
>
>                 Key: HADOOP-6788
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6788
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Balaji Rajagopalan
>         Attachments: 6788-ydist-security.patch, 6788-ydist-security.patch, HADOOP-6788.patch, HADOOP-6788.patch
>
>
> Exception exclusion functionality is not working correctly because of that tests are failing by not matching the error count.
> I debugged the issue and found that the problem with shell command which is generating in the getNumberOfMatchesInLogFile function.
> Currently building the shell command in the following way. 
> if(list != null){
>   for(int i =0; i < list.length; ++i)
>   {
>     filePattern.append(" | grep -v " + list[i] );
>   }
> }
>     String[] cmd =
>         new String[] {
>             "bash",
>             "-c",
>             "grep -c "
>                 + pattern + " " + filePattern
>                 + " | awk -F: '{s+=$2} END {print s}'" };    
> However, The above commnad won't work correctly because you are counting the exceptions in the file before excluding the known exceptions.
> In this case it gives the mismatch error counts everytime.The shell command should be in the following way to work correctly.
> if (list != null) {
>   int index = 0;
>   for (String excludeExp : list) {
>     filePattern.append((++index < list.length)? "| grep -v " : 
>             "| grep -vc " + list[i] );  
>   }
> }
> String[] cmd =
>    new String[] {
>        "bash",
>        "-c",
>        "grep "
>            + pattern + " " + filePattern
>            + " | awk -F: '{s+=$2} END {print s}'" };  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8663-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 05:57:01 2010
Return-Path: <common-issues-return-8663-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2483 invoked from network); 2 Jun 2010 05:57:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 05:57:01 -0000
Received: (qmail 44647 invoked by uid 500); 2 Jun 2010 05:57:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44624 invoked by uid 500); 2 Jun 2010 05:57:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44616 invoked by uid 99); 2 Jun 2010 05:57:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 05:57:00 +0000
X-ASF-Spam-Status: No, hits=-1488.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 05:56:59 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o525udwG029575
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 05:56:39 GMT
Message-ID: <15492793.124761275458199232.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 01:56:39 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874437#action_12874437 ] 

Jakob Homan commented on HADOOP-6796:
-------------------------------------

bq. You could also modify Path to accept Path("") or Path(null) so that instead of throwing an exception they silently convert to Path("/")

Nope - silent conversions are evil.  

The current patch  is probably the best way to go, although this new behavior needs to be documented thoroughly.  The general rule of thumb with anything implementing writable is caveat utilitor - let the user beware. Because of the write and readFields methods as well as the default constructor, it's always best practice to verify the state of the instance you're dealing with.

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8664-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 06:05:04 2010
Return-Path: <common-issues-return-8664-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4609 invoked from network); 2 Jun 2010 06:05:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 06:05:04 -0000
Received: (qmail 49867 invoked by uid 500); 2 Jun 2010 06:05:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49802 invoked by uid 500); 2 Jun 2010 06:05:04 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49793 invoked by uid 99); 2 Jun 2010 06:05:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 06:05:03 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 06:05:01 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5264d22029676
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 06:04:40 GMT
Message-ID: <21822142.124881275458679839.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 02:04:39 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874441#action_12874441 ] 

Jakob Homan commented on HADOOP-6787:
-------------------------------------

Currently the GlobFilter classes are private to their outer classes and this patch makes the new, remaining class public.  Is this necessary, or can we ratchet down the visibility to protected? What's the intended audience and stability annotations? I'd rather not make this a supported class if we don't have to and it seems pretty internal to HDFS.

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-y20s-v1.patch
>
>
> The GlobFilter class (especially the error-prone setRegex method) is duplicated in both FileContext and Filesystem. And I'd like to be able to reuse the most of the setRegex code for glob pattern matching in metrics filter configurations.
> We could at least factor out the hairy setRegex code and package them into a GlobPattern class with a minimum compile method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8665-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 06:25:00 2010
Return-Path: <common-issues-return-8665-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10905 invoked from network); 2 Jun 2010 06:24:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 06:24:59 -0000
Received: (qmail 58134 invoked by uid 500); 2 Jun 2010 06:24:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58109 invoked by uid 500); 2 Jun 2010 06:24:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58101 invoked by uid 99); 2 Jun 2010 06:24:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 06:24:59 +0000
X-ASF-Spam-Status: No, hits=-1488.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 06:24:58 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o526Ob6G029804
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 06:24:38 GMT
Message-ID: <32015933.125111275459877804.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 02:24:37 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6794) Move configuration and script files
 post split
In-Reply-To: <11039662.93131275342217664.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874449#action_12874449 ] 

Hadoop QA commented on HADOOP-6794:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446104/HADOOP-6794.patch
  against trunk revision 950309.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 12 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/72/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/72/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/72/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/72/console

This message is automatically generated.

> Move configuration and script files post split
> ----------------------------------------------
>
>                 Key: HADOOP-6794
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6794
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, scripts
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6794.patch, HADOOP-6794.patch, HADOOP-6794.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8666-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 06:27:02 2010
Return-Path: <common-issues-return-8666-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11119 invoked from network); 2 Jun 2010 06:27:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 06:27:02 -0000
Received: (qmail 61450 invoked by uid 500); 2 Jun 2010 06:27:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61334 invoked by uid 500); 2 Jun 2010 06:27:02 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61322 invoked by uid 99); 2 Jun 2010 06:27:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 06:27:01 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 06:26:59 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o526Qb7O029827
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 06:26:38 GMT
Message-ID: <26031522.125161275459997902.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 02:26:37 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874450#action_12874450 ] 

Jakob Homan commented on HADOOP-6787:
-------------------------------------

And of course by protected, I meant package private... doh.

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-y20s-v1.patch
>
>
> The GlobFilter class (especially the error-prone setRegex method) is duplicated in both FileContext and Filesystem. And I'd like to be able to reuse the most of the setRegex code for glob pattern matching in metrics filter configurations.
> We could at least factor out the hairy setRegex code and package them into a GlobPattern class with a minimum compile method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8667-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 06:45:25 2010
Return-Path: <common-issues-return-8667-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20293 invoked from network); 2 Jun 2010 06:45:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 06:45:24 -0000
Received: (qmail 79980 invoked by uid 500); 2 Jun 2010 06:45:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79935 invoked by uid 500); 2 Jun 2010 06:45:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79926 invoked by uid 99); 2 Jun 2010 06:45:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 06:45:23 +0000
X-ASF-Spam-Status: No, hits=-1488.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 06:45:23 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o526if12029962
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 06:44:42 GMT
Message-ID: <4110656.125461275461081675.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 02:44:41 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874456#action_12874456 ] 

Jakob Homan commented on HADOOP-6787:
-------------------------------------

For the love of Pete... and I meant GlobPattern. If the metrics filter configuration you mentioned above isn't within the fs package, I imagine this is the best we'll do for limiting access to this new class.

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-y20s-v1.patch
>
>
> The GlobFilter class (especially the error-prone setRegex method) is duplicated in both FileContext and Filesystem. And I'd like to be able to reuse the most of the setRegex code for glob pattern matching in metrics filter configurations.
> We could at least factor out the hairy setRegex code and package them into a GlobPattern class with a minimum compile method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8668-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 07:44:08 2010
Return-Path: <common-issues-return-8668-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36468 invoked from network); 2 Jun 2010 07:44:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 07:44:07 -0000
Received: (qmail 33383 invoked by uid 500); 2 Jun 2010 07:44:07 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33316 invoked by uid 500); 2 Jun 2010 07:44:07 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33308 invoked by uid 99); 2 Jun 2010 07:44:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 07:44:06 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 07:44:04 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o527hhwm000542
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 07:43:43 GMT
Message-ID: <18808616.126281275464623080.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 03:43:43 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6461) webapps aren't located correctly
 post-split
In-Reply-To: <428473464.1261365498148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874476#action_12874476 ] 

Vinod K V commented on HADOOP-6461:
-----------------------------------

I also verified this manually together with HADOOP-6794 having separate installation directories for the three projects. +1 from correctness POV (didn't review code myself though).

Tom, can you please commit this given Hudson also is okay?

> webapps aren't located correctly post-split
> -------------------------------------------
>
>                 Key: HADOOP-6461
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6461
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Steve Loughran
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6461.patch, HADOOP-6461.patch, HADOOP-6461.patch, HADOOP-6461.patch, hadoop-6461.patch, hadoop-6461.patch, hadoop-6461.patch, HADOOP-6461.txt, hadoop-6461.txt, hadoop-6461.txt
>
>
> Post-split, when one builds common it creates an empty build/webapps dir. If you then try to launch the NN for example using HADOOP_HDFS_HOME=/path/to/hdfs hdfs namenode, HttpServer.getWebAppsPath locates the empty common webapps dir, and the NN web UI fails to load.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8669-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 07:48:04 2010
Return-Path: <common-issues-return-8669-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37176 invoked from network); 2 Jun 2010 07:48:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 07:48:04 -0000
Received: (qmail 37012 invoked by uid 500); 2 Jun 2010 07:48:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36992 invoked by uid 500); 2 Jun 2010 07:48:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36984 invoked by uid 99); 2 Jun 2010 07:48:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 07:48:03 +0000
X-ASF-Spam-Status: No, hits=-1488.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 07:48:02 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o527lgHU000591
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 07:47:42 GMT
Message-ID: <7599508.126381275464862245.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 03:47:42 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6788) [Herriot] Exception exclusion
 functionality is not working correctly.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V updated HADOOP-6788:
------------------------------

    Summary: [Herriot] Exception exclusion functionality is not working correctly.  (was: Exception exclusion functionality is not working correctly.)

Can you please augment the title of all future herriot related issues with [Herriot]? Helps other people not concerned, avoids confusion, so... Thanks!

> [Herriot] Exception exclusion functionality is not working correctly.
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6788
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6788
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Balaji Rajagopalan
>         Attachments: 6788-ydist-security.patch, 6788-ydist-security.patch, HADOOP-6788.patch, HADOOP-6788.patch
>
>
> Exception exclusion functionality is not working correctly because of that tests are failing by not matching the error count.
> I debugged the issue and found that the problem with shell command which is generating in the getNumberOfMatchesInLogFile function.
> Currently building the shell command in the following way. 
> if(list != null){
>   for(int i =0; i < list.length; ++i)
>   {
>     filePattern.append(" | grep -v " + list[i] );
>   }
> }
>     String[] cmd =
>         new String[] {
>             "bash",
>             "-c",
>             "grep -c "
>                 + pattern + " " + filePattern
>                 + " | awk -F: '{s+=$2} END {print s}'" };    
> However, The above commnad won't work correctly because you are counting the exceptions in the file before excluding the known exceptions.
> In this case it gives the mismatch error counts everytime.The shell command should be in the following way to work correctly.
> if (list != null) {
>   int index = 0;
>   for (String excludeExp : list) {
>     filePattern.append((++index < list.length)? "| grep -v " : 
>             "| grep -vc " + list[i] );  
>   }
> }
> String[] cmd =
>    new String[] {
>        "bash",
>        "-c",
>        "grep "
>            + pattern + " " + filePattern
>            + " | awk -F: '{s+=$2} END {print s}'" };  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8670-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 08:06:07 2010
Return-Path: <common-issues-return-8670-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44584 invoked from network); 2 Jun 2010 08:06:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 08:06:06 -0000
Received: (qmail 55153 invoked by uid 500); 2 Jun 2010 08:06:06 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55112 invoked by uid 500); 2 Jun 2010 08:06:06 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55104 invoked by uid 99); 2 Jun 2010 08:06:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 08:06:05 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 08:06:03 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5285g62000906
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 08:05:42 GMT
Message-ID: <16855056.126831275465942308.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 04:05:42 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874487#action_12874487 ] 

Rodrigo Schmidt commented on HADOOP-6796:
-----------------------------------------

@Eli: I noticed this problem while writing a unit test for a different JIRA. The code didn't care much about the contents of the FileStatus object and that's why I was creating it with "new FileStatus()".

@Jakob: Besides the javadoc, where else should this be documented?

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8671-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 08:08:07 2010
Return-Path: <common-issues-return-8671-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45276 invoked from network); 2 Jun 2010 08:08:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 08:08:07 -0000
Received: (qmail 59454 invoked by uid 500); 2 Jun 2010 08:08:07 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59416 invoked by uid 500); 2 Jun 2010 08:08:07 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59399 invoked by uid 99); 2 Jun 2010 08:08:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 08:08:06 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 08:08:04 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5287hA9000949
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 08:07:43 GMT
Message-ID: <24632806.126921275466063086.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 04:07:43 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6794) Move configuration and script files
 post split
In-Reply-To: <11039662.93131275342217664.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874489#action_12874489 ] 

Vinod K V commented on HADOOP-6794:
-----------------------------------

The latest patches for the three projects look good, Tom. I could use HADOOP-6461 to successfully bring up single node clusters of both hdfs and mapreduce. I ran a simple mapreduce job too.

+1.

Thanks for taking this up!

> Move configuration and script files post split
> ----------------------------------------------
>
>                 Key: HADOOP-6794
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6794
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, scripts
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6794.patch, HADOOP-6794.patch, HADOOP-6794.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8672-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 10:04:20 2010
Return-Path: <common-issues-return-8672-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76088 invoked from network); 2 Jun 2010 10:04:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 10:04:20 -0000
Received: (qmail 27794 invoked by uid 500); 2 Jun 2010 10:04:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27694 invoked by uid 500); 2 Jun 2010 10:04:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27680 invoked by uid 99); 2 Jun 2010 10:04:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 10:04:18 +0000
X-ASF-Spam-Status: No, hits=-1489.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 10:04:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52A3u2L006906
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 10:03:57 GMT
Message-ID: <6793877.128541275473036755.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 06:03:56 -0400 (EDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6473) Add hadoop health check/diagnostics
 to run from command line, JSP pages, other tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874516#action_12874516 ] 

Steve Loughran commented on HADOOP-6473:
----------------------------------------

Some things we could check for and related bugreps/stack traces that are independent of node/client role

Network
# Network: IPv4 is in use: HADOOP-6056. Check out system property see if local ip address lookup with an IPv6 address ::1 works, see what happens on nslookup of 127.0.0.1
# DNS status. At the very least, the local hostname must resolve: HADOOP-3426. 
# Also check rdns and some nslookup of some common external addresses (hadoop.apache.org), warn if these are not available, but mention its not important if you don't want external network access
# Proxy server settings; print them, if non-null, check (hostname, port) and warn if missing

Classpath
# Print it out. 
# Check for duplicate filenames (with/without version endings?)

Dependencies
# XML engine name and version
# XML parser supports XInclude: HADOOP-5254
# XSL engine name and version

Local filesystem
# Space
# Temp dir is writeable. Write something, read it back in, verify checksum and that the file's timestamp is within a few seconds of the system clock.




> Add hadoop health check/diagnostics to run from command line, JSP pages, other tools
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6473
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6473
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Steve Loughran
>            Priority: Minor
>
> If the lifecycle ping() is for short-duration "are we still alive" checks, Hadoop still needs something bigger to check the overall system health,.This would be for end users, but also for automated cluster deployment, a complete validation of the cluster, 
> It could be a command line tool, and something that runs on different nodes, checked via IPC or JSP. the idea would be to do thorough checks with good diagnostics.  Oh, and they should be executable through JUnit too.
> For example
>  -if running on windows, check that cygwin is on the path, fail with a pointer to a wiki issue if not
>  -datanodes should check that it can create locks on the filesystem, create files, timestamps are (roughly) aligned with local time.
>  -namenodes should try and create files/locks in the filesystem
>  -task tracker should try and exec() something
>  -run through the classpath and look for problems; duplicate JARs, unsupported java, xerces versions, etc.
> * The number of tests should be extensible -rather than one single class with all the tests, there'd be something separate for name, task, data, job tracker nodes
> * They can't be in the nodes themselves, as they should be executable even if the nodes don't come up. 
> * output could be in human readable text or html, and a form that could be processed through hadoop itself in future
> * these tests could have side effects, such as actually trying to submit work to a cluster

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8673-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 10:18:06 2010
Return-Path: <common-issues-return-8673-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84088 invoked from network); 2 Jun 2010 10:18:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 10:18:06 -0000
Received: (qmail 46853 invoked by uid 500); 2 Jun 2010 10:18:06 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46818 invoked by uid 500); 2 Jun 2010 10:18:04 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46809 invoked by uid 99); 2 Jun 2010 10:18:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 10:18:04 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 10:18:01 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52AHdF0007103
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 10:17:40 GMT
Message-ID: <3542804.129011275473859815.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 06:17:39 -0400 (EDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6473) Add hadoop health check/diagnostics
 to run from command line, JSP pages, other tools
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874521#action_12874521 ] 

Steve Loughran commented on HADOOP-6473:
----------------------------------------

I think the checks/entry point should have a production/developer switch or set of maskable tests

Production
* HDFS: secondary namenode defined, hostname must resolve from NN
* DNS/rDNS must work against specifically defined hosts
* Maybe: stricter requirements about which interfaces come up on (e.g a valid range of IP addresses for each service)
* log directory space requirements
* temp dir space requirements
and failure/error code if anything isn't met, also consider running these checks on every service startup

Developer
* allow people to work on laptops with no external network, play in incomplete clusters.
* less disk space requirements

Logging also raises some questions
* Check/print log levels of the various services, warn if at DEBUG level in production
* Work out which back end to commons-logging is running, print its classname
* Print out the commons-logging, slf4j, jetty and log4j JVM config options
* print out which log4j.properties/XML file is resolving on the classpath

> Add hadoop health check/diagnostics to run from command line, JSP pages, other tools
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6473
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6473
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Steve Loughran
>            Priority: Minor
>
> If the lifecycle ping() is for short-duration "are we still alive" checks, Hadoop still needs something bigger to check the overall system health,.This would be for end users, but also for automated cluster deployment, a complete validation of the cluster, 
> It could be a command line tool, and something that runs on different nodes, checked via IPC or JSP. the idea would be to do thorough checks with good diagnostics.  Oh, and they should be executable through JUnit too.
> For example
>  -if running on windows, check that cygwin is on the path, fail with a pointer to a wiki issue if not
>  -datanodes should check that it can create locks on the filesystem, create files, timestamps are (roughly) aligned with local time.
>  -namenodes should try and create files/locks in the filesystem
>  -task tracker should try and exec() something
>  -run through the classpath and look for problems; duplicate JARs, unsupported java, xerces versions, etc.
> * The number of tests should be extensible -rather than one single class with all the tests, there'd be something separate for name, task, data, job tracker nodes
> * They can't be in the nodes themselves, as they should be executable even if the nodes don't come up. 
> * output could be in human readable text or html, and a form that could be processed through hadoop itself in future
> * these tests could have side effects, such as actually trying to submit work to a cluster

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8674-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 11:22:05 2010
Return-Path: <common-issues-return-8674-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7274 invoked from network); 2 Jun 2010 11:22:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 11:22:04 -0000
Received: (qmail 42260 invoked by uid 500); 2 Jun 2010 11:22:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41919 invoked by uid 500); 2 Jun 2010 11:22:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41887 invoked by uid 99); 2 Jun 2010 11:22:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 11:22:00 +0000
X-ASF-Spam-Status: No, hits=-1489.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 11:22:00 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52BLdt1007697
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 11:21:39 GMT
Message-ID: <10998507.130081275477699807.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 07:21:39 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6798) Align Ivy version for all Hadoop
 subprojects.
In-Reply-To: <19712300.110201275416737755.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874551#action_12874551 ] 

Hudson commented on HADOOP-6798:
--------------------------------

Integrated in Hadoop-Common-trunk #354 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/354/])
    HADOOP-6798. Align Ivy version for all Hadoop subprojects. Contributed by Konstantin Boudnik.


> Align Ivy version for all Hadoop subprojects.
> ---------------------------------------------
>
>                 Key: HADOOP-6798
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6798
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6798.patch
>
>
> HDFS-1177 and MAPREDUCE-1830 are upgrading Ivy to 2.1.0. Common still has Ivy version at 2.1.0-rc1

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8675-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 12:54:02 2010
Return-Path: <common-issues-return-8675-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42939 invoked from network); 2 Jun 2010 12:54:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 12:54:02 -0000
Received: (qmail 66342 invoked by uid 500); 2 Jun 2010 12:54:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66094 invoked by uid 500); 2 Jun 2010 12:54:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66085 invoked by uid 99); 2 Jun 2010 12:54:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 12:54:01 +0000
X-ASF-Spam-Status: No, hits=-1489.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 12:54:01 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52CreXC008386
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 12:53:40 GMT
Message-ID: <1892993.131181275483220702.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 08:53:40 -0400 (EDT)
From: "Xiao Kang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6768) RPC client can response more
 efficiently when sendParam() got IOException
In-Reply-To: <2932795.79761274091523409.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xiao Kang updated HADOOP-6768:
------------------------------

    Attachment: HADOOP-6768.patch

patch attached.

A. change org.apache.hadoop.ipc.Client

1. add call.setException(e) when sendParam() got IOException to set call.done to avoid current call wait receiveResponse() thread.

2. interrupt receiveResponse() thread to cleanup other calls on this connection



B. test case for it 

> RPC client can response more efficiently when sendParam() got IOException
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6768
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6768
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Xiao Kang
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6768.patch
>
>
> Under current RPC client implementation, when Client.Connection.sendParam() encounters IOException it just marks the exception and wait receiveResponse() thread to handle the exception. However, receiveResponse() may block ipc.ping.interval on socket read(). 
> This means that RPC client may wait ipc.ping.interval(typically 1 miniute) when exception in sendParam().
> It maybe not so reasonable in some situation. For example using "hadoop fs -put " a small file, DFSClient.close() will interupt renewLease() thread and it may wait 1 minute.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8676-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 13:02:12 2010
Return-Path: <common-issues-return-8676-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45287 invoked from network); 2 Jun 2010 13:02:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 13:02:12 -0000
Received: (qmail 74633 invoked by uid 500); 2 Jun 2010 13:02:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74580 invoked by uid 500); 2 Jun 2010 13:02:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74572 invoked by uid 99); 2 Jun 2010 13:02:11 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 13:02:11 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 13:02:09 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52D1mlm008542
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 13:01:48 GMT
Message-ID: <2439960.131291275483708077.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 09:01:48 -0400 (EDT)
From: "Xiao Kang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6768) RPC client can response more
 efficiently when sendParam() got IOException
In-Reply-To: <2932795.79761274091523409.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874572#action_12874572 ] 

Xiao Kang commented on HADOOP-6768:
-----------------------------------

Another case found for this issuse.

In mapreduce task execution, each task will report progress to TaskTracker through Ubilical RPC. When the task completes, it will interrupt and join the reporter thread, this may cause Exception for sendParam() of report progress RPC and wait ipc.ping.interval(typically 1 miniute). 

> RPC client can response more efficiently when sendParam() got IOException
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6768
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6768
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Xiao Kang
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6768.patch
>
>
> Under current RPC client implementation, when Client.Connection.sendParam() encounters IOException it just marks the exception and wait receiveResponse() thread to handle the exception. However, receiveResponse() may block ipc.ping.interval on socket read(). 
> This means that RPC client may wait ipc.ping.interval(typically 1 miniute) when exception in sendParam().
> It maybe not so reasonable in some situation. For example using "hadoop fs -put " a small file, DFSClient.close() will interupt renewLease() thread and it may wait 1 minute.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8677-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 13:42:06 2010
Return-Path: <common-issues-return-8677-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58735 invoked from network); 2 Jun 2010 13:42:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 13:42:06 -0000
Received: (qmail 16749 invoked by uid 500); 2 Jun 2010 13:42:06 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16699 invoked by uid 500); 2 Jun 2010 13:42:05 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16690 invoked by uid 99); 2 Jun 2010 13:42:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 13:42:05 +0000
X-ASF-Spam-Status: No, hits=-1489.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 13:42:04 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52Dfi5J008791
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 13:41:44 GMT
Message-ID: <18074885.131651275486104340.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 09:41:44 -0400 (EDT)
From: "Michele Catasta (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5793) High speed compression algorithm
 like BMDiff
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-5793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874576#action_12874576 ] 

Michele Catasta commented on HADOOP-5793:
-----------------------------------------

You can find a preliminary patch for BMDiff + LZO at http://github.com/pirroh/hadoop-gpl-compression/tree/branch-0.1
I opened an issue on hadoop-gpl-compression as well: http://code.google.com/p/hadoop-gpl-compression/issues/detail?id=27

Questions and feedback more than welcome.

> High speed compression algorithm like BMDiff
> --------------------------------------------
>
>                 Key: HADOOP-5793
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5793
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: elhoim gibor
>            Priority: Minor
>
> Add a high speed compression algorithm like BMDiff.
> It gives speeds ~100MB/s for writes and ~1000MB/s for reads, compressing 2.1billions web pages from 45.1TB in 4.2TB
> Reference:
> http://norfolk.cs.washington.edu/htbin-post/unrestricted/colloq/details.cgi?id=437
> 2005 Jeff Dean talk about google architecture - around 46:00.
> http://feedblog.org/2008/10/12/google-bigtable-compression-zippy-and-bmdiff/
> http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=755678
> A reference implementation exists in HyperTable.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8678-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 15:06:09 2010
Return-Path: <common-issues-return-8678-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89490 invoked from network); 2 Jun 2010 15:06:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 15:06:09 -0000
Received: (qmail 68784 invoked by uid 500); 2 Jun 2010 15:06:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68762 invoked by uid 500); 2 Jun 2010 15:06:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68753 invoked by uid 99); 2 Jun 2010 15:06:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 15:06:08 +0000
X-ASF-Spam-Status: No, hits=-1489.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 15:06:07 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52F5kBd009742
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 15:05:47 GMT
Message-ID: <29180925.132711275491146741.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 11:05:46 -0400 (EDT)
From: "stack (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5793) High speed compression algorithm
 like BMDiff
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-5793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874602#action_12874602 ] 

stack commented on HADOOP-5793:
-------------------------------

Hey Pirroh:  How would it work?  Whats the name of the compression that does both bmz and lzo?  We'd download hadoop gpl stuff, compile it and then refer to the new combo... how? (Sounds great).  This code came over from HT?

> High speed compression algorithm like BMDiff
> --------------------------------------------
>
>                 Key: HADOOP-5793
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5793
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: elhoim gibor
>            Priority: Minor
>
> Add a high speed compression algorithm like BMDiff.
> It gives speeds ~100MB/s for writes and ~1000MB/s for reads, compressing 2.1billions web pages from 45.1TB in 4.2TB
> Reference:
> http://norfolk.cs.washington.edu/htbin-post/unrestricted/colloq/details.cgi?id=437
> 2005 Jeff Dean talk about google architecture - around 46:00.
> http://feedblog.org/2008/10/12/google-bigtable-compression-zippy-and-bmdiff/
> http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=755678
> A reference implementation exists in HyperTable.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8679-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 15:12:42 2010
Return-Path: <common-issues-return-8679-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91148 invoked from network); 2 Jun 2010 15:12:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 15:12:42 -0000
Received: (qmail 82108 invoked by uid 500); 2 Jun 2010 15:12:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82085 invoked by uid 500); 2 Jun 2010 15:12:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82077 invoked by uid 99); 2 Jun 2010 15:12:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 15:12:42 +0000
X-ASF-Spam-Status: No, hits=-1489.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 15:12:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52FCLfM009829
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 15:12:22 GMT
Message-ID: <6915969.132911275491541899.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 11:12:21 -0400 (EDT)
From: "Michele Catasta (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-5793) High speed compression algorithm
 like BMDiff
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-5793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michele Catasta reassigned HADOOP-5793:
---------------------------------------

    Assignee: Michele Catasta

> High speed compression algorithm like BMDiff
> --------------------------------------------
>
>                 Key: HADOOP-5793
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5793
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: elhoim gibor
>            Assignee: Michele Catasta
>            Priority: Minor
>
> Add a high speed compression algorithm like BMDiff.
> It gives speeds ~100MB/s for writes and ~1000MB/s for reads, compressing 2.1billions web pages from 45.1TB in 4.2TB
> Reference:
> http://norfolk.cs.washington.edu/htbin-post/unrestricted/colloq/details.cgi?id=437
> 2005 Jeff Dean talk about google architecture - around 46:00.
> http://feedblog.org/2008/10/12/google-bigtable-compression-zippy-and-bmdiff/
> http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=755678
> A reference implementation exists in HyperTable.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8680-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 15:46:12 2010
Return-Path: <common-issues-return-8680-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2113 invoked from network); 2 Jun 2010 15:46:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 15:46:12 -0000
Received: (qmail 59858 invoked by uid 500); 2 Jun 2010 15:46:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59824 invoked by uid 500); 2 Jun 2010 15:46:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59816 invoked by uid 99); 2 Jun 2010 15:46:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 15:46:12 +0000
X-ASF-Spam-Status: No, hits=-1489.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 15:46:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52Fjpnc010198
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 15:45:51 GMT
Message-ID: <28934307.133571275493551327.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 11:45:51 -0400 (EDT)
From: "Michele (@pirroh) Catasta (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5793) High speed compression algorithm
 like BMDiff
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-5793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874620#action_12874620 ] 

Michele (@pirroh) Catasta commented on HADOOP-5793:
---------------------------------------------------

{quote} How would it work? {quote}

Same steps for hadoop-gpl-compression installation. I updated the build scripts and no other external libraries are needed, so it should be straightforward.

{quote} Whats the name of the compression that does both bmz and lzo?{quote}

com.hadoop.compression.bmz.BmzCodec

{quote}We'd download hadoop gpl stuff, compile it and then refer to the new combo... how? (Sounds great){quote} 

You already found out, but for HBase I already submitted HBASE-2655

{quote}This code came over from HT?{quote}

Exactly. [http://www.hypertable.org/doxygen/bmz_8h.html] - few changes here and there, but the core C lib is almost untouched (mainly HyperTable symbols stripping).


> High speed compression algorithm like BMDiff
> --------------------------------------------
>
>                 Key: HADOOP-5793
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5793
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: elhoim gibor
>            Assignee: Michele (@pirroh) Catasta
>            Priority: Minor
>
> Add a high speed compression algorithm like BMDiff.
> It gives speeds ~100MB/s for writes and ~1000MB/s for reads, compressing 2.1billions web pages from 45.1TB in 4.2TB
> Reference:
> http://norfolk.cs.washington.edu/htbin-post/unrestricted/colloq/details.cgi?id=437
> 2005 Jeff Dean talk about google architecture - around 46:00.
> http://feedblog.org/2008/10/12/google-bigtable-compression-zippy-and-bmdiff/
> http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=755678
> A reference implementation exists in HyperTable.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8681-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 15:48:06 2010
Return-Path: <common-issues-return-8681-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2745 invoked from network); 2 Jun 2010 15:48:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 15:48:05 -0000
Received: (qmail 66731 invoked by uid 500); 2 Jun 2010 15:48:05 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66628 invoked by uid 500); 2 Jun 2010 15:48:05 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66620 invoked by uid 99); 2 Jun 2010 15:48:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 15:48:05 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 15:48:03 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52FlgK4010226
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 15:47:42 GMT
Message-ID: <4738552.133631275493662243.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 11:47:42 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6461) webapps aren't located correctly
 post-split
In-Reply-To: <428473464.1261365498148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6461:
------------------------------

           Status: Resolved  (was: Patch Available)
     Hadoop Flags: [Reviewed]
    Fix Version/s:     (was: 0.22.0)
       Resolution: Fixed

I've just committed this. Thanks Todd and Steve!

> webapps aren't located correctly post-split
> -------------------------------------------
>
>                 Key: HADOOP-6461
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6461
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Steve Loughran
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6461.patch, HADOOP-6461.patch, HADOOP-6461.patch, HADOOP-6461.patch, hadoop-6461.patch, hadoop-6461.patch, hadoop-6461.patch, HADOOP-6461.txt, hadoop-6461.txt, hadoop-6461.txt
>
>
> Post-split, when one builds common it creates an empty build/webapps dir. If you then try to launch the NN for example using HADOOP_HDFS_HOME=/path/to/hdfs hdfs namenode, HttpServer.getWebAppsPath locates the empty common webapps dir, and the NN web UI fails to load.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8682-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 15:52:07 2010
Return-Path: <common-issues-return-8682-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3626 invoked from network); 2 Jun 2010 15:52:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 15:52:06 -0000
Received: (qmail 73932 invoked by uid 500); 2 Jun 2010 15:52:06 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73899 invoked by uid 500); 2 Jun 2010 15:52:06 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73891 invoked by uid 99); 2 Jun 2010 15:52:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 15:52:06 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 15:52:04 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52FpgMb010263
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 15:51:43 GMT
Message-ID: <28128762.133721275493902859.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 11:51:42 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6794) Move configuration and script files
 post split
In-Reply-To: <11039662.93131275342217664.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874624#action_12874624 ] 

Tom White commented on HADOOP-6794:
-----------------------------------

Thanks for the review, Vinod. I plan to commit these three patches soon.

> Move configuration and script files post split
> ----------------------------------------------
>
>                 Key: HADOOP-6794
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6794
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, scripts
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6794.patch, HADOOP-6794.patch, HADOOP-6794.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8683-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 16:57:05 2010
Return-Path: <common-issues-return-8683-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23993 invoked from network); 2 Jun 2010 16:57:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 16:57:05 -0000
Received: (qmail 739 invoked by uid 500); 2 Jun 2010 16:57:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 712 invoked by uid 500); 2 Jun 2010 16:57:04 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 704 invoked by uid 99); 2 Jun 2010 16:57:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 16:57:04 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 16:57:02 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52Guedd011346
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 16:56:41 GMT
Message-ID: <20037083.136261275497800780.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 12:56:40 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874658#action_12874658 ] 

Eli Collins commented on HADOOP-6796:
-------------------------------------

Sounds like the simple thing to do is have your test:
{code}
s = new FileStatus();
s.setPath("/something");
s.setXXX(...);
{code}

Or use FileStatus#read like the only other user of the default constructor (ThriftHadoopFileSystem). For the writable use case the reason the default values are null is that you know you're going to immediately overwrite them via read() and this way you don't pay the cost of instantiating an object you're just going to overwrite.

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8684-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 17:02:01 2010
Return-Path: <common-issues-return-8684-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25273 invoked from network); 2 Jun 2010 17:02:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 17:02:01 -0000
Received: (qmail 9883 invoked by uid 500); 2 Jun 2010 17:02:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9849 invoked by uid 500); 2 Jun 2010 17:02:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9841 invoked by uid 99); 2 Jun 2010 17:02:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 17:02:01 +0000
X-ASF-Spam-Status: No, hits=-1490.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 17:02:00 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52H1ePn011440
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 17:01:40 GMT
Message-ID: <12727491.136391275498100496.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 13:01:40 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6668:
------------------------------

    Attachment: HADOOP-6668.patch

Here's a new patch that makes the three interfaces Public Evolving. I think this is ready to go now. 

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: common.interface-tw-annotated.txt, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8685-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 17:04:02 2010
Return-Path: <common-issues-return-8685-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25638 invoked from network); 2 Jun 2010 17:04:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 17:04:02 -0000
Received: (qmail 11445 invoked by uid 500); 2 Jun 2010 17:04:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11417 invoked by uid 500); 2 Jun 2010 17:04:02 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11409 invoked by uid 99); 2 Jun 2010 17:04:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 17:04:02 +0000
X-ASF-Spam-Status: No, hits=-1490.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 17:04:01 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52H3f9A011515
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 17:03:41 GMT
Message-ID: <10153896.136591275498221682.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 13:03:41 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6668:
------------------------------

    Status: Patch Available  (was: Open)

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: common.interface-tw-annotated.txt, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8686-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 17:04:04 2010
Return-Path: <common-issues-return-8686-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25745 invoked from network); 2 Jun 2010 17:04:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 17:04:04 -0000
Received: (qmail 11831 invoked by uid 500); 2 Jun 2010 17:04:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11788 invoked by uid 500); 2 Jun 2010 17:04:04 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11780 invoked by uid 99); 2 Jun 2010 17:04:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 17:04:04 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 17:04:01 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52H3ecT011488
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 17:03:40 GMT
Message-ID: <12401709.136501275498220203.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 13:03:40 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6668:
------------------------------

    Status: Open  (was: Patch Available)

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: common.interface-tw-annotated.txt, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8687-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 17:06:07 2010
Return-Path: <common-issues-return-8687-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30320 invoked from network); 2 Jun 2010 17:06:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 17:06:07 -0000
Received: (qmail 15602 invoked by uid 500); 2 Jun 2010 17:06:07 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15560 invoked by uid 500); 2 Jun 2010 17:06:07 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15537 invoked by uid 99); 2 Jun 2010 17:06:07 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 17:06:07 +0000
X-ASF-Spam-Status: No, hits=-1490.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 17:06:07 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52H5kug011586
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 17:05:46 GMT
Message-ID: <33458361.136811275498346916.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 13:05:46 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874665#action_12874665 ] 

Eli Collins commented on HADOOP-6787:
-------------------------------------

I agree Jakob, GlobPattern should be package private like GlobFilter. The annotations should be the same as other package private hadoop.fs classes. Globbing is not internal to HDFS it's purely client side, used by FsShell for any file system.

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-y20s-v1.patch
>
>
> The GlobFilter class (especially the error-prone setRegex method) is duplicated in both FileContext and Filesystem. And I'd like to be able to reuse the most of the setRegex code for glob pattern matching in metrics filter configurations.
> We could at least factor out the hairy setRegex code and package them into a GlobPattern class with a minimum compile method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8688-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 17:26:07 2010
Return-Path: <common-issues-return-8688-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37129 invoked from network); 2 Jun 2010 17:26:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 17:26:07 -0000
Received: (qmail 53080 invoked by uid 500); 2 Jun 2010 17:26:07 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53047 invoked by uid 500); 2 Jun 2010 17:26:07 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53003 invoked by uid 99); 2 Jun 2010 17:26:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 17:26:07 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 17:26:04 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52HPhak011866
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 17:25:43 GMT
Message-ID: <11095652.137391275499543207.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 13:25:43 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874681#action_12874681 ] 

Hadoop QA commented on HADOOP-6668:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446156/HADOOP-6668.patch
  against trunk revision 950605.

    +1 @author.  The patch does not contain any @author tags.

    +0 tests included.  The patch appears to be a documentation patch that doesn't require tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/73/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/73/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/73/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/73/console

This message is automatically generated.

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: common.interface-tw-annotated.txt, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8689-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 17:31:59 2010
Return-Path: <common-issues-return-8689-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39188 invoked from network); 2 Jun 2010 17:31:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 17:31:59 -0000
Received: (qmail 61530 invoked by uid 500); 2 Jun 2010 17:31:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61412 invoked by uid 500); 2 Jun 2010 17:31:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61404 invoked by uid 99); 2 Jun 2010 17:31:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 17:31:58 +0000
X-ASF-Spam-Status: No, hits=-1490.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 17:31:58 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52HVbOj011957
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 17:31:38 GMT
Message-ID: <16550780.137581275499897405.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 13:31:37 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6788) [Herriot] Exception exclusion
 functionality is not working correctly.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874686#action_12874686 ] 

Konstantin Boudnik commented on HADOOP-6788:
--------------------------------------------

bq. Can you please augment the title of all future herriot related issues with [Herriot]?
Well, I'm not sure about this. Herriot never was approved/discussed by the community as an official name of the framework. Let's think about this first.

> [Herriot] Exception exclusion functionality is not working correctly.
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6788
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6788
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Balaji Rajagopalan
>         Attachments: 6788-ydist-security.patch, 6788-ydist-security.patch, HADOOP-6788.patch, HADOOP-6788.patch
>
>
> Exception exclusion functionality is not working correctly because of that tests are failing by not matching the error count.
> I debugged the issue and found that the problem with shell command which is generating in the getNumberOfMatchesInLogFile function.
> Currently building the shell command in the following way. 
> if(list != null){
>   for(int i =0; i < list.length; ++i)
>   {
>     filePattern.append(" | grep -v " + list[i] );
>   }
> }
>     String[] cmd =
>         new String[] {
>             "bash",
>             "-c",
>             "grep -c "
>                 + pattern + " " + filePattern
>                 + " | awk -F: '{s+=$2} END {print s}'" };    
> However, The above commnad won't work correctly because you are counting the exceptions in the file before excluding the known exceptions.
> In this case it gives the mismatch error counts everytime.The shell command should be in the following way to work correctly.
> if (list != null) {
>   int index = 0;
>   for (String excludeExp : list) {
>     filePattern.append((++index < list.length)? "| grep -v " : 
>             "| grep -vc " + list[i] );  
>   }
> }
> String[] cmd =
>    new String[] {
>        "bash",
>        "-c",
>        "grep "
>            + pattern + " " + filePattern
>            + " | awk -F: '{s+=$2} END {print s}'" };  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8690-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 17:36:08 2010
Return-Path: <common-issues-return-8690-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40475 invoked from network); 2 Jun 2010 17:36:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 17:36:07 -0000
Received: (qmail 67651 invoked by uid 500); 2 Jun 2010 17:36:07 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67601 invoked by uid 500); 2 Jun 2010 17:36:07 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67593 invoked by uid 99); 2 Jun 2010 17:36:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 17:36:07 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 17:36:05 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52HZhhD012012
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 17:35:43 GMT
Message-ID: <14556725.137751275500143288.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 13:35:43 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874689#action_12874689 ] 

Luke Lu commented on HADOOP-6787:
---------------------------------

The main incentive for the refactor is to make the GlobPattern reusable by other components (including but not limited to metrics configuration) in other packages (e.g., org.apache.hadoop.metrics). I  proposed in private to make it reside in org.apache.hadoop.util as it's really a general utility (a simplified and less powerful (than regex to shoot  people's foot easily) pattern matching mechanism, as found in many other languages.), and met with strong resistance. So it stays in the fs package, where it comes from. 

It seems to me that It should remain as a java public class, as it's needed by other packages, but be annotated as audience private.

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-y20s-v1.patch
>
>
> The GlobFilter class (especially the error-prone setRegex method) is duplicated in both FileContext and Filesystem. And I'd like to be able to reuse the most of the setRegex code for glob pattern matching in metrics filter configurations.
> We could at least factor out the hairy setRegex code and package them into a GlobPattern class with a minimum compile method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8691-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 17:44:02 2010
Return-Path: <common-issues-return-8691-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44318 invoked from network); 2 Jun 2010 17:44:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 17:44:02 -0000
Received: (qmail 75899 invoked by uid 500); 2 Jun 2010 17:44:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75871 invoked by uid 500); 2 Jun 2010 17:44:02 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75863 invoked by uid 99); 2 Jun 2010 17:44:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 17:44:02 +0000
X-ASF-Spam-Status: No, hits=-1490.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 17:44:01 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52Hhf50012169
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 17:43:41 GMT
Message-ID: <18939529.138131275500621385.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 13:43:41 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874693#action_12874693 ] 

Eli Collins commented on HADOOP-6787:
-------------------------------------

Agree that it's better left in o.a.h.fs.  GlobPattern should be package protection though since it's only used by o.a.h.fs right?

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-y20s-v1.patch
>
>
> The GlobFilter class (especially the error-prone setRegex method) is duplicated in both FileContext and Filesystem. And I'd like to be able to reuse the most of the setRegex code for glob pattern matching in metrics filter configurations.
> We could at least factor out the hairy setRegex code and package them into a GlobPattern class with a minimum compile method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8692-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 17:50:06 2010
Return-Path: <common-issues-return-8692-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45536 invoked from network); 2 Jun 2010 17:50:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 17:50:06 -0000
Received: (qmail 80767 invoked by uid 500); 2 Jun 2010 17:50:06 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80728 invoked by uid 500); 2 Jun 2010 17:50:06 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80720 invoked by uid 99); 2 Jun 2010 17:50:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 17:50:06 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 17:50:03 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52HngkS012227
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 17:49:42 GMT
Message-ID: <22696398.138201275500981997.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 13:49:41 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874695#action_12874695 ] 

Luke Lu commented on HADOOP-6787:
---------------------------------

I want to reuse GlobPattern in o.a.h.metrics as well. Annotate it with InterfaceAudience.Private and InterfaceStability.Evolving is probably most appropriate.

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-y20s-v1.patch
>
>
> The GlobFilter class (especially the error-prone setRegex method) is duplicated in both FileContext and Filesystem. And I'd like to be able to reuse the most of the setRegex code for glob pattern matching in metrics filter configurations.
> We could at least factor out the hairy setRegex code and package them into a GlobPattern class with a minimum compile method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8693-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 17:54:08 2010
Return-Path: <common-issues-return-8693-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46640 invoked from network); 2 Jun 2010 17:54:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 17:54:08 -0000
Received: (qmail 87556 invoked by uid 500); 2 Jun 2010 17:54:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87519 invoked by uid 500); 2 Jun 2010 17:54:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87511 invoked by uid 99); 2 Jun 2010 17:54:08 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 17:54:08 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 17:54:05 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52HriOI012280
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 17:53:44 GMT
Message-ID: <22863012.138321275501224486.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 13:53:44 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874698#action_12874698 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6787:
------------------------------------------------

@Luke, looks like that this is not only a code refactoring but also bug fixes (or behavior changes.)  Could you describe the changes in details?

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-y20s-v1.patch
>
>
> The GlobFilter class (especially the error-prone setRegex method) is duplicated in both FileContext and Filesystem. And I'd like to be able to reuse the most of the setRegex code for glob pattern matching in metrics filter configurations.
> We could at least factor out the hairy setRegex code and package them into a GlobPattern class with a minimum compile method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8694-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 18:18:03 2010
Return-Path: <common-issues-return-8694-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57059 invoked from network); 2 Jun 2010 18:18:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 18:18:02 -0000
Received: (qmail 21352 invoked by uid 500); 2 Jun 2010 18:18:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21326 invoked by uid 500); 2 Jun 2010 18:18:02 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21317 invoked by uid 99); 2 Jun 2010 18:18:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 18:18:02 +0000
X-ASF-Spam-Status: No, hits=-1490.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 18:18:02 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52IHfeu012637
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 18:17:41 GMT
Message-ID: <6678310.138941275502661328.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 14:17:41 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6661) User document for
 UserGroupInformation.doAs
In-Reply-To: <927098388.498051269554007283.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6661:
-----------------------------------------

    Status: Open  (was: Patch Available)

> User document for UserGroupInformation.doAs
> -------------------------------------------
>
>                 Key: HADOOP-6661
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6661
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6661-y20.1.patch, HADOOP-6661-y20.2.patch, HADOOP-6661.1.patch
>
>
> A user document should be added for secure impersonation feature. The document should cover a code example about how  UserGroupInformation.doAs should be used for secure impersonation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8695-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 18:26:07 2010
Return-Path: <common-issues-return-8695-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65292 invoked from network); 2 Jun 2010 18:26:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 18:26:06 -0000
Received: (qmail 34556 invoked by uid 500); 2 Jun 2010 18:26:06 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34519 invoked by uid 500); 2 Jun 2010 18:26:06 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34511 invoked by uid 99); 2 Jun 2010 18:26:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 18:26:06 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 18:26:04 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52IPgrp012696
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 18:25:43 GMT
Message-ID: <20163930.139051275503142696.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 14:25:42 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874719#action_12874719 ] 

Rodrigo Schmidt commented on HADOOP-6796:
-----------------------------------------

I'm not worried about my test. I fixed it by using a complete FileStatus object.
I just don't like the idea of leaving the code the way it is. It looks buggy and next time someone bumps into the NPE error, he or she will create a JIRA and go through all this discussion again.

@Eli: Here are a couple of options that I thought about this morning. Let me know if they sound plausible to you:

1) Prevent srcPath from being null in any non-default constructor of FileStatus() by using asserts. For the default constructor, use Path("/").

or

2) If srcPath is null, prevent any calls to methods different from readFields() with asserts. (This will have some performance impact and will be hard to maintain).



> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8696-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 18:30:04 2010
Return-Path: <common-issues-return-8696-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67050 invoked from network); 2 Jun 2010 18:30:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 18:30:04 -0000
Received: (qmail 44885 invoked by uid 500); 2 Jun 2010 18:30:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44856 invoked by uid 500); 2 Jun 2010 18:30:04 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44848 invoked by uid 99); 2 Jun 2010 18:30:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 18:30:04 +0000
X-ASF-Spam-Status: No, hits=-1490.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 18:30:03 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52IThKj012725
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 18:29:43 GMT
Message-ID: <16222784.139111275503383464.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 14:29:43 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6661) User document for
 UserGroupInformation.doAs
In-Reply-To: <927098388.498051269554007283.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6661:
-----------------------------------------

    Attachment: HADOOP-6661.2.patch

> User document for UserGroupInformation.doAs
> -------------------------------------------
>
>                 Key: HADOOP-6661
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6661
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6661-y20.1.patch, HADOOP-6661-y20.2.patch, HADOOP-6661.1.patch, HADOOP-6661.2.patch
>
>
> A user document should be added for secure impersonation feature. The document should cover a code example about how  UserGroupInformation.doAs should be used for secure impersonation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8697-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 18:32:01 2010
Return-Path: <common-issues-return-8697-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67872 invoked from network); 2 Jun 2010 18:32:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 18:32:01 -0000
Received: (qmail 47005 invoked by uid 500); 2 Jun 2010 18:32:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46974 invoked by uid 500); 2 Jun 2010 18:32:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46966 invoked by uid 99); 2 Jun 2010 18:32:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 18:32:01 +0000
X-ASF-Spam-Status: No, hits=-1490.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 18:32:01 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52IVeUC012745
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 18:31:40 GMT
Message-ID: <20553249.139131275503500794.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 14:31:40 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6661) User document for
 UserGroupInformation.doAs
In-Reply-To: <927098388.498051269554007283.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6661:
-----------------------------------------

    Status: Patch Available  (was: Open)

> User document for UserGroupInformation.doAs
> -------------------------------------------
>
>                 Key: HADOOP-6661
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6661
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6661-y20.1.patch, HADOOP-6661-y20.2.patch, HADOOP-6661.1.patch, HADOOP-6661.2.patch
>
>
> A user document should be added for secure impersonation feature. The document should cover a code example about how  UserGroupInformation.doAs should be used for secure impersonation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8698-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 18:32:07 2010
Return-Path: <common-issues-return-8698-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67935 invoked from network); 2 Jun 2010 18:32:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 18:32:06 -0000
Received: (qmail 47179 invoked by uid 500); 2 Jun 2010 18:32:06 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47136 invoked by uid 500); 2 Jun 2010 18:32:06 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47128 invoked by uid 99); 2 Jun 2010 18:32:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 18:32:06 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 18:32:04 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52IVg6W012760
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 18:31:42 GMT
Message-ID: <23746821.139181275503502651.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 14:31:42 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874722#action_12874722 ] 

Luke Lu commented on HADOOP-6787:
---------------------------------

Yes, it does include some fixes.  Summary of changes:

* Fixed ^ usage outside [...] (HADOOP-6618)
* Made the pattern more POSIX compliant: support [!...] as well as [^...]
* Made it to generate less garbage while matching by using non-capturing group.
* Removed some unnecessary checks that would throw on valid POSIX glob patterns.
* Added more comprehensive unit tests for the glob pattern matching code.



> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-y20s-v1.patch
>
>
> The GlobFilter class (especially the error-prone setRegex method) is duplicated in both FileContext and Filesystem. And I'd like to be able to reuse the most of the setRegex code for glob pattern matching in metrics filter configurations.
> We could at least factor out the hairy setRegex code and package them into a GlobPattern class with a minimum compile method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8699-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 18:44:06 2010
Return-Path: <common-issues-return-8699-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20245 invoked from network); 2 Jun 2010 18:44:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 18:44:06 -0000
Received: (qmail 70652 invoked by uid 500); 2 Jun 2010 18:44:06 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70615 invoked by uid 500); 2 Jun 2010 18:44:06 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70607 invoked by uid 99); 2 Jun 2010 18:44:06 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 18:44:06 +0000
X-ASF-Spam-Status: No, hits=-1490.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 18:44:05 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52IhjVw012899
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 18:43:45 GMT
Message-ID: <17436827.139551275504225282.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 14:43:45 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874729#action_12874729 ] 

Luke Lu commented on HADOOP-6787:
---------------------------------

In short, it's strictly less restrictive and more standard compliant than the old code. The TestGlobPaths test in HDFS passes unmodified in my environment with patched common package as well (it didn't until I wrapped the PatternSyntaxException with IOException that start with "Illegal file pattern" in the GlobFilter.)

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-y20s-v1.patch
>
>
> The GlobFilter class (especially the error-prone setRegex method) is duplicated in both FileContext and Filesystem. And I'd like to be able to reuse the most of the setRegex code for glob pattern matching in metrics filter configurations.
> We could at least factor out the hairy setRegex code and package them into a GlobPattern class with a minimum compile method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8700-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 18:52:02 2010
Return-Path: <common-issues-return-8700-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22160 invoked from network); 2 Jun 2010 18:52:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 18:52:02 -0000
Received: (qmail 85599 invoked by uid 500); 2 Jun 2010 18:52:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85565 invoked by uid 500); 2 Jun 2010 18:52:02 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85557 invoked by uid 99); 2 Jun 2010 18:52:02 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 18:52:02 +0000
X-ASF-Spam-Status: No, hits=-1491.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 18:52:02 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52IpfXi012991
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 18:51:41 GMT
Message-ID: <24060779.139791275504701490.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 14:51:41 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6787:
-------------------------------------------

    Hadoop Flags: [Incompatible change]
     Component/s: fs

The changes sound reasonable.

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-y20s-v1.patch
>
>
> The GlobFilter class (especially the error-prone setRegex method) is duplicated in both FileContext and Filesystem. And I'd like to be able to reuse the most of the setRegex code for glob pattern matching in metrics filter configurations.
> We could at least factor out the hairy setRegex code and package them into a GlobPattern class with a minimum compile method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8701-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 18:54:05 2010
Return-Path: <common-issues-return-8701-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22532 invoked from network); 2 Jun 2010 18:54:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 18:54:05 -0000
Received: (qmail 90252 invoked by uid 500); 2 Jun 2010 18:54:05 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90207 invoked by uid 500); 2 Jun 2010 18:54:05 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90125 invoked by uid 99); 2 Jun 2010 18:54:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 18:54:04 +0000
X-ASF-Spam-Status: No, hits=-1491.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 18:54:04 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52IriKr013028
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 18:53:44 GMT
Message-ID: <16422588.139901275504824118.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 14:53:44 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D128=
74740#action_12874740 ]=20

Eli Collins commented on HADOOP-6796:
-------------------------------------

I think classes like FileStatus (eg writables) are special cases where the =
user has to know that they're using a constructor that leaves the object pa=
rtially constructed. All the fields are invalid when using the default cons=
tructor, by the same logic we should also set a default owner, group etc in=
stead of using null and converting to "". The path is not special (other pa=
rts of the system are not happy with "" users). While objects like this are=
 less user friendly (they violate the rule that a constructor leaves an obj=
ect in a valid state) they prevent you from allocating a bunch of objects t=
hat you're just going to overwrite.  The default constructor for most of th=
e other objects that implement writable (eg FsPermission) leave their field=
s as null as well. Its a trade off between performance and user friendlynes=
s, the code currently sacrifies user friendlyness for performance.  If we d=
idn't do that you can see someone doing some profiling and following a jira=
 that notices "we create a lot of objects in the default constructors that =
we immediately overwrite, let's avoid that".  Not vetoing your patch, just =
pointing out that there's a reason the code works the way it does. This isn=
't a one off case where it's different, so we should be consistent.

Assertions are only enabled when the unit tests run, so there's no performa=
nce impact. I think asserts are good practice, not a maintenance burden, by=
 that's my 2c.


> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many met=
hods like write, readFields, compareTo, equals, and hashCode depend on this=
 property.

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8702-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 18:58:04 2010
Return-Path: <common-issues-return-8702-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23493 invoked from network); 2 Jun 2010 18:58:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 18:58:04 -0000
Received: (qmail 1252 invoked by uid 500); 2 Jun 2010 18:58:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1227 invoked by uid 500); 2 Jun 2010 18:58:04 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1219 invoked by uid 99); 2 Jun 2010 18:58:04 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 18:58:04 +0000
X-ASF-Spam-Status: No, hits=-1491.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 18:58:04 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52IvhbP013101
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 18:57:43 GMT
Message-ID: <33195933.140061275505063717.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 14:57:43 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874743#action_12874743 ] 

Eli Collins commented on HADOOP-6787:
-------------------------------------

I think the splitting up into GlobPattern and GlobFilter makes sense. A change like this probably should be split into two, one that's just code motion and one that changes the behavior. My bad, I should have pointed that out.   Do we need to do this as two separate jiras?

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-y20s-v1.patch
>
>
> The GlobFilter class (especially the error-prone setRegex method) is duplicated in both FileContext and Filesystem. And I'd like to be able to reuse the most of the setRegex code for glob pattern matching in metrics filter configurations.
> We could at least factor out the hairy setRegex code and package them into a GlobPattern class with a minimum compile method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8703-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 19:17:17 2010
Return-Path: <common-issues-return-8703-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33682 invoked from network); 2 Jun 2010 19:17:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 19:17:17 -0000
Received: (qmail 40698 invoked by uid 500); 2 Jun 2010 19:17:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40676 invoked by uid 500); 2 Jun 2010 19:17:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40668 invoked by uid 99); 2 Jun 2010 19:17:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 19:17:16 +0000
X-ASF-Spam-Status: No, hits=-1491.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 19:17:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52JGutF013488
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 19:16:56 GMT
Message-ID: <23110814.140941275506216322.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 15:16:56 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874749#action_12874749 ] 

Rodrigo Schmidt commented on HADOOP-6796:
-----------------------------------------

Okay! I see your point.

Maybe we should just prevent null srcPaths on constructors other than the default one. This at least makes it clear that users should not create FileStatus objects with null srcPaths.

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8704-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 19:22:58 2010
Return-Path: <common-issues-return-8704-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37078 invoked from network); 2 Jun 2010 19:22:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 19:22:58 -0000
Received: (qmail 49977 invoked by uid 500); 2 Jun 2010 19:22:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49951 invoked by uid 500); 2 Jun 2010 19:22:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49943 invoked by uid 99); 2 Jun 2010 19:22:57 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 19:22:57 +0000
X-ASF-Spam-Status: No, hits=-1491.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 19:22:57 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52JMaBq013588
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 19:22:37 GMT
Message-ID: <9457037.141241275506556853.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 15:22:36 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874753#action_12874753 ] 

Eli Collins commented on HADOOP-6796:
-------------------------------------

Seems reasonable to me.  Thanks for all the back and forth!

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8705-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 19:25:14 2010
Return-Path: <common-issues-return-8705-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37667 invoked from network); 2 Jun 2010 19:25:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 19:25:14 -0000
Received: (qmail 52681 invoked by uid 500); 2 Jun 2010 19:25:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52654 invoked by uid 500); 2 Jun 2010 19:25:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52646 invoked by uid 99); 2 Jun 2010 19:25:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 19:25:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 19:25:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52JOnoL013614
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 19:24:50 GMT
Message-ID: <16383751.141311275506689965.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 15:24:49 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874754#action_12874754 ] 

Rodrigo Schmidt commented on HADOOP-6796:
-----------------------------------------

No problem. I'll submit a new patch soon.

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8706-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 19:53:04 2010
Return-Path: <common-issues-return-8706-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42741 invoked from network); 2 Jun 2010 19:53:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 19:53:04 -0000
Received: (qmail 81631 invoked by uid 500); 2 Jun 2010 19:53:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81603 invoked by uid 500); 2 Jun 2010 19:53:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81567 invoked by uid 99); 2 Jun 2010 19:53:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 19:53:03 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 19:53:01 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52JqdKT013871
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 19:52:39 GMT
Message-ID: <283547.141841275508359541.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 15:52:39 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874763#action_12874763 ] 

Jakob Homan commented on HADOOP-6796:
-------------------------------------

Rodrigo: any constructor that converts an empty path to "/" is a change in behavior that we're not going to introduce. Again, from above and from what Eli said, Writable is fundamentally a flawed and dangerous data structure if you're expecting it to guarantee any integrity to its fields.  Checking for not null on other constructors is fine, I guess; are we also going to verify that the constituent long fields are non-negative? And yeah, go ahead and throw in asserts - they're our best bet of catching logic errors in this data structure during unit tests.

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8707-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 19:55:00 2010
Return-Path: <common-issues-return-8707-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43166 invoked from network); 2 Jun 2010 19:54:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 19:54:59 -0000
Received: (qmail 84586 invoked by uid 500); 2 Jun 2010 19:54:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84563 invoked by uid 500); 2 Jun 2010 19:54:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84555 invoked by uid 99); 2 Jun 2010 19:54:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 19:54:59 +0000
X-ASF-Spam-Status: No, hits=-1491.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 19:54:59 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52JscBL013894
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 19:54:39 GMT
Message-ID: <22095541.141891275508478939.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 15:54:38 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874764#action_12874764 ] 

Jakob Homan commented on HADOOP-6787:
-------------------------------------

bq. I want to reuse GlobPattern in o.a.h.metrics as well. Annotate it with InterfaceAudience.Private and InterfaceStability.Evolving is probably most appropriate.
Sounds good.  Thanks.

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-y20s-v1.patch
>
>
> The GlobFilter class (especially the error-prone setRegex method) is duplicated in both FileContext and Filesystem. And I'd like to be able to reuse the most of the setRegex code for glob pattern matching in metrics filter configurations.
> We could at least factor out the hairy setRegex code and package them into a GlobPattern class with a minimum compile method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8708-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 20:48:02 2010
Return-Path: <common-issues-return-8708-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57854 invoked from network); 2 Jun 2010 20:48:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 20:48:01 -0000
Received: (qmail 44671 invoked by uid 500); 2 Jun 2010 20:48:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44637 invoked by uid 500); 2 Jun 2010 20:48:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44629 invoked by uid 99); 2 Jun 2010 20:48:01 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 20:48:01 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 20:47:59 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52KlbWS014598
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 20:47:38 GMT
Message-ID: <5254831.143541275511657759.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 16:47:37 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874792#action_12874792 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6787:
------------------------------------------------

> ... Do we need to do this as two separate jiras? 

One jira probably is okay unless there is a need to split it to two.

We should emphasize that this is a bug fix and an incompatible change.  Code refactoring is a minor improvement.

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-y20s-v1.patch
>
>
> The GlobFilter class (especially the error-prone setRegex method) is duplicated in both FileContext and Filesystem. And I'd like to be able to reuse the most of the setRegex code for glob pattern matching in metrics filter configurations.
> We could at least factor out the hairy setRegex code and package them into a GlobPattern class with a minimum compile method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8709-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 21:02:03 2010
Return-Path: <common-issues-return-8709-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60857 invoked from network); 2 Jun 2010 21:02:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 21:02:03 -0000
Received: (qmail 57378 invoked by uid 500); 2 Jun 2010 21:02:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57347 invoked by uid 500); 2 Jun 2010 21:02:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57336 invoked by uid 99); 2 Jun 2010 21:02:03 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 21:02:03 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 21:02:01 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52L1dks014800
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 21:01:39 GMT
Message-ID: <31257007.143831275512499723.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 17:01:39 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-6787:
----------------------------

    Attachment: hadoop-6787-trunk-v2.patch

v2 trunk patch added annotation InterfaceAudience.Private and InterfaceStability.Evolving to the GlobPattern class.

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-trunk-v2.patch, hadoop-6787-y20s-v1.patch
>
>
> The GlobFilter class (especially the error-prone setRegex method) is duplicated in both FileContext and Filesystem. And I'd like to be able to reuse the most of the setRegex code for glob pattern matching in metrics filter configurations.
> We could at least factor out the hairy setRegex code and package them into a GlobPattern class with a minimum compile method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8710-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 21:10:03 2010
Return-Path: <common-issues-return-8710-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66092 invoked from network); 2 Jun 2010 21:10:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 21:10:02 -0000
Received: (qmail 61817 invoked by uid 500); 2 Jun 2010 21:10:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61771 invoked by uid 500); 2 Jun 2010 21:10:02 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61763 invoked by uid 99); 2 Jun 2010 21:10:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 21:10:02 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 21:10:00 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52L9cQe014878
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 21:09:39 GMT
Message-ID: <20341001.144021275512978691.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 17:09:38 -0400 (EDT)
From: "Alex Nelson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6779) Support for Ceph kernel client
In-Reply-To: <16494886.27871274310533455.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alex Nelson reassigned HADOOP-6779:
-----------------------------------

    Assignee: Alex Nelson

> Support for Ceph kernel client
> ------------------------------
>
>                 Key: HADOOP-6779
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6779
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.20.3
>            Reporter: Amandeep Khurana
>            Assignee: Alex Nelson
>             Fix For: 0.21.0
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8711-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 21:37:59 2010
Return-Path: <common-issues-return-8711-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74565 invoked from network); 2 Jun 2010 21:37:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 21:37:58 -0000
Received: (qmail 1257 invoked by uid 500); 2 Jun 2010 21:37:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1226 invoked by uid 500); 2 Jun 2010 21:37:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1218 invoked by uid 99); 2 Jun 2010 21:37:58 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 21:37:58 +0000
X-ASF-Spam-Status: No, hits=-1491.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 21:37:58 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52LbbnC015015
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 21:37:37 GMT
Message-ID: <32756581.144271275514657686.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 17:37:37 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6661) User document for
 UserGroupInformation.doAs
In-Reply-To: <927098388.498051269554007283.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874808#action_12874808 ] 

Jitendra Nath Pandey commented on HADOOP-6661:
----------------------------------------------

test patch results

     [exec] -1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
     [exec]                         Please justify why no new tests are needed for this patch.
     [exec]                         Also please list what manual steps were performed to verify this patch.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.

No tests added or modified because this patch adds a user document.

> User document for UserGroupInformation.doAs
> -------------------------------------------
>
>                 Key: HADOOP-6661
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6661
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6661-y20.1.patch, HADOOP-6661-y20.2.patch, HADOOP-6661.1.patch, HADOOP-6661.2.patch
>
>
> A user document should be added for secure impersonation feature. The document should cover a code example about how  UserGroupInformation.doAs should be used for secure impersonation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8712-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 21:37:59 2010
Return-Path: <common-issues-return-8712-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74635 invoked from network); 2 Jun 2010 21:37:59 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 21:37:59 -0000
Received: (qmail 1523 invoked by uid 500); 2 Jun 2010 21:37:59 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1483 invoked by uid 500); 2 Jun 2010 21:37:59 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1474 invoked by uid 99); 2 Jun 2010 21:37:59 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 21:37:59 +0000
X-ASF-Spam-Status: No, hits=-1491.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 21:37:58 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52LbcGb015030
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 21:37:38 GMT
Message-ID: <22611186.144321275514658708.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 17:37:38 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6661) User document for
 UserGroupInformation.doAs
In-Reply-To: <927098388.498051269554007283.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874809#action_12874809 ] 

Jitendra Nath Pandey commented on HADOOP-6661:
----------------------------------------------

ant docs was run manually to verify the patch.
ant tests is not required because it is a new user doc only.

> User document for UserGroupInformation.doAs
> -------------------------------------------
>
>                 Key: HADOOP-6661
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6661
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6661-y20.1.patch, HADOOP-6661-y20.2.patch, HADOOP-6661.1.patch, HADOOP-6661.2.patch
>
>
> A user document should be added for secure impersonation feature. The document should cover a code example about how  UserGroupInformation.doAs should be used for secure impersonation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8713-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 21:58:21 2010
Return-Path: <common-issues-return-8713-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79792 invoked from network); 2 Jun 2010 21:58:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 21:58:20 -0000
Received: (qmail 31711 invoked by uid 500); 2 Jun 2010 21:58:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31681 invoked by uid 500); 2 Jun 2010 21:58:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31671 invoked by uid 99); 2 Jun 2010 21:58:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 21:58:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 21:58:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52LvuPa015359
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 21:57:56 GMT
Message-ID: <8215141.145191275515876547.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 17:57:56 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6800) Harmonize JAR library versions
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Harmonize JAR library versions
------------------------------

                 Key: HADOOP-6800
                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Tom White
            Priority: Blocker
             Fix For: 0.21.0


Currently, multiple versions of the same library JAR are being pulled in for various reasons.
* Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
* Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8714-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 22:26:19 2010
Return-Path: <common-issues-return-8714-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91927 invoked from network); 2 Jun 2010 22:26:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 22:26:18 -0000
Received: (qmail 67162 invoked by uid 500); 2 Jun 2010 22:26:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67120 invoked by uid 500); 2 Jun 2010 22:26:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67112 invoked by uid 99); 2 Jun 2010 22:26:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 22:26:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 22:26:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52MPsvY015569
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 22:25:54 GMT
Message-ID: <5426528.145531275517554412.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 18:25:54 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6801) io.sort.mb and io.sort.factor were
 renamed and moved to mapreduce but are still in
 CommonConfigurationKeysPublic.java and used in SequenceFile.java
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

io.sort.mb and io.sort.factor were renamed and moved to mapreduce but are still in CommonConfigurationKeysPublic.java and used in SequenceFile.java
---------------------------------------------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-6801
                 URL: https://issues.apache.org/jira/browse/HADOOP-6801
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.22.0
            Reporter: Erik Steffl
            Priority: Minor
             Fix For: 0.22.0


Following configuration keys in CommonConfigurationKeysPublic.java (former CommonConfigurationKeys.java):

public static final String  IO_SORT_MB_KEY = "io.sort.mb";
public static final String  IO_SORT_FACTOR_KEY = "io.sort.factor";

are partially moved:

  - they were renamed to mapreduce.task.io.sort.mb and mapreduce.task.io.sort.factor respectively

  - they were moved to mapreduce project, documented in mapred-default.xml

However:

  - they are still listed in CommonConfigurationKeysPublic.java as quoted above

  - strings "io.sort.mb" and "io.sort.factor" are used in SequenceFile.java in Hadoop Common project

Not sure what the solution is, these constants should probably be removed from CommonConfigurationKeysPublic.java but I am not sure what's the best solution for SequenceFile.java.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8715-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 22:28:16 2010
Return-Path: <common-issues-return-8715-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92503 invoked from network); 2 Jun 2010 22:28:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 22:28:16 -0000
Received: (qmail 72278 invoked by uid 500); 2 Jun 2010 22:28:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72158 invoked by uid 500); 2 Jun 2010 22:28:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72150 invoked by uid 99); 2 Jun 2010 22:28:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 22:28:16 +0000
X-ASF-Spam-Status: No, hits=-1491.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 22:28:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52MRtXw015590
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 22:27:55 GMT
Message-ID: <1393646.145581275517675598.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 18:27:55 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6756) Clean up and add documentation for
 configuration keys in CommonConfigurationKeys.java
In-Reply-To: <23380183.20521273191468124.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874830#action_12874830 ] 

Erik Steffl commented on HADOOP-6756:
-------------------------------------

For 0.22: issue of io.sort.mb and io.sort.factor should be resolved but is outside of scope of this jira, filed https://issues.apache.org/jira/browse/HADOOP-6801 io.sort.mb and io.sort.factor were renamed and moved to mapreduce but are still in CommonConfigurationKeysPublic.java and used in SequenceFile.java

> Clean up and add documentation for configuration keys in CommonConfigurationKeys.java
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6756
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6756
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.3, 0.22.0
>            Reporter: Erik Steffl
>             Fix For: 0.20.3, 0.22.0
>
>         Attachments: jira.HADOOP-6756-0.20-1-FS_DEFAULT_NAME_KEY.patch, jira.HADOOP-6756-0.20-1.patch, jira.HADOOP-6756-0.20.patch
>
>
> Configuration keys in CommonConfigurationKeys.java should be cleaned up and documented (javadoc comments, appropriate *-default.xml descriptions).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8716-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 22:34:22 2010
Return-Path: <common-issues-return-8716-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94421 invoked from network); 2 Jun 2010 22:34:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 22:34:22 -0000
Received: (qmail 75353 invoked by uid 500); 2 Jun 2010 22:34:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75322 invoked by uid 500); 2 Jun 2010 22:34:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75306 invoked by uid 99); 2 Jun 2010 22:34:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 22:34:21 +0000
X-ASF-Spam-Status: No, hits=-1491.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 22:34:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52MY01u015638
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 22:34:00 GMT
Message-ID: <26486957.145711275518040577.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 18:34:00 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6802) Remove FS_CLIENT_BUFFER_DIR_KEY =
 "fs.client.buffer.dir" from CommonConfigurationKeys.java (not used,
 deprecated)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Remove FS_CLIENT_BUFFER_DIR_KEY = "fs.client.buffer.dir" from CommonConfigurationKeys.java (not used, deprecated)
-----------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-6802
                 URL: https://issues.apache.org/jira/browse/HADOOP-6802
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.22.0
            Reporter: Erik Steffl
             Fix For: 0.22.0


In CommonConfigurationKeys.java:

public static final String  FS_CLIENT_BUFFER_DIR_KEY = "fs.client.buffer.dir";

The variable FS_CLIENT_BUFFER_DIR_KEY and string "fs.client.buffer.dir" are not used anywhere (Checked Hadoop Common, Hdfs and Mapred projects), it seems they should be removed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8717-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 22:40:24 2010
Return-Path: <common-issues-return-8717-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95586 invoked from network); 2 Jun 2010 22:40:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 22:40:24 -0000
Received: (qmail 86307 invoked by uid 500); 2 Jun 2010 22:40:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86236 invoked by uid 500); 2 Jun 2010 22:40:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86225 invoked by uid 99); 2 Jun 2010 22:40:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 22:40:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 22:40:21 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52Me0mu015717
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 22:40:00 GMT
Message-ID: <17508627.145921275518400078.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 18:40:00 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6756) Clean up and add documentation for
 configuration keys in CommonConfigurationKeys.java
In-Reply-To: <23380183.20521273191468124.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874838#action_12874838 ] 

Erik Steffl commented on HADOOP-6756:
-------------------------------------

For 0.22: seems like FS_CLIENT_BUFFER_DIR_KEY = "fs.client.buffer.dir" should be removed, filed https://issues.apache.org/jira/browse/HADOOP-6802 Remove FS_CLIENT_BUFFER_DIR_KEY = "fs.client.buffer.dir" from CommonConfigurationKeys.java (not used, deprecated)

> Clean up and add documentation for configuration keys in CommonConfigurationKeys.java
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6756
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6756
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.3, 0.22.0
>            Reporter: Erik Steffl
>             Fix For: 0.20.3, 0.22.0
>
>         Attachments: jira.HADOOP-6756-0.20-1-FS_DEFAULT_NAME_KEY.patch, jira.HADOOP-6756-0.20-1.patch, jira.HADOOP-6756-0.20.patch
>
>
> Configuration keys in CommonConfigurationKeys.java should be cleaned up and documented (javadoc comments, appropriate *-default.xml descriptions).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8718-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 22:54:22 2010
Return-Path: <common-issues-return-8718-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99826 invoked from network); 2 Jun 2010 22:54:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 22:54:22 -0000
Received: (qmail 95983 invoked by uid 500); 2 Jun 2010 22:54:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95949 invoked by uid 500); 2 Jun 2010 22:54:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95941 invoked by uid 99); 2 Jun 2010 22:54:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 22:54:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 22:54:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52MrwVI015807
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 22:53:58 GMT
Message-ID: <3833907.146141275519238480.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 18:53:58 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6756) Clean up and add documentation for
 configuration keys in CommonConfigurationKeys.java
In-Reply-To: <23380183.20521273191468124.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Erik Steffl updated HADOOP-6756:
--------------------------------

    Attachment: HADOOP-6756-0.22.0.patch

> Clean up and add documentation for configuration keys in CommonConfigurationKeys.java
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6756
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6756
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.3, 0.22.0
>            Reporter: Erik Steffl
>             Fix For: 0.20.3, 0.22.0
>
>         Attachments: HADOOP-6756-0.22.0.patch, jira.HADOOP-6756-0.20-1-FS_DEFAULT_NAME_KEY.patch, jira.HADOOP-6756-0.20-1.patch, jira.HADOOP-6756-0.20.patch
>
>
> Configuration keys in CommonConfigurationKeys.java should be cleaned up and documented (javadoc comments, appropriate *-default.xml descriptions).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8719-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 23:12:17 2010
Return-Path: <common-issues-return-8719-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7243 invoked from network); 2 Jun 2010 23:12:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 23:12:17 -0000
Received: (qmail 12904 invoked by uid 500); 2 Jun 2010 23:12:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12782 invoked by uid 500); 2 Jun 2010 23:12:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12774 invoked by uid 99); 2 Jun 2010 23:12:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 23:12:17 +0000
X-ASF-Spam-Status: No, hits=-1491.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 23:12:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52NBuDP016005
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 23:11:57 GMT
Message-ID: <20681213.146461275520316684.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 19:11:56 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6803) ZlibCompressor hangs on close
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

ZlibCompressor hangs on close
-----------------------------

                 Key: HADOOP-6803
                 URL: https://issues.apache.org/jira/browse/HADOOP-6803
             Project: Hadoop Common
          Issue Type: Bug
          Components: io
    Affects Versions: 0.21.0, 0.22.0
            Reporter: Eli Collins
            Assignee: Eli Collins
             Fix For: 0.21.0, 0.22.0


Looking at ZlibCompressor I noticed that the finished member is never modified, and is therefore always false. This means ZlibCompressor#finished will always return false so CompressorStream#close loops indefinitely in finish:

{code} 
      while (!compressor.finished()) {
        compress();
      }
{code}

I modifed TestCodec#testGzipCodecWrite to also cover writing using the native lib and confirmed the hang with jstack. The fix is simple, ZlibCompressor should record when it's been finished.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8720-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 23:14:19 2010
Return-Path: <common-issues-return-8720-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7561 invoked from network); 2 Jun 2010 23:14:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 23:14:18 -0000
Received: (qmail 15398 invoked by uid 500); 2 Jun 2010 23:14:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15291 invoked by uid 500); 2 Jun 2010 23:14:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15276 invoked by uid 99); 2 Jun 2010 23:14:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 23:14:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 23:14:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52NDt6b016018
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 23:13:55 GMT
Message-ID: <16327732.146501275520435173.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 19:13:55 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6803) ZlibCompressor hangs on close
In-Reply-To: <20681213.146461275520316684.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6803:
--------------------------------

    Attachment: hadoop-6803-1.patch

Patch attached.

> ZlibCompressor hangs on close
> -----------------------------
>
>                 Key: HADOOP-6803
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6803
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6803-1.patch
>
>
> Looking at ZlibCompressor I noticed that the finished member is never modified, and is therefore always false. This means ZlibCompressor#finished will always return false so CompressorStream#close loops indefinitely in finish:
> {code} 
>       while (!compressor.finished()) {
>         compress();
>       }
> {code}
> I modifed TestCodec#testGzipCodecWrite to also cover writing using the native lib and confirmed the hang with jstack. The fix is simple, ZlibCompressor should record when it's been finished.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8721-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 23:16:17 2010
Return-Path: <common-issues-return-8721-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8200 invoked from network); 2 Jun 2010 23:16:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 23:16:17 -0000
Received: (qmail 18302 invoked by uid 500); 2 Jun 2010 23:16:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18272 invoked by uid 500); 2 Jun 2010 23:16:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18264 invoked by uid 99); 2 Jun 2010 23:16:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 23:16:17 +0000
X-ASF-Spam-Status: No, hits=-1491.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 23:16:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52NFu0q016041
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 23:15:56 GMT
Message-ID: <24995602.146561275520556578.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 19:15:56 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6803) ZlibCompressor hangs on close
In-Reply-To: <20681213.146461275520316684.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6803:
--------------------------------

    Status: Patch Available  (was: Open)

I confirmed the test passes locally with the native lib built.  On hudson the native test should warn and bail if the native library is not built when the test is run.

> ZlibCompressor hangs on close
> -----------------------------
>
>                 Key: HADOOP-6803
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6803
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6803-1.patch
>
>
> Looking at ZlibCompressor I noticed that the finished member is never modified, and is therefore always false. This means ZlibCompressor#finished will always return false so CompressorStream#close loops indefinitely in finish:
> {code} 
>       while (!compressor.finished()) {
>         compress();
>       }
> {code}
> I modifed TestCodec#testGzipCodecWrite to also cover writing using the native lib and confirmed the hang with jstack. The fix is simple, ZlibCompressor should record when it's been finished.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8722-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 23:22:16 2010
Return-Path: <common-issues-return-8722-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9224 invoked from network); 2 Jun 2010 23:22:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 23:22:16 -0000
Received: (qmail 21908 invoked by uid 500); 2 Jun 2010 23:22:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21797 invoked by uid 500); 2 Jun 2010 23:22:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21789 invoked by uid 99); 2 Jun 2010 23:22:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 23:22:15 +0000
X-ASF-Spam-Status: No, hits=-1491.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 23:22:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52NLtnI016102
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 23:21:55 GMT
Message-ID: <30893672.146691275520915148.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 19:21:55 -0400 (EDT)
From: "Ted Yu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6804) allow FileSystem.copyFromLocalFile()
 to execute under specified username
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

allow FileSystem.copyFromLocalFile() to execute under specified username
------------------------------------------------------------------------

                 Key: HADOOP-6804
                 URL: https://issues.apache.org/jira/browse/HADOOP-6804
             Project: Hadoop Common
          Issue Type: Improvement
          Components: util
    Affects Versions: 0.20.2
            Reporter: Ted Yu
            Priority: Minor


When the user calling FileSystem.copyFromLocalFile() doesn't have permission to write to certain hdfs path:
Thread [main] (Suspended (exception AccessControlException))   
    DFSClient.mkdirs(String, FsPermission) line: 905   
    DistributedFileSystem.mkdirs(Path, FsPermission) line: 262   
    DistributedFileSystem(FileSystem).mkdirs(Path) line: 1162   
    FileUtil.copy(FileSystem, Path, FileSystem, Path, boolean, boolean, Configuration) line: 194   
    DistributedFileSystem(FileSystem).copyFromLocalFile(boolean, boolean, Path, Path) line: 1231   
    DistributedFileSystem(FileSystem).copyFromLocalFile(boolean, Path, Path) line: 1207   
    DistributedFileSystem(FileSystem).copyFromLocalFile(Path, Path) line: 1179   
    GridM2mInstallation.copyInputFiles(FlowConfigurations$FlowConf) line: 380   

Passwordless ssh has been setup for current user, tyu, on localhost and user hadoop on hdfs

FileSystem.copyFromLocalFile() should be able to execute using a different username than effective user on localhost so that AccessControlException can be avoided.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8723-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 23:38:14 2010
Return-Path: <common-issues-return-8723-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11993 invoked from network); 2 Jun 2010 23:38:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 23:38:14 -0000
Received: (qmail 38933 invoked by uid 500); 2 Jun 2010 23:38:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38905 invoked by uid 500); 2 Jun 2010 23:38:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38897 invoked by uid 99); 2 Jun 2010 23:38:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 23:38:14 +0000
X-ASF-Spam-Status: No, hits=-1492.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 23:38:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52NbrN4016252
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 23:37:53 GMT
Message-ID: <26381719.147061275521873469.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 19:37:53 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6803) ZlibCompressor hangs on close
In-Reply-To: <20681213.146461275520316684.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874861#action_12874861 ] 

Chris Douglas commented on HADOOP-6803:
---------------------------------------

Isn't the {{finished}} member set by the JNI code?

> ZlibCompressor hangs on close
> -----------------------------
>
>                 Key: HADOOP-6803
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6803
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6803-1.patch
>
>
> Looking at ZlibCompressor I noticed that the finished member is never modified, and is therefore always false. This means ZlibCompressor#finished will always return false so CompressorStream#close loops indefinitely in finish:
> {code} 
>       while (!compressor.finished()) {
>         compress();
>       }
> {code}
> I modifed TestCodec#testGzipCodecWrite to also cover writing using the native lib and confirmed the hang with jstack. The fix is simple, ZlibCompressor should record when it's been finished.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8724-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 23:50:21 2010
Return-Path: <common-issues-return-8724-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13581 invoked from network); 2 Jun 2010 23:50:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 23:50:21 -0000
Received: (qmail 44099 invoked by uid 500); 2 Jun 2010 23:50:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44020 invoked by uid 500); 2 Jun 2010 23:50:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44010 invoked by uid 99); 2 Jun 2010 23:50:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 23:50:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 23:50:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52NnvxN016340
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 23:49:57 GMT
Message-ID: <24887544.147301275522597787.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 19:49:57 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6342) Create a script to squash a common,
 hdfs, and mapreduce tarball into a single hadoop tarball
In-Reply-To: <1136640537.1256834099525.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6342:
------------------------------

    Attachment: tar-munge

Modified version of tar-munge that works with the changes in HADOOP-6794 (and related JIRAs) to split configuration and scripts. Also updates "core" to "common". Also a few other minor changes to make tarball more similar to a 0.20 series tarball.

> Create a script to squash a common, hdfs, and mapreduce tarball into a single hadoop tarball
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6342
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6342
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>            Priority: Minor
>         Attachments: h-6342.patch, HADOOP-6342.2.patch, HADOOP-6342.patch, tar-munge, tar-munge
>
>
> It would be convenient for the transition if we had a script to take a set of common, hdfs, and mapreduce tarballs and merge them into a single tarball. This is intended just to help users who don't want to transition to split projects for deployment immediately.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8725-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 02 23:54:23 2010
Return-Path: <common-issues-return-8725-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14191 invoked from network); 2 Jun 2010 23:54:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 2 Jun 2010 23:54:22 -0000
Received: (qmail 46354 invoked by uid 500); 2 Jun 2010 23:54:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46325 invoked by uid 500); 2 Jun 2010 23:54:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46316 invoked by uid 99); 2 Jun 2010 23:54:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 23:54:22 +0000
X-ASF-Spam-Status: No, hits=-1492.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jun 2010 23:54:22 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o52Ns1MJ016402
	for <common-issues@hadoop.apache.org>; Wed, 2 Jun 2010 23:54:02 GMT
Message-ID: <22361467.147461275522841838.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 19:54:01 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6794) Move configuration and script files
 post split
In-Reply-To: <11039662.93131275342217664.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6794:
------------------------------

    Attachment: HADOOP-6794.patch

A minor change that loosens the jar glob pattern at the top-level from {{hadoop-common-*.jar}} to {{hadoop-*.jar}}. This change supports an installation where all the scripts, configuration and jars are in common directories, as produced by HADOOP-6342. I successfully ran a job on a pseudo-distributed cluster in this mode.

> Move configuration and script files post split
> ----------------------------------------------
>
>                 Key: HADOOP-6794
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6794
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, scripts
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6794.patch, HADOOP-6794.patch, HADOOP-6794.patch, HADOOP-6794.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8726-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 00:12:23 2010
Return-Path: <common-issues-return-8726-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22128 invoked from network); 3 Jun 2010 00:12:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 00:12:22 -0000
Received: (qmail 64446 invoked by uid 500); 3 Jun 2010 00:12:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64358 invoked by uid 500); 3 Jun 2010 00:12:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64350 invoked by uid 99); 3 Jun 2010 00:12:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 00:12:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 00:12:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o530BwvX016573
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 00:11:59 GMT
Message-ID: <16704886.147741275523918805.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 20:11:58 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6803) ZlibCompressor hangs on close
In-Reply-To: <20681213.146461275520316684.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874874#action_12874874 ] 

Eli Collins commented on HADOOP-6803:
-------------------------------------

Your right, I forgot that methods not fields have the "native" modifier. The "finish" field is read from the native code and "finished" is updated there so it looks like it should work.   What's strange is if I revert the change to ZlibCompressor it passes now, earlier when I first added the test and ran with the native lib it hung with jstack showing the following:

{code}
"main" prio=10 tid=0x0000000040b19000 nid=0x36f0 runnable [0x00007f1257588000]
   java.lang.Thread.State: RUNNABLE
        at org.apache.hadoop.io.compress.zlib.ZlibCompressor.deflateBytesDirect(Native Method)
        at org.apache.hadoop.io.compress.zlib.ZlibCompressor.compress(ZlibCompressor.java:359)
        - locked <0x00007f12495063e0> (a org.apache.hadoop.io.compress.GzipCodec$GzipZlibCompressor)
        at org.apache.hadoop.io.compress.CompressorStream.compress(CompressorStream.java:76)
        at org.apache.hadoop.io.compress.CompressorStream.finish(CompressorStream.java:86)
        at org.apache.hadoop.io.compress.CompressorStream.close(CompressorStream.java:97)
        at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:301)
        at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:130)
        - locked <0x00007f12495167c8> (a java.io.OutputStreamWriter)
        at java.io.OutputStreamWriter.close(OutputStreamWriter.java:216)
        at java.io.BufferedWriter.close(BufferedWriter.java:248)
        - locked <0x00007f12495167c8> (a java.io.OutputStreamWriter)
        at org.apache.hadoop.io.compress.TestCodec.testGzipCodecWrite(TestCodec.java:653)
{code}


> ZlibCompressor hangs on close
> -----------------------------
>
>                 Key: HADOOP-6803
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6803
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6803-1.patch
>
>
> Looking at ZlibCompressor I noticed that the finished member is never modified, and is therefore always false. This means ZlibCompressor#finished will always return false so CompressorStream#close loops indefinitely in finish:
> {code} 
>       while (!compressor.finished()) {
>         compress();
>       }
> {code}
> I modifed TestCodec#testGzipCodecWrite to also cover writing using the native lib and confirmed the hang with jstack. The fix is simple, ZlibCompressor should record when it's been finished.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8727-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 00:16:19 2010
Return-Path: <common-issues-return-8727-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22824 invoked from network); 3 Jun 2010 00:16:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 00:16:19 -0000
Received: (qmail 65952 invoked by uid 500); 3 Jun 2010 00:16:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65834 invoked by uid 500); 3 Jun 2010 00:16:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65826 invoked by uid 99); 3 Jun 2010 00:16:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 00:16:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 00:16:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o530FtiO016593
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 00:15:56 GMT
Message-ID: <9570740.147801275524155821.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 20:15:55 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6803) ZlibCompressor hangs on close
In-Reply-To: <20681213.146461275520316684.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6803:
--------------------------------

    Attachment: hadoop-6803-2.patch

Attached patch. Reverts the ZlibCompressor change, keeps the test.  Thanks Chris.

> ZlibCompressor hangs on close
> -----------------------------
>
>                 Key: HADOOP-6803
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6803
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6803-1.patch, hadoop-6803-2.patch
>
>
> Looking at ZlibCompressor I noticed that the finished member is never modified, and is therefore always false. This means ZlibCompressor#finished will always return false so CompressorStream#close loops indefinitely in finish:
> {code} 
>       while (!compressor.finished()) {
>         compress();
>       }
> {code}
> I modifed TestCodec#testGzipCodecWrite to also cover writing using the native lib and confirmed the hang with jstack. The fix is simple, ZlibCompressor should record when it's been finished.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8728-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 00:16:23 2010
Return-Path: <common-issues-return-8728-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22969 invoked from network); 3 Jun 2010 00:16:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 00:16:22 -0000
Received: (qmail 66672 invoked by uid 500); 3 Jun 2010 00:16:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66626 invoked by uid 500); 3 Jun 2010 00:16:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66618 invoked by uid 99); 3 Jun 2010 00:16:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 00:16:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 00:16:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o530FwxV016635
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 00:15:58 GMT
Message-ID: <33045100.147941275524158808.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 20:15:58 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6803) Add native gzip read/write coverage
 to TestCodec
In-Reply-To: <20681213.146461275520316684.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6803:
--------------------------------

       Summary: Add native gzip read/write coverage to TestCodec   (was: ZlibCompressor hangs on close)
    Issue Type: Test  (was: Bug)

> Add native gzip read/write coverage to TestCodec 
> -------------------------------------------------
>
>                 Key: HADOOP-6803
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6803
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: io
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: hadoop-6803-1.patch, hadoop-6803-2.patch
>
>
> Looking at ZlibCompressor I noticed that the finished member is never modified, and is therefore always false. This means ZlibCompressor#finished will always return false so CompressorStream#close loops indefinitely in finish:
> {code} 
>       while (!compressor.finished()) {
>         compress();
>       }
> {code}
> I modifed TestCodec#testGzipCodecWrite to also cover writing using the native lib and confirmed the hang with jstack. The fix is simple, ZlibCompressor should record when it's been finished.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8729-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 00:20:15 2010
Return-Path: <common-issues-return-8729-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23550 invoked from network); 3 Jun 2010 00:20:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 00:20:15 -0000
Received: (qmail 69565 invoked by uid 500); 3 Jun 2010 00:20:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69535 invoked by uid 500); 3 Jun 2010 00:20:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69527 invoked by uid 99); 3 Jun 2010 00:20:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 00:20:15 +0000
X-ASF-Spam-Status: No, hits=-1492.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 00:20:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o530Js98016665
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 00:19:54 GMT
Message-ID: <20309302.148011275524394346.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 20:19:54 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6803) Add native gzip read/write coverage
 to TestCodec
In-Reply-To: <20681213.146461275520316684.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874880#action_12874880 ] 

Eli Collins commented on HADOOP-6803:
-------------------------------------

I figured it out. In the earlier change I had remove the synchronized modifier from ZlibCompressor#finished which would cause the jvm to miss the update from the jni code.  

> Add native gzip read/write coverage to TestCodec 
> -------------------------------------------------
>
>                 Key: HADOOP-6803
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6803
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6803-1.patch, hadoop-6803-2.patch
>
>
> Looking at ZlibCompressor I noticed that the finished member is never modified, and is therefore always false. This means ZlibCompressor#finished will always return false so CompressorStream#close loops indefinitely in finish:
> {code} 
>       while (!compressor.finished()) {
>         compress();
>       }
> {code}
> I modifed TestCodec#testGzipCodecWrite to also cover writing using the native lib and confirmed the hang with jstack. The fix is simple, ZlibCompressor should record when it's been finished.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8730-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 00:20:16 2010
Return-Path: <common-issues-return-8730-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23557 invoked from network); 3 Jun 2010 00:20:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 00:20:15 -0000
Received: (qmail 69776 invoked by uid 500); 3 Jun 2010 00:20:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69678 invoked by uid 500); 3 Jun 2010 00:20:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69537 invoked by uid 99); 3 Jun 2010 00:20:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 00:20:15 +0000
X-ASF-Spam-Status: No, hits=-1492.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 00:20:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o530Jtgh016674
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 00:19:55 GMT
Message-ID: <23364659.148041275524395043.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 20:19:55 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6803) Add native gzip read/write coverage
 to TestCodec
In-Reply-To: <20681213.146461275520316684.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6803:
--------------------------------

               Status: Open  (was: Patch Available)
    Affects Version/s:     (was: 0.21.0)
        Fix Version/s:     (was: 0.21.0)

> Add native gzip read/write coverage to TestCodec 
> -------------------------------------------------
>
>                 Key: HADOOP-6803
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6803
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6803-1.patch, hadoop-6803-2.patch
>
>
> Looking at ZlibCompressor I noticed that the finished member is never modified, and is therefore always false. This means ZlibCompressor#finished will always return false so CompressorStream#close loops indefinitely in finish:
> {code} 
>       while (!compressor.finished()) {
>         compress();
>       }
> {code}
> I modifed TestCodec#testGzipCodecWrite to also cover writing using the native lib and confirmed the hang with jstack. The fix is simple, ZlibCompressor should record when it's been finished.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8731-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 00:20:16 2010
Return-Path: <common-issues-return-8731-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23639 invoked from network); 3 Jun 2010 00:20:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 00:20:16 -0000
Received: (qmail 69856 invoked by uid 500); 3 Jun 2010 00:20:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69828 invoked by uid 500); 3 Jun 2010 00:20:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69820 invoked by uid 99); 3 Jun 2010 00:20:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 00:20:16 +0000
X-ASF-Spam-Status: No, hits=-1492.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 00:20:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o530Jt32016689
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 00:19:55 GMT
Message-ID: <30906882.148091275524395756.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 20:19:55 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6803) Add native gzip read/write coverage
 to TestCodec
In-Reply-To: <20681213.146461275520316684.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6803:
--------------------------------

    Status: Patch Available  (was: Open)

> Add native gzip read/write coverage to TestCodec 
> -------------------------------------------------
>
>                 Key: HADOOP-6803
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6803
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6803-1.patch, hadoop-6803-2.patch
>
>
> Looking at ZlibCompressor I noticed that the finished member is never modified, and is therefore always false. This means ZlibCompressor#finished will always return false so CompressorStream#close loops indefinitely in finish:
> {code} 
>       while (!compressor.finished()) {
>         compress();
>       }
> {code}
> I modifed TestCodec#testGzipCodecWrite to also cover writing using the native lib and confirmed the hang with jstack. The fix is simple, ZlibCompressor should record when it's been finished.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8732-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 00:24:20 2010
Return-Path: <common-issues-return-8732-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24101 invoked from network); 3 Jun 2010 00:24:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 00:24:20 -0000
Received: (qmail 71845 invoked by uid 500); 3 Jun 2010 00:24:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71817 invoked by uid 500); 3 Jun 2010 00:24:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71809 invoked by uid 99); 3 Jun 2010 00:24:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 00:24:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 00:24:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o530NuJA016730
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 00:23:56 GMT
Message-ID: <5393650.148181275524636022.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 20:23:56 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6805) add buildDTServiceName method to
 SecurityUtil (as part of MAPREDUCE-1718)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

add buildDTServiceName method to SecurityUtil (as part of MAPREDUCE-1718)
-------------------------------------------------------------------------

                 Key: HADOOP-6805
                 URL: https://issues.apache.org/jira/browse/HADOOP-6805
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Boris Shkolnik
            Assignee: Boris Shkolnik




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8733-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 00:52:19 2010
Return-Path: <common-issues-return-8733-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34853 invoked from network); 3 Jun 2010 00:52:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 00:52:19 -0000
Received: (qmail 91540 invoked by uid 500); 3 Jun 2010 00:52:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91512 invoked by uid 500); 3 Jun 2010 00:52:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91504 invoked by uid 99); 3 Jun 2010 00:52:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 00:52:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 00:52:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o530ptFD016983
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 00:51:55 GMT
Message-ID: <2123021.148621275526315364.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 20:51:55 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874892#action_12874892 ] 

Eli Collins commented on HADOOP-6787:
-------------------------------------

+1  lgtm.  Confirmed that we only need the annotations for public classes.  Mind updating the jira type and summary to reflect the bug fixes?

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-trunk-v2.patch, hadoop-6787-y20s-v1.patch
>
>
> The GlobFilter class (especially the error-prone setRegex method) is duplicated in both FileContext and Filesystem. And I'd like to be able to reuse the most of the setRegex code for glob pattern matching in metrics filter configurations.
> We could at least factor out the hairy setRegex code and package them into a GlobPattern class with a minimum compile method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8734-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 01:00:20 2010
Return-Path: <common-issues-return-8734-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36247 invoked from network); 3 Jun 2010 01:00:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 01:00:19 -0000
Received: (qmail 97604 invoked by uid 500); 3 Jun 2010 01:00:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97566 invoked by uid 500); 3 Jun 2010 01:00:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97411 invoked by uid 99); 3 Jun 2010 01:00:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 01:00:19 +0000
X-ASF-Spam-Status: No, hits=-1492.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 01:00:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o530xxjr017095
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 00:59:59 GMT
Message-ID: <10057969.148831275526799156.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 20:59:59 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6791) Refresh for proxy superuser config
  (common part for HDFS-1096)
In-Reply-To: <17022862.10441274913350679.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874897#action_12874897 ] 

Jitendra Nath Pandey commented on HADOOP-6791:
----------------------------------------------

+1 for the patch.

> Refresh for proxy superuser config  (common part for HDFS-1096)
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6791
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6791
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6791.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8735-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 01:08:19 2010
Return-Path: <common-issues-return-8735-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41335 invoked from network); 3 Jun 2010 01:08:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 01:08:19 -0000
Received: (qmail 734 invoked by uid 500); 3 Jun 2010 01:08:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 701 invoked by uid 500); 3 Jun 2010 01:08:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 692 invoked by uid 99); 3 Jun 2010 01:08:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 01:08:19 +0000
X-ASF-Spam-Status: No, hits=-1492.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 01:08:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5317w3P017215
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 01:07:58 GMT
Message-ID: <13996098.148941275527278481.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 21:07:58 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Lu updated HADOOP-6787:
----------------------------

     Issue Type: Bug  (was: Improvement)
    Description: 
Refactor the glob pattern code out of FileContext and FileSystem into a package private GlobFilter and the reusable GlobPattern class (InterfaceAudience.Private)

Also fix the handling of ^ outside character class ([...]) reported in HADOOP-6618 and make the glob pattern code less restrictive (not throwing on some valid glob patterns.) and more POSIX standard compliant (support [!...]).

  was:
The GlobFilter class (especially the error-prone setRegex method) is duplicated in both FileContext and Filesystem. And I'd like to be able to reuse the most of the setRegex code for glob pattern matching in metrics filter configurations.

We could at least factor out the hairy setRegex code and package them into a GlobPattern class with a minimum compile method.


Done.

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-trunk-v2.patch, hadoop-6787-y20s-v1.patch
>
>
> Refactor the glob pattern code out of FileContext and FileSystem into a package private GlobFilter and the reusable GlobPattern class (InterfaceAudience.Private)
> Also fix the handling of ^ outside character class ([...]) reported in HADOOP-6618 and make the glob pattern code less restrictive (not throwing on some valid glob patterns.) and more POSIX standard compliant (support [!...]).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8736-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 01:48:17 2010
Return-Path: <common-issues-return-8736-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47702 invoked from network); 3 Jun 2010 01:48:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 01:48:17 -0000
Received: (qmail 35923 invoked by uid 500); 3 Jun 2010 01:48:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35852 invoked by uid 500); 3 Jun 2010 01:48:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35844 invoked by uid 99); 3 Jun 2010 01:48:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 01:48:16 +0000
X-ASF-Spam-Status: No, hits=-1492.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 01:48:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o531luPu017670
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 01:47:56 GMT
Message-ID: <31540091.150011275529676395.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 21:47:56 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6793) CLONE -fs -ls does not work if a
 path name contains the ^ character
In-Reply-To: <14195346.17471274946457778.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE resolved HADOOP-6793.
--------------------------------------------

    Resolution: Duplicate

Close as duplicate of HADOOP-6618.

> CLONE -fs -ls does not work if a path name contains the ^ character
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6793
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6793
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: haijun_li
>
> Using a wildcard, the file is found.
> {noformat}
> -bash-3.1$ hadoop fs -ls k20d2f4/bin-2\?04+1_AF650AE776488A4D
> Found 1 items
> -rw-------   3 tsz users         17 2010-03-05 19:43 /user/tsz/k20d2f4/bin-2^04+1_AF650AE776488A4D
> {noformat}
> Replace the wildcard with ^, the file is not found.
> {noformat}
> -bash-3.1$ hadoop fs -ls k20d2f4/bin-2^04+1_AF650AE776488A4D
> ls: Cannot access k20d2f4/bin-2^04+1_AF650AE776488A4D: No such file or directory.
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8737-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 01:50:21 2010
Return-Path: <common-issues-return-8737-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48145 invoked from network); 3 Jun 2010 01:50:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 01:50:21 -0000
Received: (qmail 40320 invoked by uid 500); 3 Jun 2010 01:50:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40289 invoked by uid 500); 3 Jun 2010 01:50:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40281 invoked by uid 99); 3 Jun 2010 01:50:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 01:50:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 01:50:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o531nuMJ017693
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 01:49:57 GMT
Message-ID: <32995019.150081275529796824.JavaMail.jira@thor>
Date: Wed, 2 Jun 2010 21:49:56 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6618) fs -ls does not work if a path name
 contains the ^ character
In-Reply-To: <319633239.106491267825887593.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE resolved HADOOP-6618.
--------------------------------------------

    Resolution: Duplicate

Thanks Luke for fixing this in HADOOP-6787.

> fs -ls does not work if a path name contains the ^ character
> ------------------------------------------------------------
>
>                 Key: HADOOP-6618
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6618
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Tsz Wo (Nicholas), SZE
>         Attachments: c6618_20100305.patch
>
>
> Using a wildcard, the file is found.
> {noformat}
> -bash-3.1$ hadoop fs -ls k20d2f4/bin-2\?04+1_AF650AE776488A4D
> Found 1 items
> -rw-------   3 tsz users         17 2010-03-05 19:43 /user/tsz/k20d2f4/bin-2^04+1_AF650AE776488A4D
> {noformat}
> Replace the wildcard with ^, the file is not found.
> {noformat}
> -bash-3.1$ hadoop fs -ls k20d2f4/bin-2^04+1_AF650AE776488A4D
> ls: Cannot access k20d2f4/bin-2^04+1_AF650AE776488A4D: No such file or directory.
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8738-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 06:41:18 2010
Return-Path: <common-issues-return-8738-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15803 invoked from network); 3 Jun 2010 06:41:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 06:41:18 -0000
Received: (qmail 49020 invoked by uid 500); 3 Jun 2010 06:41:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48892 invoked by uid 500); 3 Jun 2010 06:41:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48874 invoked by uid 99); 3 Jun 2010 06:41:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 06:41:15 +0000
X-ASF-Spam-Status: No, hits=-1493.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 06:41:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o536eshP020175
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 06:40:54 GMT
Message-ID: <14219813.153311275547254621.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 02:40:54 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874977#action_12874977 ] 

Todd Lipcon commented on HADOOP-6762:
-------------------------------------

Hey Sam. Is it not sufficient to simply add a "this.interrupt()" in the catch clause of sendParams?

I was running into an occasional issue with 1 minute pauses during DFSClient pipeline recovery, and I think it's due to this issue. I added this.interrupt() and it seems to pass test case, I'll see if it also fixes my 1 minute pauses.

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8739-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 08:18:22 2010
Return-Path: <common-issues-return-8739-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42716 invoked from network); 3 Jun 2010 08:18:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 08:18:22 -0000
Received: (qmail 34004 invoked by uid 500); 3 Jun 2010 08:18:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33954 invoked by uid 500); 3 Jun 2010 08:18:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33946 invoked by uid 99); 3 Jun 2010 08:18:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 08:18:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 08:18:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o538Htkn021050
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 08:17:55 GMT
Message-ID: <14761051.154291275553075675.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 04:17:55 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875003#action_12875003 ] 

Rodrigo Schmidt commented on HADOOP-6796:
-----------------------------------------

I just realized the FileStatus constructors don't throw any exception. What should we do if the parameters fail the sanity check?

Should I create a new exception? If I do so, this will have a strong impact on several parts of Hadoop.

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8740-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 09:12:24 2010
Return-Path: <common-issues-return-8740-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54557 invoked from network); 3 Jun 2010 09:12:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 09:12:24 -0000
Received: (qmail 79352 invoked by uid 500); 3 Jun 2010 09:12:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79246 invoked by uid 500); 3 Jun 2010 09:12:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79238 invoked by uid 99); 3 Jun 2010 09:12:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 09:12:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 09:12:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o539BvFA021460
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 09:11:57 GMT
Message-ID: <24783186.154751275556317303.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 05:11:57 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6461) webapps aren't located correctly
 post-split
In-Reply-To: <428473464.1261365498148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875016#action_12875016 ] 

Hudson commented on HADOOP-6461:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #274 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/274/])
    HADOOP-6461.  Webapps aren't located correctly post-split.  Contributed by Todd Lipcon and Steve Loughran.


> webapps aren't located correctly post-split
> -------------------------------------------
>
>                 Key: HADOOP-6461
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6461
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Steve Loughran
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6461.patch, HADOOP-6461.patch, HADOOP-6461.patch, HADOOP-6461.patch, hadoop-6461.patch, hadoop-6461.patch, hadoop-6461.patch, HADOOP-6461.txt, hadoop-6461.txt, hadoop-6461.txt
>
>
> Post-split, when one builds common it creates an empty build/webapps dir. If you then try to launch the NN for example using HADOOP_HDFS_HOME=/path/to/hdfs hdfs namenode, HttpServer.getWebAppsPath locates the empty common webapps dir, and the NN web UI fails to load.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8741-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 11:25:40 2010
Return-Path: <common-issues-return-8741-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90861 invoked from network); 3 Jun 2010 11:25:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 11:25:39 -0000
Received: (qmail 16920 invoked by uid 500); 3 Jun 2010 11:25:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16522 invoked by uid 500); 3 Jun 2010 11:25:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16508 invoked by uid 99); 3 Jun 2010 11:25:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 11:25:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 11:25:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53BPCfP022739
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 11:25:13 GMT
Message-ID: <15481106.156461275564312947.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 07:25:12 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6461) webapps aren't located correctly
 post-split
In-Reply-To: <428473464.1261365498148.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875061#action_12875061 ] 

Hudson commented on HADOOP-6461:
--------------------------------

Integrated in Hadoop-Common-trunk #355 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/355/])
    HADOOP-6461.  Webapps aren't located correctly post-split.  Contributed by Todd Lipcon and Steve Loughran.


> webapps aren't located correctly post-split
> -------------------------------------------
>
>                 Key: HADOOP-6461
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6461
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Steve Loughran
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6461.patch, HADOOP-6461.patch, HADOOP-6461.patch, HADOOP-6461.patch, hadoop-6461.patch, hadoop-6461.patch, hadoop-6461.patch, HADOOP-6461.txt, hadoop-6461.txt, hadoop-6461.txt
>
>
> Post-split, when one builds common it creates an empty build/webapps dir. If you then try to launch the NN for example using HADOOP_HDFS_HOME=/path/to/hdfs hdfs namenode, HttpServer.getWebAppsPath locates the empty common webapps dir, and the NN web UI fails to load.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8742-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 11:27:38 2010
Return-Path: <common-issues-return-8742-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 91393 invoked from network); 3 Jun 2010 11:27:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 11:27:36 -0000
Received: (qmail 21468 invoked by uid 500); 3 Jun 2010 11:27:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20492 invoked by uid 500); 3 Jun 2010 11:27:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20415 invoked by uid 99); 3 Jun 2010 11:27:30 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 11:27:30 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 11:27:28 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53BR6Zv022778
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 11:27:07 GMT
Message-ID: <20002294.156501275564426940.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 07:27:06 -0400 (EDT)
From: "GIRISH N IYER (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6806) While running HADOOP "Exception in
 thread "main" java.lang.NullPointerException " occurs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

While running HADOOP "Exception in thread "main" java.lang.NullPointerException " occurs
----------------------------------------------------------------------------------------

                 Key: HADOOP-6806
                 URL: https://issues.apache.org/jira/browse/HADOOP-6806
             Project: Hadoop Common
          Issue Type: Task
         Environment: CentOS  5.0 
            Reporter: GIRISH N IYER


hi,
while installin hadoop ... i can do the following step   
 "bin/hadoop namenode -format"...it got success
After when i do
 "bin/start-all.sh"....the following error occurs........

starting namenode, logging to /hadoop/hadoop/bin/../logs/hadoop-hadoop-namenode-localhost.localdomain.out
localhost: ssh: localhost: Name or service not known
hadoop@master's password: 
master: starting secondarynamenode, logging to /hadoop/hadoop/bin/../logs/hadoop-hadoop-secondarynamenode-localhost.localdomain.out
master: Exception in thread "main" java.lang.NullPointerException
master:         at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:134)
master:         at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:156)
master:         at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:160)
master:         at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:131)
master:         at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:115)
master:         at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:469)
starting jobtracker, logging to /hadoop/hadoop/bin/../logs/hadoop-hadoop-jobtracker-localhost.localdomain.out
localhost: ssh: localhost: Name or service not known


Pls help me to solve this........

Also i did not configure 

"nano conf/core-site.xml"

"nano conf/hdfs-site.xml"

"nano conf/mapred-site.xml"


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8743-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 13:36:28 2010
Return-Path: <common-issues-return-8743-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48428 invoked from network); 3 Jun 2010 13:36:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 13:36:28 -0000
Received: (qmail 76363 invoked by uid 500); 3 Jun 2010 13:36:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76333 invoked by uid 500); 3 Jun 2010 13:36:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76325 invoked by uid 99); 3 Jun 2010 13:36:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 13:36:27 +0000
X-ASF-Spam-Status: No, hits=-1494.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 13:36:26 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53Da67w029067
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 13:36:06 GMT
Message-ID: <21924662.158451275572166366.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 09:36:06 -0400 (EDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6797) DNS#getHosts() fallback leads to
 mix of network-interface addresse in case reverse lookup fails
In-Reply-To: <33178339.101931275392857496.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875105#action_12875105 ] 

Steve Loughran commented on HADOOP-6797:
----------------------------------------

adding code to catch when {{InetAddress.getByName(ips[ctr])}} was null, presumably?

> DNS#getHosts() fallback leads to mix of network-interface addresse in case reverse lookup fails
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6797
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6797
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: Johannes Zillmann
>
> org.apache.hadoop.net.DNS#getHosts():
> {noformat}
> public static String[] getHosts(String strInterface, String nameserver)
>     throws UnknownHostException {
>     String[] ips = getIPs(strInterface);
>     Vector<String> hosts = new Vector<String>();
>     for (int ctr = 0; ctr < ips.length; ctr++)
>       try {
>         hosts.add(reverseDns(InetAddress.getByName(ips[ctr]),
>                              nameserver));
>       } catch (Exception e) {
>       }
>     if (hosts.size() == 0)
>       return new String[] { InetAddress.getLocalHost().getCanonicalHostName() };
>     else
>       return hosts.toArray(new String[] {});
>   }
> {noformat}
> I have the situation where i choosing eth1 as network interface (fs.datanode.dns.interface and mapred.tasktracker.dns.interface) which is the internal interface.
> The reverse lookup fails for eth1 so the fallback:
> {noformat}
>     if (hosts.size() == 0)
>       return new String[] { InetAddress.getLocalHost().getCanonicalHostName() };
> {noformat}
> comes to action. The dns of eth0 is returned which is the external interface.
> This leads in my case to a combination of internal ip and external dns as the default ip/host.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8744-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 13:48:23 2010
Return-Path: <common-issues-return-8744-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52207 invoked from network); 3 Jun 2010 13:48:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 13:48:22 -0000
Received: (qmail 96028 invoked by uid 500); 3 Jun 2010 13:48:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95997 invoked by uid 500); 3 Jun 2010 13:48:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95989 invoked by uid 99); 3 Jun 2010 13:48:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 13:48:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 13:48:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53DlxeX029182
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 13:47:59 GMT
Message-ID: <16226985.158671275572878986.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 09:47:58 -0400 (EDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6807) Log directly from jetty to commons
 logging, bypassing SLF4J
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Log directly from jetty to commons logging, bypassing SLF4J
-----------------------------------------------------------

                 Key: HADOOP-6807
                 URL: https://issues.apache.org/jira/browse/HADOOP-6807
             Project: Hadoop Common
          Issue Type: Improvement
          Components: io
    Affects Versions: 0.21.0
            Reporter: Steve Loughran
            Priority: Minor


Jetty may default to logging through SLF4J, but don't actually need it; you can provide your own logger which goes straight to commons-logging, I've done something similar in the past:

[[http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/components/jetty/src/org/smartfrog/services/jetty/log/]]

You just need to point jetty at the relevant logger by setting up the relevant JVM property, such as {{-Dorg.mortbay.log.class=org.smartfrog.services.jetty.log.JettyLogger}}. Doing something similar in Avro/Hadoop-common would eliminate SLF4J from the dependency graph, route via commons-logging and then, usually, to Log4J, so having only one log to manage. All JVM startup scripts would need to set the relevant property, it is harmless on JVMs that don't start up Jetty6+

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8745-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 13:48:24 2010
Return-Path: <common-issues-return-8745-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52257 invoked from network); 3 Jun 2010 13:48:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 13:48:24 -0000
Received: (qmail 96184 invoked by uid 500); 3 Jun 2010 13:48:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96142 invoked by uid 500); 3 Jun 2010 13:48:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96134 invoked by uid 99); 3 Jun 2010 13:48:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 13:48:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 13:48:21 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53Dm0Lx029194
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 13:48:00 GMT
Message-ID: <177997.158711275572880525.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 09:48:00 -0400 (EDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6800) Harmonize JAR library versions
In-Reply-To: <8215141.145191275515876547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875113#action_12875113 ] 

Steve Loughran commented on HADOOP-6800:
----------------------------------------

+1 for this; I'd go for moving to the latest versions that didn't cause too much grief. 

HADOOP-6807 claims that SL4J is not needed for Jetty



> Harmonize JAR library versions
> ------------------------------
>
>                 Key: HADOOP-6800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>
> Currently, multiple versions of the same library JAR are being pulled in for various reasons.
> * Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
> * Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8746-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 13:52:22 2010
Return-Path: <common-issues-return-8746-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53683 invoked from network); 3 Jun 2010 13:52:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 13:52:22 -0000
Received: (qmail 3363 invoked by uid 500); 3 Jun 2010 13:52:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3330 invoked by uid 500); 3 Jun 2010 13:52:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3322 invoked by uid 99); 3 Jun 2010 13:52:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 13:52:22 +0000
X-ASF-Spam-Status: No, hits=-1494.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 13:52:21 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53Dq19T029273
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 13:52:01 GMT
Message-ID: <8708027.158901275573121173.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 09:52:01 -0400 (EDT)
From: "Johannes Zillmann (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6797) DNS#getHosts() fallback leads to
 mix of network-interface addresse in case reverse lookup fails
In-Reply-To: <33178339.101931275392857496.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875116#action_12875116 ] 

Johannes Zillmann commented on HADOOP-6797:
-------------------------------------------

not sure if this could happen, but probably a good idea.

> DNS#getHosts() fallback leads to mix of network-interface addresse in case reverse lookup fails
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6797
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6797
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: Johannes Zillmann
>
> org.apache.hadoop.net.DNS#getHosts():
> {noformat}
> public static String[] getHosts(String strInterface, String nameserver)
>     throws UnknownHostException {
>     String[] ips = getIPs(strInterface);
>     Vector<String> hosts = new Vector<String>();
>     for (int ctr = 0; ctr < ips.length; ctr++)
>       try {
>         hosts.add(reverseDns(InetAddress.getByName(ips[ctr]),
>                              nameserver));
>       } catch (Exception e) {
>       }
>     if (hosts.size() == 0)
>       return new String[] { InetAddress.getLocalHost().getCanonicalHostName() };
>     else
>       return hosts.toArray(new String[] {});
>   }
> {noformat}
> I have the situation where i choosing eth1 as network interface (fs.datanode.dns.interface and mapred.tasktracker.dns.interface) which is the internal interface.
> The reverse lookup fails for eth1 so the fallback:
> {noformat}
>     if (hosts.size() == 0)
>       return new String[] { InetAddress.getLocalHost().getCanonicalHostName() };
> {noformat}
> comes to action. The dns of eth0 is returned which is the external interface.
> This leads in my case to a combination of internal ip and external dns as the default ip/host.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8747-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 14:08:28 2010
Return-Path: <common-issues-return-8747-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63310 invoked from network); 3 Jun 2010 14:08:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 14:08:27 -0000
Received: (qmail 34196 invoked by uid 500); 3 Jun 2010 14:08:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34164 invoked by uid 500); 3 Jun 2010 14:08:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34156 invoked by uid 99); 3 Jun 2010 14:08:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 14:08:27 +0000
X-ASF-Spam-Status: No, hits=-1494.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 14:08:27 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53E86vN029460
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 14:08:06 GMT
Message-ID: <1672699.159081275574086563.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 10:08:06 -0400 (EDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6797) DNS#getHosts() fallback leads to
 mix of network-interface addresse in case reverse lookup fails
In-Reply-To: <33178339.101931275392857496.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875120#action_12875120 ] 

Steve Loughran commented on HADOOP-6797:
----------------------------------------

Anything can go wrong with networking given a sufficiently large number of machines. 
HADOOP-3426 covers machines throwing exceptions on getting their local name, HDFS-116 is an NPE triggered by the same machine's "issues"


> DNS#getHosts() fallback leads to mix of network-interface addresse in case reverse lookup fails
> -----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6797
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6797
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: Johannes Zillmann
>
> org.apache.hadoop.net.DNS#getHosts():
> {noformat}
> public static String[] getHosts(String strInterface, String nameserver)
>     throws UnknownHostException {
>     String[] ips = getIPs(strInterface);
>     Vector<String> hosts = new Vector<String>();
>     for (int ctr = 0; ctr < ips.length; ctr++)
>       try {
>         hosts.add(reverseDns(InetAddress.getByName(ips[ctr]),
>                              nameserver));
>       } catch (Exception e) {
>       }
>     if (hosts.size() == 0)
>       return new String[] { InetAddress.getLocalHost().getCanonicalHostName() };
>     else
>       return hosts.toArray(new String[] {});
>   }
> {noformat}
> I have the situation where i choosing eth1 as network interface (fs.datanode.dns.interface and mapred.tasktracker.dns.interface) which is the internal interface.
> The reverse lookup fails for eth1 so the fallback:
> {noformat}
>     if (hosts.size() == 0)
>       return new String[] { InetAddress.getLocalHost().getCanonicalHostName() };
> {noformat}
> comes to action. The dns of eth0 is returned which is the external interface.
> This leads in my case to a combination of internal ip and external dns as the default ip/host.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8748-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 15:10:23 2010
Return-Path: <common-issues-return-8748-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85793 invoked from network); 3 Jun 2010 15:10:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 15:10:22 -0000
Received: (qmail 22547 invoked by uid 500); 3 Jun 2010 15:10:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22515 invoked by uid 500); 3 Jun 2010 15:10:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22507 invoked by uid 99); 3 Jun 2010 15:10:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 15:10:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 15:10:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53F9wk6000045
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 15:09:58 GMT
Message-ID: <28859623.160021275577798782.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 11:09:58 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875140#action_12875140 ] 

sam rash commented on HADOOP-6762:
----------------------------------

so you mean add Thread.currentThread.interrupt() above markClosed(e) ?  I don't think this fixes the underlying problem.

The problem is if a channel is doing a wait for IO and an interrupt comes in, you get a ClosedByInterrupt exception

http://java.sun.com/javase/6/docs/api/java/nio/channels/ClosedByInterruptException.html

from the doc, this closes the channel and sets the interrupt status (ie interrupt() shouldn't have an effect--means my test doesn't repro the same thing I saw).
What I then saw was that other RPC instances using the same channel would get ChannelClosedException.  The only way to avoid this in the FileSystem case was to move the thread that uses the channel to its own so the lease checker won't interrupt it.

I'll play with the test case and see why your change makes it pass when it doesn't seem like it can't fix the underlying problem

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8749-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 15:51:21 2010
Return-Path: <common-issues-return-8749-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95740 invoked from network); 3 Jun 2010 15:51:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 15:51:20 -0000
Received: (qmail 85692 invoked by uid 500); 3 Jun 2010 15:51:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85627 invoked by uid 500); 3 Jun 2010 15:51:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85555 invoked by uid 99); 3 Jun 2010 15:51:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 15:51:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 15:51:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53Fou2q000544
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 15:50:56 GMT
Message-ID: <11535794.160831275580256575.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 11:50:56 -0400 (EDT)
From: "Jeff Whiting (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5124) A few optimizations to
 FsNamesystem#RecentInvalidateSets
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jeff Whiting updated HADOOP-5124:
---------------------------------

    Attachment: optimizeInvalidate2-0.20.2.patch

I needed this in the 0.20.2 branch and created a patch for it.  I thought it might be useful for others.

> A few optimizations to FsNamesystem#RecentInvalidateSets
> --------------------------------------------------------
>
>                 Key: HADOOP-5124
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5124
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.21.0
>
>         Attachments: optimizeInvalidate.patch, optimizeInvalidate1.patch, optimizeInvalidate2-0.20.2.patch, optimizeInvalidate2.patch
>
>
> This jira proposes a few optimization to FsNamesystem#RecentInvalidateSets:
> 1. when removing all replicas of a block, it does not traverse all nodes in the map. Instead it traverse only the nodes that the block is located.
> 2. When dispatching blocks to datanodes in ReplicationMonitor. It randomly chooses a predefined number of datanodes and dispatches blocks to those datanodes. This strategy provides fairness to all datanodes. The current strategy always starts from the first datanode.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8750-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 17:05:25 2010
Return-Path: <common-issues-return-8750-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25496 invoked from network); 3 Jun 2010 17:05:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 17:05:25 -0000
Received: (qmail 91039 invoked by uid 500); 3 Jun 2010 17:05:25 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91008 invoked by uid 500); 3 Jun 2010 17:05:25 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91000 invoked by uid 99); 3 Jun 2010 17:05:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 17:05:25 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 17:05:22 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53H51UH001588
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 17:05:01 GMT
Message-ID: <25087149.161821275584701017.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 13:05:01 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6787:
--------------------------------

    Status: Open  (was: Patch Available)

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-trunk-v2.patch, hadoop-6787-y20s-v1.patch
>
>
> Refactor the glob pattern code out of FileContext and FileSystem into a package private GlobFilter and the reusable GlobPattern class (InterfaceAudience.Private)
> Also fix the handling of ^ outside character class ([...]) reported in HADOOP-6618 and make the glob pattern code less restrictive (not throwing on some valid glob patterns.) and more POSIX standard compliant (support [!...]).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8751-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 17:05:26 2010
Return-Path: <common-issues-return-8751-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25541 invoked from network); 3 Jun 2010 17:05:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 17:05:25 -0000
Received: (qmail 91184 invoked by uid 500); 3 Jun 2010 17:05:25 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91151 invoked by uid 500); 3 Jun 2010 17:05:25 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91143 invoked by uid 99); 3 Jun 2010 17:05:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 17:05:25 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 17:05:23 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53H52HC001603
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 17:05:02 GMT
Message-ID: <23457460.161871275584702094.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 13:05:02 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6787:
--------------------------------

    Status: Patch Available  (was: Open)

Thanks Luke. Kicking hudson with the new patch.

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-trunk-v2.patch, hadoop-6787-y20s-v1.patch
>
>
> Refactor the glob pattern code out of FileContext and FileSystem into a package private GlobFilter and the reusable GlobPattern class (InterfaceAudience.Private)
> Also fix the handling of ^ outside character class ([...]) reported in HADOOP-6618 and make the glob pattern code less restrictive (not throwing on some valid glob patterns.) and more POSIX standard compliant (support [!...]).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8752-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 17:11:21 2010
Return-Path: <common-issues-return-8752-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29396 invoked from network); 3 Jun 2010 17:11:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 17:11:21 -0000
Received: (qmail 4425 invoked by uid 500); 3 Jun 2010 17:11:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4364 invoked by uid 500); 3 Jun 2010 17:11:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4341 invoked by uid 99); 3 Jun 2010 17:11:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 17:11:21 +0000
X-ASF-Spam-Status: No, hits=-1494.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 17:11:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53HB0Gk001637
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 17:11:00 GMT
Message-ID: <28243267.161931275585060005.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 13:11:00 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875193#action_12875193 ] 

sam rash commented on HADOOP-6762:
----------------------------------

maybe you can elaborate on where you are adding the interrupt() call as trying what I understand you to have suggested didn't make the test pass.

Is this what you mean for sendParam:

{code}
    public void sendParam(Call call) {
      if (shouldCloseConnection.get()) {
        return;
      }

      DataOutputBuffer d=null;
      try {
        synchronized (this.out) {
          if (LOG.isDebugEnabled())
            LOG.debug(getName() + " sending #" + call.id);
          
          //for serializing the
          //data to be written
          d = new DataOutputBuffer();
          d.writeInt(call.id);
          call.param.write(d);
          byte[] data = d.getData();
          int dataLength = d.getLength();
          out.writeInt(dataLength);      //first put the data length
          out.write(data, 0, dataLength);//write the data
          out.flush();
        }
      } catch(IOException e) {
        Thread.currentThread().interrupt(); //add this--should have no effect on ClosedByInterrupt as it sets the current thread to be interrupted already
        markClosed(e);
      } finally {
        //the buffer is just an in-memory buffer, but it is still polite to
        // close early
        IOUtils.closeStream(d);
      }
    }  
{code}

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8753-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 17:21:22 2010
Return-Path: <common-issues-return-8753-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33090 invoked from network); 3 Jun 2010 17:21:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 17:21:22 -0000
Received: (qmail 22144 invoked by uid 500); 3 Jun 2010 17:21:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22103 invoked by uid 500); 3 Jun 2010 17:21:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22095 invoked by uid 99); 3 Jun 2010 17:21:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 17:21:22 +0000
X-ASF-Spam-Status: No, hits=-1494.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 17:21:21 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53HL1pI001715
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 17:21:01 GMT
Message-ID: <6715993.162111275585661098.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 13:21:01 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875197#action_12875197 ] 

Todd Lipcon commented on HADOOP-6762:
-------------------------------------

Hey Sam. Instead of Thread.currentThread().interrupt(), I mean this.interrupt(), in order to interrupt the Connection thread (not the caller thread).

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8754-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 17:25:23 2010
Return-Path: <common-issues-return-8754-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34387 invoked from network); 3 Jun 2010 17:25:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 17:25:23 -0000
Received: (qmail 25571 invoked by uid 500); 3 Jun 2010 17:25:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25547 invoked by uid 500); 3 Jun 2010 17:25:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25539 invoked by uid 99); 3 Jun 2010 17:25:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 17:25:23 +0000
X-ASF-Spam-Status: No, hits=-1494.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 17:25:23 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53HP2ef001753
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 17:25:02 GMT
Message-ID: <16197351.162191275585902568.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 13:25:02 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875199#action_12875199 ] 

Eli Collins commented on HADOOP-6796:
-------------------------------------

If it's never ok to pass null for the path (ie a bug in the code) then you can assert path != null or throw an AssertionError. If it's OK sometimes and not others than IllegalArgumentException seems appropriate, seems like it should never be legitimate to create a FileStatus w/o a path unless using the no argument constructor. 


> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8755-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 17:27:23 2010
Return-Path: <common-issues-return-8755-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34967 invoked from network); 3 Jun 2010 17:27:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 17:27:23 -0000
Received: (qmail 28606 invoked by uid 500); 3 Jun 2010 17:27:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28576 invoked by uid 500); 3 Jun 2010 17:27:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28568 invoked by uid 99); 3 Jun 2010 17:27:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 17:27:22 +0000
X-ASF-Spam-Status: No, hits=-1494.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 17:27:22 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53HR10I001791
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 17:27:02 GMT
Message-ID: <25138637.162231275586021705.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 13:27:01 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875200#action_12875200 ] 

sam rash commented on HADOOP-6762:
----------------------------------

sorry, I'm still confused--once you catch the IOException here, the channel is closed.  How does interrupting the Connection help avoid this?

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8756-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 17:31:24 2010
Return-Path: <common-issues-return-8756-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36496 invoked from network); 3 Jun 2010 17:31:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 17:31:24 -0000
Received: (qmail 34807 invoked by uid 500); 3 Jun 2010 17:31:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34771 invoked by uid 500); 3 Jun 2010 17:31:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34763 invoked by uid 99); 3 Jun 2010 17:31:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 17:31:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 17:31:22 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53HV0rX001835
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 17:31:00 GMT
Message-ID: <5750277.162331275586260354.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 13:31:00 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875203#action_12875203 ] 

sam rash commented on HADOOP-6762:
----------------------------------

maybe my test case is not sufficiently deterministic so it works differently for you.  I replaced "Thread.currentThread.interrupt()" with "this.interrupt()" in the code I pasted above.  

The test still fails
(and regardless, I don't think that can affect the underlying problem that if something interrupts a thread that is doing RPC and that thread is blocked on the channel, it gets closed automatically)


> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8757-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 17:37:21 2010
Return-Path: <common-issues-return-8757-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37588 invoked from network); 3 Jun 2010 17:37:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 17:37:21 -0000
Received: (qmail 41453 invoked by uid 500); 3 Jun 2010 17:37:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41335 invoked by uid 500); 3 Jun 2010 17:37:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41327 invoked by uid 99); 3 Jun 2010 17:37:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 17:37:21 +0000
X-ASF-Spam-Status: No, hits=-1494.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 17:37:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53Hb06f001934
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 17:37:00 GMT
Message-ID: <21570481.162601275586620568.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 13:37:00 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6791) Refresh for proxy superuser config
  (common part for HDFS-1096)
In-Reply-To: <17022862.10441274913350679.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6791:
-----------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

committed to trunk

> Refresh for proxy superuser config  (common part for HDFS-1096)
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6791
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6791
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6791.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8758-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 17:49:20 2010
Return-Path: <common-issues-return-8758-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40078 invoked from network); 3 Jun 2010 17:49:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 17:49:20 -0000
Received: (qmail 59647 invoked by uid 500); 3 Jun 2010 17:49:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59591 invoked by uid 500); 3 Jun 2010 17:49:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 59583 invoked by uid 99); 3 Jun 2010 17:49:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 17:49:20 +0000
X-ASF-Spam-Status: No, hits=-1494.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 17:49:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53HmxiS002093
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 17:48:59 GMT
Message-ID: <30975342.163001275587339479.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 13:48:59 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6796:
------------------------------------

    Attachment: HADOOP-6796.1.patch

Does this new patch look better?

If so, I'll work on some unit tests.

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.1.patch, HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8759-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 17:53:22 2010
Return-Path: <common-issues-return-8759-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40885 invoked from network); 3 Jun 2010 17:53:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 17:53:22 -0000
Received: (qmail 70882 invoked by uid 500); 3 Jun 2010 17:53:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70859 invoked by uid 500); 3 Jun 2010 17:53:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70851 invoked by uid 99); 3 Jun 2010 17:53:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 17:53:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 17:53:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53Hqwud002130
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 17:52:58 GMT
Message-ID: <3533899.163101275587578290.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 13:52:58 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6791) Refresh for proxy superuser config
  (common part for HDFS-1096)
In-Reply-To: <17022862.10441274913350679.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875218#action_12875218 ] 

Hudson commented on HADOOP-6791:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #275 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/275/])
    HADOOP-6791.  Refresh for proxy superuser config


> Refresh for proxy superuser config  (common part for HDFS-1096)
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6791
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6791
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6791.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8760-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 18:55:18 2010
Return-Path: <common-issues-return-8760-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72004 invoked from network); 3 Jun 2010 18:55:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 18:55:18 -0000
Received: (qmail 960 invoked by uid 500); 3 Jun 2010 18:55:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 934 invoked by uid 500); 3 Jun 2010 18:55:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 926 invoked by uid 99); 3 Jun 2010 18:55:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 18:55:18 +0000
X-ASF-Spam-Status: No, hits=-1495.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 18:55:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53IsvA3003065
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 18:54:57 GMT
Message-ID: <990255.164971275591297619.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 14:54:57 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6674) Performance Improvement in Secure RPC
In-Reply-To: <375873020.645441270163787224.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6674:
-----------------------------------------

    Attachment: HADOOP-6674.1.patch

Patch for trunk.

> Performance Improvement in Secure RPC
> -------------------------------------
>
>                 Key: HADOOP-6674
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6674
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.patch, HADOOP-6674.1.patch
>
>
> This jira introduces two performance improvements in Sasl RPC
> 1. Setting of Sasl 'quality of protection' to authentication only.
> 2. Addition of BufferedOutputStream underneath SaslOutputStream.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8761-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 18:55:25 2010
Return-Path: <common-issues-return-8761-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72262 invoked from network); 3 Jun 2010 18:55:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 18:55:22 -0000
Received: (qmail 2239 invoked by uid 500); 3 Jun 2010 18:55:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2204 invoked by uid 500); 3 Jun 2010 18:55:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2196 invoked by uid 99); 3 Jun 2010 18:55:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 18:55:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 18:55:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53IswIj003083
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 18:54:58 GMT
Message-ID: <11259771.165031275591298398.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 14:54:58 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6674) Performance Improvement in Secure RPC
In-Reply-To: <375873020.645441270163787224.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6674:
-----------------------------------------

    Status: Patch Available  (was: Open)

> Performance Improvement in Secure RPC
> -------------------------------------
>
>                 Key: HADOOP-6674
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6674
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.patch, HADOOP-6674.1.patch
>
>
> This jira introduces two performance improvements in Sasl RPC
> 1. Setting of Sasl 'quality of protection' to authentication only.
> 2. Addition of BufferedOutputStream underneath SaslOutputStream.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8762-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 19:13:20 2010
Return-Path: <common-issues-return-8762-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80162 invoked from network); 3 Jun 2010 19:13:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 19:13:20 -0000
Received: (qmail 21144 invoked by uid 500); 3 Jun 2010 19:13:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21110 invoked by uid 500); 3 Jun 2010 19:13:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21102 invoked by uid 99); 3 Jun 2010 19:13:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 19:13:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 19:13:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53JCtgV003352
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 19:12:56 GMT
Message-ID: <24111693.165551275592375869.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 15:12:55 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6788) [Herriot] Exception exclusion
 functionality is not working correctly.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875255#action_12875255 ] 

Hadoop QA commented on HADOOP-6788:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446103/HADOOP-6788.patch
  against trunk revision 951081.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/74/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/74/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/74/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/74/console

This message is automatically generated.

> [Herriot] Exception exclusion functionality is not working correctly.
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6788
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6788
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Balaji Rajagopalan
>         Attachments: 6788-ydist-security.patch, 6788-ydist-security.patch, HADOOP-6788.patch, HADOOP-6788.patch
>
>
> Exception exclusion functionality is not working correctly because of that tests are failing by not matching the error count.
> I debugged the issue and found that the problem with shell command which is generating in the getNumberOfMatchesInLogFile function.
> Currently building the shell command in the following way. 
> if(list != null){
>   for(int i =0; i < list.length; ++i)
>   {
>     filePattern.append(" | grep -v " + list[i] );
>   }
> }
>     String[] cmd =
>         new String[] {
>             "bash",
>             "-c",
>             "grep -c "
>                 + pattern + " " + filePattern
>                 + " | awk -F: '{s+=$2} END {print s}'" };    
> However, The above commnad won't work correctly because you are counting the exceptions in the file before excluding the known exceptions.
> In this case it gives the mismatch error counts everytime.The shell command should be in the following way to work correctly.
> if (list != null) {
>   int index = 0;
>   for (String excludeExp : list) {
>     filePattern.append((++index < list.length)? "| grep -v " : 
>             "| grep -vc " + list[i] );  
>   }
> }
> String[] cmd =
>    new String[] {
>        "bash",
>        "-c",
>        "grep "
>            + pattern + " " + filePattern
>            + " | awk -F: '{s+=$2} END {print s}'" };  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8763-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 19:19:22 2010
Return-Path: <common-issues-return-8763-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82852 invoked from network); 3 Jun 2010 19:19:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 19:19:22 -0000
Received: (qmail 26867 invoked by uid 500); 3 Jun 2010 19:19:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26830 invoked by uid 500); 3 Jun 2010 19:19:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26822 invoked by uid 99); 3 Jun 2010 19:19:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 19:19:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 19:19:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53JIwJM003382
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 19:18:58 GMT
Message-ID: <19268101.165621275592738453.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 15:18:58 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6788) [Herriot] Exception exclusion
 functionality is not working correctly.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875257#action_12875257 ] 

Hadoop QA commented on HADOOP-6788:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446103/HADOOP-6788.patch
  against trunk revision 951081.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/559/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/559/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/559/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/559/console

This message is automatically generated.

> [Herriot] Exception exclusion functionality is not working correctly.
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6788
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6788
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Balaji Rajagopalan
>         Attachments: 6788-ydist-security.patch, 6788-ydist-security.patch, HADOOP-6788.patch, HADOOP-6788.patch
>
>
> Exception exclusion functionality is not working correctly because of that tests are failing by not matching the error count.
> I debugged the issue and found that the problem with shell command which is generating in the getNumberOfMatchesInLogFile function.
> Currently building the shell command in the following way. 
> if(list != null){
>   for(int i =0; i < list.length; ++i)
>   {
>     filePattern.append(" | grep -v " + list[i] );
>   }
> }
>     String[] cmd =
>         new String[] {
>             "bash",
>             "-c",
>             "grep -c "
>                 + pattern + " " + filePattern
>                 + " | awk -F: '{s+=$2} END {print s}'" };    
> However, The above commnad won't work correctly because you are counting the exceptions in the file before excluding the known exceptions.
> In this case it gives the mismatch error counts everytime.The shell command should be in the following way to work correctly.
> if (list != null) {
>   int index = 0;
>   for (String excludeExp : list) {
>     filePattern.append((++index < list.length)? "| grep -v " : 
>             "| grep -vc " + list[i] );  
>   }
> }
> String[] cmd =
>    new String[] {
>        "bash",
>        "-c",
>        "grep "
>            + pattern + " " + filePattern
>            + " | awk -F: '{s+=$2} END {print s}'" };  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8764-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 19:27:19 2010
Return-Path: <common-issues-return-8764-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85681 invoked from network); 3 Jun 2010 19:27:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 19:27:19 -0000
Received: (qmail 37822 invoked by uid 500); 3 Jun 2010 19:27:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37789 invoked by uid 500); 3 Jun 2010 19:27:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37781 invoked by uid 99); 3 Jun 2010 19:27:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 19:27:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 19:27:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53JQs5m003459
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 19:26:54 GMT
Message-ID: <22475506.165821275593214455.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 15:26:54 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6803) Add native gzip read/write coverage
 to TestCodec
In-Reply-To: <20681213.146461275520316684.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875261#action_12875261 ] 

Hadoop QA commented on HADOOP-6803:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446204/hadoop-6803-2.patch
  against trunk revision 951081.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/75/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/75/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/75/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/75/console

This message is automatically generated.

> Add native gzip read/write coverage to TestCodec 
> -------------------------------------------------
>
>                 Key: HADOOP-6803
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6803
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: io
>    Affects Versions: 0.22.0
>            Reporter: Eli Collins
>            Assignee: Eli Collins
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6803-1.patch, hadoop-6803-2.patch
>
>
> Looking at ZlibCompressor I noticed that the finished member is never modified, and is therefore always false. This means ZlibCompressor#finished will always return false so CompressorStream#close loops indefinitely in finish:
> {code} 
>       while (!compressor.finished()) {
>         compress();
>       }
> {code}
> I modifed TestCodec#testGzipCodecWrite to also cover writing using the native lib and confirmed the hang with jstack. The fix is simple, ZlibCompressor should record when it's been finished.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8765-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 19:37:23 2010
Return-Path: <common-issues-return-8765-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88269 invoked from network); 3 Jun 2010 19:37:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 19:37:23 -0000
Received: (qmail 49161 invoked by uid 500); 3 Jun 2010 19:37:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49110 invoked by uid 500); 3 Jun 2010 19:37:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49102 invoked by uid 99); 3 Jun 2010 19:37:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 19:37:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 19:37:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53JaxvN003533
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 19:36:59 GMT
Message-ID: <5260782.165981275593819290.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 15:36:59 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875264#action_12875264 ] 

Hadoop QA commented on HADOOP-6787:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446176/hadoop-6787-trunk-v2.patch
  against trunk revision 951081.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 2 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/560/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/560/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/560/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/560/console

This message is automatically generated.

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-trunk-v2.patch, hadoop-6787-y20s-v1.patch
>
>
> Refactor the glob pattern code out of FileContext and FileSystem into a package private GlobFilter and the reusable GlobPattern class (InterfaceAudience.Private)
> Also fix the handling of ^ outside character class ([...]) reported in HADOOP-6618 and make the glob pattern code less restrictive (not throwing on some valid glob patterns.) and more POSIX standard compliant (support [!...]).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8766-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 19:41:20 2010
Return-Path: <common-issues-return-8766-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89256 invoked from network); 3 Jun 2010 19:41:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 19:41:20 -0000
Received: (qmail 54286 invoked by uid 500); 3 Jun 2010 19:41:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 54250 invoked by uid 500); 3 Jun 2010 19:41:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 54242 invoked by uid 99); 3 Jun 2010 19:41:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 19:41:20 +0000
X-ASF-Spam-Status: No, hits=-1495.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 19:41:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53Jew5X003560
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 19:40:59 GMT
Message-ID: <6970581.166041275594058708.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 15:40:58 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6674) Performance Improvement in Secure
 RPC
In-Reply-To: <375873020.645441270163787224.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875265#action_12875265 ] 

Hadoop QA commented on HADOOP-6674:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446274/HADOOP-6674.1.patch
  against trunk revision 951081.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/76/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/76/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/76/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/76/console

This message is automatically generated.

> Performance Improvement in Secure RPC
> -------------------------------------
>
>                 Key: HADOOP-6674
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6674
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.patch, HADOOP-6674.1.patch
>
>
> This jira introduces two performance improvements in Sasl RPC
> 1. Setting of Sasl 'quality of protection' to authentication only.
> 2. Addition of BufferedOutputStream underneath SaslOutputStream.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8767-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 20:30:19 2010
Return-Path: <common-issues-return-8767-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4623 invoked from network); 3 Jun 2010 20:30:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 20:30:19 -0000
Received: (qmail 8681 invoked by uid 500); 3 Jun 2010 20:30:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8644 invoked by uid 500); 3 Jun 2010 20:30:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8636 invoked by uid 99); 3 Jun 2010 20:30:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 20:30:18 +0000
X-ASF-Spam-Status: No, hits=-1495.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 20:30:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53KTwad004114
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 20:29:58 GMT
Message-ID: <32574417.167231275596998259.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 16:29:58 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875284#action_12875284 ] 

Eli Collins commented on HADOOP-6796:
-------------------------------------

Looks reasonable to me.  What do you think Jakob?

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.1.patch, HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8768-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 21:58:21 2010
Return-Path: <common-issues-return-8768-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30018 invoked from network); 3 Jun 2010 21:58:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 21:58:21 -0000
Received: (qmail 41403 invoked by uid 500); 3 Jun 2010 21:58:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41363 invoked by uid 500); 3 Jun 2010 21:58:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41351 invoked by uid 99); 3 Jun 2010 21:58:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 21:58:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 21:58:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53LvupD004966
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 21:57:57 GMT
Message-ID: <32110815.168521275602276857.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 17:57:56 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6808) Document steps to enable
 {File|Ganglia}Context for kerberos metrics
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Document steps to enable {File|Ganglia}Context for kerberos metrics
-------------------------------------------------------------------

                 Key: HADOOP-6808
                 URL: https://issues.apache.org/jira/browse/HADOOP-6808
             Project: Hadoop Common
          Issue Type: Bug
          Components: conf
    Affects Versions: 0.20.2, 0.22.0
            Reporter: Erik Steffl
            Priority: Trivial
             Fix For: 0.20.3, 0.22.0


There was apatch adding kerberos metrics which included the following line in conf/hadoop-metrics.properties:

ugi.class=org.apache.hadoop.metrics.spi.NullContext

However to maintain consistency with dfs,jvm and mapred metrics and for documentation purpose we should include comments in the file to include steps to enable FileContext and GangliaContext as well. The following lines have to be included in conf/hadoop-metrics.properties:

# Configuration of the "ugi" context for file
#ugi.class=org.apache.hadoop.metrics.file.FileContext
#ugi.period=10
#ugi.fileName=/tmp/ugimetrics.log

# Configuration of the "ugi" context for ganglia
# ugi.class=org.apache.hadoop.metrics.ganglia.GangliaContext
# ugi.period=10
# ugi.servers=localhost:8649


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8769-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 22:58:16 2010
Return-Path: <common-issues-return-8769-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46314 invoked from network); 3 Jun 2010 22:58:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 22:58:16 -0000
Received: (qmail 87931 invoked by uid 500); 3 Jun 2010 22:58:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87900 invoked by uid 500); 3 Jun 2010 22:58:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87878 invoked by uid 99); 3 Jun 2010 22:58:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 22:58:16 +0000
X-ASF-Spam-Status: No, hits=-1495.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 22:58:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53MvtCx005466
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 22:57:55 GMT
Message-ID: <13316090.169471275605875092.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 18:57:55 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6693) Add metrics to track kerberos login
 activity
In-Reply-To: <1138587062.22291270763196714.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875339#action_12875339 ] 

Erik Steffl commented on HADOOP-6693:
-------------------------------------

If/when there is a patch for trunk the patch in https://issues.apache.org/jira/browse/HADOOP-6808 should also be included.

> Add metrics to track kerberos login activity
> --------------------------------------------
>
>                 Key: HADOOP-6693
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6693
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6693.1.patch, HADOOP-6693.patch, HADOOP-6693.rel20.1.patch, HADOOP-6693.rel20.patch
>
>
> Need metrics to track kerberos login activity such as login rate and the time taken for login.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8770-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 22:58:16 2010
Return-Path: <common-issues-return-8770-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46352 invoked from network); 3 Jun 2010 22:58:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 22:58:16 -0000
Received: (qmail 88074 invoked by uid 500); 3 Jun 2010 22:58:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88044 invoked by uid 500); 3 Jun 2010 22:58:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88036 invoked by uid 99); 3 Jun 2010 22:58:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 22:58:16 +0000
X-ASF-Spam-Status: No, hits=-1495.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 22:58:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53Mvu3N005478
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 22:57:56 GMT
Message-ID: <14113739.169511275605876015.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 18:57:56 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6808) Document steps to enable
 {File|Ganglia}Context for kerberos metrics
In-Reply-To: <32110815.168521275602276857.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Erik Steffl updated HADOOP-6808:
--------------------------------

    Attachment: HADOOP-6808-0.20.1xx.patch

> Document steps to enable {File|Ganglia}Context for kerberos metrics
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6808
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6808
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.20.2, 0.22.0
>            Reporter: Erik Steffl
>            Priority: Trivial
>             Fix For: 0.20.3, 0.22.0
>
>         Attachments: HADOOP-6808-0.20.1xx.patch
>
>
> There was apatch adding kerberos metrics which included the following line in conf/hadoop-metrics.properties:
> ugi.class=org.apache.hadoop.metrics.spi.NullContext
> However to maintain consistency with dfs,jvm and mapred metrics and for documentation purpose we should include comments in the file to include steps to enable FileContext and GangliaContext as well. The following lines have to be included in conf/hadoop-metrics.properties:
> # Configuration of the "ugi" context for file
> #ugi.class=org.apache.hadoop.metrics.file.FileContext
> #ugi.period=10
> #ugi.fileName=/tmp/ugimetrics.log
> # Configuration of the "ugi" context for ganglia
> # ugi.class=org.apache.hadoop.metrics.ganglia.GangliaContext
> # ugi.period=10
> # ugi.servers=localhost:8649

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8771-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 23:00:15 2010
Return-Path: <common-issues-return-8771-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46863 invoked from network); 3 Jun 2010 23:00:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 23:00:15 -0000
Received: (qmail 89094 invoked by uid 500); 3 Jun 2010 23:00:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89028 invoked by uid 500); 3 Jun 2010 23:00:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89014 invoked by uid 99); 3 Jun 2010 23:00:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 23:00:15 +0000
X-ASF-Spam-Status: No, hits=-1495.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 23:00:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53Mxsmj005488
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 22:59:54 GMT
Message-ID: <4129394.169531275605994214.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 18:59:54 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6808) Document steps to enable
 {File|Ganglia}Context for kerberos metrics
In-Reply-To: <32110815.168521275602276857.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875340#action_12875340 ] 

Erik Steffl commented on HADOOP-6808:
-------------------------------------

This depends on https://issues.apache.org/jira/browse/HADOOP-6693 for which there is no trunk patch, therefore this jira only has 0.20 patch at the moment.

> Document steps to enable {File|Ganglia}Context for kerberos metrics
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6808
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6808
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 0.20.2, 0.22.0
>            Reporter: Erik Steffl
>            Priority: Trivial
>             Fix For: 0.20.3, 0.22.0
>
>         Attachments: HADOOP-6808-0.20.1xx.patch
>
>
> There was apatch adding kerberos metrics which included the following line in conf/hadoop-metrics.properties:
> ugi.class=org.apache.hadoop.metrics.spi.NullContext
> However to maintain consistency with dfs,jvm and mapred metrics and for documentation purpose we should include comments in the file to include steps to enable FileContext and GangliaContext as well. The following lines have to be included in conf/hadoop-metrics.properties:
> # Configuration of the "ugi" context for file
> #ugi.class=org.apache.hadoop.metrics.file.FileContext
> #ugi.period=10
> #ugi.fileName=/tmp/ugimetrics.log
> # Configuration of the "ugi" context for ganglia
> # ugi.class=org.apache.hadoop.metrics.ganglia.GangliaContext
> # ugi.period=10
> # ugi.servers=localhost:8649

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8772-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 23:02:15 2010
Return-Path: <common-issues-return-8772-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 47260 invoked from network); 3 Jun 2010 23:02:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 23:02:15 -0000
Received: (qmail 89918 invoked by uid 500); 3 Jun 2010 23:02:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89888 invoked by uid 500); 3 Jun 2010 23:02:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89875 invoked by uid 99); 3 Jun 2010 23:02:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 23:02:15 +0000
X-ASF-Spam-Status: No, hits=-1495.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 23:02:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53N1sNu005596
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 23:01:54 GMT
Message-ID: <12427185.169681275606114239.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 19:01:54 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6661) User document for
 UserGroupInformation.doAs
In-Reply-To: <927098388.498051269554007283.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875342#action_12875342 ] 

Jakob Homan commented on HADOOP-6661:
-------------------------------------

I'm not wild about the use of Oozie in the example.  Oozie's not yet in the code source and there's no reason documentation readers would be familiar with that project.  Also, the text makes it sound as if this feature is in there for Oozie in particular.  There needs to be more explanation that this is a generic feature for different types of superusers.  Also, this is the only place we use the term "Secure Impersonation," which sounds like a poor word choice.  What about "Superusers submitting jobs on behalf of other users"?

> User document for UserGroupInformation.doAs
> -------------------------------------------
>
>                 Key: HADOOP-6661
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6661
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6661-y20.1.patch, HADOOP-6661-y20.2.patch, HADOOP-6661.1.patch, HADOOP-6661.2.patch
>
>
> A user document should be added for secure impersonation feature. The document should cover a code example about how  UserGroupInformation.doAs should be used for secure impersonation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8773-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 23:44:19 2010
Return-Path: <common-issues-return-8773-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58401 invoked from network); 3 Jun 2010 23:44:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 23:44:18 -0000
Received: (qmail 26734 invoked by uid 500); 3 Jun 2010 23:44:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26703 invoked by uid 500); 3 Jun 2010 23:44:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26694 invoked by uid 99); 3 Jun 2010 23:44:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 23:44:18 +0000
X-ASF-Spam-Status: No, hits=-1495.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 23:44:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53Nhv5f005996
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 23:43:58 GMT
Message-ID: <25027724.170681275608637748.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 19:43:57 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6661) User document for
 UserGroupInformation.doAs
In-Reply-To: <927098388.498051269554007283.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6661:
-----------------------------------------

    Attachment: HADOOP-6661.3.patch

New patch addressing the comment.

> User document for UserGroupInformation.doAs
> -------------------------------------------
>
>                 Key: HADOOP-6661
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6661
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6661-y20.1.patch, HADOOP-6661-y20.2.patch, HADOOP-6661.1.patch, HADOOP-6661.2.patch, HADOOP-6661.3.patch
>
>
> A user document should be added for secure impersonation feature. The document should cover a code example about how  UserGroupInformation.doAs should be used for secure impersonation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8774-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 23:44:19 2010
Return-Path: <common-issues-return-8774-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58440 invoked from network); 3 Jun 2010 23:44:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 23:44:19 -0000
Received: (qmail 26926 invoked by uid 500); 3 Jun 2010 23:44:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26846 invoked by uid 500); 3 Jun 2010 23:44:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26838 invoked by uid 99); 3 Jun 2010 23:44:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 23:44:18 +0000
X-ASF-Spam-Status: No, hits=-1495.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 23:44:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53Nhwpf006002
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 23:43:58 GMT
Message-ID: <19974450.170701275608638438.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 19:43:58 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6661) User document for
 UserGroupInformation.doAs
In-Reply-To: <927098388.498051269554007283.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6661:
-----------------------------------------

    Status: Open  (was: Patch Available)

> User document for UserGroupInformation.doAs
> -------------------------------------------
>
>                 Key: HADOOP-6661
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6661
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6661-y20.1.patch, HADOOP-6661-y20.2.patch, HADOOP-6661.1.patch, HADOOP-6661.2.patch, HADOOP-6661.3.patch
>
>
> A user document should be added for secure impersonation feature. The document should cover a code example about how  UserGroupInformation.doAs should be used for secure impersonation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8775-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 23:44:22 2010
Return-Path: <common-issues-return-8775-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58500 invoked from network); 3 Jun 2010 23:44:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 23:44:22 -0000
Received: (qmail 27036 invoked by uid 500); 3 Jun 2010 23:44:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27004 invoked by uid 500); 3 Jun 2010 23:44:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26996 invoked by uid 99); 3 Jun 2010 23:44:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 23:44:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 23:44:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53Nhxsm006008
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 23:43:59 GMT
Message-ID: <4646473.170721275608639030.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 19:43:59 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6661) User document for
 UserGroupInformation.doAs
In-Reply-To: <927098388.498051269554007283.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6661:
-----------------------------------------

    Status: Patch Available  (was: Open)

> User document for UserGroupInformation.doAs
> -------------------------------------------
>
>                 Key: HADOOP-6661
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6661
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6661-y20.1.patch, HADOOP-6661-y20.2.patch, HADOOP-6661.1.patch, HADOOP-6661.2.patch, HADOOP-6661.3.patch
>
>
> A user document should be added for secure impersonation feature. The document should cover a code example about how  UserGroupInformation.doAs should be used for secure impersonation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8776-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 23:46:25 2010
Return-Path: <common-issues-return-8776-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58762 invoked from network); 3 Jun 2010 23:46:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 23:46:24 -0000
Received: (qmail 28520 invoked by uid 500); 3 Jun 2010 23:46:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28477 invoked by uid 500); 3 Jun 2010 23:46:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28469 invoked by uid 99); 3 Jun 2010 23:46:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 23:46:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 23:46:22 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53Nk1Fv006030
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 23:46:01 GMT
Message-ID: <4063160.170791275608760977.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 19:46:00 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6787:
--------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]  (was: [Incompatible change])
      Resolution: Fixed

I've commited this. Thanks Luke!

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-trunk-v2.patch, hadoop-6787-y20s-v1.patch
>
>
> Refactor the glob pattern code out of FileContext and FileSystem into a package private GlobFilter and the reusable GlobPattern class (InterfaceAudience.Private)
> Also fix the handling of ^ outside character class ([...]) reported in HADOOP-6618 and make the glob pattern code less restrictive (not throwing on some valid glob patterns.) and more POSIX standard compliant (support [!...]).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8777-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 03 23:54:24 2010
Return-Path: <common-issues-return-8777-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59848 invoked from network); 3 Jun 2010 23:54:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 3 Jun 2010 23:54:24 -0000
Received: (qmail 37155 invoked by uid 500); 3 Jun 2010 23:54:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37126 invoked by uid 500); 3 Jun 2010 23:54:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37118 invoked by uid 99); 3 Jun 2010 23:54:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 23:54:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jun 2010 23:54:21 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o53Ns0jK006093
	for <common-issues@hadoop.apache.org>; Thu, 3 Jun 2010 23:54:00 GMT
Message-ID: <28821429.170931275609240095.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 19:54:00 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875369#action_12875369 ] 

Hudson commented on HADOOP-6787:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #276 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/276/])
    HADOOP-6787. Factor out glob pattern code from FileContext and FileSystem. Contributed by Luke Lu.


> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-trunk-v2.patch, hadoop-6787-y20s-v1.patch
>
>
> Refactor the glob pattern code out of FileContext and FileSystem into a package private GlobFilter and the reusable GlobPattern class (InterfaceAudience.Private)
> Also fix the handling of ^ outside character class ([...]) reported in HADOOP-6618 and make the glob pattern code less restrictive (not throwing on some valid glob patterns.) and more POSIX standard compliant (support [!...]).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8778-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 00:02:28 2010
Return-Path: <common-issues-return-8778-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61323 invoked from network); 4 Jun 2010 00:02:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 00:02:24 -0000
Received: (qmail 41606 invoked by uid 500); 4 Jun 2010 00:02:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41370 invoked by uid 500); 4 Jun 2010 00:02:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41354 invoked by uid 99); 4 Jun 2010 00:02:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:02:24 +0000
X-ASF-Spam-Status: No, hits=-1495.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:02:23 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54023KD006214
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 00:02:03 GMT
Message-ID: <20471724.171061275609723382.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 20:02:03 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6794) Move configuration and script files
 post split
In-Reply-To: <11039662.93131275342217664.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6794:
------------------------------

    Attachment: HADOOP-6794.patch

A minor change to the patches which makes HADOOP_COMMON_HOME, HADOOP_HDFS_HOME, and HADOOP_MAPRED_HOME default to HADOOP_HOME if they are not set.

> Move configuration and script files post split
> ----------------------------------------------
>
>                 Key: HADOOP-6794
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6794
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, scripts
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6794.patch, HADOOP-6794.patch, HADOOP-6794.patch, HADOOP-6794.patch, HADOOP-6794.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8779-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 00:04:26 2010
Return-Path: <common-issues-return-8779-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62342 invoked from network); 4 Jun 2010 00:04:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 00:04:24 -0000
Received: (qmail 43890 invoked by uid 500); 4 Jun 2010 00:04:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43840 invoked by uid 500); 4 Jun 2010 00:04:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43832 invoked by uid 99); 4 Jun 2010 00:04:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:04:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:04:21 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54040RS006247
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 00:04:00 GMT
Message-ID: <33549781.171171275609840436.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 20:04:00 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6661) User document for
 UserGroupInformation.doAs
In-Reply-To: <927098388.498051269554007283.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875373#action_12875373 ] 

Jakob Homan commented on HADOOP-6661:
-------------------------------------

This looks much better.  The file name and TOC entry is still for secure impersonation, however.

> User document for UserGroupInformation.doAs
> -------------------------------------------
>
>                 Key: HADOOP-6661
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6661
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6661-y20.1.patch, HADOOP-6661-y20.2.patch, HADOOP-6661.1.patch, HADOOP-6661.2.patch, HADOOP-6661.3.patch
>
>
> A user document should be added for secure impersonation feature. The document should cover a code example about how  UserGroupInformation.doAs should be used for secure impersonation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8780-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 00:08:22 2010
Return-Path: <common-issues-return-8780-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67365 invoked from network); 4 Jun 2010 00:08:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 00:08:21 -0000
Received: (qmail 50462 invoked by uid 500); 4 Jun 2010 00:08:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50340 invoked by uid 500); 4 Jun 2010 00:08:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50332 invoked by uid 99); 4 Jun 2010 00:08:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:08:21 +0000
X-ASF-Spam-Status: No, hits=-1495.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:08:21 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54081jM006318
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 00:08:01 GMT
Message-ID: <10793863.171381275610081160.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 20:08:01 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6809) rpc allow creating arbitrary size of
 objects
In-Reply-To: <4848286.171351275610080522.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6809:
-------------------------------------------

    Component/s: ipc
                     (was: io)

> rpc allow creating arbitrary size of objects
> --------------------------------------------
>
>                 Key: HADOOP-6809
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6809
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>            Reporter: Tsz Wo (Nicholas), SZE
>
> When o.a.h.ipc.Server receives a rpc method call, it reads the parameters by initializing an o.a.h.ipc.RPC.Invocation object, which read the parameter values by calling ObjectWritable.readObject(..).  However, ObjectWritable.readObject(..) does not limit the object size and may create objects with arbitrary size.  As a consequence, any rpc client may create large objects in the server by passing large parameter objects.
> For example, a user application may creates large objects in the namenode by calling DistributedFileSystem.setOwner(p, username, groupname) if username or groupname are large strings.  In such case, it could easily bring down the namenode.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8781-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 00:08:25 2010
Return-Path: <common-issues-return-8781-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67443 invoked from network); 4 Jun 2010 00:08:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 00:08:24 -0000
Received: (qmail 50710 invoked by uid 500); 4 Jun 2010 00:08:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50677 invoked by uid 500); 4 Jun 2010 00:08:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50669 invoked by uid 99); 4 Jun 2010 00:08:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:08:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:08:22 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54080Ul006309
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 00:08:00 GMT
Message-ID: <4848286.171351275610080522.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 20:08:00 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6809) rpc allow creating arbitrary size of
 objects
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

rpc allow creating arbitrary size of objects
--------------------------------------------

                 Key: HADOOP-6809
                 URL: https://issues.apache.org/jira/browse/HADOOP-6809
             Project: Hadoop Common
          Issue Type: Bug
          Components: io
            Reporter: Tsz Wo (Nicholas), SZE


When o.a.h.ipc.Server receives a rpc method call, it reads the parameters by initializing an o.a.h.ipc.RPC.Invocation object, which read the parameter values by calling ObjectWritable.readObject(..).  However, ObjectWritable.readObject(..) does not limit the object size and may create objects with arbitrary size.  As a consequence, any rpc client may create large objects in the server by passing large parameter objects.

For example, a user application may creates large objects in the namenode by calling DistributedFileSystem.setOwner(p, username, groupname) if username or groupname are large strings.  In such case, it could easily bring down the namenode.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8782-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 00:10:21 2010
Return-Path: <common-issues-return-8782-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67921 invoked from network); 4 Jun 2010 00:10:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 00:10:20 -0000
Received: (qmail 53266 invoked by uid 500); 4 Jun 2010 00:10:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53220 invoked by uid 500); 4 Jun 2010 00:10:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53212 invoked by uid 99); 4 Jun 2010 00:10:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:10:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:10:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5409uLO006328
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 00:09:56 GMT
Message-ID: <21232329.171401275610196438.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 20:09:56 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6661) User document for
 UserGroupInformation.doAs
In-Reply-To: <927098388.498051269554007283.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875377#action_12875377 ] 

Hadoop QA commented on HADOOP-6661:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446301/HADOOP-6661.3.patch
  against trunk revision 951207.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/561/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/561/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/561/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/561/console

This message is automatically generated.

> User document for UserGroupInformation.doAs
> -------------------------------------------
>
>                 Key: HADOOP-6661
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6661
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6661-y20.1.patch, HADOOP-6661-y20.2.patch, HADOOP-6661.1.patch, HADOOP-6661.2.patch, HADOOP-6661.3.patch
>
>
> A user document should be added for secure impersonation feature. The document should cover a code example about how  UserGroupInformation.doAs should be used for secure impersonation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8783-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 00:16:21 2010
Return-Path: <common-issues-return-8783-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68990 invoked from network); 4 Jun 2010 00:16:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 00:16:20 -0000
Received: (qmail 57029 invoked by uid 500); 4 Jun 2010 00:16:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56992 invoked by uid 500); 4 Jun 2010 00:16:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56983 invoked by uid 99); 4 Jun 2010 00:16:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:16:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:16:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o540Fu7W006367
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 00:15:57 GMT
Message-ID: <16113546.171501275610556931.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 20:15:56 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6762) exception while doing RPC I/O closes
 channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

sam rash updated HADOOP-6762:
-----------------------------

    Attachment: hadoop-6762-5.txt

improved test case: runs 200 threads doing rpc via the same connection.  shows that interrupting one can cause errors in the others.

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-5.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8784-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 00:18:19 2010
Return-Path: <common-issues-return-8784-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69425 invoked from network); 4 Jun 2010 00:18:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 00:18:19 -0000
Received: (qmail 58598 invoked by uid 500); 4 Jun 2010 00:18:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58565 invoked by uid 500); 4 Jun 2010 00:18:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58557 invoked by uid 99); 4 Jun 2010 00:18:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:18:18 +0000
X-ASF-Spam-Status: No, hits=-1495.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:18:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o540Hwph006393
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 00:17:58 GMT
Message-ID: <12834529.171561275610678462.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 20:17:58 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6762) exception while doing RPC I/O closes
 channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

sam rash updated HADOOP-6762:
-----------------------------

    Attachment: hadoop-6762-6.txt

had unnecessary Runnable cast

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8785-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 00:18:22 2010
Return-Path: <common-issues-return-8785-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69473 invoked from network); 4 Jun 2010 00:18:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 00:18:22 -0000
Received: (qmail 58744 invoked by uid 500); 4 Jun 2010 00:18:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58710 invoked by uid 500); 4 Jun 2010 00:18:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58702 invoked by uid 99); 4 Jun 2010 00:18:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:18:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:18:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o540HwpD006405
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 00:17:58 GMT
Message-ID: <8768122.171601275610678922.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 20:17:58 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6762) exception while doing RPC I/O closes
 channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

sam rash updated HADOOP-6762:
-----------------------------

    Attachment:     (was: hadoop-6762-5.txt)

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8786-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 00:20:19 2010
Return-Path: <common-issues-return-8786-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69734 invoked from network); 4 Jun 2010 00:20:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 00:20:19 -0000
Received: (qmail 60510 invoked by uid 500); 4 Jun 2010 00:20:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60470 invoked by uid 500); 4 Jun 2010 00:20:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60415 invoked by uid 99); 4 Jun 2010 00:20:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:20:15 +0000
X-ASF-Spam-Status: No, hits=-1496.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:20:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o540Jtmr006426
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 00:19:55 GMT
Message-ID: <22195365.171641275610795188.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 20:19:55 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875383#action_12875383 ] 

sam rash commented on HADOOP-6762:
----------------------------------

todd : you can try this.interrupt() now.  please let me know if it passes the new test case

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8787-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 00:22:16 2010
Return-Path: <common-issues-return-8787-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70011 invoked from network); 4 Jun 2010 00:22:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 00:22:16 -0000
Received: (qmail 63321 invoked by uid 500); 4 Jun 2010 00:22:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63273 invoked by uid 500); 4 Jun 2010 00:22:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63257 invoked by uid 99); 4 Jun 2010 00:22:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:22:16 +0000
X-ASF-Spam-Status: No, hits=-1496.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:22:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o540Lthi006452
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 00:21:55 GMT
Message-ID: <26104741.171691275610915125.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 20:21:55 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875384#action_12875384 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6787:
------------------------------------------------

@Eli, do you think that this is an incompatible change?

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-trunk-v2.patch, hadoop-6787-y20s-v1.patch
>
>
> Refactor the glob pattern code out of FileContext and FileSystem into a package private GlobFilter and the reusable GlobPattern class (InterfaceAudience.Private)
> Also fix the handling of ^ outside character class ([...]) reported in HADOOP-6618 and make the glob pattern code less restrictive (not throwing on some valid glob patterns.) and more POSIX standard compliant (support [!...]).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8788-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 00:26:18 2010
Return-Path: <common-issues-return-8788-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76768 invoked from network); 4 Jun 2010 00:26:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 00:26:17 -0000
Received: (qmail 66353 invoked by uid 500); 4 Jun 2010 00:26:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66327 invoked by uid 500); 4 Jun 2010 00:26:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66317 invoked by uid 99); 4 Jun 2010 00:26:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:26:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:26:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o540PrjE006484
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 00:25:53 GMT
Message-ID: <6105531.171751275611153576.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 20:25:53 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6661) User document for
 UserGroupInformation.doAs
In-Reply-To: <927098388.498051269554007283.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6661:
-----------------------------------------

    Status: Open  (was: Patch Available)

> User document for UserGroupInformation.doAs
> -------------------------------------------
>
>                 Key: HADOOP-6661
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6661
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6661-y20.1.patch, HADOOP-6661-y20.2.patch, HADOOP-6661.1.patch, HADOOP-6661.2.patch, HADOOP-6661.3.patch, HADOOP-6661.4.patch
>
>
> A user document should be added for secure impersonation feature. The document should cover a code example about how  UserGroupInformation.doAs should be used for secure impersonation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8789-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 00:26:18 2010
Return-Path: <common-issues-return-8789-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76789 invoked from network); 4 Jun 2010 00:26:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 00:26:17 -0000
Received: (qmail 66504 invoked by uid 500); 4 Jun 2010 00:26:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66470 invoked by uid 500); 4 Jun 2010 00:26:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66418 invoked by uid 99); 4 Jun 2010 00:26:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:26:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:26:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o540PsSL006490
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 00:25:54 GMT
Message-ID: <29117515.171771275611154087.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 20:25:54 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6661) User document for
 UserGroupInformation.doAs
In-Reply-To: <927098388.498051269554007283.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6661:
-----------------------------------------

    Attachment: HADOOP-6661.4.patch

New patch addressing comments. 
ant docs is tested.

> User document for UserGroupInformation.doAs
> -------------------------------------------
>
>                 Key: HADOOP-6661
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6661
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6661-y20.1.patch, HADOOP-6661-y20.2.patch, HADOOP-6661.1.patch, HADOOP-6661.2.patch, HADOOP-6661.3.patch, HADOOP-6661.4.patch
>
>
> A user document should be added for secure impersonation feature. The document should cover a code example about how  UserGroupInformation.doAs should be used for secure impersonation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8790-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 00:28:18 2010
Return-Path: <common-issues-return-8790-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77054 invoked from network); 4 Jun 2010 00:28:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 00:28:18 -0000
Received: (qmail 67401 invoked by uid 500); 4 Jun 2010 00:28:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67369 invoked by uid 500); 4 Jun 2010 00:28:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67361 invoked by uid 99); 4 Jun 2010 00:28:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:28:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:28:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o540Rs1T006512
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 00:27:54 GMT
Message-ID: <14272158.171821275611274622.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 20:27:54 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6661) User document for
 UserGroupInformation.doAs
In-Reply-To: <927098388.498051269554007283.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6661:
--------------------------------

    Hadoop Flags: [Reviewed]

+1


> User document for UserGroupInformation.doAs
> -------------------------------------------
>
>                 Key: HADOOP-6661
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6661
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6661-y20.1.patch, HADOOP-6661-y20.2.patch, HADOOP-6661.1.patch, HADOOP-6661.2.patch, HADOOP-6661.3.patch, HADOOP-6661.4.patch
>
>
> A user document should be added for secure impersonation feature. The document should cover a code example about how  UserGroupInformation.doAs should be used for secure impersonation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8791-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 00:32:18 2010
Return-Path: <common-issues-return-8791-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78090 invoked from network); 4 Jun 2010 00:32:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 00:32:18 -0000
Received: (qmail 68871 invoked by uid 500); 4 Jun 2010 00:32:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68830 invoked by uid 500); 4 Jun 2010 00:32:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68822 invoked by uid 99); 4 Jun 2010 00:32:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:32:18 +0000
X-ASF-Spam-Status: No, hits=-1496.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:32:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o540VvUq006563
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 00:31:57 GMT
Message-ID: <3003190.171961275611517593.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 20:31:57 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6787:
--------------------------------

    Hadoop Flags: [Incompatible change, Reviewed]  (was: [Reviewed])

Refactoring out the internal classes doesn't seem incompatible.  I suppose the bug fixes could affect a program eg that expected a bogus exception on a valid POSIX glob pattern, so I'll put the flag back.

> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-trunk-v2.patch, hadoop-6787-y20s-v1.patch
>
>
> Refactor the glob pattern code out of FileContext and FileSystem into a package private GlobFilter and the reusable GlobPattern class (InterfaceAudience.Private)
> Also fix the handling of ^ outside character class ([...]) reported in HADOOP-6618 and make the glob pattern code less restrictive (not throwing on some valid glob patterns.) and more POSIX standard compliant (support [!...]).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8792-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 00:42:17 2010
Return-Path: <common-issues-return-8792-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78994 invoked from network); 4 Jun 2010 00:42:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 00:42:17 -0000
Received: (qmail 77271 invoked by uid 500); 4 Jun 2010 00:42:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77233 invoked by uid 500); 4 Jun 2010 00:42:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77225 invoked by uid 99); 4 Jun 2010 00:42:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:42:16 +0000
X-ASF-Spam-Status: No, hits=-1496.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:42:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o540fuXQ006600
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 00:41:56 GMT
Message-ID: <5072363.172011275612116341.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 20:41:56 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6661) User document for
 UserGroupInformation.doAs
In-Reply-To: <927098388.498051269554007283.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan resolved HADOOP-6661.
---------------------------------

    Resolution: Fixed

I've committed this.  Thanks, Jitendra!  Resolving as fixed.

> User document for UserGroupInformation.doAs
> -------------------------------------------
>
>                 Key: HADOOP-6661
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6661
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6661-y20.1.patch, HADOOP-6661-y20.2.patch, HADOOP-6661.1.patch, HADOOP-6661.2.patch, HADOOP-6661.3.patch, HADOOP-6661.4.patch
>
>
> A user document should be added for secure impersonation feature. The document should cover a code example about how  UserGroupInformation.doAs should be used for secure impersonation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8793-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 00:54:30 2010
Return-Path: <common-issues-return-8793-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80465 invoked from network); 4 Jun 2010 00:54:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 00:54:30 -0000
Received: (qmail 87531 invoked by uid 500); 4 Jun 2010 00:54:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87490 invoked by uid 500); 4 Jun 2010 00:54:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87481 invoked by uid 99); 4 Jun 2010 00:54:30 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:54:30 +0000
X-ASF-Spam-Status: No, hits=-1496.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 00:54:30 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o540s9sm006638
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 00:54:10 GMT
Message-ID: <29372619.172061275612849595.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 20:54:09 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6661) User document for
 UserGroupInformation.doAs
In-Reply-To: <927098388.498051269554007283.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875396#action_12875396 ] 

Hudson commented on HADOOP-6661:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #277 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/277/])
    HADOOP-6661. User document for UserGroupInformation.doAs. Contributed by Jitendra Pandey.


> User document for UserGroupInformation.doAs
> -------------------------------------------
>
>                 Key: HADOOP-6661
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6661
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6661-y20.1.patch, HADOOP-6661-y20.2.patch, HADOOP-6661.1.patch, HADOOP-6661.2.patch, HADOOP-6661.3.patch, HADOOP-6661.4.patch
>
>
> A user document should be added for secure impersonation feature. The document should cover a code example about how  UserGroupInformation.doAs should be used for secure impersonation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8794-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 01:18:21 2010
Return-Path: <common-issues-return-8794-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88462 invoked from network); 4 Jun 2010 01:18:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 01:18:21 -0000
Received: (qmail 1216 invoked by uid 500); 4 Jun 2010 01:18:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1185 invoked by uid 500); 4 Jun 2010 01:18:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1177 invoked by uid 99); 4 Jun 2010 01:18:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 01:18:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 01:18:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o541Hvjw006839
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 01:17:57 GMT
Message-ID: <22928448.172391275614277500.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 21:17:57 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875406#action_12875406 ] 

Todd Lipcon commented on HADOOP-6762:
-------------------------------------

Hey Sam. Sorry, I misunderstood your point earlier. You're definitely right that interrupting one thread shouldn't take down the RPC connection.

Adding yet another thread to IPC seems a bit complicated, though. What about if we added a flag to Connection saying "no more sends on this connection", so that interrupting the sender did kill the connection but lets currently pending calls complete? Then when the queue has been quiesced the connection shuts down like it does today?

The issue I'm thinking is that we solve the interrupt problem, but don't solve the general case of exceptions during sendparam. The user can still have a Writable which eg throws an NPE, and we're back to the same problem of lack of isolation between writers.

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8795-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 01:28:15 2010
Return-Path: <common-issues-return-8795-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92935 invoked from network); 4 Jun 2010 01:28:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 01:28:15 -0000
Received: (qmail 9538 invoked by uid 500); 4 Jun 2010 01:28:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9502 invoked by uid 500); 4 Jun 2010 01:28:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9494 invoked by uid 99); 4 Jun 2010 01:28:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 01:28:15 +0000
X-ASF-Spam-Status: No, hits=-1496.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 01:28:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o541RsGV006923
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 01:27:54 GMT
Message-ID: <10378660.172601275614874129.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 21:27:54 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875411#action_12875411 ] 

sam rash commented on HADOOP-6762:
----------------------------------

Hmm, how would pending calls complete?  They already have a Connection object with a socket channel that is in bad shape.  basically there would have to be a check inside a sync block that the channel is valid before sending.  If it's not, it would have to create a new socket (or whole Connection, again all in sync block).  Does this make sense?  A bunch of threads get the Connection object and pile up on the synchronized(this.out) and if one of them is interrupted, the whole pile will get errors.  I think having the test & fix code is more complicated than using another thread actually, but I may be biased (having already done it the other way)

FWIW, we're already using this on our 0.20 branch in production where we have up to 200+ threads using the same RPC instance.

also, i don't actually think the code is complex--it's using an executor so the thread management is as simple as it can get. 
We can even get rid of the latch--it's not necessary, but I wanted the change to function exactly as it does now, so I put it in.




> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8796-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 01:30:16 2010
Return-Path: <common-issues-return-8796-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93314 invoked from network); 4 Jun 2010 01:30:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 01:30:16 -0000
Received: (qmail 10443 invoked by uid 500); 4 Jun 2010 01:30:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10326 invoked by uid 500); 4 Jun 2010 01:30:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10317 invoked by uid 99); 4 Jun 2010 01:30:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 01:30:16 +0000
X-ASF-Spam-Status: No, hits=-1496.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 01:30:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o541TtW1006945
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 01:29:55 GMT
Message-ID: <28121166.172671275614995373.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 21:29:55 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875413#action_12875413 ] 

Todd Lipcon commented on HADOOP-6762:
-------------------------------------

Hey Sam, that's a fair argument. Let me look over your patch more closely. Do you have this against 0.20 as well? The test case I have that caused me to notice this is an 0.20 based cluster test.

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8797-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 01:40:16 2010
Return-Path: <common-issues-return-8797-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93902 invoked from network); 4 Jun 2010 01:40:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 01:40:16 -0000
Received: (qmail 12125 invoked by uid 500); 4 Jun 2010 01:40:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12084 invoked by uid 500); 4 Jun 2010 01:40:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12076 invoked by uid 99); 4 Jun 2010 01:40:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 01:40:16 +0000
X-ASF-Spam-Status: No, hits=-1496.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 01:40:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o541dtC8006984
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 01:39:55 GMT
Message-ID: <23562042.172731275615595111.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 21:39:55 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875415#action_12875415 ] 

Todd Lipcon commented on HADOOP-6762:
-------------------------------------

oh, i see now that this essentially applies to 20 without changes. Will try it out and let you know.

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8798-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 01:40:20 2010
Return-Path: <common-issues-return-8798-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93949 invoked from network); 4 Jun 2010 01:40:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 01:40:19 -0000
Received: (qmail 12270 invoked by uid 500); 4 Jun 2010 01:40:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12236 invoked by uid 500); 4 Jun 2010 01:40:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12228 invoked by uid 99); 4 Jun 2010 01:40:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 01:40:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 01:40:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o541duYH006996
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 01:39:56 GMT
Message-ID: <25195592.172771275615596071.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 21:39:56 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875416#action_12875416 ] 

sam rash commented on HADOOP-6762:
----------------------------------

This patch should largely work against 0.20.  I originally patched our 0.20 and then ported to trunk (which was mostly re-writing the test case)

if this doesn't apply, let me know and i'll back-port this (including test case) to 0.20

also, I do agree on one point about the extra thread:  while I don't see it as inherently complex, I do see it as an additional resource.  if we could solve it w/o the thread and similar in code complexity, that would be the superior solution imo


> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8799-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 01:54:17 2010
Return-Path: <common-issues-return-8799-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96534 invoked from network); 4 Jun 2010 01:54:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 01:54:17 -0000
Received: (qmail 18881 invoked by uid 500); 4 Jun 2010 01:54:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18808 invoked by uid 500); 4 Jun 2010 01:54:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18800 invoked by uid 99); 4 Jun 2010 01:54:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 01:54:17 +0000
X-ASF-Spam-Status: No, hits=-1496.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 01:54:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o541rtw1007092
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 01:53:56 GMT
Message-ID: <27538911.172991275616435962.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 21:53:55 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875421#action_12875421 ] 

Todd Lipcon commented on HADOOP-6762:
-------------------------------------

Looking at the code:
- Instead of using the CountdownLatch, can you change the Runnable to a Callable<Void>() and then get back a Future<Void>? Seems a little cleaner.
- The behavior is different since we've added a timeout waiting to sendParam. Do you think this change is necessary? (under what case would we block forever waiting to write?)
- Regarding the issue I raised above with a Writable param that throws NPE, can we move the actual buffer construction back into the calling thread? Then if it throws an RTE, the user will see it (rather than it getting lost somewhere). There's still an issue that this will leave the call on the connection queue, but that's probably worth a separate jira.

Regarding the resource usage issue: can we just use a static  cachedthreadpool that's shared across all of RPC for sending params? In the common case it would only have 0 or 1 threads but it could grow as necessary and then shrink back when idle.

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8800-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 02:06:16 2010
Return-Path: <common-issues-return-8800-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2401 invoked from network); 4 Jun 2010 02:06:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 02:06:16 -0000
Received: (qmail 23480 invoked by uid 500); 4 Jun 2010 02:06:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23444 invoked by uid 500); 4 Jun 2010 02:06:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23435 invoked by uid 99); 4 Jun 2010 02:06:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 02:06:16 +0000
X-ASF-Spam-Status: No, hits=-1496.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 02:06:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5425tbq007177
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 02:05:55 GMT
Message-ID: <25731739.173081275617155305.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 22:05:55 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6768) RPC client can response more
 efficiently when sendParam() got IOException
In-Reply-To: <2932795.79761274091523409.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875423#action_12875423 ] 

Todd Lipcon commented on HADOOP-6768:
-------------------------------------

Hi Xiao,

For your test case, I think it would be much simpler to just use a Writable which throws IOE, rather than mocking out all of Socket, etc. If you look at TestIPC in trunk, there was a case added by HADOOP-6723 that fixed a similar bug for exceptions in readFields, but we didn't test writeFields. It should be easy to extend that test case and share the code there.

Also please take a look at HADOOP-6762. which solves the specific problem of InterruptedException but not the more general problem of IOE or RTE. Would appreciate your thoughts on that issue.

> RPC client can response more efficiently when sendParam() got IOException
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6768
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6768
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Xiao Kang
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6768.patch
>
>
> Under current RPC client implementation, when Client.Connection.sendParam() encounters IOException it just marks the exception and wait receiveResponse() thread to handle the exception. However, receiveResponse() may block ipc.ping.interval on socket read(). 
> This means that RPC client may wait ipc.ping.interval(typically 1 miniute) when exception in sendParam().
> It maybe not so reasonable in some situation. For example using "hadoop fs -put " a small file, DFSClient.close() will interupt renewLease() thread and it may wait 1 minute.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8801-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 02:08:19 2010
Return-Path: <common-issues-return-8801-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2539 invoked from network); 4 Jun 2010 02:08:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 02:08:19 -0000
Received: (qmail 25641 invoked by uid 500); 4 Jun 2010 02:08:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25606 invoked by uid 500); 4 Jun 2010 02:08:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25598 invoked by uid 99); 4 Jun 2010 02:08:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 02:08:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 02:08:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5427tqt007195
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 02:07:55 GMT
Message-ID: <28246099.173121275617275121.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 22:07:55 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875424#action_12875424 ] 

sam rash commented on HADOOP-6762:
----------------------------------

thanks for reviewing the patch so quickly.  

1. latch : I agree.  I'll change it to use a future & get (with timeout, see #2).

2. actually the timeout helped me debug a switch issue today.  I would see a flurry of timeouts when I saturated a switch that wasn't performing to spec.  getting the timeouts was far preferable than hanging indefinitely.  i agree it changes the behavior, but it's the same we do pings at I think.  Also, playing a game of adversary, *if* somehow the connection thread in the executor did die, the latch would hang indefinitely.

perhaps the timeout value should be something else? I choose the timeout value that was used to connect the socket (pingInterval) as it seemed appropriate.

3. ah, i misread your comment above--that's a great idea.  Only the actual push out the socket needs to be in critical section (in theory could improve perf a tiny bit).

I'll get to the changes and post another patch.

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8802-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 02:12:21 2010
Return-Path: <common-issues-return-8802-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2900 invoked from network); 4 Jun 2010 02:12:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 02:12:21 -0000
Received: (qmail 26622 invoked by uid 500); 4 Jun 2010 02:12:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26572 invoked by uid 500); 4 Jun 2010 02:12:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26564 invoked by uid 99); 4 Jun 2010 02:12:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 02:12:21 +0000
X-ASF-Spam-Status: No, hits=-1496.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 02:12:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o542C0Fw007223
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 02:12:00 GMT
Message-ID: <33352008.173181275617520004.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 22:12:00 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875426#action_12875426 ] 

Todd Lipcon commented on HADOOP-6762:
-------------------------------------

re timeout: I'm a little nervous about such a change in the semantics of IPC at this point. The ping system ensures that the other side isn't completely dead, so some people use IPCs that are *supposed* to take a really long time, and rely on ping to know it's at least still connected. Maybe if you find it useful you could introduce a new parameter for the IPC timeout, and have it default to 0 (no timeout?)

I could also see a situation where we wait for the ping time, and then print a LOG.warn("IPC call Protocol.callName to <IP> still waiting after 60000ms") once every ping interval. This would help debugging without changing behavior. (I too have often wished for such a thing)

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8803-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 02:16:20 2010
Return-Path: <common-issues-return-8803-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3365 invoked from network); 4 Jun 2010 02:16:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 02:16:20 -0000
Received: (qmail 27561 invoked by uid 500); 4 Jun 2010 02:16:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27521 invoked by uid 500); 4 Jun 2010 02:16:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27513 invoked by uid 99); 4 Jun 2010 02:16:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 02:16:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 02:16:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o542FtZR007244
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 02:15:56 GMT
Message-ID: <3807177.173221275617755948.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 22:15:55 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875427#action_12875427 ] 

Todd Lipcon commented on HADOOP-6762:
-------------------------------------

Hey Sam,

My cluster test case just spat out a nice warning and then deadlocked my DFS client:

10/06/03 18:48:54 WARN hdfs.DFSClient: Error Recovery for block blk_832465809601113490_3336818 in pipeline 192.168.42.40:11072, 192.168.42.41:11072, 192.168.42.43:11072: bad datanode 192.168.42.41:11072
Exception in thread "DataStreamer for file /user/todd/test-sync block blk_832465809601113490_3336818" java.lang.reflect.UndeclaredThrowableException
        at $Proxy1.getProtocolVersion(Unknown Source)
        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:346)
        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:383)
        at org.apache.hadoop.hdfs.DFSClient.createClientDatanodeProtocolProxy(DFSClient.java:146)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2627)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2139)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2306)
Caused by: java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1238)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:253)
        at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:526)
        at org.apache.hadoop.ipc.Client.call(Client.java:765)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
        ... 8 more

I think we need to add another catch (InterruptedException) and rethrow as IOException around the sendParam call in Client.call

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8804-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 02:16:20 2010
Return-Path: <common-issues-return-8804-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3414 invoked from network); 4 Jun 2010 02:16:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 02:16:20 -0000
Received: (qmail 27691 invoked by uid 500); 4 Jun 2010 02:16:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27664 invoked by uid 500); 4 Jun 2010 02:16:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27656 invoked by uid 99); 4 Jun 2010 02:16:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 02:16:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 02:16:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o542Fufq007256
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 02:15:56 GMT
Message-ID: <31635851.173261275617756538.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 22:15:56 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875428#action_12875428 ] 

sam rash commented on HADOOP-6762:
----------------------------------

forgot to comment on static cachedThreadPool:

This concerns me as if we don't bound the size of this, we could get a massive # of threads;  if we do bound it, then it seems to me we might have some form of deadlock possible where one RPC depends on another (indirectly) and due to limited threads, it can't complete.  Basically we would want at least one thread per Connection, but no more (which is what have now)

We have seem one case of distributed deadlock here on the IPC workers in the DN, so this isn't 100% theory

While it is an extra resource, for simplicity and safety, I *do* prefer one thread Connection.  

What do you think?



> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8805-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 02:18:15 2010
Return-Path: <common-issues-return-8805-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3705 invoked from network); 4 Jun 2010 02:18:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 02:18:15 -0000
Received: (qmail 29833 invoked by uid 500); 4 Jun 2010 02:18:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29773 invoked by uid 500); 4 Jun 2010 02:18:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29765 invoked by uid 99); 4 Jun 2010 02:18:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 02:18:15 +0000
X-ASF-Spam-Status: No, hits=-1496.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 02:18:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o542HsUW007274
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 02:17:54 GMT
Message-ID: <3941203.173301275617874415.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 22:17:54 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875429#action_12875429 ] 

sam rash commented on HADOOP-6762:
----------------------------------

yea, we do need to catch the interruptedException--our 0.20 version had that.  i'll also add that one

thx


> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8806-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 02:20:15 2010
Return-Path: <common-issues-return-8806-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3871 invoked from network); 4 Jun 2010 02:20:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 02:20:15 -0000
Received: (qmail 31345 invoked by uid 500); 4 Jun 2010 02:20:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31313 invoked by uid 500); 4 Jun 2010 02:20:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31305 invoked by uid 99); 4 Jun 2010 02:20:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 02:20:15 +0000
X-ASF-Spam-Status: No, hits=-1496.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 02:20:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o542JsFN007288
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 02:19:55 GMT
Message-ID: <12498212.173341275617994816.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 22:19:54 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875430#action_12875430 ] 

sam rash commented on HADOOP-6762:
----------------------------------

re: timeout, so if a server disappeared, the ping would fail and the RPC would fail that way?  if that's the case, then I think removing the timeout on the Future.get() is fine. 

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8807-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 02:26:21 2010
Return-Path: <common-issues-return-8807-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4481 invoked from network); 4 Jun 2010 02:26:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 02:26:21 -0000
Received: (qmail 35472 invoked by uid 500); 4 Jun 2010 02:26:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35445 invoked by uid 500); 4 Jun 2010 02:26:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35437 invoked by uid 99); 4 Jun 2010 02:26:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 02:26:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 02:26:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o542Pvn6007325
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 02:25:57 GMT
Message-ID: <18442549.173381275618357067.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 22:25:57 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875431#action_12875431 ] 

Todd Lipcon commented on HADOOP-6762:
-------------------------------------

bq. re: timeout, so if a server disappeared, the ping would fail and the RPC would fail that way? if that's the case, then I think removing the timeout on the Future.get() is fine.

Yep, that should be the case. Of course a server can stay up but be unresponsive (eg deadlocked). In those cases, while it's annoying that clients get blocked forever, I don't know that changing the behavior to be timeout based would be a change we could really make at this point without worrying that it would break lots and lots of downstream users :(

bq. We have seem one case of distributed deadlock here on the IPC workers in the DN, so this isn't 100% theory

Yep, I've seen internode deadlocks several times as well. Not pretty! However, I can't think of a situation where this could happen here -- the only thing that can block one of these sendParam calls is TCP backpressure on the socket, and that only happens when the network is stalled. I don't see a case where allowing other threads to start sending would have unstalled a prior sender.

We could actually enforce the max one thread per connection thing by synchronizing on Connection.this.out *outside* the submission of the runnable. That way we know there's only one sending going on at a time, and we're just using the thread exactly for avoiding interruption and nothing else.

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8808-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 02:34:18 2010
Return-Path: <common-issues-return-8808-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5289 invoked from network); 4 Jun 2010 02:34:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 02:34:18 -0000
Received: (qmail 38864 invoked by uid 500); 4 Jun 2010 02:34:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38837 invoked by uid 500); 4 Jun 2010 02:34:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38827 invoked by uid 99); 4 Jun 2010 02:34:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 02:34:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 02:34:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o542Xrwm007381
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 02:33:53 GMT
Message-ID: <15925425.173491275618833656.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 22:33:53 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875435#action_12875435 ] 

sam rash commented on HADOOP-6762:
----------------------------------

hmm, actually with a Future, if there is a runtime exception, it'll show up as an ExecutionException on future.get(), so there isn't a need to move the buffer construction outside (requires another try/catch clause due to IOException).   We can use getCause() on the ExecutionException to find the underlying exception and use markClosed() if it's an IOException, and re-throw if it's a runtime exception (same behavior as now)

or we can move it into the caller thread.  Using the future requires handling the ExecutionException anyway, so I sort of lean this way as it kills two birds. i don't know how much more parallelism we gain moving the buffer construction outside the sync block.

what do you think?




> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8809-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 02:36:16 2010
Return-Path: <common-issues-return-8809-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5427 invoked from network); 4 Jun 2010 02:36:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 02:36:16 -0000
Received: (qmail 39230 invoked by uid 500); 4 Jun 2010 02:36:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39191 invoked by uid 500); 4 Jun 2010 02:36:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39183 invoked by uid 99); 4 Jun 2010 02:36:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 02:36:15 +0000
X-ASF-Spam-Status: No, hits=-1496.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 02:36:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o542Zsxv007391
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 02:35:54 GMT
Message-ID: <15060049.173521275618954482.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 22:35:54 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6809) RPC allows clients to create objects
 with arbitrary size on the server
In-Reply-To: <4848286.171351275610080522.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6809:
-------------------------------------------

    Summary: RPC allows clients to create objects with arbitrary size on the server  (was: rpc allow creating arbitrary size of objects)

> RPC allows clients to create objects with arbitrary size on the server
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-6809
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6809
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>            Reporter: Tsz Wo (Nicholas), SZE
>
> When o.a.h.ipc.Server receives a rpc method call, it reads the parameters by initializing an o.a.h.ipc.RPC.Invocation object, which read the parameter values by calling ObjectWritable.readObject(..).  However, ObjectWritable.readObject(..) does not limit the object size and may create objects with arbitrary size.  As a consequence, any rpc client may create large objects in the server by passing large parameter objects.
> For example, a user application may creates large objects in the namenode by calling DistributedFileSystem.setOwner(p, username, groupname) if username or groupname are large strings.  In such case, it could easily bring down the namenode.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8810-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 02:40:16 2010
Return-Path: <common-issues-return-8810-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5703 invoked from network); 4 Jun 2010 02:40:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 02:40:15 -0000
Received: (qmail 40026 invoked by uid 500); 4 Jun 2010 02:40:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39995 invoked by uid 500); 4 Jun 2010 02:40:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39987 invoked by uid 99); 4 Jun 2010 02:40:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 02:40:15 +0000
X-ASF-Spam-Status: No, hits=-1496.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 02:40:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o542dsNG007425
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 02:39:54 GMT
Message-ID: <24586831.173591275619194744.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 22:39:54 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875437#action_12875437 ] 

Todd Lipcon commented on HADOOP-6762:
-------------------------------------

Either way seems cool. I've never seen call param serialization be a bottleneck where parallelization would really help. Whichever one makes the code cleaner :)

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8811-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 03:58:21 2010
Return-Path: <common-issues-return-8811-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19512 invoked from network); 4 Jun 2010 03:58:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 03:58:20 -0000
Received: (qmail 75564 invoked by uid 500); 4 Jun 2010 03:58:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75327 invoked by uid 500); 4 Jun 2010 03:58:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75307 invoked by uid 99); 4 Jun 2010 03:58:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 03:58:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 03:58:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o543vrAV007899
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 03:57:53 GMT
Message-ID: <10298918.174331275623873711.JavaMail.jira@thor>
Date: Thu, 3 Jun 2010 23:57:53 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6788) [Herriot] Exception exclusion
 functionality is not working correctly.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875457#action_12875457 ] 

Vinod K V commented on HADOOP-6788:
-----------------------------------

bq. Well, I'm not sure about this. Herriot never was approved/discussed by the community as an official name of the framework. Let's think about this first.
Sure Kos. My main concern is that the titles for most of these issues are fairly generic and on a surface look aren't very reflective of what they are really about, and which component.

So, yes, I'll stop messing with the titles till the name issue is resolved.

> [Herriot] Exception exclusion functionality is not working correctly.
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6788
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6788
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Balaji Rajagopalan
>         Attachments: 6788-ydist-security.patch, 6788-ydist-security.patch, HADOOP-6788.patch, HADOOP-6788.patch
>
>
> Exception exclusion functionality is not working correctly because of that tests are failing by not matching the error count.
> I debugged the issue and found that the problem with shell command which is generating in the getNumberOfMatchesInLogFile function.
> Currently building the shell command in the following way. 
> if(list != null){
>   for(int i =0; i < list.length; ++i)
>   {
>     filePattern.append(" | grep -v " + list[i] );
>   }
> }
>     String[] cmd =
>         new String[] {
>             "bash",
>             "-c",
>             "grep -c "
>                 + pattern + " " + filePattern
>                 + " | awk -F: '{s+=$2} END {print s}'" };    
> However, The above commnad won't work correctly because you are counting the exceptions in the file before excluding the known exceptions.
> In this case it gives the mismatch error counts everytime.The shell command should be in the following way to work correctly.
> if (list != null) {
>   int index = 0;
>   for (String excludeExp : list) {
>     filePattern.append((++index < list.length)? "| grep -v " : 
>             "| grep -vc " + list[i] );  
>   }
> }
> String[] cmd =
>    new String[] {
>        "bash",
>        "-c",
>        "grep "
>            + pattern + " " + filePattern
>            + " | awk -F: '{s+=$2} END {print s}'" };  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8812-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 04:08:20 2010
Return-Path: <common-issues-return-8812-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24639 invoked from network); 4 Jun 2010 04:08:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 04:08:20 -0000
Received: (qmail 80962 invoked by uid 500); 4 Jun 2010 04:08:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80873 invoked by uid 500); 4 Jun 2010 04:08:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80865 invoked by uid 99); 4 Jun 2010 04:08:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 04:08:17 +0000
X-ASF-Spam-Status: No, hits=-1496.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 04:08:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5447t0F008423
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 04:07:56 GMT
Message-ID: <12598881.174401275624475908.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 00:07:55 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6788) [Herriot] Exception exclusion
 functionality is not working correctly.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875458#action_12875458 ] 

Konstantin Boudnik commented on HADOOP-6788:
--------------------------------------------

Well, first we need to make sure that such JIRAs belong to 'test' component. And we can use something like 'test framework (Herriot)' in the description.

> [Herriot] Exception exclusion functionality is not working correctly.
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6788
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6788
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Balaji Rajagopalan
>         Attachments: 6788-ydist-security.patch, 6788-ydist-security.patch, HADOOP-6788.patch, HADOOP-6788.patch
>
>
> Exception exclusion functionality is not working correctly because of that tests are failing by not matching the error count.
> I debugged the issue and found that the problem with shell command which is generating in the getNumberOfMatchesInLogFile function.
> Currently building the shell command in the following way. 
> if(list != null){
>   for(int i =0; i < list.length; ++i)
>   {
>     filePattern.append(" | grep -v " + list[i] );
>   }
> }
>     String[] cmd =
>         new String[] {
>             "bash",
>             "-c",
>             "grep -c "
>                 + pattern + " " + filePattern
>                 + " | awk -F: '{s+=$2} END {print s}'" };    
> However, The above commnad won't work correctly because you are counting the exceptions in the file before excluding the known exceptions.
> In this case it gives the mismatch error counts everytime.The shell command should be in the following way to work correctly.
> if (list != null) {
>   int index = 0;
>   for (String excludeExp : list) {
>     filePattern.append((++index < list.length)? "| grep -v " : 
>             "| grep -vc " + list[i] );  
>   }
> }
> String[] cmd =
>    new String[] {
>        "bash",
>        "-c",
>        "grep "
>            + pattern + " " + filePattern
>            + " | awk -F: '{s+=$2} END {print s}'" };  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8813-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 04:14:55 2010
Return-Path: <common-issues-return-8813-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25157 invoked from network); 4 Jun 2010 04:14:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 04:14:55 -0000
Received: (qmail 83198 invoked by uid 500); 4 Jun 2010 04:14:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83069 invoked by uid 500); 4 Jun 2010 04:14:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83030 invoked by uid 99); 4 Jun 2010 04:14:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 04:14:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 04:14:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o544DvZK008455
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 04:13:57 GMT
Message-ID: <32188908.174451275624837102.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 00:13:57 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6788) [Herriot] Exception exclusion
 functionality is not working correctly.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik reassigned HADOOP-6788:
------------------------------------------

    Assignee: Vinay Kumar Thota  (was: Balaji Rajagopalan)

> [Herriot] Exception exclusion functionality is not working correctly.
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6788
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6788
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>             Fix For: 0.21.0
>
>         Attachments: 6788-ydist-security.patch, 6788-ydist-security.patch, HADOOP-6788.patch, HADOOP-6788.patch
>
>
> Exception exclusion functionality is not working correctly because of that tests are failing by not matching the error count.
> I debugged the issue and found that the problem with shell command which is generating in the getNumberOfMatchesInLogFile function.
> Currently building the shell command in the following way. 
> if(list != null){
>   for(int i =0; i < list.length; ++i)
>   {
>     filePattern.append(" | grep -v " + list[i] );
>   }
> }
>     String[] cmd =
>         new String[] {
>             "bash",
>             "-c",
>             "grep -c "
>                 + pattern + " " + filePattern
>                 + " | awk -F: '{s+=$2} END {print s}'" };    
> However, The above commnad won't work correctly because you are counting the exceptions in the file before excluding the known exceptions.
> In this case it gives the mismatch error counts everytime.The shell command should be in the following way to work correctly.
> if (list != null) {
>   int index = 0;
>   for (String excludeExp : list) {
>     filePattern.append((++index < list.length)? "| grep -v " : 
>             "| grep -vc " + list[i] );  
>   }
> }
> String[] cmd =
>    new String[] {
>        "bash",
>        "-c",
>        "grep "
>            + pattern + " " + filePattern
>            + " | awk -F: '{s+=$2} END {print s}'" };  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8814-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 04:14:55 2010
Return-Path: <common-issues-return-8814-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25161 invoked from network); 4 Jun 2010 04:14:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 04:14:55 -0000
Received: (qmail 83223 invoked by uid 500); 4 Jun 2010 04:14:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83076 invoked by uid 500); 4 Jun 2010 04:14:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83028 invoked by uid 99); 4 Jun 2010 04:14:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 04:14:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 04:14:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o544DvXd008470
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 04:13:57 GMT
Message-ID: <32627226.174501275624837793.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 00:13:57 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6788) [Herriot] Exception exclusion
 functionality is not working correctly.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6788:
---------------------------------------

           Status: Resolved  (was: Patch Available)
    Fix Version/s: 0.21.0
       Resolution: Fixed

I have just committed it. Thank you Vinay.

> [Herriot] Exception exclusion functionality is not working correctly.
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6788
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6788
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>             Fix For: 0.21.0
>
>         Attachments: 6788-ydist-security.patch, 6788-ydist-security.patch, HADOOP-6788.patch, HADOOP-6788.patch
>
>
> Exception exclusion functionality is not working correctly because of that tests are failing by not matching the error count.
> I debugged the issue and found that the problem with shell command which is generating in the getNumberOfMatchesInLogFile function.
> Currently building the shell command in the following way. 
> if(list != null){
>   for(int i =0; i < list.length; ++i)
>   {
>     filePattern.append(" | grep -v " + list[i] );
>   }
> }
>     String[] cmd =
>         new String[] {
>             "bash",
>             "-c",
>             "grep -c "
>                 + pattern + " " + filePattern
>                 + " | awk -F: '{s+=$2} END {print s}'" };    
> However, The above commnad won't work correctly because you are counting the exceptions in the file before excluding the known exceptions.
> In this case it gives the mismatch error counts everytime.The shell command should be in the following way to work correctly.
> if (list != null) {
>   int index = 0;
>   for (String excludeExp : list) {
>     filePattern.append((++index < list.length)? "| grep -v " : 
>             "| grep -vc " + list[i] );  
>   }
> }
> String[] cmd =
>    new String[] {
>        "bash",
>        "-c",
>        "grep "
>            + pattern + " " + filePattern
>            + " | awk -F: '{s+=$2} END {print s}'" };  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8815-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 04:24:25 2010
Return-Path: <common-issues-return-8815-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26165 invoked from network); 4 Jun 2010 04:24:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 04:24:25 -0000
Received: (qmail 88974 invoked by uid 500); 4 Jun 2010 04:24:25 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88871 invoked by uid 500); 4 Jun 2010 04:24:25 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88863 invoked by uid 99); 4 Jun 2010 04:24:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 04:24:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 04:24:22 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o544O03u008537
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 04:24:00 GMT
Message-ID: <1280981.174641275625440502.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 00:24:00 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6788) [Herriot] Exception exclusion
 functionality is not working correctly.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875462#action_12875462 ] 

Hudson commented on HADOOP-6788:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #278 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/278/])
    HADOOP-6788. [Herriot] Exception exclusion functionality is not working
correctly. Contributed by Vinay Kumar Thota.


> [Herriot] Exception exclusion functionality is not working correctly.
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6788
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6788
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>             Fix For: 0.21.0
>
>         Attachments: 6788-ydist-security.patch, 6788-ydist-security.patch, HADOOP-6788.patch, HADOOP-6788.patch
>
>
> Exception exclusion functionality is not working correctly because of that tests are failing by not matching the error count.
> I debugged the issue and found that the problem with shell command which is generating in the getNumberOfMatchesInLogFile function.
> Currently building the shell command in the following way. 
> if(list != null){
>   for(int i =0; i < list.length; ++i)
>   {
>     filePattern.append(" | grep -v " + list[i] );
>   }
> }
>     String[] cmd =
>         new String[] {
>             "bash",
>             "-c",
>             "grep -c "
>                 + pattern + " " + filePattern
>                 + " | awk -F: '{s+=$2} END {print s}'" };    
> However, The above commnad won't work correctly because you are counting the exceptions in the file before excluding the known exceptions.
> In this case it gives the mismatch error counts everytime.The shell command should be in the following way to work correctly.
> if (list != null) {
>   int index = 0;
>   for (String excludeExp : list) {
>     filePattern.append((++index < list.length)? "| grep -v " : 
>             "| grep -vc " + list[i] );  
>   }
> }
> String[] cmd =
>    new String[] {
>        "bash",
>        "-c",
>        "grep "
>            + pattern + " " + filePattern
>            + " | awk -F: '{s+=$2} END {print s}'" };  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8816-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 04:45:19 2010
Return-Path: <common-issues-return-8816-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28164 invoked from network); 4 Jun 2010 04:45:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 04:45:19 -0000
Received: (qmail 1514 invoked by uid 500); 4 Jun 2010 04:45:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1488 invoked by uid 500); 4 Jun 2010 04:45:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1480 invoked by uid 99); 4 Jun 2010 04:45:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 04:45:18 +0000
X-ASF-Spam-Status: No, hits=-1496.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 04:45:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o544ivs2008680
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 04:44:58 GMT
Message-ID: <16218814.174871275626697771.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 00:44:57 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6789) set-version should precede Herriot
 artifacts installation target
In-Reply-To: <13948127.9831274912496806.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6789:
---------------------------------------

        Status: Resolved  (was: Patch Available)
    Resolution: Invalid

Has been actually fixed in the passing by HADOOP-6404 :)

> set-version should precede Herriot artifacts installation target
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6789
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6789
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: 0.22.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>            Priority: Minor
>         Attachments: HADOOP-6789.patch
>
>
> {{set-version}} should be preceding the invocation of {{-mvn-system-install}}. Otherwise it will be failing when called on a clean workspace. This problem only affects local installation of artifacts.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8817-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 04:45:21 2010
Return-Path: <common-issues-return-8817-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28241 invoked from network); 4 Jun 2010 04:45:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 04:45:21 -0000
Received: (qmail 1713 invoked by uid 500); 4 Jun 2010 04:45:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1663 invoked by uid 500); 4 Jun 2010 04:45:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1655 invoked by uid 99); 4 Jun 2010 04:45:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 04:45:21 +0000
X-ASF-Spam-Status: No, hits=-1496.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 04:45:21 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o544j1PH008713
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 04:45:01 GMT
Message-ID: <8199211.174981275626700990.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 00:45:00 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6404) Rename the generated artifacts to
 common instead of core
In-Reply-To: <581314957.1259773100831.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875469#action_12875469 ] 

Konstantin Boudnik commented on HADOOP-6404:
--------------------------------------------

Thank Tom - looks like you've fixed HADOOP-6789 in the passing :-) Real nice!

> Rename the generated artifacts to common instead of core
> --------------------------------------------------------
>
>                 Key: HADOOP-6404
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6404
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>            Reporter: Owen O'Malley
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: HADOOP-6404.patch, HADOOP-6404.patch, HADOOP-6404.patch, HADOOP-6404.patch
>
>
> In the project split we renamed Core to Common, but failed to change the artifact names. We should do it before 0.21.0 is released.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8818-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 04:51:21 2010
Return-Path: <common-issues-return-8818-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28811 invoked from network); 4 Jun 2010 04:51:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 04:51:21 -0000
Received: (qmail 3127 invoked by uid 500); 4 Jun 2010 04:51:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3099 invoked by uid 500); 4 Jun 2010 04:51:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3091 invoked by uid 99); 4 Jun 2010 04:51:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 04:51:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 04:51:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o544ouP2008788
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 04:50:56 GMT
Message-ID: <21017385.175121275627056603.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 00:50:56 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875472#action_12875472 ] 

Jakob Homan commented on HADOOP-6796:
-------------------------------------

It's fine, I guess.  The new patch won't cause any harm and may catch some error at some point.  Since the stateSanityCheck() method is essentially one line that's called once, I'd just inline it.

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.1.patch, HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8819-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 05:11:21 2010
Return-Path: <common-issues-return-8819-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35900 invoked from network); 4 Jun 2010 05:11:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 05:11:20 -0000
Received: (qmail 20397 invoked by uid 500); 4 Jun 2010 05:11:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20197 invoked by uid 500); 4 Jun 2010 05:11:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20183 invoked by uid 99); 4 Jun 2010 05:11:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 05:11:17 +0000
X-ASF-Spam-Status: No, hits=-1496.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 05:11:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o545AuQs009276
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 05:10:56 GMT
Message-ID: <31750940.176171275628256504.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 01:10:56 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6738) Move cluster_setup.xml from MapReduce
 to Common
In-Reply-To: <26228443.22131272581753755.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6738:
------------------------------

    Attachment: HADOOP-6738.patch

> Move cluster_setup.xml from MapReduce to Common
> -----------------------------------------------
>
>                 Key: HADOOP-6738
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6738
>             Project: Hadoop Common
>          Issue Type: Test
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6738.patch, HADOOP-6738.patch
>
>
> Common half of MAPREDUCE-1404.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8820-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 05:19:24 2010
Return-Path: <common-issues-return-8820-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36873 invoked from network); 4 Jun 2010 05:19:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 05:19:23 -0000
Received: (qmail 25245 invoked by uid 500); 4 Jun 2010 05:19:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25125 invoked by uid 500); 4 Jun 2010 05:19:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25112 invoked by uid 99); 4 Jun 2010 05:19:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 05:19:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 05:19:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o545IuZJ009354
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 05:18:56 GMT
Message-ID: <18294364.176381275628736662.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 01:18:56 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875486#action_12875486 ] 

Rodrigo Schmidt commented on HADOOP-6796:
-----------------------------------------

Do you guys think I should add more checks to the method?

I could also check if the numbers are all non-negative, for example. path checking is the most important thing because it makes other methods fail.

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.1.patch, HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8821-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 07:25:26 2010
Return-Path: <common-issues-return-8821-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69950 invoked from network); 4 Jun 2010 07:25:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 07:25:26 -0000
Received: (qmail 10463 invoked by uid 500); 4 Jun 2010 07:25:26 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10378 invoked by uid 500); 4 Jun 2010 07:25:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10266 invoked by uid 99); 4 Jun 2010 07:25:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 07:25:23 +0000
X-ASF-Spam-Status: No, hits=-1497.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 07:25:23 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o547P2lC010462
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 07:25:02 GMT
Message-ID: <25559355.178761275636302954.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 03:25:02 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875518#action_12875518 ] 

Eli Collins commented on HADOOP-6796:
-------------------------------------

I think just checking the path is fine.

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.1.patch, HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8822-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 11:21:25 2010
Return-Path: <common-issues-return-8822-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29234 invoked from network); 4 Jun 2010 11:21:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 11:21:25 -0000
Received: (qmail 58926 invoked by uid 500); 4 Jun 2010 11:21:25 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58524 invoked by uid 500); 4 Jun 2010 11:21:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58500 invoked by uid 99); 4 Jun 2010 11:21:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 11:21:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 11:21:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54BKvYs011910
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 11:20:57 GMT
Message-ID: <12706261.180961275650457721.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 07:20:57 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6788) [Herriot] Exception exclusion
 functionality is not working correctly.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875563#action_12875563 ] 

Hudson commented on HADOOP-6788:
--------------------------------

Integrated in Hadoop-Common-trunk #356 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/356/])
    HADOOP-6788. [Herriot] Exception exclusion functionality is not working
correctly. Contributed by Vinay Kumar Thota.


> [Herriot] Exception exclusion functionality is not working correctly.
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-6788
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6788
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>             Fix For: 0.21.0
>
>         Attachments: 6788-ydist-security.patch, 6788-ydist-security.patch, HADOOP-6788.patch, HADOOP-6788.patch
>
>
> Exception exclusion functionality is not working correctly because of that tests are failing by not matching the error count.
> I debugged the issue and found that the problem with shell command which is generating in the getNumberOfMatchesInLogFile function.
> Currently building the shell command in the following way. 
> if(list != null){
>   for(int i =0; i < list.length; ++i)
>   {
>     filePattern.append(" | grep -v " + list[i] );
>   }
> }
>     String[] cmd =
>         new String[] {
>             "bash",
>             "-c",
>             "grep -c "
>                 + pattern + " " + filePattern
>                 + " | awk -F: '{s+=$2} END {print s}'" };    
> However, The above commnad won't work correctly because you are counting the exceptions in the file before excluding the known exceptions.
> In this case it gives the mismatch error counts everytime.The shell command should be in the following way to work correctly.
> if (list != null) {
>   int index = 0;
>   for (String excludeExp : list) {
>     filePattern.append((++index < list.length)? "| grep -v " : 
>             "| grep -vc " + list[i] );  
>   }
> }
> String[] cmd =
>    new String[] {
>        "bash",
>        "-c",
>        "grep "
>            + pattern + " " + filePattern
>            + " | awk -F: '{s+=$2} END {print s}'" };  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8824-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 11:21:27 2010
Return-Path: <common-issues-return-8824-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29301 invoked from network); 4 Jun 2010 11:21:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 11:21:26 -0000
Received: (qmail 59105 invoked by uid 500); 4 Jun 2010 11:21:26 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58989 invoked by uid 500); 4 Jun 2010 11:21:26 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58525 invoked by uid 99); 4 Jun 2010 11:21:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 11:21:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 11:21:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54BKxBJ011931
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 11:20:59 GMT
Message-ID: <23410271.181031275650459613.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 07:20:59 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6661) User document for
 UserGroupInformation.doAs
In-Reply-To: <927098388.498051269554007283.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875565#action_12875565 ] 

Hudson commented on HADOOP-6661:
--------------------------------

Integrated in Hadoop-Common-trunk #356 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/356/])
    HADOOP-6661. User document for UserGroupInformation.doAs. Contributed by Jitendra Pandey.


> User document for UserGroupInformation.doAs
> -------------------------------------------
>
>                 Key: HADOOP-6661
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6661
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6661-y20.1.patch, HADOOP-6661-y20.2.patch, HADOOP-6661.1.patch, HADOOP-6661.2.patch, HADOOP-6661.3.patch, HADOOP-6661.4.patch
>
>
> A user document should be added for secure impersonation feature. The document should cover a code example about how  UserGroupInformation.doAs should be used for secure impersonation.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8823-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 11:21:27 2010
Return-Path: <common-issues-return-8823-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29304 invoked from network); 4 Jun 2010 11:21:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 11:21:26 -0000
Received: (qmail 58980 invoked by uid 500); 4 Jun 2010 11:21:26 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58812 invoked by uid 500); 4 Jun 2010 11:21:25 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58515 invoked by uid 99); 4 Jun 2010 11:21:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 11:21:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 11:21:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54BKxpj011925
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 11:20:59 GMT
Message-ID: <28049835.181011275650459178.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 07:20:59 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6787) Factor out glob pattern code from
 FileContext and Filesystem
In-Reply-To: <2758661.46911274829813581.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875564#action_12875564 ] 

Hudson commented on HADOOP-6787:
--------------------------------

Integrated in Hadoop-Common-trunk #356 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/356/])
    HADOOP-6787. Factor out glob pattern code from FileContext and FileSystem. Contributed by Luke Lu.


> Factor out glob pattern code from FileContext and Filesystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-6787
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6787
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Luke Lu
>            Assignee: Luke Lu
>             Fix For: 0.22.0
>
>         Attachments: hadoop-6787-trunk-v1.patch, hadoop-6787-trunk-v2.patch, hadoop-6787-y20s-v1.patch
>
>
> Refactor the glob pattern code out of FileContext and FileSystem into a package private GlobFilter and the reusable GlobPattern class (InterfaceAudience.Private)
> Also fix the handling of ^ outside character class ([...]) reported in HADOOP-6618 and make the glob pattern code less restrictive (not throwing on some valid glob patterns.) and more POSIX standard compliant (support [!...]).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8825-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 11:21:27 2010
Return-Path: <common-issues-return-8825-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29329 invoked from network); 4 Jun 2010 11:21:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 11:21:27 -0000
Received: (qmail 59178 invoked by uid 500); 4 Jun 2010 11:21:26 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 59065 invoked by uid 500); 4 Jun 2010 11:21:26 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58526 invoked by uid 99); 4 Jun 2010 11:21:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 11:21:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 11:21:21 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54BKxjr011937
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 11:20:59 GMT
Message-ID: <8065837.181051275650459858.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 07:20:59 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6791) Refresh for proxy superuser config
  (common part for HDFS-1096)
In-Reply-To: <17022862.10441274913350679.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875566#action_12875566 ] 

Hudson commented on HADOOP-6791:
--------------------------------

Integrated in Hadoop-Common-trunk #356 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/356/])
    HADOOP-6791.  Refresh for proxy superuser config


> Refresh for proxy superuser config  (common part for HDFS-1096)
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6791
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6791
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6791.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8826-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 16:37:19 2010
Return-Path: <common-issues-return-8826-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29372 invoked from network); 4 Jun 2010 16:37:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 16:37:19 -0000
Received: (qmail 42487 invoked by uid 500); 4 Jun 2010 16:37:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42444 invoked by uid 500); 4 Jun 2010 16:37:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42435 invoked by uid 99); 4 Jun 2010 16:37:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 16:37:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 16:37:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54GasuJ014407
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 16:36:55 GMT
Message-ID: <31231508.184621275669414835.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 12:36:54 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6738) Move cluster_setup.xml from MapReduce
 to Common
In-Reply-To: <26228443.22131272581753755.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6738:
------------------------------

        Status: Resolved  (was: Patch Available)
    Resolution: Fixed

I've just committed this.

> Move cluster_setup.xml from MapReduce to Common
> -----------------------------------------------
>
>                 Key: HADOOP-6738
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6738
>             Project: Hadoop Common
>          Issue Type: Test
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6738.patch, HADOOP-6738.patch
>
>
> Common half of MAPREDUCE-1404.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8827-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 16:53:23 2010
Return-Path: <common-issues-return-8827-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35228 invoked from network); 4 Jun 2010 16:53:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 16:53:23 -0000
Received: (qmail 57016 invoked by uid 500); 4 Jun 2010 16:53:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56952 invoked by uid 500); 4 Jun 2010 16:53:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56944 invoked by uid 99); 4 Jun 2010 16:53:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 16:53:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 16:53:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54GqwYb014487
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 16:52:58 GMT
Message-ID: <32469486.184761275670378726.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 12:52:58 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6738) Move cluster_setup.xml from
 MapReduce to Common
In-Reply-To: <26228443.22131272581753755.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875659#action_12875659 ] 

Hudson commented on HADOOP-6738:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #279 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/279/])
    HADOOP-6738.  Move cluster_setup.xml, hod_scheduler, commands_manual from MapReduce to Common.


> Move cluster_setup.xml from MapReduce to Common
> -----------------------------------------------
>
>                 Key: HADOOP-6738
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6738
>             Project: Hadoop Common
>          Issue Type: Test
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6738.patch, HADOOP-6738.patch
>
>
> Common half of MAPREDUCE-1404.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8828-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 17:05:20 2010
Return-Path: <common-issues-return-8828-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44947 invoked from network); 4 Jun 2010 17:05:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 17:05:20 -0000
Received: (qmail 77560 invoked by uid 500); 4 Jun 2010 17:05:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77511 invoked by uid 500); 4 Jun 2010 17:05:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77500 invoked by uid 99); 4 Jun 2010 17:05:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 17:05:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 17:05:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54H4t8G014627
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 17:04:55 GMT
Message-ID: <3286554.184921275671095420.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 13:04:55 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6740) Move commands_manual.xml from
 mapreduce into common
In-Reply-To: <17589870.27071272608574801.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White resolved HADOOP-6740.
-------------------------------

    Resolution: Duplicate

Fixed by HADOOP-6738 and MAPREDUCE-1404.

> Move commands_manual.xml from mapreduce into common
> ---------------------------------------------------
>
>                 Key: HADOOP-6740
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6740
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.21.0
>            Reporter: Vinod K V
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>
> Long term, we want to split the commands manual separately into the sub-projects (MAPEDUCE-1079). But for 0.21 atleast we can move it into common. Having it in mapreduce is definitely inconvenient.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8829-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 17:33:25 2010
Return-Path: <common-issues-return-8829-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54392 invoked from network); 4 Jun 2010 17:33:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 17:33:24 -0000
Received: (qmail 15403 invoked by uid 500); 4 Jun 2010 17:33:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15368 invoked by uid 500); 4 Jun 2010 17:33:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15360 invoked by uid 99); 4 Jun 2010 17:33:24 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 17:33:24 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 17:33:22 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54HX0tv014829
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 17:33:00 GMT
Message-ID: <16397910.185351275672780515.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 13:33:00 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875675#action_12875675 ] 

sam rash commented on HADOOP-6762:
----------------------------------

I almost have an updated patch.  I loved the idea of sync on Connection.this.out around submit + wait, but this causes deadlock as the out.write() call inside is a synchronized function.  I instead sync'd on the connection object itself (seems safe and gives the same result.  1 : 1 Connection/socket => bounds the threads by the # of connections
(really, very clean/clever idea to optimize thread use)

one catch with the static executor instance:  nothing ever shuts it down.  While this isn't an issue for actual execution, it won't cause unit tests to hang, will it?  (I made a thread factory that makes the threads daemons for this purpose)

patch coming after i run a larger set of unit tests

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8830-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 17:53:20 2010
Return-Path: <common-issues-return-8830-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59633 invoked from network); 4 Jun 2010 17:53:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 17:53:19 -0000
Received: (qmail 40796 invoked by uid 500); 4 Jun 2010 17:53:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40743 invoked by uid 500); 4 Jun 2010 17:53:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40735 invoked by uid 99); 4 Jun 2010 17:53:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 17:53:19 +0000
X-ASF-Spam-Status: No, hits=-1498.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 17:53:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54HqwXS014927
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 17:52:59 GMT
Message-ID: <15901425.185511275673978977.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 13:52:58 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6762) exception while doing RPC I/O closes
 channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

sam rash updated HADOOP-6762:
-----------------------------

    Attachment: hadoop-6762-7.txt

1. changed to use Future.get() w/o timeout
2. static executor instance with cached thread instances (with daemon thread factory)
3. added catch of interruptedException around sendParams call to avoid undeclared throwable exception

note : used a dedicated lock var around the submit + wait for a sendParams call.  I didn't want to worry about reasoning through lock contention/deadlock with synchronized methods (why I didn't synchronize on "this")


> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt, hadoop-6762-7.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8831-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 17:55:21 2010
Return-Path: <common-issues-return-8831-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 60074 invoked from network); 4 Jun 2010 17:55:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 17:55:21 -0000
Received: (qmail 44027 invoked by uid 500); 4 Jun 2010 17:55:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43994 invoked by uid 500); 4 Jun 2010 17:55:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43986 invoked by uid 99); 4 Jun 2010 17:55:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 17:55:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 17:55:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54HsuEU014951
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 17:54:57 GMT
Message-ID: <25346583.185581275674096933.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 13:54:56 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875680#action_12875680 ] 

sam rash commented on HADOOP-6762:
----------------------------------

btw, is it appropriate to delete the very old patches as they aren't really relevant?  or just leave them?

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt, hadoop-6762-7.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8832-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 20:21:17 2010
Return-Path: <common-issues-return-8832-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40956 invoked from network); 4 Jun 2010 20:21:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 20:21:17 -0000
Received: (qmail 38995 invoked by uid 500); 4 Jun 2010 20:21:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38955 invoked by uid 500); 4 Jun 2010 20:21:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38947 invoked by uid 99); 4 Jun 2010 20:21:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 20:21:16 +0000
X-ASF-Spam-Status: No, hits=-1498.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 20:21:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54KKurT016345
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 20:20:56 GMT
Message-ID: <4423055.188351275682856405.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 16:20:56 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6674) Performance Improvement in Secure
 RPC
In-Reply-To: <375873020.645441270163787224.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875734#action_12875734 ] 

Devaraj Das commented on HADOOP-6674:
-------------------------------------

Some nits:
1) The configuration hadoop.rpc.protection is missing from core-default.xml
2) The if statements 
{noformat} if (dataLength == Client.PING_CALL_ID) {
            if(!useWrap) { {noformat}
    can be combined into a single statement.

> Performance Improvement in Secure RPC
> -------------------------------------
>
>                 Key: HADOOP-6674
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6674
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.patch, HADOOP-6674.1.patch
>
>
> This jira introduces two performance improvements in Sasl RPC
> 1. Setting of Sasl 'quality of protection' to authentication only.
> 2. Addition of BufferedOutputStream underneath SaslOutputStream.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8833-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 21:13:16 2010
Return-Path: <common-issues-return-8833-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58141 invoked from network); 4 Jun 2010 21:13:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 21:13:16 -0000
Received: (qmail 9225 invoked by uid 500); 4 Jun 2010 21:13:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9148 invoked by uid 500); 4 Jun 2010 21:13:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9139 invoked by uid 99); 4 Jun 2010 21:13:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 21:13:15 +0000
X-ASF-Spam-Status: No, hits=-1498.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 21:13:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54LCswG016894
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 21:12:55 GMT
Message-ID: <28110747.189531275685974847.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 17:12:54 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6649) login object in UGI should be inside
 the subject
In-Reply-To: <1899765654.380491269047307235.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6649:
-----------------------------------------

    Attachment: HADOOP-6649.1.patch

Patch for trunk.

> login object in UGI should be inside the subject
> ------------------------------------------------
>
>                 Key: HADOOP-6649
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6649
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6649-y20.1.1.patch, HADOOP-6649-y20.1.patch, HADOOP-6649.1.patch
>
>
> The login object should be moved to the subject in the UGI.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8834-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 21:13:17 2010
Return-Path: <common-issues-return-8834-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 58201 invoked from network); 4 Jun 2010 21:13:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 21:13:16 -0000
Received: (qmail 9378 invoked by uid 500); 4 Jun 2010 21:13:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9302 invoked by uid 500); 4 Jun 2010 21:13:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9288 invoked by uid 99); 4 Jun 2010 21:13:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 21:13:16 +0000
X-ASF-Spam-Status: No, hits=-1498.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 21:13:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54LCtoB016903
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 21:12:55 GMT
Message-ID: <3470844.189561275685975682.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 17:12:55 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6649) login object in UGI should be inside
 the subject
In-Reply-To: <1899765654.380491269047307235.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6649:
-----------------------------------------

    Status: Patch Available  (was: Open)

> login object in UGI should be inside the subject
> ------------------------------------------------
>
>                 Key: HADOOP-6649
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6649
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6649-y20.1.1.patch, HADOOP-6649-y20.1.patch, HADOOP-6649.1.patch
>
>
> The login object should be moved to the subject in the UGI.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8835-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 21:41:21 2010
Return-Path: <common-issues-return-8835-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63423 invoked from network); 4 Jun 2010 21:41:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 21:41:20 -0000
Received: (qmail 40177 invoked by uid 500); 4 Jun 2010 21:41:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40151 invoked by uid 500); 4 Jun 2010 21:41:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40143 invoked by uid 99); 4 Jun 2010 21:41:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 21:41:20 +0000
X-ASF-Spam-Status: No, hits=-1498.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 21:41:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54LexoE017133
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 21:40:59 GMT
Message-ID: <17748663.190111275687659709.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 17:40:59 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6649) login object in UGI should be
 inside the subject
In-Reply-To: <1899765654.380491269047307235.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875768#action_12875768 ] 

Hadoop QA commented on HADOOP-6649:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446368/HADOOP-6649.1.patch
  against trunk revision 951480.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/562/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/562/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/562/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/562/console

This message is automatically generated.

> login object in UGI should be inside the subject
> ------------------------------------------------
>
>                 Key: HADOOP-6649
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6649
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6649-y20.1.1.patch, HADOOP-6649-y20.1.patch, HADOOP-6649.1.patch
>
>
> The login object should be moved to the subject in the UGI.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8836-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 21:47:25 2010
Return-Path: <common-issues-return-8836-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64365 invoked from network); 4 Jun 2010 21:47:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 21:47:25 -0000
Received: (qmail 44825 invoked by uid 500); 4 Jun 2010 21:47:25 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44801 invoked by uid 500); 4 Jun 2010 21:47:25 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44793 invoked by uid 99); 4 Jun 2010 21:47:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 21:47:25 +0000
X-ASF-Spam-Status: No, hits=-1498.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 21:47:25 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54Ll4Af017205
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 21:47:04 GMT
Message-ID: <24244.190291275688024663.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 17:47:04 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6526) Need mapping from long principal
 names to local OS user names
In-Reply-To: <786866304.151051264870594555.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6526:
-----------------------------------

    Attachment: HADOOP-6526.patch

> Need mapping from long principal names to local OS user names
> -------------------------------------------------------------
>
>                 Key: HADOOP-6526
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6526
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 3595485.patch, c-6526-y20.patch, c-6526-y20.patch, c-6526.patch, HADOOP-6526-y20.2.patch, HADOOP-6526-y20.4.patch, HADOOP-6526.patch
>
>
> We need a configurable mapping from full user names (eg. omalley@APACHE.ORG) to local user names (eg. omalley). For many organizations it is sufficient to just use the prefix, however, in the case of shared clusters there may be duplicated prefixes. A configurable mapping will let administrators resolve the issue.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8837-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 21:47:30 2010
Return-Path: <common-issues-return-8837-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64426 invoked from network); 4 Jun 2010 21:47:29 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 21:47:29 -0000
Received: (qmail 45133 invoked by uid 500); 4 Jun 2010 21:47:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45085 invoked by uid 500); 4 Jun 2010 21:47:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45077 invoked by uid 99); 4 Jun 2010 21:47:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 21:47:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 21:47:27 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54Ll6kl017232
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 21:47:06 GMT
Message-ID: <24153392.190381275688026071.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 17:47:06 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6526) Need mapping from long principal
 names to local OS user names
In-Reply-To: <786866304.151051264870594555.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6526:
-----------------------------------

    Status: Patch Available  (was: Open)

> Need mapping from long principal names to local OS user names
> -------------------------------------------------------------
>
>                 Key: HADOOP-6526
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6526
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 3595485.patch, c-6526-y20.patch, c-6526-y20.patch, c-6526.patch, HADOOP-6526-y20.2.patch, HADOOP-6526-y20.4.patch, HADOOP-6526.patch
>
>
> We need a configurable mapping from full user names (eg. omalley@APACHE.ORG) to local user names (eg. omalley). For many organizations it is sufficient to just use the prefix, however, in the case of shared clusters there may be duplicated prefixes. A configurable mapping will let administrators resolve the issue.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8838-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 21:53:28 2010
Return-Path: <common-issues-return-8838-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65796 invoked from network); 4 Jun 2010 21:53:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 21:53:28 -0000
Received: (qmail 53731 invoked by uid 500); 4 Jun 2010 21:53:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53689 invoked by uid 500); 4 Jun 2010 21:53:28 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53681 invoked by uid 99); 4 Jun 2010 21:53:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 21:53:28 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 21:53:25 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54Lr3vt017263
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 21:53:04 GMT
Message-ID: <11010584.190441275688383694.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 17:53:03 -0400 (EDT)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6756) Clean up and add documentation for
 configuration keys in CommonConfigurationKeys.java
In-Reply-To: <23380183.20521273191468124.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875770#action_12875770 ] 

Konstantin Shvachko commented on HADOOP-6756:
---------------------------------------------

# In core-default.xml some new fields have been introduce without default values.
The default values should be there in order to define the default behavior and also as an example.
# Some fields new fields in core-default.xml are commented out. What does that mean?
If they are hidden (undocumented) variables, then they should not be core-default.xml,
if they are public, then they should not be commented out and have a reasonable default value.
# {{hadoop.security.group.mapping}} is listed twice in core-default.xml. 
Remove the one with empty value, and merge the descriptions.
# I think it makes sense to introduce the security section in core-default.xml and group all security fields under it:
{code}
 <!--- security properties -->
<property>
  <name>hadoop.security.group.mapping</name>
...
{code}
# I did not understand the description of {{io.map.index.interval}}.
# In CommonConfigurationKeysPublic I am not sure where "See src/core/core-default.xml" should lead to.
I'd propose to use something like this:
{code}
/** See  <a href="{@docRoot}/common-default.xml">common-default.xml</a> */
{code}
# The comment in CommonConfigurationKeys
{code}
//TBD: Code is not updated to use following keys.
//These keys will be used in later versions
{code}
should be investigated in terms of which variables it is referred to.
In the least it is misplaced, because {{FS_AUTOMATIC_CLOSE_KEY}} and {{FS_FTP_HOST_KEY}} are definitely used.
If it is a legacy artifact, then let's remove it.


> Clean up and add documentation for configuration keys in CommonConfigurationKeys.java
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6756
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6756
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.3, 0.22.0
>            Reporter: Erik Steffl
>             Fix For: 0.20.3, 0.22.0
>
>         Attachments: HADOOP-6756-0.22.0.patch, jira.HADOOP-6756-0.20-1-FS_DEFAULT_NAME_KEY.patch, jira.HADOOP-6756-0.20-1.patch, jira.HADOOP-6756-0.20.patch
>
>
> Configuration keys in CommonConfigurationKeys.java should be cleaned up and documented (javadoc comments, appropriate *-default.xml descriptions).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8839-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 22:09:21 2010
Return-Path: <common-issues-return-8839-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73836 invoked from network); 4 Jun 2010 22:09:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 22:09:21 -0000
Received: (qmail 66417 invoked by uid 500); 4 Jun 2010 22:09:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66297 invoked by uid 500); 4 Jun 2010 22:09:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66288 invoked by uid 99); 4 Jun 2010 22:09:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 22:09:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 22:09:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54M8uU0017426
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 22:08:56 GMT
Message-ID: <20165502.190721275689336274.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 18:08:56 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6526) Need mapping from long principal
 names to local OS user names
In-Reply-To: <786866304.151051264870594555.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875775#action_12875775 ] 

Hadoop QA commented on HADOOP-6526:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446371/HADOOP-6526.patch
  against trunk revision 951480.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 10 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    -1 javac.  The applied patch generated 1022 javac compiler warnings (more than the trunk's current 1017 warnings).

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    -1 release audit.  The applied patch generated 3 release audit warnings (more than the trunk's current 2 warnings).

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/563/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/563/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/563/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/563/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/563/console

This message is automatically generated.

> Need mapping from long principal names to local OS user names
> -------------------------------------------------------------
>
>                 Key: HADOOP-6526
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6526
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 3595485.patch, c-6526-y20.patch, c-6526-y20.patch, c-6526.patch, HADOOP-6526-y20.2.patch, HADOOP-6526-y20.4.patch, HADOOP-6526.patch
>
>
> We need a configurable mapping from full user names (eg. omalley@APACHE.ORG) to local user names (eg. omalley). For many organizations it is sufficient to just use the prefix, however, in the case of shared clusters there may be duplicated prefixes. A configurable mapping will let administrators resolve the issue.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8840-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 22:13:21 2010
Return-Path: <common-issues-return-8840-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74933 invoked from network); 4 Jun 2010 22:13:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 22:13:21 -0000
Received: (qmail 72464 invoked by uid 500); 4 Jun 2010 22:13:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72407 invoked by uid 500); 4 Jun 2010 22:13:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72399 invoked by uid 99); 4 Jun 2010 22:13:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 22:13:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 22:13:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54MCvRN017452
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 22:12:57 GMT
Message-ID: <6290265.190781275689577044.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 18:12:57 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6674) Performance Improvement in Secure RPC
In-Reply-To: <375873020.645441270163787224.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6674:
-----------------------------------------

    Attachment: HADOOP-6674.2.patch

New patch addressing the comments.

> Performance Improvement in Secure RPC
> -------------------------------------
>
>                 Key: HADOOP-6674
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6674
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.patch, HADOOP-6674.1.patch, HADOOP-6674.2.patch
>
>
> This jira introduces two performance improvements in Sasl RPC
> 1. Setting of Sasl 'quality of protection' to authentication only.
> 2. Addition of BufferedOutputStream underneath SaslOutputStream.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8841-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 22:13:22 2010
Return-Path: <common-issues-return-8841-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74981 invoked from network); 4 Jun 2010 22:13:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 22:13:21 -0000
Received: (qmail 72584 invoked by uid 500); 4 Jun 2010 22:13:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72552 invoked by uid 500); 4 Jun 2010 22:13:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72544 invoked by uid 99); 4 Jun 2010 22:13:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 22:13:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 22:13:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54MCwxA017470
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 22:12:58 GMT
Message-ID: <10643189.190841275689577986.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 18:12:57 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6674) Performance Improvement in Secure RPC
In-Reply-To: <375873020.645441270163787224.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6674:
-----------------------------------------

    Status: Open  (was: Patch Available)

> Performance Improvement in Secure RPC
> -------------------------------------
>
>                 Key: HADOOP-6674
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6674
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.patch, HADOOP-6674.1.patch, HADOOP-6674.2.patch
>
>
> This jira introduces two performance improvements in Sasl RPC
> 1. Setting of Sasl 'quality of protection' to authentication only.
> 2. Addition of BufferedOutputStream underneath SaslOutputStream.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8842-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 22:13:22 2010
Return-Path: <common-issues-return-8842-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75029 invoked from network); 4 Jun 2010 22:13:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 22:13:22 -0000
Received: (qmail 72731 invoked by uid 500); 4 Jun 2010 22:13:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72705 invoked by uid 500); 4 Jun 2010 22:13:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72692 invoked by uid 99); 4 Jun 2010 22:13:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 22:13:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 22:13:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54MCwIs017488
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 22:12:58 GMT
Message-ID: <8894946.190901275689578691.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 18:12:58 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6674) Performance Improvement in Secure RPC
In-Reply-To: <375873020.645441270163787224.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6674:
-----------------------------------------

    Status: Patch Available  (was: Open)

> Performance Improvement in Secure RPC
> -------------------------------------
>
>                 Key: HADOOP-6674
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6674
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.patch, HADOOP-6674.1.patch, HADOOP-6674.2.patch
>
>
> This jira introduces two performance improvements in Sasl RPC
> 1. Setting of Sasl 'quality of protection' to authentication only.
> 2. Addition of BufferedOutputStream underneath SaslOutputStream.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8843-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 22:39:16 2010
Return-Path: <common-issues-return-8843-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80495 invoked from network); 4 Jun 2010 22:39:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 22:39:16 -0000
Received: (qmail 2590 invoked by uid 500); 4 Jun 2010 22:39:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2551 invoked by uid 500); 4 Jun 2010 22:39:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2540 invoked by uid 99); 4 Jun 2010 22:39:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 22:39:16 +0000
X-ASF-Spam-Status: No, hits=-1499.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 22:39:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54McskO017750
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 22:38:55 GMT
Message-ID: <5956220.191581275691134905.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 18:38:54 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6674) Performance Improvement in Secure
 RPC
In-Reply-To: <375873020.645441270163787224.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875784#action_12875784 ] 

Hadoop QA commented on HADOOP-6674:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446373/HADOOP-6674.2.patch
  against trunk revision 951480.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/564/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/564/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/564/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/564/console

This message is automatically generated.

> Performance Improvement in Secure RPC
> -------------------------------------
>
>                 Key: HADOOP-6674
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6674
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.patch, HADOOP-6674.1.patch, HADOOP-6674.2.patch
>
>
> This jira introduces two performance improvements in Sasl RPC
> 1. Setting of Sasl 'quality of protection' to authentication only.
> 2. Addition of BufferedOutputStream underneath SaslOutputStream.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8844-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 22:43:15 2010
Return-Path: <common-issues-return-8844-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81389 invoked from network); 4 Jun 2010 22:43:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 22:43:14 -0000
Received: (qmail 7090 invoked by uid 500); 4 Jun 2010 22:43:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7056 invoked by uid 500); 4 Jun 2010 22:43:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7048 invoked by uid 99); 4 Jun 2010 22:43:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 22:43:14 +0000
X-ASF-Spam-Status: No, hits=-1499.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 22:43:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54MgrRi017800
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 22:42:54 GMT
Message-ID: <22677341.191711275691373941.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 18:42:53 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6780) Move Hadoop cloud scripts to Whirr
In-Reply-To: <9575050.311274374037358.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875787#action_12875787 ] 

Hudson commented on HADOOP-6780:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #280 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/280/])
    HADOOP-6780. Move Hadoop cloud scripts to Whirr.


> Move Hadoop cloud scripts to Whirr
> ----------------------------------
>
>                 Key: HADOOP-6780
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6780
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: contrib/cloud
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.21.0
>
>
> Whirr is a new Apache Incubator project for running cloud services. See http://incubator.apache.org/projects/whirr.html and http://wiki.apache.org/incubator/WhirrProposal.
> The cloud scripts in src/contrib/cloud should move to Whirr. This should be done in 0.21 too, so the scripts do not appear in that release (since having them in Hadoop for a single release would be confusing).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8845-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 23:33:18 2010
Return-Path: <common-issues-return-8845-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92282 invoked from network); 4 Jun 2010 23:33:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 23:33:17 -0000
Received: (qmail 42985 invoked by uid 500); 4 Jun 2010 23:33:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42940 invoked by uid 500); 4 Jun 2010 23:33:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42715 invoked by uid 99); 4 Jun 2010 23:33:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 23:33:17 +0000
X-ASF-Spam-Status: No, hits=-1499.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 23:33:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54NWu6Q018185
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 23:32:56 GMT
Message-ID: <18517295.192521275694376412.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 19:32:56 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6526) Need mapping from long principal
 names to local OS user names
In-Reply-To: <786866304.151051264870594555.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6526:
-----------------------------------

    Status: Open  (was: Patch Available)

> Need mapping from long principal names to local OS user names
> -------------------------------------------------------------
>
>                 Key: HADOOP-6526
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6526
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 3595485.patch, c-6526-y20.patch, c-6526-y20.patch, c-6526.patch, HADOOP-6526-y20.2.patch, HADOOP-6526-y20.4.patch, HADOOP-6526.patch
>
>
> We need a configurable mapping from full user names (eg. omalley@APACHE.ORG) to local user names (eg. omalley). For many organizations it is sufficient to just use the prefix, however, in the case of shared clusters there may be duplicated prefixes. A configurable mapping will let administrators resolve the issue.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8846-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 23:36:25 2010
Return-Path: <common-issues-return-8846-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93906 invoked from network); 4 Jun 2010 23:36:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 23:36:24 -0000
Received: (qmail 48752 invoked by uid 500); 4 Jun 2010 23:36:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48643 invoked by uid 500); 4 Jun 2010 23:36:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48625 invoked by uid 99); 4 Jun 2010 23:36:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 23:36:24 +0000
X-ASF-Spam-Status: No, hits=-1499.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 23:36:24 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54Na3d5018254
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 23:36:03 GMT
Message-ID: <29365094.192741275694563511.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 19:36:03 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5793) High speed compression algorithm
 like BMDiff
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-5793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875792#action_12875792 ] 

Luke Lu commented on HADOOP-5793:
---------------------------------

The bmz c api is designed to allow combining 2 different algorithms efficiently (minimize copies/allocations), so it would be nice to allow native compositions like bm_pack/unpack + zlib/lzma/zpaq/fastlz without having to cross the jni chasm. You also probably want to set bmz_set_out/die_proc to override the logging/error handling behavior.

Note, the bmz code is experimental (there are a lot of alternative and benchmarking code in it) and I have quite a few tweaks queued up to cleanup/cut down the code size while improving speed. Any feedback or bug reports so far regarding the c-code itself?

> High speed compression algorithm like BMDiff
> --------------------------------------------
>
>                 Key: HADOOP-5793
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5793
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: elhoim gibor
>            Assignee: Michele (@pirroh) Catasta
>            Priority: Minor
>
> Add a high speed compression algorithm like BMDiff.
> It gives speeds ~100MB/s for writes and ~1000MB/s for reads, compressing 2.1billions web pages from 45.1TB in 4.2TB
> Reference:
> http://norfolk.cs.washington.edu/htbin-post/unrestricted/colloq/details.cgi?id=437
> 2005 Jeff Dean talk about google architecture - around 46:00.
> http://feedblog.org/2008/10/12/google-bigtable-compression-zippy-and-bmdiff/
> http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=755678
> A reference implementation exists in HyperTable.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8847-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 23:36:26 2010
Return-Path: <common-issues-return-8847-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93960 invoked from network); 4 Jun 2010 23:36:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 23:36:26 -0000
Received: (qmail 48839 invoked by uid 500); 4 Jun 2010 23:36:26 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48792 invoked by uid 500); 4 Jun 2010 23:36:26 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48784 invoked by uid 99); 4 Jun 2010 23:36:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 23:36:26 +0000
X-ASF-Spam-Status: No, hits=-1499.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 23:36:25 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54Na5oq018284
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 23:36:05 GMT
Message-ID: <2085380.192841275694565722.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 19:36:05 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6526) Need mapping from long principal
 names to local OS user names
In-Reply-To: <786866304.151051264870594555.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6526:
-----------------------------------

    Attachment: HADOOP-6526-1.patch

fixed release warning for krb5.conf
javadoc and javac warnings are for "warning: sun.security.krb5.Config is Sun proprietary API and may be removed in a future release
     [exec]     [javac] import sun.security.krb5.Config; "

leaving it as is.

> Need mapping from long principal names to local OS user names
> -------------------------------------------------------------
>
>                 Key: HADOOP-6526
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6526
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 3595485.patch, c-6526-y20.patch, c-6526-y20.patch, c-6526.patch, HADOOP-6526-1.patch, HADOOP-6526-y20.2.patch, HADOOP-6526-y20.4.patch, HADOOP-6526.patch
>
>
> We need a configurable mapping from full user names (eg. omalley@APACHE.ORG) to local user names (eg. omalley). For many organizations it is sufficient to just use the prefix, however, in the case of shared clusters there may be duplicated prefixes. A configurable mapping will let administrators resolve the issue.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8848-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 23:36:31 2010
Return-Path: <common-issues-return-8848-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94106 invoked from network); 4 Jun 2010 23:36:31 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 23:36:31 -0000
Received: (qmail 50109 invoked by uid 500); 4 Jun 2010 23:36:31 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50074 invoked by uid 500); 4 Jun 2010 23:36:31 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50066 invoked by uid 99); 4 Jun 2010 23:36:31 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 23:36:31 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 23:36:28 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54Na7i5018312
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 23:36:07 GMT
Message-ID: <14729678.192931275694567456.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 19:36:07 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6526) Need mapping from long principal
 names to local OS user names
In-Reply-To: <786866304.151051264870594555.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6526:
-----------------------------------

    Status: Patch Available  (was: Open)

> Need mapping from long principal names to local OS user names
> -------------------------------------------------------------
>
>                 Key: HADOOP-6526
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6526
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 3595485.patch, c-6526-y20.patch, c-6526-y20.patch, c-6526.patch, HADOOP-6526-1.patch, HADOOP-6526-y20.2.patch, HADOOP-6526-y20.4.patch, HADOOP-6526.patch
>
>
> We need a configurable mapping from full user names (eg. omalley@APACHE.ORG) to local user names (eg. omalley). For many organizations it is sufficient to just use the prefix, however, in the case of shared clusters there may be duplicated prefixes. A configurable mapping will let administrators resolve the issue.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8849-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 23:46:27 2010
Return-Path: <common-issues-return-8849-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95769 invoked from network); 4 Jun 2010 23:46:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 23:46:27 -0000
Received: (qmail 56899 invoked by uid 500); 4 Jun 2010 23:46:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56866 invoked by uid 500); 4 Jun 2010 23:46:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56858 invoked by uid 99); 4 Jun 2010 23:46:27 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 23:46:27 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 23:46:25 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54Nk32M018420
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 23:46:03 GMT
Message-ID: <10151977.193211275695163699.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 19:46:03 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6810) Extract a subset of tests for smoke
 (DOA) validation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Extract a subset of tests for smoke (DOA) validation
----------------------------------------------------

                 Key: HADOOP-6810
                 URL: https://issues.apache.org/jira/browse/HADOOP-6810
             Project: Hadoop Common
          Issue Type: Improvement
          Components: test
    Affects Versions: 0.20.3
            Reporter: Konstantin Boudnik


Currently, there's no way to quickly find out if a new build of Hadoop is reasonably viable and can be used for future development/testing cycle.

Similarly to 'commit' testing it'd be great to have a wider subset of existing tests to be executed inside of 30 minutes. The main goal of such a subset is to have a reasonable guarantees that different parts of Hadoop's functionality aren't broken and the system is not DOA.

Full test run takes about 3+ hours and it is a too long wait time in many cases. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8850-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 23:52:26 2010
Return-Path: <common-issues-return-8850-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96979 invoked from network); 4 Jun 2010 23:52:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 23:52:26 -0000
Received: (qmail 61335 invoked by uid 500); 4 Jun 2010 23:52:25 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61265 invoked by uid 500); 4 Jun 2010 23:52:25 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61257 invoked by uid 99); 4 Jun 2010 23:52:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 23:52:25 +0000
X-ASF-Spam-Status: No, hits=-1499.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 23:52:25 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54Nq47h018445
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 23:52:05 GMT
Message-ID: <12379087.193251275695524772.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 19:52:04 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6810) Extract a subset of tests for smoke
 (DOA) validation
In-Reply-To: <10151977.193211275695163699.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875797#action_12875797 ] 

Jakob Homan commented on HADOOP-6810:
-------------------------------------

How would this be different than the current commit (run-commit-test) test suite? That suite may need to be updated, it's been a while. The Common tests already take less than 10 minutes to run...

> Extract a subset of tests for smoke (DOA) validation
> ----------------------------------------------------
>
>                 Key: HADOOP-6810
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6810
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.20.3
>            Reporter: Konstantin Boudnik
>
> Currently, there's no way to quickly find out if a new build of Hadoop is reasonably viable and can be used for future development/testing cycle.
> Similarly to 'commit' testing it'd be great to have a wider subset of existing tests to be executed inside of 30 minutes. The main goal of such a subset is to have a reasonable guarantees that different parts of Hadoop's functionality aren't broken and the system is not DOA.
> Full test run takes about 3+ hours and it is a too long wait time in many cases. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8851-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 04 23:56:26 2010
Return-Path: <common-issues-return-8851-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97432 invoked from network); 4 Jun 2010 23:56:25 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 4 Jun 2010 23:56:25 -0000
Received: (qmail 62747 invoked by uid 500); 4 Jun 2010 23:56:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62648 invoked by uid 500); 4 Jun 2010 23:56:24 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62632 invoked by uid 99); 4 Jun 2010 23:56:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 23:56:24 +0000
X-ASF-Spam-Status: No, hits=-1499.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jun 2010 23:56:24 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o54Nu31m018475
	for <common-issues@hadoop.apache.org>; Fri, 4 Jun 2010 23:56:04 GMT
Message-ID: <3408631.193321275695763950.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 19:56:03 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6780) Move Hadoop cloud scripts to Whirr
In-Reply-To: <9575050.311274374037358.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White resolved HADOOP-6780.
-------------------------------

    Fix Version/s:     (was: 0.21.0)
       Resolution: Fixed

I've just committed this.

> Move Hadoop cloud scripts to Whirr
> ----------------------------------
>
>                 Key: HADOOP-6780
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6780
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: contrib/cloud
>            Reporter: Tom White
>            Assignee: Tom White
>
> Whirr is a new Apache Incubator project for running cloud services. See http://incubator.apache.org/projects/whirr.html and http://wiki.apache.org/incubator/WhirrProposal.
> The cloud scripts in src/contrib/cloud should move to Whirr. This should be done in 0.21 too, so the scripts do not appear in that release (since having them in Hadoop for a single release would be confusing).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8852-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 05 00:04:28 2010
Return-Path: <common-issues-return-8852-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98619 invoked from network); 5 Jun 2010 00:04:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jun 2010 00:04:27 -0000
Received: (qmail 64959 invoked by uid 500); 5 Jun 2010 00:04:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64920 invoked by uid 500); 5 Jun 2010 00:04:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64912 invoked by uid 99); 5 Jun 2010 00:04:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 00:04:27 +0000
X-ASF-Spam-Status: No, hits=-1499.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 00:04:27 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o55047s0018612
	for <common-issues@hadoop.apache.org>; Sat, 5 Jun 2010 00:04:07 GMT
Message-ID: <2967945.193551275696247106.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 20:04:07 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6780) Move Hadoop cloud scripts to Whirr
In-Reply-To: <9575050.311274374037358.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875802#action_12875802 ] 

Hudson commented on HADOOP-6780:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #281 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/281/])
    HADOOP-6780. Move Hadoop cloud scripts to Whirr.


> Move Hadoop cloud scripts to Whirr
> ----------------------------------
>
>                 Key: HADOOP-6780
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6780
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: contrib/cloud
>            Reporter: Tom White
>            Assignee: Tom White
>
> Whirr is a new Apache Incubator project for running cloud services. See http://incubator.apache.org/projects/whirr.html and http://wiki.apache.org/incubator/WhirrProposal.
> The cloud scripts in src/contrib/cloud should move to Whirr. This should be done in 0.21 too, so the scripts do not appear in that release (since having them in Hadoop for a single release would be confusing).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8853-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 05 00:10:22 2010
Return-Path: <common-issues-return-8853-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3108 invoked from network); 5 Jun 2010 00:10:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jun 2010 00:10:22 -0000
Received: (qmail 65552 invoked by uid 500); 5 Jun 2010 00:10:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65513 invoked by uid 500); 5 Jun 2010 00:10:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65505 invoked by uid 99); 5 Jun 2010 00:10:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 00:10:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 00:10:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5509waK018632
	for <common-issues@hadoop.apache.org>; Sat, 5 Jun 2010 00:09:58 GMT
Message-ID: <10366540.193581275696598037.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 20:09:58 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6649) login object in UGI should be inside
 the subject
In-Reply-To: <1899765654.380491269047307235.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6649:
-----------------------------------

    Hadoop Flags: [Reviewed]

+1

> login object in UGI should be inside the subject
> ------------------------------------------------
>
>                 Key: HADOOP-6649
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6649
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6649-y20.1.1.patch, HADOOP-6649-y20.1.patch, HADOOP-6649.1.patch
>
>
> The login object should be moved to the subject in the UGI.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8854-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 05 00:12:26 2010
Return-Path: <common-issues-return-8854-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3662 invoked from network); 5 Jun 2010 00:12:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jun 2010 00:12:26 -0000
Received: (qmail 68674 invoked by uid 500); 5 Jun 2010 00:12:25 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68641 invoked by uid 500); 5 Jun 2010 00:12:25 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68633 invoked by uid 99); 5 Jun 2010 00:12:25 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 00:12:25 +0000
X-ASF-Spam-Status: No, hits=-1499.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 00:12:25 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o550C4kV018653
	for <common-issues@hadoop.apache.org>; Sat, 5 Jun 2010 00:12:05 GMT
Message-ID: <32116089.193611275696724938.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 20:12:04 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6649) login object in UGI should be inside
 the subject
In-Reply-To: <1899765654.380491269047307235.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6649:
-----------------------------------

        Status: Resolved  (was: Patch Available)
    Resolution: Fixed

committed to trunk. Thanks Jitendra.

> login object in UGI should be inside the subject
> ------------------------------------------------
>
>                 Key: HADOOP-6649
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6649
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6649-y20.1.1.patch, HADOOP-6649-y20.1.patch, HADOOP-6649.1.patch
>
>
> The login object should be moved to the subject in the UGI.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8855-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 05 00:24:29 2010
Return-Path: <common-issues-return-8855-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5232 invoked from network); 5 Jun 2010 00:24:29 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jun 2010 00:24:29 -0000
Received: (qmail 77258 invoked by uid 500); 5 Jun 2010 00:24:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77218 invoked by uid 500); 5 Jun 2010 00:24:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77194 invoked by uid 99); 5 Jun 2010 00:24:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 00:24:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 00:24:27 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o550O5Ck018718
	for <common-issues@hadoop.apache.org>; Sat, 5 Jun 2010 00:24:05 GMT
Message-ID: <9953525.193741275697445383.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 20:24:05 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6649) login object in UGI should be
 inside the subject
In-Reply-To: <1899765654.380491269047307235.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875806#action_12875806 ] 

Hudson commented on HADOOP-6649:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #282 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/282/])
    HADOOP-6649.  login object in UGI should be inside the subject (jnp via boryas)


> login object in UGI should be inside the subject
> ------------------------------------------------
>
>                 Key: HADOOP-6649
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6649
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6649-y20.1.1.patch, HADOOP-6649-y20.1.patch, HADOOP-6649.1.patch
>
>
> The login object should be moved to the subject in the UGI.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8856-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 05 00:28:26 2010
Return-Path: <common-issues-return-8856-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11662 invoked from network); 5 Jun 2010 00:28:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jun 2010 00:28:26 -0000
Received: (qmail 77809 invoked by uid 500); 5 Jun 2010 00:28:26 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77693 invoked by uid 500); 5 Jun 2010 00:28:26 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77685 invoked by uid 99); 5 Jun 2010 00:28:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 00:28:26 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 00:28:24 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o550S2ba018736
	for <common-issues@hadoop.apache.org>; Sat, 5 Jun 2010 00:28:02 GMT
Message-ID: <29051724.193781275697682525.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 20:28:02 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6810) Extract a subset of tests for smoke
 (DOA) validation
In-Reply-To: <10151977.193211275695163699.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875807#action_12875807 ] 

Konstantin Boudnik commented on HADOOP-6810:
--------------------------------------------

It is different for 0.20 branch where all tests are mixed together.
For HDFS/MR we'll need a separate JIRAs.

> Extract a subset of tests for smoke (DOA) validation
> ----------------------------------------------------
>
>                 Key: HADOOP-6810
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6810
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.20.3
>            Reporter: Konstantin Boudnik
>
> Currently, there's no way to quickly find out if a new build of Hadoop is reasonably viable and can be used for future development/testing cycle.
> Similarly to 'commit' testing it'd be great to have a wider subset of existing tests to be executed inside of 30 minutes. The main goal of such a subset is to have a reasonable guarantees that different parts of Hadoop's functionality aren't broken and the system is not DOA.
> Full test run takes about 3+ hours and it is a too long wait time in many cases. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8857-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 05 00:36:20 2010
Return-Path: <common-issues-return-8857-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12889 invoked from network); 5 Jun 2010 00:36:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jun 2010 00:36:19 -0000
Received: (qmail 82382 invoked by uid 500); 5 Jun 2010 00:36:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82342 invoked by uid 500); 5 Jun 2010 00:36:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82334 invoked by uid 99); 5 Jun 2010 00:36:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 00:36:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 00:36:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o550Ztua018811
	for <common-issues@hadoop.apache.org>; Sat, 5 Jun 2010 00:35:56 GMT
Message-ID: <31434177.193961275698155811.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 20:35:55 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6674) Performance Improvement in Secure RPC
In-Reply-To: <375873020.645441270163787224.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6674:
--------------------------------

           Status: Resolved  (was: Patch Available)
    Fix Version/s: 0.22.0
       Resolution: Fixed

I just committed this. Thanks, Jitendra!

> Performance Improvement in Secure RPC
> -------------------------------------
>
>                 Key: HADOOP-6674
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6674
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.patch, HADOOP-6674.1.patch, HADOOP-6674.2.patch
>
>
> This jira introduces two performance improvements in Sasl RPC
> 1. Setting of Sasl 'quality of protection' to authentication only.
> 2. Addition of BufferedOutputStream underneath SaslOutputStream.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8858-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 05 00:46:26 2010
Return-Path: <common-issues-return-8858-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14508 invoked from network); 5 Jun 2010 00:46:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jun 2010 00:46:26 -0000
Received: (qmail 87824 invoked by uid 500); 5 Jun 2010 00:46:26 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87803 invoked by uid 500); 5 Jun 2010 00:46:26 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87795 invoked by uid 99); 5 Jun 2010 00:46:26 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 00:46:26 +0000
X-ASF-Spam-Status: No, hits=-1499.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 00:46:25 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o550k5R2018961
	for <common-issues@hadoop.apache.org>; Sat, 5 Jun 2010 00:46:05 GMT
Message-ID: <31236999.194351275698765560.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 20:46:05 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6810) Extract a subset of tests for smoke
 (DOA) validation
In-Reply-To: <10151977.193211275695163699.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6810:
---------------------------------------

    Attachment: smoke-tests.patch

Initial patch with preliminary list of tests for MR. Core/HDFS test list to be added.

> Extract a subset of tests for smoke (DOA) validation
> ----------------------------------------------------
>
>                 Key: HADOOP-6810
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6810
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.20.3
>            Reporter: Konstantin Boudnik
>         Attachments: smoke-tests.patch
>
>
> Currently, there's no way to quickly find out if a new build of Hadoop is reasonably viable and can be used for future development/testing cycle.
> Similarly to 'commit' testing it'd be great to have a wider subset of existing tests to be executed inside of 30 minutes. The main goal of such a subset is to have a reasonable guarantees that different parts of Hadoop's functionality aren't broken and the system is not DOA.
> Full test run takes about 3+ hours and it is a too long wait time in many cases. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8859-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 05 00:48:15 2010
Return-Path: <common-issues-return-8859-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16596 invoked from network); 5 Jun 2010 00:48:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jun 2010 00:48:14 -0000
Received: (qmail 88951 invoked by uid 500); 5 Jun 2010 00:48:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88846 invoked by uid 500); 5 Jun 2010 00:48:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88838 invoked by uid 99); 5 Jun 2010 00:48:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 00:48:14 +0000
X-ASF-Spam-Status: No, hits=-1499.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 00:48:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o550lr9J018976
	for <common-issues@hadoop.apache.org>; Sat, 5 Jun 2010 00:47:53 GMT
Message-ID: <4483207.194381275698873667.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 20:47:53 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6810) Extract a subset of tests for smoke
 (DOA) validation
In-Reply-To: <10151977.193211275695163699.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875815#action_12875815 ] 

Konstantin Boudnik commented on HADOOP-6810:
--------------------------------------------

To be clean, the above patch is for 0.20.3 branch.

> Extract a subset of tests for smoke (DOA) validation
> ----------------------------------------------------
>
>                 Key: HADOOP-6810
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6810
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.20.3
>            Reporter: Konstantin Boudnik
>         Attachments: smoke-tests.patch
>
>
> Currently, there's no way to quickly find out if a new build of Hadoop is reasonably viable and can be used for future development/testing cycle.
> Similarly to 'commit' testing it'd be great to have a wider subset of existing tests to be executed inside of 30 minutes. The main goal of such a subset is to have a reasonable guarantees that different parts of Hadoop's functionality aren't broken and the system is not DOA.
> Full test run takes about 3+ hours and it is a too long wait time in many cases. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8860-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 05 00:58:19 2010
Return-Path: <common-issues-return-8860-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23517 invoked from network); 5 Jun 2010 00:58:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jun 2010 00:58:18 -0000
Received: (qmail 94615 invoked by uid 500); 5 Jun 2010 00:58:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94587 invoked by uid 500); 5 Jun 2010 00:58:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94572 invoked by uid 99); 5 Jun 2010 00:58:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 00:58:18 +0000
X-ASF-Spam-Status: No, hits=-1499.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 00:58:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o550vvOL019024
	for <common-issues@hadoop.apache.org>; Sat, 5 Jun 2010 00:57:57 GMT
Message-ID: <4632115.194431275699477539.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 20:57:57 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6687) user object in the subject in UGI
  should be reused in case of a relogin.
In-Reply-To: <191053771.27981270592254332.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6687:
-----------------------------------------

    Attachment: HADOOP-6687.2.patch

Patch for trunk.

> user object in the subject in UGI  should be reused in case of a relogin.
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6687
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6687
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6687-y20.2.patch, HADOOP-6687.2.patch
>
>
>  In case of a relogin, a new user object is created in the subject. This causes the subject to lose some state for example authentication method. Instead, a new user object should not be added to the subject if one is already present.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8861-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 05 00:58:19 2010
Return-Path: <common-issues-return-8861-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23529 invoked from network); 5 Jun 2010 00:58:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jun 2010 00:58:18 -0000
Received: (qmail 94745 invoked by uid 500); 5 Jun 2010 00:58:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94589 invoked by uid 500); 5 Jun 2010 00:58:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94571 invoked by uid 99); 5 Jun 2010 00:58:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 00:58:18 +0000
X-ASF-Spam-Status: No, hits=-1499.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 00:58:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o550vwV6019030
	for <common-issues@hadoop.apache.org>; Sat, 5 Jun 2010 00:57:58 GMT
Message-ID: <622941.194451275699478032.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 20:57:58 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6687) user object in the subject in UGI
  should be reused in case of a relogin.
In-Reply-To: <191053771.27981270592254332.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6687:
-----------------------------------------

    Status: Patch Available  (was: Open)

> user object in the subject in UGI  should be reused in case of a relogin.
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6687
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6687
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6687-y20.2.patch, HADOOP-6687.2.patch
>
>
>  In case of a relogin, a new user object is created in the subject. This causes the subject to lose some state for example authentication method. Instead, a new user object should not be added to the subject if one is already present.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8862-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 05 01:22:21 2010
Return-Path: <common-issues-return-8862-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55515 invoked from network); 5 Jun 2010 01:22:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jun 2010 01:22:21 -0000
Received: (qmail 3846 invoked by uid 500); 5 Jun 2010 01:22:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3808 invoked by uid 500); 5 Jun 2010 01:22:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3800 invoked by uid 99); 5 Jun 2010 01:22:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 01:22:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 01:22:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o551LviG019211
	for <common-issues@hadoop.apache.org>; Sat, 5 Jun 2010 01:21:57 GMT
Message-ID: <29908.194841275700917267.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 21:21:57 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6674) Performance Improvement in Secure
 RPC
In-Reply-To: <375873020.645441270163787224.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875821#action_12875821 ] 

Hudson commented on HADOOP-6674:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #283 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/283/])
    HADOOP-6674. Makes use of the SASL authentication options in the SASL RPC. Contributed by Jitendra Pandey.


> Performance Improvement in Secure RPC
> -------------------------------------
>
>                 Key: HADOOP-6674
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6674
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.patch, HADOOP-6674.1.patch, HADOOP-6674.2.patch
>
>
> This jira introduces two performance improvements in Sasl RPC
> 1. Setting of Sasl 'quality of protection' to authentication only.
> 2. Addition of BufferedOutputStream underneath SaslOutputStream.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8863-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 05 01:22:22 2010
Return-Path: <common-issues-return-8863-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55541 invoked from network); 5 Jun 2010 01:22:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jun 2010 01:22:21 -0000
Received: (qmail 4003 invoked by uid 500); 5 Jun 2010 01:22:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3955 invoked by uid 500); 5 Jun 2010 01:22:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3947 invoked by uid 99); 5 Jun 2010 01:22:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 01:22:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 01:22:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o551LwnN019220
	for <common-issues@hadoop.apache.org>; Sat, 5 Jun 2010 01:21:58 GMT
Message-ID: <8371596.194871275700918118.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 21:21:58 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6810) Extract a subset of tests for smoke
 (DOA) validation
In-Reply-To: <10151977.193211275695163699.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6810:
---------------------------------------

    Attachment: smoke-tests.patch

This version of the patch excludes commit and smoke tests from {{test-core}} to eliminate the duplication. Instead {{test-commit}} and {{test-smoke}} are included to the {{test}} along with {{test-core}}

This patch is for 0.20.3. As 0.20.3 release might not happen I am including this patch in case someone needs it eventually.

> Extract a subset of tests for smoke (DOA) validation
> ----------------------------------------------------
>
>                 Key: HADOOP-6810
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6810
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.20.3
>            Reporter: Konstantin Boudnik
>         Attachments: smoke-tests.patch, smoke-tests.patch
>
>
> Currently, there's no way to quickly find out if a new build of Hadoop is reasonably viable and can be used for future development/testing cycle.
> Similarly to 'commit' testing it'd be great to have a wider subset of existing tests to be executed inside of 30 minutes. The main goal of such a subset is to have a reasonable guarantees that different parts of Hadoop's functionality aren't broken and the system is not DOA.
> Full test run takes about 3+ hours and it is a too long wait time in many cases. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8864-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 05 01:40:21 2010
Return-Path: <common-issues-return-8864-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 59990 invoked from network); 5 Jun 2010 01:40:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jun 2010 01:40:21 -0000
Received: (qmail 10108 invoked by uid 500); 5 Jun 2010 01:40:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 10069 invoked by uid 500); 5 Jun 2010 01:40:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 10061 invoked by uid 99); 5 Jun 2010 01:40:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 01:40:21 +0000
X-ASF-Spam-Status: No, hits=-1499.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 01:40:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o551dx2k019308
	for <common-issues@hadoop.apache.org>; Sat, 5 Jun 2010 01:40:00 GMT
Message-ID: <21192741.195051275701999874.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 21:39:59 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6687) user object in the subject in UGI
  should be reused in case of a relogin.
In-Reply-To: <191053771.27981270592254332.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875827#action_12875827 ] 

Hadoop QA commented on HADOOP-6687:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446391/HADOOP-6687.2.patch
  against trunk revision 951624.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/77/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/77/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/77/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/77/console

This message is automatically generated.

> user object in the subject in UGI  should be reused in case of a relogin.
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6687
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6687
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6687-y20.2.patch, HADOOP-6687.2.patch
>
>
>  In case of a relogin, a new user object is created in the subject. This causes the subject to lose some state for example authentication method. Instead, a new user object should not be added to the subject if one is already present.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8865-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 05 02:08:18 2010
Return-Path: <common-issues-return-8865-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72526 invoked from network); 5 Jun 2010 02:08:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jun 2010 02:08:18 -0000
Received: (qmail 22814 invoked by uid 500); 5 Jun 2010 02:08:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22778 invoked by uid 500); 5 Jun 2010 02:08:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22770 invoked by uid 99); 5 Jun 2010 02:08:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 02:08:18 +0000
X-ASF-Spam-Status: No, hits=-1499.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 02:08:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5527vle019548
	for <common-issues@hadoop.apache.org>; Sat, 5 Jun 2010 02:07:57 GMT
Message-ID: <1521496.195511275703677343.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 22:07:57 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6526) Need mapping from long principal
 names to local OS user names
In-Reply-To: <786866304.151051264870594555.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875838#action_12875838 ] 

Hadoop QA commented on HADOOP-6526:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446384/HADOOP-6526-1.patch
  against trunk revision 951624.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 11 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/565/console

This message is automatically generated.

> Need mapping from long principal names to local OS user names
> -------------------------------------------------------------
>
>                 Key: HADOOP-6526
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6526
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 3595485.patch, c-6526-y20.patch, c-6526-y20.patch, c-6526.patch, HADOOP-6526-1.patch, HADOOP-6526-y20.2.patch, HADOOP-6526-y20.4.patch, HADOOP-6526.patch
>
>
> We need a configurable mapping from full user names (eg. omalley@APACHE.ORG) to local user names (eg. omalley). For many organizations it is sufficient to just use the prefix, however, in the case of shared clusters there may be duplicated prefixes. A configurable mapping will let administrators resolve the issue.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8866-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 05 03:46:16 2010
Return-Path: <common-issues-return-8866-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15430 invoked from network); 5 Jun 2010 03:46:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jun 2010 03:46:16 -0000
Received: (qmail 64378 invoked by uid 500); 5 Jun 2010 03:46:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64257 invoked by uid 500); 5 Jun 2010 03:46:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64249 invoked by uid 99); 5 Jun 2010 03:46:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 03:46:15 +0000
X-ASF-Spam-Status: No, hits=-1499.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 03:46:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o553jsT6019869
	for <common-issues@hadoop.apache.org>; Sat, 5 Jun 2010 03:45:55 GMT
Message-ID: <14612380.195841275709554768.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 23:45:54 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6403) Deprecate EC2 bash scripts
In-Reply-To: <1247543181.1259696120766.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6403:
------------------------------

    Attachment: HADOOP-6403.patch

Simple patch to add a deprecation message.

> Deprecate EC2 bash scripts
> --------------------------
>
>                 Key: HADOOP-6403
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6403
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: contrib/cloud
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6403.patch
>
>
> With the addition of python-based EC2 scripts introduced in HADOOP-6108, the bash scripts in src/contrib/ec2 should be deprecated.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8867-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 05 03:46:19 2010
Return-Path: <common-issues-return-8867-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15460 invoked from network); 5 Jun 2010 03:46:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jun 2010 03:46:19 -0000
Received: (qmail 64441 invoked by uid 500); 5 Jun 2010 03:46:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64402 invoked by uid 500); 5 Jun 2010 03:46:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64394 invoked by uid 99); 5 Jun 2010 03:46:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 03:46:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 03:46:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o553jtTV019875
	for <common-issues@hadoop.apache.org>; Sat, 5 Jun 2010 03:45:55 GMT
Message-ID: <31762841.195861275709555281.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 23:45:55 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6403) Deprecate EC2 bash scripts
In-Reply-To: <1247543181.1259696120766.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6403:
------------------------------

    Status: Patch Available  (was: Open)

> Deprecate EC2 bash scripts
> --------------------------
>
>                 Key: HADOOP-6403
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6403
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: contrib/cloud
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6403.patch
>
>
> With the addition of python-based EC2 scripts introduced in HADOOP-6108, the bash scripts in src/contrib/ec2 should be deprecated.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8868-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 05 03:48:17 2010
Return-Path: <common-issues-return-8868-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15627 invoked from network); 5 Jun 2010 03:48:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jun 2010 03:48:17 -0000
Received: (qmail 64911 invoked by uid 500); 5 Jun 2010 03:48:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64789 invoked by uid 500); 5 Jun 2010 03:48:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64773 invoked by uid 99); 5 Jun 2010 03:48:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 03:48:15 +0000
X-ASF-Spam-Status: No, hits=-1499.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 03:48:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o553lsNP019887
	for <common-issues@hadoop.apache.org>; Sat, 5 Jun 2010 03:47:55 GMT
Message-ID: <12979900.195881275709674948.JavaMail.jira@thor>
Date: Fri, 4 Jun 2010 23:47:54 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6811) Remove EC2 bash scripts
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Remove EC2 bash scripts
-----------------------

                 Key: HADOOP-6811
                 URL: https://issues.apache.org/jira/browse/HADOOP-6811
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Tom White
            Priority: Blocker
             Fix For: 0.22.0


The bash scripts are deprecated in 0.21 (HADOOP-6403) in favour of scripts in Whirr (http://incubator.apache.org/projects/whirr.html). They should be removed in 0.22. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8869-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 05 04:12:16 2010
Return-Path: <common-issues-return-8869-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21945 invoked from network); 5 Jun 2010 04:12:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jun 2010 04:12:16 -0000
Received: (qmail 73488 invoked by uid 500); 5 Jun 2010 04:12:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73372 invoked by uid 500); 5 Jun 2010 04:12:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73362 invoked by uid 99); 5 Jun 2010 04:12:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 04:12:15 +0000
X-ASF-Spam-Status: No, hits=-1499.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 04:12:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o554BshW020434
	for <common-issues@hadoop.apache.org>; Sat, 5 Jun 2010 04:11:54 GMT
Message-ID: <33548879.195911275711114459.JavaMail.jira@thor>
Date: Sat, 5 Jun 2010 00:11:54 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6403) Deprecate EC2 bash scripts
In-Reply-To: <1247543181.1259696120766.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875847#action_12875847 ] 

Hadoop QA commented on HADOOP-6403:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446397/HADOOP-6403.patch
  against trunk revision 951624.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/566/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/566/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/566/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/566/console

This message is automatically generated.

> Deprecate EC2 bash scripts
> --------------------------
>
>                 Key: HADOOP-6403
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6403
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: contrib/cloud
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6403.patch
>
>
> With the addition of python-based EC2 scripts introduced in HADOOP-6108, the bash scripts in src/contrib/ec2 should be deprecated.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8870-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 05 11:26:54 2010
Return-Path: <common-issues-return-8870-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42473 invoked from network); 5 Jun 2010 11:26:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jun 2010 11:26:54 -0000
Received: (qmail 22500 invoked by uid 500); 5 Jun 2010 11:26:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22344 invoked by uid 500); 5 Jun 2010 11:26:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22329 invoked by uid 99); 5 Jun 2010 11:26:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 11:26:50 +0000
X-ASF-Spam-Status: No, hits=-1500.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 11:26:49 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o55BQT74022160
	for <common-issues@hadoop.apache.org>; Sat, 5 Jun 2010 11:26:29 GMT
Message-ID: <14090960.197981275737189139.JavaMail.jira@thor>
Date: Sat, 5 Jun 2010 07:26:29 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6674) Performance Improvement in Secure
 RPC
In-Reply-To: <375873020.645441270163787224.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875893#action_12875893 ] 

Hudson commented on HADOOP-6674:
--------------------------------

Integrated in Hadoop-Common-trunk #357 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/357/])
    HADOOP-6674. Makes use of the SASL authentication options in the SASL RPC. Contributed by Jitendra Pandey.


> Performance Improvement in Secure RPC
> -------------------------------------
>
>                 Key: HADOOP-6674
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6674
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.bugfix.patch, HADOOP-6674-y20.1.patch, HADOOP-6674.1.patch, HADOOP-6674.2.patch
>
>
> This jira introduces two performance improvements in Sasl RPC
> 1. Setting of Sasl 'quality of protection' to authentication only.
> 2. Addition of BufferedOutputStream underneath SaslOutputStream.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8873-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 05 11:26:56 2010
Return-Path: <common-issues-return-8873-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42535 invoked from network); 5 Jun 2010 11:26:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jun 2010 11:26:55 -0000
Received: (qmail 22620 invoked by uid 500); 5 Jun 2010 11:26:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22542 invoked by uid 500); 5 Jun 2010 11:26:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22524 invoked by uid 99); 5 Jun 2010 11:26:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 11:26:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 11:26:52 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o55BQVRn022194
	for <common-issues@hadoop.apache.org>; Sat, 5 Jun 2010 11:26:31 GMT
Message-ID: <12879145.198091275737191214.JavaMail.jira@thor>
Date: Sat, 5 Jun 2010 07:26:31 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6738) Move cluster_setup.xml from
 MapReduce to Common
In-Reply-To: <26228443.22131272581753755.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875896#action_12875896 ] 

Hudson commented on HADOOP-6738:
--------------------------------

Integrated in Hadoop-Common-trunk #357 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/357/])
    HADOOP-6738.  Move cluster_setup.xml, hod_scheduler, commands_manual from MapReduce to Common.


> Move cluster_setup.xml from MapReduce to Common
> -----------------------------------------------
>
>                 Key: HADOOP-6738
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6738
>             Project: Hadoop Common
>          Issue Type: Test
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6738.patch, HADOOP-6738.patch
>
>
> Common half of MAPREDUCE-1404.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8872-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 05 11:26:56 2010
Return-Path: <common-issues-return-8872-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42531 invoked from network); 5 Jun 2010 11:26:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jun 2010 11:26:55 -0000
Received: (qmail 22639 invoked by uid 500); 5 Jun 2010 11:26:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22540 invoked by uid 500); 5 Jun 2010 11:26:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22510 invoked by uid 99); 5 Jun 2010 11:26:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 11:26:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 11:26:52 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o55BQUMA022188
	for <common-issues@hadoop.apache.org>; Sat, 5 Jun 2010 11:26:30 GMT
Message-ID: <505400.198071275737190712.JavaMail.jira@thor>
Date: Sat, 5 Jun 2010 07:26:30 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6780) Move Hadoop cloud scripts to Whirr
In-Reply-To: <9575050.311274374037358.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875895#action_12875895 ] 

Hudson commented on HADOOP-6780:
--------------------------------

Integrated in Hadoop-Common-trunk #357 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/357/])
    HADOOP-6780. Move Hadoop cloud scripts to Whirr.
HADOOP-6780. Move Hadoop cloud scripts to Whirr.


> Move Hadoop cloud scripts to Whirr
> ----------------------------------
>
>                 Key: HADOOP-6780
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6780
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: contrib/cloud
>            Reporter: Tom White
>            Assignee: Tom White
>
> Whirr is a new Apache Incubator project for running cloud services. See http://incubator.apache.org/projects/whirr.html and http://wiki.apache.org/incubator/WhirrProposal.
> The cloud scripts in src/contrib/cloud should move to Whirr. This should be done in 0.21 too, so the scripts do not appear in that release (since having them in Hadoop for a single release would be confusing).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8871-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 05 11:26:56 2010
Return-Path: <common-issues-return-8871-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42537 invoked from network); 5 Jun 2010 11:26:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Jun 2010 11:26:55 -0000
Received: (qmail 22622 invoked by uid 500); 5 Jun 2010 11:26:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22489 invoked by uid 500); 5 Jun 2010 11:26:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22359 invoked by uid 99); 5 Jun 2010 11:26:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 11:26:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 05 Jun 2010 11:26:51 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o55BQTnq022169
	for <common-issues@hadoop.apache.org>; Sat, 5 Jun 2010 11:26:29 GMT
Message-ID: <3181311.198011275737189765.JavaMail.jira@thor>
Date: Sat, 5 Jun 2010 07:26:29 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6649) login object in UGI should be
 inside the subject
In-Reply-To: <1899765654.380491269047307235.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12875894#action_12875894 ] 

Hudson commented on HADOOP-6649:
--------------------------------

Integrated in Hadoop-Common-trunk #357 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/357/])
    HADOOP-6649.  login object in UGI should be inside the subject (jnp via boryas)


> login object in UGI should be inside the subject
> ------------------------------------------------
>
>                 Key: HADOOP-6649
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6649
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6649-y20.1.1.patch, HADOOP-6649-y20.1.patch, HADOOP-6649.1.patch
>
>
> The login object should be moved to the subject in the UGI.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8874-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 06 00:06:21 2010
Return-Path: <common-issues-return-8874-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10653 invoked from network); 6 Jun 2010 00:06:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jun 2010 00:06:21 -0000
Received: (qmail 36811 invoked by uid 500); 6 Jun 2010 00:06:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36736 invoked by uid 500); 6 Jun 2010 00:06:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36728 invoked by uid 99); 6 Jun 2010 00:06:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Jun 2010 00:06:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Jun 2010 00:06:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5605uG9025034
	for <common-issues@hadoop.apache.org>; Sun, 6 Jun 2010 00:05:56 GMT
Message-ID: <7462752.201301275782756466.JavaMail.jira@thor>
Date: Sat, 5 Jun 2010 20:05:56 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6642) Fix javac, javadoc, findbugs warnings
In-Reply-To: <709708654.336271268899107220.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-6642:
----------------------------------

    Attachment: C6642-1y20-debug.patch

Remove debug stmt

> Fix javac, javadoc, findbugs warnings
> -------------------------------------
>
>                 Key: HADOOP-6642
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6642
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Arun C Murthy
>            Assignee: Chris Douglas
>             Fix For: 0.22.0
>
>         Attachments: C6642-0y20.patch, C6642-1y20-debug.patch, C6642-1y20.patch
>
>
> Unfortunately we've missed some javac, javadoc, findbugs warnings in the recent security work.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8875-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 06 00:10:15 2010
Return-Path: <common-issues-return-8875-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10901 invoked from network); 6 Jun 2010 00:10:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jun 2010 00:10:15 -0000
Received: (qmail 37236 invoked by uid 500); 6 Jun 2010 00:10:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37208 invoked by uid 500); 6 Jun 2010 00:10:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37200 invoked by uid 99); 6 Jun 2010 00:10:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Jun 2010 00:10:15 +0000
X-ASF-Spam-Status: No, hits=-1500.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Jun 2010 00:10:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5609sES025057
	for <common-issues@hadoop.apache.org>; Sun, 6 Jun 2010 00:09:55 GMT
Message-ID: <531224.201351275782994612.JavaMail.jira@thor>
Date: Sat, 5 Jun 2010 20:09:54 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5913) Allow administrators to be able to
 start and stop queues
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated HADOOP-5913:
----------------------------------

    Attachment: C5913-14y20s.patch

Backport to 0.20

> Allow administrators to be able to start and stop queues 
> ---------------------------------------------------------
>
>                 Key: HADOOP-5913
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5913
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: rahul k singh
>            Assignee: rahul k singh
>             Fix For: 0.21.0
>
>         Attachments: C5913-14y20s.patch, hadoop-5913-1.patch, hadoop-5913-10.patch, HADOOP-5913-11.patch, HADOOP-5913-12.patch, HADOOP-5913-13.patch, HADOOP-5913-14.patch, hadoop-5913-2.patch, hadoop-5913-3.patch, hadoop-5913-4.patch, hadoop-5913-5.patch, hadoop-5913-6.patch, hadoop-5913-7.patch, hadoop-5913-8.patch, hadoop-5913-9.patch
>
>
>   This feature would provide functionality to stop and start queues in Hadoop at runtime. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8876-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 06 04:36:26 2010
Return-Path: <common-issues-return-8876-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64823 invoked from network); 6 Jun 2010 04:36:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jun 2010 04:36:26 -0000
Received: (qmail 92416 invoked by uid 500); 6 Jun 2010 04:36:26 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92285 invoked by uid 500); 6 Jun 2010 04:36:25 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92272 invoked by uid 99); 6 Jun 2010 04:36:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Jun 2010 04:36:24 +0000
X-ASF-Spam-Status: No, hits=-1500.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Jun 2010 04:36:24 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o564a39d026329
	for <common-issues@hadoop.apache.org>; Sun, 6 Jun 2010 04:36:03 GMT
Message-ID: <12421855.202251275798963698.JavaMail.jira@thor>
Date: Sun, 6 Jun 2010 00:36:03 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6586) Log authentication and authorization
 failures and successes
In-Reply-To: <736883635.426721266819387914.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas resolved HADOOP-6586.
-----------------------------------

    Resolution: Fixed

Setting resolution to fixed, as the patch was committed and not reverted.

> Log authentication and authorization failures and successes
> -----------------------------------------------------------
>
>                 Key: HADOOP-6586
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6586
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6586-3.patch, HADOOP-6586-4.patch, HADOOP-6586-5.patch, HADOOP-6586-7.patch, HADOOP-6586-8-BP20-1.patch, HADOOP-6586-8-BP20.patch, HADOOP-6586-8.patch, HADOOP-6586.patch
>
>
> This jira will cover RPC authentication and SL authorizations logging.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8877-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 06 09:01:20 2010
Return-Path: <common-issues-return-8877-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12347 invoked from network); 6 Jun 2010 09:01:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jun 2010 09:01:19 -0000
Received: (qmail 47568 invoked by uid 500); 6 Jun 2010 09:01:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47469 invoked by uid 500); 6 Jun 2010 09:01:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47461 invoked by uid 99); 6 Jun 2010 09:01:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Jun 2010 09:01:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Jun 2010 09:01:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5690sKh027731
	for <common-issues@hadoop.apache.org>; Sun, 6 Jun 2010 09:00:54 GMT
Message-ID: <21664134.204551275814854626.JavaMail.jira@thor>
Date: Sun, 6 Jun 2010 05:00:54 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Moved: (HADOOP-6812) fs.inmemory.size.mb not listed in conf.
 Cluster setup page gives wrong advice.
In-Reply-To: <1859905315.553281269879147219.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas moved MAPREDUCE-1726 to HADOOP-6812:
--------------------------------------------------

              Project: Hadoop Common  (was: Hadoop Map/Reduce)
                  Key: HADOOP-6812  (was: MAPREDUCE-1726)
    Affects Version/s: 0.20.2
                       0.21.0
                       0.22.0
                           (was: 0.21.0)
                           (was: 0.22.0)
                           (was: 0.20.2)
          Component/s: documentation
                           (was: documentation)

> fs.inmemory.size.mb not listed in conf. Cluster setup page gives wrong advice.
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6812
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6812
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.20.2, 0.21.0, 0.22.0
>            Reporter: Edward Capriolo
>         Attachments: M1726-0.patch, M1726-0v20.patch
>
>
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> fs.inmemory.size.mb does not appear in any xml file
> {noformat}
> grep "fs.inmemory.size.mb" ./mapred/mapred-default.xml 
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./hdfs/hdfs-default.xml 
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./core/core-default.xml 
> {noformat}
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> Documentation error:
> Real-World Cluster Configurations
> {noformat}
> conf/core-site.xml  	io.sort.factor  	100  	More streams merged at once while sorting files.
> conf/core-site.xml 	io.sort.mb 	200 	Higher memory-limit while sorting data.
> {noformat}
> core --- io.sort.factor					-- should be mapred
> core --- io.sort.mb					-- should be mapred

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8878-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 06 09:03:24 2010
Return-Path: <common-issues-return-8878-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12629 invoked from network); 6 Jun 2010 09:03:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jun 2010 09:03:23 -0000
Received: (qmail 47927 invoked by uid 500); 6 Jun 2010 09:03:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47891 invoked by uid 500); 6 Jun 2010 09:03:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47879 invoked by uid 99); 6 Jun 2010 09:03:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Jun 2010 09:03:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Jun 2010 09:03:21 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5692xZ9027752
	for <common-issues@hadoop.apache.org>; Sun, 6 Jun 2010 09:02:59 GMT
Message-ID: <22249125.204601275814979424.JavaMail.jira@thor>
Date: Sun, 6 Jun 2010 05:02:59 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6812) fs.inmemory.size.mb not listed in
 conf. Cluster setup page gives wrong advice.
In-Reply-To: <1859905315.553281269879147219.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876026#action_12876026 ] 

Chris Douglas commented on HADOOP-6812:
---------------------------------------

Moved back to Common after MAPREDUCE-1404

> fs.inmemory.size.mb not listed in conf. Cluster setup page gives wrong advice.
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6812
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6812
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.20.2, 0.21.0, 0.22.0
>            Reporter: Edward Capriolo
>         Attachments: M1726-0.patch, M1726-0v20.patch
>
>
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> fs.inmemory.size.mb does not appear in any xml file
> {noformat}
> grep "fs.inmemory.size.mb" ./mapred/mapred-default.xml 
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./hdfs/hdfs-default.xml 
> [edward@ec src]$ grep "fs.inmemory.size.mb" ./core/core-default.xml 
> {noformat}
> http://hadoop.apache.org/common/docs/current/cluster_setup.html
> Documentation error:
> Real-World Cluster Configurations
> {noformat}
> conf/core-site.xml  	io.sort.factor  	100  	More streams merged at once while sorting files.
> conf/core-site.xml 	io.sort.mb 	200 	Higher memory-limit while sorting data.
> {noformat}
> core --- io.sort.factor					-- should be mapred
> core --- io.sort.mb					-- should be mapred

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8879-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 06 17:06:21 2010
Return-Path: <common-issues-return-8879-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21791 invoked from network); 6 Jun 2010 17:06:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jun 2010 17:06:21 -0000
Received: (qmail 64954 invoked by uid 500); 6 Jun 2010 17:06:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64925 invoked by uid 500); 6 Jun 2010 17:06:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64917 invoked by uid 99); 6 Jun 2010 17:06:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Jun 2010 17:06:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Jun 2010 17:06:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o56H5uEP004803
	for <common-issues@hadoop.apache.org>; Sun, 6 Jun 2010 17:05:56 GMT
Message-ID: <3186593.206951275843956871.JavaMail.jira@thor>
Date: Sun, 6 Jun 2010 13:05:56 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876060#action_12876060 ] 

sam rash commented on HADOOP-6762:
----------------------------------

todd : actually i want open the timeout discussion back up.  I think there might be some confusion.  The 20s timeout that was there was *not* on the RPC, but *only* on sending the RPC.  Basically this is the time to write to the socket (which there might a timeout on the actual out.write call already--i need to double-check).

The RPC itself still would have no timeout.

let me know what you think.

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt, hadoop-6762-7.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8880-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 06 20:27:16 2010
Return-Path: <common-issues-return-8880-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74854 invoked from network); 6 Jun 2010 20:27:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jun 2010 20:27:15 -0000
Received: (qmail 91428 invoked by uid 500); 6 Jun 2010 20:27:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91305 invoked by uid 500); 6 Jun 2010 20:27:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91295 invoked by uid 99); 6 Jun 2010 20:27:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Jun 2010 20:27:15 +0000
X-ASF-Spam-Status: No, hits=-1501.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Jun 2010 20:27:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o56KQrnQ005647
	for <common-issues@hadoop.apache.org>; Sun, 6 Jun 2010 20:26:54 GMT
Message-ID: <19740643.207931275856013804.JavaMail.jira@thor>
Date: Sun, 6 Jun 2010 16:26:53 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876081#action_12876081 ] 

Todd Lipcon commented on HADOOP-6762:
-------------------------------------

bq. btw, is it appropriate to delete the very old patches as they aren't really relevant? or just leave them?

Usually just leave them - makes it easier to follow the conversation and see how the patch evolved.

bq. (which there might a timeout on the actual out.write call already--i need to double-check)

Looks like there is one - a timeout is set when NetUtils.getOutputStream is called on the socket in setupIOStreams(). So I don't think we need a separate timeout in awaiting the send param.


I think the patch is good. I ran it with my test case since Friday night and didn't see any RPC hangs. The test case eventually failed with "Too many open files" but I think it's some other bug/socket leak in DFS code, not IPC. Mark Patch Available to swing it through Hudson?

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt, hadoop-6762-7.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8881-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 06 20:51:16 2010
Return-Path: <common-issues-return-8881-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78417 invoked from network); 6 Jun 2010 20:51:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jun 2010 20:51:16 -0000
Received: (qmail 99122 invoked by uid 500); 6 Jun 2010 20:51:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99102 invoked by uid 500); 6 Jun 2010 20:51:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99094 invoked by uid 99); 6 Jun 2010 20:51:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Jun 2010 20:51:16 +0000
X-ASF-Spam-Status: No, hits=-1501.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Jun 2010 20:51:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o56Kot4f005778
	for <common-issues@hadoop.apache.org>; Sun, 6 Jun 2010 20:50:55 GMT
Message-ID: <21128255.208221275857455371.JavaMail.jira@thor>
Date: Sun, 6 Jun 2010 16:50:55 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6762) exception while doing RPC I/O closes
 channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

sam rash updated HADOOP-6762:
-----------------------------

          Status: Patch Available  (was: Open)
    Hadoop Flags: [Reviewed]

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt, hadoop-6762-7.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8882-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 06 21:21:15 2010
Return-Path: <common-issues-return-8882-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87827 invoked from network); 6 Jun 2010 21:21:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 6 Jun 2010 21:21:15 -0000
Received: (qmail 12933 invoked by uid 500); 6 Jun 2010 21:21:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12898 invoked by uid 500); 6 Jun 2010 21:21:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12890 invoked by uid 99); 6 Jun 2010 21:21:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Jun 2010 21:21:15 +0000
X-ASF-Spam-Status: No, hits=-1501.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Jun 2010 21:21:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o56LKsoV005875
	for <common-issues@hadoop.apache.org>; Sun, 6 Jun 2010 21:20:54 GMT
Message-ID: <2709061.208281275859254305.JavaMail.jira@thor>
Date: Sun, 6 Jun 2010 17:20:54 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876088#action_12876088 ] 

Hadoop QA commented on HADOOP-6762:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446356/hadoop-6762-7.txt
  against trunk revision 951624.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/567/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/567/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/567/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/567/console

This message is automatically generated.

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt, hadoop-6762-7.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8883-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 04:40:16 2010
Return-Path: <common-issues-return-8883-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90477 invoked from network); 7 Jun 2010 04:40:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 04:40:16 -0000
Received: (qmail 263 invoked by uid 500); 7 Jun 2010 04:40:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 141 invoked by uid 500); 7 Jun 2010 04:40:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 133 invoked by uid 99); 7 Jun 2010 04:40:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 04:40:15 +0000
X-ASF-Spam-Status: No, hits=-1501.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 04:40:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o574dsSt008227
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 04:39:54 GMT
Message-ID: <15589974.210231275885594509.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 00:39:54 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6403) Deprecate EC2 bash scripts
In-Reply-To: <1247543181.1259696120766.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876122#action_12876122 ] 

Eli Collins commented on HADOOP-6403:
-------------------------------------

+1  


> Deprecate EC2 bash scripts
> --------------------------
>
>                 Key: HADOOP-6403
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6403
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: contrib/cloud
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6403.patch
>
>
> With the addition of python-based EC2 scripts introduced in HADOOP-6108, the bash scripts in src/contrib/ec2 should be deprecated.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8884-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 05:22:35 2010
Return-Path: <common-issues-return-8884-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99786 invoked from network); 7 Jun 2010 05:22:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 05:22:35 -0000
Received: (qmail 21331 invoked by uid 500); 7 Jun 2010 05:22:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21207 invoked by uid 500); 7 Jun 2010 05:22:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21199 invoked by uid 99); 7 Jun 2010 05:22:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 05:22:34 +0000
X-ASF-Spam-Status: No, hits=-1501.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 05:22:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o575MDec008616
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 05:22:13 GMT
Message-ID: <27500890.210771275888133632.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 01:22:13 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5438) Merge FileSystem.create and
 FileSystem.append
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876129#action_12876129 ] 

Tom White commented on HADOOP-5438:
-----------------------------------

I think this needs more work to find an API that there is broad agreement on. It should be undertaken as a larger piece of work to consolidate the create API (this would include HADOOP-3252 as Jakob pointed out), including deprecating old methods.

This change doesn't enable anything that wasn't previously possible, does it? In other words, backing it out wouldn't remove functionality for users of released versions of Hadoop. For example, append and overwrite operations are possible with the existing API. Since it hasn't been released yet, I'm +1 to reverting this while the create API gets more discussion and testing (especially with HDFS append).


> Merge FileSystem.create and FileSystem.append
> ---------------------------------------------
>
>                 Key: HADOOP-5438
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5438
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: He Yongqiang
>            Assignee: He Yongqiang
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-5438(2009-04-06).patch, Hadoop-5438-2009-03-30.patch, Hadoop-5438-2009-03-31-2.patch, Hadoop-5438-2009-03-31.patch, Hadoop-5438-2009-05-10.patch, Hadoop-5438-2009-05-15.patch, Hadoop-5438-2009-05-19.patch, Hadoop-5438-2009-05-5.patch
>
>
> Currently, when a user wants to modify a file, the user first calls exists() to know if this file is already there. And then uses create() or append() according to whether the file exists or not.
> the code looks like:
> {code}
> FSDataOutputStream out_1 = null;
> if (fs.exists(path_1))
>    out_1 = fs.append(path_1);
> else
>    out_1 = fs.create(path_1);
> {code}
> . On the performace side,It involes two RPCs. On the easy-of-use side, it is not very convient in contrast to the traditional open interface.
> It will more complicate if there is a overwrite parameter specified. I donot know whether there is a bug about 'overwrite' in 0.19, some times it takes a long time for overwrite creates to reture. So i make the write file code with overwrite param works like:
> {code}
> boolean exists = fs.exists(name);
> if (overwrite) {
>     if (exists)
>        fs.delete(name, true);
>      this.out = fs.create(name, overwrite, bufferSize, replication,
> 				    blockSize, progress);
>      this.currentRowID = 0;
>  } else {
>    if (!exists)
> 	this.out = fs.create(name, overwrite, bufferSize,
> 					replication, blockSize, progress);
>    else
> 	this.out = fs.append(name, bufferSize, progress);
> {code}
> Some code statements there are really redundant and not needed, especialy with the delete(). But without deleting first, the overwrite takes a long time to reture.
> BTW, i will create another issue about the overwrite problem. If it is not a bug at all or a duplicate, someone please close it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8885-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 05:34:20 2010
Return-Path: <common-issues-return-8885-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2467 invoked from network); 7 Jun 2010 05:34:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 05:34:20 -0000
Received: (qmail 30144 invoked by uid 500); 7 Jun 2010 05:34:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30075 invoked by uid 500); 7 Jun 2010 05:34:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30067 invoked by uid 99); 7 Jun 2010 05:34:19 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 05:34:19 +0000
X-ASF-Spam-Status: No, hits=-1501.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 05:34:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o575XwrJ008687
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 05:33:58 GMT
Message-ID: <16041764.210881275888838484.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 01:33:58 -0400 (EDT)
From: "He Yongqiang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5438) Merge FileSystem.create and
 FileSystem.append
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876130#action_12876130 ] 

He Yongqiang commented on HADOOP-5438:
--------------------------------------

i am -1 to revert.

> Merge FileSystem.create and FileSystem.append
> ---------------------------------------------
>
>                 Key: HADOOP-5438
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5438
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: He Yongqiang
>            Assignee: He Yongqiang
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-5438(2009-04-06).patch, Hadoop-5438-2009-03-30.patch, Hadoop-5438-2009-03-31-2.patch, Hadoop-5438-2009-03-31.patch, Hadoop-5438-2009-05-10.patch, Hadoop-5438-2009-05-15.patch, Hadoop-5438-2009-05-19.patch, Hadoop-5438-2009-05-5.patch
>
>
> Currently, when a user wants to modify a file, the user first calls exists() to know if this file is already there. And then uses create() or append() according to whether the file exists or not.
> the code looks like:
> {code}
> FSDataOutputStream out_1 = null;
> if (fs.exists(path_1))
>    out_1 = fs.append(path_1);
> else
>    out_1 = fs.create(path_1);
> {code}
> . On the performace side,It involes two RPCs. On the easy-of-use side, it is not very convient in contrast to the traditional open interface.
> It will more complicate if there is a overwrite parameter specified. I donot know whether there is a bug about 'overwrite' in 0.19, some times it takes a long time for overwrite creates to reture. So i make the write file code with overwrite param works like:
> {code}
> boolean exists = fs.exists(name);
> if (overwrite) {
>     if (exists)
>        fs.delete(name, true);
>      this.out = fs.create(name, overwrite, bufferSize, replication,
> 				    blockSize, progress);
>      this.currentRowID = 0;
>  } else {
>    if (!exists)
> 	this.out = fs.create(name, overwrite, bufferSize,
> 					replication, blockSize, progress);
>    else
> 	this.out = fs.append(name, bufferSize, progress);
> {code}
> Some code statements there are really redundant and not needed, especialy with the delete(). But without deleting first, the overwrite takes a long time to reture.
> BTW, i will create another issue about the overwrite problem. If it is not a bug at all or a duplicate, someone please close it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8886-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 06:30:16 2010
Return-Path: <common-issues-return-8886-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19635 invoked from network); 7 Jun 2010 06:30:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 06:30:16 -0000
Received: (qmail 65705 invoked by uid 500); 7 Jun 2010 06:30:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65578 invoked by uid 500); 7 Jun 2010 06:30:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65569 invoked by uid 99); 7 Jun 2010 06:30:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 06:30:15 +0000
X-ASF-Spam-Status: No, hits=-1501.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 06:30:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o576Tsub009007
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 06:29:54 GMT
Message-ID: <23271993.211381275892194689.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 02:29:54 -0400 (EDT)
From: "Soundararajan Velu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6707)
 BlockDecompressorStream.resetState() has a bug, which often causes the
 block decompressing not to work.
In-Reply-To: <6671890.134011271314129577.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876136#action_12876136 ] 

Soundararajan Velu commented on HADOOP-6707:
--------------------------------------------

Ruyue, This works fine for us, Please let me know if you are facing any sepecific issue with not resetting these instance variables... It will be good if you can post the exact issue here. Thanks.

> BlockDecompressorStream.resetState() has a bug, which often causes the block decompressing not to work.
> -------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6707
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6707
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Ruyue Ma
>            Assignee: Ruyue Ma
>             Fix For: 0.21.0
>
>
> BlockDecompressorStream.resetState() has a bug.
> The method of resetState() doesn't reinit the two member variables(int originalBlockSize; int noUncompressedBytes) of Class BlockDecompressorStream.  
> This bug will cause the block decompressing not to work.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8887-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 06:46:16 2010
Return-Path: <common-issues-return-8887-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21881 invoked from network); 7 Jun 2010 06:46:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 06:46:15 -0000
Received: (qmail 79218 invoked by uid 500); 7 Jun 2010 06:46:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79101 invoked by uid 500); 7 Jun 2010 06:46:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79092 invoked by uid 99); 7 Jun 2010 06:46:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 06:46:15 +0000
X-ASF-Spam-Status: No, hits=-1501.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 06:46:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o576jsSG009059
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 06:45:54 GMT
Message-ID: <6692682.211451275893154332.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 02:45:54 -0400 (EDT)
From: "Ruyue Ma (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6707)
 BlockDecompressorStream.resetState() has a bug, which often causes the
 block decompressing not to work.
In-Reply-To: <6671890.134011271314129577.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876138#action_12876138 ] 

Ruyue Ma commented on HADOOP-6707:
----------------------------------

 You have a sequencefile compressed with lzocodec. If you randomly sync(not seek) to a record block, the decompression will not work.

I guess that MapFile maybe have the bug also.

> BlockDecompressorStream.resetState() has a bug, which often causes the block decompressing not to work.
> -------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6707
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6707
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Ruyue Ma
>            Assignee: Ruyue Ma
>             Fix For: 0.21.0
>
>
> BlockDecompressorStream.resetState() has a bug.
> The method of resetState() doesn't reinit the two member variables(int originalBlockSize; int noUncompressedBytes) of Class BlockDecompressorStream.  
> This bug will cause the block decompressing not to work.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8888-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 06:54:22 2010
Return-Path: <common-issues-return-8888-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22417 invoked from network); 7 Jun 2010 06:54:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 06:54:21 -0000
Received: (qmail 84209 invoked by uid 500); 7 Jun 2010 06:54:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84088 invoked by uid 500); 7 Jun 2010 06:54:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84079 invoked by uid 99); 7 Jun 2010 06:54:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 06:54:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 06:54:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o576ruuL009081
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 06:53:56 GMT
Message-ID: <32899742.211471275893636290.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 02:53:56 -0400 (EDT)
From: "Soundararajan Velu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6092) No space left on device
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876139#action_12876139 ] 

Soundararajan Velu commented on HADOOP-6092:
--------------------------------------------

mawanqiang, is this still an issue, RawLocalFileSystem.java is more of a wrapper on Java File I/O, Please let us know if you are still facing this.

> No space left on device
> -----------------------
>
>                 Key: HADOOP-6092
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6092
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.19.0
>         Environment: ubuntu0.8.4
>            Reporter: mawanqiang
>
> Exception in thread "main" org.apache.hadoop.fs.FSError: java.io.IOException: No space left on device
>         at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:199)
>         at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
>         at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
>         at java.io.FilterOutputStream.close(FilterOutputStream.java:140)
>         at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
>         at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
>         at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.close(ChecksumFileSystem.java:339)
>         at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
>         at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
>         at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:825)
>         at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1142)
>         at org.apache.nutch.indexer.Indexer.index(Indexer.java:72)
>         at org.apache.nutch.indexer.Indexer.run(Indexer.java:92)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>         at org.apache.nutch.indexer.Indexer.main(Indexer.java:101)
> Caused by: java.io.IOException: No space left on device
>         at java.io.FileOutputStream.writeBytes(Native Method)
>         at java.io.FileOutputStream.write(FileOutputStream.java:260)
>         at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.write(RawLocalFileSystem.java:197)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8889-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 07:04:22 2010
Return-Path: <common-issues-return-8889-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24309 invoked from network); 7 Jun 2010 07:04:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 07:04:21 -0000
Received: (qmail 95223 invoked by uid 500); 7 Jun 2010 07:04:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95199 invoked by uid 500); 7 Jun 2010 07:04:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95191 invoked by uid 99); 7 Jun 2010 07:04:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 07:04:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 07:04:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5773vZk009209
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 07:03:57 GMT
Message-ID: <11773266.211631275894237287.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 03:03:57 -0400 (EDT)
From: "Ruyue Ma (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6707) BlockDecompressorStream.resetState()
 has a bug, which often causes the block decompressing not to work.
In-Reply-To: <6671890.134011271314129577.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ruyue Ma updated HADOOP-6707:
-----------------------------

    Attachment: hadoop-6707.patch

> BlockDecompressorStream.resetState() has a bug, which often causes the block decompressing not to work.
> -------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6707
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6707
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Ruyue Ma
>            Assignee: Ruyue Ma
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6707.patch
>
>
> BlockDecompressorStream.resetState() has a bug.
> The method of resetState() doesn't reinit the two member variables(int originalBlockSize; int noUncompressedBytes) of Class BlockDecompressorStream.  
> This bug will cause the block decompressing not to work.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8890-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 07:06:23 2010
Return-Path: <common-issues-return-8890-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28630 invoked from network); 7 Jun 2010 07:06:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 07:06:22 -0000
Received: (qmail 97382 invoked by uid 500); 7 Jun 2010 07:06:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97333 invoked by uid 500); 7 Jun 2010 07:06:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97325 invoked by uid 99); 7 Jun 2010 07:06:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 07:06:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 07:06:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5775vWJ009224
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 07:05:58 GMT
Message-ID: <16865110.211681275894357908.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 03:05:57 -0400 (EDT)
From: "Ruyue Ma (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6707) BlockDecompressorStream.resetState()
 has a bug, which often causes the block decompressing not to work.
In-Reply-To: <6671890.134011271314129577.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ruyue Ma updated HADOOP-6707:
-----------------------------

    Status: Patch Available  (was: Open)

> BlockDecompressorStream.resetState() has a bug, which often causes the block decompressing not to work.
> -------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6707
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6707
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Ruyue Ma
>            Assignee: Ruyue Ma
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6707.patch
>
>
> BlockDecompressorStream.resetState() has a bug.
> The method of resetState() doesn't reinit the two member variables(int originalBlockSize; int noUncompressedBytes) of Class BlockDecompressorStream.  
> This bug will cause the block decompressing not to work.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8891-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 07:26:23 2010
Return-Path: <common-issues-return-8891-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31157 invoked from network); 7 Jun 2010 07:26:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 07:26:23 -0000
Received: (qmail 11580 invoked by uid 500); 7 Jun 2010 07:26:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11456 invoked by uid 500); 7 Jun 2010 07:26:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11448 invoked by uid 99); 7 Jun 2010 07:26:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 07:26:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 07:26:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o577PvR3009420
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 07:25:57 GMT
Message-ID: <14152765.211951275895557688.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 03:25:57 -0400 (EDT)
From: "Soundararajan Velu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6718) Client does not close connection
 when an exception happens during SASL negotiation
In-Reply-To: <15471232.100441271806549356.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876144#action_12876144 ] 

Soundararajan Velu commented on HADOOP-6718:
--------------------------------------------

Devaraj, Patch looks great, but shouldn't we catch Exception instead of IOException, Just in case... and then handle the situation.

> Client does not close connection when an exception happens during SASL negotiation
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6718
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6718
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: 6718-bp20.patch
>
>
> setupSaslConnection in the RPC client might fail to successfully set up a sasl connection (e.g. if the principal is wrongly configured). It throws an exception back to the caller (setupIOstreams). setupIOstreams marks the connection as closed but it doesn't really close the socket. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8892-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 07:32:23 2010
Return-Path: <common-issues-return-8892-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32024 invoked from network); 7 Jun 2010 07:32:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 07:32:23 -0000
Received: (qmail 16704 invoked by uid 500); 7 Jun 2010 07:32:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16586 invoked by uid 500); 7 Jun 2010 07:32:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16578 invoked by uid 99); 7 Jun 2010 07:32:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 07:32:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 07:32:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o577VwWl009493
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 07:31:59 GMT
Message-ID: <24619417.212021275895918290.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 03:31:58 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6707)
 BlockDecompressorStream.resetState() has a bug, which often causes the
 block decompressing not to work.
In-Reply-To: <6671890.134011271314129577.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876146#action_12876146 ] 

Hadoop QA commented on HADOOP-6707:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446461/hadoop-6707.patch
  against trunk revision 951624.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/568/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/568/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/568/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/568/console

This message is automatically generated.

> BlockDecompressorStream.resetState() has a bug, which often causes the block decompressing not to work.
> -------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6707
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6707
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Ruyue Ma
>            Assignee: Ruyue Ma
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6707.patch
>
>
> BlockDecompressorStream.resetState() has a bug.
> The method of resetState() doesn't reinit the two member variables(int originalBlockSize; int noUncompressedBytes) of Class BlockDecompressorStream.  
> This bug will cause the block decompressing not to work.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8893-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 11:29:04 2010
Return-Path: <common-issues-return-8893-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95046 invoked from network); 7 Jun 2010 11:29:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 11:29:03 -0000
Received: (qmail 28226 invoked by uid 500); 7 Jun 2010 11:29:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28094 invoked by uid 500); 7 Jun 2010 11:29:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28081 invoked by uid 99); 7 Jun 2010 11:29:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 11:29:00 +0000
X-ASF-Spam-Status: No, hits=-1502.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 11:28:59 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57BSdxa011787
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 11:28:39 GMT
Message-ID: <15550623.311275910119004.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 07:28:39 -0400 (EDT)
From: "Uma Mahesh (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876205#action_12876205 ] 

Uma Mahesh commented on HADOOP-6762:
------------------------------------

I applied the patch and ran this test continuously.
Seems still race condition is occuring randomly.
That means this patch is not working. :-(

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt, hadoop-6762-7.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8894-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 15:16:02 2010
Return-Path: <common-issues-return-8894-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80859 invoked from network); 7 Jun 2010 15:16:02 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 15:16:02 -0000
Received: (qmail 22999 invoked by uid 500); 7 Jun 2010 15:16:02 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22973 invoked by uid 500); 7 Jun 2010 15:16:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22965 invoked by uid 99); 7 Jun 2010 15:16:01 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 15:16:01 +0000
X-ASF-Spam-Status: No, hits=-1503.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 15:16:00 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57FFdfC013643
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 15:15:40 GMT
Message-ID: <29679536.2891275923739155.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 11:15:39 -0400 (EDT)
From: "Gianmarco De Francisci Morales (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6758) MapFile.fix does not allow index
 interval definition
In-Reply-To: <14740497.57291273425048889.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876272#action_12876272 ] 

Gianmarco De Francisci Morales commented on HADOOP-6758:
--------------------------------------------------------

Sorry to ask, but how do I try to resubmit my patch?

> MapFile.fix does not allow index interval definition
> ----------------------------------------------------
>
>                 Key: HADOOP-6758
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6758
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Gianmarco De Francisci Morales
>         Attachments: HADOOP-6758.patch, HADOOP-6758.patch, HADOOP-6758.patch
>
>
> When using the static methond MapFile.fix() there is no way to override the default IndexInterval that is 128.
> The IndexInterval should be taken from the configuration that is passed to the method.
> {code}
> int indexInterval = 128; 
> indexInterval = conf.getInt(INDEX_INTERVAL, indexInterval); 
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8895-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 16:58:08 2010
Return-Path: <common-issues-return-8895-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22953 invoked from network); 7 Jun 2010 16:58:08 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 16:58:08 -0000
Received: (qmail 51469 invoked by uid 500); 7 Jun 2010 16:58:08 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51384 invoked by uid 500); 7 Jun 2010 16:58:08 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51374 invoked by uid 99); 7 Jun 2010 16:58:08 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 16:58:08 +0000
X-ASF-Spam-Status: No, hits=-1503.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 16:58:07 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57Gvlh9014641
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 16:57:47 GMT
Message-ID: <17272614.4341275929867144.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 12:57:47 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876311#action_12876311 ] 

sam rash commented on HADOOP-6762:
----------------------------------

can you elaborate on what you ran and what error messages + exceptions you saw in the test logs?  the thread that does the IO cannot receive an interrupt so if you are seeing an error, i don't think it can be the same one.


> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt, hadoop-6762-7.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8896-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 17:02:23 2010
Return-Path: <common-issues-return-8896-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23938 invoked from network); 7 Jun 2010 17:02:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 17:02:23 -0000
Received: (qmail 53811 invoked by uid 500); 7 Jun 2010 17:02:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53763 invoked by uid 500); 7 Jun 2010 17:02:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53755 invoked by uid 99); 7 Jun 2010 17:02:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 17:02:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 17:02:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57H1x1o014873
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 17:01:59 GMT
Message-ID: <12208921.4661275930119031.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 13:01:59 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6332) Large-scale Automated Test
 Framework
In-Reply-To: <36506918.1256452199424.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876312#action_12876312 ] 

Konstantin Boudnik commented on HADOOP-6332:
--------------------------------------------

Adding a reference to a scripting framework discussed in past (HADOOP-6248)

> Large-scale Automated Test Framework
> ------------------------------------
>
>                 Key: HADOOP-6332
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6332
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Arun C Murthy
>            Assignee: Konstantin Boudnik
>             Fix For: 0.21.0
>
>         Attachments: 6332-phase2.fix1.patch, 6332-phase2.fix2.patch, 6332-phase2.patch, 6332.patch, 6332.patch, 6332.patch, 6332_v1.patch, 6332_v2.patch, HADOOP-6332-MR.patch, HADOOP-6332-MR.patch, HADOOP-6332.0.22.patch, HADOOP-6332.0.22.patch, HADOOP-6332.0.22.patch, HADOOP-6332.0.22.patch, HADOOP-6332.0.22.patch, HADOOP-6332.0.22.patch, HADOOP-6332.0.22.patch, HADOOP-6332.0.22.patch, HADOOP-6332.patch, HADOOP-6332.patch
>
>
> Hadoop would benefit from having a large-scale, automated, test-framework. This jira is meant to be a master-jira to track relevant work.
> ----
> The proposal is a junit-based, large-scale test framework which would run against _real_ clusters.
> There are several pieces we need to achieve this goal:
> # A set of utilities we can use in junit-based tests to work with real, large-scale hadoop clusters. E.g. utilities to bring up to deploy, start & stop clusters, bring down tasktrackers, datanodes, entire racks of both etc.
> # Enhanced control-ability and inspect-ability of the various components in the system e.g. daemons such as namenode, jobtracker should expose their data-structures for query/manipulation etc. Tests would be much more relevant if we could for e.g. query for specific states of the jobtracker, scheduler etc. Clearly these apis should _not_ be part of the production clusters - hence the proposal is to use aspectj to weave these new apis to debug-deployments.
> ----
> Related note: we should break up our tests into at least 3 categories:
> # src/test/unit -> Real unit tests using mock objects (e.g. HDFS-669 & MAPREDUCE-1050).
> # src/test/integration -> Current junit tests with Mini* clusters etc.
> # src/test/system -> HADOOP-6332 and it's children

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8897-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 17:12:20 2010
Return-Path: <common-issues-return-8897-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30950 invoked from network); 7 Jun 2010 17:12:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 17:12:20 -0000
Received: (qmail 62840 invoked by uid 500); 7 Jun 2010 17:12:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62787 invoked by uid 500); 7 Jun 2010 17:12:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62779 invoked by uid 99); 7 Jun 2010 17:12:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 17:12:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 17:12:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57HBuGF014972
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 17:11:56 GMT
Message-ID: <4336412.4871275930716074.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 13:11:56 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5438) Merge FileSystem.create and
 FileSystem.append
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876317#action_12876317 ] 

Hairong Kuang commented on HADOOP-5438:
---------------------------------------

> Here is not about append and create. it is also create and overwrite. 
Reverting the patch does not remove the "overwrite" functionality. If we only need to support create and overwrite, a boolean flag is good enough as we have pre this patch. I do not see the need of introducing an enum.

Yongqiang, could you please explain if there is anything beyond which makes you against the idea of reverting this patch?

> Merge FileSystem.create and FileSystem.append
> ---------------------------------------------
>
>                 Key: HADOOP-5438
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5438
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: He Yongqiang
>            Assignee: He Yongqiang
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-5438(2009-04-06).patch, Hadoop-5438-2009-03-30.patch, Hadoop-5438-2009-03-31-2.patch, Hadoop-5438-2009-03-31.patch, Hadoop-5438-2009-05-10.patch, Hadoop-5438-2009-05-15.patch, Hadoop-5438-2009-05-19.patch, Hadoop-5438-2009-05-5.patch
>
>
> Currently, when a user wants to modify a file, the user first calls exists() to know if this file is already there. And then uses create() or append() according to whether the file exists or not.
> the code looks like:
> {code}
> FSDataOutputStream out_1 = null;
> if (fs.exists(path_1))
>    out_1 = fs.append(path_1);
> else
>    out_1 = fs.create(path_1);
> {code}
> . On the performace side,It involes two RPCs. On the easy-of-use side, it is not very convient in contrast to the traditional open interface.
> It will more complicate if there is a overwrite parameter specified. I donot know whether there is a bug about 'overwrite' in 0.19, some times it takes a long time for overwrite creates to reture. So i make the write file code with overwrite param works like:
> {code}
> boolean exists = fs.exists(name);
> if (overwrite) {
>     if (exists)
>        fs.delete(name, true);
>      this.out = fs.create(name, overwrite, bufferSize, replication,
> 				    blockSize, progress);
>      this.currentRowID = 0;
>  } else {
>    if (!exists)
> 	this.out = fs.create(name, overwrite, bufferSize,
> 					replication, blockSize, progress);
>    else
> 	this.out = fs.append(name, bufferSize, progress);
> {code}
> Some code statements there are really redundant and not needed, especialy with the delete(). But without deleting first, the overwrite takes a long time to reture.
> BTW, i will create another issue about the overwrite problem. If it is not a bug at all or a duplicate, someone please close it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8898-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 17:30:09 2010
Return-Path: <common-issues-return-8898-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36573 invoked from network); 7 Jun 2010 17:30:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 17:30:09 -0000
Received: (qmail 76186 invoked by uid 500); 7 Jun 2010 17:30:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76160 invoked by uid 500); 7 Jun 2010 17:30:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76149 invoked by uid 99); 7 Jun 2010 17:30:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 17:30:09 +0000
X-ASF-Spam-Status: No, hits=-1503.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 17:30:09 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57HTmCI015116
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 17:29:48 GMT
Message-ID: <25035370.5181275931788657.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 13:29:48 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876322#action_12876322 ] 

sam rash commented on HADOOP-6762:
----------------------------------

Actually, this might be an artifact of the test.  The threads in the executor's threadpool will be interrupted on shutdown.  the test doesn't wait for all of the threads to terminate gracefully before doing so.  This means there is a race in the test for rpc threads to finish before getting interrupted.  I have an updated test that uses a latch to wait for all threads to terminate before letting the test complete. I haven't been able to repro, but can post the patch this afternoon and let you try.

please do still post what exception and any relevant looking log parts here.  it will help me ascertain if this is the case

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt, hadoop-6762-7.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8899-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 17:42:30 2010
Return-Path: <common-issues-return-8899-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39266 invoked from network); 7 Jun 2010 17:42:29 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 17:42:29 -0000
Received: (qmail 87274 invoked by uid 500); 7 Jun 2010 17:42:29 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87234 invoked by uid 500); 7 Jun 2010 17:42:29 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87226 invoked by uid 99); 7 Jun 2010 17:42:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 17:42:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 17:42:27 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57Hg6Ui015265
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 17:42:06 GMT
Message-ID: <9239013.5481275932525979.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 13:42:05 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5438) Merge FileSystem.create and
 FileSystem.append
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876329#action_12876329 ] 

Jakob Homan commented on HADOOP-5438:
-------------------------------------

I would rather get this right than have to support an incorrect, deprecated version for two releases.  +1 on revert.

> Merge FileSystem.create and FileSystem.append
> ---------------------------------------------
>
>                 Key: HADOOP-5438
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5438
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: He Yongqiang
>            Assignee: He Yongqiang
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-5438(2009-04-06).patch, Hadoop-5438-2009-03-30.patch, Hadoop-5438-2009-03-31-2.patch, Hadoop-5438-2009-03-31.patch, Hadoop-5438-2009-05-10.patch, Hadoop-5438-2009-05-15.patch, Hadoop-5438-2009-05-19.patch, Hadoop-5438-2009-05-5.patch
>
>
> Currently, when a user wants to modify a file, the user first calls exists() to know if this file is already there. And then uses create() or append() according to whether the file exists or not.
> the code looks like:
> {code}
> FSDataOutputStream out_1 = null;
> if (fs.exists(path_1))
>    out_1 = fs.append(path_1);
> else
>    out_1 = fs.create(path_1);
> {code}
> . On the performace side,It involes two RPCs. On the easy-of-use side, it is not very convient in contrast to the traditional open interface.
> It will more complicate if there is a overwrite parameter specified. I donot know whether there is a bug about 'overwrite' in 0.19, some times it takes a long time for overwrite creates to reture. So i make the write file code with overwrite param works like:
> {code}
> boolean exists = fs.exists(name);
> if (overwrite) {
>     if (exists)
>        fs.delete(name, true);
>      this.out = fs.create(name, overwrite, bufferSize, replication,
> 				    blockSize, progress);
>      this.currentRowID = 0;
>  } else {
>    if (!exists)
> 	this.out = fs.create(name, overwrite, bufferSize,
> 					replication, blockSize, progress);
>    else
> 	this.out = fs.append(name, bufferSize, progress);
> {code}
> Some code statements there are really redundant and not needed, especialy with the delete(). But without deleting first, the overwrite takes a long time to reture.
> BTW, i will create another issue about the overwrite problem. If it is not a bug at all or a duplicate, someone please close it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8900-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 17:50:13 2010
Return-Path: <common-issues-return-8900-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41482 invoked from network); 7 Jun 2010 17:50:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 17:50:13 -0000
Received: (qmail 95486 invoked by uid 500); 7 Jun 2010 17:50:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95435 invoked by uid 500); 7 Jun 2010 17:50:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95427 invoked by uid 99); 7 Jun 2010 17:50:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 17:50:13 +0000
X-ASF-Spam-Status: No, hits=-1503.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 17:50:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57HnqrP015422
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 17:49:52 GMT
Message-ID: <3629970.5911275932992236.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 13:49:52 -0400 (EDT)
From: "Michele Catasta (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5793) High speed compression algorithm
 like BMDiff
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-5793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876332#action_12876332 ] 

Michele Catasta commented on HADOOP-5793:
-----------------------------------------

@Luke: great to see you here - I was planning to bug you personally and ask if I did some blatant mistakes while integrating your library!

Regarding composition: as far as I understood, BMZ supports bm_pack/unpack + LZO out of the box - so we're not crossing the JNI chasm; copies and allocations should be as few as possible. I think your point was that it would nice to support also other external libraries apart from LZO. +1 from me. 
There are already a few other Hadoop issues about FastLZ and other codecs. I'd say that I'm gonna wait a while to see what the community prefers.

 w.r.t. the code updates, are they available somewhere? If you like, I can integrate them. Otherwise, feel free to fork my github repo.

Stability wise, I compressed and shuffled around a few TBs on Hadoop, plus I've a couple of HBase tables using BMZ since a few weeks. So far, everything worked smoothly with no hiccups. I hope someone will test the lib soon and let us know.

> High speed compression algorithm like BMDiff
> --------------------------------------------
>
>                 Key: HADOOP-5793
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5793
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: elhoim gibor
>            Assignee: Michele Catasta
>            Priority: Minor
>
> Add a high speed compression algorithm like BMDiff.
> It gives speeds ~100MB/s for writes and ~1000MB/s for reads, compressing 2.1billions web pages from 45.1TB in 4.2TB
> Reference:
> http://norfolk.cs.washington.edu/htbin-post/unrestricted/colloq/details.cgi?id=437
> 2005 Jeff Dean talk about google architecture - around 46:00.
> http://feedblog.org/2008/10/12/google-bigtable-compression-zippy-and-bmdiff/
> http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=755678
> A reference implementation exists in HyperTable.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8901-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 18:04:06 2010
Return-Path: <common-issues-return-8901-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45762 invoked from network); 7 Jun 2010 18:04:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 18:04:05 -0000
Received: (qmail 17845 invoked by uid 500); 7 Jun 2010 18:04:05 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17812 invoked by uid 500); 7 Jun 2010 18:04:05 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17804 invoked by uid 99); 7 Jun 2010 18:04:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 18:04:05 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 18:04:03 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57I3f6C015713
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 18:03:41 GMT
Message-ID: <31274127.6421275933821329.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 14:03:41 -0400 (EDT)
From: "He Yongqiang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5438) Merge FileSystem.create and
 FileSystem.append
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876342#action_12876342 ] 

He Yongqiang commented on HADOOP-5438:
--------------------------------------

1) this is used in two of my previous projects, and get deployed. i don't want to go back to change things (actually i have no right to do that now).
2) i did not see any benefit here.

> Merge FileSystem.create and FileSystem.append
> ---------------------------------------------
>
>                 Key: HADOOP-5438
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5438
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: He Yongqiang
>            Assignee: He Yongqiang
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-5438(2009-04-06).patch, Hadoop-5438-2009-03-30.patch, Hadoop-5438-2009-03-31-2.patch, Hadoop-5438-2009-03-31.patch, Hadoop-5438-2009-05-10.patch, Hadoop-5438-2009-05-15.patch, Hadoop-5438-2009-05-19.patch, Hadoop-5438-2009-05-5.patch
>
>
> Currently, when a user wants to modify a file, the user first calls exists() to know if this file is already there. And then uses create() or append() according to whether the file exists or not.
> the code looks like:
> {code}
> FSDataOutputStream out_1 = null;
> if (fs.exists(path_1))
>    out_1 = fs.append(path_1);
> else
>    out_1 = fs.create(path_1);
> {code}
> . On the performace side,It involes two RPCs. On the easy-of-use side, it is not very convient in contrast to the traditional open interface.
> It will more complicate if there is a overwrite parameter specified. I donot know whether there is a bug about 'overwrite' in 0.19, some times it takes a long time for overwrite creates to reture. So i make the write file code with overwrite param works like:
> {code}
> boolean exists = fs.exists(name);
> if (overwrite) {
>     if (exists)
>        fs.delete(name, true);
>      this.out = fs.create(name, overwrite, bufferSize, replication,
> 				    blockSize, progress);
>      this.currentRowID = 0;
>  } else {
>    if (!exists)
> 	this.out = fs.create(name, overwrite, bufferSize,
> 					replication, blockSize, progress);
>    else
> 	this.out = fs.append(name, bufferSize, progress);
> {code}
> Some code statements there are really redundant and not needed, especialy with the delete(). But without deleting first, the overwrite takes a long time to reture.
> BTW, i will create another issue about the overwrite problem. If it is not a bug at all or a duplicate, someone please close it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8902-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 18:04:07 2010
Return-Path: <common-issues-return-8902-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45807 invoked from network); 7 Jun 2010 18:04:06 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 18:04:06 -0000
Received: (qmail 17985 invoked by uid 500); 7 Jun 2010 18:04:06 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17955 invoked by uid 500); 7 Jun 2010 18:04:06 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17947 invoked by uid 99); 7 Jun 2010 18:04:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 18:04:06 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 18:04:04 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57I3h1F015740
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 18:03:43 GMT
Message-ID: <6128373.6511275933823185.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 14:03:43 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6526) Need mapping from long principal
 names to local OS user names
In-Reply-To: <786866304.151051264870594555.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6526:
-----------------------------------

    Attachment: HADOOP-6526-2.patch

rebased with the current trunk

> Need mapping from long principal names to local OS user names
> -------------------------------------------------------------
>
>                 Key: HADOOP-6526
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6526
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 3595485.patch, c-6526-y20.patch, c-6526-y20.patch, c-6526.patch, HADOOP-6526-1.patch, HADOOP-6526-2.patch, HADOOP-6526-y20.2.patch, HADOOP-6526-y20.4.patch, HADOOP-6526.patch
>
>
> We need a configurable mapping from full user names (eg. omalley@APACHE.ORG) to local user names (eg. omalley). For many organizations it is sufficient to just use the prefix, however, in the case of shared clusters there may be duplicated prefixes. A configurable mapping will let administrators resolve the issue.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8903-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 18:06:03 2010
Return-Path: <common-issues-return-8903-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50028 invoked from network); 7 Jun 2010 18:06:03 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 18:06:03 -0000
Received: (qmail 20885 invoked by uid 500); 7 Jun 2010 18:06:03 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20863 invoked by uid 500); 7 Jun 2010 18:06:03 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20851 invoked by uid 99); 7 Jun 2010 18:06:03 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 18:06:03 +0000
X-ASF-Spam-Status: No, hits=-1503.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 18:06:02 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57I5gLr015772
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 18:05:42 GMT
Message-ID: <17924229.6601275933942051.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 14:05:42 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6526) Need mapping from long principal
 names to local OS user names
In-Reply-To: <786866304.151051264870594555.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6526:
-----------------------------------

    Status: Open  (was: Patch Available)

> Need mapping from long principal names to local OS user names
> -------------------------------------------------------------
>
>                 Key: HADOOP-6526
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6526
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 3595485.patch, c-6526-y20.patch, c-6526-y20.patch, c-6526.patch, HADOOP-6526-1.patch, HADOOP-6526-2.patch, HADOOP-6526-y20.2.patch, HADOOP-6526-y20.4.patch, HADOOP-6526.patch
>
>
> We need a configurable mapping from full user names (eg. omalley@APACHE.ORG) to local user names (eg. omalley). For many organizations it is sufficient to just use the prefix, however, in the case of shared clusters there may be duplicated prefixes. A configurable mapping will let administrators resolve the issue.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8904-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 18:06:07 2010
Return-Path: <common-issues-return-8904-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50092 invoked from network); 7 Jun 2010 18:06:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 18:06:07 -0000
Received: (qmail 21156 invoked by uid 500); 7 Jun 2010 18:06:07 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21128 invoked by uid 500); 7 Jun 2010 18:06:07 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21120 invoked by uid 99); 7 Jun 2010 18:06:06 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 18:06:06 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 18:06:04 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57I5hOv015799
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 18:05:43 GMT
Message-ID: <10207802.6691275933943509.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 14:05:43 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6526) Need mapping from long principal
 names to local OS user names
In-Reply-To: <786866304.151051264870594555.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6526:
-----------------------------------

    Status: Patch Available  (was: Open)

> Need mapping from long principal names to local OS user names
> -------------------------------------------------------------
>
>                 Key: HADOOP-6526
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6526
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 3595485.patch, c-6526-y20.patch, c-6526-y20.patch, c-6526.patch, HADOOP-6526-1.patch, HADOOP-6526-2.patch, HADOOP-6526-y20.2.patch, HADOOP-6526-y20.4.patch, HADOOP-6526.patch
>
>
> We need a configurable mapping from full user names (eg. omalley@APACHE.ORG) to local user names (eg. omalley). For many organizations it is sufficient to just use the prefix, however, in the case of shared clusters there may be duplicated prefixes. A configurable mapping will let administrators resolve the issue.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8905-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 18:31:21 2010
Return-Path: <common-issues-return-8905-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66087 invoked from network); 7 Jun 2010 18:31:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 18:31:21 -0000
Received: (qmail 47225 invoked by uid 500); 7 Jun 2010 18:31:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47107 invoked by uid 500); 7 Jun 2010 18:31:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47099 invoked by uid 99); 7 Jun 2010 18:31:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 18:31:20 +0000
X-ASF-Spam-Status: No, hits=-1503.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 18:31:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57IV0wm016103
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 18:31:00 GMT
Message-ID: <12167690.7421275935460215.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 14:31:00 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6526) Need mapping from long principal
 names to local OS user names
In-Reply-To: <786866304.151051264870594555.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876355#action_12876355 ] 

Hadoop QA commented on HADOOP-6526:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446507/HADOOP-6526-2.patch
  against trunk revision 951624.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 11 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    -1 javac.  The applied patch generated 1022 javac compiler warnings (more than the trunk's current 1017 warnings).

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/569/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/569/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/569/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/569/console

This message is automatically generated.

> Need mapping from long principal names to local OS user names
> -------------------------------------------------------------
>
>                 Key: HADOOP-6526
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6526
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 3595485.patch, c-6526-y20.patch, c-6526-y20.patch, c-6526.patch, HADOOP-6526-1.patch, HADOOP-6526-2.patch, HADOOP-6526-y20.2.patch, HADOOP-6526-y20.4.patch, HADOOP-6526.patch
>
>
> We need a configurable mapping from full user names (eg. omalley@APACHE.ORG) to local user names (eg. omalley). For many organizations it is sufficient to just use the prefix, however, in the case of shared clusters there may be duplicated prefixes. A configurable mapping will let administrators resolve the issue.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8906-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 18:53:04 2010
Return-Path: <common-issues-return-8906-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74124 invoked from network); 7 Jun 2010 18:53:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 18:53:04 -0000
Received: (qmail 72543 invoked by uid 500); 7 Jun 2010 18:53:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72503 invoked by uid 500); 7 Jun 2010 18:53:04 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72495 invoked by uid 99); 7 Jun 2010 18:53:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 18:53:04 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 18:53:01 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57Iqd4B016266
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 18:52:40 GMT
Message-ID: <20936064.7591275936759915.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 14:52:39 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6687) user object in the subject in UGI
  should be reused in case of a relogin.
In-Reply-To: <191053771.27981270592254332.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6687:
-----------------------------------------

    Status: Open  (was: Patch Available)

> user object in the subject in UGI  should be reused in case of a relogin.
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6687
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6687
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6687-y20.2.patch, HADOOP-6687.2.patch, HADOOP-6687.3.patch
>
>
>  In case of a relogin, a new user object is created in the subject. This causes the subject to lose some state for example authentication method. Instead, a new user object should not be added to the subject if one is already present.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8907-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 18:53:05 2010
Return-Path: <common-issues-return-8907-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74148 invoked from network); 7 Jun 2010 18:53:04 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 18:53:04 -0000
Received: (qmail 72695 invoked by uid 500); 7 Jun 2010 18:53:04 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72646 invoked by uid 500); 7 Jun 2010 18:53:04 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72519 invoked by uid 99); 7 Jun 2010 18:53:04 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 18:53:04 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 18:53:01 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57Iqe3i016272
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 18:52:40 GMT
Message-ID: <12470442.7611275936760532.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 14:52:40 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6687) user object in the subject in UGI
  should be reused in case of a relogin.
In-Reply-To: <191053771.27981270592254332.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6687:
-----------------------------------------

    Attachment: HADOOP-6687.3.patch

> user object in the subject in UGI  should be reused in case of a relogin.
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6687
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6687
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6687-y20.2.patch, HADOOP-6687.2.patch, HADOOP-6687.3.patch
>
>
>  In case of a relogin, a new user object is created in the subject. This causes the subject to lose some state for example authentication method. Instead, a new user object should not be added to the subject if one is already present.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8908-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 18:53:05 2010
Return-Path: <common-issues-return-8908-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74235 invoked from network); 7 Jun 2010 18:53:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 18:53:05 -0000
Received: (qmail 72829 invoked by uid 500); 7 Jun 2010 18:53:05 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72797 invoked by uid 500); 7 Jun 2010 18:53:05 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72789 invoked by uid 99); 7 Jun 2010 18:53:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 18:53:05 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 18:53:03 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57IqfFp016287
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 18:52:41 GMT
Message-ID: <3075268.7661275936761486.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 14:52:41 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6687) user object in the subject in UGI
  should be reused in case of a relogin.
In-Reply-To: <191053771.27981270592254332.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6687:
-----------------------------------------

    Status: Patch Available  (was: Open)

> user object in the subject in UGI  should be reused in case of a relogin.
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6687
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6687
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6687-y20.2.patch, HADOOP-6687.2.patch, HADOOP-6687.3.patch
>
>
>  In case of a relogin, a new user object is created in the subject. This causes the subject to lose some state for example authentication method. Instead, a new user object should not be added to the subject if one is already present.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8909-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 19:21:05 2010
Return-Path: <common-issues-return-8909-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88630 invoked from network); 7 Jun 2010 19:21:05 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 19:21:05 -0000
Received: (qmail 5125 invoked by uid 500); 7 Jun 2010 19:21:05 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5092 invoked by uid 500); 7 Jun 2010 19:21:05 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5084 invoked by uid 99); 7 Jun 2010 19:21:05 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 19:21:05 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 19:21:03 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57JKft5016829
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 19:20:41 GMT
Message-ID: <1135924.8691275938441643.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 15:20:41 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6687) user object in the subject in UGI
  should be reused in case of a relogin.
In-Reply-To: <191053771.27981270592254332.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876373#action_12876373 ] 

Hadoop QA commented on HADOOP-6687:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446509/HADOOP-6687.3.patch
  against trunk revision 951624.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/570/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/570/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/570/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/570/console

This message is automatically generated.

> user object in the subject in UGI  should be reused in case of a relogin.
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6687
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6687
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6687-y20.2.patch, HADOOP-6687.2.patch, HADOOP-6687.3.patch
>
>
>  In case of a relogin, a new user object is created in the subject. This causes the subject to lose some state for example authentication method. Instead, a new user object should not be added to the subject if one is already present.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8910-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 20:17:13 2010
Return-Path: <common-issues-return-8910-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15252 invoked from network); 7 Jun 2010 20:17:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 20:17:13 -0000
Received: (qmail 69006 invoked by uid 500); 7 Jun 2010 20:17:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68966 invoked by uid 500); 7 Jun 2010 20:17:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68953 invoked by uid 99); 7 Jun 2010 20:17:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 20:17:13 +0000
X-ASF-Spam-Status: No, hits=-1503.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 20:17:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57KGqE1017559
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 20:16:52 GMT
Message-ID: <5852792.10051275941812320.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 16:16:52 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5793) High speed compression algorithm
 like BMDiff
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-5793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876390#action_12876390 ] 

Todd Lipcon commented on HADOOP-5793:
-------------------------------------

Luke: did you implement this cleanroom in Hypertable? Do you hold the entire copyright thereof? If so, would you consider relicensing the code as Apache 2 license for use in Hadoop?

It would be great to have something like this that is Apache 2.0 licensed so we can freely distribute it with the rest of Hadoop, rather than making people separately compile and install a dependency.

> High speed compression algorithm like BMDiff
> --------------------------------------------
>
>                 Key: HADOOP-5793
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5793
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: elhoim gibor
>            Assignee: Michele Catasta
>            Priority: Minor
>
> Add a high speed compression algorithm like BMDiff.
> It gives speeds ~100MB/s for writes and ~1000MB/s for reads, compressing 2.1billions web pages from 45.1TB in 4.2TB
> Reference:
> http://norfolk.cs.washington.edu/htbin-post/unrestricted/colloq/details.cgi?id=437
> 2005 Jeff Dean talk about google architecture - around 46:00.
> http://feedblog.org/2008/10/12/google-bigtable-compression-zippy-and-bmdiff/
> http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=755678
> A reference implementation exists in HyperTable.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8911-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 20:25:14 2010
Return-Path: <common-issues-return-8911-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 21225 invoked from network); 7 Jun 2010 20:25:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 20:25:14 -0000
Received: (qmail 82338 invoked by uid 500); 7 Jun 2010 20:25:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82214 invoked by uid 500); 7 Jun 2010 20:25:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82206 invoked by uid 99); 7 Jun 2010 20:25:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 20:25:14 +0000
X-ASF-Spam-Status: No, hits=-1503.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 20:25:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57KOrMQ017698
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 20:24:53 GMT
Message-ID: <28493451.10431275942293068.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 16:24:53 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5438) Merge FileSystem.create and
 FileSystem.append
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-5438:
-------------------------------------------

    Hadoop Flags: [Incompatible change]

This changed ClientProtocol, making as an incompatible change.

> Merge FileSystem.create and FileSystem.append
> ---------------------------------------------
>
>                 Key: HADOOP-5438
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5438
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: He Yongqiang
>            Assignee: He Yongqiang
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-5438(2009-04-06).patch, Hadoop-5438-2009-03-30.patch, Hadoop-5438-2009-03-31-2.patch, Hadoop-5438-2009-03-31.patch, Hadoop-5438-2009-05-10.patch, Hadoop-5438-2009-05-15.patch, Hadoop-5438-2009-05-19.patch, Hadoop-5438-2009-05-5.patch
>
>
> Currently, when a user wants to modify a file, the user first calls exists() to know if this file is already there. And then uses create() or append() according to whether the file exists or not.
> the code looks like:
> {code}
> FSDataOutputStream out_1 = null;
> if (fs.exists(path_1))
>    out_1 = fs.append(path_1);
> else
>    out_1 = fs.create(path_1);
> {code}
> . On the performace side,It involes two RPCs. On the easy-of-use side, it is not very convient in contrast to the traditional open interface.
> It will more complicate if there is a overwrite parameter specified. I donot know whether there is a bug about 'overwrite' in 0.19, some times it takes a long time for overwrite creates to reture. So i make the write file code with overwrite param works like:
> {code}
> boolean exists = fs.exists(name);
> if (overwrite) {
>     if (exists)
>        fs.delete(name, true);
>      this.out = fs.create(name, overwrite, bufferSize, replication,
> 				    blockSize, progress);
>      this.currentRowID = 0;
>  } else {
>    if (!exists)
> 	this.out = fs.create(name, overwrite, bufferSize,
> 					replication, blockSize, progress);
>    else
> 	this.out = fs.append(name, bufferSize, progress);
> {code}
> Some code statements there are really redundant and not needed, especialy with the delete(). But without deleting first, the overwrite takes a long time to reture.
> BTW, i will create another issue about the overwrite problem. If it is not a bug at all or a duplicate, someone please close it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8912-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 20:33:13 2010
Return-Path: <common-issues-return-8912-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24976 invoked from network); 7 Jun 2010 20:33:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 20:33:13 -0000
Received: (qmail 93735 invoked by uid 500); 7 Jun 2010 20:33:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93703 invoked by uid 500); 7 Jun 2010 20:33:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93695 invoked by uid 99); 7 Jun 2010 20:33:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 20:33:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 20:33:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57KWnRp017790
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 20:32:49 GMT
Message-ID: <32837369.10631275942769122.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 16:32:49 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5438) Merge FileSystem.create and
 FileSystem.append
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876401#action_12876401 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-5438:
------------------------------------------------

> 1) this is used in two of my previous projects, and get deployed. i don't want to go back to change things (actually i have no right to do that now).

I am sorry that your projects have already used it.  However, your projects are using a development version, which is supposed to be unstable.

More importantly, if this is not a good API, we definitely don't want to put it on a release.  Otherwise, it is much harder to fix the problem.

Since this and the related patches were committed for more than a year, I thinks it is impossible to revert it.  I suggest we stop the discussion here and discuss this on a new issue in order to fix the problems.

> Merge FileSystem.create and FileSystem.append
> ---------------------------------------------
>
>                 Key: HADOOP-5438
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5438
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: He Yongqiang
>            Assignee: He Yongqiang
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-5438(2009-04-06).patch, Hadoop-5438-2009-03-30.patch, Hadoop-5438-2009-03-31-2.patch, Hadoop-5438-2009-03-31.patch, Hadoop-5438-2009-05-10.patch, Hadoop-5438-2009-05-15.patch, Hadoop-5438-2009-05-19.patch, Hadoop-5438-2009-05-5.patch
>
>
> Currently, when a user wants to modify a file, the user first calls exists() to know if this file is already there. And then uses create() or append() according to whether the file exists or not.
> the code looks like:
> {code}
> FSDataOutputStream out_1 = null;
> if (fs.exists(path_1))
>    out_1 = fs.append(path_1);
> else
>    out_1 = fs.create(path_1);
> {code}
> . On the performace side,It involes two RPCs. On the easy-of-use side, it is not very convient in contrast to the traditional open interface.
> It will more complicate if there is a overwrite parameter specified. I donot know whether there is a bug about 'overwrite' in 0.19, some times it takes a long time for overwrite creates to reture. So i make the write file code with overwrite param works like:
> {code}
> boolean exists = fs.exists(name);
> if (overwrite) {
>     if (exists)
>        fs.delete(name, true);
>      this.out = fs.create(name, overwrite, bufferSize, replication,
> 				    blockSize, progress);
>      this.currentRowID = 0;
>  } else {
>    if (!exists)
> 	this.out = fs.create(name, overwrite, bufferSize,
> 					replication, blockSize, progress);
>    else
> 	this.out = fs.append(name, bufferSize, progress);
> {code}
> Some code statements there are really redundant and not needed, especialy with the delete(). But without deleting first, the overwrite takes a long time to reture.
> BTW, i will create another issue about the overwrite problem. If it is not a bug at all or a duplicate, someone please close it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8913-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 21:42:09 2010
Return-Path: <common-issues-return-8913-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51284 invoked from network); 7 Jun 2010 21:42:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 21:42:09 -0000
Received: (qmail 67672 invoked by uid 500); 7 Jun 2010 21:42:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67644 invoked by uid 500); 7 Jun 2010 21:42:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67635 invoked by uid 99); 7 Jun 2010 21:42:09 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 21:42:09 +0000
X-ASF-Spam-Status: No, hits=-1503.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 21:42:08 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57LfmwJ018851
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 21:41:48 GMT
Message-ID: <30818219.12881275946908209.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 17:41:48 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876433#action_12876433 ] 

sam rash commented on HADOOP-6762:
----------------------------------

turns out this is just a really bad case of logging.  the exception was a BrokenBarrierException and I wasn't logging the real exception.  changed that and fixed the test so this doesn't occur


> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt, hadoop-6762-7.txt, hadoop-6762-8.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8914-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 21:42:12 2010
Return-Path: <common-issues-return-8914-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51363 invoked from network); 7 Jun 2010 21:42:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 21:42:12 -0000
Received: (qmail 68050 invoked by uid 500); 7 Jun 2010 21:42:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68012 invoked by uid 500); 7 Jun 2010 21:42:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68004 invoked by uid 99); 7 Jun 2010 21:42:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 21:42:12 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 21:42:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57Lfns1018863
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 21:41:49 GMT
Message-ID: <15278511.12921275946909059.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 17:41:49 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6762) exception while doing RPC I/O closes
 channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

sam rash updated HADOOP-6762:
-----------------------------

    Attachment: hadoop-6762-8.txt

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt, hadoop-6762-7.txt, hadoop-6762-8.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8915-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 21:46:37 2010
Return-Path: <common-issues-return-8915-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52347 invoked from network); 7 Jun 2010 21:46:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 21:46:37 -0000
Received: (qmail 72681 invoked by uid 500); 7 Jun 2010 21:46:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72649 invoked by uid 500); 7 Jun 2010 21:46:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72641 invoked by uid 99); 7 Jun 2010 21:46:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 21:46:37 +0000
X-ASF-Spam-Status: No, hits=-1504.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 21:46:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57LkG8f018921
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 21:46:16 GMT
Message-ID: <29303796.13071275947176327.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 17:46:16 -0400 (EDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5438) Merge FileSystem.create and
 FileSystem.append
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876435#action_12876435 ] 

Hairong Kuang commented on HADOOP-5438:
---------------------------------------

Sure, let's continue discussion at HDFS-609. 

> Merge FileSystem.create and FileSystem.append
> ---------------------------------------------
>
>                 Key: HADOOP-5438
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5438
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: He Yongqiang
>            Assignee: He Yongqiang
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-5438(2009-04-06).patch, Hadoop-5438-2009-03-30.patch, Hadoop-5438-2009-03-31-2.patch, Hadoop-5438-2009-03-31.patch, Hadoop-5438-2009-05-10.patch, Hadoop-5438-2009-05-15.patch, Hadoop-5438-2009-05-19.patch, Hadoop-5438-2009-05-5.patch
>
>
> Currently, when a user wants to modify a file, the user first calls exists() to know if this file is already there. And then uses create() or append() according to whether the file exists or not.
> the code looks like:
> {code}
> FSDataOutputStream out_1 = null;
> if (fs.exists(path_1))
>    out_1 = fs.append(path_1);
> else
>    out_1 = fs.create(path_1);
> {code}
> . On the performace side,It involes two RPCs. On the easy-of-use side, it is not very convient in contrast to the traditional open interface.
> It will more complicate if there is a overwrite parameter specified. I donot know whether there is a bug about 'overwrite' in 0.19, some times it takes a long time for overwrite creates to reture. So i make the write file code with overwrite param works like:
> {code}
> boolean exists = fs.exists(name);
> if (overwrite) {
>     if (exists)
>        fs.delete(name, true);
>      this.out = fs.create(name, overwrite, bufferSize, replication,
> 				    blockSize, progress);
>      this.currentRowID = 0;
>  } else {
>    if (!exists)
> 	this.out = fs.create(name, overwrite, bufferSize,
> 					replication, blockSize, progress);
>    else
> 	this.out = fs.append(name, bufferSize, progress);
> {code}
> Some code statements there are really redundant and not needed, especialy with the delete(). But without deleting first, the overwrite takes a long time to reture.
> BTW, i will create another issue about the overwrite problem. If it is not a bug at all or a duplicate, someone please close it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8916-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 21:50:39 2010
Return-Path: <common-issues-return-8916-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53609 invoked from network); 7 Jun 2010 21:50:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 21:50:39 -0000
Received: (qmail 80826 invoked by uid 500); 7 Jun 2010 21:50:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80803 invoked by uid 500); 7 Jun 2010 21:50:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80795 invoked by uid 99); 7 Jun 2010 21:50:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 21:50:38 +0000
X-ASF-Spam-Status: No, hits=-1504.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 21:50:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57LoICo019005
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 21:50:18 GMT
Message-ID: <30859941.13291275947418192.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 17:50:18 -0400 (EDT)
From: "Luke Lu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5793) High speed compression algorithm
 like BMDiff
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-5793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876436#action_12876436 ] 

Luke Lu commented on HADOOP-5793:
---------------------------------

It's written from scratch based on the original BM paper to scratch an itch, which turned out to be stable and fast enough. Zvents owned the copyright for this particular version, however, and I don't know the status of the transfer of the copyright to Hypertable inc. OTOH, a shorter and faster version can be written from scratch again if my current employer (Yahoo) is feeling even more generous than she already is :)

> High speed compression algorithm like BMDiff
> --------------------------------------------
>
>                 Key: HADOOP-5793
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5793
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: elhoim gibor
>            Assignee: Michele Catasta
>            Priority: Minor
>
> Add a high speed compression algorithm like BMDiff.
> It gives speeds ~100MB/s for writes and ~1000MB/s for reads, compressing 2.1billions web pages from 45.1TB in 4.2TB
> Reference:
> http://norfolk.cs.washington.edu/htbin-post/unrestricted/colloq/details.cgi?id=437
> 2005 Jeff Dean talk about google architecture - around 46:00.
> http://feedblog.org/2008/10/12/google-bigtable-compression-zippy-and-bmdiff/
> http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=755678
> A reference implementation exists in HyperTable.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8917-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 21:52:33 2010
Return-Path: <common-issues-return-8917-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54829 invoked from network); 7 Jun 2010 21:52:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 21:52:33 -0000
Received: (qmail 83309 invoked by uid 500); 7 Jun 2010 21:52:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83189 invoked by uid 500); 7 Jun 2010 21:52:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83180 invoked by uid 99); 7 Jun 2010 21:52:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 21:52:33 +0000
X-ASF-Spam-Status: No, hits=-1504.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 21:52:33 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57LqCdP019040
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 21:52:13 GMT
Message-ID: <3392237.13331275947532974.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 17:52:12 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6687) user object in the subject in UGI
  should be reused in case of a relogin.
In-Reply-To: <191053771.27981270592254332.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876438#action_12876438 ] 

Boris Shkolnik commented on HADOOP-6687:
----------------------------------------

thanks Jitendra.

> user object in the subject in UGI  should be reused in case of a relogin.
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6687
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6687
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6687-y20.2.patch, HADOOP-6687.2.patch, HADOOP-6687.3.patch
>
>
>  In case of a relogin, a new user object is created in the subject. This causes the subject to lose some state for example authentication method. Instead, a new user object should not be added to the subject if one is already present.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8918-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 21:52:36 2010
Return-Path: <common-issues-return-8918-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54875 invoked from network); 7 Jun 2010 21:52:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 21:52:36 -0000
Received: (qmail 83383 invoked by uid 500); 7 Jun 2010 21:52:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83352 invoked by uid 500); 7 Jun 2010 21:52:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83344 invoked by uid 99); 7 Jun 2010 21:52:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 21:52:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 21:52:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57LqCN7019034
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 21:52:12 GMT
Message-ID: <22707531.13311275947532477.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 17:52:12 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6687) user object in the subject in UGI
  should be reused in case of a relogin.
In-Reply-To: <191053771.27981270592254332.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6687:
-----------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

+1
committed to trunk

> user object in the subject in UGI  should be reused in case of a relogin.
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6687
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6687
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6687-y20.2.patch, HADOOP-6687.2.patch, HADOOP-6687.3.patch
>
>
>  In case of a relogin, a new user object is created in the subject. This causes the subject to lose some state for example authentication method. Instead, a new user object should not be added to the subject if one is already present.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8919-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 21:52:40 2010
Return-Path: <common-issues-return-8919-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54944 invoked from network); 7 Jun 2010 21:52:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 21:52:40 -0000
Received: (qmail 83554 invoked by uid 500); 7 Jun 2010 21:52:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83520 invoked by uid 500); 7 Jun 2010 21:52:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83512 invoked by uid 99); 7 Jun 2010 21:52:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 21:52:40 +0000
X-ASF-Spam-Status: No, hits=-1504.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 21:52:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57LqJUY019100
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 21:52:19 GMT
Message-ID: <17138053.13531275947539655.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 17:52:19 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5793) High speed compression algorithm
 like BMDiff
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-5793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876439#action_12876439 ] 

Todd Lipcon commented on HADOOP-5793:
-------------------------------------

haha, ok :) Mind reporting back here if you figure out about whether it could become Apache? This would be very useful in HBase.

> High speed compression algorithm like BMDiff
> --------------------------------------------
>
>                 Key: HADOOP-5793
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5793
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: elhoim gibor
>            Assignee: Michele Catasta
>            Priority: Minor
>
> Add a high speed compression algorithm like BMDiff.
> It gives speeds ~100MB/s for writes and ~1000MB/s for reads, compressing 2.1billions web pages from 45.1TB in 4.2TB
> Reference:
> http://norfolk.cs.washington.edu/htbin-post/unrestricted/colloq/details.cgi?id=437
> 2005 Jeff Dean talk about google architecture - around 46:00.
> http://feedblog.org/2008/10/12/google-bigtable-compression-zippy-and-bmdiff/
> http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=755678
> A reference implementation exists in HyperTable.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8920-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 22:06:33 2010
Return-Path: <common-issues-return-8920-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61281 invoked from network); 7 Jun 2010 22:06:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 22:06:33 -0000
Received: (qmail 98431 invoked by uid 500); 7 Jun 2010 22:06:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98398 invoked by uid 500); 7 Jun 2010 22:06:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98390 invoked by uid 99); 7 Jun 2010 22:06:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 22:06:32 +0000
X-ASF-Spam-Status: No, hits=-1504.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 22:06:32 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57M6BrJ019274
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 22:06:12 GMT
Message-ID: <17176494.13701275948371739.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 18:06:11 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6813) Add a new newInstance method in
 FileSystem that takes a "user" as argument
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Add a new newInstance method in FileSystem that takes a "user" as argument
--------------------------------------------------------------------------

                 Key: HADOOP-6813
                 URL: https://issues.apache.org/jira/browse/HADOOP-6813
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs
            Reporter: Devaraj Das
            Assignee: Devaraj Das
             Fix For: 0.22.0


In order to implement HDFS-1000 for trunk, I need to have a newInstance API in FileSystem that takes a "user" as an argument.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8921-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 22:08:34 2010
Return-Path: <common-issues-return-8921-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61555 invoked from network); 7 Jun 2010 22:08:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 22:08:33 -0000
Received: (qmail 99969 invoked by uid 500); 7 Jun 2010 22:08:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99943 invoked by uid 500); 7 Jun 2010 22:08:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99935 invoked by uid 99); 7 Jun 2010 22:08:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 22:08:33 +0000
X-ASF-Spam-Status: No, hits=-1504.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 22:08:33 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57M8Cxv019307
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 22:08:12 GMT
Message-ID: <26298880.13751275948492865.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 18:08:12 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6813) Add a new newInstance method in
 FileSystem that takes a "user" as argument
In-Reply-To: <17176494.13701275948371739.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6813:
--------------------------------

    Attachment: h-6813.patch

Straightforward patch. 

> Add a new newInstance method in FileSystem that takes a "user" as argument
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6813
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6813
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: h-6813.patch
>
>
> In order to implement HDFS-1000 for trunk, I need to have a newInstance API in FileSystem that takes a "user" as an argument.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8922-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 22:08:36 2010
Return-Path: <common-issues-return-8922-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61626 invoked from network); 7 Jun 2010 22:08:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 22:08:36 -0000
Received: (qmail 478 invoked by uid 500); 7 Jun 2010 22:08:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 441 invoked by uid 500); 7 Jun 2010 22:08:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 433 invoked by uid 99); 7 Jun 2010 22:08:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 22:08:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 22:08:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57M8DRH019313
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 22:08:13 GMT
Message-ID: <22860454.13771275948493220.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 18:08:13 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6813) Add a new newInstance method in
 FileSystem that takes a "user" as argument
In-Reply-To: <17176494.13701275948371739.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6813:
--------------------------------

    Status: Patch Available  (was: Open)

> Add a new newInstance method in FileSystem that takes a "user" as argument
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6813
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6813
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: h-6813.patch
>
>
> In order to implement HDFS-1000 for trunk, I need to have a newInstance API in FileSystem that takes a "user" as an argument.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8923-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 22:26:35 2010
Return-Path: <common-issues-return-8923-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69013 invoked from network); 7 Jun 2010 22:26:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 22:26:34 -0000
Received: (qmail 27466 invoked by uid 500); 7 Jun 2010 22:26:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27417 invoked by uid 500); 7 Jun 2010 22:26:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27409 invoked by uid 99); 7 Jun 2010 22:26:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 22:26:34 +0000
X-ASF-Spam-Status: No, hits=-1504.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 22:26:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57MQDK6019520
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 22:26:14 GMT
Message-ID: <19988052.14201275949573621.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 18:26:13 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6687) user object in the subject in UGI
  should be reused in case of a relogin.
In-Reply-To: <191053771.27981270592254332.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876455#action_12876455 ] 

Hudson commented on HADOOP-6687:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #284 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/284/])
    HADOOP-6687.  user object in the subject in UGI should be reused in case of a relogin.


> user object in the subject in UGI  should be reused in case of a relogin.
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6687
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6687
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6687-y20.2.patch, HADOOP-6687.2.patch, HADOOP-6687.3.patch
>
>
>  In case of a relogin, a new user object is created in the subject. This causes the subject to lose some state for example authentication method. Instead, a new user object should not be added to the subject if one is already present.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8924-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 22:29:36 2010
Return-Path: <common-issues-return-8924-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69624 invoked from network); 7 Jun 2010 22:29:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 22:29:36 -0000
Received: (qmail 30380 invoked by uid 500); 7 Jun 2010 22:29:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30347 invoked by uid 500); 7 Jun 2010 22:29:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30339 invoked by uid 99); 7 Jun 2010 22:29:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 22:29:36 +0000
X-ASF-Spam-Status: No, hits=-1504.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 22:29:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57MTFkv019573
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 22:29:15 GMT
Message-ID: <12542907.14341275949755007.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 18:29:15 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6756) Clean up and add documentation for
 configuration keys in CommonConfigurationKeys.java
In-Reply-To: <23380183.20521273191468124.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876461#action_12876461 ] 

Erik Steffl commented on HADOOP-6756:
-------------------------------------

Re https://issues.apache.org/jira/browse/HADOOP-6756?focusedCommentId=12875770&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12875770

1, 2: some fields do not have reasonable defaults therefore are commented out and are without default values (I was told that empty values are not handled well by Configuration), if you can think of reasonable defaults for these please let me know

3: will fix

4: reasonable, will do

5: Let's discuss off-line and come up with something that makes sense, it's my best guess from looking at code

6: This is a bit tricky, there's core-default.html which is built by ant docs (not ant javadocs) and resulting core-default.html is outside of javadoc docRoot. It can be referred to using "{@docRoot}/../core-default.html" (note the parent directory of docRoot). This makes the end result ok for now but it's dangerous because it makes javadoc docs rely on docs built outside of javadoc.

7: This comment was there before, thought it's out of scope for this bug, i.e. I am trying to document existing variables, not decide which ones are used or not etc.

> Clean up and add documentation for configuration keys in CommonConfigurationKeys.java
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6756
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6756
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.3, 0.22.0
>            Reporter: Erik Steffl
>             Fix For: 0.20.3, 0.22.0
>
>         Attachments: HADOOP-6756-0.22.0.patch, jira.HADOOP-6756-0.20-1-FS_DEFAULT_NAME_KEY.patch, jira.HADOOP-6756-0.20-1.patch, jira.HADOOP-6756-0.20.patch
>
>
> Configuration keys in CommonConfigurationKeys.java should be cleaned up and documented (javadoc comments, appropriate *-default.xml descriptions).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8925-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 22:33:36 2010
Return-Path: <common-issues-return-8925-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70767 invoked from network); 7 Jun 2010 22:33:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 22:33:36 -0000
Received: (qmail 36983 invoked by uid 500); 7 Jun 2010 22:33:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36948 invoked by uid 500); 7 Jun 2010 22:33:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36940 invoked by uid 99); 7 Jun 2010 22:33:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 22:33:35 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 22:33:33 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57MXBuM019602
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 22:33:12 GMT
Message-ID: <898489.14401275949991499.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 18:33:11 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6813) Add a new newInstance method in
 FileSystem that takes a "user" as argument
In-Reply-To: <17176494.13701275948371739.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876463#action_12876463 ] 

Hadoop QA commented on HADOOP-6813:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446533/h-6813.patch
  against trunk revision 952454.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/571/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/571/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/571/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/571/console

This message is automatically generated.

> Add a new newInstance method in FileSystem that takes a "user" as argument
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6813
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6813
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: h-6813.patch
>
>
> In order to implement HDFS-1000 for trunk, I need to have a newInstance API in FileSystem that takes a "user" as an argument.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8926-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 22:50:33 2010
Return-Path: <common-issues-return-8926-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73899 invoked from network); 7 Jun 2010 22:50:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 22:50:33 -0000
Received: (qmail 50437 invoked by uid 500); 7 Jun 2010 22:50:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50328 invoked by uid 500); 7 Jun 2010 22:50:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50320 invoked by uid 99); 7 Jun 2010 22:50:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 22:50:32 +0000
X-ASF-Spam-Status: No, hits=-1504.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 22:50:32 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57MoBI7019728
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 22:50:12 GMT
Message-ID: <19239921.14611275951011902.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 18:50:11 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6810) Extract a subset of tests for smoke
 (DOA) validation
In-Reply-To: <10151977.193211275695163699.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6810:
---------------------------------------

    Attachment: smoke-tests.patch

Now the list of smoke tests is formed pretty much to its final shape. Ready for a review, I suppose.

> Extract a subset of tests for smoke (DOA) validation
> ----------------------------------------------------
>
>                 Key: HADOOP-6810
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6810
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.20.3
>            Reporter: Konstantin Boudnik
>         Attachments: smoke-tests.patch, smoke-tests.patch, smoke-tests.patch
>
>
> Currently, there's no way to quickly find out if a new build of Hadoop is reasonably viable and can be used for future development/testing cycle.
> Similarly to 'commit' testing it'd be great to have a wider subset of existing tests to be executed inside of 30 minutes. The main goal of such a subset is to have a reasonable guarantees that different parts of Hadoop's functionality aren't broken and the system is not DOA.
> Full test run takes about 3+ hours and it is a too long wait time in many cases. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8927-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 22:59:35 2010
Return-Path: <common-issues-return-8927-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75475 invoked from network); 7 Jun 2010 22:59:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 22:59:35 -0000
Received: (qmail 64662 invoked by uid 500); 7 Jun 2010 22:59:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64616 invoked by uid 500); 7 Jun 2010 22:59:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64608 invoked by uid 99); 7 Jun 2010 22:59:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 22:59:35 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 22:59:33 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57MxBA8019798
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 22:59:11 GMT
Message-ID: <8789915.14731275951551658.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 18:59:11 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6813) Add a new newInstance method in
 FileSystem that takes a "user" as argument
In-Reply-To: <17176494.13701275948371739.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6813:
-----------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

+1
committed to trunk. Thanks Devaraj.

> Add a new newInstance method in FileSystem that takes a "user" as argument
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6813
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6813
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: h-6813.patch
>
>
> In order to implement HDFS-1000 for trunk, I need to have a newInstance API in FileSystem that takes a "user" as an argument.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8928-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 23:02:33 2010
Return-Path: <common-issues-return-8928-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76671 invoked from network); 7 Jun 2010 23:02:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 23:02:33 -0000
Received: (qmail 69379 invoked by uid 500); 7 Jun 2010 23:02:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69304 invoked by uid 500); 7 Jun 2010 23:02:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69295 invoked by uid 99); 7 Jun 2010 23:02:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 23:02:33 +0000
X-ASF-Spam-Status: No, hits=-1504.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 23:02:32 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57N2CsQ019878
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 23:02:12 GMT
Message-ID: <29484910.14751275951732068.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 19:02:12 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6814) Method in UGI to fet the
 authentication method of the real user.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Method in UGI to fet the authentication method of the real user.
----------------------------------------------------------------

                 Key: HADOOP-6814
                 URL: https://issues.apache.org/jira/browse/HADOOP-6814
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Jitendra Nath Pandey
            Assignee: Jitendra Nath Pandey


UGI should have a method to return the authentication method of the real user for a proxy-user scenario.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8929-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 23:02:34 2010
Return-Path: <common-issues-return-8929-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76734 invoked from network); 7 Jun 2010 23:02:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 23:02:34 -0000
Received: (qmail 69500 invoked by uid 500); 7 Jun 2010 23:02:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69454 invoked by uid 500); 7 Jun 2010 23:02:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69446 invoked by uid 99); 7 Jun 2010 23:02:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 23:02:34 +0000
X-ASF-Spam-Status: No, hits=-1504.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 23:02:33 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57N2DKq019899
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 23:02:13 GMT
Message-ID: <15204292.14821275951733765.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 19:02:13 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6810) Extract a subset of tests for smoke
 (DOA) validation
In-Reply-To: <10151977.193211275695163699.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6810:
---------------------------------------

    Attachment: smoke-tests.patch

Adding comments to the test list to separate test specific to common from hdfs

> Extract a subset of tests for smoke (DOA) validation
> ----------------------------------------------------
>
>                 Key: HADOOP-6810
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6810
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.20.3
>            Reporter: Konstantin Boudnik
>         Attachments: smoke-tests.patch, smoke-tests.patch, smoke-tests.patch, smoke-tests.patch
>
>
> Currently, there's no way to quickly find out if a new build of Hadoop is reasonably viable and can be used for future development/testing cycle.
> Similarly to 'commit' testing it'd be great to have a wider subset of existing tests to be executed inside of 30 minutes. The main goal of such a subset is to have a reasonable guarantees that different parts of Hadoop's functionality aren't broken and the system is not DOA.
> Full test run takes about 3+ hours and it is a too long wait time in many cases. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8930-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 23:17:32 2010
Return-Path: <common-issues-return-8930-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83008 invoked from network); 7 Jun 2010 23:17:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 23:17:32 -0000
Received: (qmail 78633 invoked by uid 500); 7 Jun 2010 23:17:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78544 invoked by uid 500); 7 Jun 2010 23:17:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78536 invoked by uid 99); 7 Jun 2010 23:17:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 23:17:32 +0000
X-ASF-Spam-Status: No, hits=-1504.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 23:17:32 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57NHCtF020000
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 23:17:12 GMT
Message-ID: <33404160.15011275952632277.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 19:17:12 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6813) Add a new newInstance method in
 FileSystem that takes a "user" as argument
In-Reply-To: <17176494.13701275948371739.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876475#action_12876475 ] 

Hudson commented on HADOOP-6813:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #285 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/285/])
    HADOOP-6813. Add a new newInstance method in FileSystem that takes a user as argument


> Add a new newInstance method in FileSystem that takes a "user" as argument
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6813
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6813
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: h-6813.patch
>
>
> In order to implement HDFS-1000 for trunk, I need to have a newInstance API in FileSystem that takes a "user" as an argument.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8931-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 23:23:36 2010
Return-Path: <common-issues-return-8931-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83822 invoked from network); 7 Jun 2010 23:23:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 23:23:36 -0000
Received: (qmail 83370 invoked by uid 500); 7 Jun 2010 23:23:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83326 invoked by uid 500); 7 Jun 2010 23:23:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83318 invoked by uid 99); 7 Jun 2010 23:23:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 23:23:35 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 23:23:33 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57NNBlQ020058
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 23:23:12 GMT
Message-ID: <471364.15161275952991959.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 19:23:11 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6814) Method in UGI to fet the
 authentication method of the real user.
In-Reply-To: <29484910.14751275951732068.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6814:
-----------------------------------------

    Attachment: HADOOP-6814.1.patch

> Method in UGI to fet the authentication method of the real user.
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6814
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6814
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6814.1.patch
>
>
> UGI should have a method to return the authentication method of the real user for a proxy-user scenario.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8932-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 23:23:36 2010
Return-Path: <common-issues-return-8932-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83850 invoked from network); 7 Jun 2010 23:23:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 23:23:36 -0000
Received: (qmail 83503 invoked by uid 500); 7 Jun 2010 23:23:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83469 invoked by uid 500); 7 Jun 2010 23:23:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83422 invoked by uid 99); 7 Jun 2010 23:23:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 23:23:35 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 23:23:33 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57NNCnH020064
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 23:23:12 GMT
Message-ID: <11536687.15181275952992474.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 19:23:12 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6814) Method in UGI to get the
 authentication method of the real user.
In-Reply-To: <29484910.14751275951732068.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6814:
-----------------------------------------

    Summary: Method in UGI to get the authentication method of the real user.  (was: Method in UGI to fet the authentication method of the real user.)

> Method in UGI to get the authentication method of the real user.
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6814
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6814
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6814.1.patch
>
>
> UGI should have a method to return the authentication method of the real user for a proxy-user scenario.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8933-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 23:23:36 2010
Return-Path: <common-issues-return-8933-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83903 invoked from network); 7 Jun 2010 23:23:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 23:23:36 -0000
Received: (qmail 83646 invoked by uid 500); 7 Jun 2010 23:23:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83612 invoked by uid 500); 7 Jun 2010 23:23:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83604 invoked by uid 99); 7 Jun 2010 23:23:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 23:23:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 23:23:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57NNCS3020070
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 23:23:12 GMT
Message-ID: <24604328.15201275952992748.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 19:23:12 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6814) Method in UGI to get the
 authentication method of the real user.
In-Reply-To: <29484910.14751275951732068.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6814:
-----------------------------------------

           Status: Patch Available  (was: Open)
    Fix Version/s: 0.22.0

> Method in UGI to get the authentication method of the real user.
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6814
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6814
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6814.1.patch
>
>
> UGI should have a method to return the authentication method of the real user for a proxy-user scenario.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8934-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 23:26:35 2010
Return-Path: <common-issues-return-8934-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84700 invoked from network); 7 Jun 2010 23:26:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 23:26:35 -0000
Received: (qmail 85279 invoked by uid 500); 7 Jun 2010 23:26:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85243 invoked by uid 500); 7 Jun 2010 23:26:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85234 invoked by uid 99); 7 Jun 2010 23:26:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 23:26:35 +0000
X-ASF-Spam-Status: No, hits=-1504.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 23:26:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57NQFMh020106
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 23:26:15 GMT
Message-ID: <17071845.15301275953175129.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 19:26:15 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6813) Add a new newInstance method in
 FileSystem that takes a "user" as argument
In-Reply-To: <17176494.13701275948371739.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876481#action_12876481 ] 

Chris Douglas commented on HADOOP-6813:
---------------------------------------

This issue went from created to committed in less than an hour. Changes to the {{FileSystem}} API should require more notice and explanation than was provided in this JIRA, though this looks like a reasonable convenience method to me.

> Add a new newInstance method in FileSystem that takes a "user" as argument
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6813
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6813
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: h-6813.patch
>
>
> In order to implement HDFS-1000 for trunk, I need to have a newInstance API in FileSystem that takes a "user" as an argument.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8935-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 07 23:41:36 2010
Return-Path: <common-issues-return-8935-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86761 invoked from network); 7 Jun 2010 23:41:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 7 Jun 2010 23:41:36 -0000
Received: (qmail 2248 invoked by uid 500); 7 Jun 2010 23:41:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2203 invoked by uid 500); 7 Jun 2010 23:41:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2195 invoked by uid 99); 7 Jun 2010 23:41:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 23:41:35 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Jun 2010 23:41:33 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o57NfCL5020260
	for <common-issues@hadoop.apache.org>; Mon, 7 Jun 2010 23:41:12 GMT
Message-ID: <18744129.15641275954072093.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 19:41:12 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6813) Add a new newInstance method in
 FileSystem that takes a "user" as argument
In-Reply-To: <17176494.13701275948371739.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876490#action_12876490 ] 

Devaraj Das commented on HADOOP-6813:
-------------------------------------

bq. This issue went from created to committed in less than an hour. Changes to the FileSystem API should require more notice and explanation than was provided in this JIRA, though this looks like a reasonable convenience method to me. 

Hmm.. makes sense to wait longer for such changes. The main reason for having this API is to take care of libhdfs. There is an API in libhdfs that tries to connect on behalf of users. In order to support that API, I added the method in FileSystem. The other option would be to create an UGI object and pass it to newInstance but that would complicate things a bit (to do with creating UGI objects in C code, etc.).

> Add a new newInstance method in FileSystem that takes a "user" as argument
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6813
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6813
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: h-6813.patch
>
>
> In order to implement HDFS-1000 for trunk, I need to have a newInstance API in FileSystem that takes a "user" as an argument.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8936-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 08 00:05:36 2010
Return-Path: <common-issues-return-8936-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95197 invoked from network); 8 Jun 2010 00:05:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jun 2010 00:05:36 -0000
Received: (qmail 11918 invoked by uid 500); 8 Jun 2010 00:05:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11886 invoked by uid 500); 8 Jun 2010 00:05:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11878 invoked by uid 99); 8 Jun 2010 00:05:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 00:05:35 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 00:05:33 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5805C8P020506
	for <common-issues@hadoop.apache.org>; Tue, 8 Jun 2010 00:05:12 GMT
Message-ID: <12428945.16021275955512097.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 20:05:12 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6586) Log authentication and
 authorization failures and successes
In-Reply-To: <736883635.426721266819387914.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876498#action_12876498 ] 

Allen Wittenauer commented on HADOOP-6586:
------------------------------------------

Awesome.  Now I can make my "Boris Can't Spell" JIRA.

> Log authentication and authorization failures and successes
> -----------------------------------------------------------
>
>                 Key: HADOOP-6586
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6586
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6586-3.patch, HADOOP-6586-4.patch, HADOOP-6586-5.patch, HADOOP-6586-7.patch, HADOOP-6586-8-BP20-1.patch, HADOOP-6586-8-BP20.patch, HADOOP-6586-8.patch, HADOOP-6586.patch
>
>
> This jira will cover RPC authentication and SL authorizations logging.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8937-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 08 01:43:32 2010
Return-Path: <common-issues-return-8937-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17966 invoked from network); 8 Jun 2010 01:43:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jun 2010 01:43:32 -0000
Received: (qmail 65200 invoked by uid 500); 8 Jun 2010 01:43:32 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65151 invoked by uid 500); 8 Jun 2010 01:43:32 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65143 invoked by uid 99); 8 Jun 2010 01:43:32 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 01:43:32 +0000
X-ASF-Spam-Status: No, hits=-1504.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 01:43:31 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o581hBGk021173
	for <common-issues@hadoop.apache.org>; Tue, 8 Jun 2010 01:43:11 GMT
Message-ID: <27497833.16941275961391324.JavaMail.jira@thor>
Date: Mon, 7 Jun 2010 21:43:11 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6814) Method in UGI to get the
 authentication method of the real user.
In-Reply-To: <29484910.14751275951732068.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876514#action_12876514 ] 

Hadoop QA commented on HADOOP-6814:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446545/HADOOP-6814.1.patch
  against trunk revision 952471.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/572/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/572/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/572/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/572/console

This message is automatically generated.

> Method in UGI to get the authentication method of the real user.
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6814
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6814
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6814.1.patch
>
>
> UGI should have a method to return the authentication method of the real user for a proxy-user scenario.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8938-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 08 04:31:35 2010
Return-Path: <common-issues-return-8938-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50553 invoked from network); 8 Jun 2010 04:31:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jun 2010 04:31:35 -0000
Received: (qmail 37787 invoked by uid 500); 8 Jun 2010 04:31:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37665 invoked by uid 500); 8 Jun 2010 04:31:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37657 invoked by uid 99); 8 Jun 2010 04:31:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 04:31:33 +0000
X-ASF-Spam-Status: No, hits=-1504.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 04:31:33 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o584VCZN022867
	for <common-issues@hadoop.apache.org>; Tue, 8 Jun 2010 04:31:12 GMT
Message-ID: <25747059.18481275971472368.JavaMail.jira@thor>
Date: Tue, 8 Jun 2010 00:31:12 -0400 (EDT)
From: "Uma Mahesh (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876541#action_12876541 ] 

Uma Mahesh commented on HADOOP-6762:
------------------------------------

testRPCInterrupted3:--
i also feel this may be the test problem.
 in this test, inside run method it is throwing the exception i.e java.util.concurrent.BrokenBarrierException
This is happening some times.
I added some wait to complete all the threads before assertion.
This is happenning randomly...most of the times passing the test.
Please post the latest test.

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt, hadoop-6762-7.txt, hadoop-6762-8.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8939-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 08 05:04:36 2010
Return-Path: <common-issues-return-8939-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55287 invoked from network); 8 Jun 2010 05:04:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jun 2010 05:04:36 -0000
Received: (qmail 50856 invoked by uid 500); 8 Jun 2010 05:04:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 50733 invoked by uid 500); 8 Jun 2010 05:04:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 50725 invoked by uid 99); 8 Jun 2010 05:04:35 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 05:04:35 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 05:04:33 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5854BnS023169
	for <common-issues@hadoop.apache.org>; Tue, 8 Jun 2010 05:04:11 GMT
Message-ID: <16665657.18831275973451813.JavaMail.jira@thor>
Date: Tue, 8 Jun 2010 01:04:11 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876546#action_12876546 ] 

sam rash commented on HADOOP-6762:
----------------------------------

-8 is the latest.  i ran it 100 times and did not get any failures.  do you still see a failure with rev 8?  if so, I can do some more work with latches to tighten it up (right now I have a simple sleep that should allow the threads to get warmed up, and a latch to let them gracefully exit)


> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt, hadoop-6762-7.txt, hadoop-6762-8.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8940-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 08 07:16:39 2010
Return-Path: <common-issues-return-8940-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5026 invoked from network); 8 Jun 2010 07:16:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jun 2010 07:16:39 -0000
Received: (qmail 42657 invoked by uid 500); 8 Jun 2010 07:16:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42507 invoked by uid 500); 8 Jun 2010 07:16:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42451 invoked by uid 99); 8 Jun 2010 07:16:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 07:16:35 +0000
X-ASF-Spam-Status: No, hits=-1505.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 07:16:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o587GEWC025602
	for <common-issues@hadoop.apache.org>; Tue, 8 Jun 2010 07:16:14 GMT
Message-ID: <9323227.20561275981374780.JavaMail.jira@thor>
Date: Tue, 8 Jun 2010 03:16:14 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6796:
------------------------------------

    Attachment: HADOOP-6796.2.patch

Wrote some quick unit tests

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.1.patch, HADOOP-6796.2.patch, HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8942-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 08 07:16:40 2010
Return-Path: <common-issues-return-8942-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5089 invoked from network); 8 Jun 2010 07:16:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jun 2010 07:16:40 -0000
Received: (qmail 42797 invoked by uid 500); 8 Jun 2010 07:16:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42673 invoked by uid 500); 8 Jun 2010 07:16:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42520 invoked by uid 99); 8 Jun 2010 07:16:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 07:16:37 +0000
X-ASF-Spam-Status: No, hits=-1505.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 07:16:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o587GGXt025632
	for <common-issues@hadoop.apache.org>; Tue, 8 Jun 2010 07:16:16 GMT
Message-ID: <33113785.20661275981376909.JavaMail.jira@thor>
Date: Tue, 8 Jun 2010 03:16:16 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6796:
------------------------------------

    Status: Patch Available  (was: Open)

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.1.patch, HADOOP-6796.2.patch, HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8941-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 08 07:16:40 2010
Return-Path: <common-issues-return-8941-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5081 invoked from network); 8 Jun 2010 07:16:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jun 2010 07:16:40 -0000
Received: (qmail 42731 invoked by uid 500); 8 Jun 2010 07:16:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42538 invoked by uid 500); 8 Jun 2010 07:16:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42508 invoked by uid 99); 8 Jun 2010 07:16:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 07:16:36 +0000
X-ASF-Spam-Status: No, hits=-1505.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 07:16:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o587GFSE025617
	for <common-issues@hadoop.apache.org>; Tue, 8 Jun 2010 07:16:15 GMT
Message-ID: <3375958.20611275981375931.JavaMail.jira@thor>
Date: Tue, 8 Jun 2010 03:16:15 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6796:
------------------------------------

    Status: Open  (was: Patch Available)

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.1.patch, HADOOP-6796.2.patch, HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8943-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 08 07:43:34 2010
Return-Path: <common-issues-return-8943-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13430 invoked from network); 8 Jun 2010 07:43:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jun 2010 07:43:34 -0000
Received: (qmail 93087 invoked by uid 500); 8 Jun 2010 07:43:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93060 invoked by uid 500); 8 Jun 2010 07:43:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93052 invoked by uid 99); 8 Jun 2010 07:43:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 07:43:33 +0000
X-ASF-Spam-Status: No, hits=-1505.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 07:43:32 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o587hCcN026007
	for <common-issues@hadoop.apache.org>; Tue, 8 Jun 2010 07:43:12 GMT
Message-ID: <30340740.21361275982992307.JavaMail.jira@thor>
Date: Tue, 8 Jun 2010 03:43:12 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876587#action_12876587 ] 

Hadoop QA commented on HADOOP-6796:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446567/HADOOP-6796.2.patch
  against trunk revision 952471.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 2 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/573/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/573/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/573/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/573/console

This message is automatically generated.

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.1.patch, HADOOP-6796.2.patch, HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8944-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 08 09:49:44 2010
Return-Path: <common-issues-return-8944-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45601 invoked from network); 8 Jun 2010 09:49:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jun 2010 09:49:44 -0000
Received: (qmail 25452 invoked by uid 500); 8 Jun 2010 09:49:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25331 invoked by uid 500); 8 Jun 2010 09:49:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25323 invoked by uid 99); 8 Jun 2010 09:49:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 09:49:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 09:49:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o589nDXm027332
	for <common-issues@hadoop.apache.org>; Tue, 8 Jun 2010 09:49:13 GMT
Message-ID: <3856915.22801275990553066.JavaMail.jira@thor>
Date: Tue, 8 Jun 2010 05:49:13 -0400 (EDT)
From: "Ravi Gummadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-5623) Streaming: process provided status
 messages are overwritten every 10 seoncds
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-5623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated HADOOP-5623:
---------------------------------

    Attachment: 5623.20S.patch

Patch for Y20S branch. Not for commit here.

> Streaming: process provided status messages are overwritten every 10 seoncds
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-5623
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5623
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.19.0, 0.19.1, 0.20.0
>            Reporter: Rick Cox
>            Assignee: Rick Cox
>             Fix For: 0.20.2, 0.21.0
>
>         Attachments: 5623.20S.patch, HADOOP-5623-streaming-status.patch, HADOOP-5623-streaming-status.patch, hadoop-5623-v1.patch
>
>
> Every 10 seconds (if the streaming process is producing output key/values on stdout), PipeMapRed sets the task's status string to "Records R/W=N/N". This replaces any custom task status that the streaming process may have specified using the "reporter:status:" stderr lines.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8945-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 08 11:23:40 2010
Return-Path: <common-issues-return-8945-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72389 invoked from network); 8 Jun 2010 11:23:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jun 2010 11:23:40 -0000
Received: (qmail 2959 invoked by uid 500); 8 Jun 2010 11:23:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2836 invoked by uid 500); 8 Jun 2010 11:23:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2813 invoked by uid 99); 8 Jun 2010 11:23:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 11:23:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 11:23:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o58BNClV028318
	for <common-issues@hadoop.apache.org>; Tue, 8 Jun 2010 11:23:12 GMT
Message-ID: <12775396.23591275996192773.JavaMail.jira@thor>
Date: Tue, 8 Jun 2010 07:23:12 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6687) user object in the subject in UGI
  should be reused in case of a relogin.
In-Reply-To: <191053771.27981270592254332.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876632#action_12876632 ] 

Hudson commented on HADOOP-6687:
--------------------------------

Integrated in Hadoop-Common-trunk #360 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/360/])
    HADOOP-6687.  user object in the subject in UGI should be reused in case of a relogin.


> user object in the subject in UGI  should be reused in case of a relogin.
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6687
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6687
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HADOOP-6687-y20.2.patch, HADOOP-6687.2.patch, HADOOP-6687.3.patch
>
>
>  In case of a relogin, a new user object is created in the subject. This causes the subject to lose some state for example authentication method. Instead, a new user object should not be added to the subject if one is already present.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8946-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 08 11:23:41 2010
Return-Path: <common-issues-return-8946-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72448 invoked from network); 8 Jun 2010 11:23:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jun 2010 11:23:41 -0000
Received: (qmail 3032 invoked by uid 500); 8 Jun 2010 11:23:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2843 invoked by uid 500); 8 Jun 2010 11:23:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2816 invoked by uid 99); 8 Jun 2010 11:23:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 11:23:36 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 11:23:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o58BNDm4028327
	for <common-issues@hadoop.apache.org>; Tue, 8 Jun 2010 11:23:13 GMT
Message-ID: <21528788.23621275996193205.JavaMail.jira@thor>
Date: Tue, 8 Jun 2010 07:23:13 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6813) Add a new newInstance method in
 FileSystem that takes a "user" as argument
In-Reply-To: <17176494.13701275948371739.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876633#action_12876633 ] 

Hudson commented on HADOOP-6813:
--------------------------------

Integrated in Hadoop-Common-trunk #360 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/360/])
    HADOOP-6813. Add a new newInstance method in FileSystem that takes a user as argument


> Add a new newInstance method in FileSystem that takes a "user" as argument
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6813
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6813
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: h-6813.patch
>
>
> In order to implement HDFS-1000 for trunk, I need to have a newInstance API in FileSystem that takes a "user" as an argument.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8947-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 08 17:34:39 2010
Return-Path: <common-issues-return-8947-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19726 invoked from network); 8 Jun 2010 17:34:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jun 2010 17:34:39 -0000
Received: (qmail 17702 invoked by uid 500); 8 Jun 2010 17:34:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17678 invoked by uid 500); 8 Jun 2010 17:34:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17670 invoked by uid 99); 8 Jun 2010 17:34:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 17:34:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 17:34:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o58HYEr9002439
	for <common-issues@hadoop.apache.org>; Tue, 8 Jun 2010 17:34:14 GMT
Message-ID: <5201417.28861276018454118.JavaMail.jira@thor>
Date: Tue, 8 Jun 2010 13:34:14 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5438) Merge FileSystem.create and
 FileSystem.append
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876739#action_12876739 ] 

Eli Collins commented on HADOOP-5438:
-------------------------------------

Since this a common file system API change we should probably discuss in a HADOOP jira. How about we:
* file a jira for 21 and 22 to unmerge create() and append() 
* file a jira for 22 to come up with a new API on trunk that we're comfortable releasing (eg the better create method He described above or having two RPCs only in the case where you try to append to a file that doesn't exist).



> Merge FileSystem.create and FileSystem.append
> ---------------------------------------------
>
>                 Key: HADOOP-5438
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5438
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: He Yongqiang
>            Assignee: He Yongqiang
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-5438(2009-04-06).patch, Hadoop-5438-2009-03-30.patch, Hadoop-5438-2009-03-31-2.patch, Hadoop-5438-2009-03-31.patch, Hadoop-5438-2009-05-10.patch, Hadoop-5438-2009-05-15.patch, Hadoop-5438-2009-05-19.patch, Hadoop-5438-2009-05-5.patch
>
>
> Currently, when a user wants to modify a file, the user first calls exists() to know if this file is already there. And then uses create() or append() according to whether the file exists or not.
> the code looks like:
> {code}
> FSDataOutputStream out_1 = null;
> if (fs.exists(path_1))
>    out_1 = fs.append(path_1);
> else
>    out_1 = fs.create(path_1);
> {code}
> . On the performace side,It involes two RPCs. On the easy-of-use side, it is not very convient in contrast to the traditional open interface.
> It will more complicate if there is a overwrite parameter specified. I donot know whether there is a bug about 'overwrite' in 0.19, some times it takes a long time for overwrite creates to reture. So i make the write file code with overwrite param works like:
> {code}
> boolean exists = fs.exists(name);
> if (overwrite) {
>     if (exists)
>        fs.delete(name, true);
>      this.out = fs.create(name, overwrite, bufferSize, replication,
> 				    blockSize, progress);
>      this.currentRowID = 0;
>  } else {
>    if (!exists)
> 	this.out = fs.create(name, overwrite, bufferSize,
> 					replication, blockSize, progress);
>    else
> 	this.out = fs.append(name, bufferSize, progress);
> {code}
> Some code statements there are really redundant and not needed, especialy with the delete(). But without deleting first, the overwrite takes a long time to reture.
> BTW, i will create another issue about the overwrite problem. If it is not a bug at all or a duplicate, someone please close it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8948-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 08 20:58:41 2010
Return-Path: <common-issues-return-8948-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4593 invoked from network); 8 Jun 2010 20:58:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jun 2010 20:58:40 -0000
Received: (qmail 45695 invoked by uid 500); 8 Jun 2010 20:58:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45642 invoked by uid 500); 8 Jun 2010 20:58:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45634 invoked by uid 99); 8 Jun 2010 20:58:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 20:58:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 20:58:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o58KwGwm004865
	for <common-issues@hadoop.apache.org>; Tue, 8 Jun 2010 20:58:16 GMT
Message-ID: <8440887.32721276030696711.JavaMail.jira@thor>
Date: Tue, 8 Jun 2010 16:58:16 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6756) Clean up and add documentation for
 configuration keys in CommonConfigurationKeys.java
In-Reply-To: <23380183.20521273191468124.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Erik Steffl updated HADOOP-6756:
--------------------------------

    Attachment: HADOOP-6756-0.22.0-1.patch

> Clean up and add documentation for configuration keys in CommonConfigurationKeys.java
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6756
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6756
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.3, 0.22.0
>            Reporter: Erik Steffl
>             Fix For: 0.20.3, 0.22.0
>
>         Attachments: HADOOP-6756-0.22.0-1.patch, HADOOP-6756-0.22.0.patch, jira.HADOOP-6756-0.20-1-FS_DEFAULT_NAME_KEY.patch, jira.HADOOP-6756-0.20-1.patch, jira.HADOOP-6756-0.20.patch
>
>
> Configuration keys in CommonConfigurationKeys.java should be cleaned up and documented (javadoc comments, appropriate *-default.xml descriptions).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8949-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 08 21:08:34 2010
Return-Path: <common-issues-return-8949-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12308 invoked from network); 8 Jun 2010 21:08:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jun 2010 21:08:34 -0000
Received: (qmail 68267 invoked by uid 500); 8 Jun 2010 21:08:33 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68237 invoked by uid 500); 8 Jun 2010 21:08:33 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68229 invoked by uid 99); 8 Jun 2010 21:08:33 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 21:08:33 +0000
X-ASF-Spam-Status: No, hits=-1506.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 21:08:33 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o58L8DrG005065
	for <common-issues@hadoop.apache.org>; Tue, 8 Jun 2010 21:08:13 GMT
Message-ID: <21256477.32991276031293332.JavaMail.jira@thor>
Date: Tue, 8 Jun 2010 17:08:13 -0400 (EDT)
From: "Erik Steffl (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6756) Clean up and add documentation for
 configuration keys in CommonConfigurationKeys.java
In-Reply-To: <23380183.20521273191468124.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876805#action_12876805 ] 

Erik Steffl commented on HADOOP-6756:
-------------------------------------

Patch https://issues.apache.org/jira/secure/attachment/12446625/HADOOP-6756-0.22.0-1.patch resolves issues of previous patch as described in comment https://issues.apache.org/jira/browse/HADOOP-6756?focusedCommentId=12876461&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12876461

Description of changes:

1,2: we decided upon default values for previously commented out fields, all new config parameters are uncommented now

3: fixed

4: done

5: new description, hopefully it makes sense now

6: "{@docRoot}/../core-default.html" is used to refer to core-default.xml, it's a document created by ant docs (forrest generated docs)

7: made the comment clear

> Clean up and add documentation for configuration keys in CommonConfigurationKeys.java
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6756
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6756
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.3, 0.22.0
>            Reporter: Erik Steffl
>             Fix For: 0.20.3, 0.22.0
>
>         Attachments: HADOOP-6756-0.22.0-1.patch, HADOOP-6756-0.22.0.patch, jira.HADOOP-6756-0.20-1-FS_DEFAULT_NAME_KEY.patch, jira.HADOOP-6756-0.20-1.patch, jira.HADOOP-6756-0.20.patch
>
>
> Configuration keys in CommonConfigurationKeys.java should be cleaned up and documented (javadoc comments, appropriate *-default.xml descriptions).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8950-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 08 23:28:35 2010
Return-Path: <common-issues-return-8950-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53682 invoked from network); 8 Jun 2010 23:28:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jun 2010 23:28:35 -0000
Received: (qmail 14434 invoked by uid 500); 8 Jun 2010 23:28:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 14400 invoked by uid 500); 8 Jun 2010 23:28:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 14392 invoked by uid 99); 8 Jun 2010 23:28:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 23:28:35 +0000
X-ASF-Spam-Status: No, hits=-1507.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 23:28:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o58NSDmY006705
	for <common-issues@hadoop.apache.org>; Tue, 8 Jun 2010 23:28:14 GMT
Message-ID: <22050752.36391276039693844.JavaMail.jira@thor>
Date: Tue, 8 Jun 2010 19:28:13 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6813) Add a new newInstance method in
 FileSystem that takes a "user" as argument
In-Reply-To: <17176494.13701275948371739.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6813:
--------------------------------

    Fix Version/s: 0.21.0
         Priority: Blocker  (was: Major)

Flagging for Tom to commit to 21.

> Add a new newInstance method in FileSystem that takes a "user" as argument
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6813
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6813
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: h-6813.patch
>
>
> In order to implement HDFS-1000 for trunk, I need to have a newInstance API in FileSystem that takes a "user" as an argument.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8951-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 08 23:28:40 2010
Return-Path: <common-issues-return-8951-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53907 invoked from network); 8 Jun 2010 23:28:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 8 Jun 2010 23:28:40 -0000
Received: (qmail 15326 invoked by uid 500); 8 Jun 2010 23:28:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15296 invoked by uid 500); 8 Jun 2010 23:28:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15287 invoked by uid 99); 8 Jun 2010 23:28:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 23:28:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jun 2010 23:28:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o58NSGot006720
	for <common-issues@hadoop.apache.org>; Tue, 8 Jun 2010 23:28:16 GMT
Message-ID: <4221791.36441276039696509.JavaMail.jira@thor>
Date: Tue, 8 Jun 2010 19:28:16 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6769) Add an API in FileSystem to get
 FileSystem instances based on users
In-Reply-To: <15183681.91401274128662681.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6769:
--------------------------------

        Fix Version/s: 0.21.0
    Affects Version/s: 0.21.0
                       0.22.0
             Priority: Blocker  (was: Major)

Flagging for Tom to commit to 21.

> Add an API in FileSystem to get FileSystem instances based on users
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6769
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6769
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>            Priority: Blocker
>             Fix For: 0.21.0, 0.22.0
>
>         Attachments: 6769-1.patch
>
>
> HDFS-1000 requires an API in FileSystem to get FileSystem instances based on users. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8952-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 09 10:27:43 2010
Return-Path: <common-issues-return-8952-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30264 invoked from network); 9 Jun 2010 10:27:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jun 2010 10:27:42 -0000
Received: (qmail 46489 invoked by uid 500); 9 Jun 2010 10:27:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46351 invoked by uid 500); 9 Jun 2010 10:27:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46341 invoked by uid 99); 9 Jun 2010 10:27:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 10:27:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 10:27:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o59AREuI013909
	for <common-issues@hadoop.apache.org>; Wed, 9 Jun 2010 10:27:15 GMT
Message-ID: <1387533.361276079234760.JavaMail.jira@thor>
Date: Wed, 9 Jun 2010 06:27:14 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6796:
------------------------------------

    Attachment: HADOOP-6796.3.patch

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.1.patch, HADOOP-6796.2.patch, HADOOP-6796.3.patch, HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8953-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 09 10:30:42 2010
Return-Path: <common-issues-return-8953-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32137 invoked from network); 9 Jun 2010 10:30:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jun 2010 10:30:42 -0000
Received: (qmail 51906 invoked by uid 500); 9 Jun 2010 10:30:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51821 invoked by uid 500); 9 Jun 2010 10:30:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51812 invoked by uid 99); 9 Jun 2010 10:30:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 10:30:40 +0000
X-ASF-Spam-Status: No, hits=-1508.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 10:30:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o59AUJG9013951
	for <common-issues@hadoop.apache.org>; Wed, 9 Jun 2010 10:30:19 GMT
Message-ID: <18336603.481276079419496.JavaMail.jira@thor>
Date: Wed, 9 Jun 2010 06:30:19 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6796:
------------------------------------

    Status: Patch Available  (was: Open)

Updated to junit4

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.1.patch, HADOOP-6796.2.patch, HADOOP-6796.3.patch, HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8954-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 09 10:30:42 2010
Return-Path: <common-issues-return-8954-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32192 invoked from network); 9 Jun 2010 10:30:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jun 2010 10:30:42 -0000
Received: (qmail 52006 invoked by uid 500); 9 Jun 2010 10:30:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51973 invoked by uid 500); 9 Jun 2010 10:30:42 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51965 invoked by uid 99); 9 Jun 2010 10:30:42 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 10:30:42 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 10:30:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o59AUIPA013930
	for <common-issues@hadoop.apache.org>; Wed, 9 Jun 2010 10:30:18 GMT
Message-ID: <4660483.411276079418219.JavaMail.jira@thor>
Date: Wed, 9 Jun 2010 06:30:18 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6796:
------------------------------------

    Status: Open  (was: Patch Available)

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.1.patch, HADOOP-6796.2.patch, HADOOP-6796.3.patch, HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8955-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 09 11:20:39 2010
Return-Path: <common-issues-return-8955-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48268 invoked from network); 9 Jun 2010 11:20:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jun 2010 11:20:39 -0000
Received: (qmail 36450 invoked by uid 500); 9 Jun 2010 11:20:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36421 invoked by uid 500); 9 Jun 2010 11:20:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36408 invoked by uid 99); 9 Jun 2010 11:20:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 11:20:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 11:20:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o59BKEfD014382
	for <common-issues@hadoop.apache.org>; Wed, 9 Jun 2010 11:20:14 GMT
Message-ID: <3776773.691276082414692.JavaMail.jira@thor>
Date: Wed, 9 Jun 2010 07:20:14 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877045#action_12877045 ] 

Hadoop QA commented on HADOOP-6796:
-----------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446680/HADOOP-6796.3.patch
  against trunk revision 952471.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 2 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/574/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/574/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/574/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/574/console

This message is automatically generated.

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.1.patch, HADOOP-6796.2.patch, HADOOP-6796.3.patch, HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8956-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 09 12:28:38 2010
Return-Path: <common-issues-return-8956-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62766 invoked from network); 9 Jun 2010 12:28:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jun 2010 12:28:37 -0000
Received: (qmail 12165 invoked by uid 500); 9 Jun 2010 12:28:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12135 invoked by uid 500); 9 Jun 2010 12:28:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12127 invoked by uid 99); 9 Jun 2010 12:28:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 12:28:36 +0000
X-ASF-Spam-Status: No, hits=-1508.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 12:28:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o59CSFJl015115
	for <common-issues@hadoop.apache.org>; Wed, 9 Jun 2010 12:28:15 GMT
Message-ID: <19281429.1431276086495104.JavaMail.jira@thor>
Date: Wed, 9 Jun 2010 08:28:15 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6743) Create a Writable class that stores
 a possibly incomplete list of FileStatus objects
In-Reply-To: <25158843.10621272873895914.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt resolved HADOOP-6743.
-------------------------------------

    Resolution: Won't Fix

Dhruba convinced me this should be solved inside HDFS-1111

> Create a Writable class that stores a possibly incomplete list of FileStatus objects
> ------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6743
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6743
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6743.0.patch
>
>
> HDFS-1111 requires a class that stores a list of FileStatus objects and a flag stating whether the list is complete or not. Creating a new class inside hadoop common seems to the best solution since it allows other parts of Hadoop to use this general class.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8957-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 09 16:41:50 2010
Return-Path: <common-issues-return-8957-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73586 invoked from network); 9 Jun 2010 16:41:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jun 2010 16:41:50 -0000
Received: (qmail 72484 invoked by uid 500); 9 Jun 2010 16:41:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72447 invoked by uid 500); 9 Jun 2010 16:41:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72439 invoked by uid 99); 9 Jun 2010 16:41:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 16:41:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 16:41:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o59GfPJ3017725
	for <common-issues@hadoop.apache.org>; Wed, 9 Jun 2010 16:41:25 GMT
Message-ID: <21453303.4871276101685409.JavaMail.jira@thor>
Date: Wed, 9 Jun 2010 12:41:25 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877127#action_12877127 ] 

Eli Collins commented on HADOOP-6796:
-------------------------------------

+1

I'll commit soon.

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.1.patch, HADOOP-6796.2.patch, HADOOP-6796.3.patch, HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8958-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 09 17:59:37 2010
Return-Path: <common-issues-return-8958-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2044 invoked from network); 9 Jun 2010 17:59:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jun 2010 17:59:36 -0000
Received: (qmail 76319 invoked by uid 500); 9 Jun 2010 17:59:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76264 invoked by uid 500); 9 Jun 2010 17:59:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76256 invoked by uid 99); 9 Jun 2010 17:59:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 17:59:36 +0000
X-ASF-Spam-Status: No, hits=-1509.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 17:59:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o59HxEoO018643
	for <common-issues@hadoop.apache.org>; Wed, 9 Jun 2010 17:59:15 GMT
Message-ID: <7037519.6451276106354931.JavaMail.jira@thor>
Date: Wed, 9 Jun 2010 13:59:14 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6796:
--------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

I've committed this. Thanks Rodrigo! 

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.1.patch, HADOOP-6796.2.patch, HADOOP-6796.3.patch, HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8959-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 09 18:15:38 2010
Return-Path: <common-issues-return-8959-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9592 invoked from network); 9 Jun 2010 18:15:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jun 2010 18:15:38 -0000
Received: (qmail 93172 invoked by uid 500); 9 Jun 2010 18:15:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93144 invoked by uid 500); 9 Jun 2010 18:15:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93136 invoked by uid 99); 9 Jun 2010 18:15:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 18:15:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 18:15:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o59IFE5V018821
	for <common-issues@hadoop.apache.org>; Wed, 9 Jun 2010 18:15:14 GMT
Message-ID: <12552209.6621276107314631.JavaMail.jira@thor>
Date: Wed, 9 Jun 2010 14:15:14 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877157#action_12877157 ] 

Hudson commented on HADOOP-6796:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #291 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/291/])
    HADOOP-6796. FileStatus allows null srcPath but crashes if that's done. Contributed by Rodrigo Schmidt.


> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.1.patch, HADOOP-6796.2.patch, HADOOP-6796.3.patch, HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8960-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 09 18:57:36 2010
Return-Path: <common-issues-return-8960-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29017 invoked from network); 9 Jun 2010 18:57:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jun 2010 18:57:36 -0000
Received: (qmail 56103 invoked by uid 500); 9 Jun 2010 18:57:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 56066 invoked by uid 500); 9 Jun 2010 18:57:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 56058 invoked by uid 99); 9 Jun 2010 18:57:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 18:57:36 +0000
X-ASF-Spam-Status: No, hits=-1509.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 18:57:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o59IvF6Y019274
	for <common-issues@hadoop.apache.org>; Wed, 9 Jun 2010 18:57:15 GMT
Message-ID: <1952077.7601276109835328.JavaMail.jira@thor>
Date: Wed, 9 Jun 2010 14:57:15 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6526) Need mapping from long principal
 names to local OS user names
In-Reply-To: <786866304.151051264870594555.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877167#action_12877167 ] 

Owen O'Malley commented on HADOOP-6526:
---------------------------------------

It looks good, except I don't think there is any reason to parameterize the name of the system property (java.security.krb5.conf). 

> Need mapping from long principal names to local OS user names
> -------------------------------------------------------------
>
>                 Key: HADOOP-6526
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6526
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 3595485.patch, c-6526-y20.patch, c-6526-y20.patch, c-6526.patch, HADOOP-6526-1.patch, HADOOP-6526-2.patch, HADOOP-6526-y20.2.patch, HADOOP-6526-y20.4.patch, HADOOP-6526.patch
>
>
> We need a configurable mapping from full user names (eg. omalley@APACHE.ORG) to local user names (eg. omalley). For many organizations it is sufficient to just use the prefix, however, in the case of shared clusters there may be duplicated prefixes. A configurable mapping will let administrators resolve the issue.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8961-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 09 19:30:37 2010
Return-Path: <common-issues-return-8961-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42437 invoked from network); 9 Jun 2010 19:30:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jun 2010 19:30:36 -0000
Received: (qmail 5096 invoked by uid 500); 9 Jun 2010 19:30:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5051 invoked by uid 500); 9 Jun 2010 19:30:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5043 invoked by uid 99); 9 Jun 2010 19:30:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 19:30:36 +0000
X-ASF-Spam-Status: No, hits=-1509.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 19:30:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o59JUF11019520
	for <common-issues@hadoop.apache.org>; Wed, 9 Jun 2010 19:30:16 GMT
Message-ID: <5383257.7931276111815771.JavaMail.jira@thor>
Date: Wed, 9 Jun 2010 15:30:15 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6810) Extract a subset of tests for smoke
 (DOA) validation
In-Reply-To: <10151977.193211275695163699.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877175#action_12877175 ] 

Konstantin Boudnik commented on HADOOP-6810:
--------------------------------------------

Have spoken with Boris offline and he expresses the concern that excluding commit and smoke tests from {{test-core}} target. While this is a valid concern because the set of core tests is changing by the patch without it {{test-core}} execution will increase by about 40 minutes (13 minutes from commit tests, which we already have in place, and about 30 minutes from smokes). 

Any thoughts? 

> Extract a subset of tests for smoke (DOA) validation
> ----------------------------------------------------
>
>                 Key: HADOOP-6810
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6810
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.20.3
>            Reporter: Konstantin Boudnik
>         Attachments: smoke-tests.patch, smoke-tests.patch, smoke-tests.patch, smoke-tests.patch
>
>
> Currently, there's no way to quickly find out if a new build of Hadoop is reasonably viable and can be used for future development/testing cycle.
> Similarly to 'commit' testing it'd be great to have a wider subset of existing tests to be executed inside of 30 minutes. The main goal of such a subset is to have a reasonable guarantees that different parts of Hadoop's functionality aren't broken and the system is not DOA.
> Full test run takes about 3+ hours and it is a too long wait time in many cases. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8962-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 09 21:14:37 2010
Return-Path: <common-issues-return-8962-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85652 invoked from network); 9 Jun 2010 21:14:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jun 2010 21:14:37 -0000
Received: (qmail 49502 invoked by uid 500); 9 Jun 2010 21:14:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49471 invoked by uid 500); 9 Jun 2010 21:14:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49463 invoked by uid 99); 9 Jun 2010 21:14:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 21:14:36 +0000
X-ASF-Spam-Status: No, hits=-1509.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 21:14:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o59LEFNi020513
	for <common-issues@hadoop.apache.org>; Wed, 9 Jun 2010 21:14:15 GMT
Message-ID: <23478647.9621276118055804.JavaMail.jira@thor>
Date: Wed, 9 Jun 2010 17:14:15 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6526) Need mapping from long principal
 names to local OS user names
In-Reply-To: <786866304.151051264870594555.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877208#action_12877208 ] 

Boris Shkolnik commented on HADOOP-6526:
----------------------------------------

both javadoc and javac warnings are related to this:
Constructing Javadoc information...
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:29: warning: sun.security.krb5.Config is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.Config;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:30: warning: sun.security.krb5.KrbException is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.KrbException;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:77: warning: sun.security.krb5.Config is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc]   private static Config kerbConf;


> Need mapping from long principal names to local OS user names
> -------------------------------------------------------------
>
>                 Key: HADOOP-6526
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6526
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 3595485.patch, c-6526-y20.patch, c-6526-y20.patch, c-6526.patch, HADOOP-6526-1.patch, HADOOP-6526-2.patch, HADOOP-6526-y20.2.patch, HADOOP-6526-y20.4.patch, HADOOP-6526.patch
>
>
> We need a configurable mapping from full user names (eg. omalley@APACHE.ORG) to local user names (eg. omalley). For many organizations it is sufficient to just use the prefix, however, in the case of shared clusters there may be duplicated prefixes. A configurable mapping will let administrators resolve the issue.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8963-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 09 21:31:40 2010
Return-Path: <common-issues-return-8963-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92784 invoked from network); 9 Jun 2010 21:31:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jun 2010 21:31:40 -0000
Received: (qmail 74108 invoked by uid 500); 9 Jun 2010 21:31:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74059 invoked by uid 500); 9 Jun 2010 21:31:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73926 invoked by uid 99); 9 Jun 2010 21:31:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 21:31:39 +0000
X-ASF-Spam-Status: No, hits=-1509.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 21:31:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o59LVImv020896
	for <common-issues@hadoop.apache.org>; Wed, 9 Jun 2010 21:31:19 GMT
Message-ID: <6952657.10741276119078804.JavaMail.jira@thor>
Date: Wed, 9 Jun 2010 17:31:18 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6526) Need mapping from long principal
 names to local OS user names
In-Reply-To: <786866304.151051264870594555.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6526:
-----------------------------------

    Status: Open  (was: Patch Available)

> Need mapping from long principal names to local OS user names
> -------------------------------------------------------------
>
>                 Key: HADOOP-6526
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6526
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 3595485.patch, c-6526-y20.patch, c-6526-y20.patch, c-6526.patch, HADOOP-6526-1.patch, HADOOP-6526-2.patch, HADOOP-6526-3.patch, HADOOP-6526-y20.2.patch, HADOOP-6526-y20.4.patch, HADOOP-6526.patch
>
>
> We need a configurable mapping from full user names (eg. omalley@APACHE.ORG) to local user names (eg. omalley). For many organizations it is sufficient to just use the prefix, however, in the case of shared clusters there may be duplicated prefixes. A configurable mapping will let administrators resolve the issue.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8964-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 09 21:31:41 2010
Return-Path: <common-issues-return-8964-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92993 invoked from network); 9 Jun 2010 21:31:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jun 2010 21:31:41 -0000
Received: (qmail 74929 invoked by uid 500); 9 Jun 2010 21:31:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74895 invoked by uid 500); 9 Jun 2010 21:31:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74887 invoked by uid 99); 9 Jun 2010 21:31:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 21:31:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 21:31:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o59LVHRX020869
	for <common-issues@hadoop.apache.org>; Wed, 9 Jun 2010 21:31:17 GMT
Message-ID: <23446421.10651276119077395.JavaMail.jira@thor>
Date: Wed, 9 Jun 2010 17:31:17 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6526) Need mapping from long principal
 names to local OS user names
In-Reply-To: <786866304.151051264870594555.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6526:
-----------------------------------

    Attachment: HADOOP-6526-3.patch

addressed review's comments

> Need mapping from long principal names to local OS user names
> -------------------------------------------------------------
>
>                 Key: HADOOP-6526
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6526
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 3595485.patch, c-6526-y20.patch, c-6526-y20.patch, c-6526.patch, HADOOP-6526-1.patch, HADOOP-6526-2.patch, HADOOP-6526-3.patch, HADOOP-6526-y20.2.patch, HADOOP-6526-y20.4.patch, HADOOP-6526.patch
>
>
> We need a configurable mapping from full user names (eg. omalley@APACHE.ORG) to local user names (eg. omalley). For many organizations it is sufficient to just use the prefix, however, in the case of shared clusters there may be duplicated prefixes. A configurable mapping will let administrators resolve the issue.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8965-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 09 21:31:45 2010
Return-Path: <common-issues-return-8965-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93245 invoked from network); 9 Jun 2010 21:31:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jun 2010 21:31:45 -0000
Received: (qmail 75288 invoked by uid 500); 9 Jun 2010 21:31:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 75259 invoked by uid 500); 9 Jun 2010 21:31:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 75251 invoked by uid 99); 9 Jun 2010 21:31:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 21:31:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 21:31:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o59LVL3a020933
	for <common-issues@hadoop.apache.org>; Wed, 9 Jun 2010 21:31:21 GMT
Message-ID: <9840857.10861276119081275.JavaMail.jira@thor>
Date: Wed, 9 Jun 2010 17:31:21 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6526) Need mapping from long principal
 names to local OS user names
In-Reply-To: <786866304.151051264870594555.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6526:
-----------------------------------

    Status: Patch Available  (was: Open)

> Need mapping from long principal names to local OS user names
> -------------------------------------------------------------
>
>                 Key: HADOOP-6526
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6526
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 3595485.patch, c-6526-y20.patch, c-6526-y20.patch, c-6526.patch, HADOOP-6526-1.patch, HADOOP-6526-2.patch, HADOOP-6526-3.patch, HADOOP-6526-y20.2.patch, HADOOP-6526-y20.4.patch, HADOOP-6526.patch
>
>
> We need a configurable mapping from full user names (eg. omalley@APACHE.ORG) to local user names (eg. omalley). For many organizations it is sufficient to just use the prefix, however, in the case of shared clusters there may be duplicated prefixes. A configurable mapping will let administrators resolve the issue.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8966-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 09 21:35:36 2010
Return-Path: <common-issues-return-8966-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95086 invoked from network); 9 Jun 2010 21:35:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jun 2010 21:35:36 -0000
Received: (qmail 81773 invoked by uid 500); 9 Jun 2010 21:35:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81734 invoked by uid 500); 9 Jun 2010 21:35:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81723 invoked by uid 99); 9 Jun 2010 21:35:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 21:35:36 +0000
X-ASF-Spam-Status: No, hits=-1509.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 21:35:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o59LZFaE020976
	for <common-issues@hadoop.apache.org>; Wed, 9 Jun 2010 21:35:15 GMT
Message-ID: <20039948.10941276119315214.JavaMail.jira@thor>
Date: Wed, 9 Jun 2010 17:35:15 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6815) refreshSuperUserGroupsConfiguration
 should use server side configuration for the refresh
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

refreshSuperUserGroupsConfiguration should use server side configuration for the refresh
----------------------------------------------------------------------------------------

                 Key: HADOOP-6815
                 URL: https://issues.apache.org/jira/browse/HADOOP-6815
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Boris Shkolnik
            Assignee: Boris Shkolnik




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8967-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 09 22:01:45 2010
Return-Path: <common-issues-return-8967-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1145 invoked from network); 9 Jun 2010 22:01:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jun 2010 22:01:45 -0000
Received: (qmail 13393 invoked by uid 500); 9 Jun 2010 22:01:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 13346 invoked by uid 500); 9 Jun 2010 22:01:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 13338 invoked by uid 99); 9 Jun 2010 22:01:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 22:01:45 +0000
X-ASF-Spam-Status: No, hits=-1509.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 22:01:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o59M1OaL021354
	for <common-issues@hadoop.apache.org>; Wed, 9 Jun 2010 22:01:24 GMT
Message-ID: <13408385.11681276120884418.JavaMail.jira@thor>
Date: Wed, 9 Jun 2010 18:01:24 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6526) Need mapping from long principal
 names to local OS user names
In-Reply-To: <786866304.151051264870594555.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877230#action_12877230 ] 

Hadoop QA commented on HADOOP-6526:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446722/HADOOP-6526-3.patch
  against trunk revision 953100.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 11 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    -1 javac.  The applied patch generated 1022 javac compiler warnings (more than the trunk's current 1017 warnings).

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/575/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/575/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/575/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/575/console

This message is automatically generated.

> Need mapping from long principal names to local OS user names
> -------------------------------------------------------------
>
>                 Key: HADOOP-6526
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6526
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 3595485.patch, c-6526-y20.patch, c-6526-y20.patch, c-6526.patch, HADOOP-6526-1.patch, HADOOP-6526-2.patch, HADOOP-6526-3.patch, HADOOP-6526-y20.2.patch, HADOOP-6526-y20.4.patch, HADOOP-6526.patch
>
>
> We need a configurable mapping from full user names (eg. omalley@APACHE.ORG) to local user names (eg. omalley). For many organizations it is sufficient to just use the prefix, however, in the case of shared clusters there may be duplicated prefixes. A configurable mapping will let administrators resolve the issue.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8968-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 09 22:46:39 2010
Return-Path: <common-issues-return-8968-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15588 invoked from network); 9 Jun 2010 22:46:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jun 2010 22:46:38 -0000
Received: (qmail 49330 invoked by uid 500); 9 Jun 2010 22:46:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49244 invoked by uid 500); 9 Jun 2010 22:46:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49236 invoked by uid 99); 9 Jun 2010 22:46:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 22:46:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 22:46:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o59MkE1R021732
	for <common-issues@hadoop.apache.org>; Wed, 9 Jun 2010 22:46:14 GMT
Message-ID: <18268528.12491276123574618.JavaMail.jira@thor>
Date: Wed, 9 Jun 2010 18:46:14 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6810) Extract a subset of tests for smoke
 (DOA) validation
In-Reply-To: <10151977.193211275695163699.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6810:
---------------------------------------

             Assignee: Konstantin Boudnik
    Affects Version/s: 0.21.0
                           (was: 0.20.3)

I think I have made it all confusing by setting 0.20 as the target. The attached patch is for internal Yahoo 20 release and trunk patch will be add here shortly. Trunk versions for HDFS/MR will follow shortly as well.

> Extract a subset of tests for smoke (DOA) validation
> ----------------------------------------------------
>
>                 Key: HADOOP-6810
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6810
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: smoke-tests.patch, smoke-tests.patch, smoke-tests.patch, smoke-tests.patch
>
>
> Currently, there's no way to quickly find out if a new build of Hadoop is reasonably viable and can be used for future development/testing cycle.
> Similarly to 'commit' testing it'd be great to have a wider subset of existing tests to be executed inside of 30 minutes. The main goal of such a subset is to have a reasonable guarantees that different parts of Hadoop's functionality aren't broken and the system is not DOA.
> Full test run takes about 3+ hours and it is a too long wait time in many cases. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8969-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 09 22:48:37 2010
Return-Path: <common-issues-return-8969-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16280 invoked from network); 9 Jun 2010 22:48:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jun 2010 22:48:37 -0000
Received: (qmail 49867 invoked by uid 500); 9 Jun 2010 22:48:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49824 invoked by uid 500); 9 Jun 2010 22:48:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49816 invoked by uid 99); 9 Jun 2010 22:48:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 22:48:37 +0000
X-ASF-Spam-Status: No, hits=-1509.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 22:48:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o59MmFEs021759
	for <common-issues@hadoop.apache.org>; Wed, 9 Jun 2010 22:48:16 GMT
Message-ID: <31345134.12561276123695836.JavaMail.jira@thor>
Date: Wed, 9 Jun 2010 18:48:15 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6810) Extract a subset of tests for smoke
 (DOA) validation
In-Reply-To: <10151977.193211275695163699.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877246#action_12877246 ] 

Jakob Homan commented on HADOOP-6810:
-------------------------------------

I tend to agree with Boris. With the current patch, if I run test-core, the behavior changes so that those tests in commit and smoke will *not* be run, correct?  That would be a big change from what users have grown used to.  I had assumed that commit tests are a subset of smoke tests are a subset of core-tests are a subset of test.  I had made previous comments that if you run, say ant commit-tests test, you would end up running the members of commit-tests twice, which was suboptimal.  But I'm not sure the other side of that - three separate disjoint sets - is also optimal.

Except for this behavior change, the patch looks good.

> Extract a subset of tests for smoke (DOA) validation
> ----------------------------------------------------
>
>                 Key: HADOOP-6810
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6810
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: smoke-tests.patch, smoke-tests.patch, smoke-tests.patch, smoke-tests.patch
>
>
> Currently, there's no way to quickly find out if a new build of Hadoop is reasonably viable and can be used for future development/testing cycle.
> Similarly to 'commit' testing it'd be great to have a wider subset of existing tests to be executed inside of 30 minutes. The main goal of such a subset is to have a reasonable guarantees that different parts of Hadoop's functionality aren't broken and the system is not DOA.
> Full test run takes about 3+ hours and it is a too long wait time in many cases. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8970-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 09 23:17:37 2010
Return-Path: <common-issues-return-8970-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26683 invoked from network); 9 Jun 2010 23:17:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jun 2010 23:17:37 -0000
Received: (qmail 78713 invoked by uid 500); 9 Jun 2010 23:17:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78681 invoked by uid 500); 9 Jun 2010 23:17:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78673 invoked by uid 99); 9 Jun 2010 23:17:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 23:17:37 +0000
X-ASF-Spam-Status: No, hits=-1510.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 23:17:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o59NHGRe022072
	for <common-issues@hadoop.apache.org>; Wed, 9 Jun 2010 23:17:16 GMT
Message-ID: <12718211.13091276125436254.JavaMail.jira@thor>
Date: Wed, 9 Jun 2010 19:17:16 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6810) Extract a subset of tests for smoke
 (DOA) validation
In-Reply-To: <10151977.193211275695163699.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6810:
---------------------------------------

    Attachment: smoke-tests.patch

Addressing Jacob's comment: {{core-test}} behavior is unchanged in this version. However, the overlap of running {{test-smoke}} followed by {{core-test}} will be equal to that of {{test-smoke}} tests.

> Extract a subset of tests for smoke (DOA) validation
> ----------------------------------------------------
>
>                 Key: HADOOP-6810
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6810
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: smoke-tests.patch, smoke-tests.patch, smoke-tests.patch, smoke-tests.patch, smoke-tests.patch
>
>
> Currently, there's no way to quickly find out if a new build of Hadoop is reasonably viable and can be used for future development/testing cycle.
> Similarly to 'commit' testing it'd be great to have a wider subset of existing tests to be executed inside of 30 minutes. The main goal of such a subset is to have a reasonable guarantees that different parts of Hadoop's functionality aren't broken and the system is not DOA.
> Full test run takes about 3+ hours and it is a too long wait time in many cases. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8971-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 09 23:23:35 2010
Return-Path: <common-issues-return-8971-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30696 invoked from network); 9 Jun 2010 23:23:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jun 2010 23:23:35 -0000
Received: (qmail 83265 invoked by uid 500); 9 Jun 2010 23:23:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83227 invoked by uid 500); 9 Jun 2010 23:23:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83215 invoked by uid 99); 9 Jun 2010 23:23:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 23:23:34 +0000
X-ASF-Spam-Status: No, hits=-1510.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 23:23:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o59NNESr022127
	for <common-issues@hadoop.apache.org>; Wed, 9 Jun 2010 23:23:14 GMT
Message-ID: <20908110.13201276125794094.JavaMail.jira@thor>
Date: Wed, 9 Jun 2010 19:23:14 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877262#action_12877262 ] 

Eli Collins commented on HADOOP-6762:
-------------------------------------

-8 is identical to -7, did you mean to upload a different patch?

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt, hadoop-6762-7.txt, hadoop-6762-8.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8972-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 09 23:26:38 2010
Return-Path: <common-issues-return-8972-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32418 invoked from network); 9 Jun 2010 23:26:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jun 2010 23:26:38 -0000
Received: (qmail 84749 invoked by uid 500); 9 Jun 2010 23:26:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84703 invoked by uid 500); 9 Jun 2010 23:26:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84695 invoked by uid 99); 9 Jun 2010 23:26:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 23:26:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 23:26:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o59NQEXF022163
	for <common-issues@hadoop.apache.org>; Wed, 9 Jun 2010 23:26:14 GMT
Message-ID: <8743317.13301276125974439.JavaMail.jira@thor>
Date: Wed, 9 Jun 2010 19:26:14 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6810) Extract a subset of tests for smoke
 (DOA) validation
In-Reply-To: <10151977.193211275695163699.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877263#action_12877263 ] 

Jakob Homan commented on HADOOP-6810:
-------------------------------------

+1 for Y20.

> Extract a subset of tests for smoke (DOA) validation
> ----------------------------------------------------
>
>                 Key: HADOOP-6810
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6810
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: smoke-tests.patch, smoke-tests.patch, smoke-tests.patch, smoke-tests.patch, smoke-tests.patch
>
>
> Currently, there's no way to quickly find out if a new build of Hadoop is reasonably viable and can be used for future development/testing cycle.
> Similarly to 'commit' testing it'd be great to have a wider subset of existing tests to be executed inside of 30 minutes. The main goal of such a subset is to have a reasonable guarantees that different parts of Hadoop's functionality aren't broken and the system is not DOA.
> Full test run takes about 3+ hours and it is a too long wait time in many cases. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8973-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 09 23:36:34 2010
Return-Path: <common-issues-return-8973-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34845 invoked from network); 9 Jun 2010 23:36:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 9 Jun 2010 23:36:34 -0000
Received: (qmail 89708 invoked by uid 500); 9 Jun 2010 23:36:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89682 invoked by uid 500); 9 Jun 2010 23:36:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89674 invoked by uid 99); 9 Jun 2010 23:36:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 23:36:34 +0000
X-ASF-Spam-Status: No, hits=-1510.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jun 2010 23:36:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o59NaD5x022211
	for <common-issues@hadoop.apache.org>; Wed, 9 Jun 2010 23:36:13 GMT
Message-ID: <21084038.13361276126573369.JavaMail.jira@thor>
Date: Wed, 9 Jun 2010 19:36:13 -0400 (EDT)
From: "Ted Yu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6816) enhance FsShell.dus() to include sum
 of totalSize
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

enhance FsShell.dus() to include sum of totalSize
-------------------------------------------------

                 Key: HADOOP-6816
                 URL: https://issues.apache.org/jira/browse/HADOOP-6816
             Project: Hadoop Common
          Issue Type: Improvement
          Components: util
    Affects Versions: 0.20.2
            Reporter: Ted Yu
             Fix For: 0.20.3


FsShell.dus() prints out totalSize for each file found.
It would be desirable to print sum of totalSize's at the end.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8974-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 01:28:35 2010
Return-Path: <common-issues-return-8974-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81744 invoked from network); 10 Jun 2010 01:28:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 01:28:35 -0000
Received: (qmail 64923 invoked by uid 500); 10 Jun 2010 01:28:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64886 invoked by uid 500); 10 Jun 2010 01:28:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64875 invoked by uid 99); 10 Jun 2010 01:28:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 01:28:35 +0000
X-ASF-Spam-Status: No, hits=-1510.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 01:28:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5A1SEOc023046
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 01:28:14 GMT
Message-ID: <2786644.14761276133294408.JavaMail.jira@thor>
Date: Wed, 9 Jun 2010 21:28:14 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877286#action_12877286 ] 

sam rash commented on HADOOP-6762:
----------------------------------

this is downright odd--if I click and view

https://issues.apache.org/jira/secure/attachment/12446356/hadoop-6762-7.txt
https://issues.apache.org/jira/secure/attachment/12446530/hadoop-6762-8.txt

they differ (the latter has a Thread.sleep() in the unit test)

but grabbing with wget 

wget https://issues.apache.org/jira/secure/attachment/12446530/hadoop-6762-7.txt --no-check-certificate
wget https://issues.apache.org/jira/secure/attachment/12446530/hadoop-6762-8.txt --no-check-certificate

they both have the Thread.sleep()

is something up with JIRA?  or what did I do?

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt, hadoop-6762-7.txt, hadoop-6762-8.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8975-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 06:48:46 2010
Return-Path: <common-issues-return-8975-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20352 invoked from network); 10 Jun 2010 06:48:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 06:48:43 -0000
Received: (qmail 91160 invoked by uid 500); 10 Jun 2010 06:48:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90953 invoked by uid 500); 10 Jun 2010 06:48:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90831 invoked by uid 99); 10 Jun 2010 06:48:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 06:48:40 +0000
X-ASF-Spam-Status: No, hits=-1510.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 06:48:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5A6mJrh025807
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 06:48:19 GMT
Message-ID: <12795229.18601276152499731.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 02:48:19 -0400 (EDT)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6756) Clean up and add documentation for
 configuration keys in CommonConfigurationKeys.java
In-Reply-To: <23380183.20521273191468124.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Shvachko reassigned HADOOP-6756:
-------------------------------------------

    Assignee: Erik Steffl

> Clean up and add documentation for configuration keys in CommonConfigurationKeys.java
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6756
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6756
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.3, 0.22.0
>            Reporter: Erik Steffl
>            Assignee: Erik Steffl
>             Fix For: 0.20.3, 0.22.0
>
>         Attachments: HADOOP-6756-0.22.0-1.patch, HADOOP-6756-0.22.0.patch, jira.HADOOP-6756-0.20-1-FS_DEFAULT_NAME_KEY.patch, jira.HADOOP-6756-0.20-1.patch, jira.HADOOP-6756-0.20.patch
>
>
> Configuration keys in CommonConfigurationKeys.java should be cleaned up and documented (javadoc comments, appropriate *-default.xml descriptions).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8976-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 06:48:47 2010
Return-Path: <common-issues-return-8976-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20370 invoked from network); 10 Jun 2010 06:48:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 06:48:44 -0000
Received: (qmail 91205 invoked by uid 500); 10 Jun 2010 06:48:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91082 invoked by uid 500); 10 Jun 2010 06:48:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90956 invoked by uid 99); 10 Jun 2010 06:48:41 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 06:48:41 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 06:48:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5A6mHVx025780
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 06:48:17 GMT
Message-ID: <11734702.18511276152497915.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 02:48:17 -0400 (EDT)
From: "Konstantin Shvachko (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6756) Clean up and add documentation for
 configuration keys in CommonConfigurationKeys.java
In-Reply-To: <23380183.20521273191468124.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877349#action_12877349 ] 

Konstantin Shvachko commented on HADOOP-6756:
---------------------------------------------

+1 on the patch.

> Clean up and add documentation for configuration keys in CommonConfigurationKeys.java
> -------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6756
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6756
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.3, 0.22.0
>            Reporter: Erik Steffl
>             Fix For: 0.20.3, 0.22.0
>
>         Attachments: HADOOP-6756-0.22.0-1.patch, HADOOP-6756-0.22.0.patch, jira.HADOOP-6756-0.20-1-FS_DEFAULT_NAME_KEY.patch, jira.HADOOP-6756-0.20-1.patch, jira.HADOOP-6756-0.20.patch
>
>
> Configuration keys in CommonConfigurationKeys.java should be cleaned up and documented (javadoc comments, appropriate *-default.xml descriptions).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8977-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 07:19:48 2010
Return-Path: <common-issues-return-8977-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32659 invoked from network); 10 Jun 2010 07:19:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 07:19:42 -0000
Received: (qmail 23192 invoked by uid 500); 10 Jun 2010 07:19:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21826 invoked by uid 500); 10 Jun 2010 07:19:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21800 invoked by uid 99); 10 Jun 2010 07:19:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 07:19:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 07:19:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5A7JDiF026174
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 07:19:13 GMT
Message-ID: <3770633.19001276154353682.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 03:19:13 -0400 (EDT)
From: "Wenjun Huang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6817) SequenceFile.Reader can't read gzip
 format compressed sequence file which produce by a mapreduce job without
 native compression library
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

SequenceFile.Reader can't read gzip format compressed sequence file which produce by a mapreduce job without native compression library
---------------------------------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-6817
                 URL: https://issues.apache.org/jira/browse/HADOOP-6817
             Project: Hadoop Common
          Issue Type: Bug
          Components: io
    Affects Versions: 0.20.2
         Environment: Cluster:CentOS 5,jdk1.6.0_20
Client:Mac SnowLeopard,jdk1.6.0_20
            Reporter: Wenjun Huang


An hadoop job output a gzip compressed sequence file(whether record compressed or block compressed).The client program use SequenceFile.Reader to read this sequence file,when reading the client program shows the following exceptions:

2090 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2091 [main] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new decompressor
Exception in thread "main" java.io.EOFException
	at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:207)
	at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:197)
	at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:136)
	at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:58)
	at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:68)
	at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream$ResetableGZIPInputStream.<init>(GzipCodec.java:92)
	at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream.<init>(GzipCodec.java:101)
	at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:170)
	at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:180)
	at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1520)
	at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1428)
	at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1417)
	at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1412)
	at com.shiningware.intelligenceonline.taobao.mapreduce.HtmlContentSeqOutputView.main(HtmlContentSeqOutputView.java:28)

I studied the code in org.apache.hadoop.io.SequenceFile.Reader.init method and read:
      // Initialize... *not* if this we are constructing a temporary Reader
      if (!tempReader) {
        valBuffer = new DataInputBuffer();
        if (decompress) {
          valDecompressor = CodecPool.getDecompressor(codec);
          valInFilter = codec.createInputStream(valBuffer, valDecompressor);
          valIn = new DataInputStream(valInFilter);
        } else {
          valIn = valBuffer;
        }
the problem seems to be caused by "valBuffer = new DataInputBuffer();" ,because GzipCodec.createInputStream creates an instance of GzipInputStream whose constructor creates an instance of ResetableGZIPInputStream class.When ResetableGZIPInputStream's constructor calls it base class java.util.zip.GZIPInputStream's constructor ,it trys to read the empty "valBuffer = new DataInputBuffer();" and get no content,so it throws an EOFException.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8978-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 11:23:38 2010
Return-Path: <common-issues-return-8978-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5260 invoked from network); 10 Jun 2010 11:23:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 11:23:38 -0000
Received: (qmail 65096 invoked by uid 500); 10 Jun 2010 11:23:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64966 invoked by uid 500); 10 Jun 2010 11:23:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64958 invoked by uid 99); 10 Jun 2010 11:23:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 11:23:35 +0000
X-ASF-Spam-Status: No, hits=-1511.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 11:23:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5ABNENR028226
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 11:23:15 GMT
Message-ID: <4976523.20801276168994590.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 07:23:14 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877396#action_12877396 ] 

Hudson commented on HADOOP-6796:
--------------------------------

Integrated in Hadoop-Common-trunk #362 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/362/])
    HADOOP-6796. FileStatus allows null srcPath but crashes if that's done. Contributed by Rodrigo Schmidt.


> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.1.patch, HADOOP-6796.2.patch, HADOOP-6796.3.patch, HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8979-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 16:49:36 2010
Return-Path: <common-issues-return-8979-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19619 invoked from network); 10 Jun 2010 16:49:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 16:49:36 -0000
Received: (qmail 91985 invoked by uid 500); 10 Jun 2010 16:49:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91942 invoked by uid 500); 10 Jun 2010 16:49:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91764 invoked by uid 99); 10 Jun 2010 16:49:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 16:49:35 +0000
X-ASF-Spam-Status: No, hits=-1511.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 16:49:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5AGnFrT001255
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 16:49:15 GMT
Message-ID: <4056402.25001276188555034.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 12:49:15 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877482#action_12877482 ] 

Todd Lipcon commented on HADOOP-6762:
-------------------------------------

Note that one of the path segments differs between the two URLs. I think it just pulls an attachment ID from that path segment and downloads that, regardless of the name at the end of the path (your two wget urls have the same path segment, I guess you manually edited the last bit?)

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt, hadoop-6762-7.txt, hadoop-6762-8.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8980-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 16:56:40 2010
Return-Path: <common-issues-return-8980-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20439 invoked from network); 10 Jun 2010 16:56:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 16:56:40 -0000
Received: (qmail 94744 invoked by uid 500); 10 Jun 2010 16:56:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94713 invoked by uid 500); 10 Jun 2010 16:56:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94705 invoked by uid 99); 10 Jun 2010 16:56:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 16:56:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 16:56:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5AGuF7M001396
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 16:56:16 GMT
Message-ID: <3011554.25341276188975774.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 12:56:15 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6526) Need mapping from long principal
 names to local OS user names
In-Reply-To: <786866304.151051264870594555.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6526:
-----------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

committed to trunk.

> Need mapping from long principal names to local OS user names
> -------------------------------------------------------------
>
>                 Key: HADOOP-6526
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6526
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 3595485.patch, c-6526-y20.patch, c-6526-y20.patch, c-6526.patch, HADOOP-6526-1.patch, HADOOP-6526-2.patch, HADOOP-6526-3.patch, HADOOP-6526-y20.2.patch, HADOOP-6526-y20.4.patch, HADOOP-6526.patch
>
>
> We need a configurable mapping from full user names (eg. omalley@APACHE.ORG) to local user names (eg. omalley). For many organizations it is sufficient to just use the prefix, however, in the case of shared clusters there may be duplicated prefixes. A configurable mapping will let administrators resolve the issue.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8981-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 17:00:40 2010
Return-Path: <common-issues-return-8981-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 22071 invoked from network); 10 Jun 2010 17:00:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 17:00:40 -0000
Received: (qmail 99643 invoked by uid 500); 10 Jun 2010 17:00:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99603 invoked by uid 500); 10 Jun 2010 17:00:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99595 invoked by uid 99); 10 Jun 2010 17:00:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 17:00:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 17:00:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5AH0GcJ001472
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 17:00:16 GMT
Message-ID: <32176336.25461276189216425.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 13:00:16 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877488#action_12877488 ] 

sam rash commented on HADOOP-6762:
----------------------------------

yes, I did :)

so the two patches do differ, probably in one line, Thread.sleep(1000);

As I said, I ran ant test-core in a loop 100x and did not see the failure.  If someone does, though, I can do something more sophisticated in the test (like using another latch to make sure the leader has started before interrupting it and causing the BrokenBarrierException)

let me know



> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt, hadoop-6762-7.txt, hadoop-6762-8.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8982-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 17:17:36 2010
Return-Path: <common-issues-return-8982-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29839 invoked from network); 10 Jun 2010 17:17:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 17:17:35 -0000
Received: (qmail 20003 invoked by uid 500); 10 Jun 2010 17:17:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19978 invoked by uid 500); 10 Jun 2010 17:17:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19970 invoked by uid 99); 10 Jun 2010 17:17:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 17:17:35 +0000
X-ASF-Spam-Status: No, hits=-1511.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 17:17:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5AHHF5X001676
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 17:17:15 GMT
Message-ID: <861119.25661276190235156.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 13:17:15 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6526) Need mapping from long principal
 names to local OS user names
In-Reply-To: <786866304.151051264870594555.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877493#action_12877493 ] 

Hudson commented on HADOOP-6526:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #292 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/292/])
    HADOOP-6526. Need mapping from long principal names to local OS user names


> Need mapping from long principal names to local OS user names
> -------------------------------------------------------------
>
>                 Key: HADOOP-6526
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6526
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 3595485.patch, c-6526-y20.patch, c-6526-y20.patch, c-6526.patch, HADOOP-6526-1.patch, HADOOP-6526-2.patch, HADOOP-6526-3.patch, HADOOP-6526-y20.2.patch, HADOOP-6526-y20.4.patch, HADOOP-6526.patch
>
>
> We need a configurable mapping from full user names (eg. omalley@APACHE.ORG) to local user names (eg. omalley). For many organizations it is sufficient to just use the prefix, however, in the case of shared clusters there may be duplicated prefixes. A configurable mapping will let administrators resolve the issue.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8983-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 18:38:36 2010
Return-Path: <common-issues-return-8983-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61119 invoked from network); 10 Jun 2010 18:38:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 18:38:36 -0000
Received: (qmail 16236 invoked by uid 500); 10 Jun 2010 18:38:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16212 invoked by uid 500); 10 Jun 2010 18:38:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16204 invoked by uid 99); 10 Jun 2010 18:38:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 18:38:36 +0000
X-ASF-Spam-Status: No, hits=-1512.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 18:38:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5AIcFo0002705
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 18:38:15 GMT
Message-ID: <20585750.27771276195095694.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 14:38:15 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6815) refreshSuperUserGroupsConfiguration
 should use server side configuration for the refresh
In-Reply-To: <20039948.10941276119315214.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6815:
-----------------------------------

    Status: Patch Available  (was: Open)

> refreshSuperUserGroupsConfiguration should use server side configuration for the refresh
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6815
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6815
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6815.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8984-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 18:38:39 2010
Return-Path: <common-issues-return-8984-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 61226 invoked from network); 10 Jun 2010 18:38:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 18:38:39 -0000
Received: (qmail 16810 invoked by uid 500); 10 Jun 2010 18:38:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16785 invoked by uid 500); 10 Jun 2010 18:38:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16777 invoked by uid 99); 10 Jun 2010 18:38:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 18:38:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 18:38:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5AIcEkB002696
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 18:38:15 GMT
Message-ID: <7812340.27741276195094927.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 14:38:14 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6815) refreshSuperUserGroupsConfiguration
 should use server side configuration for the refresh
In-Reply-To: <20039948.10941276119315214.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6815:
-----------------------------------

    Attachment: HADOOP-6815.patch

> refreshSuperUserGroupsConfiguration should use server side configuration for the refresh
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6815
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6815
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6815.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8985-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 19:03:39 2010
Return-Path: <common-issues-return-8985-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67306 invoked from network); 10 Jun 2010 19:03:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 19:03:39 -0000
Received: (qmail 62115 invoked by uid 500); 10 Jun 2010 19:03:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62045 invoked by uid 500); 10 Jun 2010 19:03:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62037 invoked by uid 99); 10 Jun 2010 19:03:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 19:03:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 19:03:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5AJ3EKW003110
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 19:03:15 GMT
Message-ID: <21089837.28671276196594796.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 15:03:14 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6815) refreshSuperUserGroupsConfiguration
 should use server side configuration for the refresh
In-Reply-To: <20039948.10941276119315214.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877531#action_12877531 ] 

Hadoop QA commented on HADOOP-6815:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446780/HADOOP-6815.patch
  against trunk revision 953388.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/576/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/576/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/576/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/576/console

This message is automatically generated.

> refreshSuperUserGroupsConfiguration should use server side configuration for the refresh
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6815
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6815
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6815.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8986-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 19:24:39 2010
Return-Path: <common-issues-return-8986-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77208 invoked from network); 10 Jun 2010 19:24:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 19:24:39 -0000
Received: (qmail 91057 invoked by uid 500); 10 Jun 2010 19:24:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91027 invoked by uid 500); 10 Jun 2010 19:24:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91019 invoked by uid 99); 10 Jun 2010 19:24:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 19:24:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 19:24:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5AJOFqu003331
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 19:24:15 GMT
Message-ID: <1461990.29111276197855272.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 15:24:15 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6603) Provide workaround for issue with
 Kerberos not resolving cross-realm principal
In-Reply-To: <924517533.47301267493525772.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6603:
-----------------------------------------

    Attachment: HADOOP-6603.1.patch

Patch for common trunk uploaded.

> Provide workaround for issue with Kerberos not resolving cross-realm principal
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6603
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6603
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Jakob Homan
>            Assignee: Kan Zhang
>         Attachments: fix_comment_y20.patch, HADOOP-6603-Y20S-2.patch, HADOOP-6603-Y20S-3.patch, HADOOP-6603-Y20S-4.patch, HADOOP-6603-Y20S.patch, HADOOP-6603.1.patch
>
>
> Java's SSL-Kerberos implementation does not correctly obtain the principal for cross-realm principles when clients initiate connections to servers, resulting in the client being unable to authenticate the server.  We need a work-around until this bug gets fixed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8987-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 19:24:40 2010
Return-Path: <common-issues-return-8987-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77228 invoked from network); 10 Jun 2010 19:24:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 19:24:39 -0000
Received: (qmail 91209 invoked by uid 500); 10 Jun 2010 19:24:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91172 invoked by uid 500); 10 Jun 2010 19:24:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91164 invoked by uid 99); 10 Jun 2010 19:24:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 19:24:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 19:24:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5AJOFKa003340
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 19:24:15 GMT
Message-ID: <28949152.29141276197855644.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 15:24:15 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6603) Provide workaround for issue with
 Kerberos not resolving cross-realm principal
In-Reply-To: <924517533.47301267493525772.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6603:
-----------------------------------------

    Status: Patch Available  (was: Open)

HADOOP-6603.1.patch is submitted for hudson tests.

> Provide workaround for issue with Kerberos not resolving cross-realm principal
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6603
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6603
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Jakob Homan
>            Assignee: Kan Zhang
>         Attachments: fix_comment_y20.patch, HADOOP-6603-Y20S-2.patch, HADOOP-6603-Y20S-3.patch, HADOOP-6603-Y20S-4.patch, HADOOP-6603-Y20S.patch, HADOOP-6603.1.patch
>
>
> Java's SSL-Kerberos implementation does not correctly obtain the principal for cross-realm principles when clients initiate connections to servers, resulting in the client being unable to authenticate the server.  We need a work-around until this bug gets fixed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8988-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 19:45:35 2010
Return-Path: <common-issues-return-8988-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81952 invoked from network); 10 Jun 2010 19:45:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 19:45:35 -0000
Received: (qmail 17583 invoked by uid 500); 10 Jun 2010 19:45:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17461 invoked by uid 500); 10 Jun 2010 19:45:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17453 invoked by uid 99); 10 Jun 2010 19:45:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 19:45:35 +0000
X-ASF-Spam-Status: No, hits=-1512.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 19:45:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5AJjEws003510
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 19:45:14 GMT
Message-ID: <10559678.29371276199114408.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 15:45:14 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6814) Method in UGI to get the
 authentication method of the real user.
In-Reply-To: <29484910.14751275951732068.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877545#action_12877545 ] 

Devaraj Das commented on HADOOP-6814:
-------------------------------------

Minor nit - should we rename getAuthenticationMethod in the patch to getRealAuthenticationMethod or some such (just to stress that this method returns the actual underlying authentication method).

> Method in UGI to get the authentication method of the real user.
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6814
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6814
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6814.1.patch
>
>
> UGI should have a method to return the authentication method of the real user for a proxy-user scenario.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8989-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 19:54:39 2010
Return-Path: <common-issues-return-8989-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84695 invoked from network); 10 Jun 2010 19:54:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 19:54:39 -0000
Received: (qmail 34804 invoked by uid 500); 10 Jun 2010 19:54:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 34778 invoked by uid 500); 10 Jun 2010 19:54:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 34770 invoked by uid 99); 10 Jun 2010 19:54:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 19:54:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 19:54:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5AJsF1x003569
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 19:54:15 GMT
Message-ID: <7485778.29481276199655445.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 15:54:15 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6603) Provide workaround for issue with
 Kerberos not resolving cross-realm principal
In-Reply-To: <924517533.47301267493525772.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877547#action_12877547 ] 

Hadoop QA commented on HADOOP-6603:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446784/HADOOP-6603.1.patch
  against trunk revision 953388.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    -1 javac.  The applied patch generated 1032 javac compiler warnings (more than the trunk's current 1022 warnings).

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/577/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/577/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/577/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/577/console

This message is automatically generated.

> Provide workaround for issue with Kerberos not resolving cross-realm principal
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6603
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6603
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Jakob Homan
>            Assignee: Kan Zhang
>         Attachments: fix_comment_y20.patch, HADOOP-6603-Y20S-2.patch, HADOOP-6603-Y20S-3.patch, HADOOP-6603-Y20S-4.patch, HADOOP-6603-Y20S.patch, HADOOP-6603.1.patch
>
>
> Java's SSL-Kerberos implementation does not correctly obtain the principal for cross-realm principles when clients initiate connections to servers, resulting in the client being unable to authenticate the server.  We need a work-around until this bug gets fixed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8990-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 20:13:35 2010
Return-Path: <common-issues-return-8990-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93719 invoked from network); 10 Jun 2010 20:13:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 20:13:35 -0000
Received: (qmail 67721 invoked by uid 500); 10 Jun 2010 20:13:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67637 invoked by uid 500); 10 Jun 2010 20:13:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67629 invoked by uid 99); 10 Jun 2010 20:13:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 20:13:34 +0000
X-ASF-Spam-Status: No, hits=-1512.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 20:13:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5AKDECQ003785
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 20:13:14 GMT
Message-ID: <17262845.29701276200793990.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 16:13:13 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6818) Provide a JNI-based implementation of
 GroupMappingServiceProvider
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Provide a JNI-based implementation of GroupMappingServiceProvider
-----------------------------------------------------------------

                 Key: HADOOP-6818
                 URL: https://issues.apache.org/jira/browse/HADOOP-6818
             Project: Hadoop Common
          Issue Type: Improvement
          Components: security
            Reporter: Devaraj Das
            Assignee: Devaraj Das
             Fix For: 0.22.0


The default implementation of GroupMappingServiceProvider does a fork of a unix command to get the groups of a user. Since the group resolution happens in the servers, this might be costly. This jira aims at providing a JNI-based implementation for GroupMappingServiceProvider.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8991-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 20:16:37 2010
Return-Path: <common-issues-return-8991-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94632 invoked from network); 10 Jun 2010 20:16:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 20:16:37 -0000
Received: (qmail 71153 invoked by uid 500); 10 Jun 2010 20:16:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71125 invoked by uid 500); 10 Jun 2010 20:16:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71117 invoked by uid 99); 10 Jun 2010 20:16:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 20:16:36 +0000
X-ASF-Spam-Status: No, hits=-1512.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 20:16:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5AKGF5J003851
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 20:16:16 GMT
Message-ID: <12216517.29851276200975924.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 16:16:15 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6818) Provide a JNI-based implementation of
 GroupMappingServiceProvider
In-Reply-To: <17262845.29701276200793990.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6818:
--------------------------------

    Attachment: JNIGroupMapping.patch

Patch for Y20S (not for commit). The meat of the patch is in the C and Java files. Also, the src/native/Makefile.am & src/native/configure.ac was changed to easily accommodate this native implementation in the hadoop build.

> Provide a JNI-based implementation of GroupMappingServiceProvider
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6818
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6818
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: JNIGroupMapping.patch
>
>
> The default implementation of GroupMappingServiceProvider does a fork of a unix command to get the groups of a user. Since the group resolution happens in the servers, this might be costly. This jira aims at providing a JNI-based implementation for GroupMappingServiceProvider.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8992-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 20:55:39 2010
Return-Path: <common-issues-return-8992-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7244 invoked from network); 10 Jun 2010 20:55:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 20:55:38 -0000
Received: (qmail 20431 invoked by uid 500); 10 Jun 2010 20:55:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20396 invoked by uid 500); 10 Jun 2010 20:55:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20388 invoked by uid 99); 10 Jun 2010 20:55:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 20:55:38 +0000
X-ASF-Spam-Status: No, hits=-1512.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 20:55:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5AKtHQC004365
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 20:55:17 GMT
Message-ID: <24173433.31011276203317643.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 16:55:17 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6815) refreshSuperUserGroupsConfiguration
 should use server side configuration for the refresh
In-Reply-To: <20039948.10941276119315214.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6815:
-----------------------------------

    Attachment: HADOOP-6815-1.patch

fixed javadoc warning

> refreshSuperUserGroupsConfiguration should use server side configuration for the refresh
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6815
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6815
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6815-1.patch, HADOOP-6815.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8993-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 22:26:38 2010
Return-Path: <common-issues-return-8993-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38995 invoked from network); 10 Jun 2010 22:26:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 22:26:38 -0000
Received: (qmail 28910 invoked by uid 500); 10 Jun 2010 22:26:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28879 invoked by uid 500); 10 Jun 2010 22:26:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28871 invoked by uid 99); 10 Jun 2010 22:26:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 22:26:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 22:26:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5AMQDJg005461
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 22:26:14 GMT
Message-ID: <27761451.33011276208773977.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 18:26:13 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6603) Provide workaround for issue with
 Kerberos not resolving cross-realm principal
In-Reply-To: <924517533.47301267493525772.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6603:
-----------------------------------------

    Attachment: HADOOP-6603.2.patch

Fixed a javadoc.

> Provide workaround for issue with Kerberos not resolving cross-realm principal
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6603
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6603
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Jakob Homan
>            Assignee: Kan Zhang
>         Attachments: fix_comment_y20.patch, HADOOP-6603-Y20S-2.patch, HADOOP-6603-Y20S-3.patch, HADOOP-6603-Y20S-4.patch, HADOOP-6603-Y20S.patch, HADOOP-6603.1.patch, HADOOP-6603.2.patch
>
>
> Java's SSL-Kerberos implementation does not correctly obtain the principal for cross-realm principles when clients initiate connections to servers, resulting in the client being unable to authenticate the server.  We need a work-around until this bug gets fixed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8994-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 22:26:38 2010
Return-Path: <common-issues-return-8994-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39035 invoked from network); 10 Jun 2010 22:26:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 22:26:38 -0000
Received: (qmail 29062 invoked by uid 500); 10 Jun 2010 22:26:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29024 invoked by uid 500); 10 Jun 2010 22:26:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29016 invoked by uid 99); 10 Jun 2010 22:26:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 22:26:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 22:26:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5AMQEfC005470
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 22:26:14 GMT
Message-ID: <31999574.33041276208774823.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 18:26:14 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6603) Provide workaround for issue with
 Kerberos not resolving cross-realm principal
In-Reply-To: <924517533.47301267493525772.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6603:
-----------------------------------------

    Status: Open  (was: Patch Available)

> Provide workaround for issue with Kerberos not resolving cross-realm principal
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6603
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6603
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Jakob Homan
>            Assignee: Kan Zhang
>         Attachments: fix_comment_y20.patch, HADOOP-6603-Y20S-2.patch, HADOOP-6603-Y20S-3.patch, HADOOP-6603-Y20S-4.patch, HADOOP-6603-Y20S.patch, HADOOP-6603.1.patch, HADOOP-6603.2.patch
>
>
> Java's SSL-Kerberos implementation does not correctly obtain the principal for cross-realm principles when clients initiate connections to servers, resulting in the client being unable to authenticate the server.  We need a work-around until this bug gets fixed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8995-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 22:26:39 2010
Return-Path: <common-issues-return-8995-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39077 invoked from network); 10 Jun 2010 22:26:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 22:26:39 -0000
Received: (qmail 29200 invoked by uid 500); 10 Jun 2010 22:26:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29169 invoked by uid 500); 10 Jun 2010 22:26:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29161 invoked by uid 99); 10 Jun 2010 22:26:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 22:26:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 22:26:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5AMQFf8005479
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 22:26:15 GMT
Message-ID: <22519254.33071276208775245.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 18:26:15 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6603) Provide workaround for issue with
 Kerberos not resolving cross-realm principal
In-Reply-To: <924517533.47301267493525772.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6603:
-----------------------------------------

    Status: Patch Available  (was: Open)

> Provide workaround for issue with Kerberos not resolving cross-realm principal
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6603
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6603
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Jakob Homan
>            Assignee: Kan Zhang
>         Attachments: fix_comment_y20.patch, HADOOP-6603-Y20S-2.patch, HADOOP-6603-Y20S-3.patch, HADOOP-6603-Y20S-4.patch, HADOOP-6603-Y20S.patch, HADOOP-6603.1.patch, HADOOP-6603.2.patch
>
>
> Java's SSL-Kerberos implementation does not correctly obtain the principal for cross-realm principles when clients initiate connections to servers, resulting in the client being unable to authenticate the server.  We need a work-around until this bug gets fixed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8996-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 22:50:43 2010
Return-Path: <common-issues-return-8996-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42045 invoked from network); 10 Jun 2010 22:50:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 22:50:43 -0000
Received: (qmail 45782 invoked by uid 500); 10 Jun 2010 22:50:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45752 invoked by uid 500); 10 Jun 2010 22:50:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45744 invoked by uid 99); 10 Jun 2010 22:50:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 22:50:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 22:50:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5AMoJen005704
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 22:50:19 GMT
Message-ID: <28531532.33631276210219382.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 18:50:19 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6815) refreshSuperUserGroupsConfiguration
 should use server side configuration for the refresh
In-Reply-To: <20039948.10941276119315214.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877612#action_12877612 ] 

Jitendra Nath Pandey commented on HADOOP-6815:
----------------------------------------------

1. javadoc annotation  @param cn   for method refreshSuperUserGroupsConfiguration(Configuration conf) may cause warning.
2. minor : It might be better to remove commented code //private static Configuration conf=null;


> refreshSuperUserGroupsConfiguration should use server side configuration for the refresh
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6815
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6815
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6815-1.patch, HADOOP-6815.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8997-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 22:50:44 2010
Return-Path: <common-issues-return-8997-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42090 invoked from network); 10 Jun 2010 22:50:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 22:50:44 -0000
Received: (qmail 45925 invoked by uid 500); 10 Jun 2010 22:50:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45898 invoked by uid 500); 10 Jun 2010 22:50:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45890 invoked by uid 99); 10 Jun 2010 22:50:44 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 22:50:44 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 22:50:42 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5AMoK9c005713
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 22:50:20 GMT
Message-ID: <10286081.33661276210220339.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 18:50:20 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6603) Provide workaround for issue with
 Kerberos not resolving cross-realm principal
In-Reply-To: <924517533.47301267493525772.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877613#action_12877613 ] 

Hadoop QA commented on HADOOP-6603:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446806/HADOOP-6603.2.patch
  against trunk revision 953388.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    -1 javac.  The applied patch generated 1032 javac compiler warnings (more than the trunk's current 1022 warnings).

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/578/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/578/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/578/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/578/console

This message is automatically generated.

> Provide workaround for issue with Kerberos not resolving cross-realm principal
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6603
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6603
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Jakob Homan
>            Assignee: Kan Zhang
>         Attachments: fix_comment_y20.patch, HADOOP-6603-Y20S-2.patch, HADOOP-6603-Y20S-3.patch, HADOOP-6603-Y20S-4.patch, HADOOP-6603-Y20S.patch, HADOOP-6603.1.patch, HADOOP-6603.2.patch
>
>
> Java's SSL-Kerberos implementation does not correctly obtain the principal for cross-realm principles when clients initiate connections to servers, resulting in the client being unable to authenticate the server.  We need a work-around until this bug gets fixed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8998-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 23:24:40 2010
Return-Path: <common-issues-return-8998-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51886 invoked from network); 10 Jun 2010 23:24:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 23:24:40 -0000
Received: (qmail 67535 invoked by uid 500); 10 Jun 2010 23:24:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67477 invoked by uid 500); 10 Jun 2010 23:24:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67469 invoked by uid 99); 10 Jun 2010 23:24:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 23:24:40 +0000
X-ASF-Spam-Status: No, hits=-1512.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 23:24:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5ANOJKv006115
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 23:24:19 GMT
Message-ID: <28036852.34531276212259647.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 19:24:19 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6794) Move configuration and script files
 post split
In-Reply-To: <11039662.93131275342217664.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877626#action_12877626 ] 

Hudson commented on HADOOP-6794:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #293 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/293/])
    HADOOP-6794. Move configuration and script files post split. Includes HDFS-1181, MAPREDUCE-1033.


> Move configuration and script files post split
> ----------------------------------------------
>
>                 Key: HADOOP-6794
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6794
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, scripts
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6794.patch, HADOOP-6794.patch, HADOOP-6794.patch, HADOOP-6794.patch, HADOOP-6794.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-8999-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 23:32:38 2010
Return-Path: <common-issues-return-8999-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53144 invoked from network); 10 Jun 2010 23:32:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 23:32:37 -0000
Received: (qmail 77854 invoked by uid 500); 10 Jun 2010 23:32:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77828 invoked by uid 500); 10 Jun 2010 23:32:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77820 invoked by uid 99); 10 Jun 2010 23:32:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 23:32:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 23:32:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5ANWDho006156
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 23:32:14 GMT
Message-ID: <33286627.34591276212733803.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 19:32:13 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6603) Provide workaround for issue with
 Kerberos not resolving cross-realm principal
In-Reply-To: <924517533.47301267493525772.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877629#action_12877629 ] 

Jitendra Nath Pandey commented on HADOOP-6603:
----------------------------------------------

Both javadoc and javac warnings are related to sun.security.krb5.Config 
"warning: sun.security.krb5.Config is Sun proprietary API and may be removed in a future release"

> Provide workaround for issue with Kerberos not resolving cross-realm principal
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6603
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6603
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Jakob Homan
>            Assignee: Kan Zhang
>         Attachments: fix_comment_y20.patch, HADOOP-6603-Y20S-2.patch, HADOOP-6603-Y20S-3.patch, HADOOP-6603-Y20S-4.patch, HADOOP-6603-Y20S.patch, HADOOP-6603.1.patch, HADOOP-6603.2.patch
>
>
> Java's SSL-Kerberos implementation does not correctly obtain the principal for cross-realm principles when clients initiate connections to servers, resulting in the client being unable to authenticate the server.  We need a work-around until this bug gets fixed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9000-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 23:46:35 2010
Return-Path: <common-issues-return-9000-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55572 invoked from network); 10 Jun 2010 23:46:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 23:46:35 -0000
Received: (qmail 87269 invoked by uid 500); 10 Jun 2010 23:46:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87230 invoked by uid 500); 10 Jun 2010 23:46:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87222 invoked by uid 99); 10 Jun 2010 23:46:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 23:46:34 +0000
X-ASF-Spam-Status: No, hits=-1512.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 23:46:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5ANkE5E006282
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 23:46:14 GMT
Message-ID: <21354501.34921276213574139.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 19:46:14 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6810) Extract a subset of tests for smoke
 (DOA) validation
In-Reply-To: <10151977.193211275695163699.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6810:
---------------------------------------

    Attachment: smoke-tests.patch.1

This patch goes on top of the previous one in order to have a separate target which runs test-core test set excluding that of commit and smoke tests. {{test-core}} behavior left intact and new target is added for the purpose of eliminating test run dups.
A separate combined patch for HDFS/MR trunks is coming

> Extract a subset of tests for smoke (DOA) validation
> ----------------------------------------------------
>
>                 Key: HADOOP-6810
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6810
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: smoke-tests.patch, smoke-tests.patch, smoke-tests.patch, smoke-tests.patch, smoke-tests.patch, smoke-tests.patch.1
>
>
> Currently, there's no way to quickly find out if a new build of Hadoop is reasonably viable and can be used for future development/testing cycle.
> Similarly to 'commit' testing it'd be great to have a wider subset of existing tests to be executed inside of 30 minutes. The main goal of such a subset is to have a reasonable guarantees that different parts of Hadoop's functionality aren't broken and the system is not DOA.
> Full test run takes about 3+ hours and it is a too long wait time in many cases. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9001-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 23:48:35 2010
Return-Path: <common-issues-return-9001-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55865 invoked from network); 10 Jun 2010 23:48:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 23:48:35 -0000
Received: (qmail 88230 invoked by uid 500); 10 Jun 2010 23:48:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88187 invoked by uid 500); 10 Jun 2010 23:48:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88179 invoked by uid 99); 10 Jun 2010 23:48:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 23:48:35 +0000
X-ASF-Spam-Status: No, hits=-1512.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 23:48:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5ANmEBQ006299
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 23:48:14 GMT
Message-ID: <12649214.34951276213694116.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 19:48:14 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6815) refreshSuperUserGroupsConfiguration
 should use server side configuration for the refresh
In-Reply-To: <20039948.10941276119315214.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6815:
-----------------------------------

    Attachment: HADOOP-6815-2.patch

addressed comments

> refreshSuperUserGroupsConfiguration should use server side configuration for the refresh
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6815
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6815
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6815-1.patch, HADOOP-6815-2.patch, HADOOP-6815.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9002-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 23:48:36 2010
Return-Path: <common-issues-return-9002-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55909 invoked from network); 10 Jun 2010 23:48:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 23:48:36 -0000
Received: (qmail 88374 invoked by uid 500); 10 Jun 2010 23:48:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88333 invoked by uid 500); 10 Jun 2010 23:48:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88325 invoked by uid 99); 10 Jun 2010 23:48:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 23:48:36 +0000
X-ASF-Spam-Status: No, hits=-1512.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 23:48:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5ANmFVj006311
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 23:48:15 GMT
Message-ID: <10168551.34991276213695600.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 19:48:15 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6794) Move configuration and script files
 post split
In-Reply-To: <11039662.93131275342217664.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6794:
------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

I've just committed this.

> Move configuration and script files post split
> ----------------------------------------------
>
>                 Key: HADOOP-6794
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6794
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, scripts
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6794.patch, HADOOP-6794.patch, HADOOP-6794.patch, HADOOP-6794.patch, HADOOP-6794.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9003-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 10 23:56:34 2010
Return-Path: <common-issues-return-9003-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56942 invoked from network); 10 Jun 2010 23:56:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 10 Jun 2010 23:56:34 -0000
Received: (qmail 91800 invoked by uid 500); 10 Jun 2010 23:56:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91772 invoked by uid 500); 10 Jun 2010 23:56:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91764 invoked by uid 99); 10 Jun 2010 23:56:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 23:56:34 +0000
X-ASF-Spam-Status: No, hits=-1512.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Jun 2010 23:56:33 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5ANuDaI006460
	for <common-issues@hadoop.apache.org>; Thu, 10 Jun 2010 23:56:13 GMT
Message-ID: <10517882.35421276214173535.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 19:56:13 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6403) Deprecate EC2 bash scripts
In-Reply-To: <1247543181.1259696120766.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6403:
------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

I've just committed this.

> Deprecate EC2 bash scripts
> --------------------------
>
>                 Key: HADOOP-6403
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6403
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: contrib/cloud
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6403.patch
>
>
> With the addition of python-based EC2 scripts introduced in HADOOP-6108, the bash scripts in src/contrib/ec2 should be deprecated.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9004-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 00:10:40 2010
Return-Path: <common-issues-return-9004-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63467 invoked from network); 11 Jun 2010 00:10:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 00:10:40 -0000
Received: (qmail 96372 invoked by uid 500); 11 Jun 2010 00:10:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96313 invoked by uid 500); 11 Jun 2010 00:10:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96305 invoked by uid 99); 11 Jun 2010 00:10:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 00:10:39 +0000
X-ASF-Spam-Status: No, hits=-1512.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 00:10:39 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5B0AJIj006745
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 00:10:19 GMT
Message-ID: <30113100.36061276215019152.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 20:10:19 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6810) Extract a subset of tests for smoke
 (DOA) validation
In-Reply-To: <10151977.193211275695163699.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877649#action_12877649 ] 

Jakob Homan commented on HADOOP-6810:
-------------------------------------

Updated patch looks good.  I notice line 12 is indented a couple characters too far.  I also just noticed that the smoke test list contains duplicates of from the commit list, meaning those tests will be run twice even with this change.

> Extract a subset of tests for smoke (DOA) validation
> ----------------------------------------------------
>
>                 Key: HADOOP-6810
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6810
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: smoke-tests.patch, smoke-tests.patch, smoke-tests.patch, smoke-tests.patch, smoke-tests.patch, smoke-tests.patch.1
>
>
> Currently, there's no way to quickly find out if a new build of Hadoop is reasonably viable and can be used for future development/testing cycle.
> Similarly to 'commit' testing it'd be great to have a wider subset of existing tests to be executed inside of 30 minutes. The main goal of such a subset is to have a reasonable guarantees that different parts of Hadoop's functionality aren't broken and the system is not DOA.
> Full test run takes about 3+ hours and it is a too long wait time in many cases. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9005-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 00:12:37 2010
Return-Path: <common-issues-return-9005-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63775 invoked from network); 11 Jun 2010 00:12:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 00:12:37 -0000
Received: (qmail 97891 invoked by uid 500); 11 Jun 2010 00:12:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97818 invoked by uid 500); 11 Jun 2010 00:12:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97810 invoked by uid 99); 11 Jun 2010 00:12:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 00:12:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 00:12:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5B0CDZx006763
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 00:12:13 GMT
Message-ID: <3206779.36101276215133608.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 20:12:13 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6403) Deprecate EC2 bash scripts
In-Reply-To: <1247543181.1259696120766.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877651#action_12877651 ] 

Hudson commented on HADOOP-6403:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #294 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/294/])
    HADOOP-6403.  Deprecate EC2 bash scripts.


> Deprecate EC2 bash scripts
> --------------------------
>
>                 Key: HADOOP-6403
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6403
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: contrib/cloud
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6403.patch
>
>
> With the addition of python-based EC2 scripts introduced in HADOOP-6108, the bash scripts in src/contrib/ec2 should be deprecated.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9006-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 00:19:35 2010
Return-Path: <common-issues-return-9006-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65057 invoked from network); 11 Jun 2010 00:19:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 00:19:35 -0000
Received: (qmail 281 invoked by uid 500); 11 Jun 2010 00:19:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 253 invoked by uid 500); 11 Jun 2010 00:19:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 245 invoked by uid 99); 11 Jun 2010 00:19:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 00:19:35 +0000
X-ASF-Spam-Status: No, hits=-1512.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 00:19:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5B0JECX006782
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 00:19:14 GMT
Message-ID: <3863875.36131276215554386.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 20:19:14 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6815) refreshSuperUserGroupsConfiguration
 should use server side configuration for the refresh
In-Reply-To: <20039948.10941276119315214.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877652#action_12877652 ] 

Jitendra Nath Pandey commented on HADOOP-6815:
----------------------------------------------

+1

> refreshSuperUserGroupsConfiguration should use server side configuration for the refresh
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6815
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6815
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6815-1.patch, HADOOP-6815-2.patch, HADOOP-6815.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9007-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 00:28:35 2010
Return-Path: <common-issues-return-9007-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66659 invoked from network); 11 Jun 2010 00:28:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 00:28:35 -0000
Received: (qmail 6994 invoked by uid 500); 11 Jun 2010 00:28:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6969 invoked by uid 500); 11 Jun 2010 00:28:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6961 invoked by uid 99); 11 Jun 2010 00:28:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 00:28:35 +0000
X-ASF-Spam-Status: No, hits=-1512.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 00:28:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5B0SE88006841
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 00:28:14 GMT
Message-ID: <21807138.36281276216094344.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 20:28:14 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6603) Provide workaround for issue with
 Kerberos not resolving cross-realm principal
In-Reply-To: <924517533.47301267493525772.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6603:
-----------------------------------

    Hadoop Flags: [Reviewed]

+1

> Provide workaround for issue with Kerberos not resolving cross-realm principal
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6603
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6603
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Jakob Homan
>            Assignee: Kan Zhang
>         Attachments: fix_comment_y20.patch, HADOOP-6603-Y20S-2.patch, HADOOP-6603-Y20S-3.patch, HADOOP-6603-Y20S-4.patch, HADOOP-6603-Y20S.patch, HADOOP-6603.1.patch, HADOOP-6603.2.patch
>
>
> Java's SSL-Kerberos implementation does not correctly obtain the principal for cross-realm principles when clients initiate connections to servers, resulting in the client being unable to authenticate the server.  We need a work-around until this bug gets fixed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9008-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 00:43:36 2010
Return-Path: <common-issues-return-9008-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72197 invoked from network); 11 Jun 2010 00:43:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 00:43:35 -0000
Received: (qmail 16588 invoked by uid 500); 11 Jun 2010 00:43:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16555 invoked by uid 500); 11 Jun 2010 00:43:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16547 invoked by uid 99); 11 Jun 2010 00:43:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 00:43:35 +0000
X-ASF-Spam-Status: No, hits=-1512.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 00:43:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5B0hFuG006985
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 00:43:15 GMT
Message-ID: <13344099.36651276216994982.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 20:43:14 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6603) Provide workaround for issue with
 Kerberos not resolving cross-realm principal
In-Reply-To: <924517533.47301267493525772.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6603:
--------------------------------

        Status: Resolved  (was: Patch Available)
      Assignee: Jitendra Nath Pandey  (was: Kan Zhang)
    Resolution: Fixed

I've committed this.  Resolving as fixed.

> Provide workaround for issue with Kerberos not resolving cross-realm principal
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6603
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6603
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Jakob Homan
>            Assignee: Jitendra Nath Pandey
>         Attachments: fix_comment_y20.patch, HADOOP-6603-Y20S-2.patch, HADOOP-6603-Y20S-3.patch, HADOOP-6603-Y20S-4.patch, HADOOP-6603-Y20S.patch, HADOOP-6603.1.patch, HADOOP-6603.2.patch
>
>
> Java's SSL-Kerberos implementation does not correctly obtain the principal for cross-realm principles when clients initiate connections to servers, resulting in the client being unable to authenticate the server.  We need a work-around until this bug gets fixed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9009-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 00:55:37 2010
Return-Path: <common-issues-return-9009-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77471 invoked from network); 11 Jun 2010 00:55:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 00:55:37 -0000
Received: (qmail 21437 invoked by uid 500); 11 Jun 2010 00:55:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21391 invoked by uid 500); 11 Jun 2010 00:55:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21383 invoked by uid 99); 11 Jun 2010 00:55:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 00:55:37 +0000
X-ASF-Spam-Status: No, hits=-1513.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 00:55:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5B0tGl5007121
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 00:55:16 GMT
Message-ID: <6310753.36971276217716071.JavaMail.jira@thor>
Date: Thu, 10 Jun 2010 20:55:16 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6603) Provide workaround for issue with
 Kerberos not resolving cross-realm principal
In-Reply-To: <924517533.47301267493525772.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877672#action_12877672 ] 

Hudson commented on HADOOP-6603:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #295 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/295/])
    HADOOP-6603. Provide workaround for issue with Kerberos not resolving cross-realm principal. Contributed by Kan Zhang and Jitendra Pandey.


> Provide workaround for issue with Kerberos not resolving cross-realm principal
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6603
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6603
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Jakob Homan
>            Assignee: Jitendra Nath Pandey
>         Attachments: fix_comment_y20.patch, HADOOP-6603-Y20S-2.patch, HADOOP-6603-Y20S-3.patch, HADOOP-6603-Y20S-4.patch, HADOOP-6603-Y20S.patch, HADOOP-6603.1.patch, HADOOP-6603.2.patch
>
>
> Java's SSL-Kerberos implementation does not correctly obtain the principal for cross-realm principles when clients initiate connections to servers, resulting in the client being unable to authenticate the server.  We need a work-around until this bug gets fixed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9010-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 06:58:04 2010
Return-Path: <common-issues-return-9010-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24862 invoked from network); 11 Jun 2010 04:38:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 04:38:41 -0000
Received: (qmail 27662 invoked by uid 500); 11 Jun 2010 04:38:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27534 invoked by uid 500); 11 Jun 2010 04:38:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27526 invoked by uid 99); 11 Jun 2010 04:38:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 04:38:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 04:38:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5B4cEtD008880
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 04:38:14 GMT
Message-ID: <21898350.38791276231094293.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 00:38:14 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6815) refreshSuperUserGroupsConfiguration
 should use server side configuration for the refresh
In-Reply-To: <20039948.10941276119315214.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877708#action_12877708 ] 

Vinod K V commented on HADOOP-6815:
-----------------------------------

refreshUserToGroupsMappings() is also having the same client side configuration problem. Is there a separate ticket for this or is it being done in this issue itself?

> refreshSuperUserGroupsConfiguration should use server side configuration for the refresh
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6815
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6815
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6815-1.patch, HADOOP-6815-2.patch, HADOOP-6815.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9011-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 06:59:33 2010
Return-Path: <common-issues-return-9011-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26722 invoked from network); 11 Jun 2010 05:12:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 05:12:42 -0000
Received: (qmail 41391 invoked by uid 500); 11 Jun 2010 05:12:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41297 invoked by uid 500); 11 Jun 2010 05:12:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40824 invoked by uid 99); 11 Jun 2010 05:12:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 05:12:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 05:12:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5B5CET7009146
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 05:12:14 GMT
Message-ID: <25211660.39321276233134711.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 01:12:14 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6819) [Herriot] Shell command for getting
 the new exceptions in the logs returning exitcode 1 after executing
 successfully.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

[Herriot] Shell command for getting the new exceptions in the logs returning exitcode 1 after executing successfully.
---------------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-6819
                 URL: https://issues.apache.org/jira/browse/HADOOP-6819
             Project: Hadoop Common
          Issue Type: Bug
          Components: test
            Reporter: Vinay Kumar Thota
            Assignee: Vinay Kumar Thota


I have found an corner case issue while executing the shell command for getting the new exceptions in the logs. The problem could see whenever the log files are empty and trying to get the new exceptions count, it returns the output is 0 ,however it returns the exitcode is 1 instead of 0. grep -vc in the command is not  giving the expected exitcode because its not handling the multiple in options.





-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9012-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 06:59:42 2010
Return-Path: <common-issues-return-9012-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26924 invoked from network); 11 Jun 2010 05:16:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 05:16:42 -0000
Received: (qmail 43828 invoked by uid 500); 11 Jun 2010 05:16:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 43046 invoked by uid 500); 11 Jun 2010 05:16:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 43035 invoked by uid 99); 11 Jun 2010 05:16:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 05:16:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 05:16:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5B5GF5w009184
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 05:16:15 GMT
Message-ID: <23228032.39441276233375561.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 01:16:15 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6819) [Herriot] Shell command for getting
 the new exceptions in the logs returning exitcode 1 after executing
 successfully.
In-Reply-To: <25211660.39321276233134711.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinay Kumar Thota updated HADOOP-6819:
--------------------------------------

    Attachment: 6819-ydist-security.patch

Patch for yahoo security dist branch.

> [Herriot] Shell command for getting the new exceptions in the logs returning exitcode 1 after executing successfully.
> ---------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6819
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6819
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6819-ydist-security.patch
>
>
> I have found an corner case issue while executing the shell command for getting the new exceptions in the logs. The problem could see whenever the log files are empty and trying to get the new exceptions count, it returns the output is 0 ,however it returns the exitcode is 1 instead of 0. grep -vc in the command is not  giving the expected exitcode because its not handling the multiple in options.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9013-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 07:00:11 2010
Return-Path: <common-issues-return-9013-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 27459 invoked from network); 11 Jun 2010 05:25:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 05:25:40 -0000
Received: (qmail 47933 invoked by uid 500); 11 Jun 2010 05:25:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47791 invoked by uid 500); 11 Jun 2010 05:25:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47780 invoked by uid 99); 11 Jun 2010 05:25:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 05:25:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 05:25:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5B5PEfo009276
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 05:25:14 GMT
Message-ID: <27250248.39661276233914604.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 01:25:14 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6819) [Herriot] Shell command for getting
 the new exceptions in the logs returning exitcode 1 after executing
 successfully.
In-Reply-To: <25211660.39321276233134711.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinay Kumar Thota updated HADOOP-6819:
--------------------------------------

    Attachment: HADOOP-6819.patch

Patch for trunk.

> [Herriot] Shell command for getting the new exceptions in the logs returning exitcode 1 after executing successfully.
> ---------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6819
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6819
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6819-ydist-security.patch, HADOOP-6819.patch
>
>
> I have found an corner case issue while executing the shell command for getting the new exceptions in the logs. The problem could see whenever the log files are empty and trying to get the new exceptions count, it returns the output is 0 ,however it returns the exitcode is 1 instead of 0. grep -vc in the command is not  giving the expected exitcode because its not handling the multiple in options.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9014-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 11:21:45 2010
Return-Path: <common-issues-return-9014-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85007 invoked from network); 11 Jun 2010 11:21:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 11:21:44 -0000
Received: (qmail 85280 invoked by uid 500); 11 Jun 2010 11:21:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84654 invoked by uid 500); 11 Jun 2010 11:21:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84645 invoked by uid 99); 11 Jun 2010 11:21:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 11:21:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 11:21:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BBLGW2012037
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 11:21:16 GMT
Message-ID: <24681949.42991276255276451.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 07:21:16 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6403) Deprecate EC2 bash scripts
In-Reply-To: <1247543181.1259696120766.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877785#action_12877785 ] 

Hudson commented on HADOOP-6403:
--------------------------------

Integrated in Hadoop-Common-trunk #363 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/363/])
    HADOOP-6403.  Deprecate EC2 bash scripts.


> Deprecate EC2 bash scripts
> --------------------------
>
>                 Key: HADOOP-6403
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6403
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: contrib/cloud
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6403.patch
>
>
> With the addition of python-based EC2 scripts introduced in HADOOP-6108, the bash scripts in src/contrib/ec2 should be deprecated.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9016-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 11:21:46 2010
Return-Path: <common-issues-return-9016-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85189 invoked from network); 11 Jun 2010 11:21:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 11:21:46 -0000
Received: (qmail 85627 invoked by uid 500); 11 Jun 2010 11:21:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85597 invoked by uid 500); 11 Jun 2010 11:21:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85589 invoked by uid 99); 11 Jun 2010 11:21:45 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 11:21:45 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 11:21:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BBLMKQ012107
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 11:21:22 GMT
Message-ID: <5879061.43221276255282138.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 07:21:22 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6526) Need mapping from long principal
 names to local OS user names
In-Reply-To: <786866304.151051264870594555.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877788#action_12877788 ] 

Hudson commented on HADOOP-6526:
--------------------------------

Integrated in Hadoop-Common-trunk #363 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/363/])
    HADOOP-6526. Need mapping from long principal names to local OS user names


> Need mapping from long principal names to local OS user names
> -------------------------------------------------------------
>
>                 Key: HADOOP-6526
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6526
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>         Attachments: 3595485.patch, c-6526-y20.patch, c-6526-y20.patch, c-6526.patch, HADOOP-6526-1.patch, HADOOP-6526-2.patch, HADOOP-6526-3.patch, HADOOP-6526-y20.2.patch, HADOOP-6526-y20.4.patch, HADOOP-6526.patch
>
>
> We need a configurable mapping from full user names (eg. omalley@APACHE.ORG) to local user names (eg. omalley). For many organizations it is sufficient to just use the prefix, however, in the case of shared clusters there may be duplicated prefixes. A configurable mapping will let administrators resolve the issue.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9015-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 11:21:46 2010
Return-Path: <common-issues-return-9015-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85159 invoked from network); 11 Jun 2010 11:21:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 11:21:45 -0000
Received: (qmail 85481 invoked by uid 500); 11 Jun 2010 11:21:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85337 invoked by uid 500); 11 Jun 2010 11:21:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85329 invoked by uid 99); 11 Jun 2010 11:21:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 11:21:44 +0000
X-ASF-Spam-Status: No, hits=-1513.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 11:21:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BBLNDK012132
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 11:21:23 GMT
Message-ID: <15847778.43301276255283708.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 07:21:23 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6794) Move configuration and script files
 post split
In-Reply-To: <11039662.93131275342217664.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877790#action_12877790 ] 

Hudson commented on HADOOP-6794:
--------------------------------

Integrated in Hadoop-Common-trunk #363 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/363/])
    HADOOP-6794. Move configuration and script files post split. Includes HDFS-1181, MAPREDUCE-1033.


> Move configuration and script files post split
> ----------------------------------------------
>
>                 Key: HADOOP-6794
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6794
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, scripts
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6794.patch, HADOOP-6794.patch, HADOOP-6794.patch, HADOOP-6794.patch, HADOOP-6794.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9017-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 11:21:47 2010
Return-Path: <common-issues-return-9017-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85278 invoked from network); 11 Jun 2010 11:21:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 11:21:46 -0000
Received: (qmail 85781 invoked by uid 500); 11 Jun 2010 11:21:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85742 invoked by uid 500); 11 Jun 2010 11:21:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85734 invoked by uid 99); 11 Jun 2010 11:21:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 11:21:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 11:21:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BBLMux012119
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 11:21:22 GMT
Message-ID: <7190211.43261276255282740.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 07:21:22 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6603) Provide workaround for issue with
 Kerberos not resolving cross-realm principal
In-Reply-To: <924517533.47301267493525772.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877789#action_12877789 ] 

Hudson commented on HADOOP-6603:
--------------------------------

Integrated in Hadoop-Common-trunk #363 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/363/])
    HADOOP-6603. Provide workaround for issue with Kerberos not resolving cross-realm principal. Contributed by Kan Zhang and Jitendra Pandey.


> Provide workaround for issue with Kerberos not resolving cross-realm principal
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-6603
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6603
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>            Reporter: Jakob Homan
>            Assignee: Jitendra Nath Pandey
>         Attachments: fix_comment_y20.patch, HADOOP-6603-Y20S-2.patch, HADOOP-6603-Y20S-3.patch, HADOOP-6603-Y20S-4.patch, HADOOP-6603-Y20S.patch, HADOOP-6603.1.patch, HADOOP-6603.2.patch
>
>
> Java's SSL-Kerberos implementation does not correctly obtain the principal for cross-realm principles when clients initiate connections to servers, resulting in the client being unable to authenticate the server.  We need a work-around until this bug gets fixed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9018-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 12:30:41 2010
Return-Path: <common-issues-return-9018-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12155 invoked from network); 11 Jun 2010 12:30:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 12:30:41 -0000
Received: (qmail 81880 invoked by uid 500); 11 Jun 2010 12:30:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80879 invoked by uid 500); 11 Jun 2010 12:30:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80607 invoked by uid 99); 11 Jun 2010 12:30:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 12:30:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 12:30:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BCUDlQ012644
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 12:30:13 GMT
Message-ID: <27327029.43761276259413643.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 08:30:13 -0400 (EDT)
From: "Alexander Bondar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6820) RunJar fails executing thousands JARs
 within single JVM with error "Too many open files"
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

RunJar fails executing thousands JARs within single JVM with error "Too many open files"
----------------------------------------------------------------------------------------

                 Key: HADOOP-6820
                 URL: https://issues.apache.org/jira/browse/HADOOP-6820
             Project: Hadoop Common
          Issue Type: Bug
          Components: util
    Affects Versions: 0.20.2
         Environment: OS:Linux, Linux-user limited by maximum number of open file descriptors (for example: ulimit -n shows 1024)
            Reporter: Alexander Bondar
            Priority: Minor


According to Sun JVM (up to 7) bug http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4167874 - The JarFile objects created by sun.net.www.protocol.jar.JarFileFactory never get garbage collected, even if the classloader that loaded them goes away.

So, if linux-user has limitation on maximum number of open file descriptors (for example: ulimit -n shows 1024) and performs RunJar.main(...) over thousands of JARs that include other nested JARs (also loaded by ClassLoader) within single JVM, RunJar.main(...) throws following exception: java.lang.RuntimeException: java.io.FileNotFoundException: /some-file.txt (Too many open files)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9019-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 14:02:38 2010
Return-Path: <common-issues-return-9019-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36674 invoked from network); 11 Jun 2010 14:02:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 14:02:37 -0000
Received: (qmail 95892 invoked by uid 500); 11 Jun 2010 14:02:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95763 invoked by uid 500); 11 Jun 2010 14:02:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95755 invoked by uid 99); 11 Jun 2010 14:02:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 14:02:36 +0000
X-ASF-Spam-Status: No, hits=-1514.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 14:02:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BE2G9v013329
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 14:02:16 GMT
Message-ID: <10316430.44351276264936110.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 10:02:16 -0400 (EDT)
From: "Alexander Bondar (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6820) RunJar fails executing thousands JARs
 within single JVM with error "Too many open files"
In-Reply-To: <27327029.43761276259413643.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexander Bondar updated HADOOP-6820:
-------------------------------------

    Attachment: HADOOP-6820.patch

RunJar unit-test and bug-fix for trunk

> RunJar fails executing thousands JARs within single JVM with error "Too many open files"
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6820
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: util
>    Affects Versions: 0.20.2
>         Environment: OS:Linux, Linux-user limited by maximum number of open file descriptors (for example: ulimit -n shows 1024)
>            Reporter: Alexander Bondar
>            Priority: Minor
>         Attachments: HADOOP-6820.patch
>
>
> According to Sun JVM (up to 7) bug http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4167874 - The JarFile objects created by sun.net.www.protocol.jar.JarFileFactory never get garbage collected, even if the classloader that loaded them goes away.
> So, if linux-user has limitation on maximum number of open file descriptors (for example: ulimit -n shows 1024) and performs RunJar.main(...) over thousands of JARs that include other nested JARs (also loaded by ClassLoader) within single JVM, RunJar.main(...) throws following exception: java.lang.RuntimeException: java.io.FileNotFoundException: /some-file.txt (Too many open files)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9020-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 17:30:39 2010
Return-Path: <common-issues-return-9020-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12095 invoked from network); 11 Jun 2010 17:30:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 17:30:36 -0000
Received: (qmail 98460 invoked by uid 500); 11 Jun 2010 17:30:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98262 invoked by uid 500); 11 Jun 2010 17:30:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98039 invoked by uid 99); 11 Jun 2010 17:30:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 17:30:36 +0000
X-ASF-Spam-Status: No, hits=-1514.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 17:30:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BHUEfw015006
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 17:30:15 GMT
Message-ID: <6410741.46811276277414864.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 13:30:14 -0400 (EDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6821) Document changes to memory monitoring
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Document changes to memory monitoring
-------------------------------------

                 Key: HADOOP-6821
                 URL: https://issues.apache.org/jira/browse/HADOOP-6821
             Project: Hadoop Common
          Issue Type: Bug
          Components: documentation
    Affects Versions: 0.21.0
            Reporter: Hemanth Yamijala
            Assignee: Hemanth Yamijala
             Fix For: 0.21.0


Modify the cluster_setup guide with information about memory monitoring and admin configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9021-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 17:32:36 2010
Return-Path: <common-issues-return-9021-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13128 invoked from network); 11 Jun 2010 17:32:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 17:32:35 -0000
Received: (qmail 2323 invoked by uid 500); 11 Jun 2010 17:32:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2263 invoked by uid 500); 11 Jun 2010 17:32:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2255 invoked by uid 99); 11 Jun 2010 17:32:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 17:32:35 +0000
X-ASF-Spam-Status: No, hits=-1514.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 17:32:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BHWE7H015042
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 17:32:15 GMT
Message-ID: <32814310.46891276277534836.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 13:32:14 -0400 (EDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6821) Document changes to memory
 monitoring
In-Reply-To: <6410741.46811276277414864.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877869#action_12877869 ] 

Hemanth Yamijala commented on HADOOP-6821:
------------------------------------------

This was being developed as part of MAPREDUCE-1018. Following the split of the documentation files in MAPREDUCE-1404, the cluster_setup guide moved to the Hadoop common project. Hence that part of the patch in MAPREDUCE-1018 needs to move as well.

> Document changes to memory monitoring
> -------------------------------------
>
>                 Key: HADOOP-6821
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6821
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.21.0
>            Reporter: Hemanth Yamijala
>            Assignee: Hemanth Yamijala
>             Fix For: 0.21.0
>
>
> Modify the cluster_setup guide with information about memory monitoring and admin configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9022-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 17:46:38 2010
Return-Path: <common-issues-return-9022-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18092 invoked from network); 11 Jun 2010 17:46:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 17:46:38 -0000
Received: (qmail 19000 invoked by uid 500); 11 Jun 2010 17:46:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18943 invoked by uid 500); 11 Jun 2010 17:46:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18935 invoked by uid 99); 11 Jun 2010 17:46:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 17:46:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 17:46:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BHkEft015261
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 17:46:14 GMT
Message-ID: <29819675.47431276278374403.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 13:46:14 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877878#action_12877878 ] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6796:
------------------------------------------------

The patch causes TestCopyFiles failing; see MAPREDUCE-1858.  We probably should revert it because the constructor FileStatus() should not be changed.

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.1.patch, HADOOP-6796.2.patch, HADOOP-6796.3.patch, HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9023-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 17:50:35 2010
Return-Path: <common-issues-return-9023-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20983 invoked from network); 11 Jun 2010 17:50:34 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 17:50:34 -0000
Received: (qmail 26549 invoked by uid 500); 11 Jun 2010 17:50:34 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26525 invoked by uid 500); 11 Jun 2010 17:50:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26517 invoked by uid 99); 11 Jun 2010 17:50:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 17:50:34 +0000
X-ASF-Spam-Status: No, hits=-1514.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 17:50:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BHoDuN015352
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 17:50:14 GMT
Message-ID: <12393918.47691276278613823.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 13:50:13 -0400 (EDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6821) Document changes to memory monitoring
In-Reply-To: <6410741.46811276277414864.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala updated HADOOP-6821:
-------------------------------------

    Attachment: HADOOP-6821.patch

The attached patch moves the changes to cluster_setup.xml that were being tracked in MAPREDUCE-1018 over to this JIRA. It also makes necessary adjustments to external links.

> Document changes to memory monitoring
> -------------------------------------
>
>                 Key: HADOOP-6821
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6821
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.21.0
>            Reporter: Hemanth Yamijala
>            Assignee: Hemanth Yamijala
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6821.patch
>
>
> Modify the cluster_setup guide with information about memory monitoring and admin configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9024-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 17:52:35 2010
Return-Path: <common-issues-return-9024-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23164 invoked from network); 11 Jun 2010 17:52:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 17:52:35 -0000
Received: (qmail 31084 invoked by uid 500); 11 Jun 2010 17:52:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31059 invoked by uid 500); 11 Jun 2010 17:52:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31051 invoked by uid 99); 11 Jun 2010 17:52:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 17:52:35 +0000
X-ASF-Spam-Status: No, hits=-1514.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 17:52:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BHqEje015376
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 17:52:14 GMT
Message-ID: <10024704.47711276278734159.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 13:52:14 -0400 (EDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6821) Document changes to memory monitoring
In-Reply-To: <6410741.46811276277414864.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala updated HADOOP-6821:
-------------------------------------

    Status: Patch Available  (was: Open)

> Document changes to memory monitoring
> -------------------------------------
>
>                 Key: HADOOP-6821
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6821
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.21.0
>            Reporter: Hemanth Yamijala
>            Assignee: Hemanth Yamijala
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6821.patch
>
>
> Modify the cluster_setup guide with information about memory monitoring and admin configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9025-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 17:52:38 2010
Return-Path: <common-issues-return-9025-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23212 invoked from network); 11 Jun 2010 17:52:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 17:52:38 -0000
Received: (qmail 31235 invoked by uid 500); 11 Jun 2010 17:52:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31208 invoked by uid 500); 11 Jun 2010 17:52:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31200 invoked by uid 99); 11 Jun 2010 17:52:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 17:52:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 17:52:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BHqEvW015382
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 17:52:14 GMT
Message-ID: <1799847.47731276278734684.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 13:52:14 -0400 (EDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6821) Document changes to memory monitoring
In-Reply-To: <6410741.46811276277414864.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala updated HADOOP-6821:
-------------------------------------

    Priority: Blocker  (was: Major)

Marking as blocker for 0.21, so changes can go in along with MAPREDUCE-1018. Both of these are necessary for users to make sense of the memory related options Hadoop offers.

> Document changes to memory monitoring
> -------------------------------------
>
>                 Key: HADOOP-6821
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6821
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.21.0
>            Reporter: Hemanth Yamijala
>            Assignee: Hemanth Yamijala
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6821.patch
>
>
> Modify the cluster_setup guide with information about memory monitoring and admin configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9026-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 17:54:36 2010
Return-Path: <common-issues-return-9026-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23560 invoked from network); 11 Jun 2010 17:54:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 17:54:36 -0000
Received: (qmail 32457 invoked by uid 500); 11 Jun 2010 17:54:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32332 invoked by uid 500); 11 Jun 2010 17:54:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32323 invoked by uid 99); 11 Jun 2010 17:54:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 17:54:36 +0000
X-ASF-Spam-Status: No, hits=-1514.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 17:54:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BHsFsF015417
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 17:54:15 GMT
Message-ID: <7838234.47841276278855843.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 13:54:15 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6814) Method in UGI to get the
 authentication method of the real user.
In-Reply-To: <29484910.14751275951732068.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6814:
-----------------------------------------

    Status: Open  (was: Patch Available)

> Method in UGI to get the authentication method of the real user.
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6814
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6814
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6814.1.patch, HADOOP-6814.2.patch
>
>
> UGI should have a method to return the authentication method of the real user for a proxy-user scenario.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9027-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 17:54:37 2010
Return-Path: <common-issues-return-9027-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23584 invoked from network); 11 Jun 2010 17:54:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 17:54:36 -0000
Received: (qmail 32514 invoked by uid 500); 11 Jun 2010 17:54:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32477 invoked by uid 500); 11 Jun 2010 17:54:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32469 invoked by uid 99); 11 Jun 2010 17:54:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 17:54:36 +0000
X-ASF-Spam-Status: No, hits=-1514.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 17:54:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BHsGe3015423
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 17:54:16 GMT
Message-ID: <16616188.47861276278856077.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 13:54:16 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6814) Method in UGI to get the
 authentication method of the real user.
In-Reply-To: <29484910.14751275951732068.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6814:
-----------------------------------------

    Status: Patch Available  (was: Open)

> Method in UGI to get the authentication method of the real user.
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6814
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6814
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6814.1.patch, HADOOP-6814.2.patch
>
>
> UGI should have a method to return the authentication method of the real user for a proxy-user scenario.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9028-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 17:54:38 2010
Return-Path: <common-issues-return-9028-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23650 invoked from network); 11 Jun 2010 17:54:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 17:54:38 -0000
Received: (qmail 32667 invoked by uid 500); 11 Jun 2010 17:54:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32629 invoked by uid 500); 11 Jun 2010 17:54:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32621 invoked by uid 99); 11 Jun 2010 17:54:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 17:54:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 17:54:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BHsDPW015402
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 17:54:14 GMT
Message-ID: <30403305.47791276278853548.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 13:54:13 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6814) Method in UGI to get the
 authentication method of the real user.
In-Reply-To: <29484910.14751275951732068.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6814:
-----------------------------------------

    Attachment: HADOOP-6814.2.patch

New patch addressing the comment. The name of the method is changed to getRealAuthenticationMethod.

> Method in UGI to get the authentication method of the real user.
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6814
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6814
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6814.1.patch, HADOOP-6814.2.patch
>
>
> UGI should have a method to return the authentication method of the real user for a proxy-user scenario.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9029-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 18:34:36 2010
Return-Path: <common-issues-return-9029-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51999 invoked from network); 11 Jun 2010 18:34:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 18:34:36 -0000
Received: (qmail 94428 invoked by uid 500); 11 Jun 2010 18:34:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94386 invoked by uid 500); 11 Jun 2010 18:34:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94373 invoked by uid 99); 11 Jun 2010 18:34:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 18:34:35 +0000
X-ASF-Spam-Status: No, hits=-1514.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 18:34:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BIYE7G016045
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 18:34:15 GMT
Message-ID: <32678937.49031276281254587.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 14:34:14 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6814) Method in UGI to get the
 authentication method of the real user.
In-Reply-To: <29484910.14751275951732068.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877906#action_12877906 ] 

Hadoop QA commented on HADOOP-6814:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446884/HADOOP-6814.2.patch
  against trunk revision 953517.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/78/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/78/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/78/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/78/console

This message is automatically generated.

> Method in UGI to get the authentication method of the real user.
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6814
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6814
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6814.1.patch, HADOOP-6814.2.patch
>
>
> UGI should have a method to return the authentication method of the real user for a proxy-user scenario.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9030-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 18:38:40 2010
Return-Path: <common-issues-return-9030-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53271 invoked from network); 11 Jun 2010 18:38:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 18:38:39 -0000
Received: (qmail 1803 invoked by uid 500); 11 Jun 2010 18:38:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1773 invoked by uid 500); 11 Jun 2010 18:38:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1765 invoked by uid 99); 11 Jun 2010 18:38:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 18:38:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 18:38:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BIcFgi016113
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 18:38:15 GMT
Message-ID: <27020758.49171276281495348.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 14:38:15 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6821) Document changes to memory
 monitoring
In-Reply-To: <6410741.46811276277414864.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877908#action_12877908 ] 

Hadoop QA commented on HADOOP-6821:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446883/HADOOP-6821.patch
  against trunk revision 953517.

    +1 @author.  The patch does not contain any @author tags.

    +0 tests included.  The patch appears to be a documentation patch that doesn't require tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/579/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/579/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/579/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/579/console

This message is automatically generated.

> Document changes to memory monitoring
> -------------------------------------
>
>                 Key: HADOOP-6821
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6821
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.21.0
>            Reporter: Hemanth Yamijala
>            Assignee: Hemanth Yamijala
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6821.patch
>
>
> Modify the cluster_setup guide with information about memory monitoring and admin configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9031-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 18:40:45 2010
Return-Path: <common-issues-return-9031-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55457 invoked from network); 11 Jun 2010 18:40:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 18:40:35 -0000
Received: (qmail 7829 invoked by uid 500); 11 Jun 2010 18:40:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7801 invoked by uid 500); 11 Jun 2010 18:40:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7793 invoked by uid 99); 11 Jun 2010 18:40:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 18:40:35 +0000
X-ASF-Spam-Status: No, hits=-1514.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 18:40:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BIeEsF016150
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 18:40:14 GMT
Message-ID: <32793558.49231276281614335.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 14:40:14 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6821) Document changes to memory
 monitoring
In-Reply-To: <6410741.46811276277414864.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877909#action_12877909 ] 

Tom White commented on HADOOP-6821:
-----------------------------------

+1

> Document changes to memory monitoring
> -------------------------------------
>
>                 Key: HADOOP-6821
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6821
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.21.0
>            Reporter: Hemanth Yamijala
>            Assignee: Hemanth Yamijala
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6821.patch
>
>
> Modify the cluster_setup guide with information about memory monitoring and admin configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9032-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 19:43:41 2010
Return-Path: <common-issues-return-9032-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79695 invoked from network); 11 Jun 2010 19:43:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 19:43:40 -0000
Received: (qmail 88818 invoked by uid 500); 11 Jun 2010 19:43:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88736 invoked by uid 500); 11 Jun 2010 19:43:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88728 invoked by uid 99); 11 Jun 2010 19:43:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 19:43:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 19:43:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BJhFYa016710
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 19:43:16 GMT
Message-ID: <20291600.50121276285395577.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 15:43:15 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6814) Method in UGI to get the
 authentication method of the real user.
In-Reply-To: <29484910.14751275951732068.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6814:
-----------------------------------------

    Status: Open  (was: Patch Available)

> Method in UGI to get the authentication method of the real user.
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6814
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6814
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6814.1.patch, HADOOP-6814.2.patch
>
>
> UGI should have a method to return the authentication method of the real user for a proxy-user scenario.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9033-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 19:43:41 2010
Return-Path: <common-issues-return-9033-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79691 invoked from network); 11 Jun 2010 19:43:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 19:43:40 -0000
Received: (qmail 88819 invoked by uid 500); 11 Jun 2010 19:43:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88746 invoked by uid 500); 11 Jun 2010 19:43:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88737 invoked by uid 99); 11 Jun 2010 19:43:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 19:43:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 19:43:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BJhGGB016716
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 19:43:16 GMT
Message-ID: <10554475.50141276285396280.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 15:43:16 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6814) Method in UGI to get the
 authentication method of the real user.
In-Reply-To: <29484910.14751275951732068.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jitendra Nath Pandey updated HADOOP-6814:
-----------------------------------------

    Status: Patch Available  (was: Open)

> Method in UGI to get the authentication method of the real user.
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6814
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6814
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6814.1.patch, HADOOP-6814.2.patch
>
>
> UGI should have a method to return the authentication method of the real user for a proxy-user scenario.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9034-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 20:10:49 2010
Return-Path: <common-issues-return-9034-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94954 invoked from network); 11 Jun 2010 20:10:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 20:10:49 -0000
Received: (qmail 41784 invoked by uid 500); 11 Jun 2010 20:10:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 41685 invoked by uid 500); 11 Jun 2010 20:10:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 41672 invoked by uid 99); 11 Jun 2010 20:10:48 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 20:10:48 +0000
X-ASF-Spam-Status: No, hits=-1514.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 20:10:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BKAQ1T016978
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 20:10:27 GMT
Message-ID: <352366.50481276287026952.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 16:10:26 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6814) Method in UGI to get the
 authentication method of the real user.
In-Reply-To: <29484910.14751275951732068.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877947#action_12877947 ] 

Hadoop QA commented on HADOOP-6814:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446884/HADOOP-6814.2.patch
  against trunk revision 953517.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/580/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/580/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/580/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/580/console

This message is automatically generated.

> Method in UGI to get the authentication method of the real user.
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6814
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6814
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6814.1.patch, HADOOP-6814.2.patch
>
>
> UGI should have a method to return the authentication method of the real user for a proxy-user scenario.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9035-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 20:31:50 2010
Return-Path: <common-issues-return-9035-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 551 invoked from network); 11 Jun 2010 20:31:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 20:31:50 -0000
Received: (qmail 57792 invoked by uid 500); 11 Jun 2010 20:31:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 57670 invoked by uid 500); 11 Jun 2010 20:31:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 57662 invoked by uid 99); 11 Jun 2010 20:31:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 20:31:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 20:31:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BKVPlc017161
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 20:31:25 GMT
Message-ID: <23451879.50881276288285211.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 16:31:25 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6814) Method in UGI to get the
 authentication method of the real user.
In-Reply-To: <29484910.14751275951732068.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877954#action_12877954 ] 

Jitendra Nath Pandey commented on HADOOP-6814:
----------------------------------------------

Javadoc warnings are unrelated to this patch.

Following warnings are showing up in console output

  [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:29: warning: sun.security.krb5.Config is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.Config;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:30: warning: sun.security.krb5.KrbException is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.KrbException;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:77: warning: sun.security.krb5.Config is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc]   private static Config kerbConf;
     [exec]   [javadoc]                  ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/SecurityUtil.java:31: warning: sun.security.jgss.krb5.Krb5Util is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.jgss.krb5.Krb5Util;
     [exec]   [javadoc]                              ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/SecurityUtil.java:32: warning: sun.security.krb5.Credentials is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.Credentials;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/SecurityUtil.java:33: warning: sun.security.krb5.PrincipalName is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.PrincipalName;


> Method in UGI to get the authentication method of the real user.
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6814
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6814
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6814.1.patch, HADOOP-6814.2.patch
>
>
> UGI should have a method to return the authentication method of the real user for a proxy-user scenario.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9036-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 20:35:40 2010
Return-Path: <common-issues-return-9036-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 975 invoked from network); 11 Jun 2010 20:35:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 20:35:39 -0000
Received: (qmail 62095 invoked by uid 500); 11 Jun 2010 20:35:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62043 invoked by uid 500); 11 Jun 2010 20:35:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62031 invoked by uid 99); 11 Jun 2010 20:35:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 20:35:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 20:35:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BKZEs1017175
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 20:35:15 GMT
Message-ID: <13159377.50901276288514950.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 16:35:14 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6814) Method in UGI to get the
 authentication method of the real user.
In-Reply-To: <29484910.14751275951732068.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6814:
--------------------------------

        Status: Resolved  (was: Patch Available)
    Resolution: Fixed

I just committed this. Thanks, Jitendra!

> Method in UGI to get the authentication method of the real user.
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6814
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6814
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6814.1.patch, HADOOP-6814.2.patch
>
>
> UGI should have a method to return the authentication method of the real user for a proxy-user scenario.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9037-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 20:41:36 2010
Return-Path: <common-issues-return-9037-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1922 invoked from network); 11 Jun 2010 20:41:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 20:41:36 -0000
Received: (qmail 67971 invoked by uid 500); 11 Jun 2010 20:41:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67932 invoked by uid 500); 11 Jun 2010 20:41:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67924 invoked by uid 99); 11 Jun 2010 20:41:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 20:41:35 +0000
X-ASF-Spam-Status: No, hits=-1515.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 20:41:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BKfEMW017226
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 20:41:15 GMT
Message-ID: <19381637.51031276288874891.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 16:41:14 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6819) [Herriot] Shell command for getting
 the new exceptions in the logs returning exitcode 1 after executing
 successfully.
In-Reply-To: <25211660.39321276233134711.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6819:
---------------------------------------

               Status: Patch Available  (was: Open)
         Hadoop Flags: [Reviewed]
    Affects Version/s: 0.21.0

+1 patch looks good. 
Submitting to verify the trunk version of it.

> [Herriot] Shell command for getting the new exceptions in the logs returning exitcode 1 after executing successfully.
> ---------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6819
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6819
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6819-ydist-security.patch, HADOOP-6819.patch
>
>
> I have found an corner case issue while executing the shell command for getting the new exceptions in the logs. The problem could see whenever the log files are empty and trying to get the new exceptions count, it returns the output is 0 ,however it returns the exitcode is 1 instead of 0. grep -vc in the command is not  giving the expected exitcode because its not handling the multiple in options.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9038-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 20:44:41 2010
Return-Path: <common-issues-return-9038-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2631 invoked from network); 11 Jun 2010 20:44:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 20:44:41 -0000
Received: (qmail 69690 invoked by uid 500); 11 Jun 2010 20:44:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 69638 invoked by uid 500); 11 Jun 2010 20:44:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 69630 invoked by uid 99); 11 Jun 2010 20:44:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 20:44:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 20:44:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BKiFah017269
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 20:44:16 GMT
Message-ID: <28025081.51121276289055950.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 16:44:15 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6668:
------------------------------

    Attachment: HADOOP-6668-0.21.patch
                HADOOP-6668.patch

An updated patch for trunk (the old one fell out of date), and one for the 0.21 branch.

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: common.interface-tw-annotated.txt, HADOOP-6668-0.21.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9039-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 20:53:36 2010
Return-Path: <common-issues-return-9039-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3872 invoked from network); 11 Jun 2010 20:53:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 20:53:35 -0000
Received: (qmail 79578 invoked by uid 500); 11 Jun 2010 20:53:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79555 invoked by uid 500); 11 Jun 2010 20:53:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79547 invoked by uid 99); 11 Jun 2010 20:53:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 20:53:34 +0000
X-ASF-Spam-Status: No, hits=-1515.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 20:53:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BKrDwo017305
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 20:53:14 GMT
Message-ID: <17058801.51181276289593583.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 16:53:13 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6814) Method in UGI to get the
 authentication method of the real user.
In-Reply-To: <29484910.14751275951732068.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877962#action_12877962 ] 

Hudson commented on HADOOP-6814:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #296 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/296/])
    HADOOP-6814. Adds an API in UserGroupInformation to get the real authentication method of a passed UGI. Contributed by Jitendra Pandey.


> Method in UGI to get the authentication method of the real user.
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6814
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6814
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6814.1.patch, HADOOP-6814.2.patch
>
>
> UGI should have a method to return the authentication method of the real user for a proxy-user scenario.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9040-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 21:15:42 2010
Return-Path: <common-issues-return-9040-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12658 invoked from network); 11 Jun 2010 21:15:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 21:15:42 -0000
Received: (qmail 9772 invoked by uid 500); 11 Jun 2010 21:15:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9691 invoked by uid 500); 11 Jun 2010 21:15:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9683 invoked by uid 99); 11 Jun 2010 21:15:41 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 21:15:41 +0000
X-ASF-Spam-Status: No, hits=-1515.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 21:15:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BLFKga017816
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 21:15:21 GMT
Message-ID: <33181040.52441276290920644.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 17:15:20 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6819) [Herriot] Shell command for getting
 the new exceptions in the logs returning exitcode 1 after executing
 successfully.
In-Reply-To: <25211660.39321276233134711.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877967#action_12877967 ] 

Hadoop QA commented on HADOOP-6819:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446839/HADOOP-6819.patch
  against trunk revision 953863.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/581/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/581/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/581/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/581/console

This message is automatically generated.

> [Herriot] Shell command for getting the new exceptions in the logs returning exitcode 1 after executing successfully.
> ---------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6819
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6819
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6819-ydist-security.patch, HADOOP-6819.patch
>
>
> I have found an corner case issue while executing the shell command for getting the new exceptions in the logs. The problem could see whenever the log files are empty and trying to get the new exceptions count, it returns the output is 0 ,however it returns the exitcode is 1 instead of 0. grep -vc in the command is not  giving the expected exitcode because its not handling the multiple in options.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9041-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 21:19:37 2010
Return-Path: <common-issues-return-9041-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13318 invoked from network); 11 Jun 2010 21:19:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 21:19:36 -0000
Received: (qmail 12189 invoked by uid 500); 11 Jun 2010 21:19:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12125 invoked by uid 500); 11 Jun 2010 21:19:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12110 invoked by uid 99); 11 Jun 2010 21:19:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 21:19:36 +0000
X-ASF-Spam-Status: No, hits=-1515.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 21:19:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BLJFF4017929
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 21:19:15 GMT
Message-ID: <9059326.52741276291155250.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 17:19:15 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6813) Add a new newInstance method in
 FileSystem that takes a "user" as argument
In-Reply-To: <17176494.13701275948371739.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6813:
------------------------------

     Release Note: I've just committed this to 0.21.
    Fix Version/s:     (was: 0.22.0)

> Add a new newInstance method in FileSystem that takes a "user" as argument
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6813
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6813
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: h-6813.patch
>
>
> In order to implement HDFS-1000 for trunk, I need to have a newInstance API in FileSystem that takes a "user" as an argument.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9042-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 21:19:39 2010
Return-Path: <common-issues-return-9042-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13438 invoked from network); 11 Jun 2010 21:19:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 21:19:39 -0000
Received: (qmail 12495 invoked by uid 500); 11 Jun 2010 21:19:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12424 invoked by uid 500); 11 Jun 2010 21:19:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12416 invoked by uid 99); 11 Jun 2010 21:19:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 21:19:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 21:19:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BLJE3R017920
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 21:19:14 GMT
Message-ID: <24638575.52711276291154672.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 17:19:14 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6769) Add an API in FileSystem to get
 FileSystem instances based on users
In-Reply-To: <15183681.91401274128662681.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6769:
------------------------------

    Fix Version/s:     (was: 0.22.0)

I've just committed this to 0.21.

> Add an API in FileSystem to get FileSystem instances based on users
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6769
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6769
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: 6769-1.patch
>
>
> HDFS-1000 requires an API in FileSystem to get FileSystem instances based on users. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9043-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 21:34:36 2010
Return-Path: <common-issues-return-9043-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16374 invoked from network); 11 Jun 2010 21:34:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 21:34:36 -0000
Received: (qmail 30449 invoked by uid 500); 11 Jun 2010 21:34:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30361 invoked by uid 500); 11 Jun 2010 21:34:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30352 invoked by uid 99); 11 Jun 2010 21:34:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 21:34:35 +0000
X-ASF-Spam-Status: No, hits=-1515.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 21:34:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BLYE4w018083
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 21:34:14 GMT
Message-ID: <10605187.53141276292054461.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 17:34:14 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6813) Add a new newInstance method in
 FileSystem that takes a "user" as argument
In-Reply-To: <17176494.13701275948371739.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877977#action_12877977 ] 

Hudson commented on HADOOP-6813:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #297 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/297/])
    Move entries for HADOOP-6769 and HADOOP-6813 to 0.21 section of CHANGES.txt


> Add a new newInstance method in FileSystem that takes a "user" as argument
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6813
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6813
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: h-6813.patch
>
>
> In order to implement HDFS-1000 for trunk, I need to have a newInstance API in FileSystem that takes a "user" as an argument.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9044-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 21:34:36 2010
Return-Path: <common-issues-return-9044-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16418 invoked from network); 11 Jun 2010 21:34:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 21:34:36 -0000
Received: (qmail 30542 invoked by uid 500); 11 Jun 2010 21:34:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30371 invoked by uid 500); 11 Jun 2010 21:34:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30362 invoked by uid 99); 11 Jun 2010 21:34:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 21:34:35 +0000
X-ASF-Spam-Status: No, hits=-1515.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 21:34:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BLYFdd018092
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 21:34:15 GMT
Message-ID: <14775440.53171276292055222.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 17:34:15 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6769) Add an API in FileSystem to get
 FileSystem instances based on users
In-Reply-To: <15183681.91401274128662681.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877978#action_12877978 ] 

Hudson commented on HADOOP-6769:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #297 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/297/])
    Move entries for HADOOP-6769 and HADOOP-6813 to 0.21 section of CHANGES.txt


> Add an API in FileSystem to get FileSystem instances based on users
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6769
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6769
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: 6769-1.patch
>
>
> HDFS-1000 requires an API in FileSystem to get FileSystem instances based on users. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9045-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 21:38:39 2010
Return-Path: <common-issues-return-9045-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 17365 invoked from network); 11 Jun 2010 21:38:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 21:38:39 -0000
Received: (qmail 42353 invoked by uid 500); 11 Jun 2010 21:38:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 42308 invoked by uid 500); 11 Jun 2010 21:38:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 42300 invoked by uid 99); 11 Jun 2010 21:38:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 21:38:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 21:38:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BLcEAR018135
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 21:38:14 GMT
Message-ID: <2110872.53281276292294272.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 17:38:14 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6819) [Herriot] Shell command for getting
 the new exceptions in the logs returning exitcode 1 after executing
 successfully.
In-Reply-To: <25211660.39321276233134711.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877982#action_12877982 ] 

Konstantin Boudnik commented on HADOOP-6819:
--------------------------------------------

As far as I can see all javadoc warnings are related to {{warning: sun.security.krb5.*}} and there are 6 of them. So, I don't think the patch in question could raise the number of javadoc warning considering the scope of the patch.

> [Herriot] Shell command for getting the new exceptions in the logs returning exitcode 1 after executing successfully.
> ---------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6819
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6819
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6819-ydist-security.patch, HADOOP-6819.patch
>
>
> I have found an corner case issue while executing the shell command for getting the new exceptions in the logs. The problem could see whenever the log files are empty and trying to get the new exceptions count, it returns the output is 0 ,however it returns the exitcode is 1 instead of 0. grep -vc in the command is not  giving the expected exitcode because its not handling the multiple in options.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9046-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 21:44:41 2010
Return-Path: <common-issues-return-9046-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18097 invoked from network); 11 Jun 2010 21:44:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 21:44:40 -0000
Received: (qmail 44136 invoked by uid 500); 11 Jun 2010 21:44:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44080 invoked by uid 500); 11 Jun 2010 21:44:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44068 invoked by uid 99); 11 Jun 2010 21:44:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 21:44:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 21:44:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BLiFgd018190
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 21:44:15 GMT
Message-ID: <4319727.53431276292655829.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 17:44:15 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6668:
------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

I've just committed this.

> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: common.interface-tw-annotated.txt, HADOOP-6668-0.21.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9047-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 22:34:36 2010
Return-Path: <common-issues-return-9047-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32567 invoked from network); 11 Jun 2010 22:34:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 22:34:36 -0000
Received: (qmail 9046 invoked by uid 500); 11 Jun 2010 22:34:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8948 invoked by uid 500); 11 Jun 2010 22:34:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8939 invoked by uid 99); 11 Jun 2010 22:34:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 22:34:35 +0000
X-ASF-Spam-Status: No, hits=-1515.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 22:34:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BMYEoH018610
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 22:34:15 GMT
Message-ID: <22123769.54211276295654952.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 18:34:14 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6652) ShellBasedUnixGroupsMapping shouldn't
 have a cache
In-Reply-To: <689776446.387571269113787242.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6652:
--------------------------------

    Attachment: h-6652.patch

Patch for trunk.

> ShellBasedUnixGroupsMapping shouldn't have a cache
> --------------------------------------------------
>
>                 Key: HADOOP-6652
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6652
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: groups.patch, h-6652.patch
>
>
> Since the Groups class already has a time based cache, the cache from ShellBasedUnixGroupsMapping should be removed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9048-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 22:34:37 2010
Return-Path: <common-issues-return-9048-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32605 invoked from network); 11 Jun 2010 22:34:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 22:34:36 -0000
Received: (qmail 9127 invoked by uid 500); 11 Jun 2010 22:34:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8958 invoked by uid 500); 11 Jun 2010 22:34:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8949 invoked by uid 99); 11 Jun 2010 22:34:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 22:34:36 +0000
X-ASF-Spam-Status: No, hits=-1515.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 22:34:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BMYFAK018616
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 22:34:15 GMT
Message-ID: <29781609.54231276295655558.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 18:34:15 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6652) ShellBasedUnixGroupsMapping shouldn't
 have a cache
In-Reply-To: <689776446.387571269113787242.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6652:
--------------------------------

    Status: Patch Available  (was: Open)

> ShellBasedUnixGroupsMapping shouldn't have a cache
> --------------------------------------------------
>
>                 Key: HADOOP-6652
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6652
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: groups.patch, h-6652.patch
>
>
> Since the Groups class already has a time based cache, the cache from ShellBasedUnixGroupsMapping should be removed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9049-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 22:50:35 2010
Return-Path: <common-issues-return-9049-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35417 invoked from network); 11 Jun 2010 22:50:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 22:50:35 -0000
Received: (qmail 18881 invoked by uid 500); 11 Jun 2010 22:50:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18761 invoked by uid 500); 11 Jun 2010 22:50:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18753 invoked by uid 99); 11 Jun 2010 22:50:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 22:50:34 +0000
X-ASF-Spam-Status: No, hits=-1515.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 22:50:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BMoDRv018706
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 22:50:14 GMT
Message-ID: <24536663.54401276296613718.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 18:50:13 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6620) NPE if renewer is passed as null in
 getDelegationToken
In-Reply-To: <169053026.109491267830687261.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6620:
--------------------------------

           Status: Resolved  (was: Patch Available)
    Fix Version/s: 0.22.0
       Resolution: Fixed

I've committed this.  Thanks, Jitendra!  Resolving as fixed.

> NPE if renewer is passed as null in getDelegationToken
> ------------------------------------------------------
>
>                 Key: HADOOP-6620
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6620
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6620-y20.1.patch, HADOOP-6620.2.patch
>
>
> If renewer is passed as null in getDelegationToken, an NPE is thrown. We should handle null renewer and the token must not be allowed to be renewed if the renewer is null or empty string.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9050-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 22:58:35 2010
Return-Path: <common-issues-return-9050-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36810 invoked from network); 11 Jun 2010 22:58:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 22:58:35 -0000
Received: (qmail 26879 invoked by uid 500); 11 Jun 2010 22:58:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26815 invoked by uid 500); 11 Jun 2010 22:58:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26807 invoked by uid 99); 11 Jun 2010 22:58:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 22:58:34 +0000
X-ASF-Spam-Status: No, hits=-1515.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 22:58:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BMwE1o018840
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 22:58:14 GMT
Message-ID: <25916660.54601276297094185.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 18:58:14 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6822) Provide information as to whether
 or not security is enabled on web interface
In-Reply-To: <18772311.54571276297093402.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878123#action_12878123 ] 

Jakob Homan commented on HADOOP-6822:
-------------------------------------

Currently there is no way to tell if security is enabled on JT/NN web pages (beyond the fact that the web interface access may have to be validated to get to the page, which might not get triggered every time).  It'd be nice to have a label with whether or not security is enabled.

> Provide information as to whether or not security is enabled on web interface
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-6822
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6822
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Jakob Homan
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9051-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 22:58:38 2010
Return-Path: <common-issues-return-9051-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36858 invoked from network); 11 Jun 2010 22:58:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 22:58:38 -0000
Received: (qmail 27076 invoked by uid 500); 11 Jun 2010 22:58:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26978 invoked by uid 500); 11 Jun 2010 22:58:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26969 invoked by uid 99); 11 Jun 2010 22:58:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 22:58:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 22:58:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BMwDtn018831
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 22:58:13 GMT
Message-ID: <18772311.54571276297093402.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 18:58:13 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6822) Provide information as to whether or
 not security is enabled on web interface
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Provide information as to whether or not security is enabled on web interface
-----------------------------------------------------------------------------

                 Key: HADOOP-6822
                 URL: https://issues.apache.org/jira/browse/HADOOP-6822
             Project: Hadoop Common
          Issue Type: Sub-task
            Reporter: Jakob Homan




-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9052-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 11 23:01:58 2010
Return-Path: <common-issues-return-9052-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38213 invoked from network); 11 Jun 2010 23:01:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 11 Jun 2010 23:01:58 -0000
Received: (qmail 34061 invoked by uid 500); 11 Jun 2010 23:01:58 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33811 invoked by uid 500); 11 Jun 2010 23:01:57 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33803 invoked by uid 99); 11 Jun 2010 23:01:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 23:01:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 23:01:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5BN1XHc018937
	for <common-issues@hadoop.apache.org>; Fri, 11 Jun 2010 23:01:33 GMT
Message-ID: <27341136.54691276297293751.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 19:01:33 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6823) Provide validation of
 security-related settings on startup
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Provide validation of security-related settings on startup
----------------------------------------------------------

                 Key: HADOOP-6823
                 URL: https://issues.apache.org/jira/browse/HADOOP-6823
             Project: Hadoop Common
          Issue Type: Improvement
    Affects Versions: 0.22.0
            Reporter: Jakob Homan
            Priority: Minor


There are several new config settings for security, involving keytab files, kerberos principals, security filters, etc. that need to be set for a secure cluster to come up.  It'd be nice if on startup, we checked that the required config values have been set and have something vaguely reasonable for values.  Displaying this information as well would provide a good summary for those trying to debug cluster security issues.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9053-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 00:07:39 2010
Return-Path: <common-issues-return-9053-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65112 invoked from network); 12 Jun 2010 00:07:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 00:07:39 -0000
Received: (qmail 80438 invoked by uid 500); 12 Jun 2010 00:07:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 80392 invoked by uid 500); 12 Jun 2010 00:07:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80384 invoked by uid 99); 12 Jun 2010 00:07:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 00:07:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 00:07:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5C07EIk020209
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 00:07:15 GMT
Message-ID: <16691833.56171276301234610.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 20:07:14 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6652) ShellBasedUnixGroupsMapping shouldn't
 have a cache
In-Reply-To: <689776446.387571269113787242.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6652:
--------------------------------


+1 on trunk patch.

> ShellBasedUnixGroupsMapping shouldn't have a cache
> --------------------------------------------------
>
>                 Key: HADOOP-6652
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6652
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: groups.patch, h-6652.patch
>
>
> Since the Groups class already has a time based cache, the cache from ShellBasedUnixGroupsMapping should be removed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9054-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 00:16:36 2010
Return-Path: <common-issues-return-9054-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 68597 invoked from network); 12 Jun 2010 00:16:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 00:16:36 -0000
Received: (qmail 85416 invoked by uid 500); 12 Jun 2010 00:16:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85393 invoked by uid 500); 12 Jun 2010 00:16:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85385 invoked by uid 99); 12 Jun 2010 00:16:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 00:16:35 +0000
X-ASF-Spam-Status: No, hits=-1515.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 00:16:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5C0GEpm029610
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 00:16:15 GMT
Message-ID: <6844387.56271276301774598.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 20:16:14 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6613) RPC server should check for version
 mismatch first
In-Reply-To: <1913809590.53821267667847182.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6613:
--------------------------------

           Status: Resolved  (was: Patch Available)
    Fix Version/s: 0.22.0
       Resolution: Fixed

I just committed this patch (which was manually tested) to trunk. Thanks, Kan!

> RPC server should check for version mismatch first
> --------------------------------------------------
>
>                 Key: HADOOP-6613
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6613
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>             Fix For: 0.22.0
>
>         Attachments: c6613-01.patch, HADOOP-6613-Y20S-1.patch
>
>
> Currently AuthMethod is checked before Version.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9055-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 00:37:36 2010
Return-Path: <common-issues-return-9055-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78129 invoked from network); 12 Jun 2010 00:37:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 00:37:36 -0000
Received: (qmail 94594 invoked by uid 500); 12 Jun 2010 00:37:36 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94507 invoked by uid 500); 12 Jun 2010 00:37:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94493 invoked by uid 99); 12 Jun 2010 00:37:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 00:37:35 +0000
X-ASF-Spam-Status: No, hits=-1515.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 00:37:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5C0bEet001106
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 00:37:15 GMT
Message-ID: <10857243.56421276303034692.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 20:37:14 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6613) RPC server should check for version
 mismatch first
In-Reply-To: <1913809590.53821267667847182.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878159#action_12878159 ] 

Hudson commented on HADOOP-6613:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #300 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/300/])
    HADOOP-6613. Moves the RPC version check ahead of the AuthMethod check. Contributed by Kan Zhang.


> RPC server should check for version mismatch first
> --------------------------------------------------
>
>                 Key: HADOOP-6613
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6613
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>             Fix For: 0.22.0
>
>         Attachments: c6613-01.patch, HADOOP-6613-Y20S-1.patch
>
>
> Currently AuthMethod is checked before Version.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9056-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 00:37:38 2010
Return-Path: <common-issues-return-9056-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78175 invoked from network); 12 Jun 2010 00:37:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 00:37:38 -0000
Received: (qmail 94691 invoked by uid 500); 12 Jun 2010 00:37:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94655 invoked by uid 500); 12 Jun 2010 00:37:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94647 invoked by uid 99); 12 Jun 2010 00:37:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 00:37:37 +0000
X-ASF-Spam-Status: No, hits=-1515.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 00:37:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5C0bH12001137
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 00:37:17 GMT
Message-ID: <26472278.56521276303037174.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 20:37:17 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6682) NetUtils:normalizeHostName does not
 process hostnames starting with [a-f] correctly
In-Reply-To: <1523422048.700531270511667229.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6682:
--------------------------------

    Attachment: HADOOP-6682-Y20.patch

Patch for Y20, which we forgot to upload at the time.  Not for commit.

> NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6682
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6682
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Jakob Homan
>         Attachments: HADOOP-6682-Y20.patch
>
>
>   public static String normalizeHostName(String name) {
>     if (Character.digit(name.charAt(0), 16) != -1) {
>       return name;
> This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address.  This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f].  Such names will not be converted to an ip address, but be returned unchanged.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9057-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 00:40:37 2010
Return-Path: <common-issues-return-9057-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78428 invoked from network); 12 Jun 2010 00:40:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 00:40:37 -0000
Received: (qmail 95297 invoked by uid 500); 12 Jun 2010 00:40:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95260 invoked by uid 500); 12 Jun 2010 00:40:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95251 invoked by uid 99); 12 Jun 2010 00:40:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 00:40:37 +0000
X-ASF-Spam-Status: No, hits=-1515.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 00:40:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5C0eGkO001215
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 00:40:16 GMT
Message-ID: <6774427.56611276303216153.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 20:40:16 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6682) NetUtils:normalizeHostName does not
 process hostnames starting with [a-f] correctly
In-Reply-To: <1523422048.700531270511667229.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6682:
--------------------------------

    Attachment: HADOOP-6682.patch

Patch for trunk.  Same as Y20 patch.  Just changes the base to 10 rather than 16.  

Hong, I like your approach and will open another JIRA to address the general shortcoming of the normalizeHostname function.  For this one I'd like to take as little risk as possible.

I considered writing a unit test for the fix that would try to normalize some host name that starts with an offending hostname (ask.com), but this would violate the unit-test-shouldn't-talk-to-outside-resources guideline, and couldn't think of a better way to test it.  If someone has one, let me and I'll add it.  Otherwise, I think it's good to go without one.

> NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6682
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6682
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Jakob Homan
>         Attachments: HADOOP-6682-Y20.patch, HADOOP-6682.patch
>
>
>   public static String normalizeHostName(String name) {
>     if (Character.digit(name.charAt(0), 16) != -1) {
>       return name;
> This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address.  This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f].  Such names will not be converted to an ip address, but be returned unchanged.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9058-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 00:40:41 2010
Return-Path: <common-issues-return-9058-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78517 invoked from network); 12 Jun 2010 00:40:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 00:40:40 -0000
Received: (qmail 95618 invoked by uid 500); 12 Jun 2010 00:40:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95579 invoked by uid 500); 12 Jun 2010 00:40:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95571 invoked by uid 99); 12 Jun 2010 00:40:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 00:40:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 00:40:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5C0eHr1001230
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 00:40:17 GMT
Message-ID: <32378831.56661276303217180.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 20:40:17 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6682) NetUtils:normalizeHostName does not
 process hostnames starting with [a-f] correctly
In-Reply-To: <1523422048.700531270511667229.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6682:
--------------------------------

    Status: Patch Available  (was: Open)

submitting patch.

> NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6682
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6682
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Jakob Homan
>         Attachments: HADOOP-6682-Y20.patch, HADOOP-6682.patch
>
>
>   public static String normalizeHostName(String name) {
>     if (Character.digit(name.charAt(0), 16) != -1) {
>       return name;
> This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address.  This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f].  Such names will not be converted to an ip address, but be returned unchanged.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9059-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 01:02:49 2010
Return-Path: <common-issues-return-9059-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81833 invoked from network); 12 Jun 2010 01:02:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 01:02:49 -0000
Received: (qmail 23499 invoked by uid 500); 12 Jun 2010 01:02:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23458 invoked by uid 500); 12 Jun 2010 01:02:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23450 invoked by uid 99); 12 Jun 2010 01:02:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 01:02:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 01:02:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5C12GNP001574
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 01:02:16 GMT
Message-ID: <13093432.56881276304536466.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 21:02:16 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6682) NetUtils:normalizeHostName does not
 process hostnames starting with [a-f] correctly
In-Reply-To: <1523422048.700531270511667229.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878167#action_12878167 ] 

Hadoop QA commented on HADOOP-6682:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446933/HADOOP-6682.patch
  against trunk revision 953910.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/79/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/79/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/79/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/79/console

This message is automatically generated.

> NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6682
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6682
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Jakob Homan
>         Attachments: HADOOP-6682-Y20.patch, HADOOP-6682.patch
>
>
>   public static String normalizeHostName(String name) {
>     if (Character.digit(name.charAt(0), 16) != -1) {
>       return name;
> This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address.  This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f].  Such names will not be converted to an ip address, but be returned unchanged.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9061-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 01:05:39 2010
Return-Path: <common-issues-return-9061-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86216 invoked from network); 12 Jun 2010 01:05:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 01:05:39 -0000
Received: (qmail 27594 invoked by uid 500); 12 Jun 2010 01:05:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27508 invoked by uid 500); 12 Jun 2010 01:05:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27498 invoked by uid 99); 12 Jun 2010 01:05:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 01:05:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 01:05:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5C15FvJ001592
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 01:05:15 GMT
Message-ID: <23758217.56941276304715400.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 21:05:15 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6652) ShellBasedUnixGroupsMapping shouldn't
 have a cache
In-Reply-To: <689776446.387571269113787242.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6652:
--------------------------------

    Status: Patch Available  (was: Open)

Retrying hudson.

> ShellBasedUnixGroupsMapping shouldn't have a cache
> --------------------------------------------------
>
>                 Key: HADOOP-6652
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6652
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: groups.patch, h-6652.patch
>
>
> Since the Groups class already has a time based cache, the cache from ShellBasedUnixGroupsMapping should be removed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9060-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 01:05:39 2010
Return-Path: <common-issues-return-9060-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86212 invoked from network); 12 Jun 2010 01:05:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 01:05:39 -0000
Received: (qmail 27545 invoked by uid 500); 12 Jun 2010 01:05:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27231 invoked by uid 500); 12 Jun 2010 01:05:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27170 invoked by uid 99); 12 Jun 2010 01:05:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 01:05:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 01:05:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5C15E26001586
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 01:05:14 GMT
Message-ID: <21134397.56921276304714532.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 21:05:14 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6652) ShellBasedUnixGroupsMapping shouldn't
 have a cache
In-Reply-To: <689776446.387571269113787242.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6652:
--------------------------------

    Status: Open  (was: Patch Available)

Hudson seems to be stuck on this. 

> ShellBasedUnixGroupsMapping shouldn't have a cache
> --------------------------------------------------
>
>                 Key: HADOOP-6652
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6652
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: groups.patch, h-6652.patch
>
>
> Since the Groups class already has a time based cache, the cache from ShellBasedUnixGroupsMapping should be removed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9062-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 01:07:35 2010
Return-Path: <common-issues-return-9062-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86472 invoked from network); 12 Jun 2010 01:07:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 01:07:35 -0000
Received: (qmail 28387 invoked by uid 500); 12 Jun 2010 01:07:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28333 invoked by uid 500); 12 Jun 2010 01:07:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28319 invoked by uid 99); 12 Jun 2010 01:07:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 01:07:35 +0000
X-ASF-Spam-Status: No, hits=-1515.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 01:07:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5C17E3x001615
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 01:07:14 GMT
Message-ID: <16353878.56981276304834258.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 21:07:14 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6824) Improve NetUtils:normalizeHostName
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Improve NetUtils:normalizeHostName
----------------------------------

                 Key: HADOOP-6824
                 URL: https://issues.apache.org/jira/browse/HADOOP-6824
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Jakob Homan


HADOOP-6682 revealed a bug in NetUtils:normalizeHostName, which got fixed, but it may be worth replacing the current best-effort code with a more rigorous approach, as Hong suggested.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9063-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 01:10:35 2010
Return-Path: <common-issues-return-9063-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86867 invoked from network); 12 Jun 2010 01:10:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 01:10:35 -0000
Received: (qmail 30074 invoked by uid 500); 12 Jun 2010 01:10:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30041 invoked by uid 500); 12 Jun 2010 01:10:35 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30033 invoked by uid 99); 12 Jun 2010 01:10:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 01:10:35 +0000
X-ASF-Spam-Status: No, hits=-1515.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 01:10:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5C1AE07001635
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 01:10:14 GMT
Message-ID: <14979110.57041276305014459.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 21:10:14 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6682) NetUtils:normalizeHostName does not
 process hostnames starting with [a-f] correctly
In-Reply-To: <1523422048.700531270511667229.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878172#action_12878172 ] 

Devaraj Das commented on HADOOP-6682:
-------------------------------------

+1

> NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6682
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6682
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Jakob Homan
>         Attachments: HADOOP-6682-Y20.patch, HADOOP-6682.patch
>
>
>   public static String normalizeHostName(String name) {
>     if (Character.digit(name.charAt(0), 16) != -1) {
>       return name;
> This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address.  This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f].  Such names will not be converted to an ip address, but be returned unchanged.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9064-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 01:12:39 2010
Return-Path: <common-issues-return-9064-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87028 invoked from network); 12 Jun 2010 01:12:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 01:12:38 -0000
Received: (qmail 30698 invoked by uid 500); 12 Jun 2010 01:12:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30652 invoked by uid 500); 12 Jun 2010 01:12:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30644 invoked by uid 99); 12 Jun 2010 01:12:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 01:12:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 01:12:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5C1CEu7001662
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 01:12:14 GMT
Message-ID: <27592332.57111276305134671.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 21:12:14 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6682) NetUtils:normalizeHostName does not
 process hostnames starting with [a-f] correctly
In-Reply-To: <1523422048.700531270511667229.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6682:
--------------------------------

    Status: Open  (was: Patch Available)

I call bogus on the javadoc warning, since this patch doesn't affect that.  Re-run Hudson for test failure.  I don't see it on my machine, but let's see if it re-occurs.

> NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6682
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6682
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Jakob Homan
>         Attachments: HADOOP-6682-Y20.patch, HADOOP-6682.patch
>
>
>   public static String normalizeHostName(String name) {
>     if (Character.digit(name.charAt(0), 16) != -1) {
>       return name;
> This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address.  This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f].  Such names will not be converted to an ip address, but be returned unchanged.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9065-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 01:12:39 2010
Return-Path: <common-issues-return-9065-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87069 invoked from network); 12 Jun 2010 01:12:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 01:12:39 -0000
Received: (qmail 30849 invoked by uid 500); 12 Jun 2010 01:12:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30797 invoked by uid 500); 12 Jun 2010 01:12:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30789 invoked by uid 99); 12 Jun 2010 01:12:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 01:12:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 01:12:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5C1CF3v001677
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 01:12:15 GMT
Message-ID: <31298524.57161276305135386.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 21:12:15 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6682) NetUtils:normalizeHostName does not
 process hostnames starting with [a-f] correctly
In-Reply-To: <1523422048.700531270511667229.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6682:
--------------------------------

    Status: Patch Available  (was: Open)

> NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6682
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6682
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Jakob Homan
>         Attachments: HADOOP-6682-Y20.patch, HADOOP-6682.patch
>
>
>   public static String normalizeHostName(String name) {
>     if (Character.digit(name.charAt(0), 16) != -1) {
>       return name;
> This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address.  This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f].  Such names will not be converted to an ip address, but be returned unchanged.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9066-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 01:30:38 2010
Return-Path: <common-issues-return-9066-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 89114 invoked from network); 12 Jun 2010 01:30:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 01:30:38 -0000
Received: (qmail 38601 invoked by uid 500); 12 Jun 2010 01:30:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38544 invoked by uid 500); 12 Jun 2010 01:30:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38536 invoked by uid 99); 12 Jun 2010 01:30:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 01:30:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 01:30:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5C1UDPA001766
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 01:30:13 GMT
Message-ID: <32522611.57351276306213475.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 21:30:13 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6652) ShellBasedUnixGroupsMapping
 shouldn't have a cache
In-Reply-To: <689776446.387571269113787242.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878181#action_12878181 ] 

Hadoop QA commented on HADOOP-6652:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446913/h-6652.patch
  against trunk revision 953910.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/80/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/80/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/80/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/80/console

This message is automatically generated.

> ShellBasedUnixGroupsMapping shouldn't have a cache
> --------------------------------------------------
>
>                 Key: HADOOP-6652
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6652
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: groups.patch, h-6652.patch
>
>
> Since the Groups class already has a time based cache, the cache from ShellBasedUnixGroupsMapping should be removed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9067-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 01:45:40 2010
Return-Path: <common-issues-return-9067-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 92271 invoked from network); 12 Jun 2010 01:45:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 01:45:40 -0000
Received: (qmail 47836 invoked by uid 500); 12 Jun 2010 01:45:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 47786 invoked by uid 500); 12 Jun 2010 01:45:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 47778 invoked by uid 99); 12 Jun 2010 01:45:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 01:45:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 01:45:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5C1jGXL001841
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 01:45:16 GMT
Message-ID: <33322136.57511276307116062.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 21:45:16 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6682) NetUtils:normalizeHostName does not
 process hostnames starting with [a-f] correctly
In-Reply-To: <1523422048.700531270511667229.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878183#action_12878183 ] 

Hadoop QA commented on HADOOP-6682:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446933/HADOOP-6682.patch
  against trunk revision 953910.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/81/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/81/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/81/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/81/console

This message is automatically generated.

> NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6682
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6682
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Jakob Homan
>         Attachments: HADOOP-6682-Y20.patch, HADOOP-6682.patch
>
>
>   public static String normalizeHostName(String name) {
>     if (Character.digit(name.charAt(0), 16) != -1) {
>       return name;
> This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address.  This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f].  Such names will not be converted to an ip address, but be returned unchanged.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9068-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 02:05:39 2010
Return-Path: <common-issues-return-9068-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98743 invoked from network); 12 Jun 2010 02:05:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 02:05:39 -0000
Received: (qmail 60284 invoked by uid 500); 12 Jun 2010 02:05:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 60188 invoked by uid 500); 12 Jun 2010 02:05:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 60180 invoked by uid 99); 12 Jun 2010 02:05:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 02:05:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 02:05:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5C25Eid002003
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 02:05:14 GMT
Message-ID: <13052237.57691276308314416.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 22:05:14 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878188#action_12878188 ] 

Jakob Homan commented on HADOOP-6796:
-------------------------------------

+1 on revert.  Writables are dangerous, we'll just have to live with them.

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.1.patch, HADOOP-6796.2.patch, HADOOP-6796.3.patch, HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9069-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 02:14:38 2010
Return-Path: <common-issues-return-9069-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99458 invoked from network); 12 Jun 2010 02:14:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 02:14:38 -0000
Received: (qmail 63793 invoked by uid 500); 12 Jun 2010 02:14:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63765 invoked by uid 500); 12 Jun 2010 02:14:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63757 invoked by uid 99); 12 Jun 2010 02:14:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 02:14:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 02:14:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5C2EDtB002063
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 02:14:14 GMT
Message-ID: <4398692.57811276308853972.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 22:14:13 -0400 (EDT)
From: "Jakob Homan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6682) NetUtils:normalizeHostName does not
 process hostnames starting with [a-f] correctly
In-Reply-To: <1523422048.700531270511667229.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jakob Homan updated HADOOP-6682:
--------------------------------

           Status: Resolved  (was: Patch Available)
     Hadoop Flags: [Reviewed]
         Assignee: Jakob Homan
    Fix Version/s: 0.22.0
       Resolution: Fixed

I've committed this.  Resolving as fixed.

> NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6682
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6682
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6682-Y20.patch, HADOOP-6682.patch
>
>
>   public static String normalizeHostName(String name) {
>     if (Character.digit(name.charAt(0), 16) != -1) {
>       return name;
> This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address.  This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f].  Such names will not be converted to an ip address, but be returned unchanged.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9070-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 02:59:40 2010
Return-Path: <common-issues-return-9070-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5287 invoked from network); 12 Jun 2010 02:59:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 02:59:40 -0000
Received: (qmail 88430 invoked by uid 500); 12 Jun 2010 02:59:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88325 invoked by uid 500); 12 Jun 2010 02:59:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88317 invoked by uid 99); 12 Jun 2010 02:59:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 02:59:38 +0000
X-ASF-Spam-Status: No, hits=-1515.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 02:59:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5C2xHWM002252
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 02:59:17 GMT
Message-ID: <26030125.58141276311557699.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 22:59:17 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6652) ShellBasedUnixGroupsMapping
 shouldn't have a cache
In-Reply-To: <689776446.387571269113787242.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878195#action_12878195 ] 

Hadoop QA commented on HADOOP-6652:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446913/h-6652.patch
  against trunk revision 953929.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/582/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/582/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/582/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/582/console

This message is automatically generated.

> ShellBasedUnixGroupsMapping shouldn't have a cache
> --------------------------------------------------
>
>                 Key: HADOOP-6652
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6652
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: groups.patch, h-6652.patch
>
>
> Since the Groups class already has a time based cache, the cache from ShellBasedUnixGroupsMapping should be removed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9071-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 03:04:19 2010
Return-Path: <common-issues-return-9071-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7536 invoked from network); 12 Jun 2010 03:03:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 03:03:41 -0000
Received: (qmail 91670 invoked by uid 500); 12 Jun 2010 03:03:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91553 invoked by uid 500); 12 Jun 2010 03:03:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91545 invoked by uid 99); 12 Jun 2010 03:03:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 03:03:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 03:03:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5C33FRM002337
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 03:03:15 GMT
Message-ID: <25813856.58191276311795268.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 23:03:15 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6682) NetUtils:normalizeHostName does not
 process hostnames starting with [a-f] correctly
In-Reply-To: <1523422048.700531270511667229.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878196#action_12878196 ] 

Hudson commented on HADOOP-6682:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #301 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/301/])
    HADOOP-6682. NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly.


> NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6682
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6682
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6682-Y20.patch, HADOOP-6682.patch
>
>
>   public static String normalizeHostName(String name) {
>     if (Character.digit(name.charAt(0), 16) != -1) {
>       return name;
> This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address.  This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f].  Such names will not be converted to an ip address, but be returned unchanged.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9071-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 05:43:22 2010
Return-Path: <common-issues-return-9071-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7536 invoked from network); 12 Jun 2010 03:03:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 03:03:41 -0000
Received: (qmail 91670 invoked by uid 500); 12 Jun 2010 03:03:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91553 invoked by uid 500); 12 Jun 2010 03:03:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91545 invoked by uid 99); 12 Jun 2010 03:03:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 03:03:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 03:03:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5C33FRM002337
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 03:03:15 GMT
Message-ID: <25813856.58191276311795268.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 23:03:15 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6682) NetUtils:normalizeHostName does not
 process hostnames starting with [a-f] correctly
In-Reply-To: <1523422048.700531270511667229.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878196#action_12878196 ] 

Hudson commented on HADOOP-6682:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #301 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/301/])
    HADOOP-6682. NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly.


> NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6682
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6682
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6682-Y20.patch, HADOOP-6682.patch
>
>
>   public static String normalizeHostName(String name) {
>     if (Character.digit(name.charAt(0), 16) != -1) {
>       return name;
> This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address.  This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f].  Such names will not be converted to an ip address, but be returned unchanged.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9072-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 05:43:37 2010
Return-Path: <common-issues-return-9072-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 10874 invoked from network); 12 Jun 2010 03:42:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 03:42:40 -0000
Received: (qmail 7139 invoked by uid 500); 12 Jun 2010 03:42:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5906 invoked by uid 500); 12 Jun 2010 03:42:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5898 invoked by uid 99); 12 Jun 2010 03:42:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 03:42:36 +0000
X-ASF-Spam-Status: No, hits=-1515.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 03:42:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5C3gGLM002485
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 03:42:16 GMT
Message-ID: <26550668.58471276314136054.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 23:42:16 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6679) add retry logic to
 NetUtils.createSocketAddr() when hostname resolution fails
In-Reply-To: <18346259.678881270363227220.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

sam rash reassigned HADOOP-6679:
--------------------------------

    Assignee: sam rash

> add retry logic to NetUtils.createSocketAddr() when hostname resolution fails
> -----------------------------------------------------------------------------
>
>                 Key: HADOOP-6679
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6679
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.20.0
>            Reporter: sam rash
>            Assignee: sam rash
>            Priority: Minor
>         Attachments: HADOOP-6679-patch-1.txt, HADOOP-6679-patch-2.txt
>
>
> transient DNS errors can cause the InetSocketAddress created in NetUtils.createSocketAddr() to be unresolved when the hostname is resolvable.  Add a single retry after a short sleep using a simple InetSocketAddressFactory class

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9073-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 05:43:47 2010
Return-Path: <common-issues-return-9073-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 11128 invoked from network); 12 Jun 2010 03:51:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 03:51:41 -0000
Received: (qmail 9897 invoked by uid 500); 12 Jun 2010 03:51:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9776 invoked by uid 500); 12 Jun 2010 03:51:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9768 invoked by uid 99); 12 Jun 2010 03:51:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 03:51:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 03:51:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5C3pE3f002518
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 03:51:15 GMT
Message-ID: <28288070.58551276314674922.JavaMail.jira@thor>
Date: Fri, 11 Jun 2010 23:51:14 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878203#action_12878203 ] 

Rodrigo Schmidt commented on HADOOP-6796:
-----------------------------------------

I think it's just bad initialization. permission should be initialized with the constructor value.

Not sure if it's better to revert this or add the changes and better unit tests to cover these cases.

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.1.patch, HADOOP-6796.2.patch, HADOOP-6796.3.patch, HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9074-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 10:12:41 2010
Return-Path: <common-issues-return-9074-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73317 invoked from network); 12 Jun 2010 10:12:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 10:12:41 -0000
Received: (qmail 66428 invoked by uid 500); 12 Jun 2010 10:12:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66299 invoked by uid 500); 12 Jun 2010 10:12:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66288 invoked by uid 99); 12 Jun 2010 10:12:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 10:12:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 10:12:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5CACDER005271
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 10:12:13 GMT
Message-ID: <28548891.62101276337533359.JavaMail.jira@thor>
Date: Sat, 12 Jun 2010 06:12:13 -0400 (EDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6821) Document changes to memory
 monitoring
In-Reply-To: <6410741.46811276277414864.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878246#action_12878246 ] 

Hemanth Yamijala commented on HADOOP-6821:
------------------------------------------

The javadoc warnings are related to Sun's security packages being imported and are unrelated to this patch.

> Document changes to memory monitoring
> -------------------------------------
>
>                 Key: HADOOP-6821
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6821
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.21.0
>            Reporter: Hemanth Yamijala
>            Assignee: Hemanth Yamijala
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6821.patch
>
>
> Modify the cluster_setup guide with information about memory monitoring and admin configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9075-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 11:22:42 2010
Return-Path: <common-issues-return-9075-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84270 invoked from network); 12 Jun 2010 11:22:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 11:22:42 -0000
Received: (qmail 96240 invoked by uid 500); 12 Jun 2010 11:22:41 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96069 invoked by uid 500); 12 Jun 2010 11:22:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96047 invoked by uid 99); 12 Jun 2010 11:22:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 11:22:37 +0000
X-ASF-Spam-Status: No, hits=-1516.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 11:22:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5CBMGbX005559
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 11:22:16 GMT
Message-ID: <4355978.62441276341736789.JavaMail.jira@thor>
Date: Sat, 12 Jun 2010 07:22:16 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6813) Add a new newInstance method in
 FileSystem that takes a "user" as argument
In-Reply-To: <17176494.13701275948371739.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878255#action_12878255 ] 

Hudson commented on HADOOP-6813:
--------------------------------

Integrated in Hadoop-Common-trunk #364 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/364/])
    Move entries for HADOOP-6769 and HADOOP-6813 to 0.21 section of CHANGES.txt


> Add a new newInstance method in FileSystem that takes a "user" as argument
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6813
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6813
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: h-6813.patch
>
>
> In order to implement HDFS-1000 for trunk, I need to have a newInstance API in FileSystem that takes a "user" as an argument.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9076-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 11:22:43 2010
Return-Path: <common-issues-return-9076-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84305 invoked from network); 12 Jun 2010 11:22:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 11:22:43 -0000
Received: (qmail 96291 invoked by uid 500); 12 Jun 2010 11:22:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96116 invoked by uid 500); 12 Jun 2010 11:22:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96071 invoked by uid 99); 12 Jun 2010 11:22:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 11:22:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 11:22:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5CBMFOH005538
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 11:22:15 GMT
Message-ID: <15852885.62371276341735474.JavaMail.jira@thor>
Date: Sat, 12 Jun 2010 07:22:15 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6682) NetUtils:normalizeHostName does not
 process hostnames starting with [a-f] correctly
In-Reply-To: <1523422048.700531270511667229.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878252#action_12878252 ] 

Hudson commented on HADOOP-6682:
--------------------------------

Integrated in Hadoop-Common-trunk #364 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/364/])
    HADOOP-6682. NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly.


> NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-6682
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6682
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6682-Y20.patch, HADOOP-6682.patch
>
>
>   public static String normalizeHostName(String name) {
>     if (Character.digit(name.charAt(0), 16) != -1) {
>       return name;
> This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address.  This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f].  Such names will not be converted to an ip address, but be returned unchanged.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9077-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 11:22:44 2010
Return-Path: <common-issues-return-9077-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84362 invoked from network); 12 Jun 2010 11:22:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 11:22:44 -0000
Received: (qmail 96424 invoked by uid 500); 12 Jun 2010 11:22:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96309 invoked by uid 500); 12 Jun 2010 11:22:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96078 invoked by uid 99); 12 Jun 2010 11:22:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 11:22:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 11:22:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5CBMGq4005544
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 11:22:16 GMT
Message-ID: <16606425.62391276341736117.JavaMail.jira@thor>
Date: Sat, 12 Jun 2010 07:22:16 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6613) RPC server should check for version
 mismatch first
In-Reply-To: <1913809590.53821267667847182.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878253#action_12878253 ] 

Hudson commented on HADOOP-6613:
--------------------------------

Integrated in Hadoop-Common-trunk #364 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/364/])
    HADOOP-6613. Moves the RPC version check ahead of the AuthMethod check. Contributed by Kan Zhang.


> RPC server should check for version mismatch first
> --------------------------------------------------
>
>                 Key: HADOOP-6613
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6613
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc, security
>            Reporter: Kan Zhang
>            Assignee: Kan Zhang
>             Fix For: 0.22.0
>
>         Attachments: c6613-01.patch, HADOOP-6613-Y20S-1.patch
>
>
> Currently AuthMethod is checked before Version.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9079-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 11:22:45 2010
Return-Path: <common-issues-return-9079-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84391 invoked from network); 12 Jun 2010 11:22:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 11:22:44 -0000
Received: (qmail 96735 invoked by uid 500); 12 Jun 2010 11:22:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96427 invoked by uid 500); 12 Jun 2010 11:22:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96101 invoked by uid 99); 12 Jun 2010 11:22:40 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 11:22:40 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 11:22:38 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5CBMHMA005565
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 11:22:17 GMT
Message-ID: <15051444.62461276341737057.JavaMail.jira@thor>
Date: Sat, 12 Jun 2010 07:22:17 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6620) NPE if renewer is passed as null in
 getDelegationToken
In-Reply-To: <169053026.109491267830687261.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878256#action_12878256 ] 

Hudson commented on HADOOP-6620:
--------------------------------

Integrated in Hadoop-Common-trunk #364 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/364/])
    HADOOP-6620. NPE if renewer is passed as null in getDelegationToken. Contributed by Jitendra Pandey.


> NPE if renewer is passed as null in getDelegationToken
> ------------------------------------------------------
>
>                 Key: HADOOP-6620
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6620
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6620-y20.1.patch, HADOOP-6620.2.patch
>
>
> If renewer is passed as null in getDelegationToken, an NPE is thrown. We should handle null renewer and the token must not be allowed to be renewed if the renewer is null or empty string.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9080-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 11:22:45 2010
Return-Path: <common-issues-return-9080-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84406 invoked from network); 12 Jun 2010 11:22:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 11:22:44 -0000
Received: (qmail 96856 invoked by uid 500); 12 Jun 2010 11:22:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96636 invoked by uid 500); 12 Jun 2010 11:22:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96259 invoked by uid 99); 12 Jun 2010 11:22:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 11:22:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 11:22:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5CBMJCS005592
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 11:22:19 GMT
Message-ID: <9951919.62551276341739767.JavaMail.jira@thor>
Date: Sat, 12 Jun 2010 07:22:19 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6668) Apply audience and stability
 annotations to classes in common
In-Reply-To: <1543391482.614961270068327308.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878257#action_12878257 ] 

Hudson commented on HADOOP-6668:
--------------------------------

Integrated in Hadoop-Common-trunk #364 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/364/])
    HADOOP-6668.  Apply audience and stability annotations to classes in common.


> Apply audience and stability annotations to classes in common
> -------------------------------------------------------------
>
>                 Key: HADOOP-6668
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6668
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: common.interface-tw-annotated.txt, HADOOP-6668-0.21.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch, HADOOP-6668.patch
>
>
> Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9078-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 11:22:45 2010
Return-Path: <common-issues-return-9078-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84374 invoked from network); 12 Jun 2010 11:22:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 11:22:44 -0000
Received: (qmail 96496 invoked by uid 500); 12 Jun 2010 11:22:43 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96413 invoked by uid 500); 12 Jun 2010 11:22:43 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96082 invoked by uid 99); 12 Jun 2010 11:22:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 11:22:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 11:22:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5CBMGl4005550
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 11:22:16 GMT
Message-ID: <27235297.62411276341736378.JavaMail.jira@thor>
Date: Sat, 12 Jun 2010 07:22:16 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6814) Method in UGI to get the
 authentication method of the real user.
In-Reply-To: <29484910.14751275951732068.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878254#action_12878254 ] 

Hudson commented on HADOOP-6814:
--------------------------------

Integrated in Hadoop-Common-trunk #364 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/364/])
    HADOOP-6814. Adds an API in UserGroupInformation to get the real authentication method of a passed UGI. Contributed by Jitendra Pandey.


> Method in UGI to get the authentication method of the real user.
> ----------------------------------------------------------------
>
>                 Key: HADOOP-6814
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6814
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6814.1.patch, HADOOP-6814.2.patch
>
>
> UGI should have a method to return the authentication method of the real user for a proxy-user scenario.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9081-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 12 11:22:45 2010
Return-Path: <common-issues-return-9081-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84427 invoked from network); 12 Jun 2010 11:22:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 12 Jun 2010 11:22:44 -0000
Received: (qmail 96983 invoked by uid 500); 12 Jun 2010 11:22:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96817 invoked by uid 500); 12 Jun 2010 11:22:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96410 invoked by uid 99); 12 Jun 2010 11:22:43 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 11:22:43 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Jun 2010 11:22:41 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5CBMKi2005601
	for <common-issues@hadoop.apache.org>; Sat, 12 Jun 2010 11:22:20 GMT
Message-ID: <15290784.62581276341740158.JavaMail.jira@thor>
Date: Sat, 12 Jun 2010 07:22:20 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6769) Add an API in FileSystem to get
 FileSystem instances based on users
In-Reply-To: <15183681.91401274128662681.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878258#action_12878258 ] 

Hudson commented on HADOOP-6769:
--------------------------------

Integrated in Hadoop-Common-trunk #364 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/364/])
    Move entries for HADOOP-6769 and HADOOP-6813 to 0.21 section of CHANGES.txt


> Add an API in FileSystem to get FileSystem instances based on users
> -------------------------------------------------------------------
>
>                 Key: HADOOP-6769
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6769
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: 6769-1.patch
>
>
> HDFS-1000 requires an API in FileSystem to get FileSystem instances based on users. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9082-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 13 15:08:44 2010
Return-Path: <common-issues-return-9082-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4414 invoked from network); 13 Jun 2010 04:22:42 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jun 2010 04:22:42 -0000
Received: (qmail 95937 invoked by uid 500); 13 Jun 2010 04:22:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95813 invoked by uid 500); 13 Jun 2010 04:22:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95799 invoked by uid 99); 13 Jun 2010 04:22:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jun 2010 04:22:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jun 2010 04:22:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5D4MFuc009884
	for <common-issues@hadoop.apache.org>; Sun, 13 Jun 2010 04:22:15 GMT
Message-ID: <31154686.66431276402935114.JavaMail.jira@thor>
Date: Sun, 13 Jun 2010 00:22:15 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878343#action_12878343 ] 

Eli Collins commented on HADOOP-6796:
-------------------------------------

I'll revert the change. If we want to reintroduce we should restore fix users first but it's reasonable for existing users of writables to expect null to work, we'd need to modify them first, ie can't assume they're broken.

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.1.patch, HADOOP-6796.2.patch, HADOOP-6796.3.patch, HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9083-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 13 15:09:11 2010
Return-Path: <common-issues-return-9083-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 4899 invoked from network); 13 Jun 2010 04:33:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jun 2010 04:33:38 -0000
Received: (qmail 4845 invoked by uid 500); 13 Jun 2010 04:33:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4725 invoked by uid 500); 13 Jun 2010 04:33:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4712 invoked by uid 99); 13 Jun 2010 04:33:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jun 2010 04:33:35 +0000
X-ASF-Spam-Status: No, hits=-1516.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jun 2010 04:33:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5D4XEO7009925
	for <common-issues@hadoop.apache.org>; Sun, 13 Jun 2010 04:33:15 GMT
Message-ID: <17696422.66491276403594684.JavaMail.jira@thor>
Date: Sun, 13 Jun 2010 00:33:14 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878344#action_12878344 ] 

Rodrigo Schmidt commented on HADOOP-6796:
-----------------------------------------

Sure! Go ahead and revert it please.

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.1.patch, HADOOP-6796.2.patch, HADOOP-6796.3.patch, HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9084-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 13 15:09:14 2010
Return-Path: <common-issues-return-9084-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5117 invoked from network); 13 Jun 2010 04:42:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jun 2010 04:42:41 -0000
Received: (qmail 5746 invoked by uid 500); 13 Jun 2010 04:42:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5616 invoked by uid 500); 13 Jun 2010 04:42:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5601 invoked by uid 99); 13 Jun 2010 04:42:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jun 2010 04:42:36 +0000
X-ASF-Spam-Status: No, hits=-1516.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jun 2010 04:42:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5D4gFvE009963
	for <common-issues@hadoop.apache.org>; Sun, 13 Jun 2010 04:42:15 GMT
Message-ID: <15505753.66571276404135335.JavaMail.jira@thor>
Date: Sun, 13 Jun 2010 00:42:15 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878345#action_12878345 ] 

Eli Collins commented on HADOOP-6796:
-------------------------------------

Reverted the patch, confirmed TestCopyFiles passes.


> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.1.patch, HADOOP-6796.2.patch, HADOOP-6796.3.patch, HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9085-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 13 15:09:36 2010
Return-Path: <common-issues-return-9085-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5608 invoked from network); 13 Jun 2010 04:54:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jun 2010 04:54:39 -0000
Received: (qmail 9597 invoked by uid 500); 13 Jun 2010 04:54:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9478 invoked by uid 500); 13 Jun 2010 04:54:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9470 invoked by uid 99); 13 Jun 2010 04:54:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jun 2010 04:54:36 +0000
X-ASF-Spam-Status: No, hits=-1516.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jun 2010 04:54:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5D4sEUw010042
	for <common-issues@hadoop.apache.org>; Sun, 13 Jun 2010 04:54:15 GMT
Message-ID: <9057560.66781276404854811.JavaMail.jira@thor>
Date: Sun, 13 Jun 2010 00:54:14 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878348#action_12878348 ] 

Hudson commented on HADOOP-6796:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #302 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/302/])
    HADOOP-6796. Reverting the patch.


> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.1.patch, HADOOP-6796.2.patch, HADOOP-6796.3.patch, HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9086-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 13 15:13:19 2010
Return-Path: <common-issues-return-9086-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26244 invoked from network); 13 Jun 2010 10:18:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jun 2010 10:18:40 -0000
Received: (qmail 9310 invoked by uid 500); 13 Jun 2010 10:18:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9182 invoked by uid 500); 13 Jun 2010 10:18:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9161 invoked by uid 99); 13 Jun 2010 10:18:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jun 2010 10:18:36 +0000
X-ASF-Spam-Status: No, hits=-1517.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jun 2010 10:18:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5DAIFvQ011286
	for <common-issues@hadoop.apache.org>; Sun, 13 Jun 2010 10:18:15 GMT
Message-ID: <26770631.68091276424295576.JavaMail.jira@thor>
Date: Sun, 13 Jun 2010 06:18:15 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6818) Provide a JNI-based implementation
 of GroupMappingServiceProvider
In-Reply-To: <17262845.29701276200793990.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878376#action_12878376 ] 

Vinod K V commented on HADOOP-6818:
-----------------------------------

Just a QQ. Is this a duplicate of HADOOP-4998? Or alteast forward compatible?

Lots of ideas floated around there, it'd be good if these two issues are in agreement.

> Provide a JNI-based implementation of GroupMappingServiceProvider
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6818
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6818
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: JNIGroupMapping.patch
>
>
> The default implementation of GroupMappingServiceProvider does a fork of a unix command to get the groups of a user. Since the group resolution happens in the servers, this might be costly. This jira aims at providing a JNI-based implementation for GroupMappingServiceProvider.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9087-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 13 15:13:19 2010
Return-Path: <common-issues-return-9087-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 26248 invoked from network); 13 Jun 2010 10:18:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jun 2010 10:18:40 -0000
Received: (qmail 9401 invoked by uid 500); 13 Jun 2010 10:18:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9186 invoked by uid 500); 13 Jun 2010 10:18:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9165 invoked by uid 99); 13 Jun 2010 10:18:36 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jun 2010 10:18:36 +0000
X-ASF-Spam-Status: No, hits=-1517.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jun 2010 10:18:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5DAIGbV011298
	for <common-issues@hadoop.apache.org>; Sun, 13 Jun 2010 10:18:16 GMT
Message-ID: <33283243.68131276424296216.JavaMail.jira@thor>
Date: Sun, 13 Jun 2010 06:18:16 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6748) Remove
 hadoop.cluster.administrators
In-Reply-To: <3308520.13571273038484086.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878377#action_12878377 ] 

Vinod K V commented on HADOOP-6748:
-----------------------------------

I propose we get this into 0.21, otherwise we'll need a compatibility story for 0.22.

> Remove hadoop.cluster.administrators
> ------------------------------------
>
>                 Key: HADOOP-6748
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6748
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.21.0
>            Reporter: Amareshwari Sriramadasu
>             Fix For: 0.22.0
>
>         Attachments: patch-6748.txt
>
>
> Remove hadoop.cluster.administrators in favor of having separate configuration property name for MapReduce and HDFS projects.
> See more details on MAPREDUCE-1542.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9088-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 13 15:13:47 2010
Return-Path: <common-issues-return-9088-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31377 invoked from network); 13 Jun 2010 11:22:44 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jun 2010 11:22:44 -0000
Received: (qmail 27610 invoked by uid 500); 13 Jun 2010 11:22:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 26706 invoked by uid 500); 13 Jun 2010 11:22:40 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 26695 invoked by uid 99); 13 Jun 2010 11:22:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jun 2010 11:22:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jun 2010 11:22:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5DBMFaA012332
	for <common-issues@hadoop.apache.org>; Sun, 13 Jun 2010 11:22:15 GMT
Message-ID: <1406499.68431276428135488.JavaMail.jira@thor>
Date: Sun, 13 Jun 2010 07:22:15 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878381#action_12878381 ] 

Hudson commented on HADOOP-6796:
--------------------------------

Integrated in Hadoop-Common-trunk #365 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/365/])
    HADOOP-6796. Reverting the patch.


> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.1.patch, HADOOP-6796.2.patch, HADOOP-6796.3.patch, HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9089-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 13 17:29:39 2010
Return-Path: <common-issues-return-9089-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70244 invoked from network); 13 Jun 2010 17:29:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jun 2010 17:29:39 -0000
Received: (qmail 79903 invoked by uid 500); 13 Jun 2010 17:29:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79799 invoked by uid 500); 13 Jun 2010 17:29:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79771 invoked by uid 99); 13 Jun 2010 17:29:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jun 2010 17:29:37 +0000
X-ASF-Spam-Status: No, hits=-1517.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jun 2010 17:29:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5DHTGEc018447
	for <common-issues@hadoop.apache.org>; Sun, 13 Jun 2010 17:29:17 GMT
Message-ID: <8526146.69711276450156975.JavaMail.jira@thor>
Date: Sun, 13 Jun 2010 13:29:16 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Reopened: (HADOOP-6796) FileStatus allows null srcPath but
 crashes if that's done
In-Reply-To: <8732335.101241275389076505.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE reopened HADOOP-6796:
--------------------------------------------


Reopen since the patch has already been reverted.

> FileStatus allows null srcPath but crashes if that's done
> ---------------------------------------------------------
>
>                 Key: HADOOP-6796
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6796
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>            Priority: Minor
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6796.1.patch, HADOOP-6796.2.patch, HADOOP-6796.3.patch, HADOOP-6796.patch
>
>
> FileStatus allows constructor invocation with a null srcPath but many methods like write, readFields, compareTo, equals, and hashCode depend on this property.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9090-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 13 21:14:39 2010
Return-Path: <common-issues-return-9090-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32613 invoked from network); 13 Jun 2010 21:14:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jun 2010 21:14:38 -0000
Received: (qmail 71958 invoked by uid 500); 13 Jun 2010 21:14:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71869 invoked by uid 500); 13 Jun 2010 21:14:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71861 invoked by uid 99); 13 Jun 2010 21:14:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jun 2010 21:14:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jun 2010 21:14:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5DLEDWH019594
	for <common-issues@hadoop.apache.org>; Sun, 13 Jun 2010 21:14:13 GMT
Message-ID: <28758544.71131276463653554.JavaMail.jira@thor>
Date: Sun, 13 Jun 2010 17:14:13 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6825) FileStatus needs unit tests
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

FileStatus needs unit tests
---------------------------

                 Key: HADOOP-6825
                 URL: https://issues.apache.org/jira/browse/HADOOP-6825
             Project: Hadoop Common
          Issue Type: Improvement
    Affects Versions: 0.22.0
            Reporter: Rodrigo Schmidt
            Assignee: Rodrigo Schmidt
             Fix For: 0.22.0


We need some unit tests for FileStatus to prevent problems like those we recently had on HADOOP-6796 and MAPREDUCE-1858.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9091-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 13 21:44:38 2010
Return-Path: <common-issues-return-9091-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37011 invoked from network); 13 Jun 2010 21:44:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jun 2010 21:44:38 -0000
Received: (qmail 81534 invoked by uid 500); 13 Jun 2010 21:44:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81488 invoked by uid 500); 13 Jun 2010 21:44:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81480 invoked by uid 99); 13 Jun 2010 21:44:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jun 2010 21:44:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jun 2010 21:44:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5DLiDsY019685
	for <common-issues@hadoop.apache.org>; Sun, 13 Jun 2010 21:44:14 GMT
Message-ID: <15285664.71211276465453740.JavaMail.jira@thor>
Date: Sun, 13 Jun 2010 17:44:13 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6825) FileStatus needs unit tests
In-Reply-To: <28758544.71131276463653554.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6825:
------------------------------------

    Attachment: HADOOP-6825.0.patch

> FileStatus needs unit tests
> ---------------------------
>
>                 Key: HADOOP-6825
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6825
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6825.0.patch
>
>
> We need some unit tests for FileStatus to prevent problems like those we recently had on HADOOP-6796 and MAPREDUCE-1858.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9092-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 13 21:47:40 2010
Return-Path: <common-issues-return-9092-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37284 invoked from network); 13 Jun 2010 21:47:40 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jun 2010 21:47:40 -0000
Received: (qmail 82144 invoked by uid 500); 13 Jun 2010 21:47:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82114 invoked by uid 500); 13 Jun 2010 21:47:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82106 invoked by uid 99); 13 Jun 2010 21:47:39 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jun 2010 21:47:39 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jun 2010 21:47:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5DLlF7v019697
	for <common-issues@hadoop.apache.org>; Sun, 13 Jun 2010 21:47:16 GMT
Message-ID: <10871791.71231276465635819.JavaMail.jira@thor>
Date: Sun, 13 Jun 2010 17:47:15 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6825) FileStatus needs unit tests
In-Reply-To: <28758544.71131276463653554.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rodrigo Schmidt updated HADOOP-6825:
------------------------------------

    Status: Patch Available  (was: Open)

First patch

> FileStatus needs unit tests
> ---------------------------
>
>                 Key: HADOOP-6825
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6825
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6825.0.patch
>
>
> We need some unit tests for FileStatus to prevent problems like those we recently had on HADOOP-6796 and MAPREDUCE-1858.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9093-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 13 22:11:38 2010
Return-Path: <common-issues-return-9093-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44669 invoked from network); 13 Jun 2010 22:11:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jun 2010 22:11:38 -0000
Received: (qmail 94555 invoked by uid 500); 13 Jun 2010 22:11:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94470 invoked by uid 500); 13 Jun 2010 22:11:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94462 invoked by uid 99); 13 Jun 2010 22:11:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jun 2010 22:11:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jun 2010 22:11:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5DMBDXl019839
	for <common-issues@hadoop.apache.org>; Sun, 13 Jun 2010 22:11:13 GMT
Message-ID: <15772520.71281276467073364.JavaMail.jira@thor>
Date: Sun, 13 Jun 2010 18:11:13 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6825) FileStatus needs unit tests
In-Reply-To: <28758544.71131276463653554.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878439#action_12878439 ] 

Hadoop QA commented on HADOOP-6825:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12446991/HADOOP-6825.0.patch
  against trunk revision 954165.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 2 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/583/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/583/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/583/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/583/console

This message is automatically generated.

> FileStatus needs unit tests
> ---------------------------
>
>                 Key: HADOOP-6825
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6825
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6825.0.patch
>
>
> We need some unit tests for FileStatus to prevent problems like those we recently had on HADOOP-6796 and MAPREDUCE-1858.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9094-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 13 22:23:35 2010
Return-Path: <common-issues-return-9094-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46773 invoked from network); 13 Jun 2010 22:23:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 13 Jun 2010 22:23:35 -0000
Received: (qmail 473 invoked by uid 500); 13 Jun 2010 22:23:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 432 invoked by uid 500); 13 Jun 2010 22:23:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 424 invoked by uid 99); 13 Jun 2010 22:23:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jun 2010 22:23:34 +0000
X-ASF-Spam-Status: No, hits=-1517.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Jun 2010 22:23:33 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5DMNDOd019872
	for <common-issues@hadoop.apache.org>; Sun, 13 Jun 2010 22:23:13 GMT
Message-ID: <23327165.71321276467793393.JavaMail.jira@thor>
Date: Sun, 13 Jun 2010 18:23:13 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6825) FileStatus needs unit tests
In-Reply-To: <28758544.71131276463653554.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878440#action_12878440 ] 

Rodrigo Schmidt commented on HADOOP-6825:
-----------------------------------------

warnings seem to be coming from security files, not my patch:

     [exec]   [javadoc] Constructing Javadoc information...
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:31: warning: sun.security.krb5.Config is Sun prop\
rietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.Config;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:32: warning: sun.security.krb5.KrbException is Su\
n proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.KrbException;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:81: warning: sun.security.krb5.Config is Sun prop\
rietary API and may be removed in a future release
     [exec]   [javadoc]   private static Config kerbConf;
     [exec]   [javadoc]                  ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/SecurityUtil.java:33: warning: sun.security.jgss.krb5.Krb5Util is S\
un proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.jgss.krb5.Krb5Util;
     [exec]   [javadoc]                              ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/SecurityUtil.java:34: warning: sun.security.krb5.Credentials is Sun\
 proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.Credentials;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/SecurityUtil.java:35: warning: sun.security.krb5.PrincipalName is S\
un proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.PrincipalName;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] ExcludePrivateAnnotationsStandardDoclet
     [exec]   [javadoc] Standard Doclet version 1.6.0_11
     [exec]   [javadoc] Building tree for all the packages and classes...
...
     [exec]   [javadoc] Generating /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/build/docs/api/stylesheet.css...
     [exec]   [javadoc] 6 warnings



> FileStatus needs unit tests
> ---------------------------
>
>                 Key: HADOOP-6825
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6825
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6825.0.patch
>
>
> We need some unit tests for FileStatus to prevent problems like those we recently had on HADOOP-6796 and MAPREDUCE-1858.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9095-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 14 14:31:38 2010
Return-Path: <common-issues-return-9095-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9128 invoked from network); 14 Jun 2010 05:38:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jun 2010 05:38:38 -0000
Received: (qmail 35783 invoked by uid 500); 14 Jun 2010 05:37:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35653 invoked by uid 500); 14 Jun 2010 05:37:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35639 invoked by uid 99); 14 Jun 2010 05:37:35 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Jun 2010 05:37:35 +0000
X-ASF-Spam-Status: No, hits=-1517.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Jun 2010 05:37:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5E5bEgj022156
	for <common-issues@hadoop.apache.org>; Mon, 14 Jun 2010 05:37:14 GMT
Message-ID: <30524559.73691276493834150.JavaMail.jira@thor>
Date: Mon, 14 Jun 2010 01:37:14 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6821) Document changes to memory
 monitoring
In-Reply-To: <6410741.46811276277414864.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878488#action_12878488 ] 

Vinod K V commented on HADOOP-6821:
-----------------------------------

+1 for the patch from my side too. Hemanth/Tom, please commit this to common project.

> Document changes to memory monitoring
> -------------------------------------
>
>                 Key: HADOOP-6821
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6821
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.21.0
>            Reporter: Hemanth Yamijala
>            Assignee: Hemanth Yamijala
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6821.patch
>
>
> Modify the cluster_setup guide with information about memory monitoring and admin configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9096-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 14 15:59:41 2010
Return-Path: <common-issues-return-9096-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30875 invoked from network); 14 Jun 2010 15:59:41 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jun 2010 15:59:41 -0000
Received: (qmail 83874 invoked by uid 500); 14 Jun 2010 06:59:40 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83747 invoked by uid 500); 14 Jun 2010 06:59:39 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83732 invoked by uid 99); 14 Jun 2010 06:59:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Jun 2010 06:59:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Jun 2010 06:59:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5E6xDvH022508
	for <common-issues@hadoop.apache.org>; Mon, 14 Jun 2010 06:59:14 GMT
Message-ID: <17046719.74061276498753891.JavaMail.jira@thor>
Date: Mon, 14 Jun 2010 02:59:13 -0400 (EDT)
From: "Gokul (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6746) hadoop-daemon.sh does not append to
 log files
In-Reply-To: <21404317.20031272914395859.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878495#action_12878495 ] 

Gokul commented on HADOOP-6746:
-------------------------------


i am not getting the point here.

Currently, everytime you restart the cluster the log is getting appended right??

{quote}
I believe output should be appended to the log file so logrotate copytruncate works. Currently a large log file cannot be truncated without restarting the running service, which causes issues in production environments.
{quote}

can you make it elaborate?


> hadoop-daemon.sh does not append to log files
> ---------------------------------------------
>
>                 Key: HADOOP-6746
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6746
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Travis Crawford
>
> Looking in hadoop-daemon.sh we see the start command redirects logs to the log file by creating a new file. It does not append to the log.
> nohup nice -n $HADOOP_NICENESS "$HADOOP_HOME"/bin/hadoop --config $HADOOP_CONF_DIR $command "$@" > "$log" 2>&1 < /dev/null &
> I believe output should be appended to the log file so logrotate copytruncate works. Currently a large log file cannot be truncated without restarting the running service, which causes issues in production environments.
> The improved line would be exactly the same, except use >> to open the log file in append mode.
> nohup nice -n $HADOOP_NICENESS "$HADOOP_HOME"/bin/hadoop --config $HADOOP_CONF_DIR $command "$@" >> "$log" 2>&1 < /dev/null &

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9097-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 14 18:35:39 2010
Return-Path: <common-issues-return-9097-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9053 invoked from network); 14 Jun 2010 18:35:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jun 2010 18:35:39 -0000
Received: (qmail 45979 invoked by uid 500); 14 Jun 2010 18:35:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45708 invoked by uid 500); 14 Jun 2010 18:35:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45700 invoked by uid 99); 14 Jun 2010 18:35:37 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Jun 2010 18:35:37 +0000
X-ASF-Spam-Status: No, hits=-1519.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Jun 2010 18:35:37 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5EIZHmQ029352
	for <common-issues@hadoop.apache.org>; Mon, 14 Jun 2010 18:35:17 GMT
Message-ID: <2195515.83441276540516996.JavaMail.jira@thor>
Date: Mon, 14 Jun 2010 14:35:16 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6652) ShellBasedUnixGroupsMapping
 shouldn't have a cache
In-Reply-To: <689776446.387571269113787242.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878687#action_12878687 ] 

Devaraj Das commented on HADOOP-6652:
-------------------------------------

The javadoc warning is a known issue, and not because of this patch. The patch removes unnecessary code, and has been manually tested - thus tests have not been added. I intend to commit this soon.

> ShellBasedUnixGroupsMapping shouldn't have a cache
> --------------------------------------------------
>
>                 Key: HADOOP-6652
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6652
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: groups.patch, h-6652.patch
>
>
> Since the Groups class already has a time based cache, the cache from ShellBasedUnixGroupsMapping should be removed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9098-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 14 20:05:38 2010
Return-Path: <common-issues-return-9098-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37430 invoked from network); 14 Jun 2010 20:05:38 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jun 2010 20:05:38 -0000
Received: (qmail 31310 invoked by uid 500); 14 Jun 2010 20:05:38 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31214 invoked by uid 500); 14 Jun 2010 20:05:37 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31206 invoked by uid 99); 14 Jun 2010 20:05:37 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Jun 2010 20:05:37 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Jun 2010 20:05:35 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5EK5D2B000412
	for <common-issues@hadoop.apache.org>; Mon, 14 Jun 2010 20:05:13 GMT
Message-ID: <20767844.85241276545913854.JavaMail.jira@thor>
Date: Mon, 14 Jun 2010 16:05:13 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6652) ShellBasedUnixGroupsMapping shouldn't
 have a cache
In-Reply-To: <689776446.387571269113787242.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6652:
--------------------------------

        Status: Resolved  (was: Patch Available)
    Resolution: Fixed

I just committed this. 

> ShellBasedUnixGroupsMapping shouldn't have a cache
> --------------------------------------------------
>
>                 Key: HADOOP-6652
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6652
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: groups.patch, h-6652.patch
>
>
> Since the Groups class already has a time based cache, the cache from ShellBasedUnixGroupsMapping should be removed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9099-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 14 21:19:39 2010
Return-Path: <common-issues-return-9099-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55659 invoked from network); 14 Jun 2010 21:19:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jun 2010 21:19:39 -0000
Received: (qmail 17719 invoked by uid 500); 14 Jun 2010 21:19:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17627 invoked by uid 500); 14 Jun 2010 21:19:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17619 invoked by uid 99); 14 Jun 2010 21:19:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Jun 2010 21:19:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Jun 2010 21:19:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5ELJEFT001102
	for <common-issues@hadoop.apache.org>; Mon, 14 Jun 2010 21:19:14 GMT
Message-ID: <32178964.86391276550354830.JavaMail.jira@thor>
Date: Mon, 14 Jun 2010 17:19:14 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6821) Document changes to memory monitoring
In-Reply-To: <6410741.46811276277414864.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6821:
------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

I've just committed this. Thanks Hemanth!

> Document changes to memory monitoring
> -------------------------------------
>
>                 Key: HADOOP-6821
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6821
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.21.0
>            Reporter: Hemanth Yamijala
>            Assignee: Hemanth Yamijala
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6821.patch
>
>
> Modify the cluster_setup guide with information about memory monitoring and admin configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9100-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 14 21:35:35 2010
Return-Path: <common-issues-return-9100-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 62118 invoked from network); 14 Jun 2010 21:35:35 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 14 Jun 2010 21:35:35 -0000
Received: (qmail 38669 invoked by uid 500); 14 Jun 2010 21:35:35 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38567 invoked by uid 500); 14 Jun 2010 21:35:34 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38558 invoked by uid 99); 14 Jun 2010 21:35:34 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Jun 2010 21:35:34 +0000
X-ASF-Spam-Status: No, hits=-1519.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Jun 2010 21:35:34 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5ELZD27001236
	for <common-issues@hadoop.apache.org>; Mon, 14 Jun 2010 21:35:13 GMT
Message-ID: <32820660.86571276551313420.JavaMail.jira@thor>
Date: Mon, 14 Jun 2010 17:35:13 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6821) Document changes to memory
 monitoring
In-Reply-To: <6410741.46811276277414864.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878755#action_12878755 ] 

Hudson commented on HADOOP-6821:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #309 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/309/])
    HADOOP-6821.  Document changes to memory monitoring.  Contributed by Hemanth Yamijala.


> Document changes to memory monitoring
> -------------------------------------
>
>                 Key: HADOOP-6821
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6821
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.21.0
>            Reporter: Hemanth Yamijala
>            Assignee: Hemanth Yamijala
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6821.patch
>
>
> Modify the cluster_setup guide with information about memory monitoring and admin configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9101-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 15 01:15:39 2010
Return-Path: <common-issues-return-9101-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 40439 invoked from network); 15 Jun 2010 01:15:39 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jun 2010 01:15:39 -0000
Received: (qmail 19612 invoked by uid 500); 15 Jun 2010 01:15:39 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19500 invoked by uid 500); 15 Jun 2010 01:15:38 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 19492 invoked by uid 99); 15 Jun 2010 01:15:38 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 01:15:38 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 01:15:36 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5F1FEsX003997
	for <common-issues@hadoop.apache.org>; Tue, 15 Jun 2010 01:15:14 GMT
Message-ID: <33380923.90011276564514394.JavaMail.jira@thor>
Date: Mon, 14 Jun 2010 21:15:14 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6825) FileStatus needs unit tests
In-Reply-To: <28758544.71131276463653554.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878821#action_12878821 ] 

Rodrigo Schmidt commented on HADOOP-6825:
-----------------------------------------

Eli, Jakob, can one of you review this patch. It's good to have tests for FileStatus in hadoop common (it's kind of bad to rely only on unit tests that are in HDFS or MAPREDUCE).

> FileStatus needs unit tests
> ---------------------------
>
>                 Key: HADOOP-6825
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6825
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6825.0.patch
>
>
> We need some unit tests for FileStatus to prevent problems like those we recently had on HADOOP-6796 and MAPREDUCE-1858.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9102-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 15 03:06:31 2010
Return-Path: <common-issues-return-9102-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63834 invoked from network); 15 Jun 2010 03:05:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jun 2010 03:05:51 -0000
Received: (qmail 63362 invoked by uid 500); 15 Jun 2010 03:05:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63215 invoked by uid 500); 15 Jun 2010 03:05:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63199 invoked by uid 99); 15 Jun 2010 03:05:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 03:05:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 03:05:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5F35Ott004929
	for <common-issues@hadoop.apache.org>; Tue, 15 Jun 2010 03:05:24 GMT
Message-ID: <23041363.141276571124476.JavaMail.jira@thor>
Date: Mon, 14 Jun 2010 23:05:24 -0400 (EDT)
From: "Travis Crawford (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6746) hadoop-daemon.sh does not append to
 log files
In-Reply-To: <21404317.20031272914395859.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Travis Crawford updated HADOOP-6746:
------------------------------------

          Status: Patch Available  (was: Open)
    Release Note: Open log files with append (rather than creating new files) so file truncation works. This allows log rotation tools to operate correctly without restarting Hadoop processes.

This patch is useful from an operations perspective because Hadoop processes are typically long-lived and generate large log files, risking full disks which cause node failure or pages to operations staff to resolve the issue. By opening files in append mode they can be truncated, thus freeing up disk space consumed by logs. Using a log rotation tool that (a) copies a log file aside, (b) truncates "live" file, and (c) compresses rotates file for later analysis if needed, is a very common task.

Currently log files are not opened in append mode, so truncation simply fills a file with nothing, and new bytes are written at the existing offset; disk space is not reclaimed until restarting the process.

> hadoop-daemon.sh does not append to log files
> ---------------------------------------------
>
>                 Key: HADOOP-6746
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6746
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Travis Crawford
>
> Looking in hadoop-daemon.sh we see the start command redirects logs to the log file by creating a new file. It does not append to the log.
> nohup nice -n $HADOOP_NICENESS "$HADOOP_HOME"/bin/hadoop --config $HADOOP_CONF_DIR $command "$@" > "$log" 2>&1 < /dev/null &
> I believe output should be appended to the log file so logrotate copytruncate works. Currently a large log file cannot be truncated without restarting the running service, which causes issues in production environments.
> The improved line would be exactly the same, except use >> to open the log file in append mode.
> nohup nice -n $HADOOP_NICENESS "$HADOOP_HOME"/bin/hadoop --config $HADOOP_CONF_DIR $command "$@" >> "$log" 2>&1 < /dev/null &

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9102-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 15 03:44:50 2010
Return-Path: <common-issues-return-9102-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63834 invoked from network); 15 Jun 2010 03:05:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jun 2010 03:05:51 -0000
Received: (qmail 63362 invoked by uid 500); 15 Jun 2010 03:05:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63215 invoked by uid 500); 15 Jun 2010 03:05:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63199 invoked by uid 99); 15 Jun 2010 03:05:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 03:05:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 03:05:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5F35Ott004929
	for <common-issues@hadoop.apache.org>; Tue, 15 Jun 2010 03:05:24 GMT
Message-ID: <23041363.141276571124476.JavaMail.jira@thor>
Date: Mon, 14 Jun 2010 23:05:24 -0400 (EDT)
From: "Travis Crawford (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6746) hadoop-daemon.sh does not append to
 log files
In-Reply-To: <21404317.20031272914395859.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Travis Crawford updated HADOOP-6746:
------------------------------------

          Status: Patch Available  (was: Open)
    Release Note: Open log files with append (rather than creating new files) so file truncation works. This allows log rotation tools to operate correctly without restarting Hadoop processes.

This patch is useful from an operations perspective because Hadoop processes are typically long-lived and generate large log files, risking full disks which cause node failure or pages to operations staff to resolve the issue. By opening files in append mode they can be truncated, thus freeing up disk space consumed by logs. Using a log rotation tool that (a) copies a log file aside, (b) truncates "live" file, and (c) compresses rotates file for later analysis if needed, is a very common task.

Currently log files are not opened in append mode, so truncation simply fills a file with nothing, and new bytes are written at the existing offset; disk space is not reclaimed until restarting the process.

> hadoop-daemon.sh does not append to log files
> ---------------------------------------------
>
>                 Key: HADOOP-6746
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6746
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Travis Crawford
>
> Looking in hadoop-daemon.sh we see the start command redirects logs to the log file by creating a new file. It does not append to the log.
> nohup nice -n $HADOOP_NICENESS "$HADOOP_HOME"/bin/hadoop --config $HADOOP_CONF_DIR $command "$@" > "$log" 2>&1 < /dev/null &
> I believe output should be appended to the log file so logrotate copytruncate works. Currently a large log file cannot be truncated without restarting the running service, which causes issues in production environments.
> The improved line would be exactly the same, except use >> to open the log file in append mode.
> nohup nice -n $HADOOP_NICENESS "$HADOOP_HOME"/bin/hadoop --config $HADOOP_CONF_DIR $command "$@" >> "$log" 2>&1 < /dev/null &

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9103-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 15 05:00:29 2010
Return-Path: <common-issues-return-9103-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12434 invoked from network); 15 Jun 2010 05:00:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jun 2010 05:00:28 -0000
Received: (qmail 64300 invoked by uid 500); 15 Jun 2010 03:13:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64278 invoked by uid 500); 15 Jun 2010 03:13:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64270 invoked by uid 99); 15 Jun 2010 03:13:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 03:13:45 +0000
X-ASF-Spam-Status: No, hits=-1520.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 03:13:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5F3DOg9004974
	for <common-issues@hadoop.apache.org>; Tue, 15 Jun 2010 03:13:24 GMT
Message-ID: <11210989.231276571604433.JavaMail.jira@thor>
Date: Mon, 14 Jun 2010 23:13:24 -0400 (EDT)
From: "Travis Crawford (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6746) hadoop-daemon.sh does not append to
 log files
In-Reply-To: <21404317.20031272914395859.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Travis Crawford updated HADOOP-6746:
------------------------------------

    Attachment: HADOOP-6746.patch

> hadoop-daemon.sh does not append to log files
> ---------------------------------------------
>
>                 Key: HADOOP-6746
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6746
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Travis Crawford
>         Attachments: HADOOP-6746.patch
>
>
> Looking in hadoop-daemon.sh we see the start command redirects logs to the log file by creating a new file. It does not append to the log.
> nohup nice -n $HADOOP_NICENESS "$HADOOP_HOME"/bin/hadoop --config $HADOOP_CONF_DIR $command "$@" > "$log" 2>&1 < /dev/null &
> I believe output should be appended to the log file so logrotate copytruncate works. Currently a large log file cannot be truncated without restarting the running service, which causes issues in production environments.
> The improved line would be exactly the same, except use >> to open the log file in append mode.
> nohup nice -n $HADOOP_NICENESS "$HADOOP_HOME"/bin/hadoop --config $HADOOP_CONF_DIR $command "$@" >> "$log" 2>&1 < /dev/null &

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9104-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 15 05:04:28 2010
Return-Path: <common-issues-return-9104-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14541 invoked from network); 15 Jun 2010 05:04:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jun 2010 05:04:27 -0000
Received: (qmail 68742 invoked by uid 500); 15 Jun 2010 03:17:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68624 invoked by uid 500); 15 Jun 2010 03:17:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68615 invoked by uid 99); 15 Jun 2010 03:17:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 03:17:45 +0000
X-ASF-Spam-Status: No, hits=-1520.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 03:17:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5F3HNOZ004994
	for <common-issues@hadoop.apache.org>; Tue, 15 Jun 2010 03:17:24 GMT
Message-ID: <1271473.291276571843840.JavaMail.jira@thor>
Date: Mon, 14 Jun 2010 23:17:23 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6746) hadoop-daemon.sh does not append to
 log files
In-Reply-To: <21404317.20031272914395859.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878843#action_12878843 ] 

Hadoop QA commented on HADOOP-6746:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org
  against trunk revision 954647.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 patch.  The patch command could not apply the patch.

Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/584/console

This message is automatically generated.

> hadoop-daemon.sh does not append to log files
> ---------------------------------------------
>
>                 Key: HADOOP-6746
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6746
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Travis Crawford
>         Attachments: HADOOP-6746.patch
>
>
> Looking in hadoop-daemon.sh we see the start command redirects logs to the log file by creating a new file. It does not append to the log.
> nohup nice -n $HADOOP_NICENESS "$HADOOP_HOME"/bin/hadoop --config $HADOOP_CONF_DIR $command "$@" > "$log" 2>&1 < /dev/null &
> I believe output should be appended to the log file so logrotate copytruncate works. Currently a large log file cannot be truncated without restarting the running service, which causes issues in production environments.
> The improved line would be exactly the same, except use >> to open the log file in append mode.
> nohup nice -n $HADOOP_NICENESS "$HADOOP_HOME"/bin/hadoop --config $HADOOP_CONF_DIR $command "$@" >> "$log" 2>&1 < /dev/null &

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9105-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 15 05:04:30 2010
Return-Path: <common-issues-return-9105-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 14599 invoked from network); 15 Jun 2010 05:04:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jun 2010 05:04:30 -0000
Received: (qmail 68833 invoked by uid 500); 15 Jun 2010 03:17:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68795 invoked by uid 500); 15 Jun 2010 03:17:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68784 invoked by uid 99); 15 Jun 2010 03:17:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 03:17:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 03:17:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5F3HPnW005012
	for <common-issues@hadoop.apache.org>; Tue, 15 Jun 2010 03:17:25 GMT
Message-ID: <5903587.351276571845123.JavaMail.jira@thor>
Date: Mon, 14 Jun 2010 23:17:25 -0400 (EDT)
From: "Travis Crawford (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6746) hadoop-daemon.sh does not append to
 log files
In-Reply-To: <21404317.20031272914395859.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878844#action_12878844 ] 

Travis Crawford commented on HADOOP-6746:
-----------------------------------------

This is my first contribution to Hadoop, so sorry for the noobie question. At this point (per http://wiki.apache.org/hadoop/HowToContribute) I believe the next step is submitting the patch -- however, I either don't have permissions to do so, or I'm being dense and not seeing that in the Jira interface.

If this change looks good a nudge in the right direction would be appreciated.

> hadoop-daemon.sh does not append to log files
> ---------------------------------------------
>
>                 Key: HADOOP-6746
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6746
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Travis Crawford
>         Attachments: HADOOP-6746.patch
>
>
> Looking in hadoop-daemon.sh we see the start command redirects logs to the log file by creating a new file. It does not append to the log.
> nohup nice -n $HADOOP_NICENESS "$HADOOP_HOME"/bin/hadoop --config $HADOOP_CONF_DIR $command "$@" > "$log" 2>&1 < /dev/null &
> I believe output should be appended to the log file so logrotate copytruncate works. Currently a large log file cannot be truncated without restarting the running service, which causes issues in production environments.
> The improved line would be exactly the same, except use >> to open the log file in append mode.
> nohup nice -n $HADOOP_NICENESS "$HADOOP_HOME"/bin/hadoop --config $HADOOP_CONF_DIR $command "$@" >> "$log" 2>&1 < /dev/null &

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9106-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 15 11:21:49 2010
Return-Path: <common-issues-return-9106-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50354 invoked from network); 15 Jun 2010 11:21:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jun 2010 11:21:49 -0000
Received: (qmail 2375 invoked by uid 500); 15 Jun 2010 11:21:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2238 invoked by uid 500); 15 Jun 2010 11:21:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2225 invoked by uid 99); 15 Jun 2010 11:21:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 11:21:44 +0000
X-ASF-Spam-Status: No, hits=-1520.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 11:21:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5FBLNlc008771
	for <common-issues@hadoop.apache.org>; Tue, 15 Jun 2010 11:21:23 GMT
Message-ID: <6219293.4381276600883698.JavaMail.jira@thor>
Date: Tue, 15 Jun 2010 07:21:23 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6652) ShellBasedUnixGroupsMapping
 shouldn't have a cache
In-Reply-To: <689776446.387571269113787242.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878935#action_12878935 ] 

Hudson commented on HADOOP-6652:
--------------------------------

Integrated in Hadoop-Common-trunk #367 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/367/])
    HADOOP-6652. Removes the unnecessary cache from ShellBasedUnixGroupsMapping. Contributed by Devaraj Das.


> ShellBasedUnixGroupsMapping shouldn't have a cache
> --------------------------------------------------
>
>                 Key: HADOOP-6652
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6652
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: groups.patch, h-6652.patch
>
>
> Since the Groups class already has a time based cache, the cache from ShellBasedUnixGroupsMapping should be removed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9107-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 15 11:21:50 2010
Return-Path: <common-issues-return-9107-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50412 invoked from network); 15 Jun 2010 11:21:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jun 2010 11:21:50 -0000
Received: (qmail 2445 invoked by uid 500); 15 Jun 2010 11:21:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2256 invoked by uid 500); 15 Jun 2010 11:21:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2246 invoked by uid 99); 15 Jun 2010 11:21:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 11:21:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 11:21:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5FBLObk008777
	for <common-issues@hadoop.apache.org>; Tue, 15 Jun 2010 11:21:24 GMT
Message-ID: <22656173.4401276600884559.JavaMail.jira@thor>
Date: Tue, 15 Jun 2010 07:21:24 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6821) Document changes to memory
 monitoring
In-Reply-To: <6410741.46811276277414864.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878936#action_12878936 ] 

Hudson commented on HADOOP-6821:
--------------------------------

Integrated in Hadoop-Common-trunk #367 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/367/])
    HADOOP-6821.  Document changes to memory monitoring.  Contributed by Hemanth Yamijala.


> Document changes to memory monitoring
> -------------------------------------
>
>                 Key: HADOOP-6821
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6821
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>    Affects Versions: 0.21.0
>            Reporter: Hemanth Yamijala
>            Assignee: Hemanth Yamijala
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6821.patch
>
>
> Modify the cluster_setup guide with information about memory monitoring and admin configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9108-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 15 18:25:50 2010
Return-Path: <common-issues-return-9108-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35728 invoked from network); 15 Jun 2010 18:25:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jun 2010 18:25:49 -0000
Received: (qmail 7127 invoked by uid 500); 15 Jun 2010 18:25:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6833 invoked by uid 500); 15 Jun 2010 18:25:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6824 invoked by uid 99); 15 Jun 2010 18:25:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 18:25:44 +0000
X-ASF-Spam-Status: No, hits=-1521.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 18:25:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5FIPO61013217
	for <common-issues@hadoop.apache.org>; Tue, 15 Jun 2010 18:25:24 GMT
Message-ID: <17162664.12171276626324139.JavaMail.jira@thor>
Date: Tue, 15 Jun 2010 14:25:24 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6815) refreshSuperUserGroupsConfiguration
 should use server side configuration for the refresh
In-Reply-To: <20039948.10941276119315214.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879058#action_12879058 ] 

Devaraj Das commented on HADOOP-6815:
-------------------------------------

bq. refreshUserToGroupsMappings() is also having the same client side configuration problem. Is there a separate ticket for this or is it being done in this issue itself? 
Yeah, although the conf is actually not used in the server (since the server already instantiated the Groups using the server's configuration), it is a good practice to not accept Configuration from clients for such commands... I'd propose that that issue is also addressed here...

> refreshSuperUserGroupsConfiguration should use server side configuration for the refresh
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6815
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6815
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6815-1.patch, HADOOP-6815-2.patch, HADOOP-6815.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9109-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 15 19:59:55 2010
Return-Path: <common-issues-return-9109-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75342 invoked from network); 15 Jun 2010 19:59:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jun 2010 19:59:55 -0000
Received: (qmail 33988 invoked by uid 500); 15 Jun 2010 19:59:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33915 invoked by uid 500); 15 Jun 2010 19:59:54 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33907 invoked by uid 99); 15 Jun 2010 19:59:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 19:59:54 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 19:59:52 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5FJxVOq014844
	for <common-issues@hadoop.apache.org>; Tue, 15 Jun 2010 19:59:31 GMT
Message-ID: <10842249.14951276631970984.JavaMail.jira@thor>
Date: Tue, 15 Jun 2010 15:59:30 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6794) Move configuration and script files
 post split
In-Reply-To: <11039662.93131275342217664.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879098#action_12879098 ] 

Hudson commented on HADOOP-6794:
--------------------------------

Integrated in Hadoop-Hdfs-trunk-Commit #310 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/310/])
    

> Move configuration and script files post split
> ----------------------------------------------
>
>                 Key: HADOOP-6794
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6794
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: conf, scripts
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6794.patch, HADOOP-6794.patch, HADOOP-6794.patch, HADOOP-6794.patch, HADOOP-6794.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9110-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 15 21:21:48 2010
Return-Path: <common-issues-return-9110-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44577 invoked from network); 15 Jun 2010 21:21:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jun 2010 21:21:48 -0000
Received: (qmail 22354 invoked by uid 500); 15 Jun 2010 21:21:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 22275 invoked by uid 500); 15 Jun 2010 21:21:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 22267 invoked by uid 99); 15 Jun 2010 21:21:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 21:21:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 21:21:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5FLLN69015791
	for <common-issues@hadoop.apache.org>; Tue, 15 Jun 2010 21:21:23 GMT
Message-ID: <6937853.16451276636883446.JavaMail.jira@thor>
Date: Tue, 15 Jun 2010 17:21:23 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6826) Revert FileSystem create method that
 takes CreateFlags
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Revert FileSystem create method that takes CreateFlags
------------------------------------------------------

                 Key: HADOOP-6826
                 URL: https://issues.apache.org/jira/browse/HADOOP-6826
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
            Reporter: Tom White
            Assignee: Tom White
            Priority: Blocker
             Fix For: 0.21.0


As discussed in HDFS-609 and HADOOP-5438 we should back out the FileSystem create() method that takes a set of CreateFlag objects, until the interface has been agreed upon and fully tested.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9111-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 15 21:25:47 2010
Return-Path: <common-issues-return-9111-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48855 invoked from network); 15 Jun 2010 21:25:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jun 2010 21:25:46 -0000
Received: (qmail 25514 invoked by uid 500); 15 Jun 2010 21:25:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25365 invoked by uid 500); 15 Jun 2010 21:25:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25357 invoked by uid 99); 15 Jun 2010 21:25:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 21:25:45 +0000
X-ASF-Spam-Status: No, hits=-1522.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 21:25:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5FLPOaT015815
	for <common-issues@hadoop.apache.org>; Tue, 15 Jun 2010 21:25:24 GMT
Message-ID: <26718418.16501276637124427.JavaMail.jira@thor>
Date: Tue, 15 Jun 2010 17:25:24 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6826) Revert FileSystem create method that
 takes CreateFlags
In-Reply-To: <6937853.16451276636883446.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6826:
------------------------------

    Attachment: HADOOP-6826.patch

Here's a patch that removes the public create method. It is not a straight revert of HADOOP-5438 since in the intervening time since this was committed the codebase has been split into three projects, and FileContext has been introduced which uses some of the same underlying code.

The HDFS part of this patch is in HDFS-609.

> Revert FileSystem create method that takes CreateFlags
> ------------------------------------------------------
>
>                 Key: HADOOP-6826
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6826
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6826.patch
>
>
> As discussed in HDFS-609 and HADOOP-5438 we should back out the FileSystem create() method that takes a set of CreateFlag objects, until the interface has been agreed upon and fully tested.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9112-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 15 21:25:47 2010
Return-Path: <common-issues-return-9112-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48859 invoked from network); 15 Jun 2010 21:25:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jun 2010 21:25:46 -0000
Received: (qmail 25541 invoked by uid 500); 15 Jun 2010 21:25:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 25431 invoked by uid 500); 15 Jun 2010 21:25:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 25366 invoked by uid 99); 15 Jun 2010 21:25:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 21:25:45 +0000
X-ASF-Spam-Status: No, hits=-1522.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 21:25:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5FLPPs7015827
	for <common-issues@hadoop.apache.org>; Tue, 15 Jun 2010 21:25:25 GMT
Message-ID: <19552691.16541276637125461.JavaMail.jira@thor>
Date: Tue, 15 Jun 2010 17:25:25 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6826) Revert FileSystem create method that
 takes CreateFlags
In-Reply-To: <6937853.16451276636883446.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6826:
------------------------------

    Status: Patch Available  (was: Open)

> Revert FileSystem create method that takes CreateFlags
> ------------------------------------------------------
>
>                 Key: HADOOP-6826
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6826
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6826.patch
>
>
> As discussed in HDFS-609 and HADOOP-5438 we should back out the FileSystem create() method that takes a set of CreateFlag objects, until the interface has been agreed upon and fully tested.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9113-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 15 21:33:47 2010
Return-Path: <common-issues-return-9113-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 57220 invoked from network); 15 Jun 2010 21:33:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jun 2010 21:33:46 -0000
Received: (qmail 39497 invoked by uid 500); 15 Jun 2010 21:33:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39442 invoked by uid 500); 15 Jun 2010 21:33:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39434 invoked by uid 99); 15 Jun 2010 21:33:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 21:33:46 +0000
X-ASF-Spam-Status: No, hits=-1522.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 21:33:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5FLXPge015955
	for <common-issues@hadoop.apache.org>; Tue, 15 Jun 2010 21:33:25 GMT
Message-ID: <32300319.16891276637605272.JavaMail.jira@thor>
Date: Tue, 15 Jun 2010 17:33:25 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5438) Merge FileSystem.create and
 FileSystem.append
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-5438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879127#action_12879127 ] 

Tom White commented on HADOOP-5438:
-----------------------------------

> file a jira for 21 and 22 to unmerge create() and append()

I've filed HADOOP-6826 for this.

> file a jira for 22 to come up with a new API on trunk that we're comfortable releasing (eg the better create method He described above or having two RPCs only in the case where you try to append to a file that doesn't exist).

FileContext has a create method that supports the case described in this JIRA - is this sufficient?

> Merge FileSystem.create and FileSystem.append
> ---------------------------------------------
>
>                 Key: HADOOP-5438
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5438
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>            Reporter: He Yongqiang
>            Assignee: He Yongqiang
>             Fix For: 0.21.0
>
>         Attachments: Hadoop-5438(2009-04-06).patch, Hadoop-5438-2009-03-30.patch, Hadoop-5438-2009-03-31-2.patch, Hadoop-5438-2009-03-31.patch, Hadoop-5438-2009-05-10.patch, Hadoop-5438-2009-05-15.patch, Hadoop-5438-2009-05-19.patch, Hadoop-5438-2009-05-5.patch
>
>
> Currently, when a user wants to modify a file, the user first calls exists() to know if this file is already there. And then uses create() or append() according to whether the file exists or not.
> the code looks like:
> {code}
> FSDataOutputStream out_1 = null;
> if (fs.exists(path_1))
>    out_1 = fs.append(path_1);
> else
>    out_1 = fs.create(path_1);
> {code}
> . On the performace side,It involes two RPCs. On the easy-of-use side, it is not very convient in contrast to the traditional open interface.
> It will more complicate if there is a overwrite parameter specified. I donot know whether there is a bug about 'overwrite' in 0.19, some times it takes a long time for overwrite creates to reture. So i make the write file code with overwrite param works like:
> {code}
> boolean exists = fs.exists(name);
> if (overwrite) {
>     if (exists)
>        fs.delete(name, true);
>      this.out = fs.create(name, overwrite, bufferSize, replication,
> 				    blockSize, progress);
>      this.currentRowID = 0;
>  } else {
>    if (!exists)
> 	this.out = fs.create(name, overwrite, bufferSize,
> 					replication, blockSize, progress);
>    else
> 	this.out = fs.append(name, bufferSize, progress);
> {code}
> Some code statements there are really redundant and not needed, especialy with the delete(). But without deleting first, the overwrite takes a long time to reture.
> BTW, i will create another issue about the overwrite problem. If it is not a bug at all or a duplicate, someone please close it.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9114-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 15 23:32:50 2010
Return-Path: <common-issues-return-9114-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31710 invoked from network); 15 Jun 2010 23:32:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 15 Jun 2010 23:32:50 -0000
Received: (qmail 70741 invoked by uid 500); 15 Jun 2010 23:32:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70661 invoked by uid 500); 15 Jun 2010 23:32:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70653 invoked by uid 99); 15 Jun 2010 23:32:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 23:32:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jun 2010 23:32:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5FNWPnC017259
	for <common-issues@hadoop.apache.org>; Tue, 15 Jun 2010 23:32:25 GMT
Message-ID: <19293184.19821276644745204.JavaMail.jira@thor>
Date: Tue, 15 Jun 2010 19:32:25 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6818) Provide a JNI-based implementation
 of GroupMappingServiceProvider
In-Reply-To: <17262845.29701276200793990.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879179#action_12879179 ] 

Devaraj Das commented on HADOOP-6818:
-------------------------------------

bq. Just a QQ. Is this a duplicate of HADOOP-4998? Or alteast forward compatible? 
No, this is providing a JNI implementation for just the GroupMappingService.

> Provide a JNI-based implementation of GroupMappingServiceProvider
> -----------------------------------------------------------------
>
>                 Key: HADOOP-6818
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6818
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: JNIGroupMapping.patch
>
>
> The default implementation of GroupMappingServiceProvider does a fork of a unix command to get the groups of a user. Since the group resolution happens in the servers, this might be costly. This jira aims at providing a JNI-based implementation for GroupMappingServiceProvider.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9115-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 16 00:47:49 2010
Return-Path: <common-issues-return-9115-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24827 invoked from network); 16 Jun 2010 00:47:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 00:47:49 -0000
Received: (qmail 28140 invoked by uid 500); 16 Jun 2010 00:47:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28051 invoked by uid 500); 16 Jun 2010 00:47:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28043 invoked by uid 99); 16 Jun 2010 00:47:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 00:47:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 00:47:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5G0lOss018014
	for <common-issues@hadoop.apache.org>; Wed, 16 Jun 2010 00:47:24 GMT
Message-ID: <25460038.21471276649244504.JavaMail.jira@thor>
Date: Tue, 15 Jun 2010 20:47:24 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6826) Revert FileSystem create method
 that takes CreateFlags
In-Reply-To: <6937853.16451276636883446.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879210#action_12879210 ] 

Hadoop QA commented on HADOOP-6826:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12447173/HADOOP-6826.patch
  against trunk revision 954647.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/585/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/585/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/585/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/585/console

This message is automatically generated.

> Revert FileSystem create method that takes CreateFlags
> ------------------------------------------------------
>
>                 Key: HADOOP-6826
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6826
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6826.patch
>
>
> As discussed in HDFS-609 and HADOOP-5438 we should back out the FileSystem create() method that takes a set of CreateFlag objects, until the interface has been agreed upon and fully tested.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9116-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 16 05:00:55 2010
Return-Path: <common-issues-return-9116-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 7080 invoked from network); 16 Jun 2010 05:00:52 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 05:00:52 -0000
Received: (qmail 53735 invoked by uid 500); 16 Jun 2010 04:00:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 53603 invoked by uid 500); 16 Jun 2010 04:00:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 53595 invoked by uid 99); 16 Jun 2010 04:00:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 04:00:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 04:00:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5G40OVr019318
	for <common-issues@hadoop.apache.org>; Wed, 16 Jun 2010 04:00:25 GMT
Message-ID: <20492408.23591276660824720.JavaMail.jira@thor>
Date: Wed, 16 Jun 2010 00:00:24 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6826) Revert FileSystem create method
 that takes CreateFlags
In-Reply-To: <6937853.16451276636883446.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879239#action_12879239 ] 

Tom White commented on HADOOP-6826:
-----------------------------------

There are no new tests since this patch removes code. The javadoc warning is an unrelated warning due to sun.security.krb5.* imports (from HADOOP-6526 and HADOOP-6603?) - is anyone looking at this?

> Revert FileSystem create method that takes CreateFlags
> ------------------------------------------------------
>
>                 Key: HADOOP-6826
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6826
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6826.patch
>
>
> As discussed in HDFS-609 and HADOOP-5438 we should back out the FileSystem create() method that takes a set of CreateFlag objects, until the interface has been agreed upon and fully tested.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9117-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 16 07:00:55 2010
Return-Path: <common-issues-return-9117-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71002 invoked from network); 16 Jun 2010 07:00:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 07:00:55 -0000
Received: (qmail 79960 invoked by uid 500); 16 Jun 2010 07:00:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79832 invoked by uid 500); 16 Jun 2010 07:00:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79824 invoked by uid 99); 16 Jun 2010 07:00:52 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 07:00:52 +0000
X-ASF-Spam-Status: No, hits=-1523.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 07:00:51 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5G70V4c020964
	for <common-issues@hadoop.apache.org>; Wed, 16 Jun 2010 07:00:31 GMT
Message-ID: <961168.25281276671631256.JavaMail.jira@thor>
Date: Wed, 16 Jun 2010 03:00:31 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6255) Create an rpm integration project
In-Reply-To: <1679708275.1252711317657.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879263#action_12879263 ] 

Konstantin Boudnik commented on HADOOP-6255:
--------------------------------------------

It seems like people here are fond of having a separate project from the Hadoop itself. While it seems like good idea I can see slight different approach. Here it is:

- native packaging specs (DEBs, RPMs, whatnot) are placed within the current build systems i.e. under {{$project.root/packaging/specs}}
- existing {{build.xml}} is extended with an extra target {{create-packages}} depends on perhaps {{tar}}
- an execution of 
  {{ant create-packages -Dpackage.type=RPM|DEB -Dpackage.arch=noarch|x32|x64|ARM}} 
  will exec a package creation script from {{$project.root/packaging/scripts}} using spec specified by {{package.type}}
- similarly, test packages can be produced by
  {{ant create-packages -Dpackage.type=RPM|DEB -Dpackage.class=test}}

IMO this approach should allow for reuse of some of existing build functionality without adding any extra hassle to the current build system. Besides, the packaging will be kept as a part of the project itself however will be physically separated from the build system.

Any dependencies resolvable as packages of certain type needs to be listed as external dependencies. Non-resolvable (like Avro above) will have to be included.

bq. We also need to separate the client from the server rpms
+1 on this one.


> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Owen O'Malley
>
> We should be able to create RPMs for Hadoop releases.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9118-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 16 07:06:51 2010
Return-Path: <common-issues-return-9118-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75665 invoked from network); 16 Jun 2010 07:06:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 07:06:51 -0000
Received: (qmail 83040 invoked by uid 500); 16 Jun 2010 07:06:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82923 invoked by uid 500); 16 Jun 2010 07:06:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82634 invoked by uid 99); 16 Jun 2010 07:06:47 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 07:06:47 +0000
X-ASF-Spam-Status: No, hits=-1523.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 07:06:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5G76QNU021027
	for <common-issues@hadoop.apache.org>; Wed, 16 Jun 2010 07:06:27 GMT
Message-ID: <7188648.25311276671986770.JavaMail.jira@thor>
Date: Wed, 16 Jun 2010 03:06:26 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6827) Extend the API of NetworkTopology
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Extend the API of NetworkTopology
---------------------------------

                 Key: HADOOP-6827
                 URL: https://issues.apache.org/jira/browse/HADOOP-6827
             Project: Hadoop Common
          Issue Type: New Feature
            Reporter: Rodrigo Schmidt
            Assignee: Rodrigo Schmidt


The current API for NetworkTopology is too restricted (and somehow biased towards the default block placement algorithm). 

For instance, it lacks methods to list racks or limit machines within a rack which are fundamental for HDFS-1094.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9119-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 16 17:10:29 2010
Return-Path: <common-issues-return-9119-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 679 invoked from network); 16 Jun 2010 14:43:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 14:43:28 -0000
Received: (qmail 36761 invoked by uid 500); 16 Jun 2010 14:42:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36722 invoked by uid 500); 16 Jun 2010 14:42:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36713 invoked by uid 99); 16 Jun 2010 14:42:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 14:42:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 14:42:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5GEgNh0024656
	for <common-issues@hadoop.apache.org>; Wed, 16 Jun 2010 14:42:23 GMT
Message-ID: <1988626.30181276699343718.JavaMail.jira@thor>
Date: Wed, 16 Jun 2010 10:42:23 -0400 (EDT)
From: "Arun C Murthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6827) Extend the API of NetworkTopology
In-Reply-To: <7188648.25311276671986770.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879351#action_12879351 ] 

Arun C Murthy commented on HADOOP-6827:
---------------------------------------

Rodrigo, can you please put up a concrete proposal? Thanks.

> Extend the API of NetworkTopology
> ---------------------------------
>
>                 Key: HADOOP-6827
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6827
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>
> The current API for NetworkTopology is too restricted (and somehow biased towards the default block placement algorithm). 
> For instance, it lacks methods to list racks or limit machines within a rack which are fundamental for HDFS-1094.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9120-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 16 17:14:37 2010
Return-Path: <common-issues-return-9120-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12685 invoked from network); 16 Jun 2010 16:22:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 16:22:48 -0000
Received: (qmail 99848 invoked by uid 500); 16 Jun 2010 16:22:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 99810 invoked by uid 500); 16 Jun 2010 16:22:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 99802 invoked by uid 99); 16 Jun 2010 16:22:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 16:22:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 16:22:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5GGMNof025416
	for <common-issues@hadoop.apache.org>; Wed, 16 Jun 2010 16:22:23 GMT
Message-ID: <16638806.31331276705343199.JavaMail.jira@thor>
Date: Wed, 16 Jun 2010 12:22:23 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6828) Herrior uses old way of accessing
 logs directories
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Herrior uses old way of accessing logs directories
--------------------------------------------------

                 Key: HADOOP-6828
                 URL: https://issues.apache.org/jira/browse/HADOOP-6828
             Project: Hadoop Common
          Issue Type: Bug
          Components: test
    Affects Versions: 0.21.0
            Reporter: Konstantin Boudnik
            Assignee: Sreekanth Ramakrishnan


Between 0.20 and 0.21 the way of passing log directories location has been changed from using environment variables to passing system properties. Herrior code needs to address it as well.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9125-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 16 17:20:51 2010
Return-Path: <common-issues-return-9125-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 51562 invoked from network); 16 Jun 2010 17:20:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 17:20:50 -0000
Received: (qmail 76503 invoked by uid 500); 16 Jun 2010 17:20:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76438 invoked by uid 500); 16 Jun 2010 17:20:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76430 invoked by uid 99); 16 Jun 2010 17:20:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 17:20:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 17:20:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5GHKPwu026020
	for <common-issues@hadoop.apache.org>; Wed, 16 Jun 2010 17:20:26 GMT
Message-ID: <11954733.32131276708825803.JavaMail.jira@thor>
Date: Wed, 16 Jun 2010 13:20:25 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6828) Herrior uses old way of accessing
 logs directories
In-Reply-To: <16638806.31331276705343199.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879398#action_12879398 ] 

Hadoop QA commented on HADOOP-6828:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12447236/HADOOP-6828.patch
  against trunk revision 954647.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/586/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/586/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/586/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/586/console

This message is automatically generated.

> Herrior uses old way of accessing logs directories
> --------------------------------------------------
>
>                 Key: HADOOP-6828
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6828
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Sreekanth Ramakrishnan
>         Attachments: HADOOP-6828.patch
>
>
> Between 0.20 and 0.21 the way of passing log directories location has been changed from using environment variables to passing system properties. Herrior code needs to address it as well.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9126-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 16 17:22:46 2010
Return-Path: <common-issues-return-9126-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 53829 invoked from network); 16 Jun 2010 17:22:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 17:22:46 -0000
Received: (qmail 81986 invoked by uid 500); 16 Jun 2010 17:22:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81930 invoked by uid 500); 16 Jun 2010 17:22:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81922 invoked by uid 99); 16 Jun 2010 17:22:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 17:22:45 +0000
X-ASF-Spam-Status: No, hits=-1524.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 17:22:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5GHMOoe026049
	for <common-issues@hadoop.apache.org>; Wed, 16 Jun 2010 17:22:24 GMT
Message-ID: <5459553.32201276708944498.JavaMail.jira@thor>
Date: Wed, 16 Jun 2010 13:22:24 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6829) Deprecated scripts shouldn't be used
 by Herriot
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Deprecated scripts shouldn't be used by Herriot
-----------------------------------------------

                 Key: HADOOP-6829
                 URL: https://issues.apache.org/jira/browse/HADOOP-6829
             Project: Hadoop Common
          Issue Type: Bug
          Components: test
    Affects Versions: 0.21.0
            Reporter: Konstantin Boudnik


{{hadoop-daemon.sh}} has been phased out in 0.21 and its usage for HDFS and MR was replaced by {{hdfs}} and {{mapred}} scripts.
Herriot shouldn't be using the deprecated script.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9123-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 16 17:23:31 2010
Return-Path: <common-issues-return-9123-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55329 invoked from network); 16 Jun 2010 17:23:29 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 17:23:29 -0000
Received: (qmail 71879 invoked by uid 500); 16 Jun 2010 17:16:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 71814 invoked by uid 500); 16 Jun 2010 17:16:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 71805 invoked by uid 99); 16 Jun 2010 17:16:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 17:16:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 17:16:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5GHGNCx025933
	for <common-issues@hadoop.apache.org>; Wed, 16 Jun 2010 17:16:23 GMT
Message-ID: <16436637.31871276708583720.JavaMail.jira@thor>
Date: Wed, 16 Jun 2010 13:16:23 -0400 (EDT)
From: "Gianmarco De Francisci Morales (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6758) MapFile.fix does not allow index
 interval definition
In-Reply-To: <14740497.57291273425048889.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gianmarco De Francisci Morales updated HADOOP-6758:
---------------------------------------------------

    Attachment: HADOOP-6758.patch

> MapFile.fix does not allow index interval definition
> ----------------------------------------------------
>
>                 Key: HADOOP-6758
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6758
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Gianmarco De Francisci Morales
>         Attachments: HADOOP-6758.patch, HADOOP-6758.patch, HADOOP-6758.patch, HADOOP-6758.patch
>
>
> When using the static methond MapFile.fix() there is no way to override the default IndexInterval that is 128.
> The IndexInterval should be taken from the configuration that is passed to the method.
> {code}
> int indexInterval = 128; 
> indexInterval = conf.getInt(INDEX_INTERVAL, indexInterval); 
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9127-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 16 17:24:47 2010
Return-Path: <common-issues-return-9127-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56279 invoked from network); 16 Jun 2010 17:24:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 17:24:47 -0000
Received: (qmail 85840 invoked by uid 500); 16 Jun 2010 17:24:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85705 invoked by uid 500); 16 Jun 2010 17:24:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85697 invoked by uid 99); 16 Jun 2010 17:24:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 17:24:46 +0000
X-ASF-Spam-Status: No, hits=-1524.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 17:24:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5GHOPQc026069
	for <common-issues@hadoop.apache.org>; Wed, 16 Jun 2010 17:24:25 GMT
Message-ID: <20609589.32251276709065198.JavaMail.jira@thor>
Date: Wed, 16 Jun 2010 13:24:25 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6829) Deprecated scripts shouldn't be
 used by Herriot
In-Reply-To: <5459553.32201276708944498.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879402#action_12879402 ] 

Konstantin Boudnik commented on HADOOP-6829:
--------------------------------------------

Actual changes are likely to happen in HDFS and MR components, so this is more like an umbrella feature.

> Deprecated scripts shouldn't be used by Herriot
> -----------------------------------------------
>
>                 Key: HADOOP-6829
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6829
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>
> {{hadoop-daemon.sh}} has been phased out in 0.21 and its usage for HDFS and MR was replaced by {{hdfs}} and {{mapred}} scripts.
> Herriot shouldn't be using the deprecated script.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9124-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 16 17:25:27 2010
Return-Path: <common-issues-return-9124-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56869 invoked from network); 16 Jun 2010 17:25:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 17:25:27 -0000
Received: (qmail 74298 invoked by uid 500); 16 Jun 2010 17:18:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74266 invoked by uid 500); 16 Jun 2010 17:18:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74256 invoked by uid 99); 16 Jun 2010 17:18:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 17:18:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 17:18:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5GHIN8U025943
	for <common-issues@hadoop.apache.org>; Wed, 16 Jun 2010 17:18:23 GMT
Message-ID: <8877207.31881276708703062.JavaMail.jira@thor>
Date: Wed, 16 Jun 2010 13:18:23 -0400 (EDT)
From: "Gianmarco De Francisci Morales (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6758) MapFile.fix does not allow index
 interval definition
In-Reply-To: <14740497.57291273425048889.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gianmarco De Francisci Morales updated HADOOP-6758:
---------------------------------------------------

    Attachment:     (was: HADOOP-6758.patch)

> MapFile.fix does not allow index interval definition
> ----------------------------------------------------
>
>                 Key: HADOOP-6758
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6758
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Gianmarco De Francisci Morales
>         Attachments: HADOOP-6758.patch, HADOOP-6758.patch, HADOOP-6758.patch
>
>
> When using the static methond MapFile.fix() there is no way to override the default IndexInterval that is 128.
> The IndexInterval should be taken from the configuration that is passed to the method.
> {code}
> int indexInterval = 128; 
> indexInterval = conf.getInt(INDEX_INTERVAL, indexInterval); 
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9121-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 16 17:49:45 2010
Return-Path: <common-issues-return-9121-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74326 invoked from network); 16 Jun 2010 17:49:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 17:49:45 -0000
Received: (qmail 30415 invoked by uid 500); 16 Jun 2010 16:49:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30350 invoked by uid 500); 16 Jun 2010 16:49:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30342 invoked by uid 99); 16 Jun 2010 16:49:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 16:49:44 +0000
X-ASF-Spam-Status: No, hits=-1524.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 16:49:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5GGnNAq025637
	for <common-issues@hadoop.apache.org>; Wed, 16 Jun 2010 16:49:23 GMT
Message-ID: <16282189.31441276706963764.JavaMail.jira@thor>
Date: Wed, 16 Jun 2010 12:49:23 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6828) Herrior uses old way of accessing
 logs directories
In-Reply-To: <16638806.31331276705343199.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6828:
---------------------------------------

    Attachment: HADOOP-6828.patch

Fixing the issue. Patch prepared by Sreekanth.

> Herrior uses old way of accessing logs directories
> --------------------------------------------------
>
>                 Key: HADOOP-6828
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6828
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Sreekanth Ramakrishnan
>         Attachments: HADOOP-6828.patch
>
>
> Between 0.20 and 0.21 the way of passing log directories location has been changed from using environment variables to passing system properties. Herrior code needs to address it as well.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9122-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 16 17:49:49 2010
Return-Path: <common-issues-return-9122-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 74393 invoked from network); 16 Jun 2010 17:49:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 17:49:48 -0000
Received: (qmail 30545 invoked by uid 500); 16 Jun 2010 16:49:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 30505 invoked by uid 500); 16 Jun 2010 16:49:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 30497 invoked by uid 99); 16 Jun 2010 16:49:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 16:49:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 16:49:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5GGnOc3025646
	for <common-issues@hadoop.apache.org>; Wed, 16 Jun 2010 16:49:24 GMT
Message-ID: <4485649.31471276706964175.JavaMail.jira@thor>
Date: Wed, 16 Jun 2010 12:49:24 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6828) Herrior uses old way of accessing
 logs directories
In-Reply-To: <16638806.31331276705343199.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6828:
---------------------------------------

    Status: Patch Available  (was: Open)

> Herrior uses old way of accessing logs directories
> --------------------------------------------------
>
>                 Key: HADOOP-6828
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6828
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Sreekanth Ramakrishnan
>         Attachments: HADOOP-6828.patch
>
>
> Between 0.20 and 0.21 the way of passing log directories location has been changed from using environment variables to passing system properties. Herrior code needs to address it as well.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9128-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 16 18:26:47 2010
Return-Path: <common-issues-return-9128-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95703 invoked from network); 16 Jun 2010 18:26:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 18:26:47 -0000
Received: (qmail 85307 invoked by uid 500); 16 Jun 2010 18:26:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 85243 invoked by uid 500); 16 Jun 2010 18:26:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 85235 invoked by uid 99); 16 Jun 2010 18:26:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 18:26:46 +0000
X-ASF-Spam-Status: No, hits=-1524.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 18:26:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5GIQP6X027037
	for <common-issues@hadoop.apache.org>; Wed, 16 Jun 2010 18:26:25 GMT
Message-ID: <2593415.34091276712785449.JavaMail.jira@thor>
Date: Wed, 16 Jun 2010 14:26:25 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6828) Herrior uses old way of accessing
 logs directories
In-Reply-To: <16638806.31331276705343199.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879438#action_12879438 ] 

Konstantin Boudnik commented on HADOOP-6828:
--------------------------------------------

- Test failure is unrelated
{noformat}
Problem with Trash.. Consider using -skipTrash option
rmr: Cannot move "file:/grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/build/test/data/testTrash/user/test" to the trash, as it contains the trash
{noformat}
- warnings seem to be coming either from {{SecurityUtil}} or {{KerberosName}}. I don't see any other warnings.

> Herrior uses old way of accessing logs directories
> --------------------------------------------------
>
>                 Key: HADOOP-6828
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6828
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Sreekanth Ramakrishnan
>         Attachments: HADOOP-6828.patch
>
>
> Between 0.20 and 0.21 the way of passing log directories location has been changed from using environment variables to passing system properties. Herrior code needs to address it as well.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9129-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 16 18:42:46 2010
Return-Path: <common-issues-return-9129-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 709 invoked from network); 16 Jun 2010 18:42:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 18:42:46 -0000
Received: (qmail 9701 invoked by uid 500); 16 Jun 2010 18:42:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 9592 invoked by uid 500); 16 Jun 2010 18:42:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 9584 invoked by uid 99); 16 Jun 2010 18:42:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 18:42:45 +0000
X-ASF-Spam-Status: No, hits=-1524.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 18:42:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5GIgOsL027477
	for <common-issues@hadoop.apache.org>; Wed, 16 Jun 2010 18:42:25 GMT
Message-ID: <5351640.35301276713744927.JavaMail.jira@thor>
Date: Wed, 16 Jun 2010 14:42:24 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6828) Herrior uses old way of accessing
 logs directories
In-Reply-To: <16638806.31331276705343199.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879446#action_12879446 ] 

Konstantin Boudnik commented on HADOOP-6828:
--------------------------------------------

+1 patch looks good.
In the verification of MAPREDUCE-1774 I have seen that the patch is working as expected. 
I'm going to commit this by the end of the day unless someone has onjections.

> Herrior uses old way of accessing logs directories
> --------------------------------------------------
>
>                 Key: HADOOP-6828
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6828
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Sreekanth Ramakrishnan
>         Attachments: HADOOP-6828.patch
>
>
> Between 0.20 and 0.21 the way of passing log directories location has been changed from using environment variables to passing system properties. Herrior code needs to address it as well.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9130-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 16 19:16:50 2010
Return-Path: <common-issues-return-9130-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23170 invoked from network); 16 Jun 2010 19:16:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 19:16:50 -0000
Received: (qmail 62846 invoked by uid 500); 16 Jun 2010 19:16:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62774 invoked by uid 500); 16 Jun 2010 19:16:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62763 invoked by uid 99); 16 Jun 2010 19:16:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 19:16:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 19:16:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5GJGOxk027996
	for <common-issues@hadoop.apache.org>; Wed, 16 Jun 2010 19:16:25 GMT
Message-ID: <24965972.36391276715784192.JavaMail.jira@thor>
Date: Wed, 16 Jun 2010 15:16:24 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6800) Harmonize JAR library versions
In-Reply-To: <8215141.145191275515876547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6800:
------------------------------

    Attachment: HADOOP-6800.patch

This is one of three patches (others at HDFS-1212, MAPREDUCE-1870) that removes duplicate JARs. It updates a few Apache Commons JARs, JUnit, and SLF4J.

The hardest bit was ensuring that Ant 1.6.5 and 1.7.1 don't both appear. Part of the problem here is that the Maven artifact has been relocated from the ant group to org.apache.ant. Jetty's jsp-2.1 depends on the old one, which is now explicitly excluded.

I ran a few tests with the new configuration and they passed. I also used the script in HADOOP-6342 to test that the combined tarballs had no conflicting dependencies. 

> Harmonize JAR library versions
> ------------------------------
>
>                 Key: HADOOP-6800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6800.patch
>
>
> Currently, multiple versions of the same library JAR are being pulled in for various reasons.
> * Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
> * Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9131-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 16 19:16:50 2010
Return-Path: <common-issues-return-9131-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23215 invoked from network); 16 Jun 2010 19:16:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 19:16:50 -0000
Received: (qmail 62985 invoked by uid 500); 16 Jun 2010 19:16:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62786 invoked by uid 500); 16 Jun 2010 19:16:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62777 invoked by uid 99); 16 Jun 2010 19:16:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 19:16:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 19:16:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5GJGQv9028011
	for <common-issues@hadoop.apache.org>; Wed, 16 Jun 2010 19:16:26 GMT
Message-ID: <25370099.36441276715786133.JavaMail.jira@thor>
Date: Wed, 16 Jun 2010 15:16:26 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6800) Harmonize JAR library versions
In-Reply-To: <8215141.145191275515876547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6800:
------------------------------

      Status: Patch Available  (was: Open)
    Assignee: Tom White

> Harmonize JAR library versions
> ------------------------------
>
>                 Key: HADOOP-6800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6800.patch
>
>
> Currently, multiple versions of the same library JAR are being pulled in for various reasons.
> * Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
> * Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9132-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 16 19:35:10 2010
Return-Path: <common-issues-return-9132-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33704 invoked from network); 16 Jun 2010 19:35:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 19:35:09 -0000
Received: (qmail 92471 invoked by uid 500); 16 Jun 2010 19:35:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92400 invoked by uid 500); 16 Jun 2010 19:35:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92392 invoked by uid 99); 16 Jun 2010 19:35:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 19:35:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 19:35:06 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5GJYj7J028201
	for <common-issues@hadoop.apache.org>; Wed, 16 Jun 2010 19:34:45 GMT
Message-ID: <3445701.36841276716885422.JavaMail.jira@thor>
Date: Wed, 16 Jun 2010 15:34:45 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6733) Create a test for FileSystem API
 compatibility between releases
In-Reply-To: <2563190.15351272511968608.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879474#action_12879474 ] 

Tom White commented on HADOOP-6733:
-----------------------------------

I'm going to test this and MAPREDUCE-1637 by compiling and running a few simple programs (e.g. word count) against both 0.20.2 and 0.21.0. This will also get more testing as folks try out their programs against the 0.21.0 release candidates and the release.

> Create a test for FileSystem API compatibility between releases
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6733
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6733
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs
>            Reporter: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>
> We should have an automated test for checking that programs written against an old version of the FileSystem API still run with a newer version. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9133-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 16 19:41:46 2010
Return-Path: <common-issues-return-9133-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35440 invoked from network); 16 Jun 2010 19:41:46 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 19:41:46 -0000
Received: (qmail 4151 invoked by uid 500); 16 Jun 2010 19:41:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4054 invoked by uid 500); 16 Jun 2010 19:41:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4046 invoked by uid 99); 16 Jun 2010 19:41:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 19:41:45 +0000
X-ASF-Spam-Status: No, hits=-1524.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 19:41:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5GJfOG5028246
	for <common-issues@hadoop.apache.org>; Wed, 16 Jun 2010 19:41:24 GMT
Message-ID: <5624423.36891276717284246.JavaMail.jira@thor>
Date: Wed, 16 Jun 2010 15:41:24 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6800) Harmonize JAR library versions
In-Reply-To: <8215141.145191275515876547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879476#action_12879476 ] 

Hadoop QA commented on HADOOP-6800:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12447262/HADOOP-6800.patch
  against trunk revision 954647.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/587/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/587/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/587/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/587/console

This message is automatically generated.

> Harmonize JAR library versions
> ------------------------------
>
>                 Key: HADOOP-6800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6800.patch
>
>
> Currently, multiple versions of the same library JAR are being pulled in for various reasons.
> * Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
> * Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9134-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 17 10:43:08 2010
Return-Path: <common-issues-return-9134-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5478 invoked from network); 16 Jun 2010 23:42:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 16 Jun 2010 23:42:49 -0000
Received: (qmail 15070 invoked by uid 500); 16 Jun 2010 23:42:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15013 invoked by uid 500); 16 Jun 2010 23:42:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15005 invoked by uid 99); 16 Jun 2010 23:42:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 23:42:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Jun 2010 23:42:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5GNgOGi001148
	for <common-issues@hadoop.apache.org>; Wed, 16 Jun 2010 23:42:24 GMT
Message-ID: <888126.43481276731744472.JavaMail.jira@thor>
Date: Wed, 16 Jun 2010 19:42:24 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6827) Extend the API of NetworkTopology
In-Reply-To: <7188648.25311276671986770.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879583#action_12879583 ] 

Rodrigo Schmidt commented on HADOOP-6827:
-----------------------------------------

I would like to extend it with the following API:

{code}
public String[] getRacks()  // returns the names of all racks in this topology
public int getRackSize(String  rackLocation) // returns the size of a rack
public int getIdInRack (Node  n) // returns the id (index in children) of a node n inside its rack
public Node getNodeInRack (String  rackLocation, int id) // the opposite of getIdInRack
{code}



> Extend the API of NetworkTopology
> ---------------------------------
>
>                 Key: HADOOP-6827
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6827
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>
> The current API for NetworkTopology is too restricted (and somehow biased towards the default block placement algorithm). 
> For instance, it lacks methods to list racks or limit machines within a rack which are fundamental for HDFS-1094.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9142-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 17 10:43:57 2010
Return-Path: <common-issues-return-9142-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5973 invoked from network); 17 Jun 2010 06:41:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jun 2010 06:41:28 -0000
Received: (qmail 96240 invoked by uid 500); 17 Jun 2010 06:14:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 96113 invoked by uid 500); 17 Jun 2010 06:14:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 96105 invoked by uid 99); 17 Jun 2010 06:14:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 06:14:44 +0000
X-ASF-Spam-Status: No, hits=-1526.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 06:14:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5H6EOW6005012
	for <common-issues@hadoop.apache.org>; Thu, 17 Jun 2010 06:14:24 GMT
Message-ID: <7563873.50241276755263985.JavaMail.jira@thor>
Date: Thu, 17 Jun 2010 02:14:23 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6748) Remove hadoop.cluster.administrators
In-Reply-To: <3308520.13571273038484086.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinod K V updated HADOOP-6748:
------------------------------

           Status: Open  (was: Patch Available)
         Assignee: Amareshwari Sriramadasu
    Fix Version/s: 0.21.0
                       (was: 0.22.0)

The patch needs a very minor merge in the testcase, it doesn't apply cleanly. Otherwise, there is only a trivial improvement for the patch - please add/retain the javadoc for the existing constructor in HttpServer.

> Remove hadoop.cluster.administrators
> ------------------------------------
>
>                 Key: HADOOP-6748
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6748
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.21.0
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.21.0
>
>         Attachments: patch-6748.txt
>
>
> Remove hadoop.cluster.administrators in favor of having separate configuration property name for MapReduce and HDFS projects.
> See more details on MAPREDUCE-1542.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9141-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 17 10:43:59 2010
Return-Path: <common-issues-return-9141-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 6370 invoked from network); 17 Jun 2010 06:41:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jun 2010 06:41:50 -0000
Received: (qmail 35761 invoked by uid 500); 17 Jun 2010 02:41:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 35677 invoked by uid 500); 17 Jun 2010 02:41:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 35528 invoked by uid 99); 17 Jun 2010 02:41:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 02:41:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 02:41:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5H2fOax002549
	for <common-issues@hadoop.apache.org>; Thu, 17 Jun 2010 02:41:24 GMT
Message-ID: <7525675.46611276742484281.JavaMail.jira@thor>
Date: Wed, 16 Jun 2010 22:41:24 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6815) refreshSuperUserGroupsConfiguration
 should use server side configuration for the refresh
In-Reply-To: <20039948.10941276119315214.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879637#action_12879637 ] 

Hadoop QA commented on HADOOP-6815:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12447295/HADOOP-6815-3.patch
  against trunk revision 955438.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    -1 javac.  The patch appears to cause tar ant target to fail.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    -1 contrib tests.  The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/588/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/588/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/588/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/588/console

This message is automatically generated.

> refreshSuperUserGroupsConfiguration should use server side configuration for the refresh
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6815
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6815
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6815-1.patch, HADOOP-6815-2.patch, HADOOP-6815-3.patch, HADOOP-6815.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9145-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 17 11:51:56 2010
Return-Path: <common-issues-return-9145-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2227 invoked from network); 17 Jun 2010 11:51:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jun 2010 11:51:49 -0000
Received: (qmail 1194 invoked by uid 500); 17 Jun 2010 07:47:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1170 invoked by uid 500); 17 Jun 2010 07:47:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1162 invoked by uid 99); 17 Jun 2010 07:47:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 07:47:44 +0000
X-ASF-Spam-Status: No, hits=-1526.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 07:47:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5H7lNZJ005818
	for <common-issues@hadoop.apache.org>; Thu, 17 Jun 2010 07:47:23 GMT
Message-ID: <31855693.51331276760843557.JavaMail.jira@thor>
Date: Thu, 17 Jun 2010 03:47:23 -0400 (EDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6748) Remove hadoop.cluster.administrators
In-Reply-To: <3308520.13571273038484086.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amareshwari Sriramadasu updated HADOOP-6748:
--------------------------------------------

    Attachment: patch-6748-1.txt

updated to trunk and added javadoc.

> Remove hadoop.cluster.administrators
> ------------------------------------
>
>                 Key: HADOOP-6748
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6748
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.21.0
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.21.0
>
>         Attachments: patch-6748-1.txt, patch-6748.txt
>
>
> Remove hadoop.cluster.administrators in favor of having separate configuration property name for MapReduce and HDFS projects.
> See more details on MAPREDUCE-1542.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9147-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 17 11:59:36 2010
Return-Path: <common-issues-return-9147-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15380 invoked from network); 17 Jun 2010 11:59:36 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jun 2010 11:59:36 -0000
Received: (qmail 58290 invoked by uid 500); 17 Jun 2010 10:12:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 58039 invoked by uid 500); 17 Jun 2010 10:12:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 58022 invoked by uid 99); 17 Jun 2010 10:12:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 10:12:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 10:12:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5HACOMq007150
	for <common-issues@hadoop.apache.org>; Thu, 17 Jun 2010 10:12:25 GMT
Message-ID: <10798446.52951276769544879.JavaMail.jira@thor>
Date: Thu, 17 Jun 2010 06:12:24 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6748) Remove
 hadoop.cluster.administrators
In-Reply-To: <3308520.13571273038484086.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879730#action_12879730 ] 

Hadoop QA commented on HADOOP-6748:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12447319/patch-6748-1.txt
  against trunk revision 955438.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/589/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/589/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/589/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/589/console

This message is automatically generated.

> Remove hadoop.cluster.administrators
> ------------------------------------
>
>                 Key: HADOOP-6748
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6748
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.21.0
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.21.0
>
>         Attachments: patch-6748-1.txt, patch-6748.txt
>
>
> Remove hadoop.cluster.administrators in favor of having separate configuration property name for MapReduce and HDFS projects.
> See more details on MAPREDUCE-1542.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9148-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 17 12:09:32 2010
Return-Path: <common-issues-return-9148-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35031 invoked from network); 17 Jun 2010 12:09:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jun 2010 12:09:32 -0000
Received: (qmail 73269 invoked by uid 500); 17 Jun 2010 10:22:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72702 invoked by uid 500); 17 Jun 2010 10:22:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72684 invoked by uid 99); 17 Jun 2010 10:22:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 10:22:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 10:22:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5HAMOQI007272
	for <common-issues@hadoop.apache.org>; Thu, 17 Jun 2010 10:22:24 GMT
Message-ID: <9048385.53141276770144796.JavaMail.jira@thor>
Date: Thu, 17 Jun 2010 06:22:24 -0400 (EDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6830) Javadoc warnings in trunk.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Javadoc warnings in trunk.
--------------------------

                 Key: HADOOP-6830
                 URL: https://issues.apache.org/jira/browse/HADOOP-6830
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Amareshwari Sriramadasu
             Fix For: 0.21.0


Trunk has following javadoc warnings :  
{noformat}   
    [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:31: warning: sun.security.krb5.Config is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.Config;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:32: warning: sun.security.krb5.KrbException is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.KrbException;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:81: warning: sun.security.krb5.Config is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc]   private static Config kerbConf;
     [exec]   [javadoc]                  ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/SecurityUtil.java:33: warning: sun.security.jgss.krb5.Krb5Util is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.jgss.krb5.Krb5Util;
     [exec]   [javadoc]                              ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/SecurityUtil.java:34: warning: sun.security.krb5.Credentials is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.Credentials;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/SecurityUtil.java:35: warning: sun.security.krb5.PrincipalName is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.PrincipalName;
     [exec]   [javadoc]       
{noformat}     

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9150-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 17 12:11:32 2010
Return-Path: <common-issues-return-9150-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37700 invoked from network); 17 Jun 2010 12:11:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jun 2010 12:11:32 -0000
Received: (qmail 74898 invoked by uid 500); 17 Jun 2010 10:24:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74791 invoked by uid 500); 17 Jun 2010 10:24:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74781 invoked by uid 99); 17 Jun 2010 10:24:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 10:24:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 10:24:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5HAONt4007306
	for <common-issues@hadoop.apache.org>; Thu, 17 Jun 2010 10:24:24 GMT
Message-ID: <3968526.53231276770263624.JavaMail.jira@thor>
Date: Thu, 17 Jun 2010 06:24:23 -0400 (EDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6748) Remove hadoop.cluster.administrators
In-Reply-To: <3308520.13571273038484086.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amareshwari Sriramadasu updated HADOOP-6748:
--------------------------------------------

    Attachment: patch-6748-0.21.txt

Patch for branch 0.21 with minor conflicts resolved.

test-patch for branch 0.21:
{noformat}
     [exec] +1 overall.
     [exec]
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec]
     [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
     [exec]
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec]
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec]
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec]
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec]
{noformat}

All unit tests passed on branch 0.21 as well.

> Remove hadoop.cluster.administrators
> ------------------------------------
>
>                 Key: HADOOP-6748
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6748
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.21.0
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.21.0
>
>         Attachments: patch-6748-0.21.txt, patch-6748-1.txt, patch-6748.txt
>
>
> Remove hadoop.cluster.administrators in favor of having separate configuration property name for MapReduce and HDFS projects.
> See more details on MAPREDUCE-1542.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9149-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 17 12:12:14 2010
Return-Path: <common-issues-return-9149-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38155 invoked from network); 17 Jun 2010 12:12:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jun 2010 12:12:14 -0000
Received: (qmail 74248 invoked by uid 500); 17 Jun 2010 10:22:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73097 invoked by uid 500); 17 Jun 2010 10:22:51 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72707 invoked by uid 99); 17 Jun 2010 10:22:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 10:22:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 10:22:47 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5HAMPxq007287
	for <common-issues@hadoop.apache.org>; Thu, 17 Jun 2010 10:22:25 GMT
Message-ID: <24357836.53191276770145590.JavaMail.jira@thor>
Date: Thu, 17 Jun 2010 06:22:25 -0400 (EDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6748) Remove
 hadoop.cluster.administrators
In-Reply-To: <3308520.13571273038484086.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879734#action_12879734 ] 

Amareshwari Sriramadasu commented on HADOOP-6748:
-------------------------------------------------

bq. -1 javadoc.

ant javadoc passed successfully on my machine. Looking at the console output I see the following javadoc warnings:

{noformat}
[exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:31: warning: sun.security.krb5.Config is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.Config;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:32: warning: sun.security.krb5.KrbException is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.KrbException;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/KerberosName.java:81: warning: sun.security.krb5.Config is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc]   private static Config kerbConf;
     [exec]   [javadoc]                  ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/SecurityUtil.java:33: warning: sun.security.jgss.krb5.Krb5Util is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.jgss.krb5.Krb5Util;
     [exec]   [javadoc]                              ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/SecurityUtil.java:34: warning: sun.security.krb5.Credentials is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.Credentials;
     [exec]   [javadoc]                         ^
     [exec]   [javadoc] /grid/0/hudson/hudson-slave/workspace/Hadoop-Patch-h4.grid.sp2.yahoo.net/trunk/src/java/org/apache/hadoop/security/SecurityUtil.java:35: warning: sun.security.krb5.PrincipalName is Sun proprietary API and may be removed in a future release
     [exec]   [javadoc] import sun.security.krb5.PrincipalName;
{noformat}
Raised HADOOP-6830 for this.



> Remove hadoop.cluster.administrators
> ------------------------------------
>
>                 Key: HADOOP-6748
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6748
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.21.0
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.21.0
>
>         Attachments: patch-6748-1.txt, patch-6748.txt
>
>
> Remove hadoop.cluster.administrators in favor of having separate configuration property name for MapReduce and HDFS projects.
> See more details on MAPREDUCE-1542.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9151-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 17 12:12:48 2010
Return-Path: <common-issues-return-9151-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 38480 invoked from network); 17 Jun 2010 12:12:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jun 2010 12:12:48 -0000
Received: (qmail 15936 invoked by uid 500); 17 Jun 2010 11:12:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15802 invoked by uid 500); 17 Jun 2010 11:12:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15778 invoked by uid 99); 17 Jun 2010 11:12:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 11:12:45 +0000
X-ASF-Spam-Status: No, hits=-1526.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 11:12:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5HBCNku007737
	for <common-issues@hadoop.apache.org>; Thu, 17 Jun 2010 11:12:24 GMT
Message-ID: <29298376.53991276773143869.JavaMail.jira@thor>
Date: Thu, 17 Jun 2010 07:12:23 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6828) Herrior uses old way of accessing
 logs directories
In-Reply-To: <16638806.31331276705343199.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879750#action_12879750 ] 

Hudson commented on HADOOP-6828:
--------------------------------

Integrated in Hadoop-Common-trunk #369 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/369/])
    HADOOP-6828. Herrior uses old way of accessing logs directories. Contributed by Sreekanth Ramakrishnan


> Herrior uses old way of accessing logs directories
> --------------------------------------------------
>
>                 Key: HADOOP-6828
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6828
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Sreekanth Ramakrishnan
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6828.patch
>
>
> Between 0.20 and 0.21 the way of passing log directories location has been changed from using environment variables to passing system properties. Herrior code needs to address it as well.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9138-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 17 12:44:24 2010
Return-Path: <common-issues-return-9138-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90269 invoked from network); 17 Jun 2010 12:44:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jun 2010 12:44:24 -0000
Received: (qmail 90132 invoked by uid 500); 17 Jun 2010 01:37:44 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90077 invoked by uid 500); 17 Jun 2010 01:37:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90066 invoked by uid 99); 17 Jun 2010 01:37:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 01:37:44 +0000
X-ASF-Spam-Status: No, hits=-1525.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 01:37:43 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5H1bNPr002010
	for <common-issues@hadoop.apache.org>; Thu, 17 Jun 2010 01:37:23 GMT
Message-ID: <26458678.45431276738643370.JavaMail.jira@thor>
Date: Wed, 16 Jun 2010 21:37:23 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6828) Herrior uses old way of accessing
 logs directories
In-Reply-To: <16638806.31331276705343199.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6828:
---------------------------------------

           Status: Resolved  (was: Patch Available)
     Hadoop Flags: [Reviewed]
    Fix Version/s: 0.21.0
       Resolution: Fixed

I have just committed this. Thanks Sreekanth.

> Herrior uses old way of accessing logs directories
> --------------------------------------------------
>
>                 Key: HADOOP-6828
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6828
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Sreekanth Ramakrishnan
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6828.patch
>
>
> Between 0.20 and 0.21 the way of passing log directories location has been changed from using environment variables to passing system properties. Herrior code needs to address it as well.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9143-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 17 12:48:32 2010
Return-Path: <common-issues-return-9143-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95406 invoked from network); 17 Jun 2010 12:48:31 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jun 2010 12:48:31 -0000
Received: (qmail 64743 invoked by uid 500); 17 Jun 2010 07:21:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64205 invoked by uid 500); 17 Jun 2010 07:21:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64193 invoked by uid 99); 17 Jun 2010 07:21:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 07:21:45 +0000
X-ASF-Spam-Status: No, hits=-1526.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 07:21:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5H7LOFu005546
	for <common-issues@hadoop.apache.org>; Thu, 17 Jun 2010 07:21:24 GMT
Message-ID: <7903648.50901276759284601.JavaMail.jira@thor>
Date: Thu, 17 Jun 2010 03:21:24 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6748) Remove
 hadoop.cluster.administrators
In-Reply-To: <3308520.13571273038484086.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879698#action_12879698 ] 

Vinod K V commented on HADOOP-6748:
-----------------------------------


Also to be forward compatible with changes planned in HADOOP-6715, can we change the exception message for invalid access to
{code}
 "User " + remoteUser + " is unauthorized to access this page.
 Access control list for this page: " + adminsAcl.toString() + "."
{code}
instead of the present message?

> Remove hadoop.cluster.administrators
> ------------------------------------
>
>                 Key: HADOOP-6748
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6748
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.21.0
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.21.0
>
>         Attachments: patch-6748.txt
>
>
> Remove hadoop.cluster.administrators in favor of having separate configuration property name for MapReduce and HDFS projects.
> See more details on MAPREDUCE-1542.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9139-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 17 12:50:30 2010
Return-Path: <common-issues-return-9139-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97178 invoked from network); 17 Jun 2010 12:50:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jun 2010 12:50:30 -0000
Received: (qmail 98639 invoked by uid 500); 17 Jun 2010 01:43:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98598 invoked by uid 500); 17 Jun 2010 01:43:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98585 invoked by uid 99); 17 Jun 2010 01:43:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 01:43:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 01:43:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5H1hNqJ002139
	for <common-issues@hadoop.apache.org>; Thu, 17 Jun 2010 01:43:23 GMT
Message-ID: <2228373.45821276739003669.JavaMail.jira@thor>
Date: Wed, 16 Jun 2010 21:43:23 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6733) Create a test for FileSystem API
 compatibility between releases
In-Reply-To: <2563190.15351272511968608.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879623#action_12879623 ] 

Konstantin Boudnik commented on HADOOP-6733:
--------------------------------------------

This seems to be a very reasonable and sorta cheap approach of testing backward compatibility. The only thing needs to be kept in mind that set of test applications needs to be consistent from release to release. Otherwise, such verification won't be representative enough. But I guess word count doesn't go nowhere :)

> Create a test for FileSystem API compatibility between releases
> ---------------------------------------------------------------
>
>                 Key: HADOOP-6733
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6733
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs
>            Reporter: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>
> We should have an automated test for checking that programs written against an old version of the FileSystem API still run with a newer version. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9144-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 17 12:54:32 2010
Return-Path: <common-issues-return-9144-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3670 invoked from network); 17 Jun 2010 12:54:32 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jun 2010 12:54:32 -0000
Received: (qmail 72911 invoked by uid 500); 17 Jun 2010 07:27:52 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72334 invoked by uid 500); 17 Jun 2010 07:27:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72317 invoked by uid 99); 17 Jun 2010 07:27:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 07:27:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 07:27:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5H7ROQW005625
	for <common-issues@hadoop.apache.org>; Thu, 17 Jun 2010 07:27:25 GMT
Message-ID: <2069233.51081276759644932.JavaMail.jira@thor>
Date: Thu, 17 Jun 2010 03:27:24 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6748) Remove
 hadoop.cluster.administrators
In-Reply-To: <3308520.13571273038484086.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879701#action_12879701 ] 

Vinod K V commented on HADOOP-6748:
-----------------------------------

bq. can we change the exception message for invalid access ...
Please ignore this, it is being handled in HADOOP-6715 itself.

> Remove hadoop.cluster.administrators
> ------------------------------------
>
>                 Key: HADOOP-6748
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6748
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.21.0
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.21.0
>
>         Attachments: patch-6748.txt
>
>
> Remove hadoop.cluster.administrators in favor of having separate configuration property name for MapReduce and HDFS projects.
> See more details on MAPREDUCE-1542.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9140-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 17 13:00:31 2010
Return-Path: <common-issues-return-9140-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 8401 invoked from network); 17 Jun 2010 13:00:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jun 2010 13:00:30 -0000
Received: (qmail 5577 invoked by uid 500); 17 Jun 2010 01:53:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5536 invoked by uid 500); 17 Jun 2010 01:53:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5528 invoked by uid 99); 17 Jun 2010 01:53:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 01:53:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 01:53:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5H1rO3X002263
	for <common-issues@hadoop.apache.org>; Thu, 17 Jun 2010 01:53:24 GMT
Message-ID: <23006339.46151276739604320.JavaMail.jira@thor>
Date: Wed, 16 Jun 2010 21:53:24 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6828) Herrior uses old way of accessing
 logs directories
In-Reply-To: <16638806.31331276705343199.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12879627#action_12879627 ] 

Hudson commented on HADOOP-6828:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #310 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/310/])
    HADOOP-6828. Herrior uses old way of accessing logs directories. Contributed by Sreekanth Ramakrishnan


> Herrior uses old way of accessing logs directories
> --------------------------------------------------
>
>                 Key: HADOOP-6828
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6828
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Sreekanth Ramakrishnan
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6828.patch
>
>
> Between 0.20 and 0.21 the way of passing log directories location has been changed from using environment variables to passing system properties. Herrior code needs to address it as well.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9146-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 17 13:16:33 2010
Return-Path: <common-issues-return-9146-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29642 invoked from network); 17 Jun 2010 13:16:33 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jun 2010 13:16:33 -0000
Received: (qmail 3006 invoked by uid 500); 17 Jun 2010 07:49:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 2975 invoked by uid 500); 17 Jun 2010 07:49:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 2958 invoked by uid 99); 17 Jun 2010 07:49:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 07:49:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 07:49:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5H7nNh3005833
	for <common-issues@hadoop.apache.org>; Thu, 17 Jun 2010 07:49:24 GMT
Message-ID: <15139541.51371276760963774.JavaMail.jira@thor>
Date: Thu, 17 Jun 2010 03:49:23 -0400 (EDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6748) Remove hadoop.cluster.administrators
In-Reply-To: <3308520.13571273038484086.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amareshwari Sriramadasu updated HADOOP-6748:
--------------------------------------------

    Status: Patch Available  (was: Open)

> Remove hadoop.cluster.administrators
> ------------------------------------
>
>                 Key: HADOOP-6748
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6748
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.21.0
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.21.0
>
>         Attachments: patch-6748-1.txt, patch-6748.txt
>
>
> Remove hadoop.cluster.administrators in favor of having separate configuration property name for MapReduce and HDFS projects.
> See more details on MAPREDUCE-1542.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9135-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 17 14:02:26 2010
Return-Path: <common-issues-return-9135-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64444 invoked from network); 17 Jun 2010 14:02:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jun 2010 14:02:26 -0000
Received: (qmail 55575 invoked by uid 500); 17 Jun 2010 00:35:46 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55452 invoked by uid 500); 17 Jun 2010 00:35:45 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55438 invoked by uid 99); 17 Jun 2010 00:35:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 00:35:45 +0000
X-ASF-Spam-Status: No, hits=-1525.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 00:35:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5H0ZPdg001454
	for <common-issues@hadoop.apache.org>; Thu, 17 Jun 2010 00:35:25 GMT
Message-ID: <927791.44091276734925264.JavaMail.jira@thor>
Date: Wed, 16 Jun 2010 20:35:25 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6815) refreshSuperUserGroupsConfiguration
 should use server side configuration for the refresh
In-Reply-To: <20039948.10941276119315214.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6815:
-----------------------------------

    Status: Open  (was: Patch Available)

> refreshSuperUserGroupsConfiguration should use server side configuration for the refresh
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6815
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6815
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6815-1.patch, HADOOP-6815-2.patch, HADOOP-6815-3.patch, HADOOP-6815.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9136-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 17 14:02:29 2010
Return-Path: <common-issues-return-9136-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64500 invoked from network); 17 Jun 2010 14:02:29 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jun 2010 14:02:29 -0000
Received: (qmail 55652 invoked by uid 500); 17 Jun 2010 00:35:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55602 invoked by uid 500); 17 Jun 2010 00:35:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55594 invoked by uid 99); 17 Jun 2010 00:35:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 00:35:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 00:35:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5H0ZOhT001445
	for <common-issues@hadoop.apache.org>; Thu, 17 Jun 2010 00:35:24 GMT
Message-ID: <2240944.44061276734924572.JavaMail.jira@thor>
Date: Wed, 16 Jun 2010 20:35:24 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6815) refreshSuperUserGroupsConfiguration
 should use server side configuration for the refresh
In-Reply-To: <20039948.10941276119315214.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6815:
-----------------------------------

    Attachment: HADOOP-6815-3.patch

> refreshSuperUserGroupsConfiguration should use server side configuration for the refresh
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6815
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6815
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6815-1.patch, HADOOP-6815-2.patch, HADOOP-6815-3.patch, HADOOP-6815.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9137-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 17 14:02:31 2010
Return-Path: <common-issues-return-9137-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 64510 invoked from network); 17 Jun 2010 14:02:29 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Jun 2010 14:02:29 -0000
Received: (qmail 55749 invoked by uid 500); 17 Jun 2010 00:35:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 55612 invoked by uid 500); 17 Jun 2010 00:35:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 55603 invoked by uid 99); 17 Jun 2010 00:35:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 00:35:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2010 00:35:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5H0ZPTx001463
	for <common-issues@hadoop.apache.org>; Thu, 17 Jun 2010 00:35:25 GMT
Message-ID: <6812072.44121276734925535.JavaMail.jira@thor>
Date: Wed, 16 Jun 2010 20:35:25 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6815) refreshSuperUserGroupsConfiguration
 should use server side configuration for the refresh
In-Reply-To: <20039948.10941276119315214.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6815:
-----------------------------------

    Status: Patch Available  (was: Open)

> refreshSuperUserGroupsConfiguration should use server side configuration for the refresh
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6815
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6815
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6815-1.patch, HADOOP-6815-2.patch, HADOOP-6815-3.patch, HADOOP-6815.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9152-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 18 00:53:49 2010
Return-Path: <common-issues-return-9152-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9679 invoked from network); 18 Jun 2010 00:53:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Jun 2010 00:53:49 -0000
Received: (qmail 28301 invoked by uid 500); 18 Jun 2010 00:53:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28198 invoked by uid 500); 18 Jun 2010 00:53:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28188 invoked by uid 99); 18 Jun 2010 00:53:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 00:53:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 00:53:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5I0rOLV018144
	for <common-issues@hadoop.apache.org>; Fri, 18 Jun 2010 00:53:24 GMT
Message-ID: <33409591.71081276822404558.JavaMail.jira@thor>
Date: Thu, 17 Jun 2010 20:53:24 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6810) Extract a subset of tests for smoke
 (DOA) validation
In-Reply-To: <10151977.193211275695163699.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6810:
---------------------------------------

    Attachment: smoke-tests.1

This patch addresses the issue with test duplication entries. Please review.

> Extract a subset of tests for smoke (DOA) validation
> ----------------------------------------------------
>
>                 Key: HADOOP-6810
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6810
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: smoke-tests.1, smoke-tests.patch, smoke-tests.patch, smoke-tests.patch, smoke-tests.patch, smoke-tests.patch, smoke-tests.patch.1
>
>
> Currently, there's no way to quickly find out if a new build of Hadoop is reasonably viable and can be used for future development/testing cycle.
> Similarly to 'commit' testing it'd be great to have a wider subset of existing tests to be executed inside of 30 minutes. The main goal of such a subset is to have a reasonable guarantees that different parts of Hadoop's functionality aren't broken and the system is not DOA.
> Full test run takes about 3+ hours and it is a too long wait time in many cases. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9152-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 18 01:40:19 2010
Return-Path: <common-issues-return-9152-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 9679 invoked from network); 18 Jun 2010 00:53:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Jun 2010 00:53:49 -0000
Received: (qmail 28301 invoked by uid 500); 18 Jun 2010 00:53:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28198 invoked by uid 500); 18 Jun 2010 00:53:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28188 invoked by uid 99); 18 Jun 2010 00:53:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 00:53:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 00:53:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5I0rOLV018144
	for <common-issues@hadoop.apache.org>; Fri, 18 Jun 2010 00:53:24 GMT
Message-ID: <33409591.71081276822404558.JavaMail.jira@thor>
Date: Thu, 17 Jun 2010 20:53:24 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6810) Extract a subset of tests for smoke
 (DOA) validation
In-Reply-To: <10151977.193211275695163699.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Boudnik updated HADOOP-6810:
---------------------------------------

    Attachment: smoke-tests.1

This patch addresses the issue with test duplication entries. Please review.

> Extract a subset of tests for smoke (DOA) validation
> ----------------------------------------------------
>
>                 Key: HADOOP-6810
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6810
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Boudnik
>            Assignee: Konstantin Boudnik
>         Attachments: smoke-tests.1, smoke-tests.patch, smoke-tests.patch, smoke-tests.patch, smoke-tests.patch, smoke-tests.patch, smoke-tests.patch.1
>
>
> Currently, there's no way to quickly find out if a new build of Hadoop is reasonably viable and can be used for future development/testing cycle.
> Similarly to 'commit' testing it'd be great to have a wider subset of existing tests to be executed inside of 30 minutes. The main goal of such a subset is to have a reasonable guarantees that different parts of Hadoop's functionality aren't broken and the system is not DOA.
> Full test run takes about 3+ hours and it is a too long wait time in many cases. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9154-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 18 03:17:59 2010
Return-Path: <common-issues-return-9154-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15074 invoked from network); 18 Jun 2010 02:57:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Jun 2010 02:57:51 -0000
Received: (qmail 20207 invoked by uid 500); 18 Jun 2010 02:57:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20176 invoked by uid 500); 18 Jun 2010 02:57:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20168 invoked by uid 99); 18 Jun 2010 02:57:50 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 02:57:50 +0000
X-ASF-Spam-Status: No, hits=-1529.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 02:57:50 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5I2vUhI019508
	for <common-issues@hadoop.apache.org>; Fri, 18 Jun 2010 02:57:30 GMT
Message-ID: <7109692.73021276829850062.JavaMail.jira@thor>
Date: Thu, 17 Jun 2010 22:57:30 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6255) Create an rpm integration project
In-Reply-To: <1679708275.1252711317657.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880058#action_12880058 ] 

Konstantin Boudnik commented on HADOOP-6255:
--------------------------------------------

After a good discussion with Owen and Roman here's the a better proposal:
- new top level targets to be introduced into ant build:
-- package-32
-- package-64
-- package-noarch
-- package depends on all above

It is up to build to find out about the type of OS the build is running on and either to locate appropriate packaging script and schema or fail with appropriate diagnostics.

The preliminary structure of installed packages is like this:
{noformat}
$root/
  bin/
    hadoop
    hadoop-daemon?.sh
    hdfs
    mapred
    <other user facing scripts>
  share/
    hadoop/
      bin/
        hadoop-config.sh
      lib/
        *.jar
      man/
      include/
        c++/
      sbin/
        jvsc
        taskcontroller
        runAs (for Herriot packages)
{noformat}
Some notes:
- jar files  in {{share/hadoop/lib/}} have to have their owners based on the components they are coming from (e.g. hdfs-client, hdfs-server, etc.)
- packages required by Hadoop but aren't included into its source code (LZO is a good example) shall be delivered via inter-package dependencies.

Something has to be done about configs. Shall they be placed under {{/etc/hadoop}} perhaps?

Herriot package (test for real cluster as in HADOOP-6332) shall be created separately because it requires byte-code instrumentation.

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Owen O'Malley
>
> We should be able to create RPMs for Hadoop releases.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9153-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 18 03:17:59 2010
Return-Path: <common-issues-return-9153-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15057 invoked from network); 18 Jun 2010 02:57:50 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Jun 2010 02:57:50 -0000
Received: (qmail 20158 invoked by uid 500); 18 Jun 2010 02:57:50 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 20011 invoked by uid 500); 18 Jun 2010 02:57:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20003 invoked by uid 99); 18 Jun 2010 02:57:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 02:57:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 02:57:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5I2vOCS019442
	for <common-issues@hadoop.apache.org>; Fri, 18 Jun 2010 02:57:25 GMT
Message-ID: <14839369.72821276829844606.JavaMail.jira@thor>
Date: Thu, 17 Jun 2010 22:57:24 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6762) exception while doing RPC I/O
 closes channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880057#action_12880057 ] 

Todd Lipcon commented on HADOOP-6762:
-------------------------------------

Hey Sam, one small thing in the newest patch: there are some unspecified generics - eg should be new AtomicReference<Throwable>, not just new AtomicReference. Otherwise we get some warnings compiling the tests.

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt, hadoop-6762-7.txt, hadoop-6762-8.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9155-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 18 04:18:06 2010
Return-Path: <common-issues-return-9155-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23450 invoked from network); 18 Jun 2010 04:17:55 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Jun 2010 04:17:55 -0000
Received: (qmail 72835 invoked by uid 500); 18 Jun 2010 04:17:55 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72678 invoked by uid 500); 18 Jun 2010 04:17:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72651 invoked by uid 99); 18 Jun 2010 04:17:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 04:17:52 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 04:17:49 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5I4HSCe021415
	for <common-issues@hadoop.apache.org>; Fri, 18 Jun 2010 04:17:28 GMT
Message-ID: <16715611.73701276834648027.JavaMail.jira@thor>
Date: Fri, 18 Jun 2010 00:17:28 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6255) Create an rpm integration project
In-Reply-To: <1679708275.1252711317657.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880065#action_12880065 ] 

Allen Wittenauer commented on HADOOP-6255:
------------------------------------------

Are we trying to build a package that fits the local OS or are we trying to build a package that has hadoop completely in one dir and is fairly generic?

If the former, trying to dictate where things like config files are installed is going to break.  Every OS has fairly specific rules and expectations (Linux has LSB, Solaris has filesystems(5), NeXTStep... I mean, OS X, has something documented somewhere, I'm sure, etc...).  If the latter, then I'd make the following changes:

a) drop the share level.  it doesn't seem to serve a purpose

b) etc/ should contain configs

c) lib should contain hadoop-config.sh and *.so in addtion to *.jar

d) i take it hdfs/mapred/etc is a 0.21 thing?  are we concerned about the number of people that have 'hdfs' aliased to 'hadoop dfs'?  or is it a replacement for that?

e) var/tmp and var/logs should be defined

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Owen O'Malley
>
> We should be able to create RPMs for Hadoop releases.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9156-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 18 10:02:09 2010
Return-Path: <common-issues-return-9156-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50488 invoked from network); 18 Jun 2010 10:02:09 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Jun 2010 10:02:09 -0000
Received: (qmail 83632 invoked by uid 500); 18 Jun 2010 10:02:09 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 83487 invoked by uid 500); 18 Jun 2010 10:02:06 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 83479 invoked by uid 99); 18 Jun 2010 10:02:05 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 10:02:05 +0000
X-ASF-Spam-Status: No, hits=-1529.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 10:02:05 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5IA1iP2024013
	for <common-issues@hadoop.apache.org>; Fri, 18 Jun 2010 10:01:45 GMT
Message-ID: <14900842.76961276855304905.JavaMail.jira@thor>
Date: Fri, 18 Jun 2010 06:01:44 -0400 (EDT)
From: "Vinod K V (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6748) Remove
 hadoop.cluster.administrators
In-Reply-To: <3308520.13571273038484086.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880149#action_12880149 ] 

Vinod K V commented on HADOOP-6748:
-----------------------------------

+1 for the latest patch.

> Remove hadoop.cluster.administrators
> ------------------------------------
>
>                 Key: HADOOP-6748
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6748
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.21.0
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.21.0
>
>         Attachments: patch-6748-0.21.txt, patch-6748-1.txt, patch-6748.txt
>
>
> Remove hadoop.cluster.administrators in favor of having separate configuration property name for MapReduce and HDFS projects.
> See more details on MAPREDUCE-1542.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9157-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 18 12:36:02 2010
Return-Path: <common-issues-return-9157-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 15042 invoked from network); 18 Jun 2010 12:36:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Jun 2010 12:36:01 -0000
Received: (qmail 40338 invoked by uid 500); 18 Jun 2010 12:36:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 39980 invoked by uid 500); 18 Jun 2010 12:35:58 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 39970 invoked by uid 99); 18 Jun 2010 12:35:57 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 12:35:57 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 12:35:55 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5ICZXoH025265
	for <common-issues@hadoop.apache.org>; Fri, 18 Jun 2010 12:35:33 GMT
Message-ID: <13396350.78421276864533309.JavaMail.jira@thor>
Date: Fri, 18 Jun 2010 08:35:33 -0400 (EDT)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6255) Create an rpm integration project
In-Reply-To: <1679708275.1252711317657.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880179#action_12880179 ] 

Steve Loughran commented on HADOOP-6255:
----------------------------------------

I've been pushing for this to be downstream of the initial tar process as you may want to let people build their own RPMs, with their own config files. The tar file to create the RPMs is a redistributable all of its own. If I understood source RPMs, they would probably fit into the story somehow too.



> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Owen O'Malley
>
> We should be able to create RPMs for Hadoop releases.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9158-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 18 12:40:51 2010
Return-Path: <common-issues-return-9158-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 16155 invoked from network); 18 Jun 2010 12:40:51 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Jun 2010 12:40:51 -0000
Received: (qmail 48631 invoked by uid 500); 18 Jun 2010 12:40:51 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48512 invoked by uid 500); 18 Jun 2010 12:40:49 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48503 invoked by uid 99); 18 Jun 2010 12:40:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 12:40:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 12:40:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5ICeON7025335
	for <common-issues@hadoop.apache.org>; Fri, 18 Jun 2010 12:40:25 GMT
Message-ID: <21890517.78541276864824636.JavaMail.jira@thor>
Date: Fri, 18 Jun 2010 08:40:24 -0400 (EDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6748) Remove hadoop.cluster.administrators
In-Reply-To: <3308520.13571273038484086.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amareshwari Sriramadasu updated HADOOP-6748:
--------------------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

Thanks for the review Vinod.
I just committed this.

> Remove hadoop.cluster.administrators
> ------------------------------------
>
>                 Key: HADOOP-6748
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6748
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.21.0
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.21.0
>
>         Attachments: patch-6748-0.21.txt, patch-6748-1.txt, patch-6748.txt
>
>
> Remove hadoop.cluster.administrators in favor of having separate configuration property name for MapReduce and HDFS projects.
> See more details on MAPREDUCE-1542.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9159-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 18 12:55:48 2010
Return-Path: <common-issues-return-9159-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 23346 invoked from network); 18 Jun 2010 12:55:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Jun 2010 12:55:48 -0000
Received: (qmail 77879 invoked by uid 500); 18 Jun 2010 12:55:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77756 invoked by uid 500); 18 Jun 2010 12:55:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77744 invoked by uid 99); 18 Jun 2010 12:55:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 12:55:45 +0000
X-ASF-Spam-Status: No, hits=-1530.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 12:55:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5ICtOwF025532
	for <common-issues@hadoop.apache.org>; Fri, 18 Jun 2010 12:55:24 GMT
Message-ID: <31419027.78861276865724359.JavaMail.jira@thor>
Date: Fri, 18 Jun 2010 08:55:24 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6748) Remove
 hadoop.cluster.administrators
In-Reply-To: <3308520.13571273038484086.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880188#action_12880188 ] 

Hudson commented on HADOOP-6748:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #311 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/311/])
    HADOOP-6748. Removes hadoop.cluster.administrators, cluster administrators acl is passed as parameter in constructor. Contributed by Amareshwari Sriramadasu


> Remove hadoop.cluster.administrators
> ------------------------------------
>
>                 Key: HADOOP-6748
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6748
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.21.0
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.21.0
>
>         Attachments: patch-6748-0.21.txt, patch-6748-1.txt, patch-6748.txt
>
>
> Remove hadoop.cluster.administrators in favor of having separate configuration property name for MapReduce and HDFS projects.
> See more details on MAPREDUCE-1542.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9160-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 18 14:09:38 2010
Return-Path: <common-issues-return-9160-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49462 invoked from network); 18 Jun 2010 14:09:37 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Jun 2010 14:09:37 -0000
Received: (qmail 70381 invoked by uid 500); 18 Jun 2010 14:09:37 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70277 invoked by uid 500); 18 Jun 2010 14:09:36 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Delivered-To: moderator for common-issues@hadoop.apache.org
Received: (qmail 23965 invoked by uid 99); 18 Jun 2010 10:38:12 -0000
X-ASF-Spam-Status: No, hits=-3.6 required=10.0
	tests=ASF_LIST_OPS,RCVD_IN_DNSWL_MED,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
X-SpamScore: -10
X-BigFish: VPS-10(zz9251Ka0dJ936eMzz1202hzzz2dh)
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-Class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: FW: hadoop-cluster-1.ic node errors and warnings
Date: Fri, 18 Jun 2010 16:07:15 +0530
Message-ID: <8D269469729C0848B728757E2CCF04E70341720D@wa-beexch-1-1.cybage.com>
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Thread-Topic: hadoop-cluster-1.ic node errors and warnings
thread-index: AcsOzQddkEEEk7v/RDWkrY1BvIApQgAA+bKw
From: Bikash Singhal <bikashs@cybage.com>
To: <common-dev@hadoop.apache.org>,
	<common-user-subscribe@hadoop.apache.org>,
	<common-user@hadoop.apache.org>,
	<common-issues@hadoop.apache.org>
CC: "Tina Malpure" <tinam@cybage.com>
X-OriginalArrivalTime: 18 Jun 2010 10:37:15.0950 (UTC) FILETIME=[383AF4E0:01CB0ED2]
X-Reverse-DNS: 121.242.40.50.static-pune.vsnl.net.in
X-Virus-Checked: Checked by ClamAV on apache.org

Hi folks ,

I have received this error in the hadoop cluster. Has anybody anybody
seen this . Any solution. =

--------------------- namenode errors and warning ------------------ =


2010-06-17 18:35:19,970 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Not able to place
enough replicas, still in need of 1
2010-06-17 18:35:19,972 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
ask 10.13.53.127:50010 to delete  blk_-5851901968674962584_28816
blk_-5704762921662804278_28806 blk_-3249260253840047858_28816
blk_-6785644112288085978_28816 blk_-5991019206688403229_28806
blk_3615269337590779570_28806 blk_-1181514666968321558_28806
blk_7264052281086809701_28816 blk_663323733333557946_28781
blk_5673553217372004676_28816 blk_3626732201125971756_28806
blk_8707314634323417208_28816 blk_-6887179921277000590_28812
blk_-3816018501532347017_28816 blk_-7238540159191278524_28816
blk_-7658691789280578071_28806 blk_3236948345360966700_28816
blk_-6254109926970173434_28816 blk_5195430663886472445_28816
blk_-412573738832546671_28816 blk_8109473696289375765_28806
blk_-5810373118380394822_28816 blk_2743979608816557043_28806
blk_5715402730928093013_28816 blk_8464503407273685120_28806
blk_-5062152410152898606_28816 blk_-3177617939821456482_28810
blk_-5555427680789719632_28816 blk_-5789474774332045366_28806
  blk_-5623935836874134523_28806 blk_-4337702879554700741_28806
blk_-8461056881907762006_28806 blk_9026822631507898528_28816
blk_7156167055494887627_28816 blk_-8277210689227324100_28806
blk_-3751852098002911697_28806 blk_-5585636802791023762_28806
blk_7895962823848655899_28806 blk_-990310910545048635_28816
blk_-9154860186123815888_28806 blk_-4227488676014658152_28806
blk_8936924022792123139_28816 blk_-769244617229872944_28816
blk_2688806916705051324_28816 blk_-4338943142397506199_28816
blk_-509403461442878177_28806 blk_-4842888174619311915_28815
blk_-3952777283518699846_28806 blk_-5861717742640359549_28806
blk_-3952576171995960772_28816 blk_-163789650674468878_28816
blk_-4592795213928607535_28806 blk_2333271861949132522_28816
blk_4198208275476238889_28816 blk_360552057991500806_28816
blk_5491773302900671409_28806 blk_4047676415653175342_28806
blk_-1004260037826453175_28806 blk_2767440385264091225_28816
blk_5476421277228240141_28816 blk_4407005806101654351_28816
blk_4195254311
 970318771_28816 blk_-737647080244712447_28806 blk_7438398943!
 33029124
8_28806 blk_9065794226469565702_28806 blk_-909573278974779662_28816
blk_-720057463076992315_28806 blk_-3706189772409350949_28816
blk_-7669121304118161359_28816 blk_5626381233374602629_28816
blk_3226781108133048989_28806 blk_4534712638713702794_28816
blk_-2501367440326715009_28816 blk_1182598706752583066_28806
blk_2899109514732336036_28813 blk_-3583975497823582887_28806
blk_1637573118385058229_28816 blk_6123281255845765914_28816
blk_-7325625322659708014_28816 blk_5881568644646580533_28816
blk_-3117891195870623616_28816 blk_4293852484387807756_28806
blk_1053300130581981980_28781 blk_2196698001825902338_28809
2010-06-17 18:35:19,974 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
ask 10.13.53.128:50010 to delete  blk_-5851901968674962584_28816
blk_-5991019206688403229_28806 blk_-3249260253840047858_28816
blk_-1074403248426101769_28816 blk_1102552384917878009_28816
blk_663323733333557946_28781 blk_7264052281086809701_28816
blk_-9186151702356219036_28806 blk_-6887179921277000590_28812
blk_-7403187390102989856_28814 blk_-3816018501532347017_28816
blk_-3256647808151302716_28806 blk_5195430663886472445_28816
blk_-6254109926970173434_28816 blk_-412573738832546671_28816
blk_8109473696289375765_28806 blk_2743979608816557043_28806
blk_8464503407273685120_28806 blk_-1927566815393964066_28813
blk_-910165942916728217_28816 blk_6991381516571090085_28816
blk_-5062152410152898606_28816 blk_-5555427680789719632_28816
blk_-347418184816360801_28806 blk_9026822631507898528_28816
blk_8645717205835711727_28816 blk_-8277210689227324100_28806
blk_-6820670802505184305_28809 blk_7156167055494887627_28816 =

 blk_-3751852098002911697_28806 blk_2230966748060172045_28816
blk_-9154860186123815888_28806 blk_-4227488676014658152_28806
blk_8936924022792123139_28816 blk_-3952777283518699846_28806
blk_-163789650674468878_28816 blk_2333271861949132522_28816
blk_5491773302900671409_28806 blk_4047676415653175342_28806
blk_7147022655053153746_28806 blk_5476421277228240141_28816
blk_3601702425471675469_28806 blk_-1812525521195434130_28809
blk_6208014296194515686_28816 blk_-3588835264704726772_28816
blk_5487381930889488938_28806 blk_1786827488554715485_28806
blk_6381806163859245084_28806 blk_-909573278974779662_28816
blk_-720057463076992315_28806 blk_-3706189772409350949_28816
blk_5626381233374602629_28816 blk_8812361351759351875_28806
blk_3226781108133048989_28806 blk_2626414889098125951_28816
blk_5642048906322764709_28806 blk_-3583975497823582887_28806
blk_1637573118385058229_28816 blk_-7325625322659708014_28816
blk_4293852484387807756_28806 blk_7869940696281550339_28806
blk_2196698001825902
 338_28809 blk_-6887326635813252129_28816 blk_-57047629216628!
 04278_28
806 blk_3615269337590779570_28806 blk_-6785644112288085978_28816
blk_3880871563914097435_28806 blk_-1181514666968321558_28806
blk_3626732201125971756_28806 blk_-5111860011657356431_28812
blk_5673553217372004676_28816 blk_1144230701070834232_28816
blk_8707314634323417208_28816 blk_-2478324223605041774_28808
blk_-7238540159191278524_28816 blk_-7658691789280578071_28806
blk_3236948345360966700_28816 blk_744702532009597216_28816
blk_-5810373118380394822_28816 blk_2247163771290221731_28808
blk_5715402730928093013_28816 blk_-3177617939821456482_28810
blk_-5789474774332045366_28806 blk_-5623935836874134523_28806
blk_-4682289668534212518_28816 blk_-4337702879554700741_28806
blk_-8461056881907762006_28806 blk_3225744222537772912_28816
blk_-7377461388357398241_28816 blk_-5585636802791023762_28806
blk_7787185831361186700_28816 blk_7895962823848655899_28806
blk_-990310910545048635_28816 blk_-1876758814669931378_28816
blk_-769244617229872944_28816 blk_-497469852942635196_28806 blk_-433894
 3142397506199_28816 blk_2688806916705051324_28816
blk_-6654089746586683607_28816 blk_-509403461442878177_28806
2010-06-17 18:35:19,975 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.addStoredBlock: blockMap updated: 10.13.53.127:50010 is added
to blk_151014847393206974_28824 size 41837
2010-06-17 18:35:19,978 INFO org.apache.hadoop.hdfs.StateChange: DIR*
NameSystem.completeFile: file
/var/lib/hadoop-0.20/cache/hadoop/mapred/system/job_201006090123_0068/jo
b.xml is closed by DFSClient_-1596407549
2010-06-17 18:35:19,986 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=3Dhadoop,hadoop	ip=3D/10.13.53.127	cmd=3Dopen
src=3D/var/lib/hadoop-0.20/cache/hadoop/mapred/system/job_201006090123_006
8/job.xml	dst=3Dnull	perm=3Dnull
2010-06-17 18:35:20,078 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=3Dhadoop,hadoop	ip=3D/10.13.53.127	cmd=3Dopen
src=3D/var/lib/hadoop-0.20/cache/hadoop/mapred/system/job_201006090123_006
8/job.jar	dst=3Dnull	perm=3Dnull
2010-06-17 18:35:21,581 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=3Dhadoop,hadoop	ip=3D/10.13.53.127	cmd=3Dcreate
src=3D/i2a_logs/2010-06-16/output_i2a-coll-6.ic-/_logs/history/hadoop-mast
er_1276071808358_job_201006090123_0068_hadoop_I2ALogArchiver	dst=3Dnull
perm=3Dhadoop:supergroup:rw-r--r--
2010-06-17 18:35:21,710 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=3Dhadoop,hadoop	ip=3D/10.13.53.127	cmd=3Dcreate
src=3D/i2a_logs/2010-06-16/output_i2a-coll-6.ic-/_logs/history/hadoop-mast
er_1276071808358_job_201006090123_0068_conf.xml	dst=3Dnull
perm=3Dhadoop:supergroup:rw-r--r--
2010-06-17 18:35:21,731 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.allocateBlock:
/i2a_logs/2010-06-16/output_i2a-coll-6.ic-/_logs/history/hadoop-master_1
276071808358_job_201006090123_0068_conf.xml.
blk_1315602836759907786_28826
2010-06-17 18:35:21,748 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.addStoredBlock: blockMap updated: 10.13.53.128:50010 is added
to blk_1315602836759907786_28826 size 41823
--
2010-06-17 18:45:11,591 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
PendingReplicationMonitor timed out block blk_-2915452315391280184_28822
2010-06-17 18:50:03,643 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
transactions: 676 Total time for transactions(ms): 11Number of
transactions batched in Syncs: 4 Number of syncs: 69 SyncTimes(ms): 3632

2010-06-17 18:50:03,645 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=3Dhadoop,hadoop	ip=3D/10.13.53.129	cmd=3Dcreate
src=3D/i2a_logs/2010-06-17/i2a-coll-4.ic-100617-182031-access.log.0
dst=3Dnull	perm=3Dhadoop:supergroup:rw-r--r--
2010-06-17 18:50:03,659 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.allocateBlock:
/i2a_logs/2010-06-17/i2a-coll-4.ic-100617-182031-access.log.0.
blk_-5308288487531094520_28832
2010-06-17 18:50:04,363 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.addStoredBlock: blockMap updated: 10.13.53.156:50010 is added
to blk_-5308288487531094520_28832 size 31804984
2010-06-17 18:50:04,363 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.addStoredBlock: blockMap updated: 10.13.53.128:50010 is added
to blk_-5308288487531094520_28832 size 31804984
2010-06-17 18:50:04,364 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.addStoredBlock: blockMap updated: 10.13.53.129:50010 is added
to blk_-5308288487531094520_28832 size 31804984
2010-06-17 18:50:04,366 INFO org.apache.hadoop.hdfs.StateChange: DIR*
NameSystem.completeFile: file
/i2a_logs/2010-06-17/i2a-coll-4.ic-100617-182031-access.log.0 is closed
by DFSClient_-1916945035
2010-06-17 18:50:06,172 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=3Dhadoop,hadoop	ip=3D/10.13.53.129	cmd=3Dcreate
src=3D/i2a_logs/2010-06-17/i2a-coll-4.ic-100617-182531-access.log.0
dst=3Dnull	perm=3Dhadoop:supergroup:rw-r--r--
2010-06-17 18:50:06,187 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.allocateBlock:
/i2a_logs/2010-06-17/i2a-coll-4.ic-100617-182531-access.log.0.
blk_1130168086083501329_28833
2010-06-17 18:50:07,024 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.addStoredBlock: blockMap updated: 10.13.53.127:50010 is added
to blk_1130168086083501329_28833 size 32278706

 --------------------- jobtracker errors and warning ------------------ =


2010-06-17 17:00:06,382 WARN org.apache.hadoop.conf.Configuration:
/var/lib/hadoop-0.20/cache/hadoop/mapred/local/jobTracker/job_2010060901
23_0067.xml:a attempt to override final parameter: dfs.data.dir;
Ignoring.
2010-06-17 17:00:06,448 DEBUG org.apache.hadoop.mapred.JobTracker:
Per-Task memory configuration is not set on JT. Not checking the job for
invalid memory requirements.
2010-06-17 17:00:06,449 INFO org.apache.hadoop.mapred.JobTracker:
Initializing job_201006090123_0067
2010-06-17 17:00:06,451 INFO org.apache.hadoop.mapred.JobInProgress:
Initializing job_201006090123_0067
2010-06-17 17:00:06,698 INFO org.apache.hadoop.mapred.JobInProgress:
Input size for job job_201006090123_0067 =3D 8493462348. Number of splits
=3D 288
2010-06-17 17:00:06,698 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201006090123_0067_m_000000 has split on
node:/default-rack/10.13.53.127
2010-06-17 17:00:06,699 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201006090123_0067_m_000000 has split on
node:/default-rack/10.13.53.156
2010-06-17 17:00:06,699 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201006090123_0067_m_000000 has split on
node:/default-rack/10.13.53.128
2010-06-17 17:00:06,699 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201006090123_0067_m_000001 has split on
node:/default-rack/10.13.53.127
2010-06-17 17:00:06,699 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201006090123_0067_m_000001 has split on
node:/default-rack/10.13.53.129
2010-06-17 17:00:06,699 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201006090123_0067_m_000001 has split on
node:/default-rack/10.13.53.128
--
2010-06-17 18:35:20,073 WARN org.apache.hadoop.conf.Configuration:
/var/lib/hadoop-0.20/cache/hadoop/mapred/local/jobTracker/job_2010060901
23_0068.xml:a attempt to override final parameter: dfs.data.dir;
Ignoring.
2010-06-17 18:35:20,132 DEBUG org.apache.hadoop.mapred.JobTracker:
Per-Task memory configuration is not set on JT. Not checking the job for
invalid memory requirements.
2010-06-17 18:35:20,133 INFO org.apache.hadoop.mapred.JobTracker:
Initializing job_201006090123_0068
2010-06-17 18:35:20,134 INFO org.apache.hadoop.mapred.JobInProgress:
Initializing job_201006090123_0068
2010-06-17 18:35:21,123 DEBUG org.apache.hadoop.mapred.JobTracker: Got
heartbeat from:
tracker_10.13.53.128:localhost.localdomain/127.0.0.1:35313 (restarted:
false initialContact: false acceptNewTasks: true) with responseId:
-32475
2010-06-17 18:35:21,186 DEBUG org.apache.hadoop.mapred.JobTracker: Got
heartbeat from:
tracker_10.13.53.127:localhost.localdomain/127.0.0.1:45581 (restarted:
false initialContact: false acceptNewTasks: true) with responseId:
-32074
2010-06-17 18:35:21,776 INFO org.apache.hadoop.mapred.JobInProgress:
Input size for job job_201006090123_0068 =3D 33327229. Number of splits =3D=

288
2010-06-17 18:35:21,776 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201006090123_0068_m_000000 has split on
node:/default-rack/10.13.53.127
2010-06-17 18:35:21,776 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201006090123_0068_m_000000 has split on
node:/default-rack/10.13.53.156
2010-06-17 18:35:21,777 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201006090123_0068_m_000000 has split on
node:/default-rack/10.13.53.129
2010-06-17 18:35:21,777 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201006090123_0068_m_000001 has split on
node:/default-rack/10.13.53.127

 --------------------- datanode errors and warning ------------------ =



Thanks & Regards

Bikash Singhal =




"Legal Disclaimer: This electronic message and all contents contain informa=
tion from Cybage Software Private Limited which may be privileged, confiden=
tial, or otherwise protected from disclosure. The information is intended t=
o be for the addressee(s) only. If you are not an addressee, any disclosure=
, copy, distribution, or use of the contents of this message is strictly pr=
ohibited. If you have received this electronic message in error please noti=
fy the sender by reply e-mail to and destroy the original message and all c=
opies. Cybage has taken every reasonable precaution to minimize the risk of=
 malicious content in the mail, but is not liable for any damage you may su=
stain as a result of any malicious content in this e-mail. You should carry=
 out your own malicious content checks before opening the e-mail or attachm=
ent." 
www.cybage.com



From common-issues-return-9161-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 18 17:28:48 2010
Return-Path: <common-issues-return-9161-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32673 invoked from network); 18 Jun 2010 17:28:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Jun 2010 17:28:48 -0000
Received: (qmail 46410 invoked by uid 500); 18 Jun 2010 17:28:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 46369 invoked by uid 500); 18 Jun 2010 17:28:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 46361 invoked by uid 99); 18 Jun 2010 17:28:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 17:28:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 17:28:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5IHSNDh027920
	for <common-issues@hadoop.apache.org>; Fri, 18 Jun 2010 17:28:23 GMT
Message-ID: <10641836.82171276882103782.JavaMail.jira@thor>
Date: Fri, 18 Jun 2010 13:28:23 -0400 (EDT)
From: "sam rash (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6762) exception while doing RPC I/O closes
 channel
In-Reply-To: <11173395.27001273681601359.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

sam rash updated HADOOP-6762:
-----------------------------

    Attachment: hadoop-6762-9.txt

AtomicReference typed now

> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: sam rash
>         Attachments: hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt, hadoop-6762-7.txt, hadoop-6762-8.txt, hadoop-6762-9.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9162-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 18 17:51:10 2010
Return-Path: <common-issues-return-9162-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39840 invoked from network); 18 Jun 2010 17:51:10 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Jun 2010 17:51:10 -0000
Received: (qmail 91444 invoked by uid 500); 18 Jun 2010 17:51:10 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91407 invoked by uid 500); 18 Jun 2010 17:51:09 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91399 invoked by uid 99); 18 Jun 2010 17:51:09 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 17:51:09 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 17:51:06 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5IHoj58028143
	for <common-issues@hadoop.apache.org>; Fri, 18 Jun 2010 17:50:45 GMT
Message-ID: <7963219.82761276883445117.JavaMail.jira@thor>
Date: Fri, 18 Jun 2010 13:50:45 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6255) Create an rpm integration project
In-Reply-To: <1679708275.1252711317657.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880266#action_12880266 ] 

Allen Wittenauer commented on HADOOP-6255:
------------------------------------------

After a night of sleep, the following occurred to me:

a) the stuff in sbin should actually be in libexec, along with hadoop-config.sh
b) hdfs, mapred, and hadoop-dameon.sh (assuming those are all admin tools) should be in sbin
c) why include/c++ ?  shouldn't it just be include?

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Owen O'Malley
>
> We should be able to create RPMs for Hadoop releases.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9163-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 18 20:56:49 2010
Return-Path: <common-issues-return-9163-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 94180 invoked from network); 18 Jun 2010 20:56:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Jun 2010 20:56:49 -0000
Received: (qmail 84260 invoked by uid 500); 18 Jun 2010 20:56:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84203 invoked by uid 500); 18 Jun 2010 20:56:48 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84195 invoked by uid 99); 18 Jun 2010 20:56:48 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 20:56:48 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 20:56:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5IKuOTS020258
	for <common-issues@hadoop.apache.org>; Fri, 18 Jun 2010 20:56:24 GMT
Message-ID: <7085757.85441276894584144.JavaMail.jira@thor>
Date: Fri, 18 Jun 2010 16:56:24 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6800) Harmonize JAR library versions
In-Reply-To: <8215141.145191275515876547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880322#action_12880322 ] 

Tom White commented on HADOOP-6800:
-----------------------------------

Is anyone able to review this (and related patches)? Thanks.

> Harmonize JAR library versions
> ------------------------------
>
>                 Key: HADOOP-6800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6800.patch
>
>
> Currently, multiple versions of the same library JAR are being pulled in for various reasons.
> * Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
> * Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9164-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 18 23:10:56 2010
Return-Path: <common-issues-return-9164-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32014 invoked from network); 18 Jun 2010 23:10:56 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 18 Jun 2010 23:10:56 -0000
Received: (qmail 94806 invoked by uid 500); 18 Jun 2010 23:10:56 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 94728 invoked by uid 500); 18 Jun 2010 23:10:55 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 94719 invoked by uid 99); 18 Jun 2010 23:10:55 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 23:10:55 +0000
X-ASF-Spam-Status: No, hits=-1531.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jun 2010 23:10:54 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5INAYEL002938
	for <common-issues@hadoop.apache.org>; Fri, 18 Jun 2010 23:10:34 GMT
Message-ID: <5817907.88051276902634700.JavaMail.jira@thor>
Date: Fri, 18 Jun 2010 19:10:34 -0400 (EDT)
From: "Roman Shaposhnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6255) Create an rpm integration project
In-Reply-To: <1679708275.1252711317657.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880359#action_12880359 ] 

Roman Shaposhnik commented on HADOOP-6255:
------------------------------------------

I have two points to add to the discussion:
   1. I'm wondering whether it would be useful to slice it a bit more thinly. IOW, introducing the notion of these extra 
       top level targets available for packaging:
              hadoop-core
              hadoop-client
              hadoop-daemon
              hadoop-devel
              hadoop-javadoc
     

    2. As for configs, I'd like to point out an example that Debian has established with their packaging of .20. Basically
        they created one package per node type (http://packages.qa.debian.org/h/hadoop.html) plus one package common
        among all the daemons:
              hadoop-daemons-common 
              hadoop-jobtrackerd
              hadoop-tasktrackerd
              hadoop-datanoded
              hadoop-namenoded
              hadoop-secondarynamenoded
 
         The packages themselves are pretty slim -- containing only hooks to make daemons plug into the service management
         system (init.d in Debian's case, but one would imagine Solaris/SMF or anything like that also being an option for us). 
         I also would tend to believe that these could be reasonable packages to be used for splitting the configs appropriately. 

> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Owen O'Malley
>
> We should be able to create RPMs for Hadoop releases.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9165-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 19 11:13:49 2010
Return-Path: <common-issues-return-9165-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 28560 invoked from network); 19 Jun 2010 11:13:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Jun 2010 11:13:49 -0000
Received: (qmail 87116 invoked by uid 500); 19 Jun 2010 11:13:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86976 invoked by uid 500); 19 Jun 2010 11:13:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86968 invoked by uid 99); 19 Jun 2010 11:13:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 19 Jun 2010 11:13:45 +0000
X-ASF-Spam-Status: No, hits=-1532.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 19 Jun 2010 11:13:45 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5JBDOlD007055
	for <common-issues@hadoop.apache.org>; Sat, 19 Jun 2010 11:13:24 GMT
Message-ID: <26565738.93791276946004676.JavaMail.jira@thor>
Date: Sat, 19 Jun 2010 07:13:24 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6748) Remove
 hadoop.cluster.administrators
In-Reply-To: <3308520.13571273038484086.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880470#action_12880470 ] 

Hudson commented on HADOOP-6748:
--------------------------------

Integrated in Hadoop-Common-trunk #371 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/371/])
    HADOOP-6748. Removes hadoop.cluster.administrators, cluster administrators acl is passed as parameter in constructor. Contributed by Amareshwari Sriramadasu


> Remove hadoop.cluster.administrators
> ------------------------------------
>
>                 Key: HADOOP-6748
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6748
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.21.0
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.21.0
>
>         Attachments: patch-6748-0.21.txt, patch-6748-1.txt, patch-6748.txt
>
>
> Remove hadoop.cluster.administrators in favor of having separate configuration property name for MapReduce and HDFS projects.
> See more details on MAPREDUCE-1542.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9166-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 19 20:04:48 2010
Return-Path: <common-issues-return-9166-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44309 invoked from network); 19 Jun 2010 20:04:48 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 19 Jun 2010 20:04:48 -0000
Received: (qmail 25056 invoked by uid 500); 19 Jun 2010 20:04:48 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24982 invoked by uid 500); 19 Jun 2010 20:04:47 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24974 invoked by uid 99); 19 Jun 2010 20:04:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 19 Jun 2010 20:04:47 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 19 Jun 2010 20:04:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5JK4NvD009230
	for <common-issues@hadoop.apache.org>; Sat, 19 Jun 2010 20:04:23 GMT
Message-ID: <8280048.96281276977863173.JavaMail.jira@thor>
Date: Sat, 19 Jun 2010 16:04:23 -0400 (EDT)
From: "Victor Pakhomov (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6831) "java.lang.ArithmeticException:
 Non-terminating decimal expansion; no exact representable decimal result."
 while running "hadoop jar hadoop-0.20.1+169.89-examples.jar pi 4 30"
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

"java.lang.ArithmeticException: Non-terminating decimal expansion; no exact representable decimal result." while running "hadoop jar hadoop-0.20.1+169.89-examples.jar pi 4 30"
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-6831
                 URL: https://issues.apache.org/jira/browse/HADOOP-6831
             Project: Hadoop Common
          Issue Type: Bug
          Components: util
    Affects Versions: 0.20.1
         Environment: ubuntu 10.04 lucid
sun jdk 6
hadoop packaged by cloudera
            Reporter: Victor Pakhomov
             Fix For: 0.20.1


If I run "hadoop jar hadoop-0.20.1+169.89-examples.jar pi 4 30", I get the following output:

Number of Maps  = 4
Samples per Map = 30
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Starting Job
10/06/19 21:50:34 INFO mapred.FileInputFormat: Total input paths to process : 4
10/06/19 21:50:34 INFO mapred.JobClient: Running job: job_201006192101_0003
10/06/19 21:50:35 INFO mapred.JobClient:  map 0% reduce 0%
10/06/19 21:50:45 INFO mapred.JobClient:  map 50% reduce 0%
10/06/19 21:50:51 INFO mapred.JobClient:  map 100% reduce 0%
10/06/19 21:51:00 INFO mapred.JobClient:  map 100% reduce 100%
10/06/19 21:51:02 INFO mapred.JobClient: Job complete: job_201006192101_0003
10/06/19 21:51:02 INFO mapred.JobClient: Counters: 18
10/06/19 21:51:02 INFO mapred.JobClient:   Job Counters 
10/06/19 21:51:02 INFO mapred.JobClient:     Launched reduce tasks=1
10/06/19 21:51:02 INFO mapred.JobClient:     Launched map tasks=4
10/06/19 21:51:02 INFO mapred.JobClient:     Data-local map tasks=4
10/06/19 21:51:02 INFO mapred.JobClient:   FileSystemCounters
10/06/19 21:51:02 INFO mapred.JobClient:     FILE_BYTES_READ=94
10/06/19 21:51:02 INFO mapred.JobClient:     HDFS_BYTES_READ=472
10/06/19 21:51:02 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=334
10/06/19 21:51:02 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=215
10/06/19 21:51:02 INFO mapred.JobClient:   Map-Reduce Framework
10/06/19 21:51:02 INFO mapred.JobClient:     Reduce input groups=8
10/06/19 21:51:02 INFO mapred.JobClient:     Combine output records=0
10/06/19 21:51:02 INFO mapred.JobClient:     Map input records=4
10/06/19 21:51:02 INFO mapred.JobClient:     Reduce shuffle bytes=112
10/06/19 21:51:02 INFO mapred.JobClient:     Reduce output records=0
10/06/19 21:51:02 INFO mapred.JobClient:     Spilled Records=16
10/06/19 21:51:02 INFO mapred.JobClient:     Map output bytes=72
10/06/19 21:51:02 INFO mapred.JobClient:     Map input bytes=96
10/06/19 21:51:02 INFO mapred.JobClient:     Combine input records=0
10/06/19 21:51:02 INFO mapred.JobClient:     Map output records=8
10/06/19 21:51:02 INFO mapred.JobClient:     Reduce input records=8
Job Finished in 28.593 seconds
java.lang.ArithmeticException: Non-terminating decimal expansion; no exact representable decimal result.
	at java.math.BigDecimal.divide(BigDecimal.java:1603)
	at org.apache.hadoop.examples.PiEstimator.estimate(PiEstimator.java:313)
	at org.apache.hadoop.examples.PiEstimator.run(PiEstimator.java:342)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
	at org.apache.hadoop.examples.PiEstimator.main(PiEstimator.java:351)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
	at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
	at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:156)


"hadoop jar hadoop-0.20.1+169.89-examples.jar pi 2 10" finishes fine

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9167-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 20 08:51:53 2010
Return-Path: <common-issues-return-9167-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 71634 invoked from network); 20 Jun 2010 08:51:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jun 2010 08:51:53 -0000
Received: (qmail 23750 invoked by uid 500); 20 Jun 2010 08:51:53 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 23605 invoked by uid 500); 20 Jun 2010 08:51:50 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 23596 invoked by uid 99); 20 Jun 2010 08:51:49 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 Jun 2010 08:51:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 Jun 2010 08:51:46 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5K8pO0i012317
	for <common-issues@hadoop.apache.org>; Sun, 20 Jun 2010 08:51:25 GMT
Message-ID: <23349841.99341277023884790.JavaMail.jira@thor>
Date: Sun, 20 Jun 2010 04:51:24 -0400 (EDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6826) Revert FileSystem create method
 that takes CreateFlags
In-Reply-To: <6937853.16451276636883446.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880604#action_12880604 ] 

dhruba borthakur commented on HADOOP-6826:
------------------------------------------

This revert-patch looks good to me, +1

> Revert FileSystem create method that takes CreateFlags
> ------------------------------------------------------
>
>                 Key: HADOOP-6826
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6826
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6826.patch
>
>
> As discussed in HDFS-609 and HADOOP-5438 we should back out the FileSystem create() method that takes a set of CreateFlag objects, until the interface has been agreed upon and fully tested.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9168-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 20 18:18:55 2010
Return-Path: <common-issues-return-9168-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90637 invoked from network); 20 Jun 2010 18:18:54 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 20 Jun 2010 18:18:54 -0000
Received: (qmail 68342 invoked by uid 500); 20 Jun 2010 18:18:54 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68208 invoked by uid 500); 20 Jun 2010 18:18:53 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68200 invoked by uid 99); 20 Jun 2010 18:18:53 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 Jun 2010 18:18:53 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 Jun 2010 18:18:51 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5KIIT32019705
	for <common-issues@hadoop.apache.org>; Sun, 20 Jun 2010 18:18:29 GMT
Message-ID: <9518527.102301277057909607.JavaMail.jira@thor>
Date: Sun, 20 Jun 2010 14:18:29 -0400 (EDT)
From: "Konstantin Boudnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6255) Create an rpm integration project
In-Reply-To: <1679708275.1252711317657.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880656#action_12880656 ] 

Konstantin Boudnik commented on HADOOP-6255:
--------------------------------------------

bq. a) the stuff in sbin should actually be in libexec, along with hadoop-config.sh

Perhaps for the current {{sbin/}} stuff it a better place. But why {{hadoop-config.sh}} I can see the latter to be in {{sbin/}} though

bq. b) hdfs, mapred, and hadoop-dameon.sh (assuming those are all admin tools) should be in sbin
In a sense I agree -  they are admin tools therefore need to be in {{sbin}}. And also they aren't likely to be invoked directly but rather through distro specific start/stop facilities e.g. {{init.d}} or {{service}}. So, perhaps they are indeed belong to {{sbin/}}

bq. c) why include/c++ ? shouldn't it just be include?
Agree, I guess c++ was an example of the stuff me need to put into {{include/}}


> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Owen O'Malley
>
> We should be able to create RPMs for Hadoop releases.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9169-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 21 03:27:49 2010
Return-Path: <common-issues-return-9169-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96540 invoked from network); 21 Jun 2010 03:27:49 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jun 2010 03:27:49 -0000
Received: (qmail 45866 invoked by uid 500); 21 Jun 2010 03:27:49 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 45742 invoked by uid 500); 21 Jun 2010 03:27:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 45734 invoked by uid 99); 21 Jun 2010 03:27:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Jun 2010 03:27:45 +0000
X-ASF-Spam-Status: No, hits=-1533.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Jun 2010 03:27:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5L3ROpk022079
	for <common-issues@hadoop.apache.org>; Mon, 21 Jun 2010 03:27:24 GMT
Message-ID: <6136505.104741277090844405.JavaMail.jira@thor>
Date: Sun, 20 Jun 2010 23:27:24 -0400 (EDT)
From: "Amareshwari Sriramadasu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6748) Remove hadoop.cluster.administrators
In-Reply-To: <3308520.13571273038484086.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amareshwari Sriramadasu updated HADOOP-6748:
--------------------------------------------

    Release Note: 
Removed configuration property "hadoop.cluster.administrators". Added constructor public HttpServer(String name, String bindAddress, int port, boolean findPort, Configuration conf, AccessControlList adminsAcl) in HttpServer, which takes cluster administrators acl as a parameter. 


> Remove hadoop.cluster.administrators
> ------------------------------------
>
>                 Key: HADOOP-6748
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6748
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 0.21.0
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.21.0
>
>         Attachments: patch-6748-0.21.txt, patch-6748-1.txt, patch-6748.txt
>
>
> Remove hadoop.cluster.administrators in favor of having separate configuration property name for MapReduce and HDFS projects.
> See more details on MAPREDUCE-1542.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9170-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 21 09:32:07 2010
Return-Path: <common-issues-return-9170-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96167 invoked from network); 21 Jun 2010 09:32:07 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jun 2010 09:32:07 -0000
Received: (qmail 9639 invoked by uid 500); 21 Jun 2010 09:32:05 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8536 invoked by uid 500); 21 Jun 2010 09:32:01 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8499 invoked by uid 99); 21 Jun 2010 09:32:00 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Jun 2010 09:32:00 +0000
X-ASF-Spam-Status: No, hits=0.9 required=10.0
	tests=ASF_LIST_OPS,HTML_MESSAGE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [216.32.180.31] (HELO VA3EHSOBE005.bigfish.com) (216.32.180.31)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Jun 2010 09:31:52 +0000
Received: from mail80-va3-R.bigfish.com (10.7.14.245) by
 VA3EHSOBE005.bigfish.com (10.7.40.25) with Microsoft SMTP Server id
 8.1.340.0; Mon, 21 Jun 2010 09:31:29 +0000
Received: from mail80-va3 (localhost.localdomain [127.0.0.1])	by
 mail80-va3-R.bigfish.com (Postfix) with ESMTP id 96568A0170F;	Mon, 21 Jun
 2010 09:31:29 +0000 (UTC)
X-SpamScore: -7
X-BigFish: VPS-7(zf2bjz9251Ka0dJ936eMzz1202hz31izz2dh61h)
X-Spam-TCS-SCL: 0:0
Received: from mail80-va3 (localhost.localdomain [127.0.0.1]) by mail80-va3
 (MessageSwitch) id 1277112688970156_14483; Mon, 21 Jun 2010 09:31:28 +0000
 (UTC)
Received: from VA3EHSMHS012.bigfish.com (unknown [10.7.14.245])	by
 mail80-va3.bigfish.com (Postfix) with ESMTP id E3209155004E;	Mon, 21 Jun 2010
 09:31:28 +0000 (UTC)
Received: from herald.cybage.com (121.242.40.50) by VA3EHSMHS012.bigfish.com
 (10.7.99.22) with Microsoft SMTP Server id 14.0.482.44; Mon, 21 Jun 2010
 09:31:26 +0000
Received: from wa-beexch-1-1.cybage.com ([192.168.5.207]) by herald.cybage.com
 with Microsoft SMTPSVC(6.0.3790.4675);	 Mon, 21 Jun 2010 15:00:40 +0530
X-MimeOLE: Produced By Microsoft Exchange V6.5
Content-Class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01CB1124.697FF876"
Subject: [JIRA] (COLL-117) seeing warnings "attempt to override final parameter: dfs.data.dir" in tasktracker logs
Date: Mon, 21 Jun 2010 14:56:54 +0530
Message-ID: <8D269469729C0848B728757E2CCF04E7032C5A0A@wa-beexch-1-1.cybage.com>
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
Thread-Topic: [JIRA] (COLL-117) seeing warnings "attempt to override final parameter: dfs.data.dir" in tasktracker logs
thread-index: AcsRBe1t1cjOaJSoRMOjRNzgevGCPQAHfYE5
References: <992816359.57375.1277099531336.JavaMail.jira@ent-jira-1.ic>
From: Bikash Singhal <bikashs@cybage.com>
To: "common-dev@hadoop.apache.org" <'common-dev@hadoop.apache.org'>,
	"common-user-subscribe@hadoop.apache.org" <'common-user-subscribe@hadoop.apache.org'>,
	"common-user@hadoop.apache.org" <'common-user@hadoop.apache.org'>,
	"common-issues@hadoop.apache.org" <'common-issues@hadoop.apache.org'>
CC: "Tina Malpure" <tinam@cybage.com>
X-OriginalArrivalTime: 21 Jun 2010 09:30:40.0119 (UTC) FILETIME=[69C4C870:01CB1124]
X-Reverse-DNS: 121.242.40.50.static-pune.vsnl.net.in
X-Virus-Checked: Checked by ClamAV on apache.org

------_=_NextPart_001_01CB1124.697FF876
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable


Hi Hadoopers,

I have received WARN in the hadoop cluster. Has anybody seen this . Any sol=
ution?


> 2010-06-06 01:45:04,079 WARN org.apache.hadoop.conf.Configuration: /var/l=
ib/hadoop-0.20/cache/hadoop/mapred/local/taskTracker/jobcache/job_201006060=
025_0003/job.xml:a attempt to override final parameter: dfs.data.dir;  Igno=
ring.
> ....
> 2010-06-06 01:49:41,501 WARN org.apache.hadoop.conf.Configuration: /var/l=
ib/hadoop-0.20/cache/hadoop/mapred/local/taskTracker/jobcache/job_201006060=
025_0004/job.xml:a attempt to override final parameter: dfs.data.dir;  Igno=
ring.


"Legal Disclaimer: This electronic message and all contents contain informa=
tion from Cybage Software Private Limited which may be privileged, confiden=
tial, or otherwise protected from disclosure. The information is intended t=
o be for the addressee(s) only. If you are not an addressee, any disclosure=
, copy, distribution, or use of the contents of this message is strictly pr=
ohibited. If you have received this electronic message in error please noti=
fy the sender by reply e-mail to and destroy the original message and all c=
opies. Cybage has taken every reasonable precaution to minimize the risk of=
 malicious content in the mail, but is not liable for any damage you may su=
stain as a result of any malicious content in this e-mail. You should carry=
 out your own malicious content checks before opening the e-mail or attachm=
ent." =

www.cybage.com


------_=_NextPart_001_01CB1124.697FF876--

From common-issues-return-9171-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 21 17:23:46 2010
Return-Path: <common-issues-return-9171-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25192 invoked from network); 21 Jun 2010 17:23:45 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jun 2010 17:23:45 -0000
Received: (qmail 74171 invoked by uid 500); 21 Jun 2010 17:23:45 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 74063 invoked by uid 500); 21 Jun 2010 17:23:44 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 74055 invoked by uid 99); 21 Jun 2010 17:23:44 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Jun 2010 17:23:44 +0000
X-ASF-Spam-Status: No, hits=-1534.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Jun 2010 17:23:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5LHNNRd007711
	for <common-issues@hadoop.apache.org>; Mon, 21 Jun 2010 17:23:23 GMT
Message-ID: <19001973.112581277141003401.JavaMail.jira@thor>
Date: Mon, 21 Jun 2010 13:23:23 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6832) Provide a web server plugin that uses
 a static user for the web UI
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Provide a web server plugin that uses a static user for the web UI
------------------------------------------------------------------

                 Key: HADOOP-6832
                 URL: https://issues.apache.org/jira/browse/HADOOP-6832
             Project: Hadoop Common
          Issue Type: New Feature
          Components: security
            Reporter: Owen O'Malley
            Assignee: Owen O'Malley
             Fix For: 0.22.0


We need a simple plugin that uses a static user for clusters with security that don't want to authenticate users on the web UI.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9172-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 21 17:25:47 2010
Return-Path: <common-issues-return-9172-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 25800 invoked from network); 21 Jun 2010 17:25:47 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jun 2010 17:25:47 -0000
Received: (qmail 76287 invoked by uid 500); 21 Jun 2010 17:25:47 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76256 invoked by uid 500); 21 Jun 2010 17:25:46 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76248 invoked by uid 99); 21 Jun 2010 17:25:46 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Jun 2010 17:25:46 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Jun 2010 17:25:44 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5LHPMKY007758
	for <common-issues@hadoop.apache.org>; Mon, 21 Jun 2010 17:25:23 GMT
Message-ID: <25335346.112661277141122846.JavaMail.jira@thor>
Date: Mon, 21 Jun 2010 13:25:22 -0400 (EDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6832) Provide a web server plugin that uses
 a static user for the web UI
In-Reply-To: <19001973.112581277141003401.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-6832:
----------------------------------

    Attachment: static-web-user.patch

> Provide a web server plugin that uses a static user for the web UI
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6832
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6832
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: security
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>             Fix For: 0.22.0
>
>         Attachments: static-web-user.patch
>
>
> We need a simple plugin that uses a static user for clusters with security that don't want to authenticate users on the web UI.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9173-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 21 18:42:01 2010
Return-Path: <common-issues-return-9173-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 56645 invoked from network); 21 Jun 2010 18:42:01 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 21 Jun 2010 18:42:01 -0000
Received: (qmail 97581 invoked by uid 500); 21 Jun 2010 18:42:01 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 97528 invoked by uid 500); 21 Jun 2010 18:42:00 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 97520 invoked by uid 99); 21 Jun 2010 18:42:00 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Jun 2010 18:42:00 +0000
X-ASF-Spam-Status: No, hits=-1535.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Jun 2010 18:42:00 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5LIfdvW010484
	for <common-issues@hadoop.apache.org>; Mon, 21 Jun 2010 18:41:39 GMT
Message-ID: <20068946.114471277145699569.JavaMail.jira@thor>
Date: Mon, 21 Jun 2010 14:41:39 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6255) Create an rpm integration project
In-Reply-To: <1679708275.1252711317657.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880939#action_12880939 ] 

Allen Wittenauer commented on HADOOP-6255:
------------------------------------------

I might be wrong, but I'm pretty certain that hadoop-config.sh isn't meant to be directly user-runnable.  It falls into the same category as the taskController.


> Create an rpm integration project
> ---------------------------------
>
>                 Key: HADOOP-6255
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6255
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Owen O'Malley
>
> We should be able to create RPMs for Hadoop releases.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9174-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 02:41:23 2010
Return-Path: <common-issues-return-9174-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41588 invoked from network); 22 Jun 2010 02:41:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 02:41:23 -0000
Received: (qmail 18580 invoked by uid 500); 22 Jun 2010 02:41:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18450 invoked by uid 500); 22 Jun 2010 02:41:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18421 invoked by uid 99); 22 Jun 2010 02:41:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 02:41:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 02:41:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5M2euKZ009497
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 02:40:56 GMT
Message-ID: <31342678.3821277174456005.JavaMail.jira@thor>
Date: Mon, 21 Jun 2010 22:40:56 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6833) IPC leaks call parameters when
 exceptions thrown
In-Reply-To: <21655383.3791277174455275.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6833:
--------------------------------

    Attachment: hadoop-6833.txt

> IPC leaks call parameters when exceptions thrown
> ------------------------------------------------
>
>                 Key: HADOOP-6833
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6833
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2, 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Blocker
>         Attachments: hadoop-6833.txt
>
>
> HADOOP-6498 moved the calls.remove() call lower into the SUCCESS clause of receiveResponse(), but didn't put a similar calls.remove into the ERROR clause. So, any RPC call that throws an exception ends up orphaning the Call object in the connection's "calls" hashtable. This prevents cleanup of the connection and is a memory leak for the call parameters.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9175-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 02:41:25 2010
Return-Path: <common-issues-return-9175-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41640 invoked from network); 22 Jun 2010 02:41:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 02:41:24 -0000
Received: (qmail 19407 invoked by uid 500); 22 Jun 2010 02:41:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18454 invoked by uid 500); 22 Jun 2010 02:41:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18420 invoked by uid 99); 22 Jun 2010 02:41:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 02:41:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 02:41:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5M2euwc009503
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 02:40:56 GMT
Message-ID: <23020643.3841277174456265.JavaMail.jira@thor>
Date: Mon, 21 Jun 2010 22:40:56 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6833) IPC leaks call parameters when
 exceptions thrown
In-Reply-To: <21655383.3791277174455275.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6833:
--------------------------------

    Status: Patch Available  (was: Open)

> IPC leaks call parameters when exceptions thrown
> ------------------------------------------------
>
>                 Key: HADOOP-6833
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6833
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2, 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Blocker
>         Attachments: hadoop-6833.txt
>
>
> HADOOP-6498 moved the calls.remove() call lower into the SUCCESS clause of receiveResponse(), but didn't put a similar calls.remove into the ERROR clause. So, any RPC call that throws an exception ends up orphaning the Call object in the connection's "calls" hashtable. This prevents cleanup of the connection and is a memory leak for the call parameters.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9176-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 02:41:25 2010
Return-Path: <common-issues-return-9176-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41669 invoked from network); 22 Jun 2010 02:41:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 02:41:24 -0000
Received: (qmail 19451 invoked by uid 500); 22 Jun 2010 02:41:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 19362 invoked by uid 500); 22 Jun 2010 02:41:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 18419 invoked by uid 99); 22 Jun 2010 02:41:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 02:41:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 02:41:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5M2etOR009487
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 02:40:55 GMT
Message-ID: <21655383.3791277174455275.JavaMail.jira@thor>
Date: Mon, 21 Jun 2010 22:40:55 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6833) IPC leaks call parameters when
 exceptions thrown
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

IPC leaks call parameters when exceptions thrown
------------------------------------------------

                 Key: HADOOP-6833
                 URL: https://issues.apache.org/jira/browse/HADOOP-6833
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 0.20.2, 0.21.0, 0.22.0
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
            Priority: Blocker
         Attachments: hadoop-6833.txt

HADOOP-6498 moved the calls.remove() call lower into the SUCCESS clause of receiveResponse(), but didn't put a similar calls.remove into the ERROR clause. So, any RPC call that throws an exception ends up orphaning the Call object in the connection's "calls" hashtable. This prevents cleanup of the connection and is a memory leak for the call parameters.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9177-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 03:12:22 2010
Return-Path: <common-issues-return-9177-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54059 invoked from network); 22 Jun 2010 03:12:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 03:12:22 -0000
Received: (qmail 39259 invoked by uid 500); 22 Jun 2010 03:12:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 38696 invoked by uid 500); 22 Jun 2010 03:12:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 38686 invoked by uid 99); 22 Jun 2010 03:12:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 03:12:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 03:12:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5M3Bswi009850
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 03:11:55 GMT
Message-ID: <13442550.4351277176314963.JavaMail.jira@thor>
Date: Mon, 21 Jun 2010 23:11:54 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6833) IPC leaks call parameters when
 exceptions thrown
In-Reply-To: <21655383.3791277174455275.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881035#action_12881035 ] 

Hadoop QA commented on HADOOP-6833:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12447656/hadoop-6833.txt
  against trunk revision 956710.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    -1 javac.  The applied patch generated 1017 javac compiler warnings (more than the trunk's current 1013 warnings).

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/590/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/590/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/590/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/590/console

This message is automatically generated.

> IPC leaks call parameters when exceptions thrown
> ------------------------------------------------
>
>                 Key: HADOOP-6833
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6833
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2, 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Blocker
>         Attachments: hadoop-6833.txt
>
>
> HADOOP-6498 moved the calls.remove() call lower into the SUCCESS clause of receiveResponse(), but didn't put a similar calls.remove into the ERROR clause. So, any RPC call that throws an exception ends up orphaning the Call object in the connection's "calls" hashtable. This prevents cleanup of the connection and is a memory leak for the call parameters.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9178-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 03:21:19 2010
Return-Path: <common-issues-return-9178-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 55100 invoked from network); 22 Jun 2010 03:21:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 03:21:19 -0000
Received: (qmail 40960 invoked by uid 500); 22 Jun 2010 03:21:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 40833 invoked by uid 500); 22 Jun 2010 03:21:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 40825 invoked by uid 99); 22 Jun 2010 03:21:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 03:21:16 +0000
X-ASF-Spam-Status: No, hits=-1535.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 03:21:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5M3Kt23009897
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 03:20:55 GMT
Message-ID: <13390842.4431277176855486.JavaMail.jira@thor>
Date: Mon, 21 Jun 2010 23:20:55 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6833) IPC leaks call parameters when
 exceptions thrown
In-Reply-To: <21655383.3791277174455275.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881038#action_12881038 ] 

Todd Lipcon commented on HADOOP-6833:
-------------------------------------

I think the "new warnings" are because I used UTF8 in the test, and UTF8 is deprecated. I used it to follow the pattern elsewhere in the test. Let me know if I should change the test case to use something like Text or LongWritable instead.

> IPC leaks call parameters when exceptions thrown
> ------------------------------------------------
>
>                 Key: HADOOP-6833
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6833
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2, 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Blocker
>         Attachments: hadoop-6833.txt
>
>
> HADOOP-6498 moved the calls.remove() call lower into the SUCCESS clause of receiveResponse(), but didn't put a similar calls.remove into the ERROR clause. So, any RPC call that throws an exception ends up orphaning the Call object in the connection's "calls" hashtable. This prevents cleanup of the connection and is a memory leak for the call parameters.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9179-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 04:16:24 2010
Return-Path: <common-issues-return-9179-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65511 invoked from network); 22 Jun 2010 04:16:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 04:16:24 -0000
Received: (qmail 66137 invoked by uid 500); 22 Jun 2010 04:16:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65955 invoked by uid 500); 22 Jun 2010 04:16:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64896 invoked by uid 99); 22 Jun 2010 04:16:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 04:16:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 04:16:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5M4FuQr010829
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 04:15:56 GMT
Message-ID: <24319957.5221277180156667.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 00:15:56 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6826) Revert FileSystem create method that
 takes CreateFlags
In-Reply-To: <6937853.16451276636883446.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6826:
------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

I've committed this.

> Revert FileSystem create method that takes CreateFlags
> ------------------------------------------------------
>
>                 Key: HADOOP-6826
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6826
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6826.patch
>
>
> As discussed in HDFS-609 and HADOOP-5438 we should back out the FileSystem create() method that takes a set of CreateFlag objects, until the interface has been agreed upon and fully tested.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9180-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 04:31:27 2010
Return-Path: <common-issues-return-9180-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66930 invoked from network); 22 Jun 2010 04:31:27 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 04:31:27 -0000
Received: (qmail 73006 invoked by uid 500); 22 Jun 2010 04:31:27 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72884 invoked by uid 500); 22 Jun 2010 04:31:25 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72866 invoked by uid 99); 22 Jun 2010 04:31:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 04:31:24 +0000
X-ASF-Spam-Status: No, hits=-1535.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 04:31:23 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5M4V3Gf011086
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 04:31:03 GMT
Message-ID: <22017316.5901277181063702.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 00:31:03 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6342) Create a script to squash a common,
 hdfs, and mapreduce tarball into a single hadoop tarball
In-Reply-To: <1136640537.1256834099525.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881059#action_12881059 ] 

Tom White commented on HADOOP-6342:
-----------------------------------

> This should probably only be committed to 0.21 as a compatibility hack.

Good idea - I'll commit this to the 0.21 branch in src (where there are a couple of other dev scripts).

> Create a script to squash a common, hdfs, and mapreduce tarball into a single hadoop tarball
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6342
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6342
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>            Priority: Minor
>             Fix For: 0.21.0
>
>         Attachments: h-6342.patch, HADOOP-6342.2.patch, HADOOP-6342.patch, tar-munge, tar-munge
>
>
> It would be convenient for the transition if we had a script to take a set of common, hdfs, and mapreduce tarballs and merge them into a single tarball. This is intended just to help users who don't want to transition to split projects for deployment immediately.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9181-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 04:31:28 2010
Return-Path: <common-issues-return-9181-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 66974 invoked from network); 22 Jun 2010 04:31:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 04:31:28 -0000
Received: (qmail 73097 invoked by uid 500); 22 Jun 2010 04:31:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73047 invoked by uid 500); 22 Jun 2010 04:31:27 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73024 invoked by uid 99); 22 Jun 2010 04:31:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 04:31:27 +0000
X-ASF-Spam-Status: No, hits=-1535.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 04:31:26 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5M4V6FG011140
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 04:31:06 GMT
Message-ID: <11278863.6081277181066608.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 00:31:06 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6342) Create a script to squash a common,
 hdfs, and mapreduce tarball into a single hadoop tarball
In-Reply-To: <1136640537.1256834099525.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6342:
------------------------------

    Fix Version/s: 0.21.0
         Priority: Blocker  (was: Minor)

> Create a script to squash a common, hdfs, and mapreduce tarball into a single hadoop tarball
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6342
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6342
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>            Reporter: Owen O'Malley
>            Assignee: Owen O'Malley
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: h-6342.patch, HADOOP-6342.2.patch, HADOOP-6342.patch, tar-munge, tar-munge
>
>
> It would be convenient for the transition if we had a script to take a set of common, hdfs, and mapreduce tarballs and merge them into a single tarball. This is intended just to help users who don't want to transition to split projects for deployment immediately.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9182-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 05:00:23 2010
Return-Path: <common-issues-return-9182-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 70117 invoked from network); 22 Jun 2010 05:00:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 05:00:23 -0000
Received: (qmail 89488 invoked by uid 500); 22 Jun 2010 05:00:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88917 invoked by uid 500); 22 Jun 2010 05:00:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88909 invoked by uid 99); 22 Jun 2010 05:00:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 05:00:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 05:00:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5M4xt3N011330
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 04:59:55 GMT
Message-ID: <3010147.6311277182795651.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 00:59:55 -0400 (EDT)
From: "Ravi Gummadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6826) Revert FileSystem create method
 that takes CreateFlags
In-Reply-To: <6937853.16451276636883446.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881065#action_12881065 ] 

Ravi Gummadi commented on HADOOP-6826:
--------------------------------------

This broke compilation of mapreduce trunk.

> Revert FileSystem create method that takes CreateFlags
> ------------------------------------------------------
>
>                 Key: HADOOP-6826
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6826
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6826.patch
>
>
> As discussed in HDFS-609 and HADOOP-5438 we should back out the FileSystem create() method that takes a set of CreateFlag objects, until the interface has been agreed upon and fully tested.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9183-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 05:08:20 2010
Return-Path: <common-issues-return-9183-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75261 invoked from network); 22 Jun 2010 05:08:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 05:08:20 -0000
Received: (qmail 95957 invoked by uid 500); 22 Jun 2010 05:08:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95823 invoked by uid 500); 22 Jun 2010 05:08:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95809 invoked by uid 99); 22 Jun 2010 05:08:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 05:08:16 +0000
X-ASF-Spam-Status: No, hits=-1535.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 05:08:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5M57tqe011520
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 05:07:55 GMT
Message-ID: <580801.6671277183275711.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 01:07:55 -0400 (EDT)
From: "Ravi Gummadi (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6826) Revert FileSystem create method
 that takes CreateFlags
In-Reply-To: <6937853.16451276636883446.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881067#action_12881067 ] 

Ravi Gummadi commented on HADOOP-6826:
--------------------------------------

Raised MAPREDUCE-1885 in mapreduce project to fix the compilation issue because of this FileSystem api change.

> Revert FileSystem create method that takes CreateFlags
> ------------------------------------------------------
>
>                 Key: HADOOP-6826
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6826
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6826.patch
>
>
> As discussed in HDFS-609 and HADOOP-5438 we should back out the FileSystem create() method that takes a set of CreateFlag objects, until the interface has been agreed upon and fully tested.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9184-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 05:09:19 2010
Return-Path: <common-issues-return-9184-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75570 invoked from network); 22 Jun 2010 05:09:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 05:09:18 -0000
Received: (qmail 98278 invoked by uid 500); 22 Jun 2010 05:09:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98122 invoked by uid 500); 22 Jun 2010 05:09:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98091 invoked by uid 99); 22 Jun 2010 05:09:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 05:09:15 +0000
X-ASF-Spam-Status: No, hits=-2.7 required=10.0
	tests=ASF_LIST_OPS,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [216.32.180.11] (HELO VA3EHSOBE001.bigfish.com) (216.32.180.11)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 05:09:08 +0000
Received: from mail58-va3-R.bigfish.com (10.7.14.254) by
 VA3EHSOBE001.bigfish.com (10.7.40.21) with Microsoft SMTP Server id
 8.1.340.0; Tue, 22 Jun 2010 05:08:46 +0000
Received: from mail58-va3 (localhost.localdomain [127.0.0.1])	by
 mail58-va3-R.bigfish.com (Postfix) with ESMTP id 00A50410388;	Tue, 22 Jun
 2010 05:08:45 +0000 (UTC)
X-SpamScore: 5
X-BigFish: VPS5(zz9251Ka0dJzz1202hzzz2dh182k68o)
X-Spam-TCS-SCL: 7:0
Received: from mail58-va3 (localhost.localdomain [127.0.0.1]) by mail58-va3
 (MessageSwitch) id 1277183325582433_14823; Tue, 22 Jun 2010 05:08:45 +0000
 (UTC)
Received: from VA3EHSMHS024.bigfish.com (unknown [10.7.14.245])	by
 mail58-va3.bigfish.com (Postfix) with ESMTP id 7439EA3004E;	Tue, 22 Jun 2010
 05:08:45 +0000 (UTC)
Received: from herald.cybage.com (121.242.40.50) by VA3EHSMHS024.bigfish.com
 (10.7.99.34) with Microsoft SMTP Server id 14.0.482.44; Tue, 22 Jun 2010
 05:08:42 +0000
Received: from [172.28.26.81] ([172.28.26.81] RDNS failed) by
 herald.cybage.com over TLS secured channel with Microsoft
 SMTPSVC(6.0.3790.4675);	 Tue, 22 Jun 2010 10:38:43 +0530
Message-ID: <4C204565.4080701@cybage.com>
Date: Tue, 22 Jun 2010 10:38:53 +0530
From: sonali <sonalig@cybage.com>
User-Agent: Thunderbird 2.0.0.24 (X11/20100411)
MIME-Version: 1.0
To: common-issues@hadoop.apache.org, common-user@hadoop.apache.org, 
 common-user-subscribe@hadoop.apache.org, common-dev@hadoop.apache.org
Subject: Need subscription
Content-Type: text/plain; charset="ISO-8859-1"; format=flowed
Content-Transfer-Encoding: 7bit
X-OriginalArrivalTime: 22 Jun 2010 05:08:43.0746 (UTC) FILETIME=[FC7E5420:01CB11C8]
X-Reverse-DNS: 121.242.40.50.static-pune.vsnl.net.in
X-Virus-Checked: Checked by ClamAV on apache.org



"Legal Disclaimer: This electronic message and all contents contain information from Cybage Software Private Limited which may be privileged, confidential, or otherwise protected from disclosure. The information is intended to be for the addressee(s) only. If you are not an addressee, any disclosure, copy, distribution, or use of the contents of this message is strictly prohibited. If you have received this electronic message in error please notify the sender by reply e-mail to and destroy the original message and all copies. Cybage has taken every reasonable precaution to minimize the risk of malicious content in the mail, but is not liable for any damage you may sustain as a result of any malicious content in this e-mail. You should carry out your own malicious content checks before opening the e-mail or attachment." 
www.cybage.com



From common-issues-return-9185-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 05:12:23 2010
Return-Path: <common-issues-return-9185-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 76007 invoked from network); 22 Jun 2010 05:12:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 05:12:23 -0000
Received: (qmail 5290 invoked by uid 500); 22 Jun 2010 05:12:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 5186 invoked by uid 500); 22 Jun 2010 05:12:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 5176 invoked by uid 99); 22 Jun 2010 05:12:20 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 05:12:20 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 05:12:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5M5Buw4011565
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 05:11:56 GMT
Message-ID: <19319879.6781277183516629.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 01:11:56 -0400 (EDT)
From: "Amar Kamat (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6826) Revert FileSystem create method
 that takes CreateFlags
In-Reply-To: <6937853.16451276636883446.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881069#action_12881069 ] 

Amar Kamat commented on HADOOP-6826:
------------------------------------

Tom/Dhruba,
I think this commit should be reverted. FileSystem.create() is a public api and hence cannot be removed like this. Ideally it should be deprecated. Do you agree?

> Revert FileSystem create method that takes CreateFlags
> ------------------------------------------------------
>
>                 Key: HADOOP-6826
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6826
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6826.patch
>
>
> As discussed in HDFS-609 and HADOOP-5438 we should back out the FileSystem create() method that takes a set of CreateFlag objects, until the interface has been agreed upon and fully tested.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9186-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 05:44:19 2010
Return-Path: <common-issues-return-9186-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 81027 invoked from network); 22 Jun 2010 05:44:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 05:44:19 -0000
Received: (qmail 28681 invoked by uid 500); 22 Jun 2010 05:44:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28549 invoked by uid 500); 22 Jun 2010 05:44:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28541 invoked by uid 99); 22 Jun 2010 05:44:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 05:44:16 +0000
X-ASF-Spam-Status: No, hits=-1535.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 05:44:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5M5ht0B011780
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 05:43:55 GMT
Message-ID: <9644238.7051277185435056.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 01:43:55 -0400 (EDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6826) Revert FileSystem create method
 that takes CreateFlags
In-Reply-To: <6937853.16451276636883446.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881077#action_12881077 ] 

dhruba borthakur commented on HADOOP-6826:
------------------------------------------

hi amar, this API never made it to any "release" of Hadoop. Better to remove it now than to ship a release with a bad API.

> Revert FileSystem create method that takes CreateFlags
> ------------------------------------------------------
>
>                 Key: HADOOP-6826
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6826
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6826.patch
>
>
> As discussed in HDFS-609 and HADOOP-5438 we should back out the FileSystem create() method that takes a set of CreateFlag objects, until the interface has been agreed upon and fully tested.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9187-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 05:49:22 2010
Return-Path: <common-issues-return-9187-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82833 invoked from network); 22 Jun 2010 05:49:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 05:49:22 -0000
Received: (qmail 32300 invoked by uid 500); 22 Jun 2010 05:49:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 31800 invoked by uid 500); 22 Jun 2010 05:49:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 31788 invoked by uid 99); 22 Jun 2010 05:49:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 05:49:18 +0000
X-ASF-Spam-Status: No, hits=-1535.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 05:49:18 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5M5mv0C011829
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 05:48:57 GMT
Message-ID: <12796664.7141277185737685.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 01:48:57 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6826) Revert FileSystem create method
 that takes CreateFlags
In-Reply-To: <6937853.16451276636883446.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881080#action_12881080 ] 

Tom White commented on HADOOP-6826:
-----------------------------------

Ravi, I missed that this affected MapReduce: sorry about that! Thanks for opening an issue, and creating a patch to fix MapReduce trunk.

Amar, This reverts an unreleased API, so we don't need to deprecate it. In fact, the API was broken in some cases (HDFS-609) so we shouldn't release it.

> Revert FileSystem create method that takes CreateFlags
> ------------------------------------------------------
>
>                 Key: HADOOP-6826
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6826
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6826.patch
>
>
> As discussed in HDFS-609 and HADOOP-5438 we should back out the FileSystem create() method that takes a set of CreateFlag objects, until the interface has been agreed upon and fully tested.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9188-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 07:25:24 2010
Return-Path: <common-issues-return-9188-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 12100 invoked from network); 22 Jun 2010 07:25:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 07:25:24 -0000
Received: (qmail 7117 invoked by uid 500); 22 Jun 2010 07:25:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 6905 invoked by uid 500); 22 Jun 2010 07:25:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 6871 invoked by uid 99); 22 Jun 2010 07:25:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 07:25:17 +0000
X-ASF-Spam-Status: No, hits=-1536.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 07:25:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5M7Ourn012900
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 07:24:56 GMT
Message-ID: <10707501.8491277191496663.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 03:24:56 -0400 (EDT)
From: "Ahad Rana (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6834) TFile.append compares initial key
 against null lastKey
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

TFile.append compares initial key against null lastKey  
--------------------------------------------------------

                 Key: HADOOP-6834
                 URL: https://issues.apache.org/jira/browse/HADOOP-6834
             Project: Hadoop Common
          Issue Type: Bug
          Components: io
    Affects Versions: 0.20.2, 0.20.1
            Reporter: Ahad Rana


The following code in TFile.KeyReigster.close: 


            byte[] lastKey = lastKeyBufferOS.getBuffer();
            int lastLen = lastKeyBufferOS.size();
            if (tfileMeta.getComparator().compare(key, 0, len, lastKey, 0,
                lastLen) < 0) {
              throw new IOException("Keys are not added in sorted order");
            }

compares the initial  key (passed in via  TFile.Writer.append) against a technically NULL lastKey. lastKey is not initialized until after the first call to TFile.Writer.append. The underlying RawComparator interface used for comparisons does not stipulate the proper behavior when either length 1  or length 2 is zero. In the case of LongWritable, its WritableComparator implementation does an unsafe read on the passed in byte arrays b1 and b2. Since TFile pre-allocates the buffer used for storing lastKey, this passes a valid buffer with zero count to LongWritable's comparator, which ignores length and thus produces incorrect results. 


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9189-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 11:20:28 2010
Return-Path: <common-issues-return-9189-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 77653 invoked from network); 22 Jun 2010 11:20:28 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 11:20:28 -0000
Received: (qmail 65234 invoked by uid 500); 22 Jun 2010 11:20:28 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64662 invoked by uid 500); 22 Jun 2010 11:20:25 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64452 invoked by uid 99); 22 Jun 2010 11:20:24 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 11:20:24 +0000
X-ASF-Spam-Status: No, hits=-1536.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 11:20:23 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5MBK2ba015420
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 11:20:03 GMT
Message-ID: <509473.11561277205602928.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 07:20:02 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6826) Revert FileSystem create method
 that takes CreateFlags
In-Reply-To: <6937853.16451276636883446.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881172#action_12881172 ] 

Hudson commented on HADOOP-6826:
--------------------------------

Integrated in Hadoop-Common-trunk #374 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/374/])
    HADOOP-6826.  Revert FileSystem create method that takes CreateFlags.


> Revert FileSystem create method that takes CreateFlags
> ------------------------------------------------------
>
>                 Key: HADOOP-6826
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6826
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6826.patch
>
>
> As discussed in HDFS-609 and HADOOP-5438 we should back out the FileSystem create() method that takes a set of CreateFlag objects, until the interface has been agreed upon and fully tested.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9190-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 17:30:43 2010
Return-Path: <common-issues-return-9190-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 90825 invoked from network); 22 Jun 2010 17:30:43 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 17:30:43 -0000
Received: (qmail 51751 invoked by uid 500); 22 Jun 2010 17:30:42 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 51309 invoked by uid 500); 22 Jun 2010 17:30:41 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51299 invoked by uid 99); 22 Jun 2010 17:30:40 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 17:30:40 +0000
X-ASF-Spam-Status: No, hits=-1537.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 17:30:40 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5MHUJsh019186
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 17:30:20 GMT
Message-ID: <12617036.16801277227819830.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 13:30:19 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6815) refreshSuperUserGroupsConfiguration
 should use server side configuration for the refresh
In-Reply-To: <20039948.10941276119315214.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6815:
-----------------------------------

    Attachment: HADOOP-6815-YH20-1.patch

for previous version, not for commit

> refreshSuperUserGroupsConfiguration should use server side configuration for the refresh
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6815
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6815
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6815-1.patch, HADOOP-6815-2.patch, HADOOP-6815-3.patch, HADOOP-6815-YH20-1.patch, HADOOP-6815.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9191-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 18:00:20 2010
Return-Path: <common-issues-return-9191-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96051 invoked from network); 22 Jun 2010 18:00:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 18:00:19 -0000
Received: (qmail 98134 invoked by uid 500); 22 Jun 2010 18:00:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 98053 invoked by uid 500); 22 Jun 2010 18:00:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 98038 invoked by uid 99); 22 Jun 2010 18:00:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 18:00:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 18:00:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5MHxs71019510
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 17:59:55 GMT
Message-ID: <29227177.17311277229594708.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 13:59:54 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Resolved: (HADOOP-6716) System won't start in non-secure
 mode when kerb5.conf (edu.mit.kerberos on Mac) is not present
In-Reply-To: <19456845.94831271789690859.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik resolved HADOOP-6716.
------------------------------------

    Resolution: Fixed

in trunk done as part of HADOOP-6815.

> System won't start in non-secure mode when kerb5.conf (edu.mit.kerberos on Mac) is not present
> ----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6716
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6716
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6716-BP20-3.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9192-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 18:56:18 2010
Return-Path: <common-issues-return-9192-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18501 invoked from network); 22 Jun 2010 18:56:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 18:56:18 -0000
Received: (qmail 66255 invoked by uid 500); 22 Jun 2010 18:56:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66186 invoked by uid 500); 22 Jun 2010 18:56:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66174 invoked by uid 99); 22 Jun 2010 18:56:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 18:56:17 +0000
X-ASF-Spam-Status: No, hits=-1537.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 18:56:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5MItu6d020184
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 18:55:56 GMT
Message-ID: <30382643.18861277232956513.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 14:55:56 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6815) refreshSuperUserGroupsConfiguration
 should use server side configuration for the refresh
In-Reply-To: <20039948.10941276119315214.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6815:
-----------------------------------

    Attachment: HADOOP-6815-5.patch

> refreshSuperUserGroupsConfiguration should use server side configuration for the refresh
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6815
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6815
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6815-1.patch, HADOOP-6815-2.patch, HADOOP-6815-3.patch, HADOOP-6815-5.patch, HADOOP-6815-YH20-1.patch, HADOOP-6815.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9193-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 18:56:24 2010
Return-Path: <common-issues-return-9193-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18753 invoked from network); 22 Jun 2010 18:56:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 18:56:23 -0000
Received: (qmail 67757 invoked by uid 500); 22 Jun 2010 18:56:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67715 invoked by uid 500); 22 Jun 2010 18:56:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67706 invoked by uid 99); 22 Jun 2010 18:56:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 18:56:22 +0000
X-ASF-Spam-Status: No, hits=-1537.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 18:56:22 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5MIu2fg020229
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 18:56:02 GMT
Message-ID: <29268224.19011277232962099.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 14:56:02 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6815) refreshSuperUserGroupsConfiguration
 should use server side configuration for the refresh
In-Reply-To: <20039948.10941276119315214.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6815:
-----------------------------------

    Status: Open  (was: Patch Available)

> refreshSuperUserGroupsConfiguration should use server side configuration for the refresh
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6815
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6815
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6815-1.patch, HADOOP-6815-2.patch, HADOOP-6815-3.patch, HADOOP-6815-5.patch, HADOOP-6815-YH20-1.patch, HADOOP-6815.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9194-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 18:56:26 2010
Return-Path: <common-issues-return-9194-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18969 invoked from network); 22 Jun 2010 18:56:26 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 18:56:26 -0000
Received: (qmail 67942 invoked by uid 500); 22 Jun 2010 18:56:26 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 67895 invoked by uid 500); 22 Jun 2010 18:56:26 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 67887 invoked by uid 99); 22 Jun 2010 18:56:26 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 18:56:25 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 18:56:23 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5MIu2iU020238
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 18:56:02 GMT
Message-ID: <30461395.19041277232962460.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 14:56:02 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6815) refreshSuperUserGroupsConfiguration
 should use server side configuration for the refresh
In-Reply-To: <20039948.10941276119315214.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6815:
-----------------------------------

    Status: Patch Available  (was: Open)

> refreshSuperUserGroupsConfiguration should use server side configuration for the refresh
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6815
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6815
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6815-1.patch, HADOOP-6815-2.patch, HADOOP-6815-3.patch, HADOOP-6815-5.patch, HADOOP-6815-YH20-1.patch, HADOOP-6815.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9195-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 20:26:17 2010
Return-Path: <common-issues-return-9195-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48959 invoked from network); 22 Jun 2010 20:26:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 20:26:17 -0000
Received: (qmail 95785 invoked by uid 500); 22 Jun 2010 20:26:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 95714 invoked by uid 500); 22 Jun 2010 20:26:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 95706 invoked by uid 99); 22 Jun 2010 20:26:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 20:26:16 +0000
X-ASF-Spam-Status: No, hits=-1537.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 20:26:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5MKPtP3021295
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 20:25:55 GMT
Message-ID: <24987608.21101277238355252.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 16:25:55 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6815) refreshSuperUserGroupsConfiguration
 should use server side configuration for the refresh
In-Reply-To: <20039948.10941276119315214.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881339#action_12881339 ] 

Hadoop QA commented on HADOOP-6815:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12447728/HADOOP-6815-5.patch
  against trunk revision 956710.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/591/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/591/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/591/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/591/console

This message is automatically generated.

> refreshSuperUserGroupsConfiguration should use server side configuration for the refresh
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6815
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6815
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6815-1.patch, HADOOP-6815-2.patch, HADOOP-6815-3.patch, HADOOP-6815-5.patch, HADOOP-6815-YH20-1.patch, HADOOP-6815.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9196-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 20:51:21 2010
Return-Path: <common-issues-return-9196-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 54994 invoked from network); 22 Jun 2010 20:51:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 20:51:19 -0000
Received: (qmail 32409 invoked by uid 500); 22 Jun 2010 20:51:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 32354 invoked by uid 500); 22 Jun 2010 20:51:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 32346 invoked by uid 99); 22 Jun 2010 20:51:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 20:51:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 20:51:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5MKosAt021493
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 20:50:54 GMT
Message-ID: <19041021.21501277239854524.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 16:50:54 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6834) TFile.append compares initial key
 against null lastKey
In-Reply-To: <10707501.8491277191496663.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881352#action_12881352 ] 

Hong Tang commented on HADOOP-6834:
-----------------------------------

In the first append, lastLen is 0, lastKey refers to a valid byte array whose bytes are all zero. So the behavior is determined. However, I do agree that it is possible for a customary comparator, an zero-length byte array may not be a legal key type. So it would be nice to do a first-key check.

Ahad, would you be interested in providing a patch, or perhaps share with us a sample program with customized comparator that fails TFile.Writer.append?

> TFile.append compares initial key against null lastKey  
> --------------------------------------------------------
>
>                 Key: HADOOP-6834
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6834
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Ahad Rana
>
> The following code in TFile.KeyReigster.close: 
>             byte[] lastKey = lastKeyBufferOS.getBuffer();
>             int lastLen = lastKeyBufferOS.size();
>             if (tfileMeta.getComparator().compare(key, 0, len, lastKey, 0,
>                 lastLen) < 0) {
>               throw new IOException("Keys are not added in sorted order");
>             }
> compares the initial  key (passed in via  TFile.Writer.append) against a technically NULL lastKey. lastKey is not initialized until after the first call to TFile.Writer.append. The underlying RawComparator interface used for comparisons does not stipulate the proper behavior when either length 1  or length 2 is zero. In the case of LongWritable, its WritableComparator implementation does an unsafe read on the passed in byte arrays b1 and b2. Since TFile pre-allocates the buffer used for storing lastKey, this passes a valid buffer with zero count to LongWritable's comparator, which ignores length and thus produces incorrect results. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9197-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 21:11:19 2010
Return-Path: <common-issues-return-9197-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 63521 invoked from network); 22 Jun 2010 21:11:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 21:11:18 -0000
Received: (qmail 63617 invoked by uid 500); 22 Jun 2010 21:11:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63555 invoked by uid 500); 22 Jun 2010 21:11:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63547 invoked by uid 99); 22 Jun 2010 21:11:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 21:11:18 +0000
X-ASF-Spam-Status: No, hits=-1537.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 21:11:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5MLAvu4021769
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 21:10:57 GMT
Message-ID: <17274247.21981277241057373.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 17:10:57 -0400 (EDT)
From: "Ahad Rana (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6834) TFile.append compares initial key
 against null lastKey
In-Reply-To: <10707501.8491277191496663.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881362#action_12881362 ] 

Ahad Rana commented on HADOOP-6834:
-----------------------------------

Hi Hong,

Yes I would be happy to provide a patch. As far as triggering the failure, if you append a LongWritable with a negative value as the first key value and use the LongWritable's default Comparator, this will trigger the bug. I would be happy to provide a code snippet if required. In the case of LongWritable's comparator, it interprets the uninitialized zero bytes of the lastKey as a zero (since it does not check length), and thus this causes the out-of-order insertion exception. Either LongWritable's comparator is implemented improperly, or the contract implied by RawComparator is not clear on what happens in the case where you compare a non-zero byte array to a zero byte array. Probably both LongWritable's RawComparator should be fixed, but either way, passing in a uninitialized lastKey is going to cause problems for someone else down the road, so probably best to fix this as well ?     

 

> TFile.append compares initial key against null lastKey  
> --------------------------------------------------------
>
>                 Key: HADOOP-6834
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6834
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Ahad Rana
>
> The following code in TFile.KeyReigster.close: 
>             byte[] lastKey = lastKeyBufferOS.getBuffer();
>             int lastLen = lastKeyBufferOS.size();
>             if (tfileMeta.getComparator().compare(key, 0, len, lastKey, 0,
>                 lastLen) < 0) {
>               throw new IOException("Keys are not added in sorted order");
>             }
> compares the initial  key (passed in via  TFile.Writer.append) against a technically NULL lastKey. lastKey is not initialized until after the first call to TFile.Writer.append. The underlying RawComparator interface used for comparisons does not stipulate the proper behavior when either length 1  or length 2 is zero. In the case of LongWritable, its WritableComparator implementation does an unsafe read on the passed in byte arrays b1 and b2. Since TFile pre-allocates the buffer used for storing lastKey, this passes a valid buffer with zero count to LongWritable's comparator, which ignores length and thus produces incorrect results. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9198-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 21:28:17 2010
Return-Path: <common-issues-return-9198-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 67467 invoked from network); 22 Jun 2010 21:28:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 21:28:17 -0000
Received: (qmail 82160 invoked by uid 500); 22 Jun 2010 21:28:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82079 invoked by uid 500); 22 Jun 2010 21:28:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82070 invoked by uid 99); 22 Jun 2010 21:28:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 21:28:16 +0000
X-ASF-Spam-Status: No, hits=-1537.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 21:28:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5MLRtfV021854
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 21:27:55 GMT
Message-ID: <11063946.22101277242075398.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 17:27:55 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6834) TFile.append compares initial key
 against null lastKey
In-Reply-To: <10707501.8491277191496663.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881366#action_12881366 ] 

Hong Tang commented on HADOOP-6834:
-----------------------------------

Yes, I guess LongWritable could be more defensive by throwing an exception indicating illegal format. Either way, TFile needs to be fixed.

I'd be happy to shepherd you with the patch process.

> TFile.append compares initial key against null lastKey  
> --------------------------------------------------------
>
>                 Key: HADOOP-6834
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6834
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Ahad Rana
>
> The following code in TFile.KeyReigster.close: 
>             byte[] lastKey = lastKeyBufferOS.getBuffer();
>             int lastLen = lastKeyBufferOS.size();
>             if (tfileMeta.getComparator().compare(key, 0, len, lastKey, 0,
>                 lastLen) < 0) {
>               throw new IOException("Keys are not added in sorted order");
>             }
> compares the initial  key (passed in via  TFile.Writer.append) against a technically NULL lastKey. lastKey is not initialized until after the first call to TFile.Writer.append. The underlying RawComparator interface used for comparisons does not stipulate the proper behavior when either length 1  or length 2 is zero. In the case of LongWritable, its WritableComparator implementation does an unsafe read on the passed in byte arrays b1 and b2. Since TFile pre-allocates the buffer used for storing lastKey, this passes a valid buffer with zero count to LongWritable's comparator, which ignores length and thus produces incorrect results. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9199-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 21:41:23 2010
Return-Path: <common-issues-return-9199-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69593 invoked from network); 22 Jun 2010 21:41:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 21:41:22 -0000
Received: (qmail 93451 invoked by uid 500); 22 Jun 2010 21:41:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 93384 invoked by uid 500); 22 Jun 2010 21:41:21 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 93376 invoked by uid 99); 22 Jun 2010 21:41:21 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 21:41:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 21:41:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5MLevE8021956
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 21:40:58 GMT
Message-ID: <13368115.22371277242857976.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 17:40:57 -0400 (EDT)
From: "Ahad Rana (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6834) TFile.append compares initial key
 against null lastKey
In-Reply-To: <10707501.8491277191496663.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881372#action_12881372 ] 

Ahad Rana commented on HADOOP-6834:
-----------------------------------

Hi Hong,

Sure, that would be great. What are the requirements these days ? Which version of hadoop should the patch target ? 

Ahad.

> TFile.append compares initial key against null lastKey  
> --------------------------------------------------------
>
>                 Key: HADOOP-6834
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6834
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Ahad Rana
>
> The following code in TFile.KeyReigster.close: 
>             byte[] lastKey = lastKeyBufferOS.getBuffer();
>             int lastLen = lastKeyBufferOS.size();
>             if (tfileMeta.getComparator().compare(key, 0, len, lastKey, 0,
>                 lastLen) < 0) {
>               throw new IOException("Keys are not added in sorted order");
>             }
> compares the initial  key (passed in via  TFile.Writer.append) against a technically NULL lastKey. lastKey is not initialized until after the first call to TFile.Writer.append. The underlying RawComparator interface used for comparisons does not stipulate the proper behavior when either length 1  or length 2 is zero. In the case of LongWritable, its WritableComparator implementation does an unsafe read on the passed in byte arrays b1 and b2. Since TFile pre-allocates the buffer used for storing lastKey, this passes a valid buffer with zero count to LongWritable's comparator, which ignores length and thus produces incorrect results. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9200-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 21:58:15 2010
Return-Path: <common-issues-return-9200-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 72244 invoked from network); 22 Jun 2010 21:58:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 21:58:15 -0000
Received: (qmail 12206 invoked by uid 500); 22 Jun 2010 21:58:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12078 invoked by uid 500); 22 Jun 2010 21:58:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12070 invoked by uid 99); 22 Jun 2010 21:58:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 21:58:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 21:58:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5MLvoXp022368
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 21:57:50 GMT
Message-ID: <15967793.311277243870350.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 17:57:50 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6815) refreshSuperUserGroupsConfiguration
 should use server side configuration for the refresh
In-Reply-To: <20039948.10941276119315214.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881386#action_12881386 ] 

Jitendra Nath Pandey commented on HADOOP-6815:
----------------------------------------------

+1 for the patch

> refreshSuperUserGroupsConfiguration should use server side configuration for the refresh
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6815
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6815
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6815-1.patch, HADOOP-6815-2.patch, HADOOP-6815-3.patch, HADOOP-6815-5.patch, HADOOP-6815-YH20-1.patch, HADOOP-6815.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9201-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 22:13:17 2010
Return-Path: <common-issues-return-9201-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 79063 invoked from network); 22 Jun 2010 22:13:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 22:13:16 -0000
Received: (qmail 24226 invoked by uid 500); 22 Jun 2010 22:13:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 24171 invoked by uid 500); 22 Jun 2010 22:13:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 24077 invoked by uid 99); 22 Jun 2010 22:13:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 22:13:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 22:13:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5MMCpDI022626
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 22:12:51 GMT
Message-ID: <16322304.981277244771848.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 18:12:51 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6516) Distch is not documented?
In-Reply-To: <1930452274.89871264632514793.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Allen Wittenauer updated HADOOP-6516:
-------------------------------------

    Fix Version/s: 0.22.0

> Distch is not documented?
> -------------------------
>
>                 Key: HADOOP-6516
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6516
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>            Reporter: Allen Wittenauer
>            Priority: Minor
>             Fix For: 0.22.0
>
>
> HADOOP-4661 introduced distch, but I can find no documentation on how to actually use it.  Is it really non existent or am I just missing it?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9203-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 22:18:20 2010
Return-Path: <common-issues-return-9203-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80044 invoked from network); 22 Jun 2010 22:18:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 22:18:19 -0000
Received: (qmail 27751 invoked by uid 500); 22 Jun 2010 22:18:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27652 invoked by uid 500); 22 Jun 2010 22:18:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27615 invoked by uid 99); 22 Jun 2010 22:18:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 22:18:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 22:18:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5MMHtZM022752
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 22:17:55 GMT
Message-ID: <26991293.1371277245075682.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 18:17:55 -0400 (EDT)
From: "Hong Tang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6834) TFile.append compares initial key
 against null lastKey
In-Reply-To: <10707501.8491277191496663.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881404#action_12881404 ] 

Hong Tang commented on HADOOP-6834:
-----------------------------------

You should always submit the patch for trunk. If you are using older releases, you may also backport the patch (attach here) and request it to be committed to older branches too.

> TFile.append compares initial key against null lastKey  
> --------------------------------------------------------
>
>                 Key: HADOOP-6834
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6834
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.20.1, 0.20.2
>            Reporter: Ahad Rana
>
> The following code in TFile.KeyReigster.close: 
>             byte[] lastKey = lastKeyBufferOS.getBuffer();
>             int lastLen = lastKeyBufferOS.size();
>             if (tfileMeta.getComparator().compare(key, 0, len, lastKey, 0,
>                 lastLen) < 0) {
>               throw new IOException("Keys are not added in sorted order");
>             }
> compares the initial  key (passed in via  TFile.Writer.append) against a technically NULL lastKey. lastKey is not initialized until after the first call to TFile.Writer.append. The underlying RawComparator interface used for comparisons does not stipulate the proper behavior when either length 1  or length 2 is zero. In the case of LongWritable, its WritableComparator implementation does an unsafe read on the passed in byte arrays b1 and b2. Since TFile pre-allocates the buffer used for storing lastKey, this passes a valid buffer with zero count to LongWritable's comparator, which ignores length and thus produces incorrect results. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9202-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 22:18:20 2010
Return-Path: <common-issues-return-9202-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80030 invoked from network); 22 Jun 2010 22:18:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 22:18:19 -0000
Received: (qmail 27670 invoked by uid 500); 22 Jun 2010 22:18:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27614 invoked by uid 500); 22 Jun 2010 22:18:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27606 invoked by uid 99); 22 Jun 2010 22:18:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 22:18:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 22:18:16 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5MMHtiO022743
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 22:17:55 GMT
Message-ID: <30973603.1341277245075299.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 18:17:55 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6647) balancer fails with "is not
 authorized for protocol interface NamenodeProtocol" in secure environment
In-Reply-To: <387797205.373021269030508879.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6647:
-----------------------------------

    Attachment: HADOOP-6647.patch

> balancer fails with "is not authorized for protocol interface NamenodeProtocol" in secure environment
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6647
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6647
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6647-BP20.patch, HADOOP-6647.patch
>
>
> user logs in as hdfs/something@something and tries to run balancer.
> balancer is using NameNode Protocol which authorizes based on server principal key.
> but NameNode key is hdfs/_HOST@.. now. so it fails. 
> To fix we need to compare the short names only.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9204-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 22:20:12 2010
Return-Path: <common-issues-return-9204-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80387 invoked from network); 22 Jun 2010 22:20:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 22:20:12 -0000
Received: (qmail 29787 invoked by uid 500); 22 Jun 2010 22:20:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 29756 invoked by uid 500); 22 Jun 2010 22:20:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 29748 invoked by uid 99); 22 Jun 2010 22:20:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 22:20:12 +0000
X-ASF-Spam-Status: No, hits=-1537.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 22:20:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5MMJpc7022764
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 22:19:51 GMT
Message-ID: <26364728.1411277245191543.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 18:19:51 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6647) balancer fails with "is not
 authorized for protocol interface NamenodeProtocol" in secure environment
In-Reply-To: <387797205.373021269030508879.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6647:
-----------------------------------

    Status: Patch Available  (was: Open)

> balancer fails with "is not authorized for protocol interface NamenodeProtocol" in secure environment
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6647
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6647
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6647-BP20.patch, HADOOP-6647.patch
>
>
> user logs in as hdfs/something@something and tries to run balancer.
> balancer is using NameNode Protocol which authorizes based on server principal key.
> but NameNode key is hdfs/_HOST@.. now. so it fails. 
> To fix we need to compare the short names only.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9205-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 22:30:16 2010
Return-Path: <common-issues-return-9205-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82821 invoked from network); 22 Jun 2010 22:30:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 22:30:16 -0000
Received: (qmail 37961 invoked by uid 500); 22 Jun 2010 22:30:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 37896 invoked by uid 500); 22 Jun 2010 22:30:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 37884 invoked by uid 99); 22 Jun 2010 22:30:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 22:30:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 22:30:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5MMTpP5022985
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 22:29:51 GMT
Message-ID: <12110851.2071277245791365.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 18:29:51 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Reopened: (HADOOP-6724) IPC doesn't properly handle IOEs
 thrown by socket factory
In-Reply-To: <4486259.186941272248270321.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon reopened HADOOP-6724:
---------------------------------


Think we should commit this to branch-20 as well. It's pretty nasty and a simple fix.

> IPC doesn't properly handle IOEs thrown by socket factory
> ---------------------------------------------------------
>
>                 Key: HADOOP-6724
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6724
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.20.3, 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.21.0
>
>         Attachments: hadoop-6724-20-1.patch, hadoop-6724.txt
>
>
> If the socket factory throws an IOE inside setupIOStreams, then handleConnectionFailure will be called with socket still null, and thus generate an NPE on socket.close(). This ends up orphaning clients, etc.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9206-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 22:46:13 2010
Return-Path: <common-issues-return-9206-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 85478 invoked from network); 22 Jun 2010 22:46:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 22:46:13 -0000
Received: (qmail 49503 invoked by uid 500); 22 Jun 2010 22:46:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49466 invoked by uid 500); 22 Jun 2010 22:46:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49457 invoked by uid 99); 22 Jun 2010 22:46:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 22:46:12 +0000
X-ASF-Spam-Status: No, hits=-1537.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 22:46:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5MMjp1P023207
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 22:45:51 GMT
Message-ID: <19400654.2701277246751872.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 18:45:51 -0400 (EDT)
From: "Jitendra Nath Pandey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6815) refreshSuperUserGroupsConfiguration
 should use server side configuration for the refresh
In-Reply-To: <20039948.10941276119315214.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881427#action_12881427 ] 

Jitendra Nath Pandey commented on HADOOP-6815:
----------------------------------------------

Sorry, I missed one comment which I should have mentioned earlier.

Do we need to increment the version number of RefreshUserMappingsProtocol.java?
+1 apart from that.

> refreshSuperUserGroupsConfiguration should use server side configuration for the refresh
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6815
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6815
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6815-1.patch, HADOOP-6815-2.patch, HADOOP-6815-3.patch, HADOOP-6815-5.patch, HADOOP-6815-YH20-1.patch, HADOOP-6815.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9207-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 22:51:15 2010
Return-Path: <common-issues-return-9207-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 86181 invoked from network); 22 Jun 2010 22:51:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 22:51:15 -0000
Received: (qmail 52079 invoked by uid 500); 22 Jun 2010 22:51:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52004 invoked by uid 500); 22 Jun 2010 22:51:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 51996 invoked by uid 99); 22 Jun 2010 22:51:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 22:51:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 22:51:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5MMooUO023242
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 22:50:50 GMT
Message-ID: <12297731.2771277247050456.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 18:50:50 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6815) refreshSuperUserGroupsConfiguration
 should use server side configuration for the refresh
In-Reply-To: <20039948.10941276119315214.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881430#action_12881430 ] 

Boris Shkolnik commented on HADOOP-6815:
----------------------------------------

no, it is newly introduced protocol.

> refreshSuperUserGroupsConfiguration should use server side configuration for the refresh
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6815
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6815
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6815-1.patch, HADOOP-6815-2.patch, HADOOP-6815-3.patch, HADOOP-6815-5.patch, HADOOP-6815-YH20-1.patch, HADOOP-6815.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9208-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 23:09:16 2010
Return-Path: <common-issues-return-9208-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 93256 invoked from network); 22 Jun 2010 23:09:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 23:09:15 -0000
Received: (qmail 64715 invoked by uid 500); 22 Jun 2010 23:09:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 64593 invoked by uid 500); 22 Jun 2010 23:09:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 64585 invoked by uid 99); 22 Jun 2010 23:09:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 23:09:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 23:09:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5MN8oWu023427
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 23:08:51 GMT
Message-ID: <27432039.3071277248130967.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 19:08:50 -0400 (EDT)
From: "Boris Shkolnik (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6815) refreshSuperUserGroupsConfiguration
 should use server side configuration for the refresh
In-Reply-To: <20039948.10941276119315214.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Boris Shkolnik updated HADOOP-6815:
-----------------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

javadoc warning is related to "sun.security.krb5.PrincipalName is Sun proprietary API and may be removed in a future release" issue introduces elsewhere.

committed to trunk.

> refreshSuperUserGroupsConfiguration should use server side configuration for the refresh
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6815
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6815
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6815-1.patch, HADOOP-6815-2.patch, HADOOP-6815-3.patch, HADOOP-6815-5.patch, HADOOP-6815-YH20-1.patch, HADOOP-6815.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9209-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 23:23:14 2010
Return-Path: <common-issues-return-9209-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95099 invoked from network); 22 Jun 2010 23:23:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 23:23:14 -0000
Received: (qmail 76794 invoked by uid 500); 22 Jun 2010 23:23:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76723 invoked by uid 500); 22 Jun 2010 23:23:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76715 invoked by uid 99); 22 Jun 2010 23:23:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 23:23:13 +0000
X-ASF-Spam-Status: No, hits=-1538.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 23:23:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5MNMrVe023537
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 23:22:53 GMT
Message-ID: <8013418.3371277248973204.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 19:22:53 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6800) Harmonize JAR library versions
In-Reply-To: <8215141.145191275515876547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6800:
------------------------------

    Attachment: HADOOP-6800.patch

Here's an updated version which uses Jetty's ant dependency (1.6.5) rather than paranamer's (1.7.1, which is pulled in by Avro and is not a runtime dependency). 

> Harmonize JAR library versions
> ------------------------------
>
>                 Key: HADOOP-6800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6800.patch, HADOOP-6800.patch
>
>
> Currently, multiple versions of the same library JAR are being pulled in for various reasons.
> * Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
> * Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9211-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 23:23:18 2010
Return-Path: <common-issues-return-9211-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95208 invoked from network); 22 Jun 2010 23:23:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 23:23:18 -0000
Received: (qmail 77423 invoked by uid 500); 22 Jun 2010 23:23:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77320 invoked by uid 500); 22 Jun 2010 23:23:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77311 invoked by uid 99); 22 Jun 2010 23:23:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 23:23:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 23:23:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5MNMsr8023555
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 23:22:54 GMT
Message-ID: <27324558.3431277248973997.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 19:22:53 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6800) Harmonize JAR library versions
In-Reply-To: <8215141.145191275515876547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6800:
------------------------------

    Status: Patch Available  (was: Open)

> Harmonize JAR library versions
> ------------------------------
>
>                 Key: HADOOP-6800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6800.patch, HADOOP-6800.patch
>
>
> Currently, multiple versions of the same library JAR are being pulled in for various reasons.
> * Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
> * Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9210-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 23:23:18 2010
Return-Path: <common-issues-return-9210-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 95199 invoked from network); 22 Jun 2010 23:23:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 23:23:18 -0000
Received: (qmail 77386 invoked by uid 500); 22 Jun 2010 23:23:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 77310 invoked by uid 500); 22 Jun 2010 23:23:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 77302 invoked by uid 99); 22 Jun 2010 23:23:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 23:23:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 23:23:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5MNMrNE023546
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 23:22:53 GMT
Message-ID: <31062694.3401277248973612.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 19:22:53 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6800) Harmonize JAR library versions
In-Reply-To: <8215141.145191275515876547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6800:
------------------------------

    Status: Open  (was: Patch Available)

> Harmonize JAR library versions
> ------------------------------
>
>                 Key: HADOOP-6800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6800.patch, HADOOP-6800.patch
>
>
> Currently, multiple versions of the same library JAR are being pulled in for various reasons.
> * Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
> * Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9212-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Jun 22 23:51:18 2010
Return-Path: <common-issues-return-9212-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98766 invoked from network); 22 Jun 2010 23:51:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Jun 2010 23:51:18 -0000
Received: (qmail 1662 invoked by uid 500); 22 Jun 2010 23:51:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1595 invoked by uid 500); 22 Jun 2010 23:51:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1400 invoked by uid 99); 22 Jun 2010 23:51:17 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 23:51:17 +0000
X-ASF-Spam-Status: No, hits=-1538.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jun 2010 23:51:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5MNov0J023807
	for <common-issues@hadoop.apache.org>; Tue, 22 Jun 2010 23:50:57 GMT
Message-ID: <13939964.4121277250657251.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 19:50:57 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6815) refreshSuperUserGroupsConfiguration
 should use server side configuration for the refresh
In-Reply-To: <20039948.10941276119315214.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881460#action_12881460 ] 

Hudson commented on HADOOP-6815:
--------------------------------

Integrated in Hadoop-Common-trunk-Commit #314 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/314/])
    HADOOP-6815. refreshSuperUserGroupsConfiguration should use server side configuration for the refresh


> refreshSuperUserGroupsConfiguration should use server side configuration for the refresh
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6815
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6815
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6815-1.patch, HADOOP-6815-2.patch, HADOOP-6815-3.patch, HADOOP-6815-5.patch, HADOOP-6815-YH20-1.patch, HADOOP-6815.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9213-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 00:08:12 2010
Return-Path: <common-issues-return-9213-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5596 invoked from network); 23 Jun 2010 00:08:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 00:08:12 -0000
Received: (qmail 11644 invoked by uid 500); 23 Jun 2010 00:08:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 11601 invoked by uid 500); 23 Jun 2010 00:08:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 11593 invoked by uid 99); 23 Jun 2010 00:08:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 00:08:11 +0000
X-ASF-Spam-Status: No, hits=-1538.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 00:08:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5N07oKT024038
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 00:07:50 GMT
Message-ID: <5285363.4641277251670361.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 20:07:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6800) Harmonize JAR library versions
In-Reply-To: <8215141.145191275515876547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881471#action_12881471 ] 

Hadoop QA commented on HADOOP-6800:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12447762/HADOOP-6800.patch
  against trunk revision 957074.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/82/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/82/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/82/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/82/console

This message is automatically generated.

> Harmonize JAR library versions
> ------------------------------
>
>                 Key: HADOOP-6800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6800.patch, HADOOP-6800.patch
>
>
> Currently, multiple versions of the same library JAR are being pulled in for various reasons.
> * Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
> * Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9214-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 01:31:15 2010
Return-Path: <common-issues-return-9214-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 29554 invoked from network); 23 Jun 2010 01:31:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 01:31:15 -0000
Received: (qmail 61339 invoked by uid 500); 23 Jun 2010 01:31:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61234 invoked by uid 500); 23 Jun 2010 01:31:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61226 invoked by uid 99); 23 Jun 2010 01:31:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 01:31:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 01:31:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5N1UoIl024852
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 01:30:50 GMT
Message-ID: <24891849.6481277256650299.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 21:30:50 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6647) balancer fails with "is not
 authorized for protocol interface NamenodeProtocol" in secure environment
In-Reply-To: <387797205.373021269030508879.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881508#action_12881508 ] 

Hadoop QA commented on HADOOP-6647:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12447754/HADOOP-6647.patch
  against trunk revision 957074.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/592/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/592/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/592/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/592/console

This message is automatically generated.

> balancer fails with "is not authorized for protocol interface NamenodeProtocol" in secure environment
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6647
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6647
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6647-BP20.patch, HADOOP-6647.patch
>
>
> user logs in as hdfs/something@something and tries to run balancer.
> balancer is using NameNode Protocol which authorizes based on server principal key.
> but NameNode key is hdfs/_HOST@.. now. so it fails. 
> To fix we need to compare the short names only.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9215-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 02:06:17 2010
Return-Path: <common-issues-return-9215-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39552 invoked from network); 23 Jun 2010 02:06:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 02:06:17 -0000
Received: (qmail 81578 invoked by uid 500); 23 Jun 2010 02:06:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81423 invoked by uid 500); 23 Jun 2010 02:06:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81415 invoked by uid 99); 23 Jun 2010 02:06:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 02:06:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 02:06:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5N25qO3025065
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 02:05:52 GMT
Message-ID: <15784894.6791277258752441.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 22:05:52 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6815) refreshSuperUserGroupsConfiguration
 should use server side configuration for the refresh
In-Reply-To: <20039948.10941276119315214.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881516#action_12881516 ] 

Hudson commented on HADOOP-6815:
--------------------------------

Integrated in Hadoop-Hdfs-trunk-Commit #317 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/317/])
    HDFS-1192. refreshSuperUserGroupsConfiguration should use server side configuration for the refresh (for HADOOP-6815)


> refreshSuperUserGroupsConfiguration should use server side configuration for the refresh
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6815
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6815
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6815-1.patch, HADOOP-6815-2.patch, HADOOP-6815-3.patch, HADOOP-6815-5.patch, HADOOP-6815-YH20-1.patch, HADOOP-6815.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9216-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 02:06:20 2010
Return-Path: <common-issues-return-9216-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 39598 invoked from network); 23 Jun 2010 02:06:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 02:06:20 -0000
Received: (qmail 81710 invoked by uid 500); 23 Jun 2010 02:06:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81675 invoked by uid 500); 23 Jun 2010 02:06:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81667 invoked by uid 99); 23 Jun 2010 02:06:19 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 02:06:19 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 02:06:17 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5N25usO025119
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 02:05:56 GMT
Message-ID: <1795908.6971277258756028.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 22:05:56 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Moved: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Greg Roelofs moved MAPREDUCE-469 to HADOOP-6835:
------------------------------------------------

    Project: Hadoop Common  (was: Hadoop Map/Reduce)
        Key: HADOOP-6835  (was: MAPREDUCE-469)

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>         Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9217-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 02:16:17 2010
Return-Path: <common-issues-return-9217-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 41043 invoked from network); 23 Jun 2010 02:16:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 02:16:17 -0000
Received: (qmail 84880 invoked by uid 500); 23 Jun 2010 02:16:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84819 invoked by uid 500); 23 Jun 2010 02:16:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84811 invoked by uid 99); 23 Jun 2010 02:16:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 02:16:16 +0000
X-ASF-Spam-Status: No, hits=-1538.4 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 02:16:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5N2Ftkf025203
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 02:15:55 GMT
Message-ID: <29677658.7171277259355222.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 22:15:55 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Greg Roelofs updated HADOOP-6835:
---------------------------------

    Hadoop Flags: [Incompatible change]
    Release Note: Processing of concatenated gzip files formerly stopped (quietly) at the end of the first substream/"member"; now processing will continue to the end of the concatenated stream, like gzip(1) does.
     Component/s: io

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>         Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9218-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 02:34:19 2010
Return-Path: <common-issues-return-9218-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 42549 invoked from network); 23 Jun 2010 02:34:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 02:34:19 -0000
Received: (qmail 91402 invoked by uid 500); 23 Jun 2010 02:34:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 91248 invoked by uid 500); 23 Jun 2010 02:34:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 91227 invoked by uid 99); 23 Jun 2010 02:34:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 02:34:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 02:34:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5N2Xrg3025556
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 02:33:53 GMT
Message-ID: <13823298.7851277260433404.JavaMail.jira@thor>
Date: Tue, 22 Jun 2010 22:33:53 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881522#action_12881522 ] 

Greg Roelofs commented on HADOOP-6835:
--------------------------------------

(This was MAPREDUCE-469, but Chris Douglas noted that it affects the core compression code in hadoop-common, not hadoop-mapreduce, so moved here now.)

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>         Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9219-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 05:09:14 2010
Return-Path: <common-issues-return-9219-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 84562 invoked from network); 23 Jun 2010 05:09:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 05:09:14 -0000
Received: (qmail 62974 invoked by uid 500); 23 Jun 2010 05:09:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 62849 invoked by uid 500); 23 Jun 2010 05:09:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 62841 invoked by uid 99); 23 Jun 2010 05:09:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 05:09:11 +0000
X-ASF-Spam-Status: No, hits=-1538.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 05:09:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5N58oN6027049
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 05:08:50 GMT
Message-ID: <23644117.9641277269730469.JavaMail.jira@thor>
Date: Wed, 23 Jun 2010 01:08:50 -0400 (EDT)
From: "Iyappan Srinivasan (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6836) Test Autoamtion : Method for
 restarting the daemon with new configuration.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Test Autoamtion : Method for restarting the daemon with new configuration.
--------------------------------------------------------------------------

                 Key: HADOOP-6836
                 URL: https://issues.apache.org/jira/browse/HADOOP-6836
             Project: Hadoop Common
          Issue Type: Test
          Components: test
            Reporter: Iyappan Srinivasan
            Assignee: Vinay Kumar Thota


HADOP-6772 deals with 
Common utilities for system tests.
1. A method for restarting the daemon with new configuration.
c throws Exception; 

2. A method for restarting the daemon with default configuration.
public void restart() throws Exception;

3. A method for waiting until daemon is stop.
public void waitForClusterToStop() throws Exception;

In this  some variables are made of String, instead of Long. Those needs ot be changed too. So, can this method 
"public static void restartClusterWithNewConfig(Hashtable<String,Long> props, String confFile) throws Exception;"
be generalized to accepts string too. All otehr methods should work as usual.



-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9220-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 09:02:19 2010
Return-Path: <common-issues-return-9220-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44054 invoked from network); 23 Jun 2010 09:02:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 09:02:19 -0000
Received: (qmail 61694 invoked by uid 500); 23 Jun 2010 09:02:19 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61570 invoked by uid 500); 23 Jun 2010 09:02:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61364 invoked by uid 99); 23 Jun 2010 09:02:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 09:02:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 09:02:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5N91okX029065
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 09:01:51 GMT
Message-ID: <13215418.12701277283710970.JavaMail.jira@thor>
Date: Wed, 23 Jun 2010 05:01:50 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6836) Test Autoamtion : Method for
 restarting the daemon with new configuration.
In-Reply-To: <23644117.9641277269730469.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinay Kumar Thota updated HADOOP-6836:
--------------------------------------

    Attachment: 6836-ydist-security.patch

Patch for yahoo distribution security branch.

> Test Autoamtion : Method for restarting the daemon with new configuration.
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-6836
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6836
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>            Reporter: Iyappan Srinivasan
>            Assignee: Vinay Kumar Thota
>         Attachments: 6836-ydist-security.patch
>
>
> HADOP-6772 deals with 
> Common utilities for system tests.
> 1. A method for restarting the daemon with new configuration.
> c throws Exception; 
> 2. A method for restarting the daemon with default configuration.
> public void restart() throws Exception;
> 3. A method for waiting until daemon is stop.
> public void waitForClusterToStop() throws Exception;
> In this  some variables are made of String, instead of Long. Those needs ot be changed too. So, can this method 
> "public static void restartClusterWithNewConfig(Hashtable<String,Long> props, String confFile) throws Exception;"
> be generalized to accepts string too. All otehr methods should work as usual.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9221-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 09:08:17 2010
Return-Path: <common-issues-return-9221-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49510 invoked from network); 23 Jun 2010 09:08:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 09:08:17 -0000
Received: (qmail 73731 invoked by uid 500); 23 Jun 2010 09:08:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 73602 invoked by uid 500); 23 Jun 2010 09:08:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 73592 invoked by uid 99); 23 Jun 2010 09:08:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 09:08:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 09:08:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5N97oA3029126
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 09:07:50 GMT
Message-ID: <9637179.12791277284070593.JavaMail.jira@thor>
Date: Wed, 23 Jun 2010 05:07:50 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6836) [Herriot] Test Autoamtion : Generic
 method for adding/modifying the attributes for new configuration.
In-Reply-To: <23644117.9641277269730469.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinay Kumar Thota updated HADOOP-6836:
--------------------------------------

    Summary: [Herriot] Test Autoamtion : Generic method for adding/modifying the attributes for new configuration.  (was: Test Autoamtion : Method for restarting the daemon with new configuration.)

> [Herriot] Test Autoamtion : Generic method for adding/modifying the attributes for new configuration.
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6836
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6836
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>            Reporter: Iyappan Srinivasan
>            Assignee: Vinay Kumar Thota
>         Attachments: 6836-ydist-security.patch
>
>
> HADOP-6772 deals with 
> Common utilities for system tests.
> 1. A method for restarting the daemon with new configuration.
> c throws Exception; 
> 2. A method for restarting the daemon with default configuration.
> public void restart() throws Exception;
> 3. A method for waiting until daemon is stop.
> public void waitForClusterToStop() throws Exception;
> In this  some variables are made of String, instead of Long. Those needs ot be changed too. So, can this method 
> "public static void restartClusterWithNewConfig(Hashtable<String,Long> props, String confFile) throws Exception;"
> be generalized to accepts string too. All otehr methods should work as usual.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9222-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 11:16:16 2010
Return-Path: <common-issues-return-9222-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 83196 invoked from network); 23 Jun 2010 11:16:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 11:16:16 -0000
Received: (qmail 8437 invoked by uid 500); 23 Jun 2010 11:16:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 8214 invoked by uid 500); 23 Jun 2010 11:16:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 8188 invoked by uid 99); 23 Jun 2010 11:16:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 11:16:11 +0000
X-ASF-Spam-Status: No, hits=-1539.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 11:16:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5NBFori029914
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 11:15:50 GMT
Message-ID: <29821631.13521277291750563.JavaMail.jira@thor>
Date: Wed, 23 Jun 2010 07:15:50 -0400 (EDT)
From: "Hudson (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6815) refreshSuperUserGroupsConfiguration
 should use server side configuration for the refresh
In-Reply-To: <20039948.10941276119315214.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881639#action_12881639 ] 

Hudson commented on HADOOP-6815:
--------------------------------

Integrated in Hadoop-Common-trunk #375 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/375/])
    HADOOP-6815. refreshSuperUserGroupsConfiguration should use server side configuration for the refresh


> refreshSuperUserGroupsConfiguration should use server side configuration for the refresh
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6815
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6815
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: HADOOP-6815-1.patch, HADOOP-6815-2.patch, HADOOP-6815-3.patch, HADOOP-6815-5.patch, HADOOP-6815-YH20-1.patch, HADOOP-6815.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9223-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 13:33:14 2010
Return-Path: <common-issues-return-9223-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 30421 invoked from network); 23 Jun 2010 13:33:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 13:33:14 -0000
Received: (qmail 29054 invoked by uid 500); 23 Jun 2010 13:33:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28915 invoked by uid 500); 23 Jun 2010 13:33:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28906 invoked by uid 99); 23 Jun 2010 13:33:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 13:33:11 +0000
X-ASF-Spam-Status: No, hits=-1539.6 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 13:33:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5NDWpud001254
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 13:32:51 GMT
Message-ID: <18830942.15311277299971057.JavaMail.jira@thor>
Date: Wed, 23 Jun 2010 09:32:51 -0400 (EDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6800) Harmonize JAR library versions
In-Reply-To: <8215141.145191275515876547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881691#action_12881691 ] 

Hemanth Yamijala commented on HADOOP-6800:
------------------------------------------

Tom, trying to see if I can review this, but I'm not very familiar with Ivy... Is the goal of this jira to make sure that the versions for libraries we are using  in the 3 projects are the same ? I ask because there are a few libraries with different versions in the common and mapreduce projects. For e.g. commons-logging is 1.1.1 in common and 1.0.4 in mapred, and jets3t is 0.7.1 in common and 0.6.1 in mapred. These don't seem to be excluded / modified in your patch.

Would these need to be fixed too ?

Apologies if my understanding is completely wrong... 

> Harmonize JAR library versions
> ------------------------------
>
>                 Key: HADOOP-6800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6800.patch, HADOOP-6800.patch
>
>
> Currently, multiple versions of the same library JAR are being pulled in for various reasons.
> * Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
> * Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9224-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 16:53:14 2010
Return-Path: <common-issues-return-9224-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99892 invoked from network); 23 Jun 2010 16:53:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 16:53:14 -0000
Received: (qmail 17036 invoked by uid 500); 23 Jun 2010 16:53:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 16938 invoked by uid 500); 23 Jun 2010 16:53:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 16930 invoked by uid 99); 23 Jun 2010 16:53:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 16:53:12 +0000
X-ASF-Spam-Status: No, hits=-1539.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 16:53:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5NGqqYq002898
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 16:52:52 GMT
Message-ID: <9977019.17941277311972147.JavaMail.jira@thor>
Date: Wed, 23 Jun 2010 12:52:52 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6800) Harmonize JAR library versions
In-Reply-To: <8215141.145191275515876547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6800:
------------------------------

    Status: Open  (was: Patch Available)

> Harmonize JAR library versions
> ------------------------------
>
>                 Key: HADOOP-6800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6800.patch, HADOOP-6800.patch, HADOOP-6800.patch
>
>
> Currently, multiple versions of the same library JAR are being pulled in for various reasons.
> * Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
> * Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9225-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 16:53:16 2010
Return-Path: <common-issues-return-9225-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 154 invoked from network); 23 Jun 2010 16:53:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 16:53:16 -0000
Received: (qmail 17649 invoked by uid 500); 23 Jun 2010 16:53:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17483 invoked by uid 500); 23 Jun 2010 16:53:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17475 invoked by uid 99); 23 Jun 2010 16:53:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 16:53:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 16:53:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5NGqph6002886
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 16:52:51 GMT
Message-ID: <22403940.17901277311971267.JavaMail.jira@thor>
Date: Wed, 23 Jun 2010 12:52:51 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6800) Harmonize JAR library versions
In-Reply-To: <8215141.145191275515876547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6800:
------------------------------

    Attachment: HADOOP-6800.patch

Thanks for taking a look, Hemanth. Yes, the goal is to make sure that the JAR version used between the three projects are the same. My criterion for checking this was to build the three tarballs, combine them, and visually check that there were no conflicting versions in lib. This new set of patches corrects the mismatches you saw (in some cases removing the variable since its not used in the project). 

> Harmonize JAR library versions
> ------------------------------
>
>                 Key: HADOOP-6800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6800.patch, HADOOP-6800.patch, HADOOP-6800.patch
>
>
> Currently, multiple versions of the same library JAR are being pulled in for various reasons.
> * Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
> * Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9226-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 16:56:15 2010
Return-Path: <common-issues-return-9226-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1869 invoked from network); 23 Jun 2010 16:56:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 16:56:14 -0000
Received: (qmail 21178 invoked by uid 500); 23 Jun 2010 16:56:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21128 invoked by uid 500); 23 Jun 2010 16:56:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 21120 invoked by uid 99); 23 Jun 2010 16:56:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 16:56:14 +0000
X-ASF-Spam-Status: No, hits=-1540.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 16:56:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5NGtrf8002959
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 16:55:53 GMT
Message-ID: <11410377.18081277312153163.JavaMail.jira@thor>
Date: Wed, 23 Jun 2010 12:55:53 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6800) Harmonize JAR library versions
In-Reply-To: <8215141.145191275515876547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6800:
------------------------------

    Status: Patch Available  (was: Open)

> Harmonize JAR library versions
> ------------------------------
>
>                 Key: HADOOP-6800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6800.patch, HADOOP-6800.patch, HADOOP-6800.patch
>
>
> Currently, multiple versions of the same library JAR are being pulled in for various reasons.
> * Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
> * Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9227-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 20:03:13 2010
Return-Path: <common-issues-return-9227-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 82774 invoked from network); 23 Jun 2010 20:03:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 20:03:13 -0000
Received: (qmail 15136 invoked by uid 500); 23 Jun 2010 20:03:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15074 invoked by uid 500); 23 Jun 2010 20:03:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15066 invoked by uid 99); 23 Jun 2010 20:03:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 20:03:12 +0000
X-ASF-Spam-Status: No, hits=-1540.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 20:03:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5NK2pko005368
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 20:02:51 GMT
Message-ID: <31576311.23011277323371059.JavaMail.jira@thor>
Date: Wed, 23 Jun 2010 16:02:51 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6800) Harmonize JAR library versions
In-Reply-To: <8215141.145191275515876547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881847#action_12881847 ] 

Hadoop QA commented on HADOOP-6800:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12447851/HADOOP-6800.patch
  against trunk revision 957074.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/593/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/593/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/593/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/593/console

This message is automatically generated.

> Harmonize JAR library versions
> ------------------------------
>
>                 Key: HADOOP-6800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6800.patch, HADOOP-6800.patch, HADOOP-6800.patch
>
>
> Currently, multiple versions of the same library JAR are being pulled in for various reasons.
> * Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
> * Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9228-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 21:28:24 2010
Return-Path: <common-issues-return-9228-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 88908 invoked from network); 23 Jun 2010 21:28:24 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 21:28:24 -0000
Received: (qmail 92554 invoked by uid 500); 23 Jun 2010 21:28:24 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 92480 invoked by uid 500); 23 Jun 2010 21:28:23 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 92472 invoked by uid 99); 23 Jun 2010 21:28:23 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 21:28:23 +0000
X-ASF-Spam-Status: No, hits=-1540.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 21:28:22 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5NLRweX006245
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 21:28:01 GMT
Message-ID: <29184730.24911277328478825.JavaMail.jira@thor>
Date: Wed, 23 Jun 2010 17:27:58 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881891#action_12881891 ] 

Greg Roelofs commented on HADOOP-6835:
--------------------------------------

Mildly amusing:  http://hg.openjdk.java.net/jdk7/jdk7/jdk/rev/41a5722e6e10

Looks like Sun/Oracle finally decided to fix their GZIPInputStream bug.  (Patch is dated 20100524.)

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>         Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9229-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 21:43:14 2010
Return-Path: <common-issues-return-9229-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34073 invoked from network); 23 Jun 2010 21:43:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 21:43:14 -0000
Received: (qmail 15137 invoked by uid 500); 23 Jun 2010 21:43:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 15034 invoked by uid 500); 23 Jun 2010 21:43:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 15026 invoked by uid 99); 23 Jun 2010 21:43:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 21:43:13 +0000
X-ASF-Spam-Status: No, hits=-1540.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 21:43:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5NLgqku006415
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 21:42:52 GMT
Message-ID: <7846631.25351277329372604.JavaMail.jira@thor>
Date: Wed, 23 Jun 2010 17:42:52 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6584) Provide Kerberized SSL encryption for
 webservices
In-Reply-To: <713706873.424711266804869916.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6584:
------------------------------

    Attachment: c6584-02.patch

uploading a Common patch for the current trunk

> Provide Kerberized SSL encryption for webservices
> -------------------------------------------------
>
>                 Key: HADOOP-6584
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6584
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>         Attachments: c6584-02.patch, HADOOP-6584-FixJavadoc-Y20.patch, HADOOP-6584-Y20-1.patch, HADOOP-6584-Y20-2.patch, HADOOP-6584-Y20-3.patch, HADOOP-6584-Y20-4.patch, HADOOP-6584.patch
>
>
> Some web services should be authenticated via SSL backed by Kerberos, both to provide cross-cluster secure communication and to provide intra-cluster server authentication for services such as the {Name,SecondaryName,Backup,Checkpoint}node's image transfer and balancer.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9230-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 21:46:17 2010
Return-Path: <common-issues-return-9230-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43065 invoked from network); 23 Jun 2010 21:46:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 21:46:17 -0000
Received: (qmail 17411 invoked by uid 500); 23 Jun 2010 21:46:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17350 invoked by uid 500); 23 Jun 2010 21:46:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17342 invoked by uid 99); 23 Jun 2010 21:46:16 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 21:46:16 +0000
X-ASF-Spam-Status: No, hits=-1540.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 21:46:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5NLjtFT006463
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 21:45:55 GMT
Message-ID: <39364.25511277329555778.JavaMail.jira@thor>
Date: Wed, 23 Jun 2010 17:45:55 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6584) Provide Kerberized SSL encryption for
 webservices
In-Reply-To: <713706873.424711266804869916.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6584:
------------------------------

    Status: Patch Available  (was: Open)

> Provide Kerberized SSL encryption for webservices
> -------------------------------------------------
>
>                 Key: HADOOP-6584
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6584
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>         Attachments: c6584-02.patch, HADOOP-6584-FixJavadoc-Y20.patch, HADOOP-6584-Y20-1.patch, HADOOP-6584-Y20-2.patch, HADOOP-6584-Y20-3.patch, HADOOP-6584-Y20-4.patch, HADOOP-6584.patch
>
>
> Some web services should be authenticated via SSL backed by Kerberos, both to provide cross-cluster secure communication and to provide intra-cluster server authentication for services such as the {Name,SecondaryName,Backup,Checkpoint}node's image transfer and balancer.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9231-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 21:46:19 2010
Return-Path: <common-issues-return-9231-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 43199 invoked from network); 23 Jun 2010 21:46:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 21:46:19 -0000
Received: (qmail 17732 invoked by uid 500); 23 Jun 2010 21:46:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 17628 invoked by uid 500); 23 Jun 2010 21:46:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17519 invoked by uid 99); 23 Jun 2010 21:46:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 21:46:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 21:46:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5NLjsJK006439
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 21:45:54 GMT
Message-ID: <19681562.25431277329554077.JavaMail.jira@thor>
Date: Wed, 23 Jun 2010 17:45:54 -0400 (EDT)
From: "Kan Zhang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6584) Provide Kerberized SSL encryption for
 webservices
In-Reply-To: <713706873.424711266804869916.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kan Zhang updated HADOOP-6584:
------------------------------

    Status: Open  (was: Patch Available)

> Provide Kerberized SSL encryption for webservices
> -------------------------------------------------
>
>                 Key: HADOOP-6584
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6584
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>         Attachments: c6584-02.patch, HADOOP-6584-FixJavadoc-Y20.patch, HADOOP-6584-Y20-1.patch, HADOOP-6584-Y20-2.patch, HADOOP-6584-Y20-3.patch, HADOOP-6584-Y20-4.patch, HADOOP-6584.patch
>
>
> Some web services should be authenticated via SSL backed by Kerberos, both to provide cross-cluster secure communication and to provide intra-cluster server authentication for services such as the {Name,SecondaryName,Backup,Checkpoint}node's image transfer and balancer.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9232-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 21:57:15 2010
Return-Path: <common-issues-return-9232-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78927 invoked from network); 23 Jun 2010 21:57:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 21:57:14 -0000
Received: (qmail 28189 invoked by uid 500); 23 Jun 2010 21:57:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28129 invoked by uid 500); 23 Jun 2010 21:57:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28120 invoked by uid 99); 23 Jun 2010 21:57:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 21:57:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 21:57:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5NLuncL006661
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 21:56:50 GMT
Message-ID: <33241912.25981277330209915.JavaMail.jira@thor>
Date: Wed, 23 Jun 2010 17:56:49 -0400 (EDT)
From: "Nicholas Carlini (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6837) Support for LZMA compression
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Support for LZMA compression
----------------------------

                 Key: HADOOP-6837
                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
             Project: Hadoop Common
          Issue Type: Improvement
          Components: io
            Reporter: Nicholas Carlini
         Attachments: HADOOP-6837-lzma-java-20100623.patch

Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9233-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 21:57:15 2010
Return-Path: <common-issues-return-9233-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 78999 invoked from network); 23 Jun 2010 21:57:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 21:57:15 -0000
Received: (qmail 28319 invoked by uid 500); 23 Jun 2010 21:57:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28170 invoked by uid 500); 23 Jun 2010 21:57:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28130 invoked by uid 99); 23 Jun 2010 21:57:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 21:57:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 21:57:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5NLuoVq006670
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 21:56:50 GMT
Message-ID: <20116682.26011277330210462.JavaMail.jira@thor>
Date: Wed, 23 Jun 2010 17:56:50 -0400 (EDT)
From: "Nicholas Carlini (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicholas Carlini updated HADOOP-6837:
-------------------------------------

    Attachment: HADOOP-6837-lzma-java-20100623.patch

Attached a patch. It includes an LzmaCodec, LzmaInputStream, and LzmaOutputStream. The LZMA compression/decompression uses the LZMA SDK from http://www.7-zip.org/sdk.html. The code has been tested minimally -- when used in io.SequenceFile.java, it passes the TestSetFile/TestArrayFile tests. I will attach another later on when I have fully-tested code. I will also be working on a native version of the code written in C, also based off of the LZMA SDK. It is significantly faster than than the Java code.

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9234-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 22:07:18 2010
Return-Path: <common-issues-return-9234-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 96667 invoked from network); 23 Jun 2010 22:07:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 22:07:17 -0000
Received: (qmail 44124 invoked by uid 500); 23 Jun 2010 22:07:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 44074 invoked by uid 500); 23 Jun 2010 22:07:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 44066 invoked by uid 99); 23 Jun 2010 22:07:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 22:07:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 22:07:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5NM6qqk006817
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 22:06:52 GMT
Message-ID: <15918388.26241277330812497.JavaMail.jira@thor>
Date: Wed, 23 Jun 2010 18:06:52 -0400 (EDT)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881915#action_12881915 ] 

Allen Wittenauer commented on HADOOP-6837:
------------------------------------------

The 7z SDK license is "Public Domain" and 7z LZMA is LGPL.  Is that compatible with the APL?  

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9235-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 22:14:18 2010
Return-Path: <common-issues-return-9235-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 98084 invoked from network); 23 Jun 2010 22:14:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 22:14:18 -0000
Received: (qmail 61046 invoked by uid 500); 23 Jun 2010 22:14:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 61027 invoked by uid 500); 23 Jun 2010 22:14:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 61019 invoked by uid 99); 23 Jun 2010 22:14:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 22:14:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 22:14:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5NMDrK9006930
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 22:13:53 GMT
Message-ID: <32598290.26581277331233252.JavaMail.jira@thor>
Date: Wed, 23 Jun 2010 18:13:53 -0400 (EDT)
From: "Nicholas Carlini (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Assigned: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicholas Carlini reassigned HADOOP-6837:
----------------------------------------

    Assignee: Nicholas Carlini

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9236-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 22:34:14 2010
Return-Path: <common-issues-return-9236-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1775 invoked from network); 23 Jun 2010 22:34:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 22:34:13 -0000
Received: (qmail 78894 invoked by uid 500); 23 Jun 2010 22:34:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78834 invoked by uid 500); 23 Jun 2010 22:34:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78815 invoked by uid 99); 23 Jun 2010 22:34:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 22:34:12 +0000
X-ASF-Spam-Status: No, hits=-1540.9 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 22:34:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5NMXpFW007142
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 22:33:51 GMT
Message-ID: <236781.27141277332431665.JavaMail.jira@thor>
Date: Wed, 23 Jun 2010 18:33:51 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881928#action_12881928 ] 

Greg Roelofs commented on HADOOP-6837:
--------------------------------------

7-Zip is LGPL; the LZMA SDK is not:

"License

LZMA SDK is placed in the public domain."

Given that both packages are hosted at the same site, with links to each other on the left bar, I think we can safely assume they know the difference between the two and have made a conscious decision to release them accordingly.

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9237-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 22:34:14 2010
Return-Path: <common-issues-return-9237-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 1830 invoked from network); 23 Jun 2010 22:34:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 22:34:14 -0000
Received: (qmail 79033 invoked by uid 500); 23 Jun 2010 22:34:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 78846 invoked by uid 500); 23 Jun 2010 22:34:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 78836 invoked by uid 99); 23 Jun 2010 22:34:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 22:34:13 +0000
X-ASF-Spam-Status: No, hits=-1541.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 22:34:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5NMXqMa007169
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 22:33:52 GMT
Message-ID: <12597314.27231277332432700.JavaMail.jira@thor>
Date: Wed, 23 Jun 2010 18:33:52 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881929#action_12881929 ] 

Eli Collins commented on HADOOP-6837:
-------------------------------------

Hey Nicholas,

Cool stuff.  Unfortunately LGPL in incompatible with APL so we couldn't check this in. See more at http://www.apache.org/legal/resolved.html

bq. The LGPL is ineligible primarily due to the restrictions it places on larger works, violating the third license criterion. Therefore, LGPL-licensed works must not be included in Apache products

Do you need to use this particular codec or are you just looking for something better than gzip/bzip2? If the latter HADOOP-6349 (support for FastLZ) would be a great place to direct your efforts, it's got a compatible license and like LZMA is significantly faster than gzip/bzip (and faster than the open source version of lzo). 

Thanks,
Eli


> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9238-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 22:39:18 2010
Return-Path: <common-issues-return-9238-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2590 invoked from network); 23 Jun 2010 22:39:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 22:39:18 -0000
Received: (qmail 82762 invoked by uid 500); 23 Jun 2010 22:39:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82711 invoked by uid 500); 23 Jun 2010 22:39:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82703 invoked by uid 99); 23 Jun 2010 22:39:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 22:39:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 22:39:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5NMcqZO007242
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 22:38:52 GMT
Message-ID: <19652584.27451277332732438.JavaMail.jira@thor>
Date: Wed, 23 Jun 2010 18:38:52 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881931#action_12881931 ] 

Eli Collins commented on HADOOP-6837:
-------------------------------------

If LZMA is public domain then it should safe to include. Would be good to have clarification from the author.

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9239-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 22:42:13 2010
Return-Path: <common-issues-return-9239-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2898 invoked from network); 23 Jun 2010 22:42:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 22:42:13 -0000
Received: (qmail 87150 invoked by uid 500); 23 Jun 2010 22:42:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87091 invoked by uid 500); 23 Jun 2010 22:42:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87083 invoked by uid 99); 23 Jun 2010 22:42:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 22:42:12 +0000
X-ASF-Spam-Status: No, hits=-1541.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 22:42:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5NMfqiG007278
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 22:41:52 GMT
Message-ID: <22213528.27541277332912043.JavaMail.jira@thor>
Date: Wed, 23 Jun 2010 18:41:52 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881932#action_12881932 ] 

Greg Roelofs commented on HADOOP-6837:
--------------------------------------

See the last line of their FAQ. ;-)

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9240-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 22:42:16 2010
Return-Path: <common-issues-return-9240-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 2957 invoked from network); 23 Jun 2010 22:42:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 22:42:16 -0000
Received: (qmail 87312 invoked by uid 500); 23 Jun 2010 22:42:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87242 invoked by uid 500); 23 Jun 2010 22:42:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87234 invoked by uid 99); 23 Jun 2010 22:42:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 22:42:15 +0000
X-ASF-Spam-Status: No, hits=-1541.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 22:42:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5NMfsMm007314
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 22:41:54 GMT
Message-ID: <24853540.27661277332914858.JavaMail.jira@thor>
Date: Wed, 23 Jun 2010 18:41:54 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881934#action_12881934 ] 

Greg Roelofs commented on HADOOP-6837:
--------------------------------------

LZMA is not faster than gzip/bzip2; it compresses better.  FastLZ (next item on Nicholas's plate) is faster than LZO but compresses more poorly than everything else (except maybe LZW).  They're both useful, but they address different parts of the problem domain.

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9241-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 22:44:16 2010
Return-Path: <common-issues-return-9241-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 3389 invoked from network); 23 Jun 2010 22:44:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 22:44:16 -0000
Received: (qmail 90302 invoked by uid 500); 23 Jun 2010 22:44:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 90264 invoked by uid 500); 23 Jun 2010 22:44:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 90256 invoked by uid 99); 23 Jun 2010 22:44:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 22:44:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 22:44:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5NMhqQv007357
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 22:43:52 GMT
Message-ID: <22388797.27801277333032709.JavaMail.jira@thor>
Date: Wed, 23 Jun 2010 18:43:52 -0400 (EDT)
From: "Nicholas Carlini (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881935#action_12881935 ] 

Nicholas Carlini commented on HADOOP-6837:
------------------------------------------

Per the FAQ:

"You can also read about the LZMA SDK, which is available under a more liberal license."

http://www.7-zip.org/faq.html

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9242-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Wed Jun 23 23:22:13 2010
Return-Path: <common-issues-return-9242-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 13051 invoked from network); 23 Jun 2010 23:22:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 23 Jun 2010 23:22:13 -0000
Received: (qmail 28971 invoked by uid 500); 23 Jun 2010 23:22:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 28912 invoked by uid 500); 23 Jun 2010 23:22:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 28904 invoked by uid 99); 23 Jun 2010 23:22:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 23:22:12 +0000
X-ASF-Spam-Status: No, hits=-1541.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jun 2010 23:22:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5NNLpUv007666
	for <common-issues@hadoop.apache.org>; Wed, 23 Jun 2010 23:21:51 GMT
Message-ID: <31194213.28381277335311637.JavaMail.jira@thor>
Date: Wed, 23 Jun 2010 19:21:51 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6825) FileStatus needs unit tests
In-Reply-To: <28758544.71131276463653554.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881948#action_12881948 ] 

Eli Collins commented on HADOOP-6825:
-------------------------------------

I might be missing something but doesn't this just test that the FileStatus fields are persisted, ie that the writable code works? (ie a FileStatus can be written and read back).

> FileStatus needs unit tests
> ---------------------------
>
>                 Key: HADOOP-6825
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6825
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6825.0.patch
>
>
> We need some unit tests for FileStatus to prevent problems like those we recently had on HADOOP-6796 and MAPREDUCE-1858.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9243-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 24 00:27:17 2010
Return-Path: <common-issues-return-9243-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 31434 invoked from network); 24 Jun 2010 00:27:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Jun 2010 00:27:17 -0000
Received: (qmail 66881 invoked by uid 500); 24 Jun 2010 00:27:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 66799 invoked by uid 500); 24 Jun 2010 00:27:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 66791 invoked by uid 99); 24 Jun 2010 00:27:16 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Jun 2010 00:27:16 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Jun 2010 00:27:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5O0Qp6a008274
	for <common-issues@hadoop.apache.org>; Thu, 24 Jun 2010 00:26:52 GMT
Message-ID: <21975210.29821277339211751.JavaMail.jira@thor>
Date: Wed, 23 Jun 2010 20:26:51 -0400 (EDT)
From: "Rodrigo Schmidt (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6825) FileStatus needs unit tests
In-Reply-To: <28758544.71131276463653554.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881977#action_12881977 ] 

Rodrigo Schmidt commented on HADOOP-6825:
-----------------------------------------

It also tests that an empty or partially completed FileStatus can be overwritten (problem with MAPREDUCE-1858). Simple and effective.

I think we should have at least some test for FileStatus inside hadoop-common.

At the moment, a complicated patch could obfuscate changes to FileStatus and break MAPREDUCE or HDFS.



> FileStatus needs unit tests
> ---------------------------
>
>                 Key: HADOOP-6825
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6825
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 0.22.0
>            Reporter: Rodrigo Schmidt
>            Assignee: Rodrigo Schmidt
>             Fix For: 0.22.0
>
>         Attachments: HADOOP-6825.0.patch
>
>
> We need some unit tests for FileStatus to prevent problems like those we recently had on HADOOP-6796 and MAPREDUCE-1858.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9244-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 24 01:32:13 2010
Return-Path: <common-issues-return-9244-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 44265 invoked from network); 24 Jun 2010 01:32:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Jun 2010 01:32:13 -0000
Received: (qmail 3460 invoked by uid 500); 24 Jun 2010 01:32:13 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 3434 invoked by uid 500); 24 Jun 2010 01:32:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 3426 invoked by uid 99); 24 Jun 2010 01:32:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Jun 2010 01:32:12 +0000
X-ASF-Spam-Status: No, hits=-1541.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Jun 2010 01:32:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5O1VpwP008825
	for <common-issues@hadoop.apache.org>; Thu, 24 Jun 2010 01:31:51 GMT
Message-ID: <26170675.30961277343111164.JavaMail.jira@thor>
Date: Wed, 23 Jun 2010 21:31:51 -0400 (EDT)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6584) Provide Kerberized SSL encryption
 for webservices
In-Reply-To: <713706873.424711266804869916.JavaMail.jira@brutus.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881997#action_12881997 ] 

Hadoop QA commented on HADOOP-6584:
-----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12447891/c6584-02.patch
  against trunk revision 957074.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 javadoc.  The javadoc tool appears to have generated 1 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/594/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/594/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/594/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/594/console

This message is automatically generated.

> Provide Kerberized SSL encryption for webservices
> -------------------------------------------------
>
>                 Key: HADOOP-6584
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6584
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>         Attachments: c6584-02.patch, HADOOP-6584-FixJavadoc-Y20.patch, HADOOP-6584-Y20-1.patch, HADOOP-6584-Y20-2.patch, HADOOP-6584-Y20-3.patch, HADOOP-6584-Y20-4.patch, HADOOP-6584.patch
>
>
> Some web services should be authenticated via SSL backed by Kerberos, both to provide cross-cluster secure communication and to provide intra-cluster server authentication for services such as the {Name,SecondaryName,Backup,Checkpoint}node's image transfer and balancer.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9245-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 24 04:04:19 2010
Return-Path: <common-issues-return-9245-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69843 invoked from network); 24 Jun 2010 04:04:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Jun 2010 04:04:19 -0000
Received: (qmail 66060 invoked by uid 500); 24 Jun 2010 04:04:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 65804 invoked by uid 500); 24 Jun 2010 04:04:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 65794 invoked by uid 99); 24 Jun 2010 04:04:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Jun 2010 04:04:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Jun 2010 04:04:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5O43n1X010005
	for <common-issues@hadoop.apache.org>; Thu, 24 Jun 2010 04:03:50 GMT
Message-ID: <3855828.32191277352229918.JavaMail.jira@thor>
Date: Thu, 24 Jun 2010 00:03:49 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6838) Investigate Eclipse API Tools for
 enforcing or reporting on API compatibility
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Investigate Eclipse API Tools for enforcing or reporting on API compatibility 
------------------------------------------------------------------------------

                 Key: HADOOP-6838
                 URL: https://issues.apache.org/jira/browse/HADOOP-6838
             Project: Hadoop Common
          Issue Type: Improvement
          Components: build, documentation
            Reporter: Tom White


Some links:
* http://eclipsesource.com/blogs/2010/06/14/api_tools_top_eclipse_helios_feature_8/
* http://wiki.eclipse.org/PDE/API_Tools

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9246-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 24 16:37:16 2010
Return-Path: <common-issues-return-9246-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 75326 invoked from network); 24 Jun 2010 16:37:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Jun 2010 16:37:16 -0000
Received: (qmail 1642 invoked by uid 500); 24 Jun 2010 16:37:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1594 invoked by uid 500); 24 Jun 2010 16:37:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1586 invoked by uid 99); 24 Jun 2010 16:37:14 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Jun 2010 16:37:14 +0000
X-ASF-Spam-Status: No, hits=-1542.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Jun 2010 16:37:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5OGasB7016825
	for <common-issues@hadoop.apache.org>; Thu, 24 Jun 2010 16:36:54 GMT
Message-ID: <17420960.41391277397414299.JavaMail.jira@thor>
Date: Thu, 24 Jun 2010 12:36:54 -0400 (EDT)
From: "Scott Carey (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882214#action_12882214 ] 

Scott Carey commented on HADOOP-6837:
-------------------------------------

Isn't there a new variant of LZMA (file extension xz) that uses LZMA2 and is block based (and therefore splittable)?  We should definitely make sure that is the variant we want to support.

LZMA is slower than gzip, but compresses better than both bzip2 and gzip.  It is also optimized for fast decompression -- it decompresses significantly faster than bzip2 (but not as fast as gzip).

This link is useful for understanding the performance / compression ratio differences across the various compression levels provided for each:

http://tukaani.org/lzma/benchmarks.html


LZO, FastLZ, LZF, and the like are all faster than the above three but compress at a lower ratio.  With LZMA support (hopefully .xz files, not the older 7zip) there is little reason to use bzip2 anymore -- lzma level 2 compresses as fast as bzip2 level 1, but has a compression ratio as high as bzip2 level 9.  lzma always decompresses 2 to 7 times as fast as bzip2 (only ~ half the decompression speed of gzip). 

It is the ideal archival storage format.   

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9247-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jun 24 18:01:31 2010
Return-Path: <common-issues-return-9247-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 97546 invoked from network); 24 Jun 2010 18:01:30 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 24 Jun 2010 18:01:30 -0000
Received: (qmail 36879 invoked by uid 500); 24 Jun 2010 18:01:30 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36836 invoked by uid 500); 24 Jun 2010 18:01:30 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36828 invoked by uid 99); 24 Jun 2010 18:01:29 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Jun 2010 18:01:29 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Jun 2010 18:01:27 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5OI16xw017828
	for <common-issues@hadoop.apache.org>; Thu, 24 Jun 2010 18:01:06 GMT
Message-ID: <1099594.42861277402466415.JavaMail.jira@thor>
Date: Thu, 24 Jun 2010 14:01:06 -0400 (EDT)
From: "Nicholas Carlini (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6837) Support for LZMA compression
In-Reply-To: <33241912.25981277330209915.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882250#action_12882250 ] 

Nicholas Carlini commented on HADOOP-6837:
------------------------------------------

The Java code from the SDK hasn't been updated since version 4.61 (which is as of 23 November, 2008), so support for LZMA2 would need to rely on C code, or be ported to Java. 

The compression ratios of LZMA and LZMA2 are nearly identical (+/- .01% from the tests I did). It does look like LZMA2 is block based and is splittable, so that would be a major plus for it.

On the differences between LZMA and LZMA2:

&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; LZMA2 is an extension on top of the original LZMA. LZMA2 uses
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; LZMA internally, but adds support for flushing the encoder,
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; uncompressed chunks, eases stateful decoder implementations,
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; and improves support for multithreading.

http://tukaani.org/xz/xz-file-format.txt

I did have to add support for flushing the encoder to the Java code (flushing the encoder still produces valid lzma-compressed output).

> Support for LZMA compression
> ----------------------------
>
>                 Key: HADOOP-6837
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6837
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Nicholas Carlini
>            Assignee: Nicholas Carlini
>         Attachments: HADOOP-6837-lzma-java-20100623.patch
>
>
> Add support for LZMA (http://www.7-zip.org/sdk.html) compression, which generally achieves higher compression ratios than both gzip and bzip2.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9248-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 25 08:15:16 2010
Return-Path: <common-issues-return-9248-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35225 invoked from network); 25 Jun 2010 08:15:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Jun 2010 08:15:16 -0000
Received: (qmail 12321 invoked by uid 500); 25 Jun 2010 08:15:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12180 invoked by uid 500); 25 Jun 2010 08:15:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12112 invoked by uid 99); 25 Jun 2010 08:15:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 08:15:11 +0000
X-ASF-Spam-Status: No, hits=-1544.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 08:15:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5P8Eo7n024958
	for <common-issues@hadoop.apache.org>; Fri, 25 Jun 2010 08:14:51 GMT
Message-ID: <17739127.54771277453690897.JavaMail.jira@thor>
Date: Fri, 25 Jun 2010 04:14:50 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6836) [Herriot] Test Autoamtion : Generic
 method for adding/modifying the attributes for new configuration.
In-Reply-To: <23644117.9641277269730469.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinay Kumar Thota updated HADOOP-6836:
--------------------------------------

    Attachment:     (was: 6836-ydist-security.patch)

> [Herriot] Test Autoamtion : Generic method for adding/modifying the attributes for new configuration.
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6836
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6836
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>            Reporter: Iyappan Srinivasan
>            Assignee: Vinay Kumar Thota
>         Attachments: 6836-ydist-security.patch, HADOOP-6836.patch
>
>
> HADOP-6772 deals with 
> Common utilities for system tests.
> 1. A method for restarting the daemon with new configuration.
> c throws Exception; 
> 2. A method for restarting the daemon with default configuration.
> public void restart() throws Exception;
> 3. A method for waiting until daemon is stop.
> public void waitForClusterToStop() throws Exception;
> In this  some variables are made of String, instead of Long. Those needs ot be changed too. So, can this method 
> "public static void restartClusterWithNewConfig(Hashtable<String,Long> props, String confFile) throws Exception;"
> be generalized to accepts string too. All otehr methods should work as usual.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9249-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 25 08:15:18 2010
Return-Path: <common-issues-return-9249-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 35249 invoked from network); 25 Jun 2010 08:15:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Jun 2010 08:15:17 -0000
Received: (qmail 12368 invoked by uid 500); 25 Jun 2010 08:15:17 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 12248 invoked by uid 500); 25 Jun 2010 08:15:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 12126 invoked by uid 99); 25 Jun 2010 08:15:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 08:15:12 +0000
X-ASF-Spam-Status: No, hits=-1544.3 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 08:15:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5P8EpZr024976
	for <common-issues@hadoop.apache.org>; Fri, 25 Jun 2010 08:14:51 GMT
Message-ID: <27418684.54831277453691901.JavaMail.jira@thor>
Date: Fri, 25 Jun 2010 04:14:51 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6836) [Herriot] Test Autoamtion : Generic
 method for adding/modifying the attributes for new configuration.
In-Reply-To: <23644117.9641277269730469.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinay Kumar Thota updated HADOOP-6836:
--------------------------------------

    Attachment: 6836-ydist-security.patch
                HADOOP-6836.patch

Patches for both trunk and yahoo dist security branch.

> [Herriot] Test Autoamtion : Generic method for adding/modifying the attributes for new configuration.
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6836
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6836
>             Project: Hadoop Common
>          Issue Type: Test
>          Components: test
>            Reporter: Iyappan Srinivasan
>            Assignee: Vinay Kumar Thota
>         Attachments: 6836-ydist-security.patch, HADOOP-6836.patch
>
>
> HADOP-6772 deals with 
> Common utilities for system tests.
> 1. A method for restarting the daemon with new configuration.
> c throws Exception; 
> 2. A method for restarting the daemon with default configuration.
> public void restart() throws Exception;
> 3. A method for waiting until daemon is stop.
> public void waitForClusterToStop() throws Exception;
> In this  some variables are made of String, instead of Long. Those needs ot be changed too. So, can this method 
> "public static void restartClusterWithNewConfig(Hashtable<String,Long> props, String confFile) throws Exception;"
> be generalized to accepts string too. All otehr methods should work as usual.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9250-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 25 08:25:18 2010
Return-Path: <common-issues-return-9250-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 37335 invoked from network); 25 Jun 2010 08:25:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Jun 2010 08:25:18 -0000
Received: (qmail 21514 invoked by uid 500); 25 Jun 2010 08:25:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 21387 invoked by uid 500); 25 Jun 2010 08:25:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 20496 invoked by uid 99); 25 Jun 2010 08:25:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 08:25:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 08:25:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5P8OoZO025033
	for <common-issues@hadoop.apache.org>; Fri, 25 Jun 2010 08:24:50 GMT
Message-ID: <26284266.54901277454290297.JavaMail.jira@thor>
Date: Fri, 25 Jun 2010 04:24:50 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6839) [Herriot] Implement a functionality
 for getting the user list for creating proxy users.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

[Herriot] Implement a functionality for getting the user list for creating proxy users.
---------------------------------------------------------------------------------------

                 Key: HADOOP-6839
                 URL: https://issues.apache.org/jira/browse/HADOOP-6839
             Project: Hadoop Common
          Issue Type: Task
          Components: test
            Reporter: Vinay Kumar Thota
            Assignee: Vinay Kumar Thota


Develop a new method for getting the user list. 

Method signature is 
public ArrayList<String> getHadoopMultiUsersList() throws IOException;
Add new attribute in system-test.xml  file for getting userlist path.


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9251-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 25 09:00:20 2010
Return-Path: <common-issues-return-9251-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 46990 invoked from network); 25 Jun 2010 09:00:20 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Jun 2010 09:00:20 -0000
Received: (qmail 52957 invoked by uid 500); 25 Jun 2010 09:00:20 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 52418 invoked by uid 500); 25 Jun 2010 09:00:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 52241 invoked by uid 99); 25 Jun 2010 09:00:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 09:00:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 09:00:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5P8xpXd025304
	for <common-issues@hadoop.apache.org>; Fri, 25 Jun 2010 08:59:51 GMT
Message-ID: <24609964.55211277456391582.JavaMail.jira@thor>
Date: Fri, 25 Jun 2010 04:59:51 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6839) [Herriot] Implement a functionality
 for getting the user list for creating proxy users.
In-Reply-To: <26284266.54901277454290297.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinay Kumar Thota updated HADOOP-6839:
--------------------------------------

    Attachment: 6839-ydist-security.patch

Patch for yahoo dist security branch.

> [Herriot] Implement a functionality for getting the user list for creating proxy users.
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6839
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6839
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6839-ydist-security.patch
>
>
> Develop a new method for getting the user list. 
> Method signature is 
> public ArrayList<String> getHadoopMultiUsersList() throws IOException;
> Add new attribute in system-test.xml  file for getting userlist path.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9252-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 25 09:10:18 2010
Return-Path: <common-issues-return-9252-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 52950 invoked from network); 25 Jun 2010 09:10:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Jun 2010 09:10:18 -0000
Received: (qmail 63464 invoked by uid 500); 25 Jun 2010 09:10:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 63304 invoked by uid 500); 25 Jun 2010 09:10:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 63291 invoked by uid 99); 25 Jun 2010 09:10:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 09:10:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 09:10:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5P99oBs025431
	for <common-issues@hadoop.apache.org>; Fri, 25 Jun 2010 09:09:50 GMT
Message-ID: <33105676.55301277456990336.JavaMail.jira@thor>
Date: Fri, 25 Jun 2010 05:09:50 -0400 (EDT)
From: "Vinay Kumar Thota (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6839) [Herriot] Implement a functionality
 for getting the user list for creating proxy users.
In-Reply-To: <26284266.54901277454290297.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vinay Kumar Thota updated HADOOP-6839:
--------------------------------------

    Attachment: HADOOP-6839.patch

Patch for trunk.

> [Herriot] Implement a functionality for getting the user list for creating proxy users.
> ---------------------------------------------------------------------------------------
>
>                 Key: HADOOP-6839
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6839
>             Project: Hadoop Common
>          Issue Type: Task
>          Components: test
>            Reporter: Vinay Kumar Thota
>            Assignee: Vinay Kumar Thota
>         Attachments: 6839-ydist-security.patch, HADOOP-6839.patch
>
>
> Develop a new method for getting the user list. 
> Method signature is 
> public ArrayList<String> getHadoopMultiUsersList() throws IOException;
> Add new attribute in system-test.xml  file for getting userlist path.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9253-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 25 18:40:21 2010
Return-Path: <common-issues-return-9253-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 48719 invoked from network); 25 Jun 2010 18:40:21 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Jun 2010 18:40:21 -0000
Received: (qmail 81560 invoked by uid 500); 25 Jun 2010 18:40:21 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81472 invoked by uid 500); 25 Jun 2010 18:40:20 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 81464 invoked by uid 99); 25 Jun 2010 18:40:20 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 18:40:20 +0000
X-ASF-Spam-Status: No, hits=-1545.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 18:40:19 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5PIdxF1021451
	for <common-issues@hadoop.apache.org>; Fri, 25 Jun 2010 18:39:59 GMT
Message-ID: <30801793.63301277491199317.JavaMail.jira@thor>
Date: Fri, 25 Jun 2010 14:39:59 -0400 (EDT)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882674#action_12882674 ] 

Chris Douglas commented on HADOOP-6835:
---------------------------------------

bq. Currently the new BuiltInGzipDecompressor class inherits directly from JDK Inflater, but I suspect I should extend BuiltInZlibInflater instead.

I'd lean the other way. It's not really a supertype of {{BuiltInZlibInflater}} and neither are public types. It's not really an {{Inflater}}, either; it may be worth either supporting that interface or using an {{Inflater}} member rather than inheritance, since it only calls public methods. Either way, it's an existing confusion in the compression type hierarchy, and if it calls for any additional testing it can be ignored in this issue.

bq. Is it worthwhile to encapsulate the state label and associated variables into a private inner class (BuiltInGzipDecompressor.java, first FIXME comment)?

Since the code is already implemented and tested, refactoring it for a slightly cleaner implementation of a user-opaque, RFC-compliant library doesn't seem like a reasonable condition for committing it.

bq. DecompressorStream currently supports two concatenation modes via a pseudo-ifdef ("final boolean useResetPartially"): resetPartially(), which avoids any additional buffer copies at a cost of uglifying the Decompressor interface with this new method; or regular reset() + setInput() to recopy any "excess" bytes (that is, from stream N+1) at the end of stream N. The amount of recopying in the latter case is dependent on the buffer sizes (typically 64KB around here) and sizes of the concatenated gzip streams/members, but in general it won't be much. Barring strong disagreement, I'll go with the latter approach and clean up all the resetPartially() stuff in the next (hopefully final) version of the patch.

Agreed; the penalty for re-copying once per stream is light enough that it can be endured for other codecs' API considerations.

bq. Any last-minute qualms about hardcoding the concatenation behavior? It would simplify the patch slightly and seems to be the preferred approach, so that's my plan for the next version.

Sounds fine to me. It may cause faults in some containers, but those are probably bugs covered over by this one.

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>         Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9254-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 25 18:43:14 2010
Return-Path: <common-issues-return-9254-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 49246 invoked from network); 25 Jun 2010 18:43:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Jun 2010 18:43:14 -0000
Received: (qmail 84515 invoked by uid 500); 25 Jun 2010 18:43:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 84419 invoked by uid 500); 25 Jun 2010 18:43:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 84411 invoked by uid 99); 25 Jun 2010 18:43:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 18:43:13 +0000
X-ASF-Spam-Status: No, hits=-1545.5 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 18:43:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5PIgqoT021502
	for <common-issues@hadoop.apache.org>; Fri, 25 Jun 2010 18:42:52 GMT
Message-ID: <13791553.63451277491372612.JavaMail.jira@thor>
Date: Fri, 25 Jun 2010 14:42:52 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Greg Roelofs updated HADOOP-6835:
---------------------------------

    Attachment: HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch

Updated patch with the promised changes from my third and fourth questions (i.e., reset()/setInput() rather than resetPartially(), and hardcoded concat fix); cleaned up debug/FIXME stuff; and full test files (well, the shorter versions).

This is still against Yahoo's 0.20.2xx branch; I'll port it back to trunk next.

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>         Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9255-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 25 18:48:23 2010
Return-Path: <common-issues-return-9255-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 50319 invoked from network); 25 Jun 2010 18:48:23 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Jun 2010 18:48:23 -0000
Received: (qmail 88362 invoked by uid 500); 25 Jun 2010 18:48:23 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 88325 invoked by uid 500); 25 Jun 2010 18:48:22 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 88317 invoked by uid 99); 25 Jun 2010 18:48:22 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 18:48:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 18:48:20 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5PIlwST021604
	for <common-issues@hadoop.apache.org>; Fri, 25 Jun 2010 18:47:58 GMT
Message-ID: <11124719.63761277491678886.JavaMail.jira@thor>
Date: Fri, 25 Jun 2010 14:47:58 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882678#action_12882678 ] 

Greg Roelofs commented on HADOOP-6835:
--------------------------------------

Whoops, ships passing...

bq. It's not really an Inflater, either; it may be worth either supporting that interface or using an Inflater member rather than inheritance, since it only calls public methods.

Good point.  OK, I'll look into that first, _then_ port to trunk.  Seems like a pretty lightweight change.

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>         Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9256-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 25 20:08:16 2010
Return-Path: <common-issues-return-9256-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 73917 invoked from network); 25 Jun 2010 20:08:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Jun 2010 20:08:16 -0000
Received: (qmail 82891 invoked by uid 500); 25 Jun 2010 20:08:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 82838 invoked by uid 500); 25 Jun 2010 20:08:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 82829 invoked by uid 99); 25 Jun 2010 20:08:15 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 20:08:15 +0000
X-ASF-Spam-Status: No, hits=-1545.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 20:08:14 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5PK7s1F022359
	for <common-issues@hadoop.apache.org>; Fri, 25 Jun 2010 20:07:54 GMT
Message-ID: <13433706.65271277496474337.JavaMail.jira@thor>
Date: Fri, 25 Jun 2010 16:07:54 -0400 (EDT)
From: "Greg Roelofs (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6835) Support concatenated gzip and bzip2
 files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Greg Roelofs updated HADOOP-6835:
---------------------------------

    Attachment: HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch

Final (I think) version against Yahoo 0.20.2xx branch.  Trunk version coming soon.

> Support concatenated gzip and bzip2 files
> -----------------------------------------
>
>                 Key: HADOOP-6835
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6835
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io
>            Reporter: Tom White
>            Assignee: Greg Roelofs
>         Attachments: grr-hadoop-common.dif.20100614c, grr-hadoop-mapreduce.dif.20100614c, HADOOP-6835.v3.yahoo-0.20.2xx-branch.patch, HADOOP-6835.v4.yahoo-0.20.2xx-branch.patch, MR-469.v2.yahoo-0.20.2xx-branch.patch
>
>
> When running MapReduce with concatenated gzip files as input only the first part is read, which is confusing, to say the least. Concatenated gzip is described in http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage and in http://www.ietf.org/rfc/rfc1952.txt. (See original report at http://www.nabble.com/Problem-with-Hadoop-and-concatenated-gzip-files-to21383097.html)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9257-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 25 22:07:15 2010
Return-Path: <common-issues-return-9257-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 99043 invoked from network); 25 Jun 2010 22:07:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Jun 2010 22:07:15 -0000
Received: (qmail 4146 invoked by uid 500); 25 Jun 2010 22:07:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 4079 invoked by uid 500); 25 Jun 2010 22:07:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 4071 invoked by uid 99); 25 Jun 2010 22:07:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 22:07:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 22:07:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5PM6oKI023301
	for <common-issues@hadoop.apache.org>; Fri, 25 Jun 2010 22:06:50 GMT
Message-ID: <30089846.67411277503610667.JavaMail.jira@thor>
Date: Fri, 25 Jun 2010 18:06:50 -0400 (EDT)
From: "Nicolas Spiegelberg (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6840) Support non-recursive create() in
 SequenceFile.Writer
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Support non-recursive create() in SequenceFile.Writer
-----------------------------------------------------

                 Key: HADOOP-6840
                 URL: https://issues.apache.org/jira/browse/HADOOP-6840
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs, io
    Affects Versions: 0.21.0, 0.20-append
            Reporter: Nicolas Spiegelberg
            Priority: Minor


The proposed solution for HBASE-2312 requires the sequence file to handle a non-recursive create.  This is already supported by HDFS, but needs to have an equivalent SequenceFile.Writer API.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9258-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 25 22:44:12 2010
Return-Path: <common-issues-return-9258-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 5016 invoked from network); 25 Jun 2010 22:44:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Jun 2010 22:44:12 -0000
Received: (qmail 33777 invoked by uid 500); 25 Jun 2010 22:44:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 33713 invoked by uid 500); 25 Jun 2010 22:44:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 33705 invoked by uid 99); 25 Jun 2010 22:44:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 22:44:11 +0000
X-ASF-Spam-Status: No, hits=-1546.1 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 22:44:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5PMholX023568
	for <common-issues@hadoop.apache.org>; Fri, 25 Jun 2010 22:43:50 GMT
Message-ID: <1611179.68101277505830449.JavaMail.jira@thor>
Date: Fri, 25 Jun 2010 18:43:50 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6782) TestAvroRpc fails with avro-1.3.1 and
 avro-1.3.2
In-Reply-To: <28144169.6251274389637292.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6782:
------------------------------

    Fix Version/s: 0.21.0

I was hitting this in 0.21 so I've committed the patch there.

> TestAvroRpc fails with avro-1.3.1 and avro-1.3.2
> ------------------------------------------------
>
>                 Key: HADOOP-6782
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6782
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jitendra Nath Pandey
>            Assignee: Doug Cutting
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6782.patch
>
>
> I tried to upgrade Hadoop-common to use avro-1.3.1. TestAvroRpc fails with following exceptions.
> Same exception is seen with avro-1.3.2 as well.
> org.apache.avro.AvroRuntimeException: Not in union ["string"]: org.apache.avro.AvroRuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.ipc.TestAvroRpc$TestImpl.echo(java.lang.String)
>         at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:340)
>         at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
>         at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:116)
>         at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:55)
>         at org.apache.avro.specific.SpecificResponder.writeError(SpecificResponder.java:81)
>         at org.apache.avro.ipc.Responder.respond(Responder.java:137)
>         at org.apache.hadoop.ipc.AvroRpcEngine$TunnelResponder.call(AvroRpcEngine.java:183)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:342)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1350)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1346)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:738)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1344)
> 2010-05-20 13:50:39,227 INFO  ipc.Server (Server.java:run(1358)) - IPC Server handler 0 on 54672, call call(org.apache.hadoop.ipc.AvroRpcEngine$BufferListWritable@7bcd280b) from 127.0.0.1:54673: error: java.io.IOException: org.apache.avro.AvroRuntimeException: Not in union ["string"]: org.apache.avro.AvroRuntimeException: Not in union ["string"]: org.apache.avro.AvroRuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.ipc.TestAvroRpc$TestImpl.echo(java.lang.String)
> java.io.IOException: org.apache.avro.AvroRuntimeException: Not in union ["string"]: org.apache.avro.AvroRuntimeException: Not in union ["string"]: org.apache.avro.AvroRuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.ipc.TestAvroRpc$TestImpl.echo(java.lang.String)
>         at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:340)
>         at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
>         at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:116)
>         at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:55)
>         at org.apache.avro.specific.SpecificResponder.writeError(SpecificResponder.java:81)
>         at org.apache.avro.ipc.Responder.respond(Responder.java:146)
>         at org.apache.hadoop.ipc.AvroRpcEngine$TunnelResponder.call(AvroRpcEngine.java:183)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:342)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1350)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1346)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:738)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1344)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9259-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 25 23:54:15 2010
Return-Path: <common-issues-return-9259-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 18179 invoked from network); 25 Jun 2010 23:54:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 25 Jun 2010 23:54:15 -0000
Received: (qmail 76602 invoked by uid 500); 25 Jun 2010 23:54:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 76556 invoked by uid 500); 25 Jun 2010 23:54:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 76547 invoked by uid 99); 25 Jun 2010 23:54:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 23:54:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2010 23:54:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5PNroLg024111
	for <common-issues@hadoop.apache.org>; Fri, 25 Jun 2010 23:53:50 GMT
Message-ID: <21153318.69411277510030193.JavaMail.jira@thor>
Date: Fri, 25 Jun 2010 19:53:50 -0400 (EDT)
From: "Steve McCarthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6101) CDPATH environment variable causes
 bin scripts to fail.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882772#action_12882772 ] 

Steve McCarthy commented on HADOOP-6101:
----------------------------------------

I ran into this as well.  An alternate solution would be to change the lines in the scripts to 

bin=`cd "$bin" > /dev/null; pwd`

> CDPATH environment variable causes bin scripts to fail.
> -------------------------------------------------------
>
>                 Key: HADOOP-6101
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6101
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: scripts
>    Affects Versions: 0.19.1
>            Reporter: Phil Hagelberg
>            Priority: Minor
>
> Most of the scripts in bin/* assume that cd produces no output. But when using bash (and some other shells) cd will output the destination directory if the CDPATH environment variable is set. CDPATH is very useful, and it's unfortunate to have to unset it to use Hadoop.
> The offending line (in start-all.sh, though most of the scripts exhibit the problem) is:
>   bin=`cd "$bin"; pwd`
> Adding this to the top of each affected script will fix the problem:
>   unset CDPATH

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9260-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 26 00:04:13 2010
Return-Path: <common-issues-return-9260-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 19992 invoked from network); 26 Jun 2010 00:04:13 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jun 2010 00:04:13 -0000
Received: (qmail 79847 invoked by uid 500); 26 Jun 2010 00:04:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 79796 invoked by uid 500); 26 Jun 2010 00:04:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 79787 invoked by uid 99); 26 Jun 2010 00:04:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Jun 2010 00:04:12 +0000
X-ASF-Spam-Status: No, hits=-1546.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Jun 2010 00:04:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5Q03pgD024247
	for <common-issues@hadoop.apache.org>; Sat, 26 Jun 2010 00:03:51 GMT
Message-ID: <19677123.69731277510631681.JavaMail.jira@thor>
Date: Fri, 25 Jun 2010 20:03:51 -0400 (EDT)
From: "Steve McCarthy (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6841) In Quick Start, mention requirement
 that HADOOP_HOME needs to be set
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

In Quick Start, mention requirement that HADOOP_HOME needs to be set
--------------------------------------------------------------------

                 Key: HADOOP-6841
                 URL: https://issues.apache.org/jira/browse/HADOOP-6841
             Project: Hadoop Common
          Issue Type: Improvement
          Components: documentation
    Affects Versions: 0.20.2
            Reporter: Steve McCarthy
            Priority: Minor


Ref:  http://hadoop.apache.org/common/docs/r0.20.0/quickstart.html#Local
Ref: http://wiki.apache.org/hadoop/QuickStart

As a person new to Hadoop, I was walking through the Quick Start pages referenced above.  I got stuck on the first example and had to go digging through the scripts to figure out why it wasn't working for me.

It turns out that I didn't have the HADOOP_HOME environment variable set.  It would have saved a lot of time if the instructions mentioned setting this variable.  

Thanks.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9261-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 26 00:27:12 2010
Return-Path: <common-issues-return-9261-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 32514 invoked from network); 26 Jun 2010 00:27:11 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jun 2010 00:27:11 -0000
Received: (qmail 86289 invoked by uid 500); 26 Jun 2010 00:27:11 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 86247 invoked by uid 500); 26 Jun 2010 00:27:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 86239 invoked by uid 99); 26 Jun 2010 00:27:10 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Jun 2010 00:27:10 +0000
X-ASF-Spam-Status: No, hits=-1546.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Jun 2010 00:27:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5Q0QoLg024373
	for <common-issues@hadoop.apache.org>; Sat, 26 Jun 2010 00:26:50 GMT
Message-ID: <22163171.70021277512010125.JavaMail.jira@thor>
Date: Fri, 25 Jun 2010 20:26:50 -0400 (EDT)
From: "Nicolas Spiegelberg (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6840) Support non-recursive create() in
 FileSystem & SequenceFile.Writer
In-Reply-To: <30089846.67411277503610667.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicolas Spiegelberg updated HADOOP-6840:
----------------------------------------

        Summary: Support non-recursive create() in FileSystem & SequenceFile.Writer  (was: Support non-recursive create() in SequenceFile.Writer)
    Description: The proposed solution for HBASE-2312 requires the sequence file to handle a non-recursive create.  This is already supported by HDFS, but needs to have an equivalent FileSystem & SequenceFile.Writer API.  (was: The proposed solution for HBASE-2312 requires the sequence file to handle a non-recursive create.  This is already supported by HDFS, but needs to have an equivalent SequenceFile.Writer API.)

> Support non-recursive create() in FileSystem & SequenceFile.Writer
> ------------------------------------------------------------------
>
>                 Key: HADOOP-6840
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6840
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs, io
>    Affects Versions: 0.21.0, 0.20-append
>            Reporter: Nicolas Spiegelberg
>            Priority: Minor
>
> The proposed solution for HBASE-2312 requires the sequence file to handle a non-recursive create.  This is already supported by HDFS, but needs to have an equivalent FileSystem & SequenceFile.Writer API.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9262-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 26 01:02:18 2010
Return-Path: <common-issues-return-9262-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36422 invoked from network); 26 Jun 2010 01:02:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jun 2010 01:02:18 -0000
Received: (qmail 1250 invoked by uid 500); 26 Jun 2010 01:02:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 1163 invoked by uid 500); 26 Jun 2010 01:02:17 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 1155 invoked by uid 99); 26 Jun 2010 01:02:17 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Jun 2010 01:02:17 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Jun 2010 01:02:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5Q11rDO024765
	for <common-issues@hadoop.apache.org>; Sat, 26 Jun 2010 01:01:53 GMT
Message-ID: <26395896.70981277514113890.JavaMail.jira@thor>
Date: Fri, 25 Jun 2010 21:01:53 -0400 (EDT)
From: "Steven Wong (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6842) "hadoop fs -text" does not give a
 useful text representation of MapWritable objects
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

"hadoop fs -text" does not give a useful text representation of MapWritable objects
-----------------------------------------------------------------------------------

                 Key: HADOOP-6842
                 URL: https://issues.apache.org/jira/browse/HADOOP-6842
             Project: Hadoop Common
          Issue Type: Improvement
    Affects Versions: 0.20.0
            Reporter: Steven Wong


If a sequence file contains MapWritable objects, running "hadoop fs -text" on the file prints the following for each MapWritable:

org.apache.hadoop.io.MapWritable@4f8235ed

To be more useful, it should print out the contents of the map instead. This can be done by adding a toString method to MapWritable, i.e. something like:

public String toString() {
    return (new TreeMap<Writable, Writable>(instance)).toString();
}


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9263-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 26 01:57:15 2010
Return-Path: <common-issues-return-9263-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 45976 invoked from network); 26 Jun 2010 01:57:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jun 2010 01:57:14 -0000
Received: (qmail 18061 invoked by uid 500); 26 Jun 2010 01:57:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 18001 invoked by uid 500); 26 Jun 2010 01:57:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 17985 invoked by uid 99); 26 Jun 2010 01:57:13 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Jun 2010 01:57:13 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Jun 2010 01:57:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5Q1uoqv025130
	for <common-issues@hadoop.apache.org>; Sat, 26 Jun 2010 01:56:50 GMT
Message-ID: <12452270.71901277517410023.JavaMail.jira@thor>
Date: Fri, 25 Jun 2010 21:56:50 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6843) Entries in the FileSystem's Cache
 could be cleared when they are not used
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org

Entries in the FileSystem's Cache could be cleared when they are not used
-------------------------------------------------------------------------

                 Key: HADOOP-6843
                 URL: https://issues.apache.org/jira/browse/HADOOP-6843
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Devaraj Das
             Fix For: 0.22.0


In FileSystem, there is a cache maintained for Filesystem instances. The entries in the cache are cleared only when explicit FileSystem.close is invoked. Applications are not careful on this aspect. Typically, they do FileSystem.get(), operate on the FileSystem, and then they just forget about it. Every FileSystem instance stores a reference to the Configuration object that it was created with. Over a period of time, as the cache grows, this can lead to OOM (we have seen this happening in our hadoop 20S clusters at Yahoo).

This jira aims at addressing the above issue.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9264-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 26 04:19:19 2010
Return-Path: <common-issues-return-9264-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69253 invoked from network); 26 Jun 2010 04:19:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jun 2010 04:19:19 -0000
Received: (qmail 48967 invoked by uid 500); 26 Jun 2010 04:19:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 48842 invoked by uid 500); 26 Jun 2010 04:19:16 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 48834 invoked by uid 99); 26 Jun 2010 04:19:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Jun 2010 04:19:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Jun 2010 04:19:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5Q4Ip9H026036
	for <common-issues@hadoop.apache.org>; Sat, 26 Jun 2010 04:18:51 GMT
Message-ID: <6358760.72471277525931090.JavaMail.jira@thor>
Date: Sat, 26 Jun 2010 00:18:51 -0400 (EDT)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6843) Entries in the FileSystem's Cache
 could be cleared when they are not used
In-Reply-To: <12452270.71901277517410023.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-6843:
--------------------------------

    Attachment: fs-weak-ref.4.patch

An approach to the above problem is to make the value in the map (used to implement the cache) a soft reference. Then when no one is referencing a particular FileSystem instance, and the memory is under pressure, the GC can delete the entry (and unless the Configuration object is referenced from elsewhere, that would go away too). 

Patch for Y20S (not for commit).

> Entries in the FileSystem's Cache could be cleared when they are not used
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6843
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6843
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: fs-weak-ref.4.patch
>
>
> In FileSystem, there is a cache maintained for Filesystem instances. The entries in the cache are cleared only when explicit FileSystem.close is invoked. Applications are not careful on this aspect. Typically, they do FileSystem.get(), operate on the FileSystem, and then they just forget about it. Every FileSystem instance stores a reference to the Configuration object that it was created with. Over a period of time, as the cache grows, this can lead to OOM (we have seen this happening in our hadoop 20S clusters at Yahoo).
> This jira aims at addressing the above issue.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9265-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 26 04:24:17 2010
Return-Path: <common-issues-return-9265-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 69624 invoked from network); 26 Jun 2010 04:24:17 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jun 2010 04:24:17 -0000
Received: (qmail 49486 invoked by uid 500); 26 Jun 2010 04:24:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 49399 invoked by uid 500); 26 Jun 2010 04:24:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 49391 invoked by uid 99); 26 Jun 2010 04:24:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Jun 2010 04:24:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Jun 2010 04:24:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5Q4Noei026055
	for <common-issues@hadoop.apache.org>; Sat, 26 Jun 2010 04:23:50 GMT
Message-ID: <13591390.72501277526230185.JavaMail.jira@thor>
Date: Sat, 26 Jun 2010 00:23:50 -0400 (EDT)
From: "Tsz Wo (Nicholas), SZE (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6843) Entries in the FileSystem's Cache
 could be cleared when they are not used
In-Reply-To: <12452270.71901277517410023.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo (Nicholas), SZE updated HADOOP-6843:
-------------------------------------------

    Attachment: TestFileSystemCache.java

TestFileSystemCache.java: file system cache test 

> Entries in the FileSystem's Cache could be cleared when they are not used
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6843
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6843
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Devaraj Das
>             Fix For: 0.22.0
>
>         Attachments: fs-weak-ref.4.patch, TestFileSystemCache.java
>
>
> In FileSystem, there is a cache maintained for Filesystem instances. The entries in the cache are cleared only when explicit FileSystem.close is invoked. Applications are not careful on this aspect. Typically, they do FileSystem.get(), operate on the FileSystem, and then they just forget about it. Every FileSystem instance stores a reference to the Configuration object that it was created with. Over a period of time, as the cache grows, this can lead to OOM (we have seen this happening in our hadoop 20S clusters at Yahoo).
> This jira aims at addressing the above issue.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9266-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 26 10:27:16 2010
Return-Path: <common-issues-return-9266-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33224 invoked from network); 26 Jun 2010 10:27:15 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jun 2010 10:27:15 -0000
Received: (qmail 70393 invoked by uid 500); 26 Jun 2010 10:27:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 70269 invoked by uid 500); 26 Jun 2010 10:27:13 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 70261 invoked by uid 99); 26 Jun 2010 10:27:12 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Jun 2010 10:27:12 +0000
X-ASF-Spam-Status: No, hits=-1546.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Jun 2010 10:27:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5QAQp34027566
	for <common-issues@hadoop.apache.org>; Sat, 26 Jun 2010 10:26:51 GMT
Message-ID: <11083731.73801277548011465.JavaMail.jira@thor>
Date: Sat, 26 Jun 2010 06:26:51 -0400 (EDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6800) Harmonize JAR library versions
In-Reply-To: <8215141.145191275515876547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hemanth Yamijala updated HADOOP-6800:
-------------------------------------

    Attachment: jars-in-lib.dump.merged
                ivy-declarations.dump.merged

Tom, I applied the latest patches for all three sub-projects, cleared my IVY cache, rebuilt all of them. Then, I compared the ivy library dependencies that we are declaring in the sources of the three packages and the libraries in the lib folder of the built packages. In both analyses, I still see some duplicate jars. I think most or all of the duplicates in source are coming from the HDFS Proxy ivy file. I don't know if this is reason why the libraries in the build/lib are ultimately repeating.

For e.g. here's a list of the jars in the ivy properties that have different versions:
commons-logging-api, commons-logging, ivy, jets3t, junit, mockito-all, slf4j-api, slf4j-log4j12

There are quite a few repetitions in the lib folders of the three packages:
ant,avro ,commons-http-client,commons-lang,ivy,jackson-core-asl,jackson-mapper-asl,jetty,jetty-util,junit,paranamer,paranamer-ant,paranamer-generator,qdox,servlet-api,slf4j-api,slf4j-log4j12

I am attaching two text files that are the dumps of the ivy library declarations and the build/lib folder contents for your review.

Please note that I did not run the cluster with these packages - so things are possibly fine with these duplications as well.

> Harmonize JAR library versions
> ------------------------------
>
>                 Key: HADOOP-6800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6800.patch, HADOOP-6800.patch, HADOOP-6800.patch, ivy-declarations.dump.merged, jars-in-lib.dump.merged
>
>
> Currently, multiple versions of the same library JAR are being pulled in for various reasons.
> * Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
> * Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9267-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 26 10:32:22 2010
Return-Path: <common-issues-return-9267-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 33742 invoked from network); 26 Jun 2010 10:32:22 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jun 2010 10:32:22 -0000
Received: (qmail 73089 invoked by uid 500); 26 Jun 2010 10:32:22 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 72888 invoked by uid 500); 26 Jun 2010 10:32:19 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 72844 invoked by uid 99); 26 Jun 2010 10:32:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Jun 2010 10:32:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Jun 2010 10:32:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5QAVsGb027584
	for <common-issues@hadoop.apache.org>; Sat, 26 Jun 2010 10:31:54 GMT
Message-ID: <11525185.73841277548314252.JavaMail.jira@thor>
Date: Sat, 26 Jun 2010 06:31:54 -0400 (EDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6800) Harmonize JAR library versions
In-Reply-To: <8215141.145191275515876547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882831#action_12882831 ] 

Hemanth Yamijala commented on HADOOP-6800:
------------------------------------------

One more point is that some of the subprojects in mapred seem to be using hadoop-common-*-SNAPSHOT as the dependency and som hadoop-common-*-dev-SNAPSHOT as the dependency. However, this does not seem to be a problem as far as the libraries in the packages are concerned.

> Harmonize JAR library versions
> ------------------------------
>
>                 Key: HADOOP-6800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6800.patch, HADOOP-6800.patch, HADOOP-6800.patch, ivy-declarations.dump.merged, jars-in-lib.dump.merged
>
>
> Currently, multiple versions of the same library JAR are being pulled in for various reasons.
> * Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
> * Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9268-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 26 15:20:13 2010
Return-Path: <common-issues-return-9268-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 87494 invoked from network); 26 Jun 2010 15:20:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jun 2010 15:20:12 -0000
Received: (qmail 87512 invoked by uid 500); 26 Jun 2010 15:20:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 87416 invoked by uid 500); 26 Jun 2010 15:20:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 87408 invoked by uid 99); 26 Jun 2010 15:20:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Jun 2010 15:20:11 +0000
X-ASF-Spam-Status: No, hits=-1546.8 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Jun 2010 15:20:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5QFJopO028521
	for <common-issues@hadoop.apache.org>; Sat, 26 Jun 2010 15:19:50 GMT
Message-ID: <1524367.74571277565590605.JavaMail.jira@thor>
Date: Sat, 26 Jun 2010 11:19:50 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6800) Harmonize JAR library versions
In-Reply-To: <8215141.145191275515876547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882842#action_12882842 ] 

Tom White commented on HADOOP-6800:
-----------------------------------

Hemanth, Thanks for running through this. Did you run ant with "mvn-install -Dresolvers=internal" options? When I do a clean build I don't get any of the duplicates you mention.

> Harmonize JAR library versions
> ------------------------------
>
>                 Key: HADOOP-6800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6800.patch, HADOOP-6800.patch, HADOOP-6800.patch, ivy-declarations.dump.merged, jars-in-lib.dump.merged
>
>
> Currently, multiple versions of the same library JAR are being pulled in for various reasons.
> * Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
> * Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9269-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 26 16:53:16 2010
Return-Path: <common-issues-return-9269-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 529 invoked from network); 26 Jun 2010 16:53:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jun 2010 16:53:16 -0000
Received: (qmail 36783 invoked by uid 500); 26 Jun 2010 16:53:16 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 36734 invoked by uid 500); 26 Jun 2010 16:53:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 36725 invoked by uid 99); 26 Jun 2010 16:53:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Jun 2010 16:53:15 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Jun 2010 16:53:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5QGqpjs028876
	for <common-issues@hadoop.apache.org>; Sat, 26 Jun 2010 16:52:52 GMT
Message-ID: <26173347.74981277571171922.JavaMail.jira@thor>
Date: Sat, 26 Jun 2010 12:52:51 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6800) Harmonize JAR library versions
In-Reply-To: <8215141.145191275515876547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882852#action_12882852 ] 

Tom White commented on HADOOP-6800:
-----------------------------------

Other things I did - cleared the local maven cache, and used the "veryclean" target. The -verbose ant option is useful for seeing the dependencies ivy is pulling in.

> One more point is that some of the subprojects in mapred seem to be using hadoop-common-SNAPSHOT as the dependency and som hadoop-common-dev-SNAPSHOT as the dependency. However, this does not seem to be a problem as far as the libraries in the packages are concerned.

I agree this does not seem to be causing a problem here, but should be fixed in another JIRA.

> Harmonize JAR library versions
> ------------------------------
>
>                 Key: HADOOP-6800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6800.patch, HADOOP-6800.patch, HADOOP-6800.patch, ivy-declarations.dump.merged, jars-in-lib.dump.merged
>
>
> Currently, multiple versions of the same library JAR are being pulled in for various reasons.
> * Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
> * Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9270-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sat Jun 26 19:13:12 2010
Return-Path: <common-issues-return-9270-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34219 invoked from network); 26 Jun 2010 19:13:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 26 Jun 2010 19:13:12 -0000
Received: (qmail 979 invoked by uid 500); 26 Jun 2010 19:13:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 916 invoked by uid 500); 26 Jun 2010 19:13:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 890 invoked by uid 99); 26 Jun 2010 19:13:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Jun 2010 19:13:11 +0000
X-ASF-Spam-Status: No, hits=-1547.0 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 26 Jun 2010 19:13:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5QJCoc8029495
	for <common-issues@hadoop.apache.org>; Sat, 26 Jun 2010 19:12:50 GMT
Message-ID: <21925064.75971277579570526.JavaMail.jira@thor>
Date: Sat, 26 Jun 2010 15:12:50 -0400 (EDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6800) Harmonize JAR library versions
In-Reply-To: <8215141.145191275515876547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HADOOP-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882882#action_12882882 ] 

Hemanth Yamijala commented on HADOOP-6800:
------------------------------------------

Tom, I tried again:

- Cleared Ivy cache - essentially did an rm -rf ~/.ivy2/cache
- On common, did ant veryclean jar mvn-install -- Here, I tried with and without -Dresolvers=internal
- On hdfs, did ant veryclean jar mvn-install -Dresolvers=internal. This step fails compilation, with errors liks : 
org.apache.hadoop.hdfs.server.namenode.NameNode is not abstract and does not override abstract method refreshSuperUserGroupsConfiguration(org.apache.hadoop.conf.Configuration) in org.apache.hadoop.security.RefreshUserMappingsProtocol

These errors do not come when I build the projects without the mvn-install option. Is there anything I'm doing incorrectly ?

Also, please note that of the two sets of duplicates I mentioned, one set is in the ivy configuration we're defining. So, in HDFS proxy's ivy/libraries.properties, we say junit.version=3.8.2 whereas in hdfs core, we say this is 4.8.1. Similarly for other libraries mentioned in ivy-declarations.dump.merged. Shouldn't this be resolved ?

> Harmonize JAR library versions
> ------------------------------
>
>                 Key: HADOOP-6800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-6800.patch, HADOOP-6800.patch, HADOOP-6800.patch, ivy-declarations.dump.merged, jars-in-lib.dump.merged
>
>
> Currently, multiple versions of the same library JAR are being pulled in for various reasons.
> * Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
> * Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9271-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 27 05:33:15 2010
Return-Path: <common-issues-return-9271-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 65672 invoked from network); 27 Jun 2010 05:33:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jun 2010 05:33:14 -0000
Received: (qmail 7932 invoked by uid 500); 27 Jun 2010 05:33:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 7878 invoked by uid 500); 27 Jun 2010 05:33:12 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 7865 invoked by uid 99); 27 Jun 2010 05:33:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Jun 2010 05:33:11 +0000
X-ASF-Spam-Status: No, hits=-1547.2 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Jun 2010 05:33:11 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5R5Wo1T002060
	for <common-issues@hadoop.apache.org>; Sun, 27 Jun 2010 05:32:50 GMT
Message-ID: <33511221.78001277616770670.JavaMail.jira@thor>
Date: Sun, 27 Jun 2010 01:32:50 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6800) Harmonize JAR library versions
In-Reply-To: <8215141.145191275515876547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6800:
------------------------------

    Attachment: build.sh

Hemanth,

I used the following command to find clashes in libraries.properties. The only problematic one was hdfsproxy's libraries.properties, as you pointed out, and which I've updated in HDFS-1212.

{code}
find hadoop-{common,hdfs,mapreduce}-trunk/ivy hadoop-{common,hdfs,mapreduce}-trunk/src -name libraries.properties | xargs cat | sort | uniq
{code}

However, even before this change, when running the attached build.sh script I got distinct libraries. (I suspect that the ones in HDFS were getting upgraded due to other dependencies.) I don't see the compile problem you had (this was with SVN revision 958322). Perhaps you can try the build.sh script with the patches from this JIRA, HDFS-1212, and MAPREDUCE-1870 applied to trunk? Thanks!

> Harmonize JAR library versions
> ------------------------------
>
>                 Key: HADOOP-6800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: build.sh, HADOOP-6800.patch, HADOOP-6800.patch, HADOOP-6800.patch, ivy-declarations.dump.merged, jars-in-lib.dump.merged
>
>
> Currently, multiple versions of the same library JAR are being pulled in for various reasons.
> * Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
> * Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9272-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 27 18:12:16 2010
Return-Path: <common-issues-return-9272-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 24482 invoked from network); 27 Jun 2010 18:12:16 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jun 2010 18:12:16 -0000
Received: (qmail 68211 invoked by uid 500); 27 Jun 2010 18:12:15 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 68170 invoked by uid 500); 27 Jun 2010 18:12:15 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 68162 invoked by uid 99); 27 Jun 2010 18:12:14 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Jun 2010 18:12:14 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Jun 2010 18:12:12 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5RIBoNn010336
	for <common-issues@hadoop.apache.org>; Sun, 27 Jun 2010 18:11:50 GMT
Message-ID: <141783.80251277662310623.JavaMail.jira@thor>
Date: Sun, 27 Jun 2010 14:11:50 -0400 (EDT)
From: "Hemanth Yamijala (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-6800) Harmonize JAR library versions
In-Reply-To: <8215141.145191275515876547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882971#action_12882971 ] 

Hemanth Yamijala commented on HADOOP-6800:
------------------------------------------

Tom, I applied the new patch on HDFS-1212. This removed all the duplicates in the ivy declarations. When I ran the project builds (using mvn-install as you suggested), I now see only 2 repetitions - ivy-2.0.0-rc2 and ivy-2.1.0 and jsp-2.1.jar and jsp-2.1-6.1.14.jar. In both the cases, *both* libraries were present in HDFS and mapred. So, it is not like different versions present in mapred and HDFS, but even within the project.

I thought this was close enough and wanted to build a single distribution of the three packages and test it in pseudo-distributed mode. I was trying to use the tar-munge script, but get some errors like:
mv: will not overwrite just-created `conf/configuration.xsl' with `hdfs/conf/configuration.xsl'

I am still trying to resolve this, but I thought I'd update that except for the two jars I mention, everything else seems very clean.

> Harmonize JAR library versions
> ------------------------------
>
>                 Key: HADOOP-6800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: build.sh, HADOOP-6800.patch, HADOOP-6800.patch, HADOOP-6800.patch, ivy-declarations.dump.merged, jars-in-lib.dump.merged
>
>
> Currently, multiple versions of the same library JAR are being pulled in for various reasons.
> * Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
> * Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9273-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 27 18:37:13 2010
Return-Path: <common-issues-return-9273-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 34902 invoked from network); 27 Jun 2010 18:37:12 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jun 2010 18:37:12 -0000
Received: (qmail 81822 invoked by uid 500); 27 Jun 2010 18:37:12 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 81240 invoked by uid 500); 27 Jun 2010 18:37:11 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 80929 invoked by uid 99); 27 Jun 2010 18:37:11 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Jun 2010 18:37:11 +0000
X-ASF-Spam-Status: No, hits=-1547.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Jun 2010 18:37:10 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5RIao9K010427
	for <common-issues@hadoop.apache.org>; Sun, 27 Jun 2010 18:36:50 GMT
Message-ID: <21611623.80411277663810310.JavaMail.jira@thor>
Date: Sun, 27 Jun 2010 14:36:50 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Created: (HADOOP-6844) Update docs to reflect new jar names
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Update docs to reflect new jar names
------------------------------------

                 Key: HADOOP-6844
                 URL: https://issues.apache.org/jira/browse/HADOOP-6844
             Project: Hadoop Common
          Issue Type: Task
          Components: documentation
    Affects Versions: 0.20.3
            Reporter: Eli Collins
             Fix For: 0.20.3, 0.21.0, 0.22.0


HADOOP-6382 changed the way jars are named, the docs need to be updated to reflect this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9274-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 27 18:44:15 2010
Return-Path: <common-issues-return-9274-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 36503 invoked from network); 27 Jun 2010 18:44:14 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jun 2010 18:44:14 -0000
Received: (qmail 89287 invoked by uid 500); 27 Jun 2010 18:44:14 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 89230 invoked by uid 500); 27 Jun 2010 18:44:14 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 89222 invoked by uid 99); 27 Jun 2010 18:44:13 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Jun 2010 18:44:13 +0000
X-ASF-Spam-Status: No, hits=-1547.7 required=10.0
	tests=ALL_TRUSTED,AWL
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Jun 2010 18:44:13 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5RIhrFA010487
	for <common-issues@hadoop.apache.org>; Sun, 27 Jun 2010 18:43:53 GMT
Message-ID: <27273204.80581277664233284.JavaMail.jira@thor>
Date: Sun, 27 Jun 2010 14:43:53 -0400 (EDT)
From: "Eli Collins (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6382) publish hadoop jars to apache mvn
 repo.
In-Reply-To: <1682027337.1258523559848.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/HADOOP-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HADOOP-6382:
--------------------------------

    Hadoop Flags: [Incompatible change, Reviewed]  (was: [Reviewed])
    Release Note: The hadoop jars are renamed  from previous hadoop-<version>-<name>.jar to hadoop-<name>-<version>.jar. Applications and documentation need to be updated to use the new file naming scheme. 

* Marking as an incompatible change since this patch changed how the hadoop jars are named.
* Adding a release note.
* I filed HADOOP-6844 to update the docs to reflect this change.



> publish hadoop jars to apache mvn repo.
> ---------------------------------------
>
>                 Key: HADOOP-6382
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6382
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: build
>    Affects Versions: 0.20.0
>            Reporter: Giridharan Kesavan
>            Assignee: Giridharan Kesavan
>             Fix For: 0.20.3
>
>         Attachments: h20-v3tov5.patch, HADOOP-6382-findbugs.patch, HADOOP-6382-findbugs.patch, hadoop-6382-v1.patch, hadoop-6382-v2.patch, hadoop-6382-v3.patch, hadoop-6382-v4.patch, hadoop-6382-v5.patch, hadoop-6382.patch, HADOOP-6382.script.patch
>
>
> use maven ant task to publish hadoop 20 jars to the apache maven repo

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9275-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Jun 27 23:53:19 2010
Return-Path: <common-issues-return-9275-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 20280 invoked from network); 27 Jun 2010 23:53:19 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 27 Jun 2010 23:53:19 -0000
Received: (qmail 27608 invoked by uid 500); 27 Jun 2010 23:53:18 -0000
Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org
Received: (qmail 27566 invoked by uid 500); 27 Jun 2010 23:53:18 -0000
Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-issues@hadoop.apache.org>
List-Id: <common-issues.hadoop.apache.org>
Reply-To: common-issues@hadoop.apache.org
Delivered-To: mailing list common-issues@hadoop.apache.org
Received: (qmail 27555 invoked by uid 99); 27 Jun 2010 23:53:18 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Jun 2010 23:53:18 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 27 Jun 2010 23:53:15 +0000
Received: from thor (localhost [127.0.0.1])
	by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o5RNqrS4011671
	for <common-issues@hadoop.apache.org>; Sun, 27 Jun 2010 23:52:54 GMT
Message-ID: <7464315.81831277682773799.JavaMail.jira@thor>
Date: Sun, 27 Jun 2010 19:52:53 -0400 (EDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-6800) Harmonize JAR library versions
In-Reply-To: <8215141.145191275515876547.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-6800:
------------------------------

    Attachment: tar-munge

Hemanth, thanks for your persistence - I think we're getting close.

Is the ivy JAR repetition because there is a copy of both in the hdfs/ivy directory? It's not a runtime dependency, so it's probably not a problem anyway.

The JSP one is because of the directory checked into hdfs/lib, which I think can be deleted now.

Finally, here's the latest tar-munge I've been using, which doesn't suffer the problem you describe.

> Harmonize JAR library versions
> ------------------------------
>
>                 Key: HADOOP-6800
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6800
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Tom White
>            Assignee: Tom White
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: build.sh, HADOOP-6800.patch, HADOOP-6800.patch, HADOOP-6800.patch, ivy-declarations.dump.merged, jars-in-lib.dump.merged, tar-munge
>
>
> Currently, multiple versions of the same library JAR are being pulled in for various reasons.
> * Within the same project, multiple versions of the same JAR are pulled in. E.g. Avro (used by Common) depends on Commons Lang 2.5 while Common depends on Commons Lang 2.4.
> * Dependent subprojects use different versions. E.g. Common depends on Avro 1.3.2 while MapReduce depends on 1.3.0. Since MapReduce depends on Common, this has the potential to cause a problem at runtime.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


From common-issues-return-9276-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jun 28 10:17:18 2010
Return-Path: <common-issues-return-9276-apmail-hadoop-common-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org
Received: (qmail 80764 invoked from network); 28 Jun 2010 10:17:18 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)